Wednesday, 2015-03-18

« Tuesday, 2015-03-17 Index Thursday, 2015-03-19 »
*** baoli_ has quit IRC00:00
*** ashleighfarnham has quit IRC00:00
*** baoli has joined #openstack-infra00:01
*** Longgeek has joined #openstack-infra00:01
*** dannywilson has quit IRC00:01
*** yamamoto has quit IRC00:01
*** crc32 has quit IRC00:01
*** dboik has quit IRC00:02
*** ghostpl_ has joined #openstack-infra00:02
*** dboik_ has quit IRC00:02
*** ddieterl_ has joined #openstack-infra00:03
*** prad has quit IRC00:05
*** baoli has quit IRC00:05
*** bhunter71 has joined #openstack-infra00:06
*** ddieterly has quit IRC00:06
*** hichihara has joined #openstack-infra00:07
*** ghostpl_ has quit IRC00:08
*** baoli has joined #openstack-infra00:09
*** Somay has quit IRC00:09
*** banix has quit IRC00:12
anteayafungi: that is a good reason00:12
anteayaclarkb: thanks I did00:12
*** crc32 has joined #openstack-infra00:12
*** marun has joined #openstack-infra00:13
*** Somay has joined #openstack-infra00:13
anteayamtreinish: +2 on https://review.openstack.org/#/c/164451/ you just need one more project-config core review00:14
*** baoli has quit IRC00:14
*** markvoelker has quit IRC00:15
*** achanda has quit IRC00:15
mtreinishanteaya: thanks00:16
*** dangers is now known as dangers_away00:16
*** dangers_away is now known as dangers00:16
*** achanda has joined #openstack-infra00:16
dougwigjeblair: is it a known issue for gertty to lock up for 5-15 seconds constantly?00:17
*** oomichi has quit IRC00:17
*** baoli has joined #openstack-infra00:18
*** achanda has quit IRC00:18
*** achanda has joined #openstack-infra00:18
*** Longgeek has quit IRC00:18
*** sputnik13 has quit IRC00:23
anteayamtreinish: welcome, good hunting00:23
*** bhuvan has quit IRC00:27
*** dmorita has quit IRC00:29
*** mtanino has quit IRC00:30
*** achanda has quit IRC00:30
mmedvededougwig: I thought it was a consequence of not having a separate 'thread' for ui updates while gertty is updating its db00:31
*** markvoelker has joined #openstack-infra00:32
mmedvedeI might be wrong, but I do experience the issue on initial startup00:32
anteayadougwig mmedvede how many repos do you sync with?00:32
dougwiganteaya: standby, that's about 4 clicks away, which will be a few minutes.  :-)00:33
*** yamamoto has joined #openstack-infra00:33
mmedvedeanteaya: I have about 400:33
mmedvedeIt does not hang  for 15 seconds for me though, just in bursts of 2-3 seconds00:33
dougwiganteaya: 7, including all the neutron repos.  Sync is *never* 0, even when I leave it running overnight.  the latest version is basically unusable, which is a bummer.00:33
anteaya:(00:34
blitzrageI got pinged like a boss!00:34
anteayaso looking at https://storyboard.openstack.org/#!/project/69800:34
anteayaI'm not seeing a title that exactly matches what you describe00:35
anteayadougwig: do you run it in debug mode? have you logs?00:35
jeblairdougwig: that seems like a long time for it to be non-responsive.  if you can find out what's going on during that time, it might be very helpful.00:35
*** dboik has joined #openstack-infra00:35
*** Somay has quit IRC00:35
*** mfink_ has quit IRC00:35
*** nelsnelson has quit IRC00:35
jeblairdougwig: and yeah, as anteaya says, running with "-d" might help figure out what it's stuck on00:35
dougwighow to enable debug mode/any suggestions to narrow that down for you?00:35
dougwigok, standby.00:35
anteayagertty -d00:36
*** hdd has quit IRC00:36
*** dboik_ has joined #openstack-infra00:36
jeblairdougwig: just fyi, i don't actually have a lot of time right now; i need to afk soon; but i will be happy to work with you interactively later if we need to pick it up00:36
dougwigjeblair: no urgency here.  whenever works is cool.00:37
dougwigjeblair: thanks.00:37
jeblairdougwig: also i believe tchaypo has reported that sometimes his sync counter never gets to zero; we haven't tracked that down yet either.  not sure if it's related to something being slow like you're looking into.00:37
*** dalgaaf has quit IRC00:37
tchaypowith the latest set of patches I don’t see that any more00:38
dougwigjeblair: constant gerrit traffic, and these:00:38
dougwighttps://www.irccloud.com/pastebin/tTvfQV0V00:38
tchaypoit seemed as though it would periodically decide “hey, let’s go and scan all the things!”, and that would add jobs onto the queue that seemed to duplicate the jobs already in queue00:38
dougwigtchaypo: pip version or master?00:38
tchayponeither00:38
*** dangers is now known as dangers_away00:38
dougwigtchaypo: ooh, tell me what to grab!  :-)00:38
*** dangers_away is now known as dangers00:39
tchaypohttps://www.irccloud.com/pastebin/PMwzB1qE/repo_refs.yaml00:39
jeblairdougwig: what was the previous "Run:" line before "2015-03-17 18:37:16,882 Database lock held 19.9717209339 seconds" ?00:39
*** dboik has quit IRC00:40
dougwighttps://www.irccloud.com/pastebin/jExPW2cF00:40
dougwigjeblair: ^^00:41
dougwiglooks like my pastebin got over-zealous00:41
tchaypoI’m using prep_source_repos from https://review.openstack.org/#/c/137958/ to parse that yaml and apply the patches00:41
dougwiganother fun set of lag just now, too:00:41
dougwighttps://www.irccloud.com/pastebin/r6bdvFLw00:41
dougwigand before you ask:00:42
dougwighttps://www.irccloud.com/pastebin/QpgEaBN800:42
tchaypooh, good news00:42
tchaypoall 4 of those patches have hit master, so I’m using master.00:42
dougwigahh, ok, i'll switch from pip to master.00:43
*** banix has joined #openstack-infra00:43
tchaypoOne of those changes, if I understand it correctly, catches things-to-be-queued that are already queued and doesn’t queue them again. I *think* that made a difference to me.00:44
tchaypoThe db locking change seems to have helped as well; it seems as though my bottleneck now is waiting for HTTP responses from review.openstack.org00:44
jeblairdougwig: yes, i think there are performance improvements that would have helped with the particular change you pointed me at00:45
dougwiginteresting. do i need to trash my db?00:45
dougwigalembic.util.CommandError: No such revision or branch '4a802b741d2f'00:45
dougwiguninstalled pip, used setup.py to install master.  got that on attempting to start.00:45
jeblairdougwig: no, this is a pip problem00:46
jeblairdougwig: "rm -fr gertty.egg-info"00:46
jeblairdougwig: "pip install ."00:46
*** rfolco has joined #openstack-infra00:46
jeblairdougwig: (or if you want to continue to track git more closely, you could "pip install -e ." then you don't have to install again)00:46
tchaypothat’s what I do00:47
tchaypoin a virtualenv00:47
dougwig-e all the way!00:47
dougwigok, let me let this new version crank itself down to 0 and let's see how it goes.00:48
dougwigthank you, jeblair, tchaypo, and anteaya!00:48
anteayasure, thanks for speaking up00:48
jeblairdougwig: cool, thanks.  and yeah, if you see it continuing to be slow, you now know the process for figuring out what's going on at the time.  tchaypo helped me find some really obvious missing optimizations.  :)00:49
dougwigi'll keep it running in debug and keep my eye on it.00:50
*** melwitt has quit IRC00:51
mtreinishfungi: any chance the migration might be finished? Doing a select on the test_runs table shows the new columns which is promising at least00:52
fungimtreinish: it's still not returned control to the calling terminal, for whatever that's worth00:52
mtreinishoh, I can just check the version table to see if it's updated that will let me know00:52
*** ChuckC has quit IRC00:53
tchaypowell this is fun.00:53
*** dmorita has joined #openstack-infra00:53
tchaypoI realised I probably didn’t have the latest version of some of the new dependencies, such as the new requests00:53
tchayposo I did “pip install -e .” again, and it did upgrade some things00:53
dougwiguh oh.00:53
mtreinishfungi: yeah, it's still showing the old version, I bet it just finished the big INSERT and is now re-adding the foreign key constraint to the metadata table00:54
fungimtreinish: sounds likely00:54
mtreinishwhich is probably pretty slow too00:54
tchayposo now I get this plastered on my screen:00:54
fungii'll keep an eyeball on it00:54
tchaypo /Users/james/src/stackforge/gertty/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see00:54
tchaypohttps://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.00:54
*** Ryan_Lane has joined #openstack-infra00:54
mtreinishmordred and I definitely win the award for writing super fast migrations :)00:54
fungitchaypo: oh, yep that's urllib3 trying to be a pest00:54
*** Qiming has joined #openstack-infra00:54
* tchaypo loves00:54
fungitchaypo: you can manually downgrade it00:55
* tchaypo sighs00:55
jeblairtchaypo: or you can apply 16477000:55
*** asettle has quit IRC00:55
fungior that, yes. i should review it00:55
*** Qiming_ has joined #openstack-infra00:55
*** baoli has quit IRC00:55
jeblairyeah, would love input on that, and feedback from anyone who tries it00:55
lifelessjeblair: whats the pip problem ?00:56
lifelessjeblair: (is there a bug open for it?)00:56
tchaypoexcelt that urllib3 in this case is the vendored version inside requests00:56
*** banix has quit IRC00:57
tchaypojeblair: ooh, tasty00:57
*** ghostpl_ has joined #openstack-infra00:58
*** spzala has joined #openstack-infra00:58
jeblairlifeless: mordred may be able to answer that00:58
tchaypo"You will need libssl-dev, libffi-dev, and python-dev installed on debuntu systems and their equivalents elsewhere for this to work."00:59
* tchaypo looks forward to discovering the equivlents on os x00:59
fungiapparently `pip install requests[security]` is supposed to pull in the py2.x<2.7.9 security extensions for requests?00:59
*** Longgeek has joined #openstack-infra00:59
dstufftfungi: yea00:59
fungisigmavirus24_awa is apparently away00:59
*** Qiming has quit IRC00:59
*** Qiming_ is now known as Qiming00:59
fungithat is marvellously opaque01:00
tchaypoactually it looks like I already have the equivalents installed, which is good.01:00
fungiand without it, urllib3 spam abounds01:00
dstufftWithout 2.7.9+ or 3.2+ urllib3 can't fiddle a number of knobs it needs to fiddle to ensure a secure connection01:00
dstufftit shouldn't spam though I don't believe, I think it should only do it once per python invocation01:01
anteayathat is the same issue reed was seeing earlier01:01
fungiyeah on initial import01:01
lifelessdstufft: wow, I am now wanting to stuff distributed testr testing onto pip, this is slooooow01:01
lifelessdstufft: but I shall restrain myself :)01:01
dstufftlifeless: it has concurrent testing if you enable it01:01
fungijust wondering what requests might want to do about the vendored urllib3 since requests does its own certificate validation i think?01:01
dstuffttox -e py34 -- -n 801:01
lifelessdstufft: distributed?01:01
dstufftit'll distribute it amongst N processes01:01
dstufftyou can distribute it amongst other computers if you want too I think, I never tried to do that though01:02
lifelessdstufft: e.g. I have 12 cores, 24 hardware threads across three machines01:02
clarkbrequests spams tge warning01:02
clarkbif you ^L gerrty it comes back on next sync01:02
dstufftlifeless: https://pypi.python.org/pypi/pytest-xdist01:02
clarkbits a pretty terrible user and dev experience all around01:02
clarkb"python is broken so let me annoy you until you write rust"01:03
*** ghostpl_ has quit IRC01:03
* fungi gets busy learning rust in preparation for the coming apocalypse01:03
lifelessrust is fun01:03
lifelessbut I think you need a fairly narrow problem set for it to be the right tradeoff01:04
*** salv-orl_ has quit IRC01:04
dstufftclarkb: if it's spamming that sounds like a bug fwiw01:04
dstufftunless someone's turned on always for python warnings01:04
*** dangers is now known as dangers_away01:05
fungii'm still not understanding how that error message is of any use at all to an end user01:05
clarkbfungi it isnt01:05
tchaypodeprecation warnings considered harmful01:06
dstufftyou can install pyopenssl ndg-https-client and pyasn1 to make it go away01:06
dstufftor upgrade your Python01:06
clarkbdstufft yes thats my change01:06
clarkbsee 16477001:06
*** garyh has quit IRC01:07
dstufftalternatively shit just breaks like when requests 2.5.0 was released because people's Pythons are too old to enable certain OpenSSL features (that requests had to roll back because it broke too much)01:07
fungiwell, in this case it's saying that any version of python >3 months old is deprecated so go bug your distribution to change their release model01:07
lifelessfungi: these warnings are off by default01:07
lifelessfungi: whats turned it on?01:07
clarkbfwiw the inoirt exception catch is how urllib3 suggests you deal with this01:08
dstufftnot exactly 3 months old, there's no warning on 3.2+ either01:08
dstufftwhich is.. 4 years old now?01:08
fungifair enough. any python 3.x release >3 months old01:08
fungier, 2.x release01:08
clarkb* import exception01:08
* fungi sighs at beerfingers01:08
dstufftyou can also just flat out disable the warnings via the API that urllib3 provides or the API that python provides standard01:08
fungiwell, in this case it's urllib3 as provided by requests vendoring. probably still tweakable though01:09
lifelessfungi: still - what is enabling deprecation warnings in gertty?01:09
lifelessfungi: since Python defaults them *off*01:09
clarkbdstufft I am not sure what you mean about warnings not spamming by default but nothing in pythob logging prevents that aiui01:10
fungii'm testing to see if it's being correctly raised as a deprecation warning01:10
clarkblifeless its urllib301:10
clarkblifeless saying "upgrade your python kthxbai"01:10
clarkbits not a deprecation warning01:10
lifelessoh01:10
lifelessits an error?01:11
dstufftit's not an error01:11
clarkbno its a warning01:11
dstufftthere are warnings other htan deprecation warnings01:11
dstufftclarkb: the default python warning action is "once per location"01:11
clarkbbut not a deprexation warning its an insecurepythonwarning01:11
dstufftthe warning only exists in one location, and there's no stacklevel argument01:11
dstufftso it should only show up once per Python process01:11
fungiInsecurePlatformWarning but yeah01:12
clarkbdstufft its a logging.warn() thing iirc01:12
clarkbwhich will just spam01:12
dstufftit's warning.warn01:12
dstuffthttps://github.com/shazow/urllib3/blob/master/urllib3/util/ssl_.py#L74-L8001:12
* dstufft reviewed the code01:12
clarkbdstufft is it in requests?01:12
clarkbremember thats not the source fwiw01:13
dstuffthttps://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/util/ssl_.py#L73-L7901:13
dstufftI reviewed the code when it was originally commited I mean :)01:13
dstufftso something is causing it to be shown more than once, either gerrtty is spawning multiple processes (or is otherwise resetting that state), or you have warnings set to always instead of default, or something is causing Python to not see each call as the same location01:14
clarkbor requests/urllib is doing those things01:15
*** camunoz has quit IRC01:17
dstuffthm01:17
dstufftyea01:17
* dstufft looks01:17
dstufftah01:17
dstufftyup01:17
dstuffthttps://github.com/kennethreitz/requests/blob/9adaae6785a3526319e6aabab5ed551062636970/requests/packages/urllib3/__init__.py#L58-L6001:17
dstufftthat probably wasn't intended for this particular warning01:18
dstufftthat was just an unintended consequence of using a SecurityWarning sublcass01:18
*** yamamoto has quit IRC01:19
*** Qiming_ has joined #openstack-infra01:20
*** Qiming has quit IRC01:23
dstufftclarkb: https://github.com/shazow/urllib3/issues/57401:23
clarkbdstufft thank you01:24
jeblaironce is still probably too many times for gertty01:24
dstufftthat may be true, which is why there is a bunch of different ways to silence it :)01:26
jeblairyeah, not intending to complain;  i'm happy that we can solve the spamming problem, and i think we need to do something to silence it altogether.01:27
*** MarkAtwood has joined #openstack-infra01:28
*** camunoz has joined #openstack-infra01:29
*** otter768 has joined #openstack-infra01:30
*** otter768 has quit IRC01:35
*** yamamoto has joined #openstack-infra01:42
*** tsg has joined #openstack-infra01:42
*** banix has joined #openstack-infra01:43
*** hdd has joined #openstack-infra01:45
*** adalbas has quit IRC01:45
*** stevemar has joined #openstack-infra01:47
*** Ryan_Lane has quit IRC01:48
*** Darkwan has quit IRC01:50
*** mfink_ has joined #openstack-infra01:54
fungiand that something is not convincing the world to upgrade to a new python interpreter01:55
*** ivar-laz_ has joined #openstack-infra01:55
funginot sure whether `pip install requests[security]` is necessarily a viable workaround (what will that do on later python?)01:56
*** bhunter71 has quit IRC01:56
*** ivar-lazzaro has quit IRC01:59
*** dims has quit IRC01:59
*** ivar-laz_ has quit IRC01:59
*** dhritishikhar has joined #openstack-infra02:02
*** macjack has joined #openstack-infra02:02
*** tiswanso has joined #openstack-infra02:03
*** bhuvan has joined #openstack-infra02:04
dstufftfungi: on python 2.7.9 it'll be fine02:04
dstufftI forget if that works on 3.x02:04
dstufftyea02:06
dstufftit works on 3.x02:06
*** tiswanso has quit IRC02:06
*** harlowja_ is now known as harlowja_away02:06
*** tiswanso has joined #openstack-infra02:06
*** garyh has joined #openstack-infra02:07
*** yamahata has quit IRC02:09
dstufftReminder that on Python 2.6 you're likely vulnerable to POODLE, probably using RC4, possibly export grade ciphers etc because of the ability that Python 2.6 has, in addition to the problems on Python <2.7.9 which are uh, lack of ability to disable TLS Compression (CRIME attack), lack of ability to disable SSLv3 (POODLE), lack of ability to tell OpeNSSL to use the shortest certificate path making a lot of sites require 1024 bit (and02:09
dstufft weak) roots when they could otherwise use 2048+ bit roots02:09
fungimaybe updating the requests line in gertty's requirements.txt with that will provide a viable workaround. testing in a sec02:09
clarkbwell sslv3 is disabled on the python 2.7s we interact with02:10
dstufftI think Python 2.6 had SSLv2 enabled still too02:10
clarkbso there is some ability to do these things02:11
openstackgerritQiming Teng proposed openstack-infra/project-config: new-project: stackforge/python-senlinclient  https://review.openstack.org/16496302:11
*** crc32 has quit IRC02:11
dstufftclarkb: if your system patched out SSLv3 you can do that yea02:11
*** hdd has quit IRC02:12
openstackgerritJesse DeFer proposed openstack-infra/jenkins-job-builder: Add 'Job Log Logger' wrapper  https://review.openstack.org/16433802:12
*** hdd has joined #openstack-infra02:12
fungiyeah, changing requirements.txt to requests[security]!=2.5.0 also silences that warning for me02:16
*** Qiming_ is now known as Qiming02:17
*** macjack has quit IRC02:17
*** salv-orlando has joined #openstack-infra02:18
clarkbthats probably better than my patch then02:18
*** unicell has quit IRC02:19
*** coolsvap|afk is now known as coolsvap02:19
*** camunoz has quit IRC02:20
*** crc32 has joined #openstack-infra02:20
*** erikmwilson is now known as Guest1700302:21
*** sdake__ has joined #openstack-infra02:21
*** bhunter71 has joined #openstack-infra02:22
*** asettle has joined #openstack-infra02:23
*** garyh has quit IRC02:23
*** sdake has quit IRC02:25
*** marun has quit IRC02:25
*** ayoung has quit IRC02:27
fungii don't know that it's necessarily better. it does add a heavier dependency chain dragging in cffi, cryptography, enum34, ndg-httpsclient, pyOpenSSL, pyasn1 and pycparser for your trouble02:28
fungisome of which have install-time compilation against c headers02:28
*** garyh has joined #openstack-infra02:29
*** hdd has quit IRC02:32
*** camunoz has joined #openstack-infra02:32
fungioh, wait you were installing pretty much all of that too (either directly or transitively)02:33
*** coolsvap is now known as coolsvap|afk02:33
*** kaisers1 has joined #openstack-infra02:34
*** kaisers has quit IRC02:34
dstufftthe two changes are equivilant02:36
dstufftthe importing 164770 does shouldn't be needed, requests does that on it's own02:36
fungicoolies02:37
*** otter768 has joined #openstack-infra02:37
*** baoli has joined #openstack-infra02:38
fungianyway, i'm going to make like a tree and get some sleep. or something02:38
fungiadios02:38
*** coolsvap|afk is now known as coolsvap02:45
*** garyh has quit IRC02:45
*** ghostpl_ has joined #openstack-infra02:47
*** melwitt has joined #openstack-infra02:52
*** ghostpl_ has quit IRC02:53
*** tkelsey has joined #openstack-infra02:56
*** tkelsey has quit IRC03:01
*** baoli has quit IRC03:01
*** baoli has joined #openstack-infra03:01
*** MarkAtwood has quit IRC03:02
*** marun has joined #openstack-infra03:06
*** Svedrin has quit IRC03:08
*** coolsvap is now known as coolsvap|afk03:09
*** Svedrin has joined #openstack-infra03:11
*** dboik_ has quit IRC03:13
*** tiswanso has quit IRC03:13
*** gyee has quit IRC03:15
*** bhunter71 has quit IRC03:16
*** coolsvap|afk is now known as coolsvap03:16
*** changbl has joined #openstack-infra03:20
*** banix has quit IRC03:25
*** dhritishikhar has quit IRC03:26
*** crc32 has quit IRC03:27
*** crc32 has joined #openstack-infra03:29
*** salv-orlando has quit IRC03:29
*** MarkAtwood has joined #openstack-infra03:29
*** Longgeek has quit IRC03:29
*** hdd has joined #openstack-infra03:33
*** ddieterl_ has quit IRC03:39
*** sdake__ has quit IRC03:39
*** sdake has joined #openstack-infra03:41
*** sarob has joined #openstack-infra03:41
*** tonytan4ever has quit IRC03:41
*** garyh has joined #openstack-infra03:46
*** melwitt has quit IRC03:47
*** sarob has quit IRC03:47
*** rfolco has quit IRC03:53
*** Somay has joined #openstack-infra03:55
*** garyh has quit IRC03:56
*** dmorita has quit IRC03:57
*** baoli has quit IRC03:58
*** baoli has joined #openstack-infra03:59
*** otter768 has quit IRC04:04
*** SumitNaiksatam has joined #openstack-infra04:04
*** ajmiller has joined #openstack-infra04:09
*** SumitNaiksatam has quit IRC04:11
*** SumitNaiksatam has joined #openstack-infra04:17
*** tqtran has quit IRC04:27
*** patrickeast has quit IRC04:28
*** salv-orlando has joined #openstack-infra04:28
*** yfried|afk is now known as yfried_04:30
*** Longgeek has joined #openstack-infra04:30
*** tsg has quit IRC04:30
*** koolhead17 has quit IRC04:37
*** ghostpl_ has joined #openstack-infra04:37
*** Longgeek has quit IRC04:37
*** Longgeek has joined #openstack-infra04:38
*** hdd has quit IRC04:38
*** ddieterly has joined #openstack-infra04:39
*** baoli has quit IRC04:42
*** ddieterly has quit IRC04:44
*** ghostpl_ has quit IRC04:44
*** ajmiller has quit IRC04:46
*** ajmiller has joined #openstack-infra04:46
*** adrian_otto has joined #openstack-infra04:47
*** boris-42 has joined #openstack-infra04:47
*** marun has quit IRC04:48
*** dims has joined #openstack-infra04:52
*** garyh has joined #openstack-infra04:57
*** dims has quit IRC04:57
*** tkelsey has joined #openstack-infra04:57
*** stevemar has quit IRC04:58
*** stevemar has joined #openstack-infra04:59
*** tkelsey has quit IRC05:02
*** achanda has joined #openstack-infra05:06
*** garyh has quit IRC05:07
*** marcusvrn has joined #openstack-infra05:10
*** stevemar has quit IRC05:11
*** marcusvrn1 has quit IRC05:12
*** stevemar has joined #openstack-infra05:12
*** sdake_ has quit IRC05:13
*** sdake has quit IRC05:13
*** asettle has quit IRC05:13
*** sputnik13 has joined #openstack-infra05:15
*** BharatK has joined #openstack-infra05:16
*** unicell has joined #openstack-infra05:20
*** sputnik13 has quit IRC05:23
rluciobharatk: hey - i finally got through those issues we were talking about yesterday.  I upped the FD limits and altered my jenkins jobs to disable swift and ceilometer, among other things05:24
BharatKrlucio: Great.05:24
BharatKrlucio: Is that working fine now?05:25
*** sputnik13 has joined #openstack-infra05:25
rluciobharatk: so now ive had my first full test runs passing, yea.. but still a bug remaining05:25
rluciobharatk: i can only run the jenkins job once on the slave, and then all subsequent runs fail with resource errors05:26
rluciobharatk: since these are dumb (physical) slaves and not VMs, I need to figure out how to clean up after the test run05:26
BharatKrlucio: For every run, we need to create new slave VM right?05:26
rluciobharatk: but im not launching vms05:27
BharatKrlucio: Oh... In jenkins this is the case.05:27
BharatKrlucio: I mean upstream jenkins05:27
rluciobharatk: right, at least thats the logical guess05:28
*** deepakcs has joined #openstack-infra05:28
rluciobharatk: i tried enabling a flag to unstack at the beginning of each run, but that doesn't appear to work, more investigation needed05:28
BharatKrlucio: Anyways I think you need create your own cleanup script, according to your deployment.05:29
BharatKrlucio: what are the failures?05:29
rluciobharatk: yea that's what i figured.  I tried this flag "DEVSTACK_GATE_UNSTACK" but that doesn't appear to work05:29
rluciobharatk: setting the flag i mean05:30
BharatKrlucio: Yeah as per https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L547 that should work.05:30
rluciobharatk: the errors are basically the same resource errors we saw yesterday... I think its because all of the openstack services from the previous test run are still running on the box when the new job launches05:31
*** salv-orlando has quit IRC05:31
BharatKrlucio: Oh cool. When screens (openstack services) are running, devstack will not allow to run stack.sh (install) again.05:32
BharatKrlucio: From the logs of second run, we can figureout this issue.05:32
*** markvoelker has quit IRC05:33
*** MarkAtwood has quit IRC05:34
rluciobharatk: well.. here is what the devstacklog.txt says: http://paste.openstack.org/show/193031/05:35
BharatKrlucio: Oh again :-(05:36
BharatKrlucio: It's really to difficult to estimate the things in second run on the same slave.05:36
rluciobharatk: lol yea.. i already told my manager we are going to need some beefier servers for this long-term05:36
BharatKrlucio: lol...05:37
BharatKrlucio: I think you have access to that slave machine I guess.05:37
BharatKrlucio: directly login and check openstack services are running or not.05:37
rluciobharatk: oh i already know they are, i watch output of 'top' while running the test05:38
BharatKrlucio: oh, in output of top showing openstack services right?05:39
rluciobharatk: right05:39
rluciobharatk: they are all still running after the test ends05:39
rluciobharatk: i had originally assumed the host cleanup script shut them down, but all it does is gather the logs05:40
*** ddieterly has joined #openstack-infra05:40
BharatKrlucio: Can you run "./unstack.sh" manually on the slave node. And rerun the job on the same node.05:40
*** ghostpl_ has joined #openstack-infra05:41
rluciobharatk: yep, we're on the same page.  im going to try that right now...05:41
BharatKrlucio: What does DEVSTACK_GATE_UNSTACK do?05:43
BharatKrlucio: https://review.openstack.org/#/c/117279/05:43
*** sarob has joined #openstack-infra05:43
rluciobharatk: the comment above the flag says it runs unstack.sh immediately after checking out devstack05:44
*** armax has quit IRC05:44
*** subscope_ has joined #openstack-infra05:44
*** ddieterly has quit IRC05:45
*** ghostpl_ has quit IRC05:45
BharatKrlucio: If it is first run what will be the case if this flag set to 1?05:47
rluciobharatk: oh.. yea that's not what i want at all, it does the unstack after setup, but before tempest runs05:47
*** Sukhdev has joined #openstack-infra05:47
rluciobharatk: not sure, we'll see in a minute though05:48
*** sarob has quit IRC05:49
*** __afazekas is now known as afazekas05:52
*** afazekas is now known as __afazekas05:53
*** Somay has quit IRC05:53
*** otter768 has joined #openstack-infra06:05
*** garyh has joined #openstack-infra06:07
*** otter768 has quit IRC06:09
*** garyh has quit IRC06:18
*** achanda has quit IRC06:25
*** achanda has joined #openstack-infra06:27
rluciobharatk: cool, looks like running unstack.sh at the end of the test run works06:29
rluciobharatk: yay for a simple fix06:29
BharatKrlucio: cool.06:29
*** _nadya_ has joined #openstack-infra06:30
BharatKrlucio: great. You are running CI in your local lab right?06:30
BharatKrlucio: I mean not in the upstream.06:30
*** sputnik13 has quit IRC06:31
*** liusheng has quit IRC06:31
rluciobharatk: yea06:31
*** liusheng has joined #openstack-infra06:31
rluciobharatk: for cinder volume driver CI requirements06:31
BharatKrlucio: earlier I planned to deploy CI in local, but I don't know the way, so created an upstream job06:32
BharatKrlucio: To meet the deadlines.06:32
BharatKrlucio: Can you share your experience in create CI locally.06:32
rluciobharatk: yea i can probably write something up on it06:32
BharatKrlucio: cool thanks,06:33
rluciobharatk: how did you create an upstream job?  don't they need your hardware to test?06:33
*** salv-orlando has joined #openstack-infra06:33
*** markvoelker has joined #openstack-infra06:33
BharatKrlucio: No that runs on Infra hardware. Installed and configured GlusterFS through devstack06:34
BharatKrlucio: and running tempest on top of that.06:34
rluciobharatk: oh, cool06:35
BharatKrlucio: But in future we need to maintain CI locally.06:35
rluciobharatk: i work for violin memory so ive got a couple of flash arrays in the lab to test with06:36
BharatKrlucio: in that case you can't go for upstream CI I guess :(06:38
*** markvoelker has quit IRC06:38
BharatKrlucio: Anyways now your CI is running fine right... So no problem with CI Deadline :D06:38
rluciobharatk: heh i sure hope!06:39
BharatKrlucio: Cool :D06:40
*** ddieterly has joined #openstack-infra06:41
rluciobharatk: thanks again for all of your help btw06:43
BharatKrlucio: I did nothing. But learned few things from you. Thank you :D06:44
*** sarob has joined #openstack-infra06:45
*** ddieterly has quit IRC06:46
rluciobharatk: :)06:46
*** afazekas has joined #openstack-infra06:47
*** sarob has quit IRC06:50
*** subscope_ has quit IRC06:56
*** camunoz is now known as camunoz_gone06:57
*** mestery has quit IRC06:58
*** Sukhdev has quit IRC07:03
*** mfink__ has joined #openstack-infra07:04
*** mfink_ has quit IRC07:04
*** Trozz has joined #openstack-infra07:05
*** adrian_otto has quit IRC07:05
openstackgerritMasaki Matsushita proposed openstack-dev/pbr: Add truncated lincense text  https://review.openstack.org/16531907:10
*** ghostpl_ has joined #openstack-infra07:12
*** hogepodge has quit IRC07:16
*** markvoelker has joined #openstack-infra07:17
*** garyh has joined #openstack-infra07:18
*** mpavone has joined #openstack-infra07:20
*** mrunge has joined #openstack-infra07:20
*** markvoelker has quit IRC07:21
*** scheuran has joined #openstack-infra07:24
*** garyh has quit IRC07:29
*** mfink__ has quit IRC07:33
*** salv-orlando has quit IRC07:33
*** tnovacik has joined #openstack-infra07:34
*** achanda has quit IRC07:35
*** ildikov has quit IRC07:35
*** achanda has joined #openstack-infra07:39
*** Ala has joined #openstack-infra07:42
*** ddieterly has joined #openstack-infra07:42
*** Ala has joined #openstack-infra07:42
*** salv-orlando has joined #openstack-infra07:42
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Update pip-missing-reqs to pip-check-reqs  https://review.openstack.org/16497407:43
*** LinuxJed_ is now known as LinuxJedi07:44
*** sarob has joined #openstack-infra07:46
*** ddieterly has quit IRC07:47
*** rlandy has joined #openstack-infra07:49
*** stevemar has quit IRC07:50
*** sarob has quit IRC07:51
*** jcoufal has joined #openstack-infra07:55
*** doude has joined #openstack-infra07:58
*** Bsony has joined #openstack-infra07:58
*** spredzy|afk is now known as spredzy07:58
*** Somay has joined #openstack-infra07:58
*** HeOS has quit IRC07:59
*** woodster_ has quit IRC08:00
*** jlanoux has joined #openstack-infra08:03
*** shardy has joined #openstack-infra08:04
*** otter768 has joined #openstack-infra08:06
*** belmoreira has joined #openstack-infra08:07
*** koolhead17 has joined #openstack-infra08:07
*** markus_z has joined #openstack-infra08:09
*** otter768 has quit IRC08:10
*** achanda has quit IRC08:10
*** markvoelker has joined #openstack-infra08:18
*** hyakuhei has joined #openstack-infra08:19
openstackgerritAndrey Pavlov proposed openstack/requirements: Add botocore to requirements  https://review.openstack.org/15438608:22
*** jistr has joined #openstack-infra08:22
*** markvoelker has quit IRC08:23
*** teran has quit IRC08:24
*** tkelsey has joined #openstack-infra08:25
*** hyakuhei has quit IRC08:25
*** hyakuhei has joined #openstack-infra08:25
*** LinuxJedi is now known as LinuxJedi_08:26
*** _nadya_ has quit IRC08:26
*** ildikov has joined #openstack-infra08:28
*** dtantsur|afk is now known as dtantsur08:28
*** Trozz is now known as zzTrozz08:28
*** zzTrozz is now known as Trozz08:28
*** garyh has joined #openstack-infra08:29
*** uschreiber_ has joined #openstack-infra08:31
*** yfried_ has quit IRC08:32
*** uschreiber_ has quit IRC08:33
*** jlanoux has quit IRC08:35
openstackgerritLukas Bednar proposed openstack-infra/jenkins-job-builder: publishers: groovy-postbuild changes between 1.x & 2.x  https://review.openstack.org/13427608:39
*** garyh has quit IRC08:39
*** yfried has joined #openstack-infra08:41
*** ddieterly has joined #openstack-infra08:43
openstackgerritMerged openstack-infra/release-tools: Import release notes script from oslo-incubator  https://review.openstack.org/16126508:46
openstackgerritMerged openstack-infra/release-tools: Import updates to release_notes.py from oslo-incubator  https://review.openstack.org/16518608:46
*** hashar has joined #openstack-infra08:46
*** ddieterly has quit IRC08:47
*** ghostpl_ has quit IRC08:49
GheRiveromorning all08:54
*** hyakuhei has quit IRC09:00
*** ociuhandu has quit IRC09:01
*** Hal has joined #openstack-infra09:07
*** Hal is now known as Guest9618909:08
*** boris-42 has quit IRC09:12
*** LinuxJedi_ is now known as LinuxJedi09:13
*** gampel has joined #openstack-infra09:14
gampelwho09:14
*** notnownikki has joined #openstack-infra09:15
*** andreykurilin_ has joined #openstack-infra09:15
*** hyakuhei has joined #openstack-infra09:18
*** markvoelker has joined #openstack-infra09:19
*** e0ne has joined #openstack-infra09:19
*** Somay has quit IRC09:19
*** teran has joined #openstack-infra09:20
*** bhuvan has quit IRC09:21
*** hogepodge has joined #openstack-infra09:22
*** ghostpl_ has joined #openstack-infra09:23
*** hichihara has quit IRC09:23
*** markvoelker has quit IRC09:23
*** arxcruz has joined #openstack-infra09:26
*** hyakuhei has quit IRC09:28
*** HeOS has joined #openstack-infra09:28
*** afazekas_ has joined #openstack-infra09:31
*** lucasagomes has joined #openstack-infra09:36
*** pblaho has joined #openstack-infra09:36
*** zz_johnthetubagu is now known as johnthetubaguy09:36
*** BobBall_AWOL is now known as BobBall09:37
lucasagomeshi, if you have a free time please take a look at https://review.openstack.org/#/c/163344/ , it's a small patch enabling we run the Ironic IPA deploy ramdisk in the openstack/ironic-python-agent check gate09:37
lucasagomesas non-voting09:37
*** dmellado has quit IRC09:38
*** nilasae has joined #openstack-infra09:39
*** garyh has joined #openstack-infra09:40
GheRiverolucasagomes: on my way09:41
lucasagomesGheRivero, :D hi there09:42
lucasagomesthanks09:42
*** ZZelle has quit IRC09:43
*** dims_ has joined #openstack-infra09:43
*** ZZelle has joined #openstack-infra09:43
*** ddieterly has joined #openstack-infra09:43
*** jlanoux has joined #openstack-infra09:44
*** afazekas_ has quit IRC09:45
*** dizquierdo has joined #openstack-infra09:45
*** alexpilotti has joined #openstack-infra09:47
*** nilasae has quit IRC09:47
*** ddieterly has quit IRC09:48
*** Guest96189 has quit IRC09:50
*** garyh has quit IRC09:51
*** ssam2 has joined #openstack-infra09:52
*** Qiming has quit IRC09:56
*** ominakov has joined #openstack-infra09:59
*** crc32 has quit IRC10:01
*** Somay has joined #openstack-infra10:01
*** otter768 has joined #openstack-infra10:06
*** dizquierdo has quit IRC10:08
*** dtantsur is now known as dtantsur|bbl10:08
*** otter768 has quit IRC10:11
*** LinuxJedi is now known as LinuxJedi_10:12
*** LinuxJedi_ is now known as LinuxJedi10:13
*** e0ne is now known as e0ne_10:15
*** ominakov has quit IRC10:15
*** ominakov has joined #openstack-infra10:18
*** markvoelker has joined #openstack-infra10:19
*** esker has quit IRC10:23
*** markvoelker has quit IRC10:24
*** ihrachyshka has joined #openstack-infra10:24
*** e0ne_ has quit IRC10:26
*** mestery has joined #openstack-infra10:26
*** tnovacik has quit IRC10:30
*** notnownikki has quit IRC10:32
*** notnownikki has joined #openstack-infra10:32
*** andreykurilin_ has quit IRC10:34
*** yfried is now known as yfried|afk10:37
*** e0ne has joined #openstack-infra10:38
*** yfried|afk is now known as yfried10:40
*** crc32 has joined #openstack-infra10:40
*** pelix has joined #openstack-infra10:41
*** unicell has quit IRC10:44
*** ddieterly has joined #openstack-infra10:44
*** gmann_ has joined #openstack-infra10:45
*** ddieterly has quit IRC10:49
*** boris-42 has joined #openstack-infra10:51
*** jyuso1 has quit IRC10:51
*** garyh has joined #openstack-infra10:51
*** Bsony_ has joined #openstack-infra10:54
*** yamamoto has quit IRC10:55
openstackgerritAntoine Musso proposed stackforge/python-jenkins: Add a get_nodes method  https://review.openstack.org/16306810:57
*** Bsony has quit IRC10:57
*** _nadya_ has joined #openstack-infra10:58
*** ghostpl_ has quit IRC10:59
*** ghostpl_ has joined #openstack-infra11:00
*** garyh has quit IRC11:02
*** baoli has joined #openstack-infra11:02
*** dtantsur|bbl is now known as dtantsur11:02
*** sputnik13 has joined #openstack-infra11:02
*** pc_m has joined #openstack-infra11:02
*** doude has quit IRC11:07
*** ghostpl_ has quit IRC11:07
*** ghostpl_ has joined #openstack-infra11:07
*** amotoki has joined #openstack-infra11:10
*** ghostpl_ has quit IRC11:11
*** ghostpl_ has joined #openstack-infra11:12
*** mwagner_lap has quit IRC11:12
*** juggler_ has joined #openstack-infra11:12
*** dizquierdo has joined #openstack-infra11:13
*** juggler has quit IRC11:15
*** hashar is now known as hasharFood11:16
DuncanTCan any of the gerrit guru types give me a query that gives me all the patches I have put +2 on in the past that don't have my +2 now, please? It is one missing off my dash11:18
*** tqtran has joined #openstack-infra11:18
*** tnovacik has joined #openstack-infra11:19
*** markvoelker has joined #openstack-infra11:20
*** baoli has quit IRC11:21
*** baoli has joined #openstack-infra11:21
*** yamamoto has joined #openstack-infra11:21
*** tqtran has quit IRC11:23
*** jp_at_hp has joined #openstack-infra11:24
*** markvoelker has quit IRC11:25
*** mrmartin has joined #openstack-infra11:27
*** cdent has joined #openstack-infra11:29
*** ghostpl_ has quit IRC11:30
*** ghostpl_ has joined #openstack-infra11:31
*** ominakov has quit IRC11:38
*** marcusvrn1 has joined #openstack-infra11:38
*** persia has quit IRC11:39
*** persia_ has joined #openstack-infra11:40
*** persia_ has quit IRC11:40
*** persia_ has joined #openstack-infra11:40
*** marcusvrn has quit IRC11:41
*** e0ne is now known as e0ne_11:41
*** mrda is now known as mrda-away11:42
*** persia_ has quit IRC11:44
*** achanda has joined #openstack-infra11:45
*** ddieterly has joined #openstack-infra11:45
*** persia has joined #openstack-infra11:46
*** smarcet has joined #openstack-infra11:46
*** achanda has quit IRC11:49
*** ddieterly has quit IRC11:49
*** sarob has joined #openstack-infra11:50
*** e0ne_ has quit IRC11:51
*** ldnunes has joined #openstack-infra11:52
*** smarcet has quit IRC11:53
*** erlon has joined #openstack-infra11:53
*** deepakcs has quit IRC11:54
*** dizquierdo has quit IRC11:54
*** sarob has quit IRC11:55
*** dprince has joined #openstack-infra11:56
*** e0ne has joined #openstack-infra11:59
*** dkranz has joined #openstack-infra12:00
*** woodster_ has joined #openstack-infra12:00
*** david-lyle_afk is now known as david-lyle12:00
*** BharatK has quit IRC12:01
*** markvoelker has joined #openstack-infra12:02
*** garyh has joined #openstack-infra12:03
*** mwagner_lap has joined #openstack-infra12:03
*** LinuxJedi is now known as LinuxJedi_12:04
*** mattfarina has joined #openstack-infra12:05
*** dizquierdo has joined #openstack-infra12:06
*** PaulMurray has left #openstack-infra12:07
*** otter768 has joined #openstack-infra12:07
*** mattfarina has quit IRC12:08
*** nilasae has joined #openstack-infra12:09
*** mattfarina has joined #openstack-infra12:09
*** Somay has quit IRC12:10
*** mattfarina has quit IRC12:11
*** tqtran has joined #openstack-infra12:11
*** mattfarina has joined #openstack-infra12:12
*** marun has joined #openstack-infra12:12
*** bswartz has quit IRC12:12
*** otter768 has quit IRC12:13
*** garyh has quit IRC12:13
*** BharatK has joined #openstack-infra12:13
*** eharney has joined #openstack-infra12:17
*** rfolco has joined #openstack-infra12:17
*** marun has quit IRC12:17
*** ociuhandu has joined #openstack-infra12:18
*** bradjones has joined #openstack-infra12:18
*** derekh has joined #openstack-infra12:19
*** esker has joined #openstack-infra12:20
*** kgiusti has joined #openstack-infra12:22
*** dims_ has quit IRC12:22
*** dims_ has joined #openstack-infra12:22
*** sdake has joined #openstack-infra12:23
*** dmellado has joined #openstack-infra12:26
*** ajmiller has quit IRC12:26
*** gmann_ has quit IRC12:29
*** aysyd has joined #openstack-infra12:29
*** LinuxJedi_ is now known as LinuxJedi12:31
*** ghostpl_ has quit IRC12:32
*** notnownikki has quit IRC12:33
openstackgerritPaul Michali proposed openstack-infra/project-config: Make VPNaaS StrongSwan functional gate voting  https://review.openstack.org/16539212:34
anteayaDuncanT: unfortunately not, what you did in the past isn't available for the query just the state of the patch now12:35
DuncanTanteaya: Ok, thanks. I'll see if I can add it to gerrymander instead12:35
anteayawould be great if it was12:35
anteayaokay12:36
*** sdake_ has joined #openstack-infra12:37
anteayawhat is gerrymander12:37
anteayaDuncanT: ^12:38
DuncanTanteaya: Offline gerrit tool - https://github.com/berrange/gerrymander.git12:38
*** lucasagomes is now known as lucas-hungry12:39
DuncanTanteaya: Or at least the tools for building one12:39
*** jistr is now known as jistr|biab12:39
*** sputnik13 has quit IRC12:40
*** unicell has joined #openstack-infra12:41
*** sdake has quit IRC12:41
anteayaah, cool, thanks12:44
*** BharatK has quit IRC12:45
*** ddieterly has joined #openstack-infra12:46
*** notnownikki has joined #openstack-infra12:46
*** ddieterly has quit IRC12:50
*** sarob has joined #openstack-infra12:51
*** marcusvrn1 has quit IRC12:51
*** marcusvrn has joined #openstack-infra12:52
*** tnovacik has quit IRC12:54
*** ddieterly has joined #openstack-infra12:55
*** notnownikki has quit IRC12:55
*** adalbas has joined #openstack-infra12:56
*** bswartz has joined #openstack-infra12:56
*** salv-orlando has quit IRC12:56
*** sarob has quit IRC12:56
*** salv-orlando has joined #openstack-infra12:57
*** dkliban_afk is now known as dkliban12:58
*** BharatK has joined #openstack-infra12:59
*** notnownikki has joined #openstack-infra12:59
*** ddieterly has quit IRC12:59
*** gordc has joined #openstack-infra12:59
*** sputnik13 has joined #openstack-infra13:00
*** hdd has joined #openstack-infra13:00
*** changbl has quit IRC13:00
*** sputnik13 has quit IRC13:01
*** boris-42 has quit IRC13:02
*** gordc has quit IRC13:02
*** pblaho has quit IRC13:02
*** amotoki has quit IRC13:03
*** gordc has joined #openstack-infra13:04
*** hdd has quit IRC13:04
*** spredzy is now known as spredzy|afk13:05
*** amotoki has joined #openstack-infra13:05
*** mattfarina has quit IRC13:05
*** mpaolino has joined #openstack-infra13:06
*** radez_g0n3 is now known as radez13:06
*** ildikov has quit IRC13:06
*** bknudson has quit IRC13:06
*** banix has joined #openstack-infra13:06
*** tnovacik has joined #openstack-infra13:07
*** banix has quit IRC13:07
*** mpaolino has quit IRC13:07
*** mpaolino has joined #openstack-infra13:08
*** eharney has quit IRC13:11
*** eharney has joined #openstack-infra13:11
*** dustins has joined #openstack-infra13:13
*** garyh has joined #openstack-infra13:14
dteselkinHi. Does anyone knows why database setup is skipped in slave_db.pp if in chroot?13:14
dteselkin* know13:14
*** mattfarina has joined #openstack-infra13:14
*** julim has joined #openstack-infra13:14
dteselkin* in thick_slave.pp, sorry13:15
openstackgerritMichael Johnson proposed openstack-infra/git-review: Choose tracked branch for rebase when submitting  https://review.openstack.org/15887713:17
*** tqtran has quit IRC13:18
*** mpaolino has quit IRC13:18
*** mpaolino has joined #openstack-infra13:19
*** mriedem_away has quit IRC13:24
*** garyh has quit IRC13:24
*** mpaolino has quit IRC13:24
*** mpaolino has joined #openstack-infra13:25
*** ddieterly has joined #openstack-infra13:26
*** bknudson has joined #openstack-infra13:27
*** prad has joined #openstack-infra13:27
*** ChuckC has joined #openstack-infra13:28
*** e0ne is now known as e0ne_13:30
*** mpaolino has quit IRC13:30
*** mpaolino has joined #openstack-infra13:31
*** hdd has joined #openstack-infra13:32
*** bhunter71 has joined #openstack-infra13:32
mtreinish\o/ the migration looks like it finished13:33
*** salv-orlando has quit IRC13:34
*** alexpilotti has quit IRC13:35
*** ghostpl_ has joined #openstack-infra13:35
*** mpaolino has quit IRC13:36
*** mpaolino has joined #openstack-infra13:36
mtreinishnow we just need to restart the worker daemon so that microseconds get stored13:37
*** lucas-hungry is now known as lucasagomes13:37
fungidteselkin: we override it in thick_slave.pp to enabled13:37
fungidteselkin: when we call into it from nodepool build scripts13:38
*** crc32 has quit IRC13:38
*** dkranz has quit IRC13:39
*** spredzy|afk is now known as spredzy13:39
*** dboik has joined #openstack-infra13:39
fungidteselkin: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/prepare_node_bare.sh sets export ALL_MYSQL_PRIVS='true' before calling into prepare_node.sh13:39
*** jistr|biab is now known as jistr13:39
*** e0ne_ has quit IRC13:40
fungiwhich in turn passes that value into the all_mysql_privs parameter for openstack_project::single_use_slave with puppet apply13:40
*** boris-42 has joined #openstack-infra13:41
fungidteselkin: that said, we're working on getting rid of node build-time database preconfiguration in favor of having individual projects test setups do that as needed13:42
*** mpaolino has quit IRC13:43
*** ghostpl_ has quit IRC13:43
*** mriedem has joined #openstack-infra13:43
*** mpaolino has joined #openstack-infra13:43
dteselkinSo it's normal behavior to call puppet once again from ready_script to create databases and grants if needed?13:43
fungidteselkin: i have a proof-of-concept at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/experimental-workers.yaml working out a plan to do it in a jenkins job builder macro as a stepping stone to putting it in the individual projects that need it13:43
fungidteselkin: where did i say anything about the ready script? we're currently doing it in the nodepool image prep scripts13:44
dteselkinThat's the problem :) I'm trying to create an image with DIB and because of chroot database preparation skipped13:45
fungidteselkin: though only for snapshot images. we can't do it with dib because trying to preconfigure mysql inside a chroot isn't really sane (hence my priority on moving the database setup to happen at job start time)13:45
dteselkinI thought about ready_script because it'll reduce job execution time.13:46
*** amitgandhinz has joined #openstack-infra13:47
*** cdent has quit IRC13:47
*** enikanorov_ has joined #openstack-infra13:47
fungidteselkin: yep, but the increase in job runtime is actually negligible. proof of concept demonstration http://logs.openstack.org/44/164444/1/experimental/experimental-nova-dsvm-python27/f4d9841/console.html13:47
fungidteselkin: and doing it within the context of a job means that different jobs can have their own database needs met even if they might conflict with one another13:48
*** dkranz has joined #openstack-infra13:49
fungidteselkin: also when we get to the point where we can extract database setup all the way out to the projects using that for their jobs, changes to the database configuration become self-testing, so infra has less of an opportunity to accidentally break them13:49
*** enikanorov has quit IRC13:49
*** cdent has joined #openstack-infra13:49
dteselkinYep, that's a good point too13:50
*** sandywalsh has joined #openstack-infra13:50
*** peristeri has joined #openstack-infra13:50
*** peristeri has quit IRC13:50
*** peristeri has joined #openstack-infra13:50
*** fifieldt has quit IRC13:51
*** ghostpl__ has joined #openstack-infra13:51
*** peristeri has quit IRC13:51
fungialso, between that and doing distro package installation at job runtime means we can stop having separate "bare" and "devstack" orker types, which significantly simplifies our worker pooling13:51
*** peristeri has joined #openstack-infra13:51
fungithe goal is to get down to where worker images differ only by the distro/release they're running, and all jobs are able to start up from scratch on the same images13:52
*** sarob has joined #openstack-infra13:52
*** BobH_ has quit IRC13:53
*** e0ne has joined #openstack-infra13:54
*** eharney_ has joined #openstack-infra13:54
dteselkinAgree, even if I only have two different type of nodes in my setup :)13:56
*** eharney has quit IRC13:56
*** zz_jgrimm is now known as jgrimm13:57
dteselkinfungi: thanks you13:57
dteselkin* thank you!13:57
*** sarob has quit IRC13:57
*** eharney_ is now known as eharney13:58
fungiyou're welcome. hopefully you'll be able to take advantage of some of these improvement soon13:58
*** mpaolino has quit IRC13:59
*** mpaolino has joined #openstack-infra13:59
*** sabeen1 has joined #openstack-infra13:59
*** nilasae has quit IRC14:00
fungiright now i'm polishing off some features to stackforge/bindep (which infra is formally adopting) to allow us and eventually individual projects maintain platform-independent lists of distro packages their jobs need so that we can pre-cache those in aggregate when building images and then install them selectively when running jobs14:00
*** ghostpl_ has joined #openstack-infra14:00
*** dustins_ has joined #openstack-infra14:00
*** nelsnelson has joined #openstack-infra14:01
*** hyakuhei has joined #openstack-infra14:02
*** mrmartin has quit IRC14:02
*** hyakuhei has quit IRC14:02
mtreinishfungi: now that the migration is done I think https://review.openstack.org/164730 is good to go14:02
*** ghostpl_ has quit IRC14:02
fungimtreinish: yes, it finished sometime after i passed out14:03
fungii should have thought to run it under time14:03
*** dustins has quit IRC14:03
*** mpaolino has quit IRC14:03
*** alexpilotti has joined #openstack-infra14:03
fungii should just start putting timestamps in my shell prompts14:03
*** mpaolino has joined #openstack-infra14:04
mtreinishfungi: heh, that's why I do it, because I forget to use time for things like this :)14:04
mtreinishfwiw, it looked to be done before I shut my machine down around midnight14:05
fungiyeah, i didn't last that long14:05
* fungi is an old man14:05
anteayafungi: are you ready for that patch to merge then?14:05
anteayaI can approve if you like14:05
*** hasharFood is now known as hashar14:06
fungii just did14:06
anteayaah faster than I14:06
anteayaokay14:06
*** ghostpl__ has joined #openstack-infra14:06
fungii hadn't reviewed it previously, so was just looking over it first to be sure i knew what i was signing14:06
anteayaI have to run errands uptown today and will be away for the better part of the day14:06
*** changbl has joined #openstack-infra14:06
anteayagood approach14:06
fungidon't forget to take a torch14:06
mtreinishfungi: the other follow up thing is restarting the gearman worker daemon on subunit-worker01 to use the new version14:07
anteayaI held off voting to make sure it didn't get approaved early14:07
fungimtreinish: i can do that now. hold on14:07
anteayaI wasn't thinking about it, but I guess a torch can't hurt14:07
mtreinishfungi: cool, thanks14:07
*** sigmavirus24_awa is now known as sigmavirus2414:08
mtreinishanteaya: a real torch or a queen's english torch?14:08
*** dizquierdo has quit IRC14:08
fungimtreinish: just to confirm, there are no jenkins-subunit-worker-X processes running at the moment, and we want to start all four of them, yeah?14:08
*** otter768 has joined #openstack-infra14:08
fungimtreinish: totally depends on the time period. at one point they were one and teh same after all14:08
mtreinishfungi: we just need 1 running14:08
mtreinishfungi: heh, fair point14:09
mtreinishfungi: 4 caused issues at on point and 1 seems to be able to handle the load fine14:09
fungiokay, it looks like jenkins-subunit-worker-A is the only initscript of the four enabled, now that i peer more closely14:09
fungiso i've started that one now14:10
fungiand it seems to be running still, so no instacrash anyway14:10
mtreinishfungi: awesome14:10
* mtreinish will watch the db to see if microseconds start to appear14:10
mtreinish\o/ http://paste.openstack.org/show/193122/14:11
mtreinishnow after the next image build the preseed will hopefully be useful14:11
mordredwoot!14:12
*** dhritishikhar has joined #openstack-infra14:12
*** otter768 has quit IRC14:13
*** mtanino has joined #openstack-infra14:13
openstackgerritMerged openstack-infra/project-config: Revert "Cap the subunit2sql version in the nodepool scripts"  https://review.openstack.org/16473014:13
*** BharatK has quit IRC14:14
*** alexpilotti has quit IRC14:14
*** alexpilotti has joined #openstack-infra14:15
mordredGheRivero: so - if you have interest, I made the world's worst todo list WRT the upstream ansible modules: https://etherpad.openstack.org/p/ansible-patches-to-port14:16
*** hashar has quit IRC14:17
*** hashar has joined #openstack-infra14:17
mordredGheRivero: however, the thing that could use real help is the nodepool patch that yolanda and I have both worked on: https://review.openstack.org/#/c/156322/14:17
mordredGheRivero: which is aiming at porting all of nodepool's openstack interactions to use shade14:18
*** mpaolino has quit IRC14:18
GheRiveromordred: then I'll focus on nodepool patches14:18
mordredGheRivero: it probably also means adding some things to shade - like I think nodepool is currently doing better logic than shade is with some of the floating ip stuff - but we also probably want to get shade to try to use neutron directly for floating ips14:18
yolandaGheRivero, just ping me if you have some question14:19
*** mpaolino has joined #openstack-infra14:19
yolandashade was missing the feature of releasing floating ip on delete server14:19
openstackgerritJeremy Stanley proposed openstack-infra/infra-specs: Wrap askbot migration steps in code blocks  https://review.openstack.org/16542814:19
*** dizquierdo has joined #openstack-infra14:20
yolandaGheRivero, i started working a bit on that, but i switched by the joy of puppet modules :)14:21
*** mpaolino has quit IRC14:21
GheRivero:)14:21
*** mpaolino has joined #openstack-infra14:22
*** dhritishikhar has quit IRC14:23
*** timcline has joined #openstack-infra14:24
*** garyh has joined #openstack-infra14:24
*** Somay has joined #openstack-infra14:24
*** mestery has quit IRC14:27
*** ghostpl__ has quit IRC14:29
*** ghostpl_ has joined #openstack-infra14:30
*** BharatK has joined #openstack-infra14:30
*** mpaolino has quit IRC14:30
*** tsg_ has joined #openstack-infra14:30
*** mpaolino has joined #openstack-infra14:30
*** ghostpl_ has quit IRC14:31
openstackgerritJan Provaznik proposed openstack-infra/project-config: Create os-cloud-management project on Stackforge  https://review.openstack.org/16543314:31
*** tsg_ has quit IRC14:32
*** wenlock has joined #openstack-infra14:33
*** ghostpl_ has joined #openstack-infra14:33
*** ajmiller has joined #openstack-infra14:34
*** zxiiro has quit IRC14:37
hashargood morning!14:37
hasharanyone has any idea about Nodepool cpu/io consumption ?14:37
hasharI am wondering how small / big a server needs to be to host it14:37
*** changbl has quit IRC14:38
mordredwell, I was going to connect to ours to tell you because I can't remember ...14:39
hasharthat would be great :)14:39
mordredfungi: ^^ my ssh nodepool.openstack.org is hanging ...14:39
*** BharatK has quit IRC14:39
fungimordred: interesting. works for me14:40
hasharI am wondering how many cores/cpu/mem I should plan. I guess it is mostly lightweight and a simple dual core  4GBytes ram would largely cover it14:40
mordredhashar: we're using an 8G cloud instance14:40
fungihashar: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=9414:41
lucasagomeshi, if you have a free time please take a look at https://review.openstack.org/#/c/163344/ , it's a small patch enabling we run the Ironic IPA deploy ramdisk in the openstack/ironic-python-agent check gate (as non-voting)14:41
dhellmannanteaya, do you have a few minutes to help me figure out why the requirements gate job isn't running for python-openstackclient patches? I can't see what config piece we're missing.14:41
fungihashar: most of the activity spikes you see there correspond to when we're performing daily image updates (right now for example)14:41
dhellmannanteaya: I would have expected gate-python-openstackclient-requirements to run for https://review.openstack.org/#/c/158779/ for example14:42
*** zz_dimtruck is now known as dimtruck14:42
hasharfungi: nice that is excellent!14:43
hashardoes nodepool still require 2 mysql connections per spawned vm ?14:43
fungihashar: gosh, i hope not. that sounds excessive given we spike to around 900 virtual machines at peak14:44
hasharfungi: http://ci.openstack.org/nodepool/installation.html#database :D14:45
hashar"" Nodepool will hold open a database connection for each node. Be sure to configure the database server to support at least a number of connections equal to twice the number of nodes you expect to be in use at once. ""14:45
fungihrm, yeah let me see if i can tell whether that's still the case14:45
hasharI guess that is the absolute possible spike, not the usual load14:45
hasharcacti has a low # of TCP connections for the host14:46
fungisounds like something we would have improved since that was written14:46
hasharless than a hundre14:46
hashard14:46
*** achanda has joined #openstack-infra14:47
*** mpaolino has quit IRC14:47
*** mpaolino has joined #openstack-infra14:48
*** Somay has quit IRC14:48
*** Somay has joined #openstack-infra14:49
*** mpaolino has quit IRC14:50
*** mpaolino has joined #openstack-infra14:51
*** BharatK has joined #openstack-infra14:51
*** tsufiev_ has left #openstack-infra14:51
*** achanda has quit IRC14:52
*** mfink_ has joined #openstack-infra14:52
*** mpaolino has quit IRC14:52
*** sputnik13 has joined #openstack-infra14:53
*** mpaolino has joined #openstack-infra14:53
*** sarob has joined #openstack-infra14:53
openstackgerrityolanda.robla proposed openstack-infra/project-config: Add new puppet-puppet module  https://review.openstack.org/16544214:54
*** stevemar has joined #openstack-infra14:54
*** zxiiro has joined #openstack-infra14:55
*** adrian_otto has joined #openstack-infra14:55
*** Bsony has joined #openstack-infra14:55
*** yfried is now known as yfried|afk14:56
*** nanliu has joined #openstack-infra14:56
*** Longgeek has quit IRC14:56
*** LinuxJedi is now known as LinuxJedi_14:57
*** MarkAtwood has joined #openstack-infra14:57
*** Bsony_ has quit IRC14:57
*** yamamoto has quit IRC14:58
*** vhoward has joined #openstack-infra14:58
*** sarob has quit IRC14:58
*** yamamoto has joined #openstack-infra14:59
*** yfried|afk is now known as yfried14:59
*** reed has joined #openstack-infra14:59
*** patrickeast has joined #openstack-infra15:00
*** asselin_ has joined #openstack-infra15:00
*** pc_m has quit IRC15:00
openstackgerritJames E. Blair proposed stackforge/gertty: Display warnings as a popup  https://review.openstack.org/16544715:00
openstackgerritJames E. Blair proposed stackforge/gertty: Do not display InsecurePlatformWarning  https://review.openstack.org/16544815:00
*** changbl has joined #openstack-infra15:01
jeblairclarkb, fungi, mordred, dstufft: ^ an alternate approach to the InsecurePlatformWarning problem15:01
jeblairzaro: can you take a look at that too?  i moved a recent change from you around15:01
*** sputnik13 has quit IRC15:02
mordredjeblair: I like the way you thnk15:04
*** sputnik13 has joined #openstack-infra15:04
*** amotoki has quit IRC15:06
*** Somay has quit IRC15:06
*** mrunge is now known as mrunge_away15:08
*** salv-orlando has joined #openstack-infra15:09
*** asilenkov has joined #openstack-infra15:10
*** Trozz has quit IRC15:10
asilenkovHello! Which part of openstack CI @hudson-openstack robot belongs to?15:10
*** Trozz has joined #openstack-infra15:11
*** claudiub has joined #openstack-infra15:11
*** jaypipes has quit IRC15:12
fungiasilenkov: are you talking about the ~hudson-openstack account in launchpad?15:12
asilenkovyep )15:13
fungiasilenkov: it's pretty much only used by the update_bug.py and update_blueprint.py gerrit hook scripts in the openstack-infra/jeepyb repo15:13
asilenkovthank you!15:14
*** hdd has quit IRC15:14
fungiseeing that stuck nova-docker change in the check pipeline makes me wonder if we removed the devstack-f20 node type while some jobs were still using it15:14
funginope, we still have min-ready set to 1. maybe we're just not building any15:15
*** dhritishikhar has joined #openstack-infra15:15
fungiyeah, 5 in a building state15:15
fungiall in rax-iad strangely enough15:15
*** Somay has joined #openstack-infra15:16
*** Somay has left #openstack-infra15:16
*** amotoki has joined #openstack-infra15:16
*** carl_baldwin has joined #openstack-infra15:16
*** salv-orlando has quit IRC15:17
*** pcaruana has joined #openstack-infra15:19
ctlaughI am seeing a lot of this error in the zuul webapp and zuul.log/debug.log: http://paste.ubuntu.com/10621442/.  Once it starts happening, it doesn't stop.  Is this a known bug?  Any workaround?15:19
openstackgerritJan Provaznik proposed openstack-infra/project-config: Create os-cloud-management project on Stackforge  https://review.openstack.org/16543315:21
*** crc32 has joined #openstack-infra15:26
*** ajmiller has quit IRC15:26
*** ssam2 has quit IRC15:27
*** BobH has joined #openstack-infra15:29
*** pc_m has joined #openstack-infra15:30
*** ajmiller has joined #openstack-infra15:30
*** crc32 has quit IRC15:30
*** sandywalsh has quit IRC15:31
*** yfried is now known as yfried|afk15:32
*** jp_at_hp has quit IRC15:33
*** LinuxJedi_ is now known as LinuxJedi15:34
*** yfried|afk is now known as yfried15:35
*** tnovacik has quit IRC15:37
*** armax has joined #openstack-infra15:37
*** banix has joined #openstack-infra15:37
*** ashleighfarnham has joined #openstack-infra15:38
*** amotoki has quit IRC15:38
*** johnma has joined #openstack-infra15:38
*** julim has quit IRC15:38
*** dangers_away is now known as dangers15:39
*** unicell has quit IRC15:40
*** ssam2 has joined #openstack-infra15:40
*** julim has joined #openstack-infra15:41
*** mrmartin has joined #openstack-infra15:41
*** packet has joined #openstack-infra15:41
openstackgerritMonty Taylor proposed openstack-infra/infra-specs: Add spec for puppet apply work  https://review.openstack.org/16521715:42
*** unicell has joined #openstack-infra15:42
rhe00anyone knows a reason as to why zuul would crash when trying to start the second process (I assume that's the scheduler). Nothing in zuul.log or debug.log to indicate why15:43
*** ujuc has quit IRC15:44
*** changbl has quit IRC15:44
*** changbl has joined #openstack-infra15:45
*** mpaolino has quit IRC15:45
*** mpaolino has joined #openstack-infra15:46
*** mrmartin has quit IRC15:48
fungirhe00: very broken configuration, stale pidfile...15:49
*** spzala has quit IRC15:49
fungirhe00: did you have a working zuul deployment, then upgrade zuul while trying to reuse your old config?15:49
rhe00fungi: all I did was changing the project in layout.yaml from openstack-dev/ci-sandbox to openstack/cinder15:50
rhe00and restarted zuul15:50
fungizaro: we were seeing that symptom on zuul-dev right? (zuul instantly terminates on startup before logging any details as to why) do you recall what the cause ended up being?15:50
rhe00since then I have changed the project back to openstack-dev/ci-sandbox, but it won't start with that either now15:50
jeblairrhe00: you might check the lockfile15:50
jktrhe00: sometimes, I've seen zuul not even being able to open its logs; when I'm starting it via systemd, I don't get anything in the logs either15:50
rhe00jeblair: where's the lockfile hiding?15:51
jeblairrhe00: /var/run/zuul/zuul.pid is the default15:51
*** sdake_ has quit IRC15:51
jktjeblair: any idea why I'm getting https://storyboard.openstack.org/#!/story/2000205 btw?15:51
*** dhritishikhar has quit IRC15:52
clarkbjkt: looks like another case of stale egg15:52
*** mestery has joined #openstack-infra15:52
*** sdake has joined #openstack-infra15:52
jktclarkb: I did run `pip uninstall` before installing once again, but let's see what gets left over15:53
jeblairjkt: yes,  we think that happens when a previous install uses 'python setup.py install'15:53
*** Sukhdev has joined #openstack-infra15:53
jeblairjkt: if you do 'rm -rf gertty.egg-info', then "pip install ."  or "pip install -e ." should work15:53
rhe00jeblair: that was it! I removed the lockfile and now zuul started up properly.  thanks!15:53
*** unicell has quit IRC15:53
clarkbrhe00: how did you stop zuul?15:53
jeblairrhe00: np.  we should probably detect that and print a message, eh? :)15:53
rhe00sudo service zuul stop15:54
clarkbrhe00: that should work if using our init script15:54
*** unicell has joined #openstack-infra15:54
clarkbrhe00: any idea where your init script (or equivalent) comes from?15:54
*** sarob has joined #openstack-infra15:55
*** hdd has joined #openstack-infra15:55
rhe00clarkb: when you say init script, are you referring to the layout.yaml?15:55
rhe00or how it was installed in the first place?15:55
jeblairclarkb, mordred: do we think we have the egg bug narrowed down enough we could file a proper bug report... in something?15:55
clarkbrhe00: no, the script that tells ubuntu how to manage the service15:56
jktjeblair: thanks, that fixed it15:56
rhe00clarkb: ah, sorry. I used asselins install scripts to set up the master, and I assume that set up the init script15:57
pabelangerohai! Wanted to make sure I understand properly, but when using multi-nodes in nodepool.  Sub nodes launch first, then nodes. Making the dependency order node requires a subnode.15:57
*** SumitNaiksatam has quit IRC15:57
*** MarkAtwood has quit IRC15:57
*** salv-orl_ has joined #openstack-infra15:58
*** ashleighfarnham has quit IRC15:58
jeblairpabelanger: i _think_ they all launch at the same time, but the node is not declared ready until all subnodes are15:58
openstackgerritMerged openstack-infra/infra-specs: Wrap askbot migration steps in code blocks  https://review.openstack.org/16542815:58
clarkbjeblair: no, I don't know much other than the egg info is not updated if it already exists15:58
*** bswartz has quit IRC15:59
clarkbjeblair: not sure where exactly that is failing15:59
clarkbbut if the info isn't updated then you don't get alembic migrations updated either15:59
*** thedodd has joined #openstack-infra15:59
pabelangerjeblair, okay, that helps. Planning on playing with it today to see how it works15:59
jeblairmordred: can you supply any help narrowing this down?15:59
*** deepakcs has joined #openstack-infra16:00
mordredjeblair: yes. I have a hunch on where to look in pbr ... I'll put it on my list16:00
*** sarob has quit IRC16:00
jeblairmordred: i'll file a pbr bug for now then16:00
*** bhuvan has joined #openstack-infra16:01
*** bswartz has joined #openstack-infra16:01
*** yamahata has joined #openstack-infra16:02
*** unicell has quit IRC16:02
*** LinuxJedi is now known as LinuxJedi_16:02
*** spredzy is now known as spredzy|afk16:03
*** lennyb has joined #openstack-infra16:04
*** asselin_ has quit IRC16:04
*** ayoung has joined #openstack-infra16:04
lennybkrtaylor, I wander if we tempest is a 'must' for CI, we are considering running rally instead of tempest16:05
jeblairyay for eavesdrop16:05
*** mpavone has quit IRC16:05
fungiheads up: forthcoming openssl vulnerability announcement scheduled for some time between 1100-1500 utc tomorrow16:05
krtaylorlennyb, what project are you testing against?16:06
*** changbl has quit IRC16:06
clarkbfungi: ya...16:06
lennybNova, Cinder, Neutron16:06
clarkbfungi: though I thought I had also heard it got pushed to friday16:06
lennybkrtaylor, Nova, Cinder, Neutron16:06
*** belmoreira has quit IRC16:07
krtaylorlennyb, it is up to the project to specify what tests you need to run for reporting16:07
clarkbfungi: either way it should be fun16:07
*** ibiris has joined #openstack-infra16:07
*** ja- has joined #openstack-infra16:07
*** jgrimm_ has joined #openstack-infra16:07
fungiclarkb: i haven't seen anything on oss-security about it moving to friday at least16:07
alex_xuianw: would you like looks at this https://review.openstack.org/160068 again when you have time? There is nova patch depend on that. close to freeze, sorry for push so hard. anyway thanks in advance!16:07
*** banix_ has joined #openstack-infra16:07
*** adalbas has quit IRC16:07
*** maurosr- has joined #openstack-infra16:08
lennybkrtaylor, thanks. I will ask this question on the project irc channel or search the wiki16:08
krtaylorlennyb, each have their own requirements, rally isn't really the same thing16:08
*** mjturek2 has joined #openstack-infra16:08
*** maurosr has quit IRC16:09
*** ashleighfarnham has joined #openstack-infra16:09
*** maurosr| has joined #openstack-infra16:09
*** jgrimm- has joined #openstack-infra16:09
*** otter768 has joined #openstack-infra16:09
krtaylorlennyb, that is not to say you can't use rally, it depends on your goals, if you want to report back on a patch, etc16:09
*** zz_ja has quit IRC16:09
*** maurosr| is now known as maurosr16:09
*** banix has quit IRC16:10
*** banix__ has joined #openstack-infra16:10
lennybkrtaylor, I see, thanks.16:10
jeblairclarkb, mordred: https://bugs.launchpad.net/pbr/+bug/143368116:10
openstackLaunchpad bug 1433681 in PBR "Stale egg-info dir causes alembic migrations not to be included" [Undecided,New]16:10
*** ja| has joined #openstack-infra16:10
*** mjturek3 has joined #openstack-infra16:10
*** ja| is now known as zz_ja|16:10
*** jgrimm has quit IRC16:10
jeblairclarkb, mordred: does that look correct, and have all known info?16:10
*** mjturek1 has quit IRC16:11
pabelangerBecause I don't know, which, if any, project are using multi nodes with nodepool?16:11
*** MarkAtwood has joined #openstack-infra16:11
jeblairpabelanger: i believe there's an experimental multi-node nova job16:11
jeblairpabelanger: (it's all still pretty new)16:11
clarkbjeblair: looks right other than that you install 1.0.3 twice but that should be fine16:11
mordredjeblair: yup! I see the bug16:12
clarkbinstall it twice in your reproduction steps16:12
mordredjeblair: as in, I see the bug in pbr16:12
*** Somay has joined #openstack-infra16:12
*** banix_ has quit IRC16:12
jeblairclarkb: oh, fixed.16:12
*** mjturek2 has quit IRC16:12
*** maurosr- has quit IRC16:12
*** Bsony has quit IRC16:13
jeblairclarkb: though, tbh, i do not know for certain that is not necessary to reproduce.  i'll double check.16:13
*** jgrimm_ has quit IRC16:13
*** ja- has quit IRC16:13
jeblairmordred: that was fast :)16:13
pabelangerjeblair, okay, let me see if I can hunt it down16:13
mordredjeblair: you said the magical words to describe the problem a few minutes ago16:14
mordredthat caused my brain to click16:14
*** Somay has quit IRC16:14
openstackgerritMerged stackforge/python-jenkins: Add a get_nodes method  https://review.openstack.org/16306816:14
*** otter768 has quit IRC16:14
*** spzala has joined #openstack-infra16:15
fungioh, if anyone gets a moment to have a look, ajaeger spotted a problem with check_irc_access.py https://review.openstack.org/164979 i started looking at it yesterday but was a little confused myself as to how the failure tracking was intended to work16:15
jeblairclarkb: it is required16:15
*** dustins_ has quit IRC16:15
clarkbjeblair: huh, I didn't realize16:15
*** Guest17003 is now known as erikmwilson16:16
*** koolhead17 has quit IRC16:16
mordredjeblair: tl;dr- we avoid running setup.py egg-info in many cases because if there is no git repo, running egg-info will override the version info that's in the tarball with the results of trying to find it in a non-git repo, which is failure16:16
*** lennyb has quit IRC16:16
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add release notes to the README  https://review.openstack.org/16548516:16
mordredjeblair: however, the logic to detect when it should be re-run does not include checking for alembic files16:16
*** Swami has joined #openstack-infra16:16
*** resker has joined #openstack-infra16:16
jeblairmordred: aha!16:17
mordredjeblair: SO - I think we can make it smarter16:17
*** esker has quit IRC16:17
zaromorning16:17
mordredjeblair: and perhaps always run egg-info if there is a git dir and/or re-using the pbr.json info16:17
mordredlooking in to that now16:17
*** arxcruz has quit IRC16:17
*** dustins has joined #openstack-infra16:17
jeblairfungi: looking16:17
*** HeOS has quit IRC16:20
mordredhrm16:20
jeblairclarkb, jhesketh: http://logs.openstack.org/79/164979/2/check/gate-project-config-irc-access/fa7952c/console.html16:20
*** mpavone has joined #openstack-infra16:20
*** dhritishikhar has joined #openstack-infra16:20
*** AJaeger has joined #openstack-infra16:21
AJaegerfungi, clarkb: Good morning, do either of you have a few minutes?16:21
ctlaughre: previous msg, this is the output from running 'sudo zuul show running-jobs' -- seems to be encountering the same error (as expected): http://paste.ubuntu.com/10621770/.16:23
clarkbjeblair I see what happened, will work on a fix16:23
*** adalbas has joined #openstack-infra16:23
jeblairclarkb: for what?16:24
pleia2good morning16:24
jeblairclarkb: oh, the proxy error :)16:24
fungiAJaeger: what's the question? i was just mentioning the irc access check problem to try and get more eyes on it. i'm mildly confused as to why it's doing that myself16:25
AJaegerfungi, exactly that'S the problem ;)16:26
AJaegerfungi, I'm offline most of the wiki and just have 30 mins to help ;)16:26
AJaegerfungi, glad you're on it - thanks16:26
jeblairAJaeger, fungi: one problem is that the jobs are currently timing out because "puppet-rdo" is not registered with chanserv and the script does not handle that case.16:27
*** dtantsur is now known as dtantsur|afk16:27
AJaegerjeblair, argh, we're not catching that indeed16:27
jeblairAJaeger, fungi: i will fix that.  (also, note puppet-rdo != rdo-puppet; rdo-puppet is the channel that prompted AJaeger to write this change)16:28
clarkbjeblair ya need to not append if error16:28
AJaegerjeblair, ah, so my manual "test case" is broken.16:28
*** dhritishikhar has quit IRC16:28
AJaegerjeblair, thanks for catching that - but we still should not timeout16:29
jeblairAJaeger: yes, as i said, i'm fixing that16:29
AJaegerjeblair, understood and appreciated!16:29
openstackgerritMerged openstack-infra/project-config: Add puppet-openstackci project  https://review.openstack.org/16325516:30
AJaegerjeblair, feel free to rework my patch - or do a new one and I'm fine to abandon mine16:30
jeblairAJaeger: ok -- i have not looked into the original error yet16:30
*** ddieterly has quit IRC16:31
*** scheuran has quit IRC16:31
AJaegerjeblair, one step at a time ;)16:31
fungiyep, so the question is why the run with #rdo-puppet (which is registered) isn't bailing when openstackinfra lacks an entry in the access listy16:31
*** marun has joined #openstack-infra16:31
fungii wonder if that's just tox missing the exit code from the script before it goes on to run irc_test.py16:32
jeblairfungi: i suspected so (before i branched onto this)16:32
*** hashar is now known as hasharAw16:33
*** ddieterly has joined #openstack-infra16:33
*** hasharAw is now known as hasharAway16:33
*** eglute has joined #openstack-infra16:33
*** rlandy has quit IRC16:33
fungii _think_ tox.ini is correct there so maybe we're not actually exiting nonzero when we think we should16:34
AJaegerfungi, change Ic68ba9a1c60854e68817fcadb1a5df2144944af7 added the second command to tox.ini16:34
eglutehello Infra, we could really use some Core reviewers on defcore-core https://review.openstack.org/#/admin/groups/uuid-ad95fb605fa544dab35712194df7faaa10ec7a22,members16:34
AJaegerI saw it fail before that change merged16:34
fungiwe're doing multiple commands in testenv:irc the same way as in testenv:projects16:35
*** tnovacik has joined #openstack-infra16:36
clarkbeglute who should be the first member (first member can add the rest)16:36
egluteshould be me- myself and Rob H will be co-ptl for the project16:36
jeblairfungi, AJaeger: after correcting the not registered error, it still exited 0, so i think there's a bug in the script.16:36
*** mattfarina has quit IRC16:36
*** ryanpetrello has quit IRC16:37
fungiyeah, i'm mostly confused how the cap handler works, so starting to plug some debug info into it16:38
*** mattfarina has joined #openstack-infra16:38
jeblairfungi: i'm pretty sure the "self.failed = False" in on_privnotice is wrong; digging to try to see where that came from16:39
clarkbaccount not found eglute have the account username or id?16:39
egluteoh sorry about that. email: ushnishtha@hotmail.com16:39
fungihogepodge: also can you just confirm ^ since i added that project at your direction?16:40
clarkbeglute added, you can add others now16:40
eglutethank you so much!!16:40
*** unicell1 has joined #openstack-infra16:41
AJaegerjeblair, it worked in the past - did we update the irc python code16:41
*** Bsony has joined #openstack-infra16:41
*** mpaolino has quit IRC16:42
*** sigmavirus24 is now known as sigmavirus24_awa16:42
*** ryanpetrello has joined #openstack-infra16:42
jeblairAJaeger, fungi: that line goes back to the initial commit. i suspect that it has only exited non-zero if the _last_ channel in the list failed the check.16:43
fungiAJaeger: doesn't look like it. all lines in that file originate from the change which initially added it16:43
*** gyee has joined #openstack-infra16:43
AJaegerI meant: irc python library16:43
fungioh16:43
*** ajmiller has quit IRC16:44
AJaegerJust hunting down what could have triggered this and whether that would help us16:44
*** scheuran has joined #openstack-infra16:44
odyssey4mehi guys, can we get some reviews on this please - it's been sitting for a while: https://review.openstack.org/16427116:45
AJaegerjeblair, it could also be a bug in the script. So, when I looked at it, I saw we sat the failed value to both true and false - and that gave me the idea that we might have some non-determinism16:45
AJaegerTherefore I created https://review.openstack.org/#/c/164979/16:46
jeblairAJaeger, fungi: i was close -- it would only have failed in the past if the last channel it checked had problems; however, the list is reversed, so it would have been the first channel listed in the file.16:47
AJaegermarun, could you check 164979, please?16:48
AJaegerjeblair, so, is 164979 correct?16:48
marunAJaeger: will do16:48
AJaegerjeblair, I meant 164271 for the irc_access16:49
jeblairAJaeger: possibly, but it removes an intended failure check.  i'm writing an alternate change.16:49
AJaegerjeblair, I now get a -1 with rdo-puppet as test case16:49
marunAJaeger: Hmmm, not sure I'm the person to review that16:49
AJaegerjeblair, ok16:49
AJaegermarun, I mixed up numbers16:49
AJaegerI meant odyssey4me's change:  https://review.openstack.org/164271 - it includes a skip-if16:50
AJaegermarun, ^16:50
*** unicell has joined #openstack-infra16:50
marunAJaeger: ah, got it16:50
*** Ala has quit IRC16:50
Swamiclarkb: hi16:50
clarkbhi16:51
SwamiI have some questions regarding the multinode setup for the neutron-dvr job16:51
Swamilink https://review.openstack.org/#/c/158525/16:51
*** imcsk8 has quit IRC16:51
*** imcsk8 has joined #openstack-infra16:51
marunAJaeger: lgtm16:51
SwamiFor multinode setup we need to have a DATA_IP configured between the primary and the secondary node, how do we pass that IP information. Or what would be the IP information16:52
*** Sukhdev has quit IRC16:52
*** unicell1 has quit IRC16:52
clarkbSwami: yesterday you had asked about an additional interface. The only way for additional interfaces is to do something like what the nova network job does and make some gre tunnels. So if we can avoid that because a second interface isn't necessary that would be best (it is necessary for nova network because it wants things on consistent layer 2 networks)16:53
clarkbSwami: if you just need an IP addr you should use the private IP for each node. See primary_node_private and sub_nodes_private in that file16:54
openstackgerritJeremy Stanley proposed openstack-infra/infra-manual: Your Gerrit and Launchpad username can differ  https://review.openstack.org/16550716:54
Swamiclarkb: does the primary_node_private correlate with the physical interface.16:55
Swamicarkb: The primary node has a public interface in "eth4"16:55
clarkbSwami: I do not know where eth4 comes from, none of our test slaves have an eth4 iirc16:55
*** sarob has joined #openstack-infra16:56
clarkbthe private IPs are a non world routable ip addr assigned by our clouds16:56
clarkbin hpcloud this is the only ip addr and interface given by the cloud, in rackspace it is the second interface (the first has a world routable ip addr)16:56
*** melwitt has joined #openstack-infra16:56
Swamiclarkb: yes in devstack-vm-gate.sh there is "PUBLIC_INTERFACE=eth4"16:57
*** ihrachyshka has quit IRC16:57
*** adrian_otto has left #openstack-infra16:57
openstackgerritJames E. Blair proposed openstack-infra/project-config: Fix failure detection in irc access check  https://review.openstack.org/16550816:57
jeblairAJaeger, fungi: ^16:57
clarkbSwami: you have added that16:58
clarkbSwami: I do not know where the value comes from16:58
Swamino PUBLIC_INTERFACE=eth4 is lin lin 229 and this was not added by me.16:58
*** openstackgerrit has quit IRC16:59
*** openstackgerrit has joined #openstack-infra16:59
clarkbSwami: we should probably remove that, that appears to be a xenserver thing, we don't use it at all in the multinode tests (or any other tests run upstream)17:00
Swamiclarkb: ok that makes sense, we can remove.17:00
*** sarob has quit IRC17:01
clarkbor is xapi different than xenserver? in any case I don't think its used17:01
Swamiclarkb: coming back to our discussion, on the control plane and dataplane.17:01
fungijeblair: that looks like it should work17:01
*** sarob has joined #openstack-infra17:01
SwamiIn an all-in-one node there is no separate control plane interface and dataplane interface.17:01
*** alexpilotti has quit IRC17:02
*** alexpilotti_ has joined #openstack-infra17:02
SwamiBut in multi-node we need a control plane interface and a dataplane interface.17:02
AJaegerjeblair, LGTM - thanks17:02
fungiand i grok the script a lot better at this point too, so bonus17:02
*** Ryan_Lane has joined #openstack-infra17:03
clarkbSwami: why? if you can use one in single node surelyou can use one in multinode17:03
clarkbSwami: I ask beacuse the alternative is not fun and should be avoided if possible17:03
*** kgiusti has quit IRC17:03
AJaegerLet me test it with my test case to be certain ;)17:03
Swamiclarkb: I can send you the devstack localrc config files for the primary and secondary and could you tell me if the existing setup will work for it or not.17:03
openstackgerrityolanda.robla proposed openstack-infra/project-config: Add new puppet-puppet module  https://review.openstack.org/16544217:04
*** BobH has quit IRC17:04
clarkbSwami: I am not sure if that will help me17:04
clarkbSwami: can you describe the issue with a single interface on multinode?17:04
*** dangers is now known as dangers_away17:04
*** achanda has joined #openstack-infra17:04
* fungi is popping out to consume some lunch, back in a bit17:04
SwamiIn the case of single interface, still we can see the control plane working.17:04
adam_gwe've got a pressing project-config patch up that starts to fix an ironic CI hole that could use another +2, if anyone has a sec https://review.openstack.org/#/c/163632/17:05
*** ajmiller has joined #openstack-infra17:05
*** bswartz has quit IRC17:05
*** jlanoux has quit IRC17:05
SwamiIn the "compute" node we need two interfaces, one private and on public.17:05
*** sarob has quit IRC17:05
clarkbSwami: why can't they both operate over the same interface?17:06
clarkbSwami: are they required to be different layer 2 networks?17:06
*** emagana has joined #openstack-infra17:06
clarkbin nova network they are otherwise ebtables gets really cranky17:06
openstackgerritMonty Taylor proposed openstack-dev/pbr: Don't avoid egg-info quite so often  https://review.openstack.org/16551117:06
Swamiclarkb: In both nodes we need a private and a public interface17:06
mordredclarkb, jeblair ^^17:06
mordredI think that should fix is17:07
mordredit17:07
clarkbSwami: yes, but why?17:07
*** LinuxJedi_ is now known as LinuxJedi17:07
Swamipublic interface is for the North-South testing and the private interface is for the east-west testing.17:07
*** kgiusti has joined #openstack-infra17:08
*** jistr has quit IRC17:08
*** bswartz has joined #openstack-infra17:08
*** yfried is now known as yfried|afk17:08
jeblairmordred: that logic looks good to me.  have you tested it with the gertty test case?  if not, i can do that now.17:08
clarkbSwami: is there a technical reason neutron will not work without two interfaces?17:08
mordredjeblair: yes, I have17:08
jeblairmordred: w00t!17:08
clarkbSwami: lets ignore the ideal deployment situation, will it work with one interface?17:08
mordredjeblair: however, additional testing would be nice - so you should test it too17:08
*** scheuran has quit IRC17:08
*** dmorita has joined #openstack-infra17:08
jeblairmordred: oh. kay.  :)17:09
*** deepakcs has quit IRC17:09
Swamiclarkb: An all-in-one node can just work with a single interface.17:09
clarkbSwami: if not then we can set up multiple layer 2s using the tools we use for nova network, this should be fine but it is far from ideal. So we should understand why we need this before making any changes17:09
AJaegerjeblair, your change passed all my tests - thanks!17:09
clarkbSwami: yes but can a multinode work with one interface per node?17:10
jeblairAJaeger: yay!  you're welcome, and thank you!17:10
*** e0ne is now known as e0ne_17:10
*** sigmavirus24_awa is now known as sigmavirus2417:10
Swamiclarkb: I don't think so117:11
clarkbSwami: can we find out?17:11
BobBallSwami, clarkb: Sorry - why are you proposing removing the PUBLIC_INTERFACE=eth4 part?17:11
clarkbSwami: I have a hunch that a single interface per node will be much simpler and easier to debug if we can make that work17:11
Swamiclarkb:Sure I will do a quick check and let you know today.17:11
BobBallSwami, clarkb: I'm not sure I understand; it's within a block that only runs for VIRT_DRIVER=xenapi so shouldn't affect otehr jobs?17:12
clarkbSwami: the gre tunnels for nova network are not great17:12
clarkbBobBall: its confusing, maybe we can write a better comment?17:12
Swamiclarkb: I will try it out today and tell you the different options that we have.17:12
BobBallclarkb: Ah - ok - certainly; a comment around why the block is needed or specifically why we're using ETH4 for the public interface?17:12
clarkbBobBall: specfically what is this random eth4 that does not exist on any of our test nodes17:13
BobBall*grin*17:13
*** nanliu has quit IRC17:13
*** e0ne_ is now known as e0ne17:13
*** mpavone has quit IRC17:13
*** bradjones has quit IRC17:14
BobBallSo it does exist in the XenServer test nodes in our CI but because it uses devstack-gate to run the tests it's needed in upstream17:14
*** resker has quit IRC17:14
*** mrmartin has joined #openstack-infra17:14
clarkbBobBall: maybe we can move that into the localrc append variable? as another option17:14
*** esker has joined #openstack-infra17:14
BobBallSure - happy to look at whatever option makes most sense17:15
*** nanliu has joined #openstack-infra17:15
BobBallWhich of the settings do you think should be moved to an append variable?  Clearly all of them _could_ be17:15
jeblairclarkb: oops, i was wrong, the extra pip install was not needed.  setup.py install followed by pip install is all that's necessary.17:16
jeblairclarkb, mordred: that change seems to work for me17:16
BobBallMaybe just *_INTERFACE and _BRIDGE as they are specific to the environment we run in?17:16
*** e0ne has quit IRC17:18
*** salv-orl_ has quit IRC17:19
clarkbBobBall: ya I think env specific vars17:20
*** ddieterly has quit IRC17:21
clarkbBobBall: but I would also be happy with a stronger comment as that should prevent confusion17:21
SlickNikhey guys: can I get some eyeballs on https://review.openstack.org/#/c/165161/  please?17:21
SlickNikWe're seeing some issues with slow instances and the trove functional-dsvm tests timing out because of the slowness. We will investigate making the job run faster after the FF, but want to bump up the timeout by 10 minutes to ease the current gate situation.17:21
SlickNikThanks!17:21
BobBallOk - I can do a comment today but env-specific vars will probably have to wait until I get back from vacation as I'd want to do much more testing to make sure we don't break the CI.  I'll add some env-specific comments today.17:21
*** dangers_away has quit IRC17:23
*** patrickeast has quit IRC17:23
*** ddieterly has joined #openstack-infra17:23
openstackgerritClark Boylan proposed openstack-infra/project-config: Don't dump Jenkins proxy errors to console logs  https://review.openstack.org/16552217:23
clarkbBobBall: sounds good17:23
clarkbjeblair: ^ that should fix the proxy error in the log you posted17:24
*** dmorita_ has joined #openstack-infra17:25
*** salv-orlando has joined #openstack-infra17:25
*** dmorita has quit IRC17:25
clarkbBobBall: I added a large comment in 165522 because I think that code has similar issues to the thing we were discussing :)17:26
BobBall:) Understood.17:26
*** harlowja_away is now known as harlowja_17:27
*** dangers_away has joined #openstack-infra17:27
*** garyh has quit IRC17:27
jeblairclarkb resurrects the art of the explanatory code comment17:27
cineramai like those17:28
*** bhuvan has quit IRC17:28
*** yfried|afk is now known as yfried17:29
*** dangers_away is now known as dangers17:30
*** ghostpl_ has quit IRC17:30
jeblairzaro: i pointed out errors in 16514517:30
*** markus_z has quit IRC17:30
*** sdake_ has joined #openstack-infra17:31
openstackgerritKhai Do proposed openstack-infra/system-config: Replace the Gerrit WIP icon  https://review.openstack.org/16522817:32
*** sandywalsh has joined #openstack-infra17:32
*** Hefeweizen has quit IRC17:35
*** sdake has quit IRC17:35
*** ghostpl_ has joined #openstack-infra17:36
harlowja_does anyone know if the 2.6 virtualenvs got upgraded to 2.6 so that the following doesn't happen anymore;  http://logs.openstack.org/36/164836/8/check/gate-oslo-incubator-python26/5f7bf62/console.html17:36
harlowja_*upgrade to pip 6 i mean,17:36
clarkbfungi: figured out why I thought ssl thing was friday, reddit post title mistakenly said 19th is friday17:36
*** _nadya_ has quit IRC17:36
*** johnma has left #openstack-infra17:36
jogoI have jobs stuck in experimental from 15 hours ago17:37
clarkbharlowja_: they should use whatever latest virtualenv uses17:37
jogocheck-tempest-dsvm-f20 isn't running17:37
harlowja_clarkb hmmm, maybe thats to old :-/17:38
harlowja_due to '('Expected version spec in', 'debtcollector~=0.3.0', 'at', '~=0.3.0')'17:38
clarkbjogo: I don't think there are any more f20 nodes, that job should be removed17:38
zarojeblair: thanks17:38
* harlowja_ isn't sure how any of the 2.6 virtualenvs will work with those '~=' requirements if the pip version in it doesn't support that kind of version spec17:39
*** dhritishikhar has joined #openstack-infra17:39
clarkbharlowja_: it is also possible we haven't built the base image for centos6 tests recently enough to have picked up a newer virtualenv17:39
harlowja_or maybe the centos6 virtualenv still doesn't have pip > 617:39
*** tonytan4ever has joined #openstack-infra17:39
clarkbharlowja_: we install with pip so shouldn;t matter what centos6 does17:40
harlowja_kk17:40
jogoclarkb: thanks17:40
harlowja_clarkb then i guess its probably the case u describe17:40
jogoclarkb: the stackforge nova-docker job needs updating too17:40
jogodims_: ^17:40
jogono more f20 jobs17:40
jogoerr nodes17:40
*** coolsvap is now known as coolsvap|afk17:41
jeblairjogo: fungi was looking into that earlier...17:41
*** xyang1 has joined #openstack-infra17:42
clarkbharlowja_: the oldest centos6 image is ~75 hours old17:42
clarkbharlowja_: now to cross check against pypi17:42
*** arosen1 has quit IRC17:42
clarkblooks like pip6 has been in virtualenv since december17:43
clarkbso likely something else is going on17:43
harlowja_evilness i think, lol17:43
openstackgerritJoe Gordon proposed openstack-infra/project-config: There are no more f20 nodes, remove check-tempest-dsvm-f20  https://review.openstack.org/16553217:43
jogojeblair fungi: ahh17:43
jogoclarkb: ^ so like that?17:43
clarkbjogo: should remove the jenkins job too17:43
clarkbjogo: but that change will get the experimental queue moving17:44
clarkbjogo: I will defer to fungi on it though since he was looking into it?17:44
jeblairjogo: so, i mean, there are _supposed_ to be f20 nodes17:44
clarkboh, I thought we had removed them17:44
clarkbbecause f2117:44
*** bhuvan has joined #openstack-infra17:44
morganfainbergoh god. i just chased something down the pip requirement resolution rabbit hole...17:45
* morganfainberg backs away slooooowly17:45
* morganfainberg tries to save some sanity.17:45
pleia2good luck with that17:45
* SpamapS puts on a pip mask and chases morganfainberg 17:45
*** tnovacik has quit IRC17:46
* morganfainberg runs away from SpamapS.17:46
jogoclarkb: well even if there are supposed to be f20 nodes, why would we have an experimental f20 job on tempest if there is a f21 out there17:46
harlowja_go buy http://www.customink.com/designs/stackpip/qvh0-0015-grtw/ SpamapS lol17:46
SpamapSI NEED MY REQUIREMENTS BUT I DARE YOU TO TRY AND DETERMINISTICALLY RESOLVE THEM MUAHAHAHAHHA17:46
SpamapSNICE17:46
*** julim has quit IRC17:46
openstackgerritChelsea Winfree proposed openstack-infra/project-config: Move barbican client experimental job to voting gate job  https://review.openstack.org/16553417:46
SpamapSharlowja_: I lol'd17:47
jeblairjogo: i think the change is fine but maybe remove the 'no more nodes' thing from the commit message.17:47
harlowja_pippin def not easy, :-P17:47
clarkbharlowja_: "2015-03-18 14:27:24,432 INFO nodepool.image.build.rax-ord.bare-centos6: ESC[mNotice: /Stage[main]/Openstack_project::Base/Package[virtualenv]/ensure: ensure changed '12.0.7' to '1.11.4'ESC[0m" from a centos6 image build17:47
jogomorganfainberg: that's why we want to do 'pip install --no-deps -r list-of-all-deps-including-transitive.txt'17:47
harlowja_SpamapS u want, u saw a ghost?17:47
clarkbharlowja_: so I think thats the issue17:47
morganfainbergpleia2, yeah was looking at "how do i install the minimum requirements for my project" and realized... oh this means i have to solve the transient dependency issue ... then resolve the dependency things and... then.. OH LOOK SAME PROBLEM AS WE HAVE TODAY WE ARE TRYING TO SOLVE17:47
* morganfainberg kicks caps lock17:47
morganfainbergsorry17:47
morganfainbergjogo, haha17:48
openstackgerritEric Larson proposed openstack-infra/project-config: Create oslo.cache project  https://review.openstack.org/16522417:48
openstackgerritJoe Gordon proposed openstack-infra/project-config: emove check-tempest-dsvm-f20  https://review.openstack.org/16553217:48
morganfainbergjogo, i'm... scared :(17:48
harlowja_thats one hell of a downgrade; 12.0.7 -> 1.11.4, lol17:48
pleia2morganfainberg: caps sounded intentional to me :)17:48
morganfainbergpleia2 they actually weren't :P but they worked17:48
jogomorganfainberg: the minimal reqs thing, I think someone was working on that actually17:48
*** tqtran has joined #openstack-infra17:48
pleia2hehe17:48
openstackgerritJames E. Blair proposed openstack-infra/project-config: Remove check-tempest-dsvm-f20  https://review.openstack.org/16553217:48
jogojeblair: thanks :)17:48
morganfainbergjogo, oh cool.17:48
clarkbharlowja_: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/base.pp#n6317:49
*** ParsectiX has joined #openstack-infra17:49
jeblairjogo: np :)17:49
clarkbmordred: ^ any idea why we do virtualenv that way? you are all over the git log on those lines :)17:49
jogomorganfainberg: just don't remember who17:49
adam_gmorganfainberg, pip-compile does a good job of going down that rabbit hole and finding the conflicts for you, or at least describing why you're ending up with what you're getting17:49
morganfainbergjeblair, jogo, i think "emove" was cooler -- hip like.17:49
morganfainbergjeblair, jogo, :P17:49
SpamapSharlowja_: also http://www.zazzle.com/i_unicode_tshirt-23529543844571576817:49
elarsonpleia2: thanks for reviewing the oslo.cache files. I didn't know they needed to be in alpha order17:49
clarkbharlowja_: so, I think it may be a puppet order of ops thing. If that downgrade happens before we install latest virtualenv then it works fine, if that downgrade happens after then we lose17:49
*** derekh has quit IRC17:49
pleia2elarson: no worries, it looked pretty good otherwise17:49
morganfainbergadam_g, yeah i would still need to solve the "what is the lowest versions for everything" question. it's a scary rabbit hole to dive down.17:50
harlowja_SpamapS :-P17:50
morganfainbergadam_g, but i'm poking at some ideas for making solving this less... icky. so i think i'm going to spend time on the plane chasing some related rabbits.17:50
morganfainbergor being chased by rabbits with sharp pointy teeth.17:51
openstackgerritChelsea Winfree proposed openstack-infra/project-config: Move barbican client experimental job to voting gate job  https://review.openstack.org/16553417:51
harlowja_clarkb hmmm, damn u puppet17:51
*** ChuckC has quit IRC17:51
jogomorganfainberg: I think Richard Jones may have looked into it17:51
*** ChuckC_ has joined #openstack-infra17:51
adam_gmorganfainberg, ya, sounds scary17:51
*** ChuckC_ is now known as ChuckC17:52
clarkbharlowja_: I need to step out for a few, but I think next step is figuring out why it works for eg trusty but not centos6 then fixing it across the baord17:52
harlowja_agreed17:52
morganfainbergadam_g, it looks an awful lot like this rabbit: https://41.media.tumblr.com/9ff3be3c6eeb35495d483e21f1ff1ed0/tumblr_n499zplqcj1qgsnbzo5_500.jpg17:52
clarkbharlowja_: but that line I linked is probably the center of the issue17:52
harlowja_ya, 1.1.x probably a little to old17:52
morganfainbergadam_g, i should probably run away.17:52
SpamapSmorganfainberg: it's got huge nasty pointy methods17:52
morganfainbergSpamapS, exactly!17:53
SpamapSLOOK AT THE BUGS17:53
SpamapShere's how to waste an hour, but then decide it's time to hug your tests for finding this: http://paste.ubuntu.com/10622270/17:54
SpamapSread carefuly. ;)17:54
clarkbSpamapS: apply caffeine direclty to your forehead17:55
openstackgerritKhai Do proposed openstack-infra/system-config: Fix jquery setup on Gerrit server.  https://review.openstack.org/16514517:55
jeblairSpamapS disproves the universal equivalency theorem17:56
*** notnownikki has quit IRC17:56
*** lucasagomes has left #openstack-infra17:56
SpamapSclarkb: http://us.123rf.com/450wm/stylephotographs/stylephotographs1205/stylephotographs120500200/13838234-attractive-manager-getting-needle-in-forehead-for-wrinkle-removal.jpg17:57
SpamapSsort of like that?17:57
*** sarob has joined #openstack-infra17:57
jeblairi think i will not follow that link17:57
harlowja_apply directly to the forehead17:57
harlowja_lol17:57
*** garyh has joined #openstack-infra17:58
*** ddieterly has quit IRC17:58
*** ddieterly has joined #openstack-infra17:58
*** e0ne has joined #openstack-infra17:59
greghaynesclarkb: what was the other high priority for nodepool testing? node / image deletion?18:00
*** crc32 has joined #openstack-infra18:00
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Removed grunt:server  https://review.openstack.org/16316518:01
*** ayoung is now known as ayoung-afk18:02
*** sarob has quit IRC18:02
*** mrmartin has quit IRC18:02
*** jcoufal has quit IRC18:02
dims_jogo: need me to do something?18:02
jogodims_: docker job runs f2018:03
jogoand we may or may not be moving to f21 for everything18:03
dims_jogo: ack, if it breaks, i'll fix it18:03
jogodims_: I thought we were dropping f20 all together but it may have not happened yet18:03
jogodims_: its not running right now18:03
jogofungi: ^18:03
*** AJaeger has quit IRC18:03
jogodims_: check-tempest-dsvm-f20-docker: queued (non-voting)18:04
jogo22 hours and counting18:04
*** sdake has joined #openstack-infra18:04
dims_jogo: url?18:04
*** sarob has joined #openstack-infra18:05
*** jyuso1 has joined #openstack-infra18:05
jogodims_: look on http://status.openstack.org/zuul/ and filter by docker18:06
openstackgerritgreghaynes proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492718:06
jogopatch 16519618:06
*** Hefeweizen has joined #openstack-infra18:06
dims_jogo: right, not sure if there's a way for me to nuke that one18:06
jogodims_: not sure either, but wondering if its time to move that job to f2118:07
jogobut not sure anymore18:07
dims_jogo: ack. will wait for fungi or someone to chime in :)18:07
*** sdake_ has quit IRC18:07
*** ParsectiX has quit IRC18:07
*** ParsectiX has joined #openstack-infra18:08
*** yamamoto has quit IRC18:08
*** ssam2 has quit IRC18:09
*** otter768 has joined #openstack-infra18:10
*** LinuxJedi is now known as LinuxJedi_18:11
*** Sukhdev has joined #openstack-infra18:11
*** sarob has quit IRC18:11
*** otter768 has quit IRC18:14
*** hdd has quit IRC18:16
clarkbgreghaynes: ya node and image deletion are completely uncovered right now18:17
*** SumitNaiksatam has joined #openstack-infra18:18
openstackgerritSean Dague proposed openstack-infra/os-loganalyze: fix supports_sev matching  https://review.openstack.org/16554218:18
fungidims_: jogo: clarkb: jeblair: back from eats now and looking18:20
*** Somay has joined #openstack-infra18:21
*** SumitNaiksatam has left #openstack-infra18:21
clarkbhttps://review.openstack.org/#/c/165542/ is a fix for indexing logs like gate-horizon-python27 console logs18:22
*** tonytan4ever has quit IRC18:22
*** SumitNaiksatam has joined #openstack-infra18:22
Somayclarkb: I have updated the patch as you mentioned, please review whenever you’re free https://review.openstack.org/#/c/16313718:22
clarkbharlowja_: looking at logs for trusty iamge builds we don't seem to downgrade at all18:23
harlowja_centos6 is just 'special'18:23
harlowja_lol18:23
*** mrmartin has joined #openstack-infra18:24
Somaythanks clarkb18:26
*** HeOS has joined #openstack-infra18:26
*** ildikov has joined #openstack-infra18:26
*** mrmartin has quit IRC18:27
fungijogo: clarkb: we're still apparently trying to build devstack-f20 nodes at the moment, but it's either happening very slowly or all the builds are failing for some reason (there are currently 5 building for between 0.17 and 0.83 hours)18:27
*** Somay has quit IRC18:28
*** hdd has joined #openstack-infra18:28
*** ayoung-afk is now known as ayoung18:30
*** afazekas has quit IRC18:31
*** BharatK has left #openstack-infra18:31
*** patrickeast has joined #openstack-infra18:33
*** hdd has quit IRC18:33
*** dannywilson has joined #openstack-infra18:35
jeblairfungi: yeah, looks like build timeouts: Exception: Timeout waiting for server 14cbd68b-773d-48f3-aedb-57180f303557 in rax-iad18:35
fungijeblair: yeah, that's what i just turned up too. i've seen that happen when the image can't boot successfully too18:36
jeblairfungi: and that's iad and dfw18:36
*** dustins has quit IRC18:36
fungii can try deleting today's images18:36
fungilooks like we have some successful image builds from yesterday18:37
jeblairfungi: these are in 'waitforServer', so the instance is never coming online (ie, it's an openstack problem, not a linux booting problem)18:37
*** jyuso1 has quit IRC18:37
*** dustins has joined #openstack-infra18:38
fungioh. yeah that's less fun. if we're also getting it in the same provider regions for other image types, then it's probably an issue in rackspace18:38
jeblairfungi: i only looked at that one type18:38
openstackgerritMerged openstack-infra/project-config: Don't dump Jenkins proxy errors to console logs  https://review.openstack.org/16552218:38
fungii'm broadening that grep now18:38
fungiproportionally highest in tripleo-test-cloud-rh1 actually18:40
fungii have to assume tripleo is having a bad day18:40
fungion the whole, fairly widespread in hpcloud and rax too though18:40
*** tiswanso has joined #openstack-infra18:41
fungioh, wait, i was getting deletion in there too18:41
fungi29 in rax-iad, 4 in rax-ord, 2 in rax-dfw and 1 in hpcloud-b5 so yes probably predominately devstack-f2018:42
*** garyh has quit IRC18:44
fungialso, yeah probably not today's image's fault anyway because many of those jobs have been hanging for longer than its existed18:44
*** tiswanso has quit IRC18:45
*** tiswanso has joined #openstack-infra18:45
fungistatus.rackspace.com has notices up for capacity problems in those regions for their "new compute optimize flavors" but i don't think we're using those18:46
*** dprince has quit IRC18:46
*** LinuxJedi_ is now known as LinuxJedi18:46
*** yfried has quit IRC18:47
fungii'll check for and clean up any alien nodes just in case18:47
*** pelix has quit IRC18:47
*** achanda has quit IRC18:48
*** garyh has joined #openstack-infra18:50
*** tonytan4ever has joined #openstack-infra18:50
*** julim has joined #openstack-infra18:50
*** dustins has quit IRC18:50
*** asahlin has joined #openstack-infra18:51
mordredclarkb: looking18:52
*** stevemar has quit IRC18:53
dougwigjeblair: fyi on gerrty, those fixes on master have made things quite usable again.18:53
*** stevemar has joined #openstack-infra18:53
jeblairdougwig: oh good.  please let me know if you find it hanging for a really long time (and where).18:54
*** resker has joined #openstack-infra18:54
dougwigjeblair: will do, thanks.18:55
jeblairdougwig: someday i hope we can get rid of all of the hangs, but it will require getting smarter about the database.  in the mean time, i do know of a few low hanging fruits i think i can pick off and save a few seconds.18:55
*** koolhead17 has joined #openstack-infra18:55
*** resker has quit IRC18:56
*** koolhead17 has quit IRC18:56
*** koolhead17 has joined #openstack-infra18:56
*** koolhead17 has quit IRC18:56
*** koolhead17 has joined #openstack-infra18:56
*** resker has joined #openstack-infra18:56
*** LinuxJedi is now known as LinuxJedi_18:57
*** esker has quit IRC18:57
clarkbhttps://review.openstack.org/#/c/165542/ would be a good one to get in to correct indexing failures for jobs that inadverdently match the current regex with their full console log paths18:58
*** sarob has joined #openstack-infra18:58
*** dustins has joined #openstack-infra18:58
*** dhritishikhar has quit IRC18:59
*** ghostpl_ has quit IRC18:59
*** dprince has joined #openstack-infra19:00
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: DO NOT MERGE COPY ALL OF .TESTREPOSITORY to log server  https://review.openstack.org/14971919:01
*** achanda has joined #openstack-infra19:01
*** LinuxJedi_ is now known as LinuxJedi19:01
*** _nadya_ has joined #openstack-infra19:03
*** sandywalsh has quit IRC19:03
*** sarob has quit IRC19:03
*** alexpilotti_ has quit IRC19:04
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add nodepool config-validation environment  https://review.openstack.org/16490119:04
*** liusheng has quit IRC19:05
*** andreykurilin_ has joined #openstack-infra19:06
*** liusheng has joined #openstack-infra19:06
*** emagana has quit IRC19:06
*** marun has quit IRC19:07
*** ivar-lazzaro has joined #openstack-infra19:07
SlickNikPlease can I get some eyeballs on https://review.openstack.org/#/c/165161/  when folks get a chance? Thanks!19:07
*** yamamoto has joined #openstack-infra19:09
*** emagana has joined #openstack-infra19:09
mroddenso did they start doing the ATC cutoff for the current release for this summit?19:09
mroddenor was that last time too19:09
jeblairSlickNik: that's kind of a red flag -- we've tried to have a budget of about 1 hour for gate tests19:10
*** cburgess has quit IRC19:10
fungimrodden: yes, this summit you need to have landed a contribution to an official project since the latest release. the previous summit was over the two prior dev cycles rather than just one19:10
mroddenshoot19:10
mroddendidn't realize that...19:10
*** cburgess has joined #openstack-infra19:11
* fungi counts months on his fingers...19:11
*** neoice has quit IRC19:11
*** neoice has joined #openstack-infra19:11
fungithe last release was a while ago. you've probably contributed something in that time. at least a typo correction?19:11
mroddenare you saying that "active contribution window" is still open?19:11
fungiyeah19:11
mroddenok19:11
jeblairmrodden: and patches to infra count ;)19:11
mroddenyeah i have patches up but never merged lol19:12
mroddenbashate is pretty stable19:12
mtreinishmrodden: how could the bash8 ptl not be an atc?19:12
mroddeninorite?!19:12
mroddenits been a quiet cycle for me19:12
mroddeni'll go fix up some patches19:12
jeblairmrodden: hah, your project is first to be completed!19:12
mroddenfeature complete at 0.4.019:12
fungiblue ribbon!19:12
*** MarkAtwood has quit IRC19:13
*** yamamoto has quit IRC19:13
*** Somay has joined #openstack-infra19:13
SlickNikjeblair: Yes, I know we've got  some work to do to reduce the run time of those tests — and I need to prioritize that higher.19:14
SlickNikjeblair: It's just that we've been getting a lot of slower instances of late and the tests have been timing out with around ~5 minutes left.19:14
SlickNikAnd that's been causing a lot of pain for patches to get through the gate — resulting in a lot of rechecks.19:15
*** yfried has joined #openstack-infra19:15
*** MarkAtwood has joined #openstack-infra19:15
funginote, a lot of rechecks is probably how the runtime bloat gets in there to begin with. it may not be that you're getting really slow workers, but that the changes adding extra runtime lucked out and got really fast workers after enough rechecking19:16
clarkbmordred: harlowja_ I cannot find any other place we might be installing virtualenv. My only theory is that maybe trusty has a newer virtualenv and pip refuses to downgrade on trusty?19:16
*** tkelsey has quit IRC19:17
fungiclarkb: regardless i think the virtualenv cap there is a relic of the bad old times from december and we should undo that19:17
clarkbfungi: ya19:17
SlickNikfungi: point taken — Tryng to figure out the best way to get away from this anti-pattern of multi-rechecks.19:17
dougwighi infra, any project-config cores (besides anteaya) that can take a quick look at this new neutron/lbaas job?  we have a bunch of new third-party CI's being created, and I'd like to be front and center what they're supposed to be running.  https://review.openstack.org/#/c/164390/19:18
*** bswartz has quit IRC19:19
mordredclarkb, fungi: ++19:19
mtreinishSlickNik: maybe split the jobs up into smaller chuncks that are quicker? (hasn't looked at the patch, just grasping at context from backscroll)19:19
clarkbdougwig: not sure I follow what that change has to do with third party ci19:20
fungiSlickNik: we have tried to strike a balance between noise from multiple check runs and indicating how many times they've run in the gerrit interface. note it indicates the number of rechecks in the ci results summary heading if there's more than one on the latest patchset19:20
openstackgerritMerged openstack-infra/nodepool: Add config validation option  https://review.openstack.org/16195219:21
*** ghostpl_ has joined #openstack-infra19:21
fungiSlickNik: for example https://review.openstack.org/155474 has "Jenkins check (1 rechecks)" there19:22
SlickNikfungi: Yes, I've seen that — it helps a lot, actually.19:22
*** ghostpl_ has quit IRC19:22
*** garyh has quit IRC19:22
clarkbdougwig: was there a specific reason to use the non capturing rnach regex?19:22
fungiso when you see that, toggle the ci comments and look at the earlier checks for warning signs19:22
SlickNikfungi: I haven't yet made the jump of being able to base my code reviews on that information.19:22
dougwigclarkb: i copied the neutron api job, so no specific reason beyond that.19:23
* SlickNik sees a lightbulb go off 19:23
SlickNikThat's a good warning sign to look at.19:24
*** salv-orlando has quit IRC19:24
*** spzala has quit IRC19:24
dougwigclarkb: ty19:24
*** LinuxJedi is now known as LinuxJedi_19:25
*** teran has quit IRC19:26
*** bswartz has joined #openstack-infra19:26
*** johnthetubaguy is now known as zz_johnthetubagu19:27
SlickNikmtreinish: That's a good idea (splitting that test job up). Will look into doing that so that we can make these runs faster. Hopefully we can bring that number down to <= 60.19:27
*** andreykurilin_ has quit IRC19:27
jeblairSlickNik: i'd like to get a review from sdague on 16516119:28
mtreinishSlickNik: fwiw, that's what we do in tempest for things that we've identified as slow. We regex exclude them from the gating jobs and only run them periodically and in the experimental queue19:29
mtreinishwhich is a little different from what you'd do, but it would give you a starting point for how to split things19:29
*** nikesh_vedams_ has joined #openstack-infra19:29
nikesh_vedams_hi19:30
sdaguejeblair: hmmm for most of that stuff we usually bump by > 10 minutes19:30
*** salv-orlando has joined #openstack-infra19:30
nikesh_vedams_my CI is up and running19:30
nikesh_vedams_but i want to update this https://wiki.openstack.org/wiki/ThirdPartySystems/Vedams_SCST_CI19:30
*** mrunge_away is now known as mrunge19:30
nikesh_vedams_how can i update this19:31
*** garyh has joined #openstack-infra19:31
*** doug-fish has left #openstack-infra19:32
sdaguemtreinish: manually splitting the job really only makes it take longer as you have extra overhead for all runs19:32
Ngmordred: I'm having an interesting time getting glanceclient to talk to Rax. The official word coming back from their support people is that it just plain doesn't work because of their special authentication method, which leaves me wondering how on earth you managed to do it19:32
openstackgerritMerged openstack-infra/project-config: Promote lbaasv2 api job to both the lbaas and neutron check queues  https://review.openstack.org/16439019:32
nikesh_vedams_DuncanT: there?19:32
*** MarkAtwood has quit IRC19:33
funginikesh_vedams_: the login link in the top-right corner of that page19:33
mordredNg: that is completely false19:33
*** eharney has quit IRC19:33
*** Somay has quit IRC19:33
funginikesh_vedams_: and then the edit button in the top navigation bar once you're logged in19:33
mordredNg: you must set export OS_IMAGE_API_VERSION=219:33
mtreinishsdague: well maybe it depends on how long things are really taking there is a point where the double setup cost is worth it. But, like I said I didn't actually look at anything it was just a blind idea19:33
jeblairmordred: is the 'endpoint not in catalog' problem fixed?19:34
jeblairNg: ^ potentially related to your issue19:34
mordredjeblair: yes - that's not a problem19:34
mordredjeblair: you need latest glanceclient19:34
jeblairok, neato.19:34
*** dizquierdo has quit IRC19:35
mordredNg, jeblair: these are what you need set for it to work from python-glanceclient19:35
mordredhttp://paste.openstack.org/show/19324119:35
greghaynesclarkb: What kind of node deletion failure do we encounter? Is it that node deletion fails and we fail to create new ones or just that node deletion fails and we fail to delete new ones, or something different19:36
greghaynes(or if someone else knows :))19:37
*** boris-42 has quit IRC19:37
*** edwarnicke has quit IRC19:37
SlickNikI'd appreciate it if we could bump by 10 mins for now — I will ensure that we make a concerted effort past the feature freeze to reduce this. I'm pretty sure that there is at lease a couple of things we can do with the functional tests that would help speed them up (probably by 10 minutes or so).19:37
jeblairSlickNik: done19:38
*** juggler has joined #openstack-infra19:38
*** jbryce has quit IRC19:38
openstackgerritMerged openstack-infra/system-config: Add Fedora 21 dib image  https://review.openstack.org/16398219:38
jeblairSlickNik: (i left a comment in a similar spirit to what you just said)19:38
SlickNikIdeally I'd like to get to <60 mins, but that will take a bit more redesign, which we will also look into, but might take a bit longer.19:38
*** Somay has joined #openstack-infra19:39
*** Ng has quit IRC19:39
*** stevebaker has quit IRC19:39
*** erlon has quit IRC19:39
*** juggler_ has quit IRC19:39
*** krtaylor has quit IRC19:39
*** comstud has quit IRC19:39
*** cyeoh has quit IRC19:39
*** lbragstad has quit IRC19:39
*** rainya has quit IRC19:39
clarkbgreghaynes I am not sure we get failures that are due to nodepool. mostly thinking we should test to enforce behavior with shade switch coming up19:39
*** jbryce has joined #openstack-infra19:39
*** Ng has joined #openstack-infra19:39
*** juggler_ has joined #openstack-infra19:40
greghaynesok, ill just hit a few common cases then19:40
clarkbusually failure is due to cloud then we retry with periodic cleanup cron thing19:40
*** rainya has joined #openstack-infra19:40
*** stevebaker has joined #openstack-infra19:40
*** lbragstad has joined #openstack-infra19:40
SlickNikjeblair: Thanks much!19:40
*** comstud has joined #openstack-infra19:41
*** erlon has joined #openstack-infra19:41
openstackgerritMerged openstack-infra/project-config: Create devstack-plugin-cookiecutter project  https://review.openstack.org/16188719:41
*** krtaylor has joined #openstack-infra19:41
*** cyeoh has joined #openstack-infra19:41
*** rfolco has quit IRC19:41
*** gyee has quit IRC19:41
*** edwarnicke has joined #openstack-infra19:41
*** boris-42 has joined #openstack-infra19:42
*** LinuxJedi_ is now known as LinuxJedi19:42
*** juggler has quit IRC19:42
*** Somay has quit IRC19:43
*** timcline has quit IRC19:44
mtreinishSlickNik: looking at the trove-integration suite you guys are still running that serially right? If you guys work on getting things running in parallel, that should speed things up a bit19:44
*** Somay has joined #openstack-infra19:45
*** tnovacik has joined #openstack-infra19:45
*** timcline has joined #openstack-infra19:45
*** doug-fish has joined #openstack-infra19:45
openstackgerritMerged openstack-infra/project-config: Increase trove gate job timeout  https://review.openstack.org/16516119:46
mmedvedejeblair, pleia2: any chance to get  https://review.openstack.org/#/c/155864/ approved? It is the last remaining patch from third party documentation sprint.19:46
SlickNikmtreinish: Yes! That's probably one of the first things that we'll  start looking into.19:46
jeblairmmedvede: that's merged19:47
mmedvedesorry, https://review.openstack.org/#/c/149657/19:47
mmedvedejeblair, pleia2 ^19:47
*** wenlock has quit IRC19:48
SlickNikmtreinish: Was thinking to start off with a new job that has (at least the) new tests running in parallel.19:48
pleia2mmedvede: upgraded my vote19:48
jeblairmmedvede: so we have people come into the channel all the time with problems because they follow some of those instructions; i hesitate to link to something we know is wrong.19:48
SlickNikAnd slowly migrate from the old serial way of running the tests to the new job.19:48
mtreinishSlickNik: depending on what the tests are doing it can be kinda tricky. I spent like 2-3months getting tempest working in parallel19:49
mmedvedejeblair: I see, I have no problem abandoning it19:49
SlickNikThat way it doesn't have to be all or nothing, and we can make quantized progress.19:49
jeblairmmedvede: (this is why i'm so excited about the puppet-openstackci module which should not suffer from bitrot)19:49
pleia2that was talked about at the sprint (both maintainability and the fact that we know they're wrong), I left a related comment in the review, consensus tended to be "it's the best we have at this point"19:49
mtreinishSlickNik: honestly I think a better approach is to setup a 2nd job that runs everything in parallel abd try to get that consistent19:49
pleia2but I guess linking to it does give it more "official-ness" which perhaps we want to avoid19:50
*** wenlock has joined #openstack-infra19:51
openstackgerritgreghaynes proposed openstack-infra/nodepool: Move nodepool creation in tests to common method  https://review.openstack.org/16558119:51
openstackgerritgreghaynes proposed openstack-infra/nodepool: Add node deletion tests  https://review.openstack.org/16558219:51
jeblairpleia2: yeah, maybe an intro paragraph would help, but also "We know these are wrong and get out of date, but here they are anyway, good luck." isn't really something i'd be particularly proud of either.  :)19:51
mmedvedejeblair, pleia2 : I am excited about puppet-openstackci too. Ok would abandon. Thanks19:52
*** sdake_ has joined #openstack-infra19:52
pleia2jeblair: that's fair :)19:53
*** crc32 has quit IRC19:53
SlickNikmtreinish: Perhaps that's the way to go — I suspect I might find the same once I actually start down the path of trying to parallelize the tests.19:53
*** willroberts_ has left #openstack-infra19:53
*** hasharAway is now known as hashar19:54
*** Rockyg has joined #openstack-infra19:54
*** devlaps has joined #openstack-infra19:55
*** zigo__ has joined #openstack-infra19:56
*** zigo has quit IRC19:56
*** sdake has quit IRC19:56
*** rainya has quit IRC19:57
*** roaet_ has quit IRC19:57
*** sigmavirus24 is now known as sigmavirus24_awa19:57
*** rainya has joined #openstack-infra19:58
*** subscope_ has joined #openstack-infra19:58
*** juggler has joined #openstack-infra19:58
*** roaet_ has joined #openstack-infra19:58
nikesh_vedams_fungi: done thanks19:59
*** sarob has joined #openstack-infra19:59
*** juggler has quit IRC19:59
gordchi, we're seeing "Cloning not allowed in this configuration" errors in our ceilometermiddleware gate, is it because this patch needs to be merged? https://review.openstack.org/#/c/15706019:59
*** juggler has joined #openstack-infra19:59
*** juggler_ has quit IRC19:59
*** achanda has quit IRC20:00
*** juggler_ has joined #openstack-infra20:01
*** emagana has quit IRC20:01
nikesh_vedams_fungi: "Do you have a Gerrit CI account created for you by the Infrastructure team" i have a launchpad account dedicated to CI but do i have to create account by infra team20:01
nikesh_vedams_?20:01
pleia2nikesh_vedams_: we don't create CI accounts for people anymore, it's self-service, where did you find that documentation?20:02
nikesh_vedams_https://wiki.openstack.org/wiki/ThirdPartySystems20:02
pleia2nikesh_vedams_: creating a service account docs: http://ci.openstack.org/third_party.html#creating-a-service-account20:02
nikesh_vedams_in the end20:02
openstackgerritgreghaynes proposed openstack-infra/nodepool: Move nodepool creation in tests to common method  https://review.openstack.org/16558120:03
pleia2nikesh_vedams_: ah, that's only if you already have an account and want to make a change to it, if you don't have an account that's fine :)20:03
pleia2I mean, you do need an account for 3rd party, you just don't need to worry about that statement20:04
*** sarob has quit IRC20:04
*** _nadya_ has quit IRC20:04
*** juggler has quit IRC20:04
clarkbgordc yes you need 157060, you can use depends on20:04
nikesh_vedams_pleia2: ok thanks,,now my CI is up and running and 19 th march is deadline so do i have to report someone that my CI is up and running?20:05
openstackgerritgreghaynes proposed openstack-infra/nodepool: Add node deletion tests  https://review.openstack.org/16558220:05
nikesh_vedams_nikesh_vedams:we have contributed SCST code in cinder20:05
pleia2nikesh_vedams_: what project has that deadine? you should talk to them :)20:05
nikesh_vedams_in cinder20:05
nikesh_vedams_k-320:06
*** dprince has quit IRC20:06
gordcclarkb: ah cool cool. i was wondering why i needed that patch.20:07
gordcthanks fungi20:07
fungigordc: yeah, devstack couldn't find a clone of that project but wanted one, so thought it was going to have to clone it remotely itself, which we disallow on the workers20:08
*** teran has joined #openstack-infra20:08
funginikesh_vedams_: you should definitely get more involved with the cinder developers. it's their deadline, so they're the ones you want to make aware of that20:09
gordcfungi: ah i see, good to know.20:10
*** rfolco has joined #openstack-infra20:10
nikesh_vedams_fungi:thanks20:10
*** Sukhdev has quit IRC20:11
*** otter768 has joined #openstack-infra20:11
*** spzala has joined #openstack-infra20:12
*** tiswanso has quit IRC20:13
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Allow HTTPS through iptables on zuul.openstack.org  https://review.openstack.org/16558420:14
*** openstackgerrit has quit IRC20:14
*** openstackgerrit has joined #openstack-infra20:15
*** dprince has joined #openstack-infra20:15
lifelessdhellmann: mail replied20:15
*** otter768 has quit IRC20:15
*** kiseok7_ has quit IRC20:16
*** changbl has joined #openstack-infra20:17
*** eharney has joined #openstack-infra20:18
*** aysyd has quit IRC20:19
*** mrmartin has joined #openstack-infra20:20
*** Somay has quit IRC20:21
*** aysyd has joined #openstack-infra20:21
*** sigmavirus24_awa is now known as sigmavirus2420:22
*** prad has quit IRC20:23
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Initial files for the puppet-openstackci repo  https://review.openstack.org/16558820:25
*** tkelsey has joined #openstack-infra20:25
*** maulermark has joined #openstack-infra20:25
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Initial files for the puppet-openstackci repo  https://review.openstack.org/16558820:26
*** ihrachyshka has joined #openstack-infra20:26
*** teran_ has joined #openstack-infra20:27
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Initial files for the puppet-openstackci repo  https://review.openstack.org/16558820:29
*** tkelsey has quit IRC20:29
asselinok, sorry for noise, but first patch is up! :) ^^20:29
*** teran has quit IRC20:30
*** Bsony has quit IRC20:34
fungijeblair: clarkb: ianw: was there any other wrapup we wanted to do before tagging a new nodepool release? apparently 164901 shouldn't merge until we do, and the change it needs is already merged to master20:35
jeblairfungi: oh hrm20:36
clarkbgreghaynes has a bugfix for delete image20:37
jeblairfungi: actually, i wonder if the right answer there is 'use nodepool from git'20:37
fungiheh, release schmelease20:37
jeblairfungi: because we run nodepool from git, so i think our config validator should match it20:37
fungiyeah, i'm game20:37
*** radez is now known as radez_g0n320:38
*** Bsony has joined #openstack-infra20:38
*** dkranz has quit IRC20:39
*** adalbas has quit IRC20:41
nibalizerooo20:41
greghaynesDoes that mean we want to also test config validator for project-config per nodepool commit?20:42
*** juggler_ has quit IRC20:42
fungipotentially, since that would validate changes to the validator20:42
*** juggler_ has joined #openstack-infra20:42
jeblairgreghaynes: we probably should (though we may want to make it non-voting).  (though it's actually system-config, not project-config)20:43
greghaynesRight, the repo with the things ;)20:43
jeblairgreghaynes: we do for jjb, but we don't for zuul20:43
jeblairgreghaynes: it kind of depends on the rate of change, risk, etc20:43
greghaynesYea, I dont think its a requirement for running off git - its not better or worse in that dept but just something to do20:44
asselinI added a bunch of stories to storybaord and half are now missing: https://storyboard.openstack.org/#!/story/2000101 ??20:46
pleia2asselin: do you mean tasks to a story?20:46
jeblairasselin: you may need to log into storyboard and set your preferences to display 100 items per page20:47
asselinpleia2, yes, a but of tasks20:47
*** mrda-away is now known as mrda20:47
asselinjeblair, thanks that was it!20:47
jeblairasselin: yeah, storyboard server pages results but the client doesn't actually support it :(20:48
*** achanda has joined #openstack-infra20:48
asselinok I see20:48
*** MarkAtwood has joined #openstack-infra20:49
*** nikesh_vedams_ has quit IRC20:52
*** sdake has joined #openstack-infra20:52
*** ParsectiX has quit IRC20:52
*** ParsectiX has joined #openstack-infra20:53
*** peristeri has quit IRC20:55
*** sdake_ has quit IRC20:56
*** alexpilotti has joined #openstack-infra20:56
*** mrmartin has quit IRC20:59
*** sarob has joined #openstack-infra21:00
openstackgerritKhai Do proposed openstack-infra/system-config: Replace the Gerrit WIP icon  https://review.openstack.org/16522821:04
*** hdd has joined #openstack-infra21:05
*** sarob has quit IRC21:05
*** Bsony has quit IRC21:06
*** mattfarina has quit IRC21:07
*** mattfarina has joined #openstack-infra21:08
*** mattfarina has quit IRC21:08
*** MarkAtwood has quit IRC21:09
dstuffthm, I keep getting emails from gerrit but I'm not subscribed to any projects21:09
dstuffthow do I make them stop :(21:09
*** yamamoto has joined #openstack-infra21:11
*** MarkAtwood has joined #openstack-infra21:11
*** banix__ has quit IRC21:13
ianwfungi: yeah, was going to ask about a release ... otherwise i can tag it with a specific git21:13
ianwsorry, reading scrollback ...21:13
*** ldnunes has quit IRC21:13
fungidstufft: you're probably subscribed to individual changes you've commented on21:13
*** dprince has quit IRC21:14
jeblairianw: instead of installing that way, we may want to do what we do with the zuul layout check and always install the latest commit in git21:14
fungidstufft: or changes people have added you as a requested reviewer on21:14
ianwjeblair: yeah, we almost certainly do, since we don't want to release every time we add a config option, let me fix that up21:14
dstufftfungi: hmm21:14
dstufftfungi: is there something like "all core get added as ar eviewer"21:14
funginothing automated, no21:15
fungidstufft: honestly i /dev/null just about everything i receive from gerrit and have some selective filters which just deliver a handful of useful messages to me21:15
*** yamamoto has quit IRC21:15
fungiit's a bit on the noisy side21:15
dstufftI guess I should set something up21:15
dstufftI don't use email filters normally21:15
*** kgiusti has left #openstack-infra21:15
*** ihrachyshka has quit IRC21:16
fungii would go insane if i didn't. so many mailing lists21:16
dstufftI read all of the MLs I subscribe too21:16
dstufftat least skim the subjects21:16
openstackgerritKhai Do proposed openstack-infra/system-config: Fix jquery setup on Gerrit server.  https://review.openstack.org/16514521:16
fungisame here, but sorting them into buckets helps me prioritize which ones i read first21:16
dstufft(this got a lot harder to accomplish once I subscribed to openstack-dev)21:16
jeblairdstufft: https://gerrit-review.googlesource.com/#/c/58322/  :(21:16
jeblairzaro: ^ thanks for reviewing that! :)21:17
fungiopenstack-dev used to be a lot quieter than debian-devel or cypherpunks21:17
fungibut it has grown in the past year or so21:17
*** Swami has quit IRC21:17
dstufftI'm apparently added a reviewer on all of these things21:17
openstackgerritBob Ball proposed openstack-infra/devstack-gate: XenAPI: Highlight that eth4 does not exist outside the Citrix environment  https://review.openstack.org/16560721:18
*** BobBall is now known as BobBall_AWOL21:18
dstufftI should probably figure out how to stop being a core reviewer on barbican21:18
fungidstufft: i think you can click the little x button next to your name on the ones where you don't want to be subscribed? not sure if that's an admin-only feature21:18
jeblairfungi: i believe you can remove yourself always21:18
fungiyeah, that's what i thought21:19
fungijust hard for me to tell easily because... admin21:19
*** ChuckC has quit IRC21:19
*** hdd has quit IRC21:19
*** bswartz has quit IRC21:19
*** teran__ has joined #openstack-infra21:20
*** teran_ has quit IRC21:20
openstackgerritClaudiu Belu proposed openstack-infra/project-config: Adds compute-hyperv in StackForge  https://review.openstack.org/16561121:21
*** ParsectiX has quit IRC21:21
*** ParsectiX has joined #openstack-infra21:21
openstackgerritIan Wienand proposed openstack-infra/system-config: Add nodepool config-validation environment  https://review.openstack.org/16490121:23
ianwjeblair: ^ grab nodepool from git master21:24
zarojeblair: is that a priority?  i think david p. just wants a manual verification of that before he approves it, which i can do.21:24
dstufftfungi: being an admin on things is kinda funny21:24
dstufftfungi: fun fact, if you're an admin on PyPI there's basically no system in place to prevent you from uploading to other people's projects21:24
dstufftalso there's no super easy way to really tell if someone else has a name without attempting to upload :V21:25
*** teran has joined #openstack-infra21:25
fungidstufft: yeah, on a lot of my personal systems i have two accounts for myself and only ever use the admin-capable one when i need to perform administrative functions21:26
jeblairdstufft: yeah, gerrit is actually really good about separating the _important_ things -- we can't force-merge changes or review outside our designated ranges without doing extra work to elevate our privileges.  it's the little things where they didn't bother as much.21:26
dstufftluckily the only people i uplaoded overtop of their own projects didn't get too mad D:21:26
*** __TheDodd__ has joined #openstack-infra21:27
dstufftnow I just psql into the DB and flip the admin bit to "sudo" myself21:27
*** teran__ has quit IRC21:27
*** Swami has joined #openstack-infra21:27
jeblairzaro: i don't think it's super urgent (it's been this way for years).  but it is a good idea.21:27
*** e0ne has quit IRC21:28
*** thedodd has quit IRC21:28
jeblairdstufft: careful with that db.  ;)21:28
*** mriedem is now known as mriedem_away21:28
dstufftjeblair: manually writing SQL like it aint no thang21:29
dstufftPyPI doesn't have an admin interface either21:29
mesteryHowdy infra folks! Looking at zuul, the top of the check queue is filled with neutron jobs all waiting for a node to run the check-neutron-lbaas-dsvm-api job. Something seems fishy here to me. /cc dougwig21:29
dstufftSQL is basically the admin interface21:29
* mestery wonders what is so special about that job that's causing it to backup neutron jobs in the check queue21:29
*** jamielennox is now known as jamielennox|away21:31
*** LinuxJedi has quit IRC21:31
jeblairmestery: i'll take a look21:32
*** LinuxJedi has joined #openstack-infra21:32
mesteryjeblair: Thank you sir! The poor docker job above the neutron ones has been there for 24+ hours, so I guess we're not in that much trouble yet ;)21:32
jeblairmestery: yeah, rax is unable to build f20 nodes for some reason :(21:33
mesteryjeblair: Thanks for checking, and that sucks :(21:33
*** pc_m has quit IRC21:33
openstackgerritMerged openstack-infra/puppet-cgit: Add configurable Timeout settings on git vhost  https://review.openstack.org/16087121:34
*** ihrachyshka has joined #openstack-infra21:34
fungijeblair: any idea why it's trying to build basically all of them in rax-iad? should we try dropping rax regions from that label for the time being? it looks like we have fresh images for it in hpcloud21:34
jeblairfungi: i do not know, but that seems reasonable -- did the fallback to previous images fail?21:35
fungijeblair: no, i realized that we had jobs waiting on those nodes for far longer than the most recent image was created21:36
jeblairfungi: oh ok21:36
fungiso yesterday's image can't be any better if it is an image-specific issue21:36
*** dustins has quit IRC21:36
fungii'll propose a config change for it21:36
dougwigjeblair: is that causing the lbaas-api stall?  those jobs are specified for ubuntu.21:37
*** changbl has quit IRC21:37
*** ParsectiX has quit IRC21:38
jeblairdougwig: i do not believe it was related.  it just came up in conversation.21:38
*** ParsectiX has joined #openstack-infra21:38
dougwigjeblair: got it.  i'll wait patiently.  or, as patiently as i can approximate.21:38
openstackgerritMerged openstack-infra/jenkins-job-builder: Add detailed call out about the SCP plugin  https://review.openstack.org/16091921:38
rm_workjeblair: yeah but it's weird that ALL of the neutron check jobs are still queued for lbaas-api, even when they had every other run finish long ago...21:40
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Temporarily stop adding devstack-f20 nodes in rax  https://review.openstack.org/16562021:40
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Revert "Temporarily stop adding devstack-f20 nodes in rax"  https://review.openstack.org/16562121:40
rm_workmakes me wonder if something is stuck21:40
fungijeblair: marking the second ^ wip21:40
*** Rockyg has quit IRC21:40
mesteryrm_work: Yup, that's why I pinged in here, something is really fishy21:40
*** yfried is now known as yfried|afk21:41
dougwigrm_work, mestery - my read on jeblair's reply was that he mentioned the f20 thing in relation to the docker stall, not ours, and is still looking into the lbaas-api stall.21:41
rm_workah k21:41
rm_workyeah just wanted to make sure he was aware that we thought it needed to be looked at :P21:42
*** bhunter71 has quit IRC21:42
fungimestery: dougwig: rm_work: looks like that job hasn't actually run since switching from experimental to check21:43
rm_workick21:43
mestery#sadpanda21:43
*** LinuxJedi is now known as LinuxJedi_21:43
*** devlaps has quit IRC21:43
fungiat least all the recent results i find for it on any of the jenkins masters were triggered in the experimental pipeline21:44
jeblairbuild:check-neutron-lbaas-dsvm-api:devstack-trusty32016521:44
*** asettle has joined #openstack-infra21:44
*** garyh has quit IRC21:44
dougwigjenkins is afraid.  it's such an awesome test, if it runs it, it may implode space/time.21:44
mesterylol21:45
jeblairso that's 32 jobs queued, 0 running, 165 workers21:45
jeblairfungi: when did it switch?21:46
dougwigjeblair: a few hours ago.21:46
jeblairdougwig: ah, that may be it21:46
dougwig2 hours 14 minutes, to be precise.21:46
dougwiggot queued into all existing zuul jobs about 10 minutes after that.21:46
dougwighasn't run yet, though.21:46
jeblairzuul added the job to all currently enqueued changes, but the job was added to the end of the queue21:46
rm_workah21:47
dougwigthat was my guess.21:47
jeblairdougwig: so they should start to run when all of the existing jobs at the same priority as of 2 hours 15 minutes ago have run21:47
fungijeblair: https://review.openstack.org/160915 merged 03-16 15:4221:47
fungiutc21:47
mesteryNuts, it's a traffic jam :(21:47
rm_workwow, k21:47
dougwigjeblair: ok, makes sense.21:47
jeblairdougwig: you should have told me your guess, it would have saved me time :)21:47
dougwigmestery: yes, but it'll all clear about the same time tonight in a WHOOSH.21:47
fungier, i'm clearly looking at the wrong change21:48
dougwigjeblair: sorry, i figured i was way too green on infra to be right.  :)21:48
*** tqtran is now known as tqtran_afk21:48
fungiyeah, it was https://review.openstack.org/164390 a couple hours ago21:48
fungiso coincidence i guess21:48
*** hdd has joined #openstack-infra21:49
dougwigthe humans timed out before the machines.21:49
*** xyang1 has quit IRC21:49
fungizuul's been too starved for the right workers for that to get to them since they're behind more than two hours of waiting queue21:49
*** ParsectiX has quit IRC21:49
*** spzala has quit IRC21:50
*** flaper87 has quit IRC21:50
*** persia has quit IRC21:50
fungioh, jeblair already said all that while i was git logging21:50
jeblairchanges from 2:45 ago are getting nodes now, so it shouldn't be too much longer.21:50
*** spzala has joined #openstack-infra21:51
*** mwagner_lap has quit IRC21:51
*** pcrews has quit IRC21:51
*** persia has joined #openstack-infra21:51
*** persia has quit IRC21:51
*** persia has joined #openstack-infra21:51
*** pcrews has joined #openstack-infra21:51
*** sigmavirus24 is now known as sigmavirus24_awa21:51
*** yamahata has quit IRC21:52
*** jkt has quit IRC21:52
*** asettle has quit IRC21:53
*** yamahata has joined #openstack-infra21:53
*** jkt has joined #openstack-infra21:53
*** flaper87 has joined #openstack-infra21:53
*** waynr has quit IRC21:55
*** waynr has joined #openstack-infra21:55
jeblairStevenK: ping me when you are online21:57
*** camunoz_gone has quit IRC21:57
zigo_It's looking like to me that review.openstack.org isn't available from its advertized IPv6.21:58
zigo_Could someone look into this?21:58
sc68calzigo_: looking21:58
*** sarob has joined #openstack-infra21:58
zigo_I had to edit my /etc/hosts to be able to reach it.21:58
zigo_Maybe my bad, checking again from another host.21:58
*** Rockyg has joined #openstack-infra21:59
sc68calzigo_: I can ping it from my v6 env21:59
sc68cal2001:4800:7815:104:3bc3:d7f6:ff03:bf5d21:59
*** bswartz has joined #openstack-infra21:59
openstackgerritMerged openstack/requirements: Update Magic Search version  https://review.openstack.org/16066121:59
fungizigo_: sc68cal: i can reach it at that ip address22:00
fungivia https22:00
nibalizerv6 works for me22:00
zigo_Weird, seems my ISP ipv6 is down...22:00
zigo_Sorry guys for the trouble.22:00
fungino problem. i too wish isps would get better about ipv622:00
*** asselin_ has joined #openstack-infra22:00
*** yfried|afk is now known as yfried22:01
sc68calI've had one that advertises a route from one LLA, then later it is moved to a different LLA22:01
fungithe whole "happy eyeballs" idea is a circular argument. nobody will use ipv6 while isps make no effort to keep it working, and the isps will see no incentive to keep it working when nobody uses it22:01
sc68calso I manually restart rtsol every hour22:01
openstackgerritKhai Do proposed stackforge/python-jenkins: Add utf8 to headers  https://review.openstack.org/16562822:01
sc68calkimsufi hosting has wacky ipv622:02
*** Ryan_Lane has quit IRC22:02
*** boris-42 has quit IRC22:02
dstufftfungi: I learned the other day that 14% of the traffic to google.com from the US is via ipv622:02
fungineat!22:03
dstuffthttps://www.google.com/intl/en/ipv6/statistics.html#tab=per-country-ipv6-adoption22:03
fungithat's certainly on the rise then22:03
dstufftfound the link22:03
dstufftwell, dunno if that's google.com or all google web properties22:04
*** resker has quit IRC22:04
fungiit's probably mostly the traffic for their nsa backdoor ;)22:04
*** nelsnelson has quit IRC22:05
sc68calfungi: hahaha... awwww :*(22:05
dstufftlifeless: it wouldn't be a packaging tool release if we didn't break something ;/22:05
*** asselin_ has quit IRC22:05
openstackgerritJames E. Blair proposed openstack-infra/puppet-httpd: Reduce version to 0.0.0  https://review.openstack.org/16401322:06
zarojeblair: could you please be more specific on which points have not been address in this? https://review.openstack.org/#/c/164264/22:06
*** dangers is now known as dangers_away22:07
*** asettle has joined #openstack-infra22:07
openstackgerritMerged openstack-infra/puppet-graphite: Add missing storage subfolders  https://review.openstack.org/16167622:07
*** hashar has quit IRC22:07
jeblairzaro: i think the point about possibly conflicting with gerrit::mysql and also whether we want it in that module or in openstack-project (though the latter is a discussion point i think)22:08
lifelessdstufft: oh, w.r.t. pbr discussion? ancient history now :)22:08
dstufftlifeless: ya22:08
dstufftspeaking of PEP 440, we should mark that as no longer provisoonal22:08
dstufftI think all the issues we're likely to be able to solve are solved now22:08
jheskethMorning22:09
fungizaro: jeblair: yes, rereviewing the latest patchset, those two items22:09
*** bknudson has quit IRC22:09
fungizaro: jeblair: depending on the outcome of the discussion point, the resource conflict might be trivially solved too22:09
jeblairya22:09
*** ZZelle_ has joined #openstack-infra22:09
*** camunoz_gone has joined #openstack-infra22:10
*** marun has joined #openstack-infra22:10
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Add support to configure boolean parameter in PBT  https://review.openstack.org/14666522:10
*** carl_baldwin has quit IRC22:11
*** yfried is now known as yfried|afk22:11
*** sabeen1 has quit IRC22:11
*** asettle has quit IRC22:12
*** otter768 has joined #openstack-infra22:12
fungiin light of that, i'm in favor of having it in openstack_project::gerrit in the system-config repo22:12
*** jamielennox|away is now known as jamielennox22:12
*** asettle has joined #openstack-infra22:13
*** Bsony has joined #openstack-infra22:13
fungibut also because i think it's very much something specific to our deployment and not a decision we should be making for downstream deployers22:13
*** sputnik13 has quit IRC22:13
jeblairfungi: yeah, i'm still a little on the fence because i think it's a good decision :).  i could get behind that though.22:14
*** sputnik13 has joined #openstack-infra22:14
*** LinuxJedi_ is now known as LinuxJedi22:14
*** stevemar has quit IRC22:15
fungiwell, for people who are doing remote databases it's probably not something they should implement without also changing their server end, and if we want to enforce it in configuration for anyone using local databases that part probably needs to be managed through the puppet mysql module's config class22:15
*** LinuxJed_ has joined #openstack-infra22:16
fungiso doing it "correctly" in the puppet gerrit module gets decidedly more complex than making it something specific to our deployment22:16
*** otter768 has quit IRC22:17
fungibut i wasn't -1'ing over the choice of repo for the change, just wanting to make sure it's discussed some22:17
*** dkranz has joined #openstack-infra22:17
*** Bsony has quit IRC22:18
*** dboik has quit IRC22:18
*** packet has quit IRC22:19
nibalizerand as the 'openstackci' module takes shape, the puppet-<service> modules are gonna have to get flexible22:19
*** LinuxJedi has quit IRC22:19
nibalizersince thats targeted at a single allinone node, so not everyone can manage the mysql database, they'll have to cooperate22:20
nibalizerer the mysql service, and each will have to manage a database22:20
*** shardy is now known as shardy_z22:20
*** LinuxJed_ is now known as LinuxJedi22:21
*** Rockyg has quit IRC22:21
clarkbjeblair: any opposition to me self approving 155458?22:26
clarkbthat should get a whole stack of new tests in22:26
*** amitgandhinz has quit IRC22:27
j^2if i’d like openstack to post new bugs reported via launchpad is there a way i can put that in?22:27
j^2the irc bot that is ;) ^22:28
clarkbj^2: not currently, launchpad doesn't have a good event feed. THe one hack that has been used by some is to subscribe a bot to email from bugs then have that bot talk to irc22:28
j^2clarkb: ah lame, i’ve noticed in #maas it’s does basiclly what i’d like22:29
jeblairclarkb: go for it!22:29
j^2ok, thanks :)22:29
*** dimtruck is now known as zz_dimtruck22:30
openstackgerritMerged openstack-infra/nodepool: Add nodepool command tests  https://review.openstack.org/15545822:33
openstackgerritMerged openstack-infra/nodepool: Check image-list output in cmd tests  https://review.openstack.org/16435622:33
openstackgerritMerged openstack-infra/nodepool: Indicate error when deleting an invalid snapshot  https://review.openstack.org/16447222:33
*** melwitt has quit IRC22:34
notmynameout of curiosity, is it possible to configure the CI jobs per feature branch? eg run one set on master and a different on feature/foo. or run one config with master and a different one for feature/foo (for the same test)22:34
mordrednotmyname: yes - zuul is fully branch aware22:34
notmynamemordred: cool22:34
mordrednotmyname: you can see examples related to stable/ branches22:34
*** Sukhdev has joined #openstack-infra22:35
notmynameah ok. that makes sense22:35
Sukhdevfungi: can I ask you a question22:35
notmynamemordred: and that would be configured in the zuul config yaml file? ie the same place the jobs themselves are defined?22:35
*** ajmiller_ has joined #openstack-infra22:36
mordrednotmyname: yes to zuul layout.yaml22:36
clarkbI am looking at http://logs.openstack.org/42/165542/1/check/gate-os-loganalyze-python27/7549f7c/console.html now22:36
notmynamemordred: thanks22:36
mordrednotmyname: sure nuff!22:36
*** timcline has quit IRC22:36
openstackgerritMerged openstack-infra/nodepool: Test snapshot image deletion command  https://review.openstack.org/16447322:37
fungiSukhdev: you just did!22:37
*** tqtran_afk is now known as tqtran22:37
fungiSukhdev: better to just ask your actual question than to ask me a question about whether you can ask me a question22:37
Sukhdevfungi: Ha ha funny:-) In that case can I ask a follow on quesiton :-)22:38
Sukhdevfungi: I have stackforge/networking-arista project22:38
fungibut clearly the answer is yes, you definitely seem to be able to ask me a question22:38
jeblairclarkb: what do you think about all those InsecurePlatformWarnings?22:38
SukhdevI want to create a branch (in preparation for kilo), I am having hard time doing it - how do I do it?22:39
*** ajmiller has quit IRC22:39
fungiSukhdev: just a sec and i'll look at the acl for it to see if it's set to allow you to do that22:39
clarkbjeblair: its the same thing that hit gertty22:39
openstackgerritKhai Do proposed stackforge/python-jenkins: Fix examples  https://review.openstack.org/16563522:39
clarkbjeblair: they are mostly just annoying but go away in the swift uploaded file because they happen after we get the console so haven't been a priority to fix22:39
*** dkranz has quit IRC22:39
*** dboik has joined #openstack-infra22:41
clarkbjeblair: we could add the ignore code for that to the script though22:42
*** dkliban is now known as dkliban_afk22:42
*** dboik_ has joined #openstack-infra22:43
*** mrmartin has joined #openstack-infra22:44
fungiSukhdev: yeah, insufficient permissions. you need to propose a change to http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/stackforge/networking-arista.config adding [access "refs/heads/*"] create = group networking-arista-release22:45
*** bhunter71 has joined #openstack-infra22:45
*** camunoz_gone has quit IRC22:45
fungiSukhdev: and once that merges, members of the networking-arista-release group will be able to create branches on that project22:45
*** dboik has quit IRC22:46
*** gyee has joined #openstack-infra22:48
*** andreykurilin_ has joined #openstack-infra22:48
*** ddieterly has quit IRC22:50
*** koolhead17 has quit IRC22:51
*** aysyd has quit IRC22:55
*** Rockyg has joined #openstack-infra22:55
ianwfungi: you ok with git ref in https://review.openstack.org/#/c/164901/  (Add nodepool config-validation environment)?22:56
*** Ryan_Lane has joined #openstack-infra22:56
openstackgerritClark Boylan proposed openstack-infra/os-loganalyze: fix supports_sev matching  https://review.openstack.org/16554222:57
clarkbjeblair: fungi that should actually pass tests now and solve our issue22:57
*** camunoz_gone has joined #openstack-infra22:57
ianwalso, jebliar / fungi : https://review.openstack.org/#/c/164036/ (Generate nodepool logging configuration) <- this is what you want for split nodepool logs?22:57
fungiianw: we should probably use zuul-cloner so that we're not hitting the git server for that?22:57
openstackgerritMerged openstack-infra/devstack-gate: add ceilometermiddleware  https://review.openstack.org/15706022:58
jeblairclarkb: can you double check me on my assumption in 163650?22:58
*** gordc has quit IRC22:58
clarkbjeblair: looking at it now22:59
fungiianw: i'm hunting for a good example now23:00
maulermarkfungi: asselin: I got my WSGI log server working yesterday with your help.  It turned out the scripts didn't work if you used the default document root /var/www/html.  Thanks for the help.23:00
*** Ryan_Lane has quit IRC23:01
*** Ryan_Lane has joined #openstack-infra23:02
clarkbjeblair: ya looking at cgconfig docs from redhat it also seems to be fine23:02
*** bhunter71 has quit IRC23:03
*** ihrachyshka has quit IRC23:03
fungiianw: actually, i guess our other scripts are actually just cloning directly from our git servers, so while specifying a git url in a requirements file is something we try very hard to avoid in python projects it may be fine in system-config23:04
*** _nadya_ has joined #openstack-infra23:04
ianwfungi: that was what i thought looking at tools/run-layout.sh23:05
jeblairfungi: yeah, it could be improved.  in the mean time, our volume is much lower.  and we tend not to yell at us if our servers break.  :)23:05
fungiianw: the only place we seem to really be doing it is in the apply test23:05
fungijeblair: i totally yell at me when our servers break23:05
openstackgerritMerged openstack-infra/puppet-httpd: Reduce version to 0.0.0  https://review.openstack.org/16401323:06
*** garyh has joined #openstack-infra23:06
*** YorikSar has joined #openstack-infra23:07
anteayamtreinish: I took a queen's english torch, I was hoping that fungi was expressing concern that I return home safely should I happen to be out after dark23:07
greghaynesI took a swing at the puppet and zuul config for resurrecting pypi-mirror for wheel mirrors and am wondering if theres any glaring holes in https://review.openstack.org/#/c/165240/ and https://review.openstack.org/#/c/164927/ - Also not sure if theres some sane way to be testing that as I go or what23:07
nanliunibalizer, I'm not seeing the error regarding the inline_template23:07
*** subscope_ has quit IRC23:07
anteayadhellmann: sorry I was away all day, do you still need help with the gate-python-openstackclient-requirements job?23:08
fungianteaya: i was actually just wanting to make sure you blended in with the angry mob. though i guess you'd have needed a pitchfork too23:08
nanliuI can hardcode the checksum, but would force users to specify checksum instead of being able to download and verify23:08
nanliuwhich might be a step back.23:08
anteayafungi: why would I need to blend in with an angry mob?23:08
fungidunno. canada has those right? or am i thinking of guatemala?23:08
mtreinishanteaya: heh, ok. Just less cool than a flaming stick :)23:08
anteayaHaliburton in march isn't much for mobs23:08
anteayaJuly is the mob season23:09
mordredanteaya: if you don't blen in with the angry mob - then the angry mob chases you23:09
anteayafungi: when we win hockey yes23:09
YorikSarHello, #openstack-infra! I see an issue with Gerrit permissions. From the looks of it I can Workflow+1 this change: https://review.openstack.org/164755 (the option is in Reply form), but I shouldn't have such permission.23:09
*** _nadya_ has quit IRC23:09
mordredanteaya: didn't canada lose all the hockeys lasst year?23:09
anteayamtreinish: way less cool, but cleaner to put in my backpack23:09
clarkbjeblair: thinking about those warnings, I think reasonable approach would be to do warnings.simplefilter('ignore', InsecurePlatformWarning)23:09
clarkbjeblair: unless you think its good to have so we can see the warnings go away if/when python is fixed23:09
anteayamordred: yes that is the pattern I am used to, crowds don't do much for me23:10
anteayamordred: I don't know exactly, we were doing well in the juniors around christmas but I think I lost interest before I found out the final outcome23:10
fungijeblair: i'm planning to rename the private foundation-board ml to foundation-board-confidential tomorrow and create a new public foundation-board ml to replace it (after jbryce warns them all it's finally happening). any words of warning as someone who has spent a long time wrangling mailman?23:10
fungihttp://wiki.list.org/DOC/4.70%20How%20do%20I%20change%20the%20name%20of%20(rename)%20a%20list%3F looks fairly straightforward23:11
mordredanteaya: I know very little - although I've been thinking that the people here really like it, so i should go learn things23:11
mordredfungi: run away very very quietly23:11
nibalizernanliu: http://logs.openstack.org/08/164908/3/check/gate-infra-puppet-apply-precise/d30c9a5/console.html#_2015-03-17_02_50_55_64523:11
nibalizerour apply test logs are not easy to read23:11
fungimordred: i tend to just back away slowly23:11
nibalizerit is truly sad23:11
nanliunibalizer, found it23:12
nibalizerthere is so much improvement to do there23:12
clarkbnibalizer: we can blame puppet3 for that :)23:12
mordrednibalizer: I blame ... uhm ...23:12
mordredyes! puppet223:12
mordred323:12
clarkbnibalizer: I swear everything is a warning in puppet323:12
mordredI blame puppet3!23:12
anteayamordred: take me to a game and I'll teach you23:12
nanliuok, might be issue with server verification of ssl cert.23:12
*** yamamoto has joined #openstack-infra23:12
anteayamordred: or you can just go to a game and try to pick it uop23:12
anteayaup23:12
nibalizeralso just for the record, hax23:12
mordredanteaya: ok. will you teach me how to properly join a hockey brawl? or will you only teach me how the game works?23:12
nanliucan't diag more unless I know more about test box :(23:12
nibalizerthat line 33 is mordred worthy23:12
clarkbYorikSar: not just workflow -1 for wip?23:13
*** hichihara has joined #openstack-infra23:13
nibalizernanliu: note that the apply test is running in noop so the file wont actually be pulled down23:13
mordrednibalizer: line 33?23:13
YorikSarclarkb: I see all 3 options: -1, 0, +1.23:13
nibalizermordred: https://review.openstack.org/#/c/164908/ line 33 of the proposed23:13
nanliuwell the inline_template is server side processing23:13
nibalizerof init.pp23:13
nanliuso it is doing the download of the checksum file on compile23:13
anteayamordred: to properly join a hockey brawl? climb over the boards and start hitting23:14
anteayadone23:14
nanliuLet me double check again.23:14
clarkbYorikSar: the acl hasn't been changed since february23:14
nibalizerwait so you're going out over the network inside inline_template?23:14
*** tnovacik has quit IRC23:14
mordredanteaya: neat! I wasn't sure if there was protocol to follow23:14
clarkbYorikSar: so maybe a group change happened?23:14
nibalizeri thought you were opening the file23:14
anteayaif you want to know how the game works I can teach you that23:14
*** sputnik13 has quit IRC23:14
JayFanteaya: mordred: stick, gloves, then shirt23:14
anteayamordred: that is the protocol23:14
pleia2anteaya: haha23:14
anteayaJayF: you've studied23:15
JayFI'm a HUGE hockey fan. And I liked the Mighty Ducks movie even before that :P23:15
YorikSarclarkb: Nothing should've changed about me... And I tried to find a group that might have W+1 right with me in it and there isn't one.23:15
clarkbYorikSar: I cannot +1,0,-1 workflow so I don't think this is a general problem23:16
anteayaJayF: nice23:16
pleia2I've been to some Sharks games, but hockey here is very different than it was in Maine (which is kind of honorary Canada)23:16
openstackgerritMerged openstack-infra/puppet-jenkins: Use @varname to supress warnings  https://review.openstack.org/16365023:16
*** ajmiller_ is now known as ajmiller23:16
anteayaJayF: who is your team?23:16
anteayapleia2: yes23:16
nanliunibalizer, yes, see: https://gist.github.com/nanliu/80b1abbb3fe15f79ab1523:16
JayFanteaya: I am a native of NC, and became a fan watching Carolina Hurricanes. Since being in SF for 3y, I've been watching more of the San Jose Sharks + have even got season tix this year23:16
anteayawho is going to be in montreal for pycon?23:17
JayFanteaya: but we're about to miss the playoffs so that's no fun23:17
pleia2JayF: aw :\23:17
*** yamamoto has quit IRC23:17
nibalizermaybe the require failed?23:17
nibalizeron the test box?23:17
anteayaJayF: nice on the tix :( on missing the playoffs23:17
nanliuopen-uri is stdlib in ruby23:17
nanliuso the certificate failure is likely: https://stackoverflow.com/questions/23190868/opensslsslsslerror-hostname-does-not-match-the-server-certificate23:18
nanliumaybe the test server ruby bundled cert is out of date23:18
mordrednanliu: does open-uri obey proxy settings?23:18
*** yamamoto has joined #openstack-infra23:19
mordredJayF: I've been considering adopting the rangers, seeing as how I'm in ny and all23:19
*** LinuxJedi has quit IRC23:19
YorikSarclarkb: Is there a safe way to test if I actually have that right or is it just an UI issue?23:19
jeblairJayF: oh i'm from nc too.  and i nearly live in sf (berkeley)23:19
clarkbYorikSar: thats a good question. I don't think so23:19
clarkbYorikSar: if you have the rights you can leave the vote and it should stick23:19
nanliumordred, not sure23:19
mordredJayF: I'm also from NC - but, you know NYC23:19
clarkbYorikSar: which isn't super safe23:19
jogojeblair: when skip-if was added you wanted to hold off before a lot of projects adopt it23:20
jogojeblair: is that still the case? (to work out the kinks)23:20
mordrednanliu: we have consumers of these modules who are behind terrible firewalls - (hi nibalizer) so we should doublecheck that23:20
*** juggler has joined #openstack-infra23:20
JayFjeblair: nice23:20
nibalizeri think its okay to just not be that fancy23:20
JayFmordred: fuggedaboutit23:20
mordredJayF: :)23:20
anteayajogo: last I saw there were three projects using it23:20
nibalizerif whomever wants to bump the version of es they can get the checksum manually and put it in the change that bumps the version23:20
jeblairjogo: i think if marun is happy we can go ahead.  i think a few more have added it and i don't think there have been huge issues.23:20
StevenKjeblair: Hai23:21
nibalizerpass it in as a prameter or look it up in hiera or whatever23:21
jogoanteaya: I count 3 as well23:21
*** sputnik13 has joined #openstack-infra23:21
clarkbYorikSar: I am going to poke at the database to see if that reveals any info23:21
jogojeblair: makes sense, thanks23:21
nibalizermaybe thats madness23:21
jeblairStevenK: hi there!  pleia2, mordred, and i wanted to chat about zanata client packages23:21
*** claudiub has quit IRC23:21
clarkbYorikSar: can you give me your account id number? its on your main settings page in gerrit23:21
jeblairpleia2: around?23:21
YorikSarclarkb: I see that option in every CR where I'm an owner.23:21
nibalizersince i think of the checksum as being more than just about integerity of the data after download and also ensuring it is the file we actually wanted23:21
YorikSarclarkb: 70823:21
clarkbYorikSar: huh23:22
*** Swami has quit IRC23:22
clarkbcan anyone else check if they get the behavior that YorikSar sees?23:22
StevenKjeblair: Chat here, or in a hangout, or somewhere else?23:22
*** juggler_ has quit IRC23:22
clarkbI don't get that behavior on an older oslo.rootwrap change that I have up23:22
pleia2jeblair: hey23:23
jeblairclarkb: i have no outgoing changes to a repo where i am not core :(23:23
pleia2StevenK: here is good23:23
greghaynesclarkb: So thought I have for nodepool tests - rather than our if 'fake-' return FakeClient else get_client we could monkeypatch our fake client over the real client in the tests?23:23
jeblairStevenK: here is fine23:23
*** juggler_ has joined #openstack-infra23:23
clarkbgreghaynes: ya thats what we should be doing23:23
greghayneskk, patches incoming23:23
nanliunibalizer, ok I'll change it to best effort obtaining checksum23:23
clarkbgreghaynes: and just do it across the board in the base test class23:23
greghaynesah, nice23:24
*** sputnik13 has quit IRC23:24
mordredStevenK: here is ok23:24
nanliuif you specify, we will check, if not will try to pull checksum, and if all fails no checksum.23:24
openstackgerritKhai Do proposed openstack-infra/system-config: Replace the Gerrit WIP icon  https://review.openstack.org/16522823:24
* mordred thought he'd add his thoughts on here to keep up with jeblair and pleia223:24
jeblairStevenK: i hear you're making packages -- are you intending them just for infra, or for ubuntu?23:24
*** mjturek3 has quit IRC23:25
*** andreykurilin_ has quit IRC23:25
StevenKjeblair: I am trying to. :-)23:25
*** esheffield__ has quit IRC23:25
*** esheffield_ has joined #openstack-infra23:25
*** LinuxJedi has joined #openstack-infra23:25
YorikSarclarkb: If I check this on some other change where there's not enough CR+2's and it actually allows me to put W+1, will it trigger Zuul merge machinery?23:25
clarkbYorikSar: you use change screen 2?23:25
*** mrodden has quit IRC23:25
clarkbYorikSar: if there is one +2 then it will trigger the machinery23:26
StevenKjeblair: My plan was to propose them for Debian/Ubuntu if I feel they are high enough quality23:26
clarkbYorikSar: I am wondering if this is a change screen 2 bug23:26
clarkbzaro: ^23:26
*** sputnik13 has joined #openstack-infra23:26
YorikSarclarkb: Yeah, New change screen. Let me verify the old one...23:26
*** juggler has quit IRC23:26
*** mrodden has joined #openstack-infra23:27
openstackgerritJoe Gordon proposed openstack-infra/project-config: Don't run neutron-large-ops on neutron advanced services  https://review.openstack.org/16564823:27
YorikSarclarkb: Ok, there's no W+1 on old change screen...23:27
jeblairStevenK: okay cool.  i think when we wrote down "package the client" at the summit and put it in the spec, we might have extra context that we didn't write down23:27
StevenKjeblair: But that requires me to stop drowning in dependency hell23:27
clarkbYorikSar: I think it must be a bug in the new change screen23:27
clarkbYorikSar: I do not think it will actually let you post a W+123:28
jeblairStevenK: if you're in dependency hell, you might like where i'm going with this :)23:28
YorikSarclarkb: Yeah, it seems so. I'll try to do it on a CR w/o any votes.23:28
nanliumordred, yes follows proxy in test env.23:28
clarkbYorikSar: let me know how that goes23:28
jeblairStevenK: so the thing about "packages for infra" is that we don't really have a package building infrastructure for ourselves yet.  i know we could use a ppa, but that doesn't really work with our workflow23:28
YorikSarclarkb: "Applying label "Workflow": 1 is restricted"23:29
YorikSarclarkb: Yay, system works! :)23:29
clarkbYorikSar: perfect, though that is an annoying UI issue23:29
jeblairStevenK: we would _love_ to have such a thing (and i think we may have talked about this briefly when we met in person)23:29
StevenKjeblair: Right. And I'm happy to be involved with that effort as well. :-)23:29
zarofungi, jeblair : i don't think the gerrit::mysql is even used.  or at least i couldn't find it being called anywhere.23:29
YorikSarclarkb: Sorry for the fuzz. I remember new change screen is not officially supported in our Gerrit.23:29
jeblairStevenK: but that's a sizable effort, and we don't necessarily need zanata to block on it23:29
*** dims__ has joined #openstack-infra23:29
clarkbzaro: its not used by us because we trove, however I would not be surprised if we stop troving23:30
clarkbwe seem to have trouble building nodes again23:30
jeblairStevenK: yeah, so i think if we/you want to build that packaging infrastructure i think that's great, and let's look into doing it in its own right23:30
*** boris-42 has joined #openstack-infra23:30
jeblairStevenK: but we probably shouldn't make it a requirement for zanata23:31
mordredjeblair: ++23:31
pleia2agreed23:31
*** camunoz_gone has quit IRC23:31
clarkbwe seem to have a good number of leaked nodes in hpcloud23:31
fungizaro: it's not used by _us_ but for someone who does use it, it will probably break if we're declaring files also declared by the puppet mysqldb resource used by the gerrit::mysql class23:31
openstackgerritMerged openstack-infra/jenkins-job-builder: updated conditional_publisher to support plugin changes  https://review.openstack.org/15551423:31
jeblairStevenK, pleia2: i think for zanata, we can probably just have puppet install the client war file or whatever, right?23:31
*** sputnik13 has quit IRC23:31
StevenKjeblair: It's not a war file from what I can see23:31
pleia2jeblair: yeah, I can add it to our todo list for puppet23:31
jeblairStevenK, pleia2: we have some other software with that installation pattern, more or less, so it's somewhat of a known quantity23:31
clarkbmordred: jeblair maye we can inquire about any recent updates? because this is a really annoying situation. I will work on cleaning out the aliens now23:31
* pleia2 nods23:32
mordredclarkb: I got a note taht the public cloud taem are looking in to port leaking23:32
jeblairStevenK: oh, what form does it take?23:32
clarkbmordred: this isn't port leakage, its node leakage23:32
openstackgerritIan Wienand proposed openstack-infra/system-config: Generate nodepool logging configuration  https://review.openstack.org/16403623:32
mordredclarkb: taht I don't know anything about23:32
clarkbmordred: nodepool has been pretty good about leaking nodes in the past23:32
ianwjeblair: ^ template now, same as the existing23:33
*** beecee has joined #openstack-infra23:33
*** dims_ has quit IRC23:33
clarkbmordred: if I had to guess they return ERROR state, nodepool says delete and they are "deleted" then sometime later the cloud says "lol just kidding here have that node afterall"23:34
marunjeblair, jogo: We haven't seen problems past the initial nova/neutron match conflict, it's been stable.23:34
mordredclarkb: wait - and you're saying you don't think that's friendly?23:34
StevenKjeblair: The dist.tar.gz on maven-central is a whole lot of .jar's and three files in bin/23:34
clarkbmordred: not if I am billed for it :)23:34
mordredStevenK: is it's mvn installable then?23:35
jogomarun: cool, I was thinking of adding skip-if to other projects now as well23:35
pleia2jeblair: but the install directions are: sudo ln -s bin/zanata-cli /usr/local/bin/zanata-cli23:35
pleia2jeblair: so it's just a single bin file that we really need23:35
*** ashleighfarnham has quit IRC23:35
pleia2afaik23:35
StevenKmordred: Probably, but I'm allergic to mvn install downloading the entire internet23:36
*** wenlock has quit IRC23:36
pleia2so for us in puppet that probably means: grab the .tar.gz from maven.org, unpack, symlink23:36
StevenKI am happy to package up the .dist.tar.gz as something to put on static.o.o23:36
jeblairwhich sounds terrible, but we've actually gotten pretty good at that sort of thing23:36
StevenKBut it offends my Debian sensibilities :-)23:37
zarofungi, clarkb :got it.23:37
marunjeblair: cool23:37
pleia2re: http://zanata.org/help/cli/cli-install/23:37
*** hdd has quit IRC23:37
pleia2or try the mvn install thing, but I defer to someone who knows something about that23:37
*** sputnik13 has joined #openstack-infra23:37
*** sputnik13 has quit IRC23:38
zaroYorikSar, clarkb : lots of stuff don't work right with new screen in Gerrit 2.823:38
jeblairStevenK: we can probably fetch the file directly from maven.org; though perhaps we should try 'mvn install' and see what it does?23:38
*** schang has quit IRC23:38
mordredStevenK, jeblair, pleia2: apt-get install ivy ; wget https://raw.githubusercontent.com/zanata/zanata-client-ivy/master/zanata-cli /usr/local/bin/zanata-cli23:39
anteayadhellmann: so far I'm coming up empty, I'm comparing what you have with python-saharaclient who has running requirements jobs and I can't see the difference yet between your code and theirs23:39
mordredor - obviously, clone the git repo and copy the file if you're doing it for real23:39
ianwnibalizer: does chaining parameters work with puppet classes, or is it undefined behaviour.  e.g. $log_dir='foo', $bar_dir='$log_dir/bar'23:39
jeblairmordred: what's ivy?23:39
mordredjeblair: it's an "agile dependency manager"23:39
SpamapSwe don't need ivy23:39
SpamapSthe tar with the binaries is on maven23:40
SpamapSand a .asc to verify its integrity23:40
SpamapSSomebody needs to verify the signer's identity23:40
mordredsure. just saying - if you don't want to wget a tarball - you can ivy23:40
clarkbStevenK: I would describe debian as anything but sensible :P23:40
pleia2maybe not sensible all the time, but it is maintainable23:40
pleia2download tarballs is a bit :(23:40
pleia2but it does work for us for now23:40
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper  https://review.openstack.org/15668123:40
mordrednevermind- don't do ivy - it seems to do dependency things EVERY TIME YOU RUN IT23:40
StevenKclarkb: Pppffft23:41
SpamapShttp://paste.ubuntu.com/10624101/23:41
SpamapSivy is for devving23:41
pleia2mordred: hah23:41
nibalizerzaro: so im trying to understand 16514523:41
*** schang has joined #openstack-infra23:41
mordredSpamapS: nod. I have learned something23:41
nibalizerwhat does touching the GerritSiteHeader.html do23:41
SpamapSSo we just need to find somebody who has met Ding-Yi Chen and can verify that signature. ;)23:41
nibalizerdoes gerrit watch taht file and reload?23:41
mordredSpamapS: of course, I don't know Ding-Yi Chen23:42
SpamapShe has no sigs in the global key db23:42
openstackgerritJoe Gordon proposed openstack-infra/project-config: Split grenade out of integrated-gate template  https://review.openstack.org/16565123:42
openstackgerritJoe Gordon proposed openstack-infra/project-config: Remove grenade jobs from projects that grenade doesn't test  https://review.openstack.org/16565223:42
StevenKHah23:42
nibalizeras written, we'll compress jquery every puppet run, and propogate through whatever changes touching the gerrit file do23:42
mordredsigh23:42
mordredseriously maven central isn't https?23:42
StevenK"web" of trust23:42
jeblairStevenK: he's in Brisbane, Australia according to github.  you're up!  :)23:42
pleia2hehe23:42
StevenKHah23:42
mordredoh - https://search.maven.org/remotecontent?filepath=org/zanata/zanata-cli/3.4.2/zanata-cli-3.4.2-dist.tar.gz works23:43
StevenKIt takes like 12 hours to drive there23:43
zaronibalizer: compression of jquery should only be done once.23:43
anteayaif I want to check why an expected jenkins job isn't running, are there logs someone can check if I trigger the patch with recheck?23:43
StevenKAnd then, even worse, I'll be in Brisbane.23:43
nibalizerzaro: thats not how I think that code behaves23:43
zaronibalizer: it's just a javascript library23:43
*** e0ne has joined #openstack-infra23:43
nibalizerthat exec resource will fire on every puppet run is my point23:43
pleia2StevenK: I sympathize, people think they can pop by and visit me when they're in Los Angeles23:43
StevenKHahaha23:43
*** BobBall_AWOL has quit IRC23:44
StevenKBut leaving LA is hard, I've seen Escape from LA23:44
* StevenK hides from SpamapS 23:44
pleia2hah23:44
anteayaEscape from New York is better23:44
zaronibalizer: is that due to notify => Exec['reload_gerrit_header'],23:44
fungianteaya: gotta agree with you on that one. one of carpenter's best works23:44
jeblairStevenK, pleia2, mordred: so probably fetch from maven central, verify signature if SpamapS reviews this change, then unpack and put in place?23:44
pleia2jeblair: +123:45
StevenKanteaya: An Australian comedy show lampooned a short for 'Escape from Aunty Flo's Backyard BBQ'23:45
anteayaclassic23:45
mordred++23:45
pleia2and back burner the run-our-own-debs-system for a spec at a later date23:45
anteayaStevenK: oooh, was it any good?23:45
SpamapSpleia2: what, LA is only 5 hours away. ;)23:45
mordredpleia2: yah23:45
clarkbwe have 164 leaked nodes by my count23:45
pleia2SpamapS: yeah, 6 if you drive like me23:45
anteayaclarkb: wonderful23:45
*** BobBall_AWOL has joined #openstack-infra23:46
clarkbin hpcloud alone23:46
anteayawoooo23:46
SpamapSpleia2: or 18 if you take the 1. ;)23:46
jeblairStevenK: are you interested in puppeting that?  and maybe starting on the build-debs-in-infra spec in your spare time?23:46
clarkbI will start deletes for that, anyone want the list for talking to hpcloud?23:46
pleia2SpamapS: but it's the prettiest 18 hours :)23:46
StevenKjeblair: I can package up the -dist.tar.gz into a .deb in about 15 minutes if that makes things a little easier -- but I'd prefer to do it from source if we're doing that route23:46
mordredStevenK: the problem with debs right now is that we have no place to put them23:46
anteayacan I convince anyone to look at some jenkins logs for https://review.openstack.org/#/c/158779/ if I fire a recheck, I don't understand why the check-requirements job isn't working23:46
anteayathe code looks correct to me23:47
jogoclarkb: out of curiosity what is a leaked node? a node you see in an error state, quota out of whack etc.23:47
*** camunoz_gone has joined #openstack-infra23:47
StevenKmordred: My terrible terrible plan with that was a tiny directory on static.o.o that we've run dpkg-scanpackages over, but that may offend your sensibilities :-)23:47
SpamapSpleia2: evar23:47
clarkbjogo: its a node in the tenant that nodepool uses and nodepool has no record of it23:48
jeblairclarkb: i can try to talk to hpcloud tomorrow23:48
mordredStevenK: what I'm saying is that it gets us nothing over puppet with wget  and an ln -s except complexity23:48
*** maulermark has quit IRC23:48
clarkbjogo: this means that whatever happened to the node nodepool believes it was successfully deleted23:48
openstackgerritSpencer Krum proposed openstack-infra/system-config: Manage hiera.yaml on all hosts  https://review.openstack.org/16487023:48
mordredStevenK: I think we all want a system with which we can properly build and host and update debs23:48
nibalizerzaro: yes23:48
nibalizerzaro: well kindof23:48
*** mfink_ has quit IRC23:49
mordredStevenK: but just a file in a directory would still leave questions as to how the deb got built, how we rebuild it, how we update it, etc23:49
nibalizerwhat it needs is an unless => '', clause or a subscribe => 'somefile' + refreshonly => true,23:49
jogoclarkb: ahh so its probably not an underlying nova issue that we should fix?23:49
mordredStevenK: puppet doing wget + tar xfz + ln -s is a thing that we can update by changing the url23:50
clarkbjogo: I think it probably is23:50
mordredand it can be in place and working while we figure out the more complex task of "how do we build, maintain and update debian and rpm packages in infra"23:50
clarkbjeblair: my list can be found in nodepool.o.o:~clarkb/hpcloudaliens23:50
mordredwhich ultimately is the actual problem a debian repository system aroudn here needs to solve23:50
anteayadhellmann: sorry I'm at a loss as to why check-requirements isn't running on python-openstackclient, pick up with me tomorrow and maybe we can make some progress23:51
nibalizermordred: also the archive module is a formalization of the wget/tar pattern23:51
*** pelix has joined #openstack-infra23:51
mordrednibalizer: yes. that too23:51
jogoclarkb: do you get leaked nodes in rax as well?23:51
zaronibalizer: please suggest on review.23:51
nibalizerzaro: kk23:51
clarkbjogo: I am about to check but I don't think so23:52
jogoclarkb: if none in rax, it may be a odd HP bug that has been fixed already or something23:52
greghaynesSpamapS: Hey - is your using fakes from nova experiment working out?23:52
greghaynesSpamapS: im running into a situation where I may want to look at doing something similar23:53
clarkbjogo: just after a quick glance most of the reported aliens in rax-* are semi expected. Bug reports subnodes as aliens and we have a couple nodes booted by hand in those regions23:53
nibalizerzaro: reviewed23:54
jogoclarkb: cool23:55
mordredjeblair, pleia2, StevenK: make sense? ^^23:55
*** ajmiller has quit IRC23:55
pleia2mordred: yeah23:55
StevenKmordred: Yes. I didn't think we ran puppet on random machines created by nodepool, though23:56
mordredStevenK: we run it on all of the machines23:56
SpamapSgreghaynes: well so far it is in that shade's tests that I wrote to use them continue to work in the check queue.23:56
pleia2StevenK: sorry I sent you down the scary java packaging hole23:56
mordredStevenK: we create the base image in the nodepool scripts using puppet23:56
anteayapleia2: this is your zanata packaging conversation?23:57
greghayneshah, good enough for me23:57
SpamapSgreghaynes: I'm trying to take it one step further by making nova actually consume them to build replies for a fake api server23:57
StevenKI even sent a mail to debian-java@ last night23:57
pleia2anteaya: yes23:57
SpamapSgreghaynes: rather than fake virt and novaclient fakes being two things.. make it one thing. :)23:57
greghaynesoh, wow, ok so youre going all the way down the fake rabbit hole23:57
anteayapleia2: let me know the tl;dr when you figure it out23:57
jeblairStevenK: and actually, i think most of the jobs will be on proposal.slave.openstack.org, which is (for the moment) a long running machine23:57
SpamapSgreghaynes: yes I want a single process nova that replies in consistent and expected ways23:57
mordredStevenK: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/puppet23:57
StevenKjeblair: I thought the client would be run from the slaves themselves23:57
jeblairStevenK: it is a slave, it's just a long running slave that we use for translation jobs because it needs gerrit credentials in order to propose a change23:58
*** yamamoto has quit IRC23:59
« Tuesday, 2015-03-17 Index Thursday, 2015-03-19 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!