Monday, 2014-12-29

*** openstack has joined #openstack-infra16:03
fungidstufft: cool--thanks16:03
*** dimtruck is now known as zz_dimtruck16:03
*** rfolco has quit IRC16:04
*** zz_dimtruck is now known as dimtruck16:04
fungi2014-12-25 05:40:10     <--     openstack (~openstack@openstack/openstack) has quit (Remote host closed the connection)16:06
fungiand nobody noticed until krotscheck went to start the storyboard meeting16:06
fungi"you know it's the holiday season when..."16:07
mordredfungi: hah. nice16:07
fungii guess it was taking off for christmas, boxing day and the weekend16:07
*** otter768 has joined #openstack-infra16:07
*** skolekonov has quit IRC16:07
*** garyh has joined #openstack-infra16:07
*** david-ly_ is now known as david-lyle16:08
*** ddieterly has quit IRC16:09
anteayahope it is refreshed now then16:09
*** otter768 has quit IRC16:12
*** e0ne has quit IRC16:14
fungino idea. checking the its daemon log now to see if i can figure out why it wanted a few days off16:15
fungiwe're also missing the openstackstatus bot16:15
fungii've restarted it just now too16:16
anteayaI was disconnected twice from freenode between the 26th and the 29th16:16
*** ivar-lazzaro has joined #openstack-infra16:16
*** openstackstatus has joined #openstack-infra16:16
*** ChanServ sets mode: +v openstackstatus16:16
*** rfolco has joined #openstack-infra16:17
fungime too. interestingly openstackgerrit survived. i think it's been a month or more since anyone needed to manually restart it16:17
boris-42fungi: hi16:17
*** e0ne has joined #openstack-infra16:18
anteayaopenstackgerrit what do you do?16:18
*** ihrachyshka has joined #openstack-infra16:18
*** dimtruck is now known as zz_dimtruck16:18
fungiboris-42: what's up?16:19
*** amitgandhinz has quit IRC16:19
boris-42fungi: how are you? how were holidays?)16:19
*** ddieterly has joined #openstack-infra16:20
*** rfolco has quit IRC16:22
boris-42fungi: Could you help me with getting this patch in infra https://review.openstack.org/#/c/143879/, it adds new job template for rally-mistral jobs and adds it to experimental queue of rally16:22
boris-42fungi: without it we can't test this patch https://review.openstack.org/#/c/144320/16:23
boris-42fungi: thanks16:23
*** Ala has quit IRC16:23
*** stevemar has joined #openstack-infra16:23
*** e0ne has quit IRC16:24
*** henditan has joined #openstack-infra16:24
*** jerryz has quit IRC16:25
*** e0ne has joined #openstack-infra16:29
*** achuprin_ has quit IRC16:32
*** e0ne has quit IRC16:32
*** wenchma has quit IRC16:32
*** rfolco has joined #openstack-infra16:34
fungiboris-42: great holiday fun over here, thanks! hope yours was similar. i'll take a quick look16:34
boris-42fungi: actually in Russia we have everything different=)16:35
boris-42fungi: so holidays will just start on 1 Jan and will continue till 12 =)16:35
fungiboris-42: in russia holidays celebrate you?16:35
fungiboris-42: oh fun! then i hope your preparations are proceeding according to plan ;)16:36
*** tobiash has quit IRC16:36
*** tobiash has joined #openstack-infra16:36
boris-42fungi: yep yep I already bought a lot of alcohol!=)16:36
*** e0ne has joined #openstack-infra16:36
fungithat is a necessary component of any holiday16:36
boris-42fungi: totally agree on that=)16:37
*** rfolco has quit IRC16:38
*** rfolco has joined #openstack-infra16:39
*** Masahiro has joined #openstack-infra16:41
bknudsonlooking at a jenkins failure in icehouse -- happens during pip installing and says "ImportError: No module named command.install" -- http://logs.openstack.org/16/130916/9/check/gate-tempest-dsvm-neutron-src-keystonemiddleware-icehouse/2dfb21b/console.html#_2014-12-29_16_17_21_06816:42
*** rfolco has quit IRC16:43
gordcbknudson: was just looking at that myself. seems to happen when installing glanclient16:43
*** garyk has quit IRC16:43
*** wuhg has quit IRC16:44
*** _shaps__ has quit IRC16:44
bknudsonlooks pretty consistent in that all keystonemiddleware jobs are failing the same way16:44
fungibknudson: gordc: that's a weird race we can get into when pip uninstalls setuptools to upgrade it and then in between tries to install something else which uses it16:44
funginote that it mentions uninstalling setuptools right before it vomits a traceback calling into it16:45
*** achuprin_ has joined #openstack-infra16:45
bknudsonfungi: that makes sense from the log output.16:45
boris-42fungi: thank you for review16:45
gordcfungi: darn races... yeah i noticed it tries to uninstall 8.2.1 but installs 9.0.1 a bit earlier.16:45
*** Masahiro has quit IRC16:46
fungibknudson: gordc: mordred has a stable backport proposed to devstack which should fix that16:46
bknudsonmust be https://review.openstack.org/#/c/144161/16:47
fungihttps://review.openstack.org/#/q/I9996bbc76ea81dd5bdded3bc8a5bf0542ade95f1,n,z16:47
fungiyeah, looks like it still needs some help16:47
zaromorning16:47
bknudsonhttp://logs.openstack.org/62/144162/2/check/check-grenade-dsvm-ironic-sideways/75eb9bf/logs/grenade.sh.txt.gz#_2014-12-26_18_05_11_308 -- "mkdir: cannot create directory '/opt/stack/new/.cache': Permission denied"16:49
jrollbknudson: fix is +A'd for that and in the gate queue16:50
*** e0ne has quit IRC16:50
fungibknudson: yep, that's fixed by one of the sudo -H patches16:50
bknudsonsweet16:51
openstackgerritMerged openstack-infra/project-config: Add Rally performace job template for Mistral  https://review.openstack.org/14387916:51
bknudsonany time you get permission denied, try sudo.16:51
*** amitgandhinz has joined #openstack-infra16:51
sigmavirus24permission denied? challenge accepted. sudo !!16:52
fungibknudson: actually, sudo is what has created the permission denied in this case16:52
fungibknudson: pip 6.0 now stores stuff in a ~/.cache directory16:52
fungibknudson: so if you sudo pip something you end up with a root-owned .cache in your homedir rather than in ~root16:53
* anteaya has been called to go on an adventure16:53
bknudsonfungi: I set PIP_DOWNLOAD_CACHE16:54
dstufftfungi: obviously the solution is to just prefix every command with sudo16:54
fungibknudson: that's one possible workaround, yes16:54
bknudsonmaybe I don't need to do that anymore.16:54
zarofungi: do you know why ci generates version info with something like this?  <version>.postX16:54
zarofungi: example http://docs-draft.openstack.org/32/144232/1/check/gate-jenkins-job-builder-docs/f7703cd//doc/build/html/16:55
fungibknudson: but passing -H to sudo will allow pip to properly resolve $HOME to ~root16:55
*** e0ne has joined #openstack-infra16:55
*** rfolco has joined #openstack-infra16:55
zarofungi: i believe it's something to do with pbr, older builds have <version>.<git_sha>16:55
fungizaro: pbr does that to be pep 440 compliant for post-versioned projects16:55
*** yfried_ has quit IRC16:55
dstufftbknudson: no you don't (infact the "download cache", which was a custom bit of caching code that was janky is deprecated and no-oped). In pip 6 there's the HTTP cache which is an opt-out cache of all HTTP access tha respects Cache-Control headers and also supports conditional HTTP GETs even if the cache object has expired (although it only supports cache for hosts that don't require the --trusted-host flag to not get warnings)16:56
fungizaro: embedding the git sha in the version string causes setuptools to misbehave (it sorts the resulting version in a non-intuitive manner)16:56
*** e0ne has quit IRC16:57
bknudsondstufft: interesting... I do have a ~/.cache/pip/ already and updated recently.16:58
zarofungi: thanks.16:58
fungizaro: that change was introduced circa pbr 0.10.3 as part of the setuptools 8 coping scramble16:59
dstufftbknudson: (the one caveat to "we respect cache-control headers" bit is we limit index page caches (not package files themselves) to a 10 minute cache time, though it's smart enough to sitll do a conditional get)16:59
*** rfolco has quit IRC17:00
fungizaro: *moght have been more like 0.10.4 or 0.10.5... it's all a blur to me now)17:00
fungis/*moght/(might/17:01
*** e0ne has joined #openstack-infra17:01
*** marcusvrn1 has quit IRC17:01
*** rfolco has joined #openstack-infra17:02
*** _nadya_ has joined #openstack-infra17:03
*** marcusvrn has joined #openstack-infra17:04
*** henditan has quit IRC17:05
*** _nadya_ has quit IRC17:05
*** rfolco has quit IRC17:06
bknudsonjroll: does https://review.openstack.org/#/c/144216/ need to be backported to juno and icehouse?17:08
*** marcusvrn has quit IRC17:08
*** marcusvrn has joined #openstack-infra17:08
*** andreaf has joined #openstack-infra17:09
bknudsonmaybe just include it in https://review.openstack.org/#/c/144162/.17:09
jrollbknudson: I have no idea, was just letting you know :)17:10
jrolladam_g: ^^17:10
fungilooks like 144216 is about to merge. the neutron change ahead of it just succeeded it's remaining job'17:10
fungier, its17:10
*** lttrl has joined #openstack-infra17:11
fungiand there it goes. so at least nova should be unblocked on master now17:14
*** e0ne has quit IRC17:15
bknudsonI can do the backport easy enough... include it in https://review.openstack.org/#/c/144162/ ? mordred17:16
bknudsonI assume that all these changes are required to get juno and icehouse working again17:16
*** garyh has quit IRC17:17
*** rfolco has joined #openstack-infra17:19
*** e0ne has joined #openstack-infra17:19
*** SumitNaiksatam has quit IRC17:21
mordredzaro: we might should figure out how to pull the sha from the pbr.json file and show it on docs publications17:22
mordredzaro: but I don't think that's urgent probably17:22
mordredbknudson: looking17:22
mordredbknudson: feel free to take that over - I'm mostly AFK at the moment17:22
openstackgerritKevin L. Mitchell proposed stackforge/python-jenkins: Raise an exception on 404  https://review.openstack.org/13915817:23
bknudsonmordred: ok, I'll push a new patch set to https://review.openstack.org/#/c/144162/ and stable that includes https://review.openstack.org/#/c/144216/.17:23
mordredbknudson: cool17:23
*** rfolco has quit IRC17:23
*** tonytan4ever has quit IRC17:24
*** tonytan4ever has joined #openstack-infra17:25
*** achanda has joined #openstack-infra17:27
*** e0ne has quit IRC17:29
zaromordred: so i just noticed that pbr VersionInfo no longer works?  the published jjb docs has an old version http://ci.openstack.org/jenkins-job-builder/17:29
*** tonytan4ever has quit IRC17:29
*** vigneshvar has joined #openstack-infra17:31
zaroit should be 1.1.0 instead of 1.0.017:31
*** pc_m has quit IRC17:32
*** e0ne has joined #openstack-infra17:33
openstackgerritMerged openstack-infra/storyboard: Fix i18n in exception module.  https://review.openstack.org/14399217:34
openstackgerritMerged openstack-infra/storyboard: Extended titles  https://review.openstack.org/14398417:35
*** rfolco has joined #openstack-infra17:35
openstackgerritMerged openstack-infra/storyboard: All aborts now have message  https://review.openstack.org/14346517:36
*** e0ne has quit IRC17:38
*** rfolco has quit IRC17:40
*** rfolco has joined #openstack-infra17:40
krotscheckfungi: Ready when you are.17:42
*** e0ne has joined #openstack-infra17:42
*** ddieterly has quit IRC17:43
*** rfolco has quit IRC17:45
fungikrotscheck: so there was an outstanding puppet review, yeah?17:47
krotscheckfungi: Yep. https://review.openstack.org/#/c/140466/17:50
*** e0ne has quit IRC17:52
*** teran_ has joined #openstack-infra17:52
fungikrotscheck: okay, so the to do list there is i need to set up a dedicated trove db for it and put the relevant db (and some generated rabbitmq creds too) into hiera, then try to launch a fresh instance with that change applied17:55
fungii'll get started on the trove/hiera bits now17:55
krotscheckfungi: Thanks. It’ll need to be a MySQL 5.6 db17:55
*** teran__ has joined #openstack-infra17:55
*** e0ne has joined #openstack-infra17:55
krotscheckOr, well, something-that-supports-mysql-syntax-and-does-fulltext-on-innodb17:56
fungikrotscheck: i think we just get whatever we get, but it'll be their latest which is probably 5.6 or later17:56
*** teran has quit IRC17:56
krotscheckfungi: Got it.17:56
*** e0ne has quit IRC17:57
*** rfolco has joined #openstack-infra17:57
*** teran_ has quit IRC17:57
dkranzfungi: I no longer see the tempest log file in build results. Was this intentional?17:59
fungidkranz: i do not know18:00
fungimtreinish: ^ ?18:00
dkranzfungi: k, I'll track it down18:00
dkranzfungi: Do you know which of our projects has the code that controls which log files are shown?18:01
*** vigneshvar has quit IRC18:01
*** e0ne has joined #openstack-infra18:01
*** rfolco has quit IRC18:02
fungidkranz: devstack-gate decides what files to archive, but in most cases it's just archiving anything left in the devstack logdir so devstack and/or tempest could have ceased creating that file or started putting it somewhere else18:02
*** hdd has joined #openstack-infra18:02
dkranzfungi: Ah, thanks. I have a clue as to what is going on.18:02
fungi(and ultimately it's really a jenkins job publisher macro which does the retrieval/archiving)18:03
*** zz_dimtruck is now known as dimtruck18:04
*** yamahata has joined #openstack-infra18:04
*** dims_ has quit IRC18:08
*** dims has joined #openstack-infra18:08
*** otter768 has joined #openstack-infra18:08
*** achanda has quit IRC18:09
*** achanda has joined #openstack-infra18:09
*** armax has joined #openstack-infra18:10
*** isviridov is now known as isviridov_away18:13
*** otter768 has quit IRC18:13
*** imcsk8 has quit IRC18:13
*** imcsk8 has joined #openstack-infra18:13
*** achanda has quit IRC18:14
*** ddieterly has joined #openstack-infra18:14
*** rfolco has joined #openstack-infra18:14
*** mfink has joined #openstack-infra18:14
*** HeOS has quit IRC18:14
*** hdd has quit IRC18:16
*** ddieterl_ has joined #openstack-infra18:16
*** ddieterly has quit IRC18:16
dstuffthmm18:17
dstuffttchaypo: mordred sudo python -c "import os.path,getpass; print((os.path.expanduser('~'), os.path.expanduser('~{}'.format(getpass.getuser()))))" -> ('/Users/dstufft', '/var/root')18:17
dstufftusing ~user instead of ~ bypasses looking at the HOME dir18:18
dstufftnot sure if that's better or not18:18
dstuffter HOME env18:18
*** ivar-lazzaro has quit IRC18:18
*** ddieterl_ has quit IRC18:20
fungidstufft: well, i can see that also working around similar problems where an admin may use sudo to call pip as another (non-root) user and would thus fail to create .cache due to permissions on their homedir18:20
*** armax has quit IRC18:21
*** yfried_ has joined #openstack-infra18:22
*** hdd has joined #openstack-infra18:22
dstufftfungi: yea, my only concern with that is it'll break people setting $HOME to something different and expecting that to work since we won't inspect the environment anymore18:22
*** cinerama has quit IRC18:22
fungidstufft: good point18:22
fungisudo -H is still a preferable way of ensuring it18:23
fungii mean, that's the entire reason that sudo option exists18:23
*** cinerama has joined #openstack-infra18:24
*** patrickeast has joined #openstack-infra18:25
dstufftI *think* what pip should do here is just detect if the directory we're attempting to write to is owned by the user we're executing under, and just disable things that write to the home directory if it's not owned by the same user18:25
fungidstufft: that seems like a very safe fallback. i'm in favor18:26
dstufftthe only time I can think that would "fail", is if someone has their home directory set to something like /whatever/that/is/not/owned/by/them/ but where they can create directories, we won't automatically create the directory and will turn off stuff... but I don't think that's a very large number of users18:27
*** unicell has quit IRC18:27
*** mfink__ has quit IRC18:27
*** yolanda has quit IRC18:27
fungidstufft: then the discussion becomes "to get proper caching with pip when called from sudo, make sure to use -H or otherwise set a reasonable homedir owned by the target account"18:27
*** rfolco has quit IRC18:28
*** yolanda has joined #openstack-infra18:29
fungirather than "why is pip creating root-owned files in my homedir?!?"18:29
*** rfolco_ has joined #openstack-infra18:29
*** gingerjiang has quit IRC18:29
dstufftand we'll "fail soft" with a warning that will let them mkdir -p to fix it, instead of failing hard on sudo pip (often times with unrelated commands later on)18:29
*** dstufft has quit IRC18:29
*** NikitaKonovalov has quit IRC18:29
*** yjiang5 has joined #openstack-infra18:29
*** dstufft has joined #openstack-infra18:30
*** Masahiro has joined #openstack-infra18:30
*** antonym has quit IRC18:30
fungifreenode brings the lart to bear18:30
*** NikitaKonovalov has joined #openstack-infra18:31
*** antonym has joined #openstack-infra18:31
dstufftwtf excess flood18:31
*** unicell has joined #openstack-infra18:31
fungilag-induced flood?18:31
*** achanda has joined #openstack-infra18:31
dstufftprobably18:33
dkranzfungi: sorry to bug you again. I presume it is known that stable/icehouse builds started failing on or about the 25th?18:33
*** mikal has quit IRC18:33
*** mikal_ has joined #openstack-infra18:33
*** rfolco_ has quit IRC18:33
dstufftfungi: tchaypo mordred recorded here: https://github.com/pypa/pip/issues/2285 gonna see about making a fix for it18:34
fungidkranz: i'm surprised they worked prior to that, but yes i think bknudson, gordc, jroll and mordred were discussing the current patches proposed to get it back on track18:34
*** dhritishikhar_ has quit IRC18:34
*** Masahiro has quit IRC18:34
dkranzfungi: ok, thanks. This passed on the 23rd https://review.openstack.org/#/c/137239/18:35
*** rfolco_ has joined #openstack-infra18:35
*** e0ne has quit IRC18:36
dkranzfungi: It is an unfortunate consequence that all tempest builds will fail until the issue is fixed18:36
fungidkranz: i think that was when we finally managed to un-break ourselves enough after the pip 6.0 release to get new nodepool images built18:37
*** armax has joined #openstack-infra18:37
dkranzfungi: ah, a short window or workingness :)18:37
*** tonytan4ever has joined #openstack-infra18:38
fungidkranz: well, things were "working" to some extent before that because we were too broken to build images with the new pip/virtualenv/setuptools releases18:38
fungiso you were effectively getting a stale build environment with older versions18:39
*** andreykurilin has joined #openstack-infra18:39
*** Longgeek has quit IRC18:39
*** rfolco_ has quit IRC18:40
dstufftfungi: I think #7 here has been fixed now yea? https://etherpad.openstack.org/p/pydistutils-issues18:41
fungidstufft: yep, i believe so18:42
fungidstufft: at least for our ci it's fixed18:43
dstufftsetuptools is on 9.1 or some such thing now18:43
dstufftsetuptools is actually an experiment in CD versioning for OSS18:43
fungidstufft: as opposed to my software which is usually an experiment in ocd versioning18:44
dstufftfungi: :D18:44
dstufftI already have a list of things I want to do in a "Son of PEP 440" PEP :/18:45
dstufftbackwards compat now though!18:45
fungirevenge of the return of son of pep 440's revenge18:46
dstuffthttps://gist.github.com/dstufft/88261e1e252756ba1cd918:46
fungii'm in favor of all three of those ideas18:47
dstufftlol18:49
dstufftpeople are apparentling import random crap in pip for use in their setup.py's18:49
dstufftdon't do this people ;((18:49
*** marun has joined #openstack-infra18:50
*** Longgeek has joined #openstack-infra18:51
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Corrected type casting in OAuth Response  https://review.openstack.org/14434118:51
*** bdpayne has joined #openstack-infra18:51
krotscheckProduction problem on storyboard ^^18:52
yolandahi people, are you working today?18:54
*** armax has quit IRC18:55
* zaro is18:55
*** achanda has quit IRC18:55
*** Longgeek has quit IRC18:55
yolandahi Khai, going to take a look at that jjb stuff18:55
fungidstufft: you mean like pbr? (though pbr only calls pip.__init__)18:56
krotscheckhi hi18:56
*** armax has joined #openstack-infra18:56
yolandahi krotscheck, how is christmas going?18:56
fungiyolanda: i'm not people, but i guess i'm working as much as i ever do18:56
*** Ryan_Lane has joined #openstack-infra18:57
krotscheckyolanda: Spiffy!18:57
krotscheckyolanda: You?18:57
yolandamy children had a new Furby Boom for christmas, so they don't want to go outside, and do anything but play with furby :(18:57
dstufftfungi: I think pbr shells out to ``python -m pip.__init__`` or something like that right?18:57
yolandaso going to take a look at reviews18:57
dstufftI doubt we'll ever break that tbh18:57
fungidstufft: yeah, sort of an entrypoint18:57
krotscheckyolanda: Can you take a quick look at https://review.openstack.org/#/c/144341/18:57
krotscheckProduction issue on storyboard :)18:57
yolandasure18:57
dstufftfungi: more like "I imported random functions from pip, used thme in setup.py, and now my setup.py is broke because the signature of that function changed"18:58
dstufftif y'all were doing that I'd feel no remorse breaking it ;)18:58
yolandakrotscheck, makes sense to me, when did it happen?18:58
krotscheckyolanda: That happened when Aleksey’s six merges went in.18:58
krotscheckAbout an hour or so ago18:58
yolandaok, +118:59
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Corrected type casting in OAuth Response  https://review.openstack.org/14434118:59
krotscheckdagnabbit. pep hates me.18:59
*** pcaruana has quit IRC19:00
fungidkranz: bknudson: gordc: jroll: mordred: i held an unused devstack-precise node (the kind we use for those failing icehouse jobs) and confirmed that setuptools 9.0 is preinstalled on our images, so something must be downgrading it to 8.2.1 within the job (maybe a transitive dep?)19:00
krotscheckQuestion about Py2 vs Py3. I know that under the hood, str() is very different, but if I do a check in Py2 type(val) is str, shouldn’t that behave the ‘same’ way in Py3?19:00
*** achanda has joined #openstack-infra19:00
fungikrotscheck: easy enough to test. also "is" and "==" have slight differences in behavior as comparison operators19:01
* krotscheck facepalms.19:01
krotscheckgoody19:01
zaroyolanda: furby thing looks a little creepy19:02
*** rockyg has joined #openstack-infra19:02
krotscheckSeems to work :/19:02
*** pcrews has joined #openstack-infra19:02
dstufftkrotscheck: probably you need to define what exactly you're trying to check for19:03
dstuffttext or bytes19:03
dstufft:D19:03
fungikrotscheck: in some places you can use == to mean "equivalent to" while "is" pretty much always means "identical to"19:04
fungikrotscheck: though typevals seem to always be distinct even with ==19:04
dstufftfungi: well ``is`` is the identity oeprator and ``==`` is the equality operator, ``is`` generally implies ``==`` but ``==` doesn't imply ``is``19:06
openstackgerritMerged openstack-infra/storyboard: Corrected type casting in OAuth Response  https://review.openstack.org/14434119:07
dstufftI say generally because you can override __eq__ on a class and make == mean whatever you want19:07
*** rockyg has quit IRC19:08
*** dtroyer has quit IRC19:08
fungikrotscheck: i suppose if you want to be able to operate on the text content of an object regardless of type you can test with hasattr for '__str__' (or just try to use it and rely on exception handling)19:09
* dims scrolls back to see if anyone has seen "ImportError: No module named command.install"19:09
fungidims: that's the "pip removes setuptools in preparation for upgrading it, but then tries to install something else which uses it in between)19:09
fungidims: i think we have something in icehouse with a (possibly transitive) dep on setuptools<8.319:10
funginot immediately obvious from the log what19:11
fungioh!19:11
dimsfungi: ack thanks!19:11
fungicreate: /opt/stack/new/tempest/.tox/full19:11
*** ddieterly has joined #openstack-infra19:11
fungithis is the tempest virtualenv getting created with setuptools 8.2.1 and then trying to upgrade to latest setuptools because pip is called with -U and something getting installed has a dependency declared on setuptools19:12
fungithe cryptography package possibly19:13
fungiCollecting setuptools from http://pypi.DFW.openstack.org/packages/3.4/s/setuptools/setuptools-9.0.1-py2.py3-none-any.whl#md5=fba66e93e17d5a8c2dc9a8f0beddc5a9 (from cryptography>=0.2.1->pyOpenSSL>=0.11->python-glanceclient!=0.14.0,<0.15,>=0.9.0->-r /opt/stack/new/tempest/requirements.txt (line 13))19:13
dstufftfungi: hm19:13
dstufftfungi: I thought we fixed that19:13
*** aberezin has joined #openstack-infra19:14
dimsdstufft: still seeing hits - http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSW1wb3J0RXJyb3I6IE5vIG1vZHVsZSBuYW1lZCBjb21tYW5kLmluc3RhbGxcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiODY0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDE5ODgwMDY1Mjk0fQ==19:14
fungidstufft: regression? http://logs.openstack.org/61/144161/5/check/check-tempest-dsvm-full/a62d8cb/console.html#_2014-12-29_17_49_19_17519:14
dstufftdims: sorry I don't mean "we" as in openstack, I mean "we" as in I thought pip had previously fixed that problem, I could be remembering wrong19:14
dims:)19:14
fungidstufft: we _mostly_ work around that by preinstalling the latest setuptools before we do anything else19:15
dstufftit could alos be a symptom of the hacks we put into place for the distribute -> setuptools upgrade merging with the change I made to install packages in the reverse order19:15
fungibut it's possible that pip also installed it19:15
fungier, also fixed it19:15
fungiand then perhaps regressed19:15
dstufft(reverse order being -> "mostly try to install dependencies before installing the things that depend on them")19:15
fungidims: so anyway, it's possible we can work around this by preinstalling latest setuptools in tempest's tox venv19:16
*** adalbas has quit IRC19:18
dimsfungi: all the hits do seem to be dsvm jobs, so +1!19:18
dstufftPython Packaging: At least we validate some TLS now19:19
fungidims: yeah, i think this is only cropping up in the venv tempest uses. so workaround for it goes in tempest19:19
fungidims: i'll propose it and see what happens19:20
fungidims: is there a bug number?19:22
dimsfungi: no19:22
dimsjust spotted it19:22
dstufftfungi: some day, there is going to be a world where you can take some code, shove it in a tarball, give it to other people, and things will just work and you won't need 20 layers of work arounds in 5 different projects just to reimplement wget, tar, and cp19:22
fungidstufft: don't forget ar and cpio ;)19:22
dimsdstufft: We can only wish!19:23
dstufftin happier news, my virtualenv rewrite can create virtual environments now for 2.7 and 3.3+ on OSX! Probably not on Windows and maybe not on Linux and probably there's a bunch of edge cases that I rebroke by rewriting things19:24
dstufftbut hey, I can make a virtual environment and install things into it19:24
*** dtroyer has joined #openstack-infra19:24
fungidkranz: bknudson: gordc: jroll: mordred: dims: dstufft: stable compat jobs on https://review.openstack.org/144347 should tell us if it's a viable workaround19:27
gordcfungi: cool cool. i'll keep an eye on that patch.19:27
*** lttrl has quit IRC19:28
fungilooks like maybe we have bug 1405579 tracking it19:28
uvirtbotLaunchpad bug 1405579 in python-heatclient "tempest-dsvm-neutron-src-python-heatclient-icehouse failed due to ERROR:   smoke: could not install deps" [Undecided,New] https://launchpad.net/bugs/140557919:28
bknudsonadding the sudo -H change to mordred's change ( https://review.openstack.org/#/c/144162 ) didn't seem to help... should I revert it back?19:32
fungibknudson: which error were you attempting to resolve that way?19:32
bknudsonfungi: that's supposed to fix the issue in  https://review.openstack.org/#/c/144162/ with check-grenade-dsvm-ironic-sideways with "mkdir: cannot create directory '/opt/stack/new/.cache': Permission denied" -- let me check if that's gone away19:35
*** andreykurilin has quit IRC19:35
bknudsonfungi: that problem did not go away: 2014-12-29 17:43:31.795 | mkdir: cannot create directory '/opt/stack/new/.cache': Permission denied19:35
*** reed has joined #openstack-infra19:36
adam_gbknudson, i think grenade's functions-common needsa a sync from devstack19:36
*** armax has quit IRC19:37
adam_ggrenade's prep-base does some pip installing, currently without -H it looks19:37
zaroashp: ping19:38
fungibknudson: adam_g: that sounds probable19:39
adam_ghavne't had a chance to look closely at the delta, but this might help https://review.openstack.org/14435019:40
bknudsonadam_g: the file is in sync now and it's got a change to sudo -H.19:44
fungibknudson: in the grenade repo?19:45
adam_gbknudson, in grenade? doesnt look like it19:45
bknudsonfungi: adam_g: with this change https://review.openstack.org/#/c/144350/19:45
adam_gim not sure i follow? there's definitely a diff, https://review.openstack.org/#/c/144350/1/functions-common :)19:46
adam_gsimilar update probably needs to happen in grenades stable/juno too19:46
*** HeOS has joined #openstack-infra19:47
fungioh, i see. _with_ 144350 the file has no diff against the copy which is in devstack19:47
*** _nadya_ has joined #openstack-infra19:48
*** hdd has quit IRC19:48
bknudsonstable/juno has functions rather than functions-common19:49
adam_gyeah, the sync into stable/juno looks a bit hairy19:50
adam_gi thought it had been synced before cutting that branch19:50
adam_ghttps://review.openstack.org/14435219:52
fungikrotscheck: okay, i've finally gotten around to adding the trove db and getting all the credentials into hiera19:55
fungikrotscheck: trying out that change on a fresh instance momentarily19:56
krotscheckfungi: Excellent, thanks19:56
*** annegentle has quit IRC19:56
openstackgerrityolanda.robla proposed openstack-infra/storyboard-webclient: Add timeout to the blur event for story inline edition  https://review.openstack.org/14435419:57
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Interpolate params inside of params.  https://review.openstack.org/14313819:58
*** dhritishikhar_ has joined #openstack-infra19:58
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Added cron plugin to clean old access tokens.  https://review.openstack.org/14053620:01
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard: Consolidated OAuth Configuration, added enable flag to oauth cleaner.  https://review.openstack.org/14435520:01
*** dhritishikhar_ has quit IRC20:01
*** e0ne has joined #openstack-infra20:05
dtroyerI'm +2 on both Grenade reviews…I'm not sure any other cores are about today…20:07
*** otter768 has joined #openstack-infra20:09
openstackgerritMichael Krotscheck proposed openstack-infra/puppet-storyboard: Added Cron and Token Cleanup to configuration.  https://review.openstack.org/14435620:09
*** armax has joined #openstack-infra20:11
*** armax has quit IRC20:13
*** otter768 has quit IRC20:13
*** ddieterl_ has joined #openstack-infra20:15
*** ddieterly has quit IRC20:19
*** Masahiro has joined #openstack-infra20:19
*** fifieldt has quit IRC20:24
*** Masahiro has quit IRC20:24
*** fifieldt has joined #openstack-infra20:24
*** e0ne has quit IRC20:26
openstackgerritBrant Knudson proposed openstack/requirements: Remove configobj  https://review.openstack.org/14436320:28
fungikrotscheck: Error: /Stage[main]/Storyboard::Cert/File[/etc/ssl/certs/storyboard.pem]: Could not evaluate: Could not retrieve information from environment fungi source(s) file:/etc/ssl/certs/ssl-cert-snakeoil.pem20:29
fungikrotscheck: i'm looking into the puppet module now to see if i can tell why it's trying to retrieve that file20:30
*** annegentle has joined #openstack-infra20:30
krotscheckfungi: It’s trying to use it as the SSL cert for the VHost, but I believe that apache probably hasn’t been installe dyet.20:30
dkranzfungi: I gave your change in tempest a +2.20:31
dkranzfungi: But I am not really a pip guru.20:31
fungidkranz: ooh, did the jobs finish for it?20:31
dkranzfungi: yes20:32
dkranzfungi: If you need me to +A because no one else is around, I can do that if you are sure it is right or get +1 from others more knowing20:32
fungidkranz: great--does indeed seem to work around that race. i assume icehouse backward compat jobs for other tempest changes fail?20:33
dkranzfungi: yes, you can see here https://review.openstack.org/#/q/project:openstack/tempest+status:open,n,z20:33
dkranzfungi: That is actually how I noticed this20:33
fungiyep20:33
dkranzfungi: I have to run out for a bit but let me know what you want me to do more, if anything.20:34
fungidkranz: i'll leave it to tempest cores to decide. i consider it an ugly workaround to a bug which probably needs to be addressed in pip, but can't think of a better option to get you un-stuck for now20:34
dkranzfungi: ok, thanks. We will see what happens with the tempest core APAC folks tonight20:36
fungii suppose i could stand to update the commit message linking to the corresponding tempest bug report in lp, and then open something about the regression in pip's tracker20:36
dkranzfungi: Yeah, and I am going to send a heads up to the ml so the tempest cores know what is going on. Do you have a link to the bug for reference?20:37
fungidkranz: i just updated the commit message with it20:37
dkranzfungi: thanks!20:37
fungididn't have it handy when i originally pushed the workaround test20:37
fungii'll try to hunt down the pip bug and add it as a comment in that lp bug20:38
fungikrotscheck: okay, i see what's going on. storyboard::cert tries to copy the specified file to /etc/ssl/certs/storyboard.whatever instead of having the vhost point to the file directly20:41
krotscheckfungi: That makes sense, since the file may come from several different origins.20:42
*** annegentle has quit IRC20:43
fungikrotscheck: templates/storyboard_https.vhost.erb could probably use storyboard::cert::ssl_cert_file et cetera, and then cert.pp could set that variable if it doesn't exist?20:43
krotscheckfungi: Can we guarantee that the file permissions will work though? Copying it gives us control over the where-and-who of those certs.20:44
fungikrotscheck: that way for a dev server the vhost ends up obviously using /etc/ssl/private/ssl-cert-snakeoil.* when configured to do so20:44
fungikrotscheck: the file permissions are set by the apache package so that they can be read by the user running apache20:45
fungikrotscheck: that's basically what the modules for review-dev, groups-dev, openstackid-dev, et cetera do20:45
fungirather, how the modules which implement those handle it20:46
fungikrotscheck: for example http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/gerrit/templates/gerrit.vhost.erb and http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/gerrit/manifests/init.pp20:47
fungikrotscheck: basically of content is specified, then try to create the file using that content, otherwise assume the file must already exist at the indicated path, but use it the same way in tje vhost config regardless20:48
fungis/tje/the/20:48
fungimy typing is going to hell this afternoon20:49
krotscheckfungi: That seems like a lot of work for a server :)20:50
krotscheckfungi: Also, I don’t exactly agree with the “Assume the file exists” philosophy. I prefer having storyboard try to copy the file into a known location, which also enables us to trigger a server restart if someone futzes with the cert.20:53
krotscheckThe downside of that though is that it doesn’t like it if the apache package isn’t installed yet, which I don’t really have a solution for yet because of how the apache module is designed.20:54
krotscheckOr rather, how the out-of-date version of the apache module that clarkb loves is designed.20:54
krotscheckI can’t explicitly reference “Hey this is dependent on the package install” :/20:54
fungikrotscheck: i see a chicken-and-egg issue there20:57
fungikrotscheck: those files are set to be created before the storyboard::application class20:58
fungikrotscheck: which is in turn where the apache module is being added20:58
krotscheckfungi: Yep. Moving it anywhere else causes dependency loops.20:59
fungikrotscheck: perhaps an "include apache" in storyboard:cert and then require apache in each file resource solves this?20:59
krotscheckfungi: I thought I tried that, but I can again.21:00
*** gordc has quit IRC21:02
clarkbso apache package being installed should not matter at all for certs21:03
fungikrotscheck: regardless, i'm not sure i understand the objection to having a dynamic certificate file name in the vhost config... have the module let you specify the name of the file and optionally its content. we do it for a lot of other servers and it trivially supports using the snakeoil files21:03
*** dkranz has quit IRC21:03
clarkbthey are completely independent of each other since it turns out you need certs on a machine for things other than apache21:03
fungiclarkb: it matters because the snakeoil cert files are installed by the apache package maintscripts, and storyboard's puppet module wants to copy them to a new name before referencing them in the vhost config21:04
krotscheckclarkb: So apache shouldn’t install the snakeoil certs? Gotcha.21:04
*** dkranz has joined #openstack-infra21:04
clarkbfungi: no they aren't iirc21:04
clarkbfungi: they are installed by a different package which you can independently install and reinstall and so on21:04
clarkbkrotscheck: no, just saying apache isn't really the problem here. Directly depend on that stuff if you want21:05
fungiclarkb: oh, that package is being installed as a dep of apache221:05
clarkbkrotscheck: but don't blame apache21:05
clarkbbut yes I agree with fungi. I am not sure I understand why we would want to use a different mechanism here when the existing one seems to work well across all of the other servers21:05
*** _nadya_ has quit IRC21:06
krotscheckclarkb: I’m blaming the apache module, which rolls its internal dependencies in such a way that I can’t say: Hey, first pull the apache _package_, then do stuff, then set up a vhost.21:06
krotscheck(apache puppet module)21:06
krotscheckclarkb, fungi: Well, I’d be more than happy to review a patch to puppet-storyboard that does that.21:06
fungikrotscheck: well, that's because puppet is configuration management, and you seem to want an orchestration engine21:06
fungikrotscheck: i'll whip one up now--you seemed allergic to the idea moments ago or i would have offered then21:07
*** andreaf has quit IRC21:07
*** andreaf has joined #openstack-infra21:07
*** jaypipes has quit IRC21:15
*** ihrachyshka has quit IRC21:16
*** ihrachyshka has joined #openstack-infra21:16
clarkbfungi: for https://review.openstack.org/#/c/144347/2 do we know what depends on setuptools?21:18
clarkbfungi: mostly curious if its one of openstack's libs as we should fix that if so21:19
fungiclarkb: cryptography (via something else via glanceclient i think)21:19
fungiclarkb: it says in the pip output what teh dependency chain is which caused setuptools to get upgraded21:19
clarkbfungi: thanks21:20
clarkbno alex_gaynor here but maybe we can find out why they have done that21:20
fungiclarkb: from cryptography>=0.2.1->pyOpenSSL>=0.11->python-glanceclient!=0.14.0,<0.15,>=0.9.021:21
fungiclarkb: so i take that to mean that latest version of cryptography has an explicit dependency on setuptools (unversioned)21:21
*** mase_x200 has joined #openstack-infra21:22
*** ihrachyshka has quit IRC21:22
*** andreykurilin has joined #openstack-infra21:23
clarkbya their setup.py seems to explicity require setuptools21:25
openstackgerritJeremy Stanley proposed openstack-infra/puppet-storyboard: Allow overriding the cert/key file paths  https://review.openstack.org/14436721:27
fungikrotscheck: clarkb: ^ is what i was proposing21:27
*** gordc has joined #openstack-infra21:29
*** mase_x200 has quit IRC21:31
*** dkranz has quit IRC21:33
*** dkranz has joined #openstack-infra21:34
nibalizerkrotscheck: does this change help? https://review.openstack.org/#/c/126086/12/manifests/cert.pp21:34
nibalizeri have that on a review that does other things21:34
* nibalizer hasn't actually read the scrollback21:34
dimsfungi: we had neglected to cut branches for stackforge/nova-docker, is there an easy way to do them now if i have the commit ids?21:35
fungidims: if you have a group with create perms for refs/heads/* in your gerrit acl then those people can just do it21:35
*** weshay has quit IRC21:35
dimscool, will try21:36
funginibalizer: yep, that's an alternative workaround, though not consistent with how we've implemented in other modules (doesn't mean it's bad, just means additional divergence between our modules so hopefully for a good reason)21:36
fungianyway, gonna go grab an early dinner, then come back and see what else needs help21:38
krotscheckfungi: One comment on your patch, some var c/p issue.21:38
fungikrotscheck: great catch. thanks! fixing21:39
openstackgerritJeremy Stanley proposed openstack-infra/puppet-storyboard: Allow overriding the cert/key file paths  https://review.openstack.org/14436721:40
krotscheckTHis is going to require some coordination with a simila patch on system-config, no?21:40
fungikrotscheck: i haven't checked yet to see whether the production server is overriding any of those21:40
krotschecknibalizer: What fungi said. I’m all for simplifying things so that habituated brain patterns encourage use :)21:41
* fungi will bbiab21:41
*** tonytan4ever has quit IRC21:42
jheskethMorning21:44
*** zz_sabari is now known as sabari21:44
anteayamorning jhesketh21:45
*** gondoi is now known as zz_gondoi21:45
*** achanda has quit IRC21:47
*** ihrachyshka has joined #openstack-infra21:48
*** mikal_ is now known as mikal21:52
*** ivar-lazzaro has joined #openstack-infra21:55
*** ihrachyshka has quit IRC21:55
*** tonytan4ever has joined #openstack-infra22:01
*** Sukhdev has joined #openstack-infra22:02
*** sabari is now known as zz_sabari22:04
*** krtaylor has quit IRC22:06
*** Masahiro has joined #openstack-infra22:08
*** otter768 has joined #openstack-infra22:10
*** ihrachyshka has joined #openstack-infra22:10
*** erw_ is now known as erw22:11
*** Masahiro has quit IRC22:12
*** otter768 has quit IRC22:15
*** teran_ has joined #openstack-infra22:16
*** krtaylor has joined #openstack-infra22:17
*** teran__ has quit IRC22:17
*** ivar-lazzaro has quit IRC22:18
*** ihrachyshka has quit IRC22:23
*** ihrachyshka has joined #openstack-infra22:26
*** tonytan4ever has quit IRC22:27
*** andreykurilin has quit IRC22:28
*** andreykurilin has joined #openstack-infra22:29
*** dimtruck is now known as zz_dimtruck22:31
*** cnesa has quit IRC22:40
*** zz_sabari is now known as sabari22:47
*** ihrachyshka has quit IRC22:50
*** ihrachyshka has joined #openstack-infra22:51
*** ihrachyshka has quit IRC22:55
*** sabari is now known as zz_sabari22:57
*** zz_dimtruck is now known as dimtruck23:00
adam_gfungi, dtroyer one last backport that i think we need to fix the ironic grenade job https://review.openstack.org/#/c/144374/23:01
*** zz_sabari is now known as sabari23:02
*** teran has joined #openstack-infra23:02
*** gordc has quit IRC23:03
openstackgerritKhai Do proposed openstack-infra/storyboard: Allow overriding test database from an environment variable  https://review.openstack.org/14437523:04
*** teran_ has quit IRC23:06
*** sabari is now known as zz_sabari23:09
*** mfink has quit IRC23:12
*** zz_sabari is now known as sabari23:15
*** amitgandhinz has quit IRC23:16
*** mfink has joined #openstack-infra23:17
*** andreaf has quit IRC23:18
*** andreaf has joined #openstack-infra23:19
openstackgerritJeremy Stanley proposed openstack-infra/puppet-storyboard: Allow overriding the cert/key file paths  https://review.openstack.org/14436723:20
openstackgerritKhai Do proposed openstack-infra/storyboard: Allow overriding test database from an environment variable  https://review.openstack.org/14437523:24
*** dims_ has joined #openstack-infra23:25
*** sigmavirus24 is now known as sigmavirus24_awa23:26
*** dims has quit IRC23:28
openstackgerritKhai Do proposed openstack-infra/storyboard: Allow overriding test database from an environment variable  https://review.openstack.org/14437523:30
*** lttrl has joined #openstack-infra23:31
*** sabari is now known as zz_sabari23:33
*** zz_sabari is now known as sabari23:33
*** sigmavirus24_awa is now known as sigmavirus2423:34
*** sabari is now known as zz_sabari23:35
*** dims_ has quit IRC23:39
*** otter768 has joined #openstack-infra23:40
*** lttrl has quit IRC23:43
*** lttrl has joined #openstack-infra23:43
*** dims__ has joined #openstack-infra23:50
*** unicell has quit IRC23:52
*** dmsimard is now known as dmsimard_away23:52
*** Masahiro has joined #openstack-infra23:56
*** zz_sabari is now known as sabari23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!