| mordred | clarkb: woot | 00:01 |
|---|---|---|
| lifeless | mordred: you use it from pypi right ? | 00:01 |
| *** arborism has quit IRC | 00:04 | |
| clarkb | I need a firefox plugin that won't let me have two tab of the same uri open at the same time unless I click some override | 00:07 |
| lifeless | clarkb: ^ | 00:07 |
| lifeless | do we use django-openid-auth from pypi | 00:07 |
| lifeless | or debs | 00:07 |
| clarkb | lifeless: pypi | 00:07 |
| lifeless | cool | 00:08 |
| lifeless | thanks | 00:08 |
| lifeless | mordred: ttx: mail sent. | 00:08 |
| mordred | lifeless: yes | 00:10 |
| mordred | lifeless: thank you! | 00:10 |
| clarkb | mordred: https://jenkins.openstack.org/job/periodic-mirror-python27/142/consoleFull did not copy pycparser to the mirror | 00:10 |
| mordred | clarkb: why not? | 00:10 |
| clarkb | mordred: no idea | 00:10 |
| * clarkb does some testing | 00:10 | |
| mordred | clarkb: oh. | 00:11 |
| *** mkirk_ has quit IRC | 00:11 | |
| mordred | clarkb: I betcha it's a setup_requires | 00:11 |
| lifeless | mordred: so run-mirror; brass tacks | 00:11 |
| lifeless | mordred: is it concurrency safe? | 00:11 |
| mordred | clarkb: we still do not have a systemic solution for those other than adding them directly to requirements | 00:11 |
| mordred | lifeless: uhm. I'm not sure I've even thought through that | 00:11 |
| clarkb | mordred: ok, I will do that if I can confirm it is a setup requires | 00:11 |
| lifeless | mordred: also I'm thinking the sequence is this: during install of dib, we install run-mirror and setup an initial mirror with jeeypb in it. | 00:11 |
| clarkb | lifeless: no | 00:11 |
| *** nati_ueno has joined #openstack-infra | 00:12 | |
| clarkb | I think they will fight over the same pip cache | 00:12 |
| mordred | well, I guess the question is "is the pip cache concurrency safe" | 00:12 |
| clarkb | which may not be a problem in most cases but if you feed two different inputs into the system one will win | 00:12 |
| lifeless | mordred: then inside the image when we build it we install jeepyb from the mirror, and then update it | 00:12 |
| lifeless | or perhaps we do it from outside. Not sure yet. | 00:12 |
| mordred | lifeless: I'm in favor of both or either of those choices | 00:13 |
| dstufft | I'm almost 100% that the pip cache doesn't do anything to make itself concurrency safe | 00:13 |
| dstufft | if it is safe it's likely safe by accident | 00:13 |
| clarkb | mordred: pycparser is a cffi install_requires | 00:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add hieraedit.py https://review.openstack.org/38003 | 00:14 |
| mordred | clarkb: ooh. that should have been picked up then | 00:15 |
| jeblair | ^ "hieraedit key new-value" | 00:15 |
| mordred | jeblair: nice | 00:15 |
| jeblair | the first run it's going to change everything by getting rid of a bunch of quotes and alphabetizing things. :| | 00:15 |
| jeblair | there was only so much "make it look like what we have now" i could reasonably do. | 00:16 |
| jeblair | (though i did get some things) | 00:16 |
| clarkb | mordred: installing xattr 0.7.1 locally has pycparser showing up in a pip freeze | 00:16 |
| mordred | clarkb: GAH | 00:16 |
| jeblair | also "hieraedit key" will print the value for that key | 00:17 |
| mordred | jeblair: I'm fine with making the file more sensible in that way | 00:17 |
| clarkb | I will try running the mirror script again out of lazyness and being fresh out of ideas | 00:17 |
| jog0 | mordred: any simple way to test that change in devstack? | 00:17 |
| dstufft | mordred: instead of using pip cache you could jsut run a local proxy that hits PyPI and shoves everything it fetches into a directory | 00:17 |
| dstufft | so pip and such would be configured to use th elocal proxy in the mirror build script | 00:18 |
| dstufft | local proxy grabs from PyPI, serves it and stashes in a directory | 00:18 |
| dstufft | tarball directory, you haz a mirror | 00:18 |
| *** nati_ueno has quit IRC | 00:20 | |
| *** cppcabrera has quit IRC | 00:22 | |
| fungi | clarkb: looking back at puppet, seems not to be failing on review.o.o but the latest jeepyb installed by vcsrepo is f98a3a4 which is about a dozen commits behind the tip of master even though the vcsrepo config in jeepyb.pp ensures latest | 00:22 |
| reed | it's beer time | 00:23 |
| reed | bye, see (some of) you in Portland | 00:23 |
| *** reed has quit IRC | 00:23 | |
| mordred | dstufft: tried that. none of the proxy programs are good enough | 00:24 |
| mordred | dstufft: like, I emperically attemped a re-write of our mirror software by using one of the proxies and none of them could do it. I was a sad panda | 00:25 |
| mordred | jog0: it will, in fact, run a devstack-gate test | 00:25 |
| mordred | jog0: because we gate pbr on devstack :) | 00:25 |
| dstufft | mordred: alternative: kill with fire anything that isn't hosted on pypi | 00:25 |
| clarkb | mordred: wsgi-ref and pycparser are not ending up in the pip cache | 00:25 |
| clarkb | they are both cffi requirements | 00:25 |
| *** nati_ueno has joined #openstack-infra | 00:25 | |
| mordred | clarkb: wow. what the hell? | 00:25 |
| clarkb | mordred: fungi: I need to take a break caffeinate/beer. it would be great if someone else could look into this | 00:26 |
| lifeless | are the explicit or transitive? | 00:26 |
| clarkb | mordred: https://jenkins.openstack.org/job/gate-keystone-pep8/6428/console is how it manifests itself | 00:26 |
| mordred | lifeless: transitive | 00:26 |
| jog0 | mordred: nice, I like it when things just do what you would expect | 00:26 |
| clarkb | lifeless: xattr -> cffi -> pycparser/wsgi-ref | 00:26 |
| mordred | lifeless: but run-mirror is designed ot handle transitive depends | 00:26 |
| lifeless | mordred: ok | 00:26 |
| mordred | until today, the only transitive depends we miss are things that are setup_requires and not install_requires | 00:26 |
| clarkb | mordred: I am going to remove xattr 0.7* from our mirror now. Which will allow things to work until the mirror jobs run again | 00:27 |
| mordred | clarkb: +100 | 00:27 |
| lifeless | how/why do setup_requires get missed? | 00:27 |
| *** linuxmohan has joined #openstack-infra | 00:27 | |
| mordred | lifeless: because they don't get installed into the venv | 00:27 |
| jog0 | mordred: lokks like its working so far | 00:27 |
| mordred | lifeless: so we don't have a way of finding them | 00:27 |
| lifeless | mordred: but they do get sucked down right | 00:27 |
| *** dirk has quit IRC | 00:27 | |
| lifeless | ? | 00:27 |
| mordred | yes. easy_install grabs them and sticks an egg file into the temporary build dir | 00:28 |
| mordred | but that dir goes away | 00:28 |
| lifeless | does easy_install support a download cache dir? | 00:28 |
| mordred | we have discussed setting up a passthrough caching pypi mirror that we configure in .pydistutils.cfg | 00:28 |
| lifeless | cause if it did you could tell it to use such a dir | 00:28 |
| mordred | to allow us to grab things that get easy_installed | 00:28 |
| mordred | lifeless: it does not | 00:28 |
| lifeless | suck | 00:28 |
| mordred | lifeless: easy_install is the worst pile of shit | 00:29 |
| mordred | ever infliceted on python | 00:29 |
| mordred | which is why pbr kills it with fire | 00:29 |
| clarkb | mordred: I have removed new xattr from the mirror. I am going to afk for a bit now | 00:29 |
| mordred | clarkb: thank you | 00:29 |
| *** nati_ueno has quit IRC | 00:29 | |
| mordred | lifeless: dstufft and I spoke briefly a while back about trying to figure ohow to get pip to grok that a setup_requires was needed when pip installing things | 00:30 |
| fungi | the caching proxy solution would need some belts and braces too... first, not everything we download via pip actually comes from pypi (some might only be indexed there). also, are we guaranteed that all the download mechanisms in play respect proxy settings? | 00:30 |
| mordred | fungi: we are guaranteed that we can tell easy_install to use the proxy | 00:31 |
| fungi | okay, then there's that at least | 00:31 |
| mordred | and when I tested it, it _does_ do a good job of catching all traffic | 00:31 |
| dstufft | wsgiref ships with Python itself doesn't it? | 00:31 |
| mordred | the main problem is that it can only be a supplimental | 00:31 |
| mordred | because it's not good enough to actualy build our whole mirror | 00:31 |
| dstufft | mordred: why | 00:31 |
| *** linuxmohan has quit IRC | 00:31 | |
| mordred | dstufft: because it breaks | 00:31 |
| mordred | dstufft: each one in different ways | 00:31 |
| dstufft | specifics? | 00:31 |
| dstufft | lol | 00:31 |
| dstufft | fucking packaging | 00:31 |
| mordred | one of them didn't like case insensitivity | 00:32 |
| dstufft | oh wait | 00:32 |
| dstufft | I see what you're saying now | 00:32 |
| mordred | one of them thinks that being clever with async downloading backends is cute | 00:32 |
| dstufft | mordred: did you try devpi? | 00:32 |
| mordred | yes | 00:32 |
| dstufft | IT's borken? | 00:32 |
| mordred | it's the 'clever async' one | 00:32 |
| * dstufft hasn't actually used it | 00:32 | |
| mordred | yup | 00:32 |
| dstufft | ah | 00:32 |
| mordred | it _mostly_ works | 00:32 |
| mordred | except sometimes it returns not found | 00:32 |
| jeblair | jenkins.openstack.org cert replaced | 00:33 |
| mordred | when it shouldn't | 00:33 |
| dstufft | packaging is the worst | 00:33 |
| mordred | yup | 00:33 |
| dstufft | mordred: how much of your stuff you install isn't hosted on pypi | 00:33 |
| mordred | dstufft: no idea | 00:33 |
| mordred | dstufft: I don't think it would help if it all was though | 00:33 |
| mordred | we need to be able to not depend on external network-based resources within a test run | 00:34 |
| dstufft | mordred: if it was all you could implement a caching local index within like a 100 lines of flask app | 00:34 |
| dstufft | it's the requirement to hit other urls that makes that hard | 00:34 |
| mordred | why would it matter? | 00:34 |
| mordred | oh - gotcha | 00:34 |
| mordred | well, that's the other thing | 00:34 |
| mordred | about devpi | 00:35 |
| mordred | it talks to pypi itself | 00:35 |
| mordred | rather than using pip | 00:35 |
| mordred | to talk to pypi | 00:35 |
| mordred | I think it's actually a super simple problem | 00:35 |
| lifeless | mordred: dunno if you saw, but dib now has a --offline mode. | 00:35 |
| mordred | write the passthrough caching proxy and have that thing use pip on the backend to actually fetch the request content | 00:35 |
| mordred | lifeless: SWEET | 00:35 |
| mordred | dstufft: but once you start doing direct curl calls to pypi simple indexes yourself | 00:36 |
| lifeless | mordred: which is not as strict as you'll want | 00:36 |
| dstufft | I wouldn't use pip on the backend tbh | 00:36 |
| mordred | no? | 00:36 |
| mordred | why not? | 00:36 |
| lifeless | mordred: but its a long step in the right direction - its 'use caches and don't refresh them' | 00:36 |
| lifeless | mordred: a) yum, b) apt, c) we haven't implemented the pip cache yet. | 00:36 |
| dstufft | mordred: pip is a mess, it just happens to be less a mess than anything | 00:36 |
| *** prad has joined #openstack-infra | 00:36 | |
| dstufft | but if you restrict yourself to pypi itself | 00:36 |
| dstufft | the logic is fairly simply | 00:36 |
| *** nati_ueno has joined #openstack-infra | 00:37 | |
| mordred | well, I'm pretty sure that won't be possible | 00:37 |
| mordred | and we're REALLY REALLY close to having this all be good | 00:37 |
| dstufft | mordred: That might change when people start getting warnings that their installs are insecure :D | 00:37 |
| mordred | I think our problem right now is 3 level deep transitive depends | 00:37 |
| *** nati_ueno has quit IRC | 00:37 | |
| mordred | xattr -> cffi -> pycparser | 00:38 |
| *** linuxmohan has joined #openstack-infra | 00:38 | |
| mordred | I think pip install xattr winds up with pycparser being pulled by easy_install, yeah? | 00:38 |
| jeblair | fungi, clarkb, mordred: puppet is disabled on review | 00:39 |
| dstufft | uh | 00:39 |
| dstufft | looking | 00:39 |
| jeblair | fungi, clarkb, mordred: who did that and why? | 00:39 |
| mordred | which, if true, might mean that devpi as an easy_install trap might be worth it? | 00:39 |
| *** rwsu is now known as rwsu-away | 00:39 | |
| jeblair | fungi, clarkb, mordred: (istr it's because of the jeepyb thing?) | 00:39 |
| mordred | jeblair: clarkb disabled puppet in places earlier for some reason | 00:39 |
| mordred | clarkb | I have reenabled puppet in all of the places that I had turned it off (mirror slaves, logstash.o.o, and the elasticsearch nodes) | 00:40 |
| dstufft | mordred: pycparser should be installed by pip | 00:40 |
| dstufft | it's an install_requires | 00:40 |
| mordred | dstufft: hrm | 00:40 |
| mordred | dstufft: right. but it's in install_requires of an install_requires | 00:40 |
| dstufft | still pip | 00:40 |
| jeblair | not really pleased with this | 00:40 |
| mordred | ok. well, for some reason it's not showing up. | 00:41 |
| * mordred will dig further | 00:41 | |
| dstufft | pip will download xattr, run egg_info, parse requires.txt, download those (cffi), run egg_info on those, parse requires.txt, download those (pycparser) | 00:41 |
| dstufft | just verified | 00:42 |
| dstufft | Downloading/unpacking pycparser (from cffi>=0.4->xattr) | 00:42 |
| dstufft | Using download cache from /Users/dstufft/.pip/cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpycparser%2Fpycparser-2.09.1.tar.gz | 00:42 |
| dstufft | Running setup.py egg_info for package pycparser | 00:42 |
| fungi | jeblair: i'll hunt down when and why puppet got disabled | 00:42 |
| mordred | dstufft: 2013-07-19 23:06:51.083 | Running pycparser-2.09.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-dSAMlW/pycparser-2.09.1/egg-dist-tmp-_T3qkP | 00:42 |
| dstufft | wat | 00:42 |
| mordred | dstufft: https://jenkins.openstack.org/job/periodic-mirror-python27/142/consoleFull | 00:42 |
| mordred | dstufft: look at log - search for pycparser | 00:42 |
| *** linuxmohan has quit IRC | 00:42 | |
| dstufft | https://gist.github.com/dstufft/6043323 | 00:42 |
| dstufft | looking | 00:42 |
| mordred | dstufft: your gist is what I expect to happen | 00:43 |
| jeblair | Jul 20 00:38:46 review puppet-agent[1004]: Finished catalog run in 21.64 seconds | 00:43 |
| jeblair | Jul 20 00:38:49 review puppet-agent[24964]: Skipping run of Puppet configuration client; administratively disabled; use 'puppet Puppet configuration client --enable' to re-enable. | 00:43 |
| jeblair | fungi: ^ those 2 log lines really don't make any sense | 00:43 |
| jeblair | fungi: (the second is from my cmdline invocation) | 00:43 |
| jeblair | fungi: the first is presumably the running puppet agent which claims to be doing things | 00:43 |
| dstufft | mordred: is the script that this job runs available anywhere | 00:44 |
| *** koudaddy1981 has joined #openstack-infra | 00:44 | |
| mordred | mordred@mirror27:~$ pip --version | 00:44 |
| mordred | pip 1.0 from /usr/lib/python2.7/dist-packages (python 2.7) | 00:44 |
| mordred | perhaps we should upgrade pip on our mirror builder slaves? | 00:44 |
| dstufft | hum | 00:44 |
| dstufft | perhaps | 00:44 |
| *** sarob has joined #openstack-infra | 00:44 | |
| dstufft | pip 1.0 is pretty old | 00:44 |
| dstufft | it's also vulnerable to MITM | 00:44 |
| mordred | dstufft: yes. https://github.com/openstack-infra/jeepyb/blob/master/jeepyb/cmd/run_mirror.py | 00:45 |
| mordred | jeblair: when you have a moment (puppet on review is more important) I think we should upgrade pip on mirror building slaves, but want to get your take | 00:45 |
| fungi | jeblair: it doesn't look (from the agent reports on dashboard) like puppet is or was disabled on review.o.o any time recently | 00:46 |
| dstufft | mordred: -M is slow and insecure too btw | 00:46 |
| jeblair | mordred: do we have a choice? | 00:47 |
| jeblair | fungi: i will --enable and move on. | 00:47 |
| mordred | jeblair: I'm going to go with "no" - also because of the security aspect | 00:48 |
| mordred | jeblair: since pip 1.0 does not use ssl | 00:48 |
| jeblair | mordred: shouldn't there be an os package update? | 00:48 |
| mordred | jeblair: not on precise | 00:48 |
| jeblair | mordred: it's not secure enough for us to use, but it's not insecure enough for them to push an update? | 00:48 |
| *** prad is now known as prad-afk | 00:49 | |
| fungi | jeblair: ahh, it's puppet configuration client and not puppet agent that log says is disabled... i wonder what the configuration client is? looking it up... | 00:49 |
| dstufft | jeblair: well fixing it would require them to upgrade from 1.0 to 1.3 | 00:49 |
| dstufft | Not sure if ubuntu is willing to do that to an already released version? | 00:49 |
| mordred | which is against policy | 00:49 |
| mordred | no, I believe they would consider this "working as designed when precise was released" | 00:50 |
| jeblair | mordred: so basically, we decided, hey we want to stop pip installing things globally via puppet and only use packages for that, including packages from the cloud archive | 00:50 |
| dstufft | maybe I should figure out how to make a ppa of pip | 00:50 |
| jeblair | mordred: so if we could install that with dpkg, that'd be great | 00:50 |
| mordred | jeblair: how about this... | 00:50 |
| mordred | jeblair: how about we have run_mirror upgrade pip in the virtualenv it uses to do the work | 00:51 |
| fungi | jeblair: apparently puppet sometimes erroneously claims that... http://projects.puppetlabs.com/issues/13299 (i suspect there's more than one way to get it to think that's the case) | 00:51 |
| mordred | jeblair: and use the pip from that virtualenv to do its work | 00:51 |
| mordred | rather than the system pip | 00:51 |
| mordred | since we're going to want it to use pip 1.4 when that comes out anyway | 00:51 |
| jeblair | mordred: well, that doesn't address the security aspect. | 00:51 |
| jeblair | mordred: i suspect i have missed the actual problem you're trying to solve. :) | 00:52 |
| mordred | jeblair: there are a couple | 00:52 |
| jeblair | mordred: fyi, i have not been paying attention. | 00:52 |
| mordred | jeblair: I know. I was tyring to tl;dr you - so I did not present a full thing | 00:52 |
| jeblair | roger | 00:52 |
| jeblair | review.openstack.org cert replaced | 00:53 |
| jeblair | fungi, clarkb, mordred: puppet agent --test reported no other changes, so i think the disable was benign. | 00:54 |
| mordred | jeblair: good | 00:54 |
| fungi | jeblair: yeah, it seems like it's sometimes confused. i also see reports a stale lockfile or second/hung concurrent agent run can cause that message | 00:55 |
| jeblair | mordred: i recall that at one point we had slave machines that had pip installed pip and they were completely hosed | 00:55 |
| * clarkb catches up on scrollback | 00:55 | |
| mordred | jeblair: the _main_ think is that pip 1.0 is not doing transitive depends properly so we're winding up with 2-deep transitive depends being handled by easy_install and thus not ending up in the mirror | 00:55 |
| mordred | jeblair: newer pip behaves properly | 00:55 |
| jeblair | mordred: that makes upgrading in a venv for the mirror script sound like a reasonable solution | 00:56 |
| clarkb | jeblair: they don't get completely hosed. what happend was pip provider in puppet ensured latest and it caused puppet to report it did things on every run | 00:56 |
| mordred | jeblair: we also will want a a strategy for getting new pip/virtualenv/tox when they come out, because we'll need them to do pre-release filtering and wheel support | 00:56 |
| clarkb | because puppet uses pip freeze to see what it needs to install and pip does not report itself in a pip freeze | 00:56 |
| mordred | but we don't need to solve that right now | 00:56 |
| clarkb | I have filed a bug upstream for this | 00:56 |
| dstufft | mordred: would a PPA be useful for you guys in that realm? | 00:56 |
| jeblair | clarkb: istr that oneiric hosts were actually hosed too because pip reported incorrect versions for other things as well | 00:56 |
| mordred | dstufft: possibly? | 00:57 |
| jeblair | mordred, dstufft: .debs are ideal for that | 00:57 |
| dstufft | Been thinking about fucking around with getting a ppa that tracks pip stable | 00:57 |
| dstufft | haven't gotten around to it | 00:57 |
| jeblair | mordred: i believe at that point those are good candidates for the cloud archive tho | 00:57 |
| mordred | dstufft: ++ | 00:57 |
| jeblair | mordred: and don't forget rpms. :) | 00:57 |
| mordred | jeblair: /me cries | 00:58 |
| dstufft | if it fixes things for you guys I can priotiize it higher | 00:58 |
| mordred | dstufft: I think we've got a temp fix for this we can do with venvs for now ... so don't go out of your way | 00:58 |
| jeblair | mordred: cloud archive because that'll be a priority for doing openstack development (it'll be beyond just infra at that point) | 00:58 |
| mordred | zul: we need you to put things in cloud archive | 00:58 |
| clarkb | I did not disable puppet on review. I had puppet disabled on logstash to fix the log swap stuff, on the mirror slaves because I manually applied the libffi-dev stuff and on elasticsearch bceause I needed to get my emergency changes into puppet, but I have enabled it everywhere I turned it off before the weekend | 00:58 |
| jeblair | clarkb: yeah, fungi said it's possibly a puppet error | 00:59 |
| *** nati_ueno has joined #openstack-infra | 01:02 | |
| clarkb | mordred: jeblair fungi do we have a fix then? is it to use virtualenv? | 01:02 |
| pabelanger | So, I was checking out pbr and sphinx when I noticed the follow with nova-docs | 01:02 |
| pabelanger | https://jenkins.openstack.org/view/All/job/nova-docs/7557/consoleText | 01:02 |
| fungi | i see no actual signs it is or was disabled, and at least a couple bugs about that message appearing erroneously | 01:03 |
| pabelanger | there is a ton of warnings, but looks like the job still passes | 01:03 |
| mordred | clarkb: yeah. I'm making a jeepyb patch righ tnow | 01:03 |
| pabelanger | something not getting parsed right I guess | 01:03 |
| clarkb | mordred: why not patch jjb configs? | 01:03 |
| mordred | clarkb: what? | 01:03 |
| jeblair | wiki.openstack.org cert replaced | 01:03 |
| clarkb | mordred: do the venv stuff in the jenkins job | 01:04 |
| fungi | clarkb: but back to your comment about jeepyb on review.o.o being stale, ensure=>latest in vcsrepo should track the tip of master, right? it's not (a dozen commits behind now), and puppet's not erroring | 01:04 |
| clarkb | fungi: it should | 01:04 |
| fungi | oh! wait | 01:05 |
| mordred | clarkb: the pip in jeepyb for the mirror is part of the venv that's used | 01:05 |
| mordred | clarkb: jenkins jobs do not have enough context | 01:05 |
| *** sarob has quit IRC | 01:05 | |
| fungi | it's up to date in /opt, but its pip-installed system-wide on review.o.o from about a dozen commits ago | 01:05 |
| clarkb | mordred: ok, I await the patch | 01:05 |
| fungi | i guess vcsrepo doesn't do a 'pip install /opt/jeepyb' when it updates | 01:06 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Upgrade pip in the mirror building venv https://review.openstack.org/38004 | 01:06 |
| mordred | clarkb: hrm. I'm lying. the pip in the virtualenv on mirror27 is 1.3.1 already | 01:07 |
| mordred | dammit | 01:08 |
| mordred | dstufft: ^^ | 01:08 |
| dstufft | mordred: :[ | 01:08 |
| mordred | the behavior in that log makes no sense! | 01:08 |
| jeblair | etherpad.openstack.org cert replaced | 01:08 |
| jeblair | clarkb, fungi, mordred, pleia2: ssl cert for git.o.o added to hiera under the usual pattern | 01:08 |
| dstufft | mordred: I'm on a phone call right now, can you get me the exact command pip runs? and what repo i'd need to do to run it locally? | 01:09 |
| fungi | jeblair: awesome. we'll make sure it gets integrated | 01:09 |
| mordred | dstufft: yes. I will get you infos | 01:09 |
| fungi | clarkb: out of curiosity, how does the install of jeepyb pip freeze shows (which is not a version released to pypi). currently reported as jeepyb==0.0.122.f98a3a4 which is not as new as what's in /opt/jeepyb | 01:10 |
| mordred | I KNOW WHAT IT IS | 01:10 |
| clarkb | \o/ | 01:10 |
| mordred | it's not fixable | 01:10 |
| fungi | mordred: the glorious eureka moment | 01:10 |
| clarkb | NNNOOOOO | 01:10 |
| fungi | why is it not fixable? | 01:11 |
| dstufft | mordred: wat is it | 01:11 |
| mordred | it's the local egg thing | 01:11 |
| mordred | setup_requires triggers easy_install to grab the stuff | 01:11 |
| mordred | which puts eggs in the current dir | 01:11 |
| mordred | which causes pip to think that the software is there | 01:11 |
| dstufft | Yes, but pip shouldn't look at that else it would break for the simple test case of ``pip install xattr`` | 01:11 |
| *** bknudson has quit IRC | 01:12 | |
| mordred | hrm | 01:12 |
| mordred | the thing is - this: Writing /tmp/easy_install-CITY8u/pycparser-2.09.1/setup.cfg | 01:13 |
| pabelanger | Oooh, I see. You need to add [pbr]\nwarnerrors = True\n to get sphinx to error on warnings | 01:13 |
| mordred | is happening in the Running setup.py egg_info for package xattr block | 01:13 |
| mordred | which leads to Installed /tmp/tmpRL4RxH/build/xattr/pycparser-2.09.1-py2.7.egg | 01:13 |
| dstufft | mordred: it should install it into the buold directory | 01:14 |
| dstufft | whichi makes it available to setup.py | 01:14 |
| dstufft | but not to pip | 01:14 |
| dstufft | https://gist.github.com/dstufft/6043323#file-gistfile1-txt-L36 | 01:14 |
| dstufft | because the build directory isn't on sys.path for pip, but pip subprocesses out to setup.py, so it is on the sys.path for that | 01:15 |
| mordred | yup. for some reason this is not the behavior that I'm seeing in the pip.log | 01:15 |
| dstufft | mordred: why does this install everything twice | 01:18 |
| dstufft | oh I see | 01:18 |
| dstufft | nevermind | 01:18 |
| dstufft | acutally no I don't see | 01:18 |
| dstufft | mordred: Downloading/unpacking xattr>=0.4 (from -r /tmp/tmpoO19Px (line 67)) | 01:19 |
| dstufft | do we know what the value of /tmp/tmpoO19Px is | 01:19 |
| dstufft | uh | 01:21 |
| dstufft | mordred: | 01:22 |
| dstufft | 2013-07-19 23:13:12.486 | Bad md5 hash for package https://pypi.python.org/packages/source/c/colorama/colorama-0.2.5.zip#md5=db7356055c2ed4f5fe9f386490943f20 (from https://pypi.python.org/simple/colorama/) | 01:22 |
| dstufft | 2013-07-19 23:13:12.486 | Storing complete log in /home/jenkins/.pip/pip.log | 01:22 |
| dstufft | looks like pip exited in the middle of the build? | 01:22 |
| dstufft | but the miror script didn't stop | 01:22 |
| *** cppcabrera has joined #openstack-infra | 01:28 | |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/zuul: Enable warnerrors for pbr to catch Sphinx warnings https://review.openstack.org/38006 | 01:29 |
| mordred | dstufft: that's correct. we try to grab as much as we can, because we're talking to the internet and something might bork mid-way through | 01:29 |
| mordred | that may not be a great idea though | 01:29 |
| dstufft | mordred: it failed on coloroma on every run it looks like | 01:30 |
| dstufft | so it might not be that pycparser wasn't installed | 01:30 |
| dstufft | pip might just not have gotten to it yet | 01:30 |
| mordred | because of that reason | 01:30 |
| mordred | yes | 01:30 |
| dstufft | before it bailed out | 01:30 |
| mordred | so - I have a patch that should address that | 01:30 |
| mordred | https://review.openstack.org/#/c/34146/ | 01:30 |
| mordred | which, instead of installing the whole file | 01:30 |
| mordred | does a per-line pip call | 01:31 |
| mordred | which means each depend should succeed or fail | 01:31 |
| dstufft | that will have a different behavior that installing the whole file fwiw | 01:31 |
| dstufft | you might end up with different versions of things installed | 01:31 |
| dstufft | I think | 01:31 |
| dstufft | then again pip has a horrible dep resolver | 01:31 |
| dstufft | so maybe not | 01:31 |
| mordred | the "package in middle of file fails so we lose everything else" seems like a potentially greater (and currently manifesting) problem | 01:32 |
| dstufft | is there more to this log then is shown here | 01:33 |
| dstufft | 2013-07-19 23:03:30.094 | Started by user Clark Boylan | 01:34 |
| dstufft | 2013-07-19 23:03:30.095 | [EnvInject] - Loading node environment variables. | 01:34 |
| dstufft | 2013-07-19 23:03:30.184 | Building remotely on mirror27 in workspace /home/jenkins/workspace/periodic-mirror-python27 | 01:34 |
| dstufft | 2013-07-19 23:03:30.219 | [periodic-mirror-python27] $ /bin/bash -xe /tmp/hudson5451042873003930760.sh | 01:34 |
| dstufft | 2013-07-19 23:03:30.229 | + /usr/local/bin/run-mirror -c /home/jenkins/pypimirror/etc/pypi-mirror.yaml | 01:34 |
| dstufft | 2013-07-19 23:06:51.044 | Installing pip requires for https://github.com/openstack/requirements:remotes/origin/master failed. | 01:34 |
| dstufft | shouldn't there have been output | 01:34 |
| dstufft | before that fail | 01:34 |
| dstufft | mordred: I ask becasue every attempted install of colorrama that is listed in the log was pulling it from the download cache | 01:35 |
| dstufft | so obviously whatever populated that download cache put something bad in there | 01:35 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Use precisepy3k instead of precise3k. https://review.openstack.org/38007 | 01:35 |
| dstufft | and then pip, being the stupid git it is, just blindly used it and failed instead of getting it from the soruce | 01:35 |
| dstufft | oh hey | 01:37 |
| dstufft | https://github.com/pypa/pip/pull/968 | 01:37 |
| *** sandywalsh has quit IRC | 01:37 | |
| dstufft | looks like another pip 1.4 fix :V | 01:37 |
| jeblair | #status log replaced ssl certs for jenkins, review, wiki, and etherpad | 01:41 |
| clarkb | jeblair: woot | 01:42 |
| clarkb | mordred: so, should we pin xattr for now or are we actually going to fix this? | 01:42 |
| clarkb | mordred: or, and I don't care for this option too much, do we want to manually update the mirror? | 01:42 |
| *** nati_ueno has quit IRC | 01:44 | |
| mordred | clarkb: well... it seems the real problem is not related to xattr | 01:49 |
| *** koobs has quit IRC | 01:49 | |
| *** koobs has joined #openstack-infra | 01:49 | |
| mordred | clarkb: I think at the VERY least we should delete the current colorrama from the pip cache | 01:49 |
| dstufft | mordred: does that cache get cleaned between builds | 01:49 |
| dstufft | or does it stick around | 01:49 |
| mordred | dstufft: no. it does not | 01:49 |
| mordred | it sticks around | 01:49 |
| clarkb | it sticks around so that it doesn't take an hour each time | 01:50 |
| clarkb | I am not currently on a computer with ssh keys into stuff, but want to make sure this doesn't get lost over the weekend while everyone is traveling and/or in dufur middle of nowhere oregon | 01:51 |
| mordred | clarkb: I'm going to delete colorama | 01:51 |
| mordred | and re-run | 01:51 |
| dstufft | mordred: pip 1.4 will make this case better then | 01:51 |
| dstufft | not sure why it would have a bad cache though | 01:51 |
| mordred | dstufft: 1.4 will solve everything | 01:52 |
| dstufft | looks like the last change to colorama was 6-15 | 01:52 |
| mordred | dstufft: we have noticed from time to time that we get bad md5 things | 01:52 |
| mordred | it's possible this has been a problem since then | 01:52 |
| mordred | it's one of the reasons I wrote the above back to jeepyb | 01:52 |
| dstufft | mordred: happens with cached downloads when someone deletes a file and reuploads a different one | 01:52 |
| mordred | patch | 01:52 |
| mordred | not back | 01:52 |
| mordred | dstufft: which should be completely disallowed | 01:52 |
| dstufft | I agree | 01:52 |
| dstufft | that's on my list to fight for again | 01:52 |
| dstufft | I tried to argue that a year and a half ago | 01:53 |
| dstufft | almost 2 years now | 01:53 |
| clarkb | dstufft: I will sit in the trenches with you if you need more bodies | 01:53 |
| clarkb | because bad bad bad | 01:53 |
| mordred | dstufft: people argue AGAINST you? | 01:53 |
| dstufft | and a lot of shit was throw "well they can do X or Y instead if they really want too" | 01:53 |
| dstufft | so i'm systematically getting rid of those X and Y's | 01:53 |
| dstufft | :D | 01:53 |
| mordred | dstufft: have any of these people ever used computers? | 01:53 |
| mordred | :)rant rant rant | 01:53 |
| dstufft | mordred: well to give an example, one of the people arguging against me was PJE, creator of setuptools/easy_install | 01:54 |
| dstufft | he's on the opposing side of me a lot | 01:54 |
| dstufft | :[ | 01:54 |
| fungi | but... people are sloppy and make mistakes, and too lazy or embarrassed to make a new release/version bump. can't everyone cater to their whims? | 01:54 |
| mordred | I cannot imagine anyone opposing restricting the re-uplloading of a new thing with the same version | 01:54 |
| mordred | I have re-kicked mirror-python27 ... | 01:55 |
| mordred | https://jenkins.openstack.org/job/periodic-mirror-python27/144/console | 01:55 |
| mordred | in case anyone wants to follow along at home | 01:55 |
| pleia2 | o/ | 01:55 |
| dstufft | mordred: https://gist.github.com/dstufft/409306ae13c6f076719d | 01:55 |
| clarkb | pleia2: you have landed? | 01:55 |
| mordred | dstufft: stab stab stab | 01:55 |
| pleia2 | clarkb: checked in at hotel | 01:55 |
| fungi | pleia2: safe of mind and body? | 01:55 |
| clarkb | pleia2: are you in portland? | 01:55 |
| pleia2 | it's summer here :) | 01:55 |
| pleia2 | clarkb: yep | 01:55 |
| dstufft | fungi: literally one of the arguments against me was "sometimes we make mistakes and we can't just increment the version becasue we have to write blog posts and stuff for new versions" | 01:56 |
| dstufft | which my response was | 01:56 |
| clarkb | pleia2: it has been perfect in the NW the last week or so | 01:56 |
| dstufft | test your fucking shit before you upload it then | 01:56 |
| mordred | "get better automation" | 01:56 |
| *** zaro0508 has quit IRC | 01:56 | |
| fungi | dstufft: my response would have been *thppphbbbht* (a la bill the cat) | 01:56 |
| dstufft | version numbers are not a limited resource :V | 01:56 |
| clarkb | pleia2: I can recommend cascade barrel house for beering | 01:56 |
| * fungi makes more numbers | 01:56 | |
| pleia2 | clarkb: cool, thanks | 01:56 |
| dstufft | they are not hand crafted by orphan children in a diamond mine | 01:56 |
| clarkb | or hair of the dog, or breakside or brass horse or pints or baileys | 01:56 |
| clarkb | ^ makes me miss portland | 01:57 |
| mordred | soon we can stop mirroring colorama | 01:57 |
| *** sandywalsh has joined #openstack-infra | 01:57 | |
| fungi | pleia2: what day is your preso? don't want to miss it | 01:57 |
| mordred | because we can get rid of openstack.nose_plugin | 01:57 |
| mordred | once we stop using it | 01:57 |
| clarkb | mordred: and nosehtmloutput :) | 01:57 |
| mordred | yup | 01:57 |
| mordred | and nose | 01:57 |
| pleia2 | fungi: thursday at 11:30 | 01:57 |
| fungi | we can finally cut off our nose to spite our face? | 01:58 |
| mordred | fungi: YES | 01:58 |
| dstufft | mordred: but yea, that gist I pasted all happened on 6-15-2013 so it's likely your mirror script has been broken since then | 01:58 |
| fungi | pleia2: with bells on. adding it to my personal schedule | 01:58 |
| mordred | dstufft: yup. I believe you are right | 01:58 |
| clarkb | pleia2: pints is really cool because their fermentor and boiler are in the same room | 01:58 |
| mordred | dstufft: and I would love to land my patch - which would at least not have broken colorama kill the rest of the mirroring | 01:58 |
| * mordred looks at clarkb and fungi | 01:58 | |
| pleia2 | clarkb: ooh, fun! | 01:58 |
| clarkb | pleia2: baileys has ~20 rotating taps and a json api | 01:59 |
| clarkb | hair of the dog has aged beers | 01:59 |
| fungi | mordred: i think we were just looking for some rough timing comparisons (i need to go back over it again and see if i missed an update) | 01:59 |
| clarkb | breakside is just good beer. brass horse is old school pub. and cascade does sour beer. So many choices | 01:59 |
| mordred | fungi: yeah. sorry, I should have given you some timing on that | 01:59 |
| dstufft | mordred: I have no control over that :) It'd worry me you'll get bugs based on not actually installing a set of requirements and instead installing X number of individual requirement sets, so if different projects have different overlapping requirements you might end up with a busted mirror. But that's a personal choice which way you guys would rather guy | 02:00 |
| dstufft | go | 02:00 |
| pleia2 | clarkb: ended up staying just south of pioneer square, so tonight I might search around here | 02:01 |
| dstufft | mordred: you might want some sort of notification when the pip install fails if you continue to continue on in he mirror script regardless | 02:01 |
| dstufft | to prevent issues like this from going undetected for a month+ | 02:01 |
| clarkb | pleia2: baileys is what you want then | 02:01 |
| pleia2 | clarkb: ok cool | 02:01 |
| clarkb | pleia2: it is nearby and awesome. Will probably be busy on a friday night but they have board games and did I mention a json api? | 02:01 |
| pleia2 | hehe | 02:02 |
| clarkb | pleia2: http://www.baileystaproom.com/ | 02:02 |
| dstufft | hm | 02:02 |
| dstufft | guess openstack/requirements doesn't install on OSX | 02:02 |
| clarkb | pleia2: they have vertigo raspberry wheat! that is brewed just down the street from my brother's house. perfect summer beer. | 02:03 |
| pleia2 | nice | 02:03 |
| clarkb | dstufft: mordred: you could setup the jenkins job to fail and send email when one package does not install, but still rsync the packages that were pulled in | 02:04 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr https://review.openstack.org/38009 | 02:06 |
| clarkb | mordred: so before I AFK for the night, what needs doing? are we going to push your run_mirror change through, pin xattr or? | 02:08 |
| mordred | clarkb: GAH! | 02:10 |
| fungi | clarkb: if that run completes, then it was just a corrupt package lingering in the cache | 02:10 |
| fungi | i guess it didn't work? | 02:10 |
| clarkb | oh is that still going? | 02:10 |
| clarkb | oh it seems to have grabbed pycparser but not web* | 02:11 |
| mordred | so, that was the first problem | 02:11 |
| mordred | NOW we have a problem with a setuptools upgrade problem | 02:11 |
| clarkb | wow | 02:11 |
| dstufft | probably need to run pip install -U setuptools in the venv | 02:12 |
| mordred | I mean, I don't want to slam in code at 7pm on a friday- but there is a functional reason I wrote that patch :) | 02:12 |
| mordred | and it's because our mirror hasn't been working in a month or so - I just keep forgetting | 02:12 |
| mordred | lemme make a patch to do the setuptools upgrade step | 02:13 |
| clarkb | I'm all for fixing the mirror but I thought there were some genuine concerns that were not addressed | 02:13 |
| mordred | questions on run time | 02:13 |
| mordred | which, to be fair, I have not responded to | 02:13 |
| fungi | i'll be around to review a setuptools upgrade change | 02:14 |
| * mordred not trying to passive aggressive the patch in - just hitting the point where tracking down intermediary problems where we know there is an existing logic error ... | 02:14 | |
| dstufft | I'll be around too but that doesn't win you anything :) | 02:14 |
| mordred | lemme just run the patch and get timing | 02:14 |
| dstufft | hm | 02:14 |
| dstufft | looks like less things then I thought require external links | 02:15 |
| dstufft | from openstack/requirements/requirements.txt | 02:15 |
| *** koudaddy1981 has quit IRC | 02:16 | |
| mordred | dstufft: oh yeah? | 02:17 |
| mordred | dstufft: did you make a list? | 02:17 |
| dstufft | mordred: yea sec the install is still running | 02:17 |
| dstufft | pip 1.4 lets you pass --no-allow-external | 02:17 |
| dstufft | which disables external by default | 02:17 |
| dstufft | (which is the befault behavior in 1.5) | 02:18 |
| dstufft | waiting on things to compile zzzzzz | 02:18 |
| dstufft | probably shoudln't have ran this on a micro instance | 02:18 |
| mordred | haha | 02:18 |
| dstufft | Using the requirement list I stole from openstacl/requirements/requirements.txt (https://gist.github.com/dstufft/6043583) this is what is required to successfully install it with --no-allow-external -- pip install -r req.txt --no-allow-external --allow-external dnspython --allow-external lockfile --allow-external netifaces --allow-external psutil | 02:20 |
| dstufft | I would bet those are all completely unverified downloads too, but I can tell you that too! | 02:20 |
| *** pcrews has joined #openstack-infra | 02:22 | |
| dstufft | so yea | 02:24 |
| dstufft | looks like they are all insecure downloads too | 02:24 |
| dstufft | so anyone installing that set right now is vulnerable to code injection via MITM :V | 02:24 |
| dstufft | mordred: just for completeness sake, here's the command it would take to install openstack/requirements/requirements.txt in pip 1.5 (where these default to disallowed) pip install -r req.txt --allow-external dnspython --allow-external lockfile --allow-external netifaces --allow-external psutil --allow-insecure dnspython --allow-insecure lockfile --allow-insecure netifaces --allow-insecure psutil | 02:26 |
| dstufft | Although you could cheat and replace all of the --allow-external with a single --allow-all-external | 02:26 |
| mordred | dstufft: oh wow | 02:26 |
| mordred | so there's really only 4 bad apples | 02:26 |
| dstufft | yup | 02:26 |
| dstufft | for openstack/requirements/requirements.txt | 02:26 |
| mordred | I wonder if we really need all 4 of them | 02:26 |
| dstufft | I didn't look at the test reuqirements or anything | 02:26 |
| *** cppcabrera has quit IRC | 02:27 | |
| dstufft | but all I did was install pip1.4rc5 and run pip install --no-allow-external to turn off scraping of external links and ried to install, and as it ailed each time I added another --allow-external | 02:28 |
| * Alex_Gaynor kind of wants to propose getting rid of netifaces | 02:28 | |
| dstufft | mordred: at the very least you might want to disallow *new* bad apples, and once pip 1.4 is released switch to a command like that where you have to explicitly whitelist packages that hit external urls | 02:28 |
| clarkb | mordred: your jeepyb change has my +2. feel free to approve if youthink it will improve the situation | 02:31 |
| mordred | clarkb: thanks. I'm running it in a timing test right now | 02:31 |
| mordred | dstufft: ++ | 02:31 |
| mordred | also, dnspython is a pile of turdsoup as well | 02:31 |
| mordred | so maybe we should consider being in that list a knock against something | 02:31 |
| dstufft | mordred: test-requirements.txt and dev-requirements.txt are the only other requirements lists? | 02:32 |
| dstufft | I'll do the same thing against them too | 02:32 |
| mordred | dstufft: yes | 02:32 |
| dstufft | ok | 02:32 |
| dstufft | gimme a few and i'll see if there's anything external only there | 02:32 |
| mordred | dstufft: dev-requirements.txt is just on ours | 02:32 |
| dstufft | ah | 02:32 |
| mordred | dstufft: so test-requirements.txt would be the only thing to check | 02:32 |
| dstufft | ok i'll check that then | 02:32 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Install python-sphinx on slave nodes. https://review.openstack.org/38011 | 02:33 |
| mordred | fungi: ^^ whyfore? | 02:33 |
| mordred | oy | 02:34 |
| clarkb | fungi: why would we be using a system installed jinja2? | 02:34 |
| clarkb | is that salt? | 02:34 |
| mordred | actually - we added -U to the tox.ini for nova | 02:34 |
| mordred | so that shouldn't be a problem anymore | 02:34 |
| clarkb | yeah there shouldn't be interference between system and virtualenv packages | 02:35 |
| fungi | clarkb: nova | 02:36 |
| clarkb | fungi: I think we fixed nova | 02:36 |
| fungi | clarkb: oh? | 02:36 |
| clarkb | we might need to backport that change to folsom and grizzly | 02:36 |
| fungi | nova no longer uses site packages? | 02:37 |
| fungi | i missed that somewhere in scrollback mountain i guess | 02:37 |
| clarkb | fungi: it does but we added a -U to the install line so pip gives nova the version it wants | 02:37 |
| fungi | ahh, so maybe they can just uncap requests and go on their merry way | 02:37 |
| clarkb | I think so | 02:37 |
| fungi | i'll let them know | 02:37 |
| clarkb | fungi: you should be able to test simply by proposing that change yourself | 02:38 |
| *** linuxmohan has joined #openstack-infra | 02:38 | |
| dstufft | mordred: pip install -r test-reqs.txt --no-allow-external --no-allow-insecure --allow-external pysendfile --allow-insecure pysendfile | 02:42 |
| dstufft | looks like that's all that's required | 02:42 |
| *** linuxmohan has quit IRC | 02:42 | |
| fungi | clarkb: that was the plan, in fact | 02:43 |
| fungi | i keep forgetting how fast nova moves... 565 commits to master since my last pull | 02:44 |
| *** pcrews has quit IRC | 02:45 | |
| *** koobs has quit IRC | 02:45 | |
| clarkb | fungi: how long ago was that? | 02:46 |
| fungi | clarkb: not sure. i guess i don't pull nova too often | 02:46 |
| dstufft | mordred: ok confirmed requirements.txt = dnspython,lockfile,netifaces,psutil test-requirements.txt = pysendfile are the externally hosted ones, they will give users warnings in pip 1.4 and will not install by default in 1.5 | 02:48 |
| *** linuxmohan has joined #openstack-infra | 02:48 | |
| dstufft | so unless you guys want broken by default for end users via pip 1.5 you'll probably want to either pester authors to upload, upload yourself under diff names, or replace | 02:49 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Switch to flake8+hacking https://review.openstack.org/38013 | 02:50 |
| *** linuxmohan has quit IRC | 02:52 | |
| fungi | clarkb: 565 commits in 18 days (that was the last time i pulled, based on reflog) | 02:54 |
| mordred | real33m38.729s | 02:54 |
| mordred | why do we need pysendfile for _test_ requirements? | 02:55 |
| fungi | mordred: that's a run of the new or old algorithm? | 02:55 |
| mordred | fungi, clarkb: script runs in 33 m with new algorithm | 02:55 |
| mordred | and does get pycparser==2.09.1 | 02:55 |
| mordred | in the freeze output | 02:55 |
| fungi | mordred: on your machine or on a mirrorXX.slave.o.o? | 02:55 |
| mordred | fungi: on mirror27.slave.o.o running as the jenkins user | 02:56 |
| mordred | but not in the normal workspace (I made a venv with that jeepyb patch) | 02:56 |
| fungi | okay, so roughly twice as long. not too bad in my opinion | 02:56 |
| mordred | no - not to get functionality where it was broke before | 02:56 |
| mordred | lemme paste the freeze output | 02:56 |
| mordred | http://paste.openstack.org/show/41007/ | 02:57 |
| fungi | oh! has it been running in ~15 minutes because it was breaking partway through? i didn't think about that | 02:57 |
| mordred | ah! yes in deed | 02:57 |
| mordred | I say let's land the chance | 02:58 |
| mordred | change | 02:58 |
| mordred | in fact | 02:58 |
| mordred | I just +1 APRV'd it | 02:58 |
| fungi | i had +2'd it days ago anyway. wfm | 02:58 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Be more agressive trying to install requirements. https://review.openstack.org/34146 | 02:59 |
| fungi | according to russellb's stats script i give away +2 votes like hallowe'en candy | 02:59 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Switch to flake8+hacking https://review.openstack.org/38013 | 02:59 |
| dstufft | mordred: you know about --download right? | 03:00 |
| mordred | dstufft: oh yeah. we use the crap out of it | 03:01 |
| mordred | dstufft: if you want to read evil at some point - you should read the pbr integration test :) | 03:01 |
| mordred | dstufft: https://github.com/openstack-dev/pbr/blob/master/tools/integration.sh | 03:02 |
| mordred | dstufft: also, I've been brainstorming ways in which we can run integration tests with your tip of master | 03:02 |
| mordred | dstufft: and report back to you if something breaks | 03:02 |
| dstufft | of pip? | 03:02 |
| mordred | dstufft: yeah | 03:02 |
| dstufft | that'd be cool, though it'd be develop | 03:02 |
| mordred | sure | 03:02 |
| mordred | whichever thing is the thing it should be | 03:02 |
| mordred | all of my ideas so far are bad | 03:03 |
| mordred | but I'm working on it | 03:03 |
| dstufft | that'd be really awesome. Openstack probably stresses pip far more than most folks do | 03:03 |
| dstufft | mordred: btw | 03:04 |
| dstufft | switching to pip install --no-allow-external --no-allow-insecure with those 4 things white listed cuts the download time in almost half | 03:05 |
| mordred | dstufft: just sent a message to our dev list with the 5 evil external folks | 03:05 |
| mordred | dstufft: AWESOME | 03:05 |
| dstufft | whitelisted only externals = 1:36.52 total | 03:05 |
| dstufft | all externals = 2:18.14 total | 03:05 |
| * Alex_Gaynor isn't sure dstufft knows what half is | 03:06 | |
| dstufft | Alex_Gaynor: lol if you think I looked at anything but the first number | 03:06 |
| clarkb | Alex_Gaynor: maths are hard :) | 03:07 |
| dstufft | there's a half of something in there | 03:08 |
| dstufft | somewhere | 03:08 |
| dstufft | :{ | 03:08 |
| dstufft | hm | 03:10 |
| dstufft | openstack-dev gets a lot of traffic | 03:10 |
| dstufft | not sure my inbox can handle more | 03:10 |
| mordred | it seems that some people actually read it all | 03:10 |
| mordred | which is A-MAZING | 03:10 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Switch to flake8+hacking https://review.openstack.org/38013 | 03:10 |
| clarkb | mordred: what? how is that possible? | 03:11 |
| dstufft | mordred: I read every messages that comes into my inbox atm, which is ~300 a day or so | 03:11 |
| *** pcrews has joined #openstack-infra | 03:11 | |
| clarkb | dstufft: I gave up reading it all long ago | 03:11 |
| * mordred lets people point me at openstack-dev messages | 03:12 | |
| dstufft | I wish I could subscribe to specific threads | 03:12 |
| dstufft | Because i'd follow packaging related ones in openstack :V | 03:13 |
| fungi | i skim and use filters (and the filters could use some more work, undoubtedly) | 03:13 |
| mordred | dstufft: well, to be fair, almost all of the packaging traffic happens in here with you :) | 03:14 |
| dstufft | mordred: :) | 03:14 |
| fungi | fwiw i've had to start ignoring most of what's traversed distutils-sig since i hopped over there when package-sig folded in | 03:14 |
| clarkb | mordred: are you rerunning the mirror job with new script yet? | 03:14 |
| mordred | clarkb: not yet. I need to wait for puppet to run right? | 03:14 |
| dstufft | fungi: I'm pretty sure the bulk of distutils-sig traffic is me going "no no no no no no" | 03:14 |
| dstufft | or "dear god no" | 03:14 |
| fungi | heh, the ones from you i open usually are, yes | 03:15 |
| Alex_Gaynor | dstufft: ocasionally "I have root on that box. Quite simply: no." | 03:15 |
| mordred | clarkb, fungi: all of the logs on eavesdrop have mimetypes such that they want me to download them | 03:15 |
| mordred | I feel like they used to be browseable ttw | 03:15 |
| clarkb | mordred: ya, still works with chromium | 03:15 |
| clarkb | mordred: not sure what happened there but noticed it today | 03:16 |
| mordred | clarkb: not with firefox | 03:16 |
| clarkb | ya firefox is derping | 03:16 |
| mordred | one more thing for the list :) | 03:16 |
| * fungi checks | 03:16 | |
| dstufft | Alex_Gaynor: yes that too :V | 03:16 |
| fungi | mordred: clarkb: firefox displays http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-07-16-19.01.log.txt just fine | 03:17 |
| fungi | what's the issue? | 03:17 |
| fungi | aha! | 03:17 |
| fungi | saving with a .log extension in the channel logs and .log.txt extension in the meeting logs | 03:18 |
| zul | mordred: its being done next week | 03:18 |
| pabelanger | Hmm | 03:18 |
| pabelanger | Did we recently add gate-jenkins-job-builder-python26 ? | 03:19 |
| pabelanger | because I think my commits are the first time it is running | 03:19 |
| fungi | pabelanger: the job number on them is 1? | 03:19 |
| clarkb | pabelanger: we did | 03:20 |
| clarkb | pabelanger: did it fail? | 03:20 |
| fungi | pabelanger: yeah, 4 runs ever, all failed | 03:20 |
| pabelanger | Yup | 03:20 |
| pabelanger | I thought it was my code | 03:20 |
| fungi | earliest was about an hour ago | 03:20 |
| pabelanger | but looks likes failing in general | 03:21 |
| pabelanger | I don't have a py26 to test with :) | 03:21 |
| fungi | AttributeError: 'TestCaseModulePublisher' object has no attribute | 03:21 |
| pabelanger | Yup | 03:21 |
| openstackgerrit | Evgeny Fadeev proposed a change to openstack-infra/askbot-theme: added maintenance state template https://review.openstack.org/38015 | 03:21 |
| *** melwitt has quit IRC | 03:21 | |
| pabelanger | assertMultiLineEqual I guess is not for py26 | 03:22 |
| openstackgerrit | A change was merged to openstack-infra/askbot-theme: added maintenance state template https://review.openstack.org/38015 | 03:23 |
| clarkb | pabelanger: can you try changing that to an assertEqual? | 03:23 |
| clarkb | failures won't be as pretty but it should work for both versions of python | 03:23 |
| pabelanger | testing | 03:24 |
| fungi | i recommend a debian/wheezy vm with python{2.{6,7},3.{2,3}} installed (the last has to be pinned from jessie for now) | 03:25 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Switch to flake8+hacking https://review.openstack.org/38013 | 03:25 |
| pabelanger | ya, I need to get my chroots up again | 03:26 |
| clarkb | lifeless: does the testscenarios test class derive from testtools? I know testscenarios depends on testtools | 03:26 |
| pabelanger | clarkb, okay, that seems to work | 03:27 |
| pabelanger | getting a patch up now | 03:28 |
| *** nati_ueno_2 has quit IRC | 03:29 | |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Add python26 support https://review.openstack.org/38017 | 03:29 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr https://review.openstack.org/38009 | 03:30 |
| *** nati_ueno has joined #openstack-infra | 03:31 | |
| clarkb | mordred: I don't see any new mirror runs. Should I kick some off? | 03:31 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Switch to flake8+hacking https://review.openstack.org/38013 | 03:32 |
| fungi | clarkb: aha! caught the jeepyb install problem in action just now... http://puppet-dashboard.openstack.org:3000/reports/654004 | 03:35 |
| fungi | i'll see if i can get more from a cli invocation | 03:35 |
| *** nati_ueno has quit IRC | 03:36 | |
| *** pcrews has quit IRC | 03:37 | |
| fungi | i think maybe it started with pbrization of jeepyb | 03:37 |
| *** woodspa has quit IRC | 03:38 | |
| fungi | our jeepyb puppet module seems to just do 'python setup.py install' which has now started returning error: /usr/bin/python -m pip.__init__ install returned 1 | 03:38 |
| *** SergeyLukjanov has joined #openstack-infra | 03:39 | |
| fungi | do we need to change the jeepyb puppet module to exec something like 'pip install -U .' instead? | 03:40 |
| fungi | er, sorry, i meant currently returning... You must give at least one requirement to install (see "pip help install") | 03:41 |
| pabelanger | fungi, we don't use pip for jeepyb I think, we install from git | 03:43 |
| fungi | pabelanger: yes, but now that pbr invokes pip behind the scenes from setup.py, the way we install from the git clone has stopped working | 03:44 |
| fungi | pabelanger: doing 'pip install -U .' would ask pip to install jeepyb from the source in the current working directory | 03:45 |
| clarkb | fungi: I would try running that by hand and see what happens | 03:45 |
| pabelanger | Ah, ya | 03:45 |
| fungi | pabelanger: https://github.com/openstack-infra/config/blob/master/modules/jeepyb/manifests/init.pp#L68 | 03:46 |
| fungi | clarkb: running what by hand? i already ran the current exec by hand to get the error detail i pasted above | 03:46 |
| clarkb | fungi: pip install -U . | 03:47 |
| fungi | oh, actually also not a great idea since that will pull in/upgrade dependencies from pypi | 03:49 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Setup pbr to gate on sphinx https://review.openstack.org/38018 | 03:49 |
| clarkb | fungi: thoughts on running the mirror script now? | 03:49 |
| clarkb | s/script/jobs/ | 03:49 |
| fungi | clarkb: worth retriggering to test, i agree | 03:52 |
| clarkb | python27 started | 03:52 |
| fungi | mordred: if you're still around, thoughts on a post-pbr alternative to the command we're exec'ing at https://github.com/openstack-infra/config/blob/master/modules/jeepyb/manifests/init.pp#L68 | 03:53 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/config: Drop gate-jenkins-job-builder-pyflakes in favor of flake8 https://review.openstack.org/38019 | 03:54 |
| fungi | mordred: currently failing with http://paste.openstack.org/show/41008/ | 03:54 |
| pabelanger | wow, been a while since I hacked for -infra | 03:56 |
| pabelanger | need more time in the day | 03:56 |
| fungi | pabelanger: it's always appreciated! | 03:56 |
| pabelanger | fungi, Ya, I wish I could help more full-time maybe after this contract wraps up in the next 2 weeks. I should be able to get back into it then | 03:59 |
| *** dkliban has quit IRC | 04:00 | |
| clarkb | fungi: looks like django nose is still having trouble with setuptools | 04:07 |
| clarkb | but otherwise the script is running | 04:08 |
| *** vogxn has joined #openstack-infra | 04:08 | |
| fungi | radical | 04:12 |
| clarkb | fungi: mordred but I think dstufft was right it let stuff in that isn't normally allowed by our requirements | 04:12 |
| clarkb | eg sqlalchemy 0.8 | 04:13 |
| clarkb | so yeah :/ | 04:13 |
| fungi | hrm, right, that's probably not good | 04:13 |
| fungi | as opposed to processing the requires list as a unit, when in theory it would have been disallowed/satisfied with a lower rev | 04:14 |
| dstufft | yea | 04:16 |
| fungi | so this will potentially pull in newer versions of some things than the caps in openstack/requirements when they're dependencies of something else in the list | 04:16 |
| dstufft | because each indivdiual install runs on it's own | 04:16 |
| dstufft | line | 04:16 |
| dstufft | w/e | 04:16 |
| fungi | though i guess it's not actually dangerous since we'll also get the versions we want from the direct entries | 04:17 |
| dstufft | fungi: you don't actually test they all installed together though | 04:17 |
| dstufft | FWIW | 04:17 |
| fungi | and the projects will download only versions matching their requirements ranges from the mirror | 04:17 |
| fungi | dstufft: we actually do, just not there | 04:17 |
| dstufft | or that you have a single set that can satisfy the entire deps | 04:17 |
| dstufft | fungi: oh | 04:17 |
| dstufft | ok | 04:17 |
| fungi | dstufft: we have a separate job which tests installing them all together to confirm that works | 04:18 |
| dstufft | gotcha | 04:18 |
| * dstufft doesn't know the infra that well | 04:18 | |
| * fungi neither | 04:18 | |
| * dstufft still thinks the safer route is to install the entire thing at once and fail the job if it fails to install anything | 04:18 | |
| fungi | if we go that route, we probably also need to do notifications from failed runs since they're periodic or spawned from post-merge events | 04:20 |
| clarkb | fungi: the danger is we can no longer enforce that people follow openstack/requirements | 04:20 |
| clarkb | fungi: they are free to include other stuff (we do have that special test that does help though) | 04:21 |
| clarkb | tl;dr its bad but not world ending bad | 04:21 |
| clarkb | dstufft: I agree | 04:21 |
| fungi | clarkb: i thought we had jobs which admonished them roundly when they tried to add requirements which weren't in the main list | 04:22 |
| clarkb | mordred: I think we should consider reverting that change and cleaning up the contents of pip cache and the mirror. thankfully zaro updated the mirror jobs to have rsync itemize changes so you can see what needs cleaning | 04:22 |
| fungi | as in refused to let them merge such a change | 04:22 |
| clarkb | fungi: we do but the tests enforce via zuul. the mirror enforces via a big giant wall you can't cross | 04:22 |
| fungi | sure, just saying it's only removing a redundant barrier there | 04:23 |
| *** dkliban has joined #openstack-infra | 04:23 | |
| fungi | unless i'm missing something | 04:24 |
| clarkb | fungi: its a lot easier for projects to get around the requirements test eg by having code install deps | 04:24 |
| clarkb | fungi: horizon does this for example | 04:24 |
| fungi | true | 04:24 |
| clarkb | with their django 1.4 testing | 04:24 |
| clarkb | (you can get around the mirror thing too, but it poses a taller wall to climb over) | 04:25 |
| pabelanger | Hmm, when does sphinx generate the autodocs for api's? | 04:26 |
| pabelanger | I cannot get it to trigger | 04:26 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Setup pbr to gate on sphinx https://review.openstack.org/38018 | 04:26 |
| clarkb | pabelanger: if running them locally `tox -evenv -- python setup.py build_sphinx` | 04:26 |
| dstufft | clarkb: fwiw pip 1.4 (due on monday) would have silently kept working and done the right thing for the colorama error | 04:27 |
| pabelanger | clarkb, ya, I am but it is not getting built | 04:27 |
| fungi | clarkb: yeah, so a revert may be in order, but probably not so urgent that we need to do it late on a friday night when everyone's heading out on planes in the next day or so | 04:27 |
| pabelanger | doc/source/api is not created and empty | 04:27 |
| clarkb | fungi: ya I hadn't planned on doing it now :) | 04:27 |
| clarkb | pabelanger: oh that | 04:27 |
| clarkb | pabelanger: what project? | 04:28 |
| pabelanger | JJB | 04:28 |
| *** mrodden has quit IRC | 04:28 | |
| clarkb | pabelanger: I don't think jjb gets an api dir | 04:28 |
| clarkb | pabelanger: and it if does its empty because of doc building stuff in pbr | 04:28 |
| pabelanger | Hmm | 04:28 |
| pabelanger | unless I was doing it for zuul | 04:28 |
| clarkb | you should get a bunch of documentation in doc/build/ though | 04:29 |
| pabelanger | ya, it exists in html | 04:29 |
| pabelanger | it is the chicken and egg thing. with warnerrors enabled, it complain that api/* does not exist if you include api/autoindex in the toctree. but it is built after the warning check | 04:30 |
| pabelanger | oh well, I think my brain is shutting down to find a solution for tonight | 04:31 |
| clarkb | fungi: we should note that http://logs.openstack.org/periodic/periodic-mirror-python27/145/ contains the rsync itemization of extra crap we don't want | 04:31 |
| clarkb | and now it is in our handy dandy IRC logs | 04:31 |
| fungi | excellente | 04:31 |
| clarkb | and with that, good night | 04:31 |
| fungi | night! | 04:31 |
| fungi | i'll probably wind up here in a bit too | 04:31 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jenkins-job-builder: Setup pbr to gate on sphinx https://review.openstack.org/38018 | 04:33 |
| pabelanger | CSS broke on http://zuul.openstack.org/ ? | 04:34 |
| Alex_Gaynor | pabelanger: you looking for status.openstack.org/zuul/ ? | 04:35 |
| pabelanger | Alex_Gaynor, no | 04:35 |
| pabelanger | zuul.o.o | 04:35 |
| pabelanger | http://zuul.openstack.org/bootstrap/css/bootstrap.min.css is missing | 04:35 |
| Alex_Gaynor | looks like it's the same data on the page | 04:35 |
| pabelanger | and http://zuul.openstack.org/bootstrap/css/bootstrap-responsive.min.css | 04:36 |
| pabelanger | a path issue I guess | 04:36 |
| pabelanger | oh, I guess zuul.o.o is not valid :) | 04:36 |
| pabelanger | Alex_Gaynor, right? | 04:37 |
| *** dkliban has quit IRC | 04:37 | |
| Alex_Gaynor | pabelanger: I don't know the official details (like why woudl zuul.o.o even load if you weren't supposed to use it?) but status.o.o/zuul seems to have the same data so :shrug: | 04:37 |
| pabelanger | I think it was zuul.o.o at one time | 04:37 |
| pabelanger | A few months ago | 04:38 |
| pabelanger | likely the dns was never removed | 04:38 |
| pabelanger | clarkb, fungi ^ | 04:38 |
| fungi | pabelanger: yeah now it's just a demo of the default status dashboard we ship in the zuul source i think. the interface we use is the one at http://status.openstack.org/zuul/ like Alex_Gaynor said | 04:39 |
| fungi | same data, but with window dressing | 04:40 |
| fungi | back when the openstack-themed version was still being served directly from zuul it was still a preview--always intended to integrate into the status site later (which it eventually was) | 04:44 |
| *** linuxmohan has joined #openstack-infra | 04:49 | |
| *** linuxmohan has quit IRC | 04:53 | |
| *** linuxmohan has joined #openstack-infra | 04:59 | |
| *** linuxmohan has quit IRC | 05:03 | |
| *** SergeyLukjanov has quit IRC | 05:14 | |
| *** changbl has joined #openstack-infra | 05:39 | |
| *** locke105 has quit IRC | 05:44 | |
| *** locke105 has joined #openstack-infra | 05:46 | |
| *** nati_ueno has joined #openstack-infra | 05:57 | |
| *** vogxn has quit IRC | 06:18 | |
| *** dirk has joined #openstack-infra | 06:31 | |
| *** vogxn has joined #openstack-infra | 06:47 | |
| *** linuxmohan has joined #openstack-infra | 06:51 | |
| *** changbl has quit IRC | 06:56 | |
| *** nati_ueno has quit IRC | 07:02 | |
| *** linuxmohan has quit IRC | 07:04 | |
| *** linuxmohan has joined #openstack-infra | 07:05 | |
| mordred | clarkb, fungi crappit | 07:11 |
| *** linuxmohan has quit IRC | 07:20 | |
| *** linuxmohan has joined #openstack-infra | 07:21 | |
| *** linuxmohan has quit IRC | 07:25 | |
| *** linuxmohan has joined #openstack-infra | 07:25 | |
| *** linuxmohan has joined #openstack-infra | 07:43 | |
| *** arosen has quit IRC | 07:46 | |
| *** koobs has joined #openstack-infra | 07:51 | |
| *** linuxmohan has quit IRC | 08:15 | |
| *** linuxmohan has joined #openstack-infra | 08:17 | |
| *** dkehn_ has joined #openstack-infra | 08:20 | |
| *** linuxmohan has quit IRC | 08:21 | |
| *** Reapster_ has joined #openstack-infra | 08:25 | |
| *** guitarza1 has joined #openstack-infra | 08:26 | |
| *** koolhead17 has quit IRC | 08:30 | |
| *** jpeeler has quit IRC | 08:30 | |
| *** afrittoli has quit IRC | 08:30 | |
| *** Reapster has quit IRC | 08:30 | |
| *** dkehn_away has quit IRC | 08:30 | |
| *** zaro has quit IRC | 08:30 | |
| *** akscram has quit IRC | 08:30 | |
| *** guitarzan has quit IRC | 08:30 | |
| *** vogxn has quit IRC | 08:30 | |
| *** afrittoli has joined #openstack-infra | 08:31 | |
| *** dirk_ has joined #openstack-infra | 08:31 | |
| *** dirk_ has joined #openstack-infra | 08:32 | |
| *** dkehn__ has joined #openstack-infra | 08:32 | |
| *** mancdaz_ has joined #openstack-infra | 08:34 | |
| *** Reapster has joined #openstack-infra | 08:37 | |
| *** vogxn has joined #openstack-infra | 08:37 | |
| *** zaro has joined #openstack-infra | 08:38 | |
| *** koolhead17 has joined #openstack-infra | 08:38 | |
| *** akscram has joined #openstack-infra | 08:39 | |
| *** Shrews_ has joined #openstack-infra | 08:40 | |
| *** arosen has joined #openstack-infra | 08:42 | |
| *** jpeeler1 has joined #openstack-infra | 08:45 | |
| *** Reapster_ has quit IRC | 08:46 | |
| *** dkehn_ has quit IRC | 08:46 | |
| *** dirk has quit IRC | 08:46 | |
| *** openstackgerrit has quit IRC | 08:46 | |
| *** dtroyer has quit IRC | 08:46 | |
| *** mancdaz has quit IRC | 08:46 | |
| *** Ng has quit IRC | 08:46 | |
| *** Shrews has quit IRC | 08:46 | |
| *** mancdaz_ is now known as mancdaz | 08:46 | |
| *** Ng has joined #openstack-infra | 08:49 | |
| *** dtroyer has joined #openstack-infra | 08:49 | |
| *** zul has quit IRC | 08:54 | |
| *** yamahata has quit IRC | 08:54 | |
| *** yamahata has joined #openstack-infra | 08:57 | |
| *** zul has joined #openstack-infra | 08:57 | |
| *** Ng_ has joined #openstack-infra | 08:59 | |
| *** mtaylor has joined #openstack-infra | 09:00 | |
| *** LinuxJedi has quit IRC | 09:06 | |
| *** mordred has quit IRC | 09:06 | |
| *** jeremyb has quit IRC | 09:06 | |
| *** Ng has quit IRC | 09:06 | |
| *** dtroyer has quit IRC | 09:06 | |
| *** ttx has quit IRC | 09:06 | |
| *** hughsaunders has quit IRC | 09:06 | |
| *** annegentle_ has quit IRC | 09:06 | |
| *** juice has quit IRC | 09:06 | |
| *** cp16net|away has quit IRC | 09:06 | |
| *** sileht has quit IRC | 09:06 | |
| *** briancline has quit IRC | 09:06 | |
| *** EntropyWorks has quit IRC | 09:06 | |
| *** jcooley has quit IRC | 09:06 | |
| *** jpeeler1 has quit IRC | 09:06 | |
| *** devananda has quit IRC | 09:06 | |
| *** blamar has quit IRC | 09:06 | |
| *** Guest64828 has quit IRC | 09:06 | |
| *** simonmcc has quit IRC | 09:06 | |
| *** beagles has quit IRC | 09:06 | |
| *** cody-somerville has quit IRC | 09:06 | |
| *** andreaf has quit IRC | 09:06 | |
| *** echohead_ has quit IRC | 09:06 | |
| *** akscram has quit IRC | 09:06 | |
| *** guitarza1 has quit IRC | 09:06 | |
| *** koobs has quit IRC | 09:06 | |
| *** plomakin has quit IRC | 09:06 | |
| *** mikal has quit IRC | 09:06 | |
| *** jjmb has quit IRC | 09:06 | |
| *** dkranz has quit IRC | 09:06 | |
| *** jd__ has quit IRC | 09:06 | |
| *** saper has quit IRC | 09:06 | |
| *** dhellmann_ has quit IRC | 09:06 | |
| *** Adri2000 has quit IRC | 09:06 | |
| *** pleia2 has quit IRC | 09:06 | |
| *** jeblair has quit IRC | 09:06 | |
| *** sdague has quit IRC | 09:06 | |
| *** thingee has quit IRC | 09:06 | |
| *** cyeoh has quit IRC | 09:06 | |
| *** maurosr has quit IRC | 09:06 | |
| *** SlickNik has quit IRC | 09:06 | |
| *** arosen has quit IRC | 09:06 | |
| *** locke105 has quit IRC | 09:06 | |
| *** klrmn has quit IRC | 09:06 | |
| *** rwsu-away has quit IRC | 09:06 | |
| *** sdake_ has quit IRC | 09:06 | |
| *** dirk has joined #openstack-infra | 09:07 | |
| *** dtroyer_zz has joined #openstack-infra | 09:07 | |
| *** jpeeler1 has joined #openstack-infra | 09:07 | |
| *** arosen has joined #openstack-infra | 09:07 | |
| *** akscram has joined #openstack-infra | 09:07 | |
| *** guitarza1 has joined #openstack-infra | 09:07 | |
| *** koobs has joined #openstack-infra | 09:07 | |
| *** locke105 has joined #openstack-infra | 09:07 | |
| *** plomakin has joined #openstack-infra | 09:07 | |
| *** mikal has joined #openstack-infra | 09:07 | |
| *** jjmb has joined #openstack-infra | 09:07 | |
| *** klrmn has joined #openstack-infra | 09:07 | |
| *** devananda has joined #openstack-infra | 09:07 | |
| *** dkranz has joined #openstack-infra | 09:07 | |
| *** blamar has joined #openstack-infra | 09:07 | |
| *** rwsu-away has joined #openstack-infra | 09:07 | |
| *** Guest64828 has joined #openstack-infra | 09:07 | |
| *** jd__ has joined #openstack-infra | 09:07 | |
| *** saper has joined #openstack-infra | 09:07 | |
| *** dhellmann_ has joined #openstack-infra | 09:07 | |
| *** simonmcc has joined #openstack-infra | 09:07 | |
| *** Adri2000 has joined #openstack-infra | 09:07 | |
| *** beagles has joined #openstack-infra | 09:07 | |
| *** cody-somerville has joined #openstack-infra | 09:07 | |
| *** andreaf has joined #openstack-infra | 09:07 | |
| *** echohead_ has joined #openstack-infra | 09:07 | |
| *** pleia2 has joined #openstack-infra | 09:07 | |
| *** jcooley has joined #openstack-infra | 09:07 | |
| *** EntropyWorks has joined #openstack-infra | 09:07 | |
| *** briancline has joined #openstack-infra | 09:07 | |
| *** sileht has joined #openstack-infra | 09:07 | |
| *** cp16net|away has joined #openstack-infra | 09:07 | |
| *** juice has joined #openstack-infra | 09:07 | |
| *** annegentle_ has joined #openstack-infra | 09:07 | |
| *** hughsaunders has joined #openstack-infra | 09:07 | |
| *** ttx has joined #openstack-infra | 09:07 | |
| *** jeblair has joined #openstack-infra | 09:07 | |
| *** sdague has joined #openstack-infra | 09:07 | |
| *** thingee has joined #openstack-infra | 09:07 | |
| *** cyeoh has joined #openstack-infra | 09:07 | |
| *** maurosr has joined #openstack-infra | 09:07 | |
| *** sdake_ has joined #openstack-infra | 09:07 | |
| *** SlickNik has joined #openstack-infra | 09:07 | |
| *** jeremyb has joined #openstack-infra | 09:07 | |
| *** ttx has quit IRC | 09:08 | |
| *** hughsaunders has quit IRC | 09:08 | |
| *** annegentle_ has quit IRC | 09:08 | |
| *** juice has quit IRC | 09:08 | |
| *** cp16net|away has quit IRC | 09:08 | |
| *** sileht has quit IRC | 09:08 | |
| *** briancline has quit IRC | 09:08 | |
| *** EntropyWorks has quit IRC | 09:08 | |
| *** jcooley has quit IRC | 09:08 | |
| *** ttx has joined #openstack-infra | 09:10 | |
| *** hughsaunders has joined #openstack-infra | 09:10 | |
| *** annegentle_ has joined #openstack-infra | 09:10 | |
| *** juice has joined #openstack-infra | 09:10 | |
| *** cp16net|away has joined #openstack-infra | 09:10 | |
| *** sileht has joined #openstack-infra | 09:10 | |
| *** briancline has joined #openstack-infra | 09:10 | |
| *** EntropyWorks has joined #openstack-infra | 09:10 | |
| *** jcooley has joined #openstack-infra | 09:10 | |
| *** dirk_ has quit IRC | 09:11 | |
| *** LinuxJedi has joined #openstack-infra | 09:12 | |
| *** echohead1 has joined #openstack-infra | 09:18 | |
| *** dtroyer has joined #openstack-infra | 09:18 | |
| *** linuxmohan has joined #openstack-infra | 09:20 | |
| *** Vivek has joined #openstack-infra | 09:21 | |
| *** Vivek is now known as Guest43078 | 09:21 | |
| *** blamar_ has joined #openstack-infra | 09:23 | |
| *** beagles` has joined #openstack-infra | 09:24 | |
| *** dtroyer_zz has quit IRC | 09:24 | |
| *** jpeeler1 has quit IRC | 09:24 | |
| *** devananda has quit IRC | 09:24 | |
| *** blamar has quit IRC | 09:24 | |
| *** Guest64828 has quit IRC | 09:24 | |
| *** simonmcc has quit IRC | 09:24 | |
| *** beagles has quit IRC | 09:24 | |
| *** cody-somerville has quit IRC | 09:24 | |
| *** andreaf has quit IRC | 09:24 | |
| *** echohead_ has quit IRC | 09:24 | |
| *** blamar_ is now known as blamar | 09:24 | |
| *** devananda has joined #openstack-infra | 09:24 | |
| *** simonmcc has joined #openstack-infra | 09:25 | |
| *** andreaf has joined #openstack-infra | 09:26 | |
| *** linuxmohan has quit IRC | 09:27 | |
| *** koobs has quit IRC | 09:31 | |
| *** jpeeler1 has joined #openstack-infra | 09:31 | |
| *** cody-somerville has joined #openstack-infra | 09:31 | |
| *** cody-somerville has joined #openstack-infra | 09:31 | |
| *** koobs has joined #openstack-infra | 09:31 | |
| *** linuxmohan has joined #openstack-infra | 09:34 | |
| *** linuxmohan has quit IRC | 09:34 | |
| *** odyssey4me has joined #openstack-infra | 10:44 | |
| *** linuxmohan has joined #openstack-infra | 10:44 | |
| *** linuxmohan has quit IRC | 10:50 | |
| *** arosen has quit IRC | 10:50 | |
| *** koolhead17 has quit IRC | 11:01 | |
| *** koobs has quit IRC | 11:02 | |
| *** dtroyer has quit IRC | 11:02 | |
| *** LinuxJedi has quit IRC | 11:02 | |
| *** arosen has joined #openstack-infra | 11:02 | |
| *** dtroyer has joined #openstack-infra | 11:02 | |
| *** koobs has joined #openstack-infra | 11:03 | |
| *** LinuxJedi has joined #openstack-infra | 11:04 | |
| *** vogxn has quit IRC | 11:13 | |
| *** linuxmohan has joined #openstack-infra | 11:15 | |
| *** arosen has quit IRC | 11:19 | |
| *** odyssey4me has quit IRC | 11:33 | |
| *** arosen has joined #openstack-infra | 11:41 | |
| *** mikal has quit IRC | 11:45 | |
| *** mikal has joined #openstack-infra | 11:47 | |
| *** linuxmohan has quit IRC | 11:49 | |
| *** linuxmohan has joined #openstack-infra | 11:49 | |
| *** linuxmohan has quit IRC | 11:54 | |
| kiall | FYI Looks like openstackgerrit is on the wrong side of a split for the last 3 hours.. Maybe that server went down and it failed to reconnect... | 11:55 |
| *** mikal has quit IRC | 12:06 | |
| *** mikal has joined #openstack-infra | 12:08 | |
| *** SergeyLukjanov has joined #openstack-infra | 12:44 | |
| *** linuxmohan has joined #openstack-infra | 12:49 | |
| fungi | kiall: i'll kick it in a sec | 12:51 |
| *** openstackgerrit has joined #openstack-infra | 12:54 | |
| fungi | kiall: it's back now | 12:55 |
| *** linuxmohan has quit IRC | 12:55 | |
| kiall | Cool :) Gotta love buggy IRC reconnect logic ;) | 12:55 |
| *** rfolco has joined #openstack-infra | 12:58 | |
| openstackgerrit | Thomas Goirand proposed a change to openstack/requirements: Removes the cap for SQLAlchemy https://review.openstack.org/38035 | 12:59 |
| *** linuxmohan has joined #openstack-infra | 13:07 | |
| *** SergeyLukjanov has quit IRC | 13:09 | |
| *** prad-afk has quit IRC | 13:13 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add gate job for CI docs https://review.openstack.org/36295 | 13:13 |
| *** SergeyLukjanov has joined #openstack-infra | 13:23 | |
| *** prad has joined #openstack-infra | 13:30 | |
| *** jaypipes has quit IRC | 13:34 | |
| *** SergeyLukjanov has quit IRC | 13:37 | |
| *** prad has left #openstack-infra | 13:59 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:13 | |
| *** SergeyLukjanov has quit IRC | 14:14 | |
| clarkb | fungi when do you sleep? | 14:16 |
| *** CaptTofu has quit IRC | 14:18 | |
| *** CaptTofu has joined #openstack-infra | 14:18 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:20 | |
| *** CaptTofu has quit IRC | 14:23 | |
| *** sandywalsh has quit IRC | 14:28 | |
| *** rfolco has quit IRC | 14:29 | |
| *** dkehn__ is now known as dkehn | 14:32 | |
| *** ianw has quit IRC | 14:32 | |
| *** koolhead17 has joined #openstack-infra | 14:34 | |
| mtaylor | clarkb: why would he sleep? | 14:58 |
| clarkb | Human beings require some amount of sleep. are you implying he is a robot? | 15:01 |
| clarkb | mtaylor: are we reverting the mirror change? | 15:02 |
| mtaylor | clarkb: hrm | 15:02 |
| mtaylor | clarkb: so which breakage to we hate more? | 15:02 |
| mtaylor | clarkb: do we hate having extra items in our mirror? | 15:02 |
| mtaylor | clarkb: or less? | 15:02 |
| clarkb | I am fine with extra as I argued the mirror should exist for test reliability and not requirement enforcement | 15:03 |
| mtaylor | :) | 15:03 |
| clarkb | but jeblair disagrees and I think he will want less | 15:03 |
| mtaylor | so maybe we revert and then go back to fixing the individual error | 15:04 |
| mtaylor | errors | 15:04 |
| mtaylor | and knowing that from time to time some asshat might re-upload a package | 15:04 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Revert "Be more agressive trying to install requirements." https://review.openstack.org/38036 | 15:06 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Upgrade pip and setuptools in the mirror venv https://review.openstack.org/38004 | 15:06 |
| mtaylor | clarkb: that ^^ should get us past the next problem in the reverted world view | 15:06 |
| mtaylor | clarkb: to clean the current mirror, we need to clean the pip cache and the mirror itself, yeah? | 15:07 |
| clarkb | yes | 15:09 |
| mtaylor | clarkb: hang on... | 15:10 |
| mtaylor | https://jenkins.openstack.org/job/periodic-mirror-python27/142/console | 15:10 |
| mtaylor | look at the rsync log | 15:10 |
| mtaylor | why do we think that we got things that we didn't want to get? | 15:11 |
| clarkb | mtaylor: s/142/145/ | 15:11 |
| mtaylor | I see it transferring cffi, pand xattr | 15:11 |
| clarkb | mtaylor: it wans't until I ran build 145 that extra stuff copied | 15:11 |
| mtaylor | ah. ok | 15:11 |
| clarkb | there is a neat gerrit plugin that adds reviewers based on the results of git blaming around the lines you have changed | 15:13 |
| mtaylor | oh neat | 15:16 |
| mtaylor | clarkb: I am testing the two patches above to see if they can build the mirror | 15:16 |
| *** dklyle has joined #openstack-infra | 15:16 | |
| clarkb | that should tell you if the pip and setuptools upgrade fixes the problem right? | 15:17 |
| *** SergeyLukjanov has quit IRC | 15:18 | |
| mtaylor | yup | 15:22 |
| *** linuxmohan has quit IRC | 15:24 | |
| *** UtahDave has joined #openstack-infra | 15:25 | |
| mtaylor | clarkb: appears to have worked | 15:31 |
| mtaylor | clarkb: I think we should land the patches, and then I can clean the cache and mirror on both slaves | 15:31 |
| *** dklyle has quit IRC | 15:32 | |
| jeblair | quick suggestion | 15:32 |
| mtaylor | jeblair: yeah? | 15:33 |
| mtaylor | ooh! it's jeblair | 15:33 |
| jeblair | stick a --download-cache option on those pip install commands? | 15:33 |
| jeblair | make it a bit faster? | 15:33 |
| mtaylor | oh - on the setuptools and pip upgrade commands? | 15:33 |
| jeblair | i assume we'd be okay with the versions of setuptools and pip that the mirror builder is using ending up in the mirror | 15:33 |
| jeblair | mtaylor: yep | 15:33 |
| jeblair | mtaylor: so we don't have to download those 2 every time | 15:33 |
| mtaylor | jeblair: ++ | 15:34 |
| mtaylor | jeblair: also, fwiw, dstufft suggested that we not use -M | 15:35 |
| jeblair | mtaylor: which one is that? | 15:36 |
| *** ewindisch has joined #openstack-infra | 15:36 | |
| clarkb | mirrors | 15:36 |
| jeblair | ah, yeah, they don't always seem to be super helpful do they? :) | 15:37 |
| *** dirk has quit IRC | 15:38 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Remove -M from the pip install command https://review.openstack.org/38038 | 15:39 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Use print function for python3 https://review.openstack.org/38039 | 15:39 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Upgrade pip and setuptools in the mirror venv https://review.openstack.org/38004 | 15:39 |
| jeblair | mtaylor: merge conflict? | 15:39 |
| mtaylor | god, really? | 15:39 |
| mtaylor | what a fun race condition | 15:39 |
| *** pcrews has joined #openstack-infra | 15:40 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Upgrade pip and setuptools in the mirror venv https://review.openstack.org/38004 | 15:42 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Use print function for python3 https://review.openstack.org/38039 | 15:42 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Remove -M from the pip install command https://review.openstack.org/38038 | 15:42 |
| clarkb | http://openstackreactions.tumblr.com/ the jenkins gate passed gif made me laugh | 15:42 |
| mtaylor | I rebased them on top of the revert | 15:42 |
| *** mikal has quit IRC | 15:43 | |
| mtaylor | haha | 15:43 |
| mtaylor | "When I still have -1 after 15 patch-sets…" | 15:43 |
| *** mikal has joined #openstack-infra | 15:44 | |
| clarkb | mtaylor: jeblair http://engineering.cloudscaling.com/happy-3rd-birthday-openstack/ any idea where those numbers come from? they seem low | 15:45 |
| clarkb | I think we ran >5k jobs on Tuesday alone | 15:45 |
| mtaylor | clarkb: no clue | 15:46 |
| *** ewindisch has quit IRC | 15:50 | |
| mtaylor | clarkb: so, I can has some +2's? | 15:57 |
| mtaylor | https://review.openstack.org/#/q/status:open+project:openstack-infra/jeepyb,n,z | 15:57 |
| *** jaypipes has joined #openstack-infra | 15:59 | |
| Alex_Gaynor | Has anyone else seen "error: invalid command 'egg_info'" seems to be affecting all devstack builds | 16:01 |
| mtaylor | Alex_Gaynor: I was just trying to reproduce that from another thing - is it hitting all devstacks now? | 16:05 |
| mtaylor | Alex_Gaynor: and is it around warlock? | 16:05 |
| Alex_Gaynor | Yes, and yes. | 16:05 |
| mtaylor | Alex_Gaynor: well yes, I have seen it, and I have not been able to reproduce it | 16:06 |
| mtaylor | (I'm actually working on that right now) | 16:07 |
| mtaylor | http://logs.openstack.org/00/38000/2/check/gate-tempest-devstack-vm-full/34051/console.html.gz | 16:07 |
| mtaylor | was triggered by the same change as | 16:07 |
| mtaylor | http://logs.openstack.org/00/38000/2/check/gate-pbr-devstack-vm-rawinstall/231/console.html.gz | 16:07 |
| mtaylor | one passes, the other fails | 16:07 |
| mtaylor | both run devstack the same way | 16:07 |
| mtaylor | I have no idea why the one that's failing is failing | 16:08 |
| mtaylor | because if you instal those things, from that mirror, in that order, it works finee | 16:08 |
| Alex_Gaynor | Yeah I can't reproduce it either, bizzare | 16:11 |
| clarkb | mtaylor: +2's given | 16:11 |
| mtaylor | clarkb: thanks. | 16:11 |
| mtaylor | clarkb: also, we're having weird gate failures | 16:11 |
| mtaylor | Alex_Gaynor: you got a link to another one that's failing? | 16:12 |
| clarkb | mtaylor: your update setuptools everywhere change get onto d-g images yet? | 16:12 |
| Alex_Gaynor | https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/34083/consoleText | 16:12 |
| mtaylor | Alex_Gaynor: thank you | 16:12 |
| jeblair | mirror changes aprvd | 16:13 |
| * mtaylor trying to diff the failing and non-failing run to see what's different | 16:16 | |
| mtaylor | although probably logstash would make this easier | 16:16 |
| clarkb | maybe, I am not sure how good it is at diffing | 16:16 |
| *** linuxmohan has joined #openstack-infra | 16:17 | |
| mtaylor | - Installing /usr/local/lib/python2.7/dist-packages/oslo.config-1.1.1-nspkg.pth | 16:18 |
| mtaylor | + Installing /usr/local/lib/python2.7/dist-packages/oslo.config-1.1.1-py2.7-nspkg.pth | 16:18 |
| mtaylor | that's interesting | 16:18 |
| mtaylor | -Searching for distribute==0.6.24dev-r0 | 16:20 |
| mtaylor | -Best match: distribute 0.6.24dev-r0 | 16:20 |
| mtaylor | -Adding distribute 0.6.24dev-r0 to easy-install.pth file | 16:20 |
| mtaylor | +Searching for setuptools==0.9.6 | 16:20 |
| mtaylor | +Best match: setuptools 0.9.6 | 16:20 |
| mtaylor | +Adding setuptools 0.9.6 to easy-install.pth file | 16:21 |
| *** CaptTofu has joined #openstack-infra | 16:21 | |
| *** SergeyLukjanov has joined #openstack-infra | 16:22 | |
| mtaylor | nope. red herrir | 16:23 |
| mtaylor | herring | 16:23 |
| mtaylor | jeez | 16:25 |
| mtaylor | I have NO CLUE | 16:25 |
| clarkb | mtaylor: I wonder if it has to do with ensureing latest setuptools everywhere | 16:25 |
| clarkb | I am assuming we have finally built new d-g images since them | 16:25 |
| mtaylor | nope | 16:25 |
| mtaylor | there are different versions of setuptools or distribute on passing and failing nodes | 16:26 |
| mtaylor | like, the is not a correlation | 16:26 |
| mtaylor | I have a passing test with setuptools 0.9.6 and a failing test with 0.9.6 | 16:26 |
| mtaylor | https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/34083/consoleText fails - 0.9.6 | 16:27 |
| mtaylor | http://logs.openstack.org/00/38000/2/check/gate-tempest-devstack-vm-full/34051/console.html.gz passes - 0.9.6 | 16:28 |
| jeblair | all 3 update image scripts completed successfully, so the d-g nodes are probably the same across all zones. but don't completely discount that. | 16:29 |
| mtaylor | kk | 16:30 |
| jeblair | mtaylor: and your 2 examples are both from az2 after the most recent image build | 16:30 |
| mtaylor | I cannot find what's different between them | 16:30 |
| jeblair | oh, but let me make sure the _nodes_ came from the same image | 16:30 |
| mtaylor | ah | 16:30 |
| mtaylor | oh wait- I'm wrong | 16:32 |
| mtaylor | http://logs.openstack.org/00/38000/2/check/gate-tempest-devstack-vm-full/34051/console.html.gz passes and has distribute 0.6.24 | 16:32 |
| jeblair | success job node would have come from the previous image, not the most current one. failed job node came from the current image. | 16:33 |
| jeblair | mtaylor: clarkb's theory sounding probable? | 16:33 |
| *** changbl has joined #openstack-infra | 16:34 | |
| mtaylor | wow. it seems like upstream setuptools has perhaps removed the ability to do an upgrade from distribute | 16:34 |
| mtaylor | dstufft: ^^^ | 16:34 |
| mtaylor | " | 16:34 |
| mtaylor | Currently, Distribute disallows installing Setuptools 0.7+ over Distribute. You must first uninstall any active version of Distribute first (see Uninstalling)." | 16:34 |
| mtaylor | https://pypi.python.org/pypi/setuptools/0.9.6 | 16:34 |
| mtaylor | what an f-ing mess everything about this is | 16:35 |
| *** beagles` has quit IRC | 16:36 | |
| mtaylor | so, but why is this working for me locally in a venv with setuptools 0.9.6 installed? | 16:39 |
| mtaylor | jeblair: | 16:45 |
| mtaylor | gah | 16:45 |
| jeblair | mtaylor: doh | 16:45 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add hieraedit.py https://review.openstack.org/38003 | 16:48 |
| *** SergeyLukjanov has quit IRC | 16:49 | |
| mtaylor | now I'm just trying random things | 16:49 |
| jeblair | oh, i forgot to aprv the revert change; just did that. | 16:49 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Revert "Be more agressive trying to install requirements." https://review.openstack.org/38036 | 16:50 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Upgrade pip and setuptools in the mirror venv https://review.openstack.org/38004 | 16:50 |
| mtaylor | jeblair: btw - do you have any idea where the double timestamp is coming from? | 16:50 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Remove -M from the pip install command https://review.openstack.org/38038 | 16:50 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Use print function for python3 https://review.openstack.org/38039 | 16:50 |
| mtaylor | 2013-07-20 16:50:32.216 | 2013-07-20 16:50:32 stack.sh log /opt/stack/new/devstacklog.txt.2013-07-20-165032 | 16:50 |
| jeblair | mtaylor: devstack prints timestamps, and jenkins console log wrapper adds them as well | 16:51 |
| mtaylor | jeblair: what is it about devstack that causes it to print timestamps? | 16:51 |
| jeblair | mtaylor: i don't know, but i think it's pretty cool | 16:51 |
| * mtaylor wasn't sure what to look for in stack.sh | 16:51 | |
| mtaylor | I do think it's pretty cool | 16:51 |
| mtaylor | it would be nice if we could disable it for devstack gate runs, since we already have timestamps on those lines | 16:51 |
| jeblair | indeed | 16:51 |
| jeblair | though that might affect the timestamps in the devstack log file | 16:52 |
| mtaylor | set +o xtrace | 16:52 |
| jeblair | </speculation> | 16:52 |
| mtaylor | ? | 16:52 |
| jeblair | that should just cause things to be printed; not necessarily with timestamps | 16:52 |
| mtaylor | hrm | 16:52 |
| *** vogxn has joined #openstack-infra | 16:52 | |
| mtaylor | ok. I found the place in stack.sh | 16:55 |
| jeblair | mtaylor: so do you want to revert the setuptools patch to puppet in order to get new d-g images that work? | 16:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Allow specification of public UDP ports https://review.openstack.org/37506 | 16:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Set up basic Asterisk configuration https://review.openstack.org/37505 | 16:58 |
| mtaylor | jeblair: one sec - I'm trying a patch to devstack | 16:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable inbound SIP calls https://review.openstack.org/37608 | 16:58 |
| jeblair | k | 16:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable basic conferencing support https://review.openstack.org/37705 | 16:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Install additional sound prompts https://review.openstack.org/37708 | 16:58 |
| mtaylor | jeblair: gonna try a couple of things - if none of them work, then we'll revert | 16:58 |
| mtaylor | and I'll scream at people on monday | 16:59 |
| jeblair | mtaylor: my availibility to help is limited; so just wanted to pitch in while i can. :) | 16:59 |
| jeblair | mtaylor: if you're comfortable spending more of your saturday dealing with breakage, is ok with me. :) | 16:59 |
| mtaylor | thank you! | 16:59 |
| clarkb | yeah, I need to get in a car and start driving soon | 16:59 |
| mtaylor | well, I mean, right now I'm just angry that this is broken | 16:59 |
| mtaylor | in a very deep way | 16:59 |
| jeblair | your anger gives you power. embrace your anger. | 17:00 |
| jeblair | or was it the other way around? i can never remember. | 17:00 |
| mtaylor | jeblair: you're going to come back in this evening and I will have re-written all of openstack in C++ | 17:00 |
| jeblair | mtaylor: wfm. i think compiling is really cool. :) | 17:01 |
| jeblair | #status alert devstack-gate jobs broken due to setuptools brokenness; fix in progress. | 17:02 |
| openstackstatus | NOTICE: devstack-gate jobs broken due to setuptools brokenness; fix in progress. | 17:02 |
| *** ChanServ changes topic to "devstack-gate jobs broken due to setuptools brokenness; fix in progress." | 17:02 | |
| *** changbl has quit IRC | 17:02 | |
| mtaylor | ok. let's revert | 17:03 |
| mtaylor | none of this makes even the foggiest bit of senes | 17:03 |
| *** pcrews has quit IRC | 17:04 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Revert "Ensure latest setuptools is present." https://review.openstack.org/38041 | 17:05 |
| *** linuxmohan has quit IRC | 17:08 | |
| *** linuxmohan has joined #openstack-infra | 17:08 | |
| * mtaylor trying one more thing... | 17:09 | |
| mtaylor | BWAHAHAHAHAHAHAHA | 17:11 |
| mtaylor | jeblair: evil laugh evil laugh | 17:11 |
| mtaylor | jeblair: apt-get remove -y python-setuptools in devstack is helpful | 17:11 |
| *** CaptTofu has quit IRC | 17:13 | |
| *** CaptTofu has joined #openstack-infra | 17:13 | |
| mtaylor | jeblair: https://review.openstack.org/38040 | 17:14 |
| mtaylor | dtroyer, sdague ^^ | 17:14 |
| mtaylor | clarkb: ^^ | 17:14 |
| clarkb | mtaylor: jeblair -1 because it seems to be debian derivative specific | 17:17 |
| *** CaptTofu has quit IRC | 17:17 | |
| mtaylor | clarkb: oh right | 17:18 |
| mtaylor | people use this elsehwere dont' they? | 17:18 |
| mtaylor | also, virtualenv isn't in our mirror | 17:18 |
| clarkb | mtaylor: and it just failed everythin. is that because virtualenv is not in our mirror? | 17:18 |
| mtaylor | yes | 17:18 |
| mtaylor | let me amend | 17:18 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Update the default extension https://review.openstack.org/37755 | 17:19 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Update asterisk documentation https://review.openstack.org/37762 | 17:19 |
| mtaylor | clarkb: updated | 17:20 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add voipms configuration https://review.openstack.org/37765 | 17:20 |
| openstackgerrit | A change was merged to openstack-infra/config: Update the default extension https://review.openstack.org/37755 | 17:21 |
| *** SergeyLukjanov has joined #openstack-infra | 17:21 | |
| openstackgerrit | A change was merged to openstack-infra/config: Update asterisk documentation https://review.openstack.org/37762 | 17:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Add voipms configuration https://review.openstack.org/37765 | 17:22 |
| mtaylor | clarkb, jeblair: ok. I just updated the patch one more time, moving the setuptools removal to before the "if virtualenv" block | 17:24 |
| mtaylor | it's not important for the gate, but if someone did want to use the track depends feature, the other thing would have made their life weird | 17:25 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jeepyb: Cause virtualenv to be installed into the mirror https://review.openstack.org/38042 | 17:30 |
| mtaylor | also that | 17:30 |
| mtaylor | clarkb, jeblair https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/34096/console https://review.openstack.org/38040 is past the install stagees of things | 17:30 |
| jeblair | mtaylor: you could probably go ahead and approve it then | 17:34 |
| mtaylor | jeblair: ok. I just wanted a double check from you guys on it - clarkb's catch about deb specific was a good one | 17:34 |
| mtaylor | jeblair: have I mentioned in the last couple of minutes that this is a mess? :) | 17:37 |
| mtaylor | Alex_Gaynor: ^^ I believe we're landing a fix for the devstack gate | 17:37 |
| clarkb | mtaylor: is it called python-setuptools on fedora and friends? | 17:38 |
| mtaylor | Alex_Gaynor, dstufft: I do not fully grok the root cause, the but the symptoms seem to revolve around distro-installed old distribute in /usr/lib/ combined with pip installed new setuptools | 17:38 |
| mtaylor | clarkb: thankfully, yes | 17:38 |
| clarkb | wow | 17:38 |
| clarkb | that never happens | 17:39 |
| clarkb | +1 from me | 17:39 |
| clarkb | mtaylor: https://bugs.launchpad.net/openstack-ci/+bug/1203340 | 17:39 |
| mtaylor | GAH. it broke | 17:40 |
| mtaylor | uhm | 17:41 |
| mtaylor | http://logs.openstack.org/40/38040/7/check/gate-tempest-devstack-vm-full/34096/console.html | 17:41 |
| mtaylor | wth? | 17:41 |
| clarkb | mtaylor: I don't think that is you, instead its one of the many failures the gate runs into | 17:42 |
| clarkb | mtaylor: however your grenade job in the gate failed | 17:42 |
| mtaylor | clarkb: all of the jobs failed | 17:42 |
| mtaylor | oh. good | 17:44 |
| mtaylor | clarkb, jeblair: I think we're going to have to revert the setuptools everywhere change and rebuild the pool | 17:44 |
| jeblair | mtaylor: ok | 17:45 |
| clarkb | wfm | 17:45 |
| mtaylor | clarkb: then we can land changes in grenade and devstack (since we have to land them in both) | 17:45 |
| mtaylor | which means we can get help tracking down why a new version of setuptools causes cinder and euca to fail | 17:45 |
| jeblair | did you abandon your revert patch? | 17:45 |
| mtaylor | https://review.openstack.org/#/c/38041/ | 17:46 |
| mtaylor | yu[p | 17:46 |
| mtaylor | restored | 17:46 |
| jeblair | taking clarkb's wfm as approval and aprvd it | 17:46 |
| mtaylor | ++ | 17:46 |
| jeblair | mtaylor: next step is to trigger image build jobs on all zones; then when they complete, run the reap jobs on all zones with "all servers" checked | 17:47 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Ensure latest setuptools is present." https://review.openstack.org/38041 | 17:47 |
| mtaylor | hrm | 17:47 |
| jeblair | hrm? | 17:47 |
| mtaylor | jeblair: sorry - different thing - the grenade gate job triggered by devstack changes does not run the copy of devstack proposed for change | 17:48 |
| mtaylor | jeblair: for trigger image build jobs - is there a better way than just clicking in jenkins? | 17:49 |
| jeblair | mtaylor: just did it | 17:49 |
| mtaylor | jeblair: oh! you're quick | 17:49 |
| clarkb | jeblair is an email and test ninja | 17:50 |
| clarkb | it sounds like this should be settled shortly. I am going to shift gears into driving to the dalles | 17:50 |
| mtaylor | clarkb: have fun | 17:50 |
| jeblair | mtaylor: i'm about to have to run out, so when the image jobs build, you'll want to run https://jenkins.openstack.org/job/devstack-reap-vms-hpcloud-az1/build?delay=0sec and check 'all servers', and do that for az2 and az3 as well. | 17:54 |
| mtaylor | jeblair: ok. I will do | 17:54 |
| mtaylor | thank you for your help | 17:54 |
| jeblair | mtaylor: image build: https://jenkins.openstack.org/job/devstack-update-vm-image-hpcloud-az1/ | 17:54 |
| jeblair | mtaylor: np | 17:54 |
| *** pcrews has joined #openstack-infra | 17:59 | |
| *** pabelanger has quit IRC | 18:17 | |
| mtaylor | reap jobs running | 18:20 |
| *** changbl has joined #openstack-infra | 18:21 | |
| *** pcrews has quit IRC | 18:28 | |
| mtaylor | #status ok devstack gate should be back to normal | 18:28 |
| *** mtaylor has quit IRC | 18:28 | |
| *** mtaylor has joined #openstack-infra | 18:28 | |
| mtaylor | #status ok devstack gate should be back to normal | 18:29 |
| mtaylor | hrm | 18:29 |
| *** vogxn has quit IRC | 18:29 | |
| mtaylor | why statusbot no like me? | 18:30 |
| *** pabelanger has joined #openstack-infra | 18:31 | |
| clarkb | you need to be mordred | 18:31 |
| *** mriedem has joined #openstack-infra | 18:35 | |
| jeblair | #status ok devstack gate should be back to normal | 18:38 |
| openstackstatus | NOTICE: devstack gate should be back to normal | 18:38 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer Infrastructure | docs http://ci.openstack.org | bugs https://launchpad.net/openstack-ci/+milestone/grizzly | https://github.com/openstack-infra/config" | 18:38 | |
| *** dripton has quit IRC | 18:41 | |
| *** dripton has joined #openstack-infra | 18:44 | |
| dstufft | mtaylor: sorry just woke up, whats up? | 18:49 |
| *** jjmb has quit IRC | 18:51 | |
| *** linuxmohan has quit IRC | 18:52 | |
| mtaylor | dstufft: oh, all sort of stuff | 18:53 |
| mtaylor | oh! why am I mtaylor? | 18:53 |
| *** mtaylor is now known as mordred | 18:53 | |
| mordred | weird | 18:53 |
| Alex_Gaynor | mordred: that's a bit of an existential question, dont' ya think? | 18:54 |
| mordred | dstufft: fascinating interaction between distro-installed old distribute and upgraded setuptools via pip | 18:54 |
| *** linuxmohan has joined #openstack-infra | 18:54 | |
| dstufft | mordred: your client realized the supremacy of first initial + last name | 18:54 |
| mordred | dstufft: and by fascinating, I do not actually mean fascinating | 18:54 |
| mordred | heh | 18:54 |
| dstufft | oh goody | 18:54 |
| dstufft | clashes | 18:54 |
| mordred | it's SUPER exciting | 18:55 |
| mordred | dstufft: https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/34083/consoleText | 18:55 |
| mordred | dstufft: fwiw | 18:55 |
| mordred | dstufft: we've worked around it for now | 18:55 |
| mordred | but you'll see near the end that installing warlock fails when because setup.py egg_info goes to hell | 18:56 |
| dstufft | Hum | 18:56 |
| dstufft | That's normally indicative that setuptools went awol | 18:57 |
| dstufft | Sicne that comes from setuptools | 18:57 |
| dstufft | also | 18:57 |
| dstufft | /usr/local/lib/python2.7/dist-packages/d2to1/core.py:67: UserWarning: Unknown distribution option: 'install_requires' | 18:57 |
| dstufft | and the others | 18:57 |
| dstufft | you were using plain old distutils… for some reason | 18:57 |
| *** jjmb has joined #openstack-infra | 18:58 | |
| *** UtahDave has quit IRC | 19:02 | |
| fungi | wow, i step away to pack and run pre-travel errands and miss all the fun! | 19:03 |
| fungi | i seem to pick just the wrong (right?) times to be afk | 19:03 |
| mordred | dstufft: well, we weren't doing that on purpose | 19:04 |
| dstufft | mordred: yea not saying you were | 19:05 |
| mordred | :) | 19:05 |
| mordred | dstufft: in a venv with latest setuptools it works, and it works without upgrading setuptools, and it works if I apt-get remove python-setuptools | 19:05 |
| * mordred stabs self in eyeholes | 19:05 | |
| dstufft | is apt-get python-setuptools actually distribute | 19:06 |
| mordred | yes | 19:07 |
| Alex_Gaynor | amazing | 19:07 |
| fungi | after i slept on last night's mirroring behavior change, it dawned on me that we probably only ended up with older and newer versions of stuff in or depended on by openstack/requirements, and with the current wisdom of removing version caps wherever we can as early as we can i'm unconvinced it's as big hole in our gating as i first believed | 19:08 |
| dstufft | I wonder if upgrading distribute first would fix it | 19:08 |
| dstufft | lol fucking shitshows | 19:08 |
| dstufft | and people wonder why I want pip to just vendor setuptools | 19:09 |
| Alex_Gaynor | dstufft: has setuptools/commands/easy_install been deleted yet? | 19:10 |
| dstufft | No | 19:10 |
| * mordred cires | 19:10 | |
| mordred | cries | 19:10 |
| mordred | setuptools/commands/easy_install should die in a fire in a fire while dying in a fire | 19:11 |
| mordred | also I don't like it | 19:11 |
| dstufft | There is a PR to make pip not depend on setuptools | 19:12 |
| dstufft | sort of | 19:12 |
| fungi | mordred: while we're on the subject of python packaging breakage, the exec we have at https://github.com/openstack-infra/config/blob/master/modules/jeepyb/manifests/init.pp#L68 is failing on at least our gerrit servers with this error... http://paste.openstack.org/show/41008/ | 19:12 |
| fungi | trying to figure out if we need to change the exec or if that's indicative of something else gone awry on the servers | 19:13 |
| fungi | seems to have started breaking when the pbrization change merged: https://review.openstack.org/31439 | 19:15 |
| *** hashar has joined #openstack-infra | 19:17 | |
| *** ladquin has joined #openstack-infra | 19:21 | |
| fungi | so a little over two weeks now | 19:36 |
| *** linuxmohan has quit IRC | 19:40 | |
| *** SergeyLukjanov has quit IRC | 19:42 | |
| mordred | fungi: fuckit | 20:13 |
| mordred | fungi: that's going to be an old version of pbr | 20:13 |
| mordred | on the box | 20:13 |
| fungi | aha | 20:13 |
| fungi | so we need to pip install latest pbr i guess? | 20:13 |
| mordred | fungi: or we need to get an apt repo that we fill with things that we produce | 20:14 |
| mordred | fungi: that we use for our server | 20:14 |
| mordred | servers | 20:14 |
| fungi | should jeepyb require a newer pbr then? | 20:14 |
| mordred | maybe so | 20:15 |
| fungi | oh, right, we don't install the dependencies via pip | 20:15 |
| mordred | I don't thin that will accomplish what you might think it will though | 20:15 |
| fungi | so in the short term, we need to have puppet install a new enough pbr on those servers | 20:16 |
| fungi | probably in the jeepyb puppet module | 20:16 |
| fungi | i | 20:16 |
| fungi | er, i'll propose something to get us past this for the moment so jeepyb starts updating there again | 20:16 |
| mordred | dammit | 20:30 |
| mordred | https://jenkins.openstack.org/job/periodic-mirror-python27/146/console | 20:30 |
| mordred | also | 20:30 |
| mordred | didn't we add libffi-dev to the slaves? | 20:30 |
| mordred | http://logs.openstack.org/00/38000/2/check/gate-pbr-devstack-vm-rawinstall/232/console.html.gz | 20:30 |
| mordred | isrunning on a devstack node | 20:30 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Add libffi-dev https://review.openstack.org/38049 | 20:36 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Replace entry_points console_scripts https://review.openstack.org/38000 | 20:36 |
| mordred | dhellmann_: totally agree on tests... | 20:36 |
| mordred | dhellmann_: any ideas on what we should do for the second one? | 20:37 |
| *** dhellmann_ is now known as dhellmann | 20:37 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Serve IRC log files as MIME type text/plain. https://review.openstack.org/38050 | 20:37 |
| dhellmann | well... | 20:37 |
| mordred | dhellmann: I mean, testing the setuptools override seems simple enough | 20:38 |
| dhellmann | how about an integration test that packages a dummy app, installs it in a venv, and then runs the script? | 20:38 |
| mordred | hrm | 20:38 |
| mordred | we could do that | 20:38 |
| dhellmann | we have some shell tests for pbr already, right? | 20:38 |
| mordred | a very large one | 20:38 |
| dhellmann | yeah, I promised to split that up using shunit2 and it's still on my list | 20:39 |
| mordred | :) | 20:39 |
| dhellmann | mordred: have you seen pbr emit a message like "'e' not a regular file -- skipping" when creating an sdist? | 20:50 |
| dhellmann | I get that for several letters, apparently alphabetically | 20:50 |
| *** linuxmohan has joined #openstack-infra | 20:50 | |
| dhellmann | but not *every* letter | 20:51 |
| *** linuxmohan has quit IRC | 20:56 | |
| *** mriedem has quit IRC | 21:04 | |
| dhellmann | ah, found it, I had the "scripts" list in the wrong part of the metadata | 21:09 |
| *** hashar has quit IRC | 21:12 | |
| mordred | dhellmann: hehe | 21:12 |
| mordred | nice | 21:12 |
| dhellmann | yeah, it turns out the letters were sorted(set('virtualenvwrapper.shvirtualenvwrapper_lazy.sh')) | 21:13 |
| dhellmann | if d2to1 is going to be picky about where things show up in the input file, it should not use the values from the wrong places :-/ | 21:14 |
| Alex_Gaynor | mordred: Somedays I wish openstack didn't use its own pypi server, so people could see how dumb download counts are on pypi. | 21:14 |
| mordred | Alex_Gaynor: heh | 21:14 |
| mordred | dhellmann: we can fix that now | 21:14 |
| dhellmann | yep | 21:14 |
| mordred | dhellmann: god. there is no great place from which to test this | 21:16 |
| mordred | dhellmann: I can create a fake python package and shell out to python setup.py | 21:17 |
| mordred | which seems to be the _best_ option | 21:17 |
| mordred | which makes me want to die inside | 21:17 |
| dhellmann | or write another bash script that does it, and I'll convert it along with the existing script? | 21:17 |
| dhellmann | that's what I meant before | 21:17 |
| mordred | yeah - I was just trying to figure out how to do this with, you know, a unit test | 21:19 |
| mordred | dhellmann: but I think adding it to the integration test will be the best bed | 21:20 |
| mordred | bet | 21:20 |
| dhellmann | yeah :-/ | 21:20 |
| *** changbl has quit IRC | 21:31 | |
| *** dhellmann is now known as dhellmann_ | 21:40 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Replace entry_points console_scripts https://review.openstack.org/38000 | 21:48 |
| mordred | dhellmann_: ok. there's tests ^^ | 21:48 |
| *** dhellmann_ is now known as dhellmann | 21:58 | |
| * dhellmann wonders if mordred gets extra points for rolling the odometer over on the review # | 21:58 | |
| Alex_Gaynor | dhellmann: not until 40k | 21:59 |
| dhellmann | haha | 21:59 |
| Alex_Gaynor | dhellmann: and if diff 40k isn't a warhammer joke, vengene be upon the uploader | 21:59 |
| dhellmann | Alex_Gaynor: that sounds like something worth ensuring | 22:00 |
| dhellmann | write something to watch gerrit events for patch 39999 and automatically submit one? | 22:00 |
| *** sdake has quit IRC | 22:11 | |
| mordred | haha | 22:14 |
| mordred | btw - that's just a whole lot of patches | 22:14 |
| Alex_Gaynor | 40k, or the 2k more to get there? | 22:14 |
| mordred | 40k | 22:16 |
| mordred | although gerrit's gerrit has 48k and android's has 62k ... so we've got some catching up to do still | 22:16 |
| Alex_Gaynor | those are much older, aren't they? | 22:17 |
| Alex_Gaynor | And the acceleration probably isn't as high. | 22:17 |
| mordred | they are. android has been around for quite a while :) | 22:18 |
| Alex_Gaynor | dhellmann: FWIW motivation for that tweet was https://review.openstack.org/#/c/38014/ and me being gnerally angry at py2k :) | 22:18 |
| *** dhellmann is now known as dhellmann_ | 22:18 | |
| dhellmann_ | Alex_Gaynor: I like the -1 without any comment | 22:19 |
| Alex_Gaynor | dhellmann_: Peter and I discussed it for a fwe minutes on IRC last night, I assume he just didn't want it merged without a discussion but it was too late for him to articulate a better direction | 22:20 |
| *** dhellmann_ is now known as dhellmann | 22:20 | |
| * dhellmann can't keep up with his IRC status | 22:20 | |
| dhellmann | Alex_Gaynor: ah, ok | 22:20 |
| *** holdenweb has joined #openstack-infra | 22:23 | |
| *** dkliban has joined #openstack-infra | 22:26 | |
| *** holdenweb has quit IRC | 22:40 | |
| *** dhellmann is now known as dhellmann_ | 22:44 | |
| *** jjmb has quit IRC | 22:45 | |
| *** jjmb has joined #openstack-infra | 22:45 | |
| *** ewindisch has joined #openstack-infra | 23:25 | |
| *** ewindisch has quit IRC | 23:26 | |
| *** CaptTofu has joined #openstack-infra | 23:49 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!