| clarkb | nati_ueno: markmcclain was working that one by fixing the problem in novaclient iirc | 00:00 |
|---|---|---|
| nati_ueno | clarkb: so mark was working on that to update remove cap for nova | 00:00 |
| nati_ueno | clarkb: yes that's one | 00:00 |
| clarkb | ya, not sure if there was a bug for that | 00:00 |
| lifeless | jeblair: if you want to avoid a new C binary, we could use lmirror, which is python and has a wsgi server that streams quite efficiently. | 00:00 |
| clarkb | fungi may know as he was involved with the requests pinning | 00:00 |
| lifeless | jeblair: (just jumping back to one of your other comments) | 00:00 |
| lifeless | jeblair: that would run over https too and that might avoid concerns about folk mitming the content) | 00:00 |
| nati_ueno | clarkb: Thanks. fungi: do you know something? | 00:01 |
| jeblair | lifeless: disregarding my concerns (which are more around risk/reward, and i'm coming over to understanding the reward part), which do you think would be better? | 00:02 |
| fungi | clarkb: nati_ueno: the reason we had originally pinned requests in nova is that nova was tested with site-packages enabled, nova depended on another openstack project which depended on requests. requests 1.2.1 pulled in sphinx as a new dependency which in turn depended on something else which depended on a newer jinja2 than the globally-installed rpm on centos 6.4 | 00:02 |
| *** CaptTofu has quit IRC | 00:02 | |
| *** CaptTofu has joined #openstack-infra | 00:03 | |
| fungi | that's as "briefly" as i can summarize the original issue | 00:03 |
| clarkb | fungi: that sentence is full of ugh | 00:03 |
| fungi | switching tests to pip install -U dependencies solved this because we started pulling in a sufficiently new jinja2 in the virtualenv and were able to stop capping requests in nova as a result | 00:04 |
| *** CaptTofu has quit IRC | 00:04 | |
| fungi | nova never directly depended on requests anyway | 00:05 |
| fungi | or hadn't prior to sticking it in as a workaround for that specific situation | 00:05 |
| *** CaptTofu has joined #openstack-infra | 00:05 | |
| nati_ueno | fungi: thanks. Do you have bug report for this issue? I wanna know it is solved or not | 00:05 |
| jeblair | lifeless: yes, we can certainly revisit that decision, and i think you make a compelling case that the tripleo program has a real general need for this. | 00:06 |
| fungi | nati_ueno: there's a chain from the requirements cap revert to the nova cap revert to the original patch which closed the bug. i'll hunt it down | 00:06 |
| jeblair | lifeless: i'm now on board with running rsync or lmirror as appropriate, and segregating the hosts from static.o.o as needed for improved security. | 00:07 |
| nati_ueno | fungi: Thanks. so is the change finished? I see mark's patch is merged | 00:07 |
| clarkb | nati_ueno: which patch merged? | 00:08 |
| jeblair | lifeless: so i'd just want to make sure the other coremudgeons agree with the scope expansion, though it seems like they have been. :) | 00:08 |
| clarkb | I think one thing rsync has going for it is the puppet module... if we start using https on static I can see lmirror being useful | 00:09 |
| fungi | nati_ueno: it | 00:09 |
| fungi | er | 00:09 |
| fungi | nati_ueno: it's bug 1182271 | 00:09 |
| uvirtbot | Launchpad bug 1182271 in nova "Nova unit tests fail on CentOS 6 when python-jinja2 package is installed" [Medium,In progress] https://launchpad.net/bugs/1182271 | 00:09 |
| jeblair | this does dovetail into the earlier https for pypi.o.o discussion | 00:09 |
| nati_ueno | fungi: Thansk! | 00:09 |
| *** dina_belova has joined #openstack-infra | 00:09 | |
| locke105 | pip forces SSL? | 00:11 |
| nati_ueno | clarkb: sorry my mistake. It looks it is not merged yet https://review.openstack.org/#/c/40483/ | 00:11 |
| locke105 | (sorry was reading HTTPS discussion) | 00:11 |
| clarkb | locke105: yes, after it became an internet meme that pip was insecure (I think everyone sort of knew about this before hand...) dstufft and I assume the other pypi/pip folks worked to fix it | 00:12 |
| clarkb | locke105: it did other bad things like not enfoce md5sum checks | 00:12 |
| *** CaptTofu has quit IRC | 00:12 | |
| locke105 | my internal "mirror" doesn't use that... | 00:12 |
| locke105 | HTTPS that is | 00:12 |
| clarkb | locke105: s/pip/pypi/ | 00:13 |
| *** CaptTofu has joined #openstack-infra | 00:13 | |
| locke105 | oooh | 00:13 |
| locke105 | yeah that makes sense | 00:13 |
| locke105 | my squid busted after they did that | 00:13 |
| locke105 | which is why i am rebuilding a full internal mirror instead of some packages | 00:13 |
| fungi | pip doesn't enforce https, pypi now redirects from http to https as of a month or so | 00:13 |
| fungi | ahh, right | 00:13 |
| clarkb | jeblair: I think there is value in https everywhere. We could create our own CA so that nodes trust each other properly while still self signing | 00:13 |
| * locke105 hates dealing with certs | 00:14 | |
| jeblair | clarkb: i am in favor of https everywhere; i'm less in favor of filing expense reports for buying certs for transient servers with numbers in their names. :/ | 00:14 |
| fungi | clarkb: mordred already suggested that once today. having managed certificate authorities before, i'm glad you two are volunteering to run this one instead ;) | 00:14 |
| jeblair | thus the current 'https most places'. | 00:14 |
| *** dina_belova has quit IRC | 00:14 | |
| clarkb | fungi: oh man I raelly don't want to manage a CA >_> | 00:15 |
| locke105 | yep... | 00:15 |
| clarkb | but it is a possibility | 00:15 |
| locke105 | although the nova-cert service theorectically turns most of it into an API | 00:15 |
| locke105 | its kind of cool | 00:15 |
| jeblair | if we expand the scope of pypi as lifeless is suggesting, that increases my desire for it to have a ca-signed cert | 00:15 |
| fungi | the mechanics of ca management aren't that bad actually. easy enough to script. doing proper due diligence to secure your ca key is another matter entirely | 00:16 |
| locke105 | yeah thats true | 00:16 |
| clarkb | jeblair: I agree for pypi.o.o scope changing wants a proper cert | 00:16 |
| locke105 | what exactly is the scope change? | 00:17 |
| *** weshay has joined #openstack-infra | 00:17 | |
| *** emagana has joined #openstack-infra | 00:17 | |
| clarkb | locke105: providing this as a service to downstreams outside of our infrastructure | 00:17 |
| locke105 | ah... | 00:18 |
| clarkb | locke105: so that people doing openstack dev/deployments/whatever can use our mirror instead of pypi | 00:18 |
| locke105 | i probably would still need to maintain an internal mirror for bandwidth reasons | 00:18 |
| ianw | harlowja: do you use the -p command on ever step? | 00:18 |
| harlowja | nah, its actually cached :) | 00:18 |
| *** reed has quit IRC | 00:18 | |
| harlowja | just the first one | 00:18 |
| jeblair | locke105: lifeless wants to make that easier for you by supporting rsync or something like it. | 00:18 |
| locke105 | yeah that would kick ass | 00:18 |
| harlowja | ianw /etc/anvil/passwords.cfg , yes its not secure, but oh well, ha | 00:18 |
| clarkb | fungi: jeblair: https://review.openstack.org/#/c/40456/1/modules/openstack_project/files/gerrit/acls/stackforge/puppet-modules.config does /refs/heads/* not match /regs/heads/stable/* ? | 00:19 |
| harlowja | sorry i mean /etc/anvil/settings.yaml , the passwords should be somewhat secure, ha | 00:19 |
| fungi | clarkb: /regs/heads/stable/* is a more specific match and is present in all-projects so needs to be overridden there | 00:19 |
| fungi | er, refs | 00:20 |
| jeblair | clarkb: the exclusive on global refs/heads/stable matches more closely, so it needs an even more closer match (project is closer than all-projects) | 00:20 |
| ianw | hmm, ok, install step is failing to find openstack-cinder,cinder==2013.2.a154.g8aac388, openstack-glance,glance==2013.2.a54.g463f322, ... and pretty much everything else | 00:20 |
| jeblair | what fungi said :) | 00:20 |
| *** salv-orlando has quit IRC | 00:20 | |
| fungi | jeblair said it better | 00:20 |
| clarkb | I see, /me looks at all projects | 00:20 |
| harlowja | ianw u did, ./smithy -a prepare, ./smithy -a build, ./smithy -a install right | 00:21 |
| harlowja | no skipping :-p | 00:21 |
| *** nijaba has quit IRC | 00:21 | |
| locke105 | i have noticed that with my mirror lagging behind on reqs sometimes, we catch a lot of bugs | 00:21 |
| ianw | no skipping :) | 00:21 |
| fungi | clarkb: it's our blanket rule to grant the stable release managers exclusive control to those branches | 00:21 |
| jeblair | clarkb: there ought to be similar existing samples | 00:21 |
| harlowja | $ ls /etc/yum.repos.d/anvil* | 00:21 |
| jeblair | harlowja: wow that's really close to "smitty" and i was having aix flashbacks | 00:21 |
| clarkb | jeblair: there are. It just occurred to me that it shouldn't be necessary but the blanket rule explains it | 00:21 |
| harlowja | ianw ^ ? | 00:21 |
| harlowja | lol | 00:21 |
| *** nijaba has joined #openstack-infra | 00:22 | |
| harlowja | smithy is anvils blacksmith jeblair , ha | 00:22 |
| *** derekh has quit IRC | 00:22 | |
| ianw | harlowja: not found ... so hmm | 00:22 |
| fungi | jeblair: good flashbacks! filesystem/block device management with smitty was great | 00:22 |
| lifeless | jeblair: sorry, ELOCAL. | 00:22 |
| harlowja | hmmmm, ianw wanna sudo rm -rf ~/openstack then do the steps again | 00:23 |
| harlowja | those 2 repo files should exist after the build step | 00:23 |
| ianw | harlowja: hang on, i think i *did* forget build | 00:23 |
| harlowja | :) | 00:23 |
| harlowja | the install step actually verifies that all packages it wants to install are there, its the late catch anything missing thing | 00:23 |
| ianw | yeah, sorry, lost track | 00:24 |
| openstackgerrit | A change was merged to openstack-infra/config: Allow puppet-manager-core to review stable branches https://review.openstack.org/40456 | 00:24 |
| harlowja | np :) | 00:24 |
| harlowja | https://github.com/stackforge/anvil/blob/master/anvil/packaging/yum.py#L674 | 00:24 |
| lifeless | jeblair: uhm, so I think the set of functional requirements we want: - streaming; no content alteration by MITMers; multi-concurrent-writer-safe. | 00:24 |
| locke105 | smitty can die in a fire along with AIX and its crappy network device model | 00:24 |
| harlowja | lol | 00:24 |
| lifeless | jeblair: I designed lmirror for single writer millions-of-files, it's not a great fit here - only mentioned it due to your C daemon comment. | 00:25 |
| harlowja | ianw there was a time ago that anvil would run the dependent operations, but it was just split into these steps, for better or worse :-p | 00:25 |
| harlowja | time long ago, ha | 00:25 |
| ianw | smithy --help giving an overview of the steps might be good | 00:26 |
| ianw | i think i forgot because i had the rst of the docs in less | 00:26 |
| ianw | and it all looks the same :) | 00:26 |
| harlowja | https://anvil.readthedocs.org/en/latest/topics/gettingstarted.html#pre-installing | 00:27 |
| harlowja | ;) | 00:27 |
| harlowja | but ya smithy --help would make sense | 00:27 |
| jeblair | hrm, just thought of something else... | 00:28 |
| jeblair | supporting rsync makes it more difficult to store the mirror in object storage (and front with, eg a wsgi app), if we wanted to go that way. | 00:28 |
| lifeless | jeblair: so I think stunnel in inetd rather than rsyncd running; use rsync -e stunnel ... to connect to the rsync server. | 00:29 |
| clarkb | jeblair: it does | 00:29 |
| lifeless | jeblair: it does, thats true; how many thousand files is the mirror? | 00:29 |
| jeblair | but if we did that, we could more easily provide the (accurate) index files lifeless originally wanted | 00:29 |
| lifeless | jeblair: so I only went to index files after talking to clarkb here. | 00:29 |
| lifeless | jeblair: rsync was my first thought, I think :) | 00:30 |
| jeblair | find -type f |wc -l | 00:30 |
| jeblair | 890 | 00:30 |
| clarkb | I didn't like rsync without more discussion for the reasons jeblair describes | 00:30 |
| lifeless | jeblair: teeny :) | 00:30 |
| jeblair | lifeless: expact that to quintuple due to added branches. | 00:30 |
| clarkb | I did however think hosting a few more files would be fine ... | 00:30 |
| jeblair | but yes, still teeny. | 00:30 |
| lifeless | jeblair: so we could always stash that in object storage as well, just a cron job on the host to sync regularly; the size is small enough that that wouldn't have scaling/overhead issues. | 00:31 |
| *** zul has quit IRC | 00:32 | |
| clarkb | I am going to walk home now. Back later | 00:32 |
| jeblair | lifeless: i think if we used object storage, we wouldn't want to support rsync because we'd need the whole tree locally; we'd want to use object storage exclusive of local storage (except for caching) | 00:32 |
| lifeless | jeblair: I'm saying that the 'whole tree locally' only becomes an issue at scale; at a scale we aren't at. | 00:33 |
| clarkb | crazy idea before I go: I believe swift allows you to replicate containers. Might be possible to do something with that? | 00:33 |
| lifeless | jeblair: [and have no reason to think we'll reach for a long time] | 00:33 |
| jeblair | lifeless: well, its absence will be an issue. | 00:33 |
| lifeless | jeblair: I don't follow | 00:33 |
| clarkb | replicate to lifeless' container and have him do what he wants? it will be an auth nightmware though | 00:33 |
| jeblair | because i really don't want to have an authoritative copy on object storage and one locally. this is supposed to make things simpler. | 00:34 |
| lifeless | jeblair: in that if the scale is small enough one can make a local tree in cron on a cloud server | 00:34 |
| Alex_Gaynor | mordred: man, today is just bringing first class bummers, nonstop | 00:35 |
| jeblair | lifeless: yes, and rsync is inaccurate until the cron runs; it may be empty when the server spins up, and different servers may return different values. | 00:35 |
| lifeless | jeblair: I see; I presumed we didn't put things into rotation until they'd set themselves up. | 00:36 |
| *** sandywalsh has quit IRC | 00:36 | |
| jeblair | lifeless: having dumber application servers that rely entirely on cloud services is a direction we'd like to head. it's going to take a while to get there, but some of the static content servers are serious candidates. logs or talballs may be first. pypi could be on the list. | 00:36 |
| lifeless | jeblair: ack on the skew between cron window thing, but the empty dir case is much more of an issue | 00:36 |
| lifeless | jeblair: there's no in-principle reason that swift can't support rsync. Heck, S3 supports bittorrent. | 00:37 |
| jeblair | lifeless: we haven't engineered the system yet, but i'd certainly want to keep it as simple as possible. | 00:38 |
| locke105 | i think the problem is that we want to rsync a tree of objects | 00:38 |
| locke105 | not just the objects | 00:38 |
| locke105 | swift uses/used rsync for object sync already | 00:39 |
| locke105 | the problem we have is the one git solves | 00:39 |
| clarkb | jeblair ++ | 00:39 |
| locke105 | only in git, the backing store is gzipped files... | 00:40 |
| jeblair | i have to run now | 00:40 |
| fungi | there's an idea... publish the pypi mirror via bt | 00:41 |
| locke105 | i'm not sure how that would work with a backing object store and wsgi frontend still... | 00:43 |
| locke105 | plus if the file data changes you need to build a new torrent | 00:44 |
| locke105 | where did the object store requirement come from? | 00:46 |
| lifeless | fungi: I would be happy with that too :) | 00:46 |
| fungi | locke105: the object store benefit is something we've discussed in the past. being able to turn up a new server and just point it at the data in a provider rather than needing to restore copies from backup when replacing one | 00:48 |
| dkehn | clarkb, well the devstack with neutron looks worthless again | 00:48 |
| locke105 | so use the list of objects in a container as your index | 00:48 |
| fungi | yep | 00:48 |
| locke105 | and the objects are the tarballs or zips etc | 00:48 |
| locke105 | yeah i like that | 00:48 |
| harlowja | ianw let me know how it goes :) | 00:49 |
| harlowja | hopefully ok this time, ha | 00:49 |
| fungi | but that's a long-term possibility for our static and semi-static content | 00:49 |
| locke105 | i believe that is how pypi.crate.io works | 00:49 |
| locke105 | serves out of a cdn with wsgi | 00:49 |
| fungi | near term we're still serving off filesystems on local block devices | 00:49 |
| fungi | mainly because we haven't had time to reimplement something browseable like apache's autoindex | 00:50 |
| locke105 | right you need something to do the translation from object store to file-like system | 00:50 |
| locke105 | can't imagine it would be too hard to hack up | 00:51 |
| fungi | it's not a super-complex problem, but we have plenty of other places we need to spend our time improving higher priority things | 00:51 |
| locke105 | yeah i have the same issue :( | 00:51 |
| locke105 | hosting the current file set with git or rsync would be easiest | 00:53 |
| fungi | huh, git's a possibility i hadn't considered. it's not great for larger binary blobs though so it would need to be something more like bup | 00:54 |
| locke105 | yeah you guys said mirror + rsync and my mind jumped to the Gentoo portage tree | 00:54 |
| locke105 | which uses rsync to mirror around | 00:55 |
| locke105 | they have been trying to move it to Git for some time... | 00:55 |
| locke105 | thats a lot of tiny files | 00:55 |
| locke105 | not binaries | 00:55 |
| locke105 | git would work well, since it would only grab the latest set in the tree | 00:56 |
| locke105 | if you did shallow clones | 00:56 |
| locke105 | but at the same time we don't want to keep the large binaries in history | 00:56 |
| fungi | that's basically what bup does, but it's not plain git (so that it can optimize for those issues) | 00:57 |
| locke105 | never heard of bup... looking into it now... | 00:58 |
| fungi | we're in the process of rolling it out for server backups | 00:58 |
| fungi | it's a very neat implementation | 00:58 |
| locke105 | It uses a rolling checksum algorithm (similar to rsync) to split large files into chunks. The most useful result of this is you can backup huge virtual machine (VM) disk images, databases, and XML files incrementally, even though they're typically all in one huge file, and not use tons of disk space for multiple versions. | 00:59 |
| locke105 | first bullet point and i'm sold | 00:59 |
| lifeless | locke105: the pypi CDN is backed by a regular server with a regular filesystem. | 00:59 |
| lifeless | locke105: it's just an aggressive HTTP cache in front of everything. | 00:59 |
| locke105 | the crate.io one? | 00:59 |
| locke105 | not python.org | 00:59 |
| lifeless | oh, sorry, not sure. | 01:00 |
| lifeless | but IIUC dstufft is moving the python.org one onto the same codebase. | 01:00 |
| lifeless | so, same same. | 01:00 |
| locke105 | ah | 01:00 |
| dstufft | it's a whole new codebase | 01:00 |
| dstufft | the crate.io codebase sucks | 01:00 |
| locke105 | lol | 01:01 |
| dstufft | first systems etc | 01:01 |
| dstufft | I've learned a ton since I wrote it | 01:01 |
| dstufft | lots of mistakes | 01:01 |
| fungi | proof of concept ;) | 01:01 |
| dstufft | basically | 01:01 |
| lifeless | fungi: jeblair: so, I'd be happy with a few-minutes-delayed rsyncable tree. | 01:01 |
| locke105 | +1 | 01:01 |
| dstufft | the PyPI CDN is basically just varnish | 01:01 |
| fungi | like when you're making pancakes, the first one never looks as good as the rest | 01:01 |
| lifeless | all the systems that need latest-and-greatest could happily run off an object storage container. | 01:01 |
| lifeless | Both would be eventually consistent given the cache properties of object stores. | 01:02 |
| lifeless | so - my suggestion is stunnel in inetd wired up to invoke rsync - getting us end to end encryption. | 01:02 |
| locke105 | tbh my mirror lags behind further than a few minutes (more like days) and its fine | 01:02 |
| lifeless | I don't know if the rsync module from puppet labs does that. | 01:02 |
| dstufft | But yea Crate stores it's files in S3 (and PyPI will eventually too) | 01:03 |
| dstufft | or rather not S3 for PyPI, but an object store | 01:03 |
| clarkb | dstufft: are you going to host your own? | 01:03 |
| dstufft | clarkb: probably, our infra lead doesn't like depending on external things if we can easily host it ourselves | 01:03 |
| fungi | lifeless: we could wire up stunnel to rsync fairly easily on a second port | 01:03 |
| lifeless | FWIW lmirror could happily back onto S3 - it has a simple file based journal that clients can interpret; you could run the server against it. | 01:03 |
| lifeless | fungi: or that, whatever you prefer. | 01:04 |
| dstufft | clarkb: We've talked about using Ceph or Swift, and we want to switch to OpenStack for our ifnra anyways | 01:04 |
| dstufft | right now using Ganeti because OSUOL recommended it when we setup there | 01:05 |
| fungi | dstufft: simultaneously stick copies in several swift providers and that way you can weather outages of a provider | 01:05 |
| clarkb | dstufft: ya its what Lance prefers | 01:05 |
| clarkb | ganeti is nice and simple | 01:05 |
| Alex_Gaynor | fungi: "why would a highly available object store ever go down" (snrk) | 01:05 |
| * locke105 googles ganeite | 01:06 | |
| dstufft | I don't have access to Ganeti so I don't know anything about it | 01:06 |
| fungi | Alex_Gaynor: i have seen what happens when a tornado cuts a top-notch data center in half | 01:06 |
| * locke105 tries again with correct spelling... | 01:06 | |
| clarkb | unlike openstack it is highly opinionated about what your setup should look like | 01:06 |
| dstufft | other than it doesn't support LXC and we want to use LXC for isolation | 01:06 |
| clarkb | iirc KVM only with only local storage for VM images | 01:06 |
| Alex_Gaynor | fungi: I'm curious how swifts new global replication would weather a tornado | 01:07 |
| clarkb | that makes it simple to setup configure etc | 01:07 |
| dtroyer | ganeti is great for smallish relatively static clusters…with live drbd-based failover | 01:07 |
| dtroyer | s/failover/migration/ | 01:07 |
| *** sandywalsh has joined #openstack-infra | 01:07 | |
| fungi | Alex_Gaynor: yes, globally distributed in a single provider is almost as good as distributing across multiple providers | 01:07 |
| clarkb | dstufft: I am guessing the fact that a majority of their employees are students helps influence those types of decisions | 01:07 |
| harlowja | ianw https://review.openstack.org/#/c/40529/ :) | 01:08 |
| *** weshay has quit IRC | 01:09 | |
| dstufft | clarkb: probably yea, There was some misconceptions about how much OSOUL was going to actively do so we'll likely be moving to something our members have more experience with and know people to complain at when things break ;) | 01:10 |
| dstufft | plus written in python! | 01:10 |
| *** dina_belova has joined #openstack-infra | 01:10 | |
| locke105 | +2 | 01:10 |
| dtroyer | ganeti is python | 01:10 |
| locke105 | looks like a lot of .hs files... | 01:12 |
| locke105 | and .pys | 01:12 |
| locke105 | weird | 01:12 |
| dstufft | dtroyer: welp I refer to the above I don't know anything about it other than it doesn't support LXC :V | 01:13 |
| dtroyer | dstufft: no, and the fact that it doesn't use libvirt makes that a more difficult problem to fix. but it was rock solid for me for a couple of years running basic infra stuff. it solves a basic virt manager problem, not clouds | 01:15 |
| *** dina_belova has quit IRC | 01:15 | |
| clarkb | dtroyer: exactly | 01:15 |
| clarkb | it is actually a pretty awesome tool, just different | 01:15 |
| clarkb | one of the OSUOSLers wrote the web front end for it during an internship at google iirc. which is how it first ended up in their infra iirc | 01:16 |
| dstufft | Yea I'm saying it's bad I don't have any data to claim that :) I just know we're looking at moving to OpenStack in general | 01:16 |
| locke105 | tbh... who isn't looking at moving to openstack... | 01:17 |
| *** mestery_ has joined #openstack-infra | 01:17 | |
| locke105 | i have a meeting tomorrow with some internal labops guys that want to know how to set one up | 01:18 |
| fungi | i always keep an extra openstack in my back pocket. you never know when you might need one | 01:19 |
| *** mestery has quit IRC | 01:19 | |
| locke105 | fungi: ++ i have around 3 or 4 now... | 01:19 |
| locke105 | they tend to be pretty popular and fill up fast :) | 01:20 |
| locke105 | i wish i could get stats of ubuntu/rhel installs with openstack vs ESXi in our labs and how they have changed over the past year | 01:21 |
| *** nijaba has quit IRC | 01:22 | |
| *** nijaba has joined #openstack-infra | 01:22 | |
| *** nijaba has quit IRC | 01:22 | |
| *** nijaba has joined #openstack-infra | 01:22 | |
| clarkb | locke105: glance/nova doesn't keep track? | 01:25 |
| locke105 | clarkb: idk. how? | 01:26 |
| clarkb | locke105: I am guessing logs | 01:27 |
| locke105 | oh | 01:28 |
| clarkb | which you could then process and spit numbers out to graphite with | 01:28 |
| locke105 | the lab hardware is either xcat provisioned or manually provisioned (yes sad panda i know) | 01:28 |
| locke105 | thats what i would be interested in getting stats off of | 01:28 |
| locke105 | i could probably whip up some xcat magic | 01:29 |
| fungi | btw, reinstalling python-pip on all the centos servers seems not to have rebroken after the puppet update circulated | 01:30 |
| clarkb | hurray | 01:30 |
| fungi | though pbx and centos6-dev1 have separate issues on top of that so they're still erroring | 01:30 |
| jog0 | clarkb: where is the code for the gerrit irc bot? | 01:30 |
| jog0 | I wantto make it less wordy | 01:31 |
| fungi | jog0: openstack-infra/gerritbot | 01:31 |
| jog0 | fungi: thanks | 01:31 |
| clarkb | jog0: less wordy? | 01:31 |
| clarkb | jog0: its already only one message per announcement | 01:31 |
| clarkb | whcih is what matters in IRC | 01:31 |
| jog0 | I have word wrap :( | 01:31 |
| jog0 | as do others I bet | 01:31 |
| clarkb | I do... doesn't bother me | 01:32 |
| jog0 | clarkb: I find it breaks up conversations sometimes | 01:33 |
| clarkb | maybe a client thing? | 01:33 |
| jog0 | clarkb: I was thinking instead of %s propsed a change to %s ... | 01:34 |
| jog0 | just listing everything | 01:34 |
| *** gyee has quit IRC | 01:35 | |
| jog0 | mayvbe %s proposed %s ... | 01:35 |
| *** yaguang has joined #openstack-infra | 01:35 | |
| clarkb | jog0: I am not really picky. The current behavior hasn't bothered me even on my phone... So I am open to whatever you think will help | 01:36 |
| *** markmcclain has quit IRC | 01:37 | |
| fungi | it will be an invitation for bikeshed painters | 01:37 |
| jog0 | lets paint it red | 01:38 |
| clarkb | my gut is telling me it is the wrong kind of optimization | 01:38 |
| clarkb | because it doesn't matter from IRC's perspective | 01:38 |
| locke105 | +1 for red | 01:38 |
| clarkb | but because I feel that it doesn't matter I will withhold the -1s :) | 01:38 |
| * fungi left his paintbrush in his other pants | 01:38 | |
| locke105 | :P | 01:38 |
| *** pcrews has quit IRC | 01:39 | |
| clarkb | as long as you are under 512 bytes it is good :) | 01:39 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/gerritbot: Make proposed and merged messages more terse https://review.openstack.org/40531 | 01:39 |
| jog0 | nice and wordy ^ | 01:39 |
| jog0 | the nova room has been empty lately and i think gerrit bot is to blame | 01:40 |
| jog0 | thats just my hunch | 01:40 |
| clarkb | jog0: you can turn it off in the channel too if you think it might help | 01:40 |
| clarkb | we are still spitting the same info to openstack-dev iirc | 01:40 |
| jog0 | clarkb: this seemed like a less drastic first step | 01:40 |
| jog0 | clarkb: I already ignore openstack-dev because of gerritbot though :) | 01:40 |
| fungi | -dev may only be getting merge comments | 01:40 |
| fungi | we actually make use of the gerritbot comments in here, and to some degree they drive additional discussion | 01:41 |
| fungi | i would miss them | 01:41 |
| clarkb | jog0: so the proper way to deal with this in IRC is to set up filters/ignores | 01:41 |
| jog0 | fungi: sure but this is a much smaller team, | 01:41 |
| clarkb | don't not sit in a channel because a bot is chatty, just ignore the bot | 01:42 |
| jog0 | clarkb: perhaps, but right around when we started using gerritbot in openstack-nova I think it became less used | 01:42 |
| fungi | but a channel of 162 even if most lurk, and we probably get at least as many patches across our collection of projects as nova+novaclient | 01:42 |
| jog0 | fungi: how many patches a day do you get? | 01:43 |
| clarkb | fungi: I think jeblair counted once and we are about half of nova | 01:43 |
| clarkb | fungi: which is chattier than any other project iirc | 01:43 |
| fungi | ahh, okay, so not quite as high as i though | 01:43 |
| fungi | t | 01:43 |
| clarkb | so not nova bad but not quiet either | 01:43 |
| *** changbl_ has joined #openstack-infra | 01:44 | |
| fungi | it's within the same order of magnitude ;) | 01:44 |
| jog0 | either way I don't see why the bot needs to be so verbose. I do see value in the bot, figure this would be a good safe experiment | 01:44 |
| *** zaro has quit IRC | 01:44 | |
| *** gyee has joined #openstack-infra | 01:44 | |
| *** yaguang has quit IRC | 01:45 | |
| jog0 | before gerritbot I wired up growl (on OS X) to do the same thing | 01:45 |
| *** yaguang has joined #openstack-infra | 01:45 | |
| *** zaro has joined #openstack-infra | 01:46 | |
| *** adalbas has quit IRC | 01:47 | |
| *** zaro0508 has joined #openstack-infra | 01:52 | |
| *** nati_uen_ has joined #openstack-infra | 01:54 | |
| *** nati_ueno has quit IRC | 01:56 | |
| *** gyee has quit IRC | 01:57 | |
| *** davidhadas has quit IRC | 01:57 | |
| *** davidhadas has joined #openstack-infra | 01:58 | |
| *** thomasm has joined #openstack-infra | 01:59 | |
| *** anteaya has quit IRC | 02:04 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/gitdm: Add fungi@yuggoth.org to foundation group file https://review.openstack.org/40533 | 02:05 |
| *** enikanorov_ has quit IRC | 02:06 | |
| *** soren has quit IRC | 02:10 | |
| *** soren has joined #openstack-infra | 02:10 | |
| *** prad has joined #openstack-infra | 02:10 | |
| *** dina_belova has joined #openstack-infra | 02:10 | |
| *** dina_belova has quit IRC | 02:15 | |
| *** Ryan_Lane has joined #openstack-infra | 02:16 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: Raise warlock requirement https://review.openstack.org/37616 | 02:16 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: Raise eventlet to 0.13.0 https://review.openstack.org/37996 | 02:16 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: Add pysnmp needed for Ceilometer/HardwareAgent/SNMPInspector https://review.openstack.org/30746 | 02:16 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: Bump pbr requirment to 0.5.21 https://review.openstack.org/40117 | 02:16 |
| *** Ryan_Lane has quit IRC | 02:19 | |
| *** emagana has quit IRC | 02:20 | |
| *** Ryan_Lane has joined #openstack-infra | 02:21 | |
| *** nijaba has quit IRC | 02:21 | |
| *** nijaba has joined #openstack-infra | 02:22 | |
| *** nijaba has quit IRC | 02:22 | |
| *** nijaba has joined #openstack-infra | 02:22 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: Update keyring minimum version https://review.openstack.org/38358 | 02:27 |
| *** Ryan_Lane has quit IRC | 02:30 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: remove netifaces as a requirement https://review.openstack.org/38482 | 02:31 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: Raise psutil requirement https://review.openstack.org/38429 | 02:31 |
| Alex_Gaynor | Is there a script to add the apache header to files? | 02:33 |
| clarkb | Alex_Gaynor: I don't think so | 02:34 |
| clarkb | hacking does check that it exists in files longer than 10 lines though | 02:34 |
| * Alex_Gaynor gets his copy-paste hat on | 02:34 | |
| * locke105 is not aware of any script that does that either | 02:34 | |
| clarkb | for file in `git status --porcelain | grep ^(A | 02:36 |
| clarkb | bah | 02:36 |
| * clarkb tries again | 02:36 | |
| fungi | grr... non-py3k-compatible global requirements are preventing us from building a pypi mirror on mirror33: https://jenkins.openstack.org/job/post-mirror-python33/4/console | 02:39 |
| locke105 | i've tried to :wq my irc so many times its not funny | 02:39 |
| clarkb | for file in `git status --porcelain | grep ^(A|\?\?) | sed -n 's/(A|\?\?)\(.*\)/\1/p'` do ; cp $HEADER_FILE $TMPFILE && cat $file >> $TMPFILE && mv $TMPFILE $file ; done | 02:39 |
| fungi | +++ATH | 02:40 |
| clarkb | so uh I am sure the regexes in that are wrong, but you get the idea | 02:40 |
| clarkb | locke105: I recently switch bash to vi mode. It has been fun | 02:41 |
| locke105 | clarkb: wut? | 02:41 |
| clarkb | locke105: best of all dd works | 02:41 |
| *** emagana has joined #openstack-infra | 02:41 | |
| clarkb | and hitting v in command mode opens vim | 02:41 |
| locke105 | haha | 02:41 |
| locke105 | thats freakin' awesome... must try... | 02:42 |
| clarkb | locke105: `set -o vi` iirc | 02:42 |
| clarkb | locke105: hjkl work too | 02:44 |
| clarkb | I haven't figured out the shortcut for clear though | 02:44 |
| locke105 | wow awesome | 02:44 |
| locke105 | except it opens nano for some reason on my ubuntu machine.... | 02:44 |
| locke105 | must have not set EDITOR | 02:45 |
| locke105 | wow this is great | 02:45 |
| *** sarob has joined #openstack-infra | 02:46 | |
| fungi | update-alternatives --set editor /usr/bin/vim.basic | 02:46 |
| clarkb | it has definitely made not learning emacs things easy | 02:46 |
| clarkb | as I try random vi things and they just work :) | 02:47 |
| locke105 | yep | 02:47 |
| locke105 | clarkb: you rock, thanks for the tip | 02:47 |
| clarkb | fungi: do you vi mode too? | 02:47 |
| clarkb | at times it feels half backed but I don't use a lot of the emacs stuff because it isn't familiar to me | 02:47 |
| fungi | clarkb: not previously... already know a lot of classic readline shortcuts anyway | 02:47 |
| fungi | assuming you mean vi mode in bash (as opposed to vi mode in emacs) | 02:48 |
| clarkb | yup | 02:48 |
| locke105 | i got so excited i forgot what i was doing now | 02:48 |
| clarkb | fungi: also if doing emacs in vi mode you should use elvis because the name is awesome | 02:49 |
| clarkb | I really want an ex mode when you type : | 02:50 |
| mordred | fungi: we could land my "install what you can" patch | 02:50 |
| clarkb | mordred: your global venv change failed in the gate :( | 02:50 |
| fungi | mordred: it does suddenly become much more compelling | 02:50 |
| mordred | clarkb: ossume! | 02:50 |
| mordred | fungi: see... I had reasons for everything... ;) | 02:50 |
| clarkb | fungi: mordred doesn't it already install what it can? | 02:51 |
| mordred | no | 02:51 |
| clarkb | I mean things blow up and it still copies the packages | 02:51 |
| fungi | clarkb: it gives up on the first exception | 02:52 |
| mordred | well, it blows up at first error | 02:52 |
| *** melwitt has quit IRC | 02:52 | |
| clarkb | i see | 02:52 |
| mordred | which means we get predictably broken things | 02:52 |
| clarkb | and that is a pip thing right? | 02:52 |
| fungi | and then copies however far it made it, but doesn't try to install anything else | 02:52 |
| mordred | this has repeatedly been a problem for me in pbr gate | 02:52 |
| * dstufft thinks it should just blow up if pip install fails, not silently error | 02:52 | |
| *** emagana has quit IRC | 02:53 | |
| clarkb | mordred: have you seen http://eat24hours.com/ ? | 02:53 |
| clarkb | this is going to be seriously dangerous for my food intake | 02:53 |
| pleia2 | I use that + delivery.com | 02:55 |
| mordred | clarkb: I use seamless - but I'm in NY :) | 02:55 |
| mordred | dstufft: it does that already | 02:55 |
| mordred | dstufft: BUT | 02:55 |
| clarkb | pleia2: there is another one I have bene told about to use in SEA but I forget what it is | 02:55 |
| mordred | dstufft: if a pip blows up in teh woods and there is nobody to see it... | 02:56 |
| clarkb | pleia2: I am trying to cook more and amazon fresh helps on that front | 02:56 |
| dstufft | mordred: no I mean the job should fail if pip blows up | 02:56 |
| mordred | dstufft: right... but who is going to notice | 02:56 |
| mordred | dstufft: these are periodic jo bs | 02:56 |
| mordred | when it breaks, it regularly stays broken for weks | 02:56 |
| mordred | weeks | 02:56 |
| clarkb | dstufft: I thought so too, but we had this discussion and we want to the breakage to percolate downstream so that the projects fix it themselves | 02:56 |
| mordred | what I want is as much of the mirror mirrored as possible | 02:57 |
| clarkb | because swift is in a better position to make a decision about exploding xattr than I am | 02:57 |
| mordred | and I don't want a breakage in prettytable | 02:57 |
| pleia2 | clarkb: no amazon fresh but we have instacart.com here, can order from a few local grocery stores, very nice :) | 02:57 |
| mordred | to affect whether or not we can mirror eventlet | 02:57 |
| mordred | dstufft: it's possible we're not using all of our words thogh | 02:57 |
| clarkb | pleia2: the other problem I have is five point is a block and a half away | 02:57 |
| mordred | dstufft: the current process does a pip install -d . -r requirements.txt | 02:57 |
| clarkb | pleia2: so whenever I am too lazy to make somethign I can just walk over there and eat horrible^HHHHHHHH awesome greasy pub food | 02:58 |
| mordred | dstufft: to grab _all_ of the thigns that opentsack needs in the mirror | 02:58 |
| mordred | dstufft: and as soon as it can't download any of them, it stops trying | 02:58 |
| pleia2 | clarkb: haha, doh :) | 02:58 |
| *** amotoki has joined #openstack-infra | 02:58 | |
| clarkb | fungi: have you been to five point? | 02:58 |
| clarkb | if not maybe we should beer there | 02:58 |
| dstufft | mordred: yea, and then it just moves onto the next pip install -d right? | 02:58 |
| mordred | no | 02:58 |
| clarkb | because it is awesome | 02:58 |
| mordred | there is no next install -d | 02:58 |
| fungi | clarkb: i have not, but certainly want to | 02:58 |
| dstufft | mordred: isn't there multiple mirrors? | 02:59 |
| mordred | there is a single, global, requirements list with every requirement needed by all of openstack | 02:59 |
| dstufft | grizzly, havana etc | 02:59 |
| mordred | not yet, no | 02:59 |
| dstufft | oh | 02:59 |
| dstufft | Maybe my brain is remembering it wrong | 02:59 |
| fungi | working on that this week | 02:59 |
| dstufft | or I didn't understand the script well enough | 02:59 |
| mordred | it _also_ tries putting the other branches into that mirror | 02:59 |
| mordred | so yes, it will tryu those too | 02:59 |
| mordred | but | 02:59 |
| mordred | they are all in alpah order | 03:00 |
| fungi | soon it will be putting those branches in separate mirrors | 03:00 |
| clarkb | we have worked around it so far because the old versions of thigns worked with the stable branches | 03:00 |
| mordred | so it's likely that grizzly's version will fail in the same place | 03:00 |
| clarkb | and we don't remove old packages | 03:00 |
| mordred | but in any case- we wound up with a case where a thing early in the list was bombing on cffi | 03:00 |
| dstufft | mordred: I think installing each item with pip seperately is going to bite you fwiw but I think I've mentioned that before. But it's your pain if it does :) | 03:00 |
| fungi | in this case, boto does not play well with (py3k) others | 03:01 |
| mordred | dstufft: why? | 03:01 |
| fungi | File "./boto/pyami/config.py", line 185 | 03:01 |
| fungi | print s.getvalue() | 03:01 |
| mordred | dstufft: if we do each individually, will we not wind up with the things we asked for in the list, plus a myriad of other crap? | 03:02 |
| fungi | mordred: dstufft: the main difference i've seen it cause is pulling in newer packages than what our requirements list caps at (in addition to the versions we do want) | 03:03 |
| mordred | fungi: I think I'd rather have that, tbh | 03:03 |
| mordred | as long as we do get the ones we aske for too | 03:03 |
| clarkb | mordred: for the reason that it did bite us | 03:03 |
| fungi | which shouldn't itself cause issues with our requirements sync anywa | 03:03 |
| mordred | btw- clarkb was right | 03:03 |
| fungi | y | 03:03 |
| clarkb | I was right about something? what was I right about? | 03:04 |
| clarkb | mordred: the biting happens because pip doesn't resolve dependencies together. It isn't great at that but individual installs are even worse | 03:04 |
| mordred | clarkb: well, at this point, we are enforcing versions strongly through other means | 03:04 |
| mordred | so, as logn as the versions that are in the per-project lists are in the mirror | 03:04 |
| dstufft | mordred: Yea becasue of higher versions, and that if project A depends on Foo<=1.0, and project B depends on Bar, which Depends on Foo w/o a version spec | 03:04 |
| mordred | our installation of _those_ all in one go should get us the right hting | 03:04 |
| dstufft | thenw hen you install project B alone from the mirror you get a different set of versions thenw hen you install project A and Project B together | 03:05 |
| mordred | I do not think that's solvable by the m irror | 03:05 |
| mordred | which is what clarkb was saying originally | 03:05 |
| mordred | what Iwant the mirror to be now | 03:05 |
| mordred | is a thing that makes us not dependent on the state of an external network resrouces for the gate | 03:05 |
| mordred | and let version mnanagement happen in the projects | 03:06 |
| mordred | where order of install _is_ curated | 03:06 |
| mordred | so a large corpus of versions in the mirror is great | 03:06 |
| *** dkliban has quit IRC | 03:06 | |
| mordred | I don't need the mirror to protect me unnaturally from a situation that would occur in the wild | 03:06 |
| dstufft | Gotcha | 03:07 |
| clarkb | mordred: that makes me so happy :) | 03:07 |
| mordred | clarkb: see! you were right! | 03:07 |
| *** prad has quit IRC | 03:07 | |
| *** emagana has joined #openstack-infra | 03:07 | |
| mordred | I disagreed with you before the current batch of requireents wrk | 03:07 |
| mordred | work | 03:07 |
| clarkb | and I understand why | 03:07 |
| mordred | because we did _not_ protect strongly enough | 03:07 |
| mordred | I mean - yeek | 03:07 |
| mordred | we still barely do | 03:07 |
| clarkb | there was definitely the large step that needed to be made to get from here to there | 03:07 |
| *** emagana has quit IRC | 03:07 | |
| dstufft | mordred: You might have less pain not using pip to mirror then once you no longer have deps hosted externally to PyPI | 03:07 |
| mordred | yeah - less pain ... but still - if we depend on external resources | 03:08 |
| mordred | then we're at your personal mercy | 03:08 |
| mordred | should something go down | 03:08 |
| dstufft | no I mean | 03:08 |
| mordred | not that I don't like you :) | 03:08 |
| dstufft | using something other than pip | 03:08 |
| mordred | like what? | 03:08 |
| mordred | EVERY other piece of software I've seen and tried is TERRIBLE | 03:08 |
| mordred | I mean UNWORKABLE | 03:09 |
| dstufft | mordred: You've tried bandersnatch? | 03:09 |
| mordred | and non functional | 03:09 |
| mordred | and doesn't do anything | 03:09 |
| mordred | omg. there's another one? /me goes to see in what ways it does not work... | 03:09 |
| dstufft | mordred: it's the new recommended mirroring client over pep381client | 03:09 |
| fungi | pip and friends will figure out our dependency tree. bandersnatch will try to mirror all of pypi right? | 03:09 |
| mordred | dstufft: what does it mirror? | 03:10 |
| clarkb | I am not opposed to mirroring all of pypi if external links go away | 03:10 |
| dstufft | mordred: It'll mirror PyPI (hence the once you no longer have deps hosted externally) | 03:10 |
| dstufft | the benefit is it'll be WAY faster | 03:10 |
| clarkb | that avoids the branch problem if we restrict versions through some other mechanism (which was my original argument :) ) | 03:10 |
| dstufft | well the intial sync will suck | 03:10 |
| dstufft | but after that you can just run the mirror in a cronjob once a minute or once every 5 minutes or so, and not have to rebuild the entire mirror | 03:11 |
| dstufft | You'll get your own minature version of PyPI, which will only help you once all your deps are PyPI hosted only | 03:11 |
| *** dina_belova has joined #openstack-infra | 03:11 | |
| mordred | do we know where we are on that? | 03:11 |
| dstufft | I haven't looked recently. I know some of them moved over though | 03:12 |
| dstufft | I can take a look though | 03:12 |
| fungi | it was previously suggested that we wanted to be able to identify and curate our transitive dependencies, and mirroring the entire dependency tree from our direct requirements list accomplishes that | 03:12 |
| mordred | fungi: right. I believe that's an out of date desire | 03:12 |
| dstufft | fungi: so does pip instaling into a venv and using pip freeze whenever you need to see the entire dep tree :) | 03:12 |
| mordred | which was a thing we wanted before we had strong reuqirements gating | 03:12 |
| mordred | however - _I_ like run-mirror because I can run it on my laptop | 03:13 |
| mordred | and get a usable mirror for when I'm on a plane :) | 03:13 |
| dtroyer | +++++ | 03:13 |
| mordred | and that's why lifeless is wanting to add rsync support ot our mirror | 03:13 |
| * mordred might spend a lot of time in planes | 03:14 | |
| dstufft | mordred: I run bandersnatch on my laptop :D, takes up above 53GB though | 03:15 |
| lifeless | rsync++ | 03:15 |
| lifeless | dstufft: mordred: so what I want is a /mini/ pyPI with just the transitive deps in it. | 03:15 |
| lifeless | because 53GB to install tripleo is bad. | 03:15 |
| mordred | dstufft: 57M/home/mordred/cache/pypi/ | 03:16 |
| lifeless | 100MB would be about right. | 03:16 |
| mordred | 57 | 03:16 |
| mordred | 57M is the size | 03:16 |
| *** dina_belova has quit IRC | 03:16 | |
| mordred | and that's with old things in it, because I jus update it | 03:16 |
| fungi | so 3 orders of magnitude reduction | 03:16 |
| dstufft | possible the author of bandersnatch would support partial mirroring | 03:16 |
| dstufft | dunno | 03:16 |
| dstufft | just an idea so your mirroring is way faster :) | 03:16 |
| lifeless | dstufft: rsync is widely available | 03:16 |
| dstufft | lifeless: you're not gonna get rsync from pypi | 03:16 |
| lifeless | dstufft: I'd rather use widely available tools even if they aren't domain specific. | 03:17 |
| mordred | dstufft: he will from our mirror | 03:17 |
| lifeless | dstufft: I don't care about pypi, just pypi.o.o. | 03:17 |
| dstufft | sure, I don't care what you do from your mirror, I'm saying to get things onto your mirror | 03:17 |
| fungi | bandertorrent | 03:17 |
| mordred | ah - well, in general, mirror speed _in_ to our mirror is not a problem | 03:18 |
| mordred | yikes! | 03:18 |
| mordred | 448Mopenstack | 03:18 |
| mordred | that's the size of our curent mirror | 03:18 |
| mordred | lifeless: ^^ | 03:18 |
| lifeless | mordred: thats tolerable | 03:18 |
| mordred | lifeless: just letting you know | 03:18 |
| lifeless | mordred: base ubuntu image is within margin-of-error of that. | 03:18 |
| mordred | lifeless: we have literally never deleted anything from it | 03:18 |
| lifeless | mordred: 250MB or whatever. | 03:18 |
| fungi | i intend to clean that up with the branch-specific mirrors anyway, so we'll only carry 6 months of cruft per mirror once that's working | 03:18 |
| lifeless | fungi: whats this per-mirror thing? | 03:19 |
| fungi | lifeless: separate mirror for stable branches of openstack/requirements | 03:19 |
| lifeless | fungi: do you mean the havana/grizzly/ etc stuff? | 03:19 |
| fungi | yep | 03:19 |
| mordred | lifeless: yah | 03:19 |
| lifeless | fungi: I thought it was folded intothe same tree? | 03:19 |
| lifeless | fungi: so there is no redundancy? | 03:19 |
| dstufft | mordred: okies, well just an idea, bandersnatch is also smarter about not accepting old versions from the CDN and such but if you're rebuilding your mirror periodically that's less important because you just pay the cost overtime by needing to redo the entire mirroring operation | 03:19 |
| *** mestery_ is now known as mestery | 03:19 | |
| mordred | yeah. I mean, bandersnatch seems to be a MUCH better choice fora full mirror | 03:20 |
| fungi | lifeless: originally it was folded together out of necessity, because we needed to see it in action before adding the necessary complexity. also we haven't been diverging requirements enforcement for new releases yet | 03:20 |
| lifeless | I don't see how bandersnatch could do partial for us anyhow, not unless I convince mordred and jeblair to list the transitive deps | 03:20 |
| mordred | the problem is - there are still external deps | 03:20 |
| mordred | and transitive deps | 03:21 |
| lifeless | fungi: so, I think folding it together is a good thing. | 03:21 |
| mordred | so, bandersnatch is unlikely to grab either for a while | 03:21 |
| mordred | during which time, we'll have ot continue to engineer our current thing | 03:21 |
| lifeless | fungi: don't need to download the same file twice.. | 03:21 |
| mordred | fungi: honestly, I'd like to revisit per-pbranch mirrors with jeblair and you in themorning | 03:22 |
| *** nijaba has quit IRC | 03:22 | |
| fungi | lifeless: well, that's a new possibility we've just started discussing... basically relying on comparison of requirements lists alone and not expecting the mirror scope to limit access to packages | 03:22 |
| mordred | fungi: I think it's a project we started before the requirements stuff changed | 03:22 |
| mordred | and I want to make sure it's still a rabbit we need | 03:22 |
| lifeless | fungi: right. | 03:22 |
| *** nijaba has joined #openstack-infra | 03:23 | |
| fungi | mordred: i agree that it's worth reevaluating, but i'm not strongly invested in either position at the moment | 03:23 |
| mordred | in fact, the more I think about it, th emore I think it might be counter productive, because it won't tell us how our requirmeents lists fare in teh cold light of public pypi | 03:23 |
| lifeless | zigactly | 03:23 |
| mordred | fungi: me either - I'm only just now rethining my position | 03:23 |
| *** nati_uen_ has quit IRC | 03:23 | |
| mordred | fungi: speaking of - should I look at the rsync patch? | 03:24 |
| fungi | the last time the topic surfaced there was fairly strong dissent. but the requirements enforcement has lurched forward since | 03:24 |
| fungi | mordred: you can look at the rsync patch if you like. i think jeblair is still unconvinced it's a safe move (or is maybe still on the fence) | 03:24 |
| fungi | the change to implement it is extremely trivial | 03:25 |
| mordred | he didn't -1 though - just asked the question, which I tihnk its fair | 03:25 |
| mordred | I'm going to +2 but ping him in the morning before aprv | 03:25 |
| fungi | it got discussed heavily in here after that, i just haven't recapped it there yet | 03:26 |
| mordred | oh - did it? | 03:26 |
| lifeless | yes | 03:26 |
| mordred | tl;dr ? | 03:26 |
| lifeless | we should add stunnel | 03:26 |
| mordred | ok | 03:26 |
| lifeless | so we have ssl - prevent hostile attackers modifying packages via MITM | 03:26 |
| mordred | ++ I'm on board with that | 03:26 |
| fungi | also possibly move pypi off to a separate server again. i think that's where he was still waffling | 03:27 |
| lifeless | can still have anonymous non-ssl rsync, but tripleo would want to be more secure, all things being equal. | 03:27 |
| fungi | then we moved on to discussing how having a filesystem rsync could grok might be counter to any future move of the mirror to an object storage backend | 03:28 |
| fungi | and looking into rsync alternatives which could be integrated into the web services layer instead | 03:29 |
| mordred | hrm | 03:30 |
| fungi | and then he had to bail for a prior engagement | 03:30 |
| lifeless | I would like to ignore that though, as a 500M mirror isn't a stateless-scaling-problem; it's what - < 60 seconds to mirror from an existing server, or put it on a block device. | 03:31 |
| mordred | nod. well, not to be my brazen self - but the object storage part reminds me of the gerrit folks and "we might have a generalized workflow engine some day"... | 03:31 |
| mordred | and what lifeless said | 03:31 |
| mordred | I think our build logs need to go in object storage badly | 03:32 |
| mordred | the mirror - meh | 03:32 |
| mordred | but the stunnel and the put-mirror-on-own-box seem quite legit | 03:33 |
| *** sarob has quit IRC | 03:35 | |
| *** sarob has joined #openstack-infra | 03:35 | |
| *** arosen1 has joined #openstack-infra | 03:37 | |
| mordred | wow. I just tweeted a link to gerrit docs about prolog | 03:38 |
| fungi | you need to get out more ;) | 03:38 |
| *** dkliban has joined #openstack-infra | 03:39 | |
| mordred | fungi: I just had dinner and drinks in brazil! how much more out can I get? | 03:39 |
| fungi | space | 03:40 |
| *** sarob has quit IRC | 03:40 | |
| fungi | if sabdfl can do it... | 03:41 |
| *** changbl_ has quit IRC | 03:42 | |
| fungi | though i hear the dehydrated ice cream and tang gets tiresome | 03:43 |
| mordred | I love dehydrated ice cream | 03:45 |
| clarkb | mordred: I like the context of that tweet too | 03:47 |
| mordred | clarkb: trying one more devstack patch | 03:48 |
| *** sdake has quit IRC | 03:48 | |
| *** enikanorov has joined #openstack-infra | 03:51 | |
| *** ladquin has quit IRC | 03:53 | |
| *** SergeyLukjanov has joined #openstack-infra | 03:59 | |
| *** kpepple has quit IRC | 04:00 | |
| mordred | ok. I have actually read the scrollback on the earlier discussion around rsync and object store and I have more context | 04:01 |
| lifeless | mordred: devstack - dstufft had a suggestion for the same thing when it bit tripleo. | 04:01 |
| mordred | and I do think a general look at the system design we'd like to be at is warranted | 04:01 |
| lifeless | pip install -U setuptools after installing distribute. | 04:01 |
| mordred | lifeless: what? | 04:01 |
| mordred | lifeless: right. we already do that | 04:01 |
| mordred | lifeless: it breaks on redhat | 04:01 |
| lifeless | actually, not dstufft, was rslage or something - for fixing redhat. | 04:02 |
| lifeless | mordred: interesting, since it was submitted to fix redhat :) | 04:02 |
| mordred | lifeless: it turns out that "redhat" are a conflicting set of needs | 04:02 |
| mordred | lifeless: I have at least three diffrent patches around devstack that fix devstack for one person's use on fedora and breaks it for anothers | 04:02 |
| * mordred tries to remember why pip install -U setuptools was insufficient | 04:03 | |
| lifeless | noice | 04:03 |
| dstufft | it breaks yum | 04:04 |
| mordred | lifeless: what would be great is if I could figure out why starting the services in devstack does not work with the software installed in a virtualenv | 04:04 |
| mordred | dstufft: is that what it is? | 04:04 |
| dstufft | yea I think so | 04:04 |
| dstufft | because it uninstalsl setuptools that yum installed | 04:04 |
| dstufft | and installs a new version | 04:05 |
| mordred | oh. right | 04:05 |
| dstufft | but doesn't tell yum it did that | 04:05 |
| mordred | because pip uninstalls then reinstalls | 04:05 |
| lifeless | oh yay. Well, I'm sure the redhatters will followup ;) | 04:05 |
| mordred | lifeless: I'm surrounded by redhatters | 04:05 |
| lifeless | mordred: where art thou? | 04:05 |
| mordred | I'm more and more in favor of a single global virtualenv | 04:05 |
| mordred | lifeless: brazil | 04:05 |
| lifeless | mordred: I knew that... | 04:05 |
| mordred | I'm not literally surrounded by them | 04:05 |
| mordred | I'm not actually surrounded by them | 04:06 |
| lifeless | mordred: one venv per service :> | 04:06 |
| mordred | I'm figuratively surrounded by them | 04:06 |
| lifeless | mordred: because that way conflicting requirements between services isn't a problem. | 04:06 |
| mordred | no, I very specifically do not want that | 04:06 |
| *** zul has joined #openstack-infra | 04:06 | |
| mordred | we've already solved that | 04:06 |
| mordred | conflicting requirements is fixed | 04:06 |
| dstufft | a virtualenv is pretty similar to a software collection really | 04:06 |
| dstufft | except a software collection is controlled by rpm and handles more than python | 04:06 |
| lifeless | mordred: I don't think you have in general. You have for the 'running trunk of everything' special case. | 04:06 |
| mordred | lifeless: we're not designing a general deployment solution - you are | 04:07 |
| lifeless | mordred: doh, how did I end up doing that :) | 04:07 |
| mordred | lifeless: in this case, _I_ need to care that we prevent conflicting dependencies | 04:07 |
| lifeless | mordred: you may need to revisit the 'single global venv' when we start gating on trunk vs release scenarios. | 04:07 |
| *** sarob has joined #openstack-infra | 04:07 | |
| mordred | as otherwise the distros will kill me | 04:08 |
| mordred | lifeless: why? | 04:08 |
| lifeless | mordred: because the last release of e.g. nova may have conflicting requirements after 5 months. | 04:08 |
| mordred | lifeless: we already do this in grenade | 04:08 |
| lifeless | mordred: grenade installs e.g. released nova, trunk nova client ? | 04:09 |
| mordred | lifeless: the only thing we don't try to do is install different verions on the same machine simultaneously | 04:09 |
| mordred | lifeless: yes | 04:09 |
| mordred | actually | 04:09 |
| mordred | and that should always work :) | 04:09 |
| lifeless | cause I thought grenade tested upgrades. | 04:09 |
| mordred | it does | 04:09 |
| mordred | the client libraries do not have stable releases | 04:09 |
| mordred | I'm picking a nit | 04:09 |
| lifeless | clearly I need to learn more about it | 04:09 |
| * lifeless is shocked | 04:09 | |
| mordred | BUT | 04:10 |
| mordred | I'm also speaking flippantly | 04:10 |
| mordred | right now, it does not entirely do that, because grizzly has some bad arbitrary version caps | 04:10 |
| lifeless | in | 04:10 |
| lifeless | $jeepybvenv/bin/run-mirror -b remotes/origin/master --verbose -c $tmpdir/mirror.yaml --no-process | 04:10 |
| lifeless | what is the branch used for ? | 04:10 |
| lifeless | do I need a checkout of each thing whose requirements I'm going to mirror ? | 04:10 |
| mordred | lifeless: there are some historical bits in there ... | 04:11 |
| mordred | it was originally designed to work on the list of projects in our yaml file | 04:11 |
| *** pcrews has joined #openstack-infra | 04:11 | |
| mordred | now we only operate on the requirments repo | 04:11 |
| *** dina_belova has joined #openstack-infra | 04:11 | |
| mordred | but still treat it with the wider logic | 04:12 |
| lifeless | ok, so I clone openstack-infra/requirements? | 04:12 |
| mordred | run-mirror should do that for you | 04:13 |
| mordred | if you list it in your mirror.yaml file | 04:13 |
| lifeless | oh, so in projects in the yaml I list the github repo? | 04:13 |
| mordred | yes | 04:13 |
| mordred | (and it's openstack/requirements - not openstack-infra/requirements) | 04:13 |
| mordred | lifeless: there's a simple example of usage... | 04:13 |
| mordred | hrm. maybe it's not that simple | 04:14 |
| mordred | https://github.com/openstack-dev/pbr/blob/master/tools/integration.sh#L56-L95 | 04:14 |
| lifeless | for project in mirror['projects']: | 04:14 |
| lifeless | KeyError: 'projects' | 04:14 |
| lifeless | oh, damn copy paste fail | 04:15 |
| *** dina_belova has quit IRC | 04:16 | |
| *** changbl has joined #openstack-infra | 04:17 | |
| *** nayward has quit IRC | 04:20 | |
| lifeless | what does ffi need to build ? | 04:20 |
| lifeless | sorry, xattr | 04:20 |
| lifeless | Running setup.py egg_info for package xattr | 04:20 |
| lifeless | c/_cffi_backend.c:14:17: fatal error: ffi.h: No such file or directory | 04:20 |
| lifeless | compilation terminated. | 04:20 |
| *** nijaba has quit IRC | 04:22 | |
| lifeless | mordred: ^ | 04:22 |
| mordred | lifeless: you need libffi-dev | 04:23 |
| *** nijaba has joined #openstack-infra | 04:23 | |
| *** nijaba has quit IRC | 04:23 | |
| *** nijaba has joined #openstack-infra | 04:23 | |
| mordred | lifeless: https://github.com/openstack-dev/pbr/blob/master/tools/integration.sh#L27 | 04:23 |
| *** reed has joined #openstack-infra | 04:23 | |
| mordred | lifeless: current list there, in case it prevents you from chasing further | 04:24 |
| lifeless | thanks | 04:25 |
| Alex_Gaynor | mordred: gerrit requires like a server to run :effort:. ZaaS please | 04:25 |
| lifeless | I'll note that this is one of my objections to run-mirror ;) | 04:25 |
| mordred | Alex_Gaynor: gimme a little bit, I'm working on it... | 04:25 |
| Alex_Gaynor | mordred: A+ awesome :) | 04:26 |
| mordred | and with that ... I'm off to bed | 04:26 |
| lifeless | Alex_Gaynor: disk-image-builder + heat. doit. | 04:27 |
| Alex_Gaynor | lifeless: Sounds like a thing someone who actually enjoys ops should do | 04:27 |
| lifeless | lol | 04:29 |
| lifeless | File "/tmp/tmpWHXU3n/venv/local/lib/python2.7/site-packages/pip/index.py", line 887, in filename | 04:29 |
| lifeless | assert name, ('URL %r produced no filename' % self.url) | 04:29 |
| lifeless | something wrong there... | 04:29 |
| lifeless | AssertionError: URL 'file://' produced no filename | 04:29 |
| Alex_Gaynor | lifeless: file a bug :) | 04:29 |
| lifeless | Alex_Gaynor: sure, but what on... | 04:29 |
| lifeless | Alex_Gaynor: dunno cause yet. | 04:29 |
| Alex_Gaynor | lifeless: pip I imagine, things should never die with an assertion error | 04:30 |
| lifeless | well, dunno reproduction instructions yet | 04:30 |
| lifeless | Alex_Gaynor: this is a reasonable assertion, the issue for me is the input | 04:30 |
| lifeless | gotta run | 04:30 |
| Alex_Gaynor | dstufft: Look a thing is broken! | 04:31 |
| *** dina_belova has joined #openstack-infra | 04:33 | |
| *** nati_ueno has joined #openstack-infra | 04:33 | |
| *** afazekas has joined #openstack-infra | 04:35 | |
| dstufft | Alex_Gaynor: :[ | 04:36 |
| *** sarob has quit IRC | 04:36 | |
| dstufft | Alex_Gaynor: 1.4.1 is gonna come out tomorrow btw | 04:36 |
| dstufft | with fixes for stuff | 04:36 |
| Alex_Gaynor | dstufft: wassit do? | 04:36 |
| *** sarob has joined #openstack-infra | 04:36 | |
| Alex_Gaynor | dstufft: stuff I filed I guess? | 04:36 |
| *** dina_belova has quit IRC | 04:36 | |
| dstufft | ya | 04:36 |
| dstufft | https://github.com/pypa/pip/issues?milestone=4&page=1&state=closed | 04:37 |
| *** pcrews has quit IRC | 04:39 | |
| *** sarob has quit IRC | 04:41 | |
| *** sdake has joined #openstack-infra | 04:49 | |
| *** reed has quit IRC | 04:50 | |
| *** sdake_ has quit IRC | 04:50 | |
| *** sarob has joined #openstack-infra | 05:00 | |
| *** dina_belova has joined #openstack-infra | 05:05 | |
| *** emagana has joined #openstack-infra | 05:09 | |
| *** shhu has quit IRC | 05:09 | |
| *** boris-42 has joined #openstack-infra | 05:11 | |
| *** jhesketh has quit IRC | 05:14 | |
| *** jhesketh has joined #openstack-infra | 05:14 | |
| *** nijaba has quit IRC | 05:22 | |
| *** dina_belova has quit IRC | 05:22 | |
| *** nijaba has joined #openstack-infra | 05:23 | |
| *** nijaba has quit IRC | 05:23 | |
| *** nijaba has joined #openstack-infra | 05:23 | |
| *** dina_belova has joined #openstack-infra | 05:24 | |
| *** dina_belova has quit IRC | 05:24 | |
| *** sarob has quit IRC | 05:29 | |
| *** sarob has joined #openstack-infra | 05:30 | |
| jhesketh | Hi guys, how do I get gerrit/jenkins to recheck a patchset? | 05:32 |
| *** sarob has quit IRC | 05:34 | |
| pleia2 | jhesketh: this should help https://wiki.openstack.org/wiki/GerritJenkinsGithub#Test_Failures | 05:39 |
| jhesketh | thanks pleia2, my search foo failed me | 05:40 |
| pleia2 | you're welcome | 05:41 |
| jhesketh | so why do we keep running gates that seem to fail/not work that are non-voting? | 05:41 |
| *** pcrews has joined #openstack-infra | 05:43 | |
| *** nicedice has quit IRC | 05:43 | |
| *** sarob has joined #openstack-infra | 05:44 | |
| *** SergeyLukjanov has quit IRC | 05:48 | |
| *** sarob has quit IRC | 05:49 | |
| *** sarob has joined #openstack-infra | 05:49 | |
| clarkb | the goal is they eventually pass and we make them vote | 05:51 |
| clarkb | it helps development of new tests. the tempest testr test for example | 05:51 |
| Alex_Gaynor | clarkb: I was suggesting to mordred the other week that if a team wants to add a non-gating test they should be required to provide a description, a timeline, and under what conditions it'll be upgraded to gating | 05:52 |
| clarkb | thats not a bad idea | 05:52 |
| *** sarob has quit IRC | 05:53 | |
| clarkb | there are legit reasons for somethibg to always be non voting. our jjb diff job is that way. it provides a binary state with logs to help reviewers decide if the change is passing or failing | 05:54 |
| Alex_Gaynor | that's definitely the exception, not the rule though | 05:54 |
| jhesketh | so why not use the silent pipeline for non-voting jobs? | 05:55 |
| clarkb | jhesketh because we want them to have visibility | 05:56 |
| clarkb | the failure rate shoulr motivate you to fix it | 05:56 |
| clarkb | :) | 05:56 |
| *** nayward has joined #openstack-infra | 05:56 | |
| jhesketh | heh, okay | 05:56 |
| clarkb | and yes the jjb diff job is an exception | 05:57 |
| *** arosen1 has quit IRC | 05:57 | |
| *** vogxn has joined #openstack-infra | 05:57 | |
| clarkb | Alex_Gaynor: the swift functional tests graduated to gating status this week \o/ | 05:58 |
| Alex_Gaynor | yay! | 05:58 |
| clarkb | jhesketh: you will probably start seeing non voting py3k testa soon too | 05:59 |
| jhesketh | cool | 05:59 |
| clarkb | again we dont expect them to pass but should provide good info to careful reviewers | 05:59 |
| *** arosen1 has joined #openstack-infra | 06:00 | |
| jhesketh | so I think I follow that zuul triggers jenkins through gearman and jenkins-job-builder builds configurations for jenkins, but I'm unsure where the job code itself is stored | 06:17 |
| jhesketh | for example, where is the code that gate-nova-python27 runs? | 06:18 |
| lifeless | hmm python-cinderclient not mirroring properly | 06:19 |
| lifeless | clarkb: so run-mirror, do you guys basically run it again and again and again until it works ? | 06:19 |
| *** sarob has joined #openstack-infra | 06:20 | |
| *** nijaba has quit IRC | 06:22 | |
| *** nijaba has joined #openstack-infra | 06:23 | |
| *** nijaba has quit IRC | 06:23 | |
| *** nijaba has joined #openstack-infra | 06:23 | |
| *** sarob has quit IRC | 06:26 | |
| lifeless | mordred: oh btw services in venvs not starting - we had a minor glitchon that in neutron with hardcoded paths not being linked into the global path, but I suspect we were doing something unique... | 06:27 |
| lifeless | mordred: what symptoms do you have? | 06:28 |
| *** marun has quit IRC | 06:31 | |
| *** marun has joined #openstack-infra | 06:35 | |
| *** marun has quit IRC | 06:42 | |
| *** sarob has joined #openstack-infra | 06:52 | |
| *** sarob has quit IRC | 06:59 | |
| *** sdake_ has joined #openstack-infra | 07:01 | |
| *** sdake_ has joined #openstack-infra | 07:01 | |
| *** salv-orlando has joined #openstack-infra | 07:12 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added Robot Framework reposts publisher https://review.openstack.org/39939 | 07:13 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added Robot Framework reports publisher https://review.openstack.org/39939 | 07:14 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added pre-scm-buildstep wrapper https://review.openstack.org/39338 | 07:16 |
| *** nijaba has quit IRC | 07:23 | |
| *** woodspa has joined #openstack-infra | 07:23 | |
| *** nijaba has joined #openstack-infra | 07:24 | |
| *** nijaba has quit IRC | 07:24 | |
| *** nijaba has joined #openstack-infra | 07:24 | |
| *** sarob has joined #openstack-infra | 07:25 | |
| *** sarob has quit IRC | 07:29 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:30 | |
| *** jhesketh has quit IRC | 07:31 | |
| *** psedlak has joined #openstack-infra | 07:32 | |
| *** afazekas_ has joined #openstack-infra | 07:33 | |
| *** afazekas has quit IRC | 07:36 | |
| *** afazekas_ is now known as afazekas | 07:36 | |
| *** dina_belova has joined #openstack-infra | 07:40 | |
| *** derekh has joined #openstack-infra | 07:40 | |
| *** woodspa has quit IRC | 07:41 | |
| *** psedlak has quit IRC | 07:43 | |
| *** psedlak has joined #openstack-infra | 07:45 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added more options to the Build Timeout plugin https://review.openstack.org/39329 | 07:49 |
| *** BobBall_Away is now known as BobBall | 07:52 | |
| *** vogxn has quit IRC | 07:53 | |
| *** odyssey4me has joined #openstack-infra | 08:01 | |
| *** Ryan_Lane has joined #openstack-infra | 08:02 | |
| *** fbo_away is now known as fbo | 08:02 | |
| *** jpich has joined #openstack-infra | 08:12 | |
| *** derekh has quit IRC | 08:13 | |
| *** odyssey4me has quit IRC | 08:16 | |
| *** derekh has joined #openstack-infra | 08:18 | |
| *** psedlak_ has joined #openstack-infra | 08:20 | |
| *** odyssey4me has joined #openstack-infra | 08:21 | |
| *** nijaba has quit IRC | 08:24 | |
| *** psedlak has quit IRC | 08:24 | |
| *** nijaba has joined #openstack-infra | 08:24 | |
| *** sarob has joined #openstack-infra | 08:25 | |
| *** sarob has quit IRC | 08:30 | |
| *** Ryan_Lane has quit IRC | 08:33 | |
| openstackgerrit | Roman Podolyaka proposed a change to openstack-infra/config: Modify running of sqlalchemy-migrate tests https://review.openstack.org/39304 | 08:35 |
| *** ruhe has joined #openstack-infra | 08:37 | |
| *** vogxn has joined #openstack-infra | 08:43 | |
| lifeless | mordred: https://bugs.launchpad.net/openstack-ci/+bug/1209132 | 08:45 |
| uvirtbot | Launchpad bug 1209132 in openstack-ci "run-mirror fails to mirror cinderclient" [Undecided,New] | 08:45 |
| *** odyssey4me has quit IRC | 08:47 | |
| *** odyssey4me has joined #openstack-infra | 08:55 | |
| *** dina_belova has quit IRC | 09:12 | |
| *** ruhe has quit IRC | 09:15 | |
| *** ruhe has joined #openstack-infra | 09:17 | |
| *** nijaba has quit IRC | 09:23 | |
| *** nijaba has joined #openstack-infra | 09:24 | |
| *** nijaba has joined #openstack-infra | 09:24 | |
| *** sarob has joined #openstack-infra | 09:26 | |
| *** sarob has quit IRC | 09:31 | |
| *** dina_belova has joined #openstack-infra | 09:42 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Edit README to include amended initial data instructions. https://review.openstack.org/40014 | 09:48 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: projects view now accepts projects with spaces https://review.openstack.org/37977 | 09:49 |
| *** dina_belova has quit IRC | 09:51 | |
| *** GlaciErr has joined #openstack-infra | 09:56 | |
| *** Ryan_Lane has joined #openstack-infra | 10:03 | |
| *** nijaba has quit IRC | 10:24 | |
| *** nijaba has joined #openstack-infra | 10:25 | |
| *** nijaba has quit IRC | 10:25 | |
| *** nijaba has joined #openstack-infra | 10:25 | |
| *** ruhe has quit IRC | 10:25 | |
| *** sarob has joined #openstack-infra | 10:26 | |
| *** dina_belova has joined #openstack-infra | 10:31 | |
| *** sarob has quit IRC | 10:31 | |
| *** vogxn has quit IRC | 10:31 | |
| *** vogxn has joined #openstack-infra | 10:32 | |
| *** yaguang has quit IRC | 10:39 | |
| *** emagana has quit IRC | 10:41 | |
| BobBall | sdague / anyone: What/where is EXTRA_CONF? I'm probably be being blind... but I don't see it | 10:42 |
| *** nati_ueno has quit IRC | 10:42 | |
| *** ruhe has joined #openstack-infra | 10:55 | |
| *** nayward has quit IRC | 10:55 | |
| *** Ryan_Lane has quit IRC | 10:55 | |
| lifeless | mordred: btw https://review.openstack.org/#/c/39811/ just needs +2's AFAICT | 11:01 |
| *** nayward has joined #openstack-infra | 11:01 | |
| *** ruhe has quit IRC | 11:02 | |
| *** weshay has joined #openstack-infra | 11:02 | |
| *** fbo is now known as fbo_away | 11:04 | |
| *** emagana has joined #openstack-infra | 11:11 | |
| *** nayward has quit IRC | 11:17 | |
| *** boris-42 has quit IRC | 11:19 | |
| *** emagana has quit IRC | 11:19 | |
| *** nijaba has quit IRC | 11:24 | |
| *** adalbas has joined #openstack-infra | 11:25 | |
| *** nijaba has joined #openstack-infra | 11:25 | |
| *** nijaba has quit IRC | 11:25 | |
| *** nijaba has joined #openstack-infra | 11:25 | |
| *** sarob has joined #openstack-infra | 11:27 | |
| *** chuckieb has quit IRC | 11:28 | |
| *** woodspa has joined #openstack-infra | 11:29 | |
| *** lcestari has joined #openstack-infra | 11:30 | |
| *** sarob has quit IRC | 11:32 | |
| *** CliMz has joined #openstack-infra | 11:33 | |
| *** GlaciErr has quit IRC | 11:36 | |
| *** CliMz has quit IRC | 11:37 | |
| *** vogxn has quit IRC | 11:38 | |
| *** CliMz has joined #openstack-infra | 11:41 | |
| *** prad_ has joined #openstack-infra | 11:43 | |
| sdague | BobBall: actually it's called EXTRA_OPTS | 11:44 |
| *** ArxCruz has joined #openstack-infra | 11:44 | |
| BobBall | ah | 11:44 |
| BobBall | I searched for CONF too *grin* | 11:44 |
| sdague | we should probably restructure those variables for clarity and get them into the README | 11:44 |
| BobBall | thanks | 11:44 |
| sdague | yeh, my bad | 11:44 |
| sdague | I think it used to be more documented than it currently is, the libraryizing of devstack hid some things away that were in stack.sh | 11:45 |
| *** nayward has joined #openstack-infra | 11:47 | |
| *** ruhe has joined #openstack-infra | 11:58 | |
| *** sandywalsh has quit IRC | 12:01 | |
| *** dkehn_ has joined #openstack-infra | 12:01 | |
| *** mdurnosvistov has joined #openstack-infra | 12:02 | |
| *** rfolco has joined #openstack-infra | 12:02 | |
| *** psedlak_ has quit IRC | 12:02 | |
| *** psedlak_ has joined #openstack-infra | 12:02 | |
| *** dkehn has quit IRC | 12:02 | |
| *** afazekas has quit IRC | 12:02 | |
| sdague | mordred: when you get in I've got a bunch of requirements rebases that need +A | 12:05 |
| sdague | https://review.openstack.org/#/q/status:open+project:openstack/requirements+branch:master+topic:requirements_rebase,n,z | 12:05 |
| *** CaptTofu has quit IRC | 12:06 | |
| *** CaptTofu has joined #openstack-infra | 12:07 | |
| *** mriedem has joined #openstack-infra | 12:07 | |
| *** CaptTofu has quit IRC | 12:08 | |
| *** CaptTofu has joined #openstack-infra | 12:08 | |
| *** fbo_away is now known as fbo | 12:09 | |
| *** prad_ has quit IRC | 12:10 | |
| *** sandywalsh has joined #openstack-infra | 12:14 | |
| *** afazekas has joined #openstack-infra | 12:14 | |
| *** nijaba has quit IRC | 12:25 | |
| *** nijaba has joined #openstack-infra | 12:27 | |
| *** nijaba has quit IRC | 12:27 | |
| *** nijaba has joined #openstack-infra | 12:27 | |
| *** dprince has joined #openstack-infra | 12:29 | |
| *** boris-42 has joined #openstack-infra | 12:30 | |
| *** afazekas has quit IRC | 12:36 | |
| *** psedlak__ has joined #openstack-infra | 12:41 | |
| *** psedlak_ has quit IRC | 12:45 | |
| mordred | sdague: on it | 12:50 |
| *** adalbas has quit IRC | 12:50 | |
| *** adalbas has joined #openstack-infra | 12:55 | |
| *** sarob has joined #openstack-infra | 12:58 | |
| *** afazekas has joined #openstack-infra | 13:00 | |
| sdague | mordred: and this time I was smart, and made them a patch series :) | 13:01 |
| sdague | though I needed to do a lot of cherry-pick --strategy none because it wanted to merge into the test file instead | 13:01 |
| mordred | sdague: yes! well done | 13:02 |
| mordred | and all +A'd | 13:02 |
| *** sarob has quit IRC | 13:02 | |
| *** anteaya has joined #openstack-infra | 13:03 | |
| openstackgerrit | Monty Taylor proposed a change to openstack/requirements: Remove unneeded tests dir reference https://review.openstack.org/40002 | 13:05 |
| openstackgerrit | Monty Taylor proposed a change to openstack/requirements: Update ourselves to ourselves https://review.openstack.org/39963 | 13:05 |
| openstackgerrit | Monty Taylor proposed a change to openstack/requirements: Add pep8 checks https://review.openstack.org/39967 | 13:05 |
| mordred | sdague: ^^ fixed/rebased those | 13:06 |
| *** weshay has quit IRC | 13:06 | |
| sdague | mordred: ok, cool, pep8 was still waiting on landing it as a gate check though, right? | 13:07 |
| sdague | and the rest sat on that | 13:07 |
| mordred | sdague: oh - right | 13:07 |
| mordred | I thought it was on clark's print_fuction quesiton | 13:07 |
| mordred | one sec | 13:07 |
| sdague | no look at clarkb's comments on 39967 | 13:07 |
| mordred | yeah - I remember now. stil pre-coffee | 13:08 |
| sdague | yep, no worries | 13:08 |
| sdague | that's the reason I didn't bother to go through those last night though | 13:08 |
| sdague | also, tempest gate on requirements is fantastic | 13:09 |
| sdague | it means the sqla version uncap gets tested that it blows up before we break anyone | 13:09 |
| sdague | that made me very happy last night | 13:09 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Added pep8 checks to requirements https://review.openstack.org/40615 | 13:11 |
| *** krtaylor has quit IRC | 13:11 | |
| *** nayward has quit IRC | 13:13 | |
| mordred | sdague: +100 | 13:16 |
| BobBall | sdague: https://review.openstack.org/#/c/40583/1 - devstack will fail to run if the directories permissions are 0700 (https://github.com/openstack-dev/devstack/blob/master/functions#L1501). I assumed that 777 was needed because there's no guarantee that any daemons we run will be in our group if they aren't our user? | 13:19 |
| *** pentameter has joined #openstack-infra | 13:21 | |
| *** vogxn has joined #openstack-infra | 13:21 | |
| *** nijaba has quit IRC | 13:25 | |
| *** nijaba has joined #openstack-infra | 13:26 | |
| *** nijaba has quit IRC | 13:26 | |
| *** nijaba has joined #openstack-infra | 13:26 | |
| *** larsks has quit IRC | 13:26 | |
| *** dkehn_ is now known as dkehn | 13:27 | |
| *** afazekas has quit IRC | 13:35 | |
| zaro | mordred: would you be able to take a look? https://review.openstack.org/#/c/40455/ | 13:36 |
| *** vijendar has joined #openstack-infra | 13:36 | |
| openstackgerrit | A change was merged to openstack/requirements: Raise warlock requirement https://review.openstack.org/37616 | 13:36 |
| openstackgerrit | will soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin https://review.openstack.org/40621 | 13:37 |
| *** weshay has joined #openstack-infra | 13:38 | |
| *** dina_belova has quit IRC | 13:38 | |
| *** prad_ has joined #openstack-infra | 13:41 | |
| *** ruhe has quit IRC | 13:42 | |
| sdague | BobBall: what about just turning on rx bits? | 13:43 |
| *** changbl has quit IRC | 13:43 | |
| BobBall | If that's all that's needed I'm happy - I was being paranoid :) | 13:43 |
| openstackgerrit | Dirk Mueller proposed a change to openstack/requirements: Always specify a lower bound on packages https://review.openstack.org/37911 | 13:44 |
| *** ruhe has joined #openstack-infra | 13:44 | |
| *** markmcclain has joined #openstack-infra | 13:47 | |
| openstackgerrit | A change was merged to openstack/requirements: Bump pbr requirment to 0.5.21 https://review.openstack.org/40117 | 13:49 |
| *** pabelanger_ has joined #openstack-infra | 13:49 | |
| *** pabelanger_ has quit IRC | 13:50 | |
| *** pabelanger_ has joined #openstack-infra | 13:50 | |
| BobBall | actually, convinced myself that 755 is fine - that's what devstack dir uses and all of the dirs it pulls down | 13:50 |
| *** pabelanger has quit IRC | 13:50 | |
| BobBall | so root directory being 755 is fine | 13:50 |
| *** pabelanger_ is now known as pabelanger | 13:50 | |
| *** pabelanger_ has joined #openstack-infra | 13:50 | |
| *** pabelanger has quit IRC | 13:50 | |
| openstackgerrit | will soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin https://review.openstack.org/40621 | 13:50 |
| *** pabelanger has joined #openstack-infra | 13:51 | |
| *** krtaylor has joined #openstack-infra | 13:51 | |
| *** cppcabrera has joined #openstack-infra | 13:52 | |
| *** cppcabrera has left #openstack-infra | 13:52 | |
| *** datsun180b has joined #openstack-infra | 13:53 | |
| openstackgerrit | will soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin https://review.openstack.org/40621 | 13:54 |
| *** wu_wenxiang has joined #openstack-infra | 13:54 | |
| *** woodspa has quit IRC | 13:56 | |
| *** woodspa has joined #openstack-infra | 13:57 | |
| *** sarob has joined #openstack-infra | 13:58 | |
| openstackgerrit | will soula proposed a change to openstack-infra/jenkins-job-builder: Adding support for the Warnings plugin https://review.openstack.org/40621 | 13:59 |
| BobBall | arghhhhh - sdague are you there? or someone else who knows what's going on with setuptools... | 14:02 |
| BobBall | Trying to run devstack on CentOS - it used to work but now the code in lib/infra for unfubar_setuptools fails because after we've yum removed it, pip has gone too | 14:02 |
| *** sarob has quit IRC | 14:03 | |
| BobBall | so pip_install of setuptools doesn't work :) | 14:03 |
| *** dina_belova has joined #openstack-infra | 14:05 | |
| *** avtar has joined #openstack-infra | 14:06 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/storyboard: Introducing project groups https://review.openstack.org/40626 | 14:07 |
| *** tiamar has quit IRC | 14:07 | |
| mordred | BobBall: yes. we're having rh/centos issues | 14:09 |
| BobBall | okay | 14:09 |
| BobBall | known problem then :) | 14:09 |
| mordred | try commenting out the removal | 14:09 |
| mordred | and see if that breaks more or less for you | 14:10 |
| *** _TheDodd_ has joined #openstack-infra | 14:10 | |
| *** ^d has joined #openstack-infra | 14:11 | |
| *** ^d has joined #openstack-infra | 14:11 | |
| *** ruhe has joined #openstack-infra | 14:15 | |
| BobBall | less so far... ;) | 14:24 |
| mordred | w00t | 14:24 |
| *** nijaba has quit IRC | 14:25 | |
| *** nijaba has joined #openstack-infra | 14:26 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/storyboard: Introducing project groups https://review.openstack.org/40626 | 14:27 |
| sdague | BobBall: so there is a proposed fix for it that dtroyer wrote | 14:28 |
| sdague | BobBall: try this on for size - https://review.openstack.org/#/c/39827/ | 14:29 |
| BobBall | I've got confused in the last few days - too many similar issues going round in my head | 14:29 |
| sdague | BobBall: dude, I'm with you | 14:29 |
| sdague | I've been down this rabbit hole for 3 weeks now | 14:29 |
| *** enikanorov-w has joined #openstack-infra | 14:29 | |
| *** burt has joined #openstack-infra | 14:29 | |
| sdague | but I think we actually have a light at the end of the tunnel | 14:29 |
| BobBall | Ah - not still going down then :D | 14:29 |
| BobBall | heh | 14:30 |
| BobBall | I like his final comment | 14:30 |
| BobBall | stack.sh runs to completion on F18 and CentOS6 but in my environment | 14:30 |
| BobBall | instances do not have working networks | 14:30 |
| BobBall | ;) | 14:30 |
| BobBall | who needs networking?! ;) | 14:30 |
| chmouel | does any of you guys have by any chances the latest standard license header? | 14:32 |
| *** jbjohnso_ has quit IRC | 14:34 | |
| sdague | chmouel: it's in hacking/hacking/core.py | 14:34 |
| clarkb | chmouel: http://www.apache.org/licenses/LICENSE-2.0.txt should be in every repo | 14:34 |
| sdague | as we have a check for it | 14:34 |
| clarkb | the header is at the end of the text | 14:34 |
| *** emagana has joined #openstack-infra | 14:35 | |
| chmouel | cool thanks | 14:35 |
| *** Ryan_Lane has joined #openstack-infra | 14:36 | |
| chmouel | if there is some vim: stuff in the files can i add my emacs stuff there as well ? | 14:36 |
| chmouel | (it's a troll) | 14:36 |
| sdague | chmouel: I do :) | 14:37 |
| *** _TheDodd_ has left #openstack-infra | 14:38 | |
| sdague | though emacs normally doesn't need other stuff to make it not scramble the files :) | 14:38 |
| *** _TheDodd_ has joined #openstack-infra | 14:38 | |
| fungi | a proper troll would have been to ask whether you can add ms-project metadata | 14:38 |
| fungi | sorry, visual studio | 14:38 |
| fungi | i get all their product names mixed up | 14:39 |
| zaro | morning fungi | 14:39 |
| fungi | i also don't actually like the vim comments in files, even though i use vim. i have y vim configured for "the one true style" for whatever file type i happen to be editing | 14:40 |
| fungi | morning zaro! | 14:40 |
| clarkb | fungi ++ annotations in file bother me for some reason | 14:42 |
| *** changbl has joined #openstack-infra | 14:45 | |
| *** rcleere has joined #openstack-infra | 14:46 | |
| *** afazekas has joined #openstack-infra | 14:48 | |
| openstackgerrit | A change was merged to openstack/requirements: Raise eventlet to 0.13.0 https://review.openstack.org/37996 | 14:50 |
| openstackgerrit | A change was merged to openstack/requirements: Add pysnmp needed for Ceilometer/HardwareAgent/SNMPInspector https://review.openstack.org/30746 | 14:50 |
| openstackgerrit | A change was merged to openstack/requirements: Update keyring minimum version https://review.openstack.org/38358 | 14:50 |
| openstackgerrit | A change was merged to openstack/requirements: Raise psutil requirement https://review.openstack.org/38429 | 14:50 |
| openstackgerrit | A change was merged to openstack/requirements: remove netifaces as a requirement https://review.openstack.org/38482 | 14:50 |
| *** mdurnosvistov has quit IRC | 14:52 | |
| *** mdurnosvistov has joined #openstack-infra | 14:52 | |
| *** dina_belova has quit IRC | 14:54 | |
| Mithrandir | fungi: clearly you mean MS word? | 14:57 |
| chmouel | fungi: +1 i am not sure why people do that since it's not like people switch between editor configs all the time | 14:57 |
| fungi | Mithrandir: the only ones i remember using were ms lookout! and m sexchange | 14:57 |
| fungi | though i've heard of m sword, it sounds rather dangerous | 14:58 |
| *** sarob has joined #openstack-infra | 14:59 | |
| *** nayward has joined #openstack-infra | 14:59 | |
| dtroyer | BobBall: :) It was late when I wrote that… IOW I know it isn't done but I wanted to get it out for incremental testing. And more verification that Ububntu is working. | 15:02 |
| ArxCruz | Hello, does anyone know where can I find the hiera data in order to use puppet to install some openstack infra projects? | 15:02 |
| ArxCruz | I'm trying to run puppet and I'm getting this error: | 15:02 |
| ArxCruz | Could not find data item sysadmins in any Hiera data file and no default supplied at /opt/config/production/manifests/site.pp:7 on node river.stglabs.ibm.com | 15:02 |
| *** boris-42 has quit IRC | 15:03 | |
| *** boris-42_ has joined #openstack-infra | 15:03 | |
| *** dina_belova has joined #openstack-infra | 15:03 | |
| clarkb | ArxCruz: you have to provide it or swap out the hiera calls with your data | 15:04 |
| *** sarob has quit IRC | 15:04 | |
| clarkb | sysadmins is a list of email addresses to get root mail iirc | 15:04 |
| ArxCruz | clarkb: thanks, but where can I find an hiera sample ? | 15:04 |
| fungi | ArxCruz: we have some suggestions at http://ci.openstack.org/sysadmin.html#making-a-change-in-puppet | 15:04 |
| fungi | the local.pp example there is of swapping out variables we store in hiera with local definitions | 15:05 |
| clarkb | ArxCruz: we don't currently have a sample file. we could add one or comment values in site.pp about what their contents should be | 15:06 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Improved the gerrit trigger https://review.openstack.org/40644 | 15:06 |
| clarkb | ArxCruz: the only place we use hiera is in site.pp to avoid needing complicated samples requiring hiera | 15:06 |
| *** HenryG has quit IRC | 15:06 | |
| *** krtaylor has quit IRC | 15:07 | |
| ArxCruz | fungi: clarkb thanks | 15:07 |
| ArxCruz | having a sample file it's a good idea, mainly for people starting to use puppet and openstack like me :) | 15:07 |
| *** UtahDave has joined #openstack-infra | 15:07 | |
| *** boris-42_ is now known as boris-42 | 15:08 | |
| clarkb | it doesn't hurt, but it is an implementation detail and not a requirement. I think it may be better to properly document out class parameters | 15:08 |
| fungi | ArxCruz: well, bear in mind our puppet configuration repo at openstack-infra/config will not help you set up openstack in any way. it will help you set up servers like we use to provide resources to our developer community (code review, wiki, pastebin, et cetera) | 15:09 |
| fungi | ArxCruz: if you're interested in using puppet to set up openstack, see the various puppet-* repositories at https://github.com/stackforge/ | 15:10 |
| *** dprince has quit IRC | 15:10 | |
| *** vogxn has quit IRC | 15:13 | |
| *** jjmb has joined #openstack-infra | 15:15 | |
| *** pabelanger has quit IRC | 15:15 | |
| *** krtaylor has joined #openstack-infra | 15:16 | |
| *** CliMz has quit IRC | 15:17 | |
| *** rnirmal has joined #openstack-infra | 15:18 | |
| *** dina_belova has quit IRC | 15:20 | |
| *** SergeyLukjanov has quit IRC | 15:20 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added on-cache option https://review.openstack.org/40651 | 15:23 |
| *** mrodden1 has quit IRC | 15:25 | |
| *** nijaba has quit IRC | 15:26 | |
| *** vogxn has joined #openstack-infra | 15:26 | |
| *** nijaba has joined #openstack-infra | 15:27 | |
| *** nijaba has quit IRC | 15:27 | |
| *** nijaba has joined #openstack-infra | 15:27 | |
| *** vijendar has quit IRC | 15:31 | |
| *** sarob has joined #openstack-infra | 15:31 | |
| *** Ryan_Lane has quit IRC | 15:32 | |
| *** mrodden has joined #openstack-infra | 15:35 | |
| *** HenryG has joined #openstack-infra | 15:36 | |
| *** emagana has quit IRC | 15:36 | |
| *** sarob has quit IRC | 15:37 | |
| *** Ryan_Lane has joined #openstack-infra | 15:37 | |
| *** ruhe has quit IRC | 15:41 | |
| *** zaro0508 has quit IRC | 15:41 | |
| *** Ryan_Lane has quit IRC | 15:44 | |
| *** reed has joined #openstack-infra | 15:52 | |
| *** olaph has quit IRC | 15:54 | |
| chmouel | somebody can help with some weirdness with testr/jenkins? | 15:55 |
| *** salv-orlando has quit IRC | 15:55 | |
| clarkb | sure | 15:57 |
| clarkb | chmouel what are you seeing? | 15:57 |
| chmouel | clarkb: welll that's for this bug https://review.openstack.org/#/c/40601/ | 15:57 |
| chmouel | I am getting an error in test_auth_token_middleware.py:CertDownloadMiddlewareTest | 15:58 |
| chmouel | which is cool and dandy | 15:58 |
| chmouel | but launching the tox manually shows me no error | 15:58 |
| chmouel | http://paste.openstack.org/raw/43442/ | 15:58 |
| chmouel | launching nosetests test_auth_token_middleware.py pass as well | 15:58 |
| chmouel | but launching the test manually fail : | 15:59 |
| clarkb | chmouel: don't use nosetests :) | 15:59 |
| chmouel | https://gist.github.com/chmouel/6175403 | 15:59 |
| chmouel | yep was just invesgiating :) | 15:59 |
| clarkb | so, testr forks and execs testr runners and collects the results over a subunit stream | 15:59 |
| *** ruhe has joined #openstack-infra | 15:59 | |
| clarkb | this means that your tests are partitioned and run in multiple processes concurrently | 16:00 |
| *** krtaylor has quit IRC | 16:00 | |
| clarkb | the other thing testr does is it may not make those partitions deterministic especially across file systems | 16:00 |
| clarkb | all of that to say when you run tox locally the tests may run in a different order and in a different set of processes than they do on jenkins | 16:01 |
| *** olaph has joined #openstack-infra | 16:01 | |
| *** w_ has joined #openstack-infra | 16:01 | |
| *** w_ has quit IRC | 16:01 | |
| chmouel | humm so how do I fix that? | 16:02 |
| clarkb | chmouel: the fact that running the test individual with nose (you can do this with tox by doing `tox -epy27 $testname` where $testname is a regex that is used for matching the name) fails implies that there is some hidden dependency in the test | 16:02 |
| chmouel | rerun the tests until that succed? | 16:02 |
| clarkb | chmouel: no! :) you fix the test to remove the hidden dependency | 16:02 |
| chmouel | :) | 16:03 |
| chmouel | fun times | 16:03 |
| *** CliMz has joined #openstack-infra | 16:03 | |
| clarkb | the test is broken if it cannot run on its own, you will need to figure out what that dependency is and correct it | 16:03 |
| clarkb | the fact that it passes when run by nose with the rest of the suite implies there is some other test that is satisfying the dependency implicitly | 16:04 |
| chmouel | the thing is that i can't reproduce the randomness | 16:04 |
| fungi | or rattle the appropriate cages of people more incented to fix the test, at least | 16:04 |
| clarkb | chmouel: I thought you said you could reproduce by running the test individually? | 16:04 |
| clarkb | chmouel: that should be sufficient to debug | 16:04 |
| chmouel | clarkb: oh i guess, will try | 16:04 |
| chmouel | thanks | 16:04 |
| clarkb | chmouel: cms.subprocess is None | 16:05 |
| *** emagana has joined #openstack-infra | 16:05 | |
| clarkb | chmouel: my guess is somewhere another test is either importing the proper bits or instantiating that object | 16:05 |
| clarkb | and this test needs to do the same | 16:05 |
| chmouel | ah ok yeah i see | 16:06 |
| *** odyssey4me has quit IRC | 16:07 | |
| clarkb | Some days i really want a jenkins job that runs each test in its own process | 16:08 |
| zul | can we get a newer oslo.config that builds with python3 on pypi? | 16:08 |
| clarkb | zul: markmc does those releases iirc | 16:08 |
| zul | clarkb: k | 16:08 |
| chmouel | clarkb: I think I have seem someone doing that with docker | 16:10 |
| clarkb | chmouel: _ensure_subprocess needs to be called in keystoneclient.common.cms | 16:11 |
| clarkb | chmouel: that is a private method so you will need to call one of the things the sideffects it | 16:12 |
| chmouel | clarkb: cool you had it all debugged for me :) | 16:12 |
| chmouel | i was filling up the bug report https://bugs.launchpad.net/python-keystoneclient/+bug/1209300 | 16:12 |
| uvirtbot | Launchpad bug 1209300 in python-keystoneclient "concurrency error in auth_token error" [Undecided,New] | 16:12 |
| chmouel | prob need a better subject | 16:12 |
| clarkb | chmouel: I think it is a fast fix :) | 16:13 |
| chmouel | feel free to send it over :) | 16:13 |
| *** CliMz has quit IRC | 16:13 | |
| chmouel | if you like I don't mind doing it... | 16:14 |
| clarkb | I am just trying to figure out the best way to do this | 16:15 |
| clarkb | It may require calling the private function directly which seems a little dirty | 16:15 |
| ArxCruz | fungi: ty | 16:15 |
| *** pabelanger_ has quit IRC | 16:16 | |
| *** pabelanger has joined #openstack-infra | 16:16 | |
| *** marun has joined #openstack-infra | 16:16 | |
| pleia2 | morning | 16:17 |
| zaro | morning pleia2 | 16:17 |
| zaro | fungi: would you be able to take a look? https://review.openstack.org/#/c/40455/ | 16:18 |
| clarkb | chmouel: there is a chicken and egg | 16:18 |
| clarkb | the assertRaises is calling the method that will init subprocess | 16:18 |
| clarkb | but it is referring to the exception prior to calling it :) | 16:18 |
| zaro | clarkb: not coming in today. | 16:18 |
| clarkb | zaro: ok | 16:18 |
| clarkb | chmouel: there are other tests that call the same method, if they run first the test passes | 16:18 |
| chmouel | i see | 16:19 |
| fungi | clarkb: calling private methods from tests doesn't seem that dirty to me. they're tests. sometimes you may even want to directly test a private method | 16:19 |
| clarkb | chmouel: I will push a naive patch and people can argue if it is good enough :) | 16:19 |
| chmouel | heh | 16:19 |
| *** dprince has joined #openstack-infra | 16:20 | |
| clarkb | running tox now | 16:21 |
| chmouel | clarkb: let me know when you have done so I can review | 16:23 |
| *** krtaylor has joined #openstack-infra | 16:23 | |
| *** vijendar has joined #openstack-infra | 16:25 | |
| *** nijaba has quit IRC | 16:26 | |
| *** nijaba has joined #openstack-infra | 16:27 | |
| *** nijaba has quit IRC | 16:27 | |
| *** nijaba has joined #openstack-infra | 16:27 | |
| clarkb | chmouel: https://review.openstack.org/40671 | 16:28 |
| clarkb | chmouel: I have confirmed tox -epy27 test_request_no_token_dummy fails without the patch and passes with it | 16:29 |
| clarkb | fungi: it is dirty because we want the side effect. we aren't actually directly testing it | 16:30 |
| chmouel | clarkb: cool that was quick | 16:30 |
| fungi | clarkb: in that case, calling any method purely for an unintended side effect seems specious | 16:30 |
| clarkb | chmouel: if you like we can move that into the setUp() call | 16:31 |
| chmouel | I would be honest I don't really understand very well the auth_token middleware structure and inheritance to be confident how it works | 16:32 |
| clarkb | you probably understand it better than me :) | 16:32 |
| chmouel | the test i mean | 16:32 |
| chmouel | I think I wrote the first version of those | 16:33 |
| chmouel | but things has much changed since then :) | 16:33 |
| *** vogxn has quit IRC | 16:33 | |
| CaptTofu | devananda: ping! | 16:33 |
| clarkb | chmouel: updated the commit message with the bug | 16:34 |
| clarkb | completely forgot to do that in my haste | 16:34 |
| chmouel | clarkb: but I think in seTup sounds like a better place | 16:34 |
| clarkb | gah ok :) I will push that | 16:34 |
| chmouel | since there is only one test in this class | 16:34 |
| chmouel | but probably would have to the same bug if there is a new one | 16:35 |
| *** BobBall is now known as BobBall_AFK | 16:36 | |
| clarkb | yup | 16:36 |
| *** derekh has quit IRC | 16:37 | |
| chmouel | clarkb: thanks | 16:38 |
| clarkb | chmouel: third patchset pushed, should be good now | 16:38 |
| chmouel | perfect will ping the keystone guys to get this merged quickly | 16:39 |
| fungi | clarkb: jeblair: mordred: i have temporarily stopped puppet and apache2 on cacti.openstack.org pending a usn and precise security update | 16:41 |
| fungi | clarkb: jeblair: mordred: http://sourceforge.net/mailarchive/message.php?msg_id=31258868 http://www.openwall.com/lists/oss-security/2013/08/07/7 | 16:41 |
| clarkb | thanks | 16:42 |
| fungi | at your option, i can also temporarily comment out the contents of /etc/cron.d/cacti to stop polling, depending on how uncomfortable you are with the security of the systems we're monitoring | 16:44 |
| fungi | but it will of course leave gaps in all the graphs if i do | 16:44 |
| *** boris-42 has quit IRC | 16:45 | |
| fungi | and it doesn't look like any of the exploitation vectors are in the return values from the oids being polled, unless i'm misreading the patches, so probably not a risk | 16:46 |
| fungi | (my php is a bit rusty these days, thankfully) | 16:47 |
| jeblair | fungi: thanks; i think leaving the poller running is ok | 16:51 |
| fungi | i'll keep banging on the usn list until i see an update | 16:52 |
| clarkb | running tests individually is really slow | 16:52 |
| clarkb | I could see this potentially being a periodic job | 16:53 |
| fungi | clarkb: run them individually in parallel on separate machines ;) | 16:53 |
| fungi | that should be really, really fast. you just need really, really many machines | 16:53 |
| *** HenryG has quit IRC | 16:53 | |
| *** HenryG has joined #openstack-infra | 16:54 | |
| * clarkb AFKs for a bit | 17:00 | |
| *** ruhe has quit IRC | 17:01 | |
| *** nayward has quit IRC | 17:03 | |
| *** fbo is now known as fbo_away | 17:03 | |
| *** dkranz has quit IRC | 17:04 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Unset ZUUL_PROJECT on periodic devstack jobs https://review.openstack.org/40677 | 17:04 |
| *** ^demon has joined #openstack-infra | 17:07 | |
| *** ^d is now known as Guest80968 | 17:08 | |
| *** psedlak__ has quit IRC | 17:09 | |
| *** Guest80968 has quit IRC | 17:09 | |
| *** ^demon is now known as ^d | 17:09 | |
| jeblair | fungi: did you want to review https://review.openstack.org/#/c/40468/ ? | 17:10 |
| fungi | i'm sure i did want to. i'll do that now ;) | 17:11 |
| jeblair | fungi, clarkb: https://review.openstack.org/#/c/40677/ is a futher operational fix needed by the stable jobs | 17:11 |
| jeblair | s/stable/bitrot/ | 17:11 |
| *** ruhe has joined #openstack-infra | 17:11 | |
| fungi | added it next in line | 17:12 |
| clarkb | jeblair: in https://review.openstack.org/#/c/40677/1 you are setting ZUUL_BRANCH which seems to conflict with teh commit message | 17:16 |
| clarkb | one of the two is set not both or none | 17:16 |
| jeblair | clarkb: wow, that was a poor commit message wasn't it? | 17:17 |
| *** nati_ueno has joined #openstack-infra | 17:17 | |
| clarkb | my brain still feels like it is far too early in the morning, wanted to make sure I wasn't mis parsing something | 17:17 |
| *** jpich has quit IRC | 17:17 | |
| fungi | yeah, i'm confused by that one as well | 17:17 |
| *** sarob has joined #openstack-infra | 17:18 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Unset ZUUL_PROJECT on periodic devstack jobs https://review.openstack.org/40677 | 17:18 |
| openstackgerrit | A change was merged to openstack-infra/gearman-plugin: Add OFFLINE_NODE_WHEN_COMPLETE option https://review.openstack.org/40468 | 17:18 |
| fungi | from the change i gather zuul is passing ZUUL_PROJECT in the environment and d-g is bailing on it | 17:18 |
| jeblair | clarkb, fungi: forget everything i said earlier and try that instead. :) | 17:18 |
| *** dkranz has joined #openstack-infra | 17:18 | |
| fungi | rereading | 17:18 |
| fungi | that makes waaaay more sense | 17:19 |
| *** hub_cap has left #openstack-infra | 17:19 | |
| *** pabelanger has quit IRC | 17:20 | |
| fungi | also, our gerrit triggers seem not to be running (no trivial rebase comment there about only the commit message changing, several recent bugs which didn't get updated when patches merged). i'm going to start looking into whether we've broken jeepyb again | 17:21 |
| fungi | oh, nevermind. it did comment, just took longer than usual | 17:21 |
| fungi | i still suspect we may have update_bug.py issues though | 17:21 |
| zaro | jeblair: does review 40041 fix bug 1082803? | 17:24 |
| uvirtbot | Launchpad bug 1082803 in openstack-ci "Manage jenkins global config and plugin list" [Medium,Triaged] https://launchpad.net/bugs/1082803 | 17:24 |
| _TheDodd_ | fungi, I hope not. Out of curiosity, what do you think is broken? | 17:24 |
| sarob | anyone up for helping with validate.py errors? | 17:24 |
| jeblair | zaro: oh, i guess it's a partial fix | 17:24 |
| fungi | _TheDodd_: no idea yet. i'm looking at tracebacks fromit in the gerrit logs ow and will paste a summary link here in a bit | 17:24 |
| zaro | cool thing with that uvirtbot showing the bug info. can we do same thing with review info? | 17:25 |
| zaro | jeblair: would you mind updating bug with what else is needed for full fix? | 17:26 |
| *** afazekas has quit IRC | 17:26 | |
| clarkb | zaro: you could extend gerritbot to do that, right now it does not read from irc, it only writes, but reading and responding with review info shouldn't be too hard to add | 17:26 |
| *** nijaba has quit IRC | 17:26 | |
| clarkb | I think some folks feel that gerritbot is too chatty so it will need to be configurable on a per channel basis | 17:27 |
| *** odyssey4me has joined #openstack-infra | 17:27 | |
| *** reed has quit IRC | 17:27 | |
| *** nijaba has joined #openstack-infra | 17:27 | |
| _TheDodd_ | fungi: great. I will try to stay in the loop as much as possible. | 17:29 |
| zaro | clarkb: i just like that it prints more info to irc channel when bug XXX is mentioned. what does that have to do with gerrit being chatty? | 17:29 |
| fungi | _TheDodd_: maybe false alarm. this is the most recent exception i saw it throw, which is potentially legit (not sure which project it was for) http://paste.openstack.org/show/43463/ | 17:29 |
| *** wu_wenxiang has quit IRC | 17:29 | |
| clarkb | zaro: because having extra info about reviews would make that bot more chatty | 17:29 |
| clarkb | zaro: I like the extra info as well and think there are better ways for removing noise if individuals wish (IRC clients come with ignore/filter tools) | 17:30 |
| *** ruhe has quit IRC | 17:30 | |
| fungi | i think we would probably want to run gerritbot not-on-the-review-server if it's acting on random input | 17:30 |
| *** sarob_ has joined #openstack-infra | 17:30 | |
| zaro | 17:22:22 zaro | jeblair: does review 40041 fix bug 1082803? | 17:31 |
| uvirtbot | Launchpad bug 1082803 in openstack-ci "Manage jenkins global config and plugin list" [Medium,Triaged] https://launchpad.net/bugs/1082803 | 17:31 |
| zaro | 17:22:24 uvirtbot | Launchpad bug 1082803 in openstack-ci "Manage jenkins global config and plugin list" | 17:31 |
| uvirtbot | Launchpad bug 1082803 in openstack-ci "Manage jenkins global config and plugin list" [Medium,Triaged] https://launchpad.net/bugs/1082803 | 17:31 |
| zaro | | [Medium,Triaged] https://launchpad.net/bugs/1082803 | 17:31 |
| zaro | opps! | 17:31 |
| fungi | i'm uncomfortable enough with it running there now and acting on semi-dangerous input from the commit message texts, but at least that's more sanitized and leaves a much more substantial audit trail | 17:32 |
| clarkb | fungi: I would be completely open to moving it | 17:32 |
| *** sarob_ has quit IRC | 17:33 | |
| *** sarob__ has joined #openstack-infra | 17:33 | |
| clarkb | fungi: eavesdrop is probably a half decent place for it as the only stuff hosted there is irc and meeting logs (maybe we should treat that data as more important than I am implying though) | 17:33 |
| fungi | in my opinion that would be a prerequisite for turning it into a read+write ircbot | 17:33 |
| fungi | (moving it off review.o.o) | 17:34 |
| clarkb | ++ | 17:34 |
| *** sarob has quit IRC | 17:34 | |
| jeblair | yes, we should put it on eavesdrop | 17:35 |
| zaro | bug 1209335 entered. | 17:36 |
| uvirtbot | Launchpad bug 1209335 in openstack-ci "gerritbot should print review info to irc channel" [Undecided,New] https://launchpad.net/bugs/1209335 | 17:36 |
| *** ruhe has joined #openstack-infra | 17:39 | |
| jeblair | jog0: if the actual issue is that gerritbot interrupts conversations in #openstack-nova, have you considered either removing it from openstack-nova, tweaking its config so it announces only interesting things (only merges? only new patches?), or giving it the ability to be defer announcements while conversations are happening? | 17:40 |
| _TheDodd_ | fungi, a quick look through the stack trace that you linked above seems to indicate that the problem is an authorization issue, where, perhaps, the dev who submitted the code is unauthorized to make changes to that particular bugs status. | 17:43 |
| jog0 | jeblair: i do like having gerritbot and didn't want to remove it | 17:44 |
| _TheDodd_ | fungi, you obviously know the code significantly better than I, but that is what I gathered upon initial evaluation with my current knowledge of our systems. | 17:44 |
| jog0 | jeblair: but not sure what I would keep and remove if tweaking it | 17:44 |
| jog0 | merges aren't that important, as everything is done | 17:44 |
| jog0 | new patches are interesting, as are updates I think | 17:45 |
| *** sarob__ has quit IRC | 17:45 | |
| jog0 | jeblair: and sometimes we refer to bot msgs, like you guys too | 17:45 |
| *** sarob has joined #openstack-infra | 17:45 | |
| *** sarob has quit IRC | 17:45 | |
| *** harlowja has quit IRC | 17:45 | |
| *** nayward has joined #openstack-infra | 17:46 | |
| *** sarob has joined #openstack-infra | 17:46 | |
| *** ruhe has quit IRC | 17:46 | |
| jeblair | jog0: turning of merge messages is easy | 17:46 |
| jog0 | jeblair: in that lets turn of merges | 17:47 |
| jog0 | as I don't think they really are relevant to discussion usually | 17:47 |
| fungi | _TheDodd_: yeah, that's why i said it's probably a legitimate exception and not indicative of a problem in the script | 17:47 |
| jeblair | jog0: openstack-infra/config/modules/gerritbot/files/gerritbot_channel_config.yaml | 17:48 |
| *** harlowja has joined #openstack-infra | 17:48 | |
| mordred | jeblair, jog0: I think that might be a good change to make for #openstack-dev too | 17:48 |
| mordred | for the same reasons | 17:48 |
| jog0 | mordred: will do | 17:49 |
| jeblair | mordred: maybe openstack-dev doesn't want announcements at all. | 17:49 |
| jeblair | [though we'll just miss zuul merge-bombs. :( ] | 17:49 |
| fungi | i find those amusing, if probably extremely disruptive to conversation on -dev | 17:50 |
| *** boris-42 has joined #openstack-infra | 17:51 | |
| jeblair | i actually like the "don't interrupt people" idea and think it could be quite workable. | 17:51 |
| fungi | i concur | 17:52 |
| fungi | ooh, new topic. look at the graphs on http://status.openstack.org/zuul/ | 17:53 |
| fungi | i think we've plateaued at a maximum throughput for new jobs there ;) | 17:53 |
| pleia2 | jeblair: replied to your comment on https://review.openstack.org/#/c/37794/ - if there is time a few of us should probably do some brainstorming to get this finally sorted | 17:54 |
| *** odyssey4me has quit IRC | 17:54 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Don't post merges in openstack-dev and openstack-nova https://review.openstack.org/40685 | 17:55 |
| jeblair | pleia2: i don't quite understand. are you saying puppet can't handle passing a string with "${name}" in it? | 17:55 |
| jeblair | pleia2: even one in single quotes? | 17:55 |
| pleia2 | jeblair: so ${name} in the manifest means something different than name in a template for gerrit | 17:56 |
| *** odyssey4me has joined #openstack-infra | 17:56 | |
| pleia2 | it's a specific variable | 17:56 |
| *** sarob has quit IRC | 17:57 | |
| pleia2 | so I'm pretty sure puppet will expand it to it's default value in puppet in the manifest, and pass that wrong thing to the template rather than passing the string itself | 17:57 |
| mgagne | pleia2: ping | 17:57 |
| pleia2 | mgagne: hey | 17:57 |
| *** sarob has joined #openstack-infra | 17:57 | |
| jeblair | doesn't using single quotes cause it to not be substituted? or isn't there a way to escape it? | 17:57 |
| mgagne | pleia2: can I be of any help? | 17:57 |
| pleia2 | jeblair: that's where puppet-lint gets us, it won't let us | 17:58 |
| *** mrodden1 has joined #openstack-infra | 17:58 | |
| mgagne | jeblair: puppet-lint will cry | 17:58 |
| jeblair | but it really works? | 17:58 |
| jeblair | with puppet? | 17:58 |
| *** avtar has quit IRC | 17:58 | |
| pleia2 | it should | 17:58 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Remove oslo gerritbot posts from openstack-nova https://review.openstack.org/40686 | 17:58 |
| mgagne | jeblair: you can use single-quote to avoid variable interpolation | 17:58 |
| mgagne | jeblair: but puppet-lint won't let you unless you globally disable the check | 17:58 |
| jeblair | so, ah, kill puppet lint? :) | 17:58 |
| pleia2 | puppet-lint is usually lovely :) | 17:58 |
| jeblair | i mean, what we have here is an inability to actually do the work that we want to do | 17:59 |
| mgagne | no_single_quote_string_with_variables | 17:59 |
| pleia2 | I wouldn't want to disable it for gerrit.pp, that's a complicated file and link checks are nice | 17:59 |
| pleia2 | lint too | 17:59 |
| mgagne | pleia2: you can disable this specific check until we (or someone else) find a way to fix it in a nicer way | 17:59 |
| *** mrodden has quit IRC | 18:00 | |
| jeblair | i wouldn't want to hinder our ability to provide a useful way to configure gerrit that doesn't assume specific git repo layouts | 18:00 |
| pleia2 | anyway, in my patch I used fungi's fix of simply putting ${name}.git in the template, it works fine, but now we're stuck on true/false | 18:00 |
| mgagne | pleia2: maybe with the addition of a "#nolint" flag for puppet-lint | 18:00 |
| jeblair | mgagne: that would be lovely | 18:00 |
| *** sarob_ has joined #openstack-infra | 18:00 | |
| jeblair | pleia2: so what's the true/false issue? | 18:00 |
| mgagne | pleia2: what's up with the bool? | 18:00 |
| pleia2 | same thing, we can't single quote them | 18:01 |
| pleia2 | but I guess in this case it'll pass the right thing, so maybe it's ok | 18:01 |
| jeblair | puppet lint doesn't let you pass the string 'true' ? | 18:01 |
| mgagne | jeblair: it won't allow you | 18:01 |
| pleia2 | single quotes bad | 18:01 |
| mgagne | jeblair: true will be true anyway in the template | 18:02 |
| *** sarob has quit IRC | 18:02 | |
| pleia2 | https://review.openstack.org/#/c/37794/10/modules/gerrit/templates/replication.config.erb | 18:02 |
| jeblair | mgagne: and likewise false, i assume? | 18:02 |
| mgagne | jeblair: yes | 18:02 |
| pleia2 | so that's the solution for ${name}.git we came up with | 18:02 |
| jeblair | ok, that seems fine then; i think it makes sense for these args to feel more puppet-native anyway. | 18:03 |
| *** odyssey4me has quit IRC | 18:03 | |
| mgagne | jeblair: unless you *need* True, in which case you can use @var.to_s.capitalize if puppet-lint cries too much | 18:03 |
| pleia2 | ok, and I'll drop the quotes around true/false, since there is no scary variable substitution, it'll just stay true/false | 18:03 |
| pleia2 | mgagne: the word "true" needs to be assigned to the variable "mirror" to be passed into a template | 18:04 |
| mgagne | pleia2: how are bools managed by gerrit? does it have to be true or True? | 18:04 |
| *** sarob_ has quit IRC | 18:04 | |
| pleia2 | mgagne: true is fine | 18:05 |
| mgagne | pleia2: The true puppet boolean will be substituted to true in a template. you should be fine. | 18:05 |
| pleia2 | great, thanks | 18:05 |
| * pleia2 gets to patching | 18:05 | |
| *** dina_belova has joined #openstack-infra | 18:06 | |
| jeblair | pleia2, mgagne: thanks! | 18:06 |
| *** sarob has joined #openstack-infra | 18:06 | |
| *** nayward has quit IRC | 18:07 | |
| jeblair | fungi: i believe that graph should more correctly be labeled "Zuul Jobs Completed (per Hour)" | 18:08 |
| jeblair | (it does still include aborted jobs) | 18:09 |
| fungi | agreed | 18:12 |
| fungi | also a more interesting metric than started anyway | 18:13 |
| jeblair | it's about the same, but time shifted | 18:13 |
| *** dina_belova has quit IRC | 18:14 | |
| fungi | would be really neat to make it multiline and also incorporate successful vs failed vs lost vs unstable vs... but that would maybe be better applied to specific pipeline stats (gate in particular) | 18:14 |
| *** pabelanger has joined #openstack-infra | 18:17 | |
| *** svarnau has joined #openstack-infra | 18:18 | |
| fungi | too much noise in the check pipeline, whereas any bias away from success in the gate pipeline is an interesting statistic | 18:18 |
| clarkb | fungi: we should cleanup the non voting jobs in the gate if we do that | 18:19 |
| clarkb | they will just make noise | 18:19 |
| pleia2 | hmm, in replication.config, what happens if we set "authGroup = " no definition, for stanzas that don't need it? or can we do if statements in there? | 18:19 |
| jeblair | yep. the data should all be there. i think i'd like to try something like sparklines in the pipeline headers. | 18:19 |
| pleia2 | ie: http://paste.openstack.org/show/43481/ | 18:19 |
| clarkb | fungi: but those numbers would be good to see | 18:19 |
| jeblair | pleia2: you can put if statements in there, there should be in-tree examples | 18:20 |
| fungi | clarkb: ahh, tracking voting vs. non-voting to filter those out maybe | 18:20 |
| pleia2 | jeblair: ok thanks, I'll look around | 18:20 |
| jeblair | pleia2: (that compare to the empty string which should be exactly what you want) | 18:20 |
| jeblair | clarkb, fungi: ok we're not tracking voting/non-voting separately. | 18:20 |
| jeblair | i'm not entirely sure we'd want to... | 18:21 |
| clarkb | jeblair: fungi I was thinking of pulling them out of the layout file sections for gate | 18:21 |
| jeblair | clarkb: ? | 18:21 |
| clarkb | jeblair: just don't run them in the gate queue | 18:21 |
| jeblair | clarkb: ah yes | 18:22 |
| jeblair | clarkb: i think that's mostly the case right now? | 18:22 |
| clarkb | I think now that neutron is voting again the instance of that is low | 18:22 |
| jeblair | clarkb: anyway, yes, in that case, agreed, and should be simple. | 18:22 |
| *** ladquin has joined #openstack-infra | 18:23 | |
| *** nijaba has quit IRC | 18:24 | |
| emagana | folks, is devstack broken? | 18:24 |
| emagana | keep getting this error: | 18:25 |
| emagana | sqlalchemy.exc.OperationalError: (OperationalError) (1005, "Can't create table 'neutron_linux_bridge.routerroutes' (errno: 150)") '\nCREATE TABLE routerroutes | 18:25 |
| clarkb | emagana: it runs about a thousands times per day on the devstack gate slaves, but there have been issues around setuptools particularly on fedora/rhel/centos | 18:25 |
| clarkb | emagana: that particular error is not one I am familiar with. Perhaps your mysql/postgres permissions are not quite right? | 18:26 |
| mgagne | Do LP project groups have a bug tracker or is it just a container for other projects? | 18:26 |
| emagana | clarkb: I just install a new ubuntu 12.04 and run devstack | 18:26 |
| emagana | clarkb: I was expecting no errors at all :-( I am using neutron with linuxbridge plugin | 18:27 |
| *** dkranz has quit IRC | 18:27 | |
| *** nijaba has joined #openstack-infra | 18:27 | |
| *** nijaba has quit IRC | 18:27 | |
| *** nijaba has joined #openstack-infra | 18:27 | |
| clarkb | emagana: can you check which version of sqlalchemy is isntalled? `pip freeze` should show it | 18:27 |
| jeblair | mgagne: i _think_ it's just a container; possibly mordred or ttx could confirm with more authority | 18:28 |
| clarkb | I don't expect that to be the issue but there was recently talk of bumping the sqlalchemy version | 18:28 |
| mordred | mgagne: I belive it's just a contained - I think it might show you bugs associated with al lof its sub containers thought | 18:28 |
| emagana | clarkb: sqlalchemy-migrate==0.7.2 | 18:28 |
| *** dina_belova has joined #openstack-infra | 18:28 | |
| clarkb | emagana: cool that is the version we want | 18:28 |
| clarkb | emagana: the neutron devstack testing is still fairly rudimentary aiui. Do you have anything special in your localrc related to neutron? It is possible devstack is trying to do something we don't test | 18:29 |
| emagana | clarkb: will try another plugin, maybe this is neutron linuxbridge related | 18:29 |
| HenryG | When I do "git review -d 34642" I don't get the same code as I see here: https://review.openstack.org/#/c/34642/13 | 18:30 |
| clarkb | emagana: http://logs.openstack.org/71/40671/3/gate/gate-tempest-devstack-vm-neutron/e4e8ee0/logs/localrc.txt.gz is the localrc we use in testing | 18:30 |
| HenryG | What am I doing wrong? | 18:30 |
| clarkb | emagana: note some of those options you will not want because they are related to testing like ERROR_ON_CLONE, but most of them should be fine if you want to copy that | 18:30 |
| clarkb | HenryG: the sha1's are different? | 18:31 |
| *** SergeyLukjanov has joined #openstack-infra | 18:31 | |
| dansmith | I'm seeing a new nova unit test failure related to neutron that seems to be affecting a lot of folks: https://jenkins01.openstack.org/job/gate-nova-python27/86/console | 18:31 |
| dansmith | I don't understand it, and on top of the various other things going on right now, my patches are getting nailed 100% of the time | 18:31 |
| HenryG | clarkb: I did not check that. Where do I look? | 18:32 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix puppet-lint for bare puppet modules https://review.openstack.org/40314 | 18:32 |
| clarkb | HenryG: `git log -1 --oneline` will show you the local one | 18:32 |
| clarkb | HenryG: then you can compare to the on on gerrit | 18:32 |
| emagana | clarkb: checking out the localrc | 18:32 |
| clarkb | dansmith: did neutronclient change their MockCLient version or something? | 18:32 |
| jeblair | HenryG: i tried locally and get the same sha1 in gerrit def56cfaeb9a093888396c5daae891431258cf2e | 18:33 |
| *** nayward has joined #openstack-infra | 18:33 | |
| clarkb | dansmith: it almost looks like neutron's code base expects one thing and nova's another | 18:33 |
| dansmith | clarkb: I dunno, I thought we were all unified now | 18:33 |
| clarkb | dansmith: mostly unified | 18:33 |
| dansmith | ah | 18:33 |
| dansmith | is that like "mostly pregnant" ? | 18:33 |
| dansmith | "officer, I'm mostly sober" | 18:33 |
| clarkb | we allow ranges and don't currently require you update all projects when a requirement changes | 18:34 |
| dansmith | ah | 18:34 |
| clarkb | that is changing though in part to avoid issues like the supposed one above | 18:34 |
| dansmith | clarkb: so the real question is.. can we blame this on sdague somehow? | 18:34 |
| clarkb | dansmith: no, he is fixing this :) | 18:34 |
| dansmith | crap, okay. | 18:34 |
| sdague | is this on unit tests ? | 18:35 |
| HenryG | clarkb: jeblair: thanks. I did "git review -u -d 34642" and that seems to have fixed it. | 18:35 |
| jeblair | sdague: yes | 18:35 |
| sdague | right, I was hoping phase 2 was going to be able to wait longer, but I guess not | 18:35 |
| sdague | dansmith: where's the review link? | 18:35 |
| jeblair | sdague: run every projects unit tests as requirements gate? | 18:35 |
| dansmith | https://review.openstack.org/#/c/40648 | 18:36 |
| clarkb | dansmith: sdague actually, looking closer it appears that nova is mocking neutronclient? | 18:36 |
| dansmith | sdague: I checked a few of the ones in the check queue with failing unit tests and they're because of that | 18:36 |
| clarkb | and the mock isn't complete enough to include EXTED_PLURALS | 18:36 |
| clarkb | dansmith: so possibly an cross test reaction where someone isn't cleaning up their mocks properly | 18:36 |
| jeblair | what's an exted? | 18:36 |
| clarkb | jeblair: no idea | 18:37 |
| clarkb | oh even better | 18:38 |
| clarkb | https://review.openstack.org/#/c/33187/ | 18:38 |
| *** vijendar1 has joined #openstack-infra | 18:38 | |
| clarkb | dansmith: neutronclient changed the internals of the thing being mocked so the existing mock is no longer sufficient? | 18:38 |
| *** vijendar1 has quit IRC | 18:38 | |
| *** vijendar1 has joined #openstack-infra | 18:38 | |
| *** vijendar has quit IRC | 18:38 | |
| emagana | clarkb: It seems that the problem is with linuxbridge plugin, basically something is broken during the alembic.migration | 18:38 |
| jeblair | clarkb: https://pypi.python.org/pypi/python-neutronclient/2.2.5 uploaded today | 18:39 |
| clarkb | jeblair: that'll do it | 18:39 |
| *** dkranz has joined #openstack-infra | 18:39 | |
| jeblair | dansmith, sdague: so possibly neutronclient released a new version that broke backwards compatability with nova unit tests | 18:39 |
| sdague | jeblair: yeh, sounds like | 18:40 |
| dansmith | boo | 18:40 |
| sdague | so this was one of the edge cases I was worried about that we hadn't hit yet, that python clients are out of git in the devstack nodes, but not in the unit tests | 18:40 |
| sdague | so these kinds of breaks end up being allowed | 18:40 |
| clarkb | sdague: I think this would've been allowed anyways | 18:41 |
| clarkb | sdague: because the fail would show up after the thing causing the breakage | 18:41 |
| clarkb | (anyways = even if you pulled client from git) | 18:41 |
| sdague | right, the issue is on the client landing side | 18:41 |
| jeblair | sdague: yeah, we are following the 'treat clients as external releases' model for unit tests | 18:41 |
| sdague | jeblair right... so we've got another asymetric gate | 18:41 |
| sdague | the clients actually need to be gated on the unit tests of the projects that consume them as well | 18:42 |
| clarkb | its asymetric but making it symetric does not fix this particular edge case | 18:42 |
| sdague | which has never been in the equation | 18:42 |
| fungi | sdague: it would be asymmetric regardless, as clarkb points out | 18:42 |
| clarkb | it may fix others though | 18:42 |
| clarkb | sdague: oh, yes that would fix it | 18:42 |
| sdague | fungi: not if python-neutronclient gates on nova unit tests | 18:42 |
| jeblair | sdague: well, i think it's more subtle than that; we've never treated unit tests as integration tests. | 18:42 |
| clarkb | ++ | 18:43 |
| fungi | sdague: oh, okay granted | 18:43 |
| clarkb | jeblair: and I think it is a good thing we don't | 18:43 |
| sdague | jeblair: it is... however we're at 20 git trees that all need to work together | 18:43 |
| clarkb | because running `tox` locally shouldn't be complicated | 18:43 |
| jeblair | sdague: i think i'm in full agreement with you on the technical aspects; this can be solved in the way you describe. | 18:43 |
| clarkb | so one thing lifeless suggested in a mail list thread was that each clinet/server provide mocks to other people | 18:43 |
| jeblair | s/solved/improved/ | 18:44 |
| clarkb | then nova imports netronclient.mock | 18:44 |
| *** CaptTofu has quit IRC | 18:44 | |
| clarkb | then we can gate the mocks with an integration test | 18:44 |
| sdague | clarkb: I think it would just change the surface of the break | 18:44 |
| clarkb | sdague: it changes the surface of what you need to test | 18:44 |
| *** CaptTofu has joined #openstack-infra | 18:44 | |
| clarkb | and gives us one mock per projects instead of nova's, keystone's etc that will each needfixing | 18:45 |
| *** dkranz has quit IRC | 18:45 | |
| sdague | oh what a tangle web we weave.... | 18:46 |
| clarkb | it doesn't outright solve the problem though | 18:46 |
| jeblair | dansmith: since we're about to get into the weeds as we do; do you have the actionable info you need? i think the choices are (a) pin the neutron requirement (bad) (b) update nova to match (c) ask neutronclient to fix/revert and release an update | 18:46 |
| *** sarob has quit IRC | 18:47 | |
| sdague | jeblair: I think he just ran to lunch | 18:47 |
| jeblair | sdague: probably a good choice | 18:47 |
| sdague | heh | 18:47 |
| *** sarob has joined #openstack-infra | 18:47 | |
| *** sarob has quit IRC | 18:48 | |
| clarkb | and if you want to live with a more limited set of corner cases run neutronclient unittests against that mock too | 18:48 |
| clarkb | with the theory being that sufficient test coverage will uncover problems locally | 18:48 |
| jeblair | nati_ueno: it looks like the neutronclient release that included this commit has broken nova unit tests: https://review.openstack.org/#/c/33187/ | 18:49 |
| *** sarob has joined #openstack-infra | 18:49 | |
| jeblair | nati_ueno: you can see the failure here: https://jenkins01.openstack.org/job/gate-nova-python27/86/console | 18:49 |
| nati_ueno | jeblair: gotcha do you have bug report? | 18:49 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add replication of git from gerrit to git.o.o https://review.openstack.org/37794 | 18:49 |
| jeblair | nati_ueno: i haven't seen one | 18:50 |
| clarkb | dansmith might | 18:50 |
| nati_ueno | OK I'll register it | 18:50 |
| dansmith | I didn't | 18:50 |
| sdague | jeblair: heh, I just ran over the the neutron channel to ask about it | 18:50 |
| sdague | but now we have nati_ueno here, so it's all good | 18:50 |
| nati_ueno | Thank you for your sharing issues | 18:51 |
| nati_ueno | jeblair: how did you find https://review.openstack.org/#/c/33187/ is the cause? | 18:53 |
| clarkb | nati_ueno: it added new things to the client that nova needed to mock but hadn't | 18:54 |
| clarkb | nati_ueno: https://jenkins01.openstack.org/job/gate-nova-python27/86/console | 18:54 |
| nati_ueno | I registerd https://bugs.launchpad.net/nova/+bug/1209364 | 18:54 |
| uvirtbot | Launchpad bug 1209364 in neutron "AttributeError: 'MockClient' object has no attribute 'EXTED_PLURALS' " [Critical,In progress] | 18:54 |
| clarkb | notice that EXTED_PLURALS is missing at line 38, 33187 seemed to add that to neutronclient | 18:54 |
| nati_ueno | gotcha let's me check | 18:56 |
| nati_ueno | OK I confirmed the patch is the cause | 18:57 |
| *** ruhe has joined #openstack-infra | 18:58 | |
| *** ^d has quit IRC | 18:58 | |
| nati_ueno | markmcclain: I would like to revert 33187. because FWaaS is not fully merged on server side. | 18:59 |
| nati_ueno | markmcclain: IMO, we should fix nova unit test before the patch is merged | 18:59 |
| nati_ueno | markmcclain: How do you think? | 18:59 |
| *** avtar has joined #openstack-infra | 18:59 | |
| *** dkranz has joined #openstack-infra | 19:00 | |
| lifeless | clarkb: actually I suggested each project export *test doubles*, not mocks per se : actual mocks are per-test, not-per-codebase -> you have to fix every single mock on every single API change always | 19:01 |
| lifeless | mordred: so - run-mirror sadness. | 19:02 |
| mordred | booo | 19:02 |
| lifeless | mordred: I'd like a hand with it, unless rsync is unblocked [I need something working] | 19:02 |
| *** changbl has quit IRC | 19:05 | |
| *** vijendar has joined #openstack-infra | 19:05 | |
| markmcclain | nati_ueno: looking | 19:05 |
| nati_ueno | markmcclain: I updated bug report https://bugs.launchpad.net/nova/+bug/1209364 | 19:05 |
| uvirtbot | Launchpad bug 1209364 in neutron "AttributeError: 'MockClient' object has no attribute 'EXTED_PLURALS' " [Critical,In progress] | 19:05 |
| *** vijendar has quit IRC | 19:06 | |
| lifeless | clarkb: (but some times things that are not 'mocks' are labelled Mock* :) | 19:06 |
| clarkb | lifeless: I think that is the case here :) | 19:06 |
| *** vijendar has joined #openstack-infra | 19:06 | |
| lifeless | mordred: some scrollback we should follow up on - https://bugs.launchpad.net/openstack-ci/+bug/1209132 and also https://review.openstack.org/#/c/39811/ and also you mentioned services in venvs not starting - do you need a hand? what symptoms do you have? | 19:07 |
| uvirtbot | Launchpad bug 1209132 in openstack-ci "run-mirror fails to mirror cinderclient" [Undecided,New] | 19:07 |
| *** vijendar1 has quit IRC | 19:08 | |
| *** ^d has joined #openstack-infra | 19:10 | |
| openstackgerrit | A change was merged to openstack-infra/config: Unset ZUUL_PROJECT on periodic devstack jobs https://review.openstack.org/40677 | 19:12 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added no-cache option https://review.openstack.org/40651 | 19:12 |
| fungi | on the topic of run-mirror, this is probably a good time to rehash last night's discussion with mordred and lifeless about requirements enforcement/stable branch mirrors in the presence of clarkb, jeblair and sdague | 19:12 |
| clarkb | fungi: we are in a call, but happy to in about 45 minutes | 19:13 |
| fungi | oh, no sweat. i can wait ;) | 19:13 |
| jeblair | lunch calls are evil | 19:14 |
| fungi | evilly...delicious? | 19:14 |
| pleia2 | less evil than 7AM :) | 19:14 |
| lifeless | It *is* a 7AM call. | 19:14 |
| pleia2 | sorry lifeless | 19:14 |
| lifeless | pleia2: :) At least it's not a 6AM call. | 19:15 |
| pleia2 | hehe | 19:15 |
| nati_ueno | markmcclain: I pushed patch, if you don't want revert i'll abandon this patch https://review.openstack.org/40704 | 19:15 |
| mgagne | puppet-lint is finally working on bare puppet modules :D | 19:15 |
| jgriffith | fungi: I do have strong feelings about the double negative logic statements :) | 19:15 |
| fungi | jgriffith: you are entitled to them! | 19:16 |
| fungi | sometimes code review is like group therapy, with red foam bats labeled "-1" | 19:17 |
| jgriffith | fungi: haha | 19:17 |
| jgriffith | fungi: stuff like that I just give my 2 cents but still +1 | 19:18 |
| jgriffith | fungi: not important enough or something that we have a set standard on. Hopefully folks rethink, and often they do | 19:18 |
| jgriffith | good enough for me | 19:18 |
| sdague | man, chasing the project deps makes me understand the facebook model of every line of code in the company being in one git tree | 19:18 |
| jgriffith | sdague: :) | 19:19 |
| *** vipul is now known as vipul-away | 19:19 | |
| *** vipul-away is now known as vipul | 19:19 | |
| fungi | better than every line of code being in it own separate git tree | 19:19 |
| * jgriffith likes that idea | 19:19 | |
| mgagne | atm, is it normal/expected to have a ~45m delay between a "recheck" and the results? | 19:25 |
| clarkb | mgagne: http://status.openstack.org/zuul/ we are seeing resource starvation because everyone else is doing the same thing :) | 19:26 |
| clarkb | mgagne: expected definitely when load ramps up like this. Maybe not normal though | 19:26 |
| openstackgerrit | A change was merged to openstack-infra/config: Prepare to test git-review https://review.openstack.org/40319 | 19:26 |
| *** nijaba has quit IRC | 19:27 | |
| fungi | it's times like this when i wish i didn't have to keep cacti unviewable, because i'd love to see what the actual resource utilization is like on zuul and the jenkins masters | 19:27 |
| fungi | still no updated cacti packages though | 19:27 |
| *** mrmartin has joined #openstack-infra | 19:28 | |
| * fungi uses ssh | 19:28 | |
| jeblair | zuul load average: 0.31, 0.72, 0.79 | 19:28 |
| markmcclain | sdague: we're going to make the code in neutronclient a little more forgiving of a bad mock for the time being… it is the least bad of the solutions | 19:28 |
| *** nijaba has joined #openstack-infra | 19:28 | |
| markmcclain | sdague: long term we need to clean the mock in Nova but I'd like to fix a few warts in the client before doing so | 19:28 |
| *** mrodden has joined #openstack-infra | 19:29 | |
| jeblair | fungi: the jenkins webuis seem responsive, and there is no backlog on devstack-gate update jobs | 19:29 |
| *** moted has joined #openstack-infra | 19:29 | |
| mgagne | clarkb: according to the graph, it isn't "normal". I mean, "patchset created" isn't abnormally high atm. Unless slaves are clogged running lengthy tests. ^^' | 19:29 |
| fungi | jeblair: yeah, the control servers seem to not be heavily loaded, so it's just availability of slave servers/pool size i guess | 19:30 |
| *** ruhe has quit IRC | 19:30 | |
| *** mrodden1 has quit IRC | 19:31 | |
| jeblair | we could probably increase the number of slaves slightly, but i'd like to avoid doing it too much yet until we have a new devstack-gate system | 19:31 |
| fungi | the launch slaves also seem not at all loaded | 19:32 |
| clarkb | mgagne: I think that is a big part of it, there has been an extended period of running long running tests | 19:32 |
| *** thomasbiege1 has joined #openstack-infra | 19:33 | |
| mgagne | clarkb: would there be a way to buy a flash pass to avoid waiting in queue? :D | 19:33 |
| thomasbiege1 | hi | 19:33 |
| clarkb | thomasbiege1: hello | 19:34 |
| fungi | mgagne: the approve button is the priority customer line | 19:34 |
| *** avtar has quit IRC | 19:34 | |
| fungi | mgagne: puts you in the gate queue which gets first dibs on available test resources | 19:35 |
| *** sandywalsh has quit IRC | 19:37 | |
| thomasbiege1 | I am investigating ways to add security tests (against a deployed openstack setup) to the openstack infrastructure, but for me it is not very clear where it would fit best. There is devstack, tempest and smokestack. Additionall deployment tests seem just to focus on integration validation. Maybe you have a hint for me? | 19:39 |
| *** ^d has quit IRC | 19:41 | |
| jeblair | thomasbiege1: perhaps they would be a good fit for tempest, which has a number of different test groupings (including long-running stress tests, for example) | 19:41 |
| jeblair | thomasbiege1: you may want to start by talking to sdague | 19:41 |
| *** ^d has joined #openstack-infra | 19:41 | |
| thomasbiege1 | jeblair: thanks! | 19:42 |
| fungi | thomasbiege1: also, a lot of the discussion around planning tests takes place in #openstack-qa so you might want to ask in there | 19:42 |
| jeblair | thomasbiege1: are you thinking of tests such as fuzz-testing, or something completely different? | 19:42 |
| markmcclain | sdague: fix working its way through the gate | 19:43 |
| *** sarob has quit IRC | 19:43 | |
| thomasbiege1 | fungi: ah, thanks for the hint | 19:43 |
| *** sarob has joined #openstack-infra | 19:43 | |
| thomasbiege1 | jeblair: yes, fuzz testing and web-security testing (based on the OWASP testing guide) | 19:44 |
| *** changbl has joined #openstack-infra | 19:44 | |
| jeblair | thomasbiege1: do you have a really rough idea of how long these tests might take (minutes, hours, days?) | 19:44 |
| thomasbiege1 | jeblair: well the web-security test w/o fuzzing only need minutes | 19:45 |
| * fungi imagines a devstack-gate based test which stands up horizon, installs a local openvas and scans it to generate audit reports | 19:45 | |
| thomasbiege1 | fungi: sounds good :) | 19:46 |
| jeblair | neat | 19:46 |
| *** ^d has quit IRC | 19:46 | |
| jeblair | thomasbiege1: so at least some of this could be considered for check and gate jobs; maybe the fuzz testing would need to run nightly due to length | 19:46 |
| fungi | i guess actually something light like whisker would be sufficient against horizon alone, but openvas might be great to bombard all the devstack services | 19:47 |
| jeblair | thomasbiege1: at any rate, when you find the right home for them, we'll be happy to run them, and it sounds like it shouldn't be too much trouble. | 19:47 |
| *** sarob has quit IRC | 19:47 | |
| thomasbiege1 | jeblair: what is the official/openstack way of introducing such test-suites? | 19:48 |
| fungi | i'm personally extremely excited by the prospect of automated vulnerability scans against proposed changes, if it turns out to be feasable in check and gate jobs | 19:48 |
| dtroyer | thomasbiege1: be aware that DevStack's deployment of OpenStack is not considered secure out of the box. I would love to change that | 19:48 |
| jeblair | thomasbiege1: we strongly favor consolidation into tempest if feasible (sdague and folks in #openstack-qa should be able to help determine that) | 19:49 |
| *** sandywalsh has joined #openstack-infra | 19:49 | |
| sdague | markmcclain: cool, thanks much | 19:49 |
| jeblair | thomasbiege1: it will make it much easier to run that set of tests, as tempest is already well integrated into the infrastructure; and we run tempest on a deployment made by devstack -- so that's the kind of environment you can expect it to test against | 19:50 |
| sdague | thomasbiege1: yes, tempest would be the right place. Probably either in the scenario subdir, or if it grows enough, have a dedicated security directory | 19:50 |
| fungi | thomasbiege1: another good candidate which might be better placed as per-project testing would be static analysis for risky code (looking for potential format string vulrabilities for example) | 19:50 |
| thomasbiege1 | *nod* | 19:51 |
| fungi | either run the way we do coverage reporting, or maybe as a flake8 plugin if the false-positive rate is low enough | 19:51 |
| thomasbiege1 | fungi: code scanning is also on my list | 19:51 |
| jeblair | ++ | 19:51 |
| thomasbiege1 | fungi: but I thnik this can go directly into Jenkins (at least this is the way we implemented it at our company) | 19:52 |
| fungi | thomasbiege1: potentially, though depending on how quickly it runs, being able to lump it in with other static analysis tests might help avoid unnecessary additional resource consumption | 19:53 |
| jeblair | thomasbiege1: tempest has developer documentation here: http://docs.openstack.org/developer/tempest/ | 19:54 |
| thomasbiege1 | ah, thanks for the pointer | 19:54 |
| jeblair | thomasbiege1: and if you want to read up on the project infrastructure, our docs are here: http://ci.openstack.org/ | 19:54 |
| thomasbiege1 | jeblair: read it already :) | 19:55 |
| jeblair | thomasbiege1: the sections on devstack-gate, jenkins job builder, and zuul may be interesting to help you ... | 19:55 |
| jeblair | ok | 19:55 |
| jeblair | :) | 19:55 |
| thomasbiege1 | thanks | 19:55 |
| jeblair | thomasbiege1: that's never happened. that's really cool. :) | 19:55 |
| thomasbiege1 | hahaha :-D | 19:55 |
| fungi | you've just blown jeblair's mind by reading documentation before asking questions | 19:56 |
| *** sandywalsh has quit IRC | 19:56 | |
| *** UtahDave has quit IRC | 19:56 | |
| thomasbiege1 | nevertheless knowing now the best place to start this little project helps me a lot. thank you guys | 19:56 |
| jeblair | thomasbiege1: you're welcome. looking forward to it! | 19:56 |
| fungi | you bet! i'm excited to see this come about | 19:57 |
| thomasbiege1 | :) | 19:57 |
| *** dprince has quit IRC | 19:57 | |
| *** sarob has joined #openstack-infra | 20:00 | |
| mordred | woot! | 20:01 |
| lifeless | mordred: ok so... | 20:01 |
| mordred | fungi: ok - clarkb and I are back - | 20:01 |
| *** krtaylor has quit IRC | 20:01 | |
| *** isviridov has joined #openstack-infra | 20:01 | |
| mordred | fungi: so we can talk about mirrors if you want | 20:01 |
| fungi | mordred: cool | 20:01 |
| mordred | also, apparently run-mirror doesn't work for lifeless | 20:01 |
| fungi | yes, that's the bug he opened i guess? | 20:02 |
| fungi | i haven't tried to reproduce that yet | 20:02 |
| mordred | bugs? | 20:02 |
| mordred | weird. | 20:02 |
| fungi | jeblair: clarkb: sdague: it was suggested last night that maybe the newer strong requirements enforcement we have going into the gate obviates any need for separate stable-branch pypi mirrors, and that the gains from keeping that collapsed into one tree are worth not giving up | 20:03 |
| clarkb | and if that is the case we can just run a full pypi mirror using something snatch | 20:04 |
| clarkb | or snap or snarf | 20:04 |
| clarkb | I forget the name | 20:04 |
| jeblair | fungi: i think i agree with that. | 20:04 |
| jeblair | fungi: the _current_ stable branches are worth considering... | 20:04 |
| fungi | i know we've beaten the topic around previously, but the last time it came up was before the requirements enforcement work lurched forward so significantly | 20:04 |
| mordred | yeah. basically, I think I agree with what clarkb was originally saying | 20:05 |
| mordred | that the mirror should not act as a part of the enforcement of the requirements | 20:05 |
| sdague | fungi: ok, honestly I'd need to think through it. And I'm only a few days away from vacation, so my thinking hat is probably not in the state of usefulness | 20:05 |
| jeblair | in that they are not currently subject to brutal requirements enforcement | 20:05 |
| clarkb | jeblair: good point | 20:05 |
| mordred | jeblair: this is true - however, if we have a mirror which has less things than the outside world... | 20:05 |
| mordred | we won't be testing what life will be like out of our world (which I think is what clarkb was originally trying to tell us) | 20:06 |
| jeblair | mordred: yes -- something needs to enforce requirements, it no longer needs to be the mirror. | 20:06 |
| mordred | sdague: I know your thinking hat isn't on ... | 20:06 |
| jeblair | mordred: i don't know what you're trying to say there talking about worlds | 20:06 |
| mordred | sdague: but what do you thinkabout backporting strong requirements enforcement | 20:06 |
| fungi | sdague: keep your party hat on (i recommend using a lampshade, if available) | 20:06 |
| jeblair | mordred: perhaps i should finish what i was saying... | 20:06 |
| mordred | jeblair: ha. yes lease | 20:06 |
| sdague | mordred: I'm +1 to unified requirements on stable | 20:07 |
| sdague | if that's the question you are asking | 20:07 |
| mordred | sdague: do you think we should just try backporting the patch to devstack stable/grizzly ? | 20:07 |
| sdague | yes | 20:07 |
| sdague | we have requirements stable/grizzly/ | 20:08 |
| sdague | ? | 20:08 |
| clarkb | sdague: iirc yes | 20:08 |
| * clarkb double checks | 20:08 | |
| fungi | sdague: i don't think the requirements list even needs to be unified on stable branches, more just that the mirror itself might not need to be separated out. but i want to wait for whatever jeblair's going to say which makes my assumptions silly after all | 20:08 |
| clarkb | yup | 20:08 |
| sdague | so, here's the thing, I expect that requirements stable/grizzly is totally crap based on the state requirements was in, so my expectation is iteration would be required | 20:08 |
| jeblair | we should probably _not_ clean up old packages in the mirror yet, which stable branches may be relying on | 20:08 |
| mordred | jeblair: +1000 | 20:09 |
| clarkb | sdague: probably so, the initial seed was very naive | 20:09 |
| *** sandywalsh has joined #openstack-infra | 20:09 | |
| mordred | then additionally, I believe we were going to try to cap the folsom branch | 20:09 |
| mordred | so that one should actually be easyish | 20:09 |
| jeblair | and yeah, if we want to backport enforcement to devstack, and spruce up the requirements repo, that could be nice. | 20:09 |
| sdague | clarkb: it turns out we're saved in the devstack-gate case by the fact that we never install the minimum, but always the maximum (unless we've got an apt dep) | 20:09 |
| sdague | but in real people's devstacks, it causes havoc | 20:10 |
| sdague | that was the reason to push through all those updates last night, after figuring out with mtreinish why g-r broke him | 20:10 |
| clarkb | gotcha | 20:10 |
| mordred | btw - I'm going to re-push requirement update patches | 20:10 |
| fungi | g-r... git-review? something else? | 20:10 |
| sdague | global-requirements | 20:10 |
| fungi | THAT. okay | 20:11 |
| clarkb | so it sounds like we have agreement on not worrying about stable branch mirrors if we backport and cleanup | 20:12 |
| clarkb | what about possibly mirroring all the things if external links go away? | 20:12 |
| mordred | I think I'd lke to revisit that if/when external links go away | 20:13 |
| jeblair | clarkb: that would be fine, but possibly in contradiction to our tentatively accepted scope expansion of the mirror as a resource for devs and maybe deployers for things openstack needs. | 20:13 |
| clarkb | ++ has the list gotten any shorter since we started advertising weshouldn't use those projects or get them to push to pypi properly? | 20:13 |
| mordred | clarkb: I believe some of them have hosted now | 20:13 |
| clarkb | jeblair: yes, I think we would want to make it a proper dedicated host in that case | 20:13 |
| clarkb | jeblair: it is definitely not as simple as running that mirror script | 20:14 |
| mordred | I mena- the question becomes, is a partial mirror more helpful than a full mirror | 20:14 |
| mordred | to our devs | 20:14 |
| jeblair | clarkb: i don't think that would help -- i don't believe lifeless and others want to rsync all of pypi.python.org | 20:14 |
| clarkb | my personal interest is for a full mirror so that I don't have to change settings locally | 20:14 |
| mordred | agree | 20:14 |
| jeblair | clarkb: change settings locally? | 20:14 |
| fungi | if they did want that, there are already scripts to do it (several at least) | 20:14 |
| clarkb | I am also not on the other side of the planet so I don't need a local cache | 20:14 |
| clarkb | jeblair: I use our mirror locally, but when touching python things that are not mirrored by our mirror I have to edit files | 20:15 |
| lifeless | clarkb: Full pypi is 53GB | 20:15 |
| lifeless | clarkb: I want the 400MB one that pypi.o.o is. | 20:15 |
| *** adalbas has quit IRC | 20:15 | |
| mordred | yeah - without partial mirror, the pbr integration test would be hard | 20:15 |
| mordred | I think that's my main usecase for the mirror being both small and reconsumable | 20:15 |
| *** dkranz has quit IRC | 20:15 | |
| mordred | if I can grab a copy of the mirror at the top of that | 20:15 |
| clarkb | what if we had two mirrors (I am hating this already) the full mirror which is consumed by the subest mirror | 20:15 |
| jeblair | clarkb: lifeless being on the other side of the planet is not a good reason for us to run a mirror, but he cited a need of devs who are on this side that could benefit from it, and others chimed in and said it would help with their site-local mirror needs too. | 20:15 |
| fungi | clarkb: you shouldn't need to use our mirror any longer since we're not mirroring unreleased tarballs on it any longer, which was the only reason i used to point to it for tests | 20:15 |
| mordred | and then just inject the local to-be-tested pbr | 20:16 |
| mordred | that would be great | 20:16 |
| clarkb | fungi: I use our mirror because it is better than pypi | 20:16 |
| clarkb | you should try it, much more reliable and fast | 20:16 |
| fungi | ah, that's a different argument entirely ;) | 20:16 |
| mordred | if I have to try to inject the to-be-tested pbr intoa full mirror | 20:16 |
| mordred | to test that packages can find and install it automatically properly | 20:16 |
| mordred | then things just got honestly quite difficult | 20:16 |
| mordred | so, since we have a partial mirror pretty much working, I would be quite sad if it went away | 20:17 |
| mordred | because it would make testing some things harder | 20:17 |
| clarkb | mordred: agreed | 20:17 |
| jeblair | mordred: when we can rsync the mirror, perhaps that should be a devstack-gate image prep step so most packages are local. | 20:17 |
| clarkb | I am beginning to think that a full mirror would need to support a partial mirror too which is extra work | 20:17 |
| clarkb | since what we need is the partial mirror | 20:17 |
| mordred | jeblair: +100 | 20:17 |
| mordred | jeblair: I think that would be great | 20:17 |
| fungi | in general i concede that we probably do want to continue having a partial mirror. the question at hand is whether we want more than one | 20:18 |
| thomasbiege1 | cu! | 20:18 |
| jeblair | thomasbiege1: drop in anytime! | 20:18 |
| jeblair | fungi: one is enough for me! | 20:18 |
| mordred | fungi: I'm not sure I can state any currently compelling arguments for two | 20:18 |
| mordred | or more | 20:18 |
| fungi | jeblair: by "more than one" i mean mirror per stable branch | 20:18 |
| mordred | I do think we need to backport reqs stuff to stable/* - but I'll work on htat | 20:18 |
| clarkb | mordred: my only compelling argument is that a full mirror makes local use of the mirror easy | 20:19 |
| fungi | so anyway, based on last night's rethink, i mostly need to know whether i continue down the path of separating out different mirrors for each branch after all, or whether that effort would be better spent on something else (improving the mirroring implementation in general for more robustness, requirements enforcement improvements, whatever) | 20:19 |
| clarkb | mordred: for general pythoness and that is low priority because it isn't strictly openstack related | 20:19 |
| mordred | clarkb: yup | 20:19 |
| mordred | fungi: I think the two things in ()s in there would be more useful overall | 20:19 |
| jeblair | fungi: yep, i say abort. | 20:20 |
| fungi | okay, so i'll update the existing bug on branch mirrors as being an obsolete direction, and then reroute my efforts to those other bits accordingly | 20:20 |
| clarkb | ++ | 20:20 |
| * clarkb can find lunch now | 20:21 | |
| fungi | now, as to lifeless's run-mirror bug report... any ideas on that issue? i can try to reproduce and dissect, but we likely want to narrow that down to a more specific failure | 20:22 |
| *** pabelanger has quit IRC | 20:22 | |
| *** krtaylor has joined #openstack-infra | 20:22 | |
| *** thomasbiege1 has quit IRC | 20:22 | |
| *** thomasbiege1 has joined #openstack-infra | 20:23 | |
| *** sdake has quit IRC | 20:23 | |
| lifeless | so my run-mirror thing is so that I can get a test mirror up and running and not be blocked. It's probably more interesting to push an official mirror forward faster than spend time on my issue. Buttt.... if the official mirror needs a few more days, I'd really appreciate a pointer/aid on that bug. | 20:23 |
| fungi | bug 1209132 | 20:23 |
| uvirtbot | Launchpad bug 1209132 in openstack-ci "run-mirror fails to mirror cinderclient" [Undecided,New] https://launchpad.net/bugs/1209132 | 20:23 |
| *** boris-42 has quit IRC | 20:24 | |
| fungi | and it's also worth repeating for the present crowd that run-mirror's current failure modes make assumptions which aren't actually good when it comes to py3k support | 20:25 |
| fungi | example run: https://jenkins.openstack.org/job/post-mirror-python33/4/console | 20:25 |
| lifeless | ohho this might be it | 20:27 |
| lifeless | cat .pydistutils.cfg | 20:27 |
| lifeless | [easy_install] | 20:27 |
| lifeless | index_url = file:// | 20:27 |
| lifeless | rogue configu files | 20:27 |
| fungi | ah! | 20:27 |
| *** nijaba has quit IRC | 20:27 | |
| lifeless | lets see | 20:27 |
| fungi | that looks like an easy_install optimization to try and use a local on-disk replica | 20:27 |
| *** sdake has joined #openstack-infra | 20:28 | |
| fungi | and does seem to correspond with the exception you quoted | 20:28 |
| *** nijaba has joined #openstack-infra | 20:28 | |
| *** nijaba has quit IRC | 20:28 | |
| *** nijaba has joined #openstack-infra | 20:28 | |
| *** dkranz has joined #openstack-infra | 20:29 | |
| fungi | ha, the script output in your paste even says "Please make sure you remove any previous custom paths from your /home/ubuntu/.pydistutils.cfg file. | 20:29 |
| lifeless | hah | 20:30 |
| lifeless | well there was so much crap in there how could I tell :) | 20:30 |
| *** sdake has quit IRC | 20:30 | |
| fungi | it is rather wordy | 20:30 |
| *** sdake has joined #openstack-infra | 20:31 | |
| *** dina_belova has quit IRC | 20:34 | |
| fungi | this is a beautiful sight, btw... http://logs.openstack.org/04/35104/17/check/gate-git-review-python33/2c67cdb/console.html | 20:37 |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/jenkins-job-builder: Make parameter description optional https://review.openstack.org/40721 | 20:37 |
| clarkb | fungi: !!!!! | 20:38 |
| fungi | yes, !!!1!eleven | 20:38 |
| *** sdake has quit IRC | 20:38 | |
| *** sdake has joined #openstack-infra | 20:38 | |
| *** sdake has quit IRC | 20:38 | |
| *** sdake has joined #openstack-infra | 20:38 | |
| fungi | clarkb: though for some reason subunit2html.py is still not generating a testr_results.html when run from .tox/py33/bin/python | 20:40 |
| *** cppcabrera has joined #openstack-infra | 20:41 | |
| lifeless | I think it would be interesting to stream the subunit straight out and process offline into html | 20:41 |
| *** cppcabrera has left #openstack-infra | 20:41 | |
| *** ^d has joined #openstack-infra | 20:41 | |
| fungi | yeah, and subunit_log.txt exists, so it's something non-obvious happening within subunit2html.py to cause it to silently fail there | 20:41 |
| lifeless | next thing... | 20:43 |
| lifeless | UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 62: ordinal not in range(128) | 20:43 |
| lifeless | pip install did not indicate success | 20:43 |
| lifeless | uhhhh | 20:43 |
| lifeless | looks like it's compiling an LDAP module | 20:43 |
| fungi | too fun | 20:44 |
| fungi | and yes, it will compile things. many nasty things | 20:44 |
| *** mrmartin has quit IRC | 20:45 | |
| fungi | because it has to run setup for each thing it downloads to make sure it catches any other python packages which get pulled in by $random_maintainers_setup_madness in them | 20:45 |
| *** ^d has quit IRC | 20:46 | |
| fungi | one of those things which will hopefully become much simpler in that utopian future where every python package uses consistent tooling | 20:46 |
| s1rp | lifeless: just a heads up: in nova at least ldap as test-requirement is going away https://review.openstack.org/#/c/40261/ | 20:48 |
| fungi | i suspect it's not going away for keystone though | 20:49 |
| *** thomasbiege1 has quit IRC | 20:53 | |
| openstackgerrit | Monty Taylor proposed a change to openstack/requirements: Revert "remove netifaces as a requirement" https://review.openstack.org/40730 | 20:53 |
| mordred | sdague: ^^ | 20:54 |
| *** ^d has joined #openstack-infra | 20:54 | |
| *** ^d has joined #openstack-infra | 20:54 | |
| lifeless | s1rp: sure, but what binary package do I need to be able to build it / | 20:54 |
| clarkb | fungi: I can take a look at the subunit stuff | 20:56 |
| *** pabelanger has joined #openstack-infra | 20:57 | |
| clarkb | fungi: I am just going to go into that workspace and fiddle with running subunit2html. I assume this is ok as you won't be running that test a whole bunch | 20:57 |
| sdague | mordred: ok, why? | 20:59 |
| fungi | clarkb: yeah, i'm actually running it a few times, but it runs quickly | 20:59 |
| mordred | sdague: swift still uses it | 20:59 |
| mordred | sdague: the patch to remove it has not, in fact, landed | 20:59 |
| mordred | and is being contested | 20:59 |
| sdague | oh, ok | 20:59 |
| mordred | sdague: https://review.openstack.org/#/c/38415/ for the record | 20:59 |
| *** melwitt has joined #openstack-infra | 20:59 | |
| sdague | so basically right now the gate is bust if any nova patches go to it | 21:00 |
| sdague | any idea where that fix is sitting for this? | 21:00 |
| jeblair | sdague: i believe https://review.openstack.org/#/c/40706/ is the fix | 21:01 |
| jeblair | sdague: which is at the head | 21:01 |
| jeblair | sdague: though i believe a neutronclient release will need to be made after that | 21:01 |
| sdague | ok, but that requires a neutron client release as well | 21:01 |
| sdague | there was no way to fix it on the nova side? | 21:01 |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Fix python-ldap mirroring. https://review.openstack.org/40732 | 21:02 |
| jeblair | sdague: i don't know if anyone looked into that | 21:02 |
| sdague | oh, ok, that's what I thought was being looked at... :) | 21:02 |
| *** ArxCruz has quit IRC | 21:02 | |
| sdague | oh well | 21:02 |
| clarkb | fungi: I am beginning to think it may be a subunit bug | 21:02 |
| clarkb | fungi: the script runs to completion. No exception thrown (that bubble out at least) | 21:03 |
| clarkb | unless maybe it has to do with some weird bytes vs string thing | 21:03 |
| fungi | clarkb: is the file opened with an encoding specified (that's usually the first thing to bite me)? | 21:04 |
| *** thomasm has quit IRC | 21:04 | |
| *** vipul is now known as vipul-away | 21:05 | |
| *** vipul-away is now known as vipul | 21:05 | |
| dansmith | sdague, clarkb: it's scrolled off my buffer now I think, but what was the outcome? | 21:05 |
| fungi | though i think that's one of the methods which are new in 2.7 so we probably need a conditional | 21:05 |
| clarkb | fungi: no, but the write encodes strings | 21:05 |
| jeblair | mordred: ping | 21:05 |
| sdague | dansmith: neutron client revert, then new release | 21:05 |
| sdague | then things should be ok again | 21:05 |
| clarkb | fungi: and that should be compatible with 2.6 | 21:05 |
| mordred | jeblair: pong | 21:05 |
| fungi | yeah, hrm | 21:05 |
| dansmith | so, we're just sunk right now? | 21:05 |
| sdague | though I actually think it would be fixable with just nova patches as well | 21:05 |
| sdague | dansmith: yep | 21:05 |
| dansmith | alrighty then | 21:06 |
| lifeless | well, that patch won't be enough, but will iterate | 21:06 |
| jeblair | mordred: so i'm starting rough-in work on the new devstack-gate pooling code | 21:06 |
| jeblair | clarkb, fungi: you too | 21:06 |
| mordred | jeblair: ooh, neat | 21:06 |
| fungi | awesome! | 21:06 |
| jeblair | will you take a look at this proposed config file: http://paste.openstack.org/show/43495/ | 21:06 |
| mordred | yup | 21:07 |
| jeblair | and tell me if that looks reasonable... | 21:07 |
| jeblair | mordred: i want to make sure i accomodate a possible move to dib | 21:07 |
| *** isviridov has quit IRC | 21:07 | |
| jeblair | mordred: so i'm expecting the "configuration:" entry to be a shell script that does what the image script currently does for now | 21:07 |
| mordred | yup | 21:08 |
| jeblair | mordred: but possibly something else that specifies how dib could build the image in the future | 21:08 |
| fungi | jeblair: yaml! looks like what we want--has the right info and organized sanely | 21:08 |
| mordred | yeah - we could even collapse _this_ file in to not having a configuration entry | 21:08 |
| mordred | OR | 21:08 |
| jeblair | mordred: so in addition to anything else you think of, specific attention to forward-comptability there would be good | 21:08 |
| mordred | we could keep this as a config file for both things | 21:08 |
| mordred | if we put the elements list into configuration for dib | 21:09 |
| jeblair | mordred: i think (without knowing the terms) that's what i was thinking | 21:09 |
| *** lcestari has quit IRC | 21:10 | |
| fungi | jeblair: i think you inadvertently duplicated your providers key under targets images | 21:10 |
| jeblair | mordred: so maybe we add something like "elements: [puppet, git, pypi-mirror]" later? | 21:10 |
| clarkb | jeblair: can we abstract the provider such that maybe d-g can spin up and connect to gearman directly? | 21:10 |
| clarkb | jeblair: maybe that means having a provider type | 21:10 |
| jeblair | fungi: yes, copy/paste error there | 21:10 |
| clarkb | I might be thinking too far ahead here | 21:11 |
| mordred | then base-image could be the base image to use - which might want us to have an image-name | 21:11 |
| mordred | oh - but there is one | 21:11 |
| mordred | nevermind - I think that's all we need | 21:11 |
| clarkb | but something like that would make it useful for general worker pool management | 21:11 |
| clarkb | eg logstash | 21:11 |
| *** SergeyLukjanov has quit IRC | 21:11 | |
| mordred | jeblair: yes | 21:11 |
| sdague | so... I had this other idea, last one for the day before I run off | 21:11 |
| fungi | jeblair: but otherwise lgtm and should be extensible enough if we want to add more there later | 21:11 |
| jeblair | fungi: https://etherpad.openstack.org/UrYq1SBwwx | 21:11 |
| sdague | to make sure that global-requirements actually worked at the minimums, have a version of the job which runs tempest gate with min versions | 21:12 |
| sdague | because in the base case we basically run at max | 21:12 |
| sdague | so we only test one end of the range | 21:12 |
| jeblair | sdague: sounds good | 21:12 |
| fungi | jeblair: okay, cool. that's what i assumed you meant there | 21:12 |
| mordred | sdague: neat | 21:12 |
| * zaro is addicted to looking at activity on jenkins01 & jenkins02 | 21:13 | |
| sdague | ok, maybe I can actually get that nailed before I start road tripping next week :) | 21:13 |
| fungi | sdague: that would need more work on the mirror script to make sure we actually provided those minimum package versions | 21:14 |
| sdague | fungi: yes, probably | 21:14 |
| jeblair | clarkb: so maybe we do that in the way i just added? | 21:14 |
| sdague | it's going to need a version of update that provides just the min versions | 21:14 |
| jeblair | clarkb: though specifically for logstash, that might be something that we could just handle with heat | 21:14 |
| sdague | that should provide the contents of the list | 21:14 |
| sdague | anyway, I'll start digging on it tomorrow | 21:14 |
| jeblair | clarkb: though if we start spinning up single-use workers that aren't attached to jenkins, we'd need that | 21:15 |
| clarkb | the etherpad looks good | 21:16 |
| clarkb | we can key off the jenkins-url and have a gearman-server uri | 21:16 |
| jeblair | clarkb, mordred: i added a 'cleanup' line to the provider-image section -- so maybe we do something like that to tell it to try to run kexec to cleanup | 21:16 |
| clarkb | something like I added | 21:16 |
| *** vijendar1 has joined #openstack-infra | 21:17 | |
| *** dkranz has quit IRC | 21:17 | |
| mordred | jeblair: yeah. maybe so | 21:17 |
| fungi | sdague: one more wrench in the works... what's the minimum version to test for a package with no specified minimum version in the requirements list? | 21:17 |
| jeblair | clarkb: presumably a logstash node would know how to connect to the logstash gearman server though | 21:17 |
| *** anteaya has quit IRC | 21:17 | |
| clarkb | jeblair: maybe? I am not exactly sure how much data you plan to push into the node from here | 21:18 |
| clarkb | I think either way works | 21:18 |
| jeblair | clarkb: i don't want to be heat. :) | 21:18 |
| clarkb | thats fair | 21:18 |
| jeblair | and when/if heat can maintain pools, i'm happy to start using that | 21:19 |
| jeblair | (though our case is more subtle -- we want pools of _ready/idle_ nodes, not just pools of live nodes) | 21:19 |
| fungi | sdague: also the minimum version specified may provide a lower bound while not specifying an actual released version number explicitly (for example >=2 where 2.0.0 was the actual released package version) | 21:19 |
| *** vipul is now known as vipul-away | 21:20 | |
| *** vijendar has quit IRC | 21:20 | |
| jeblair | fungi, sdague: at least for the case of gating a new requirement, we can reject ones that don't match or don't have a min specified. | 21:21 |
| jeblair | fungi: the issue still holds for something that aimed to run all the requirements at min; so that may be a cleanup item. | 21:21 |
| fungi | jeblair: fair. and maybe we go back and clean up the existing ones lacking a lower bound or without a lower bound on a real released version | 21:21 |
| jeblair | mordred, clarkb, fungi: ok, thanks. back to my hacking hole. | 21:22 |
| mordred | jeblair: in the hole! | 21:22 |
| fungi | the hacking hole. has a nice ring to it | 21:22 |
| mordred | jeblair: actually - I think our case isn't too subtle... | 21:23 |
| fungi | might be a good name for a beer bar/coffee house/cyber cafe/hacker space venture | 21:23 |
| mordred | jeblair: or different. "live" for us means "in a state ready to run devstack" | 21:23 |
| clarkb | fungi: progress! pdb is my friend | 21:23 |
| clarkb | fungi: in _generate_report_test there is string/byte decode/encode happening | 21:23 |
| clarkb | and that breaks under py3k with an AttributeError. Not sure why that doesn't bubble out or what is causing it yet | 21:24 |
| lifeless | kindy run,back soon | 21:24 |
| jeblair | mordred: i don't either, but the thing that checks the low water mark needs to know that something that has started to run a devstack test is not live anymore (despite it needing to exist as a nova server for another hour at least, and possibly might become live again after kexec) | 21:24 |
| mordred | jeblair: + | 21:25 |
| *** nicedice_ has joined #openstack-infra | 21:25 | |
| fungi | clarkb: huh--good catch | 21:27 |
| *** nijaba has quit IRC | 21:27 | |
| *** nijaba has joined #openstack-infra | 21:28 | |
| *** nijaba has quit IRC | 21:28 | |
| *** nijaba has joined #openstack-infra | 21:28 | |
| clarkb | fungi: I think I can just remove that section. I am testing | 21:29 |
| mordred | jog0: ping | 21:31 |
| jog0 | mordred: pong | 21:31 |
| fungi | i hadn't thought about it this way before, but our devstack pools are managed in much the same algorithm as apache's worker process pools... try to maintain at least x available workers and don't let it exceed y total workers | 21:32 |
| fungi | and also don't keep more than z available workers | 21:32 |
| fungi | thought i guess we don't do that last bit (yet at least) | 21:33 |
| *** UtahDave has joined #openstack-infra | 21:33 | |
| mordred | jog0: wanted to do high bandwidth about requirements sync for hacking | 21:33 |
| fungi | but would probably want to if we ever start moving nodes back from the used pool into the available pool | 21:33 |
| mordred | jog0: two things - a, we're close to landing a script that will auto-propose requirements patches when requirements changes | 21:34 |
| mordred | jog0: b- I can do a larger commit message this time, but moving forward, it's going to be a bot making these, that ok with you? | 21:34 |
| jog0 | mordred: yes and yes | 21:34 |
| mordred | jog0: ossum | 21:35 |
| jog0 | mordred: I figured as much, I am just want to make sure got logs are readable by someone who doens't know the context | 21:35 |
| fungi | after the first one, he can simply berate the bot for not providing sufficient context in the commit message ;) | 21:36 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/hacking: Synced with global requirements https://review.openstack.org/40065 | 21:36 |
| jeblair | fungi: we have bots that can berate our bots. :) | 21:36 |
| * jeblair loves the idea of jog0's avatar in hacking -1ing monty's avatar in requirements-update. | 21:37 | |
| mordred | jeblair, clarkb: could I get a hell yeah on : https://review.openstack.org/#/c/40615/ | 21:37 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: ignore instead of fail when reading empty config files https://review.openstack.org/38520 | 21:37 |
| jeblair | mordred: does it work? | 21:38 |
| jog0 | jeblair: :) | 21:38 |
| mordred | jeblair: it will once we land the pep8 job patch to requirements | 21:38 |
| *** jerryz has joined #openstack-infra | 21:38 | |
| mordred | jeblair: which I've been asked to hold off on until we land this patch | 21:38 |
| jeblair | mordred: which is ready? the intent is to block in on that. | 21:38 |
| jeblair | ok | 21:38 |
| mordred | yup | 21:38 |
| jeblair | i aprvd | 21:39 |
| mordred | thx | 21:39 |
| *** avtar has joined #openstack-infra | 21:40 | |
| zaro | fungi: would you mind triaging bug 1208901 ? | 21:40 |
| uvirtbot | Launchpad bug 1208901 in openstack-ci "deploy jenkins plugin pom.xml file " [Undecided,New] https://launchpad.net/bugs/1208901 | 21:40 |
| fungi | zaro: you bet | 21:41 |
| zaro | fungi: i've already got a patch on gerrit but comments don't show up on the bug. | 21:41 |
| fungi | zaro: that seems wrong. how about instead i track down why that didn't get updated | 21:41 |
| zaro | jeblair: got time to review https://review.openstack.org/#/c/40455/ ? | 21:42 |
| fungi | zaro: which was the patch? | 21:42 |
| zaro | fungi: ohh yeah. that would be nice. | 21:42 |
| zaro | fungi: one i just mentioned to jeblair | 21:42 |
| fungi | ahh, the one you just linked | 21:42 |
| fungi | yeah | 21:42 |
| openstackgerrit | A change was merged to openstack-infra/config: Added pep8 checks to requirements https://review.openstack.org/40615 | 21:43 |
| zaro | fungi: thought it was because bug needed to be triaged first, but that does seem wrong. | 21:43 |
| zaro | fungi: you can review that one too if you've got time. | 21:44 |
| fungi | zaro: right, update_bug.py does not need the bug to be triaged | 21:44 |
| *** datsun180b has quit IRC | 21:44 | |
| *** sarob has quit IRC | 21:44 | |
| fungi | i'm going hunting in the gerrit logs again | 21:44 |
| *** dina_belova has joined #openstack-infra | 21:44 | |
| *** sarob has joined #openstack-infra | 21:45 | |
| *** avtar has quit IRC | 21:45 | |
| *** dina_belova has quit IRC | 21:47 | |
| *** krtaylor has quit IRC | 21:47 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make subunit2html.py py3k and python2 compatible. https://review.openstack.org/40743 | 21:48 |
| clarkb | fungi: ^ I used the wrong topic branch for that >_> but that shouldn't affect anything | 21:48 |
| *** sarob has quit IRC | 21:49 | |
| *** vipul-away is now known as vipul | 21:50 | |
| *** avtar has joined #openstack-infra | 21:51 | |
| *** woodspa has quit IRC | 21:51 | |
| markmcclain | ok.. new neutronclient in pypi that should enable the nova unittests to pass | 21:53 |
| mordred | markmcclain: woot! | 21:53 |
| mordred | just so you all know - I'm currently testing the update-requirements script by running it by hand and havingit submit changes | 21:55 |
| *** rfolco has quit IRC | 21:56 | |
| mgagne | Do you mind if I tag bugs related to JJB with jjb? | 21:57 |
| mgagne | but I don't think there's more than 1 bug tbh | 21:58 |
| clarkb | mgagne: fine wiht me | 21:58 |
| fungi | mgagne: works for me. doesn't look like we have an existing tag for jenkins job builder | 21:58 |
| fungi | so jjb's as good as anything there, i suppose | 21:58 |
| mgagne | Is jjb fine with you or do people prefer the long form? | 21:58 |
| *** nayward has quit IRC | 21:59 | |
| nati_ueno | sdague: jeblair: neutronclient fix is merged. Nova is fixed now? | 21:59 |
| *** vijendar1 has quit IRC | 22:01 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make subunit2html.py py3k and python2 compatible. https://review.openstack.org/40743 | 22:02 |
| clarkb | fungi: Alex_Gaynor ^ now with proper topic and file closing | 22:02 |
| *** dkranz has joined #openstack-infra | 22:02 | |
| Alex_Gaynor | clarkb: Awesome, I'm no longer qualified to review :) | 22:03 |
| clarkb | fungi: I have generated html files with one subunit log under python 2.7 and python3.2 and under python2.7 using the old version | 22:03 |
| clarkb | there is no diff between new and old python27 outputs, There is a slight diff with py3k and I think it is related to how classes are repr'd as strings | 22:03 |
| fungi | makes sense | 22:04 |
| *** mriedem has quit IRC | 22:04 | |
| fungi | also, neat use of "with" to effect the equivalent of close() | 22:04 |
| clarkb | fungi: I prefer it in the general case so that you don't have to worry about exceptions and branching logic | 22:05 |
| fungi | ahh, yeah leaves it up to runtime gc to take care of for you. very neat | 22:06 |
| fungi | i'll have to remember to leverage that trick in the future | 22:06 |
| *** mrodden has quit IRC | 22:08 | |
| *** fbo_away is now known as fbo | 22:08 | |
| *** _TheDodd_ has quit IRC | 22:09 | |
| *** carlp-away is now known as carlp | 22:09 | |
| *** arosen1 has quit IRC | 22:11 | |
| *** arosen1 has joined #openstack-infra | 22:14 | |
| pabelanger | Not sure the place to ask, hopefully somebody will point me in the right direction. Am I correct is seeing that test repository (the test runner specifically) launches unit tests in parallel by default? | 22:16 |
| zaro | clarkb, fungi : talk about https://review.openstack.org/#/c/38663/ | 22:16 |
| clarkb | pabelanger: yes | 22:17 |
| clarkb | pabelanger: well not by default, but by how we run it (with the --parallel flag) | 22:17 |
| clarkb | pleia2: it will fork X test runners where X is the number of available cpus giving each runner some partition of the test set | 22:17 |
| zaro | fungi: sorry but i'm not a fan of your suggestion. putting a job in a time window doesn't seem good to me. | 22:17 |
| clarkb | pabelanger: ^ whoops | 22:17 |
| clarkb | zaro: I don't like the suggestion either as 24 hours off isn't major imo | 22:18 |
| zaro | clarkb: would it be better to create a project with python-grizzly-bitrot-jobs and folsom-bitrot-jobs then call the job in layout.yaml | 22:19 |
| clarkb | zaro: that said, I think this is a less problematic issue when we push all of this extra requirements checking down into the stable branches | 22:19 |
| pabelanger | clarkb, okay, I'm actually trying to get them to run sequentially but actually seeing some run in parallel for some reason | 22:19 |
| pabelanger | moving to nosetest works as I expect | 22:19 |
| clarkb | pabelanger: we want them to run in parallel :) if you really need sequential for debugging reasons you can source the venv and do `testr run` | 22:19 |
| *** fbo is now known as fbo_away | 22:20 | |
| mordred | pabelanger: why you want sequential? | 22:20 |
| clarkb | zaro: I am beginning to think we may be able to just let this die on the vine | 22:20 |
| *** burt has quit IRC | 22:20 | |
| zaro | clarkb: ohhh?? | 22:20 |
| clarkb | zaro: because we will end up testing this stuff through other mechanisms | 22:20 |
| clarkb | zaro: see the discussion that began at about 1pm today about the mirror and stable branches | 22:20 |
| clarkb | zaro: basically we will add extra machinery around the stable branches to keep them from bitrotting against the requirements | 22:21 |
| zaro | clarkb: cool. how do we let stuff die in gerrit? | 22:21 |
| *** sarob has joined #openstack-infra | 22:22 | |
| clarkb | zaro: I would make sure fungi agrees as he has all of this stuff in his head right now. But you can abandon that change and update the bug if this is the route we want to take | 22:22 |
| pabelanger | clarkb, mordred: I should have prefaced, this is for my own app. Dealing with an race condition to an external redis server (yes, integration testing using unit tests) | 22:23 |
| pabelanger | moving sequential is the simple solution for now | 22:23 |
| zaro | fungi: d'you get that? | 22:24 |
| clarkb | pabelanger: I would solve that by adding different redis queues for each test runner | 22:24 |
| clarkb | pabelanger: each runner can create one called runner_$PID or something | 22:24 |
| clarkb | pabelanger: then they won't trample each other | 22:24 |
| clarkb | s/queues/channel/ or whatever redis calls it | 22:24 |
| *** carlp is now known as carlp-away | 22:25 | |
| clarkb | jeblair: https://review.openstack.org/#/c/39995/ do you want to do any additonal testing on that before we approve it? | 22:25 |
| clarkb | jeblair: I am not terribly worried about it, maybe we merge it during a quiet time | 22:25 |
| clarkb | pabelanger: that is the approach we have been working towards when tests need to talk to DBs. each process gets its own schema | 22:26 |
| jeblair | clarkb: agreed | 22:26 |
| pabelanger | clarkb, Ya, thats likely what I will end up doing. Just got side tracked with testr and the race condition. Thanks for the pointers | 22:27 |
| *** nijaba has quit IRC | 22:27 | |
| mordred | sdague: we have a bug in the update script | 22:28 |
| *** jhesketh has joined #openstack-infra | 22:28 | |
| *** nijaba has joined #openstack-infra | 22:29 | |
| fungi | zaro: sorry, had stepped away for just a moment | 22:29 |
| *** changbl has quit IRC | 22:30 | |
| fungi | zaro: i'm not sure putting the bitrot jobs as children of the periodic mirror jobs in layout.yaml will work. i suspect that zuul's timed triggers may be more naive than that, but jeblair will know for sure | 22:31 |
| jog0 | anyone seeing this kind of error: http://logs.openstack.org/33/40333/2/check/gate-nova-python27/fa6a098/console.html | 22:31 |
| fungi | zaro: clarkb: i'm not convinced that backporting requirements changes to the stable branches will solve the bitrot-job-catches-upstream-release-breakage issues which the bug in question was filed to address by running bitrot jobs sooner after mirror updates | 22:33 |
| clarkb | jog0: yes, it was caused by a recent neutronclient release | 22:33 |
| *** prad_ has quit IRC | 22:33 | |
| *** weshay has quit IRC | 22:33 | |
| clarkb | jog0: neutron has since released a new version of neutronclient that reverts the changes made to the client that break nova testing | 22:33 |
| clarkb | jog0: tests after the new version is in the mirror (which is now I think) should pass | 22:34 |
| * clarkb double checks the mirror | 22:34 | |
| clarkb | http://pypi.openstack.org/openstack/python-neutronclient/ verion 2.2.6 is the one we want and it is present | 22:34 |
| fungi | zaro: clarkb: basically my interpretation of that bug report was an attempt to shorten the window between when the mirror pulls in new upstream releases and when the bitrot jobs report that something broke because of it, so having them not run almost 24 hours after the mirror update would give more of a leg up on timely response and fix | 22:35 |
| clarkb | fungi: correct, but as part of the no stable mirror branches we will be directly testing this stuff | 22:36 |
| jog0 | clarkb: gah thanks | 22:36 |
| clarkb | fungi: so we don't need to run the bitrot jobs after $thing happens, there will be dedicated jobs that servce this purpose | 22:36 |
| Alex_Gaynor | Do we keep track of how many fails occur in the gating queue and on which projects/test suites/etc.? | 22:36 |
| fungi | clarkb: i can probably be convinced that we'll be running the equivalent of bitrot jobs more often than every 24 hours, and that makes the bitrot jobs themselves no longer necessary | 22:37 |
| clarkb | fungi: at least that was my understanding of this afternoons conversation | 22:37 |
| clarkb | Alex_Gaynor: we do | 22:37 |
| *** thomasbiege1 has joined #openstack-infra | 22:37 | |
| clarkb | Alex_Gaynor: let me get you a link | 22:37 |
| clarkb | Alex_Gaynor: | 22:37 |
| clarkb | http://graphite.openstack.org/render/?width=586&height=308&_salt=1375457125.913&target=alias%28summarize%28stats_counts.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.FAILURE%2C%221h%22%29%2C%20%27Failure%27%29&target=alias%28summarize%28stats_counts.zuul.pipeline.gate.job.gate-tempest-devstack-vm-neutron.SUCCESS%2C%221h%22%29%2C%20%27Success%27%29&title=Neutron%20Gate%20Results%20per%20Hour | 22:37 |
| clarkb | for example | 22:37 |
| jeblair | Alex_Gaynor: you can go spelunking in http://graphite.openstack.org/ for more | 22:37 |
| fungi | death by url | 22:37 |
| clarkb | fungi: sorry | 22:37 |
| Alex_Gaynor | thanks | 22:37 |
| Alex_Gaynor | has anyone looked at setting up a gdash instance with common graphs? | 22:38 |
| Alex_Gaynor | https://github.com/ripienaar/gdash | 22:38 |
| * fungi has never heard of such a thing | 22:38 | |
| clarkb | I haven't but it looks shiny | 22:38 |
| fungi | ho, nifty | 22:39 |
| jeblair | Alex_Gaynor: the graphs at the bottom of the zuul status page come from graphite; i think it would be nice to add them to other pages there | 22:39 |
| *** thomasbiege1 has quit IRC | 22:39 | |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Fix python-ldap mirroring. https://review.openstack.org/40732 | 22:39 |
| fungi | it does indeed seem full of fancy | 22:39 |
| jeblair | Alex_Gaynor: the jquery module it uses makes it pretty easy | 22:40 |
| clarkb | jeblair: I am ferrying people from the airport tonight/this evening and don't expect to have much keyboard time until Sunday... I can approve 39995 on Sunday if no one beats me to it | 22:44 |
| jeblair | clarkb: roger | 22:44 |
| fungi | zaro: your change for bug 1208901 seems not to have resulted in any exceptions from update_bug.py so i wonder if maybe your bug mention in the commit message is insufficient to trigger it to update the bug at all | 22:46 |
| dansmith | clarkb: so we should be rechecking now? | 22:46 |
| uvirtbot | Launchpad bug 1208901 in openstack-ci "deploy jenkins plugin pom.xml file " [Undecided,New] https://launchpad.net/bugs/1208901 | 22:46 |
| Alex_Gaynor | mordred: I assume it's not intentional that https://review.openstack.org/#/c/40749/1 has 4 commit-ids? | 22:47 |
| zaro | fungi: that would be odd, because that's my typical message. i've written the same message on many other patches. | 22:49 |
| clarkb | dansmith: I think so, maybe try one change first and if that passes do them all | 22:49 |
| dansmith | clarkb: was there a bug filed? | 22:49 |
| clarkb | dansmith: I didn't file one but I think nati_ueno did | 22:49 |
| zaro | fungi: it usually goes like "This is a fix for bug ..." | 22:49 |
| clarkb | dansmith: https://bugs.launchpad.net/nova/+bug/1209364 | 22:49 |
| uvirtbot | Launchpad bug 1209364 in python-neutronclient "AttributeError: 'MockClient' object has no attribute 'EXTED_PLURALS' " [Critical,Fix committed] | 22:49 |
| dansmith | clarkb: thanks | 22:49 |
| nati_ueno | dansmith: could you share if it working now? | 22:50 |
| nati_ueno | dansmith: code is fixed | 22:50 |
| clarkb | mirror is updated too so all the pieces should be in place | 22:50 |
| dansmith | nati_ueno: just reverified this: https://review.openstack.org/#/c/40648/ | 22:50 |
| dansmith | nati_ueno: so we'll see :) | 22:51 |
| zaro | fungi: however.. i do notice a period at the end of the bug number, not sure if that makes a difference but i don't usually end with a period. | 22:51 |
| nati_ueno | dansmith: Thanks! | 22:51 |
| clarkb | zaro: that may break the regex | 22:52 |
| clarkb | zaro: also you should use the new syntax :) | 22:52 |
| fungi | zaro: with _TheDodd_'s recent header implementation in the script, it might no longer match on precisely the same sentences it did before | 22:52 |
| clarkb | closes-bug: #XXXXXX | 22:52 |
| fungi | what clarkb said | 22:52 |
| fungi | i suspect the "to" between "fix" and "bug" might be the culprit there | 22:53 |
| fungi | but need to look back over the new patterns more closely | 22:54 |
| fungi | i think <prefix> is one word bounded by whitespace | 22:56 |
| *** rfolco has joined #openstack-infra | 22:56 | |
| fungi | https://github.com/openstack-infra/jeepyb/blob/master/jeepyb/cmd/update_bug.py#L335 | 22:56 |
| fungi | so if you had "Fixes bug bug 1172419." at the start of a line (that's what i used to have in mine as a one-liner paraghraph) it would match | 22:57 |
| uvirtbot | Launchpad bug 1172419 in openstack-ci "build bitrot jobs after mirror update" [Low,In progress] https://launchpad.net/bugs/1172419 | 22:57 |
| fungi | er, s/bug bug/bug/ there (paste error on my part) | 22:57 |
| zaro | fungi: check this one out: https://review.openstack.org/#/c/32263/ | 22:58 |
| zaro | fungi: that one worked with period and all. message didn't even start with "Fixes.." | 22:59 |
| zaro | fungi: but in any case i will start using new syntax. | 22:59 |
| fungi | zaro: yeah, remember that the new regular expressions went into update_bug.py last week. that change you just linked is a couple months old | 22:59 |
| zaro | fungi: i see. | 23:00 |
| fungi | what i'm trying to say is that _TheDodd_'s implementation was only mostly backward compatible, not 100% backward compatible | 23:00 |
| clarkb | pleia2: fungi: https://review.openstack.org/#/c/40253/ any reason to not go ahead and approve that? | 23:00 |
| pleia2 | clarkb: that would should be fine | 23:01 |
| pleia2 | 37794 will require baby sitting (if it messes up the gerrit replication file for git, we will all be unhappy) | 23:01 |
| clarkb | 40253 approved | 23:02 |
| pleia2 | \o/ | 23:02 |
| clarkb | pleia2: just people that use github >_> | 23:02 |
| clarkb | I suppose the local replica would be affected too... | 23:02 |
| fungi | i'm okay with babysitting and manually fixing gerrit this evening if 37794 causes issues | 23:02 |
| pleia2 | clarkb: heh, yeah | 23:02 |
| clarkb | fungi: let me review 37794 then | 23:02 |
| clarkb | and you can babysit it :) | 23:02 |
| pleia2 | fungi: that would be great, I'll be around most of the evening (have a 12:40 flight tonight to philly, so I have my evening before airport) | 23:02 |
| mgagne | clarkb: XML element is empty as in <description/> | 23:03 |
| openstackgerrit | A change was merged to openstack-infra/config: Add httpd ssl support to git.openstack.org https://review.openstack.org/40253 | 23:03 |
| clarkb | mgagne: is that not equivalent? | 23:03 |
| clarkb | my knowledge of the intricasies of xml is not good | 23:03 |
| mgagne | clarkb: a pedantic would say no, but in the end, I guess it makes no difference | 23:04 |
| fungi | but i can try pulling 37794 onto review-dev first for good measure | 23:04 |
| pleia2 | fungi: if you have time, it may be worthwhile | 23:04 |
| mgagne | clarkb: I'll test and see if the UI is impacted | 23:04 |
| fungi | pleia2: i'm making time to do that right now | 23:05 |
| clarkb | mgagne: perfect | 23:05 |
| pleia2 | fungi: thanks :) | 23:05 |
| *** nayward has joined #openstack-infra | 23:06 | |
| *** pentameter has quit IRC | 23:06 | |
| *** lcheng has quit IRC | 23:08 | |
| fungi | pleia2: bad news :( | 23:08 |
| clarkb | pleia2: 37794 reviewed | 23:08 |
| fungi | pleia2: " | 23:08 |
| fungi | err: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter replication at /opt/config/fungi/modules/openstack_project/manifests/gerrit.pp:176 on node review-dev.openstack.org | 23:08 |
| clarkb | pleia2: fungi: I think one of my comments is half the story there | 23:08 |
| clarkb | pleia2: you have $giturl whcih should be removed and need $replication | 23:08 |
| pleia2 | clarkb: ah, I missed one! | 23:09 |
| fungi | pleia2: while i'm testing, if you can get that change uploaded fairly soon i'll re-test | 23:09 |
| *** nayward has quit IRC | 23:10 | |
| fungi | we'll get the configuration applying cleanly and then i'm happy to approve and keep an eye on it | 23:10 |
| *** wu_wenxiang has joined #openstack-infra | 23:10 | |
| * fungi will brb | 23:11 | |
| clarkb | dstufft: I am going to put my mordred hat on | 23:15 |
| clarkb | dstufft: how is https://github.com/pypa/pip/issues/989 not a bug if you may need to require new pip | 23:15 |
| clarkb | dstufft: specifically mordred wants to require new pip everywhere. `pip freeze` should reflect that | 23:15 |
| clarkb | also pip list and pip freeze seem redundant | 23:16 |
| dstufft | pip list does more than just show the current installed version | 23:16 |
| dstufft | we talked about making freeze be pip list --freeze | 23:17 |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/jenkins-job-builder: Fix handling of optional parameter description https://review.openstack.org/40721 | 23:17 |
| dstufft | but nobody ever did it and those names already existed | 23:17 |
| clarkb | list is new | 23:17 |
| clarkb | I had to upgrade to 1.4 to get it | 23:17 |
| clarkb | anyways meh | 23:17 |
| dstufft | clarkb: er I mean pip freeze already existed | 23:18 |
| clarkb | mordred: ^ that may potentially be a problem if you want ot require pip 1.4 everywhere | 23:18 |
| dstufft | clarkb: I'm not sure how that'd work though | 23:18 |
| dstufft | if you're on pip 1.3, and you did pip install -r requirements.txt and it has pip==1.4.1 | 23:18 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add replication of git from gerrit to git.o.o https://review.openstack.org/37794 | 23:18 |
| dstufft | what is pip suposed to do? | 23:18 |
| clarkb | dstufft: upgrade pip | 23:18 |
| dstufft | and then exit? | 23:18 |
| dstufft | it can't upgrade the currently running code | 23:19 |
| clarkb | no, I think pip has already established it cannot handle upgradin setup requires and setting up a thing in the same process | 23:19 |
| clarkb | dstufft: the current behavior that requires you to run pip twice is probably fine | 23:19 |
| clarkb | dstufft: eg if you require new setuptools | 23:19 |
| dstufft | clarkb: I don't think there's a sane way of doing it otherwise, we'd have to somehow completely unload pip, evict it from the sys.modules cache, uninstall and install pip, then reimport pip and finish processing the requirements.txt file | 23:20 |
| dstufft | all without closing the process | 23:20 |
| clarkb | dstufft: the sane way is you fork and exec yourself | 23:20 |
| clarkb | but the current behavior is established | 23:21 |
| dtroyer | are we still requiring new setuptools? I thought I saw yesterday that the distro setuptools was generally ok now that d2to1 and friends are gone. | 23:21 |
| clarkb | dtroyer: we are not requiring new setuptools, but mordred does still want to require new pip iirc | 23:21 |
| clarkb | dstufft: and by sane I mean probably full of problems that will need work arounds | 23:22 |
| dtroyer | clarkb: that's what I thought. btw, I've switched to installing pip 1.4 from source in tools/installpip.sh and it appears to be working cleanly everywhere | 23:22 |
| dstufft | oh yea, I need to make a new release of pip in half an hour | 23:22 |
| clarkb | dstufft: but today if you want to upgrade anything in the path of installing stuff (pip,setuptools,distribute,etc) you must do two passes | 23:22 |
| clarkb | dstufft: which is essentially a fork and exec | 23:22 |
| clarkb | I just happen to do it with my shell instead :) | 23:23 |
| dstufft | clarkb: well upgrading setuptools you only needed to because of a bug in pip | 23:23 |
| dstufft | well sort of a bug | 23:23 |
| clarkb | dstufft: or when one of your dependencies thinks you need it eg mysql-python | 23:23 |
| dstufft | clarkb: no that should be fixed in pip 1.4 | 23:23 |
| clarkb | nice | 23:24 |
| clarkb | I should upgrade pip on my local machine then | 23:24 |
| clarkb | maybe after you push that new release with the bug fixes :) | 23:25 |
| notmyname | mordred: is https://review.openstack.org/#/c/40286/ supposed to have 2 change ids? | 23:25 |
| Alex_Gaynor | notmyname: you only got 2 change-ids? I saw one with 5 of them :P I assume it's not intentional, but I have no idea if it causes problems | 23:26 |
| notmyname | Alex_Gaynor: heh | 23:26 |
| dstufft | clarkb: the problem was sort of a perfect storm of things, basically because distribute was empty and forced install to setuptools, and the way pip handles upgrading setuptools and such it meant that if you were using distribute already and you tried to upgrade distribute (which means either setuptools or distribute becasue distribute <= 0.7 internally rewrote setuptools to distribute) it would uninsall distribute (which you were using) | 23:26 |
| dstufft | and attempt to install the "blank" distribute (which fails) | 23:26 |
| dstufft | fails because you no longer have a ``import setuptools`` to handle the installation | 23:27 |
| dstufft | normally upgrading ditribute/setuptools was handled just by making the CWD the build directory so it ``import`` the setuptools package from the to be installed setuptools/distribute, but the blank one didn't have that | 23:27 |
| fungi | notmyname: Alex_Gaynor: that definitely looks like scriptfail (on its way to botfail if you hadn't pointed it out) | 23:27 |
| dstufft | clarkb: if that makes sense? It's kind of crazy | 23:28 |
| clarkb | notmyname: Alex_Gaynor: if you look at patchset one it has a single change id. I bet it was unintentional. It won't break anything other than looking up the change in gerrit might take a couple tries :) | 23:28 |
| clarkb | dstufft: I am still trying to grok. It sounds crazy :) | 23:28 |
| *** nijaba has quit IRC | 23:28 | |
| clarkb | dstufft: I see. Using the CWD broke because it was empty | 23:29 |
| *** nijaba has joined #openstack-infra | 23:29 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Fix double-declaration of cgit class https://review.openstack.org/40770 | 23:29 |
| pleia2 | ^^ pretty sure that fixes what puppet is hiccuping on for ssl | 23:29 |
| uvirtbot | pleia2: Error: "^" is not a valid command. | 23:29 |
| * pleia2 pets uvirtbot | 23:29 | |
| clarkb | pleia2: you fixed one resource, where is the second? | 23:30 |
| clarkb | oh there it is | 23:30 |
| clarkb | pleia2: you need to remove line 30 | 23:30 |
| dstufft | clarkb: yea; pip 1.4 does some hackish shit to fix that. But it should get even better in the future. But upgrading setuptools setuptools should basically work (otherwise it's a bug) because we basically do fork/execute (subprocess) | 23:31 |
| fungi | pleia2: still, yay though! http://paste.openstack.org/show/43512 | 23:31 |
| clarkb | dstufft: it does mostly basically work | 23:31 |
| pleia2 | clarkb: hm, how does that work? | 23:32 |
| pleia2 | fungi: that makes me happy! | 23:32 |
| clarkb | dstufft: the issue is when a thing you are installing requires a newer version of setuptools. pip will upgrade it but pip has already imported the old version so the new one doesn't get used and dependency bombs out | 23:32 |
| dstufft | clarkb: pip doesn't import setuptools | 23:32 |
| dstufft | at all | 23:32 |
| clarkb | dstufft: sorry not pip, the setup.py of the things being installed | 23:32 |
| fungi | pleia2: though i think you need to wrap whole lines in the template rather than just the values? | 23:33 |
| pleia2 | clarkb: actually, I get it, nm :) | 23:33 |
| pleia2 | fungi: ooh, yeah | 23:33 |
| pleia2 | fungi: fixing | 23:33 |
| dstufft | clarkb: yea, that'll bomb because it got uninstalled before installing the new one (fixed in 1.4 supposidly, I wasn't the one working on that issue though) | 23:33 |
| clarkb | dstufft: when I noticed it it bombed because the version was wrong and not because the old one got uninstalled | 23:34 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Fix double-declaration of cgit class https://review.openstack.org/40770 | 23:34 |
| clarkb | dstufft: eg it upgraded just fine but the old stuff was loaded into memory | 23:34 |
| dstufft | clarkb: old stuff shouldn't be loaded into memory, each setup.py invocation is it's own subprocess so it's own process and it's own memory | 23:34 |
| fungi | pleia2: actually, it looks like maybe you did wrap the entire line for each of those in a conditional check, but for some reason != "" is matching even when the value is nonempty | 23:34 |
| pleia2 | fungi: yeah | 23:35 |
| clarkb | dstufft: yeah but there is only one invocation | 23:35 |
| pleia2 | hrm | 23:35 |
| fungi | er, even when the value is empty | 23:35 |
| dstufft | clarkb: no, it does setup.py egg_info to get a list of deps, then later on it does setup.py install in a seperate process | 23:35 |
| pleia2 | copied that logic from gerrit.config.erb | 23:35 |
| *** changbl has joined #openstack-infra | 23:35 | |
| clarkb | dstufft: but dependencies install in the setup.py install process right? | 23:35 |
| fungi | pleia2: yeah, i'm not sure what rubbet devilry is afoot there | 23:35 |
| dstufft | clarkb: no | 23:36 |
| dstufft | clarkb: well they do for openstack | 23:36 |
| dstufft | becuase you guys call pip yourself | 23:36 |
| pleia2 | heh, rubbet :) | 23:36 |
| dstufft | instead of letting pip handle things | 23:36 |
| dstufft | IIRC | 23:36 |
| clarkb | dstufft: well we saw this with tox before the pbr madness | 23:36 |
| pleia2 | fungi: ooh, I think I have some syntax trouble, fixing up | 23:36 |
| pleia2 | hm, no | 23:36 |
| clarkb | tox calls pip -r file and mysql-python cannot install because it requires distribute >= 06.28 abut virtualenv bundled 0.6.lessthan28 | 23:36 |
| clarkb | pip was installing 0.6.28 it just wasn't imported into the process doing the execution | 23:37 |
| fungi | pleia2: oh, yes i think you have -%> in some places you shouldn't? | 23:37 |
| clarkb | the eventual work around there was to tell everyone to upgrade virtualenv to get a newer distribute bundle | 23:37 |
| pleia2 | fungi: that's what I thought, but it actually seems consistent | 23:37 |
| dstufft | clarkb: i'd have to look to see closer. gotta run for a few minutes right now. But if you get it again and can reproduce it let me know | 23:38 |
| pleia2 | -%> should be in the if statement, %> otherwise | 23:38 |
| clarkb | but another workaround was to upgrade distribute first in the virtualenv then run tox | 23:38 |
| clarkb | s/tox/pip or tox/ | 23:38 |
| clarkb | dstufft: let me try | 23:38 |
| dstufft | clarkb: but that should be 100% supported IMO without hacks on your end (assuming the pbr stuff isn't making it happen) | 23:38 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/config: A few updates for RHEL based systems https://review.openstack.org/37789 | 23:38 |
| dstufft | clarkb: I'll be back in 10min or so | 23:38 |
| pleia2 | fungi: I'm thinking maybe the variables aren't being expanded when it's <% rather than <%= | 23:38 |
| *** rfolco has quit IRC | 23:39 | |
| fungi | oh, perhaps | 23:40 |
| pleia2 | I'm just pattern matching though, I have no idea how this template magic works :) | 23:41 |
| fungi | i'm actually wondering if != "" will ever match a hash value | 23:42 |
| pleia2 | ah, yeah, what would an empty hash value look like? | 23:42 |
| fungi | it's possible it gets translated on insertion but doesn't straight up compare against a string | 23:44 |
| * fungi is stabbing in the dark, being somewhat of a clueless ruby rube | 23:44 | |
| pleia2 | looks like ruby uses .empty .nil syntax | 23:44 |
| pleia2 | so perhaps something like: <% if replication['replicatePermissions'].empty -% | 23:45 |
| pleia2 | > | 23:45 |
| pleia2 | err, not empty :) | 23:45 |
| *** rnirmal has quit IRC | 23:45 | |
| pleia2 | don't know how to do notempty | 23:45 |
| fungi | i'll give it a try | 23:45 |
| fungi | i'm trying if not replication['replicationDelay'].empty | 23:46 |
| * pleia2 nods | 23:46 | |
| mgagne | why is this not enough? <% if @replication['replicatePermissions'] -%> | 23:46 |
| clarkb | dstufft: reproduced in a horrid way. Install virtualenv 1.7.2 so that you get ancient distribute bundled (I did this in a venv), create a new virtualenv using version 1.7.2 using the --distribute flag. Deactivate initial venv, source new one made by old virtualenv, pip install mysql-python. It will bomb out and complain about the distribute version | 23:46 |
| clarkb | dstufft: now to test with pip 1.4 to see if it fixed it | 23:47 |
| pleia2 | mgagne: oh, if that would work it would be nice | 23:47 |
| clarkb | nope pip 1.4 still breaks | 23:47 |
| pleia2 | mgagne: including the @ ? (it's not used elsewhere | 23:47 |
| pleia2 | mgagne is totally my puppet hero today | 23:48 |
| fungi | pleia2: looks like it wants .nil instead of .empty anyway | 23:48 |
| pleia2 | fungi: heh, naturally | 23:48 |
| fungi | but i'll try mgagne's suggestion | 23:48 |
| clarkb | FYI requirements approvals that don't pass pep8 will keep failing in the gate. | 23:48 |
| notmyname | anyone know why https://review.openstack.org/#/c/28892/ (ie migrate swift to pbr) has pbr v0.5.16 in requirements but 0.5.20 in setup.py? | 23:49 |
| clarkb | https://review.openstack.org/#/c/39967/ needs to merge before other requirements changes go in | 23:49 |
| clarkb | still waiting on check tests to come back for it though | 23:49 |
| clarkb | notmyname: because it took so long that things skewed. It should probably be 0.5.20 in both | 23:50 |
| notmyname | k | 23:50 |
| clarkb | notmyname: I would -1 that and ask mordred to correct it | 23:50 |
| notmyname | clarkb: ok, thanks | 23:50 |
| mgagne | pleia2: erb template variables now require @ notation (since Puppet 3.2): http://projects.puppetlabs.com/issues/19058 | 23:50 |
| mgagne | pleia2: both methods were supported before. | 23:51 |
| fungi | pleia2: mgagne: conditional matching on the lookup is also causing it to not pass the variables we want explicitly set = false | 23:51 |
| pleia2 | fungi: *facepalm* | 23:51 |
| *** rcleere has quit IRC | 23:51 | |
| mgagne | fungi: true, if it's a bool value... ^^' | 23:52 |
| fungi | pleia2: mgagne: so i suspect i really do need to match not lookup['key'].nil | 23:52 |
| mgagne | pleia2: @ refers to an instance variable in ruby. Without @, it's referring to a method. This way of accessing ERB variables got deprecated. | 23:52 |
| pleia2 | mgagne: thanks, might as well update this template while I'm in it, since we'll go to 3.2 at some point | 23:53 |
| fungi | mgagne: what am i doing wrong with the syntax there? getting undefined method `nil' for nil:NilClass | 23:54 |
| mgagne | fungi: your key does not exist. Can't call nil method on a nil. | 23:54 |
| mgagne | fungi: testing... | 23:55 |
| fungi | how do i match on nil? just != nil | 23:55 |
| fungi | trying | 23:55 |
| mgagne | fungi: better only check if the key exists | 23:56 |
| mgagne | lookup.has_key?('key') | 23:56 |
| fungi | ahh | 23:56 |
| mgagne | fungi: testing | 23:56 |
| fungi | well, in this case != nil did exactly what we want, but if has_key?() is safer i'll give it a shot and see | 23:57 |
| mgagne | fungi: should work if your hash uses string for keys and not ruby symbols | 23:57 |
| mgagne | fungi: http://paste.openstack.org/show/43513/ | 23:58 |
| clarkb | dstufft: I am going to walk home now. Will be AFK for ~20 minutes. Let me know if you need more info abuot the above mysql-python + virtualenv + pip + distribute stuff | 23:59 |
| clarkb | I will get back to it from hom | 23:59 |
| mgagne | pleia2: I can check to update puppet templates later this week if people don't mind to use @ notation | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!