| anteaya | fungi: in that patch does project come in as /neutron or does it come in as /openstack/neutron? | 00:00 |
|---|---|---|
| anteaya | the path needs the second option | 00:00 |
| anteaya | I can't see in the code which one gets passed in | 00:01 |
| fungi | anteaya: you could look at a bug updated by gerrit | 00:01 |
| fungi | i'll find a good example | 00:01 |
| StevenK | https://bugs.launchpad.net/bugs/1257295 | 00:01 |
| uvirtbot | Launchpad bug 1257295 in python-novaclient "openstack is full of misspelled words" [Low,In progress] | 00:01 |
| fungi | the man of the hour | 00:01 |
| StevenK | https://bugs.launchpad.net/nova/+bug/1257295/comments/7 in fact | 00:01 |
| *** jcooley_ has quit IRC | 00:02 | |
| fungi | anteaya: so there you can see that the first %s gets orgname/reponame and the second %s gets the commit sha | 00:02 |
| *** jcooley_ has joined #openstack-infra | 00:02 | |
| fungi | anteaya: so you can try replacing the bits of the format string which i replaced and see if that gets you a usable page | 00:02 |
| fungi | (something i would have tested myself if i weren't so slack) | 00:03 |
| pleia2 | github.org uses the same path ie openstack/neutron so it's an easy one to eyeball | 00:03 |
| fungi | yup | 00:03 |
| *** yamahata_ has quit IRC | 00:04 | |
| fungi | i *did* at least pull up a sample cgit commit page for sanity | 00:04 |
| *** fbo is now known as fbo_away | 00:04 | |
| anteaya | StevenK: thank you | 00:04 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Pin Sphinx to <1.2 https://review.openstack.org/61172 | 00:05 |
| anteaya | fungi: yes I can see the value for the first and second %s in the original, and they make sense now for the change | 00:05 |
| fungi | anteaya: good review! | 00:05 |
| anteaya | thanks | 00:05 |
| * anteaya shakes off the dust | 00:05 | |
| anteaya | pleia2: yes, that is true | 00:06 |
| *** jcooley_ has quit IRC | 00:07 | |
| anteaya | trying to find a rhythm again | 00:07 |
| anteaya | StevenK: coming in to review 61290? | 00:10 |
| *** jcooley_ has joined #openstack-infra | 00:13 | |
| *** jcooley_ has quit IRC | 00:13 | |
| *** jcooley_ has joined #openstack-infra | 00:13 | |
| StevenK | anteaya: Have just done so | 00:16 |
| *** jcooley_ has quit IRC | 00:18 | |
| anteaya | :D | 00:18 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Fix the legend in the e-r page https://review.openstack.org/61295 | 00:18 |
| *** jergerber has quit IRC | 00:18 | |
| *** Bada has quit IRC | 00:19 | |
| openstackgerrit | A change was merged to openstack-infra/config: Install libtidy on slaves https://review.openstack.org/60872 | 00:21 |
| *** herndon has quit IRC | 00:21 | |
| *** pmathews has quit IRC | 00:25 | |
| *** jhesketh_ has joined #openstack-infra | 00:29 | |
| *** salv-orlando has joined #openstack-infra | 00:30 | |
| *** ^d is now known as ^demon|away | 00:31 | |
| *** jhesketh_ is now known as jhesketh | 00:33 | |
| *** jhesketh_ has joined #openstack-infra | 00:33 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Cap Sphinx at <1.2 to avoid distutils problems. https://review.openstack.org/61297 | 00:34 |
| *** sarob has joined #openstack-infra | 00:34 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Document adding bug signatures to e-r. https://review.openstack.org/61298 | 00:34 |
| clarkb | that will close another one of our bugs \o/ | 00:35 |
| sdague | clarkb: is e-r publishing public docs yet? | 00:35 |
| fungi | lgtm | 00:36 |
| clarkb | sdague: unsure | 00:36 |
| sdague | man, this fact that gerrit only returns 500 results at a time is annoying | 00:36 |
| clarkb | sdague: doesn't look like it, it is just a couple jobs in the zuul layout file to make it happen | 00:36 |
| sdague | clarkb: any chance that you could propose that? | 00:36 |
| clarkb | sure | 00:36 |
| sdague | hey, while there are knowledgable folks around. Anyone know the magic invocation to use upstream nova client to spin up rax nodes? | 00:37 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Cap Sphinx at <1.2 to avoid distutils problems. https://review.openstack.org/61297 | 00:37 |
| fungi | sdague: stop using the rax-specific auth and use an account password | 00:38 |
| fungi | sdague: example at https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst#n100 | 00:38 |
| clarkb | did we stop publishing zuul docs? | 00:39 |
| sdague | fungi: so I tried that, and it didn't work | 00:39 |
| sdague | I can try it again | 00:39 |
| clarkb | oh nevermind it is a special job | 00:39 |
| fungi | sdague: OS_USERNAME,OS_PASSWORD,OS_TENANT_NAME,OS_AUTH_URL,OS_REGION_NAME are all i use with latest released novaclient from pypi | 00:39 |
| fungi | sdague: and remove any vars specific to rax special auth methods | 00:40 |
| sdague | hmm... let me test again | 00:40 |
| fungi | since those now require pulling their novaclient plugin off github | 00:40 |
| anteaya | sdague: any objections to me submitting an e-r patch clean up the empty spaces? | 00:41 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Publish elastic-recheck documentation. https://review.openstack.org/61300 | 00:41 |
| clarkb | sdague: ^ | 00:41 |
| sdague | anteaya: which empty spaces? | 00:41 |
| anteaya | been looking for https://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries for a while | 00:41 |
| anteaya | 1252514, 1251920, 1251784, 1251512 ... | 00:42 |
| anteaya | the bugs with no matches | 00:42 |
| *** datsun180b has quit IRC | 00:44 | |
| *** cyeoh_ has joined #openstack-infra | 00:46 | |
| *** cyeoh_ is now known as cyeoh | 00:46 | |
| *** dims has joined #openstack-infra | 00:46 | |
| *** salv-orlando has quit IRC | 00:46 | |
| anteaya | or is there something obvious which I am failing to understand? | 00:46 |
| sdague | fungi: ok, maybe I mis understand what OS_TENANT_NAME should be | 00:48 |
| *** sarob has quit IRC | 00:48 | |
| *** sarob has joined #openstack-infra | 00:49 | |
| fungi | sdague: for our rackspace accounts, it's an integer | 00:49 |
| fungi | seemingly unique, so probably tenant id number | 00:49 |
| *** ^demon|away has quit IRC | 00:50 | |
| fungi | i can poke around the webui to see what it matches | 00:50 |
| sdague | ok, that was the magic bit of info I needed | 00:51 |
| sdague | it's your account # | 00:51 |
| fungi | sdague: yup, if you log into https://mycloud.rackspace.com/ it's next to your account name in the top-right corner | 00:51 |
| fungi | for our hpcloud accounts it's a project name, so i'm guessing it's just a provider-specific id field each cloud implements to tie into whatever their other systems want as a unique key | 00:52 |
| anteaya | clarkb: one suggestion on 61298 | 00:53 |
| *** sarob has quit IRC | 00:53 | |
| *** yamahata_ has joined #openstack-infra | 00:55 | |
| clarkb | anteaya: responded | 00:55 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: update documentation with specific of rax setup https://review.openstack.org/61301 | 00:55 |
| *** yamahata_ has quit IRC | 00:55 | |
| sdague | anteaya: oh, so there are patches in flight for dropping those bugs right now | 00:56 |
| *** yamahata_ has joined #openstack-infra | 00:56 | |
| sdague | fungi: ok, so above is a doc patch to help the next poor slob that doesn't understand this the way I failed to understand it | 00:56 |
| fungi | sdague: thanks! | 00:57 |
| anteaya | sdague: ah perhaps I can review them /me looks at elastic-recheck patches | 00:58 |
| *** jhesketh_ has quit IRC | 00:58 | |
| fungi | sdague: i run those steps soooo often that i now have a set of scripts... one which allows me to select provider/region/account/flavor/image/servername, a second which spins up the vm and applies all the puppetry, and a third which uploads a tempest job shell stub and patch and runs with those against the project/branch i specify | 00:59 |
| fungi | i probably did 50 by-hand tempest runs against various patches over the course of last weekend | 01:00 |
| anteaya | clarkb: back at 'cha | 01:00 |
| sdague | fungi: you got those scripts somewhere? | 01:00 |
| fungi | sdague: i can upload the secnod and thord (and some sub-scripts they call), though the first script is basically credentials | 01:01 |
| sdague | fungi: also, you know what the performance flavors are called | 01:01 |
| sdague | fungi: right :) | 01:01 |
| fungi | sdague: nova flavor-list? | 01:01 |
| anteaya | clarkb: here is my use case: https://wiki.openstack.org/wiki/Network/Meetings#Bugs | 01:01 |
| sdague | ah, right | 01:01 |
| anteaya | I'm going through all the gate blocking bugs for neutron and since I am doing such a great job, now the critical neutron bugs too | 01:02 |
| fungi | sdague: i can slap those into a git repo and cram them up to github or something, but not right this moment. i think ttx may also have some similar since he runs a lot of them for the same reasons i do | 01:02 |
| clarkb | anteaya: I guess I am not seeing how they are different other than the mechanical steps. I will change step 3 to suggest pasting the logstash url instead of just the query | 01:02 |
| anteaya | having a link to logstash in each bug report just helps me sort out which bugs are currently an issue and which aren't | 01:02 |
| sdague | fungi: yeh, seems like that would be a handy set of things to share. | 01:02 |
| sdague | possibly even as a subdir of d-g | 01:02 |
| devananda | clarkb: hi! have a minute or two for a chat about ironic testing? | 01:03 |
| anteaya | clarkb: this bug report has both the fingerprint and the url: https://bugs.launchpad.net/neutron/+bug/1251448 | 01:03 |
| uvirtbot | Launchpad bug 1251448 in neutron "BadRequest: Multiple possible networks found, use a Network ID to be more specific. " [High,New] | 01:03 |
| anteaya | it is just faster when I have to go through about 7 or 8 bug reports, that is all | 01:03 |
| clarkb | devananda: sure | 01:03 |
| clarkb | anteaya: ya, I will update | 01:03 |
| anteaya | clarkb: and thanks | 01:03 |
| *** jhesketh_ has joined #openstack-infra | 01:03 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Document adding bug signatures to e-r. https://review.openstack.org/61298 | 01:04 |
| clarkb | anteaya: ^ | 01:04 |
| fungi | sdague: definitely--i just need to generalize them a little more first, but at the moment i have a bunch of advisories to put together for tomorrow morning | 01:04 |
| sdague | no worries | 01:05 |
| sdague | I just feel accomplished to have figured out this part :) | 01:05 |
| *** mrodden has joined #openstack-infra | 01:05 | |
| devananda | clarkb: remember the conversations we had a while back about testing nova-baremetal inside of devstack-gate | 01:06 |
| fungi | sdague: that's how i felt once i got that writeup cut-n-pasteable enough for what i was doing. that's honestly most of the battle with replicating tests | 01:06 |
| devananda | clarkb: eg, set up a network bridge, create a few VMs, and let baremetal control them | 01:06 |
| clarkb | devananda: ya | 01:06 |
| devananda | clarkb: all that got canned when it was apaprent that tripleo couldn't be tested that way because nested VMs are too slow | 01:06 |
| devananda | clarkb: but that doesn't apply to the testing of ironic | 01:06 |
| clarkb | devananda: right | 01:07 |
| devananda | clarkb: we just need to bring up a tiny cirros image | 01:07 |
| devananda | clarkb: how feasible do you think it would be today to revive that effort? | 01:07 |
| clarkb | devananda: I think it is very feasible (I thought we were already going down that route) | 01:07 |
| devananda | clarkb: and get tempest exercising the whole deployment path with ironic in d-g pipeline | 01:08 |
| devananda | clarkb: well, pleia2 was the last one working on that stuff, and afaik she stopped when it didn't fit tripleo's needs | 01:08 |
| clarkb | devananda: right I don't think anyone is working on it | 01:08 |
| *** sandywalsh_ has quit IRC | 01:09 | |
| pleia2 | there was a recent mailing list thread about it, and I spoke with someone else the other day | 01:09 |
| devananda | some folks from mirantis are now looking at testing ironic. i'm not sure how easily i can explain this to them.... but it seems better than having them run off and do something else entirely | 01:09 |
| * pleia2 digs up notes | 01:09 | |
| *** julim has joined #openstack-infra | 01:09 | |
| clarkb | devananda: ++ | 01:09 |
| devananda | pleia2: possibly vkozhukalov ? I pulled him in here at one point in relation to some of this | 01:09 |
| pleia2 | devananda: I think he's the one I corresponded with over email, someone else in irc | 01:10 |
| pleia2 | 10:31 < pleia2> mestery: vkozhukalov collected links from our current work, I replied here: http://lists.openstack.org/pipermail/openstack-dev/2013-December/021399.html | 01:10 |
| pleia2 | ah, it was mestery :) | 01:10 |
| anteaya | clarkb: +1'd and \o/ | 01:10 |
| clarkb | the way I would start is to have a small pretest devstack-gate hook that does the setup for the "baremetal" nodes | 01:10 |
| clarkb | then tempest will run and have that available to it | 01:10 |
| fungi | mestery: is cisco, not mirantis | 01:11 |
| devananda | pleia2: yep! saw that, thanks | 01:11 |
| pleia2 | yeah, vkozhukalov is from the mirantis crew | 01:11 |
| clarkb | since devstack itself doesn't need to know about this right? this is purely contrived test env setup | 01:11 |
| pleia2 | so we've got some mirantis folks, and at least one guy from cisco interested | 01:11 |
| devananda | pleia2: i don't see the tripleo-ci stuff as in any way overlapping (or likely even being reusable) for this | 01:12 |
| devananda | but maybe i'm missing something | 01:12 |
| anteaya | mestery mostly needs multi-node testing for neutron stuff, to the best of my knowledge | 01:12 |
| clarkb | I think multinode testin will need to go another route | 01:12 |
| anteaya | he is having a meeting for third party testing on Thursday at 1700utc in -alt, mostly aimed at -neutron plugin testing | 01:13 |
| devananda | clarkb: well. something needs to switch on the flag(s) in devstack to use ironic, configure it the right way, and enroll the VMs that were pre-prepared | 01:13 |
| pleia2 | devananda: yeah, perhaps not | 01:13 |
| anteaya | the multi-node testing will probably come up | 01:13 |
| clarkb | devananda: that is what devstack-gate does | 01:13 |
| clarkb | devananda: I guess I am trying to differentiate between make sure baremetal nodes are avaialble and setup a cloud | 01:13 |
| devananda | clarkb: right. devstack itself does need some changes for testing ironic (some or most of those may already be in place) | 01:14 |
| devananda | clarkb: and d-g clearly needs most of the work | 01:14 |
| clarkb | ya | 01:14 |
| devananda | clarkb: i think that the existing tempest tests which exercise clouds should be fine. | 01:14 |
| devananda | AND | 01:14 |
| devananda | this is all separate from the tempest tests which we've already proposed | 01:14 |
| devananda | those explicitly perform CRUD operations on ironic's API | 01:15 |
| devananda | which won't be tested by devstack in teh above discussion -- they'll be _used_ by d-g in the preparation of the run | 01:15 |
| clarkb | devananda: so one potential problem there is the existing tempest tests can start a ton of VMs, we will need to make sure the "baremetal" nodes have enough capacity | 01:15 |
| *** eharney has quit IRC | 01:15 | |
| devananda | hum | 01:15 |
| devananda | clarkb: are they starting "fake" vms? | 01:15 |
| clarkb | no real qemu VMs today | 01:15 |
| devananda | ah, k | 01:15 |
| devananda | in principle, there shouldn't be any more overhead for ironic to manage a bunch of qemu VMs | 01:16 |
| clarkb | devananda: we just need to size the fake baremetal env appropriately | 01:16 |
| clarkb | this won't be 2 or 3 nodes, probably more like 100 | 01:16 |
| devananda | well. correction. the nova scheduler may be a bit mroe of a bottleneck -- it'll see each qemu VM as a separate resource | 01:16 |
| clarkb | sdague and mtreinish would have a better handle on tempest resource consumption | 01:17 |
| devananda | cool | 01:17 |
| *** oubiwan__ has joined #openstack-infra | 01:17 | |
| sdague | devananda: what image are you going to boot on the fake baremetal? | 01:17 |
| devananda | my point above is that teh CRUD of ironic resources still needs to be handled separately by tempest | 01:17 |
| sdague | today we do really tiny guests because we boot the cirros image | 01:17 |
| devananda | since that won't be covered by the traditional tempest-runs-against-a-cloud stuff | 01:17 |
| sdague | that's the only place I think we might get in trouble | 01:18 |
| devananda | sdague: cirros is fine for testing whether ironic booted something | 01:18 |
| sdague | devananda: that will pxe? | 01:18 |
| devananda | why wouldn't it? | 01:18 |
| devananda | it has a kernel & ramdisk | 01:18 |
| lifeless | sdague: cirros is above the PXE layer | 01:18 |
| sdague | ok, fair | 01:18 |
| lifeless | sdague: cirros is the workload, not the VM :) | 01:18 |
| sdague | right :) | 01:18 |
| sdague | ok, I'm on the same page now | 01:18 |
| devananda | :) | 01:18 |
| sdague | I guess in your fake baremetal layer, how are you specifying qemu memory? | 01:19 |
| devananda | sdague: ironic is not creating VMs | 01:19 |
| clarkb | devananda: for the other stuff you would just have tempest tests tagged with ironic or whatever that only run when ironic is available. similar to how we do ?neutron? again sdague knows best | 01:19 |
| devananda | sdague: in this environment, d-g scripts would create all the tiny VMs ahead of time and just enroll them with ironic -- feed the MAC addresses of the NIC of each qemu VM to our API | 01:20 |
| clarkb | sdague: right so something in d-g (I think a hook) would setup a ton of qemu VMs 64MB large each then register them into ironic via devstack or whatever | 01:20 |
| sdague | devananda: ok, gotcha | 01:20 |
| devananda | clarkb: it'd have to be after devstack is started (or at least after the ironic services are up) | 01:20 |
| clarkb | devananda: yup | 01:21 |
| *** sandywalsh_ has joined #openstack-infra | 01:21 | |
| sdague | right, so as long as that mem size is tunable sanely in d-g life is probably good | 01:21 |
| *** dims has quit IRC | 01:21 | |
| devananda | botgh the mem size and # of qemu VMs | 01:21 |
| clarkb | I am thinking devstack needs to know about enrollment beacse you may want to use devstack to do ironic | 01:21 |
| clarkb | but devstack shouldn't start a ton of small VMs to be baremetal proxies, because that is implementation specific | 01:21 |
| sdague | now that's an interesting idea | 01:21 |
| lifeless | clarkb: btw we have that code in tripleo incubator already | 01:21 |
| lifeless | clarkb: we should reuse it | 01:21 |
| sdague | honestly, that might be a good devstack patch | 01:22 |
| clarkb | lifeless: perfect | 01:22 |
| lifeless | we might want to rehome it for this | 01:22 |
| lifeless | but it shouldn't live in devstack | 01:22 |
| devananda | clarkb: so, maybe have d-g create the VMs and dump their MACs into a csv file | 01:22 |
| devananda | clarkb: then have devstack suck up that file as part of lib/ironic or such? | 01:23 |
| sdague | lifeless: so why wouldn't you have this live in devstack? | 01:23 |
| devananda | sdague: because tripleo does the same thing | 01:23 |
| lifeless | sdague: because that would make tripleo-gate depend on devstack | 01:23 |
| *** sarob has joined #openstack-infra | 01:23 | |
| sdague | if I was a developer, and I wanted to work on ironic, this would be really useful | 01:23 |
| devananda | lifeless: this sounds a lot like what bmposeur initially aimed to do | 01:23 |
| clarkb | devananda: that is what I am thinking | 01:23 |
| lifeless | sdague: right, so we should make it more accessible yes | 01:23 |
| lifeless | sdague: but the devstack codebase is a thing that takes over your machine, and thats very much not what the tripleo devtest cycle does - we explicitly avoid that | 01:24 |
| lifeless | sdague: because the workaround - running in a VM - is horribly slow for tripleo | 01:24 |
| *** salv-orlando has joined #openstack-infra | 01:25 | |
| sdague | hmmm... ok, I guess I'll see the patches when they come | 01:25 |
| lifeless | sdague: devstack can reuse things, right? :) | 01:26 |
| clarkb | devananda: so I think we start with those two bits. Something to spin up a ton of small qemu VMs and devstack learning to enroll them as baremetal nodes | 01:26 |
| lifeless | devananda: yes, but it overreached and became a reimplementation of virsh | 01:26 |
| clarkb | once we have that we can add a job that runs tempest against a "baremetal" cloud | 01:26 |
| devananda | lifeless: agreed | 01:27 |
| clarkb | on top of that we add the additional tests | 01:27 |
| *** talluri has quit IRC | 01:27 | |
| lifeless | sdague: devstack combines deployment tooling with developer setup - and thats fine for its mission, the problem is if we want to reuse just the developer setup, it's really hard because its conflated with the deployments tuff | 01:27 |
| devananda | clarkb: sounds good to me | 01:27 |
| clarkb | because the underlying cloud should have all of the dependencies you need to run those tests at that point | 01:27 |
| lifeless | sdague: which is why we have similar but different stuff in the tripleo space :( | 01:27 |
| *** sarob has quit IRC | 01:28 | |
| *** salv-orlando has quit IRC | 01:28 | |
| lifeless | one approach I think might work well would be to suck the tripleo-incubator into devstack for some of these functions [we'd want symettric gating for it for that, of course] | 01:28 |
| sdague | lifeless: well it would be interesting to understand how that could be sliced better, because having a reimplementation in tripleo space seems suboptimal, even if different on a few things | 01:28 |
| *** nosnos has joined #openstack-infra | 01:29 | |
| *** salv-orlando has joined #openstack-infra | 01:29 | |
| lifeless | sdague: I think you should have a solid play with tripleo, it would help you see where there is overlap-that-should-be-converged and overlap-that-shouldn't-be | 01:29 |
| sdague | possibly, but with the stuff on my plate, it will be a while :) | 01:30 |
| lifeless | sdague: sure :) | 01:30 |
| lifeless | sdague: it's just kinda hard to give a useful handwave description | 01:30 |
| sdague | plus I'm more hardware constrained than previously at the moment. Will fix that, but it means playing with this is somewhat bounded. | 01:30 |
| devananda | clarkb: I need to step away and start prepping for dinner soon. this has been very helpful; i'm sending a few notes to the ML. thanks! | 01:31 |
| lifeless | sdague: point is, devstack's mission is developer productivity with some deployment capability as option, tripleos is production deployment with developer-productivity as option | 01:31 |
| clarkb | devananda: np I need to head home in a minute too | 01:31 |
| clarkb | I will try to keep up with the ML thread | 01:31 |
| lifeless | devananda: what do you want tonnes of VMs to assess? | 01:32 |
| sdague | ok, fair. Seems like we're going to have some duping, but some times that's what it is. | 01:33 |
| devananda | lifeless: i don't. it's apparently a requirement of tempest now | 01:33 |
| *** yaguang has joined #openstack-infra | 01:33 | |
| devananda | lifeless: for the general let-tempest-exercise-openstack tests | 01:34 |
| lifeless | oh, due to the concurrency? | 01:34 |
| clarkb | lifeless: concurrency and async cleanup iirc | 01:34 |
| lifeless | kk | 01:35 |
| lifeless | so should be identical to qemu | 01:35 |
| *** dims has joined #openstack-infra | 01:36 | |
| clarkb | new propose reqs failure :( https://jenkins.openstack.org/job/propose-requirements-updates/40/console | 01:37 |
| clarkb | I can't believe there is a conflict, unless an out of band requirements update went in | 01:37 |
| jeblair | clarkb: 2013-12-11 01:23:35.782 | Rebasing (1/108) | 01:39 |
| clarkb | jeblair: right it did a rebase for some reason, I think possibly out of band requirement | 01:40 |
| jeblair | clarkb: that's kind of a red flag. is it possible there's some git repo cleanup in the workspace not happening? | 01:40 |
| *** julim has quit IRC | 01:40 | |
| clarkb | jeblair: well it is pulling the existing change from gerrit and basing on that | 01:40 |
| *** talluri has joined #openstack-infra | 01:40 | |
| clarkb | jeblair: I think we may need to pull that change from gerrit for diffing only | 01:40 |
| clarkb | go back to basing new patchset on $branch then at the end do a git diff HEAD and a git diff $oldpatchset from gerrit. If they both differ then push | 01:41 |
| clarkb | I need to run, but that may fix the probem | 01:42 |
| jeblair | clarkb: i'm not sure it checkout out the havana branch of ceilometer | 01:43 |
| jeblair | wow | 01:43 |
| jeblair | clarkb: i'm not sure it checked out the havana branch of ceilometer | 01:43 |
| *** michchap_ has quit IRC | 01:46 | |
| *** sarob has joined #openstack-infra | 01:47 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix branch logic in req propose script https://review.openstack.org/61304 | 01:47 |
| jeblair | clarkb: ^ | 01:47 |
| *** michchap has joined #openstack-infra | 01:47 | |
| anteaya | clarkb: on 61300 I am just curious about '{name}-docs' I don't see it anywhere else in projects.yaml, why here? | 01:47 |
| anteaya | oh he left | 01:48 |
| *** reed has quit IRC | 01:58 | |
| *** herndon has joined #openstack-infra | 02:00 | |
| *** rongze has joined #openstack-infra | 02:01 | |
| *** guohliu has joined #openstack-infra | 02:02 | |
| *** AaronGr is now known as AaronGr_Zzz | 02:03 | |
| *** bingbu has joined #openstack-infra | 02:03 | |
| *** jcooley_ has joined #openstack-infra | 02:04 | |
| *** mriedem has joined #openstack-infra | 02:04 | |
| *** mriedem has quit IRC | 02:05 | |
| *** mriedem has joined #openstack-infra | 02:06 | |
| *** xchu has joined #openstack-infra | 02:10 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Ability to Soft Delete Stale Bugs from elastic-recheck https://review.openstack.org/61149 | 02:10 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Fix the legend in the e-r page https://review.openstack.org/61295 | 02:10 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Delete stale bugs from elastic-recheck page https://review.openstack.org/61186 | 02:14 |
| *** ryanpetrello has quit IRC | 02:17 | |
| *** herndon has quit IRC | 02:20 | |
| *** ryanpetrello has joined #openstack-infra | 02:28 | |
| *** sarob has quit IRC | 02:29 | |
| *** sarob has joined #openstack-infra | 02:30 | |
| *** rongze has quit IRC | 02:31 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: use grenade gate numbers, not check queue numbers https://review.openstack.org/61307 | 02:32 |
| *** slong has joined #openstack-infra | 02:33 | |
| *** sarob_ has joined #openstack-infra | 02:34 | |
| *** sarob has quit IRC | 02:35 | |
| *** krotscheck has quit IRC | 02:37 | |
| *** jcooley_ has quit IRC | 02:43 | |
| *** rongze has joined #openstack-infra | 02:46 | |
| sarob_ | i requested a new stackforge project | 02:49 |
| sarob_ | stackforge/milk has no master, i can't push the first patch | 02:51 |
| *** sarob_ is now known as sarob | 02:51 | |
| sarob | Errors running git rebase -i remotes/gerrit/master | 02:52 |
| sarob | fatal: Needed a single revision | 02:52 |
| sarob | invalid upstream remotes/gerrit/master | 02:52 |
| *** Ryan_Lane has quit IRC | 02:52 | |
| *** Ryan_Lane has joined #openstack-infra | 02:53 | |
| *** nati_ueno has quit IRC | 02:54 | |
| *** Ryan_Lane has quit IRC | 02:58 | |
| *** marun has joined #openstack-infra | 03:00 | |
| *** sarob has quit IRC | 03:06 | |
| *** sarob has joined #openstack-infra | 03:06 | |
| *** ryanpetrello has quit IRC | 03:11 | |
| *** pcrews has quit IRC | 03:11 | |
| *** sarob has quit IRC | 03:11 | |
| *** ladquin has quit IRC | 03:14 | |
| *** AaronGr_Zzz is now known as AaronGr | 03:19 | |
| *** paul-- has joined #openstack-infra | 03:21 | |
| *** zhiyan has joined #openstack-infra | 03:22 | |
| zaro | anteaya: '{name}-docs' expands to 'elastic-recheck-docs' which is found in layout.yaml | 03:23 |
| *** Ryan_Lane has joined #openstack-infra | 03:25 | |
| * StevenK peers at https://wiki.openstack.org/wiki/Governance/Approved | 03:28 | |
| *** Ryan_Lane has quit IRC | 03:30 | |
| fungi | if sarob returns, i've cleared and successfully reimported stackforge/milk, but unfortunately that means our race is probably still lurking (don't have time to go spelunking in logs just at the moment) | 03:30 |
| clarkb | fungi: if you are still on review.o.o check that jeepyb is up to date there | 03:32 |
| fungi | good idea | 03:32 |
| clarkb | I think it has had trouble updating which may explain it (could be the race is just still there and ugh) | 03:32 |
| fungi | jeepyb==1caeeae | 03:34 |
| fungi | according to pip freeze | 03:34 |
| clarkb | that is up to date :/ | 03:35 |
| clarkb | I am pretty sure the problem is related to creating groups. Anytime a new project creates a new group everything after that fails | 03:35 |
| clarkb | subsequent runs work | 03:35 |
| clarkb | maybe the logic in group creation is actualyl the problem? should probably beef up logging and hit review-dev when it is back | 03:36 |
| *** hcc has joined #openstack-infra | 03:50 | |
| *** gyee has quit IRC | 03:55 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:17 | |
| *** SergeyLukjanov has quit IRC | 04:21 | |
| *** dstanek_afk has joined #openstack-infra | 04:22 | |
| *** ArxCruz has quit IRC | 04:22 | |
| *** ArxCruz has joined #openstack-infra | 04:24 | |
| *** slong_ has joined #openstack-infra | 04:24 | |
| *** slong has quit IRC | 04:24 | |
| *** yamahata_ has quit IRC | 04:24 | |
| *** dstanek has quit IRC | 04:24 | |
| *** plomakin has quit IRC | 04:24 | |
| *** Mithrandir has quit IRC | 04:24 | |
| *** Mithrandir has joined #openstack-infra | 04:25 | |
| *** plomakin has joined #openstack-infra | 04:25 | |
| *** hcc is now known as hdd_ | 04:25 | |
| clarkb | fungi: I think the problem may be elsewhere looking at the new mange-projects, if project_created: push_to_gerrit() then later if acl_config: process_acls() if has-github: create_github_project() | 04:27 |
| *** SergeyLukjanov has joined #openstack-infra | 04:27 | |
| *** yamahata_ has joined #openstack-infra | 04:28 | |
| clarkb | the github project was created so those other things must be independently failing now | 04:28 |
| *** ArxCruz has quit IRC | 04:29 | |
| *** dstanek_afk has quit IRC | 04:31 | |
| *** mriedem has quit IRC | 04:31 | |
| *** ArxCruz has joined #openstack-infra | 04:32 | |
| *** changbl has quit IRC | 04:33 | |
| clarkb | looking at the git log for the project it looks like project creation simply failed | 04:34 |
| *** changbl has joined #openstack-infra | 04:36 | |
| *** dstanek_afk has joined #openstack-infra | 04:36 | |
| fungi | did you confirm the github mirror wasn't created on my manual re-run? | 04:39 |
| clarkb | fungi: I checked it about 15 minutes before you reran it | 04:40 |
| fungi | ah... okay | 04:40 |
| clarkb | fungi: assuming your statement was near when you reran it | 04:40 |
| clarkb | fungi: I wonder if the clone of the seed repo is failing | 04:41 |
| fungi | yup | 04:41 |
| *** blamar has quit IRC | 04:47 | |
| *** talluri has quit IRC | 04:48 | |
| *** rongze has quit IRC | 04:50 | |
| *** dstanek_afk is now known as dstanek | 04:52 | |
| *** AaronGr is now known as AaronGr_Zzz | 04:53 | |
| *** oubiwan__ has quit IRC | 04:56 | |
| *** ryanpetrello has joined #openstack-infra | 04:58 | |
| *** talluri has joined #openstack-infra | 05:04 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 05:09 | |
| *** _SergeyLukjanov has quit IRC | 05:10 | |
| *** SergeyLukjanov has joined #openstack-infra | 05:14 | |
| SergeyLukjanov | fungi, clarkb, hey guys :) | 05:15 |
| fungi | SergeyLukjanov: hey there | 05:15 |
| SergeyLukjanov | that's funny to have a 12h time difference :) | 05:16 |
| * fungi should get to sleep soon | 05:17 | |
| *** oubiwan__ has joined #openstack-infra | 05:17 | |
| StevenK | Sleep is for the week | 05:17 |
| *** oubiwan__ is now known as oubiwann-lambda | 05:17 | |
| *** boris-42 has joined #openstack-infra | 05:18 | |
| *** rongze has joined #openstack-infra | 05:20 | |
| *** oubiwann-lambda has quit IRC | 05:23 | |
| *** ryanpetrello has quit IRC | 05:25 | |
| *** ryanpetrello_ has joined #openstack-infra | 05:25 | |
| *** nosnos_ has joined #openstack-infra | 05:31 | |
| *** jcooley_ has joined #openstack-infra | 05:33 | |
| *** nosnos has quit IRC | 05:34 | |
| *** rongze has quit IRC | 05:35 | |
| *** nicedice has quit IRC | 05:36 | |
| *** jhesketh has quit IRC | 05:38 | |
| *** xchu has quit IRC | 05:45 | |
| openstackgerrit | Steve Martinelli proposed a change to openstack/requirements: Add oauthlib library to replace oauth2 https://review.openstack.org/61198 | 05:53 |
| openstackgerrit | ChangBo Guo proposed a change to openstack-dev/pbr: Fix typos in documents https://review.openstack.org/61110 | 05:53 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 05:54 |
| *** juice has quit IRC | 05:57 | |
| *** devananda has quit IRC | 05:57 | |
| *** davidlenwell_ has joined #openstack-infra | 05:57 | |
| *** devananda has joined #openstack-infra | 05:57 | |
| *** Ghe_HPDiscover has quit IRC | 05:57 | |
| *** vipuls has quit IRC | 05:58 | |
| *** davidlenwell has quit IRC | 05:58 | |
| *** echohead has quit IRC | 05:58 | |
| *** LinuxJedi has quit IRC | 05:58 | |
| *** NobodyCam has quit IRC | 05:58 | |
| *** xchu has joined #openstack-infra | 05:59 | |
| *** echohead has joined #openstack-infra | 05:59 | |
| *** LinuxJedi has joined #openstack-infra | 06:00 | |
| *** rongze has joined #openstack-infra | 06:03 | |
| *** juice has joined #openstack-infra | 06:03 | |
| *** nosnos_ has quit IRC | 06:04 | |
| *** nosnos has joined #openstack-infra | 06:05 | |
| *** xchu has quit IRC | 06:06 | |
| *** vipul has joined #openstack-infra | 06:07 | |
| *** rongze has quit IRC | 06:07 | |
| *** vkozhukalov has quit IRC | 06:09 | |
| *** paul-- has quit IRC | 06:13 | |
| *** paul-- has joined #openstack-infra | 06:13 | |
| *** NobodyCam has joined #openstack-infra | 06:14 | |
| *** rongze has joined #openstack-infra | 06:15 | |
| *** paul-- has quit IRC | 06:17 | |
| *** odyssey4me has joined #openstack-infra | 06:19 | |
| *** hdd_ has quit IRC | 06:20 | |
| *** SergeyLukjanov has quit IRC | 06:21 | |
| *** zhiyan has quit IRC | 06:22 | |
| *** xchu has joined #openstack-infra | 06:25 | |
| *** echohead has quit IRC | 06:31 | |
| *** Ghe_HPDiscover has joined #openstack-infra | 06:35 | |
| *** jcooley_ has quit IRC | 06:39 | |
| *** senk has joined #openstack-infra | 06:40 | |
| *** denis_makogon has joined #openstack-infra | 06:42 | |
| *** thomasbiege has joined #openstack-infra | 06:45 | |
| *** SushilKM has joined #openstack-infra | 06:46 | |
| *** zhiyan has joined #openstack-infra | 06:51 | |
| *** ryanpetrello_ has quit IRC | 06:52 | |
| *** ryanpetrello has joined #openstack-infra | 06:56 | |
| *** dstanek has quit IRC | 07:02 | |
| *** lifeless has quit IRC | 07:05 | |
| *** jhesketh_ has quit IRC | 07:09 | |
| *** jcooley_ has joined #openstack-infra | 07:09 | |
| *** sdake_ has quit IRC | 07:13 | |
| *** jcooley_ has quit IRC | 07:15 | |
| *** guohliu has quit IRC | 07:18 | |
| *** boris-42 has quit IRC | 07:22 | |
| *** guohliu has joined #openstack-infra | 07:25 | |
| *** nati_ueno has joined #openstack-infra | 07:36 | |
| *** thomasbiege has quit IRC | 07:37 | |
| *** yolanda has joined #openstack-infra | 07:37 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:49 | |
| *** sarob has joined #openstack-infra | 07:51 | |
| *** xchu has quit IRC | 07:52 | |
| *** DinaBelova has joined #openstack-infra | 07:54 | |
| *** thomasbiege has joined #openstack-infra | 07:55 | |
| *** Abhishek_ has joined #openstack-infra | 07:57 | |
| *** vkozhukalov has joined #openstack-infra | 07:57 | |
| *** senk1 has joined #openstack-infra | 07:58 | |
| *** senk has quit IRC | 08:00 | |
| *** jcoufal has joined #openstack-infra | 08:04 | |
| *** jcoufal has quit IRC | 08:04 | |
| *** jcooley_ has joined #openstack-infra | 08:05 | |
| *** bingbu has quit IRC | 08:05 | |
| *** jcoufal has joined #openstack-infra | 08:05 | |
| *** xchu has joined #openstack-infra | 08:05 | |
| *** jcooley_ has quit IRC | 08:09 | |
| *** flaper87|afk is now known as flaper87 | 08:13 | |
| *** ruhe has joined #openstack-infra | 08:16 | |
| *** fbo_away is now known as fbo | 08:18 | |
| *** ruhe has quit IRC | 08:21 | |
| *** ruhe has joined #openstack-infra | 08:22 | |
| *** bingbu has joined #openstack-infra | 08:25 | |
| *** ruhe has quit IRC | 08:25 | |
| *** guohliu has quit IRC | 08:35 | |
| *** guohliu has joined #openstack-infra | 08:35 | |
| *** nati_ueno has quit IRC | 08:38 | |
| *** yongli is now known as yongli_away | 08:41 | |
| ttx | StevenK: about that wiki URL, it's pretty stale | 08:45 |
| *** yassine has joined #openstack-infra | 08:46 | |
| ttx | StevenK: we are phasing it out and using https://github.com/openstack/governance instead | 08:46 |
| *** rongze has quit IRC | 08:46 | |
| *** rongze has joined #openstack-infra | 08:47 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/61286 | 08:52 |
| *** apevec has joined #openstack-infra | 08:52 | |
| *** nprivalova has joined #openstack-infra | 08:52 | |
| apevec | fungi, ping - is there ci lp bug# for sphinx issue? | 08:54 |
| apevec | https://review.openstack.org/61236 was merged but I don't see automatic proposals James mentioned? | 08:54 |
| SergeyLukjanov | apevec, https://review.openstack.org/#/q/owner:jenkins,n,z | 08:55 |
| openstackgerrit | ChangBo Guo proposed a change to openstack-dev/hacking: Fix typos of comment in module core https://review.openstack.org/61111 | 08:55 |
| SergeyLukjanov | ^^ here are the auto proposed requirements updates | 08:55 |
| SergeyLukjanov | and here is the bug for sphinx | 08:55 |
| SergeyLukjanov | https://bugs.launchpad.net/openstack-ci/+bug/1259511 | 08:55 |
| uvirtbot | Launchpad bug 1259511 in openstack-ci "sphinx docs gating jobs are broken (source_dir must be a directory)" [Critical,Fix released] | 08:55 |
| apevec | SergeyLukjanov, thanks! | 08:55 |
| *** roeyc has joined #openstack-infra | 08:56 | |
| DinaBelova | SergeyLukjanov, thank you so much | 08:56 |
| apevec | SergeyLukjanov, so that's all for master | 08:56 |
| apevec | what about stable/havana | 08:56 |
| SergeyLukjanov | apevec, it'll be fixed after one fix merged | 08:57 |
| SergeyLukjanov | https://review.openstack.org/#/c/61304/ | 08:57 |
| openstackgerrit | ChangBo Guo proposed a change to openstack-dev/pbr: Fix typos in documents https://review.openstack.org/61110 | 08:58 |
| apevec | SergeyLukjanov ah ha! thanks for the pointers | 08:58 |
| SergeyLukjanov | np | 08:58 |
| *** jcooley_ has joined #openstack-infra | 09:00 | |
| *** afazekas has joined #openstack-infra | 09:02 | |
| *** jcooley_ has quit IRC | 09:04 | |
| *** senk1 has quit IRC | 09:05 | |
| *** yolanda has quit IRC | 09:06 | |
| *** boris-42 has joined #openstack-infra | 09:06 | |
| *** apevec has quit IRC | 09:06 | |
| *** yolanda has joined #openstack-infra | 09:07 | |
| *** Abhishek_ has quit IRC | 09:08 | |
| *** hashar has joined #openstack-infra | 09:11 | |
| *** jpich has joined #openstack-infra | 09:14 | |
| *** bingbu has quit IRC | 09:14 | |
| mattymo | SergeyLukjanov, ping | 09:16 |
| *** Abhishek_ has joined #openstack-infra | 09:17 | |
| *** locke105 has quit IRC | 09:17 | |
| *** avishayb has joined #openstack-infra | 09:18 | |
| *** hashar has quit IRC | 09:18 | |
| *** hashar has joined #openstack-infra | 09:22 | |
| StevenK | ttx: And it's only on github because? | 09:23 |
| *** derekh has joined #openstack-infra | 09:23 | |
| SergeyLukjanov | mattymo, pong | 09:24 |
| ttx | StevenK: it's not only on github, just mirrored there | 09:26 |
| ttx | http://git.openstack.org/cgit/openstack/governance/ | 09:26 |
| StevenK | Not very user-friendly. But okay | 09:28 |
| ttx | StevenK: the goal is to get it publiched somewhere from there | 09:28 |
| StevenK | However, the core developer policy is under that wiki hierachy, and it isn't in that git repo | 09:28 |
| ttx | published* | 09:28 |
| ttx | StevenK: link? | 09:29 |
| ttx | there are lots of very old wiki pages that may or may not still be relevant | 09:29 |
| StevenK | ttx: https://wiki.openstack.org/wiki/Governance/Approved/CoreDevProcess | 09:31 |
| ttx | StevenK: so that one predates the notion of PTL | 09:32 |
| ttx | since then we established that PTLs have ultimate authority on a project matters | 09:33 |
| ttx | so they are free to pick whatever method they like to establish their core team | 09:33 |
| ttx | that said, most still follow the ideas described in this document | 09:33 |
| *** guohliu has quit IRC | 09:33 | |
| ttx | So that's superseded by https://wiki.openstack.org/wiki/Governance/Foundation/TechnicalCommittee - section on Program Leads | 09:34 |
| ttx | that page being a copy of http://git.openstack.org/cgit/openstack/governance/tree/reference/charter | 09:35 |
| ttx | Governance/Approved/CoreDevProcess should no longer be linked from anywhere | 09:36 |
| StevenK | ttx: I did a quick google this morning about core developer -- everything you're mentioning doesn't look to be well discoverable | 09:36 |
| *** nosnos_ has joined #openstack-infra | 09:36 | |
| *** tsufiev has joined #openstack-infra | 09:37 | |
| *** sergmelikyan has joined #openstack-infra | 09:37 | |
| tsufiev | hi there! do you know about problem with new branch creation in review.openstack.org? | 09:37 |
| ttx | StevenK: https://wiki.openstack.org/ >> Governance >> Technical Committee | 09:38 |
| SergeyLukjanov | tsufiev, which problem? | 09:38 |
| ttx | StevenK: but yeah, Google will find stale pages on the wiki for sure | 09:38 |
| tsufiev | we've just tried to create a new branch for our murano repositories, but could not do that | 09:38 |
| SergeyLukjanov | tsufiev, have you checked ACL? | 09:38 |
| ttx | StevenK: and we should remove all non longer applicable pages, or at least move them to some attic/ | 09:39 |
| sergmelikyan | We created new branches around 2 month ago and all was OK, there was some changes? | 09:39 |
| tsufiev | SergeyLukjanov: not yet. Where can it be done? | 09:39 |
| SergeyLukjanov | sergmelikyan, I don't think so | 09:40 |
| SergeyLukjanov | tsufiev, https://review.openstack.org/#/admin/projects/stackforge/murano-repository,access | 09:40 |
| *** jooools has joined #openstack-infra | 09:40 | |
| *** nosnos has quit IRC | 09:40 | |
| SergeyLukjanov | tsufiev, do you have any error message? | 09:40 |
| ttx | StevenK: there were lots of changes in the governance over the last 3 years, and the wiki still contains artifacts from that. If you're interested in history, I wrote up a summary of the evolution: http://fnords.wordpress.com/2013/06/20/history-of-openstack-governance/ | 09:41 |
| sergmelikyan | http://i41.tinypic.com/sw5nio.jpg | 09:41 |
| sergmelikyan | Error message :) | 09:42 |
| *** denis_makogon has quit IRC | 09:42 | |
| sergmelikyan | Simply Application Error: Cannot create refs/heads/release-0.4 | 09:42 |
| tsufiev | SergeyLukjanov: actually, there is no special section for branch creation in Access section | 09:42 |
| *** lifeless has joined #openstack-infra | 09:43 | |
| tsufiev | wait a minute... can this change have caused it https://review.openstack.org/gitweb?p=stackforge/murano-repository.git;a=commitdiff;h=c95e646aca6c6f12fcc9eed12c681e0efd219229 ? | 09:44 |
| sergmelikyan | https://github.com/openstack-infra/config/commit/9bbd07ee06fe656ad7687dd4bb725a5d69843600 | 09:46 |
| sergmelikyan | Issue with ACL :( Thx! | 09:46 |
| *** johnthetubaguy has joined #openstack-infra | 09:56 | |
| SergeyLukjanov | sergmelikyan, np, exactly, you remove the ability to create refs | 09:57 |
| *** jcoufal has quit IRC | 09:58 | |
| *** shardy_afk is now known as shardy | 10:01 | |
| *** katyafervent has joined #openstack-infra | 10:03 | |
| *** xchu has quit IRC | 10:04 | |
| *** senk has joined #openstack-infra | 10:05 | |
| *** senk has quit IRC | 10:10 | |
| *** heyongli has joined #openstack-infra | 10:10 | |
| *** rossella_s has joined #openstack-infra | 10:14 | |
| *** markmc has joined #openstack-infra | 10:14 | |
| *** bogdando has joined #openstack-infra | 10:15 | |
| *** Bada has joined #openstack-infra | 10:15 | |
| *** apevec has joined #openstack-infra | 10:17 | |
| *** apevec has joined #openstack-infra | 10:17 | |
| *** dizquierdo has joined #openstack-infra | 10:17 | |
| *** DinaBelova has quit IRC | 10:20 | |
| *** nprivalova has quit IRC | 10:20 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 10:23 | |
| bogdando | hello. What is typical work-flow for creating WIP branches in gerrit for Openstack projects? | 10:24 |
| bogdando | should I send PRs to https://github.com/openstack-infra/config? What exactly should be changed? Any help with docs? Thank you. | 10:25 |
| *** sarob has quit IRC | 10:26 | |
| sdague | ttx: we should build a docs job for the governance repo | 10:31 |
| sdague | so it publishes out to somewhere on openstack.org in pretty html | 10:31 |
| ttx | sdague: yes, or get them mirrored on the wiki: https://bugs.launchpad.net/openstack-ci/+bug/1233059 | 10:31 |
| uvirtbot | Launchpad bug 1233059 in openstack-ci "Set up governance wiki pages as read-only copies of reference repo" [High,Triaged] | 10:31 |
| ttx | if not on wiki, we'll have to set up redirects -- the wiki URLs are cited in governance docs like bylaws. | 10:32 |
| sdague | ttx: honestly, I think the docs jobs are a better bet. Republishing to the wiki seems weird. | 10:33 |
| lifeless | why publish somewhere folk can edit and then say they can't edit | 10:34 |
| ttx | sdague: agreed, we just need to preserve links | 10:35 |
| ttx | so replacing charter with link to actual charter | 10:35 |
| ttx | lifeless: fwiw those pages are protected. they are on the wiki because that was the easiest option, not the best. | 10:36 |
| ttx | lifeless: I'm more scared to see the Foundation *bylaws* on the wiki, tbh | 10:36 |
| *** senk has joined #openstack-infra | 10:43 | |
| *** che-arne has joined #openstack-infra | 10:43 | |
| *** senk has quit IRC | 10:47 | |
| openstackgerrit | Timur Sufiev proposed a change to openstack-infra/config: Enable new branch creation for murano. https://review.openstack.org/61369 | 10:47 |
| *** senk has joined #openstack-infra | 10:48 | |
| *** jd__ has quit IRC | 10:48 | |
| *** jd__ has joined #openstack-infra | 10:48 | |
| tsufiev | mordred, fungi, clarkb: could you please review this change https://review.openstack.org/61369 ? | 10:50 |
| *** bogdando has quit IRC | 10:51 | |
| *** mgagne has quit IRC | 10:52 | |
| *** Bada has quit IRC | 10:52 | |
| *** nosnos_ has quit IRC | 10:57 | |
| *** ativelkov has joined #openstack-infra | 10:57 | |
| *** nprivalova has joined #openstack-infra | 10:57 | |
| *** nosnos has joined #openstack-infra | 10:57 | |
| *** yaguang has quit IRC | 10:59 | |
| *** yhe_ has joined #openstack-infra | 11:04 | |
| *** zhiyan has quit IRC | 11:05 | |
| *** mgagne has joined #openstack-infra | 11:06 | |
| *** mgagne has quit IRC | 11:06 | |
| *** mgagne has joined #openstack-infra | 11:06 | |
| *** zhiyan has joined #openstack-infra | 11:06 | |
| *** heyongli has quit IRC | 11:06 | |
| *** zhiyan has quit IRC | 11:08 | |
| sdague | hmmm... I'm still getting that sphinx issue from yesterday, even with the pin locally | 11:08 |
| *** nprivalova has quit IRC | 11:09 | |
| sdague | oh, that's why, sphinx 1.2b3 is out on pypi | 11:11 |
| sdague | I wonder why that only sometimes was getting pulled | 11:11 |
| apevec | oh no again beta < release issue | 11:13 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: change sphinx requirement to < 1.1.99 because 1.2b3 exists https://review.openstack.org/61378 | 11:13 |
| sdague | yep | 11:14 |
| apevec | ok, so just local builds, gate is not affected | 11:14 |
| *** sarob has joined #openstack-infra | 11:16 | |
| *** lcestari has joined #openstack-infra | 11:16 | |
| *** Bada has joined #openstack-infra | 11:17 | |
| sdague | apevec: yeh, that's what it seems like | 11:19 |
| sdague | maybe we don't mirror betas to the gate? | 11:19 |
| sdague | which would actually be bad | 11:19 |
| sdague | ttx: https://review.openstack.org/#/c/61380/ | 11:20 |
| apevec | yeah, interesting to check how is mirroring comparing versions | 11:20 |
| ttx | sdague: nice | 11:20 |
| sdague | tox -evenv -- python setup.py build_sphinx | 11:21 |
| sdague | to test it | 11:21 |
| sdague | and see what the output looks like | 11:21 |
| *** rongze has quit IRC | 11:21 | |
| *** nati_ueno has joined #openstack-infra | 11:21 | |
| sdague | if we go down that path, I'll provide some patches to get the tree into something which builds nicely for all the docs | 11:21 |
| *** senk has quit IRC | 11:21 | |
| *** rfolco has joined #openstack-infra | 11:22 | |
| apevec | sdague, could you propose 61378 to stable/havana too? <1.2 hasn't been propagated yet to havana, so better we take yours version | 11:22 |
| sdague | apevec: oh, sure | 11:22 |
| *** Abhishek_ has quit IRC | 11:23 | |
| *** jcoufal has joined #openstack-infra | 11:24 | |
| *** roeyc has quit IRC | 11:24 | |
| *** talluri has quit IRC | 11:26 | |
| *** talluri has joined #openstack-infra | 11:27 | |
| sdague | apevec: https://review.openstack.org/#/c/61383/ | 11:27 |
| apevec | sdague, I'd prefer the same changeid | 11:29 |
| apevec | like https://review.openstack.org/#/q/I0487b4eca8f2755b882689289e3cdf429729b1fb,n,z | 11:29 |
| *** talluri has quit IRC | 11:31 | |
| *** yhe_ is now known as heyongli | 11:33 | |
| sdague | apevec: ok | 11:33 |
| *** bogdando has joined #openstack-infra | 11:33 | |
| *** yamahata_ has quit IRC | 11:33 | |
| tsufiev | jeblair: could you please review the change https://review.openstack.org/#/c/61369/ ? | 11:34 |
| sdague | apevec: I wasn't sure how we were doing backports like that, fixed now | 11:35 |
| sdague | tsufiev: it's at least 5 hrs before jeblair is awake | 11:35 |
| sdague | and you just pinged about that 40 minutes ago | 11:36 |
| *** paul-- has joined #openstack-infra | 11:36 | |
| *** paul-- has quit IRC | 11:37 | |
| *** talluri has joined #openstack-infra | 11:40 | |
| *** jcooley_ has joined #openstack-infra | 11:43 | |
| *** rongze has joined #openstack-infra | 11:44 | |
| openstackgerrit | A change was merged to openstack-infra/release-tools: Initial setup of python project layout https://review.openstack.org/60840 | 11:46 |
| *** jcooley_ has quit IRC | 11:47 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Add mpcut.sh for milestone-proposed branch cutting https://review.openstack.org/61389 | 11:48 |
| *** sarob has quit IRC | 11:48 | |
| *** DinaBelova has joined #openstack-infra | 11:57 | |
| *** hashar has quit IRC | 12:04 | |
| *** nprivalova has joined #openstack-infra | 12:04 | |
| tsufiev | sdague: I did not actually know what timezone he lives in :(. Just tried my luck. | 12:07 |
| tsufiev | sdague: anyway, thank you for the info | 12:09 |
| *** SergeyLukjanov has joined #openstack-infra | 12:10 | |
| *** [1]avishayb has joined #openstack-infra | 12:11 | |
| *** adalbas has joined #openstack-infra | 12:12 | |
| *** [2]avishayb has joined #openstack-infra | 12:12 | |
| openstackgerrit | Timur Sufiev proposed a change to openstack-infra/config: Enable new branch creation for murano. https://review.openstack.org/61369 | 12:13 |
| *** sarob has joined #openstack-infra | 12:13 | |
| *** avishayb has quit IRC | 12:13 | |
| *** [2]avishayb is now known as avishayb | 12:13 | |
| BobBall | sdague: do you know if only re-running the failed jobs on recheck/reverify has been considered? | 12:14 |
| sdague | BobBall: meaning? | 12:14 |
| *** [1]avishayb has quit IRC | 12:15 | |
| BobBall | well - we've got so many jobs running tempest now (and that number is likely to increase) if we have a reverify issued we currently re-run everything. So if there was a failure rate of 1 in 5 and we ran 5 jobs every time, you're going to hit it on every recheck on average. | 12:16 |
| BobBall | We're increasing the number of runs that we do - therefore we're increasing the chance of hitting an issue - even if it's a hypervisor issue | 12:16 |
| ArxCruz | ALL: Does anyone saw this error before? http://paste.openstack.org/show/54815/ | 12:16 |
| BobBall | just thinking about reducing the chances that an instability would wedge check/gate jobs :) | 12:17 |
| BobBall | of course, we want to get rid of all instabilities too and it could be argued that making them have less of an impact would work against them getting fixed | 12:18 |
| *** hashar has joined #openstack-infra | 12:21 | |
| *** nprivalova has quit IRC | 12:21 | |
| *** nprivalova has joined #openstack-infra | 12:26 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: don't have grenade hit the network with apt, as the network breaks https://review.openstack.org/61399 | 12:26 |
| sdague | BobBall: so the right thing to do there is help get rid of the races :) | 12:27 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Make savanna-ml able to create branches https://review.openstack.org/61400 | 12:27 |
| sdague | yeh, I think you'll find the qa and infra teams would be opposed to making it easier to ignore the system races. | 12:27 |
| sdague | ArxCruz: that means you have a syntax error in a test somewhere | 12:29 |
| ArxCruz | sdague: I just ran stack.sh in F19 | 12:29 |
| *** Bada has quit IRC | 12:29 | |
| ArxCruz | checking the logs I don't see anything wrong in devstack installation | 12:30 |
| sdague | ArxCruz: that's a syntax error in the tempest tree somewhere | 12:30 |
| ArxCruz | there's a bunch of warnings like: warning: LocalManifestMaker: standard file '-c' not found | 12:30 |
| BobBall | I understand and agree with that - I'm just concerned that as we grow the chances of hitting gate blocking issues - or even just check blocking issues - could increase out of proportion, which is why I was asking if it had been considered ;) | 12:31 |
| sdague | BobBall: actually, the intent is make smaller and smaller races blocking on the gate | 12:31 |
| *** jcoufal has quit IRC | 12:31 | |
| sdague | so they get resolved | 12:31 |
| sdague | because they don't otherwise | 12:32 |
| BobBall | but it's not just races / bugs in OS that I'm thinking about | 12:32 |
| BobBall | but anyway | 12:32 |
| BobBall | it's a hypothetical discussion ATM ;) | 12:33 |
| BobBall | I'm thinking that by the end of IceHouse we could easily have hyper-v, XS, KVM and vmware tested in the check jobs - and any bugs in those hypervisors would cause issues for everyone | 12:34 |
| BobBall | not just in the drivers | 12:34 |
| sdague | BobBall: they are check jobs though, so they aren't blocking. If it turns out they are racing they can be overridden by the reviewers | 12:38 |
| sdague | jeblair: for when you get up, after all that ... openstack-ci-bugs is a closed group? :P | 12:39 |
| *** ativelkov has quit IRC | 12:41 | |
| *** nprivalova has quit IRC | 12:42 | |
| BobBall | btw, sdague, while I've got you here... | 12:43 |
| *** dkliban has quit IRC | 12:43 | |
| BobBall | How would you / can you trace back from an instance/cinder device to the test that left it hanging around? | 12:43 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Improve storyboard gate jobs https://review.openstack.org/61119 | 12:43 |
| *** dcramer_ has quit IRC | 12:43 | |
| BobBall | I can easily get back to the request sent to cinder + nova but then how do you track it back to the actual test? | 12:43 |
| sdague | BobBall: I don't think you can get the test, you can get the class if you are running with tenant isolation | 12:44 |
| *** beagles has quit IRC | 12:44 | |
| sdague | as the classname of the test class is part of the tenant name | 12:44 |
| BobBall | yeah - I've got to that in a number of cases... but I'm still struggling to figure out why/how these aren't being cleaned up sometimes | 12:45 |
| BobBall | :) | 12:45 |
| *** sarob has quit IRC | 12:45 | |
| sdague | http://youtu.be/qrzFuYyr8c4?t=20m46s | 12:48 |
| sdague | BobBall: so I'd look at the tear down paths for those classes | 12:48 |
| sdague | to make sure they are actually cleaning up correctly | 12:48 |
| BobBall | they seem to be from my naieve understanding | 12:49 |
| BobBall | I'll dig more :) | 12:49 |
| BobBall | I was hoping for some magic way to join the dots | 12:49 |
| sdague | yeh, honestly, I'm not entirely convinced that cinder always cleans up right | 12:49 |
| sdague | the above youtube video is best I've got, that's where I talk about this at summit | 12:50 |
| BobBall | I'm watching ATM | 12:50 |
| sdague | it's a little unfortunate that the charts are so blown out on that | 12:51 |
| *** nprivalova has joined #openstack-infra | 12:51 | |
| *** dkliban has joined #openstack-infra | 12:51 | |
| *** senk has joined #openstack-infra | 12:53 | |
| BobBall | I wonder if we should have a test that runs explicitely at the end to confirm there are no test tenants/VMs/volumes left lying around | 12:53 |
| sdague | yeh, we've talked about that. Realistically it would be in the final tearDown on the basic class where we purge the tenant | 12:54 |
| sdague | needs someone to actually do it | 12:54 |
| BobBall | assuming that you do purge the tenant... I'm left with many tenants | 12:54 |
| BobBall | many = a dozen or so, depending on what actually went wrong | 12:54 |
| sdague | yeh, I expect the tenant purge probably fails when it has resources | 12:54 |
| BobBall | The strange thing is I'm sure I've seen tenants + volumes left when a test passes | 12:55 |
| BobBall | but anyway - needs more investigation before I can tie it down | 12:55 |
| * BobBall loves random failures | 12:55 | |
| sdague | BobBall: yeh, so because deletes are async | 12:55 |
| *** odyssey4me has quit IRC | 12:56 | |
| *** jcoufal-mob has joined #openstack-infra | 12:56 | |
| sdague | they all pass with 202 return | 12:56 |
| BobBall | ah | 12:56 |
| sdague | so unless you explicitly wait around and check, they may not actually delete | 12:56 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: don't have grenade hit the network with apt, as the network breaks https://review.openstack.org/61399 | 12:58 |
| *** thomasbiege has quit IRC | 12:58 | |
| *** eharney has joined #openstack-infra | 12:58 | |
| *** [1]avishayb has joined #openstack-infra | 12:59 | |
| sdague | fungi: so when you get up - https://review.openstack.org/#/c/61399/ probably should fastpath | 13:00 |
| *** thomasbiege has joined #openstack-infra | 13:00 | |
| *** vkozhukalov has quit IRC | 13:00 | |
| *** [2]avishayb has joined #openstack-infra | 13:00 | |
| *** tma996 has joined #openstack-infra | 13:01 | |
| *** avishayb has quit IRC | 13:02 | |
| *** [2]avishayb is now known as avishayb | 13:02 | |
| *** [1]avishayb has quit IRC | 13:03 | |
| *** nprivalova has quit IRC | 13:04 | |
| * fungi is not really here. stack of advisories going out, but i'll look once those are rolling | 13:05 | |
| *** nprivalova has joined #openstack-infra | 13:05 | |
| *** [1]avishayb has joined #openstack-infra | 13:06 | |
| *** sandywalsh_ has quit IRC | 13:07 | |
| sdague | fungi: so if advisories are going to include trying to land patches, that grenade patch should get looked at | 13:08 |
| *** jcoufal-mob_ has joined #openstack-infra | 13:08 | |
| * fungi grumbles | 13:08 | |
| sdague | well that d-g change should get looked at | 13:08 |
| *** avishayb has quit IRC | 13:09 | |
| *** [1]avishayb is now known as avishayb | 13:09 | |
| sdague | because it's bouncing things on rax iad servers atm | 13:09 |
| fungi | i take it grenade is wedging the gate? | 13:09 |
| fungi | too fun | 13:09 |
| sdague | well... only partially | 13:09 |
| fungi | well, rax nodes are check-only, not gate | 13:09 |
| *** b3nt_pin has joined #openstack-infra | 13:10 | |
| *** sarob has joined #openstack-infra | 13:10 | |
| *** dprince has joined #openstack-infra | 13:11 | |
| *** Bada has joined #openstack-infra | 13:11 | |
| sdague | ok, not just rax | 13:11 |
| sdague | http://logs.openstack.org/27/61227/3/gate/gate-grenade-dsvm/3156471/console.html | 13:11 |
| *** jcoufal-mob has quit IRC | 13:11 | |
| sdague | that was an hp cloud one | 13:11 |
| sdague | basically apt.puppetlabs.com is timing out some times | 13:12 |
| *** b3nt_pin has quit IRC | 13:12 | |
| *** b3nt_pin has joined #openstack-infra | 13:12 | |
| sdague | and... it turns out that grenade it actually doing an apt-get update under the covers | 13:12 |
| fungi | yeah, i think we already had a bug open with ideas toward a different solution | 13:13 |
| fungi | in particular, remove the puppetlabs apt repo (and any other unnecessary ones) at the end of the image prep in nodepool scripts | 13:13 |
| sdague | ok, any reason to not just revoke the apt-get update like I did it? | 13:13 |
| fungi | which would allow us to still hit the vendor mirrors | 13:14 |
| sdague | right, but the vendor mirrors will be up to date within 24hrs, right? | 13:14 |
| jog0 | ohh externeneral dependencies | 13:14 |
| jog0 | external* | 13:14 |
| fungi | more or less. it seems like an okay compromise... do we not do apt upgrades in other dsvm jobs? | 13:14 |
| fungi | just grenade? | 13:15 |
| *** nprivalova has quit IRC | 13:17 | |
| sdague | hmmm... we do it in devstack as well | 13:17 |
| fungi | looks like bug 1253774 was prompted by a grenade job as well | 13:17 |
| uvirtbot | Launchpad bug 1253774 in openstack-ci "Reduce number of apt sources that must be up for gate to work" [Medium,Confirmed] https://launchpad.net/bugs/1253774 | 13:17 |
| fungi | i want to say bug 1259911 is a duplicate of that one | 13:18 |
| uvirtbot | Launchpad bug 1259911 in openstack-ci "apt.puppetlabs.com connectivity problem from rax breaks grenade job" [Undecided,In progress] https://launchpad.net/bugs/1259911 | 13:18 |
| *** b3nt_pin is now known as beagles | 13:19 | |
| fungi | i guess i'd want to know, first, whether there are good reasons to be performing an apt-get update/upgrade at all in any of those jobs. i'm okay with considering https://review.openstack.org/61399 as a stop-gap if that's where it's hurting us most at the moment, but i'd rather solve this correctly (whatever correctly means in this case) and consistently once there's time to discuss further | 13:20 |
| sdague | fungi: so, no, this hits us everywhere - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkZhaWxlZCB0byBmZXRjaCA8YSBocmVmPSdodHRwOi8vYXB0LnB1cHBldGxhYnMuY29tL2Rpc3RzL3ByZWNpc2UvUmVsZWFzZS5ncGcnPmh0dHA6Ly9hcHQucHVwcGV0bGFicy5jb20vZGlzdHMvcHJlY2lzZS9SZWxlYXNlLmdwZzwvYT5cIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4Njc2Nzk2MTk3OSwibW9kZSI6InRlcm1zIiwiYW5 | 13:20 |
| sdague | hbHl6ZV9maWVsZCI6ImJ1aWxkX25hbWUifQ== | 13:20 |
| *** yaguang has joined #openstack-infra | 13:20 | |
| *** sandywalsh_ has joined #openstack-infra | 13:20 | |
| sdague | so let me make that generically apply | 13:20 |
| *** Bada has quit IRC | 13:21 | |
| *** prad has joined #openstack-infra | 13:23 | |
| sdague | hmmm... now that I think about it, that fix is probably not the right one. As it could mean an out of data index file for installing from the local mirrors | 13:25 |
| *** talluri has quit IRC | 13:25 | |
| sdague | fungi: so your idea of removing the extra repos is probably a good one. Where should I do that. | 13:25 |
| sdague | This is something we've hit a lot apparently | 13:26 |
| fungi | sdague: https://bugs.launchpad.net/openstack-ci/+bug/1253774/comments/1 | 13:26 |
| uvirtbot | Launchpad bug 1253774 in openstack-ci "Reduce number of apt sources that must be up for gate to work" [Medium,Confirmed] | 13:26 |
| fungi | details there | 13:26 |
| fungi | should be as simple as just deleting the additional repo files. any files in /etc/apt/sources.list.d/ are fair game i think... | 13:27 |
| *** talluri has joined #openstack-infra | 13:27 | |
| fungi | though if you look on one of your d-g nodes you spun up yesterday for manual tests, see what is in sources.list and sources.list.d which might not be needed for the tests to run | 13:28 |
| *** smarcet has joined #openstack-infra | 13:28 | |
| * fungi puts his other hat back on for a couple hours | 13:29 | |
| sdague | hmm... interestingly.... it only seems to trigger the fail on the grenade jobs | 13:29 |
| sdague | the error is passed by in the devstack jobs most of the time | 13:30 |
| *** jcooley_ has joined #openstack-infra | 13:30 | |
| *** dstanek has joined #openstack-infra | 13:35 | |
| *** sarob has quit IRC | 13:35 | |
| *** jcooley_ has quit IRC | 13:35 | |
| *** talluri has quit IRC | 13:38 | |
| sdague | fungi: so - https://review.openstack.org/61418 might actually be a sensible fix as well | 13:38 |
| *** talluri has joined #openstack-infra | 13:38 | |
| *** jorisroovers has joined #openstack-infra | 13:39 | |
| fungi | sdague: that seems a little more resilient... if the update fails and you end up with outdated sources which fail to be able to install particular packages, you'll still get failures later but that can be diagnosed sanely i think | 13:39 |
| sdague | yeh | 13:40 |
| sdague | this actually is only a grenade failure... after looking at logstash | 13:40 |
| sdague | we set -o errexit on all the upgrade scripts | 13:40 |
| sdague | and we do this a lot in them | 13:40 |
| sdague | so it's actually basically ignored effectively in all the devstack jobs | 13:41 |
| sdague | just the grenade jobs make it a failure | 13:41 |
| sdague | also, I thought we linked reviews to launchpad - https://bugs.launchpad.net/openstack-ci/+bug/1259911 ? | 13:42 |
| uvirtbot | Launchpad bug 1259911 in openstack-ci "apt.puppetlabs.com connectivity problem from rax breaks grenade job" [Undecided,In progress] | 13:42 |
| *** talluri has quit IRC | 13:43 | |
| sdague | or is that just launchpad racing and not taking updates | 13:43 |
| *** paul-- has joined #openstack-infra | 13:44 | |
| sdague | ok, if https://review.openstack.org/61418 passes I'll push it through | 13:45 |
| sdague | fungi: please put your +1 on that if you like it. | 13:46 |
| fungi | could be lp. i can add that to the list of things to look at later | 13:46 |
| fungi | will do | 13:46 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Add a check for file with only comments https://review.openstack.org/59090 | 13:46 |
| fungi | i take it that's what devstack already does to get around it elsewhere? | 13:46 |
| sdague | fungi: I dug in because this actually bounced the requirements updates | 13:46 |
| sdague | fungi: well, we don't error there | 13:46 |
| fungi | grr | 13:46 |
| fungi | ahh, okay | 13:47 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix branch logic in req propose script https://review.openstack.org/61304 | 13:47 |
| sdague | grenade is way more aggressive on using set -o errexit | 13:47 |
| sdague | so in devstack... the update returns bad, but we don't check, and don't care | 13:47 |
| fungi | works for me. in for a penny, in for a pound | 13:48 |
| sdague | in grenade, errexit means even if we don't check, bash exits | 13:48 |
| *** dkranz has joined #openstack-infra | 13:48 | |
| fungi | i'd rather we be consistently broken about it than have inconsistent workarounds. makes it easier if we choose differently later | 13:48 |
| sdague | we seem slightly resource starved, the job didn't start yet, waiting on d-g nodes | 13:50 |
| sdague | oh, just got them | 13:51 |
| fungi | more just busy than starved, looks like | 13:51 |
| *** bpokorny has joined #openstack-infra | 13:54 | |
| *** thomasbiege1 has joined #openstack-infra | 13:54 | |
| *** thomasbiege2 has joined #openstack-infra | 13:55 | |
| *** thomasbiege2 has quit IRC | 13:57 | |
| *** thomasbiege has quit IRC | 13:58 | |
| *** thomasbiege1 has quit IRC | 13:58 | |
| sdague | well, now just the waiting | 14:00 |
| *** dkliban has quit IRC | 14:01 | |
| *** NikitaKonovalov has joined #openstack-infra | 14:02 | |
| sdague | jog0: so yuo back in the states now? | 14:02 |
| *** mfer has joined #openstack-infra | 14:04 | |
| *** bogdando has left #openstack-infra | 14:05 | |
| *** prad has quit IRC | 14:06 | |
| *** sarob has joined #openstack-infra | 14:06 | |
| *** nosnos has quit IRC | 14:06 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: use grenade gate numbers, not check queue numbers https://review.openstack.org/61307 | 14:08 |
| *** ryanpetrello has quit IRC | 14:09 | |
| *** ryanpetrello has joined #openstack-infra | 14:09 | |
| *** thomasem has joined #openstack-infra | 14:11 | |
| *** prad has joined #openstack-infra | 14:12 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add graph for gate hit count https://review.openstack.org/61429 | 14:13 |
| jog0 | sdague: nope, in Barcelona but actually able to do work today | 14:14 |
| *** dkliban has joined #openstack-infra | 14:14 | |
| *** sarob has quit IRC | 14:14 | |
| *** paul-- has quit IRC | 14:15 | |
| *** mriedem has joined #openstack-infra | 14:16 | |
| *** jcoufal has joined #openstack-infra | 14:17 | |
| sdague | man, you're as bad as mordred | 14:18 |
| *** NikitaKonovalov has quit IRC | 14:19 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: refactor graphite stanzas for readability https://review.openstack.org/61433 | 14:20 |
| katyafervent | fungi, Hi! | 14:20 |
| jog0 | sdague: I am seeing him on 3 continents in less then 10 days | 14:21 |
| katyafervent | We realized why we can't create a new branch and prepare a fix https://review.openstack.org/#/c/61369/ | 14:21 |
| katyafervent | Please take a look at that. We needed in new branch) | 14:22 |
| *** michchap has quit IRC | 14:24 | |
| jog0 | sdague: I was just about to work on http://paste.openstack.org/show/54592/ | 14:24 |
| jog0 | you want to do that one instead, your js is better then mine | 14:24 |
| *** jcooley_ has joined #openstack-infra | 14:25 | |
| sdague | I actually wanted to get back to the datamine on gerrit for our categorization rate :) | 14:25 |
| sdague | was just doing this while babysitting the grenade patch | 14:25 |
| jog0 | sdague: cool | 14:25 |
| jog0 | i'll give it a whack then | 14:25 |
| sdague | jog0: but if you can layer on top of that last change I put out there | 14:25 |
| sdague | it would be good, as that should clean things up a bunch | 14:26 |
| sdague | jog0: yuo mind if I just land 61433 | 14:27 |
| *** jgrimm has joined #openstack-infra | 14:27 | |
| *** jergerber has joined #openstack-infra | 14:27 | |
| *** jcoufal-mob_ has quit IRC | 14:27 | |
| *** sarob has joined #openstack-infra | 14:27 | |
| jog0 | go fir it | 14:28 |
| *** weshay has joined #openstack-infra | 14:29 | |
| *** jcooley_ has quit IRC | 14:29 | |
| *** eharney has quit IRC | 14:30 | |
| *** sarob has quit IRC | 14:32 | |
| *** zhiyan has joined #openstack-infra | 14:32 | |
| *** nati_ueno has quit IRC | 14:33 | |
| *** [1]avishayb has joined #openstack-infra | 14:34 | |
| *** rnirmal has joined #openstack-infra | 14:34 | |
| *** oubiwan__ has joined #openstack-infra | 14:36 | |
| *** HenryG has quit IRC | 14:36 | |
| *** avishayb has quit IRC | 14:37 | |
| *** [1]avishayb is now known as avishayb | 14:37 | |
| *** mrodden has quit IRC | 14:39 | |
| anteaya | zaro: ah thank you | 14:41 |
| *** alcabrera has joined #openstack-infra | 14:41 | |
| *** odyssey4me has joined #openstack-infra | 14:41 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: refactor graphite stanzas for readability https://review.openstack.org/61433 | 14:44 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add graph for gate hit count https://review.openstack.org/61429 | 14:44 |
| *** jergerber has quit IRC | 14:47 | |
| *** esker has joined #openstack-infra | 14:47 | |
| *** ativelkov has joined #openstack-infra | 14:47 | |
| *** jergerber has joined #openstack-infra | 14:48 | |
| sdague | fungi: do we have a way to move changes in the queue yet? | 14:48 |
| fungi | sdague: nope | 14:48 |
| *** mattray has joined #openstack-infra | 14:49 | |
| fungi | we have ways to blow away and reload the queue in a chosen order, but it's fairly time-intensive | 14:49 |
| sdague | yeh, ok | 14:49 |
| sdague | a lot more is failing in the gate than I expected, slowing that grenade change in getting anywhere | 14:49 |
| *** jorisroovers has quit IRC | 14:49 | |
| fungi | trust me, i feel your pain. much more today than usual | 14:50 |
| *** SushilKM has quit IRC | 14:50 | |
| sdague | yep. | 14:50 |
| *** mrodden has joined #openstack-infra | 14:50 | |
| *** yaguang has quit IRC | 14:51 | |
| sdague | I also need to figure out why the web console updates only seem to take some times | 14:51 |
| fungi | i think there must be something going on with rackspace's network, because the change i merged to get stable requirements updates working doesn't seem to be making it to the proposal slave. timeouts from puppet agent trying to talk to the master | 14:52 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add check pipeline graphs https://review.openstack.org/61444 | 14:52 |
| fungi | gonna have to hand-patch it in | 14:52 |
| fungi | err: Could not retrieve catalog from remote server: Connection reset by peer - SSL_connect | 14:52 |
| *** marun has quit IRC | 14:53 | |
| *** dhellmann has joined #openstack-infra | 14:53 | |
| katyafervent | fungi, Thanks a lot! | 14:54 |
| mattray | what's the proper way to get a repository removed from GitHub? Specifically https://github.com/openstack/openstack-chef | 14:54 |
| mattray | dprince is on board with this, this repo is out of date and unmaintained, we prefer that people use the StackForge cookbooks | 14:54 |
| fungi | mattray: it's a historical artifact. we have an unresolved on-again/off-again discussion about how we should treat projects which we host, which are now defunct | 14:55 |
| fungi | that is one of several | 14:55 |
| mattray | fungi: so we just update the README with that for now? | 14:55 |
| fungi | mattray: i think so. bit shouty warnings all over | 14:55 |
| fungi | er, big | 14:56 |
| mattray | dprince: I'll send you a shouty pull request? | 14:56 |
| dprince | mattray: cool | 14:56 |
| fungi | mattray: gerrit review | 14:56 |
| mattray | fungi: is that repo even under gerrit review? | 14:56 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable new branch creation for murano. https://review.openstack.org/61369 | 14:57 |
| fungi | that's part of the problem. it's a project in gerrit | 14:57 |
| *** jcoufal-mob has joined #openstack-infra | 14:59 | |
| sdague | fungi: that apt.puppetlabs.com problem is happing from hp-az3 as well | 15:03 |
| sdague | so it's probably puppet's fault | 15:03 |
| mattray | dprince: https://review.openstack.org/#/c/61450 | 15:03 |
| mattray | fungi: ^^ | 15:03 |
| dprince | mattray: approved | 15:03 |
| *** alcabrera is now known as alcabrera|afk | 15:03 | |
| *** SergeyLukjanov_ has joined #openstack-infra | 15:03 | |
| mattray | thanks! | 15:03 |
| *** yaguang has joined #openstack-infra | 15:04 | |
| katyafervent | fungi, Do we need to restart anything or changes in acl configs applies right away? | 15:05 |
| fungi | katyafervent: they'll probably take about another 10 minutes | 15:05 |
| sdague | fungi: fwiw, the top of the gate went in 3h 45m ago | 15:05 |
| *** freyes has joined #openstack-infra | 15:05 | |
| katyafervent | fungi, Thanks! | 15:05 |
| fungi | sdague: yeah, doesn't bode well for all these dozen patches i have going into various projects | 15:06 |
| *** SergeyLukjanov has quit IRC | 15:06 | |
| sdague | yeh | 15:06 |
| sdague | how much of the gate is yours right now? | 15:06 |
| *** vkozhukalov has joined #openstack-infra | 15:08 | |
| *** boris-42 has quit IRC | 15:08 | |
| fungi | honestly, no idea. they'll begin triclking in as stable branch maintainers and core reviewers reapply their preapproval from the bugs | 15:08 |
| *** jcoufal has quit IRC | 15:08 | |
| *** eharney has joined #openstack-infra | 15:09 | |
| *** jcoufal-mob has quit IRC | 15:10 | |
| *** SergeyLukjanov has joined #openstack-infra | 15:10 | |
| *** SergeyLukjanov_ has quit IRC | 15:12 | |
| *** HenryG has joined #openstack-infra | 15:13 | |
| jog0 | http://paste.openstack.org/show/54834 | 15:13 |
| jog0 | gate failure rate = 57.76% | 15:13 |
| sdague | jog0: yeh | 15:13 |
| jog0 | sdague: pragmatically calculated this time | 15:14 |
| *** spredzy has joined #openstack-infra | 15:14 | |
| pabelanger | Guess you guys are fighting with apt.puppetlabs.com being down? | 15:15 |
| *** jcoufal has joined #openstack-infra | 15:15 | |
| *** mrmartin has joined #openstack-infra | 15:16 | |
| mrmartin | hi | 15:17 |
| *** jcooley_ has joined #openstack-infra | 15:18 | |
| *** odyssey4me has quit IRC | 15:19 | |
| *** mattray has left #openstack-infra | 15:21 | |
| *** sarob has joined #openstack-infra | 15:21 | |
| *** talluri has joined #openstack-infra | 15:22 | |
| mattymo | fungi, did you see the issue my colleague Roman Vyalov (rvyalov@mirantis.com) reported? | 15:22 |
| *** julim has joined #openstack-infra | 15:23 | |
| *** jcooley_ has quit IRC | 15:23 | |
| sdague | pabelanger: well, there is a work around trying to get through the gate | 15:23 |
| *** NikitaKonovalov has joined #openstack-infra | 15:24 | |
| *** sarob has quit IRC | 15:26 | |
| *** mrmartin has quit IRC | 15:26 | |
| *** sarob has joined #openstack-infra | 15:27 | |
| fungi | mattymo: i haven't caught up on scrollback, mailing list or non-security bugs yet today. i'll try to have a look in another hour or so | 15:28 |
| mattymo | fungi, in gerrit rvyalov@mirantis.com has two entries and neither can be selected as Reviewer | 15:29 |
| mattymo | I imagine somehow he registered twice | 15:29 |
| fungi | mattymo: it's not uncommon to find duplicates in there. i can clean it up later today | 15:29 |
| mattymo | that's perfect | 15:30 |
| mattymo | thanks | 15:30 |
| fungi | gerrit considers e-mail addresses to be unique between accounts, and then has trouble when that assumption is invalid | 15:30 |
| fungi | but there are places where it doesn't prevent address duplication. mainly when it comes in via openid i think | 15:31 |
| mattymo | more like openevil | 15:31 |
| fungi | so having more than one launchpad account or changing/merging lp accounts has a tendency to cause this | 15:31 |
| *** sarob has quit IRC | 15:32 | |
| *** oubiwan__ is now known as oubiwann-lambda | 15:35 | |
| *** datsun180b has joined #openstack-infra | 15:35 | |
| *** rcleere has joined #openstack-infra | 15:36 | |
| *** leifmadsen has quit IRC | 15:36 | |
| mordred | omg backscroll | 15:40 |
| *** jorisroovers has joined #openstack-infra | 15:40 | |
| *** heyongli has quit IRC | 15:41 | |
| sdague | tl;dr - apt.puppetlabs.com periodic flakiness causing grenade job issues - fix in gate - fix probably 4 - 6 hrs from landing based on current reset rate | 15:41 |
| *** jorisroovers has quit IRC | 15:46 | |
| *** jroovers has joined #openstack-infra | 15:46 | |
| *** changbl has quit IRC | 15:46 | |
| *** dims has quit IRC | 15:46 | |
| mordred | wow | 15:48 |
| pabelanger | sdague, Ya, some of my boxes are failing to run puppet, since it is down. | 15:49 |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/nodepool: Support install phase with nodepool https://review.openstack.org/61463 | 15:50 |
| sdague | mordred: also, sphinx req fix was wrong | 15:50 |
| sdague | because 1.2b3 is a pypi version | 15:50 |
| *** senk has quit IRC | 15:51 | |
| jeblair | sdague: what's the puppetlabs fix? | 15:51 |
| sdague | jeblair: https://review.openstack.org/#/c/61418/ | 15:51 |
| mordred | sdague: wait - what? | 15:52 |
| sdague | basically grenade runs a lot more under set -o errexit | 15:52 |
| *** dizquierdo has quit IRC | 15:52 | |
| mordred | sdague: what does "is a pypi version" mean | 15:52 |
| jeblair | you know, if any infra-cores want to review https://review.openstack.org/#/c/60644/ we could push changes to the top of the queue. | 15:53 |
| mordred | jeblair: looking | 15:53 |
| sdague | mordred: pip install sphinx < 1.2 installs 1.2b3 | 15:53 |
| sdague | I have no idea why it doesn't do it in the gate some times | 15:53 |
| matel | jeblair: a patch regading to node install is uploaded: https://review.openstack.org/61463 - could you look at it to see if I'm on the right path or not? | 15:53 |
| mordred | jeblair: gah - I keep opening these to look at them and then keep getting distracteed | 15:53 |
| mordred | sdague: pip version | 15:53 |
| sdague | 1.4.1 | 15:54 |
| jeblair | matel: yes, but probably not right away | 15:54 |
| mordred | it should ignore that. | 15:54 |
| sdague | well it didn't | 15:54 |
| *** julim has quit IRC | 15:54 | |
| matel | jeblair: sure. | 15:54 |
| *** rcleere has quit IRC | 15:55 | |
| sdague | I couldn't add the docs build to openstack/governance until I pinned at 1.1.99 | 15:55 |
| mordred | sdague: and you're sure that docs build was running somewhere sensible? that had pip 1.4? | 15:56 |
| sdague | mordred: on my laptop | 15:56 |
| sdague | under tox | 15:56 |
| mordred | wth. I just reproduced | 15:57 |
| sdague | or... not laptop. my desktop. but my primary computer for dev :) | 15:57 |
| sdague | so used to saying laptop | 15:57 |
| mordred | wait. no I didn't | 15:58 |
| mordred | http://paste.openstack.org/show/54840/ | 15:58 |
| *** yolanda has quit IRC | 15:59 | |
| pabelanger | jeblair, new hotness! Great feature | 15:59 |
| *** jorisroovers has joined #openstack-infra | 15:59 | |
| *** changbl has joined #openstack-infra | 15:59 | |
| clarkb | sdague see my -2 | 15:59 |
| sdague | mordred: I don't know, I went around and around on that one | 16:00 |
| sdague | mordred: sure it acts the right way under tox? | 16:00 |
| sdague | and pbr | 16:00 |
| *** mestery has quit IRC | 16:00 | |
| clarkb | it doesnt see my -2 :) | 16:00 |
| sdague | clarkb: which -2 | 16:00 |
| clarkb | on the requirements change to <1.199 | 16:01 |
| *** rcleere has joined #openstack-infra | 16:01 | |
| sdague | ok | 16:01 |
| *** dims has joined #openstack-infra | 16:01 | |
| clarkb | portante and I sorted this for swift yesterday | 16:01 |
| sdague | ok | 16:02 |
| *** AaronGr_Zzz is now known as AaronGr | 16:02 | |
| sdague | so there are then plenty of trees that need this tox update | 16:02 |
| *** jroovers has quit IRC | 16:03 | |
| clarkb | possibly | 16:03 |
| fungi | yes. most of them in fact | 16:03 |
| clarkb | mordred can weigh in on that? | 16:03 |
| fungi | i did get the requirements proposal re-run with jeblair's latest patch in place, but haven't circled back around to see whether it did the right thing | 16:04 |
| clarkb | mordred usually proposes changes like this across the world. was this just missed? | 16:04 |
| sdague | so I copied tempest bits to governance, so tempest definitely doesn't have this | 16:05 |
| *** SergeyLukjanov has quit IRC | 16:05 | |
| *** NikitaKonovalov has quit IRC | 16:05 | |
| sdague | there were others that were commenting on the patch that they were hitting this issue as well | 16:05 |
| sdague | as it's also a backport, it means that doing tox patches on havana branches as well | 16:05 |
| *** afrittoli has quit IRC | 16:06 | |
| *** DinaBelova has quit IRC | 16:06 | |
| *** rcarrillocruz has joined #openstack-infra | 16:06 | |
| sdague | I'm happy if the answer is mordred fixes the tox world :) | 16:06 |
| *** andreaf has joined #openstack-infra | 16:06 | |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/nodepool: Support install phase with nodepool https://review.openstack.org/61463 | 16:07 |
| yaguang | hi , for this bug sphinx docs gating jobs are broken | 16:07 |
| *** dizquierdo has joined #openstack-infra | 16:08 | |
| yaguang | after fix is merged , but why stable cinder gating still fails | 16:08 |
| clarkb | mordred: have you left us? curious if that was a nova only workaround for site packages? | 16:08 |
| yaguang | bug is here https://bugs.launchpad.net/openstack-ci/+bug/1259511 | 16:09 |
| uvirtbot | Launchpad bug 1259511 in openstack-ci "sphinx docs gating jobs are broken (source_dir must be a directory)" [Critical,Fix released] | 16:09 |
| *** elasticio has joined #openstack-infra | 16:09 | |
| clarkb | yaguang was the fix applied to the stable branch? | 16:09 |
| *** alcabrera|afk is now known as alcabrera | 16:11 | |
| *** dstanek has quit IRC | 16:11 | |
| apevec | yaguang, "Updated from global requirements" aren't merged yet https://review.openstack.org/#/q/owner:jenkins+branch:stable/havana,n,z | 16:11 |
| apevec | fungi, ^^^ btw how come only cinder, ceilo and glance?? | 16:11 |
| apevec | should be all projects, rihgt? | 16:11 |
| *** jorisroovers has quit IRC | 16:11 | |
| fungi | apevec: i've got another patch coming--that fix was insufficient | 16:11 |
| apevec | bad day to go public with cves, heh | 16:12 |
| fungi | apevec: no kidding | 16:12 |
| yaguang | apevec, thanks to point it out, and I think we still need to fix stable/grizzly right ? | 16:12 |
| apevec | yaguang, yes, grizzly needs iso8601 fix and this sphinx cap in each project | 16:13 |
| apevec | latter would be manual job, I don't think "updates from global" is working for stable/grizzly | 16:14 |
| *** dstanek has joined #openstack-infra | 16:14 | |
| fungi | apevec: keep in mind that sphinx fixes for grizzly will probably need to be piecemeal. the global requirements implementation for grizzly is a work in progress since we didn't have any enforcement back then | 16:14 |
| jeblair | apevec: it may or may not work; but the change to reqs for sphinx hasn't landed because of the iso8601 issue: https://review.openstack.org/#/c/61237/ | 16:15 |
| apevec | fungi, ack | 16:15 |
| yaguang | apevec, iso8601 blocks any nova backport to stable/grizzly | 16:15 |
| fungi | and thus there is no strong consistency between requirements lists on grizzly branches of projects | 16:15 |
| apevec | right, re. iso8601 I've asked Glance core to chime in https://review.openstack.org/55939 | 16:15 |
| *** jcooley_ has joined #openstack-infra | 16:15 | |
| apevec | now waiting for their PTL to approve | 16:15 |
| apevec | if anyone sees markwash send him to that review :) | 16:16 |
| jeblair | clarkb: what was your question for mordred? maybe we can figure out the answer. | 16:16 |
| clarkb | jeblair: looking for possible reasons line 9 of nova's tox.ini isn't ubiquitous | 16:17 |
| jeblair | clarkb, mordred, fungi: and if anyone wants to +2 https://review.openstack.org/#/c/60644/ i'll restart zuul and reorder the queue. | 16:17 |
| clarkb | it overrides the tox pip install command disabling prerelease dependencies by default | 16:18 |
| *** jcooley_ has quit IRC | 16:20 | |
| *** weshay has quit IRC | 16:22 | |
| jeblair | clarkb: https://review.openstack.org/#/c/42178/ | 16:22 |
| jeblair | clarkb: my recollection of his plan was that that change should go everywhere | 16:22 |
| jeblair | clarkb: i see no evidence he proposed a change like that to any other project though. | 16:22 |
| jeblair | clarkb: oh, here's 1: https://review.openstack.org/#/c/57170/ | 16:23 |
| mordred | ah. yes. we should put that everywhere | 16:23 |
| mordred | I forgot about that | 16:24 |
| jeblair | mordred: are you in a position to script that now? | 16:24 |
| mordred | jeblair: my internet here is TERRIBLE | 16:24 |
| jeblair | mordred: okay, i will work on spamming all projects with the tox change | 16:25 |
| *** julim has joined #openstack-infra | 16:25 | |
| clarkb | thanks. I am reviewing zuul change now | 16:25 |
| jeblair | does the tox change need to include the sphinx <1.2 version pin? | 16:25 |
| *** reed has joined #openstack-infra | 16:26 | |
| apevec | pip -U prevents prereleases? | 16:27 |
| jeblair | apevec: i think the default tox command allows pre-release, and we're omitting something from it. | 16:28 |
| *** talluri has quit IRC | 16:28 | |
| *** boris-42 has joined #openstack-infra | 16:28 | |
| *** talluri has joined #openstack-infra | 16:28 | |
| *** jcoufal has quit IRC | 16:28 | |
| clarkb | jeblair this is only a problem for folks at home so it can be a separate change | 16:29 |
| jeblair | clarkb: ok, thx | 16:29 |
| clarkb | the mirror protects the gate iirc | 16:29 |
| clarkb | sdague ^ that right? | 16:29 |
| apevec | default: | 16:30 |
| apevec | pip install --pre {opts} {packages} | 16:30 |
| *** prad_ has joined #openstack-infra | 16:31 | |
| sdague | clarkb: it was definitely not being seen in the gate, and I expect that's why | 16:31 |
| *** prad has quit IRC | 16:31 | |
| *** prad_ is now known as prad | 16:31 | |
| jeblair | apevec: i believe '--pre' means 'allow prerelease versions' | 16:32 |
| apevec | yep | 16:32 |
| *** cody-somerville has joined #openstack-infra | 16:32 | |
| *** cody-somerville has quit IRC | 16:32 | |
| *** cody-somerville has joined #openstack-infra | 16:32 | |
| apevec | --pre | 16:32 |
| apevec | Include pre-release and development versions. By default, pip only finds stable versions. | 16:32 |
| *** talluri has quit IRC | 16:33 | |
| jeblair | clarkb: this lgty? https://review.openstack.org/#/c/61473/ | 16:33 |
| *** jcooley_ has joined #openstack-infra | 16:33 | |
| *** yaguang has quit IRC | 16:34 | |
| jeblair | oh, i should probably amend the commit msg to mention the --pre thing | 16:34 |
| clarkb | ++ | 16:34 |
| jog0 | jeblair: can you convert some code into js for me? | 16:34 |
| *** Ryan_Lane has joined #openstack-infra | 16:34 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Check that the target branch exists https://review.openstack.org/61475 | 16:35 |
| fungi | jeblair: clarkb: ^ latest attempt to fix the reqs proposal for stable branches | 16:35 |
| jog0 | jeblair: my javascript is non existent | 16:35 |
| fungi | jeblair: clarkb: current state with the patch from last night... https://jenkins.openstack.org/job/propose-requirements-updates/41/console | 16:35 |
| jog0 | jeblair: http://paste.openstack.org/show/54841 | 16:36 |
| *** weshay has joined #openstack-infra | 16:36 | |
| mordred | apevec: tox decided that backwards compat behavior by default was more important than sanity :) | 16:36 |
| jog0 | code to calculate gate failure rate. sdague ^ same | 16:36 |
| clarkb | jeblair: for the zuul change. if line 556 of the scheduler throws a key error won't that crash the scheduler? A test seems to cover this with less horrible results. I guess the scheduler has a catch all exception handler | 16:36 |
| apevec | mordred, that's... toxic | 16:36 |
| mordred | :) | 16:36 |
| jog0 | jeblair sdague: I got this far and am stuck with js http://paste.openstack.org/show/54842 | 16:37 |
| sdague | jeblair: so I think I can replicate the elastic-recheck page without the pickle or the daemon through gerrit bulk query that could run on cron | 16:37 |
| jeblair | clarkb, mordred: review commit msg: https://review.openstack.org/#/c/61473/ | 16:38 |
| jeblair | sdague: ++ | 16:38 |
| mordred | jeblair: ++ | 16:38 |
| sdague | I'm writing some classes for looking up things in gerrit that elastic search needs, like which bugs are people rechecking against that aren't on our list | 16:38 |
| jeblair | jog0: sorry, firefighting now; will look later if you're still stuck | 16:39 |
| *** SushilKM has joined #openstack-infra | 16:39 | |
| clarkb | jeblair lgtm | 16:39 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Refactor reconfiguration into a management event https://review.openstack.org/60623 | 16:40 |
| jog0 | jeblair: no problem thanks. there is no rush, its more of I don't knwo any javascript and throwing this over the fence. And its something I think we all want | 16:40 |
| jog0 | I will put up a review as as WIP for it | 16:40 |
| jeblair | clarkb: see line 697 for the exception handler that will catch that case and send the traceback to the client | 16:41 |
| fungi | wow, gerrit's diff algorithm made a serious mess of 61475 in the webui | 16:41 |
| sdague | jog0: I'm not really quite sure that that is, but WIP with a big explaination of what the intent is | 16:41 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: WIP: Add gate failure rate number to status page https://review.openstack.org/61480 | 16:43 |
| jog0 | sdague: ^ | 16:43 |
| zaro | good morning | 16:44 |
| *** rcleere has quit IRC | 16:45 | |
| clarkb | jeblair see inline comment | 16:45 |
| clarkb | once I wrap my head around the inline comment there I think this may be ready to go | 16:45 |
| clarkb | zaro: morning | 16:46 |
| *** rcleere has joined #openstack-infra | 16:46 | |
| *** afazekas has quit IRC | 16:46 | |
| jeblair | clarkb: rplied | 16:47 |
| *** ^d has joined #openstack-infra | 16:48 | |
| fungi | i love it when changes merge while i'm reviewing them, and i don't even notice for 10 minutes. reminds me of just how painfully slow my code reading comprehension really is | 16:50 |
| clarkb | jeblair: +2 | 16:50 |
| jeblair | fungi: or how thorough you are | 16:50 |
| *** SushilKM has quit IRC | 16:51 | |
| fungi | well, i also don't like to hold up the boat | 16:51 |
| *** rongze_ has joined #openstack-infra | 16:51 | |
| *** dcramer_ has joined #openstack-infra | 16:51 | |
| jeblair | fungi: that doomed ship's gotta sail! | 16:52 |
| fungi | undeed | 16:52 |
| *** rongze__ has joined #openstack-infra | 16:52 | |
| *** rongze has quit IRC | 16:52 | |
| * fungi hoists the mainsail | 16:53 | |
| *** gothicmindfood has joined #openstack-infra | 16:54 | |
| fungi | 60644 lgtm | 16:54 |
| *** SushilKM has joined #openstack-infra | 16:56 | |
| *** mestery has joined #openstack-infra | 16:56 | |
| *** rongze_ has quit IRC | 16:56 | |
| jeblair | clarkb: several projects do have the new tox settings (ceilo, keystone) but not all | 16:56 |
| *** klrmn has joined #openstack-infra | 16:57 | |
| clarkb | fungi: for 61475, is it possible the branch won't be in the packed refs? we may want to try a git branch -a instead? | 16:57 |
| *** SushilKM has quit IRC | 16:58 | |
| *** SushilKM has joined #openstack-infra | 16:59 | |
| fungi | clarkb: good question. the ones i tested manually, the only way to confirm the remote branches existed without changing into the directory and using git commands was to grep through the pack. i'll do a new patchset with what i think the git branch -a solution would look like instead | 17:00 |
| clarkb | fungi: loose refs like loose objects are technically possible I think | 17:00 |
| jeblair | what is 'distribute=False' in a tox.ini? | 17:00 |
| fungi | i thought that told tox not to install distribute (install setuptools instead) | 17:01 |
| fungi | since it needs to make the virtualenv with one or the other | 17:01 |
| jeblair | i've only seen it in oslo.config, not any others | 17:01 |
| *** herndon has joined #openstack-infra | 17:02 | |
| *** herndon has quit IRC | 17:04 | |
| *** pmathews has joined #openstack-infra | 17:04 | |
| sdague | jog0: I take it you would probably react poorly to me adding 37 MB of gerrit test data to er? | 17:04 |
| *** acabrera has joined #openstack-infra | 17:04 | |
| *** HenryG has quit IRC | 17:04 | |
| *** dcramer_ has quit IRC | 17:05 | |
| *** HenryG has joined #openstack-infra | 17:05 | |
| zaro | mordred: questions for you on https://review.openstack.org/#/c/60348 | 17:05 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Check that the target branch exists https://review.openstack.org/61475 | 17:06 |
| fungi | clarkb: ^ git branch -a alternative | 17:06 |
| *** alcabrera has quit IRC | 17:07 | |
| zaro | clarkb: too cold to come in today | 17:07 |
| clarkb | fungi: ty | 17:08 |
| mordred | jeblair: it's not needed anymore. it used to be needed | 17:08 |
| *** acabrera is now known as alcabrera | 17:08 | |
| clarkb | zaro: :( I am headed in shortly | 17:08 |
| jeblair | mordred: ok. i'll probably just ignore for now; but we could probably do a pass through and remove that. | 17:09 |
| zaro | clarkb: have a nice walk :) | 17:09 |
| mordred | zaro: we do the publisher in the check jobs so that we can install the war produced on review-dev without needing to land the change first | 17:09 |
| mordred | zaro: you'll notice that it publishes to a subdir | 17:10 |
| *** tma996 has quit IRC | 17:10 | |
| mordred | zaro: 'tarballs/ci/test/' | 17:10 |
| *** rongze__ has quit IRC | 17:11 | |
| *** freyes has quit IRC | 17:11 | |
| zaro | ok. i see the reason. but don't you want to install war after gate ? | 17:11 |
| mordred | nope | 17:11 |
| mordred | because a lot of times the way we test it for the gate is by instaling the war on review-dev | 17:12 |
| mordred | and that might need a few iterations | 17:12 |
| zaro | gotcha. will add deployment to the check job then. | 17:12 |
| mordred | thanks! | 17:12 |
| *** nprivalova has joined #openstack-infra | 17:12 | |
| mordred | the other thing I'm meh on - I'm fine with it being there | 17:13 |
| sdague | 763 reviews have seen activity in the last 24hrs.... if I make my dataset 4hrs I can cut it down to about 350 reviews ... still kind of impressive | 17:13 |
| *** bodepd has quit IRC | 17:13 | |
| *** bodepd has joined #openstack-infra | 17:13 | |
| *** resker has joined #openstack-infra | 17:14 | |
| *** weshay has quit IRC | 17:14 | |
| *** hashar has quit IRC | 17:14 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add a promote client command https://review.openstack.org/60644 | 17:15 |
| *** esker has quit IRC | 17:15 | |
| *** weshay has joined #openstack-infra | 17:16 | |
| sdague | woot | 17:16 |
| *** rongze has joined #openstack-infra | 17:17 | |
| sdague | jeblair: 61418,2 is the one we want on top | 17:17 |
| fungi | zaro: part of the backstory there is that sometimes the (old) gerrit build process would pack javascript improperly and result in broken warfiles | 17:18 |
| *** johnthetubaguy has quit IRC | 17:18 | |
| fungi | seemingly at random | 17:18 |
| dkranz | mordred: Do you have the gerrit link for the "gate on client library stability" patch? | 17:19 |
| mordred | dkranz: https://review.openstack.org/#/c/41931/ | 17:19 |
| dkranz | mordred: Thx | 17:20 |
| *** thomasem has quit IRC | 17:20 | |
| mordred | dkranz: if you fix it, I'll buy you a puppy | 17:20 |
| jeblair | sdague: ack, will start on that in ~10 mins | 17:20 |
| *** SergeyLukjanov has joined #openstack-infra | 17:20 | |
| jeblair | sdague: is there a bug for the apt.puppetlabs issue? | 17:24 |
| jeblair | fungi: ^? | 17:24 |
| tsufiev | what can still prevent murano from creating new branches? Though https://review.openstack.org/#/c/61369/ is merged, we still don't see 'create' permission in refs/heads/*, nor can actually create a branch | 17:24 |
| sdague | jeblair: https://bugs.launchpad.net/openstack-ci/+bug/1259911 | 17:25 |
| uvirtbot | Launchpad bug 1259911 in openstack-ci "apt.puppetlabs.com connectivity problem from rax breaks grenade job" [Undecided,In progress] | 17:25 |
| fungi | jeblair: bug 1253774 and bug 1259911 | 17:26 |
| uvirtbot | Launchpad bug 1253774 in openstack-ci "Reduce number of apt sources that must be up for gate to work" [Medium,Confirmed] https://launchpad.net/bugs/1253774 | 17:26 |
| uvirtbot | Launchpad bug 1259911 in openstack-ci "apt.puppetlabs.com connectivity problem from rax breaks grenade job" [Undecided,In progress] https://launchpad.net/bugs/1259911 | 17:26 |
| fungi | just found them in scrollback | 17:26 |
| fungi | oh, i guess sdague already did too | 17:26 |
| jeblair | i think 1259911 is the one we want to promote for reverifies | 17:27 |
| * zaro wonders if it's kosher to share projects via a review? | 17:27 | |
| fungi | tsufiev: it's possible there's a problem with puppet applying that change on gerrit | 17:27 |
| jeblair | zaro: what do you mean? | 17:27 |
| ativelkov | fungi: is there anything we can do to fix it? | 17:28 |
| *** SushilKM has quit IRC | 17:28 | |
| fungi | oh, right, clarkb: jeblair: mordred: also seen this morning but haven't had time to troubleshoot yet. puppet timeouts on ssl interactions with ci-puppetmaster. i worked around it on proposal.slave where i saw it by hand-applying the change i was waiting on | 17:28 |
| jeblair | fungi: we should check ci-puppetmaster load | 17:28 |
| fungi | but there's every chance it's hitting us elsewhere too--i haven't had time to look yet | 17:28 |
| zaro | jeblair: i wanted to share my modified jjb project with mgagne . | 17:28 |
| fungi | having a look now | 17:29 |
| jeblair | zaro: what do you mean by project? | 17:29 |
| zaro | jeblair: changes i guess. changes that i would never want merged. | 17:29 |
| fungi | ativelkov: patience. we've got a stack of problems in the infrastructure and gating we're trying to resolve all at once | 17:29 |
| zaro | jeblair: so maybe upload then abandone | 17:29 |
| fungi | load average on the puppet master is trivial, but maybe it wasn't earlier. checking graphs | 17:30 |
| ativelkov | fungi: thanks | 17:30 |
| mgagne | zaro: draft :D | 17:30 |
| *** blamar has joined #openstack-infra | 17:30 | |
| *** gyee has joined #openstack-infra | 17:30 | |
| mordred | zaro: yeah - or draft - which you can then explicitly give permission to someone else | 17:30 |
| zaro | what's the difference? | 17:30 |
| jeblair | mordred: can you look at tox.ini in savanna-dashboard? they have an extra testenv section and i wonder what's going on. | 17:31 |
| *** ^demon has joined #openstack-infra | 17:31 | |
| *** ^demon has joined #openstack-infra | 17:31 | |
| mordred | yup | 17:32 |
| *** reed has quit IRC | 17:32 | |
| SergeyLukjanov | jeblair, mordred, is the tox.ini incorrect in savannas-dashboard? | 17:32 |
| mordred | SergeyLukjanov: it's weird - what do you guys use the tests env for? | 17:33 |
| *** ^demon has quit IRC | 17:33 | |
| *** dims has quit IRC | 17:33 | |
| SergeyLukjanov | mordred, does you mean [testenv:tests]? | 17:33 |
| mordred | SergeyLukjanov: and why do you have sitepackages=true? is that on purpose? | 17:33 |
| SergeyLukjanov | mordred, if yes - it's for selenium tests | 17:33 |
| mordred | SergeyLukjanov: yes | 17:33 |
| *** ^demon has joined #openstack-infra | 17:33 | |
| mordred | SergeyLukjanov: how do you call it? | 17:33 |
| mordred | oh. duh. brain dead | 17:34 |
| SergeyLukjanov | mordred, tox -e tests, but we're executing it in savanna-ci | 17:34 |
| SergeyLukjanov | mordred, checking file history to understand why sitepackages=true was used | 17:34 |
| *** ^d has quit IRC | 17:35 | |
| jeblair | we run selenium for django | 17:35 |
| jeblair | gah | 17:35 |
| jeblair | we run selenium for horizon | 17:35 |
| openstackgerrit | James E. Blair proposed a change to openstack-dev/hacking: Add -U to pip install command in tox.ini https://review.openstack.org/61517 | 17:36 |
| *** nprivalova has quit IRC | 17:36 | |
| *** ^demon has quit IRC | 17:36 | |
| jeblair | okay, i've done all the openstack projects in projects.txt | 17:36 |
| *** ^d has joined #openstack-infra | 17:36 | |
| jog0 | sdague: haha yes | 17:37 |
| SergeyLukjanov | mordred, I'll remove sitepackages=true asap | 17:38 |
| mordred | SergeyLukjanov: cool | 17:38 |
| mordred | SergeyLukjanov: you can probably just remove all traces of it | 17:39 |
| *** dims has joined #openstack-infra | 17:39 | |
| jeblair | SergeyLukjanov: can you look at what we do for horizon and see if you can run the selenium tests in the openstack gate? | 17:39 |
| SergeyLukjanov | jeblair, I do | 17:39 |
| SergeyLukjanov | jeblair, we have someone assigned to it and verdict was that - nope, we can't | 17:39 |
| *** oubiwann-lambda has quit IRC | 17:40 | |
| SergeyLukjanov | jeblair, oh, I remember, this tests are not functional, but integration - we're creating hadoop clusters and run jobs on them | 17:40 |
| *** chandankumar has quit IRC | 17:41 | |
| jeblair | fungi: i logged into zuul; puppet has not updated the git repo since last night | 17:42 |
| jeblair | fungi: continual ssl errors | 17:43 |
| *** thomasem has joined #openstack-infra | 17:43 | |
| *** thomasem has quit IRC | 17:44 | |
| *** thomasem has joined #openstack-infra | 17:44 | |
| fungi | jeblair: yes, it looks like the puppet master process was restarted this morning at 05:37 (haven't narrowed down the cause yet, assuming cron job, maybe log rotation, no mention of it in the dpkg log), and after that the graphs for the master flatlined | 17:44 |
| fungi | trying to restart it but i think i may have the wrong initscript for the master process though... i guess it's not puppetqd? | 17:45 |
| SergeyLukjanov | jeblair, thanks for the tox.ini update | 17:45 |
| SergeyLukjanov | mordred, done, will be merged soom | 17:46 |
| SergeyLukjanov | soon* | 17:46 |
| jeblair | fungi: apt.puppetlabs being down plus this is a strange coincidenc. | 17:46 |
| fungi | jeblair: i feel the same | 17:46 |
| SergeyLukjanov | is there still problems with propose-requirements-updates on non-master branch? | 17:47 |
| *** dstanek has quit IRC | 17:47 | |
| jeblair | restarting zuul | 17:47 |
| mordred | fungi: puppetmaster is run through apache | 17:47 |
| mordred | it's a passeger thing | 17:47 |
| fungi | SergeyLukjanov: yes, https://review.openstack.org/61475 | 17:47 |
| mordred | fungi: restarting apache is what you want | 17:47 |
| *** dstanek has joined #openstack-infra | 17:47 | |
| fungi | mordred: aha! thanks muchj | 17:48 |
| sdague | grrr.... gerritlib is driving me nuts if I search with multiple queries it's only returning data for the first one | 17:48 |
| *** jooools has quit IRC | 17:48 | |
| SergeyLukjanov | fungi, oh | 17:48 |
| fungi | mordred: looks like you already did. thanks again | 17:48 |
| mordred | fungi: I did not | 17:49 |
| mordred | fungi: I did not touch anything - I just logged in to look at the load | 17:49 |
| *** senk has joined #openstack-infra | 17:49 | |
| fungi | er | 17:49 |
| fungi | perhaps apache started new master threads on its own | 17:49 |
| fungi | s/threads/processes/ | 17:49 |
| fungi | restarting now in that case | 17:50 |
| *** rcarrillocruz1 has joined #openstack-infra | 17:50 | |
| fungi | in case it's an apache issue | 17:50 |
| pabelanger | I've had a heck of a time with puppetmaster-passenger recently, so much so, I actually when back to just using the native puppetmaster | 17:50 |
| *** rcarrillocruz has quit IRC | 17:50 | |
| jeblair | zuul restarted and queue reloaded | 17:52 |
| fungi | jeblair: clarkb: mordred: after a good clean restart of apache on ci-puppetmaster, i can pull puppet configs from it successfully again | 17:52 |
| mordred | fungi: that's SPECTACULAR | 17:52 |
| *** acabrera has joined #openstack-infra | 17:52 | |
| fungi | not so much spectacular as really annoying | 17:53 |
| fungi | the other spectacular | 17:53 |
| sdague | jeblair: neat | 17:53 |
| sdague | with any luck we'll start landing things quickly | 17:53 |
| *** senk has quit IRC | 17:53 | |
| *** alcabrera has quit IRC | 17:55 | |
| *** SushilKM has joined #openstack-infra | 17:56 | |
| *** acabrera is now known as alcabrera | 17:56 | |
| *** derekh has quit IRC | 17:57 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove 'reverify no bug' https://review.openstack.org/61522 | 17:59 |
| mordred | jeblair: can I +A that? or should I just +2? | 18:01 |
| jeblair | mordred: +A at will | 18:02 |
| *** sdake_ has joined #openstack-infra | 18:02 | |
| mgagne | mordred: ironically, gate will fail and reverify no bug will be used. | 18:02 |
| fungi | the -dev ml got a couple days notice | 18:02 |
| jeblair | #status alert Grenade gate infra issues: use "reverify bug 1259911" | 18:03 |
| openstackstatus | NOTICE: Grenade gate infra issues: use "reverify bug 1259911" | 18:03 |
| uvirtbot | Launchpad bug 1259911 in openstack-ci "apt.puppetlabs.com connectivity problem from rax breaks grenade job" [Undecided,In progress] https://launchpad.net/bugs/1259911 | 18:03 |
| *** ChanServ changes topic to "Grenade gate infra issues: use "reverify bug 1259911"" | 18:03 | |
| sdague | jog0: how about 4M of json? that's about as small as I can get and still have it be a real slice with enough variants | 18:03 |
| pmathews | mordred: Can I get you to review this: https://review.openstack.org/#/c/59875/3 | 18:04 |
| jog0 | sdague: I can live with that as long as its not 4k files or something | 18:04 |
| sdague | it's one file | 18:05 |
| jog0 | cool, and I assume you will document how it was created so when the schema we use gets out of date and we need a new file | 18:05 |
| sdague | yep, | 18:05 |
| *** che-arne has quit IRC | 18:06 | |
| *** reed has joined #openstack-infra | 18:06 | |
| sdague | I mostly want to have this in the tree so I can write the rest of the things that go against this all with a fake backend, so I'm not constantly waiting 60 seconds to get the result set back over the network | 18:06 |
| *** cody-somerville has quit IRC | 18:08 | |
| jeblair | sdague: 60 seconds? please consider gerrit server load. :) | 18:08 |
| sdague | jeblair: well, that's why I'm trying to get a local dump :) | 18:09 |
| jeblair | sdague: that's great; i'm just worried about running it production too if there are queries that take 60 seconds to return | 18:10 |
| jeblair | sdague: not saying don't do it -- just asking you to keep it in mind, and we can evaluate or tune later as needed of course | 18:10 |
| sdague | sure | 18:10 |
| sdague | let me get real timings, it might just feel like 60s | 18:11 |
| sdague | it looks like gerrit is saying these are 8, 4, and 1 s queries (need all three) for last 4hrs of active patches | 18:12 |
| jog0 | I wonder what the load on gerrit is | 18:13 |
| sdague | the 2w query is probably going to agregate to 60s, as gerrit comes back with only 500 rows at a time, and you need to go across cursors to them | 18:13 |
| sdague | yeh, it's a good question. fwiw I've been doing this all morning, a lot | 18:13 |
| sdague | finally got far enough that a cache is going to work | 18:14 |
| jog0 | what are you querying gerrit for anyway? | 18:14 |
| sdague | all open, merged, or abandoned patches that have since activity in the last "foo" time period (2w in the real world, but 4h in this smaller set) | 18:15 |
| sdague | to gather up all the recheck comments in the time period that er runs | 18:15 |
| sdague | to cross reference them, and see that 50 bugs that people rechecked against that we don't have fingerprints for | 18:15 |
| sdague | it will be cron runnable low frequency | 18:16 |
| fungi | i'm willing to bet at least half of those are people still incorrectly rechecking against the bug their patch fixes | 18:16 |
| sdague | fungi: probably | 18:16 |
| sdague | I could check :) | 18:17 |
| *** Protux has quit IRC | 18:17 | |
| fungi | that would be a funny e-r "hey numbskull" check | 18:17 |
| *** ^d has quit IRC | 18:18 | |
| jog0 | sdague: why not look at status.openstack.org/rechecks ? | 18:18 |
| *** ^d has joined #openstack-infra | 18:18 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup tooling and jobs for openstack Gerrit ver 2.8 and beyond https://review.openstack.org/60348 | 18:19 |
| sdague | jog0: because that's extra fragile code that keeps a local pickle database, and reads the gerrit stream | 18:19 |
| sdague | and if it hangs it just looses events until someone notices | 18:19 |
| sdague | which it does, plenty | 18:19 |
| sdague | the point was to get rid of that code base | 18:19 |
| clarkb | right we are trying to move more towards e-r instead | 18:19 |
| *** Protux has joined #openstack-infra | 18:20 | |
| jeblair | if it's low frequency, users won't notice new bugs until the next run | 18:20 |
| jog0 | sdague: heh it is really fragile, while your at it want to track the recheck no bugs as well? | 18:20 |
| jeblair | sdague: a local cache would be a great way to avoid running the 2w query often. | 18:20 |
| sdague | jeblair: so the reality is | 18:21 |
| sdague | the 2w query is only 3x - 4x the 4hr query | 18:21 |
| sdague | which is sort of due to the longevity of a patch | 18:22 |
| *** senk has joined #openstack-infra | 18:22 | |
| sdague | so the small queries may not really help on load, but we can figure that all out later | 18:22 |
| fungi | so, the grenade change merged. and i see lots near completion in teh gate with successful grenade jobs | 18:24 |
| clarkb | woot | 18:25 |
| jeblair | sdague: gerrit currently handles about one query per second; moreover, for some reason the people who write these bots start a new ssh connection for each query. there is a sizable load from unoptimized query bots. i'd like it if we, who actually know how this works, could do a little better. :) | 18:25 |
| sdague | jeblair: sure | 18:25 |
| sdague | let me get the functionality fleshed out first, then we can figure out the right optimization | 18:26 |
| jeblair | sdague: ++ | 18:26 |
| sdague | jeblair: also, is gerritlib correctly reusing the ssh connection? | 18:26 |
| jeblair | sdague: i'm attempting to help by brainstorming ideas, not prematurely reviewing. :) | 18:26 |
| *** senk has quit IRC | 18:26 | |
| sdague | my intent was to punt to that as assuming it does the right thing | 18:27 |
| jeblair | sdague: it does not. gerritlib needs updates from zuul.lib.gerrit to do that. | 18:28 |
| jeblair | sdague: (and then zuul should depend on gerritlib) | 18:28 |
| sdague | ok, that would be cool to do | 18:29 |
| jeblair | there's some low hanging python-hacking fruit ^ | 18:29 |
| clarkb | it does for event watching but nothing else | 18:29 |
| fungi | and would have a pretty sizable impact, both for random people interating with us via their own bots/automation and also reducing load on our gerrit | 18:29 |
| fungi | er, integrating | 18:30 |
| fungi | we'd trade ssh auth and setup/teardown overhead for more perpetually-open connections, but i suspect that'll still be a net win | 18:31 |
| *** ^demon has joined #openstack-infra | 18:31 | |
| *** ^d has quit IRC | 18:32 | |
| *** ^demon is now known as ^d_ | 18:32 | |
| *** ^d_ is now known as ^d | 18:32 | |
| jeblair | fungi: yeah, most of these are effectively perpetually open anyway. | 18:33 |
| dims | huh! bug 1259907 marked as incomplete | 18:33 |
| uvirtbot | Launchpad bug 1259907 in openstack-ci "check-grenade-dsvm marked as FAILED - n-api/g-api Logs have errors" [Undecided,New] https://launchpad.net/bugs/1259907 | 18:33 |
| fungi | dims: i hit it just a few minutes ago on a keystone patch | 18:34 |
| *** praneshp has joined #openstack-infra | 18:35 | |
| sdague | fungi: which job? | 18:35 |
| fungi | ah! dolphm suggests it's actually bug 1260015 | 18:35 |
| uvirtbot | Launchpad bug 1260015 in openstack-ci "PKI token contains the string "ERROR"" [Undecided,New] https://launchpad.net/bugs/1260015 | 18:35 |
| fungi | that's so craaaaaazy | 18:35 |
| *** ^d has quit IRC | 18:35 | |
| fungi | dkranz: you'll like that one | 18:36 |
| *** markmc has quit IRC | 18:36 | |
| *** krotscheck has joined #openstack-infra | 18:36 | |
| praneshp | Can someone please review https://review.openstack.org/#/c/60132/ ? It was approved, then automatic merge failed, and I merged changes. Now waiting for review and approval again. | 18:36 |
| openstackgerrit | A change was merged to openstack-infra/config: Check that the target branch exists https://review.openstack.org/61475 | 18:36 |
| fungi | sdague: on check-tempest-dsvm-full | 18:36 |
| morganfainberg | LOL seriously, 'ERROR' in the token's string | 18:37 |
| sdague | fungi: yeh, I never saw it fail those other jobs | 18:37 |
| morganfainberg | thats a new one. | 18:37 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove 'reverify no bug' https://review.openstack.org/61522 | 18:37 |
| jeblair | sdague: doesn't e-r use stream-events already? | 18:38 |
| *** simonmcc has quit IRC | 18:38 | |
| *** dizquierdo has quit IRC | 18:38 | |
| jeblair | (for the real-time identification, not graphs, obviously) | 18:39 |
| sdague | jeblair: yes | 18:39 |
| sdague | but it keeps no state | 18:39 |
| *** nati_ueno has joined #openstack-infra | 18:40 | |
| jeblair | sdague: how about combining some ideas here, have that do real time updates, but also have it run queries at startup to make sure you don't miss data | 18:40 |
| sdague | if we want to put a database in front of gerrit, we can do that. But it's a different thing. | 18:40 |
| jeblair | sdague: a cache. | 18:40 |
| sdague | so I think basically we just need to decide are the er tools stateless, because they can discover everything from gerrit / er / lp | 18:41 |
| jeblair | sdague: maybe a mem. cache. | 18:41 |
| *** blamar has quit IRC | 18:41 | |
| jeblair | d. | 18:41 |
| sdague | yeh, let's not build a caching architecture until we actually get the set of things it's doing done :) | 18:41 |
| *** blamar has joined #openstack-infra | 18:41 | |
| sdague | especially not if it's 0.1% of the gerrit load with all those other bots out there | 18:42 |
| zaro | fungi: could you tell me what do i need to change to add my key for review-dev? | 18:42 |
| dkranz | fungi: Heh. That is obviously a bug I should fix. | 18:42 |
| *** SushilKM has quit IRC | 18:42 | |
| jeblair | sdague: we're already at caching scale. we're already caching. you're about to drop that and lose functionality in the process. i fell like we've already done the 'sketch out what this would look like quickly' bit, we're at 'do it right'. | 18:43 |
| *** simonmcc has joined #openstack-infra | 18:43 | |
| jeblair | so again, i'm trying to brainstorm ideas so that we don't have to keep rewriting this. | 18:43 |
| fungi | zaro: yeah, there's a couple files you need to change | 18:43 |
| *** AaronGr is now known as AaronGr_afk | 18:43 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Document adding bug signatures to e-r. https://review.openstack.org/61298 | 18:43 |
| fungi | zaro: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/users.pp | 18:44 |
| fungi | zaro: put a user entry for yourself there with your ssh public key | 18:44 |
| dims | pki token ERROR - LOL | 18:44 |
| fungi | zaro: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/review_dev.pp | 18:45 |
| *** rossella_s has quit IRC | 18:45 | |
| sdague | jeblair: so I actually think the set of data queries we're going to want isn't yet well defined, so I'd rather actually work out that code first, then figure out the right performance afterwards. | 18:45 |
| *** dcramer_ has joined #openstack-infra | 18:45 | |
| fungi | zaro: add a relize for your user in that file similar to the one in http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/groups_dev.pp#n24 | 18:46 |
| fungi | zaro: also add a review comment once you submit it, stating that you've read and agree to follow http://ci.openstack.org/sysadmin.html#ssh-access | 18:46 |
| *** ^d has joined #openstack-infra | 18:47 | |
| *** jpich has quit IRC | 18:47 | |
| *** pcrews has joined #openstack-infra | 18:47 | |
| sdague | anyway, I'm way past lunch here, so I got to go do that. biab | 18:47 |
| jeblair | sdague: i think the queries are orthogonal to the architecture. please consider designing it to at least handle the scale the current tools do. that's all i'm asking. | 18:48 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add packstack to gerritbot https://review.openstack.org/60240 | 18:48 |
| jeblair | sdague: it doesn't have to be fragile. zuul and gerritbot, for instance, are extremely reliable. | 18:48 |
| clarkb | I am just goint to approve 60240, trying to flush the list of changes that have previously been determined to be mergeable. but it will conflict with entropy >_> so I will wait a minute I guess | 18:50 |
| praneshp | jeblair, clarkb: can you approve https://review.openstack.org/#/c/60132/ | 18:52 |
| praneshp | that's the change to entropy, and it was already approved once, just a merge issue | 18:52 |
| clarkb | praneshp: it has been approved | 18:52 |
| praneshp | that way your new approval of 60240 will be fine I think | 18:52 |
| clarkb | praneshp: no I need to rebase again | 18:53 |
| praneshp | oh thanks clarkb | 18:53 |
| fungi | stable branch proposals seem to be working better now... https://jenkins.openstack.org/job/propose-requirements-updates/42/console | 18:53 |
| clarkb | fungi: woot | 18:53 |
| fungi | projects with no stable/havana are being skipped properly, and projects with are getting checked out and processed | 18:54 |
| fungi | success! | 18:54 |
| *** thomasem has quit IRC | 18:54 | |
| fungi | apevec: see if your havana requirements update changes look any better now | 18:54 |
| apevec | much better https://review.openstack.org/#/q/status:open+branch:stable/havana+topic:openstack/requirements,n,z | 18:55 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: give zaro access to review-dev server https://review.openstack.org/61542 | 18:55 |
| fungi | i think no fewer than four contributors touched and iteratively fixed bugs in that proposal script over the past 24 hours | 18:55 |
| zaro | fungi^ | 18:55 |
| zaro | thanks | 18:55 |
| clarkb | fungi: now we need to do the same with manage-projects | 18:56 |
| fungi | clarkb: and run-mirror | 18:56 |
| clarkb | I am not sure how popular this opinion may be, but I am half tempted to revert all of the changes that have been made to it to go back to the stateless version of the script | 18:56 |
| clarkb | then we can add the statefulness back in | 18:56 |
| fungi | i'm on the fence there. i think there are also improvements interspersed (additional debug logging for example) | 18:57 |
| clarkb | right, but we didn't need that additional logging until after we added all of the features >_> | 18:58 |
| fungi | heh | 18:58 |
| jeblair | clarkb: lost me -- what does the propose script have to do with manage-projects? | 18:58 |
| clarkb | jeblair: nothing directly, just that manage-projects is still apparently broken | 18:58 |
| clarkb | I was suggesting we throw bodies at the problem until it goes away | 18:58 |
| jeblair | clarkb: ah. okay. should we stop approving new projects? | 18:59 |
| *** prad has quit IRC | 18:59 | |
| fungi | as we discovered last night after sarob popped in to point out that stackforge/milk was uncloneable | 18:59 |
| clarkb | jeblair: no, I think the failure mode is much cleaner right now | 18:59 |
| jeblair | clarkb: uncloneable does not sound clean. | 18:59 |
| clarkb | jeblair: well it means we don't need to fix zuul by hand | 18:59 |
| jeblair | clarkb: how does it get fixed? | 19:00 |
| *** AaronGr_afk is now known as AaronGr | 19:00 | |
| clarkb | manage-projects? we need to sort out why gerrit project creation is failing these projects | 19:00 |
| fungi | i've gotten a bit faster about being able to clear and reimport repositories for gerrit and check for broken mirrors on review, zuul and the git server farm | 19:00 |
| openstackgerrit | A change was merged to openstack-infra/config: Added new project, entropy https://review.openstack.org/60132 | 19:00 |
| jeblair | clarkb: no, how doe 'uncloneable' get fixed? | 19:00 |
| clarkb | jeblair: rerun manage-projects is what fungi did | 19:01 |
| fungi | well, delete several repository copies on gerrit and then rerun manage-projects | 19:01 |
| fungi | have to delete the gerrit, jeepyb and apache copies there | 19:01 |
| jeblair | keeping in mind that stackforge projects are not our priority, i don't think we should create new ones as long as the new project automation is not working. | 19:01 |
| clarkb | jeblair: wfm | 19:02 |
| fungi | can't argue with that logic | 19:02 |
| clarkb | fungi: there was a gerrit copy? | 19:02 |
| fungi | clarkb: there was, it was just empty | 19:02 |
| clarkb | fungi: project creation seemed to have completely failed, but it may have appeared that way to me because you removed the repo | 19:02 |
| fungi | clarkb: i think it got git-init'd and then nothing pushed into it | 19:02 |
| clarkb | fungi: no refs/meta/config? | 19:02 |
| clarkb | the first commit in refs/meta/config should always happen automagically by gerrit | 19:03 |
| fungi | it did have refs/meta/config but no heads | 19:03 |
| clarkb | gotcha, so project creation did work to that extent | 19:03 |
| clarkb | fungi: did the local jeepyb cache repo have any content in it? | 19:04 |
| *** mrodden has quit IRC | 19:04 | |
| fungi | clarkb: honestly i didn't look. i will copy off all three of the repos on gerrit next time i witness it | 19:04 |
| jeblair | reopened bug 1242569 | 19:05 |
| uvirtbot | Launchpad bug 1242569 in openstack-ci "manage-projects error on new project creation" [Critical,Triaged] https://launchpad.net/bugs/1242569 | 19:05 |
| *** nati_uen_ has joined #openstack-infra | 19:05 | |
| jeblair | mordred: ^ you may be interested in this; manage-projects is still broken | 19:05 |
| *** klrmn has left #openstack-infra | 19:05 | |
| *** prad has joined #openstack-infra | 19:06 | |
| jeblair | AaronGr: if you're looking for a project, that's a good one ^. just getting things set up to be able to test that will get you familiar with a lot of stuff. | 19:06 |
| clarkb | ++ | 19:07 |
| fungi | i'll see if entropy encounters any issues, since it should be getting imported right about now | 19:07 |
| jeblair | AaronGr: (you would need to set up a gerrit) | 19:07 |
| clarkb | fungi: if it does, can you move the various repos aside so that we can refer back to them while debugging? | 19:07 |
| fungi | oh, nevermind. entropy doesn't have an upstream, so it probably won't break | 19:07 |
| clarkb | fungi: will be interesting to see if ACLs apply though | 19:08 |
| *** nati_ueno has quit IRC | 19:08 | |
| clarkb | fungi: if they do I think we can rule out problems iwth group management | 19:08 |
| fungi | we *can* test with 61250 (openstack-infra/openstackid), but i want to rebase it not on top of the project.yaml realphabetize change (and abandon that one until we have a test for it) | 19:09 |
| fungi | i'll do that now | 19:09 |
| *** rcleere_ has joined #openstack-infra | 19:09 | |
| *** rcleere has quit IRC | 19:10 | |
| *** rcleere_ is now known as rcleere | 19:10 | |
| zaro | gerrit user summit: https://sites.google.com/site/gerritsummit2014 | 19:12 |
| *** rongze has quit IRC | 19:13 | |
| *** danger_fo_away is now known as dangers | 19:13 | |
| *** mrodden has joined #openstack-infra | 19:13 | |
| *** reed has quit IRC | 19:14 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add packstack to gerritbot https://review.openstack.org/60240 | 19:15 |
| clarkb | before any more conflicts happen I am approving that :) | 19:15 |
| fungi | wfm | 19:15 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Import the openstackid authentication project https://review.openstack.org/61250 | 19:15 |
| *** rongze has joined #openstack-infra | 19:15 | |
| clarkb | I think that covers things that hit merge conflicts, back to normal review | 19:16 |
| *** jorisroovers has joined #openstack-infra | 19:17 | |
| *** zhiyan has left #openstack-infra | 19:18 | |
| clarkb | jeblair: fungi: out of curiousity are requirements and hacking in your gerrit project watch list? I am debating removing them from my watchlist | 19:19 |
| *** zhiyan has joined #openstack-infra | 19:19 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add packstack to gerritbot https://review.openstack.org/60240 | 19:20 |
| fungi | clarkb: i don't think either requirements or hacking are in my watch list since there seemed to already be plenty of people reviewing them | 19:20 |
| *** crank_ has joined #openstack-infra | 19:20 | |
| *** dcramer_ has quit IRC | 19:21 | |
| clarkb | fungi: they also have a lot of noise >_> | 19:21 |
| fungi | yes. noise i do not need | 19:21 |
| *** crank_ has quit IRC | 19:21 | |
| *** rongze has quit IRC | 19:22 | |
| *** resker has quit IRC | 19:23 | |
| *** esker has joined #openstack-infra | 19:23 | |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Adds Mercurial SCM support https://review.openstack.org/61547 | 19:28 |
| *** CaptTofu has quit IRC | 19:29 | |
| *** CaptTofu has joined #openstack-infra | 19:30 | |
| *** nati_uen_ has quit IRC | 19:30 | |
| *** rcarrillocruz has joined #openstack-infra | 19:31 | |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Fix format in installation.rst https://review.openstack.org/61548 | 19:31 |
| clarkb | I went ahead and removed them from my watchlist, we'll see if this helps | 19:31 |
| *** yolanda has joined #openstack-infra | 19:31 | |
| *** rcarrillocruz1 has quit IRC | 19:32 | |
| *** dcramer_ has joined #openstack-infra | 19:32 | |
| *** openstackgerrit has quit IRC | 19:34 | |
| *** CaptTofu has quit IRC | 19:34 | |
| *** openstackgerrit has joined #openstack-infra | 19:34 | |
| *** rnirmal has quit IRC | 19:36 | |
| *** nati_ueno has joined #openstack-infra | 19:40 | |
| *** prad has quit IRC | 19:40 | |
| *** herndon has joined #openstack-infra | 19:41 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60961 | 19:41 |
| *** prad has joined #openstack-infra | 19:43 | |
| *** reed has joined #openstack-infra | 19:46 | |
| *** pblaho has joined #openstack-infra | 19:47 | |
| *** pblaho has quit IRC | 19:48 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Serve the statusbot alert file from eavesdrop https://review.openstack.org/60582 | 19:50 |
| jeblair | fungi, clarkb: i'm starting to think we should make status.o.o its own host so it can run all those things we want to run (irc bots, cron jobs, etc) without having to worry about artifact hosting. | 19:51 |
| clarkb | jeblair: ++ | 19:51 |
| openstackgerrit | A change was merged to openstack-infra/statusbot: Add an alert file publisher https://review.openstack.org/60568 | 19:52 |
| fungi | jeblair: that makes sense | 19:53 |
| *** _david_ has joined #openstack-infra | 19:53 | |
| *** thomasem has joined #openstack-infra | 19:58 | |
| *** AaronGr is now known as AaronGr_afk | 19:58 | |
| *** sdake_ has quit IRC | 19:58 | |
| *** sdake_ has joined #openstack-infra | 19:59 | |
| *** dcramer__ has joined #openstack-infra | 19:59 | |
| apevec | fungi, ttx - here's draft relnotes for 2013.2.1 https://wiki.openstack.org/wiki/ReleaseNotes/2013.2.1 - please check I didn't miss any CVEs | 20:02 |
| *** SergeyLukjanov has quit IRC | 20:02 | |
| *** openstackstatus has quit IRC | 20:02 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add status.openstack.org vm https://review.openstack.org/61552 | 20:02 |
| *** openstackstatus_ has joined #openstack-infra | 20:02 | |
| *** openstackstatus_ is now known as openstackstatus | 20:02 | |
| apevec | we should probably have script generate it :) | 20:02 |
| apevec | fungi, do you keep CVE metadata in some machine readable format? | 20:02 |
| *** openstackstatus has quit IRC | 20:03 | |
| *** openstackstatus_ has joined #openstack-infra | 20:03 | |
| *** openstackstatus_ is now known as openstackstatus | 20:03 | |
| *** dcramer_ has quit IRC | 20:03 | |
| *** openstackstatus has quit IRC | 20:04 | |
| *** openstackstatus_ has joined #openstack-infra | 20:04 | |
| *** openstackstatus_ is now known as openstackstatus | 20:04 | |
| *** rcarrillocruz has quit IRC | 20:05 | |
| *** balar has quit IRC | 20:05 | |
| *** openstackstatus has quit IRC | 20:05 | |
| *** openstackstatus_ has joined #openstack-infra | 20:05 | |
| *** openstackstatus_ is now known as openstackstatus | 20:05 | |
| *** openstackstatus has quit IRC | 20:06 | |
| *** openstackstatus_ has joined #openstack-infra | 20:06 | |
| *** openstackstatus_ is now known as openstackstatus | 20:06 | |
| *** herndon has quit IRC | 20:07 | |
| *** openstackstatus_ has joined #openstack-infra | 20:07 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60961 | 20:08 |
| *** openstackstatus has quit IRC | 20:08 | |
| *** openstackstatus_ is now known as openstackstatus | 20:08 | |
| *** openstackstatus has quit IRC | 20:08 | |
| *** apevec has quit IRC | 20:08 | |
| *** openstackstatus has joined #openstack-infra | 20:08 | |
| sdague | fungi: finally found proof that the grenade change helped - http://logs.openstack.org/65/46065/24/gate/gate-grenade-dsvm/a1b5b14/console.html#_2013-12-11_16_52_05_198 | 20:09 |
| sdague | that would have failed previously | 20:09 |
| sdague | puppet's network also got better about the same time we merged that, so I wanted to find an instance of that working | 20:09 |
| *** dcramer___ has joined #openstack-infra | 20:09 | |
| mtreinish | jeblair: were we hitting an issue which prompted?: https://review.openstack.org/#/c/61513/ | 20:10 |
| jeblair | mtreinish: sphinx pre-release versions broke local use of tox (not a gate issue) | 20:11 |
| *** nicedice has joined #openstack-infra | 20:11 | |
| jeblair | mtreinish: that change, however, should have been propogated months ago, but was dropped. about 1/3 of the projects already had it. | 20:11 |
| *** jroovers has joined #openstack-infra | 20:12 | |
| *** jorisroovers has quit IRC | 20:12 | |
| mtreinish | jeblair: yeah, I think I forgot to add it when I updated the tox.ini min-version | 20:12 |
| *** _david_ has quit IRC | 20:12 | |
| *** thedodd has joined #openstack-infra | 20:13 | |
| *** freyes has joined #openstack-infra | 20:13 | |
| *** dcramer__ has quit IRC | 20:13 | |
| *** jooools has joined #openstack-infra | 20:15 | |
| jeblair | #status ok | 20:16 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 20:18 | |
| *** dcramer__ has joined #openstack-infra | 20:20 | |
| *** vipul is now known as vipul-away | 20:21 | |
| *** vipul-away is now known as vipul | 20:21 | |
| *** dcramer___ has quit IRC | 20:23 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Switch devstack checkout to use git.o.o https://review.openstack.org/61556 | 20:28 |
| *** vkozhukalov has quit IRC | 20:29 | |
| *** dcramer___ has joined #openstack-infra | 20:29 | |
| *** mrodden has quit IRC | 20:30 | |
| *** gothicmindfood has quit IRC | 20:31 | |
| *** mrodden has joined #openstack-infra | 20:32 | |
| *** dcramer__ has quit IRC | 20:33 | |
| *** dcramer__ has joined #openstack-infra | 20:33 | |
| dhellmann | fungi: I'm still having some trouble with that stevedore repo you fixed a day or two ago, do you have a sec to help? | 20:35 |
| pleia2 | following up on a couple bug day things, do we want to give shrini the green light to set up an sss demo, or..? https://bugs.launchpad.net/openstack-ci/+bug/1203137 | 20:35 |
| uvirtbot | Launchpad bug 1203137 in openstack-ci "Run log analyzer for all IRC logs" [Low,Triaged] | 20:35 |
| fungi | dhellmann: sure--what's the symptom? | 20:35 |
| dhellmann | fungi: clone a fresh repo, create a branch, create a commit, git review, merge failure reported by jenkins: https://review.openstack.org/#/c/61559/ | 20:36 |
| dhellmann | fungi: the review contains a brand new file (.gitreview) | 20:36 |
| *** vipul is now known as vipul-away | 20:37 | |
| pleia2 | jeblair: any thoughts on this one? https://bugs.launchpad.net/openstack-ci/+bug/1221842 argument was made that "mailman isn't secure anyway" but is adding an ssl cert for it really more much trouble/expense than it's worth? | 20:37 |
| *** dcramer___ has quit IRC | 20:37 | |
| uvirtbot | Launchpad bug 1221842 in openstack-ci "Add ssl cert to lists.openstack.org" [Medium,Triaged] | 20:37 |
| fungi | dhellmann: i'll check zuul's clone... probably something's wrong with it from when we were working through fixing these before | 20:37 |
| dhellmann | fungi: that's what I figured, too | 20:37 |
| fungi | dhellmann: in good news, more recent project additions haven't been hitting that specific problem | 20:38 |
| fungi | so probably just pending cleanup i missed there | 20:38 |
| *** zhiyan has quit IRC | 20:38 | |
| *** dprince has quit IRC | 20:38 | |
| dhellmann | fungi: good, I have several more to submit once I have a new stevedore release done :-) | 20:38 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/jeepyb: Add new hook flags for gerrit 2.5 https://review.openstack.org/61560 | 20:38 |
| *** dcramer__ has quit IRC | 20:39 | |
| *** _david_ has joined #openstack-infra | 20:39 | |
| pabelanger | So, taking suggestions how to address the new flags for gerrit 2.5 hooks | 20:39 |
| jeblair | pleia2: i will reply to 1203137 | 20:40 |
| pleia2 | jeblair: thank you | 20:40 |
| *** sandywalsh_ has quit IRC | 20:42 | |
| jeblair | pleia2: as long as we realize that mailman sends out passwords in plain text. :) i'm okay adding it though. | 20:43 |
| jeblair | pleia2: assigned to me | 20:43 |
| *** oubiwan__ has joined #openstack-infra | 20:43 | |
| *** yolanda has quit IRC | 20:44 | |
| reed | fungi, re: https://review.openstack.org/#/c/61250/ is the code going to stay on github or can we move it to our git repo? | 20:45 |
| fungi | reed: in this case, "upstream" really means "import-from" | 20:45 |
| reed | fungi, so once that erb is executed we will have it on git.openstack.org? | 20:45 |
| *** lcestari has quit IRC | 20:46 | |
| fungi | reed: so approving that change is going to effectively move the repository to start being hosted from gerrit as the source of truth, and it will be mirrored to the openstack-infra orgs on git.o.o and github, yes | 20:46 |
| reed | sweet | 20:46 |
| pleia2 | jeblair: thanks, I know it's terribly insecure in general | 20:46 |
| reed | fungi, will we need to write jenkins jobs? in other words, now there are not gate checks, right? | 20:46 |
| *** krotscheck has quit IRC | 20:47 | |
| fungi | reed: right now there are no jenkins jobs for it (excepting a no-op which just basically returns true on any change). we can add as many as we want though | 20:47 |
| reed | sweet | 20:47 |
| *** krotscheck has joined #openstack-infra | 20:49 | |
| fungi | pleia2: yes, even though https prevents someone from sniffing your list password when you log in to moderate posts or update your settings, it in no way helps the attacker who is trying to sniff your traffic from also clicking the "send my password" button and then snagging it out of the ensuing e-mail message | 20:49 |
| *** AaronGr_afk is now known as AaronGr | 20:49 | |
| fungi | er, in no way helps thwart | 20:49 |
| pleia2 | fungi: right, but I'd argue that the passive sniffing over login is a much more common attack vector, with entry-level tools you can use on windows in starbucks | 20:50 |
| pleia2 | other vectors take a targeted attack | 20:50 |
| fungi | though i suppose opportunistic smtp-over-ssl/tls does raise the bar for some message, requiring the attacker to actually mitm the connection (still not too hard since smtps almost never performs cert validation) | 20:51 |
| *** praneshp has quit IRC | 20:51 | |
| pleia2 | yeah | 20:51 |
| fungi | also, how did you know i sniff passwords at starbucks? coffee-scented passwords smell great | 20:52 |
| pleia2 | everybody's doing it ;) | 20:52 |
| fungi | dhellmann: okay, looks like it's fixed now. i got a test to run to completion for it (the change should report here soon-ish once there are available slaves to run the remaining jobs) | 20:54 |
| *** sandywalsh_ has joined #openstack-infra | 20:54 | |
| dhellmann | fungi: thanks! | 20:54 |
| *** hashar has joined #openstack-infra | 20:57 | |
| *** paul-- has joined #openstack-infra | 20:59 | |
| *** ^demon has joined #openstack-infra | 21:01 | |
| *** ^demon has joined #openstack-infra | 21:01 | |
| *** denis_makogon_ has joined #openstack-infra | 21:02 | |
| *** ^demon has quit IRC | 21:02 | |
| *** hashar has quit IRC | 21:03 | |
| *** ^demon has joined #openstack-infra | 21:04 | |
| *** ^d has quit IRC | 21:05 | |
| *** yassine has quit IRC | 21:08 | |
| SpamapS | hm.. given http://paste.ubuntu.com/6558026/ and http://paste.ubuntu.com/6558028/ ... why is my pypi-mirror job downloading and installing httpretty over and over? | 21:09 |
| SpamapS | oh n/m | 21:10 |
| SpamapS | Using download cache from /var/cache/pypi-mirror/pypi/download/pip/openstack/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fh%2Fhttpretty%2Fhttpretty-0.7.0.tar.gz | 21:10 |
| *** denis_makogon_ is now known as denis_makogon | 21:11 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Don't re-exec in check-dg-tempest-dsvm-full https://review.openstack.org/61569 | 21:15 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't fix disks before rexecing https://review.openstack.org/60961 | 21:15 |
| jeblair | that change is now ready for review ^ | 21:15 |
| clarkb | jeblair: https://review.openstack.org/#/c/55829/9 can you expand on what you mean by your last comment? or possible review the latest patchset and indicate if it looks good | 21:17 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add status.openstack.org vm https://review.openstack.org/61552 | 21:17 |
| *** dstanek has quit IRC | 21:17 | |
| jeblair | clarkb: yes; if you read my ealier comments you'll see what i was on about (i just explained more in irc); i'll review now though. | 21:18 |
| *** dstanek has joined #openstack-infra | 21:18 | |
| jeblair | lgtm | 21:19 |
| jeblair | #alert test | 21:20 |
| jeblair | #status alert test | 21:20 |
| openstackstatus | NOTICE: test | 21:20 |
| *** ChanServ changes topic to "test" | 21:20 | |
| fungi | ...if this had been an actual emergency... | 21:20 |
| *** rfolco has quit IRC | 21:21 | |
| jeblair | #status ok | 21:21 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 21:21 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: wip: work in progress on bulk gerrit query https://review.openstack.org/61573 | 21:24 |
| clarkb | jeblair: line 548 https://review.openstack.org/#/c/60961/8/devstack-vm-gate-wrap.sh could that be run out of the workspace instead? | 21:25 |
| *** vipul-away is now known as vipul | 21:25 | |
| *** thomasem has quit IRC | 21:25 | |
| openstackgerrit | A change was merged to openstack-infra/config: Serve the statusbot alert file from eavesdrop https://review.openstack.org/60582 | 21:27 |
| *** dkliban has quit IRC | 21:27 | |
| *** alcabrera has quit IRC | 21:28 | |
| jeblair | clarkb: possibly, but i find it's easier to reason about just treating it the same as the rest of the repos at that point in the script. | 21:29 |
| *** hashar has joined #openstack-infra | 21:29 | |
| clarkb | jeblair: ok | 21:29 |
| *** paul-- has quit IRC | 21:29 | |
| jeblair | clarkb: (it should be the same because it should update the workspace copy iff this or a change ahead changes it, and otherwise, tip-of-master should be fine) <-- but that's the sort of thing you don't have to think too much about if we don't do that. :) | 21:30 |
| *** thomasem has joined #openstack-infra | 21:30 | |
| jeblair | clarkb: oh, also, i think that makes it easier to use in full-on manual dev mode. | 21:31 |
| *** thomasem has quit IRC | 21:31 | |
| clarkb | jeblair: should be the same meaning run out of workspace or run out of /opt? | 21:32 |
| *** dstanek has quit IRC | 21:32 | |
| mgagne | zaro: in its current state (git tag based versionning), how are maven project versions generated when a snapshot is built? (if any) What's the format used? | 21:32 |
| jeblair | clarkb: yes | 21:32 |
| *** openstackstatus has quit IRC | 21:32 | |
| clarkb | heh | 21:33 |
| *** openstackstatus has joined #openstack-infra | 21:33 | |
| jeblair | #status alert test | 21:34 |
| openstackstatus | NOTICE: test | 21:34 |
| *** ChanServ changes topic to "test" | 21:34 | |
| *** jcooley_ has quit IRC | 21:34 | |
| clarkb | mgagne: PROJECT_VER="$SCM_TAG.$COMMITS_SINCE_TAG.$SCM_SHA"; it doesn't do snapshots, it does actual increasing version numbers | 21:34 |
| jeblair | #status ok | 21:34 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 21:34 | |
| mgagne | clarkb: "snapshots" are therefore considered sub-releases of a major/minor release previously done. | 21:35 |
| jeblair | #status alert test | 21:35 |
| openstackstatus | NOTICE: test | 21:35 |
| *** ChanServ changes topic to "test" | 21:35 | |
| clarkb | mgagne: right, it does what our python clients do for example | 21:35 |
| jeblair | #status ok | 21:35 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 21:35 | |
| *** ^demon has quit IRC | 21:35 | |
| *** smarcet has quit IRC | 21:36 | |
| hdd | so this change was approved, but failed at the gate: https://review.openstack.org/#/c/60980/ | 21:36 |
| mgagne | clarkb: I had an idea to address the concern someone (I forgot the name) raised yesterday and I want to fully grasp the current concept first. | 21:36 |
| clarkb | mgagne: it was sharwell | 21:36 |
| mgagne | clarkb: thanks | 21:36 |
| hdd | and changes for python-glanceclient are all failing to verify because of it | 21:36 |
| hdd | fungi, clarkb: any ideas? | 21:37 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Add missing import directives https://review.openstack.org/61575 | 21:37 |
| jeblair | clarkb, fungi: ^ i'm going to self-approve that; it's, ah, tested in production. | 21:38 |
| clarkb | hdd: looks like review.openstack.org fell over then those tests tried to clone from it | 21:38 |
| clarkb | jeblair: :) go for it | 21:38 |
| fungi | lgtm | 21:38 |
| openstackgerrit | A change was merged to openstack-infra/statusbot: Add missing import directives https://review.openstack.org/61575 | 21:38 |
| mgagne | clarkb: how are those "snapshots" used? Anything having dependencies against it? Are they published to any Maven repo? | 21:38 |
| clarkb | mgagne: nothing should have dependencies on it and they shouldn't be published to Maven repos (at least we don't do that for other projects that use the process today) | 21:39 |
| *** jcooley_ has joined #openstack-infra | 21:39 | |
| mgagne | clarkb: what's the point of building snapshots if nothing uses it? Are they simply published to a scp site and manually consumed by anyone wishing to test it? | 21:40 |
| clarkb | mgagne: yup | 21:41 |
| clarkb | hdd: 0609 was 15 and a half hours ago? I wonder fi there was a network burp | 21:41 |
| mgagne | clarkb: and I guess they are published to a repo only when built by pre-release/release pipelines. | 21:41 |
| clarkb | mgagne: right | 21:41 |
| mgagne | clarkb: alright. thanks! | 21:41 |
| openstackgerrit | Steve Baker proposed a change to openstack-infra/devstack-gate: Register Fedora 20 for DEVSTACK_GATE_TEMPEST_HEAT_SLOW https://review.openstack.org/61576 | 21:41 |
| *** fbo is now known as fbo_away | 21:43 | |
| sharwell | mgagne: snapshots are used in 2 situations: 1) snapshots can be built on a local development machine and then referenced by another project to test the behavior of the dependent project with the new build, and 2) a snapshot can be deployed to a special sonatype repository to allow early testing of a build by other individuals (similar to #1, but publicly accessible) | 21:43 |
| clarkb | jeblair: so I understand. You are saying running it out of $BASE makes sense so that we don't have to worry about complicated d-g updates? imo doing it out of $WORKSPACE avoids that problem because we know $WORKSPACE is up to date | 21:43 |
| hdd | clarkb, how can I get that change reverified? No more "reverify no bug" (-: | 21:43 |
| mgagne | sharwell: yes, I correctly guessed how they were used by your team. We have the same (or very similar) workflow. thanks. | 21:44 |
| clarkb | hdd: submit a bug against openstack-ci about tests failing to fetch/clone git stuff due to possible network problems | 21:44 |
| mgagne | sharwell: I want to understand how each team uses the artifact/versioning to I can adjust my proposition/idea to fit the needs of both. | 21:44 |
| mgagne | sO* | 21:44 |
| clarkb | hdd: reverify against that. This is a bug, it just happens to probably be one very trick to correct | 21:44 |
| hdd | got it; thanks | 21:45 |
| *** ArxCruz has quit IRC | 21:46 | |
| *** jhesketh has joined #openstack-infra | 21:46 | |
| *** jooools has quit IRC | 21:46 | |
| jeblair | clarkb: if you want to do that then i think we should remove d-g from PROJECTS and move the re-exec setup_project call out one level. then i'd feel better about knowing it's always updated. | 21:48 |
| jeblair | clarkb: ugh | 21:48 |
| jeblair | clarkb: more later... | 21:48 |
| clarkb | I don't feel super strongly about it, just pointing out how I think the pieces fit together in my head | 21:49 |
| fungi | so... does anyone remember if we reverted anything https://review.openstack.org/56660 depends on? | 21:50 |
| clarkb | I believe the setup_projects call that happens before we call into $BASE will update the project in $BASE for us, but that happens as secondary step | 21:50 |
| clarkb | fungi: pretty sure we did | 21:50 |
| clarkb | fungi: we reverted the change that told select-mirror where to find wheels | 21:51 |
| devananda | clarkb: you guys manage global-requirements, ya? | 21:51 |
| clarkb | because wheels were not acting case insensitive. | 21:51 |
| clarkb | devananda: no the requirements core group manages them | 21:51 |
| devananda | ah | 21:51 |
| devananda | clarkb: i see. ci team is included, thus my presumption | 21:52 |
| mgagne | clarkb: if pbr handles the versioning using git tags, how is there a version in setup.cfg? What's the purpose? https://github.com/openstack/nova/blob/master/setup.cfg#L3 Is it the next major/minor version the project is working/looking toward to release? | 21:53 |
| clarkb | mgagne: ya, pbr supports two different types of versioning. Pre versioning where you are working up to a version and post versioning where you are tacking on point releases to an old version. If you specify the version in setup.cfg that tells pbr that you are using pre versioning and are working up to that version | 21:54 |
| mgagne | clarkb: Maven (project.version in pom.xml) is post-versioning right? | 21:55 |
| clarkb | mgagne: yes | 21:55 |
| mgagne | just checking | 21:55 |
| clarkb | np | 21:55 |
| *** freyes has quit IRC | 21:57 | |
| mgagne | clarkb: ... unless -SNAPSHOT is used | 21:57 |
| *** ^d has joined #openstack-infra | 21:57 | |
| clarkb | ? | 21:57 |
| clarkb | or would that be the distinction on pre vs post for maven? | 21:58 |
| mgagne | clarkb: if 1.0-SNAPSHOT is used (instead of 1.0) in project.version, this would mean the project is working toward a release of 1.0 | 21:58 |
| *** dstanek has joined #openstack-infra | 21:58 | |
| mgagne | clarkb: and I guess, even with git tag based releases, pbr has the same "limitation": Someone has to manually bump that version to indicate a new major version is about to be released someday. | 21:59 |
| clarkb | mgagne: right (careful pretty quick here I will be complaining about the date based versioning :) ) | 22:00 |
| fungi | preversioning also has a fairly major logistical drawback. you may decide at the last minute that the version number you thoughth you wanted isn't the one you actually wanted, so those pre-versions become sort of a lie | 22:00 |
| mgagne | fungi: true | 22:01 |
| lifeless | s/sort of// | 22:01 |
| clarkb | preversioning is something ttx and I disagree on for many reasons aiui | 22:01 |
| fungi | snapshot of "the version which never was" | 22:01 |
| *** praneshp has joined #openstack-infra | 22:01 | |
| lifeless | the problem is that version numbers as most folk know them don't have the concept of 'intended version' | 22:02 |
| fungi | well, i actually think our use of date-based versioning and preversioning for time-based releases isn't entirely outlandish. if your release model is openbsd-style "we release what's ready on time no matter what" then it seems acceptable | 22:02 |
| lifeless | more sophisticated grammars do, but they aren't widely adopted and people don't recognise them given an arbitrary string. | 22:02 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Run solum functional tests https://review.openstack.org/61582 | 22:02 |
| *** ^d has quit IRC | 22:02 | |
| clarkb | fungi: it is outlandish because it doesn't provide me with any non redudnant info | 22:03 |
| mgagne | if you ever backtrack on your intented version, any published artifacts will forever be more recent than any new builds of the new intended version as it will always be lower than the old one. | 22:03 |
| *** dstanek has quit IRC | 22:03 | |
| clarkb | the version control system knows when a version was cut, it shouldn't be part of the version | 22:03 |
| clarkb | instead versions should carry semantic info that conveys to users useful data | 22:04 |
| *** dkliban has joined #openstack-infra | 22:04 | |
| mgagne | clarkb: my intention wasn't to open a pandora'S box but I fear I did. | 22:04 |
| fungi | well, yes, in our case the date-based version numbers are non-semantic and arbitrary, and just happen to include a year for the major part | 22:05 |
| clarkb | mgagne: sorry, I get where you were taking this, I just can't help but bring this up when people want to do the other thing :) | 22:05 |
| mgagne | clarkb: np =) | 22:05 |
| fungi | clarkb: so... digging in the d-g git log, i don't think we reverted the select-mirror patch there after all? wasn't that the one which got proposed but check jobs showed it didn't help? | 22:06 |
| *** ^d has joined #openstack-infra | 22:06 | |
| mgagne | clarkb: I hope you guessed where I'm going with my questions | 22:06 |
| *** jroovers has quit IRC | 22:06 | |
| clarkb | mgagne: yes, I think you are going to suggest we support both pre and post versioning schemes based on a stated version somewhere and the contents of the git repo. I think we can do that, but for reasons above think we shouldn't | 22:07 |
| clarkb | but I may lose that argument for the same reason I lose when talking nova versioining :) | 22:07 |
| clarkb | fungi: we reverted a change to select-mirror itself | 22:07 |
| fungi | ohhhhh | 22:07 |
| fungi | i'll find it | 22:07 |
| clarkb | fungi: 343cff816b9ffd6d25d76481ba2a6a0eb7495352 in config | 22:08 |
| fungi | want to get these reproposed if needed and back on track. we didn't leave enough breadcrumbs in the outstanding commits | 22:08 |
| *** esker has quit IRC | 22:08 | |
| fungi | right, we killed Add support for per-distro wheel mirrors | 22:09 |
| clarkb | fungi: careful about straight up reproposing that change, I think $distro wasn't set in one place it was used and the sed on $distro is wrong compared to what we have in the mirror | 22:09 |
| clarkb | ++ to reproposing but for that particular one I think we need to fix a couple things | 22:09 |
| fungi | oh, i completely expect anything which needs reproposing to probably also need fixing. | 22:09 |
| pmathews | jeblair: ping? | 22:09 |
| fungi | just trying to make sure things like 56660 don't get approved and then somehow break everything | 22:10 |
| anteaya | clarkb: a patch hit two different bugs, can I use `recheck bug 1258848 and bug 1257641` ? | 22:10 |
| uvirtbot | Launchpad bug 1258848 in nova "vif doesn't exist Error message in tempest runs" [Critical,Triaged] https://launchpad.net/bugs/1258848 | 22:10 |
| uvirtbot | Launchpad bug 1257641 in nova "Quota exceeded for instances: Requested 1, but already used 10 of 10 instances" [Undecided,Confirmed] https://launchpad.net/bugs/1257641 | 22:10 |
| clarkb | anteaya: no, just pick one | 22:10 |
| anteaya | okay thanks | 22:10 |
| *** dkranz has quit IRC | 22:11 | |
| mgagne | clarkb: unless I misunderstand something, the version in setup.cfg is the equivalent of having 2014.1-SNAPSHOT in pom.xml | 22:11 |
| jeblair | pmathews: pong | 22:11 |
| jeblair | anteaya: sometimes i leave both in two comments | 22:11 |
| jeblair | anteaya: that way recheckwatch seets both. but recheckwatch may not live much longer so it probably doesn't matter. | 22:11 |
| clarkb | mgagne: almost, instead you remove -SNAPSHOT and the machinery sorts out what the actual version is | 22:12 |
| pmathews | jeblair: Can I get you to take a look at https://review.openstack.org/#/c/59875/3 | 22:12 |
| jeblair | pmathews: not my area, sorry. | 22:12 |
| anteaya | jeblair: okay, I just made a note of the other bug in a follow-up comment, not sure if it is the correct syntax for recheckwatch, I doubt it: https://review.openstack.org/#/c/61234/ | 22:13 |
| fungi | pmathews: you probably will fare better looking for devstack developers on #openstack-qa | 22:13 |
| *** elasticio has quit IRC | 22:15 | |
| *** oubiwa___ has joined #openstack-infra | 22:17 | |
| *** oubiwa___ is now known as oubiwann-lambda | 22:18 | |
| *** esker has joined #openstack-infra | 22:18 | |
| *** esker has quit IRC | 22:19 | |
| *** oubiwan__ has quit IRC | 22:20 | |
| openstackgerrit | A change was merged to openstack-infra/config: Switch some infra jobs to use bare precise nodes https://review.openstack.org/60344 | 22:21 |
| jeblair | ooh | 22:21 |
| clarkb | dun dun dun | 22:21 |
| mgagne | clarkb: alright. time based release makes this process easier I guess... not so much for libraries which can be released or bump the minor/major version at any time :-/ | 22:21 |
| mgagne | clarkb: my plan is falling apart :O | 22:22 |
| clarkb | :( | 22:22 |
| *** resker has joined #openstack-infra | 22:22 | |
| mgagne | clarkb: I could still propose it and see if someone can come up with a solution | 22:23 |
| fungi | okay, i'm going to disappear to grab some dinner, then back for more fun | 22:24 |
| *** oubiwann-lambda has quit IRC | 22:24 | |
| clarkb | mgagne: I am leaning towards just forgoeing automagic releases if the old version scheming is hat important | 22:25 |
| *** dims has quit IRC | 22:26 | |
| clarkb | do releases by hand since the rest of it needs hand holding | 22:26 |
| *** hashar has left #openstack-infra | 22:26 | |
| jeblair | releases by hand is not okay. | 22:26 |
| jeblair | i don't care about versioning, but publishing a release artifact has to be automatic. it can't depend on someone uploading something somewhere manually. | 22:26 |
| mgagne | jeblair, clarkb: the idea was to implement the same magic of pbr in Maven | 22:27 |
| clarkb | fungi: oh crud, I just approved a projects.yaml change. Should I wait on that to do your alphebetizing? | 22:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Set launchpad project mapping for all -api repos https://review.openstack.org/60597 | 22:27 |
| clarkb | fungi: I think I will forge ahead and get those through then rebase your change if necessary | 22:28 |
| *** resker has quit IRC | 22:28 | |
| *** herndon has joined #openstack-infra | 22:28 | |
| *** dstanek has joined #openstack-infra | 22:30 | |
| openstackgerrit | A change was merged to openstack-infra/config: Set launchpad mappings for rest openstack/* repos https://review.openstack.org/60598 | 22:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Set launchpad mappings for all cookbook-* repos https://review.openstack.org/60599 | 22:30 |
| mgagne | clarkb: BUT hey, they would have bumped that version manually anyway. | 22:30 |
| mgagne | clarkb: my plan still stand | 22:31 |
| *** michchap has joined #openstack-infra | 22:31 | |
| clarkb | mgagne: right I think we could have something that emulated pre and post versioning for maven projects | 22:31 |
| clarkb | that might be one way of tackling the problem if we are very attached to pre versioning | 22:31 |
| mgagne | clarkb: I'm writing a FAQ or some kind of user stories if you want that explains how this system would work. | 22:32 |
| openstackgerrit | A change was merged to openstack-infra/config: Set launchpad mappings for the rest projects https://review.openstack.org/60600 | 22:32 |
| jeblair | 0o644 is the new right way to do octal in python, right? | 22:33 |
| mgagne | clarkb: I however don't know anything about the inner working of Maven. I know Maven has plugin but don't know the extend of the possibilities and informations they can intercept/modify. | 22:33 |
| clarkb | jeblair: yes | 22:34 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Set world-readable permissions on alert file https://review.openstack.org/61588 | 22:34 |
| clarkb | mgagne: ya I am pretty light on maven internal knowledge as well. zaro seems to grok good portions of it | 22:34 |
| mgagne | clarkb: zaro is aware of the idea but he looks busy/away atm =) | 22:35 |
| *** mriedem has quit IRC | 22:37 | |
| *** ryanpetrello has quit IRC | 22:39 | |
| *** dims has joined #openstack-infra | 22:39 | |
| mgagne | clarkb: what happens if after a release has been tagged, someone forgets to bump the version? Which version is generated? Is it still against the same version or is the last digit increased? | 22:40 |
| zaro | clarkb: mgagne , so been away for lunch. need to catch up on scrollback | 22:40 |
| zaro | mgagne: version is auto bumped | 22:40 |
| mgagne | zaro: idea is pbr in maven. asking questions about pbr and current maven/pbr versioning system in openstack. | 22:41 |
| anteaya | I'm obviously doing something wrong, I did recheck bug <bug number> on https://review.openstack.org/#/c/61234/ and I don't see it in the check queue and zuul had 0 events | 22:41 |
| zaro | mgagne: ohh wait tag is manually set by person making tag. | 22:41 |
| mgagne | zaro: yes | 22:42 |
| anteaya | 10 events now, but it did have 0 when I last looked for 61234 in the queue | 22:42 |
| *** dangers is now known as danger_fo_away | 22:42 | |
| mgagne | zaro: you create that tag. A release is done. version in setup.cfg is not bumped. Someone merge a change. Which version is it? | 22:42 |
| zaro | mgagne: takling about this? https://github.com/ktoso/maven-git-commit-id-plugin | 22:42 |
| mgagne | zaro: no, pbr | 22:43 |
| clarkb | mgagne: still the old version | 22:43 |
| clarkb | mgagne: it would look like the next snapshot against the previous version | 22:43 |
| mgagne | clarkb: so you could still build snapshots of an already released version. | 22:43 |
| clarkb | I think | 22:43 |
| mgagne | clarkb: alright | 22:43 |
| anteaya | my bad, I just found it | 22:47 |
| anteaya | d'oh | 22:47 |
| *** adrian_otto has joined #openstack-infra | 22:48 | |
| *** mfer has quit IRC | 22:48 | |
| mgagne | zaro, clarkb: therefore, the only thing that needs to be fixed is: find a way to override $project.version with tag when releasing and generating a proper snapshot version based on commit since last tag. Nothing else needs to be done to address the workflow of the dev which remains unchanged. | 22:52 |
| mgagne | zaro, clarkb: this can already be done with mvn versions:set | 22:53 |
| *** CaptTofu has joined #openstack-infra | 22:53 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix serving alert json file on eavesdrop https://review.openstack.org/61593 | 22:53 |
| zaro | mgagne: i'm wondering if it can always be set to -SNAPSHOT? | 22:54 |
| mgagne | zaro: it should always be set to -SNAPSHOT | 22:54 |
| mgagne | zaro: X.Y.Z-SNAPSHOT where X.Y.Z is the next intended version. | 22:54 |
| *** paul-- has joined #openstack-infra | 22:55 | |
| *** dizquierdo has joined #openstack-infra | 22:56 | |
| zaro | mgagne: yeah, i would agree. | 22:56 |
| zaro | mgagne: it's set by default right? so just mvn package | 22:57 |
| zaro | mgagne: so are we back to maven plugin idea? | 22:58 |
| *** krotscheck has quit IRC | 22:58 | |
| mgagne | zaro: no plugin, just mvn versions:set | 22:58 |
| zaro | mgagne: is that a phase? | 23:00 |
| mgagne | zaro: it's a target | 23:00 |
| mgagne | zaro: afaik | 23:00 |
| mgagne | zaro: http://mojo.codehaus.org/versions-maven-plugin/set-mojo.html | 23:00 |
| *** paul-- has quit IRC | 23:01 | |
| clarkb | jeblair: zuul status page idea, when a job completes if you hover over the SUCCESS or FAILURE result work have it display the total time it took for the job to run | 23:04 |
| jeblair | clarkb: ++ | 23:04 |
| *** paul-- has joined #openstack-infra | 23:04 | |
| clarkb | btw I have approved a few simple changes with only one +2 (doc updates and gerritbot project watches) | 23:06 |
| zaro | mgagne: ahh, i see. maven set is doing the exact same thing as we do now. except we don't need to have a variable for the project version. NICE! | 23:06 |
| mgagne | zaro: yep | 23:06 |
| zaro | clarkb: jeblair . i think mgagne is on to something | 23:07 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Fix README file https://review.openstack.org/60579 | 23:07 |
| mgagne | zaro: updated: https://etherpad.openstack.org/p/java-release-process | 23:07 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Improve README.rst formatting https://review.openstack.org/60955 | 23:07 |
| zaro | clarkb: jeblair, mgagne : that's a simple fix that may make everyone happyy. | 23:07 |
| clarkb | cool | 23:08 |
| clarkb | trying to flush the queue of as many simple things as possible | 23:08 |
| zaro | mgagne: does that fix it for you and your peeps? | 23:08 |
| mgagne | zaro: ironically, we already have this system here, except the version isn't coming from a git tag but a script incrementing the last digit found in $project.version. That version is then bumped by this same script and committed in the repo so the release manager doesn't have to do it. This process is trigged by someone launching a jenkins job. | 23:10 |
| zaro | sharwell: your thoughts on mgagne proposal? | 23:11 |
| mgagne | zaro: well, it's a jenkins job (which can be parameterized), no a git tag. that's the point. | 23:11 |
| zaro | mgagne: i assume you still deploy with release plugin? | 23:13 |
| mgagne | zaro: I think so. I usually stay away from this system as they are used by projects I do not manage. | 23:13 |
| mgagne | zaro: the missing puzzle piece in the previous system was Versions Maven Plugin. | 23:14 |
| clarkb | jeblair: fungi: curious about what you think of https://review.openstack.org/#/c/61204/ | 23:16 |
| sharwell | mgagne: (sorry i'm a little behind, reading your messages now) if you remove -SNAPSHOT for the release commit, and then forget to make a new commit bumping the version and proceed with other work, you will have more than one commit in source control which produces what looks like a final release output. the commit immediately after removing -SNAPSHOT should be a new commit which bumps up the version number and adds -SNAPSHOT | 23:16 |
| mgagne | sharwell: please read the proposition, you never ever have a version without -SNAPSHOT | 23:16 |
| *** flaper87 is now known as flaper87|afk | 23:17 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix GitHub pull request builder plugin support https://review.openstack.org/60673 | 23:17 |
| fungi | clarkb: catching up now, but i abandoned the alphabetizing change. there's really no sane way to keep that in shape unless we have a check for it | 23:17 |
| clarkb | fungi: ok | 23:17 |
| *** bpokorny has quit IRC | 23:19 | |
| zaro | mgagne: i assume that's a bump version and commit back to git? | 23:19 |
| jeblair | mgagne: does your proposal support not having version info in the source code at all (in a post-versioning system)? | 23:20 |
| mgagne | zaro: it's like pbr. you git tag, release is made by jenkins. you commit the next intended version with -SNAPSHOT. rinse and repeat. | 23:20 |
| jeblair | mgagne: maybe you just set $project.version to something and never touch it? | 23:20 |
| sharwell | mgagne: in your proposal, what in the content of the <version> element in pom.xml for a *tagged* commit | 23:20 |
| sharwell | the way I read it, it would contain x.y.z-SNAPSHOT | 23:21 |
| mgagne | jeblair: I believe the sharwell's team have other projects depending on this artifact. They are using local maven repository to allow other project to download and build against this snapshot version. Without $project.version, they won't have any version to depend on. | 23:21 |
| sharwell | which would not be correct. if a user checks out a tagged commit and builds, it should produce x.y.z, not x.y.z-SNAPSHOT | 23:21 |
| *** herndon has quit IRC | 23:21 | |
| zaro | sharwell: it's whatever we want it to be. the ci system just passes it thru. | 23:21 |
| zaro | sharwell: i believe the version plugin gives us the power. | 23:22 |
| mgagne | sharwell: a tagged commit would still have X.Y.Z-SNAPSHOT. (unfortunately) | 23:22 |
| sharwell | mgagne: zaro: I have to run for now,. I'll review again this evening and can continue discussing then or tommorrow | 23:22 |
| *** ryanpetrello has joined #openstack-infra | 23:22 | |
| mgagne | sharwell: as Jenkins won't commit the change it made with mvn versions:set. | 23:22 |
| jeblair | mgagne: i'm not sure that's the case, but that doesn't concern me at the moment; i just want to know if a post-versioned project can operate purely on tags, never updating a commit. | 23:23 |
| *** paul-- has quit IRC | 23:23 | |
| zaro | mgagne: actually i was thinking it would be X.Y.Z-$git_sha | 23:23 |
| mgagne | jeblair: yes it can | 23:23 |
| mgagne | jeblair: with mvn versions:set, you set anything you want. | 23:23 |
| praneshp | can someone help me get +2 review powers on https://review.openstack.org/#/admin/projects/stackforge/entropy,access | 23:24 |
| mgagne | jeblair: and commit (by Jenkins) is not required. | 23:24 |
| *** thedodd has quit IRC | 23:24 | |
| zaro | mgagne: hmm, that doesn't seem right. | 23:24 |
| zaro | mgagne: doesn't jenkins need to commit to bump the version? | 23:24 |
| *** thedodd has joined #openstack-infra | 23:24 | |
| *** sergmelikyan has quit IRC | 23:24 | |
| mgagne | zaro: snapshots made by Jenkins can have the format you want. it can be the one already used ("$SCM_TAG.$COMMITS_SINCE_TAG.$SCM_SHA") | 23:25 |
| zaro | mgagne: maybe not jenkins but the version plugin? | 23:25 |
| mgagne | zaro: jenkins doesn't need to commit anything to be able to build an artifact | 23:25 |
| zaro | mgagne: how does the version get bumped if not thru a commit? | 23:25 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Description Setter Plugin support added https://review.openstack.org/60674 | 23:26 |
| zaro | mgagne: when dev clones they are expecting the bumped version right? | 23:26 |
| openstackgerrit | A change was merged to openstack-infra/config: Add launchpad entries for openstack-manuals https://review.openstack.org/60943 | 23:26 |
| mgagne | zaro: version isn't "bumped" per se, it is a passthrough. you git tag, you take that version, you run mvn versions:set to set the value, you build the thing and you are done. | 23:26 |
| mgagne | zaro: the proposed system has the same limitations as pbr. someone needs to bump the version AFTER a release. | 23:27 |
| *** dstanek has quit IRC | 23:27 | |
| zaro | mgagne: ohhh. i see. /me not real familiar with pbr. | 23:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Add puppet_openstack_builder to gerritbot config https://review.openstack.org/61010 | 23:28 |
| jeblair | mgagne: are you talking about pre- or post versions? | 23:28 |
| mgagne | zaro: clarkb explained that the version stays the same in setup.cfg until someone bumps it. | 23:28 |
| jeblair | i really think it's important to specify. | 23:28 |
| jeblair | mgagne: that's true for pre-versioned projects. not post-versioned. | 23:28 |
| openstackgerrit | A change was merged to openstack-infra/config: Improve storyboard gate jobs https://review.openstack.org/61119 | 23:28 |
| jeblair | mgagne: all of our java projects will be post-versioned. are yours pre-versioned? | 23:29 |
| mgagne | jeblair: pom.xml is pre-versioning IF $project.version is set to X.Y.X-SNAPSHOT. If it's X.Y.Z, it's post-versioning. | 23:29 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Re-enable patchset-created https://review.openstack.org/61605 | 23:29 |
| jeblair | mgagne: pbr does not require a version in setup.cfg at all for post versioning. | 23:29 |
| fungi | praneshp: i've added you as the initial member of the entropy-core group. you can add/remove other members at https://review.openstack.org/#/admin/groups/252,members | 23:30 |
| *** thedodd has quit IRC | 23:30 | |
| mgagne | jeblair: true, it is however present in all openstack projects I checked | 23:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Gerritbot: watch master on python-climateclient https://review.openstack.org/61220 | 23:31 |
| jeblair | mgagne: did you check the client projects? they are all post-versioned. the servers are all pre-versioned. | 23:31 |
| praneshp | thanks fungi | 23:31 |
| mgagne | jeblair: didn't check client projects | 23:31 |
| jeblair | mgagne: anyway -- bear with me here for a minute. do you need pre-versioning? | 23:31 |
| *** rcleere has quit IRC | 23:32 | |
| mgagne | jeblair: I don't need anything, I'm helping people finding a way to answer their needs =) | 23:32 |
| fungi | mgagne: compare http://git.openstack.org/cgit/openstack/nova/tree/setup.cfg to http://git.openstack.org/cgit/openstack/python-novaclient/tree/setup.cfg | 23:32 |
| jeblair | mgagne: can you tell me why we're talking about pre-versioning at all here then? | 23:32 |
| jeblair | i'm trying to make this conversation simpler. :) | 23:33 |
| mgagne | jeblair: ok, to clarify, 2014.1 is pre-versioning right? | 23:33 |
| *** ^demon has joined #openstack-infra | 23:33 | |
| jeblair | mgagne: yes | 23:33 |
| *** ^demon has quit IRC | 23:33 | |
| *** ^demon has joined #openstack-infra | 23:33 | |
| mgagne | jeblair: this is what I'm proposing for java projects. | 23:33 |
| jeblair | mgagne: for yours or ours? | 23:33 |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Gerrit-2.8: Add migration path document https://review.openstack.org/61606 | 23:33 |
| *** jgrimm has quit IRC | 23:33 | |
| fungi | i'm not sure why this is suggested for java projects, unless simply because "java people expect preversioning schemes" | 23:33 |
| mgagne | jeblair: forget my projects. I'm referring to java projects managed by openstack | 23:34 |
| jeblair | mgagne: all of our projects will be post-versioned. | 23:34 |
| mgagne | jeblair: what fungi said. | 23:34 |
| clarkb | fungi: want to address https://review.openstack.org/#/c/61290/ really quickly and I will approve that change? | 23:34 |
| fungi | clarkb: sure | 23:34 |
| mgagne | jeblair: ok, I didn't know about the desire to move to "post-versioning" | 23:34 |
| *** dims has quit IRC | 23:34 | |
| *** Adri2000 has quit IRC | 23:35 | |
| mgagne | jeblair: I assumed what was in master of server-side projects (nova, cinder, etc.) was the way to go (pre-versioning) | 23:35 |
| jeblair | mgagne: those are pre-versioned because they are part of a time-based release process | 23:35 |
| *** openstackgerrit has quit IRC | 23:35 | |
| *** ^d has quit IRC | 23:36 | |
| *** openstackgerrit has joined #openstack-infra | 23:36 | |
| jeblair | mgagne: but we have no java projects that do time-based releases, so they should all be post-versioned | 23:36 |
| mgagne | jeblair: ok, I better understand now. then we have a clash. | 23:36 |
| jeblair | mgagne: (openstack client libraries are not part of the time-based release, so they are post-versioned) | 23:36 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/jeepyb: Link git.o.o in bug updates https://review.openstack.org/61290 | 23:36 |
| fungi | clarkb: ^ thanks! | 23:36 |
| *** Adri2000 has joined #openstack-infra | 23:36 | |
| *** Adri2000 has quit IRC | 23:36 | |
| *** Adri2000 has joined #openstack-infra | 23:36 | |
| mgagne | jeblair: java projects are pre-versioned. | 23:36 |
| lifeless | I'm sorry, I can't help but read that as perversioned | 23:36 |
| jeblair | mgagne: not in zaro's system | 23:36 |
| mgagne | jeblair: zaro's system is none of them. there is no version at all. | 23:37 |
| fungi | lifeless: i'm RIGHT there with you | 23:37 |
| clarkb | fungi: ty | 23:37 |
| *** _david_ has quit IRC | 23:38 | |
| jeblair | mgagne: so i'm interested in hearing more about host post-versioning works in your system with mvn versions plugin | 23:38 |
| fungi | mgagne: well, there is a version in the release artifacts, but not in the files which are tracked by revision control | 23:38 |
| jeblair | mgagne: because it sounds like your idea is similar to zaros but would put a nicer version field in a locally built copy | 23:39 |
| mgagne | jeblair: I'll have to dig more on that one. One thing is sure, jenkins HAS to commit something on behalf of the release manager. (to bump version) | 23:39 |
| *** blamar has quit IRC | 23:39 | |
| jeblair | mgagne: that's something we definitely don't want to do. this system should never have a commit to change a version number. | 23:39 |
| mgagne | jeblair: we would be back to my first iteration of the idea. A new Maven plugin doing what pbr does. | 23:39 |
| fungi | having files under revision control track the state of the same revision control system gets into gödelian incompleteness territory | 23:40 |
| mgagne | jeblair: so devs can build SNAPSHOT with version based on git tags found in the repository | 23:40 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Link git.o.o in bug updates https://review.openstack.org/61290 | 23:40 |
| *** ryanpetrello has quit IRC | 23:41 | |
| jeblair | mgagne: i'm not sure why the version used for something locally built by a developer matters. i think it would be nice if it showed a fully-correct version like pbr, but if it is something like "DEV" that doesn't bother me too much, especially if there's a script someone can run to build a copy with the correct git tag based version number. | 23:41 |
| fungi | ("those who do not learn from mathematics are doomed to iterate infinitely") | 23:42 |
| mgagne | jeblair: that "script" should be a first class citizen in maven otherwise devs will have to change their workflow. | 23:42 |
| zaro | jeblair: i think it's because it gets integrated with maven's dependency management system. | 23:42 |
| zaro | jeblair: SNAPSHOT is very special. | 23:43 |
| mgagne | zaro, jeblair: yes. you have to fully grasp their workflow and how they use those snapshots, dependencies to understand the motivation being it. | 23:43 |
| jeblair | zaro: okay, so DEV-SNAPSHOT. or 0.0.0-SNAPSHOT. | 23:43 |
| *** ryanpetrello has joined #openstack-infra | 23:44 | |
| jeblair | nothing should be depending on an unreleased locally-built version. | 23:44 |
| mgagne | jeblair: that's how maven works. | 23:44 |
| *** dstanek has joined #openstack-infra | 23:44 | |
| fungi | from outside, it seems very much like someone codified cargo-cult behavior into a standardized workflow for these details to matter so much | 23:44 |
| zaro | jeblair: actually the developer is dependent on it. | 23:44 |
| zaro | jeblair: because anything built locally gets place into local maven repo and that gets reused by downstream projects. | 23:45 |
| jeblair | zaro: what downstream projects? | 23:45 |
| zaro | jeblair: any other project that has a dependency on foo-x.y-SNAPSHOT | 23:45 |
| fungi | "downstream" sounds like it's javaspeak for "reverse dependency" | 23:45 |
| fungi | ahh, good guess ;) | 23:46 |
| *** paul-- has joined #openstack-infra | 23:47 | |
| jeblair | mgagne, zaro: a maven plugin that behaves like pbr might be a good idea, but given that zaro's system achieves the goals we set out some time ago, and already works with some projects we have; i'd like to finish moving clouddocs-maven-plugin to it, and then possibly iterating from there | 23:48 |
| mgagne | zaro: not sure if it's a good comparison, but I think SNAPSHOT and local maven repo could be compared to python setup.py develop | 23:48 |
| zaro | ++ | 23:48 |
| fungi | for development to depend so heavily on locally-rebuilt dependencies smacks of a compile-everything-and-embed-rather-than-decouple development mentality | 23:49 |
| fungi | seems dangerous, but maybe that's normal in javaland | 23:50 |
| clarkb | jeblair: not to change the subject but https://review.openstack.org/#/c/61463/ feels a lot like working around the lack of public glance which I thought we were against doing | 23:50 |
| clarkb | jeblair: which is why dib hasn't become a thing for us | 23:50 |
| mgagne | fungi: sorry, that's how it works with maven | 23:50 |
| *** masayukig has joined #openstack-infra | 23:51 | |
| jeblair | clarkb: i don't think that's the case. the devstack-gate setup scripts have always been about creating our own images. also, mordred had a change to move nodepool to dib but he abandoned it. | 23:52 |
| *** vipul is now known as vipul-away | 23:52 | |
| jeblair | mgagne: so how about setting the version to 0.0.0-SNAPSHOT, and then having the downstream projects depend on that? | 23:53 |
| *** bodepd has left #openstack-infra | 23:54 | |
| clarkb | jeblair: right, but they aren't doing the takeover node thing and completely rewriting the underlying disk | 23:54 |
| clarkb | caching small changes to existing known working images seems a lot less fragile than dd'ing over a image writing arbitrary bits to it and snapshotting that. However, this is the only practical way of moving forward so maybe I just deal with that | 23:55 |
| *** adrian_otto has quit IRC | 23:55 | |
| *** jhesketh__ has joined #openstack-infra | 23:55 | |
| jeblair | clarkb: wait, i'm confused. | 23:55 |
| zaro | mgagne: it can be anything really, but i believe they just release a ver 12.1 or something like that. so it would be 12.1.0-SNAPSHOT | 23:55 |
| jeblair | clarkb: the change you pointed me at doesn't do anything with dd. | 23:55 |
| clarkb | jeblair: it allows for dd | 23:56 |
| *** openstackgerrit has quit IRC | 23:56 | |
| zaro | jeblair: oops, what i said to mgagne | 23:56 |
| *** blamar has joined #openstack-infra | 23:56 | |
| jeblair | clarkb: it allows you to upgrade a kernel... | 23:56 |
| *** openstackgerrit has joined #openstack-infra | 23:56 | |
| *** datsun180b has quit IRC | 23:56 | |
| jeblair | clarkb: it allows for a lot of things. why are we talking about dd? | 23:56 |
| mgagne | jeblair: this would require changes in pom.xml of the other projects. Those projects have range of supported versions in pom.xml (it's like requirements.txt) With maven, you can create profile (in your home directory) which can be used to allow download of SNAPSHOT instead of release. This allows a dev to test their projects against bleeding edge versions of a dependency they just locally built 15 minutes ago without having to release them and | 23:56 |
| clarkb | jeblair: I believe the intention is feed nodepool an install script, nodepool runs install script, that script dd's over image with Xen bits, node reboots does more stuff to configure Xen bits, snapshot | 23:56 |
| clarkb | jeblair: we can already update the kernel without that feature | 23:57 |
| jeblair | mgagne: you got cut off at "having to release them and" | 23:57 |
| clarkb | the existing nodepool scriptage does that for us | 23:57 |
| mgagne | and publish them to a public maven repo. | 23:57 |
| jeblair | mgagne: right, so once you have a downstream saying "i want to use something-SNAPSHOT", it doesn't really matter what the 'something' is, does it? | 23:57 |
| mgagne | jeblair: public maven repo being the equivalent of pypi in python | 23:57 |
| mgagne | jeblair: it does | 23:58 |
| jeblair | mgagne: why? the thing you are indicating is "i want to run the most recent locally built version" | 23:58 |
| mgagne | jeblair: Do you have versions in requirements.txt? It's the same with Maven. You just can't tell the dev to sed everything to 0.0.0. | 23:58 |
| zaro | jeblair: it depends on the version # | 23:58 |
| jeblair | mgagne: what would those different numbers be? how can you have more than one most recent locally built version? | 23:59 |
| jeblair | 00:01 < mgagne> jeblair: Do you have versions in requirements.txt? It's the same with Maven. You just can't tell the dev to sed everything to 0.0.0. | 23:59 |
| jeblair | 00:01 < zaro> jeblair: it depends on the version # | 23:59 |
| jeblair | 00:02 < jeblair> mgagne: what would those different numbers be? how can you have more than one most recent locally built version? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
