| clarkb | yup they hit it with a hammer recently | 00:00 |
|---|---|---|
| clarkb | JayF: see the purple graph at the bottom of the status page | 00:00 |
| dstufft | (IAD is the best DC because it's close to me you see) | 00:00 |
| *** harlowja has quit IRC | 00:00 | |
| jeblair | clarkb, fungi, angu5: i think we're at a point were we could stop using the mirror for enforcement, but that requires replacing it with a different mirror, eg, bandersnatch | 00:00 |
| clarkb | jeblair: yes I think we are close | 00:00 |
| clarkb | especially now that no external link deps exist | 00:00 |
| JayF | clarkb: aight, nice to know people are working on things. If we ever get any onmetal hardware to you guys I might wield the hammer :) | 00:00 |
| clarkb | and we can't add them because pip won't let us by default | 00:00 |
| clarkb | JayF: though looking at that graph we should be mostly reocvred. I am looking to see why that job hasn't run | 00:01 |
| clarkb | I have 5 bare-trusty nodes ready | 00:01 |
| jeblair | angu5: so that's not an immediate solution, but when someone gets around to puppeting bandersnatch installation, we're probably pretty close to what you need | 00:01 |
| clarkb | hrm | 00:01 |
| angu5 | thanks jeblair | 00:01 |
| angu5 | if that happens can you let us know in #solum? | 00:02 |
| *** lcheng has joined #openstack-infra | 00:02 | |
| clarkb | jeblair: ideas on why ironic-python-agent-buildimage-coreos hasn't run yet? | 00:02 |
| anteaya | angu5: you are better reading the infra-meeting logs | 00:03 |
| openstackgerrit | Angus Salkeld proposed a change to openstack/requirements: Remove solum from the projects list https://review.openstack.org/102389 | 00:03 |
| anteaya | since that is were progress will be discussed | 00:03 |
| clarkb | argle bargle | 00:04 |
| clarkb | more errors | 00:04 |
| clarkb | this change is fun | 00:04 |
| jeblair | clarkb: i take it you don't need my help :) | 00:04 |
| *** lcheng has quit IRC | 00:04 | |
| clarkb | jeblair: well maybe | 00:05 |
| clarkb | I will post an error | 00:05 |
| clarkb | jeblair: http://paste.openstack.org/show/84836/ | 00:05 |
| clarkb | but the code explicitly checks for hasattr(change, 'branch') | 00:06 |
| *** hogepodge has joined #openstack-infra | 00:06 | |
| *** e0ne has joined #openstack-infra | 00:07 | |
| jeblair | clarkb: i think you need a () | 00:07 |
| clarkb | oh crap | 00:08 |
| clarkb | patch incoming | 00:08 |
| jeblair | clarkb: i think it's doing "if (True and foo or bar)" and that becomes "if ((True and foo) or bar)" | 00:08 |
| jeblair | except pretend i said 'False' in those examples :) | 00:09 |
| *** prad_ has quit IRC | 00:09 | |
| jeblair | so doing explicit "if (False and (foo or bar))" should be what we want | 00:09 |
| *** pfallenop has joined #openstack-infra | 00:09 | |
| *** e0ne has quit IRC | 00:10 | |
| jeblair | i blame pep8 for making this hard to read :) | 00:10 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Properly group check for branch in zuul functions https://review.openstack.org/102390 | 00:10 |
| clarkb | jeblair: yup ^ | 00:10 |
| *** angu5 is now known as asalkeld | 00:12 | |
| *** UtahDave has quit IRC | 00:12 | |
| JayF | What needs to be done to have that job try to requeue? | 00:12 |
| JayF | once that gets in and applied | 00:12 |
| dstufft | clarkb: oh man did openstack finally get to the point where it doesn't need externals? | 00:12 |
| clarkb | JayF: I think it should happen automagically | 00:12 |
| JayF | I like automagic | 00:12 |
| fungi | dstufft: about a month ago | 00:13 |
| dstufft | awesome :3 | 00:13 |
| clarkb | JayF: I think zuul is currently looping going this don't work lets try again | 00:13 |
| clarkb | JayF: if we make it work then it should just go through | 00:13 |
| fungi | dstufft: after that last push, most (all?) of the stragglers moved to pypi | 00:13 |
| *** hogepodge has quit IRC | 00:13 | |
| dstufft | it's fun wielding the pip and/or PyPI hammer to force change | 00:13 |
| clarkb | dstufft: fungi yup and now our run mirror script won't allow any new ones in | 00:13 |
| fungi | dstufft: i think the "not installed by default" in latest pip had a lot to do with it ;) | 00:13 |
| clarkb | because the overrides we had for pip have been removed | 00:14 |
| clarkb | ya that | 00:14 |
| clarkb | so thank you dstufft for using the big hammer | 00:14 |
| fungi | fist of DOOM | 00:14 |
| clarkb | JayF: yup according to my tail -f this is happening over and over, once we teach it how to work it should just go through like magic | 00:14 |
| JayF | Very nice. | 00:15 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add subunit2sql to stackforge https://review.openstack.org/102392 | 00:15 |
| *** annegent_ has joined #openstack-infra | 00:15 | |
| dstufft | clarkb: ;) Now just to finish that PEP to finally abolish the last remnants of external hosting | 00:16 |
| *** bknudson has joined #openstack-infra | 00:17 | |
| *** marcoemorais has quit IRC | 00:18 | |
| *** marcoemorais has joined #openstack-infra | 00:18 | |
| *** ociuhandu has quit IRC | 00:18 | |
| fungi | looks like sqlalchemy 0.9.6 was uploaded yesterday, and i've seen no new screaming, so i guess it fixed where 0.9.5 bit us | 00:19 |
| *** marcoemorais has quit IRC | 00:19 | |
| *** marcoemorais has joined #openstack-infra | 00:19 | |
| anteaya | oh yay | 00:20 |
| *** annegent_ has quit IRC | 00:20 | |
| fungi | yep, brown bag. the changelog references just one fix added in the new release ;) | 00:21 |
| clarkb | yup | 00:21 |
| *** jhesketh has quit IRC | 00:22 | |
| *** Ryan_Lane has joined #openstack-infra | 00:22 | |
| fungi | oh poo, angu5 left. i finally spotted why https://review.openstack.org/100538 isn't getting tested. i'll just add a note in the review | 00:23 |
| asalkeld | fungi: just changed my nick | 00:24 |
| fungi | asalkeld: oh, there you are! | 00:24 |
| asalkeld | (fighting with irc client) | 00:24 |
| fungi | asalkeld: missed adding the project to zuul's layout.yaml... http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml | 00:24 |
| *** bknudson has left #openstack-infra | 00:25 | |
| *** praneshp has quit IRC | 00:25 | |
| asalkeld | fungi: julienvey has a review up for that | 00:25 |
| *** rlandy has quit IRC | 00:25 | |
| asalkeld | fungi: https://review.openstack.org/#/c/102246/ | 00:25 |
| openstackgerrit | A change was merged to openstack-infra/config: Properly group check for branch in zuul functions https://review.openstack.org/102390 | 00:25 |
| fungi | asalkeld: cool. looks like it got missed by the author and reviewers in https://review.openstack.org/98237 | 00:26 |
| asalkeld | yeah, we chatted about in our meeting this morning | 00:26 |
| fungi | asalkeld: awesome. one more mystery solved | 00:27 |
| asalkeld | less black magic | 00:27 |
| mtreinish | clarkb: do you not have a python34 definition in the python-jobs group yet? :) | 00:30 |
| *** Sukhdev has joined #openstack-infra | 00:31 | |
| clarkb | nope | 00:32 |
| clarkb | thats later | 00:32 |
| *** bknudson has joined #openstack-infra | 00:34 | |
| *** jaypipes has quit IRC | 00:35 | |
| *** matsuhashi has joined #openstack-infra | 00:35 | |
| *** Ryan_Lane has quit IRC | 00:36 | |
| mtreinish | clarkb: fair enough I can wait on adding subunit2sql to stackforge | 00:36 |
| *** Ryan_Lane has joined #openstack-infra | 00:37 | |
| *** Sukhdev has quit IRC | 00:38 | |
| *** jhesketh has joined #openstack-infra | 00:39 | |
| openstackgerrit | Jay Faulkner proposed a change to openstack-infra/config: Add -y to apt-get for IPA builder https://review.openstack.org/102400 | 00:44 |
| *** ramashri has quit IRC | 00:44 | |
| JayF | jroll: clarkb: https://review.openstack.org/102400 | 00:44 |
| jroll | hi | 00:44 |
| JayF | clarkb: it ran, but our stuff failed | 00:44 |
| JayF | My conference is over, and I'm in need of dinner. I'm going now. Thanks for your help in getting the job running. | 00:44 |
| clarkb | mtreinish: you need py34 for that? | 00:45 |
| mtreinish | I've been writing the unit tests with py34 | 00:46 |
| mtreinish | I guess I can just add that later and only run py27 in the gate | 00:46 |
| clarkb | mtreinish: well it would just work if you added the test | 00:47 |
| clarkb | mtreinish: we don't have to add it everywhere for you to use it | 00:47 |
| mtreinish | actually now that I think about I've been hand testing it with py27 and unit testing with py34 | 00:48 |
| clarkb | JayF: jroll we might want to consider moving most of that job into a script in IPA then have jenkins call that | 00:48 |
| mtreinish | so I probably should make sure it works with py34 before I start requiring that it passes the tests | 00:48 |
| clarkb | JayF: jroll then you don't have to wait on us to fix it for you | 00:48 |
| *** dstanek is now known as dstanek_zzz | 00:48 | |
| mtreinish | so just py27 is probably for the best up front | 00:48 |
| clarkb | oh I never sent mail about this to the list | 00:49 |
| * clarkb drafts a thing | 00:49 | |
| jroll | clarkb: +9000 | 00:49 |
| zaro | fungi: what's up? | 00:54 |
| *** rloo_gone has quit IRC | 00:55 | |
| *** rloo_gone has joined #openstack-infra | 00:55 | |
| clarkb | fungi: jeblair jhesketh morganfainberg_L SergeyLukjanov https://etherpad.openstack.org/p/move-to-trusty-mail | 00:56 |
| jogo | jhesketh:ping | 00:56 |
| clarkb | everyone else is welcome to edit that too | 00:56 |
| *** rloo_gone has quit IRC | 00:56 | |
| *** morganfainberg is now known as morganfainberg_Z | 00:57 | |
| *** rloo_gone has joined #openstack-infra | 00:57 | |
| *** maxbit_ has quit IRC | 00:58 | |
| *** sarob has quit IRC | 00:58 | |
| *** sarob has joined #openstack-infra | 00:59 | |
| *** rloo_gone has quit IRC | 00:59 | |
| *** rloo_gone has joined #openstack-infra | 01:00 | |
| anteaya | clarkb: I have a question on line 8 of the etherpad | 01:01 |
| anteaya | jees, we're all red | 01:01 |
| *** devkulkarni has joined #openstack-infra | 01:01 | |
| *** devkulkarni has quit IRC | 01:01 | |
| *** achanda has quit IRC | 01:01 | |
| anteaya | clarkb: yes that reads better now | 01:02 |
| anteaya | clarkb: did you want to mention python3 at all in the email? | 01:02 |
| *** rloo_gone has quit IRC | 01:02 | |
| clarkb | no python3 is somewhat independent for now | 01:03 |
| clarkb | unless we want to make a decision on py33 testing nowish? | 01:03 |
| *** rloo_gone has joined #openstack-infra | 01:03 | |
| clarkb | basically we can't py33 test on trusty sanely. So we can keep py33 on precise and do py34 on trusty or just make everything py34 | 01:03 |
| anteaya | I'm fine leaving it out | 01:03 |
| openstackgerrit | Angus Salkeld proposed a change to openstack/requirements: Remove solum from the projects list https://review.openstack.org/102389 | 01:03 |
| anteaya | just recognize it probably will be an early question | 01:03 |
| clarkb | I lean toward py34 simply because py33's install base seems small | 01:03 |
| anteaya | so having a preemptive response can't hurt | 01:03 |
| *** sarob has quit IRC | 01:03 | |
| anteaya | sounds like a ml question to me | 01:04 |
| anteaya | but the deciding factor is that we can't test py33 on trusty sanely | 01:04 |
| *** dstanek_zzz is now known as dstanek | 01:04 | |
| anteaya | you might mention that is an obstacle and that a larger discussion needs to happen? | 01:04 |
| clarkb | anteaya: well for that reason I don't think the ml needs to be involved | 01:05 |
| anteaya | or just leave it out and wait for someone to ask | 01:05 |
| clarkb | its an infra thing | 01:05 |
| anteaya | true | 01:05 |
| anteaya | how about py3 testing struture is a discussion infra will have soon and will follow up with another email | 01:05 |
| anteaya | or is that digging a hole? | 01:06 |
| clarkb | thats digging a hole | 01:06 |
| anteaya | kk | 01:06 |
| clarkb | opens it to discussion :) | 01:06 |
| anteaya | forget what I said then | 01:06 |
| anteaya | okay email looks fine | 01:07 |
| * anteaya predicts a py3 question in response to the post though | 01:07 | |
| *** e0ne has joined #openstack-infra | 01:07 | |
| jhesketh | jogo: pong | 01:08 |
| jhesketh | clarkb: email lgtm | 01:09 |
| *** rkukura_ has joined #openstack-infra | 01:10 | |
| anteaya | you're peach | 01:10 |
| anteaya | someone was mint green | 01:10 |
| *** rkukura has quit IRC | 01:10 | |
| *** rkukura_ is now known as rkukura | 01:10 | |
| *** e0ne has quit IRC | 01:12 | |
| jogo | jhesketh: so about the idea of not running a auto recheck on a negative non-jenkins comment | 01:12 |
| jogo | I talked to jeblair in here the other day and he had no objections to it | 01:12 |
| jogo | and considering how many -1s I gave out today, I think it would be very helpful | 01:13 |
| clarkb | didn't that change merge? | 01:13 |
| clarkb | does fedora20 do py33 or 34? | 01:13 |
| *** yaguang has joined #openstack-infra | 01:14 | |
| fungi | clarkb: draft e-mail lgtm. no edits/suggestions | 01:15 |
| clarkb | thanks | 01:15 |
| fungi | clarkb: i think the consensus in our summit discussions was that we would drop python 3.3 testing and move to 3.4, since we haven't officially supported 3.3 yet anyway | 01:16 |
| clarkb | looks like f20 may be 3.3 | 01:16 |
| *** annegent_ has joined #openstack-infra | 01:16 | |
| clarkb | ya I seem to recall that was the case | 01:16 |
| clarkb | now that you mention it | 01:16 |
| clarkb | and sles was the only one that bothered? | 01:17 |
| fungi | so no deprecation period/branch pins needed for it | 01:17 |
| clarkb | this is great because it simplifies the number of images we need | 01:17 |
| clarkb | except for pypy | 01:17 |
| clarkb | but I bet we can pypy on trusty too | 01:17 |
| clarkb | Alex_Gaynor: ^ can you comment on that? | 01:17 |
| fungi | pypy should be fine on trusty, based on what he said earlier | 01:17 |
| clarkb | awesome | 01:18 |
| fungi | so that had been part of our loose migration plan | 01:18 |
| mtreinish | clarkb: arch is 3.4 so I vote for that, just so I can run things locally easily :) | 01:18 |
| fungi | and then the condensing of dsvm/bare node types will allow us to reduce further still | 01:18 |
| clarkb | mtreinish: you run arch locally? | 01:18 |
| clarkb | mtreinish: I gave up long before the systemd switch | 01:18 |
| clarkb | fungi: yup | 01:18 |
| *** praneshp has joined #openstack-infra | 01:18 | |
| mtreinish | I've got 5 machines that all run arch | 01:18 |
| clarkb | and with centos7/rhel7 being a thing now we can definitely deprecate python26 in K as we said at the summit | 01:19 |
| mtreinish | I have to do reinstalls about once a year because I forget to update and missed the window of breaking upgrade | 01:19 |
| mtreinish | but I deal with it... | 01:19 |
| clarkb | this is so exciting | 01:19 |
| clarkb | we probably can't drop 26 for juno though | 01:19 |
| * clarkb digs up russellb's tweet | 01:20 | |
| clarkb | 1https://twitter.com/russellbryant/status/466241078472228864 | 01:20 |
| clarkb | ya so in october we can put centos6 and python26 out to pasture | 01:20 |
| *** annegent_ has quit IRC | 01:21 | |
| clarkb | 1https is my new protocol name for https1.x | 01:21 |
| clarkb | >_> | 01:21 |
| anteaya | ha ha ha | 01:21 |
| anteaya | it was clickable | 01:21 |
| anteaya | just don't toss the 1 at the end | 01:21 |
| clarkb | that brings up the question of centos7 | 01:22 |
| clarkb | does it possibly make more sense to use that instead of say f20 for tripleo and tempest testing? | 01:22 |
| *** wenlock has joined #openstack-infra | 01:22 | |
| clarkb | clearly we can't do that today because no release | 01:22 |
| clarkb | But it may be viable real soon now | 01:22 |
| anteaya | lifeless is sure to have an opinion | 01:23 |
| *** reed has quit IRC | 01:23 | |
| jhesketh | jogo: okay, my memory is a bit hazy sorry.. you want to trigger rechecks for bots that leave -1? | 01:23 |
| jogo | no | 01:24 |
| jogo | so currently retrigger 'jenkins' on a comment and if the previous jenkins vote is 72 hours old | 01:24 |
| *** Ryan_Lane has quit IRC | 01:24 | |
| *** rloo_gone has quit IRC | 01:24 | |
| jogo | I want to change that to retrigger on only non-negative comments and if the previous jenkisn vote is 72 hours old | 01:25 |
| *** rloo_gone has joined #openstack-infra | 01:25 | |
| *** sweston_ has quit IRC | 01:25 | |
| *** Ryan_Lane1 has joined #openstack-infra | 01:26 | |
| clarkb | anteaya: I expext the red hatters to chime in | 01:26 |
| clarkb | I know they use fedora as a dev platform and expect everyone else to | 01:26 |
| anteaya | clarkb: I don't believe you will be disappointed | 01:26 |
| anteaya | yes | 01:26 |
| clarkb | but imo fedora is pretty bad for what we do however much better than the current ubuntu state of affairs | 01:26 |
| anteaya | I will let them wave that flag | 01:26 |
| ianw | clarkb: that's the plan :) | 01:26 |
| clarkb | ianw: centos7 is the plan? that makes me so happy | 01:26 |
| anteaya | ianw: there you are | 01:27 |
| anteaya | that was easy | 01:27 |
| jogo | jhesketh: modules/openstack_project/files/zuul/layout.yaml as defined here | 01:27 |
| *** jyuso has joined #openstack-infra | 01:27 | |
| clarkb | though apparently you can get away with not using yum on fedora now? | 01:27 |
| clarkb | that might make it worthwhile :) | 01:27 |
| anteaya | really | 01:27 |
| anteaya | how do you install packages then? | 01:27 |
| ianw | clarkb: yeah, i mean there's two parts ... a lot of devs use devstack on fedora and having it broken wastes a lot of time. that's why i've been focusing on bringing up f20 as an immediate issue | 01:28 |
| clarkb | there is a different package manager | 01:28 |
| anteaya | what is it called? | 01:28 |
| clarkb | I forget | 01:28 |
| anteaya | handy | 01:28 |
| jhesketh | jogo: oh that's right | 01:28 |
| jhesketh | yeah okay | 01:28 |
| jogo | so do I file a bug or just by you a beer in germany ;) | 01:28 |
| ianw | clarkb: but, we're going to want to build centos images with dib, right? | 01:28 |
| clarkb | ianw: so to me running devsdtack on fedora is weird, but I wouldn't run fedora anywhere | 01:28 |
| clarkb | ianw: if mordred gets that sorted in a reasonable amount of time yes | 01:28 |
| *** harlowja_ has quit IRC | 01:28 | |
| clarkb | ianw: but I raelly don't want to have dib trouble stall forward progress anymore | 01:29 |
| *** melwitt has quit IRC | 01:29 | |
| *** oomichi has quit IRC | 01:29 | |
| clarkb | ianw: the last week or so has shown me that there are enough problems without adding dib that we probably should've been doing this stuff weeks ago | 01:30 |
| *** marcoemorais has quit IRC | 01:30 | |
| clarkb | but I am very excited to see what mordred whips up for "thick" slaves and dib | 01:30 |
| anteaya | searching but just getting yum/rpm | 01:30 |
| ianw | clarkb: so if that's not ready, do you think using some sort of partner images from rackspace/hp cloud will be the way for centos? | 01:30 |
| clarkb | ianw: for hpcloud probably. rax has been good about giving us images straight up | 01:31 |
| ianw | i need to bootstrap myself on the dib work, as i'm pretty clueless (so forgive any stupid questions :) | 01:31 |
| *** rloo_gone has quit IRC | 01:31 | |
| clarkb | ianw: ^ is the trusty situation | 01:31 |
| jyuso | anteaya: hi. | 01:31 |
| anteaya | jyuso: hello | 01:31 |
| jyuso | anteaya: sorry,my id is changed.I asked you about upstream tempest cases,yesterday. | 01:31 |
| *** rloo_gone has joined #openstack-infra | 01:31 | |
| clarkb | ianw: the dib stuff is actually raelly cool. It just needs a lot more massaging than I expected | 01:32 |
| anteaya | xy<something> | 01:32 |
| *** praneshp_ has joined #openstack-infra | 01:32 | |
| anteaya | jyuso: with intel right? | 01:32 |
| *** harlowja has joined #openstack-infra | 01:32 | |
| ianw | clarkb: ok, that might make things a lot easier. redhatci is trying it's best to keep devstack on rhel7 stable so we're ready to go when things are released | 01:32 |
| clarkb | ianw: one of the really great things about it is your image building host can cache all the things. So you your builds go really quickly | 01:32 |
| jyuso | anteaya: yes,correct | 01:32 |
| clarkb | ianw: that and anyone can build their own identical images | 01:32 |
| jyuso | anteaya: Coud you give me any advice?thx:) | 01:32 |
| anteaya | jyuso: I made this wikipage for you: https://wiki.openstack.org/wiki/ThirdPartySystems/Intel-PCI-CI | 01:32 |
| *** camunoz has quit IRC | 01:32 | |
| *** rloo_gone has quit IRC | 01:32 | |
| *** oomichi has joined #openstack-infra | 01:33 | |
| anteaya | jyuso: can you fill it out and feel free to put material below the box | 01:33 |
| ianw | clarkb: while we're talking about it, where's the best reviews to start looking at? | 01:33 |
| *** rloo_gone has joined #openstack-infra | 01:33 | |
| anteaya | jyuso: I don't know enough about your system or what you are testing to really understand your question | 01:33 |
| anteaya | jyuso: and you had asked for a wikipage | 01:33 |
| *** amcrn_ has quit IRC | 01:33 | |
| *** dstanek is now known as dstanek_zzz | 01:34 | |
| clarkb | ianw: https://review.openstack.org/#/c/88479/27 | 01:34 |
| clarkb | ianw: I would clone that on a throwaway host as dib wants roots and stuff | 01:34 |
| clarkb | ianw: but if you run the build-image script in that change it should make you a devstack-precise image that works | 01:34 |
| clarkb | ianw: you can tweak the values to work on getting a centos7 image for example | 01:35 |
| *** praneshp has quit IRC | 01:35 | |
| *** praneshp_ is now known as praneshp | 01:35 | |
| clarkb | I don't expect centos7 to work out of the gate just due to different versions of puppet and stuff | 01:35 |
| clarkb | but have to start somewhere | 01:35 |
| ianw | clarkb: ok, so it would be helpful if i was say bringing up a f20 image (just cause that's out there ATM)? | 01:35 |
| clarkb | ya that actually should work | 01:35 |
| clarkb | and you can test it and make sure it does | 01:35 |
| mordred | clarkb: I'm very close. most of my current problems are puppet derp | 01:35 |
| clarkb | mordred: great news | 01:36 |
| anteaya | mordred: hi | 01:36 |
| mordred | also. I hate all things | 01:36 |
| anteaya | I could tell | 01:36 |
| mordred | anteaya: :) | 01:36 |
| mattoliverau | anteaya, clark, fungi, jeblair: FYI, the Wednesday of the mid-cycle (16th July) is my birthday, so I hope you all and the other infra peeps at the mid-cycle would join me for drinks or dinner that night, although I hope we all catch up for drinks and dinner every night, but that night a have a license to be extra sarcastic! :P | 01:36 |
| anteaya | mattoliverau: awesome | 01:36 |
| anteaya | yay | 01:36 |
| ianw | clarkb: ok, i'll take that as an action item to do, and i'll report back maybe to openstack-infra list? is it discussed anywhere else? | 01:36 |
| anteaya | we can be extra silly for your birthday | 01:37 |
| clarkb | mattoliverau: yes drinks ! | 01:37 |
| clarkb | ianw: its mostly been discussed on irc and in those changes | 01:37 |
| anteaya | and send pictures back for your wife | 01:37 |
| clarkb | ianw: but usually because it is nice to have low latency "why does dib not work in this case?" discussions | 01:37 |
| mattoliverau | anteaya: thanks I'd like that ;) | 01:37 |
| anteaya | mattoliverau: count on it | 01:37 |
| mattoliverau | lol | 01:37 |
| anteaya | another cancer, we be everywhere | 01:37 |
| jhesketh | jogo: heh, German beer sounds good to me! | 01:38 |
| ianw | clarkb: ok, got it, thanks. fair bit of bootstrapping for me to do but i'll get back to you guys :) | 01:39 |
| *** rloo_gone has quit IRC | 01:39 | |
| clarkb | ya it was for me too | 01:39 |
| clarkb | I basically blocked off a day and learned all about dib | 01:40 |
| *** nati_ueno has quit IRC | 01:40 | |
| clarkb | mordred: is https://review.openstack.org/#/c/90565/9/modules/openstack_project/files/nodepool/elements/slave-db/post-install.d/99-install-databases mostly working? | 01:40 |
| jyuso | anteaya: hi,this options "what tools are you using",did this means jenkins and "gerrit trigger"?Do you know did other 3rd part testing have such wiki pages?Maybe I could have a look as reference:) | 01:41 |
| clarkb | mordred: it doesn't seem to put the DBs on sockets only but that can come later | 01:41 |
| ianw | clarkb: it sounds more fun than a day, say, debugging vm connectivity issues with neutron :) | 01:41 |
| clarkb | ianw: debugging why tmpfs didn't work was fun | 01:41 |
| clarkb | I am pretty sure mordred figured that out but I couldn't find any reason not to | 01:41 |
| clarkb | so I was tmpfsing and nothing worked | 01:41 |
| clarkb | silly chattr | 01:41 |
| clarkb | ryanpetrello: woot looks like I finally fixed your issues. Sorry for taking so long | 01:42 |
| *** praneshp has quit IRC | 01:43 | |
| clarkb | ryanpetrello: and welcome to the exclusive my tests run on trusty group. You are the only person there outside of integration testing | 01:43 |
| clarkb | well I am going to fire off that email | 01:43 |
| clarkb | jeblair must be AFK for the evening and I would rather not wait letting people know trsuty is a thing | 01:43 |
| *** asalkeld has left #openstack-infra | 01:44 | |
| *** camunoz has joined #openstack-infra | 01:44 | |
| mordred | clarkb: it does not yet do sockets ... there is another break to fix first | 01:45 |
| *** homeless has quit IRC | 01:46 | |
| jogo | jhesketh: excellent, a beer it is | 01:47 |
| ryanpetrello | clarkb: no problem at all :) | 01:47 |
| ryanpetrello | glad to be part of the trusty party :D | 01:47 |
| anteaya | jyuso: https://wiki.openstack.org/wiki/ThirdPartySystems/IBMPowerKVMCI | 01:47 |
| jyuso | anteaya: thx. | 01:48 |
| anteaya | jyuso: yes tools can be jenkins and the gerrit plugin | 01:48 |
| anteaya | jyuso: yours will be an example too | 01:48 |
| anteaya | jyuso: so I would like to help you get it usable | 01:48 |
| clarkb | ok mail sent | 01:49 |
| jyuso | anteaya: OK,thx,I'll try to fill out the table. | 01:50 |
| anteaya | jyuso: good place to start | 01:50 |
| anteaya | do your best and save so I can look too | 01:51 |
| *** nosnos has joined #openstack-infra | 01:51 | |
| *** unicell has joined #openstack-infra | 01:55 | |
| *** alexpilotti has quit IRC | 01:55 | |
| *** nosnos has quit IRC | 01:56 | |
| *** nosnos has joined #openstack-infra | 01:57 | |
| StevenK | clarkb: \o/ (re: Trusty dsvm) | 01:57 |
| clarkb | StevenK: I will probably need to grab some of your tripleo folks to switch you guys | 01:57 |
| clarkb | but I think you are relatively low priority as you are sorting out other CI issues | 01:58 |
| clarkb | those issues are more important imo | 01:58 |
| anteaya | jyuso: it is 10pm my time | 02:03 |
| anteaya | jyuso: I am going to have to sign off soon | 02:04 |
| clarkb | wow it is that late | 02:04 |
| anteaya | for me | 02:04 |
| anteaya | 7 for u ya? | 02:04 |
| clarkb | ya I didn't realize it was 7pm pdt | 02:04 |
| anteaya | tis | 02:04 |
| *** morganfainberg_Z is now known as morganfainberg | 02:06 | |
| jyuso | anteaya: okanteaya: OK.Maybe contact you tomorrow,good night:) | 02:13 |
| anteaya | jyuso: okay | 02:13 |
| anteaya | I'll read the wikipage in the morning | 02:13 |
| anteaya | i might have to edit it | 02:14 |
| jyuso | anteaya: Yes.I've finished it. | 02:14 |
| anteaya | oh I'll read it | 02:14 |
| jyuso | anteaya: Oh,thanks very much! | 02:15 |
| anteaya | okay I made some grammar edits | 02:16 |
| anteaya | okay so you want to test nova | 02:17 |
| anteaya | mikal: are you about? | 02:17 |
| jyuso | anteaya: yes,just nova now. | 02:17 |
| anteaya | jyuso: okay trying to see if the nova ptl has time to meet you | 02:17 |
| jyuso | anteaya: maybe we will add more component later | 02:17 |
| anteaya | are you in the nova channel? | 02:18 |
| anteaya | let's start small and get it right | 02:18 |
| jyuso | anteaya: not yet | 02:18 |
| anteaya | okay please /join #openstack-nova | 02:18 |
| *** jyuso has quit IRC | 02:18 | |
| *** signed8bit has joined #openstack-infra | 02:18 | |
| *** jyuso has joined #openstack-infra | 02:18 | |
| jyuso | anteaya: OK,I've joined the nova channel. | 02:19 |
| anteaya | jyuso: ah you are on both | 02:19 |
| *** talluri has joined #openstack-infra | 02:20 | |
| *** jamielennox is now known as jamielennox|away | 02:22 | |
| *** arnaud__ has quit IRC | 02:23 | |
| *** fanhe has joined #openstack-infra | 02:25 | |
| *** dstanek_zzz is now known as dstanek | 02:26 | |
| *** dims__ has quit IRC | 02:28 | |
| *** openstackgerrit has quit IRC | 02:31 | |
| *** signed8bit has quit IRC | 02:32 | |
| *** changbl has joined #openstack-infra | 02:32 | |
| *** talluri has quit IRC | 02:33 | |
| *** otherwiseguy has joined #openstack-infra | 02:33 | |
| *** talluri has joined #openstack-infra | 02:34 | |
| *** adrian_otto has joined #openstack-infra | 02:34 | |
| *** timrc is now known as timrc-afk | 02:35 | |
| *** r1chardj0n3s has quit IRC | 02:36 | |
| *** talluri has quit IRC | 02:38 | |
| *** basha has joined #openstack-infra | 02:39 | |
| *** otherwiseguy has quit IRC | 02:40 | |
| *** otherwiseguy has joined #openstack-infra | 02:41 | |
| *** annegent_ has joined #openstack-infra | 02:42 | |
| *** timrc-afk is now known as timrc | 02:42 | |
| ryanpetrello | man | 02:45 |
| ryanpetrello | whatever this bug is, it’s killing me: https://jenkins04.openstack.org/job/gate-pecan-tox-storyboard-tip/4/console | 02:45 |
| ryanpetrello | I think sdague mentioned the other day that it was some sort of mirror bug | 02:46 |
| ryanpetrello | I see this *all the time* w/ pecan tests now, and it’s not uncommon for me to have to recheck like 3 or 4 times to get everything passing :\ | 02:46 |
| clarkb | ryanpetrello: wow | 02:46 |
| clarkb | ryanpetrello: so that is pypi connectivity going away for your slave | 02:46 |
| clarkb | dstufft: ^ | 02:46 |
| *** annegent_ has quit IRC | 02:46 | |
| clarkb | arg we don't do the ip capture in that test | 02:47 |
| clarkb | are we only doing that for tempest? | 02:47 |
| *** fifieldt_ is now known as fifieldt | 02:47 | |
| dstufft | do you specifically see the 503 error, or do you normaly see a Connection Reset by Peer error | 02:47 |
| ryanpetrello | here are two more: | 02:48 |
| ryanpetrello | http://logs.openstack.org/44/94244/17/check/gate-pecan-tox-ceilometer-tip/112b7a2/console.html | 02:48 |
| ryanpetrello | http://logs.openstack.org/44/94244/17/check/gate-pecan-tox-ironic-stable/7a46f4c/console.html | 02:48 |
| dstufft | hrm | 02:50 |
| dstufft | how often is all the time (e.g. what's the rate of change) | 02:50 |
| clarkb | I do note that those are all hpcloud nodes | 02:51 |
| ryanpetrello | sec, generating some more data for you :) | 02:51 |
| dstufft | a 503 error means Fastly -> PyPI broke | 02:52 |
| ryanpetrello | here’s another: http://logs.openstack.org/44/94244/17/check/gate-pecan-tox-ironic-tip/d8b6567/console.html | 02:52 |
| dstufft | Connection Errors means Node -> Fastly broke | 02:52 |
| dstufft | is it possible to get these run with -vvv added to pip | 02:53 |
| dstufft | like is that real hard | 02:53 |
| dstufft | thanks to bullshit external link crap we "fail soft" when an error happens | 02:53 |
| ryanpetrello | know of a way to make tox do that? | 02:53 |
| ryanpetrello | because these look like tox dep installs to me | 02:53 |
| *** sweston_ has joined #openstack-infra | 02:54 | |
| *** dims__ has joined #openstack-infra | 02:54 | |
| clarkb | ryanpetrello: there is a way to overload the install commadn with pip iirc | 02:54 |
| *** gokrokve has joined #openstack-infra | 02:54 | |
| clarkb | in tox | 02:54 |
| *** Ryan_Lane1 has quit IRC | 02:54 | |
| dstufft | install_command = pip install --pre {opts} -vvv {packages} | 02:54 |
| dstufft | add that to the tox.ini underneath the test envs | 02:55 |
| clarkb | https://git.openstack.org/cgit/openstack/nova/tree/tox.ini#n9 | 02:55 |
| dstufft | like [py27]pip install --pre {opts} {packages} | 02:55 |
| clarkb | ryanpetrello: ^ like that bu add the -vvv | 02:55 |
| dstufft | yea there you go | 02:55 |
| *** otherwiseguy has quit IRC | 02:55 | |
| *** otherwiseguy has joined #openstack-infra | 02:56 | |
| *** gokrokve_ has joined #openstack-infra | 02:56 | |
| *** dims__ has quit IRC | 02:58 | |
| *** gokrokve has quit IRC | 02:59 | |
| ryanpetrello | so, I actually *can’t* do this :\ | 03:00 |
| ryanpetrello | because what I’m doing with tox | 03:00 |
| ryanpetrello | is I’m actually *downloading* HEAD or a tag of some other project, e.g., ironic | 03:00 |
| ryanpetrello | and calling `tox` for that project | 03:01 |
| ryanpetrello | (by injecting the pecan patchset into the virtualenv) | 03:01 |
| *** asalkeld has joined #openstack-infra | 03:01 | |
| ryanpetrello | so I actually don’t have control over the tox.ini of the projects I’m gating against | 03:01 |
| ryanpetrello | which is (I suspect) where these pip installs sometimes fail | 03:01 |
| dstufft | uh | 03:02 |
| dstufft | can you set an environment variable | 03:02 |
| ryanpetrello | e.g., https://github.com/stackforge/pecan/blob/master/tox.ini#L154 | 03:02 |
| ryanpetrello | hrm, maybe | 03:02 |
| ryanpetrello | *looks at tox documentation | 03:02 |
| *** e0ne has joined #openstack-infra | 03:02 | |
| clarkb | dstufft: is pip -vvv something we should do everywhere anyways? or is it too chatty? | 03:02 |
| *** praneshp has joined #openstack-infra | 03:04 | |
| *** harlowja is now known as harlowja_away | 03:04 | |
| ryanpetrello | dstufft: looks to me like it’s only passed in via tox config, not an environment variable: https://bitbucket.org/hpk42/tox/src/cefc0fd28dda72ac76a9170b4c586e0eb3f1d124/tox/_venv.py?at=default#cl-264 | 03:04 |
| *** jpeeler has joined #openstack-infra | 03:05 | |
| *** jpeeler has quit IRC | 03:05 | |
| *** jpeeler has joined #openstack-infra | 03:05 | |
| *** jpeeler has quit IRC | 03:05 | |
| *** e0ne has quit IRC | 03:05 | |
| *** jpeeler has joined #openstack-infra | 03:06 | |
| *** jpeeler has quit IRC | 03:06 | |
| *** jpeeler has joined #openstack-infra | 03:06 | |
| dstufft | ryanpetrello: no I mean if you set PIP_VERBOSE=on that'll get you -v (not -vvv) at least | 03:06 |
| ryanpetrello | ah, I see | 03:06 |
| mordred | clarkb, dstufft perhaps we should configure our servers to have vvv set on globally? | 03:06 |
| mordred | dstufft: is there a way to set -vvv through env? | 03:06 |
| dstufft | mordred: clarkb it adds extra stuff to the log, it's chattier but idk if it's signifcantly so, it'll print out every link it looks at and stuff like that | 03:07 |
| dstufft | I don't think you can set -vvv through env unfortinately | 03:07 |
| *** e0ne has joined #openstack-infra | 03:07 | |
| *** otter768 has quit IRC | 03:07 | |
| dstufft | our env vars are created magically from the CLI options, and it appears that doesn't handle the additive options very well | 03:07 |
| dstufft | like -vvv and -qqq | 03:08 |
| mordred | hrm | 03:08 |
| ryanpetrello | https://review.openstack.org/102418 | 03:08 |
| ryanpetrello | like so? | 03:08 |
| mordred | yah. I'd really love to not have to set that on every tox ini | 03:09 |
| dstufft | yea I think so | 03:09 |
| mordred | but that looks right | 03:10 |
| mordred | ryanpetrello: you could also override the install command ... | 03:10 |
| ryanpetrello | mordred: see above | 03:10 |
| ryanpetrello | http://screens.objects.dreamhost.com/06-24-2014-23-10-47.png | 03:10 |
| dstufft | mordred: you guys grab files when you do a run don't you? | 03:10 |
| dstufft | actually scratch that, it won't be useful yet | 03:11 |
| mordred | ryanpetrello: OH. jeez | 03:11 |
| ryanpetrello | yea | 03:11 |
| mordred | ryanpetrello: I grok all of the words you're saying | 03:11 |
| *** camunoz has quit IRC | 03:11 | |
| ryanpetrello | the things I do do ensure compatability | 03:11 |
| mordred | yah. which is awesome | 03:11 |
| ryanpetrello | also, it’s late and I just said do do | 03:11 |
| mordred | hehe | 03:12 |
| *** signed8bit has joined #openstack-infra | 03:12 | |
| *** e0ne has quit IRC | 03:12 | |
| ryanpetrello | but yea, I have a handful of jobs that do e.g., https://github.com/stackforge/pecan/blob/master/tox.ini#L186 | 03:12 |
| ryanpetrello | download dependency source, build its tox27 *without running tests*, install pecan-dev there, actually run the tests w/ the previously built virtualenv | 03:13 |
| ryanpetrello | er, *dependant project* source, that is | 03:13 |
| ryanpetrello | so I do this for any incubated+ plus project that uses pecan | 03:14 |
| ryanpetrello | now this is odd | 03:19 |
| ryanpetrello | https://jenkins01.openstack.org/job/gate-pecan-tox-storyboard-tip/7/console | 03:19 |
| ryanpetrello | this failed, but I don’t see a pip failure | 03:19 |
| ryanpetrello | but again, also hpcloud | 03:19 |
| ryanpetrello | I’m seeing that pattern here, at least | 03:19 |
| ryanpetrello | oh, no, wait | 03:20 |
| ryanpetrello | I see it now, I wasn’t looking at the full log | 03:20 |
| ryanpetrello | dstuff ^ | 03:20 |
| ryanpetrello | this one is with PIP_VERBOSE on | 03:20 |
| ryanpetrello | *dstufft ^ | 03:21 |
| ryanpetrello | (sorry, typoed your name the first time) | 03:22 |
| ryanpetrello | okay | 03:22 |
| ryanpetrello | I’m going to bed | 03:22 |
| ryanpetrello | will resume investigation tomorrow :D | 03:22 |
| dstufft | Connection Reset by Peer'd | 03:23 |
| dstufft | known issue :( | 03:23 |
| dstufft | Can't figure out WTF is causing it | 03:23 |
| *** craigbr has quit IRC | 03:24 | |
| dstufft | mordred: clarkb any idea how wide spread the problem is? Connection Reset by Peer? because pip 1.6 has some mitigations but it's not released yet.. | 03:24 |
| *** camunoz has joined #openstack-infra | 03:25 | |
| *** achanda has joined #openstack-infra | 03:25 | |
| *** nati_ueno has joined #openstack-infra | 03:25 | |
| ryanpetrello | dstufft: seems widespread enough to me | 03:26 |
| ryanpetrello | if I run a pecan patchset w/ 7-8 jobs | 03:26 |
| *** talluri has joined #openstack-infra | 03:26 | |
| mordred | we've been talking about expanding our mirroring to be per-cloud region and to apply to everything | 03:26 |
| ryanpetrello | sometimes I have to recheck the patch 2-3 times to get all the tests to pass | 03:27 |
| mordred | :( | 03:27 |
| mordred | yeah - I think it's about time to attack that problem | 03:27 |
| ryanpetrello | granted, I’m kind of a special case | 03:27 |
| ryanpetrello | I’m downloading a *ton* of packages on each pull request | 03:27 |
| ryanpetrello | since I’m gating against 5-6 projects with unique dependency lists themselves | 03:27 |
| *** devkulkarni has joined #openstack-infra | 03:28 | |
| mordred | ryanpetrello: special case or not, you're an interesting case study to try to solve | 03:29 |
| *** r1chardj0n3s has joined #openstack-infra | 03:30 | |
| *** talluri has quit IRC | 03:30 | |
| *** arnaud__ has joined #openstack-infra | 03:31 | |
| dstufft | blast | 03:31 |
| dstufft | setuptools foils me again | 03:31 |
| *** talluri has joined #openstack-infra | 03:31 | |
| * dstufft is trying to rip setuptools out of pip | 03:31 | |
| mordred | dstufft: it's the worst thing ever | 03:31 |
| dstufft | I have an implementation of PEP440 for pip | 03:32 |
| *** dstanek is now known as dstanek_zzz | 03:32 | |
| dstufft | including the new specifiers | 03:32 |
| dstufft | like ~= and ==.* | 03:32 |
| dstufft | er ==2.* | 03:32 |
| dstufft | but it won't work because I don't ALSO have an implementation that will look at the installed files on disk | 03:32 |
| dstufft | and we need that to see if the installed version matches ;( | 03:33 |
| *** talluri_ has joined #openstack-infra | 03:33 | |
| *** talluri has quit IRC | 03:33 | |
| *** weshay has quit IRC | 03:34 | |
| *** basha has quit IRC | 03:35 | |
| r1chardj0n3s | dstufft: yuck | 03:35 |
| dstufft | oh hey, I forgot r1chardj0n3s was here \o/ | 03:35 |
| dstufft | r1chardj0n3s: second? or uh, third day? :] | 03:36 |
| r1chardj0n3s | dstufft: third :) | 03:36 |
| dstufft | living in the future and what not | 03:37 |
| r1chardj0n3s | yes! | 03:38 |
| *** matsuhashi has quit IRC | 03:42 | |
| *** matsuhashi has joined #openstack-infra | 03:43 | |
| *** matsuhashi has quit IRC | 03:47 | |
| *** wenlock has quit IRC | 03:47 | |
| *** zhiyan_ is now known as zhiyan | 03:49 | |
| *** nosnos has quit IRC | 03:51 | |
| *** signed8bit has quit IRC | 03:51 | |
| *** craigbr has joined #openstack-infra | 03:54 | |
| *** dims__ has joined #openstack-infra | 03:55 | |
| *** mmaglana has quit IRC | 03:58 | |
| *** mmaglana has joined #openstack-infra | 03:58 | |
| *** dims__ has quit IRC | 03:59 | |
| *** devkulkarni has quit IRC | 04:03 | |
| *** mmaglana has quit IRC | 04:03 | |
| *** e0ne has joined #openstack-infra | 04:07 | |
| *** nati_ueno has quit IRC | 04:08 | |
| *** nati_ueno has joined #openstack-infra | 04:09 | |
| *** garyh has quit IRC | 04:09 | |
| *** e0ne has quit IRC | 04:10 | |
| dstufft | mordred: you might enjoy this http://d.stufft.io/image/0I042h1D1t04 | 04:13 |
| *** otherwiseguy has quit IRC | 04:13 | |
| *** otherwiseguy has joined #openstack-infra | 04:14 | |
| *** annegent_ has joined #openstack-infra | 04:16 | |
| *** annegent_ has quit IRC | 04:21 | |
| *** asalkeld has left #openstack-infra | 04:23 | |
| *** mrmartin has joined #openstack-infra | 04:24 | |
| *** dstanek_zzz is now known as dstanek | 04:25 | |
| *** yfried has quit IRC | 04:25 | |
| *** nati_ueno has quit IRC | 04:29 | |
| *** zehicle_at_dell has joined #openstack-infra | 04:31 | |
| *** matsuhashi has joined #openstack-infra | 04:31 | |
| *** basha has joined #openstack-infra | 04:34 | |
| *** melwitt has joined #openstack-infra | 04:34 | |
| *** dstanek is now known as dstanek_zzz | 04:35 | |
| *** nosnos has joined #openstack-infra | 04:37 | |
| *** otherwiseguy has quit IRC | 04:39 | |
| *** achanda has quit IRC | 04:39 | |
| *** gokrokve_ has quit IRC | 04:39 | |
| *** zehicle_at_dell has quit IRC | 04:42 | |
| *** zehicle_at_dell has joined #openstack-infra | 04:42 | |
| *** ramashri has joined #openstack-infra | 04:42 | |
| *** gokrokve_ has joined #openstack-infra | 04:43 | |
| *** basha has quit IRC | 04:44 | |
| *** Sukhdev has joined #openstack-infra | 04:44 | |
| *** ramashri has quit IRC | 04:49 | |
| *** ramashri has joined #openstack-infra | 04:54 | |
| *** dims__ has joined #openstack-infra | 04:56 | |
| *** dstanek_zzz is now known as dstanek | 04:56 | |
| *** achanda has joined #openstack-infra | 04:59 | |
| *** yfried has joined #openstack-infra | 04:59 | |
| *** dims__ has quit IRC | 05:00 | |
| *** gyee has quit IRC | 05:02 | |
| *** gokrokve_ has quit IRC | 05:02 | |
| *** craigbr has quit IRC | 05:05 | |
| *** dstanek is now known as dstanek_zzz | 05:06 | |
| *** jhesketh has quit IRC | 05:08 | |
| *** morganfainberg is now known as morganfainberg_Z | 05:14 | |
| *** annegent_ has joined #openstack-infra | 05:16 | |
| *** markmcclain has joined #openstack-infra | 05:17 | |
| *** nati_ueno has joined #openstack-infra | 05:17 | |
| *** markmcclain1 has joined #openstack-infra | 05:18 | |
| r1chardj0n3s | hey clarkb you there? | 05:18 |
| clarkb | yes | 05:18 |
| r1chardj0n3s | I'm looking to respond to you / adrian in the ML about pypi but need to clear up some of my openstack-noobness so I don't confuse everyone ;) | 05:19 |
| clarkb | sure | 05:19 |
| r1chardj0n3s | what do you mean by "requirements enforcement via checks"? | 05:19 |
| r1chardj0n3s | is that just pinning versions in requirements.txt? | 05:19 |
| clarkb | r1chardj0n3s: so we have a project called openstack/requirements at https://git.openstack.org/cgit/openstack/requirements/tree/ | 05:19 |
| *** cody-somerville has quit IRC | 05:20 | |
| clarkb | r1chardj0n3s: the global-requirements file determines both the contents of our mirror (only those packages and their dependencies are mirrored) and the valid values for requirements.txt and test-requirements.txt in projects | 05:20 |
| clarkb | r1chardj0n3s: we need to enforce that those are the only valid values even if we extend to a proper bandersnatched mirror | 05:20 |
| clarkb | r1chardj0n3s: the other point I made was that we would probably need to add some way of building a platform dependent wheel mirror | 05:21 |
| clarkb | maybe, I don't think tests use that today btu I use it on my laptop :) | 05:21 |
| *** annegent_ has quit IRC | 05:21 | |
| r1chardj0n3s | clarkb: how is the current pypi.openstack updated? | 05:21 |
| dstufft | giant hacks | 05:22 |
| clarkb | r1chardj0n3s: using https://git.openstack.org/cgit/openstack-infra/pypi-mirror/tree/ | 05:22 |
| *** markmcclain has quit IRC | 05:22 | |
| clarkb | yes giant hacks beacuse pypi mirroring was/is terrible | 05:22 |
| r1chardj0n3s | clarkb: I'm trying to figure out how it and a bandersnatched mirror would be different :) | 05:22 |
| clarkb | r1chardj0n3s: well for one thing we will mirror external links contents | 05:22 |
| clarkb | not just the links themselves | 05:22 |
| r1chardj0n3s | clarkb: surely just pinning the versions in global-requirements.txt would solve a bunch of problems? | 05:22 |
| clarkb | no | 05:22 |
| r1chardj0n3s | clarkb: ah, but you're now in a position to ignore external links? | 05:23 |
| clarkb | we already pin everywhere and that is a bit orthogonal | 05:23 |
| clarkb | yup we depend on no external linked projects | 05:23 |
| clarkb | but that is recent | 05:23 |
| r1chardj0n3s | ok | 05:23 |
| clarkb | our mirror also builds platform dependent wheels for all of our packages | 05:23 |
| * mrodden dislikes external links a great great deal.... | 05:23 | |
| clarkb | because wheels neglected to accomodate linux as an end user this is annoying | 05:23 |
| clarkb | and we have to do it ourselve | 05:23 |
| clarkb | eg http://pypi.openstack.org/openstack/CentOS-6.5/ | 05:24 |
| r1chardj0n3s | ok, we seriously need to have a conversation about devpi ;) | 05:24 |
| clarkb | and http://pypi.openstack.org/openstack/Ubuntu-12.04/ | 05:24 |
| *** rwsu has quit IRC | 05:24 | |
| r1chardj0n3s | yep | 05:24 |
| clarkb | well devpi we looked at too | 05:24 |
| clarkb | and it had the same external links issue | 05:24 |
| clarkb | so we basically ignored it back when | 05:24 |
| r1chardj0n3s | but external links aren't an issue now | 05:24 |
| clarkb | right but thats new :) | 05:24 |
| clarkb | like really really new | 05:24 |
| r1chardj0n3s | yes, and I also am new ;) | 05:25 |
| dstufft | clarkb: I would be very happy to have a thing like bandersnatch that instead of mirroring created a platform specific wheel server | 05:25 |
| clarkb | and now we need to take stock again | 05:25 |
| r1chardj0n3s | hence my suggestions are relevant to the new :) | 05:25 |
| clarkb | there was another issue with devpi I am trying to remember what it was | 05:25 |
| r1chardj0n3s | I am going to be at the infra meetup BTW | 05:25 |
| clarkb | I think it maybe didn't figure out dependency trees in weird corner cases? | 05:25 |
| dstufft | devpi is on demand mirroring instead of creating a mirror | 05:25 |
| dstufft | and it doesn't build wheels AFAIK | 05:25 |
| clarkb | dstufft: right and I want to say it doesn't properly spider | 05:25 |
| clarkb | or didnt | 05:25 |
| clarkb | because you actually have to execute code to get all the dependencies in all cases | 05:26 |
| dstufft | I don't think devpi actually spiders at all | 05:26 |
| clarkb | ya so that would be the other issue | 05:26 |
| r1chardj0n3s | devpi doesn't need to spider | 05:26 |
| clarkb | r1chardj0n3s: how does it build a complete mirror of what you need then? | 05:26 |
| dstufft | I think it just does if you ask for /simple/requests/, it caches /simple/requests/ | 05:26 |
| dstufft | it does it on demand | 05:26 |
| r1chardj0n3s | clarkb: by building a complete mirror | 05:26 |
| dstufft | sort of like varnish | 05:26 |
| clarkb | oh its a proxy? | 05:26 |
| *** nati_ueno has quit IRC | 05:26 | |
| dstufft | yes, except smart-er | 05:26 |
| r1chardj0n3s | clarkb: yes | 05:26 |
| dstufft | because it'll use the changelog API to keep it up to date | 05:27 |
| r1chardj0n3s | but it's also smarter, as dstufft says | 05:27 |
| clarkb | so why use devpi over say bandersnatch? | 05:27 |
| r1chardj0n3s | it can also have *overrides* and *extensions* to the index from the root source | 05:27 |
| *** ildikov has quit IRC | 05:27 | |
| clarkb | I think bandersnatch fits our use case better which is mirror everything | 05:27 |
| r1chardj0n3s | clarkb: because what I just said :) | 05:27 |
| clarkb | rather than being on demand which is subject to the failures ryanpetrello is seeing | 05:27 |
| clarkb | r1chardj0n3s: but why do I need those? | 05:27 |
| dstufft | I'm not sure openstack the project needs overrides or extensions | 05:28 |
| clarkb | I am not doing anything in private with private packages | 05:28 |
| clarkb | that is a very corporate sekret sauce use case :) | 05:28 |
| r1chardj0n3s | you guys saw the email in the list just now, yes? | 05:28 |
| clarkb | totally exists as a use case | 05:28 |
| r1chardj0n3s | :) | 05:28 |
| clarkb | r1chardj0n3s: yes I responded to it | 05:28 |
| clarkb | r1chardj0n3s: but devpi doesn't fix that | 05:28 |
| dstufft | hm | 05:28 |
| dstufft | I must not be on this list | 05:28 |
| clarkb | in fact it would just work today as is with no work | 05:28 |
| dstufft | or I ignored it | 05:28 |
| *** mrmartin has quit IRC | 05:28 | |
| clarkb | but you would be subject to pypi.python.org being down | 05:28 |
| clarkb | so solving ^ is the problem at hand | 05:28 |
| dstufft | clarkb: only if you didn't already have it cached | 05:29 |
| r1chardj0n3s | clarkb: there's information I don't have, but I believe devpi could be a solution | 05:29 |
| r1chardj0n3s | note that devpi can have cache pre-loaded | 05:29 |
| r1chardj0n3s | which could include *all* of pypi if you desire | 05:29 |
| clarkb | dstufft: r1chardj0n3s: right so lets back up a bit and assess options | 05:29 |
| clarkb | bandersnatch would just run and give me everything all the time | 05:29 |
| clarkb | that is simple. a bit hammerish but direct and simple | 05:29 |
| dstufft | (for the record, I think bandersnatch is probably more on point for what openstack wants for pypi.openstack.org given what I know it uses it for) | 05:30 |
| clarkb | devpi would add fancy features and do mostly the same thing | 05:30 |
| clarkb | so either would probably work but devpi sounds more complicated | 05:30 |
| clarkb | one concern I have with treating stackforge in a special devpi way is managing that | 05:30 |
| clarkb | its nice to just give people access ot everything and never worry about it | 05:31 |
| r1chardj0n3s | if you can figure out a way to make everything work against a single index of everything, then yes, bandersnatch is your solution | 05:31 |
| r1chardj0n3s | but my understanding was that that is not an option | 05:31 |
| *** amitpp has joined #openstack-infra | 05:31 | |
| clarkb | r1chardj0n3s: it definitely is an option | 05:32 |
| r1chardj0n3s | but again my understanding is about 4 hours old ;) | 05:32 |
| clarkb | with two additional things we need to figure out (maybe just one) | 05:32 |
| clarkb | first is enforcement of openstack/requirements/global-requirements in openstack projects | 05:32 |
| clarkb | we have a lot of that in place today even with the mirror that exists today | 05:32 |
| r1chardj0n3s | how does that enforcement work? | 05:32 |
| clarkb | for example devstack overwrites what any projects requirements may say with what global-requirements says | 05:33 |
| clarkb | and we do testing of that | 05:33 |
| clarkb | we also make sure that any updates to requirements.txt and test-requirements.txt in projects match the global requirements | 05:33 |
| dstufft | r1chardj0n3s: the current mirror is built using pip install --download from the requirements files, so if the mirror contains it, then it's allowed to be used ala global requirements more or less | 05:33 |
| clarkb | and that is the third enforcement mechanism | 05:33 |
| clarkb | its possible the first 2 are sufficient | 05:33 |
| r1chardj0n3s | that's pretty much what I understood - but you're proposing to move to a mirror of *everything* so that won't hold any longer | 05:34 |
| clarkb | r1chardj0n3s: we would rely on the first two mechanisms | 05:34 |
| *** markmc has joined #openstack-infra | 05:34 | |
| clarkb | and possible add more if those are insufficient | 05:34 |
| dstufft | hm | 05:35 |
| dstufft | you know | 05:35 |
| *** adrian_otto has quit IRC | 05:35 | |
| clarkb | even with a devpi setup we couldn't get rid of those two mechanism | 05:35 |
| r1chardj0n3s | but global-requirements doesn't pin versions | 05:35 |
| clarkb | r1chardj0n3s: in cases without a pin we allow up to whenver | 05:35 |
| clarkb | the reason we need those two mechanisms is pip dep resolution essentially | 05:35 |
| dstufft | r1chardj0n3s: so global requirements isn't "you must install this exact version", it "your version spec must match what global requirements says the version soec is" | 05:35 |
| *** penguinRaider has joined #openstack-infra | 05:36 | |
| dstufft | so if global requirements says requests>=1.5, then you install_requires has to be requests>=1.5 | 05:36 |
| r1chardj0n3s | sure, but that means there's no upper bound - the upper bound is only limited by the current partial mirror | 05:36 |
| clarkb | if we let any project differ in integration testing we may test somethign different than the global reqs because pip can install whatever in that case | 05:36 |
| clarkb | r1chardj0n3s: correct but that is true of a complete mirror too | 05:36 |
| *** timrc is now known as timrc-afk | 05:36 | |
| clarkb | that doesnt' chagne anything | 05:36 |
| clarkb | we try to leave the top end open to discover when we break | 05:36 |
| dstufft | clarkb: this proof of concept I have for pip could be leveraged so that pip properly handles multiple projects depending on the same thing, not in 1.6 timeline, maybe 1.7 timeline (not sure, something to think about anyways) | 05:36 |
| clarkb | otherwise you end up needinging to do massive migrations in deps constantly | 05:37 |
| r1chardj0n3s | clarkb: wha? your current mirror has an explicitly restricted set of vetted versions, and you're proposing moving to a mirror that has no restricted set. not the same thing at all? | 05:37 |
| clarkb | r1chardj0n3s: no that isn't how it works | 05:37 |
| clarkb | r1chardj0n3s: the restricted mirror builds use that exact list | 05:37 |
| r1chardj0n3s | (BTW if there is something I can just read instead of being annoying here just let me know) | 05:37 |
| clarkb | r1chardj0n3s: which if open ended will be the same as a complete mirror on the top end | 05:37 |
| clarkb | r1chardj0n3s: I mean you can read lots of ugly shell and stuff :P | 05:38 |
| clarkb | if there is a pin then the non mirror enforcement mechanisms should enforce that | 05:38 |
| clarkb | if there isn't a pin it doesn't matter | 05:38 |
| dstufft | r1chardj0n3s: the pypi.o.o rebuilds every hour? (I think?) so if the version spec in global requirements is open ended, then the mirror will get whatever the latest is | 05:38 |
| dstufft | modulo lag time for the mirror to update | 05:38 |
| clarkb | dstufft: it is once a day at least. more if we need to update it based on our own releases | 05:38 |
| r1chardj0n3s | oh! someone told me that only vetted versions get into pypi.o.o | 05:38 |
| clarkb | r1chardj0n3s: that is wrong | 05:39 |
| r1chardj0n3s | right! | 05:39 |
| r1chardj0n3s | hence my confusion | 05:39 |
| clarkb | and it is done this way by design | 05:39 |
| dstufft | nah, no vetting other than it has to match the version spec in global requirements | 05:39 |
| clarkb | so that we can discover when we are broken early rather than when debian says you must update this dep | 05:39 |
| *** markmcclain1 has quit IRC | 05:39 | |
| *** basha has joined #openstack-infra | 05:40 | |
| r1chardj0n3s | ok, so switching pypi.o.o to bandersnatch seems a no-brainer because it'll be just the same, but with more packages that aren't used. and the limitation on allowed packages is enforced by current global-requirements.txt etc | 05:41 |
| clarkb | right | 05:41 |
| r1chardj0n3s | *however* that doesn't solve your wheels issue, which devpi would ;) | 05:41 |
| clarkb | I think we just need to do a santiy check that the package enforcement works | 05:41 |
| dstufft | r1chardj0n3s: does devpi build wheels? | 05:41 |
| *** basha has quit IRC | 05:41 | |
| r1chardj0n3s | dstufft: is *building* the wheels the issue? | 05:41 |
| clarkb | r1chardj0n3s: yes | 05:42 |
| clarkb | because pypi.python.org only hosts platform independent linx wheels | 05:42 |
| r1chardj0n3s | right, then neither bandersnatch not devpi help ;) | 05:42 |
| *** amitpp has quit IRC | 05:42 | |
| dstufft | you'd want something like bandernsatch, but instead of mirroring it downloaded + built wheels | 05:42 |
| clarkb | so if we need centos and ubuntu trusty wheels we have to build them ourselves | 05:42 |
| r1chardj0n3s | that's a *hosting* issue, not a *building* issue ;) | 05:42 |
| clarkb | r1chardj0n3s: well its a bit of both. imo a bug in the wheel pep | 05:42 |
| r1chardj0n3s | I'm trying to separate the concerns here | 05:42 |
| r1chardj0n3s | 'cos you're smooshing them together which I think is a mistake | 05:42 |
| clarkb | r1chardj0n3s: lets talk in terms of what we have today | 05:42 |
| dstufft | clarkb: (the wheel problem for linux will be fixed btw, it's on the stack :|) | 05:43 |
| clarkb | we want http://pypi.openstack.org/openstack/Ubuntu-12.04/ | 05:43 |
| clarkb | and http://pypi.openstack.org/openstack/CentOS-6.5/ | 05:43 |
| r1chardj0n3s | that looks a helluva lot like a devpi url ;) | 05:43 |
| clarkb | r1chardj0n3s: ok, where does devpi get those wheels? | 05:43 |
| r1chardj0n3s | something builds and submits it to devpi | 05:43 |
| clarkb | ok so devpi is independent of the problem then | 05:44 |
| *** timrc-afk is now known as timrc | 05:44 | |
| clarkb | as the problem is building them | 05:44 |
| clarkb | (hosting is easy) | 05:44 |
| *** ildikov has joined #openstack-infra | 05:44 | |
| r1chardj0n3s | clarkb *except* that it provides a very nice mechanism for namespaces which can be things like "ubuntu-12.04" and "centos-6.5" | 05:44 |
| dstufft | well the filesystem and nginx does that too :) | 05:44 |
| clarkb | right, I don't want to jump on a tool because it makes easy things easy | 05:45 |
| clarkb | if it makes hard things easy ++ lets do it | 05:45 |
| r1chardj0n3s | dstufft it doesn't also bring in all the rest of PyPI packages into the same view | 05:45 |
| clarkb | (/me is still bitter about dib taking so long for our slave images) | 05:45 |
| dstufft | r1chardj0n3s: eh, pip does that | 05:45 |
| r1chardj0n3s | dstufft: uh, yuck | 05:45 |
| dstufft | pip supports multple index urls for a long time | 05:45 |
| clarkb | also doesn't devpi have its own client? | 05:45 |
| clarkb | or would we setup pip to do ^ | 05:45 |
| r1chardj0n3s | clarkb for some devpi operations yes, but it is just an index to pip | 05:46 |
| dstufft | devpi shines a lot when you want to promote packages from one index to another | 05:46 |
| dstufft | or you want a partial mirror | 05:46 |
| *** ihrachyshka has joined #openstack-infra | 05:46 | |
| *** ihrachyshka has quit IRC | 05:46 | |
| r1chardj0n3s | dstufft: multiple index support in tools can cause pain, is all I gonna say | 05:46 |
| *** praneshp_ has joined #openstack-infra | 05:47 | |
| dstufft | heh | 05:47 |
| clarkb | so to recap. I think devpi is a possibility. I don't think it solves any actual problems in a way that would make it a prefered too | 05:47 |
| dstufft | sure :) | 05:47 |
| clarkb | *tool | 05:47 |
| clarkb | it solves lots of other problems that people have but not problems we have | 05:47 |
| dstufft | it's packaging, you're not looking for painfree, you're looking for tolerable pain | 05:47 |
| dstufft | :D | 05:47 |
| r1chardj0n3s | does any of this solve the solum issue? :) | 05:48 |
| clarkb | r1chardj0n3s: no, solum is a non issue | 05:48 |
| dstufft | what | 05:48 |
| clarkb | its already solved | 05:49 |
| dstufft | is the solum issue | 05:49 |
| clarkb | 2 years ago | 05:49 |
| r1chardj0n3s | \o/ | 05:49 |
| *** praneshp has quit IRC | 05:49 | |
| *** praneshp_ is now known as praneshp | 05:49 | |
| clarkb | or whenever we added the mirror | 05:49 |
| clarkb | dstufft: solum wants to use pypi.openstack.org and pypi.python.org | 05:49 |
| dstufft | oh is the issue that distributed networks are hard | 05:49 |
| clarkb | dstufft: it will just work today | 05:49 |
| dstufft | gotcha | 05:49 |
| clarkb | as long as pypi.python.org doesn't 503 :P | 05:49 |
| dstufft | ;( | 05:49 |
| clarkb | dstufft: http://lists.openstack.org/pipermail/openstack-dev/2014-June/038609.html | 05:50 |
| r1chardj0n3s | openstack really shouldn't be depending full-time on pypi.p.o ;) | 05:50 |
| dstufft | I hate glusterfs | 05:50 |
| clarkb | r1chardj0n3s: we don't | 05:50 |
| clarkb | r1chardj0n3s: we depend on python.openstack.org | 05:50 |
| clarkb | er pypi.openstack.org | 05:50 |
| clarkb | typing hard | 05:50 |
| r1chardj0n3s | clarkb: well, except solum | 05:50 |
| clarkb | and fallback to pypi.python.org in non enforced projects | 05:50 |
| clarkb | r1chardj0n3s: no arg | 05:50 |
| clarkb | I feel like there is a massive misunderstanding somewhere | 05:50 |
| clarkb | and my exaplaning in the thread doesn't clear it up :( | 05:51 |
| dstufft | clarkb: fwiw once we get access to OnMetal I'm going to be looking at figuring out what's the best way to get glusterfs some consistent IOPS | 05:51 |
| clarkb | all testing on on giant botnet will use pypi.openstack.org first | 05:51 |
| dstufft | glusterfs gets really cranky when the IOPS arn't consistent | 05:51 |
| clarkb | if you are not part of openstack/requirements you will fall back on pypi.python.org | 05:51 |
| clarkb | dstufft: nice, I was thinking about switching to the IO onmetal for ES too :) but haven't figured out what is involved in that | 05:52 |
| r1chardj0n3s | clarkb: based on what I've learned in here, your response in the ML is sound. | 05:52 |
| dstufft | (Also I'm of the opinion that you *should* be able to rely on PyPI not 503ing or being crappy, or at least "PyPI access is inconsistent" shouldn't be a reasonf or someone not to just use PyPI) | 05:52 |
| r1chardj0n3s | clarkb: for solum, they'll just point at pypi.o.o and it'll all just work, even for their bizarro dependencies. assuming they can get their requirements.txt accepted by whatever vetting is done. | 05:52 |
| clarkb | r1chardj0n3s: no | 05:52 |
| *** basha has joined #openstack-infra | 05:52 | |
| r1chardj0n3s | darn, I thought I had it there | 05:53 |
| clarkb | well that is one possiblility. my suggestion was to just remove them from the requirements projects list and fall back on pypi.python.org | 05:53 |
| clarkb | as that will work right now | 05:53 |
| r1chardj0n3s | apparently removing them from requirements/projects.txt is Not Gonna Happen | 05:53 |
| clarkb | the other thing requires review and vetting etc | 05:53 |
| clarkb | r1chardj0n3s: oh well then that is their problem | 05:53 |
| clarkb | there is nothing preventing them from doing that | 05:53 |
| r1chardj0n3s | :) | 05:53 |
| clarkb | they can still have enforcement on the non mirror side | 05:54 |
| clarkb | and do hacks to get mistral as a req | 05:54 |
| clarkb | but really if you want to say you are going to be openstack but not openstack it doesn't make sense to me to bend over backwards to circumvent the checks in place | 05:54 |
| clarkb | this is working as designed with a clear out | 05:54 |
| r1chardj0n3s | clarkb: the problem is that the "and do hacks" part can get mighty complicuated when you involved tox et al | 05:55 |
| clarkb | right so I don't suggest they do that | 05:55 |
| clarkb | I suggest they remove themselves from the projects list | 05:55 |
| dstufft | clarkb: forgive me but I forget, the reason for the global reqs crap was that pip sucks at handling the case where two things depend on the same project with different version specs yea? | 05:55 |
| clarkb | dstufft: partially yes | 05:55 |
| clarkb | dstufft: basically if you installed openstack you had no idea what versions of things would actually be used | 05:55 |
| clarkb | because pip dep resolution was order dependent and that chagned every other day | 05:56 |
| dstufft | yea | 05:56 |
| clarkb | so we standardized across the board and made it explicit | 05:56 |
| dstufft | I'm thinking about that at the moment, because I think I can fix that | 05:56 |
| clarkb | the other reason for it is the distros | 05:56 |
| *** basha has quit IRC | 05:56 | |
| *** Sukhdev has quit IRC | 05:56 | |
| clarkb | it makes it much easier for them to keep track of what is needed | 05:56 |
| clarkb | as we have tested that specific list | 05:56 |
| *** dims__ has joined #openstack-infra | 05:56 | |
| clarkb | (one pip resolved version of it at least) | 05:56 |
| *** dstanek_zzz is now known as dstanek | 05:57 | |
| clarkb | r1chardj0n3s: so even if we go the bandersnatch/devpi route it sounds like solum won't be happy | 05:57 |
| clarkb | r1chardj0n3s: I think that is because they have set up an unresolvable problem. Want to be openstack but not | 05:57 |
| r1chardj0n3s | clarkb: yeah, the issue bign that their requirements.txt isn't gonna validate. unless a special-case is added to the validation to allow unvalidated-requirements.txt or some such | 05:58 |
| clarkb | right | 05:58 |
| clarkb | which is counter to the whole point of the thing | 05:58 |
| *** mmaglana has joined #openstack-infra | 05:58 | |
| *** gokrokve has joined #openstack-infra | 05:59 | |
| r1chardj0n3s | yup | 05:59 |
| clarkb | I get wanting to haev a complete pypi mirror so that falling back to pypi is less scary | 05:59 |
| clarkb | I don't get ^ | 05:59 |
| *** chandan_kumar has quit IRC | 06:00 | |
| clarkb | way back when there was a less standard set of requiremetns files and all sorts of fun things happened with them | 06:00 |
| clarkb | my favorite was installing test requirements always... | 06:00 |
| clarkb | but ya would be good to avoid that adhoc way of approaching dependencies | 06:00 |
| clarkb | dstufft: can pip solve that problem if I say install nova now, then 20 minute later install swift? | 06:01 |
| clarkb | dstufft: pip won't be aware of the intersection between them until it is too late. Or would you try to backfill when you install swift? | 06:01 |
| dstufft | clarkb: so there is two different problems | 06:01 |
| *** dims__ has quit IRC | 06:01 | |
| dstufft | one is that if you ``pip install nova swift`` that pip is really stupid and if they both depend on the s\ame thing, it'll just use the first defintion of that it finds | 06:02 |
| dstufft | instead of combining them | 06:02 |
| clarkb | right | 06:02 |
| dstufft | that is easier to fix | 06:02 |
| dstufft | the second one is wjhat you said | 06:02 |
| dstufft | pip install nova && pip install swift | 06:02 |
| dstufft | that should still be solvable | 06:02 |
| *** mmaglana has quit IRC | 06:03 | |
| dstufft | but it requires more work, because we'll need to make sure we preservce the version specs for each individual project when we install them | 06:03 |
| dstufft | and then make sure that we don't invalidate the version specs even for installed stuff | 06:03 |
| *** gokrokve has quit IRC | 06:04 | |
| clarkb | mordred: foobie bletch was abandoned | 06:04 |
| clarkb | mordred: I love getting those emails :) | 06:05 |
| dstufft | clarkb: so pip has all the mechanisms in place to actually handle this I think, we'd just need to actually record the data + populate it at the beginning of a pip install. Mostly it'd be changing the concept from "pip go do this stuff" to "pip modify my environment to add this extra stuff and re-resolve the dependencies" | 06:06 |
| clarkb | like a package manager :) | 06:07 |
| *** dstanek is now known as dstanek_zzz | 06:07 | |
| clarkb | thats cool though, it would solve a lot of problems | 06:07 |
| *** sweston__ has joined #openstack-infra | 06:10 | |
| *** achanda has quit IRC | 06:10 | |
| *** sweston_ has quit IRC | 06:10 | |
| *** afazekas_ has joined #openstack-infra | 06:13 | |
| *** achanda has joined #openstack-infra | 06:15 | |
| *** annegent_ has joined #openstack-infra | 06:16 | |
| *** sweston__ has quit IRC | 06:18 | |
| *** annegent_ has quit IRC | 06:21 | |
| clarkb | one potential headache that we could see with a move to a true mirror is pip dependency resolution allowing common dependencies to be installed at incompatible versions making things die | 06:21 |
| clarkb | so the above would help with that | 06:21 |
| *** achanda has quit IRC | 06:22 | |
| *** penguinRaider has quit IRC | 06:24 | |
| dstufft | plus the whole thing about the fact that pip should just work correctly | 06:24 |
| dstufft | and not be crazy | 06:24 |
| clarkb | (note that doesn't affect the correctness of things as we should fail properly, we will just have a new class of failure) | 06:24 |
| dstufft | clarkb: openstack might like the PEP 440 specifiers at least :D little things like < is exclusive so like foo<3 will not match 3.0.dev1 and such, also ~= is awesome, I love ~= | 06:25 |
| *** penguinRaider has joined #openstack-infra | 06:25 | |
| *** _nadya_ has joined #openstack-infra | 06:25 | |
| clarkb | arg it is too hot in seattle to sleep but I must try | 06:27 |
| clarkb | good night | 06:27 |
| r1chardj0n3s | g'night! | 06:27 |
| clarkb | r1chardj0n3s: I hope that huge dump of info helps | 06:27 |
| r1chardj0n3s | yes! thankyou :) | 06:27 |
| clarkb | r1chardj0n3s: poking around in the openstack/requirements repo may make it a bit more clear | 06:28 |
| *** basha has joined #openstack-infra | 06:28 | |
| *** sweston_ has joined #openstack-infra | 06:30 | |
| *** sweston__ has joined #openstack-infra | 06:31 | |
| *** gcha has quit IRC | 06:34 | |
| *** sweston_ has quit IRC | 06:35 | |
| *** gcha has joined #openstack-infra | 06:35 | |
| *** asettle has quit IRC | 06:37 | |
| *** nati_ueno has joined #openstack-infra | 06:42 | |
| *** andreykurilin_ has joined #openstack-infra | 06:43 | |
| *** kashyap has quit IRC | 06:45 | |
| *** arnaud__ has quit IRC | 06:46 | |
| *** arnaud__ has joined #openstack-infra | 06:48 | |
| *** kashyap has joined #openstack-infra | 06:48 | |
| *** r1chardj0n3s has quit IRC | 06:51 | |
| *** kashyap` has joined #openstack-infra | 06:52 | |
| *** pfallenop has quit IRC | 06:52 | |
| *** pfallenop has joined #openstack-infra | 06:54 | |
| *** kashyap has quit IRC | 06:56 | |
| *** dims__ has joined #openstack-infra | 06:57 | |
| *** dstanek_zzz is now known as dstanek | 06:57 | |
| *** camunoz is now known as camunoz_gone | 06:59 | |
| *** camunoz_gone is now known as camunoz | 06:59 | |
| *** camunoz is now known as camunoz_gone | 07:00 | |
| *** zehicle_at_dell has quit IRC | 07:00 | |
| *** gokrokve has joined #openstack-infra | 07:00 | |
| *** praneshp has quit IRC | 07:01 | |
| *** _nadya_ has quit IRC | 07:01 | |
| *** rgerganov has joined #openstack-infra | 07:03 | |
| *** dims__ has quit IRC | 07:04 | |
| *** gokrokve has quit IRC | 07:04 | |
| *** kashyap` is now known as kashyap | 07:05 | |
| *** cody-somerville has joined #openstack-infra | 07:06 | |
| *** dstanek is now known as dstanek_zzz | 07:07 | |
| *** roeyc has joined #openstack-infra | 07:10 | |
| *** pfallenop has quit IRC | 07:10 | |
| *** mrmartin has joined #openstack-infra | 07:12 | |
| *** ramashri has quit IRC | 07:13 | |
| *** freyes has joined #openstack-infra | 07:15 | |
| *** annegent_ has joined #openstack-infra | 07:16 | |
| *** freyes has quit IRC | 07:16 | |
| *** tkelsey has joined #openstack-infra | 07:18 | |
| *** markwash has joined #openstack-infra | 07:19 | |
| *** freyes has joined #openstack-infra | 07:21 | |
| *** annegent_ has quit IRC | 07:23 | |
| *** jcoufal has joined #openstack-infra | 07:30 | |
| *** skolekonov has joined #openstack-infra | 07:33 | |
| *** basha has quit IRC | 07:34 | |
| *** Longgeek has joined #openstack-infra | 07:37 | |
| kashyap | Is there a way to convince paste.openstack.org to wrap lines? e.g. I have to horizontally scroll -- http://paste.openstack.org/show/84854/ | 07:37 |
| *** Longgeek has quit IRC | 07:37 | |
| *** Longgeek has joined #openstack-infra | 07:37 | |
| *** _nadya_ has joined #openstack-infra | 07:38 | |
| *** e0ne has joined #openstack-infra | 07:40 | |
| *** Hal_ has joined #openstack-infra | 07:44 | |
| *** basha has joined #openstack-infra | 07:48 | |
| *** sweston_ has joined #openstack-infra | 07:48 | |
| *** andreykurilin_ has quit IRC | 07:50 | |
| *** jpich has joined #openstack-infra | 07:50 | |
| *** sweston__ has quit IRC | 07:51 | |
| *** arnaud__ has quit IRC | 07:51 | |
| *** e0ne has quit IRC | 07:51 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 07:53 | |
| *** matsuhashi has quit IRC | 07:57 | |
| *** hashar has joined #openstack-infra | 07:58 | |
| *** matsuhashi has joined #openstack-infra | 07:58 | |
| *** dstanek_zzz is now known as dstanek | 07:58 | |
| *** mmaglana has joined #openstack-infra | 07:58 | |
| *** melwitt1 has joined #openstack-infra | 07:59 | |
| *** e0ne has joined #openstack-infra | 08:00 | |
| *** gokrokve has joined #openstack-infra | 08:00 | |
| *** IvanBerezovskiy has joined #openstack-infra | 08:01 | |
| *** doude has joined #openstack-infra | 08:02 | |
| *** melwitt has quit IRC | 08:02 | |
| *** mrda is now known as mrda-away | 08:02 | |
| *** mmaglana has quit IRC | 08:03 | |
| *** ihrachyshka has joined #openstack-infra | 08:03 | |
| *** katyafervent is now known as katyafervent_awa | 08:03 | |
| *** matsuhashi has quit IRC | 08:05 | |
| *** jlibosva has joined #openstack-infra | 08:05 | |
| *** gokrokve has quit IRC | 08:06 | |
| *** timrc is now known as timrc-afk | 08:07 | |
| *** kashyap is now known as kashyap_lunch | 08:07 | |
| *** dstanek is now known as dstanek_zzz | 08:08 | |
| *** zehicle_at_dell has joined #openstack-infra | 08:09 | |
| *** dizquierdo has joined #openstack-infra | 08:09 | |
| *** matsuhashi has joined #openstack-infra | 08:10 | |
| *** pblaho has joined #openstack-infra | 08:15 | |
| *** timrc-afk is now known as timrc | 08:16 | |
| *** annegent_ has joined #openstack-infra | 08:16 | |
| *** roeyc has quit IRC | 08:17 | |
| *** matsuhashi has quit IRC | 08:17 | |
| *** matsuhashi has joined #openstack-infra | 08:18 | |
| *** annegent_ has quit IRC | 08:21 | |
| *** srenatus has quit IRC | 08:21 | |
| *** nijaba has joined #openstack-infra | 08:21 | |
| *** nijaba has joined #openstack-infra | 08:22 | |
| *** srenatus has joined #openstack-infra | 08:22 | |
| *** isviridov|away is now known as isviridov | 08:26 | |
| *** jdahlin has quit IRC | 08:27 | |
| *** matsuhashi has quit IRC | 08:31 | |
| *** timrc is now known as timrc-afk | 08:34 | |
| *** pelix has joined #openstack-infra | 08:36 | |
| *** andreaf has joined #openstack-infra | 08:39 | |
| *** melwitt1 has quit IRC | 08:40 | |
| *** katyafervent_awa is now known as katyafervent | 08:41 | |
| *** jedimike has joined #openstack-infra | 08:43 | |
| *** nosnos has quit IRC | 08:45 | |
| therve | openstackgerrit has disappeared, if someone knows how to fix it | 08:50 |
| therve | SergeyLukjanov perharps? :) | 08:51 |
| *** openstackgerrit has joined #openstack-infra | 08:53 | |
| *** nijaba has quit IRC | 08:56 | |
| *** dstanek_zzz is now known as dstanek | 08:59 | |
| *** gokrokve has joined #openstack-infra | 09:01 | |
| *** dims has joined #openstack-infra | 09:02 | |
| *** _nadya_ has quit IRC | 09:03 | |
| *** e0ne has quit IRC | 09:03 | |
| *** gokrokve has quit IRC | 09:06 | |
| *** habib has joined #openstack-infra | 09:06 | |
| *** jp_at_hp has joined #openstack-infra | 09:06 | |
| *** e0ne has joined #openstack-infra | 09:07 | |
| *** jp_at_hp has quit IRC | 09:07 | |
| *** jp_at_hp has joined #openstack-infra | 09:07 | |
| *** e0ne_ has joined #openstack-infra | 09:08 | |
| *** dims has quit IRC | 09:08 | |
| *** pblaho is now known as pblaho|afk | 09:09 | |
| *** dstanek is now known as dstanek_zzz | 09:09 | |
| *** kashyap_lunch is now known as kashyap | 09:10 | |
| *** e0ne has quit IRC | 09:12 | |
| *** annegent_ has joined #openstack-infra | 09:16 | |
| *** cnesa has joined #openstack-infra | 09:18 | |
| *** jyuso has quit IRC | 09:18 | |
| *** pfallenop has joined #openstack-infra | 09:20 | |
| *** annegent_ has quit IRC | 09:21 | |
| *** amotoki has quit IRC | 09:24 | |
| *** nati_ueno has quit IRC | 09:31 | |
| *** lcheng has joined #openstack-infra | 09:33 | |
| *** habib has quit IRC | 09:34 | |
| *** _nadya_ has joined #openstack-infra | 09:34 | |
| *** habib has joined #openstack-infra | 09:34 | |
| *** nosnos has joined #openstack-infra | 09:36 | |
| *** matsuhashi has joined #openstack-infra | 09:36 | |
| *** nijaba has joined #openstack-infra | 09:37 | |
| *** nijaba has joined #openstack-infra | 09:37 | |
| *** habib has quit IRC | 09:42 | |
| *** ihrachyshka has quit IRC | 09:44 | |
| *** _nadya_ has quit IRC | 09:52 | |
| SergeyLukjanov | therve, I've restarted it again :) | 09:53 |
| SergeyLukjanov | devananda, lifeless, folks, could you please confirm that https://review.openstack.org/#/c/100063 could be approved | 09:53 |
| *** matsuhashi has quit IRC | 09:54 | |
| *** dtantsur|afk is now known as dtantsur | 09:54 | |
| *** matsuhashi has joined #openstack-infra | 09:55 | |
| *** pblaho|afk is now known as pblaho | 09:56 | |
| *** habib has joined #openstack-infra | 09:56 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/storyboard-webclient: Display dates in timeago format https://review.openstack.org/96713 | 09:57 |
| *** matsuhas_ has joined #openstack-infra | 09:57 | |
| *** matsuhashi has quit IRC | 09:57 | |
| *** unicell has quit IRC | 09:58 | |
| *** dstanek_zzz is now known as dstanek | 10:00 | |
| *** gokrokve has joined #openstack-infra | 10:02 | |
| *** fanhe has quit IRC | 10:07 | |
| *** gokrokve has quit IRC | 10:07 | |
| *** yamahata has quit IRC | 10:08 | |
| *** dstanek is now known as dstanek_zzz | 10:10 | |
| *** mkerrin has quit IRC | 10:12 | |
| *** sabeen has quit IRC | 10:15 | |
| *** annegent_ has joined #openstack-infra | 10:16 | |
| *** ihrachyshka has joined #openstack-infra | 10:17 | |
| *** _nadya_ has joined #openstack-infra | 10:18 | |
| *** Hal_ has quit IRC | 10:19 | |
| *** jpich has quit IRC | 10:20 | |
| *** jpich has joined #openstack-infra | 10:20 | |
| *** annegent_ has quit IRC | 10:21 | |
| *** ihrachyshka has quit IRC | 10:21 | |
| *** ihrachyshka has joined #openstack-infra | 10:22 | |
| *** rm_work is now known as rm_work|away | 10:23 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/storyboard-webclient: Display dates in timeago format https://review.openstack.org/96713 | 10:25 |
| *** penguinRaider has left #openstack-infra | 10:27 | |
| *** _nadya_ has quit IRC | 10:27 | |
| *** _nadya_ has joined #openstack-infra | 10:28 | |
| *** matsuhas_ has quit IRC | 10:28 | |
| *** matsuhashi has joined #openstack-infra | 10:28 | |
| *** amitpp has joined #openstack-infra | 10:28 | |
| *** ociuhandu has joined #openstack-infra | 10:29 | |
| johnthetubaguy | is there anyone that can talk about auto-abandoning patches, and how to turn that on for the nova-specs repos, or maybe its already on? | 10:31 |
| *** lcheng has quit IRC | 10:31 | |
| *** habib has quit IRC | 10:31 | |
| *** habib__ has joined #openstack-infra | 10:34 | |
| sdague | johnthetubaguy: it's off for everything now | 10:35 |
| johnthetubaguy | sdague: ah, I did not no that | 10:36 |
| johnthetubaguy | sdague: curious as to why, was it annoying too many people? | 10:36 |
| *** matsuhashi has quit IRC | 10:37 | |
| johnthetubaguy | s/no/know/ | 10:37 |
| sdague | johnthetubaguy: there was some feeling that with the new dashboard abilities, it wouldn't be needed. Personally, I'd like it back on some things as well. | 10:37 |
| *** matsuhashi has joined #openstack-infra | 10:37 | |
| sdague | +2s have abandon authority now though | 10:37 |
| sdague | which means you could write a script to do it client side | 10:37 |
| *** gokrokve has joined #openstack-infra | 10:37 | |
| johnthetubaguy | sdague: yeah, I noticed the extra button, thinking about lots of old specs with −2s on them | 10:38 |
| BobBall | Anyone seen a problem with git clone failing because a .coveragerc file exists? devstack is failing for me and I'm confused why it's not failing for everyone :) I've certainly not changed anything to do with coverage | 10:38 |
| *** gokrokve has quit IRC | 10:38 | |
| BobBall | 2014-06-25 06:37:25.238 | error: Untracked working tree file '.coveragerc' would be overwritten by merge. | 10:38 |
| BobBall | 00:33:37.624 2014-06-25 06:37:25.238 | fatal: unable to checkout working tree | 10:38 |
| sdague | BobBall: what tree? | 10:38 |
| johnthetubaguy | sdague: anyways, thanks for the update, will work around that | 10:38 |
| BobBall | it's an internal tree we make by automatically merging the Citrix patches with upstream. There is a .coveragerc (in swift) but I don't get why devstack fails to checkout to the local directory which _should_ have been empty before... | 10:39 |
| sdague | BobBall: well if you are using devstack-gate, it's not empty | 10:40 |
| sdague | it's synced from pre populated trees | 10:41 |
| BobBall | not using d-g in this setup - just running devstack from supposedly fresh | 10:41 |
| BobBall | https://gist.github.com/BobBall/e2ea98ae344d708a1ed2#file-gistfile1-txt-L7308 is the error I see | 10:41 |
| BobBall | does something in devstack create the coverage files? | 10:41 |
| BobBall | I'm just confused... | 10:41 |
| *** ihrachyshka has quit IRC | 10:47 | |
| *** ihrachyshka has joined #openstack-infra | 10:47 | |
| *** dtantsur is now known as dtantsur|lunch | 10:48 | |
| *** roeyc has joined #openstack-infra | 10:52 | |
| *** unicell has joined #openstack-infra | 10:53 | |
| *** habib__ has quit IRC | 10:53 | |
| *** habib__ has joined #openstack-infra | 10:54 | |
| *** talluri_ has quit IRC | 10:56 | |
| *** talluri has joined #openstack-infra | 10:57 | |
| openstackgerrit | Ana Krivokapic proposed a change to openstack/requirements: Add Pint for Horizon https://review.openstack.org/97224 | 10:57 |
| *** dstanek_zzz is now known as dstanek | 11:00 | |
| *** talluri has quit IRC | 11:01 | |
| *** dims has joined #openstack-infra | 11:03 | |
| dhellmann | clarkb: I noticed discussion of python versions in the scrollback from last night. The notes from the summit session have some additional info about when we expect to be able to drop 2.6 and seem to suggest 3.4 is going to be a reasonable target soon: https://etherpad.openstack.org/p/juno-cross-project-future-of-python | 11:04 |
| *** dims has quit IRC | 11:08 | |
| *** gokrokve has joined #openstack-infra | 11:08 | |
| *** gokrokve_ has joined #openstack-infra | 11:10 | |
| *** gokrokve has quit IRC | 11:13 | |
| *** akrivoka has joined #openstack-infra | 11:15 | |
| *** gokrokve_ has quit IRC | 11:15 | |
| *** yamahata has joined #openstack-infra | 11:15 | |
| anteaya | dstufft: that is the greatest prompt | 11:15 |
| *** jedimike has quit IRC | 11:16 | |
| *** annegent_ has joined #openstack-infra | 11:16 | |
| *** dtantsur|lunch is now known as dtantsur | 11:16 | |
| *** dims_ has joined #openstack-infra | 11:19 | |
| *** habib__ has quit IRC | 11:20 | |
| *** srenatus has quit IRC | 11:20 | |
| *** srenatus has joined #openstack-infra | 11:21 | |
| *** annegent_ has quit IRC | 11:21 | |
| *** yaguang has quit IRC | 11:21 | |
| *** dims__ has joined #openstack-infra | 11:22 | |
| sdague | BobBall: it looks like /opt/stack/swift already exists | 11:24 |
| sdague | is that possible? | 11:24 |
| *** dims_ has quit IRC | 11:25 | |
| BobBall | I don't know how... it's a fresh install each time, but I'm digging into it. It failed 4 times with the same error (different hosts) but now seems to be working again | 11:26 |
| BobBall | very bizzare | 11:26 |
| BobBall | why do you think it existed? It seems that line 534 (the [ ! -d $GIT_DEST ]) will only pass if it does not exist? | 11:28 |
| *** michchap has quit IRC | 11:30 | |
| *** michchap has joined #openstack-infra | 11:30 | |
| *** tkelsey has quit IRC | 11:33 | |
| *** yfried_ has joined #openstack-infra | 11:35 | |
| *** yfried has quit IRC | 11:35 | |
| *** gokrokve has joined #openstack-infra | 11:37 | |
| *** doug-fish has joined #openstack-infra | 11:39 | |
| *** gokrokve has quit IRC | 11:42 | |
| *** adrian_otto has joined #openstack-infra | 11:45 | |
| *** pblaho is now known as pblaho|afk | 11:48 | |
| *** ArxCruz has joined #openstack-infra | 11:50 | |
| *** pblaho|afk is now known as pblaho | 11:51 | |
| *** mwagner_lap has quit IRC | 11:53 | |
| *** lcostantino has joined #openstack-infra | 11:53 | |
| *** adrian_otto has quit IRC | 11:54 | |
| *** habib__ has joined #openstack-infra | 11:55 | |
| *** adalbas has joined #openstack-infra | 11:58 | |
| *** flaper87|afk is now known as flaper87 | 11:59 | |
| *** rkukura has quit IRC | 12:05 | |
| *** lcostantino has quit IRC | 12:06 | |
| *** ominakov has joined #openstack-infra | 12:08 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: cloner to easily clone dependent repositories https://review.openstack.org/70373 | 12:08 |
| *** dprince has joined #openstack-infra | 12:12 | |
| *** alexpilotti has joined #openstack-infra | 12:15 | |
| *** annegent_ has joined #openstack-infra | 12:16 | |
| *** weshay has joined #openstack-infra | 12:17 | |
| bookwar | do we have drafts enabled at Gerrit? we get " ! [remote rejected] HEAD -> refs/drafts/master/test_patching (cannot upload drafts)" error for stackforge/fuel-main repo | 12:18 |
| *** dkranz has joined #openstack-infra | 12:18 | |
| *** amitpp has quit IRC | 12:19 | |
| *** amitpp has joined #openstack-infra | 12:19 | |
| *** annegent_ has quit IRC | 12:21 | |
| *** amitpp has quit IRC | 12:22 | |
| *** lcostantino has joined #openstack-infra | 12:23 | |
| anteaya | bookwar: drafts are disabled | 12:24 |
| anteaya | they do evil things when trying to perform operations on patchsets that started out as a draft | 12:24 |
| anteaya | as a way of avoided merge complications, we have disabled drafts | 12:24 |
| *** maxbit has joined #openstack-infra | 12:25 | |
| *** rfolco has joined #openstack-infra | 12:25 | |
| anteaya | submit the patch and mark it work in progress | 12:25 |
| *** Hal_ has joined #openstack-infra | 12:25 | |
| bookwar | anteaya: ok, thanks, maybe you can update this page also https://wiki.openstack.org/wiki/GerritJenkinsGit ? | 12:27 |
| *** rlandy has joined #openstack-infra | 12:27 | |
| *** dkranz has quit IRC | 12:27 | |
| anteaya | I will do so | 12:27 |
| anteaya | thanks bookwar | 12:27 |
| *** yfried_ has quit IRC | 12:29 | |
| *** dims__ has quit IRC | 12:31 | |
| vponomaryov | fungi: Hello, could you please remove obsolete branch from Manilaclient project repo: https://github.com/stackforge/python-manilaclient/tree/cinder-to-manila | 12:31 |
| *** dims__ has joined #openstack-infra | 12:31 | |
| *** basha has quit IRC | 12:32 | |
| *** amalagon has joined #openstack-infra | 12:32 | |
| *** radez_g0n3 is now known as radez | 12:36 | |
| *** pdmars has joined #openstack-infra | 12:37 | |
| *** CaptTofu_ has quit IRC | 12:37 | |
| *** CaptTofu_ has joined #openstack-infra | 12:38 | |
| *** gokrokve has joined #openstack-infra | 12:38 | |
| fungi | vponomaryov: i have removed the cinder-to-manila branch. its last commit was d20a98e6926a3c313830e1e9bd68b6a0505c1ad3 | 12:38 |
| vponomaryov | fungi: Thanks! | 12:38 |
| *** dkranz has joined #openstack-infra | 12:39 | |
| *** rcarrillocruz has joined #openstack-infra | 12:40 | |
| *** CaptTofu_ has quit IRC | 12:42 | |
| *** rcarrill` has quit IRC | 12:42 | |
| *** yfried has joined #openstack-infra | 12:43 | |
| *** gokrokve has quit IRC | 12:43 | |
| *** sweston_ has quit IRC | 12:44 | |
| *** pdmars has quit IRC | 12:44 | |
| *** annegent_ has joined #openstack-infra | 12:45 | |
| *** pdmars has joined #openstack-infra | 12:45 | |
| annegentle | fungi: morning! do you know the ftp server for the docs site is in the jenkins job builder files? I can't find it... | 12:49 |
| annegentle | fungi: nevermind, found it | 12:50 |
| *** eharney has joined #openstack-infra | 12:52 | |
| *** rloo has joined #openstack-infra | 12:52 | |
| *** gokrokve has joined #openstack-infra | 12:53 | |
| *** dstanek is now known as dstanek_zzz | 12:53 | |
| *** annegent_ has quit IRC | 12:54 | |
| *** pblaho is now known as pblaho|afk | 12:55 | |
| *** flaper87 is now known as flaper87|afk | 12:55 | |
| fungi | annegentle: great! easiest question i've answered all morning | 12:55 |
| anteaya | off to a great start | 12:56 |
| *** prad_ has joined #openstack-infra | 12:56 | |
| *** _nadya_ has quit IRC | 12:57 | |
| *** amitpp has joined #openstack-infra | 12:58 | |
| *** matsuhashi has quit IRC | 12:59 | |
| *** tkelsey has joined #openstack-infra | 12:59 | |
| *** matsuhashi has joined #openstack-infra | 12:59 | |
| *** dstanek_zzz is now known as dstanek | 13:00 | |
| *** enikanorov_ has quit IRC | 13:00 | |
| fungi | i guess if anyone sees Ryan_Lane in here today, we should inquire about http://lists.wikimedia.org/pipermail/mediawiki-announce/2014-June/000154.html | 13:00 |
| sdague | fungi: yeh | 13:00 |
| sdague | ok, ES has gone completely off the rails again and not catching up | 13:00 |
| sdague | :( | 13:00 |
| *** enikanorov_ has joined #openstack-infra | 13:00 | |
| sdague | which is especially a bummer because there has been an error rate uptick since trusty roll out, and I expect the new code is expressing a different timing pattern | 13:01 |
| *** pblaho|afk is now known as pblaho | 13:02 | |
| *** doug-fish1 has joined #openstack-infra | 13:02 | |
| *** mugsie has quit IRC | 13:03 | |
| *** doug-fish has quit IRC | 13:03 | |
| *** julim has joined #openstack-infra | 13:03 | |
| anteaya | fungi sdague I just made some changes to this wikipage, do you mind taking a peek so I'm saying the right thing? https://wiki.openstack.org/wiki/GerritJenkinsGit | 13:04 |
| *** yfried has quit IRC | 13:04 | |
| *** matsuhashi has quit IRC | 13:04 | |
| fungi | sdague: elasticsearch-head and bigdesk both report cluster status as green. digging into the detailed metrics now | 13:06 |
| fungi | all 6 es cluster nodes and all 20 ls workers present and accounted for. no shards replicating currently | 13:08 |
| *** CaptTofu_ has joined #openstack-infra | 13:08 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add fingerprint for Dump was interrupted bug https://review.openstack.org/102517 | 13:08 |
| *** matsuhashi has joined #openstack-infra | 13:10 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add fingerprint for Dump was interrupted bug https://review.openstack.org/102517 | 13:11 |
| sdague | fungi: yeh, this is just the disk slow thing | 13:12 |
| sdague | we're 60k events behind | 13:12 |
| sdague | see the status graph in zuul | 13:12 |
| sdague | I think 5k events is typically a linear hour of time delay | 13:12 |
| BobBall | Can someone remove http://status.openstack.org/rechecks/ ? :) I know the link has gone, but I've been there several times and keep forgetting it's obsolete :P | 13:13 |
| fungi | anteaya: lgtm | 13:13 |
| anteaya | fungi: thanks | 13:14 |
| fungi | BobBall: it probably needs a permanent redirect rewrite rule in teh apache config so everyone who bookmarked it gets forwarded to the new hotness | 13:14 |
| fungi | BobBall: somewhere around http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/status.vhost.erb#n18 | 13:15 |
| *** adrian_otto has joined #openstack-infra | 13:15 | |
| *** mikeh has joined #openstack-infra | 13:15 | |
| *** mriedem has joined #openstack-infra | 13:15 | |
| *** mikeh is now known as jedimike | 13:16 | |
| *** mwagner_lap has joined #openstack-infra | 13:16 | |
| BobBall | thanks fungi | 13:16 |
| fungi | or maybe remove those lines and add the rewrite down around line 50 where the elastic-recheck is configured | 13:16 |
| *** pblaho has quit IRC | 13:17 | |
| *** yaguang has joined #openstack-infra | 13:17 | |
| *** salv-orlando_ has joined #openstack-infra | 13:18 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: remove /rechecks https://review.openstack.org/102524 | 13:18 |
| sdague | there it is | 13:18 |
| sdague | fungi: how does that look? | 13:18 |
| sdague | fungi: is our allocator off? | 13:19 |
| *** mkerrin has joined #openstack-infra | 13:20 | |
| sdague | I'm surprised how long er is taking to get nodes | 13:20 |
| sdague | oh, there was a gate reset it looks like | 13:20 |
| sdague | guess it's just recovering | 13:20 |
| *** dstanek is now known as dstanek_zzz | 13:20 | |
| *** salv-orlando has quit IRC | 13:20 | |
| *** salv-orlando_ is now known as salv-orlando | 13:20 | |
| *** ociuhandu has quit IRC | 13:20 | |
| *** dstanek_zzz is now known as dstanek | 13:23 | |
| *** gokrokve has quit IRC | 13:25 | |
| fungi | sdague: based on my reading, it looks like you have to specify permanent or else it will be a temporary 302 redirect instead. i always end up using mod_rewrite instead of mod_alias for redirects, so i had to look up the syntax | 13:26 |
| *** annegent_ has joined #openstack-infra | 13:26 | |
| *** miqui has joined #openstack-infra | 13:29 | |
| *** basha has joined #openstack-infra | 13:29 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add fingerprint for Dump was interrupted bug https://review.openstack.org/102517 | 13:30 |
| *** gokrokve has joined #openstack-infra | 13:30 | |
| *** amitpp has quit IRC | 13:30 | |
| *** ociuhandu has joined #openstack-infra | 13:32 | |
| *** matsuhashi has quit IRC | 13:33 | |
| *** gokrokve has quit IRC | 13:33 | |
| *** matsuhashi has joined #openstack-infra | 13:33 | |
| *** basha has quit IRC | 13:34 | |
| dhellmann | mordred, lifeless, fungi: I'm contemplating a pbr 0.9.0 release this morning to roll out the fix for that dependency chaining issue with distutils monkeypatching. I see 3 failed merges, which I'll babysit today, but is there anything else you'd like me to wait for? | 13:36 |
| *** markmcclain has joined #openstack-infra | 13:36 | |
| *** radez is now known as radez_g0n3 | 13:36 | |
| *** esker has quit IRC | 13:36 | |
| fungi | dhellmann: nothing i'm aware of aside from my fix for bug 1326682 | 13:37 |
| uvirtbot | Launchpad bug 1326682 in pbr "pbr doesn't check if git is not installed and/or not in the PATH" [Undecided,In progress] https://launchpad.net/bugs/1326682 | 13:37 |
| fungi | dhellmann: which you already seem to have spotted | 13:37 |
| dhellmann | fungi: yeah, I'll try to get that to merge first | 13:37 |
| fungi | i'll add to my to-do list to figure out how to get a regression test added for that condition | 13:37 |
| *** markmcclain1 has joined #openstack-infra | 13:37 | |
| *** matsuhashi has quit IRC | 13:38 | |
| fungi | but am fine doing so in a separate patch after the release | 13:38 |
| *** markmcclain1 has quit IRC | 13:38 | |
| dhellmann | fungi: sounds good | 13:38 |
| *** markmcclain has quit IRC | 13:40 | |
| *** rcarrill` has joined #openstack-infra | 13:40 | |
| *** habib__ has quit IRC | 13:40 | |
| fungi | that one was an odd corner case anyway (you got a git repository onto your machine without having the git utility installed?!?) | 13:40 |
| *** zz_gondoi is now known as gondoi | 13:40 | |
| *** habib__ has joined #openstack-infra | 13:41 | |
| *** rcarrillocruz has quit IRC | 13:43 | |
| *** nosnos has quit IRC | 13:43 | |
| *** matsuhashi has joined #openstack-infra | 13:44 | |
| *** dkliban_afk is now known as dkliban | 13:44 | |
| *** matsuhashi has quit IRC | 13:44 | |
| *** matsuhashi has joined #openstack-infra | 13:45 | |
| *** homeless has joined #openstack-infra | 13:45 | |
| *** maxbit has quit IRC | 13:45 | |
| *** joa has joined #openstack-infra | 13:46 | |
| joa | hello | 13:46 |
| anteaya | hello joa what is on your mind? | 13:47 |
| *** rloo has quit IRC | 13:47 | |
| joa | I'm tackling the 3rd-party integration for Cinder (I'm from Scality); I read in the docs that one could (or should?) request a service account for the CI system. I feel like it's mostly to associate a service with an existing accoun | 13:47 |
| joa | account* | 13:47 |
| ryanpetrello | if an approved review has passed the check queue, but failed the gate, is there a way “regate” ? | 13:47 |
| *** rloo has joined #openstack-infra | 13:48 | |
| *** rloo has quit IRC | 13:48 | |
| ryanpetrello | (without running going into the check queue again first) | 13:48 |
| anteaya | reverify bug <bug number> | 13:48 |
| joa | Is it necessary to request this service account or can it be enough to only create a separate account for the CI system ? | 13:48 |
| ryanpetrello | thanks :) | 13:48 |
| anteaya | it will go through check again | 13:48 |
| *** rloo has joined #openstack-infra | 13:48 | |
| ryanpetrello | oh | 13:48 |
| ryanpetrello | okay | 13:48 |
| anteaya | it is to allow the service account to vote verified | 13:49 |
| anteaya | which user accounts can't do | 13:49 |
| ryanpetrello | I wanted to avoid going through check again, because it already pass the gate | 13:49 |
| ryanpetrello | pass the check queue, that is | 13:49 |
| *** lcostantino has quit IRC | 13:49 | |
| *** rloo has quit IRC | 13:49 | |
| anteaya | ryanpetrello: I don't know of a way to do that | 13:49 |
| ryanpetrello | okay | 13:49 |
| ryanpetrello | thanks :) | 13:49 |
| *** rloo has joined #openstack-infra | 13:49 | |
| *** matsuhashi has quit IRC | 13:49 | |
| anteaya | ryanpetrello: every patch has to go through check on the way to merge | 13:49 |
| ryanpetrello | yea, understood | 13:49 |
| ryanpetrello | pecan is suffering from some issues I discussed w/ monty last night, and I basically can’t get patches through :\ | 13:50 |
| anteaya | :( | 13:50 |
| anteaya | that hurst | 13:50 |
| anteaya | hurts | 13:50 |
| ryanpetrello | seems like 3/4 of the time, I get pypi failures | 13:50 |
| anteaya | joa: does that make sense to you? | 13:50 |
| anteaya | eww | 13:50 |
| anteaya | anyone looking at that bug? | 13:50 |
| ryanpetrello | I’ve managed to get a few to pass the check queue, but it seems almost impossible to get them to pass check *and* gate one after another | 13:50 |
| *** mrmartin has quit IRC | 13:51 | |
| anteaya | joa: so send an email to openstack-infra@lists.openstack.org and the name of your account can be Scality CI and username scality-ci | 13:52 |
| joa | anteaya: yeah that makes sense; so I shall submit my request :) | 13:52 |
| anteaya | joa: great | 13:52 |
| joa | just created the account, currently reading all the 3rd-party docs | 13:53 |
| anteaya | joa: do you expect you will need more than one gerrit ci account for scality at any point in the future? | 13:53 |
| *** gokrokve has joined #openstack-infra | 13:53 | |
| *** yfried has joined #openstack-infra | 13:53 | |
| anteaya | joa: uhhhh you mean requested the account? | 13:53 |
| anteaya | you can't create ci accounts yourself | 13:53 |
| joa | anteaya: Don't think so, honestly; if one account can run the tests for multiple projects it's enough | 13:53 |
| fungi | ryanpetrello: i assume pecan isn't using the openstack mirror | 13:53 |
| anteaya | okay great | 13:53 |
| joa | anteaya: I meant created launchpad/review account; but should I delete/remove it instead ? | 13:54 |
| *** yfried has quit IRC | 13:54 | |
| *** yfried has joined #openstack-infra | 13:54 | |
| fungi | ryanpetrello: yeah, not in http://git.openstack.org/cgit/openstack/requirements/tree/projects.txt | 13:54 |
| joa | I thought of the request as an "upgrade" of the review account, I might be mistaken ? | 13:54 |
| anteaya | joa: you need to send an email to openstack-infra@lists.openstack.org | 13:54 |
| *** rloo has quit IRC | 13:55 | |
| anteaya | joa: can I ask what you are reading that gave you that idea? | 13:55 |
| *** nati_ueno has joined #openstack-infra | 13:55 | |
| *** rloo has joined #openstack-infra | 13:55 | |
| *** habib__ has quit IRC | 13:55 | |
| joa | anteaya: http://ci.openstack.org/third_party.html#requesting-a-service-account ; I was pointed to it by http://www.joinfu.com/2014/02/setting-up-an-external-openstack-testing-system/comment-page-1/#comment-216394 | 13:56 |
| anteaya | right | 13:56 |
| fungi | ryanpetrello: interestingly, since pecan *is* in openstack's list of global requirements, all its current (released) requirements should already be met in our mirror because those become transitive deps of openstack projects anyway | 13:56 |
| anteaya | what line says that this is an upgrade to a current gerrit account? | 13:56 |
| *** wenlock has joined #openstack-infra | 13:56 | |
| ryanpetrello | fungi: so the majority of my failures are actually tests where I’m gating against *other* projects | 13:56 |
| anteaya | joa: I need to fix that line if that is how you read it | 13:56 |
| joa | no line says it, but no line says it isn't either, kind of blurry, if you see what I mean ? | 13:56 |
| *** wenlock has quit IRC | 13:56 | |
| *** wenlock has joined #openstack-infra | 13:56 | |
| ryanpetrello | e.g., I have jobs where I download the source of storyboard, create a tox -epy27 from *its* source, install pecan-dev there, and run the tests for storyboard using tox | 13:57 |
| fungi | ryanpetrello: oh. good point, to gate on those they'd also need to have their dependencies met in our mirror | 13:57 |
| ryanpetrello | and pip installs for that tox are failing | 13:57 |
| joa | anteaya: let me re-read it, double checking... | 13:57 |
| ryanpetrello | so the failures I’m seeing are pretty much always these types of jobs | 13:57 |
| joa | anteaya: "to arrange setting up a dedicated user (so your system can post reviews and vote using a system name rather than your user name)." | 13:57 |
| ryanpetrello | i.e., I don’t think I’ve seen my py26-33 jobs fail at all | 13:58 |
| anteaya | joa: ah dedicated user | 13:58 |
| joa | To me it felt like: You could do it the other way too. It's interpretation, but nothing says it isn't | 13:58 |
| anteaya | what do you mean the other way? | 13:58 |
| fungi | ryanpetrello: according to dstufft you should never get download failures from pypi.python.org ;) though in this case it's more likely broken nats/proxies in our cloud providers | 13:58 |
| joa | that you could use your own username :) | 13:58 |
| anteaya | what two ways do you think it is saying | 13:58 |
| ryanpetrello | fungi: another thing I noticed - I tested a bit last night, and couldn’t find a pip failure anywhere outside of hp cloud | 13:58 |
| joa | and not have a dedicated account | 13:59 |
| joa | s/account/user | 13:59 |
| anteaya | joa: can you suggest how I can change the document so it is clearer | 13:59 |
| anteaya | since there is no way other than having a dedicated ci account for third party ci | 13:59 |
| joa | I'd explicit the requirement by writing along the lines of "In order to post reviews and vote using a third-party CI system, you will need a dedicated account" | 14:00 |
| *** akrivoka has left #openstack-infra | 14:00 | |
| anteaya | joa: I will submit a patch to that effect, thank you | 14:00 |
| fungi | ryanpetrello: not at all surprising. i think they oversubscribe some of their network hardware. it seems to mostly just impact http/https which makes me wonder if it's a "web accelerator" proxy inline with their tenant network border | 14:00 |
| joa | anteaya: maybe also adding that this dedicated account should not be created manually ? | 14:00 |
| *** lcostantino has joined #openstack-infra | 14:00 | |
| anteaya | joa: also please review: https://review.openstack.org/#/c/101013/ and https://review.openstack.org/#/c/101227/ | 14:00 |
| anteaya | joa: can not be | 14:01 |
| anteaya | but that can be explict | 14:01 |
| joa | well, what will happen if I created a matching user account ? Will it conflict ? | 14:01 |
| anteaya | if you use the same email it will conflict | 14:02 |
| *** timrc-afk is now known as timrc | 14:02 | |
| anteaya | gerrit needs unique emails | 14:02 |
| joa | okay so I need to delete it first :o should have asked first :D | 14:02 |
| *** hashar has quit IRC | 14:03 | |
| anteaya | joa: you can't | 14:05 |
| anteaya | you can't delete a gerrit account | 14:05 |
| joa | well, seems like I'm in a bind then ? :D | 14:05 |
| anteaya | we have to do it for you | 14:05 |
| anteaya | stay calm | 14:05 |
| joa | ahah I'm not worrying too much, you are here to help me | 14:05 |
| anteaya | what is the user id and username associated with the gerrit account you would like deleted | 14:05 |
| anteaya | yes, I am trying | 14:05 |
| joa | username scal-ci | 14:06 |
| *** ildikov has quit IRC | 14:06 | |
| anteaya | and does that user account have any history, like reviewing patches or submitting any patches? | 14:06 |
| joa | nope, it was just created before I came here | 14:06 |
| anteaya | joa: go to the settings page for that account | 14:06 |
| anteaya | what is the user id for that account please | 14:06 |
| joa | account id: 12123 | 14:07 |
| anteaya | thanks | 14:07 |
| anteaya | fungi when next you can query the gerrit db, account id: 12123 username: scal-ci with no history needs to be deleted | 14:07 |
| *** wenlock has quit IRC | 14:07 | |
| anteaya | joa: sign out of the account and don't sign back into it, please | 14:08 |
| joa | okay | 14:08 |
| *** Hal_ has quit IRC | 14:08 | |
| fungi | anteaya: gerrit doesn't have a way to delete accounts, though i can blank out the e-mail address and other external ids for it | 14:08 |
| sdague | ryanpetrello: do you have a representative error message from the console for the pip failure? | 14:09 |
| *** rloo has quit IRC | 14:09 | |
| fungi | but yes, that does require manually running update/delete queries in the mysql backend for it | 14:09 |
| * joa feels sorry about that. | 14:09 | |
| *** rloo has joined #openstack-infra | 14:09 | |
| ryanpetrello | sdague: one second, let me grab a few | 14:09 |
| fungi | joa: it's far from the first... don't sweat it ;) | 14:09 |
| joa | well, I guess so | 14:10 |
| mordred | dhellmann, fungi: new pbr works for me | 14:10 |
| joa | at least it takes part in improving 3rd-party doc \o/ | 14:10 |
| *** rgerganov has quit IRC | 14:11 | |
| *** signed8bit has joined #openstack-infra | 14:11 | |
| ryanpetrello | sdague: https://gist.github.com/ryanpetrello/167dbda2dbb4740999ef | 14:11 |
| ryanpetrello | all seem to be HP Cloud | 14:12 |
| *** rgerganov has joined #openstack-infra | 14:12 | |
| *** otherwiseguy has joined #openstack-infra | 14:12 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Point search requests to search endpoints https://review.openstack.org/102536 | 14:12 |
| sdague | ryanpetrello: are we logstashing those runs? | 14:13 |
| ryanpetrello | no idea; is that something I can adjust in Jenkins config? | 14:13 |
| joa | anteaya, fungi: So in my ci account request, should I tell that there is a pending delete for the conflicting user account ? | 14:14 |
| ryanpetrello | talked to dstufft last night and he instructed me to make pip verbose: https://review.openstack.org/#/c/102418/ | 14:14 |
| sdague | ryanpetrello: it's something you need to do config/ | 14:15 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcImVycm9yOiBbRXJybm8gMTA0XSBDb25uZWN0aW9uIHJlc2V0IGJ5IHBlZXJcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwMzcwNTY2OTI3MywibW9kZSI6InNjb3JlIiwiYW5hbHl6ZV9maWVsZCI6ImJ1aWxkX25vZGUifQ== | 14:15 |
| sdague | so that's the last 48hrs of results | 14:15 |
| *** rloo has quit IRC | 14:15 | |
| sdague | it does look like it's more often hp cloud than rax | 14:15 |
| *** rloo has joined #openstack-infra | 14:15 | |
| ryanpetrello | it seems like we see two sorts of pip failures, “error: [Errno 104] Connection reset by peer” and “distutils.errors.DistutilsError: Can't download <url>: 503 Service Unavailable” | 14:15 |
| ryanpetrello | ” | 14:15 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Clarifies that CI accounts must request accounts from infra https://review.openstack.org/102538 | 14:15 |
| fungi | sdague: though worth noting we have a lot more hpcloud nodes than rax | 14:15 |
| sdague | fungi: any idea what the mix ratio is now? | 14:16 |
| fungi | sdague: just a sec and i'll get you a breakdown | 14:16 |
| *** nati_ueno has quit IRC | 14:16 | |
| *** rloo has quit IRC | 14:16 | |
| *** rloo has joined #openstack-infra | 14:16 | |
| anteaya | joa: you can if you wish | 14:16 |
| sdague | ryanpetrello: we definitely aren't doing storyboard in logstash it seems | 14:17 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcImVycm9yOiBbRXJybm8gMTA0XSBDb25uZWN0aW9uIHJlc2V0IGJ5IHBlZXJcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwMzcwNTU3MDk3MywibW9kZSI6InNjb3JlIiwiYW5hbHl6ZV9maWVsZCI6InByb2plY3QifQ== | 14:17 |
| joa | anteaya: okay ! thanks for the trouble :) | 14:17 |
| anteaya | joa: reference that fungi (Jeremy Stanley) has it on his todo list | 14:17 |
| joa | thanks/sorry | 14:17 |
| joa | okay | 14:17 |
| anteaya | joa: well at least you are talking to us | 14:17 |
| anteaya | joa: so that is a good place | 14:17 |
| anteaya | joa: can you review the patch I put up? | 14:17 |
| joa | yup I was looking at those | 14:17 |
| anteaya | https://review.openstack.org/102538 | 14:18 |
| anteaya | that is the one that addresses your concern | 14:18 |
| anteaya | or tries to | 14:18 |
| anteaya | it will conflict with one that is already up *sigh* | 14:18 |
| joa | not exactly sure how to review (comment format as I guessed), so reading the associated doc | 14:18 |
| anteaya | but I will figure that out | 14:18 |
| joa | oh :/ | 14:18 |
| anteaya | it is just there is a backlog of patches | 14:18 |
| anteaya | the changes need to go in | 14:18 |
| anteaya | joa: you haven't reviewed before? | 14:19 |
| anteaya | joa: let me get you something to read | 14:19 |
| *** maxbit has joined #openstack-infra | 14:19 | |
| anteaya | joa: try this: http://anteaya.info/blog/2013/03/21/reviewing-an-openstack-patch/ | 14:19 |
| fungi | sdague: http://paste.openstack.org/show/84874/ (roughly 2:1) | 14:19 |
| anteaya | joa: ask if you have any questions | 14:19 |
| joa | anteaya: well, I'm currently working on the side, planning to get active soon, but I didn't take time for that yet, no | 14:19 |
| anteaya | joa: it is a great place to learn | 14:20 |
| sdague | fungi: cool | 14:20 |
| anteaya | you learn more from doing reviews than from submitting patches | 14:20 |
| joa | anteaya: that's why I love active IRC channels :) | 14:20 |
| anteaya | good | 14:20 |
| anteaya | you'll fit in well | 14:20 |
| *** sileht has quit IRC | 14:20 | |
| sdague | yeh, honestly, that's on my todo list to actually build all the ES stuff to be in consumable percentages | 14:20 |
| *** annegent_ has quit IRC | 14:21 | |
| fungi | so if we're seeing these sorts of failures twice as often in hpcloud compared to rax, then that's just due to the node distribution weighting them | 14:21 |
| *** annegent_ has joined #openstack-infra | 14:21 | |
| fungi | if we're seeing them 10 times more often there, then it's something else | 14:22 |
| *** roeyc has quit IRC | 14:22 | |
| *** yaguang has quit IRC | 14:22 | |
| dstufft | so | 14:23 |
| dstufft | Connection Reset by Peer is a thing that's happening lately | 14:23 |
| dstufft | not openstack specific | 14:23 |
| *** sileht has joined #openstack-infra | 14:23 | |
| dstufft | I've been working withFastly to try and nail down why it occurs | 14:23 |
| dstufft | fungi: ryanpetrello ^ | 14:23 |
| *** Hal_ has joined #openstack-infra | 14:24 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/config: Only enabling rsyslog service if not chroot https://review.openstack.org/101562 | 14:25 |
| joa | anteaya: do I need to add myself as reviewer before I can vote -1/0/+1 ? that's the part I can't figure out (or missed the piece in the docs) | 14:26 |
| joa | mhh no scratch that | 14:26 |
| joa | found the right button. | 14:26 |
| *** craigbr has joined #openstack-infra | 14:26 | |
| anteaya | joa: great | 14:26 |
| ryanpetrello | okay, looks like I managed to get a pecan change through the gate queue just now; maybe I was just having a run of bad luck when I was trying last night | 14:27 |
| anteaya | hey craigbr | 14:27 |
| craigbr | hi anteaya | 14:27 |
| anteaya | craigbr: how is the day treating you so far? | 14:27 |
| sdague | ryanpetrello: there are times when things blip as well | 14:27 |
| ryanpetrello | yep | 14:27 |
| craigbr | anteaya: trying to recover everything. We got hammered by lightening last night | 14:28 |
| anteaya | craigbr: nooo | 14:28 |
| dstufft | ryanpetrello: last night you might have been hit by glusterfs | 14:28 |
| *** IvanBerezovskiy has left #openstack-infra | 14:28 | |
| anteaya | craigbr: at home or at work? | 14:28 |
| anteaya | craigbr: or both? | 14:28 |
| craigbr | at work. | 14:28 |
| anteaya | craigbr: noooo | 14:28 |
| craigbr | at home, I meant | 14:28 |
| anteaya | ah | 14:28 |
| anteaya | still | 14:28 |
| craigbr | I usually work at home on wednesdays | 14:28 |
| anteaya | that is aweful | 14:28 |
| anteaya | yep | 14:28 |
| ryanpetrello | dstufft: sounds like something I should get some medication for | 14:28 |
| anteaya | my neighbours tree got hit a few years ago | 14:29 |
| anteaya | took out their tv and a bunch of other things | 14:29 |
| craigbr | It just took out our power multiple times for hours. Part of life where we live. At least the garden didn't get destroyed by hail like some of our friends | 14:29 |
| anteaya | wow | 14:29 |
| anteaya | so you are in cleanup and on the telephone with insurance mode | 14:29 |
| anteaya | got it | 14:29 |
| dstufft | ryanpetrello: PyPI uses gluterfs on two Rackspace VMs to back PyPI... but glusterfs gets real cranky if the IOPS are inconsistent so every so often gluterfs just completely craps itself and blocks for a long time | 14:30 |
| *** ildikov has joined #openstack-infra | 14:30 | |
| craigbr | No, not quite that bad. It just crashed the mac, our power, the router, etc. But, I think I have it all cleaned up now | 14:30 |
| *** james_li has joined #openstack-infra | 14:30 | |
| mordred | dstufft: wow. glusterfs sounds great | 14:31 |
| dstufft | mordred: I don't like it | 14:31 |
| anteaya | craigbr: wow | 14:31 |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482 | 14:31 |
| dstufft | I want to switch to an object store | 14:31 |
| dstufft | but I haven't had time to do it yet | 14:31 |
| mordred | nod | 14:31 |
| craigbr | anteaya: Should we switch to @openstack-dev? Do you have some more time? | 14:31 |
| anteaya | sure let's go there | 14:32 |
| fungi | dstufft: ryanpetrello: sdague: given that the pypi cdn is geo-distribited, we might see higher incidence from hpcloud simply by virtue of it being "closer" to one or more problem nodes in fastly's farm | 14:32 |
| fungi | dstufft: ryanpetrello: sdague: one thing we could try is to configure our jobs to do an initial dns lookup for pypi.python.org and then hard-code that into /etc/hosts (to prevent it resolving to something else over the course of the job) and report what we're using. then afterward we could run a correlation on the failures to see if it maps back to specific fastly nodes | 14:33 |
| *** wenlock has joined #openstack-infra | 14:33 | |
| dstufft | fungi: how hard would it be to use a custom pip | 14:34 |
| *** dstanek is now known as dstanek_zzz | 14:34 | |
| mordred | depends on which jobs | 14:34 |
| dstufft | I have a patch I can apply to pip that'll print the IP address it's actually connecting too | 14:34 |
| fungi | yeah, we'd basically override the get-pip machinery to do that | 14:35 |
| mordred | it would be pretty hard to use a custom pip inside of the tox unittest jobs | 14:35 |
| lifeless | PIIIIP | 14:35 |
| mordred | look, it's a lifeless | 14:35 |
| fungi | oh, right, pip vendored into those venvs | 14:35 |
| fungi | dstufft: how hard would it be to release a new pip with a command line option or envvar to make it report that for us? | 14:35 |
| *** yaguang has joined #openstack-infra | 14:36 | |
| sdague | fungi: the dns should remain stable over the course of the run, right? | 14:36 |
| fungi | sdague: no guarantees | 14:36 |
| fungi | sdague: totally depends on the record ttls | 14:36 |
| dstufft | fungi: not super duper hard, the major thing is it's actually patch against urllib3 so I have to put it in there and then get requests to pull it down, and then get requests to do a new release :| | 14:36 |
| dstufft | FWIW Fastly returns multiple A records | 14:37 |
| dstufft | and it's geo dns | 14:37 |
| dstufft | I can probably go get the patch into urllib3 today though | 14:37 |
| fungi | dstufft: okay, i'd classify that as "hard" in the spectrum of "only modifying pip" vs "dependencies need patching" | 14:37 |
| sdague | oh, this is for jobs not hitting our mirrors | 14:37 |
| mordred | yah | 14:37 |
| fungi | sdague: right | 14:37 |
| joa | anteaya: reviewed your patches :) great clarifications btw | 14:38 |
| *** yaguang has quit IRC | 14:38 | |
| anteaya | joa: great thank you | 14:38 |
| mordred | which is why I think the real answer for us is to add bandersnatch mirrors to each cloud region | 14:38 |
| fungi | sdague: for finding out whether there are problem fastly cdn nodes in the pypi.python.org farm so we can report that back to the pypi maintainers | 14:38 |
| anteaya | in -dev right now I am working with craigbr who is learning how to review a certain kind of file in infra patches | 14:38 |
| mordred | because we've learned consistently that anything that hits the internets is going to fail at some point | 14:38 |
| anteaya | joa: you are welcome to join us | 14:38 |
| *** _nadya_ has joined #openstack-infra | 14:38 | |
| joa | anteaya: openstack-dev ? | 14:39 |
| sdague | mordred: right | 14:39 |
| dstufft | the good news is | 14:39 |
| *** radez_g0n3 is now known as radez | 14:39 | |
| dstufft | pip 1.6 should make this way less frequent | 14:39 |
| fungi | oh? | 14:39 |
| mordred | ++ | 14:39 |
| dstufft | since it does automatic retries of connection errors by default | 14:39 |
| fungi | dstufft: nice! | 14:39 |
| openstackgerrit | Pablo Andres Fuente proposed a change to openstack-infra/config: Adding Tempest tests to Blazar Gate Job https://review.openstack.org/102303 | 14:39 |
| dstufft | (up to 3 times per connection) | 14:39 |
| dstufft | if I have time before we actually release 1.6, it'll also retry on 503 errors | 14:39 |
| dstufft | but that needs work still | 14:39 |
| *** sabeen has joined #openstack-infra | 14:40 | |
| *** skolekonov has quit IRC | 14:40 | |
| anteaya | joa: yes #openstack-dev | 14:40 |
| joa | thanks, joined :) | 14:40 |
| dstufft | (I have no idea if 3 is good default, the personw ho made the PR picked 3 :V) | 14:40 |
| *** morganfainberg_Z is now known as morganfainberg | 14:42 | |
| sdague | dstufft: it would be nice if it was configurable | 14:42 |
| dstufft | sdague: it is | 14:42 |
| dstufft | just defaults to 3 | 14:43 |
| sdague | dstufft: ok, cool | 14:43 |
| dstufft | --retries 1000 | 14:43 |
| dstufft | or whatever | 14:43 |
| dstufft | I think its --retries | 14:43 |
| *** zhiyan is now known as zhiyan_ | 14:43 | |
| *** esker has joined #openstack-infra | 14:43 | |
| *** rloo has quit IRC | 14:44 | |
| *** rloo has joined #openstack-infra | 14:45 | |
| *** dstanek_zzz is now known as dstanek | 14:45 | |
| *** rcarrillocruz has joined #openstack-infra | 14:45 | |
| *** rcarrill` has quit IRC | 14:48 | |
| *** markmcclain has joined #openstack-infra | 14:48 | |
| *** sweston_ has joined #openstack-infra | 14:48 | |
| *** gokrokve has quit IRC | 14:49 | |
| *** rwsu has joined #openstack-infra | 14:49 | |
| *** gokrokve has joined #openstack-infra | 14:49 | |
| *** sweston_ has quit IRC | 14:50 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Remove old queries for fixed bugs https://review.openstack.org/102547 | 14:52 |
| *** oomichi has quit IRC | 14:52 | |
| *** rloo has quit IRC | 14:52 | |
| *** rloo has joined #openstack-infra | 14:53 | |
| dhellmann | is the "a2ensite pypi" failures in check-requirements-integration-dsvm a known issue? http://logs.openstack.org/73/98273/1/check/check-requirements-integration-dsvm/8a5fe9b/console.html | 14:53 |
| dhellmann | s/is/are | 14:53 |
| * dhellmann grammars better post-tea | 14:53 | |
| *** gokrokve has quit IRC | 14:53 | |
| *** rloo has quit IRC | 14:54 | |
| *** devkulkarni has joined #openstack-infra | 14:55 | |
| *** rloo has joined #openstack-infra | 14:55 | |
| *** rloo has quit IRC | 14:55 | |
| *** rloo has joined #openstack-infra | 14:56 | |
| *** david-lyle has joined #openstack-infra | 14:57 | |
| *** yaguang has joined #openstack-infra | 14:57 | |
| *** rloo has quit IRC | 14:57 | |
| *** rloo has joined #openstack-infra | 14:57 | |
| sdague | dhellmann: if only we had a system for asking that question :) | 14:59 |
| *** mwagner_lap has quit IRC | 15:00 | |
| openstackgerrit | enikanorov proposed a change to openstack-infra/elastic-recheck: Add query for bug #1332500 https://review.openstack.org/102551 | 15:00 |
| uvirtbot | Launchpad bug 1332500 in neutron "Lock wait timeout inserting row into routers table" [High,Confirmed] https://launchpad.net/bugs/1332500 | 15:00 |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/config: Only enabling rsyslog service if not chroot https://review.openstack.org/101562 | 15:01 |
| *** rloo_ has joined #openstack-infra | 15:03 | |
| *** rloo has quit IRC | 15:04 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: remove /rechecks https://review.openstack.org/102524 | 15:04 |
| sdague | fungi: should be fixed | 15:04 |
| sdague | dhellmann: realistically, I think a2ensite pypi is a bug in the tests | 15:05 |
| *** dkranz has quit IRC | 15:05 | |
| sdague | which will have to be fixed to work on trusty | 15:05 |
| dhellmann | sdague: logstash isn't showing me any hits for "Site pypi does not exist" but I have lots of jobs failing with that error. I assumed it was a (recent) bug introduced by an infra change, so I thought I'd ask if someone was already working on it before I spent a bunch of time debugging. | 15:05 |
| sdague | dhellmann: well ES is backed up, so we're at least 12 hrs behind | 15:06 |
| *** rloo_ has quit IRC | 15:06 | |
| *** rloo has joined #openstack-infra | 15:06 | |
| dhellmann | sdague: ok, I figured that was the case | 15:07 |
| sdague | the crux of it is that apache 2.4 requires that conf files end in .conf | 15:07 |
| * dhellmann loves upgrades | 15:07 | |
| sdague | http://logs.openstack.org/73/98273/1/check/check-requirements-integration-dsvm/8a5fe9b/console.html#_2014-06-25_13_52_46_732 needs to include the .conf | 15:08 |
| fungi | oh, yeah, i just upgraded my debian servers to testing and got bitten by that with all my vhosts | 15:08 |
| sdague | we've had those work around in devstack for a while, but this appears to do a bunch of that itself | 15:08 |
| fungi | took me a while to realize the reason it wasn't serving *anything* besides the default site was because none of my vhost snippets were in files ending in .conf | 15:09 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for heat bug 1333556 https://review.openstack.org/102554 | 15:09 |
| *** rloo has quit IRC | 15:09 | |
| uvirtbot | Launchpad bug 1333556 in heat "SQLAlchemy >= 0.9.5 breaks tests" [Critical,Fix committed] https://launchpad.net/bugs/1333556 | 15:09 |
| *** adrian_otto has quit IRC | 15:09 | |
| *** rloo has joined #openstack-infra | 15:09 | |
| fungi | i used to just name the files exactly the same as the (sub)domains they were serving. added .conf to the end of all of them and *poof* | 15:09 |
| fungi | well, not so much *poof* as *moved on to address other 2.2->2.4 upgrade issues in my apache configs) | 15:10 |
| sdague | fungi: the access rules change is awesome as well | 15:10 |
| fungi | yup | 15:10 |
| fungi | at least that one was well marked in apache's upgrade notes | 15:10 |
| fungi | the config file thing i think is a debianism, so apache hadn't documented that as a potential upgrade gotcha | 15:11 |
| sdague | mriedem: instead of building new slqa 0.9.5 bugs, can we just consolidate them? | 15:11 |
| sdague | I was just adding new queries to the one breaks the world bu | 15:11 |
| sdague | bug | 15:11 |
| *** dangers_away is now known as dangers | 15:12 | |
| vponomaryov | mordred, SergeyLukjanov, jeblair: hello, need your assistance on review of little commit: https://review.openstack.org/#/c/97747/ getting this change upstream would be very helpful for testing purposes of Manila project. | 15:13 |
| *** mwagner_lap has joined #openstack-infra | 15:16 | |
| *** dkranz has joined #openstack-infra | 15:18 | |
| mriedem | sdague: http://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries/1333410.yaml | 15:20 |
| mriedem | yeah | 15:20 |
| mriedem | sdague: although it is a separate heat bug | 15:21 |
| mriedem | but sure | 15:21 |
| sdague | mriedem: I just marked it duplicate | 15:21 |
| openstackgerrit | enikanorov proposed a change to openstack-infra/elastic-recheck: Add query for bug #1332500 https://review.openstack.org/102551 | 15:21 |
| uvirtbot | Launchpad bug 1332500 in neutron "Lock wait timeout inserting row into routers table" [High,Confirmed] https://launchpad.net/bugs/1332500 | 15:21 |
| sdague | it's still just a 0.9.5 fail right? | 15:21 |
| mriedem | sdague: well looks like that triggered something bad in the heat code | 15:21 |
| mriedem | so sqla 0.9.5 was the trigger | 15:21 |
| sdague | different from the bad in neutron and nova? | 15:22 |
| mriedem | i think so | 15:22 |
| mriedem | nova/neutron had an issue where results weren't returned for queries | 15:22 |
| mriedem | heat has an issue with setdefault | 15:22 |
| mriedem | resulting in NoneType errors | 15:22 |
| *** ihrachyshka has quit IRC | 15:23 | |
| *** rcarrill` has joined #openstack-infra | 15:24 | |
| *** e0ne_ has quit IRC | 15:25 | |
| dhellmann | sdague, fungi : I opened https://bugs.launchpad.net/openstack-ci/+bug/1334326 about the a2nesite thing. I can work on a patch if someone isn't already on it. | 15:25 |
| uvirtbot | Launchpad bug 1334326 in openstack-ci "check-requirements-integration-dsvm failing with "Site pypi does not exist"" [Undecided,New] | 15:25 |
| fungi | dhellmann: not aware of anyone tackling that one yet, so i say go for it | 15:25 |
| dhellmann | ok | 15:25 |
| *** gokrokve has joined #openstack-infra | 15:26 | |
| *** rcarrillocruz has quit IRC | 15:26 | |
| fungi | mriedem: are you still seeing that in heat tests with 0.9.6? | 15:26 |
| *** gokrokve has quit IRC | 15:26 | |
| fungi | mriedem: or did it clear up when that release landed in our mirror? | 15:26 |
| mriedem | fungi: the heat team already fixed the issue in their code | 15:26 |
| fungi | oh | 15:27 |
| fungi | got it | 15:27 |
| *** mfer has joined #openstack-infra | 15:27 | |
| fungi | yeah, because the 0.9.6 release notes didn't mention anything besides just the one regression fix | 15:27 |
| *** andreaf_ has joined #openstack-infra | 15:27 | |
| therve | sdague, I saw you closed bug #1333556 as a duplicate of bug #1333410 but 0.9.6 doesn't work for heat either | 15:27 |
| uvirtbot | Launchpad bug 1333556 in heat "SQLAlchemy >= 0.9.5 breaks tests" [Critical,Fix committed] https://launchpad.net/bugs/1333556 | 15:27 |
| uvirtbot | Launchpad bug 1333410 in nova "sqla 0.9.5 breaks the world" [Critical,In progress] https://launchpad.net/bugs/1333410 | 15:27 |
| mriedem | fungi: i'm thinking it's not a bug in sqla, it was bad code in heat triggered by a change in sqla | 15:27 |
| fungi | mriedem: agreed | 15:28 |
| sdague | therve: ok, sorry, I thought it was the same bug | 15:28 |
| mriedem | therve: i removed the dupe | 15:28 |
| therve | mriedem, Ah thanks | 15:28 |
| sdague | therve: feel free to undup | 15:28 |
| mriedem | sdague: and re-opened my e-r query | 15:28 |
| openstackgerrit | enikanorov proposed a change to openstack-infra/elastic-recheck: Add query for neutron bug 1334109 https://review.openstack.org/102558 | 15:28 |
| uvirtbot | Launchpad bug 1334109 in neutron "Lock wait timeout updating router's gw_port_id" [High,New] https://launchpad.net/bugs/1334109 | 15:28 |
| *** andreaf_ has quit IRC | 15:28 | |
| mriedem | enikanorov: remove the \r in that yaml file please | 15:29 |
| sdague | mriedem: as you are someone that probably is more strict on reviewing the queries than most, you have time to write down your basic review criteria, and I'll figure out if I can test encode it sanely | 15:30 |
| *** gyee has joined #openstack-infra | 15:31 | |
| fungi | weird. i reached a bounce score of 5 for the -dev ml back on the 18th. the last message to bump me up was when my mailserver thought http://lists.openstack.org/pipermail/openstack-dev/2014-June/038104.html matched clamav's signature for Worm.Mydoom-27 | 15:32 |
| fungi | i see several other recipients also bounced that, according to the mailman logs | 15:32 |
| *** reed has joined #openstack-infra | 15:33 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Clarifies that CI accounts must request accounts from infra https://review.openstack.org/102538 | 15:34 |
| mriedem | sdague: sure....email? | 15:34 |
| mriedem | sdague: you're looking to add unit tests for what we review? | 15:35 |
| zaro | morning | 15:35 |
| *** adrian_otto has joined #openstack-infra | 15:35 | |
| sdague | mriedem: yeh | 15:35 |
| anteaya | morning zaro | 15:36 |
| anteaya | thanks fungi | 15:36 |
| sdague | also trying to get a little consistency in the queries so that when we have to refactor them, it isn't massively bonkers | 15:36 |
| sdague | mriedem: or an etherpad | 15:36 |
| * anteaya votes for an etherpad | 15:37 | |
| mriedem | oh right etherpads are things | 15:38 |
| fungi | zaro: what was it you were suggesting for granting stackforge projects the ability to delete branches? mattoliverau tested just adding create permissions on refs/* but that didn't seem to add the delete button. clarkb suggested it may also require push --force on refs/heads/* (which obviously is something we'd rather not grant), though that might have just been for deleting via the git cli | 15:38 |
| mriedem | sdague: well i probably wouldn't have much more than what we put into the readme last week http://docs.openstack.org/infra/elastic-recheck/readme.html#queries | 15:39 |
| *** annegent_ has quit IRC | 15:40 | |
| *** rloo has quit IRC | 15:40 | |
| openstackgerrit | enikanorov proposed a change to openstack-infra/elastic-recheck: Add query for neutron bug 1334109 https://review.openstack.org/102558 | 15:40 |
| uvirtbot | Launchpad bug 1334109 in neutron "Lock wait timeout updating router's gw_port_id" [High,New] https://launchpad.net/bugs/1334109 | 15:40 |
| mriedem | no wildcards, use tags, etc | 15:40 |
| zaro | fungi: don't remember now, give me few minsy to refresh my memory | 15:40 |
| mriedem | sdague: i think the best thing we could have for testing is actually running the query against -es | 15:40 |
| *** _nadya_ has quit IRC | 15:40 | |
| mriedem | sdague: and if it doesn't have hits or it has successful jobs in hits, we -1 | 15:40 |
| mriedem | that's always the first thing i do, check the query in logstash | 15:41 |
| *** rloo has joined #openstack-infra | 15:41 | |
| openstackgerrit | enikanorov proposed a change to openstack-infra/elastic-recheck: Add query for bug #1332500 https://review.openstack.org/102551 | 15:41 |
| sdague | mriedem: yeh, let me ponder. Because in the unit tests we don't get the patch diff | 15:41 |
| uvirtbot | Launchpad bug 1332500 in neutron "Lock wait timeout inserting row into routers table" [High,Confirmed] https://launchpad.net/bugs/1332500 | 15:41 |
| mriedem | sdague: we'd need some way to tell the tests to not fail on successful job hits though since we have some of those by design, like libvirt and neutron db lock fails | 15:41 |
| *** rloo has quit IRC | 15:41 | |
| sdague | mriedem: right | 15:42 |
| *** rloo has joined #openstack-infra | 15:42 | |
| *** Sukhdev has joined #openstack-infra | 15:42 | |
| mriedem | sdague: seems like we'd need a new type of jenkins job for that kind of testing | 15:42 |
| sdague | yeh, I'm going to noodle on this for a bit. I'll try to propose something this afternoon | 15:42 |
| *** mmaglana has joined #openstack-infra | 15:42 | |
| sdague | we could use absolute limits | 15:42 |
| mriedem | checking for wildcards and tags vs filenames in unit tests is easy | 15:42 |
| mriedem | sdague: sure, percentages | 15:43 |
| jeblair | sdague: current thoughts on 88289 ? | 15:43 |
| sdague | like 60% of everything needs to return true | 15:43 |
| *** annegent_ has joined #openstack-infra | 15:43 | |
| *** annegent_ has quit IRC | 15:43 | |
| sdague | jeblair: I think I'd like to see what the failure rate looks like after the trusty bump. | 15:44 |
| sdague | which is currently hard because we're at least 12 hrs behind in ES | 15:44 |
| sdague | so we have pretty minimal seed data in there that's trusty | 15:44 |
| mtreinish | sdague: what about jogo's graphite thing? | 15:44 |
| jeblair | sdague: is it currently nv or experimental? | 15:44 |
| jeblair | sdague: can you leave a comment to that effect in the review? | 15:45 |
| sdague | it's nv | 15:45 |
| sdague | sure | 15:45 |
| mordred | jeblair, fungi: I think yolanda's work with adding dib to nodepool is about ready - I think it's working for her locally | 15:47 |
| *** rgerganov has quit IRC | 15:47 | |
| fungi | mordred: yolanda: awesome! | 15:47 |
| zaro | fungi: has he tried project owner permission. looks like project owners have that permission. | 15:47 |
| *** james_li has quit IRC | 15:47 | |
| morganfainberg | sdague, would http://bit.ly/1lPSbZn be [used a shortener here cause logstash urls are looong" be too wide a cast for a recheck classifier? it's what i'm generally seeing | 15:47 |
| jeblair | mordred, yolanda: serendipity -- i was just starting to review dib nodepool patches | 15:48 |
| zaro | fungi: https://gerrit-review.googlesource.com/Documentation/access-control.html#project_owners | 15:48 |
| mordred | fungi, jeblair: we just chatted about adding a few more tests - but I think the overall patch https://review.openstack.org/#/c/46482/ is ready for review | 15:48 |
| fungi | mordred: yolanda: sync'ed up with clarkb? i know he was struggling to get mysql/postgres configuration happening on the base images under dib | 15:48 |
| morganfainberg | but i don't want to propose useless classifier searches | 15:48 |
| *** jcoufal is now known as jcoufal_bbl | 15:48 | |
| mordred | and I think we can land that one before we finish the config dib patches | 15:48 |
| *** yaguang has quit IRC | 15:48 | |
| mordred | fungi: there's two sides | 15:48 |
| yolanda | mordred, fungi, i tested locally with nodepool and dib, adding devstack nodes, they work fine | 15:48 |
| fungi | zaro: ahh, yeah, i thought you'd also suggested something else previously since we don't want to grant owner permission on refs (as it comes with lots more besides just branch deletes) | 15:49 |
| jeblair | mordred: should i stop reviewing your config patches? | 15:49 |
| mordred | fungi: one is adding support to nodepool to know how to configure and run diskimage-builder and upload to galnce | 15:49 |
| *** dkranz has quit IRC | 15:49 | |
| mordred | jeblair: the patches up through .. you know, let me mark the one that's not ready WIP real quick | 15:49 |
| sdague | morganfainberg: no, that seems fair | 15:49 |
| *** dprince has quit IRC | 15:50 | |
| sdague | morganfainberg: though do we already have a bug for that? | 15:50 |
| mriedem | sdague: anteaya: quick and dirty etherpad here https://etherpad.openstack.org/p/elastic-recheck-testing | 15:50 |
| morganfainberg | sdague, ok let me file a bug and propose the classifier, what is the typical window you like to see on the ER classifiers? | 15:50 |
| morganfainberg | sdague, can't find one, i've been searching up and down | 15:50 |
| *** rloo has quit IRC | 15:50 | |
| sdague | morganfainberg: whats the tempest test fail look like under that case? | 15:50 |
| * fungi needs to step out for lunch. bbiab | 15:50 | |
| *** dizquierdo has quit IRC | 15:50 | |
| mordred | jeblair: done. it's "add additional node types" that needs more work - devstack works fine - it's the thick slaves for unittests that are tricky right now | 15:50 |
| *** rloo has joined #openstack-infra | 15:50 | |
| morganfainberg | sdague, http://logs.openstack.org/11/101611/10/check/check-tempest-dsvm-postgres-full/0284f87/console.html#_2014-06-25_05_19_59_116 | 15:51 |
| sdague | I'm surprised with the # of hits that there isn't one | 15:51 |
| mordred | jeblair: but I figure that gives us some lattitude to roll out node type at a time | 15:51 |
| *** david-lyle has quit IRC | 15:51 | |
| morganfainberg | sdague and other similar ones. it's not always on ip address | 15:51 |
| *** david-lyle has joined #openstack-infra | 15:51 | |
| morganfainberg | seeing it as #1 on http://status.openstack.org/elastic-recheck/data/uncategorized.html forgate-tempest-dsvm-full : 8 Uncategorized Fails. 87.3% Classification Rate (63 Total Fails) | 15:52 |
| morganfainberg | or at least appears to be the same thing | 15:52 |
| sdague | morganfainberg: yep, new bug is good | 15:52 |
| sdague | that actually looks specific enough to fix | 15:52 |
| morganfainberg | sdague, ++ cool, thanks :) | 15:52 |
| morganfainberg | sdague, 24h the standard window for an ER classifier? | 15:53 |
| *** rloo has quit IRC | 15:53 | |
| anteaya | in about an hour, I will disappear for the afternoon | 15:53 |
| anteaya | in case anyone cares | 15:53 |
| *** rloo has joined #openstack-infra | 15:53 | |
| mriedem | morganfainberg: 7 days | 15:53 |
| *** gokrokve has joined #openstack-infra | 15:53 | |
| morganfainberg | mriedem, ++ will do | 15:53 |
| morganfainberg | anteaya, as long as you're eventually back and not disappearing for awful reasons :) | 15:54 |
| *** lttrl has joined #openstack-infra | 15:54 | |
| *** rloo has quit IRC | 15:54 | |
| *** rloo has joined #openstack-infra | 15:55 | |
| yolanda | fungi, my changes are more related with nodepol and dib, i didn't work on the config stuff, that work has been done mostly by mordred | 15:55 |
| jeblair | anteaya: thx | 15:55 |
| *** rloo has quit IRC | 15:56 | |
| anteaya | morganfainberg: all good, errands to run, y'know life stuff | 15:56 |
| morganfainberg | anteaya, then it's a good reason to disappear :) | 15:56 |
| anteaya | jeblair: kk | 15:56 |
| anteaya | morganfainberg: yep, have to do it every once in a while | 15:56 |
| *** david-lyle has quit IRC | 15:56 | |
| *** david-lyle has joined #openstack-infra | 15:57 | |
| *** gondoi is now known as zz_gondoi | 15:57 | |
| *** rloo has joined #openstack-infra | 15:57 | |
| *** rloo has quit IRC | 15:58 | |
| *** _nadya_ has joined #openstack-infra | 15:58 | |
| *** rloo has joined #openstack-infra | 15:58 | |
| *** yamahata has quit IRC | 15:58 | |
| sdague | sadly it looks like the apt retry thing didn't impact the apt issue at all | 15:59 |
| sdague | https://review.openstack.org/#/c/101004/ | 15:59 |
| *** david-lyle has quit IRC | 16:00 | |
| *** gokrokve has quit IRC | 16:00 | |
| *** dkranz has joined #openstack-infra | 16:00 | |
| *** rloo has quit IRC | 16:00 | |
| *** rloo has joined #openstack-infra | 16:00 | |
| *** david-lyle has joined #openstack-infra | 16:00 | |
| *** rloo has quit IRC | 16:04 | |
| *** rloo has joined #openstack-infra | 16:04 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/elastic-recheck: Add query for nova bug 1334345 https://review.openstack.org/102571 | 16:05 |
| uvirtbot | Launchpad bug 1334345 in nova "State expected powering off but is None" [Undecided,New] https://launchpad.net/bugs/1334345 | 16:05 |
| *** dtantsur is now known as dtantsur|afk | 16:06 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: add network_sanity_check https://review.openstack.org/101976 | 16:07 |
| *** hogepodge has joined #openstack-infra | 16:07 | |
| sdague | fungi: I added some more to that which would hopefully have helped me debug the apt issue further | 16:07 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for heat bug 1333556 https://review.openstack.org/102554 | 16:10 |
| uvirtbot | Launchpad bug 1333556 in heat "SQLAlchemy >= 0.9.5 breaks tests" [Critical,Fix committed] https://launchpad.net/bugs/1333556 | 16:10 |
| dhellmann | sdague: this a2ensite bug appears to be related to devstack, but I can't for the life of me find where anything is setting up this pypi config in apache. If you have a few minutes to give me some hints, I would appreciate it. | 16:10 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/elastic-recheck: Clarifies documentation about wildcards in queries https://review.openstack.org/102573 | 16:10 |
| *** e0ne has joined #openstack-infra | 16:12 | |
| *** zehicle_at_dell has quit IRC | 16:12 | |
| *** praneshp has joined #openstack-infra | 16:14 | |
| *** rcarrillocruz has joined #openstack-infra | 16:14 | |
| JayF | If someone has a second, I'd love to get this merged and approved. It already has one +2. https://review.openstack.org/#/c/102400/ | 16:15 |
| *** rcarrill` has quit IRC | 16:16 | |
| *** david-lyle has quit IRC | 16:17 | |
| *** UtahDave has joined #openstack-infra | 16:17 | |
| *** david-lyle has joined #openstack-infra | 16:17 | |
| sdague | dhellmann: it's not related to devstack | 16:19 |
| *** david-lyle has quit IRC | 16:19 | |
| *** david-ly_ has joined #openstack-infra | 16:19 | |
| dhellmann | sdague: yeah, I just realized devstack is finishing | 16:19 |
| sdague | it's in the requirements job somewhere | 16:19 |
| sdague | which I think is in the pbr code base | 16:20 |
| dhellmann | this console log doesn't have enough information for me to tell what script is being run | 16:20 |
| sdague | for... odd reasons | 16:20 |
| sdague | you probably have to go back to zuul definitions | 16:20 |
| mordred | jeblair: if you havne't started reviewing the dib-nodepool config patch, I'm going to split it into a few pieces real quick | 16:20 |
| dhellmann | I have litterally grepped our entire source base for some of these commands and I'm not finding them, even in snippets | 16:20 |
| dhellmann | maybe I just fail at grep | 16:20 |
| jeblair | mordred: i have only barely started on 88479, feel free | 16:21 |
| dhellmann | sdague: ffs, pbr? | 16:22 |
| sdague | dhellmann: hey, don't look at me | 16:22 |
| mordred | dhellmann: hysterical raisins | 16:22 |
| sdague | I was surprised when I figured that out | 16:22 |
| mordred | dhellmann: I think we should redesign that .. but it's 'working' so far so it's been low priority | 16:22 |
| *** david-ly_ has quit IRC | 16:22 | |
| dhellmann | ok | 16:22 |
| sdague | yeh, it should actually be in the requirements repo | 16:23 |
| sdague | I think a forklift is probably in order | 16:23 |
| *** david-lyle has joined #openstack-infra | 16:23 | |
| jeblair | but who wants to put stuff there -- it's so much more typing than pbr! | 16:23 |
| *** rloo has quit IRC | 16:24 | |
| *** packet has joined #openstack-infra | 16:24 | |
| dhellmann | ah, I see why I couldn't tell what script | 16:24 |
| *** rloo has joined #openstack-infra | 16:25 | |
| *** rloo has quit IRC | 16:25 | |
| *** rloo has joined #openstack-infra | 16:25 | |
| dhellmann | jeblair: if I went through all of the builders and added "now running foo builder from file blah" messages to each to help debug stuff like this, would you hate that patch? | 16:26 |
| *** amotoki has joined #openstack-infra | 16:26 | |
| *** afazekas_ has quit IRC | 16:26 | |
| *** rloo has quit IRC | 16:26 | |
| *** rloo has joined #openstack-infra | 16:26 | |
| *** rloo has quit IRC | 16:27 | |
| *** rloo has joined #openstack-infra | 16:27 | |
| *** david-lyle has quit IRC | 16:27 | |
| anteaya | our code base needs more foo | 16:27 |
| *** crc32 has joined #openstack-infra | 16:27 | |
| *** ramashri has joined #openstack-infra | 16:28 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Be verbose in gate hoook for requirements job https://review.openstack.org/102576 | 16:28 |
| *** sweston_ has joined #openstack-infra | 16:29 | |
| jeblair | dhellmann: i think i might like that patch | 16:29 |
| dhellmann | jeblair: ok, I'll work on an example | 16:29 |
| sdague | dhellmann: depending on how tricky you want to be, it might be worth doing the echo_summary thing that devstack does | 16:30 |
| jeblair | dhellmann: i may be a bit ambivalent on the "from file" part (it sounds like it has moderate bitrot potential), but we'll see :) | 16:30 |
| sdague | which actually lets us have summary and full logs | 16:30 |
| *** devkulkarni has quit IRC | 16:30 | |
| *** Hal_ has quit IRC | 16:31 | |
| dhellmann | jeblair: does jjb know where a builder was defined originally after it has parsed it? we could use a variable for the filename | 16:31 |
| *** mmaglana has quit IRC | 16:31 | |
| jeblair | dhellmann: nope | 16:31 |
| dhellmann | jeblair: ok | 16:31 |
| *** briancurtin has left #openstack-infra | 16:32 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Be verbose in gate hoook for requirements job https://review.openstack.org/102576 | 16:32 |
| dhellmann | jeblair: ^^ includes a message like what I'm thinking of | 16:33 |
| sdague | jeblair: https://review.openstack.org/#/c/100728/ - gets kind of rebase sad with other changes in flight if you want to take a poke. Mostly just adding some safety. | 16:33 |
| jeblair | why is xenserver ci commenting on it? | 16:35 |
| *** rcarrill` has joined #openstack-infra | 16:36 | |
| craigbr | For my new project change, https://review.openstack.org/99767, which adds the new project python-monascaclient: since I am adding the group monasca and adding the file modules/openstack_project/files/gerrit/acls/stackforge/monasca.config, do I need a line: "acl-config: /home/gerrit2/acls/stackforge/monasca.config" in modules/openstack_project/files/review.projects.yaml? | 16:36 |
| craigbr | 16:36 | |
| *** timrc is now known as timrc-afk | 16:36 | |
| jeblair | i have sent them mail asking them to stop | 16:38 |
| *** rcarrillocruz has quit IRC | 16:38 | |
| *** crc32 has quit IRC | 16:40 | |
| sdague | jeblair: you can ask BobBall, but I think because they have a full zuul stand up including using devstack-gate | 16:40 |
| sdague | and they often need changes, or our changes break them | 16:40 |
| jeblair | that's cool, but i don't want 3rd party ci reports on devstack-gate | 16:40 |
| *** arnaud has joined #openstack-infra | 16:40 | |
| * anteaya makes a note | 16:41 | |
| jeblair | if everyone who used it reported, it would be quite a mess | 16:41 |
| sdague | jeblair: sure, though there is xen specific code in there | 16:41 |
| jeblair | sdague: oh right -- they were working on running that upstream | 16:42 |
| *** pblaho has joined #openstack-infra | 16:42 | |
| jeblair | probably needs to be one thing or the other -- upstream or downstream | 16:43 |
| jeblair | this middle ground where we have upstream support for running downstream tests is not our long-term goal | 16:43 |
| sdague | it looks like the voting is relatively recent phenomena - https://review.openstack.org/#/q/devstack-gate+label:Verified%253E%253D1%252Copenstack%2540citrix.com,n,z | 16:44 |
| *** timrc-afk is now known as timrc | 16:44 | |
| jeblair | sdague: it started the day i left on vacation ;) | 16:44 |
| jeblair | sly... | 16:44 |
| sdague | see, you just can't go on vacation any more :) | 16:45 |
| jeblair | nice try | 16:45 |
| *** signed8bit has quit IRC | 16:45 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Creates a list of no-test repositories https://review.openstack.org/102581 | 16:46 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Split the database grants into their own class https://review.openstack.org/98586 | 16:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add support for disk-image-builder in nodepool https://review.openstack.org/88479 | 16:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Only enabling rsyslog service if not chroot https://review.openstack.org/101562 | 16:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add additional node types to dib elements https://review.openstack.org/90565 | 16:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Don't manage iptables if we're in a choot https://review.openstack.org/102582 | 16:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add pre-requisites needed for diskimage-builder https://review.openstack.org/102583 | 16:47 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Creates a list of no-test repositories https://review.openstack.org/102581 | 16:47 |
| sdague | ah, damn, already a merge conflict on 100728 | 16:47 |
| sdague | let me rebase fast | 16:47 |
| anteaya | hit the bomb with bookends | 16:47 |
| jeblair | anteaya: if we were to include that, i'd say "openstack-infra/*". but actually i think that should be covered by the fact that we want ptls to make the decision about when something is ready to vote on a repo | 16:47 |
| anteaya | that doestn' happen often | 16:48 |
| anteaya | right but that doesn't stop the wild commenting | 16:48 |
| mordred | jeblair: ok. I split the above ^^ into a few more discreet pieces - the first 4 of which I think we can land safely without the other pieces | 16:48 |
| *** crc32 has joined #openstack-infra | 16:48 | |
| anteaya | systems that have gone wild I am trying to get them to talk to ptls before they comment on repos | 16:48 |
| anteaya | but so far that hasn't worked well | 16:48 |
| jeblair | anteaya: i don't think xenserver is a 'gone wild' thing | 16:49 |
| anteaya | since the only instance is Longgeek | 16:49 |
| anteaya | I can abandon if you want | 16:49 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: double up [ for safety https://review.openstack.org/100728 | 16:49 |
| jeblair | anteaya: i'm just thinking that the expectation should be that you should not comment unless invited | 16:49 |
| sdague | jeblair: ok, rebased | 16:49 |
| jeblair | or vote, rather | 16:49 |
| anteaya | they were voting on devstack? | 16:49 |
| anteaya | devstack-gate? | 16:49 |
| Longgeek | anteaya: ? | 16:49 |
| anteaya | Longgeek: hi | 16:50 |
| sdague | anteaya: xenserver remains voting on devstack, and I'm fine with that | 16:50 |
| Longgeek | anteaya: hi | 16:50 |
| sdague | they've been doing that for a while | 16:50 |
| anteaya | devstack-gate was the issue | 16:50 |
| jeblair | anteaya: yeah. though actually, i don't want either. i suppose we have set the expectation that they can comment without invitation... | 16:50 |
| anteaya | Longgeek: no, i was saying I had asked you to talk to the qa ptl to ask to comment on patches | 16:50 |
| anteaya | Longgeek: and that hadn't gone well, since noone knows what you are testing | 16:50 |
| jeblair | anteaya: your approach may be necessary after all :) | 16:50 |
| anteaya | jeblair: I have to jet | 16:51 |
| anteaya | we can steep on it | 16:51 |
| anteaya | and come back to it | 16:51 |
| Longgeek | anteaya: I have closed my CI system. | 16:51 |
| *** gokrokve has joined #openstack-infra | 16:51 | |
| *** doude has quit IRC | 16:52 | |
| clarkb | morning | 16:52 |
| jeblair | anteaya: ++ | 16:52 |
| mordred | clarkb: I just rebased the dib-nodepool patches to config - I think the first 4 can land already | 16:52 |
| *** mmaglana has joined #openstack-infra | 16:52 | |
| mordred | I _actually_ think the fifth probably can too | 16:52 |
| *** signed8bit has joined #openstack-infra | 16:53 | |
| clarkb | mordred: fifth is the DB one? | 16:53 |
| clarkb | also that is exciting | 16:53 |
| mordred | clarkb: no. fifth is devstack-gate | 16:53 |
| mordred | clarkb: sixth is WIP and is where we need to deal with thick_slave | 16:53 |
| anteaya | Longgeek: oh okay, does that mean we can disable the THSTACK CI account? | 16:53 |
| anteaya | Longgeek: we would like to disable if you aren't going to use it | 16:54 |
| mordred | but devstack-gate nodes do not need db solved | 16:54 |
| *** mwagner_lap has quit IRC | 16:54 | |
| clarkb | mordred: right | 16:54 |
| clarkb | I will try to rereview in between doing trusty things | 16:54 |
| * clarkb has secret goal to trusty everything before next meeting | 16:54 | |
| mordred | clarkb: anyway - yeah, since you've got good context, I thnk the early patches in the series should be easy reads | 16:54 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: Update integration script for Apache 2.4 https://review.openstack.org/102584 | 16:54 |
| mordred | clarkb: also, turns out notify => undef is a bad thing and notify => [] needs to be used :( | 16:55 |
| *** devkulkarni has joined #openstack-infra | 16:55 | |
| clarkb | ya | 16:55 |
| clarkb | I was worried about things like that | 16:55 |
| Longgeek | anteaya: No, I want to take the time to adjust my CI system, wait until the right time I'll tell you, until you let me open. | 16:55 |
| clarkb | so much of it is dark voodoo undocumented puppet hax | 16:55 |
| Longgeek | anteaya: But please don't close my account | 16:56 |
| mordred | clarkb: yah. that's actually one of the reasons I just split it into a series - so each of the things can be reasoned about | 16:56 |
| dhellmann | mordred, sdague, jeblair : Please take a look at ^^ to unblock the requirements-integration tests | 16:56 |
| *** ominakov has quit IRC | 16:57 | |
| clarkb | mordred: jeblair: any interest in reviewing https://review.openstack.org/#/c/101303/ before I push it through? | 16:58 |
| mordred | clarkb: looking | 16:58 |
| clarkb | that is next step in trustifying everything | 16:58 |
| jeblair | dhellmann: so on 102576, i'm less excited about that because it's naming the job-specific builder... i thought you were thinking of adding lines to the global builders | 16:58 |
| *** jpich has quit IRC | 16:59 | |
| dhellmann | jeblair: *all* of the builders, because as someone who isn't in these files all the time, I don't know where anything is defined and it makes it harder for me to help debug. | 16:59 |
| clarkb | once we have that I can begin to move a projectish at a time | 16:59 |
| mordred | clarkb: lgtm | 16:59 |
| jeblair | dhellmann: sure, but i mean it's just printing the job name | 17:00 |
| *** achanda has joined #openstack-infra | 17:00 | |
| dhellmann | jeblair: in the log output, it's very very hard to tell what is part of the job and what is part of some other script | 17:00 |
| *** david-lyle has joined #openstack-infra | 17:01 | |
| clarkb | mordred: thanks | 17:03 |
| dhellmann | jeblair: it's also not always apparent how to convert the job name to the template name, unless you are familiar with the templates (in this case the job is check-* but it's defined from a template so that name never appears in the jenkins input files) | 17:03 |
| *** _david_ has joined #openstack-infra | 17:03 | |
| *** sarob has joined #openstack-infra | 17:04 | |
| *** otherwiseguy has quit IRC | 17:04 | |
| _david_ | zaro your question on repo-discuss: username mismatch? | 17:04 |
| jeblair | dhellmann: so the issue is that it was not obvious how to find the job definition for check-requirements-integration-dsvm? | 17:04 |
| *** doude has joined #openstack-infra | 17:05 | |
| *** basha has joined #openstack-infra | 17:05 | |
| *** andreykurilin_ has joined #openstack-infra | 17:05 | |
| mordred | jeblair: yah. | 17:05 |
| dhellmann | jeblair: it was not obvious from the log output of that job what was actually being run. Maybe some of this is just me having to learn how to debug the jobs better, and starting with the zuul file, but yes, it would have taken me a while to find that job's real definition. | 17:06 |
| *** rloo has quit IRC | 17:06 | |
| *** rloo has joined #openstack-infra | 17:06 | |
| jeblair | okay, that's a different problem than what i thought you were trying to solve | 17:06 |
| dhellmann | jeblair: we have lots of "bash -x" throughout, but for anything that isn't in a script there's no filename to look at | 17:06 |
| *** adrian_otto has quit IRC | 17:07 | |
| jeblair | (i thought you were trying to identify which builder was running a specific command, not which template was responsible for a job defn) | 17:07 |
| dhellmann | both | 17:07 |
| jeblair | well, in this case, i'm pretty sure that if you found the job, you would have found "bash -xe $BASE/new/pbr/tools/integration.sh" | 17:07 |
| *** rloo has quit IRC | 17:07 | |
| dhellmann | I'm trying to make it easier to tell from the log output when a command fails, what caused that command to be run so you can tell what you need to edit to change how it is run. | 17:07 |
| *** rloo has joined #openstack-infra | 17:08 | |
| NobodyCam | morning infra | 17:08 |
| *** rloo has quit IRC | 17:08 | |
| clarkb | NobodyCam: morning | 17:08 |
| dhellmann | yes, that's true | 17:08 |
| *** Ryan_Lane has joined #openstack-infra | 17:08 | |
| *** rloo has joined #openstack-infra | 17:08 | |
| NobodyCam | just stopping by to plug https://review.openstack.org/#/c/100063 (already has two +2's) | 17:08 |
| NobodyCam | morning clarkb | 17:08 |
| clarkb | jeblair: you have been onboard with the changes to do trusty so far so I am going to approve 101303 :) you +2'd a variant of it last week too | 17:08 |
| dhellmann | clarkb: trusty appears to have broken the requirements check jobs: https://bugs.launchpad.net/pbr/+bug/1334326 | 17:09 |
| uvirtbot | Launchpad bug 1334326 in pbr "check-requirements-integration-dsvm failing with "Site pypi does not exist"" [Critical,In progress] | 17:09 |
| clarkb | dhellmann: cool | 17:09 |
| *** rloo has quit IRC | 17:09 | |
| jeblair | clarkb: i'm sure i'm okay with it, whatever it is! :) | 17:09 |
| clarkb | dhellmann: that may be the first actual broken not related to zuul or jenkins | 17:09 |
| clarkb | NobodyCam: so does tripleo even have enough nodes to run those tests? | 17:10 |
| *** rloo has joined #openstack-infra | 17:10 | |
| * dhellmann goes for lunch | 17:10 | |
| *** rloo has quit IRC | 17:10 | |
| clarkb | dhellmann: oh apache differences | 17:10 |
| NobodyCam | lifeless: asked for that test to be moved. | 17:10 |
| clarkb | I am less worried about the unittests now | 17:10 |
| *** eharney has quit IRC | 17:10 | |
| NobodyCam | lifeless: ^^^ clarkb's question | 17:10 |
| *** rloo has joined #openstack-infra | 17:11 | |
| clarkb | NobodyCam: if I pull up my handy graph you are still running on one cloud | 17:11 |
| clarkb | (I don't really care myself how long it takes for things to report back on your changes, just double checking you are ok with it consideriong the current cloud state) | 17:12 |
| *** rloo_ has joined #openstack-infra | 17:12 | |
| clarkb | mordred: did you see dhellmann's bug? | 17:12 |
| clarkb | mordred: it looks like a new apache problem | 17:13 |
| jeblair | dhellmann: okay, i think i'd like to try to find a way to accomplish what you want that doesn't involve changing hundreds of job definitions | 17:13 |
| mordred | clarkb: yes. he submitted a patch too | 17:13 |
| clarkb | there it is | 17:13 |
| clarkb | I had seen the config change not the pbr change | 17:13 |
| *** harlowja_away is now known as harlowja | 17:13 | |
| openstackgerrit | Radoslav Gerganov proposed a change to openstack-infra/config: Add button that shows/hides CI comments in Gerrit https://review.openstack.org/95743 | 17:14 |
| mordred | jeblair: what if we made a utility which could be pointed to a jjb dir and could be given a job name and would return info about where it came from? | 17:14 |
| mordred | jeblair: or, alternately, had jjb jobs put in a location comment header into shell snippets "# this is from template {name}-requirements-integration" | 17:14 |
| * mordred thinking out loud | 17:14 | |
| *** rloo has quit IRC | 17:16 | |
| clarkb | jeblair: 101303 is the change to allow trusty for unittesting in jobs | 17:17 |
| clarkb | jeblair: so once that is in I can make zuul layout changes that switch projects to trusty as the default node | 17:17 |
| *** jaypipes has joined #openstack-infra | 17:17 | |
| jeblair | mordred: yeah, i'm thinking a jjb enhancement may be a good idea | 17:18 |
| *** devkulkarni has quit IRC | 17:20 | |
| mordred | jeblair: ++ | 17:20 |
| *** jlibosva has quit IRC | 17:21 | |
| *** rloo_ has quit IRC | 17:23 | |
| *** devkulkarni has joined #openstack-infra | 17:23 | |
| *** _david_ has quit IRC | 17:23 | |
| *** rloo has joined #openstack-infra | 17:23 | |
| *** arnaud__ has joined #openstack-infra | 17:25 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Install curl when logstash watchdog is used. https://review.openstack.org/102588 | 17:25 |
| clarkb | jeblair: mordred ^ that is sort of the easy way out on curl | 17:25 |
| clarkb | it looks like we install it where it is required rather than installing it everywhere | 17:25 |
| clarkb | I can work on a different change to just install it everywhere but I wanted to start with being consistent | 17:25 |
| openstackgerrit | A change was merged to openstack-infra/config: Prep JJB jobs for running on bare-trusty https://review.openstack.org/101303 | 17:26 |
| clarkb | woot the next puppet run will catch that and I can start moving projects to trusty /me runs off to test nova, neutron, and ceilomter | 17:26 |
| *** SumitNaiksatam has joined #openstack-infra | 17:27 | |
| enikanorov | mriedem: removed \r in those yamls, i was creating those under windows, forgot to dos2unix them :) | 17:28 |
| lifeless | clarkb: whats the q ? | 17:29 |
| clarkb | lifeless: https://review.openstack.org/#/c/100063/2 adds jobs to your resource constrained env | 17:30 |
| mordred | clarkb: I'm good with installing where needed - but I'm starting to also be a big fan of just installing it everywhere | 17:30 |
| clarkb | lifeless: there have previously been chagnes to remove jobs from that env | 17:30 |
| jeblair | clarkb: 97389 has 4 +2s; you want to give #5 and aprv? | 17:30 |
| clarkb | jeblair: sure let me review real quick | 17:30 |
| mordred | jeblair: are we doing 5x+2 now? :) | 17:30 |
| clarkb | done | 17:31 |
| *** otherwiseguy has joined #openstack-infra | 17:31 | |
| *** rcarrillocruz has joined #openstack-infra | 17:31 | |
| jeblair | mordred: makes all the changes 5x better! | 17:31 |
| NobodyCam | lifeless: question was are we sure we have the resources in our pool to run the undercloud job | 17:31 |
| jeblair | mordred: (it's a root access change) | 17:31 |
| *** dprince has joined #openstack-infra | 17:31 | |
| *** rcarrill` has quit IRC | 17:32 | |
| clarkb | I have held 567434 which is a used bare-trusty node to do testing on | 17:33 |
| clarkb | once it is done running its current job I am going to dig into that | 17:34 |
| *** e0ne has quit IRC | 17:34 | |
| clarkb | my plan is to start with nova, neutron and ceilometer because they seem to have the biggest benefits to updating to trusty | 17:34 |
| clarkb | new libvirt, mongodb, kernels, ovs and so on | 17:34 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Added a puppet module for livegrep. https://review.openstack.org/96877 | 17:34 |
| *** e0ne has joined #openstack-infra | 17:35 | |
| *** markwash has quit IRC | 17:36 | |
| *** rloo has quit IRC | 17:36 | |
| *** rloo has joined #openstack-infra | 17:37 | |
| sdague | dhellmann: is that also going to work on stable jobs? | 17:37 |
| sdague | I don't think pbr actually tests the stable case | 17:37 |
| *** homeless has quit IRC | 17:37 | |
| *** dkehn_ has joined #openstack-infra | 17:37 | |
| *** rloo has quit IRC | 17:38 | |
| *** e0ne has quit IRC | 17:38 | |
| *** rloo has joined #openstack-infra | 17:38 | |
| clarkb | sdague: oh good question | 17:39 |
| clarkb | mordred: ^ | 17:39 |
| *** e0ne has joined #openstack-infra | 17:40 | |
| *** rloo has quit IRC | 17:40 | |
| *** signed8bit has quit IRC | 17:40 | |
| sdague | clarkb: got to run out to do some errands now, but if you have time to dig into ES backup, it would be appreciated. We're at least 12 hrs behind from what I can tell | 17:40 |
| *** rloo has joined #openstack-infra | 17:40 | |
| sdague | and don't seem to be catching up | 17:40 |
| *** james_li has joined #openstack-infra | 17:40 | |
| clarkb | sdague: ya its on my list. I think it may be a combo of volumes being bad, and no curl on machines, and java OOMing | 17:40 |
| *** e0ne has quit IRC | 17:41 | |
| clarkb | because when the volumes go bad everything snowballs | 17:41 |
| sdague | yeh | 17:41 |
| *** e0ne has joined #openstack-infra | 17:41 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:41 | |
| *** ramashri has quit IRC | 17:41 | |
| *** jedimike has quit IRC | 17:41 | |
| sdague | if only some friendly rax people would give us some IO baremetal nodes to "test" :) | 17:41 |
| mordred | clarkb: what about the change would need to interact with stable? | 17:42 |
| *** marcoemorais has joined #openstack-infra | 17:42 | |
| clarkb | mordred: stable stll runs on precise which is apache2 | 17:42 |
| *** Sukhdev has quit IRC | 17:42 | |
| clarkb | mordred: so if that test runs against stable it would have to work with both versions | 17:42 |
| *** dkehnx has quit IRC | 17:42 | |
| mordred | right. it doesn't | 17:42 |
| clarkb | cool | 17:42 |
| mordred | this is a test ot build the mirror | 17:42 |
| *** rloo has left #openstack-infra | 17:42 | |
| mordred | but also, I think the change will work with old apache as well - old apache did not care what the files were called | 17:42 |
| clarkb | gotcha | 17:43 |
| *** homeless has joined #openstack-infra | 17:43 | |
| *** flaper87|afk is now known as flaper87 | 17:43 | |
| *** e0ne has quit IRC | 17:43 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Add templated uploading jobs for doc8 project https://review.openstack.org/96631 | 17:44 |
| *** jaypipes has quit IRC | 17:44 | |
| *** e0ne has joined #openstack-infra | 17:44 | |
| *** jaypipes has joined #openstack-infra | 17:45 | |
| *** dkehn__ has joined #openstack-infra | 17:45 | |
| *** shayneburgess has joined #openstack-infra | 17:45 | |
| *** dkehn__ is now known as dkehnx | 17:45 | |
| *** jerryz_ has joined #openstack-infra | 17:45 | |
| sdague | mordred: it doesn't strip the .conf though | 17:45 |
| sdague | I thought | 17:45 |
| *** devkulkarni has quit IRC | 17:46 | |
| *** e0ne has quit IRC | 17:47 | |
| sdague | mordred: https://github.com/openstack-dev/devstack/blob/master/lib/apache#L85-L91 | 17:47 |
| mordred | sdague: does that matter? | 17:47 |
| sdague | we just went through this in gorry detail in devstack | 17:47 |
| mordred | gotcha. well, in this case, we're not doing a2ensite since there is only one site | 17:48 |
| mordred | but good to know | 17:48 |
| sdague | mordred: no, you are | 17:48 |
| sdague | https://review.openstack.org/#/c/102584/1/tools/integration.sh | 17:48 |
| *** atiwari has joined #openstack-infra | 17:48 | |
| sdague | the a2ensite was the thing that was failing | 17:48 |
| sdague | now it will fail the other way | 17:48 |
| mordred | oh. piddle | 17:48 |
| mordred | GAH | 17:48 |
| clarkb | in other news once I switch nova to trusty we can install python-libvirt via pypi | 17:49 |
| *** dkehn_ has quit IRC | 17:49 | |
| clarkb | mordred: ^ | 17:49 |
| morganfainberg | mordred, yeah you need to check if it's apache 2.2 or 2.4 the way devstack does :( | 17:49 |
| mordred | well ... ACTUALLY - we only ever run this in one place | 17:49 |
| mordred | so the new patch will fix it | 17:49 |
| mordred | we can completely rework this in the future to take advantage of libraries in devstack I think | 17:50 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files https://review.openstack.org/48783 | 17:50 |
| fungi | dhellmann: that requirements integration job is currently in the pbr repo because it started out as a "try to install everything to make sure this pbr patch isn't going to break us horribly" test | 17:50 |
| *** e0ne has joined #openstack-infra | 17:50 | |
| mordred | it was also the best place to ensure that changes to the test got self-gated | 17:51 |
| mordred | because we don't gate anything on the config repo | 17:51 |
| mordred | so it's a TERRIBLE place to put it - but kinda makes some weird sense | 17:51 |
| bookwar | when you use tox for testing packages, do you add tox itself to test-requirements.txt? I wonder what is the common practice? | 17:52 |
| clarkb | bookwar: no | 17:52 |
| clarkb | tox is assumed to already exist since it can't really bootstrap itself | 17:52 |
| bookwar | clarkb: i see, thanks | 17:53 |
| clarkb | bookwar: since tox is the thing we use to install test-requirements in many cases we end up in a chicken and egg situation | 17:54 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fixed referenced before assignment in zuul module https://review.openstack.org/92186 | 17:55 |
| zaro | _david_: thanks for your help. i've reposted, not sure if it's a bug or by design. | 17:56 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix a broken include in the configuration documentation. https://review.openstack.org/100838 | 17:56 |
| openstackgerrit | A change was merged to openstack-infra/config: Add -y to apt-get for IPA builder https://review.openstack.org/102400 | 17:57 |
| fungi | heh... http://www.infoworld.com/t/private-cloud/joyent-vows-beat-openstack-its-own-game-244874 | 17:57 |
| fungi | the node.js people are getting into writing a cloud platform now? | 17:58 |
| jeblair | "Folks developing OpenStack have not actually put their own butts on the line by operating an OpenStack-based public cloud." | 17:58 |
| sdague | mordred: we run this job on stable/icehouse requirements as well | 17:59 |
| sdague | so I think this breaks our ability to update requirements in any stable branch | 17:59 |
| fungi | jeblair: i like that they see having a diverse community pushing competing viewpoints as a liability | 17:59 |
| fungi | because we now have... how many examples of "handful of people design their own cloud platform and assume it will meet everyone's needs"? | 18:00 |
| *** nati_ueno has joined #openstack-infra | 18:00 | |
| TheJulia | fungi: they've been getting into it for ~2 years, I actually have a slice on a smartos machine. It leaves some things to be desired though. | 18:00 |
| *** e0ne has quit IRC | 18:00 | |
| fungi | TheJulia: smartos actually doesn't seem that bad, based on what i've read. bigger OoO is at the "smartdatacenter 7" base concept | 18:01 |
| clarkb | nova develop-inst with tox is really slow | 18:01 |
| TheJulia | fungi: That I've not yet seen. Really the problems I encountered were all package management related. :( | 18:02 |
| *** Sukhdev has joined #openstack-infra | 18:02 | |
| *** rm_work|away is now known as rm_work | 18:02 | |
| sdague | fungi: in fairness, joyent is a bunch of the smart people that were solaris devs back in the day | 18:02 |
| clarkb | which explains why they don't care about package management :) | 18:03 |
| sdague | which also explains the not wanting to play with others | 18:03 |
| fungi | sdague: back in which day? back in the pre-solaris sunos days, or back in the opensolaris days, or back in the flailingoraclesunisdying days? | 18:03 |
| sdague | ok, time to really leave and do errands :) | 18:03 |
| mordred | do joyent know what our game is? | 18:03 |
| fungi | seems like no | 18:03 |
| mordred | because I'm going to guess they don't | 18:03 |
| sdague | fungi: pre open solaris | 18:03 |
| *** chianingwang has joined #openstack-infra | 18:04 | |
| sdague | they mostly are folks that fled around the oracle buyout | 18:04 |
| * TheJulia likes the OmniOS folks better... they have liquor on their desks | 18:04 | |
| fungi | especially with cto quotes like "it's easy to overemphasize that [open source orchestration layer] value, especially when that substrate is allowing vendors to add their own proprietary hooks" | 18:04 |
| *** eharney has joined #openstack-infra | 18:04 | |
| fungi | TheJulia: that certainly makes it easier to deal with some classes of problems | 18:05 |
| *** arnaud__ has quit IRC | 18:06 | |
| *** arnaud has quit IRC | 18:06 | |
| TheJulia | That was their point of view as well. | 18:06 |
| *** arnaud has joined #openstack-infra | 18:06 | |
| *** arnaud__ has joined #openstack-infra | 18:07 | |
| *** ominakov has joined #openstack-infra | 18:07 | |
| fungi | TheJulia: also, we missed you at the last rtp osug meetup. i probably won't get to many more of them now that i've moved, but you'll need to start going so you can harass Shrews and olaph | 18:09 |
| *** ramashri has joined #openstack-infra | 18:09 | |
| *** basha has quit IRC | 18:10 | |
| *** arnaud has quit IRC | 18:10 | |
| *** arnaud__ has quit IRC | 18:10 | |
| Shrews | olaph is too busy with his kids/cows/chickens/goats/<insert_animal_here> to attend anymore | 18:11 |
| *** tkelsey has quit IRC | 18:12 | |
| *** markmc has quit IRC | 18:12 | |
| bodepd | jeblair: thanks for the verify! | 18:12 |
| *** ominakov has quit IRC | 18:12 | |
| TheJulia | fungi: Yeah :( Last month I started coming down with a cold that afternoon so I was asleep shortly after 5 PM. I'm planning on going to the birthday party meetup tomorrow though. | 18:12 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Patch writexml for non standard XML DOM implementations https://review.openstack.org/93198 | 18:13 |
| dhellmann | fungi, mordred : that makes sense, when put that way | 18:13 |
| *** e0ne has joined #openstack-infra | 18:13 | |
| *** afazekas_ has joined #openstack-infra | 18:13 | |
| dhellmann | jeblair: yeah, I'll take that line out of the current patch and keep thinking about it | 18:13 |
| devananda | SergeyLukjanov: 100063 has +1's from both lifeless and myself. I've it's GTG | 18:14 |
| TheJulia | fungi: In any event, somebody told me traffic was absoloutely horrible heading east bound through RTP that evening | 18:14 |
| fungi | TheJulia: i thought the b'day party for the triangle osug was next month? | 18:14 |
| devananda | *It's GTG | 18:14 |
| fungi | TheJulia: when is traffic on i-40 not horrible in the evenings? ;) | 18:14 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Be verbose in gate hoook for requirements job https://review.openstack.org/102576 | 18:15 |
| TheJulia | heh, it is next month. Derp | 18:15 |
| fungi | TheJulia: yeah, i'll be at the main party in portland, or i'd come out to the triangle for that (since it'll be at a local brewery) | 18:15 |
| *** adrian_otto has joined #openstack-infra | 18:16 | |
| fungi | TheJulia: though the tripleo mid-cycle sprint is in raleigh that week too, so they should totally crash that party | 18:17 |
| jeblair | check-rally-dsvm-neutron-neutron? | 18:17 |
| * TheJulia fixes her calendar and adds a reminder to suggest crashing the party | 18:17 | |
| * jeblair assumes neutron-neutron has meaning | 18:18 | |
| fungi | a confusing meaning | 18:19 |
| fungi | perhaps it's part of a deuterium nucleus? | 18:19 |
| fungi | still need a proton in the mix though | 18:19 |
| *** signed8bit has joined #openstack-infra | 18:20 | |
| fungi | would have to be tritium (two neutrons one proton) i guess | 18:20 |
| dhellmann | clarkb: are you the one doing the config work on the trusty machines? | 18:20 |
| clarkb | jeblair: I think it means they are profiling neutron in a devstack like deployment with neutron running instead of nova network | 18:20 |
| clarkb | dhellmann: yes | 18:20 |
| clarkb | well some of it | 18:20 |
| clarkb | mordred did a lot of the work too | 18:20 |
| dhellmann | clarkb: it looks like we're missing something kerberos wants: http://logs.openstack.org/84/102584/1/check/check-requirements-integration-dsvm/8fe24b9/console.html | 18:20 |
| zaro | clarkb: we ready to merge build-timeout usage? https://review.openstack.org/#/c/95933 | 18:21 |
| dhellmann | clarkb: I'm trying to figure out what it is now, but thought I'd mention it in case you had seen that one before | 18:21 |
| clarkb | zaro: probably. the first change looks happy | 18:21 |
| jeblair | libkrb5-dev: /usr/bin/krb5-config | 18:22 |
| jeblair | ? | 18:22 |
| jeblair | on precise | 18:22 |
| clarkb | that looks maybe right | 18:22 |
| clarkb | is devstack not installing that for us on trusty? | 18:23 |
| * clarkb updates devstack repo | 18:23 | |
| clarkb | git grep krb returns nothing | 18:23 |
| clarkb | does that imply we got it transitively on precise? | 18:23 |
| jeblair | is in trusty too http://packages.ubuntu.com/trusty/amd64/libkrb5-dev/filelist | 18:23 |
| jeblair | clarkb: i was guessing the same thing | 18:24 |
| dhellmann | it's not in the default set of packages in the cloud image I have on DH, and I think we got that from canonical | 18:24 |
| dhellmann | installing it does allow pip to install the kerberos package | 18:24 |
| clarkb | dhellmann: ya that test depends on devstack to install the C deps it needs | 18:24 |
| clarkb | so devstack probably needs an update and this is a weird corner case where something else pulled it in on precise | 18:25 |
| dhellmann | ok, thanks, I'll submit a patch against devstack | 18:25 |
| jeblair | yeah, it was definitely being installed on precise | 18:26 |
| fungi | either transitively or it was on all our precise base images. though the latter seems specious | 18:26 |
| dhellmann | weird, it's in the global requirements list but not mentioned in any of the integrated projects | 18:26 |
| jeblair | fungi: transitively | 18:26 |
| jeblair | http://logs.openstack.org/71/101171/2/check/check-requirements-integration-dsvm/e1c144d/logs/devstacklog.txt.gz | 18:26 |
| fungi | ahh, yep | 18:26 |
| jeblair | it's getting installed there but is not in the apt-get cmdline | 18:26 |
| fungi | something like ldap was probably dragging it in | 18:27 |
| *** mwagner_lap has joined #openstack-infra | 18:27 | |
| *** rcarrillocruz has quit IRC | 18:28 | |
| jeblair | libpq-dev as it turns out | 18:28 |
| openstackgerrit | A change was merged to openstack-infra/config: give zaro access to jenkins-dev server https://review.openstack.org/97389 | 18:28 |
| *** hogepodge has quit IRC | 18:28 | |
| mriedem | is the ES cluster down? not seeing e-r reporting on known queries | 18:29 |
| openstackgerrit | A change was merged to openstack-infra/config: remove unused jenkins plugins from masters https://review.openstack.org/96891 | 18:29 |
| clarkb | mriedem: its behind | 18:29 |
| jeblair | http://packages.ubuntu.com/precise/libpq-dev | 18:29 |
| jeblair | http://packages.ubuntu.com/trusty/libpq-dev | 18:29 |
| jeblair | switched to krb5-multidev | 18:29 |
| mriedem | clarkb: ah ok, thanks | 18:29 |
| clarkb | mriedem: current working theory is when rax volumes stop giving us performance it needs everything snowballs and we end up in this state | 18:30 |
| jeblair | Development files for MIT Kerberos without Heimdal conflict | 18:30 |
| james_li | Hi clarkb and fungi : will docker be installed if setting DEVSTACK_GATE_VIRT_DRIVER=docker ? | 18:30 |
| clarkb | mriedem: processes OOM as they queue and so on | 18:30 |
| * zaro afk, bbl | 18:30 | |
| clarkb | james_li: probably not, but maybe if devstack does the right thing in that case | 18:30 |
| fungi | clarkb: also, for some reason elasticsearch-head no longer gives me any performance graphs... am i doing something stupid? | 18:30 |
| dhellmann | clarkb, jeblair, fungi : https://review.openstack.org/102600 | 18:30 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Added a puppet module for livegrep. https://review.openstack.org/96877 | 18:30 |
| clarkb | fungi: are you connected to 01? 01 is "off" | 18:31 |
| fungi | clarkb: er, bigdesk not eshead | 18:31 |
| fungi | clarkb: connected to 06 | 18:31 |
| james_li | ok, so setting DEVSTACK_GATE_VIRT_DRIVER=docker will cause a devstack error if docker is not installed in my contrib code | 18:31 |
| *** morganfainberg_L has quit IRC | 18:31 | |
| james_li | clarkb: ^^ | 18:31 |
| mordred | james_li: yah. I believe that's right. there are some examples of having things drop in appropriate devstack code to get things installed | 18:32 |
| clarkb | fungi: works for me. try clicking the connect button on the page? | 18:32 |
| jeblair | dhellmann: +2 | 18:32 |
| *** isviridov is now known as isviridov|away | 18:32 | |
| dhellmann | jeblair: thanks! | 18:32 |
| clarkb | fungi: I used 02 though as the command was in my history | 18:32 |
| *** lcostantino has quit IRC | 18:33 | |
| *** rcarrillocruz has joined #openstack-infra | 18:33 | |
| james_li | thanks mordred ! | 18:34 |
| fungi | clarkb: it was already connected from earlier today, but i disconnected and reconnected and still nothing. then switched from the nodes display to the cluster pack display and back again and got graphs. good enough i guess | 18:34 |
| clarkb | fungi: cool | 18:36 |
| *** devkulkarni has joined #openstack-infra | 18:36 | |
| *** ociuhandu has quit IRC | 18:37 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: Update integration script for Apache 2.4 https://review.openstack.org/102584 | 18:38 |
| *** e0ne has quit IRC | 18:38 | |
| *** e0ne has joined #openstack-infra | 18:39 | |
| *** pelix has quit IRC | 18:39 | |
| dhellmann | sdague, fungi : ^^ should address your concerns about supporting stable tests | 18:41 |
| *** afazekas_ has quit IRC | 18:41 | |
| *** CaptTofu_ has quit IRC | 18:41 | |
| *** CaptTofu_ has joined #openstack-infra | 18:42 | |
| *** CaptTof__ has joined #openstack-infra | 18:42 | |
| *** jp_at_hp has quit IRC | 18:44 | |
| clarkb | nova py27 on trusty works | 18:44 |
| clarkb | going to check docs and flake8 but should have patch up to move nova shortly assuming those still work | 18:45 |
| *** zehicle_at_dell has joined #openstack-infra | 18:45 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove apache-services tempest check https://review.openstack.org/97638 | 18:46 |
| *** CaptTofu_ has quit IRC | 18:47 | |
| *** andreykurilin_ has quit IRC | 18:47 | |
| *** wenlock has quit IRC | 18:50 | |
| *** basha has joined #openstack-infra | 18:52 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Display dates in timeago format https://review.openstack.org/96713 | 18:52 |
| dhellmann | I wonder why none of the other projects are being hit by this requirements check issue? What's special about oslo? Maybe it's just pbr's job configuration? | 18:52 |
| *** harlowja has quit IRC | 18:52 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add puppet-openstack project puppet-openstack_extras https://review.openstack.org/97647 | 18:53 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Create zuul template translation-jobs https://review.openstack.org/98252 | 18:53 |
| openstackgerrit | A change was merged to openstack-infra/config: Make install_puppet.sh more robust against failure https://review.openstack.org/97665 | 18:53 |
| openstackgerrit | A change was merged to openstack-infra/config: Keep devstack's localrc for manila jobs https://review.openstack.org/97747 | 18:54 |
| *** _nadya_ has quit IRC | 18:54 | |
| *** chuckC has quit IRC | 18:56 | |
| *** rcarrill` has joined #openstack-infra | 18:56 | |
| *** jerryz_ has quit IRC | 18:57 | |
| *** basha has quit IRC | 18:58 | |
| *** rcarrillocruz has quit IRC | 18:58 | |
| *** marcoemorais has quit IRC | 18:59 | |
| *** basha has joined #openstack-infra | 18:59 | |
| *** marcoemorais has joined #openstack-infra | 18:59 | |
| *** marcoemorais has quit IRC | 19:00 | |
| *** marcoemorais has joined #openstack-infra | 19:00 | |
| *** marcoemorais has quit IRC | 19:01 | |
| *** marcoemorais has joined #openstack-infra | 19:02 | |
| *** marcoemorais has quit IRC | 19:02 | |
| *** marcoemorais has joined #openstack-infra | 19:02 | |
| *** mwagner_lap has quit IRC | 19:02 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for nova libvirt connection reset snapshot bug 1334398 https://review.openstack.org/102608 | 19:03 |
| uvirtbot | Launchpad bug 1334398 in nova "snapshot hangs when libvirt connection is reset" [High,New] https://launchpad.net/bugs/1334398 | 19:03 |
| clarkb | mriedem: is ^ a new bug related to new libvirt? | 19:03 |
| *** afazekas_ has joined #openstack-infra | 19:03 | |
| clarkb | nova flake8 is slow | 19:04 |
| mriedem | clarkb: similar to bug 1255624 | 19:04 |
| uvirtbot | Launchpad bug 1255624 in nova "libvirtError: Unable to read from monitor: Connection reset by peer" [High,Triaged] https://launchpad.net/bugs/1255624 | 19:04 |
| mriedem | different point of failure | 19:04 |
| clarkb | jogo: does flake8 run in parallel? | 19:04 |
| *** openstackgerrit has quit IRC | 19:04 | |
| *** Longgeek has quit IRC | 19:04 | |
| mriedem | clarkb: looks like that really spiked yesterday | 19:05 |
| mriedem | that libvirt snapshot connection reset bu | 19:05 |
| mriedem | *bug | 19:05 |
| clarkb | mriedem: cool maybe it is related :) | 19:05 |
| clarkb | or at least made worse by | 19:05 |
| mriedem | clarkb: what happened yesterday? | 19:05 |
| dprince | clarkb: we have another TripleO CI job running on repeat https://review.openstack.org/#/c/84384/4 | 19:05 |
| clarkb | mriedem: we switched to trusty | 19:05 |
| *** CaptTof__ has quit IRC | 19:05 | |
| mriedem | clarkb: newer libvirt? | 19:05 |
| clarkb | mriedem: yes libvirt 1.2.2 now | 19:05 |
| *** CaptTofu_ has joined #openstack-infra | 19:06 | |
| *** basha has quit IRC | 19:06 | |
| clarkb | dprince: huh and that chaneg isn't even a weird one like lifeless' change that looped | 19:06 |
| *** openstackgerrit has joined #openstack-infra | 19:06 | |
| dprince | clarkb: my guess is this is a different issue than last week, but wondering if there is any job killing you could do | 19:06 |
| mriedem | clarkb: spikes yesterday and then drops off today | 19:06 |
| afazekas_ | Can you review this change: https://review.openstack.org/#/c/100103/ | 19:06 |
| mriedem | clarkb: did you turn off trusty today? | 19:06 |
| clarkb | jeblair: fungi is there any investigation of 84384 that we should do before I try to fix it? | 19:06 |
| clarkb | mriedem: no | 19:06 |
| clarkb | mriedem: we are moving full steam ahead on trusty | 19:06 |
| mriedem | ok | 19:06 |
| mriedem | well that's a weird blip then | 19:07 |
| clarkb | maybe it was caused by something else | 19:07 |
| clarkb | woot nova flake8 runs on trusty | 19:07 |
| clarkb | now docs | 19:07 |
| *** yjiang5 is now known as yjiang5_away | 19:08 | |
| *** hogepodge has joined #openstack-infra | 19:08 | |
| *** Longgeek has joined #openstack-infra | 19:08 | |
| openstackgerrit | A change was merged to openstack-infra/config: Initial commit for the AWS EC2 and VPC API standalone service project https://review.openstack.org/97795 | 19:08 |
| openstackgerrit | A change was merged to openstack-infra/config: Add rally-neutron-neutron job https://review.openstack.org/97805 | 19:08 |
| *** wenlock has joined #openstack-infra | 19:08 | |
| jeblair | woo, my config backlog is down to 2 digits! | 19:09 |
| fungi | clarkb: looks like it's not updating its vote when it loops. this could be similar to the one yesterday which was looping (much more rapidly) on a merge conflict | 19:09 |
| clarkb | jeblair: wow | 19:10 |
| clarkb | fungi: I think the tripleo test interval is making it go slowly | 19:10 |
| clarkb | fungi: it is commenting on the latest patchset | 19:10 |
| *** CaptTofu_ has quit IRC | 19:10 | |
| jeblair | oh, yeah, let's investigate before we do anything | 19:10 |
| clarkb | ok I won't hit it with a hammer | 19:10 |
| clarkb | it does seem like it could be related to the -1 not doing rechecks change | 19:11 |
| fungi | clarkb: yeah, yesterday's was too... and was also triggered by a comment for stale recheck | 19:11 |
| jeblair | (i have a pending change to zuul which would help a lot) | 19:11 |
| clarkb | as it did a -1 | 19:11 |
| *** melwitt has joined #openstack-infra | 19:11 | |
| clarkb | jeblair: is that a hint I should review it? /me does that | 19:11 |
| *** hashar has joined #openstack-infra | 19:11 | |
| jeblair | i mean, it won't help with this | 19:11 |
| clarkb | if it is the log one it has been approved | 19:11 |
| jeblair | yep that one | 19:11 |
| fungi | clarkb: abandoning the change yesterday halted the loop because it triggered the refusal to enqueue a closed change | 19:11 |
| jeblair | i mean, the next time something like this happened | 19:12 |
| jeblair | happens | 19:12 |
| *** ociuhandu has joined #openstack-infra | 19:12 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added tests for Project Groups API https://review.openstack.org/98745 | 19:12 |
| *** achanda_ has joined #openstack-infra | 19:14 | |
| *** sarob has quit IRC | 19:14 | |
| openstackgerrit | A change was merged to openstack-infra/config: upgrade jenkins build-timeout plugin due to a performance bug https://review.openstack.org/97651 | 19:15 |
| *** e0ne has quit IRC | 19:15 | |
| *** sarob has joined #openstack-infra | 19:15 | |
| *** rcarrillocruz has joined #openstack-infra | 19:15 | |
| clarkb | jeblair: I can review that stack. I take it it is safe to approve as I go? | 19:15 |
| krotscheck | mordred: You mentione that puppi had problematic dependencies, do you happen to remember which ones? | 19:16 |
| jeblair | clarkb: yeah | 19:16 |
| *** rcarrill` has quit IRC | 19:17 | |
| *** ramashri has quit IRC | 19:17 | |
| *** achanda has quit IRC | 19:17 | |
| fungi | ahh, yeah i got partway through that stack. finishing now. i'm so very, very behind on reviews | 19:17 |
| *** lttrl has quit IRC | 19:18 | |
| james_li | clarkb: mordred : is there a way that I can set VIRT_DRIVER in localrc? I export DEVSTACK_GATE_VIRT_DRIVER in pre_test_hook, but did not work. | 19:18 |
| *** Sukhdev has quit IRC | 19:18 | |
| james_li | should I export DEVSTACK_GATE_VIRT_DRIVER in gate_hook? | 19:19 |
| clarkb | james_li: exporting in the hooks is problematic due to the use of sudo | 19:19 |
| clarkb | you actually have to get it written to the localrc | 19:19 |
| clarkb | which devstack gate should do for you but maybe it is specific | 19:19 |
| mrodden | busted out laughin in the middle of a meeting today because of this http://lists.openstack.org/pipermail/openstack-infra/2014-June/001415.html | 19:19 |
| *** sarob has quit IRC | 19:19 | |
| *** lttrl has joined #openstack-infra | 19:20 | |
| james_li | clarkb: do you mean I should export DEVSTACK_GATE_VIRT_DRIVER in localrc? | 19:21 |
| james_li | or export VIRT_DRIVER in localrc? | 19:22 |
| *** basha has joined #openstack-infra | 19:22 | |
| clarkb | james_li: I mean export it before you run devstack gate and have it write the value to localrc | 19:22 |
| clarkb | I think it already does this | 19:22 |
| *** chuckC has joined #openstack-infra | 19:23 | |
| james_li | clarkb: think I tried that way in pre_test_hook, but did not work | 19:23 |
| clarkb | no not in pretest hook | 19:24 |
| clarkb | that is already too late | 19:24 |
| clarkb | before you run devstack gate | 19:24 |
| jeblair | fungi: was the other looping change also tripleo related? | 19:24 |
| fungi | jeblair: oh, good point... checking | 19:24 |
| *** _nadya_ has joined #openstack-infra | 19:24 | |
| fungi | jeblair: ironic i think, but... | 19:25 |
| jeblair | fungi: the specific thing i'm chasing down is participation in the tripleo pipelines | 19:25 |
| openstackgerrit | A change was merged to openstack-infra/config: Fetch graphitejs zuul dependency https://review.openstack.org/98018 | 19:25 |
| clarkb | jeblair: oh interesting | 19:25 |
| james_li | clarkb: so export in config / solum.yaml ? | 19:25 |
| fungi | jeblair: ironic... https://review.openstack.org/73005 | 19:25 |
| clarkb | james_li: ya, you may want to read devstack gate's use of that variable first | 19:25 |
| clarkb | james_li: to double check it writes it to localrc | 19:25 |
| fungi | jeblair: but possible it was troggering because of one of the tripleo pipelines for ironic changes? | 19:26 |
| fungi | troggering. heh | 19:26 |
| *** basha has quit IRC | 19:26 | |
| jeblair | fungi: yeah, it participates in that | 19:26 |
| jeblair | it was totally troggering | 19:27 |
| fungi | indeed it does. so sayeth the layout.yam | 19:27 |
| fungi | l | 19:27 |
| james_li | clarkb: ok thanks! | 19:27 |
| *** jcoufal_bbl has quit IRC | 19:28 | |
| jeblair | fungi: we couldn't tell with the ironic, change, but on this one, we can see that it's the tripleo pipeline that's looping | 19:28 |
| jeblair | (actually, we could, due to the lack of start messages, but it was just that much less obvious) | 19:28 |
| fungi | ahh, yep. it's a pipeline which doesn't remove the old jenkins vote | 19:29 |
| *** Energy has joined #openstack-infra | 19:29 | |
| *** rfolco has quit IRC | 19:30 | |
| Energy | hello all bitch ! | 19:30 |
| Energy | pussy ! | 19:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Added docs-on-rtfd for MagnetoDB https://review.openstack.org/99039 | 19:30 |
| *** Sukhdev has joined #openstack-infra | 19:31 | |
| *** ChanServ sets mode: +o jeblair | 19:31 | |
| *** Energy was kicked by jeblair (Energy) | 19:31 | |
| *** Energy has joined #openstack-infra | 19:31 | |
| Energy | suck my dick jeblair ! | 19:31 |
| *** jeblair sets mode: +b *!~Energy@* | 19:31 | |
| *** Energy was kicked by jeblair (Energy) | 19:32 | |
| *** alexpilotti has quit IRC | 19:32 | |
| *** lttrl has quit IRC | 19:33 | |
| *** jeblair sets mode: -o jeblair | 19:34 | |
| *** dstanek is now known as dstanek_zzz | 19:34 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for nova libvirt connection reset snapshot bug 1334398 https://review.openstack.org/102608 | 19:35 |
| uvirtbot | Launchpad bug 1334398 in nova "snapshot hangs when libvirt connection is reset" [High,New] https://launchpad.net/bugs/1334398 | 19:35 |
| clarkb | jeblair: this first change for that zuul stack is really dense | 19:36 |
| clarkb | or maybe I am spending too much time looking at the yaml | 19:36 |
| fungi | clarkb: i'm on that one now, but i reviewed them in reverse order. the rest are really teensy | 19:36 |
| krotscheck | jeblair: Regarding your comment about /opt vs. /usr/src in the storyboard puppet module: I’m guessing that’s a deliberate step away from the linux filesystem hierarchy standard? | 19:36 |
| clarkb | fungi: cool | 19:36 |
| jeblair | clarkb: i tried to make the yaml simple and just focus on one thing at a time | 19:37 |
| fungi | krotscheck: fhs would have you put it somewhere in /opt or /usr/local | 19:37 |
| krotscheck | fungi: Really? Source code that is eventually installed via pip? | 19:37 |
| clarkb | jeblair: yeah once I realized that it became easier | 19:37 |
| fungi | krotscheck: /opt being for unstructured local software additions and /usr/local being for structured software additions | 19:37 |
| clarkb | jeblair: but I was still trying to grok the differences betweenthem | 19:37 |
| jeblair | krotscheck: what fungi said; i'd personally prefer /usr/local/src, but we should take the opportunity to use the ephemeral drive to save space when we can | 19:37 |
| fungi | krotscheck: well, usually /opt would be software deployed with some comingled executables, data and configuration run directly from that location | 19:38 |
| *** pcrews has quit IRC | 19:39 | |
| *** pcrews_ has joined #openstack-infra | 19:39 | |
| fungi | krotscheck: /usr/src would tend to contain packaged source trees | 19:39 |
| *** Ryan_Lane has quit IRC | 19:39 | |
| fungi | krotscheck: er, system/distro-package-managed source trees | 19:39 |
| jeblair | fungi, clarkb: i understand the zuul loop | 19:39 |
| clarkb | that is great news | 19:39 |
| fungi | krotscheck: though i agree traditionally on *bsd /usr/src would just be any old sources you happened to want to compile | 19:39 |
| clarkb | jeblair: still want these chanegs to be approved as I go? | 19:40 |
| jeblair | clarkb: sure, it's not a zuul bug (per se) | 19:40 |
| fungi | jeblair: some sort of unforseen interaction in our configuration choices i guess? | 19:40 |
| krotscheck | fungi jeblair: Hrm. Ok, so… the directory doesn’t matter too much to me, I’m just concerned about outward appearances of something that’s hopefully going to end up on puppetforge some day. | 19:40 |
| krotscheck | …though at that point it’ll make more sense to have it pip-packaged first. | 19:41 |
| *** CaptTofu_ has joined #openstack-infra | 19:41 | |
| krotscheck | …which means this isn’t actually an issue. | 19:41 |
| krotscheck | …which means it’ll never matter. | 19:41 |
| krotscheck | Alllrightey! | 19:41 |
| * krotscheck loves it when he wins an argument against itself. | 19:41 | |
| krotscheck | *himself | 19:41 |
| *** ihrachyshka has joined #openstack-infra | 19:41 | |
| fungi | krotscheck: also we've previously been dragging git trees into /opt before deploying on our long-lived servers, so in some ways it's just a matter of historical consistency (not a strong argument in and of itself) | 19:42 |
| jeblair | fungi, clarkb: check-tripleo runs if the most recent jenkins vote is older than 120h; the most recent vote is older than that, but negative, so the normal check queue will not run again (because of jhesketh's change), so it will never have a newer vote unless someone does a manual recheck. | 19:42 |
| *** rfolco has joined #openstack-infra | 19:43 | |
| krotscheck | jeblair: Will /opt/src/ work? | 19:43 |
| krotscheck | jeblair: Or would you just prefer /opt/storyboard | 19:43 |
| fungi | krotscheck: /opt/storyboard would be consistent with our existing puppet vcsrepo usage | 19:44 |
| krotscheck | Works for me. | 19:44 |
| *** crc32 has quit IRC | 19:45 | |
| fungi | we clone into /opt/jeepyb, /opt/zuul, /opt/nodepool and so on for our continuously-deployed systems | 19:45 |
| *** dstanek_zzz is now known as dstanek | 19:45 | |
| jeblair | clarkb, fungi: i'm not sure the check-tripleo freshness thing ever worked as intended; i think the only time it would have worked is if the check queue was backlogged more than 48 hours | 19:45 |
| fungi | heh, great point | 19:46 |
| *** signed8bit is now known as signed8bit_ZZZzz | 19:46 | |
| clarkb | oh | 19:46 |
| fungi | again, because of the reliance on a shared vote | 19:46 |
| *** e0ne has joined #openstack-infra | 19:46 | |
| openstackgerrit | A change was merged to openstack-infra/config: Adding #openstack-nfv channel https://review.openstack.org/99421 | 19:46 |
| clarkb | so this could be fixed by non binding votes | 19:46 |
| fungi | well, semi-shared (one pipeline didn't vote at all) | 19:46 |
| fungi | clarkb: exactly what i was thinking | 19:46 |
| fungi | so for a working freshness check, the pipeline which is being refreshed needs its own vote | 19:47 |
| jeblair | clarkb: yeah, we could give that pipeline a user; or i think we might be able to do something where we trigger on vote values there... | 19:47 |
| fungi | jeblair: expand the vote range and turn it into a bitfield? | 19:47 |
| * fungi is only slightly joking there | 19:48 | |
| jeblair | clarkb, fungi: like: enqueue on a +1 vote from jenkins (that's an especially attractive option if we want to say that tripleo shouldn't even do an initial run until jenkins comes back with a positive vote) | 19:48 |
| *** pblaho has quit IRC | 19:48 | |
| fungi | oh, though a bitfield would get complicated with trinary votes | 19:48 |
| openstackgerrit | A change was merged to openstack-infra/config: Added the Surveil project to gerritbot, zuul and stackforge config https://review.openstack.org/99746 | 19:48 |
| clarkb | jeblair: I like that | 19:49 |
| jeblair | clarkb, fungi: but even if we still enqueue on upload, the chances of it running twice are fairly small | 19:49 |
| fungi | we'd have to do base 3 conversions | 19:49 |
| *** basha has joined #openstack-infra | 19:49 | |
| fungi | jeblair: clarkb: i do like the cascading queue idea there | 19:49 |
| *** crc32 has joined #openstack-infra | 19:49 | |
| fungi | way better than my base 3 math idea ;) | 19:49 |
| *** e0ne has quit IRC | 19:50 | |
| clarkb | fungi: since you said you are reviewing that same zuul change I won't approve it yet | 19:50 |
| clarkb | but +2 from me | 19:50 |
| fungi | clarkb: i'm +2 on all the children of that change, and am almost through going through it too | 19:50 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove freshness check from tripleo https://review.openstack.org/102618 | 19:50 |
| clarkb | ok I will review children really quickly and can approve them | 19:51 |
| jeblair | clarkb, fungi: i think we should speedy approve https://review.openstack.org/102618 | 19:51 |
| fungi | so i can approve 102029 in a moment if it lgtm, and you can do the same with the rest i guess | 19:51 |
| *** ihrachyshka has quit IRC | 19:51 | |
| jeblair | then think about how to put it back | 19:51 |
| clarkb | jeblair: +2'd | 19:51 |
| clarkb | fungi: sounds good | 19:51 |
| *** ihrachyshka has joined #openstack-infra | 19:51 | |
| fungi | jeblair: yep | 19:51 |
| clarkb | then once I am through this stack I need to find lunch | 19:52 |
| jeblair | i did 'recheck no bug' on https://review.openstack.org/#/c/84384/ which should stop the loop | 19:52 |
| *** e0ne has joined #openstack-infra | 19:52 | |
| fungi | that's way better than abandoning | 19:52 |
| jeblair | and yeah, lunch now for me | 19:52 |
| openstackgerrit | A change was merged to openstack-infra/config: Pre-cache UCA packages during nodepool img build https://review.openstack.org/99740 | 19:53 |
| fungi | the cascading enqueue idea might also save tripleo some worker resources | 19:53 |
| *** andreaf_ has joined #openstack-infra | 19:53 | |
| fungi | at the expense of more result latency of course | 19:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Add pylint job for swift3 https://review.openstack.org/99816 | 19:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Add the python-monascaclient https://review.openstack.org/99767 | 19:54 |
| clarkb | hrm https://review.openstack.org/#/c/99740/ may be mostly a noop with trusty | 19:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Create compass-monit project https://review.openstack.org/99727 | 19:54 |
| clarkb | but probably doesn't hurt to have either | 19:54 |
| *** zehicle_at_dell has quit IRC | 19:54 | |
| fungi | clarkb: unless there are stable dsvm jobs using uca on precise still | 19:55 |
| clarkb | fungi: just ironic which I odn't think so | 19:55 |
| *** e0ne has quit IRC | 19:55 | |
| fungi | yeah, then probably a solution to an obsolete problem | 19:55 |
| jeblair | there's some nice framework in there though | 19:55 |
| jeblair | (for next time around) | 19:56 |
| fungi | agreed, for caching things for alternate package repositories | 19:56 |
| fungi | when we end up needing a ppa for specific jobs or something | 19:56 |
| *** unicell has quit IRC | 19:57 | |
| clarkb | jeblair: ya I don't think it hurts | 19:57 |
| *** sabeen1 has joined #openstack-infra | 19:58 | |
| *** sabeen1 has quit IRC | 19:58 | |
| *** sabeen1 has joined #openstack-infra | 19:58 | |
| *** cnesa has quit IRC | 19:59 | |
| *** otherwiseguy has quit IRC | 19:59 | |
| *** sabeen has quit IRC | 19:59 | |
| clarkb | ok stack is approved | 20:00 |
| *** ominakov has joined #openstack-infra | 20:00 | |
| clarkb | nova docs build on trusty. I am going to do nova specs at the same time just to reduce headache on their end | 20:00 |
| fungi | yep, just approved the bottom change too | 20:00 |
| clarkb | so testing nova specs next | 20:00 |
| *** devkulkarni has quit IRC | 20:00 | |
| *** denis_makogon has joined #openstack-infra | 20:03 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Make rally-install-bare-centos6 voting again https://review.openstack.org/102620 | 20:03 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Move nova default test node to trusty. https://review.openstack.org/102621 | 20:04 |
| clarkb | fungi: jeblair ^ I am going to find lunch so up to you if you want to approve that quickly. But reviews would be great and I can approve later | 20:04 |
| *** openstackgerrit has quit IRC | 20:05 | |
| *** Longgeek has quit IRC | 20:05 | |
| *** sarob has joined #openstack-infra | 20:05 | |
| *** openstackgerrit has joined #openstack-infra | 20:06 | |
| *** CaptTofu_ has quit IRC | 20:08 | |
| *** CaptTofu_ has joined #openstack-infra | 20:08 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Split the database grants into their own class https://review.openstack.org/98586 | 20:09 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Cleanup approval requirement testing https://review.openstack.org/102029 | 20:12 |
| *** CaptTofu_ has quit IRC | 20:13 | |
| *** ArxCruz has quit IRC | 20:13 | |
| *** CaptTofu_ has joined #openstack-infra | 20:15 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add support for disk-image-builder in nodepool https://review.openstack.org/88479 | 20:15 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Only enabling rsyslog service if not chroot https://review.openstack.org/101562 | 20:15 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Don't manage iptables if we're in a choot https://review.openstack.org/102582 | 20:15 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add pre-requisites needed for diskimage-builder https://review.openstack.org/102583 | 20:15 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add additional node types to dib elements https://review.openstack.org/90565 | 20:15 |
| *** timrc is now known as timrc-afk | 20:17 | |
| *** mwagner_lap has joined #openstack-infra | 20:18 | |
| *** achanda has joined #openstack-infra | 20:18 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove freshness check from tripleo https://review.openstack.org/102618 | 20:19 |
| *** achanda_ has quit IRC | 20:20 | |
| *** jdahlin has joined #openstack-infra | 20:20 | |
| *** afazekas_ has quit IRC | 20:21 | |
| *** timrc-afk is now known as timrc | 20:25 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for tempest get_console_output bug 1329563 https://review.openstack.org/102629 | 20:26 |
| uvirtbot | Launchpad bug 1329563 in tempest "test_suspend_server_invalid_state fails with 400 response" [High,In progress] https://launchpad.net/bugs/1329563 | 20:26 |
| *** ramashri has joined #openstack-infra | 20:26 | |
| *** shayneburgess has quit IRC | 20:28 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Modularized StoryBoard Module https://review.openstack.org/98007 | 20:29 |
| mordred | krotscheck: you submitted a modularized module | 20:29 |
| mordred | krotscheck: something about that makes me unreaonably happy | 20:29 |
| krotscheck | mordred: It’s modules all the way down | 20:29 |
| mordred | krotscheck: are all of them modularized? | 20:29 |
| krotscheck | mordred: Only to make sure they’re modular. | 20:30 |
| hashar | hello folks. | 20:30 |
| *** lcheng has joined #openstack-infra | 20:32 | |
| hashar | <ad> I have a few tiny patches for Zuul that could use some review / approval. They are adding some new helpers to the Repo object. They should be harmless :) | 20:32 |
| hashar | I would love some review, even if it is nitpick / typo / whatever : https://review.openstack.org/#/q/is:open+project:openstack-infra/zuul+topic:merger,n,z | 20:32 |
| jogo | clarkb: the next version will | 20:33 |
| *** ramashri has quit IRC | 20:33 | |
| jogo | I added a patch to do that | 20:33 |
| jogo | and it just came out on Sunday! | 20:34 |
| jogo | 2.2.0 | 20:34 |
| krotscheck | Is this an intermittent pip failure? http://logs.openstack.org/45/98745/5/check/gate-storyboard-python27/a650da4/console.html | 20:34 |
| BobBall | jeblair, sdague, anteaya: Sorry - I should have talked to the PTL before voting on devstack-gate. We are indeed dependent on devstack gate for the XenServer CI and there was one instance (possibly two) where a d-g change broke the CI. I'm very keen to move more towards using all upstream repos, but that implies that more should be added to the test matrix rather than less? Short term, I actually had to branch d-g a week or so ago to | 20:34 |
| *** otherwiseguy has joined #openstack-infra | 20:34 | |
| *** maxbit has quit IRC | 20:34 | |
| *** ramashri has joined #openstack-infra | 20:34 | |
| mordred | krotscheck: what does the python class do and where does it come from? | 20:35 |
| *** maxbit has joined #openstack-infra | 20:35 | |
| mordred | BobBall: just fwiw, I have started to review your nodepool patch like 10 times and each time I went "eek, my brain is not big enough right now" | 20:36 |
| BobBall | I know :( | 20:36 |
| BobBall | it sucks | 20:36 |
| mordred | BobBall: it looks mostly ok - but I feel like I need to actually engage fully with the change - which doesn't help you, but I didn't want you to think I was ignoring you in my own head | 20:36 |
| jeblair | BobBall, mordred: i'm thinking maybe that will get a lot easier with dib? | 20:36 |
| mordred | BobBall: also - have you followed/seen the dib-nodepool work? | 20:36 |
| mordred | jeblair: jinx | 20:36 |
| BobBall | I'm afraid I'll admit to knowing _nothing_ about the dib-nodepool stuff | 20:37 |
| mordred | BobBall: it MAY make things easier | 20:37 |
| mordred | it may not - I'm not sure | 20:37 |
| krotscheck | mordred: Oh, crap. Zombie package, good catch. | 20:37 |
| krotscheck | mordred: Oh wait, I’m wrong. | 20:37 |
| krotscheck | mordred: See install_modules.sh | 20:37 |
| mordred | BobBall: but the idea is to allow us to express nodepool images using diskimage-builder and to build them locally and upload them via glance API | 20:37 |
| mordred | BobBall: which might make the work you're doing in your patch easier? | 20:38 |
| BobBall | possibly... but maybe not. Getting things uploaded through glance will be a lot easier - but getting them built through diskimage-builder may be just as painful | 20:38 |
| mordred | krotscheck: does puppi pull in python? | 20:38 |
| krotscheck | mordred: No, stankevitch-python is in install_modules. | 20:38 |
| *** devkulkarni has joined #openstack-infra | 20:38 | |
| mordred | BobBall: well, diskimage-builder can start with arbitrary images and can do as few as 0 additional things to them | 20:38 |
| * krotscheck should add that to the modulefile... | 20:39 | |
| mordred | BobBall: so if there is an image you're pulling from the internets, we might just be able to say "oh, hi, please pull that image and also put these keys on it kthxbai" | 20:39 |
| mordred | BobBall: or I may be oversimplifying greatly :) | 20:39 |
| BobBall | Because XS is running nested it can't access xenstore, which is where the IP address is put by RAX. As such, we actually have an ubuntu and XS image side by side so Ubuntu can get the IP then reboot to XS to use it | 20:39 |
| *** andreaf_ has quit IRC | 20:39 | |
| *** e0ne has joined #openstack-infra | 20:39 | |
| *** achanda has quit IRC | 20:39 | |
| *** shayneburgess has joined #openstack-infra | 20:39 | |
| mordred | oh holy bejeezus | 20:39 |
| BobBall | I see - then we may have a winner there. If I can make an image that 'just works' and give it to DIB or dib-nodepool or whatever then I'm happy | 20:39 |
| sdague | clarkb: so it looks like there might be a libvirt issue | 20:39 |
| mordred | BobBall: ++ | 20:40 |
| mordred | BobBall: but, you know, I should undersatnd your patch too | 20:40 |
| BobBall | please don't hurt me mordred ... the pain of having to do this is bad enough | 20:40 |
| clarkb | sdague ya | 20:40 |
| mordred | so taht we're actually talking about the same things | 20:40 |
| *** eharney has quit IRC | 20:40 | |
| sdague | so before rolling any more trusty nodes, it would be cool if we could actually sort out the blindspot in the ES data | 20:40 |
| clarkb | I disagree. es is weeks of work at current rate | 20:41 |
| BobBall | The patch is simple... When doing the install, we have to reboot the VM several times and so we just need to wait for the VM to say 'I'm done' after starting the script | 20:41 |
| mordred | krotscheck: oh! I see it now | 20:41 |
| clarkb | unless pvo magically appears | 20:41 |
| BobBall | but if we can avoid all of that pain and just upload an image to glance that we nab from somewhere then it'll be great | 20:41 |
| sdague | clarkb: well, the problem is we've got a 12 - 18hr delay in knowing what's actually going on right nwo | 20:41 |
| clarkb | sure I dont see that changing | 20:41 |
| sdague | ok, that just makes things terribly difficult to fix | 20:41 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Modularized StoryBoard Module https://review.openstack.org/98007 | 20:42 |
| krotscheck | There. | 20:42 |
| *** signed8bit_ZZZzz is now known as signed8bit | 20:42 | |
| clarkb | I mean I can hammer it then next time volumes slow down same issue | 20:42 |
| BobBall | mordred: do you know when dib-nodepool stuff might be usable / testable? | 20:42 |
| krotscheck | Added it to the Modulefile, but it should also be in install-modules. | 20:42 |
| BobBall | If it's near-future then perhaps it's worthwhile ignoring the nodepool changes on the assumption they will soon be obsoleted? | 20:42 |
| james_li | hi clarkb solum gate node was changed to trusty which causes our builds failed: http://logs.openstack.org/14/102614/1/check/gate-solum-devstack-dsvm/dba2917/console.html | 20:43 |
| krotscheck | Whooooey, things aren’t looking good for Ecuador. | 20:43 |
| *** e0ne has quit IRC | 20:43 | |
| mordred | krotscheck: I have given you some small feedbacks | 20:44 |
| clarkb | sdague: the issue is we need to change how we host the thing. this is hard without a quota bump | 20:44 |
| mordred | BobBall: yolanda is using it currently | 20:44 |
| clarkb | I can do it but expect at least a week to cycle through | 20:44 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: Update integration script for Apache 2.4 https://review.openstack.org/102584 | 20:44 |
| * BobBall will talk to yolanda at some point then | 20:44 | |
| jogo | clarkb: just confirmed the new flake8 version works in parallel as expected | 20:44 |
| mordred | BobBall: it still needs a couple of test cases and whatnot ... but she's used it to build devstack-gate nodes | 20:44 |
| clarkb | and even then we might still be in a bad spot | 20:44 |
| jogo | clarkb: guess I should cut another hacking this week ;) | 20:44 |
| cody-somerville | Hey. Does anyone have a link to the wiki page that lists mid-cycle sprints? | 20:45 |
| *** denis_makogon has quit IRC | 20:45 | |
| jogo | (fully backwards compat version with *no* new rules | 20:45 |
| krotscheck | mordred: I think your comments are attached to patchset 6 and are thus still in draft. | 20:45 |
| clarkb | sdague: so no magic bullet unfortunately | 20:45 |
| BobBall | Sounds useful - thanks mordred. | 20:45 |
| clarkb | might be able to limp better though | 20:46 |
| *** BobBall is now known as BobBall-Away | 20:46 | |
| sdague | clarkb: sure. I'm just really concerned we're running a 25% fail rate on tempest full at this point because the libvirt thing, and we don't have our diagnostic tools to figure out if we've got it sorted | 20:47 |
| james_li | hi clarkb could you revert solum config? | 20:47 |
| fungi | james_li: database configuration differences on trusty? (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`solum`.`assembly`, CONSTRAINT `assembly_ibfk_1` FOREIGN KEY (`plan_id`) REFERENCES `plan` (`id`))') 'DELETE FROM plan WHERE plan.id = %s' (1L,) | 20:47 |
| *** flaper87 is now known as flaper87|afk | 20:47 | |
| *** marcoemorais has quit IRC | 20:48 | |
| clarkb | sdague sure but its a known thing | 20:48 |
| clarkb | ES wouldnt help a ton either right? | 20:49 |
| clarkb | libvirt 1.2.2 breaks snapshots | 20:49 |
| clarkb | it was a risk associated with the switch | 20:49 |
| clarkb | james_li idealy no | 20:49 |
| sdague | clarkb: the problem is knowing if we fixed it | 20:50 |
| clarkb | we would revert everything together since these are integration tests | 20:50 |
| *** yjiang5_away is now known as yjiang5 | 20:52 | |
| mordred | krotscheck: nope. I made them on patchset 7 | 20:53 |
| james_li | fungi: no. thats because of a messaging problem, our services cannot get messages from queue | 20:53 |
| krotscheck | mordred: I see no inline comments :/ | 20:53 |
| james_li | fungi: we see that on trusty | 20:53 |
| fungi | james_li: oh, right, this is the rabbitmq issue you encountered on trusty which doesn't seem to manifest with other queue consuming services | 20:54 |
| clarkb | sdague: I have food now. let me hit things with a hammer afterwards and see if it is happier | 20:54 |
| clarkb | sdague: but the actual fix is either ssd's or baremetal | 20:54 |
| mordred | krotscheck: hrm | 20:54 |
| clarkb | both of which require some intervention on rax's part to not suck | 20:54 |
| sdague | clarkb: ok | 20:54 |
| *** ramashri has quit IRC | 20:54 | |
| james_li | fungi: right. thats why we want to delay the migration to trusty | 20:54 |
| mordred | krotscheck: well, just imagine the comments I made, and if you imagine the right things, you can address them ;) | 20:55 |
| clarkb | we can go to ssds without rax intervention but it will mean running the ES cluster in a degraded state while we do that | 20:55 |
| james_li | fungi: I am preparing a patch to go with F20 | 20:55 |
| clarkb | james_li: the problemm with delaying is devstack will be tested on trusty | 20:55 |
| clarkb | james_li: and everything else | 20:55 |
| *** eharney has joined #openstack-infra | 20:55 | |
| mordred | krotscheck: nope.you were right. I voted on 7, but commented on 6 | 20:55 |
| mordred | submitted | 20:55 |
| clarkb | james_li: so you will be very open to something else breaking on it | 20:55 |
| clarkb | this is why we did all the integration testing at once | 20:55 |
| clarkb | because it is integrated | 20:55 |
| *** ramashri has joined #openstack-infra | 20:56 | |
| *** ramashri has quit IRC | 20:56 | |
| james_li | clarkb: the problem is either with rabbit or solum code, but it will take time to identify, could we just switch back to precise? | 20:56 |
| *** radez is now known as radez_g0n3 | 20:56 | |
| james_li | as all out patches will see build failures | 20:57 |
| james_li | s/out/our | 20:57 |
| *** signed8bit has quit IRC | 20:57 | |
| clarkb | you can but then anything else may break you | 20:57 |
| clarkb | imo the known problem is much better | 20:57 |
| *** gokrokve_ has joined #openstack-infra | 20:58 | |
| *** devkulkarni has quit IRC | 20:58 | |
| james_li | clarkb: can we change to F20 for now? | 20:59 |
| clarkb | james_li: technicaly you can do whatever you want | 20:59 |
| clarkb | I am just trying to provide feedback on what would be useful beyond this week | 20:59 |
| *** devkulkarni has joined #openstack-infra | 20:59 | |
| clarkb | I can put you back to precise for that test | 20:59 |
| clarkb | then you can switch it to f20 | 21:00 |
| *** timrc is now known as timrc-afk | 21:00 | |
| clarkb | or whatever | 21:00 |
| clarkb | or we can make that job non voting | 21:00 |
| clarkb | and you can add an f20 job | 21:00 |
| krotscheck | mordred: What do you think an appropriate default in storyboard::application would be, emtpystring? | 21:00 |
| clarkb | that way you can test the fixes | 21:00 |
| openstackgerrit | james-li-3 proposed a change to openstack-infra/config: Migrating solum gate to F20 https://review.openstack.org/94994 | 21:01 |
| *** gokrokve has quit IRC | 21:01 | |
| clarkb | james_li: unfortunately that won't work with zuul picking nodes now | 21:01 |
| clarkb | it would be better to add a second job with f20 in the name | 21:01 |
| clarkb | and mark the existing job non voting | 21:02 |
| *** srenatus has quit IRC | 21:02 | |
| jesusaurus | clarkb: can you point me at the bit of code that retries a job if jenkins loses its connection to a slave? | 21:02 |
| *** basha has quit IRC | 21:02 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Modularized StoryBoard Module https://review.openstack.org/98007 | 21:02 |
| mordred | krotscheck: hrm. that a good question. yeah - that - or no default - what would a storyboard install look like without an openid endpoint right now? is that valid? | 21:02 |
| *** srenatus has joined #openstack-infra | 21:02 | |
| krotscheck | mordred: Nope, it’s not valid. | 21:03 |
| mordred | krotscheck: yeah. then I'd say no default. | 21:03 |
| krotscheck | mordred: Already done. | 21:03 |
| mordred | neat! | 21:03 |
| *** ramashri has joined #openstack-infra | 21:03 | |
| *** ramashri has quit IRC | 21:04 | |
| *** sabeen1 has quit IRC | 21:05 | |
| *** achanda has joined #openstack-infra | 21:05 | |
| *** ramashri has joined #openstack-infra | 21:07 | |
| *** rfolco has quit IRC | 21:08 | |
| *** timrc-afk is now known as timrc | 21:08 | |
| mordred | nibalizer: we should fix stankevitch-python at some point | 21:09 |
| *** ramashri has quit IRC | 21:09 | |
| *** ramashri has joined #openstack-infra | 21:09 | |
| clarkb | sdague: http://paste.openstack.org/show/84928/ is almost certainly part of the problem | 21:09 |
| *** mfer has quit IRC | 21:09 | |
| Alex_Gaynor | Anyone around who can help with a puppet issue? | 21:10 |
| james_li | clarkb: you mean the latest patch on https://review.openstack.org/94994 can not work? | 21:10 |
| clarkb | james_li: correct | 21:10 |
| Alex_Gaynor | Not sure how to understand http://logs.openstack.org/77/96877/29/check/gate-config-puppet-apply-precise/884c63f/console.html | 21:10 |
| *** pdmars has quit IRC | 21:10 | |
| mordred | Alex_Gaynor: gate-config-puppet-apply-precise doesn't work yet | 21:11 |
| *** ramashri has quit IRC | 21:11 | |
| clarkb | james_li: I can whip up a change once I eat through the es stuff | 21:11 |
| Alex_Gaynor | mordred: oh yeah? The error looks like it's about my patch, so I assume it was my fault :-) | 21:11 |
| mordred | oh - well, it might be | 21:11 |
| sdague | clarkb: interesting | 21:11 |
| *** ramashri has joined #openstack-infra | 21:11 | |
| james_li | clarkb: sure. if you can make a comment that will be great. | 21:11 |
| sdague | morganfainberg: do you know you are eventlet stack tracing like there is no tomorrow in keystone? | 21:11 |
| mordred | Alex_Gaynor: there is not livegrep::livegrep | 21:12 |
| morganfainberg | sdague, really? | 21:12 |
| Alex_Gaynor | mordred: I think it has something to do with https://review.openstack.org/#/c/96877/29/modules/openstack_project/manifests/livegrep.pp but I'm not sure | 21:12 |
| sdague | http://logs.openstack.org/66/99766/2/check/check-grenade-dsvm/9fd33e1/logs/old/screen-key.txt.gz?level=INFO | 21:12 |
| mordred | Alex_Gaynor: you have a livegrep | 21:12 |
| Alex_Gaynor | mordred: I totally have a module named livegrep, and a class in it named livegrep, am I doing it wrong? | 21:12 |
| sdague | you might want to download that | 21:12 |
| Alex_Gaynor | (I don't know how puppet) | 21:12 |
| sdague | it will probably kill your browser | 21:12 |
| mordred | Alex_Gaynor: nope. that's correct - but that's just livegrep | 21:13 |
| morganfainberg | sdague, nah, browser is just fine | 21:13 |
| Alex_Gaynor | mordred: so I remove the ::livegrep and it all works? | 21:13 |
| mordred | Alex_Gaynor: and when you invoke it, you want "class 'livegrep' { " not "livegrep::livegrep {" | 21:13 |
| mordred | or, rather | 21:13 |
| thingee | sdague: https://review.openstack.org/#/c/102568/1 - grenade-dsvm passed, safe to assume grenade-dsvm-partial-n-cpu was just a fluke? | 21:13 |
| mordred | Alex_Gaynor: class { '::livegrep': | 21:13 |
| morganfainberg | sdague, /me goes on record about dislikeing eventlet a lot (even more now) | 21:13 |
| Alex_Gaynor | mordred: and then where do I put the name? | 21:14 |
| sdague | thingee: no idea, look at the logs :) | 21:14 |
| mordred | lemme read your code a little more | 21:14 |
| morganfainberg | sdague, this is icehouse or havana i assume (from the /old/ path)? is juno doing the same thing? | 21:14 |
| thingee | sdague: c-api didn't come up, and thus there are no logs of c-api. All I know is it failed the sanity check. | 21:14 |
| mordred | Alex_Gaynor: you need a name param. the form you're doing is similar to a define not a class | 21:14 |
| jesusaurus | Alex_Gaynor: it looks to me like you might be creating a livegrep class and then trying to use it as a livegrep define | 21:15 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Update examples for i18n https://review.openstack.org/97344 | 21:15 |
| mordred | Alex_Gaynor: listen to jesusaurus - he's smarter in the head than me | 21:15 |
| morganfainberg | sdague, interesting broken pipe | 21:15 |
| Alex_Gaynor | jesusaurus: uhh, seems likely! | 21:15 |
| mordred | Alex_Gaynor: welcome to puppet - nothing is what you think it is going to be | 21:15 |
| sdague | morganfainberg: https://review.openstack.org/#/c/99766/ it would have been icehouse | 21:15 |
| Alex_Gaynor | mordred: I'm going to enjoy my stay | 21:15 |
| *** sballe has joined #openstack-infra | 21:16 | |
| mordred | hell yes you are | 21:16 |
| sdague | clarkb: fwiw, that's a fail though | 21:16 |
| *** julim has quit IRC | 21:16 | |
| sdague | morganfainberg: it's icehouse on trusty | 21:17 |
| sdague | let me find a working run | 21:17 |
| clarkb | sdague: yeah that was just an anomoly I noticed | 21:17 |
| clarkb | it had monopolized a worker for hours | 21:17 |
| *** weshay has quit IRC | 21:17 | |
| *** signed8bit has joined #openstack-infra | 21:18 | |
| morganfainberg | sdague, this looks like we're htting a limit (buffer size? | 21:18 |
| morganfainberg | did trusty lower limits by default? | 21:18 |
| sdague | http://logs.openstack.org/76/101976/4/check/check-grenade-dsvm/e400c7d/logs/old/screen-key.txt.gz?level=INFO is a working run | 21:18 |
| openstackgerrit | james-li-3 proposed a change to openstack-infra/config: Migrating solum gate to F20 https://review.openstack.org/94994 | 21:18 |
| sdague | it's a ton smaller | 21:18 |
| sdague | 2338 lines vs 905418 lines | 21:19 |
| *** signed8bit has quit IRC | 21:19 | |
| Alex_Gaynor | mordred, jesusaurus: So I think http://bpaste.net/show/407484/ is the way I'm supposed to do this? Now how do I pass $openstack_repos to the class? | 21:19 |
| clarkb | and I have found at least 3 nodes with no workers | 21:19 |
| clarkb | I think they may have hit files like ^ and OOMed | 21:19 |
| morganfainberg | ok hmm | 21:19 |
| *** signed8bit has joined #openstack-infra | 21:20 | |
| mordred | Alex_Gaynor: nope - you changed too much | 21:20 |
| sdague | morganfainberg: so that's a weird run, but it's an interesting boundary case | 21:20 |
| *** Ryan_Lane has joined #openstack-infra | 21:20 | |
| mordred | Alex_Gaynor: the param list in the class definition was correct | 21:20 |
| sdague | clarkb: so what's a normal max lines? | 21:20 |
| Alex_Gaynor | mordred: doh! | 21:20 |
| *** julim has joined #openstack-infra | 21:20 | |
| *** ominakov has quit IRC | 21:20 | |
| sdague | maybe we put max lines into os-loganalyze ? | 21:20 |
| mordred | Alex_Gaynor: hang on - I need to afk for a sec - I'll send you a little thing in a sec | 21:20 |
| morganfainberg | sdague, sure. its very odd though. if we weren't seeing that on precise, i'm very curious what changed | 21:20 |
| clarkb | sdague: in the 10s of thousands | 21:20 |
| clarkb | sdague: we could try something like that | 21:21 |
| *** marcoemorais has joined #openstack-infra | 21:21 | |
| sdague | clarkb: so if we only fetched the first 50k lines we'd maybe be safe? | 21:21 |
| clarkb | sdague: except gzip | 21:21 |
| clarkb | or do you mean drop the remaining lines? | 21:21 |
| sdague | yeh, drop the rest | 21:21 |
| clarkb | yeah that would probably help | 21:21 |
| sdague | &max=50000 | 21:21 |
| morganfainberg | sdague, i also notice it's almost all in /auth/tokens | 21:21 |
| sdague | &limit=50000 actually | 21:21 |
| *** dkranz has quit IRC | 21:22 | |
| morganfainberg | sdague, this... is almost screaming to me token is too big and is hittint the limit that can be buffered/sent | 21:22 |
| sdague | morganfainberg: hah | 21:22 |
| sdague | yeh, you guys exceed 8k? | 21:22 |
| Alex_Gaynor | mordred: ok, thanks a ton! | 21:22 |
| sdague | I think that's the default evenlet limit | 21:22 |
| morganfainberg | sdague, sometimes | 21:22 |
| morganfainberg | sdague, but yes. | 21:22 |
| sdague | morganfainberg: so... token size | 21:23 |
| morganfainberg | sdague it's the whole reason for the compressed token provider | 21:23 |
| morganfainberg | sdague, well more to the point, catalog | 21:23 |
| *** dprince has quit IRC | 21:23 | |
| *** HenryG has quit IRC | 21:23 | |
| *** julim has quit IRC | 21:24 | |
| sdague | so, can we get on backporting that then? Because that's kind of a big problem | 21:24 |
| sdague | if keystone just decides to stop sending tokens if the catalog is of medium size | 21:24 |
| morganfainberg | sdague, yeah :( we need to not put the catalog in the token, period | 21:24 |
| morganfainberg | that is the real issue | 21:24 |
| morganfainberg | even with compressed tokens, we could easily hit the cap | 21:25 |
| dhellmann | morganfainberg: +1 | 21:25 |
| morganfainberg | (not as easily) | 21:25 |
| sdague | morganfainberg: can you take that in as a critical issue. This seems pretty problematic | 21:25 |
| sdague | because the fail case is cloud death | 21:25 |
| sdague | as seen here | 21:25 |
| morganfainberg | sdague, i'm trying to think if we can backport PKIZ | 21:25 |
| sdague | clarkb: ok, I'm going to work on the limit fix | 21:25 |
| morganfainberg | sdague, the simplest solution is " UUID tokens" as sucky as they are (for different reasons) | 21:25 |
| morganfainberg | sdague, this is likely an issue that would need to go back to H too, right? | 21:26 |
| morganfainberg | or was H V3-ignorant enough to not matter? | 21:26 |
| morganfainberg | sdague, yeah i'll see what we can do. | 21:27 |
| morganfainberg | sdague, i really hope we can "fix this" easily | 21:27 |
| sdague | H is close enough to EOL we can probably not care | 21:27 |
| *** harlowja has joined #openstack-infra | 21:28 | |
| sdague | but icehouse also needs fixing | 21:28 |
| jeblair | sdague: your opinion on 100033 would be lovely | 21:28 |
| sdague | jeblair: looking | 21:28 |
| clarkb | sdague: ok I sweeped through the nodes and kicked them | 21:28 |
| clarkb | sdague: we should be at full strength for workers now | 21:29 |
| *** dangers is now known as dangers_away | 21:29 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Updated AngularJS to 1.2.18 https://review.openstack.org/102651 | 21:30 |
| *** devkulkarni has quit IRC | 21:30 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Search use case 1: Quicknav https://review.openstack.org/102652 | 21:30 |
| sdague | jeblair: -1, with reasons. I'm actually surprised jogo didn't abandon that one | 21:30 |
| *** Sukhdev has quit IRC | 21:31 | |
| nibalizer | mordred: what's the issue on stankevitch-python that needs to be fixed? | 21:31 |
| morganfainberg | sdague, ok i'm fine with that. i'll figure out what we can do, and i'll bring back my spec to remove the catalog from the token for J | 21:31 |
| *** andreykurilin_ has joined #openstack-infra | 21:31 | |
| morganfainberg | sdague that, i don't think i could even remotely pretend to backport to i | 21:31 |
| morganfainberg | if it is accepted | 21:31 |
| anteaya | so I'm home now but not really useful | 21:32 |
| jogo | sdague: oh I left that one around because I thought we were really close to full neutron tempest | 21:33 |
| jogo | but we are not really close so moving to WIP | 21:33 |
| anteaya | so I think I will be offline until tomorrow | 21:33 |
| openstackgerrit | james-li-3 proposed a change to openstack-infra/config: Migrating solum gate to F20 https://review.openstack.org/94994 | 21:33 |
| anteaya | my oestopath uncovered more that either of us thought, so I just need to be quiet now and not think using words | 21:34 |
| sdague | honestly, I think that whole conversation becomes a germany conversation. Especially the data we need to make decisions. | 21:34 |
| *** jaypipes has quit IRC | 21:34 | |
| *** ihrachyshka has quit IRC | 21:35 | |
| *** Ryan_Lane has quit IRC | 21:37 | |
| *** mrda-away is now known as mrda | 21:37 | |
| *** chianingwang has quit IRC | 21:38 | |
| *** esker has quit IRC | 21:38 | |
| *** Ryan_Lane has joined #openstack-infra | 21:38 | |
| jeblair | sdague: 100103 ? | 21:38 |
| *** e0ne has joined #openstack-infra | 21:39 | |
| *** miqui has quit IRC | 21:40 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] Added search interface https://review.openstack.org/99975 | 21:41 |
| clarkb | jesusaurus: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/model.py#n209 is what does it | 21:42 |
| clarkb | jesusaurus: that removes the build forcing zuul to do another build. So anywhere setResult is called after setting retry to true will do this | 21:43 |
| clarkb | jesusaurus: if you look in the gearman trigger you can see where it sets retry to true | 21:43 |
| *** gokrokve_ has quit IRC | 21:44 | |
| *** e0ne has quit IRC | 21:44 | |
| *** gokrokve has joined #openstack-infra | 21:44 | |
| openstackgerrit | A change was merged to openstack-infra/config: Move check-tripleo-ironic-undercloud-precise check queue https://review.openstack.org/100063 | 21:44 |
| clarkb | james_li: ok your turn, patch incoming shortly | 21:45 |
| *** markwash has joined #openstack-infra | 21:45 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Added tests for Project Groups API https://review.openstack.org/98745 | 21:46 |
| james_li | clarkb: is the latest one good? | 21:46 |
| clarkb | james_li: it is closer, we need to tell zuul to run that job and to run it on f20 | 21:47 |
| clarkb | james_li: I will push an update to your change | 21:47 |
| james_li | clarkb: cool thanks! | 21:47 |
| clarkb | james_li: do you want to run on trusty as non voting? | 21:47 |
| clarkb | james_li: this may help you fix the rabbit problems | 21:47 |
| james_li | clarkb: yes as non voting | 21:48 |
| james_li | thanks | 21:48 |
| *** gokrokve_ has joined #openstack-infra | 21:48 | |
| clarkb | ok | 21:48 |
| openstackgerrit | Kurt Taylor proposed a change to openstack-infra/config: Script to upload log files to swift https://review.openstack.org/102655 | 21:48 |
| *** gokrokve has quit IRC | 21:49 | |
| *** chuckC has quit IRC | 21:49 | |
| sdague | jeblair: yeh, and the ML thread that created that | 21:49 |
| *** mmaglana has quit IRC | 21:50 | |
| morganfainberg | sdague, do you have a bug i can reference on the eventlet OMG THATS A HUGE LOG issue? | 21:51 |
| *** plars has quit IRC | 21:51 | |
| jeblair | sdague: don't understand; oh i think i may have erred in abbreviating my question earlier... | 21:51 |
| jeblair | sdague: expanded: can you also please take a look at 100103 when you have a moment? | 21:51 |
| *** _nadya_ has quit IRC | 21:52 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Migrating solum gate to F20 https://review.openstack.org/94994 | 21:52 |
| clarkb | jeblair: fungi ^ something like that should unblock solum | 21:52 |
| jeblair | sdague: (i'm guessing you read that as 100033 and thought i was continuing the previous convo) | 21:52 |
| krtaylor | clarkb, fyi, that last patchset of mine is the swift upload script we use, it would need to be configured for our community service | 21:52 |
| *** gokrokve_ has quit IRC | 21:52 | |
| jeblair | clarkb: i'm only reviewing change numbers that are composed entirely of the numerals 0,1, and 3 right now. | 21:52 |
| sdague | jeblair: oh, sorry, yeh, read it wrong | 21:52 |
| *** gokrokve has joined #openstack-infra | 21:52 | |
| sdague | +1 on heat slow voting | 21:53 |
| clarkb | jeblair: ok | 21:53 |
| *** r1chardj0n3s has joined #openstack-infra | 21:54 | |
| jeblair | (i was joking, but it turns out i actually have more changes that match that pattern to review) | 21:55 |
| *** plars has joined #openstack-infra | 21:55 | |
| fungi | clarkb: should we take their nv job out of the gate pipeline for now? | 21:55 |
| clarkb | fungi: whose? | 21:56 |
| clarkb | oh sure | 21:56 |
| clarkb | sorry I am juggling all the things right now | 21:56 |
| fungi | clarkb: 94994 | 21:56 |
| clarkb | brain is sad | 21:56 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Migrating solum gate to F20 https://review.openstack.org/94994 | 21:56 |
| clarkb | fungi: ^ thanks | 21:56 |
| fungi | clarkb: lgtm | 21:57 |
| * fungi needs to get to new house chores, then get back to updating the atc scripts for new gerrit db schema and new governance programs.yaml format | 21:57 | |
| james_li | Thanks fungi clarkb | 21:58 |
| *** marcoemorais has quit IRC | 21:58 | |
| *** david-lyle has quit IRC | 21:59 | |
| SlickNik | fungi / clarkb: Question when you guys have a minute. | 21:59 |
| *** marcoemorais has joined #openstack-infra | 21:59 | |
| *** marcoemorais has quit IRC | 21:59 | |
| SlickNik | fungi / clarkb: Have we turned of upload of drafts to review.o.o? | 21:59 |
| clarkb | SlickNik: go for it, before I start the next thing :) | 21:59 |
| clarkb | SlickNik: yes | 21:59 |
| *** marcoemorais has joined #openstack-infra | 21:59 | |
| *** david-lyle has joined #openstack-infra | 21:59 | |
| SlickNik | clarkb: Okay. That was easy. :) | 21:59 |
| *** plars has quit IRC | 22:00 | |
| clarkb | sdague: the backlog is slowly falling. I think this means that when everything is working we can stay afloat | 22:00 |
| clarkb | sdague: trouble is when things break... | 22:00 |
| *** dims__ has quit IRC | 22:01 | |
| SlickNik | clarkb: Was it breaking something? Curious to what the reasoning for that was. | 22:01 |
| clarkb | SlickNik: yes drafts break everything | 22:02 |
| SlickNik | Not that it's something that I can't live without. :) | 22:02 |
| *** yjiang5 is now known as yjiang5_away | 22:02 | |
| clarkb | they provided an inconsistent user experience with false assumptions of security | 22:02 |
| *** yjiang5_away has quit IRC | 22:02 | |
| clarkb | this leads to people using them for purposes tehy aren't suited for and confusion when drafts are in use and reviewers can't see draft patchsets | 22:02 |
| clarkb | SlickNik: there was a thread on the ML about it a month or so ago | 22:03 |
| *** andreykurilin_ has quit IRC | 22:03 | |
| SlickNik | clarkb: makes sense; I've run into exactly those sort of scenarios. | 22:03 |
| clarkb | specifically if you go to review a draft and can't see the draft patchset you don't get vote buttons | 22:03 |
| clarkb | that confuses people | 22:03 |
| clarkb | another point of confusion is people think drafts are secure and private, but anyone can fetch them | 22:03 |
| *** david-lyle has quit IRC | 22:03 | |
| clarkb | you can also base a non draft on a draft | 22:04 |
| openstackgerrit | A change was merged to openstack-infra/config: heat slow voting again https://review.openstack.org/100103 | 22:04 |
| clarkb | that really breaks things when it comes to merging | 22:04 |
| clarkb | and as I discovered yesterday draft changes ona branch prevent you from deleting that branch | 22:04 |
| *** nikhil___ is now known as nikhil___|afk | 22:04 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fixed typo: 'project.yaml' -> 'projects.yaml' https://review.openstack.org/100305 | 22:04 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Add berrange to nova-specs-core https://review.openstack.org/102659 | 22:05 |
| *** hogepodge has quit IRC | 22:05 | |
| SlickNik | Wow, turning it off definitely seems a good idea. | 22:05 |
| SlickNik | I missed that thread on the ML; thanks for the update. :) | 22:05 |
| *** shayneburgess has quit IRC | 22:06 | |
| jeblair | clarkb: is 101088 an ongoing concern? | 22:07 |
| *** plars has joined #openstack-infra | 22:07 | |
| *** ramashri has quit IRC | 22:07 | |
| jeblair | (i've moved to 0,1,8 btw; an 8 is like 2 3's) | 22:07 |
| clarkb | jeblair: it was when I pushed teh change. I can double check | 22:08 |
| clarkb | I know mordred was going to tell peopel to not dothat | 22:08 |
| clarkb | maybe they fixed it | 22:08 |
| *** ramashri has joined #openstack-infra | 22:08 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 22:08 | |
| *** markmcclain has quit IRC | 22:08 | |
| *** james_li has quit IRC | 22:08 | |
| jeblair | clarkb: i guess i'm partly asking because it seems like a change that would cause us not to build new images in a week so i would have expected us to approve that quickly -- but maybe we're so behind that even urgent changes sit for a bit | 22:09 |
| *** signed8bit_ZZZzz is now known as signed8bit | 22:09 | |
| *** eharney has quit IRC | 22:09 | |
| *** wenlock has quit IRC | 22:09 | |
| mattoliverau | Morning all | 22:10 |
| clarkb | jeblair: confirmed Ubuntu Server 12.04 LTS (amd64 20140606) - Partner Image is the image to use the image we were using does not exist | 22:10 |
| clarkb | sdague: we have fallen ~13k events since I kicked the workers | 22:12 |
| *** chuckC has joined #openstack-infra | 22:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: Migrating solum gate to F20 https://review.openstack.org/94994 | 22:15 |
| *** jhesketh has joined #openstack-infra | 22:16 | |
| jhesketh | Morning | 22:17 |
| *** shayneburgess has joined #openstack-infra | 22:17 | |
| clarkb | jhesketh: o/ | 22:17 |
| *** sabeen1 has joined #openstack-infra | 22:17 | |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Remove irc notification from puppet-openstack_dev_env https://review.openstack.org/100653 | 22:18 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add berrange to nova-specs-core https://review.openstack.org/102659 | 22:18 |
| *** achanda has quit IRC | 22:18 | |
| *** _nadya_ has joined #openstack-infra | 22:19 | |
| *** achanda_ has joined #openstack-infra | 22:20 | |
| jesusaurus | clarkb: sweet, thanks for the link | 22:23 |
| fungi | clarkb: for future reference, this is the uuid list (and canonical publishes detached signatures for it too)... https://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:hpcloud.json | 22:24 |
| fungi | clarkb: sigs are in the same dir i beieve | 22:24 |
| fungi | believe | 22:24 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "disable voting on gate-tempest-dsvm-virtual-ironic" https://review.openstack.org/100623 | 22:24 |
| clarkb | fungi: the image we are switching to isn't on that list :/ | 22:25 |
| fungi | clarkb: that's... troubling | 22:25 |
| *** jlk has left #openstack-infra | 22:25 | |
| fungi | zul: ^ ? | 22:25 |
| clarkb | ya | 22:25 |
| *** dims__ has joined #openstack-infra | 22:25 | |
| clarkb | 'Ubuntu Server 12.04 LTS (amd64 20140606) - Partner Image' is the name hpcloud side | 22:25 |
| reed | anteaya, regarding https://review.openstack.org/#/c/101227/, is there somewhere also specified the ultimate goal of a 3rd party CI? | 22:25 |
| zul | fungi: hmm? | 22:25 |
| openstackgerrit | A change was merged to openstack-infra/config: Create a marconi-specs repository https://review.openstack.org/100633 | 22:26 |
| fungi | zul: hpcloud claims to have an ubuntu-provided partner image which is not in the signed list | 22:26 |
| reed | anteaya, for example, where do we explain folks that setting one up is good for the whole community, 3rd party included? | 22:26 |
| *** ramashri has quit IRC | 22:26 | |
| zul | fungi: lemme ask around | 22:26 |
| fungi | clarkb: do you have the uuid handy? | 22:26 |
| clarkb | 75d47d10-fef8-473b-9dd1-fe2f7649cb41 | 22:26 |
| reed | anteaya, I'd like to make it clear to anyone who's setting up a CI that they're expected to participate in openstack, not just dump something out there to check a box | 22:26 |
| sdague | clarkb: win | 22:27 |
| sdague | ok, I need to start making dinner here. I'll work on limit= for os-loganalyze in the morning with a clear head | 22:28 |
| clarkb | sdague: I will make it my "coffee" activity to scan the workers in the morning | 22:28 |
| krtaylor | reed, I was poking at that a bit, thinking maybe we could at least require a person associated with a service account to have signed the icla | 22:28 |
| clarkb | and hopefully we can identify any other kilelrs of workers and work through them | 22:28 |
| sdague | clarkb: sounds great | 22:28 |
| sdague | thanks much | 22:28 |
| openstackgerrit | ChingWei Chang proposed a change to openstack-infra/config: Prepared for new project puppet-n1k-vsm https://review.openstack.org/102662 | 22:29 |
| reed | krtaylor, why would that be needed? | 22:29 |
| reed | krtaylor, I mean, if they have to commit code to our repos they have to (currently) | 22:30 |
| krtaylor | reed, its not, I was just thinking of ways to push forward what you were saying | 22:30 |
| krtaylor | reed, right | 22:30 |
| reed | krtaylor, oh, I see... well, I am not sure that more burocracy would result in them understanding anything more than more burocracy | 22:30 |
| reed | I think we should have a conversation on the list though | 22:31 |
| krtaylor | reed, I think it would come down to whether they were active in the community as to whether they would be allowed to turn on voting | 22:31 |
| krtaylor | reed, this is why we need to write this stuff down | 22:32 |
| reed | krtaylor, yep, we need to write this stuff down | 22:32 |
| *** devkulkarni has joined #openstack-infra | 22:32 | |
| krtaylor | not sure what happened to anteaya, she had to call it quits for today, hope she's ok | 22:32 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for neutron bug 1334109 https://review.openstack.org/102558 | 22:34 |
| uvirtbot | Launchpad bug 1334109 in neutron "Lock wait timeout updating router's gw_port_id" [High,New] https://launchpad.net/bugs/1334109 | 22:34 |
| *** sarob has quit IRC | 22:34 | |
| *** adrian_otto has quit IRC | 22:35 | |
| *** doude has quit IRC | 22:35 | |
| *** openstackgerrit has quit IRC | 22:35 | |
| *** devkulkarni has quit IRC | 22:35 | |
| *** devkulkarni has joined #openstack-infra | 22:35 | |
| *** openstackgerrit has joined #openstack-infra | 22:36 | |
| *** markwash_ has joined #openstack-infra | 22:37 | |
| *** lcheng has quit IRC | 22:37 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Move neutron default test node to trusty. https://review.openstack.org/102663 | 22:37 |
| *** e0ne has joined #openstack-infra | 22:39 | |
| clarkb | I am only proposing those changes after I have done work on my side to test on bare-trusty nodes | 22:40 |
| *** markwash has quit IRC | 22:40 | |
| *** markwash_ is now known as markwash | 22:40 | |
| clarkb | but maybe we hold off while nova discusses libvirt | 22:40 |
| *** hashar has quit IRC | 22:40 | |
| *** e0ne has quit IRC | 22:44 | |
| jeblair | clarkb: is that discussion happening now? | 22:44 |
| jeblair | clarkb: i'm about to +2 102621; should i avoid aprving it? | 22:44 |
| *** lcheng has joined #openstack-infra | 22:44 | |
| *** markmcclain has joined #openstack-infra | 22:45 | |
| jeblair | jhesketh: see 102655 | 22:45 |
| *** hogepodge has joined #openstack-infra | 22:45 | |
| clarkb | jeblair: ya lets wait on 102621 it is happening now | 22:45 |
| jeblair | clarkb: where? | 22:45 |
| clarkb | though it has wound down without much decisioning | 22:45 |
| clarkb | jeblair: over in -nova | 22:45 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove pushMerge from keystoneclient refs/for/refs acl https://review.openstack.org/101591 | 22:45 |
| *** jdahlin has quit IRC | 22:46 | |
| openstackgerrit | A change was merged to openstack-infra/config: Update HPCloud Precise image name https://review.openstack.org/101088 | 22:46 |
| openstackgerrit | A change was merged to openstack-infra/config: Configure apt to retry more on failures https://review.openstack.org/101944 | 22:46 |
| openstackgerrit | A change was merged to openstack-infra/config: remove /rechecks https://review.openstack.org/102524 | 22:46 |
| *** serverascode has quit IRC | 22:47 | |
| clarkb | jeblair: I can reorder and we can do neutron first potentially | 22:47 |
| jhesketh | jeblair: interesting way of getting the full console log... might be worth pulling that part into the other script(s) we have | 22:48 |
| clarkb | or maybe we just keep pushing for trusty to make nova fix it >_> | 22:48 |
| clarkb | jhesketh: oh? is this krtaylors thing? | 22:48 |
| clarkb | I haven't looked at it yet | 22:48 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for tempest get_console_output bug 1329563 https://review.openstack.org/102629 | 22:48 |
| uvirtbot | Launchpad bug 1329563 in tempest "test_suspend_server_invalid_state fails with 400 response" [High,In progress] https://launchpad.net/bugs/1329563 | 22:48 |
| mordred | clarkb, jeblair: I asked them to not do that, but they're probably going to keep doing it | 22:48 |
| jeblair | mordred: breaking libvirt? | 22:49 |
| mordred | jeblair: oh, I was talking about cloud image names changing | 22:49 |
| *** markmcclain has quit IRC | 22:49 | |
| jhesketh | clarkb: yep | 22:49 |
| jeblair | mordred: ask all the people to stop breaking all the things all the time | 22:49 |
| clarkb | ianw: fwiw I am very excited for your nodepool change after an initial skim | 22:49 |
| *** jdahlin has joined #openstack-infra | 22:49 | |
| mordred | jeblair: but now that I think about it - libvirt will also keep breaking | 22:49 |
| jeblair | i reviewed 100 changes today. i'm done. | 22:49 |
| mordred | jeblair: woot | 22:49 |
| clarkb | ianw: I need to properly review it but I THink the approach is very good | 22:50 |
| clarkb | jeblair: nice | 22:50 |
| *** wenlock has joined #openstack-infra | 22:50 | |
| *** serverascode has joined #openstack-infra | 22:50 | |
| *** sabeen1 has quit IRC | 22:50 | |
| *** dkliban is now known as dkliban_afk | 22:51 | |
| *** signed8bit has quit IRC | 22:52 | |
| mordred | jeblair: quick follow up question on your review on the ansible stack ... | 22:53 |
| mordred | jeblair: one of the options that is available on the host list is to have an inventory provider that would know how to do "puppet cert list" | 22:53 |
| mordred | jeblair: but doing that dynamically clearly would have a perf hit on using it as a tool | 22:53 |
| *** mriedem has quit IRC | 22:54 | |
| mordred | jeblair: would you prefer I fix it in that way which will always be perfect but slightly slower - or in a way that's created by puppet/cron and written to the file? | 22:54 |
| *** srenatus has quit IRC | 22:55 | |
| *** srenatus has joined #openstack-infra | 22:56 | |
| *** asettle has joined #openstack-infra | 22:56 | |
| openstackgerrit | A change was merged to openstack-infra/config: Update jenkins gearman plugin version https://review.openstack.org/102019 | 22:56 |
| openstackgerrit | A change was merged to openstack-infra/config: Add rally devstack integration to rally-cli job https://review.openstack.org/101570 | 22:56 |
| openstackgerrit | A change was merged to openstack-infra/config: Install curl when logstash watchdog is used. https://review.openstack.org/102588 | 22:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Make Marconi python33 job voting https://review.openstack.org/101709 | 22:59 |
| *** mdenny has quit IRC | 22:59 | |
| *** pcrews_ has quit IRC | 23:00 | |
| *** pcrews has joined #openstack-infra | 23:00 | |
| *** markmcclain has joined #openstack-infra | 23:01 | |
| openstackgerrit | A change was merged to openstack-infra/config: Added pylint job to Blazar https://review.openstack.org/102344 | 23:02 |
| *** marcoemorais has quit IRC | 23:06 | |
| *** marcoemorais has joined #openstack-infra | 23:06 | |
| *** marcoemorais has joined #openstack-infra | 23:07 | |
| *** UtahDave has quit IRC | 23:09 | |
| *** otherwiseguy has quit IRC | 23:14 | |
| *** Sukhdev has joined #openstack-infra | 23:14 | |
| *** adrian_otto has joined #openstack-infra | 23:19 | |
| ianw | clarkb: thanks; i feel like it could be extended to things like your PID idea easily enough. but i wanted to make something easily digested first :) | 23:21 |
| *** timrc is now known as timrc-afk | 23:23 | |
| clarkb | ianw: and just reviewed | 23:24 |
| clarkb | ianw: yup I like it | 23:24 |
| clarkb | ianw: I did -1 beacuse one of the test things (do_it) wasn't clear to me | 23:24 |
| clarkb | and was hoping for more explanation in the code | 23:24 |
| clarkb | but other than that mostly trying to have a discussion about the code but it looks good | 23:24 |
| clarkb | ianw: also it is entirely possible my brain has melted and my -1 would have been avoided if I groked what you were saying with the existing comment there | 23:25 |
| *** r1chardj0n3s has quit IRC | 23:25 | |
| *** atiwari has quit IRC | 23:27 | |
| ianw | clarkb: do_it() simulates the rounds of allocation ; it's analagous to getNeededNodes https://github.com/openstack-infra/nodepool/blob/master/nodepool/nodepool.py#L1223 | 23:29 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Move ceilometer default test node to trusty. https://review.openstack.org/102672 | 23:30 |
| clarkb | ianw: right but why does that happen in setUp | 23:30 |
| *** rlandy has quit IRC | 23:30 | |
| clarkb | ianw: it doesn't seem to assert anything. | 23:30 |
| *** timrc-afk is now known as timrc | 23:30 | |
| clarkb | ianw: or is that to reduce the amount of code needed in tests themselves? | 23:30 |
| *** r1chardj0n3s has joined #openstack-infra | 23:30 | |
| *** oomichi has joined #openstack-infra | 23:30 | |
| clarkb | eglynn-reg-offic: ^ 102672 should interest you | 23:31 |
| mordred | clarkb: this is all it takes: http://paste.openstack.org/show/84933/ | 23:31 |
| mordred | clarkb: to do the dynamic inventory for ansible based on puppet cert list output | 23:31 |
| mordred | clarkb: it's possible I should have just tried writing that before | 23:31 |
| clarkb | mordred: :) cool | 23:32 |
| *** adrian_otto has quit IRC | 23:32 | |
| ianw | clarkb: i guess it falls out of using the scenarios plugin ... setUp is creating self.allocations[] which we then check via RoundRobinTestCase https://review.openstack.org/#/c/101110/6/nodepool/tests/__init__.py | 23:33 |
| *** Ryan_Lane has quit IRC | 23:33 | |
| clarkb | ianw: I see | 23:34 |
| clarkb | you are doing the test in setUp then asserting results in the actual "test" | 23:34 |
| clarkb | clearly my brain is toast. Could you udpate the comment to say something like self.allocations will be checked in RoundRobinTestCase.fo? | 23:34 |
| ianw | clarkb: yeah ... i really just followed along with everything else in there | 23:35 |
| clarkb | ianw: ^ or am I still overthinking this? | 23:35 |
| clarkb | ianw: if you think the comment is good as is i can update my vote | 23:35 |
| ianw | clarkb: well maybe in isolation my change seems odd, but i think if you look at the rest of the tests in that file, it is consistent? | 23:36 |
| * clarkb looks | 23:36 | |
| clarkb | ah yup | 23:36 |
| clarkb | they just don't do it with a closure. I will update my review | 23:37 |
| clarkb | done | 23:37 |
| *** sballe has quit IRC | 23:38 | |
| *** e0ne has joined #openstack-infra | 23:39 | |
| *** reed has quit IRC | 23:39 | |
| *** SumitNaiksatam has quit IRC | 23:39 | |
| ianw | clarkb: i did want to use assertListEqual as suggested by lifeless, but it's not 2.6 compatible and i wasn't sure about the unittest2 situation | 23:39 |
| phschwartz | Hello from 33k feet. | 23:39 |
| clarkb | ianw: is that in testtools? | 23:40 |
| clarkb | we tend to prefer testtools over unittest2 because of the python26 situation | 23:40 |
| clarkb | phschwartz: hello | 23:40 |
| ianw | clarkb: i think so, but that would require more of a rewrite? | 23:41 |
| clarkb | ianw: maybe? | 23:41 |
| clarkb | we already use testtools | 23:41 |
| *** wenlock has quit IRC | 23:41 | |
| notmyname | is there such thing as a "release" of a -specs repo? | 23:41 |
| clarkb | notmyname: I don't think so | 23:42 |
| notmyname | ok | 23:42 |
| notmyname | I'm wondering about AUTHORS and ChangeLog files in a -specs repo and what that even means | 23:42 |
| clarkb | notmyname: I believe the intent is to continuously publish them | 23:42 |
| notmyname | right | 23:42 |
| *** lcheng has quit IRC | 23:43 | |
| *** e0ne has quit IRC | 23:43 | |
| clarkb | ianw: fwiw I don't think little nits like that are worth it on a change like yours | 23:43 |
| clarkb | ianw: the actual allocations are what is interesting | 23:44 |
| ianw | clarkb: it was actually a prior change (https://review.openstack.org/#/c/101068/). but now i want to understand why testtools assertListEqual wasn't picked up in http://logs.openstack.org/10/101110/5/check/gate-nodepool-python26/c3c2d03 | 23:45 |
| clarkb | ianw: I think you may need https://testtools.readthedocs.org/en/latest/for-test-authors.html#equals | 23:46 |
| *** r-daneel has joined #openstack-infra | 23:48 | |
| *** hemna is now known as hemna_ | 23:49 | |
| clarkb | mordred: what do you think about moving nova to trusty for unittests? | 23:49 |
| ianw | yeah, i thought it would "just work" ... :All of the assertions that you can find in Python standard unittest can be found in testtools (remember, testtools extends unittest)" | 23:49 |
| clarkb | mordred: should I possibly flip the order so that neutron and ceilometer happe nfirst? | 23:49 |
| clarkb | ianw: ya I would have expected it to work too | 23:49 |
| morganfainberg | clarkb, sdague, do we capture dmesg in gate runs? grenate / dsvm specifically? | 23:49 |
| clarkb | we capture syslog and kern.log | 23:50 |
| * morganfainberg doesn't see a dmesg, but... should ask instead of assuming we don't | 23:50 | |
| morganfainberg | ah | 23:50 |
| morganfainberg | ok | 23:50 |
| ianw | morganfainberg: yeah see -> https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L437 | 23:51 |
| morganfainberg | ianw, yeah found it | 23:51 |
| morganfainberg | oomkiller would show up in kern? or syslog right? | 23:52 |
| clarkb | ya syslog | 23:52 |
| morganfainberg | darn, trying to figure out that EPIPE issue | 23:52 |
| morganfainberg | still | 23:52 |
| *** r-daneel has quit IRC | 23:52 | |
| morganfainberg | it's not eventlet, it's not actually token size | 23:52 |
| morganfainberg | it's the client hanging up | 23:52 |
| morganfainberg | eventlet is just letting us know (in an obnoxious way) the client hung up. | 23:53 |
| *** asettle has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
