| fungi | oh wow. that seems beyond inefficient | 00:00 |
|---|---|---|
| * clarkb is currently wrapping his head around readPacket to attempt at rewriting it with larger read buffers | 00:00 | |
| jeblair | clarkb: i think we may want to do that, but i'm also not convinced this is a big enough problem to explain what we're seeing | 00:00 |
| jeblair | fungi: thoughts on whether to roll back? | 00:01 |
| *** bknudson has joined #openstack-infra | 00:02 | |
| *** bknudson has quit IRC | 00:02 | |
| *** dims has quit IRC | 00:02 | |
| *** bknudson has joined #openstack-infra | 00:02 | |
| fungi | i think it depends on whether we're able to reproduce outside production. would we benefit from some more targeted debugging.profiling data from nodepool-dev before switching back? | 00:02 |
| fungi | debugging/profiling | 00:02 |
| *** Sukhdev has quit IRC | 00:02 | |
| jeblair | fungi: i think we've got what we need for the moment; i'm optimistic we can produce it outside of production | 00:03 |
| *** Sukhdev has joined #openstack-infra | 00:03 | |
| jeblair | also, switching will be a little easier now -- i think we can leave zuul+gear as-is, so we only have to change nodepool | 00:04 |
| *** ddieterly has quit IRC | 00:04 | |
| *** pc_m has quit IRC | 00:05 | |
| fungi | in that case, i'm happy to help switch back now, unless there's a significantly increased chance we'll be able to hack nodepool-dev the rest of the way to happy in short order | 00:05 |
| clarkb | jeblair: agreed, zuul is fine as is | 00:05 |
| jeblair | clarkb: keep in mind that method services both blocking and non-blocking io, and we're now concerned with its use in the blocking context | 00:07 |
| *** e0ne has quit IRC | 00:07 | |
| jeblair | clarkb, fungi: okay, shall we revert nodepool? | 00:07 |
| jeblair | now | 00:07 |
| fungi | i can stop nodepoold on nodepool-dev and begin deleting ready/building nodes | 00:08 |
| clarkb | jeblair: yup. Also I notice that isComplete() for admin requests isn't super performant either | 00:08 |
| clarkb | we basically go one byte at a time and check against the end of message bytes | 00:09 |
| jeblair | i think that looks like: stop new nodepool, delete ready/building; start old nodepool; wait an hour and delete the remaing nodes? | 00:09 |
| *** zz_sabari_ is now known as sabari | 00:09 | |
| fungi | yep, i can stick around an hour or so to do the cleanup too | 00:09 |
| fungi | i didn't have anything pressing this evening | 00:10 |
| fungi | any objection to me stopping nodepoold on nodepool-dev and getting started with that sequence now? | 00:10 |
| *** rkukura has quit IRC | 00:10 | |
| clarkb | jeblair: that sounds right | 00:10 |
| clarkb | fungi: none from me | 00:10 |
| jeblair | clarkb: sure, there are lots of things that could be improved, but i really don't think that's the underlying cause -- these things are all true on the precise server too | 00:11 |
| clarkb | jeblair: yup | 00:11 |
| jeblair | fungi: sounds good | 00:11 |
| *** e0ne has joined #openstack-infra | 00:11 | |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking https://review.openstack.org/140163 | 00:12 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/devstack-gate: Introduce support for running IPv6 networking in Neutron https://review.openstack.org/140128 | 00:12 |
| jeblair | clarkb, fungi: i think i will clone the current prod nodepool to another server and use it to benchmark the fake tests | 00:12 |
| *** boris-42 has quit IRC | 00:12 | |
| *** annegent_ has quit IRC | 00:13 | |
| *** armax has quit IRC | 00:13 | |
| jeblair | or perhaps spin up a new one and try to get the versions to match | 00:13 |
| fungi | ready and building nodes are on their way out now | 00:14 |
| jeblair | i have to run out for a bit; bbl | 00:15 |
| *** pc_m has joined #openstack-infra | 00:15 | |
| fungi | k | 00:16 |
| *** sarob has quit IRC | 00:16 | |
| *** otter768 has joined #openstack-infra | 00:18 | |
| *** ivar-laz_ has joined #openstack-infra | 00:18 | |
| *** ivar-laz_ has quit IRC | 00:19 | |
| *** sabari is now known as zz_sabari | 00:20 | |
| *** ivar-laz_ has joined #openstack-infra | 00:20 | |
| *** ivar-lazzaro has quit IRC | 00:21 | |
| *** otter768 has quit IRC | 00:22 | |
| *** lttrl has quit IRC | 00:24 | |
| *** rkukura has joined #openstack-infra | 00:25 | |
| *** craigbr has quit IRC | 00:25 | |
| dougwig | mtreinish: that ironic/ssh failure is odd. you don't have the lbaas_agent_scheduler extension listed, which the test is checking for. | 00:26 |
| *** signed8bit is now known as signed8bit_ZZZzz | 00:26 | |
| *** armax has joined #openstack-infra | 00:28 | |
| mtreinish | dougwig: it looks like the conditional I put in devstack-gate doesn't affect the ironic job: http://logs.openstack.org/64/140164/3/check/check-tempest-dsvm-ironic-pxe_ssh/607617f/logs/tempest_conf.txt.gz | 00:30 |
| mtreinish | it's not getting set in the config file | 00:30 |
| dougwig | ahh, that'd explain it. | 00:30 |
| fungi | clarkb: all ready/building nodes on nodepool-dev have been deleted now | 00:31 |
| *** salv-orlando has joined #openstack-infra | 00:31 | |
| *** armax has quit IRC | 00:31 | |
| mtreinish | dougwig: actually, I'm not sure it's doing for any of the jobs: http://logs.openstack.org/64/140164/3/check/check-tempest-dsvm-neutron-full/bd124a7/logs/tempest_conf.txt.gz | 00:31 |
| fungi | clarkb: we're set for me to start nodepoold on nodepool.o.o? | 00:32 |
| fungi | going once | 00:32 |
| fungi | twice | 00:32 |
| * anteaya hands fungi a gavel | 00:32 | |
| *** armax has joined #openstack-infra | 00:33 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/gear: Read more bytes per readPacket iteration https://review.openstack.org/140186 | 00:33 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Create an experimental job for Neutron with IPv6 Networking https://review.openstack.org/140163 | 00:33 |
| clarkb | jeblair: fungi ^ something like that maybe? it does pass unittests | 00:34 |
| *** sarob has joined #openstack-infra | 00:34 | |
| clarkb | fungi: sold! | 00:34 |
| *** sarob has quit IRC | 00:34 | |
| *** Sukhdev_ has joined #openstack-infra | 00:34 | |
| fungi | clarkb: started | 00:34 |
| dougwig | mtreinish: hmm, we haven't broken openstack/neutron yet, so that ironic failure is likely a transient thing. | 00:34 |
| fungi | in ~ an hour i'll clean up remaining nodes on nodepool-dev | 00:35 |
| dougwig | but, not good that the change isn't being applied. | 00:35 |
| *** sarob has joined #openstack-infra | 00:35 | |
| *** Sukhdev_ has quit IRC | 00:35 | |
| mtreinish | dougwig: no you turned off the services in features.yaml | 00:35 |
| mtreinish | so it's the same as breaking it | 00:35 |
| clarkb | fungi: sounds good let me know if you want me to babysit that so you can have an evening | 00:35 |
| *** Sukhdev has quit IRC | 00:36 | |
| dougwig | ahh, right. | 00:36 |
| fungi | clarkb: if you have time to keep an eye on things that's great, though i'm also around and near the computer if you want to keep hacking on gear/nodepool | 00:36 |
| *** stevemar has quit IRC | 00:36 | |
| fungi | mostly just catching up on e-mail since i didn't get around to it earlier | 00:36 |
| clarkb | fungi: I think my gear patch is as good as it will get for now until people review it | 00:37 |
| *** stevemar has joined #openstack-infra | 00:37 | |
| clarkb | the non admin stuff was easy beacuse packets have a length header | 00:37 |
| mtreinish | dougwig: got it, it's the whitespace without quotes in the localrc: http://logs.openstack.org/64/140164/3/check/check-tempest-dsvm-neutron-full/bd124a7/logs/localrc.txt.gz | 00:37 |
| mtreinish | err, maybe | 00:37 |
| *** Masahiro has joined #openstack-infra | 00:38 | |
| dougwig | heh, i love debugging CI's with long cycle times. | 00:38 |
| fungi | dougwig: very simple solution to that problem--make openstack lots faster (solution left as an exercise for the reader) | 00:38 |
| dougwig | /dev/potato -- writes are very fast. | 00:39 |
| *** Sukhdev has joined #openstack-infra | 00:40 | |
| mtreinish | dougwig: yep that's eactly what it was: http://logs.openstack.org/64/140164/3/check/check-tempest-dsvm-neutron-full/bd124a7/console.html#_2014-12-08_23_41_43_841 | 00:41 |
| mtreinish | I'll drop the whitespace and try again | 00:42 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas https://review.openstack.org/140164 | 00:42 |
| mtreinish | dougwig: ^^^ | 00:42 |
| *** Masahiro has quit IRC | 00:43 | |
| *** annegent_ has joined #openstack-infra | 00:43 | |
| dougwig | i bet you're right. | 00:43 |
| *** amitgandhinz has joined #openstack-infra | 00:44 | |
| *** andreykurilin_ has quit IRC | 00:44 | |
| clarkb | wait we were testing lbaas, fwaas, and vpnaas? | 00:45 |
| clarkb | mtreinish: and we can straight up revert that change when lbaas et al are working? | 00:46 |
| mestery | Question for infra folks: I'm trying to do a merge commit for a stackforge project, collapsing a local feature branch into master, and getting an error around commits from other authors. Anyone seen this? http://paste.openstack.org/show/147562/ | 00:46 |
| mestery | fungi: IF you have a moment sir ^^^ | 00:46 |
| fungi | mestery: yes, trying to push commits which lack you as the git committer will get rejected due to lack of "forge commiter identity" permission in the acl | 00:48 |
| *** amitgandhinz has quit IRC | 00:48 | |
| mtreinish | clarkb: sigh, yes. In fact for a long time the hack of an lbaas scenario was the top gate bug | 00:48 |
| fungi | mestery: this is why we have a repo import feature in project creation | 00:48 |
| mtreinish | clarkb: I'm afraid the number of patches that added testing were too many for a revert | 00:49 |
| mestery | fungi: Yes :) | 00:49 |
| mtreinish | I'd be fine with just pushing a delete patch for all the testing | 00:49 |
| dougwig | hi infra, i've noticed an issue with the single use slaves. they mask versioning problems in the global requirements. known issue? my CI has broken several times this week. | 00:49 |
| clarkb | mtreinish: thats fine, just want to make sure I understand how this fits together | 00:49 |
| clarkb | dougwig: can you expand on that? | 00:49 |
| mestery | fungi: I think I may just create a single patch then, the history isn't very meaningful here anyways :) | 00:49 |
| clarkb | dougwig: I think that is a feature of single use slaves | 00:49 |
| clarkb | dougwig: so want to make sure I understand your concern :) | 00:49 |
| *** ivar-lazzaro has joined #openstack-infra | 00:50 | |
| fungi | mestery: probably yes. either that or you need me or another gerrit admin to push in the initial commits for you, which we generally lack time to support for stackforge projects | 00:50 |
| dougwig | clarkb: oslo.concurrency released a 1.3.0 package with a major namespace change. global requirements was set at 1.1.0. ceilometer started using 1.3.0 features. openstack jenkins was peachy keen, since the single use nodes grab the latest version. my slaves were somewhat less happy, being at 1.2.0 of that package, which devstack thought satisfied the | 00:50 |
| dougwig | requirements just fine. | 00:50 |
| mestery | fungi: Completely understand, thanks for the input fungi! | 00:50 |
| *** ivar-lazzaro has quit IRC | 00:51 | |
| mtreinish | clarkb: if you want to facepalm, look at: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_load_balancer_basic.py | 00:51 |
| dougwig | clarkb: so yes, it's a feature, but there might be room for a job that does a simple sanity test using the minimum required versions in openstack/requirements. | 00:51 |
| mtreinish | mestery: actually what would be your opinion on just deleting the adv services tests from tempest? | 00:51 |
| clarkb | dougwig: ya so that is actually an issue in how we update requirements not single use slaves. We specifically use single use slaves to prevent tests from leaking across each other | 00:51 |
| *** ivar-laz_ has quit IRC | 00:51 | |
| fungi | dougwig: yeah, pinning oslo libraries in global requirements currently doesn't work with devstack-gate's desire to clone those libraries from git rather than install them from pypi i think | 00:51 |
| clarkb | dougwig: right, we need something to keep the lower bounds honest | 00:51 |
| mestery | mtreinish: My opinion would be we could do that, and then have each new team work to get them running again. | 00:52 |
| dougwig | mtreinish: that facepalm might break your skull. | 00:52 |
| *** annegent_ has quit IRC | 00:52 | |
| *** ivar-lazzaro has joined #openstack-infra | 00:52 | |
| mestery | mtreinish: Does that make this easier? | 00:52 |
| *** yamamoto_ has joined #openstack-infra | 00:52 | |
| fungi | dougwig: oh, and yeah what you describe | 00:52 |
| mtreinish | mestery: yes, basically we're missing the infrastructure in devstack-gate to do it correctly. (salv-orlando has it up as wip) | 00:53 |
| mtreinish | mestery: the current approach is to just kinda hack in it | 00:53 |
| mestery | mtreinish: I say we nuke it and make each new services team do it the right way. | 00:53 |
| *** craigbr has joined #openstack-infra | 00:53 | |
| mestery | salv-orlando: ^^^ | 00:53 |
| fungi | dougwig: at a minimum the environment should be rebuilt for each job to catch things which have changed, but as a result it will mask the inability of people who don't rebuild those environments to get magically updated to cope | 00:53 |
| mtreinish | mestery, dougwig: ok do you guys want to push a patch to remove those tests (that way I can +2 it) | 00:54 |
| clarkb | dougwig: unfortunately pip makes that almost impossible to do properly | 00:54 |
| mestery | dougwig: What say you? | 00:54 |
| dougwig | clarkb: well, my weaksauce third-party CI has you partially covered then. | 00:54 |
| clarkb | mestery: I mean that seems like a reasonable way to test? | 00:55 |
| clarkb | er | 00:55 |
| clarkb | mtreinish: ^ | 00:55 |
| *** jaypipes has quit IRC | 00:55 | |
| mtreinish | clarkb: well sort of, did you get to the netcat part? | 00:56 |
| clarkb | mtreinish: no I just ready the docstrings | 00:56 |
| *** yamamoto_ has quit IRC | 00:56 | |
| salv-orlando | mtreinish: api, scenario or both tests? | 00:56 |
| *** baoli has joined #openstack-infra | 00:56 | |
| mtreinish | because it's cirros in the gate and doesn't have any services running, they start up netcat and make it pretend to be an http server | 00:56 |
| mtreinish | salv-orlando: everything that would fail if they're not present | 00:57 |
| fungi | dougwig: we gave up reusing devstack workers years ago, because if you're testing patches to software which runs as root, it's a game of russian roulette before a change under testing completely breaks the operating system (or worse, clouds the results of subsequent jobs being run on that worker) | 00:57 |
| mtreinish | so both | 00:57 |
| * markmcclain catches up on scrollback | 00:57 | |
| jeblair | clarkb: i need to know how you would like me to prioritize work on nodepool | 00:57 |
| salv-orlando | mtreinish: I marun and markmcclain are ok with it, mostly because we realize that we can always revert this decision if we find out its a terrible thing | 00:57 |
| jeblair | clarkb: because as i have said, i do not think that the performance improvements you are looking into are the cause of our problems. yet it will take a _considerably_ long time for me to review the patch you have proposed | 00:58 |
| salv-orlando | but I don't think so because as these services are frozen anyway nothing should break them | 00:58 |
| *** dims has joined #openstack-infra | 00:58 | |
| clarkb | jeblair: I think you should continue the work you were doing | 00:58 |
| clarkb | jeblair: because I agree its unlikely that the recv(1) is the root cause | 00:58 |
| mtreinish | salv-orlando: well the only issue with a straight up revert would be bit-rot from being out of tree | 00:59 |
| *** mmaglana has quit IRC | 00:59 | |
| mtreinish | salv-orlando: but, it seems like the better solution, especially since we've been basically blocking new adv services tests anyway | 00:59 |
| dougwig | mtreinish: https://review.openstack.org/140192 | 00:59 |
| clarkb | jeblair: so spinning up a precise lookalike then sorting out how python is acting differently between trusty and precise still sounds like a good plan to me | 00:59 |
| clarkb | my change is likely to be an optimization that we can worry about later | 01:00 |
| clarkb | I just wanted to write that while the code was still fresh in my head | 01:00 |
| mtreinish | dougwig: it's got to be all of them, and the service clients too, otherwise it doesn't really fix it | 01:00 |
| *** armax has quit IRC | 01:00 | |
| dougwig | mtreinish: ok, so we expect the reasonable tests to go back in in a few days, organized in separate jobs properly? | 01:01 |
| *** e0ne has quit IRC | 01:02 | |
| mtreinish | dougwig: well, maybe. I've been on the fence about having testing for the adv services in tempest. We haven't allowed new ones in a while | 01:02 |
| dougwig | mtreinish: that's really less gross than the two-line hack that would also only last a few days? | 01:02 |
| clarkb | jeblair: does that line up with your thoughts on this? | 01:03 |
| jeblair | clarkb: yes. can you wip that patch until you test it? | 01:03 |
| *** yaguang has joined #openstack-infra | 01:03 | |
| clarkb | jeblair: yup | 01:03 |
| *** Sukhdev_ has joined #openstack-infra | 01:03 | |
| clarkb | and done | 01:03 |
| dougwig | mtreinish: where do they land if they go out of tempest? in the service repos, and then indirectly referenced for tempest runs? or? | 01:03 |
| jeblair | clarkb: (unit tests are probably useless here :) | 01:03 |
| mtreinish | dougwig: well read the NOTE on the bit I added, it really is a layer violation because we're missing the extension selection bit from devstack-gate | 01:03 |
| mtreinish | dougwig: yeah, you bake them into the service repos and set up the jobs to run them directly | 01:04 |
| *** Sukhdev has quit IRC | 01:04 | |
| clarkb | jeblair: agreed | 01:04 |
| mtreinish | dougwig: barbican is doing something like that right now for their functional testing | 01:04 |
| jeblair | my precise server has built, i will attempt to match packages, etc on it now | 01:04 |
| *** yaguang_ has joined #openstack-infra | 01:04 | |
| dougwig | does each repo setup its own devstack, or is there some way they're sharing one install? | 01:04 |
| mtreinish | dougwig: http://git.openstack.org/cgit/openstack/barbican/tree/functionaltests | 01:05 |
| mtreinish | with the import tempest bits, eventually moving to tempest-lib when that's ready | 01:05 |
| *** spzala has joined #openstack-infra | 01:05 | |
| mtreinish | dougwig: you can reuse devstack, since the adv services bits are already there | 01:06 |
| *** e0ne has joined #openstack-infra | 01:06 | |
| *** Sukhdev_ has quit IRC | 01:06 | |
| mtreinish | but if you need extras you can keep them locally in each repo | 01:06 |
| *** avozza is now known as zz_avozza | 01:06 | |
| *** Sukhdev has joined #openstack-infra | 01:06 | |
| mtreinish | mestery: ^^^ any thoughts on that? | 01:06 |
| mestery | mtreinish: I'm onboard with that plan, seems like a good way to move forward and get these tests under the control of each group responsible for them. | 01:07 |
| *** signed8bit_ZZZzz is now known as signed8bit | 01:07 | |
| dougwig | ok, i'm willing to make this change, but i'm wondering if it's the best idea to be compounding picking up the pieces of this split with switching how we're doing testing. i'd rather make it as a separate change, rather than making this snowball larger. | 01:07 |
| *** kumartin has joined #openstack-infra | 01:08 | |
| *** yaguang has quit IRC | 01:08 | |
| mestery | dougwig: I think we're headed in this direction, and this will catalyze that. | 01:08 |
| *** zz_sabari is now known as sabari | 01:08 | |
| *** Sukhdev_ has joined #openstack-infra | 01:08 | |
| *** yaguang_ has quit IRC | 01:08 | |
| jeblair | clarkb: that node is very slow :/ i'm going to build another one | 01:09 |
| *** ZZelle_ has quit IRC | 01:09 | |
| mestery | dougwig: Agreed on the snowball comment. We need to find a way to move forward here that works for everyone. | 01:09 |
| mestery | mtreinish: ^^^ | 01:09 |
| mtreinish | mestery: well in the short term I guess we can go with the hacked list of neutron extensions: https://review.openstack.org/140164 | 01:10 |
| mestery | mtreinishr: ++ | 01:10 |
| *** Sukhdev has quit IRC | 01:11 | |
| mtreinish | it's just that leaves a hole open, so I don't think we should land any unrelated neutron changes (in any repos) until it's ready to be removed | 01:12 |
| *** jyuso has quit IRC | 01:14 | |
| dougwig | mtreinish: i don't plan to land any services code besides repo fixes and turning those back on in the next couple of days. | 01:14 |
| mtreinish | dougwig: the exposure is in new code that manipulates non-adv services extensions | 01:14 |
| *** melwitt has quit IRC | 01:14 | |
| mtreinish | which could be tempest tests, a new extension in neutron, or devstack code to manipulate neutron config | 01:15 |
| dougwig | mtreinish: i'm not sure that i understand. you're leaving the non-adv.services extensions enabled. | 01:15 |
| *** SumitNaiksatam has quit IRC | 01:15 | |
| mtreinish | dougwig: it's 2-fold, first that list will be used on the stable branches too. (which is incorrect, since master has new extensions) Although this really is no different from today | 01:17 |
| mtreinish | the second aspect is new extension additions, since the list is hard coded, if something is added tempest won't think it's enabled and if tests are added the gate won't catch it | 01:18 |
| mtreinish | it's really unlikely that either would cause issues, but you never know | 01:18 |
| *** mestery has quit IRC | 01:18 | |
| dougwig | neutron is in the middle of refactoring the rest server. the odds of a new extension are really low, but i can make extra sure to watch the incoming reviews. | 01:18 |
| *** gyee has quit IRC | 01:19 | |
| mtreinish | dougwig: yeah, it's probably a super low likelihood but you'd be surprised how often things manage to splip through when we have a hole exposed | 01:20 |
| *** mestery has joined #openstack-infra | 01:20 | |
| *** salv-orlando has quit IRC | 01:20 | |
| mtreinish | clarkb, fungi: any chance we can promote this: https://review.openstack.org/#/c/140196/ | 01:21 |
| dougwig | the alternative is neutron being entirely broken and locked. :) | 01:21 |
| mtreinish | dougwig: heh, it's happened many times before :) | 01:21 |
| dougwig | lol | 01:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/gear: Read more bytes per readPacket iteration https://review.openstack.org/140186 | 01:22 |
| fungi | mtreinish: looking | 01:23 |
| clarkb | mtreinish: how did that merge if it errors? | 01:23 |
| *** mestery has quit IRC | 01:23 | |
| mtreinish | fungi: it's a straight up revert of the most recent tempest patch which was wrong | 01:24 |
| * anteaya is offline until the brand new third party meeting which starts in 6 hours and 37 minutes | 01:24 | |
| anteaya | 36 | 01:24 |
| mtreinish | clarkb: I'm not sure, it looks like they're non-fatal: https://jenkins05.openstack.org/job/check-tempest-dsvm-neutron-full/12091/console | 01:24 |
| clarkb | ianw: did you see my comments on https://review.openstack.org/#/c/138250/ | 01:24 |
| fungi | mtreinish: what's the impact if it running like that a couple more hours while the change gates? | 01:24 |
| *** obondarev has quit IRC | 01:25 | |
| dougwig | mtreinish: i have to step out for a bit. i'll be back in 2 hours. | 01:25 |
| mtreinish | fungi: well, since it looks non fatal, then just much longer console logs which eat up more space | 01:26 |
| ianw | clarkb: yeah, sorry haven't got back to updating it but will do | 01:26 |
| *** rkukura has quit IRC | 01:26 | |
| mtreinish | dougwig: ok, I'll be offline by then, just ping me in the morning if we need to work through some more stuff | 01:26 |
| dougwig | ok. is one of the tempest changes going to land tonight? | 01:26 |
| fungi | mtreinish: by what factor are we talking about? order of magnitude? multiple orders of magnitude? there's really not a lot running right now | 01:26 |
| *** marun has quit IRC | 01:27 | |
| ianw | clarkb: you're probably right that there's some other issue if you fixed the dict | 01:27 |
| *** mriedem1 has quit IRC | 01:27 | |
| *** markmcclain has quit IRC | 01:28 | |
| *** andreaf has quit IRC | 01:28 | |
| *** shakamunyi has quit IRC | 01:28 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:29 | |
| *** andreaf has joined #openstack-infra | 01:29 | |
| mtreinish | fungi: from looking at 1 sample each it looks like it goes from ~342k to 1M | 01:29 |
| *** Sukhdev_ has quit IRC | 01:30 | |
| mtreinish | so not the end of the world at light load I guess | 01:30 |
| *** pc_m has quit IRC | 01:31 | |
| grue_pm | /////////////////////////////////// | 01:32 |
| *** jyuso has joined #openstack-infra | 01:32 | |
| *** otter768 has joined #openstack-infra | 01:33 | |
| mtreinish | clarkb: ^^^ do you think we can handle that jump for every tempest run between now and when 140196 makes it through the gate? :) | 01:33 |
| *** tnovacik has quit IRC | 01:35 | |
| *** e0ne has quit IRC | 01:37 | |
| clarkb | ya that should be fine | 01:39 |
| *** emagana has quit IRC | 01:40 | |
| *** sarob has quit IRC | 01:41 | |
| *** e0ne has joined #openstack-infra | 01:41 | |
| *** kumartin has quit IRC | 01:41 | |
| *** jerryz has joined #openstack-infra | 01:42 | |
| *** alexpilotti has joined #openstack-infra | 01:43 | |
| *** wenlock has quit IRC | 01:43 | |
| fungi | looks like it's safe for me to remove the remaining nodepool-dev nodes now | 01:43 |
| fungi | and there they go | 01:44 |
| *** ryanpetrello has joined #openstack-infra | 01:45 | |
| *** zz_dimtruck is now known as dimtruck | 01:49 | |
| *** ChuckC has quit IRC | 01:50 | |
| *** Ryan_Lane has quit IRC | 01:50 | |
| *** Ryan_Lane has joined #openstack-infra | 01:51 | |
| *** dannywil_ has quit IRC | 01:56 | |
| *** signed8bit has quit IRC | 02:02 | |
| *** signed8bit has joined #openstack-infra | 02:03 | |
| jeblair | clarkb: comment on your change | 02:05 |
| jeblair | (not an in depth review) | 02:06 |
| *** alexpilotti has quit IRC | 02:06 | |
| clarkb | jeblair: thanks, I already found at least one bug. Also it turns out that when you run the reverser test job on non trivial amounts of data xfce4-terminal gets unhappy | 02:08 |
| *** dims has quit IRC | 02:08 | |
| jeblair | ha | 02:08 |
| jeblair | mordred: i have renewed appreciation for your dib-for-non-nodepool-infra work as i try to figure out why python-cheetah is installed on a new machine i spun up but not on our old nodepool server | 02:09 |
| clarkb | so I think I may be done for the niht because all my terminals died | 02:09 |
| clarkb | :/ | 02:09 |
| jeblair | clarkb: i think you should heed the signs | 02:10 |
| *** shashankhegde has quit IRC | 02:10 | |
| *** sabari is now known as zz_sabari | 02:10 | |
| jeblair | The following packages will be REMOVED: cloud-init python-cheetah | 02:10 |
| jeblair | oh. that explains it. :( | 02:10 |
| *** ryanpetrello has quit IRC | 02:10 | |
| clarkb | but initial results after I fixed that first bug are reasonable. I do know that geard/client/workers can move >4096 bytes with that change | 02:11 |
| clarkb | and that is what kills terminals | 02:11 |
| *** ddieterly has joined #openstack-infra | 02:11 | |
| *** ChuckC has joined #openstack-infra | 02:12 | |
| jeblair | The following packages will be REMOVED: cloud-utils euca2ools python-m2crypto | 02:12 |
| jeblair | m2crypto is apparently still not dead? | 02:12 |
| *** signed8bit has quit IRC | 02:13 | |
| clarkb | jeblair: in what cases would conn.recv(1) return b'' instead of throwing an error? and in those cases shouldn't we preserve the old already read raw_bytes and try again? | 02:15 |
| clarkb | jeblair: thinking out loud here in response to your comment | 02:15 |
| clarkb | or are we saying if something that weird happens just give up and start from scratch on the buffer? | 02:15 |
| *** hdd has quit IRC | 02:15 | |
| jeblair | clarkb: i think in the case of connection close while reading | 02:16 |
| *** ChuckC has quit IRC | 02:16 | |
| *** jyuso has quit IRC | 02:16 | |
| *** ChuckC has joined #openstack-infra | 02:16 | |
| clarkb | hrm ok | 02:16 |
| *** jyuso has joined #openstack-infra | 02:17 | |
| clarkb | I will need to think about that then, because I think we should try to keep the old buffer around then let recv throw an error in that case so that we can process any unprocessed packets. But that may not be necessary | 02:17 |
| *** ivar-lazzaro has quit IRC | 02:17 | |
| jeblair | clarkb: i don't think recv throws errors in all cases | 02:17 |
| *** ayoung has joined #openstack-infra | 02:17 | |
| clarkb | oh ok | 02:17 |
| *** hdd has joined #openstack-infra | 02:17 | |
| clarkb | though my latest version of code is better about only reading for new data when it actually needs new data so we should handle this properly | 02:18 |
| clarkb | I will address your comment tomorrow. thanks | 02:18 |
| *** e0ne has quit IRC | 02:20 | |
| openstackgerrit | Steven Weston proposed openstack-infra/system-config: Adds puppet system-config module split script https://review.openstack.org/137991 | 02:21 |
| *** signed8bit has joined #openstack-infra | 02:23 | |
| *** yamamoto_ has joined #openstack-infra | 02:25 | |
| *** Masahiro has joined #openstack-infra | 02:26 | |
| *** julim has quit IRC | 02:27 | |
| *** Ryan_Lane has quit IRC | 02:30 | |
| *** Ryan_Lane has joined #openstack-infra | 02:31 | |
| *** Masahiro has quit IRC | 02:31 | |
| *** stevemar has quit IRC | 02:33 | |
| *** yaguang has joined #openstack-infra | 02:36 | |
| *** SumitNaiksatam has quit IRC | 02:39 | |
| *** Ryan_Lane has quit IRC | 02:41 | |
| *** dims has joined #openstack-infra | 02:50 | |
| *** stevemar has joined #openstack-infra | 02:50 | |
| *** markmcclain has joined #openstack-infra | 02:54 | |
| *** jyuso has quit IRC | 02:55 | |
| *** salv-orlando has joined #openstack-infra | 02:55 | |
| *** armax has joined #openstack-infra | 02:56 | |
| *** signed8bit has quit IRC | 02:57 | |
| *** mestery has joined #openstack-infra | 02:57 | |
| *** patrickeast has quit IRC | 03:01 | |
| *** signed8bit has joined #openstack-infra | 03:10 | |
| *** talluri has joined #openstack-infra | 03:12 | |
| *** nelsnelson has quit IRC | 03:12 | |
| *** dims has quit IRC | 03:13 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:14 | |
| *** dims has joined #openstack-infra | 03:14 | |
| *** dims has quit IRC | 03:18 | |
| *** signed8bit has quit IRC | 03:23 | |
| krtaylor | sweston, ping, I just saw your email, you beat me to a proposal my just a few minutes :) | 03:23 |
| krtaylor | sweston, I am sending choices now, can you change the moderator page for the vote? | 03:24 |
| *** david-lyle is now known as david-lyle_afk | 03:24 | |
| *** mmaglana has joined #openstack-infra | 03:26 | |
| *** ryanpetrello has joined #openstack-infra | 03:27 | |
| jeblair | clarkb, fungi: good news and bad news on the reproducibility front; when i started builds of 1000 fake servers on precise and trusty, it looked like trusty was lagging behind measurably. but then i ended up with 999 threads deadlocked in "/usr/local/lib/python2.7/dist-packages/Crypto/Random/_UserFriendlyRNG.py" which has had a significant negative impact on my ability to test | 03:28 |
| jeblair | clarkb, fungi: pycrypto is 2.6.1 on trusty and 2.4.1 on precise | 03:29 |
| jeblair | clarkb, fungi: (it's not in the OS, that's just the versions we happened to have on those respective hosts) | 03:29 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Move zuul_swift_upload into postbuildscript https://review.openstack.org/140218 | 03:30 |
| jhesketh | jeblair, clarkb: if you guys have a second to review ^ | 03:30 |
| *** hdd has quit IRC | 03:30 | |
| jeblair | clarkb, fungi: oh neat, it wasn't a deadlock; it unwedged | 03:31 |
| clarkb | jhesketh: what do you think about updating that macro to be a post step? | 03:31 |
| clarkb | jhesketh: then moving zuul-swift-logs to publishers | 03:31 |
| jhesketh | clarkb: good idea | 03:32 |
| jhesketh | please hold | 03:32 |
| *** jaypipes has joined #openstack-infra | 03:32 | |
| clarkb | jeblair: do we just not have the same entropy on trusty? | 03:33 |
| *** koolhead17 has joined #openstack-infra | 03:35 | |
| *** koolhead17 has joined #openstack-infra | 03:35 | |
| jeblair | interestingly pycrypto 2.6.1 is a release to fix a cve related to entropy: https://www.dlitz.net/software/pycrypto/ | 03:35 |
| jeblair | (of course we had no idea about that) | 03:36 |
| jeblair | however, i don't think it affects us | 03:36 |
| jeblair | clarkb: it reads from urandom, so it shouldn't be blocking on that | 03:37 |
| jeblair | i will dig into the logs and check the thread dump from the first unwedged thread | 03:37 |
| *** koolhead_ has joined #openstack-infra | 03:39 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Create zuul_swift_upload publishers https://review.openstack.org/140218 | 03:41 |
| jhesketh | clarkb: ^ okay, try that :-) | 03:41 |
| *** Masahiro has joined #openstack-infra | 03:42 | |
| *** koolhead17 has quit IRC | 03:43 | |
| *** annegent_ has joined #openstack-infra | 03:46 | |
| *** Masahiro has quit IRC | 03:47 | |
| *** annegent_ has quit IRC | 03:48 | |
| *** annegent_ has joined #openstack-infra | 03:48 | |
| *** erlon has quit IRC | 03:50 | |
| *** erlon has joined #openstack-infra | 03:51 | |
| *** bradjones has quit IRC | 03:51 | |
| jeblair | clarkb, fungi: when i did the thread dump, 186 threads were trying to acquire the lock, 14 were doing computation outside the lock, 1 held the lock and was doing computation, and 800 threads were releasing the lock | 03:52 |
| clarkb | jhesketh: can we have overlapping publishers and builders macros with the same names? | 03:53 |
| jeblair | clarkb, fungi: what i'm getting from this is that we weren't stuck waiting on entropy, but rather, when threads released the lock, they stopped getting scheduled. | 03:53 |
| jeblair | this feels like the same kind of behavior we're observing in production | 03:53 |
| jhesketh | clarkb: I don't see why not, the jjb diff looks correct http://logs.openstack.org/18/140218/2/check/project-config-compare-xml/145dbae/console.html | 03:53 |
| jhesketh | they are referred in different sections (ie publishers or builders) | 03:54 |
| jeblair | (i'm imagining that basically all of the threads doing crypto work ended up being serialized before nodepool started making progress again) | 03:54 |
| clarkb | jhesketh: I was just going to look at the diff | 03:54 |
| clarkb | jeblair: huh | 03:54 |
| clarkb | jhesketh: does publisher: zuul_swift_upload do anything? | 03:55 |
| jeblair | jhesketh, clarkb: it's been a while, but unless something's changed, i think they should be able to have the same name. of course we may or may not want to do that for our own maintenance sanity, but as a transitional step i think it should be fine | 03:55 |
| clarkb | jhesketh: it seems to run build: zuul_swift_upload with no args | 03:55 |
| jhesketh | clarkb: right, but runs it as a publisher rather than a build step | 03:56 |
| jhesketh | well as a postbuildscript anyway | 03:56 |
| clarkb | jhesketh: ya but it takes no args and the underlying builder wants an arg | 03:57 |
| jhesketh | clarkb: ah, hold on, I see my mistake | 03:57 |
| *** armax has quit IRC | 03:58 | |
| *** koolhea__ has joined #openstack-infra | 03:58 | |
| jeblair | clarkb, fungi: also, nodepool uses 3x as much memory on trusty | 03:59 |
| clarkb | jhesketh: also the zuul_swift_uploads_with_console et al builders embed the scripts needed to get the console logs | 03:59 |
| clarkb | jhesketh: so in your publishers I think we can reduce that down | 03:59 |
| *** achanda has joined #openstack-infra | 03:59 | |
| *** koolhead_ has quit IRC | 03:59 | |
| jhesketh | clarkb: yep | 03:59 |
| clarkb | though looking at the diff it seems to have deduped? | 04:00 |
| *** salv-orlando has quit IRC | 04:00 | |
| *** mestery has quit IRC | 04:00 | |
| clarkb | oh right beacuse yaml | 04:00 |
| clarkb | I think | 04:00 |
| clarkb | jeblair: wow | 04:00 |
| clarkb | jhesketh: oh and you need to rename the builders in the publishers | 04:01 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Create zuul_swift_upload publishers https://review.openstack.org/140218 | 04:01 |
| jhesketh | clarkb: like that ^ ? | 04:01 |
| *** markmcclain has quit IRC | 04:02 | |
| *** ryanpetrello has quit IRC | 04:02 | |
| jeblair | i think i'm going to eod and sleep on this. but i think my next step is going to be some pretty basic python threading tests. | 04:02 |
| clarkb | jhesketh: yup I think thats it | 04:02 |
| *** baoli has quit IRC | 04:03 | |
| clarkb | I am waiting on xml diff to review | 04:03 |
| *** koolhea__ has quit IRC | 04:03 | |
| *** koolhead17 has joined #openstack-infra | 04:04 | |
| *** spzala has quit IRC | 04:08 | |
| *** koolhead17 has quit IRC | 04:08 | |
| clarkb | jhesketh: +2 | 04:09 |
| jhesketh | clarkb: cheers | 04:09 |
| jhesketh | it must be late for you yeah? | 04:09 |
| clarkb | 8pm ish | 04:10 |
| *** zz_sabari is now known as sabari | 04:10 | |
| jhesketh | ah, I had my timezones wrong then | 04:11 |
| jhesketh | clarkb: are you going to be around if I self-approve that change? | 04:11 |
| jhesketh | if you're heading off I'll wait for the proper 2*2+ | 04:11 |
| clarkb | I should be around for another hour or so | 04:11 |
| clarkb | jhesketh: I am pretty confident in that change since there is no diff with the centos6 experimental change | 04:12 |
| jhesketh | ditto | 04:12 |
| jhesketh | happy for me to break the world then^W^W^W^Wgive it a try | 04:12 |
| *** pc_m has joined #openstack-infra | 04:14 | |
| *** ddieterly has quit IRC | 04:14 | |
| clarkb | ya go for it | 04:15 |
| *** viglesias has quit IRC | 04:15 | |
| *** viglesias has joined #openstack-infra | 04:15 | |
| *** koolhead17 has joined #openstack-infra | 04:16 | |
| *** achanda has quit IRC | 04:17 | |
| *** achanda has joined #openstack-infra | 04:17 | |
| jhesketh | clarkb: I can't remember... do slave scripts get built into the images or pushed out from jenkins masters? | 04:19 |
| clarkb | the scripts themselves in /usr/local/jenkins are on the images | 04:19 |
| jhesketh | actually, nevermind, this won't change the slave scripts, it'll just reconfigure the masters | 04:19 |
| clarkb | but the stuff you are changing here is just the masters | 04:19 |
| jhesketh | yeah, which is done every 15min ? | 04:19 |
| clarkb | ya | 04:20 |
| jhesketh | cool, thanks | 04:20 |
| clarkb | its a */15 ish cron with some splay bcause ansible does ~10 nodes at a time iirc | 04:20 |
| *** salv-orlando has joined #openstack-infra | 04:20 | |
| clarkb | jhesketh: http://puppetboard.openstack.org should show you when they update if you filter on jenkins | 04:20 |
| jhesketh | do you mean s/ansible/puppet? | 04:20 |
| clarkb | jhesketh: sort of. the */15 cron runs ansible which runs puppet :) | 04:21 |
| *** koolhead17 has quit IRC | 04:21 | |
| jhesketh | ah, fair enough | 04:21 |
| clarkb | ansible is ther eto make sure that we do gitXX.o.o then gerrit then everything else | 04:21 |
| clarkb | so that adding new projects happens in a safe order | 04:21 |
| *** armax has joined #openstack-infra | 04:21 | |
| jhesketh | ah I see | 04:21 |
| openstackgerrit | Merged openstack-infra/project-config: Create zuul_swift_upload publishers https://review.openstack.org/140218 | 04:21 |
| mordred | jeblair: it is truly remarkable just how crazy all the base images are | 04:21 |
| *** Sukhdev has joined #openstack-infra | 04:22 | |
| clarkb | and I believe its a max of 10 nodes at a time that ansilbe will run puppet on | 04:22 |
| mordred | jeblair: I'm _very_ much looking forward to a solid base image | 04:22 |
| *** mestery has joined #openstack-infra | 04:22 | |
| mordred | clarkb: aroo? | 04:22 |
| *** achanda has quit IRC | 04:22 | |
| mordred | ah - yes. although, sake of argument, that's a configurable number | 04:22 |
| clarkb | ya its like xargs -P | 04:22 |
| clarkb | jhesketh: so I expect that by 0445UTC the masters will be up to date | 04:23 |
| jhesketh | yep | 04:23 |
| *** sarob has joined #openstack-infra | 04:25 | |
| *** bswartz has joined #openstack-infra | 04:27 | |
| *** pcrews has quit IRC | 04:29 | |
| *** rushiagr_away is now known as rushiagr | 04:34 | |
| sweston | krtaylor: apologies, It was getting late so I thought I'd take it off your hands | 04:34 |
| sweston | krtaylor: would you like a question modified? It is a collaborative poll, you can add your own questions as well. | 04:36 |
| *** rkukura has joined #openstack-infra | 04:36 | |
| *** achanda has joined #openstack-infra | 04:37 | |
| sweston | krtaylor: I see your email, I'll make the changes | 04:38 |
| *** otter768 has quit IRC | 04:39 | |
| *** achanda has quit IRC | 04:40 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: DO NOT MERGE - testing breakage https://review.openstack.org/140226 | 04:41 |
| clarkb | jhesketh: looks like all of the jenkins masters updated | 04:42 |
| jhesketh | yep, thanks :-) | 04:42 |
| *** Masahiro has joined #openstack-infra | 04:43 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: DO NOT MERGE - testing breakage https://review.openstack.org/140226 | 04:45 |
| *** ddieterly has joined #openstack-infra | 04:45 | |
| *** salv-orlando has quit IRC | 04:46 | |
| *** armax has quit IRC | 04:47 | |
| *** Masahiro has quit IRC | 04:47 | |
| clarkb | http://logs.openstack.org/26/140226/1/check/project-config-compare-xml/7025a52/console.html?source=swift itworked for a successful job | 04:48 |
| *** salv-orlando has joined #openstack-infra | 04:49 | |
| *** armax has joined #openstack-infra | 04:49 | |
| *** ddieterly has quit IRC | 04:49 | |
| jhesketh | clarkb: and a failing job: http://logs.openstack.org/26/140226/2/check/gate-project-config-projects-yaml/2caa1e4/console.html?source=swift | 04:49 |
| openstackgerrit | David Pursehouse proposed openstack-infra/zuul: Document that the "Stream Events" permission is needed. https://review.openstack.org/132645 | 04:51 |
| openstackgerrit | David Pursehouse proposed openstack-infra/zuul: Require python-daemon 1.6 or higher https://review.openstack.org/132743 | 04:51 |
| *** talluri has quit IRC | 04:52 | |
| openstackgerrit | Steven Weston proposed openstack-infra/system-config: Adds puppet system-config module split script https://review.openstack.org/137991 | 04:52 |
| jhesketh | clarkb: so the next step is a little more intersting.. turning off existing publishers to see if we can just use swift | 04:53 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Test only using swift for log storing https://review.openstack.org/140229 | 04:54 |
| *** zhiwei has joined #openstack-infra | 04:55 | |
| jhesketh | clarkb: ^ is pretty harmless if you wouldn't mind taking a quick look and I'll self-approve | 04:55 |
| zhiwei | hi, is any one here? | 04:55 |
| clarkb | jhesketh: looking | 04:56 |
| clarkb | zhiwei: yes there are at least a couple of us around | 04:56 |
| zhiwei | Gem::RemoteFetcher::FetchError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://rubygems.global.ssl.fastly.net/gems/coderay-1.1.0.gem) | 04:56 |
| zhiwei | how to handle this error. | 04:56 |
| zhiwei | http://logs.openstack.org/15/140215/1/check/gate-cookbook-openstack-common-chef-lint/5f572a7/console.html | 04:56 |
| clarkb | jhesketh: +2 | 04:56 |
| clarkb | zhiwei: Ithink rubygems needs to fix their cert | 04:56 |
| *** dannywilson has joined #openstack-infra | 04:57 | |
| zhiwei | Building remotely on bare-precise-rax-iad-3363170 (bare-precise) is the bare-precise machine? | 04:57 |
| zhiwei | I can fetch the gems locally. | 04:58 |
| clarkb | yes that is the jenkins machine that that ran on | 04:58 |
| zhiwei | How to fix this on jenkins machine? | 04:58 |
| *** Sukhdev has quit IRC | 04:58 | |
| clarkb | I don't think it is something that can be fixed on jenkins | 04:58 |
| clarkb | fastly and/or rubygems served a cert that could not be verified | 04:59 |
| zhiwei | I saw some of them uses http not https. | 04:59 |
| zhiwei | is there a workaround method? | 05:01 |
| *** dannywilson has quit IRC | 05:01 | |
| zhiwei | how to retrigger the jinkens job by not submit new patch set? | 05:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Check env-vars is a dictionary https://review.openstack.org/138250 | 05:02 |
| *** koolhead17 has joined #openstack-infra | 05:02 | |
| clarkb | zhiwei: leave a comment of 'recheck' | 05:02 |
| *** gema_ has quit IRC | 05:03 | |
| zhiwei | thanks! | 05:03 |
| *** chandankumar has joined #openstack-infra | 05:03 | |
| clarkb | jhesketh: when we make the switch we will need to tell the logstash stuff to not retry on console.html files | 05:03 |
| zhiwei | clarkb: bare-precise-rax-dfw-3363512 success and bare-precise-rax-iad-3363170 failed. | 05:04 |
| clarkb | jhesketh: since they should always be complete by the time logstash gets to them under swift and currently it will retry until a timeout or until it finds text that will never show up unless maybe the text will show up | 05:04 |
| clarkb | zhiwei: probably ruby gems or fastly having intermittent issues | 05:04 |
| clarkb | zhiwei: I don't think the test machine has anything to do with it | 05:05 |
| zhiwei | ok | 05:05 |
| zhiwei | I will retrigger the jobs. | 05:05 |
| *** gema has joined #openstack-infra | 05:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Test only using swift for log storing https://review.openstack.org/140229 | 05:06 |
| zhiwei | clarkb: can you open this URL? I think I misunderstand your replies. https://review.openstack.org/#/c/140215/ | 05:07 |
| clarkb | jhesketh: I think content-type=text/plain actually does the right thing there so nevermind about that | 05:07 |
| clarkb | zhiwei: that is correct. http://status.openstack.org/zuul/ you can see it there | 05:07 |
| clarkb | same failure though. I would check ith rubygems | 05:08 |
| jhesketh | clarkb: hmm, so how does logstash currently know when to fetch a log? | 05:08 |
| *** rushiagr is now known as rushiagr_away | 05:09 | |
| zhiwei | failed again. | 05:09 |
| zhiwei | thanks clarkb. | 05:09 |
| clarkb | jhesketh: it starts when the jobs finish. with current console.html uploads the upload may finish after the jobs finishes so it will look for the ending </pre> tag with a timeout | 05:10 |
| clarkb | jhesketh: but since swift uploads happen entirely within a job this isnt' an issue | 05:10 |
| clarkb | jhesketh: also http://logs.openstack.org/26/140226/2/check/gate-project-config-projects-yaml/2caa1e4/?source=swift seems to be the apache rendered index.html | 05:10 |
| *** smoser has quit IRC | 05:11 | |
| clarkb | jhesketh: http://logs.openstack.org/26/140226/2/check/gate-project-config-projects-yaml/2caa1e4/index.html?source=swift does the correct thing though | 05:11 |
| clarkb | I guess that is why your most recent chagne is valuable | 05:11 |
| *** ddieterly has joined #openstack-infra | 05:11 | |
| *** koolhead_ has joined #openstack-infra | 05:14 | |
| jhesketh | okay | 05:14 |
| *** koolhead17 has quit IRC | 05:14 | |
| jhesketh | clarkb: yeah the apache rules will load the directory since it is on disk (therefore it doesn't hit osloganalyze) | 05:14 |
| jhesketh | however if hte directory doesn't exist it'll try osloganalyze and if the object doesn't exist it'll append index.html and try again | 05:14 |
| jhesketh | so it should work once we turn off disk logging | 05:14 |
| clarkb | yup | 05:14 |
| *** ddieterly has quit IRC | 05:16 | |
| *** harlowja is now known as harlowja_away | 05:16 | |
| *** sarob has quit IRC | 05:18 | |
| jhesketh | clarkb: close:http://logs.openstack.org/79/133179/1/experimental/experimental-swift-logs-system-config-pep8-centos6/f3b73cc | 05:21 |
| clarkb | I guess that is progress :) | 05:21 |
| *** dimtruck is now known as zz_dimtruck | 05:22 | |
| *** koolhead_ has quit IRC | 05:24 | |
| *** koolhead17 has joined #openstack-infra | 05:25 | |
| *** koolhead17 has quit IRC | 05:29 | |
| clarkb | jhesketh: do the logs you have from before you removed normal console uploads have the data you need? | 05:31 |
| clarkb | jhesketh: http://logs.openstack.org/79/133179/1/experimental/experimental-swift-logs-system-config-pep8-centos6/c326e8b/console.html.gz#_2014-12-04_20_30_31_370 looks like it may | 05:31 |
| *** hdd has joined #openstack-infra | 05:34 | |
| jhesketh | clarkb: I can still get the full console log off the jenkins node too | 05:36 |
| jhesketh | clarkb: I'm wondering if I compiled jjb wrong | 05:36 |
| *** shashankhegde has joined #openstack-infra | 05:37 | |
| *** stevemar has quit IRC | 05:37 | |
| clarkb | jhesketh: ya I think its not properly passing through the upload source | 05:38 |
| *** stevemar has joined #openstack-infra | 05:38 | |
| jhesketh | clarkb: it might be a bug with jjb actually... not sure if I can pass through the upload source to a postbuildscript | 05:38 |
| clarkb | jhesketh: I think you can since its template substition happening a layer above | 05:39 |
| clarkb | jhesketh: let me try a thing | 05:39 |
| jhesketh | clarkb: right and I can see an example of it too | 05:40 |
| *** mrmartin has joined #openstack-infra | 05:40 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Fix yaml indentation for zuul swift log upload https://review.openstack.org/140235 | 05:40 |
| clarkb | jhesketh: ^ I think that that may be at fault | 05:40 |
| *** talluri has joined #openstack-infra | 05:40 | |
| clarkb | the diff should tell us though | 05:40 |
| jhesketh | clarkb: I would have thought the column after the previous line wouldn't parse if that was the case | 05:41 |
| jhesketh | also have I mentioned I don't like yaml | 05:41 |
| clarkb | jhesketh: I think it shouldn't parse but python yaml is :( | 05:42 |
| *** groknix has joined #openstack-infra | 05:42 | |
| *** pc_m has quit IRC | 05:42 | |
| jhesketh | clarkb: that did it though | 05:44 |
| clarkb | yup :/ | 05:44 |
| jhesketh | good spotting :-) | 05:44 |
| *** Masahiro has joined #openstack-infra | 05:44 | |
| * jhesketh yaml_hate += 1 | 05:44 | |
| jhesketh | clarkb: I +2'd if you're comfortable self-approving | 05:44 |
| clarkb | yup /me approves | 05:44 |
| *** craigbr has quit IRC | 05:45 | |
| jhesketh | cheers | 05:46 |
| *** Masahiro has quit IRC | 05:48 | |
| *** jyuso has joined #openstack-infra | 05:50 | |
| *** talluri has quit IRC | 05:55 | |
| *** mestery has quit IRC | 05:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix yaml indentation for zuul swift log upload https://review.openstack.org/140235 | 05:57 |
| *** salv-orlando has quit IRC | 05:58 | |
| *** armax has quit IRC | 05:58 | |
| *** salv-orlando has joined #openstack-infra | 06:01 | |
| *** sabari is now known as zz_sabari | 06:02 | |
| *** sabeen1 has quit IRC | 06:03 | |
| *** zz_sabari is now known as sabari | 06:05 | |
| *** salv-orlando has quit IRC | 06:06 | |
| *** markmcclain has joined #openstack-infra | 06:11 | |
| *** shashankhegde has quit IRC | 06:11 | |
| *** ddieterly has joined #openstack-infra | 06:11 | |
| *** odyssey4me has quit IRC | 06:12 | |
| *** freyes has quit IRC | 06:12 | |
| *** armax has joined #openstack-infra | 06:12 | |
| *** Nakato has quit IRC | 06:12 | |
| *** koolhead17 has joined #openstack-infra | 06:12 | |
| *** koolhead17 has joined #openstack-infra | 06:12 | |
| *** d34dh0r53 has quit IRC | 06:12 | |
| *** fungi has quit IRC | 06:12 | |
| *** odyssey4me has joined #openstack-infra | 06:12 | |
| *** cloudnull has quit IRC | 06:13 | |
| *** d34dh0r53 has joined #openstack-infra | 06:13 | |
| *** cloudnull has joined #openstack-infra | 06:13 | |
| *** fungi has joined #openstack-infra | 06:13 | |
| *** freyes has joined #openstack-infra | 06:13 | |
| *** rushiagr_away is now known as rushiagr | 06:14 | |
| *** talluri has joined #openstack-infra | 06:14 | |
| *** Nakato has joined #openstack-infra | 06:14 | |
| *** talluri has quit IRC | 06:14 | |
| *** talluri has joined #openstack-infra | 06:15 | |
| *** markmcclain has quit IRC | 06:15 | |
| jhesketh | clarkb: there we go :-) http://logs.openstack.org/79/133179/1/experimental/experimental-swift-logs-system-config-pep8-centos6/cc75c20 | 06:15 |
| *** ddieterly has quit IRC | 06:16 | |
| clarkb | cool | 06:16 |
| jhesketh | clarkb: how can I tell if logstash got it? | 06:20 |
| clarkb | jhesketh: one sec | 06:21 |
| clarkb | jhesketh: search for build_short_uuid:cc75c20 | 06:21 |
| clarkb | looks like it got it | 06:22 |
| jhesketh | awesome | 06:23 |
| jhesketh | so now we just need to discuss turning off disk logging for infra jobs, evalulate that and do it for the world | 06:23 |
| clarkb | yup I think that would be the next step | 06:24 |
| jhesketh | okay, I'll leave that for the meeting tomorrow | 06:24 |
| grue_pm | ugh getting used to this desktop - I minimized the internal course window and not sure where to look for it now | 06:27 |
| *** ildikov_ is now known as ildikov_afk | 06:28 | |
| *** annegent_ has quit IRC | 06:28 | |
| zhiwei | hi, how to set depends to other project? I have a patch in project A, and the other patch in project B depends on this patch in project A. | 06:28 |
| clarkb | zhiwei: gerrit doesn't support this directly. Typically people just document it in commit messages/change comments or write the tests such that they only pass if changes merge in teh correct order | 06:30 |
| zhiwei | ok, thanks. | 06:30 |
| jyuso | hi,guys,I notice that the content of compressed logs("xxx.txt.gz") from many 3rd part CI could be previewed on browser.But in my environment, such compressed file only could be downloaded after clicked. If I want to apply such preview function on Apache server,what settings should I do? | 06:37 |
| *** otter768 has joined #openstack-infra | 06:40 | |
| *** hdd has quit IRC | 06:41 | |
| anteaya | jyuso: that is a great question for our third party meeting taking place at 0800 utc in #openstack-meeting | 06:42 |
| anteaya | jyuso: why don't you join and when we get to the section on asking questions, you can ask then | 06:42 |
| *** russellb has quit IRC | 06:43 | |
| jyuso | anteaya: OK,thx. | 06:44 |
| anteaya | thank you | 06:44 |
| *** russellb has joined #openstack-infra | 06:44 | |
| *** otter768 has quit IRC | 06:44 | |
| *** sabari is now known as zz_sabari | 06:49 | |
| *** reed has quit IRC | 06:56 | |
| *** armax has quit IRC | 06:56 | |
| *** fandi has quit IRC | 06:57 | |
| *** Longgeek has joined #openstack-infra | 06:57 | |
| *** mrmartin has quit IRC | 06:59 | |
| *** salv-orlando has joined #openstack-infra | 07:02 | |
| *** fandi has joined #openstack-infra | 07:02 | |
| dougwig | jyuso: i'm not going to be up for the 1am meeting, but here's the apache config that i use: | 07:02 |
| dougwig | https://www.irccloud.com/pastebin/t1Yx0e8y | 07:02 |
| anteaya | dougwig: thanks | 07:05 |
| anteaya | actually a big part of the third party meeting is designed around asking questions like this | 07:05 |
| jyuso | dougwig: really thank you:) | 07:05 |
| anteaya | to help third party operators focus on talking to each other, rather than asking the same question over and over again in -infra | 07:05 |
| *** teran_ has joined #openstack-infra | 07:05 | |
| *** salv-orlando has quit IRC | 07:07 | |
| *** teran has quit IRC | 07:07 | |
| anteaya | dougwig: what is your status? | 07:09 |
| dougwig | the devstack-gate change is failing due to a neutron bug. i have another qa review out to skip the single failing (non-service) bug. waiting on approval for that (https://review.openstack.org/#/c/140225/), so I can recheck the tempest review (https://review.openstack.org/#/c/140164). | 07:10 |
| dougwig | i also have reviews for all three service repos to cleanup little things and get their basic jenkins plumbing passing. | 07:11 |
| *** ddieterly has joined #openstack-infra | 07:11 | |
| anteaya | cool | 07:12 |
| anteaya | cyeoh: are you able to review a tempest patch? https://review.openstack.org/#/c/140225/ | 07:12 |
| dougwig | recheck the devstack-gate review, i meant above. | 07:13 |
| openstackgerrit | Merged openstack-infra/project-config: Make gate-heat-templates-* jobs voting https://review.openstack.org/138939 | 07:14 |
| anteaya | kk | 07:14 |
| *** k4n0 has joined #openstack-infra | 07:15 | |
| *** AJaeger has joined #openstack-infra | 07:15 | |
| *** ddieterly has quit IRC | 07:16 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove dsvm-cells tests from places they don't do anything https://review.openstack.org/140011 | 07:17 |
| *** teran has joined #openstack-infra | 07:18 | |
| *** teran_ has quit IRC | 07:20 | |
| *** tkelsey has joined #openstack-infra | 07:20 | |
| *** tkelsey has quit IRC | 07:24 | |
| *** nuritv has joined #openstack-infra | 07:26 | |
| *** e0ne has joined #openstack-infra | 07:27 | |
| *** annegent_ has joined #openstack-infra | 07:28 | |
| *** Masahiro has joined #openstack-infra | 07:33 | |
| *** mrunge has joined #openstack-infra | 07:33 | |
| *** annegent_ has quit IRC | 07:33 | |
| *** Masahiro has quit IRC | 07:37 | |
| *** mrmartin has joined #openstack-infra | 07:39 | |
| *** belmoreira has joined #openstack-infra | 07:55 | |
| *** tnovacik has joined #openstack-infra | 07:58 | |
| *** jlibosva has joined #openstack-infra | 08:00 | |
| *** xianghui has joined #openstack-infra | 08:02 | |
| *** teran has quit IRC | 08:02 | |
| *** vigneshvar has joined #openstack-infra | 08:03 | |
| *** nuritv has quit IRC | 08:06 | |
| anteaya | jhesketh: able to attend the third-party meeting tonight? | 08:07 |
| jhesketh | anteaya: umm, when is it? | 08:07 |
| anteaya | now | 08:07 |
| jhesketh | err, sorry, guests are about to arrive for dinner | 08:08 |
| jhesketh | I didn't realise a time was set yet | 08:08 |
| *** HeOS has quit IRC | 08:08 | |
| AJaeger | good morning ;), anteaya and jhesketh. | 08:09 |
| anteaya | AJaeger: morning | 08:09 |
| anteaya | jhesketh: enjoy dinner | 08:09 |
| *** wuhg has joined #openstack-infra | 08:09 | |
| anteaya | jhesketh: see you next week maybe | 08:09 |
| jhesketh | anteaya: you only set this time an hour ago (/me catches up on email) | 08:10 |
| jhesketh | I didn't think we had concensus sorry | 08:10 |
| jhesketh | anyway, I'll observe as much as I can and read the logs | 08:10 |
| anteaya | no I set it last week | 08:10 |
| anteaya | the thread kind of went sideways | 08:10 |
| anteaya | which is unfortunately a pattern with the group | 08:10 |
| anteaya | part of what I am trying to change | 08:10 |
| anteaya | jhesketh: thanks | 08:10 |
| jhesketh | right, I missed the definitive decision sorry | 08:11 |
| *** trinaths has joined #openstack-infra | 08:11 | |
| jhesketh | I'll make sure it's in my calendar for next week | 08:11 |
| *** ddieterly has joined #openstack-infra | 08:11 | |
| anteaya | not to worry | 08:13 |
| anteaya | and thank you | 08:13 |
| *** ddieterly has quit IRC | 08:14 | |
| *** jgallard has joined #openstack-infra | 08:15 | |
| *** ddieterly has joined #openstack-infra | 08:15 | |
| *** teran has joined #openstack-infra | 08:16 | |
| *** afazekas has joined #openstack-infra | 08:18 | |
| *** arxcruz has joined #openstack-infra | 08:18 | |
| *** yfried_ is now known as yfried|afk | 08:19 | |
| *** ddieterly has quit IRC | 08:19 | |
| *** mpaolino has joined #openstack-infra | 08:19 | |
| *** jistr has joined #openstack-infra | 08:20 | |
| *** yfried|afk is now known as yfried_ | 08:21 | |
| *** teran has quit IRC | 08:22 | |
| *** andreaf has quit IRC | 08:23 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Conditionally check for targets in reconfigureManagers https://review.openstack.org/129560 | 08:23 |
| *** ihrachyshka has joined #openstack-infra | 08:25 | |
| *** dtantsur|afk is now known as dtantsur | 08:27 | |
| *** mpaolino has quit IRC | 08:27 | |
| *** amuller has joined #openstack-infra | 08:30 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Move image uploads out of ProviderManager https://review.openstack.org/137401 | 08:31 |
| *** trinaths has quit IRC | 08:32 | |
| *** asettle has quit IRC | 08:33 | |
| *** jcoufal has joined #openstack-infra | 08:33 | |
| *** salv-orlando has joined #openstack-infra | 08:33 | |
| *** Masahiro has joined #openstack-infra | 08:33 | |
| *** zz_avozza is now known as avozza | 08:33 | |
| *** luqas has joined #openstack-infra | 08:34 | |
| *** avozza is now known as zz_avozza | 08:36 | |
| *** salv-orlando has quit IRC | 08:38 | |
| *** Masahiro has quit IRC | 08:38 | |
| *** yamamoto_ has quit IRC | 08:39 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Move image uploads out of ProviderManager https://review.openstack.org/137401 | 08:39 |
| *** otter768 has joined #openstack-infra | 08:41 | |
| *** mpaolino has joined #openstack-infra | 08:42 | |
| *** doude has joined #openstack-infra | 08:43 | |
| *** MaxV has joined #openstack-infra | 08:44 | |
| *** vigneshvar_ has joined #openstack-infra | 08:44 | |
| *** ihrachyshka has quit IRC | 08:45 | |
| *** otter768 has quit IRC | 08:45 | |
| *** markus_z has joined #openstack-infra | 08:47 | |
| *** vigneshvar has quit IRC | 08:48 | |
| *** berendt has joined #openstack-infra | 08:49 | |
| *** ala_ has joined #openstack-infra | 08:51 | |
| *** e0ne has quit IRC | 08:53 | |
| *** zz_avozza is now known as avozza | 08:55 | |
| *** jcoufal_ has joined #openstack-infra | 08:55 | |
| odyssey4me | interesting - the external CI build (on physical servers) seems to have taken an hour less for https://review.openstack.org/#/c/139264/ versus https://review.openstack.org/#/c/140096/ | 08:58 |
| odyssey4me | unless my morning math is squiff, which is entirely likely | 08:58 |
| *** jcoufal has quit IRC | 08:59 | |
| *** Hal_ has joined #openstack-infra | 08:59 | |
| *** dizquierdo has joined #openstack-infra | 09:01 | |
| *** mpaolino has quit IRC | 09:01 | |
| *** mmaglana has quit IRC | 09:03 | |
| openstackgerrit | Ann Kamyshnikova proposed openstack/requirements: Set alembic>=0.7.1 https://review.openstack.org/140275 | 09:04 |
| odyssey4me | whoops - wrong channel :p | 09:04 |
| *** mpaolino has joined #openstack-infra | 09:04 | |
| *** luqas has quit IRC | 09:11 | |
| *** luqas has joined #openstack-infra | 09:12 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Move image uploads out of ProviderManager https://review.openstack.org/137401 | 09:14 |
| *** avozza is now known as zz_avozza | 09:15 | |
| *** zz_avozza is now known as avozza | 09:16 | |
| *** skolekonov has joined #openstack-infra | 09:20 | |
| *** tkelsey has joined #openstack-infra | 09:22 | |
| *** HeOS has joined #openstack-infra | 09:26 | |
| *** tkelsey has quit IRC | 09:26 | |
| odyssey4me | AJaeger ping | 09:32 |
| AJaeger | hi odyssey4me ! | 09:32 |
| odyssey4me | good morning :) | 09:32 |
| AJaeger | good morning to you, odyssey4me ! | 09:32 |
| *** Longgeek has quit IRC | 09:32 | |
| odyssey4me | I wonder if you know - is it possible to get interactive access to an instance built through the zuul/jenkins pipeline for a job before it completes. | 09:33 |
| odyssey4me | ie can we prevent the job from exiting and cleaning up, and ssh to the instance and troubleshoot on it | 09:33 |
| AJaeger | odyssey4me, I don't know ;( | 09:33 |
| odyssey4me | I've wasted several days trying to figure out why a build isn't working, even though it works as-is on a public cloud server I built | 09:34 |
| AJaeger | odyssey4me, I expect that this is not possible for the jobs we run in the OpenStack CI but I'm not authoritative here. | 09:34 |
| odyssey4me | ok, any thoughts on who may know? | 09:34 |
| *** jedimike has joined #openstack-infra | 09:34 | |
| *** derekh has joined #openstack-infra | 09:35 | |
| AJaeger | odyssey4me, I suggest to ask on #openstack-qa for help. In 4 hours time sdague and fungi should be here as well, I hope either of them can help. | 09:35 |
| *** nfedotov has joined #openstack-infra | 09:35 | |
| *** Longgeek has joined #openstack-infra | 09:35 | |
| odyssey4me | thanks AJaeger :) we really do need to get a better spread of -infra skills across time zones | 09:36 |
| gilliard | Morning Infra! Do you take care of stackalytics? | 09:36 |
| AJaeger | odyssey4me, are you volunteering? ;) | 09:36 |
| AJaeger | gilliard, good morning. No, infra is not taking care of stackalytics | 09:36 |
| AJaeger | gilliard, what'S your problem? | 09:37 |
| odyssey4me | lol AJaeger - I am interested, but I need a bit of time to settle in to a new role | 09:37 |
| gilliard | AJaeger: thanks. The problem is that there doesn't seem to be an authoritative list of person->irc_nick anywhere. | 09:37 |
| gilliard | I thought it might live in stackalytics? | 09:37 |
| gilliard | https://wiki.openstack.org/wiki/People is quite out of date | 09:38 |
| AJaeger | gilliard, I'm not aware of any such list. | 09:38 |
| AJaeger | gilliard, there's none in the tools AFAIK. Didn't know about the page, willl add myself now ;) | 09:39 |
| gilliard | AJaeger: OK cool :) I'm not there either | 09:40 |
| gilliard | yet. | 09:40 |
| *** isviridov_away is now known as isviridov | 09:41 | |
| gilliard | I'll stick something out on openstack-dev about it. | 09:41 |
| *** jp_at_hp has joined #openstack-infra | 09:42 | |
| *** andreykurilin_ has joined #openstack-infra | 09:42 | |
| AJaeger | gilliard, I'm added, now is your turn ;) | 09:42 |
| * gilliard on it | 09:43 | |
| *** e0ne has joined #openstack-infra | 09:44 | |
| *** berendt has quit IRC | 09:44 | |
| *** luqas has quit IRC | 09:46 | |
| *** luqas has joined #openstack-infra | 09:46 | |
| *** _nadya_ has joined #openstack-infra | 09:48 | |
| *** pelix has joined #openstack-infra | 09:48 | |
| *** andreykurilin_ has quit IRC | 09:49 | |
| *** andreykurilin_ has joined #openstack-infra | 09:49 | |
| *** dtantsur is now known as dtantsur|brb | 09:49 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: (WIP) Trying tests (WIP) https://review.openstack.org/140282 | 09:49 |
| *** Vivek has quit IRC | 09:49 | |
| AJaeger | gilliard, now you need to fill https://wiki.openstack.org/w/index.php?title=Gilliard&action=edit&redlink=1 with content ;) | 09:50 |
| AJaeger | Or remove the link to the page like many others did | 09:50 |
| *** punith has joined #openstack-infra | 09:50 | |
| gilliard | Err yeah I'll just remove the link I think :) | 09:50 |
| *** teran has joined #openstack-infra | 09:51 | |
| *** tkelsey has joined #openstack-infra | 09:52 | |
| *** ociuhandu has quit IRC | 09:54 | |
| *** Vivek has joined #openstack-infra | 09:56 | |
| *** ihrachyshka has joined #openstack-infra | 09:56 | |
| *** Vivek is now known as Guest27629 | 09:57 | |
| *** teran has quit IRC | 10:00 | |
| *** andreykurilin__ has joined #openstack-infra | 10:00 | |
| *** andreykurilin_ has quit IRC | 10:00 | |
| *** teran has joined #openstack-infra | 10:00 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: (WIP) Trying tests (WIP) https://review.openstack.org/140282 | 10:01 |
| *** stevemar has quit IRC | 10:04 | |
| *** amuller has quit IRC | 10:04 | |
| *** katyafervent has left #openstack-infra | 10:06 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: (WIP) Trying tests (WIP) https://review.openstack.org/140282 | 10:08 |
| *** fandi has quit IRC | 10:08 | |
| *** dizquierdo has quit IRC | 10:09 | |
| *** fandi has joined #openstack-infra | 10:20 | |
| *** jistr has quit IRC | 10:21 | |
| *** Masahiro has joined #openstack-infra | 10:22 | |
| *** viglesias has quit IRC | 10:22 | |
| *** mpaolino has quit IRC | 10:24 | |
| *** cdent has joined #openstack-infra | 10:24 | |
| *** ihrachyshka has quit IRC | 10:25 | |
| *** MaxV has quit IRC | 10:26 | |
| *** MaxV has joined #openstack-infra | 10:26 | |
| *** Masahiro has quit IRC | 10:27 | |
| *** MaxV has quit IRC | 10:33 | |
| *** viglesias has joined #openstack-infra | 10:33 | |
| *** MaxV has joined #openstack-infra | 10:33 | |
| *** MaxV has quit IRC | 10:33 | |
| *** AJaeger has quit IRC | 10:33 | |
| *** MaxV has joined #openstack-infra | 10:33 | |
| *** alex7376 has joined #openstack-infra | 10:35 | |
| *** ldnunes has joined #openstack-infra | 10:37 | |
| *** mpaolino has joined #openstack-infra | 10:39 | |
| *** signed8bit has joined #openstack-infra | 10:41 | |
| *** otter768 has joined #openstack-infra | 10:42 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: (WIP) Trying tests (WIP) https://review.openstack.org/140282 | 10:43 |
| *** nelsnelson has joined #openstack-infra | 10:45 | |
| *** otter768 has quit IRC | 10:46 | |
| *** mrmartin has quit IRC | 10:46 | |
| *** mrmartin has joined #openstack-infra | 10:47 | |
| *** mrmartin has quit IRC | 10:47 | |
| *** andreykurilin__ has quit IRC | 10:48 | |
| *** ruhe has quit IRC | 10:48 | |
| *** SergeyLukjanov has quit IRC | 10:48 | |
| *** dmitryme has quit IRC | 10:48 | |
| *** alazarev has quit IRC | 10:48 | |
| *** DinaBelova has quit IRC | 10:48 | |
| *** NikitaKonovalov has quit IRC | 10:48 | |
| *** DinaBelova has joined #openstack-infra | 10:48 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: (WIP) Trying tests (WIP) https://review.openstack.org/140282 | 10:49 |
| *** jistr has joined #openstack-infra | 10:49 | |
| *** alazarev has joined #openstack-infra | 10:49 | |
| *** dmitryme has joined #openstack-infra | 10:49 | |
| *** NikitaKonovalov has joined #openstack-infra | 10:49 | |
| *** ruhe has joined #openstack-infra | 10:50 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:50 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: (WIP) Trying tests (WIP) https://review.openstack.org/140282 | 10:51 |
| *** luqas__ has joined #openstack-infra | 10:54 | |
| *** luqas has quit IRC | 10:54 | |
| *** Longgeek has quit IRC | 10:54 | |
| *** luqas__ has quit IRC | 10:55 | |
| *** Longgeek has joined #openstack-infra | 10:56 | |
| *** luqas__ has joined #openstack-infra | 10:56 | |
| *** ZZelle has quit IRC | 10:57 | |
| *** jgallard has quit IRC | 10:57 | |
| *** ZZelle has joined #openstack-infra | 10:57 | |
| *** luqas___ has joined #openstack-infra | 10:58 | |
| *** luqas__ has quit IRC | 10:58 | |
| *** luqas__ has joined #openstack-infra | 11:00 | |
| *** pblaho has joined #openstack-infra | 11:01 | |
| *** luqas___ has quit IRC | 11:01 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: (WIP) Trying tests (WIP) https://review.openstack.org/140282 | 11:02 |
| openstackgerrit | Sergey Skripnick proposed openstack-infra/project-config: Fix publishing rally html results https://review.openstack.org/140296 | 11:03 |
| *** ihrachyshka has joined #openstack-infra | 11:03 | |
| *** luqas has joined #openstack-infra | 11:04 | |
| *** luqas__ has quit IRC | 11:04 | |
| *** salv-orlando has joined #openstack-infra | 11:04 | |
| *** rlandy has joined #openstack-infra | 11:05 | |
| *** luqas has quit IRC | 11:05 | |
| *** vponomaryov has quit IRC | 11:05 | |
| *** luqas has joined #openstack-infra | 11:05 | |
| *** Hal_ has quit IRC | 11:05 | |
| *** luqas has quit IRC | 11:09 | |
| *** luqas has joined #openstack-infra | 11:09 | |
| *** salv-orlando has quit IRC | 11:09 | |
| *** MaxV has quit IRC | 11:10 | |
| openstackgerrit | Tomas Sedovic proposed openstack-infra/project-config: Use the no-mergepy templates in a TripleO job https://review.openstack.org/139102 | 11:11 |
| *** luqas has quit IRC | 11:11 | |
| *** luqas has joined #openstack-infra | 11:12 | |
| *** luqas has quit IRC | 11:14 | |
| *** luqas__ has joined #openstack-infra | 11:14 | |
| *** nikil22 has joined #openstack-infra | 11:14 | |
| *** punith has quit IRC | 11:14 | |
| *** luqas__ has quit IRC | 11:16 | |
| *** luqas has joined #openstack-infra | 11:16 | |
| *** alexpilotti has joined #openstack-infra | 11:17 | |
| *** luqas has quit IRC | 11:17 | |
| *** _nadya_ has quit IRC | 11:18 | |
| *** luqas has joined #openstack-infra | 11:18 | |
| *** luqas has quit IRC | 11:20 | |
| *** luqas has joined #openstack-infra | 11:20 | |
| *** MaxV has joined #openstack-infra | 11:22 | |
| *** SumitNaiksatam has quit IRC | 11:22 | |
| *** luqas has quit IRC | 11:23 | |
| *** ociuhandu has joined #openstack-infra | 11:23 | |
| *** luqas has joined #openstack-infra | 11:23 | |
| *** SumitNaiksatam has joined #openstack-infra | 11:26 | |
| *** luqas__ has joined #openstack-infra | 11:26 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: (WIP) Trying tests (WIP) https://review.openstack.org/140282 | 11:27 |
| *** luqas has quit IRC | 11:27 | |
| *** talluri has quit IRC | 11:27 | |
| *** hashar has joined #openstack-infra | 11:27 | |
| *** luqas___ has joined #openstack-infra | 11:28 | |
| *** luqas__ has quit IRC | 11:29 | |
| openstackgerrit | Jean-Frédéric proposed openstack-infra/jenkins-job-builder: Add Mercurial plugin feature to jenkins-job-builder https://review.openstack.org/139136 | 11:29 |
| *** luqas__ has joined #openstack-infra | 11:31 | |
| *** luqas___ has quit IRC | 11:31 | |
| *** signed8bit has quit IRC | 11:32 | |
| *** amuller has joined #openstack-infra | 11:32 | |
| *** luqas__ has quit IRC | 11:33 | |
| *** luqas__ has joined #openstack-infra | 11:33 | |
| *** AJaeger has joined #openstack-infra | 11:34 | |
| *** AJaeger has joined #openstack-infra | 11:34 | |
| *** luqas__ has quit IRC | 11:36 | |
| *** luqas__ has joined #openstack-infra | 11:36 | |
| *** dmsimard_away is now known as dmsimard | 11:37 | |
| *** tkelsey has quit IRC | 11:39 | |
| *** luqas has joined #openstack-infra | 11:39 | |
| *** signed8bit has joined #openstack-infra | 11:40 | |
| *** luqas__ has quit IRC | 11:40 | |
| *** aysyd has joined #openstack-infra | 11:43 | |
| *** MaxV has quit IRC | 11:44 | |
| *** dtantsur|brb is now known as dtantsur | 11:46 | |
| *** dims has joined #openstack-infra | 11:48 | |
| *** e0ne has quit IRC | 11:49 | |
| *** hashar has quit IRC | 11:53 | |
| *** andreaf has joined #openstack-infra | 11:53 | |
| *** e0ne has joined #openstack-infra | 11:55 | |
| *** tnovacik has quit IRC | 11:58 | |
| *** cdent has quit IRC | 11:59 | |
| *** MaxV has joined #openstack-infra | 12:00 | |
| *** adalbas has quit IRC | 12:01 | |
| *** vponomaryov has joined #openstack-infra | 12:02 | |
| *** luqas has quit IRC | 12:02 | |
| *** luqas has joined #openstack-infra | 12:02 | |
| *** adalbas has joined #openstack-infra | 12:04 | |
| *** adalbas has quit IRC | 12:06 | |
| *** zhiwei has quit IRC | 12:06 | |
| *** baoli has joined #openstack-infra | 12:07 | |
| *** teran has quit IRC | 12:08 | |
| *** teran has joined #openstack-infra | 12:09 | |
| *** teran_ has joined #openstack-infra | 12:10 | |
| *** Masahiro has joined #openstack-infra | 12:12 | |
| *** adalbas has joined #openstack-infra | 12:12 | |
| *** cnesa has joined #openstack-infra | 12:13 | |
| openstackgerrit | Tomas Sedovic proposed openstack-infra/project-config: Use the no-mergepy templates in a TripleO job https://review.openstack.org/139102 | 12:13 |
| *** teran has quit IRC | 12:13 | |
| *** baoli has quit IRC | 12:15 | |
| *** Masahiro has quit IRC | 12:15 | |
| *** baoli has joined #openstack-infra | 12:16 | |
| *** bradjones has joined #openstack-infra | 12:20 | |
| *** luqas has quit IRC | 12:24 | |
| *** luqas has joined #openstack-infra | 12:24 | |
| *** chandankumar has quit IRC | 12:25 | |
| *** luqas has quit IRC | 12:26 | |
| *** luqas has joined #openstack-infra | 12:27 | |
| *** smoser has joined #openstack-infra | 12:27 | |
| *** talluri has joined #openstack-infra | 12:28 | |
| *** luqas has quit IRC | 12:29 | |
| *** jamielennox is now known as jamielennox|away | 12:29 | |
| *** luqas has joined #openstack-infra | 12:29 | |
| *** obondarev has joined #openstack-infra | 12:29 | |
| *** dizquierdo has joined #openstack-infra | 12:30 | |
| *** yfried_ is now known as yfried|afk | 12:30 | |
| *** signed8b_ has joined #openstack-infra | 12:31 | |
| *** e0ne has quit IRC | 12:31 | |
| *** luqas has quit IRC | 12:31 | |
| *** luqas has joined #openstack-infra | 12:32 | |
| *** talluri has quit IRC | 12:32 | |
| *** luqas has quit IRC | 12:34 | |
| *** luqas has joined #openstack-infra | 12:34 | |
| *** signed8bit has quit IRC | 12:35 | |
| *** oomichi has quit IRC | 12:36 | |
| *** yfried|afk is now known as yfried_ | 12:36 | |
| *** luqas has quit IRC | 12:36 | |
| *** luqas has joined #openstack-infra | 12:36 | |
| *** luqas__ has joined #openstack-infra | 12:39 | |
| *** luqas has quit IRC | 12:40 | |
| *** e0ne has joined #openstack-infra | 12:40 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (status) https://review.openstack.org/140316 | 12:40 |
| *** xianghui has quit IRC | 12:41 | |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: builders: add cmakebuilder plugin support https://review.openstack.org/134199 | 12:42 |
| *** otter768 has joined #openstack-infra | 12:42 | |
| *** hashar has joined #openstack-infra | 12:45 | |
| *** alexpilotti has quit IRC | 12:46 | |
| *** otter768 has quit IRC | 12:47 | |
| *** yamamoto has joined #openstack-infra | 12:48 | |
| *** teran_ has quit IRC | 12:49 | |
| *** ddieterly has joined #openstack-infra | 12:50 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (status) https://review.openstack.org/140316 | 12:51 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: builders: add cmakebuilder plugin support https://review.openstack.org/134199 | 12:53 |
| *** denis_makogon has joined #openstack-infra | 12:53 | |
| *** talluri has joined #openstack-infra | 12:54 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Fix Designate release publication https://review.openstack.org/140322 | 12:55 |
| *** hashar has quit IRC | 12:56 | |
| ttx | Kiall: ^ | 12:56 |
| Kiall | ttx: +1 :) | 12:57 |
| *** Longgeek has quit IRC | 12:58 | |
| *** mbacchi has joined #openstack-infra | 13:00 | |
| *** mpaolino has quit IRC | 13:03 | |
| *** Longgeek has joined #openstack-infra | 13:04 | |
| *** weshay has joined #openstack-infra | 13:06 | |
| *** salv-orlando has joined #openstack-infra | 13:06 | |
| *** signed8b_ has quit IRC | 13:06 | |
| *** dprince has joined #openstack-infra | 13:08 | |
| *** xianghui has joined #openstack-infra | 13:08 | |
| *** lttrl has joined #openstack-infra | 13:10 | |
| *** Kiall has quit IRC | 13:11 | |
| *** Kiall has joined #openstack-infra | 13:11 | |
| *** salv-orlando has quit IRC | 13:11 | |
| *** mfink has quit IRC | 13:12 | |
| *** xianghui has quit IRC | 13:14 | |
| *** obondarev has quit IRC | 13:14 | |
| *** teran_ has joined #openstack-infra | 13:14 | |
| *** salv-orlando has joined #openstack-infra | 13:17 | |
| *** skolekonov has quit IRC | 13:17 | |
| *** enikanorov__ has joined #openstack-infra | 13:18 | |
| *** radez_g0n3 is now known as radez | 13:18 | |
| *** tnurlygayanov has quit IRC | 13:18 | |
| *** enikanorov_ has quit IRC | 13:18 | |
| *** skolekonov has joined #openstack-infra | 13:18 | |
| *** doug-fish has joined #openstack-infra | 13:18 | |
| *** tnurlygayanov has joined #openstack-infra | 13:20 | |
| AJaeger | ttx, SergeyLukjanov, looking at 140322: sahara also uses publish_to_pypi - is that usage ok? | 13:21 |
| SergeyLukjanov | AJaeger, hey, lookng | 13:23 |
| *** obondarev has joined #openstack-infra | 13:23 | |
| ttx | AJaeger: probably not | 13:24 |
| AJaeger | ttx, I'll do a followup on your patch and create a server-release template... | 13:25 |
| *** bswartz has quit IRC | 13:25 | |
| AJaeger | ttx, do you want to amend your patch or should we do a second one? | 13:25 |
| *** alexpilotti has joined #openstack-infra | 13:25 | |
| ttx | AJaeger: I can amend | 13:25 |
| AJaeger | ttx, let'S do it so that I can +2 ;) | 13:26 |
| ttx | hehe | 13:26 |
| *** vigneshvar_ has quit IRC | 13:31 | |
| *** yfried_ is now known as yfried|afk | 13:32 | |
| *** talluri has quit IRC | 13:32 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Introduce openstack-server-release-jobs template https://review.openstack.org/140326 | 13:35 |
| AJaeger | ttx, I'll rebase on top of your amend ^ | 13:35 |
| *** Adri2000 has quit IRC | 13:36 | |
| AJaeger | ttx, what about tuskar ? IT has some upload jobs as well | 13:36 |
| ttx | not integrated/incubated, so I don't care as much | 13:36 |
| *** bknudson has quit IRC | 13:37 | |
| AJaeger | ;) | 13:37 |
| ttx | checking all the ones in my list | 13:38 |
| AJaeger | ttx, tuskar is part of TripleO - seems in the wrong section in the file, time for another housekeeping patch | 13:39 |
| *** _nadya_ has joined #openstack-infra | 13:39 | |
| *** miqui has joined #openstack-infra | 13:42 | |
| AJaeger | I wonder whether we need these for neutron-fwaas and friends as well? Or how will those get released? | 13:42 |
| ttx | it's an open question I'll ask mestery later today | 13:43 |
| AJaeger | thanks, ttx! | 13:43 |
| *** mfink has joined #openstack-infra | 13:43 | |
| *** kgiusti has joined #openstack-infra | 13:43 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Fix release publication for server projects https://review.openstack.org/140322 | 13:44 |
| *** vigneshvar_ has joined #openstack-infra | 13:44 | |
| ttx | AJaeger: should be good now ^ | 13:44 |
| *** rushiagr is now known as rushiagr_away | 13:44 | |
| ttx | Also added the missing merge-release-tags jobs, so your sshould be noop | 13:44 |
| AJaeger | ttx, will update my patch now and then double check everything. thanks! | 13:45 |
| *** ddieterly has quit IRC | 13:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Introduce openstack-server-release-jobs template https://review.openstack.org/140326 | 13:48 |
| *** Hal_ has joined #openstack-infra | 13:49 | |
| *** dkliban_afk is now known as dkliban | 13:49 | |
| *** vigneshvar_ has quit IRC | 13:49 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Introduce openstack-server-release-jobs template https://review.openstack.org/140326 | 13:50 |
| *** Longgeek_ has joined #openstack-infra | 13:52 | |
| *** Longgeek has quit IRC | 13:52 | |
| *** yfried|afk is now known as yfried_ | 13:52 | |
| *** Adri2000 has joined #openstack-infra | 13:52 | |
| *** Adri2000 has quit IRC | 13:52 | |
| *** Adri2000 has joined #openstack-infra | 13:52 | |
| AJaeger | SergeyLukjanov, do you want to double check 140322 since now Sahara was changed? | 13:52 |
| *** xyang1 has joined #openstack-infra | 13:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Introduce openstack-server-release-jobs template https://review.openstack.org/140326 | 13:57 |
| *** bswartz has joined #openstack-infra | 13:58 | |
| *** julim has joined #openstack-infra | 13:58 | |
| *** annegent_ has joined #openstack-infra | 13:58 | |
| *** Sukhdev has joined #openstack-infra | 13:59 | |
| *** mestery has joined #openstack-infra | 13:59 | |
| *** Masahiro has joined #openstack-infra | 14:00 | |
| *** koolhead17 has quit IRC | 14:00 | |
| *** koolhead17 has joined #openstack-infra | 14:01 | |
| *** hashar has joined #openstack-infra | 14:01 | |
| *** chandankumar has joined #openstack-infra | 14:01 | |
| *** yfried_ is now known as yfried|afk | 14:02 | |
| *** Masahiro has quit IRC | 14:04 | |
| *** koolhead17 has quit IRC | 14:05 | |
| *** baoli has quit IRC | 14:05 | |
| *** annegent_ has quit IRC | 14:06 | |
| derekh | jeblair: if you get a chance would you mind revisiting https://review.openstack.org/#/c/124334/4 | 14:09 |
| *** markmcclain has joined #openstack-infra | 14:10 | |
| *** luqas__ has quit IRC | 14:10 | |
| *** luqas__ has joined #openstack-infra | 14:10 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (storyboard) https://review.openstack.org/140335 | 14:11 |
| *** luqas__ has quit IRC | 14:12 | |
| *** luqas has joined #openstack-infra | 14:13 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (storyboard) https://review.openstack.org/140335 | 14:13 |
| *** luqas has quit IRC | 14:15 | |
| *** luqas has joined #openstack-infra | 14:15 | |
| *** dustins has joined #openstack-infra | 14:15 | |
| *** habib has joined #openstack-infra | 14:16 | |
| *** luqas has quit IRC | 14:17 | |
| *** jgallard has joined #openstack-infra | 14:17 | |
| *** luqas has joined #openstack-infra | 14:17 | |
| *** ddieterly has joined #openstack-infra | 14:17 | |
| *** luqas has quit IRC | 14:19 | |
| *** luqas has joined #openstack-infra | 14:20 | |
| nikil22 | hi is there any existing puppet scripts with will install jenkins,zuul,JJB for third party CI system | 14:20 |
| *** changbl has quit IRC | 14:20 | |
| *** k4n0 has quit IRC | 14:21 | |
| *** luqas has quit IRC | 14:22 | |
| *** Longgeek_ has quit IRC | 14:22 | |
| *** luqas has joined #openstack-infra | 14:22 | |
| odyssey4me | nikil22: this blog post takes you through it, and has puppet modules referenced: http://www.joinfu.com/2014/02/setting-up-an-external-openstack-testing-system/ | 14:23 |
| *** salv-orlando has quit IRC | 14:23 | |
| odyssey4me | (I haven't gone through it myself) | 14:23 |
| *** luqas has quit IRC | 14:24 | |
| *** cpowell has joined #openstack-infra | 14:24 | |
| *** luqas has joined #openstack-infra | 14:24 | |
| nikil22 | odyssey4me:I am trying the same, and it tried to download some script called install_puppet.sh form openstack-infra repo and the path fails. So i am not able to proceed with it. I have raised the details issue in his github issue here https://github.com/jaypipes/os-ext-testing-data/issues/6 | 14:25 |
| jaypipes | nikhil_k: there is a big warning at the top of that blog post about this... | 14:25 |
| jaypipes | nikhil_k: bottom line, I have not had the time to update the blog entries and retry the system with the updated upstream CI tooling/puppet modules. | 14:26 |
| *** ryanpetrello has joined #openstack-infra | 14:26 | |
| *** luqas__ has joined #openstack-infra | 14:27 | |
| *** eharney has joined #openstack-infra | 14:27 | |
| nikil22 | jaypipes: thanks let me check for it | 14:28 |
| *** luqas has quit IRC | 14:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Introduce openstack-server-release-jobs template https://review.openstack.org/140326 | 14:29 |
| *** KanagarajM has joined #openstack-infra | 14:29 | |
| *** luqas__ has quit IRC | 14:29 | |
| *** Longgeek has joined #openstack-infra | 14:29 | |
| *** mestery has quit IRC | 14:29 | |
| *** luqas has joined #openstack-infra | 14:29 | |
| nikil22 | jaypipes: do you know what is the exact new path for install_puppet.sh in openstack-infra repo? Atleast i can try to remodify the path and execute it again | 14:31 |
| jaypipes | nikhil_k: one sec | 14:31 |
| *** markmcclain has quit IRC | 14:31 | |
| *** avozza is now known as zz_avozza | 14:31 | |
| openstackgerrit | Merged openstack/requirements: Add oslo.context https://review.openstack.org/140141 | 14:31 |
| *** signed8bit has joined #openstack-infra | 14:31 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/devstack-gate: Add tooz in project list https://review.openstack.org/140343 | 14:32 |
| *** mwagner_lap has joined #openstack-infra | 14:32 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 14:32 | |
| *** luqas__ has joined #openstack-infra | 14:32 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 14:32 | |
| *** baoli has joined #openstack-infra | 14:32 | |
| openstackgerrit | Julien Danjou proposed openstack/requirements: Add tooz to project list to update https://review.openstack.org/140344 | 14:33 |
| *** luqas has quit IRC | 14:33 | |
| *** mattfarina has joined #openstack-infra | 14:33 | |
| jaypipes | nikhil_k: https://github.com/openstack-infra/system-config/blob/master/install_puppet.sh | 14:33 |
| jaypipes | nikhil_k: it's in the system-config repo now. | 14:33 |
| *** luqas__ has quit IRC | 14:34 | |
| *** luqas___ has joined #openstack-infra | 14:34 | |
| nikil22 | jaypipes: thanks let me modify the path and try to run the script | 14:35 |
| *** [HeOS] has joined #openstack-infra | 14:36 | |
| *** zz_avozza is now known as avozza | 14:36 | |
| *** HeOS has quit IRC | 14:36 | |
| *** luqas___ has quit IRC | 14:37 | |
| *** luqas has joined #openstack-infra | 14:37 | |
| *** bknudson has joined #openstack-infra | 14:38 | |
| *** luqas has quit IRC | 14:39 | |
| *** luqas has joined #openstack-infra | 14:39 | |
| *** Longgeek has quit IRC | 14:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix release publication for server projects https://review.openstack.org/140322 | 14:40 |
| *** wenlock has joined #openstack-infra | 14:42 | |
| *** otter768 has joined #openstack-infra | 14:43 | |
| *** Longgeek has joined #openstack-infra | 14:44 | |
| *** craigbr has joined #openstack-infra | 14:44 | |
| *** hdd has joined #openstack-infra | 14:45 | |
| nikil22 | hi is there any way to wget the file directory from the repo tree "http://git.openstack.org/cgit/openstack-infra/system-config/tree/install_puppet.sh"? | 14:47 |
| *** otter768 has quit IRC | 14:48 | |
| *** rushiagr_away is now known as rushiagr | 14:48 | |
| *** e0ne has quit IRC | 14:49 | |
| openstackgerrit | Maish Saidel-Keesing proposed openstack-infra/project-config: Add rss.opml file to repository https://review.openstack.org/140041 | 14:50 |
| *** e0ne has joined #openstack-infra | 14:51 | |
| *** bdpayne has joined #openstack-infra | 14:53 | |
| *** yapeng_ has joined #openstack-infra | 14:54 | |
| *** alexpilotti_ has joined #openstack-infra | 14:54 | |
| fungi | odyssey4me: if all you want to do is reproduce a job run, it should be fairly straightforward. you need an appropriately sized cloud vm of whatever distro, run a couple scripts from the openstack-infra/system-config repo and then run the commands from the builders in your job. no need for any additional systems | 14:55 |
| fungi | gilliard: i replied on the ml thread too, but foundation member profiles can already be linked to irc nicks | 14:56 |
| AJaeger | fungi, I checked my profile and had my IRC nick in there - still didn't remember it ;( | 14:56 |
| odyssey4me | fungi: ok - the trouble I'm hitting is that I have a perfectly working build script on a standard cloud server, but inside openstack-ci is fails every time... so I'm trying to figure out what's interfering | 14:56 |
| *** zz_dimtruck is now known as dimtruck | 14:56 | |
| fungi | odyssey4me: have a link to the log? i might just be able to spot the issue | 14:57 |
| *** alexpilotti has quit IRC | 14:57 | |
| *** alexpilotti_ is now known as alexpilotti | 14:57 | |
| *** signed8b_ has joined #openstack-infra | 14:58 | |
| odyssey4me | fungi sure http://logs.openstack.org/29/139629/31/check/check-os-ansible-deployment-dsvm-aio-build/8754434/console.html | 14:58 |
| *** _nadya_ has quit IRC | 14:59 | |
| odyssey4me | effectively what I've worked through thus far is that from inside the galera container, a connection to mysql works - from the host of all containers, a connection to mysql works - but from inside the keystone container on that host, a connection to mysql (in the galera container) doesn't work | 14:59 |
| fungi | odyssey4me: so potentially networking between containers, eh? | 15:00 |
| *** yfried|afk is now known as yfried_ | 15:00 | |
| *** eharney has quit IRC | 15:00 | |
| *** rushiagr is now known as rushiagr_away | 15:00 | |
| *** luqas has quit IRC | 15:00 | |
| *** mpaolino has joined #openstack-infra | 15:00 | |
| *** luqas has joined #openstack-infra | 15:01 | |
| *** zz_jgrimm is now known as jgrimm | 15:01 | |
| odyssey4me | fungi so it would seem - note that the containers are all bridged onto a bridge device which has no bridge_port - it's all on one host so they shouldn't require a physical device backing them up | 15:01 |
| *** signed8bit has quit IRC | 15:01 | |
| fungi | odyssey4me: and it's expecting to reach mysql on the "internal_lb_vip_address" according to that log | 15:01 |
| *** k4n0 has joined #openstack-infra | 15:01 | |
| *** armax has joined #openstack-infra | 15:02 | |
| *** luqas has quit IRC | 15:02 | |
| odyssey4me | fungi: yes - that address is on the host (ie the instance) on the br-mgmt bridge | 15:02 |
| *** luqas has joined #openstack-infra | 15:02 | |
| *** eharney has joined #openstack-infra | 15:03 | |
| *** obondarev has quit IRC | 15:03 | |
| odyssey4me | there is a test that checks that the mysql connection from the host through that address to the container works | 15:03 |
| nikhil_k | jaypipes: guessing none those messages were for me ? | 15:03 |
| *** [HeOS] has quit IRC | 15:03 | |
| *** shakamunyi has joined #openstack-infra | 15:04 | |
| *** shakamunyi has quit IRC | 15:04 | |
| fungi | odyssey4me: and this failure happened on a trusty instance in hpcloud. have you witnessed identical failures in rackspace too (jobs run in arbitrary service providers so if you have a few failing runs you should be able to check the node id in the third line to see which provider it was in) | 15:04 |
| *** shakamunyi has joined #openstack-infra | 15:05 | |
| odyssey4me | fungi: yep - it's always failing at the exact same point every time... but if I test it in a rackspace cloud account it works just fine | 15:05 |
| *** avozza is now known as zz_avozza | 15:05 | |
| odyssey4me | at the exact same time in both providers | 15:05 |
| *** zz_avozza is now known as avozza | 15:05 | |
| odyssey4me | it may be useful for me to try running it after running the appropriate puppet scripts used in infra first? | 15:06 |
| *** [HeOS] has joined #openstack-infra | 15:07 | |
| *** amitgandhinz has joined #openstack-infra | 15:07 | |
| fungi | odyssey4me: git clone stackforge/os-ansible-deployment; cd os-ansible-deployment; sudo pip install ansible==1.6.10; sudo scripts/os-ansible-aio-check.sh | 15:07 |
| fungi | odyssey4me: that's the rough steps to reproduce presumably? | 15:07 |
| *** jgrimm is now known as zz_jgrimm | 15:08 | |
| *** pblaho has quit IRC | 15:08 | |
| sdague | the git servers seem to have fallen over again | 15:09 |
| odyssey4me | fungi: skip the ansible installation, the script will do it for you - the rest is correct | 15:09 |
| odyssey4me | fungi: which flavor & image do the instances use ? | 15:10 |
| fungi | sdague: that's probably us (or other people running nodepool with the same image update schedule). when i was doing it manually yesterday, it was something like 78 images being built in parallel across 8 providers/regions, all cloning git repositories | 15:11 |
| sdague | fungi: so... that seems problematic if our git servers fall over on our node rebuilds | 15:11 |
| *** _nadya_ has joined #openstack-infra | 15:11 | |
| fungi | sdague: i agree. we might want to consider moving our update time and seeing if the problem follows, or if it's (maybe multiples of) someone else copying our present configuration | 15:12 |
| fungi | sdague: long term, dib solves this because we don't reclone, and we do it once per image type rather than multiplied by every provider/region | 15:13 |
| sdague | well that mitigates this issue | 15:13 |
| sdague | for people using it | 15:13 |
| fungi | sdague: part of the current problem is that we moved the image updates to a time when there are more people around, except now the load from image updates gets to compete with the load from the people who are around | 15:14 |
| *** rushiagr_away is now known as rushiagr | 15:14 | |
| fungi | odyssey4me: "Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM)" on an 8gb performance flavor | 15:14 |
| *** ChuckC_ has joined #openstack-infra | 15:15 | |
| *** ChuckC has quit IRC | 15:15 | |
| clarkb | odyssey4me: fungi: iptables | 15:16 |
| odyssey4me | fungi: thanks - going to build another one and test again - can you gist (or elsewhere) the prep steps that would be done via infra before executing the build? | 15:16 |
| *** yapeng_ is now known as yapeng | 15:16 | |
| odyssey4me | clarkb: iptables is configured in the base image or by puppet somewhere? | 15:17 |
| clarkb | by puppet in our base images | 15:17 |
| fungi | odyssey4me: clarkb may be right. iptables is applied on the instance interfaces so if you're not opening ports you're trying to hit there, it could complicate matters | 15:17 |
| *** pradk has joined #openstack-infra | 15:17 | |
| *** luqas has quit IRC | 15:18 | |
| clarkb | it has got to be iptables if you are connecting to not localhost | 15:18 |
| *** luqas has joined #openstack-infra | 15:18 | |
| odyssey4me | clarkb fungi that would do it then, and it would account for the difference between my other tests and openstack-ci | 15:18 |
| *** esker has joined #openstack-infra | 15:19 | |
| odyssey4me | so the best would be for me to add some rules to allow connections between the bridges | 15:19 |
| fungi | odyssey4me: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/single_use_slave.pp#n29 | 15:19 |
| fungi | odyssey4me: that's the base ruleset we add | 15:19 |
| fungi | odyssey4me: if you look at devstack, it does rather a lot of iptables manipulation when it sets itself up | 15:19 |
| *** luqas has quit IRC | 15:20 | |
| *** dustins_ has joined #openstack-infra | 15:20 | |
| *** luqas has joined #openstack-infra | 15:20 | |
| *** obondarev has joined #openstack-infra | 15:20 | |
| *** david-lyle has joined #openstack-infra | 15:21 | |
| *** gokrokve has joined #openstack-infra | 15:21 | |
| *** ChuckC_ has quit IRC | 15:21 | |
| fungi | odyssey4me: for example http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/ironic#n605 | 15:22 |
| odyssey4me | fungi clarkb thank you both - you rock | 15:22 |
| *** Sukhdev has quit IRC | 15:22 | |
| *** dustins has quit IRC | 15:22 | |
| mordred | fungi: you know - we could _Also_ add an rsync server on our git mirrors | 15:23 |
| fungi | odyssey4me: you may want to adjust your os-ansible-aio-check.sh script to run iptables -L (and ip6tables -L if relevant, and whatever else) and iterate from that | 15:23 |
| mordred | fungi: so that people, like us, who want to clone _everything_ can just rsync the git tree - almost certainly more effecient than letting git do it | 15:23 |
| fungi | mordred: or we could put them in afs and then mount that on the nodepool server. so many options | 15:23 |
| mordred | yah | 15:23 |
| *** salv-orlando has joined #openstack-infra | 15:23 | |
| mordred | adding a public read-only rsync server that serves that tree is probably the quickest and most non-invasive though :) | 15:24 |
| clarkb | so we should probably add another server if we didnt hit 800mbps on the haproxy node | 15:24 |
| mordred | ++ | 15:24 |
| clarkb | iirc 800mbps is our peak bw on that public ip | 15:24 |
| clarkb | and nova is the bad repo | 15:24 |
| fungi | mordred: also using afs would solve the "not all nodes are guaranteed to be in sync because they replicate independently" problem, and would be able to distribute across them more evenly (no hash persistence needed) | 15:24 |
| clarkb | we might look at ways of making nova less bad | 15:25 |
| *** shakamunyi has quit IRC | 15:25 | |
| *** shakamunyi has joined #openstack-infra | 15:25 | |
| *** salv-orlando has quit IRC | 15:25 | |
| *** salv-orl_ has joined #openstack-infra | 15:26 | |
| *** yamamoto has quit IRC | 15:26 | |
| *** avozza is now known as zz_avozza | 15:26 | |
| *** salv-orl_ has quit IRC | 15:26 | |
| *** obondarev has quit IRC | 15:26 | |
| mtreinish | clarkb: what's bad about it? Just size? | 15:26 |
| clarkb | it is the repo that makes git upload pack sit and use all the memory | 15:26 |
| clarkb | so yes likely size | 15:27 |
| fungi | number of objects | 15:27 |
| *** yfried_ is now known as yfried|afk | 15:27 | |
| *** zz_avozza is now known as avozza | 15:27 | |
| clarkb | thats the other thing to look at swap usage during this time period | 15:28 |
| clarkb | now that we have swap we may be digging in to it | 15:28 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add Networking-L2GW project https://review.openstack.org/140369 | 15:29 |
| fungi | whereas before we just had the kernel oom killer running wild aborting transfers in progress | 15:31 |
| *** zz_jgrimm is now known as jgrimm | 15:32 | |
| *** gokrokve has quit IRC | 15:32 | |
| *** gokrokve has joined #openstack-infra | 15:33 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas https://review.openstack.org/140164 | 15:33 |
| *** jerryz has quit IRC | 15:33 | |
| *** mpaolino has quit IRC | 15:33 | |
| *** wuhg has quit IRC | 15:33 | |
| fungi | clarkb: doesn't look like we're even under memory pressure on any of the git servers, and definitely no swap getting used | 15:34 |
| clarkb | so maybe not the same issue then | 15:35 |
| *** spzala has joined #openstack-infra | 15:36 | |
| *** afazekas has quit IRC | 15:36 | |
| *** koolhead17 has joined #openstack-infra | 15:37 | |
| *** koolhead17 has joined #openstack-infra | 15:37 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (summit) https://review.openstack.org/140374 | 15:37 |
| *** sdake has quit IRC | 15:38 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=862&rra_id=all ok we may actually be hitting our bw limits again | 15:39 |
| *** luqas has quit IRC | 15:39 | |
| clarkb | also, git.o.o would use the public interface for its backend connections too | 15:39 |
| *** rkukura has quit IRC | 15:39 | |
| clarkb | we could switch those connections to the "flat/internal/whatever" interfaces | 15:39 |
| *** luqas has joined #openstack-infra | 15:39 | |
| mtreinish | mordred: anything I can do to cajole you to review the mysql-proxy patch? I'd really like to stop using the fungi-proxy. | 15:39 |
| *** koolhead17 has quit IRC | 15:40 | |
| *** koolhead17 has joined #openstack-infra | 15:40 | |
| *** sdake has joined #openstack-infra | 15:41 | |
| *** sdake has quit IRC | 15:41 | |
| *** sdake has joined #openstack-infra | 15:41 | |
| *** baoli has quit IRC | 15:42 | |
| fungi | the fungi-proxy is inefficient and buggy | 15:45 |
| fungi | unreliable too | 15:45 |
| *** sabeen1 has joined #openstack-infra | 15:45 | |
| fungi | it's for experimental use only | 15:45 |
| mtreinish | fungi: heh, the few times I've used it though it works well :) | 15:45 |
| mordred | mtreinish: I'll look in just a bit | 15:47 |
| *** yfried|afk is now known as yfried_ | 15:47 | |
| *** baoli has joined #openstack-infra | 15:47 | |
| mtreinish | mordred: cool, thanks | 15:48 |
| *** Masahiro has joined #openstack-infra | 15:48 | |
| *** mrmartin has joined #openstack-infra | 15:48 | |
| *** reed has joined #openstack-infra | 15:49 | |
| *** ryanpetrello has quit IRC | 15:49 | |
| clarkb | I can stp puppet on git.o.o, update config and do a haproxy reload to switch over to the 10net addresses for backends. If that helps we can then merge the puppet change to do that during a quiet time because puppet will haproxy restart | 15:49 |
| fungi | clarkb: sounds fine to me | 15:50 |
| clarkb | but ya swap does not appera to be an issue or memory use at all during that time period | 15:52 |
| *** hdd has quit IRC | 15:53 | |
| fungi | probably ought to first double-check that git.o.o can _reach_ those addresses/ports on the rfc-1918 backend though | 15:53 |
| *** Masahiro has quit IRC | 15:53 | |
| *** armax has quit IRC | 15:54 | |
| *** ChuckC_ has joined #openstack-infra | 15:54 | |
| clarkb | yup | 15:54 |
| *** tonytan4ever has joined #openstack-infra | 15:54 | |
| clarkb | I have confirmed that http and git:// work for git01 so far will get to the rest a bit later | 15:55 |
| clarkb | also no telnet or nc | 15:55 |
| clarkb | but using git directly works | 15:55 |
| *** mestery has joined #openstack-infra | 15:55 | |
| *** yaguang has quit IRC | 15:55 | |
| mrmartin | morning | 15:55 |
| *** salv-orlando has joined #openstack-infra | 15:56 | |
| *** talluri has joined #openstack-infra | 15:57 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (subunit_worker) https://review.openstack.org/140383 | 15:57 |
| *** obondarev has joined #openstack-infra | 15:57 | |
| mrmartin | fungi, clarkb: https://storyboard.openstack.org/#!/story/2000071 the puppet modules seems to be broken on precise gating | 15:57 |
| jeblair | derekh: done, thanks | 15:57 |
| *** yfried_ is now known as yfried|afk | 15:57 | |
| derekh | jeblair: no prob, thank you | 15:58 |
| *** ChuckC_ has quit IRC | 15:59 | |
| *** zz_sabari has quit IRC | 16:00 | |
| *** afazekas has joined #openstack-infra | 16:01 | |
| *** jcoufal_ has quit IRC | 16:02 | |
| *** Sukhdev has joined #openstack-infra | 16:03 | |
| *** rkukura has joined #openstack-infra | 16:04 | |
| *** talluri has quit IRC | 16:05 | |
| *** zz_sabari has joined #openstack-infra | 16:05 | |
| *** wenlock_ has joined #openstack-infra | 16:06 | |
| *** pcrews has joined #openstack-infra | 16:06 | |
| *** talluri has joined #openstack-infra | 16:07 | |
| *** markmcclain has joined #openstack-infra | 16:07 | |
| *** belmoreira has quit IRC | 16:09 | |
| *** pc_m has joined #openstack-infra | 16:12 | |
| *** wenlock_ has quit IRC | 16:12 | |
| *** ChuckC_ has joined #openstack-infra | 16:12 | |
| *** nfedotov has quit IRC | 16:12 | |
| *** ChuckC_ is now known as ChuckC | 16:13 | |
| *** Hal_ has quit IRC | 16:14 | |
| *** devlaps has joined #openstack-infra | 16:15 | |
| *** cnesa has quit IRC | 16:17 | |
| *** achanda has joined #openstack-infra | 16:17 | |
| clarkb | mrmartin: so I think the problem there is iterating bash arrays in order is hard | 16:18 |
| *** chandankumar has quit IRC | 16:18 | |
| *** cnesa has joined #openstack-infra | 16:18 | |
| clarkb | mrmartin: so we don't get consistent install orders and if we do them in some orders they fail and if we do them in other orders they pass | 16:18 |
| mrmartin | yep, it is a by design feature of bash associative arrays, so if we can predict some order, I guess it can work | 16:19 |
| sdague | bash arrays don't have an order | 16:19 |
| *** marun has joined #openstack-infra | 16:19 | |
| clarkb | sdague: right | 16:19 |
| mrmartin | the ultimate solution can be something that deploy modules in a proper dependency order. but a manual workaround could be total ok for me | 16:20 |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (subunit_worker) https://review.openstack.org/140383 | 16:20 |
| *** david-lyle has quit IRC | 16:20 | |
| clarkb | sdague: you can do things like use `sort` iirc so its not all pain | 16:20 |
| mrmartin | so maybe, adding an separate array that holds the keys in proper order is not nice, but can work. | 16:20 |
| *** armax has joined #openstack-infra | 16:20 | |
| mrmartin | clarkb: but we don't want to sort by name, we need an order by dependency | 16:21 |
| clarkb | mrmartin: correct | 16:21 |
| clarkb | mrmartin: so I think we need an extra dimension of order here | 16:21 |
| clarkb | jesusaurus: and/or yolanda were looking at it at one time | 16:21 |
| *** devlaps has quit IRC | 16:21 | |
| sdague | so my not just build MODULES_TO_INSTALL | 16:21 |
| sdague | as a lit | 16:21 |
| sdague | list | 16:21 |
| sdague | and iterate over that instead | 16:22 |
| sdague | use the others for attribute storage only | 16:22 |
| *** groknix has quit IRC | 16:22 | |
| *** lttrl has quit IRC | 16:22 | |
| yolanda | ah clarkb, i filed a change once but then the way of install in upstream was very different so i abandoned it | 16:22 |
| yolanda | we keep that change downstream for our use | 16:22 |
| mrmartin | yep, I write a patch for that, but need a careful review, I don't want to broke the entire module installation :) | 16:23 |
| clarkb | mrmartin: thankfully it is somewhat self testing via the check/gate tests | 16:23 |
| *** waynr has quit IRC | 16:23 | |
| *** devlaps has joined #openstack-infra | 16:23 | |
| *** tnovacik has joined #openstack-infra | 16:25 | |
| *** yamamoto has joined #openstack-infra | 16:26 | |
| ashp | Does infra have puppet modules for running pypi stashed anywhere, wasn't sure if you were using some other named pypi clone or something (shows what I know about pypi right there, I imagine) | 16:27 |
| *** changbl has joined #openstack-infra | 16:28 | |
| *** SumitNaiksatam has quit IRC | 16:28 | |
| fungi | ashp: what do you mean "running pypi"? | 16:29 |
| fungi | ashp: a mirror? | 16:29 |
| ashp | it looks like we're going to be building and pushing our own stuff into a pypi server, so I guess we need something capable of doing that, not just a mirror | 16:29 |
| *** talluri has quit IRC | 16:30 | |
| *** enikanorov has quit IRC | 16:30 | |
| *** dannywilson has joined #openstack-infra | 16:30 | |
| *** enikanorov has joined #openstack-infra | 16:30 | |
| fungi | ashp: ahh, no we don't do anything like that. we push packages into pypi.python.org and then mirror that in our infrastructure for performance reasons | 16:30 |
| *** ryanpetrello has joined #openstack-infra | 16:30 | |
| ashp | you have the nice benefit of not having private packages! That's cool, I'll do this the hard way, was just hoping to be lazy :) | 16:31 |
| fungi | ashp: you may want https://pypi.python.org/pypi/devpi (i think it's supposed to support adding custom packages) | 16:31 |
| *** yamamoto has quit IRC | 16:31 | |
| *** arxcruz has quit IRC | 16:32 | |
| ashp | fungi: awesome, I'll check that, there's like 500 clones of pypi it seems | 16:32 |
| *** shakamunyi has quit IRC | 16:33 | |
| fungi | ashp: it's the most common/popular tool in the python packaging community for managing custom pypi repos (partial mirrors, modified mirrors, et cetera), from what i've seen | 16:33 |
| *** talluri has joined #openstack-infra | 16:33 | |
| *** k4n0 has quit IRC | 16:34 | |
| ashp | That's where I'll start then, thanks! | 16:34 |
| *** esker has quit IRC | 16:34 | |
| *** sweston1 has joined #openstack-infra | 16:35 | |
| ashp | I sure wish I was writing this automation in ansible, not chef | 16:35 |
| *** gokrokve has quit IRC | 16:36 | |
| fungi | ashp: yeah, configuration management systems are not orchestration systems. it's taken years for that to really sink in for me | 16:36 |
| ashp | I did our jenkins build slaves in ansible as it's "easier" to just pip install ansible than to try and get chef on build slaves | 16:37 |
| ashp | and it was just really pleasant | 16:37 |
| openstackgerrit | Douglas Mendizábal proposed openstack-infra/project-config: Add docs to python-barbicanclient gates https://review.openstack.org/139259 | 16:37 |
| ashp | I love that you can register the results of actions then do things based on them, that kind of stuff is nightmarish in puppet | 16:37 |
| fungi | ashp: tell me about it. that was really the first thing to really push us to start using ansible for some stuff... coordinating cross-server dependent tasks | 16:38 |
| ashp | I just sent my boss https://github.com/ccollicutt/swiftacular | 16:38 |
| *** esker has joined #openstack-infra | 16:39 | |
| ashp | we were talking about firing up a small cluster with ansible using upstream packages for tinkering and bam | 16:39 |
| fungi | ashp: and now we actually use ansible to apply our puppet manifests on all our long-lived servers | 16:39 |
| clarkb | I am checking git.o.o connectivity to internal bcakends now | 16:39 |
| clarkb | will probably reload git.o.o haproxy in two passes so that we have some overlap with public and intenral IPs rather than a straight cutover | 16:39 |
| *** AJaeger has quit IRC | 16:39 | |
| fungi | clarkb: sounds good | 16:39 |
| fungi | clarkb: as for growing this further, i wonder whether adding a second haproxy node and doing round-robin address records in dns between them would help | 16:40 |
| clarkb | fungi: it may if the bandwidth on the haproxy public ip is the problem | 16:40 |
| *** waynr has joined #openstack-infra | 16:40 | |
| fungi | it would be a much coarser persistence, but probably enough | 16:40 |
| fungi | also keep the ttl low on those records so the resolution order shifts more often on leaf resolvers | 16:42 |
| *** ryanpetrello has quit IRC | 16:42 | |
| clarkb | rax lets us do 5min ttls which should be plenty | 16:42 |
| clarkb | though in the past with similar set ups we always did 30s | 16:42 |
| fungi | 5min is fine for this case i think. just not 1 day or something | 16:43 |
| clarkb | ya | 16:43 |
| clarkb | I think we are real close to having nodepool figured out though. jeblair work from yesterday sounded promising | 16:44 |
| clarkb | and once we have that I/we can push really hard on dib | 16:44 |
| *** ayoung has quit IRC | 16:44 | |
| *** otter768 has joined #openstack-infra | 16:44 | |
| *** ayoung has joined #openstack-infra | 16:44 | |
| fungi | now that i've been playing with dnssec a lot more (specifically for publishing sshfp records securely) i'm getting pretty annoyed by the fact that rackspace's recursors don't dnssec (as we've already observed months ago) | 16:44 |
| pleia2 | good morning | 16:45 |
| fungi | not only am i running my own authoritative servers now, but i'm going to resort to running recursive servers too it looks like | 16:46 |
| *** markus_z has quit IRC | 16:46 | |
| fungi | oh, also my isp breaks dnssec, and so does dd-wrt's built-in dnsmasq version. so much fail everywhere | 16:46 |
| *** Mithrandir has quit IRC | 16:46 | |
| *** krtaylor has quit IRC | 16:46 | |
| clarkb | fungi: :/ I gave up on ddwrt when I moved so that I could have ipv6 | 16:47 |
| clarkb | pleia2: morning | 16:47 |
| *** gyee has joined #openstack-infra | 16:47 | |
| fungi | the ugliness in getting dnssec and ipv6 going at home is going to drive me back to running an openbsd firewall on some low-power commodity something or other | 16:47 |
| *** shakamunyi has joined #openstack-infra | 16:47 | |
| pleia2 | heh | 16:48 |
| *** otter768 has quit IRC | 16:49 | |
| fungi | i had a great, seamless, manageable setup for years on openbsd with ipv6, dnssec, openvpn, ipsec, accelerating proxies, you name it | 16:49 |
| fungi | thought once i moved all my servers out of my basement i could switch to a much simpler firewall platform, but signs point to no | 16:49 |
| *** jlibosva has quit IRC | 16:50 | |
| *** Mithrandir has joined #openstack-infra | 16:51 | |
| jeblair | clarkb: what do you think our bandwidth limit is? | 16:51 |
| fungi | 800mbps right? | 16:51 |
| clarkb | jeblair: te published limit is 800mbps | 16:51 |
| fungi | tied to the instance flavor | 16:51 |
| jeblair | i'm trying to look that up and i'm having trouble because the rax names are all changing | 16:51 |
| clarkb | jeblair: the flavor gives us 1600mbps total half of which you can use on the public interface | 16:52 |
| *** AJaeger has joined #openstack-infra | 16:52 | |
| *** AJaeger has quit IRC | 16:52 | |
| *** AJaeger has joined #openstack-infra | 16:52 | |
| jeblair | there's now "standard" and "general purpose" in their docs | 16:52 |
| *** SumitNaiksatam has joined #openstack-infra | 16:52 | |
| jeblair | but yeah, the cp says: Network | 16:52 |
| jeblair | 1.6 Gb / s | 16:52 |
| *** wenlock_ has joined #openstack-infra | 16:53 | |
| jeblair | ah, "general purpose" "replaces performance 1" | 16:53 |
| fungi | gotta love marketing numbers | 16:53 |
| jeblair | and io replaces performance 2 | 16:53 |
| jeblair | and then 'compute' and 'memory' are new | 16:54 |
| jeblair | "Maximum network bandwidth on General Purpose Cloud Servers is 10,000 Mbps to divide between public and private networks as you choose.." | 16:54 |
| jeblair | that does not seem to be the case | 16:54 |
| clarkb | we aren't quite there on the cacti graphs like we were in the past but its close enough that I am willing to attribute that to hand waving | 16:54 |
| clarkb | and I figure this is a low overhead change that we can attempt to see if it helps | 16:55 |
| *** yfried|afk is now known as yfried_ | 16:56 | |
| clarkb | I am read to switch to talking to git01 and git02 via the 10net addrs. jeblair anything else you want to look at before I do that? | 16:56 |
| clarkb | I have confirmed that git.o.o can talk to them via those addresses with all three protocols that we use | 16:56 |
| jeblair | ah, there's some hidden mouseover fine-print: | 16:57 |
| jeblair | (sorry, i have to type it because i can't copy/paste mouseover very easily) | 16:57 |
| *** sarob has joined #openstack-infra | 16:57 | |
| jeblair | "...maximum outbound public bandwidth is limited to 40% of the aggregrate..." | 16:58 |
| jeblair | (also, i can't see it while i type. grr) | 16:58 |
| jeblair | so i think that means our outbound limit is 640? | 16:58 |
| clarkb | oh I thought I read 50% but if its 40% then ya that puts us right up against the limit | 16:58 |
| clarkb | though if only outbound is measured my change is likely to have a small impact | 16:59 |
| jeblair | why? isn't it the outbound from git0X that's the problem? | 16:59 |
| clarkb | jeblair: no its outbound from git.o.o | 16:59 |
| clarkb | jeblair: the git0X servers peak around 150mbps each | 17:00 |
| clarkb | git.o.o hits the ~640mbps, but git.o.o is doing outbound to git0X too | 17:00 |
| clarkb | so this would be removing the git.o.o outbound to git0X which is a smaller chunk of the data | 17:00 |
| jeblair | clarkb: this will subtract the git0X inbound traffic from the git.o outbound public traffic | 17:01 |
| clarkb | correct | 17:01 |
| jeblair | that's about 2.5mbps | 17:01 |
| clarkb | ya its not quite worth it | 17:01 |
| jeblair | average | 17:01 |
| *** achanda has quit IRC | 17:01 | |
| jeblair | maybe 15mbps peak | 17:02 |
| jeblair | (if they all peak at once) | 17:02 |
| jeblair | (and they usually don't) | 17:02 |
| *** isviridov is now known as isviridov_away | 17:03 | |
| *** luqas__ has joined #openstack-infra | 17:03 | |
| *** luqas has quit IRC | 17:03 | |
| jeblair | we have 2 git.o.o's in nova | 17:03 |
| clarkb | gah thats my fault | 17:04 |
| clarkb | I did not delete the old one after I switched to the new one | 17:04 |
| *** luqas__ has quit IRC | 17:05 | |
| *** esker has quit IRC | 17:05 | |
| *** luqas__ has joined #openstack-infra | 17:05 | |
| jeblair | the load balancers claim to be able to push 9gb/sec | 17:05 |
| *** afazekas has quit IRC | 17:05 | |
| *** baoli has quit IRC | 17:05 | |
| jeblair | i wonder if we can use them yet? | 17:06 |
| clarkb | 8d14fa13-f69e-4890-a6f7-24594710a9a7 can be deleted I will do that momentarily | 17:06 |
| clarkb | jeblair: we would need the ability to set queue lengths on git:// tcp backends | 17:07 |
| clarkb | jeblair: at least that is why we didn't use their load balancers when we made the haproxy machine | 17:07 |
| jeblair | clarkb: why is that? | 17:07 |
| *** luqas__ has quit IRC | 17:07 | |
| *** luqas__ has joined #openstack-infra | 17:07 | |
| clarkb | jeblair: because otherwise you get no queuing and git daemon falls over | 17:07 |
| jeblair | ah, okay, to make it large, not small, got it :) | 17:08 |
| *** KanagarajM has quit IRC | 17:08 | |
| jeblair | clarkb: what problem are we seeing today? should i go read scrollback? | 17:08 |
| *** achanda has joined #openstack-infra | 17:08 | |
| clarkb | jeblair: during nodepool image builds jenkins jobs are unable to git reliably | 17:08 |
| * jeblair spins up a load balancer to poke at | 17:09 | |
| *** MaxV_ has joined #openstack-infra | 17:09 | |
| *** MaxV has quit IRC | 17:09 | |
| *** krtaylor has joined #openstack-infra | 17:09 | |
| *** luqas___ has joined #openstack-infra | 17:09 | |
| *** luqas__ has quit IRC | 17:09 | |
| jeblair | which is surprisingly slow | 17:10 |
| *** ala_ has quit IRC | 17:10 | |
| jeblair | clarkb: is this the same problem we saw before? i thought before git.o was unable to communicate with git0x | 17:11 |
| clarkb | jeblair: it was slightly different in the past. Previously OOMKiller would be invoked because we would have lots of processes serving data eating up all the memory | 17:11 |
| clarkb | jeblair: at the time I believed that this was partially due to bandwidth limitations. Basically the limits backed everything up killing connections and preventing new connections | 17:12 |
| *** esker has joined #openstack-infra | 17:12 | |
| clarkb | jeblair: this time around we did not see swapping or OOMKiller | 17:12 |
| *** luqas__ has joined #openstack-infra | 17:12 | |
| jeblair | so why is the bandwidth limit affecting the git client? is it doing tcp resets or something? | 17:12 |
| *** luqas___ has quit IRC | 17:13 | |
| openstackgerrit | garyk proposed openstack-infra/project-config: Add VMware-NSX to StackForge https://review.openstack.org/140409 | 17:13 |
| *** ryanpetrello has joined #openstack-infra | 17:13 | |
| clarkb | jeblair: I was not able to confirm that but I was able to confirm that the connections that failed never reached git.o.o so I thought something like that may be happening | 17:13 |
| clarkb | jeblair: so something in the middle was saying lolno and it coincided with when we were at or near bandwidth limitations | 17:14 |
| fungi | chances are they're not doing something graceful like red and are just dropping packets that don't fit in the buffer | 17:14 |
| *** luqas__ has quit IRC | 17:14 | |
| *** luqas__ has joined #openstack-infra | 17:14 | |
| zaro | morning | 17:14 |
| jeblair | fungi: and doing so often enough that tcp isn't recovering? | 17:14 |
| *** avozza is now known as zz_avozza | 17:15 | |
| clarkb | fungi: I think git specifically reported connection was reset by peer | 17:15 |
| jeblair | i don't see any options about queue depth; do we know anyone at rax working on lbaas? | 17:15 |
| clarkb | fungi: but haproxy never reported having a connection to that host | 17:15 |
| anteaya | morning zaro | 17:16 |
| fungi | jeblair: hrm, maybe. i guess it depends on by what factor we're exceeding the limits | 17:16 |
| *** luqas__ has quit IRC | 17:16 | |
| clarkb | in other news. I am checking the haproxy log on old git.o.o and IBM and Freescale will be very unhappy when I delete this node | 17:16 |
| clarkb | anteaya: who do we know at IBM and freescale? | 17:16 |
| *** luqas has joined #openstack-infra | 17:16 | |
| fungi | so, yeah, maybe they have a traffic shaper which rejects/disconnects select sessions rather than simply dropping some percentage of the packets | 17:16 |
| jeblair | fungi, clarkb: yeah, a tcp rst would do that i think. | 17:17 |
| *** esker has quit IRC | 17:17 | |
| jeblair | clarkb: sounds like "java dns" :) | 17:17 |
| anteaya | clarkb: trinaths and krtaylor | 17:17 |
| clarkb | anteaya: thank you | 17:17 |
| anteaya | thank you for asking | 17:17 |
| jeblair | so the 60gb i/o flavor (was performance2) has 5gb/s network | 17:17 |
| clarkb | krtaylor: I need to clean up an old load balancer node and IBM has connections to it still. Any chance we can work on getting those connections moved to what DNS currently points at? | 17:18 |
| clarkb | krtaylor: as jeblair says it may be something silly like java dns caching results forever | 17:18 |
| *** baoli has joined #openstack-infra | 17:18 | |
| *** luqas has quit IRC | 17:18 | |
| jeblair | clarkb: (standard instances have 50% of aggregate public network) | 17:19 |
| clarkb | jeblair: and performance are 40%? | 17:19 |
| *** luqas has joined #openstack-infra | 17:19 | |
| *** amuller has quit IRC | 17:19 | |
| jeblair | yep | 17:19 |
| clarkb | gotcha | 17:20 |
| clarkb | krtaylor: anyways when you see this ping me and we can work on getting this cleaned up more gracefully. I won't delete this on you quite yet :) | 17:20 |
| anteaya | clarkb: you can also try jungleboyj in -cinder, he works with IBM though not with a ci | 17:21 |
| anteaya | clarkb: might trim down the waiting time | 17:21 |
| clarkb | anteaya: thanks, I am not in a super rush and krtaylor tends to be around so I think I can wait a bit | 17:21 |
| *** mmaglana has joined #openstack-infra | 17:21 | |
| jeblair | clarkb, fungi: so if we want to ~double our network throughput (2.5gbps) by staying on a single server, we need to ~4x our ram to 30g (either compute or io flavors will do) | 17:21 |
| jeblair | clarkb: or i think you mentioned maybe round-robining? | 17:21 |
| *** _nadya_ has quit IRC | 17:21 | |
| clarkb | jeblair: that was fungi's idea. we could use a DNS round robin | 17:22 |
| jeblair | that might let us ~double bandwidth by ~doubling ram footprint | 17:22 |
| clarkb | and have 2 git haproxy servers | 17:22 |
| jeblair | fungi: oh sorry | 17:22 |
| anteaya | clarkb: sounds good | 17:22 |
| fungi | if it's a bad suggestion, i'll let clarkb take the credit ;) | 17:22 |
| clarkb | the one potential probelm with that is haproxy hashing not being consistent | 17:22 |
| clarkb | but haproxy may have ways to sync that sort of info between servers | 17:22 |
| clarkb | so that they use consistent hashing | 17:22 |
| jeblair | clarkb: it's not based on math? | 17:23 |
| *** esker has joined #openstack-infra | 17:23 | |
| jeblair | (as in, hash of ip address?) | 17:23 |
| fungi | it's based on math, just not math from _our_ universe | 17:23 |
| *** charz has quit IRC | 17:23 | |
| jeblair | also, if the image builds start at around the same time, they might end up with the same ip address anyway | 17:23 |
| fungi | but yes, i don't understand the words you strung together there clarkb. can you try new ones? | 17:23 |
| jeblair | but hapefully the overall load will be less even in that case | 17:23 |
| clarkb | jeblair: it is based on math but the math changes if one node detects a node down and the other doesnt | 17:24 |
| anteaya | fungi: do you have 5 minutes to review a devstack-gate patch to lay a foundation for ipv6 functional testing? https://review.openstack.org/#/c/140128/ | 17:24 |
| clarkb | which probably happens infrequently enough that we can just not worry about it | 17:24 |
| fungi | clarkb: oh, i get you. regardless i don't see why that would matter. your session is first balanced to a proxy based on the order your resolver got, then proxied to an end node based on a hash | 17:24 |
| jeblair | well, it could be an issue of the backends are out of sync, right? | 17:25 |
| jeblair | and you get bumped mid-operation | 17:25 |
| clarkb | fungi: right but if you are not using java you could resolve DNS twice getting different haproxies | 17:25 |
| clarkb | and potentially get different backends which if out of sync then you break | 17:25 |
| *** gokrokve has joined #openstack-infra | 17:25 | |
| clarkb | which seems unlikely enough that maybe I shouldn't worry about it until it happens | 17:25 |
| fungi | i guess i find myself wondering the degree to which we expect synchronization to be out of alignment, and even if it's within an order of magnitude or two of the ttl that's a problem anyway yeah? | 17:26 |
| jeblair | so that's only a problem if the backends are out of sync, and one of them is down, and you do a dns query mid-operation | 17:26 |
| clarkb | jeblair: ya :) | 17:26 |
| *** MaxV_ has quit IRC | 17:26 | |
| jeblair | (and if they are out of sync and one of them goes down, you still might have this problem even with one lb, since it might shift you) | 17:26 |
| *** MaxV has joined #openstack-infra | 17:27 | |
| fungi | my thoughts as well | 17:27 |
| *** eharney has quit IRC | 17:27 | |
| fungi | it seems like a corner case, and similar to corner cases we already can't avoid | 17:27 |
| *** MaxV has quit IRC | 17:27 | |
| fungi | and which we can solve better by using a synchronous backend for the git farm | 17:27 |
| *** MaxV has joined #openstack-infra | 17:27 | |
| jeblair | fungi: ooh, i've got one! :) | 17:28 |
| fungi | or is there a reason why afs read-only on the git servers would be async still? | 17:28 |
| clarkb | +1 | 17:28 |
| jeblair | fungi: no reason afaik | 17:28 |
| jeblair | clarkb, fungi: i think adding a new fe is worth trying; it should be fairly easy and completely non-disruptive | 17:28 |
| clarkb | jeblair: agreed. | 17:28 |
| fungi | that was one of my suggested use cases for afs deployment anyway, so maybe i just start poking at it | 17:28 |
| jeblair | easier than rebuilding onto a larger server at least | 17:28 |
| jeblair | fungi: cool | 17:29 |
| jeblair | clarkb: i also think your change to use private net should not hurt if we want to continue with that | 17:29 |
| clarkb | jeblair: it shouldn't hurt but I don't think it is valuable since you have to dance around puppet haproxy restarts | 17:29 |
| fungi | maybe first i go fix our out-of-sync uids/gids so we can merge that puppet patch. and then someone besides me might ought to review the awesome afs documentation jeblair wrote, which has been rotting in gerrit for some time now | 17:30 |
| *** MaxV has quit IRC | 17:30 | |
| jeblair | fungi: sounds like a plan | 17:30 |
| *** e0ne has quit IRC | 17:31 | |
| *** charz has joined #openstack-infra | 17:31 | |
| clarkb | which puppet patch? | 17:31 |
| *** eharney has joined #openstack-infra | 17:31 | |
| *** eharney has joined #openstack-infra | 17:31 | |
| jeblair | clarkb: the one that sets gids/uids for users | 17:32 |
| clarkb | I thought I did review that /me looks again | 17:32 |
| jeblair | clarkb: oh i think it's reviewed, we just forgot to merge it after fungi did the work to change things on filesystems | 17:32 |
| clarkb | fungi: your doc update changes have completely ruined my ability to serach through your open changes :) | 17:33 |
| fungi | so i am working on finding and updating the systems which need manual normalization before i approve it | 17:33 |
| fungi | clarkb: it's a jeblair change anyway | 17:34 |
| *** ChuckC has quit IRC | 17:34 | |
| jeblair | i used to write changes... so long ago... | 17:34 |
| fungi | the puppet uid mapping change and afs documentation change are both jeblair | 17:34 |
| *** tonytan4ever has quit IRC | 17:34 | |
| *** melwitt has joined #openstack-infra | 17:34 | |
| * clarkb reviews as gerrit says I have not done so yet | 17:35 | |
| *** ayoung is now known as ayoung-lunch | 17:35 | |
| *** wenlock_ has quit IRC | 17:36 | |
| *** wenlock_ has joined #openstack-infra | 17:37 | |
| *** vigneshvar has joined #openstack-infra | 17:37 | |
| *** Masahiro has joined #openstack-infra | 17:37 | |
| *** shashankhegde has joined #openstack-infra | 17:38 | |
| *** skolekonov has quit IRC | 17:38 | |
| clarkb | nibalizer's comment sort of got lost in https://review.openstack.org/#/c/129476/2/modules/user/manifests/virtual/localuser.pp but I think it should be addressed | 17:38 |
| *** armax has quit IRC | 17:38 | |
| clarkb | jeblair: if you just do $uid,\n$guid,\n that will make it so that those values are not options | 17:38 |
| clarkb | er not optional | 17:39 |
| clarkb | but otherwise it looks good. fungi hasalready give me my new ids on most servers | 17:39 |
| jeblair | clarkb: could you write a follow-up patch to do that? | 17:40 |
| fungi | yeah, i just dug up the script i used on the old servers. now just scanning everything ansible knows about to see which ones are nonconforming | 17:40 |
| jeblair | fungi: you can ignore jeblairtest* :) | 17:40 |
| jeblair | those are my nodepool test nodes | 17:40 |
| fungi | jeblair: can do | 17:40 |
| clarkb | jeblair: in a new change or you want me to update your change? | 17:40 |
| fungi | i was going to put together a list first anyway | 17:40 |
| clarkb | and yes I can write it | 17:40 |
| jeblair | clarkb: a new change; that way we don't add any more hurdles to merging this one | 17:40 |
| *** cnesa has quit IRC | 17:41 | |
| clarkb | jeblair: well I am not sure the = unset will even parse | 17:41 |
| clarkb | jeblair: unless you have tested it in a non noop fashion | 17:41 |
| *** AJaeger_ has joined #openstack-infra | 17:41 | |
| mrmartin | clarkb, fungi, I'm writing this puppet module order script, but something is hell strange to me: puppetlabs-puppetdb => 3.0.1 and it depends exactly on a puppetlabs-stdlb 2.2.0, but we are defining puppetlabs-stdlib => 4.3.2, so puppetlabs-puppetdb also fails to deploy. how was that working before? | 17:41 |
| jeblair | clarkb: oh, well if you think we should update it, then cool. | 17:41 |
| jeblair | clarkb: i thought we had tests that did that | 17:41 |
| clarkb | jeblair: they run under noop mode which catches a lot of stuff but not everything | 17:41 |
| *** jgallard_ has joined #openstack-infra | 17:41 | |
| jeblair | clarkb: it was so long ago, i can't recall what testing i may have done :( | 17:42 |
| *** nikil22_ has joined #openstack-infra | 17:42 | |
| *** Masahiro has quit IRC | 17:42 | |
| fungi | mrmartin: i think nibalizer figured that out and implemented a workaround | 17:42 |
| *** stevemar has joined #openstack-infra | 17:42 | |
| *** jgallard has quit IRC | 17:43 | |
| mrmartin | nibalizer: are you here? | 17:43 |
| *** armax has joined #openstack-infra | 17:43 | |
| jeblair | i need to add a feature to gertty to toggle hiding comments (for those reviews where there are 300 inline comments already) | 17:43 |
| clarkb | jeblair: I will use a logstash-worker as a test machine really quickly. if the current patchset works I will just write a follow up change | 17:44 |
| jeblair | clarkb: thanks | 17:44 |
| mrmartin | and it also fail with puppetlabs postgresql 'puppetlabs-puppetdb' (v3.0.1) requires 'puppetlabs-postgresql' (>= 3.1.0 <4.0.0), and we are using exactly 4.0.0 from pgsql | 17:44 |
| *** [HeOS] has quit IRC | 17:45 | |
| *** jgallard_ has quit IRC | 17:46 | |
| *** shashankhegde has quit IRC | 17:46 | |
| *** cnesa has joined #openstack-infra | 17:46 | |
| *** esker has quit IRC | 17:46 | |
| *** esker has joined #openstack-infra | 17:47 | |
| *** ChuckC has joined #openstack-infra | 17:47 | |
| mrmartin | other reviews also failing: https://review.openstack.org/#/c/136597/ it is from Monty, same stdlib issue | 17:47 |
| nikil22_ | is ubuntu-trusty supported for CI system or at present only ubuntu-precise is supported | 17:47 |
| *** tonytan4ever has joined #openstack-infra | 17:48 | |
| fungi | nikil22_: we use both | 17:48 |
| *** hdd has joined #openstack-infra | 17:48 | |
| *** pc_m has quit IRC | 17:48 | |
| fungi | nikil22_: depending on what part of the ci system you're talking about | 17:48 |
| *** pc_m has joined #openstack-infra | 17:48 | |
| clarkb | jeblair: ok that works. writing follow up change now | 17:49 |
| reed | mrmartin, I'm assuming that using Trusty instead would make things even more complex, right? | 17:49 |
| *** markmcclain has quit IRC | 17:49 | |
| nikil22_ | fungi: I am trying to setup the similar CI system of openstack, not third party. just need to replicate the same setup. And i see in docs "OpenStack-CI puppetmaster due to new Ruby and older Puppet not being compatible, so be sure to use an older release - e.g. Ubuntu Precise." | 17:50 |
| mrmartin | reed, I don't want to use Trusty in that specific case, because askbot runs well on precise, and want to decrease the potential failure points, we have enough without Trusty. But I guess this specific problem is not precise specific, it is a module dependency problem. | 17:50 |
| fungi | nikil22_: right, that's not specific to our ci but our project infrastructure in general | 17:50 |
| openstackgerrit | Merged openstack-infra/project-config: Add Networking-L2GW project https://review.openstack.org/140369 | 17:50 |
| reed | thanks mrmartin | 17:50 |
| *** sarob_ has joined #openstack-infra | 17:50 | |
| mrmartin | I suggest to move to new server with precise, and if it works well for a while, upgrade to Trusty | 17:50 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas https://review.openstack.org/140164 | 17:50 |
| *** yfried_ is now known as yfried|afk | 17:50 | |
| clarkb | jeblair: I think what = unset does is try to set them to uid of 'unset' and gid of 'unset' which works in our case because we override it everywhere we use it | 17:51 |
| clarkb | jeblair: so its fine for us but for completeness we should have the follow up change to make them properly required params | 17:51 |
| fungi | nikil22_: right now we're running our puppetmaster on trusty, but other long-lived systems on precise and centos 6 depending on their purpose, and in the process of testing moving nodepool onto trusty. we build workers for a variety of distros/releases though including precise and trusty but lots of others as well | 17:51 |
| nikil22_ | fungi: so if i tried to use the same project infra and use those scripts to set up the similar CI system i need to use ubuntu-precise ? | 17:52 |
| *** ryanpetrello has quit IRC | 17:52 | |
| nikil22_ | fungi: ok so trust also supports thanks. | 17:52 |
| fungi | nikil22_: yes, i would say deploy your puppetmaster (if you're using one) on trusty but put zuul, nodepool and jenkins on precise | 17:52 |
| openstackgerrit | Merged openstack-infra/project-config: Remove tripleo control variables https://review.openstack.org/124334 | 17:52 |
| *** gokrokve has quit IRC | 17:52 | |
| *** sarob__ has joined #openstack-infra | 17:53 | |
| *** sarob_ has quit IRC | 17:53 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Make localuser gid and uid required params https://review.openstack.org/140415 | 17:53 |
| fungi | nikil22_: unless you're adventurous and want to help us test those on trusty/submit patches to fix the bugs you find | 17:53 |
| clarkb | I am going to test 140415 in the same way right now | 17:53 |
| *** sarob__ has quit IRC | 17:53 | |
| *** gokrokve has joined #openstack-infra | 17:53 | |
| *** ryanpetrello has joined #openstack-infra | 17:54 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas https://review.openstack.org/140164 | 17:54 |
| nikil22_ | fungi: i can't use different systems . i am trying to use in same machine and deploy puppetmaster,zuul,jenkins etc . Let me try in trusty itself and see whats happening | 17:54 |
| *** cnesa has quit IRC | 17:54 | |
| clarkb | and that looks good as well /me updates reviews | 17:54 |
| clarkb | note we already know that nodepool has trouble of some sort on trusty | 17:55 |
| clarkb | with scheduling threads. jeblair was looking into this yesterday | 17:55 |
| clarkb | but you ma not have trouble if not operating at quite th same scale or on a differnet cloud etc | 17:56 |
| openstackgerrit | Merged openstack-infra/project-config: Use a job-template for the tripleo jobs https://review.openstack.org/124335 | 17:56 |
| clarkb | fungi: jeblair: oh heads up. jhesketh got swift uploads working as the only log uploads in the experimental job used to test that. logstash indexed the console.html too \o/ | 17:57 |
| clarkb | so I think jhesketh is going to propose we switch to swift only for infra jobs in todays meeting | 17:57 |
| fungi | i saw--stellar! | 17:57 |
| nikil22_ | so fungi: clarkb if we use precis for puppet master and for jenkins,nodepool etc it should be ok ? | 17:57 |
| *** cnesa has joined #openstack-infra | 17:57 | |
| clarkb | nikil22_: I think so. precise has ruby 1.9 which new puppet will want | 17:58 |
| clarkb | nikil22_: but give trusty a go too | 17:58 |
| *** koolhead17 has quit IRC | 17:58 | |
| *** david-lyle_afk is now known as david-lyle | 17:58 | |
| nikil22_ | thanks i will try | 17:58 |
| *** koolhead17 has joined #openstack-infra | 17:59 | |
| *** packet has joined #openstack-infra | 17:59 | |
| *** luqas has quit IRC | 18:00 | |
| armax | hi there, whenever someone has a spare second, https://storyboard.openstack.org/#!/story/2000072 many thanks! | 18:00 |
| clarkb | fungi: I think jeblair's change is ready to merge as soon as you are happy with the less puppety side of the uid/gid move | 18:00 |
| anteaya | armax is asking to be added to a new gerrit group | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| *** sarob has quit IRC | 18:01 | |
| jeblair | krotscheck: i just followed a link to https://storyboard.openstack.org/#!/story/2000072 and got no data | 18:01 |
| jeblair | krotscheck: doing it again works | 18:01 |
| zaro | hashar: are you subscribed to the python-jenkins changes? | 18:01 |
| jeblair | krotscheck: is there any debugging/log capturing we should do? | 18:01 |
| clarkb | armax: that groups hasn't been created yet. Can you point at the change that adds it? | 18:01 |
| hashar | zaro: yeah all of them | 18:01 |
| armax | clarkb: the change just merged, I would imagine there’s a bit of a lag? https://review.openstack.org/#/c/140369/ | 18:02 |
| clarkb | armax: that explains it. Things should happen over the next 10 minutes or so | 18:02 |
| hashar | zaro: I noticed some patches from last thursday, I am a bit busy though :/ | 18:02 |
| zaro | hashar: ok, i might just approval all those single +1 reviews if someone doesn't come along and keep me company :) | 18:02 |
| armax | clarkb: gotcha, thanks, it makes sense…sorry if I jumped the gun :) | 18:02 |
| clarkb | armax: no problem just wanting to make sure I hae the context | 18:03 |
| fungi | clarkb: is nodepool-dev not puppeting? | 18:03 |
| clarkb | fungi: it is not | 18:03 |
| fungi | clarkb: looks like it's not in the hostfile | 18:03 |
| *** Sukhdev has quit IRC | 18:03 | |
| *** koolhead17 has quit IRC | 18:03 | |
| clarkb | that was somewhat intentional as we keep flipping back and forth and installing weird gear and so on | 18:03 |
| *** jp_at_hp has quit IRC | 18:04 | |
| fungi | clarkb: jeblair: any other servers i should check which aren't added to the ansible hostfile on puppetmaster yet? | 18:04 |
| *** Ryan_Lane has joined #openstack-infra | 18:04 | |
| *** armax has quit IRC | 18:04 | |
| clarkb | fungi: you should probably update nodepool-dev as we may keep that host as is and make it the new host. But other than that host I can't think of anything | 18:04 |
| *** cnesa has quit IRC | 18:05 | |
| fungi | clarkb: right, that's why i was looking for it | 18:05 |
| *** dtantsur is now known as dtantsur|afk | 18:05 | |
| zaro | hashar: would like to get your quick opinion on this one though, https://review.openstack.org/#/c/122462/ | 18:05 |
| *** ivar-lazzaro has joined #openstack-infra | 18:05 | |
| fungi | clarkb: jeblair: so it appears that the only stragglers i need to normalize are nodepool-dev.openstack.org, git.openstack.org and subunit-worker01.openstack.org | 18:06 |
| clarkb | that roughly coincides with my list of new servers so sounds good | 18:07 |
| clarkb | I thought we had more than one subunit worker for some reason but I think just one is correct | 18:07 |
| mtreinish | clarkb: nah, it's just the 1, we started 4 daemons on it | 18:07 |
| *** koolhead17 has joined #openstack-infra | 18:08 | |
| mtreinish | but it looks like 1 is handling the load better than 4 did | 18:08 |
| mtreinish | so I'm not sure we'll ever need to grow it out | 18:08 |
| *** jamielennox|away is now known as jamielennox | 18:09 | |
| clarkb | cool | 18:11 |
| *** bdpayne has quit IRC | 18:11 | |
| *** rushiagr is now known as rushiagr_away | 18:12 | |
| *** shashankhegde has joined #openstack-infra | 18:12 | |
| *** dustins has joined #openstack-infra | 18:12 | |
| *** dustins_ has quit IRC | 18:13 | |
| *** zz_sabari is now known as sabari | 18:14 | |
| *** ddieterly has quit IRC | 18:15 | |
| hashar | zaro: awaited for dinner sorry :-/ | 18:15 |
| *** jamielennox is now known as jamielennox|away | 18:16 | |
| jogo | anteaya: ping https://review.openstack.org/#/c/139167/ | 18:17 |
| hashar | zaro: that is all about https://bugs.launchpad.net/python-jenkins/+bug/1177831 not sure why it would need a POST via the rest api :( | 18:17 |
| uvirtbot | Launchpad bug 1177831 in python-jenkins "build_job with parameters fails since POST is not used." [Undecided,In progress] | 18:17 |
| jogo | anteaya: where do you recommend I make the change? I agree with your -1 just not sure where to put it | 18:17 |
| hashar | zaro: might be some plugin | 18:17 |
| clarkb | btw that networking-l2gw change merged one non core non release group | 18:17 |
| hashar | zaro: anyway sorry, dinner + sleep time :-/ | 18:17 |
| *** hashar has quit IRC | 18:17 | |
| clarkb | I have added the initial member as requested but we should consider renaming that group and creating a second group for tags | 18:17 |
| clarkb | anteaya: AJaeger ^ | 18:17 |
| *** amuller has joined #openstack-infra | 18:18 | |
| *** ddieterly has joined #openstack-infra | 18:18 | |
| anteaya | sorry did I approve a repo with an acl that gave tagging to core? | 18:18 |
| fungi | and it doesn't look like we've had anything create new users or groups on any servers in the >2000 range besides those listed in the puppet change (wanted to make absolutely sure for the sake of sanity) | 18:19 |
| clarkb | anteaya: sort of, they didn't even have a core group just 'networking-l2gw' | 18:19 |
| anteaya | jogo: I don't know, sorry I wish I did | 18:19 |
| jogo | fungi: in the change to making 3rd party CI self service I think we lost the 'don't email everyone' change we made to the old 3rd party group | 18:19 |
| clarkb | fungi: sanity is good | 18:19 |
| *** mwagner_lap has quit IRC | 18:19 | |
| anteaya | jogo: I don't know enough about devstack-gate to say where would be better | 18:19 |
| fungi | though pleia2 and ttx are in the list of users in puppet and don't seem to have shell accounts in any of our servers | 18:20 |
| jogo | anteaya: can I make a job template with a specific topology? | 18:20 |
| fungi | jogo: that shouldn't have changed at all | 18:20 |
| clarkb | jogo: ya there is no overarching third party ci group any longer | 18:20 |
| fungi | jogo: no idea what "don't send email" change you think we made | 18:20 |
| clarkb | jogo: but we requie they all name themselves Something Something CI so you can filter on that | 18:20 |
| anteaya | clarkb: I will ask armax his intentions, he got kicked off the network but is down the hall | 18:20 |
| jogo | fungi: let me look what it was | 18:20 |
| clarkb | fungi: there was an acl that said deny email group third-party ci | 18:21 |
| anteaya | jogo: that might work | 18:21 |
| pleia2 | fungi: I was added in anticipation of helping with pootle and zanata testing, but in the end clarkb did pootle and I just tossed zanata demo on one of my hpcloud instances to avoid waiting longer | 18:21 |
| jogo | fungi: https://review.openstack.org/#/c/107589/ | 18:21 |
| fungi | clarkb: oh! i didn't even know that was possible (or i've forgotten) | 18:21 |
| clarkb | fungi: but since third-party ci is no longer a thing it isn't something we can take advantage of | 18:21 |
| pleia2 | (still am using my own hpcloud account for zanata puppet config testing) | 18:21 |
| fungi | clarkb: can we add all the per-project groups as included groups in that hidden group? | 18:21 |
| clarkb | fungi: that only covers voting CI | 18:22 |
| clarkb | so we could add a non voting CI group but I really don't think that is benefiial | 18:22 |
| fungi | oh, right | 18:22 |
| clarkb | espeically since as soon as they become voting they can email again | 18:22 |
| fungi | maybe an exim filter? | 18:22 |
| fungi | that seems gross though | 18:22 |
| jeblair | i do not know why this would have stopped since there is still a third-party ci group | 18:23 |
| clarkb | ya we could do something like that on the server itself. Personally I don't grok why filtering at the client side is always such a fight | 18:23 |
| *** patrickeast has joined #openstack-infra | 18:23 | |
| clarkb | jeblair: the acls for those groups were removed as prt of the initial switch | 18:23 |
| jogo | jeblair: I can give an example of where it didn't work | 18:23 |
| clarkb | jeblair: so that we can remove members of those groups and put them in "inactive" state | 18:23 |
| jeblair | https://review.openstack.org/gitweb?p=All-Projects.git;a=commitdiff;h=ff53035c8f4e8a42bb563a1226df6a4183d11b20 | 18:23 |
| jogo | "VMware NSX CI" on https://review.openstack.org/123394 | 18:23 |
| jeblair | ah there it is | 18:23 |
| jeblair | jogo: i believe you :) | 18:24 |
| jogo | I got an email about that bot. | 18:24 |
| *** hdd has quit IRC | 18:24 | |
| jogo | jeblair: I wasn't sure if I believed myself | 18:24 |
| *** radez is now known as radez_g0n3 | 18:24 | |
| *** jlibosva has joined #openstack-infra | 18:24 | |
| clarkb | we could add a non voting group back in and sub group them to the central group just for that rule | 18:25 |
| AJaeger_ | clarkb: oops, sorry | 18:25 |
| *** armax has joined #openstack-infra | 18:26 | |
| AJaeger_ | anteaya: do you want to fix it or shall I? | 18:27 |
| jeblair | we could make that group owned by a group that includes all the groups that manage the voting groups, and people can just stick accounts in there as it comes up. | 18:27 |
| jeblair | or we could make an exim filter :) | 18:27 |
| anteaya | talking with armax stand by | 18:27 |
| clarkb | jeblair: thats another good suggestion (the first one) | 18:28 |
| clarkb | but I find that this sort of thing is honestly best dealt with client side | 18:28 |
| clarkb | because filtering isn't going to meet everyones needs if we do it one way centrally | 18:28 |
| * AJaeger_ is fixing the network-L2GW now... | 18:29 | |
| clarkb | AJaeger_: hold on, we should rename that group in gerrit so that we don't have an orphaned group | 18:29 |
| AJaeger_ | anteaya: I'm standing by waiting for your comments, thanks | 18:29 |
| clarkb | AJaeger_: so I think propose the cahnge but WIP it | 18:29 |
| AJaeger_ | clarkb: Ok, will do | 18:29 |
| anteaya | AJaeger_: armax will offer the patch | 18:30 |
| *** jlibosva has quit IRC | 18:30 | |
| *** esker has quit IRC | 18:30 | |
| jogo | clarkb: have a quick jjb question https://review.openstack.org/#/c/139167/1 | 18:30 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix ACL groups for networking L2GW https://review.openstack.org/140426 | 18:30 |
| *** harlowja_away is now known as harlowja | 18:30 | |
| *** habib has quit IRC | 18:30 | |
| AJaeger_ | anteaya: Shall I abandon the above? | 18:30 |
| jogo | can I do '{pipeline}-tempest-dsvm-{topology}{branch-designator}' | 18:30 |
| AJaeger_ | anteaya: I'll wait for armax' and then abandon mine | 18:31 |
| jogo | can I do '{pipeline}-tempest-dsvm-aiopcpu{branch-designator}' for aiopcpu specific things? | 18:31 |
| clarkb | jogo: yes | 18:31 |
| jogo | clarkb: would that inherit from '{pipeline}-tempest-dsvm-{topology}{branch-designator}' | 18:31 |
| *** dims has quit IRC | 18:31 | |
| clarkb | jogo: no | 18:31 |
| armax | AJaeger_: if you have it up then there’s no need to wait for mine | 18:31 |
| jogo | clarkb: thanks | 18:31 |
| clarkb | there is no inheritance | 18:31 |
| armax | AJaeger_: I was about to push it though | 18:31 |
| anteaya | AJaeger_: thanks | 18:31 |
| *** mrunge has quit IRC | 18:31 | |
| jogo | clarkb: didn't think so, thanks | 18:32 |
| anteaya | AJaeger_ armax you work it out | 18:32 |
| jeblair | clarkb, fungi: http://paste.openstack.org/show/148067/ | 18:32 |
| fungi | jogo: instead you could have a conditional block on $topology | 18:32 |
| AJaeger_ | armax: please review https://review.openstack.org/140426 | 18:32 |
| jeblair | clarkb, fungi: that script takes 3 seconds on our precise nodepool config, and 1 minute 12 seconds on trusty | 18:32 |
| *** jlibosva has joined #openstack-infra | 18:32 | |
| clarkb | jeblair: woah | 18:32 |
| jogo | fungi: ohh, even better I see | 18:32 |
| fungi | wow! | 18:32 |
| *** talluri has quit IRC | 18:32 | |
| jeblair | clarkb, fungi: it exhibits the same behavior where lots of threads are apparently releasing a lock at the same time that i saw in my artificial test yesterday | 18:33 |
| jeblair | clarkb, fungi: i do not know if that is the _cause_ of our actual problems, or if it _shares the cause_ | 18:33 |
| fungi | jogo: alternatively, two job templates reusing a common builder macro, but that still likely results in more duplication than embedding a conditional in the job's main builder (or in a script it's calling for that matter) | 18:33 |
| *** sabari is now known as zz_sabari | 18:33 | |
| clarkb | fungi: jogo: honestly I would just rename those jobs to be aiopcpu specific | 18:33 |
| clarkb | since right now they are specific and not worry about passing topology around as a var | 18:34 |
| jeblair | in production, we only saw a handful of threads in those exact functions when we did the thread dump | 18:34 |
| *** achanda has quit IRC | 18:34 | |
| anteaya | clarkb: ++ | 18:34 |
| jeblair | nonetheless, i think it may be related and it seems similar enough to where we thought the problem might be that it's worth pursuing | 18:34 |
| clarkb | jeblair: +1 | 18:35 |
| clarkb | jeblair: maybe try it on a trusty non pvhvm node too | 18:35 |
| *** r-daneel has joined #openstack-infra | 18:35 | |
| clarkb | jeblair: or is your precise node pvhvm? | 18:35 |
| jeblair | clarkb: precise is non pvhm | 18:35 |
| jogo | clarkb: makes sense, like this: http://paste.openstack.org/show/148068 | 18:36 |
| clarkb | I think pvhvm makes a difference with kernel things so might be worth trying if it is easy to do so | 18:36 |
| nibalizer | mrmartin: hrm yea that makes sense | 18:36 |
| nibalizer | kindof | 18:36 |
| *** kumartin has joined #openstack-infra | 18:36 | |
| fungi | clarkb: jeblair: yeah, i'm curious if the underlying cause can be further isolated | 18:36 |
| nibalizer | let me look | 18:36 |
| clarkb | jogo: ya, then you have to update the places where you sued the old name | 18:36 |
| mrmartin | nibalizer: I made a patch for puppet module order | 18:36 |
| anteaya | jogo: I agree with clarkb's suggestions and your paste thus far | 18:37 |
| *** denis_makogon has quit IRC | 18:37 | |
| AJaeger_ | clarkb, anteaya: regarding https://review.openstack.org/140426, should we rename the existing group as my patch does or leave it and add just a -release? I thought we would rename... | 18:37 |
| mrmartin | but puppetlabs-puppetdb seems to be not satisfying the dependency requirements | 18:37 |
| clarkb | AJaeger_: we should rename so that we have a -core and a -release | 18:37 |
| anteaya | jogo: I'll be looking for edtis to zuul/layout.yaml | 18:37 |
| clarkb | AJaeger_: then we will stop puppet, do the rename in gerrit manually, then start puppet to apply the acl | 18:37 |
| anteaya | clarkb: you don't care we will have an orphaned group in gerrit? | 18:38 |
| *** baoli has quit IRC | 18:38 | |
| nibalizer | mrmartin: well i dont really want to fix it with order hacks | 18:38 |
| clarkb | anteaya: ^ prevents an orphaned group | 18:38 |
| nibalizer | because thats how you get broken | 18:38 |
| AJaeger_ | anteaya: clarkb will do some magic ;) | 18:38 |
| *** esker has joined #openstack-infra | 18:38 | |
| anteaya | clarkb: okay let's let clarkb do his magic | 18:38 |
| clarkb | anteaya: we can rename groups we just have to make sure that manage-projects doesn't create the group for us before we can rename to it | 18:38 |
| jeblair | clarkb, fungi: i can reproduce on precise with merely an upgrade of pycrypto from 2.4.1 to 2.6.1 | 18:38 |
| armax | just for my sanity…if having different teams recommended or necessary? | 18:38 |
| mrmartin | nibalizer: and what you suggest as a solution? | 18:38 |
| anteaya | clarkb: so you like AJaeger_'s patch as is? | 18:38 |
| clarkb | jeblair: wow | 18:38 |
| clarkb | jeblair: so at least that narrows it down a bit | 18:38 |
| anteaya | armax: yes a smaller group for tagging | 18:38 |
| *** Sukhdev has joined #openstack-infra | 18:39 | |
| armax | would it make sense to add a check on the tests? I could volunteer to write the test | 18:39 |
| armax | :) | 18:39 |
| clarkb | anteaya: ya, | 18:39 |
| *** shakamunyi has quit IRC | 18:39 | |
| anteaya | armax: let's have your test | 18:39 |
| clarkb | anteaya: but we shouldn't merge that until one of the gerrit admins can do the rename and juggle puppet things | 18:39 |
| anteaya | clarkb: okay I'll change my vote | 18:39 |
| nibalizer | mrmartin: i think we should bump puppetdb to 4.1.0 | 18:39 |
| nibalizer | but I haven't tested if that works yet | 18:39 |
| anteaya | clarkb: I'll +2 you can +A | 18:39 |
| nibalizer | let me do some poking | 18:39 |
| anteaya | I'll make a note on the patch | 18:39 |
| *** marun has quit IRC | 18:39 | |
| clarkb | I won't commit to that today since I already have a bunch of stuff going on but likely can do it tomorrow | 18:39 |
| AJaeger_ | clarkb: shall I leave it as WIP, or do you want to add a WIP yourself and then remove it whenever you have time? | 18:40 |
| clarkb | AJaeger_: so if you leave it as WIP for now maybe we can coordinate tomorrow morning to get that in | 18:40 |
| mrmartin | nibalizer, ok, give me a sec, I test with 4.1.0 | 18:40 |
| armax | sorry if I caused unnecessary churn! | 18:40 |
| *** denis_makogon has joined #openstack-infra | 18:40 | |
| fungi | jeblair: is that coming into play for the socket security then? | 18:40 |
| AJaeger_ | clarkb: I would prefer if you WIP it, less coordination ;) | 18:40 |
| clarkb | AJaeger_: ok let me wip it | 18:40 |
| AJaeger_ | thanks, clarkb ! | 18:40 |
| anteaya | armax: you didnt, this is my mistake | 18:40 |
| *** cpowell has quit IRC | 18:40 | |
| clarkb | AJaeger_: done you can remove your -W now | 18:41 |
| jeblair | fungi: don't understand your question | 18:41 |
| *** yfried|afk is now known as yfried_ | 18:41 | |
| nibalizer | mrmartin: okay yea report back on that | 18:41 |
| AJaeger_ | clarkb: done | 18:41 |
| fungi | jeblair: wondering why pycrypto is impacting thread lock behavior | 18:41 |
| AJaeger_ | armax: I didn't catch it in review - my bad ;( | 18:41 |
| armax | AJaeger_: my understanding was that it wasn’t mandatory and I didn’t want t cause the extra complexity | 18:42 |
| anteaya | AJaeger_: we both missed it | 18:42 |
| fungi | jeblair: wondering if it's underlying ssl/tls layer implementation and getting hooked when polling sockets | 18:42 |
| zaro | fungi: gerrit not linking stories? https://review.openstack.org/#/c/140120/1//COMMIT_MSG | 18:42 |
| armax | AJaeger_: but if it is, then I am more than happy to add a test so that we catch these types of issues | 18:42 |
| fungi | zaro: not yet anyway | 18:42 |
| *** e0ne has joined #openstack-infra | 18:43 | |
| clarkb | fungi: jeblair: for when fungi is done with the uid and gid stuff and we get the puppet change merged. I can spin up the second haproxy host. Call it something like git-frontend02.openstack.org? | 18:43 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Make aiopcpu not use config drive https://review.openstack.org/139167 | 18:43 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Stop using topology variable https://review.openstack.org/140429 | 18:43 |
| zaro | fungi: looks to be defined in system-config gerrit.pp file why doesn't it work? | 18:43 |
| AJaeger_ | armax: http://docs.openstack.org/infra/manual/creators.html#add-gerrit-permissions has a "should". | 18:43 |
| AJaeger_ | armax: do we need to make it clearer? | 18:44 |
| fungi | zaro: no clue | 18:44 |
| *** baoli has joined #openstack-infra | 18:44 | |
| armax | AJaeger_: should isn’t not a must :) | 18:44 |
| fungi | clarkb: yep, the normalization is done, i'm just pulling up the change to approve now | 18:44 |
| armax | AJaeger_: my bad | 18:44 |
| * clarkb steps out before meeting back in time for that though | 18:44 | |
| jeblair | fungi: well, this test program doesn't do anything with sockets, and neither does our use of it in nodepool (via paramiko) | 18:45 |
| *** wznoinsk has joined #openstack-infra | 18:45 | |
| armax | AJaeger_: I’ll propose a check to verify this condition on the acl files | 18:45 |
| *** ZZelle_ has joined #openstack-infra | 18:45 | |
| fungi | jeblair: oh, right. just looked back at the test script | 18:45 |
| *** otter768 has joined #openstack-infra | 18:45 | |
| fungi | hadn't looked at it closely as i was distracted working on other tasks | 18:46 |
| dougwig | hi infra, can we get some reviews for https://review.openstack.org/#/c/140164/ ? just passed jenkins, needed by neutron split. | 18:46 |
| armax | to make sure we’ll noticed in the future, unless you think it’s easier to be more prescriptive on the docs | 18:46 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Add puppet-recheckwatch as split out module https://review.openstack.org/140430 | 18:46 |
| dougwig | (devstack-gate patch) | 18:46 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/system-config: Split out recheckwatch module https://review.openstack.org/140431 | 18:46 |
| asselin | nibalizer, do you have a few minutes to talk about puppet module split? | 18:47 |
| mrmartin | nibalizer: so, this is the result: http://paste.openstack.org/show/148078/ need to upgrade puppetlabs-firewall to at least 1.1.3 and puppetlabs-inifile to 1.1.3 also | 18:47 |
| anteaya | armax: this was my and AJaeger_'s doing, we didn't catch it in review | 18:47 |
| fungi | jeblair: so guessing pycrypto changed its rsa key generation algorithm in the later release | 18:48 |
| zaro | fungi: ahh i see the problem. capital 'S' won't work. | 18:48 |
| *** _shaps_ has quit IRC | 18:48 | |
| *** marun has joined #openstack-infra | 18:48 | |
| armax | anteaya: well, I feel mostly responsible, I should have paid more attention to the docs | 18:49 |
| fungi | jeblair: maybe using some sort of shared (blocking?) entropy source it previously didn't? | 18:49 |
| *** jistr has quit IRC | 18:49 | |
| jeblair | fungi: the internal entropy source is definitely shared and blocking (it holds a threading.lock()), though the external source it reads from is urandom and should not block | 18:50 |
| openstackgerrit | Khai Do proposed openstack-infra/storyboard-webclient: Added millisecond delay to reduce the number of queries. https://review.openstack.org/140120 | 18:50 |
| *** baoli has quit IRC | 18:50 | |
| *** otter768 has quit IRC | 18:50 | |
| jeblair | fungi, clarkb: the behavior is present in 2.5 too, so pycryto 2.4.1 -> 2.5 is the change | 18:50 |
| anteaya | armax: it's okay, we have a plan to fix, we're good | 18:51 |
| fungi | was about to ask if you'd bisected to a specific version. looking at the changelog now | 18:51 |
| armax | anteaya: indeed, I’ll work on the test to help us catch this in the future | 18:51 |
| fungi | jeblair: several items related to rsa key generation in the 2.6 release notes | 18:52 |
| fungi | jeblair: er, 2.5 | 18:52 |
| jeblair | fungi: only 64 commits to bisect :) | 18:52 |
| jeblair | but i'm going to start by reading | 18:53 |
| anteaya | armax: thank you :D | 18:53 |
| *** pelix has quit IRC | 18:53 | |
| fungi | can you name that tune in 6 bisects? | 18:53 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/storyboard: Converted application to use utcnow() https://review.openstack.org/140148 | 18:53 |
| *** esker has quit IRC | 18:53 | |
| mordred | fungi: are we supposing that the nodepool issue may have actually been caused by a pycrypto algorithm change? | 18:54 |
| jhesketh | Morning | 18:54 |
| pleia2 | g'day jhesketh | 18:55 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/136504 | 18:55 |
| mordred | jhesketh: where are you? | 18:55 |
| *** cpowell has joined #openstack-infra | 18:55 | |
| fungi | mordred: or concurrency-related changes around entropy gathering... something | 18:56 |
| jhesketh | mordred: home (Aust) | 18:56 |
| anteaya | morning jhesketh | 18:56 |
| anteaya | asselin: your patch set is looking good, nice job on the commit messages | 18:57 |
| *** markmcclain has joined #openstack-infra | 18:57 | |
| anteaya | asselin: why are you offering a patch and the upstream repo is on sweston's account? | 18:57 |
| fungi | jeblair: i'm ready to approve 129476 if you want to lift your -2 | 18:57 |
| anteaya | asselin: that has the potential for unnecessary need for co-ordination for me | 18:57 |
| asselin | anteaya, thanks. Using yet-to-be-merge spec updates by mmedvede | 18:57 |
| jeblair | fungi: done | 18:58 |
| fungi | thanks! | 18:58 |
| anteaya | asselin: in case the upstream repo needs to be re-generated | 18:58 |
| asselin | anteaya, we're trying to simplify the process. sweston's repo is keeping everything up-to-date, so anyone can use it | 18:58 |
| anteaya | asselin: well they work, so good job | 18:58 |
| anteaya | asselin: do you have push rights to his repo? | 18:58 |
| asselin | anteaya, no need to | 18:58 |
| *** yapeng is now known as yapeng_ | 18:58 | |
| anteaya | asselin: please expand | 18:58 |
| *** jdandrea has quit IRC | 18:59 | |
| anteaya | because based on the number of times dougwig had to re-generate repos yesterday, I disagree | 18:59 |
| *** rushiagr_away is now known as rushiagr | 18:59 | |
| *** andreykurilin_ has joined #openstack-infra | 18:59 | |
| asselin | anteaya, https://review.openstack.org/#/c/137991/ | 19:00 |
| asselin | anteaya, is dougwig using sweston's repos? | 19:00 |
| pleia2 | meeting time :) | 19:00 |
| *** bdpayne has joined #openstack-infra | 19:00 | |
| anteaya | what is special about sweston's repos? | 19:00 |
| asselin | anteaya, we discussed this last infra meeting (or two) | 19:00 |
| anteaya | refresh my memory | 19:01 |
| asselin | anteaya, his script is keeping the subtreed repos continously up to date | 19:01 |
| anteaya | so it is constantly updateing the repos? | 19:01 |
| anteaya | what is the trigger? | 19:01 |
| asselin | correct | 19:01 |
| asselin | anteaya, we can ask sweston | 19:02 |
| jeblair | asselin: i'm pining you in #openstack-meeting :) | 19:02 |
| *** radez_g0n3 is now known as radez | 19:03 | |
| openstackgerrit | Merged openstack-infra/system-config: Manage UID/GID with puppet https://review.openstack.org/129476 | 19:04 |
| nibalizer | asselin: i will at utc 21:00 | 19:05 |
| *** Ryan_Lane has quit IRC | 19:06 | |
| *** bdpayne has quit IRC | 19:07 | |
| *** ivar-lazzaro has quit IRC | 19:08 | |
| *** Ryan_Lane has joined #openstack-infra | 19:08 | |
| *** armax has quit IRC | 19:09 | |
| *** ivar-lazzaro has joined #openstack-infra | 19:10 | |
| *** zz_sabari is now known as sabari | 19:10 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: make gerrit commentlink case insensitive for bug and story https://review.openstack.org/140439 | 19:12 |
| *** nfedotov has joined #openstack-infra | 19:13 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Fix puppet module dependency order https://review.openstack.org/140440 | 19:13 |
| mrmartin | nibalizer, fungi: the solution for puppet module dependency problem ^^^ | 19:14 |
| *** mwagner_lap has joined #openstack-infra | 19:14 | |
| fungi | mrmartin: thanks! i'll take a look after the infra weekly meeting we're in right now | 19:15 |
| *** baoli has joined #openstack-infra | 19:15 | |
| nibalizer | mrmartin: so that doesn't work | 19:15 |
| nibalizer | because infra doesn't rm -fr before running install_modules.sh | 19:15 |
| nibalizer | so you cant do the requirements hack of ordering to fix it | 19:15 |
| *** koolhead17 has quit IRC | 19:15 | |
| nibalizer | we have to just actually fix it | 19:15 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Make aiopcpu jobs run tempest-full https://review.openstack.org/140445 | 19:15 |
| mrmartin | nibalizer: oh thats a point | 19:15 |
| *** viglesias has quit IRC | 19:16 | |
| *** viglesias has joined #openstack-infra | 19:16 | |
| *** bradjones has quit IRC | 19:16 | |
| mrmartin | so it means I can get a different result if I run this on an existing node or a new one started from scratch | 19:17 |
| nibalizer | ya | 19:18 |
| nibalizer | exactly | 19:18 |
| nibalizer | the existing infra modules directory provides kindof a base, and any changes we want to make happen need to work as diffs on that | 19:18 |
| nibalizer | mrmartin: so did bumping puppetdb not fix it? | 19:19 |
| mrmartin | nibalizer: if you want to upgrade puppetdb, you need to upgrade both the firewall and inifile modules. | 19:19 |
| nibalizer | sounds good | 19:19 |
| nibalizer | i think the firewall module is completely unused? and inifile is fine to version bump | 19:20 |
| *** Longgeek has quit IRC | 19:20 | |
| nibalizer | cus its inifile | 19:20 |
| *** sweston1 has quit IRC | 19:20 | |
| mrmartin | nibalizer: my original problem was this failure: http://logs.openstack.org/43/140043/4/check/gate-infra-puppet-apply-precise/84ae501/console.html | 19:21 |
| *** hdd has joined #openstack-infra | 19:21 | |
| *** viglesias has joined #openstack-infra | 19:21 | |
| *** jlibosva has quit IRC | 19:22 | |
| mrmartin | and the first problem was that: Error: Could not install module 'puppetlabs-stdlib' (v4.3.2), and it happens only when some other module already deployed a 4.4.0 | 19:22 |
| nibalizer | mrmartin: im trying to focus on the meeting | 19:22 |
| clarkb | nibalizer: I don't follow why the rm matters | 19:22 |
| nibalizer | lets talk about this after that | 19:22 |
| mrmartin | km | 19:22 |
| *** ihrachyshka has quit IRC | 19:22 | |
| mrmartin | ok. | 19:22 |
| clarkb | nibalizer: we upgrade and upgrades should happen in the same order as specified for installs | 19:22 |
| *** yfried_ is now known as yfried|afk | 19:23 | |
| *** amuller has quit IRC | 19:25 | |
| *** Masahiro has joined #openstack-infra | 19:26 | |
| *** bradjones has joined #openstack-infra | 19:29 | |
| *** yamamoto has joined #openstack-infra | 19:29 | |
| *** teran_ has quit IRC | 19:29 | |
| *** sabari is now known as zz_sabari | 19:30 | |
| *** zz_sabari is now known as sabari | 19:30 | |
| *** Masahiro has quit IRC | 19:30 | |
| *** rushiagr is now known as rushiagr_away | 19:31 | |
| *** zul has quit IRC | 19:31 | |
| *** melwitt has quit IRC | 19:33 | |
| *** yamamoto has quit IRC | 19:34 | |
| *** zul has joined #openstack-infra | 19:34 | |
| *** dprince has quit IRC | 19:34 | |
| *** zul has quit IRC | 19:35 | |
| *** zul has joined #openstack-infra | 19:37 | |
| mestery | As we continue our neutron services split work, would be great to get some infra love on this patch: https://review.openstack.org/#/c/140164 | 19:41 |
| *** MaxV has joined #openstack-infra | 19:41 | |
| mestery | That one temporarily disables services tests in devstack-gate, which lets us merge the neutron patch removing them. | 19:41 |
| *** kumartin has quit IRC | 19:41 | |
| fungi | mestery: i'll have a look once the infra weekly meeting finishes' | 19:42 |
| mestery | fungi: Thank you sir! | 19:42 |
| *** eharney has quit IRC | 19:43 | |
| *** shayneburgess has joined #openstack-infra | 19:44 | |
| *** cnesa has joined #openstack-infra | 19:44 | |
| *** dizquierdo has quit IRC | 19:45 | |
| *** [HeOS] has joined #openstack-infra | 19:45 | |
| *** _nadya_ has joined #openstack-infra | 19:50 | |
| *** rushiagr_away is now known as rushiagr | 19:52 | |
| *** mmaglana_ has joined #openstack-infra | 19:53 | |
| *** mmaglana has quit IRC | 19:53 | |
| *** changbl has quit IRC | 19:53 | |
| *** cnesa2 has joined #openstack-infra | 19:55 | |
| *** cnesa has quit IRC | 19:55 | |
| *** baoli has quit IRC | 19:56 | |
| ttx | fungi: I probably still have access to the odsreg host (previously known as summit.o.o) | 19:58 |
| fungi | ttx: yep, but it's not puppeted i don't think | 19:58 |
| ttx | no it's not | 19:58 |
| *** changbl has joined #openstack-infra | 20:00 | |
| *** ayoung-lunch is now known as ayoung | 20:00 | |
| fungi | ttx: so anyway, once it _is_ puppeted, your inclusion in the list of users in our puppet manifest will serve a purpose ;) | 20:01 |
| jedimike | hi, two super quick things from me before i sign off for the day | 20:01 |
| pleia2 | btw, I'm heading out in about an hour for an afternoon meeting, but will be back later this evening | 20:01 |
| jedimike | I have a spec that clarkb had +1'd but wanted jeblair to look at before any +2's went on it - https://review.openstack.org/129255 , and a patch that has 2 +2's - https://review.openstack.org/136281 | 20:01 |
| mrmartin | ttx: what you want to puppetize? | 20:01 |
| *** CIBuilds has joined #openstack-infra | 20:02 | |
| *** melwitt has joined #openstack-infra | 20:03 | |
| *** wenlock_ has quit IRC | 20:04 | |
| *** melwitt1 has joined #openstack-infra | 20:05 | |
| *** vigneshvar has quit IRC | 20:06 | |
| *** gyee has quit IRC | 20:06 | |
| *** cpowell_ has joined #openstack-infra | 20:06 | |
| *** changbl has quit IRC | 20:07 | |
| krotscheck | fungi: So, about storyboard-dev.openstack.org. Should I just start with a system-config patch? | 20:07 |
| *** melwitt has quit IRC | 20:08 | |
| mmedvede | anteaya: your concern regarding the sweston 's upstream for module split is valid. I think the auto-update script is supposed to be running all the time, but s human still would need to check how valid the new module upstream is before landing module split patches. | 20:08 |
| *** melwitt1 is now known as melwitt | 20:08 | |
| mmedvede | asselin, sweston: is the module split script is running now? I do not see recent changes in sweston 's repository | 20:09 |
| anteaya | mmedvede: also we need to define what running all the time means | 20:09 |
| anteaya | as if the repo is in the middle of an update while we try to get it as seed | 20:09 |
| anteaya | there is a potential for interference | 20:09 |
| *** cpowell has quit IRC | 20:10 | |
| fungi | krotscheck: yep | 20:10 |
| mmedvede | anteaya: yes, there is always that | 20:10 |
| fungi | krotscheck: add the server in the global site manifest and we'll go over the storyboard module to make sure it's parameterized appropriately to support having a dev instance | 20:10 |
| anteaya | so if we know what the trigger is, we can not trigger during the patch merge process | 20:10 |
| *** Rockyg has joined #openstack-infra | 20:11 | |
| mmedvede | anteaya: I might have misunderstood what you mean by the trigger. Trigger for updating the seed? | 20:12 |
| anteaya | yes | 20:12 |
| anteaya | do we have a trigger? or is it a cron job? | 20:12 |
| anteaya | what does "running all the time" mean | 20:12 |
| anteaya | I don't know | 20:12 |
| fungi | krotscheck: once the patch is in shape (or before if i get time) i'll create the additional database and hiera keys. also i'll need your help generating a sample dataset | 20:13 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Fix gerrit mimetype references https://review.openstack.org/140464 | 20:13 |
| mmedvede | anteaya: as I understand (could be wrong), the script was running as fast as it could to create the split modules. Of course, if there are no changes to the module, nothing would get pushed | 20:14 |
| anteaya | so it is triggered on a merged change to certain watched dirs and files? | 20:14 |
| fungi | krotscheck: or we can just start with a blank canvas and let people sandbox whatever examples they want into it, since that's what the data will probably evolve into anyway over time | 20:14 |
| mmedvede | anteaya: that we need to ask sweston or asselin | 20:15 |
| *** MaxV has quit IRC | 20:15 | |
| anteaya | if so, that would fix the concern about triggering a respin during the patch merges | 20:15 |
| krotscheck | fungi: Yeah, I’m torn on that. A good seed of data would be nice I think. | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| krotscheck | fungi: But after that it doesn’t really matter that much. | 20:15 |
| anteaya | mmedvede: sounds good | 20:15 |
| *** changbl has joined #openstack-infra | 20:16 | |
| fungi | krotscheck: up to you. if you want a seed db though it'll need a volunteer to create it (i'm happy to do the final step of sourcing it into mysql once it's ready) | 20:16 |
| clarkb | ok Third-Party CI is back to normalish and owned by Third-Party Coordinators. Third-Party Coordinators is also back to normal and I put krtaylor and asselin in it | 20:17 |
| clarkb | looking at attendance for the last meetings mmedvede and sweston should probably go in there too but I don't know their email addrs | 20:18 |
| *** vigneshvar has joined #openstack-infra | 20:18 | |
| clarkb | and finally jogo: would you like to be added since yu didn't like the emails you were getting? | 20:18 |
| openstackgerrit | Weidong Shao proposed openstack-infra/project-config: Add compass-install project to stackforge https://review.openstack.org/140465 | 20:18 |
| mestery | fungi: Apologies for nagging, any chance you can look at this one now that the infra meeting is done? https://review.openstack.org/#/c/140164 | 20:19 |
| *** aysyd has quit IRC | 20:19 | |
| clarkb | mestery: I thought mtreinish was removing the tests from tempest? | 20:20 |
| *** Longgeek has joined #openstack-infra | 20:20 | |
| clarkb | maybe I misunderstood yesterday's discussion but I thought the d-g was no longer necessary | 20:20 |
| mestery | clarkb: Long term that's the goal, short term we just need to get neutron running again :) | 20:20 |
| clarkb | mestery: gotcha | 20:20 |
| *** andreykurilin_ has quit IRC | 20:20 | |
| fungi | mestery: yep, was just looking back over the review history for it to double-check i;m not missing any significant objections | 20:21 |
| *** markmcclain1 has joined #openstack-infra | 20:21 | |
| clarkb | I will review it now too | 20:21 |
| *** salv-orl_ has joined #openstack-infra | 20:21 | |
| mestery | awesome, thanks guys! | 20:21 |
| mmedvede | clarkb: my email is mihailmed@gmail.com (that is the one I use for mailing lists) | 20:21 |
| *** eharney has joined #openstack-infra | 20:21 | |
| *** salv-orlando has quit IRC | 20:22 | |
| *** markmcclain has quit IRC | 20:22 | |
| clarkb | mmedvede: thanks you are added | 20:22 |
| mmedvede | clarkb: sweet | 20:22 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added manifest for storyboard-dev.openstack.org https://review.openstack.org/140466 | 20:22 |
| *** shashankhegde has quit IRC | 20:23 | |
| mestery | fungi: Thanks sir! clarkb, now it's your turn ;) | 20:23 |
| *** MaxV has joined #openstack-infra | 20:23 | |
| *** nikil22_ has quit IRC | 20:23 | |
| mtreinish | clarkb: yeah, we want to migrate the tests out of tempest, but before that we want to make sure the transition doesn't break | 20:23 |
| *** eharney has quit IRC | 20:25 | |
| *** yolanda has quit IRC | 20:25 | |
| *** eharney has joined #openstack-infra | 20:25 | |
| dougwig | fungi, clarkb, mtreinish - thank you! | 20:25 |
| *** Longgeek has quit IRC | 20:26 | |
| mtreinish | clarkb: so the short term is skip, unskip. Then delete | 20:26 |
| clarkb | thanks approved | 20:26 |
| clarkb | and with that time for lunch | 20:27 |
| *** rushiagr is now known as rushiagr_away | 20:27 | |
| clarkb | I will still try to boot up the new git server today if no one beats me to it | 20:27 |
| clarkb | but feel free if you have spare cycles | 20:27 |
| mestery | thanks clarkb and fungi! | 20:27 |
| mestery | mtreinish too | 20:27 |
| krotscheck | fungi: ^^ | 20:28 |
| jogo | clarkb: works for me | 20:28 |
| *** yolanda has joined #openstack-infra | 20:29 | |
| clarkb | jogo: done | 20:29 |
| krotscheck | ARGH TABS | 20:29 |
| * krotscheck nukes his IDE from orbit. | 20:29 | |
| *** eharney has quit IRC | 20:29 | |
| jogo | clarkb: thanks | 20:30 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added manifest for storyboard-dev.openstack.org https://review.openstack.org/140466 | 20:30 |
| *** eharney has joined #openstack-infra | 20:30 | |
| *** eharney has quit IRC | 20:30 | |
| *** eharney has joined #openstack-infra | 20:30 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added manifest for storyboard-dev.openstack.org https://review.openstack.org/140466 | 20:30 |
| openstackgerrit | Merged openstack-infra/system-config: Fix groups-dev ssl certificate defaults https://review.openstack.org/138012 | 20:30 |
| *** salv-orl_ has quit IRC | 20:31 | |
| *** salv-orlando has joined #openstack-infra | 20:31 | |
| *** esker has joined #openstack-infra | 20:32 | |
| *** pc_m has quit IRC | 20:33 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Configure dev draft client to use storyboard-dev. https://review.openstack.org/140469 | 20:34 |
| krotscheck | And there’s part 2 | 20:34 |
| *** pc_m has joined #openstack-infra | 20:34 | |
| krotscheck | fungi: So, those two patches will only work on precise, as trusty will require landing https://review.openstack.org/#/c/132105/ first. | 20:35 |
| *** pc_m_ has joined #openstack-infra | 20:35 | |
| *** mmaglana has joined #openstack-infra | 20:36 | |
| fungi | krotscheck: thanks--adding to my priority list | 20:36 |
| krotscheck | fungi: Thanks. lmk when you’re ready to build that box. | 20:37 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add test for validating certain gerrit ACL rules https://review.openstack.org/140470 | 20:37 |
| *** mmaglana has quit IRC | 20:38 | |
| *** armax has joined #openstack-infra | 20:38 | |
| armax | AJaeger_: a first stub at checking ACL’s https://review.openstack.org/#/c/140470/, comments welcome | 20:38 |
| *** mmaglana_ has quit IRC | 20:38 | |
| *** Sukhdev has quit IRC | 20:38 | |
| AJaeger_ | armax: thanks, will check soon | 20:39 |
| *** pc_m has quit IRC | 20:39 | |
| armax | AJaeger_: thank you! | 20:39 |
| krtaylor | clarkb, re Third-Party Coordinators, what are we coordinating, is this about the email deny list for third-party ci? | 20:45 |
| clarkb | krtaylor: ya | 20:46 |
| krtaylor | clarkb, so it is adding ci systems to the deny list, once fungi sets that up? | 20:46 |
| clarkb | the deny is in place | 20:46 |
| *** otter768 has joined #openstack-infra | 20:46 | |
| clarkb | and yes this is basically there so that we can deny emails | 20:46 |
| krtaylor | right, needed for sure | 20:46 |
| *** teran has joined #openstack-infra | 20:46 | |
| clarkb | krtaylor: two groups involved have their descriptions set. Let me know if you think they weren't descriptive enough and/or you can update them now | 20:47 |
| *** shashankhegde has joined #openstack-infra | 20:48 | |
| *** melwitt has quit IRC | 20:48 | |
| krtaylor | mmedvede, ^^^ | 20:48 |
| *** yfried|afk is now known as yfried_ | 20:49 | |
| openstackgerrit | Weidong Shao proposed openstack-infra/project-config: Add compass-install project to stackforge https://review.openstack.org/140465 | 20:51 |
| krtaylor | clarkb, I have term overload, by groups, you mean CI teams? or gerrit admin groups? | 20:51 |
| *** otter768 has quit IRC | 20:51 | |
| *** melwitt has joined #openstack-infra | 20:51 | |
| clarkb | krtaylor: 'Third-Party CI' and 'Third-Party Coordinators' gerrit groups | 20:51 |
| *** dims has joined #openstack-infra | 20:51 | |
| krtaylor | ok, gotcha | 20:51 |
| ttx | mrmartin: nothing. | 20:52 |
| *** bradjones has quit IRC | 20:52 | |
| *** rkukura_ has joined #openstack-infra | 20:53 | |
| *** rkukura has quit IRC | 20:53 | |
| *** rkukura_ is now known as rkukura | 20:53 | |
| *** obondarev has quit IRC | 20:54 | |
| *** alex7376 is now known as asettle | 20:54 | |
| *** JayJ has joined #openstack-infra | 20:54 | |
| openstackgerrit | Sukhdev Kapur proposed openstack-infra/project-config: Add networking_arisa project to StackForge https://review.openstack.org/140473 | 20:54 |
| *** obondarev has joined #openstack-infra | 20:55 | |
| *** mrmartin has quit IRC | 20:56 | |
| anteaya | okay the room has taken a turn for the worse here with marun finding names like leather | 20:56 |
| anteaya | mikal your video didn't help | 20:56 |
| *** Sukhdev has joined #openstack-infra | 20:56 | |
| mikal | anteaya: I was seeking to explain my problem with using Lynx | 20:57 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add test for validating certain gerrit ACL rules https://review.openstack.org/140470 | 20:57 |
| *** dprince has joined #openstack-infra | 20:57 | |
| anteaya | I understand your problem | 20:57 |
| anteaya | mikal: I get the explaination | 20:57 |
| *** subscope has joined #openstack-infra | 20:57 | |
| fungi | also, pretty certain lynx would go nowhere in the trademark search | 20:57 |
| anteaya | yay | 20:58 |
| anteaya | guess we are piling back onto love then | 20:58 |
| *** AJaeger_ has quit IRC | 20:58 | |
| clarkb | lanz! | 20:59 |
| fungi | no can haz lanz | 20:59 |
| *** jamielennox|away is now known as jamielennox | 20:59 | |
| clarkb | but it has a 'z' | 21:00 |
| clarkb | therefore it should win | 21:00 |
| *** _nadya_ has quit IRC | 21:00 | |
| *** jamespage_ has joined #openstack-infra | 21:00 | |
| anteaya | zeds win | 21:00 |
| *** armax has left #openstack-infra | 21:00 | |
| openstackgerrit | Merged openstack-infra/puppet-storyboard: Disabled rabbit repo management. https://review.openstack.org/132105 | 21:00 |
| *** changbl has quit IRC | 21:01 | |
| *** kgiusti has quit IRC | 21:01 | |
| *** rlandy has quit IRC | 21:01 | |
| fungi | especially on a triple-word score | 21:01 |
| anteaya | and qs | 21:02 |
| *** btran has joined #openstack-infra | 21:02 | |
| *** teran_ has joined #openstack-infra | 21:02 | |
| *** dims has quit IRC | 21:03 | |
| *** changbl has joined #openstack-infra | 21:03 | |
| *** dims has joined #openstack-infra | 21:03 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Temporarily disable testing lbaas, fwaas, and vpnaas https://review.openstack.org/140164 | 21:04 |
| *** harlowja is now known as harlowja_away | 21:04 | |
| btran | can't add "add-apt-repository cloud-archive:juno" on my ubuntu 12.04 server. Can't find solution online, anyone has any idea how I can add this repo? | 21:04 |
| fungi | btran: this isn't likely a good channel to ask, but i don't think ubuntu has a juno cloud archive for 12.04, only 14.04 | 21:05 |
| *** eharney has quit IRC | 21:05 | |
| *** teran has quit IRC | 21:05 | |
| btran | fungi: ok thanks | 21:06 |
| fungi | btran: anyway, this is the channel where we run the project community infrastructure, not where we discuss building an infrastructure for/with openstack | 21:06 |
| *** eharney has joined #openstack-infra | 21:06 | |
| fungi | krotscheck: okay, i've cleaned up storyboard.o.o's repository list. this is going to downgrade rabbitmq-server from 3.3.5-1 to 2.7.1-0ubuntu4 you're okay with that? anything i need to stop first or restart after i'm done? | 21:07 |
| *** markmcclain1 has quit IRC | 21:07 | |
| *** dims has quit IRC | 21:08 | |
| krotscheck | fungi: Hrm... | 21:08 |
| krotscheck | leeeet me verify that. | 21:08 |
| jeblair | clarkb, fungi: going to back to pycrypto thing | 21:09 |
| *** dmsimard is now known as dmsimard_away | 21:09 | |
| * fungi nods | 21:10 | |
| mattoliverau | Morning | 21:10 |
| *** nfedotov has quit IRC | 21:10 | |
| zaro | morning mattoliverau | 21:11 |
| anteaya | morning mattoliverau | 21:11 |
| *** jamespage_ has quit IRC | 21:12 | |
| *** harlowja_away is now known as harlowja | 21:12 | |
| grue_pm | morning <-: | 21:12 |
| *** baoli has joined #openstack-infra | 21:13 | |
| *** markmcclain has joined #openstack-infra | 21:13 | |
| anteaya | morning grue_pm | 21:13 |
| krotscheck | fungi: Is there the option to upgrade to Trusty? (checking my options here) | 21:14 |
| *** Masahiro has joined #openstack-infra | 21:15 | |
| fungi | krotscheck: we could... rebuild storyboard.o.o on trusty | 21:15 |
| fungi | krotscheck: in theory the only relevant state is in trove, so nothing to copy over | 21:15 |
| krotscheck | fungi: Ok, just checking. | 21:15 |
| krotscheck | fungi: It might not be necessary. | 21:16 |
| krotscheck | fungi: Honestly, a simpler answer may be to allow the module to do the repo management based on what it’s installed on. | 21:16 |
| *** melwitt has quit IRC | 21:16 | |
| *** MaxV has quit IRC | 21:16 | |
| * krotscheck is still downloading a precise image. | 21:16 | |
| *** MaxV has joined #openstack-infra | 21:17 | |
| openstackgerrit | Merged openstack-infra/system-config: Add puppet-httpd to modules.env https://review.openstack.org/136962 | 21:17 |
| *** julim has quit IRC | 21:18 | |
| nibalizer | asselin: im around if you want to chat | 21:19 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Update git frontend regex to support more nodes https://review.openstack.org/140480 | 21:19 |
| nibalizer | about split out | 21:19 |
| asselin | nibalizer, was just about to ping you | 21:19 |
| clarkb | fungi: jeblair jhesketh ^ that is a prerequisite to building a new git.o.o haproxy frontend | 21:19 |
| nibalizer | did mrmartin leave? | 21:19 |
| clarkb | nibalizer: if you get a chance can you double check m puppet regex there | 21:19 |
| *** Masahiro has quit IRC | 21:19 | |
| nibalizer | clarkb: sure | 21:19 |
| asselin | nibalizer, ok, so I was just doing another module split, and I'm thinking about proposing a single patch that does most of the project-config work | 21:19 |
| jeblair | clarkb, fungi: oh, it's not the version of pycrypto, it's that we're switching from os-installed to pip installed: | 21:20 |
| jeblair | clarkb, fungi: warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath. | 21:20 |
| asselin | nibalizer, basically everything except the change to gerrit/projects.yaml | 21:20 |
| clarkb | jeblair: ooohhh | 21:20 |
| nibalizer | clarkb: might i suggest just calling the node def /git/ ? | 21:20 |
| clarkb | nibalizer: no we have gitXX.o.o as well | 21:21 |
| clarkb | nibalizer: that are different | 21:21 |
| nibalizer | ah | 21:21 |
| *** nfedotov has joined #openstack-infra | 21:21 | |
| asselin | nibalizer, this way all those small changes are done up front. as far as I know, there shouldn't be any conflicts | 21:21 |
| nibalizer | for every module you mean? | 21:21 |
| *** Longgeek has joined #openstack-infra | 21:22 | |
| jeblair | clarkb: and we can rotate git.o.o out after adding fe01 and fe02 | 21:22 |
| fungi | jeblair: fast math sounds like it would be, more fast | 21:22 |
| asselin | nibalizer, yes, do all of them in one patch, for everything in project-config except the gerrit/projects.yaml change | 21:22 |
| jeblair | clarkb: can we call them git-fe01? :) | 21:22 |
| *** ociuhandu has quit IRC | 21:22 | |
| asselin | nibalizer, then it's a simpler dual change to project-config & system-config to do the switch | 21:22 |
| * jeblair always wants shorter hostnames :) | 21:22 | |
| fungi | jeblair: so maybe we need to install distro packages for libgmp-dev or libmpir-dev (or whatever the names of the header packages are)? | 21:23 |
| nibalizer | asselin: i'm okay with that i think | 21:23 |
| nibalizer | im wondering if anything else busts if you don't have the gerrit project | 21:23 |
| clarkb | jeblair sure feXX works then I can restrict puppet regex after two new nodes are built | 21:23 |
| jeblair | fungi: yeah, i'm about to explore that, but i'm going to dig into whether and why this appears to have any releation to thread scheduling | 21:23 |
| asselin | nibalizer, as far as I know, it should be ok.....but wanted a 2nd opinion | 21:23 |
| nibalizer | ya we'll need to get core's thoughts on that | 21:24 |
| jeblair | asselin: can you do a wip patch that does what you propose for one project? then we can double check that and okay the approach | 21:25 |
| asselin | jeblair, ok I'll do it on my existing patch that's open. split it out into two and go from there. that's a good idea. | 21:25 |
| *** Longgeek has quit IRC | 21:26 | |
| nibalizer | clarkb: your puppet regex looks good, reviewed | 21:26 |
| clarkb | thanks | 21:26 |
| nibalizer | btw this is how i tested that http://paste.ubuntu.com/9447002/ | 21:27 |
| jeblair | nibalizer: thanks for the tip | 21:28 |
| *** yamamoto has joined #openstack-infra | 21:31 | |
| *** esker has quit IRC | 21:32 | |
| *** yfried_ is now known as yfried|afk | 21:32 | |
| *** sabari is now known as zz_sabari | 21:33 | |
| *** alexpilotti has quit IRC | 21:35 | |
| *** zz_sabari is now known as sabari | 21:35 | |
| *** yamamoto has quit IRC | 21:35 | |
| *** nfedotov1 has joined #openstack-infra | 21:35 | |
| openstackgerrit | Merged openstack-infra/system-config: Update git frontend regex to support more nodes https://review.openstack.org/140480 | 21:36 |
| *** mestery_ has joined #openstack-infra | 21:37 | |
| *** nfedotov has quit IRC | 21:38 | |
| *** pc_m_ has quit IRC | 21:38 | |
| *** mestery has quit IRC | 21:38 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add test for validating certain gerrit ACL rules https://review.openstack.org/140470 | 21:39 |
| clarkb | oh hrm jhesketh +2'd and fungi approved before I could s/frontend/fe/ | 21:39 |
| * clarkb writes a follow up patch | 21:39 | |
| *** Sukhdev_ has joined #openstack-infra | 21:39 | |
| fungi | clarkb: oh, i thought you were merely abbreviating for the sake of irc | 21:39 |
| *** Sukhdev has quit IRC | 21:39 | |
| fungi | clarkb: sorry about that! | 21:39 |
| jhesketh | clarkb: wouldn't be too late to change surely? | 21:40 |
| jhesketh | (ie in a new patch) | 21:40 |
| jeblair | morganfainberg: labels and prolog rules are in section 4 here: https://review.openstack.org/Documentation/index.html | 21:40 |
| *** armax has joined #openstack-infra | 21:40 | |
| clarkb | jhesketh: it isn't I just need to push a new change | 21:40 |
| clarkb | not a big deal | 21:40 |
| jhesketh | right :-) | 21:40 |
| fungi | can't we convince gerrit to allow scripting in lisp too? | 21:40 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Use 'fe' instead of 'frontend' https://review.openstack.org/140486 | 21:41 |
| fungi | (and forth? lua?) | 21:41 |
| clarkb | jhesketh: fungi ^ | 21:41 |
| clarkb | once that gets in I will build two new hosts | 21:41 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Add puppet-recheckwatch as split out module https://review.openstack.org/140430 | 21:41 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Add puppet-recheckwatch as split out module Part 1 https://review.openstack.org/140487 | 21:41 |
| *** subscope has quit IRC | 21:42 | |
| jeblair | clarkb: 2 commits for the price of one, nice | 21:42 |
| jeblair | fungi, clarkb: so my best theory is that what i've been seeing in my testing is just pycrypto hogging the cpu by doing lots of math while holding the GIL | 21:42 |
| fungi | that sounds plausible | 21:42 |
| clarkb | jeblair: if you install pycrypto with the fastmath on trusty does it go quicker? | 21:43 |
| fungi | sounded like that was the next thing to find out | 21:43 |
| jeblair | and what looks like threads stuck releasing the lock is threads being starved (it locks and releases a lot, so many of the releases are not at the same point in the cycle) | 21:43 |
| fungi | jeblair: mordred: are we still keeping the afs/kdc servers out of teh ansible hostlist for any particular reason? | 21:43 |
| jeblair | clarkb: adding libgmp-dev makes it faster with pip installed on precise; about to test with trusty | 21:44 |
| jeblair | fungi: haven't we switched back to nova inventory? | 21:44 |
| fungi | jeblair: i must have missed that patch | 21:45 |
| *** baoli has quit IRC | 21:45 | |
| fungi | jeblair: but i'm thinking no, judging from puppetboard | 21:45 |
| jeblair | that seems like a serious omission :( | 21:46 |
| *** baoli has joined #openstack-infra | 21:46 | |
| jeblair | i think it was intended to be temporary for the puppet3 move | 21:46 |
| jeblair | i'm certainly very un-keen on a system where we have to remember to add systems to secret non-version-controlled files for them to be updated :( | 21:47 |
| fungi | if i'm reading between the lines correctly, the idea is that the ansible run script would just look at nova list output? | 21:47 |
| clarkb | no it should look at puppet cert list output iirc | 21:47 |
| fungi | ahh | 21:47 |
| clarkb | and that code exists let me go find it | 21:47 |
| *** esker has joined #openstack-infra | 21:47 | |
| fungi | maybe it's in the infinite review backlog i'm still wading through | 21:47 |
| jeblair | well, whatever we did before we turned it off for the puppet move | 21:48 |
| jeblair | i thought mordred had it consulting nova | 21:48 |
| jeblair | but either way | 21:48 |
| clarkb | ya let me figure out how to flip it back | 21:48 |
| mordred | what did I so? | 21:48 |
| krotscheck | fungi: Looks like I have to modify the puppet module. Lemme revert what I had and present a new patch. | 21:49 |
| *** resker has joined #openstack-infra | 21:49 | |
| nibalizer | it should be consulting puppet cert list irrc | 21:49 |
| nibalizer | er iirc | 21:49 |
| clarkb | this is actually something I dislike a lot about ansible. The config is so obtuse | 21:49 |
| mordred | ah - so, it's supposed to query puppet cert for now - but we removed that for the puppet3 move | 21:49 |
| mordred | clarkb: 0_o ? | 21:49 |
| nibalizer | ya puppet3 move introduced complexity, but it can be removed | 21:49 |
| mordred | we just need to put the puppet cert inventory back | 21:49 |
| jeblair | clarkb: yes, gmp makes go fast | 21:49 |
| clarkb | mordred: yes "just" | 21:49 |
| *** esker_ has joined #openstack-infra | 21:50 | |
| mordred | clarkb: it's not hard. it's a file. it's very clear | 21:50 |
| mordred | and then, when I've fixed launch_node, we can move to the "query nova" version | 21:50 |
| clarkb | mordred: yes but how do you configure that. It looks like hostfile in ansilble.cfg wants ot point at a host file | 21:50 |
| clarkb | mordred: so how to tell it run a thing, take stdout, and use use? | 21:50 |
| mordred | clarkb: it can also point to an executable | 21:50 |
| clarkb | mordred: that is obtuse :) | 21:51 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Revert "Disabled rabbit repo management." https://review.openstack.org/140489 | 21:51 |
| clarkb | hostfile being a hostfile or a script is not very clear | 21:51 |
| mordred | http://docs.ansible.com/intro_configuration.html#hostfile | 21:51 |
| *** mestery_ is now known as mestery | 21:51 | |
| fungi | krotscheck: okay, in a holding pattern now | 21:51 |
| *** jedimike has quit IRC | 21:51 | |
| clarkb | anyway I think if we just puppet the puppetmaster we will get what we want /me tries this with --noop | 21:51 |
| *** mbacchi has quit IRC | 21:52 | |
| *** esker has quit IRC | 21:52 | |
| clarkb | that doesn't work at all :( | 21:52 |
| asselin | nibalizer, jeblair this patch (test out puppet-module split project config change into 2 patches, with first that can be done for all modules as once) passed Jenkins: https://review.openstack.org/#/c/140487/1 | 21:52 |
| mordred | clarkb: what are you trying to do? | 21:52 |
| clarkb | mordred: `puppet agent --test --noop` on the puppetmaster | 21:53 |
| clarkb | asthat should fix the ansible.cfg | 21:53 |
| mordred | clarkb: right - but to do what? | 21:53 |
| clarkb | to fix the ansible.cfg | 21:53 |
| *** resker has quit IRC | 21:53 | |
| clarkb | I think the puppet.conf on puppetmaster is incomplete | 21:53 |
| *** esker_ has quit IRC | 21:54 | |
| mordred | you need to update :q | 21:54 |
| mordred | gah | 21:54 |
| mordred | you need to update modules/openstack_project/manifests/puppetmaster.pp | 21:54 |
| *** dkliban is now known as dkliban_afk | 21:54 | |
| mordred | we overrode the hostfile param in teh ansible class here: ansible_hostfile => '/etc/ansible/hostfile', | 21:55 |
| mordred | remove that line, then run puppet on puppetmaster and it should go back to being dynamic | 21:55 |
| clarkb | yes but first I need to make puppet work on the puppetmaster | 21:55 |
| mordred | kk | 21:55 |
| *** vigneshvar has quit IRC | 21:57 | |
| openstackgerrit | Merged openstack-infra/system-config: Use 'fe' instead of 'frontend' https://review.openstack.org/140486 | 21:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Install libgmp-dev on nodepool https://review.openstack.org/140490 | 21:57 |
| *** esker has joined #openstack-infra | 21:58 | |
| *** esker has quit IRC | 21:59 | |
| *** esker has joined #openstack-infra | 21:59 | |
| clarkb | jeblair: did you want to rebuild the trusty nodepool or just manaully reinstall pycrypto there? | 21:59 |
| jeblair | clarkb: manual should be fine; i'll do real quick | 22:00 |
| clarkb | ok so adding certname and server to puppet.conf on puppetmaster gets it to run under noop, but the only thing it does is remove those config items | 22:00 |
| clarkb | going to sort out why that is, push fix, and push ansible thing on top of that | 22:00 |
| krotscheck | fungi: As soon as https://review.openstack.org/140489 you can proceed with precise. I’m adjusting the module to then handle trusty | 22:01 |
| mordred | clarkb: it usually does that when fqdn is somehow set poorly | 22:01 |
| mordred | fwiw | 22:01 |
| fungi | jeblair: -1 on that change, see inline... typo? | 22:01 |
| *** esker has quit IRC | 22:01 | |
| jeblair | oops | 22:01 |
| fungi | okay, i shouldn't doubt myself then ;) | 22:02 |
| clarkb | mordred: in this case it is because we check the version of puppet | 22:02 |
| clarkb | nibalizer: can you help unravel this? | 22:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Install libgmp-dev on nodepool https://review.openstack.org/140490 | 22:02 |
| dougwig | hi infra. neutron split again. this job failed grenade (upgrade testing), which his not surprising: https://review.openstack.org/#/c/139901/ | 22:02 |
| fungi | krotscheck: okay, fast-approved the revert. thanks | 22:02 |
| krotscheck | thanks | 22:02 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/puppet.conf.erb#n6 is the problem. puppet is version 3.7 | 22:02 |
| dougwig | i'm thinking we need to make that non-voting for a week or two. | 22:02 |
| clarkb | so it doesn't install the hostname | 22:02 |
| *** harlowja has quit IRC | 22:02 | |
| clarkb | also I hate puppet | 22:03 |
| fungi | dougwig: probably would help to get sdague/mtreinish to weigh in | 22:03 |
| *** bswartz has quit IRC | 22:03 | |
| dougwig | assuming yes to that, and since that was the only failure, is there any objection to ninja merging that commit, so the folks at the sprint aren't basing their reviews on a repo with a bunch of service files? | 22:03 |
| clarkb | nibalizer: are factpath and templatedir the only thing that should be in that if block? | 22:04 |
| clarkb | nibalizer: pull server and certname and pluginsync out of it? | 22:04 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add test for validating certain gerrit ACL rules https://review.openstack.org/140470 | 22:04 |
| *** shashankhegde has quit IRC | 22:04 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Add Gerrit tracking id for storyboard https://review.openstack.org/140492 | 22:04 |
| mtreinish | dougwig: well, what is the upgrade story here? I think grenade raises a good question. | 22:04 |
| *** amitgandhinz has quit IRC | 22:05 | |
| *** cpowell_ has quit IRC | 22:05 | |
| *** ryanpetrello has quit IRC | 22:05 | |
| dougwig | it does, and the plan is for the four repos are going to be packaged together for kilo. and the grenade test can be modified to do the same, but that won't work until those three sub-repos are working, which will be a few days yet. | 22:06 |
| *** mattfarina has quit IRC | 22:06 | |
| dougwig | mtreinish: ^^ | 22:07 |
| mestery | dougwig: ++ | 22:07 |
| *** zz_avozza is now known as avozza | 22:07 | |
| nibalizer | clarkb: are we talking aobut this ? https://review.openstack.org/#/c/134362/1 | 22:07 |
| mtreinish | dougwig: this is what concerns me we keep having to turn off more and more testing in a rush to get you guys unblocked. But, it feels like this part of the plan hasn't been thought out | 22:07 |
| mestery | mtreinish: This is all work we know we have to do, but the first step is getting the 4 repos working again. | 22:07 |
| jeblair | clarkb: pycrypto updated on nodepool-dev | 22:08 |
| dougwig | mtreinish: you're right, we did. i totally spaced testing, and now i'm trying to get neutron unblocked, then pick up the pieces. | 22:08 |
| mtreinish | mestery: I'm just concerned by having a window where upgrades aren't being tested at all | 22:08 |
| openstackgerrit | Merged openstack-infra/puppet-storyboard: Revert "Disabled rabbit repo management." https://review.openstack.org/140489 | 22:08 |
| nibalizer | so this is the real problem and fix https://review.openstack.org/#/c/137246/ | 22:09 |
| *** andreykurilin_ has joined #openstack-infra | 22:09 | |
| mestery | mtreinish: Understood, and I agree completely. I will ensure the window is as small as possible. | 22:09 |
| nibalizer | clarkb: jeblair so the first time we did this I screwed up the variable | 22:09 |
| mestery | mtreinish: Basically, dougwig and I are on this until it's all working and we'll get grenade enabled ASAP once the 4 repos are back, whatever we have to do. | 22:09 |
| *** dustins has quit IRC | 22:09 | |
| mtreinish | dougwig: are you sure that grenade failure is related to the split? | 22:10 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/elastic-recheck: Add query for bug #1400902 https://review.openstack.org/140495 | 22:10 |
| uvirtbot | Launchpad bug 1400902 in tempest "test_update_router_admin_state failing against ironic " [Undecided,New] https://launchpad.net/bugs/1400902 | 22:10 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Properly set master and certname on newer puppet https://review.openstack.org/140496 | 22:10 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Use puppet inventory when running ansible. https://review.openstack.org/140497 | 22:10 |
| clarkb | nibalizer: ^ something like that | 22:10 |
| dougwig | mtreinish: is there an option to disable checking on lbaas/fwaas/vpnaas in the grenade tests, instead of turning them non-voting? | 22:10 |
| clarkb | jeblair: thanks, should we try switching back to it today? | 22:10 |
| jeblair | clarkb: i'm experimenting with running nodepool fakes with that in place | 22:11 |
| jeblair | i'll get back to you on that | 22:11 |
| clarkb | jeblair: sounds good and thank yo | 22:11 |
| mtreinish | dougwig: it should already be disabled by devstack-gate: http://logs.openstack.org/01/139901/4/check/check-grenade-dsvm-neutron/d657bff/logs/new/tempest_conf.txt.gz | 22:11 |
| *** alexpilotti has joined #openstack-infra | 22:11 | |
| fungi | i'm happy to help with another nodepool cut-over later this evening if desired, though i'm going to disappear for dinner for a while first | 22:11 |
| dougwig | huh, looks like maybe not. | 22:11 |
| *** esker has joined #openstack-infra | 22:11 | |
| nibalizer | clarkb: i think we want 137246 | 22:11 |
| clarkb | ya I have to disappear at ~4:30 local time. 0030UTC | 22:11 |
| dougwig | ok, me being full of crap is full of crap. let me run a recheck. | 22:11 |
| mestery | We're learning dougwig. :) | 22:11 |
| *** esker has quit IRC | 22:11 | |
| dougwig | (that's twice as much crap. sigh.) :) | 22:12 |
| fungi | once you're full of enough crap, you can have my job ;) | 22:12 |
| clarkb | nibalizer: I mean sure except that that sort of funnily fixes the problem | 22:12 |
| mestery | fungi: lol | 22:12 |
| *** esker has joined #openstack-infra | 22:12 | |
| *** Sincler has joined #openstack-infra | 22:12 | |
| dougwig | lol | 22:12 |
| mtreinish | dougwig: yeah, looking at the logs from the run it failed on a test which isn't related to lbaas, fwaas, or whatever the other one is | 22:12 |
| dougwig | thank you, sorry for fire drill. | 22:13 |
| clarkb | nibalizer: +2 let us see if we can get fungi or jhesketh or jeblair to review too | 22:13 |
| jhesketh | clarkb: link? | 22:13 |
| nibalizer | clarkb: what do you mean? | 22:13 |
| *** kumartin has joined #openstack-infra | 22:13 | |
| fungi | clarkb: nibalizer: yep, was just reviewing that | 22:13 |
| clarkb | nibalizer: the bug is a different change | 22:13 |
| clarkb | nibalizer: this fix is fixing the bug + other stuff | 22:13 |
| clarkb | jhesketh: https://review.openstack.org/#/c/137246/ | 22:13 |
| *** melwitt has joined #openstack-infra | 22:14 | |
| *** dmsimard_away is now known as dmsimard | 22:14 | |
| nibalizer | clarkb: i see what you're saying | 22:14 |
| nibalizer | but that change is kinda old at this point | 22:14 |
| fungi | jeblair: should i hold off approving the libgmp-dev addition until your manual load-testing is completed? | 22:15 |
| clarkb | ya I think its fine to go with that one | 22:15 |
| *** dmsimard is now known as dmsimard_away | 22:15 | |
| nibalizer | i identified this problem a while back and fixed it and some other problems | 22:15 |
| clarkb | fungi has approved it | 22:15 |
| nibalizer | okay cool | 22:15 |
| clarkb | so now we wait then I try puppetting again | 22:15 |
| nibalizer | ya also i made a mistacke when we went to puppet3 | 22:15 |
| anteaya | clarkb fungi before y'all leave can I bug you for a devstack-gate review? https://review.openstack.org/#/c/140128/ | 22:15 |
| nibalizer | $puppet_version isn't a variable $puppetversion is, im not sure how our puppetmaster templating ever worked | 22:16 |
| clarkb | anteaya: will you be sad if I -2 that change? | 22:16 |
| clarkb | anteaya: devstack should just do it (tm) | 22:16 |
| anteaya | clarkb: any review is a good review | 22:16 |
| anteaya | silence is the only thing that doesn't help | 22:16 |
| clarkb | anteaya: is sean around? | 22:16 |
| anteaya | sean dague or collins | 22:16 |
| anteaya | sc68cal: ping | 22:16 |
| anteaya | sc68cal: <- sean collins | 22:17 |
| sdague | fungi / clarkb - where are we reporting things like apt mirror mass failures for ER bugs? | 22:17 |
| sdague | openstack-ci bugs seem to be non existant now | 22:17 |
| *** KurtMartin is now known as kmartin | 22:17 | |
| fungi | sdague: openstack-gate | 22:17 |
| *** JayJ has quit IRC | 22:17 | |
| *** esker has quit IRC | 22:17 | |
| sdague | fungi: thanks! | 22:17 |
| clarkb | anteaya: I commented I was nicer and left a -1 | 22:17 |
| *** nfedotov1 has quit IRC | 22:17 | |
| sc68cal | hello | 22:17 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add test for validating certain gerrit ACL rules https://review.openstack.org/140470 | 22:18 |
| *** JayJ has joined #openstack-infra | 22:18 | |
| clarkb | sc68cal: hey, I don't think that devstack-gate change should be necessary. devstack should just use ipv6 and any devstack-gate changes we make should be to make that happen | 22:18 |
| clarkb | sc68cal: rather than having an explicit flag we must toggle | 22:18 |
| openstackgerrit | Merged openstack-infra/system-config: Remove logic from puppet.conf https://review.openstack.org/137246 | 22:19 |
| sc68cal | clarkb: that would involve setting IP_VERSION=6+4 as a default | 22:19 |
| clarkb | sc68cal: yes that is what I am saying | 22:19 |
| *** claco has joined #openstack-infra | 22:19 | |
| clarkb | I do not think this should be a non default flag | 22:19 |
| clarkb | it should just go | 22:19 |
| clarkb | sc68cal: and we should be able to get that in as a self testing change to devstack | 22:20 |
| clarkb | also is ipv6 currently only a neutron thing? | 22:21 |
| clarkb | (just curious about that) | 22:21 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: add query for hash sum mismatch https://review.openstack.org/140500 | 22:22 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-storyboard: Rabbit repository management https://review.openstack.org/140501 | 22:22 |
| sc68cal | clarkb: the only concern i have is setting 4+6 without having the fix for #1400823 merged | 22:22 |
| sc68cal | https://bugs.launchpad.net/devstack/+bug/1400823 | 22:22 |
| uvirtbot | Launchpad bug 1400823 in devstack "DevStack needs to set an IPv6 address on PUBLIC_BRIDGE if no ipv6 address has been set" [Undecided,In progress] | 22:22 |
| *** JayJ has quit IRC | 22:22 | |
| sdague | clarkb: I think 6+4 can't be a devstack default due to the fact that lots of people put this on non IPv6 aware networks | 22:22 |
| sc68cal | I hit this on my vagrant VM - because VirtualBox does not spawn an radvd process on the host even if you set an IPv6 prefix on the virtualbox network | 22:22 |
| sc68cal | ^ +1 | 22:23 |
| clarkb | sdague: they can override to be v4 only | 22:23 |
| clarkb | sdague: sc68cal if we are going to be serious about ipv6 we should require it by default everywhere | 22:23 |
| sdague | clarkb: the defaults should work | 22:23 |
| *** nfedotov has joined #openstack-infra | 22:23 | |
| jeblair | krotscheck: is storyboard on precise and storyboard-dev trusty? | 22:23 |
| clarkb | sdague: they will work | 22:23 |
| clarkb | and we will test it | 22:23 |
| sdague | not for most people | 22:23 |
| sc68cal | clarkb: I agree - my hope is to use the experimental job to find the big landmines | 22:23 |
| *** changbl has quit IRC | 22:23 | |
| sc68cal | then switch it to a default very soon | 22:23 |
| krotscheck | jeblair: I believe fungi will give more insight on this, but I believe that both are going to be on precise for the time being. | 22:23 |
| clarkb | sc68cal: right and I think an experimental job is the wrong approach | 22:24 |
| krotscheck | jeblair: I would prefer we move entirely to trusty. | 22:24 |
| sdague | clarkb: I'm good with this is d-g default | 22:24 |
| clarkb | sc68cal: mostly beacuse this should be easily self testing with existing jobs | 22:24 |
| sdague | but I don't think it's a good devstack default | 22:24 |
| fungi | jeblair: krotscheck: yes, i'm happy to install whichever wherever but most of the time, excepting testing transitions, they should hopefully match | 22:24 |
| sdague | because the majority of people run devstack in environments where v6 isn't working correctly | 22:25 |
| clarkb | sdague: do they though? most peopl run single node devstacks on their laptops | 22:25 |
| sc68cal | even Vagrant/VirtualBox doesn't do it right | 22:25 |
| clarkb | sdague: and ipv6 should work there all the time | 22:25 |
| jeblair | fungi: yeah, my question was really "are we starting the upgrade to trusty via storyboard-dev" | 22:25 |
| sdague | clarkb: nope | 22:25 |
| sc68cal | clarkb: are laptops configured with v6 correctly? | 22:25 |
| fungi | jeblair: krotscheck: if we want to take storyboard-dev as an opportunity to stand it up on trusty for a bit and then follow suit replacing production with trusty, i'm good with that plan too | 22:25 |
| jeblair | krotscheck: so you're just laying groundwork for trusty then? | 22:25 |
| sc68cal | if the network they're attached to isn't, they'd need to configure a ULA prefix | 22:26 |
| sc68cal | and spin up radvd | 22:26 |
| clarkb | sc68cal: it doesn't matter if the laptop is... you are creating a fake set of networking anyways | 22:26 |
| krotscheck | I’d suggest we get both onto precise, test upgrade dev, then if that works without a problem we can upgrade prod | 22:26 |
| clarkb | sc68cal: yes that is why we have devstack... | 22:26 |
| sc68cal | clarkb: right -but that fake networking, like vboxnet doesn't work right | 22:26 |
| asselin | mordred, ping re: rework-launch-node | 22:26 |
| sc68cal | doesn't advertise the prefix, etc.. | 22:26 |
| jeblair | krotscheck: i like that plan too. fungi? | 22:26 |
| clarkb | sc68cal: sdague is that a problem in single node or just multinode? | 22:27 |
| clarkb | want to make sure I understand what breaks | 22:27 |
| sc68cal | clarkb: single - take a look at my pastes in https://bugs.launchpad.net/devstack/+bug/1400823 | 22:27 |
| uvirtbot | Launchpad bug 1400823 in devstack "DevStack needs to set an IPv6 address on PUBLIC_BRIDGE if no ipv6 address has been set" [Undecided,In progress] | 22:27 |
| clarkb | sc68cal: right ut thats a devstack bug that we can fix | 22:27 |
| fungi | krotscheck: jeblair: works for me | 22:27 |
| clarkb | sc68cal: it doesn't appear to be a virtualbox will never make this work issue | 22:27 |
| sdague | clarkb: so right now, I don't have the energy to deal with all the people freaking out when we break them on that default change | 22:28 |
| sdague | because this gets run in all sorts of environments that definitely don't have ipv6 working | 22:28 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config: Fix neutron jobs for tempest on stable branches https://review.openstack.org/140503 | 22:28 |
| mtreinish | clarkb: heh, I had the same comment on rev 1 of that patch :) | 22:28 |
| clarkb | sdague: ok, I think d-g default is also a reasonable step to take | 22:29 |
| sdague | right, for d-g it's environment that we know does ipv6 correctly | 22:29 |
| clarkb | sdague: sc68cal: so next question: is this only a neutron thing? because I would expect DEVSTACK_IPV6 to just work for every service we run or at least for that to be the goal | 22:29 |
| sdague | clarkb: good question | 22:30 |
| sc68cal | clarkb: I think IP_VERSION does changes to other services to have them listen on v6 | 22:30 |
| sc68cal | let me grab a link to diff | 22:30 |
| sdague | yeh, I remember that landing | 22:30 |
| clarkb | sdague: sc68cal ok, can we update that change then to not be neutron specific and just set it always in localrc | 22:30 |
| clarkb | then see what breaks | 22:30 |
| sc68cal | hmmm | 22:30 |
| clarkb | you may have to switch on branch though since icehouse and havana may not be happy | 22:30 |
| sdague | sure, though tempest is probably only testing via v4 connections | 22:31 |
| sc68cal | the patchset that landed did not change other services to listen on v6, but an earlier version did | 22:31 |
| clarkb | sdague: I think thats ok for a first step | 22:31 |
| sdague | the service catalog is probably only v4 as well | 22:31 |
| sdague | ok | 22:31 |
| clarkb | sdague: right but if they can at least listen on v6 without falling over we have made progress | 22:31 |
| sdague | just realize it's not actually testing it :) | 22:31 |
| sdague | sure | 22:31 |
| clarkb | sdague: then all future changes from there will be self testing | 22:31 |
| sdague | yeh, I'm good with that | 22:31 |
| *** teran has joined #openstack-infra | 22:31 | |
| sc68cal | so what is our consensus? | 22:32 |
| clarkb | sc68cal: I think d-g change to make this the default for >=juno assuming juno works. just >= kilo otherwise | 22:32 |
| fungi | jeblair: should i hold off approving the libgmp-dev addition until your manual load-testing is completed? | 22:32 |
| clarkb | sc68cal: then everything else can be self testing and we can make progress on this without guessing about the gate or needing to trigger special jobs | 22:33 |
| jeblair | fungi: no, i think it should go in regardless | 22:33 |
| sc68cal | clarkb: you may need to walk me through this a bit | 22:33 |
| sc68cal | which jobs to modify | 22:33 |
| *** esker has joined #openstack-infra | 22:33 | |
| *** dprince has quit IRC | 22:33 | |
| *** esker has quit IRC | 22:33 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: add query for hash sum mismatch https://review.openstack.org/140500 | 22:34 |
| sc68cal | sdague: are you in agreement? | 22:34 |
| fungi | jeblair: done--thanks | 22:34 |
| sdague | sc68cal: yeh, default for devstack-gate seems fine | 22:34 |
| *** changbl has joined #openstack-infra | 22:34 | |
| sdague | it should be self testing even, right? | 22:34 |
| clarkb | sc68cal: don't modify any jobs | 22:35 |
| *** teran_ has quit IRC | 22:35 | |
| sc68cal | clarkb: ah I think I see | 22:35 |
| sc68cal | let me grab a link | 22:35 |
| clarkb | sc68cal: instead in https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n39 that function just make it be on everywhere it has a hope of working | 22:35 |
| jeblair | clarkb, fungi: there's still a measurable behavior change between these two systems: on trusty, after the provider manager has started threads for 500-600 fake machines, its thread is partially starved (processes 1 machine per second istead of 10) until about 300-400 are finished, then it resumes at normal speed | 22:35 |
| clarkb | sdague: yup self testing | 22:35 |
| jeblair | clarkb, fungi: (on precise it plows through at full speed) | 22:35 |
| clarkb | jeblair: huh | 22:36 |
| *** esker has joined #openstack-infra | 22:36 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Add Gerrit tracking id for storyboard https://review.openstack.org/140492 | 22:36 |
| *** nfedotov1 has joined #openstack-infra | 22:37 | |
| *** ryanpetrello has joined #openstack-infra | 22:37 | |
| jeblair | so we could cross our fingers and hope that the pycrypto change was exacerbating this enough to be responsible for most of the pain, but there's still likely an underlying change that may or may not still be a problem | 22:37 |
| fungi | jeblair: so there's still likely something else compounding it. and you see it all snappy on precise even with newer pip-installed pycrypto? | 22:37 |
| sc68cal | clarkb: ok - I was looking at putting it under line 87 | 22:37 |
| sc68cal | clarkb: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n87 | 22:37 |
| *** SumitNaiksatam has quit IRC | 22:37 | |
| clarkb | sc68cal: the nice thing about this approach is everything should just be self testing and as you land fixes or new v6 support regressing that will be hard | 22:37 |
| *** harlowja has joined #openstack-infra | 22:37 | |
| *** esker has quit IRC | 22:37 | |
| clarkb | sc68cal: no because that is only neutron | 22:37 |
| clarkb | sc68cal: and well we do a lot more than just neutron | 22:38 |
| fungi | okay, disappearing for dinner... back soonish | 22:38 |
| sc68cal | IP_VERSION currently just does stuff for neutron | 22:38 |
| jeblair | fungi: i'm back at system installed 2.4.1 on precise; i could try 2.6.1 on precise if you like | 22:38 |
| clarkb | sc68cal: :/ | 22:38 |
| *** esker has joined #openstack-infra | 22:38 | |
| sc68cal | clarkb: also, there are issues with DVR and ipv6 | 22:38 |
| clarkb | sc68cal: but still I think make it global then devstack can grow support for the other things | 22:38 |
| fungi | jeblair: nah, just wondering whether it's still pycrypto contributing to the difference, or something else different between precise and trusty | 22:38 |
| clarkb | sc68cal: I don't think we test dvr | 22:38 |
| sc68cal | clarkb: ok - I'll give it a try higher up in that function | 22:39 |
| jeblair | fungi: doing it anyway, will have an answer in about a minute | 22:39 |
| *** nfedotov has quit IRC | 22:39 | |
| fungi | clarkb: sc68cal: i think proper dvr testing was waiting on multi-node devstack | 22:39 |
| clarkb | fungi: correct | 22:39 |
| openstackgerrit | Merged openstack-infra/system-config: Install libgmp-dev on nodepool https://review.openstack.org/140490 | 22:39 |
| jeblair | fungi: yeah, no change with 261 | 22:40 |
| sc68cal | clarkb: so basically I'll be abandoning https://review.openstack.org/#/c/140163/ | 22:40 |
| fungi | okay, so definitely a compound set of differences | 22:40 |
| fungi | 2+ needles in this haystack i guess | 22:41 |
| clarkb | sc68cal: thats my goal, but we should see what gets shaken out with the updated d-g change | 22:41 |
| clarkb | sc68cal: but really I don't think we need special jobs for this. I think we just get in what works and go from there | 22:41 |
| jeblair | clarkb, fungi: so probably i should do the following: spin up trusty non-pvhvm and try that out. and while it's building, start upgrading python packages on precise one by one | 22:41 |
| *** sabari is now known as zz_sabari | 22:41 | |
| clarkb | jeblair: sounds good | 22:41 |
| *** pc_m has joined #openstack-infra | 22:41 | |
| *** pradk has quit IRC | 22:41 | |
| fungi | jeblair: a painful, but systematic approach | 22:41 |
| fungi | sounds like a logical next step | 22:42 |
| jeblair | that should eventually remove pvhvm and python packages from the list, leaving us with kernel, base os packages, and python versions as differences | 22:42 |
| * fungi will be back soon | 22:42 | |
| jeblair | and then it gets really painful :( | 22:42 |
| *** krtaylor has quit IRC | 22:42 | |
| *** ryanpetrello has quit IRC | 22:44 | |
| *** changbl has quit IRC | 22:45 | |
| *** ryanpetrello has joined #openstack-infra | 22:46 | |
| *** shashankhegde has joined #openstack-infra | 22:46 | |
| openstackgerrit | Sean M. Collins proposed openstack-infra/devstack-gate: Set IP_VERSION in DevStack to run dual stack https://review.openstack.org/140128 | 22:46 |
| *** otter768 has joined #openstack-infra | 22:47 | |
| clarkb | sc68cal: ya that looks good. Now we will see how hard that falls over but I expect it to do reasonably well | 22:48 |
| openstackgerrit | Merged openstack-infra/project-config: Stop using topology variable https://review.openstack.org/140429 | 22:48 |
| *** SumitNaiksatam has joined #openstack-infra | 22:48 | |
| *** dannywilson has quit IRC | 22:49 | |
| sc68cal | clarkb: yup | 22:49 |
| * sc68cal grabs popcorn | 22:49 | |
| *** dannywilson has joined #openstack-infra | 22:49 | |
| sc68cal | i'll also look into making other parts of devstack honor IP_VESRION | 22:49 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Use puppet inventory when running ansible. https://review.openstack.org/140497 | 22:50 |
| clarkb | fungi: jeblair jhesketh ^ ok I rebased that now that nibalizer's change is in | 22:50 |
| *** markmcclain has quit IRC | 22:50 | |
| *** zz_sabari is now known as sabari | 22:50 | |
| clarkb | puppetmaster is puppeting itself now so that should apply as soon as it merges | 22:50 |
| *** mfink has quit IRC | 22:50 | |
| * krotscheck is pondering just pinning the versoin | 22:50 | |
| *** patrickeast_ has joined #openstack-infra | 22:51 | |
| *** mjturek has quit IRC | 22:52 | |
| *** otter768 has quit IRC | 22:52 | |
| clarkb | ok I am going to spin up git-fe01.openstack.org now | 22:52 |
| *** patrickeast_ has quit IRC | 22:52 | |
| *** patrickeast has quit IRC | 22:53 | |
| *** dannywilson has quit IRC | 22:54 | |
| krotscheck | Oh. That doesn’t do what I want it to. Dagnabbit | 22:54 |
| *** marun has quit IRC | 22:54 | |
| *** obondarev has quit IRC | 22:54 | |
| krotscheck | nibalizer: I’m trying to figure out how to disable the rabbit ppa in the case that we’re running in trusty, but I can’t figure out whether I have to include rabbitmq::repo::apt to explicitly disable it (because of the namevar). Any pointers? | 22:56 |
| *** cnesa2 has quit IRC | 22:57 | |
| *** mfink has joined #openstack-infra | 22:58 | |
| *** devlaps has quit IRC | 22:58 | |
| *** gokrokve has quit IRC | 22:58 | |
| *** jgrimm is now known as zz_jgrimm | 22:59 | |
| *** gokrokve has joined #openstack-infra | 22:59 | |
| *** kumartin has quit IRC | 23:00 | |
| *** freyes has quit IRC | 23:00 | |
| *** StevenK has quit IRC | 23:00 | |
| *** armax has quit IRC | 23:01 | |
| jeblair | clarkb, fungi: strik phvhm from the list of potential culprits; behavior is same | 23:01 |
| *** jaypipes has quit IRC | 23:01 | |
| *** freyes has joined #openstack-infra | 23:02 | |
| mtreinish | clarkb: are you guys round robin dns'ing multiple haproxy frontends to get around the single node bandwidth limitations in rax? | 23:02 |
| jeblair | (on the plus side, we have 2 trusty hosts behaving this way in my test) | 23:02 |
| jeblair | mtreinish: yes | 23:02 |
| *** alexpilotti has quit IRC | 23:02 | |
| mtreinish | jeblair: heh, ok cool | 23:02 |
| *** tonytan4ever has quit IRC | 23:03 | |
| *** bknudson has quit IRC | 23:03 | |
| jeblair | mtreinish: that's the best question i've seen asked in a long time. i have nothing to add to that. except it would be cool if we could set the settings we needed in the lbaas product and use that | 23:03 |
| *** Masahiro has joined #openstack-infra | 23:04 | |
| *** Rockyg has quit IRC | 23:04 | |
| *** marun has joined #openstack-infra | 23:04 | |
| nibalizer | krotscheck: code sample? | 23:04 |
| jeblair | (growing the server is an option but seems silly) | 23:04 |
| clarkb | 23.253.252.78 2001:4800:7818:104:be76:4eff:fe04:6c24 is fe01 I am going to test it really quickly and add some dns records if git works | 23:04 |
| *** shakamunyi has joined #openstack-infra | 23:04 | |
| clarkb | then 02 and updating dns | 23:04 |
| krotscheck | nibalizer: I’m looking at this - http://paste.openstack.org/show/148228/ | 23:05 |
| *** obondarev has joined #openstack-infra | 23:05 | |
| *** jamielennox is now known as jamielennox|away | 23:05 | |
| mtreinish | jeblair: well I saw clarkb's comment and saw a patch mentioning round robin. But, I missed the earlier discussion, I assumed it was around the bandwidth limitation, but was curious | 23:05 |
| nibalizer | krotscheck: doesn't the rabbitmq class have a 'manage-repo' ? | 23:06 |
| krotscheck | nibalizer: honestly I’m super leery about doing the management for a repo that’s injected by another module. | 23:06 |
| krotscheck | nibalizer: Except that the rabbitmq module is, how did you put it… a bit problematic? | 23:06 |
| krotscheck | nibalizer: That only adds it. | 23:06 |
| krotscheck | nibalizer: If we upgrade from precise to trusty it never removes it. | 23:06 |
| nibalizer | ya we have a change up | 23:06 |
| nibalizer | krotscheck: well the box won't be upgraded | 23:07 |
| nibalizer | jeblair and co will provision a new trusty machine ya? | 23:07 |
| *** imcsk8 has quit IRC | 23:07 | |
| *** imcsk8 has joined #openstack-infra | 23:07 | |
| krotscheck | nibalizer: We _could_ do it that way. | 23:08 |
| jeblair | oh! | 23:08 |
| *** Masahiro has quit IRC | 23:08 | |
| mtreinish | jeblair: well, if it was for my personal use, I'd weigh the cost of scaling a single node up vs horizontally across multiple nodes | 23:08 |
| krotscheck | In the interest of being a good open source citizen though I’d like to have a clean upgrade path if someone runs do-release-upgrade | 23:08 |
| jeblair | clarkb: i got precise to exhibit the behavior with python package changes | 23:08 |
| mtreinish | But I guess this is more resilient in the long run, and I'm guess the quotas are big enough cost doesn't really matter either | 23:08 |
| *** rmcall has joined #openstack-infra | 23:09 | |
| jeblair | mtreinish: we're on an 8g node and would need to move to a 30g node, so this is actually the less expensive route | 23:09 |
| clarkb | jeblair: progress! | 23:09 |
| *** craigbr has quit IRC | 23:09 | |
| jeblair | clarkb: but now i have to bisect which of these it is: | 23:09 |
| jeblair | cryptography==0.6.1 cffi, pycparser ecdsa==0.11 lockfile==0.10.2 lxml==3.3.3 pytz==2014.10 requests==2.2.1 urllib3==1.7.1 paramiko==1.15.1 pyOpenSSL==0.14 | 23:09 |
| jeblair | :( | 23:09 |
| mtreinish | jeblair: hmm, ok well then it's a win all the way around :) | 23:10 |
| jeblair | cause, you know, i grouped them because i wasn't actually expceting this | 23:10 |
| clarkb | jeblair: gah that list is rather large | 23:10 |
| jeblair | it was like a quarter of the list | 23:10 |
| *** Longgeek has joined #openstack-infra | 23:10 | |
| clarkb | jeblair: fungi: jhesketh: ok fe01 looks good so I am addings its DNS records then will go modify git.openstack.org as well. Also we should revoke the git.openstack.org puppet cert so that ansible doens't run twice on the same nodes | 23:11 |
| andreaf | jeblair: when you have a moment, I'm having trouble with gertty: https://storyboard.openstack.org/#!/story/2000073 - thanks | 23:11 |
| jeblair | krotscheck, nibalizer: yes, we will almost certainly provision a replacement machine | 23:11 |
| jeblair | krotscheck: it did that thing where it didn't load any data again | 23:11 |
| nibalizer | krotscheck: well if you know that you're not telling rabbitmq to manage the apt repo then you also know that you can re-define that repo | 23:12 |
| krotscheck | jeblair: Looks like it’s a race condition in token resolution. | 23:12 |
| nibalizer | it also might make sense to patch the rabbitmq module so that it can accept an 'enable_repo => purged' as opposed to just slicing it off the dag | 23:12 |
| nibalizer | that second thing might be the winner | 23:13 |
| *** nfedotov1 has left #openstack-infra | 23:13 | |
| clarkb | nibalizer: how do I make a puppet cert not active anymore? | 23:13 |
| clarkb | I can google if you are busy | 23:13 |
| krotscheck | nibalizer: Well, I’ve got this -> https://review.openstack.org/#/c/140501/ | 23:13 |
| nibalizer | puppet cert clean; puppet cert deactivate | 23:13 |
| clarkb | nibalizer: do they do different things? | 23:13 |
| krotscheck | Which, assuming we’re provisioning new machines, should work. | 23:13 |
| clarkb | jeblair: ready for me to add fe01 to git.o.o A and AAAA records? | 23:14 |
| *** ivar-laz_ has joined #openstack-infra | 23:14 | |
| nibalizer | clarkb: yes | 23:14 |
| *** ivar-laz_ has quit IRC | 23:14 | |
| nibalizer | one revokes the cert | 23:14 |
| *** e0ne has quit IRC | 23:14 | |
| nibalizer | the other tells puppetdb that node is deactivated | 23:15 |
| nibalizer | i dont know if puppetdb removes that data or not | 23:15 |
| *** e0ne has joined #openstack-infra | 23:15 | |
| nibalizer | i think its a bit weird | 23:15 |
| *** Longgeek has quit IRC | 23:15 | |
| clarkb | nibalizer: thank you | 23:15 |
| nibalizer | but point is puppet node deactivate makes the node not show up in puppetboard any more | 23:15 |
| *** mikedillion has joined #openstack-infra | 23:16 | |
| jeblair | andreaf: oh, this seems familiar; it's not a code bug, it's an installation problem... but i guess we haven't had a bug logged with it... | 23:16 |
| jeblair | andreaf: trying to dig up/remember details | 23:16 |
| *** ivar-lazzaro has quit IRC | 23:17 | |
| clarkb | I am revoking git.o.o's puppet cert now so that ansible does the right thing with it | 23:18 |
| *** eharney has quit IRC | 23:18 | |
| *** esker has quit IRC | 23:18 | |
| mtreinish | is there a recommended place to put images for developer docs? Should I just dump them in the project repo? | 23:18 |
| jeblair | andreaf: oh, maybe it's that 'pip install .' does not work... | 23:18 |
| jeblair | andreaf: try making a new venv; rm ~/.gertty.db; pip install -e . | 23:19 |
| jeblair | andreaf: (with the -e (editable) flag) | 23:19 |
| mtreinish | jeblair: why wouldn't it work with a regular install? | 23:19 |
| jeblair | andreaf: (in which case, it probably is a code bug we need to deal with somehow) | 23:19 |
| jeblair | mtreinish: mumble alembic mumble? | 23:20 |
| clarkb | nibalizer: puppet cert deactivate is not a thing | 23:20 |
| jeblair | mtreinish: it works with pypi and editable | 23:20 |
| jeblair | mtreinish: my poor memory says it might not work with non-editable local install | 23:20 |
| nibalizer | er puppet node deactivate | 23:20 |
| nibalizer | my bad | 23:20 |
| *** esker has joined #openstack-infra | 23:20 | |
| nibalizer | clarkb: ^ | 23:20 |
| *** baoli has quit IRC | 23:21 | |
| mtreinish | jeblair: heh, ok. I had a similar issue in subunit2sql with my alembic stuff, it was looking for an alembic.ini which didn't get copied by install. | 23:21 |
| clarkb | nibalizer: thanks | 23:21 |
| mtreinish | I just had to add it to the manifest file and that fixed it | 23:21 |
| clarkb | ok git.o.o is a round robin now right git.o.o and git-fe01 in it | 23:21 |
| andreaf | jeblair: thanks I'll give it a try | 23:21 |
| clarkb | when fe02 is done building and I have finsihed testing it I will add it to the round robin and remove git.o.o | 23:21 |
| clarkb | then we can delete both git.o.o hosts when people stop using them | 23:22 |
| mtreinish | but I have to wrap alembic to use oslo.db stuff in migrations so it might just be something specific to that | 23:22 |
| jeblair | mtreinish: ooh ooh that sounds a lot like what andreaf reported | 23:22 |
| sc68cal | clarkb: looks like tempest tests are failing - reading the logs of https://jenkins03.openstack.org/job/check-tempest-dsvm-ironic-pxe_ssh/499/consoleText they're doing a comparison, and it's failing on 2 != 1, in verify gateway port - going to guess it's expecting only a single IP address :) | 23:22 |
| *** markmcclain has joined #openstack-infra | 23:22 | |
| *** esker has quit IRC | 23:22 | |
| *** mikedillion has quit IRC | 23:23 | |
| mtreinish | jeblair: http://git.openstack.org/cgit/openstack-infra/subunit2sql/commit/?id=fe24b4a39684e2792aa49fec214a7f97fb62ec71 | 23:23 |
| mtreinish | I'll check the gertty repo if something similar is needed there too | 23:23 |
| jeblair | mtreinish: thanks! | 23:23 |
| *** mikedillion has joined #openstack-infra | 23:23 | |
| *** jamielennox|away is now known as jamielennox | 23:24 | |
| *** esker has joined #openstack-infra | 23:24 | |
| clarkb | sc68cal: I wonder ifthat is just the ironic tests? sorry juggling a couple things right now but will try to take a look shrotly | 23:24 |
| *** teran_ has joined #openstack-infra | 23:24 | |
| andreaf | jeblair: install -e worked fine | 23:24 |
| sc68cal | clarkb: nope - take a look at this http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/network/test_routers.py#n341 | 23:25 |
| *** ddieterly has quit IRC | 23:25 | |
| sc68cal | they expect only one fixed IP on a port | 23:25 |
| clarkb | nibalizer: so puppet help node showd deavtivate but not that you should pass any hosts to hit and man puppet-node doens't have deactivate... | 23:25 |
| *** esker has quit IRC | 23:25 | |
| sc68cal | dual stacked router means two fixed ips - one for v4 and one for v6 | 23:25 |
| *** e0ne has quit IRC | 23:26 | |
| nibalizer | clarkb: https://docs.puppetlabs.com/puppetdb/latest/maintain_and_tune.html#deactivate-decommissioned-node | 23:26 |
| sc68cal | hmmm but the others are passing :-\ | 23:26 |
| *** teran has quit IRC | 23:27 | |
| clarkb | ok puppet master and puppetdb things done for git.o.o | 23:27 |
| clarkb | now to test fe02 | 23:27 |
| clarkb | v6 is faster for me than v4 | 23:28 |
| clarkb | I wonder if that is comcasts way of saying please just use v6 | 23:28 |
| *** krtaylor has joined #openstack-infra | 23:29 | |
| anteaya | jogo: so taking a look at config-drive in the devstack lib/nova it appears that FALSE turns off the config-drive: http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n80 | 23:29 |
| sc68cal | clarkb: we've seen some good performance improvements v4 vs. v6 - with same workload | 23:30 |
| mtreinish | jeblair: yeah that looks like it's the issue, patch incoming | 23:30 |
| clarkb | sc68cal: ya I am super happy that rax lets us v6 | 23:30 |
| mtreinish | is there a format to use for storyboard issues in the commit message? | 23:30 |
| anteaya | jogo: so any value other than 1 triggers the else in devstack-gate? | 23:30 |
| clarkb | ok fe02 is looking good. updating dns for it now | 23:30 |
| krotscheck | mtreinish: “Story: ###" | 23:30 |
| mtreinish | krotscheck: thanks | 23:31 |
| jogo | anteaya: https://review.openstack.org/#/c/136504/10/devstack-vm-gate-wrap.sh,cm | 23:31 |
| openstackgerrit | Matthew Treinish proposed stackforge/gertty: Add manifest file and include alembic.ini https://review.openstack.org/140522 | 23:31 |
| mtreinish | jeblair, andreaf: ^^^ That should fix the issue | 23:31 |
| *** patrickeast has joined #openstack-infra | 23:32 | |
| *** dannywilson has joined #openstack-infra | 23:32 | |
| jogo | anteaya: I think so, DEVSTACK_GATE_CONFIGDRIVE=0 is used elsewhere in jenkins/jobs/devstack-gate.yaml | 23:33 |
| *** gyee has joined #openstack-infra | 23:33 | |
| nibalizer | clarkb: i think its comcast having gear that does qos better on v6 | 23:34 |
| nibalizer | :P | 23:34 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Prepare project-config for puppet module split https://review.openstack.org/140523 | 23:35 |
| asselin | nibalizer, when you have a chance, please take a look at ^^ and see if it makes sence. I think there are some modules that should be removed. | 23:36 |
| mtreinish | clarkb: the ipv6 devstack gate stuff is all to make sure you get better tv service in the long run... :) | 23:36 |
| nibalizer | oh hrm | 23:37 |
| nibalizer | yea | 23:37 |
| jeblair | mtreinish: thanks! | 23:37 |
| jeblair | clarkb: reduced to: set {paramiko, ecdsa} | 23:37 |
| nibalizer | asselin: i kindof agree, doing things one at a time allows us to evaluate if that project should exist at all, or if it should be snuckerd off into openstack_project somewhere | 23:37 |
| clarkb | jeblair: fungi jhesketh mtreinish dns is all done we should be round robining across fe01 and fe02 within 5 minutes (the ttl on those records) | 23:37 |
| clarkb | I am going to push a cacti change for them now | 23:38 |
| krtaylor | asselin, nice, big bang approach | 23:38 |
| krtaylor | nibalizer, each will still require a patch, is that enough for evaluation? | 23:38 |
| nibalizer | krtaylor: exactly | 23:39 |
| nibalizer | when it comes to that module and its time to split it out | 23:39 |
| grue_pm | morning nibalizer - do you ever sleep? | 23:39 |
| *** Sukhdev_ has quit IRC | 23:39 | |
| nibalizer | if the 'split out' is 'refactor it to not exist' then we can propose a patch to project-config that undoes a section of asselin's big patch | 23:39 |
| nibalizer | works for me | 23:39 |
| nibalizer | grue_pm: i sleep lots! | 23:39 |
| asselin | nibalizer, ok, so if it's decided to *not* split out, then we can remove those portions of the file | 23:39 |
| asselin | nibalizer, right | 23:39 |
| *** shakamunyi has quit IRC | 23:39 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Add new git haproxy frontends to cacti https://review.openstack.org/140526 | 23:40 |
| clarkb | jhesketh: fungi jeblair ^ | 23:40 |
| krtaylor | nibalizer, asselin, lgtm | 23:40 |
| *** salv-orlando has quit IRC | 23:40 | |
| asselin | krtaylor, nibalizer I set it to WIP. After you review and fix anything obviously wrong, I'll remove it. | 23:40 |
| *** kumartin has joined #openstack-infra | 23:41 | |
| asselin | krtaylor, nibalizer (and fix the jenkins -1 issues) | 23:41 |
| *** camunoz_gone is now known as camunoz | 23:43 | |
| jeblair | asselin: lgtm, i think it'll work. | 23:43 |
| clarkb | sc68cal: was that the only failure? | 23:43 |
| clarkb | sc68cal: I think that is pretty good if so | 23:43 |
| *** dims has joined #openstack-infra | 23:43 | |
| clarkb | krtaylor: did you see my comment about IBM continuing to use our old git.o.o host | 23:43 |
| sc68cal | clarkb: no I'm seeing the same failure in the check-tempest-dsvm-neutron-full | 23:44 |
| clarkb | krtaylor: we are switching to a two node round robin so you will likely want to make sure that you are talking to those | 23:44 |
| clarkb | sc68cal: but its only that one tempest test? | 23:44 |
| sc68cal | clarkb: that section - I think I may have a fix - https://review.openstack.org/140528 | 23:44 |
| anteaya | jogo: okay thanks | 23:44 |
| clarkb | krtaylor: I can keep the old host around long enough for you to move if we move in the near future. But I don't want to wait too long because holidas and stuff | 23:44 |
| *** shakamunyi has joined #openstack-infra | 23:45 | |
| clarkb | sc68cal: hrm might be better to make it add 1s up for each ip version | 23:46 |
| * clarkb awaits interplanetary IP | 23:46 | |
| *** bswartz has joined #openstack-infra | 23:46 | |
| clarkb | sc68cal: but that way you don't have to dualstack when ipv6ing to get tempest working | 23:46 |
| mtreinish | clarkb: the issue is tempest doesn't know whether you're dual stacking or not | 23:46 |
| mtreinish | it only has access to the apis which I don't think can tell you that | 23:47 |
| mtreinish | sc68cal: it looks like a couple scenario tests fail too | 23:47 |
| clarkb | mtreinish: arg | 23:47 |
| clarkb | we should fix that | 23:47 |
| *** MaxV has quit IRC | 23:47 | |
| mtreinish | clarkb: heh, add it to the list of how the openstack apis are broken... | 23:48 |
| krtaylor | clarkb, thanks, I will communicate that to the IBM teams immediately | 23:49 |
| mtreinish | sc68cal: but I agree with clarkb's comment that patch will probably fail in a ipv6 only env | 23:49 |
| jogo | anteaya: thanks | 23:49 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config: Adds experimental job for testing Nova V2.1 API https://review.openstack.org/138951 | 23:49 |
| *** mestery has quit IRC | 23:50 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: rename subnode log directory https://review.openstack.org/140529 | 23:51 |
| anteaya | jogo: thank you | 23:52 |
| clarkb | ok git.o.o as a round robin looks happy | 23:52 |
| clarkb | no jobs falling over | 23:52 |
| *** Hefeweizen has quit IRC | 23:52 | |
| sc68cal | mtreinish: yeah - we're in a bind here, depending on env will determine fixed_ip count | 23:52 |
| clarkb | jeblair: any chance you can review https://review.openstack.org/#/c/140526/1/modules/openstack_project/manifests/cacti.pp before tomorrow so that we have data with the next round of nodepool image builds? | 23:53 |
| *** dimtruck is now known as zz_dimtruck | 23:53 | |
| nibalizer | clarkb: neato | 23:53 |
| clarkb | sc68cal: mtreinish: I just find it amazing that you can't ask your network gear how many ips you should be getting | 23:53 |
| clarkb | sc68cal: mtreinish: what if we we do this | 23:53 |
| clarkb | sc68cal: mtreinish: test checks that you have at least one of ipv4/ipv6 and having one of each is also ok | 23:54 |
| *** shakamunyi has quit IRC | 23:54 | |
| anteaya | who wants to take point on the names of the proposed puppet modules | 23:54 |
| *** esker has joined #openstack-infra | 23:54 | |
| anteaya | as in hypens and underscores in the name | 23:54 |
| jeblair | clarkb: aprvd | 23:55 |
| clarkb | awesome thanks | 23:55 |
| sc68cal | clarkb: ok - i'll try that after test returns on the tempest one | 23:56 |
| *** harlowja has quit IRC | 23:56 | |
| sc68cal | just realized RAX, you may get more than 1 ipv6 addr | 23:56 |
| *** salv-orlando has joined #openstack-infra | 23:57 | |
| JayF | sc68cal: I don't think you should get multiple v6 that are globally scoped? | 23:57 |
| JayF | sc68cal: like most you'll get v6 link-local scope and a global scope c6 | 23:58 |
| JayF | *v6 | 23:58 |
| openstackgerrit | Merged openstack-infra/project-config: Make aiopcpu not use config drive https://review.openstack.org/139167 | 23:58 |
| *** armax has joined #openstack-infra | 23:59 | |
| *** esker has quit IRC | 23:59 | |
| *** mestery has joined #openstack-infra | 23:59 | |
| *** koolhead17 has joined #openstack-infra | 23:59 | |
| *** koolhead17 has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!