| *** tjones has joined #openstack-infra | 00:03 | |
| *** hemna_ is now known as hemnafk | 00:11 | |
| *** rockyg has quit IRC | 00:11 | |
| *** rockyg has joined #openstack-infra | 00:14 | |
| *** melwitt has quit IRC | 00:22 | |
| *** dkliban has quit IRC | 00:25 | |
| *** blamar has quit IRC | 00:28 | |
| *** julim has joined #openstack-infra | 00:41 | |
| *** CaptTofu has joined #openstack-infra | 00:42 | |
| *** CaptTofu has quit IRC | 00:47 | |
| *** reed has quit IRC | 00:47 | |
| *** tjones has quit IRC | 00:54 | |
| *** CaptTofu has joined #openstack-infra | 00:56 | |
| *** dims has quit IRC | 01:08 | |
| *** nati_ueno has quit IRC | 01:10 | |
| *** rcleere has joined #openstack-infra | 01:12 | |
| *** rcleere has quit IRC | 01:15 | |
| *** senk has joined #openstack-infra | 01:16 | |
| *** ericw has quit IRC | 01:22 | |
| *** ericw has joined #openstack-infra | 01:22 | |
| jog0 | wow with tempest being so flaky its amazing we can merge anything at all | 01:22 |
|---|---|---|
| *** dims has joined #openstack-infra | 01:23 | |
| clarkb | jog0: the gif of the toddler trying to get the block on the bucket and eventually giving up and taking the cover off comes to mind for some reason | 01:36 |
| clarkb | Alex_Gaynor: mordred I have realized that you guys twitter a lot | 01:36 |
| mordred | clarkb: we do | 01:36 |
| clarkb | also, its like IRC where people don't need to subscribe to the channel to hear what you have to say. Seems dangerous :) | 01:37 |
| *** ericw has quit IRC | 01:39 | |
| clarkb | pleia2: you too | 01:41 |
| clarkb | jeblair: fungi: we need a support group or something | 01:41 |
| pleia2 | clarkb: yeah, twitter is multicast irc | 01:43 |
| lifeless | unlike regular irc | 01:43 |
| pleia2 | and I use bitlbee to read twitter feed and often tweet, so it's in irssi too ;) | 01:43 |
| clarkb | IRC has the benefit of allowing ~500 characters >_> | 01:45 |
| lifeless | and hyperlinks | 01:45 |
| lifeless | mordred: btw http://www.technicolor.com/en/solutions-services/connected-home/modems-gateways/xdsl-modems-gateways | 01:47 |
| mordred | lifeless: bonghits | 01:48 |
| clarkb | is someone in the market for a dsl modem? | 01:48 |
| *** senk has quit IRC | 01:48 | |
| lifeless | mordred: I know right | 01:48 |
| lifeless | clarkb: I upgraded to vDSL recently | 01:48 |
| lifeless | clarkb: 10Mbps up 50Mbps down | 01:49 |
| clarkb | nice | 01:49 |
| lifeless | clarkb: but the ISP shipped a technicolor modem | 01:49 |
| lifeless | clarkb: Technicolor, known for dsl modems? ... not really. | 01:49 |
| *** davidhadas_ has quit IRC | 01:49 | |
| clarkb | http://www.versatek.com/vx-veb165-long-reach-ip-vdsl2-modem.html is what I have. seems to mostly work | 01:50 |
| *** davidhadas_ has joined #openstack-infra | 01:50 | |
| * mordred has fios | 01:50 | |
| lifeless | the technicolor stuff is shite | 01:50 |
| clarkb | mordred: I bet my vdsl has better bandwidth than your fios :) | 01:51 |
| lifeless | mordred: have you seen """ File "/opt/stack/venvs/python-glanceclient/local/lib/python2.7/site-packages/pip/backwardcompat/__init__.py", line 90, in fwrite | 01:51 |
| clarkb | I get about 70/25mbps out of it | 01:51 |
| lifeless | f.write(s) | 01:51 |
| lifeless | ValueError: I/O operation on closed file | 01:51 |
| lifeless | """ | 01:51 |
| lifeless | from pip | 01:51 |
| lifeless | clarkb: I'm not talking speed, rather quality. | 01:51 |
| lifeless | clarkb: not linux, has to reboot to disable dhcp on an interface | 01:51 |
| mordred | lifeless: yes. but I do not remember what causes it | 01:51 |
| lifeless | haphazard UI | 01:51 |
| mordred | dstufft: do you remember ^^ ? | 01:51 |
| mordred | dstufft: I remember we both looked at that liek a month ago or so... | 01:52 |
| lifeless | hmm 0.2% packet loss out of the freecloud dc | 01:52 |
| lifeless | the first modem they gave me ran for 24 hours then started rebooting in a loop | 01:53 |
| dstufft | mordred: I don't remember | 01:55 |
| dstufft | clarkb: mordred I also have fios, I get like 60/25 and if I felt like paying like $70 more a month I could get 150/35 | 01:56 |
| dstufft | or something like that | 01:56 |
| dstufft | I think I can get up to 300 :V | 01:56 |
| *** schwicht has quit IRC | 01:57 | |
| clarkb | yeah I had fios in portlandish | 01:57 |
| clarkb | it was good too. I have also had comcast which was not fun | 01:57 |
| dstufft | we used to have comcast | 01:58 |
| dstufft | it was terribad | 01:58 |
| pleia2 | I had fios in PA, but back to comcast (but it's ok, I have a backup connection too) | 01:58 |
| clarkb | my favorite thing was that they knew when I needed it and would always have an outage. Once I did a 5am remote course thing and of course outage | 01:58 |
| clarkb | not in any way prepared for debugging connectivity issues at 5am | 01:59 |
| clarkb | pleia2: your condo is multihomed? | 01:59 |
| *** lifeless_ has joined #openstack-infra | 02:00 | |
| *** lifeless has quit IRC | 02:00 | |
| clarkb | I think pleia2 wins this competition | 02:00 |
| pleia2 | clarkb: yeah, we have p2p microwave that goes to the building | 02:00 |
| pleia2 | 100M for the whole building, so not spectacular but it's not bad | 02:01 |
| pleia2 | especially good during the day when no one else is home :) | 02:01 |
| *** lifeless has joined #openstack-infra | 02:06 | |
| *** lifeless_ has quit IRC | 02:07 | |
| clarkb | dstufft: haven't gotten twine into place yet, but should soon. very happy twine is a thing | 02:07 |
| dstufft | clarkb: :D | 02:08 |
| dstufft | clarkb: lemme know how it goes | 02:08 |
| dstufft | I plan to make a new upload API for pypi evnetually | 02:08 |
| dstufft | and will update twine for it too | 02:08 |
| clarkb | the change to use it conflicts with another change we want to get in, so fungi and I need to sort out how we want to get those in | 02:08 |
| dstufft | it conflicts? | 02:08 |
| clarkb | dstufft: git conflicts | 02:09 |
| dstufft | ahh | 02:09 |
| dstufft | gotcha | 02:09 |
| clarkb | but nevermind fungi's change merged so I get to rebase \o/ | 02:09 |
| clarkb | which is good because now I know which direction we want to order them | 02:09 |
| *** notmyname_ has joined #openstack-infra | 02:15 | |
| clarkb | dstufft: https://review.openstack.org/#/c/48507/ | 02:15 |
| *** cyeoh_ has joined #openstack-infra | 02:16 | |
| dstufft | +1'd | 02:17 |
| *** notmyname has quit IRC | 02:17 | |
| *** cyeoh has quit IRC | 02:17 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Use twine to upload python package to pypi. https://review.openstack.org/48507 | 02:17 |
| *** HenryG has quit IRC | 02:17 | |
| *** notmyname_ is now known as notmyname | 02:17 | |
| clarkb | dstufft: thanks | 02:18 |
| dstufft | clarkb: np! | 02:19 |
| *** tizzo has joined #openstack-infra | 02:21 | |
| clarkb | just found the thing on raw_input in setup.py and people wonder why pbr exists :) | 02:26 |
| justinabrahms | Do you guys use a fork of gerrit? | 02:31 |
| justinabrahms | or just configure it differently? | 02:31 |
| mordred | clarkb: :) | 02:31 |
| mordred | dstufft: which is the pip that no-one should ever run earlier than due to security? | 02:31 |
| mordred | justinabrahms: both | 02:32 |
| clarkb | justinabrahms: it is a slight fork unfortunately. But we are very close to being off of the fork | 02:32 |
| mordred | justinabrahms: we're working on re-aligning with upstream though | 02:32 |
| mordred | what clarkb said | 02:32 |
| justinabrahms | What's the main reason for the fork? | 02:32 |
| clarkb | justinabrahms: the only major things in the fork that you would probably notice are the work in progress feature and a second default dashboard | 02:32 |
| dstufft | mordred: anything older than 1.3 oesn't verify SSL | 02:32 |
| dstufft | mordred: anything older than 1.4 doesn't give you the option to not download shit from random hosts | 02:33 |
| clarkb | justinabrahms: the main reason is that gerrit didn't want our work in progress feature. But now gerrit has plugins that can do that for us and _david_ is writing one so win | 02:33 |
| dstufft | mordred: anything older than 1.5 (not relesased) doesn't not download random shit by defailt | 02:33 |
| dstufft | mordred: so take your pick :) | 02:33 |
| justinabrahms | I'm going to be setting up our gerrit instance with chef (currently have a mostly vanilla setup at this point) and looking to mimic some of the things you guys have, just trying to get a sense of what is different. :) | 02:33 |
| clarkb | justinabrahms: upstream adopted our default css stuff so look and feel should be similar from upstream | 02:34 |
| justinabrahms | mostly trying to ape the: CI must approve, push result to github aspects. | 02:34 |
| clarkb | justinabrahms: we also add at least one extra review category (approval) but that is configuration not code | 02:34 |
| mordred | justinabrahms: fwiw (not trying to get into a chef v. puppet debate or anything- they're both fine) but we do have a completely puppet repo with all of our puppet | 02:35 |
| clarkb | justinabrahms: we use three review categories. verified which is where CI systems vote and has ranges -2 to +2, code review which is where humans can vote, we allow anyone to +/-1 and core reviewers to +/-2. And lastly approved which is +1 only | 02:35 |
| justinabrahms | mordred: understood. If I didn't already have a chef instance, I'd just use what you have. | 02:35 |
| mordred | as well as docs - so please feel free to steal/crib/borrow whatever is useful! | 02:35 |
| mordred | http://ci.openstack.org/ in case you haven't found it | 02:35 |
| mordred | we've also got a bunch of management scripts to do things like manage gerrit projects via yaml files - and of course there is zuul :) | 02:36 |
| morganfainberg | mordred, i so want to crib / use more of it! but not yet :( | 02:36 |
| clarkb | typically workflow is as soon as you submit code Jenkins et al test it and vote +/-1 based on the results. Humans then review it and when you have two +2 votes from humans reviewers can hit the +1 approved vote which puts things in the gate queue | 02:36 |
| mordred | happy to have you as part of the gerrit family! | 02:36 |
| dstufft | fwiw python-dev is discussing PEP453 (pep bootstrap) a few folks have expressed concern over the 2.7 backport, so if you feel strongly about it one way or the other you might want to weigh in | 02:36 |
| clarkb | the gate queue tests that change as it would merge if it passes tests and if tests pass you get merged | 02:36 |
| dstufft | pip bootstrap* | 02:36 |
| clarkb | oh yes manage_projects | 02:37 |
| morganfainberg | mordred, any idea when/if we'll be upgrading to a later version of gerrit? i want some of the OS niceityies (trivial rebase, WIP) but i made the mistake of going with a later version early on | 02:37 |
| morganfainberg | and i'm concerned about trying to downgrade. | 02:37 |
| mordred | morganfainberg: soonish. we're actively working on it | 02:37 |
| clarkb | mordred: https://review.openstack.org/#/c/48312/ and https://review.openstack.org/#/c/48314/ make manage_projects better | 02:38 |
| morganfainberg | mordred, awesome. that will make me quite happy. | 02:38 |
| mordred | morganfainberg: between _david_'s new plugin and zaro's patches I think we'l l get somewhere soon | 02:38 |
| morganfainberg | (and likely my boss too) | 02:38 |
| mordred | clarkb: neat! | 02:38 |
| justinabrahms | clarkb: is it the case that one of those +2s has to be from jenkins? Or can 2 core humans +2 and get it into zuul? | 02:38 |
| clarkb | morganfainberg: _david_ (works on gerrit and libreoffice iirc) has a WIP gerrit plugin in the works and zaro has a gerrit patch to extend the gerrit ACL system to make it useable for us | 02:38 |
| morganfainberg | and when i have time… new zuul for internal work..but thats currently a beast. | 02:38 |
| justinabrahms | also, if zuul fails, does it reopen the review? | 02:38 |
| morganfainberg | (for me) | 02:38 |
| morganfainberg | clarkb, awesome. i'll go look for it | 02:39 |
| mordred | justinabrahms: jenkins only votes in the verified column | 02:39 |
| morganfainberg | the ACL patch that is | 02:39 |
| clarkb | justinabrahms: jenkins must vote +2 in the verified column so you need one +2 in verified and two +2s in code review (though only one +2 in code review is enforced) | 02:39 |
| mordred | what clarkb said | 02:39 |
| clarkb | justinabrahms: also if you just want to fiddle with gerrit sans the super fancy CI stuff you can push to review-dev.openstack.org | 02:39 |
| mordred | justinabrahms: and the code reviews don't close until jenkins/zuul close them - so, yes, if the test fail, a negative verified vote is posted | 02:39 |
| justinabrahms | got it. | 02:40 |
| clarkb | also you want to enable ssh so that git-review will work (it doesn't support pushing via https yet) | 02:41 |
| justinabrahms | To you guys support a "To Be Reviewed" status? | 02:42 |
| justinabrahms | ie: I need to fix this prod bug right now, please submit a changeset for later review? | 02:42 |
| justinabrahms | (Google had that, but wasn't sure if you guys did) | 02:42 |
| clarkb | morganfainberg: I think usptream new gerrit can do trivial rebase or cherry picking | 02:42 |
| morganfainberg | clarkb, i am fair certain cherry pick yes, trvial rebase detection wasn't there. | 02:43 |
| morganfainberg | or well.. trvial rebase issues | 02:43 |
| morganfainberg | i might be one rev behind though at this point. | 02:43 |
| clarkb | morganfainberg: oh trivial rebase detection. that is just a hook iirc you should be able to do it with your gerrit | 02:43 |
| morganfainberg | so, might have progressed | 02:43 |
| clarkb | justinabrahms: is that something that gets merged now and reviewed later? (trying to understand the workflow) | 02:43 |
| morganfainberg | clarkb, yeah, i've loathed to try and patch gerrit though (I'm not a java dev and each time i try i get … lets just say bad results). I figured i'd wait for OS to catch up to where I am and move over and then stay in lock-step | 02:44 |
| justinabrahms | clarkb: correct. | 02:44 |
| justinabrahms | it was for things like: "I need to fix this at 3am. We can review it in the morning." | 02:44 |
| mordred | justinabrahms: we do not support that concept here | 02:44 |
| mordred | I have no idea if gerrit itself can do it | 02:45 |
| justinabrahms | I think that's fine. If it were massively important, presumably you could circumvent the review state all together and manually go over changes. | 02:45 |
| mordred | yup | 02:45 |
| morganfainberg | justinabrahms, I am unsure how well that would work. you could get into a strange bind if things got based on that patch.. or piled on. | 02:45 |
| mordred | although we don't give those acls to any of our devs | 02:45 |
| mordred | but yeah | 02:46 |
| clarkb | mordred: http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/trivial_rebase.py run it as a patchset created hook | 02:46 |
| clarkb | gah morganfainberg ^ | 02:46 |
| morganfainberg | clarkb, heheeh took me a while to get the keystone people to type 4 characters :P | 02:46 |
| clarkb | justinabrahms: also check out jeepyb, it is a collection of tools we built around gerrit. Gerrit Project Builder -> GPB -> jeepyb | 02:47 |
| clarkb | justinabrahms: it isn't perfect though, it unfortunately encodes a bunch of what we do, but patches are welcome (and mordred has been trying to genericize it too) | 02:47 |
| dstufft | mordred: clarkb btw we found the problem that was causing thrift to timeout | 02:48 |
| morganfainberg | clarkb, wish i could be 100% dedicated to writing openstack code (and contribute to infra/qa more). maybe when i have more staff here i can do that :) | 02:48 |
| dstufft | the thrift install | 02:48 |
| dstufft | it'll take awhile for a deploy to fix it to be fully deployed though :[ | 02:49 |
| clarkb | morganfainberg: I wish I had more time to do the random housekeeping nice to have stuff | 02:50 |
| justinabrahms | Yea, I think it makes less sense for infrastructural things like openstack, but might make more sense for an application. | 02:50 |
| justinabrahms | Looking over this stuff, there are a few things I think I'll need. generate project on disk from chef-like location, shuffle commits to github (I think jeepyb has this?), notify jenkins that a thing needs to be tested, wire jenkins to let gerrit know how to deal with things.. that may be it? | 02:52 |
| clarkb | shuffling commits to github is done with gerrit's replication feature (now in a plugin), jeepyb creates projects in gerrit based on config and an optional location. wiring jenkins is good and can be done with the gerrit jenkins plugin or zuul. And git-review. I really think general opinion of gerrit would be better if git review were used more :) | 02:54 |
| justinabrahms | (as background, this will be for a closed-source project that will be used by ~6 engineers) | 02:54 |
| lifeless | https://github.com/pypa/pip/issues/219 clarkb mordred | 02:55 |
| lifeless | looks suspiciously like it, in that it's coming out of 'log.py' | 02:55 |
| justinabrahms | I think I might punt on a lot of the ACL stuff for now (at least until we migrate our open source tools over) | 02:55 |
| clarkb | justinabrahms: I think it is common to go about it in steps. Don't need everything all at once for it to be useful | 02:56 |
| clarkb | lifeless: yes looks like the thing. The last stacktrace especially (I didn't look closely at the others) | 02:57 |
| clarkb | I really need to make time to go give the infra overview talk in portland | 02:57 |
| clarkb | justinabrahms: eventually you end up with http://status.openstack.org/zuul/ | 02:58 |
| *** markmcclain1 has quit IRC | 02:59 | |
| *** basha has joined #openstack-infra | 02:59 | |
| mordred | dstufft: any chance you know when pip 1.4.2 is coming out? | 03:00 |
| dstufft | mordred: no set date, whenever one of us proposes it with no arguments from the rest I gues | 03:01 |
| mordred | dstufft: I think someone from tripleo is desperately waiting for it | 03:01 |
| mordred | because of the fix to the on-disk permissions on install | 03:01 |
| mordred | and also there's a patch that landed 8 days ago that might make lifeless happy | 03:01 |
| *** basha has quit IRC | 03:02 | |
| dstufft | mordred: is it in the 1.4.2 milestone? | 03:02 |
| mordred | dstufft: one is this: https://github.com/pypa/pip/commit/eee40f9dc6f650b17c5d7c46450f5e6e9a6e56f9 | 03:02 |
| mordred | that's the lifeless one | 03:02 |
| mordred | dstufft: he's hitting that bug right now | 03:02 |
| mordred | the other was merged a WHILE ago | 03:03 |
| mordred | this one: https://github.com/pypa/pip/issues/1133 | 03:04 |
| dstufft | mordred: the one you linked looks like it's only on the develop branch so it'd need backported but that's not a big deal | 03:04 |
| mordred | is the one that will make tripleo guys happy | 03:04 |
| mordred | which seems to be in 1.4.2 | 03:04 |
| dstufft | mordred: can you send me an email (donald@stufft.io) with any tickets you want in 1.4.2 and i'll see about getting them backported to 1.4.x if they aren't and propose making a release | 03:05 |
| dstufft | done or not | 03:05 |
| dstufft | just so I don't forget^ | 03:05 |
| mordred | dstufft: yah. thanks! | 03:05 |
| lifeless | dstufft: mordred: it *might* be different | 03:05 |
| lifeless | but it's super repeatable | 03:05 |
| dstufft | lifeless: test it with develop and see if it happens? | 03:06 |
| lifeless | dstufft: it's deep in dib | 03:06 |
| dstufft | oh | 03:06 |
| lifeless | dstufft: on a remote datacentre machine with packet loss | 03:06 |
| lifeless | dstufft: not an idea develop environment | 03:06 |
| mordred | lifeless: sudo pip install -e git://github.com/pypa/pip ?? :) | 03:10 |
| *** basha has joined #openstack-infra | 03:11 | |
| clarkb | mordred: what heppens when pip breaks itsefl? pipception? | 03:11 |
| *** basha has quit IRC | 03:11 | |
| dstufft | clarkb: I've done that | 03:12 |
| dstufft | if it's not in a venv it's terrible | 03:13 |
| dstufft | there's a reason why a goal of mine is to make pip bundle or not depend on setuptool | 03:13 |
| mordred | clarkb: what happens is you learn a lot more than you knew before about installing and uninstalling python packages | 03:14 |
| *** mrodden1 has joined #openstack-infra | 03:14 | |
| lifeless | huh | 03:14 |
| lifeless | I added debug output to source-repositories. | 03:14 |
| lifeless | ah no, it did not fix it :) | 03:14 |
| *** jerryz has quit IRC | 03:14 | |
| lifeless | now why, with the pypi mirror element in place | 03:15 |
| lifeless | is Downloading/unpacking jsonschema>=0.7,<3 (from warlock>=1.0.1,<2->-r requirements.txt (line 7)) | 03:15 |
| *** mestery_ has joined #openstack-infra | 03:15 | |
| lifeless | Error <urlopen error [Errno 101] Network is unreachable> while getting http://pypi.openstack.org/openstack/jsonschema/jsonschema-2.1.0.tar.gz (from http://pypi.openstack.org/openstack/jsonschema/) | 03:15 |
| lifeless | happening | 03:15 |
| dstufft | http:// | 03:16 |
| dstufft | wat | 03:16 |
| dstufft | how old is your pip | 03:16 |
| * mordred hangs head | 03:16 | |
| *** mrodden has quit IRC | 03:16 | |
| *** mestery has quit IRC | 03:16 | |
| lifeless | dstufft: I'm expecting file:// here. | 03:16 |
| mordred | we still don't have https on our mirror | 03:16 |
| lifeless | dstufft: there is a local mirror. | 03:16 |
| dstufft | oh dur | 03:16 |
| lifeless | dstufft: so tis is astonishing me. | 03:16 |
| dstufft | i didn't read the whole thing | 03:16 |
| mordred | lifeless: how do you have the local mirror configured? | 03:16 |
| dstufft | I just say http://pypi | 03:16 |
| dstufft | and was like wat | 03:16 |
| mordred | hehe | 03:16 |
| dstufft | ssw* | 03:17 |
| dstufft | saw* | 03:17 |
| dstufft | typing is hard | 03:17 |
| dstufft | good thing I don't type for a living | 03:17 |
| mordred | right? | 03:17 |
| mordred | lifeless: can you point me at source code? | 03:17 |
| lifeless | mordred: sure, tripleo-image-elements/elements/pypi | 03:18 |
| lifeless | ahhhh found it | 03:18 |
| lifeless | PIP_INDEX_URL=http://pypi.openstack.org/openstack/ <- naughty script | 03:18 |
| mordred | ooh. that is a naughty script | 03:19 |
| lifeless | I missed that in elements/openstack-clients/install.d/51-openstack-clients | 03:19 |
| lifeless | mordred: but this should hint to you how shoddy the networking I'm getting is | 03:19 |
| lifeless | mordred: pypi.openstack.org was failing consistently... | 03:19 |
| mordred | lifeless: that's impressive | 03:21 |
| mordred | we should CDN pypi.openstack.org | 03:22 |
| clarkb | dstufft: the mirror was born prior to pypi.python.org https so we figured there wasn't much reason for https. We should probably rethink that | 03:22 |
| dstufft | clarkb: +1 | 03:23 |
| dstufft | in pip 1.5 it'll start printing messages | 03:23 |
| dstufft | telling you it's insecure | 03:23 |
| *** CaptTofu has quit IRC | 03:24 | |
| clarkb | it will help fix several other issues too | 03:24 |
| clarkb | within our CI systems | 03:24 |
| lifeless | other fun stuff today | 03:24 |
| lifeless | was finding out that you can't change dhcp allocation pools in neutron | 03:25 |
| lifeless | it's not 'in the spec' | 03:25 |
| lifeless | https://bugs.launchpad.net/neutron/+bug/1111572 | 03:25 |
| uvirtbot | Launchpad bug 1111572 in neutron "quantum subnet-update can't update allocation-pool " [High,Invalid] | 03:25 |
| lifeless | bbiaw | 03:26 |
| clarkb | lifeless: can you delete create? | 03:27 |
| clarkb | the other ip address management tool I have used in the past required that you delete create, was annoying | 03:27 |
| clarkb | but worked | 03:27 |
| mordred | clarkb: I think we don't want https until we figure out a way to get lifeless rsync mirroring | 03:29 |
| mordred | clarkb: because otherwise we'll break his squid | 03:29 |
| clarkb | mordred: we can do both | 03:30 |
| mordred | yah | 03:30 |
| mordred | I'm just saying, we should do both or neither | 03:30 |
| mordred | not just one | 03:30 |
| clarkb | ++ | 03:30 |
| lifeless | clarkb: not when it's in use | 03:30 |
| *** ericw has joined #openstack-infra | 03:34 | |
| *** rcleere has joined #openstack-infra | 03:34 | |
| *** basha has joined #openstack-infra | 03:43 | |
| *** pabelanger has quit IRC | 03:44 | |
| *** yonglihe_ has joined #openstack-infra | 03:45 | |
| *** yongli has quit IRC | 03:45 | |
| *** tian has quit IRC | 03:45 | |
| *** tian has joined #openstack-infra | 03:45 | |
| *** ryanpetrello has joined #openstack-infra | 03:48 | |
| *** pabelanger has joined #openstack-infra | 03:49 | |
| mordred | ok. jenkins and jenkins01 restarted | 03:59 |
| *** ryanpetrello has quit IRC | 04:10 | |
| *** tizzo has quit IRC | 04:15 | |
| *** ryanpetrello has joined #openstack-infra | 04:16 | |
| jog0 | wow I go away for a few hours and gate queue goes down by 2 | 04:17 |
| clarkb | progress >_> | 04:17 |
| jog0 | clarkb: this is what it reminds me of https://www.youtube.com/watch?v=0Au_8GMUxVs | 04:19 |
| *** nati_ueno has joined #openstack-infra | 04:22 | |
| *** rcleere has quit IRC | 04:23 | |
| *** ericw has quit IRC | 04:30 | |
| *** sarob has joined #openstack-infra | 04:49 | |
| *** Ryan_Lane has quit IRC | 05:00 | |
| *** HenryG has joined #openstack-infra | 05:09 | |
| *** cody-somerville has quit IRC | 05:13 | |
| *** AlexF has joined #openstack-infra | 05:15 | |
| *** SlickNik_ has joined #openstack-infra | 05:20 | |
| *** Ryan_Lane has joined #openstack-infra | 05:31 | |
| *** sarob has quit IRC | 05:34 | |
| *** sarob has joined #openstack-infra | 05:34 | |
| *** Ryan_Lane has quit IRC | 05:39 | |
| *** sarob has quit IRC | 05:39 | |
| *** SlickNik_ has quit IRC | 05:42 | |
| *** AlexF has quit IRC | 05:44 | |
| *** tjones has joined #openstack-infra | 05:45 | |
| *** ryanpetrello has quit IRC | 05:45 | |
| *** tjones has quit IRC | 05:49 | |
| *** HenryG has quit IRC | 06:02 | |
| *** Ryan_Lane has joined #openstack-infra | 06:07 | |
| *** Ryan_Lane has quit IRC | 06:11 | |
| *** SergeyLukjanov has joined #openstack-infra | 06:16 | |
| *** Ryan_Lane has joined #openstack-infra | 06:37 | |
| *** Ryan_Lane has quit IRC | 06:41 | |
| *** fbo is now known as fbo_away | 06:45 | |
| *** sarob has joined #openstack-infra | 06:45 | |
| *** sarob has quit IRC | 06:50 | |
| *** dmakogon_ has joined #openstack-infra | 06:54 | |
| *** Ryan_Lane has joined #openstack-infra | 07:07 | |
| *** SpamapS has quit IRC | 07:08 | |
| *** Ryan_Lane has quit IRC | 07:11 | |
| mordred | jog0: woot! | 07:24 |
| *** fbo_away is now known as fbo | 07:30 | |
| *** SergeyLukjanov has quit IRC | 07:31 | |
| *** alexpilotti has joined #openstack-infra | 07:34 | |
| *** Ryan_Lane has joined #openstack-infra | 07:38 | |
| *** nati_ueno has quit IRC | 07:38 | |
| *** Ryan_Lane has quit IRC | 07:43 | |
| *** nati_ueno has joined #openstack-infra | 07:51 | |
| *** alexpilotti has quit IRC | 07:52 | |
| *** Ryan_Lane has joined #openstack-infra | 08:09 | |
| *** dguitarbite has joined #openstack-infra | 08:11 | |
| *** Ryan_Lane has quit IRC | 08:13 | |
| *** nati_ueno has quit IRC | 08:18 | |
| *** Ryan_Lane has joined #openstack-infra | 08:39 | |
| *** Ryan_Lane has quit IRC | 08:43 | |
| *** Ryan_Lane has joined #openstack-infra | 09:10 | |
| *** Ryan_Lane has quit IRC | 09:14 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack/requirements: Revert "Describe SQLAlchemy constraint more clearly." https://review.openstack.org/48779 | 09:16 |
| *** tizzo has joined #openstack-infra | 09:20 | |
| *** tizzo has quit IRC | 09:25 | |
| *** AlexF has joined #openstack-infra | 09:27 | |
| *** tvb has joined #openstack-infra | 09:32 | |
| *** tizzo has joined #openstack-infra | 09:33 | |
| *** Ryan_Lane has joined #openstack-infra | 09:40 | |
| *** Ryan_Lane has quit IRC | 09:44 | |
| *** AlexF has quit IRC | 09:45 | |
| *** dguitarbite has quit IRC | 09:58 | |
| *** boris-42 has joined #openstack-infra | 09:59 | |
| *** SlickNik_ has joined #openstack-infra | 10:01 | |
| *** dmakogon_ has quit IRC | 10:03 | |
| *** SlickNik_ has quit IRC | 10:05 | |
| *** Ryan_Lane has joined #openstack-infra | 10:11 | |
| *** Ryan_Lane has quit IRC | 10:15 | |
| *** jtzl has quit IRC | 10:17 | |
| *** jtzl has joined #openstack-infra | 10:17 | |
| *** SpamapS has joined #openstack-infra | 10:25 | |
| *** mkirk_ has quit IRC | 10:39 | |
| *** mkirk has joined #openstack-infra | 10:40 | |
| *** Ryan_Lane has joined #openstack-infra | 10:41 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files https://review.openstack.org/48783 | 10:44 |
| *** tvb has quit IRC | 10:48 | |
| *** Ryan_Lane has quit IRC | 10:50 | |
| *** tvb has joined #openstack-infra | 10:56 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: start voting on neutron-pg job https://review.openstack.org/48784 | 11:00 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add neutron-pg testing to neutronclient https://review.openstack.org/48785 | 11:00 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: double jobs on neutron to prevent races from landing https://review.openstack.org/48786 | 11:00 |
| *** CaptTofu has joined #openstack-infra | 11:06 | |
| *** tizzo has quit IRC | 11:06 | |
| *** Ryan_Lane has joined #openstack-infra | 11:16 | |
| *** Ryan_Lane has quit IRC | 11:20 | |
| *** michchap has quit IRC | 11:40 | |
| *** michchap_ has joined #openstack-infra | 11:41 | |
| *** michchap has joined #openstack-infra | 11:41 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: add an explanation to the sqla requirement https://review.openstack.org/48789 | 11:41 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Provide default ConfigParser object https://review.openstack.org/48790 | 11:45 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files https://review.openstack.org/48783 | 11:45 |
| *** michchap_ has quit IRC | 11:45 | |
| *** Ryan_Lane has joined #openstack-infra | 11:47 | |
| dims | so, jgriffith has a review that bumps up num_iscsi_retries to try to get to root cause of https://bugs.launchpad.net/tempest/+bug/1226337 - BUT the problem is check-tempest-devstack-vm-neutron does not run the test we need tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern - Any ideas on how to reproduce the problem? Please leave hints on the https://review.openstack.org/#/c/48752/ review | 11:49 |
| uvirtbot | Launchpad bug 1226337 in tempest "tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern flake failure" [High,Triaged] | 11:49 |
| *** Ryan_Lane has quit IRC | 11:51 | |
| *** CaptTofu has quit IRC | 11:51 | |
| sdague | dims: the fails are triggered on the non neutron jobs | 11:52 |
| *** jhesketh has quit IRC | 11:55 | |
| dims | sdague, thx! i see it now | 11:58 |
| *** schwicht has joined #openstack-infra | 12:03 | |
| *** CaptTofu has joined #openstack-infra | 12:12 | |
| *** Ryan_Lane has joined #openstack-infra | 12:17 | |
| *** Ryan_Lane has quit IRC | 12:21 | |
| guohliu | hello, has that pypersing or cliff issue solved on stable/grizzly? | 12:41 |
| *** CaptTofu has quit IRC | 12:42 | |
| *** Ryan_Lane has joined #openstack-infra | 12:48 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: Revert "Enable q-vpn service" https://review.openstack.org/48793 | 12:52 |
| *** Ryan_Lane has quit IRC | 12:52 | |
| sdague | guohliu: I don't believe so | 12:52 |
| sdague | fungi, jeblair, mordred, clarkb: if any of you guys end up on today - https://review.openstack.org/48793 might get neutron back under control | 12:53 |
| fungi | sdague: thanks for the heads up. i'll try to have a look after breakfast | 12:54 |
| *** tvb|afk has joined #openstack-infra | 12:57 | |
| sdague | fungi: cool, thanks | 12:59 |
| *** tvb has quit IRC | 12:59 | |
| sdague | ok, time to get ready for some bike riding and good tea shop, I may check back in late in the day | 13:04 |
| fungi | have fun | 13:07 |
| guohliu | sdague: ok, thx | 13:10 |
| guohliu | sdague: and we really can't bypass this issue by update that quantumclient requirments temporarily? | 13:12 |
| *** Ryan_Lane has joined #openstack-infra | 13:18 | |
| soren | fungi: fwiw: https://code.launchpad.net/~soren/python-jenkins/add-crumb/+merge/188155 | 13:19 |
| soren | fungi: That'll let us all enable csrf protection without breaking jenkins-job-builder. | 13:20 |
| *** Ryan_Lane has quit IRC | 13:23 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:32 | |
| *** michchap has quit IRC | 13:45 | |
| *** michchap has joined #openstack-infra | 13:45 | |
| *** Protux has quit IRC | 13:49 | |
| *** Ryan_Lane has joined #openstack-infra | 13:49 | |
| *** adalbas has quit IRC | 13:49 | |
| *** michchap_ has joined #openstack-infra | 13:50 | |
| *** Protux has joined #openstack-infra | 13:51 | |
| *** mestery has joined #openstack-infra | 13:53 | |
| *** Ryan_Lane has quit IRC | 13:53 | |
| *** michchap has quit IRC | 13:54 | |
| *** mestery_ has quit IRC | 13:54 | |
| *** dmakogon_ has joined #openstack-infra | 13:58 | |
| *** adalbas has joined #openstack-infra | 14:01 | |
| *** dmakogon_ has quit IRC | 14:02 | |
| *** dguitarbite has joined #openstack-infra | 14:04 | |
| *** adalbas has quit IRC | 14:13 | |
| *** adalbas has joined #openstack-infra | 14:13 | |
| jeblair | sdague: fyi https://review.openstack.org/#/c/48767/ | 14:14 |
| sdague | jeblair: ah, oops. Guess that what happens when it doesn't like to launchpad | 14:17 |
| sdague | jeblair: actually, any idea why Related-Bug didn't link that one? | 14:18 |
| *** Ryan_Lane has joined #openstack-infra | 14:19 | |
| jeblair | sdague: maybe it's still looking at the project->lp mapping and refusing to link to something other than openstack-ci | 14:19 |
| jeblair | sdague: what do you think of the results from salvatore's change? | 14:21 |
| jeblair | i'm confused by the disagreement between jog0's bot and salvatore | 14:22 |
| *** Ryan_Lane has quit IRC | 14:23 | |
| jeblair | clarkb, jog0: i registered the 'openstackrecheck' nick and added the password to hiera | 14:46 |
| jeblair | clarkb: (email address info, etc, in the usual place) | 14:47 |
| jeblair | fungi, mordred: fyi ^ | 14:47 |
| jeblair | fungi, mordred, clarkb: i also restructured our automated system related addresses to all be an alias for one account (info is in the usual place). | 14:48 |
| jeblair | soren: do you need a crumb for every request? or can you get a crumb and reuse it for several requests? | 14:49 |
| jeblair | soren: (i'm wondering about doubling the number of http requests overall) | 14:49 |
| *** Ryan_Lane has joined #openstack-infra | 14:49 | |
| sdague | jeblair: so I think the issue is there are a few different neutron races going on | 14:51 |
| sdague | and as far as I can tell they are all basically related to bad db management. I actually think they are doing what nova used to do which is only send ids on the wire then have the otherside go look things up in the db. | 14:52 |
| *** dguitarbite has quit IRC | 14:53 | |
| *** Ryan_Lane has quit IRC | 14:54 | |
| sdague | yeh, honestly, there are a few different things happening, and I think that the elastic-recheck checks tend to be broader than they really should be | 14:54 |
| jeblair | ok, so the distable qvpn route still seems promising? | 14:54 |
| jeblair | (btw, that explanation does make sense of the logstash observation it happened a few times in the past and then greatly increased in frequency several days ago) | 14:55 |
| sdague | so this is the deadlock history - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkxvY2sgd2FpdCB0aW1lb3V0IGV4Y2VlZGVkOyB0cnkgcmVzdGFydGluZyB0cmFuc2FjdGlvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzgwMzgwMjAwNTA4fQ== | 14:57 |
| *** dizquierdo has joined #openstack-infra | 14:59 | |
| sdague | man, you know what I really want... corrolating the specific fails we get in the gate against all the fails we get in the gate, to know if we've got classifications for all of them | 14:59 |
| sdague | dammit, I need more hours in the day to write code :) | 14:59 |
| sdague | jeblair: also, when you get a chance - https://review.openstack.org/#/c/48787/ is a really important narrowing of that fail | 15:00 |
| sdague | and points us back to a keystoneclient issue | 15:00 |
| sdague | so it would be good to +2 it into elastic-recheck | 15:01 |
| *** AlexF has joined #openstack-infra | 15:02 | |
| sdague | ok... biking... later all | 15:04 |
| jeblair | we need to make that a yaml file so we can end the quoting madness | 15:04 |
| *** tvb|afk has quit IRC | 15:05 | |
| *** HenryG has joined #openstack-infra | 15:05 | |
| *** alexpilotti has joined #openstack-infra | 15:09 | |
| jeblair | restarting zuul | 15:09 |
| *** fbo is now known as fbo_away | 15:10 | |
| *** Ryan_Lane has joined #openstack-infra | 15:20 | |
| *** Ryan_Lane has quit IRC | 15:24 | |
| Alex_Gaynor | jeblair, mordred, clarkb: FYI, fastly found the issue that was causing some timeouts on installing thrift, they're in the processing of rolling out a fix | 15:33 |
| *** SergeyLukjanov has quit IRC | 15:36 | |
| *** Ryan_Lane has joined #openstack-infra | 15:50 | |
| *** Ryan_Lane has quit IRC | 15:55 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: doc: fix up strict errors https://review.openstack.org/47861 | 15:56 |
| *** thomasm has joined #openstack-infra | 16:15 | |
| *** tvb has joined #openstack-infra | 16:16 | |
| *** tvb has joined #openstack-infra | 16:16 | |
| *** changbl has quit IRC | 16:16 | |
| *** tvb has quit IRC | 16:20 | |
| *** Ryan_Lane has joined #openstack-infra | 16:21 | |
| *** Ryan_Lane has joined #openstack-infra | 16:21 | |
| *** AlexF has quit IRC | 16:24 | |
| *** Ryan_Lane has quit IRC | 16:25 | |
| *** dizquierdo has left #openstack-infra | 16:27 | |
| openstackgerrit | A change was merged to openstack-infra/gerritlib: Add method to list Gerrit Groups. https://review.openstack.org/48312 | 16:29 |
| *** SnowDust has joined #openstack-infra | 16:38 | |
| *** ericw has joined #openstack-infra | 16:41 | |
| *** salv-orlando_ has joined #openstack-infra | 16:46 | |
| *** ericw has quit IRC | 16:46 | |
| *** ericw has joined #openstack-infra | 16:47 | |
| *** salv-orlando has quit IRC | 16:49 | |
| *** salv-orlando_ is now known as salv-orlando | 16:49 | |
| *** Ryan_Lane has joined #openstack-infra | 16:52 | |
| fungi | soren: thanks for the link! | 16:53 |
| *** Ryan_Lane has quit IRC | 16:56 | |
| fungi | jeblair: what was the upshot of the discussion with sdague on https://review.openstack.org/48793 vs https://review.openstack.org/48767 ? i had approved 48793 before noticing 48767 so went back and -2'd it, but i'll switch back to a +2 there if it's one we still want to go forward with | 16:56 |
| *** ericw has quit IRC | 16:57 | |
| fungi | nevermind, looks like you +2'd it after discussing with sdague rather than before. switching my vote back so it can hopefully merge | 17:11 |
| * fungi disappears to run errands... bbl | 17:14 | |
| SnowDust | http://paste.openstack.org/show/47666/ anyone ? | 17:14 |
| *** dguitarbite has joined #openstack-infra | 17:17 | |
| *** alexpilotti has quit IRC | 17:24 | |
| *** thomasm has quit IRC | 17:24 | |
| *** thomasm has joined #openstack-infra | 17:36 | |
| *** dguitarbite has quit IRC | 17:37 | |
| mordred | soren: neat re: crsf | 17:42 |
| mordred | SnowDust: we don't really do anything with redstack | 17:46 |
| *** DennyZhang has joined #openstack-infra | 17:46 | |
| dstufft | mordred: looks like you'll be getting a 1.4.2 in the near future | 17:50 |
| mordred | dstufft: woot! | 17:50 |
| mordred | I know some humans who will be happy | 17:51 |
| dstufft | the pip dev channel is talking about it, we're gonna see about knocking out the issues | 17:51 |
| dstufft | then see if the RM wants me to make a release or wants to (and is able) to do it himself again | 17:51 |
| * mordred gets prepared to force zul to get 1.4.2 rammed in to the next ubuntu | 17:53 | |
| ekarlso | 1.4.2 of ? | 17:54 |
| mordred | ekarlso: pip | 18:02 |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Rename jeepyb to pypi-mirror https://review.openstack.org/45726 | 18:04 |
| *** cody-somerville has joined #openstack-infra | 18:06 | |
| *** salv-orlando has quit IRC | 18:06 | |
| mordred | dstufft: do I need to explicitly depend on wheel to get the pip wheel commands? or did you guys vendor it? | 18:09 |
| dstufft | it's not vendored | 18:09 |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Start building wheels in the mirror https://review.openstack.org/45727 | 18:10 |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Strip out the things we don't need from jeepyb https://review.openstack.org/45728 | 18:10 |
| *** DennyZhang has quit IRC | 18:11 | |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Remove spurious uses of output return values https://review.openstack.org/46348 | 18:11 |
| *** dims has quit IRC | 18:12 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Rework run_shell_command https://review.openstack.org/42337 | 18:16 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Do not pass unicode where byte strings are wanted https://review.openstack.org/48355 | 18:18 |
| *** schwicht has quit IRC | 18:22 | |
| *** sarob has joined #openstack-infra | 18:28 | |
| *** basha has joined #openstack-infra | 18:28 | |
| *** adalbas has quit IRC | 18:29 | |
| justinabrahms | question on gerrit replication: how do you guys have github replication setup? I'm looking through the docs and I'm not seeing anything in the replication config that is per-project where you'd indicate the other repo. | 18:30 |
| *** Ryan_Lane has joined #openstack-infra | 18:30 | |
| justinabrahms | Do you guys not use gerrit replication and instead use zuul? | 18:30 |
| clarkb | justinabrahms: we use gerrit replication. we have a 1:1 mapping in gerrit project name to github project so openstack/nova in our gerrit maps to github.com/openstack/nova | 18:31 |
| justinabrahms | That makes sense. | 18:31 |
| clarkb | and the gerrit user can push to ssh://github.com/openstack/nova | 18:31 |
| justinabrahms | How is that setup? Is it via something like github deploy keys? | 18:31 |
| mordred | we have a service account at github with an ssh key | 18:32 |
| mordred | https://git.openstack.org/cgit/openstack-infra/config/tree/modules/gerrit/templates/replication.config.erb is the file - although probably completely looks nonsensicle there | 18:32 |
| mordred | justinabrahms: here's what it looks like expanded: http://paste.openstack.org/show/47667/ | 18:33 |
| justinabrahms | actually looks reasonable there. | 18:34 |
| justinabrahms | I think chef has broken my brain. | 18:34 |
| mordred | :) | 18:34 |
| *** sarob has quit IRC | 18:35 | |
| mordred | authGroup means "replicate all repos that can be seen by this group" - mirror means "force push everything all the time" | 18:35 |
| *** sarob has joined #openstack-infra | 18:35 | |
| justinabrahms | actually, looking at the docs, mirror also means to delete any refs which aren't present in the gerrit config | 18:35 |
| clarkb | yeah it is destructive if you use the remote end for anything else | 18:36 |
| *** tvb has joined #openstack-infra | 18:38 | |
| *** tvb has quit IRC | 18:38 | |
| *** tvb has joined #openstack-infra | 18:38 | |
| *** sarob has quit IRC | 18:40 | |
| mordred | we don't use the remote end for anyting else - we really only touch github because if we don't mirror it there, someone else will | 18:41 |
| mordred | (true story, btw, that happened before we did) | 18:41 |
| clarkb | mordred: they still own the favicon account :) | 18:42 |
| clarkb | mordred: which is on my list to fix but wordpress ugh | 18:42 |
| mordred | they do? wait - what? | 18:42 |
| mordred | what favicon account? | 18:42 |
| clarkb | mordred: for openstack/ | 18:42 |
| mordred | what's a favicon account? | 18:42 |
| clarkb | er not favicon | 18:42 |
| clarkb | whatever the damn thing is that makes the pictuere show up and requires wordpress | 18:43 |
| clarkb | clearly I am not awesome enough to remember | 18:43 |
| mordred | gravatar | 18:43 |
| clarkb | that thing | 18:43 |
| mordred | oh. and yeah. gravatar | 18:43 |
| mordred | and wordpress | 18:43 |
| *** basha has quit IRC | 18:45 | |
| mordred | clarkb: ok. I'm taking care of that right now while I'm thinking about it | 18:46 |
| clarkb | mordred: the reason we want to change it (well one reason) is to get a higher res logo | 18:47 |
| mordred | clarkb: do you think we need to make a specific email alias for the gravatar accout? or use one of the current ones? | 18:47 |
| clarkb | mordred: so make sure you don't copy pasta the old one | 18:47 |
| clarkb | mordred: I think review.o.o or similar is fine | 18:47 |
| clarkb | er review at o.o | 18:47 |
| mordred | clarkb: any idea where where's a good high res I should use? | 18:50 |
| clarkb | mordred: http://www.openstack.org/brand/openstack-logo/ | 18:50 |
| clarkb | mordred: I think even the small png for the first logo is better res | 18:50 |
| clarkb | not sure how gravatar will scale if you use the large png | 18:51 |
| openstackgerrit | A change was merged to openstack/requirements: Revert "Describe SQLAlchemy constraint more clearly." https://review.openstack.org/48779 | 18:52 |
| mordred | clarkb: done | 18:54 |
| clarkb | looks much better | 18:55 |
| clarkb | how horrible was the wordpress stuff? | 18:55 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation https://review.openstack.org/48803 | 18:57 |
| mordred | clarkb: not bad | 18:57 |
| mordred | clarkb: I added an account and put it in the file | 18:57 |
| mordred | clarkb: the above pbr change may be interesting :) | 18:58 |
| clarkb | are we building wheel mirrors yet? | 18:59 |
| mordred | nope. but the new pypi-mirror command is ready for it | 18:59 |
| mordred | that change above should test that it will work | 18:59 |
| clarkb | cool | 18:59 |
| mordred | if it works, then I'll propose a change to config to use pypi-mirror instead of jeepyb | 18:59 |
| mordred | which will mean we'll start building them | 18:59 |
| mordred | then we can roll out consumption of them | 18:59 |
| mordred | (the mirror building should continue to build old tarball mirrors too) | 19:00 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: log info on job deletion https://review.openstack.org/48804 | 19:01 |
| *** cody-somerville has quit IRC | 19:02 | |
| mordred | clarkb: the neat thing is, I believe that we will not have to make any changes to devstack to take advantage | 19:06 |
| clarkb | does pip expect wheels to live under those dirs on the mirror? | 19:06 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Update tox config to use latest upgrades https://review.openstack.org/47702 | 19:07 |
| mordred | clarkb: it does when we change the pip.conf to tell it to look there | 19:07 |
| mordred | Alex_Gaynor: uhm. around? | 19:08 |
| mordred | Alex_Gaynor: http://logs.openstack.org/37/42337/8/check/gate-pbr-pypy/00b5ded/console.html | 19:08 |
| Alex_Gaynor | mordred: yeah | 19:08 |
| mordred | I'm confuseled by that error | 19:08 |
| Alex_Gaynor | mordred: waaa, never seen anything like this before; does it reproduce? | 19:09 |
| mordred | Alex_Gaynor: dunno. let me re-run | 19:09 |
| Alex_Gaynor | mordred: I could theoretically imagine how if this patch changed when a stream was closed it could affect it? | 19:10 |
| mordred | ok. the gate is still pretty flaky | 19:10 |
| mordred | Alex_Gaynor: it really shouldn't have | 19:11 |
| Alex_Gaynor | elastic recheck is pretty great | 19:13 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Add pypi-mirror to the list of pre-cloned repos https://review.openstack.org/48805 | 19:14 |
| mordred | Alex_Gaynor: yeah it is | 19:15 |
| *** SnowDust has quit IRC | 19:15 | |
| mordred | clarkb: ^^ the above is needed for further work/testing on wheel patches | 19:15 |
| clarkb | yes super flaky, one of the changes to help make it less flaky flaked out... I am trying to find a bug to reverify against | 19:15 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: KeepLongStdio argument for JUnit publisher https://review.openstack.org/48431 | 19:18 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: parameters self expansion https://review.openstack.org/48476 | 19:21 |
| clarkb | and reverified, hopefully it merges this time | 19:23 |
| justinabrahms | woo! first part of replication setup! https://github.com/justinabrahms/gerrit-demo/commit/1721315657dbd77c6b0a0d1b839418208ca858ec | 19:26 |
| justinabrahms | ^_^ | 19:26 |
| uvirtbot | justinabrahms: Error: "_^" is not a valid command. | 19:26 |
| mordred | justinabrahms: woot! | 19:27 |
| *** Ryan_Lane has quit IRC | 19:28 | |
| soren | jeblair: I don't know, to be honest. Let me try. | 19:45 |
| *** sarob has joined #openstack-infra | 19:46 | |
| *** sarob has quit IRC | 19:51 | |
| soren | jeblair: That totally seems to work. I thought it was supposed to be a nonce, but I guess not. | 19:52 |
| *** weshay has quit IRC | 19:58 | |
| justinabrahms | how do you guys manage the SSH key of your gerrit box? Is that managed via puppet? or do you just add a new one on each deploy, then upload it to github? | 20:00 |
| justinabrahms | second option seems more secure & more work (which I suppose is a truism of "more secure") | 20:00 |
| soren | jeblair: Patch updated to only fetch crumb once. Neat. Thanks for the suggestion. | 20:01 |
| pabelanger | justinabrahms, did the same thing you are, mine are stored in hiera (puppet) | 20:02 |
| justinabrahms | chef has encrypted cookbooks, iirc, which seem like they would be a reasonable fit here. | 20:02 |
| *** salv-orlando has joined #openstack-infra | 20:03 | |
| mordred | justinabrahms: yeah. we put them in heira for puppet | 20:07 |
| mordred | which serves a similar role | 20:07 |
| *** basha has joined #openstack-infra | 20:08 | |
| mordred | so we have a thing that holds our passwords and keys, and then our puppet manifests reference those by name | 20:08 |
| *** AlexF has joined #openstack-infra | 20:14 | |
| *** rickerc has quit IRC | 20:18 | |
| *** DennyZhang has joined #openstack-infra | 20:22 | |
| *** basha has quit IRC | 20:37 | |
| *** jhesketh_ has quit IRC | 20:39 | |
| *** rickerc has joined #openstack-infra | 20:40 | |
| clarkb | mordred: you may find http://www.redrideapp.com/ useful | 20:41 |
| *** thomasm has quit IRC | 20:45 | |
| *** changbl has joined #openstack-infra | 20:50 | |
| mordred | clarkb: fascinating | 20:51 |
| *** tvb has quit IRC | 20:55 | |
| *** AlexF has quit IRC | 21:09 | |
| *** SergeyLukjanov has joined #openstack-infra | 21:12 | |
| *** davidhadas_ has quit IRC | 21:14 | |
| *** SergeyLukjanov has quit IRC | 21:25 | |
| *** tvb has joined #openstack-infra | 21:26 | |
| *** SergeyLukjanov has joined #openstack-infra | 21:26 | |
| jog0 | sdague: its hard to find an accurate query to use for elstic-recheck until we know the root cause. so the bugs being to high level makes sense. Hopefully we will get better at this over time as the number of bugs drops and we reign in transient bugs more | 21:30 |
| jog0 | sdague: just pulled down the change for https://review.openstack.org/#/c/48787/ so it will be running now. | 21:33 |
| jog0 | also sdague to tell if there are unknown failures in the gate we dump out warnings to openstack-qa saying unknown bug | 21:34 |
| *** tvb has quit IRC | 21:34 | |
| jog0 | jeblair: let me know when you want to start running the bot in infra under the new user | 21:34 |
| jog0 | so I can kill my bot | 21:34 |
| *** SergeyLukjanov has quit IRC | 21:44 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Revert "Enable q-vpn service" https://review.openstack.org/48793 | 21:55 |
| *** tjones has joined #openstack-infra | 22:00 | |
| *** tvb has joined #openstack-infra | 22:01 | |
| *** tjones has quit IRC | 22:04 | |
| *** DennyZhang has quit IRC | 22:31 | |
| *** alexpilotti has joined #openstack-infra | 22:51 | |
| sdague | jeblair / mordred - http://logs.openstack.org/74/48474/1/gate/gate-requirements-install/63e24af/console.html - SSL error on pypi... making requirements tests "fun" | 22:58 |
| Alex_Gaynor | sdague: this is a known bug in fastly (PyPi's CDN), they've lidentified it and are in the process of deploying a fix | 22:58 |
| sdague | Alex_Gaynor: ok, any idea on ETA? | 22:59 |
| sdague | it's bad enough to be fighting our own bugs, then to be taken down by external ones :) | 22:59 |
| Alex_Gaynor | sdague: They're in the process of landing and deploying it, it'll take some time to get it rolled out to their full fleet of PoPs, but a) I imagine US pops will come first so it'll be effectively resolved for us soon, b) it's intermitent so it's still possble to proceed, just really annoying :(, it's not totally impossible to install thrift :/ | 23:00 |
| sdague | Alex_Gaynor: well it bounced me 2 out of 3 times today on requirements, so it's pretty bad | 23:01 |
| Alex_Gaynor | Sorry, I realize it sucks :/ | 23:01 |
| Alex_Gaynor | sdague: agreed :/ | 23:01 |
| sdague | yep, no worries, it's a weekend so it's not too bad. Just hope it's fixed by monday, otherwise we're going to have lots of challenges | 23:02 |
| sdague | I've gotten used to the fact that openstack infrastructure is basically the best monitoring framework for the health of the internet. :) All these services that you think are always up, turn out to be down more than you realize | 23:02 |
| Alex_Gaynor | heh | 23:03 |
| notmyname | sdague: http://www.downforeveryoneorjustme.com is really a bot that just asks the -infra team, isn't it? | 23:04 |
| sdague | that my understanding. It just texts mordred | 23:04 |
| sdague | Alex_Gaynor: well at least one of the fails was something other than thrift - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIlRoZSByZWFkIG9wZXJhdGlvbiB0aW1lZCBvdXQgd2hpbGUgZ2V0dGluZyBcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDQxMDA5MTQxMn0= | 23:15 |
| Alex_Gaynor | dstufft: FYI https://bugs.launchpad.net/openstack-ci/+bug/1232592 | 23:19 |
| uvirtbot | Launchpad bug 1232592 in openstack-ci "pypi CDN bugs cause SSL errors when trying to install packages" [Undecided,New] | 23:19 |
| dstufft | Alex_Gaynor: :[ | 23:20 |
| dstufft | Alex_Gaynor: Not sure I have a better answer then what you have already said | 23:20 |
| Alex_Gaynor | dstufft: sure | 23:20 |
| dstufft | Fastly always does phased roll outs so they don't take down their entire network at once with a bad deploy | 23:21 |
| dstufft | sdague: Alex_Gaynor I can give you a temporary bypass the CDN if that'd help | 23:22 |
| dstufft | well no I lie I can't | 23:22 |
| dstufft | I don't have DNS control | 23:22 |
| sdague | dstufft: if you think it's solved within 24 hrs, it's fine to wait, we'll just recheck | 23:23 |
| sdague | if you think it's going to bleed into the work week, we might want to figure out a bypass | 23:23 |
| dstufft | sdague: a full rollout takes a week or two, Not sure at what rate their rollout actually occurs though (e.g. is 50% of the hosts got the fix within 24 hours? I have no idea) | 23:24 |
| dstufft | sdague: this is running PR tests? | 23:24 |
| sdague | PR? | 23:25 |
| dstufft | pull requests | 23:25 |
| dstufft | I don't know the gerrit terms | 23:25 |
| dstufft | does this operate in a trusted context | 23:25 |
| sdague | it's testing proposed patches | 23:25 |
| dstufft | sdague: I ask because I do have an url that bypasses the CDN, but it also does not have TLS | 23:26 |
| dstufft | so you're open for MITM | 23:26 |
| sdague | these are devstack nodes, so they are single use | 23:26 |
| dstufft | if this is during a job that is testing random crap, that's not a big deal because you already have to deal with untrusted code in that context | 23:26 |
| dstufft | if this runs on a trusted node, that is a bad thing | 23:26 |
| sdague | the jobs that have bounced are all untrusted | 23:27 |
| dstufft | sdague: pypi.int.python.org bypasses the CDN but there is zero SLA on that url so if we decide to move things around or something that url may stop working (but it probably won't in the next week or so) | 23:27 |
| sdague | I'll let mordred or jeblair comment on whether an untrusted url for this would be a good solution come the work week | 23:27 |
| dstufft | it also bypasses the LB | 23:27 |
| dstufft | so I don't suggest using it unless you need to | 23:28 |
| sdague | yeh, so I think mostly we only hit it on jobs which bypass our mirror | 23:28 |
| dstufft | sdague: I'm sorry we're affecting you though | 23:28 |
| sdague | yeh, no worries, stuff happens | 23:28 |
| dstufft | Varnish isn't really designed to serve big files | 23:28 |
| dstufft | if you have jobs that are hitting pypi directly (besides the obvious mirror jobs) you might want to setup a global mirror | 23:30 |
| sdague | so this only actually seems to be affecting us on our global requirements repo, that's small enough of a surface that it's easy enough to work around | 23:30 |
| dstufft | one that just tracks PyPI in entirity | 23:30 |
| dstufft | sdague: ok | 23:30 |
| sdague | dstufft: we actually use the fact that we have a limitted mirror to prevent packages from sneaking in new dependencies | 23:31 |
| sdague | it's a second line of defense | 23:31 |
| dstufft | sdague: ya | 23:31 |
| sdague | thanks for the options though, if it becomes a bigger issue might want to sort out switching the mirror | 23:32 |
| sdague | time for me to pumkin for the night, later folks | 23:33 |
| *** nati_ueno has joined #openstack-infra | 23:37 | |
| *** ericw has joined #openstack-infra | 23:39 | |
| *** reed has joined #openstack-infra | 23:40 | |
| *** nati_ueno has quit IRC | 23:41 | |
| fungi | right, we figured hitting pypi to test changes to the set of packages from which we build our mirror was an infrequent enough job that having to *occasionally* recheck because of network screwiness was acceptable. though it wasn't until the thrift download problems that i realized just how often we do change the requirements list | 23:48 |
| *** reed has quit IRC | 23:54 | |
| boris-42 | fungi hi | 23:54 |
| boris-42 | fungi does gate work properly?) | 23:54 |
| boris-42 | fungi I mean it's empty!=0 | 23:55 |
| fungi | boris-42: it works best when empty ;) | 23:55 |
| boris-42 | fungi =)) | 23:55 |
| openstackgerrit | A change was merged to openstack/requirements: Removed a reference to quantum and removed a dead file https://review.openstack.org/48474 | 23:57 |
| *** ianw has quit IRC | 23:58 | |
| *** ericw has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!