| *** ddieterly has quit IRC | 00:01 | |
| *** dimsum__ has joined #openstack-infra | 00:06 | |
| *** sdake_ has joined #openstack-infra | 00:08 | |
| *** sdake_ has quit IRC | 00:08 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 00:09 |
|---|---|---|
| *** sdake_ has joined #openstack-infra | 00:09 | |
| *** zz_dimtruck is now known as dimtruck | 00:10 | |
| *** camunoz has joined #openstack-infra | 00:11 | |
| *** sdake has quit IRC | 00:12 | |
| *** dimsum__ has quit IRC | 00:14 | |
| *** yamamoto has joined #openstack-infra | 00:18 | |
| *** amotoki has joined #openstack-infra | 00:19 | |
| openstackgerrit | lifeless proposed openstack-infra/project-config: Check resolver pip on global-requirements changes https://review.openstack.org/181774 | 00:20 |
| *** otter768 has joined #openstack-infra | 00:23 | |
| lifeless | oh ugh | 00:24 |
| lifeless | we do have a complete copy of the same build-everything script in requirements | 00:25 |
| lifeless | why? | 00:25 |
| *** jogo has quit IRC | 00:25 | |
| *** ddieterly has joined #openstack-infra | 00:28 | |
| *** weshay has quit IRC | 00:28 | |
| *** Alexandra_ has quit IRC | 00:32 | |
| *** dimsum__ has joined #openstack-infra | 00:39 | |
| *** hdd has quit IRC | 00:40 | |
| tchaypo | Which script? | 00:40 |
| *** ivar-laz_ has joined #openstack-infra | 00:40 | |
| *** woodster_ has quit IRC | 00:40 | |
| *** ivar-la__ has joined #openstack-infra | 00:42 | |
| *** ivar-lazzaro has quit IRC | 00:42 | |
| *** ivar-laz_ has quit IRC | 00:42 | |
| *** trey has joined #openstack-infra | 00:43 | |
| openstackgerrit | lifeless proposed openstack/requirements: Allow testing with specific pips https://review.openstack.org/181777 | 00:44 |
| lifeless | tchaypo: oh hai, so lets talk environment markers | 00:45 |
| lifeless | tchaypo: tools/integration.sh in pbr and in requirements. | 00:45 |
| tchaypo | lifeless: also, would you have some time this afternoon (ie, at least 2 hours from now) to talk about... | 00:46 |
| tchaypo | Hah, you beat me to it | 00:46 |
| lifeless | I will have time anytime until 3.5hrs from now | 00:47 |
| lifeless | at which time i need to go out and pick up C etc | 00:48 |
| *** dimtruck is now known as zz_dimtruck | 00:48 | |
| tchaypo | Okay good, I'll ping you as soon as I'm free. Hopefully it will be less than 2 hours | 00:49 |
| *** yamahata has quit IRC | 00:55 | |
| *** zz_dimtruck is now known as dimtruck | 00:55 | |
| *** craigbr has joined #openstack-infra | 00:56 | |
| *** craigbr has quit IRC | 00:58 | |
| *** fhubik_lunch has joined #openstack-infra | 01:01 | |
| *** mriedem has joined #openstack-infra | 01:02 | |
| *** dtantsur|pto has quit IRC | 01:03 | |
| *** dtantsur has joined #openstack-infra | 01:03 | |
| *** fhubik_afk has quit IRC | 01:04 | |
| *** mriedem1 has quit IRC | 01:04 | |
| *** julim has joined #openstack-infra | 01:05 | |
| *** dtantsur has quit IRC | 01:10 | |
| *** dtantsur has joined #openstack-infra | 01:11 | |
| *** fhubik_lunch has quit IRC | 01:11 | |
| *** fhubik_lunch has joined #openstack-infra | 01:11 | |
| *** fhubik_lunch is now known as fhubik_afk | 01:11 | |
| *** otter768 has quit IRC | 01:12 | |
| fungi | lifeless: there used to be one script, in the pbr repo. sdague forked them so that he could adjust the requirements integration job to not test things which were only relevant to pbr's integration testing | 01:19 |
| lifeless | fungi: well, sure - I know that history | 01:20 |
| lifeless | fungi: but - I don't see what value testing that everything can build successfully on a global requirements change brings | 01:20 |
| fungi | i suspect the one in the requirements repo simply hasn't been stripped down as far as it could be | 01:21 |
| *** zhiwei has joined #openstack-infra | 01:25 | |
| *** zhiwei has left #openstack-infra | 01:30 | |
| *** markvoelker has quit IRC | 01:35 | |
| *** markvoelker has joined #openstack-infra | 01:36 | |
| *** banix has quit IRC | 01:37 | |
| *** markvoelker has quit IRC | 01:37 | |
| *** baoli has joined #openstack-infra | 01:39 | |
| *** baoli has quit IRC | 01:44 | |
| *** ddieterly has quit IRC | 01:45 | |
| *** julim has quit IRC | 01:48 | |
| *** hdd has joined #openstack-infra | 01:50 | |
| *** mriedem has quit IRC | 01:53 | |
| *** achanda has joined #openstack-infra | 01:55 | |
| *** banix has joined #openstack-infra | 01:56 | |
| *** mahito has quit IRC | 01:57 | |
| *** dimtruck is now known as zz_dimtruck | 01:59 | |
| *** Shrews has joined #openstack-infra | 02:00 | |
| *** oomichi has joined #openstack-infra | 02:01 | |
| *** oomichi has quit IRC | 02:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Straighten up keystone endpoint management https://review.openstack.org/181783 | 02:01 |
| mordred | lifeless: the reasoning is that sometimes a requirements bump causes a knock-on effect elsewhere which causes pain | 02:03 |
| lifeless | mordred: I presume its done so in the past? | 02:04 |
| mordred | lifeless: like, "I bumped the six version and it caused me to not be able to install eventlet" - I believe is the thing that was trying to be trapped for | 02:04 |
| mordred | lifeless: yah | 02:04 |
| mordred | lifeless: now, whether the current state can still fail in that manner is an excellent question | 02:04 |
| lifeless | so we have the coinstallability test | 02:04 |
| lifeless | which is up the top of the script | 02:04 |
| lifeless | and separate to the map over projects | 02:04 |
| mordred | yah. which is likely a better test from what we had before | 02:04 |
| lifeless | I'm not questioning the coinstallability test | 02:04 |
| lifeless | just the map over projects | 02:05 |
| mordred | lifeless: does coinstallablility of the total list imply coinstallability of all arbitrary sub-lists? | 02:05 |
| lifeless | certainly the multiple variations of setup.py per project | 02:05 |
| lifeless | mordred: with my resolver branch yes | 02:05 |
| mordred | but not with non-resolver | 02:05 |
| lifeless | I can imagine corner cases with non-resolver | 02:06 |
| mordred | I think that's the thing - without resolver, a sub-list may exhibit a different coinstallability solution because of ordering | 02:06 |
| mordred | and the brute-force solution is to try each likely sublist to be sure | 02:06 |
| lifeless | if two components A and B depend on (bad, good) versions of C | 02:06 |
| mordred | and then to not trust different setup.py invocations to not alter the ordering | 02:06 |
| *** mahito has joined #openstack-infra | 02:06 | |
| lifeless | and C was in the global list | 02:06 |
| lifeless | but not in the sublist | 02:06 |
| mordred | yah | 02:06 |
| mordred | lifeless: have I mentioned how excited I am for resolver? | 02:07 |
| lifeless | very | 02:07 |
| lifeless | (I hope) | 02:07 |
| *** ddieterly has joined #openstack-infra | 02:07 | |
| lifeless | oh man, this virus has my body messed up. Sitting and shaking now..... hope I'm well by Vancouver time | 02:07 |
| mordred | lifeless: I recommend not getting viruses | 02:08 |
| lifeless | mordred: while you're here, I'd like to remove the requirement of pip from pr | 02:08 |
| lifeless | pr | 02:08 |
| lifeless | pbr | 02:08 |
| lifeless | since we no longer call pip ourselves | 02:08 |
| mordred | I think we don't need it anymore now that we're not ... yeah | 02:08 |
| mordred | and that gets us back down to 0 runtime depends, which is nice | 02:08 |
| *** camunoz has quit IRC | 02:08 | |
| lifeless | though I still see | 02:08 |
| lifeless | pbr/packaging.py: cmd = [sys.executable, '-m', 'pip.__init__', 'install'] | 02:08 |
| lifeless | whats that about I wonder | 02:09 |
| mordred | lifeless: it's vestigal and used in a branch that never gets touched really | 02:10 |
| lifeless | can we nuke it? | 02:10 |
| mordred | lifeless: the attempt to install test requirements before running setup.py tet | 02:10 |
| mordred | lifeless: the attempt to install test requirements before running setup.py test | 02:10 |
| lifeless | so same situation | 02:10 |
| mordred | yah - need to remove _PipInstallTestRequires | 02:10 |
| mordred | and then you can remove _pip_install | 02:10 |
| mordred | it can really be one big happy patch | 02:11 |
| lifeless | we could fold the test requires in our projects into an extra | 02:11 |
| lifeless | and then install .[test] | 02:11 |
| mordred | we could - but we do not use that feature | 02:11 |
| lifeless | ok, so if we don't use it | 02:11 |
| mordred | lifeless: last I checked, extra is not additive | 02:11 |
| lifeless | extras is a set | 02:11 |
| mordred | lifeless: so it would need to be pip install . .[test] no? | 02:11 |
| lifeless | and its additive | 02:11 |
| mordred | it is? ok. that's way less useless than I thought | 02:11 |
| mordred | I though each extras list was separate | 02:12 |
| *** dimsum__ has quit IRC | 02:12 | |
| lifeless | try pip install requests[security] | 02:12 |
| lifeless | in a clean venv :) | 02:12 |
| lifeless | they are | 02:12 |
| mordred | in any case- I support removing _pip_install, its consumers, and the requirement | 02:12 |
| lifeless | the install is the union of the thing and its extras | 02:12 |
| mordred | awesome | 02:12 |
| mordred | that's what I want it to be | 02:12 |
| lifeless | oh nice | 02:13 |
| lifeless | we can nuke more stuff | 02:13 |
| mordred | yah | 02:14 |
| mordred | nuking stuff from pbr is always a happy | 02:14 |
| *** otter768 has joined #openstack-infra | 02:14 | |
| mordred | it means something in life has improved | 02:14 |
| lifeless | oh | 02:15 |
| lifeless | hooks/backwards uses parse_dependency_links | 02:15 |
| mordred | yah. there is a bunch of stuff in there that can die | 02:15 |
| lifeless | complex enough I'm shelving that aspect | 02:15 |
| mordred | yah - but I'd like to come back and nuke a large chunk of that | 02:15 |
| mordred | it's leftover from like 3 years ago and we don't support the thing it was written to support anymore :) | 02:16 |
| lifeless | ditto | 02:16 |
| lifeless | but one bite | 02:16 |
| mordred | yu | 02:16 |
| mordred | lifeless: I can send you a patch for the removes if you want | 02:17 |
| lifeless | sec | 02:17 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Finish removing invocations of pip. https://review.openstack.org/181785 | 02:19 |
| lifeless | mordred: sure, ^ has the pip bit | 02:19 |
| lifeless | I'll do a 1.0.0 tomorrow (so that infra folk are around), with whatever we have in trunk | 02:19 |
| *** mmedvede_ has quit IRC | 02:20 | |
| mordred | lifeless: you misssed _pip_install - comment left - also, there's a SKIP_PIP_INSTALL reference still in the docs | 02:21 |
| *** jogo has joined #openstack-infra | 02:21 | |
| *** camunoz has joined #openstack-infra | 02:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove n-novnc from enabled services https://review.openstack.org/181736 | 02:22 |
| mordred | lifeless: also, 181777 is unhappy with pypy, fwiw | 02:24 |
| *** zz_dimtruck is now known as dimtruck | 02:27 | |
| lifeless | 181777 needs the config change done | 02:27 |
| mordred | nod | 02:33 |
| lifeless | I mean, it may have more issues | 02:34 |
| lifeless | but there's no point fiddling until the new job is up and done - merging it first would be bad :) | 02:34 |
| *** dimtruck is now known as zz_dimtruck | 02:40 | |
| *** yamamoto has quit IRC | 02:40 | |
| *** Alexandra_ has joined #openstack-infra | 02:40 | |
| lifeless | mordred: - updated | 02:40 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Finish removing invocations of pip. https://review.openstack.org/181785 | 02:40 |
| *** yamamoto has joined #openstack-infra | 02:41 | |
| *** amotoki has quit IRC | 02:43 | |
| *** zz_dimtruck is now known as dimtruck | 02:44 | |
| *** sputnik13 has joined #openstack-infra | 02:46 | |
| tchaypo | lifeless: okay, I’m settled. | 02:49 |
| *** yamamoto_ has joined #openstack-infra | 02:49 | |
| *** mpopow has joined #openstack-infra | 02:49 | |
| lifeless | tchaypo: ok so hi | 02:50 |
| * tchaypo loads backscroll from friday | 02:51 | |
| *** yamamoto has quit IRC | 02:52 | |
| *** patrickeast has quit IRC | 02:52 | |
| *** mpopow has quit IRC | 02:54 | |
| lifeless | tchaypo: also add this thread to your reading list | 02:55 |
| lifeless | https://groups.google.com/forum/#!topic/pypa-dev/2UFQKqSHc_s | 02:56 |
| *** amotoki has joined #openstack-infra | 02:56 | |
| lifeless | tchaypo: http://paste.ubuntu.com/11072233/ may be useful too | 02:56 |
| *** boris-42 has quit IRC | 02:58 | |
| *** dimtruck is now known as zz_dimtruck | 02:59 | |
| *** camunoz has quit IRC | 03:00 | |
| *** achanda has quit IRC | 03:01 | |
| tchaypo | Oh goody | 03:03 |
| tchaypo | I end up at https://github.com/pypa/pip/issues/1433 which was filed by haypo | 03:03 |
| *** camunoz has joined #openstack-infra | 03:04 | |
| *** fedexo has quit IRC | 03:04 | |
| lifeless | yup | 03:04 |
| *** achanda_ has joined #openstack-infra | 03:04 | |
| tchaypo | Currently reading https://github.com/pypa/pip/pull/2153/files | 03:04 |
| lifeless | oh don't | 03:04 |
| lifeless | stay shallow for now | 03:04 |
| lifeless | whats your understanding so far | 03:04 |
| *** sdake has joined #openstack-infra | 03:05 | |
| lifeless | (and do you know what environment markers even are?) | 03:05 |
| *** sdake has quit IRC | 03:05 | |
| tchaypo | but from the release notes on 6.1.1 and your thread it sounds like this is already implemented in pip/setup_tools/distutils | 03:05 |
| *** fedexo has joined #openstack-infra | 03:05 | |
| *** sdake has joined #openstack-infra | 03:05 | |
| tchaypo | my understanding so far is that environment markers are the thing to say “if the environment looks like X, i require Y" | 03:05 |
| lifeless | https://www.python.org/dev/peps/pep-0426/#environment-markers | 03:06 |
| lifeless | yeah | 03:06 |
| tchaypo | obviously useful for python2 vs python3, but I suspect possibly also useful for “if redhat require blah, if debian require other blah" | 03:06 |
| *** sdake_ has quit IRC | 03:06 | |
| lifeless | less so, but perhaps | 03:06 |
| tchaypo | ah, the pep has “if windows” and “if i386” as examples | 03:06 |
| *** sputnik13 has quit IRC | 03:07 | |
| lifeless | ok | 03:07 |
| lifeless | so - have a look at my pastebin | 03:07 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add python-cloudpulseclient project for CloudPulse https://review.openstack.org/181761 | 03:08 |
| tchaypo | I’m noticing lots of mentions of wheels here. My memory says that wheels only have very limited ways to indicate what they’re compatible with | 03:08 |
| *** sdake has quit IRC | 03:08 | |
| lifeless | you'll need a python2.6 virtualenv. https://launchpad.net/~fkrull/+archive/ubuntu/deadsnakes is a good way to get that | 03:09 |
| lifeless | wheels have all the same requirements capabilities as the rest of python packages | 03:09 |
| *** sdake has joined #openstack-infra | 03:09 | |
| lifeless | you're remembering the applicability discusssion about used library files | 03:09 |
| lifeless | e.g. was it compiled on rhel vs debian | 03:09 |
| lifeless | thats a related but distinct topic, ignore for now | 03:10 |
| tchaypo | yes, that sounds right - it’s an issue for wheels but not for source, right? | 03:10 |
| lifeless | yes, but what we're interested in has to apply for both | 03:10 |
| lifeless | ok, so - get my pastebin applied to a new git tree and try installing it in a clean venv of 2.6 | 03:12 |
| lifeless | ditto a clean venv of 2.7 | 03:12 |
| *** sputnik13 has joined #openstack-infra | 03:12 | |
| *** sputnik13 has quit IRC | 03:13 | |
| *** sputnik13 has joined #openstack-infra | 03:13 | |
| *** panda|afk has quit IRC | 03:13 | |
| *** panda|afk has joined #openstack-infra | 03:14 | |
| *** achanda_ has quit IRC | 03:14 | |
| *** ddieterly has quit IRC | 03:14 | |
| *** mahito has quit IRC | 03:16 | |
| *** alex_xu has quit IRC | 03:20 | |
| *** baoli has joined #openstack-infra | 03:21 | |
| *** alex_xu has joined #openstack-infra | 03:21 | |
| *** marcusvrn has joined #openstack-infra | 03:21 | |
| lifeless | tchaypo: <crickets> | 03:22 |
| *** yamamoto_ has quit IRC | 03:22 | |
| tchaypo | sorry. adding the ppa and setting up the venvs. | 03:23 |
| *** shashankhegde has joined #openstack-infra | 03:24 | |
| lifeless | tchaypo: ok, for clarity:I'm doing nothing other than discuss this with you atm | 03:24 |
| *** mahito has joined #openstack-infra | 03:24 | |
| lifeless | tchaypo: if I do something else i'll context switch and be gone for an hour | 03:24 |
| * tchaypo waits for packages to install | 03:25 | |
| tchaypo | i hope you have some music playing or something | 03:25 |
| lifeless | lupe fiasco | 03:25 |
| *** baoli has quit IRC | 03:26 | |
| *** mahito has quit IRC | 03:29 | |
| *** mahito has joined #openstack-infra | 03:30 | |
| tchaypo | my first attempt at applying the patch gave me a file called -.rej | 03:34 |
| tchaypo | good work james | 03:34 |
| tchaypo | second attempt worked better | 03:34 |
| tchaypo | okay, installing in clean 2.6 venv | 03:35 |
| tchaypo | that seems fine. trying 2.7.. | 03:35 |
| tchaypo | and that works fine. didn’t install argparse, as expected | 03:35 |
| lifeless | right | 03:37 |
| lifeless | so | 03:37 |
| lifeless | the thing we want | 03:37 |
| lifeless | is some machine mergable requirements thing | 03:37 |
| lifeless | requirements.txt gets machine gardened with human approvals | 03:38 |
| lifeless | via update.py in requirements - have you looked at that? | 03:38 |
| tchaypo | briefly | 03:38 |
| lifeless | extras_require in setup.py is not machine gardened today | 03:38 |
| lifeless | and since its editing-python-source less easily machine gardened than either requirements.txt or setup.cfg | 03:39 |
| lifeless | since the setup.cfg stuff is still in nightmare territory | 03:39 |
| lifeless | but | 03:39 |
| lifeless | is the way of the future | 03:39 |
| lifeless | I'm going to suggest an impl strategy, which is to teach pbr to read extra requires from setup.cfg | 03:39 |
| *** sputnik13 has quit IRC | 03:40 | |
| *** marcusvrn has quit IRC | 03:40 | |
| lifeless | and inject that like requirements.txt is injected into the Distribution metadata | 03:40 |
| tchaypo | stepping back | 03:40 |
| tchaypo | why are the extras_require in setup.py? | 03:41 |
| tchaypo | oh. because pbr doesn’t know how to read them from setup.cfg | 03:41 |
| lifeless | bada bing bada boom | 03:41 |
| tchaypo | I might be confused about what extras_require is then | 03:41 |
| *** sputnik13 has joined #openstack-infra | 03:41 | |
| lifeless | its a dict of conditional requirements | 03:42 |
| lifeless | prior to environment markers the condition was only activated one way: by someone requesting one or more of the conditions | 03:42 |
| tchaypo | nope, I’m not confused, I just hadn’t seen it used for this | 03:42 |
| lifeless | e.g. requests[security] | 03:42 |
| tchaypo | the usage I’m familiar with is.. | 03:42 |
| tchaypo | exactly, requests[security] | 03:42 |
| lifeless | the extra '' is == install_requires | 03:43 |
| lifeless | environment markers make it possible to automatically activate additional requirements | 03:43 |
| tchaypo | and it makes sense to put the env-marker stuff in extras_requires | 03:43 |
| lifeless | by having an extra of '' with the condition <whatever> | 03:43 |
| lifeless | expressed as ':condition' | 03:43 |
| tchaypo | because the stuff that’s in install_requires always needs to be installed, the stuff in extras only needs to be installed sometimes | 03:44 |
| tchaypo | okay | 03:44 |
| lifeless | its batshit insane as far as UI goes | 03:44 |
| lifeless | but there it is | 03:44 |
| tchaypo | so, next question | 03:44 |
| *** yamamoto has joined #openstack-infra | 03:44 | |
| tchaypo | "The micro-language behind this is a simple subset of Python: it compares only strings, with the == and in operators (and their opposites), and with the ability to combine expressions" | 03:44 |
| tchaypo | you’re proposing to teach pbr to understand a simple subset of python? | 03:45 |
| lifeless | no | 03:45 |
| lifeless | I'm proposing to teach pbr about extras | 03:45 |
| tchaypo | oh. | 03:45 |
| lifeless | in a way that will be compatible with the broad needs of the requirements update process | 03:45 |
| tchaypo | and if pbr can read the extras, it can just pass the strings through as-is and let the existing machinery handle the parsing | 03:46 |
| lifeless | yes, that is what pbr does for requirements files too, FWIW | 03:47 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Update Kolla Meeting https://review.openstack.org/181790 | 03:48 |
| *** hdd has quit IRC | 03:49 | |
| tchaypo | okay. So what would these look like in setup.cfg.. | 03:51 |
| *** banix has quit IRC | 03:51 | |
| tchaypo | hrm. This sounds like it depends on the work to make pbr read install_requires from setup.cfg rather than requirements.txt | 03:53 |
| lifeless | I had an implementation | 03:53 |
| lifeless | let me push it back up (its forpip, but the schema and input code is what you'll need | 03:53 |
| *** sdake_ has joined #openstack-infra | 03:56 | |
| lifeless | tchaypo: just having to deal with some conflicts | 03:57 |
| lifeless | tchaypo: no, install_requires moving to setup.cfg is entirely separate | 03:57 |
| *** camunoz has quit IRC | 03:58 | |
| *** rkukura has quit IRC | 03:59 | |
| *** rkukura has joined #openstack-infra | 03:59 | |
| *** sdake has quit IRC | 04:00 | |
| *** sputnik13 has quit IRC | 04:02 | |
| tchaypo | okay. that seems backwards to me | 04:02 |
| lifeless | tchaypo: https://github.com/pypa/pip/pull/2603 (the discussion is irrelevant for now) | 04:02 |
| tchaypo | having extras_requires in setup.cfg but not install_requires | 04:02 |
| lifeless | thats not a goal | 04:02 |
| lifeless | its just not a dependency | 04:02 |
| tchaypo | *thinks* on the other hand, pbr can already handle install requires somewhat, but it can’t handle extras at all | 04:02 |
| *** sputnik13 has joined #openstack-infra | 04:03 | |
| lifeless | https://github.com/pypa/pip/pull/2603/files#diff-e804de345dd8db5b181b697998b44ab9R485 | 04:03 |
| lifeless | is the format bit | 04:03 |
| lifeless | https://github.com/pypa/pip/pull/2603/files#diff-abf8962da0a79184dd2c4bf311fc9b57R159 | 04:04 |
| lifeless | was the parsing bit | 04:04 |
| lifeless | BUT | 04:04 |
| lifeless | you probably want to keep things in string format | 04:05 |
| lifeless | perhaps | 04:05 |
| lifeless | - I'll leave that to you to explore :) | 04:05 |
| lifeless | tchaypo: and yes, install_requires works - its reflected into by pbr already. | 04:05 |
| *** mpopow has joined #openstack-infra | 04:05 | |
| lifeless | mordred: around ? | 04:06 |
| tchaypo | what I’m not seeing there is an example of an env marker inside setup.cfg | 04:06 |
| lifeless | tchaypo: of course, since I wasn't working on env markers | 04:06 |
| lifeless | [extras] | 04:07 |
| lifeless | :python_version=='2.6' = | 04:07 |
| lifeless | argparse | 04:07 |
| lifeless | EOF | 04:07 |
| tchaypo | yep, that’s what I was just coming to | 04:07 |
| lifeless | it might fall into a screaming heap | 04:07 |
| lifeless | depending on iniparse | 04:07 |
| tchaypo | there’s really no difference between the env-marker and just-a-label case as far as pbr is concerned though | 04:07 |
| tchaypo | either way it’s a string label to pass on, and The Machinery will handle it appropriately | 04:08 |
| lifeless | ConfigParser.ParsingError: File contains parsing errors: setup.cfg | 04:09 |
| lifeless | [line 7]: ':foo = \n' | 04:09 |
| lifeless | [line 8]: ' bar\n' | 04:09 |
| lifeless | quotes work | 04:09 |
| lifeless | ish | 04:10 |
| lifeless | oh wow | 04:10 |
| lifeless | thats messed up | 04:10 |
| *** camunoz has joined #openstack-infra | 04:10 | |
| lifeless | [extras] | 04:11 |
| lifeless | ":foo" = | 04:11 |
| lifeless | bar | 04:11 |
| lifeless | gets me | 04:11 |
| lifeless | >>> r.items('extras') | 04:11 |
| lifeless | [('"', 'foo" =\nbar')] | 04:11 |
| lifeless | yes, a key of " .... | 04:11 |
| lifeless | single quotes fail similarly | 04:12 |
| lifeless | ; is a comment | 04:12 |
| lifeless | | works | 04:12 |
| lifeless | >>> r.items('extras') | 04:13 |
| lifeless | [('|foo', '\nbar')] | 04:13 |
| tchaypo | so to extend on that, it sounds like we don’t have to teach pbr about env markers as such | 04:13 |
| lifeless | tchaypo: we will need to | 04:14 |
| tchaypo | we need to teach it about reading extras in setup.cfg, and that should mean it now supports env markers | 04:14 |
| lifeless | tchaypo: because markerlib / pkg_resources know : as the separator | 04:14 |
| lifeless | so we'll need to translate '|foo' to ':foo' | 04:14 |
| *** ddieterly has joined #openstack-infra | 04:15 | |
| lifeless | we don't need to write an interpreter or anything, if thats what you were speculating about | 04:15 |
| tchaypo | hrm | 04:16 |
| tchaypo | so env markers have to be specified with a : in setup.py but a | in setup.cfg? I don’t like that much. | 04:16 |
| lifeless | I suspect that this may have cascading failure modes once we get it out there with the use of ini file keys as extra keys | 04:16 |
| lifeless | tchaypo: they are specified with a ; in requirements.txt | 04:16 |
| lifeless | tchaypo: so don't let it both you too much | 04:16 |
| lifeless | tchaypo: you know the three step program :) | 04:17 |
| tchaypo | At least it should be safe to assume that we’re not going to worry about a package called “|foo”, right | 04:17 |
| tchaypo | ? | 04:17 |
| tchaypo | right? Please say I’m right | 04:17 |
| lifeless | | is not a legal package element TBBOMK | 04:17 |
| tchaypo | I’m just going to go with that. | 04:18 |
| lifeless | though I can make an import a package called | | 04:18 |
| lifeless | but, I'm evil. | 04:18 |
| lifeless | >>> __import__('|') | 04:19 |
| lifeless | <module '|' from '|/__init__.py'> | 04:19 |
| * tchaypo looks at birds in the park | 04:19 | |
| *** ddieterly has quit IRC | 04:19 | |
| tchaypo | much prettier | 04:19 |
| lifeless | >>> __import__('#|').name | 04:20 |
| lifeless | 'hashpipe' | 04:20 |
| tchaypo | reading https://github.com/pypa/pip/pull/2603/files#diff-abf8962da0a79184dd2c4bf311fc9b57R159 it seems like pkg_resources.parse_requirements already handles the requirements | 04:20 |
| lifeless | yes, the problem is marshalling | 04:20 |
| tchaypo | and it should be fairly trivial to iterate over them and push the result into dm | 04:20 |
| lifeless | and verification | 04:20 |
| lifeless | the _dep_map thing I do there is irrelvant for pbr | 04:20 |
| lifeless | look at pbr's existing code to see how marshalling is done on pbr's side | 04:21 |
| tchaypo | I understand “marshalling” to mean converting a data type from the form used by one language into the form used by a different language | 04:21 |
| lifeless | s/language/thing/ | 04:21 |
| lifeless | and yes | 04:21 |
| lifeless | setuptools doesn't know about requirements.txt. pbr does. pbr marshalls the install_requires from requirements.txt into a form setuptools understands | 04:22 |
| tchaypo | not sure why that’s relevant here - doesn’t setuptools already know about setup.cfg? | 04:23 |
| *** coolsvap|afk is now known as coolsvap | 04:23 | |
| lifeless | ish | 04:23 |
| lifeless | it doesn't process dependency data from it | 04:23 |
| *** sks has joined #openstack-infra | 04:23 | |
| lifeless | you can make the argument that you should do this in setuptools | 04:24 |
| tchaypo | pip/2603 was an attempt to add it in there | 04:24 |
| lifeless | no | 04:24 |
| lifeless | 2603 is about pip, defining something I hope to also add to setuptools | 04:24 |
| lifeless | there's a mess of chicken egg around this | 04:24 |
| lifeless | regardless | 04:25 |
| lifeless | pbr needs to kep doing requirements.txt indefinitely for backwards compat | 04:25 |
| lifeless | I am not sure that you could sanely do requires in setup.cfg and not have pbr stomp over stuff by mistake | 04:25 |
| lifeless | also | 04:26 |
| lifeless | I know the pbr codebase well | 04:26 |
| lifeless | and can support you | 04:26 |
| tchaypo | am hunting for the bit where the marshalling happens in pbr | 04:26 |
| lifeless | setuptools I have not yet tackled | 04:26 |
| lifeless | and I'm told its worse than pip was before I started cleavering | 04:27 |
| *** yamamoto has quit IRC | 04:28 | |
| *** ildikov has quit IRC | 04:28 | |
| tchaypo | D1_D2_SETUP_ARGS | 04:29 |
| lifeless | yeah | 04:29 |
| lifeless | there was a thing call d2to1 | 04:29 |
| lifeless | pbr is sortof a fork of that | 04:29 |
| lifeless | it may be that somewhere in its guts it supports extras already | 04:30 |
| tchaypo | my memory is a bit hazy | 04:30 |
| lifeless | the author claimed that but I could not figure it out | 04:30 |
| tchaypo | but memory says there was a thing called setuptools, which got forked to make distutils, and they developed seperately; and then distutils2 was formed by merging distutils with setuptools | 04:30 |
| *** yamamoto has joined #openstack-infra | 04:30 | |
| lifeless | those all other different things than d2to1 | 04:31 |
| *** ildikov has joined #openstack-infra | 04:31 | |
| lifeless | distutils came first, setuptools layers on top of distutils, distribute was a fork of setuptools, distutils2 was a short lived thing before they went back to the name setuptools | 04:31 |
| tchaypo | in unrelated news, the birds in the park have stopped being pretty | 04:32 |
| tchaypo | one of them seems to be emulating a car alarm | 04:32 |
| tchaypo | so d2 is distutils2? | 04:32 |
| lifeless | d2to1 | 04:32 |
| lifeless | no | 04:32 |
| tchaypo | wheeee | 04:32 |
| lifeless | https://pypi.python.org/pypi/d2to1 | 04:32 |
| lifeless | https://pypi.python.org/pypi/Distutils2 | 04:33 |
| lifeless | different | 04:33 |
| lifeless | thats scary 1327 downloads in the last month | 04:33 |
| lifeless | of distutils2 | 04:33 |
| lifeless | WTeverlastingF | 04:33 |
| tchaypo | d2to1 (the ‘d’ is for ‘distutils’) | 04:33 |
| lifeless | yes | 04:34 |
| tchaypo | it’s using distutils2 setup.cfg in a distutils1 setup.py | 04:34 |
| tchaypo | anyway | 04:34 |
| lifeless | not quite | 04:34 |
| lifeless | setuptools setup.py | 04:34 |
| lifeless | and actually we're in the 'pbr setup.py' context | 04:34 |
| *** sdake has joined #openstack-infra | 04:34 | |
| lifeless | if you think too hard on this, you'll be ready for our tentacled master to rise from the cthonian deeps | 04:35 |
| *** otter768 has quit IRC | 04:35 | |
| tchaypo | did lovecraft ever play with using extra spacing between paragraphs for dramatic effect? | 04:36 |
| tchaypo | I started reading a book yesterday that has two newlines between paragraphs, and it turns out that’s enough space to have me mentally reset and ready for a new scene, and so I keep getting surprised when the new scene has the same setting and the same characters continuing the same conversation | 04:36 |
| *** sputnik13 has quit IRC | 04:36 | |
| tchaypo | it makes me feel like I (or maybe the characters) keep passing out in the middle of the action | 04:37 |
| lifeless | sure thats not your sleep apnea? | 04:37 |
| tchaypo | it’s a very similar feeling to what I used to get | 04:37 |
| *** ivar-la__ has quit IRC | 04:37 | |
| *** markvoelker has joined #openstack-infra | 04:37 | |
| *** sputnik13 has joined #openstack-infra | 04:38 | |
| *** ivar-lazzaro has joined #openstack-infra | 04:38 | |
| *** sdake_ has quit IRC | 04:38 | |
| tchaypo | so I’m ignoring d2to1 for now | 04:38 |
| tchaypo | it looks like extras is going to need to be added to MULTI_FIELDS | 04:39 |
| mordred | lifeless: sup? | 04:39 |
| lifeless | mordred: the pbr change failed because I deleted requirements.txt AFAICT | 04:39 |
| lifeless | mordred: I wanted a quick touchstone on WHY WE CAN"T HAVE NICE THINGS. | 04:39 |
| tchaypo | is the comment at http://git.openstack.org/cgit/openstack-dev/pbr/tree/pbr/util.py#n46 about this possibly evolving with distutils2 still current? or is that a hangover from the d2to1 legacy? | 04:40 |
| mordred | hrm | 04:40 |
| lifeless | mordred: I note that http://git.openstack.org/cgit/openstack/governance/tree/reference/project-testing-interface.rst#n53 | 04:40 |
| mordred | lifeless: well, we can't have nice things because eek | 04:40 |
| lifeless | mordred: now specifies requirements.txt | 04:40 |
| mordred | yes | 04:40 |
| mordred | it does | 04:41 |
| lifeless | mordred: I think we do need to specify where, because of update.py | 04:41 |
| mordred | aroo? | 04:41 |
| lifeless | mordred: but I don't think our tooling should ever need to install *install_requires* via -r requirements.txt of a repo | 04:41 |
| mordred | tchaypo: hangover | 04:41 |
| mordred | lifeless: well, we currently do in tox.ini | 04:41 |
| mordred | lifeless: although I largely agree with your premise | 04:41 |
| lifeless | mordred: - concretely, since I want move us to having deps in setup.cfg anyhow | 04:41 |
| *** stevemar has joined #openstack-infra | 04:42 | |
| mordred | lifeless: yah - once we're ready for that, I think we can take a PTI change to the TC | 04:42 |
| lifeless | mordred: so, there are two things | 04:42 |
| mordred | whee! two things! | 04:42 |
| lifeless | mordred: one we need a PTI change for (and update.py changes) | 04:42 |
| lifeless | mordred: which is 'how do we introspect in the requirements-management tooling to do updates/consistency checks' | 04:43 |
| lifeless | mordred: the other one is 'how do we install stuff' - and that should just be pip install .[test] everywhere | 04:43 |
| lifeless | mordred: which, once I've validated (and fixed probably) pip for that, isn't a PTI change | 04:43 |
| lifeless | mordred: - IMO anyhow | 04:43 |
| mordred | lifeless: yah. once we're to the point where that works - I think that's a great change to the places where we're doing -r requirements.txt | 04:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add in extra connections tests https://review.openstack.org/169300 | 04:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Refactor sources out of triggers https://review.openstack.org/118993 | 04:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Document the new connections in zuul https://review.openstack.org/169270 | 04:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add gerrit reviews into patchset approvals https://review.openstack.org/97390 | 04:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for negative requirements https://review.openstack.org/102726 | 04:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Configure triggers dynamically https://review.openstack.org/119534 | 04:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for 'connection' concept https://review.openstack.org/121528 | 04:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for triggers https://review.openstack.org/119532 | 04:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for sources https://review.openstack.org/119531 | 04:44 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for reporters https://review.openstack.org/119530 | 04:44 |
| mordred | *patchbomb* | 04:44 |
| lifeless | mordred: so -r requirements.txt should be replaceable by pip install . --only-dependencies | 04:44 |
| lifeless | mordred: -r test-requirements, pip install .[test] --only-dependencies | 04:44 |
| lifeless | mordred: great | 04:44 |
| lifeless | shoved onto my stack | 04:44 |
| mordred | lifeless: right - but we generally don't need to install the deps without the thing | 04:44 |
| mordred | lifeless: EXCEPT | 04:45 |
| *** ivar-lazzaro has quit IRC | 04:45 | |
| lifeless | mordred: in test :) | 04:45 |
| *** ivar-laz_ has joined #openstack-infra | 04:45 | |
| mordred | lifeless: for setup_requires of pbr | 04:45 |
| lifeless | mordred: thats not in requirements.txt today anyhow | 04:45 |
| *** ddieterly has joined #openstack-infra | 04:45 | |
| mordred | lifeless: which is one of the reasons we do -r requirements -r test-requirements in tox.ini - because tox does it before it touches the setup.py | 04:45 |
| mordred | whereas pip install . touches the setup.py before it touches test-requirements.txt | 04:46 |
| mordred | which can trigger easy_install | 04:46 |
| lifeless | bad news for you | 04:46 |
| mordred | I love bad news | 04:46 |
| lifeless | many of our projects don't have a runtime dep on pbr | 04:46 |
| lifeless | so we're vulnerable to easy-install of pbr unless something in their deps happens to bring it in as a runtime dep | 04:47 |
| mordred | this is true - I'm not saying it's a great plan or even correct :) | 04:47 |
| lifeless | today | 04:47 |
| mordred | yah | 04:47 |
| lifeless | so, I'm going to ignore it and treat that as something to explicitly handle | 04:47 |
| mordred | that would be hacking | 04:47 |
| lifeless | pip install pbr .[test] | 04:47 |
| mordred | hacking is in everything's test-requirements.txt | 04:47 |
| mordred | it has a runtime on pbr | 04:47 |
| lifeless | ok | 04:47 |
| mordred | HOWEVER | 04:47 |
| mordred | I like your plan in general | 04:47 |
| mordred | just making sure I say all the things I know about past context in case there are things underneath rocks | 04:48 |
| lifeless | appreciated | 04:48 |
| lifeless | thats why I pinged you | 04:48 |
| lifeless | because I hate rocks. | 04:48 |
| mordred | I have rocks in my head :) | 04:48 |
| lifeless | and cthulu | 04:48 |
| lifeless | wat https://github.com/testing-cabal/testtools/pull/141 | 04:48 |
| mordred | lifeless: so - are you going to make pbr grok test-requirements.txt into an extras_require test as a stepping stone? | 04:49 |
| lifeless | mordred: no | 04:49 |
| mordred | cool | 04:49 |
| lifeless | mordred: I'm going to drink heavily | 04:49 |
| mordred | lifeless: that's the right choice | 04:49 |
| tchaypo | lifeless: so it looks like the bulk of the marshalling is going to happen around http://git.openstack.org/cgit/openstack-dev/pbr/tree/pbr/util.py#n323; and it looks like a way to test this will be to look at the kwargs that function returns | 04:50 |
| *** ddieterly has quit IRC | 04:50 | |
| tchaypo | mordred: while he’s drinking, I’m going to try to make pbr grok extras_require from setup.cfg | 04:50 |
| tchaypo | I think | 04:50 |
| mordred | tchaypo: neat | 04:50 |
| mordred | tchaypo: you'll need alcohol after taht | 04:50 |
| lifeless | tchaypo: sounds plausible | 04:51 |
| lifeless | tchaypo: oh yeah, and about 1000 folk are going to owe us drinks after this. | 04:51 |
| mordred | tchaypo: btw - there are many mentions in the codebase about being compatible with various things - most of those mentions are hysterical | 04:51 |
| mordred | tchaypo: so, you know, if they seem odd - don't be wary of challenging their existence - many may have grown _other_ reasons for being important, but there is some deep dark cruft up in there | 04:52 |
| * mordred hands tchaypo a headlamp and a pickaxe | 04:52 | |
| lifeless | (or just ask me, I am fairly current) | 04:52 |
| mordred | well, that's what I meant | 04:52 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Finish removing invocations of pip. https://review.openstack.org/181785 | 04:53 |
| lifeless | lets see if thats happier | 04:53 |
| lifeless | mordred: so the roadmap thats slowly forming in my head is | 04:53 |
| lifeless | mordred: pbr extras support in setup.cfg | 04:54 |
| lifeless | mordred: update.py extras support, using setup.cfg | 04:54 |
| lifeless | mordred: pip --only-dependencies and pip .[test] support. | 04:54 |
| mordred | lifeless: sounds good - what does "pip --only-dependencies" support mean? | 04:54 |
| lifeless | mordred: pull the trigger on removing all references to *requirements*.txt outside of global-requirements processing | 04:55 |
| mordred | as in - what does supporting that look like other than just typing --only-dependencies ? | 04:55 |
| lifeless | mordred: it won't install the thing itself | 04:55 |
| lifeless | mordred: pip install -r requirements.txt | 04:55 |
| lifeless | mordred: == pip install --only-dependencies . | 04:55 |
| *** afazekas has joined #openstack-infra | 04:55 | |
| lifeless | mordred: (for us) | 04:55 |
| mordred | yes. grok that | 04:55 |
| mordred | just saying - is there _work_ to be done to support that? | 04:55 |
| lifeless | yes, there's an open, approved in principle issue for it | 04:56 |
| lifeless | its fairly shallow | 04:56 |
| mordred | ok. cool | 04:56 |
| lifeless | there's already --ignore-deps | 04:56 |
| lifeless | so this probably becomes --only-deps | 04:56 |
| mordred | AH. | 04:56 |
| lifeless | and is symmetric | 04:56 |
| mordred | the work is "add an option to pip called --only-dependencies" | 04:56 |
| lifeless | yes | 04:56 |
| mordred | I wasn't sure if the work was in pbr or in pip - thanks - I am with you now | 04:56 |
| lifeless | (or $*(bikeshed)) | 04:57 |
| *** yamamoto has quit IRC | 04:57 | |
| mordred | s/$*(bikeshed)/red/ | 04:57 |
| lifeless | a separate arc is to get environment markers in place | 04:57 |
| lifeless | which I've convinced tchaypo is contained enough and relevant enough he's poking at now | 04:57 |
| lifeless | when thats in place, we can start the crusade to remove requirements$version.txt | 04:58 |
| tchaypo | first I’m making sure I can run tests on pbr | 04:58 |
| lifeless | another arc is to get install_requires into setup.cfg | 04:58 |
| tchaypo | then I’ll add a test case that has a setup.cfg with extra_requires and looks for them in kwargs | 04:58 |
| tchaypo | then I’ll make that test not fail | 04:58 |
| lifeless | which will be the last one I think, thats the point where we can move everything into setup.cfg | 04:58 |
| tchaypo | then tomorrow we can see if this translates to something useful | 04:58 |
| *** deepakcs has joined #openstack-infra | 04:58 | |
| lifeless | tchaypo: coool | 04:59 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add in extra connections tests https://review.openstack.org/169300 | 05:00 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Document the new connections in zuul https://review.openstack.org/169270 | 05:00 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Configure triggers dynamically https://review.openstack.org/119534 | 05:00 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for 'connection' concept https://review.openstack.org/121528 | 05:00 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for triggers https://review.openstack.org/119532 | 05:00 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for sources https://review.openstack.org/119531 | 05:00 |
| mordred | lifeless: I'd say we should replace setup.cfg with setup.yaml so that we wouldn't have to play the awful parsing games ... except yaml isn't in the stdlib :( | 05:00 |
| mordred | and json is not for humans | 05:00 |
| lifeless | mordred: yaml is an awful parsing game | 05:00 |
| lifeless | mordred: stdlib inclusion is irrelevant here | 05:00 |
| mordred | lifeless: possibly - but it's a very well defined one | 05:00 |
| lifeless | mordred: with objects. Honestly. | 05:00 |
| mordred | lifeless: as opposed to the meaning we have to put into intra-line syntax in an ini file | 05:00 |
| lifeless | anyhow, we can use arbitrary libraries if we want to, once setup-requires is fixed | 05:01 |
| mordred | well, let's ignore the abritrary object serialization madness | 05:01 |
| mordred | this is try | 05:01 |
| mordred | true | 05:01 |
| mordred | very true | 05:01 |
| mordred | I just don't live in that world in my head yet | 05:01 |
| lifeless | <- has vision | 05:01 |
| mordred | because I'm terrified that if I believe it it'll not come true | 05:01 |
| tchaypo | Could not find a version that satisfies the requirement fixtures>=0.3.14 (from -r /Users/james/src/openstack-dev/pbr/test-requirements.txt (line 3)) (from versions: ) | 05:02 |
| tchaypo | No matching distribution found for fixtures>=0.3.14 (from -r /Users/james/src/openstack-dev/pbr/test-requirements.txt (line 3)) | 05:02 |
| mordred | that sees unlikely | 05:03 |
| tchaypo | it says it couldn’t find fixtures on http://localhost:3141/root/pypi/+simple/fixtures/ (my local devpi instance) | 05:03 |
| tchaypo | but I can see it. I’m assuming it was a timeout.. | 05:03 |
| tchaypo | yep, running it again and it works fine now. | 05:03 |
| mordred | btw - I've been uploading an image to a cloud in china for the last 7 hours or so - impressive thing is - it's still uploading | 05:03 |
| mordred | tchaypo: devpi's transparent grabbing of missing things in the background can occasionally lead to timeouts | 05:04 |
| *** sputnik13 has quit IRC | 05:05 | |
| tchaypo | usually when I’ve seen it fail it has been pretty opaque | 05:05 |
| mordred | yah | 05:05 |
| lifeless | BANDERSNATCH | 05:06 |
| tchaypo | as in, pretty evident what happened. | 05:06 |
| mordred | speaking of bandersnatch - we really need to get jeblair's patch to that landed | 05:06 |
| tchaypo | lifeless: doesn’t do external packages, and even if it did, I’m not wasting >90Gb of laptop space on packages I don’t use. | 05:06 |
| mordred | next time anyone complains about patch velocity in openstack ... | 05:06 |
| tchaypo | bandersnatch has its place, but my laptop isn’t that place | 05:06 |
| lifeless | do you have an SSD ? | 05:07 |
| lifeless | mordred: oh? link me | 05:07 |
| tchaypo | yes, i have an ssd | 05:07 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Add nodejs to the source modules https://review.openstack.org/98029 | 05:07 |
| *** sputnik13 has joined #openstack-infra | 05:08 | |
| lifeless | mordred: how often does HP buy its staff laptops? | 05:08 |
| mordred | lifeless: https://bitbucket.org/pypa/bandersnatch/pull-request/12/add-option-to-dir-hash-index-files/diff | 05:08 |
| mordred | lifeless: almost never | 05:08 |
| lifeless | mordred: 3 years? 4? I have a vague memory of 4. | 05:09 |
| *** chlong has quit IRC | 05:09 | |
| mordred | lifeless: yeah - I think the lifecycle length is actually 4 | 05:09 |
| mordred | which is, of course, _crazy_ | 05:09 |
| mordred | I usually trash a laptop after about 1 year | 05:09 |
| mordred | this one ahs been going for almost 1.5 and is about done | 05:09 |
| lifeless | Christian is normally pretty good | 05:10 |
| tchaypo | my previous laptop is 4 and a bit years old and just starting to act a little tired | 05:10 |
| lifeless | I suspect bitbucket UI fail | 05:10 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/puppet-zuul: Use the latest jquery on zuul https://review.openstack.org/181793 | 05:10 |
| lifeless | jeblair: have you tried pinging ctheune (e.g. via the bitbucket send message feature) ? | 05:11 |
| mordred | lifeless: yah. the last time christian responded, jeblair responded in 2 days. that was in december - since then the codebase has changed and produced conflicts :) | 05:11 |
| mordred | lifeless: which is fine - it happens to the best of us | 05:12 |
| mordred | it's just on the TDL and you said bandersnatch which reminded me | 05:12 |
| *** sdake_ has joined #openstack-infra | 05:13 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool: Remove the need for zmq in proper demand calcs. https://review.openstack.org/98306 | 05:13 |
| mordred | okie. I will now go to the bed | 05:13 |
| *** emagana has joined #openstack-infra | 05:15 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Assume trailing slashes want to load indexes https://review.openstack.org/141260 | 05:16 |
| *** sdake has quit IRC | 05:16 | |
| *** stevemar has quit IRC | 05:17 | |
| *** yamamoto has joined #openstack-infra | 05:17 | |
| tchaypo | ho hum. | 05:17 |
| tchaypo | … actually, no ho. | 05:18 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Assume trailing slashes want to load indexes https://review.openstack.org/141260 | 05:19 |
| *** BharatK has joined #openstack-infra | 05:19 | |
| *** unicell1 has joined #openstack-infra | 05:20 | |
| *** unicell has quit IRC | 05:21 | |
| lifeless | mordred: night | 05:21 |
| *** woodster_ has joined #openstack-infra | 05:24 | |
| *** Alexandra_ has quit IRC | 05:24 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Create devstack plugin for testing os-loganalyze https://review.openstack.org/180553 | 05:25 |
| openstackgerrit | Merged openstack-infra/zuul: Update the description for grey icons on status pg https://review.openstack.org/154690 | 05:26 |
| *** sputnik13 has quit IRC | 05:28 | |
| *** ibiris_away is now known as ibiris | 05:29 | |
| *** heyongli has quit IRC | 05:29 | |
| openstackgerrit | James Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras from setup.cfg https://review.openstack.org/181795 | 05:29 |
| *** yamamoto has quit IRC | 05:30 | |
| *** sputnik13 has joined #openstack-infra | 05:30 | |
| *** heyongli has joined #openstack-infra | 05:31 | |
| tchaypo | so let me go flag that as WIP properly | 05:32 |
| tchaypo | but lifeless does that file format look right to you? | 05:32 |
| *** ildikov has quit IRC | 05:32 | |
| *** camunoz has quit IRC | 05:33 | |
| lifeless | yes | 05:33 |
| *** yamamoto has joined #openstack-infra | 05:33 | |
| lifeless | awful | 05:33 |
| lifeless | but right | 05:33 |
| lifeless | please put a blank line between the sections though | 05:33 |
| *** achanda has joined #openstack-infra | 05:37 | |
| mgedmin | ugh, won't the first '=' be interpreted by the ini parser? | 05:37 |
| *** yamamoto has quit IRC | 05:37 | |
| *** koolhead17 has joined #openstack-infra | 05:39 | |
| *** jogo has quit IRC | 05:39 | |
| lifeless | mgedmin: sufficiently not, that it works | 05:39 |
| lifeless | mgedmin: its ninth circle of hell territory, but wtf, deal the cards we have | 05:39 |
| mgedmin | I can't parse that | 05:39 |
| *** jogo has joined #openstack-infra | 05:39 | |
| *** ibiris is now known as ibiris_away | 05:40 | |
| lifeless | mgedmin: its not interpreted by safeconfigparser | 05:40 |
| lifeless | mgedmin: others might interpret it | 05:40 |
| mgedmin | ah, you mean pbr doesn't use configparser | 05:40 |
| lifeless | we use it | 05:40 |
| mgedmin | my experiments contradict your assertio | 05:43 |
| *** ibiris_away is now known as ibiris | 05:43 | |
| mgedmin | https://dpaste.de/KEw1 | 05:44 |
| *** camunoz has joined #openstack-infra | 05:46 | |
| *** ddieterly has joined #openstack-infra | 05:46 | |
| tchaypo | mgedmin: i dislike your experiment | 05:47 |
| *** btully has joined #openstack-infra | 05:47 | |
| tchaypo | but all the same I’d rather find out now than later | 05:47 |
| mgedmin | it's not the most elegant code I've ever written | 05:47 |
| *** yamamoto has joined #openstack-infra | 05:47 | |
| lifeless | mgedmin: use items, not options | 05:48 |
| lifeless | mgedmin: your experiment is broken :) | 05:48 |
| mgedmin | I get the same result if I replace for k in p.options(s): with for k, v in p.items(s) | 05:49 |
| openstackgerrit | James Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras from setup.cfg https://review.openstack.org/181795 | 05:49 |
| lifeless | mgedmin: so I tested this :) | 05:49 |
| *** shashankhegde has quit IRC | 05:49 | |
| * tchaypo re-marks as wip, since we can’t have nice things | 05:49 | |
| tchaypo | lifeless: the bits you pasted in channel didn’t seem to have == in | 05:49 |
| * mgedmin would like to register an objection to calling that syntax "nice" | 05:49 | |
| lifeless | ut perhaps not enough | 05:49 |
| lifeless | mgedmin: thank you for being paranoid | 05:50 |
| *** ddieterly has quit IRC | 05:50 | |
| mgedmin | s/paranoid/curious | 05:51 |
| lifeless | samesame | 05:51 |
| *** mpopow has quit IRC | 05:51 | |
| mgedmin | I've used 'foo = value that contains embedded = signs' in the past, I know what to expect | 05:51 |
| lifeless | more grist for the approach of putting these things on extras being a tad fugly | 05:53 |
| *** sdake has joined #openstack-infra | 05:57 | |
| *** mahito has quit IRC | 05:58 | |
| *** shashankhegde has joined #openstack-infra | 05:59 | |
| *** Longgeek_ has joined #openstack-infra | 05:59 | |
| tchaypo | hrm. My results differ. | 06:00 |
| *** rmcall has quit IRC | 06:00 | |
| lifeless | tchaypo: from mgedmins? | 06:01 |
| lifeless | I've verified his | 06:01 |
| *** tnovacik has joined #openstack-infra | 06:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/181800 | 06:01 |
| *** sdake_ has quit IRC | 06:01 | |
| tchaypo | https://dpaste.de/oGzh | 06:01 |
| tchaypo | i get different results with options and items | 06:01 |
| tchaypo | neither particularly useful | 06:02 |
| tchaypo | … no I don’t. I was just reading them wrong | 06:02 |
| lifeless | we could possibly use ConfigObj | 06:03 |
| lifeless | if we vendor it | 06:03 |
| mgedmin | question is: do you want to push something like this to be a Python-wide standard? | 06:03 |
| mgedmin | or just a pbr thing | 06:03 |
| lifeless | mgedmin: gosh no | 06:03 |
| *** Longgeek has quit IRC | 06:03 | |
| lifeless | mgedmin: I want something to unbreak the logjam we're in | 06:03 |
| *** mahito_ has joined #openstack-infra | 06:04 | |
| lifeless | mgedmin: which we can preserve as ugly back-compat here indefinitely | 06:04 |
| lifeless | mgedmin: and learn about the problems its going to have (like this!) | 06:04 |
| *** luqas has joined #openstack-infra | 06:04 | |
| mgedmin | maybe namespace the section name then? [pbr:extras] |py26 = ... | 06:05 |
| lifeless | mgedmin: thats a not-unreasonable idea, though we're not namespaced for metadata either | 06:05 |
| lifeless | and thats much more intrusive and widely used | 06:05 |
| tchaypo | it’s not unreasonable, but | 06:06 |
| tchaypo | if we want to support all the flexibility of https://www.python.org/dev/peps/pep-0426/#environment-markers | 06:06 |
| *** yamamoto has quit IRC | 06:06 | |
| lifeless | I've done some experiments | 06:06 |
| lifeless | and interpolation doesn't work in keys | 06:06 |
| lifeless | so we seem to have the following options to me: | 06:06 |
| lifeless | - embed all the extras in one text field | 06:07 |
| tchaypo | it’s easy enough to map “|py26” to “python_version==2.6”, but we need to support an arbitrary subset of python there.. | 06:07 |
| lifeless | - make the keys be ignored, put everything in the text fields | 06:07 |
| lifeless | - use a different container format | 06:07 |
| *** yamamoto has joined #openstack-infra | 06:08 | |
| tchaypo | can that test field be json? I can’t imagine anyone being upset with json embedded in an ini</sarcasm> | 06:08 |
| lifeless | - use a different ini file parser [e.g. configobj with nested things] | 06:08 |
| tchaypo | s/test/text/ | 06:08 |
| lifeless | tchaypo: it probably would be yes | 06:08 |
| tchaypo | I don’t like the idea of relying on the quirks of one particular ini file parser | 06:08 |
| *** koolhead17 has quit IRC | 06:08 | |
| lifeless | tchaypo: the reasons to embed in a text field would be to avoid the dependencies implied by having a nice format for a new file (such as yaml) | 06:08 |
| tchaypo | one of the reasons i get so upset every time someone says “the config file is a standard ini file” is because N developers have N+3 ideas of what “standard ini file format” is | 06:09 |
| mgedmin | haha yes | 06:09 |
| mgedmin | at least in Python land we have ConfigParser in the stdlib | 06:09 |
| *** luqas has quit IRC | 06:09 | |
| *** mrunge has joined #openstack-infra | 06:09 | |
| tchaypo | so if we’re going to use ini files, I’d prefer to stick with ConfigParser. | 06:10 |
| openstackgerrit | Sirushti Murugesan proposed openstack-infra/project-config: Add grenade jobs for Heat https://review.openstack.org/178352 | 06:10 |
| mgedmin | what was the pep that defined environment markers? | 06:10 |
| lifeless | 426 | 06:11 |
| tchaypo | https://www.python.org/dev/peps/pep-0426/#environment-markers | 06:11 |
| lifeless | the markerlib impl is a little incomplete | 06:11 |
| lifeless | its missing <= and < for instance | 06:11 |
| lifeless | but, if we can get the dots joined so things tunnel together properly | 06:11 |
| lifeless | we'll be in good shape | 06:11 |
| lifeless | tchaypo: I think we should get something working | 06:11 |
| tchaypo | yaml is out because using yaml would mean you can’t reliably run setup.py without having a yaml parser installed first. | 06:11 |
| lifeless | tchaypo: and then do a spec and get input on the right way to make it nice | 06:12 |
| lifeless | tchaypo: no, thats not why yaml is out ;) | 06:12 |
| tchaypo | lifeless: you mean, it’s not the only reason yaml is out, or that it’s not a a reason why yaml is out? | 06:12 |
| lifeless | tchaypo: yaml is out because having pbr install_require other libraries leads to the corner cases of easy_install affecting other things | 06:12 |
| lifeless | tchaypo: we already have this dep on easy_install for pbr | 06:13 |
| tchaypo | yep. if yaml was in core it wouldn’t be an issue | 06:13 |
| lifeless | tchaypo: I may be splitting hairs. But your statement was incorrect. | 06:13 |
| lifeless | You can run it reliably. Just not safely. | 06:13 |
| mgedmin | what if you invert the syntax? instead of [extras] env_marker = pkg you do [metadata] requires = foo [env_marker] bar [env_marker] ... ? | 06:14 |
| *** zhiwei has joined #openstack-infra | 06:14 | |
| *** rmcall has joined #openstack-infra | 06:14 | |
| mgedmin | [ and ] are not valid inside env markers (unless inside strings) | 06:14 |
| lifeless | [] define extras | 06:14 |
| mgedmin | darn, they're used for extras | 06:14 |
| mgedmin | uh, {env_marker}? | 06:15 |
| lifeless | so | 06:15 |
| lifeless | the setuptools UI for this is terrible IM | 06:15 |
| lifeless | IMO | 06:15 |
| *** sdake_ has joined #openstack-infra | 06:15 | |
| lifeless | because its not per requirement | 06:15 |
| lifeless | but I think thats been done for graceful backwards comapt | 06:15 |
| lifeless | because parse_requirements has no mechanism defined to make a requirements list contain markers | 06:16 |
| tchaypo | https://dpaste.de/ipJv | 06:16 |
| tchaypo | putting them as a single key seems to work… | 06:16 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add python-cloudpulseclient project for CloudPulse https://review.openstack.org/181761 | 06:17 |
| tchaypo | hrm. | 06:17 |
| lifeless | I don't recall if there's a inter requirement separator | 06:17 |
| lifeless | foo, bar | 06:17 |
| *** sdake__ has joined #openstack-infra | 06:18 | |
| *** sdake has quit IRC | 06:18 | |
| *** koolhead17 has joined #openstack-infra | 06:18 | |
| lifeless | tchaypo: https://dpaste.de/JSNH | 06:19 |
| lifeless | tchaypo: you can parse that as a human. Its complex enough I'd want to write a proper parser for it I think | 06:19 |
| lifeless | tchaypo: we avoid most of the complexity today by one-line-per-requirements, and use parse_requirements | 06:19 |
| tchaypo | that’s slightly different to my idea | 06:20 |
| tchaypo | one moment | 06:20 |
| lifeless | tchaypo: that was representing one extra | 06:21 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/181800 | 06:21 |
| lifeless | tchaypo: with a marker and four dependencies | 06:21 |
| clarkb | what happened to $requirement>=x.y ; python == 2.6 ? I want to say that was supposed to be the thing? (or is it just too hard to make pip honor that right now) | 06:21 |
| *** sdake_ has quit IRC | 06:21 | |
| tchaypo | https://dpaste.de/787N | 06:21 |
| lifeless | clarkb: thats the syntax for requirements.txt | 06:21 |
| lifeless | clarkb: which is not that for setuptools | 06:21 |
| lifeless | clarkb: also its python_version=="2.6" :) | 06:22 |
| clarkb | ugh so we get two different overlapping things that are both bad? | 06:22 |
| lifeless | clarkb: three or four eventually probably | 06:22 |
| lifeless | so there are two bad things here | 06:22 |
| lifeless | firstly, setuptools doesn't support environment markers on requires | 06:23 |
| lifeless | you can't say 'requirement[EXTRAS] VERSIONSTUFF ; marker' | 06:23 |
| BharatK | Hi, After running GlusterFS CI I am getting all "/var/log/glusterfs" logs of ZERO size. http://logs.openstack.org/00/180400/10/check/check-tempest-dsvm-full-glusterfs-nv/10729fb/logs/glusterfs/ | 06:23 |
| *** yfried_ is now known as yfried|afk | 06:24 | |
| lifeless | secondly | 06:24 |
| lifeless | you can currently only talk to setuptools via setup.cfg | 06:24 |
| *** yfried|afk is now known as yfried_ | 06:24 | |
| lifeless | erm | 06:24 |
| lifeless | setup.py | 06:24 |
| lifeless | setup.cfg [metadata] and requirements.txt are both all pbr things | 06:24 |
| tchaypo | *checks* yep, my version seems to work | 06:24 |
| tchaypo | https://dpaste.de/L4wb | 06:24 |
| *** jamespage_ has quit IRC | 06:24 | |
| lifeless | we need to do related but distinct things | 06:25 |
| tchaypo | by work, I mean that ‘ordinary’ extras can be simple key=value lines, it’s only env markers that need special handling | 06:25 |
| lifeless | a) we need to make it possible for users to define extras in the pbr world outside of setup.py, including environment markers. | 06:25 |
| clarkb | yes a should be my example | 06:25 |
| lifeless | b) we need to reflect environment markers in requirements.txt requirements into the extras | 06:25 |
| clarkb | pbr can inject whatever setuptools wants | 06:26 |
| lifeless | pbr has no means to define extras today at all | 06:26 |
| clarkb | buts use my example as a source then you dont write this twice in two terrible languages | 06:26 |
| lifeless | the subset of extras that is environment markers on install_requires | 06:26 |
| lifeless | clarkb: there is no defined language for extras for pbr today | 06:26 |
| *** scheuran has joined #openstack-infra | 06:26 | |
| clarkb | Im not talking extras, just markers | 06:26 |
| lifeless | clarkb: we're talking extras | 06:27 |
| clarkb | then why all the effort with markers? | 06:27 |
| clarkb | you will just end up with two places defining the same info | 06:27 |
| lifeless | clarkb: because a new thing to do extras that can't also do markers will be immediately obsolete | 06:27 |
| lifeless | clarkb: no we won't | 06:27 |
| lifeless | clarkb: I think there's a disconnect here. I don't want to have a back-and-fro debate while we tease it out | 06:28 |
| clarkb | yes any pip markers will go in requirements.txt then again in setup.cfg for setuptools | 06:28 |
| lifeless | clarkb: can I ask you let us roll with it and assume some clue? | 06:28 |
| clarkb | sure | 06:28 |
| lifeless | thanks | 06:28 |
| lifeless | to answer that concern, you wouldn't put an install_requires entry in via markers if you're using pbr | 06:28 |
| lifeless | because pbr will be getting taught to marshall that smartly | 06:29 |
| lifeless | but we have a feature request for pbr to do markers for things like backends | 06:29 |
| lifeless | s/markers/extras/ | 06:29 |
| lifeless | e.g. sqlite deps if you ask for the sqlite backend | 06:29 |
| lifeless | mysql for mysql etc | 06:29 |
| lifeless | *those* also need to support markers. | 06:29 |
| lifeless | So we need a good solid proper implementation of extras that supports markers. | 06:30 |
| tchaypo | Are we still planning to have pbr be able to read install_requires from setup.cfg instead of requirements.txt | 06:31 |
| tchaypo | ? | 06:31 |
| lifeless | tchaypo: thats a separate arc of work | 06:31 |
| *** jcoufal has joined #openstack-infra | 06:31 | |
| lifeless | tchaypo: its marginal benefit, long term alignment as a goal there. | 06:31 |
| clarkb | I would say its desireable for consistency but also separate | 06:31 |
| tchaypo | but if it happens, we’d need to end up being able to read the markers out of setup.cfg anyway, right? | 06:31 |
| lifeless | tchaypo: yes, and that suggests another option | 06:32 |
| tchaypo | I agree that it’s seperate, but also it seems desireable to end up with a format that’s still going to make sense when we’re reading install_requires from setup.cfg | 06:32 |
| lifeless | tchaypo: which is to do the somewhat convluted logic of translating things upfront | 06:32 |
| clarkb | tchaypo only if we assume extras go in setup.cfg | 06:33 |
| lifeless | tchaypo: the install_requires 'format' of one-per-line is much easier than extras which is a dict of one-per-line things | 06:33 |
| tchaypo | agreed. ConfigParser seems to be pretty insistent on splitting on = though | 06:34 |
| lifeless | tchaypo: not in values | 06:34 |
| clarkb | which is a decent assumption except for how the initial work around this happened | 06:34 |
| tchaypo | argparser>1.0 ; python_version==2.6 <<- splits on the first = | 06:34 |
| lifeless | clarkb: there was a thread about moving to setup.cfg on the list; if you disagree, could you chime in there? | 06:34 |
| lifeless | tchaypo: yes, but thats only if you give it as a key=value | 06:35 |
| lifeless | tchaypo: when its all a value its fine | 06:35 |
| tchaypo | oh sure | 06:35 |
| tchaypo | when it’s *all* value there’s no problem | 06:35 |
| tchaypo | except that we need to manually re-parse later | 06:35 |
| clarkb | I dont disagree | 06:35 |
| lifeless | tchaypo: install_requires = \n argparse>1.0;python_version=="2.6" | 06:35 |
| clarkb | I am just saying that oslo wrnt in a very different direction to start | 06:35 |
| tchaypo | … oslo has done work on this already? | 06:35 |
| lifeless | clarkb: oslo.config? | 06:36 |
| clarkb | lifeless oslo in general, the $foothing-requirements.txt | 06:36 |
| *** otter768 has joined #openstack-infra | 06:36 | |
| StevenK | lifeless: Not trying to distract you, but you say your unpolished resolver "chews away at the problem" -- is it actually slow at calculating everything? | 06:36 |
| lifeless | clarkb: AFAIK thats all pbr | 06:36 |
| lifeless | StevenK: 60 seconds | 06:36 |
| clarkb | they even wrote a change to pbr that was rejected to better support ^ as a general standin for extrad | 06:36 |
| clarkb | lifeless oslo was using it for oslo | 06:36 |
| StevenK | Longer than I thought, yikes. | 06:36 |
| lifeless | clarkb: yes, oslo designed that in isolation | 06:37 |
| clarkb | I know | 06:37 |
| lifeless | clarkb: I'm aware of it, and meeting their needs for extras is why we're doing this... | 06:37 |
| *** soren_ has joined #openstack-infra | 06:37 | |
| lifeless | clarkb: I'm not sure what your point is. | 06:37 |
| clarkb | yes... | 06:37 |
| clarkb | my point is that we assume install reqs should go in setup.cfg but only because we are putting extras there and that isnt required | 06:38 |
| clarkb | its merely how we have chosen to solve this | 06:38 |
| lifeless | if you look at the backscroll | 06:39 |
| lifeless | we're not assuming that | 06:39 |
| lifeless | we were talking around different tradeoffs just before you joined | 06:39 |
| lifeless | no, where were we | 06:39 |
| lifeless | tchaypo: so - let me put a sketch together | 06:40 |
| *** shardy has joined #openstack-infra | 06:40 | |
| *** otter768 has quit IRC | 06:41 | |
| *** shashankhegde has quit IRC | 06:41 | |
| *** Ala has joined #openstack-infra | 06:43 | |
| *** yamamoto_ has joined #openstack-infra | 06:45 | |
| lifeless | tchaypo: https://dpaste.de/rb3f | 06:46 |
| lifeless | mgedmin: ^ | 06:46 |
| *** ddieterly has joined #openstack-infra | 06:47 | |
| mgedmin | this is nicer than the original suggestion | 06:47 |
| mgedmin | one thing to check: doesn't ConfigParser intepret ; as comments and strip everything from ; until EOL? | 06:48 |
| mgedmin | (hopefully not) | 06:48 |
| tchaypo | no | 06:48 |
| tchaypo | i tested that | 06:48 |
| *** yamamoto has quit IRC | 06:48 | |
| tchaypo | https://www.irccloud.com/pastebin/rms3TG37 | 06:48 |
| tchaypo | mgedmin: that last string there isn’t useful, but it at least shows configparser didn’t treat things after the ; as a comment | 06:49 |
| tchaypo | lifeless: we’re going to have to de-dupe keys when building that dict | 06:51 |
| lifeless | tchaypo: union | 06:51 |
| lifeless | tchaypo: yes | 06:51 |
| lifeless | tchaypo: we need to do that for the install_requires marker handling in requirements.txt too | 06:51 |
| lifeless | tchaypo: we might want to do that patch as the first in the series | 06:51 |
| mgedmin | tchaypo, I really hate to contradict you ... | 06:51 |
| *** ddieterly has quit IRC | 06:52 | |
| mgedmin | https://dpaste.de/nquG | 06:52 |
| *** mpopow has joined #openstack-infra | 06:52 | |
| lifeless | mgedmin: does : work ? | 06:52 |
| *** mpaolino has joined #openstack-infra | 06:52 | |
| mgedmin | : more or less the same as = to configparser, i.e. you can have key: value instead of key = value; additional : inside value aren't special | 06:52 |
| mgedmin | apparently ; is valid inside a key but is treated as a comment inside the value | 06:53 |
| lifeless | mgedmin: but : should work | 06:54 |
| mgedmin | yes | 06:55 |
| * mgedmin tests | 06:55 | |
| mgedmin | reqs = foo : marker came through unscathed | 06:55 |
| *** mpopow has quit IRC | 06:56 | |
| *** ildikov has joined #openstack-infra | 06:57 | |
| lifeless | yeah | 06:59 |
| tchaypo | bah. | 07:00 |
| tchaypo | naturally I only tested with it in a key. | 07:00 |
| lifeless | so we map requirements to extras_require={'': requirements_txt_contents} | 07:00 |
| lifeless | and the transform to expandout markers from inside that, taking either : or ; to delimit | 07:00 |
| lifeless | then we can add native extras support trivially | 07:00 |
| *** markus_z has joined #openstack-infra | 07:02 | |
| lifeless | ahha | 07:04 |
| *** dizquierdo has joined #openstack-infra | 07:04 | |
| *** sergsh has joined #openstack-infra | 07:05 | |
| *** luqas has joined #openstack-infra | 07:05 | |
| lifeless | tchaypo: right, will pickup discussion tomorrow - I think we've broken the back of this. Thanks mgedmin and clarkb | 07:09 |
| tchaypo | yep. | 07:09 |
| *** hashar has joined #openstack-infra | 07:09 | |
| *** luqas has quit IRC | 07:10 | |
| *** panda|afk has quit IRC | 07:13 | |
| *** ibiris is now known as ibiris_away | 07:14 | |
| tchaypo | in a moment, my review will be updated with an example of setup.cfg that I think is what we want | 07:14 |
| *** panda|afk has joined #openstack-infra | 07:14 | |
| *** fedexo has quit IRC | 07:14 | |
| *** achanda has quit IRC | 07:15 | |
| openstackgerrit | James Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras from setup.cfg https://review.openstack.org/181795 | 07:15 |
| *** jtomasek has joined #openstack-infra | 07:15 | |
| tchaypo | I’ve given it it two test+env_marker and one plain test, just to check that we’re making the union properly, and I’ve added another example of an extra that has several requriements but no env markers to make sure we handle that too | 07:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Propose translations: Fix handling of deleted files https://review.openstack.org/181810 | 07:17 |
| lifeless | tchaypo: so I suggest you use made up names | 07:18 |
| lifeless | like python26_only_one and python26_only_two | 07:18 |
| lifeless | tchaypo: (also see my review) | 07:18 |
| tchaypo | which review? | 07:19 |
| lifeless | 181795 | 07:19 |
| *** jamespage_ has joined #openstack-infra | 07:20 | |
| tchaypo | oh, your comment on my review | 07:21 |
| *** ibiris_away is now known as ibiris | 07:22 | |
| lifeless | StevenK: resolving is NP-C in the limit, so slower is expected. | 07:24 |
| lifeless | StevenK: its not been profiled yet | 07:24 |
| StevenK | lifeless: Yeah, I'm not sure what number I was expecting, but ~60 seconds surprised me. | 07:25 |
| StevenK | Maybe I'm spoiled rotten by apt/aptitude and their C++-ness | 07:25 |
| openstackgerrit | James Polley proposed openstack-dev/pbr: WIP: Teach pbr to read extras from setup.cfg https://review.openstack.org/181795 | 07:25 |
| lifeless | StevenK: see my tweet | 07:26 |
| lifeless | StevenK: I put comparison data up | 07:26 |
| lifeless | StevenK: they also have a much more constrained DB | 07:26 |
| lifeless | StevenK: and cheap local data | 07:26 |
| StevenK | I don't use twitter, but let's see if I can scrape it | 07:26 |
| lifeless | StevenK: pip has hundreds of versions of some packages | 07:26 |
| lifeless | StevenK: https://twitter.com/rbtcollins/status/597536282164596737 | 07:27 |
| StevenK | Hah, when pip develop already takes 2 minutes ... | 07:28 |
| *** sdake__ has quit IRC | 07:28 | |
| *** lennyb has joined #openstack-infra | 07:29 | |
| *** mpavone has joined #openstack-infra | 07:30 | |
| *** arxcruz has joined #openstack-infra | 07:31 | |
| StevenK | lifeless: However, that tweet doesn't show up on https://twitter.com/rbtcollins | 07:31 |
| *** ildikov has quit IRC | 07:31 | |
| *** koolhead17 has quit IRC | 07:31 | |
| StevenK | But clearly that page is not what is supposed to be consumed. | 07:31 |
| *** spredzy|afk is now known as spredzy | 07:32 | |
| *** yamamoto_ has quit IRC | 07:33 | |
| *** soren_ has quit IRC | 07:33 | |
| *** soren_ has joined #openstack-infra | 07:33 | |
| *** luqas has joined #openstack-infra | 07:33 | |
| *** soren_ is now known as soren | 07:33 | |
| lifeless | StevenK: yeah that page is a lie | 07:34 |
| *** AJaeger has quit IRC | 07:35 | |
| *** yfried_ is now known as yfried|afk | 07:36 | |
| *** koolhead_ has joined #openstack-infra | 07:37 | |
| lifeless | tchaypo: ok, so ping me when you get up tomorrow | 07:37 |
| *** mahito_ has quit IRC | 07:38 | |
| *** imcsk8 has joined #openstack-infra | 07:38 | |
| *** luqas has quit IRC | 07:38 | |
| *** yfried|afk is now known as yfried_ | 07:38 | |
| *** jamespage_ has quit IRC | 07:39 | |
| lifeless | tchaypo: yes, that looks plausible | 07:40 |
| *** woodster_ has quit IRC | 07:40 | |
| *** ihrachyshka has joined #openstack-infra | 07:43 | |
| *** achanda has joined #openstack-infra | 07:45 | |
| *** ivar-laz_ has quit IRC | 07:45 | |
| lennyb | Hi, We've started to get 'ERROR zuul.GerritEventConnector: Received unrecognized event type 'ref-replicated' from Gerrit. ' after recent Gerrit update and our zuul update. any ideas | 07:45 |
| *** ivar-lazzaro has joined #openstack-infra | 07:46 | |
| *** mahito has joined #openstack-infra | 07:46 | |
| *** yfried_ has quit IRC | 07:46 | |
| *** jlanoux has joined #openstack-infra | 07:46 | |
| *** ildikov has joined #openstack-infra | 07:47 | |
| *** AJaeger has joined #openstack-infra | 07:48 | |
| *** AJaeger has joined #openstack-infra | 07:48 | |
| *** ddieterly has joined #openstack-infra | 07:48 | |
| *** rmcall has quit IRC | 07:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate: [WIP] run centos7 with neutron https://review.openstack.org/179689 | 07:50 |
| *** dizquierdo has quit IRC | 07:50 | |
| *** rmcall has joined #openstack-infra | 07:52 | |
| *** mpopow has joined #openstack-infra | 07:52 | |
| *** ddieterly has quit IRC | 07:53 | |
| mrda | Hey infra, wondering if you could help me with a problem I'm seeing. I added "appdirs>=1.4.0" to requirements.txt in python-ironicclient and now I'm getting an error that "...site-packages/pip-1.5.4.dist-info/METADATA: No such file or directory". See http://logs.openstack.org/74/173674/9/check/gate-python-ironicclient-pypy/c570d4a/console.html Any ideas on what I'm doing wrong? | 07:53 |
| mrda | Or has anyone seen anything like it in the gate before? | 07:54 |
| *** yamamoto has joined #openstack-infra | 07:54 | |
| mrda | (btw, works locally in a pypy tox job, but that's with pip 6.1.1 rather than 1.5.4 in gate) | 07:56 |
| *** mpopow has quit IRC | 07:57 | |
| *** fifieldt has joined #openstack-infra | 07:57 | |
| *** fifieldt has quit IRC | 07:58 | |
| *** jistr has joined #openstack-infra | 07:58 | |
| *** achanda has quit IRC | 08:01 | |
| *** devvesa has joined #openstack-infra | 08:03 | |
| lifeless | mrda: thats due to too-old setuptools in that venv | 08:03 |
| lifeless | mrda: it was discussed on the list | 08:03 |
| lifeless | mrda: they're looking for volunteers to debug and fix | 08:03 |
| mrda | oh, missed that one | 08:03 |
| mrda | thanks for answering! | 08:04 |
| lifeless | mrda: and clarkb has put up a patch that may fix it (but may need some iterations) | 08:04 |
| lifeless | mrda: http://lists.openstack.org/pipermail/openstack-dev/2015-May/063720.html | 08:04 |
| mrda | lifeless: you are a very useful engine | 08:05 |
| lifeless | mrda: thanks, I think | 08:05 |
| mrda | ;-) | 08:05 |
| lifeless | mrda: https://review.openstack.org/#/c/181558/ | 08:06 |
| *** nithyag_ has quit IRC | 08:12 | |
| *** koolhead_ is now known as koolhead17 | 08:13 | |
| *** koolhead17 has joined #openstack-infra | 08:13 | |
| lennyb | mrda: we solved a similar issue by removing cache in ~jenkins/.cache ... | 08:14 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update Kolla Meeting https://review.openstack.org/181790 | 08:14 |
| openstackgerrit | Qin Zhao proposed openstack/requirements: Update sqlalchemy-migrate to 0.9.6 https://review.openstack.org/181821 | 08:15 |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Query all jobs once https://review.openstack.org/181364 | 08:21 |
| *** derekh has joined #openstack-infra | 08:22 | |
| *** yfried has joined #openstack-infra | 08:24 | |
| *** Hal has joined #openstack-infra | 08:25 | |
| *** Hal is now known as Guest13826 | 08:25 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/jenkins-job-builder: Query all jobs once https://review.openstack.org/181364 | 08:26 |
| *** AJaeger has quit IRC | 08:26 | |
| *** ibiris is now known as ibiris_away | 08:27 | |
| *** AJaeger has joined #openstack-infra | 08:28 | |
| *** AJaeger has joined #openstack-infra | 08:28 | |
| *** btully has quit IRC | 08:28 | |
| *** rossella_s has quit IRC | 08:28 | |
| *** fhubik has joined #openstack-infra | 08:28 | |
| *** ihrachyshka has quit IRC | 08:30 | |
| *** dguitarbite has joined #openstack-infra | 08:32 | |
| *** Ala has quit IRC | 08:34 | |
| *** MaxV_ has joined #openstack-infra | 08:35 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add in F21 nonha job https://review.openstack.org/181136 | 08:37 |
| *** otter768 has joined #openstack-infra | 08:37 | |
| *** gilliard_afk is now known as gilliard | 08:37 | |
| *** rossella_s has joined #openstack-infra | 08:41 | |
| *** otter768 has quit IRC | 08:42 | |
| *** ajo has joined #openstack-infra | 08:42 | |
| *** ociuhandu has joined #openstack-infra | 08:44 | |
| *** yamamoto has quit IRC | 08:45 | |
| *** achanda has joined #openstack-infra | 08:45 | |
| *** ddieterly has joined #openstack-infra | 08:49 | |
| openstackgerrit | Cyril Roelandt proposed openstack-infra/project-config: Enable a non-voting py34 check for Neutron https://review.openstack.org/181046 | 08:50 |
| *** fawadkhaliq has joined #openstack-infra | 08:52 | |
| *** ddieterly has quit IRC | 08:53 | |
| *** Ala has joined #openstack-infra | 09:00 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Start using cgit::lb class https://review.openstack.org/181829 | 09:01 |
| *** ociuhandu has quit IRC | 09:03 | |
| *** dizquierdo has joined #openstack-infra | 09:03 | |
| openstackgerrit | Stuart McLaren proposed openstack/requirements: Bump eventlet version to 0.17.4 https://review.openstack.org/181830 | 09:04 |
| *** ssam2 has joined #openstack-infra | 09:05 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Start using puppet-cgit to manage gitrc https://review.openstack.org/181835 | 09:09 |
| *** hichihara has quit IRC | 09:09 | |
| *** mrunge has quit IRC | 09:09 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Start using puppet-cgit to manage gitrc https://review.openstack.org/181835 | 09:13 |
| *** samuelBartel has joined #openstack-infra | 09:13 | |
| *** mrunge has joined #openstack-infra | 09:16 | |
| *** koolhead17 has quit IRC | 09:22 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Upgrade selinux version to 0.5.3 https://review.openstack.org/181841 | 09:22 |
| *** koolhead17 has joined #openstack-infra | 09:22 | |
| *** koolhead17 has quit IRC | 09:22 | |
| *** koolhead17 has joined #openstack-infra | 09:22 | |
| *** yamamoto has joined #openstack-infra | 09:24 | |
| *** sergsh is now known as sshnaidm | 09:25 | |
| *** pblaho has joined #openstack-infra | 09:25 | |
| *** derekh has quit IRC | 09:27 | |
| *** ildikov has quit IRC | 09:27 | |
| *** koolhead17 has quit IRC | 09:27 | |
| *** derekh has joined #openstack-infra | 09:28 | |
| openstackgerrit | Merged openstack-infra/storyboard: Email Outbox https://review.openstack.org/151413 | 09:29 |
| openstackgerrit | Merged openstack-infra/storyboard: Email base plugin. https://review.openstack.org/151948 | 09:29 |
| *** cdent has joined #openstack-infra | 09:31 | |
| *** aswadr has joined #openstack-infra | 09:33 | |
| shardy | Hi all, we have a problem with gate-python-heatclient-pypy, it's failing with "error: option --single-version-externally-managed not recognized", appears to be distutils related | 09:34 |
| shardy | I was wondering if anyone knows of any openstack-wide issues similar to that, or how we may fix it | 09:34 |
| shardy | It looks a lot like bug #1290562, but that was closed many months ago | 09:35 |
| openstack | bug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] https://launchpad.net/bugs/1290562 - Assigned to Ildiko Vancsa (ildiko-vancsa) | 09:35 |
| shardy | I raised bug #1453095 against heatclient so I can track figuring out what's causing the failure | 09:35 |
| openstack | bug 1453095 in python-heatclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,New] https://launchpad.net/bugs/1453095 | 09:35 |
| *** achanda has quit IRC | 09:40 | |
| *** mpopow has joined #openstack-infra | 09:41 | |
| *** ZZelle has joined #openstack-infra | 09:42 | |
| *** xnox has quit IRC | 09:42 | |
| *** ildikov has joined #openstack-infra | 09:43 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Update to search UI. https://review.openstack.org/178003 | 09:43 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Renamed result-set-size directive https://review.openstack.org/178004 | 09:45 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Result set paging update. https://review.openstack.org/178005 | 09:46 |
| *** mpopow has quit IRC | 09:46 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Story Search Results now show loading spinner. https://review.openstack.org/178867 | 09:47 |
| *** zhiwei has quit IRC | 09:48 | |
| *** xnox has joined #openstack-infra | 09:48 | |
| *** pdangur has joined #openstack-infra | 09:48 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Initial proposal for integration tests https://review.openstack.org/156217 | 09:48 |
| *** ddieterly has joined #openstack-infra | 09:49 | |
| pdangur | hello, i need some help with jjb: i'd like to delete jobs that were removed in the YAML conf | 09:50 |
| pdangur | can anyone please help? | 09:50 |
| openstackgerrit | Merged openstack-infra/storyboard: Email Send Cron Worker https://review.openstack.org/151416 | 09:52 |
| openstackgerrit | Merged openstack-infra/storyboard: Redirect Browsers from API to Client https://review.openstack.org/155551 | 09:53 |
| openstackgerrit | Merged openstack-infra/storyboard: Added email preferences. https://review.openstack.org/179507 | 09:53 |
| *** rlandy has joined #openstack-infra | 09:53 | |
| *** ddieterly has quit IRC | 09:53 | |
| AJaeger | pdangur: you want to remove jobs from jenkins? | 09:54 |
| AJaeger | jjb does not handle that | 09:54 |
| openstackgerrit | Merged openstack-infra/storyboard: Added worker resource resolution helper. https://review.openstack.org/179508 | 09:54 |
| openstackgerrit | Merged openstack-infra/storyboard: Updated oslo.config to 1.11.0 https://review.openstack.org/179509 | 09:54 |
| pdangur | AJaeger: i know. i'd like to come up with a rather quick way to examine changes to infer which jobs to remove | 09:54 |
| pdangur | AJaeger: is there a way to get the list of jobs configured in yaml? | 09:55 |
| pdangur | AJaeger: (not from jenkins) | 09:55 |
| AJaeger | pdangur: sure, have a look at how we gate project-config. | 09:56 |
| AJaeger | for example these tests: http://logs.openstack.org/50/181750/1/check/project-config-compare-xml/03a98b1/console.html | 09:56 |
| *** fawadkhaliq has quit IRC | 09:56 | |
| AJaeger | or http://logs.openstack.org/50/181750/1/check/gate-project-config-layout/fefee78/console.html - both generate the list of current jobs | 09:57 |
| *** ihrachyshka has joined #openstack-infra | 10:00 | |
| pdangur | AJaeger: not sure what i'm seeing here | 10:00 |
| *** Guest13826 has quit IRC | 10:01 | |
| AJaeger | pdangur: invocations that show you how to get the list of jobs | 10:01 |
| pdangur | AJaeger: is there a summary of the current job somewhere in this console? | 10:01 |
| AJaeger | pdangur: no | 10:01 |
| AJaeger | pdangur: you can run these locally using tox on project-config repo | 10:01 |
| *** dimsum__ has joined #openstack-infra | 10:02 | |
| therve | So pypy jobs are failing accross the board, we've noticed 2 different errors | 10:04 |
| therve | They started happening on thursday | 10:04 |
| therve | Any idea what might be going on? | 10:04 |
| *** shardy_ has joined #openstack-infra | 10:04 | |
| *** shardy has quit IRC | 10:06 | |
| *** shardy_ has quit IRC | 10:10 | |
| pdangur | AJaeger: can you please point me to the actual invocation? i can't find it | 10:10 |
| *** shardy has joined #openstack-infra | 10:10 | |
| *** MaxV_ has quit IRC | 10:12 | |
| AJaeger | pdangur: What exactly do you mean? cd openstack-infra/project-config;tox -e jjb will show you the jobs | 10:12 |
| *** pc_m has joined #openstack-infra | 10:13 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/git-review: Allow to specify which commit to send https://review.openstack.org/139701 | 10:14 |
| *** Ala has quit IRC | 10:15 | |
| *** NithyaG has joined #openstack-infra | 10:20 | |
| *** dguitarbite has quit IRC | 10:24 | |
| *** dguitarbite has joined #openstack-infra | 10:27 | |
| *** pelix has joined #openstack-infra | 10:30 | |
| *** fawadkhaliq has joined #openstack-infra | 10:31 | |
| *** samueldmq has joined #openstack-infra | 10:33 | |
| samueldmq | morning | 10:33 |
| *** mahito has quit IRC | 10:36 | |
| samueldmq | hmm, now gerrit defines a size metric for reviews | 10:36 |
| *** mahito has joined #openstack-infra | 10:36 | |
| samueldmq | what are the thresholds ? | 10:36 |
| *** baoli has joined #openstack-infra | 10:37 | |
| *** otter768 has joined #openstack-infra | 10:38 | |
| *** baoli has quit IRC | 10:41 | |
| *** moshele has joined #openstack-infra | 10:41 | |
| *** otter768 has quit IRC | 10:43 | |
| *** mahito has quit IRC | 10:44 | |
| moshele | hi, how can I create branch in my stackforge repository? I tried to follow http://docs.openstack.org/infra/manual/drivers.html but is seem I don't have admin in the gerrit | 10:45 |
| *** ams__ has quit IRC | 10:47 | |
| *** coolsvap is now known as coolsvap|afk | 10:48 | |
| *** ddieterly has joined #openstack-infra | 10:50 | |
| *** ddieterly has quit IRC | 10:55 | |
| openstackgerrit | Merged openstack-dev/pbr: Advertise support for Python3.4 https://review.openstack.org/181525 | 10:56 |
| *** tnovacik has quit IRC | 10:57 | |
| AJaeger | moshele: which repo is this? | 10:58 |
| moshele | AJaeger: networking-mlnx | 10:58 |
| AJaeger | moshele: you're right, there's no permission at https://review.openstack.org/#/admin/projects/stackforge/networking-mlnx,access | 10:59 |
| *** ildikov has quit IRC | 10:59 | |
| moshele | AJaeger: ok, can you add me? | 11:00 |
| *** _nadya_ has joined #openstack-infra | 11:00 | |
| AJaeger | moshele: feel free to review my patch https://review.openstack.org/#/c/181646/ and send a patch for the project-config repo to update your repository | 11:00 |
| moshele | AJaeger: ok thanks | 11:00 |
| AJaeger | moshele: and then create permission is for the release group - so talk with Nurit Vilosny (unless that's you ;) | 11:01 |
| moshele | AJaeger: ok thanks | 11:01 |
| *** ibiris_away is now known as ibiris | 11:03 | |
| moshele | AJaeger: also can you change that I will be ptl of networking-mlnx instead of Nurit Vilosny | 11:06 |
| AJaeger | moshele: Nurit can add you and remove himself | 11:07 |
| moshele | AJaeger: ok thanks | 11:08 |
| AJaeger | moshele: just tell her/him to use the URL above and click on the group | 11:08 |
| moshele | AJaeger: ok | 11:08 |
| *** dalgaaf has joined #openstack-infra | 11:08 | |
| AJaeger | and then add you | 11:09 |
| *** panda|afk has quit IRC | 11:13 | |
| *** panda|afk has joined #openstack-infra | 11:14 | |
| *** ildikov has joined #openstack-infra | 11:15 | |
| *** mpaolino has quit IRC | 11:15 | |
| *** _nadya_ has quit IRC | 11:17 | |
| *** doug-fish has joined #openstack-infra | 11:18 | |
| *** fhubik has quit IRC | 11:19 | |
| *** fhubik_afk has quit IRC | 11:19 | |
| *** fhubik has joined #openstack-infra | 11:21 | |
| *** marcusvrn has joined #openstack-infra | 11:22 | |
| *** marcusvrn1 has joined #openstack-infra | 11:24 | |
| *** mrmartin has joined #openstack-infra | 11:24 | |
| *** fhubik is now known as fhubik_afk | 11:26 | |
| *** marcusvrn has quit IRC | 11:27 | |
| openstackgerrit | Moshe Levi proposed openstack-infra/project-config: Add create rule to networking-mlnx https://review.openstack.org/181862 | 11:27 |
| *** EmilienM|afk is now known as EmilienM | 11:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Check resolver pip on global-requirements changes https://review.openstack.org/181774 | 11:28 |
| openstackgerrit | Merged openstack-infra/project-config: Enable a non-voting py34 check for Neutron https://review.openstack.org/181046 | 11:28 |
| *** mpopow has joined #openstack-infra | 11:30 | |
| *** mmedvede_ has joined #openstack-infra | 11:30 | |
| AJaeger | sdague, fungi, pleia2: Could you review https://review.openstack.org/181810, please? | 11:31 |
| *** jaypipes has joined #openstack-infra | 11:32 | |
| sdague | does that mean deletes are never sent? | 11:32 |
| openstackgerrit | Merged openstack-infra/project-config: Add in F21 nonha job https://review.openstack.org/181136 | 11:33 |
| sdague | oh, I think I understand now | 11:33 |
| *** moshele has left #openstack-infra | 11:35 | |
| *** ociuhandu has joined #openstack-infra | 11:35 | |
| *** ldnunes has joined #openstack-infra | 11:36 | |
| openstackgerrit | Mikhail Dubov proposed openstack-infra/project-config: Add Rally job to Murano https://review.openstack.org/181343 | 11:37 |
| *** zul has quit IRC | 11:37 | |
| *** zul has joined #openstack-infra | 11:38 | |
| *** mpopow has quit IRC | 11:41 | |
| sdague | so, the pypy jobs are also exploding requirements at the moment | 11:45 |
| mordred | sdague: they seem to be exploding across the board | 11:49 |
| *** deepakcs has quit IRC | 11:49 | |
| *** notnownikki has joined #openstack-infra | 11:49 | |
| sdague | is there a compelling reason to not just remove pypy from our tool chain? | 11:49 |
| *** ddieterly has joined #openstack-infra | 11:51 | |
| hughsaunders | Hey, I'm getting a gerrit modal with the message "Intraline difference not available due to timeout." when viewing the diff for this change: https://review.openstack.org/#/c/180296/9/etc/openstack_deploy/openstack_user_config.yml.example Other diffs seem fine. | 11:52 |
| *** dizquierdo has quit IRC | 11:52 | |
| *** masayukig has quit IRC | 11:53 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make pypy non-voting across the board https://review.openstack.org/181868 | 11:54 |
| openstackgerrit | Merged openstack-infra/nodepool: Clarify project-id meaning for Rackspace deployment https://review.openstack.org/167881 | 11:54 |
| sdague | mordred / AJaeger you guys want to fast approve that? Then we can stop blocking on pypy | 11:54 |
| *** ddieterly has quit IRC | 11:55 | |
| mordred | sdague: on it | 11:55 |
| *** dkranz has joined #openstack-infra | 11:56 | |
| *** baoli has joined #openstack-infra | 11:56 | |
| jhesketh | sdague, mordred: looking too | 11:56 |
| mordred | sdague: I mean, ultimately getting pypy _working_ acrss everything has the potential to be pretty neat | 11:57 |
| mordred | especially for the places where people say they want better performance | 11:57 |
| mordred | however, in general, nobody actually cares | 11:57 |
| jhesketh | I'm a little torn here to be honest... disabling the gate when it doesn't work is quite dangerous | 11:57 |
| jhesketh | but it maybe true that nobody actually cares about pypy | 11:58 |
| mordred | yeah- in this case I'm a little more inclined to do it than in others | 11:58 |
| sdague | jhesketh: so, the gate can't be a thing when I show up 2 years ago, get a test working once, and now make the community have to support it for the rest of existance | 11:58 |
| sdague | there is no pypy champion that I can tell | 11:58 |
| *** claudiub has joined #openstack-infra | 11:58 | |
| mordred | because I'm not sure we have a champion for pypy at the moment | 11:58 |
| sdague | which means it should be deleted | 11:58 |
| mordred | heh | 11:58 |
| *** fhubik_afk is now known as fhubik | 11:58 | |
| mordred | we both said champion | 11:59 |
| sdague | :) | 11:59 |
| mordred | I'm going to send a quick mailing list message | 11:59 |
| sdague | mordred: there is a thread | 11:59 |
| jhesketh | yep, I don't disagree | 11:59 |
| *** masayukig has joined #openstack-infra | 11:59 | |
| jhesketh | but that's a bigger call to make imo | 11:59 |
| sdague | and I think, realistically, we should probably get ahead of everything where we feel like we don't have an usptream champion and start removing those jobs. | 12:00 |
| jhesketh | agreed | 12:00 |
| jhesketh | given that list thread was started 2 days ago and nobody has cried out I'm more inclined to disable it | 12:00 |
| jhesketh | mind you, it has been a weekend | 12:00 |
| sdague | yeh, but honestly, no one is going to own this for real | 12:01 |
| mordred | I mean, I've been arguing that we should delete postgres for the same reason, I'm pretty sure I need to be consistent here :) | 12:01 |
| *** gordc has joined #openstack-infra | 12:01 | |
| sdague | mordred: you haven't yet argued that at a global level :) | 12:01 |
| sdague | btw, I love the new size bars in gerrit ui | 12:02 |
| *** dprince has joined #openstack-infra | 12:02 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Make all pypy jobs non-voting https://review.openstack.org/181870 | 12:02 |
| *** dimsum__ has quit IRC | 12:03 | |
| *** woodster_ has joined #openstack-infra | 12:03 | |
| *** boris-42 has joined #openstack-infra | 12:04 | |
| sdague | jhesketh: I also think it's *far* more dangerous to not be fast about removing failing tests that are completely unrelated to code patches, because if there is a test remove penalty, which things take too long to delete when they are no longer relevant, that's a massive disincentive to creating new an interesting tests in the first place | 12:04 |
| *** _nadya_ has joined #openstack-infra | 12:05 | |
| jhesketh | sdague: Yep, especially since most projects just have pypy because it was there when they set up their jobs | 12:06 |
| openstackgerrit | Merged openstack-infra/project-config: Add stackforge/networking-6wind project https://review.openstack.org/180979 | 12:06 |
| jhesketh | I would like to give the pypy folks more time to weigh in, but I think that's likely to hold up too much | 12:06 |
| *** luqas has joined #openstack-infra | 12:06 | |
| jhesketh | perhaps the consequence here is that if there are people serious about pypy they'll need to fix the jobs and anything else that breaks while they are disabled | 12:07 |
| *** BharatK has quit IRC | 12:07 | |
| *** mwagner_lap has quit IRC | 12:07 | |
| jhesketh | mordred, sdague: I'd like to see another infra person weigh in if possible before merging.. however if that doesn't happen shortly I'd be in favour of merging. (I'm definately not blocking if you want to merge between yourselves) | 12:08 |
| *** wwwBUKOLAYcom has joined #openstack-infra | 12:08 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/puppet-askbot: Askbot module refactor https://review.openstack.org/181873 | 12:09 |
| sdague | jhesketh: treating failing jobs as punishment works pretty poorly | 12:09 |
| sdague | until this merges, requirements are fronze | 12:09 |
| sdague | frozen | 12:09 |
| *** ddieterly has joined #openstack-infra | 12:09 | |
| sdague | tons of interesting working can't proceed | 12:09 |
| jhesketh | sdague: by shortly I meant < 1hr | 12:10 |
| sdague | sure | 12:10 |
| sdague | I'm not sure what you expect to change in the next hour, but so be it | 12:10 |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:10 | |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:10 | |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:11 | |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:11 | |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:11 | |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:11 | |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:11 | |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:11 | |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:11 | |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:11 | |
| * wwwBUKOLAYcom Click http://www.bukolay.com I Come. | 12:11 | |
| sdague | oh hey, someone want to kickban ? | 12:11 |
| jhesketh | well I'm dubious too, and I'm also happy for it to merge, I'm just being somewhat cautious | 12:11 |
| * jhesketh can't sorry | 12:11 | |
| *** weshay has joined #openstack-infra | 12:12 | |
| *** wwwBUKOLAYcom has quit IRC | 12:12 | |
| openstackgerrit | Merged openstack-infra/shade: Fix delete_server when wait=True https://review.openstack.org/180659 | 12:12 |
| *** claudiub has quit IRC | 12:13 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/puppet-askbot: Askbot module refactor https://review.openstack.org/181873 | 12:13 |
| openstackgerrit | Merged openstack-infra/project-config: Cleanup openstack documentation Gerrit ACLs https://review.openstack.org/181655 | 12:13 |
| *** _dguerri is now known as dguerri | 12:13 | |
| *** esker has quit IRC | 12:14 | |
| openstackgerrit | Merged openstack-infra/project-config: Update jenkins jobs for networking-ovn https://review.openstack.org/181487 | 12:14 |
| openstackgerrit | Merged openstack-infra/project-config: Adding create permissions to networking-vsphere https://review.openstack.org/181172 | 12:14 |
| *** claudiub has joined #openstack-infra | 12:16 | |
| *** rfolco has joined #openstack-infra | 12:16 | |
| *** koolhead17 has joined #openstack-infra | 12:16 | |
| openstackgerrit | Merged openstack-infra/shade: Return Bunch objects instead of plain dicts https://review.openstack.org/180269 | 12:17 |
| *** fhubik has quit IRC | 12:18 | |
| *** fhubik has joined #openstack-infra | 12:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Configure to use gem mirrors https://review.openstack.org/181427 | 12:19 |
| openstackgerrit | Merged openstack-infra/project-config: Remove integrated-gate template from sahara https://review.openstack.org/181211 | 12:19 |
| openstackgerrit | Merged openstack-infra/project-config: Explicitly enable all required services in rally https://review.openstack.org/181391 | 12:19 |
| openstackgerrit | Merged openstack-infra/project-config: Stop hacking integration test from cloning across the network https://review.openstack.org/179438 | 12:20 |
| openstackgerrit | Merged openstack-infra/project-config: Enable upload to pypi for Shaker project https://review.openstack.org/168047 | 12:20 |
| mordred | greghaynes: responded to https://review.openstack.org/#/c/181742/ | 12:20 |
| *** sks has quit IRC | 12:21 | |
| *** banix has joined #openstack-infra | 12:22 | |
| mordred | greghaynes: and the next one in the stack | 12:23 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Return command status in tsfilter https://review.openstack.org/178905 | 12:24 |
| *** TheJulia has joined #openstack-infra | 12:24 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add errexit to setup & cleanup functions https://review.openstack.org/177017 | 12:24 |
| *** _nadya_ has quit IRC | 12:26 | |
| *** fawadkhaliq has quit IRC | 12:26 | |
| *** banix has quit IRC | 12:31 | |
| *** yfried is now known as yfried|afk | 12:32 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make pypy non-voting across the board https://review.openstack.org/181868 | 12:33 |
| sdague | fixed to pass layout tests, I missed the fact that there was a name clash | 12:33 |
| *** Hal has joined #openstack-infra | 12:33 | |
| *** _nadya_ has joined #openstack-infra | 12:33 | |
| *** Hal is now known as Guest21373 | 12:34 | |
| *** aysyd has joined #openstack-infra | 12:34 | |
| *** Ala has joined #openstack-infra | 12:35 | |
| sdague | gah, once more here | 12:36 |
| *** sdake has joined #openstack-infra | 12:38 | |
| *** bswartz has quit IRC | 12:38 | |
| *** otter768 has joined #openstack-infra | 12:39 | |
| *** sdake_ has joined #openstack-infra | 12:40 | |
| *** mrmartin has quit IRC | 12:41 | |
| jhesketh | sdague, mordred: okay, doesn't look like others are around and I need to run | 12:42 |
| jhesketh | sdague, mordred: you guys okay with me merging it and running? Worst case it can be undone | 12:42 |
| sdague | yeh, though I need a new patch that will pass | 12:42 |
| sdague | one second | 12:42 |
| *** BharatK has joined #openstack-infra | 12:43 | |
| *** otter768 has quit IRC | 12:43 | |
| *** fawadkhaliq has joined #openstack-infra | 12:44 | |
| fungi | busy night! almost 1k lines of scrollback in this channel. warning: i'm not going to read any of it aside from nick highlights | 12:44 |
| *** sdake has quit IRC | 12:44 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Enable pacemaker via the resource registry https://review.openstack.org/181522 | 12:46 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make pypy non-voting across the board https://review.openstack.org/181868 | 12:46 |
| sdague | fungi: yeh, so honestly, the kind of important one is probably ^^^ | 12:46 |
| sdague | which I'm still not sure I got right | 12:46 |
| sdague | pypy ruins everything | 12:46 |
| *** marcusvrn1 has quit IRC | 12:47 | |
| *** bcourt has joined #openstack-infra | 12:47 | |
| jhesketh | sdague: your latest change only disables pypy for icehouse? | 12:47 |
| sdague | does it? | 12:48 |
| sdague | I though that was 'don't run on icehouse' | 12:48 |
| sdague | also don't vote | 12:48 |
| *** Longgeek_ has quit IRC | 12:48 | |
| sdague | but, I could be wrong | 12:48 |
| jhesketh | sdague: oh, I may have mis-read.. | 12:49 |
| *** Longgeek has joined #openstack-infra | 12:49 | |
| jhesketh | sdague: yep, you're right | 12:49 |
| sdague | the don't run on icehouse was there from a long time ago | 12:49 |
| *** _nadya_ has quit IRC | 12:49 | |
| jhesketh | fungi: you normally read scrollback? | 12:49 |
| *** koolhead17 has quit IRC | 12:49 | |
| *** marcusvrn has joined #openstack-infra | 12:50 | |
| *** fawadk has joined #openstack-infra | 12:50 | |
| *** mwagner_lap has joined #openstack-infra | 12:50 | |
| jhesketh | sdague: okay so lgtm.. I'll leave it for fungi and mordred to take from here | 12:50 |
| jhesketh | night all :-) | 12:50 |
| *** fawadkhaliq has quit IRC | 12:50 | |
| sdague | night | 12:51 |
| fungi | sdague: i saw | 12:51 |
| fungi | fun times | 12:51 |
| sdague | well, it's sort of to be expected from bit rot. Much like building a bridge, you have to occationally do some maintenance to it or it falls down. | 12:52 |
| *** marcusvrn has quit IRC | 12:54 | |
| *** pdangur has left #openstack-infra | 12:54 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/system-config: Use jenkins default file from puppet-jenkins https://review.openstack.org/178760 | 12:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/puppet-jenkins: Adds jenkins default file https://review.openstack.org/178749 | 12:55 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Correct misleading comment about pypy jobs https://review.openstack.org/181882 | 12:55 |
| fungi | sdague: approved, follow-on to correct typo ^ | 12:55 |
| *** xyang1 has joined #openstack-infra | 12:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Propose translations: Fix handling of deleted files https://review.openstack.org/181810 | 12:55 |
| *** sdake_ is now known as sdake | 12:55 | |
| *** mrunge has quit IRC | 12:56 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config: Adds pypi-jobs to cloudbase-init https://review.openstack.org/181883 | 12:56 |
| fungi | i wouldn't normally be picky about comment text, but when it says the opposite of what it means, it can be confusing | 12:57 |
| *** marcusvrn has joined #openstack-infra | 12:57 | |
| *** btully has joined #openstack-infra | 12:58 | |
| *** radez_g0n3 is now known as radez | 13:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Straighten up keystone endpoint management https://review.openstack.org/181783 | 13:01 |
| *** bknudson has quit IRC | 13:01 | |
| *** smarcet has joined #openstack-infra | 13:03 | |
| *** MaxV has joined #openstack-infra | 13:03 | |
| sdague | fungi: yep, good catch | 13:03 |
| *** dkranz has quit IRC | 13:05 | |
| *** koolhead17 has joined #openstack-infra | 13:05 | |
| *** ddieterly has quit IRC | 13:06 | |
| yolanda | hi, any infra cores to help debug an issue with storyboard? | 13:06 |
| fungi | yolanda: what's broken? | 13:06 |
| yolanda | fungi, we cannot login, it redirects to the authorize page, then redirects to home | 13:07 |
| yolanda | so we cannot see any error on xhr or in the html itself | 13:07 |
| yolanda | we'd need some logs to get a clue | 13:08 |
| fungi | hrm, yeah i'm getting the same behavior... checking the logs now | 13:08 |
| *** fawadkhaliq has joined #openstack-infra | 13:08 | |
| openstackgerrit | Russell Bryant proposed openstack/requirements: global-requirements: Add ovs https://review.openstack.org/177891 | 13:08 |
| fungi | yolanda: would errors end up somewhere besides the apache logs? | 13:10 |
| *** fawadk has quit IRC | 13:11 | |
| *** amitgandhinz has joined #openstack-infra | 13:11 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:11 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/system-config: Use common jenkins master implementation https://review.openstack.org/177373 | 13:12 |
| fungi | it looks like https://storyboard.openstack.org/#!/auth/authorize is 303 redirecting back to https://storyboard.openstack.org/ | 13:13 |
| yolanda | fungi, they shoud be on apache, storyboard-error.log | 13:13 |
| fungi | yeah, no errors | 13:13 |
| yolanda | :( | 13:13 |
| fungi | i'm guessing something has changed recently in the storyboard openid callout implementation or configuration | 13:13 |
| yolanda | let me check what landed today | 13:13 |
| fungi | that was going to be my next suggestion ;) | 13:14 |
| yolanda | a bunch of changes actually | 13:14 |
| fungi | yolanda: to help narrow it down, looks like 09:36:09 utc today was teh last successful login via launchpad openid | 13:15 |
| fungi | so you have fewer than 4 hours of potential changes breaking it | 13:15 |
| yolanda | all changes landed starting at 09:43utc | 13:15 |
| *** Longgeek has quit IRC | 13:15 | |
| *** Longgeek has joined #openstack-infra | 13:16 | |
| mordred | I saw a patch, I don't know if it landed, but something about redirecting API calls to storyboard | 13:16 |
| fungi | then it's almost certainly one of those | 13:16 |
| mordred | https://review.openstack.org/#/c/155551/ | 13:16 |
| mordred | ? | 13:16 |
| yolanda | fungi, that looks as a potential guilty | 13:17 |
| mordred | ttx landed that 4 hours ago | 13:17 |
| *** fifieldt has joined #openstack-infra | 13:17 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/storyboard: Revert "Redirect Browsers from API to Client" https://review.openstack.org/181889 | 13:17 |
| *** fifieldt has quit IRC | 13:17 | |
| *** eharney has joined #openstack-infra | 13:17 | |
| yolanda | ok, let's start by it | 13:17 |
| openstackgerrit | Merged openstack-infra/zuul: Status: Use $DEST_DIR instead of $BASE_DIR/public_html https://review.openstack.org/169955 | 13:18 |
| openstackgerrit | Merged openstack-infra/project-config: make pypy non-voting across the board https://review.openstack.org/181868 | 13:19 |
| openstackgerrit | Merged openstack-infra/project-config: Correct misleading comment about pypy jobs https://review.openstack.org/181882 | 13:19 |
| *** bswartz has joined #openstack-infra | 13:20 | |
| fungi | yolanda: it'll need a rebase to deal with conflicts on other changes touching nearby lines | 13:20 |
| ttx | mordred: agreed that's the most... likely suspect | 13:20 |
| yolanda | oh, bad... i don't have the time now, it will need to wait for a bit , unless some others come and help | 13:21 |
| *** BobBall_AWOL is now known as BobBall | 13:21 | |
| fungi | i can do it | 13:21 |
| sdague | fungi: so... that's a thing | 13:21 |
| fungi | unless mordred wants to. it's his patch | 13:21 |
| yolanda | many thx, fungi | 13:21 |
| sdague | zuul retroactively applies voting rules? | 13:21 |
| sdague | I don't think I've ever seen that before | 13:22 |
| mordred | uhm. I guess I'll do it | 13:22 |
| *** bknudson has joined #openstack-infra | 13:22 | |
| *** chlong has joined #openstack-infra | 13:22 | |
| *** cinerama has quit IRC | 13:22 | |
| *** dkranz has joined #openstack-infra | 13:22 | |
| fungi | sdague: i believe those are evaluated real-time/on reporting back to the change. not positive | 13:22 |
| sdague | we landed the non voting change, zuul applied the new rules to the existing gate, it reset back into mostly green state | 13:23 |
| fungi | yep | 13:23 |
| sdague | yeh, I guess I shouldn't be totally surprised, I'd just never seen this edge case before | 13:23 |
| sdague | neat | 13:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/storyboard: Revert "Redirect Browsers from API to Client" https://review.openstack.org/181889 | 13:23 |
| fungi | i can't say i've paid close enough attention to notice it before myself | 13:23 |
| mordred | that's neat | 13:24 |
| *** craigbr has joined #openstack-infra | 13:24 | |
| *** dustins has joined #openstack-infra | 13:24 | |
| *** tiswanso has joined #openstack-infra | 13:25 | |
| fungi | huh... new in gerrit 2.10: ls-projects lists an "All-Users" project which can't be cloned | 13:26 |
| *** cinerama has joined #openstack-infra | 13:26 | |
| *** ibiris is now known as ibiris_away | 13:27 | |
| sdague | label:Code-Review<=-1,group=xxxxxxxxxxx still doesn't work | 13:27 |
| sdague | which makes me sad | 13:27 |
| fungi | https://review.openstack.org/gitweb?p=All-Users.git;a=summary | 13:28 |
| *** dizquierdo has joined #openstack-infra | 13:28 | |
| fungi | gitweb lists it too, but you can't browse it | 13:28 |
| mordred | wow. that's a fantastic degredation of functionality | 13:28 |
| mordred | so now we have to put a grep -v in everywhere we use ls-projects | 13:28 |
| *** fawadkhaliq has quit IRC | 13:29 | |
| mordred | zaro: ^^ I would contend that this is a bug | 13:29 |
| *** peristeri has joined #openstack-infra | 13:29 | |
| fungi | yep. in fact i noticed it because my daily remote update and clone new repos cron job on my workstation spat an error about it to my inbox | 13:29 |
| mordred | bleh | 13:30 |
| fungi | i agree, you would think that since ls-projects doesn't list the All-Projects pseudoproject, it shouldn't list All-Users either | 13:30 |
| *** baoli has quit IRC | 13:30 | |
| *** isviridov_away is now known as isviridov | 13:31 | |
| *** baoli_ has joined #openstack-infra | 13:31 | |
| *** jgrimm is now known as zz_jgrimm | 13:32 | |
| sdague | fungi: if you get to a slow point, I'd be kind of curious to figure out if there are errors being thrown in the gerrit logs on the group queries | 13:32 |
| *** esker has joined #openstack-infra | 13:32 | |
| *** koolhead17 has quit IRC | 13:33 | |
| *** amitgandhinz has quit IRC | 13:33 | |
| *** koolhead17 has joined #openstack-infra | 13:33 | |
| *** koolhead17 has quit IRC | 13:33 | |
| *** koolhead17 has joined #openstack-infra | 13:33 | |
| *** koolhead17 has quit IRC | 13:34 | |
| *** koolhead17 has joined #openstack-infra | 13:34 | |
| *** koolhead17 has quit IRC | 13:34 | |
| *** koolhead17 has joined #openstack-infra | 13:34 | |
| *** amitgandhinz has joined #openstack-infra | 13:34 | |
| *** isviridov is now known as isviridov_away | 13:34 | |
| *** koolhead17 has quit IRC | 13:35 | |
| *** ddieterly has joined #openstack-infra | 13:36 | |
| *** mrmartin has joined #openstack-infra | 13:36 | |
| *** craigbr has quit IRC | 13:37 | |
| *** mpaolino has joined #openstack-infra | 13:38 | |
| *** yamamoto has quit IRC | 13:39 | |
| *** yamamoto has joined #openstack-infra | 13:39 | |
| openstackgerrit | Merged openstack-infra/zuul: Status: Move external dependencies into public_html/lib https://review.openstack.org/169956 | 13:39 |
| *** dimsum__ has joined #openstack-infra | 13:39 | |
| *** prad has joined #openstack-infra | 13:39 | |
| sdake | is 30 minutes enough for the centos puppet apply job? http://logs.openstack.org/61/181761/3/gate/gate-infra-puppet-apply-centos6/4f43626/console.html#_2015-05-11_11_57_07_811 | 13:40 |
| sdake | seems to be timing out | 13:40 |
| *** teran has quit IRC | 13:40 | |
| openstackgerrit | Merged openstack-infra/storyboard: Revert "Redirect Browsers from API to Client" https://review.openstack.org/181889 | 13:40 |
| *** ivar-lazzaro has quit IRC | 13:41 | |
| *** pabelanger has joined #openstack-infra | 13:41 | |
| *** ivar-lazzaro has joined #openstack-infra | 13:41 | |
| sdake | also this error looks suspicious: http://logs.openstack.org/61/181761/3/gate/gate-infra-puppet-apply-centos6/4f43626/console.html#_2015-05-11_11_32_41_879 | 13:41 |
| fungi | sdake: centos 6.x git is broken. ever since the "ghost" vulnerability update we've seen random hangs of remote git operations on our bare-centos6 workers | 13:42 |
| sdake | fungi thanks, so recheck is the solution for the moment then? | 13:42 |
| fungi | i've so far been unable to reproduce it on demand, so haven't managed to get it to happen under a debugger | 13:42 |
| *** mriedem has joined #openstack-infra | 13:42 | |
| *** julim has joined #openstack-infra | 13:42 | |
| openstackgerrit | Merged openstack/requirements: add what-broke.py analayzer https://review.openstack.org/177889 | 13:43 |
| fungi | sdake: yes, and soon probably some timeout+retry logic in zuul-cloner (and somewhat longer term, stop caring about centos 6) | 13:43 |
| *** emagana has quit IRC | 13:43 | |
| sdake | ya, I was going to suggest just moving to centos7 | 13:43 |
| sdake | its pretty stable | 13:43 |
| fungi | sdake: i hear our python 2.6 jobs would run great on centos 7 | 13:44 |
| sdake | lol ;) | 13:44 |
| fungi | (which is the only thing we keep centos 6 workers around for) | 13:44 |
| sdake | i see | 13:44 |
| fungi | as soon as we can drop python 2.6 testing, we will be dropping centos 6 like a hot potato | 13:44 |
| *** yamamoto has quit IRC | 13:44 | |
| *** craigbr has joined #openstack-infra | 13:44 | |
| *** yfried|afk has quit IRC | 13:44 | |
| *** yfried|afk has joined #openstack-infra | 13:45 | |
| *** MaxV has quit IRC | 13:47 | |
| *** prad has quit IRC | 13:47 | |
| *** Ala has quit IRC | 13:48 | |
| *** prad has joined #openstack-infra | 13:48 | |
| openstackgerrit | Merged openstack-infra/project-config: Add python-cloudpulseclient project for CloudPulse https://review.openstack.org/181761 | 13:49 |
| *** dizquierdo has quit IRC | 13:52 | |
| *** dimsum__ has quit IRC | 13:53 | |
| *** mattfarina has joined #openstack-infra | 13:54 | |
| openstackgerrit | Merged openstack/requirements: Bump futures up to 3.0 or greater https://review.openstack.org/179967 | 13:56 |
| *** markvoelker has quit IRC | 13:56 | |
| openstackgerrit | Merged openstack/requirements: Add Bandit as a global test requirement https://review.openstack.org/167126 | 13:56 |
| *** kgiusti has joined #openstack-infra | 13:57 | |
| *** Mmike has quit IRC | 13:57 | |
| sdague | zaro: is there an updated set of instructions somewhere to get a working gerrit eclipse environment? I'd like to try to solve this groups thing as this would really help our review flow | 13:58 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Disable puppet-askbot resources temporarily https://review.openstack.org/181894 | 13:58 |
| openstackgerrit | Merged openstack-infra/system-config: Enable logging of #openstack-defcore https://review.openstack.org/181577 | 13:58 |
| sdague | the existing instructions seems to die for me after running buck on the failure to find watchman | 13:58 |
| *** Mmike has joined #openstack-infra | 13:58 | |
| *** Mmike has joined #openstack-infra | 13:58 | |
| fungi | sdague: how does eclipse fit into the picture? | 13:58 |
| sdague | fungi: because navigating large sets of java code without eclipse is... more challenge then I'm looking for | 13:59 |
| fungi | aha | 13:59 |
| yolanda | <yolanda> fungi, how much time shall puppet take? | 13:59 |
| yolanda | <yolanda> it's still failing | 13:59 |
| sdague | https://review.typo3.org/Documentation/dev-buck.html#build | 13:59 |
| fungi | yolanda: up to an hour | 13:59 |
| fungi | probably between 30-45 minutes after the change merges though | 13:59 |
| *** Longgeek has quit IRC | 14:00 | |
| yolanda | ok, let's hope is that | 14:00 |
| *** Longgeek has joined #openstack-infra | 14:01 | |
| fungi | i'm testing workarounds for the openid callback double-slash problem in new gerrit, and not having any luck. i had hoped to convince apache to rewrite double-slashes to single slashes, but it seems not to do so | 14:01 |
| *** stevemar has joined #openstack-infra | 14:01 | |
| *** ibiris_away is now known as ibiris | 14:01 | |
| *** wznoinsk has joined #openstack-infra | 14:02 | |
| *** smccully has quit IRC | 14:02 | |
| yolanda | mordred, i'm supper happy with your review speed for my changes , it's great :) | 14:02 |
| *** smccully has joined #openstack-infra | 14:02 | |
| *** yamamoto has joined #openstack-infra | 14:02 | |
| *** Longgeek_ has joined #openstack-infra | 14:03 | |
| sdague | fungi: so... gerrit 2.10 means we can remove the merge check facility in zuul, right? | 14:03 |
| sdague | as gerrit computes it directly itself | 14:04 |
| *** Longgeek has quit IRC | 14:04 | |
| fungi | sdague: in theory, yes. in practice we need a little time to confirm it's working as advertised | 14:04 |
| sdague | ok, cool | 14:04 |
| *** ibiris is now known as ibiris_away | 14:05 | |
| *** ibiris_away is now known as ibiris | 14:06 | |
| *** ssam2 has quit IRC | 14:06 | |
| *** openstackgerrit has quit IRC | 14:06 | |
| mordred | fungi: we may have lost gerritbot - I just pushed two patches and no notification | 14:06 |
| *** Guest21373 has quit IRC | 14:07 | |
| *** openstackgerrit has joined #openstack-infra | 14:07 | |
| *** craigbr has quit IRC | 14:07 | |
| mordred | oh. nm. he just left and rejoined | 14:07 |
| sdague | https://review.openstack.org/#/q/status:open+NOT+is:mergeable+AND+label:Verified%253E%253D1%252Cjenkins - fwiw, that's the list of things gerrit now doesn't count as mergable that Jenkins does. Going the other way will be a little hard to auto compute | 14:07 |
| mordred | oh! is:mergable is a thing | 14:07 |
| sdague | yes | 14:08 |
| sdague | it is now | 14:08 |
| mordred | I was looking for it over teh weekend | 14:08 |
| sdague | and the Status column in gerrit now displays Merge Conflict as a status | 14:08 |
| mordred | yah. well, that I knew | 14:09 |
| mordred | I just couldn't find the flag to search for it over teh weekend | 14:09 |
| *** rossella_s has quit IRC | 14:09 | |
| mordred | not sure why | 14:09 |
| *** mpaolino has quit IRC | 14:09 | |
| *** rossella_s has joined #openstack-infra | 14:09 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Disable puppet-askbot resources temporarily https://review.openstack.org/181894 | 14:09 |
| sdague | I always just go and read - https://review.openstack.org/Documentation/user-search.html once the new release is out there | 14:10 |
| openstackgerrit | Merged openstack/requirements: Require flake8 2.4.0 https://review.openstack.org/157985 | 14:10 |
| sdague | the other nice add is size:<50 | 14:10 |
| sdague | so less than 50 lines of change | 14:10 |
| *** Hal has joined #openstack-infra | 14:10 | |
| *** yamamoto has quit IRC | 14:11 | |
| *** Hal is now known as Guest28311 | 14:11 | |
| sdague | and... the secondary indexes for full text search on message seem to actually work now | 14:11 |
| *** dboik has joined #openstack-infra | 14:11 | |
| *** Ala has joined #openstack-infra | 14:11 | |
| *** hashar has quit IRC | 14:11 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/puppet-askbot: Askbot module refactor https://review.openstack.org/181873 | 14:11 |
| *** zz_dimtruck is now known as dimtruck | 14:11 | |
| fungi | oh, that's marvellous | 14:12 |
| mordred | nibalizer, EmilienM: I reworked EmilienM's patch that nibalizer had ameneded, basically reverting the patch back to PS9 and then maing nibalizer's changes a new patchset | 14:12 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add kilo compat jobs to tempest https://review.openstack.org/181907 | 14:13 |
| mordred | the combined patch was too big for me to grok what was going on - just to let you know why I did that | 14:13 |
| sdague | but I still really want reviews by group filter | 14:13 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements https://review.openstack.org/181908 | 14:13 |
| sdague | which the documentation says works, but the fact that all the examples reference ldap means I bet they only have tested the ldap case | 14:14 |
| EmilienM | mordred: ack | 14:15 |
| *** sdake_ has joined #openstack-infra | 14:15 | |
| *** dims has joined #openstack-infra | 14:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Rework mirrors to be more general https://review.openstack.org/181903 | 14:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Create rubygems mirror from rubygems.org https://review.openstack.org/178026 | 14:17 |
| mordred | hrm. ok. there we go | 14:17 |
| *** openstack has joined #openstack-infra | 14:20 | |
| openstackgerrit | Merged openstack/requirements: Add cachetools requirement https://review.openstack.org/152953 | 14:21 |
| *** ddieterly has joined #openstack-infra | 14:21 | |
| openstackgerrit | Merged openstack/requirements: Add ironic-discoverd https://review.openstack.org/158210 | 14:23 |
| *** mpaolino has joined #openstack-infra | 14:25 | |
| *** bnemec has joined #openstack-infra | 14:25 | |
| *** kmartin has quit IRC | 14:26 | |
| yolanda | mordred, fungi, SB is working, many thanks | 14:27 |
| *** wenlock has joined #openstack-infra | 14:28 | |
| * krotscheck likes the new gerrit | 14:29 | |
| *** miqui has joined #openstack-infra | 14:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use jenkins default file from puppet-jenkins https://review.openstack.org/178760 | 14:31 |
| *** ddieterly has quit IRC | 14:31 | |
| *** sdake has joined #openstack-infra | 14:32 | |
| mordred | krotscheck: ++ | 14:32 |
| krotscheck | I have to say though, there's now four columns I have to read to figure out the status of a patch. | 14:33 |
| mordred | krotscheck: you could fix that with a fifth column which summarizes most of the info of the other columns | 14:33 |
| mriedem | was there a specific bug that fixed this? http://logs.openstack.org/10/158210/3/gate/gate-requirements-pypy/7122683/console.html#_2015-05-11_13_01_25_526 | 14:34 |
| krotscheck | mordred: You'd need a sixth to explain the rationale. | 14:34 |
| mordred | krotscheck: or maybe an animation that bounces and the frequency of the boucing combined with the shade of the fill color would tell you the most important column to look at | 14:34 |
| krotscheck | mordred: It'd need to blink. | 14:35 |
| mordred | yah | 14:35 |
| *** mtanino_away has quit IRC | 14:35 | |
| mordred | clearly | 14:35 |
| mordred | I can think of no reasons to not implement that | 14:35 |
| notnownikki | have you seen the new change screen that cyanogenmod are using on their gerrit? | 14:35 |
| * krotscheck runs off to StoryBoard to file a strory. | 14:35 | |
| *** mtanino has joined #openstack-infra | 14:35 | |
| mordred | notnownikki: nope! I shall go look | 14:35 |
| *** sdake_ has quit IRC | 14:35 | |
| notnownikki | it's a bit rearranged and has the size graphs by each file | 14:36 |
| notnownikki | a little jarring to look at, at first | 14:36 |
| mordred | notnownikki: you can have that change on our gerrit too | 14:36 |
| mordred | notnownikki: go to your user preferences and select "new screen" (we have it turned off by default) | 14:36 |
| notnownikki | ah, i see it now :) | 14:37 |
| *** prad has quit IRC | 14:37 | |
| *** zz_jgrimm is now known as jgrimm | 14:37 | |
| fungi | yeah, i'm not a fan. then again, gertty is way better than either the old or new webui | 14:37 |
| *** prad has joined #openstack-infra | 14:37 | |
| *** deepakcs has joined #openstack-infra | 14:38 | |
| *** mestery has quit IRC | 14:38 | |
| *** ddieterly has joined #openstack-infra | 14:38 | |
| *** otter768 has joined #openstack-infra | 14:40 | |
| *** annegentle has joined #openstack-infra | 14:40 | |
| *** emagana has joined #openstack-infra | 14:42 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for pypy job bug 1290562 https://review.openstack.org/181919 | 14:42 |
| openstack | bug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] https://launchpad.net/bugs/1290562 - Assigned to Ildiko Vancsa (ildiko-vancsa) | 14:42 |
| *** anteaya has joined #openstack-infra | 14:43 | |
| *** koolhead17 has joined #openstack-infra | 14:43 | |
| *** otter768 has quit IRC | 14:44 | |
| *** amotoki_ has joined #openstack-infra | 14:45 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for pypy job bug 1290562 https://review.openstack.org/181919 | 14:46 |
| openstack | bug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] https://launchpad.net/bugs/1290562 - Assigned to Ildiko Vancsa (ildiko-vancsa) | 14:46 |
| *** hemnafk is now known as hemna | 14:51 | |
| *** teran has joined #openstack-infra | 14:51 | |
| *** banix has joined #openstack-infra | 14:52 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for pypy job bug 1290562 https://review.openstack.org/181919 | 14:52 |
| openstack | bug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] https://launchpad.net/bugs/1290562 - Assigned to Ildiko Vancsa (ildiko-vancsa) | 14:52 |
| *** teran has quit IRC | 14:55 | |
| *** asselin_ has joined #openstack-infra | 14:55 | |
| *** koolhead17 has quit IRC | 14:56 | |
| *** sarob has joined #openstack-infra | 14:58 | |
| *** koolhead_ has joined #openstack-infra | 15:00 | |
| krotscheck | StoryBoard Meeting in #openstack-meeting-3 | 15:00 |
| *** mpopow has joined #openstack-infra | 15:01 | |
| *** dkranz has quit IRC | 15:02 | |
| *** tiswanso has quit IRC | 15:02 | |
| *** markvoelker has joined #openstack-infra | 15:03 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for pypy job bug 1290562 https://review.openstack.org/181919 | 15:04 |
| openstack | bug 1290562 in python-ceilometerclient "failing pypy jobs with "option --single-version-externally-managed not recognized"" [Undecided,Fix committed] https://launchpad.net/bugs/1290562 - Assigned to Ildiko Vancsa (ildiko-vancsa) | 15:04 |
| *** sabeen1 has joined #openstack-infra | 15:04 | |
| fungi | ugh. i consider myself relatively adept at apache redirects and rewrites, but cannot for the life of me get a rewrite of https://review-dev.openstack.org// to https://review-dev.openstack.org/ working | 15:05 |
| *** dboik has quit IRC | 15:05 | |
| anteaya | fungi: do you know if jeblair has posted the infra schedule to the main schedule yet? | 15:06 |
| mordred | fungi: I'm going to blame systemd | 15:06 |
| anteaya | the third party folks are asking | 15:06 |
| mordred | anteaya: I believe so - I have subscribed to infra events | 15:06 |
| anteaya | mordred: you are the one behind the colourful logs | 15:06 |
| mordred | anteaya: yes I am | 15:06 |
| anteaya | mordred: have you a link, so far I can't find a url for it | 15:06 |
| mordred | http://libertydesignsummit.sched.org/ | 15:06 |
| anteaya | thank you | 15:07 |
| jeblair | fungi: maybe apache/mod_rewrite is too smart and is normalizing that for you before processing? | 15:08 |
| fungi | jeblair: that's all i can figure | 15:08 |
| *** tim_o has joined #openstack-infra | 15:09 | |
| fungi | anteaya: btw don't direct normal conferencegoers to that url. the https://www.openstack.org/summit/vancouver-2015/schedule/ one omits the design summit sessions so that they don't go wandering in unless they know they need to be there | 15:10 |
| anteaya | fungi: yes thank you | 15:11 |
| *** nelsnelson has joined #openstack-infra | 15:11 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: networking-ovn: disable n-novnc https://review.openstack.org/181930 | 15:11 |
| anteaya | this is a part of the third party meeting (now in progress in -meeting) | 15:11 |
| anteaya | to direct and support those operators willing to learn to contribute and help the effort | 15:11 |
| *** dims has quit IRC | 15:12 | |
| fungi | sounds great | 15:12 |
| mordred | fungi: whoever I should thank should be thanked for adding the conference sessions to teh design summit sched.org | 15:12 |
| mordred | fungi: I actually can deal with my schedule non-insanely for the first time EVER | 15:13 |
| fungi | mordred: ttx and/or lsell i think | 15:13 |
| mordred | ttx: thank you | 15:13 |
| nibalizer | mordred: okay | 15:13 |
| *** panda|afk has quit IRC | 15:13 | |
| mordred | also, tell lsell I'd thank her but she's not in here | 15:13 |
| anteaya | fungi: thanks | 15:13 |
| mordred | nibalizer: the patch looks good - it was just too much at once for my reptile brain | 15:13 |
| *** aswadr has quit IRC | 15:14 | |
| *** yamahata has joined #openstack-infra | 15:14 | |
| *** panda|afk has joined #openstack-infra | 15:14 | |
| ttx | mordred: that was a very selfish move | 15:15 |
| annegentle | I'm getting a 503 for http://libertydesignsummit.sched.org/ and downforeveryoneorjustme.com also says "it looks down from here" -- is this a known issue? | 15:15 |
| ttx | but I guess my problem was also the others problem | 15:15 |
| mordred | annegentle: it was up for me a second ago ... it seems unhealthy this instant | 15:15 |
| annegentle | ttx: definitely a good move | 15:15 |
| mordred | ttx: yah. it's too hard of a schedule merge | 15:16 |
| annegentle | ttx: call it altruistic, it's ok | 15:16 |
| mordred | I mean, I'm still 95% in the summit - but the 3 things I'm doing in the conference show up well | 15:16 |
| annegentle | mordred: ok thanks for looking | 15:16 |
| sdague | annegentle: yeh, seems down atm | 15:16 |
| *** baoli_ has quit IRC | 15:16 | |
| fungi | there must be a conference going on | 15:16 |
| annegentle | mordred: ttx: sdague: next step to notify <insert name>? | 15:16 |
| fungi | or they're testing out their auto-crashing feature in preparation for next monday | 15:17 |
| sdague | annegentle: no idea, it's a 3rd party service | 15:17 |
| ttx | annegentle: I'm on it, although it's just sched | 15:17 |
| *** hashar has joined #openstack-infra | 15:17 | |
| annegentle | ttx: yeah sure it's out of our hands directly (sched being sched) | 15:17 |
| sdague | oh, it's back | 15:17 |
| ttx | wat | 15:18 |
| *** unicell has joined #openstack-infra | 15:18 | |
| ttx | oh, and it's fixed too. Cool | 15:18 |
| sdague | also, reseting my password | 15:20 |
| *** dannywilson has joined #openstack-infra | 15:20 | |
| *** mtanino has quit IRC | 15:20 | |
| *** unicell1 has quit IRC | 15:20 | |
| sdague | the captchas are so scrambled at this point it was 10 captchas before I found one I could do | 15:20 |
| nibalizer | mordred: you're not the only one rewriting cloud init https://github.com/coreos/coreos-cloudinit :P | 15:20 |
| openstackgerrit | Jaume Devesa proposed openstack-infra/project-config: Define MidoNet v2 devstack-gate jobs https://review.openstack.org/181932 | 15:20 |
| *** bhunter71 has joined #openstack-infra | 15:20 | |
| nibalizer | mordred: ya good feedback on size i've been making epic code bombs so I need to get better at breaking it up | 15:21 |
| *** Krinkle|detached is now known as Krinkle | 15:21 | |
| mordred | nibalizer: I have the same tendency myself | 15:21 |
| mordred | nibalizer: yeah - I'd show them my version that's in rust - but somethign tells me they're obsessed with go :) | 15:22 |
| *** fifieldt has joined #openstack-infra | 15:22 | |
| *** fifieldt has quit IRC | 15:23 | |
| annegentle | what's rust good for? | 15:23 |
| *** sabeen3 has joined #openstack-infra | 15:23 | |
| annegentle | (not trolling) | 15:23 |
| mordred | annegentle: same things as go | 15:23 |
| annegentle | ah ok | 15:23 |
| fungi | writing software | 15:23 |
| mordred | annegentle: it statically compiles, is designed for high-performance high-concurrency | 15:23 |
| * fungi _was_ sort of trolling | 15:24 | |
| morganfainberg | annegentle: rust is pretty cool. | 15:24 |
| mordred | annegentle: main difference being that go is written by google and rust is written by mozilla | 15:24 |
| clarkb | mordred I think they forget reseting host keys too :) we are all doomed to repeat the same mistakes | 15:24 |
| mordred | and rust reminds me more of python/ruby while go reminds me more of java | 15:24 |
| * morganfainberg likes some of the rust-isms more than go-isms personally. | 15:24 | |
| mordred | yah | 15:24 |
| *** sabeen1 has quit IRC | 15:24 | |
| mordred | I _really_ like the matcher stuff they borroed from haskell | 15:24 |
| mordred | annegentle: the mozilla folks seem to have taken the best features from all the langauges people love and made a nice language | 15:25 |
| fungi | and like most languages, it contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of common lisp ;) | 15:26 |
| *** bhunter71 has quit IRC | 15:26 | |
| morganfainberg | Ahaha | 15:26 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Rewrite extension checking methods https://review.openstack.org/181935 | 15:27 |
| *** harlowja_at_home has joined #openstack-infra | 15:27 | |
| *** bhunter71 has joined #openstack-infra | 15:28 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Initial commit of ask-staging.o.o site https://review.openstack.org/172880 | 15:28 |
| *** sputnik13 has quit IRC | 15:29 | |
| *** sks has joined #openstack-infra | 15:29 | |
| clarkb | dguerri: question, why would checking if extensions exist be a private method? | 15:30 |
| mordred | clarkb: because it's an implementation detail | 15:31 |
| *** mtanino has joined #openstack-infra | 15:31 | |
| mordred | it's not related to managing a resource ina cloud | 15:31 |
| clarkb | yes it is... | 15:31 |
| mordred | and honestly, that anyone has to ever call it is a bug in openstack | 15:31 |
| dguerri | yup, thanks mordred | 15:31 |
| clarkb | if that resource is part of an extension | 15:31 |
| clarkb | eg floating IPs | 15:31 |
| mordred | nope | 15:31 |
| mordred | totally disagree | 15:31 |
| mordred | clouds have some features and not others | 15:32 |
| mordred | whether they are implemented as api extensions or not is immaterial | 15:32 |
| mordred | either a cloud can perform a function or it can't | 15:32 |
| clarkb | sure so that I agree with, except that openstack has decided to have these things | 15:32 |
| clarkb | and its part of openstack service apis | 15:32 |
| *** ivar-lazzaro has quit IRC | 15:32 | |
| mordred | right. but openstack has no UI | 15:32 |
| mordred | it's terrible | 15:32 |
| mordred | I'm not going to write a library that exposes the mistakes of openstack's insanity | 15:33 |
| mordred | I'm going to write a library that works | 15:33 |
| *** ivar-lazzaro has joined #openstack-infra | 15:33 | |
| *** armax has joined #openstack-infra | 15:33 | |
| clarkb | I don't think you are exposing a mistake, you are exposing info to the user | 15:33 |
| *** hdd has joined #openstack-infra | 15:33 | |
| *** Ala has quit IRC | 15:33 | |
| mordred | I disagree that the user should ever need to know that | 15:33 |
| mordred | the user wants a floating ip | 15:33 |
| mordred | we should give them one | 15:33 |
| clarkb | 99% of the time the happily don't need that info, but the first time a shade user needs that info and they find its hidden away they will decide shade is just as broken as openstack | 15:33 |
| mordred | whether we need to talk to an api extension to do so is who cares | 15:33 |
| mordred | clarkb: that's fine | 15:33 |
| mordred | they can use something else | 15:33 |
| clarkb | ok | 15:34 |
| clarkb | (I mostly bring this up because you shouldn't change the publicit of a method like that fwiw) | 15:34 |
| mordred | yah. I agree | 15:34 |
| clarkb | mordred: or did we not tag that release yet? | 15:34 |
| mordred | 0.x | 15:34 |
| clarkb | mordred: you said you were tagging a thing last week I thought | 15:34 |
| *** scheuran has quit IRC | 15:34 | |
| mordred | yup. I did | 15:34 |
| mordred | none of our known consumers use that method | 15:34 |
| mordred | unknown consumers are using an 0.x and should know what they are getting | 15:35 |
| *** koolhead_ has quit IRC | 15:35 | |
| *** tmcpeak has joined #openstack-infra | 15:35 | |
| *** tiswanso has joined #openstack-infra | 15:35 | |
| *** dboik has joined #openstack-infra | 15:35 | |
| *** ssam2 has quit IRC | 15:35 | |
| tmcpeak | hey all - are there any provisions/processes for pruning global-requirements? | 15:35 |
| mordred | tmcpeak: with gusto! | 15:35 |
| *** wenlock has quit IRC | 15:36 | |
| tmcpeak | :) | 15:36 |
| tmcpeak | mordred: so we're fairly sure all of the things on there are used? | 15:36 |
| clarkb | mordred: ok, /me notes for the record that we are basically perpetuating problems that the clients have with shade by making changes like this. | 15:36 |
| *** annegentle has quit IRC | 15:37 | |
| mordred | clarkb: I'm not sure I follow you? | 15:37 |
| *** markus_z has quit IRC | 15:37 | |
| clarkb | mordred: the api changes, lack of documentation of the change/deprecation, and "no you can't do this because silly user" | 15:37 |
| clarkb | mordred: its basically everything I hate about using eg novaclient | 15:37 |
| *** koolhead17 has joined #openstack-infra | 15:37 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/reviewstats: Fix compat with gerrit 2.9 https://review.openstack.org/181938 | 15:38 |
| *** annegentle has joined #openstack-infra | 15:38 | |
| *** wenlock has joined #openstack-infra | 15:38 | |
| mordred | clarkb: sure. right now I'm trying to get to a supportable 1.0 API contract as quickly as possible | 15:38 |
| mordred | clarkb: so that we can actually support everything with deprecations | 15:38 |
| dguerri | clarkb who is relying on a 0.x version? | 15:38 |
| mordred | clarkb: a few things have snuck in and we're trying to trim them out before we cut the 1.0 - which I'd like to do soon honestsly | 15:39 |
| clarkb | dguerri: you can't know, and thats sort of why thinking about this is important | 15:39 |
| mordred | because I agree with you | 15:39 |
| dguerri | that method was not even documented clarkb | 15:39 |
| *** yamamoto has joined #openstack-infra | 15:39 | |
| mordred | currently there are 2 consumers I am willing to support- nodepool and ansible - outside of those I'll support everyone esle as soon as we cut 1.0 | 15:39 |
| sdague | fungi: fyi, resume_sortkey was removed for gerrit cli api | 15:39 |
| clarkb | dguerri: it doesn't matter, it was part of the public api and now it is gone, anyone that has used it few they may be will break on your next release | 15:40 |
| *** Swami has joined #openstack-infra | 15:40 | |
| *** soren has quit IRC | 15:40 | |
| sdague | so everywhere that was used we'll need to move to --start n | 15:40 |
| dguerri | define part of public api please? | 15:40 |
| sdague | like - https://review.openstack.org/#/c/181938/ | 15:40 |
| mordred | dstufft: no leading _ | 15:40 |
| mordred | gah | 15:40 |
| clarkb | dguerri: it was a public method has_extensions() | 15:40 |
| mordred | dguerri: no leading _ | 15:40 |
| *** gyee has joined #openstack-infra | 15:41 | |
| anteaya | clarkb: can you join us in -meeting, have you seen the just posted gerrit error before? | 15:41 |
| anteaya | I have not | 15:41 |
| mordred | clarkb: so, yes - I agree with you - that breaks the public interface. but I quite explicitly do not care until the version is not 0.x - else 0.x does not mean anything either. "0" as a major number means "I make zero assurance about interface" | 15:41 |
| mordred | but - I also agree with you that this is not friendly, thus want to make a 1.x as soon as possible | 15:42 |
| clarkb | ok | 15:42 |
| dguerri | That's mean it is a public method. I don't know guys, to me a public API should documented. | 15:42 |
| clarkb | I wouldn't be so cranky about this if it wasn't shade | 15:42 |
| *** dboik_ has joined #openstack-infra | 15:42 | |
| clarkb | but shade specifically exists to make life easier for users and these are specific problems users have with the existing clients | 15:43 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config: Have gate-ironic-python34 be a voting job https://review.openstack.org/181034 | 15:43 |
| dguerri | can be called != should be called | 15:43 |
| clarkb | dguerri: sorry, that isn't how python works | 15:43 |
| dguerri | clarkb ^ that's how humans work | 15:43 |
| dguerri | I can jump out of the window right now clarkb | 15:44 |
| dguerri | but I'd rather not | 15:44 |
| *** dboik has quit IRC | 15:44 | |
| *** mpaolino has quit IRC | 15:44 | |
| mordred | dguerri: sure. but I think the rule of thumb needs to be that if the method is not marked with a _ we need to assume it's being used even if not documented | 15:45 |
| clarkb | also it will be documented if using doc generation | 15:45 |
| mordred | and we need to make a full pass through the library before we 1.0 to make sure that we're happy with everything that doesn't have a _ | 15:45 |
| * clarkb goes to find the shade docs | 15:45 | |
| *** markvoelker has quit IRC | 15:45 | |
| mordred | clarkb: and totally agree re: being more cranky and pedantic with shade | 15:46 |
| openstackgerrit | Merged openstack/requirements: Update python-cinderclient to 1.2.1 https://review.openstack.org/181083 | 15:46 |
| *** baoli has joined #openstack-infra | 15:46 | |
| clarkb | doesn't look like shade's doc api generation is working at all fwiw | 15:47 |
| clarkb | http://ci.openstack.org/shade/py-modindex.html | 15:47 |
| *** rwsu has joined #openstack-infra | 15:48 | |
| *** ssam2 has joined #openstack-infra | 15:48 | |
| mordred | clarkb: you're looking in the wrong place: http://docs.openstack.org/infra/shade/ | 15:49 |
| mordred | http://docs.openstack.org/infra/shade/usage.html#module-shade | 15:49 |
| clarkb | mordred: you should clean up the old docs then | 15:49 |
| mordred | clarkb: I believe we have an outstanding patch to do that | 15:49 |
| mordred | that fungi wrote I think? | 15:49 |
| clarkb | cool | 15:49 |
| *** _nadya_ has joined #openstack-infra | 15:49 | |
| mordred | to add the redirects from ci.o.o to docs.o.o/infra | 15:49 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Root the Change-Id commentlink explicitly https://review.openstack.org/181949 | 15:49 |
| fungi | clarkb: mordred: yeah we still have a couple of projects we need to trigger documentation jobs for and then we can turn on the redirects | 15:50 |
| fungi | just got swamped with other things in the middle of that | 15:50 |
| clarkb | fungi: for 181949 do we need to root the query below the one you have updated too? | 15:51 |
| clarkb | line 181 | 15:51 |
| mordred | fungi: I can't find the change itself ... | 15:51 |
| fungi | clarkb: oh, so we do. i missed that because it was an html replacement. fixing | 15:52 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Root the Change-Id and commit commentlinks https://review.openstack.org/181949 | 15:53 |
| fungi | anyway, tested and working on review-dev, so should get around the problem for now while we figure out what new behavior in 2.10 (or possibly even 2.9) caused it | 15:54 |
| *** eharney_ has joined #openstack-infra | 15:54 | |
| *** wenlock has quit IRC | 15:54 | |
| *** hashar has quit IRC | 15:55 | |
| *** eharney has quit IRC | 15:56 | |
| *** ildikov has quit IRC | 15:56 | |
| *** ildikov has joined #openstack-infra | 15:56 | |
| *** wenlock has joined #openstack-infra | 15:57 | |
| *** e0ne has joined #openstack-infra | 15:57 | |
| *** e0ne is now known as e0ne_ | 15:58 | |
| *** mpaolino has joined #openstack-infra | 15:58 | |
| *** e0ne_ is now known as e0ne | 15:58 | |
| clarkb | mordred: dguerri properly reviewed now, one small issue with removing that import | 15:58 |
| mordred | cool | 15:58 |
| mordred | thanks clarkb | 15:58 |
| mgagne | Am I the only one not able to load this change? https://review.openstack.org/#/c/149942/ | 15:59 |
| jeblair | mgagne: you are not the only one | 15:59 |
| *** baoli has quit IRC | 15:59 | |
| *** annegentle has quit IRC | 15:59 | |
| mordred | me neither | 15:59 |
| mgagne | I didn't scrollback far in backlog, any known problem? | 15:59 |
| mgagne | (related to it, not in general) | 16:00 |
| clarkb | fungi: +2 thanks for editing the second query link thing | 16:00 |
| jeblair | that change # shows up in the error log | 16:00 |
| *** annegentle has joined #openstack-infra | 16:00 | |
| jeblair | com.google.gerrit.server.patch.PatchSetInfoNotAvailableException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 3af29bcd3c88c49978e1dae55852b0a3fff12616 | 16:00 |
| fungi | gertty can't retrieve it either | 16:01 |
| openstackgerrit | Merged openstack/requirements: Using django-nose>=1.2 needed by murano-dashboard https://review.openstack.org/125651 | 16:01 |
| fungi | "Error: Unable to sync change." | 16:01 |
| mriedem | yeah it's everything, git review -d fails too | 16:01 |
| *** unicell has quit IRC | 16:01 | |
| jeblair | remote_revision['commit']['message'], remote_commit, | 16:01 |
| jeblair | KeyError: 'commit' | 16:01 |
| jeblair | gertty ^ | 16:02 |
| dguerri | clarkb thanks | 16:02 |
| jeblair | it's a nova change? | 16:02 |
| mgagne | yes | 16:02 |
| fbo | Hi, I tried to add a task inside a storyboard story but the task never appears in the task list. And on the other side I can see the event in the events timeline list. Is it a known issue or I misuse the form ? here is the related story https://storyboard.openstack.org/#!/story/2000101 | 16:02 |
| mgagne | found in bug #1293540 | 16:02 |
| openstack | bug 1293540 in OpenStack Compute (nova) "nova should make sure the bridge exists before resuming a VM after an offline snapshot" [Low,Confirmed] https://launchpad.net/bugs/1293540 - Assigned to Luo Gangyi (luogangyi) | 16:02 |
| jeblair | git show refs/changes/42/149942/1 | 16:03 |
| *** lennyb has quit IRC | 16:03 | |
| jeblair | that works on gerrit's git repo | 16:03 |
| *** asselin_ has quit IRC | 16:03 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Update server API for get/list/search interface https://review.openstack.org/180293 | 16:03 |
| *** banix has quit IRC | 16:03 | |
| fungi | | I43fd09c6c36aa401b53a1af77fcb6775313797c1 | 2015-01-26 07:54:26 | 2015-05-03 11:46:54 | 0034dae2000249b6 | 13630 | openstack/nova | refs/heads/master | Y | n | 12 | Ensure bridge exists before resuming a VM after an offline snapshot | Bug1293540 | 110 | 149942 | df04ec843a9d32d30ec99ba07b5e757281dfdbb5 | Y | | 16:03 |
| *** spredzy is now known as spredzy|afk | 16:03 | |
| fungi | from the changes table | 16:04 |
| fungi | status is 'n' | 16:04 |
| fungi | so not merged yet | 16:05 |
| clarkb | I can fetch taht change from git.o.o just fine | 16:05 |
| *** jlanoux has quit IRC | 16:05 | |
| *** craigbr has joined #openstack-infra | 16:05 | |
| clarkb | I am guessing its jgit having a hard time | 16:05 |
| jeblair | clarkb, fungi: i kind of want to see if it persists after a gerrit restart | 16:05 |
| jeblair | since we do repack the repo behind gerrit's back | 16:05 |
| fungi | oh, yep | 16:06 |
| clarkb | jeblair: that seems like a reasonable check | 16:06 |
| mgagne | thanks for the link to git, will check now | 16:06 |
| *** annegentle has quit IRC | 16:06 | |
| fungi | that sounds familiar to the jgit bug about cgit replacing pack files while retaining the same name causing confusion | 16:06 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Initial commit of ask-staging.o.o site https://review.openstack.org/172880 | 16:06 |
| jeblair | clarkb, fungi: shall i execute? | 16:07 |
| jeblair | $ git fetch gerrit refs/changes/42/149942/1 | 16:07 |
| jeblair | fatal: git fetch_pack: expected ACK/NAK, got 'ERR want e3a4b7ea669b8905da337df3ceb68ca40996963e not valid' | 16:07 |
| jeblair | also ^ | 16:07 |
| clarkb | zuul merges are ~14 minutes out so now is a reasonable time to restart gerrit | 16:07 |
| clarkb | actualy wait | 16:08 |
| jeblair | that's via ssh, so to gerrit's git repo via jgit | 16:08 |
| zaro | morning | 16:08 |
| *** baoli has joined #openstack-infra | 16:08 | |
| clarkb | jeblair: maybe try flushing all gerrit caches | 16:08 |
| *** fitoduarte has quit IRC | 16:08 | |
| clarkb | perhaps that will get it to rebuild its internal data structures around git repos, and if it works we can do that post repack? | 16:08 |
| jeblair | clarkb: flushing | 16:09 |
| clarkb | there are diff and projects caches according to the docs which may be at play here | 16:09 |
| *** mpavone has quit IRC | 16:09 | |
| jeblair | this is slow | 16:09 |
| zaro | sdague: gerrit eclipse setup, https://gerrit-review.googlesource.com/Documentation/dev-eclipse.html | 16:10 |
| jeblair | the error log is flooded with this: | 16:10 |
| jeblair | java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 6c337867abda3d53da25187dfdfaf948eb89461a | 16:10 |
| fungi | zaro: to update you to where we are, there seem to be several known issues post upgrade. 1. account registration dates all got reset (presumably an oversight in a db migration?), 2. the openid callback url adds a redundant trailing slash (/) which confuses some relative hyperlinks, 3. gerrit fails to find commits for some changes (possibly a jgit bug triggered by repacking the git repos) | 16:10 |
| *** yfried|afk is now known as yfried | 16:10 | |
| sdague | zaro: yeh, none of that worked for me | 16:11 |
| sdague | or, more importantly, I'm stuck at https://gerrit-review.googlesource.com/Documentation/dev-eclipse.html#_site_initialization | 16:11 |
| jeblair | clarkb: flishing done | 16:11 |
| sdague | because buck won't run | 16:11 |
| jeblair | i think that made things worse :/ | 16:11 |
| sdague | won't complete successfully | 16:11 |
| clarkb | jeblair: :( | 16:11 |
| sdague | because watchman is missing, even though not in the docs anywhere, which makes me wonder if I'm on the wrong buck version | 16:12 |
| jeblair | clarkb: there are no visible nova changes now | 16:12 |
| mordred | oh. that's not good | 16:12 |
| clarkb | wow | 16:12 |
| mriedem | heh | 16:12 |
| jeblair | ready for me to stop gerrit? | 16:12 |
| mriedem | all my -workflow changes are not showing | 16:12 |
| clarkb | jeblair: yup | 16:12 |
| zaro | fungi: anybody working on any of those issues? i can take a look at the trailing slash issue. | 16:12 |
| jeblair | so all those entries in the log are due to all nova changes being impacted by the 'missing' object | 16:12 |
| fungi | so perhaps we need to emergency restart gerrit and temporarily disable our repack cronjob for gerrit's backend repos | 16:13 |
| clarkb | its kind of amazing that flushing caches makes a stale cache inside gerrit become more problematic | 16:13 |
| jeblair | fungi: if the restart fixes it, possibly so | 16:13 |
| jeblair | starting now | 16:13 |
| fungi | zaro: i've got a config change proposed to root the commentlinks which works around the openid callback being slightly wrong, but haven't opened an upstream bug about it yet, so please do | 16:13 |
| jeblair | okay, nova changes exist again :) | 16:13 |
| clarkb | we do repack on sundays, so it would have popped up yesterday at some point | 16:13 |
| clarkb | I can write a change to disable that cron | 16:14 |
| jeblair | https://review.openstack.org/#/c/149942/ works now | 16:14 |
| zaro | sdague: have you run the ./tools/eclipse/project.py script? | 16:14 |
| clarkb | I wonder what Gerrit suggests users do to repack then :/ | 16:14 |
| fungi | zaro: i did at least confirm that changing the canonicalweburl to not include a trailing "/" doesn't seem to prevent the extra one either | 16:15 |
| jeblair | we could go all windows on this and repack && restart | 16:15 |
| *** MmikeM has joined #openstack-infra | 16:15 | |
| *** ashleighfarnham has joined #openstack-infra | 16:15 | |
| jeblair | clarkb, fungi, zaro: but maybe the "nfs repack" bug isn't actually nfs related; perhaps it is triggered on local fs's too | 16:16 |
| clarkb | jeblair: it is certainly looking like it | 16:16 |
| fungi | clarkb: background gerrit task to repack via jgit? no clue | 16:16 |
| sdague | /home/sdague/code/buck/bin/buck_common: line 64: watchman: command not found | 16:17 |
| sdague | 16:17 | |
| sdague | so... I'm assuming that watchman is thus a hard dep | 16:17 |
| *** jistr has quit IRC | 16:17 | |
| sdague | that's not called out? | 16:17 |
| sdague | anyway, need to run an errand, back in a bit | 16:17 |
| *** jcoufal has quit IRC | 16:18 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-gerrit: Disable Gerrit git repo repacking cron https://review.openstack.org/181965 | 16:18 |
| *** jcoufal has joined #openstack-infra | 16:18 | |
| clarkb | jeblair: fungi zaro ^ | 16:18 |
| fungi | toward the end of https://groups.google.com/forum/#!msg/repo-discuss/Qdmbl-YZ4NU/nMhm6IwQic0J the discussion is mostly just about jgit not handling repacks where the packfile name doesn't change, unrelated to choice of filesystem | 16:18 |
| *** alexsyip has joined #openstack-infra | 16:18 | |
| clarkb | also this appears to be very similar to the issue GitPython has where it loads up the git tree layout in memory and if it every changes on disk you lose | 16:19 |
| clarkb | s/every/ever/ | 16:19 |
| *** Krinkle is now known as Krinkle|detached | 16:19 | |
| zaro | hmm, i see that on our build of gerrit as well: http://logs.openstack.org/23/176523/1/check/check-gerrit-buck/1e1f702/console.html.gz | 16:19 |
| *** banix has joined #openstack-infra | 16:19 | |
| zaro | sdague: ^ but it seems to continue and finish ok | 16:19 |
| *** jogo has quit IRC | 16:19 | |
| *** devvesa has quit IRC | 16:21 | |
| *** baoli has quit IRC | 16:22 | |
| *** kmartin has joined #openstack-infra | 16:22 | |
| *** koolhead17 has quit IRC | 16:22 | |
| *** claudiub|2 has joined #openstack-infra | 16:22 | |
| jeblair | is there an issue in the tracker for the jgit thing? | 16:22 |
| zaro | sdague: make sure you are cloning buck from same location as our build: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n547 | 16:23 |
| *** jcoufal has quit IRC | 16:23 | |
| *** jcoufal has joined #openstack-infra | 16:23 | |
| openstackgerrit | Obulpathi proposed openstack-infra/project-config: Removes Pypy jobs for Poppy at Jenkins gate https://review.openstack.org/181970 | 16:24 |
| *** claudiub has quit IRC | 16:25 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add floating IP pool resource methods https://review.openstack.org/180255 | 16:25 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Rewrite extension checking methods https://review.openstack.org/181935 | 16:25 |
| fungi | still looking, haven't found one yet at least | 16:25 |
| clarkb | fungi: can I get a +A on https://review.openstack.org/#/c/181558/ I can rebuild a bare-trusty image this morning and babysit | 16:25 |
| *** sdake_ has joined #openstack-infra | 16:25 | |
| *** ddieterl_ has joined #openstack-infra | 16:26 | |
| Clint | https://code.google.com/p/gerrit/issues/detail?id=81 is super old | 16:26 |
| *** pcm_ has joined #openstack-infra | 16:27 | |
| clarkb | sdague: do you know if jhesketh has written a change to run the new os-loganalyze devstack plugin against os-loganalyze changes? | 16:27 |
| fungi | yeah, fixed in 2.0.11 so at best a regression of that bug, but more likely some new bug with similar symptoms | 16:27 |
| *** ddieterl_ has quit IRC | 16:27 | |
| dguerri | clarkb, btw old implementation didn't work (self._extension_cache used before assignment). I get the general point of yours, but at least in this case I guess nobody were using that method | 16:27 |
| dguerri | *was | 16:28 |
| *** obulpathi has joined #openstack-infra | 16:28 | |
| mordred | or, couldn't have been, since it flat out did not work | 16:28 |
| *** baoli has joined #openstack-infra | 16:28 | |
| clarkb | fungi: Clint what is old is new again/ | 16:28 |
| *** ddieterly has quit IRC | 16:29 | |
| *** BharatK has quit IRC | 16:29 | |
| *** pc_m has quit IRC | 16:29 | |
| *** pfallenop has quit IRC | 16:29 | |
| *** antonym has quit IRC | 16:29 | |
| *** alaski has quit IRC | 16:29 | |
| *** gothicmindfood has quit IRC | 16:29 | |
| *** marun has joined #openstack-infra | 16:29 | |
| *** sdake has quit IRC | 16:29 | |
| *** tim_o has quit IRC | 16:29 | |
| * mordred cries | 16:29 | |
| *** smccully has quit IRC | 16:29 | |
| *** ldnunes has quit IRC | 16:29 | |
| *** mmedvede_ has quit IRC | 16:29 | |
| *** tmcpeak has quit IRC | 16:30 | |
| clarkb | fungi: danke | 16:30 |
| *** tiswanso has quit IRC | 16:30 | |
| fungi | going over that thread in more detail, there is indeed a suggestion of using jgit to do the repacking' | 16:31 |
| *** pfallenop has joined #openstack-infra | 16:31 | |
| *** tiswanso_ has joined #openstack-infra | 16:31 | |
| *** gothicmindfood has joined #openstack-infra | 16:31 | |
| *** rmcall has quit IRC | 16:31 | |
| clarkb | fungi: any indication if we can do that without writing a special java utility just to do it? | 16:31 |
| clarkb | eg java -jar jgit.jar repace /home/gerrit2/review_site/git ? | 16:32 |
| *** rmcall has joined #openstack-infra | 16:32 | |
| *** _nadya_ has quit IRC | 16:32 | |
| mordred | fungi: just java it. in the docker. | 16:33 |
| clarkb | dguerri: ah ok, missed that, commit message says it didn't work but not because of the NameError | 16:33 |
| greghaynes | mordred: re: https://review.openstack.org/#/c/181750 I think simple-init should have that exact behavior | 16:34 |
| clarkb | we should jython | 16:34 |
| *** unicell has joined #openstack-infra | 16:34 | |
| zaro | fungi: Could that jgit thing be causing failures to view WIP changes? i don't see that problem on review-dev.o.o | 16:34 |
| jeblair | surely the problem is open files / stale internal cache, not a jgit-git incompatibility | 16:35 |
| greghaynes | mordred: and I think (or it should be) as easy as dependon on the cloud-init-datasources and exporting that var | 16:35 |
| *** alaski has joined #openstack-infra | 16:35 | |
| greghaynes | I can poke at that and see.. | 16:35 |
| *** Sukhdev has joined #openstack-infra | 16:35 | |
| clarkb | zaro: the problem appears to be with repacking behind jgits back and doing so in a way that jgit does not expect | 16:35 |
| mordred | greghaynes: so, I completely disagree with you | 16:35 |
| mordred | greghaynes: but it's possible I'm missing something or understanding you wrong | 16:35 |
| jeblair | (eg, if you did something like "java -jar jgit.jar repack" i would expect it to still break) | 16:35 |
| clarkb | zaro: basically C git can reuse the pack name but make a new pack, jgit explodes when that happens | 16:35 |
| jeblair | clarkb: oooh, i see what you're thinking | 16:36 |
| mordred | greghaynes: that behavior is about configuring how cloud-init works | 16:36 |
| clarkb | jeblair: I think the thread indicates ^ breaks gerrit, it expects a new pack name | 16:36 |
| mordred | greghaynes: I dont' think simple-init should have that behavior, because cloud-init should not exist on images with simple-init | 16:36 |
| jeblair | clarkb: gotcha, so it is an incompatibility, of sorts. | 16:36 |
| fungi | that's my reading as well | 16:36 |
| clarkb | ya | 16:36 |
| *** sdake has joined #openstack-infra | 16:36 | |
| clarkb | I suppose its possible newer C git introduces this for us too ? | 16:36 |
| fungi | http://permalink.gmane.org/gmane.comp.version-control.repo/13743 also discusses a possible workaround of not replacing packfiles | 16:36 |
| clarkb | since we did upgrade to trusty | 16:36 |
| greghaynes | mordred: Since we build off cloud images for most of the distros I think it would be nice to work with that setup somehow (just the ubuntu and simple-init element as an example) | 16:37 |
| greghaynes | mordred: and its not hard... | 16:37 |
| openstackgerrit | Merged openstack-infra/puppet-diskimage_builder: Update metadata to be diskimage_builder https://review.openstack.org/180850 | 16:37 |
| greghaynes | mordred: we just have to do exactly what that patch does | 16:37 |
| mordred | greghaynes: right - but ... so you're saying you think that if someone installs simple-init _with_ cloud-init, that they should have simple-init configure cloud-init to only use config-drive? | 16:38 |
| mordred | greghaynes: rather tahn having the simple-init element uninstall cloud-init if present? | 16:38 |
| greghaynes | oh what, sorry, I thought it did disable it | 16:39 |
| fungi | trying to figure out of git-index-pack's --keep option does this | 16:39 |
| mordred | greghaynes: right - so if it disables it, then what is the point of putting something in the data sources config? | 16:39 |
| fungi | hrm, perhaps not | 16:39 |
| *** yamamoto has quit IRC | 16:39 | |
| greghaynes | mordred: er, I thought that patch I was reading was there to disable it, youre right, it sets it to be configdrive | 16:39 |
| greghaynes | mordred: I think currently we dont disalbe cloud-init at all | 16:40 |
| *** deepakcs has quit IRC | 16:40 | |
| greghaynes | (in dib) | 16:40 |
| mordred | greghaynes: I agree with you | 16:40 |
| *** sdake_ has quit IRC | 16:40 | |
| mordred | greghaynes: so - a) I agree that simple-init should have a sane behavior on top of the distro-provided ubuntu element | 16:40 |
| *** otter768 has joined #openstack-infra | 16:40 | |
| *** dangers is now known as dangers_away | 16:41 | |
| clarkb | fungi: jeblair maybe the -F flag to repack would fix this? | 16:41 |
| clarkb | it will cost much cpu time though | 16:41 |
| samueldmq | hi, gerrit does not show any info for https://review.openstack.org/#/c/151677/ | 16:41 |
| fungi | aha, that's what i was looking for | 16:41 |
| samueldmq | just a blank page ^ : / | 16:41 |
| *** BharatK has joined #openstack-infra | 16:41 | |
| *** kgiusti has quit IRC | 16:42 | |
| clarkb | gah maybe repacking isn't causing the problems | 16:42 |
| *** antonym has joined #openstack-infra | 16:42 | |
| fungi | okay, maybe this isn't repack related if we've now got new failures since the restart | 16:42 |
| *** mmedvede_ has joined #openstack-infra | 16:42 | |
| *** smccully has joined #openstack-infra | 16:42 | |
| jeblair | org.eclipse.jgit.errors.MissingObjectException: Missing unknown 906fca0ba6642bc2673b32f17cd31190b90805eb | 16:42 |
| greghaynes | mordred: ok, so im fine with the change, sorry about that | 16:43 |
| jeblair | now the logs are full of that ^ | 16:43 |
| mordred | greghaynes: cool. I figured we were just missing each other somewhere | 16:43 |
| *** ldnunes has joined #openstack-infra | 16:43 | |
| mordred | jeblair: oh wow. thats so exciting | 16:43 |
| samueldmq | jeblair, oh, someone committed eclipse files ? | 16:43 |
| mordred | quite literally exactly what we wanted to discover 2 days after the upgrade | 16:44 |
| mordred | samueldmq: nah - that's the signature for jgit | 16:44 |
| *** derekh_ has joined #openstack-infra | 16:44 | |
| *** ssam2 has quit IRC | 16:44 | |
| mordred | samueldmq: it's a library that exists within the eclipse org | 16:44 |
| clarkb | jeblair: looks the same client side too, gerrit responds with a 304 unchanged for the change details | 16:45 |
| samueldmq | mordred, ++ makes sense | 16:45 |
| openstackgerrit | Merged openstack-infra/system-config: Use latest current virtualenv https://review.openstack.org/181558 | 16:45 |
| samueldmq | mordred, btw, do we have any known issues with gerrit today ? | 16:45 |
| clarkb | which is a really weird return code for "server failed" | 16:45 |
| *** otter768 has quit IRC | 16:45 | |
| *** banix has quit IRC | 16:46 | |
| samueldmq | I can't see any info for https://review.openstack.org/#/c/151677 | 16:46 |
| *** mpaolino has quit IRC | 16:46 | |
| mordred | samueldmq: yup. we're seeing that problem ^^ | 16:46 |
| *** ZZelle is now known as ZZelle_ | 16:46 | |
| mordred | samueldmq: which is what jeblair and clarkb are working on diagnosing right now | 16:46 |
| clarkb | and fungi | 16:46 |
| *** ericksonfgds has joined #openstack-infra | 16:46 | |
| jeblair | i'm looking in the log for a triggering event | 16:46 |
| samueldmq | mordred, oh ... maybe what I said is a side-effect | 16:46 |
| *** mleroy has joined #openstack-infra | 16:46 | |
| *** esker has quit IRC | 16:46 | |
| clarkb | I am going to go get caffeine before more debugging, back in a few | 16:46 |
| *** tnovacik has joined #openstack-infra | 16:47 | |
| *** derekh has quit IRC | 16:47 | |
| *** shashankhegde has joined #openstack-infra | 16:49 | |
| jeblair | [2015-05-11 16:30:13,687] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.pack is corrupt, removing it from pack list | 16:49 |
| jeblair | org.eclipse.jgit.errors.CorruptObjectException: Object at 87,706,216 in /home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.pack has bad zlib stream | 16:49 |
| jeblair | that happens right after the first "missing" error | 16:49 |
| fungi | seeing what git fsck reports for it | 16:50 |
| jeblair | fungi: maybe don't operate on the actual repo | 16:51 |
| *** derekh_ has quit IRC | 16:51 | |
| jeblair | fungi: i have one going on an rsync copy | 16:51 |
| fungi | ahh, yours has probably been running longer | 16:51 |
| fungi | i'm guessing jgit ran into some unexpected failure and didn't handle it well, and assumed a corrupt packfile | 16:52 |
| fungi | and that fsck will say everything's okay | 16:52 |
| *** dguitarbite has quit IRC | 16:52 | |
| fungi | similar to what's discussed in the commit message for https://gerrit-review.googlesource.com/63453 | 16:52 |
| *** harlowja_at_home has quit IRC | 16:53 | |
| jeblair | well, the fsck output exceeded my terminal buffer | 16:53 |
| fungi | and the https://code.google.com/p/gerrit/issues/detail?id=3094 bug linked from it | 16:53 |
| *** banix has joined #openstack-infra | 16:54 | |
| *** afazekas has quit IRC | 16:55 | |
| jeblair | the fsck output is: | 16:56 |
| jeblair | error: invalid parameter: expected sha1, got '.' | 16:56 |
| jeblair | followed by 1246 lines like: | 16:56 |
| jeblair | dangling blob 831000ca7365a61e700460703a6cc65d22355e03 | 16:56 |
| jeblair | or | 16:56 |
| *** tmcpeak has joined #openstack-infra | 16:56 | |
| jeblair | dangling commit 0a5380df6ccf793b87dd985c8b657d0b7d40d22c | 16:56 |
| jeblair | it's in review.o.o:~corvus/nova.git/fsck-out | 16:56 |
| *** davideagnello has joined #openstack-infra | 16:58 | |
| *** yamahata has quit IRC | 16:58 | |
| *** banix has quit IRC | 16:59 | |
| *** yamahata has joined #openstack-infra | 16:59 | |
| *** eharney_ is now known as eharney | 16:59 | |
| jeblair | but it doesn't say anything about a bad zlib stream | 16:59 |
| fungi | no mention of the 93ad57004de887eb835b2bd4df2d7c3f6a5c394b packfile there | 16:59 |
| *** amotoki_ has quit IRC | 16:59 | |
| clarkb | does git show on 906fca0ba6642bc2673b32f17cd31190b90805eb work from your repo copy? | 17:00 |
| *** davideagnello has quit IRC | 17:00 | |
| fungi | yep | 17:00 |
| *** davideagnello has joined #openstack-infra | 17:00 | |
| fungi | looks like a normal commit | 17:00 |
| *** sshnaidm has quit IRC | 17:00 | |
| clarkb | so git porcelain commands are able to go find it | 17:01 |
| *** _nadya_ has joined #openstack-infra | 17:01 | |
| *** whoops has joined #openstack-infra | 17:01 | |
| *** dizquierdo has joined #openstack-infra | 17:01 | |
| fungi | starting to wonder if the issues fixed in the jgit upgrade that got rolled back are biting us, and whether we want to add that commit that brings in jgit 4 | 17:01 |
| *** dkranz has joined #openstack-infra | 17:01 | |
| fungi | revisiting the details from that now | 17:01 |
| jeblair | git show-index < pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.idx|grep 87706216 | 17:02 |
| jeblair | 87706216 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118 (b60fcf57) | 17:02 |
| jeblair | git show 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118 works too | 17:02 |
| fungi | basically, we are in theory running a jgit affected by https://bugs.eclipse.org/bugs/show_bug.cgi?id=422988 | 17:03 |
| openstack | bugs.eclipse.org bug 422988 in JGit "checkReferencedIsReachable fails to find a tree which is not sent as part of a pack" [Normal,Reopened] - Assigned to jgit.core-inbox | 17:03 |
| jeblair | running git verify-pack pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.idx | 17:04 |
| fungi | also i didn't get "error: invalid parameter: expected sha1, got '.'" from my git fsck. might have had to do with invocation? i just ran `git fsck` while my cwd was in my raw copy of the repo | 17:04 |
| clarkb | looks like https://git.eclipse.org/r/#/c/46486/2 is the newly proposed fix? | 17:04 |
| jeblair | fungi: oh, yeah, that was probably it, sorry | 17:04 |
| *** annegentle has joined #openstack-infra | 17:05 | |
| jeblair | git verify-pack pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.idx returned without error or output | 17:05 |
| *** shashankhegde has quit IRC | 17:05 | |
| fungi | seems like jgit is confused and thinks the packfile is corrupt when in fact it probably hit some other transient problem | 17:06 |
| fungi | and then defensively stops retrying to read it on subsequent requests, persisting in its belief it's corrupt | 17:06 |
| clarkb | fungi: so restarting the server may help just to reset that "corrupt" flag | 17:07 |
| clarkb | and I suppose its possible for packing to move things such that we hit the bug, but is not the cause of the bug | 17:07 |
| zaro | i've restored the update to jgit 4.0 in case we need it: https://review.openstack.org//#/c/180781/ | 17:07 |
| *** aduarte has joined #openstack-infra | 17:07 | |
| clarkb | zaro: well I think we will want https://git.eclipse.org/r/#/c/46486/2 once it is ready since that is supposed to fix the performance issues | 17:08 |
| jeblair | fungi: that patch seems to be related to receiving packs, so i wonder if it's relevant to current situation (it may be through some mechanism i don't understand, but on the surface, i would guess not?) | 17:08 |
| clarkb | zaro: but starting with the bad performing patch is likely a good start | 17:08 |
| clarkb | jeblair: thats how you push code, so could happen when new patchsets are pushed? | 17:08 |
| fungi | jeblair: yeah, seems at best tangentially related | 17:08 |
| openstackgerrit | Sergey Lukjanov proposed openstack-infra/project-config: Grant sahara-ci group permissions to verify all sahara repos https://review.openstack.org/181988 | 17:09 |
| * fungi was just grasping at the one known jgit issue we had discussed in our upgrade planning | 17:09 | |
| jeblair | clarkb: ah, hrm. | 17:09 |
| *** banix has joined #openstack-infra | 17:09 | |
| fungi | http://git.eclipse.org/c/jgit/jgit.git/commit/?h=stable-3.7&id=f02e4a61887d89bf97dffa8e626735c3675cabe3 was apparently the landed revert for the performance problem | 17:10 |
| clarkb | jeblair: `git upload-pack` is the C git thing that does it iirc | 17:10 |
| *** baoli has quit IRC | 17:10 | |
| mleroy | Hi all. The changes for the creation of the networking-6wind project have been merged. Can you please add me to both groups (networking-6wind-core, networking-6wind-release) in gerrit? Or should I send an email to the mailing list for that ? Maxime Leroy | 17:11 |
| clarkb | oh wait, send-pack | 17:11 |
| *** _nadya_ has quit IRC | 17:11 | |
| *** baoli has joined #openstack-infra | 17:12 | |
| *** claudiub has joined #openstack-infra | 17:12 | |
| *** ivar-lazzaro has quit IRC | 17:12 | |
| *** e0ne has quit IRC | 17:13 | |
| *** sputnik13 has joined #openstack-infra | 17:13 | |
| jeblair | the action that appears to have triggered the zlib-related failure appears to be an ssh query operation -- so the direct cause does not appear to be an upload, but i suppose it could have had a delayed impact? | 17:13 |
| *** btully has quit IRC | 17:14 | |
| *** Guest28311 has quit IRC | 17:14 | |
| *** prad_ has joined #openstack-infra | 17:14 | |
| clarkb | mleroy: we are currently debugging issues with the gerrit upgrade, sending email might be best | 17:14 |
| clarkb | mleroy: so it doesn't get lost in scrollback | 17:15 |
| *** jcoufal has quit IRC | 17:15 | |
| mleroy | clarkb: ok thanks | 17:15 |
| *** claudiub|2 has quit IRC | 17:15 | |
| *** turul has joined #openstack-infra | 17:15 | |
| clarkb | jeblair: perhaps, I can imagine that a push via send-pack would make the repo unhappy, then the next query goes to look in the repo and throws the exception | 17:15 |
| *** turul is now known as afazekas | 17:15 | |
| *** dguitarbite has joined #openstack-infra | 17:15 | |
| jeblair | http://paste.openstack.org/show/220035/ | 17:15 |
| *** ddieterly has joined #openstack-infra | 17:16 | |
| jeblair | there's the relevant log snippet; the main error is in the middle | 17:16 |
| Kiall | In case anyone hasn't noticed it already, the review merged -> launchpad updates stuff seems flaky - https://bugs.launchpad.net/designate/+bug/1437699/comments/6 | 17:16 |
| openstack | Launchpad bug 1437699 in Designate "mDNS should Handle Bad File Descriptor" [Medium,Fix committed] - Assigned to Kiall Mac Innes (kiall) | 17:16 |
| *** prad has quit IRC | 17:17 | |
| *** tiswanso_ has quit IRC | 17:18 | |
| *** cdent has quit IRC | 17:18 | |
| *** jogo has joined #openstack-infra | 17:19 | |
| *** jogo has quit IRC | 17:19 | |
| *** jogo has joined #openstack-infra | 17:19 | |
| fungi | Kiall: looks like that may be unrelated to the gerrit upgrade. it's from a tag merge change, so probably enumerating all the commits between stable/kilo and the tag merge point on master | 17:19 |
| fungi | Kiall: if you find other examples which aren't from a merge change, then i agree we may have a deeper issue | 17:20 |
| clarkb | jeblair: reading that it appears the diff timed out because it couldn't get the object because the pack was "corrupt" ? is that how you read the log? | 17:21 |
| *** jamesmcarthur has joined #openstack-infra | 17:21 | |
| jeblair | clarkb: i suspect the diff timeout is unrelated, they show up fairly frequently normally | 17:21 |
| *** tiswanso_ has joined #openstack-infra | 17:21 | |
| *** ddieterl_ has joined #openstack-infra | 17:22 | |
| fungi | https://bugzilla.redhat.com/show_bug.cgi?id=1085328 | 17:22 |
| openstack | bugzilla.redhat.com bug 1085328 in Business Central "Git: cannot clone repositories from business-central to different machine" [High,Closed: currentrelease] - Assigned to porcelli | 17:22 |
| *** ddieterly has quit IRC | 17:22 | |
| fungi | has to do with jgit timeouts causing "packfile truncated" errors | 17:22 |
| Kiall | fungi: ah, Okay.. that makes sense.. I assumed upgrade related ;) | 17:22 |
| fungi | so possible that a timeout reading could cause jgit to get confused and think it's corrupt | 17:23 |
| pleia2 | :\ | 17:23 |
| jamesmcarthur | clarkb: fungi: - clarkb alerted me one of our server IPs being blocked over the weekend for excessive Gerrit queries. I took a look at the queries and we are paging. However, they could use a little refining. We are pulling merged changes for local stats for the Foundation (# of countries merging, % merging, etc…). We’re going to work to clean up the queries a bit so we’re not hitting you all so frequently. | 17:24 |
| *** patrickeast has joined #openstack-infra | 17:24 | |
| *** emagana has quit IRC | 17:24 | |
| *** ihrachyshka has quit IRC | 17:24 | |
| *** krtaylor has quit IRC | 17:24 | |
| *** arxcruz has quit IRC | 17:24 | |
| clarkb | jamesmcarthur: thank you, I think we can refine by adding an age parameter so that we aren't getting the entire history of all merges | 17:25 |
| fungi | jamesmcarthur: thanks. it looked like the frequency of the queries looking for merges was the biggest drain. it was causing a fair amount of drag on gerrit performance too | 17:25 |
| jamesmcarthur | fungi: clarkb: One thing I noticed is that we’re not limiting by project. Am I correct in limiting it to only projects starting with openstack/ | 17:25 |
| jamesmcarthur | And yes clarkb: - we’re adding an age param as well | 17:25 |
| jamesmcarthur | fungi: My apologies for the drag on Gerrit :| No bueno. | 17:25 |
| *** emagana has joined #openstack-infra | 17:26 | |
| fungi | jamesmcarthur: not really. the repository naming is only mostly indicative of openstack officialness. there's actually a structured data file maintained by the technical committee which enumerates them http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml | 17:26 |
| jeblair | clarkb: there are 3 pack file removals in the log; two of them are near a diff timeout, one is not. | 17:26 |
| *** dizquierdo has quit IRC | 17:27 | |
| fungi | jamesmcarthur: so checking all projects hosted in gerrit is likely necessary | 17:27 |
| jeblair | clarkb: the first removal is: | 17:27 |
| jeblair | [2015-05-11 02:59:02,155] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /home/gerrit2/review_site/git/stackforge/fuel-stats.git/objects/pack/pack-64b011fb58533468215ff41155731039b6cd56e1.pack was deleted, removing it from pack list | 17:27 |
| fungi | jamesmcarthur: depending on what you're trying to determine with that query anyway | 17:28 |
| jamesmcarthur | fungi: thanks jeremy, I found that this monring. However a lot of those we don’t need data on. For instance stackforge projects. | 17:28 |
| clarkb | jeblair: huh, thats not when we repack, weird that it would decide the pack was deleted | 17:28 |
| jamesmcarthur | Just trying to figure out if there is some common thread we can use to pull a more limited data set. | 17:28 |
| fungi | clarkb: unless that was the first time since the repack that it tried to access that file | 17:28 |
| clarkb | fungi: good point | 17:28 |
| jeblair | jamesmcarthur: depending on what you want, you may want to include stackforge projects. also, if you have in the past, i certainly would not stop now, especially not with the big-tent direction the project is taking. | 17:29 |
| jeblair | clarkb: the second and third times were for the same nova pack file (and that was across our restart) | 17:29 |
| jeblair | org.eclipse.jgit.errors.CorruptObjectException: Object at 89,109,081 in /home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.pack has bad zlib stream | 17:29 |
| *** baoli has quit IRC | 17:29 | |
| jeblair | [2015-05-11 02:59:02,155] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /home/gerrit2/review_site/git/stackforge/fuel-stats.git/objects/pack/pack-64b011fb58533468215ff41155731039b6cd56e1.pack was deleted, removing it from pack list | 17:29 |
| jeblair | oops, sorry wrong paste | 17:29 |
| jeblair | meant to say: | 17:30 |
| jeblair | org.eclipse.jgit.errors.CorruptObjectException: Object at 89,109,081 in /home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.pack has bad zlib stream | 17:30 |
| jeblair | org.eclipse.jgit.errors.CorruptObjectException: Object at 87,706,216 in /home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-93ad57004de887eb835b2bd4df2d7c3f6a5c394b.pack has bad zlib stream | 17:30 |
| *** craigbr has quit IRC | 17:30 | |
| jeblair | so same pack file, different indexes | 17:30 |
| clarkb | jeblair: and one before restart one after right? | 17:30 |
| jeblair | yep | 17:30 |
| *** sks has quit IRC | 17:31 | |
| *** moshele has joined #openstack-infra | 17:31 | |
| jeblair | clarkb, fungi: i'm willing to discount the fuel one for the moment since it was after a repack | 17:31 |
| *** vipul has quit IRC | 17:31 | |
| jeblair | but i'm starting to wonder if there really is something about the pack-93 file that jgit doesn't like? | 17:31 |
| clarkb | jeblair: seems possible | 17:32 |
| jogo | I am getting a fatal internal server error when I call 'ssh jogo@review.openstack.org -p 29418 gerrit query --format=JSON is:watched status:merged | 17:33 |
| jeblair | jogo: likely related to our current investigation | 17:33 |
| jeblair | it might be time for an alert | 17:33 |
| mordred | agree | 17:33 |
| clarkb | https://wiki.eclipse.org/JGit/User_Guide#Running_the_JGit_CLI jgit does have a cli interface but no fsck (was hoping we could compare fscks between C git and jgit) | 17:33 |
| zaro | fungi: that '//' issue appears on review-dev.o.o as well but the link seems to work fine. wondering what's different | 17:33 |
| *** baoli has joined #openstack-infra | 17:34 | |
| *** shashankhegde has joined #openstack-infra | 17:34 | |
| *** notnownikki has quit IRC | 17:34 | |
| fungi | zaro: restart gerrit there. i was testing my proposed fix | 17:35 |
| jogo | jeblair: thanks | 17:35 |
| zaro | fungi: ahh ok. | 17:35 |
| jeblair | status alert We have discovered post-upgrade issues with Gerrit affecting nova (and potentially other projects). Some changes will not appear and some actions, such as queries, may return an error. We are continuing to investigate. | 17:35 |
| jeblair | does that look good ^ ? | 17:36 |
| clarkb | jeblair: yup | 17:36 |
| pleia2 | thumbs up | 17:36 |
| jeblair | #status alert We have discovered post-upgrade issues with Gerrit affecting nova (and potentially other projects). Some changes will not appear and some actions, such as queries, may return an error. We are continuing to investigate. | 17:36 |
| openstackstatus | jeblair: sending alert | 17:36 |
| *** krtaylor has joined #openstack-infra | 17:36 | |
| *** yfried has quit IRC | 17:37 | |
| fungi | post-facto lgtm | 17:38 |
| clarkb | I have installed jgit-cli locally and am trying to see if it gives us any interfaces into sanity | 17:38 |
| -openstackstatus- NOTICE: We have discovered post-upgrade issues with Gerrit affecting nova (and potentially other projects). Some changes will not appear and some actions, such as queries, may return an error. We are continuing to investigate. | 17:38 | |
| *** ChanServ changes topic to "We have discovered post-upgrade issues with Gerrit affecting nova (and potentially other projects). Some changes will not appear and some actions, such as queries, may return an error. We are continuing to investigate." | 17:38 | |
| jeblair | clarkb: ++ thx | 17:38 |
| fungi | ooh, i didn't realize jgit-cli was a thing | 17:38 |
| fungi | good call | 17:38 |
| jeblair | i need to afk for about 10 mins. i have emitted our problem into the #gerrit channel, no response so far. back asap. | 17:39 |
| clarkb | but confirmed no fsck is available, do have a gc though | 17:39 |
| fungi | gerrit seems to do its own gc as well, if i'm interpreting the files in ~gerrit2/review_site/logs correctly | 17:40 |
| *** moshele has quit IRC | 17:40 | |
| mordred | is that new since we added our cronjob gc? | 17:40 |
| clarkb | mordred: we cronjob repack not gc which happens to also prune | 17:40 |
| fungi | [2015-05-11 16:13:36,603] INFO : Ignoring missing gc schedule configuration | 17:40 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Add pep8 job to nova-specs https://review.openstack.org/181994 | 17:40 |
| openstackstatus | jeblair: finished sending alert | 17:40 |
| fungi | looks like maybe it _would_ do them if we configured it to | 17:40 |
| *** mestery has joined #openstack-infra | 17:41 | |
| fungi | seems to add that to the log each time gerrit starts | 17:41 |
| tmcpeak | hey guys, what's the best way to find out which OpenStack projects are using a certain requirement? | 17:42 |
| clarkb | tmcpeak: clone all of the projects then find -name \*requirements.txt -exec grep $requirement {}; | 17:42 |
| tmcpeak | clarkb: *all the projects*? | 17:42 |
| fungi | tmcpeak: eventually something like livegrep. for now i have a cron job which keeps updated clones of everything from gerrit and then i loop over them ang git grep | 17:42 |
| fungi | er, and | 17:43 |
| tmcpeak | fungi: ahh ok, I guess where's a good comprehensive list to point it at? | 17:43 |
| fungi | tmcpeak: the gerrit api | 17:43 |
| tmcpeak | fungi: cool, thanks | 17:43 |
| *** annegentle has quit IRC | 17:43 | |
| *** baoli has quit IRC | 17:43 | |
| fungi | rm_work: ssh -p 29418 review.openstack.org gerrit ls-projects | 17:44 |
| tmcpeak | fungi: cool, thank you | 17:44 |
| *** mestery has quit IRC | 17:44 | |
| openstackgerrit | Steve Leon proposed openstack-infra/project-config: Add devstack gate job for Cue project https://review.openstack.org/180774 | 17:45 |
| fungi | tmcpeak: my hackish script i run from cron is http://paste.openstack.org/show/220044/ if you want ideas | 17:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Update images API for get/list/search interface https://review.openstack.org/180600 | 17:45 |
| fungi | rm_work: sorry, meant to highlight tmcpeak and my tab-complete failed me | 17:46 |
| *** mestery has joined #openstack-infra | 17:46 | |
| tmcpeak | fungi: perfect, thanks! | 17:46 |
| clarkb | do we know what change fc05ab545c014f31078b3d53f843e299e95b8a51 belongs to in nova so I can fetch it? | 17:47 |
| fungi | tmcpeak: obviously beware of hardcoded dragons, like my callouts to keychain and embedded username in my git remote | 17:47 |
| clarkb | http://git.openstack.org/cgit/openstack/nova/commit/?id=fc05ab545c014f31078b3d53f843e299e95b8a51 that actualy worked surprisingly well | 17:47 |
| tmcpeak | fungi: yep, I don't copy and paste *anything* ;) | 17:48 |
| tmcpeak | from untrusted sources anyway, no offense | 17:48 |
| fungi | tmcpeak: here, wget this and pipe it through sudo bash ;) | 17:48 |
| fungi | tmcpeak: i expect nothing less! | 17:48 |
| tmcpeak | +1 for a good time | 17:48 |
| clarkb | using the jgit-cli shipped by trusty I can `jgit fetch --fsck https://git.openstack.org/openstack/nova refs/changes/63/111563/4` and that works, now to see if I can build the same jgit gerrit is using | 17:49 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Use shade for image uploads https://review.openstack.org/168633 | 17:50 |
| openstackgerrit | Louis Taylor proposed openstack-infra/project-config: Add experimental glance_store functional test job https://review.openstack.org/182000 | 17:52 |
| *** Krinkle|detached is now known as Krinkle | 17:52 | |
| fungi | zuul is dealing interestingly with one of the broken changes... (see top of integrated gate change queue in status_ | 17:53 |
| *** moshele has joined #openstack-infra | 17:53 | |
| fungi | or i should say not dealing well at all | 17:53 |
| fungi | looks like that has gummed it up entirely | 17:53 |
| openstackgerrit | Swaminathan Vasudevan proposed openstack-infra/project-config: Make neutron-dvr job voting at the gate https://review.openstack.org/180230 | 17:53 |
| mordred | fungi: you know I have a code indexer running, right? just waiting on reviews of the puppet before I start handing end-users the url ... | 17:53 |
| *** achanda has joined #openstack-infra | 17:54 | |
| fungi | mordred: yep, just couldn't remember the name of it | 17:54 |
| greghaynes | mordred: openstack cross reference? | 17:54 |
| mordred | fungi: I say that not because it's important right now, but because you just said "eventually something like livegrep" - but there is actually one running | 17:54 |
| *** cdent has joined #openstack-infra | 17:54 | |
| mordred | greghaynes: yes | 17:54 |
| mordred | well, an index of all of the source code in gerrit is what it actually is - but yeah | 17:54 |
| fungi | mordred: more specifically, that's why i said "something like" (because i couldn't remember what the new thing you got working is called) | 17:55 |
| mordred | https://review.openstack.org/#/q/status:open+branch:master+topic:hound | 17:55 |
| mordred | fungi: :) | 17:55 |
| openstackgerrit | Louis Taylor proposed openstack-infra/project-config: Add experimental glance_store functional test job https://review.openstack.org/182000 | 17:55 |
| *** ibiris is now known as ibiris_away | 17:55 | |
| fungi | oh, right, hound | 17:55 |
| jeblair | i think it would be a good use of time to start preparing for a downgrade | 17:56 |
| clarkb | zaro: I am thinking we should try to reproduce these issues on review-dev. Maybe we put nova on that server. Then we upgrade to gerrit-v2.10.2.24.92f609a.war and see if it fixes | 17:56 |
| clarkb | jeblair: not ^ | 17:56 |
| clarkb | ? | 17:56 |
| *** esker has joined #openstack-infra | 17:56 | |
| mordred | jeblair: oh wow | 17:56 |
| *** ivar-lazzaro has joined #openstack-infra | 17:56 | |
| jeblair | nova is basically dead in the water | 17:56 |
| *** mestery has quit IRC | 17:56 | |
| mordred | maybe we do the two in parallel? | 17:57 |
| jeblair | mordred: yes, that's what i'm suggesting | 17:57 |
| mordred | ok | 17:57 |
| mordred | then yes - I totally agree | 17:57 |
| *** prad_ is now known as prad | 17:57 | |
| jeblair | clarkb: what's .24? | 17:57 |
| clarkb | jeblair: https://review.openstack.org/#/c/180781/ that change | 17:57 |
| jeblair | jgit4? | 17:57 |
| clarkb | yes | 17:57 |
| *** baoli has joined #openstack-infra | 17:58 | |
| jeblair | i believe the main thing we need to do for a downgrade would be to reverse the schema changes | 17:58 |
| *** pelix has quit IRC | 17:58 | |
| clarkb | jeblair: sounds right, we have to reindex which will build that from scratch | 17:58 |
| jeblair | clarkb: agreed | 17:59 |
| fungi | yeah, i'm on board | 17:59 |
| jeblair | would someone like to go and prepare the sql to reverse the schema changes? | 17:59 |
| mordred | so - worth looking at the schema changes to make sure we actually do need to schema downgrade? | 17:59 |
| jeblair | mordred: i'm pretty sure we need to do the work to create the downgrade ourselves; i don't know of a way to ask gerrit to downgrade to a certain version | 18:00 |
| fungi | i'm sort of wondering how we reproduce still, because i get the impression that gerrit restarts clear up the changes it's thinking are broken | 18:00 |
| mordred | jeblair: yes. I agree that we will need to do that | 18:00 |
| clarkb | I think mordred is suggesting that the curernt schema may be compat | 18:00 |
| clarkb | and we should check | 18:00 |
| *** dguitarbite has quit IRC | 18:00 | |
| fungi | at least that's what we saw so far | 18:00 |
| mordred | but yes ^^ | 18:00 |
| mordred | we may not need to manipulate the db | 18:00 |
| * mordred is going to read source code | 18:00 | |
| jeblair | mordred: so the next step after writing the sql to downgrade is to see what the delta is and think about ti | 18:00 |
| *** btully has joined #openstack-infra | 18:01 | |
| jeblair | mordred: but step 1 regardless, is to figure that out | 18:01 |
| mordred | yes | 18:01 |
| mordred | I agree with you - and I'm looking at it now | 18:01 |
| jeblair | mordred: and that's what i'm asking for a volunteer for; is that you? :) | 18:01 |
| mordred | yah | 18:01 |
| jeblair | mordred: we were on schema 86 before | 18:02 |
| jeblair | mordred: thanks | 18:02 |
| *** ajmiller has joined #openstack-infra | 18:02 | |
| clarkb | zaro: so jgit 4.0 is basically https://git.eclipse.org/r/#/c/46447/ are you suggesting that there are other fixes in 4.0 on top of our current 3.x state which does not include that patch? | 18:02 |
| clarkb | zaro: if so, do you expect those changes to fix this problem? | 18:02 |
| * clarkb is trying to figure out if he should spend time with review-dev to reproduce and apply the .24 war | 18:03 | |
| jeblair | clarkb: i think we'll need part or all of the database on review-dev too, so that we can look at changes in gerrit and cause it to try to access git | 18:03 |
| mordred | where are the migrations? | 18:03 |
| *** dprince has quit IRC | 18:03 | |
| jeblair | gerrit-server/src/main/java/com/google/gerrit/server/schema/ | 18:04 |
| mordred | thank you | 18:04 |
| jeblair | mordred: and just to confirm, we're currently on schema 98. | 18:05 |
| mordred | AWEWSOME | 18:05 |
| *** btully has quit IRC | 18:05 | |
| mordred | 13 java schema migrations | 18:05 |
| zaro | clarkb: yes, i believe 4.0 contains more than that fix but i have no idea what the extra fixes are so don't know if it will fix our particular problem. | 18:05 |
| *** obulpathi has left #openstack-infra | 18:05 | |
| fungi | i'm currently reading through the jgit changelog to see if anything looks relevant | 18:06 |
| clarkb | https://github.com/eclipse/jgit/commit/ae4b72d50e84f9af7978075d4919ad4d2b1d018a is one fix that may be handy that is in 4.0 | 18:07 |
| clarkb | fungi: me too :) | 18:07 |
| mordred | jeblair: so - the first migration, 87, seems like it might be hard to revert | 18:07 |
| mordred | jeblair: as it deletes SYSTEM groups | 18:07 |
| jeblair | mordred: so we might restore those from our dump | 18:08 |
| mordred | jeblair: yah | 18:08 |
| clarkb | fungi: thats the only one I am seeing between 3.7.x and 4.0.0 that looks helpful | 18:09 |
| fungi | how can i easily tell which jgit we've currently built on? the build log or is there a way to ask gerrit? | 18:10 |
| clarkb | oh but we use 3.7.0 and there is 3.7.2 | 18:10 |
| clarkb | fungi: https://review.openstack.org/#/c/180781/1/lib/jgit/BUCK | 18:10 |
| clarkb | fungi: old side is what we use now | 18:10 |
| fungi | ahh, yep, just ended up back tehre myself. thanks | 18:10 |
| fungi | so basically 65c379e..v4.0.0.201505050340-m2 | 18:12 |
| *** annegentle has joined #openstack-infra | 18:12 | |
| clarkb | that looks correct | 18:13 |
| fungi | eek, https://github.com/eclipse/jgit/commit/4feffb3 | 18:14 |
| clarkb | I really hope that --0700 is meant to be +0700 but that doesn't seem likely | 18:15 |
| *** emagana has quit IRC | 18:15 | |
| fungi | also https://github.com/eclipse/jgit/commit/6e5c71b | 18:16 |
| fungi | no idea if that plays into this at all | 18:16 |
| clarkb | https://github.com/eclipse/jgit/commit/3cd7d0d85a5fe6ba03f218898e866f06258b7405 is fun | 18:17 |
| *** emagana has joined #openstack-infra | 18:17 | |
| fungi | clarkb: the irony is in the end of the last file in that diff | 18:18 |
| *** achanda has quit IRC | 18:18 | |
| clarkb | indeed | 18:18 |
| mordred | jeblair: ok -I've gone through the schema versions | 18:19 |
| mordred | jeblair, zaro clarkb: what's up with the empty ones? there are SEVERAL of them that just seem to increment the version and do nothing else | 18:19 |
| *** yfried has joined #openstack-infra | 18:19 | |
| *** yfried has quit IRC | 18:19 | |
| jeblair | mordred: i have no idea; maybe git blame and check out the commits? | 18:19 |
| clarkb | mordred: perhaps thats how they reserve them for older point releases? | 18:19 |
| mordred | clarkb: that's right | 18:20 |
| *** arielb has joined #openstack-infra | 18:20 | |
| *** arielb has joined #openstack-infra | 18:20 | |
| fungi | yeah, solution for later migration injection | 18:20 |
| *** tonytan4ever has joined #openstack-infra | 18:20 | |
| clarkb | fungi: zaro so I am not seeing any bugfixes that standout as fixing this (except possibly the first one I linked) | 18:20 |
| *** baoli has quit IRC | 18:21 | |
| clarkb | fungi: zaro but there does seem to be a decent sized set of bug fixes in general | 18:21 |
| *** baoli has joined #openstack-infra | 18:21 | |
| *** funzo_ is now known as funzo | 18:21 | |
| mordred | ah. nope. it's more isidious | 18:22 |
| clarkb | so probably worth trying out on review-dev if we can reproduce the failures with nova there | 18:22 |
| mordred | insidious | 18:22 |
| mordred | whatever | 18:22 |
| clarkb | mordred: darth insidious? | 18:22 |
| mordred | the java infers all the columns that need to exist | 18:22 |
| mordred | the migration code only handles other things | 18:22 |
| fungi | oh, ew | 18:22 |
| mordred | yah | 18:22 |
| fungi | that's going to be a pain to unwind | 18:22 |
| mordred | sigh. | 18:22 |
| clarkb | mordred: so you have an empty migration if the java changed? | 18:22 |
| mordred | yah | 18:22 |
| mordred | 97 is a good example | 18:23 |
| mordred | it adds a column in teh java model | 18:23 |
| mordred | and then has an empty migration | 18:23 |
| fungi | so basically empty migration means "java magic happens here" | 18:23 |
| jeblair | well, all migrations mean that, right? empty ones mean that no transformations were necessary | 18:23 |
| fungi | er, right | 18:23 |
| mordred | well, I think what will happen is when you go to run migrations, it will look at the current data model, compute needed additional columns and apply all of them | 18:23 |
| mordred | then run through the schema migrations one at a time to handle things that aren't in teh java data model | 18:24 |
| mordred | like indexes or data transforms | 18:24 |
| fungi | so basically any migration might also have unspecified implications in the java source | 18:24 |
| clarkb | mordred: so additions are ok as long as it doesn't also remove things? | 18:24 |
| *** hemna_ has joined #openstack-infra | 18:24 | |
| *** unicell has quit IRC | 18:25 | |
| *** alaski has quit IRC | 18:25 | |
| *** dboik_ has quit IRC | 18:25 | |
| *** mpopow has quit IRC | 18:25 | |
| *** sarob has quit IRC | 18:25 | |
| *** bnemec has quit IRC | 18:25 | |
| *** kaisers has quit IRC | 18:25 | |
| *** Guest64787 has quit IRC | 18:25 | |
| *** smallbig has quit IRC | 18:25 | |
| *** morganfainberg has quit IRC | 18:25 | |
| *** russellb has quit IRC | 18:25 | |
| *** adreznec has quit IRC | 18:25 | |
| *** mfink- has quit IRC | 18:25 | |
| *** b^_^d has quit IRC | 18:25 | |
| *** SotK has quit IRC | 18:25 | |
| *** tsufiev has quit IRC | 18:25 | |
| *** mrda has quit IRC | 18:25 | |
| *** ianw has quit IRC | 18:25 | |
| mordred | yah. I mean I think we can run on a half-migrated db - but for reference: | 18:25 |
| mordred | http://git.openstack.org/cgit/openstack-infra/gerrit/tree/gerrit-reviewdb/src/main/java/com/google/gerrit/reviewdb/client/AccountDiffPreference.java?id=a64d689c6beb7b97f1ccde217e2e1da9695cb85b#n156 | 18:25 |
| jeblair | one thing to keep in mind is that if we run the old version without a schema downgrade, when we upgrade again, we may be in a pickle. | 18:25 |
| mordred | jeblair: yup | 18:25 |
| *** samueldmq_ has joined #openstack-infra | 18:26 | |
| mordred | lines 156 and 157 got added in the same commit | 18:26 |
| *** larsks|alt has joined #openstack-infra | 18:26 | |
| mordred | as added http://git.openstack.org/cgit/openstack-infra/gerrit/tree/gerrit-server/src/main/java/com/google/gerrit/server/schema/Schema_97.java?id=a64d689c6beb7b97f1ccde217e2e1da9695cb85b#n20 | 18:26 |
| *** mjg has joined #openstack-infra | 18:26 | |
| *** radez_g0n3 has joined #openstack-infra | 18:26 | |
| mordred | so 156 and 157 add the column | 18:26 |
| *** emagana has quit IRC | 18:26 | |
| *** gilliard has quit IRC | 18:26 | |
| *** larsks has quit IRC | 18:26 | |
| *** samueldmq has quit IRC | 18:26 | |
| *** rcarrill` has quit IRC | 18:26 | |
| *** dmellado has quit IRC | 18:26 | |
| *** tlbr has quit IRC | 18:26 | |
| *** hemna has quit IRC | 18:26 | |
| *** slagle has quit IRC | 18:26 | |
| *** phschwartz has quit IRC | 18:26 | |
| *** claudiub|2 has joined #openstack-infra | 18:26 | |
| *** claudiub has quit IRC | 18:26 | |
| *** yfried has joined #openstack-infra | 18:26 | |
| *** Shrews has quit IRC | 18:26 | |
| *** zaneb has quit IRC | 18:26 | |
| *** bkero has quit IRC | 18:26 | |
| *** hamzy has quit IRC | 18:26 | |
| *** waynr has quit IRC | 18:26 | |
| *** radez has quit IRC | 18:26 | |
| *** hamzy_ has joined #openstack-infra | 18:26 | |
| mordred | and Schema_97 does nothing | 18:26 |
| *** samueldmq_ is now known as samueldmq | 18:26 | |
| *** waynr has joined #openstack-infra | 18:27 | |
| *** zaneb has joined #openstack-infra | 18:27 | |
| *** slagle has joined #openstack-infra | 18:27 | |
| *** alaski has joined #openstack-infra | 18:27 | |
| *** unicell has joined #openstack-infra | 18:27 | |
| *** dboik_ has joined #openstack-infra | 18:27 | |
| *** mpopow has joined #openstack-infra | 18:27 | |
| *** sarob has joined #openstack-infra | 18:27 | |
| *** bnemec has joined #openstack-infra | 18:27 | |
| *** kaisers has joined #openstack-infra | 18:27 | |
| *** smallbig has joined #openstack-infra | 18:27 | |
| *** b^_^d has joined #openstack-infra | 18:27 | |
| *** morganfainberg has joined #openstack-infra | 18:27 | |
| *** russellb has joined #openstack-infra | 18:27 | |
| *** adreznec has joined #openstack-infra | 18:27 | |
| *** mfink- has joined #openstack-infra | 18:27 | |
| *** SotK has joined #openstack-infra | 18:27 | |
| *** tsufiev has joined #openstack-infra | 18:27 | |
| *** mrda has joined #openstack-infra | 18:27 | |
| *** ianw has joined #openstack-infra | 18:27 | |
| *** dmellado has joined #openstack-infra | 18:27 | |
| *** ddieterl_ has quit IRC | 18:27 | |
| *** tlbr has joined #openstack-infra | 18:27 | |
| *** radez_g0n3 is now known as radez | 18:27 | |
| *** slagle has quit IRC | 18:27 | |
| *** slagle has joined #openstack-infra | 18:27 | |
| *** emagana has joined #openstack-infra | 18:28 | |
| *** rcarrillocruz has joined #openstack-infra | 18:28 | |
| *** phschwartz has joined #openstack-infra | 18:28 | |
| *** yamamoto has joined #openstack-infra | 18:28 | |
| *** larsks|alt is now known as larsks | 18:28 | |
| mordred | devananda, SpamapS: isn't there a useful schema diff tool in maatkit? | 18:28 |
| *** Shrews has joined #openstack-infra | 18:28 | |
| zaro | jeblair: i don't think you can even run old version on current schea | 18:28 |
| *** tmcpeak has quit IRC | 18:29 | |
| *** annegentle has quit IRC | 18:29 | |
| *** dan| has joined #openstack-infra | 18:29 | |
| jeblair | zaro: mordred is looking into the differences and evaluating whether it's possible/advisable and working on constructing a downgrade plan | 18:29 |
| *** annegentle has joined #openstack-infra | 18:29 | |
| *** bkero has joined #openstack-infra | 18:29 | |
| mordred | I've got a list of the data transformations - I'm going to grab the db backup, and a current dump and see if percona-toolkit can make me a schema diff | 18:29 |
| jeblair | zaro, fungi, clarkb: what jgit version were we running with 2.8? | 18:30 |
| mordred | at least one of the migrations invovled cross-referencing with file information from git though ... so I'm not going to be 100% happy with doing this | 18:30 |
| * clarkb checks | 18:30 | |
| *** bkero has quit IRC | 18:30 | |
| *** bkero has joined #openstack-infra | 18:30 | |
| mordred | but - working on it | 18:30 |
| clarkb | jeblair: VERS = '3.2.0.201312181205-r' appears to be the version on 2.8 HEAD | 18:30 |
| zaro | yup, i see the same | 18:31 |
| fungi | i wonder if downgrading jgit with gerrit 2.10 would work | 18:31 |
| fungi | yeah | 18:31 |
| *** jodah- has joined #openstack-infra | 18:31 | |
| *** Krinkle is now known as Krinkle|detached | 18:32 | |
| *** jodah- has quit IRC | 18:32 | |
| zaro | i can try to do a build with older version to see if it even builds | 18:32 |
| *** Krinkle|detached is now known as Krinkle | 18:33 | |
| fungi | zaro: i think that would be a useful datapoint | 18:33 |
| *** ibiris_away is now known as ibiris | 18:33 | |
| zaro | ok trying now | 18:33 |
| *** yamamoto has quit IRC | 18:33 | |
| *** jodah has joined #openstack-infra | 18:33 | |
| *** ildikov has quit IRC | 18:34 | |
| *** e0ne has joined #openstack-infra | 18:35 | |
| *** Sukhdev has quit IRC | 18:35 | |
| fungi | #gerrit is a complete ghost town | 18:36 |
| *** ZZelle has joined #openstack-infra | 18:36 | |
| zaro | you'll get quicker response from mailing list. | 18:36 |
| jeblair | https://etherpad.openstack.org/p/ZIepsHViGk | 18:38 |
| jeblair | i started looking into where the zlib-related exception is coming from | 18:39 |
| jeblair | there is a difference in that function between jgit 3.2 and 3.7 | 18:39 |
| *** annegentle has quit IRC | 18:40 | |
| *** yamahata has quit IRC | 18:40 | |
| *** annegentle has joined #openstack-infra | 18:41 | |
| *** otter768 has joined #openstack-infra | 18:41 | |
| *** ibiris is now known as ibiris_away | 18:41 | |
| fungi | introduced in v3.5.0.201409071800-rc1 | 18:42 |
| fungi | from what i can see | 18:42 |
| jeblair | i just added a link to the EP with the commit that changed that function | 18:42 |
| clarkb | jeblair: big difference I see is old code checked that dstoff is less than length | 18:43 |
| clarkb | jeblair: current code seems to assume everything is aligned and will == length to end | 18:43 |
| fungi | yeah, that's the commit i was looking at. so definitely new in 3.5 | 18:43 |
| clarkb | but it only checks that if headerOnly is set | 18:43 |
| mordred | I just installed something from CPAN | 18:44 |
| mordred | I havent' done that in so long | 18:44 |
| jeblair | clarkb: have you gotten jgit-cli to barf yet? | 18:46 |
| *** otter768 has quit IRC | 18:46 | |
| SpamapS | mordred: maatkit is percona toolkit now | 18:46 |
| mordred | SpamapS: yah. and best I can tell there is no schema diff in the version that's in ubuntu at least | 18:47 |
| clarkb | jeblair: no, sorry haven't built a local copy of the exact version will try that now (I got completely confused by BUCK) | 18:47 |
| SpamapS | mordred: yeah I don't remember one. | 18:47 |
| mordred | jeblair: note for next time - when we do the backup - we should make 2 backups - one that's the backup, and one that's a schema-only backup | 18:47 |
| mordred | jeblair: as I currently have no choice but to restore all the data on my laptop as well | 18:47 |
| jeblair | mordred: ++ | 18:47 |
| SpamapS | grep -v ^INSERT ? | 18:48 |
| mordred | SpamapS: not a bad idea! | 18:48 |
| radez | anyone know if there is openid connect support on openstackid.org? | 18:48 |
| * mordred gives SpamapS a jellybean | 18:48 | |
| SpamapS | mordred: unfortunately backups were done with systemd so they're binary and you're screwed </troll> | 18:48 |
| *** ddieterly has joined #openstack-infra | 18:48 | |
| fungi | radez: smarcet or mrmartin may know | 18:49 |
| clarkb | oh jgit uses maven | 18:49 |
| radez | thx fungi | 18:49 |
| fungi | radez: i think the answer is "no" but i'll defer to them | 18:49 |
| *** dan| has quit IRC | 18:49 | |
| radez | smarcet: mrmartin: does openstackid.org support openid connect? | 18:49 |
| fungi | (or rather, "not yet") | 18:49 |
| *** tmcpeak has joined #openstack-infra | 18:50 | |
| clarkb | jeblair: I have a jgit build going now | 18:50 |
| radez | I've got Kilo happy on the new TryStack gear and wanted to integrate openstackid.org instead of facebook for the auth | 18:50 |
| *** BharatK has quit IRC | 18:50 | |
| *** baoli has quit IRC | 18:51 | |
| jeblair | clarkb: i'm curious if you cat jgit-show 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118 | 18:51 |
| jeblair | clarkb: s/cat/can/ | 18:52 |
| clarkb | for those following along gerrit apparently uses buck to build jgit but jgit itself uses maven | 18:52 |
| *** harlowja has quit IRC | 18:52 | |
| *** harlowja has joined #openstack-infra | 18:52 | |
| mordred | oh wow | 18:52 |
| mordred | of course it does | 18:52 |
| jeblair | clarkb: what version of jgit-cli did you install earlier? | 18:53 |
| mrmartin | radez: hi, as I know, not yet | 18:53 |
| *** Krinkle is now known as Krinkle|detached | 18:53 | |
| radez | mrmartin: kk, thx | 18:53 |
| mrmartin | radez: just plane openid and oauth | 18:53 |
| clarkb | jeblair: the version shipped by trusty | 18:53 |
| mordred | http://paste.openstack.org/show/220088 | 18:53 |
| clarkb | jeblair: 3.3.1.201403241930-r | 18:53 |
| *** soren_ has joined #openstack-infra | 18:53 | |
| mordred | ok - that's what mysqldiff says is the difference between the old and new dbs | 18:54 |
| mordred | that does NOT include the data transformations | 18:54 |
| radez | mrmartin: and only oauth2 correct? not v1? | 18:54 |
| jeblair | clarkb: is there a jgit show command? | 18:54 |
| mrmartin | oauth2 | 18:54 |
| *** dan_ has joined #openstack-infra | 18:54 | |
| jeblair | clarkb: if so, can you verify that 'jgit show 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118' works? (also, are you doing this on an exact replica of the repo on review?) | 18:55 |
| *** dan_ is now known as Guest25580 | 18:55 | |
| zaro | clarkb: gerrit builds jgit? | 18:55 |
| jeblair | clarkb: i would expect it to work since that version of jgit is pretty close to what we ran with 2.8; i will be very happy if you are able to build a jgit >= 3.7 and 'jgit show 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118' fails | 18:55 |
| mrmartin | fungi, all of the askbot-staging patches available now in the gerrit queue: https://review.openstack.org/181894 https://review.openstack.org/181873 https://review.openstack.org/172880 | 18:55 |
| zaro | clarkb: doesn't only reference it as a dependency with buck? | 18:56 |
| clarkb | jeblair: I was using a git clone from git.o.o | 18:56 |
| clarkb | jeblair: I can try on a copied replica too | 18:56 |
| zaro | i'm having a hard time building gerrit with older ver 3.2.0.201312181205-r | 18:57 |
| jeblair | clarkb: yeah, i think an exact replica would be best | 18:57 |
| *** moshele has left #openstack-infra | 18:57 | |
| mrmartin | basically we did a guerrilla staging site setup, and the deployment works well there, it is consuming askbot directly from github repo and we can fine-tune the branches | 18:57 |
| clarkb | jeblair: http://git.openstack.org/cgit/openstack/nova/commit/?id=4ad6d89dc9ea435eef448a15a301c9ae4e7e3118 that ref doesn't seem to exist, is that an object for a ref? | 18:57 |
| mrmartin | but the ideal way would be to move it entirely under infra and also upgrade the ask.o.o to match the new branching model | 18:57 |
| jeblair | clarkb: i think it's a blob | 18:58 |
| clarkb | jeblair: ok, git show only works against commits and refs right? probably need to figure out what the blob belongs to and go from there | 18:59 |
| jeblair | clarkb: git show works on blobs, when i do that with plain git on an rsync of the nova repo, i get a file | 18:59 |
| jeblair | clarkb: something about ec2 :) | 18:59 |
| clarkb | ah ok | 18:59 |
| fungi | mrmartin: excellent news. i'll try to have a look once we have our gerrit upgrade issues settled | 19:00 |
| mrmartin | great, thank you | 19:00 |
| clarkb | jeblair: ~corvus/nova.git is your copy? | 19:01 |
| *** kgiusti has joined #openstack-infra | 19:02 | |
| jeblair | clarkb: yep | 19:03 |
| jeblair | git cat-file -t 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118 | 19:03 |
| jeblair | blob | 19:03 |
| *** Rockyg has joined #openstack-infra | 19:03 | |
| *** jamesmcarthur has quit IRC | 19:03 | |
| *** tmcpeak has left #openstack-infra | 19:03 | |
| jeblair | (so yes -- confirmed it's a blob) | 19:03 |
| *** yamahata has joined #openstack-infra | 19:04 | |
| *** emagana has quit IRC | 19:05 | |
| *** emagana has joined #openstack-infra | 19:05 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Remove custom taskflow sqlalchemy environments https://review.openstack.org/181687 | 19:06 |
| clarkb | jeblair: `org.eclipse.jgit.pgm/target/jgit --git-dir ~/tmp/jgit-debug/nova.git show 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118` works with my local jgit version that was built using the version 2.10 is using | 19:06 |
| pabelanger | jeblair, are you fine with grafyaml living under openstack-infra like JJB does? | 19:07 |
| jeblair | clarkb: :( | 19:07 |
| jeblair | pabelanger: yep | 19:07 |
| pabelanger | jeblair, Roger | 19:08 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Remove extra unused environments from tooz jobs https://review.openstack.org/182021 | 19:08 |
| jeblair | clarkb: i think i want to restart gerrit again and see if we get the same error in the same file at the same offset | 19:08 |
| clarkb | jeblair: ok | 19:09 |
| openstackgerrit | Julia Kreger proposed openstack-infra/project-config: Add bifrost to OpenStack https://review.openstack.org/182022 | 19:09 |
| clarkb | jeblair: also, considering jgit show works it is possible this is a bug in how gerrit uses jgit | 19:09 |
| fungi | sort of wondering the same. if we can confirm this is persistent vs random we might be slightly closer to finding the conditions under which it emerges | 19:09 |
| jeblair | i copied the nova.git i have been using to: http://tarballs.openstack.org/ci/nova.git.tar.bz2 | 19:09 |
| mordred | http://paste.openstack.org/show/220089 <-- this is the latest - I stripped the autoinc changes out, and I added in or accounted for the data transformations | 19:10 |
| jeblair | so if anyone else wants to download that and try things out locally ^ | 19:10 |
| mordred | with 3 exceptions that are noted at the bottom, which I'm now working on | 19:10 |
| fungi | thanks--that looks quite concise | 19:10 |
| fungi | i was expecting something hairier | 19:11 |
| jeblair | added to etherpad | 19:11 |
| jeblair | restarting gerrit | 19:11 |
| clarkb | gerrit itself does not seem to use the WindowCursor or inflate method, so unlikely a gerrit api consumption issue | 19:11 |
| *** eharney has quit IRC | 19:12 | |
| *** dguerri is now known as _dguerri | 19:12 | |
| *** soren_ has quit IRC | 19:12 | |
| *** BharatK has joined #openstack-infra | 19:13 | |
| *** panda|afk has quit IRC | 19:13 | |
| jeblair | so now we just need to wait (probably 20 mins at most) for something to trigger it | 19:13 |
| clarkb | also if anyone else builds jgit you will need maven3 not maven2 | 19:13 |
| *** panda|afk has joined #openstack-infra | 19:14 | |
| jeblair | java.lang.StringIndexOutOfBoundsException: String index out of range: -17 | 19:15 |
| jeblair | those exceptions are curious. i am assuming they are not related. | 19:15 |
| mordred | I agree. -17 is out of range | 19:15 |
| clarkb | mordred: can you check the old db dump for SYSTEM groups? | 19:17 |
| fungi | i'm working on identifying what commit is responsible for the 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118 blob, to see if we can retrigger it faster | 19:17 |
| clarkb | mordred: I want to say we have just a couple and could add them back easily enough | 19:17 |
| pabelanger | jeblair, So, I assume I would open a review into openstack/governance first adding grafyaml under the openstack-infra namespace? | 19:17 |
| mordred | clarkb: yes - that's what I'm doing now | 19:18 |
| clarkb | mordred: registered users and administraors? | 19:18 |
| *** smccully has quit IRC | 19:18 | |
| fungi | pabelanger: yep, and then mention it in the project-config change | 19:18 |
| jeblair | pabelanger: you can create the repo in project-config first and have the governance change depend on it | 19:18 |
| mordred | clarkb: (I did SpamapS's trick to only load schema before - loading data now so that I can poke at those data changes) | 19:18 |
| pabelanger | jeblair, fungi: okay | 19:18 |
| jeblair | pabelanger, fungi: we reversed that for project additions to existing programs | 19:18 |
| AJaeger | pabelanger: all explained in http://docs.openstack.org/infra/manual/creators.html | 19:19 |
| pabelanger | AJaeger, thanks, that's what I was looking for | 19:19 |
| fungi | jeblair: yep, so no need to use a depends on in the project-config change, but having a pointer to the governance change is still useful | 19:19 |
| AJaeger | pabelanger: jeblair's comment is in http://docs.openstack.org/infra/manual/creators.html#add-new-repository-to-the-governance-repository | 19:19 |
| *** freerunner has joined #openstack-infra | 19:19 | |
| *** hamzy_ is now known as hamzy | 19:22 | |
| *** e0ne has quit IRC | 19:22 | |
| clarkb | I am going to grab lunch while we wait on post restart results | 19:22 |
| *** btully has joined #openstack-infra | 19:25 | |
| *** spzala has joined #openstack-infra | 19:26 | |
| *** BharatK has quit IRC | 19:26 | |
| *** jtomasek has quit IRC | 19:27 | |
| fungi | fwiw, https://review.openstack.org/151677 which was blank before the restart has content again | 19:27 |
| *** BharatK has joined #openstack-infra | 19:27 | |
| *** rlandy_ has joined #openstack-infra | 19:28 | |
| *** btully has quit IRC | 19:29 | |
| *** rlandy has quit IRC | 19:30 | |
| mordred | ok. I updated the migration script | 19:30 |
| mordred | I believe it's complete now | 19:30 |
| mordred | and not particularly scary | 19:30 |
| mordred | I believe we'll also need to manually delete the All-Users.git repo | 19:30 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Templatize zuul: Use python*-jobs https://review.openstack.org/182034 | 19:32 |
| jeblair | mordred: just delete it from disk? | 19:32 |
| mordred | yeah. I don't believe it has entries anywhere else | 19:32 |
| jeblair | mordred: or are there any database tendrils... ok | 19:32 |
| mordred | jeblair: now - obviously running that script without proper testing is mildly terrifying | 19:33 |
| mordred | so I'm still hoping that you find jgit solutions that make it not needed | 19:34 |
| fungi | well, running that script shouldn't be entirely terrifying. we'll backup the database before we try it | 19:35 |
| fungi | though testing separately first would still be good | 19:35 |
| *** koolhead17 has joined #openstack-infra | 19:35 | |
| *** koolhead17 has quit IRC | 19:35 | |
| *** koolhead17 has joined #openstack-infra | 19:35 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Stop leaking server objects https://review.openstack.org/181513 | 19:38 |
| *** _dguerri is now known as dguerri | 19:38 | |
| *** j^2 has joined #openstack-infra | 19:42 | |
| fungi | aha! my recursive digging has turned up "b286872 Allow volume filtering" as the commit responsible for that blob | 19:43 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Single event email worker. https://review.openstack.org/179542 | 19:43 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Factory subject is now a template file. https://review.openstack.org/181763 | 19:43 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Revert "Email Send Cron Worker" https://review.openstack.org/182042 | 19:43 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Revert "Email Outbox" https://review.openstack.org/182043 | 19:43 |
| j^2 | what is the process to open up #openstack-meeting-5 ? | 19:43 |
| fungi | j^2: we'd need to register it. we've had some backpressure on people sticking to somewhat compatible meeting times by limiting the number of channels, but as many meetings as we've added it may be time to consider a 5th channel for them now | 19:45 |
| j^2 | fungi: yeah, i’m trying to get the openstack-chef meeting at 1500UTC and it seems that all the meetings are full on Mondays:( | 19:46 |
| *** hashar has joined #openstack-infra | 19:46 | |
| j^2 | there is no reason why we could have N nummber of meeting rooms right? | 19:47 |
| fungi | j^2: other than then everyone would schedule their meetings in the same timeslot and it would become a lot harder to participate in multiple meetings because of increased conflicts | 19:47 |
| j^2 | ahh makes sense | 19:47 |
| fungi | and also it increases the number of channels we need to lurk in should something come up, though that's a lesser issue | 19:48 |
| j^2 | ohh yeah that is a challange | 19:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add grafyaml into openstack-infra https://review.openstack.org/182045 | 19:49 |
| fungi | j^2: if there aren't any great compromise options open for a new meeting time, i'd recommend starting a thread on the openstack-dev ml to gauge interest in/concern over growing the count of meeting channels again | 19:49 |
| fungi | since it's a bit of a balancing act, but we do acknowledge that we need to increase them from time to time | 19:49 |
| *** oomichi has joined #openstack-infra | 19:49 | |
| j^2 | fungi: the challange is that getting sacntioned we are requeride to use the meeting rooms, which we havent been doing, but we have a community that is used to meeting at a specific time | 19:50 |
| j^2 | if that makes sense | 19:50 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: NPM-Run jobs are now voting. https://review.openstack.org/182049 | 19:50 |
| *** oomichi has quit IRC | 19:50 | |
| fungi | the infra team is not the (sole) decision maker over that, so it's useful to get some additional input as it affects the whole community to a degree | 19:51 |
| j^2 | hmm ok | 19:51 |
| *** mrmartin has quit IRC | 19:52 | |
| mordred | I just did a scan - I will say that the overlap betweenthe current 1500UTC Monday meetings and chef is pretty low | 19:52 |
| mordred | in terms of exclusion of people who'd want to be in both | 19:52 |
| clarkb | I wonder, what if we are looking at the wrong level. Could it be that zlib has the bug? | 19:52 |
| jeblair | [2015-05-11 19:52:42,639] WARN com.google.gerrit.server.git.SubmoduleOp : Cannot update gitlinks for openstack/openstack,refs/heads/stable/grizzly:python-openstackclient follows openstack/python-openstackclient,refs/heads/master due to The branch was probably deleted from the subscriber repository | 19:53 |
| jeblair | [2015-05-11 19:52:42,642] INFO com.google.gerrit.server.git.SubmoduleOp : Deleted incorrect submodule subscription(s) [openstack/openstack,refs/heads/stable/grizzly:python-openstackclient follows openstack/python-openstackclient,refs/heads/master] | 19:53 |
| *** koolhead17 has quit IRC | 19:53 | |
| mordred | but - let me tell you something I don't want to get into the business of | 19:53 |
| krotscheck | clarkb: Did vinz ever get backed? | 19:53 |
| j^2 | mordred: yeah it’s a handful of meetings, all the offical channels are taken up :) | 19:53 |
| jeblair | mordred: ^ maybe file that for later | 19:53 |
| mordred | jeblair: oh my | 19:53 |
| fungi | okay, so in theory trying to access commit b286872 (Change-Id: Icd915f30be8120a9c3fde602be66b442e4460709) should cause gerrit to rerequest our previously failing blob 4ad6d89dc9ea435eef448a15a301c9ae4e7e3118 | 19:53 |
| fungi | should i try it? | 19:53 |
| jeblair | fungi: go for it | 19:53 |
| *** tvander has joined #openstack-infra | 19:54 | |
| clarkb | hrm that comes from java.util.zip so likely not at issue and I should have the same version as review.o.o | 19:54 |
| clarkb | krotscheck: no :/ | 19:54 |
| *** Krinkle|detached is now known as Krinkle | 19:54 | |
| krotscheck | clarkb: sadpanda | 19:54 |
| fungi | no dice https://review.openstack.org/62350 | 19:54 |
| mordred | clarkb: as in - they chose to not back it? or they have not chosen? | 19:54 |
| crinkle | is the infra meeting agenda the authoritative list for infra's priority efforts? i found https://etherpad.openstack.org/p/infrastructure-priorities but it seems a bit out of date | 19:54 |
| jeblair | fungi: thx | 19:54 |
| jeblair | clarkb: yes, the agenda is the current auth list | 19:55 |
| jeblair | crinkle: ^ sorry | 19:55 |
| crinkle | jeblair: thank you | 19:55 |
| *** tvander has left #openstack-infra | 19:55 | |
| clarkb | mordred: we did not get selected | 19:55 |
| clarkb | mordred: I think even college seniors are smart enough to avoid the javascript :) | 19:55 |
| jeblair | :( | 19:56 |
| fungi | crinkle: we sort of semi-intentionally allow it to dwindle as the summit approaches, so that there's room to fill it back up with the new priorities we settle on | 19:56 |
| mordred | :( | 19:56 |
| greghaynes | yeaa :/ | 19:56 |
| clarkb | fungi: so I take it that means gerrit hasn't fallen over on nova yet? | 19:57 |
| fungi | krotscheck: in good news, we have an outreachy intern starting soon. you could try to convince them that's an interesting project ;) | 19:57 |
| *** freerunner has quit IRC | 19:57 | |
| *** pabelanger has quit IRC | 19:57 | |
| zaro | i think i might have figured out how to build with older version of jgit if we are still interested in that. | 19:57 |
| clarkb | zaro: perhaps put details on https://etherpad.openstack.org/p/ZIepsHViGk ? | 19:58 |
| fungi | clarkb: seems not. so viewing the change,patchset responsible for that blob, including looking at diffs of the files changed by it, doesn't seem to have retriggered the failure we saw earlier | 19:58 |
| fungi | clarkb: suggesting that it' | 19:58 |
| fungi | s not a problem with that particular packfile i think? | 19:58 |
| *** jamesmcarthur has joined #openstack-infra | 20:01 | |
| jeblair | fungi: it seems that way; so maybe something about gerrit's internal state causes that method to fail... | 20:01 |
| *** spredzy|afk is now known as spredzy | 20:02 | |
| *** Krinkle is now known as Krinkle|detached | 20:03 | |
| *** dims has joined #openstack-infra | 20:03 | |
| *** pabelanger has joined #openstack-infra | 20:04 | |
| *** pabelanger has joined #openstack-infra | 20:04 | |
| *** sabeen3 has quit IRC | 20:05 | |
| *** tiswanso_ has quit IRC | 20:06 | |
| krotscheck | clarkb: I will shiv you | 20:06 |
| *** mwagner_lap has quit IRC | 20:07 | |
| openstackgerrit | Sergey Lukjanov proposed openstack-infra/project-config: Add non-voting coverage job to sahara check pipeline https://review.openstack.org/182059 | 20:07 |
| openstackgerrit | Khai Do proposed openstack-infra/gerrit: Use jgit ver 3.2 https://review.openstack.org/182060 | 20:08 |
| krotscheck | We should probably just kill the project then, so that it doesn't suck up resources. | 20:08 |
| jeblair | [2015-05-11 20:08:00,161] WARN com.google.gerrit.server.plugins.PluginLoader : Plugin </home/gerrit2/review_site/plugins/javamelody.jar> was disabled, because another plugin </home/gerrit2/review_site/plugins/javamelody-4744bfb.jar> with the same name <javamelody> already exists | 20:08 |
| jeblair | what ^? | 20:08 |
| jeblair | zaro: maybe a puppet bug there ^. no idea why it emitted that _now_. | 20:09 |
| clarkb | krotscheck: I don't think its sucking up any resources currently beyond what they will suck up if moved to the attic | 20:10 |
| *** sdake_ has joined #openstack-infra | 20:10 | |
| krotscheck | clarkb: Well, it's currently in my way because it's using js-build templates that I want to get rid of. | 20:10 |
| krotscheck | And, well, the repo is empty. | 20:10 |
| zaro | ohh odd, i would expected to see that during the upgrade as well. i'll take a look. | 20:10 |
| jeblair | krotscheck: just remove the jobs | 20:10 |
| zaro | but quick fix is to just remove older one from plugins folder | 20:10 |
| krotscheck | jeblair: That's what I'm doing right now | 20:11 |
| *** doude has quit IRC | 20:11 | |
| jeblair | okay heads up, i'm seeing missing commits | 20:11 |
| *** soren_ has joined #openstack-infra | 20:11 | |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Add appdirs requirement https://review.openstack.org/182061 | 20:11 |
| *** doude has joined #openstack-infra | 20:11 | |
| *** _nadya_ has joined #openstack-infra | 20:11 | |
| mordred | jeblair: ooh! | 20:11 |
| sdague | so.... I'm going to assume this is all part of the jgit bug, but my search dashboards are returning a dramatically small number of hits (mostly missing stuff from April 28th forward) | 20:12 |
| zaro | just built gerrit with older jgit, pushed a change, updated etherpad, and now testing it out. | 20:12 |
| jeblair | [2015-05-11 20:10:48,957] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /home/gerrit2/review_site/git/openstack/neutron.git/object | 20:12 |
| jeblair | s/pack/pack-5d1fdaf97301469c0d4cd0233b3dee590fed72bc.pack is corrupt, removing it from pack list | 20:12 |
| jeblair | org.eclipse.jgit.errors.CorruptObjectException: Object at 88,537,491 in /home/gerrit2/review_site/git/openstack/neutron.git/objects/pack/pack-5d1fdaf97301469 | 20:12 |
| jeblair | c0d4cd0233b3dee590fed72bc.pack has bad zlib stream | 20:12 |
| sdague | in case that is a useful piece of information | 20:12 |
| clarkb | interesting thing I notice about the jgit code, is it keeps a long start position and long current position in its windows but then when it takes the delta between them casts to an int | 20:13 |
| jeblair | sdague: thx, we're seeing a lot of failures; i would actually expect your dashboard to completely fail. it might be related or there might be something else going on. | 20:13 |
| clarkb | I suppose if the pack file is gigantic we could be running into 2^31 rollovers in java ints | 20:13 |
| sdague | jeblair: it mostly works, just a lot of stuff missing when I try to filter by label: | 20:13 |
| jeblair | clarkb: when it outputs that error, is that the long or the int? | 20:13 |
| *** sdake has quit IRC | 20:13 | |
| sdague | that seems to be the trigger for the filters dumping a ton of stuff into the void | 20:14 |
| jeblair | so the latest error is in a different repo, but has a similarly high index | 20:15 |
| clarkb | jeblair: when it does position += window.setInput(position, inf); The return value of setInput is an int | 20:16 |
| *** _nadya_ has quit IRC | 20:16 | |
| clarkb | if it rollsover we could in theroy be adding a negative value, this moves position backwards and the inflator may return n = 0 at that point | 20:16 |
| * clarkb is reading docs on the inflator | 20:16 | |
| fungi | so far we've seen this hit nova and neutron which are "really big" (commit, change and history wise) repos so that seems like a possible correlation | 20:17 |
| clarkb | oh and inflate can throw its own DataFormatExceptions directly | 20:18 |
| jeblair | clarkb: the length in bytes of the nova pack file is much smaller than 2^31 -- 325720228 | 20:19 |
| *** ildikov has joined #openstack-infra | 20:19 | |
| jeblair | clarkb: i assume you mean Inflater.inflate can throw those, yeah? | 20:21 |
| clarkb | jeblair: ya it can throw, but rereading the stacktrace I don't think it did | 20:21 |
| jeblair | clarkb: agreed, it was WindowCursor.inflate that threw | 20:21 |
| zaro | looks like the build with older jgit seems to generally work, i can upload a new patch and merge it thru. was wondering if anybody wanted this on review-dev for further eval? | 20:25 |
| *** teran has joined #openstack-infra | 20:28 | |
| jeblair | the neutron error was also on its largest pack file | 20:29 |
| jeblair | (the nova error was on its largest pack file) | 20:30 |
| clarkb | jeblair my java is bad but pretty sure the issue is in the headerOnly check, need to see if that is not set in places | 20:30 |
| clarkb | jeblair: we read dstbuf.length - dstoff bytes each time then update dstoff | 20:30 |
| clarkb | jeblair: eventually that will go to zero, at which point inflating should return zero bytes except if headerOnly is false that means we can be unfinished and not need more input | 20:31 |
| clarkb | not sure what a good way to test/confirm that is other than maybe dumping the values of dstbuf.length and dstoff somehow (debugger, logger) | 20:31 |
| jeblair | clarkb: yeah, or possibly constructing an artificial test case if we think we know what kind of input may trigger it | 20:32 |
| clarkb | jeblair: note that the old code properly ended when that buffer was full without raising | 20:32 |
| *** pcrews has quit IRC | 20:33 | |
| *** mtanino has quit IRC | 20:34 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Add pep8 job to nova-specs https://review.openstack.org/181994 | 20:34 |
| mikal | Am I correct in assuming that we don't bump the versions of dependancies in stable releases? | 20:34 |
| * krotscheck just found stackforge/merlin | 20:34 | |
| krotscheck | Zee Javazcreeept, eet ees eeeverywhere! | 20:35 |
| clarkb | jeblair: if (curs.inflate(this, position, dstbuf, false) != sz) is the call whee we raise from | 20:35 |
| clarkb | jeblair: so its not setting true there | 20:35 |
| mordred | mikal: we try our best not to - I imagine we would if we needed to for a security update | 20:35 |
| clarkb | I am not quite sure what spearce added the headerOnly flag for, trying to grok that now | 20:36 |
| *** radez is now known as radez_g0n3 | 20:36 | |
| clarkb | I mean, if you pass in a buffer and don't grow the buffer for the caller then ou can only ever read in that many bytes... | 20:36 |
| jeblair | clarkb: he said that the buffer needs to be able to support the full decompressed data size | 20:37 |
| jlvillal | Does anyone know if in project-config/zuul/layout.yaml if I can use a template with a branch restriction? | 20:37 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: NPM-Run jobs are now voting. https://review.openstack.org/182049 | 20:37 |
| jlvillal | It was suggested I change my patch to use a template, but I also have a branch restriction. | 20:38 |
| *** craigbr has joined #openstack-infra | 20:38 | |
| clarkb | jeblair: that seems like a really bad way to fail when you don't then... | 20:38 |
| AJaeger | jlvillal: which change? | 20:38 |
| jlvillal | AJaeger: https://review.openstack.org/#/c/181034/ | 20:38 |
| clarkb | "Something wrong with your zip file except not really you actually need a bigger buffer" | 20:38 |
| jlvillal | AJaeger: You were the suggester :) | 20:38 |
| AJaeger | jlvillal: A template just expands, so no restrictions | 20:39 |
| *** ams__ has joined #openstack-infra | 20:39 | |
| jeblair | clarkb: so if you assume that the conditions in the javadoc hold (the supplied buffer is large enough), then i don't think there's a logic error, as inf.finished() should return true | 20:39 |
| clarkb | jeblair: correct | 20:39 |
| jlvillal | AJaeger: I don't understand if that means I can use it with a branch restriction??? | 20:39 |
| mikal | mordred: ahhh, ok cool. That's kind of where I was going. | 20:40 |
| *** baoli has joined #openstack-infra | 20:40 | |
| jeblair | clarkb: i agree that if the supplied buffer is not large enough, we'll see the error we got (and that could be because of a programming error or because the stream actually is corrupt and is spewing data) | 20:40 |
| jlvillal | AJaeger: Or should I update my patch to put the check in both the gate and check steps? | 20:40 |
| clarkb | jeblair: there is a lot of masking to determine the value of the buffer size | 20:41 |
| fungi | jeblair: clarkb: is that buffer filled with compressed or decompressed data? | 20:41 |
| AJaeger | jlvillal: You should run the voting python34 jobs in both check and gate. And instead of adding it manually in the file, use the template. | 20:41 |
| clarkb | fungi: decompressed (its receiving the inflated data) | 20:41 |
| *** rmcall has quit IRC | 20:41 | |
| jlvillal | AJaeger: What about the branch restriction? | 20:41 |
| AJaeger | The branch restriction you have will work on the template the same way as it works on the jobs. Just assume that we do template expansion first and then the branch restrictions. | 20:42 |
| jlvillal | AJaeger: Thanks! | 20:42 |
| *** ildikov has quit IRC | 20:42 | |
| fungi | clarkb: okay, that's what it seemed like. so we need to compare the raw sizes i guess, not the current on-disk size | 20:42 |
| AJaeger | The templates are just macros - syntactic sugar - they don't limit you | 20:42 |
| *** otter768 has joined #openstack-infra | 20:42 | |
| jeblair | we also have erorrs in ironic-specs, keystonemiddleware, murano-agent and os-collect-config now | 20:43 |
| jeblair | looking to see if they follow the pattern | 20:43 |
| *** openstackgerrit_ has joined #openstack-infra | 20:43 | |
| *** baoli has quit IRC | 20:44 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config: Have gate-ironic-python34 be a voting job https://review.openstack.org/181034 | 20:44 |
| *** tonytan4ever has quit IRC | 20:44 | |
| jlvillal | AJaeger: Done I think :) ^^^^ | 20:44 |
| openstackgerrit | Khai Do proposed openstack-infra/gerrit: Use jgit ver 3.2 https://review.openstack.org/182060 | 20:44 |
| jeblair | the last two have tiny pack files, which seems to suggest pack file size is not related | 20:44 |
| *** baoli has joined #openstack-infra | 20:44 | |
| AJaeger | jlvillal: that won't work. Leave the branch restriction where it was | 20:45 |
| AJaeger | jlvillal: and remove gate-ironic-python3 from the check queue | 20:45 |
| jlvillal | AJaeger: Okay. working.... | 20:45 |
| jeblair | also, ironic-specs and keystonemiddleware had 2 errors apiece at different offsets but with the same log timestamp | 20:46 |
| jeblair | possibly different threads? | 20:46 |
| *** craigbr has quit IRC | 20:46 | |
| clarkb | fungi: yes, and it appears to be doing it per deleta | 20:46 |
| jeblair | in all cases, it's still the largest pack file (also meaning it's probably the most used) | 20:46 |
| clarkb | but its completely undocumented so hard to follow | 20:46 |
| *** otter768 has quit IRC | 20:47 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config: Have gate-ironic-python34 be a voting job https://review.openstack.org/181034 | 20:47 |
| *** achanda has joined #openstack-infra | 20:48 | |
| jlvillal | AJaeger: The layout.yaml file is starting to make sense to me. Thanks for your help. | 20:48 |
| openstackgerrit | Sean Roberts proposed openstack-infra/project-config: update akanda acls to support branching https://review.openstack.org/182079 | 20:48 |
| AJaeger | jlvillal: LGTM, +2 | 20:49 |
| clarkb | org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:813) is where the size calculation happens | 20:49 |
| jogo | anywhere I can file UI complaints for the new gerrit? | 20:49 |
| clarkb | jogo: upstream against gerrit | 20:49 |
| *** tonytan4ever has joined #openstack-infra | 20:49 | |
| * jogo checks config first to see if he can turn off the ugly | 20:49 | |
| jogo | listing conflicts with by default is distracting to me | 20:49 |
| jlvillal | AJaeger: Thanks | 20:49 |
| fungi | jogo: example? | 20:50 |
| clarkb | fungi: thats new change screen iirc | 20:50 |
| fungi | jogo: is that just in the new change screen, or old change screen but new behavior in 2.10? | 20:50 |
| fungi | ahh, the former. got it | 20:50 |
| clarkb | fungi: it tells you what other changes you conflict with | 20:50 |
| fungi | interesting | 20:50 |
| ZZelle | Hi everyone | 20:51 |
| ZZelle | i get the following error in gerrit: | 20:51 |
| ZZelle | org.eclipse.jgit.errors.MissingObjectException: Missing unknown 8cbfa31e60d5d202da4fa89f942fb17704f52fea | 20:51 |
| clarkb | jeblair: but basically the only way I can make it raise that error while not being finished or needing more input is if the buffer provided is not large enough | 20:51 |
| ZZelle | with https://review.openstack.org/#/c/179955/4/neutron/ipam/subnet_alloc.py | 20:51 |
| jogo | fungi: only checked the new screen | 20:51 |
| jogo | fungi: https://review.openstack.org/#/c/163798/ | 20:51 |
| *** Sukhdev has joined #openstack-infra | 20:52 | |
| jeblair | jogo: can it wait until we have dealt with the git problems? | 20:52 |
| jeblair | ZZelle: yep, that's the error we're working on (referenced in channel topic) | 20:52 |
| ZZelle | jeblair, neutron is also concerned | 20:53 |
| jogo | jeblair: very much so yes, that is why I was hoping there was some place I could put a comment | 20:53 |
| *** claudiub has joined #openstack-infra | 20:53 | |
| jeblair | jogo: upstream gerrit issue tracker? | 20:53 |
| clarkb | I hvae a doctors appointment I need to go to now, back after | 20:54 |
| *** carl_baldwin has joined #openstack-infra | 20:55 | |
| jogo | jeblair: doing that now | 20:55 |
| *** tnovacik has quit IRC | 20:55 | |
| *** claudiub|2 has quit IRC | 20:55 | |
| *** sks has joined #openstack-infra | 20:56 | |
| sdague | jogo: honestly, I really like having conflicts with. That's going to help a lot in pruning duplicate fixes in the nova backlog. | 20:56 |
| fungi | so, i'm seeing that gerrit has a core.packedGitWindowSize config option "Default on JGit is 8 KiB on all platforms." | 20:56 |
| fungi | should we be considering tuning that? | 20:56 |
| *** ddieterly has quit IRC | 20:56 | |
| sdague | it's also good to see that every vmware patch is a 20 way conflict with all the other ones. | 20:57 |
| zaro | jogo: what do you want to see differently? turned off? | 20:57 |
| jeblair | zaro: can we try to keep the discussion related to the current problem please | 20:57 |
| openstackgerrit | Ruslan Kamaldinov proposed openstack-infra/project-config: Add functional test job for python-muranoclient https://review.openstack.org/182082 | 20:57 |
| *** ildikov has joined #openstack-infra | 20:58 | |
| jeblair | i will set up #openstack-infra-incident for the next time this happens, but it is not logged now, and i think it's important that this be logged | 20:58 |
| fungi | also core.packedGitLimit, core.deltaBaseCacheLimit, core.packedGitOpenFiles, core.streamFileThreshold, core.packedGitMmap... | 20:58 |
| jogo | zaro: https://code.google.com/p/gerrit/issues/detail?id=3362&thanks=3362&ts=1431377929 | 20:58 |
| fungi | wondering if maybe android's gerrit has those tuned to larger values and so isn't running into this | 20:59 |
| jeblair | fungi: if i understand the windowcursor correctly, i think the windowsize just means the size of reads that it performs within the inflate loop | 20:59 |
| jogo | zaro: yeah or at least minimized to just a number that you can click on to expand | 20:59 |
| fungi | ahh, okay | 20:59 |
| *** moshele has joined #openstack-infra | 21:00 | |
| *** ociuhandu has quit IRC | 21:00 | |
| jogo | sdague: yeah it should help a lot for duplicates, I just don't know if I want to see a massive list every time | 21:00 |
| * jogo wishes gerrit had a better UX | 21:00 | |
| jeblair | sdague, jogo, zaro: please take that discussion to another channel | 21:00 |
| jeblair | it's seriously distracting | 21:00 |
| jogo | jeblair: sorry | 21:00 |
| *** marcusvrn has quit IRC | 21:01 | |
| anteaya | jeblair: I'm sort of out of the loop but trying to catch up, tell me if there is anything I can do to help | 21:01 |
| jeblair | fungi: i don't understand any of the other constants you mentioned; do they look likely to be related? | 21:02 |
| *** mwagner_lap has joined #openstack-infra | 21:02 | |
| jeblair | fungi: i think the fact that we haven't seen this in exactly the same place twice suggests that there's some transient operational aspect involved and maybe not a simple "constant not big enough" error | 21:03 |
| openstackgerrit | Merged openstack-infra/project-config: Adds pypi-jobs to cloudbase-init https://review.openstack.org/181883 | 21:03 |
| *** eharney has joined #openstack-infra | 21:03 | |
| *** sabeen1 has joined #openstack-infra | 21:04 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added NPM-based unit tests for merlin https://review.openstack.org/182084 | 21:04 |
| *** sdake has joined #openstack-infra | 21:04 | |
| openstackgerrit | Sean Roberts proposed openstack-infra/project-config: update akanda acls to support branching https://review.openstack.org/182079 | 21:05 |
| jeblair | fungi, mordred, zaro: when should we make a decision to revert or not? | 21:05 |
| fungi | jeblair: agreed, since the impact is seemingly random i don't suspect this is intended/expected behavior from using default tuning values. just wondering if of the large gerrit implementations out there running 2.10+ others haven't hit this issue because they have some of these tuned to non-default values | 21:05 |
| *** sabeen2 has joined #openstack-infra | 21:05 | |
| mordred | jeblair: I think we should do it with enough time that we have time to do it before west coast EOD | 21:05 |
| *** hichihara has joined #openstack-infra | 21:05 | |
| *** dustins has quit IRC | 21:05 | |
| mordred | otherwise we're going to be trying to do it completely burned | 21:05 |
| zaro | no interest using gerrit with older jgit? | 21:06 |
| *** shardy is now known as shardy_z | 21:06 | |
| openstackgerrit | Sean Roberts proposed openstack-infra/project-config: update akanda acls to support branching https://review.openstack.org/182079 | 21:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Register openstack-infra-incident with accessbot https://review.openstack.org/182086 | 21:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Log openstack-infra-incident https://review.openstack.org/182087 | 21:07 |
| fungi | if gerrit 2.10 with jgit 3.2 works (zaro's test build) it might be worth giving that a go to get us through until we can get some feedback from gerrit devs, but there's a good chance that they bumped the included jgit because newer gerrit depends on newer jgit features | 21:07 |
| openstackgerrit | Ruslan Kamaldinov proposed openstack-infra/project-config: Add functional test job for python-muranoclient https://review.openstack.org/182082 | 21:07 |
| jeblair | fungi, zaro: yeah, i'm worried there a design issue lurking in there, even if it appears to work... | 21:07 |
| *** sdake_ has quit IRC | 21:08 | |
| fungi | if i were more confident we might reliably reproduce this condition on review-dev i'd be inclined to start the rollback immediately | 21:08 |
| jeblair | fungi: i think to do that we will need to make -dev a full production staging environment | 21:09 |
| *** sabeen1 has quit IRC | 21:09 | |
| fungi | my worry is that even with the same data and configuration and some sort of fabricated load generation we'll be missing a key element to get it to happen there | 21:09 |
| jeblair | fungi: very possible | 21:09 |
| mordred | but - it does motivate making it a real staging env | 21:09 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Removed old js-based test job. https://review.openstack.org/182088 | 21:09 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Deprecated js-test build. https://review.openstack.org/182089 | 21:09 |
| *** moshele has quit IRC | 21:10 | |
| *** mtanino has joined #openstack-infra | 21:10 | |
| *** annegent_ has joined #openstack-infra | 21:10 | |
| *** tonytan4ever has quit IRC | 21:11 | |
| *** esker has quit IRC | 21:11 | |
| fungi | so, that said, i'm at a loss as to what else to try to get this back on track. unless anyone else has ideas, we should probably start the rollback | 21:13 |
| *** pcrews has joined #openstack-infra | 21:14 | |
| fungi | is there any additional data about the failure we should be considering collecting which would be destroyed when we downgrade? | 21:14 |
| * anteaya stands by to keep the channel clear | 21:14 | |
| *** annegentle has quit IRC | 21:15 | |
| *** baoli has quit IRC | 21:15 | |
| pleia2 | anteaya: yeah, -incident will be nice moving forward to avoid the interruptions :) | 21:15 |
| fungi | also, what's the jgit version in latest gerrit 2.9.x? (not that we should likely consider trying to downgrade to 2.9, nor immediately upgrade from 2.8 to 2.9 before we have more time to plan/test) | 21:15 |
| mordred | fungi, jeblair: I'm the more insane of us - but I'm tempted by the new-gerrit-old-jgit - purely because java is strongly typed ... and I can't imagine it being much worse | 21:15 |
| anteaya | pleia2: agreed | 21:15 |
| anteaya | pleia2: didn't want to side tracking making it now though | 21:15 |
| *** baoli has joined #openstack-infra | 21:15 | |
| anteaya | track | 21:15 |
| pleia2 | anteaya: need to wait until it's logged anyway | 21:15 |
| anteaya | yes | 21:16 |
| mordred | reviews welcome on those two patches... | 21:16 |
| jeblair | fungi, zaro: fwiw, review-dev has logged errors relating to packs being deleted (which we think is related to repack and we understand), but not zlib errors | 21:16 |
| pleia2 | they look fine, just waiting on jenkins | 21:16 |
| pleia2 | feeble human eye review | 21:16 |
| mordred | jeblair: I've lost track - which combo is review-dev running now? | 21:17 |
| *** rlandy_ has quit IRC | 21:17 | |
| jeblair | zaro: ? | 21:18 |
| zaro | combo? | 21:18 |
| zaro | i haven't changed anything. | 21:18 |
| fungi | zaro: so gerrit 2.10 and jgit 3.7 still, i think | 21:18 |
| zaro | yes | 21:18 |
| jeblair | mordred: but that was a few weeks ago, so i'm not actually sure what was running at the time | 21:18 |
| fungi | sort of wishing the person who reported https://groups.google.com/forum/#!topic/repo-discuss/fTx5piOjcJ4 had followed up in the past week | 21:18 |
| *** sks has quit IRC | 21:18 | |
| fungi | jkt: ^ you commented on that post as well | 21:19 |
| zaro | it was 2.10 & jgit 3.7 as well | 21:19 |
| zaro | 2.10 has been running on review-dev for a while | 21:19 |
| *** tonytan4ever has joined #openstack-infra | 21:19 | |
| *** mpopow has quit IRC | 21:20 | |
| mordred | so - maybe start 2.10 + 3.2 on review-dev real quick and make sure it doesn't just flat delete things | 21:21 |
| mordred | ? | 21:21 |
| *** mpopow has joined #openstack-infra | 21:21 | |
| mordred | and if it doesn't delete things, try running it on review instead of 3.7 and see if the error triggers, if not, rollback? | 21:21 |
| jeblair | i think i can live with that if that doesn't terrify anyone else | 21:22 |
| mordred | basically aLL of of the options terrify me | 21:22 |
| jeblair | fungi: thoughts? | 21:22 |
| fungi | i'll just make sure to have a teddy bear and a security blanket handy | 21:22 |
| fungi | but i'm in | 21:22 |
| fungi | i'm here all night | 21:22 |
| fungi | we have backups | 21:22 |
| fungi | wors that happens is people reupload changes from the past couple days | 21:23 |
| zaro | my build is on review-dev.o.o ready to go | 21:23 |
| jeblair | zaro: go for it | 21:23 |
| zaro | ok | 21:23 |
| jeblair | zaro: do you have a change ready for production? | 21:23 |
| smarcet | smarcet | 21:25 |
| *** smarcet has quit IRC | 21:25 | |
| zaro | https://review.openstack.org/#/c/182060/ | 21:25 |
| *** vhoward has left #openstack-infra | 21:26 | |
| *** bswartz has quit IRC | 21:27 | |
| *** rbradfor has quit IRC | 21:27 | |
| mordred | it's got 2 +2s now - +A will build and upload an artifact somewhere, yeah? | 21:27 |
| jeblair | mordred: i +Ad and yes | 21:28 |
| mordred | cool | 21:28 |
| fungi | lgtm | 21:28 |
| jeblair | how about i start on a message to the repo list? | 21:28 |
| *** craigbr has joined #openstack-infra | 21:29 | |
| mordred | kk | 21:29 |
| zaro | done. ver 2.10.2-23-g039a170-dirty | 21:30 |
| *** dkranz has quit IRC | 21:30 | |
| *** julim has quit IRC | 21:31 | |
| mordred | I confirm that that is the dirty version running there | 21:31 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Finish removing invocations of pip. https://review.openstack.org/181785 | 21:31 |
| mordred | I can browse thigns at least | 21:31 |
| *** peristeri has quit IRC | 21:32 | |
| anteaya | I can sign in and leave a comment | 21:33 |
| *** emagana has quit IRC | 21:34 | |
| *** dangers_away is now known as dangers | 21:34 | |
| jeblair | push up a new change; new patchset, etc? | 21:35 |
| *** emagana has joined #openstack-infra | 21:35 | |
| fungi | done and done | 21:35 |
| fungi | merged a couple too | 21:35 |
| *** waynr has quit IRC | 21:36 | |
| *** jamesmcarthur has quit IRC | 21:36 | |
| zaro | i also tested creating new projects on the UI as well. | 21:39 |
| openstackgerrit | Merged openstack-infra/project-config: networking-ovn: disable n-novnc https://review.openstack.org/181930 | 21:39 |
| *** Rockyg has quit IRC | 21:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Add create rule to networking-mlnx https://review.openstack.org/181862 | 21:39 |
| *** cdent has quit IRC | 21:41 | |
| adam_g | anyone know where to configure the list of channels that the 'openstack' lp bug bot hangs out in? | 21:41 |
| *** mattfarina has quit IRC | 21:42 | |
| fungi | adam_g: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/eavesdrop.pp#n39 | 21:42 |
| *** hashar has quit IRC | 21:42 | |
| adam_g | fungi, ah, cool | 21:43 |
| *** achanda has quit IRC | 21:43 | |
| adam_g | fungi, while ive got you: how does the gerrit bot resolve a gerrit patch's 'Closes-bug' to LP project? does it just go by $repo_name->$lp_project? wondeirng how i can get several sub-projects updating bugs in a single place, similar to neuron and the neutron-* sub-projects | 21:44 |
| *** baoli has quit IRC | 21:44 | |
| fungi | adam_g: it's configurable. come back when we're not trying to work through gerrit upgrade issues | 21:44 |
| greghaynes | fungi: Hypothesis maybe for why you couldnt re-trigger the pack corrupt error - when gerrit "removes from packlist" does gerrit keep the ObjectDirectory instance alive with the removed reference to that pack? | 21:44 |
| adam_g | fungi, sure, thanks | 21:45 |
| bhunter71 | a question; what's the basic process upstream follows when creating a new stable release branch? or, more specifically, are the branches created in gerrit first, and then replicated to github.com? | 21:45 |
| jeblair | greghaynes: i think when fungi was trying to retrigger it, it was after a restart? | 21:45 |
| greghaynes | ah, if so then there goes that theory | 21:45 |
| anteaya | bhunter71: our workflow is that gerrit is the canonical repo and git.o.o and github are mirrors | 21:46 |
| fungi | yeah, so far i'm not seeing any problems using review-dev with older jgit | 21:46 |
| bhunter71 | anteaya; thanks. Are there cases where you have to manually 'gerrit replicate', or is it smart enough to hook the branch creation event? | 21:47 |
| fungi | errors in its log seem to just be due to missing/misconfigured hooks, so nothing alarming so far | 21:48 |
| fungi | bhunter71: it happens automatically. same with tags | 21:48 |
| anteaya | bhunter71: I don't follow, our toolchain tests and merges to the canonical git repos on gerrit, once a merge has taken place in a repo the change is replicated to the mirrors | 21:48 |
| zaro | fungi: anything issues with replication? | 21:48 |
| anteaya | bhunter71: are we talking hypothetically or do you have a problem? | 21:49 |
| fungi | zaro: no replication errors that i can see either, no | 21:49 |
| jeblair | fungi, zaro, mordred: draft message in etherpad | 21:49 |
| *** funzo has quit IRC | 21:49 | |
| fungi | zaro: hrm... though changes that i merged aren't showing up in github https://github.com/gtest-org/test2/commits/master | 21:50 |
| *** sdake has quit IRC | 21:50 | |
| mordred | jeblair: lgtm | 21:51 |
| anteaya | have we decided to downgrade our jgit on review.o.o? | 21:51 |
| anteaya | I didn't see the draft email mention that | 21:51 |
| anteaya | did it and I missed it? | 21:51 |
| mordred | jeblair: maybe ... when you mention the pack file, you make it sound like it's related to one pack file - rather than that it can happen in any repo but when it does the file is the same? | 21:51 |
| *** baoli has joined #openstack-infra | 21:52 | |
| jeblair | Host gem.DFW.openstack.org not found: 3(NXDOMAIN) | 21:52 |
| *** annegent_ has quit IRC | 21:52 | |
| jeblair | ready scripts are failing with that error | 21:52 |
| *** pcrews has quit IRC | 21:52 | |
| fungi | jeblair: to disambiguate this from recent discussion about stale file handles on nfs, maybe mention that this is a regular (ext4) filesystem? | 21:52 |
| jeblair | can someone fix that please? ^ | 21:52 |
| * ianw still catching up ... is pushing changes, etc, generally making things worse, or is this only affecting some git trees? | 21:52 | |
| *** mpopow has quit IRC | 21:52 | |
| zaro | anteaya: msg is to gerrit mailing list. | 21:52 |
| bhunter71 | anteaya: I think you're saying that if I create a branch in the gerrit UI, it will just replicate automaticaly (whereby my replication is currently explicitly only the master branch, but when I lift that, it should just work) | 21:52 |
| *** sdake has joined #openstack-infra | 21:52 | |
| anteaya | zaro: ah sorry, I thought it was to -dev | 21:53 |
| *** annegentle has joined #openstack-infra | 21:53 | |
| anteaya | my mistake | 21:53 |
| *** jamesmcarthur has joined #openstack-infra | 21:53 | |
| jeblair | mordred, fungi: ack will address | 21:53 |
| fungi | ianw: completely unknown | 21:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Revert "Configure to use gem mirrors" https://review.openstack.org/182099 | 21:54 |
| fungi | mordred: thanks | 21:54 |
| anteaya | my replication is currently explictly only master branch, where did you specify that gerrit is to replicate only your master branch to the mirrors? | 21:54 |
| zaro | fungi: IIRC there's a bit of a delay from when the change merges before it appears on github. | 21:55 |
| mordred | sorry. that was my bad - I left off teh depends-on | 21:55 |
| *** kgiusti has quit IRC | 21:55 | |
| *** mriedem is now known as mriedem_away | 21:56 | |
| morganfainberg | fungi, clarkb, do we have a definitive list of repos that are under investigation post gerrit upgrade? | 21:57 |
| fungi | morganfainberg: it doesn't seem to be repo-specific | 21:58 |
| * morganfainberg isn't seeing a ML topic...but might be blind. | 21:58 | |
| mordred | morganfainberg: current theory is gerrit and/or jgit issue that's fundamental but not immediate in nature | 21:58 |
| morganfainberg | fungi: ah, ok. Keystonemiddleware is generating 500s for me in the gerrit ui in some cases now which is why i was asking | 21:58 |
| morganfainberg | mordred: ah | 21:58 |
| fungi | morganfainberg: yep, we saw errors related to that repo since the latest restart | 21:58 |
| morganfainberg | lbragstad, ^^ | 21:58 |
| morganfainberg | lbragstad: there is your answer. | 21:58 |
| fungi | morganfainberg: i think so far the only message to the ml was from me in reply to a reply to the maintenance completion e-mail | 21:59 |
| *** stevemar has quit IRC | 21:59 | |
| morganfainberg | ok cool. | 21:59 |
| bhunter71 | anteaya; sorry- I have some replication config in modules/<openstack_project>/manifests/review.pp | 21:59 |
| lbragstad | ah, cool, just checking | 21:59 |
| bhunter71 | push => [ | 21:59 |
| bhunter71 | '+refs/heads/master:refs/heads/master', | 21:59 |
| bhunter71 | '+refs/changes/*:refs/changes/*', | 21:59 |
| bhunter71 | '+refs/meta/config:refs/meta/config', | 21:59 |
| bhunter71 | ] | 21:59 |
| morganfainberg | just checking on things. | 21:59 |
| lbragstad | thanks fungi! | 21:59 |
| anteaya | bhunter71: use a paste service please | 21:59 |
| fungi | we don't have any great answers yet, just more questions | 21:59 |
| morganfainberg | fungi, mordred: thanks. will keep an eye on convos here as well | 21:59 |
| jeblair | bhunter71: we're dealing with an incident right now, can you hold your questions for another time please? | 21:59 |
| bhunter71 | anteaya: I think the point of clarification is you're saying "once a merge has taken place", and I'm wondering where this merge happens when I'm creating a branch | 22:00 |
| anteaya | my fault sorry | 22:00 |
| anteaya | bhunter71: jeblair has asked this discussion to take place later | 22:00 |
| jeblair | (and i imagine we're probably about a day or two away from having our incident channel logged because we can't use any nodes right now) | 22:00 |
| *** emagana has quit IRC | 22:00 | |
| bhunter71 | gotcha, sorry, ttyl. thanks anteaya, out. | 22:00 |
| *** annegentle has quit IRC | 22:00 | |
| anteaya | thanks | 22:01 |
| fungi | zaro: so... the commits i merged 25 minutes ago, not on github and no replication tasks oending in the show-queue output | 22:01 |
| mordred | jeblair: perhaps we should back that change out on nodepool - we're not going to get a ready node to land the read node fix | 22:01 |
| jeblair | mordred: i think if you force-merged your commit it would fix the problem | 22:01 |
| mordred | yah | 22:01 |
| jeblair | mordred: i think you should do that | 22:01 |
| mordred | sorry - that's what I meant - I'm not using english | 22:01 |
| mordred | doing now | 22:01 |
| fungi | zaro: not sure if things are just weird with replication on review-dev for unrelated reasons, or something with this gerrit+jgit version combo | 22:01 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Configure to use gem mirrors" https://review.openstack.org/182099 | 22:02 |
| mordred | k | 22:02 |
| *** waynr has joined #openstack-infra | 22:03 | |
| mordred | jeblair: ok. I kicked puppet on nodepool to boot - the gem change revert ahs been applied | 22:03 |
| mordred | so we should get nodes again | 22:03 |
| zaro | fungi: taking a look | 22:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Revert "Revert "Configure to use gem mirrors"" https://review.openstack.org/182101 | 22:04 |
| mordred | ok. that can be left for later | 22:04 |
| *** baoli has quit IRC | 22:05 | |
| *** emagana has joined #openstack-infra | 22:05 | |
| *** ddieterly has joined #openstack-infra | 22:06 | |
| *** bhunter71 has quit IRC | 22:07 | |
| zaro | fungi: hmm, the replication logs used to be pretty noisy. but don't see anything in the logs now. | 22:07 |
| *** aysyd has quit IRC | 22:08 | |
| zaro | fungi: also i remember afs not working so it would throw a bunch of errors, unless that's been fixed of course. | 22:08 |
| fungi | zaro: my bad, they are queued | 22:08 |
| fungi | i was looking at show-queue from review not review-dev | 22:08 |
| zaro | ahh | 22:08 |
| fungi | lots of queued up retries to push to github | 22:08 |
| jeblair | mordred, fungi: msg updated; changed paragraphs in bold | 22:09 |
| fungi | so maybe we just don't have the keys puppeted properly there yet | 22:09 |
| zaro | yup, ran into that problem last time | 22:09 |
| *** Krinkle|detached is now known as Krinkle | 22:09 | |
| zaro | i remember there was a change to add the key. but maybe invalid still? | 22:09 |
| fungi | jeblair: looks great--hopefully that will get a faster response than the non-response in irc | 22:10 |
| *** amitgandhinz has quit IRC | 22:10 | |
| mordred | jeblair: looking | 22:10 |
| zaro | fungi: change to setup keys https://review.openstack.org/#/c/174133/ | 22:10 |
| mordred | jeblair: yes. that's excellent | 22:11 |
| anteaya | jeblair: did you want to give timelines? that the upgrade took place Saturday, and perhaps how many restarts today | 22:11 |
| *** banix has quit IRC | 22:12 | |
| jeblair | anteaya: i don't think it would help | 22:12 |
| anteaya | okay | 22:12 |
| fungi | well, today was the first day there was any significant activity on the server since the upgrade | 22:12 |
| anteaya | then everything else makes sense to me | 22:12 |
| anteaya | yes was thinking that might be worth conveying | 22:12 |
| fungi | so for all intents and purposes, "immediately after upgrade" | 22:12 |
| anteaya | okay fine, just thought I would mention it | 22:13 |
| zaro | jeblair: lgtm | 22:13 |
| *** moshele has joined #openstack-infra | 22:13 | |
| jeblair | anteaya, yeah, it just means i have to spend a paragraph explaining why "two days later" is "immediately", so i'd like to avoid the extra data. main thing is that when under load, it happens shortly after restarts, and that's in there. | 22:13 |
| anteaya | okay | 22:13 |
| *** whoops has quit IRC | 22:17 | |
| fungi | zaro: the ssh key on the old review-dev still works, and is not the same as the one on new review-dev | 22:17 |
| fungi | i'll fix it real quick so we can make sure it's working as intended with gerrit 2.10 and jgit 3.7 | 22:18 |
| jeblair | fungi: thx | 22:18 |
| *** bswartz has joined #openstack-infra | 22:18 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/project-config: Add bifrost to OpenStack https://review.openstack.org/182022 | 22:18 |
| jeblair | mordred: Host gem.DFW.openstack.org not found: 3(NXDOMAIN) | 22:18 |
| jeblair | is still happening | 22:18 |
| jeblair | mordred: oh, it's because ready scripts are built into images, yeah? | 22:19 |
| jeblair | so this is a delete-current-image situation | 22:19 |
| *** rkukura_ has joined #openstack-infra | 22:19 | |
| mordred | gah. | 22:19 |
| mordred | on it | 22:19 |
| mordred | why did I think ready scripts were copied at run time? | 22:19 |
| jeblair | because we keep wishing they were :) | 22:20 |
| mordred | one of these days... | 22:20 |
| jeblair | (i think we can fix that, but later) | 22:20 |
| * mordred stops hacking on the fix to nodepool | 22:20 | |
| jeblair | mordred: so you're deleting <24h old images? | 22:20 |
| mordred | yah | 22:20 |
| *** claudiub has quit IRC | 22:20 | |
| mordred | WOW there are some old images up in here | 22:21 |
| jeblair | yeah, and i think they are the only reason anything is working :) | 22:21 |
| fungi | try not to think about it | 22:21 |
| *** tim_o has joined #openstack-infra | 22:21 | |
| fungi | i think gerrit caches this key too, so restarting it on review-dev real fast | 22:21 |
| *** dimtruck is now known as zz_dimtruck | 22:22 | |
| fungi | okay, it's back up and running again | 22:22 |
| *** rkukura has quit IRC | 22:22 | |
| *** rkukura_ is now known as rkukura | 22:22 | |
| mordred | fungi: fwiw, bare-centos6 seems to have not built in a very long time | 22:22 |
| fungi | mordred: in hpcloud? rax? both? | 22:23 |
| fungi | mordred: nevermind--i'll think about it later ;) | 22:23 |
| mordred | hp for now | 22:23 |
| *** pcm_ has quit IRC | 22:23 | |
| fungi | but thanks for the heads up | 22:23 |
| fungi | replication success https://github.com/gtest-org/test2/commits/master | 22:23 |
| mordred | thought for later - graphs tracking image age per image | 22:24 |
| fungi | after restart it queued up some new replication pushes, but those have completed | 22:24 |
| *** jgrimm is now known as zz_jgrimm | 22:24 | |
| fungi | mordred: maybe just a table of images available and their ages as part of the new nodepool status api | 22:25 |
| *** rwsu has quit IRC | 22:25 | |
| mordred | yah | 22:25 |
| mordred | fungi: nodepool delete-all older-than:1day | 22:25 |
| mordred | also - I'm a moron. why have I not thought about nodepool image-list | sort before now | 22:25 |
| mordred | since, you know, image ids are autoinc ints | 22:26 |
| jeblair | mordred: i hope you are deleting all that are _newer_ than 1 day | 22:26 |
| mordred | yah | 22:26 |
| mordred | :) | 22:26 |
| fungi | okay, i have no identified usage problems with gerrit 2.10 and jgit 3.2 on review-dev | 22:26 |
| jeblair | okay. there is never a reason to delete images _older_ than 1 day | 22:26 |
| mordred | jeblair: well, when you want to cause havoc | 22:26 |
| *** tonytan4ever has quit IRC | 22:26 | |
| jeblair | okay, should we just ninja the same build onto production? | 22:27 |
| jeblair | because at this point we might be waiting a while to do it the right way, due to the bad images | 22:27 |
| *** arielb has quit IRC | 22:27 | |
| fungi | zaro: get up with me later to check that the replication key for review-dev is getting puppeted successfully (since that change to add it merged a couple weeks ago) and that it's entered correctly in hiera | 22:27 |
| zaro | fungi: ok | 22:27 |
| *** soren_ has quit IRC | 22:27 | |
| fungi | thanks. i'll likely forget otherwise | 22:27 |
| zaro | ++ ninja | 22:28 |
| fungi | right now i just copied it from the old review-dev manually to make sure it worked | 22:28 |
| jeblair | also, i'm thinking we should change the puppet manifest to not reindex on upgrade; i think that's causing us more problems than it's worth. | 22:28 |
| lifeless | anyone have objections to a pbr release happening now ? | 22:28 |
| fungi | lifeless: you are a funny one | 22:28 |
| fungi | in for a penny, in for a pound. if we've tested this dirty build of gerrit, then we might as well roll it into production while we're in fact-finding mode | 22:29 |
| fungi | especially if we're talking an hour or more to get workers to build a clean one from the proposed patch | 22:29 |
| lifeless | fungi: ? | 22:29 |
| mordred | lifeless: we're in the middle of a day-long incident | 22:29 |
| fungi | lifeless: channel /topic | 22:30 |
| anteaya | lifeless: yes, now would not be a good time | 22:30 |
| lifeless | fungi: ack | 22:30 |
| mordred | lifeless: however, THANK YOU for asking | 22:30 |
| jeblair | fungi: sounds good to me. you want to do it? | 22:30 |
| fungi | lifeless: yes, sorry, i actually thought you were making a joke ;) | 22:30 |
| zaro | jeblair: hmm, i think reindex is required *sometimes* | 22:30 |
| lifeless | fungi: nope, twas not :) | 22:30 |
| fungi | zaro: going between essentially the same gerrit code just with different jgit lib version? | 22:31 |
| jeblair | zaro: it is sometimes, it's just that whenever we want to make bugfixes like this, we almost never want it. so it hinders our ability to use puppet the way we want and forces us to circumvent it. | 22:31 |
| zaro | ohh i see, you mean just for this time. | 22:31 |
| jeblair | or the last time | 22:31 |
| jeblair | (last 3 times actually) | 22:31 |
| jeblair | anyway, can hash it out later | 22:32 |
| *** ociuhandu has joined #openstack-infra | 22:32 | |
| fungi | so i think as long as i stop gerrit, copy the dirty build to ~gerrit2/review_site/etc/gerrit.war and then start it again, we'll avoid teh reindex for now | 22:33 |
| fungi | and shouldn't need to disable puppet agent? | 22:33 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: post_test_hook is not executed when the gate test returns an error https://review.openstack.org/148863 | 22:33 |
| jeblair | fungi: agreed | 22:33 |
| fungi | er, bin not etc | 22:33 |
| fungi | but ykwim | 22:33 |
| mordred | yah | 22:34 |
| fungi | copying it to the server now | 22:34 |
| zaro | fungi: you need to run init | 22:34 |
| fungi | zaro: ahh, thanks. what's the easiest way to do that? just copy the comment from the gerrit exec? | 22:35 |
| jeblair | why run init? | 22:35 |
| fungi | does it do anything besides redownload libs? | 22:35 |
| fungi | and run migrations and whatever? | 22:35 |
| fungi | like, anything we'd need to actually have updated between these two wars? | 22:35 |
| jeblair | (because if we run init, we'll need to clean up after it before starting) | 22:36 |
| zaro | fungi: cmd is 'java -jar $jar_file init --batch --no-auto-start -d review_site' | 22:36 |
| zaro | need to run init because it will copy jgit libs into review_site | 22:36 |
| zaro | ohh wait, maybe it just copies the gerrit.war to review_site | 22:37 |
| zaro | ohh so, maybe just copy the war to review_site/bin | 22:37 |
| fungi | that's what i'm doing | 22:37 |
| zaro | safer just to run the init command. | 22:38 |
| jeblair | not safer | 22:38 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/devstack-gate: Allow for git base override https://review.openstack.org/151034 | 22:38 |
| zaro | ohh ok. | 22:38 |
| jeblair | again, if you run init, you need to clean up after it | 22:38 |
| jeblair | let's just go with fungi's original plan, unless someone can state why init needs to run | 22:38 |
| zaro | i'm ok with that. but make sure it's the bin folder not the etc folder as you mentioned earlier | 22:39 |
| fungi | yep | 22:39 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/system-config: Initial manual for infra-cloud https://review.openstack.org/180796 | 22:39 |
| fungi | i've initially copied it to ~gerrit2/gerrit-wars/gerrit-v2.10.2.23.g039a170.dirty.war for ease of access | 22:39 |
| fungi | in case it comes up later | 22:39 |
| fungi | should we do it asap, or do we need to send out an update in irc first? | 22:40 |
| jeblair | fungi: let's just do it | 22:40 |
| fungi | stoppign gerrit now | 22:40 |
| jeblair | it will not make things much worse for many people :) | 22:40 |
| *** pcrews has joined #openstack-infra | 22:40 | |
| zaro | i need to catch a carpool home, should be back online in 40 mins. | 22:40 |
| mordred | zaro: hopefully there will be nothing interesting going on at that time | 22:41 |
| fungi | dirty gerrit build is on its way up now | 22:41 |
| jeblair | tbh, i sort of expect that this will not solve the problem | 22:41 |
| mordred | jeblair: I REALLY hope that you're wrong | 22:42 |
| fungi | with our luck | 22:42 |
| jeblair | me too. it's just a hunch. | 22:42 |
| mordred | yah | 22:42 |
| mordred | well, sadly you tend to have good hunches | 22:42 |
| fungi | confirmed gerrit is up and working on "2.10.2-23-g039a170-dirty" per the webui | 22:42 |
| jeblair | (based on where we got to looking at jgit changes and possible causes) | 22:42 |
| *** moshele has left #openstack-infra | 22:42 | |
| *** esker has joined #openstack-infra | 22:42 | |
| *** otter768 has joined #openstack-infra | 22:43 | |
| jeblair | that did not take long | 22:44 |
| jeblair | [2015-05-11 22:14:40,084] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user openstack-project-creator account 6199) during git-upload-pack '/opensta | 22:45 |
| jeblair | ck/neutron' | 22:45 |
| jeblair | org.eclipse.jgit.errors.PackProtocolException: internal revision error | 22:45 |
| mordred | wow | 22:45 |
| mordred | well, at least we didnt' have to wait | 22:45 |
| fungi | huh. didn't see any of that on review-dev | 22:45 |
| mordred | so - revert to normal 2.10 while we ready ourselves for reverting? | 22:45 |
| fungi | yep | 22:45 |
| jeblair | mordred: i think so | 22:45 |
| mordred | oh god | 22:45 |
| fungi | stopping gerrit again now | 22:45 |
| mordred | I'm so unhappy about this | 22:45 |
| *** achanda has joined #openstack-infra | 22:46 | |
| mordred | jeblair: I feel like there was a config patch we applied that will need to be reverted ... | 22:46 |
| *** mpopow has joined #openstack-infra | 22:46 | |
| fungi | gerrit's on its way back up with normal 2.10 again | 22:47 |
| *** nelsnelson has quit IRC | 22:47 | |
| anteaya | do we need an etherpad for the rollback? | 22:47 |
| jeblair | i don't see the same zlib triggering error | 22:47 |
| jeblair | [2015-05-11 22:43:51,636] WARN com.google.gerrit.server.patch.PatchListLoader : Error computing PatchListKey[openstack/nova BASE..b3ee6a14c2dc1ced7ce775a6806da74a0f84b423 IGNORE_NONE] | 22:47 |
| jeblair | java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 19a0c41a3fb38875259a6353c76c90ee3433c195 | 22:47 |
| *** otter768 has quit IRC | 22:47 | |
| jeblair | is first error | 22:48 |
| jeblair | er | 22:48 |
| jeblair | i wonder if we should clear the cache? | 22:48 |
| jeblair | so two thoughts: | 22:49 |
| fungi | does the cache persist between restarts? | 22:49 |
| mordred | I didn't think so? | 22:49 |
| jeblair | 1) did not clearing the cache cause the -dirty rollback to fail? | 22:49 |
| mordred | oh. | 22:49 |
| jeblair | 2) did not clearing the cache cause the zlib problem we're seeing? | 22:49 |
| jeblair | it does in review_site/cache | 22:49 |
| jeblair | it caches diffs | 22:49 |
| fungi | ohhh | 22:49 |
| mordred | hrm | 22:49 |
| *** dangers is now known as dangers_away | 22:50 | |
| fungi | i can restart it, moving that out of the way | 22:50 |
| jeblair | i have no idea if cache inconsistency could relate to missing object errors | 22:50 |
| mordred | me either | 22:50 |
| jeblair | and i guess while we're at it: 3) could the repack have caused the cache to get out of sync causing the zlib errors | 22:50 |
| jeblair | fungi: let's try -dirty again with no cache | 22:50 |
| *** trey has quit IRC | 22:50 | |
| jeblair | or should we try -clean again with no cache? | 22:51 |
| *** mpopow has quit IRC | 22:51 | |
| fungi | okay, so move ~gerrit2/review_site/cache to cache.old? or just delete the files in it? | 22:51 |
| jeblair | fungi: lets move | 22:51 |
| *** Sukhdev has quit IRC | 22:52 | |
| jeblair | and let's try -dirty | 22:52 |
| jeblair | since it's likely to fail quickly if the problem is not cache related :) | 22:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add SQL file for emergency rollback https://review.openstack.org/182113 | 22:52 |
| jeblair | (it's just that if dirty works, we need to try clean again with no cache) | 22:52 |
| mordred | yah | 22:53 |
| fungi | okay, doing now | 22:53 |
| fungi | it's starting again on dirty with no cache | 22:54 |
| fungi | confirmed, running what i think it should | 22:54 |
| jeblair | it has not immediately blown up | 22:56 |
| *** trey has joined #openstack-infra | 22:56 | |
| *** bknudson has quit IRC | 22:56 | |
| mordred | neat! | 22:56 |
| *** esker has quit IRC | 22:57 | |
| jeblair | er, my stream events just failed and now i'm getting rejected connections | 22:57 |
| jhesketh | Morning | 22:57 |
| fungi | yep, big traceback in the error log | 22:57 |
| anteaya | morning jhesketh | 22:57 |
| fungi | jetty seems to have gone belly-up | 22:58 |
| *** craigbr has quit IRC | 22:58 | |
| anteaya | jhesketh: troubleshooting the gerrit upgrade: https://etherpad.openstack.org/p/ZIepsHViGk | 22:58 |
| fungi | blow away the cache and switch to the clean 2.10 build now? | 22:58 |
| jhesketh | Yeah just saw some rollback. This doesn't look fun | 22:58 |
| jeblair | fungi: sure. i don't know what to make of this. | 22:58 |
| anteaya | jhesketh: not so much | 22:58 |
| jhesketh | Let me know how I can help (although I'm only partially here today) | 22:59 |
| zaro | that's weird i would expect something like that to happen on review-dev as well. | 22:59 |
| jeblair | 1) No index versions ready; run Reindex | 22:59 |
| *** reed_ has joined #openstack-infra | 23:00 | |
| fungi | Starting Gerrit Code Review: FAILED | 23:00 |
| fungi | start a reindex? | 23:00 |
| zaro | we still on dirty? | 23:00 |
| fungi | or put the original cache back? | 23:01 |
| jeblair | fungi: yeah, maybe move the index dir out of the way just to be sure | 23:01 |
| fungi | doing | 23:01 |
| *** gordc has quit IRC | 23:01 | |
| jeblair | fungi: i would not put the cache back; i'd also delete the current cache | 23:01 |
| jeblair | so clear cache, clear index, reindex, start 2.10-clean | 23:01 |
| fungi | yep, done and reindex underway | 23:01 |
| fungi | the original cache from before we started this particular experiment is still cache.old and index is moved to index.old | 23:02 |
| fungi | clean war is in place and reindex is currently running | 23:02 |
| anteaya | zaro: you home now, that was only 20 minutes? | 23:02 |
| fungi | so starting from empty cache and empty index | 23:03 |
| jeblair | fungi: think we ought to do a git fsck on nova while that's running, just to make sure it wasn't damaged? | 23:03 |
| zaro | anteaya: neat thing about carpool is that you can use your computer as a passenger | 23:03 |
| anteaya | zaro: k | 23:03 |
| fungi | jeblair: on it | 23:03 |
| fungi | i'll fsck a cp -ax of it as a test | 23:03 |
| *** emagana has quit IRC | 23:03 | |
| *** funzo has joined #openstack-infra | 23:04 | |
| jeblair | fungi: ok | 23:04 |
| fungi | reindex is about 1/3 done | 23:04 |
| *** yamamoto has joined #openstack-infra | 23:05 | |
| *** ashleighfarnham has quit IRC | 23:06 | |
| fungi | git fsck shows nothing out of the ordinary besides the dangling blobs and commits we saw earlier | 23:08 |
| *** reed_ has quit IRC | 23:08 | |
| jeblair | ok | 23:08 |
| fungi | reindex is about half done now | 23:08 |
| *** reed_ has joined #openstack-infra | 23:08 | |
| fungi | nearly there | 23:10 |
| *** amitgandhinz has joined #openstack-infra | 23:10 | |
| *** jamesmcarthur has quit IRC | 23:12 | |
| *** dboik_ has quit IRC | 23:12 | |
| *** reed_ has quit IRC | 23:12 | |
| fungi | ...so...slow... | 23:13 |
| *** ldnunes has quit IRC | 23:13 | |
| *** panda|afk has quit IRC | 23:13 | |
| *** panda|afk has joined #openstack-infra | 23:14 | |
| *** ddieterly has quit IRC | 23:14 | |
| fungi | okay, clean gerrit started now with empty cache and fresh indexes | 23:14 |
| jeblair | it seems to be able to receive patchsets | 23:17 |
| mordred | that'sa good start | 23:17 |
| *** ivar-laz_ has joined #openstack-infra | 23:18 | |
| *** hemna_ is now known as hemnafk | 23:19 | |
| *** ivar-laz_ has quit IRC | 23:19 | |
| *** ivar-laz_ has joined #openstack-infra | 23:19 | |
| *** Longgeek_ has quit IRC | 23:20 | |
| *** Daviey has quit IRC | 23:21 | |
| *** carl_baldwin has quit IRC | 23:21 | |
| *** Daviey has joined #openstack-infra | 23:21 | |
| *** ivar-lazzaro has quit IRC | 23:21 | |
| *** lhcheng has joined #openstack-infra | 23:22 | |
| *** esker has joined #openstack-infra | 23:22 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for keep-records parameter for plots-plugin https://review.openstack.org/182116 | 23:24 |
| *** rrepaka has joined #openstack-infra | 23:24 | |
| jeblair | [2015-05-11 23:24:15,695] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : ERROR: Exception caught while accessing pack file /home/gerrit2/review_site/git/stackforge/fuel-web.git/objects/pack/pack-a814e22337395838b039d1e8ef3eef47ff3cd9fe.pack, the pack file might be corrupt | 23:24 |
| jeblair | java.io.EOFException: Short compressed stream at 25,794,933 | 23:24 |
| jeblair | that's different | 23:24 |
| rrepaka | lifeless: Hi | 23:24 |
| jeblair | and it did not remove the pack file | 23:24 |
| jeblair | i'm going to check out the file | 23:25 |
| * mordred keeps fingers crossed | 23:25 | |
| lifeless | rrepaka: hi? | 23:25 |
| rrepaka | lifeless: Pbr is returning a wrong version string https://bugs.launchpad.net/pbr/+bug/1453996 | 23:25 |
| openstack | Launchpad bug 1453996 in PBR "pbr-0.11.0 Returns wrong version string for oslo-config" [Undecided,New] | 23:25 |
| zaro | just noticed that label queries now need to be set with 'label', so this '-Verified-1' now needs to be '-label:Verified-1' | 23:25 |
| rrepaka | lifeless: Can this be fixed quickly ? | 23:26 |
| rrepaka | lifeless:Just checking if it takes longer i will go ahead with patching my build to use pbr 0.10 | 23:26 |
| jeblair | [2015-05-11 23:26:12,079] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory : Pack file /home/gerrit2/review_site/git/openstack/glance.git/objects/pack/pack-f | 23:27 |
| jeblair | b9c038839e5238ba128a173570ee7787905edfd.pack is corrupt, removing it from pack list | 23:27 |
| lifeless | rrepaka: it works for me | 23:27 |
| jeblair | org.eclipse.jgit.errors.CorruptObjectException: Object at 26,164,967 in /home/gerrit2/review_site/git/openstack/glance.git/objects/pack/pack-fb9c038839e5238ba128a173570e | 23:27 |
| jeblair | e7787905edfd.pack has bad zlib stream | 23:27 |
| jeblair | there's our usual error | 23:27 |
| lifeless | rrepaka: what does 'git describe' output? | 23:27 |
| mordred | lifeless, rrepaka: if y'all could move the pbr discussion elsewhere, that would be helpful | 23:27 |
| fungi | yep, a ton of MissingObjectException errors now | 23:27 |
| rrepaka | lifeless: its 1.4.0 | 23:27 |
| mordred | well, at least we learned that it's not the cache | 23:27 |
| fungi | so i think we've probably done all we can short of rollback | 23:28 |
| lifeless | rrepaka: -> #openstack-oslo please. | 23:28 |
| fungi | no reply on the repo-discuss ml yet either | 23:29 |
| mordred | so - https://review.openstack.org/#/c/182113/ has the script | 23:29 |
| mordred | do we need to revert https://review.openstack.org/#/c/172534/ ? | 23:30 |
| fungi | i believe so. zaro? ^ | 23:30 |
| mordred | as well as the version rollback and the database rollback? | 23:30 |
| *** teran has quit IRC | 23:31 | |
| fungi | oh, wait, that's version-detecting now | 23:31 |
| fungi | i think we can just roll back the war version and not have to touch 172534 | 23:31 |
| jeblair | i'm seeing a lot of 'short compressed stream' errors now | 23:31 |
| fungi | yeah, we landed it well before the upgrade | 23:31 |
| mordred | neat | 23:31 |
| jeblair | i haven't found any actual git problems yet; i'll continue to look | 23:31 |
| jeblair | feel free to proceed with prepping downgrade | 23:32 |
| fungi | where was our upgrade etherpad? anyone have the url handy? | 23:32 |
| *** wenlock has quit IRC | 23:32 | |
| jeblair | https://etherpad.openstack.org/p/gerrit-2.10-upgrade | 23:32 |
| fungi | thanks, just found it in my history too | 23:32 |
| anteaya | hopefully zaro should be home soon, 50 minutes since his alert he was afk | 23:33 |
| fungi | i'll propose a revert of 155463 now | 23:33 |
| zaro | looking | 23:33 |
| jeblair | there are 'short compressed stream' errors in the log from before we switched to -dirty, so i don't think they are entirely new | 23:33 |
| mordred | oh - I just thought of something else that shoudl be in the migration | 23:34 |
| mordred | the schema version is recorded in it isn't it? | 23:34 |
| *** dims has quit IRC | 23:34 | |
| jeblair | yeah, is a single row in a table | 23:34 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Revert "Upgrade Gerrit from 2.8 to ver 2.10 on review.o.o" https://review.openstack.org/182117 | 23:35 |
| zaro | should not have to rever that change. made it so that it will install correct libs for either 2.8 or 2.10 | 23:35 |
| mordred | yah. I think the script should set it when it's done | 23:35 |
| jeblair | mordred: can you double check the version table itself didn't change? it has 2 columns now, and i can't remember if it did before | 23:35 |
| fungi | zaro: thanks for verifying. i came to the same conclusion after i thought about it | 23:35 |
| *** lhcheng has left #openstack-infra | 23:35 | |
| mordred | jeblair: it did not show up in the schema diff | 23:35 |
| fungi | shall i go ahead and disable puppet on review.o.o? | 23:36 |
| anteaya | so I'm clear, are we going back to 2.8? | 23:36 |
| fungi | for now, yes | 23:36 |
| anteaya | thank you | 23:36 |
| *** tim_o has quit IRC | 23:37 | |
| fungi | until we come up with additional evidence through testing on review-dev as to the cause or get some feedback from the gerrit devs | 23:37 |
| *** esker has quit IRC | 23:37 | |
| *** marun has quit IRC | 23:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add SQL file for emergency rollback https://review.openstack.org/182113 | 23:37 |
| anteaya | fungi: right | 23:37 |
| anteaya | thanks | 23:38 |
| ianw | "This did not work as planned" | 23:38 |
| anteaya | yeah :( | 23:38 |
| *** dims has joined #openstack-infra | 23:39 | |
| fungi | ianw: i am in some universes known as "cap'n obvious" | 23:39 |
| anteaya | jhesketh: want to learn something about how to downgrade a gerrit? | 23:39 |
| jhesketh | anteaya: what's that? | 23:40 |
| * jhesketh has been watching the conversation | 23:40 | |
| anteaya | jhesketh: we are downgrading to gerrit 2.8 | 23:40 |
| anteaya | ah cool | 23:40 |
| anteaya | wasn't sure | 23:40 |
| jhesketh | I just don't have much to add but if there is anything I can do to help let me know | 23:41 |
| anteaya | jhesketh: this was the upgrade etherpad: https://etherpad.openstack.org/p/gerrit-2.10-upgrade | 23:41 |
| anteaya | jhesketh: well was poking you along those lines, as it will likely be we're done and offline | 23:41 |
| anteaya | over to you jhesketh | 23:41 |
| anteaya | wanted to make sure you were playing along at home | 23:42 |
| jhesketh | Yep as best I can | 23:42 |
| anteaya | thanks | 23:42 |
| mordred | I DON'T want to learn something about downgrading a gerrit | 23:42 |
| mordred | but I'm going to anyway | 23:42 |
| anteaya | well that is a good point | 23:42 |
| jhesketh | Heh | 23:43 |
| fungi | everything we never wanted to know about gerrit downgrades but have been forced to find out anyway | 23:43 |
| anteaya | will you have a book signing at summit? | 23:43 |
| *** dims has quit IRC | 23:43 | |
| jhesketh | fungi: there's a conference talk topic | 23:43 |
| fungi | did we update the puppet exec timeout for the reindex? do we want to see about doing this with a puppet dev env so that we can patch in a change to avoid automatic reindex exec? | 23:44 |
| anteaya | do we have a list of steps for the downgrade? | 23:44 |
| anteaya | so we can keep track of done things? | 23:44 |
| fungi | anteaya: i'm working on it in https://etherpad.openstack.org/p/ZIepsHViGk | 23:45 |
| *** miqui has quit IRC | 23:45 | |
| anteaya | thank you | 23:45 |
| greghaynes | Do we want to do another db dump before the downgrade migration? | 23:46 |
| mordred | greghaynes: yes | 23:46 |
| greghaynes | heh, the author of the downgrade seems keen on that, I wonder why ;) | 23:46 |
| *** yamamoto has quit IRC | 23:46 | |
| mordred | ETERROR | 23:47 |
| *** ams__ has quit IRC | 23:47 | |
| fungi | *quaking in virtual boots* | 23:48 |
| mordred | these virtual boots are not keeping my feet warm | 23:48 |
| mordred | these pretzels are making me thirsty | 23:50 |
| *** gmann has joined #openstack-infra | 23:50 | |
| anteaya | zaro are you about and able to offer thoughts on the downgrade plan unfolding on https://etherpad.openstack.org/p/ZIepsHViGk | 23:51 |
| jeblair | status alert Gerrit is going offline while we perform an emergency downgrade to version 2.8. | 23:51 |
| jeblair | look good ^ ? | 23:51 |
| fungi | jeblair: lgtm | 23:51 |
| anteaya | yup | 23:51 |
| jeblair | #status alert Gerrit is going offline while we perform an emergency downgrade to version 2.8. | 23:51 |
| openstackstatus | jeblair: sending alert | 23:51 |
| fungi | mordred: if you're doing the db commands on the gerrit server, you want to do the other local tasks there so that we don't step on one another's virtual boots? | 23:52 |
| mordred | fungi: sure | 23:52 |
| *** oomichi has joined #openstack-infra | 23:52 | |
| jeblair | i'm writing an email update for the -dev list; we can finish and review it later, just didn't want anyone else to start on one | 23:52 |
| anteaya | thanks | 23:53 |
| mordred | I'm going to to ahead and disable puppet on review.o.o | 23:53 |
| fungi | thanks | 23:53 |
| mordred | done | 23:53 |
| -openstackstatus- NOTICE: Gerrit is going offline while we perform an emergency downgrade to version 2.8. | 23:53 | |
| *** ChanServ changes topic to "Gerrit is going offline while we perform an emergency downgrade to version 2.8." | 23:53 | |
| fungi | seems to not be running either. that's goof | 23:54 |
| fungi | er, good | 23:54 |
| mordred | fungi: actually - my mysql things are as root - you can do the things that happen as gerrit2 perhaps? | 23:54 |
| zaro | anteaya: i've comment on the changes | 23:54 |
| fungi | mordred: sure thing | 23:54 |
| anteaya | zaro: thank you | 23:55 |
| mordred | fungi: especially since you've been doing those the last few times | 23:55 |
| clarkb | I am almost home can help once thete | 23:55 |
| *** dboik has joined #openstack-infra | 23:55 | |
| fungi | mordred: which is basically nothing except handling the cache and reindex, but happy to do so | 23:55 |
| anteaya | clarkb: thank you, travel safe | 23:55 |
| *** ZZelle has quit IRC | 23:55 | |
| fungi | clarkb: we have plenty of hands on deck, and there will almost certainly be aftermath, so there's no rush | 23:56 |
| openstackstatus | jeblair: finished sending alert | 23:56 |
| clarkb | kk | 23:56 |
| fungi | carnage awaits us all | 23:56 |
| mordred | EAFTERMATH | 23:56 |
| *** dboik_ has joined #openstack-infra | 23:56 | |
| mordred | except Exception: raise Aftermath() | 23:56 |
| fungi | anybody want to +2 https://review.openstack.org/182117 ? then i'll submit it and make sure it's showing up on the puppetmaster | 23:57 |
| *** sdake_ has joined #openstack-infra | 23:57 | |
| jeblair | fungi: done | 23:57 |
| mordred | two of us even | 23:58 |
| fungi | merging | 23:59 |
| *** banix has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!