| *** harlowja has joined #openstack-infra | 00:00 | |
| zaro | fungi: odd, why do i need to do that? i see the gerrit stuff right after i clone jjb. | 00:00 |
|---|---|---|
| *** melwitt has joined #openstack-infra | 00:00 | |
| *** Loquacity has joined #openstack-infra | 00:00 | |
| *** loq_mac has quit IRC | 00:00 | |
| zaro | fungi: yeah, that fixed it. but i don't know why i have to do that. | 00:00 |
| *** dolphm has joined #openstack-infra | 00:01 | |
| fungi | zaro: git won't have a remote named "gerrit" until the first time git-review runs. when you push a tag you're not running git-review, so if you've *never* run git-review in that copy of the repository you'd be missing the remote | 00:01 |
| arosen | fungi: hrm passlib is in keystone's requirements.txt so it should get installed but i don't see that happening here: | 00:01 |
| arosen | http://logs.openstack.org/51/66451/2/check/check-tempest-dsvm-neutron-pg-isolated/6b3bee7/console.html | 00:01 |
| jamielennox | fungi/arosen: that doesn't look like something that would come from keystoneclient tagging | 00:02 |
| *** sarob has quit IRC | 00:02 | |
| dolphm | arosen: maybe it's getting a version of passlib without passlib.hash? | 00:02 |
| zaro | fungi: thanks. | 00:03 |
| dolphm | python-keystoneclient requirements.txt diff from 0.4.2 to 0.4.1 http://pasteraw.com/rnxcju8hjq049m3y870uda5pbdcjl1 | 00:04 |
| jamielennox | passlib should have nothing to do with keystoneclient | 00:04 |
| dolphm | i was about to say -- where's passlib? lol | 00:04 |
| fungi | arosen: jamielennox: dolphm: yeah, i'm stumped, but honestly haven't had time to look closely at the log | 00:04 |
| *** yamahata has joined #openstack-infra | 00:06 | |
| zaro | mgagne: would the fix be to be more open to the types of acceptable parameters? | 00:07 |
| zaro | mgagne: do you want to propose something? | 00:07 |
| mgagne | zaro: not sure how to handle it as I'm not sure how we are supposed to handle deprecation | 00:07 |
| zaro | is that jenkins deprecsted or only jjb deprecated? | 00:08 |
| *** sarob has joined #openstack-infra | 00:09 | |
| mgagne | zaro: not sure if the plugin got updated and now accepts additional parameters as implemented by the committer or not. I would have to check. I'm probably running an old version of the pipeline plugin | 00:09 |
| * zaro takes a look on latest ver | 00:10 | |
| mgagne | zaro: I'm running 1.3.3. 1.4.2 looks to be the latest. | 00:11 |
| *** sarob has quit IRC | 00:13 | |
| *** sarob has joined #openstack-infra | 00:13 | |
| *** sandywalsh has quit IRC | 00:15 | |
| *** bknudson has joined #openstack-infra | 00:16 | |
| openstackgerrit | Sabari Murugesan proposed a change to openstack/requirements: nova api validation fw requires jsonschema >= 2.0.0 https://review.openstack.org/66464 | 00:17 |
| zaro | mgagne: 1.4.2 allows additional params: boolean, current params, params from properties file, and predefined params | 00:18 |
| *** sarob has quit IRC | 00:18 | |
| *** thuc has joined #openstack-infra | 00:19 | |
| *** thuc has quit IRC | 00:19 | |
| *** thuc has joined #openstack-infra | 00:20 | |
| *** thuc has quit IRC | 00:20 | |
| *** hogepodge has quit IRC | 00:20 | |
| *** salv-orlando has joined #openstack-infra | 00:21 | |
| *** thuc has joined #openstack-infra | 00:21 | |
| *** thuc has quit IRC | 00:22 | |
| *** mozawa has joined #openstack-infra | 00:22 | |
| *** thuc has joined #openstack-infra | 00:22 | |
| *** tian has quit IRC | 00:23 | |
| *** salv-orlando has quit IRC | 00:26 | |
| *** gyee_ has quit IRC | 00:28 | |
| *** boris-42 has quit IRC | 00:32 | |
| *** praneshp has joined #openstack-infra | 00:33 | |
| *** sdake has quit IRC | 00:34 | |
| jerryz | fungi: https://bugs.launchpad.net/openstack-ci/+bug/1266603 please add the initial reviewers if you have time. thanks | 00:36 |
| uvirtbot | Launchpad bug 1266603 in openstack-ci "Please add initial core reviewers for stackforge/compass project" [Undecided,New] | 00:36 |
| *** erfanian has joined #openstack-infra | 00:40 | |
| *** yamahata has quit IRC | 00:44 | |
| *** yamahata has joined #openstack-infra | 00:44 | |
| *** dkranz has joined #openstack-infra | 00:49 | |
| zaro | fungi: build failed on tag.. https://jenkins01.openstack.org/view/All/job/gearman-plugin-hpi-artifact/ | 00:49 |
| zaro | fungi: I can't repro on my machine and it hasn't failed before. would you be able to re-run? | 00:50 |
| fungi | zaro: weird. i can retrigger it and see if it continues to happen | 00:51 |
| *** CaptTofu has quit IRC | 00:51 | |
| *** sandywalsh has joined #openstack-infra | 00:51 | |
| *** dcramer_ has joined #openstack-infra | 00:53 | |
| fungi | zaro: https://jenkins01.openstack.org/view/All/job/gearman-plugin-hpi-artifact/5/console | 00:54 |
| fungi | "ERROR: No artifacts found that match the file pattern "target/gearman-plugin-${PROJECT_VER}.hpi". Configuration error?" | 00:55 |
| zaro | fungi: actually it's a test that fails which causes no artifact to be created. | 00:56 |
| fungi | oh, also i see it says "cp ./target/gearman-plugin.hpi ./target/gearman-plugin-.hpi" so it's missing a version value | 00:56 |
| fungi | got it | 00:56 |
| fungi | testStartStopThread(hudson.plugins.gearman.AbstractWorkerThreadTest) | 00:56 |
| fungi | zaro: it ran on precise5 both times. do you think there's any benefit to clearing the workspace for the job on that slave and trying again? | 00:57 |
| zaro | fungi: is it possible to run on another machine? | 00:57 |
| fungi | sure, i can mark that slave temporarily offline and retrigger it again | 00:57 |
| zaro | fungi: we could try but chances are low that will fix it. | 00:58 |
| *** nati_uen_ has quit IRC | 00:58 | |
| zaro | fungi: more likely a timing issue is my guess. | 00:58 |
| fungi | zaro: running on precise25 this time... https://jenkins01.openstack.org/view/All/job/gearman-plugin-hpi-artifact/6/console | 00:59 |
| fungi | and i've brought precise5 back online now | 00:59 |
| *** wenlock has quit IRC | 00:59 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: refactor bot to be based on argparse https://review.openstack.org/66401 | 01:00 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: moving readiness checks into stream https://review.openstack.org/65961 | 01:00 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: fix bot to actually use super() https://review.openstack.org/66405 | 01:00 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: create more sane logging for the er bot https://review.openstack.org/66435 | 01:00 |
| *** melwitt has quit IRC | 01:01 | |
| *** ryanpetrello has joined #openstack-infra | 01:02 | |
| *** nati_ueno has joined #openstack-infra | 01:02 | |
| *** nati_ueno has quit IRC | 01:02 | |
| *** nati_ueno has joined #openstack-infra | 01:04 | |
| zaro | fungi: it passed on precise25 | 01:04 |
| fungi | zaro: all set for me to trigger gearman-plugin-jenkinsci-upload for tat tag then? | 01:05 |
| zaro | fungi: that would be great. | 01:05 |
| *** nati_uen_ has joined #openstack-infra | 01:05 | |
| zaro | fungi: what should i do in this case? if anything. | 01:06 |
| fungi | zaro: success https://jenkins.openstack.org/job/gearman-plugin-jenkinsci-upload/4/console | 01:06 |
| fungi | zaro: well, it would be great if you had a theory, from the logs, as to why it failed consistently on precise5 (2 out of 2 times, for whatever that sample size is worth) | 01:07 |
| *** melwitt has joined #openstack-infra | 01:07 | |
| zaro | fungi: i could formulate a theory, but not much more. i can't repo so it would be difficult to prove and fix. | 01:08 |
| *** nati_ueno has quit IRC | 01:08 | |
| fungi | zaro: mainly wondering whether i should be suspecting the workspace, or the installed packages/files on the server | 01:09 |
| *** rnirmal has joined #openstack-infra | 01:10 | |
| *** senk has quit IRC | 01:12 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: create more sane logging for the er bot https://review.openstack.org/66435 | 01:12 |
| *** ^d has quit IRC | 01:13 | |
| *** nosnos has joined #openstack-infra | 01:13 | |
| zaro | fungi: my theory is that precise5 is maybe signficantly faster or slower than the other machines. i think it's timing related rather than bad file or something. | 01:13 |
| *** mrda has quit IRC | 01:14 | |
| clarkb | sdague fungi it is easy to determine a lag value for es with public data | 01:14 |
| fungi | oh, too weird | 01:14 |
| clarkb | sdague fungi every event has a received at field take the difference of that and the timestamp value | 01:15 |
| clarkb | note that since files are batch processed you only want that delta for the end of files | 01:15 |
| zaro | fungi: is there a history of gearman-plugin builds on precise5? | 01:15 |
| *** zhiwei has joined #openstack-infra | 01:15 | |
| zaro | clarkb: _david_ fixed the javamelody security flaw. | 01:16 |
| *** senk has joined #openstack-infra | 01:16 | |
| zaro | clarkb: new build is up on review-dev.o.o | 01:16 |
| clarkb | fungi you can also connect to localhost:4730 on logstash.o.o via tcp and run the 'status' command to see the queue depth which we should get into graphite soon | 01:16 |
| fungi | clarkb: oh, good point | 01:16 |
| clarkb | zaro: cool | 01:16 |
| *** rnirmal_ has joined #openstack-infra | 01:16 | |
| fungi | clarkb: and i probably knew about the 4730/tcp interface but should have taken better notes to remind myself | 01:17 |
| fungi | clarkb: i've turned up four worker processes each on 05 and 06 now, since the first four servers were spending a lot of time with cpu pegged | 01:17 |
| *** senk has quit IRC | 01:17 | |
| *** rnirmal has quit IRC | 01:17 | |
| *** rnirmal_ is now known as rnirmal | 01:17 | |
| *** vkozhukalov has joined #openstack-infra | 01:18 | |
| clarkb | fungi awesome make sure you restart logstash-indexer when you do that | 01:18 |
| clarkb | otherwise y2k | 01:18 |
| fungi | ahh, good to know. also adding to my notes | 01:18 |
| fungi | clarkb: which host is logstash-indexer on? | 01:20 |
| clarkb | fungi its on the individual worker hosts | 01:20 |
| clarkb | the 4 gearman processes talk to a single local logstash-indexer | 01:21 |
| *** jerryz has left #openstack-infra | 01:21 | |
| *** jerryz has joined #openstack-infra | 01:21 | |
| fungi | i'll keep hunting for the initscript then | 01:21 |
| clarkb | its an upstart job >_> | 01:21 |
| *** melwitt has quit IRC | 01:21 | |
| *** harlowja has quit IRC | 01:22 | |
| fungi | thanks! restarted | 01:22 |
| *** nati_uen_ has quit IRC | 01:22 | |
| *** marcoEmbrane has left #openstack-infra | 01:22 | |
| *** nati_ueno has joined #openstack-infra | 01:23 | |
| clarkb | np and thank you | 01:24 |
| *** sarob has joined #openstack-infra | 01:24 | |
| *** sarob_ has joined #openstack-infra | 01:24 | |
| fungi | push-log 16 16 24 | 01:25 |
| fungi | i'm sure that means something | 01:26 |
| * fungi reads | 01:26 | |
| *** sarob_ has quit IRC | 01:26 | |
| clarkb | first number is queue length second is active workers last is total workers | 01:26 |
| clarkb | so that looks good | 01:26 |
| *** sarob_ has joined #openstack-infra | 01:26 | |
| fungi | that makes sense, and those numbers look right | 01:27 |
| *** mrda has joined #openstack-infra | 01:27 | |
| fungi | at least the total workers number looks correct | 01:27 |
| fungi | with the addition of the 8 more earlier | 01:27 |
| fungi | zaro: we don't keep much build history on our jenkins masters, so all you get is https://jenkins01.openstack.org/view/All/job/gearman-plugin-hpi-artifact/buildTimeTrend unless you know somewhere else i should look | 01:28 |
| *** sarob has quit IRC | 01:28 | |
| *** praneshp has quit IRC | 01:29 | |
| fungi | clarkb: i tried a couple more new projects today and got the same symptoms, after confirming that the globally-installed jeepyb is new enough to include the hoped fix, so back to the drawing board | 01:29 |
| clarkb | fungi: for adding projects? | 01:30 |
| fungi | yep | 01:30 |
| *** kraman has quit IRC | 01:30 | |
| clarkb | :/ | 01:30 |
| fungi | clarkb: one of them used an existing acl, so no new groups were needed | 01:30 |
| fungi | clarkb: when puppet ran it, never even got as far as cloning into /var/lib/jeepyb | 01:30 |
| *** sarob_ has quit IRC | 01:30 | |
| fungi | clarkb: when i reran it myself, worked right off the bat | 01:31 |
| clarkb | maybe fixing each consequitive problem exposes a new bug? | 01:31 |
| fungi | same for one which was adding two new groups. failed under puppet, worked when i reran | 01:31 |
| fungi | it's possible | 01:31 |
| fungi | the one which had new groups to add never got as far as creating either group the first time around | 01:32 |
| fungi | also, i'm trying to beat pypi-mirror into submission for the requirements integration runs, but am running into some weird behavior with how it's populating the local mirror in /var/www on the slave | 01:33 |
| *** harlowja has joined #openstack-infra | 01:33 | |
| zaro | fungi: since that failing test has been there for a long time, i was hoping to see if any gearman-plugin builds have passed on precise5. that history didn't help and i don't know where else to look. | 01:34 |
| fungi | zaro: yeah, sorry. me either | 01:34 |
| clarkb | it is tempting to rewrite from scratch ... throw away the cruft and start again with what we have learned | 01:34 |
| *** gokrokve has quit IRC | 01:35 | |
| *** rakhmerov has quit IRC | 01:35 | |
| fungi | on the pypi-mirror integration runs, it hit an issue where a project wanted oslo.messaging, which doesn't have a release on public pypi yet. that should still be in the local mirror it created from all the git checkouts, but the mirror is only getting populated with release versions from pypi | 01:36 |
| *** gokrokve has joined #openstack-infra | 01:36 | |
| fungi | it's strange | 01:36 |
| *** pballand has quit IRC | 01:36 | |
| fungi | it's building wheels from the same versions too | 01:36 |
| *** gokrokve_ has joined #openstack-infra | 01:37 | |
| fungi | and md5sums of the tarballs match what's hosted on pypi, so it's downloading those, not building them from scratch | 01:38 |
| *** praneshp has joined #openstack-infra | 01:38 | |
| *** yaguang has joined #openstack-infra | 01:39 | |
| fungi | that may be the intended behavior, i have to read back through, but if so it's not working with the -f link to prerelease oslo.messaging on tarballs.o.o | 01:39 |
| clarkb | is the dev requirements list being used properly? | 01:39 |
| fungi | it's possible that's the issue | 01:40 |
| dstufft | Are y'all using -f in a requirements.txt that you're shuttling through a setup.py using pbr? | 01:40 |
| *** ryanpetrello has quit IRC | 01:40 | |
| *** senk has joined #openstack-infra | 01:40 | |
| *** gokrokve has quit IRC | 01:41 | |
| fungi | dstufft: until we get in good enough shape to upload prerelease wheels to pypi, i think | 01:41 |
| dstufft | fungi: are you using pip 1.5 or did you rollback to 1.4? | 01:41 |
| fungi | dstufft: in this specific case, i think it's using 1.5 but i need to dissect it a bit more to be able to tell | 01:41 |
| dstufft | if I recall pbr shuttled -f <tarball link> through a setup.py using dependency links | 01:42 |
| dstufft | which 1.5 disabled by default | 01:42 |
| dstufft | --process-dependency-links will turn them back on | 01:42 |
| *** gokrokve_ has quit IRC | 01:42 | |
| dstufft | if that's the case anyways | 01:42 |
| fungi | dstufft: oh, cool--thanks! i'll give that a shot | 01:42 |
| dstufft | or the envvar would be something like PIP_PROCESS_DEPENDENCY_LINKS=1 | 01:42 |
| fungi | dstufft: what version of pip added --process-dependency-links? | 01:43 |
| dstufft | fungi: 1.5 I think, it was a quick dprecation because of the security concern and the fact that most projects just didn't use them | 01:43 |
| fungi | dstufft: makes sense. envvar will be a saner solution in this case. thanks again | 01:44 |
| dstufft | pip 1.5, breaking all the things :[ | 01:44 |
| dstufft | fungi: np :) | 01:44 |
| dstufft | I think the last thing we have left that pip doesn't have locked down about as good as it can get, is setup_requires, need to figure out how to wrest control of those from setuptools still :/ | 01:45 |
| fungi | clarkb: a quick git grep through pypi-mirror says it has no reference at all to dev-requirements.txt | 01:45 |
| fungi | clarkb: which makes sense in the normal case... we don't want to be pulling those into our actual mirror | 01:46 |
| fungi | so i'm beginning to suspect it's as dstufft says, we're probably getting pip 1.5 there, the -f is being ignored in the individual projects' requirements.txt files, and so i may be able to patch up the calling environment in the integration test script to work around that | 01:48 |
| *** hogepodge has joined #openstack-infra | 01:48 | |
| *** dkliban has joined #openstack-infra | 01:48 | |
| sdague | fungi: honestly, we should shoot dev-requirements in the head | 01:48 |
| sdague | I really think it's the wrong approach | 01:48 |
| sdague | if people want prerelease, they go in the integrated gate | 01:49 |
| fungi | sdague: well, that's a big worry i have. we've manufactured another "solution" which now breaks until pip 1.5, so even if i work around it in the integration job, it's still broken for the rest of the world | 01:49 |
| fungi | er, under pip 1.5 | 01:49 |
| sdague | so if deleting the contents of dev requirements fixes things, we should do that | 01:49 |
| fungi | sdague: in this case, i can't install glance with pip 1.5 without passing an extra option so that the -f link to oslo.messaging on tarballs.o.o will be respected | 01:50 |
| sdague | fungi: so there is no real reason for that right | 01:50 |
| sdague | it's just that they have a totally old requirement | 01:51 |
| fungi | sdague: so we're talking about not just deleting dev-requirements.txt, but also all the -f links in the various projects requirements files | 01:51 |
| sdague | yep | 01:51 |
| sdague | this is just because glance was lazy and never updated it | 01:51 |
| sdague | that line predates havana release | 01:51 |
| fungi | okay, i'll get a patch proposed. how is anyone else using oslo.messaging though if it's not released yet? (and why have we not even reserved that one on pypi?) | 01:52 |
| sdague | fungi: actually, I'm working ont he glance patch | 01:53 |
| sdague | fungi: because it's in /opt/stack | 01:53 |
| fungi | sdague: okay, awesome. i'll leave you to it | 01:53 |
| dstufft | fungi: for the record I grumped about the use of dependency links in pbr back when Ifirst noticed it ;P | 01:54 |
| * dstufft will get all the packages to work how he wants them to :D | 01:54 | |
| fungi | dstufft: i believe it was added to support a use case we had and wanted to divest ourselves of. i don't think it was ever intended to be kept supported in there long-term (though i could be mistaken) | 01:54 |
| *** fifieldt has joined #openstack-infra | 01:55 | |
| sdague | fungi: though I think you might need to do some ninja merging | 01:56 |
| sdague | because I think that will fail on the requirements test | 01:56 |
| sdague | oh, wait... | 01:57 |
| fungi | sdague: what i'm working on is the requirements integration job which runs against the requirements project changes as well as pypi-mirror and pbr | 01:57 |
| sdague | yeh, but we won't be able to change a glance requirement | 01:58 |
| fungi | changes to glance should only just get checked that they use sane requirements entries | 01:58 |
| sdague | because it will fail the integration job | 01:58 |
| sdague | oh... maybe | 01:58 |
| fungi | unless we also need to change the openstack/requirements repository, it shouldn't be an issue | 01:58 |
| fungi | sdague: my bigger concern is that i may need to fix this simultaneously in openstack/requirements, openstack-infra/pypi-mirror and possibly also openstack-dev/pbr, depending on what else i hit while testing it | 01:58 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: remove oslo.messaging -f urls https://review.openstack.org/66474 | 02:00 |
| fungi | sdague: on that earlier item though, do you know of any reason i should *not* register oslo.messaging on pypi? it is intended to get released there, right? would be a shame for someone else to squat it | 02:00 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: remove oslo.messaging -f urls https://review.openstack.org/66474 | 02:02 |
| sdague | hmmm... yeh, and oslo.messaging was never released? | 02:03 |
| sdague | that seems wrong | 02:03 |
| fungi | seems wrong to me too. no harm in uploading prereleases to pypi until there's an actual initial release | 02:04 |
| sdague | any library that is olso.* should be release on pypi | 02:04 |
| fungi | it's not like you're going to get a pip 1.3 user accidentally "upgrading to a prerelease from a release version if there never was a release version before | 02:04 |
| *** ryanpetrello has joined #openstack-infra | 02:04 | |
| sdague | I'm so seriously confused that they had 12 1.2.0a releases, and are now doing 1.3.0a tags | 02:05 |
| sdague | and nothing to pypi | 02:05 |
| sdague | so we probably need dhellmann in the midst of all of this | 02:05 |
| sdague | because I incorrectly assumed we actually saw a 1.2.0 at havana | 02:05 |
| clarkb | ya no release | 02:06 |
| fungi | well, i've gone ahead and reserved it on pypi with a 0 version | 02:06 |
| sdague | and that we were not having core openstack things rely on unreleased libraries | 02:06 |
| fungi | so that we don't have to try to take it over from someone or rename later | 02:06 |
| sdague | yeh | 02:06 |
| fungi | but correct, the tags here never had a non-prerelease version http://git.openstack.org/cgit/openstack/oslo.messaging/refs/ | 02:07 |
| fungi | started at 1.2.0a1 and then skipped from 1.2.0a11 to 1.3.0a1 with no 1.2.0 ever existing | 02:08 |
| *** jhesketh has joined #openstack-infra | 02:08 | |
| sdague | yeh, so I actually don't think the rest of this is solvable until they release | 02:08 |
| sdague | not in any sane way | 02:08 |
| fungi | agreed | 02:08 |
| fungi | dead | 02:08 |
| fungi | in | 02:09 |
| fungi | water | 02:09 |
| sdague | enough so, that I'd say it's worth an email thread | 02:09 |
| sdague | the -f pre release urls never sat well with me anyway | 02:09 |
| sdague | so I'm glad to see them gone | 02:09 |
| fungi | and yes, your proposed change to glance will simply break for everyone, because there is no release of oslo.messaging | 02:09 |
| fungi | until they tag a non-prerelease version | 02:10 |
| zhiwei | fungi: hi | 02:10 |
| fungi | zhiwei: hi there | 02:10 |
| sdague | fungi: yep | 02:10 |
| sdague | it will actually pass requirements and tempest though :) | 02:11 |
| zhiwei | clarkb told me I should contact you about renaming stackforge project issue. | 02:11 |
| *** nati_uen_ has joined #openstack-infra | 02:11 | |
| fungi | zhiwei: right, i saw in scrollback, he said he'd be around on the 18th. i'll bring it up in tomorrow's team meeting and see whether there are any objections to doing it then | 02:11 |
| clarkb | fungi I can help with rename on saturday | 02:11 |
| *** thuc_ has joined #openstack-infra | 02:12 | |
| zhiwei | ok, thanks clark and fungi. | 02:12 |
| fungi | clarkb: works for me. i'll be here and i can do it. just don't want to risk it if i'm the only one around. appreciated! | 02:12 |
| sdague | clarkb: so any idea what the average lag time is on the logstashing? because I'm running about 26 minutes and not finding logs for jobs most of the time | 02:12 |
| *** sdake has joined #openstack-infra | 02:13 | |
| *** sdake has quit IRC | 02:13 | |
| *** sdake has joined #openstack-infra | 02:13 | |
| zhiwei | because I will have my Chinese New Year holiday from next Friday, so just want to get it done before holiday. | 02:13 |
| *** yamahata has quit IRC | 02:13 | |
| clarkb | sdague: I dont have an idea have you oooked at the received at field and compared to timestamp? | 02:14 |
| clarkb | sdague we will be graphiting queue length soon which will give us hard numbers | 02:14 |
| sdague | clarkb: so we honestly might be loosing things | 02:15 |
| *** nati_ueno has quit IRC | 02:15 | |
| fungi | sdague: http://logstash.openstack.org/#eyJzZWFyY2giOiIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6OTAwLCJncmFwaG1vZGUiOiJjb3VudCJ9 | 02:15 |
| sdague | clarkb: like I'm pretty sure - http://logstash.openstack.org/#eyJzZWFyY2giOiJmaWxlbmFtZTpcImNvbnNvbGUuaHRtbFwiIEFORCBtZXNzYWdlOlwiW1NDUF0gQ29weWluZyBjb25zb2xlIGxvZ1wiIEFORCBidWlsZF9zdGF0dXM6XCJGQUlMVVJFXCIgQU5EIGJ1aWxkX2NoYW5nZTo2NjM3NiBBTkQgYnVpbGRfcGF0Y2hzZXQ6MiBBTkQgYnVpbGRfbmFtZTpcImdhdGUtbm92YS1wZXA4XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzO | 02:15 |
| sdague | Dk2NjU2ODEwNDh9 | 02:15 |
| sdague | should exist | 02:15 |
| *** thuc has quit IRC | 02:15 | |
| clarkb | you are going to make me boot a proper machine :P | 02:15 |
| fungi | clarkb: filename:"console.html" AND message:"[SCP] Copying console log" AND build_status:"FAILURE" AND build_change:66376 AND build_patchset:2 AND build_name:"gate-nova-pep8" | 02:16 |
| *** yamahata has joined #openstack-infra | 02:16 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Add mailing list for OpenStack Ambassadors https://review.openstack.org/66478 | 02:16 |
| *** thuc_ has quit IRC | 02:17 | |
| fungi | clarkb: sdague do we load non-dsvm jobs into logstash? | 02:17 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9uYW1lOmdhdGUtaG9yaXpvbi1wZXA4IiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI4NjQwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODk2NjU3ODk4MjN9 | 02:17 |
| sdague | some times | 02:17 |
| fungi | when did we start putting unit test console logs in there? | 02:17 |
| *** esker has joined #openstack-infra | 02:17 | |
| sdague | I assumed we were doing it all the time | 02:17 |
| fungi | yeah, looks like we do | 02:18 |
| sdague | so if you tell me we only do it for gate and not check, that would be useful info to have | 02:18 |
| sdague | because I am assuming it's always around, and if it's not, that would explain some other things | 02:19 |
| clarkb | we have always put all console.html files in there | 02:19 |
| sdague | like right now I'm waiting for check-tempest-dsvm-full 66371,1 to show up | 02:19 |
| *** pballand has joined #openstack-infra | 02:19 | |
| sdague | clarkb: ok, great | 02:19 |
| sdague | so that pep8 console.html above should have gone in a couple hours ago | 02:19 |
| sdague | and it didn't | 02:20 |
| sdague | and there is a gate-horizon-pep8 I'm waiting on that the job ended 40 minutes ago | 02:20 |
| sdague | and the last gate-horizon-pep8 jobs are from yesterday | 02:21 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9uYW1lOmdhdGUtaG9yaXpvbi1wZXA4IiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI4NjQwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODk2NjU3ODk4MjN9 | 02:21 |
| clarkb | sdague: I am not follwing? which one? | 02:21 |
| lifeless | fungi: just ssling up the new cloud now | 02:21 |
| fungi | lifeless: exciting! | 02:21 |
| sdague | clarkb: 2014-01-13 20:52:19 WARNING [recheckwatchbot] Console logs not available after 0:26:49.363097s for gate-horizon-pep8 66465,1 | 02:21 |
| sdague | that was the fail that never showed up | 02:21 |
| sdague | or a fail that never showed up | 02:21 |
| sdague | when I see gate-horizon-pep8 jobs in logstash, they are indexed in 1s | 02:22 |
| *** jishaom has joined #openstack-infra | 02:22 | |
| sdague | comparing received_at vs. @timestamp | 02:22 |
| fungi | lifeless: is the auth-url going to need a change at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb#n199 ? | 02:23 |
| sdague | anyway, I'm being called out for the evening | 02:23 |
| lifeless | fungi: yah | 02:23 |
| sdague | so enjoy the rest of it | 02:23 |
| lifeless | ci-overcloud.tripleo.org , on https | 02:23 |
| clarkb | sdague: so first hunch is that that file isn't making it into the log archive quick enough and the logstash machinery is giving up on it | 02:23 |
| fungi | lifeless: okay, submitting a review for that. you said the username, password and project-id will stay the same though? | 02:23 |
| *** bknudson has quit IRC | 02:24 | |
| lifeless | fungi: I *think* you have a set in hiera or somewhere for cd-overcloud... | 02:24 |
| clarkb | sdague: or, when did the logstash client get restarted. its possible we missed things during that process restart. we can definitely make both cases more robust | 02:24 |
| fungi | lifeless: we do (the ruby variables on those lines above it are coming from hiera) | 02:24 |
| lifeless | fungi: ah, history - there are three clouds in question | 02:24 |
| lifeless | fungi: the poc, which had one-off creds | 02:25 |
| lifeless | fungi: cd-overcloud, which has automated creds | 02:25 |
| clarkb | can do a reload that fork and execs itself for new code while keeping sockets open | 02:25 |
| lifeless | fungi: and ci-overcloud which has those from cd-overcloud, I hope | 02:25 |
| lifeless | fungi: though right now, I'm failing to talk to it, which is worrying | 02:25 |
| clarkb | and can do more of an exponential backoff for logs if they don't become available | 02:25 |
| clarkb | fungi: ^ | 02:25 |
| fungi | lifeless: oh, okay. yeah we just have poc.tripleo.org in teh nodepool config, but i believe i remember adding the other creds of which you speak | 02:26 |
| lifeless | fungi: so the other creds, are the ones to duplicate | 02:26 |
| *** hogepodge has quit IRC | 02:26 | |
| fungi | lifeless: noted | 02:26 |
| fungi | clarkb: i'll get the timestamps | 02:26 |
| *** jasondotstar has joined #openstack-infra | 02:28 | |
| fungi | clarkb: sdague: Jan 13 20:56:11 logstash sudo: fungi : TTY=pts/1 ; PWD=/home/fungi ; USER=root ; COMMAND=/usr/sbin/invoke-rc.d jenkins-log-client restart | 02:29 |
| clarkb | hmm that is about half an hour after the test finished | 02:30 |
| clarkb | which makes the other thing a possibility | 02:30 |
| clarkb | the console log isn't being available before the timeout | 02:31 |
| clarkb | fungi: sdague: we can grep for te change number in all of the logstash worker logs to see if it timed out | 02:31 |
| clarkb | I need to start packing up soon. need to checkout in a bit. I can poke at this more when back in seattle | 02:32 |
| clarkb | fungi: sdague: but I do agree that things seem to be missing rather than late which is helpful because it narrows where we need to look | 02:32 |
| *** dkliban has quit IRC | 02:33 | |
| *** ArxCruz has quit IRC | 02:34 | |
| clarkb | the logs are quite verbose so we should be able to see what is going on. I am going to afk now for a bit | 02:39 |
| *** jhesketh has quit IRC | 02:39 | |
| *** pballand has quit IRC | 02:40 | |
| *** dkranz has quit IRC | 02:42 | |
| clarkb | oh quick thought before I go, the new scp plugin may have changed behavior in a subtle way that tickles a logstash bug or vice versa | 02:43 |
| *** dkranz has joined #openstack-infra | 02:43 | |
| clarkb | or | 02:43 |
| clarkb | new jenkins masters... | 02:43 |
| fungi | also possibilities | 02:45 |
| fungi | we do now have 5 production jenkins masters instead of 3, and 3 of those 5 are running a newer version of jenkins too | 02:46 |
| clarkb | looks like log archive returned a 404 | 02:50 |
| * clarkb will paste logs in a minute | 02:50 | |
| *** nati_uen_ has quit IRC | 02:50 | |
| clarkb | fungi: I think it is a new race exposed by new scp plugin | 02:51 |
| *** nati_ueno has joined #openstack-infra | 02:51 | |
| clarkb | fungi: before the file would always exist thoguh often in a 0 byte form | 02:51 |
| clarkb | now the file may not exist when logstash processes the events | 02:51 |
| fungi | ahh | 02:51 |
| fungi | so the processes need to wait and recheck i guess | 02:52 |
| clarkb | I lean toweards making the scp plugin a little less async | 02:52 |
| clarkb | http://paste.openstack.org/show/61156/ | 02:52 |
| *** coolsvap has quit IRC | 02:53 | |
| clarkb | fungi: but we can also tackle this in the worker if the retry flag is sset | 02:53 |
| clarkb | in any case, I think that isolates the problem | 02:53 |
| clarkb | and I am really going to afk now | 02:53 |
| fungi | l | 02:53 |
| fungi | k | 02:53 |
| openstackgerrit | Aaron Rosen proposed a change to openstack/requirements: Require httplib2>=0.7.5 https://review.openstack.org/66486 | 02:54 |
| clarkb | to solve in scp plugin we can have a sync point across both threads after the file has first been touched | 02:54 |
| clarkb | then allow the two threads to run completely independently | 02:54 |
| clarkb | zaro: ^ | 02:54 |
| *** krotscheck has quit IRC | 02:59 | |
| *** vipul is now known as vipul-away | 02:59 | |
| *** thomasem has joined #openstack-infra | 02:59 | |
| *** thomasem has quit IRC | 02:59 | |
| *** nati_ueno has quit IRC | 03:00 | |
| *** AaronGr is now known as AaronGr_Zzz | 03:02 | |
| *** jerryz has quit IRC | 03:09 | |
| *** pballand has joined #openstack-infra | 03:10 | |
| *** emagana has quit IRC | 03:14 | |
| *** dcramer_ has quit IRC | 03:14 | |
| *** tian has joined #openstack-infra | 03:16 | |
| *** vipul-away is now known as vipul | 03:19 | |
| lifeless | fungi: ok so I found a machine with dead fan | 03:19 |
| lifeless | fungi: redeploying now | 03:19 |
| fungi | lifeless: sounds fun. should we continue launching nodepool nodes in the poc cloud, or switch to using this one going forward? | 03:20 |
| lifeless | fungi: I deleted the POC cloud today | 03:20 |
| lifeless | fungi: it didn't have the right network setup at all | 03:20 |
| fungi | guess my question is answered | 03:20 |
| lifeless | fungi: the new cloud API endpoint is https://ci-overcloud.tripleo.org:13000/v2.0 | 03:21 |
| lifeless | fungi: it's not up yet, but your EOD is approaching | 03:21 |
| lifeless | fungi: so if you can copy the cd-overcloud creds to ci-overcloud, and that API endpoint | 03:21 |
| lifeless | fungi: it should all come together smoothly | 03:21 |
| lifeless | fungi: I have to run now to pick up C, I'll be back in a few hours when you should be asleep :) | 03:21 |
| lifeless | fungi: once it's up, I'm going to want to get you to allocate a floating IP in the cloud for the broker | 03:22 |
| *** senk has quit IRC | 03:22 | |
| fungi | lifeless: okay | 03:23 |
| fungi | and yeah, eod was more like 5 hours ago, but that doesn't seem to be stopping me | 03:27 |
| *** ^d has joined #openstack-infra | 03:28 | |
| *** pballand has quit IRC | 03:32 | |
| *** rnirmal has quit IRC | 03:33 | |
| *** jasondotstar has quit IRC | 03:34 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Update TripleO Cloud API endpoint for Nodepool https://review.openstack.org/66491 | 03:39 |
| fungi | lifeless: ^ (whenever you're back) | 03:42 |
| *** ^d is now known as ^demon|away | 03:43 | |
| *** jasondotstar has joined #openstack-infra | 03:46 | |
| *** ^demon|away has quit IRC | 03:51 | |
| *** senk has joined #openstack-infra | 03:57 | |
| *** slong has quit IRC | 03:57 | |
| *** jasondotstar has quit IRC | 04:00 | |
| *** weshay has quit IRC | 04:02 | |
| *** mrda has quit IRC | 04:07 | |
| *** mrda has joined #openstack-infra | 04:08 | |
| *** nati_ueno has joined #openstack-infra | 04:10 | |
| *** praneshp has quit IRC | 04:10 | |
| *** reed has quit IRC | 04:13 | |
| *** esker has quit IRC | 04:13 | |
| *** reed has joined #openstack-infra | 04:14 | |
| *** nati_ueno has quit IRC | 04:15 | |
| openstackgerrit | Jun Jie Nan proposed a change to openstack/requirements: Update python-keystoneclient to 0.4.2 https://review.openstack.org/66494 | 04:25 |
| *** noorul has quit IRC | 04:29 | |
| *** dkliban has joined #openstack-infra | 04:35 | |
| *** coolsvap has joined #openstack-infra | 04:39 | |
| *** slong has joined #openstack-infra | 04:49 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:52 | |
| *** slong has quit IRC | 04:53 | |
| *** nati_ueno has joined #openstack-infra | 04:53 | |
| *** nati_ueno has quit IRC | 04:55 | |
| *** nati_ueno has joined #openstack-infra | 04:55 | |
| *** praneshp has joined #openstack-infra | 04:56 | |
| *** dstufft has quit IRC | 05:00 | |
| *** slong has joined #openstack-infra | 05:05 | |
| *** dcramer_ has joined #openstack-infra | 05:05 | |
| *** dcramer__ has joined #openstack-infra | 05:10 | |
| *** dcramer_ has quit IRC | 05:12 | |
| *** praneshp_ has joined #openstack-infra | 05:18 | |
| *** pballand has joined #openstack-infra | 05:19 | |
| *** praneshp has quit IRC | 05:20 | |
| *** praneshp_ is now known as praneshp | 05:20 | |
| *** chandankumar has joined #openstack-infra | 05:24 | |
| *** erfanian has quit IRC | 05:24 | |
| *** pballand has quit IRC | 05:39 | |
| *** nati_uen_ has joined #openstack-infra | 05:49 | |
| *** nati_uen_ has quit IRC | 05:50 | |
| *** nati_uen_ has joined #openstack-infra | 05:50 | |
| *** nati_ueno has quit IRC | 05:53 | |
| *** mrda has quit IRC | 06:00 | |
| *** dkranz has quit IRC | 06:03 | |
| *** dkranz has joined #openstack-infra | 06:03 | |
| *** denis_makogon has joined #openstack-infra | 06:06 | |
| *** coolsvap has quit IRC | 06:07 | |
| *** dcramer__ has quit IRC | 06:08 | |
| *** dkliban has quit IRC | 06:09 | |
| *** SergeyLukjanov has quit IRC | 06:12 | |
| *** dstanek has quit IRC | 06:13 | |
| *** dkliban has joined #openstack-infra | 06:14 | |
| *** pelix has left #openstack-infra | 06:17 | |
| *** boris-42 has joined #openstack-infra | 06:18 | |
| *** qs201 has joined #openstack-infra | 06:22 | |
| *** qs201 has quit IRC | 06:23 | |
| *** senk has quit IRC | 06:26 | |
| *** senk has joined #openstack-infra | 06:27 | |
| mordred | sdague, fungi: re: oslo.messaging - once we start publishing wheels, we're ok and can deal with it better | 06:29 |
| mordred | sdague, fungi: it's myu fault that we're not yet | 06:29 |
| *** dkliban has quit IRC | 06:29 | |
| clarkb | mordred you are up late | 06:31 |
| *** coolsvap has joined #openstack-infra | 06:32 | |
| mordred | clarkb: it's only 10:30 | 06:34 |
| mordred | clarkb: where are you? | 06:34 |
| mordred | wow. this is backwards | 06:34 |
| clarkb | oh right you are west coast now | 06:34 |
| clarkb | mordred still in perth flying back tonight | 06:34 |
| clarkb | visiting the museum of western australia currently | 06:35 |
| mordred | clarkb: nice. oz good? | 06:35 |
| mordred | woot | 06:35 |
| clarkb | ya the weather finally cooled | 06:35 |
| *** reed has quit IRC | 06:37 | |
| *** nati_ueno has joined #openstack-infra | 06:38 | |
| *** nati_uen_ has quit IRC | 06:41 | |
| *** zhiwei has left #openstack-infra | 06:45 | |
| *** gokrokve has joined #openstack-infra | 06:47 | |
| *** dkliban has joined #openstack-infra | 06:51 | |
| *** gokrokve has quit IRC | 06:53 | |
| *** vkozhukalov has quit IRC | 06:53 | |
| lifeless | fungi: ok way to late, but the API is up on SSL | 06:55 |
| *** slong has quit IRC | 06:55 | |
| *** marun has quit IRC | 07:01 | |
| *** boris-42 has quit IRC | 07:04 | |
| *** DinaBelova has joined #openstack-infra | 07:04 | |
| *** senk has quit IRC | 07:05 | |
| *** dkliban has quit IRC | 07:05 | |
| lifeless | clarkb: are you perchance around ? | 07:06 |
| *** senk has joined #openstack-infra | 07:06 | |
| *** jamielennox is now known as jamielennox|away | 07:07 | |
| *** yolanda has joined #openstack-infra | 07:12 | |
| *** tian has quit IRC | 07:20 | |
| *** DinaBelova has quit IRC | 07:22 | |
| clarkb | lifeless: sorry no. museuming | 07:22 |
| *** nati_uen_ has joined #openstack-infra | 07:26 | |
| *** nati_uen_ has quit IRC | 07:26 | |
| *** nati_uen_ has joined #openstack-infra | 07:27 | |
| *** tian has joined #openstack-infra | 07:28 | |
| *** nati_ueno has quit IRC | 07:30 | |
| *** DinaBelova has joined #openstack-infra | 07:34 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:35 | |
| *** praneshp has quit IRC | 07:38 | |
| *** jcoufal has quit IRC | 07:48 | |
| *** gokrokve has joined #openstack-infra | 07:48 | |
| *** gokrokve has quit IRC | 07:52 | |
| *** harlowja is now known as harlowja_away | 08:00 | |
| *** jishaom has quit IRC | 08:04 | |
| *** jcoufal has joined #openstack-infra | 08:08 | |
| fifieldt | russellbryant.net appears to be down! | 08:09 |
| * fifieldt craves reviewstats | 08:09 | |
| *** bauzas has joined #openstack-infra | 08:12 | |
| *** fbo_away is now known as fbo | 08:20 | |
| *** nati_ueno has joined #openstack-infra | 08:21 | |
| *** nati_uen_ has quit IRC | 08:24 | |
| *** nati_ueno has quit IRC | 08:25 | |
| *** nati_ueno has joined #openstack-infra | 08:25 | |
| *** fbo is now known as fbo_away | 08:26 | |
| *** jishaom has joined #openstack-infra | 08:27 | |
| *** vkozhukalov has joined #openstack-infra | 08:27 | |
| *** flaper87|afk is now known as flaper87 | 08:36 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 08:42 |
| *** denis_makogon has quit IRC | 08:42 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 08:43 |
| *** pblaho has joined #openstack-infra | 08:46 | |
| *** gokrokve has joined #openstack-infra | 08:48 | |
| *** ruhe has joined #openstack-infra | 08:49 | |
| *** _ruhe has joined #openstack-infra | 08:53 | |
| *** gokrokve has quit IRC | 08:53 | |
| *** jasondotstar has joined #openstack-infra | 08:53 | |
| *** ruhe has quit IRC | 08:53 | |
| *** che-arne has quit IRC | 08:53 | |
| *** _ruhe is now known as ruhe | 08:54 | |
| *** jpich has joined #openstack-infra | 09:00 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Add manuals-upstream-translation-update to api-site https://review.openstack.org/66519 | 09:04 |
| *** coolsvap has quit IRC | 09:05 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 09:07 |
| *** NikitaKonovalov has joined #openstack-infra | 09:07 | |
| *** jishaom has quit IRC | 09:07 | |
| *** emagana has joined #openstack-infra | 09:08 | |
| *** mkerrin has joined #openstack-infra | 09:08 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add project for savanna-ci configs https://review.openstack.org/66521 | 09:10 |
| *** yassine has joined #openstack-infra | 09:11 | |
| *** dizquierdo has joined #openstack-infra | 09:12 | |
| *** emagana has quit IRC | 09:13 | |
| *** mozawa has quit IRC | 09:13 | |
| *** fbo_away is now known as fbo | 09:18 | |
| *** nati_uen_ has joined #openstack-infra | 09:20 | |
| *** mancdaz_away is now known as mancdaz | 09:20 | |
| *** mancdaz is now known as mancdaz_away | 09:21 | |
| *** nati_ueno has quit IRC | 09:23 | |
| *** dpyzhov has joined #openstack-infra | 09:24 | |
| *** afazekas has joined #openstack-infra | 09:24 | |
| BobBallAway | lifeless: great - thanks. That's what I wanted to know :) | 09:29 |
| *** BobBallAway is now known as BobBall | 09:29 | |
| *** dpyzhov has quit IRC | 09:29 | |
| *** derekh has joined #openstack-infra | 09:36 | |
| *** fbo is now known as fbo_away | 09:36 | |
| *** coolsvap has joined #openstack-infra | 09:38 | |
| *** dpyzhov has joined #openstack-infra | 09:39 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 09:39 |
| *** ruhe is now known as ruhe_away | 09:41 | |
| *** ruhe_away is now known as ruhe | 09:41 | |
| *** fbo_away is now known as fbo | 09:41 | |
| *** ruhe is now known as ruhe_away | 09:42 | |
| *** mancdaz_away is now known as mancdaz | 09:45 | |
| *** gokrokve has joined #openstack-infra | 09:48 | |
| *** markmc has joined #openstack-infra | 09:48 | |
| *** ruhe_away is now known as ruhe | 09:49 | |
| *** gokrokve has quit IRC | 09:53 | |
| *** ruhe is now known as ruhe_away | 10:00 | |
| *** ruhe_away is now known as ruhe | 10:01 | |
| *** dpyzhov has quit IRC | 10:06 | |
| *** flaper87 is now known as flaper87|afk | 10:06 | |
| *** dpyzhov has joined #openstack-infra | 10:06 | |
| *** nati_ueno has joined #openstack-infra | 10:07 | |
| *** nati_uen_ has quit IRC | 10:11 | |
| *** nati_ueno has quit IRC | 10:13 | |
| *** nati_ueno has joined #openstack-infra | 10:13 | |
| *** flaper87|afk is now known as flaper87 | 10:13 | |
| *** NikitaKonovalov has quit IRC | 10:24 | |
| *** NikitaKonovalov has joined #openstack-infra | 10:27 | |
| *** coolsvap has quit IRC | 10:27 | |
| *** ruhe is now known as ruhe_away | 10:31 | |
| *** lcestari has joined #openstack-infra | 10:31 | |
| *** mrda has joined #openstack-infra | 10:35 | |
| *** ruhe_away is now known as _ruhe | 10:41 | |
| *** gokrokve has joined #openstack-infra | 10:48 | |
| *** gokrokve has quit IRC | 10:52 | |
| *** dpyzhov has quit IRC | 10:52 | |
| *** dpyzhov has joined #openstack-infra | 10:59 | |
| *** nati_uen_ has joined #openstack-infra | 11:01 | |
| mkoderer | lifeless: ping | 11:03 |
| *** nati_ueno has quit IRC | 11:05 | |
| *** boris-42 has joined #openstack-infra | 11:07 | |
| *** mrda has quit IRC | 11:07 | |
| *** _ruhe is now known as ruhe | 11:11 | |
| *** flaper87 is now known as flaper87|afk | 11:16 | |
| *** andreaf has joined #openstack-infra | 11:19 | |
| *** ArxCruz has joined #openstack-infra | 11:23 | |
| *** flaper87|afk is now known as flaper87 | 11:26 | |
| *** yaguang has quit IRC | 11:29 | |
| *** nprivalova has quit IRC | 11:35 | |
| *** ildikov_ has joined #openstack-infra | 11:41 | |
| *** amotoki_ has quit IRC | 11:46 | |
| *** jcoufal has quit IRC | 11:47 | |
| *** gokrokve has joined #openstack-infra | 11:48 | |
| *** hashar has joined #openstack-infra | 11:52 | |
| *** gokrokve has quit IRC | 11:53 | |
| *** jcoufal has joined #openstack-infra | 11:54 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file https://review.openstack.org/66280 | 11:55 |
| *** hashar has quit IRC | 11:58 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Get rid of Django https://review.openstack.org/66016 | 11:59 |
| *** rfolco has joined #openstack-infra | 11:59 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: [WIP] Add tests for Alembic migrations https://review.openstack.org/66414 | 12:00 |
| *** yamahata has quit IRC | 12:01 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Load projects from yaml file https://review.openstack.org/66280 | 12:07 |
| *** nati_uen_ has quit IRC | 12:12 | |
| *** dstufft has joined #openstack-infra | 12:12 | |
| *** HenryG_ has quit IRC | 12:13 | |
| openstackgerrit | Alexandre Levine proposed a change to openstack-infra/config: Adding empty gce-api project to stackforge https://review.openstack.org/61954 | 12:13 |
| *** mfink has quit IRC | 12:15 | |
| *** nati_ueno has joined #openstack-infra | 12:17 | |
| *** SergeyLukjanov has quit IRC | 12:18 | |
| openstackgerrit | Eric Guo proposed a change to openstack/requirements: Whitelist external netifaces requirement https://review.openstack.org/66549 | 12:19 |
| *** SergeyLukjanov has joined #openstack-infra | 12:19 | |
| ttx | fungi: gate apparently blocked on queueing | 12:20 |
| ttx | https://bugs.launchpad.net/openstack-ci/+bug/1268892 | 12:20 |
| uvirtbot | Launchpad bug 1268892 in openstack-ci "Zuul is stuck - not starting a lot of jobs in check/gate" [Undecided,New] | 12:20 |
| ttx | lacking slaves | 12:21 |
| *** iv_m_ has joined #openstack-infra | 12:21 | |
| sdague | ttx: yeh, I think node pool exploded | 12:23 |
| *** coolsvap has joined #openstack-infra | 12:24 | |
| sdague | I never gathered if clarkb was back in the states today | 12:24 |
| sdague | I think jeblair was flying today | 12:24 |
| *** dstanek has joined #openstack-infra | 12:25 | |
| *** dstanek is now known as dstanekafk | 12:27 | |
| *** dstanekafk is now known as dstanek | 12:28 | |
| ttx | #status notice Gate currently blocked due to slave node exhaustion | 12:29 |
| openstackstatus | NOTICE: Gate currently blocked due to slave node exhaustion | 12:29 |
| fifieldt | ouch | 12:31 |
| ttx | sdague: don't think there is much we can do except praying fungi will get up early | 12:31 |
| sdague | ttx: you need to make sure that jonathan realizes how much fungi has been holding down the fort this week. | 12:31 |
| ttx | sdague: I'll do my best to convey that. | 12:31 |
| ttx | too bad we got so low on Monday and now we've piled up a queue again | 12:32 |
| * fifieldt will also convey this | 12:32 | |
| ttx | sdague: given when the starvation started, it looks like we could use some alerting | 12:35 |
| ttx | because i'm pretty sure we had infra-core around when it actually started | 12:35 |
| sdague | could be | 12:35 |
| ttx | (looking at the test nodes graph at the bottom of zuul status) | 12:35 |
| EmilienM | sdague: Hi Sean, I need to go ahead with Tempest / Neutron tests that I'm doing https://review.openstack.org/#/c/66143/, but I face to a blocking patch for devstack-gate. Could have a look when you can of course ? https://review.openstack.org/#/c/66142/ | 12:35 |
| sdague | EmilienM: so I don't yet have d-g authority | 12:37 |
| EmilienM | sdague: oh, my bad. | 12:37 |
| sdague | ttx: yeh, could be. | 12:37 |
| *** sandywalsh has quit IRC | 12:38 | |
| *** heyongli has joined #openstack-infra | 12:39 | |
| *** dpyzhov has quit IRC | 12:39 | |
| *** jasondotstar has quit IRC | 12:42 | |
| *** ruhe is now known as ruhe_away | 12:42 | |
| *** mozawa has joined #openstack-infra | 12:45 | |
| *** coolsvap has quit IRC | 12:46 | |
| *** portante has quit IRC | 12:46 | |
| *** gokrokve has joined #openstack-infra | 12:48 | |
| *** yamahata has joined #openstack-infra | 12:48 | |
| *** sandywalsh has joined #openstack-infra | 12:51 | |
| *** dpyzhov has joined #openstack-infra | 12:51 | |
| *** ruhe_away is now known as ruhe | 12:51 | |
| *** SergeyLukjanov has quit IRC | 12:53 | |
| *** gokrokve has quit IRC | 12:53 | |
| *** SergeyLukjanov has joined #openstack-infra | 12:54 | |
| *** nati_ueno has quit IRC | 12:54 | |
| *** yaguang has joined #openstack-infra | 12:57 | |
| *** smarcet has joined #openstack-infra | 13:00 | |
| *** bingbu has joined #openstack-infra | 13:06 | |
| *** mkerrin has quit IRC | 13:07 | |
| *** afazekas has quit IRC | 13:07 | |
| *** yaguang has quit IRC | 13:09 | |
| *** yaguang has joined #openstack-infra | 13:09 | |
| *** bingbu has quit IRC | 13:12 | |
| *** afazekas has joined #openstack-infra | 13:19 | |
| *** esker has joined #openstack-infra | 13:20 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 13:20 |
| *** ruhe is now known as ruhe_away | 13:23 | |
| *** yamahata has quit IRC | 13:23 | |
| *** yamahata has joined #openstack-infra | 13:23 | |
| *** jishaom has joined #openstack-infra | 13:24 | |
| *** esker has quit IRC | 13:24 | |
| *** ruhe_away is now known as _ruhe | 13:31 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: move to static LOG https://review.openstack.org/66564 | 13:32 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: expose on channel when we timeout on logs https://review.openstack.org/66565 | 13:32 |
| *** dstanek has quit IRC | 13:32 | |
| *** _ruhe is now known as ruhe | 13:32 | |
| *** jishaom has quit IRC | 13:34 | |
| vkozhukalov | hey guys, is it ok to use libvirt python API inside tempest scenarios? I ask because here http://docs.openstack.org/developer/tempest/overview.html is written that I am not allowed to hit hypervisors directly, but using libvirt is not hitting hypervisors directly. I need to create several tiny VMs in my testing scenario (ironic functional testing). | 13:38 |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 13:40 | |
| *** pblaho has quit IRC | 13:40 | |
| *** _SergeyLukjanov has quit IRC | 13:40 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations https://review.openstack.org/66414 | 13:41 |
| *** mfink has joined #openstack-infra | 13:42 | |
| *** hashar has joined #openstack-infra | 13:43 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations https://review.openstack.org/66414 | 13:46 |
| *** gokrokve has joined #openstack-infra | 13:48 | |
| ttx | fungi: for when you get up, uregtn action needed on slave pool exhaution blocking gate | 13:48 |
| *** weshay has joined #openstack-infra | 13:51 | |
| *** miqui has joined #openstack-infra | 13:53 | |
| *** gokrokve has quit IRC | 13:53 | |
| *** afazekas has quit IRC | 13:57 | |
| *** jasondotstar has joined #openstack-infra | 13:58 | |
| *** dkliban has joined #openstack-infra | 14:01 | |
| *** flaper87 has quit IRC | 14:01 | |
| *** iv_m_ has quit IRC | 14:01 | |
| *** thomasem has joined #openstack-infra | 14:03 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: [Don not review] Add tests for Alembic migrations https://review.openstack.org/66414 | 14:03 |
| *** dprince has joined #openstack-infra | 14:04 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: [Don not review] Add tests for Alembic migrations https://review.openstack.org/66414 | 14:05 |
| *** heyongli has quit IRC | 14:05 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:06 | |
| *** dkranz has quit IRC | 14:08 | |
| *** changbl has quit IRC | 14:09 | |
| *** thuc has joined #openstack-infra | 14:10 | |
| fungi | ttx: looking | 14:11 |
| fungi | wow, that's some graph | 14:12 |
| sdague | fungi: yep | 14:12 |
| ttx | fun heh | 14:12 |
| *** dkliban has quit IRC | 14:14 | |
| *** afazekas has joined #openstack-infra | 14:15 | |
| *** max_lobur has joined #openstack-infra | 14:15 | |
| *** ArxCruz has quit IRC | 14:15 | |
| fungi | looks like it's a bug in nodepool | 14:17 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations https://review.openstack.org/66414 | 14:17 |
| fungi | i'm collecting some state information before i shut it down, comment out the affected provider and start it back again | 14:17 |
| SergeyLukjanov | fungi, morning | 14:17 |
| *** ArxCruz has joined #openstack-infra | 14:19 | |
| *** dims has quit IRC | 14:22 | |
| *** julim has joined #openstack-infra | 14:22 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: webapp: set cache-control headers to prevent caching https://review.openstack.org/66583 | 14:22 |
| fungi | okay, nodepool thread dump captured, puppet agent stopped, config file altered and daemon started cleanly again. it claims to be launching new nodes now, so should hopefully have some picking up jobs in 10-15 minutes | 14:23 |
| *** esker has joined #openstack-infra | 14:23 | |
| fungi | good afternoon, SergeyLukjanov | 14:23 |
| *** dims has joined #openstack-infra | 14:23 | |
| hashar | good morning | 14:24 |
| fungi | good afternoon, hashar | 14:24 |
| hashar | if you feel like reviewing some HTTP headers for Zuul web app : https://review.openstack.org/#/c/66583/ :-D | 14:24 |
| *** tma996 has joined #openstack-infra | 14:25 | |
| fungi | hashar: i would love to do that. we'll see how far i get dealing with broken things first | 14:25 |
| hashar | it doesn't send cache-controls which would cause the status.json to be cached by Varnish :-] | 14:25 |
| hashar | ah take your time, that is definitely not urgent | 14:25 |
| *** saschpe has quit IRC | 14:25 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Use venv to build documentation https://review.openstack.org/65987 | 14:27 |
| *** afazekas has quit IRC | 14:28 | |
| *** saschpe has joined #openstack-infra | 14:29 | |
| *** thuc has quit IRC | 14:29 | |
| *** flaper87|afk has joined #openstack-infra | 14:29 | |
| *** thuc has joined #openstack-infra | 14:30 | |
| *** flaper87|afk is now known as flaper87 | 14:30 | |
| *** mrodden has joined #openstack-infra | 14:30 | |
| *** oubiwann has joined #openstack-infra | 14:31 | |
| *** saschpe has quit IRC | 14:31 | |
| *** saschpe has joined #openstack-infra | 14:31 | |
| *** flaper87 has quit IRC | 14:32 | |
| *** flaper87 has joined #openstack-infra | 14:32 | |
| *** thuc has quit IRC | 14:34 | |
| fungi | the nodepool issue is being tracked in bug 1269001 | 14:35 |
| uvirtbot | Launchpad bug 1269001 in nodepool "Nodepool stops building any new nodes when one provider is down" [High,Triaged] https://launchpad.net/bugs/1269001 | 14:35 |
| sdague | fungi: man, that nodepool graph is a thing :) | 14:38 |
| fungi | thing and a half | 14:39 |
| *** bknudson has joined #openstack-infra | 14:39 | |
| sdague | so, unrelated, we are definitely completely loosing logs going into ES | 14:39 |
| sdague | which I'll hound clarkb about when he's back | 14:39 |
| *** burt has joined #openstack-infra | 14:40 | |
| fungi | sdague: yes, did you catch clarkb's comments on that? he found tracebacks suggesting that the jenkins master signalled the log was ready, but returned a 404 when an attempt was made to retrieve it. most likely a race in the jenkins plugin | 14:41 |
| sdague | oh, interesting | 14:41 |
| sdague | no, missed that | 14:41 |
| fungi | probably introduced/exposed with the threading fixes we added to the plugin so it would work on newer jenkins | 14:41 |
| sdague | cool | 14:42 |
| *** mfer has joined #openstack-infra | 14:42 | |
| sdague | well that explains some things | 14:42 |
| fungi | so it's likely been happening since the upgrade to jenkins 1.5.43 a couple weeks back | 14:42 |
| sdague | yeh | 14:42 |
| fungi | er, 1.543, whatever | 14:42 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/gear: inline helps for interactive commands https://review.openstack.org/66591 | 14:43 |
| fungi | he was going to get zaro to look into adding a sync point between the threads for that | 14:43 |
| sdague | cool | 14:44 |
| sdague | well at least having the problem uncovered means we can address it | 14:44 |
| *** vogxn has joined #openstack-infra | 14:44 | |
| sdague | the newest code I wrote for the bot will also log those timeouts in channel | 14:44 |
| *** afazekas has joined #openstack-infra | 14:44 | |
| *** dkliban has joined #openstack-infra | 14:44 | |
| sdague | I need to get jog0 and mtreinish to review through all that code | 14:45 |
| sdague | but it should make things a ton more reliable | 14:45 |
| *** senk has quit IRC | 14:46 | |
| * hashar curses Mac OS python for not including poll() :-( | 14:48 | |
| *** gokrokve has joined #openstack-infra | 14:48 | |
| dhellmann | sdague, fungi : yes, PLEASE reserve oslo.messaging on pypi | 14:49 |
| fungi | hashar: curse apple's disdain for some free software licenses | 14:49 |
| hashar | well it is BSD based so we might use kqueue() :D | 14:50 |
| fungi | dhellmann: i did last night, so it won't get squatted (or rather, openstack is now squatting it) | 14:50 |
| dhellmann | fungi: thanks | 14:50 |
| *** eharney has joined #openstack-infra | 14:50 | |
| *** nosnos has quit IRC | 14:50 | |
| dhellmann | fungi: I think we were assuming the "oslo" prefix would protect us | 14:50 |
| fungi | dhellmann: mordred: also, the lack of an oslo.messaging release seems to be my current blocker for getting the requirements integration job working, since pip 1.5 no longer follows -f urls without additional prompting | 14:51 |
| *** vogxn has quit IRC | 14:51 | |
| fungi | well, without passing an additional option/envvar | 14:52 |
| markmc | <markmc> dhellmann, do we have the publish-beta-releases-to-pypi thing yet? | 14:52 |
| fungi | so anyone installing, for example, glance from source using 'pip install .' is currently going to break on being unable to find oslo.messaging on pypi unless they know to add the right magic incantation | 14:53 |
| dhellmann | markmc: do you still consider messaging beta? | 14:53 |
| markmc | dhellmann, yes | 14:53 |
| dhellmann | can we just call it 0.1? | 14:53 |
| *** gokrokve has quit IRC | 14:53 | |
| *** senk has joined #openstack-infra | 14:53 | |
| markmc | dhellmann, I wouldn't be very keen on going backwards in release numbers | 14:54 |
| * dhellmann looks at the current version # | 14:54 | |
| dhellmann | oh, right | 14:54 |
| fungi | i'm going to continue trying to work around it in the pypi-mirror script for now, but just be aware that pip 1.5 is unfond of -f urls | 14:55 |
| markmc | fungi, ok, thanks | 14:55 |
| markmc | fungi, mordred had a plan for this, just don't know what the status is | 14:55 |
| sdague | realistically, it's going to really impact other users as well because of pip 1.5 behavior | 14:55 |
| dhellmann | we were going to start building wheels | 14:55 |
| fungi | my pleasure. wanted to be sure someone knew the urgency crept upward on that | 14:55 |
| *** chandankumar has quit IRC | 14:55 | |
| markmc | the wheels thing was just to avoid affecting <1.4 users | 14:56 |
| markmc | doesn't apply to oslo.messaging, really | 14:56 |
| dhellmann | yeah, but that was how we were going to do alpha and beta releases | 14:56 |
| dhellmann | so everything was consistent | 14:56 |
| dhellmann | what's standing in the way of calling 1.3 done? | 14:56 |
| markmc | all the outstanding reviews and time for things to settle down | 14:57 |
| sdague | or release 1.2.0 off the last 1.2.0a | 14:57 |
| sdague | then let out 1.3 when it's ready | 14:57 |
| markmc | sdague, 1.2.0a wasn't api stable | 14:57 |
| *** dkliban has quit IRC | 14:57 | |
| dhellmann | yeah, I don't think we want people using that one | 14:57 |
| sdague | dhellmann: well the already are | 14:58 |
| sdague | glance users are using it in the field today if they are CDing | 14:58 |
| *** senk has quit IRC | 14:58 | |
| sdague | so the cats out of the ba | 14:58 |
| sdague | bag | 14:58 |
| markmc | icehouse only | 14:58 |
| sdague | markmc: yeh, so HP & Rax clouds | 14:58 |
| sdague | no one important :) | 14:58 |
| *** senk has joined #openstack-infra | 14:59 | |
| markmc | I don't want API unstable stuff on pypi if it isn't marked as beta | 14:59 |
| markmc | we can do that, even if we can't publish wheels yet | 14:59 |
| *** dolphm has left #openstack-infra | 14:59 | |
| markmc | i.e. wheel vs beta-pypi-publishing is mostly orthogonal | 15:00 |
| *** sarob has joined #openstack-infra | 15:00 | |
| *** sarob has quit IRC | 15:01 | |
| fungi | yes, we can for example add a pypi-upload job for it in the pre-release pipeline for now, then strip that out at or immediately following the first stable release | 15:02 |
| *** sarob has joined #openstack-infra | 15:02 | |
| markmc | fungi, can pypi-upload mark it as beta | 15:02 |
| markmc | fungi, I forget how that works in pypi ? | 15:02 |
| dstufft | dhellmann: you know i've debated making it possible for a project to request registration of a namespace like oslo.* | 15:02 |
| fungi | the risk is in pip-1.3 and earlier unexpectedly upgrading someone from a stable release to a newer pre-release, but if there's been no first stable release it's moot | 15:02 |
| markmc | fungi, or maybe it's not a pypi metadata thing, but just how pip interprets version numbers ? | 15:02 |
| sdague | markmc: so we're going to need another work around for that as well, right | 15:02 |
| sdague | because pip 1.4 doesn't install beta levels by default now | 15:03 |
| *** sdake has quit IRC | 15:03 | |
| *** sandywalsh has quit IRC | 15:03 | |
| fungi | markmc: use a beta version string (like it's doing now) instead of a release one. there's also a trove classifier for project maturity, though it's probably completely ignored for any real purpose | 15:03 |
| *** coolsvap has joined #openstack-infra | 15:04 | |
| markmc | sdague, it does if you ask for e.g. >=1.3.0a2 AFAIR | 15:04 |
| markmc | sdague, i.e. you explicitly ask for a beta version | 15:04 |
| fungi | markmc: right | 15:04 |
| *** vkozhukalov has quit IRC | 15:04 | |
| sdague | ok | 15:04 |
| markmc | fungi, ok, I'm up for that | 15:04 |
| *** prad has joined #openstack-infra | 15:04 | |
| *** ruhe is now known as ruhe_away | 15:05 | |
| fungi | so really, it's just as simple as us adding a few more lines to zuul's layout.yaml so that for now, the alpha/beta releases of oslo.messaging end up on pypi and then in the future we remove that upload job for pre-release tags when it's time for the first real release | 15:05 |
| fungi | getting mordred's buy off on that as a stop-gap while he's working on the last bits of the wheels stuff would be good before i go forward with it, but i don't see any reason why it wouldn't work | 15:06 |
| markmc | fungi, yep | 15:06 |
| markmc | fungi, I'll take a look at the infra config change | 15:06 |
| *** dkranz has joined #openstack-infra | 15:07 | |
| fungi | awesome. thanks markmc | 15:07 |
| markmc | fungi, thank you | 15:07 |
| dstufft | fungi: yes the classifer is completely ignored except for display in the ui | 15:08 |
| dhellmann | dstufft: that would be a nice feature | 15:08 |
| *** ruhe_away is now known as _ruhe | 15:11 | |
| *** _ruhe is now known as ruhe | 15:12 | |
| *** dpyzhov has quit IRC | 15:12 | |
| *** rakhmerov has joined #openstack-infra | 15:14 | |
| *** sdake has joined #openstack-infra | 15:15 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: expose on channel when we timeout on logs https://review.openstack.org/66565 | 15:15 |
| *** sarob has quit IRC | 15:17 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: expose on channel when we timeout on logs https://review.openstack.org/66565 | 15:18 |
| *** herndon_ has joined #openstack-infra | 15:19 | |
| *** sandywalsh has joined #openstack-infra | 15:20 | |
| *** dcramer__ has joined #openstack-infra | 15:20 | |
| *** dpyzhov has joined #openstack-infra | 15:21 | |
| *** praneshp has joined #openstack-infra | 15:22 | |
| mfer | fungi did you hear back from the foundation on using openstack in a stackforge name? | 15:23 |
| mordred | fungi: what did I do? | 15:23 |
| mordred | mfer: tell me the openstack/stackforge question | 15:23 |
| fungi | mfer: staff meeting is in ~30 minutes so i plan to get some confirmation | 15:23 |
| fungi | but with mordred around he can probably also help on that | 15:23 |
| mfer | I'm curious because I'm thinking of changing the launchpad project for the go binding to be something more sane | 15:23 |
| fungi | board member and all | 15:23 |
| *** gokrokve has joined #openstack-infra | 15:24 | |
| *** dkliban has joined #openstack-infra | 15:24 | |
| mfer | mordred on https://review.openstack.org/#/c/62069/ I proposed a name for the php sdk as openstack-sdk-php. is it ok to use the name openstack in a stackforge project like this? | 15:24 |
| mordred | hrm | 15:24 |
| *** praneshp has quit IRC | 15:25 | |
| mfer | this is in the same pattern as aws, azure, and others use for their sdks. for DX and findability it's the best name | 15:25 |
| mordred | yeah - actually, let fungi get confirm from jonathan | 15:25 |
| fungi | problem being, we've already let several stackforge projects in like that (puppet-openstack, cl-openstack, openstack-chef, et cetera) | 15:25 |
| mfer | that's why i'm trying | 15:25 |
| mordred | I _think_ it's fine | 15:25 |
| mordred | the main issue is more in usage | 15:25 |
| mfer | usage? | 15:25 |
| mordred | "PHP SDK for OpenStack" vs. "OpenStack PHP SDK" | 15:26 |
| fungi | i think it's fine too, but if for some reason it's not fine, then i'd like to make sure we start correcting it. i know there have been recent stirrings about "pollution" of the trademark in light of the "what is core" spider/defcore discussions | 15:26 |
| mordred | one indicates that it's an SDK in PHP for OpenStack - the second could be seen to indicate that it is a php sdk that is _part_ of OpenStack | 15:26 |
| BobBall | Is there a way to get a change's dependency tree from gerrit's ssh interface? The json that I'm getting back doesn't seem to have the dependency information in it... which is somewhat surprising | 15:26 |
| mordred | BobBall: hrm. you know, I'm not sure. perhaps not | 15:27 |
| *** rakhmerov has quit IRC | 15:27 | |
| fungi | BobBall: the dependencies are actually the git tree itself. so if you can identify the parent commit sha, then you can query it for a change number as well | 15:27 |
| mfer | mordred this is a PHP SDK for OpenStack. It's to communitcate with APIs and such. the naming pattern for the project is based around other expectations. For example aws does aws-sdk-lang | 15:27 |
| *** gokrokve_ has joined #openstack-infra | 15:27 | |
| mfer | mordred i'm trying to just follow the common patterns to make it findable for devs who want to write php apps against openstack apis | 15:28 |
| mordred | yup | 15:28 |
| mfer | whether you find it sad or beautiful... php is the most widely use language on the web by a long long long shot. | 15:28 |
| mordred | mfer: I'm 95% sure what you're doing is fine | 15:28 |
| *** gokrokve has quit IRC | 15:28 | |
| BobBall | Not what I was hoping to hear fungi ... :) | 15:28 |
| BobBall | but thank | 15:28 |
| mordred | but we _should_ for sure get jonathan's buy in | 15:28 |
| BobBall | +s | 15:28 |
| fungi | mordred: in other news, and why i invoked the summoning ritual earlier, any objections to adjusting oslo.messaging's pre-release pipeline to upload tarballs to pypi for now, until they're ready for the first stable release? | 15:29 |
| mordred | hrm | 15:29 |
| *** rakhmerov has joined #openstack-infra | 15:29 | |
| fungi | there's no risk of pip 1.3 "upgrading" a user from a stable release to a pre-release if there never was a stable release | 15:29 |
| mordred | I personally don't mind that. the key will be fixing it to not upload tarballs to pypi on pre-releases any more once they do make a release | 15:29 |
| mordred | but hopefully we'll have wheel uploads finished by then | 15:29 |
| *** yaguang has quit IRC | 15:30 | |
| fungi | right, i made sure to highlight the need to make that adjustment come release time | 15:30 |
| mordred | I'm largely in favor of removing support for external -f urls myself | 15:31 |
| *** krotscheck has joined #openstack-infra | 15:31 | |
| *** rakhmerov has left #openstack-infra | 15:32 | |
| *** rakhmerov has joined #openstack-infra | 15:32 | |
| *** sandywalsh has quit IRC | 15:32 | |
| fungi | it's currently flumoxing the pypi-mirror run inside the requirements integration job, since the mirror is getting built from upstream pypi downloads, but it can't find a release of oslo.messaging on pypi. then later when it tries to satisfy glance install dependencies from that mirror, and pip 1.5 refuses to follow the -f url in the requirements.txt, looks in the local mirror it built instead and | 15:33 |
| fungi | sees no oslo.messaging, then dies | 15:33 |
| fungi | at least, this is the behavior i'm currently able to get out of it once i whitelist netifaces, psutil, et cetera | 15:34 |
| *** ildikov_ has quit IRC | 15:35 | |
| mordred | fungi: have you figured out how to land the whitelist into the mirror build? | 15:41 |
| mordred | fungi: is it purely a patch to run-mirror? | 15:41 |
| *** kraman has joined #openstack-infra | 15:41 | |
| fungi | mordred: so far it looks like https://review.openstack.org/66364 | 15:42 |
| mordred | fungi: wow. that's gross | 15:43 |
| fungi | the most recent logs there show the behavior i'm running into with glance's -f requirement on unreleased oslo.messaging | 15:43 |
| fungi | well, that's a hack to see what else is broken | 15:43 |
| fungi | it could certainly be refined | 15:43 |
| mordred | well, at least we KNOW we're not going to land additional broken reqs now :) | 15:43 |
| fungi | heh | 15:43 |
| mordred | fungi: I'd +2 that patch, but my team access is all gone | 15:44 |
| mordred | :) | 15:44 |
| fungi | note, some of those pip commands might not need the whitelists. i just splatted them on all four so i could keep moving, since it takes so long to run | 15:44 |
| fungi | but so far it appears it's just those four requirements, now that netaddr is getting uploaded | 15:44 |
| *** sandywalsh has joined #openstack-infra | 15:44 | |
| kraman | mordred: ping | 15:45 |
| fungi | mordred: as for gerrit group memberships, jeblair queried and bulk-removed you in the db. i can add you back now, but need to hunt down where he saved the list | 15:46 |
| *** jgrimm has joined #openstack-infra | 15:46 | |
| fungi | so probably not until after the conference call i'm on for the next hour | 15:46 |
| *** wenlock has joined #openstack-infra | 15:46 | |
| fifieldt | yay, conference call | 15:46 |
| mordred | fungi: it's no rush - just letting you know, I can't +2 that patch | 15:47 |
| fungi | mordred: i wouldn't want you to +2 that patch right now anyway. not until i can get the job to pass (that's why i have it marked wip) | 15:47 |
| kraman | mordred: can you take a look at https://github.com/kraman/zuul/compare/solum_hacks . would like to discuss some of the changes there to see if there are better ways to do them | 15:48 |
| fungi | fifieldt: btw, finally got around to peeking at gerrit logs... "[patchset-created] output: welcome-message: error: too few arguments" | 15:49 |
| *** thedodd has joined #openstack-infra | 15:49 | |
| fifieldt | thanks fungi, I really appreciate you took the time to look | 15:49 |
| fifieldt | things have been busy for you of late :) | 15:49 |
| fungi | fifieldt: i suspect the script is looking for arguments which the patchset-created hook does not normally pass | 15:49 |
| fungi | fifieldt: yes, yes they have | 15:50 |
| fungi | scattered flurry of insane | 15:50 |
| fifieldt | and not the tasty, sugary kind of flurry, either | 15:50 |
| fungi | not that anybody's told me anyway | 15:50 |
| *** NikitaKonovalov has quit IRC | 15:53 | |
| *** SergeyLukjanov has quit IRC | 15:53 | |
| fifieldt | argh, I'm terrible | 15:55 |
| fifieldt | I copied ssh client methodology from one script, and the arguments for a different method | 15:55 |
| *** ^d has joined #openstack-infra | 15:56 | |
| *** pballand has joined #openstack-infra | 15:56 | |
| *** senk has quit IRC | 15:57 | |
| fungi | it happens | 15:58 |
| *** reed has joined #openstack-infra | 15:59 | |
| *** ruhe is now known as ruhe_away | 16:00 | |
| *** eharney_ has joined #openstack-infra | 16:01 | |
| *** eharney has quit IRC | 16:02 | |
| *** eharney_ is now known as eharney | 16:02 | |
| *** senk has joined #openstack-infra | 16:02 | |
| *** rcleere has joined #openstack-infra | 16:02 | |
| fifieldt | btw fungi, should we create a role id similar to trivial_rebase_role_id => 'trivial-rebase@review.openstack.org' for welcome message ? | 16:04 |
| fifieldt | at the moment it doesn't have one | 16:05 |
| fifieldt | I guess it's needed so gerrit gets the labels right | 16:05 |
| *** DinaBelova has quit IRC | 16:05 | |
| fifieldt | or maybe it isn't used | 16:06 |
| * fifieldt shrugs | 16:06 | |
| *** sarob has joined #openstack-infra | 16:06 | |
| *** afazekas has quit IRC | 16:09 | |
| *** ruhe_away is now known as _ruhe | 16:09 | |
| fungi | fifieldt: hrm... yeah we probably do need a service account created for it so that it can have its own name in gerrit comments | 16:12 |
| fifieldt | should I send a message to the mailing list? | 16:12 |
| fifieldt | doing the request | 16:12 |
| fifieldt | argh, will also need to know where the SSH key will live | 16:13 |
| fungi | fifieldt: either ml post or a openstack-ci bug to help keep me from forgetting, sure | 16:13 |
| fifieldt | since the private key needs to live on gerrit, I guess I'll just file a bug | 16:14 |
| fungi | yeah, i can generate and add it to hiera | 16:15 |
| *** AaronGr_Zzz is now known as AaronGr | 16:16 | |
| fifieldt | https://bugs.launchpad.net/openstack-ci/+bug/1269057 | 16:16 |
| uvirtbot | Launchpad bug 1269057 in openstack-ci "Make a service account for welcome-message" [Undecided,New] | 16:16 |
| fifieldt | :) | 16:16 |
| *** jcoufal has quit IRC | 16:16 | |
| *** changbl has joined #openstack-infra | 16:18 | |
| *** dpyzhov has quit IRC | 16:21 | |
| *** boris-42 has quit IRC | 16:21 | |
| *** markmcclain has joined #openstack-infra | 16:23 | |
| fungi | fifieldt: thanks! | 16:24 |
| *** thuc has joined #openstack-infra | 16:24 | |
| *** praneshp has joined #openstack-infra | 16:25 | |
| *** markmcclain has quit IRC | 16:25 | |
| *** thuc has quit IRC | 16:25 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Fix CLI args for welcome-message https://review.openstack.org/66623 | 16:25 |
| *** thuc has joined #openstack-infra | 16:26 | |
| *** moted has quit IRC | 16:26 | |
| *** thuc has quit IRC | 16:26 | |
| *** thuc has joined #openstack-infra | 16:27 | |
| *** markmcclain has joined #openstack-infra | 16:28 | |
| *** rakhmerov has quit IRC | 16:28 | |
| *** ArxCruz has quit IRC | 16:29 | |
| *** rfolco has quit IRC | 16:32 | |
| *** bauzas has left #openstack-infra | 16:32 | |
| *** ArxCruz has joined #openstack-infra | 16:33 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Fix up `zuul --version` https://review.openstack.org/66629 | 16:34 |
| *** MarkAtwood has joined #openstack-infra | 16:35 | |
| *** rfolco has joined #openstack-infra | 16:36 | |
| anteaya | sorry for coming in blind, I have not read any backscroll :/ can gerrit talk to the foundation website yet, or do we still get 404? https://bugs.launchpad.net/openstack-org/+bug/1268254 | 16:38 |
| uvirtbot | Launchpad bug 1268254 in openstack-org "no communication with gerrit" [Undecided,New] | 16:38 |
| *** sarob has quit IRC | 16:38 | |
| anteaya | I'm just going through my email | 16:38 |
| *** SergeyLukjanov has joined #openstack-infra | 16:41 | |
| *** markmcclain has quit IRC | 16:41 | |
| anteaya | mikal: we missing one, tail-f has a -1 on this one: https://review.openstack.org/#/c/56184/17 can you give me a hand clearing, or shall I take care of this one? | 16:43 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: change # of tries back to 20 https://review.openstack.org/66631 | 16:45 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: expose on channel when we timeout on logs https://review.openstack.org/66565 | 16:45 |
| SergeyLukjanov | does anyone use bip and can share config file? looks like I have a bad karma on setting up bouncers ;) | 16:45 |
| fungi | anteaya: i have no idea--did the user you talked to retry? | 16:45 |
| *** jbryce has joined #openstack-infra | 16:45 | |
| *** gyee_ has joined #openstack-infra | 16:45 | |
| fungi | i have been too busy to check e-mail bugs | 16:46 |
| sdague | jog0: if you have plane review time, check out the elastic recheck queue I've got going | 16:46 |
| sdague | the bot will be tons more reliable after all of this | 16:46 |
| *** ^demon has joined #openstack-infra | 16:47 | |
| *** ^d has quit IRC | 16:49 | |
| *** beekneemech is now known as bnemec | 16:50 | |
| *** _ruhe is now known as ruhe | 16:51 | |
| *** blamar has quit IRC | 16:53 | |
| anteaya | fungi: understood | 16:53 |
| *** gothicmindfood has joined #openstack-infra | 16:54 | |
| anteaya | fungi: I had advised him to wait until at least Monday to check in, I will see if I can find him | 16:54 |
| anteaya | and if so, I will ask him | 16:54 |
| *** sandywalsh_ has joined #openstack-infra | 16:54 | |
| *** fifieldt has quit IRC | 16:55 | |
| *** blamar has joined #openstack-infra | 16:55 | |
| *** krotscheck has quit IRC | 16:55 | |
| fungi | ahh, originally going to ask him to retry, but then i never heard the outcome | 16:56 |
| fungi | er, you originally said you were going to, i mean | 16:56 |
| fungi | anyway, i mentioned it to toddmorey and jbryce and if we can get some detailed timestamps of the incidents it will help | 16:57 |
| anteaya | fungi: he did retry and got the same error | 16:57 |
| fungi | k | 16:57 |
| anteaya | since we were face to face he may have told me on the walk to lunch and we never shared with you | 16:57 |
| *** mrodden has quit IRC | 16:57 | |
| anteaya | sorry about that, the problem of face to face | 16:58 |
| jbryce | are there full error logs from the client side? | 16:58 |
| jbryce | with timestamps, full stack trace, etc? | 16:58 |
| *** mrodden has joined #openstack-infra | 16:58 | |
| jbryce | i know the first one was about 3:50 utc sunday. what about the second? | 16:59 |
| anteaya | jbryce: shortly after that one, within the hour I believe | 16:59 |
| fungi | jbryce: i will dig on the gerrit end and see if it logged anything | 16:59 |
| *** tma996 has quit IRC | 16:59 | |
| anteaya | but I don't have a specific time frame for it, sorry | 16:59 |
| jbryce | fungi: thanks | 17:00 |
| * fungi needs to switch computers real quick and will dig for more detail | 17:00 | |
| jbryce | has anyone signed up successfully since then? | 17:00 |
| *** esker has quit IRC | 17:01 | |
| *** gothicmindfood_ has joined #openstack-infra | 17:01 | |
| *** esker has joined #openstack-infra | 17:01 | |
| *** gothicmindfood has quit IRC | 17:02 | |
| fungi | that's a good question, i should be able to do a date range query on that field in the gerrit db | 17:05 |
| *** portante has joined #openstack-infra | 17:05 | |
| *** sarob has joined #openstack-infra | 17:05 | |
| *** esker has quit IRC | 17:06 | |
| *** vkozhukalov has joined #openstack-infra | 17:06 | |
| *** ruhe is now known as ruhe_away | 17:07 | |
| *** rakhmerov has joined #openstack-infra | 17:08 | |
| fungi | jbryce: here's what one of the exceptions looks like... http://paste.openstack.org/show/61199/ | 17:09 |
| zaro | morning | 17:09 |
| fungi | java likes to get very wordy with its backtraces | 17:09 |
| *** woodspa has joined #openstack-infra | 17:09 | |
| fungi | morning zaro... did you see clarkb's comments in scrollback about the scp plugin? | 17:10 |
| jbryce | fungi: looking | 17:10 |
| sdague | zaro: I can definitely confirm we are dropping log files going to ES | 17:11 |
| *** gokrokve_ has quit IRC | 17:11 | |
| *** woodspa has quit IRC | 17:11 | |
| *** herndon_ has quit IRC | 17:11 | |
| *** gokrokve has joined #openstack-infra | 17:11 | |
| *** Craig has joined #openstack-infra | 17:12 | |
| fungi | jbryce: it must be working at least some of the time, because i see 8 accounts which have added/updated contact info via that api successfully since midnight utc | 17:13 |
| *** mozawa has quit IRC | 17:13 | |
| *** Craig is now known as cp16net | 17:16 | |
| *** gokrokve has quit IRC | 17:16 | |
| *** sandywalsh_ has quit IRC | 17:17 | |
| *** ruhe_away is now known as _ruhe | 17:17 | |
| anteaya | jeblair: there is another 3rd party testing meeting scheduled: http://lists.openstack.org/pipermail/openstack-dev/2014-January/024297.html | 17:17 |
| zaro | sdague: what is ES? | 17:17 |
| fungi | centos6-3 workspace for gate-ceilometer-python26 seems to be broken... https://jenkins01.openstack.org/job/gate-ceilometer-python26/3135/console (offlining and cleaning up now) | 17:17 |
| sdague | zaro: elastic search | 17:17 |
| fungi | zaro: elasticsearch | 17:17 |
| anteaya | I bet it conflicts but wanted to share with you | 17:18 |
| *** DinaBelova has joined #openstack-infra | 17:18 | |
| *** herndon has joined #openstack-infra | 17:18 | |
| zaro | fungi: yeah, i noticed the comment. but not sure how i could repro. | 17:18 |
| *** ildikov_ has joined #openstack-infra | 17:18 | |
| fungi | zaro: i'm guessing it happens intermittently so probably not reproducible 100% of the time | 17:19 |
| *** thomasem has quit IRC | 17:19 | |
| zaro | fungi: i mean what do i need to setup to even attempt to repro? | 17:20 |
| sdague | I think it's probably around 5 - 10% of the time based on rough estimates of what I'm seeing | 17:20 |
| zaro | i'm not real familiar with ES under the hood. | 17:20 |
| fungi | zaro: probably something to listen for the ready event and then try immediately to access the url on the master | 17:20 |
| *** yassine has quit IRC | 17:20 | |
| fungi | probably don't need the whole logstash-jenkins-client worker to reproduce it | 17:21 |
| zaro | thought i would just wait for clarkb | 17:21 |
| zaro | return | 17:21 |
| fungi | zaro: sure. mainly just making sure you saw his comments | 17:21 |
| zaro | didn't he fly back yesterday? | 17:21 |
| fungi | zaro: no idea what yesterday means in this context. he was sightseeing in au while we were asleep | 17:22 |
| fungi | so may be on a plane now, but i don't really know | 17:22 |
| zaro | fungi: ohh i heard something about packing and checking in.. maybe just flying to another part of au. | 17:23 |
| *** thomasem has joined #openstack-infra | 17:23 | |
| zaro | well i'll catch him next time he's in channel. | 17:23 |
| sdague | fungi: I think he's in the air now | 17:23 |
| sdague | from what I remember of his comments | 17:24 |
| *** Ryan_Lane has quit IRC | 17:24 | |
| *** boris-42 has joined #openstack-infra | 17:26 | |
| SpamapS | ok I have another missing commit from git.openstack.org | 17:27 |
| * SpamapS saves local repo in a tarball | 17:27 | |
| SpamapS | clarkb: want that tarball? | 17:27 |
| fungi | SpamapS: what't the project name and commit id? | 17:27 |
| fungi | one of the servers in the farm is probably lagging behind on replication for some reason | 17:28 |
| SpamapS | fungi: heat, 1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26 | 17:28 |
| * fungi checks | 17:28 | |
| *** max_lobur is now known as max_lobur_afk | 17:28 | |
| mordred | SpamapS: you're better at doing that than anyone else | 17:30 |
| fungi | SpamapS: all four of the git servers behind git.o.o have that ref in their openstack/heat repo mirrors | 17:30 |
| *** pcrews_ has quit IRC | 17:31 | |
| fungi | SpamapS: is it missing on some specific branch? | 17:31 |
| * fungi checks master | 17:32 | |
| fungi | i see it in the history of the master branch on each of them | 17:33 |
| *** hashar has quit IRC | 17:33 | |
| SpamapS | fungi: let me try pulling again | 17:33 |
| SpamapS | error: Unable to find 1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26 under https://git.openstack.org/cgit/openstack/heat | 17:33 |
| SpamapS | Cannot obtain needed commit 1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26 | 17:33 |
| fungi | weird. i'll try cloning from them | 17:34 |
| SpamapS | fungi: I'll push the tarball of my repo up | 17:34 |
| SpamapS | ugggh | 17:34 |
| SpamapS | -rw-r--r-- 1 clint clint 259M Jan 14 09:29 heat-missing-1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26.tar.gz | 17:34 |
| mordred | SpamapS: use the cloud | 17:35 |
| BobBall | Would someone be able to review our nodepool changes? They are needed to get XenAPI tested by jenkins which in turn is needed ASAP (pre I-2 if possible) to stop a nova patch to deprecate XenAPI driver. Three nodepool changes at https://review.openstack.org/#/dashboard/5044 | 17:36 |
| fungi | able to clone from the filesystem on all of them too | 17:36 |
| SpamapS | mordred: I'm using the xz at this point | 17:38 |
| jbryce | fungi, anteaya: no errors at that timestamp either | 17:40 |
| fungi | jbryce: what are the chances that server sits behind a proxy at the provider? | 17:40 |
| jbryce | fungi: 1,000,000% | 17:40 |
| fungi | wondering whether the 404 is coming from something in front of where you have logs | 17:41 |
| SpamapS | -rw-r--r-- 1 clint clint 171M Jan 14 09:29 heat-missing-1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26.tar.xz | 17:44 |
| SpamapS | not bad xz.. not bad | 17:44 |
| SpamapS | fungi: I'm uploading that to HP Cloud's us east object storage. Perhaps you have a VM there you can inspect it from? | 17:45 |
| *** markmcclain has joined #openstack-infra | 17:46 | |
| fungi | SpamapS: perhaps. is the url only reachable from within that cloud region? | 17:46 |
| *** jerryz has joined #openstack-infra | 17:46 | |
| SpamapS | fungi: no it is public | 17:46 |
| SpamapS | fungi: but it will be 10Gb in that region. ;) | 17:47 |
| *** NikitaKonovalov has joined #openstack-infra | 17:47 | |
| SpamapS | fungi: but it will be 10Gb in that region. ;) | 17:47 |
| SpamapS | doh | 17:47 |
| SpamapS | tcp 0 52308 192.168.0.21:48486 15.126.26.71:443 ESTABLISHED 6097/firefox | 17:47 |
| SpamapS | healthy Send-Q there :) | 17:47 |
| fungi | it'll be 50mbps at my house, and that's plenty for a 171mib file | 17:47 |
| jbryce | fungi: the http logs are actually sourced from the proxy because it's load-balancing multiple servers behind it. it's also possible there's some fault that's not being logged because it's dying in a different part of the response. if that's the case, there's actually a different error page than the standard 404 that gets served, so if we can replicate in real-time or capture the body of the 404 response we could nail that dow | 17:47 |
| jbryce | however that is usually caused by a failure of one of the load-balanced web nodes to respond and then there's usually an app error logged... | 17:47 |
| SpamapS | fungi: cool | 17:47 |
| jbryce | if anyone runs into an error with the gerrit interface to www.openstack.org you can feel free to contact me directly asap (210-317-2438) | 17:49 |
| sdague | SpamapS: do you have a corp firewall between you and the git server? | 17:50 |
| fungi | jbryce: okay, thanks. i'm getting in touch with the user fifield reported having issues at samsung, to confirm whether they're getting an error on contact form submission or not | 17:50 |
| sdague | we found some git ops were getting disrupted by a layer 7 attempted injection by the IBM corp firewall | 17:51 |
| sdague | back when I had to deal with such things | 17:51 |
| *** NikitaKonovalov has quit IRC | 17:51 | |
| SpamapS | sdague: no | 17:51 |
| SpamapS | it is https direct to git.o.o | 17:51 |
| sdague | SpamapS: yes, this was over https | 17:51 |
| sdague | there was some ssl issue | 17:52 |
| SpamapS | well I'm also at home. but.. wow.. that is _eeeeevil_ | 17:52 |
| *** NikitaKonovalov has joined #openstack-infra | 17:52 | |
| sdague | yeh, agreed | 17:52 |
| *** markmcclain1 has joined #openstack-infra | 17:52 | |
| sdague | just throwing it out there as a possibility :) | 17:52 |
| *** jcoufal-m has joined #openstack-infra | 17:53 | |
| *** derekh has quit IRC | 17:54 | |
| *** markmcclain has quit IRC | 17:55 | |
| SpamapS | fungi: https://region-b.geo-1.objects.hpcloudsvc.com/v1/42841271810042/clint-fewbar/heat-missing-1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26.tar.xz?temp_url_sig=42841271810042:G6JETLFJDYYT771NJWPB:ffda56fc309016c67433e361fc85b843a2431f53&temp_url_expires=1389723024 | 17:55 |
| fungi | SpamapS: thanks, i'll have a look | 17:56 |
| SpamapS | fungi: https://region-b.geo-1.objects.hpcloudsvc.com/v1/42841271810042/clint-fewbar/heat-missing-1bcd6625b44b70b1fddef4b7d46ccf32a7fa7a26.tar.xz should work too (horizon just giving me tempurls for the hell of it) | 17:57 |
| *** NikitaKonovalov has quit IRC | 17:57 | |
| fungi | i'll try that one. the first was giving me " | 17:57 |
| fungi | HTTP request sent, awaiting response... 401 Unauthorized" | 17:57 |
| *** mancdaz is now known as mancdaz_away | 17:57 | |
| *** NikitaKonovalov has joined #openstack-infra | 17:57 | |
| fungi | that worked | 17:57 |
| jog0 | sdague: I'm taking a redeye and taking today off so will review that stuff in canada | 18:00 |
| sdague | ok, cool | 18:00 |
| *** NikitaKonovalov has quit IRC | 18:01 | |
| *** harlowja_away is now known as harlowja | 18:03 | |
| *** changbl has quit IRC | 18:05 | |
| *** changbl has joined #openstack-infra | 18:05 | |
| *** gokrokve has joined #openstack-infra | 18:06 | |
| *** NikitaKonovalov has joined #openstack-infra | 18:07 | |
| *** jcoufal has joined #openstack-infra | 18:09 | |
| *** NikitaKonovalov has quit IRC | 18:10 | |
| *** mrodden has quit IRC | 18:10 | |
| harlowja | clarkb yt, or others that know more about the 'openstack_citest' database on the slaves, just a few questions releated to it | 18:13 |
| fungi | SpamapS: preliminary confirmation... i can't remote update your broken local copy (same error you reported, same commit at fault), but i can make a fresh clone from the same exact remote and those commits are included intact | 18:13 |
| fungi | harlowja: i may be able to answer questions about that | 18:14 |
| harlowja | ah, sweet fungi | 18:14 |
| *** NikitaKonovalov has joined #openstack-infra | 18:15 | |
| harlowja | so intersting question, testr runs in parallel right, and say each test uses 'openstack_citest' as its database, then basically each test needs to be serialized to avoid messing up that database, is that expected, does the ''openstack_citest' have access to create other temp databases to avoid that serialization? | 18:15 |
| *** ianw has quit IRC | 18:15 | |
| *** dcramer__ has quit IRC | 18:15 | |
| harlowja | if each test creates/drops it, then they could collide :-/ | 18:15 |
| *** gyee_ has quit IRC | 18:16 | |
| harlowja | but if each test made ''openstack_citest-$XYZ' then they could avoid that, but it doesn't appear that the ''openstack_citest' has those capabilties | 18:16 |
| *** markmcclain1 has quit IRC | 18:16 | |
| harlowja | https://github.com/openstack-infra/config/blob/master/modules/jenkins/manifests/slave.pp#L230 | 18:17 |
| fungi | harlowja: it's possible with postgres, but not with mysql right now (there's been talk of switching from mysql to drizzle to overcome that) | 18:17 |
| *** Ajaeger1 has joined #openstack-infra | 18:17 | |
| fungi | harlowja: basically, drop isn't grantable without making the user a full mysql admin | 18:17 |
| harlowja | ah | 18:18 |
| fungi | harlowja: so if we're trying to reign in the privs of unit test scope activities, that presents an issue | 18:18 |
| harlowja | correct | 18:18 |
| fungi | but it's mostly a shortcoming of mysql | 18:19 |
| *** krotscheck has joined #openstack-infra | 18:19 | |
| harlowja | i'm fine with serializing it to avoid screwing up each test, if thats how it has to go :) | 18:19 |
| SpamapS | fungi: that seems like "software bug" to me. | 18:19 |
| SpamapS | fungi: as in, either local client is doing someting wonky on fetch, or reporting the wrong problem, or cgit is not responding properly. | 18:20 |
| fungi | harlowja: i think that's the route oslo was talking about, but there has been some concern raised that it flies in the face of parallelizing these things. i'd recommend ambushing mordred and clarkb to get into the weeds on that decision | 18:20 |
| harlowja | fungi although i'm unsure how nova is doing it, they seem to be dropping and recreating :-/ | 18:20 |
| harlowja | http://git.openstack.org/cgit/openstack/nova/tree/nova/tests/db/test_migrations.py#n287 | 18:20 |
| *** jpich has quit IRC | 18:20 | |
| harlowja | if no drop permissions wonder how thats worked for them :-/ | 18:20 |
| * mordred reads scrollback | 18:20 | |
| fungi | SpamapS: i did a git gc and git fsck on it and am getting a persistently dangling commit 0c83ac25a912843cfdee3ecd8bfb70921d326974 which is unreadab;e | 18:20 |
| fungi | unreadable | 18:20 |
| fungi | harlowja: entirely possible i'm mistaken. if so mordred will correct me | 18:21 |
| harlowja | fungi np | 18:21 |
| sdague | harlowja: you could also declare locks around the tests | 18:21 |
| harlowja | nova also doesn't seem to be ensuring that each test locks a file to avoid colliding, so sorta weird, ha | 18:21 |
| harlowja | sdague ya, if i have to :) | 18:21 |
| sdague | fortunately we have this nice oslo lock infrastruture :) | 18:22 |
| sdague | I think in the nova case if you back it out far enough, there really is only one test | 18:22 |
| harlowja | sdague ah :) | 18:22 |
| sdague | which is why the parallel isn't the problem | 18:22 |
| sdague | it's the migrations test | 18:22 |
| sdague | it's huge, but only actually one test | 18:22 |
| harlowja | sdague ya, taskflow uses part of the oslo locking stuff, the one problem with the olso locking code is https://github.com/openstack/oslo-incubator/blob/master/openstack/common/lockutils.py#L34 (the sucking in of oslo.logging, which brings in oslo.config) | 18:23 |
| *** mrodden has joined #openstack-infra | 18:23 | |
| harlowja | also lockutils itself seems to suck in oslo.config | 18:23 |
| harlowja | anyways, if have to do process lock per test, thats fine | 18:23 |
| harlowja | sdague the other part that confuses me is if fungi is right, then nova shouldn't be able to drop/create the openstack_citest db :-/ | 18:24 |
| sdague | why not? | 18:24 |
| *** atiwari has joined #openstack-infra | 18:25 | |
| harlowja | ah, nm, i think its fine | 18:25 |
| fungi | harlowja: i believe we create that db using puppet | 18:25 |
| *** esker has joined #openstack-infra | 18:25 | |
| *** andreaf has quit IRC | 18:25 | |
| fungi | harlowja: if you git grep inside openstack-infra/config i believe you'll find it there | 18:25 |
| harlowja | fungi ya, the nova code then must be legacy | 18:25 |
| harlowja | *the nova code that tries to create it | 18:25 |
| sdague | I actually think it's created both ways | 18:26 |
| sdague | and that's allowed | 18:26 |
| harlowja | k, nm then :) | 18:26 |
| mfer | fungi we're you able to get feedback on using the openstack name? | 18:26 |
| sdague | once the permissions are set in mysql, nova can drop and recreate to it's hearts content | 18:26 |
| mordred | sdague: that is correct | 18:26 |
| harlowja | sdague ya, good point | 18:26 |
| harlowja | k, this answered my question, will add some shared process locks on taskflow tests that are gonna touch that db | 18:27 |
| fungi | mfer: the initial answer was that it's a case-by-case basis. i'm sending mark collier the details right now and he's going to determine whether it needs to go before foundation legal counsel | 18:27 |
| *** jamielennox|away is now known as jamielennox | 18:27 | |
| mfer | fungi thanks. if we could know a general answer for SDKs that would be useful for any others as well | 18:27 |
| *** markmcclain has joined #openstack-infra | 18:27 | |
| *** NikitaKonovalov has quit IRC | 18:28 | |
| fungi | mfer: i agree. it's my understanding that if it's a generalized approach allowing open participation and isn't vendor-focused or otherwise supporting a specific commercial endeavor then it's non-infringing | 18:29 |
| fungi | so i suspect sdks fall into that category for the most part, but i am not a lawyer ;) | 18:29 |
| mfer | fungi for my stuff it won't be vendor focused but will work cleanly for vendors | 18:29 |
| fungi | yeah, i think as long as it tries to work for everyone equally and not one particular company, that's the crux of the requirement | 18:30 |
| mfer | and i agree whole heartedly with that | 18:30 |
| *** _ruhe is now known as ruhe | 18:30 | |
| *** NikitaKonovalov has joined #openstack-infra | 18:33 | |
| *** vipul is now known as vipul-away | 18:35 | |
| *** krtaylor has quit IRC | 18:36 | |
| lifeless | mkoderer: pong | 18:37 |
| *** NikitaKonovalov has quit IRC | 18:37 | |
| *** NikitaKonovalov has joined #openstack-infra | 18:38 | |
| *** krtaylor has joined #openstack-infra | 18:39 | |
| *** hogepodge has joined #openstack-infra | 18:40 | |
| *** pcrews_ has joined #openstack-infra | 18:40 | |
| *** hogepodge has quit IRC | 18:41 | |
| *** ryanpetrello has quit IRC | 18:41 | |
| *** krotscheck has quit IRC | 18:42 | |
| mkoderer | lifeless: hi | 18:42 |
| mkoderer | lifeless: I already wrote an email | 18:42 |
| *** gyee_ has joined #openstack-infra | 18:42 | |
| mkoderer | lifeless: I tried to define a decorator for the testscenarios | 18:43 |
| *** hartsocks has joined #openstack-infra | 18:44 | |
| hartsocks | hi, I have an identity crisis | 18:44 |
| mordred | hartsocks: I recommend spending some time alone in the desert to sort that out | 18:44 |
| fungi | hartsocks: double-account in gerrit? | 18:44 |
| hartsocks | I am both http://www.stackalytics.com/report/users/hartsocks and http://www.stackalytics.com/report/users/hartsocks | 18:44 |
| hartsocks | fungi: yeah | 18:44 |
| hartsocks | http://www.stackalytics.com/report/users/hartsocks | 18:45 |
| hartsocks | both hartsock and hartsocks … not sure how I managed to screw that up | 18:45 |
| fungi | i seem to recall you had both as separate accounts at one point, maybe a 6mos to a year ago | 18:45 |
| fungi | because i think we got asked to combine them back then | 18:46 |
| hartsocks | fungi: yeah, so I wanted to fold everything into hartsocks IIRC. | 18:46 |
| hartsocks | fungi: but I think I left my launchpad id hartsock or something… oops. | 18:46 |
| *** krotscheck has joined #openstack-infra | 18:47 | |
| fungi | yeah, we don't control stackalytics and i'm not sure how they're doing account classification | 18:47 |
| *** NikitaKonovalov has quit IRC | 18:47 | |
| hartsocks | so, on launchpad… should I change my id to hartsocks? will that break things worse? | 18:48 |
| fungi | at first glance, the gerrit activity counted for hartsock and hartsocks seem to map back to the same e-mail address and gerrit account number | 18:49 |
| *** hogepodge has joined #openstack-infra | 18:49 | |
| hartsocks | btw on review.openstack.org when you search "shawn" I show up twice too... | 18:49 |
| hartsocks | not sure what's going on there. | 18:49 |
| hartsocks | (that's on the "add reviewer" dialog) | 18:50 |
| *** jamielennox is now known as jamielennox|away | 18:50 | |
| fungi | hartsocks: that means there are still two accounts in gerrit. i'll make sure the e-mail address gets cleared from the old one | 18:50 |
| fungi | duplicate e-mail addresses tends to cause issues when selecting a reviewer for a change or group membership | 18:51 |
| hartsocks | fungi: sorry to be a pain. I should have just kept my hartsock@acm.org identity | 18:51 |
| *** jcoufal-m has quit IRC | 18:51 | |
| fungi | hartsocks: part of this is that gerrit wasn't designed well to deal with openid providers, and lp does unexpected things with openid | 18:52 |
| fungi | so it's not really your fault | 18:52 |
| *** NikitaKonovalov has joined #openstack-infra | 18:52 | |
| hartsocks | fungi: I want to clean it up, and I also don't want to make it worse... | 18:53 |
| lifeless | fungi: hi | 18:53 |
| fungi | lifeless: hi there | 18:53 |
| lifeless | fungi: so, new cloud :) | 18:53 |
| *** vipul-away is now known as vipul | 18:53 | |
| fungi | lifeless: yeah | 18:53 |
| lifeless | fungi: I believe it to be all setup | 18:54 |
| *** rnirmal has joined #openstack-infra | 18:55 | |
| *** rnirmal_ has joined #openstack-infra | 18:57 | |
| *** rnirmal_ has quit IRC | 18:57 | |
| fungi | hartsocks: should be all set now, hopefully. i nulled-out the inactive account's preferred e-mail address and reassigned the stray external id to your active account | 18:57 |
| *** rnirmal_ has joined #openstack-infra | 18:57 | |
| *** NikitaKonovalov has quit IRC | 18:57 | |
| *** rnirmal_ has quit IRC | 18:57 | |
| fungi | lifeless: excellent. also, you helped us expose bug 1269001 | 18:57 |
| uvirtbot | Launchpad bug 1269001 in nodepool "Nodepool stops building any new nodes when one provider is down" [High,Triaged] https://launchpad.net/bugs/1269001 | 18:57 |
| *** rnirmal_ has joined #openstack-infra | 18:57 | |
| lifeless | fungi: ahahahahaha, sorry. | 18:57 |
| fungi | you can't be held accountable for our bugs | 18:58 |
| hartsocks | fungi: thanks. Should I edit my launchpad ID from https://launchpad.net/~hartsock to hartsocks or will that break something? | 18:58 |
| mordred | fungi: hold him accountable | 18:58 |
| *** SergeyLukjanov has quit IRC | 18:58 | |
| *** NikitaKonovalov has joined #openstack-infra | 18:58 | |
| fungi | hartsocks: i would leave it as is. changing your username in lp has been seen to generate yet another new openid | 18:58 |
| hartsocks | fungi: thanks again. | 18:58 |
| *** SergeyLukjanov_ has joined #openstack-infra | 18:59 | |
| fungi | hartsocks: don't mention it | 18:59 |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: maven-deploy: don't require id https://review.openstack.org/66033 | 18:59 |
| *** atiwari has quit IRC | 18:59 | |
| lifeless | fungi: so I'd like to get confirmation that you can use said cloud | 18:59 |
| *** rnirmal has quit IRC | 18:59 | |
| *** rnirmal_ is now known as rnirmal | 18:59 | |
| fungi | lifeless: me too. hopefully after the tc meeting i can give it a whirl | 19:00 |
| fungi | infra meeting time now | 19:00 |
| *** pblaho has joined #openstack-infra | 19:00 | |
| *** nicedice has quit IRC | 19:02 | |
| *** SergeyLukjanov_ has quit IRC | 19:03 | |
| *** nicedice has joined #openstack-infra | 19:03 | |
| *** NikitaKonovalov has quit IRC | 19:03 | |
| *** SergeyLukjanov has joined #openstack-infra | 19:03 | |
| *** krtaylor has quit IRC | 19:05 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Remove unnecessary headers https://review.openstack.org/62137 | 19:05 |
| openstackgerrit | A change was merged to openstack-dev/hacking: checking for metaclass to be Python 3.x compatible https://review.openstack.org/56890 | 19:06 |
| *** MarkAtwood has quit IRC | 19:07 | |
| *** NikitaKonovalov has joined #openstack-infra | 19:10 | |
| *** gyee_ has quit IRC | 19:10 | |
| *** elasticio has joined #openstack-infra | 19:11 | |
| *** ryanpetrello has joined #openstack-infra | 19:12 | |
| *** NikitaKonovalov has quit IRC | 19:14 | |
| *** dizquierdo has quit IRC | 19:15 | |
| *** markmcclain has quit IRC | 19:15 | |
| *** NikitaKonovalov has joined #openstack-infra | 19:19 | |
| *** yolanda has quit IRC | 19:20 | |
| *** dcramer_ has joined #openstack-infra | 19:21 | |
| *** herndon has quit IRC | 19:24 | |
| *** dcramer_ has quit IRC | 19:27 | |
| *** vkozhukalov has quit IRC | 19:29 | |
| *** ruhe is now known as ruhe_away | 19:30 | |
| *** ruhe_away is now known as ruhe | 19:33 | |
| *** NikitaKonovalov has quit IRC | 19:34 | |
| *** beagles is now known as beagles_brb | 19:35 | |
| *** NikitaKonovalov has joined #openstack-infra | 19:38 | |
| *** krtaylor has joined #openstack-infra | 19:45 | |
| *** NikitaKonovalov has quit IRC | 19:46 | |
| *** rfolco has quit IRC | 19:50 | |
| *** NikitaKonovalov has joined #openstack-infra | 19:51 | |
| *** vipul is now known as vipul-away | 19:52 | |
| *** vipul-away is now known as vipul | 19:52 | |
| *** dkliban is now known as dkliban_afk | 19:54 | |
| *** pballand has quit IRC | 19:55 | |
| *** dcramer_ has joined #openstack-infra | 19:56 | |
| *** CaptTofu has joined #openstack-infra | 19:56 | |
| *** ianw has joined #openstack-infra | 19:59 | |
| *** denis_makogon_ has joined #openstack-infra | 19:59 | |
| *** markmcclain has joined #openstack-infra | 19:59 | |
| *** jcoufal has quit IRC | 20:00 | |
| *** sdake has quit IRC | 20:01 | |
| *** pblaho has quit IRC | 20:01 | |
| *** sdake has joined #openstack-infra | 20:01 | |
| *** sdake has quit IRC | 20:01 | |
| *** sdake has joined #openstack-infra | 20:01 | |
| *** markmcclain1 has joined #openstack-infra | 20:01 | |
| *** markmcclain1 has quit IRC | 20:01 | |
| *** markmcclain1 has joined #openstack-infra | 20:01 | |
| *** madmike has joined #openstack-infra | 20:02 | |
| *** jcoufal has joined #openstack-infra | 20:02 | |
| *** yolanda has joined #openstack-infra | 20:03 | |
| *** mfink has quit IRC | 20:03 | |
| *** markmcclain has quit IRC | 20:04 | |
| *** ArxCruz has quit IRC | 20:05 | |
| *** beagles_brb is now known as beagles | 20:05 | |
| *** dkranz has quit IRC | 20:05 | |
| *** dkranz has joined #openstack-infra | 20:05 | |
| *** denis_makogon_ is now known as denis_makogon | 20:06 | |
| *** sparkycollier has joined #openstack-infra | 20:06 | |
| *** NikitaKonovalov has quit IRC | 20:06 | |
| *** vipul is now known as vipul-away | 20:08 | |
| *** esker has quit IRC | 20:08 | |
| *** thuc has quit IRC | 20:10 | |
| *** resker has joined #openstack-infra | 20:10 | |
| *** thuc has joined #openstack-infra | 20:10 | |
| *** senk has quit IRC | 20:11 | |
| *** CaptTofu has quit IRC | 20:12 | |
| *** vipul-away is now known as vipul | 20:12 | |
| *** NikitaKonovalov has joined #openstack-infra | 20:13 | |
| *** thuc has quit IRC | 20:15 | |
| *** Ajaeger1 has quit IRC | 20:18 | |
| *** gothicmindfood_ has left #openstack-infra | 20:20 | |
| *** gothicmindfood has joined #openstack-infra | 20:21 | |
| *** sarob has quit IRC | 20:23 | |
| *** HenryG has joined #openstack-infra | 20:24 | |
| *** NikitaKonovalov has quit IRC | 20:25 | |
| *** niska has quit IRC | 20:27 | |
| *** vipul is now known as vipul-away | 20:27 | |
| *** yolanda has quit IRC | 20:29 | |
| *** dkliban_afk is now known as dkliban | 20:32 | |
| *** NikitaKonovalov has joined #openstack-infra | 20:33 | |
| *** herndon has joined #openstack-infra | 20:34 | |
| *** NikitaKonovalov has quit IRC | 20:35 | |
| *** gyee has quit IRC | 20:38 | |
| *** gyee has joined #openstack-infra | 20:38 | |
| *** gothicmindfood has left #openstack-infra | 20:46 | |
| *** miqui has quit IRC | 20:46 | |
| *** gothicmindfood has joined #openstack-infra | 20:47 | |
| *** sarob has joined #openstack-infra | 20:48 | |
| *** sarob has quit IRC | 20:50 | |
| *** sarob has joined #openstack-infra | 20:51 | |
| *** senk has joined #openstack-infra | 20:53 | |
| *** niska has joined #openstack-infra | 20:53 | |
| *** senk1 has joined #openstack-infra | 20:55 | |
| *** senk has quit IRC | 20:57 | |
| *** dcramer_ has quit IRC | 20:58 | |
| *** DinaBelova has quit IRC | 20:59 | |
| *** markmcclain has joined #openstack-infra | 21:02 | |
| fungi | mordred: i've added you back to the administrators group in gerrit, so you should be able to add yourself to the other groups you need (i couldn't find where jeblair kept a record of what groups he removed you from) | 21:03 |
| *** AaronGr is now known as AaronGr_afk | 21:03 | |
| fungi | mordred: i'll add you back in lp groups and github orgs shortly | 21:03 |
| *** markmcclain2 has joined #openstack-infra | 21:03 | |
| fungi | assuming you've changed passwords anywhere you feel you need to do so | 21:03 |
| mordred | fungi: awesome. thanks | 21:04 |
| *** markmcclain1 has quit IRC | 21:04 | |
| *** ^d has joined #openstack-infra | 21:05 | |
| *** ^d has joined #openstack-infra | 21:05 | |
| *** dprince has quit IRC | 21:06 | |
| *** julim has quit IRC | 21:06 | |
| *** markmcclain has quit IRC | 21:07 | |
| *** ^demon has quit IRC | 21:08 | |
| mikal | jog0: can I get a new revision of https://review.openstack.org/#/c/63478/ from you please? | 21:13 |
| *** mrda has joined #openstack-infra | 21:14 | |
| *** praneshp has quit IRC | 21:15 | |
| *** mfink has joined #openstack-infra | 21:16 | |
| *** madmike has quit IRC | 21:17 | |
| *** Hefeweizen has quit IRC | 21:17 | |
| fungi | mordred: added you back to openstack-ci-admins on lp for now. jeblair will have to add you back to openstack-admins later (he's the only admin of that group now) | 21:18 |
| *** senk1 has quit IRC | 21:18 | |
| mordred | fungi: k. thanks! | 21:18 |
| mordred | if anybody is bored and looking for things to do - https://review.openstack.org/#/c/56760/ | 21:19 |
| mordred | that ^^ doesn't actually work, but it would be nice and magical if a fairy flew in and fixed it | 21:19 |
| * fungi is fresh out of pixie dust | 21:21 | |
| *** gothicmindfood has quit IRC | 21:21 | |
| *** melwitt has joined #openstack-infra | 21:21 | |
| fungi | mordred: also remember that this revert happened... https://review.openstack.org/60042 | 21:23 |
| fungi | and https://review.openstack.org/56660 needed it | 21:24 |
| mordred | oh right | 21:24 |
| * fungi has been trying to keep state of all that in his notes | 21:25 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Stop running manage-projects https://review.openstack.org/66692 | 21:25 |
| mordred | fungi: well, luckily - wheels are enabled by default in pip 1.5 .. so the matrix of things we have to screw with is smaller to totally land everything | 21:25 |
| fungi | excellent | 21:25 |
| mordred | fungi: ^^ the patch above should block manage-projects from running until we get it fixed | 21:25 |
| mordred | fungi: I think that for now we can batch up and run by hand, and then figure out why it works by hand but not in puppet | 21:26 |
| mordred | fungi: yeah? | 21:26 |
| *** vipul-away is now known as vipul | 21:26 | |
| fungi | yeah | 21:26 |
| fungi | mordred: why does gerrit::remotes need to be absent? | 21:27 |
| *** _david_ has joined #openstack-infra | 21:27 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations https://review.openstack.org/66414 | 21:28 |
| mordred | fungi: to remote the cron job that runs manage-projects | 21:28 |
| fungi | THAT | 21:28 |
| fungi | right | 21:28 |
| fungi | i keep forgetting it's there, and then i see it in the syslog when i go to debug, and then i forget again | 21:28 |
| *** thuc has joined #openstack-infra | 21:28 | |
| *** thuc has quit IRC | 21:29 | |
| *** thuc has joined #openstack-infra | 21:29 | |
| mordred | :) | 21:30 |
| dstufft | fungi: mordred clarkb whoever, we're down to basically 1 issue before we start the 1.5.1/1.11/1 RC process for real, and that is basically just waiting on a guy to release a new version of distlib so we can vendor it | 21:30 |
| dstufft | just as an ETA kind of thing | 21:30 |
| mordred | dstufft: thank you! | 21:31 |
| *** senk has joined #openstack-infra | 21:31 | |
| *** ildikov_ has quit IRC | 21:32 | |
| dstufft | mordred: I know openstack deals with helping people new to python get involved some too, pip 1.5.1 comes with a much much much simplified bootstrap instruction too | 21:32 |
| mordred | dstufft: oh yeah? what is? | 21:32 |
| dstufft | get-pip.py basically handles everything for you, setuptools, pip, the whole 9 yards | 21:32 |
| mordred | dstufft: WOOT! | 21:33 |
| mordred | that's excellent | 21:33 |
| dstufft | it won't install virtualenv by default, but it'll get the package manager stuff setup for you | 21:33 |
| fungi | dstufft: stellar. sounds like our existing get-pip machinations will get even simpler | 21:33 |
| mordred | AaronGr_afk: ^^ the puppet module we have that hacks up the puppet pip provider should be able to be smaller now | 21:33 |
| fungi | yes, that | 21:33 |
| dstufft | this basically happened because pip itself no longer depends on setuptools to install from a Wheel | 21:34 |
| dstufft | (in 1.5.1 that is) | 21:34 |
| fungi | die setuptools die? | 21:34 |
| dstufft | it still uses setuptools for sdists | 21:34 |
| dstufft | but it's able to install setuptools itself now | 21:34 |
| dstufft | instead of needing to have it installed before hand | 21:34 |
| mordred | fungi: https://review.openstack.org/#/dashboard/9660 both of these are nice cleanups you may want to look at - AaronGr_afk is starting to walk through cleanup refactoring stuff | 21:34 |
| fungi | excellent. one more chicken-and-egg hatched before its laid | 21:34 |
| fungi | mordred: AaronGr_afk: i already had the first one starred. other added now. last week was not conducive to getting much reviewed | 21:35 |
| mordred | fungi: :) | 21:36 |
| mordred | fungi: no, I know it - and I could not be happier that you did, you know, real things | 21:36 |
| fungi | i think the main thing i accomplished was to not get around to approving more changes which could break everything ;) | 21:36 |
| mordred | hehe | 21:38 |
| *** rnirmal has quit IRC | 21:40 | |
| fungi | mordred: added you back to all the owner teams in github again | 21:40 |
| mordred | fungi: tahnks! | 21:44 |
| fungi | yw | 21:44 |
| *** senk has quit IRC | 21:45 | |
| *** senk has joined #openstack-infra | 21:46 | |
| mordred | fungi: also - AaronGr_afk and I are working on some package building stuff - what's teh status of the stuff from pabelanger? | 21:46 |
| mordred | I'd ove to not overlap | 21:46 |
| *** thomasem has quit IRC | 21:47 | |
| fungi | mordred: i think we added a packaging project parallel to one of our infra projects... zuul maybe? | 21:47 |
| mordred | fungi: and he has shown me https://github.com/rcrowley/freight#readme | 21:47 |
| mordred | fungi: which is apparently a modern and easy way to manage the package repos | 21:47 |
| fungi | mordred: http://git.openstack.org/cgit/openstack-infra/zuul-packaging/ | 21:47 |
| fungi | freight, huh? | 21:48 |
| pabelanger | I have some reviews up, but not much exposure on it: https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul-packaging,n,z | 21:48 |
| mordred | AaronGr_afk: we shoudl sync with pabelanger | 21:48 |
| mordred | because the machinery is something that we should want to be useful to infra's repos too | 21:48 |
| *** jhesketh_ has joined #openstack-infra | 21:48 | |
| fungi | mordred: so the high-level idea was a packaging repo per project being packaged, and each distro+release gets its own branch | 21:49 |
| *** markmc has quit IRC | 21:49 | |
| mordred | fungi: hrm. why repo per project? | 21:49 |
| mordred | oh | 21:49 |
| mordred | gah | 21:49 |
| mordred | wait | 21:49 |
| mordred | "packaging repo" | 21:49 |
| mordred | yes | 21:49 |
| mordred | with you | 21:49 |
| mordred | 100% | 21:49 |
| fungi | yes, sorry, git repo | 21:50 |
| mordred | no - you said it properly - I can't read | 21:50 |
| fungi | since, for example, packaging for debian-jessie would end up being potentially different from debian-wheezy and definitely different than fedora 20 | 21:50 |
| fungi | but they could still benefit from diverging from some common branches (branch debian-jessie off from debian-sid for example) | 21:50 |
| fungi | so it makes keeping the commit history a little easier | 21:51 |
| pabelanger | indeed | 21:51 |
| fungi | the thought was that pabelanger and others bang on the zuul-packaging git repo until the workflow emerges on its own, then codify that and start propagating more packaging git repos | 21:52 |
| *** AaronGr_afk is now known as AaronGr | 21:53 | |
| mordred | k. well, fyi - I have an internal deadline to have tools for it soonish | 21:53 |
| fungi | mordred: that's awesome | 21:53 |
| pabelanger | Ya, I haven't had much time to play with it lately. However, that will be changing shortly | 21:53 |
| mordred | so hopefully we can take a stab at workflow and stuff | 21:54 |
| *** dkliban has quit IRC | 21:56 | |
| pabelanger | one item on the workflow is how to handle chroots for building packages. I've used pbuilder in the past for debian / ubuntu, but others use cowbuilder. Honestly I can go either way, however building / maintain the chroot takes some time and energy. So, I think we'd need to get a specific jenkins-slave snapshot that runs up to date chroots nightly to avoid delays in builds | 21:57 |
| *** markmcclain2 has quit IRC | 21:58 | |
| *** gothicmindfood has joined #openstack-infra | 21:58 | |
| fungi | pabelanger: i'd lean toward a nodepool image which gets an updated set of chroots built nightly, then we can launch single-use slaves from it as needed | 21:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Added new SSH key for Monty https://review.openstack.org/66343 | 22:00 |
| fungi | mordred: ^ | 22:00 |
| mordred | woot | 22:00 |
| pabelanger | fungi, Ya, I think that might be best | 22:01 |
| mordred | fungi: ++ on single-use-slaves | 22:01 |
| fungi | long-running slaves have proven to be a liability, so the fewer of them we have doing generic tasks, the safer we'll be | 22:02 |
| *** jasondotstar has quit IRC | 22:02 | |
| *** sparkycollier has quit IRC | 22:04 | |
| *** smarcet has left #openstack-infra | 22:04 | |
| EmilienM | fungi: Hi, could you please tell me who else is devstack-gate core ? I would need https://review.openstack.org/#/c/65822/ and https://review.openstack.org/#/c/66142/ merged when possible. | 22:04 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations https://review.openstack.org/66414 | 22:04 |
| *** resker has quit IRC | 22:05 | |
| fungi | EmilienM: me, mordred, clarkb and jeblair at the moment | 22:08 |
| EmilienM | fungi: ok, I need it because tomorrow we start the neutron/tempest sprint, and we will need those plugins. | 22:09 |
| EmilienM | mordred: if you have 2 minutes for that, I would appreciate :-) | 22:09 |
| *** praneshp has joined #openstack-infra | 22:10 | |
| *** ^d has quit IRC | 22:11 | |
| fungi | EmilienM: if you could get some buy-off from neutron and qa on those patches, i'd be inclined to just approve them | 22:11 |
| *** praneshp_ has joined #openstack-infra | 22:11 | |
| *** ^d has joined #openstack-infra | 22:12 | |
| fungi | EmilienM: i wasn't involved in the previous disabling of those services in tempest, so don't have a lot of history/perspective on whether the situation has resolved itself | 22:12 |
| fungi | EmilienM: also, as i noted in review, those changes are going to conflict with one another, so will need to rebase one of them when the other merges (or rebase one on the other in review if you prefer) | 22:13 |
| EmilienM | fungi: yeah when one will be merged, I'll rebase | 22:14 |
| EmilienM | fungi: some buy-off? | 22:14 |
| fungi | EmilienM: confirmation from qa that this isn't going to cause the same problems which led to it being disabled the first time | 22:15 |
| *** praneshp has quit IRC | 22:15 | |
| *** praneshp_ is now known as praneshp | 22:15 | |
| EmilienM | fungi: they are not disabled, they just are not enabled by default in devstack :-) | 22:15 |
| *** sarob has quit IRC | 22:15 | |
| EmilienM | and in neutron too. | 22:15 |
| EmilienM | it's just plugins to add in neutron | 22:16 |
| *** sarob has joined #openstack-infra | 22:16 | |
| fungi | EmilienM: right. i gather we were testing them and had to stop because it was disruptive, but i don't recall the details. i think clarkb and sdague were involved in that | 22:16 |
| EmilienM | fungi: both plugins are working. I just don't know what to do more. We have to perform tempest tests, but if the plugins are disabled, i don't know what to do more :-) | 22:17 |
| fungi | mainly just wanted confirmation from neutron and qa devs that it was okay to reenable again | 22:17 |
| EmilienM | fungi: you have a +1 from enikanorov_ who is neutron-core | 22:18 |
| fungi | preferably who would confirm with a comment in the review that whatever cause us to remove it last time (whatever that was) has been addressed | 22:19 |
| *** lcestari has quit IRC | 22:20 | |
| *** jamielennox|away is now known as jamielennox | 22:20 | |
| fungi | i guess i'll see if i can git pickaxe to where it was removed before and see if i can find a bug or commit message to explain the circumstances | 22:20 |
| *** ruhe is now known as _ruhe | 22:21 | |
| EmilienM | fungi: tbh, i'm not sure the plugins have already been enabled in the past. metering & fwaas are quite new plugins in neutron | 22:22 |
| *** ^demon has joined #openstack-infra | 22:23 | |
| *** ^d has quit IRC | 22:24 | |
| *** Hefeweizen has joined #openstack-infra | 22:25 | |
| *** _david_ has left #openstack-infra | 22:25 | |
| *** julim has joined #openstack-infra | 22:29 | |
| fungi | EmilienM: agreed, hunting in git history of config, devstack-gate, devstack and tempest i'm not finding any record of q-fwaas having been previously disabled in some way. bug search is also turning up nothing, so i must have imagined it | 22:31 |
| *** mfer has quit IRC | 22:31 | |
| *** dkliban has joined #openstack-infra | 22:31 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Bump python-savannaclient version to >=0.4.1 https://review.openstack.org/66723 | 22:32 |
| *** ArxCruz has joined #openstack-infra | 22:33 | |
| SergeyLukjanov | fungi, mordred, could you please take a look on ^^ | 22:33 |
| *** dcramer_ has joined #openstack-infra | 22:34 | |
| *** ArxCruz has quit IRC | 22:35 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/pypi-mirror: Whitelist external packages https://review.openstack.org/66364 | 22:35 |
| *** SergeyLukjanov has quit IRC | 22:36 | |
| anteaya | voted | 22:36 |
| fungi | SergeyLukjanov: it looks like a more general requirements review, not anything infra-specific (there's a fairly large requirements-core group to review those). also, it's not going to make it through the requirements integration job in the gate until pypi-mirror fixes are in place | 22:36 |
| EmilienM | fungi: once rebased, I'll ping you to approve the metering one | 22:37 |
| fungi | EmilienM: okay | 22:37 |
| fungi | mordred: latest iteration of 66364 has the --process-dependency-links option on | 22:38 |
| *** sandywalsh has quit IRC | 22:38 | |
| fungi | mordred: i wonder now, though, whether that's going to cause old pip to fail to upgrade itself to new pip | 22:38 |
| fungi | guess i'll find out | 22:39 |
| * fungi needs to take a break to cook dinner... bbiab | 22:39 | |
| *** dcramer_ has quit IRC | 22:41 | |
| *** dims has quit IRC | 22:42 | |
| *** jcoufal has quit IRC | 22:43 | |
| openstackgerrit | A change was merged to openstack-infra/config: Set up openstackid module https://review.openstack.org/63316 | 22:46 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Add SQLalchemy database model https://review.openstack.org/62239 | 22:48 |
| *** mfink has quit IRC | 22:48 | |
| *** CaptTofu has joined #openstack-infra | 22:49 | |
| *** oubiwann has quit IRC | 22:50 | |
| *** herndon has quit IRC | 22:51 | |
| *** pcrews_ has quit IRC | 22:52 | |
| *** julim has quit IRC | 22:52 | |
| *** oubiwann has joined #openstack-infra | 22:52 | |
| *** hogepodge has quit IRC | 22:53 | |
| *** sandywalsh has joined #openstack-infra | 22:54 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added .gitignore for IntelliJ specific configuration files https://review.openstack.org/62956 | 22:54 |
| *** hogepodge has joined #openstack-infra | 22:55 | |
| *** jhesketh__ has joined #openstack-infra | 22:56 | |
| jhesketh__ | Morning | 22:56 |
| *** herndon has joined #openstack-infra | 22:57 | |
| *** dims_ has joined #openstack-infra | 22:57 | |
| *** markmcclain has joined #openstack-infra | 23:00 | |
| *** CaptTofu has quit IRC | 23:04 | |
| *** bknudson has quit IRC | 23:06 | |
| *** pelix has joined #openstack-infra | 23:07 | |
| *** rcleere has quit IRC | 23:08 | |
| *** prad has quit IRC | 23:09 | |
| *** CaptTofu has joined #openstack-infra | 23:10 | |
| *** burt has quit IRC | 23:11 | |
| *** kraman has quit IRC | 23:12 | |
| pelix | zaro: Re: https://review.openstack.org/#/c/48783/ removing the leading '/' results in the specified file being left out of the API documentation. I suspect it's because sphinx assumes paths starting with '/' are relative to the top level build dir, so if documentation is being built at two different levels (html/ and html/api/) you can't use purely relative paths. | 23:13 |
| *** denis_makogon has quit IRC | 23:16 | |
| *** dkranz has quit IRC | 23:17 | |
| *** CaptTofu has quit IRC | 23:18 | |
| *** CaptTofu has joined #openstack-infra | 23:19 | |
| fungi | jhesketh__: long time no see! | 23:19 |
| *** CaptTofu has quit IRC | 23:19 | |
| jhesketh__ | fungi: indeed.. I've been slack^W on leave and then at LCA | 23:19 |
| *** elasticio has quit IRC | 23:20 | |
| fungi | jhesketh__: no worries. mikal broke all your stuff while you were gone... enjoy ;) | 23:20 |
| jhesketh__ | heh, yeah, I saw | 23:20 |
| jhesketh__ | on the bright side, he's nearly done improving it | 23:21 |
| fungi | (not really--he did an admirable job) | 23:21 |
| jhesketh__ | indeed, I kinda turned on voting and ran away | 23:21 |
| fungi | i've pointed lots of random people at turbo-hipster, so maybe you'll get some outside contributions | 23:22 |
| jhesketh__ | yes, thanks for that :-) | 23:22 |
| fungi | mostly people who are running jenkins and question all the bloat when they use basically none of its extended features | 23:23 |
| fungi | also, it's written in a language which isn't java | 23:24 |
| jhesketh__ | :-) | 23:26 |
| *** sarob has quit IRC | 23:32 | |
| *** mrda has quit IRC | 23:33 | |
| *** eharney has quit IRC | 23:33 | |
| *** mfink has joined #openstack-infra | 23:34 | |
| *** CaptTofu has joined #openstack-infra | 23:36 | |
| *** ^d has joined #openstack-infra | 23:36 | |
| *** mrda has joined #openstack-infra | 23:36 | |
| *** ^demon has quit IRC | 23:38 | |
| *** nati_ueno has joined #openstack-infra | 23:38 | |
| *** dkliban has quit IRC | 23:42 | |
| *** thedodd has quit IRC | 23:42 | |
| zaro | pelix: wow, thanks. i didn't notice that. curently most of the jjb plugins that are using literal include do not have the leading '/' . probably should refactor that. | 23:45 |
| zaro | pelix: would you like to review 52080? then maybe i'll get another core to see if it passes muster. | 23:48 |
| *** herndon has quit IRC | 23:51 | |
| *** changbl has quit IRC | 23:54 | |
| *** markmcclain has quit IRC | 23:54 | |
| *** mrodden has quit IRC | 23:55 | |
| *** thuc has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!