| dougwig | jeblair: ok, thank you, i'll try it. is that exception in the log anything to worry about, or just noise? | 00:00 |
|---|---|---|
| adam_g | clarkb, who ends up using ssh keys/etc stashed to $BASE/new/.ssh/ in dsg ? $STACK_USER? | 00:01 |
| clarkb | adam_g: ya $BASE/new is the stack users homedir | 00:01 |
| clarkb | so it uses that key by default | 00:02 |
| adam_g | clarkb, ah, gotcha | 00:02 |
| *** emagana has quit IRC | 00:02 | |
| *** emagana has joined #openstack-infra | 00:02 | |
| *** Viswanath has joined #openstack-infra | 00:04 | |
| *** baoli has quit IRC | 00:04 | |
| harlowja | thx mtreinish | 00:05 |
| jeblair | dougwig: it's probably due to an earlier failed api call while syncing, it should recover the next time it tries to sync the affected change | 00:05 |
| fungi | ianw: clarkb: dougwig had an interesting suggestion on that change as well... we might should prepend to existing $PATH in case a platform expects unusual additional entries in it | 00:05 |
| jeblair | dougwig: (which you can force by hitting ctrl-r) | 00:05 |
| fungi | clarkb: also, i will be around in the morning to help with nodepool maintenance | 00:06 |
| *** melwitt has quit IRC | 00:06 | |
| clarkb | fungi: awesome | 00:07 |
| *** emagana has quit IRC | 00:07 | |
| *** emagana has joined #openstack-infra | 00:08 | |
| fungi | though for tonight i think i'm packing it in early | 00:08 |
| *** ddieterly has quit IRC | 00:08 | |
| ianw | fungi dougwig: response inline, but the point is more to standarise and not rely on funny things being done | 00:08 |
| *** Viswanath has quit IRC | 00:09 | |
| ianw | fungi: how many hours from now is morning for you? | 00:09 |
| clarkb | 11ish | 00:09 |
| fungi | ianw: sometime after i sleep and wake up again | 00:10 |
| *** emagana has quit IRC | 00:10 | |
| dougwig | ianw: unless you can standardize how the vendors generate their images (and you can't), carrying forward their PATH plus your minimum is my idea of standardizing. | 00:10 |
| *** emagana has joined #openstack-infra | 00:10 | |
| clarkb | er wait thats not right. 13ish? anyways jet lag has ruined my mornings | 00:10 |
| fungi | clarkb: roughly, yeah | 00:10 |
| fungi | also i've been unable to artificially recreate the manage-projects failure mode by leaving a MySQLdb connect() open to the trove instance from review.o.o | 00:11 |
| *** zz_dimtruck is now known as dimtruck | 00:11 | |
| fungi | so i think something is causing that socket to get prematurely closed, it's not a general timeouty | 00:11 |
| ianw | fungi: ok cool, very interested in the nodepool update for obvious reasons but i don't think there's much i can do | 00:11 |
| fungi | i can try manually applying the fix from 136079 on review.o.o and approving some project creation changes to see if it helps, but i'd like to have a better idea of why we're actually losing the socket | 00:12 |
| *** banix has joined #openstack-infra | 00:12 | |
| *** dimtruck is now known as zz_dimtruck | 00:16 | |
| clarkb | mtreinish: harlowja: I approved the stable compat job revert change | 00:16 |
| clarkb | so now we wait for that then we can put the taskflow ervert through | 00:16 |
| harlowja | k | 00:16 |
| notmyname | how does one test translations? also, where's the better place to ask that question? | 00:16 |
| harlowja | https://review.openstack.org/#/c/136174/ still going through the tubes | 00:16 |
| clarkb | notmyname: you can change $LANG for the processes you want to test with translations to the language you want to test | 00:17 |
| clarkb | notmyname: the translations list may have better ideas, but if you do that you should get text back that is in the language you specified | 00:17 |
| notmyname | clarkb: ie start it up with `LANG=foo /path/to/some/server` ? | 00:18 |
| clarkb | notmyname: yup | 00:18 |
| *** ryanpetrello has quit IRC | 00:18 | |
| *** pc_m has joined #openstack-infra | 00:18 | |
| *** r-daneel has quit IRC | 00:20 | |
| *** ZZelle_ has quit IRC | 00:21 | |
| *** alexpilotti has joined #openstack-infra | 00:21 | |
| lifeless | clarkb: notmyname: you can set domain at runntime, if this is for automated tests | 00:22 |
| notmyname | clarkb: lifeless: more for hand-tests right now | 00:23 |
| notmyname | clarkb: hmm...that's not working for me | 00:23 |
| notmyname | clarkb: does it have to be installed that way? | 00:24 |
| notmyname | or is it the whole systme locale? | 00:24 |
| *** tgohad has quit IRC | 00:24 | |
| *** ryanpetrello has joined #openstack-infra | 00:24 | |
| *** jungleboyj has joined #openstack-infra | 00:25 | |
| clarkb | notmyname: you do have to make sure that the .po files are installed in places where gettext looks for them | 00:26 |
| clarkb | notmyname: I am pretty sure that a pip install does not do this | 00:26 |
| *** radez is now known as radez_g0n3 | 00:27 | |
| lifeless | you may need 'mo's | 00:27 |
| openstackgerrit | Merged openstack-infra/project-config: Temporarily to remove stable compat jobs on taskflow https://review.openstack.org/136185 | 00:27 |
| clarkb | lifeless: I think it will compile them at run time but I may be mistaken about that | 00:27 |
| lifeless | *blink* _testcapi is present in my python2.7 distro install. | 00:29 |
| lifeless | Colour me verra surprised | 00:29 |
| *** david-lyle is now known as david-lyle_afk | 00:33 | |
| *** jungleboyj has quit IRC | 00:34 | |
| mtreinish | harlowja: ^^^ the stable compat revert landed if you want to fast approve the revert | 00:36 |
| mtreinish | then we can promote it to the top of the queue | 00:36 |
| harlowja | mtreinish https://review.openstack.org/#/c/136174/ still going | 00:36 |
| harlowja | or should i recheck? | 00:37 |
| *** ryanpetrello_ has joined #openstack-infra | 00:37 | |
| harlowja | *recheck to chop off the icehouse jobs that are still active | 00:37 |
| mtreinish | harlowja: it should be ok it doesn't look like the icehouse jobs are running | 00:37 |
| mtreinish | at least according to status.o.o | 00:37 |
| *** ryanpetrello has quit IRC | 00:38 | |
| *** ryanpetrello_ is now known as ryanpetrello | 00:38 | |
| harlowja | mtreinish k, lets wait, they hopefully won't start running | 00:38 |
| harlowja | https://pypi.python.org/pypi/gerrit-view#czuul :) | 00:38 |
| *** yamamoto has joined #openstack-infra | 00:43 | |
| *** ryanpetrello_ has joined #openstack-infra | 00:44 | |
| *** unicell has joined #openstack-infra | 00:44 | |
| dougwig | jeblair: confirmed your suggestion, thank you | 00:46 |
| *** bdpayne has quit IRC | 00:46 | |
| *** ryanpetrello has quit IRC | 00:47 | |
| *** emagana has quit IRC | 00:47 | |
| *** emagana has joined #openstack-infra | 00:48 | |
| *** yamamoto has quit IRC | 00:48 | |
| notmyname | clarkb: lifeless: https://wiki.openstack.org/wiki/Translations/HowToTest | 00:49 |
| *** sputnik13 has quit IRC | 00:50 | |
| *** ryanpetrello_ has quit IRC | 00:50 | |
| *** sputnik13 has joined #openstack-infra | 00:50 | |
| *** alexpilotti has quit IRC | 00:50 | |
| *** emagana has quit IRC | 00:52 | |
| *** sputnik13 has quit IRC | 00:52 | |
| *** sputnik13 has joined #openstack-infra | 00:53 | |
| clarkb | notmyname: so you did need to compile to mo and copy to a different location, but it worked after that? | 00:53 |
| clarkb | thats exciting :) | 00:54 |
| notmyname | clarkb: yup | 00:54 |
| notmyname | clarkb: and those instructions weren't written down anywhere (that was findable) | 00:54 |
| notmyname | I pulled them from a gerrit comment on a patch merged last year for a locale unit test we have in swift. fortunately the dev who wrote the test was sitting physically next to me and remembered it | 00:55 |
| *** banix has quit IRC | 00:58 | |
| *** wuhg has joined #openstack-infra | 01:01 | |
| *** sarob has quit IRC | 01:02 | |
| *** ddieterly has joined #openstack-infra | 01:13 | |
| mtreinish | harlowja: looks like 136174 came back if you want to +A it so it can get promoted | 01:14 |
| wuhg | 2014-11-20 14:52:25.909 | [SCP] uploading file: '/srv/static/logs/73/135273/6/experimental/check-tempest-dsvm-full-ceph/d80a5a8/logs/horizon_error.txt.gz' | 01:14 |
| wuhg | it jenkis run log http://logs.openstack.org/73/135273/6/experimental/check-tempest-dsvm-full-ceph/d80a5a8/console.html | 01:14 |
| wuhg | where can i get the horizon_error.txt.gz? | 01:15 |
| harlowja | mtreinish will do | 01:17 |
| harlowja | mtreinish +Aed | 01:18 |
| clarkb | wuhg: if you go to http://logs.openstack.org/73/135273/6/experimental/check-tempest-dsvm-full-ceph/d80a5a8/ you should have a logs dir | 01:18 |
| mtreinish | harlowja: cool, thanks | 01:18 |
| clarkb | wuhg: and in that dir will be the horizon error log | 01:18 |
| mtreinish | clarkb: ^^^ when you get a chance if you could promote 136174 | 01:18 |
| *** sputnik13 has quit IRC | 01:19 | |
| clarkb | it doesn't need promotion | 01:19 |
| mtreinish | oh it's in a separate queue, I didn't realize that | 01:19 |
| *** sputnik13 has joined #openstack-infra | 01:19 | |
| wuhg | clarkb:yes , i see it ,but can't open , | 01:20 |
| clarkb | wuhg: its empty | 01:21 |
| wuhg | horizon_error.txt.gz should be downloaded ,right | 01:21 |
| clarkb | it is but its empty | 01:21 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Revert "Temporarily to remove stable compat jobs on taskflow" https://review.openstack.org/136204 | 01:21 |
| *** sputnik13 has quit IRC | 01:21 | |
| *** sputnik13 has joined #openstack-infra | 01:22 | |
| mtreinish | wuhg: the horrizon error log is normally empty. There is only 1 thing during the run which uses horizon (it just logs in) and if that went wrong the test would fail | 01:22 |
| *** sputnik13 has quit IRC | 01:22 | |
| mtreinish | wuhg: but looking at the log that job hit the jenkins timeout well before the horizon test would have gotten executed | 01:23 |
| jamielennox | mtreinish: does the taskflow wrapt thing only affect stable, seems to break keystoneclient but only against neutron | 01:23 |
| *** jaypipes has quit IRC | 01:24 | |
| mtreinish | jamielennox: it should only affect jobs that use devstack icehouse (which is a surprising number of things) | 01:24 |
| mtreinish | jamielennox: looking at: https://review.openstack.org/#/c/97681/ it was the icehouse jobs that failed | 01:25 |
| wuhg | mtreinish:i have a horizon patch that jenkins have a verify error from check-tempest-dsvm-full-ceph | 01:25 |
| clarkb | wuhg: I think thats ok. that job is not voting and is experimental | 01:26 |
| jamielennox | mtreinish: ok, yea - it's the neutron/icehouse jobs that are failing all through there | 01:27 |
| wuhg | mtreinish:https://review.openstack.org/#/c/135273/, the only error in check-tempest-dsvm-full-ceph related to horizon is horizon_error.txt.gz | 01:27 |
| clarkb | wuhg: you can ignore that test, it has never passed | 01:27 |
| wuhg | ok ,thank you clark: mtreinish: | 01:28 |
| wuhg | clarkb: the the quenston is that jenkins desn't verify my patch https://review.openstack.org/#/c/135273/ | 01:30 |
| wuhg | clarkb:but all Jenkins check are success | 01:31 |
| clarkb | wuhg: its in the queue. see http://status.openstack.org/zuul/ | 01:31 |
| *** yamamoto_ has joined #openstack-infra | 01:32 | |
| *** dannywilson has quit IRC | 01:34 | |
| *** banix has joined #openstack-infra | 01:35 | |
| *** banix has quit IRC | 01:35 | |
| wuhg | clarkb:many thanks | 01:36 |
| wuhg | mtreinish:many thanks for the explain | 01:37 |
| *** jungleboyj has joined #openstack-infra | 01:38 | |
| *** sputnik13 has joined #openstack-infra | 01:38 | |
| *** jaypipes has joined #openstack-infra | 01:38 | |
| *** yaguang has joined #openstack-infra | 01:39 | |
| *** baoli has joined #openstack-infra | 01:40 | |
| *** salv-orlando has quit IRC | 01:41 | |
| *** rmcall has quit IRC | 01:45 | |
| *** banix has joined #openstack-infra | 01:57 | |
| *** ryanpetrello has joined #openstack-infra | 02:04 | |
| *** SumitNaiksatam has quit IRC | 02:06 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add mysql-proxy to enable read-only access to a db https://review.openstack.org/122311 | 02:06 |
| mtreinish | clarkb: ^^^ if you're still around do I need to add an iptables rule for the proxy on logstash.o.o there? | 02:07 |
| openstackgerrit | Abhishek Chanda proposed openstack/requirements: Add docker-py to global requirements https://review.openstack.org/136212 | 02:08 |
| *** Daisy has joined #openstack-infra | 02:09 | |
| *** yolanda has joined #openstack-infra | 02:09 | |
| *** ChuckC has quit IRC | 02:10 | |
| *** wenlock has joined #openstack-infra | 02:11 | |
| *** nosnos has joined #openstack-infra | 02:12 | |
| *** shashankhegde has quit IRC | 02:14 | |
| *** dims has joined #openstack-infra | 02:14 | |
| *** adalbas has quit IRC | 02:15 | |
| *** baoli has quit IRC | 02:16 | |
| *** stevemar has joined #openstack-infra | 02:18 | |
| *** banix has quit IRC | 02:19 | |
| *** krtaylor has quit IRC | 02:21 | |
| *** stevemar2 has quit IRC | 02:21 | |
| *** stevemar has quit IRC | 02:23 | |
| *** pc_m has quit IRC | 02:23 | |
| *** wenlock has quit IRC | 02:25 | |
| *** teran has quit IRC | 02:26 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Magnum is not ready for requirements check https://review.openstack.org/136214 | 02:26 |
| *** mudassirlatif has quit IRC | 02:28 | |
| *** krtaylor has joined #openstack-infra | 02:33 | |
| harlowja | mtreinish alright, https://review.openstack.org/#/c/136174/ is good to go | 02:35 |
| harlowja | so i guess can fix whatever tempest stuff is needed ? | 02:35 |
| *** dmsimard is now known as dmsimard_away | 02:35 | |
| harlowja | *or someone can fix* :-P | 02:35 |
| *** banix has joined #openstack-infra | 02:36 | |
| *** Longgeek has joined #openstack-infra | 02:37 | |
| *** teran has joined #openstack-infra | 02:38 | |
| *** rmcall has joined #openstack-infra | 02:38 | |
| *** patrickeast has quit IRC | 02:40 | |
| *** teran has quit IRC | 02:43 | |
| *** tsg has joined #openstack-infra | 02:43 | |
| *** Longgeek has quit IRC | 02:46 | |
| *** Ryan_Lane has quit IRC | 02:46 | |
| *** banix has quit IRC | 02:52 | |
| *** ivar-laz_ has joined #openstack-infra | 02:52 | |
| wuhg | what is the column Workflow in the Reviewer table from https://review.openstack.org/#/c/136156/ | 02:53 |
| wuhg | what does the column Workflow do? | 02:54 |
| nibalizer | its where the 'work in progress' flag would be | 02:54 |
| nibalizer | or the 'approved' flag | 02:54 |
| *** ivar-lazzaro has quit IRC | 02:55 | |
| *** ivar-laz_ has quit IRC | 02:56 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:00 | |
| *** Daisy has quit IRC | 03:00 | |
| *** ryanpetrello has quit IRC | 03:01 | |
| *** nelsnelson has quit IRC | 03:01 | |
| *** ryanpetrello has joined #openstack-infra | 03:04 | |
| wuhg | ok | 03:05 |
| *** ujuc has joined #openstack-infra | 03:05 | |
| *** sarob has joined #openstack-infra | 03:06 | |
| *** stevemar has joined #openstack-infra | 03:06 | |
| *** koolhead17 has joined #openstack-infra | 03:08 | |
| *** camunoz has quit IRC | 03:11 | |
| *** dims has quit IRC | 03:11 | |
| *** pcrews has quit IRC | 03:13 | |
| *** dims has joined #openstack-infra | 03:13 | |
| mtreinish | harlowja: yep, the tempest on icehouse fix was already in progress https://review.openstack.org/136154 | 03:14 |
| mtreinish | I'm not sure how prevelent it was but it bounced the reqs change from the gate twice | 03:14 |
| *** ryanpetrello has quit IRC | 03:16 | |
| *** yamahata has joined #openstack-infra | 03:16 | |
| *** amcrn has quit IRC | 03:17 | |
| *** ryanpetrello has joined #openstack-infra | 03:17 | |
| *** Daisy has joined #openstack-infra | 03:19 | |
| *** nosnos has quit IRC | 03:24 | |
| zxiiro | Hi, is anyone here available to help me with my account? | 03:26 |
| *** iain-king has quit IRC | 03:27 | |
| *** dannywilson has joined #openstack-infra | 03:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Cleanup gerrit ACLs https://review.openstack.org/136222 | 03:31 |
| *** ryanpetrello has quit IRC | 03:31 | |
| *** dannywil_ has joined #openstack-infra | 03:32 | |
| zxiiro | nevermind i figured it out :) | 03:33 |
| *** banix has joined #openstack-infra | 03:33 | |
| *** dannywilson has quit IRC | 03:34 | |
| *** dannywil_ has quit IRC | 03:37 | |
| *** dannywilson has joined #openstack-infra | 03:38 | |
| *** teran has joined #openstack-infra | 03:39 | |
| *** SumitNaiksatam has quit IRC | 03:39 | |
| *** harlowja is now known as harlowja_away | 03:39 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Revert "Temporarily to remove stable compat jobs on taskflow" https://review.openstack.org/136204 | 03:41 |
| *** dannywilson has quit IRC | 03:42 | |
| *** Longgeek has joined #openstack-infra | 03:42 | |
| *** teran has quit IRC | 03:44 | |
| *** SumitNaiksatam has quit IRC | 03:46 | |
| *** banix has quit IRC | 03:47 | |
| *** Longgeek has quit IRC | 03:47 | |
| *** teran has joined #openstack-infra | 03:48 | |
| *** dims has quit IRC | 03:48 | |
| *** teran_ has joined #openstack-infra | 03:48 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch dsvm-docker as a Nova check jobs https://review.openstack.org/136115 | 03:50 |
| *** SumitNaiksatam has joined #openstack-infra | 03:50 | |
| *** teran has quit IRC | 03:52 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for Config File Provider to Maven Project module https://review.openstack.org/136225 | 03:54 |
| *** ChuckC has joined #openstack-infra | 03:58 | |
| *** otherwiseguy has joined #openstack-infra | 04:00 | |
| *** Viswanath has joined #openstack-infra | 04:09 | |
| *** mbacchi has quit IRC | 04:11 | |
| *** Viswanath has quit IRC | 04:13 | |
| *** banix has joined #openstack-infra | 04:24 | |
| *** oomichi_ has joined #openstack-infra | 04:27 | |
| *** dannywilson has joined #openstack-infra | 04:27 | |
| *** sdake has quit IRC | 04:28 | |
| *** nosnos has joined #openstack-infra | 04:29 | |
| *** banix has quit IRC | 04:29 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for Config File Provider to Maven Project module https://review.openstack.org/136225 | 04:35 |
| *** rmcall has quit IRC | 04:37 | |
| *** AlexF has joined #openstack-infra | 04:38 | |
| *** Daisy has quit IRC | 04:40 | |
| *** dannywilson has quit IRC | 04:41 | |
| *** koolhead17 has quit IRC | 04:41 | |
| *** dannywilson has joined #openstack-infra | 04:41 | |
| *** rushiagr_away is now known as rushiagr | 04:43 | |
| *** david-ly_ has joined #openstack-infra | 04:43 | |
| *** david-lyle_afk has quit IRC | 04:43 | |
| *** Longgeek has joined #openstack-infra | 04:43 | |
| *** dannywilson has quit IRC | 04:46 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: New Gerrit ACL for puppet-openstack-cloud https://review.openstack.org/136098 | 04:46 |
| *** AlexF has quit IRC | 04:47 | |
| *** BharatK has joined #openstack-infra | 04:47 | |
| *** Longgeek has quit IRC | 04:48 | |
| *** dims has joined #openstack-infra | 04:49 | |
| *** stevemar has quit IRC | 04:51 | |
| *** stevemar has joined #openstack-infra | 04:52 | |
| *** oomichi_ has quit IRC | 04:52 | |
| wuhg | hi, all , how do i know if bugs Fix has published in a milestone? | 04:54 |
| *** ryanpetrello has joined #openstack-infra | 04:54 | |
| *** dims has quit IRC | 04:55 | |
| *** sdake has joined #openstack-infra | 04:56 | |
| *** sdake has quit IRC | 04:56 | |
| *** sdake has joined #openstack-infra | 04:56 | |
| *** sarob has quit IRC | 04:59 | |
| *** AlexF has joined #openstack-infra | 04:59 | |
| *** sarob has joined #openstack-infra | 05:00 | |
| *** Daisy has joined #openstack-infra | 05:01 | |
| *** marun has quit IRC | 05:02 | |
| *** sdake has quit IRC | 05:02 | |
| *** fandi has quit IRC | 05:03 | |
| *** sarob has quit IRC | 05:04 | |
| *** AlexF has quit IRC | 05:05 | |
| *** yfried has quit IRC | 05:05 | |
| *** yfried has joined #openstack-infra | 05:05 | |
| *** sdake has joined #openstack-infra | 05:06 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add nodepool script to preseed testrepository from subunit2sql https://review.openstack.org/136234 | 05:07 |
| *** dannywilson has joined #openstack-infra | 05:08 | |
| *** marun has joined #openstack-infra | 05:09 | |
| *** garyh has quit IRC | 05:09 | |
| *** sarob has joined #openstack-infra | 05:11 | |
| *** otherwiseguy has quit IRC | 05:11 | |
| *** sarob has quit IRC | 05:12 | |
| *** nosnos has quit IRC | 05:14 | |
| *** koolhead17 has joined #openstack-infra | 05:15 | |
| *** koolhead17 has joined #openstack-infra | 05:15 | |
| *** sdake has quit IRC | 05:15 | |
| *** nosnos has joined #openstack-infra | 05:16 | |
| *** shashankhegde has joined #openstack-infra | 05:18 | |
| *** zul has joined #openstack-infra | 05:20 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for Config File Provider to Maven Project module https://review.openstack.org/136225 | 05:29 |
| *** dannywilson has quit IRC | 05:32 | |
| *** dannywilson has joined #openstack-infra | 05:32 | |
| *** AlexF has joined #openstack-infra | 05:33 | |
| *** mrmartin has joined #openstack-infra | 05:36 | |
| openstackgerrit | daisy-ycguo proposed openstack-infra/project-config: Add a new project in stackforge named as i18n-test to maintain the test codes for i18n and l10n functions. https://review.openstack.org/135857 | 05:37 |
| *** dannywilson has quit IRC | 05:37 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for Config File Provider to Maven Project module https://review.openstack.org/136225 | 05:38 |
| *** AlexF has quit IRC | 05:40 | |
| *** AlexF has joined #openstack-infra | 05:40 | |
| *** garyh has joined #openstack-infra | 05:44 | |
| *** sdake has joined #openstack-infra | 05:48 | |
| *** iain-king has joined #openstack-infra | 05:51 | |
| *** harlowja_at_home has joined #openstack-infra | 05:56 | |
| *** zul has quit IRC | 05:57 | |
| *** nosnos has quit IRC | 05:59 | |
| *** teran_ has quit IRC | 05:59 | |
| *** nosnos has joined #openstack-infra | 06:00 | |
| *** jcoufal has joined #openstack-infra | 06:03 | |
| *** chandankumar has joined #openstack-infra | 06:04 | |
| *** yamahata has quit IRC | 06:04 | |
| *** yamahata has joined #openstack-infra | 06:05 | |
| *** Longgeek has joined #openstack-infra | 06:09 | |
| *** AlexF has quit IRC | 06:10 | |
| *** harlowja_at_home has quit IRC | 06:13 | |
| *** tsg has quit IRC | 06:15 | |
| *** yfried is now known as yfried|afk | 06:16 | |
| *** k4n0 has joined #openstack-infra | 06:17 | |
| *** shashankhegde has quit IRC | 06:20 | |
| *** talluri has joined #openstack-infra | 06:28 | |
| *** yfried|afk is now known as yfried | 06:36 | |
| *** AlexF has joined #openstack-infra | 06:40 | |
| *** ildikov has quit IRC | 06:42 | |
| *** changbl has joined #openstack-infra | 06:44 | |
| *** koolhead17 has quit IRC | 06:44 | |
| *** koolhead17 has joined #openstack-infra | 06:45 | |
| *** koolhead17 has quit IRC | 06:46 | |
| *** koolhead17 has joined #openstack-infra | 06:46 | |
| *** henditan has joined #openstack-infra | 06:56 | |
| *** bdpayne has joined #openstack-infra | 06:56 | |
| *** afazekas has joined #openstack-infra | 06:58 | |
| *** AlexF has quit IRC | 06:59 | |
| *** bogdando has quit IRC | 07:00 | |
| *** yfried is now known as yfried|afk | 07:00 | |
| *** henditan has quit IRC | 07:02 | |
| *** henditan has joined #openstack-infra | 07:03 | |
| *** dannywilson has joined #openstack-infra | 07:05 | |
| *** ryanpetrello has quit IRC | 07:05 | |
| *** henditan has quit IRC | 07:07 | |
| *** henditan has joined #openstack-infra | 07:08 | |
| *** e0ne has joined #openstack-infra | 07:08 | |
| *** yfried|afk is now known as yfried | 07:09 | |
| *** dannywil_ has joined #openstack-infra | 07:10 | |
| *** jgjl has joined #openstack-infra | 07:13 | |
| *** henditan has quit IRC | 07:13 | |
| *** dannywilson has quit IRC | 07:13 | |
| *** stevemar has quit IRC | 07:16 | |
| *** talluri has quit IRC | 07:16 | |
| *** henditan has joined #openstack-infra | 07:18 | |
| *** mmaglana has quit IRC | 07:19 | |
| *** yfried is now known as yfried|afk | 07:20 | |
| *** henditan has quit IRC | 07:20 | |
| *** jgjl has quit IRC | 07:27 | |
| *** e0ne has quit IRC | 07:28 | |
| *** dannywil_ has quit IRC | 07:30 | |
| *** dannywilson has joined #openstack-infra | 07:31 | |
| *** koolhead17 has quit IRC | 07:33 | |
| *** ildikov has joined #openstack-infra | 07:34 | |
| *** hdd has joined #openstack-infra | 07:34 | |
| *** dannywilson has quit IRC | 07:35 | |
| *** achanda has quit IRC | 07:38 | |
| *** bhuvan has quit IRC | 07:39 | |
| *** bradjones has quit IRC | 07:40 | |
| *** yfried|afk is now known as yfried | 07:40 | |
| *** e0ne has joined #openstack-infra | 07:40 | |
| *** HeOS has quit IRC | 07:40 | |
| *** bradjones has joined #openstack-infra | 07:40 | |
| *** e0ne has quit IRC | 07:41 | |
| *** unicell has quit IRC | 07:41 | |
| *** unicell has joined #openstack-infra | 07:42 | |
| *** yamamoto_ has quit IRC | 07:44 | |
| *** yfried is now known as yfried|afk | 07:50 | |
| *** henditan has joined #openstack-infra | 07:53 | |
| *** teran has joined #openstack-infra | 07:55 | |
| *** changbl has quit IRC | 07:56 | |
| *** nadya_ has joined #openstack-infra | 07:58 | |
| *** henditan has quit IRC | 07:59 | |
| *** henditan has joined #openstack-infra | 07:59 | |
| *** nadya_ has quit IRC | 08:00 | |
| *** henditan has quit IRC | 08:00 | |
| *** koolhead17 has joined #openstack-infra | 08:00 | |
| *** henditan has joined #openstack-infra | 08:01 | |
| *** yamamoto_ has joined #openstack-infra | 08:01 | |
| *** ZZelle has quit IRC | 08:01 | |
| *** achuprin_ has quit IRC | 08:02 | |
| *** ZZelle has joined #openstack-infra | 08:02 | |
| *** koolhead_ has joined #openstack-infra | 08:11 | |
| *** koolhead17 has quit IRC | 08:11 | |
| *** sarob has joined #openstack-infra | 08:12 | |
| *** achuprin_ has joined #openstack-infra | 08:16 | |
| *** sarob has quit IRC | 08:17 | |
| *** Daisy has quit IRC | 08:19 | |
| *** henditan has quit IRC | 08:19 | |
| *** henditan has joined #openstack-infra | 08:20 | |
| *** otter768 has quit IRC | 08:20 | |
| *** reed has joined #openstack-infra | 08:21 | |
| *** matrohon has joined #openstack-infra | 08:21 | |
| *** mrmartin has quit IRC | 08:23 | |
| *** nosnos has quit IRC | 08:24 | |
| *** henditan has quit IRC | 08:24 | |
| *** nosnos has joined #openstack-infra | 08:24 | |
| *** henditan has joined #openstack-infra | 08:24 | |
| *** salv-orlando has joined #openstack-infra | 08:24 | |
| *** yfried|afk is now known as yfried | 08:28 | |
| *** bogdando has joined #openstack-infra | 08:29 | |
| *** iain-king has quit IRC | 08:30 | |
| *** reed_ has joined #openstack-infra | 08:37 | |
| *** MaxV has joined #openstack-infra | 08:37 | |
| *** reed_ has quit IRC | 08:38 | |
| *** achanda has joined #openstack-infra | 08:38 | |
| *** yfried is now known as yfried|afk | 08:38 | |
| *** links has joined #openstack-infra | 08:39 | |
| *** achanda_ has joined #openstack-infra | 08:40 | |
| *** achanda has quit IRC | 08:40 | |
| *** achanda_ is now known as achanda | 08:41 | |
| *** arxcruz has joined #openstack-infra | 08:45 | |
| *** jgallard_ has joined #openstack-infra | 08:46 | |
| *** yamamoto_ has quit IRC | 08:46 | |
| *** henditan has quit IRC | 08:47 | |
| *** BharatK has quit IRC | 08:47 | |
| *** henditan has joined #openstack-infra | 08:48 | |
| *** talluri has joined #openstack-infra | 08:48 | |
| *** henditan has quit IRC | 08:48 | |
| *** henditan has joined #openstack-infra | 08:49 | |
| *** yamamoto_ has joined #openstack-infra | 08:49 | |
| *** armax has quit IRC | 08:49 | |
| *** henditan has quit IRC | 08:49 | |
| *** nosnos has quit IRC | 08:51 | |
| *** nosnos has joined #openstack-infra | 08:51 | |
| *** jedimike has joined #openstack-infra | 08:51 | |
| *** jpich has joined #openstack-infra | 08:57 | |
| *** pblaho has joined #openstack-infra | 08:59 | |
| *** HeOS has joined #openstack-infra | 09:03 | |
| *** nosnos has quit IRC | 09:06 | |
| *** e0ne has joined #openstack-infra | 09:06 | |
| *** Guest63270 has joined #openstack-infra | 09:06 | |
| *** ujuc has quit IRC | 09:06 | |
| *** yamamoto_ has quit IRC | 09:07 | |
| *** koolhead_ has quit IRC | 09:07 | |
| *** koolhead17 has joined #openstack-infra | 09:08 | |
| *** koolhead_ has joined #openstack-infra | 09:09 | |
| *** BharatK has joined #openstack-infra | 09:10 | |
| *** mrmartin has joined #openstack-infra | 09:10 | |
| *** nadya_ has joined #openstack-infra | 09:10 | |
| *** koolhead17 has quit IRC | 09:12 | |
| *** SumitNaiksatam has quit IRC | 09:16 | |
| *** SumitNaiksatam has joined #openstack-infra | 09:16 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/devstack-gate: Add is_debian and is_ubuntu_or_debian functions https://review.openstack.org/130768 | 09:18 |
| openstackgerrit | Julie Pichon proposed openstack-infra/project-config: Add integration tests to the Horizon check gate https://review.openstack.org/136269 | 09:20 |
| *** Daisy has joined #openstack-infra | 09:23 | |
| *** jlibosva has joined #openstack-infra | 09:24 | |
| *** katyafervent is now known as katyafervent_awa | 09:27 | |
| *** andreykurilin_ has joined #openstack-infra | 09:30 | |
| *** IvanBerezovskiy has joined #openstack-infra | 09:30 | |
| *** dmellado_ has joined #openstack-infra | 09:31 | |
| *** dmellado_ has quit IRC | 09:32 | |
| *** achanda has quit IRC | 09:33 | |
| *** nosnos has joined #openstack-infra | 09:34 | |
| *** BharatK has quit IRC | 09:34 | |
| *** derekh has joined #openstack-infra | 09:35 | |
| *** dtantsur|afk is now known as dtantsur | 09:37 | |
| *** talluri has quit IRC | 09:39 | |
| *** BharatK has joined #openstack-infra | 09:39 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:40 | |
| *** jp_at_hp has joined #openstack-infra | 09:42 | |
| *** katyafervent_awa is now known as katyafervent | 09:43 | |
| *** mpaolino has joined #openstack-infra | 09:47 | |
| *** yamahata has quit IRC | 09:50 | |
| *** Longgeek has quit IRC | 09:51 | |
| *** dmellado has joined #openstack-infra | 09:52 | |
| *** k4n0 has quit IRC | 09:53 | |
| openstackgerrit | Mike Heald proposed openstack-infra/zuul: Fix tests for GitPython>3.2.0.RC1 https://review.openstack.org/136281 | 09:54 |
| *** Daisy has quit IRC | 09:56 | |
| *** koolhead_ has quit IRC | 09:56 | |
| *** koolhead17 has joined #openstack-infra | 09:57 | |
| *** koolhead_ has joined #openstack-infra | 09:58 | |
| *** vryzhenkin has joined #openstack-infra | 09:58 | |
| *** Masahiroo has joined #openstack-infra | 09:59 | |
| openstackgerrit | Mike Heald proposed openstack-infra/zuul: Fix tests for GitPython>0.3.2.RC1 https://review.openstack.org/136281 | 09:59 |
| *** ihrachyshka has joined #openstack-infra | 09:59 | |
| *** vryzhenkin has quit IRC | 10:01 | |
| *** vrutkovs has joined #openstack-infra | 10:01 | |
| *** vryzhenkin has joined #openstack-infra | 10:01 | |
| *** rlandy has joined #openstack-infra | 10:01 | |
| *** koolhead17 has quit IRC | 10:02 | |
| openstackgerrit | Merged openstack-infra/zuul: Enforce ref only for gerrit events that supply a ref https://review.openstack.org/114510 | 10:03 |
| *** Longgeek has joined #openstack-infra | 10:04 | |
| *** salv-orlando has quit IRC | 10:06 | |
| *** yamamoto_ has joined #openstack-infra | 10:07 | |
| *** ihrachyshka has quit IRC | 10:08 | |
| *** yamamoto_ has quit IRC | 10:12 | |
| openstackgerrit | Mike Heald proposed openstack-infra/zuul: Fix tests for GitPython>0.3.2.RC1 https://review.openstack.org/136281 | 10:15 |
| *** talluri has joined #openstack-infra | 10:15 | |
| *** talluri has quit IRC | 10:17 | |
| *** talluri_ has joined #openstack-infra | 10:17 | |
| *** talluri_ has quit IRC | 10:18 | |
| *** talluri has joined #openstack-infra | 10:18 | |
| *** unicell has quit IRC | 10:19 | |
| *** andreykurilin_ has quit IRC | 10:20 | |
| *** otter768 has joined #openstack-infra | 10:21 | |
| *** Masahiroo has quit IRC | 10:21 | |
| *** andreykurilin_ has joined #openstack-infra | 10:22 | |
| *** yfried|afk is now known as yfried | 10:24 | |
| *** jcoufal has quit IRC | 10:24 | |
| *** jcoufal has joined #openstack-infra | 10:25 | |
| *** otter768 has quit IRC | 10:25 | |
| *** salv-orlando has joined #openstack-infra | 10:27 | |
| *** ldnunes has joined #openstack-infra | 10:35 | |
| *** dtantsur is now known as dtantsur|brb | 10:41 | |
| *** sergsh_away is now known as sergsh | 10:43 | |
| *** isviridov_away is now known as isvirido | 10:43 | |
| *** isvirido is now known as isviridov | 10:43 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-storyboard: Integration Testing https://review.openstack.org/126086 | 10:49 |
| openstackgerrit | Ann Kamyshnikova proposed openstack/requirements: Upgrade oslo.db>=1.1.0 https://review.openstack.org/136296 | 10:53 |
| *** skolekonov has joined #openstack-infra | 11:02 | |
| *** mase_x200 has joined #openstack-infra | 11:03 | |
| *** dims has joined #openstack-infra | 11:11 | |
| *** cnesa has joined #openstack-infra | 11:14 | |
| *** jgallard_ has quit IRC | 11:15 | |
| *** adalbas has joined #openstack-infra | 11:16 | |
| *** pelix has joined #openstack-infra | 11:19 | |
| *** mase_x200 has quit IRC | 11:22 | |
| *** yfried is now known as yfried|afk | 11:27 | |
| *** pblaho has quit IRC | 11:27 | |
| *** Guest63270 has quit IRC | 11:30 | |
| *** mase_x200 has joined #openstack-infra | 11:33 | |
| *** talluri has quit IRC | 11:34 | |
| *** che-arn5 has joined #openstack-infra | 11:35 | |
| *** che-arne has joined #openstack-infra | 11:35 | |
| *** che-arne has quit IRC | 11:35 | |
| *** hashar has joined #openstack-infra | 11:37 | |
| *** doug-fish has joined #openstack-infra | 11:40 | |
| *** hashar has quit IRC | 11:41 | |
| *** pc_m has joined #openstack-infra | 11:42 | |
| bookwar | why this patch here https://review.openstack.org/#/c/131770/1 is not merged? I don't see it in zuul queues or something.. | 11:43 |
| nibalizer | oh thats super old | 11:45 |
| nibalizer | maybe try a recheck ? | 11:46 |
| *** yfried|afk is now known as yfried | 11:47 | |
| *** HeOS has quit IRC | 11:51 | |
| *** asettle has quit IRC | 11:56 | |
| *** mpaolino has quit IRC | 11:57 | |
| *** nadya_ has quit IRC | 11:58 | |
| *** jcoufal has quit IRC | 11:58 | |
| *** mase_x200 has quit IRC | 12:01 | |
| *** HeOS has joined #openstack-infra | 12:01 | |
| dims | sdague: quick one when you get a chance - need to disable requirements job - https://review.openstack.org/#/c/136214/ | 12:02 |
| sdague | dims: +A | 12:05 |
| dims | thanks! | 12:05 |
| *** BharatK has left #openstack-infra | 12:07 | |
| *** BharatK has joined #openstack-infra | 12:08 | |
| *** uvirtbot has quit IRC | 12:09 | |
| *** asettle has joined #openstack-infra | 12:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Magnum is not ready for requirements check https://review.openstack.org/136214 | 12:12 |
| *** alexpilotti has joined #openstack-infra | 12:12 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client https://review.openstack.org/136004 | 12:13 |
| *** rediskin has joined #openstack-infra | 12:18 | |
| *** talluri has joined #openstack-infra | 12:19 | |
| *** dmellado is now known as dmellado|afk | 12:22 | |
| *** otter768 has joined #openstack-infra | 12:22 | |
| *** talluri has quit IRC | 12:24 | |
| *** dmellado|afk is now known as dmellado | 12:24 | |
| *** otter768 has quit IRC | 12:27 | |
| *** BharatK has left #openstack-infra | 12:29 | |
| *** dmsimard_away is now known as dmsimard | 12:33 | |
| *** baoli has joined #openstack-infra | 12:33 | |
| *** baoli has quit IRC | 12:34 | |
| *** baoli has joined #openstack-infra | 12:34 | |
| *** che-arn5 has quit IRC | 12:41 | |
| *** cnesa has quit IRC | 12:42 | |
| *** andreykurilin_ has quit IRC | 12:44 | |
| *** jamespage_ has joined #openstack-infra | 12:45 | |
| openstackgerrit | Nikita Konovalov proposed openstack-infra/project-config: Setting up a StoryBoard python client https://review.openstack.org/136004 | 12:49 |
| *** dmsimard is now known as dmsimard_away | 12:49 | |
| *** hdd has quit IRC | 12:49 | |
| *** weshay has joined #openstack-infra | 12:50 | |
| *** dprince has joined #openstack-infra | 12:50 | |
| *** julim has joined #openstack-infra | 12:51 | |
| *** katyafervent is now known as katyafervent_awa | 12:51 | |
| *** katyafervent_awa is now known as katyafervent | 12:52 | |
| openstackgerrit | Nikolay Fedotov proposed openstack-infra/system-config: Remove 'git_dir' and 'zuul_url' from [zuul] section https://review.openstack.org/136325 | 12:52 |
| *** mpaolino has joined #openstack-infra | 12:56 | |
| *** yfried is now known as yfried|afk | 12:59 | |
| *** mrmartin has quit IRC | 13:07 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-storyboard: Integration Testing https://review.openstack.org/126086 | 13:08 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-storyboard: Integration Testing https://review.openstack.org/126086 | 13:09 |
| *** yamamoto has joined #openstack-infra | 13:10 | |
| *** weshay has quit IRC | 13:13 | |
| *** doug-fish has quit IRC | 13:13 | |
| *** doug-fish has joined #openstack-infra | 13:14 | |
| *** bswartz has quit IRC | 13:15 | |
| *** jamespage_ has quit IRC | 13:15 | |
| *** yamamoto has quit IRC | 13:15 | |
| NikitaKonovalov | hi, can anyone help registering a project on pypi? | 13:16 |
| *** nosnos has quit IRC | 13:16 | |
| NikitaKonovalov | The project is python-storyboardclient https://review.openstack.org/#/c/136004 | 13:16 |
| *** nadya_ has joined #openstack-infra | 13:16 | |
| *** yfried|afk is now known as yfried | 13:19 | |
| *** talluri has joined #openstack-infra | 13:20 | |
| *** jamielennox is now known as jamielennox|away | 13:20 | |
| *** yamamoto has joined #openstack-infra | 13:20 | |
| *** ala_ has joined #openstack-infra | 13:22 | |
| *** bknudson has quit IRC | 13:23 | |
| *** tsg_ has joined #openstack-infra | 13:24 | |
| *** talluri has quit IRC | 13:24 | |
| *** koolhead_ has quit IRC | 13:27 | |
| *** dtantsur|brb is now known as dtantsur | 13:27 | |
| *** koolhead17 has joined #openstack-infra | 13:27 | |
| *** mbacchi has joined #openstack-infra | 13:27 | |
| *** akuznetsova has joined #openstack-infra | 13:27 | |
| *** dmakogon_ is now known as denis_makogon | 13:28 | |
| *** dmsimard_away is now known as dmsimard | 13:28 | |
| *** yfried is now known as yfried|afk | 13:29 | |
| *** yamamoto has quit IRC | 13:29 | |
| akuznetsova | Hi guys! I have a question. I need that "Closes-Bug" works on commit to some repo. Do you know what I have to configure for that? | 13:29 |
| *** koolhead17 has quit IRC | 13:32 | |
| *** dmellado is now known as dmellado|afk | 13:34 | |
| *** hdd has joined #openstack-infra | 13:37 | |
| *** rushiagr is now known as rushiagr_away | 13:38 | |
| tchaypo | I don’t know the answer | 13:41 |
| tchaypo | but http://ci.openstack.org/gerrit.html#launchpad-bug-integration looks relevant | 13:41 |
| *** dkranz has joined #openstack-infra | 13:41 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Temporarily to remove stable compat jobs on taskflow" https://review.openstack.org/136204 | 13:42 |
| tchaypo | from http://ci.openstack.org/jeepyb.html#launchpad-bug-integration it sounds like you need to set up a commit hook. the doc is silent on how to do that (or at least, doesn’t link to the docs on how to do that) | 13:43 |
| *** banix has joined #openstack-infra | 13:48 | |
| *** jgallard_ has joined #openstack-infra | 13:49 | |
| *** dims is now known as dimsum__ | 13:49 | |
| *** bknudson has joined #openstack-infra | 13:51 | |
| openstackgerrit | Julie Pichon proposed openstack-infra/project-config: Add integration tests to the Horizon check gate https://review.openstack.org/136269 | 13:53 |
| *** mpaolino has quit IRC | 13:53 | |
| *** radez_g0n3 is now known as radez | 13:55 | |
| *** che-arne has joined #openstack-infra | 13:55 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: New Gerrit ACL for puppet-openstack-cloud https://review.openstack.org/136098 | 13:56 |
| *** julim has quit IRC | 13:56 | |
| *** cpowell has joined #openstack-infra | 13:56 | |
| *** dkranz has quit IRC | 13:57 | |
| *** tnovacik has joined #openstack-infra | 13:58 | |
| anteaya | akuznetsova tchaypo a patch to update the commit message syntax and hooks merged last august/september and I have been looking for it but so far I can't find it | 13:58 |
| anteaya | and I don't know where that code lives otherwise I would point to it | 13:58 |
| *** dmellado has joined #openstack-infra | 13:58 | |
| *** julim has joined #openstack-infra | 13:59 | |
| *** mjturek has joined #openstack-infra | 13:59 | |
| anteaya | NikitaKonovalov: I have never registered a project on pypi, so I don't know, the only part that needs to be registered is the name as far as I know | 13:59 |
| anteaya | NikitaKonovalov: are you being asked for information other than the name of the project? | 13:59 |
| *** rlandy has quit IRC | 14:00 | |
| anteaya | NikitaKonovalov: for instance here is the python-novaclient page: https://pypi.python.org/pypi/python-novaclient/2.20.0 | 14:01 |
| *** weshay has joined #openstack-infra | 14:01 | |
| *** salv-orlando has quit IRC | 14:02 | |
| *** ryanpetrello has joined #openstack-infra | 14:03 | |
| *** jistr has joined #openstack-infra | 14:03 | |
| tchaypo | NikitaKonovalov: I’ve done this on pypi recently, what help do you need? | 14:03 |
| *** hashar has joined #openstack-infra | 14:03 | |
| tchaypo | *reads review* | 14:04 |
| *** jistr is now known as jistr|mtg | 14:04 | |
| tchaypo | Oh, I see. | 14:05 |
| tchaypo | you can use the form at https://pypi.python.org/pypi?%3Aaction=submit_form to register the package name | 14:05 |
| *** ryanpetrello_ has joined #openstack-infra | 14:05 | |
| openstackgerrit | Dmitry Teselkin proposed openstack-infra/nodepool: Add per-provider env vars https://review.openstack.org/116993 | 14:05 |
| tchaypo | https://packaging.python.org/en/latest/distributing.html#uploading-your-project-to-pypi talks about this in a bit more detail | 14:06 |
| *** Guest63270 has joined #openstack-infra | 14:06 | |
| *** persia has quit IRC | 14:07 | |
| *** persia has joined #openstack-infra | 14:07 | |
| *** persia has quit IRC | 14:07 | |
| *** persia has joined #openstack-infra | 14:07 | |
| *** ddieterly has quit IRC | 14:08 | |
| *** ryanpetrello has quit IRC | 14:08 | |
| *** ryanpetrello_ is now known as ryanpetrello | 14:08 | |
| *** vrutkovs has left #openstack-infra | 14:08 | |
| *** mpaolino has joined #openstack-infra | 14:09 | |
| *** bswartz has joined #openstack-infra | 14:09 | |
| *** eharney has joined #openstack-infra | 14:12 | |
| *** jamespage_ has joined #openstack-infra | 14:13 | |
| *** Guest63270 has quit IRC | 14:13 | |
| *** cnesa has joined #openstack-infra | 14:13 | |
| *** Hal_ has joined #openstack-infra | 14:13 | |
| *** banix has quit IRC | 14:13 | |
| *** rainya has quit IRC | 14:15 | |
| *** ctracey has quit IRC | 14:15 | |
| *** rainya has joined #openstack-infra | 14:16 | |
| *** sweston has quit IRC | 14:16 | |
| *** rossella_s has quit IRC | 14:16 | |
| *** msuriar has quit IRC | 14:16 | |
| *** matt__ has joined #openstack-infra | 14:16 | |
| *** mattoliverau has quit IRC | 14:16 | |
| *** msuriar has joined #openstack-infra | 14:16 | |
| *** sweston_ has joined #openstack-infra | 14:17 | |
| *** cnesa2 has joined #openstack-infra | 14:17 | |
| fungi | NikitaKonovalov: i'll take care of it if nobody has done so yet | 14:17 |
| *** ctracey has joined #openstack-infra | 14:17 | |
| *** cnesa has quit IRC | 14:17 | |
| *** rossella_s has joined #openstack-infra | 14:17 | |
| openstackgerrit | Merged openstack-infra/project-config: New Gerrit ACL for puppet-openstack-cloud https://review.openstack.org/136098 | 14:18 |
| *** zul has joined #openstack-infra | 14:19 | |
| fungi | NikitaKonovalov: actually, it'll have to wait a few minutes... pypi.python.org is having some sort of trouble and i can't log into it | 14:20 |
| EmilienM | fungi: can I be admin of the new Gerrit group called "puppet-openstack-cloud" please? | 14:21 |
| fungi | akuznetsova: can you point me to the review which failed to update some bug and i can tell you what's missing/needed | 14:21 |
| *** cnesa2 is now known as cnesa | 14:23 | |
| *** otter768 has joined #openstack-infra | 14:23 | |
| *** dkranz has joined #openstack-infra | 14:23 | |
| akuznetsova | fungi, for example this one https://review.openstack.org/#/c/133067/ | 14:23 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make devstack docs job only run on master https://review.openstack.org/136362 | 14:23 |
| *** mriedem has joined #openstack-infra | 14:23 | |
| fungi | EmilienM: the acl for stackforge/puppet-openstack-cloud currently only lists the puppet-manager-core group (which you're in). since https://review.openstack.org/136098 only merged a moment ago it hasn't been applied yet. i'll add you once it's there in a few minutes | 14:24 |
| *** changbl has joined #openstack-infra | 14:24 | |
| *** dkliban_afk is now known as dkliban | 14:25 | |
| *** zz_gondoi is now known as gondoi | 14:25 | |
| fungi | akuznetsova: that bug is for the python-mistralclient project in launchpad, but your patch attempting to close it was to the stackforge/mistral project in gerrit | 14:26 |
| EmilienM | fungi: cool no problem. It's not urgent. Thanks :) | 14:26 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: centos7 devstack jobs should not be run on icehouse https://review.openstack.org/136363 | 14:26 |
| *** rushiagr_away is now known as rushiagr | 14:27 | |
| akuznetsova | fungi, I thought that it is not important and bug is searching by id | 14:27 |
| *** otter768 has quit IRC | 14:27 | |
| *** chandankumar has quit IRC | 14:28 | |
| *** Sincler has quit IRC | 14:28 | |
| akuznetsova | fungi, but thanks, now it makes sense | 14:28 |
| fungi | akuznetsova: the script requires the project name for the patch in gerrit (without the part before the / separator) to match the project name in launchpad. you can override that in gerrit/projects.yaml by giving the gerrit project a groups entry matching the correct launchpad project name but that's a one-way mapping | 14:29 |
| *** uvirtbot has joined #openstack-infra | 14:30 | |
| fungi | akuznetsova: it's a safeguard against committers accidentally mis-typing a bug number, to avoid our automated systems updating bugs for projects outside our sphere of control and annoying random people on the internet | 14:30 |
| *** yamamoto has joined #openstack-infra | 14:30 | |
| *** mwagner_lap has joined #openstack-infra | 14:31 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard: Add details to event subscription https://review.openstack.org/136364 | 14:31 |
| akuznetsova | fungi, thanks for clarification | 14:31 |
| *** HeOS has quit IRC | 14:31 | |
| fungi | anteaya: btw, that 136098 change you approved modifies gerrit/projects.yaml and i seem to remember you sending some announcement that we were not approving those until https://storyboard.openstack.org/#!/story/2000017 was fixed? (no harm done though--i've got my proposed fix hand-patched in and wanted to see if it helped anyway) | 14:32 |
| *** yfried|afk has quit IRC | 14:32 | |
| sdague | so exactly when did -nv jobs get to be voting? | 14:32 |
| *** HeOS has joined #openstack-infra | 14:33 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard-webclient: Add more details on event dashboard https://review.openstack.org/135954 | 14:33 |
| openstackgerrit | Merged openstack-infra/project-config: Add integration tests to the Horizon check gate https://review.openstack.org/136269 | 14:33 |
| fungi | anteaya: also, it seems to have worked! (or else we got lucky and didn't trigger all the broken this time) | 14:33 |
| anteaya | fungi: my understanding was that the problem was only with new repo creation | 14:33 |
| fungi | sdague: give me a specific job name and i should be able to tell you. chances are it's modified by a matching expression in the jobs list after the one at the top setting them all non-voting | 14:34 |
| anteaya | if the problem encompasses all of gerrit/projects.yaml then yes, I did approve in error, but I didn't know that it did | 14:34 |
| sdague | check-tempest-dsvm-ironic-pxe_ssh-nv | 14:34 |
| sdague | fungi: yeh, I think it's a bad regex thing | 14:34 |
| anteaya | thank you for bringing that to my attention and please accept my apologies | 14:34 |
| *** vhoward has joined #openstack-infra | 14:34 | |
| fungi | anteaya: well, the bug is exhibited from new gerrit group creation, but yeah you're right it only breaks nodepool image updates if that coincides with adding a new project | 14:34 |
| sdague | fungi: https://review.openstack.org/#/c/136150/ | 14:34 |
| *** dizquierdo has joined #openstack-infra | 14:35 | |
| anteaya | fungi: and I am glad my mistake did not trigger the error | 14:35 |
| anteaya | ah new gerrit group creation, which https://review.openstack.org/#/c/136098 does | 14:35 |
| anteaya | yes, you are correct and I am wrong | 14:35 |
| anteaya | I will update the email if only for my own benefit | 14:35 |
| fungi | anteaya: i'm hoping that's a sign that my proposed fix works. unfortunately i can't seem to reproduce the bug in a simulated test | 14:35 |
| *** yamamoto has quit IRC | 14:35 | |
| *** jungleboyj has quit IRC | 14:36 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: -nv jobs should remain non voting https://review.openstack.org/136367 | 14:36 |
| mriedem | clarkb: i will say, that was a nice 4th quarter last night for what looked to be a stinky game | 14:36 |
| fungi | anteaya: well, don't. i want to experiment a little more, but this may be fixed by the review i pushed up | 14:36 |
| fungi | in which case you'll be wasting the time it takes to write that e-mail ;) | 14:36 |
| *** mrmartin has joined #openstack-infra | 14:37 | |
| fungi | EmilienM: i've added you to puppet-openstack-cloud-core now | 14:37 |
| EmilienM | fungi: thank you | 14:37 |
| anteaya | fungi: okay I won't send the email | 14:37 |
| fungi | sdague: so you're asking about check-tempest-dsvm-ironic-pxe_ssh-nv et cetera? | 14:38 |
| anteaya | fungi: and your patch isn't merged but it is on the server? | 14:38 |
| fungi | anteaya: right | 14:38 |
| anteaya | ah okay | 14:38 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make devstack docs job only run on master https://review.openstack.org/136362 | 14:38 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: centos7 devstack jobs should not be run on icehouse https://review.openstack.org/136363 | 14:38 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: -nv jobs should remain non voting https://review.openstack.org/136367 | 14:38 |
| fungi | anteaya: i manually edited gerritdb.py on review.openstack.org as a test because i was having trouble recreating the failure condition | 14:38 |
| sdague | fungi: yeh | 14:38 |
| anteaya | fungi: makes sense | 14:38 |
| *** jamespage_ has quit IRC | 14:38 | |
| sdague | fungi: mostly, I was looking at the enormous job growth on devstack which just isn't sane in my opinion, and trying to prune back | 14:39 |
| nibalizer | good morning infra | 14:39 |
| fungi | anteaya: normally i would avoid doing that, but couldn't find a better option | 14:39 |
| anteaya | fungi: well anytime you are ready to test it with new repo creation we have some lined up, they are -2'd | 14:39 |
| fungi | anteaya: oh, awesome--thanks! i'll ping you in a bit | 14:39 |
| anteaya | fungi: yes that makes sense | 14:40 |
| anteaya | nibalizer: morning | 14:40 |
| anteaya | fungi: since there was an idea floated at some point that this same mysql port timeout (if that is what this is) might be affecting paste as well | 14:40 |
| nibalizer | did fungi's change merge and/or fix the problem with projects un-creating? | 14:40 |
| anteaya | fungi: would there be any point in testing this on paste? | 14:41 |
| fungi | sdague: i think it's this line overriding it... http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n559 | 14:41 |
| anteaya | nibalizer: not merged yet, but he changed the code on the server to test it | 14:41 |
| sdague | fungi: yeh | 14:41 |
| anteaya | nibalizer: he can't recreate the conditions so this was the only option left | 14:41 |
| *** ddieterly has joined #openstack-infra | 14:41 | |
| sdague | fungi: so https://review.openstack.org/#/c/136367/2/zuul/layout.yaml,cm fixes it, right? | 14:41 |
| fungi | anteaya: possibly something similar could be inserted into http://git.openstack.org/cgit/openstack-infra/lodgeit/tree/lodgeit/database.py | 14:42 |
| fungi | sdague: i _think_ so but we should be able to tell from the job results | 14:43 |
| sdague | I really wish cgit ui wasn't the most terrible thing in the world, basic lacking features like line highlighting make me sad | 14:43 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Fix nonglobal variables in template names https://review.openstack.org/136370 | 14:43 |
| *** JayJ has joined #openstack-infra | 14:43 | |
| mtreinish | nibalizer: oh, I didn't even realize that was a define. (I thought I used class there) I was confused by you earlier comment but just went with it, but now it makes sense... | 14:43 |
| *** tnovacik has quit IRC | 14:43 | |
| fungi | sdague: i believe the zuul parser as invoked in the layout job will mention non-voting status | 14:44 |
| sdague | fungi: ok | 14:44 |
| *** JayJ has quit IRC | 14:44 | |
| nibalizer | mtreinish: i dont have any experience with mysql proxy | 14:44 |
| nibalizer | but it kindof looks like it call all just go in a mysql_proxy class | 14:44 |
| *** JayJ has joined #openstack-infra | 14:44 | |
| nibalizer | unless there are multiple configuration stanzas or something | 14:45 |
| *** salv-orlando has joined #openstack-infra | 14:45 | |
| mtreinish | nibalizer: no there shouldn't be, my intent was to use a class there, I don't know where I copy and pasted using define there from. | 14:45 |
| anteaya | fungi: what version of sqlalchemy is paste using? | 14:47 |
| fungi | anteaya: i don't know that it is using sqlalchemy at all. did you see a reference to it in the source there? | 14:48 |
| fungi | anteaya: oh, i see it's importing there | 14:48 |
| *** wenlock has joined #openstack-infra | 14:49 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add mysql-proxy to enable read-only access to a db https://review.openstack.org/122311 | 14:49 |
| *** JayJ has quit IRC | 14:49 | |
| fungi | anteaya: yeah, so it's likely the fix wouldn't really be similar... the problem may not even be the same. sqlalchemy is supposed to automatically reconnect if it detects a problem. the bigger issue there may be implicit transaction rollback | 14:49 |
| *** JayJ has joined #openstack-infra | 14:50 | |
| anteaya | I did find this in my searching: http://stackoverflow.com/questions/18054224/python-sqlalchemy-mysql-server-has-gone-away | 14:50 |
| fungi | SQLAlchemy==0.8.1 is installed on paste.o.o | 14:51 |
| anteaya | thanks | 14:51 |
| anteaya | how would we detect what would be triggering a rollback? | 14:51 |
| anteaya | or is that a default behaviour | 14:51 |
| *** AlexF has joined #openstack-infra | 14:52 | |
| fungi | anteaya: mysqldb's ping documents that if it gets a disconnect mid-transaction, the software should take into account that any previously completed parts of the transaction will likely get rolled back | 14:52 |
| fungi | anteaya: so it's possible lodgeit isn't properly retrying when a write fails | 14:53 |
| anteaya | ah | 14:53 |
| *** r-daneel has joined #openstack-infra | 14:54 | |
| fungi | anteaya: i probably need to go digging and find the full traceback from one of the paste failures to have a better idea of where and what is breaking | 14:55 |
| *** yamahata has joined #openstack-infra | 14:55 | |
| openstackgerrit | Dmitry Teselkin proposed openstack-infra/nodepool: Test CI https://review.openstack.org/136375 | 14:55 |
| *** patrickeast has joined #openstack-infra | 14:55 | |
| anteaya | fungi: sure, when you have time it feels like you have a backlog already, in the meantime I am going to read up on the sqlalchemy Pool class: http://docs.sqlalchemy.org/en/rel_0_8/core/pooling.html#sqlalchemy.pool.Pool | 14:56 |
| *** AJaeger has joined #openstack-infra | 14:56 | |
| anteaya | it feels aplicable somehow | 14:56 |
| *** signed8bit has joined #openstack-infra | 14:56 | |
| AJaeger | good morning! | 14:57 |
| *** yaguang has quit IRC | 14:57 | |
| anteaya | morning AJaeger | 14:57 |
| AJaeger | sdague: regarding https://review.openstack.org/136362 - don't we need to block docs publishing of stable/* for all projects? | 14:57 |
| sdague | AJaeger: honestly, I don't know | 14:58 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: hipchat bug fix https://review.openstack.org/136376 | 14:58 |
| sdague | but I know for devstack, we definitely don't want it | 14:58 |
| *** dansmith is now known as superdan | 14:58 | |
| sdague | and I was just trying to fix that bit | 14:58 |
| AJaeger | sdague: ;) | 14:59 |
| AJaeger | I assume that we don't want it for e.g. nova either - the docs on docs.openstack.org/developer/nova are contantly updated | 14:59 |
| AJaeger | and we don't want icehouse there but trunk | 14:59 |
| reed | fungi, I see aptira's blog post multiple times on planet, I swear :) | 15:00 |
| *** tsg_ has quit IRC | 15:00 | |
| *** mattfarina has joined #openstack-infra | 15:00 | |
| AJaeger | fungi, anteaya: I'm not seeing any disabling of the stable docs publishing - shouldn't this be done? | 15:00 |
| AJaeger | sdague: I prefer to understand the big picture first ;) | 15:00 |
| fungi | reed: checking | 15:01 |
| sdague | reed: I assume they are using terrible blog post software that violates rss | 15:02 |
| anteaya | AJaeger: this is a good question | 15:03 |
| anteaya | AJaeger: I don't know either | 15:03 |
| reed | sdague, so you see it multiple times, too? | 15:03 |
| anteaya | AJaeger: what does annegentle say on the topic? | 15:03 |
| sdague | yeh Nov 19th and Nov 16th | 15:03 |
| fungi | reed: ahh, yep, the "Product Management lies dreaming" post | 15:03 |
| reed | i'm not crazy? | 15:03 |
| sdague | so their blog software didn't keep the guid | 15:03 |
| *** koolhead17 has joined #openstack-infra | 15:03 | |
| AJaeger | fungi, reed: Yeah, I noticed that one as well in my RSS feed | 15:03 |
| * reed is not crazy ahahhahha iahihiihihi hahhahah | 15:04 | |
| *** nelsnelson has joined #openstack-infra | 15:04 | |
| fungi | reed: i'm not sure this proves anything at all with regard to your sanity, but i do agree the blog post is duplicated. hopefully only in our cache? | 15:04 |
| *** barnaby has joined #openstack-infra | 15:04 | |
| fungi | i can delete and regen | 15:04 |
| reed | LOL | 15:04 |
| AJaeger | anteaya: for openstack-manuals we publish a few documents to /juno - that is all special cased... | 15:04 |
| anteaya | fungi: sqlalchemy does document a connection timeout workaround: http://docs.sqlalchemy.org/en/rel_0_8/dialects/mysql.html#connection-timeouts | 15:04 |
| *** bcourt has quit IRC | 15:04 | |
| sdague | fungi: yeh, if you delete and regen, the old one isn't in their rss | 15:05 |
| anteaya | AJaeger: then if the default case is publish master branch documentation only, I can get behind a patch that makes that happen | 15:05 |
| sdague | notice the link for the older one goes to something else entirely | 15:05 |
| fungi | anteaya: cool. if this is timeout related that may help. i'm unconvinced it's a timeout though, but would want to figure out who in rax to ask about how their trove instances are tuned for idle sockets | 15:05 |
| dteselkin | Hi. It looks like something is wrong with test for nodepool progect - I made a simple commit to nodepool (touch ci.test) and it failed. https://review.openstack.org/#/c/136375/ | 15:05 |
| *** talluri has joined #openstack-infra | 15:06 | |
| dstufft | mordred: pfft, you're in Philly and you didn't even tell me? ;P | 15:06 |
| anteaya | fungi: I had an offer from johnthetubaguy yesterday to find out and felt that yesterday was premature | 15:06 |
| anteaya | fungi: I'll go hunting for a name | 15:06 |
| sdague | dstufft: it was on the twitters | 15:06 |
| mordred | dstufft: yeah. oh. crap. right. you live in philly | 15:06 |
| *** jamespage_ has joined #openstack-infra | 15:07 | |
| anteaya | johnthetubaguy: so yes, we are now ready to have a chat with any rax folks that tune idle sockets on trove instances | 15:07 |
| mordred | dstufft: we accomplished good drinking, BTW | 15:07 |
| anteaya | johnthetubaguy: how would I go about finding such a person? | 15:07 |
| fungi | dteselkin: i agree, it looks like the allocator tests have bitrotted. perhaps triggered by an update in a dependency | 15:07 |
| mordred | fungi: you know mysql my.cnf params can be changed via trove api ... | 15:08 |
| fungi | mordred: ooh! can they also be introspected? | 15:08 |
| johnthetubaguy | anteaya: good questions, probably a silly question, look for trove commits with rackspace email addresss? | 15:08 |
| mordred | fungi: yes | 15:08 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: adding support for the custom tools plugin https://review.openstack.org/134011 | 15:08 |
| anteaya | johnthetubaguy: I will start there | 15:08 |
| fungi | mordred: i will seek out documentation | 15:08 |
| dstufft | sdague: I didn't notice apparently :( | 15:08 |
| mordred | fungi: show global variables | 15:08 |
| dstufft | mordred: ah well, at least good drinking was had :D | 15:08 |
| fungi | johnthetubaguy: anteaya: mordred says we ought to be able to check and if necessary tune them ourselves | 15:09 |
| johnthetubaguy | anteaya: I can't remember the names of folks right now, which is annoying, I spoke to them | 15:09 |
| AJaeger | anteaya: doing a proper patch would be great - just take care that it does not touch openstack-manuals jobs | 15:09 |
| anteaya | ah the best way | 15:09 |
| clarkb | people still use twitter? | 15:09 |
| * dstufft uses twitter | 15:09 | |
| dteselkin | fungi, ok, just wanted to know that someone else knows about the problem :) | 15:09 |
| AJaeger | clarkb: yeah, when travelling ;) | 15:09 |
| jedimike | oooh, cores are awake! :) https://review.openstack.org/#/c/136281/ should be a quick review to fix zuul for the newer GitPython release, if anyone can take a look :) | 15:09 |
| anteaya | johnthetubaguy: not to worry, I will hunt someone down if we still need to, thanks | 15:09 |
| zhiyan | fungi: hi, could you pls help take a look on this failure, i got no idea :/ http://logs.openstack.org/39/136039/2/check/gate-tempest-dsvm-src-glance_store/39b81bf/logs/devstacklog.txt.gz#_2014-11-21_14_23_45_411 | 15:10 |
| johnthetubaguy | anteaya: cools, let me know if the trail goes cold, and I can try from inside | 15:10 |
| anteaya | AJaeger: sounds like you are willing to let me offer the patch, you must be busy | 15:10 |
| anteaya | AJaeger: I can do that | 15:10 |
| anteaya | johnthetubaguy: thanks I will do | 15:10 |
| AJaeger | anteaya: sitting in Florida at the pool, soon shopping and then flying back home... | 15:10 |
| sdague | dimsum__: what's up with the docker job on nova check? | 15:10 |
| *** talluri has quit IRC | 15:10 | |
| AJaeger | anteaya: I can also do it next week | 15:10 |
| anteaya | AJaeger: dude, get off irc and enjoy the pool | 15:11 |
| anteaya | AJaeger: I'm mixing up a patch | 15:11 |
| clarkb | jedimike I am not convinced that is all we need to change. a little worried that those exceptions can be thrown in other code paths breaking zuul. but if we dont have test coverage for that its hard to know :) at the very least we can start with that change then look for other oddities | 15:11 |
| fungi | zhiyan: you should check out sdague's podcast today. it's all about debugging jobs like that one | 15:11 |
| AJaeger | anteaya: ;) | 15:11 |
| AJaeger | anteaya: thanks! | 15:12 |
| jedimike | clarkb, thanks, I can look through GitPython and see where those exceptions are thrown and go through zuul and see where we call those codepaths. Sound sensible? | 15:12 |
| zhiyan | fungi: ok, i met this begin yday | 15:12 |
| *** primemin2sterp has quit IRC | 15:12 | |
| *** primemin1sterp has quit IRC | 15:12 | |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: hipchat bug fix https://review.openstack.org/136376 | 15:13 |
| dimsum__ | sdague: at the moment a sequencing problem (3 changes in nova broke docker-dsvm) - https://review.openstack.org/#/c/136071/ | 15:13 |
| clarkb | jedimike ya a quick scan of places where that might be thrown may be good | 15:13 |
| *** primeministerp has joined #openstack-infra | 15:13 | |
| *** primemin1sterp has joined #openstack-infra | 15:13 | |
| jedimike | clarkb, great, I'll make a start | 15:13 |
| *** primemin2sterp has joined #openstack-infra | 15:13 | |
| dimsum__ | sdague: waiting for 136071 to merge | 15:13 |
| *** wenlock has quit IRC | 15:13 | |
| *** primeministerp has quit IRC | 15:14 | |
| *** primemin2sterp has quit IRC | 15:14 | |
| *** primemin1sterp has quit IRC | 15:14 | |
| zhiyan | fungi: can you show me a link for the podcase? sdague | 15:15 |
| fungi | zhiyan: i think that's a new job which just got applied to glance-store and perhaps devstack/devstack-gate are not set up to pre-clone it | 15:15 |
| zhiyan | fungi: ok, can you disable it a while before devstack-gate fix it? | 15:16 |
| dimsum__ | sdague: if we dont think nova should have a docker-dsvm, then we should just rip it out and not run it silently (which was what was happening for a while now) | 15:16 |
| fungi | zhiyan: it's in about 4 hours... http://lists.openstack.org/pipermail/openstack-dev/2014-November/050922.html | 15:16 |
| zhiyan | fungi: or how can i do it now? | 15:16 |
| *** timrc-afk is now known as timrc | 15:17 | |
| fungi | sdague: mtreinish: any objections to reverting the -src jobs on glance-store in the short term? do you agree that's the diagnosis? | 15:17 |
| dimsum__ | fungi: when changes from git://git.openstack.org/openstack-infra/zuul get merged, does the zuul job get restarted? | 15:17 |
| fungi | dimsum__: no, it only applies to changes which are not yet in the pipeline | 15:17 |
| fungi | dimsum__: oh, depending on what the change is though | 15:18 |
| *** ildikov has quit IRC | 15:18 | |
| dimsum__ | fungi: y, the one i was poking at was the gerrit owner in status.json | 15:18 |
| fungi | dimsum__: but in general, dependent queues will not be rearranged for changes already running jobs i don't think | 15:18 |
| timrc | It looks like the way etherpad-lite is getting deployed, the repo is being checked out on the 'develop' branch at whatever the latest rev happens to be and once it's present on the node, it'll not get updated. So, how does openstack-infra update these applications? Is it as simple as rm -rf'ing the checkout on the node and letting puppet recheck out the latest rev? | 15:18 |
| *** amitgandhinz has joined #openstack-infra | 15:19 | |
| fungi | dimsum__: ahh, i think it depends on whether zuul has re-queried gerrit for that metadata | 15:19 |
| clarkb | timrc you can change the ref in puppet | 15:19 |
| mtreinish | fungi: sorry I wasn't following, what's up with glance_store? | 15:19 |
| timrc | clarkb, Is that how you do it? | 15:19 |
| reed | say a group wants to use Storyboard but doesn't need gerrit or git... how can they? | 15:19 |
| dimsum__ | fungi: right. will keep an eye on it | 15:19 |
| fungi | timrc: we manually update the ref and test it on an etherpad-dev server first, because epl has a tendency to be broken in random ways so you sometimes have to pick a "sweet spot" in the commit history | 15:20 |
| clarkb | that ^ | 15:20 |
| clarkb | but once you find a spot yup | 15:20 |
| fungi | mtreinish: http://logs.openstack.org/39/136039/2/check/gate-tempest-dsvm-src-glance_store/39b81bf/logs/devstacklog.txt.gz#_2014-11-21_14_23_45_411 | 15:20 |
| timrc | clarkb, Cool | 15:20 |
| timrc | fungi, Thanks | 15:20 |
| mordred | dstufft: also, cheesesteak eggrolls | 15:20 |
| fungi | mtreinish: zhiyan just brought it to our attention | 15:20 |
| clarkb | also I want to switch to trusty nodejs at some point | 15:21 |
| zhiyan | fungi: thanks | 15:21 |
| clarkb | its non suck on trusty iirc | 15:21 |
| fungi | mtreinish: i believe that means it's not in devstack-gate's list of projects to put in the devstack git cache | 15:21 |
| rcarrillocruz | o-k | 15:21 |
| rcarrillocruz | found a bug in nodepool | 15:21 |
| rcarrillocruz | it's now supposed to be filed on storyboard, right? | 15:21 |
| rcarrillocruz | (morning) | 15:21 |
| mtreinish | fungi: yeah that's what it looks like | 15:21 |
| *** tonytan4ever has joined #openstack-infra | 15:22 | |
| clarkb | yes storyboard. | 15:22 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: add support to git for changelog against branch https://review.openstack.org/134014 | 15:22 |
| fungi | rcarrillocruz: https://storyboard.openstack.org/#!/project/668 | 15:22 |
| clarkb | reed I think there is a way but you still add it to projects.yaml | 15:22 |
| rcarrillocruz | k, thx | 15:22 |
| fungi | mtreinish: so i guess we need to back those two jobs out of layout.yaml for glance-store temporarily while we get support landed in devstack-gate, then revert the disablement | 15:23 |
| clarkb | reed what is it for? | 15:23 |
| clarkb | fungi or just update d-g? | 15:23 |
| mordred | we should support that case for sure | 15:24 |
| mtreinish | fungi: is it blocking us from landing the d-g (or devstack) changes we need to cache it? | 15:24 |
| fungi | clarkb: d-g changes have to walk the gauntlet for a while | 15:24 |
| mordred | it'll come up more than once | 15:24 |
| fungi | mtreinish: no, it's just blocking glance-store devs from getting patches reviewed/merged | 15:24 |
| clarkb | mordred I thought jeepyb already supported it | 15:24 |
| reed | clarkb, the Win The Enterprise group, trying to evaluate how painful it would be for me to lower their expectations on Storyboard or teach them how to use a wiki or ... just let them go buy something else somewhere else or ... | 15:24 |
| clarkb | a storyboard only project type. maybe I am imagining a wanted feature as an implemented feature | 15:25 |
| clarkb | reed right but what is use case? | 15:25 |
| mtreinish | fungi: the issue with dropping the src jobs is I think it's the only functional testing on the project, if we drop it they;re opened up to landing broken code | 15:25 |
| fungi | clarkb: reed: "bugs with no associated code?" | 15:25 |
| reed | at the moment I'm ambivalent between teach them to use the wiki and give them a storyboard.. the use case is filing and tracking tasks | 15:25 |
| clarkb | we can throw people at bug trackers and wikis or buy tools but without an intended use case hard to know whats best | 15:26 |
| reed | no bugs, no code... more like a 'trello' with no process | 15:26 |
| fungi | mtreinish: great point | 15:26 |
| clarkb | reed gotcha, we should put them in storyboard then | 15:26 |
| timrc | clarkb, btw, hipchat is way less hip without you on it :~( | 15:26 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: adding support for the custom tools plugin https://review.openstack.org/134011 | 15:27 |
| fungi | timrc: now he's on shoot-from-the-hip-chat | 15:27 |
| *** jamespage_ has quit IRC | 15:27 | |
| clarkb | hipchat now with less hip and more gifs! | 15:27 |
| timrc | fungi, Nice :P | 15:27 |
| fungi | clarkb: that sounds like 4chan | 15:27 |
| *** kgiusti has joined #openstack-infra | 15:27 | |
| AJaeger | bye for now! | 15:27 |
| *** AJaeger has quit IRC | 15:28 | |
| *** ayoung is now known as ayoung-dentist | 15:28 | |
| *** ayoung-dentist has quit IRC | 15:28 | |
| reed | clarkb, I'd be start a conversation on a review... should I send a change to projects.yaml and discuss there? any clue what the syntax should be? | 15:28 |
| *** jungleboyj has joined #openstack-infra | 15:29 | |
| clarkb | reed no and it may not support it yet. but if you push a wip change we can add that in | 15:29 |
| fungi | mtreinish: oddly, openstack/glance_store is already in the projects list in devstack-gate | 15:29 |
| *** rmcall has joined #openstack-infra | 15:29 | |
| reed | clarkb, ok, let me ask krotscheck too. | 15:30 |
| mtreinish | fungi: hmm, could it be that sdague's patch to construct the git path is wrong and it's trying to clone something by mistake? | 15:30 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/nodepool: Set region_name kwarg to None if provider does not have a region https://review.openstack.org/136392 | 15:33 |
| fungi | mtreinish: looking at the setup workspace log for that job, it did indeed sync in openstack/glance_store and update it | 15:33 |
| *** jamespage_ has joined #openstack-infra | 15:34 | |
| *** sdake has quit IRC | 15:34 | |
| *** zz_jgrimm is now known as jgrimm | 15:34 | |
| *** tsg has joined #openstack-infra | 15:34 | |
| *** sdake has joined #openstack-infra | 15:34 | |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: add support for pyton and tox matrix project axis https://review.openstack.org/134025 | 15:35 |
| *** liam_ has joined #openstack-infra | 15:35 | |
| *** liam_ has quit IRC | 15:35 | |
| fungi | mtreinish: yeah, i'm suspecting something is odd here deeper inside devstack | 15:36 |
| *** andreykurilin_ has joined #openstack-infra | 15:36 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Documentation publication should be based on master https://review.openstack.org/136395 | 15:36 |
| *** otherwiseguy has joined #openstack-infra | 15:36 | |
| sdague | fungi: which thing? | 15:37 |
| *** dmellado has quit IRC | 15:37 | |
| fungi | sdague: http://logs.openstack.org/39/136039/2/check/gate-tempest-dsvm-src-glance_store/39b81bf/logs/devstacklog.txt.gz#_2014-11-21_14_23_45_411 | 15:37 |
| fungi | sdague: devstack-gate definitely sync'd it into /opt/stack/new but devstack is trying to clone it anyway | 15:38 |
| sdague | fungi: yep, the fix is here - https://review.openstack.org/#/c/135711/ | 15:38 |
| *** dmellado|afk is now known as dmellado | 15:38 | |
| fungi | aha, grenade tanked it because of trouble on juno? | 15:38 |
| sdague | it borked for some reason | 15:39 |
| sdague | javelin seems to have died, don't know why | 15:39 |
| fungi | or just $other_random_failure perhaps | 15:39 |
| anteaya | why is the trove api documentation on github? | 15:40 |
| anteaya | SlickNik: ^^ | 15:40 |
| *** dangers_away is now known as dangers | 15:40 | |
| mtreinish | sdague: http://logs.openstack.org/11/135711/3/gate/gate-grenade-dsvm/7e251d2/logs/javelin.txt.gz#_2014-11-20_23_36_00_415 | 15:40 |
| mtreinish | sdague: looks like server create died | 15:41 |
| sdague | mtreinish: because we couldn't allocate memory - http://logs.openstack.org/11/135711/3/gate/gate-grenade-dsvm/7e251d2/logs/old/screen-n-cpu.txt.gz#_2014-11-20_23_35_59_282 | 15:42 |
| fungi | reed: planet should be cleaned of the extra aptira post now | 15:42 |
| anteaya | configuration-parameter-show Shows details of a configuration parameter. | 15:43 |
| reed | thanks fungi | 15:43 |
| *** mpaolino has quit IRC | 15:43 | |
| *** IvanBerezovskiy has left #openstack-infra | 15:43 | |
| *** fandi has joined #openstack-infra | 15:44 | |
| sdague | fungi: anyway, the glance_store issue has a fix there with tests so it doesn't revert. But given how many jobs are defined on devstack... someone probably just needs to recheck babysit that all the way through. | 15:44 |
| *** dimsum__ has quit IRC | 15:45 | |
| mtreinish | sdague: hmm, that error looks familiar | 15:45 |
| sdague | that fix was created 2 days ago | 15:45 |
| mtreinish | but I didn't see it in e-r | 15:45 |
| *** dimsum__ has joined #openstack-infra | 15:45 | |
| mordred | anteaya: link? | 15:45 |
| anteaya | https://github.com/openstack/python-troveclient | 15:46 |
| *** wenlock has joined #openstack-infra | 15:46 | |
| mordred | anteaya: (like, where is the trove-api docs?) | 15:46 |
| anteaya | mordred: from http://docs.openstack.org/developer/trove/#developer-resources | 15:46 |
| sdague | fungi / mordred / et al | 15:46 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: hipchat bug fix https://review.openstack.org/133997 | 15:46 |
| fungi | mordred: http://docs.openstack.org/developer/trove/ also says "Trove API Documentation on github" | 15:46 |
| fungi | er, what anteaya said | 15:46 |
| mordred | ah | 15:47 |
| sdague | zuul does the extra tests on changes, how difficult would it be to add the inverse to drop tests on changes | 15:47 |
| *** stevemar has joined #openstack-infra | 15:47 | |
| sdague | because, it occurred to me this morning, that running any dsvm* jobs on project requirements changes is actually of 0 value | 15:47 |
| sdague | because devstack ignores the contents of that file | 15:47 |
| fungi | sdague: what do you mean by extra tests on changes? | 15:47 |
| fungi | sdague: oh, you mean if file path is changed, run these additional tests | 15:48 |
| sdague | fungi: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L639-L644 | 15:48 |
| sdague | yeh | 15:48 |
| mtreinish | sdague: so on a change which just touches requirements only run the reqs job and nothing else | 15:48 |
| sdague | mtreinish: no, still run the unit tests, because those matter | 15:48 |
| fungi | sdague: so the trick would probably be adding a feature to match when *only* files in a specific path change | 15:48 |
| mordred | wow. the trove README has a whole other project's readme in it | 15:48 |
| fungi | sdague: or a way to do inverse matches i guess (only run this job if files outside this list of paths change) | 15:49 |
| sdague | because every time we land a requirements change now we kick off about 400 tempest jobs, that actually have 0 value | 15:49 |
| *** zz_dimtruck is now known as dimtruck | 15:49 | |
| sdague | so we mostly just block actual work from getting done, which is sadness | 15:50 |
| fungi | sdague: and we definitely want to run things like style checks and unit tests, but i agree devstack-gate-based jobs don't use those files | 15:50 |
| sdague | fungi: yes, agreed | 15:50 |
| *** ddieterly has quit IRC | 15:50 | |
| sdague | this is mostly about reducing the storm of really long and provably non useful tests | 15:50 |
| *** ddieterly has joined #openstack-infra | 15:51 | |
| *** armax has joined #openstack-infra | 15:51 | |
| *** markmcclain has joined #openstack-infra | 15:52 | |
| sdague | fungi: also like... https://review.openstack.org/#/c/136086/ seems silly that it runs tempest tests | 15:54 |
| mordred | fungi, anteaya, SlickNik https://review.openstack.org/136400 | 15:54 |
| *** pcrews has joined #openstack-infra | 15:54 | |
| jeblair | i think you could do that today with a negative lookahead on the file regex, but that would probably only work for one file. doing it right by adding a new negative match term would not be hard. | 15:55 |
| mordred | sdague: hang on ... don't we do the requirements syncing in devstack anymore? | 15:55 |
| *** nfedotov has left #openstack-infra | 15:55 | |
| mordred | OH | 15:55 |
| *** emagana has joined #openstack-infra | 15:55 | |
| mordred | nevermind | 15:55 |
| *** ildikov has joined #openstack-infra | 15:55 | |
| jeblair | mordred: i'm assuming sdague is referring to changes to individual project requirements (eg tempest), rather than to global reqs? | 15:55 |
| mordred | still coffeeing | 15:55 |
| mordred | yeah | 15:56 |
| mordred | I now fully undersatnd | 15:56 |
| jeblair | (aiui, devstack tests of changes to global reqs is still useful) | 15:56 |
| fungi | jeblair: mordred: which we generally propose en masse and generate a storm of jobs when that happens | 15:56 |
| *** achanda has joined #openstack-infra | 15:56 | |
| mordred | yah | 15:56 |
| sdague | jeblair: yes | 15:56 |
| fungi | (e.g. when a change lands on global reqs) | 15:56 |
| * mordred agrees about not running dsvm then | 15:56 | |
| sdague | like I said, this is about not doing a wasteful thing that we *know* isn't testing anything | 15:57 |
| *** mpaolino has joined #openstack-infra | 15:57 | |
| sdague | especially as it consumes close to 50% of our capacity when it happens | 15:57 |
| *** markmcclain1 has joined #openstack-infra | 15:57 | |
| clarkb | I almost ready to start building new nodepool host | 15:58 |
| *** skolekonov has quit IRC | 15:59 | |
| *** sabeen2 has joined #openstack-infra | 15:59 | |
| *** markmcclain has quit IRC | 15:59 | |
| *** sabeen1 has quit IRC | 16:00 | |
| sdague | jeblair: you got a pointer as to where I might hack on that? | 16:00 |
| *** che-arne has quit IRC | 16:01 | |
| jeblair | sdague: yeah, i think all of the occurences of the word 'files' in model.py and scheduler.py are the relevant bits to copy/past/invert | 16:01 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/nodepool: Set region_name kwarg to None if provider does not have a region https://review.openstack.org/136392 | 16:01 |
| *** barnaby has quit IRC | 16:01 | |
| *** lttrl has joined #openstack-infra | 16:02 | |
| *** achanda has quit IRC | 16:02 | |
| mestery | anteaya: The ODL CI has two succesful sandbox runs now under tykeal's gerrit account: https://review.openstack.org/#/c/136142/ and https://review.openstack.org/#/c/127460/ | 16:03 |
| *** andreykurilin_ has quit IRC | 16:03 | |
| *** andreykurilin__ has joined #openstack-infra | 16:03 | |
| mestery | anteaya: What else do we need to do to get it enabled again? | 16:03 |
| tykeal | anteaya: the IP for both of those was 23.253.119.7 | 16:03 |
| tykeal | anteaya: display name in gerrit is 'Andrew Grimberg' | 16:04 |
| *** bhunter71 has joined #openstack-infra | 16:04 | |
| jeblair | i have to run to an appointment; back in a few hours | 16:05 |
| *** sergsh has quit IRC | 16:05 | |
| *** liam_ has joined #openstack-infra | 16:05 | |
| *** talluri has joined #openstack-infra | 16:06 | |
| *** jamespage_ has quit IRC | 16:07 | |
| *** xyang0 has joined #openstack-infra | 16:08 | |
| openstackgerrit | Dmitry Teselkin proposed openstack-infra/nodepool: Add per-provider env vars https://review.openstack.org/116993 | 16:08 |
| *** rediskin has left #openstack-infra | 16:09 | |
| *** RicoLin has joined #openstack-infra | 16:09 | |
| anteaya | tykeal: okay please respond to this thread: http://lists.openstack.org/pipermail/third-party-announce/2014-November/000092.html | 16:10 |
| tykeal | anteaya: can do | 16:10 |
| anteaya | tykeal: include this url https://jenkins.opendaylight.org/ovsdb/job/openstack-gerrit/ | 16:10 |
| tykeal | also can do :) | 16:10 |
| anteaya | tykeal: which links to the sandbox runs | 16:10 |
| *** Sincler has joined #openstack-infra | 16:10 | |
| anteaya | ping emagana and find one other operator to reply to your post stating they feel you should be re-enabled | 16:11 |
| anteaya | once I see those posts, we can re-enable you | 16:11 |
| *** talluri has quit IRC | 16:11 | |
| anteaya | also stay close to emagana for times when he needs someone else to verify someone's system | 16:11 |
| anteaya | that helps us all | 16:11 |
| *** Viswanath has joined #openstack-infra | 16:11 | |
| *** wenlock has quit IRC | 16:12 | |
| emagana | tykeal: I am around if you need to chat | 16:13 |
| tykeal | emagana: k, I'm just about to finish sending the note to the list per anteaya's request | 16:13 |
| emagana | tykeal: ;-)_ | 16:13 |
| tykeal | and away it goes | 16:14 |
| *** Viswanath has quit IRC | 16:14 | |
| tykeal | though it's awaiting moderation, wee! | 16:14 |
| *** RicoLin has quit IRC | 16:15 | |
| *** barnaby has joined #openstack-infra | 16:16 | |
| tykeal | emagana: for quick reference the ODL CI was disabled earlier this week. We've resolved the errors causing bad builds and got a couple of successful runs triggered off of the sandbox available for viewing here: https://jenkins.opendaylight.org/ovsdb/job/openstack-gerrit/ | 16:16 |
| *** MarkAtwood has joined #openstack-infra | 16:17 | |
| *** lttrl has quit IRC | 16:19 | |
| *** jamespage_ has joined #openstack-infra | 16:19 | |
| *** dannywilson has joined #openstack-infra | 16:19 | |
| *** salv-orlando has quit IRC | 16:20 | |
| *** dannywilson has quit IRC | 16:20 | |
| *** dannywilson has joined #openstack-infra | 16:20 | |
| *** chandankumar has joined #openstack-infra | 16:20 | |
| clarkb | fungi: ok I think I am ready to start booting the trusty nodepool node. | 16:21 |
| clarkb | fungi: anything else you want to see done first? | 16:21 |
| fungi | clarkb: nope, i'm on deck if you run into trouble | 16:22 |
| clarkb | my current plan is spin it up. It should then start building images then get min ready nodes | 16:22 |
| clarkb | at that point I change dns and shutdown old server | 16:22 |
| clarkb | and finally clean up alien things | 16:22 |
| clarkb | that should just about cover it | 16:22 |
| fungi | the clean up alien things should wait a couple hours after cut-over | 16:22 |
| clarkb | ++ | 16:23 |
| fungi | to avoid killing running jobs by deleting nodes out from under them | 16:23 |
| fungi | however, we're likely going to see serious over-quota errors in the meantime | 16:24 |
| *** otter768 has joined #openstack-infra | 16:24 | |
| clarkb | fungi: its friday though so it shouldn't be too bad on the over quota error front | 16:24 |
| fungi | yeah, hoping that the node utilization is starting to trend downward around the time we do the cut | 16:25 |
| *** jlibosva has quit IRC | 16:25 | |
| *** dimsum__ has quit IRC | 16:25 | |
| *** yolanda has quit IRC | 16:25 | |
| *** dimsum__ has joined #openstack-infra | 16:26 | |
| *** jamespage_ has quit IRC | 16:26 | |
| *** HeOS has quit IRC | 16:26 | |
| *** yolanda has joined #openstack-infra | 16:26 | |
| *** ala_ has quit IRC | 16:26 | |
| clarkb | ok starting image build now | 16:27 |
| *** otter768 has quit IRC | 16:28 | |
| *** dkranz has quit IRC | 16:29 | |
| *** dimsum__ has quit IRC | 16:30 | |
| *** Hal_ has quit IRC | 16:30 | |
| clarkb | puppet is running | 16:31 |
| *** yolanda has quit IRC | 16:33 | |
| ryanpetrello | clarkb: any idea if the tox infra is using is > 1.8 ? | 16:33 |
| ryanpetrello | the new parametrized/generative stuff is pretty cool and I'd like to use it for pecan :) | 16:33 |
| *** yolanda has joined #openstack-infra | 16:33 | |
| *** henditan has joined #openstack-infra | 16:34 | |
| krotscheck | Anyone out there able to look at storyboard logs for me? I’m looking for a failed login attempt from mbitard, it might be throwing some kind of an encoding error | 16:35 |
| clarkb | hrm I think pyzmq failed to build. we will see if puppet treats that as a failure and I have to rebuild or if I can get on the node and see post puppet | 16:35 |
| clarkb | ryanpetrello: it should be latest | 16:35 |
| ryanpetrello | okay, awesome | 16:35 |
| clarkb | ryanpetrello: we pinned for a while but upstream fixed the bugs that hit us so now we install latest | 16:36 |
| clarkb | krotscheck: if no one beats me to it I can as soon as this node build is done | 16:36 |
| krotscheck | clarkb: Thanks. I’ve got an appointment at 9, so if you can’t do it by then then I’ve got after 10. | 16:36 |
| krotscheck | I mean it can wait until after 10 | 16:36 |
| clarkb | kk | 16:37 |
| krotscheck | Seriously, sometimes I feel that my fingers are dyslexic. | 16:37 |
| *** HeOS has joined #openstack-infra | 16:37 | |
| *** emagana has quit IRC | 16:37 | |
| *** emagana has joined #openstack-infra | 16:37 | |
| *** luqas has quit IRC | 16:38 | |
| clarkb | x86_64-linux-gnu-gcc: error trying to exec 'cc1plus': execvp: No such file or directory we may just be missing the compiler? or puppet order of ops is biting us? | 16:39 |
| *** henditan has quit IRC | 16:40 | |
| *** jgallard_ has quit IRC | 16:40 | |
| *** emagana has quit IRC | 16:42 | |
| *** tnovacik has joined #openstack-infra | 16:43 | |
| *** jaypipes is now known as leakypipes | 16:43 | |
| mtreinish | fungi, clarkb: do you think we'll have time today to give setting up the mysql proxy a try? | 16:45 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Install build-essential for nodepool installation https://review.openstack.org/136418 | 16:47 |
| *** groknix has quit IRC | 16:49 | |
| *** groknix has joined #openstack-infra | 16:50 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make devstack docs job only run on master https://review.openstack.org/136362 | 16:50 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: centos7 devstack jobs should not be run on icehouse https://review.openstack.org/136363 | 16:50 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: -nv jobs should remain non voting https://review.openstack.org/136367 | 16:50 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: move nova-tox-functional to experimental until there is content https://review.openstack.org/136420 | 16:50 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Install build deps for nodepool installation https://review.openstack.org/136418 | 16:50 |
| *** matrohon has quit IRC | 16:50 | |
| clarkb | fungi: ok I am going to attempt starting nodepool on new host now | 16:51 |
| fungi | mtreinish: i can try later on today if things get quiet | 16:51 |
| clarkb | hrm isn't happy | 16:51 |
| mtreinish | fungi: ok sure, no worries. It's not really a high prio | 16:51 |
| clarkb | the number of packages we need to install this stuff is really much higher than I expected... | 16:52 |
| clarkb | more patches coming | 16:52 |
| fungi | clarkb: yeah, i think we determined that older/other base images included all of build-essential but some newer (trusty? pvhvm?) images do not, and we only really ensure it present on slaves not general purpose servers | 16:52 |
| *** jpich has quit IRC | 16:53 | |
| *** jp_at_hp has quit IRC | 16:53 | |
| clarkb | and I need libffi-dev and libssl-dev | 16:54 |
| *** HeOS has quit IRC | 16:54 | |
| clarkb | and maybe more | 16:54 |
| fungi | oh, yep | 16:54 |
| *** david-ly_ is now known as david-lyle | 16:56 | |
| mtreinish | clarkb: heh, maybe just use: http://git.openstack.org/cgit/openstack-dev/devstack/tree/files/apts/general | 16:57 |
| mtreinish | granted most of that probably isn't needed :) | 16:57 |
| *** markmcclain1 has quit IRC | 16:58 | |
| *** jamespage_ has joined #openstack-infra | 16:58 | |
| *** markmcclain has joined #openstack-infra | 16:58 | |
| fungi | mtreinish: yeah in this case i think some of it is related to the nodepool->swiftclient->keystoneclient->lxml dependency chain, and then whatever the thing is wanting a c++ compiler | 16:59 |
| clarkb | yes novaclient and six and pycryto and we can't have nice things | 17:00 |
| fungi | er, nodepool->novaclient->keystoneclient->lxml i mean | 17:00 |
| clarkb | AttributeError: 'NoneType' object has no attribute 'db' something is hosed | 17:00 |
| clarkb | I wonder if that means we haven't puppeted the db stuff for nodepool | 17:00 |
| fungi | i was conflating it with zuul->swifclient->keystoneclient->lxml which is similarly annoying | 17:00 |
| clarkb | oh unless is the db connection info different? | 17:00 |
| fungi | clarkb: why would the db connection info be different? it's local, right? | 17:01 |
| clarkb | ya its local and it seems to be configured | 17:02 |
| clarkb | so I have no idea what is going on there yet | 17:02 |
| *** patrickeast has quit IRC | 17:03 | |
| clarkb | I just cofirmed I can connect with the nodepool db uri | 17:03 |
| clarkb | so thats not the issue | 17:03 |
| *** markmcclain has quit IRC | 17:04 | |
| clarkb | (it uses localhost) | 17:04 |
| *** banix has joined #openstack-infra | 17:04 | |
| *** emagana has joined #openstack-infra | 17:05 | |
| *** jamespage_ has quit IRC | 17:07 | |
| *** arxcruz has quit IRC | 17:07 | |
| clarkb | nodepool schema has 4 tables in it which matches the old host | 17:08 |
| clarkb | table slook sane so I think the initial db bootstrapping was fine | 17:09 |
| *** dtantsur is now known as dtantsur|afk | 17:10 | |
| *** shashankhegde has joined #openstack-infra | 17:10 | |
| *** amitgandhinz has quit IRC | 17:11 | |
| *** dizquierdo has quit IRC | 17:11 | |
| *** dizquierdo has joined #openstack-infra | 17:11 | |
| jogo | anteaya: morning | 17:12 |
| anteaya | jogo: hello there | 17:12 |
| *** amitgandhinz has joined #openstack-infra | 17:12 | |
| tykeal | anteaya: got an ok (sort of?) from Doug on that thread... just waiting on emagana I suppose? | 17:12 |
| anteaya | tykeal: yes, let's wait for edgar | 17:12 |
| jogo | anteaya: have you been hit by the freezing cold weather? | 17:12 |
| anteaya | tykeal: that test is supposed to fail btw, that is the point, two patches, one expected to pass, one expected to fail | 17:13 |
| anteaya | tykeal: edgar's invention | 17:13 |
| jogo | the same system that dumped several feet in buffalo | 17:13 |
| tykeal | anteaya: ah, ok, I just wanted to be sure! | 17:13 |
| anteaya | jogo: yes, cold and snowy here, but doesn't compare | 17:13 |
| anteaya | jogo: yes, we just have the top edge of it, only about 2 feet here | 17:13 |
| anteaya | normal weather | 17:13 |
| anteaya | tykeal: of course, thanks for the confirmation run on the failing patch :D | 17:14 |
| tykeal | :D | 17:14 |
| jogo | anteaya: haha 2 feet is normal wow | 17:14 |
| anteaya | tykeal: the only thing that concerns me is the mention of setting it up and going on vacation | 17:14 |
| jogo | anteaya: anyway wanted to poke you about https://review.openstack.org/#/c/136115/2 | 17:14 |
| anteaya | tykeal: happy vacation, but is someone able to admin whilst you are gone? | 17:14 |
| tykeal | anteaya: I'm on pager duty, even if I'm on vacation | 17:14 |
| jogo | anteaya: I was surprised to see a change to nova's tests land without a nova-core +1 | 17:14 |
| * anteaya loves working whislt into a sentence | 17:14 | |
| anteaya | tykeal: boo | 17:14 |
| tykeal | anteaya: and yes, there are other admins that can deal with it, I'm just point for ODL wee! | 17:15 |
| superdan | jogo: anteaya: yeah, same | 17:15 |
| tykeal | anteaya: the only time I get real vacation time is if I a) go out of country or b) go camping ;) | 17:15 |
| anteaya | tykeal: do that | 17:16 |
| *** chandankumar has quit IRC | 17:16 | |
| *** mriedem is now known as mriedem_lunch | 17:16 | |
| anteaya | superdan jogo okay well just speaking for myself here, since I haven't discussed this with other cores | 17:16 |
| tykeal | anteaya: and it turned out the reason why the CI kept failing was a job configuration which the project committers ( mestery being one of ) can modify on their own | 17:16 |
| *** amitgandhinz has quit IRC | 17:16 | |
| *** kgiusti has quit IRC | 17:16 | |
| *** mriedem_lunch is now known as mriedem_vittles | 17:16 | |
| jogo | anteaya: and was wondering if there is some sort of informal policy around adding tests and requiring +1s from the core teams | 17:16 |
| clarkb | fungi: looking at how the db is configured I think some of the other reconfigure* calls are failing which prevents us from setting the db config | 17:16 |
| anteaya | but for a silent or experiemental non-voting moving to a check non-voting I haven't been alerting repo cores/ptls | 17:17 |
| clarkb | fungi: but there are no other stacktraces that I see... | 17:17 |
| anteaya | but I can for nova if that is your wish | 17:17 |
| anteaya | but I do for moving a non-voting to voting job | 17:17 |
| *** AlexF has quit IRC | 17:17 | |
| superdan | anteaya: I think that'd be good | 17:17 |
| anteaya | sure I have no problem with that | 17:17 |
| anteaya | would you like that patch reverted? | 17:17 |
| jogo | anteaya: thanks, ahh that makes sense | 17:17 |
| jogo | anteaya: after discussion in openstack-nova it sounds like we are fine with the patch | 17:17 |
| anteaya | okay | 17:18 |
| jogo | anteaya: but were just a bit surprised | 17:18 |
| superdan | it's not my preference, personally, but not worth a revert at this point | 17:18 |
| anteaya | so going forward, does nova want to weigh in on non-voting experiemental/silent jobs moving to non-voting check? | 17:18 |
| anteaya | superdan says yes | 17:18 |
| anteaya | what do you want jogo? | 17:18 |
| superdan | I think any non-silent change is probably good to get input on | 17:19 |
| anteaya | superdan: I can do that | 17:19 |
| anteaya | and post to infra to express your wishes | 17:19 |
| jogo | anteaya: yes that would be great, thanks! | 17:19 |
| anteaya | okay | 17:19 |
| * anteaya composes a post | 17:19 | |
| fungi | clarkb: which reconfigure calls? | 17:20 |
| clarkb | fungi: in nodepool.nodepool:updateConfig() | 17:20 |
| clarkb | except | 17:21 |
| clarkb | that we have a wrapper try except that says "Exception in startup" and that is not hit | 17:21 |
| clarkb | so I am really confused | 17:21 |
| clarkb | 104.130.154.112 is the new node btw | 17:21 |
| openstackgerrit | Bradley Klein proposed openstack-infra/project-config: Add puppet-monasca acls and review group. https://review.openstack.org/136432 | 17:21 |
| fungi | clarkb: oh, is it possible we're depending on manual schema changes we made in the db to support some recent feature addition and neglected to adjust the table creation calls accordingly? | 17:21 |
| fungi | hopping in | 17:22 |
| jesusaurus | clarkb: are you trying to deploy nodepool on trusty? | 17:22 |
| * jesusaurus is trying to follow along | 17:23 | |
| fungi | jesusaurus: yep | 17:23 |
| *** salv-orlando has joined #openstack-infra | 17:23 | |
| clarkb | fungi: no I don't think that is what is breaking us. sqlalchemy appears to have populated the tables properly | 17:24 |
| *** bnemec is now known as beekneemech | 17:24 | |
| clarkb | jesusaurus: ya, I have a new patch I need to push up to cover the install stuff | 17:24 |
| clarkb | maybe I push that up and delete this node then try again. I suppose its possible that my efforts to cleanly reinstall have failed | 17:24 |
| clarkb | fungi: ^ what do you think? I guess I should let you poke around a little first | 17:24 |
| anteaya | jogo superdan https://etherpad.openstack.org/p/wjSZahen7z is my email draft | 17:26 |
| anteaya | clarkb fungi ^^ | 17:26 |
| *** dprince has quit IRC | 17:26 | |
| *** dprince has joined #openstack-infra | 17:26 | |
| *** dimsum__ has joined #openstack-infra | 17:26 | |
| fungi | clarkb: hrm, yeah a fresh start is probably a good idea. if nodepool dependencies didn't all actually get properly installed you could see a myriad of weird issues | 17:27 |
| clarkb | fungi: ya that is what I am thinking let me push this patchset up, get that merged then give it another go | 17:27 |
| fungi | clarkb: i don't entirely trust a new node launch until i get puppet to run clean on it in the first pass | 17:27 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Install build deps for nodepool installation https://review.openstack.org/136418 | 17:29 |
| clarkb | fungi: ^ that should give us a clean node launch | 17:29 |
| anteaya | devananda: can we get a rebase on https://review.openstack.org/#/c/126627/ | 17:30 |
| jogo | anteaya: looks good to me! | 17:30 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: adding support for the custom tools plugin https://review.openstack.org/134011 | 17:30 |
| anteaya | jogo superdan and then can we get a core to weigh in on https://review.openstack.org/#/c/126627/ | 17:31 |
| anteaya | jogo: thank you | 17:31 |
| clarkb | fungi: but I will put it in my dev env and try it from there | 17:31 |
| * clarkb deletes the current node | 17:31 | |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: adding support for the custom tools plugin https://review.openstack.org/134011 | 17:31 |
| fungi | clarkb: yeah, lgtm but not sure if there's any effort to keep puppet-nodepool working on non-ubuntu | 17:32 |
| jogo | anteaya: done | 17:32 |
| fungi | also that might conflict with puppet-zuul if someone tries to coinstall them (which some third-party ci operators do) | 17:32 |
| *** MaxV has quit IRC | 17:32 | |
| anteaya | jogo: thank you | 17:32 |
| *** ihrachyshka has joined #openstack-infra | 17:32 | |
| *** mudassirlatif has joined #openstack-infra | 17:33 | |
| * asselin is paying attention...we install nodepool and zuul in one | 17:33 | |
| clarkb | damnit puppet | 17:33 |
| clarkb | that is all I have to say about that :) | 17:33 |
| anteaya | fairly clear | 17:33 |
| asselin | but we're on ubuntu :) | 17:33 |
| anteaya | clarkb fungi any objection to me sending out: https://etherpad.openstack.org/p/wjSZahen7z | 17:34 |
| anteaya | jeblair: if you are back from your appointment ^^ | 17:34 |
| clarkb | fungi: second build is running | 17:35 |
| fungi | anteaya: sorry, too many simultaneous discussions in different channels. i've generally tried to get the ptl to weigh in on new voting (or even non-voting, experimental) job additions and changes of any of them to voting, but in absence of a ptl at least a couple regular core reviewers from the project(s) in question | 17:35 |
| clarkb | anteaya: my only concern is that there will be times where we change nova's jobs without asking nova. sorry | 17:35 |
| clarkb | anteaya: like when we removed py26 | 17:35 |
| clarkb | anteaya: or when we added requirements checking | 17:35 |
| anteaya | jogo superdan ^^ | 17:35 |
| anteaya | I can't box infra into a corner here, but I am trying to play nice | 17:36 |
| fungi | clarkb: i think when infra needs to make changes to a project's jobs, we do that and give them a heads up | 17:36 |
| *** tonytan4ever has quit IRC | 17:36 | |
| clarkb | fungi: sure, but this is all very much phrased in the way of "we must ask for permission" | 17:36 |
| fungi | i see this as more of a "when some interested party proposes a job change and is not a core reviewer for the project that job is voting on" | 17:36 |
| *** e0ne has quit IRC | 17:36 | |
| superdan | clarkb: you understand the intent here, right? | 17:37 |
| anteaya | fungi: if that is the case, I need to chage the language of the email | 17:37 |
| fungi | exempting infra/qa/release where we'd just let them know and sort out our differences after the fact | 17:37 |
| clarkb | superdan: yes I think this specific case the correct thing to do is ask nova core | 17:37 |
| fungi | anteaya: so anyway, what i've tried to do in the past is use common sense. unfortunately common sense is hard to codify in policy | 17:38 |
| superdan | clarkb: We didn't really ask for an email or a policy change, we'd just like to be consulted on things like this particular thing | 17:38 |
| clarkb | sure I think thats fair | 17:38 |
| superdan | cool | 17:38 |
| clarkb | but I don't want this to become a policy of *-core must +1 any job changes that affect their project outside of the silent queue | 17:38 |
| anteaya | fungi: yes | 17:38 |
| *** dkehn has quit IRC | 17:39 | |
| anteaya | I don't have to send an email | 17:39 |
| clarkb | (which is what jogo's comment implies he wants) | 17:39 |
| anteaya | however SergeyLukjanov isn't around much and andreas is offline right now | 17:39 |
| anteaya | so while I can say yes for me I don't know what they will do | 17:39 |
| anteaya | and unfortunately SergeyLukjanov hasn't been reading commit messages lately when he approves code | 17:40 |
| *** sarob has joined #openstack-infra | 17:40 | |
| fungi | anteaya: i think a reminder to the group is fair, that if it looks like there's a non-urgent job change (where urgency is generally related to release process and gate breakage), then get some reasonably authoritative input from the projects affected and try to make sure it's reflected somehow in the review prior to approving | 17:40 |
| superdan | the py26 change was well understood and socialized, right? | 17:41 |
| superdan | we all knew it was coming | 17:41 |
| superdan | this particular change landed in eight hours, and it wasn't urgent, AFAICT | 17:41 |
| fungi | superdan: yeah, we announced it in advance in multiple venues and projects had plenty of time to object | 17:41 |
| fungi | the py26 change i mean | 17:41 |
| superdan | right, needing a +1 on the actual final patch isn't at all the point | 17:42 |
| clarkb | superdan: ok I may have misread what jogo wanted | 17:42 |
| *** dkehnm has quit IRC | 17:42 | |
| superdan | clarkb: well, I'm not speaking for him, but I don't think that's really what he meant | 17:42 |
| clarkb | "I am just a little surprised this landed without a +1 from nova core" | 17:43 |
| *** akuznetsova has left #openstack-infra | 17:43 | |
| fungi | that's one of those where i would have sought out mikal to +1 or at least two or three nova core reviewers | 17:43 |
| superdan | clarkb: well, because that was an eight hour window and a single revision | 17:43 |
| *** links has quit IRC | 17:43 | |
| superdan | clarkb: I don't think he meant that any patch that lands needs a +1 from us, I think he meant "WAT?" | 17:43 |
| *** shashankhegde has quit IRC | 17:44 | |
| clarkb | AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult' | 17:44 |
| anteaya | yay for fast turn around on project-config | 17:44 |
| *** AlexF has joined #openstack-infra | 17:44 | |
| clarkb | via novaclient | 17:44 |
| jogo | superdan: yes I meant 'WAT' | 17:46 |
| *** dkehn has joined #openstack-infra | 17:46 | |
| anteaya | jogo: you try slowing down andreas and his reviews | 17:46 |
| clarkb | sure enough no SplitResult | 17:46 |
| anteaya | jogo: I rarely see a review he hasn't hit first | 17:46 |
| fungi | clarkb: seems like that may be the wrong version of six? | 17:46 |
| jogo | anteaya: heh | 17:47 |
| fungi | clarkb: maybe it's pinned in requirements.txt or in the reqs of a dep? | 17:47 |
| anteaya | but yeah, I hear the "WAT" | 17:47 |
| anteaya | just not sure how to rope in those sleeping in the way we mean | 17:47 |
| clarkb | fungi: no Ithink oslo.utils is broken | 17:48 |
| clarkb | maybe not still digging | 17:48 |
| *** Longgeek has quit IRC | 17:49 | |
| clarkb | oslo.utils requires six>=1.7.0 | 17:50 |
| clarkb | we have 1.5.2 | 17:50 |
| clarkb | now to find out why | 17:50 |
| fungi | yeah, that's exactly what i was suspecting | 17:50 |
| * clarkb should make a rule that you must support six 1.0 | 17:51 | |
| tykeal | anteaya: and emagana FYI I need to hit the road, I saw that mestery responded about the concern over the log showing a fail but the job giving a pass, hopefully y'all can work that out ;). I will be checking back in on the mail thread later today so if our CI account gets reactivated I can get it running again. Many thanks for the help! | 17:51 |
| pleia2 | good morning | 17:51 |
| pleia2 | digging out of piles of email today, but I'm around :) | 17:51 |
| anteaya | tykeal: enjoy vacation | 17:51 |
| anteaya | pleia2: morning | 17:51 |
| tykeal | just a single day, mostly so I can drive up to my parents place for an early thanksgiving tomorrow ;) | 17:51 |
| *** harlowja_away is now known as harlowja | 17:52 | |
| clarkb | doing a local install to a venv to see if that makes grep work better | 17:52 |
| mestery | tykeal: Enjoy Thanksgiving! | 17:52 |
| * tykeal waves | 17:52 | |
| anteaya | tykeal: nice | 17:53 |
| *** AlexF has quit IRC | 17:53 | |
| anteaya | tykeal: I hope you have a nice drive | 17:53 |
| anteaya | I am about to go for a walk, back later | 17:53 |
| pleia2 | enjoy | 17:53 |
| anteaya | thanks | 17:53 |
| fungi | clarkb: i suspect you're going to find something (possibly transitively) listed after six in requirements.txt is causing 1.7.0 to get downgraded to 1.5.2 | 17:53 |
| mordred | fungi, clarkb, jeblair: https://review.openstack.org/#/c/90150/ I made some comments on it, but you guys may want to as well | 17:54 |
| clarkb | fungi: ya | 17:54 |
| clarkb | dhellmann: btw this is why I say don't pin six | 17:54 |
| mordred | I believe that it's a step towards what we want out of nova+neutron+dns - but there are some specific real-world implications what's in the spec mgight have on us | 17:54 |
| *** chandankumar has joined #openstack-infra | 17:55 | |
| dhellmann | clarkb: I support not pinning six. | 17:55 |
| clarkb | hrm in a virtualenv I get 1.8 I bet trusty has six 1.5.2 | 17:55 |
| clarkb | fungi: I am just going to try a manual upgrade of six | 17:55 |
| * dhellmann also supports not using distro packages | 17:55 | |
| mordred | dhellmann: ++ | 17:55 |
| fungi | clarkb: oh! is the base image coming with python-six already installed maybe? | 17:56 |
| clarkb | dhellmann: mordred the issue is that pip without --upgrade says meh | 17:56 |
| clarkb | even if the version isn't correct | 17:56 |
| clarkb | because you know that would be too helpful | 17:56 |
| clarkb | nodepool is now running with six 1.8.0 | 17:56 |
| clarkb | but | 17:56 |
| clarkb | still broken | 17:56 |
| *** AlexF has joined #openstack-infra | 17:57 | |
| fungi | clarkb: same traceback? | 17:57 |
| clarkb | fungi: ya | 17:57 |
| *** derekh has quit IRC | 17:57 | |
| fungi | clarkb: did you manage to uninstall the distro package cleanly? | 17:57 |
| clarkb | ideally pip would at least fail | 17:57 |
| clarkb | and say use the --upgrade flag | 17:57 |
| clarkb | fungi: no, I just installed six via pip with --upgrade | 17:57 |
| fungi | yeah, python is quite likely finding the old copy | 17:57 |
| clarkb | really? hrm I can try nuking old six | 17:58 |
| *** dkehn has quit IRC | 17:58 | |
| fungi | since debs and pip put them in different places, it will depend on library search path order | 17:58 |
| *** nadya_ has quit IRC | 17:58 | |
| clarkb | lol this will remove cloud init | 17:58 |
| fungi | ahh, yep. i now remember, cloud-init depends on python-six | 17:59 |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Add iptables to log dropped packets https://review.openstack.org/136439 | 17:59 |
| clarkb | along with requests and urllib3 | 17:59 |
| nibalizer | adam_g: what about this https://review.openstack.org/136439 ? | 17:59 |
| nibalizer | instead of the other thing | 17:59 |
| *** ayoung has joined #openstack-infra | 17:59 | |
| fungi | which we remove all of in our worker builds, right? | 17:59 |
| nibalizer | sin the other thing affects every server in infra | 17:59 |
| clarkb | fungi: maybe | 17:59 |
| *** dkehn has joined #openstack-infra | 17:59 | |
| clarkb | I am going to give it a go and see what happens | 17:59 |
| adam_g | nibalizer, that chain gets managed centrally via puppet | 18:00 |
| *** hashar has quit IRC | 18:00 | |
| mordred | clarkb: we don't need cloudinit | 18:00 |
| mordred | clarkb: once the node is booted, if puppet causes cloud-init to be removed, that's fine | 18:00 |
| adam_g | nibalizer, also that will append it to the bottom of the chain, after the -j REJECT | 18:00 |
| clarkb | nope no luck | 18:00 |
| mordred | the _only_ thing we use it for for real is getting ssh keys onto machines | 18:00 |
| clarkb | I think something is properly broken in nodepool | 18:01 |
| nibalizer | adam_g: well we can figure out how to do the insert thing to iptables | 18:01 |
| clarkb | maybe sqlalchemy related? | 18:01 |
| nibalizer | but the way nodepool works is puppet runs once on the nodepool node then doesn't touch the machine again, the tests run, then the machine is turned off forever | 18:01 |
| nibalizer | i think you probably actually want to log much more packets than 2/minute, and you can do that if you do it in the test right there | 18:02 |
| clarkb | I have stopped the nodepool service on the new host | 18:02 |
| clarkb | need a break and caffeine | 18:02 |
| *** emagana has quit IRC | 18:02 | |
| adam_g | nibalizer, i dont understand, whats the issue with doing it in puppet? thats the way that chain has been updated in the past | 18:02 |
| clarkb | (but /var/log/nodepool/debug.log on 104.130.138.17 is full on infos) | 18:03 |
| adam_g | and its persistent across all devstack slaves | 18:03 |
| *** emagana has joined #openstack-infra | 18:03 | |
| nibalizer | adam_g: if yo uchange it there then it goes to all infra server nodes as well | 18:04 |
| nibalizer | so the gerrit and paste servers will be logging ip traffic to syslog | 18:04 |
| adam_g | nibalizer, ah | 18:04 |
| *** mpaolino has quit IRC | 18:04 | |
| nibalizer | so i think we can hoist your change to a different place and i'll be much happier | 18:04 |
| *** ChuckC has quit IRC | 18:04 | |
| clarkb | dstufft: should I file a bug for pip not upgrading packages or not erroring when a package is already installed but at an older version than what was requseted? | 18:05 |
| *** lttrl has joined #openstack-infra | 18:05 | |
| anteaya | actually as I think about it, I would put the onus on the owner of the patch changing the job to source agreement from the repo in question | 18:05 |
| adam_g | nibalizer, yeah | 18:05 |
| *** jedimike has quit IRC | 18:06 | |
| *** patrickeast has joined #openstack-infra | 18:06 | |
| anteaya | perhaps even going so far as marking the patch wip until that communication has been aquired in a comment | 18:06 |
| *** cpowell has quit IRC | 18:06 | |
| * anteaya disappears again | 18:06 | |
| *** sdake has quit IRC | 18:07 | |
| *** wenlock has joined #openstack-infra | 18:07 | |
| *** rcarrillocruz has quit IRC | 18:07 | |
| *** otherwiseguy has quit IRC | 18:07 | |
| *** emagana has quit IRC | 18:07 | |
| *** rcarrillocruz has joined #openstack-infra | 18:07 | |
| *** emagana has joined #openstack-infra | 18:08 | |
| nibalizer | adam_g: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/single_use_slave.pp#n38 here i think | 18:08 |
| *** sdake has joined #openstack-infra | 18:08 | |
| adam_g | nibalizer, yah, pushing a new rev now | 18:08 |
| nibalizer | ok woot | 18:09 |
| clarkb | I also tried downgrading sqlalchemy to the version that was on the precise server. no luck there either | 18:09 |
| openstackgerrit | Bradley Klein proposed openstack-infra/project-config: Add puppet-monasca acls and review group. https://review.openstack.org/136432 | 18:09 |
| *** ihrachyshka has quit IRC | 18:09 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/system-config: Add iptables rule to log dropped packets https://review.openstack.org/135694 | 18:10 |
| *** shashankhegde has joined #openstack-infra | 18:10 | |
| *** bhuvan has joined #openstack-infra | 18:11 | |
| clarkb | also I have confirmed that the same version of nodepool is installed on both servers | 18:12 |
| clarkb | oh this is curious | 18:12 |
| clarkb | libmysqlclient-dev is not installed | 18:12 |
| *** tonytan4ever has joined #openstack-infra | 18:13 | |
| clarkb | how did MySQL-Python get installed? | 18:13 |
| fungi | hrm, we do need that for the mysqld module | 18:13 |
| clarkb | does MySQL-Python no longer link against libmysqlclient-dev? | 18:13 |
| fungi | i doubt it did fully get installed. it likely errored out in the pip install and didn't exit nonzero so pip assumed it was successful when it's only really partly there | 18:13 |
| clarkb | fungi: except that sqlalchemy is able to put the tables into the db | 18:14 |
| clarkb | so its able to do some stuff? | 18:14 |
| fungi | oh, hrm | 18:14 |
| mtreinish | clarkb: is there an ubuntu package for it that doesn't depend on libmysqlclient-dev? | 18:14 |
| clarkb | mtreinish: oh maybe let me see | 18:14 |
| fungi | there's a possibility... is the deb for that lib installed? | 18:14 |
| openstackgerrit | Merged openstack/requirements: Require dnspython>=1.12.0 https://review.openstack.org/126356 | 18:14 |
| clarkb | ah yup | 18:14 |
| fungi | that s'plains it | 18:14 |
| clarkb | but its called python-mysqldb | 18:14 |
| clarkb | the same version is on both nodes too (1.2.3) | 18:15 |
| fungi | yeah, debian python packaging guidelines are generally that the module/package provided for import should be prefixed with python- to make the deb name, rather than whatever upstream named their project | 18:15 |
| clarkb | shouldn't it be python-MySQL-python? | 18:15 |
| fungi | so since you import MySQLdb from it, debian/ubuntu call it python-mysqldb | 18:15 |
| clarkb | because I really really want to see the distros do that :) | 18:15 |
| fungi | python-<whatever the import name is> | 18:16 |
| fungi | you don't import MySQL-python | 18:16 |
| fungi | oh, and they translate _ to - in the module name too when making the package name | 18:16 |
| clarkb | I am stumped then | 18:16 |
| clarkb | should take a braek | 18:17 |
| fungi | do we need a newer mysqldb than what we're trying to use there? | 18:17 |
| fungi | though i got the impression that module moved pretty glacially upstream | 18:17 |
| clarkb | ya its slower than glaciers | 18:18 |
| clarkb | and that is the same version as on the old box | 18:18 |
| *** achanda has joined #openstack-infra | 18:18 | |
| *** achanda has quit IRC | 18:19 | |
| *** achanda has joined #openstack-infra | 18:19 | |
| *** MarkAtwood has quit IRC | 18:22 | |
| *** emagana has quit IRC | 18:22 | |
| *** emagana has joined #openstack-infra | 18:23 | |
| *** cpowell has joined #openstack-infra | 18:23 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:24 | |
| *** otter768 has joined #openstack-infra | 18:25 | |
| *** amcrn has joined #openstack-infra | 18:25 | |
| *** gyee_ has joined #openstack-infra | 18:25 | |
| clarkb | back in a bit. /me finds food | 18:26 |
| *** jogo is now known as flashgordon | 18:26 | |
| *** achanda_ has joined #openstack-infra | 18:26 | |
| mrmartin | er | 18:26 |
| *** emagana has quit IRC | 18:27 | |
| *** dkranz has joined #openstack-infra | 18:27 | |
| mrmartin | hi. I need some review for this: https://review.openstack.org/#/c/135708/ and fungi if you had time to create a self-signed cert for groups-dev portal, the cert could be go to proper hiera variables | 18:27 |
| fungi | mrmartin: sorry, i meant to, and it's still on my to do list so hopefully today | 18:28 |
| *** AlexF has quit IRC | 18:28 | |
| mrmartin | ok, thnx. | 18:28 |
| *** nadya_ has joined #openstack-infra | 18:28 | |
| *** otter768 has quit IRC | 18:29 | |
| *** emagana has joined #openstack-infra | 18:30 | |
| *** achanda has quit IRC | 18:30 | |
| *** yamamoto has joined #openstack-infra | 18:33 | |
| *** ihrachyshka has joined #openstack-infra | 18:34 | |
| *** achanda_ is now known as achanda | 18:34 | |
| *** pelix has quit IRC | 18:35 | |
| *** reed has quit IRC | 18:36 | |
| *** Ryan_Lane has joined #openstack-infra | 18:36 | |
| *** zul has quit IRC | 18:37 | |
| *** nadya_ has quit IRC | 18:37 | |
| *** yamamoto has quit IRC | 18:38 | |
| *** dizquierdo has quit IRC | 18:39 | |
| *** andreykurilin__ has quit IRC | 18:42 | |
| *** andreykurilin_ has joined #openstack-infra | 18:42 | |
| *** emagana has quit IRC | 18:42 | |
| *** lttrl has quit IRC | 18:42 | |
| *** rmcall has quit IRC | 18:43 | |
| *** emagana has joined #openstack-infra | 18:43 | |
| zaro | fungi: did you still need me to help you with that launchpad thing? | 18:43 |
| *** wuhg has quit IRC | 18:43 | |
| fungi | zaro: yes please, if you're not too busy | 18:44 |
| *** belmoreira has joined #openstack-infra | 18:44 | |
| zaro | fungi: what did you need again? | 18:44 |
| *** ChuckC has joined #openstack-infra | 18:44 | |
| fungi | zaro: on https://launchpad.net/gearman-plugin change the maintainer to "OpenStack Continuous Integration Administrators (openstack-ci-admins)" | 18:46 |
| fungi | zaro: that way i'll be able to complete the lp->sb migration and lock down lp bug reporting | 18:47 |
| *** emagana has quit IRC | 18:47 | |
| *** mattfarina has quit IRC | 18:49 | |
| *** armax has quit IRC | 18:50 | |
| boris-42 | fungi: hi could you please take a look at infra patch that adds rally job to zaqar (already has PTL +1 and one +2) https://review.openstack.org/#/c/135132/ thanks | 18:50 |
| *** alexpilotti has quit IRC | 18:53 | |
| *** _buzz has joined #openstack-infra | 18:53 | |
| *** alexpilotti has joined #openstack-infra | 18:53 | |
| zaro | fungi: done for gearman-plugin, what was the other project? | 18:54 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 18:54 | |
| *** _buzz_ has quit IRC | 18:56 | |
| *** emagana has joined #openstack-infra | 18:56 | |
| fungi | zaro: that was it, thanks! | 18:59 |
| zaro | fungi: cool | 19:01 |
| adam_g | crinkle, hogepodge looks like that PPA i was telling you about in paris got seeded for kilo: https://launchpad.net/~openstack-ubuntu-testing/+archive/ubuntu/kilo | 19:01 |
| *** mpaolino has joined #openstack-infra | 19:02 | |
| *** emagana has quit IRC | 19:02 | |
| *** emagana has joined #openstack-infra | 19:02 | |
| crinkle | adam_g: awesome! | 19:02 |
| *** yamahata has quit IRC | 19:03 | |
| *** alexpilotti has quit IRC | 19:03 | |
| *** teran_ has joined #openstack-infra | 19:03 | |
| *** jistr|mtg has quit IRC | 19:04 | |
| *** SumitNaiksatam has quit IRC | 19:04 | |
| *** weshay has quit IRC | 19:04 | |
| *** annegent_ has joined #openstack-infra | 19:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Add non voting rally performance job to zaqar project https://review.openstack.org/135132 | 19:05 |
| *** teran has quit IRC | 19:06 | |
| *** tsg has quit IRC | 19:06 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu https://review.openstack.org/136158 | 19:08 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT COMMIT disable config drive https://review.openstack.org/135831 | 19:08 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/134639 | 19:08 |
| openstackgerrit | Kyle Rockman proposed openstack-infra/jenkins-job-builder: adding support for the custom tools plugin https://review.openstack.org/134011 | 19:08 |
| *** matrohon has joined #openstack-infra | 19:10 | |
| flashgordon | zaro: I have a new gerrit UI question | 19:12 |
| flashgordon | zaro: how do I tell what revision I am looking at? I keep looking at the wrong one | 19:12 |
| *** amcrn has quit IRC | 19:13 | |
| *** amitgandhinz has joined #openstack-infra | 19:15 | |
| dhellmann | pleia2, anteaya : I'm interested in working on turning https://wiki.openstack.org/wiki/Oslo/CreatingANewLibrary into a section of the infra manual during the upcoming sprint. Is that something you're still interested in having or are you going in a different direction with that guide? | 19:17 |
| *** otter768 has joined #openstack-infra | 19:19 | |
| *** emagana has quit IRC | 19:19 | |
| *** armax has joined #openstack-infra | 19:19 | |
| *** emagana has joined #openstack-infra | 19:19 | |
| *** andreykurilin_ has quit IRC | 19:19 | |
| pleia2 | dhellmann: perhaps not straight conversion (might be too oslo-specific), but as I understand it that is the kind of material we do want to pull in | 19:20 |
| dstufft | clarkb: what's the issue? | 19:20 |
| *** andreykurilin_ has joined #openstack-infra | 19:20 | |
| dhellmann | pleia2: right, removing the oslo-ness is the "work on" part :-) | 19:20 |
| dhellmann | pleia2: but good, I'll block out some time those days | 19:20 |
| *** radez is now known as radez_g0n3 | 19:21 | |
| pleia2 | dhellmann: great, thank you :) | 19:21 |
| *** tonytan4ever has quit IRC | 19:21 | |
| *** teran_ has quit IRC | 19:22 | |
| clarkb | dstufft: there is a system install of six==1.5.2 for cloud init. We install software via `pip install $pathtorepo` that repo has a transitive dep on six>=1.7.0 | 19:22 |
| clarkb | dstufft: the end result is pip returns 0 and leaves six at 1.5.2 | 19:23 |
| dhellmann | pleia2: I may want to pick your brain about how best to deal with the oslo-specific parts -- I'd hate to have the reader have to go back and forth between a wiki page and the manual, for example. Is there room for the oslo workflow in the manual as a separate chapter? That would make it easier to share content. | 19:23 |
| clarkb | dstufft: imo it should update to >=1.7.0 OR return an error | 19:23 |
| dstufft | clarkb: does anything have a transitive dependency on "six" | 19:23 |
| clarkb | dstufft: yes. in this case nodepool depends on novaclient which depends on oslo.utils which depends on six>=1.7.0 | 19:23 |
| *** ildikov has quit IRC | 19:23 | |
| dstufft | clarkb: no I mean just "six" | 19:23 |
| dstufft | no qualifier | 19:23 |
| clarkb | dstufft: no | 19:23 |
| clarkb | dstufft: at least not that I found | 19:23 |
| dstufft | or a qualifier that includes 1.5.2 | 19:23 |
| clarkb | dstufft: yes nodepool itself says >=1.4.1 iirc | 19:24 |
| dstufft | ok | 19:24 |
| dstufft | known bug then :/ | 19:24 |
| dstufft | there's a ticket | 19:24 |
| dstufft | sec | 19:24 |
| *** emagana has quit IRC | 19:24 | |
| dstufft | clarkb: https://github.com/pypa/pip/issues/988 | 19:26 |
| *** emagana has joined #openstack-infra | 19:26 | |
| *** mriedem_vittles is now known as mriedem | 19:27 | |
| *** ihrachyshka has quit IRC | 19:27 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:27 | |
| fungi | dstufft: i think the concern here is the (arguably surprising) behavior in pip where it won't upgrade an already installed dependency, even if a later one is requested, unless you explicitly provide the -U option. i get that it's likely not solvable (or not worth solving) outside of whatever the solution for 988 ends up being | 19:27 |
| *** ivar-lazzaro has quit IRC | 19:27 | |
| *** tonytan4ever has joined #openstack-infra | 19:28 | |
| *** ivar-lazzaro has joined #openstack-infra | 19:28 | |
| *** e0ne has joined #openstack-infra | 19:28 | |
| *** zul has joined #openstack-infra | 19:29 | |
| pleia2 | dhellmann: I haven't had a very close look at it, but just going over it quickly it seems like sections 5-7 can be pulled in directly (with some edits to 7 for a world with Storyboard), 8 can be generalized for releases and I wonder if 10 could happen earlier in the docs to prevent some skipping around | 19:29 |
| dstufft | fungi: right, the behavior is super wrong | 19:29 |
| *** dimsum__ has quit IRC | 19:29 | |
| dstufft | I'm not arguing that :) | 19:29 |
| *** amitgandhinz has quit IRC | 19:29 | |
| dstufft | just saying you don't need to file another ticket b/c we have one for that issue already | 19:29 |
| pleia2 | dhellmann: well, maybe not "directly" but with few edits :) | 19:30 |
| clarkb | does pdb not step across threads? | 19:30 |
| clarkb | also yes I am running nodepool in pdb now :/ | 19:30 |
| *** teran has joined #openstack-infra | 19:30 | |
| *** MarkAtwood has joined #openstack-infra | 19:32 | |
| dstufft | clarkb: fungi the low level problem here is pip is stupid and it just uses the "highest" tier requirement in the face of duplicate requirements | 19:32 |
| dstufft | e.g. nodepool depends on six>=1.4.1 which depends on something which depends on six>1.7.wahtever, so the nodepool dependency on six is "higher" in the dep tree | 19:33 |
| clarkb | so if we update nodepool it should be happier? | 19:33 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu https://review.openstack.org/136158 | 19:33 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT COMMIT disable config drive https://review.openstack.org/135831 | 19:33 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/134639 | 19:33 |
| *** zul has quit IRC | 19:34 | |
| dhellmann | pleia2: yeah, it doesn't have to say "oslo" if the process is generally applicable to any library. Some of the steps won't apply to applications, though, I think (e.g., 9). 8 only applies to libraries, but Oslo isn't the only project managing libraries now so I think you're right I can generalize that. I'm not sure if 10 can happen earlier or not -- chicken and egg. But if it's so late, and the middle step of the process is "go follow the | 19:34 |
| dhellmann | instructions in this part of the infra guide" then maybe that doesn't matter and then we can keep this wiki page with the oslo-specific parts at the beginning and end. I'll spend more time on it, maybe it's not as bad as I feared as far as flipping back and forth. | 19:34 |
| *** salv-orlando has quit IRC | 19:34 | |
| dstufft | clarkb: if you switch nodepool's six dependency to be six>=1.7.0 it'll do what you're hoping for right now | 19:35 |
| dstufft | without the use of -U | 19:35 |
| clarkb | ok | 19:35 |
| fungi | dstufft: even if pip finds that six is already installed? that's the situation here | 19:35 |
| openstackgerrit | Merged openstack-infra/reviewday: Escape double quotes for the subject field. https://review.openstack.org/134739 | 19:36 |
| *** yjiang5_away is now known as yjiang5 | 19:36 | |
| *** sdake has quit IRC | 19:36 | |
| *** sdake has joined #openstack-infra | 19:37 | |
| dstufft | fungi: yes, the problem is that pip find's nodepool's "six>=1.4.1" and goes "ok six 1.5.1 satisifies that" and it ignores the transitive dep on six>=1.7.0 | 19:37 |
| fungi | dstufft: ubuntu cloud server image with a deb of cloud-init installed, which depends on a deb of python-six so that's also installed. then we pip install something which wants newer six without uninstalling the python-six deb first (which arguably also has other negative side effects) | 19:37 |
| *** sdake has quit IRC | 19:37 | |
| *** sdake has joined #openstack-infra | 19:37 | |
| *** sdake has quit IRC | 19:37 | |
| *** sdake has joined #openstack-infra | 19:37 | |
| dstufft | if nodepools six dep is changes to six>=1.7.0 it'll look and go "ok 1.5.1 doens't satisify that" uninstall it and install a version of six that matches six>=1.7.0 | 19:37 |
| fungi | dstufft: neat, even without -U? | 19:38 |
| *** markmcclain has joined #openstack-infra | 19:38 | |
| fungi | not that it will be able to completely successfully uninstall the deb of six, though, so not sure what the actual outcome ends up being there | 19:38 |
| dstufft | fungi: yes even without the -U | 19:39 |
| dstufft | the only reason the -U works is because that tells it to unconditionally upgrade "six>=1.4.1" | 19:39 |
| dstufft | which happens to be also be >=1.7.0 | 19:39 |
| *** annegent_ has quit IRC | 19:39 | |
| dstufft | if nodepool's six dep was six<=1.6 instead, then -U wouldn't fix it | 19:39 |
| fungi | i suppose also reordering the requirements.txt so that whatever's depending on newer six is before the nodepool dep entry for six itself, then it will have a similar effect | 19:40 |
| fungi | since the first entry it will encounter for six will ask for newer than it finds currently installed | 19:41 |
| *** doug-fish has quit IRC | 19:41 | |
| *** doug-fish has joined #openstack-infra | 19:41 | |
| dstufft | I forget exactly how pip picks which "duplicate" specifier it's going to pay attention to and which it's going to ignore, but it's something like "first one we've seen" or "closest to the root of the tree" or something like that | 19:42 |
| dstufft | basically it's terrible | 19:42 |
| fungi | heh. thanks for the insight as usual! | 19:42 |
| dstufft | if my PEP 440 branch lands I might be able to fix it for the easy case without going full blown depsolver | 19:42 |
| dstufft | maybe | 19:43 |
| dstufft | it gets hard because when you start combining multiple specifiers from multiple projects, as you get more specifiers you might have to backtrack and redo it | 19:44 |
| *** matrohon has quit IRC | 19:44 | |
| dstufft | and then you're basically writing a SAT solver | 19:44 |
| fungi | i'd imagine the tricky part is working out dependencies specified dynamically based on execution of setup.py | 19:45 |
| dstufft | we tell setup.py to build us a static representation of it | 19:45 |
| clarkb | ok I think this is hanging because the thread that sets up the DB config is getting stuck waiting for gearman connection to go through which should be blocked by firewalls | 19:45 |
| *** AlexF has joined #openstack-infra | 19:46 | |
| fungi | clarkb: oh! spooky thread interactions. that seems like not a great bootstrapping story | 19:46 |
| clarkb | fungi: ^ I am going to propose an update to nodepool to use newer six. Then I am going to rebuild and if that works except for the current issue I will try removing gearman from nodepools config until we are ready to switch | 19:46 |
| *** AlexF has quit IRC | 19:46 | |
| clarkb | fungi: ya its not great... | 19:46 |
| *** markvan has quit IRC | 19:47 | |
| arosen | fungi: any idea where this is coming from? http://paste.openstack.org/show/136065/ | 19:47 |
| *** asettle has quit IRC | 19:47 | |
| *** markvan_ has joined #openstack-infra | 19:47 | |
| arosen | I'm wondering if it the failure is because: Host pypi.PA.openstack.org not found: 3(NXDOMAIN) | 19:47 |
| *** markmcclain1 has joined #openstack-infra | 19:47 | |
| *** markmcclain1 has quit IRC | 19:47 | |
| arosen | I set: scripts/configure_mirror.sh:NODEPOOL_PYPI_MIRROR=${NODEPOOL_PYPI_MIRROR:-http://pypi.dfw.openstack.org/simple/} | 19:47 |
| fungi | dstufft: i'm unable to picture how that works... A requires B, B runs setup.py to express that it requires C so you only find that out after downloading B, C requires D, and D requires a different version of B than you've already downloaded... lather, rinse, funny farm | 19:48 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Match novaclients transitive six dep https://review.openstack.org/136474 | 19:48 |
| clarkb | fungi: ^ review on that would be awesome | 19:48 |
| dstufft | fungi: yea that's when you start needing a SAT solver | 19:48 |
| dstufft | fungi: particularly one that can handle iteratively adding things to it for best performance | 19:49 |
| *** sdake_ has joined #openstack-infra | 19:49 | |
| *** sdake_ has quit IRC | 19:49 | |
| *** sdake_ has joined #openstack-infra | 19:49 | |
| openstackgerrit | Jaime Flynn proposed openstack-infra/jenkins-job-builder: adds safe-output to email-ext in publishers.py https://review.openstack.org/136475 | 19:49 |
| arosen | fungi: doh | 19:49 |
| arosen | sorry i see. | 19:49 |
| fungi | dstufft: well, the whole "setup.py might calculate different dependencies on different platforms" situation implies you could wind up downloading all of pypi to figure it out, unless this is only going to be implemented for wheels | 19:49 |
| clarkb | fungi: self.gearman_client.waitForServer() | 19:50 |
| dstufft | fungi: we'll you'd calculate it at runtime whne you do pip install | 19:50 |
| clarkb | so ya nodepool just sits there waiting for the connection to happen if it is configured to gearman. if it is not configured to gearman it won't do that. so I will remove its gearman config until images are built | 19:50 |
| clarkb | then add it back once we flip flop the servers | 19:50 |
| *** matrohon has joined #openstack-infra | 19:50 | |
| *** mpaolino has quit IRC | 19:50 | |
| dstufft | fungi: so setup.py should be deterministic within a single pip install run (unless you're doing something crazy in your setup.py, in which case glhf) | 19:50 |
| fungi | dstufft: yeah, which gets fun when it's a runtime-determined dep of something you haven't downloaded yet | 19:51 |
| *** e0ne has quit IRC | 19:51 | |
| *** kgiusti has joined #openstack-infra | 19:51 | |
| dstufft | fungi: we basically won't support runtime-determined deps that change based on anything other than properties of the actual system it's being installed into | 19:52 |
| dstufft | (OS, Python version, etc) | 19:52 |
| dstufft | if you do something like that you get undefined behavior | 19:52 |
| clarkb | I need to run mail out to the mail box before it gets picked up. Back shortly | 19:52 |
| fungi | dstufft: yeah, requiring it to be declared in some other metadata instead so that the determining mechanism can be abstracted out of setup.py altogether seems like the sanest route | 19:52 |
| *** mudassirlatif has quit IRC | 19:53 | |
| dstufft | (more or less it'll be implemented like, "the first time we download B-1.0.tar.gz in a particular pip install run, we'll run setup.py egg_info which will print requires.txt inside the egg-info directory we'll parse that and that's your dependencies for B 1.0 in this particular install run, if your setup.py dynamically adjusts install_requires in a way that subsequent setup.py egg_info's would give idfferent results on the same machine | 19:53 |
| dstufft | within minutes of each other, glhf) | 19:53 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu https://review.openstack.org/136158 | 19:54 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT COMMIT disable config drive https://review.openstack.org/135831 | 19:54 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/134639 | 19:54 |
| fungi | dstufft: right, i guess the point is just that because dependencies _can_ be declared run-time, you can't calculate them in advance before downloading things, and therefore can't prevent possibly downloading lots of packages you end up not using because later requirements ended up run-time expressing a need for different versions of them | 19:56 |
| dstufft | fungi: yup | 19:56 |
| *** nibalizer is now known as nibalizer_change | 19:56 | |
| dstufft | we'll just end up downloading a bunch of stuff | 19:56 |
| dstufft | when things aren't wheels anyways | 19:56 |
| dstufft | or sdist 2.0 whenever that happens | 19:56 |
| *** Sincler has quit IRC | 19:57 | |
| dstufft | both of those will have static metadata whch will be exposed directly in a future PyPI api | 19:57 |
| fungi | clarkb: i'm uncertain whether 136474 is necessarily the ideal solution, but we can iterate on better fixes the bleeding stops. i think for example this will cause people running nodepool with earlier novaclient releases to upgrade to a newer six that they actually need (because nodepool works with novaclient versions which didn't need such a new six) | 19:57 |
| *** nibalizer_change has left #openstack-infra | 19:58 | |
| *** nibalizer_change has quit IRC | 19:58 | |
| clarkb | fungi: ya but six does forward compat so it should be ok | 19:59 |
| *** ildikov has joined #openstack-infra | 19:59 | |
| clarkb | fungi: if not ideal | 19:59 |
| *** tsg has joined #openstack-infra | 19:59 | |
| fungi | clarkb: oh, except we can't land that patch because nodepool tests bitrotted and i haven't gotten to the bottom of it yet | 20:00 |
| *** asettle has joined #openstack-infra | 20:00 | |
| fungi | clarkb: something to do with the allocator tests | 20:00 |
| fungi | AttributeError: 'RoundRobinTestCase' object has no attribute 'results' | 20:00 |
| fungi | i'm thinking that's a testing-cabal new release of something | 20:01 |
| fungi | AttributeError: 'AllocatorTestCase' object has no attribute 'results' | 20:01 |
| fungi | et cetera | 20:01 |
| mtreinish | fungi: do you have a full stack trace I can give you a hand | 20:02 |
| *** bdpayne has joined #openstack-infra | 20:02 | |
| fungi | mtreinish: http://logs.openstack.org/74/136474/1/check/gate-nodepool-python27/ca32976/console.html | 20:02 |
| *** sabeen1 has joined #openstack-infra | 20:02 | |
| krotscheck | clarkb: You free enough to look at those logs for me? | 20:02 |
| fungi | mtreinish: is this something you recently corrected in tempest? | 20:02 |
| *** sabeen2 has quit IRC | 20:02 | |
| *** nibalizer has joined #openstack-infra | 20:02 | |
| fungi | krotscheck: what specifically did you need in the sb logs? something about a failed login... | 20:02 |
| *** banix has quit IRC | 20:03 | |
| mtreinish | fungi: we had an issue were attrs we set in tempest weren't getting added to the test_id | 20:03 |
| krotscheck | fungi: Yeah, mbitard was getting a 500 error when logging in, and all the weird test cases I’ve tried to build locally have all passed :/ | 20:03 |
| mtreinish | which broke regex filtering... | 20:03 |
| mtreinish | this looks to be different | 20:03 |
| *** blkperl has quit IRC | 20:04 | |
| *** Hunner has quit IRC | 20:05 | |
| fungi | krotscheck: you expect to see it mentioned in /var/log/apache2/storyboard-error.log or somewhere else? | 20:05 |
| *** bdpayne_ has joined #openstack-infra | 20:05 | |
| *** mudassirlatif has joined #openstack-infra | 20:05 | |
| krotscheck | fungi: Yep. | 20:05 |
| *** nibalizer has quit IRC | 20:05 | |
| fungi | aha, storyboard-ssl-error.log actually | 20:05 |
| *** emagana has quit IRC | 20:06 | |
| fungi | krotscheck: mbitard is not mentioned in that log file. have a timestamp i should be looking at? | 20:06 |
| *** emagana has joined #openstack-infra | 20:06 | |
| krotscheck | fungi: Wish I did, but no. Any 500’s in there? | 20:06 |
| *** mudassirlatif has quit IRC | 20:07 | |
| fungi | krotscheck: http response codes aren't listed in there unfortunately | 20:07 |
| krotscheck | fungi: Dagnabbit | 20:07 |
| *** bdpayne has quit IRC | 20:07 | |
| *** nadya has joined #openstack-infra | 20:08 | |
| krotscheck | fungi: Ok, last test. Do you see oauth_validator in there somewhere? | 20:08 |
| *** nadya is now known as Guest90907 | 20:08 | |
| fungi | krotscheck: i find tons of 500 responses in the access log though (just not echoed in the error log) | 20:08 |
| krotscheck | fungi: Oh yeah, that makes me feel happy | 20:09 |
| fungi | krotscheck: yeah, tracebacks mentioning oauth_validator | 20:09 |
| * krotscheck might be sarcastic. | 20:09 | |
| krotscheck | fungi: Can you grab one of those? | 20:09 |
| krotscheck | fungi: I believe that’s probably the culprit. | 20:09 |
| fungi | krotscheck: most recent is 18:56:22 utc today (a little over an hour ago) | 20:09 |
| fungi | krotscheck: can do, just a sec | 20:09 |
| *** emagana has quit IRC | 20:11 | |
| mtreinish | fungi: from what I can tell those failures are happening because the tests are running in the super class and they didn't used to | 20:14 |
| clarkb | fungi: ya likely testtools release | 20:14 |
| fungi | krotscheck: http://paste.openstack.org/show/136087 | 20:14 |
| * clarkb looks | 20:14 | |
| mtreinish | my guess is a discovery change in testtools when it moved to unittest2 | 20:14 |
| *** Hunner has joined #openstack-infra | 20:15 | |
| *** emagana has joined #openstack-infra | 20:15 | |
| fungi | krotscheck: that's everything which happened at 18:56:22 in the error log | 20:15 |
| krotscheck | fungi: Cool. That’s not the issue we’re looking for, but it’s probably the issue that’s preventing people from remaining logged in. | 20:15 |
| *** MarkAtwood has quit IRC | 20:15 | |
| j^2 | is it possible to get bugs reported to my #openstack-chef channel? | 20:15 |
| j^2 | when people open them that is | 20:15 |
| clarkb | mtreinish: oh something like that ok | 20:15 |
| *** nibalizer has joined #openstack-infra | 20:16 | |
| krotscheck | fungi: If I fix that it’ll hopefully clear up the lgo a bit. | 20:16 |
| fungi | j^2: someone would need to write a bug watcher bot. we don't have one (yet!) | 20:16 |
| clarkb | ya I think its finding the test cases in __init__.py despite it being __init__.py | 20:16 |
| j^2 | fungi: ah nice, yeah i figured; i couldnt find the docs on it ;) | 20:16 |
| j^2 | fungi: shouldn’t be *too* hard ;) | 20:16 |
| *** HeOS has joined #openstack-infra | 20:17 | |
| *** ldnunes has quit IRC | 20:17 | |
| *** r-daneel has quit IRC | 20:17 | |
| clarkb | mtreinish: so do we work around that by not inherting the base test class until we absolutely need it? | 20:17 |
| fungi | j^2: something which watched a pop/imap mailbox subscribed to a project in lp would probably be sufficient | 20:17 |
| *** zul has joined #openstack-infra | 20:18 | |
| j^2 | interesting idea | 20:18 |
| fungi | j^2: i _think_ that's the closest thing it has to a filtered event stream | 20:18 |
| mtreinish | clarkb: I can try to come up with something, I mean a simple fix is change the test runner regex to exclude __init__.py | 20:18 |
| mtreinish | they'll still get discovered by the tests won't be run | 20:18 |
| fungi | that seems like a reasonable workaround | 20:19 |
| clarkb | I think it would be nicer to see if we can change the class hierarchy to make it stop | 20:19 |
| clarkb | also I hate testtools2 | 20:19 |
| clarkb | I am going to refer to unitest2 + testtools as testtools2 | 20:19 |
| *** Guest90907 has quit IRC | 20:20 | |
| clarkb | let me whip something up quickly | 20:20 |
| *** mudassirlatif has joined #openstack-infra | 20:20 | |
| mtreinish | clarkb: yeah that works too, just move the code from tests/__init__.py to test.py | 20:20 |
| mtreinish | and change the imports | 20:21 |
| *** chandankumar has quit IRC | 20:21 | |
| *** mudassirlatif has quit IRC | 20:21 | |
| *** dprince has quit IRC | 20:22 | |
| *** annegent_ has joined #openstack-infra | 20:23 | |
| *** emagana has quit IRC | 20:24 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Fix test discovery by new testtools https://review.openstack.org/136481 | 20:24 |
| clarkb | mtreinish: ^ | 20:24 |
| *** emagana has joined #openstack-infra | 20:24 | |
| barnaby | hi, quick nodepool question, if I'm building my nodepool images from existing images (not using dib) is the assumption that I have pre-baked the ssh keys into it? I don't see any configuration in nodepool to specify the keypair or the security group. | 20:25 |
| mtreinish | clarkb: yeah that looks like it'll work. | 20:25 |
| mtreinish | clarkb: except I always get confused with the super() calls in setUp() are they going to be handled correctly? | 20:25 |
| clarkb | barnaby: the assumption is that the build scripst will bake an ssh ke into the nodes | 20:25 |
| fungi | barnaby: the expectation is that something your prep scripts or dib elements to to the image will install authorized keys entries for the relevant accounts on the filesystem | 20:26 |
| clarkb | mtreinish: ya that should work properly | 20:26 |
| *** annegent_ has quit IRC | 20:26 | |
| *** Hunner has quit IRC | 20:26 | |
| clarkb | mtreinish: jenkins should confirm or deny :) | 20:26 |
| fungi | barnaby: in our case, we do it by having prep scripts/dib apply a puppet manifest which sets up accounts and ~/.ssh/authorized_keys files | 20:26 |
| mtreinish | clarkb: heh, fair enough :) | 20:27 |
| barnaby | fungi, so the providers: images: <image-name>: setup: script.sh is what would do that? | 20:27 |
| fungi | barnaby: if you're not using diskimages (dib) instead, then yes script.sh should run something either directly or indirectly to set up accounts on your image | 20:28 |
| barnaby | great, thanks | 20:29 |
| *** tonytan4ever has quit IRC | 20:29 | |
| fungi | barnaby: if you're using diskimages, then you'd instead have a dib element run something to accomplish that | 20:29 |
| fungi | but same basic principle | 20:29 |
| *** hashar has joined #openstack-infra | 20:29 | |
| barnaby | yup, I just needed to understand the principal, I am too used to specify the keypair & security group when I launch instances | 20:30 |
| *** e0ne has joined #openstack-infra | 20:30 | |
| clarkb | barnaby: the potential problem with that is in many clouds it depends on metadata server | 20:31 |
| clarkb | which never works | 20:31 |
| barnaby | makes perfect sense | 20:31 |
| fungi | or more generally, it's hard to make portable across multiple cloud providers | 20:31 |
| fungi | some are using metadata service, some disk injection, some config drive, you have to hope cloud-init or whatever similar thing yuo're using knows where to find the data, et cetera | 20:32 |
| clarkb | mtreinish: my understanding of how it works is that you are called in breadth first order as long as each init function properly calls super | 20:32 |
| fungi | and as clarkb points out, for metadata service in particular, if it's flaky then that means unnecessarily discarding a lot of nodes | 20:32 |
| *** dangers is now known as dangers_away | 20:32 | |
| fungi | clarkb: it worked, so rushed approval in and will recheck your reqs bump momentarily | 20:33 |
| clarkb | tahnks | 20:33 |
| *** dimsum__ has joined #openstack-infra | 20:34 | |
| *** packet has joined #openstack-infra | 20:34 | |
| *** mudassirlatif has joined #openstack-infra | 20:35 | |
| mtreinish | clarkb: ok, that makes sense. My concern was on the super going to object(which would cause an attr error) instead of TestCase. | 20:35 |
| fungi | clarkb: also, i think we should go ahead soon and review/merge/tag a new gerritbot. it's been super-stable since i upgraded it on review.o.o | 20:35 |
| *** yamamoto has joined #openstack-infra | 20:35 | |
| *** mudassirlatif has quit IRC | 20:36 | |
| clarkb | fungi: awesome. what did we change? | 20:36 |
| clarkb | fungi: or did we just deplio from source instaed of a tag? | 20:36 |
| krotscheck | AGH. I cannot figure out this issue that mbitard is running into :/ | 20:37 |
| openstackgerrit | Merged openstack-infra/nodepool: Fix test discovery by new testtools https://review.openstack.org/136481 | 20:37 |
| krotscheck | I’ve tried every permutation of weird character, and they all work. | 20:37 |
| *** andreykurilin_ has quit IRC | 20:37 | |
| fungi | clarkb: well, we were running an oooooold release from 2012 for starters, and i can't remember now why we rolled back, but i upgraded to latest master plus dimsum__'s reconnect fix and it's been rock-solid for more than a week now | 20:37 |
| clarkb | awesome | 20:37 |
| *** ddieterly has quit IRC | 20:38 | |
| dimsum__ | i attribute it all to the latest master :) | 20:38 |
| anteaya | dhellmann: would love to have your content and presence during the infra manual sprint, thank you | 20:38 |
| dhellmann | anteaya: watch for patches :-) | 20:38 |
| clarkb | fungi: once the six dep change merges I am going to build a new host | 20:38 |
| anteaya | dhellmann: /me rubs her hands with glee | 20:39 |
| *** Sincler has joined #openstack-infra | 20:39 | |
| *** yamamoto has quit IRC | 20:40 | |
| fungi | dimsum__: i'm half expecting that whatever stability issues i vaguely recall us seeing with the more recent gerritbot are fixed by your patch | 20:40 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu https://review.openstack.org/136158 | 20:40 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT COMMIT disable config drive https://review.openstack.org/135831 | 20:40 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/134639 | 20:40 |
| dimsum__ | fungi: thanks! | 20:40 |
| mordred | krotscheck: have you tilted your head to the side? | 20:41 |
| fungi | krotscheck: can you get a reasonably precise timestamp and/or source ip address? i can try to narrow it down in the logs | 20:41 |
| mordred | krotscheck: have you tried turning it off and on again? | 20:41 |
| * krotscheck slaps mordred around a bit with a large trout. | 20:41 | |
| *** rcarrillocruz has quit IRC | 20:41 | |
| *** achanda has quit IRC | 20:41 | |
| mordred | mmm. fish | 20:42 |
| krotscheck | fungi: Maybe? The bug was reported in the channel at 16:12:22 UTC. | 20:42 |
| fungi | krotscheck: i'll try to see if anything around that timeframe is telling | 20:42 |
| krotscheck | fungi: Thanks | 20:43 |
| *** otherwiseguy has joined #openstack-infra | 20:43 | |
| *** AlexF has joined #openstack-infra | 20:43 | |
| fungi | mordred: can you weigh in on http://lists.openstack.org/pipermail/openstack-infra/2014-November/002126.html ? i'd like to go ahead and knock out a bunch of pending third-party ci account requests | 20:43 |
| mordred | fungi: it sounds very unlike something I would normally to do actually be useful | 20:44 |
| *** rcarrillocruz has joined #openstack-infra | 20:46 | |
| *** emagana has quit IRC | 20:46 | |
| *** koolhead17 has quit IRC | 20:46 | |
| anteaya | okay so Yongli He and Shane Wang and my contacts from Intel for setting up ci for SR-IOV for nfv | 20:46 |
| anteaya | if either of them show up, do find me | 20:46 |
| arosen | fungi: finally got my dib node pool nodes showing up in jenkins :) | 20:46 |
| *** koolhead17 has joined #openstack-infra | 20:47 | |
| mordred | arosen: woot! | 20:47 |
| morganfainberg | arosen, AHA you're hiding here! | 20:47 |
| anteaya | they know zip about openstack and have to get a ci running on their hardware | 20:47 |
| morganfainberg | arosen, i was trying to figure out where to find you. | 20:47 |
| arosen | morganfainberg: i'm in openstack-dev too :P | 20:47 |
| mordred | anteaya: that's fine - we don't know anything about openstack either :) | 20:47 |
| anteaya | awesome | 20:47 |
| anteaya | this is the place then | 20:47 |
| anteaya | flashgordon: I will need you as well when these folks show up | 20:47 |
| morganfainberg | arosen, i must have checked at the wrong time. | 20:48 |
| clarkb | mordred: I dunno this multinode stuff has me in a position where I can deplo opensatck with nova network | 20:48 |
| arosen | morganfainberg: what's up? | 20:48 |
| fungi | anteaya: their employer is our newest platinum board seat provider, so hopefully they learn quickly? | 20:48 |
| clarkb | mordred: its weird to finally understand how some of this stuff works :) | 20:48 |
| anteaya | here's hoping | 20:48 |
| mordred | clarkb: gross! | 20:48 |
| flashgordon | anteaya: let me know what I can do to help | 20:48 |
| anteaya | though I was hoping that intel would hire a bunch of folks like hp did and have found out that they are only hiring for santa clara | 20:48 |
| *** MarkAtwood has joined #openstack-infra | 20:49 | |
| anteaya | flashgordon: so far be on the lookout for anyone who resembles these names Yongli He and Shane Wang | 20:49 |
| anteaya | and welcome them | 20:49 |
| clarkb | I could drive out to hillsboro and find keith | 20:49 |
| anteaya | I have no idea what they know | 20:49 |
| clarkb | security would probably throw me out | 20:49 |
| *** julim has quit IRC | 20:49 | |
| anteaya | so getting them on irc and introduced around is my first step | 20:49 |
| anteaya | and thanks flashgordon | 20:49 |
| *** rushiagr is now known as rushiagr_away | 20:50 | |
| *** e0ne has quit IRC | 20:50 | |
| *** mudassirlatif has joined #openstack-infra | 20:50 | |
| *** cnesa has quit IRC | 20:50 | |
| anteaya | and I just found out they are both in china | 20:50 |
| *** mudassirlatif has quit IRC | 20:50 | |
| anteaya | how is your overlap with china flashgordon? | 20:51 |
| *** koolhead17 has quit IRC | 20:51 | |
| *** hdd has quit IRC | 20:51 | |
| openstackgerrit | Merged openstack-infra/nodepool: Match novaclients transitive six dep https://review.openstack.org/136474 | 20:51 |
| clarkb | fungi: ^ danke I am rebuilding new nodepool.o.o now | 20:51 |
| *** banix has joined #openstack-infra | 20:51 | |
| fungi | clarkb: w00t | 20:51 |
| clarkb | if it all comes together then we will have a non gearman listening nodepool building images | 20:52 |
| *** cnesa has joined #openstack-infra | 20:52 | |
| *** alexpilotti has joined #openstack-infra | 20:53 | |
| openstackgerrit | Terry Howe proposed openstack/requirements: Add in python-openstacksdk to requirements.txt https://review.openstack.org/136486 | 20:53 |
| clarkb | mtreinish: re ^ did you ever look into working with them | 20:53 |
| fungi | krotscheck: if i were really useful i'd open stories for all the various errors in this log | 20:53 |
| clarkb | I still have a hunch that tempest clients are what should be used | 20:53 |
| *** Hunner has joined #openstack-infra | 20:54 | |
| clarkb | since they are tested a million times a day | 20:54 |
| *** afazekas has quit IRC | 20:54 | |
| fungi | krotscheck: File does not exist: /var/lib/storyboard/www/config.jsonFile does not exist: /var/lib/storyboard/www/config.json | 20:54 |
| krotscheck | fungi: That would be really useful :) | 20:54 |
| fungi | krotscheck: it's like we get that on lots and lots of requests | 20:54 |
| krotscheck | fungi: yep, that’s by design. | 20:54 |
| mtreinish | clarkb: I asked jamielennox|away about it | 20:54 |
| fungi | krotscheck: by design we report that as an error? | 20:55 |
| krotscheck | fungi: Well, no. | 20:55 |
| krotscheck | fungi: But that’s an apache 404 log error, no? | 20:55 |
| fungi | krotscheck: possibly. as i said the error log doesn't echo http response codes | 20:55 |
| krotscheck | fungi: So it’s reporting accurately that the file isn’t there, but the client knows how to recover. | 20:55 |
| clarkb | puppet is running | 20:55 |
| fungi | krotscheck: Hostname storyboard.openstack.org. provided via SNI and hostname storyboard.openstack.org provided via HTTP are different | 20:56 |
| fungi | krotscheck: tons of that over and over too | 20:56 |
| *** cpowell has quit IRC | 20:56 | |
| *** tsufiev has quit IRC | 20:56 | |
| mtreinish | clarkb: there wasn't any overlap, they're building on top of the keystone token stuff from keystone client | 20:56 |
| *** tonytan4ever has joined #openstack-infra | 20:56 | |
| krotscheck | fungi: Huhn. That’s… interesteing. | 20:56 |
| clarkb | mtreinish: well the overlap is common sdk for openstack | 20:56 |
| fungi | krotscheck: SNI (whatever that is) seems to append a trailing . | 20:56 |
| *** cpowell has joined #openstack-infra | 20:56 | |
| clarkb | mtreinish: which you all have basically implemented | 20:57 |
| *** rakhmerov has quit IRC | 20:57 | |
| fungi | krotscheck: which under normal circumstances should be the same as long as you don't assume the possibility of relative hostnames (e.g. fqdn only) | 20:57 |
| flashgordon | anteaya: with china? not very good I think | 20:58 |
| clarkb | mordred: if I said the sky is blue would you agree with me on that even more? | 20:58 |
| flashgordon | I am California | 20:58 |
| *** AlexF has quit IRC | 20:58 | |
| *** prad has joined #openstack-infra | 20:58 | |
| clarkb | * except in the NW where it is gray | 20:58 |
| anteaya | flashgordon: yeah, not ideal | 20:59 |
| anteaya | hmmm anyone know when jhesketh is expected back on line? | 20:59 |
| *** tsufiev has joined #openstack-infra | 20:59 | |
| clarkb | anteaya: ~3 weeks post summit | 20:59 |
| clarkb | so after US thanksgiving I think | 20:59 |
| anteaya | kk thanks | 20:59 |
| jhesketh | anteaya: heya, still on holiday until next week | 20:59 |
| anteaya | jhesketh: heya | 20:59 |
| jhesketh | I should be back in aciton by the 26th | 20:59 |
| anteaya | okay well enjoy holiday | 20:59 |
| annegentle | accio jhesketh! Nice one anteaya :) | 21:00 |
| anteaya | I have a lineup of apac stuff to dump on you once you are here | 21:00 |
| clarkb | fungi: woot that got nodepool up and running but now the db thing. I am going to stop nodepool, edit config file then start it again | 21:00 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Setup /etc/hosts for aiopcpu https://review.openstack.org/136158 | 21:00 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT COMMIT disable config drive https://review.openstack.org/135831 | 21:00 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: testing aiopcpu with tempest full https://review.openstack.org/134639 | 21:00 |
| anteaya | annegentle: :D it works! | 21:00 |
| *** emagana has joined #openstack-infra | 21:00 | |
| anteaya | jhesketh: so rest up | 21:00 |
| anteaya | :D | 21:00 |
| jhesketh | currently in front of an open fire with a red in France | 21:00 |
| jhesketh | so doing well ;-) | 21:00 |
| anteaya | ohhhh | 21:00 |
| anteaya | nice | 21:00 |
| mordred | clarkb: with you being in portland I would find it hard to agree with a statement from you about a blue sky | 21:00 |
| anteaya | cheese? | 21:00 |
| anteaya | jhesketh: is there cheese involved? | 21:00 |
| jhesketh | anteaya: there was cheese... | 21:00 |
| clarkb | woot its doing stuff | 21:01 |
| anteaya | jhesketh: I'm jealous | 21:01 |
| clarkb | mordred: did you see my disclaimer :) but ya the rain came back | 21:01 |
| clarkb | we had nice sunny weather for a while was cold though | 21:01 |
| *** AlexF has joined #openstack-infra | 21:01 | |
| fungi | krotscheck: so no python exceptions in the error log for a couple hours before the problem was mentioned in irc then | 21:01 |
| clarkb | images are building \o/ | 21:01 |
| krotscheck | fungi: Got it. | 21:01 |
| *** rakhmerov has joined #openstack-infra | 21:01 | |
| krotscheck | fungi: That SNI error is related to running https as a virtual host apparently. | 21:02 |
| fungi | krotscheck: if that was a fresh observation then it's not resulting in any obviously related error messages. nothing but file not founds | 21:02 |
| *** mpaolino has joined #openstack-infra | 21:02 | |
| krotscheck | fungi: And, in particular, the hostname certificate resolution. | 21:02 |
| clarkb | ianw: I know it is your weekend but hopefully I can have dib building centos7 images next week | 21:02 |
| *** mpaolino has quit IRC | 21:02 | |
| krotscheck | fungi: Great. Dagnabbit. | 21:02 |
| clarkb | but now that I have said that I know it will fail :/ | 21:02 |
| *** shakamunyi_ has joined #openstack-infra | 21:03 | |
| krotscheck | fungi: Ok, I’ll circle around with the user again on monday and see where that leaves us. | 21:03 |
| fungi | krotscheck: if you can get a source ip address i can filter down the access log for any obvious non-error content | 21:03 |
| krotscheck | fungi: Thanks for your help. | 21:03 |
| clarkb | we have a lot of images to build... | 21:03 |
| clarkb | poor git farm | 21:03 |
| *** cnesa has quit IRC | 21:03 | |
| anteaya | and a book I ordered came in so I am going to zip up and pick it up, back in a bit | 21:03 |
| *** packet has quit IRC | 21:03 | |
| krotscheck | fungi: I’ll see if I can get that plus the URL in quesiton. | 21:03 |
| fungi | clarkb: is qemu-utils installed? https://review.openstack.org/125515 | 21:04 |
| clarkb | fungi: it is | 21:04 |
| *** mudassirlatif has joined #openstack-infra | 21:05 | |
| *** mudassirlatif has quit IRC | 21:05 | |
| fungi | (which btw i -1'd due to redundancy) | 21:06 |
| fungi | arosen: ^ feedback? | 21:06 |
| clarkb | fungi: whats redundant? | 21:06 |
| clarkb | oh ya | 21:07 |
| fungi | clarkb: python-yaml as we already have PyYAML in the requirements.txt | 21:07 |
| clarkb | good catch | 21:07 |
| *** ddieterly has joined #openstack-infra | 21:07 | |
| morganfainberg | so, looks like we (keystone) are missing our logs in gate now. https://review.openstack.org/#/c/116404/3 this was abandoned but the devstack screen thing isn't being captured. | 21:07 |
| clarkb | morganfainberg: what does the abandoned change have to do with it? | 21:09 |
| morganfainberg | clarkb, it was trying to capture the apache logs | 21:09 |
| clarkb | morganfainberg: we are capturing all apache logs and should be getting all screen logs too | 21:09 |
| morganfainberg | directly | 21:09 |
| morganfainberg | hm... | 21:10 |
| fungi | morganfainberg: picking a recently completed devstack-gate job at random, i agree | 21:10 |
| clarkb | morganfainberg: can you link to example? | 21:10 |
| morganfainberg | clarkb, sec. | 21:10 |
| fungi | http://logs.openstack.org/48/136248/7/gate/gate-tempest-dsvm-neutron-pg-full/2e2f826/logs/ | 21:10 |
| bknudson | http://logs.openstack.org/99/136399/2/check/check-tempest-dsvm-full/d9f0e7d/logs/ | 21:10 |
| morganfainberg | http://logs.openstack.org/84/118884/7/check/check-tempest-dsvm-full/29c5132/logs/ | 21:10 |
| bknudson | there's no horizon logs either, right? | 21:10 |
| bknudson | oh, there's horizon_error.txt.gz | 21:11 |
| fungi | bknudson: screen-h-* | 21:11 |
| clarkb | or mabe I only +2'd that change | 21:11 |
| morganfainberg | http://logs.openstack.org/84/118884/7/check/check-tempest-dsvm-full/29c5132/logs/screen-key.txt.gz | 21:11 |
| clarkb | fungi: thats heat | 21:11 |
| fungi | d'oh! | 21:11 |
| clarkb | so I am going to guess a devstack change | 21:11 |
| fungi | we need to avoid projects starting with the same error. there are already too many ;) | 21:11 |
| clarkb | or maybe the multinode thing | 21:11 |
| clarkb | except that the other screen logs are retrieved | 21:12 |
| clarkb | which really makes me think devsatck | 21:12 |
| morganfainberg | yeah | 21:12 |
| morganfainberg | i honestly don't know when these stopped being captured | 21:13 |
| morganfainberg | just noticed it today | 21:13 |
| morganfainberg | 2 things, yay for keystone not breaking things | 21:13 |
| fungi | http://logs.openstack.org/48/136248/7/gate/gate-tempest-dsvm-neutron-pg-full/2e2f826/logs/devstacklog.txt.gz#_2014-11-21_20_27_24_080 | 21:13 |
| morganfainberg | boo for making it hard to find. | 21:13 |
| fungi | morganfainberg: so running under apache we should be collecting the apache logs instead i guess? | 21:13 |
| morganfainberg | fungi, yeah. | 21:13 |
| clarkb | there is a change to get those logs | 21:14 |
| clarkb | morganfainberg: what happened to the screen logs? | 21:14 |
| *** emagana has quit IRC | 21:14 | |
| clarkb | I thought we had those even with apache | 21:14 |
| morganfainberg | clarkb, i don't know. i was sure we did for a while. | 21:14 |
| bknudson | I'll try run devstack with keystone in httpd here and see if I get some logs. | 21:14 |
| *** emagana has joined #openstack-infra | 21:14 | |
| clarkb | https://review.openstack.org/#/c/133881/ | 21:14 |
| bknudson | screens | 21:14 |
| clarkb | thats the one we want I think | 21:15 |
| clarkb | hrm I approved it why didn't it merge | 21:15 |
| fungi | conveniently, already approved | 21:15 |
| morganfainberg | clarkb, depends on a non-approved patch? | 21:15 |
| fungi | clarkb: dependency | 21:15 |
| morganfainberg | https://review.openstack.org/#/c/133914/5 | 21:15 |
| bknudson | http://logs.openstack.org/81/133881/6/check/check-tempest-dsvm-full/72e1059/logs/apache/ | 21:15 |
| fungi | clarkb: reviewing now, you already have added you blessing | 21:15 |
| bknudson | has keystone.txt and keystone_access.txt | 21:16 |
| fungi | your | 21:16 |
| *** zul has quit IRC | 21:16 | |
| arosen | fungi: i'm seeing this warning from jenkins about my dib images;: Disk space is too low. Only 0.993GB left on /home/jenkins. i guess that is fine? | 21:16 |
| clarkb | arosen: depends on how much disk you need to run your tests | 21:16 |
| clarkb | it also does things like complain if there is no swap | 21:16 |
| clarkb | jenkins is funny | 21:16 |
| morganfainberg | yeah http://logs.openstack.org/81/133881/6/check/check-tempest-dsvm-full/72e1059/logs/apache/ that does have the output | 21:17 |
| morganfainberg | from the one you were asking why it hadn't merged | 21:17 |
| clarkb | morganfainberg: ya I know I had checked it before I approved | 21:17 |
| morganfainberg | ahh | 21:17 |
| arosen | clarkb: is disksize configurable somewhere? | 21:17 |
| clarkb | arosen: when you boot your nodes with the flavor | 21:17 |
| *** Sincler has quit IRC | 21:17 | |
| fungi | morganfainberg: that should be headed into the gate pipeline nowish | 21:18 |
| *** AlexF has quit IRC | 21:18 | |
| clarkb | arosen: your dib images should autoexpand to fill the entire disk space available to the VM | 21:18 |
| morganfainberg | fungi, ++ thanks :) | 21:18 |
| arosen | clarkb: hrm in my experience the disk size is defined by: qemi-image create xxx 40G | 21:18 |
| *** amitgandhinz has joined #openstack-infra | 21:18 | |
| *** emagana has quit IRC | 21:19 | |
| clarkb | arosen: thats another way to do it | 21:19 |
| clarkb | arosen: but not the way dib does it | 21:19 |
| clarkb | bceause you don't know where you will be booting these | 21:19 |
| *** mudassirlatif has joined #openstack-infra | 21:20 | |
| arosen | clarkb: right now my flavor for Disk says 0 | 21:20 |
| arosen | probably need to update that in the cloud to something other than 0 i guess. | 21:20 |
| clarkb | arosen: probably | 21:20 |
| *** mudassirlatif has quit IRC | 21:20 | |
| clarkb | most of our nodes come with a 20 or 30GB / then a much larger ephemeral disk that we mount under /opt | 21:21 |
| clarkb | but /opt has nothing to do with the dib images in those cases | 21:21 |
| fungi | arosen: you're uncovering bugs we don't see, as we don't maintain a cloud and define our own flavors | 21:21 |
| fungi | we're stuck with whatever the service provider has set forth | 21:22 |
| *** ivar-lazzaro has quit IRC | 21:26 | |
| *** shakamunyi_ has quit IRC | 21:26 | |
| sdague | anyone up for pruning a few jobs - https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:devstack_pruning,n,z | 21:26 |
| *** ivar-lazzaro has joined #openstack-infra | 21:26 | |
| arosen | fungi: :) | 21:27 |
| *** e0ne has joined #openstack-infra | 21:27 | |
| clarkb | sdague: how did video go? | 21:28 |
| sdague | clarkb: I think well, it's up on the youtubes now | 21:29 |
| sdague | I just send the link to the list | 21:29 |
| clarkb | I dont think https://review.openstack.org/#/c/136367/3 is right | 21:29 |
| *** mrmartin has quit IRC | 21:30 | |
| clarkb | you want to restrict the branch for the nonvoting jobs too | 21:30 |
| *** e0ne has quit IRC | 21:30 | |
| clarkb | sdague: fungi ^ | 21:31 |
| fungi | clarkb: perhaps, but if so it needs a separate section | 21:32 |
| *** jungleboyj has quit IRC | 21:32 | |
| fungi | clarkb: the problem is that entry was shadowing the ^.*-nv entry and unsetting the voting:false | 21:32 |
| clarkb | ya but now you hvae broken all of those tjos | 21:32 |
| fungi | clarkb: i think it causes check-tempest-dsvm-ironic-pxe_ssh-nv to run non-voting on stable/icehouse | 21:33 |
| clarkb | right which isn't a cleanup | 21:33 |
| fungi | clarkb: right, potentially undesirable, but not blocking anything. still i'll unapprove for edits | 21:34 |
| fungi | done. feel free to -1 with details | 21:34 |
| *** mudassirlatif has joined #openstack-infra | 21:35 | |
| *** bitblt has joined #openstack-infra | 21:35 | |
| *** achanda has joined #openstack-infra | 21:35 | |
| *** mudassirlatif has quit IRC | 21:35 | |
| clarkb | well I think second nodepool has derped the world | 21:36 |
| fungi | eh? | 21:36 |
| openstackgerrit | Merged openstack-infra/project-config: centos7 devstack jobs should not be run on icehouse https://review.openstack.org/136363 | 21:36 |
| clarkb | look at zuul status | 21:36 |
| clarkb | I do not yet understand how this could have happened | 21:37 |
| clarkb | but its happening | 21:37 |
| clarkb | unless hrm | 21:37 |
| clarkb | maybe its the log server? | 21:37 |
| clarkb | and I am just keyed in on nodepool | 21:37 |
| fungi | looking | 21:38 |
| sdague | clarkb: so the ironic folks have a bunch of compounding logic here I need to unwind? | 21:38 |
| clarkb | logs claims to have tons of disk space | 21:38 |
| clarkb | sdague: yes | 21:38 |
| * sdague looks at adam_g sternly | 21:38 | |
| clarkb | sdague: you basically need to preserve the exclusion of the stable branch on the -nv jobs. just add a second section | 21:38 |
| *** wenlock has quit IRC | 21:38 | |
| *** cnesa has joined #openstack-infra | 21:38 | |
| fungi | clarkb: xvdk seems to be f'd | 21:39 |
| clarkb | woo | 21:39 |
| fungi | clarkb: dmesg is spammed out of existence on static.o.o | 21:39 |
| fungi | clarkb: i'll see if i can recover, but this is going to be hella messy | 21:39 |
| clarkb | ok let me know if I need to undistract myself from nodepool | 21:40 |
| fungi | end_request: I/O error, dev xvdk, sector blah | 21:40 |
| fungi | all over | 21:40 |
| fungi | i'm wondering how best to contain the damage, because a fsck is going to take a _while_ | 21:40 |
| clarkb | and xvdk is one of the volumes behind /srv/logs? | 21:41 |
| fungi | yah | 21:41 |
| clarkb | fungi: we might just want to stop zuul | 21:41 |
| clarkb | as much as that sucks | 21:41 |
| fungi | we could put all teh jenkins masters in prepare for shutdown maybe | 21:41 |
| fungi | and let zuul queue a ton of crap | 21:41 |
| clarkb | ya that would also quiet things | 21:41 |
| clarkb | and potentially allow zuul to catch back up over weekend | 21:41 |
| fungi | so that it doesn't get lost | 21:41 |
| clarkb | +1 | 21:42 |
| fungi | i'll start that now | 21:42 |
| clarkb | ok | 21:42 |
| fungi | #status alert gating is going offline while we deal with a broken block device, eta unknown | 21:42 |
| openstackstatus | fungi: sending alert | 21:42 |
| *** mudassirlatif has joined #openstack-infra | 21:44 | |
| -openstackstatus- NOTICE: gating is going offline while we deal with a broken block device, eta unknown | 21:44 | |
| *** ChanServ changes topic to "gating is going offline while we deal with a broken block device, eta unknown" | 21:44 | |
| openstackstatus | fungi: finished sending alert | 21:46 |
| fungi | okay, all 8 masters are in prepare for shutdown now | 21:47 |
| fungi | i'm going to start by unmounting everything in /srv | 21:47 |
| fungi | and put my plans for a spicy friday run drink on hold ;) | 21:48 |
| adam_g | sdague, clarkb there is a patch to name those properly https://review.openstack.org/#/c/134620/ tho im having trouble convincing people thats correct | 21:48 |
| *** funzo has joined #openstack-infra | 21:48 | |
| sdague | adam_g: so, to be honest, I don't think we should be running those 2 jobs on devstack changes | 21:48 |
| arosen | clarkb: fungi changing the size value on flavor seemed to fix this disk issue i was having :) | 21:48 |
| clarkb | fungi: I hve the cinder uuid handy if you need it | 21:48 |
| *** bswartz has quit IRC | 21:49 | |
| clarkb | fungi: for that particular volume | 21:49 |
| adam_g | sdague, how about one? | 21:49 |
| sdague | I think we need to isolate the parts for setting up ironic correctly and run a smaller thing | 21:49 |
| *** sandywalsh has quit IRC | 21:49 | |
| adam_g | sdague, that jobs been broken multiple times by changes to devstack prior to it being voting | 21:49 |
| clarkb | fungi: and show on that volume doesn't say much | 21:49 |
| sdague | so, let's figure out what parts of devstack need to be constrained, and test those parts | 21:50 |
| clarkb | I guess cinder doesn't have the same sort of state machine for states like nova | 21:50 |
| *** matrohon has quit IRC | 21:50 | |
| *** whoops has joined #openstack-infra | 21:50 | |
| sdague | we don't make this system any less fragile by co-gating everything | 21:50 |
| adam_g | sdague, that job is still currently using the tempest regex to target a small subset of tests, i'd be happy limiting it further to only tempest.api.baremetal.* and tempest.scenario.*baremetal* | 21:50 |
| clarkb | fungi: also you might want a drink :P | 21:51 |
| jgriffith | clarkb: we have *no* state machine :( | 21:51 |
| fungi | i've taken static.o.o out of /etc/ansible/hostfile on puppetmaster.o.o so i can disable the bandersnatch cron | 21:51 |
| sdague | anyway, it's basically friday. I'll cut you a deal of +2 a change that gets us down to 1 job for now, but realize I do think that devstack shouldn't run every config under the sun. We need to pick a few cross sections, and just hotfix the rest when it's an issue. | 21:51 |
| fungi | though that should just get deleted soon anyway | 21:52 |
| clarkb | fungi: you should be able to just puppet agent --disable? | 21:52 |
| clarkb | on satic.o.o | 21:52 |
| fungi | clarkb: thanks, i'll do that and put it back | 21:52 |
| adam_g | sdague, hot fixing that stuff after the fact is what we were doing in the past and it was insanely painful--ironic was basically unable to merge code for days at a time | 21:53 |
| clarkb | jgriffith: so question for you | 21:53 |
| *** doug-fish has left #openstack-infra | 21:53 | |
| jgriffith | clarkb: sure | 21:53 |
| sdague | adam_g: it is also insanely painful for me not to be able to fix other project's issues with devstack because ironic is racing | 21:53 |
| clarkb | jgriffith: rax says use raid over the top of your volumes. Does cinder make sure that volumes for a single tenant do not all end up on the same underlying device? | 21:53 |
| clarkb | jgriffith: because us doing raid is pretty useless if cinder doesn't do that | 21:53 |
| jgriffith | clarkb: nope, but I have a solution for that :) | 21:54 |
| sdague | so all you are saying is it's better for other people to feel pain than you... I get everyone feels that way, however that's not leading us to a less fragile world | 21:54 |
| * fungi hopes its a snarky solutiojn | 21:54 | |
| clarkb | ok so rax shouldn't actually suggest that like it will make the world better for us | 21:54 |
| clarkb | jgriffith: good to know | 21:54 |
| jgriffith | clarkb: in the general case no, but RAX is different | 21:54 |
| jgriffith | clarkb: RAX backend for CBS is kinda unique | 21:55 |
| clarkb | jgriffith: except that we run on openstack | 21:55 |
| jgriffith | clarkb: touchet | 21:55 |
| clarkb | and shouldn't take advantage of that too much | 21:55 |
| jgriffith | clarkb: we're working on DRBD | 21:55 |
| *** dkranz has quit IRC | 21:56 | |
| jgriffith | clarkb: also still have a BP somewhere to auto create two vols on seperate backends via a special type and do the raid thing | 21:56 |
| jgriffith | clarkb: if there's interest from anybody but me I'd be thrilled to have an excuse to work on the patch for it | 21:56 |
| adam_g | sdague, thats not what i am saying, and singling out ironic in that regard isn't fair, sorry | 21:57 |
| clarkb | jgriffith: I think we mostly want a better story for resiliency in our 14x1TB volume lvm fs | 21:57 |
| *** belmoreira has quit IRC | 21:57 | |
| clarkb | jgriffith: unfortunately cinder doesn't give us good tools for that | 21:57 |
| clarkb | I think we are almost btter off running ceph on cinder | 21:57 |
| *** emagana has joined #openstack-infra | 21:57 | |
| clarkb | across tenants | 21:57 |
| jgriffith | clarkb: yeah, I've been waiting for that | 21:57 |
| sdague | adam_g: you are not getting singled out, you are just first | 21:57 |
| clarkb | but even then we don't actually get assurances that our data doesn't all end up on the same underlying device | 21:58 |
| jgriffith | clarkb: TBH that may be true, however... | 21:58 |
| sdague | because ironic actually got in the way of fixing real issues this week | 21:58 |
| jgriffith | clarkb: if you wait another month or so we'll have a better answer for you | 21:58 |
| jgriffith | clarkb: the DRBD work is coming along, but there's a backlog of things in front of it | 21:58 |
| clarkb | jgriffith: and with that cinder basically runs with a backup mirror for each volume? | 21:59 |
| jgriffith | clarkb: yup | 21:59 |
| *** sarob has quit IRC | 21:59 | |
| fungi | okay, i've managed to cleanly umount everything on that vg | 21:59 |
| jgriffith | clarkb: and this time the linbit guys are working on it with the newer version of DRBD | 21:59 |
| jgriffith | clarkb: when I tried it last summer it was a 40% perf hit :( | 21:59 |
| jgriffith | but they know how to fix that | 22:00 |
| jgriffith | clarkb: anyway, don't know what your timelines are like or if that even works for you or not | 22:00 |
| *** salv-orlando has joined #openstack-infra | 22:00 | |
| *** banix has quit IRC | 22:00 | |
| jgriffith | clarkb: I'm assuming your proposal is the "undercloud" or whatever it's called these days | 22:01 |
| clarkb | jgriffith: well we just dropped a volume | 22:01 |
| jgriffith | clarkb: and the gating defaults remain the same, just on top of something a bit more robust | 22:01 |
| clarkb | jgriffith: so its topical :) | 22:01 |
| jgriffith | clarkb: not sure what you mean 'dropped a volume' | 22:01 |
| clarkb | poor fungi is in the trenches fixing it | 22:01 |
| clarkb | jgriffith: xvdk died on us | 22:01 |
| jgriffith | oh | 22:02 |
| jgriffith | bleh | 22:02 |
| *** emagana has quit IRC | 22:02 | |
| fungi | vgchange -a n is hanging on me | 22:02 |
| *** prad has quit IRC | 22:02 | |
| adam_g | sdague, how is ironic different here than any of the other known gate issues causing rechecks? i know there is a bug atm and ive spent the better part of the week trying to get to the bottom of it, but there are many others that happen much more frequently that aren't being threatened with removal | 22:02 |
| *** eharney has quit IRC | 22:02 | |
| clarkb | fungi: the underlying volume may still be dead? | 22:03 |
| clarkb | we may need to ping rax | 22:03 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Update ironic pxe job names to reflect voting status https://review.openstack.org/134620 | 22:03 |
| sdague | adam_g: I've been looking at all kinds of jobs to remove this week | 22:04 |
| clarkb | jeblair: you may be interested if your morning things are done | 22:04 |
| sdague | and I've been deleting and skipping tests left right and center the last 2 weeks when they are unreliable | 22:04 |
| *** sarob has joined #openstack-infra | 22:04 | |
| fungi | clarkb: one step ahead of you... already logged into their ticket system | 22:04 |
| arosen | clarkb: fungi have you guys tested running 1 jenkins with multiple different nodepool machines? | 22:04 |
| clarkb | fungi: fi you want volume uuid I still have | 22:04 |
| clarkb | arosen: doing it right now | 22:04 |
| clarkb | arosen: it mostly sort of does work | 22:05 |
| adam_g | sdague, 134620 updated to remove the postgres job and update naming to reflect reality | 22:05 |
| arosen | clarkb: it looks like that State doesn't always go to used :( | 22:05 |
| fungi | it had to timeout with writes to /dev/xvdk1 but i was finally able to get the vg deactivated | 22:05 |
| sdague | adam_g: it also shouldn't add to the nova gate | 22:05 |
| *** dangers_away is now known as dangers | 22:05 | |
| sdague | honestly, all the pg jobs should be removed | 22:05 |
| arosen | i wonder if the zmq stuff sends it to the wrong nodepool node maybe? | 22:05 |
| sdague | testing that db layer should be back in ironic to assume equivalency | 22:05 |
| *** matrohon has joined #openstack-infra | 22:06 | |
| arosen | clarkb: you see anything like this? | 22:06 |
| clarkb | arosen: no | 22:06 |
| clarkb | arosen: jenkins uses a pubsub so all nodes should get the onstarted events | 22:06 |
| adam_g | sdague, fine by me--the nova job was added during review b/c there seems to be an underwritten policy of ensuring voting check jobs run in the gate, happy to remove it | 22:06 |
| *** MarkAtwood has quit IRC | 22:07 | |
| arosen | clarkb: hrm maybe it's just a weird bug if you don't have enough workers rready or something. | 22:07 |
| sdague | adam_g: yeh, honestly, that symetry really isn't needed with clean check, and reducing the spurious gate resets is worth running a lighter set on the gate | 22:07 |
| arosen | it seems to be working now | 22:07 |
| *** emagana has joined #openstack-infra | 22:07 | |
| arosen | tried it a few additional times but i still have a node in ready state that jenkins is using | 22:07 |
| clarkb | arosen: it is possible for the first event to be missed | 22:08 |
| adam_g | sdague, that sounds appropriate to me and was under the impression thats idea for devstack/tempest/dsg--the projects i wasn't sure of | 22:08 |
| *** shashankhegde has quit IRC | 22:08 | |
| clarkb | so there is a race there we have mostly ignored since we offline on jenkins side then cleanup after 8 hours | 22:08 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: move nova-tox-functional to experimental until there is content https://review.openstack.org/136420 | 22:09 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make devstack docs job only run on master https://review.openstack.org/136362 | 22:09 |
| fungi | clarkb: you grabbed the xvdk uuid reported by cinderclient? | 22:10 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Update ironic pxe job names to reflect voting status https://review.openstack.org/134620 | 22:10 |
| clarkb | fungi: ya want it? | 22:10 |
| clarkb | fungi: 53ae3137-de6b-4da0-9281-e1d86660f1d9 claims to be xvdk when you run show on it | 22:10 |
| adam_g | sdague, updated to clean out postgres from everywhere other thanir | 22:10 |
| fungi | clarkb: you bet. will save me several seconds since i want to paste it into the ticket | 22:10 |
| fungi | clarkb: thanks! | 22:10 |
| sdague | adam_g: coolio | 22:10 |
| sdague | adam_g: ok, I dropped mine | 22:11 |
| sdague | +2 | 22:11 |
| clarkb | new nodepool has almost built all the images | 22:11 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: DO NOT MERGE: pulling in unmerged devstack patch https://review.openstack.org/136504 | 22:11 |
| *** shashankhegde has joined #openstack-infra | 22:11 | |
| fungi | i've uncommented all the lvm-contained filesystems in /etc/fstab and am about to poweroff and recycle static.o.o | 22:12 |
| fungi | just fyi | 22:12 |
| clarkb | fungi: I am seeing jobs starting on some jenkinses | 22:12 |
| clarkb | fungi: uncommented? | 22:12 |
| clarkb | so they will mount and fsck? | 22:12 |
| fungi | clarkb: commented out, sorry | 22:13 |
| clarkb | I am going to look at jenkinses and shutdown mode them more | 22:13 |
| clarkb | fungi: kk | 22:13 |
| adam_g | sdague, plz dont think the racing job isn't on my radar--i've been trying to get more debug out of the gate and have been running the test in a tight loop for the last 3 days trying to reproduce locally. i will end up bribing clarkb with margaritas next week to hold a node for me if nothing else | 22:13 |
| mordred | mmm. margaritas | 22:13 |
| sdague | adam_g: right, I don't actually think it's not on your radar | 22:13 |
| fungi | i will manually activate the vg and if possible fsck the lvs once i recycle it | 22:13 |
| clarkb | fungi: awesome | 22:13 |
| clarkb | fungi: godspeed | 22:13 |
| clarkb | or cthuluspeed? | 22:13 |
| sdague | but I think we've gotten in this reactive mode where any break between systems means we glue everything into 1 ball | 22:13 |
| fungi | clarkb: i pray to the great old ones | 22:13 |
| sdague | which we need to break the habit of | 22:14 |
| fungi | clarkb: may their insanity spur me onward | 22:14 |
| mordred | sdague: (btw, reading scrollback, we should totally delete postgres) | 22:14 |
| *** cpowell has quit IRC | 22:14 | |
| adam_g | sdague, +1 | 22:14 |
| * adam_g relocates to cafe | 22:14 | |
| sdague | and be much more selective about "what happened that caused the break, how do we specifically isolate that issue" | 22:14 |
| clarkb | mordred: are you catching the great turkey day log outage of 2014 in scrollback too? | 22:14 |
| clarkb | I am going to try and take advantage to speed up the nodepool switches | 22:15 |
| clarkb | since idle jenkinses means I can clean up old nodepool more aggressively | 22:15 |
| fungi | this might be the great purge of all existing static web content if we're particularly unlucky | 22:15 |
| mordred | clarkb: I am not - I am still very out of touch with anything that is happening | 22:15 |
| *** _buzz has quit IRC | 22:15 | |
| fungi | mordred: me too! | 22:15 |
| clarkb | fungi: :/ | 22:15 |
| sdague | speaking of which.... on the libraries from pypi front, I'm realizing that we probably don't need to do both n-net and neutron jobs | 22:16 |
| sdague | that if we just do neutron we should cover things | 22:16 |
| clarkb | does n-net not use libraries? | 22:16 |
| sdague | it does | 22:16 |
| sdague | but it also has unit tests | 22:16 |
| sdague | so ... the likelihood that a library changes in a way that *only* breaks the 4% of nova code in n-net daemon, and nothing else in openstack, seems small | 22:17 |
| *** dkliban is now known as dkliban_afk | 22:17 | |
| *** ryanpetrello has quit IRC | 22:17 | |
| sdague | again, back to this, stop using an elephant gun to swatch a fly issue | 22:18 |
| fungi | Couldn't find device with uuid UI8f7L-pZ6O-Wv8n-OJs3-3492-QIEh-Dv3VJi. | 22:18 |
| fungi | that's after a hard reboot | 22:18 |
| fungi | i'll see if cinder detach/attach helps | 22:18 |
| clarkb | sdague: so I like it theory problem is we have proven that the untangling doesn't work | 22:18 |
| clarkb | we have alrady tripped over like 5 things this week alone | 22:18 |
| *** emagana has quit IRC | 22:18 | |
| sdague | clarkb: we have proven that edge cases still exist | 22:19 |
| *** emagana has joined #openstack-infra | 22:19 | |
| sdague | but, honestly, on paper you could have seen the stable branch issue with oslo coming | 22:19 |
| barnaby | ok, I think one last nodepool (or related question jenkins) question. What is the groovy script to take a builder offline in order to ensure that another job doesn't run on a node before nodepool can take it out of rotation? | 22:19 |
| clarkb | barnaby: we don't use one. instead the gearman plugin we use has the option to put the node offline as part of starting jobs | 22:20 |
| sdague | but we need to fix edge cases with edge fixes, not with giant bunkers | 22:20 |
| *** groknix has quit IRC | 22:21 | |
| *** groknix has joined #openstack-infra | 22:21 | |
| fungi | had to rescue ~root/ci-launch/cinder.sh out of the ci-puppetmaster:~root tarball | 22:21 |
| clarkb | sdague: I like it in theory I just think we have only made it more ocmplicated that the old thing though | 22:22 |
| sdague | clarkb: we have clearly made it more complicated | 22:22 |
| clarkb | debugging these issues is a lot more convoluted and requires a lot more lockstep process than it did before | 22:22 |
| sdague | however, the old way doesn't work unless we have 10x more nodes | 22:23 |
| sdague | and it honestly wasn't helping a ton on debugging | 22:23 |
| *** emagana has quit IRC | 22:24 | |
| *** melwitt has joined #openstack-infra | 22:24 | |
| *** sabeen1 has quit IRC | 22:24 | |
| clarkb | fungi: what does that do? | 22:25 |
| *** dims_ has joined #openstack-infra | 22:26 | |
| fungi | clarkb: http://ci.openstack.org/sysadmin.html#cinder | 22:26 |
| clarkb | oh its the env file. I just used the normal launch env | 22:26 |
| fungi | clarkb: i've detached and reattached that volume now | 22:26 |
| clarkb | for my cinder list and cinder show | 22:26 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: only test libraries in a neutron config https://review.openstack.org/136512 | 22:26 |
| *** SumitNaiksatam has quit IRC | 22:27 | |
| *** hashar has quit IRC | 22:27 | |
| fungi | clarkb: and pvs no longer throws errors about xvdk1 which is helpful | 22:27 |
| clarkb | yay | 22:27 |
| clarkb | fungi: you going to fsck ya? | 22:27 |
| fungi | clarkb: yep, able to vgchange -a y so that's next on the agenda | 22:27 |
| sdague | ok, time for beers with mtreinish for his birthday | 22:28 |
| clarkb | (note nodepool is near a consistent state since jenkinses are not doing anything so I can easily help with the volume fire or take advantage and flip the nodes over) | 22:28 |
| clarkb | oh I never dropped the DNS record TTL /me does that now | 22:28 |
| arosen | What component tells jenkins to run a job? I see commits are queuing up on zuul but jenkins isn't running them ? | 22:29 |
| fungi | fsck'd all other lvs in that vg, and /dev/mapper/main-logs is running a fsck now | 22:29 |
| *** dimsum__ has quit IRC | 22:29 | |
| clarkb | arosen: zuul. we have stopped running any jobs until we can log them | 22:29 |
| *** melwitt1 has joined #openstack-infra | 22:29 | |
| clarkb | as the broken log fs has resulted in failed jos across the board | 22:30 |
| clarkb | fungi is doing his magic to fix it | 22:30 |
| arosen | clarkb: this wouldn't have done anything on my end though would it? | 22:30 |
| clarkb | (super interseted in a run down of what you did when its all done) | 22:30 |
| clarkb | arosen: no | 22:30 |
| fungi | it's not _my_ magic. i just push the buttons. someone else wrote the magic | 22:30 |
| *** matrohon has quit IRC | 22:31 | |
| *** SumitNaiksatam has joined #openstack-infra | 22:31 | |
| *** melwitt has quit IRC | 22:31 | |
| * fungi does not like to take credit for the awesome of others | 22:31 | |
| clarkb | A and AAAA records for nodepool.o.o shortened to 5 minutes | 22:31 |
| fungi | good call | 22:31 |
| fungi | were they an hour before? | 22:32 |
| clarkb | ya | 22:32 |
| clarkb | :/ | 22:32 |
| clarkb | I got all excited by the volume thing and forgot that I need to update DNS | 22:32 |
| fungi | so in an hour you can switch dns rrs i guess | 22:32 |
| clarkb | ya and that when the majority of the switch happens. will allow me to connect via gearman to zuul and get zmq pub sub on jenkinses | 22:34 |
| fungi | clarkb: good news! we'll be running "very few" jobs at that point (where "very few" tends toward 0) | 22:35 |
| fungi | i suspect the fsck will be running for a while neyond | 22:35 |
| fungi | beyond | 22:35 |
| clarkb | kk | 22:36 |
| *** ZZelle_ has joined #openstack-infra | 22:36 | |
| fungi | it recovered the journal and made it into pass 1 (checking inodes, blocks, and sizes) a few minutes ago | 22:36 |
| fungi | fsck of a mostly-full and somewhat fragmented ~14tb filesystem takes "a while" | 22:37 |
| clarkb | when all running jobs are done I am going to turn off old nodepool | 22:37 |
| clarkb | then I can start deleting the nodes it had control of (but won't clean up images yet) | 22:37 |
| fungi | sounds good | 22:37 |
| *** tsg has quit IRC | 22:37 | |
| patrickeast | my ci started to fail with “Connecting to tarballs.openstack.org (tarballs.openstack.org)|192.237.219.31|:80... failed: Connection refused.” is that related to the issues you guys are working on right now? | 22:38 |
| clarkb | patrickeast: yes | 22:38 |
| fungi | i guess this is the weekend where we break all the things. luckily i'm not scheduled to be anywhere special | 22:38 |
| patrickeast | clarkb: ok cool, ill just turn it off until this is resolved ; ) | 22:38 |
| *** amitgandhinz has quit IRC | 22:38 | |
| patrickeast | thx | 22:38 |
| fungi | patrickeast: yeah, catastrophic cinder backend failure in rackspace i;m working through now | 22:38 |
| fungi | while this runs, i'll open the provider trouble ticket | 22:39 |
| fungi | phschwartz: around? | 22:40 |
| *** sarob has quit IRC | 22:40 | |
| clarkb | all images are rebuilt on new node side \o/ | 22:40 |
| * anteaya is back | 22:41 | |
| anteaya | how can I help? | 22:41 |
| fungi | anteaya: redirect people asking about problems with the sites served from static.o.o or jobs not running/zuul pending everything to the channel topic | 22:42 |
| anteaya | very good | 22:42 |
| fungi | ooh! fsck is rewriting error blocks now!!! | 22:42 |
| fungi | "Error reading block <foo> (Attempt to read block from filesystem resulted in short read) while reading inode and block bitmaps. Ignore error? yes Force rewrite? yes" | 22:44 |
| fungi | it only scrolled by for a minute or so, therefore guessing we only lost somewhere in the neighborhood of 10k files | 22:45 |
| fungi | a pittance, ultimately | 22:45 |
| clarkb | fungi's secret ability: eyeballing lost files during fsck | 22:46 |
| * anteaya is reading scrollback and agrees 10k logs is a pittance | 22:46 | |
| *** tsg has joined #openstack-infra | 22:46 | |
| fungi | clarkb: i have muddled through filesystem recovery on servers for much more of my life than i like to admit. it's old hat now | 22:47 |
| clarkb | and this is why I want the fungi's secrets to file system recovery rundown :P | 22:47 |
| *** gondoi is now known as zz_gondoi | 22:48 | |
| *** xyang0 has quit IRC | 22:49 | |
| *** JayJ has quit IRC | 22:49 | |
| *** JayJ has joined #openstack-infra | 22:50 | |
| fungi | i'll see about incorporating it into my memoirs, right after the section on repairing out-of-warranty scsi tape library robots | 22:50 |
| anteaya | ha ha ha | 22:50 |
| *** tonytan4ever has quit IRC | 22:50 | |
| clarkb | I am going to take advantage of this too to clean up the jenkins offline nodes | 22:50 |
| fungi | clarkb: feel free. by the time everythig is running again, none of it will hold relevance | 22:51 |
| *** e0ne has joined #openstack-infra | 22:52 | |
| *** achanda has quit IRC | 22:53 | |
| *** achanda has joined #openstack-infra | 22:54 | |
| *** mestery has quit IRC | 22:55 | |
| *** JayJ has quit IRC | 22:56 | |
| *** mestery has joined #openstack-infra | 22:56 | |
| *** packet has joined #openstack-infra | 22:56 | |
| fungi | for the record, rax ticket 141121-ord-0001241 has been opened to track this issue | 22:56 |
| *** JayJ has joined #openstack-infra | 22:56 | |
| fungi | first signs of trouble in static's syslog were at 21:07:56 utc | 22:57 |
| *** emagana has joined #openstack-infra | 22:57 | |
| mordred | fungi, clarkb: wow, I'm missing all the fun | 22:57 |
| mordred | anything I can do to be helpful? | 22:57 |
| anteaya | you caught ia at about 21:39 | 22:57 |
| fungi | mordred: i have fun for you if you want any ;) | 22:57 |
| clarkb | ya fungi's things aer more urgent than mine | 22:58 |
| *** bknudson has quit IRC | 22:58 | |
| fungi | an lp (because of elastic-recheck) bug mentioning this stuff will be helpful once we get everything back online | 22:58 |
| clarkb | fungi: e-r won't be able to catch any of it | 22:59 |
| clarkb | for better or worse because it relies on logs that were never written | 22:59 |
| fungi | clarkb: fair point, no logs | 22:59 |
| clarkb | so storyboard should be fine | 22:59 |
| mordred | ok | 22:59 |
| mordred | so just a storyboard story saying that there was a massive cinder backend failure at rackspace? | 22:59 |
| *** otherwiseguy has quit IRC | 23:00 | |
| openstackgerrit | Surojit Pathak proposed openstack-dev/hacking: Fixing broken loop to import a module in hacking https://review.openstack.org/136517 | 23:00 |
| *** e0ne has quit IRC | 23:01 | |
| clarkb | mordred: static.o.o lost xvdk, unmounting everything, rebooting, then manually bringing everything back online after detaching and attaching the volume seems to be correcting things | 23:01 |
| *** signed8bit has quit IRC | 23:01 | |
| *** odyi has quit IRC | 23:01 | |
| fungi | not sure i'd publically classify it as massive | 23:01 |
| fungi | hit 1 out of 14 of our volumes | 23:02 |
| fungi | the impact on _us_ was massive | 23:02 |
| *** sarob has joined #openstack-infra | 23:02 | |
| *** emagana has quit IRC | 23:03 | |
| fungi | but for all i know it's a hardware failure in some tiny component among a fleet of arrays at rax | 23:03 |
| *** mjturek has quit IRC | 23:03 | |
| fungi | and we may have just been "lucky" | 23:03 |
| *** emagana has joined #openstack-infra | 23:03 | |
| openstackgerrit | Surojit Pathak proposed openstack-dev/hacking: Fixing broken loop to import a module in hacking In function, is_module_for_sure(), the while loop to parse through the fully qualified module name was broken. In every first attempt, it used to get an ImportError, the fallback exception handling, which w https://review.openstack.org/136517 | 23:04 |
| *** odyi has joined #openstack-infra | 23:04 | |
| anteaya | quite the title | 23:04 |
| *** emagana_ has joined #openstack-infra | 23:04 | |
| *** emagana has quit IRC | 23:04 | |
| anteaya | flashgordon: you want to do the honours ^^ | 23:04 |
| mordred | clarkb, fungi https://storyboard.openstack.org/#!/story/2000022 | 23:05 |
| flashgordon | anteaya: hacking should -1 that automatically | 23:05 |
| fungi | thanks mordred! | 23:05 |
| *** jgrimm is now known as zz_jgrimm | 23:05 | |
| fungi | fwiw i expect the other fsck phases to take a while | 23:05 |
| fungi | because, yaknow, 14tb | 23:06 |
| JayF | Is it expected that tarballs.o.o is impacted by this as well? Getting 404s where things used to exist. | 23:06 |
| clarkb | yes | 23:06 |
| jeblair | fungi, mordred, clarkb: i'm back, though not at 100%. i'm catching up now | 23:06 |
| openstackgerrit | Surojit Pathak proposed openstack-dev/hacking: Fixing broken loop to import a module in hacking. https://review.openstack.org/136517 | 23:06 |
| fungi | JayF: yep, served from static.openstack.org as an apache vhost | 23:06 |
| jeblair | mordred: i read your story. it's a sad story. :( | 23:06 |
| fungi | jeblair: i recommend taking the evening off ;) | 23:06 |
| JayF | fungi: that was my presumption but wanted to be sure. Thanks and gl. | 23:06 |
| *** jeblair is now known as corvus | 23:06 | |
| corvus | done! | 23:07 |
| *** mmaglana has joined #openstack-infra | 23:07 | |
| anteaya | corvus: rest well | 23:07 |
| mordred | corvus: I really only like making art that produces tears | 23:07 |
| anteaya | explains some of your patches | 23:07 |
| *** corvus is now known as jeblair | 23:07 | |
| mordred | I believe anteaya just won Friday | 23:07 |
| jeblair | anteaya: nice! :) | 23:07 |
| clarkb | burn | 23:07 |
| fungi | zing! | 23:08 |
| anteaya | I won a Friday | 23:08 |
| fungi | you can take your extra friday tomorrow. i insist | 23:08 |
| anteaya | ha ha ha | 23:08 |
| jeblair | fungi: do you think this is similar to the event that took the afs volume offline? | 23:09 |
| fungi | jeblair: since i don't work at rackspace supporting their disk arrays, i honestly have no idea. it's i/o errors | 23:09 |
| fungi | so, yeah, superficially similar i suppose | 23:09 |
| JayF | fungi: Have you guys filed a ticket or anything? Gotten a notice from Rackspace that it's down? | 23:10 |
| anteaya | JayF: https://storyboard.openstack.org/#!/story/2000022 | 23:10 |
| fungi | JayF: Ticket ID | 23:10 |
| fungi | 141121-ord-0001241 | 23:10 |
| JayF | I won't go look at your tickets because I wouldn't know what to do with them anyway; just wanted to make sure we had done the right thing and told you or you guys had made sure we knew | 23:10 |
| fungi | JayF: still in new status for the moment | 23:11 |
| JayF | hmm. | 23:11 |
| fungi | JayF: as you should expect, i spent my time first doing what i could to get our systems on the road to recovery, _then_ opened the ticket, so it's not terribly old yet | 23:11 |
| jeblair | i don't see a proactive email for this one | 23:12 |
| fungi | nor i | 23:12 |
| JayF | Lets say I don't wanna say I pinged someone to look at that ticket | 23:13 |
| openstackgerrit | Surojit Pathak proposed openstack-dev/hacking: Fixing broken loop to import a module in hacking. https://review.openstack.org/136517 | 23:13 |
| fungi | they totally need to recommend raid on top of their raid again though. that one never gets old | 23:13 |
| JayF | but that's definately the sort of thing I'd do ;) | 23:13 |
| fungi | JayF: your "inaction" is appreciated ;) | 23:14 |
| *** MarkAtwood has joined #openstack-infra | 23:15 | |
| clarkb | I am just about ready to stop old nodepool and begin deleting its nodes | 23:15 |
| *** barnaby has quit IRC | 23:15 | |
| clarkb | and in about 15 minutes I can do the DNS record update | 23:15 |
| anteaya | oh does new nodepool on trusty work? | 23:15 |
| clarkb | anteaya: yes I hit it with a hammer long enough | 23:15 |
| anteaya | well done | 23:15 |
| *** dims_ has quit IRC | 23:16 | |
| *** dimtruck is now known as zz_dimtruck | 23:16 | |
| clarkb | ok old nodepool service is stopped | 23:16 |
| fungi | i'll make sure to take all our gating offline for you ;) | 23:17 |
| fungi | done! | 23:17 |
| anteaya | well on the bright side we didn't have to schedule a downtime to switch over to nodepool on trusty | 23:17 |
| fungi | unscheduled downtime is so much more sexy | 23:17 |
| fungi | we can collect pity points on it | 23:18 |
| mordred | I feel like we could have fixed or avoided this by using docker in some manner | 23:18 |
| nibalizer | >.> | 23:18 |
| fungi | mordred: we could have avoided this by growing handlebar mustaches on our handlebar mustaches? | 23:18 |
| fungi | in reality, we would have avoided this by moving whole hog to swift log storage. too bad the timing was off | 23:19 |
| *** dimsum__ has joined #openstack-infra | 23:19 | |
| *** dmellado is now known as dmellado|afk | 23:19 | |
| clarkb | ok I have a for loop deleting nodes that were associated with old nodepool now | 23:20 |
| clarkb | I will leave images alone for now so we can fall back onto it | 23:20 |
| clarkb | if necessary | 23:20 |
| *** andreykurilin_ has joined #openstack-infra | 23:21 | |
| *** alexpilotti has quit IRC | 23:22 | |
| clarkb | we are using 20% of /opts disk space on new nodepool but centos7 dib image built | 23:22 |
| clarkb | this is without vpc images | 23:22 |
| fungi | that seems safe but we may want a cinder volume later | 23:22 |
| clarkb | ya | 23:22 |
| fungi | since we're only just beginning our foray into converting our images | 23:23 |
| fungi | ultimately that will probably be pushing it | 23:23 |
| clarkb | Iam just excited that that bit seems to have just worked | 23:23 |
| clarkb | overridden tmpdir and cache dir and all | 23:23 |
| * fungi too! | 23:23 | |
| fungi | sorry i didn't help more | 23:23 |
| clarkb | np | 23:23 |
| clarkb | you jumped on the big fire of the day :) | 23:23 |
| *** otherwiseguy has joined #openstack-infra | 23:23 | |
| *** dannywilson has quit IRC | 23:24 | |
| fungi | i was chilly | 23:24 |
| anteaya | yeah, you will never ever get the trophy for not-doing-enough fungi | 23:24 |
| fungi | i can dream | 23:24 |
| *** packet has quit IRC | 23:25 | |
| *** dannywilson has joined #openstack-infra | 23:25 | |
| anteaya | if volumes come back | 23:26 |
| clarkb | I should've parallelized this a bit more. oh well. no rush considering the other thing | 23:26 |
| fungi | yeah, i think you can take as much time as you need | 23:27 |
| fungi | i'm going to take a break to start a batch of sourdough bread while this continues running | 23:27 |
| anteaya | yep | 23:28 |
| *** SumitNaiksatam has quit IRC | 23:31 | |
| *** timrc is now known as timrc-afk | 23:31 | |
| *** davideagnello has joined #openstack-infra | 23:32 | |
| *** mudassirlatif_ has joined #openstack-infra | 23:32 | |
| *** patrickeast_ has joined #openstack-infra | 23:32 | |
| *** patrickeast has quit IRC | 23:32 | |
| *** patrickeast_ is now known as patrickeast | 23:32 | |
| davideagnello | hello, why is there no content in "http://tarballs.openstack.org/"? | 23:33 |
| anteaya | hi davideagnello | 23:33 |
| jeblair | maybe we should update the status to mention that and point to the story | 23:33 |
| davideagnello | is there some maintenance, should I wait? | 23:33 |
| anteaya | take a moment and read the channel topic | 23:33 |
| anteaya | jeblair: sounds good | 23:33 |
| jeblair | anteaya: to be fair, the topic does not mention tarballs :) | 23:33 |
| anteaya | jeblair: fair enough | 23:33 |
| davideagnello | ups, thanks | 23:33 |
| anteaya | davideagnello: we are dealing with a storage failure, unexpected | 23:34 |
| fungi | jeblair: in actuality, i could probably bring everything besides logs.o.o back online. the rest of the filesystems checked clean and could be mounted again for now | 23:34 |
| *** sarob has quit IRC | 23:34 | |
| jeblair | fungi: that's probably a good idea then | 23:34 |
| anteaya | davideagnello: unknown time for finishing | 23:34 |
| fungi | doing | 23:34 |
| *** mudassirlatif has quit IRC | 23:34 | |
| *** mudassirlatif_ is now known as mudassirlatif | 23:34 | |
| jeblair | fungi: is your fsck in screen? | 23:34 |
| clarkb | there is something wrong with the fedora21 dib build though | 23:35 |
| clarkb | TypeError: execve() arg 3 contains a non-string value | 23:35 |
| clarkb | I think that means its env is bot set correctly | 23:35 |
| fungi | jeblair: no, it should have been but i wasn't thinking ahead | 23:35 |
| fungi | okay, tarballs and docs-draft and whatever else is not on logs.o.o is back online now, apache restarted and all | 23:37 |
| anteaya | awesome | 23:37 |
| jeblair | fungi: :( just thinking of ways to get you off the hook and drinking again, but i'm guessing we'll want to know what fsck says when it's done | 23:37 |
| fungi | jeblair: i'm committed to sticking around anyway | 23:38 |
| anteaya | sometime in the next hour I need to go outside and load the stove, will probably take about 20 minutes | 23:39 |
| fungi | i think we'll manage | 23:40 |
| fungi | stove comes first | 23:40 |
| anteaya | okay I'll go now | 23:40 |
| * fungi findly recalls his childhood, when the cast iron wood stove was the primary heat source through the winters | 23:40 | |
| fungi | fondly | 23:40 |
| jroll | random question that just occurred to me, y'all ever consider serving tarballs.o.o over ssl? | 23:41 |
| *** ddieterly has quit IRC | 23:41 | |
| fungi | jroll: considered, yes, though so far it's not been our primary distribution channel | 23:41 |
| *** MarkAtwood has quit IRC | 23:41 | |
| *** FallenPegasus has joined #openstack-infra | 23:41 | |
| fungi | jroll: likely worth doing soon anyway, so it's on my radar | 23:41 |
| *** bhuvan has quit IRC | 23:42 | |
| fungi | jroll: biggest challenge is that it would require either a separate instance or a special cert (wildcard or aliased) | 23:42 |
| clarkb | ok I am updating nodepool.o.o DNS records now | 23:42 |
| jeblair | fungi: the fsck is using 3g of ram | 23:43 |
| jroll | fungi: I see, cool | 23:43 |
| jroll | thanks | 23:43 |
| fungi | jroll: since it shares an ip address with other vhosts right now | 23:43 |
| fungi | jeblair: ext4 is dirty business | 23:43 |
| jroll | fungi: right, and not everything supports SNI because sadface | 23:43 |
| *** dmsimard is now known as dmsimard_away | 23:43 | |
| clarkb | forward dns is done | 23:45 |
| *** bswartz has joined #openstack-infra | 23:45 | |
| clarkb | and reverse is done | 23:46 |
| jeblair | clarkb, fungi: since fungi was hogging all the fsck fun, i fixed the afs server | 23:46 |
| fungi | jeblair: yay!!! | 23:46 |
| jeblair | stop; unmount; fsck; mount; start | 23:46 |
| jeblair | was clean | 23:46 |
| clarkb | I will confirm ability to hit jenkins for zmq and zuul for gearman shortly and once that is confirmed restart nodepool | 23:46 |
| clarkb | we should consider deleting the fedora21 image once things settle | 23:46 |
| clarkb | its making ENOISE | 23:46 |
| *** achanda has quit IRC | 23:48 | |
| *** FallenPegasus has quit IRC | 23:49 | |
| *** yamahata has joined #openstack-infra | 23:50 | |
| *** armax has quit IRC | 23:51 | |
| clarkb | ok I am stopping nodepool on new nodepool, reenabling gearman, then starting it up | 23:51 |
| *** MarkAtwood has joined #openstack-infra | 23:52 | |
| fungi | k | 23:52 |
| clarkb | ok its doing things | 23:53 |
| kevinbenton | hi folks. Can someone give me a quick pointer to where the gate-rally-dsvm-neutron-neutron job is configured? | 23:53 |
| *** achanda has joined #openstack-infra | 23:53 | |
| kevinbenton | i'm going to add a new rally scenario but i'm not sure how the current one is being discovered | 23:53 |
| clarkb | note I have enabled puppet on the new node without first merging my change to add the new deps | 23:54 |
| clarkb | but it should be just fine since they are only package installs and are already there | 23:55 |
| *** dimsum__ has quit IRC | 23:55 | |
| clarkb | kevinbenton: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/rally.yaml#n212 | 23:55 |
| *** dimsum__ has joined #openstack-infra | 23:55 | |
| kevinbenton | clarkb: thanks. i take it devstack-rally-gate probably has the logic i'm looking for | 23:56 |
| kevinbenton | ? | 23:56 |
| *** dimsum__ has quit IRC | 23:56 | |
| *** julim has joined #openstack-infra | 23:56 | |
| *** dimsum__ has joined #openstack-infra | 23:57 | |
| *** MarkAtwood has quit IRC | 23:57 | |
| kevinbenton | clarkb: yeah, i think i see how it works now. thanks for the pointer | 23:57 |
| *** shashankhegde has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!