| clarkb | notmyname: any other gotchas to consider? | 00:00 |
|---|---|---|
| notmyname | who told you that? | 00:00 |
| clarkb | notmyname: krtaylor | 00:00 |
| jgriffith | clarkb: hmm.. so that *should* work no? I'm trying to figure out how to get rid of: http://paste.openstack.org/show/191718/ | 00:00 |
| notmyname | clarkb: I don't think that's right, but let me check | 00:00 |
| clarkb | jgriffith: oh thats from the pretty_tox.sh script that mtreinish wrote | 00:00 |
| *** banix has joined #openstack-infra | 00:01 | |
| clarkb | jgriffith: I think there is a flag to turn that off in the script let me check | 00:01 |
| *** _nadya_ has joined #openstack-infra | 00:01 | |
| *** sabeen has quit IRC | 00:01 | |
| anteaya | notmyname: context 19:04:42 <jeblair> #topic Priority Efforts (Swift logs) http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-03-10-19.01.log.html | 00:01 |
| *** michchap_ has quit IRC | 00:01 | |
| *** Longgeek has joined #openstack-infra | 00:03 | |
| jgriffith | clarkb: I thought that was calling testr directly? | 00:03 |
| clarkb | jhesketh: subunit-trace appears to be the source of that but I do not see where that comes from | 00:03 |
| *** otter768 has joined #openstack-infra | 00:03 | |
| clarkb | er jgriffith ^ | 00:03 |
| clarkb | jgriffith: its calling testr but piping all output through subunit-trace which does the rendering | 00:04 |
| jgriffith | clarkb: got ya... | 00:04 |
| jgriffith | so run_tests script does that | 00:04 |
| *** ujuc has joined #openstack-infra | 00:05 | |
| clarkb | jgriffith: https://git.openstack.org/cgit/openstack/cinder/tree/run_tests.sh#n153 ya it uses subunit-trace too | 00:05 |
| clarkb | but if I install python-subunit I don't get a subunit-trace | 00:05 |
| jhesketh | notmyname: https://review.openstack.org/#/c/163188/ | 00:05 |
| *** _nadya_ has quit IRC | 00:05 | |
| clarkb | lifeless: ^ any idea where subunit-trace comes from? | 00:05 |
| *** annegentle has joined #openstack-infra | 00:06 | |
| jgriffith | clarkb: haha... don't try just simply removing the wrapper :) | 00:06 |
| *** bhuvan has quit IRC | 00:06 | |
| *** Longgeek has quit IRC | 00:07 | |
| clarkb | looks like tempest lib installs subunit-trace, that is not super obvious | 00:07 |
| jgriffith | clarkb: hehe.. yeah, and then that in turn has the print output always | 00:08 |
| *** otter768 has quit IRC | 00:08 | |
| clarkb | jgriffith: https://github.com/openstack/tempest-lib/blob/master/tempest_lib/cmd/subunit_trace.py#L231-L244 those appear to be your options | 00:09 |
| jgriffith | clarkb: aha! Excellent | 00:09 |
| jgriffith | thanks | 00:09 |
| clarkb | I don't think any of them provide the don't print stderr/stdout while running | 00:09 |
| clarkb | but you can probably add taht in | 00:09 |
| clarkb | (or stop using pretty_tox.sh) | 00:09 |
| clarkb | but pretty_tox.sh seems to be a thing now :/ | 00:09 |
| jgriffith | clarkb: well, I'm still unclear on how we get to prettytox TBH | 00:10 |
| notmyname | clarkb: you can update the object expiry time | 00:10 |
| clarkb | jgriffith: tox.ini does it at https://git.openstack.org/cgit/openstack/cinder/tree/tox.ini#n16 then run_tests.sh emulates it but does not call it directly | 00:10 |
| clarkb | notmyname: cool thats even better | 00:10 |
| clarkb | notmyname: are there any gotchas you would point out? | 00:10 |
| jgriffith | clarkb: aha! | 00:11 |
| notmyname | clarkb: if you update the expiry time very close to the original delete time, you might not catch it in time | 00:11 |
| clarkb | notmyname: I think we can live with that (we were also prepared to lie with not being able to update) | 00:12 |
| notmyname | clarkb: well, let me check one thing on that | 00:12 |
| greghaynes | clarkb: not sure if you saw my comment but my deadlock issues seem to happen only if I run testtools.run instead of tix | 00:12 |
| greghaynes | er, instead of tox | 00:12 |
| jgriffith | wonder if maybe I'm chasing this needlessly... regardless, clarkb thanks for all the info | 00:12 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-jenkins: Use @varname to supress warnings https://review.openstack.org/163650 | 00:12 |
| clarkb | greghaynes: I did, not sure why that would be other than that the .testr.conf sets an env up that may be different than what you get via testtools directly | 00:13 |
| greghaynes | hrmm | 00:13 |
| notmyname | clarkb: oh, nevermind. that corner case is covered in the swift code. not edge cases there | 00:13 |
| clarkb | greghaynes: specifically log and stdout/stderr capturing | 00:13 |
| notmyname | clarkb: ie if you change the exipry time one second before it was set to expire, it won't expire | 00:13 |
| greghaynes | clarkb: hrmmm | 00:13 |
| clarkb | notmyname: nice | 00:14 |
| SpamapS | hrm, there seems to be something deep within pyOpenSSL that doesn't work with the gear py3 tests | 00:14 |
| clarkb | greghaynes: try running it with those env vars set? see .testr.conf for the full list | 00:14 |
| clarkb | greghaynes: curious if that makes a difference | 00:14 |
| *** fdegir has quit IRC | 00:14 | |
| *** hdd_ has joined #openstack-infra | 00:14 | |
| anteaya | notmyname: thanks, did you want to leave a review on 163188? | 00:15 |
| *** boris-42 has joined #openstack-infra | 00:15 | |
| jlvillal | clarkb: What is the correct procedure to ask for an infra patch to be reviewed that is blocking the openstack/python-ironicclient gate? This is the patch: https://review.openstack.org/163600 | 00:15 |
| notmyname | anteaya: hang on! I'm still looking at stuff! ;-) | 00:15 |
| anteaya | notmyname: sorry don't mean to rush | 00:15 |
| *** xgerman has quit IRC | 00:16 | |
| notmyname | clarkb: so, no, I don't think there are any corner cases to worry about. after the object expires, swift will return a 404 for it and a background process will eventually reap it | 00:16 |
| clarkb | notmyname: that is awesome | 00:16 |
| jgriffith | clarkb: think you meant me... and yep | 00:16 |
| jgriffith | clarkb: setting those to False/0 eliminates the capture | 00:17 |
| jgriffith | clarkb: thanks a bunch for the help | 00:17 |
| clarkb | jgriffith: np | 00:17 |
| jgriffith | clarkb: think I'll add an option with the default to eat them | 00:17 |
| clarkb | greghaynes: I just ran `.tox/py27/bin/python -m testtools.run nodepool.tests.test_nodepool` without trouble on your change | 00:18 |
| greghaynes | :/ | 00:18 |
| clarkb | is that how you invoke testtools.run? | 00:18 |
| clarkb | maybe i should try the specific test metho? | 00:18 |
| greghaynes | I was singleshotting the method | 00:18 |
| clarkb | jlvillal: requesting here is fine I will review it | 00:18 |
| jlvillal | clarkb: Thank you :) | 00:19 |
| greghaynes | sounds like it might just be that I have some derp in my test environment | 00:19 |
| greghaynes | which is very plausible | 00:19 |
| *** Longgeek has joined #openstack-infra | 00:19 | |
| clarkb | ya `.tox/py27/bin/python -m testtools.run nodepool.tests.test_nodepool.TestNodepool.test_dib_upload_fail` works too | 00:20 |
| clarkb | jlvillal: maybe we should stop using httpretty? | 00:20 |
| clarkb | jlvillal: aiui its a constant source of problems | 00:21 |
| *** bdpayne has quit IRC | 00:21 | |
| *** melwitt has quit IRC | 00:21 | |
| jlvillal | clarkb: Uh.... That isn't really my call. Probably would have to be something for the ironic-core team and any other projects which use it to decide. | 00:21 |
| *** mwagner_lap has joined #openstack-infra | 00:21 | |
| clarkb | jlvillal: sure | 00:21 |
| * ianw happy i think i came to the same conclusion as lifeless , just half an hour later | 00:22 | |
| notmyname | clarkb: anteaya: +1 in gerrit | 00:22 |
| anteaya | notmyname: thank you! | 00:22 |
| anteaya | and thanks for checking the update object options | 00:22 |
| *** ddieterly has joined #openstack-infra | 00:23 | |
| anteaya | jhesketh: since I'm looking at the swift logging things, did you want to address clarkb's comment? https://review.openstack.org/#/c/163276/3 | 00:24 |
| *** ddieterl_ has joined #openstack-infra | 00:24 | |
| *** ddieterly has quit IRC | 00:24 | |
| *** Longgeek has quit IRC | 00:26 | |
| jhesketh | clarkb, anteaya: Replied to the comment on https://review.openstack.org/#/c/163276/ | 00:26 |
| *** banix has quit IRC | 00:26 | |
| *** patrickeast has joined #openstack-infra | 00:27 | |
| anteaya | jhesketh: thanks | 00:27 |
| clarkb | jhesketh: oh in that case I can flip m vote | 00:28 |
| anteaya | mtreinish: you still around to look at this patch which adds a new non-voting job to tempest that just tests the database tempest tests with trove: https://review.openstack.org/#/c/163521/1 | 00:28 |
| jhesketh | if you're happy with it | 00:28 |
| clarkb | yup | 00:28 |
| *** patrickeast has quit IRC | 00:29 | |
| mtreinish | anteaya: sure I'll take a look now | 00:29 |
| anteaya | any reason 163276 can't merge? | 00:29 |
| anteaya | mtreinish: thanks | 00:29 |
| mtreinish | jgriffith: did you figure out your issue with subunit-trace? | 00:29 |
| anteaya | clarkb: I'm mostly concerned with timing on the image builds | 00:30 |
| *** mpopow__ has joined #openstack-infra | 00:30 | |
| *** gyee has quit IRC | 00:31 | |
| openstackgerrit | Merged openstack-infra/project-config: Add option to set swift delete after header https://review.openstack.org/163188 | 00:31 |
| *** ashleighfarnham has quit IRC | 00:31 | |
| *** annegentle has quit IRC | 00:31 | |
| mtreinish | jgriffith: looking at the backscroll if you use --no-failure-debug that will stop printing stdout during the run | 00:32 |
| *** patrickeast has joined #openstack-infra | 00:32 | |
| mtreinish | jgriffith: just make sure you have -f flag set too otherwise you won't see any tracebacks on failure | 00:32 |
| SpamapS | dhellmann: around? | 00:33 |
| SpamapS | dhellmann: https://github.com/python/cpython/blob/master/Lib/idlelib/PyShell.py#L1342 <-- that looks like a bug | 00:33 |
| SpamapS | dhellmann: shouldn't that be bytes, not str, for python3? | 00:34 |
| *** mpopow_ has quit IRC | 00:34 | |
| mtreinish | anteaya: +1 on both commits in that series | 00:35 |
| *** ghostpl_ has joined #openstack-infra | 00:35 | |
| SpamapS | ahh n/m | 00:35 |
| SpamapS | dhellmann: ignore me, had it backwards | 00:35 |
| * SpamapS is still confused why files want strings, and not bytes.. gah | 00:35 | |
| *** dimtruck is now known as zz_dimtruck | 00:36 | |
| *** esker has joined #openstack-infra | 00:36 | |
| *** annegentle has joined #openstack-infra | 00:38 | |
| clarkb | anteaya: they will build in the morning so first thing to do is check on it tomorrow | 00:38 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/gear: Resolve python3 test failures https://review.openstack.org/158147 | 00:38 |
| clarkb | anteaya: shouldn't be a major problem | 00:38 |
| anteaya | mtreinish: thank you | 00:39 |
| anteaya | clarkb: k thanks | 00:40 |
| anteaya | clarkb: I'm thinnking after these go in we should wait for the python-tests one | 00:40 |
| anteaya | to see and make sure these work as expected | 00:40 |
| anteaya | what do you think? | 00:40 |
| *** woodster_ has joined #openstack-infra | 00:41 | |
| *** vishy has quit IRC | 00:41 | |
| clarkb | anteaya: yup | 00:41 |
| *** vishy has joined #openstack-infra | 00:42 | |
| *** ghostpl_ has quit IRC | 00:42 | |
| anteaya | k | 00:43 |
| *** dannywilson has joined #openstack-infra | 00:44 | |
| *** kiseok7_ has quit IRC | 00:46 | |
| *** mtanino has quit IRC | 00:47 | |
| *** tsg_ has quit IRC | 00:47 | |
| *** annegentle has quit IRC | 00:48 | |
| *** annegentle has joined #openstack-infra | 00:49 | |
| *** zz_dimtruck is now known as dimtruck | 00:50 | |
| *** kiseok7_ has joined #openstack-infra | 00:50 | |
| openstackgerrit | Merged openstack-infra/project-config: zuul-swift-upload use logging for debug msgs https://review.openstack.org/163276 | 00:51 |
| *** bhuvan has joined #openstack-infra | 00:51 | |
| *** tiswanso has joined #openstack-infra | 00:51 | |
| *** esker has quit IRC | 00:52 | |
| *** esker has joined #openstack-infra | 00:52 | |
| *** tiswanso has quit IRC | 00:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting trove tempest job https://review.openstack.org/163521 | 00:54 |
| *** tiswanso has joined #openstack-infra | 00:55 | |
| *** chlong has quit IRC | 00:55 | |
| *** chlong has joined #openstack-infra | 00:55 | |
| *** rfolco_ has quit IRC | 00:59 | |
| *** marun has joined #openstack-infra | 01:00 | |
| *** dimtruck is now known as zz_dimtruck | 01:01 | |
| openstackgerrit | Brian Curtin proposed openstack/requirements: Add httpretty 0.8.8 https://review.openstack.org/163662 | 01:03 |
| *** rwsu is now known as rwsu-afk | 01:04 | |
| *** marun has quit IRC | 01:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Add skip if for tempest running tempest on tempest commits https://review.openstack.org/162993 | 01:05 |
| *** annegentle has quit IRC | 01:08 | |
| clarkb | about to sign off for the day but I notice that the status: ERROR messages from hpcloud seem to have stopped | 01:12 |
| *** SumitNaiksatam has quit IRC | 01:13 | |
| clarkb | so we likely need to figure out why/how ports leak | 01:13 |
| lifeless | clarkb: subunit-trace is in tempestlib | 01:13 |
| clarkb | lifeless: ya I finally tracked that down | 01:14 |
| lifeless | I wrotes it for tempest | 01:14 |
| lifeless | its being reused now | 01:14 |
| lifeless | ianw: my conclusion about? | 01:14 |
| *** gokrokve has quit IRC | 01:15 | |
| anteaya | clarkb: happy evening and yay cloud | 01:15 |
| clarkb | lifeless: the testtools 1.7.0 thing | 01:15 |
| *** AJaeger has quit IRC | 01:18 | |
| lifeless | oh right, I see the mail. | 01:18 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 01:19 | |
| clarkb | mordred: did ansible fix ever get pushed? | 01:20 |
| clarkb | mordred: we should try to get that in soon (tomorrow at this point) so that puppet works | 01:20 |
| *** mtanino has joined #openstack-infra | 01:21 | |
| openstackgerrit | Merged openstack/requirements: Avoid httpretty 0.8.8 as it can break unittests https://review.openstack.org/163600 | 01:22 |
| *** unicell has quit IRC | 01:25 | |
| *** markvoelker has quit IRC | 01:27 | |
| *** ivar-laz_ has joined #openstack-infra | 01:27 | |
| *** markvoelker has joined #openstack-infra | 01:27 | |
| *** ivar-lazzaro has quit IRC | 01:30 | |
| *** AJaeger has joined #openstack-infra | 01:30 | |
| *** ivar-laz_ has quit IRC | 01:32 | |
| *** markvoelker has quit IRC | 01:32 | |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Add 'shared_lock' package https://review.openstack.org/163666 | 01:32 |
| *** hdd_ has quit IRC | 01:34 | |
| *** ayoung has joined #openstack-infra | 01:35 | |
| *** asettle-mtg is now known as asettle | 01:37 | |
| *** macjack has joined #openstack-infra | 01:37 | |
| *** Alexandra_ has joined #openstack-infra | 01:38 | |
| *** sdake_ has quit IRC | 01:42 | |
| *** Longgeek has joined #openstack-infra | 01:52 | |
| *** dims_ has quit IRC | 01:53 | |
| *** Longgeek has quit IRC | 01:56 | |
| *** salv-orlando has quit IRC | 01:58 | |
| *** yamahata has quit IRC | 01:59 | |
| *** armax has quit IRC | 02:01 | |
| *** macjack has quit IRC | 02:03 | |
| *** otter768 has joined #openstack-infra | 02:04 | |
| *** mpopow__ has quit IRC | 02:05 | |
| *** tqtran_ has quit IRC | 02:06 | |
| *** shardy has quit IRC | 02:07 | |
| *** shardy has joined #openstack-infra | 02:07 | |
| *** armax has joined #openstack-infra | 02:08 | |
| *** otter768 has quit IRC | 02:08 | |
| *** nelsnelson has joined #openstack-infra | 02:10 | |
| *** banix has joined #openstack-infra | 02:11 | |
| *** hdd_ has joined #openstack-infra | 02:14 | |
| *** patrickeast has quit IRC | 02:18 | |
| *** yfried|afk is now known as yfried|pto | 02:20 | |
| *** Qiming has joined #openstack-infra | 02:21 | |
| *** ivar-lazzaro has joined #openstack-infra | 02:23 | |
| *** jyuso1 has joined #openstack-infra | 02:25 | |
| *** ghostpl_ has joined #openstack-infra | 02:26 | |
| *** yfried|pto is now known as yfried|afk | 02:30 | |
| ianw | lifeless: version pins for unittest2 | 02:33 |
| *** ghostpl_ has quit IRC | 02:36 | |
| jeblair | clarkb, mordred, fungi: it looks like over the post few hours we've been able to create many more hpcloud nodes | 02:42 |
| *** achanda has quit IRC | 02:42 | |
| jeblair | clarkb, mordred, fungi: so i think the port/ip leakages were probably the bulk of the problem | 02:42 |
| *** nelsnelson has quit IRC | 02:42 | |
| *** Ryan_Lane has quit IRC | 02:43 | |
| *** zz_dimtruck is now known as dimtruck | 02:43 | |
| *** ivar-lazzaro has quit IRC | 02:43 | |
| *** david-lyle is now known as david-lyle_afk | 02:45 | |
| *** sdake_ has joined #openstack-infra | 02:48 | |
| *** yfried|afk is now known as yfried|pto | 02:50 | |
| *** oomichi has joined #openstack-infra | 02:52 | |
| *** unicell has joined #openstack-infra | 02:53 | |
| *** marun has joined #openstack-infra | 02:53 | |
| *** dims_ has joined #openstack-infra | 02:54 | |
| *** gokrokve has joined #openstack-infra | 02:54 | |
| *** spzala has quit IRC | 02:55 | |
| *** salv-orlando has joined #openstack-infra | 02:59 | |
| *** dims_ has quit IRC | 02:59 | |
| *** yfried|pto is now known as yfried|afk | 03:00 | |
| *** zz_jgrimm is now known as jgrimm | 03:03 | |
| openstackgerrit | Huang Rui proposed openstack-infra/project-config: Create nova-zvm-virt-driver project on StackForge https://review.openstack.org/163687 | 03:07 |
| *** ddieterl_ has quit IRC | 03:09 | |
| *** dboik_ has quit IRC | 03:10 | |
| *** dkehn has quit IRC | 03:10 | |
| *** salv-orlando has quit IRC | 03:10 | |
| *** Qiming_ has joined #openstack-infra | 03:11 | |
| *** yfried|afk is now known as yfried|pto | 03:12 | |
| *** Qiming__ has joined #openstack-infra | 03:12 | |
| *** Qiming has quit IRC | 03:14 | |
| *** Ryan_Lane has joined #openstack-infra | 03:14 | |
| *** dkehn has joined #openstack-infra | 03:15 | |
| *** Qiming_ has quit IRC | 03:16 | |
| *** koolhead17 has joined #openstack-infra | 03:17 | |
| *** salv-orlando has joined #openstack-infra | 03:19 | |
| *** nelsnelson has joined #openstack-infra | 03:19 | |
| *** ivar-lazzaro has joined #openstack-infra | 03:19 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:22 | |
| *** radez_g0n3 is now known as radez | 03:22 | |
| *** salv-orlando has quit IRC | 03:23 | |
| *** sputnik13 has quit IRC | 03:24 | |
| *** mtanino has quit IRC | 03:24 | |
| *** salv-orlando has joined #openstack-infra | 03:24 | |
| *** resker has joined #openstack-infra | 03:27 | |
| *** salv-orlando has quit IRC | 03:29 | |
| *** esker has quit IRC | 03:31 | |
| *** dannywilson has quit IRC | 03:31 | |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Modifying devstack plugin job template https://review.openstack.org/163323 | 03:33 |
| *** Longgeek has joined #openstack-infra | 03:37 | |
| *** yfried|pto is now known as yfried|afk | 03:38 | |
| *** sdake_ has quit IRC | 03:38 | |
| *** Ryan_Lane has quit IRC | 03:40 | |
| *** yfried|afk is now known as yfried|pto | 03:41 | |
| *** banix has quit IRC | 03:41 | |
| *** hdd_ has quit IRC | 03:42 | |
| *** claudiub has quit IRC | 03:46 | |
| *** nelsnelson has quit IRC | 03:51 | |
| *** yfried|pto is now known as yfried|afk | 03:55 | |
| *** yfried|afk is now known as yfried|pto | 03:56 | |
| *** Ryan_Lane has joined #openstack-infra | 04:00 | |
| *** funzo has joined #openstack-infra | 04:01 | |
| *** Longgeek has quit IRC | 04:03 | |
| *** otter768 has joined #openstack-infra | 04:05 | |
| *** funzo has quit IRC | 04:06 | |
| *** sdake_ has joined #openstack-infra | 04:08 | |
| *** garyh has quit IRC | 04:09 | |
| *** ddieterly has joined #openstack-infra | 04:10 | |
| *** otter768 has quit IRC | 04:10 | |
| *** oomichi has quit IRC | 04:13 | |
| *** ddieterly has quit IRC | 04:14 | |
| *** Qiming_ has joined #openstack-infra | 04:16 | |
| *** gokrokve_ has joined #openstack-infra | 04:18 | |
| *** Qiming__ has quit IRC | 04:19 | |
| *** gokrokve has quit IRC | 04:21 | |
| *** radez is now known as radez_g0n3 | 04:21 | |
| *** ghostpl_ has joined #openstack-infra | 04:21 | |
| *** amotoki has joined #openstack-infra | 04:22 | |
| *** gokrokve_ has quit IRC | 04:22 | |
| *** devlaps has quit IRC | 04:25 | |
| *** ghostpl_ has quit IRC | 04:31 | |
| *** achanda has joined #openstack-infra | 04:31 | |
| *** ivar-lazzaro has quit IRC | 04:33 | |
| *** ivar-lazzaro has joined #openstack-infra | 04:34 | |
| *** macjack has joined #openstack-infra | 04:37 | |
| *** ivar-lazzaro has quit IRC | 04:39 | |
| *** ddieterly has joined #openstack-infra | 04:40 | |
| *** erlon has quit IRC | 04:41 | |
| *** ddieterly has quit IRC | 04:44 | |
| *** unicell has quit IRC | 04:44 | |
| *** unicell has joined #openstack-infra | 04:45 | |
| *** marun has quit IRC | 04:47 | |
| *** pblaho has quit IRC | 04:47 | |
| *** salv-orlando has joined #openstack-infra | 04:49 | |
| *** marun has joined #openstack-infra | 04:50 | |
| *** yfried|pto is now known as yfried|afk | 04:50 | |
| *** nelsnelson has joined #openstack-infra | 04:53 | |
| *** sdake__ has joined #openstack-infra | 04:55 | |
| *** Qiming_ is now known as Qiming | 04:56 | |
| *** Alexandra_ has joined #openstack-infra | 04:57 | |
| *** gokrokve has joined #openstack-infra | 04:58 | |
| *** sdake_ has quit IRC | 04:58 | |
| *** salv-orlando has quit IRC | 05:00 | |
| *** hdd_ has joined #openstack-infra | 05:02 | |
| *** teran_ has joined #openstack-infra | 05:02 | |
| *** Qiming has quit IRC | 05:03 | |
| *** dimtruck is now known as zz_dimtruck | 05:04 | |
| *** yamahata has joined #openstack-infra | 05:04 | |
| *** gokrokve_ has joined #openstack-infra | 05:04 | |
| *** achanda has quit IRC | 05:04 | |
| *** david-ly_ has joined #openstack-infra | 05:04 | |
| *** omnibus7 has joined #openstack-infra | 05:05 | |
| *** macjack1 has joined #openstack-infra | 05:05 | |
| *** vryzhenkin has joined #openstack-infra | 05:05 | |
| *** dvorak has quit IRC | 05:05 | |
| *** kevinbenton_ has joined #openstack-infra | 05:05 | |
| *** jd__` has joined #openstack-infra | 05:05 | |
| *** spiffxp- has joined #openstack-infra | 05:05 | |
| *** _qb has quit IRC | 05:06 | |
| *** devanand1 has joined #openstack-infra | 05:07 | |
| *** GheRiver1 has joined #openstack-infra | 05:07 | |
| *** mestery_ has joined #openstack-infra | 05:07 | |
| *** SumitNaiksatam has quit IRC | 05:07 | |
| *** ubuntu2 has joined #openstack-infra | 05:08 | |
| *** sdake__ has quit IRC | 05:08 | |
| *** mestery has quit IRC | 05:08 | |
| *** camunoz_ has joined #openstack-infra | 05:08 | |
| *** SlickN1k has joined #openstack-infra | 05:09 | |
| *** terryw has joined #openstack-infra | 05:09 | |
| *** jyuso has quit IRC | 05:09 | |
| *** freerunner has quit IRC | 05:09 | |
| *** david-lyle_afk has quit IRC | 05:09 | |
| *** adreznec has quit IRC | 05:09 | |
| *** SlickNik has quit IRC | 05:09 | |
| *** jyuso has joined #openstack-infra | 05:09 | |
| *** camunoz has quit IRC | 05:09 | |
| *** otherwiseguy has quit IRC | 05:09 | |
| *** sreshetnyak has quit IRC | 05:09 | |
| *** cebruns has quit IRC | 05:09 | |
| *** sreshetnyak has joined #openstack-infra | 05:09 | |
| *** mikal_ has joined #openstack-infra | 05:09 | |
| *** devananda has quit IRC | 05:09 | |
| *** gokrokve has quit IRC | 05:09 | |
| *** macjack has quit IRC | 05:09 | |
| *** jd__ has quit IRC | 05:09 | |
| *** davidlenwell has quit IRC | 05:09 | |
| *** kevinbenton has quit IRC | 05:09 | |
| *** teran has quit IRC | 05:09 | |
| *** spiffxp_ has quit IRC | 05:09 | |
| *** aarefiev has quit IRC | 05:09 | |
| *** gmann has quit IRC | 05:09 | |
| *** GheRivero has quit IRC | 05:09 | |
| *** frickler has quit IRC | 05:09 | |
| *** ubuntu1 has quit IRC | 05:09 | |
| *** mikal has quit IRC | 05:09 | |
| *** frickler has joined #openstack-infra | 05:09 | |
| *** raginbajin has quit IRC | 05:09 | |
| *** SlickN1k is now known as SlickNik | 05:09 | |
| *** jd__` is now known as jd__ | 05:09 | |
| *** cebruns has joined #openstack-infra | 05:09 | |
| *** adreznec has joined #openstack-infra | 05:10 | |
| *** gmann has joined #openstack-infra | 05:10 | |
| *** aarefiev has joined #openstack-infra | 05:10 | |
| *** yfried|afk is now known as yfried|pto | 05:11 | |
| *** raginbajin has joined #openstack-infra | 05:13 | |
| *** dvorak has joined #openstack-infra | 05:14 | |
| openstackgerrit | Huang Rui proposed openstack-infra/project-config: Create nova-zvm-virt-driver project on StackForge https://review.openstack.org/163687 | 05:19 |
| *** marun has quit IRC | 05:19 | |
| *** yfried|pto has quit IRC | 05:20 | |
| *** marun has joined #openstack-infra | 05:22 | |
| *** markvoelker has joined #openstack-infra | 05:25 | |
| *** achanda has joined #openstack-infra | 05:28 | |
| *** markvoelker has quit IRC | 05:30 | |
| *** reed has quit IRC | 05:30 | |
| *** WendellM_ has quit IRC | 05:32 | |
| *** BharatK has joined #openstack-infra | 05:37 | |
| *** enikanorov has joined #openstack-infra | 05:40 | |
| *** woodster_ has quit IRC | 05:40 | |
| *** harlowja_ is now known as harlowja_away | 05:41 | |
| *** ddieterly has joined #openstack-infra | 05:41 | |
| *** enikanorov__ has quit IRC | 05:41 | |
| *** marun has quit IRC | 05:42 | |
| *** sputnik13 has joined #openstack-infra | 05:42 | |
| *** esker has joined #openstack-infra | 05:44 | |
| *** deepakcs has joined #openstack-infra | 05:44 | |
| *** nelsnelson has quit IRC | 05:44 | |
| *** Longgeek has joined #openstack-infra | 05:45 | |
| *** ddieterly has quit IRC | 05:45 | |
| *** resker has quit IRC | 05:46 | |
| *** macjack1 has quit IRC | 05:46 | |
| *** yamamoto has joined #openstack-infra | 05:47 | |
| *** woodster_ has joined #openstack-infra | 05:49 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Add a python-ironicclient src job https://review.openstack.org/163632 | 05:50 |
| *** wuhg has joined #openstack-infra | 05:56 | |
| *** Ryan_Lane has quit IRC | 05:56 | |
| *** hdd_ has quit IRC | 06:00 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/163703 | 06:01 |
| *** mrmartin has joined #openstack-infra | 06:01 | |
| *** sdake_ has joined #openstack-infra | 06:02 | |
| *** sdake__ has joined #openstack-infra | 06:04 | |
| *** otter768 has joined #openstack-infra | 06:06 | |
| *** sdake_ has quit IRC | 06:08 | |
| *** sdake_ has joined #openstack-infra | 06:10 | |
| *** bknudson has quit IRC | 06:10 | |
| *** otter768 has quit IRC | 06:11 | |
| *** sdake__ has quit IRC | 06:13 | |
| *** sdake__ has joined #openstack-infra | 06:14 | |
| *** Longgeek has quit IRC | 06:18 | |
| *** sdake_ has quit IRC | 06:18 | |
| *** gokrokve has joined #openstack-infra | 06:20 | |
| *** gokrokve_ has quit IRC | 06:23 | |
| *** afazekas is now known as __afazekas | 06:25 | |
| *** sputnik13 has quit IRC | 06:26 | |
| *** markvoelker has joined #openstack-infra | 06:26 | |
| *** macjack has joined #openstack-infra | 06:26 | |
| *** yamahata has quit IRC | 06:26 | |
| *** sputnik13 has joined #openstack-infra | 06:28 | |
| *** markvoelker has quit IRC | 06:32 | |
| *** dims_ has joined #openstack-infra | 06:32 | |
| *** _nadya_ has joined #openstack-infra | 06:35 | |
| *** _nadya_ has quit IRC | 06:35 | |
| *** sdake_ has joined #openstack-infra | 06:37 | |
| *** krisgan has quit IRC | 06:37 | |
| *** dims_ has quit IRC | 06:38 | |
| *** Alexandra_ has quit IRC | 06:38 | |
| *** sabari has quit IRC | 06:40 | |
| *** sdake__ has quit IRC | 06:41 | |
| *** ddieterly has joined #openstack-infra | 06:42 | |
| *** ddieterly has quit IRC | 06:46 | |
| *** funzo has joined #openstack-infra | 06:47 | |
| *** sabari has joined #openstack-infra | 06:49 | |
| *** yfried|pto has joined #openstack-infra | 06:51 | |
| *** funzo has quit IRC | 06:51 | |
| *** camunoz_ has quit IRC | 06:53 | |
| *** yfried|pto has quit IRC | 06:55 | |
| *** yfried|pto has joined #openstack-infra | 06:56 | |
| *** yfried|pto is now known as yfried|afk | 06:57 | |
| *** sdake__ has joined #openstack-infra | 06:58 | |
| *** yfried|afk is now known as yfried|pto | 07:00 | |
| *** yfried|pto is now known as yfried | 07:00 | |
| *** sdake_ has quit IRC | 07:02 | |
| *** mrunge has joined #openstack-infra | 07:02 | |
| *** sabari has quit IRC | 07:04 | |
| *** armax has quit IRC | 07:06 | |
| *** sdake__ has quit IRC | 07:08 | |
| *** sdake has quit IRC | 07:08 | |
| *** achanda has quit IRC | 07:11 | |
| *** sabari has joined #openstack-infra | 07:13 | |
| *** sdake has joined #openstack-infra | 07:19 | |
| *** garyk has joined #openstack-infra | 07:19 | |
| *** stevemar has quit IRC | 07:22 | |
| *** fdegir has joined #openstack-infra | 07:22 | |
| *** achanda has joined #openstack-infra | 07:24 | |
| *** HeOS__ is now known as HeOS_ | 07:25 | |
| *** jcoufal has joined #openstack-infra | 07:27 | |
| *** salv-orlando has joined #openstack-infra | 07:27 | |
| *** markvoelker has joined #openstack-infra | 07:28 | |
| *** markvoelker has quit IRC | 07:33 | |
| *** sabari has quit IRC | 07:33 | |
| *** yfried is now known as yfried|afk | 07:34 | |
| *** yfried|afk is now known as yfried | 07:35 | |
| *** salv-orlando has quit IRC | 07:35 | |
| *** sputnik13 has quit IRC | 07:37 | |
| *** salv-orlando has joined #openstack-infra | 07:39 | |
| *** sputnik13 has joined #openstack-infra | 07:39 | |
| *** garyk has quit IRC | 07:41 | |
| *** garyk has joined #openstack-infra | 07:41 | |
| *** tnovacik has joined #openstack-infra | 07:41 | |
| *** salv-orl_ has joined #openstack-infra | 07:42 | |
| *** ujuc has quit IRC | 07:42 | |
| *** ddieterly has joined #openstack-infra | 07:42 | |
| *** sdake_ has joined #openstack-infra | 07:42 | |
| *** salv-orlando has quit IRC | 07:43 | |
| *** achanda has quit IRC | 07:44 | |
| *** salv-orl_ has quit IRC | 07:46 | |
| *** ddieterly has quit IRC | 07:47 | |
| *** terryw has quit IRC | 07:48 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/devstack-gate: Provide option to tests Nova against v2 API https://review.openstack.org/163725 | 07:49 |
| *** sdake_ has quit IRC | 07:49 | |
| *** sdake has quit IRC | 07:49 | |
| *** stevelle has left #openstack-infra | 07:50 | |
| *** sabari has joined #openstack-infra | 07:51 | |
| *** otherwiseguy has joined #openstack-infra | 07:53 | |
| *** HeOS has quit IRC | 07:53 | |
| *** jlanoux has joined #openstack-infra | 07:54 | |
| *** sdake has joined #openstack-infra | 07:56 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-jenkins: Add configuration for HipChat in jjb https://review.openstack.org/162152 | 07:58 |
| *** scheuran has joined #openstack-infra | 07:59 | |
| *** woodster_ has quit IRC | 08:00 | |
| *** salv-orlando has joined #openstack-infra | 08:01 | |
| *** Longgeek has joined #openstack-infra | 08:03 | |
| *** jistr has joined #openstack-infra | 08:03 | |
| *** sputnik13 has quit IRC | 08:03 | |
| *** ghostpl_ has joined #openstack-infra | 08:04 | |
| *** markus_z has joined #openstack-infra | 08:04 | |
| *** salv-orlando has quit IRC | 08:06 | |
| *** afazekas_ has joined #openstack-infra | 08:06 | |
| *** sputnik13 has joined #openstack-infra | 08:07 | |
| *** otter768 has joined #openstack-infra | 08:07 | |
| *** ghostpl_ has quit IRC | 08:10 | |
| *** bhuvan has quit IRC | 08:10 | |
| *** otter768 has quit IRC | 08:11 | |
| *** Longgeek has quit IRC | 08:12 | |
| *** gokrokve has quit IRC | 08:14 | |
| *** gokrokve has joined #openstack-infra | 08:14 | |
| *** GheRiver1 is now known as GheRivero | 08:15 | |
| *** _shaps_ has joined #openstack-infra | 08:16 | |
| *** gokrokve has quit IRC | 08:19 | |
| *** yfried is now known as yfried|afk | 08:20 | |
| *** hashar has joined #openstack-infra | 08:22 | |
| *** koolhead17 has quit IRC | 08:24 | |
| *** garyk has quit IRC | 08:25 | |
| *** garyk has joined #openstack-infra | 08:25 | |
| *** sdake_ has joined #openstack-infra | 08:25 | |
| *** mrda is now known as mrda-away | 08:27 | |
| *** markvoelker has joined #openstack-infra | 08:29 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config: Remove Nova v21 neutron job & make one for v2.0 https://review.openstack.org/163731 | 08:29 |
| *** markvoelker has quit IRC | 08:34 | |
| *** salv-orlando has joined #openstack-infra | 08:34 | |
| *** doug-fish has quit IRC | 08:36 | |
| *** arxcruz has joined #openstack-infra | 08:39 | |
| *** mpavone has joined #openstack-infra | 08:39 | |
| *** yfried|afk is now known as yfried | 08:40 | |
| *** sputnik13 has quit IRC | 08:41 | |
| *** ddieterly has joined #openstack-infra | 08:43 | |
| *** mpaolino has joined #openstack-infra | 08:44 | |
| *** gokrokve has joined #openstack-infra | 08:45 | |
| openstackgerrit | Mike Heald proposed openstack-infra/project-config: Project for puppet-bandersnatch https://review.openstack.org/163402 | 08:45 |
| *** ddieterly has quit IRC | 08:48 | |
| *** gokrokve has quit IRC | 08:50 | |
| *** yfried is now known as yfried|afk | 08:50 | |
| *** Ala has joined #openstack-infra | 08:51 | |
| *** spredzy|afk is now known as spredzy | 08:52 | |
| *** sdake__ has joined #openstack-infra | 08:54 | |
| *** yfried|afk is now known as yfried | 08:54 | |
| *** skolekonov has joined #openstack-infra | 08:55 | |
| *** sdake has quit IRC | 08:57 | |
| GheRivero | morning all | 09:00 |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config: Add networking-bagpipe-l2 project https://review.openstack.org/161290 | 09:01 |
| *** ghostpl_ has joined #openstack-infra | 09:06 | |
| *** HeOS has joined #openstack-infra | 09:07 | |
| yfried | anyone knows about personalize gerrit boards? | 09:08 |
| *** sputnik13 has joined #openstack-infra | 09:13 | |
| *** _nadya_ has joined #openstack-infra | 09:14 | |
| *** gokrokve has joined #openstack-infra | 09:14 | |
| *** ghostpl_ has quit IRC | 09:14 | |
| *** gokrokve has quit IRC | 09:15 | |
| *** gokrokve has joined #openstack-infra | 09:16 | |
| *** che-arne has joined #openstack-infra | 09:17 | |
| *** ominakov_ has joined #openstack-infra | 09:19 | |
| *** gokrokve has quit IRC | 09:20 | |
| openstackgerrit | Max Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper https://review.openstack.org/156681 | 09:21 |
| *** yamamoto has quit IRC | 09:21 | |
| *** derekh has joined #openstack-infra | 09:23 | |
| *** teran_ has quit IRC | 09:24 | |
| openstackgerrit | Matthias Runge proposed openstack/requirements: Raise cap for Django https://review.openstack.org/155353 | 09:25 |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Split auth in api and change user_tokens https://review.openstack.org/153591 | 09:29 |
| *** markvoelker has joined #openstack-infra | 09:30 | |
| *** dizquierdo has joined #openstack-infra | 09:31 | |
| *** ihrachyshka has joined #openstack-infra | 09:32 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Added story types https://review.openstack.org/163412 | 09:33 |
| *** markvoelker has quit IRC | 09:35 | |
| *** asselin has quit IRC | 09:37 | |
| *** asselin has joined #openstack-infra | 09:38 | |
| *** _shaps_ has quit IRC | 09:38 | |
| openstackgerrit | Max Rasskazov proposed openstack-infra/jenkins-job-builder: Add support of multiple credentials for ssh-agent-credentials wrapper https://review.openstack.org/156681 | 09:38 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Set up ssh_known_host based on hostname https://review.openstack.org/136596 | 09:39 |
| *** sarob has quit IRC | 09:40 | |
| *** sarob has joined #openstack-infra | 09:42 | |
| *** ddieterly has joined #openstack-infra | 09:44 | |
| *** Qiming has joined #openstack-infra | 09:46 | |
| *** sarob has quit IRC | 09:47 | |
| *** ddieterly has quit IRC | 09:48 | |
| *** yfried is now known as yfried|afk | 09:49 | |
| *** sputnik13 has quit IRC | 09:49 | |
| *** yfried|afk is now known as yfried | 09:50 | |
| *** yamamoto has joined #openstack-infra | 09:51 | |
| *** dims__ has joined #openstack-infra | 09:54 | |
| *** teran has joined #openstack-infra | 09:54 | |
| *** Longgeek has joined #openstack-infra | 09:54 | |
| *** e0ne has joined #openstack-infra | 09:55 | |
| *** ssam2 has joined #openstack-infra | 09:56 | |
| *** e0ne is now known as e0ne_ | 09:56 | |
| *** macjack has quit IRC | 09:57 | |
| *** ujuc has joined #openstack-infra | 09:58 | |
| *** teran has quit IRC | 09:58 | |
| *** ghostpl_ has joined #openstack-infra | 10:01 | |
| *** e0ne_ has quit IRC | 10:05 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config: Add networking-bagpipe-l2 project https://review.openstack.org/161290 | 10:05 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Enable Neutron in neutronclient functional test job https://review.openstack.org/163746 | 10:06 |
| *** zz_johnthetubagu is now known as johnthetubaguy | 10:07 | |
| *** otter768 has joined #openstack-infra | 10:08 | |
| *** e0ne has joined #openstack-infra | 10:09 | |
| *** e0ne has quit IRC | 10:09 | |
| *** bharath_ has joined #openstack-infra | 10:10 | |
| *** e0ne has joined #openstack-infra | 10:10 | |
| bharath_ | Hi, can we add launchpad group as a user in this file https://github.com/stackforge/stackalytics/blob/master/etc/default_data.json? | 10:10 |
| *** sputnik13 has joined #openstack-infra | 10:12 | |
| *** jp_at_hp has joined #openstack-infra | 10:13 | |
| *** otter768 has quit IRC | 10:13 | |
| *** Longgeek has quit IRC | 10:14 | |
| *** sputnik13 has quit IRC | 10:14 | |
| *** gokrokve has joined #openstack-infra | 10:16 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Install puppet modules on the right order https://review.openstack.org/163752 | 10:17 |
| *** jlanoux has quit IRC | 10:18 | |
| *** salv-orl_ has joined #openstack-infra | 10:18 | |
| *** sputnik13 has joined #openstack-infra | 10:18 | |
| *** imcsk8 is now known as imcsk8|zZz | 10:18 | |
| *** salv-orlando has quit IRC | 10:20 | |
| *** Qiming has quit IRC | 10:21 | |
| *** gokrokve has quit IRC | 10:21 | |
| *** fdegir has quit IRC | 10:22 | |
| *** fdegir has joined #openstack-infra | 10:22 | |
| *** salv-orl_ has quit IRC | 10:22 | |
| *** jedimike has joined #openstack-infra | 10:23 | |
| *** kiseok7_ has quit IRC | 10:24 | |
| *** andreykurilin_ has joined #openstack-infra | 10:26 | |
| *** markvoelker has joined #openstack-infra | 10:31 | |
| *** BobBall_AWOL is now known as BobBall | 10:33 | |
| *** yamamoto has quit IRC | 10:35 | |
| *** markvoelker has quit IRC | 10:36 | |
| *** jistr has quit IRC | 10:36 | |
| *** ominakov_ has quit IRC | 10:38 | |
| *** che-arne has quit IRC | 10:38 | |
| *** che-arne has joined #openstack-infra | 10:39 | |
| *** pelix has joined #openstack-infra | 10:41 | |
| *** teran has joined #openstack-infra | 10:42 | |
| *** sarob has joined #openstack-infra | 10:43 | |
| *** teran_ has joined #openstack-infra | 10:44 | |
| *** ddieterly has joined #openstack-infra | 10:45 | |
| *** teran has quit IRC | 10:46 | |
| *** mpaolino has quit IRC | 10:48 | |
| *** ddieterly has quit IRC | 10:49 | |
| *** sarob has quit IRC | 10:49 | |
| AJaeger | bharath_, infra team does not run stackalytics - not sure what'S the appropriate channel is | 10:51 |
| *** pc_m has joined #openstack-infra | 10:51 | |
| *** skraynev has quit IRC | 10:52 | |
| *** jlanoux has joined #openstack-infra | 10:52 | |
| *** skraynev has joined #openstack-infra | 10:52 | |
| *** skraynev has quit IRC | 10:53 | |
| *** jistr has joined #openstack-infra | 10:53 | |
| *** skraynev has joined #openstack-infra | 10:53 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-zuul: Add fetch_url and poll parametesr to zuul.conf https://review.openstack.org/162591 | 10:54 |
| jedimike | i'm writing the initial version of puppet-bandersnatch, bit confused about names of dependencies. I see storyboard has openstackci-httpd as a dependency, does that provide apache? I can't find that module anywhere, the naming conventions seem a bit weird because i can find puppet-httpd which calls itself openstackinfra-httpd... can anyone explain? | 10:54 |
| vponomaryov | bharath_: you can contribute to stackalytics as well as to any other openstack projects proposing gerrit change. | 10:58 |
| *** erlon has joined #openstack-infra | 10:58 | |
| bharath_ | vponomaryov, AJaeger okay thank you, I will do that | 10:59 |
| *** kaisers has joined #openstack-infra | 11:00 | |
| kaisers | Hi! | 11:02 |
| kaisers | Does anybody working on a thirdparty CI here have issues fetching the gerrit eventstream today? | 11:02 |
| *** skraynev_ has joined #openstack-infra | 11:03 | |
| *** skraynev_ has quit IRC | 11:03 | |
| *** garyk has quit IRC | 11:04 | |
| *** garyk has joined #openstack-infra | 11:05 | |
| *** rlandy has joined #openstack-infra | 11:06 | |
| *** ominakov_ has joined #openstack-infra | 11:13 | |
| *** gokrokve has joined #openstack-infra | 11:14 | |
| *** andreykurilin_ has quit IRC | 11:18 | |
| *** gokrokve has quit IRC | 11:19 | |
| *** Qiming has joined #openstack-infra | 11:20 | |
| Qiming | can some infra-cores help review this, thanks! https://review.openstack.org/#/c/163292/ | 11:21 |
| *** ominakov_ has quit IRC | 11:22 | |
| *** Qiming_ has joined #openstack-infra | 11:27 | |
| *** Qiming has quit IRC | 11:31 | |
| *** yfried is now known as yfried|afk | 11:31 | |
| openstackgerrit | Sergey Lukjanov proposed openstack-infra/project-config: Add py27-scenario-unit tox non-voting job for Sahara https://review.openstack.org/163776 | 11:32 |
| *** markvoelker has joined #openstack-infra | 11:33 | |
| *** hichihara has quit IRC | 11:34 | |
| *** ldnunes has joined #openstack-infra | 11:36 | |
| *** ominakov_ has joined #openstack-infra | 11:37 | |
| *** markvoelker has quit IRC | 11:37 | |
| BobBall | doesn't look like it kaisers | 11:41 |
| BobBall | mine looks fine | 11:41 |
| *** yfried|afk is now known as yfried | 11:42 | |
| *** ominakov_ has quit IRC | 11:42 | |
| *** esker has quit IRC | 11:44 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Split auth in api and change user_tokens https://review.openstack.org/153591 | 11:45 |
| *** ddieterly has joined #openstack-infra | 11:45 | |
| *** scheuran_ has joined #openstack-infra | 11:46 | |
| *** doug-fish has joined #openstack-infra | 11:47 | |
| *** markus_z has quit IRC | 11:49 | |
| *** mpavone has quit IRC | 11:49 | |
| *** scheuran has quit IRC | 11:49 | |
| *** ddieterly has quit IRC | 11:50 | |
| *** markus_z has joined #openstack-infra | 11:50 | |
| *** isviridov_vction is now known as isviridov | 11:54 | |
| *** rfolco has joined #openstack-infra | 11:56 | |
| *** kgiusti has joined #openstack-infra | 11:58 | |
| *** spzala has joined #openstack-infra | 11:58 | |
| openstackgerrit | Merged openstack-infra/storyboard: Removed Process Pool execution. https://review.openstack.org/161374 | 11:59 |
| *** Longgeek has joined #openstack-infra | 11:59 | |
| openstackgerrit | Merged openstack-infra/storyboard: Added python-dateutil https://review.openstack.org/161017 | 11:59 |
| *** tsg_ has joined #openstack-infra | 11:59 | |
| *** salv-orlando has joined #openstack-infra | 12:00 | |
| *** mpavone has joined #openstack-infra | 12:01 | |
| *** adalbas has joined #openstack-infra | 12:02 | |
| *** yfried is now known as yfried|afk | 12:03 | |
| *** cdent has joined #openstack-infra | 12:03 | |
| *** yfried|afk is now known as yfried | 12:04 | |
| *** mwagner_lap has quit IRC | 12:04 | |
| *** salv-orlando has quit IRC | 12:08 | |
| *** baoli has joined #openstack-infra | 12:08 | |
| *** salv-orlando has joined #openstack-infra | 12:08 | |
| *** otter768 has joined #openstack-infra | 12:09 | |
| *** funzo has joined #openstack-infra | 12:11 | |
| *** otter768 has quit IRC | 12:14 | |
| *** gokrokve has joined #openstack-infra | 12:14 | |
| *** markvoelker has joined #openstack-infra | 12:15 | |
| *** esker has joined #openstack-infra | 12:15 | |
| *** gokrokve has quit IRC | 12:16 | |
| *** marcusvrn has joined #openstack-infra | 12:16 | |
| *** gokrokve has joined #openstack-infra | 12:16 | |
| *** marun has joined #openstack-infra | 12:17 | |
| *** Longgeek has quit IRC | 12:19 | |
| *** salv-orlando has quit IRC | 12:20 | |
| *** gokrokve has quit IRC | 12:21 | |
| *** skraynev has quit IRC | 12:23 | |
| *** skraynev has joined #openstack-infra | 12:23 | |
| *** sdake has joined #openstack-infra | 12:23 | |
| *** sdake__ has quit IRC | 12:23 | |
| *** tiswanso has quit IRC | 12:29 | |
| *** marcusvrn has quit IRC | 12:30 | |
| *** marcusvrn has joined #openstack-infra | 12:32 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 12:35 | |
| *** e0ne is now known as e0ne_ | 12:35 | |
| *** dims__ has quit IRC | 12:35 | |
| *** dims_ has joined #openstack-infra | 12:36 | |
| *** spredzy is now known as spredzy|afk | 12:36 | |
| *** bswartz has quit IRC | 12:37 | |
| *** dprince has joined #openstack-infra | 12:37 | |
| kaisers | BobBall: ok, thanks! | 12:39 |
| *** Longgeek has joined #openstack-infra | 12:39 | |
| openstackgerrit | Matthias Runge proposed openstack/requirements: Raise cap for Django https://review.openstack.org/155353 | 12:41 |
| *** marun has quit IRC | 12:41 | |
| *** aysyd has joined #openstack-infra | 12:43 | |
| *** michchap has joined #openstack-infra | 12:43 | |
| *** e0ne_ has quit IRC | 12:45 | |
| mkj_wk | fungi: found that neither pexpect nor local reimplementation will help with my test case problem, because the output is stolen for logging by the test suite | 12:45 |
| *** ddieterly has joined #openstack-infra | 12:45 | |
| mkj_wk | fungi: wondering whether there's something in the testware that would let me wait for logged output | 12:46 |
| *** deepakcs has quit IRC | 12:46 | |
| *** ujuc has quit IRC | 12:46 | |
| *** vipul has quit IRC | 12:48 | |
| *** radez_g0n3 is now known as radez | 12:48 | |
| *** sdake has quit IRC | 12:48 | |
| *** vipul has joined #openstack-infra | 12:48 | |
| *** sdake has joined #openstack-infra | 12:49 | |
| *** devanand1 has quit IRC | 12:49 | |
| *** devananda has joined #openstack-infra | 12:50 | |
| *** esker has quit IRC | 12:50 | |
| *** bharath_ has quit IRC | 12:51 | |
| *** hdd_ has joined #openstack-infra | 12:51 | |
| *** mwagner_lap has joined #openstack-infra | 12:52 | |
| kaisers | BobBall: FYI - Seems to be project related. I was using ci-sandbox for testing which had the issue. Experimenting with cinder works... | 12:52 |
| BobBall | Fair enough | 12:54 |
| *** zhiwei has joined #openstack-infra | 12:56 | |
| *** amotoki has quit IRC | 12:57 | |
| *** esker has joined #openstack-infra | 12:59 | |
| *** zhiwei has quit IRC | 12:59 | |
| *** yfried_ has joined #openstack-infra | 13:00 | |
| *** saper has joined #openstack-infra | 13:01 | |
| *** pblaho has joined #openstack-infra | 13:01 | |
| *** pblaho has quit IRC | 13:02 | |
| *** _nadya_ has quit IRC | 13:02 | |
| *** pblaho_ has joined #openstack-infra | 13:02 | |
| fungi | mkj_wk: did you see my question yesterday about python 2.x vs 3.x (input vs raw_input)? | 13:02 |
| *** _nadya_ has joined #openstack-infra | 13:02 | |
| *** dalgaaf has joined #openstack-infra | 13:03 | |
| *** esker has quit IRC | 13:03 | |
| *** yfried__ has joined #openstack-infra | 13:03 | |
| *** dboik has joined #openstack-infra | 13:03 | |
| *** hdd_ has quit IRC | 13:03 | |
| *** yfried has quit IRC | 13:03 | |
| fungi | it seemed like you had said something earlier about it working in 3.x and not 2.x (or was it the other way around?) | 13:03 |
| *** bswartz has joined #openstack-infra | 13:04 | |
| *** hdd_ has joined #openstack-infra | 13:05 | |
| *** ddieterly has quit IRC | 13:05 | |
| *** dboik_ has joined #openstack-infra | 13:05 | |
| *** yfried_ has quit IRC | 13:05 | |
| mkj_wk | fungi: I didn't see the question; my only 2 vs. 3 thing was whether pexpect was already in the stack | 13:06 |
| mkj_wk | but now I understand that pexpect wouldn't fix it either | 13:06 |
| mkj_wk | even unbuffered Popen and a prod.stdout.read(1) hangs forever | 13:07 |
| mkj_wk | when the timeout hits, all the output from the process is in the test log | 13:07 |
| *** asselin_ has joined #openstack-infra | 13:07 | |
| *** mestery_ has quit IRC | 13:08 | |
| mkj_wk | I don't yet understand how that happens with subprocess.Popen since neither Popen nor subprocess.PIPE is mocked | 13:08 |
| *** dboik has quit IRC | 13:08 | |
| mkj_wk | fungi: all the advice I've seen on teh intarwebs about testing raw_input in subprocesses is "doctor doctor it hurts when I do this / don't do that then" advice that says you should not be invoking a subprocess, and should instead call the main method and mock out raw_input | 13:09 |
| mkj_wk | when I'm writing functional tests for my own work that's what I do | 13:09 |
| fungi | mkj_wk: got it, so input() isn't problematic but raw_input() is? | 13:10 |
| mkj_wk | but I hesitate to convert the entire git-review functional test suite to invoke main | 13:10 |
| mkj_wk | well, I was only searching regarding raw_input :) | 13:10 |
| mkj_wk | since that's what do_input is an alias for | 13:10 |
| fungi | point being we choose one or the other depending on which python major version is calling it | 13:11 |
| fungi | yeah | 13:11 |
| mkj_wk | if the functional tests wrapped cmd:_main() in a chdir we could just mock do_input | 13:11 |
| mkj_wk | (that's what I do) | 13:11 |
| *** dboik_ has quit IRC | 13:11 | |
| mkj_wk | but I'm guessing there's a reason that git-review doesn't do that; I just don't know what it is | 13:12 |
| fungi | well, i'm not at all opposed to a refactoring of the testsuite that's there. it's sort of hackish to start with but was really just a placeholder for better structured testing | 13:12 |
| mkj_wk | if you know the right way to get at the stolen output, I could do an "expect" o ntop of that | 13:12 |
| *** yamamoto has joined #openstack-infra | 13:13 | |
| mkj_wk | I also hook up coverage reporting -- helps me be sure I covered at least all the lines when adding new code | 13:13 |
| fungi | (before that functional test set it was me running a script to exercise git-review against a dev gerrit from the command line, and before that it was people running through an actual set of testing instructions) | 13:14 |
| *** gokrokve has joined #openstack-infra | 13:14 | |
| fungi | we do have a coverage job we can add to it pretty trivially i think, though with the current structure i don't know if it would report very accurately | 13:15 |
| mkj_wk | I don't know how what I do would work under tox/testr | 13:16 |
| jedimike | i'm writing a new module, puppet-bandersnatch. what dependencies should i use in the metadata to make sure i have apache and logrotate available? (some of the naming conventions are confusing me) | 13:17 |
| mkj_wk | Using nose, I always invoke with --with-coverage --cover-package <mypackage> | 13:17 |
| mkj_wk | at rPath we used a --patch-coverage that we used to make sure that all the lines we had added or modified were covered... :) | 13:18 |
| fungi | mkj_wk: there may already be a cover env specified in tox.ini, but if there's not it's easy to add | 13:18 |
| fungi | jedimike: if you use the apache module for puppet (or our httpd fork of it) then it will install apache automatically | 13:18 |
| *** gokrokve has quit IRC | 13:18 | |
| *** stevemar has joined #openstack-infra | 13:18 | |
| fungi | not sure about logrotate, i'll have to hunt | 13:19 |
| jedimike | fungi, so I see puppet-httpd, but it lists its name as openstackinfra-httpd, and other modules seem to use openstackci-httpd, which is correct? | 13:19 |
| fungi | jedimike: good question. i don't know if we've actually started using it as a dep anywhere yet. nibalizer could probably tell you | 13:20 |
| *** mestery has joined #openstack-infra | 13:20 | |
| jedimike | thanks :) | 13:20 |
| fungi | i think at the moment we're still just using the apache module everywhere | 13:20 |
| jedimike | i'll grep the existing modules, see what i can find out and get working | 13:21 |
| fungi | and yeah, if you grep for logrotate you'll likely see how we're pulling it in elsewhere, i just don't remember off the top of my head | 13:21 |
| mestery | Infra folks: Can someone help merge this one: https://review.openstack.org/#/c/163746/ | 13:21 |
| *** dkranz has joined #openstack-infra | 13:21 | |
| *** pblaho_ has quit IRC | 13:21 | |
| mestery | We need that to get the python-neutronclient functional job running to move out our CLI tests from tempest /cc mtreinish | 13:21 |
| *** _nadya_ has quit IRC | 13:22 | |
| fungi | mkj_wk: yeah, no cover env specified in tox.ini there yet, i'll try adding a typical one and see what i get | 13:23 |
| mkj_wk | cool -- this is my introduction to both testr and tox so I'm a bit out of depth :) | 13:24 |
| fungi | mkj_wk: bear in mind that git-review is one of our least rigorously tested projects and could certainly use an overhaul of its test setup | 13:24 |
| *** fdegir has quit IRC | 13:24 | |
| fungi | so it's probably a poor intro there | 13:24 |
| *** Longgeek has quit IRC | 13:24 | |
| openstackgerrit | MORITA Kazutaka proposed openstack-infra/project-config: Add non-voting Sheepdog test job to cinder check pipeline https://review.openstack.org/153868 | 13:24 |
| *** bknudson has joined #openstack-infra | 13:26 | |
| *** peristeri has joined #openstack-infra | 13:27 | |
| mkj_wk | fungi: I'll try to make fewer assumptions then :) | 13:27 |
| fungi | yeah, we're spread pretty thin and that project needs some extra love | 13:27 |
| mkj_wk | fungi: I've been grovelling through /proc looking to see if I can find what process has a file descriptor that is logging the output... | 13:27 |
| mkj_wk | but no luck so far | 13:27 |
| mkj_wk | do you happen to know how it happens that the output gets logged from the processes being started under the test suite?? | 13:28 |
| mkj_wk | (it's not in git-review itself, it's just a detail I've never needed to know for testing other things) | 13:28 |
| fungi | i don't but should be able to find it | 13:28 |
| mkj_wk | I'm feeling like an idiot after over ten years of writing tests for practically all my python code :) | 13:29 |
| fungi | oh, that. i think subprocess.popen() gets wrapped in one of the testing tools | 13:29 |
| fungi | clarkb just had a patch to nodepool related to that. looking | 13:29 |
| *** spredzy|afk is now known as spredzy | 13:30 | |
| mkj_wk | thanks! | 13:31 |
| *** dboik has joined #openstack-infra | 13:32 | |
| *** dboik has quit IRC | 13:32 | |
| * mkj_wk has been trying really hard not to ask you to "take a free puppy" with this new feature ☺ | 13:32 | |
| *** dboik has joined #openstack-infra | 13:32 | |
| *** tiswanso has joined #openstack-infra | 13:34 | |
| *** ddieterly has joined #openstack-infra | 13:36 | |
| *** mriedem_away is now known as mriedem | 13:36 | |
| *** annegentle has joined #openstack-infra | 13:37 | |
| fungi | mkj_wk: ahh, it was jeblair's patch... so anyway we added popen logging to nodepool's tests like https://review.openstack.org/#/c/162673/2/nodepool/tests/__init__.py | 13:37 |
| *** Longgeek has joined #openstack-infra | 13:37 | |
| fungi | i'm not immediately spotting a similar construct in teh git-review tests | 13:38 |
| fungi | we do mock out run_command_exc() | 13:40 |
| fungi | but just in ConfigTestCase class | 13:41 |
| mkj_wk | yeah | 13:41 |
| fungi | and not adding any explicit logging even there | 13:41 |
| mkj_wk | thus why I think that what is happening is in unittest or testtools | 13:42 |
| mkj_wk | not like I haven't used subprocess.Popen a lot... | 13:42 |
| fungi | so if you look at run_command_status() we do have an explicit print call when VERBOSE is set | 13:42 |
| fungi | my money says we set VERBOSE when running under the testsuite | 13:43 |
| *** gordc has joined #openstack-infra | 13:43 | |
| *** tnovacik has quit IRC | 13:43 | |
| mkj_wk | if I don't put in the time.sleep(1) and instead do proc.stdout.read(1), the 2-minute timeout fires, and it shows the output I was trying to wait for | 13:43 |
| fungi | self._run_git_review('-v') | 13:44 |
| mkj_wk | so clearly, the output of the subprocess is going somewhere, and it's not to the stdout=subprocess.PIPE | 13:44 |
| *** e0ne has joined #openstack-infra | 13:44 | |
| mkj_wk | if I could just wait for the prompt to show up before writing the answer, the whole thing would work | 13:45 |
| mkj_wk | since it prints out the output after the test times out, the output clearly lives *somewhere* | 13:46 |
| *** saper has quit IRC | 13:46 | |
| mkj_wk | if I could just figure out where, I could expect/send instead of time.sleep(1) | 13:46 |
| *** esker has joined #openstack-infra | 13:46 | |
| *** esker has quit IRC | 13:47 | |
| *** sarob has joined #openstack-infra | 13:48 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Upgrade openstackid to 1.0.5 https://review.openstack.org/163820 | 13:49 |
| *** dkliban_afk is now known as dkliban | 13:50 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:51 | |
| *** sarob has quit IRC | 13:53 | |
| mkj_wk | fungi: oh. testr is opening a pty | 13:53 |
| mkj_wk | I think testr is reporting output on that pty | 13:53 |
| *** marcusvrn has quit IRC | 13:53 | |
| *** marcusvrn has joined #openstack-infra | 13:54 | |
| mkj_wk | is git-review opening /dev/tty?? | 13:54 |
| mkj_wk | doesn't look like it | 13:54 |
| fungi | not sure. i believe testr wants to grab stdio so it can attach it to the multiplexed subunit stream of test results, but i'm not much of a testing guru | 13:55 |
| mkj_wk | oh, huh | 13:55 |
| fungi | as soon as clarkb's awake he may have some ideas | 13:55 |
| mrmartin | er | 13:56 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Conditionally manage iptables https://review.openstack.org/163829 | 13:57 |
| mrmartin | fungi: if you are here, we need a +2 on this patch, https://review.openstack.org/#/c/163820/ because openstack.org changed to openstackid as auth, and some users cannot login until a bug is fixed that is landed in 1.0.5 | 13:57 |
| mkj_wk | fungi: at least it clarifies to me that the output is not available in process; it's only being held in the supervisory testr process | 13:58 |
| mkj_wk | fungi: it explains why I couldn't find it when I walked up the stack printing out the locals() at each frame :) | 13:58 |
| fungi | mrmartin: done--thanks for the quick heads up | 13:58 |
| mrmartin | fungi thnx | 13:58 |
| AJaeger | anteaya, please review https://review.openstack.org/#/c/159512/ again - both ' and " are valid quotation marks. Reviews by others on it are welcome as well ;) | 13:59 |
| *** baoli has quit IRC | 14:00 | |
| *** bhunter71 has joined #openstack-infra | 14:01 | |
| *** baoli_ has joined #openstack-infra | 14:02 | |
| *** baoli_ has quit IRC | 14:02 | |
| *** r-daneel has joined #openstack-infra | 14:03 | |
| *** baoli_ has joined #openstack-infra | 14:04 | |
| *** esker has joined #openstack-infra | 14:04 | |
| *** esker has quit IRC | 14:04 | |
| *** nelsnelson has joined #openstack-infra | 14:06 | |
| *** zz_dimtruck is now known as dimtruck | 14:06 | |
| *** nelsnelson has quit IRC | 14:07 | |
| *** yamamoto has quit IRC | 14:07 | |
| *** mattfarina has joined #openstack-infra | 14:08 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Ensure filepath exists https://review.openstack.org/163834 | 14:09 |
| mordred | fungi: ^^ fix ansible please? | 14:09 |
| *** pblaho_ has joined #openstack-infra | 14:09 | |
| *** otter768 has joined #openstack-infra | 14:09 | |
| openstackgerrit | Merged openstack-infra/system-config: Upgrade openstackid to 1.0.5 https://review.openstack.org/163820 | 14:10 |
| fungi | mrmartin: ^ i'll go ahead and pull that on openstackid.org right away | 14:10 |
| mrmartin | fungi: will you do a puppet apply ? | 14:10 |
| fungi | yes | 14:11 |
| openstackgerrit | Somay Jain proposed openstack-infra/jenkins-job-builder: Adding more configurable options in Notifications plugin https://review.openstack.org/163137 | 14:11 |
| openstackgerrit | Marc Koderer proposed openstack-dev/specs-cookiecutter: Fix example template https://review.openstack.org/163835 | 14:11 |
| fungi | mrmartin: it's upgraded now. see if that's any better | 14:12 |
| *** amotoki has joined #openstack-infra | 14:12 | |
| fungi | Notice: /Stage[main]/Openstackid/Exec[update-site]/returns: executed successfully | 14:12 |
| mrmartin | nice | 14:14 |
| mrmartin | thnx. | 14:14 |
| fungi | once we have an openstackweb-dev hooked up to openstackid-dev there hopefully won't be scrambles like this | 14:14 |
| *** gokrokve has joined #openstack-infra | 14:14 | |
| fungi | but that's awesome that www.o.o is now authenticating against openstackid.org. i had no idea they were that close to switching over | 14:14 |
| *** krotscheck has quit IRC | 14:14 | |
| *** yfried__ has quit IRC | 14:14 | |
| *** otter768 has quit IRC | 14:15 | |
| mordred | ++ | 14:15 |
| *** BrianB_ has joined #openstack-infra | 14:16 | |
| fungi | the upshot being that was the blocker to being able to start removing any of the password management bits from the foundation website software, so that eventually it and openstackid.org no longer need to share a backend database | 14:17 |
| *** krotscheck has joined #openstack-infra | 14:17 | |
| fungi | eventually the www site should just be asking openstackid for those bits | 14:18 |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Ensure filepath exists https://review.openstack.org/163834 | 14:19 |
| *** gokrokve has quit IRC | 14:19 | |
| fungi | (e.g. whatever bits openstackid is handling should be its and its alone) | 14:19 |
| *** yamamoto has joined #openstack-infra | 14:19 | |
| mrmartin | awesome | 14:20 |
| *** yfried__ has joined #openstack-infra | 14:20 | |
| *** zhiwei has joined #openstack-infra | 14:20 | |
| *** ominakov_ has joined #openstack-infra | 14:21 | |
| *** yfried__ has quit IRC | 14:21 | |
| BrianB_ | I could use help with an error I am getting when doing git review, fatal: ICLA contributor agreement requires current contact information. | 14:22 |
| *** mrmartin has quit IRC | 14:23 | |
| BrianB_ | Looking at bug/1317957, I verified my email, agreement signed, | 14:23 |
| fungi | BrianB_: https://ask.openstack.org/question/56720 | 14:23 |
| *** garyh has joined #openstack-infra | 14:24 | |
| fungi | BrianB_: see if your profile is community or speaker or something rather than foundation member | 14:24 |
| *** funzo has quit IRC | 14:24 | |
| *** vhoward has left #openstack-infra | 14:24 | |
| *** ominakov_ has quit IRC | 14:25 | |
| *** ominakov_ has joined #openstack-infra | 14:26 | |
| *** yfried has joined #openstack-infra | 14:27 | |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Don't display project column in project change list https://review.openstack.org/163843 | 14:27 |
| *** timcline has joined #openstack-infra | 14:28 | |
| openstackgerrit | Merged openstack-dev/specs-cookiecutter: Fix example template https://review.openstack.org/163835 | 14:28 |
| fungi | jeblair: ooh! room for useful data on my tiny screens ;) | 14:28 |
| BrianB_ | fungi: my profile shows Current Member Level: Foundation Member | 14:28 |
| *** wenlock has joined #openstack-infra | 14:29 | |
| fungi | BrianB_: and you have tried to submit contact information at https://review.openstack.org/#/settings/contact ? | 14:29 |
| jeblair | fungi: heh, sorry about that. apparently i broke that a while ago. | 14:30 |
| openstackgerrit | Merged openstack-infra/project-config: Enable Neutron in neutronclient functional test job https://review.openstack.org/163746 | 14:30 |
| fungi | jeblair: oh, i missed that it had ever worked any other way | 14:30 |
| jeblair | fungi: yeah, i'm not a fan of crowding that screen | 14:31 |
| *** mattfarina has quit IRC | 14:31 | |
| BrianB_ | fungi: my contack info matches the info from git config --list (name and email) | 14:31 |
| fungi | BrianB_: that page says "Contact information last updated on [some date and time]"? | 14:32 |
| *** gokrokve has joined #openstack-infra | 14:33 | |
| fungi | jeblair: oddly, the updated column is now 9 characters wide but needs 10 to fit a date without wrapping to a second line | 14:33 |
| *** prad has joined #openstack-infra | 14:34 | |
| fungi | jeblair: not sure which patch is bringing that in... i can bisect if you aren't sure | 14:34 |
| jeblair | fungi: i don't think that change wolud have affected that (it wraps 2 chars onto the next line if i do a query that causes the project column to appear) | 14:35 |
| jeblair | fungi: my guess is that happened when we switched to iso dates | 14:35 |
| *** yfried is now known as yfried|afk | 14:35 | |
| jeblair | fungi: all the values in that column used to be shorter | 14:35 |
| *** yfried|afk is now known as yfried | 14:35 | |
| *** yfried has quit IRC | 14:35 | |
| fungi | ahh, yep, latest release does it too | 14:35 |
| fungi | i'll confirm | 14:35 |
| jeblair | fungi: but xkcd issued a comic so now we have no choice | 14:35 |
| *** marun has joined #openstack-infra | 14:35 | |
| *** achanda has joined #openstack-infra | 14:36 | |
| fungi | heh, well i already set my LC_TIME=en_DK.UTF-8 | 14:36 |
| *** eharney has joined #openstack-infra | 14:39 | |
| *** _nadya_ has joined #openstack-infra | 14:40 | |
| *** mpaolino has joined #openstack-infra | 14:40 | |
| fungi | yep, that change is the culprit. i'll see if i can figure out a quick fix | 14:41 |
| *** r-daneel has quit IRC | 14:42 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Change Story to StorySummary in search https://review.openstack.org/163852 | 14:42 |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Switch "Updated" column to fixed width https://review.openstack.org/163853 | 14:42 |
| jeblair | fungi: ^ | 14:42 |
| fungi | you're fast! | 14:43 |
| *** woodster_ has joined #openstack-infra | 14:43 | |
| *** unicell1 has joined #openstack-infra | 14:43 | |
| openstackgerrit | Aleksey Ripinen proposed openstack-infra/storyboard: Change Story to StorySummary in search https://review.openstack.org/163852 | 14:43 |
| *** unicell has quit IRC | 14:43 | |
| *** achanda has quit IRC | 14:44 | |
| fungi | jeblair: that works great | 14:44 |
| *** Somay has joined #openstack-infra | 14:44 | |
| *** r-daneel has joined #openstack-infra | 14:45 | |
| *** sarob has joined #openstack-infra | 14:48 | |
| Somay | Hello! Are there any commiters around who could review my patch for the openstack-infra/jenkins-job-builder project? https://review.openstack.org/#/c/163137/ | 14:49 |
| *** carl_baldwin has joined #openstack-infra | 14:49 | |
| *** sabeen has joined #openstack-infra | 14:49 | |
| *** tonytan4ever has joined #openstack-infra | 14:53 | |
| *** thedodd has joined #openstack-infra | 14:53 | |
| *** sarob has quit IRC | 14:54 | |
| *** Longgeek has quit IRC | 14:55 | |
| BrianB_ | fungi:I added my address to contact and saved it, now it is working. Thanks for your help | 14:56 |
| *** xgerman has joined #openstack-infra | 14:57 | |
| *** mpaolino has quit IRC | 14:57 | |
| clarkb | fungi: mkj_wk yes testr expects the individual test runners it starts to feed subunit back to it over stdout | 14:58 |
| *** dboik_ has joined #openstack-infra | 14:58 | |
| fungi | clarkb: does any spawned subprocess stdout pipe get renamed to something else? | 14:58 |
| *** yfried has joined #openstack-infra | 14:59 | |
| *** ominakov_ has quit IRC | 14:59 | |
| *** tiswanso has quit IRC | 14:59 | |
| fungi | BrianB_: you're welcome. it looks like the bit about entering contact information in the cla subsection of http://docs.openstack.org/infra/manual/developers.html#account-setup could stand to be a little more explicit. i'll try to remember to update that documentation | 14:59 |
| *** Guest88753 has quit IRC | 15:00 | |
| clarkb | mordred: fungi ansible seems to be working for puppet now. Just need to install puppet3 on the zm05-zm08 machines | 15:00 |
| *** amitgandhinz has joined #openstack-infra | 15:00 | |
| *** baoli_ has quit IRC | 15:00 | |
| *** fdegir has joined #openstack-infra | 15:00 | |
| *** mattfarina has joined #openstack-infra | 15:00 | |
| clarkb | fungi: no, you should still be able to use stdout and the subunit parser passes that data through if it isn't subunit | 15:00 |
| *** dboik has quit IRC | 15:00 | |
| mordred | clarkb: woot! | 15:00 |
| *** e0ne is now known as e0ne_ | 15:01 | |
| *** Hal has joined #openstack-infra | 15:01 | |
| *** claudiub has joined #openstack-infra | 15:01 | |
| *** Hal is now known as Guest30394 | 15:01 | |
| clarkb | I think that is just a simple matter of updating the package pin file and doing an apt-get update apt-get install puppet | 15:04 |
| clarkb | will give that a go shortly | 15:04 |
| *** BharatK has quit IRC | 15:06 | |
| *** mpopow has joined #openstack-infra | 15:06 | |
| *** e0ne_ has quit IRC | 15:06 | |
| *** gokrokve_ has joined #openstack-infra | 15:07 | |
| clarkb | we are using 588 ports in hpcloud region b | 15:07 |
| *** tiswanso has joined #openstack-infra | 15:07 | |
| *** baoli has joined #openstack-infra | 15:08 | |
| *** MarkAtwood has joined #openstack-infra | 15:08 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 15:09 | |
| jeblair | wow, uh, so do we need to run that script continuously? | 15:09 |
| *** gokrokve has quit IRC | 15:11 | |
| clarkb | maybe? I/we should probably spend a bit more time trying to understand how ports can leak | 15:11 |
| *** mpaolino has joined #openstack-infra | 15:11 | |
| *** esp has left #openstack-infra | 15:12 | |
| clarkb | also looks like we may be trying to use most of our capacity so its possible that this is normal use. Worth watching | 15:12 |
| *** erikmwil_ has joined #openstack-infra | 15:13 | |
| *** erikmwilson has quit IRC | 15:13 | |
| *** erikmwil_ is now known as erikmwilson | 15:13 | |
| *** erikmwilson_ has joined #openstack-infra | 15:13 | |
| AJaeger | jeblair, your proposal script works now: https://review.openstack.org/#/c/163703/ | 15:13 |
| jeblair | yay! | 15:14 |
| AJaeger | jeblair, is that something we want to treat like many projects treat translation imports: A single +2 is enough - or do we want to do the usual 2 cores? | 15:14 |
| clarkb | AJaeger: jeblair I didn't get a chance to review the job, does it handle cases where the project may not be in gerrit yet? | 15:14 |
| jeblair | clarkb: no it does not. so i think we should check that before approving. | 15:14 |
| jeblair | AJaeger: ^ so i think as long as we double check that the import actually happened, single +2 is probably ok. | 15:15 |
| AJaeger | jeblair, ok. | 15:16 |
| jeblair | AJaeger, clarkb: and maybe we can extend the job to check that automatically | 15:16 |
| clarkb | sounds good | 15:16 |
| AJaeger | and if it fails, we need an email telling us about it ;/ | 15:16 |
| openstackgerrit | Chris Dent proposed openstack-infra/project-config: Add functional test job for python-ceilometerclient https://review.openstack.org/163865 | 15:16 |
| *** yamahata has joined #openstack-infra | 15:17 | |
| *** prad has quit IRC | 15:17 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Make neutronclient-dsvm-functional gating for neutronclient https://review.openstack.org/163868 | 15:18 |
| *** MarkAtwood_ has joined #openstack-infra | 15:19 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add query for bug 1431408 https://review.openstack.org/163871 | 15:19 |
| openstack | bug 1431408 in tempest "Race in tenant isolation unit tests" [High,Triaged] https://launchpad.net/bugs/1431408 | 15:19 |
| *** BharatK has joined #openstack-infra | 15:20 | |
| *** dannywilson has joined #openstack-infra | 15:20 | |
| *** e0ne has joined #openstack-infra | 15:21 | |
| clarkb | ok zm05-zm08 should have puppet3 now and ansible will puppet them | 15:21 |
| clarkb | fungi: ask.o.o needs to be added to ansible's known hosts file | 15:22 |
| clarkb | ci-backup-rs-ord is intentionally not key accepted | 15:22 |
| *** dannywil_ has joined #openstack-infra | 15:22 | |
| clarkb | we should probably run puppet agent --disable on it too if we haven't already | 15:22 |
| clarkb | pleia2: ^ thoughts | 15:22 |
| *** dannywilson has quit IRC | 15:23 | |
| *** david-ly_ is now known as david-lyle | 15:23 | |
| jeblair | clarkb: we're using a hosts file? i thought we were using puppet cert inventory | 15:23 |
| mordred | clarkb: I've got a plan for ansible-side disabling that I think we can probably get landed very soon - let me go clean up a couple of patches | 15:23 |
| mordred | jeblair: we are - but ssh host keys need to be accepted | 15:23 |
| *** tonytan4ever has quit IRC | 15:23 | |
| jeblair | oh that sorry | 15:23 |
| *** dkranz has quit IRC | 15:24 | |
| jeblair | the known_hosts file, not the file of known hosts ;) | 15:24 |
| mordred | jeblair: words are important | 15:24 |
| fungi | clarkb: i wouldn't add ask.o.o yet since it's not updated in dns? | 15:24 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/163703 | 15:24 |
| fungi | clarkb: the puppeted ask.o.o is not the one in dns | 15:25 |
| clarkb | jeblair: the ssh known hosts file | 15:25 |
| clarkb | fungi: oh good point | 15:25 |
| fungi | testing the data import is on my to do list for today | 15:26 |
| fungi | and then if that checks out, we'll schedule a brief outage to shut down the current service, do one last export, import that into the replacement server and update dns | 15:26 |
| *** ChuckC has joined #openstack-infra | 15:26 | |
| *** mpopow has quit IRC | 15:30 | |
| *** mpopow has joined #openstack-infra | 15:30 | |
| *** zhiwei has left #openstack-infra | 15:31 | |
| *** [HeOS] has joined #openstack-infra | 15:31 | |
| *** ddieterly has quit IRC | 15:31 | |
| *** HeOS has quit IRC | 15:31 | |
| *** eharney has quit IRC | 15:32 | |
| *** rwsu-afk is now known as rwsu | 15:33 | |
| *** dhritishikhar has joined #openstack-infra | 15:33 | |
| *** eharney has joined #openstack-infra | 15:33 | |
| *** reed has joined #openstack-infra | 15:34 | |
| fungi | need to think hard about whether i'm putting in a proposal for all things open in october. i'd get back with possibly one day before heading out to tokyo if so | 15:34 |
| fungi | Shrews should totally submit an abstract since he lives there | 15:34 |
| *** kaisers has quit IRC | 15:35 | |
| clarkb | notmyname: `swift stat containername objectname` should show me the X-Delete-At info if it was set correct? | 15:35 |
| * clarkb is trying to confirm we are properly setting that value on the objects now | 15:35 | |
| *** prad has joined #openstack-infra | 15:35 | |
| *** asselin_ has quit IRC | 15:36 | |
| openstackgerrit | Anthony Michon proposed openstack-infra/project-config: Add cerberus-dashboard, python-cerberusclient, stick, stick-dashboard, python-stickclient https://review.openstack.org/163876 | 15:36 |
| *** zul has quit IRC | 15:40 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for bug 1431408 https://review.openstack.org/163871 | 15:40 |
| openstack | bug 1431408 in tempest "Race in tenant isolation unit tests" [High,Triaged] https://launchpad.net/bugs/1431408 | 15:40 |
| *** zul has joined #openstack-infra | 15:41 | |
| *** mrmartin has joined #openstack-infra | 15:41 | |
| *** Somay has quit IRC | 15:44 | |
| *** BharatK has quit IRC | 15:46 | |
| *** armax has joined #openstack-infra | 15:46 | |
| *** amitgandhinz has quit IRC | 15:47 | |
| pleia2 | good morning | 15:48 |
| pleia2 | clarkb: wfm, I hadn't run disable on the agent | 15:48 |
| pleia2 | (can do that now) | 15:48 |
| *** nelsnelson has joined #openstack-infra | 15:48 | |
| *** apevec has joined #openstack-infra | 15:48 | |
| AJaeger | morning, pleia2 | 15:49 |
| apevec | hi, I need infra help, looks like I got this Icehouse backport into limbo: https://review.openstack.org/141412 | 15:49 |
| pleia2 | g'day AJaeger | 15:49 |
| apevec | it was in gate when my stable_freeze mistriggered and put -2 | 15:49 |
| apevec | it was approved and just about to merge | 15:50 |
| apevec | now re-approval keeps saying it's in merge conflict even after I rebased it | 15:50 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: Add script for creating stable releases of libraries https://review.openstack.org/163880 | 15:50 |
| fungi | apevec: i'll double-check the rebase | 15:50 |
| apevec | so looks like it is somehow half-merged? | 15:50 |
| apevec | fungi, thanks! | 15:50 |
| *** dhritishikhar has quit IRC | 15:50 | |
| *** amitgandhinz has joined #openstack-infra | 15:51 | |
| *** ihrachyshka has quit IRC | 15:52 | |
| *** asselin_ has joined #openstack-infra | 15:53 | |
| pleia2 | clarkb: elasticon was really great, they said they'd be getting videos online soon, sadly the security one was about one of their very few proprietary offerings (a security plugin), but when they went through what it covered it was *mostly* orthoginal to what we do since none of our data is secret | 15:54 |
| clarkb | I have confirmed that the X-Delete-After script change was used to upload http://logs.openstack.org/91/158891/1/check/gate-infra-puppet-apply-trusty/4e71f1b/ but swift stat on that object does not show the X-Delete-After info | 15:54 |
| clarkb | pleia2: nice (though less nice no the security thing) | 15:55 |
| clarkb | s/no/on/ | 15:55 |
| fungi | apevec: interesting, it's definitely a fast-forward merge (its current parent is the tip of the target branch) | 15:55 |
| fungi | apevec: i'll go fishing in the zuul logs for clues | 15:56 |
| *** Longgeek has joined #openstack-infra | 15:56 | |
| *** amitgandhinz has quit IRC | 15:56 | |
| apevec | yeah, looks like I confused zuul | 15:56 |
| clarkb | also apple swift makes googling for 'swift object *' really difficult | 15:56 |
| clarkb | you get results on object reflection in the language | 15:56 |
| *** dkranz has joined #openstack-infra | 15:56 | |
| *** sputnik13 has quit IRC | 15:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix return code of irc_tests.py https://review.openstack.org/163884 | 15:57 |
| clarkb | notmyname: suggestions on double checking the X-Delete-At info much appreciated /me continues to dig | 15:57 |
| AJaeger | My irc_tests for catching missing accessbot works fine - but always returns 0. Could I get some quick review, please? ^ | 15:58 |
| fungi | apevec: in the meantime, i've rechecked it and zuul seems to have been able to calculate a merge commit for it just fine | 15:58 |
| *** mrmartin has quit IRC | 15:58 | |
| *** AJaeger has quit IRC | 15:59 | |
| clarkb | using the most privileged account instead of the rw/ro swift specific accounts doesn't help either | 15:59 |
| clarkb | I can still list things just without seeing X-Delete-At info | 15:59 |
| *** BharatK has joined #openstack-infra | 15:59 | |
| clarkb | fungi: apevec that may be similar to the thing I was looking at yesterday | 16:00 |
| *** asselin_ has quit IRC | 16:00 | |
| clarkb | if the merger reconnects then it is treated as a merge failure for an in progress job. jeblair has a change up to get better logging around that, I should go review it now | 16:00 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: Add script for creating stable releases of libraries https://review.openstack.org/163880 | 16:00 |
| *** thedodd has quit IRC | 16:00 | |
| jeblair | mtreinish: does this look correct for the trove-only tempest job: http://logs.openstack.org/19/125819/6/check/gate-tempest-dsvm-trove/e4d2879/console.html ? | 16:00 |
| jeblair | 6 tests, apparently | 16:01 |
| *** carl_baldwin has quit IRC | 16:01 | |
| mtreinish | jeblair: yeah, there are only 6 tests for trove in tempest | 16:01 |
| mtreinish | and there mostly just sanity checks | 16:01 |
| jeblair | mtreinish: also, how can i find out the failure count of those tests with subunit2sql? | 16:01 |
| openstackgerrit | Qiming Teng proposed openstack-infra/project-config: new-project: stackforge/senlin https://review.openstack.org/163292 | 16:02 |
| mtreinish | jeblair: I have instructions up on: https://wiki.openstack.org/wiki/QA/Tempest-test-removal | 16:02 |
| mtreinish | in this case it would be something like: select * from tests where test_id like "tempest.api.database%"; | 16:02 |
| *** ashleighfarnham has joined #openstack-infra | 16:02 | |
| *** cody-somerville has joined #openstack-infra | 16:03 | |
| *** cody-somerville has quit IRC | 16:03 | |
| *** cody-somerville has joined #openstack-infra | 16:03 | |
| jeblair | yep: http://paste.openstack.org/show/191865/ | 16:03 |
| *** tnovacik has joined #openstack-infra | 16:03 | |
| fungi | man, a day of zuul debug logs uncompressed is up to ~1gb | 16:03 |
| *** ddieterly has joined #openstack-infra | 16:03 | |
| *** andreykurilin has quit IRC | 16:03 | |
| *** prad has quit IRC | 16:04 | |
| mtreinish | heh, I didn't expect any failures on those | 16:04 |
| *** tqtran has joined #openstack-infra | 16:04 | |
| jeblair | mtreinish: there is a .00011781401451254451 fail percentage | 16:04 |
| *** tonytan4ever has joined #openstack-infra | 16:04 | |
| *** bhuvan has joined #openstack-infra | 16:06 | |
| mtreinish | jeblair: heh, even so I was expecting 0, the tests that just do a bunch of api lists :) | 16:07 |
| jeblair | also, they take 3.4 seconds to run | 16:08 |
| *** cmluciano has left #openstack-infra | 16:08 | |
| *** krisgan has joined #openstack-infra | 16:09 | |
| *** claudiub has quit IRC | 16:09 | |
| *** scheuran_ has quit IRC | 16:09 | |
| *** eharney has quit IRC | 16:09 | |
| jeblair | i approved the change that flips that to voting | 16:09 |
| *** tiswanso has quit IRC | 16:09 | |
| *** unicell1 has quit IRC | 16:09 | |
| clarkb | out of curiousity, if they only take 3.4 seconds to run why do we run them in a special job that needs ~25 minutes of setup? | 16:09 |
| clarkb | why not just run those 3.4 seconds of test along with the 35 minutes of test for the other things? | 16:10 |
| zaro | morning | 16:10 |
| *** dboik_ has quit IRC | 16:10 | |
| *** tiswanso has joined #openstack-infra | 16:10 | |
| *** Somay has joined #openstack-infra | 16:10 | |
| *** otter768 has joined #openstack-infra | 16:10 | |
| *** dboik has joined #openstack-infra | 16:10 | |
| jeblair | clarkb: that's a good question. | 16:11 |
| *** mpaolino has quit IRC | 16:11 | |
| jeblair | clarkb: i guess it's the principle? | 16:11 |
| *** ParsectiX has joined #openstack-infra | 16:12 | |
| mtreinish | jeblair: oh, it looks like the failures were from before we fixed the check queue filtering, I pulled to check: http://logs.openstack.org/52/135152/1/check/gate-tempest-dsvm-src-python-openstackclient/c37e7ac/ | 16:12 |
| jeblair | ah | 16:12 |
| *** e0ne is now known as e0ne_ | 16:13 | |
| jeblair | clarkb: but yeah, people have been saying that gating on too many projects is a problem, but it's hard to imagine how that would be so for trove. | 16:13 |
| clarkb | jeblair: ya it just seems really odd to me. Even nodepool has more testing :) | 16:13 |
| jeblair | well, now :) | 16:14 |
| * clarkb notes that pip is failing to install keyring on static.o.o | 16:14 | |
| *** yfried has quit IRC | 16:14 | |
| clarkb | which is used by os_loganalyze | 16:15 |
| clarkb | will have to sort that out after breakfast | 16:15 |
| *** otter768 has quit IRC | 16:15 | |
| *** dpyzhov_ has joined #openstack-infra | 16:15 | |
| *** dangers_away is now known as dangers | 16:16 | |
| mtreinish | jeblair, clarkb: fwiw, when we get the tempest-lib bits in place for the trove client those tests will probably be removed from tempest | 16:16 |
| ctlaugh | What's the best way to test a 3rd-party CI setup once it's running? I used the os-ext scripts to deploy, and it's currently pointed at sandbox, but there's nothing out there. Looking for recommendations of the best way to test that everything is running together correctly. | 16:16 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/project-config: Add python3-jobs for gear https://review.openstack.org/158148 | 16:16 |
| *** tsg_ has quit IRC | 16:17 | |
| *** bswartz has quit IRC | 16:17 | |
| clarkb | ctlaugh push code to sandbox let it exercise your system | 16:17 |
| *** e0ne_ is now known as e0ne | 16:17 | |
| clarkb | also you can run a silent pipeline against an actual project and check the results | 16:17 |
| jeblair | mtreinish: yeah, and when that's the case, and we move devstack bits into trove, it should be (automatically) removed from the integrated change queue | 16:18 |
| jeblair | mtreinish: (because we won't be running trove jobs on devstack and tempest) | 16:18 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add initial compute functional tests to Shade https://review.openstack.org/158463 | 16:18 |
| clarkb | I can install keyring using the version of pip on static.o.o locally | 16:19 |
| *** koolhead17 has joined #openstack-infra | 16:19 | |
| *** koolhead17 has quit IRC | 16:19 | |
| *** koolhead17 has joined #openstack-infra | 16:19 | |
| *** mrmartin has joined #openstack-infra | 16:19 | |
| asselin | jeblair, please +1 this change: https://review.openstack.org/#/c/163246/ | 16:20 |
| *** BrianB_ has quit IRC | 16:20 | |
| jeblair | asselin: done | 16:20 |
| jeblair | thx | 16:20 |
| *** claudiub has joined #openstack-infra | 16:20 | |
| asselin | thank you | 16:20 |
| *** Qiming_ has quit IRC | 16:22 | |
| *** eharney has joined #openstack-infra | 16:22 | |
| *** mrmartin has quit IRC | 16:22 | |
| *** prad has joined #openstack-infra | 16:23 | |
| clarkb | dhellmann: did aeromancer acls sort themselves out? | 16:25 |
| *** rkukura has quit IRC | 16:25 | |
| ctlaugh | clarkb: What makes a pipeline silent? Just not having a "success:" or "failure:" I like the idea of being able to run like that for a while, but I definitely don't want to generate noise. | 16:25 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 16:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Make trove-only tempest job voting https://review.openstack.org/163522 | 16:26 |
| *** rfolco has quit IRC | 16:26 | |
| *** Longgeek has quit IRC | 16:26 | |
| clarkb | ctlaugh: yes, not having reporters for success and fail that talk to gerrit. we have a silent pipeline in our zuul config at https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n110 | 16:26 |
| clarkb | ctlaugh: it is otherwise the same as the check pipeline just doesn't report anything to gerrit | 16:26 |
| *** afazekas_ has quit IRC | 16:27 | |
| clarkb | looks like the latest release of keyring may have broken us? ok really need food now, back in a bit to continue debugging that | 16:28 |
| *** baoli has quit IRC | 16:28 | |
| *** jlanoux has quit IRC | 16:28 | |
| dhellmann | clarkb: let me check... | 16:30 |
| *** salv-orlando has joined #openstack-infra | 16:31 | |
| dhellmann | clarkb: yes, thanks! | 16:31 |
| clarkb | great it was just the cache then | 16:31 |
| asselin | ctlaugh, silent is the best way to go. You can also configure e-mail notifications. http://ci.openstack.org/zuul/zuul.html#smtp http://ci.openstack.org/zuul/reporters.html#smtp-configuration | 16:32 |
| *** garyk has quit IRC | 16:33 | |
| asselin | ctlaugh, you can pass in smtp host to zuul here: https://github.com/rasselin/os-ext-testing/blob/master/puppet/install_master.sh#L170 | 16:33 |
| ctlaugh | clarkb: asselin: excellent - thank you | 16:37 |
| *** rkukura has joined #openstack-infra | 16:38 | |
| *** baoli has joined #openstack-infra | 16:38 | |
| *** arxcruz has quit IRC | 16:39 | |
| *** tnovacik has quit IRC | 16:40 | |
| *** BharatK has quit IRC | 16:40 | |
| *** tkelsey has joined #openstack-infra | 16:41 | |
| clarkb | hrm looks like nodepool isn't getting job demand | 16:42 |
| pleia2 | cinerama: while you're about, can we chat quickly about the zanata patch? | 16:42 |
| *** rfolco has joined #openstack-infra | 16:43 | |
| cinerama | pleia2: you bet. btw, clarkb, saw your comments - thanks for the feedback@ | 16:43 |
| cinerama | for values of '@' that equal '!' | 16:43 |
| pleia2 | cinerama: I read the meeting minutes, thanks for providing an update | 16:43 |
| *** _qb has joined #openstack-infra | 16:43 | |
| pleia2 | cinerama: I think re: apache proxy we might want to hold off on that until a later patch (like with haproxy for git, we got the base git working first) | 16:44 |
| cinerama | pleia2: yep, i actually have a working patch in the wings ready to go | 16:44 |
| pleia2 | I kind of want to get most basic thing working initially, and things like that should be optional in this module anyway | 16:44 |
| pleia2 | great | 16:45 |
| *** jistr has quit IRC | 16:45 | |
| clarkb | nodepool is calculating demand just not showing very large deficits /me looks furhter up the chain | 16:45 |
| pleia2 | cinerama: and we'll have to think about clamav, the fact that it's a stressed part of the install docs gives me reason to pause, but it's not really a think we've done (as noted in the meeting) | 16:46 |
| *** tonytan4ever has quit IRC | 16:46 | |
| pleia2 | s/think/thing | 16:46 |
| clarkb | it appears that the zuul mergers may have not done a whole lot of work for a while? | 16:46 |
| cinerama | pleia2: in the meeting iirc we decided that for our purposes clamav wasn't really a so much thing because the content to be uploaded is already reviewed code | 16:47 |
| pleia2 | cinerama: plus, I think that's a 2nd patch kind of thing too :) like email (fwiw, we use exim, but postfix is more popular for ubuntu things as a default) | 16:47 |
| pleia2 | cinerama: ah yes, you're right | 16:47 |
| *** yamahata has quit IRC | 16:47 | |
| cinerama | the main thing i am having trouble with is just separating out what goes in the base module vs what goes in a site file, etc | 16:47 |
| clarkb | hrm looks like lots of nova merges which are slower | 16:48 |
| clarkb | is that merge-check then? | 16:48 |
| pleia2 | and regarding all the translations imports, AJaeger is the one who's familiar with all of our current scripts, so he's going to be the one to chat with about that (indeed, I think he volunteered to lead up the work there for Zanata) | 16:48 |
| cinerama | also i briefly started work on openid integration...so it's supported, but it doesn't get along with openstackid.org yet | 16:48 |
| pleia2 | cinerama: yeah, that's always tricky for us :\ | 16:48 |
| *** bdpayne has joined #openstack-infra | 16:48 | |
| *** markus_z has quit IRC | 16:48 | |
| *** yamamoto has quit IRC | 16:48 | |
| *** carl_baldwin has joined #openstack-infra | 16:48 | |
| clarkb | cinerama: even after fungi fixed the cert it still complains? | 16:49 |
| cinerama | having a more formal convention might be good here, both as a guideline for developers who want to help infra, and as a yardstick to review by | 16:49 |
| pleia2 | cinerama: it's kind of an evolving discussion for us, historically we've been bad at separating (all of our puppet modules were all in system-config until not too long ago) | 16:50 |
| cinerama | clarkb: yeah i was testing against openstackid-dev.o.o, which gives a different error. the returned xrds is slightly different from, say, login.lp.n, but i don't know enough about it to know whether that would make a difference | 16:50 |
| clarkb | cinerama: gotcha | 16:50 |
| cinerama | clarkb: i tried standing up my own openstackid to play around, but then i got lost in a twisty maze of stuff | 16:50 |
| pleia2 | fwiw, auth can be a "in the next patch" thing too | 16:50 |
| cinerama | pleia2: yes that is in a separate patch | 16:50 |
| SpamapS | hr | 16:51 |
| SpamapS | hrm | 16:51 |
| pleia2 | woo, on the same page \o/ | 16:51 |
| *** Somay has quit IRC | 16:51 | |
| clarkb | and I am happy to help with the translation import/export as I have some familiarity with the existing scripts too | 16:51 |
| SpamapS | zuul.openstack.org makes my firefox sad | 16:51 |
| pleia2 | SpamapS: how about status.openstack.org/zuul ? | 16:51 |
| pleia2 | actually, zuul.openstack.org kind of makes chrome sad too, seems to be loading up a lot of stuff | 16:52 |
| pleia2 | (I hadn't navigated to it in forever) | 16:52 |
| cinerama | clarkb: that would be awesome if you could help with that. it's kind of early days for me so i am just starting to think past the immediate goal of getting it up and running | 16:52 |
| *** Somay has joined #openstack-infra | 16:52 | |
| SpamapS | pleia2: yeah that one works fine | 16:52 |
| clarkb | pleia2: SpamapS its working here with chromium | 16:52 |
| SpamapS | actually no | 16:52 |
| SpamapS | that one also breaks in jquery | 16:52 |
| pleia2 | doh | 16:53 |
| SpamapS | making me stop/debug/wait | 16:53 |
| SpamapS | ah but at least that one eventually works | 16:53 |
| SpamapS | zuul.o.o just spins forever | 16:53 |
| cinerama | pleia2: it would be good if you could think about anything else we're missing that this will need - i want to have scope on it as early as possible even if we don't deliver in full all at once | 16:53 |
| *** ParsectiX has quit IRC | 16:54 | |
| pleia2 | cinerama: agreed, I'll work to make that happen by the end of the week (probably in a thread to the openstack-infra mailing list) | 16:54 |
| clarkb | cinerama: basically we have two sets of scripts. The first set run when code merges, they update the project's pot file (which contains a list of strings that need to be translated) and pushes that up into transifex. The second set run once a day and use the transifex api/client to pull any translations that have been made and propose them as a new patch in gerrit | 16:54 |
| *** ParsectiX has joined #openstack-infra | 16:54 | |
| clarkb | cinerama: so we basically need to take that second set and run them against transifex but instead of proposing them as a new patch in gerrit push them into zanata | 16:54 |
| clarkb | cinerama: let me get a link to those scripts | 16:55 |
| pleia2 | there's also the migration of the less translated (under 75%) transifex translations to the new system | 16:55 |
| clarkb | pleia2: ya thats the above | 16:55 |
| clarkb | we need to tweak the scripts to grab all translations too | 16:56 |
| cinerama | clarkb: ooh, sound like we need some machinery too | 16:56 |
| pleia2 | but I haven't put a lot of thought into the scripts, since I've been trying to focus on the server bit for now | 16:56 |
| pleia2 | (this is a big project, dealing with it in chunks helps me :)) | 16:56 |
| clarkb | cinerama: pleia2 https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts the set of propose_translation_update scripts | 16:57 |
| pleia2 | cinerama: also, please feel free to update the spec if things are unclear | 16:57 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/propose_translation_update.sh#n37 is the line to update (just remove the flag for minimum percentage) to get all the rtanslations | 16:57 |
| cinerama | pleia2: it's not unclear, but it could be more detailed possibly | 16:58 |
| pleia2 | cinerama: adding more details is good too :) | 16:58 |
| *** ihrachyshka has joined #openstack-infra | 16:58 | |
| *** mwagner_lap has quit IRC | 16:59 | |
| pleia2 | bbiab, appointment with painter | 16:59 |
| *** afazekas has joined #openstack-infra | 16:59 | |
| *** Somay has quit IRC | 17:03 | |
| clarkb | pretty sure at this point that a large number of nova merge jobs essentially backed up zuuls ability to do other work since the merger workers were all busy doing nova things | 17:03 |
| *** jlanoux has joined #openstack-infra | 17:04 | |
| *** Somay has joined #openstack-infra | 17:04 | |
| *** Ryan_Lane has joined #openstack-infra | 17:04 | |
| clarkb | which is basically as designed but maybe we need more resources, so back to debugging keyring install on static.o.o | 17:04 |
| *** thingee has joined #openstack-infra | 17:04 | |
| *** bhuvan has quit IRC | 17:04 | |
| *** dimtruck is now known as zz_dimtruck | 17:05 | |
| *** esker has joined #openstack-infra | 17:05 | |
| *** zz_dimtruck is now known as dimtruck | 17:06 | |
| *** harlowja_away is now known as harlowja_ | 17:06 | |
| clarkb | o_O manually running `sudo puppet agent --test` on static.o.o got it to install properly | 17:06 |
| *** Ala has quit IRC | 17:06 | |
| clarkb | I really have no idea why it failed under ansible puppet | 17:06 |
| clarkb | other than what the pip log said about it failing to run setup.py install | 17:07 |
| *** thingee has quit IRC | 17:07 | |
| *** yamahata has joined #openstack-infra | 17:09 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/devstack-gate: Add oslo.versionedobjects to devstack-vm-gate-wrap.sh https://review.openstack.org/163908 | 17:09 |
| *** esker has quit IRC | 17:10 | |
| *** woodster_ has quit IRC | 17:10 | |
| dhellmann | dims_: that's a dupe of https://review.openstack.org/#/c/162218/ which is having trouble landing apparently | 17:11 |
| fungi | cinerama: the cert chain on openstackid.org is at least no longer causing a problem though, right? | 17:11 |
| dims_ | dhellmann: ah thanks | 17:12 |
| cinerama | fungi: looks fine now. i've torn down my environment where i was working on openid stuff at the moment but i can check it later | 17:12 |
| dhellmann | dims_: it looks like my patch was hit by the unittest/testtools version conflict thing | 17:12 |
| fungi | cinerama: okay, thanks for confirming | 17:12 |
| dims_ | dhellmann: right, forgot about that one | 17:12 |
| dims_ | that should pass now | 17:12 |
| dhellmann | yep | 17:13 |
| *** Guest30394 has quit IRC | 17:13 | |
| cinerama | fungi: thanks for chasing that up...got to the point where us plebs couldn't quite work out what was happening :) | 17:15 |
| *** bswartz has joined #openstack-infra | 17:16 | |
| *** patrickeast has joined #openstack-infra | 17:16 | |
| *** tonytan4ever has joined #openstack-infra | 17:17 | |
| notmyname | clarkb: did you get any resolution to checking the delete-at header? | 17:19 |
| *** tonytan4ever has quit IRC | 17:21 | |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Log the job name info when running test mode https://review.openstack.org/163912 | 17:22 |
| *** melwitt has joined #openstack-infra | 17:22 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 17:25 | |
| *** devlaps has joined #openstack-infra | 17:26 | |
| *** apevec has quit IRC | 17:27 | |
| clarkb | notmyname: no | 17:28 |
| clarkb | jeblair: fwiw I think we have only leaked 3 ports so far | 17:28 |
| *** wuhg has quit IRC | 17:29 | |
| *** bhuvan has joined #openstack-infra | 17:29 | |
| *** jp_at_hp has quit IRC | 17:31 | |
| *** achanda has joined #openstack-infra | 17:32 | |
| *** ghostpl_ has quit IRC | 17:32 | |
| *** andreykurilin_ has joined #openstack-infra | 17:33 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:33 | |
| *** mtanino has joined #openstack-infra | 17:34 | |
| SpamapS | rcarrillocruz: Whats going on with the shade func tests? I thought you were adding some debug help? | 17:35 |
| clarkb | jeblair: I think zuul maybe merging in repo git tree deps directly rather than just merging the HEAD which brings along the parents | 17:36 |
| clarkb | jeblair: and that is what is causing the oscilations in zuul with the nova changes currently in check | 17:37 |
| *** e0ne has quit IRC | 17:37 | |
| clarkb | jeblair: seeings lots of merges across several mergers at around 1730 for 163356 | 17:37 |
| clarkb | functionally it is correct but its slow | 17:37 |
| *** amitgandhinz has joined #openstack-infra | 17:38 | |
| *** Somay has quit IRC | 17:39 | |
| *** cody-somerville has quit IRC | 17:40 | |
| *** woodster_ has joined #openstack-infra | 17:41 | |
| *** andreykurilin_ has quit IRC | 17:41 | |
| *** mpavone has quit IRC | 17:41 | |
| *** yfried has joined #openstack-infra | 17:42 | |
| *** tonytan4ever has joined #openstack-infra | 17:43 | |
| jesusaurus | fungi: I just discovered that adding the notes fetch line to ~/.gitconfig causes a bad remote to get created on clone, which breaks 'git review -s' | 17:43 |
| SpamapS | You know, why don't we just use zmq in our cloud? | 17:43 |
| SpamapS | I know we lurve gearman. | 17:43 |
| SpamapS | But zmq is already written. Maybe we can just help improve it. | 17:44 |
| clarkb | SpamapS: I missed the discussion wher we said no rabbit | 17:44 |
| clarkb | also zmq is :( | 17:44 |
| clarkb | I would not use it again for anything | 17:44 |
| SpamapS | clarkb: can you sumarize :( in 140 characters? | 17:45 |
| SpamapS | or less | 17:45 |
| fungi | jesusaurus: oh, great point, git-review automatically assumes it doesn't need to create that remote any longer. could probably just name it something other than gerrit | 17:45 |
| SpamapS | clarkb: the discussion on no rabbit was guided by the same things sdague mentioned in the ops summary. Operating it is painful. | 17:45 |
| clarkb | SpamapS: it completely hides all error conditions from you then fails to act on errors properly | 17:45 |
| SpamapS | clarkb: is that by design, or a bad python implementation? | 17:46 |
| clarkb | SpamapS: by design (its the C lib that does it has nothing to do with python) | 17:46 |
| SpamapS | Like, we had the same thing with the python gearman lib right? | 17:46 |
| sdague | right, so during the ops session on rabbit basically it was 'rabbit is bad, everything else is worse' | 17:46 |
| SpamapS | sdague: Right, and I'm worried that gearman could be a second system, while zmq might be something that could be improved. | 17:47 |
| jesusaurus | fungi: but isnt that telling git which remote to fetch from? | 17:47 |
| clarkb | SpamapS: so if it breaks in a way it can't recover it does not tell you and it is very hard to discover that it has broken. Basically oyu learn a day later when none of your services work despite being happily green | 17:47 |
| clarkb | SpamapS: so then you restart everything because zmq | 17:47 |
| SpamapS | but it sounds like zmq might just be fundamentally bad, not poorly maintained or misunderstood. | 17:47 |
| sdague | SpamapS: zmq has no HA story iirc | 17:47 |
| *** skolekonov has quit IRC | 17:47 | |
| SpamapS | sdague: zmq is inherently HA because there is no SPOF. | 17:47 |
| clarkb | sdague: correct you have to implement that on top of it | 17:47 |
| jesusaurus | fungi: it looks like a fresh clone creates a gerrit remote with no uri | 17:47 |
| clarkb | SpamapS: no :/ | 17:47 |
| clarkb | SpamapS: thats not how HA works | 17:47 |
| *** dustins has joined #openstack-infra | 17:47 | |
| fungi | jesusaurus: ahh, right. probably have to duplicate a little extra data to make that work | 17:47 |
| clarkb | SpamapS: you would have to implement your own retries and so on | 17:48 |
| fungi | jesusaurus: like add connection details to the remote itself | 17:48 |
| clarkb | SpamapS: which is hard beacuse zmq doesn't tell you when it breaks | 17:48 |
| greghaynes | SpamapS: It might be better nowdays, but back when we were getting salt to operate using zmq there was a lot of issues where messages would get accepted and then go into a black hole and never be reported as such. Then there was a lot of finger pointing between zmq and the pyzmq folks and both ended up being at fault | 17:48 |
| fungi | jesusaurus: at which point i guess you've devolved into needing to know repo-specific details | 17:48 |
| SpamapS | zomg wow | 17:48 |
| greghaynes | this happened on like 3 separate occasions with 3 separate bugs | 17:48 |
| SpamapS | ok now I see | 17:48 |
| fungi | jesusaurus: this is actually something we can probably solve in git-review (like have it auto-add the notes fetch, perhaps make that a configurable option) | 17:49 |
| sdague | it was actually a really fascinating session to be in, and see all the work arounds that folks have had | 17:49 |
| SpamapS | clarkb: I do think HA works if you eliminate all the spofs. It sounds like it's not HA that is lacking, but basic reliability of individual components. | 17:49 |
| *** bhuvan has quit IRC | 17:49 | |
| SpamapS | I'd expect a library purporting to be about simplifying async comms to be crazy pedantic about errors. | 17:50 |
| jesusaurus | fungi: oh good idea | 17:50 |
| SpamapS | Sounds like it's the opposite. | 17:50 |
| greghaynes | yep. There was a continual "next major version will be better about this" | 17:51 |
| fungi | jesusaurus: though i should also test just overriding origin. i'm pretty sure the notes get replicated so they don't _have_ to be fetched directly from gerrit | 17:51 |
| clarkb | SpamapS: eliminating spofs does not imply HA | 17:51 |
| SpamapS | sdague: I quite enjoyed the cron-to-restart-the-things comment. It's like Windows NT all over again. | 17:51 |
| sdague | out of curiosity, what would the gear story be here? Because based on the event drops I've seen on zuul restarts that seems equally problematic on the HA side | 17:51 |
| clarkb | sdague: yes you would have to implement your own retries in gear | 17:51 |
| SpamapS | clarkb: If you have no spof, what challenges your availability? | 17:51 |
| clarkb | sdague: but gear will at least tell you when it fails | 17:51 |
| jesusaurus | fungi: oh? do the notes get replicated out to the mirrors? i assumed they didnt | 17:52 |
| clarkb | SpamapS: error handling | 17:52 |
| clarkb | SpamapS: if I don't have a single point of failure but they can't handle errors then what good is that | 17:52 |
| clarkb | SpamapS: my classic example is network load balancer arp storm explosion. There is no SPOF but also no HA | 17:53 |
| SpamapS | clarkb: I have not ever included that in my discussions of HA. If something is unreliable as a standalone component, that is just plain suckage. | 17:53 |
| *** claudiub has quit IRC | 17:54 | |
| sdague | SpamapS: ummm... well from what I can gather that's a probably with a ton of this software space | 17:54 |
| *** bhuvan has joined #openstack-infra | 17:54 | |
| sdague | s/probably/problem/ | 17:54 |
| SpamapS | clarkb: yeah, cascading fail is indeed the one area where 0 SPOF can still mean a failure to be available. Good point. :) | 17:55 |
| *** _nadya_ has quit IRC | 17:55 | |
| *** dpyzhov_ has quit IRC | 17:55 | |
| *** sputnik13 has joined #openstack-infra | 17:55 | |
| sdague | so... http://kafka.apache.org/ was also brought up in discussion a little bit. No idea if that's a more fruitful target than gear. | 17:55 |
| SpamapS | But I think if you have a standalone component that handles errors poorly and is unreliable, I put that in a bucket further away from desirability than "it isn't HA". That's in the "It isn't worth considering." bucket. | 17:55 |
| sdague | I do know that the fanout queues get used enough that having to implement rebroadcase on top of gear might be challenging | 17:56 |
| greghaynes | ++ on the kafka, just pretty out of scope for the infra cloud | 17:56 |
| clarkb | sdague: ya I think kafka is likely a good alternative | 17:56 |
| SpamapS | sdague: fanout isn't the same thing as broadcast. Or is it? | 17:56 |
| clarkb | sdague: especially with the reliability requirements that the openstack projects expect | 17:56 |
| *** dizquierdo has quit IRC | 17:56 | |
| SpamapS | To me fanout means "you get a job. Now you get a different job. Now you get a different job." No? | 17:56 |
| fungi | jesusaurus: here's my ~/.gitconfig and it seems to work as intended http://paste.openstack.org/show/191887/ | 17:57 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:57 | |
| sdague | SpamapS: honestly, you're getting me a bit out of my depth in all the details of this part | 17:57 |
| SpamapS | sdague: mmk. :) | 17:57 |
| mordred | I like gear over kafka purely because we currently very successfully run gear at scale | 17:57 |
| SpamapS | Because that is one thing gearman is crazy good at (job distribution) | 17:57 |
| mordred | which is not to say that kafka wouldn't work | 17:57 |
| jesusaurus | fungi: looks like you can fetch the notes from the git mirrors, so using origin Just Worksâ„¢ | 17:57 |
| mordred | but we _know_ gear works well | 17:57 |
| fungi | jesusaurus: when i clone and remote update/pull/fetch it still does the usual | 17:58 |
| sdague | mordred: well, for a particular application | 17:58 |
| mordred | and until I know the same about kafka, it will remain the new shiny hipster toy to me | 17:58 |
| mordred | sdague: sure | 17:58 |
| mordred | not saying it's the right or wrong choice for openstack | 17:58 |
| sdague | which doesn't require reliable messaing | 17:58 |
| *** cdent has quit IRC | 17:58 | |
| sdague | and I agree, kafka is completely unknown here as well | 17:59 |
| *** SumitNaiksatam has joined #openstack-infra | 17:59 | |
| SpamapS | sdague: ok I was wrong, and keep getting this wrong. Fanout == broadcast. | 18:00 |
| *** annegentle has quit IRC | 18:00 | |
| greghaynes | thats what it means in rmq land also | 18:00 |
| *** yfried is now known as yfried|afk | 18:00 | |
| *** bhuvan has quit IRC | 18:00 | |
| *** purp is now known as purp_away | 18:00 | |
| fungi | jesusaurus: and after a remote update in any existing clones i also now have notes in my git log et cetera | 18:01 |
| SpamapS | fanout seems to be used in 3 places in nova | 18:01 |
| SpamapS | 1: scheduler for resource updates | 18:01 |
| *** katyafervent is now known as katyafervent_awa | 18:01 | |
| SpamapS | 2: cells for all the things | 18:01 |
| jeblair | clarkb: were you suggesting that the factorial merges from a long nova series was keeping the mergers busy enough that we weren't launching jobs in a timely manner? | 18:01 |
| SpamapS | 3: nova-network for ... I refuse to spend any time caring about nova network. | 18:01 |
| clarkb | jeblair: yes | 18:02 |
| clarkb | jeblair: I haven't been able to confirm this by reading code but the zm0X logs seem to imply this is the case | 18:02 |
| jesusaurus | fungi: yep, I'm seeing the same thing after changing "gerrit" to "origin" | 18:02 |
| clarkb | jeblair: note the building spikes in the node graph, that should be relatively smooth | 18:02 |
| *** derekh has quit IRC | 18:03 | |
| jeblair | clarkb: i believe it will be the case, yes, though i would not necessarily expect the merging of the parents to take long, that should be very simple. | 18:03 |
| *** dims__ has joined #openstack-infra | 18:03 | |
| jeblair | clarkb: my guess is that a nova tree reset takes 30-60 seconds at this point, and actually merging the commit should be very fast. let me investigate. | 18:03 |
| SpamapS | sdague: so anyway, for our cloud, we're not using cells, we're not using nova network, and we're not using host aggregates (the piece of the scheduler that uses fanout). So we might not even get a chance to find out if our fanout sucks. | 18:03 |
| *** dims__ has quit IRC | 18:03 | |
| *** AJaeger has joined #openstack-infra | 18:04 | |
| clarkb | jeblair: ok, anything else I should be looking at to help? | 18:04 |
| *** katyafervent_awa is now known as katyafervent | 18:04 | |
| *** dims__ has joined #openstack-infra | 18:04 | |
| jeblair | clarkb: i don't think so | 18:04 |
| jeblair | it's recovered now, so... | 18:04 |
| *** koolhead17 has quit IRC | 18:05 | |
| AJaeger | clarkb, could you review https://review.openstack.org/#/c/163884/ , please? That improves our gating on project-config | 18:05 |
| sdague | SpamapS: yeh, especially as neutron beats up the message bus more than nova if I understand it correctly | 18:05 |
| clarkb | AJaeger: ya I will dive into reviews now that jeblair is looking at the zuul thing | 18:05 |
| *** dpyzhov_ has joined #openstack-infra | 18:05 | |
| AJaeger | clarkb: thanks! | 18:05 |
| *** mwagner_lap has joined #openstack-infra | 18:06 | |
| *** jlanoux has quit IRC | 18:06 | |
| clarkb | huh sys.exit(True) has a return code of 1 | 18:07 |
| *** dims_ has quit IRC | 18:07 | |
| clarkb | I guess True is coerced to an int value? | 18:07 |
| jeblair | clarkb: the individual changes are not fast. | 18:07 |
| *** esker has joined #openstack-infra | 18:07 | |
| clarkb | jeblair: right but its overdoing them. Shouldn't it only do one per check queue item? instead it appears to do all of them | 18:08 |
| AJaeger | clarkb: yeah... | 18:08 |
| clarkb | notmyname: still around? sorry I keep getting distracted. Is there a simple way to check the X-Delete-At value for a specific object? | 18:09 |
| SpamapS | sdague: analyzing neutron now, looks like it uses fanout for security groups updates.. which is.. interesting | 18:09 |
| jeblair | clarkb: if you upload a stack of 8 changes, there will be 8 merge jobs, one for each change. the first will merge A. the second A,B, then A,B,C, etc. | 18:09 |
| *** dims__ has quit IRC | 18:09 | |
| SpamapS | ah and for DHCP agent updates | 18:09 |
| SpamapS | which makes sense since the DHCP agent is HA | 18:09 |
| clarkb | jeblair: yes 8 merge jobs but each job is running a merge for each change instead of just merging the latest one | 18:10 |
| SpamapS | (or.. is it.... #letsNotDebateHATOdaykthxbai) | 18:10 |
| sdague | SpamapS: heh | 18:10 |
| clarkb | jeblair: I think we can optimize out all of those unnecessary merges if the git tree already reflects them | 18:10 |
| *** dpyzhov_ has quit IRC | 18:10 | |
| mordred | SpamapS: well, we won't do many security group updates | 18:10 |
| jeblair | clarkb: correct. give me some more time to analyze this :) | 18:10 |
| mordred | SpamapS: but dhcp agent might get updated more than occasionally | 18:10 |
| SpamapS | mordred: right, but we will do 2x DHCP updates per boot | 18:10 |
| mordred | yup | 18:10 |
| SpamapS | and 2x per delete | 18:10 |
| SpamapS | well maybe 1x per boot and delete.. not sure | 18:11 |
| mordred | SpamapS: fanout is "send message A to every recipient" right? | 18:11 |
| clarkb | wait why does dhcp need any updates like that? | 18:11 |
| SpamapS | mordred: correct | 18:11 |
| clarkb | node boots, it asks for IP, some dhcp server will win and respond | 18:11 |
| *** Longgeek has joined #openstack-infra | 18:11 | |
| *** otter768 has joined #openstack-infra | 18:11 | |
| clarkb | node is deleted, short lease expires a minute later all is happy | 18:11 |
| SpamapS | clarkb: so you can have two capable of responding? | 18:11 |
| mordred | clarkb: the dhcp server has to be told what IP to give to the vm that just booted | 18:11 |
| *** annegentle has joined #openstack-infra | 18:12 | |
| clarkb | mordred: wait is that how its implemented but why | 18:12 |
| clarkb | the whole point with dhcp in a dynamic environment is to not worry about that | 18:12 |
| mordred | clarkb: beacuse neutron owns the ips - and you can request via api call of neutron what to do with them | 18:12 |
| SpamapS | oh now this is even more interesting | 18:12 |
| mordred | clarkb: no, that's not the whole point of it | 18:12 |
| clarkb | mordred: no neutron owns the networks | 18:12 |
| SpamapS | Neutron dosn't use oslo.messaging's built in notifier thingy, but sends fanouts on the RPC bus | 18:12 |
| mordred | clarkb: the whoel point is a standard mechanism to communicate this information to clients | 18:12 |
| mordred | clarkb: the information may be statically or dynamically configured on the DHCP server side | 18:13 |
| clarkb | mordred: thats really easy to determine, check arp | 18:13 |
| mordred | like, you may be saying "give me a random ip from the range" | 18:13 |
| clarkb | or ask libvirt | 18:13 |
| mordred | clarkb: or you may be saying "give me the IP that my MAC Address is mapped to" | 18:13 |
| mordred | because your machine's mac address may have an allocated IP | 18:13 |
| clarkb | basically you make this a whole lot more reliable if you let it converge to where it goes rather than deciding upfront where it should go | 18:13 |
| mordred | that doesn't change over time | 18:13 |
| SpamapS | clarkb: I'm not sure what you're arguing for. Neutron has information about which address each MAC should have. This is a useful thing. We don't want dynamic allocation at the DHCP server level. | 18:14 |
| mordred | SpamapS: ++ | 18:14 |
| clarkb | SpamapS: I am saying neutron should not assign IPs like that | 18:14 |
| clarkb | SpamapS: DHCP should assign it | 18:14 |
| mordred | clarkb: it is | 18:14 |
| mordred | clarkb: DHCP is assigning what neutron tells it to | 18:14 |
| SpamapS | and we should poll DHCP for the current state of the world? | 18:14 |
| clarkb | mordred: but only assigning what neutron has told it | 18:14 |
| mordred | it HAS to work that way | 18:14 |
| notmyname | clarkb: ya, I'm getting distracted too. checking now | 18:14 |
| clarkb | no it does not need to work that way | 18:14 |
| *** koolhead17 has joined #openstack-infra | 18:14 | |
| * clarkb has built a large cloud thing that absolutely did not work this way | 18:14 | |
| mordred | clarkb: "neutron create-network A ; nova boot --network=A" | 18:14 |
| mordred | clarkb: did your network have API-driven network config? | 18:15 |
| clarkb | mordred: yes that should send a dhcp request on the l2 broadcast addr | 18:15 |
| *** tkelsey has quit IRC | 18:15 | |
| *** annegentle has quit IRC | 18:15 | |
| mordred | clarkb: yes. which is what it does | 18:15 |
| clarkb | mordred: then dhcp servers fight to assign an ip addr for the ips that live on that l2 netowrk | 18:15 |
| mordred | clarkb: right | 18:15 |
| SpamapS | clarkb: and how did your clients determine what address their box got? | 18:15 |
| clarkb | mordred: this IP can be arbitrary as long as it is on the network that belongs to the l2 net | 18:15 |
| mordred | clarkb: how did the client mac get mapped to the right network? | 18:15 |
| *** annegentle has joined #openstack-infra | 18:15 | |
| *** baoli has quit IRC | 18:16 | |
| SpamapS | some kind of REST API in front of all the DHCP servers providing a union of all the address assignments? | 18:16 |
| mordred | clarkb: and also SpamapS wuestion | 18:16 |
| clarkb | mordred: because your broadcast comes in on interfaces that can be assigned to networks | 18:16 |
| clarkb | mordred: SpamapS one thing at a time | 18:16 |
| clarkb | so you get l2 broadcast on interface foo for l2 network foo. you then give response ack for IP on l3 network foo | 18:16 |
| *** otter768 has quit IRC | 18:16 | |
| fungi | jesusaurus: are you overriding core.notesRef to refs/notes/review like i'm doing, or is there a saner solution to the fact that gerrit's not stashing its notes in the normal tree? | 18:16 |
| clarkb | VM says awesome my IP is now foo[ip] | 18:16 |
| clarkb | then later when you want to say what is VM's ip address you ask arp | 18:17 |
| clarkb | done | 18:17 |
| SpamapS | who asks arp? | 18:17 |
| clarkb | SpamapS: your control plane in this case it would be neutron | 18:17 |
| notmyname | clarkb: ah, I think I see the issue | 18:17 |
| *** achanda has quit IRC | 18:17 | |
| clarkb | SpamapS: mordred but that is completely ditributed it converges to a stable state and does not have any broadcast rpc issues in the control plane | 18:17 |
| clarkb | SpamapS: mordred it turns out that netowrk engineers from the 80s have solved these problems | 18:17 |
| SpamapS | clarkb: yeah, I don't think that is any better than how neutron does it today, which is to just have neutron tell the dhcp server what mac<->ip mapping to use. | 18:17 |
| *** dprince has quit IRC | 18:18 | |
| mordred | SpamapS: no, I agree with clarkb that his is better | 18:18 |
| clarkb | SpamapS: thats a lot less reliable because it requires you to have multiple pieces of software all agree with each other in lockstep | 18:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Don't check out a commit if it is already checked out https://review.openstack.org/163922 | 18:18 |
| jeblair | clarkb: ^ | 18:18 |
| clarkb | jeblair: thanks looking | 18:18 |
| mordred | SpamapS: but I do not think that clarkb's solution solves all of the use cases neutron is solving | 18:18 |
| clarkb | notmyname: cool | 18:18 |
| notmyname | clarkb: `swift stat <cont> <obj>` reports the x-delete-at value. internally, an x-delete-after is translated to x-delete-at (a definitive timestamp). so you can get x-delete-at, but you won't see x-delete-after | 18:18 |
| SpamapS | mordred: hah, well I think we can agree we don't have most of the use cases neutron solves. :) | 18:18 |
| notmyname | clarkb: and I think it's a good idea that you're setting it with x-delete-after | 18:18 |
| mordred | SpamapS: totally | 18:18 |
| clarkb | notmyname: I don't even get x-delete-at though, does that mean I am not setting either? | 18:19 |
| clarkb | notmyname: its entirely possible we aren't setting this properlywhich I am trying to confirm | 18:19 |
| notmyname | clarkb: x-delete-after is seconds until it's deleted. x-delete-at is a time to delete it at | 18:19 |
| SpamapS | clarkb: ok, so I agree, you have a lower consistency requirement on the frontend, and thus gain scale. I do agree with that. | 18:19 |
| notmyname | clarkb: ah. maybe | 18:19 |
| jeblair | clarkb: i know this isn't exactly what you were thinking, but let me explain my thoughts. here's the log from a recent big merge: http://paste.openstack.org/show/191893/ | 18:19 |
| notmyname | clarkb: it's working on my box :-) | 18:19 |
| mordred | clarkb: I was going to argue that the reason you want to have neutron tell the dhcp server what the mapping is was so that you can handle flloating-ips, which are api-assigned ips to server mappings | 18:19 |
| clarkb | notmyname: ok :) I think that means we did something wrong. thank you for confirming | 18:19 |
| mordred | clarkb: when I Remembered that floating ips don't dhcp | 18:19 |
| mordred | clarkb: because everyone is crazy | 18:19 |
| clarkb | mordred: :) | 18:19 |
| SpamapS | clarkb: except I very often want to assign an exact IP to a mac, which requires that we do things the way we have now AFAICT. | 18:19 |
| mordred | clarkb: so - I'm on board with you being right | 18:19 |
| mordred | SpamapS: no - but we dont' reallyd o that | 18:20 |
| mordred | SpamapS: see above re: floating-ips | 18:20 |
| mordred | SpamapS: that's teh only mechanism in openstack to assign a specific ip to a specific mac | 18:20 |
| mordred | SpamapS: and we do that wrong too | 18:20 |
| SpamapS | mordred: Yes, yes we do. :-/ | 18:20 |
| mordred | SpamapS: because we do that with firewall rules, aiui | 18:20 |
| *** [HeOS] has quit IRC | 18:20 | |
| *** yfried|afk is now known as yfried | 18:20 | |
| SpamapS | Though you can actually cheat and get specific assignment if you can define your own subnets. | 18:20 |
| clarkb | mordred: yes its iptables NAT | 18:20 |
| SpamapS | because you can make 1 IP subnets. | 18:20 |
| mordred | when that SHOULD be done with dhcp - because of the reasons | 18:20 |
| jeblair | clarkb: if you look at it, it's the "checking out" part that takes a long time. that's doing a working tree reset. but it's resetting to the commit that it just created, so it shouldn't actually have to do anything. yes, it's true that in the case of a git-parent, we could just check out the last thing in the tree, which is what we used to do, but i think it's much simpler if we can just be naive about it and treat git and commit-message | 18:21 |
| SpamapS | But that's crazy sauce anyway. ;) | 18:21 |
| mordred | clarkb: you know - neutron I hear is crazy pluggable | 18:21 |
| clarkb | jeblair: oh is this related to the change I merged in zuul to do a reset? | 18:21 |
| mordred | clarkb: we could write a proper dhcp/arp based neutron plugin | 18:21 |
| clarkb | jeblair: it made sense for the zuul-cloner but I didn't consider if it would break the merger workers | 18:21 |
| mordred | clarkb: that also does floating-ips but does them via dhcp too | 18:21 |
| mordred | clarkb: and only uses neutron to configure broad-level dhcp network ranges and whatnot | 18:22 |
| SpamapS | That part probably isn't as pluggable as you'd need. | 18:22 |
| mordred | SpamapS: I believe _everything_ in neutron is pluggable, no? | 18:22 |
| SpamapS | No | 18:22 |
| greghaynes | clarkb: this is a bit beyond my networking fu, but if youre using something like DVR are you sure to have a consistent arp table? | 18:22 |
| mordred | SpamapS: I betcha I could write a patch to add a plugin point | 18:22 |
| mordred | greghaynes: you dont need DVR with this | 18:22 |
| greghaynes | yes, we done | 18:22 |
| greghaynes | er, we dont | 18:22 |
| clarkb | greghaynes: if arp is inconsistent then nothing will work | 18:22 |
| jeblair | clarkb: so i think as long as we don't do those checkouts, the actual git merge (which takes 0.2 seconds) will be a trivial amount of extra work. also, it is possible that later merges can re-use the refs for changes created halfway through the stack. so it may not be completely wasted (but probably is) | 18:22 |
| sdague | maybe the neutron is wrong conversation might be better had in the neutron channel with neutron folks? | 18:22 |
| clarkb | greghaynes: so I think by definition arp must be consistent | 18:22 |
| openstackgerrit | Merged openstack-infra/project-config: New project: cue-dashboard to stackforge https://review.openstack.org/162804 | 18:22 |
| greghaynes | clarkb: I mean what if two nodes only have partial view of the world | 18:23 |
| *** che-arne has quit IRC | 18:23 | |
| mordred | sdague: nah - beacuse we'd jsut be yelling at them | 18:23 |
| SpamapS | mordred: L2 network management is highly pluggable. The rest is pretty rigid, and DHCP looks to be almost hard coded. | 18:23 |
| clarkb | greghaynes: thats not how l2 networks work | 18:23 |
| *** achanda has joined #openstack-infra | 18:23 | |
| jeblair | clarkb: i don't think it's related to that change, and i don't think we've restarted the mergers since then? | 18:23 |
| clarkb | greghaynes: even when virtualized every node on an l2 network should see the others via arp whohas | 18:23 |
| *** tsg_ has joined #openstack-infra | 18:23 | |
| openstackgerrit | Max Rydahl Andersen proposed openstack-infra/jenkins-job-builder: Add support for 'raw' to inject direct xml to job configs. https://review.openstack.org/160158 | 18:23 |
| jeblair | clarkb: also, you are having _three_ conversations right now, right? :) | 18:23 |
| clarkb | jeblair: oh good point on restarting the workers would've happened on the 4th | 18:23 |
| mordred | jeblair: he's good like that | 18:23 |
| greghaynes | clarkb: arp replys are directed to a single mac IIRC? | 18:23 |
| clarkb | jeblair: yes sorry my brain is about to melt | 18:23 |
| *** achanda has quit IRC | 18:23 | |
| mordred | jeblair: we made him rage-converse :) | 18:23 |
| clarkb | greghaynes: its a broadcast send then response unicast | 18:23 |
| greghaynes | hrm, so yes, per l2 | 18:24 |
| *** achanda has joined #openstack-infra | 18:24 | |
| sdague | mordred: maybe, but it would also mean folks that actually know neutron internals would be able to comment about which parts are factual and which aren't instead of speculation | 18:24 |
| mordred | jeblair: he's so correct though that I almost want to write the patch to neutron to enable this | 18:24 |
| mordred | sdague: bah. see, now you're talking about being productive | 18:24 |
| SpamapS | the speculation right now is that fanout would suck w/ gearman | 18:24 |
| jeblair | mordred: you'll probably write it faster than clarkb since he's busy talking | 18:24 |
| SpamapS | it actually might be incredibly awesome. :) | 18:25 |
| clarkb | jeblair: ha | 18:25 |
| * jeblair goes back to read two other conversations | 18:25 | |
| notmyname | clarkb: I'm told that our new feature branch (feature/crypto) isn't having patches reported in the channel. is there a config somewhere that I need to tickle to get that to happen? | 18:25 |
| mordred | clarkb: right now, I want max's threaded IRC thing ... | 18:25 |
| *** ssam2 has quit IRC | 18:25 | |
| greghaynes | mordred: lol, its up | 18:25 |
| clarkb | notmyname: yes, we need to add that branch to the list of branches to report in the bot config | 18:25 |
| greghaynes | mordred: its just not open sauce | 18:26 |
| mordred | greghaynes: oh. well, I have no interest then | 18:26 |
| clarkb | notmyname: willget a link for you | 18:26 |
| notmyname | clarkb: ah, thanks. what's the repo/file? ya, a like would be swell | 18:26 |
| greghaynes | mordred: yea, I gave a vigorous fist shaking at that | 18:26 |
| SpamapS | anyway, to summarize: dynamic allocation done all on dhcpd's side: #win ; static allocation done without NAT: #win | 18:26 |
| SpamapS | go win | 18:26 |
| clarkb | notmyname: https://git.openstack.org/cgit/openstack-infra/project-config/tree/gerritbot/channels.yaml#n974 update the branches list there | 18:26 |
| *** afazekas has quit IRC | 18:26 | |
| notmyname | clarkb: thanks | 18:26 |
| mordred | SpamapS: I agree with your tl;dr | 18:27 |
| SpamapS | so, reliable delivery in gearman is now something I hadn't really considered but needs considering. | 18:28 |
| SpamapS | it's actually pretty hard to know, just looking at call sites, if things require reliable delivery or not | 18:29 |
| mordred | SpamapS: I think the usual pattern is for the client to maintain context on the payload and to re-submit if the server goes away | 18:29 |
| SpamapS | which means we'd have ot assume they always do | 18:29 |
| mordred | SpamapS: no? | 18:29 |
| SpamapS | mordred: yes | 18:29 |
| mordred | which is also how multi-master gearman-server for HA works | 18:29 |
| SpamapS | mordred: and we can do that all deep within oslo.messaging, I think. | 18:29 |
| jeblair | SpamapS: so the interface to o.m is "here, deliver this message", and o.m makes a gearman job, submits it, resubmits it if needed, etc? | 18:30 |
| SpamapS | but I think there are bits that expect .cast(...) to return immediately, but that the message will eventually be delivered come hell or high water. | 18:30 |
| openstackgerrit | John Dickinson proposed openstack-infra/project-config: gerritbot report on swift's feature/crypto branch https://review.openstack.org/163927 | 18:30 |
| SpamapS | jeblair: right | 18:30 |
| notmyname | clarkb: ^ | 18:30 |
| SpamapS | so we might have to write a store-and-forward worker | 18:30 |
| *** yfried is now known as yfried|afk | 18:30 | |
| SpamapS | which is actually really easy to write and I've written before. :) | 18:30 |
| mordred | jeblair, SpamapS: this is almost going to be like me saying AFS is the answer to everything ... | 18:30 |
| mordred | jeblair, SpamapS: but tell me why Cyrus IMAPd isn't the right answer? | 18:31 |
| *** yfried|afk is now known as yfried | 18:31 | |
| SpamapS | hah, store the message on AFS. ;) | 18:31 |
| *** Longgeek has quit IRC | 18:31 | |
| *** ujuc has joined #openstack-infra | 18:31 | |
| clarkb | jeblair: your change lgtm and +1 to being naive and simple if possible | 18:31 |
| mordred | since you want "hey, deliver this message kthxbai" | 18:31 |
| SpamapS | mordred: well that would be exim + Cyrus IMAPd right? ;) | 18:31 |
| mordred | SpamapS: nope | 18:31 |
| mordred | SpamapS: have oslo.messaging do direct message injection | 18:31 |
| SpamapS | oh just pop it into INBOX | 18:31 |
| mordred | no need for actual smtp mail routing | 18:31 |
| mordred | yah | 18:32 |
| mordred | OR | 18:32 |
| mordred | to a "shared queue" | 18:32 |
| mordred | for things that need to be seen by multiple people | 18:32 |
| mordred | which would be a shared mailbox | 18:32 |
| jeblair | funny thing -- cyrus does not actually work on afs due to memmap and range locking | 18:32 |
| SpamapS | mordred: on shared mailboxes, does each account get their own record of read/unread ? | 18:32 |
| * mordred may have distracted with AFS mention | 18:32 | |
| mordred | SpamapS: I dunno - jeblair ? | 18:32 |
| SpamapS | if not, then you really need nntp for fanout.. and yeah, I'm trolling myself at this point. | 18:33 |
| *** vhoward has joined #openstack-infra | 18:33 | |
| jeblair | SpamapS: i believe so, lemme double check | 18:33 |
| mordred | SpamapS: beacuse with a Cyrus Murder - you get HA - and you get scalability | 18:33 |
| *** koolhead17 has quit IRC | 18:34 | |
| nibalizer | jedimike: yep its not really used just yet i think | 18:34 |
| SpamapS | mordred: don't you get a lot of IO per message too? | 18:34 |
| nibalizer | and we will publish all the modules under openstackinfra so thats why the change | 18:34 |
| nibalizer | gotta roll some updates through that | 18:34 |
| mordred | SpamapS: maybe - but it's scalable to VERY VERY large | 18:34 |
| notmyname | clarkb: jeblair: will that be available as soon as it lands, or does it take a bot/gerrit restart? | 18:34 |
| mordred | SpamapS: and not insane to operate | 18:34 |
| clarkb | notmyname: it takes a bot restart which is automated by puppet as soon as it runs on that node | 18:34 |
| *** esker has quit IRC | 18:34 | |
| SpamapS | mordred: well a lot of IO per message may be unavoidable if reliable delivery is really truly necessary. | 18:35 |
| notmyname | clarkb: cool. thanks | 18:35 |
| mordred | SpamapS: yup | 18:35 |
| *** dims has joined #openstack-infra | 18:35 | |
| mordred | SpamapS: in fact, that's one of the rabbit problems I hear | 18:35 |
| mordred | is that you get message loss in an HA event | 18:35 |
| mordred | so - you know - it's lying about guaranteed delivery | 18:35 |
| SpamapS | but I'm kind of wondering if it is. I'd bet 90% of the time it isn't, and we would do well to annotate that in call sites for all of the messaging backends to consume. | 18:35 |
| mordred | SpamapS: ++ | 18:35 |
| sdague | mordred: so it sounded more like the major issue wasn't that | 18:35 |
| mordred | sdague: sure - but that _Was_ an issue that got brought up | 18:35 |
| sdague | but that in an HA event something doesn't reconnect correctly | 18:35 |
| mordred | sdague: that too | 18:36 |
| mordred | yeah | 18:36 |
| mordred | being the main one | 18:36 |
| sdague | so rabbit believes it's sending things | 18:36 |
| sdague | but no one is getting them | 18:36 |
| SpamapS | mordred: yes thats why the recommended method of failover for RabbitMQ is active/passive with shovel if you care about losing messages. | 18:36 |
| *** claudiub has joined #openstack-infra | 18:36 | |
| SpamapS | But like sdague said, people aren't feeling that problem. | 18:36 |
| * mordred is going to laugh if we wind up writing an IMAP plugin for oslo.messaging | 18:36 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Add a python-ironicclient src job https://review.openstack.org/163632 | 18:37 |
| *** yolanda has quit IRC | 18:37 | |
| sdague | ``Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.'' | 18:37 |
| *** yolanda has joined #openstack-infra | 18:38 | |
| dims | a.k.a the "Dague Rule" :) | 18:38 |
| jeblair | mordred, SpamapS: i think shared "Seen" is actually configurable as either per-user or per-message.... | 18:38 |
| clarkb | speaking of I think I saw a thread on gerrit list that there is someone working on better email integration | 18:38 |
| sdague | no, that's jwz | 18:38 |
| clarkb | so you can email your patches and reviews | 18:38 |
| dims | :) | 18:38 |
| *** Sukhdev has joined #openstack-infra | 18:38 | |
| SpamapS | jeblair: so that would suffice for tracking fanout recipients then. | 18:39 |
| mordred | jeblair: awesome | 18:39 |
| * dhellmann wonders why mordred, SpamapS, and sdague are talking about changes to oslo.messaging here instead of in #openstack-oslo | 18:39 | |
| *** yolanda has quit IRC | 18:39 | |
| *** skraynev has quit IRC | 18:39 | |
| mordred | dhellmann: we haven't actually gotten past trolling yet | 18:39 |
| jeblair | mordred, SpamapS: and that configuration is actually per-mailbox :) | 18:39 |
| mordred | dhellmann: when we have an actual suggestion that's worth oslo's time, we'll move there | 18:39 |
| SpamapS | dhellmann: yeah we don't want to start our reliationship with this insanity. | 18:39 |
| mordred | jeblair: AWESOME | 18:39 |
| mordred | dhellmann: although we might be getting close ... I may be coming at you with a spec for an imap backend ... | 18:40 |
| dhellmann | SpamapS: fair enough :-) | 18:40 |
| SpamapS | jeblair: so that's tracked in the mail store which is cyrus's special mdir thing? | 18:40 |
| clarkb | notmyname: ok, I have mostly confirmed at this point we must not be setting X-Delete-After properly on our uploads | 18:40 |
| dhellmann | mordred: talk to sileht | 18:40 |
| clarkb | notmyname: I did one by hand (I shoudl've just started with this) and it does have X-Delete-At in stat info | 18:40 |
| notmyname | clarkb: ok. what are you doing? is it that patch I added my +1 to? | 18:40 |
| clarkb | notmyname: ya | 18:40 |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Modifying devstack plugin job template https://review.openstack.org/163323 | 18:40 |
| jeblair | SpamapS: yeah, it'll be in one of the binary databases which hold message metadata | 18:41 |
| clarkb | notmyname: so its likely either a logic bug or something we did wrong with requests | 18:41 |
| notmyname | clarkb: well that's what you get for trusting my reviews!! ;-) | 18:41 |
| SpamapS | mordred: the main reason this might not fly is that people will make fun of us. ;) | 18:41 |
| mordred | SpamapS: yes | 18:41 |
| *** mtanino has quit IRC | 18:41 | |
| *** zul has quit IRC | 18:41 | |
| mordred | SpamapS: that is, in fact, the only reason it won't fly | 18:41 |
| jeblair | i hope that won't actually stop us :/ | 18:41 |
| SpamapS | No | 18:41 |
| SpamapS | we're nerds, we're used to being made fun of :) | 18:41 |
| mordred | jeblair, SpamapS: oh, it won't stop us :) | 18:42 |
| mkj_wk | clarkb: thanks for the confirmation (hours ago...) that testr is the process swallowing the output. If the test case can't wait for the subprocess output any way, I have trouble knowing what to do other than time.sleep(1)... I'd like your thoughts if you have a moment some time... | 18:42 |
| clarkb | mkj_wk: I don't think I understand the underlying issue, you can write to stdout just fine | 18:43 |
| *** tkelsey has joined #openstack-infra | 18:43 | |
| SpamapS | Ok, it's worth an investigation and an experiment or two. Especially since it wouldn't require anything except an oslo.messaging driver | 18:43 |
| mkj_wk | I can't *read* it from the parent process | 18:43 |
| *** zul has joined #openstack-infra | 18:43 | |
| fungi | mordred: clarkb: SpamapS: i gather the use case for floating ips is to be able to move (float) them between machines without needing to make interface changes. so while i agree that being able to dedicate static ip addresses to virtual machines would be a great feature independent of fip, i think that what fip does is the sane solution for what i assume was its primary use case | 18:43 |
| SpamapS | also with this if we want to inspect the queue we can do it with roundcube or thunderbird. ;) | 18:43 |
| mkj_wk | I tried proc.stdout.read(1) with unbuffered pipes, and it hung until the timeout hit | 18:43 |
| mordred | fungi: but you can do that with arp | 18:44 |
| mkj_wk | at which point testr printed out the prompt it had helpfully intercepted | 18:44 |
| mordred | fungi: I used to do it all the time with heartbeat-based HA systems | 18:44 |
| fungi | mordred: well, with hsrp/vrrp/carp sure | 18:44 |
| jeblair | SpamapS: also "less", since the actuall messages are still maildir | 18:44 |
| SpamapS | mordred: gratuitous arp on a shared network of VMs _might_ be something we don't want to encourage. :) | 18:44 |
| dhellmann | mordred, SpamapS, jeblair : the thing more likely to stop you is the paucity of reviewers contributing to oslo.messaging these days | 18:44 |
| mordred | SpamapS: fair | 18:44 |
| clarkb | mkj_wk: I see | 18:44 |
| mordred | dhellmann: good point | 18:44 |
| fungi | mordred: that however requires coordination between the machines you're moving the address between though, while doing it at a router does not need the machines to even know | 18:44 |
| mordred | fungi: good point | 18:45 |
| clarkb | mkj_wk: I am not sure how to handle that better other than maybe give it a different fd to use? | 18:45 |
| mordred | fungi: I think I want both things, you know? | 18:45 |
| SpamapS | dhellmann: count me in as a reviewer for now. We can't let this infrastructure crumble. :) | 18:45 |
| fungi | mordred: i do too | 18:45 |
| mordred | fungi: I want "I know best, put this IP on this machine" | 18:45 |
| mordred | fungi: and "please do a current floating-ip like thing and put the ip elsewhere" | 18:45 |
| fungi | mordred: and i think the "add some addresses i can bind to my interface" nonexistent feature is the more useful one, unfortunately | 18:45 |
| mordred | fungi: of course, the floating-ip case is REALLY the simple case of load balacing | 18:45 |
| mordred | fungi: so you'd expect a good LBaaS thing to be able to handle the current floating-ip case | 18:46 |
| mkj_wk | clarkb: well, a hack is to wait until the prompt would have been issued | 18:46 |
| mkj_wk | clarkb: which was what I did | 18:46 |
| mordred | fungi: and then the rest to be able to use dhcp like sane people | 18:46 |
| *** dprince has joined #openstack-infra | 18:46 | |
| fungi | mordred: it goes beyond load balancing, but yes it provides a good basis for simple failover solutions | 18:46 |
| mkj_wk | clarkb: the alternative is to completely change how the test suite works not to open a subprocess but instead call cmd._main(), and mock out lots more | 18:46 |
| *** claudiub has quit IRC | 18:47 | |
| SpamapS | every good failover-floating-ip thing I've implemented has ended up on the load balancers. :) | 18:47 |
| dhellmann | SpamapS: it'll be good to have the help :-) | 18:47 |
| mkj_wk | clarkb: it's how I write my own python test suites for integration testing | 18:47 |
| mordred | SpamapS: yup | 18:47 |
| mkj_wk | (when I'm trying to test the main function, of course) | 18:47 |
| *** tkelsey has quit IRC | 18:47 | |
| clarkb | mkj_wk: ya that similar to have I have started to write nodepool cli command tests | 18:47 |
| clarkb | *how I have | 18:48 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add 'Build Flow Test Aggregator' publisher https://review.openstack.org/158083 | 18:49 |
| * SpamapS goes off in search of food | 18:50 | |
| mordred | SpamapS: get me some | 18:51 |
| *** jedimike has quit IRC | 18:51 | |
| purp_away | SpamapS: me, too, please? =D | 18:52 |
| fungi | ooh, if you're going out for burritos i want fried fish and cabbage with that habanero puree they use | 18:52 |
| *** ghostpl_ has joined #openstack-infra | 18:52 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add tests for yaml anchor behaviour https://review.openstack.org/158350 | 18:52 |
| mkj_wk | clarkb: I hesitate to take on restructuring the entire command line testing in order to get this feature in, though :/ | 18:53 |
| *** jcoufal has quit IRC | 18:54 | |
| mkj_wk | I'm hoping that an explained time.sleep() workaround can be good enough for now... :/ | 18:55 |
| *** esker has joined #openstack-infra | 18:55 | |
| clarkb | mkj_wk: I will defer to fungi on that but generally if its documented (commented) I can be convinced that hacks are ok | 18:55 |
| *** thedodd has joined #openstack-infra | 18:55 | |
| fungi | mkj_wk: yeah, while it would be awesome for git-review to get a better testsuite, i totally understand that being more work than someone would want to sign up for, so pragmatic workarounds are fine by me as long as they don't make the job flaky ;) | 18:57 |
| fungi | mkj_wk: as it stands, the testsuite it has now wasn't exactly what i envisioned, but since i wasn't the one doing the work to write it and some testing is better than none, i'm just glad we have it at all | 18:58 |
| mkj_wk | thanks | 18:59 |
| fungi | before git-review had testing, it tended to be a poster child for the "if it's not tested it's broken" mantra | 18:59 |
| mkj_wk | :) | 18:59 |
| fungi | we had at least a few rather embarrassing broken releases | 18:59 |
| mkj_wk | I hate horribly to add new hacks, especially in the first feature I try to provide | 18:59 |
| mkj_wk | I'll make sure the comment is clear so it doesn't entrench... | 19:00 |
| AJaeger | anteaya: thanks for the review. could you review https://review.openstack.org/163884 as well, please? | 19:01 |
| fungi | mkj_wk: also i keep meaning to test out the corner cases i'm worried about with 158877, so hopefully i can give you feedback on that later today | 19:02 |
| wenlock | hi, can anyone point me to any examples of where we create a tarball in a jenkins job builder template? Is there a publisher or builder for that? | 19:02 |
| fungi | wenlock: the surprisingly-named "tarball" macro at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n352 | 19:03 |
| fungi | wenlock: the scripts you see it running there are at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts | 19:04 |
| *** yfried is now known as yfried|afk | 19:04 | |
| openstackgerrit | Merged openstack-infra/project-config: gerritbot report on swift's feature/crypto branch https://review.openstack.org/163927 | 19:04 |
| wenlock | fungi, i looked at that guy too... but it seemed like it was copying a tar bal... not creating one. | 19:04 |
| mkj_wk | fungi: cool, I'll submit a newer version now with a more detailed comment so by the time you get to it Jenkins should be happy with it. ☺ | 19:04 |
| wenlock | fungi, is that what archive does? how do you tell it the input or format in which to call the tar ? | 19:05 |
| openstackgerrit | Michael Johnson proposed openstack-infra/git-review: Choose correct branch for rebase when submitting https://review.openstack.org/158877 | 19:05 |
| *** yfried|afk is now known as yfried | 19:06 | |
| fungi | wenlock: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/run-tarball.sh relies on a properly configured tox.ini and setup.py/setup.cfg to build a python package tarball | 19:06 |
| wenlock | fungi, ok, leme see if i can hunt a project down that uses that....otherwise, if the projects not python, i have to make something up? | 19:07 |
| fungi | wenlock: yeah, our "tarball" builder assumes you're making a python sdist tarball specifically | 19:08 |
| wenlock | fungi, ok cool, thanks! | 19:08 |
| wenlock | btw, is sdist something made in openstack ? | 19:08 |
| SlickNik | jeblair / fungi: I have some folks from RedHat who want to run trove tests on fedora to make sure that they pass. Is there any OpenStack policy/guidance existing in that area? Do they need to set up a 3rd party CI to do it? Any docs you can point me to? Thanks! | 19:08 |
| fungi | wenlock: the standard git repository template we use for python packaging can be found at http://git.openstack.org/cgit/openstack-dev/cookiecutter/tree/%7b%7bcookiecutter.repo_name%7d%7d/ | 19:09 |
| wenlock | fungi, awesome | 19:09 |
| *** apevec has joined #openstack-infra | 19:09 | |
| *** tgohad has joined #openstack-infra | 19:09 | |
| fungi | SlickNik: we have fedora job workers. do you have any more specific requirements from them? | 19:10 |
| *** carl_baldwin has quit IRC | 19:10 | |
| *** garyk has joined #openstack-infra | 19:12 | |
| clarkb | notmyname: so in my debugging as best as I can tell we are handing the X-Delete-After header to requests.post just fine. Could it be because we are using tempurl to upload these files that that header doesn't stick? | 19:12 |
| SlickNik | fungi: Specifically they want to be able to run the tests that run as part of the current http://logs.openstack.org/27/159627/8/check/gate-trove-functional-dsvm-mysql/081de39/ gate on Fedora. | 19:13 |
| *** tsg_ has quit IRC | 19:13 | |
| SlickNik | gate-trove-functional-dsvm-mysql* | 19:13 |
| SlickNik | (too eager with the copy paste) | 19:13 |
| clarkb | notmyname: looks like the server side may be configured with incoming_remove_headers | 19:13 |
| clarkb | notmyname: any idea if we can whitelist things ourselves on a per tempurl or tenant/account/project basis | 19:14 |
| *** pmackinn has joined #openstack-infra | 19:15 | |
| dhellmann | has anyone else reported seeing import errors with glob2 in zuul_swift_upload.py at the end of jobs? | 19:15 |
| clarkb | dhellmann: not yet, can you link to an example? | 19:16 |
| dhellmann | clarkb: https://jenkins01.openstack.org/job/check-dg-tempest-dsvm-full-reexec/364/console | 19:16 |
| clarkb | also I am guessing that means the change merged that probably shouldn't have merged | 19:16 |
| clarkb | oh wow that means we have a really old node | 19:17 |
| fungi | SlickNik: yeah, i think as long as the devstack-f21 workers we have are viable, that should be easy to add. might want to hit up ianw to find out if that platform's usable for us yet but we do seem to have some jobs running on it... e.g. https://jenkins04.openstack.org/job/check-tempest-dsvm-f21/296/console running right this moment | 19:17 |
| * clarkb goes to check devstack-trusty in hpcloud-b3 | 19:17 | |
| clarkb | dhellmann: huh our image there is ~4 hours old which is plenty new enough that it should have glob2 installed. THis requires more investigation | 19:18 |
| dhellmann | clarkb: when the current job fails should I reverify, or wait for you to do that investigation? | 19:18 |
| SlickNik | fungi: Awesome, thanks — I also see the jobs in project-config, and I can point them in that direction. | 19:19 |
| clarkb | dhellmann: you can reverify and roll dice that you get an image with glob2 in that venv | 19:19 |
| dhellmann | clarkb: ok, thanks | 19:19 |
| clarkb | dhellmann: it is working on some of the images I think because in general the infra jobs do work ok but msot don't run on devstack-* nodes | 19:19 |
| openstackgerrit | Merged openstack-infra/project-config: Fix return code of irc_tests.py https://review.openstack.org/163884 | 19:21 |
| apevec | fungi, https://review.openstack.org/141412 is now in gate and all jobs are SKIPPED ?? | 19:21 |
| apevec | what does that mean? | 19:21 |
| jeblair | apevec: if you mouse-over the black dot next to it you'll see the reason | 19:21 |
| cinerama | hmm. so i'm a bit confused about where is the best place to put the mysql-related bits of my stuff for zanata given that the different puppet modules vary a bit on how they handle it. | 19:22 |
| apevec | so again merge conflict... | 19:22 |
| jeblair | apevec: the ceilo change has a merge conflict | 19:22 |
| jeblair | yeah | 19:22 |
| apevec | jeblair, but it doesn't | 19:22 |
| *** ParsectiX has quit IRC | 19:22 | |
| *** crc32 has joined #openstack-infra | 19:22 | |
| fungi | yeah, it's parent (according to gerrit) is the branch tip | 19:22 |
| *** ParsectiX has joined #openstack-infra | 19:22 | |
| jeblair | i'll look that up | 19:23 |
| fungi | i went digging in the debug log to see if i could tell why zuul kicked it out for a merge conflict earlier and had no luck | 19:23 |
| *** esker has quit IRC | 19:24 | |
| clarkb | cinerama: I would have a zanata::mysql like you do, just don't put that in zanata's init.pp | 19:24 |
| clarkb | cinerama: that way you can choose to use it or choose not to use it when you compose your puppet manifests | 19:25 |
| apevec | jeblair, it was in the gate yesterday when I accidentaly set freeze -2, maybe that confused zuul? | 19:25 |
| krtaylor | clarkb, what is the problem you are seeing with X-Delete-After? | 19:25 |
| clarkb | krtaylor: its not applying with the tempurl, I am pretty sure its because we use tempurl and rax must not allow us to | 19:25 |
| jeblair | apevec: it should have removed it from the gate at that point, but it shouldn't matter now | 19:25 |
| cinerama | clarkb: at least some of the modules seem to include their mysql stuff rather than bringing it in in the manifest | 19:25 |
| *** pelix has quit IRC | 19:25 | |
| clarkb | krtaylor: I have been able to confirm that the header val is being set correctly before being given to requests.post | 19:26 |
| krtaylor | clarkb, you should be able to do a test and then go check the metadata on the object store console | 19:26 |
| clarkb | cinerama: I think we are trying to get away from that as a default | 19:26 |
| *** claudiub has joined #openstack-infra | 19:26 | |
| clarkb | krtaylor: yes I have already checked `swift stat` for tempurl uploads and manaul not temprul uploads and its not set for the tempurl upload but is set otherwise | 19:26 |
| cinerama | clarkb:oh, ok. do we have a...thing...about this i should have read already? | 19:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Log merger job uuids https://review.openstack.org/163936 | 19:27 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:27 | |
| jeblair | fungi: ^ we should merge that and its parent; it will greatly help with debugging this | 19:27 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:28 | |
| krtaylor | clarkb, in our swift upload we do it directly (not with the middleware), so hard to say, but it sounds like you may be right | 19:28 |
| *** cody-somerville has joined #openstack-infra | 19:29 | |
| * krtaylor needs to go revisit the zuul swift upload helper | 19:29 | |
| fungi | jeblair: thanks, i was just looking for that since clarkb mentioned it earlier | 19:29 |
| *** weshay has quit IRC | 19:30 | |
| krtaylor | clarkb, I wish I could give you temp access to our swift store so you could see if it really is rax | 19:31 |
| fungi | cinerama: i don't think we've documented that goal specifically, it's more just institutional knowledge that some deployments may want to have their database server somewhere other than on the application server (for example we might want to hook the application up to a "trove" saas db in the same service provider where the app server resides for ease of manageability) | 19:31 |
| fungi | cinerama: so it's great flexibility to include | 19:32 |
| *** bdpayne has quit IRC | 19:32 | |
| clarkb | krtaylor: its fine I can probably poke rax to get a concrete answer | 19:32 |
| clarkb | but need to figure out these iamges first | 19:32 |
| cinerama | fungi: right, i'm just saying that if folks are working off the existing examples then they are not necessarily going to have a clear picture of that | 19:33 |
| cinerama | fungi: also it depends on whether we are developing for our own purposes at present, or for a more general use case where others would adopt it | 19:33 |
| fungi | cinerama: i totally agree, it would probably be a low-hanging spec for someone to tackle (make database installation optional in existing modules) | 19:33 |
| clarkb | dhellmann: ok I think I understand what is going on and your job should work fine if it runs on a rax node | 19:34 |
| fungi | cinerama: we always try to develop for others to use | 19:34 |
| clarkb | fungi: jeblair: et al new dib fun. If dib fails to build an image it appears that it will just upload the most recently succesfully build image during the daily image update | 19:34 |
| dhellmann | clarkb: let's hope that's where it's scheduled, then :-) | 19:34 |
| clarkb | fungi: jeblair so while it looks like we have a current dib image for hpcloud devstack-trusty we do not, its really old | 19:34 |
| dhellmann | oh, that's a fun failure mode | 19:35 |
| clarkb | and we are hitting that ruby issue that asselin ran into | 19:35 |
| clarkb | dhellmann: ya :/ | 19:35 |
| clarkb | asselin: what happeed with that ruby issue? /me needs to go figure that out now apparently | 19:35 |
| fungi | cinerama: we've found that assuming up-front that others will want to run your software means you make early design choices to support that assumption which might be a lot harder to fix later once the code grows more complex | 19:35 |
| *** ParsectiX has quit IRC | 19:36 | |
| clarkb | dhellmann: so ya we are using an older image there that did not install glob2 into that venv yet | 19:36 |
| *** pmackinn has left #openstack-infra | 19:36 | |
| clarkb | dhellmann: once I figure out why the builds are failing I can trigger some rebuilds to get things moving, then work on making nodepool make these failures more obvious | 19:36 |
| fungi | clarkb: oh, ick! | 19:37 |
| dhellmann | clarkb: sounds reasonable | 19:37 |
| clarkb | but its also lunchtime so I might eat first >_> | 19:37 |
| *** timcline has quit IRC | 19:38 | |
| clarkb | mordred: ^ I think this is related to how we cache packages in dib builds, I am going to try and reproduce in my local test bed to debug but maybe it stands out to you | 19:38 |
| openstackgerrit | Merged openstack-infra/zuul: Improve merge client logging https://review.openstack.org/163110 | 19:38 |
| mordred | clarkb: I'm excited about this? | 19:38 |
| *** pc_m has quit IRC | 19:38 | |
| clarkb | mordred: well I think ou understand how we cache stuff with dib better than * | 19:38 |
| *** e0ne has joined #openstack-infra | 19:39 | |
| *** openstackgerrit has quit IRC | 19:40 | |
| *** openstackgerrit has joined #openstack-infra | 19:40 | |
| greghaynes | clarkb: are there logs of the bulid fails somewhere? | 19:40 |
| greghaynes | or is this the kind of thing you just have to hope you can repro | 19:41 |
| fungi | greghaynes: http://nodepool.openstack.org/ for image build logs | 19:41 |
| greghaynes | ahoy! | 19:41 |
| asselin | clarkb, I worked around it..... | 19:42 |
| asselin | clarkb, greghaynes fyi: disabled set -e and added --ignore-missing http://paste.openstack.org/show/191911/ | 19:43 |
| greghaynes | asselin: :/ | 19:44 |
| greghaynes | asselin: what was the pakage it was failing on again? ruby1.8? | 19:45 |
| clarkb | on trusty | 19:45 |
| clarkb | on precise its docker.io | 19:45 |
| rcarrillocruz | SpamapS: i didn't want to pollute the change with the debug statemnts, latest patchset made the func test class to derive from TestCase, that has all fixtures OS_STDOUT_CAPTURE goodness | 19:46 |
| asselin | sorry...I needed to get dib working first. | 19:46 |
| rcarrillocruz | but my devstack-gate VM is giving me pain, i'll put the debug statements on the change and push for review shortly, let's hope i get something useful from output... | 19:46 |
| asselin | I can take another look since it's reproducible on my dev box | 19:46 |
| *** teran_ has quit IRC | 19:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Add networking-bagpipe-l2 project https://review.openstack.org/161290 | 19:47 |
| greghaynes | clarkb: oh, youre hitting that. What I traced it down to is the trema deps file in devstack, but it seems thats been removed | 19:49 |
| greghaynes | clarkb: and looks like now ruby1.8 is no longer a dep in devstack | 19:49 |
| greghaynes | at least not explicitly | 19:49 |
| cloudnull | hey all, i want to convert the os-ansible-deployment project in stackforge to using specs instead of continuing to use Launchpad bp's. Is there a defacto way creating a specs repo for a stackforge project that I can read about without bothering you guys to much? | 19:49 |
| greghaynes | asselin: Yea, I wonder if its just 'fixed' now for you | 19:50 |
| anteaya | cloudnull: http://docs.openstack.org/infra/manual/creators.html | 19:50 |
| *** yfried is now known as yfried|afk | 19:51 | |
| * cloudnull reading | 19:51 | |
| cloudnull | thanks anteaya | 19:51 |
| asselin | greghaynes, no..it's still broken...I saw the log message on newer dib builds | 19:51 |
| clarkb | ya it failed today for us | 19:51 |
| greghaynes | huh | 19:51 |
| anteaya | you're welcome | 19:51 |
| asselin | greghaynes, just that the script keeps going. I justified it b/c if it's really needed, it will be installed during the job itself | 19:52 |
| greghaynes | https://review.openstack.org/#/c/162455/ looks like its the commit that removed the dep | 19:52 |
| asselin | using the not cached version.... | 19:52 |
| greghaynes | asselin: and its still ruby1.8? | 19:52 |
| asselin | greghaynes, yes..let me see if I still have logs | 19:52 |
| *** [HeOS] has joined #openstack-infra | 19:53 | |
| *** gokrokve_ has quit IRC | 19:53 | |
| *** gokrokve has joined #openstack-infra | 19:54 | |
| *** marun has quit IRC | 19:54 | |
| *** marcusvrn has quit IRC | 19:54 | |
| greghaynes | clarkb: fungi so with those nodepool logs, is that multiple image bulids concatenated together? | 19:55 |
| clarkb | yes, just grep out the one you want | 19:55 |
| fungi | greghaynes: interleaved, but since dib builds serially they look concatenated | 19:55 |
| *** gokrokve_ has joined #openstack-infra | 19:55 | |
| greghaynes | is splitting that out a desired feature? looking for excuses to poke at things | 19:56 |
| fungi | greghaynes: yes, ianw is working on a possible solution there i believe | 19:56 |
| *** dprince has quit IRC | 19:56 | |
| greghaynes | ah | 19:56 |
| mtreinish | jogo: on https://review.openstack.org/#/c/160933/4 do you know where in the logs to look and see if everything is working? | 19:57 |
| fungi | greghaynes: but in parallel to that, we also want to dramatically reduce the number of images we're building anyway | 19:57 |
| fungi | and have efforts underway to make that happen too | 19:57 |
| greghaynes | ah, right, ive seen those patches | 19:57 |
| *** gokrokve has quit IRC | 19:59 | |
| pabelanger | Did gitweb ever run directly on gerrit for -infra? Specifically behind the http proxy? Trying to figure out the apache syntax to get it working with anonymous gerrit users | 19:59 |
| fungi | pabelanger: you mean like https://review.openstack.org/gitweb?p=openstack-infra%2Fzuul.git | 19:59 |
| pabelanger | fungi, yup | 20:00 |
| *** timcline has joined #openstack-infra | 20:00 | |
| pabelanger | for me, if unauthenticated user of gerrit access that URL, they get 404 | 20:00 |
| pabelanger | I'm still doing 2.5 for gerrit | 20:00 |
| fungi | pabelanger: i believe it was working for us anonymously even before 2.4 | 20:01 |
| *** imcsk8|zZz is now known as imcsk8 | 20:01 | |
| pabelanger | I suspect because you are using external cgit, it is working differently then my setup | 20:01 |
| pabelanger | fungi, okay, cool. Checking git logs for info to see what apache config you did | 20:01 |
| *** melwitt_ has joined #openstack-infra | 20:02 | |
| fungi | pabelanger: i don't see anything particularly special in our https://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/templates/gerrit.vhost.erb | 20:02 |
| *** mikal_ is now known as mikal | 20:02 | |
| *** melwitt has quit IRC | 20:02 | |
| *** melwitt_ is now known as melwitt | 20:02 | |
| *** garyk has quit IRC | 20:03 | |
| pabelanger | fungi, same. | 20:03 |
| pabelanger | my is a little different, but same concept | 20:03 |
| pabelanger | let me try using that syntax and see what happens | 20:03 |
| *** baoli has joined #openstack-infra | 20:03 | |
| *** taron has quit IRC | 20:04 | |
| *** cody-somerville has quit IRC | 20:04 | |
| *** baoli has quit IRC | 20:04 | |
| *** taron has joined #openstack-infra | 20:05 | |
| dougwig | clarkb: sorry i haven't gotten back to you on the unit test madness thing. it's just a conversation that can wait until after kilo, and i've been buried. i will close that loop when it gets a little saner. | 20:05 |
| *** baoli has joined #openstack-infra | 20:05 | |
| anteaya | dougwig: it never gets saner | 20:05 |
| anteaya | in 2 years I have never seen it get saner | 20:06 |
| *** arxcruz has joined #openstack-infra | 20:06 | |
| *** jgrimm is now known as zz_jgrimm | 20:07 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: neutron-*aas pylint jobs moving into pep8 job https://review.openstack.org/160915 | 20:08 |
| *** jebertol has joined #openstack-infra | 20:08 | |
| dougwig | anteaya: i'd settle for brief lull. :) | 20:09 |
| *** AJaeger has quit IRC | 20:10 | |
| *** yfried|afk is now known as yfried | 20:11 | |
| *** gordc has quit IRC | 20:11 | |
| anteaya | saw it once during grizzly | 20:11 |
| *** otter768 has joined #openstack-infra | 20:12 | |
| jebertol | Hi - I'm getting an 'invalid committer' error when doing a git review. Could use some help. Looks like my git config and contacts on review.openstack.org match though. | 20:12 |
| fungi | jebertol: can you post the output at http://paste.openstack.org/ and link it here? | 20:13 |
| fungi | jebertol: chances are you used a different e-mail address when you ran 'git commit' or else you're trying to submit commits which aren't yours | 20:14 |
| *** zz_jgrimm is now known as jgrimm | 20:14 | |
| jebertol | fungi: here you go - http://paste.openstack.org/show/191930/ | 20:15 |
| *** Longgeek has joined #openstack-infra | 20:16 | |
| fungi | jebertol: is it possible you had quotes around the e-mail address in your git config and fixed those after you ran git commit? | 20:17 |
| *** kgiusti has left #openstack-infra | 20:17 | |
| *** otter768 has quit IRC | 20:17 | |
| jebertol | fungi: hmmm - yes I did | 20:18 |
| fungi | jebertol: if so, a 'git commit --amend' should fix that | 20:18 |
| *** sambetts has quit IRC | 20:18 | |
| jebertol | fungi - ok then do the git review after the amend? | 20:18 |
| fungi | jebertol: yep | 20:19 |
| jebertol | fungi: thanks some much - that worked | 20:20 |
| *** woodster_ has quit IRC | 20:20 | |
| fungi | jebertol: great--let us know if you run into any other problems | 20:20 |
| jebertol | fungi: so much ... | 20:20 |
| *** sambetts has joined #openstack-infra | 20:20 | |
| *** yfried is now known as yfried|afk | 20:21 | |
| *** cody-somerville has joined #openstack-infra | 20:22 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add initial compute functional tests to Shade https://review.openstack.org/158463 | 20:22 |
| SpamapS | rcarrillocruz: hah, you forgot the print statements in 19. ;) | 20:23 |
| *** Longgeek has quit IRC | 20:23 | |
| *** dkranz has quit IRC | 20:24 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Log stdout from build_wheels in cache_devstack https://review.openstack.org/163958 | 20:24 |
| mtreinish | jogo: ^^^ | 20:25 |
| openstackgerrit | Kevin Carter proposed openstack-infra/project-config: Updates for os-ansible-deployment adding specs and docs repos https://review.openstack.org/163960 | 20:26 |
| mtreinish | clarkb, fungi: If you get a sec ^^^ is needed to debug what is going on with the build_wheels script on nodepool | 20:27 |
| mtreinish | all it says in the logs is running, but jogo's d-g patch isn't seeing the dir being created | 20:27 |
| *** dkranz has joined #openstack-infra | 20:27 | |
| fungi | jeblair: zaro: mordred: i have a custom trove config with character_set_server=utf8, collation_server=utf8_bin and wait_timeout=28800 which i can switch the review-dev instance to use (in place of the one it currently has to just override wait_timeout). does that cover what we were considering trying? | 20:28 |
| jeblair | fungi: i believe so | 20:28 |
| *** e0ne has quit IRC | 20:30 | |
| jogo | mtreinish: ok running your patch locally in a docker env | 20:30 |
| *** maxbit has joined #openstack-infra | 20:30 | |
| openstackgerrit | Jan Hruban proposed openstack-infra/jenkins-job-builder: Add support for Sonar analysis builder https://review.openstack.org/163961 | 20:31 |
| *** teran has joined #openstack-infra | 20:31 | |
| *** Somay has joined #openstack-infra | 20:31 | |
| openstackgerrit | Merged openstack-infra/nodepool: Accomodate ready nodes in min-ready checks https://review.openstack.org/162714 | 20:32 |
| mtreinish | jogo: I have a suspicion it's something with setting wheelhouse in the env for the subprocess, but that's just wild speculation | 20:32 |
| openstackgerrit | Merged openstack-infra/nodepool: Don't allocate nodes to providers without images https://review.openstack.org/162255 | 20:33 |
| openstackgerrit | Steve Baker proposed openstack-infra/project-config: Custom OVERRIDE_ENABLED_SERVICES for heat-dsvm-functional https://review.openstack.org/162487 | 20:33 |
| *** e0ne has joined #openstack-infra | 20:34 | |
| jogo | mtreinish: hmm it worked for me | 20:34 |
| *** maxbit has left #openstack-infra | 20:34 | |
| openstackgerrit | Merged openstack-infra/nodepool: Test that dib failures don't stop node builds https://review.openstack.org/162237 | 20:34 |
| * mtreinish scatches his head | 20:34 | |
| openstackgerrit | Merged openstack-infra/nodepool: Gracefully handle upload failures https://review.openstack.org/163239 | 20:35 |
| jogo | mtreinish: but my env isn't the same as the one this is being run in | 20:36 |
| jogo | so it could be subtle | 20:36 |
| jogo | hmm this is failing in 15 seconds in nodepool | 20:37 |
| jogo | which is pretty fast | 20:37 |
| jogo | err finishing* | 20:37 |
| fungi | jeblair: zaro: mordred: looks like it works, but the trove instance needs a restart after the config is chosen (i did the trove instance restart with gerrit stopped for safety): https://review-dev.openstack.org/5258 | 20:37 |
| jogo | I wonder if pip is installed | 20:38 |
| *** Somay has quit IRC | 20:38 | |
| *** Sukhdev has quit IRC | 20:38 | |
| anteaya | SpamapS: I recall not so long ago you waxing philosophical about a mythical village with a village square | 20:39 |
| anteaya | it was a good email | 20:39 |
| jeblair | fungi: woot; so want to just do this next time gerrit is stopped for any reason (which is looking like next weekend for the os move anyway)? | 20:39 |
| anteaya | my vision got trampled by the crowd long ago | 20:39 |
| fungi | jeblair: yep | 20:39 |
| anteaya | there was a part of me that hoped you still could maintain yours | 20:39 |
| mtreinish | jogo: I thought get_pip was done during nodepool but that's based on nothing but a blind assumption | 20:40 |
| fungi | jeblair: i'll try to make sure we remember to switch the config to that one (it's called "sanity") | 20:40 |
| jogo | mtreinish: nodepool/scripts/fix_pip.sh:if | 20:40 |
| jeblair | fungi: i have an idea for what to call the next one | 20:40 |
| openstackgerrit | Merged openstack-infra/nodepool: Remove duplicate code https://review.openstack.org/163234 | 20:40 |
| SpamapS | anteaya: a village with a square sounds nice. :) | 20:40 |
| fungi | jeblair: unsanity! | 20:40 |
| anteaya | SpamapS: tell if you find one | 20:41 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-kibana: Update kibana project url https://review.openstack.org/162727 | 20:41 |
| jogo | mtreinish: but we also run http://logs.openstack.org/33/160933/4/check/check-tempest-dsvm-full/b1e7aa7/logs/devstacklog.txt.gz#_2015-03-12_18_27_23_739 | 20:41 |
| mtreinish | jogo: we might just have to wait until tomorrow's image build after my stdout patch lands | 20:42 |
| SpamapS | anteaya: in this scenario, the village is hustling and bustling which is good, but do we really need the cobblers and taylors to run around in the street telling us about the suits and shoes they made this week? :) | 20:42 |
| *** mrda-away is now known as mrda | 20:42 | |
| mtreinish | jogo: but is it doing anything, the trace doesn't show anything being run, just some games with cd | 20:43 |
| *** krisgan has quit IRC | 20:43 | |
| *** erikmwilson has quit IRC | 20:43 | |
| mtreinish | jogo: I also know pip is there eventually because I pip install somethings for the subunit2sql side. But it could be an ordering thing | 20:44 |
| jogo | 2015-03-12 14:38:16,261 INFO nodepool.image.build.rax-dfw.devstack-precise: Installing setuptools, pip...done. | 20:44 |
| *** tonytan4ever has quit IRC | 20:44 | |
| anteaya | SpamapS: ah your end of the village sounds pretty good, and yeah less noise would be good | 20:44 |
| jogo | 2015-03-12 14:32:00,513 INFO nodepool.image.build.rax-dfw.devstack-precise: Running: ['bash', 'tools/build_wheels.sh'] | 20:45 |
| jogo | so it maybe an ordering thing | 20:45 |
| mtreinish | jogo: yep, an ordering issue | 20:45 |
| mtreinish | we probably will need to move the build_wheels call later in the process | 20:45 |
| anteaya | I've been trying to pick up the garbage and realize that just makes space for more people who can't find a garbage can | 20:46 |
| openstackgerrit | Merged openstack-infra/project-config: Run nodepool-coverage as a check job https://review.openstack.org/163237 | 20:46 |
| openstackgerrit | Merged openstack-infra/nodepool: Remove unneeded Label.*_providers https://review.openstack.org/152757 | 20:47 |
| jogo | hmm nodepool/scripts/prepare_node.sh:wget https://git.openstack.org/cgit/openstack-infra/system-config/plain/install_puppet.sh | 20:48 |
| *** ddieterly has quit IRC | 20:48 | |
| clarkb | ok I am back now to debug image builds | 20:48 |
| pleia2 | anteaya: that should be the description of being a sysadmin :) | 20:48 |
| *** ddieterl_ has joined #openstack-infra | 20:48 | |
| openstackgerrit | Kevin Carter proposed openstack-infra/project-config: Updates for os-ansible-deployment adding specs and docs repos https://review.openstack.org/163960 | 20:48 |
| *** carl_baldwin has joined #openstack-infra | 20:48 | |
| jogo | mtreinish: so what setsthe order? | 20:48 |
| openstackgerrit | jose-idar proposed openstack-infra/project-config: Adding pep8 gate check to Opencafe https://review.openstack.org/159642 | 20:49 |
| jeblair | clarkb: you said gertty made something hard to detect in 161952; can you expand on that? | 20:50 |
| anteaya | pleia2: well you get to clean up the machines | 20:50 |
| *** thedodd has quit IRC | 20:50 | |
| openstackgerrit | Merged openstack-infra/nodepool: Do not try to upload building disk images https://review.openstack.org/161482 | 20:50 |
| clarkb | jeblair: ya, the diff view makes it raelly hard to see where the left edge of the "page" is to count whitespace towards the right | 20:50 |
| anteaya | cleaning the garbage that _makes_ it into the can is way better | 20:50 |
| clarkb | jeblair: if we could draw a line on the right side of the line numbers that would help a bunch | 20:50 |
| *** Somay has joined #openstack-infra | 20:50 | |
| anteaya | rather than trying to train them to hit the can | 20:50 |
| pleia2 | anteaya: ah, indeed | 20:50 |
| clarkb | jeblair: because right now at the top you have 1lotsofspace | 20:50 |
| clarkb | jeblair: then at the bottom ou have 2123littlespace | 20:51 |
| clarkb | asselin: so I think your solution may actually be best | 20:51 |
| clarkb | asselin: the ignore-missing flag, want to propose that? | 20:51 |
| clarkb | asselin: the reason I think that is we look at all devstack branches regardless of base distro and release even though we know some images won't be used with some branches | 20:52 |
| clarkb | asselin: rather than encode that in the image builds I think we take the simple route of just ignoring missing packages | 20:52 |
| mtreinish | jogo: IIRC it's somewhere in system-config (or at least it used to be before the split) | 20:52 |
| jeblair | clarkb: ah, we should right-align the numbers first, which would help a lot. and if you want more, we could give them a background color, then there will be a nice line | 20:52 |
| mtreinish | jogo: or the first script called was in there | 20:52 |
| clarkb | jeblair: that sounds good | 20:52 |
| *** thedodd has joined #openstack-infra | 20:53 | |
| *** thedodd has quit IRC | 20:54 | |
| *** marun has joined #openstack-infra | 20:54 | |
| *** jebertol has quit IRC | 20:54 | |
| jogo | ahh entry point is prepare_node_devstack | 20:54 |
| *** yfried|afk is now known as yfried | 20:55 | |
| *** Somay has quit IRC | 20:55 | |
| mtreinish | jogo: yeah | 20:55 |
| *** _nadya_ has joined #openstack-infra | 20:55 | |
| jogo | hmm maybe not a ordering thing | 20:57 |
| jogo | 2015-03-12 14:16:28,444 INFO nodepool.image.build.rax-dfw.devstack-precise: Successfully installed pip-6.0.8 setuptools-14.0 | 20:57 |
| mtreinish | well fungi just +A'd the patch so we can wait until the next image build to see what's going on | 20:58 |
| fungi | mtreinish: i figured it was good output to have regardless | 20:58 |
| *** dboik_ has joined #openstack-infra | 20:59 | |
| clarkb | so are we still trying to build the wheels in the images? | 20:59 |
| clarkb | can we instaed work on the mirroring of wheels? | 20:59 |
| jogo | clarkb: I don't see why not | 20:59 |
| jogo | lets do both and see who finishes first | 21:00 |
| *** _nadya_ has quit IRC | 21:00 | |
| clarkb | because of all the problems with it... | 21:00 |
| clarkb | and I would rather not duplicate work, instead just do the correct work | 21:00 |
| clarkb | jogo: how are you building wheels without installing deps first? | 21:00 |
| clarkb | mtreinish: ^ | 21:01 |
| clarkb | if we can solve that problem I would be more on board but I am not sure that is solveable in a nice wa | 21:01 |
| jogo | pip wheel | 21:01 |
| clarkb | jogo: no, you need dev headers of system packages to link against | 21:01 |
| mtreinish | clarkb: IIRC the the build_wheel script runs apt | 21:01 |
| clarkb | libxml2-dev for example | 21:01 |
| mtreinish | which is probably terrible | 21:01 |
| *** ghostpl_ has quit IRC | 21:01 | |
| clarkb | mtreinish: yes and thats a problem | 21:01 |
| clarkb | mtreinish: we need to not do that | 21:02 |
| *** dboik has quit IRC | 21:02 | |
| jogo | so this still comes back to the point that I don't think we should have one test to run devstack agaainst an empty box | 21:02 |
| jogo | and that would be gating for devstack | 21:02 |
| mtreinish | jogo: it already is | 21:02 |
| openstackgerrit | Stephanie Miller proposed openstack-infra/puppet-zanata: Initial commit of Zanata-specific sections https://review.openstack.org/147947 | 21:02 |
| mtreinish | it runs stack.sh and then unstack.sh and makes sure both exit 0 | 21:03 |
| clarkb | fungi: I think we need to consider reverting all of that wheels stuff instead of making more changes to it, we cannot sanely build wheels without installing a bunch of things we explicitly don't want to install | 21:03 |
| jogo | and another with as much as possible pre-installed/cached etc. and that would be for everything else | 21:03 |
| mtreinish | clarkb: fwiw, it doesn't look like it's doing anything now anyway it fails early for whatever reason | 21:03 |
| jogo | mtreinish: oh right, what base image does that use? | 21:03 |
| *** harlowja_ has quit IRC | 21:03 | |
| fungi | clarkb: i built a temporary duplicate of git-fe02 at 104.239.141.161 and it looks like it dtrt with regard to login.defs but take a quick look if you don't mind | 21:03 |
| clarkb | fungi: will do | 21:03 |
| mtreinish | clarkb: I'm fine with a revert as long as we start working on mirroring the wheels (not sure where to start on that though) | 21:04 |
| *** rlandy has quit IRC | 21:04 | |
| clarkb | mtreinish: that would be my preference | 21:04 |
| *** harlowja has joined #openstack-infra | 21:04 | |
| *** bswartz has quit IRC | 21:04 | |
| fungi | clarkb: so it seems to me like the timing theory is validated, and our current configuration does what we want tof user/group creation now, we just have a handful of systems from that timeframe which need cleanup | 21:04 |
| *** rfolco has quit IRC | 21:04 | |
| jogo | long as we have something in place the entire time | 21:04 |
| clarkb | fungi: ya it seems to be correct there | 21:05 |
| fungi | awesome. i'll go ahead and delete that instance now | 21:05 |
| jogo | clarkb: so if we have a test that makes sure devstack installs on a generic base image | 21:05 |
| *** patrickeast has quit IRC | 21:05 | |
| *** patrickeast_ has joined #openstack-infra | 21:05 | |
| jogo | what is wrong with pre-installing all the packages etc? | 21:05 |
| clarkb | jogo: so the whole point is that devstack can always deploy openstack exactly how it is now | 21:05 |
| clarkb | jogo: if we want to change that that is likely a qusetion for dtroyer and sdague | 21:06 |
| jogo | clarkb: I don't follow | 21:06 |
| jogo | clarkb: I am referring to the apt-get packages and things like that | 21:06 |
| mtreinish | clarkb: it's not something that's going to change | 21:06 |
| clarkb | jogo: the way we test today with devstack ensure to a high degree that devstack can always deploy openstack at all times | 21:06 |
| *** sarob has joined #openstack-infra | 21:06 | |
| clarkb | jogo: if you pull HEAD | 21:06 |
| jogo | we still get updates etc. just more is in the image we start from | 21:06 |
| clarkb | as soon as you start doing work for devstack you stop testing that stuff | 21:07 |
| clarkb | and open yourself to races between devtsack changes and openstack changes breaking in those portions of devstack | 21:07 |
| jogo | sorry I don't follow | 21:07 |
| jogo | we make the stack.sh/unstack.sh test use a blank image | 21:07 |
| jogo | to make sure we test the full standard devstack flow | 21:07 |
| asselin | clarkb, ok | 21:08 |
| clarkb | jogo: but thats only testing at a point in time | 21:08 |
| clarkb | jogo: so you oepn yourself to breaking | 21:08 |
| clarkb | asselin: I am about to test it locally for my own sanity but go ahead and push the change | 21:08 |
| *** yfried is now known as yfried|afk | 21:08 | |
| clarkb | jogo: the way we gate is that you test in lockstep as you go, this has the nice benefit of meaning cloning devstack and running it is almost always going to work on a clean machine | 21:08 |
| clarkb | jogo: that is not true if image builds or devstack-gate start doing devstack's job for it. We know this because we have broken devstack in the pst when devstack did too much | 21:09 |
| *** aysyd has quit IRC | 21:09 | |
| jogo | clarkb: how do we open our self up to breaking? | 21:10 |
| jogo | we still gate devstack on running devtack from an empty state | 21:10 |
| *** Ryan_Lane has quit IRC | 21:10 | |
| jogo | can you give a specific example | 21:10 |
| clarkb | jogo: because devstack runs about 20 something projects | 21:10 |
| jogo | and ... ? | 21:10 |
| clarkb | jogo: so its eg an interaction between nova and devstack | 21:10 |
| clarkb | we never tested that because we only tested last nights devstack against latest nova | 21:11 |
| clarkb | now on next image build the entire gate is wedged | 21:11 |
| jogo | no that isn't what I am saying | 21:11 |
| jogo | I am saying we preinstall the following: | 21:11 |
| clarkb | also on a related note we are explicitly trying to stop having specialized images | 21:11 |
| *** ghostpl_ has joined #openstack-infra | 21:11 | |
| clarkb | we want the simplest image possible to run all the tests | 21:11 |
| jogo | packages, python stuff | 21:11 |
| clarkb | this lends itself well to using the mirror | 21:11 |
| jogo | and devstack runs an update on them anyway (or can be made to) | 21:11 |
| clarkb | jogo: yes today that is devstacks job | 21:11 |
| clarkb | so lets contrive an example | 21:12 |
| jogo | today we don't pre-install any packages or any python packages | 21:12 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Ignore missing packages https://review.openstack.org/163976 | 21:12 |
| clarkb | say today devstack installs package A and nova depends on package A | 21:12 |
| jogo | optimizing for a single image isn't the right thing IMHO | 21:12 |
| jogo | we should optimize for run time | 21:12 |
| clarkb | devstack and nova work together bceause A is installed by devstack | 21:12 |
| asselin | clarkb, ^^ I still need to double-check set -e is ok with --ignore-missing | 21:13 |
| greghaynes | asselin: if it ends up not being you can just || true, then we still have set -e for the rest of the script | 21:13 |
| fungi | hrm... i do want to discuss how we pre-cache things on nodepool images (so that we can do it in a generic way for all projects' jobs rather than just for devstack-based jobs) but i need to pop out to dinner. i'll check scrollback in a little while and see where this goes ;) | 21:14 |
| greghaynes | asselin: didnt want to shed on that but youll end up making a change anyways if it doesnt work ;) | 21:14 |
| *** woodster_ has joined #openstack-infra | 21:14 | |
| asselin | greghaynes, thanks for the heads up | 21:14 |
| clarkb | jogo: now if package A is removed from devstack nova still works because using an old image | 21:14 |
| *** thedodd has joined #openstack-infra | 21:14 | |
| openstackgerrit | James E. Blair proposed stackforge/gertty: Right align line numbers https://review.openstack.org/163979 | 21:15 |
| clarkb | jogo: your test guards against that if devstack is gated by running from scratch | 21:15 |
| clarkb | so everything still good | 21:15 |
| jeblair | clarkb: ^ er, that took about 4 times as long as i thought it would, but i think it's good. ;) | 21:15 |
| fungi | i do agree at least that pre-building virtualenvs for the things we want tested is going to end in sorrow though | 21:15 |
| fungi | anyway, back in a little while | 21:15 |
| clarkb | jogo: but, what if say that package is installed by the image build script similar to how we are doing | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| jogo | if you remove package A from devstack, the devstack test that doesn't pre-install anything would catch it | 21:16 |
| clarkb | jogo: then your simple base image is inadverdently broken allowing devstack change to merge | 21:16 |
| clarkb | jogo: from this point on anyone using devstack is broken | 21:16 |
| *** ghostpl_ has quit IRC | 21:16 | |
| jogo | clarkb: see my comment above | 21:16 |
| clarkb | jogo: yes I provided an example where that is insufficient | 21:17 |
| jogo | how? | 21:17 |
| jogo | I still don't follow | 21:17 |
| clarkb | the problem is in the managing of the images | 21:17 |
| clarkb | its hard to prevent leaking | 21:17 |
| jogo | you made a change to devstack | 21:17 |
| jogo | devstack tests catch the issue | 21:17 |
| clarkb | yes keep reaidng | 21:17 |
| *** yfried|afk is now known as yfried | 21:17 | |
| openstackgerrit | Kevin Carter proposed openstack-infra/project-config: Updates for os-ansible-deployment adding specs and docs repos https://review.openstack.org/163960 | 21:18 |
| jogo | so your saying having multiple base images is why we can't do this? | 21:18 |
| clarkb | jogo: its one reason ye | 21:18 |
| jogo | what is another reason? | 21:18 |
| jogo | we should really be optimizing this for shortest build time | 21:19 |
| clarkb | yes wheels on the mirror do that | 21:19 |
| clarkb | is what I am trying to say, we know how to solve this problem better | 21:19 |
| *** mattfarina has quit IRC | 21:19 | |
| clarkb | we just need to do it | 21:19 |
| jogo | its not just wheels, deb packages | 21:20 |
| clarkb | the other reason is that you can get clunky gate wedges by testing like this | 21:20 |
| clarkb | jogo: those can be mirrored too | 21:20 |
| clarkb | jogo: but that is less of a problem because we can cache them reliably | 21:20 |
| jogo | networks break on us all the time | 21:20 |
| jogo | and installing things takes time | 21:20 |
| clarkb | unlike wheels you just shove the ar files in dirs | 21:20 |
| jogo | even if its a simple install | 21:20 |
| *** erikmwilson has joined #openstack-infra | 21:20 | |
| *** tiswanso has quit IRC | 21:21 | |
| jogo | anyway ... | 21:21 |
| jeblair | we have to have working network to at least be able to update from reliable mirrers (probably that we run) | 21:21 |
| jogo | I agree mirroring wheels is better then what mtreinish and I are trying to to today | 21:21 |
| jeblair | we can't depend on having everything cached locally -- partly because we regularly go through phases where we can't build images for a week or more at a time | 21:22 |
| jogo | clarkb: just not even sure where to start for that one | 21:22 |
| dstufft | clarkb: you can totally just wheels in a dir | 21:22 |
| dstufft | just shove* | 21:22 |
| jogo | jeblair: whoa why can't we build images for wheeks on end? | 21:22 |
| clarkb | dstufft: yes but the deps required to do that are a pita | 21:22 |
| jeblair | we _should_ try to have everything locally though so that we don't saturate the network and to reduce the chances of impact | 21:22 |
| jeblair | jogo: because the world is a terrible place :( | 21:22 |
| clarkb | dstufft: we don't want to install libxml2-dev to build lxml for example | 21:22 |
| jeblair | jogo: all kinds of things | 21:22 |
| *** bdpayne has joined #openstack-infra | 21:23 | |
| *** erikmwilson has quit IRC | 21:23 | |
| jeblair | jogo: random github failures, external things change, base images change | 21:23 |
| greghaynes | or specifically, you dont want that installed on the resulting image, right? | 21:23 |
| *** erikmwilson_ is now known as erikmwilson | 21:23 | |
| greghaynes | I mention that because I think ive heard other hacky solutions, like using a second chroot for building the wheels | 21:23 |
| clarkb | greghaynes: ya | 21:23 |
| *** ayoung has quit IRC | 21:24 | |
| clarkb | greghaynes: and yes we could do hacky things like that but that becomes hard to debug, one chroot is hard enough | 21:24 |
| greghaynes | yep, agreed | 21:24 |
| jogo | jeblair: I guess that is where we take all the hits for external network issues, that makes sense | 21:24 |
| greghaynes | and at the end of the day I think its more work than mirroring. Then if you want to cache a wheel in the built image you can just download the wheel from the mirror at build time | 21:24 |
| clarkb | greghaynes: +1000 | 21:25 |
| clarkb | so ya how to mirror | 21:25 |
| jeblair | jogo: we try to keep the network traffic local though -- to our own mirrors. that helps tremendously | 21:25 |
| jeblair | jogo: most of the network issues we see now, when talking to our own mirrors, are due to capacity limitations in neutron | 21:25 |
| clarkb | we have a tool called pypi-mirror that will build wheels for every package given a list of deps. We can run this on centos/ubuntu/fedora nodes to make wheels when rsync them into our mirrors somehow | 21:25 |
| *** dkranz has quit IRC | 21:25 | |
| jeblair | jogo: apparently, there's only so much traffic you can shove through a virtual router | 21:25 |
| greghaynes | :D You know I love me some pypi-mirror | 21:26 |
| jogo | jeblair: :( boo neutron | 21:26 |
| clarkb | it will also build them in a way that is distro release specific so that solves the problem that pypi itself isn't tackling | 21:26 |
| clarkb | it may need a change to handle arch details if we decide that is necessary. Right now I think its only distro-release specific and doesn't account for arch | 21:26 |
| dstufft | um | 21:27 |
| greghaynes | oh? I know it makes an arch specific dir | 21:27 |
| dstufft | it acoutns for arch I think | 21:27 |
| clarkb | oh if it does even better | 21:27 |
| dstufft | or well wheels do I mean | 21:27 |
| dstufft | you can stick 32bit and 64bit linux wheels in the same dir | 21:27 |
| clarkb | dstufft: wheels do, pypi-mirror doesnt | 21:27 |
| clarkb | but maybe thats a non issue if wheels just work if in the same dir | 21:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Add Fedora 21 dib image https://review.openstack.org/163982 | 21:28 |
| *** koolhead17 has joined #openstack-infra | 21:28 | |
| *** koolhead17 has quit IRC | 21:29 | |
| jogo | interesting, never seen this faiure before http://logs.openstack.org/58/163958/1/gate/gate-infra-puppet-apply-centos6/7ffba5f/console.html#_2015-03-12_21_16_09_456 | 21:29 |
| rcarrillocruz | SpamapS: i think i found the problem: https://jenkins04.openstack.org/job/check-shade-dsvm-functional/1/console | 21:29 |
| *** tgohad has quit IRC | 21:29 | |
| rcarrillocruz | when we source openrc from devstack in the job, it contains OS_CACERT | 21:29 |
| rcarrillocruz | and that's loaded by os-client-config | 21:29 |
| clarkb | jogo: thats the git on centos6 is funny after libc update | 21:29 |
| jogo | clarkb: lol wow | 21:29 |
| rcarrillocruz | i'll put a change on the shade job to unset that variable after sourcing openrc | 21:30 |
| rcarrillocruz | we should be good then | 21:30 |
| greghaynes | clarkb: There were some fixes to make it work on an arch specific setup recently, so it might 'just work' | 21:30 |
| * jogo wonders if redhat can fix it | 21:30 | |
| *** koolhead_ has joined #openstack-infra | 21:30 | |
| greghaynes | http://git.openstack.org/cgit/openstack-infra/pypi-mirror/commit/pypi_mirror/cmd/run_mirror.py?id=4591e2abbc55c01a86180d1642a824f0188ffa8f | 21:30 |
| clarkb | jogo: we have been unable to root cause but git client deadlocks quite often since the great libc update | 21:30 |
| *** kevinbenton_ is now known as kevinbenton | 21:30 | |
| *** koolhead_ has quit IRC | 21:30 | |
| clarkb | jogo: it started right at the same time so we correlated the two but that may not be the actual cause | 21:31 |
| rcarrillocruz | or in the post-test-hook script rather... | 21:31 |
| clarkb | I really need a cron that keeps my local dib cache hot | 21:31 |
| *** teran has quit IRC | 21:32 | |
| greghaynes | well, if youre always building images with dib is stays pretty warm ;) | 21:32 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add initial compute functional tests to Shade https://review.openstack.org/158463 | 21:32 |
| *** timcline has quit IRC | 21:32 | |
| greghaynes | clarkb: so what would actually kick off the pypi-mirror runs? | 21:34 |
| clarkb | woot I wasn't crazywith my idea why the shade test failed | 21:34 |
| clarkb | greghaynes: zuul/jenkins | 21:34 |
| greghaynes | oh, just add a timer trigger? | 21:34 |
| clarkb | ya | 21:34 |
| greghaynes | \O/ my new found zuul knowledge is becoming useful | 21:34 |
| clarkb | and possibly trigger on global-requirements updates since we would likely just wheel the entirety of g-r | 21:34 |
| *** hdd_ has quit IRC | 21:36 | |
| jeblair | clarkb: is there a way to get wheels if available and fall back to what's in bandersnatch otherwise? | 21:37 |
| clarkb | yup | 21:37 |
| clarkb | jeblair: you set your main mirror index to be the wheels location (based on your distro release) then have an extra index url that points to the simple/ dir of the bandersnatch | 21:38 |
| jeblair | clarkb: so we'd still get new things instantly, and new versions instantly if someone had a req that required the new version. we'd get new versions that match open ended reqs daily? | 21:38 |
| *** ujuc has quit IRC | 21:38 | |
| clarkb | exactly | 21:38 |
| clarkb | and as greghaynes mentions the image builds could cache the wheels on the images from the mirror too | 21:39 |
| jeblair | oh nice | 21:39 |
| jeblair | though if a project with an open ended req and a wheel breaks us, we end up not finding out for a day | 21:39 |
| *** tqtran has quit IRC | 21:40 | |
| clarkb | jeblair: no I think it would grab the latest from bandersnatch mirror and fail early | 21:40 |
| greghaynes | hrm? youd do the same thing there, just extra-index-url into the on disk cache | 21:40 |
| openstackgerrit | Kevin Carter proposed openstack-infra/project-config: updates for os-ansible-deployment adding specs and docs repos https://review.openstack.org/163960 | 21:40 |
| clarkb | jeblair: because pip basically lists all the available options then picks the best match | 21:40 |
| greghaynes | yep | 21:40 |
| jeblair | clarkb: oh, i figured it would check wheels first and stop when it found a match there | 21:40 |
| jeblair | okay, then basically there's nothing lost? | 21:40 |
| cloudnull | sorry for all the reviews, that should be the "last one"-(tm). | 21:40 |
| greghaynes | AIUI that is the difference between using index-url and extra-index-url | 21:41 |
| greghaynes | not so much that wheels win, but whether the first match in an index wins | 21:41 |
| greghaynes | but I imagine dstufft will tell me im wrong ;) | 21:41 |
| dstufft | index-url replaces the default, extra-index-urla dds one | 21:42 |
| greghaynes | also, DIB has a pypi element to configure this kind of thing already (having multiple pip indexes) | 21:42 |
| dstufft | pip priortizes verison, then wheel, then order it finds it | 21:42 |
| jeblair | dstufft: so if we ask for foo >= 2.0 and index-url has egg 3.0 and extra-index-url has wheel 2.0, we get egg 3.0? | 21:43 |
| *** dboik_ has quit IRC | 21:43 | |
| dstufft | well pip doesn't support egg, but if you pretend egg is sdist then yes | 21:43 |
| jeblair | dstufft: let's prentend i'm bad at picking the right words sometimes, yes :) | 21:44 |
| *** dboik has joined #openstack-infra | 21:44 | |
| *** tqtran has joined #openstack-infra | 21:44 | |
| jeblair | and speeling them too | 21:44 |
| clarkb | unlike gerrit caches I think this sort of caching will do what we want :) | 21:44 |
| *** Sukhdev has joined #openstack-infra | 21:44 | |
| *** ihrachyshka has quit IRC | 21:44 | |
| clarkb | the important thing in dstufft's explanation of the best match is version is the highest weight | 21:44 |
| jeblair | sounds like a plan to me :) | 21:45 |
| dstufft | yea, we basically shove everytihng in a list in the order we find them, then do sorted(filter(ENSURE_MATCHES, the_list), key=lambda x: (x.version, x.is_wheel))[0] | 21:46 |
| openstackgerrit | Merged openstack-infra/project-config: Log stdout from build_wheels in cache_devstack https://review.openstack.org/163958 | 21:46 |
| dstufft | since pythons sort is stable that means we prefer version, format, then order we found it | 21:46 |
| clarkb | so ya if we can get a job template for running pypi-mirror to just make a wheel mirror I am sure we can resurrect the long lived slaves to run those jobs on | 21:47 |
| pleia2 | nibalizer: about? I have a question re: https://review.openstack.org/#/c/163267/ | 21:48 |
| clarkb | woah did all of those nodepool test changes merge when I wasn't looking? thats awesome. time for me to rebase the cli tests change | 21:48 |
| *** amotoki has quit IRC | 21:49 | |
| jogo | mtreinish: your patch is in https://review.openstack.org/163958 | 21:49 |
| jogo | mtreinish: now we wait and see / hope clarkb does his thing first | 21:50 |
| nibalizer | pleia2: lookin | 21:51 |
| pleia2 | nibalizer: so, I don't actually understand how it works now :) | 21:51 |
| nibalizer | haha | 21:51 |
| *** prad has quit IRC | 21:52 | |
| pleia2 | nibalizer: I would assume that like $sysadmins we'd want to do something like = [], but apparently I don't know what's going on | 21:52 |
| nibalizer | we do that on line 5? | 21:52 |
| pleia2 | I mean, $root_rsa_key would be the same | 21:53 |
| pleia2 | since it's a variable from hiera, I think | 21:53 |
| pleia2 | setting it to xxx would break things, no? | 21:53 |
| nibalizer | there are a couple ways to break | 21:53 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/project-config: Ignore missing packages https://review.openstack.org/163976 | 21:53 |
| nibalizer | if you just want a the puppetmaster class to apply so you can prototype/test/kick | 21:53 |
| nibalizer | putting AAAA into /root/.ssh/id_rsa is fine | 21:53 |
| asselin | greghaynes, clarkb ^^ | 21:53 |
| nibalizer | if you really want infra to come up, you'll need to pipe a real rsa key into that parameter | 21:54 |
| pleia2 | nibalizer: yeah, so right now it doesn't magically pull in from hiera? | 21:54 |
| *** bknudson has quit IRC | 21:55 | |
| pleia2 | (I swear, I am trying to say magic less WRT puppet) | 21:55 |
| pleia2 | in short: right now it wouldn't actually work | 21:56 |
| *** annegentle has quit IRC | 21:56 | |
| *** mriedem is now known as mriedem_away | 21:56 | |
| clarkb | asselin: cool, I had just started a dib run to test that | 21:56 |
| clarkb | asselin: will rerun with latest patch as soon as current one fails | 21:56 |
| asselin | clarkb, ok | 21:57 |
| nibalizer | pleia2: it gets looked up here http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n175 | 21:57 |
| *** mjturek1 has left #openstack-infra | 21:58 | |
| *** che-arne has joined #openstack-infra | 21:58 | |
| pleia2 | nibalizer: right, so wouldn't hard-setting it in the config like that reset the value to xxx for our infra? | 21:58 |
| nibalizer | but in the second box http://ci.openstack.org/puppet.html it says to use 'include o_p:puppetmaster' | 21:58 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Add nodepool command tests https://review.openstack.org/155458 | 21:58 |
| nibalizer | which doesn't work any more | 21:58 |
| nibalizer | pleia2: no if you explicitly pass in a value that takes precedence over the default | 21:59 |
| pleia2 | ah, ok, so it will prefer the hiera value | 21:59 |
| jeblair | nibalizer: so it wasn't mordred's fault... i did a password reset by email, and still can't log in to puppet forge | 22:00 |
| pleia2 | nibalizer: I think I understand now, thanks :) | 22:00 |
| *** asettle has quit IRC | 22:00 | |
| clarkb | jeblair: fungi greghaynes anyone else interested in nodepool testing ^ that above change should cover a good portion of our command line tools | 22:01 |
| clarkb | wil need more tests but that lays the groundwork for them | 22:01 |
| *** melwitt_ has joined #openstack-infra | 22:01 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:02 | |
| *** melwitt_ has quit IRC | 22:02 | |
| *** harlowja has quit IRC | 22:02 | |
| openstackgerrit | Merged openstack-infra/zuul: Log merger job uuids https://review.openstack.org/163936 | 22:02 |
| *** melwitt has quit IRC | 22:03 | |
| clarkb | jroll: JayF trying to determine if rackspace swift implementation allows us to set X-Delete-At/X-Delete-After on tempurl posted files. Should I file a ticket to figure that out? | 22:03 |
| *** peristeri has quit IRC | 22:03 | |
| JayF | clarkb: I honestly don't know for 100% sure but I suspect yes; I know we expose functionality that seems similar to that in our UI (which is mostly how I interact with swift these days) | 22:04 |
| jroll | clarkb: that's the best way, but hang on lemme check | 22:04 |
| jeblair | nibalizer: hah, it was mordred's fault ;) the account was not confirmed. | 22:04 |
| *** sarob has quit IRC | 22:04 | |
| *** harlowja has joined #openstack-infra | 22:04 | |
| *** e0ne has quit IRC | 22:05 | |
| rcarrillocruz | weeee | 22:05 |
| rcarrillocruz | https://jenkins06.openstack.org/job/check-shade-dsvm-functional/2/console | 22:05 |
| rcarrillocruz | SpamapS: clarkb ^ | 22:05 |
| rcarrillocruz | unsetting OS_CACERT did it | 22:06 |
| jeblair | nibalizer, mordred: i have confirmed the puppetforge account and reset the password back to what's in hiera | 22:06 |
| clarkb | rcarrillocruz: nice | 22:06 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: fix test in include-raw-escape001.yaml https://review.openstack.org/163995 | 22:06 |
| clarkb | rcarrillocruz: I will have to rereview that change now | 22:06 |
| jeblair | nibalizer: what do i need to do next? create an openstackinfra-httpd module? | 22:06 |
| rcarrillocruz | yeah, i will address your sync/async thing, was really focused on getting it working | 22:06 |
| rcarrillocruz | so feel free to re-review that again | 22:07 |
| clarkb | rcarrillocruz: ok | 22:07 |
| rcarrillocruz | will also address GheRivero comments as well for the util module | 22:07 |
| *** amitgandhinz has quit IRC | 22:07 | |
| jeblair | nibalizer: or will that happen automatically when we tag / upload? and my only task was to fix the password? :) | 22:07 |
| clarkb | http://logs.openstack.org/58/155458/3/check/nodepool-coverage/4794ce2/cover/ | 22:07 |
| clarkb | jeblair: ^ we can use that to focus what we test and track our progress | 22:08 |
| jeblair | clarkb: neato! | 22:08 |
| jeblair | ah, that's the rebase cool | 22:08 |
| *** Longgeek has joined #openstack-infra | 22:08 | |
| pleia2 | are we ok with adding bots our to #rdo- channels? https://review.openstack.org/#/c/163331/ | 22:09 |
| clarkb | yup so it has coverage for the cli commands | 22:09 |
| greghaynes | oh hey, my test image uploading deal merged | 22:09 |
| greghaynes | so theres coverage on that too now | 22:09 |
| anteaya | pleia2: ah not sure we want to bot all of freenode | 22:10 |
| nibalizer | jeblair: you must manually create the module on the web thing yes | 22:11 |
| nibalizer | then tag /should/ upload | 22:11 |
| jhesketh | Morning | 22:11 |
| nibalizer | or when you push a tag it should auto upload | 22:11 |
| jeblair | nibalizer: it wants me to upload a tarball | 22:11 |
| pleia2 | anteaya: indeed | 22:11 |
| anteaya | pleia2: not exactly sure where the line is, but pretty sure that is on the other side | 22:11 |
| anteaya | morning jhesketh | 22:11 |
| nibalizer | jeblair: orly | 22:11 |
| pleia2 | anteaya: that's my feeling too | 22:11 |
| crinkle | nibalizer: yeah you can't just create an empty module | 22:11 |
| nibalizer | jeblair: it wont let you 'create' without uploading a tarball | 22:11 |
| jeblair | nibalizer: http://paste.openstack.org/show/191960/ | 22:11 |
| jeblair | nibalizer: text from page | 22:11 |
| nibalizer | gahhhhhhhh puppetforge y u so terrible | 22:11 |
| clarkb | jhesketh: good morning. The change for X-Delete-After got in and didn't cause uploads to break but it isn't setting that header on objects. I think because rackspace swift may not allow us to set that header wen using tempurls | 22:11 |
| *** tsg has joined #openstack-infra | 22:12 | |
| openstackgerrit | Sumit Naiksatam proposed openstack-infra/project-config: Create experimental job for group-based-policy https://review.openstack.org/161511 | 22:12 |
| crinkle | nibalizer: i don't think they thought having an empty module was a valid use case | 22:12 |
| jeblair | crinkle, nibalizer: but maybe an initial upload via the automation will work? | 22:12 |
| clarkb | jhesketh: swift has several config options to limit headers for tempurl usage. My next step was to submit a ticket with rackspace to confirm | 22:12 |
| nibalizer | jeblair: worth a shot | 22:12 |
| crinkle | is there documentation on the blacksmith thing about it? | 22:12 |
| clarkb | jhesketh: unless you have other ideas? I also ran the upload script locally commenting out the request.post to check all the parameters to requests.post and they look good | 22:12 |
| jeblair | nibalizer: I39bafac1e7f85b0d14ec5d6626779985093c54f7 has merged; is the repo ready to go? | 22:12 |
| *** ghostpl_ has joined #openstack-infra | 22:12 | |
| * anteaya goes for a walk | 22:13 | |
| jeblair | nibalizer: version number? semver style? 0.1.0 ? | 22:13 |
| *** otter768 has joined #openstack-infra | 22:13 | |
| pleia2 | anteaya: enjoy | 22:13 |
| anteaya | thanks | 22:13 |
| * jeblair has finished typing his passphrase and is standing by :) | 22:14 | |
| nibalizer | jeblair: version i metadata.json is 0.0.1 | 22:15 |
| clarkb | greghaynes: looking at the coverage node deletion isn't well handled at all and the gearman/zmq/ssh client stuff is pretty untouched. Those bug me less though because they do work and aren't touched much | 22:15 |
| nibalizer | and forge will listen to whats in metadata.json | 22:15 |
| jeblair | nibalizer: okay, will tag 0.0.1 then | 22:15 |
| greghaynes | clarkb: ah. I was going to try and poking at a few more command tests | 22:15 |
| jhesketh | clarkb: hmm, a ticket is probably a good idea. mattoliverau had mentioned in the past that it was enabled on rax. | 22:15 |
| greghaynes | clarkb: since those are low hanging | 22:15 |
| jeblair | nibalizer: er, have we figured out how to put that into the json file during the tag build job? | 22:15 |
| *** Ryan_Lane has joined #openstack-infra | 22:15 | |
| jeblair | i'm very keen on tag-based post versioning | 22:16 |
| jhesketh | mattoliverau: what do you know of X-Delete-After? | 22:16 |
| nibalizer | you can also cd into puppet-httpd and 'puppet module build' and that will make a tarball and that tarball could be the inital upload if plan A doesn't work | 22:16 |
| ianw | jeblair: are your comments on https://review.openstack.org/#/c/153904/ (Add generate-log-config command) a -2? it's not clear to me if there is a discussion to be had or it's not going in | 22:16 |
| mattoliverau | Lots, but do you mean regarding formpost? | 22:16 |
| *** eharney has quit IRC | 22:16 | |
| nibalizer | jeblair: we're managin the json file in git... so unless we want to do the pbr thing i thing we just have to be responsible and make sure they line up | 22:16 |
| mattoliverau | jhesketh: ^ | 22:16 |
| jeblair | nibalizer: we want to do the pbr thing | 22:16 |
| clarkb | greghaynes: go for it. But I definitely think node deletion should be high on our list. Also the method that is run by the cron thing to periodically update images is not well covered | 22:17 |
| *** nelsnels_ has joined #openstack-infra | 22:17 | |
| ianw | clarkb: if you want to test all your new features, https://review.openstack.org/#/c/163982/ (https://review.openstack.org/#/c/163982/) ... works for me but always willing to be surprised by nodepool in production... | 22:17 |
| clarkb | greghaynes: so low hanging fruit is good as is covering deletion and running the cron function | 22:17 |
| jhesketh | mattoliverau: yes and it's deployment in public cloud | 22:17 |
| greghaynes | clarkb: sounds good | 22:17 |
| jeblair | nibalizer: it doesn't have to be the full pbr thing, but if we can omit the version info from metadata.json in git, and then have the build job insert the correct version info, that will be enough | 22:17 |
| clarkb | ianw: I will need to restart nodepool daemon first | 22:17 |
| clarkb | ianw: I can do that nowish | 22:17 |
| jeblair | nibalizer: we've done something similar with java, i believe. possibly other systems | 22:18 |
| crinkle | blacksmith has a 'bump' task and a 'tag' task http://kartar.net/2014/10/using-puppet-blacksmith/ | 22:18 |
| crinkle | maybe relevant | 22:18 |
| *** carl_baldwin has quit IRC | 22:18 | |
| *** otter768 has quit IRC | 22:18 | |
| mattoliverau | jhesketh: let me check to see if rax supports it.. X-delete-after in formpost has been in the code base for a year now! | 22:18 |
| *** dimtruck is now known as zz_dimtruck | 22:19 | |
| jhesketh | mattoliverau: cool, thanks | 22:19 |
| clarkb | mattoliverau: so the behavior I see is that if I `swift upload --header 'X-Delete-After:300' container object` that works fine | 22:19 |
| *** nelsnelson has quit IRC | 22:19 | |
| nibalizer | jeblair: so the metadata.json not being it git is a problem for anyone trying to use the module from git | 22:19 |
| nibalizer | so two bad options i guess | 22:19 |
| ianw | clarkb: sure; new-ish redirects to images have made the f21 build just look like everything else | 22:19 |
| jeblair | nibalizer: does it need to have the version info in there? | 22:19 |
| ianw | which is nice | 22:19 |
| clarkb | mattoliverau: but the tempurl posts that we do doesn't seem to and I found taht swift has config options to whitelist/blacklist specific headers when using tempurl | 22:19 |
| crinkle | jeblair: yes | 22:19 |
| nibalizer | jeblair: yea | 22:19 |
| clarkb | jhesketh: tempurl is what we are using right? | 22:19 |
| *** ghostpl_ has quit IRC | 22:20 | |
| jeblair | nibalizer, crinkle: does it have to be accurate? | 22:20 |
| crinkle | yes, the forge depends on it | 22:20 |
| clarkb | ianw: did your yaml check thing get merged too? /me goes to check on it | 22:20 |
| jeblair | crinkle: no not the forge -- the forge gets the correct version when we build the release and upload it | 22:20 |
| *** hashar has quit IRC | 22:20 | |
| jeblair | i'm just talking about people using something from git | 22:21 |
| ianw | clarkb: not yet, when it does i'll work on adding a job to use it | 22:21 |
| crinkle | oh, in that case there's not a technical reason for it to be accurate, but it might be confusing if it's not | 22:21 |
| jhesketh | clarkb: not quite, we're using the formpost middleware | 22:21 |
| mattoliverau | clarkb, well we are using the form post middleware, but needs the tempurl.. For x-delete-after in form post you pass it in via the form | 22:21 |
| clarkb | mattoliverau: OH we pass it via the form not the header? | 22:21 |
| jeblair | crinkle: it can't be accurate anyway. the last 10 commits have all claimed to be version 0.0.1, but only the last one really is. | 22:21 |
| jhesketh | Ah, so it just needs to be part of the payload | 22:22 |
| clarkb | jhesketh: so thats probably it then we need to put the value in the payload | 22:22 |
| *** yfried is now known as yfried|afk | 22:22 | |
| mattoliverau | But it'll only work if rax is using the latest version of form post | 22:22 |
| jhesketh | clarkb: yep | 22:22 |
| clarkb | jhesketh: also other bug I noticed is that since the command like arg is a string if delete_after when delete_after is '0' will be true | 22:22 |
| jhesketh | mattoliverau: do you have a way of determining that? | 22:22 |
| *** luqas has joined #openstack-infra | 22:22 | |
| clarkb | jhesketh: we should have a check there that says and delete_after != '0' | 22:22 |
| crinkle | jeblair: i'm confused, do you want to have a new version on every commit? | 22:22 |
| clarkb | mattoliverau: jhesketh what is the purpose behind only accepting these headers in the form? | 22:23 |
| mattoliverau | Yup, I'll take a look via info I hope.. Or rax docs or failing that just try | 22:23 |
| clarkb | I think I am missing the utility of that functionality | 22:23 |
| *** dkliban is now known as dkliban_afk | 22:23 | |
| mattoliverau | Its so you can simply use tempurl key + a simple http forms to push data | 22:24 |
| jeblair | crinkle: so the way we have all of our python (and some other stuff like java, etc) set up is that when you tag the repo, our ci system automatically builds the package and the version embedded in the package is the name of the git tag. | 22:24 |
| jeblair | crinkle: so all i have to do to release zuul is "git tag -s 3.0.0; git push gerrit 3.0.0". | 22:24 |
| jhesketh | clarkb: can't we make argparse give us an int? | 22:24 |
| jeblair | crinkle: no commits to change the version number that have to land in the right order, etc... | 22:24 |
| nibalizer | we could generate metadata.json from a template pretty easily | 22:24 |
| mattoliverau | Form post pulls the data out of the form data and passes it along | 22:25 |
| jeblair | crinkle: so my main concern is that i want that to be the release process for the puppet modules too | 22:25 |
| clarkb | jhesketh: ya we could do that too then coerce to string when we put it in the payload | 22:25 |
| jeblair | crinkle: strictly speaking, that just means that the metadata.json embedded in the tarball that the tag job creates needs to get the version number from the tag | 22:25 |
| nibalizer | crinkle: do you think people who clone from git are expecting a metadata.json file? i think it is reasonable to expect that but I'm not sure what exactly would break | 22:25 |
| clarkb | jhesketh: pretty sure we need a string/bytes in the payload because http | 22:25 |
| clarkb | jhesketh: either way will work | 22:25 |
| jeblair | crinkle, nibalizer: beyond that, i guess, whatever we need for git to work :) | 22:26 |
| nibalizer | jeblair: my plan was to have the person doing the tag make sure that the tag matched metadata.json | 22:26 |
| crinkle | nibalizer: i don't think anything would break | 22:26 |
| mordred | nibalizer: the problem is that we have this code review system | 22:26 |
| jeblair | nibalizer: we're not smart enough to remember to do that | 22:26 |
| mordred | aslo, what jeblair said | 22:26 |
| jhesketh | clarkb: oh the payload needs to be a string.. | 22:26 |
| mordred | version numbers in files in git repos is ew | 22:26 |
| clarkb | jhesketh: ya, but we can coerce after checking thats easy enough | 22:26 |
| jhesketh | Yep | 22:26 |
| nibalizer | so the way they do it on github is they make a review 'prep for 0.0.2 release' | 22:26 |
| crinkle | i'm used to bumping versions being a code review process | 22:27 |
| nibalizer | which bumps metadata.json, changelog, whatevs | 22:27 |
| clarkb | jhesketh: should I write the change or would you like to? | 22:27 |
| nibalizer | then when that lands they release it | 22:27 |
| crinkle | because someone could be like "no that's a feature bump not a bugfix bump" or something | 22:27 |
| jhesketh | clarkb: want me to make these changes since you seem busy | 22:27 |
| jhesketh | Heh, I'm happy to :-) | 22:27 |
| crinkle | yeah what nibalizer said | 22:27 |
| nibalizer | for us, it would be 'when that lands, pull it, tag, push' | 22:27 |
| clarkb | jhesketh: sure, I am only juggling a couple things at the moment :) | 22:27 |
| clarkb | jhesketh: but ya it would help | 22:27 |
| mattoliverau | jhesketh, clarkb: according to RAX cloudfiles docs, the X-Delete-At is in formpost.. good indication they are using the later formpost: http://docs.rackspace.com/files/api/v1/cf-devguide/content/Create_the_Form-d1a777.html | 22:27 |
| jeblair | crinkle: so continuing the story about python -- we wrote a thing called pbr that we include in all our python projects which knows how to calculate a unique version number for something in a git repo, so you can still run a program and have it report a version, without having to have that version in code itself (it reads tags and appends a discriminator to the most recent version) | 22:28 |
| jhesketh | clarkb: I'll take a look shortly (meeting first) | 22:28 |
| jeblair | crinkle: we don't necessarily have to have something like that for puppet modules -- only if there's some reason that modules installed from git need to have a version number | 22:28 |
| mordred | nibalizer: right - thing is, we tend to think that process is ugly and convoluted | 22:28 |
| mordred | nibalizer: because the correct info is in the tag - so repeating it in a file is a waste of effort and prone to error | 22:29 |
| clarkb | +1000 | 22:29 |
| clarkb | git is a version control system | 22:29 |
| mordred | yup | 22:29 |
| mordred | it's great at controlling versions | 22:29 |
| jeblair | once you release something with 'git push' you'll never want to release something any other way | 22:29 |
| nibalizer | right and things like cvs and rcs had a magic flag that you could write into a file, and it would be expanded to version whenever it was checked out | 22:29 |
| greghaynes | Yes, can confirm - the number of times I have tagged a release and forgotten to bump a version is > 0 | 22:29 |
| mordred | ++ | 22:29 |
| nibalizer | git doesn't have that | 22:29 |
| *** patrickeast_ has quit IRC | 22:29 | |
| mordred | nibalizer: nope. don't want | 22:29 |
| nibalizer | mordred: its hacking around the same problem | 22:30 |
| mordred | nibalizer: that's all a hack for compatability with when tarballs were the main method of things | 22:30 |
| mordred | nibalizer: there is no problem if the tools aren't broken | 22:30 |
| mordred | and trying to exist in 1994 | 22:30 |
| mordred | nibalizer: I mean, dont' get me wrong ... the "let's put a version number ina file our git repo" is a pretty common pattern | 22:31 |
| *** camunoz has joined #openstack-infra | 22:31 | |
| mordred | nibalizer: we've just fixed it for our other languages/repos | 22:31 |
| clarkb | 161952 is another dib-nodepool test related change would be good to get that one reviewed | 22:31 |
| *** andreykurilin_ has joined #openstack-infra | 22:32 | |
| mordred | nibalizer: I'd be happy to write a patch for $something to have $something grok the pattern | 22:32 |
| mordred | but I'm not sure what $something would be | 22:32 |
| nibalizer | ya im trying to think of what the consumng points are | 22:32 |
| mordred | woudl it be the puppet module command? | 22:32 |
| *** Longgeek has quit IRC | 22:32 | |
| mordred | becasue, the only thing metadata.json in a git repo would be used for programmatically is figuring out depends, yeah? | 22:33 |
| nibalizer | probably | 22:33 |
| crinkle | yes | 22:33 |
| nibalizer | so if you remove metadata.json and run 'puppet module list' it gives ├── concat (???) instead of ├── puppetlabs-concat (v1.1.2) | 22:33 |
| nibalizer | unicodepipebars are theirs not mine | 22:34 |
| mordred | if that's true - then perhaps a patch to puppet module that says "if metadata.json has no version number and it's in a git repo, look for git tag output instead" | 22:34 |
| mordred | nibalizer: aww... I thought you'd typed the pipebars :) | 22:34 |
| jeblair | did we decide whether the version key is necessary in the metadata.json file in the git repo? so could we try removing that key and then injecting it into the file during the tarball build? | 22:34 |
| nibalizer | jeblair: ill try that | 22:34 |
| openstackgerrit | Merged openstack/requirements: Bump rtslib version to have config persistence https://review.openstack.org/161291 | 22:34 |
| mordred | if it works, I'll work on trying to make a patch that says the thing I said above | 22:35 |
| jeblair | nibalizer: i'll look into modifying the tarball job | 22:35 |
| nibalizer | jeblair: haha thats actually worse | 22:35 |
| *** tiswanso has joined #openstack-infra | 22:35 | |
| nibalizer | when metadata.json isn't present 'puppet module' kind rolls with it, when metadata.json doesn't have 'verision' it breaks pretty hard | 22:35 |
| nibalizer | pretend for a second that i can spell | 22:36 |
| *** mpopow_ has joined #openstack-infra | 22:36 | |
| *** melwitt has joined #openstack-infra | 22:36 | |
| *** mpopow has quit IRC | 22:36 | |
| *** fdegir has quit IRC | 22:36 | |
| nibalizer | the thigns we produce are 1) a git repo and 2) a forge module | 22:36 |
| crinkle | so what if you just modified the job to sed the version in the metadata.json before building? | 22:36 |
| crinkle | then it would be accurate and puppet module would be happy | 22:36 |
| jeblair | so we could (a) remove it and generate the whole thing from a template (yaml file? ;) or (b) stick "version": "0.0.0" in it and then overwrite it | 22:36 |
| mordred | nibalizer, crinkle: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/module_tool/metadata.rb is the place I'd do it if I wanted to try to make module smarterer | 22:36 |
| jroll | clarkb: "Just to be clear, you want to know if you can add them when using FormPOST" | 22:37 |
| mordred | jeblair: (b) sounds like it would break the least number of things right now | 22:37 |
| mordred | ? | 22:37 |
| nibalizer | the things that consume 1) and 2) are a) r10k b) puppet-librarian c) puppet-module tool d) random shell scripts | 22:37 |
| jeblair | and i think (b) is also crinkle's suggestion? | 22:37 |
| mordred | or - rather - less work, and break the same amount of things | 22:37 |
| nibalizer | i like (b) | 22:37 |
| crinkle | they are not super open to the idea of puppet module talking to git https://tickets.puppetlabs.com/browse/BKR-21 | 22:38 |
| clarkb | jroll apparently. I got educated and mattoliverau says we can so I think we should be good | 22:38 |
| jroll | clarkb: oh, ok, cool | 22:38 |
| *** krisgan has joined #openstack-infra | 22:38 | |
| mordred | crinkle: oh. that makes me sad | 22:38 |
| clarkb | jroll just need to move from header to form data | 22:38 |
| jroll | ah | 22:38 |
| nibalizer | yea 0.0.0 as version is enough to say 'ya you did something funky, have fun with that' | 22:39 |
| *** thedodd has quit IRC | 22:39 | |
| mordred | crinkle: it seems they aren't in favor of even using puppet mpdule | 22:39 |
| mordred | cinerama: and instead say "if you aren't using r10k go away" | 22:40 |
| mordred | gah | 22:40 |
| mordred | crinkle: ^^ | 22:40 |
| mordred | (at least, that's my reading of that issue) | 22:40 |
| *** tiswanso has quit IRC | 22:40 | |
| nibalizer | yea everybody loves r10k | 22:40 |
| * nibalizer likes our shell script | 22:40 | |
| crinkle | yeah i'm not sure they're considering `puppet module build` and `puppet module list` | 22:41 |
| *** tim_o_ has joined #openstack-infra | 22:41 | |
| crinkle | which aren't in r10k yet afaik | 22:41 |
| jeblair | nibalizer: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/puppet-module-jobs.yaml#n62 | 22:41 |
| fungi | every time i see r10k i think of sgi mips systems | 22:41 |
| jeblair | nibalizer: is it possible for a project to override "rake module:push" to do something locally? | 22:41 |
| mordred | nibalizer: I like our shell script too | 22:41 |
| fungi | (e.g. risc 10000) | 22:41 |
| fungi | my age is showing i guess | 22:42 |
| jeblair | fungi: isn't that usually rs10k? | 22:42 |
| *** yfried|afk is now known as yfried | 22:42 | |
| fungi | yep | 22:42 |
| fungi | or r10000 | 22:42 |
| fungi | the oxygen series | 22:42 |
| nibalizer | jeblair: probably.. but i think you just want to do the metadata.json munging before you call that | 22:42 |
| * jeblair scores a fungus | 22:42 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Add details on using git-review over https https://review.openstack.org/161078 | 22:43 |
| jeblair | nibalizer: i was actually asking for a different reason... | 22:43 |
| cinerama | aww i was all excited someone was talking to me :( | 22:43 |
| pleia2 | cinerama: thanks for the latest patchset! :) | 22:43 |
| jeblair | nibalizer: we have a rube-goldberg thing in place for most of the build/upload jobs where we build on one node, put it in tarballs.o.o, then run another job on the pypi node which downloads that and uploads it | 22:43 |
| jeblair | nibalizer: because if someone can run arbitrary code on the pypi node, then they can steal the creds | 22:44 |
| cinerama | pleia2: more on the way...just duking it out with Krusty's Non Narkotik Kough Syrup | 22:44 |
| fungi | heyhey kids | 22:44 |
| pleia2 | cinerama: understandable, gives me time to scope out what exactly we want from this specific patch before we call it "finished" anyway :) (fwiw, I think we're close) | 22:44 |
| *** tkelsey has joined #openstack-infra | 22:44 | |
| jeblair | nibalizer: so i think we probably need to do that for this too, and then replace "rake module:push" with something that doesn't run arbitrary code | 22:44 |
| nibalizer | jeblair: nothing in rake module:push is executing arbitrary code though | 22:45 |
| nibalizer | if you consider blacksmith trusted i guess | 22:45 |
| nibalizer | which we may not | 22:45 |
| fungi | i didn't get a photo with krusty, but i do have one from last week of me with duffman. i'll have to dig that up | 22:45 |
| jeblair | nibalizer: can you change Rakefile to make it do something else on "rake module:push" ? | 22:45 |
| jeblair | nibalizer: i think we can trust blacksmith | 22:45 |
| jeblair | nibalizer: it's the local repo we can't trust | 22:45 |
| mordred | cinerama: I mean, you can be involved in the r10k conversation if you'd like! | 22:45 |
| *** dims_ has joined #openstack-infra | 22:46 | |
| jeblair | cinerama: rs10k! | 22:46 |
| *** dims has quit IRC | 22:46 | |
| mordred | I want to install AIX on one of our nodes now ... | 22:46 |
| pleia2 | haha | 22:47 |
| fungi | ultrix ftw | 22:47 |
| nibalizer | crinkle: ^? can you sneak in to a rakefile? | 22:47 |
| crinkle | nibalizer: i'm experimenting | 22:48 |
| *** dboik has quit IRC | 22:48 | |
| dstufft | mordred: oh man life decisions | 22:48 |
| jesusaurus | mordred: you could make a bare-aix... for.... puppet compatibility... i guess | 22:48 |
| dstufft | mordred: I have a TODO list to figure out how to run CI on AIX | 22:48 |
| *** dboik has joined #openstack-infra | 22:48 | |
| *** r-daneel has quit IRC | 22:48 | |
| nibalizer | jeblair: we could just overwrite the rakefile with only enough smarts to do blacksmith | 22:48 |
| dstufft | and QNX | 22:48 |
| nibalizer | obviously we are not gonna need to run lint tests or anything else at that point | 22:48 |
| dstufft | and a bunch of other random things | 22:48 |
| * fungi now wonders if irix is still getting any love since it was so rudely abandoned | 22:48 | |
| cinerama | boy i sure haven't done aix in a while | 22:48 |
| *** tkelsey has quit IRC | 22:49 | |
| mordred | nibalizer: does the rakefile need to be in the module upload? | 22:49 |
| *** amotoki has joined #openstack-infra | 22:49 | |
| * fungi worries he has accidentally brought on friday an roughly hour early | 22:49 | |
| crinkle | yeah you can totally overload it | 22:49 |
| mordred | nibalizer: by that, I mean, does rake module:push make a tarball? or can it operate on a tarball that's been made already | 22:49 |
| crinkle | gross | 22:49 |
| *** ayoung has joined #openstack-infra | 22:50 | |
| *** nelsnels_ has quit IRC | 22:50 | |
| fungi | computering is a gross and untidy business | 22:50 |
| * mordred avoids computering as much as possible and washes his hands afterwards | 22:51 | |
| nibalizer | mordred: no the rakefile doesn't need to be in the artificat that is uploaded | 22:51 |
| nibalizer | also i dont think there is really value in splitting module build and module upload | 22:52 |
| nibalizer | they are both determinisitc tasks | 22:52 |
| fungi | debconf cfp is out now, if anybody wants to see heidelberg in august | 22:52 |
| *** yfried is now known as yfried|afk | 22:52 | |
| nibalizer | the only question is can they be overrided | 22:52 |
| *** achanda has quit IRC | 22:52 | |
| jeblair | nibalizer: well, that's the only reason we split them :) | 22:52 |
| jeblair | and it's important | 22:52 |
| mordred | nibalizer: so, you're suggesting have build and publish go like "system writes out new Rakefile, does build, does upload" | 22:53 |
| *** achanda has joined #openstack-infra | 22:53 | |
| crinkle | nibalizer: yes you can add a task to the rakefile with the name module:push and rake will run both | 22:53 |
| mordred | jeblair: I like the "write out Rakefile and don't use the one from the repo" approach | 22:53 |
| jeblair | because "cat /path/to/pypi/credentials; cat /path/to/puppetforge/credentials" would be a bad think to put in the rakefile | 22:53 |
| clarkb | wow google code is now deadish | 22:54 |
| fungi | eh?!? | 22:54 |
| jeblair | clarkb: i think it may have been dead for a few years | 22:54 |
| fungi | you mean even less usable than it used to be? | 22:54 |
| mordred | clarkb: but code hosting run by single companies is a great idea! | 22:54 |
| jeblair | okay, so we either need to start the job by overwriting the rakefile, or doing the split thing we do elsewhere | 22:54 |
| mordred | ++ | 22:54 |
| *** amotoki has quit IRC | 22:55 | |
| fungi | either of those options is fine by me | 22:55 |
| fungi | both should be safe | 22:55 |
| clarkb | no new projects as of today | 22:55 |
| mordred | and if we do the split thing, we need to not use rake module:upload in the second job | 22:55 |
| clarkb | and offline january of next year | 22:55 |
| jeblair | right, i forgot to add that includes something like a curl-based uploader | 22:55 |
| fungi | clarkb: woah! so add to the codehaus and gitlab die-offs then | 22:55 |
| crinkle | jeblair: what is the risk when someone overrides the task? the task is only called when someone pushes a tag, correct? so only trusted people can run it? | 22:55 |
| *** baoli has quit IRC | 22:55 | |
| jeblair | crinkle: yeah, but we don't trust everyone who can push a tag; we want this to be available to anyone on the system | 22:56 |
| *** achanda_ has joined #openstack-infra | 22:56 | |
| jeblair | crinkle: so not just infra projects, but openstack and stackforge ones too | 22:56 |
| crinkle | oh I see | 22:56 |
| *** annegentle has joined #openstack-infra | 22:56 | |
| *** achanda_ has quit IRC | 22:56 | |
| *** achanda_ has joined #openstack-infra | 22:56 | |
| fungi | er, not gitlab but gitorious (the former bought and is closing down the latter) | 22:57 |
| *** ddieterl_ has quit IRC | 22:57 | |
| jesusaurus | gitorious is shutting down? | 22:57 |
| jesusaurus | :( | 22:57 |
| *** achanda has quit IRC | 22:57 | |
| pleia2 | jesusaurus: yes :( | 22:57 |
| nibalizer | so i think echo > Rakefile | 22:57 |
| nibalizer | maybe we need to require it | 22:58 |
| nibalizer | ill do some testing | 22:58 |
| *** melwitt has quit IRC | 22:59 | |
| *** sdake has quit IRC | 23:00 | |
| openstackgerrit | Stephanie Miller proposed openstack-infra/puppet-zanata: Add apache configuration for zanata https://review.openstack.org/164011 | 23:00 |
| *** dims_ has quit IRC | 23:00 | |
| *** otter768 has joined #openstack-infra | 23:00 | |
| *** dims has joined #openstack-infra | 23:00 | |
| *** erlon has quit IRC | 23:01 | |
| *** dpyzhov_ has joined #openstack-infra | 23:02 | |
| *** annegentle has quit IRC | 23:02 | |
| clarkb | https://review.openstack.org/#/c/163976/ builds an image locally after it was failing locally | 23:02 |
| clarkb | mordred: jeblair fungi pleia2 ^ any chance we can get a second reviewer to look at that? I can kick off new dib build for precise and trusty after that merges | 23:03 |
| fungi | looking now | 23:03 |
| zaro | clarkb: i get these errors when running nodepool tests locally: "Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)". | 23:03 |
| clarkb | zaro: you need to have a mysql server running | 23:03 |
| zaro | do i need to install something to get it to work? | 23:03 |
| zaro | need a local mysql? | 23:03 |
| *** dangers is now known as dangers_away | 23:03 | |
| clarkb | zaro: I have a docker container with mysql in it that I start and run tests in | 23:03 |
| mordred | clarkb, fungi: +2 - I left off +A because fungi was looking | 23:04 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-httpd: Reduce version to 0.0.0 https://review.openstack.org/164013 | 23:04 |
| *** MarkAtwood has quit IRC | 23:04 | |
| *** melwitt has joined #openstack-infra | 23:05 | |
| *** dims has quit IRC | 23:05 | |
| *** dboik has quit IRC | 23:05 | |
| zaro | clarkb: you run test in it? why not just connect to it? | 23:06 |
| zaro | or is that not possible? | 23:06 |
| fungi | clarkb: asselin: i'm cool with 163976 and curious to know what packages specifically were causing grief there, but also wondering if we should look to do something similar with yum | 23:06 |
| clarkb | zaro: it is possible, its just easier to run in it because it looks for that unix socket | 23:07 |
| clarkb | fungi: ruby1.8 on trusty and docker.io on precise | 23:07 |
| fungi | clarkb: is that a recent change? | 23:07 |
| jeblair | nibalizer: i'm almost done with a job config change | 23:07 |
| jeblair | nibalizer: i'm futzing with some regexes | 23:07 |
| *** ayoung has quit IRC | 23:07 | |
| clarkb | fungi: the problem seems to be we take all packages for all devstack branches then try to cache them on a single release of ubuntu when reality is devstack isn't going to run all branches against both precise and trusty | 23:07 |
| *** andreykurilin_ has quit IRC | 23:07 | |
| mordred | clarkb: ++ | 23:08 |
| clarkb | fungi: I think it will only complicate caching if we try to encode which branches go to which releases during our image builds | 23:08 |
| clarkb | instead best effort caching is ideal | 23:08 |
| fungi | clarkb: ah! as soon as i whip bindep into shape i should look at integrating it into devstack | 23:08 |
| mordred | clarkb: devstack already has trusty/precise support internally | 23:08 |
| clarkb | fungi: its likely that yum will have similar problems | 23:08 |
| *** achanda has joined #openstack-infra | 23:08 | |
| mordred | clarkb: it should not be terribly hard to use it | 23:08 |
| clarkb | fungi: but I don't know enough about yum to make a good answer about it but I can look into it | 23:08 |
| mordred | clarkb: I mean, I thnk this patch is great | 23:08 |
| fungi | clarkb: that way devstack can declare distro release specific deps | 23:08 |
| mordred | because $things_happen | 23:09 |
| mordred | clarkb: it already does | 23:09 |
| mordred | gah | 23:09 |
| mordred | fungi: it already does | 23:09 |
| *** hichihara has joined #openstack-infra | 23:09 | |
| fungi | oh, but not in a way our nodepool scripts are able to consume | 23:09 |
| clarkb | ianw: any idea how to tell yum to download only and not complain too much if it can't download something in particular? | 23:09 |
| * clarkb tries a centos7 build locally | 23:09 | |
| mordred | fungi: disagree. I'll make a patch | 23:09 |
| nibalizer | jeblair: so what the way you detect the tag to use? | 23:10 |
| fungi | one of the next items on my list is to get nodepool running bindep against projects to determine what to cache, so if devstack used bindep to describe those then we could pretty easily just reuse the same in our nodepool image builds | 23:10 |
| mordred | fungi: http://git.openstack.org/cgit/openstack-dev/devstack/tree/files/debs/heat#n1 | 23:11 |
| jeblair | nibalizer: $ZUUL_REF | 23:11 |
| nibalizer | oh | 23:11 |
| ianw | clarkb: yumdownloader | 23:11 |
| mordred | fungi: all we have to do is read the dist: entry when we loop | 23:11 |
| nibalizer | im glad i asked :) | 23:11 |
| mordred | fungi: now - sure to bindep | 23:11 |
| nibalizer | would have taken a while | 23:11 |
| mordred | just saying, it's not really a hard thing to process | 23:11 |
| *** achanda_ has quit IRC | 23:11 | |
| ianw | clarkb: in fact yum not complaining if it can't download has been an issue for devstack, although not sure about network errors v actually missing packages | 23:11 |
| clarkb | ianw: thanks, I will look into that if the centos7 build breaks like precise/trusty were | 23:12 |
| fungi | mordred: oh, so it does have distro release identifiers and we were just ignoring them? yeah that's a quick-ish fix | 23:12 |
| mordred | yes | 23:12 |
| clarkb | fungi: mordred I wouldn't trust those completely | 23:12 |
| clarkb | the reason is master/kilo is currently only tested on trusty | 23:12 |
| clarkb | so we don't actually test if any of those deps are trusty only | 23:12 |
| fungi | i guess ideally if projects used by devstack had bindep manifests then devstack might be able to drop its rpms and debs lists entirely at that point | 23:12 |
| clarkb | same problem with icehouse being precise | 23:13 |
| clarkb | devstack is going to best effort it but I wouldn't count on it | 23:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Update forge-upload job to use tags https://review.openstack.org/164016 | 23:13 |
| mordred | clarkb: sure- which is why I think the patch we just did was a good idea | 23:13 |
| jeblair | nibalizer: https://review.openstack.org/164016 | 23:13 |
| mordred | clarkb: BUT - we can also only cache what devstack asks for | 23:13 |
| *** nelsnelson has joined #openstack-infra | 23:13 | |
| jeblair | nibalizer: i totally just guessed at what the rakefile should look like :) | 23:13 |
| clarkb | mordred: yup | 23:13 |
| jeblair | ianw: did i ever answer your question? | 23:14 |
| ianw | jeblair: which question? | 23:14 |
| *** patrickeast has joined #openstack-infra | 23:14 | |
| jeblair | ianw: about the logging patch | 23:14 |
| jeblair | i am apparently not able to carry on as many conversations as clarkb | 23:14 |
| ianw | jeblair: not yet, that's fine, i only meant it to go into a queue while i was looking at my reviews | 23:15 |
| openstackgerrit | Merged openstack-infra/project-config: Ignore missing packages https://review.openstack.org/163976 | 23:15 |
| clarkb | jeblair: really my brain is melted. Also, I cleaned the shower earlier today and the bleach may have given me super powers | 23:16 |
| *** ghostpl_ has joined #openstack-infra | 23:16 | |
| jeblair | ianw: i'm not going to -2 it if other people think that's the best approach, but i really would prefer one of the other approaches i listed. | 23:16 |
| * fungi is reminded of a dead milkmen song about drinking bleach | 23:16 | |
| ianw | jeblair: i think my issue is finding anyone else to give an opinion | 23:17 |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Improvements to forge uploading process https://review.openstack.org/164018 | 23:17 |
| *** yamamoto has joined #openstack-infra | 23:18 | |
| fungi | jeblair: ianw: i'm good with just hand-maintaining a logging config for the separate images with the understanding that we're working toward substantially reducing their count in the near future anyway | 23:18 |
| nibalizer | jeblair: crinkle https://review.openstack.org/#/c/164018/ | 23:18 |
| nibalizer | oh you tried too | 23:18 |
| nibalizer | oops | 23:18 |
| clarkb | jeblair: ianw: I agree with jeblair that the logging config shouldn't be autogened by puppet and put in place. However, I don't having a utility like that that we can use to create and propose the logging config in gerrit changes | 23:19 |
| ianw | fungi: i'm not sure if i pasted the output of that command, it's not really maintainable if you split up by provider IMO | 23:19 |
| *** crc32 has quit IRC | 23:20 | |
| jeblair | clarkb: if we want a utility like that, it should probably be in system-config, not in nodepool itself | 23:20 |
| jeblair | since it's _our_ logging config | 23:20 |
| clarkb | jeblair: that would be fine | 23:20 |
| clarkb | *I don't mind having a utility | 23:20 |
| fungi | yeah, that seems reasonable. again it would be hand-maintained configuration with a helper tool we can use to submit the updates | 23:21 |
| ianw | ok that's where i started with https://review.openstack.org/#/c/147368/ | 23:22 |
| ianw | before the config split, that could rather easily create the config file | 23:23 |
| *** asettle has joined #openstack-infra | 23:23 | |
| ianw | i will move it | 23:23 |
| nibalizer | jeblair: i've abandoned 164018 for 164016, and commented | 23:24 |
| *** ghostpl_ has quit IRC | 23:25 | |
| morganfainberg | dhellmann, jogo, sdague, so i have an idea for a hacking check i need to implement - but i don't want to wedge everyone. basically i need a way to say "don't consume the options in the [keystone_authtoken] section of the config if a project is using auth_token middleware. I am thinking we expose this hacking check via an entrypoint in middleware. but i want to do it "right" - however that is. | 23:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Update forge-upload job to use tags https://review.openstack.org/164016 | 23:26 |
| jeblair | nibalizer, crinkle: ^ | 23:26 |
| clarkb | any opposition to me restarting nodepool now to pick up all the code fixes and things? | 23:26 |
| *** otter768 has quit IRC | 23:26 | |
| SpamapS | clarkb: Hey I just noticed that automatic nodepool coverage report link you pasted. I kind of wonder why we don't have that turned on in python-jobs.. ? | 23:26 |
| jeblair | clarkb: i want to restart zuul to get the logging changes | 23:26 |
| clarkb | SpamapS: because its super slow for many projects | 23:26 |
| jeblair | clarkb: let's coordinate! | 23:26 |
| clarkb | jeblair: +1 :) | 23:27 |
| SpamapS | clarkb: ahh | 23:27 |
| SpamapS | too bad | 23:27 |
| morganfainberg | dhellmann, jogo, sdague, the key is projects can't rely on those options for anything besides middleware doing token validates (no you can't use those same options to talk to nova or neutron). any thoughts on the best way to approach this... softly to start | 23:27 |
| *** tnovacik has joined #openstack-infra | 23:27 | |
| clarkb | SpamapS: (it is in python-jobs on the jenkins side but not the zuul side, you can add it individually if you want) | 23:27 |
| clarkb | jeblair: what sort of coordinating should we do? | 23:27 |
| *** wenlock has quit IRC | 23:28 | |
| jeblair | clarkb: okay, i either need to open 8 terminal windows for all the zuul mergers or finally set up ansible locally | 23:28 |
| clarkb | jeblair: you can use ansible on the puppetmaster | 23:28 |
| clarkb | but I don't know the exact incantation | 23:29 |
| jogo | morganfainberg: ping me tomorrow and I may be able to help brainstorm | 23:29 |
| jeblair | clarkb: oh good idea.. how do i do that? | 23:29 |
| clarkb | jeblair: let me look it up | 23:29 |
| morganfainberg | jogo, sounds good1 | 23:29 |
| jogo | morganfainberg: but not sure if hacking is the right thing for that etc. | 23:29 |
| *** yamamoto has quit IRC | 23:29 | |
| morganfainberg | jogo, this is why i'm asking, hacking was my first thought. but of course open to others | 23:29 |
| jeblair | clarkb: oh, bash history! | 23:29 |
| clarkb | jeblair: ansible 'zm0.*' -m shell -a '/etc/init.d/zuul-merger restart' -f8 | 23:30 |
| clarkb | jeblair: something like that | 23:30 |
| jeblair | what's the -f8? | 23:30 |
| clarkb | jeblair: its the parallelism | 23:30 |
| clarkb | it can be omitted | 23:30 |
| *** zz_dimtruck is now known as dimtruck | 23:30 | |
| clarkb | also we still only have 3 leaked ports so that number hasn't changed over the course of the day \o/ | 23:31 |
| jeblair | clarkb: okay. so i'm going to save the queue, stop zuul, stop mergers; then you can stop nodepool, set all nodes to delete, start nodepool; then i'll start zuul, start mergers, reload queue. | 23:31 |
| jeblair | clarkb: sound good? | 23:31 |
| *** achanda has quit IRC | 23:32 | |
| jeblair | clarkb: actually, let's modify that | 23:32 |
| clarkb | jeblair: I think so. I have to figure a good way to set all nodes to delete | 23:32 |
| *** achanda has joined #openstack-infra | 23:32 | |
| jeblair | clarkb: mysql: "update node set state=4; update sub_node set state=4" | 23:32 |
| clarkb | jeblair: thanks | 23:32 |
| jeblair | clarkb: i'll save the queue, stop zuul, stop mergers; then you can stop nodepool, set all nodes to delete; i'll start geard; you start nodepool; wait until things have settled a bit, then i'll stop geard, start zuul, start mergers, reload queue. | 23:33 |
| clarkb | jeblair: sounds good, let me know when it is my turn | 23:33 |
| jeblair | clarkb: zuul is stopped; go for it | 23:34 |
| *** psedlak has quit IRC | 23:34 | |
| jeblair | clarkb: and geard is running so you can start nodepool whenever | 23:35 |
| clarkb | ok starting it now | 23:35 |
| SpamapS | hrm.. | 23:35 |
| SpamapS | trying to track down how to pass --omit's to coverage report generation | 23:35 |
| *** melwitt has quit IRC | 23:35 | |
| jeblair | clarkb: so now let's wait until we have some new nodes so all the jobs get registered | 23:35 |
| clarkb | it is doing the things so we should just need to wait at this point | 23:36 |
| jeblair | it's fun to see geard as the countdown for that | 23:36 |
| jeblair | DEBUG:gear.Server:Polling 199 connections | 23:36 |
| jeblair | DEBUG:gear.Server:Polling 189 connections | 23:36 |
| clarkb | SpamapS: I think the easiest way is via a .coveragerc | 23:36 |
| SpamapS | clarkb: ah yeah thats the ticket | 23:37 |
| clarkb | SpamapS: pbr may also support it | 23:37 |
| SpamapS | clarkb: .coveragerc is perfect actually | 23:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-logstash: Flesh out the readme https://review.openstack.org/153820 | 23:37 |
| jeblair | DEBUG:gear.Server:Polling 28 connections | 23:38 |
| jeblair | i think that was the nadir | 23:38 |
| clarkb | jeblair: http://paste.openstack.org/show/191971/ I think that happens when we switch a node to delete before it has a uuid? | 23:40 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/project-config: Turn on oslo.messaging coverage report https://review.openstack.org/164022 | 23:40 |
| clarkb | jeblair: I am guessing we might want to delete that row specifically then do an alien listing to kill aliens later? | 23:40 |
| jeblair | clarkb: i think that might be a database race, where the row was deleted out from under it | 23:40 |
| clarkb | jeblair: gotcha, I will see if it goes away then | 23:41 |
| *** david-lyle is now known as david-lyle_afk | 23:41 | |
| jeblair | clarkb: as in, we started a thread saying "delete node id foo" and by the time it tried to delete it, the row in the db was already gone | 23:41 |
| jeblair | clarkb: i think we have enough node types online now. | 23:41 |
| jeblair | i'll restart zuul | 23:41 |
| clarkb | ok | 23:41 |
| ianw | jeblair / clarkb / fungi : so do you want https://git.openstack.org/cgit/openstack-infra/puppet-nodepool/tree/manifests/init.pp to a) have the generated log-config checked into it b) not deploy a log-config & leave it up to system-config c) something else? | 23:44 |
| *** Sukhdev has quit IRC | 23:44 | |
| jeblair | sdague: the logging to help figure out what's going on when a change is removed from the check queue because "a parent failed to merge" is in place; let me know if you see it again. | 23:44 |
| jeblair | ianw: puppet-nodepool should also not have our logging configuration in it | 23:45 |
| *** asettle has quit IRC | 23:45 | |
| fungi | ianw: system-config should be supplying it | 23:45 |
| fungi | ianw: system-config is intended to be "systems configuration for the servers run by openstack infra" (ignoring for the moment that lots of people seem to want to also genericize that module to run copies of all our infrastructure without forking it) | 23:47 |
| ianw | fungi: any pointers on how to get puppet to use the template from system-config, rather than the local one? is it like puppet://modules/system-config/path/to/template.erb maybe? | 23:47 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: zuul-swift-upload fix delete-after https://review.openstack.org/164023 | 23:47 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: zuul-swift-upload ensure delete after is an int https://review.openstack.org/164024 | 23:47 |
| jhesketh | clarkb: ^ | 23:47 |
| clarkb | ok I have results for centos7 build. It caches packages ok but the wheel thing explodes on it so fails anyways | 23:47 |
| fungi | ianw: we could make the logging config file path parameterized in the puppet-nodepool module, defaulting to the current one, but then supply an alternative path when we instantiate that module from the openstack_project module | 23:48 |
| clarkb | jhesketh: does requests.post with the data parameter coerce the types to the appropriate thing? | 23:48 |
| clarkb | jhesketh: looking at the other ints we have in there I am guessing so | 23:48 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Turn on post taskflow coverage report https://review.openstack.org/164025 | 23:48 |
| jhesketh | clarkb: yeah, that's what I assumed | 23:49 |
| ianw | fungi: yep, that's sort of my idea, if you can think of anything else that does that i'd just like to see the syntax. i'll dig though | 23:49 |
| *** unicell has joined #openstack-infra | 23:49 | |
| jeblair | fungi: i just traced one of the merge failures down to the merger itself quickly. | 23:50 |
| jeblair | 2015-03-12 23:43:57,325 INFO zuul.MergeClient: Merge <gear.Job 0x7fefe83b9650 handle: H:127.0.0.1:79 name: merger:merge unique: 263d5fb242094ee2b9292b8d6207fbbb> complete, merged: False, updated: False, comm | 23:50 |
| jeblair | it: None | 23:50 |
| jeblair | 2015-03-12 23:43:57,325 DEBUG zuul.Scheduler: Adding merge complete event for build set: <BuildSet item: <QueueItem 0x7fefe83b9610 for <Change 0x7fefea397bd0 154513,26> in gate> #builds: 0 merge state: PENDI | 23:50 |
| jeblair | NG> | 23:50 |
| fungi | jeblair: oh! i was not having a lot of luck tracking down which merger processed which task previously | 23:50 |
| jeblair | fungi: yeah, i just restarted zuul with the new debug lines | 23:50 |
| fungi | got it | 23:50 |
| jeblair | fungi: so check " unique: 263d5fb242094ee2b9292b8d6207fbbb" from there | 23:50 |
| jeblair | ansible zm* -m shell -a 'grep 263d5fb242094ee2b9292b8d6207fbbb /var/log/zuul/merger-debug.log' | 23:50 |
| jeblair | then that ^ | 23:51 |
| clarkb | jeblair: did you also restart the workers with your checkout change? | 23:51 |
| jeblair | zm05.openstack.org | success | rc=0 >> | 23:51 |
| jeblair | 2015-03-12 23:42:59,338 DEBUG zuul.MergeServer: Got merge job: 263d5fb242094ee2b9292b8d6207fbbb | 23:51 |
| jeblair | then that ^ | 23:51 |
| fungi | aha | 23:51 |
| jeblair | and then read down a bit to find: | 23:51 |
| jeblair | fatal: merge program failed' | 23:51 |
| jeblair | stderr: 'error: Merge requires file-level merging | 23:51 |
| jeblair | so, real merge failure that one :) | 23:51 |
| fungi | yeah, distributed grep across the logs of all the workers is an awesome idea. i didn't think of that anyway | 23:51 |
| *** dboik has joined #openstack-infra | 23:52 | |
| *** vladan has left #openstack-infra | 23:52 | |
| jeblair | we could probably report the worker in the server log line too; i didn't think of that. but this is a lot better than before :) | 23:52 |
| fungi | absolutely | 23:52 |
| *** yamamoto has joined #openstack-infra | 23:52 | |
| jeblair | so next time we get a merge failure that doesn't appear to be a merge failure, we should be better equipped | 23:52 |
| *** dboik_ has joined #openstack-infra | 23:52 | |
| jeblair | clarkb: i did not, because apparently that change fails all kinds of tests | 23:52 |
| jeblair | clarkb: so it may require actual thought | 23:53 |
| jeblair | i am not prepared to give it that right at this moment :) | 23:53 |
| fungi | i wonder where the "requires file-level merging" is coming from. if it's the ceilometer icehouse patch apevec is trying to get merged, that was a pure ff | 23:53 |
| clarkb | jeblair: oh fun | 23:53 |
| *** asettle has joined #openstack-infra | 23:53 | |
| jeblair | fungi: oh, that was just the neutron change near the top failing | 23:54 |
| apevec | fungi, it merged | 23:54 |
| fungi | apevec: did it? strange | 23:54 |
| apevec | yeah... | 23:54 |
| jeblair | apevec: good for you, bad for our debugging :) | 23:54 |
| apevec | https://review.openstack.org/141412 | 23:54 |
| fungi | the only thing i hate more than bugs which won't go away is bugs which silently go away | 23:55 |
| apevec | ps2 failed then merged in 2. attempt | 23:55 |
| *** reed has quit IRC | 23:55 | |
| clarkb | I have started a devstack-trusty dib build on nodepool.o.o and will upload it as soon as its done | 23:55 |
| clarkb | dhellmann: ^ this should fix your d-g test fail | 23:55 |
| *** mtanino has joined #openstack-infra | 23:55 | |
| * SpamapS silently goes away | 23:56 | |
| *** gyee has joined #openstack-infra | 23:56 | |
| *** ChuckC has quit IRC | 23:56 | |
| *** dboik has quit IRC | 23:56 | |
| *** sabeen has quit IRC | 23:57 | |
| *** nelsnelson has quit IRC | 23:57 | |
| *** annegentle has joined #openstack-infra | 23:58 | |
| *** dannywil_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!