| *** vipul is now known as vipul-away | 00:00 | |
| jeblair | clarkb, jog0: maybe instead of e-r reading the zmq stream, we should have it be a gearman worker with a function that the logstash client can invoke. | 00:00 |
|---|---|---|
| *** sarob has joined #openstack-infra | 00:01 | |
| jeblair | clarkb, jog0: this occured to me since those two programs are basically implementing very similar logic to decide when to start working. | 00:01 |
| clarkb | yeah I have been considering the various options | 00:01 |
| *** thingee is now known as thingee_zzz | 00:01 | |
| clarkb | gearman seems nice because we can scale it if necessary | 00:02 |
| *** emagana has joined #openstack-infra | 00:03 | |
| jeblair | clarkb: and should be easy to make the logstash client call elastic-recheck when it's done stuffing things into logstash, right? | 00:03 |
| *** emagana has quit IRC | 00:03 | |
| clarkb | sort of. there is no indication of when elasticsearch is done indexing but we can ger it really close | 00:03 |
| *** emagana has joined #openstack-infra | 00:04 | |
| clarkb | which is better than the current poll situation today | 00:04 |
| *** DennyZhang has quit IRC | 00:04 | |
| *** vipul-away is now known as vipul | 00:04 | |
| clarkb | so ++ to gearman | 00:05 |
| *** Cyril2 has joined #openstack-infra | 00:06 | |
| *** sarob has quit IRC | 00:07 | |
| *** jpipes has quit IRC | 00:08 | |
| *** sarob has joined #openstack-infra | 00:08 | |
| *** sarob has quit IRC | 00:10 | |
| *** sarob has joined #openstack-infra | 00:10 | |
| *** AlexF has quit IRC | 00:15 | |
| jog0 | ++ from me too | 00:18 |
| jeblair | jog0: is there a script to run the queries and output counts? | 00:20 |
| jog0 | jeblair: there will be in 10 minutes | 00:21 |
| jog0 | jeblair: thats on the TODO list | 00:21 |
| jog0 | will do it now | 00:21 |
| *** UtahDave has joined #openstack-infra | 00:21 | |
| *** svarnau has quit IRC | 00:25 | |
| *** matsuhashi has joined #openstack-infra | 00:25 | |
| jog0 | clarkb: ElasticSearch is still slow | 00:34 |
| jog0 | and elastic-recheck is hung again | 00:34 |
| *** emagana_ has joined #openstack-infra | 00:39 | |
| clarkb | jog0 ya until my longer timeout patch goes in and your change to timeout goes in this will be a problem | 00:41 |
| jog0 | clarkb: ahh | 00:41 |
| *** julim has quit IRC | 00:42 | |
| jog0 | and your patch hasn't landed yet? | 00:42 |
| jog0 | and the load isn't down? | 00:42 |
| *** emagana has quit IRC | 00:43 | |
| clarkb | my patch hasn't even been reviewed I don't think | 00:43 |
| clarkb | I just pushed it :) | 00:43 |
| clarkb | and load may be related to the crazy find we run | 00:44 |
| jeblair | clarkb: i reviewed it! | 00:44 |
| jeblair | clarkb: what load? | 00:44 |
| jeblair | clarkb: and we don't run it during the week | 00:45 |
| clarkb | jeblair hypothesis is that load domewhere between slave and static makes file copy slow | 00:45 |
| jeblair | clarkb: did i miss a conversation? | 00:45 |
| clarkb | what is happening is after 60 seconds console.html is not copied to static. sort of I mention it briefly in my commit message I think | 00:46 |
| jeblair | clarkb: possibly the jenkins master, since all file copies are funneled through it. and we do hundreds of them at once. | 00:46 |
| clarkb | but I wanted more time to investigate before digging deeply | 00:46 |
| clarkb | jeblair oh good point | 00:46 |
| *** dcramer_ has joined #openstack-infra | 00:48 | |
| *** nosnos has joined #openstack-infra | 00:49 | |
| jog0 | jeblair: turns out https://review.openstack.org/#/c/49246/1 | 00:51 |
| jog0 | generates what you want, running it now | 00:51 |
| jeblair | jog0: cool | 00:52 |
| jog0 | these queries are a little slow when open ended | 00:54 |
| *** ryanpetrello has joined #openstack-infra | 00:54 | |
| jgriffith | clarkb: jeblair quick question... was going to propose on ML but thought I'd ask in here first :) | 00:56 |
| jgriffith | seems we've identified a number of bug fixes that are actuallyin progress to fix some of the failures that are choking up gates | 00:56 |
| jgriffith | any chance we could/should start systematically ONLY allowing those changes in for the time being? | 00:57 |
| *** adalbas has quit IRC | 00:58 | |
| *** darraghb has joined #openstack-infra | 00:58 | |
| *** weshay has quit IRC | 00:59 | |
| jog0 | jgriffith: ++ | 01:00 |
| jog0 | ttx: kicked that idea around before too | 01:00 |
| jog0 | ttx kicked * | 01:01 |
| jgriffith | ahh...ok, so not totally off the wall then | 01:01 |
| jog0 | jeblair jgriffith: here are the top offending bugs http://paste.openstack.org/show/47807/ | 01:01 |
| jog0 | unsorted | 01:01 |
| jgriffith | jog0: here's my homework assignment http://paste.openstack.org/show/47798/ | 01:02 |
| jgriffith | jog0: trying to get the last cinder one in since 9 this morning :( | 01:02 |
| jgriffith | but not sure how frequent it is of late | 01:02 |
| jog0 | nice | 01:02 |
| *** schwicht has quit IRC | 01:03 | |
| jgriffith | jog0: BTW I think those top list items have changed through the day | 01:04 |
| jgriffith | anyway... | 01:04 |
| jog0 | jgriffith: I just ran the check momemnts ago | 01:05 |
| jog0 | so the paste I gave is fresh | 01:05 |
| jgriffith | jog0: oh... then never mind :) | 01:05 |
| clarkb | jog0 jgriffith I think you fix that socially... we could extend zuul to have change based acls but asking folks to play nice seems simple | 01:05 |
| *** melwitt has quit IRC | 01:05 | |
| jgriffith | clarkb: seems like it, however i"m finding otherwise | 01:05 |
| jgriffith | ie "My driver has a misspelled commet, here's my change" | 01:06 |
| jog0 | jgriffith: bug 1226337 at 389 | 01:06 |
| uvirtbot | Launchpad bug 1226337 in cinder "tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern flake failure" [Undecided,Fix committed] https://launchpad.net/bugs/1226337 | 01:06 |
| jog0 | bug 1229475 at 500! | 01:06 |
| uvirtbot | Launchpad bug 1229475 in python-neutronclient "terminate_instance(): RuntimeError: Second simultaneous read on fileno 16 detected" [Undecided,In progress] https://launchpad.net/bugs/1229475 | 01:06 |
| jog0 | bug 1223469 at 109 | 01:07 |
| uvirtbot | Launchpad bug 1223469 in cinder "Volume create tgtadmin fails with error code 22" [High,Fix committed] https://launchpad.net/bugs/1223469 | 01:07 |
| jgriffith | jog0: got ya | 01:08 |
| jgriffith | jog0: so the good thing is 1223469 shouldn't be causing failures anymore | 01:08 |
| jgriffith | jog0: but at this point it's small taters | 01:08 |
| *** ryanpetrello has quit IRC | 01:08 | |
| jog0 | impressively 2 should be closed and one is on its way | 01:08 |
| jgriffith | cool... I just be patient :) | 01:09 |
| jgriffith | I really want to get Cinder off of your list ;) | 01:09 |
| jog0 | so after 2 weeks when we remove the logs from logstash we will see the bug hasn't happend and remove it from elastic-recheck | 01:09 |
| jog0 | jgriffith: ^_^ | 01:09 |
| darraghb | has anyone had a chance to see if the single entry point assumption in the following change for JJB https://review.openstack.org/#/c/49102/ is valid? | 01:09 |
| jog0 | I know there are some nova issues not on that list | 01:10 |
| jog0 | and maybe neutron | 01:10 |
| jgriffith | jog0: so maybe I need to change the code to not print an error message if the recovery works | 01:10 |
| jog0 | that makes sense | 01:11 |
| *** DennyZhang has joined #openstack-infra | 01:12 | |
| clarkb | darraghb: I haven't had a chance to look | 01:12 |
| jog0 | jgriffith: ther is also this one bug 1205344 | 01:13 |
| uvirtbot | Launchpad bug 1205344 in nova "mkfs error in test_stamp_pattern" [Medium,Confirmed] https://launchpad.net/bugs/1205344 | 01:13 |
| jog0 | which we are disabling the test for | 01:13 |
| jgriffith | jog0: got ya | 01:13 |
| jog0 | and will re-enable early in Icehouse | 01:13 |
| jog0 | that maybe a nova issue too? | 01:14 |
| jgriffith | Yeah, I was looking at that one, it's a big funky | 01:14 |
| *** changbl has joined #openstack-infra | 01:14 | |
| jgriffith | jog0: not a ton to go on | 01:15 |
| jog0 | jgriffith: :( it keeps happening though | 01:15 |
| jog0 | but this is post havana | 01:15 |
| jgriffith | sighh | 01:16 |
| *** ryanpetrello has joined #openstack-infra | 01:17 | |
| jog0 | jgriffith: anyway I have a plane to catch | 01:18 |
| jog0 | just think when the gate is stable and we do a better job of catching these issues you won't have this pain again | 01:19 |
| jgriffith | jog0: safe travels | 01:19 |
| *** ryanpetrello has quit IRC | 01:20 | |
| *** ryanpetrello has joined #openstack-infra | 01:22 | |
| mriedem | does anyone know why this doesn't fall back to get thrift from pypi.openstack.org? http://logs.openstack.org/39/48739/4/check/gate-requirements-install/6a6ce59/console.html | 01:28 |
| *** sarob has quit IRC | 01:28 | |
| mriedem | pypi.python.org is not responding, but the thrift package is in the openstack mirror | 01:28 |
| *** sarob has joined #openstack-infra | 01:29 | |
| clarkb | mriedm: requirements doesn't use our mirror | 01:32 |
| clarkb | its a chicken and egg problem | 01:32 |
| *** sarob has quit IRC | 01:32 | |
| clarkb | requirements builds the mirror so needs to be tested against pypi | 01:32 |
| dstufft | I see bumping the timeout just made it 503 | 01:33 |
| dstufft | I was worried it might | 01:33 |
| dstufft | I'm sorry for that ^, upstream Fastly is still investigating | 01:33 |
| *** dims has quit IRC | 01:40 | |
| *** emagana_ has quit IRC | 01:52 | |
| *** ArxCruz has quit IRC | 01:52 | |
| *** sandywalsh has joined #openstack-infra | 01:53 | |
| hub_cap | mordred: its coming, albeit rather slowly https://gist.github.com/hub-cap/6787885 | 01:54 |
| *** sarob has joined #openstack-infra | 01:55 | |
| *** ArxCruz has joined #openstack-infra | 01:56 | |
| *** ryanpetrello has quit IRC | 01:58 | |
| *** DennyZhang has quit IRC | 02:10 | |
| *** sarob_ has joined #openstack-infra | 02:15 | |
| *** jerryz has quit IRC | 02:17 | |
| *** sarob has quit IRC | 02:18 | |
| *** sarob_ has quit IRC | 02:20 | |
| *** ryanpetrello has joined #openstack-infra | 02:25 | |
| mriedem | clarkb: ok, thanks, will recheck when i can get the tar myself | 02:26 |
| darraghb | Hoping to resurrect the thread patch I uploaded previously that was abandoned https://review.openstack.org/#/c/45371/1 | 02:33 |
| darraghb | Could do with some suggestions on how to ensure the CI tests will use multiple threads to catch issues while the code defaults to single threaded behaviour to ensure end use of the change requires explicit enabling? | 02:34 |
| *** AlexF has joined #openstack-infra | 02:36 | |
| *** dkehn_ has joined #openstack-infra | 02:37 | |
| *** dkehn has quit IRC | 02:41 | |
| *** markmcclain has joined #openstack-infra | 02:42 | |
| *** gyee has quit IRC | 02:52 | |
| anteaya | I'm looking for the reason for this grenade-devstack-vm failure and so far I can't find it: https://review.openstack.org/#/c/48971/ | 02:54 |
| anteaya | anyone willing to take a peek and offer an opinion? | 02:54 |
| *** AlexF has quit IRC | 02:59 | |
| mriedem | anteaya: maybe because it can't find the test results log files it's looking for? | 03:03 |
| *** ryanpetrello has quit IRC | 03:03 | |
| mriedem | [SCP] '**/subunit_log.txt.gz' doesn't match anything: '**' exists but not '**/subunit_log.txt.gz' | 03:03 |
| anteaya | really? | 03:03 |
| anteaya | that would create a failure? | 03:04 |
| clarkb | no that won't cause a failure | 03:04 |
| anteaya | I need to get it back in the queue, but somehow reverify no bug would feel like I was missing something | 03:06 |
| anteaya | but for the life of me, I can't find what caused that test to fail | 03:06 |
| mriedem | anteaya: clarkb: i'm just comparing to this which didn't fail: http://logs.openstack.org/71/48971/2/gate/gate-grenade-devstack-vm/632df89/ | 03:07 |
| mriedem | that was from a previous run, it doesn't fail and the log gz files are there | 03:07 |
| mriedem | although it has a ton more logs here too: http://logs.openstack.org/71/48971/2/gate/gate-grenade-devstack-vm/632df89/logs/ | 03:07 |
| anteaya | yes | 03:08 |
| clarkb | grenade is returning 1 for some reason | 03:08 |
| anteaya | a large description in the amount and format of the logs | 03:08 |
| anteaya | 2013-10-02 02:43:13.040 | + return 1 in console.html | 03:09 |
| anteaya | a bit slim to go on for debugging | 03:09 |
| anteaya | at this timestamp as well: 2013-10-02 02:43:13.027 | + return 1 | 03:10 |
| mriedem | anteaya: i see 138 hits for "return 1" :) | 03:10 |
| anteaya | 2013-10-02 02:43:13.015 | + return 0 above that | 03:10 |
| anteaya | that is quite a few | 03:11 |
| anteaya | mriedem: how many do you get in this file: http://logs.openstack.org/71/48971/2/gate/gate-grenade-devstack-vm/632df89/console.html | 03:12 |
| anteaya | I get about 190 | 03:13 |
| anteaya | so return 1 might not be a good indicator of what is unusual in the console.html for the failure | 03:13 |
| mriedem | 191 for me | 03:14 |
| mriedem | ~same | 03:14 |
| mriedem | off topic, but this could use another infra +2, i've been rechecking the hell out of it today: https://review.openstack.org/#/c/48739/ | 03:14 |
| mriedem | clarkb: ^? | 03:14 |
| anteaya | I'm going to swing 48971 back onto the pile with no bug | 03:15 |
| anteaya | I'll hang my head in shame as I do it | 03:15 |
| mriedem | i think it was already rechecked | 03:16 |
| mriedem | reverified i mean | 03:16 |
| anteaya | oh morganfainberg already did it | 03:16 |
| anteaya | yup | 03:16 |
| anteaya | I'll hang my head in shame another time | 03:16 |
| mriedem | anteaya: yeah, about 20 min ago | 03:16 |
| anteaya | fine | 03:16 |
| anteaya | I couldn't find the failure anyhow | 03:16 |
| mriedem | anteaya: let that be your shame | 03:17 |
| anteaya | I'm not very experienced at reading the grenade failure files | 03:17 |
| anteaya | yay, I have a target for the shame | 03:17 |
| anteaya | so someday I will learn how to read the grenade failure files | 03:17 |
| clarkb | mriedem: I would look but am on a phone. about to afk | 03:18 |
| mriedem | clarkb: ok, thanks anyway | 03:19 |
| *** hogepodge has quit IRC | 03:26 | |
| lifeless | clarkb: how do I go about getting a service account ? | 03:26 |
| *** mriedem has quit IRC | 03:28 | |
| clarkb | lifeless: file a bug/request one via email/irc and provide pubkey | 03:30 |
| jog0 | jgriffith: are you still thinking of asking the ML to be careful about approving patches? | 03:30 |
| jog0 | If so I am all for it | 03:31 |
| lifeless | clarkb: so I'd like one | 03:31 |
| *** UtahDave has quit IRC | 03:32 | |
| lifeless | clarkb: 'tripleo-cd-bot', pubkey is linked from https://etherpad.openstack.org/tripleo-mvp1 | 03:32 |
| clarkb | lifeless: ok. I would help now but am trying to get off this phone :) probably easiest since most of us are afk to submit a bug or send email to infra list | 03:32 |
| *** darraghb has quit IRC | 03:33 | |
| lifeless | maybe mordred will do it when he gets back :) | 03:33 |
| lifeless | I will file bug now | 03:33 |
| jgriffith | jog0: well... not sure but it looks like we may be rounding a corner | 03:34 |
| jgriffith | unless I'm just being overly optimistic | 03:34 |
| lifeless | clarkb: https://bugs.launchpad.net/openstack-ci/+bug/1233961 | 03:35 |
| uvirtbot | Launchpad bug 1233961 in openstack-ci "Please add a service account" [Undecided,New] | 03:35 |
| clarkb | lifeless: perfect thanks | 03:36 |
| mordred | what did I miss? | 03:42 |
| *** Ryan_Lane has joined #openstack-infra | 03:44 | |
| jgriffith | jog0: last four Cinder patches have passed, haven't seen that in a while | 03:49 |
| jgriffith | jog0: like I said, don't want to jinx it, but maybe we're rounding the corner | 03:49 |
| *** darraghb has joined #openstack-infra | 03:49 | |
| *** dkliban has joined #openstack-infra | 03:50 | |
| lifeless | mordred: https://bugs.launchpad.net/openstack-ci/+bug/1233961 | 03:50 |
| uvirtbot | Launchpad bug 1233961 in openstack-ci "Please add a service account" [Undecided,New] | 03:50 |
| *** Ryan_Lane has quit IRC | 03:50 | |
| mordred | jgriffith: woot | 03:54 |
| mordred | lifeless: nod | 03:54 |
| anteaya | jgriffith: oh I hope so | 03:54 |
| anteaya | I need to hear that somebody is happy today | 03:54 |
| * jgriffith is always happy | 03:56 | |
| jgriffith | mordred: hey.. btw, how do I recheck smokestack (only had to ever do it once and that was like a year ago) | 03:56 |
| anteaya | yay | 03:57 |
| mordred | jgriffith: there's no way to without poking dan | 03:58 |
| mordred | jgriffith: we don't run smokestack | 03:58 |
| jgriffith | mordred: that's whatI thought.. is it non-voting? | 03:58 |
| mordred | yes | 03:58 |
| jgriffith | mordred: excellent.. thanks | 03:59 |
| morganfainberg | anteaya, the granade error was actually in the build, wasn't anything really wrong with the code (mysql timed out it looked like) | 03:59 |
| anteaya | morganfainberg: can you show me the line/timestamp where you saw that? | 03:59 |
| morganfainberg | sure, sec. | 03:59 |
| morganfainberg | anteaya, 2013-10-02 02:22:59.965 in console.html also notice how the other logs were really really empty compared to others. | 04:00 |
| morganfainberg | other grenade runs that is | 04:00 |
| morganfainberg | oh hah, wasn't even mysql time out was a total 60 minute timeout | 04:01 |
| anteaya | yes I did notice the absence of logs and content | 04:01 |
| morganfainberg | haven't seen that much in granade tbh | 04:01 |
| anteaya | me either | 04:01 |
| anteaya | a 60 minute timeout? | 04:01 |
| morganfainberg | yeah, thats a long time to be spinning on the build. | 04:02 |
| anteaya | I am about to stick my foot in my mouth, the build of what | 04:02 |
| *** UtahDave has joined #openstack-infra | 04:02 | |
| anteaya | what was building | 04:02 |
| anteaya | that was given 60 minutes to do so and couldn't manage it | 04:02 |
| anteaya | devstack? | 04:03 |
| morganfainberg | anteaya, setup maybe? | 04:03 |
| morganfainberg | anteaya, thats what it looked like | 04:03 |
| morganfainberg | anteaya, it looked transient at worst, just a bad setup. | 04:03 |
| morganfainberg | anteaya, hence why i went with no bug. (I haven't seen that happen really ever) | 04:03 |
| anteaya | well it got it back on the pile | 04:03 |
| anteaya | so that is good | 04:03 |
| anteaya | but I find this ... odd | 04:04 |
| *** tkammer has joined #openstack-infra | 04:04 | |
| morganfainberg | anteaya, yep. i'll babysit it some more tonight. I've been babysitting RC bugs through gate for keystone for the last couple weeks at night | 04:04 |
| anteaya | now it might be odd because I don't know how to read grenade logs | 04:04 |
| anteaya | cool | 04:04 |
| morganfainberg | anteaya, in this case, the lack of logs make it a particularly bad example :P | 04:04 |
| anteaya | I'm also babysitting the glance one | 04:04 |
| anteaya | that is true | 04:04 |
| anteaya | so we are on the night watch | 04:05 |
| morganfainberg | anteaya, otherwise granade isn't a whole lot different than most tempest like logs (devstack) | 04:05 |
| morganfainberg | anteaya, Winter is coming? | 04:05 |
| morganfainberg | >.> | 04:05 |
| anteaya | apparently | 04:05 |
| anteaya | my nephew has a shirt that says so | 04:05 |
| morganfainberg | anteaya, but i didn't want to take the black :( i don't want to go to the wall :P | 04:05 |
| * morganfainberg stops w/ the ASoIF (or GoT) references... | 04:05 | |
| anteaya | I feel I am missing something | 04:06 |
| anteaya | I feel my nephew would get it though | 04:06 |
| morganfainberg | anteaya, Game of Thrones | 04:06 |
| anteaya | ah | 04:06 |
| anteaya | no idea | 04:06 |
| anteaya | I can tell you about guitarzan though | 04:06 |
| morganfainberg | anteaya, if you like low-fantasy and very political (kings, queens, subterfuge, royalty, etc) its a decent series to read. And if you like HBO and the "HBO-Style" shows, it's and entertaining watch. | 04:07 |
| morganfainberg | but the show/books aren't for everyone | 04:08 |
| anteaya | yay 49029 succeeded | 04:08 |
| * morganfainberg cheers for 49029 | 04:08 | |
| anteaya | go swift and horizon patches | 04:08 |
| *** harlowja has quit IRC | 04:08 | |
| anteaya | merge, merge, merge | 04:08 |
| morganfainberg | hehe | 04:08 |
| anteaya | it has been so long since I watched a TV series | 04:08 |
| anteaya | I get disappointed in the writing | 04:09 |
| morganfainberg | at least we aren't up to the oslo-policy sync reverify's for the keystone merges yet. https://review.openstack.org/#/c/48111/ (it got silly) | 04:09 |
| morganfainberg | anteaya, ah yeah. i don't have TV service anymore. I only watch the shows I want (~2) via like itunes. | 04:09 |
| anteaya | woo Sept 28 was a memorable time for you watching ZuulTV | 04:10 |
| anteaya | I don't blame you | 04:11 |
| morganfainberg | hehe | 04:11 |
| anteaya | actually the writing for ZuulTV is quite good | 04:11 |
| anteaya | production quality is impressive | 04:11 |
| morganfainberg | anteaya, hehehehe ++ | 04:11 |
| anteaya | :D | 04:11 |
| *** markmcclain1 has joined #openstack-infra | 04:15 | |
| *** markmcclain has quit IRC | 04:15 | |
| *** markmcclain1 has quit IRC | 04:19 | |
| *** Ryan_Lane has joined #openstack-infra | 04:19 | |
| *** markmcclain has joined #openstack-infra | 04:19 | |
| *** emagana has joined #openstack-infra | 04:20 | |
| *** Ryan_Lane has quit IRC | 04:23 | |
| *** dkliban has quit IRC | 04:23 | |
| *** tkammer has quit IRC | 04:28 | |
| *** tkammer has joined #openstack-infra | 04:29 | |
| *** esker has joined #openstack-infra | 04:41 | |
| lifeless | mordred: I am really wondering how hard it would be to tell jjb to make three jobs for each gate job definition | 04:47 |
| lifeless | mordred: and zuul to schedule all three | 04:47 |
| lifeless | as a low-overhead variant of 'run N', just to see what it scares up | 04:48 |
| mordred | lifeless: so, I believe we've landed the patch now to allow us to tell zuul to schedule multiple runs of a given job for a patch | 04:49 |
| mordred | lifeless: before, we were explicitly collapsing duplicates in the job list | 04:49 |
| *** Ryan_Lane has joined #openstack-infra | 04:49 | |
| morganfainberg | mordred, running multiples could be beneficial | 04:49 |
| mordred | morganfainberg: yes. it's definitely on the list | 04:49 |
| mordred | once we get the current thing under control, we'd liek to put multiple runs in to try to prevent new things from coming in | 04:50 |
| morganfainberg | mordred, i'm going to start taking a hard look at some of the infra stuff once i have a few cycles, see if i can contribute some help along those lines. | 04:50 |
| morganfainberg | or something in here :) | 04:50 |
| mordred | morganfainberg: yay! | 04:50 |
| mordred | morganfainberg: I like it when people contribute! | 04:51 |
| morganfainberg | mordred, yah! and i like contributing! | 04:51 |
| *** Ryan_Lane has quit IRC | 04:54 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:58 | |
| *** boris-42 has joined #openstack-infra | 04:59 | |
| markmcclain | is there something wrong with jenkins01? it has been SCP'ing horizon doc files for nearly 90mins | 05:01 |
| morganfainberg | markmcclain, it's also been slow loading the consoles/info pages on the jobs | 05:02 |
| *** UtahDave has quit IRC | 05:03 | |
| anteaya | it seems to be processing the files | 05:09 |
| anteaya | perhaps it just has a lot of files to process | 05:09 |
| markmcclain | no more than files than the other horizon doc jobs that have already completed | 05:13 |
| morganfainberg | markmcclain, maybe just a bad vm/slave that is misbehaving from an I/O standpoint? | 05:14 |
| mordred | markmcclain: quantumclient released | 05:14 |
| markmcclain | mordred: thanks | 05:14 |
| anteaya | markmcclain: true it may be slow compared to other vms, but it is still processing | 05:15 |
| anteaya | I'm not a fan of waiting either, but it isn't stuck | 05:16 |
| morganfainberg | anteaya, but uploading 3-4 files per minute is … very slow. | 05:16 |
| anteaya | yes | 05:16 |
| morganfainberg | just as long as it completes and doesn't fail. | 05:16 |
| anteaya | right | 05:16 |
| morganfainberg | i'll be ok with it | 05:16 |
| anteaya | since there is no intervention that will make it go faster | 05:16 |
| anteaya | of which I am aware | 05:16 |
| morganfainberg | I shall glare menacingly | 05:17 |
| morganfainberg | maybe it'll go faster | 05:17 |
| anteaya | glare away | 05:18 |
| anteaya | I hope it works | 05:18 |
| anteaya | I'm browsing reddit | 05:18 |
| anteaya | let's see if that helps | 05:18 |
| anteaya | my ability to think passed long ago | 05:18 |
| morganfainberg | anteaya, i'm getting my grumpycat fix in :P | 05:19 |
| anteaya | not a meme I ever felt akin too | 05:19 |
| anteaya | but if grumpycat does it for you, knock yourself out | 05:20 |
| *** Ryan_Lane has joined #openstack-infra | 05:20 | |
| morganfainberg | anteaya, not the meme , but the actual cat videos pics w/o the text :) | 05:20 |
| anteaya | this one I don't know | 05:20 |
| *** Ryan_Lane has quit IRC | 05:24 | |
| *** tvb|afk has joined #openstack-infra | 05:25 | |
| *** tvb|afk has quit IRC | 05:30 | |
| anteaya | mordred, it is like you just tailed off and hit send on that one | 05:42 |
| *** jaypipes has joined #openstack-infra | 05:46 | |
| anteaya | how can this be merged: https://review.openstack.org/#/c/49006/ | 05:46 |
| anteaya | when it is at the end of the gate queue and failing spectacularly? | 05:46 |
| anteaya | mordred ^ | 05:46 |
| *** pblaho has joined #openstack-infra | 05:47 | |
| SergeyLukjanov | anteaya, Patch Set 1: Verified; Looks good to me (core reviewer); Approved | 05:47 |
| anteaya | status is merged | 05:48 |
| anteaya | UploadedSep 30, 2013 12:42 PM | 05:48 |
| anteaya | UpdatedOct 2, 2013 1:13 AM | 05:48 |
| anteaya | StatusMerged | 05:48 |
| anteaya | but 1:13 was when it was approved | 05:48 |
| anteaya | and it is in the bottom of the gate queue | 05:49 |
| anteaya | failing on every devstack test so far | 05:49 |
| mordred | anteaya: I force merged it | 05:49 |
| mordred | anteaya: it will never pass tests | 05:49 |
| anteaya | oh okay | 05:49 |
| anteaya | I see | 05:50 |
| anteaya | like it isn't passing tests now | 05:50 |
| mordred | it's the quantumclient branch, which will be spectacularly broken in our rig - but we don't care :) | 05:50 |
| mordred | yah | 05:50 |
| anteaya | oh we don't care | 05:50 |
| anteaya | tomorrow I will ask why we don't care | 05:50 |
| anteaya | but not tonight | 05:50 |
| *** Ryan_Lane has joined #openstack-infra | 05:50 | |
| *** odyssey4me has joined #openstack-infra | 05:50 | |
| anteaya | tonight I want the squirrels on the Jenkins01 doc scp job to keep rolling | 05:51 |
| mordred | mmm. squirrels | 05:52 |
| *** tvb|afk has joined #openstack-infra | 05:54 | |
| *** Ryan_Lane has quit IRC | 05:54 | |
| *** SergeyLukjanov has quit IRC | 05:55 | |
| anteaya | oh this glacial doc job had better pass | 05:57 |
| jog0 | anteaya: I am watching it too | 05:57 |
| anteaya | we will hopefully clear about about 19 patches | 05:57 |
| anteaya | jog0: any thoughts on why that doc job is taking so long? | 05:57 |
| jog0 | last I saw it was on scp for the past hour or so but not sure | 05:58 |
| anteaya | yup | 05:58 |
| anteaya | still scp'ing | 05:58 |
| jog0 | lol | 05:58 |
| jog0 | wonder if its rax issue? | 05:58 |
| *** esker has quit IRC | 05:58 | |
| jog0 | other pieces are slow today too | 05:58 |
| anteaya | it is a rax vm? | 05:58 |
| jog0 | anteaya: not sure | 05:58 |
| anteaya | I havent' been paying attention to speed today | 05:59 |
| *** odyssey4me has quit IRC | 05:59 | |
| *** esker has joined #openstack-infra | 05:59 | |
| anteaya | for the logs we are discussing this patch: https://review.openstack.org/#/c/44724/ | 05:59 |
| anteaya | all done 'cept gate-horizon-docs | 06:00 |
| anteaya | which is holding up everything else | 06:00 |
| anteaya | though still running | 06:00 |
| jog0 | anteaya: pretty amazing its still running | 06:00 |
| anteaya | it is | 06:00 |
| anteaya | the 20 patches behind it are all finished (passed or failed) | 06:01 |
| Alex_Gaynor | it's got like 20 minutes until it times out I believe | 06:04 |
| morganfainberg | Alex_Gaynor, i hope it doesn't time out... | 06:06 |
| *** odyssey4me has joined #openstack-infra | 06:08 | |
| anteaya | Alex_Gaynor: how long does it have total? | 06:09 |
| Alex_Gaynor | anteaya: I think the timeout is 3hours? | 06:09 |
| anteaya | wow | 06:09 |
| anteaya | yea, we are close to that | 06:10 |
| morganfainberg | is there anyway we can prevent the timeout from happening? | 06:10 |
| anteaya | since that is a non failing item I'm betting that it will be a gate reset | 06:10 |
| anteaya | I have no power | 06:10 |
| anteaya | it is in the code I do believe | 06:11 |
| anteaya | not a trigger on the vm | 06:11 |
| morganfainberg | nod | 06:11 |
| anteaya | I still feel awful sitting here, watching the timeout about to happen | 06:11 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Handle null in this function https://review.openstack.org/49312 | 06:12 |
| anteaya | I'm curious as to the reason why for 49312 | 06:14 |
| anteaya | since we are here | 06:14 |
| Alex_Gaynor | anteaya: ugh, I should improve that commit message | 06:15 |
| anteaya | k | 06:15 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Handle null in this function https://review.openstack.org/49312 | 06:16 |
| *** tvb|afk_ has joined #openstack-infra | 06:16 | |
| anteaya | mordred you still up? | 06:19 |
| *** tvb|afk has quit IRC | 06:20 | |
| mordred | anteaya: yup | 06:20 |
| anteaya | grand | 06:20 |
| *** Ryan_Lane has joined #openstack-infra | 06:20 | |
| *** shardy_afk is now known as shardy | 06:23 | |
| *** odyssey4me2 has joined #openstack-infra | 06:25 | |
| *** odyssey4me has quit IRC | 06:28 | |
| anteaya | Build has been executing for 3 hr 4 min on precise7 | 06:28 |
| anteaya | and still running | 06:28 |
| anteaya | whew | 06:28 |
| *** Ryan_Lane has quit IRC | 06:29 | |
| morganfainberg | very close to being done | 06:29 |
| anteaya | oh i hope we make it | 06:29 |
| *** odyssey4me2 has quit IRC | 06:29 | |
| Alex_Gaynor | Apparently it's not a 3 hour cutoff? | 06:30 |
| morganfainberg | ~60 more files to upload | 06:30 |
| *** tvb|afk_ is now known as tvb | 06:31 | |
| morganfainberg | Alex_Gaynor i think that is because it's in upload mode | 06:31 |
| *** tvb has quit IRC | 06:31 | |
| *** tvb has joined #openstack-infra | 06:31 | |
| morganfainberg | different phase of the build | 06:31 |
| Alex_Gaynor | morganfainberg: oh, can it not time out at this stage? | 06:31 |
| morganfainberg | not "execution" of the build | 06:31 |
| Alex_Gaynor | I guess that makes sense, it's outside the main script | 06:31 |
| morganfainberg | Alex_Gaynor, no idea, but jenkins handles that separately | 06:31 |
| morganfainberg | so that is my guess | 06:31 |
| morganfainberg | Alex_Gaynor, in either case this is a bit nerve wracking because of how many successful gate jobs are behind it | 06:33 |
| Alex_Gaynor | indeed | 06:33 |
| anteaya | 20 completed patches behind it | 06:35 |
| anteaya | 16 successful | 06:35 |
| morganfainberg | woo, last 8 files or so | 06:38 |
| anteaya | go go go | 06:39 |
| anteaya | yayaya | 06:39 |
| morganfainberg | and success | 06:39 |
| anteaya | finished success | 06:39 |
| anteaya | yay | 06:39 |
| anteaya | there goes horizon | 06:40 |
| morganfainberg | and phew, that means the RC1 patch made it in for keystone :) | 06:40 |
| *** emagana has quit IRC | 06:40 | |
| anteaya | yay | 06:41 |
| sld | what patch? | 06:41 |
| morganfainberg | uhoh | 06:41 |
| anteaya | 48971 | 06:41 |
| morganfainberg | is zuul … unhappy? | 06:41 |
| anteaya | zuul just needs a moment | 06:41 |
| anteaya | recalculating | 06:42 |
| morganfainberg | ok | 06:42 |
| anteaya | and uploading logs to comments | 06:42 |
| anteaya | give him a minute | 06:42 |
| anteaya | he has had to entertain us for over 3 hours | 06:42 |
| anteaya | the god need s a break | 06:42 |
| sld | this? https://review.openstack.org/#/c/48971/ ? | 06:43 |
| morganfainberg | sld yeah | 06:43 |
| anteaya | gate-horizon-docs SUCCESS in 3h 16m 20s | 06:46 |
| morganfainberg | woot | 06:47 |
| anteaya | I'm so glad we made it | 06:47 |
| AJaeger | congrats! | 06:47 |
| anteaya | AJaeger: thanks | 06:47 |
| anteaya | feels like a victory | 06:47 |
| anteaya | though I did nothing but watch zuulTV | 06:48 |
| anteaya | and click | 06:48 |
| sld | 3 hours? geez. | 06:48 |
| anteaya | yeah, no kidding | 06:48 |
| anteaya | compared with this horizon patch: https://review.openstack.org/#/c/49249/ | 06:48 |
| anteaya | gate-horizon-docs SUCCESS in 35m 32s | 06:49 |
| AJaeger | anteaya, it's not only the zuulTV but all the buildup over the last 6 months. But without zuulTV you would had less fun ;) | 06:50 |
| anteaya | much less fun indeed | 06:50 |
| anteaya | yes, the culmination of a grand effort | 06:50 |
| *** darraghb has left #openstack-infra | 06:51 | |
| *** afazekas_no_irq has joined #openstack-infra | 06:55 | |
| *** Ryan_Lane has joined #openstack-infra | 06:55 | |
| *** uvirtbot has quit IRC | 06:56 | |
| *** odyssey4me has joined #openstack-infra | 06:56 | |
| *** uvirtbot has joined #openstack-infra | 06:56 | |
| *** Ryan_Lane has quit IRC | 06:59 | |
| *** odyssey4me2 has joined #openstack-infra | 07:00 | |
| *** vipul is now known as vipul-away | 07:02 | |
| *** odyssey4me has quit IRC | 07:02 | |
| *** boris-42 has quit IRC | 07:06 | |
| jd__ | zuul doesn't seem very active according to http://status.openstack.org/zuul/ | 07:11 |
| jd__ | I hope it's just a glitch on the web interface :) | 07:11 |
| anteaya | we just waited a long time for a job at the head of the queue to finish | 07:12 |
| morganfainberg | jd__, i am hoping it's just catching up from the ~16 patchest that completed gate all at once. | 07:12 |
| anteaya | I am hoping that the pause is due to writing logs and recalculating the pipeline | 07:13 |
| jd__ | anteaya: yeah but now… I've approved a patch and asked for a recheck 5 minutes ago already | 07:13 |
| anteaya | yeah what morganfainberg said | 07:13 |
| jd__ | i'm just scared nothing shows up, not even the event queue length | 07:13 |
| anteaya | oh goodness | 07:14 |
| anteaya | I just refreshed my page | 07:14 |
| * ttx yawns | 07:14 | |
| morganfainberg | anteaya, yeah | 07:14 |
| anteaya | no queue | 07:14 |
| morganfainberg | that is what made me go "uhhhhhh" | 07:14 |
| anteaya | morning ttx | 07:14 |
| lifeless | it'll be ttx again | 07:14 |
| ttx | anteaya: still up? | 07:14 |
| lifeless | he broke it yesterday when he got up :) | 07:14 |
| anteaya | yeah | 07:14 |
| anteaya | it be broke | 07:15 |
| morganfainberg | =/ | 07:15 |
| lifeless | check the json | 07:15 |
| anteaya | http://zuul.openstack.org/status.json | 07:15 |
| anteaya | saved a copy | 07:16 |
| anteaya | not sure what to do with it though | 07:16 |
| ttx | anteaya: was the same yesterday | 07:16 |
| anteaya | :/ | 07:16 |
| anteaya | this was the queue | 07:16 |
| anteaya | 44724,4 running | 07:16 |
| anteaya | 49249,1 | 07:16 |
| anteaya | 49155,1 | 07:16 |
| anteaya | 47969,8 | 07:16 |
| anteaya | 48608,1 | 07:16 |
| lifeless | well, it's not empty :) | 07:16 |
| anteaya | 47463,3 | 07:16 |
| anteaya | 49029,1 glance passed | 07:16 |
| anteaya | 46747,5 | 07:16 |
| anteaya | 47503,16 | 07:16 |
| anteaya | 49266,1 | 07:16 |
| anteaya | 49083,1 | 07:17 |
| anteaya | 48794,2 | 07:17 |
| anteaya | 49105,1 | 07:17 |
| ttx | I saved json for jeblair but the json wasn't really weird | 07:17 |
| anteaya | 48968,1 failed | 07:17 |
| anteaya | 48628,7 failed | 07:17 |
| anteaya | 49275,1 | 07:17 |
| anteaya | 49019,3 | 07:17 |
| anteaya | 49191,1 failed | 07:17 |
| anteaya | 48993,2 | 07:17 |
| anteaya | 48971,2 keystone passed | 07:17 |
| anteaya | 49287,1 failed | 07:17 |
| anteaya | 47453,2 running | 07:17 |
| anteaya | 48836,1 | 07:17 |
| anteaya | 49233,1 | 07:17 |
| lifeless | anteaya: uhm enough :) | 07:17 |
| anteaya | 49058,3 | 07:17 |
| anteaya | 48891,1 | 07:17 |
| lifeless | anteaya: pastebin is better for that | 07:17 |
| anteaya | 47130,11 | 07:17 |
| anteaya | 49088,3 | 07:17 |
| anteaya | 48387,5 | 07:17 |
| anteaya | 49006,1 failed | 07:17 |
| anteaya | 48528,8 | 07:17 |
| ttx | anteaya: i'm reading from the json :) | 07:17 |
| anteaya | I think perhaps there were so many failed branches zuul might have given up | 07:17 |
| anteaya | lifeless: yes you are correct | 07:17 |
| anteaya | I'm tired and lazy, my bad | 07:18 |
| lifeless | anteaya: zuul creates that json; the failure is only in the UI | 07:18 |
| anteaya | http://paste.openstack.org/show/47821/ | 07:18 |
| anteaya | I guess what I am saying is that patch 14 and 15 failed | 07:19 |
| *** SergeyLukjanov has joined #openstack-infra | 07:19 | |
| anteaya | and patch 18 | 07:19 |
| anteaya | and patch 21 | 07:19 |
| anteaya | so 3 different branches for nnfi | 07:19 |
| anteaya | I wonder if that was too much for it to calculate | 07:19 |
| anteaya | so when 44724 completed, it took a long break | 07:20 |
| *** flaper87|afk is now known as flaper87 | 07:20 | |
| anteaya | everything prior to patch 22 had completed | 07:20 |
| ttx | anteaya: go to bed, we'll babysit it from here :) | 07:20 |
| anteaya | k | 07:20 |
| anteaya | thanks good night | 07:20 |
| ttx | good night! | 07:20 |
| anteaya | :D | 07:21 |
| *** anteaya has quit IRC | 07:21 | |
| *** davidhadas has quit IRC | 07:23 | |
| ttx | jd__: zuul status page was broken in the same way same time yesterday | 07:26 |
| *** Ryan_Lane has joined #openstack-infra | 07:26 | |
| *** AlexF has joined #openstack-infra | 07:29 | |
| *** Ryan_Lane has quit IRC | 07:30 | |
| *** vipul-away is now known as vipul | 07:30 | |
| lifeless | ttx: did you file a bug yesterday ? | 07:31 |
| *** odyssey4me2 has quit IRC | 07:32 | |
| *** jcoufal has joined #openstack-infra | 07:34 | |
| *** DinaBelova has joined #openstack-infra | 07:41 | |
| ttx | lifeless: no, I only informally signalled it to jeblair | 07:42 |
| ttx | and sent him a couple status.jsons | 07:42 |
| ttx | lifeless: I wonder if it's not time-sensitive. Zuul's display waits for jeblair to go to bed and then fails. | 07:43 |
| Mithrandir | so we should just get jeblair a coffee drip? | 07:43 |
| *** flaper87 is now known as flaper87|afk | 07:43 | |
| ttx | Mithrandir: that's a known solution to many problems. | 07:44 |
| ttx | Mithrandir: i've been encouraging jeblair to move to France. | 07:44 |
| ttx | that would solve it. | 07:44 |
| Mithrandir | because of the stronger coffee? | 07:45 |
| ttx | Mithrandir: stronger coffee and the resulting 24-h core infra coverage | 07:45 |
| *** odyssey4me has joined #openstack-infra | 07:47 | |
| lifeless | ttx: ITYM infra-admins or whatever the root team name is :)( | 07:47 |
| ttx | lifeless: starting to wonder if it's actually not the periodic jobs f*cking things up. They finish around 0600 UTC which is fail time | 07:48 |
| ttx | at the very least they add complexity to the display | 07:48 |
| *** yassine has joined #openstack-infra | 07:48 | |
| lifeless | it is rather coincidental | 07:49 |
| ttx | lifeless: because the rest of the gate is not significantly busier now | 07:49 |
| openstackgerrit | lifeless proposed a change to openstack-dev/cookiecutter: Ignore editor files. https://review.openstack.org/49318 | 07:50 |
| *** Ryan_Lane has joined #openstack-infra | 07:56 | |
| *** flaper87|afk is now known as flaper87 | 07:59 | |
| *** jcoufal_ has joined #openstack-infra | 07:59 | |
| morganfainberg | ugh i think something went wonky w/ zuul's gate queue. prior to the UI going out, https://review.openstack.org/#/c/48971/ and it's preceeding list appeared to have hit full success. now the json is claiming it's running again | 08:00 |
| *** Ryan_Lane has quit IRC | 08:01 | |
| morganfainberg | either ui was misleading or something partially reset the queue somewhere between the 3hr horizon doc job and ~16 successes down the list | 08:01 |
| *** hashar has joined #openstack-infra | 08:01 | |
| *** jpich has joined #openstack-infra | 08:01 | |
| *** jcoufal has quit IRC | 08:02 | |
| *** davidhadas has joined #openstack-infra | 08:03 | |
| *** thomasbiege has joined #openstack-infra | 08:07 | |
| *** thomasbiege has quit IRC | 08:09 | |
| *** flaper87 is now known as flaper87|afk | 08:09 | |
| *** flaper87|afk is now known as flaper87 | 08:12 | |
| *** davidhadas has quit IRC | 08:13 | |
| *** boris-42 has joined #openstack-infra | 08:15 | |
| *** derekh has joined #openstack-infra | 08:16 | |
| *** AlexF has quit IRC | 08:17 | |
| *** davidhadas has joined #openstack-infra | 08:19 | |
| *** basha has joined #openstack-infra | 08:23 | |
| *** resker has joined #openstack-infra | 08:26 | |
| *** tvb has quit IRC | 08:26 | |
| *** Ryan_Lane has joined #openstack-infra | 08:27 | |
| *** matsuhashi has quit IRC | 08:28 | |
| *** flaper87 is now known as flaper87|afk | 08:28 | |
| *** matsuhashi has joined #openstack-infra | 08:28 | |
| *** esker has quit IRC | 08:29 | |
| *** Ryan_Lane has quit IRC | 08:31 | |
| *** OlivierSanchez has joined #openstack-infra | 08:32 | |
| *** AlexF has joined #openstack-infra | 08:32 | |
| *** mrmartin has joined #openstack-infra | 08:32 | |
| *** sld has quit IRC | 08:33 | |
| *** dizquierdo has joined #openstack-infra | 08:36 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/gerritlib: Simplify GerritWatcher creation. https://review.openstack.org/49326 | 08:37 |
| openstackgerrit | lifeless proposed a change to openstack-infra/gerritlib: Ignore editor files. https://review.openstack.org/49327 | 08:37 |
| *** bnemec has quit IRC | 08:39 | |
| *** flaper87|afk is now known as flaper87 | 08:39 | |
| *** johnthetubaguy has joined #openstack-infra | 08:41 | |
| *** AlexF has quit IRC | 08:42 | |
| *** DinaBelova has quit IRC | 08:44 | |
| hashar | lifeless: awesome :-) | 08:44 |
| hashar | lifeless: I was wondering yesterday wether gerritlib is still used/maintained | 08:45 |
| *** basha has quit IRC | 08:46 | |
| ttx | zuul status is back | 08:47 |
| lifeless | hashar: well, I have no idea :) | 08:47 |
| *** davidhadas_ has joined #openstack-infra | 08:47 | |
| lifeless | hashar: but zuul does use it | 08:48 |
| *** johnthetubaguy1 has joined #openstack-infra | 08:48 | |
| hashar | lifeless: gasp, yet another dependency I will have to package :D | 08:48 |
| lifeless | hashar: package? | 08:49 |
| *** yolanda has joined #openstack-infra | 08:49 | |
| *** johnthetubaguy has quit IRC | 08:50 | |
| *** davidhadas has quit IRC | 08:50 | |
| yolanda | hi, i'm trying to setup permissions on non-interactive users group using git, but i find that non-interactive users group is not listed in groups file. As a consequence, permissions aren't updated until i save them from the UI, then Non-Interactive Users group is created. Any idea on what can be happening? other groups such as Administrators, Anonymous, etc... are in groups file | 08:51 |
| jd__ | ttx: that's good news :) | 08:51 |
| jd__ | and I see a lot of success | 08:51 |
| jd__ | it's like the sky went cleared out of the (bad) clouds | 08:52 |
| hashar | lifeless: at Wikimedia we rely almost exclusively on Debian packages. So whenever there is new python module, I got to package it to be able to install the module in production :-] | 08:55 |
| ttx | jd__: a few things are running extremely slowly though. That test we are waiting on at the top of the queue is not even finished installing devstack | 08:56 |
| ttx | been running for more than one hour now | 08:56 |
| jd__ | :-/ | 08:57 |
| jd__ | ceilometer should be ready soon otherwise | 08:57 |
| *** Ryan_Lane has joined #openstack-infra | 08:57 | |
| lifeless | hashar: ah | 08:58 |
| *** DinaBelova has joined #openstack-infra | 08:59 | |
| *** davidhadas has joined #openstack-infra | 08:59 | |
| hashar | lifeless: that is partly why I haven't upgraded Zuul yet :-] | 09:00 |
| *** Ryan_Lane has quit IRC | 09:01 | |
| *** davidhadas_ has quit IRC | 09:01 | |
| *** tkammer has quit IRC | 09:01 | |
| *** markmc has joined #openstack-infra | 09:01 | |
| *** DinaBelova has quit IRC | 09:08 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/gerritlib: Allow ctrl-Cing gerritlib programs. https://review.openstack.org/49333 | 09:17 |
| *** michchap_ has joined #openstack-infra | 09:26 | |
| *** tkammer has joined #openstack-infra | 09:27 | |
| *** tvb has joined #openstack-infra | 09:27 | |
| *** Ryan_Lane has joined #openstack-infra | 09:27 | |
| *** michchap has quit IRC | 09:29 | |
| *** dizquierdo has left #openstack-infra | 09:30 | |
| *** flaper87 is now known as flaper87|afk | 09:31 | |
| *** tvb has quit IRC | 09:31 | |
| *** Ryan_Lane has quit IRC | 09:32 | |
| *** AlexF has joined #openstack-infra | 09:32 | |
| ttx | grmbl, fail at top of queue after running the test for 2 hours | 09:33 |
| ttx | so it's not just flaky tests. That one took 2 hours to run (and fail): https://jenkins01.openstack.org/job/gate-tempest-devstack-vm-full/17906/ | 09:36 |
| ttx | and then another hour to get the failure noticed and processed. | 09:37 |
| ttx | ...and counting | 09:41 |
| ttx | Jenkins is not properly returning a FAIL for that job, so the whole queue is stuck | 09:41 |
| *** flaper87|afk is now known as flaper87 | 09:42 | |
| ttx | Yep, all results on jenkins01 are stuck | 09:42 |
| ttx | We lost it | 09:43 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: missing parameter error are no more meaningfull https://review.openstack.org/49336 | 09:44 |
| ttx | #status notice One of our Jenkins masters is failing to return results, so the gate is currently stuck. | 09:46 |
| openstackstatus | NOTICE: One of our Jenkins masters is failing to return results, so the gate is currently stuck. | 09:46 |
| ttx | #status notice Jenkins01 is not failing, it's just very slow at the moment... so the gate is not completely stuck. | 09:54 |
| openstackstatus | NOTICE: Jenkins01 is not failing, it's just very slow at the moment... so the gate is not completely stuck. | 09:54 |
| ttx | Note: the slowness apparently caused OK tests to fail: http://logs.openstack.org/29/49029/1/gate/gate-tempest-devstack-vm-full/7355145/console.html | 09:55 |
| *** Ryan_Lane has joined #openstack-infra | 09:58 | |
| *** DinaBelova has joined #openstack-infra | 10:00 | |
| *** Ryan_Lane has quit IRC | 10:02 | |
| *** tkammer has quit IRC | 10:04 | |
| *** darraghb has joined #openstack-infra | 10:05 | |
| markmc | ttx, oh dear | 10:08 |
| *** davidhadas_ has joined #openstack-infra | 10:09 | |
| ttx | There are test jobs like gate-cinder-docs that have been running for 1h22min, waiting for jenkins01 to notice they completed 1h20 min ago | 10:09 |
| ttx | (top-left on zuul status page) | 10:10 |
| ttx | err.. actually that job is still running, which is even more puzzling. | 10:11 |
| *** pblaho has quit IRC | 10:11 | |
| ttx | Lots of proxy errors to jenkins01 too | 10:12 |
| *** davidhadas has quit IRC | 10:12 | |
| *** pblaho has joined #openstack-infra | 10:12 | |
| *** matsuhashi has quit IRC | 10:16 | |
| *** tkammer has joined #openstack-infra | 10:17 | |
| *** SergeyLukjanov has quit IRC | 10:27 | |
| *** schwicht has joined #openstack-infra | 10:27 | |
| *** tvb has joined #openstack-infra | 10:28 | |
| *** dizquierdo has joined #openstack-infra | 10:28 | |
| *** Ryan_Lane has joined #openstack-infra | 10:28 | |
| *** AlexF has quit IRC | 10:30 | |
| *** dims has joined #openstack-infra | 10:30 | |
| *** pblaho has quit IRC | 10:30 | |
| *** tvb has quit IRC | 10:32 | |
| *** Ryan_Lane has quit IRC | 10:32 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:33 | |
| *** pcm_ has joined #openstack-infra | 10:33 | |
| *** pcm_ has quit IRC | 10:34 | |
| *** pcm_ has joined #openstack-infra | 10:34 | |
| *** yaguang has joined #openstack-infra | 10:35 | |
| *** dizquierdo has quit IRC | 10:36 | |
| *** AlexF has joined #openstack-infra | 10:38 | |
| *** sdake_ has quit IRC | 10:40 | |
| *** sdake has quit IRC | 10:41 | |
| *** sdake_ has joined #openstack-infra | 10:41 | |
| *** sdake_ has joined #openstack-infra | 10:41 | |
| *** sdake has joined #openstack-infra | 10:41 | |
| ttx | and/or scp being dead slow | 10:44 |
| *** jeblair has quit IRC | 10:47 | |
| *** jeblair has joined #openstack-infra | 10:48 | |
| *** AlexF has quit IRC | 10:50 | |
| *** weshay has joined #openstack-infra | 10:53 | |
| *** AlexF has joined #openstack-infra | 10:55 | |
| *** Ryan_Lane has joined #openstack-infra | 10:58 | |
| *** fbo is now known as fbo_away | 10:59 | |
| *** dizquierdo has joined #openstack-infra | 11:00 | |
| *** johnthetubaguy1 is now known as johnthetubaguy | 11:01 | |
| openstackgerrit | Endre Karlson proposed a change to openstack-infra/config: Add RTFD hook to trigger auto-build of docs https://review.openstack.org/48437 | 11:01 |
| *** alexpilotti has joined #openstack-infra | 11:02 | |
| *** Ryan_Lane has quit IRC | 11:03 | |
| *** sdake_ has quit IRC | 11:03 | |
| *** sdake_ has joined #openstack-infra | 11:03 | |
| *** sdake_ has joined #openstack-infra | 11:03 | |
| *** markmc has quit IRC | 11:04 | |
| openstackgerrit | Endre Karlson proposed a change to openstack-infra/config: Add RTFD hook to trigger auto-build of docs https://review.openstack.org/48437 | 11:05 |
| *** darraghb has left #openstack-infra | 11:06 | |
| *** SergeyLukjanov has quit IRC | 11:08 | |
| *** dims has quit IRC | 11:09 | |
| *** SergeyLukjanov has joined #openstack-infra | 11:10 | |
| *** yaguang has quit IRC | 11:12 | |
| *** adalbas has joined #openstack-infra | 11:15 | |
| *** nosnos has quit IRC | 11:16 | |
| ttx | yes, bad karma confirmed for jenkins01: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=40 | 11:16 |
| ttx | starting around 1900 UTC yesterday | 11:17 |
| jd__ | damn :( | 11:19 |
| *** julim has joined #openstack-infra | 11:19 | |
| *** afazekas_no_irq has quit IRC | 11:20 | |
| *** luhrs1 is now known as che-arne | 11:21 | |
| sdague | hmmmm | 11:23 |
| *** tvb has joined #openstack-infra | 11:23 | |
| sdague | also this is an interesting thing - https://review.openstack.org/#/c/49006/ | 11:23 |
| sdague | the change was ninja merged by mordred | 11:23 |
| sdague | but garyk ran reverify | 11:23 |
| sdague | so now it's an unpassable change in the gate | 11:23 |
| sdague | and because it's already merged, my normal ninja "new review" trick can't snipe it out | 11:24 |
| *** CaptTofu has quit IRC | 11:25 | |
| *** CaptTofu has joined #openstack-infra | 11:26 | |
| *** Ryan_Lane has joined #openstack-infra | 11:29 | |
| *** Ryan_Lane has quit IRC | 11:33 | |
| ttx | sdague: I think jenkins01 weird condition is the top fire atm | 11:35 |
| ttx | sdague: creates lots of slowness, superfluous fails and edge behavior | 11:35 |
| *** dims has joined #openstack-infra | 11:36 | |
| ttx | sdague: looks like we could use an alarm on that. It's been burning CPU for the last 16 hours now | 11:37 |
| ttx | going slightly better now as jenkins02 is handling most of the load, but some jobs are still getting on jenkins01 tarpit | 11:39 |
| *** dims has quit IRC | 11:41 | |
| *** dims has joined #openstack-infra | 11:42 | |
| *** sandywalsh has quit IRC | 11:49 | |
| *** flaper87 is now known as flaper87|afk | 11:53 | |
| *** CaptTofu has quit IRC | 11:54 | |
| *** CaptTofu has joined #openstack-infra | 11:54 | |
| *** Ryan_Lane has joined #openstack-infra | 11:59 | |
| *** derekh has quit IRC | 11:59 | |
| *** fbo_away is now known as fbo | 12:01 | |
| *** Ryan_Lane has quit IRC | 12:03 | |
| ttx | jenkins01 seemes to have recovered a bit | 12:05 |
| ttx | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=816&rra_id=all | 12:06 |
| ttx | must be smelling that its masters are about to wake up | 12:07 |
| ttx | we should be hiring a servant that doesn't get drunk all night while not being supervised | 12:08 |
| *** AlexF has quit IRC | 12:08 | |
| AJaeger | ttx, just hire somebody in a different timezone ;) | 12:08 |
| *** dprince has joined #openstack-infra | 12:09 | |
| *** DinaBelova has quit IRC | 12:09 | |
| jd__ | good point :) | 12:10 |
| *** hashar has quit IRC | 12:12 | |
| *** mriedem has joined #openstack-infra | 12:12 | |
| *** OlivierSanchez has quit IRC | 12:16 | |
| *** flaper87|afk is now known as flaper87 | 12:17 | |
| mriedem | mordred: thanks a bunch! https://review.openstack.org/#/c/49006/ | 12:18 |
| *** dcramer_ has quit IRC | 12:21 | |
| *** hashar has joined #openstack-infra | 12:26 | |
| *** pblaho has joined #openstack-infra | 12:27 | |
| *** hashar_ has joined #openstack-infra | 12:29 | |
| *** pblaho has quit IRC | 12:29 | |
| *** thomasm has joined #openstack-infra | 12:30 | |
| *** Ryan_Lane has joined #openstack-infra | 12:30 | |
| *** pblaho has joined #openstack-infra | 12:30 | |
| *** osanchez has joined #openstack-infra | 12:31 | |
| *** hashar has quit IRC | 12:32 | |
| *** hashar_ is now known as hashar | 12:32 | |
| *** Ryan_Lane has quit IRC | 12:34 | |
| *** pblaho has quit IRC | 12:34 | |
| *** matty_dubs|gone is now known as matty_dubs | 12:35 | |
| *** markmcclain1 has joined #openstack-infra | 12:38 | |
| *** markmcclain has quit IRC | 12:38 | |
| *** dkranz has joined #openstack-infra | 12:42 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added globbed parameters to the job specification https://review.openstack.org/48688 | 12:48 |
| *** acabrera has joined #openstack-infra | 12:53 | |
| *** thomasbiege1 has joined #openstack-infra | 12:56 | |
| *** thomasbiege1 has quit IRC | 12:56 | |
| *** Ryan_Lane has joined #openstack-infra | 13:00 | |
| *** Ryan_Lane has quit IRC | 13:05 | |
| *** giulivo has joined #openstack-infra | 13:06 | |
| giulivo | guys, I'm not really sure what is failing here? | 13:06 |
| giulivo | https://review.openstack.org/#/c/47508/ | 13:06 |
| *** jcoufal_ is now known as jcoufal | 13:08 | |
| giulivo | I don't think it is the tests but to make sure we address it properly with the recheck | 13:08 |
| giulivo | jog0, | 13:09 |
| ttx | giulivo: could be an artifact of jenkins01 being CPU-bound a few hours ago | 13:09 |
| ttx | giulivo: I would "recheck no bug" | 13:09 |
| ttx | (in that precise case) | 13:09 |
| giulivo | thanks ttx :) | 13:09 |
| giulivo | jog0, I noticed people started to thank you for the hints about the bug they're hitting in the reviews :) | 13:10 |
| *** markmcclain1 has quit IRC | 13:11 | |
| *** pblaho has joined #openstack-infra | 13:11 | |
| *** markmcclain has joined #openstack-infra | 13:12 | |
| *** markmcclain has left #openstack-infra | 13:12 | |
| *** markmcclain has joined #openstack-infra | 13:12 | |
| *** mrmartin has quit IRC | 13:13 | |
| sdague | fungi up yet? | 13:22 |
| ttx | fungi: up? | 13:22 |
| *** changbl has quit IRC | 13:22 | |
| ttx | sdague: wow. | 13:22 |
| sdague | heh | 13:22 |
| ttx | sdague: WE SHARE THE SAME BRAIN | 13:22 |
| ttx | At 3:22 pm we both got some synapse deciding fungi is usually up at that time. | 13:23 |
| *** weshay has quit IRC | 13:24 | |
| sdague | :) | 13:25 |
| *** acabrera has quit IRC | 13:25 | |
| *** ryanpetrello has joined #openstack-infra | 13:25 | |
| *** dkehn_ is now known as dkehn | 13:26 | |
| *** cppcabrera has joined #openstack-infra | 13:27 | |
| mriedem | i'm trying to figure out why this failed, but it's not obvious to me: http://logs.openstack.org/72/48772/1/check/check-tempest-devstack-vm-neutron/db02571/console.html | 13:27 |
| mriedem | does leaked process file descriptors in jenkins cause jobs to fail? | 13:27 |
| *** michchap_ has quit IRC | 13:27 | |
| jd__ | mriedem: no | 13:28 |
| *** alcabrera has joined #openstack-infra | 13:28 | |
| mriedem | jd__: i didn't think so, otherwise nothing glaring as far as errors or failures | 13:28 |
| *** bnemec has joined #openstack-infra | 13:28 | |
| jd__ | mriedem: is this really supposed to the end of the script at this stage? | 13:29 |
| jd__ | I wonder if it didn't exited early | 13:30 |
| *** thomasbiege1 has joined #openstack-infra | 13:30 | |
| mriedem | jd__: that's what it looks like, anteaya was asking about the same last night in another patch | 13:30 |
| mriedem | jd__: this one: https://review.openstack.org/#/c/48971/ | 13:31 |
| *** Ryan_Lane has joined #openstack-infra | 13:31 | |
| mriedem | it was just reverified and passed | 13:31 |
| jd__ | it's possible, there has been issues with jenkins01 recently that could cause such failure IMHO | 13:31 |
| jd__ | you should try a recheck | 13:31 |
| mriedem | ok, in that other patch it was a different devstack job (grenade vs neutron) but the same kind of mystery, no errors, just exited early | 13:32 |
| *** michchap has joined #openstack-infra | 13:34 | |
| *** yassine has quit IRC | 13:34 | |
| *** yassine has joined #openstack-infra | 13:35 | |
| mriedem | hmm, thinking it's more than a fluke, failed here too: https://review.openstack.org/#/c/48300/ | 13:35 |
| mriedem | both are stable/grizzly patches, using quantumclient | 13:35 |
| *** Ryan_Lane has quit IRC | 13:35 | |
| *** rfolco has joined #openstack-infra | 13:35 | |
| mriedem | i don't even see logs for nova compute | 13:36 |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Moved get_scenarios() function to a base module to make it easier to reuse https://review.openstack.org/49267 | 13:37 |
| ttx | sdague: in the gate queue, there is a failed change not triggering a reset: 49006,1 | 13:39 |
| ttx | note sure how to read that | 13:40 |
| ttx | not* | 13:40 |
| *** DinaBelova has joined #openstack-infra | 13:41 | |
| mriedem | https://bugs.launchpad.net/openstack-ci/+bug/1234181 | 13:45 |
| uvirtbot | Launchpad bug 1234181 in openstack-ci "stable/grizzly patches are failing jenkins in check-tempest-devstack-vm-neutron" [Undecided,New] | 13:45 |
| fungi | finally caught up on scrollback. geez you people like to talk | 13:47 |
| fungi | and i find all the problems right at the end of the scrollback. lovely | 13:47 |
| fungi | i need to find a way to read this stuff in reverse from now on | 13:48 |
| * fungi checks to see if things are still broken | 13:49 | |
| fungi | jenkins01 seems to be running jobs okay at the moment. jenkins02 is being very slow to give me a webui | 13:50 |
| fungi | finally showed up (could have been at my end), and is also running jobs | 13:51 |
| fungi | at first glance everything seems to be okay--what are the currently observed issues, if any? or did they all self-correct? | 13:52 |
| ttx | fungi: you should read scrollback in reverse order | 13:52 |
| fungi | ttx: it feels like reading top-posted e-mail to me... breaks my brain somehow | 13:52 |
| ttx | fungi: so yeah, jenkins01 was REALLY SLOW and broke a number of things (extra failed tests, extra slowness) | 13:53 |
| ttx | SCP still seems a bit slow but no longer as much blocking | 13:53 |
| * fungi digs through the cacti graphs on the jenkinses | 13:53 | |
| ttx | fungi: still wondering about 49006,1 status at the gate | 13:54 |
| ttx | it's failed but not triggering any reset | 13:54 |
| ttx | been like this all day | 13:54 |
| ttx | fungi cacti graph at http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=816&rra_id=all | 13:54 |
| ttx | fungi: was actually in a bad condition yesterday in your afternoon | 13:55 |
| *** dcramer_ has joined #openstack-infra | 13:55 | |
| ttx | sounds like we could use an alert on that one | 13:55 |
| *** dkliban has joined #openstack-infra | 13:55 | |
| fungi | ttx: note that mordred force-merged 49006 since it was for quantumclient (not neutronclient) and is as such untestable by the neutronclient jobs | 13:55 |
| fungi | zuul just doesn't realize it's merged, and chucked it into the queue anyway | 13:55 |
| ttx | fungi: hmm, ok, will it just break stuff when at top of queue ? | 13:56 |
| fungi | ttx: that's a very good question, and one that i suspect i will have to wait to discover the answer for | 13:56 |
| fungi | since we do that so infrequently | 13:57 |
| *** weshay has joined #openstack-infra | 13:58 | |
| *** thedodd has joined #openstack-infra | 13:59 | |
| *** DennyZhang has joined #openstack-infra | 13:59 | |
| fungi | i definitely find the cpu/load history on jenkins01 disturbing, and the memory utilization on jenkins02 | 13:59 |
| *** yassine has quit IRC | 14:00 | |
| ttx | fungi: I can tell you it was pretty bad. Proxy errors trying to access it, then a docs job would take 2 hours to run just due to scp | 14:00 |
| ttx | (or appear so) | 14:00 |
| *** yassine has joined #openstack-infra | 14:01 | |
| fungi | yeah, clearly something was eating as much cpu (user) as it could get | 14:01 |
| *** Ryan_Lane has joined #openstack-infra | 14:01 | |
| fungi | well, spiked to 80% user, looks like, and then plateaued there for about 14 hours | 14:02 |
| *** thomasbiege1 has quit IRC | 14:03 | |
| fungi | whatever it was also seemed to free up a good 5gib ram when it cleared up too | 14:04 |
| fungi | maybe the start and/or end times of the plateau will loosely correlate to cron jobs on the machine | 14:05 |
| *** thomasbiege1 has joined #openstack-infra | 14:05 | |
| *** Ryan_Lane has quit IRC | 14:06 | |
| *** AlexF has joined #openstack-infra | 14:07 | |
| ttx | fungi: looks like classic jvm garbage collection metafail to me, but then I hate java | 14:08 |
| *** thomasbiege1 has quit IRC | 14:09 | |
| fungi | yeah, nothing suspiciously timed in various crontabs. i wonder if it has a monthly jvm cleanup task set to fire around 2100utc on the first of the month or something | 14:11 |
| fungi | nothing in the process list which seems to have restarted around the end of the plateau either | 14:12 |
| *** AJaeger has quit IRC | 14:12 | |
| fungi | i'm starting to try and read documentation about profiling and tuning jvm garbage collection, and my brain is already turning to mush. this might require more coffee | 14:15 |
| *** pblaho has quit IRC | 14:15 | |
| mriedem | ttx: fungi: just getting caught up too, https://review.openstack.org/#/c/49006/ was merged to unblock stable/grizzly patches | 14:21 |
| mriedem | but now i'm seeing at least 4 stable/grizzly patches all failing mysteriously in check-tempest-devstack-vm-neutron | 14:22 |
| mriedem | when setting up quantum | 14:22 |
| jeblair | ttx: i've confirmed that the status page bug you observed is due to periodic jobs and is fixed by this patch from Alex_Gaynor: https://review.openstack.org/49312 | 14:22 |
| *** tvb has quit IRC | 14:22 | |
| *** markmc has joined #openstack-infra | 14:24 | |
| jeblair | fungi: we lacked a number of graphs for jenkins0[12], including network; i just added them. | 14:24 |
| ttx | jeblair: cool. | 14:24 |
| fungi | jeblair: yeah, i was wondering about that--what was the fix for the missing graphs? were they not being polled or just not in the config to be displayed? | 14:25 |
| jeblair | fungi: not being polled; it looks like when they were added, the snmp query to get interfaces/disks didn't run; so i manually triggered the queries and then ran the script again to create the graphs | 14:26 |
| *** pblaho has joined #openstack-infra | 14:29 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Make test_required_files.py real unit tests https://review.openstack.org/49265 | 14:30 |
| fungi | got iy | 14:30 |
| fungi | it | 14:30 |
| *** Ryan_Lane has joined #openstack-infra | 14:32 | |
| *** Guest49600 is now known as oubiwann_ | 14:32 | |
| *** resker has quit IRC | 14:32 | |
| *** pblaho has quit IRC | 14:33 | |
| *** ruhe has joined #openstack-infra | 14:33 | |
| *** ryanpetrello has quit IRC | 14:35 | |
| *** ryanpetrello has joined #openstack-infra | 14:35 | |
| ttx | jeblair: for some reason just before you get up, everything starts to behave properly again | 14:36 |
| *** ryanpetrello has quit IRC | 14:36 | |
| *** Ryan_Lane has quit IRC | 14:36 | |
| fungi | i believe it senses his impending presence | 14:36 |
| *** ryanpetrello has joined #openstack-infra | 14:37 | |
| *** rcleere has joined #openstack-infra | 14:37 | |
| ttx | fungi: I suspect Jenkins is drinking all night long to forget about the daylight abuse | 14:38 |
| *** dkliban has quit IRC | 14:38 | |
| fungi | certainly a plausible explanation | 14:39 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Make test_queries save stdout in tox testr https://review.openstack.org/49080 | 14:40 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add mox fixture to base TestCase https://review.openstack.org/49264 | 14:40 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Cleanup README https://review.openstack.org/49281 | 14:40 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1232748 https://review.openstack.org/49280 | 14:40 |
| uvirtbot | Launchpad bug 1232748 in nova "[Postgres] DB error: (OperationalError) could not translate host name "localhost" to address" [Undecided,Confirmed] https://launchpad.net/bugs/1232748 | 14:40 |
| *** ryanpetrello has quit IRC | 14:40 | |
| *** DinaBelova has quit IRC | 14:41 | |
| *** ryanpetrello has joined #openstack-infra | 14:41 | |
| *** anteaya has joined #openstack-infra | 14:42 | |
| *** dkehn_ has joined #openstack-infra | 14:42 | |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Added support for Git shallow clone parameter https://review.openstack.org/49268 | 14:43 |
| *** changbl has joined #openstack-infra | 14:44 | |
| *** dkehn has quit IRC | 14:45 | |
| *** pblaho has joined #openstack-infra | 14:48 | |
| *** changbl has quit IRC | 14:48 | |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Added support for Git shallow clone parameter https://review.openstack.org/49268 | 14:49 |
| *** rnirmal has joined #openstack-infra | 14:50 | |
| *** datsun180b has joined #openstack-infra | 14:51 | |
| *** AlexF has quit IRC | 14:51 | |
| *** odyssey4me has quit IRC | 14:56 | |
| *** thomasbiege1 has joined #openstack-infra | 14:56 | |
| *** tkammer has quit IRC | 15:00 | |
| *** changbl has joined #openstack-infra | 15:00 | |
| *** Cyril2 has quit IRC | 15:01 | |
| *** changbl has quit IRC | 15:01 | |
| *** dkliban has joined #openstack-infra | 15:02 | |
| *** Ryan_Lane has joined #openstack-infra | 15:02 | |
| *** Ryan_Lane has quit IRC | 15:06 | |
| *** adalbas has quit IRC | 15:07 | |
| *** changbl has joined #openstack-infra | 15:08 | |
| *** Ryan_Lane has joined #openstack-infra | 15:09 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Remove shebang line from git_review/cmd.py https://review.openstack.org/49383 | 15:09 |
| *** thomasbiege1 has quit IRC | 15:11 | |
| *** mrodden has quit IRC | 15:15 | |
| *** thomasbiege1 has joined #openstack-infra | 15:17 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-dev/hacking: Allow APL text to start on the first text column https://review.openstack.org/49387 | 15:23 |
| *** esker has joined #openstack-infra | 15:23 | |
| *** tvb has joined #openstack-infra | 15:23 | |
| *** tvb has quit IRC | 15:23 | |
| *** tvb has joined #openstack-infra | 15:23 | |
| *** oubiwann_ is now known as oubiwann | 15:26 | |
| *** tvb has quit IRC | 15:30 | |
| *** Ryan_Lane has quit IRC | 15:31 | |
| *** jomara has joined #openstack-infra | 15:32 | |
| *** flaper87 is now known as flaper87|afk | 15:33 | |
| *** ruhe has quit IRC | 15:33 | |
| sdague | fungi: you up for a ninja merge? | 15:34 |
| *** ruhe has joined #openstack-infra | 15:34 | |
| fungi | rebooting precise29 as it was also unreachable and exhibiting cpu lockups in the console | 15:35 |
| fungi | sdague: sure | 15:35 |
| sdague | dims: what's the review again? | 15:35 |
| fungi | sdague: what are the circumstances for this one? | 15:35 |
| sdague | neutron has too many race issues to land a python-neutronclient fix for one of them | 15:35 |
| dims | https://review.openstack.org/#/c/49210/ | 15:35 |
| sdague | because it can't pass tempest 4 times | 15:36 |
| * fungi is looking it over | 15:36 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Update elasticRecheck imports in bot.py https://review.openstack.org/48878 | 15:37 |
| dims | fungi, httplib2 has a bug running under eventlet which causes bad fd(s) as rpc and httplib2 stomp all over each other. simple fix was to cleanup the connection pool in httplib2 | 15:38 |
| fungi | so it looks like it's only failing the neutron/neutronclient-only jobs. if it does somehow contribute to making those even less stable, it's not likely to destabilize gating for any other projects at least | 15:38 |
| ttx | jeblair: weird optimization idea: when change 2 at the gate is all successful (while change 1 is still running tests), would it make sense to merge both immediately ? | 15:38 |
| ttx | (see current state at the zuul status) | 15:39 |
| ttx | (the trick being, they sometimes don't run the same tests) | 15:39 |
| jeblair | ttx: change 1 could still fail an unrelated test, like a unit test | 15:39 |
| jeblair | ttx: exactly :/ | 15:39 |
| dims | fungi, right. | 15:40 |
| ttx | jeblair: yes, so it would have to be extra smart. Like only assume test success on tests it actually ran | 15:40 |
| ttx | yay, looks like we can have glance rc1 now | 15:40 |
| *** pblaho has quit IRC | 15:40 | |
| ttx | jeblair: just thinking out loud, igore me | 15:40 |
| fungi | dims: can we get at least one neutron core to +2 that first? | 15:41 |
| *** flaper87|afk is now known as flaper87 | 15:41 | |
| fungi | i would feel a little less hesitant to force a patch into their project if they were complicit | 15:41 |
| openstackgerrit | A change was merged to openstack/requirements: Raise Babel requirements to >= 1.3 https://review.openstack.org/48739 | 15:41 |
| *** dizquierdo has quit IRC | 15:41 | |
| *** dizquierdo has joined #openstack-infra | 15:42 | |
| mriedem | mordred: does your fancy bot cascade this out now? ^ | 15:42 |
| jeblair | mriedem: yeah, it should be automatic (it's a jenkins job, it'll run in a few mins) | 15:42 |
| dims | fungi, there was +2 from Salvatore Orlando yday | 15:43 |
| fungi | mriedem: it should open or update the requirements patches for all projects subscribed to them in a bit | 15:43 |
| anteaya | ttx yay | 15:43 |
| fungi | dims: ahh, i see you only updated the commit message since then, which is what cleared his +2. good enough for me | 15:43 |
| dims | thanks fungi | 15:44 |
| jog0 | jeblair fungi: can someone restart elastic-recheck | 15:45 |
| jog0 | it got borked due to some issues on ElasticSearch being slow and the fix patches aren't in yet | 15:45 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Reorganize tests into unit and functional tests https://review.openstack.org/49391 | 15:46 |
| jeblair | jog0: will do | 15:46 |
| jog0 | jeblair: thanks | 15:47 |
| jeblair | jog0: done | 15:47 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/puppet-dashboard: Changing the module to use the sodabrew fork of puppet-dashboard https://review.openstack.org/49392 | 15:47 |
| *** AlexF has joined #openstack-infra | 15:48 | |
| fungi | dims: sdague: merged | 15:48 |
| *** oubiwann has quit IRC | 15:49 | |
| dims | fungi, sdague thanks! | 15:49 |
| *** jcoufal has quit IRC | 15:49 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Stop using While True to wait for ElasticSearch https://review.openstack.org/48766 | 15:49 |
| *** AlexF has quit IRC | 15:50 | |
| *** marun has quit IRC | 15:50 | |
| jog0 | jeblair: it was related to console.html file being cut off | 15:50 |
| *** Ryan_Lane has joined #openstack-infra | 15:50 | |
| hashar | !launchpad nodepool | 15:50 |
| openstack | hashar: Error: "launchpad" is not a valid command. | 15:50 |
| hashar | argh | 15:50 |
| *** ruhe has quit IRC | 15:50 | |
| *** thingee_zzz is now known as thingee | 15:51 | |
| *** yassine has quit IRC | 15:52 | |
| *** boris-42 has quit IRC | 15:54 | |
| *** mrodden has joined #openstack-infra | 15:55 | |
| hashar | jeblair: hey :] I can't report bug against nodepool! https://launchpad.net/nodepool | 15:55 |
| hashar | [Report a bug] is gray | 15:55 |
| jeblair | hashar: that's because it's impossible for there to be bugs! :) | 15:56 |
| jeblair | hashar: i'll set that up real quick, sorry. | 15:56 |
| *** prad has joined #openstack-infra | 15:56 | |
| hashar | it is ok | 15:56 |
| jeblair | hashar: also, i know i still have a lot to do with nodepool -- like write docs. :( | 15:56 |
| hashar | i don't even have a gear backed Zuul ... | 15:56 |
| *** thomasbiege1 has quit IRC | 15:57 | |
| hashar | anyway I noticed nodepool/myjenkins.py which had methods to play with Jenkins nodes. Wanted to request that to be upstreamed. Might do it myself though | 15:57 |
| jeblair | hashar: yeah, i've been meaning to do that too. if you get to it first, that would be great. | 15:58 |
| jeblair | hashar: bug tracker configured | 15:59 |
| hashar | jeblair: it is all about finding out the original authors + reading the bzr man page | 15:59 |
| *** Ryan_Lane has quit IRC | 15:59 | |
| hashar | awesome | 15:59 |
| fungi | jeblair: clarkb: per the thread on the infra list, the fuel devs are wanting to rename several of their projects. objections to adding them to the batch on saturday if they're ready in time? | 15:59 |
| jeblair | fungi: wfm, thx. | 16:00 |
| fungi | i'll reply to the list with the details then | 16:00 |
| fungi | also, precise29 is back online in jenkins and successfully running jobs after the reboot | 16:01 |
| *** gyee has joined #openstack-infra | 16:03 | |
| *** hogepodge has joined #openstack-infra | 16:03 | |
| *** hemnafk is now known as hemna_ | 16:04 | |
| *** oubiwann has joined #openstack-infra | 16:06 | |
| *** portante is now known as portante|afk | 16:06 | |
| *** AlexF has joined #openstack-infra | 16:07 | |
| *** prad has left #openstack-infra | 16:09 | |
| anteaya | welcome back precise29 | 16:10 |
| *** markmc has quit IRC | 16:10 | |
| *** hogepodge has quit IRC | 16:10 | |
| *** matty_dubs is now known as matty_dubs|lunch | 16:12 | |
| *** jpich has quit IRC | 16:12 | |
| *** changbl has quit IRC | 16:12 | |
| *** hogepodge has joined #openstack-infra | 16:12 | |
| *** AlexF has quit IRC | 16:13 | |
| hogepodge | pabelanger: ping | 16:14 |
| russellb | dang this new zuul scheduler is the hotness | 16:14 |
| jeblair | oh thank goodness someone likes it | 16:15 |
| russellb | heh, haters gonna hate | 16:16 |
| russellb | i'm an infra fanboy i think | 16:16 |
| notmyname | russellb: how can one not be? | 16:16 |
| russellb | exactly | 16:16 |
| notmyname | russellb: and then you say, "I know, I'll work on a non-openstack project". now you have code and ci problems :-) | 16:17 |
| russellb | usually complete-lack-of-ci problems, heh | 16:17 |
| notmyname | ya :-) | 16:18 |
| fungi | an absolute vacuum of ci | 16:18 |
| fungi | echoing sounds | 16:18 |
| pabelanger | hogepodge, pong | 16:19 |
| hogepodge | Is there much more I should do on https://review.openstack.org/#/c/49020/ | 16:20 |
| *** SergeyLukjanov has quit IRC | 16:20 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Add hits_by_query method https://review.openstack.org/49398 | 16:20 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: add check_success tool https://review.openstack.org/49246 | 16:20 |
| *** hashar has quit IRC | 16:20 | |
| pabelanger | hogepodge, nope, I think we are good. | 16:22 |
| pabelanger | Hopefully fungi or clarkb can approve | 16:22 |
| hogepodge | probably swamped with release 2 weeks away :-) | 16:22 |
| jeblair | hogepodge: we have 2 more weeks of this! yikes. | 16:23 |
| fungi | yeah, it's been insane. somehow i feel like i've been completely busy and also accomplished absolutely nothing (or if i did, can't remember at all what it was) | 16:23 |
| fungi | seems somehow substantially busier than 6 (much less 12) months ago | 16:24 |
| hogepodge | HK feels bigger | 16:24 |
| jeblair | pabelanger, hogepodge: left a comment; based on the commit msg, i think you want to remove the lines i pointed to, but i may be missing context. | 16:25 |
| hogepodge | jeblair: targets master by default? so not necessary? | 16:26 |
| hogepodge | there could be a branch in the future to handle stable/(grizzly|havana) | 16:26 |
| hogepodge | But for now the module is simple enough to be universal. | 16:27 |
| jeblair | hogepodge: i would expect a future branch to be able to run those tests, so no qualification is necessary | 16:27 |
| jeblair | hogepodge: (what's currently there means "only run this job on master") | 16:28 |
| jeblair | hogepodge: (which is redundant as long as the only branch is master) | 16:28 |
| hogepodge | I see. I'll pull it out. Thanks. | 16:28 |
| clarkb | fungi: iknowright! | 16:28 |
| pabelanger | doh, nice catch | 16:28 |
| clarkb | I am going to be mostly afk today. driving back to seattle land. weather is much better should be fun :) | 16:29 |
| fungi | clarkb: good luck | 16:29 |
| clarkb | jog0 I plan on doing the exponential.backoff thing tonight | 16:29 |
| jeblair | clarkb: sounds nice! | 16:29 |
| hogepodge | jeblair pebelanger maybe I can catch up with you at the summit and learn a bit more about the inner workings of the infra project | 16:29 |
| anteaya | clarkb: I hope you have a safe drive | 16:30 |
| jeblair | hogepodge: i would love nothing more than to share that with you and help you assimilate^Mcontribute! | 16:30 |
| clarkb | ++ | 16:30 |
| jeblair | hogepodge: in case you haven't seen it, http://ci.openstack.org/ has some info. much of it is correct. | 16:30 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Stop using While True to wait for ElasticSearch https://review.openstack.org/48766 | 16:31 |
| jog0 | clarkb: ! | 16:31 |
| hogepodge | jeblair been swimming in that for a week or so now :-) | 16:31 |
| jog0 | enjoy the drive | 16:31 |
| *** dmakogon_ipod has joined #openstack-infra | 16:32 | |
| clarkb | notmyname: I think I am going to have to quote you on that :) | 16:32 |
| hogepodge | jeblair I learn better by doing, so at some point I want to dive into http://ci.openstack.org/running-your-own.html | 16:35 |
| *** dmakogon_ has joined #openstack-infra | 16:36 | |
| *** dizquierdo has left #openstack-infra | 16:36 | |
| fungi | hogepodge: also we're starting to curate our conference presentations at http://docs.openstack.org/infra/publications/ if you like slides | 16:36 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Stop using While True to wait for ElasticSearch https://review.openstack.org/48766 | 16:37 |
| *** dmakogon_ipod has quit IRC | 16:38 | |
| *** marun has joined #openstack-infra | 16:39 | |
| *** emagana has joined #openstack-infra | 16:39 | |
| jog0 | jeblair: I see you toook over patch 49246 | 16:40 |
| jog0 | my same comments from before still apply to it | 16:40 |
| jeblair | jog0: not really (at least not yet); i just needed a function from it, so i pulled out the function and rebased | 16:41 |
| jog0 | ohh | 16:41 |
| *** tvb has joined #openstack-infra | 16:41 | |
| jeblair | jog0: sorry, i should leave a comment about that, will do real quick | 16:41 |
| *** dmakogon_ has quit IRC | 16:42 | |
| *** dmakogon_ipod has joined #openstack-infra | 16:42 | |
| jog0 | just did it | 16:42 |
| jog0 | where is your dependant patch? | 16:42 |
| jeblair | jog0: i'm writing it | 16:42 |
| jog0 | ahh what will it do? | 16:42 |
| jeblair | jog0: make graphs. :) | 16:43 |
| jog0 | jeblair: awesome sauce | 16:43 |
| jeblair | yeah, i think it's going to be really useful | 16:43 |
| jog0 | so one thing that would be neat is to show of all failures what percentege we have classified | 16:44 |
| jog0 | will the graphs be time series as well? | 16:44 |
| *** julim has quit IRC | 16:44 | |
| jeblair | jog0: yes, i'm making a page of 'bug hits over time' for each bug. | 16:44 |
| *** adalbas has joined #openstack-infra | 16:44 | |
| jog0 | cool, kibnana has that if you know what to search | 16:45 |
| *** Ryan_Lane has joined #openstack-infra | 16:45 | |
| jog0 | which is to say people don't know what to search | 16:45 |
| jeblair | jog0: but that's still one bug at a time, right? | 16:45 |
| *** tvb has quit IRC | 16:46 | |
| jog0 | jeblair: yes, its not that useful but maybe a way to use that as part of your solution | 16:46 |
| jog0 | jeblair: for example bug 1230407 | 16:48 |
| uvirtbot | Launchpad bug 1230407 in neutron "VMs can't progress through state changes because Neutron is deadlocking on it's database queries, and thus leaving networks in inconsistent states" [Critical,Confirmed] https://launchpad.net/bugs/1230407 | 16:48 |
| jog0 | http://logstash.openstack.org/#eyJzZWFyY2giOiIgICAgQG1lc3NhZ2U6XCJMb2NrIHdhaXQgdGltZW91dCBleGNlZWRlZDsgdHJ5IHJlc3RhcnRpbmcgdHJhbnNhY3Rpb25cIiAgICAgQU5EIEBmaWVsZHMuZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1xLXN2Yy50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDczMjQ3ODYwN30= | 16:48 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Update elasticRecheck imports in bot.py https://review.openstack.org/48878 | 16:48 |
| *** afazekas_no_irq has joined #openstack-infra | 16:48 | |
| *** SergeyLukjanov has joined #openstack-infra | 16:49 | |
| jog0 | or http://logstash.openstack.org/#eyJzZWFyY2giOiIgICAgQG1lc3NhZ2U6XCJ0ZW1wZXN0LnNjZW5hcmlvLnRlc3RfbmV0d29ya19iYXNpY19vcHMgQXNzZXJ0aW9uRXJyb3I6IFRpbWVkIG91dCB3YWl0aW5nIGZvclwiICAgICBBTkQgQGZpZWxkcy5maWxlbmFtZTpcImNvbnNvbGUuaHRtbFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzgwNzMyNTUxNDcxfQ== | 16:49 |
| jog0 | shows 1224001 is mostly fixed | 16:49 |
| jeblair | jog0: *nod*, i'm patterning it after that, but i don't think i can use it directly for a number of reasons. that's definitely the info i want to see. | 16:49 |
| jeblair | i should have something shortly. | 16:50 |
| jog0 | jeblair: cool, yeah this page will be really really useful | 16:50 |
| *** julim has joined #openstack-infra | 16:52 | |
| *** sarob has joined #openstack-infra | 16:53 | |
| *** tstevenson has joined #openstack-infra | 16:55 | |
| *** emagana has quit IRC | 16:55 | |
| *** dcramer_ has quit IRC | 16:55 | |
| *** hogepodge has quit IRC | 16:55 | |
| *** ryanpetrello_ has joined #openstack-infra | 16:57 | |
| *** ryanpetrello has quit IRC | 16:57 | |
| *** ryanpetrello_ is now known as ryanpetrello | 16:57 | |
| *** julim has quit IRC | 16:58 | |
| *** matty_dubs|lunch is now known as matty_dubs | 16:59 | |
| *** fbo is now known as fbo_away | 17:01 | |
| *** hogepodge has joined #openstack-infra | 17:02 | |
| *** davidhadas_ has quit IRC | 17:04 | |
| *** tstevenson has quit IRC | 17:04 | |
| jeblair | jog0: http://imgur.com/iFNa6nS | 17:05 |
| jog0 | jeblair: nice | 17:06 |
| jog0 | can you add the query in there too | 17:06 |
| jog0 | and the bug title | 17:06 |
| *** sarob has quit IRC | 17:07 | |
| jeblair | jog0: i was planning on adding the query (and linking it to logstash); we'd have to go to lp to get the bug title, yeah? | 17:07 |
| *** sarob has joined #openstack-infra | 17:07 | |
| jog0 | yeah | 17:08 |
| jeblair | jog0: let's leave that for the next round then. :) | 17:08 |
| jog0 | link to query would be cool | 17:08 |
| jog0 | jeblair: works for me | 17:08 |
| openstackgerrit | Chris Hoge proposed a change to openstack-infra/config: Adding puppet-vswitch project to Stackforge https://review.openstack.org/49020 | 17:08 |
| fungi | i wonder if that will be more useful as a percentage of failures found in overall jobs run by the same name (though that gets hairy since the same error could appear in more than one similar yet distinct job) | 17:09 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Reorganize tests into unit and functional tests https://review.openstack.org/49391 | 17:09 |
| *** DinaBelova has joined #openstack-infra | 17:09 | |
| fungi | because as it stands, an occurrence tally will rise and fall with overall job volume | 17:09 |
| *** harlowja has joined #openstack-infra | 17:10 | |
| jeblair | fungi: i think you're correct | 17:10 |
| fungi | or is that y axis already a proportion rather than a linear count? | 17:11 |
| jeblair | fungi: nope, it's a histogram | 17:11 |
| fungi | hard to tell | 17:11 |
| fungi | h | 17:11 |
| fungi | k | 17:11 |
| *** dkranz has quit IRC | 17:12 | |
| *** sarob has quit IRC | 17:12 | |
| *** jerryz has joined #openstack-infra | 17:20 | |
| jeblair | jog0: know how to make one of those base64 encoded urls using python? | 17:21 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add hits_by_query method https://review.openstack.org/49398 | 17:22 |
| *** emagana has joined #openstack-infra | 17:22 | |
| mtreinish | jeblair: something like: base64.urlsafe_b64encode(s) | 17:24 |
| mtreinish | just guessing | 17:24 |
| jeblair | mtreinish: yeah, though i think the input for logstash/kibana is slightly different than the elasticsearch query | 17:24 |
| mtreinish | jeblair: I could definitely see it being that way | 17:25 |
| jog0 | jeblair: not sure | 17:26 |
| jeblair | it looks like a pretty straightforward translation, shouldn't be a big deal. | 17:26 |
| jog0 | probablly can dig up the kibana source code | 17:27 |
| *** dkranz has joined #openstack-infra | 17:29 | |
| *** tvb has joined #openstack-infra | 17:30 | |
| jeblair | got it | 17:31 |
| *** emagana has quit IRC | 17:32 | |
| *** sarob has joined #openstack-infra | 17:32 | |
| *** DennyZhang has quit IRC | 17:34 | |
| *** dkehn_ is now known as dkehn | 17:35 | |
| matty_dubs | mordred: jeblair: I appreciate the help yesterday with looking at AskBot. FWIW I'm bringing in jomara to lend some of his Puppet expertise to this endeavor, so he might have some questions as well. | 17:35 |
| ryanpetrello | potentially dumb question - is there a reason you can't $ pip install horizon from PyPI? | 17:36 |
| *** sarob has quit IRC | 17:36 | |
| *** fbo_away is now known as fbo | 17:37 | |
| *** Ajaeger has joined #openstack-infra | 17:37 | |
| jomara | mordred: hi! when he says expertise, he doesnt mean that | 17:37 |
| *** melwitt has joined #openstack-infra | 17:39 | |
| *** tzumainn has joined #openstack-infra | 17:40 | |
| anteaya | jomara: any level of puppet experience is welcome, however you want to characterize it | 17:41 |
| *** davidhadas has joined #openstack-infra | 17:42 | |
| *** yolanda has quit IRC | 17:44 | |
| *** DennyZhang has joined #openstack-infra | 17:44 | |
| jomara | anteaya: i will characterise it as, "can use sledgehammer to tear down wall" | 17:45 |
| fungi | ryanpetrello: good question actually... we've generally only released client packages onto pypi, as server packages are expected to get installed from distribution vendors instead | 17:46 |
| *** tzumainn has left #openstack-infra | 17:46 | |
| ryanpetrello | yea, figured | 17:46 |
| fungi | ryanpetrello: ...or potentially from source if you're doing continuous deployment | 17:46 |
| ryanpetrello | my predicament is that I'm writing a Horizon extension for personal use, and need to set up tox to install *just* Horizon | 17:46 |
| ryanpetrello | I suppose I could point at Github... | 17:46 |
| fungi | ryanpetrello: we do currently publish release tarballs of the server projects on launchpad.net | 17:47 |
| *** flaper87 is now known as flaper87|afk | 17:48 | |
| fungi | ryanpetrello: and if you need branch-tip tarballs those can be found on tarballs.openstack.org | 17:48 |
| ryanpetrello | yep, that's probably the route I should go (the release tarball) | 17:48 |
| fungi | pip will deal with those tarballs fine since they're already sdist'ed. if you're wanting to get source from github.com for some reason, DO NOT use their tarball feature | 17:49 |
| pleia2 | jomara: I think that's how everyone uses puppet :) | 17:49 |
| ryanpetrello | fungi: yep, http://tarballs.openstack.org/horizon/horizon-master.tar.gz is probably what I want, then | 17:50 |
| fungi | ryanpetrello: also, depending on your distro, you may simply be able to install their horizon release packages, but presumably either your distro is lacking there or you've already ruled that option out for other reasons | 17:50 |
| ryanpetrello | sure | 17:50 |
| ryanpetrello | the trickiness is that I'm just installing into a virtualenv via tox | 17:50 |
| ryanpetrello | so I can run tests against my code that uses horizon server | 17:50 |
| ryanpetrello | but a direct tarball link should work | 17:50 |
| fungi | ryanpetrello: yes, .*-master.tar.gz get rebuilt on every change merged to a project, so pretty close to cd | 17:50 |
| ryanpetrello | great - thanks! | 17:51 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Cleanup tests https://review.openstack.org/48764 | 17:52 |
| fungi | jomara: my sledge hammer is fairly worn at this point | 17:52 |
| *** boris-42 has joined #openstack-infra | 17:52 | |
| anteaya | jomara: duly filed as such | 17:54 |
| mriedem | if i want to change something in the check-tempest-devstack-vm-neutron job, is that in devstack or devstack-gate? | 17:55 |
| anteaya | so it appears we could do with a fresh sledge, wonderful | 17:55 |
| Ajaeger | jeblair, fungi: any words of wisdom on this one https://review.openstack.org/#/c/48128/ ? I'm not sure which way to go... | 17:55 |
| anteaya | and a popular choice | 17:56 |
| *** sgordon has joined #openstack-infra | 17:56 | |
| jeblair | Ajaeger: so i understand, why did you want 4 jobs? and if you reduced to 2 jobs, what would they be? | 17:56 |
| fungi | Ajaeger: yeah, so it seems like some of those tests being in separate jobs is simply so they'll get run in parallel to increase throughput/decrease overall run time | 17:57 |
| Ajaeger | jeblair: we have four different tasks we do: Syntax check, check for deletions and building of books - and a niceness check (whitespace) | 17:58 |
| Ajaeger | We can run them in parallel if we split them up. | 17:58 |
| jeblair | Ajaeger: do they take long? | 17:58 |
| *** changbl has joined #openstack-infra | 17:58 | |
| fungi | but there are also potentially other ways to do that without having them all run on separate machines (for example if the scripts could be run in parallel on the same machine and then results collated) | 17:58 |
| Ajaeger | On the other hand, the build job is very long (2 mins for one book, 10 for all), the others are small (seconds) | 17:58 |
| Ajaeger | building is done already in parallel on one machine | 17:59 |
| *** julim has joined #openstack-infra | 18:00 | |
| Ajaeger | I can merge the three voting jobs into one, no problem at all. We (-docu) just considered it better to have one gate for each major test we do. | 18:00 |
| *** flaper87|afk is now known as flaper87 | 18:01 | |
| Ajaeger | fungi, jeblair I don't think it makes much difference whether its 4 or 2 timewise. | 18:01 |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Added some options and pep8 compliance https://review.openstack.org/49406 | 18:01 |
| jeblair | i'm okay with 4, especially if they represent distinct tests... | 18:03 |
| Ajaeger | jeblair: 4 tasks - but run from the same python script | 18:03 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Fix fuel repos layout https://review.openstack.org/49407 | 18:04 |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Added some options and pep8 compliance https://review.openstack.org/49406 | 18:05 |
| *** sarob has joined #openstack-infra | 18:05 | |
| jeblair | Ajaeger: yeah, it sort of sounds like how we run several things (flake8, unit tests) from tox; those are separate jobs | 18:06 |
| jeblair | Ajaeger: so that's why i think it's fine; but here's a test you can use to help decide -- | 18:06 |
| jeblair | Ajaeger: think about how the results will be reported to the user; whether or not it would help to have a separate line for that job in the report on gerrit | 18:07 |
| *** sarob has quit IRC | 18:09 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Add graph script https://review.openstack.org/49410 | 18:09 |
| jeblair | jog0: ^ part 1/2 | 18:10 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Add graph script https://review.openstack.org/49410 | 18:10 |
| fungi | Ajaeger: jeblair: yep, as long as there's a reason for job proliferation (particularly when it comes to improving usefulness of developer feedback), i'm good with it | 18:10 |
| mtreinish | jeblair: cool | 18:11 |
| jeblair | Ajaeger: i think you should remove the failure and success messages for the niceness check -- they don't indicate anything out of the ordinary | 18:11 |
| Ajaeger | annegentle_, sgordon: What do you think? I'm not 100 % convinced that having them as four jobs will *always* help. | 18:11 |
| mtreinish | jeblair: I'm wondering how many of the hypothetical todos I put in the summit session description will be done by summit | 18:12 |
| Ajaeger | jeblair: ok, will do. | 18:12 |
| jeblair | mtreinish: ha, i haven't read it. :) | 18:12 |
| jeblair | mtreinish: i just felt like i kept wanting to run queries to look at graphs and realized that wasn't the best use of everyone's time. :) | 18:12 |
| fungi | Ajaeger: i think as long as you feel like it will help sometimes, that's good enough. no need to burn more engineering time on an otherwise excellent review | 18:13 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Create new gates for openstack-manuals https://review.openstack.org/48128 | 18:13 |
| Ajaeger | fungi: YEs, I think it will help sometimes. | 18:14 |
| fungi | wfm | 18:14 |
| Ajaeger | jeblair: I removed the the failure/success message and send a new version ^ | 18:14 |
| Ajaeger | jeblair, fungi : While I have your attention: sgordon mentioned that you had some other comments on how to improve the docs jenkins jobs. Is there anything you like me to look at? | 18:15 |
| *** osanchez has quit IRC | 18:16 | |
| *** pblaho has joined #openstack-infra | 18:16 | |
| fungi | Ajaeger: well, the biggest thing for me is that many of them build with maven even though they're not java programs... seems a little odd, but probably a pragmatic choice at some point in the past which has not been reevaluated lately? | 18:17 |
| fungi | Ajaeger: also, the way a lot of the documentation shares common target locations makes those places more likely to accumulate cruft since there's no safe way to tell which files are safe to remove | 18:18 |
| Ajaeger | fungi: regarding maven: no clue about that one | 18:19 |
| Ajaeger | fungi: yes, the cruft is bad. We would need to run rsync --delete-after instead of uploading via ftp | 18:19 |
| Ajaeger | but there's no publisher for it ;( | 18:19 |
| jeblair | Ajaeger: can you propose a summit session to talk about infra needs for docs (if there isn't one already)? | 18:20 |
| Ajaeger | jeblair: I'm not coming to the summit ;( | 18:20 |
| jeblair | Ajaeger: nooo! | 18:21 |
| Ajaeger | jeblair: next time ;) | 18:21 |
| jeblair | Ajaeger: want to try to remote | 18:21 |
| jeblair | remote-participate, or is the timezone too bad for you? | 18:21 |
| fungi | Ajaeger: well, yes there are technical challenges with cleaning up cruft now, but if the jobs aren't designed to provide any hope of making that possible then there won't be a lot of impetus to solve the technical issues (different publisher mechanisms, maybe different hosting systems) | 18:21 |
| * Ajaeger needs some time to see whether anything can be done here... | 18:23 | |
| Ajaeger | perhaps some kind of filelist to iterate over? | 18:23 |
| Ajaeger | jeblair: let me check what's possible remote, hadn't thought about it | 18:24 |
| fungi | Ajaeger: refactoring the manuals into separate git repositories with another repository for the theming/template pieces might also be helpful. right now a typo in one manual means we rebuild and reupload 20 other manuals | 18:24 |
| Ajaeger | fungi: indeed | 18:24 |
| fungi | and extracting out templates/theme bits also makes them more generally reusable elsewhere | 18:25 |
| Ajaeger | the books share common files (documentation files shared by guides) and some build tools. | 18:27 |
| sgordon | Ajaeger, i think what fungi just described is what i was talking about when i said we build too much | 18:27 |
| sgordon | the validate/build on the review only builds impacted stuff | 18:27 |
| sgordon | but publishing builds it all | 18:27 |
| Ajaeger | sgordon: agreed | 18:27 |
| sgordon | and yeah common content is an on-going issue | 18:28 |
| Ajaeger | fungi, jeblair thanks for the food for thought, will see what I can do... | 18:28 |
| sgordon | could be handled via a git submodule though | 18:28 |
| fungi | sgordon: no need for submodules there | 18:28 |
| fungi | sgordon: the build job can easily grab code from more than one git repository | 18:29 |
| Ajaeger | fungi: still, it needs to be easy for us humans as well | 18:29 |
| fungi | (and also there's the issue that much of our ci tooling in its current form can't support submodules, nor do we want it to) | 18:29 |
| sgordon | yeah but most existing editors and validators wont | 18:29 |
| sgordon | so you screw the authors | 18:30 |
| sgordon | ok so that probably explains how we ended up where we are with all the books in one repo alongside the common content folder ;) | 18:30 |
| fungi | i'm sure my limited view comes from writing and editing content abstracted completely from presentation | 18:30 |
| *** johnthetubaguy has quit IRC | 18:30 | |
| *** sdake has quit IRC | 18:30 | |
| *** sdake has joined #openstack-infra | 18:30 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Add graph script https://review.openstack.org/49410 | 18:32 |
| Ajaeger | fungi: still it might help us look at it from a differentangle | 18:33 |
| fungi | sgordon: another alternative to submodules which allows you to provide the common template/boilerplate/formatting pieces in-tree with the unique content is the approach which oslo-incubator uses | 18:34 |
| jog0 | jeblair: I think elastic-recheck bot is hung again | 18:34 |
| jog0 | its working locally on my box and found two more bugs that it didn't | 18:34 |
| fungi | sgordon: basically keep a master copy of ommon files in a separate repository and have a script which copies that into the document to update it, then make sure reviewers insist that changes to the common parts go to the common repository | 18:35 |
| *** dkranz has quit IRC | 18:36 | |
| Ajaeger | fungi: is there a way to run only some gate jobs based on the output of some script? | 18:36 |
| Ajaeger | We have a script that figure out which books to build - but don't use it for the publishing jobs to limit which to run | 18:37 |
| jog0 | sdague mtreinish: can I get some eyes on https://review.openstack.org/#/c/48766/ | 18:37 |
| sgordon | fungi, the second half is actually the part i suspect we'll be able to handle sooner | 18:37 |
| fungi | Ajaeger: there is a way to run specific jobs when changes touch files matching a list of patterns | 18:37 |
| jog0 | it should make elastic-recheck more robust | 18:37 |
| sgordon | i have on my list to write a check that shared content is in the common directory (or more accurately that shared content does not exist outside of it) | 18:37 |
| sgordon | the former is going to require more invasive work in clouddocs-maven which im not as familiar with | 18:38 |
| fungi | Ajaeger: that way it might be possible to only build manual-x when only files in its directory are changed, but rebuild all manuals when files in the shared content directory change | 18:39 |
| fungi | Ajaeger: but since i don't know a lot about how the source repository for the manuals is organized, i have no idea whether that will help | 18:39 |
| Ajaeger | fungi: let me investigate that one, there might be some lowhanging fruit here | 18:39 |
| *** mrmartin has joined #openstack-infra | 18:40 | |
| Ajaeger | fungi: do you have a pointer or example for this? | 18:41 |
| jog0 | fungi: ^^^ elastic-recheck needs a light kick | 18:41 |
| fungi | jog0: restarting it now | 18:41 |
| *** portante|afk is now known as portante | 18:41 | |
| jog0 | fungi: thanks! not sure what happened it kept working fine for me | 18:41 |
| fungi | Ajaeger: i'll get you a link momentarily | 18:41 |
| fungi | jog0: restarted | 18:42 |
| sdague | jog0: looking | 18:42 |
| jog0 | sdague fungi thanks | 18:43 |
| lifeless | mordred: so I made git@github.com:rbtcollins/gerrit-hook.git | 18:43 |
| sdague | jog0: sure, I think we need to fix it in a real way later, but I'm ok with the patch for now | 18:43 |
| lifeless | mordred: I want to have a release of it and put it in gerrit; is it ok to put it in openstack/ or should it go somewhere else? | 18:44 |
| jog0 | sdague: part of a real fix is to move away from polling logstash | 18:44 |
| jog0 | all together | 18:44 |
| *** DinaBelova has quit IRC | 18:44 | |
| sdague | clarkb: so what are the odds we could add the cloud-region to the logstash queries? | 18:45 |
| sdague | that would be kind of interesting | 18:45 |
| fungi | Ajaeger: the files job parameter is described in http://ci.openstack.org/zuul/zuul.html#jobs and you can see it in use at https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n229 | 18:46 |
| *** sarob has joined #openstack-infra | 18:46 | |
| Ajaeger | fungi: thanks! | 18:46 |
| Ajaeger | will bookmark and read later | 18:46 |
| * Ajaeger needs to do some other stuff now... | 18:46 | |
| Ajaeger | fungi: yeah, I see some low-hanging fruit! | 18:47 |
| fungi | Ajaeger: pick it when it's ripe! | 18:47 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Stop using While True to wait for ElasticSearch https://review.openstack.org/48766 | 18:47 |
| fungi | or maybe pick it early and let it ripen? i really don't know what idiom i was going for there | 18:47 |
| Ajaeger | fungi: will do either way ;) Thanks a lot! | 18:48 |
| fungi | lifeless: we already have some of those in openstack-infra/jeepyb... could we replace them with your thing or break them out of jeepyb and add them into yours? if so, i'd say it fits well in the openstack-infra org | 18:50 |
| *** jerryz has quit IRC | 18:51 | |
| *** dkranz has joined #openstack-infra | 18:51 | |
| jog0 | sdague: should be doable. ask clarkb when he gets back from driving | 18:53 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Add graph script https://review.openstack.org/49410 | 18:54 |
| sdague | man, sorting python dicts by value elements always gets me turned around | 18:54 |
| lifeless | fungi: I'm easy | 18:54 |
| lifeless | fungi: I just want a place to put it and do releases from | 18:54 |
| lifeless | fungi: if you say 'openstack-infra with openstack-infra-core as reviewers' thats fine, I'll submit the config patch for that | 18:55 |
| *** dcramer_ has joined #openstack-infra | 18:56 | |
| *** vipul is now known as vipul-away | 18:57 | |
| fungi | lifeless: i think the deciding factor there is "if it's useful to the infra team or for users/devs interacting with openstack's developer community infrastructure" | 18:59 |
| *** sarob has quit IRC | 19:00 | |
| fungi | so probably, but i'd get a second core opinion just for confirmation | 19:00 |
| *** sarob has joined #openstack-infra | 19:00 | |
| *** danger_fo_away is now known as danger_fo | 19:00 | |
| *** sarob has quit IRC | 19:01 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add elastic-recheck graphs to status https://review.openstack.org/49417 | 19:03 |
| jeblair | jog0: ^ change 2/2 | 19:03 |
| jeblair | jog0, sdague, mtreinish, clarkb: http://status.openstack.org/elastic-recheck/ | 19:03 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add check_success tool https://review.openstack.org/49246 | 19:04 |
| hub_cap | what version of centos is py26 run on? | 19:05 |
| jeblair | hub_cap: CentOS release 6.4 (Final) | 19:06 |
| jomara | jeblair: hey, n ot sure if you saw matty dubs introduction, but he handed off the askbot thing to me and i need to get up to speed | 19:06 |
| zul | coolio looks like we might be able to making python3 gate voting for python-novaclient soon | 19:06 |
| hub_cap | <3 jeblair | 19:06 |
| jeblair | jomara: how can i help? (i'm about to grab lunch though, i may have to help you when i get back) | 19:07 |
| *** DinaBelova has joined #openstack-infra | 19:08 | |
| *** changbl has quit IRC | 19:10 | |
| jomara | jeblair: no problem, you can lunch on | 19:10 |
| jomara | jeblair: i have basically no details other than the goal i need to accomplish and a tiny bit of history, so i just know i need to perform the installation with puppet :) | 19:11 |
| *** thingee is now known as thingee_zzz | 19:19 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add check_success tool https://review.openstack.org/49246 | 19:22 |
| jeblair | lifeless: the name of your tool confused me... | 19:25 |
| jeblair | lifeless: gerrit has hooks, so i thought it was a tool to help you run things using a gerrit hook | 19:26 |
| lifeless | fungi: have you looked at the code? Were you similarly confused with jeblair ? | 19:26 |
| jeblair | lifeless: but instead, this watches the event stream and performs actions based on that | 19:26 |
| fungi | lifeless: oh, i made assumptions based on the name/readme | 19:26 |
| fungi | hadn't looked closely yet | 19:27 |
| lifeless | fungi: the readme is clear I think | 19:27 |
| lifeless | the name I did not realise would be confusing | 19:27 |
| fungi | i may have read into it what i expected, in that case | 19:27 |
| jeblair | lifeless: which reminds me that we have a tool that does something similar... have you considered using zuul + https://github.com/rcbau/turbo-hipster ? | 19:27 |
| lifeless | jeblair: yes, its more moving parts than I want to consider for now | 19:27 |
| lifeless | jeblair: since we /want/ serialisation | 19:27 |
| fungi | lifeless: my confusion is that gerrit has hooks, more or less like git hooks, which are usually shell scripts it runs on specific gerrit events | 19:28 |
| lifeless | jeblair: a future iteration of the CD cloud may well move to zuul | 19:28 |
| lifeless | fungi: what context do they run in? | 19:28 |
| fungi | lifeless: on the gerrit server | 19:28 |
| jeblair | lifeless: ok. that's possible with the independent queue manager + single worker. also, new queue managers are possible. | 19:28 |
| lifeless | ok, so this is remote | 19:28 |
| fungi | yeah, we have gerrit hooks (it even explicitly calls them hooks) which we run on gerrit servers | 19:28 |
| lifeless | jeblair: yes, I'm aware it's possible, and am interested in that in future, but higher entry cost | 19:29 |
| fungi | so i read far too much into the name and let it cloud my interpretation of the description | 19:29 |
| lifeless | jeblair: which may just be my ignorance of the easiness of zuul | 19:29 |
| lifeless | so in some ways I'd like this to be on stackforge | 19:29 |
| lifeless | to indicate it's a thing but maybe it's not a Thing | 19:30 |
| lifeless | but if it sstuck around I don't want you guys to pay the overhead of moving it in future | 19:30 |
| fungi | yes, stackforge seems more appropriate if its future is uncertain and we're not using it in the upstream infrastructure | 19:30 |
| fungi | and i agree this seems far more like something we'd use zuul for if we did need it in the upstream infrastructure | 19:31 |
| *** sarob has joined #openstack-infra | 19:31 | |
| lifeless | right, and I have zero interest in replacing zuul; this is totally about having a low cost quick-experiment facility | 19:32 |
| lifeless | jeblair: fungi: any suggestions for a better name? | 19:32 |
| lifeless | gerrit-stream-reactor? | 19:32 |
| *** sarob has quit IRC | 19:32 | |
| fungi | dont-cross-the-streams | 19:32 |
| *** sarob has joined #openstack-infra | 19:33 | |
| fungi | gerrit-eventerator | 19:33 |
| * fungi sucks at names | 19:34 | |
| annegentle_ | gerrit-rivulet (new name for me, means a stream) :) | 19:35 |
| * ttx closes shop for today | 19:36 | |
| jeblair | lifeless: if it's a quick experiment with an uncertain future (actually, you seem to be saying it has a probable dead-end future), why put it in gerrit at all? | 19:36 |
| jeblair | ttx: see http://status.openstack.org/elastic-recheck/ ? | 19:36 |
| *** sarob has quit IRC | 19:37 | |
| *** julim has quit IRC | 19:37 | |
| lifeless | jeblair: if it's not in gerrit, it's 10 times harder to collaborate on; if it doesn't have releases it's much harder for folk to reason about use of it | 19:37 |
| lifeless | jeblair: and we have a team of 20ish folk driving tripleo so those factors do matter to us | 19:38 |
| *** sarob has joined #openstack-infra | 19:39 | |
| sdague | jeblair: so you might want to liberate some stuff from the check_success tool | 19:39 |
| sdague | threw some comments in your review | 19:39 |
| jeblair | sdague: well, i clearly liberated most of the structure from check_success. thanks. :) want a co-authored-by line? | 19:40 |
| sdague | heh | 19:40 |
| sdague | well I just did a refactor, and found some other issues :) | 19:40 |
| sdague | you might want to reliberate | 19:40 |
| sdague | like my set counting was error prone | 19:40 |
| jeblair | sdague: oh, ok, the uuid is for de-duplication then? | 19:41 |
| sdague | yep | 19:41 |
| jeblair | sdague, jog0: oh, i thought you wanted entry points rather than standalone tools... | 19:41 |
| sdague | jeblair: I think there is debate, I thought we were moving to a tools model | 19:42 |
| jeblair | sdague: i like entry points because, as you'll see in https://review.openstack.org/#/c/49417/ i can just install e-r, and then call the script from cron | 19:43 |
| jeblair | (without having to run it out of the repo) | 19:43 |
| *** DinaBelova has quit IRC | 19:43 | |
| jeblair | sdague: but i'll go with the flow if you guys don't want that. | 19:43 |
| sdague | sure, would there be an issue to have the entry points over in tools? | 19:43 |
| *** dkranz has quit IRC | 19:44 | |
| sdague | well things are moving fast, so lets say there is creative collision :) | 19:44 |
| lifeless | how about | 19:44 |
| lifeless | gerrit-client-hooks ? | 19:44 |
| lifeless | fungi: jeblair: ^ | 19:44 |
| jeblair | sdague: i don't think tools would be installed... | 19:44 |
| jeblair | sdague: (since i think setup.cfg just says to install elastic_recheck) | 19:45 |
| sdague | right, it does now. Sorry I hadn't thought about the entrypoints bit yet | 19:45 |
| fungi | lifeless: it's at least as good as echo `shuf /usr/share/dict/words|grep '^[a-z]*$'|head -n$(($RANDOM%3+1))`|sed 's/ /-/g' | 19:45 |
| lifeless | fungi: high praise! | 19:45 |
| mordred | fungi likes giving out the praise | 19:46 |
| jeblair | lifeless: that is less confusing (it would at least make me think twice; i couldn't say for sure that i'd get the idea right away) | 19:46 |
| lifeless | fungi: more importantly, does it reduce the confusion with gerrit serverside hooks ? | 19:46 |
| lifeless | ok cool | 19:46 |
| lifeless | I will rename it and submit a config request for it for stackforge | 19:46 |
| fungi | lifeless: probably, yes | 19:46 |
| sdague | jeblair: let me fix my patch | 19:47 |
| sdague | ls | 19:48 |
| jeblair | sdague: the graph script is designed to keep track of all job results, so i think it's okay that success shows up (as long as it's getting categorized properly) | 19:49 |
| jeblair | sdague: does that make sense, or am i missing something? | 19:49 |
| sdague | success showing up I think is fine | 19:49 |
| sdague | but I thought there was still a hit count issue, as you are just using results | 19:49 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add check_success tool https://review.openstack.org/49246 | 19:50 |
| jeblair | sdague: yeah, i'll fix that | 19:50 |
| sdague | ok, so that does it with entry points, but also lives outside the elastic_recheck dir | 19:50 |
| jeblair | sdague: oh, that works? | 19:51 |
| sdague | jeblair: yep, did for me | 19:51 |
| jeblair | neato | 19:51 |
| jeblair | didn't expect that | 19:51 |
| annegentle_ | jeblair: sdague: by the way, how do you do a co-authored line on a patch? | 19:53 |
| jeblair | annegentle_: Co-Authored-By: foo <foo@example.com> | 19:53 |
| annegentle_ | jeblair: cool I had a need for that a bit ago | 19:54 |
| mordred | sdague: I don't think that's going to work as well as you think in a pure install | 19:54 |
| mordred | sdague: because tools.check_success is not going to get installed | 19:54 |
| *** oubiwann has quit IRC | 19:55 | |
| sdague | mordred: ok | 19:55 |
| jeblair | sdague: elastic_recheck/cmd/check_success | 19:55 |
| mordred | sdague: so the entry point created for elastic-recheck-success will point to an unresolvable path. however, when you run develop and you're still in that dir, the entry point will find it | 19:55 |
| mordred | sorry | 19:55 |
| *** vipul-away is now known as vipul | 19:55 | |
| jeblair | (i think that's a convention i've seen elsewhere) | 19:55 |
| sdague | mordred: ok, so if we move it to cmd I'm good with it | 19:55 |
| annegentle_ | jeblair: does that "do" anything (adding co-authored-by)? | 19:55 |
| sdague | I just want some separation, and I really do want to be able to run some of the stuff from in tree | 19:56 |
| annegentle_ | jeblair: I'll add info to https://wiki.openstack.org/wiki/GitCommitMessages | 19:56 |
| jeblair | annegentle_: nope, but it's a convention, and we've been encouraging the stats tools to observe it | 19:56 |
| jeblair | annegentle_: i thought it was there! | 19:56 |
| annegentle_ | jeblair: cool, just did a search in the wiki and that page and couldn't find it | 19:56 |
| jeblair | annegentle_: i agree it's not. i'm confused. | 19:56 |
| sdague | mordred: so where should I fully fill my head with entry point info, so I don't be silly in the future? | 19:57 |
| mordred | sdague: muh... | 19:58 |
| mordred | sdague: you really don't want to know more :) | 19:58 |
| mordred | sdague: actually, it's easy as implemented by pbr | 19:58 |
| mordred | sdague: what happens is that, for the thing listed as the target of the entrypoint | 19:58 |
| sdague | ok, I just do want to find a way to run from inside the tree easily as well | 19:58 |
| *** adalbas has quit IRC | 19:58 | |
| mordred | a very small script is generated that does "from %(target_lib)s import %(target_object)s ; def main(): return %(target_object)s" | 19:59 |
| *** dcramer_ has quit IRC | 19:59 | |
| mordred | sdague: so you could actually have a script in tools that just does that if the entry_point generation is causing you angst | 19:59 |
| mordred | there is no other magic - it's just a way to avoid committing boilerplate code to tree at this point | 20:00 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add check_success tool https://review.openstack.org/49246 | 20:00 |
| sdague | mordred: ok, so better? | 20:01 |
| mordred | sdague: (if you look in /usr/local/bin, you should see the script that pbr generated - it should work unmodified if copied into tyour local dir) | 20:01 |
| mordred | > | 20:01 |
| mordred | I don't see this as about Lauren either. But, it is a real world example which I think is a useful benchmark. I worry we spend a huge amount of time anticipating all the bad things that *COULD* happen vs. what the reality is. I see a different analogy here. It is not uncommon for a corporate BoD to have members of the company on the board. So, there are clearly examples of both options. It is up for | 20:01 |
| mordred | us to decide. I happen to think being more inclusive is better. So, I am in favor of allowing foundation employees on the board as suggested. | 20:01 |
| *** julim has joined #openstack-infra | 20:01 | |
| mordred | sdague: perfect | 20:01 |
| mordred | sdague: also, it looks like running "python elastic_recheck/cmd/check_success.py" should also work like a champ | 20:02 |
| *** dmakogon_ipod has left #openstack-infra | 20:04 | |
| jomara | jeblair: he mentioned something about there being an issue w/ setting up postgres | 20:04 |
| jomara | jeblair: is it just missing puppet classes? | 20:04 |
| *** oubiwann has joined #openstack-infra | 20:05 | |
| jeblair | jomara: for the full context, you might want to read http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2013-09-30.log starting at 2013-09-30T20:24:58 | 20:05 |
| *** tvb has quit IRC | 20:06 | |
| jomara | jeblair: ahh, that will help, thanks | 20:06 |
| *** dprince has quit IRC | 20:07 | |
| *** dkliban has quit IRC | 20:08 | |
| jog0 | jeblair: nice! | 20:08 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Run some manual post jobs only if needed https://review.openstack.org/49429 | 20:08 |
| Ajaeger | fungi: here's a first ripe fruit | 20:09 |
| jog0 | although the bar saying success of failure is a little long | 20:09 |
| Ajaeger | sgordon: adding patterns for common + the directory should be easy once you cleaned up the references | 20:09 |
| jog0 | jeblair: also can you make the bug numbers links to launchpad | 20:09 |
| jeblair | jog0: yeah, LOTS of things could be made better | 20:09 |
| jog0 | jeblair: the links to launchpad is really simple | 20:10 |
| jeblair | jog0: there are links to launchad under the graph, next to the link to logstash | 20:10 |
| jog0 | jeblair: just use 'https://bugs.launchpad.net/bugs/%s' % bug_number | 20:10 |
| jog0 | ohh didn't see that nice | 20:10 |
| jeblair | i'd like to focus on structural issues and get this merged, then it'll be easier for others to hack on it | 20:11 |
| jog0 | jeblair: ++ | 20:11 |
| fungi | jog0: you can even omit the bugs subdomain and launchpad.net's director will figure it out from the url pattern | 20:11 |
| jog0 | fungi: hehe nice | 20:11 |
| jog0 | bug 1224001 is impressive | 20:11 |
| uvirtbot | Launchpad bug 1224001 in neutron "test_network_basic_ops fails waiting for network to become available" [Critical,In progress] https://launchpad.net/bugs/1224001 | 20:11 |
| jog0 | in your graph jeblair | 20:11 |
| jog0 | looks like it dropped off recently | 20:11 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Add graph script https://review.openstack.org/49410 | 20:12 |
| *** dcramer_ has joined #openstack-infra | 20:13 | |
| jeblair | sdague: i did uuid de-duplication a little differently than you ^ | 20:13 |
| sdague | jeblair: ok, sure | 20:13 |
| sdague | honestly, next up is to write a class to handle the results in some more sane way. 5 levels of dictionaries, with leaves than might be strings or might be arrays is pretty ugly to carry around in the code | 20:14 |
| jomara | jeblair: i feel like this conversation is happening around me! ive given each of you voices | 20:14 |
| *** julim has quit IRC | 20:15 | |
| mtreinish | jog0: that drop off is because it only affects neutron now. sdague turned off tenant isolation off again for the neutron runs on everyone else | 20:15 |
| jeblair | jomara: we all have voices too! | 20:15 |
| sdague | mtreinish is correct | 20:15 |
| jog0 | mtreinish: oh right, its still neat to see though | 20:15 |
| sdague | jeblair: I'd still vote sorting the graphs by failures | 20:16 |
| jeblair | sdague: yes, that (and many other things) would be nice, but i'd like to get this much merged so other people can start hacking/using it. | 20:17 |
| jog0 | jeblair: ++ | 20:18 |
| sdague | jeblair: sure, I'm game | 20:19 |
| jomara | jeblair: it looks like the first thing proposed was , "run on openshift until we have time to setup a proper puppet install w/ postgres on openstack-infra" | 20:19 |
| jomara | then migrate the db and put in a redirect or something | 20:20 |
| jomara | i think matty tried to mess wtih getting it to work w/ mysql but didnt have much luck, its probably not worth haxing up askbott | 20:21 |
| mriedem | dtroyer: sdague: fyi https://review.openstack.org/#/c/49409/ | 20:23 |
| mriedem | that's an attempt at fixing one of the common recheck bugs | 20:23 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add check_success tool https://review.openstack.org/49246 | 20:24 |
| sdague | mriedem: I'm +2, if I don't hear from dtroyer before I leave in 30 mins, I'll +A it | 20:24 |
| mriedem | sdague: thanks | 20:24 |
| sdague | then it should get in overnight | 20:25 |
| *** Ajaeger has quit IRC | 20:33 | |
| *** DennyZhang has quit IRC | 20:34 | |
| jog0 | jeblair sdague: about to merge https://review.openstack.org/#/c/49410/ | 20:34 |
| jog0 | jeblair: want to make sure you think its ready first | 20:35 |
| jog0 | fungi: somethign is funny with the elastic-recheck bot running on infra | 20:36 |
| jog0 | jeblair: ^ | 20:36 |
| jog0 | can we check the logs to see whats going on | 20:36 |
| fungi | jog0: sure | 20:39 |
| jog0 | fungi: a local copy just detected https://review.openstack.org/#/c/46979/2 | 20:39 |
| jog0 | Found bug: https://bugs.launchpad.net/bugs/1232748 | 20:39 |
| uvirtbot | Launchpad bug 1232748 in devstack "[Postgres] DB error: (OperationalError) could not translate host name "localhost" to address" [Undecided,In progress] | 20:39 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add gerrit-client-hooks to our infrastructure. https://review.openstack.org/49432 | 20:40 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Better recommendations for adding sections. https://review.openstack.org/49433 | 20:40 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Better sample jobs. https://review.openstack.org/49434 | 20:40 |
| *** dkranz has joined #openstack-infra | 20:41 | |
| lifeless | IIRC we can do new things w/out downtime, right ? | 20:41 |
| fungi | jog0: last entry in the debug log is 2013-10-02 20:40:55,033 DEBUG gerrit.GerritWatcher: Received data from Gerrit event stream | 20:41 |
| fungi | lifeless: yes | 20:41 |
| lifeless | ok, ^ plox :) | 20:41 |
| lifeless | but I had to guess at the jobs list due to skew between practice and reality | 20:42 |
| lifeless | which I also captured as new docs. | 20:42 |
| *** AlexF has joined #openstack-infra | 20:42 | |
| fungi | jog0: no mention of 46979 in the debug log | 20:42 |
| jog0 | fungi: that may be misleading because gerritlib writes to logs and runs in the background | 20:42 |
| jog0 | can you pastebin me the last 1000 lines or so? | 20:43 |
| fungi | last update to channel was at 19:49 | 20:43 |
| fungi | jog0: sure | 20:43 |
| jog0 | fungi: TBH not sure what I am looking for either | 20:43 |
| *** zul has quit IRC | 20:44 | |
| fungi | jog0: http://paste.openstack.org/show/47867 | 20:44 |
| fungi | jog0: oh, wait, you want debug. just a sec | 20:45 |
| fungi | jog0: http://paste.openstack.org/show/47868 | 20:45 |
| *** SergeyLukjanov has quit IRC | 20:48 | |
| *** pblaho has quit IRC | 20:49 | |
| fungi | jog0: the git clone of elastic-recheck on logstash.o.o is also current with master | 20:49 |
| *** sarob has quit IRC | 20:50 | |
| *** sarob has joined #openstack-infra | 20:51 | |
| jog0 | fungi: hmm no idea whats going on | 20:51 |
| jog0 | fungi: can I get debug logs without gerritlib or irc? | 20:52 |
| fungi | jog0: netstat on both ends seems to think the socket to gerrit is still open | 20:52 |
| jog0 | last 1000 lines | 20:52 |
| fungi | jog0: eh, probably not easily. there's way too much multiline stuff in that log | 20:52 |
| jog0 | fungi: in that case last 10k lines in debug? | 20:52 |
| fungi | coming right up. or i can do a time range if you need | 20:53 |
| jog0 | last two hours? | 20:53 |
| *** alcabrera has quit IRC | 20:53 | |
| fungi | 10k lines is http://paste.openstack.org/show/47870 and goes back a little over 2h | 20:54 |
| fungi | jog0: ^ | 20:54 |
| *** sarob has quit IRC | 20:55 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Cleanup tests https://review.openstack.org/48764 | 20:57 |
| fungi | jog0: filtered out all multiline continuations and gerritlib/irc client entries... http://paste.openstack.org/show/47871 | 20:59 |
| fungi | no idea if that view helps | 20:59 |
| *** alexpilotti has quit IRC | 20:59 | |
| fungi | i suspect tweaks to the logging.conf might be in order to break the debug logs out in the future | 21:00 |
| *** matty_dubs is now known as matty_dubs|gone | 21:00 | |
| *** datsun180b has quit IRC | 21:01 | |
| jog0 | fungi: going to add more debug logs | 21:01 |
| *** freyes has quit IRC | 21:02 | |
| fungi | okay | 21:02 |
| *** alexpilotti has joined #openstack-infra | 21:07 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add more debug logs to classify https://review.openstack.org/49442 | 21:08 |
| jog0 | fungi ^ that should help | 21:08 |
| jog0 | sdague: can I get a quick review on ^ | 21:08 |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Added some options and pep8 compliance https://review.openstack.org/49406 | 21:10 |
| dtroyer | sdague: +A in the localhost bit | 21:11 |
| *** sarob has joined #openstack-infra | 21:12 | |
| *** changbl has joined #openstack-infra | 21:12 | |
| jog0 | fungi: if you can review my patch above we can rekick e-r after that lands | 21:13 |
| fungi | you bet | 21:14 |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Adapting for generic use https://review.openstack.org/49445 | 21:15 |
| mtreinish | jog0: are the logs from the bot posted somewhere? | 21:16 |
| jog0 | mtreinish: fungi was doing it by hand | 21:17 |
| jog0 | see scrollback | 21:17 |
| *** prad has joined #openstack-infra | 21:17 | |
| *** jerryz has joined #openstack-infra | 21:19 | |
| *** dcramer_ has quit IRC | 21:19 | |
| *** sgordon has quit IRC | 21:21 | |
| zaro | fungi: would you be able to tell the purpose for gerrit::remotes module? | 21:22 |
| *** thomasbiege1 has joined #openstack-infra | 21:24 | |
| fungi | zaro: we have some projects for which we are maintaining forks, so that updates branches containing the upstream versions | 21:25 |
| *** thomasbiege1 has quit IRC | 21:26 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add more debug logs to classify https://review.openstack.org/49442 | 21:26 |
| zaro | fungi: thnx. | 21:26 |
| *** mrmartin has quit IRC | 21:27 | |
| *** weshay has quit IRC | 21:28 | |
| fungi | jog0: restarted on top of that new commit now | 21:30 |
| jog0 | fungi: thanks! hopefully that will give us a little more insight | 21:31 |
| fungi | here's hoping | 21:31 |
| *** thomasm has quit IRC | 21:34 | |
| *** danger_fo is now known as danger_fo_away | 21:34 | |
| *** thingee_zzz is now known as thingee | 21:39 | |
| *** pcm_ has quit IRC | 21:45 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/gerritlib: Allow ctrl-Cing gerritlib programs. https://review.openstack.org/49333 | 21:45 |
| openstackgerrit | lifeless proposed a change to openstack-infra/gerritlib: Ignore editor files. https://review.openstack.org/49327 | 21:45 |
| openstackgerrit | lifeless proposed a change to openstack-infra/gerritlib: Simplify GerritWatcher creation. https://review.openstack.org/49326 | 21:45 |
| *** CaptTofu has quit IRC | 21:46 | |
| *** CaptTofu has joined #openstack-infra | 21:47 | |
| *** dkliban has joined #openstack-infra | 21:48 | |
| *** zul has joined #openstack-infra | 21:49 | |
| *** rfolco has quit IRC | 21:49 | |
| *** zul has quit IRC | 21:49 | |
| *** zul has joined #openstack-infra | 21:50 | |
| *** nati_ueno has joined #openstack-infra | 21:53 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Add graph script https://review.openstack.org/49410 | 21:55 |
| *** CaptTofu has quit IRC | 21:57 | |
| *** afazekas_no_irq has quit IRC | 21:57 | |
| *** CaptTofu has joined #openstack-infra | 21:58 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add graph script https://review.openstack.org/49410 | 22:03 |
| openstackgerrit | A change was merged to openstack-infra/config: Add elastic-recheck graphs to status https://review.openstack.org/49417 | 22:06 |
| *** julim has joined #openstack-infra | 22:10 | |
| *** mriedem has quit IRC | 22:15 | |
| openstackgerrit | A change was merged to openstack-infra/config: Publish atom.xml for manuals https://review.openstack.org/47897 | 22:17 |
| *** mrodden has quit IRC | 22:17 | |
| lifeless | jeblair: would you like me to rebase my new project under the doc changes ? | 22:22 |
| jeblair | lifeless: that would be swell, thanks. | 22:22 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module https://review.openstack.org/49448 | 22:23 |
| *** flaper87 is now known as flaper87|afk | 22:24 | |
| jeblair | mordred, zaro: can the two of you get together and talk about https://review.openstack.org/#/c/48058/ and https://review.openstack.org/45329 | 22:25 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Better sample jobs. https://review.openstack.org/49434 | 22:25 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Better recommendations for adding sections. https://review.openstack.org/49433 | 22:25 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add gerrit-client-hooks to our infrastructure. https://review.openstack.org/49432 | 22:25 |
| zaro | mordred: does 48058 mean that 45329 should be abandoned? | 22:28 |
| zul | wtf? why do we have to freaking update the requirements now i thought we were in a freeze? | 22:29 |
| mordred | zaro: nope. you keep yours. I was wrong about mine | 22:30 |
| mordred | zul: aroo? | 22:30 |
| mordred | zul: so - openstack/requirements is frozen - this is aligning the specific projects to that | 22:31 |
| mordred | zul: you should be happy about that, no? | 22:31 |
| mordred | (we did it for you) | 22:31 |
| fungi | zul: which change? specifically? | 22:31 |
| zul | mordred/fungi: https://github.com/openstack/cinder/commit/56d3392e2dd66710a0876a8ba4d567e5d89086fb | 22:31 |
| mordred | zul: also, moving forward, they should drastically cease to diverge | 22:31 |
| mordred | zul: ah - that's a specific bug | 22:32 |
| mordred | zul: Bable 0.9.6 breaks with our current translations, 1.1 doesn't recognize zh_CN - we had to bump to 1.3 to get translation processing that actually worked with our repos | 22:32 |
| zul | mordred: thats a really really big change for us at this close to release | 22:32 |
| mordred | zul: :( | 22:33 |
| openstackgerrit | A change was merged to openstack-infra/publications: Update talk with new information and images. https://review.openstack.org/48151 | 22:33 |
| mordred | zul: lemme find the notes for you on what was broken | 22:33 |
| dstufft | is github showing notes a new thing? o.O | 22:33 |
| mordred | zul: https://review.openstack.org/#/c/48739/ | 22:34 |
| mordred | zul: are the notes on it | 22:34 |
| openstackgerrit | A change was merged to openstack-infra/publications: Add logstash slide. https://review.openstack.org/48152 | 22:34 |
| *** prad has quit IRC | 22:35 | |
| mordred | zul: since you're not installing via pip, you could probably ignore that, looking at it | 22:35 |
| zul | yeah | 22:35 |
| zul | just get to carry another patch whee!! | 22:35 |
| * zul goes get alcohol poising | 22:35 | |
| mordred | zul: no, I mean, you should not need a patch | 22:36 |
| mordred | zul: you're doing SKIP_PIP_INSTALL in your rules file, yeah? | 22:36 |
| openstackgerrit | A change was merged to openstack-infra/publications: Make zuul simulation reflect NNFI. https://review.openstack.org/48153 | 22:36 |
| zul | mordred: SKIP_PIP_INSTALL? | 22:36 |
| mordred | zul: if you do that, then the contents of the requirements files are meaningless to you | 22:36 |
| mordred | zomg | 22:36 |
| mordred | nobody reads the docs I wrote for them | 22:36 |
| mordred | http://docs.openstack.org/developer/pbr/packagers.html | 22:37 |
| mordred | http://docs.openstack.org/developer/pbr/packagers.html#dependencies | 22:37 |
| zul | mordred: sorry we dont ;) | 22:37 |
| zul | mordred: i wasnt aware of them | 22:37 |
| mordred | zul: put SKIP_PIP_INSTALL in your debian/rules files | 22:37 |
| *** oubiwann has quit IRC | 22:37 | |
| mordred | zul: because you do not WANT pkg_resources enforcing depends - you're doing that better with packages | 22:37 |
| zul | mordred: how does that affect the tarball thats already created? | 22:38 |
| mordred | zul: it affects what happens at setup.py install time | 22:38 |
| zul | mordred: ok ill play around with it then | 22:38 |
| mordred | at install time, it will skip both the pip step, and the inject-requirmeents-into-install-requirements step | 22:38 |
| *** zaro0508 has joined #openstack-infra | 22:38 | |
| mordred | so that not only will pip not attempt to even read the requirements.txt file | 22:38 |
| mordred | but also, the installed package will never complain at the python level that the version of a thing is wrong | 22:39 |
| mordred | zul: I wrote that feature for you personally | 22:39 |
| zul | mordred: problem with that if the debian/control is not complete then we might end up with broken packages | 22:39 |
| zul | mordred: thanks i feel honored | 22:39 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Remove usage of d2to1 https://review.openstack.org/48294 | 22:39 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Use python logging isntead of print https://review.openstack.org/48313 | 22:39 |
| mordred | zul: hrm. | 22:39 |
| mordred | zul: so, hrm | 22:40 |
| dstufft | you probably want to still populate install_requires | 22:40 |
| *** rnirmal has quit IRC | 22:40 | |
| dstufft | just not install crap | 22:40 |
| mordred | dstufft: yeah - but with versionless stuf | 22:40 |
| mordred | dstufft: so that whatever the version of the os says, that's right | 22:40 |
| mordred | yeah? | 22:40 |
| fungi | zul: how are these sorts of bugs normally solved in distros? carrying a patch to python-babel backporting the chinese locale compilation fix? we just can't patch the babel on pypi so unfortunately, fix means later lib | 22:40 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Partial sync with OpenStack requirements. https://review.openstack.org/48295 | 22:41 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Wait for groups to be created before getting uuids https://review.openstack.org/48314 | 22:41 |
| dstufft | mordred: meh, I wouldn't bother, adding a patch for requirements.txt in the debian package shouldn't be that hard | 22:41 |
| dstufft | mordred: by forcing a packager to make a patch, it's a signal that maybe the versions aren't compat | 22:42 |
| jeblair | jog0, Alex_Gaynor: what's the status of the issue referenced in https://review.openstack.org/#/c/48352/1 ? | 22:42 |
| mordred | indeed | 22:42 |
| dstufft | besides the "patch the source" stuff in the Os packages certainly has way more real world use than SKIP_PIP_INSTALL :) | 22:43 |
| dstufft | mordred: I would provide the option to not reach out and install stuff though | 22:43 |
| dstufft | though setuptools already has an option for that, so maybe you could just piggyback off that | 22:44 |
| mordred | well, we can't | 22:44 |
| mordred | because we subvert setuptools' installation anyway | 22:44 |
| mordred | because we hate easy_install | 22:44 |
| dstufft | mordred: no actually I think you already do this | 22:44 |
| mordred | I believe we do | 22:44 |
| jog0 | jeblair: as far as I know still the same, pypy breaks | 22:44 |
| mordred | we always use distutils.install (--single-version-externally-managed) which skips setuptools trying to do anything | 22:45 |
| dstufft | --single-version-externally-managed is the cannonical way for setuptools to disable dependency installation, so any non openstack project I believe Os packages are likely to already be using them | 22:45 |
| mordred | yup. we enable --single-version-externally-managed at all times | 22:45 |
| dstufft | so I'd assume (maybe i'm wrong?) that they are already using that flag | 22:45 |
| mordred | well, they may or may not be - it's irrelvant | 22:45 |
| zul | fungi: usually the patch is isolated and then its backported | 22:45 |
| mordred | WE do a pip install ourselves in pbr | 22:46 |
| openstackgerrit | A change was merged to openstack-infra/gear: Fix geard logging config https://review.openstack.org/48334 | 22:46 |
| mordred | which is what SKIP_PIP_INSTALL skips | 22:46 |
| dstufft | mordred: but wether it's pip or setuptools doing the install is irrelevant | 22:46 |
| mordred | zul, dstufft: I was wrong, btw, the install_requires still gets populated | 22:46 |
| mordred | dstufft: it's very relevant for us | 22:46 |
| mordred | but not for this | 22:46 |
| dstufft | mordred: I mean for a packager :) | 22:46 |
| mordred | dstufft: yes. the packagers want it to not happen :)_ | 22:46 |
| dstufft | they either want the install to install deps, or they don't want it | 22:46 |
| *** thedodd has quit IRC | 22:47 | |
| dstufft | in what case would that want easy_install to install dependencies but not pip | 22:47 |
| dstufft | they* | 22:47 |
| *** anteaya has quit IRC | 22:47 | |
| mordred | yup. so they need to set SKIP_PIP_INSTALL - and honestly, we should probably get that added to dh_python | 22:47 |
| mordred | dstufft: oh - you're saying that we should honor --single-version-externally-managed as way to switch the SKIP_PIP_INSTALL logic? | 22:47 |
| fungi | zul: makes sense. but with most of the third-party modules we use not providing stable backport versions for bugs we encounter, the alternative (also not without merit) is to declare that close to a release we'll refuse to fix any bugs which require newer versions of dependencies | 22:47 |
| dstufft | mordred: yes | 22:48 |
| mordred | dstufft: interesting. good idea. I will see if I can do it :) | 22:48 |
| mordred | (setuptools. arg parsgin. evil. crazy. AAAAGGGGHHHH.) | 22:48 |
| dstufft | mordred: at least that would make you act more like a normal python package, just your installs happen with pip instead of setuptools, but unless someone wants to install dependencies specifically with setuptools I don't see a need for a special command to specifically disable pip | 22:49 |
| zul | fungi: right i know we need to find a balance but the way that dpkg/pydist/setuptools drives me batty/causes pain | 22:49 |
| fungi | zul: i think it just requires more of a dedication on the upstream side to accept that there are certain bugs which are better left in the release if doing otherwise means more dependency churn for packagers | 22:49 |
| mordred | dstufft: I guess the thing is is that --single-version-externally-managed as a mechanism to disable internet depends seems like relying on an impl hack | 22:49 |
| mordred | dstufft: --single-version-externally-managed is suppoed to turn off egg installs | 22:50 |
| mordred | as a result, it does it by punting to distutils | 22:50 |
| mordred | which does not have internet downloading of depends | 22:50 |
| *** esker has quit IRC | 22:50 | |
| zul | fungi: sure but you are dealing with different distros with different intests different release schedules | 22:50 |
| mordred | so using that flag as a flag to mean "please skip downloading things from the internet" seem VERY FAR from being something that a sane person would be expected to know | 22:50 |
| mordred | even though it does have that side effect | 22:50 |
| dstufft | mordred: python setup.py install --help says -> | 22:51 |
| dstufft | --single-version-externally-managed used by system package builders to | 22:51 |
| dstufft | create 'flat' eggs | 22:51 |
| *** thingee is now known as thingee_zzz | 22:51 | |
| dstufft | so it's called out in the help as the thing system package builders should use | 22:51 |
| dstufft | I mean it's name is terrible | 22:51 |
| fungi | zul: getting everyone on board with "no new features requiring new(er versions of) dependencies allowed close to release time" was fairly easy. the "no fixing bugs" proposition will be a harder sell | 22:51 |
| mordred | yes. it says nothing about "disables downloading from the internet" | 22:51 |
| mordred | even though it does that | 22:52 |
| lifeless | dstufft: the help doesn't imply the thing we care about... | 22:52 |
| zul | bbl need to put people to bed | 22:52 |
| mordred | SKIP_PIP_INSTALL is at least clear as to _exactly_ what it actually does | 22:52 |
| mordred | (again, not arguing that I shouldn't also respond to it for ease of use ...) | 22:52 |
| *** alexpilotti has quit IRC | 22:52 | |
| dstufft | mordred: lifeless to make matters worse, python setup.py develop as a -N // --no-deps | 22:53 |
| dstufft | has a* | 22:53 |
| mordred | dstufft: yah. I tried to use that first | 22:53 |
| mordred | until I discovered it was only for develop | 22:53 |
| mordred | jog0: https://review.openstack.org/#/c/48352/ | 22:53 |
| dstufft | mordred: I mean I don't care if you keep the SKIP_PIP_INSTALL thing, but I think that it won't be that useful because anyone who doesn't want to install things probably doesn't want to install things for any python package and is already using --single-version-externally-managed | 22:54 |
| mordred | dstufft: I think you'd be surpised | 22:54 |
| dstufft | maybe i'm wrong and OS packagers are just downloading crap from the internet | 22:54 |
| mordred | they all complain at me | 22:54 |
| *** zaro0508 has quit IRC | 22:54 | |
| mordred | BUT - possibly it's a knock on and I'm only noticing it because I'm not honoring single-version-externally-managed | 22:55 |
| dstufft | I think you are aren't you? | 22:55 |
| jog0 | mordred: what about https://review.openstack.org/#/c/48352/ ? | 22:55 |
| dstufft | now that I think about it | 22:55 |
| dstufft | else you'd do some wierd recursive shit from pip? | 22:55 |
| mordred | I mean, the flag has no effect on our packages | 22:55 |
| fungi | some prefer to build packages in a chroot with no internet access so they can watch if there are failures related to missing dependencies, since pip install will try to retrieve them | 22:55 |
| mordred | we _always_ don't make eggs, and we _always_ use pip to processes dependencies | 22:56 |
| dstufft | mordred: right, but when you ``pip install foo`` you don't disable pbr's pip stuff then? | 22:56 |
| mordred | single-version-externally-managed has zero impact on anything | 22:56 |
| dstufft | where foo is a pbr thing | 22:56 |
| mordred | nope | 22:56 |
| dstufft | wat | 22:56 |
| mordred | it works | 22:56 |
| dstufft | don't do that | 22:56 |
| mordred | becuse the calling pip does the egg_info | 22:56 |
| mordred | and installs the list | 22:56 |
| dstufft | it still calls python setup.py install | 22:57 |
| dstufft | and uses --single-version-externally-managed to keep it from installing stuff | 22:57 |
| dstufft | (among other things) | 22:57 |
| mordred | HAHAHAHAHAHAHAHAHAAHHAAH | 22:57 |
| mordred | ok | 22:57 |
| mordred | seriously | 22:57 |
| mordred | I'll do the flag | 22:57 |
| mordred | but please tell me you realize that interface is ludicrous and non-documented and magic | 22:57 |
| mordred | and there is NO WAY I could be expected to know that | 22:57 |
| *** fbo is now known as fbo_away | 22:58 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add initial files for tripleo in the gate. https://review.openstack.org/49454 | 22:58 |
| pleia2 | I get a gold star \o/ | 22:58 |
| pleia2 | (it probably needs work) | 22:58 |
| lifeless | \o/ | 22:58 |
| fungi | pleia2: the star is foil paper with water-activated adhesive backing. enjoy | 22:59 |
| mordred | jog0: a) why is it failing pypy, b) isn't the point of the pypy gate to prevent things from breaking on pypy? | 22:59 |
| pleia2 | fungi: goodie! | 22:59 |
| dstufft | mordred: everything about setuptools is fucking terrible | 22:59 |
| mordred | dstufft: so terrible | 22:59 |
| dstufft | I don't expect anyone to understand how it works | 22:59 |
| dstufft | hell I don't fully understand how it works | 23:00 |
| mordred | dstufft: so seriously? pip --single-version-externally-managed the things that it installs via depends? | 23:00 |
| dstufft | and I spend a not insignicant amount of time working on it | 23:00 |
| dstufft | mordred: yes, sec I'll show you the code | 23:00 |
| mordred | wow | 23:00 |
| * mordred loves that as much as he thinks he understands about the weird back corners of things, he's always able to learn something new | 23:00 | |
| dstufft | https://github.com/pypa/pip/blob/develop/pip/req.py#L611 | 23:00 |
| *** anteaya has joined #openstack-infra | 23:01 | |
| jog0 | mordred: why don't know and Alex_Gaynor didn't either | 23:02 |
| jog0 | b) yes, but if we don't know the issue and can't fix it soon (when all RCs are out) don't see why we should hold this up for a possible pypy bug | 23:02 |
| dstufft | mordred: https://github.com/openstack-dev/pbr/blob/master/pbr/packaging.py#L357-L358 ? | 23:03 |
| *** sarob_ has joined #openstack-infra | 23:04 | |
| dstufft | mordred: a little scatter brained today, I'm sick :( | 23:04 |
| *** sarob_ has quit IRC | 23:05 | |
| *** sarob_ has joined #openstack-infra | 23:05 | |
| *** sarob has quit IRC | 23:06 | |
| *** dkliban has quit IRC | 23:08 | |
| *** sarob_ has quit IRC | 23:08 | |
| *** AlexF has quit IRC | 23:14 | |
| mordred | jog0: ok. | 23:15 |
| mordred | dstufft: yuck! sick seems to be going around this week | 23:15 |
| clarkb | is elastic-recheck bot still derping? I am finally home, eating lunch then will push in gearman worker changes | 23:16 |
| jog0 | fungi: I think the issue is gerritlib is dropping things | 23:19 |
| jog0 | I am seeing patches pop up on the bot that I don't see | 23:19 |
| *** boris-42 has quit IRC | 23:19 | |
| jog0 | hmm actually I see the last patch the bot detected | 23:20 |
| jog0 | https://review.openstack.org/#/c/49435/2 | 23:20 |
| *** mdenny has quit IRC | 23:20 | |
| jog0 | fungi: can I get the logs for everything after that | 23:20 |
| jog0 | mtreinish: ^ | 23:22 |
| *** rcleere has quit IRC | 23:22 | |
| *** julim has quit IRC | 23:23 | |
| *** dcramer_ has joined #openstack-infra | 23:30 | |
| *** rfolco has joined #openstack-infra | 23:30 | |
| *** giulivo has quit IRC | 23:31 | |
| *** jergerber has quit IRC | 23:31 | |
| anteaya | clarkb: glad you are home safe | 23:34 |
| *** Ryan_Lane has quit IRC | 23:44 | |
| *** gyee has quit IRC | 23:46 | |
| jog0 | clarkb: it is its workign locally for me (non-bot mode) | 23:46 |
| jog0 | clarkb: also status.openstack.org/elastic-recheck/ | 23:48 |
| jog0 | jeblair did that one | 23:48 |
| clarkb | jog0: nice | 23:50 |
| clarkb | jog0: the irc bot is not reporting on stuff right? I am about to pull out the laptop and start poking at it | 23:50 |
| jog0 | clarkb: correct | 23:50 |
| *** rfolco has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!