| *** saschpe has quit IRC | 00:00 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/zuul: Fix jshint linting errors https://review.openstack.org/82308 | 00:01 |
|---|---|---|
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/zuul: Use javascript toolchain https://review.openstack.org/82307 | 00:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Stop logging obsolete channel #savanna https://review.openstack.org/82587 | 00:01 |
| mordred | jhesketh_: ^^ rebased on your stuff there | 00:02 |
| *** Hefeweizen has joined #openstack-infra | 00:02 | |
| jhesketh | thanks mordred | 00:02 |
| mordred | jhesketh_: I'm guessing I should probably add some docs on how to work with it or something | 00:03 |
| mordred | jhesketh: for now - https://github.com/emonty/zuul-status/blob/master/README.md | 00:05 |
| mordred | if you want to poke at it | 00:05 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add check-rally-install-* template and jobs https://review.openstack.org/82617 | 00:05 |
| mordred | jhesketh: krotscheck can also answer all of your questions | 00:05 |
| *** saschpe has joined #openstack-infra | 00:05 | |
| krotscheck | …. did you just throw something over the wall at me? | 00:06 |
| *** rhsu1 has quit IRC | 00:06 | |
| jhesketh | mordred: oh cool, you had actually started reworking the status page already? | 00:06 |
| mordred | jhesketh: yes | 00:08 |
| mordred | krotscheck: and yes | 00:08 |
| mordred | jhesketh: I reworked it as a POC to see what the new fancy javascript tools from krotscheck would look like on a codebase we grokked | 00:08 |
| mordred | we put it on hold for a bit to also explore what single-repo would look like | 00:09 |
| jhesketh | ah neat | 00:09 |
| *** melwitt has quit IRC | 00:09 | |
| jhesketh | yeah separating out the UI from zuul makes sense | 00:09 |
| jhesketh | similarly I'd like to see zuuls webapp removed and use RPC to get the details it needs | 00:10 |
| mordred | well - I think I could argue both ways | 00:10 |
| mordred | both that having it separate is good, and also having it in the same repo is good | 00:10 |
| mordred | but - krotscheck - I was meaning to ask you | 00:10 |
| mordred | if the zuul project wanted to havea default non-opensatck branded status page (either in the same or in a different repo) | 00:10 |
| mordred | and then the openstack installation of zuul wanted to add, ou know, headers and footers and css | 00:11 |
| mordred | krotscheck: should I assume that there is a well understood and sane way to go about doing that? | 00:11 |
| mordred | (and when I say status page, I mean the status webapp) | 00:12 |
| krotscheck | mordred: There are several approaches we can take for that, yes. | 00:12 |
| mordred | woot | 00:12 |
| * mordred figured that this was not a unique-to-use problem | 00:12 | |
| krotscheck | mordred: If you can move the branding into stylesheets, it’s as simple as allowing someone to include a custom CSS file. | 00:12 |
| krotscheck | mordred: Fair warning, legacy code can often pose tricky bits. | 00:13 |
| mordred | krotscheck: I believe there is also a branding header section though | 00:13 |
| mordred | krotscheck: which was making me think of (bear with me while I say all the wrong words...) | 00:13 |
| mordred | krotscheck: making zuul-status into an angular app | 00:13 |
| mordred | then making an openstack-zuul-status app that dependeded on that app and had its own index.html + header + footer sections | 00:14 |
| * mordred cringes at how much he probably got wrong there | 00:14 | |
| krotscheck | mordred: Nope, that’s exactly right | 00:14 |
| krotscheck | mordred: In a nutshell, bower/npm will allow you to set up a git repostiory as a dependency. | 00:16 |
| krotscheck | mordred: From there it’s a fairly simple step to create a whitelable template that imports the default app and overrides the styles, templates, etc. | 00:16 |
| *** rediskin has left #openstack-infra | 00:17 | |
| jeblair | mordred: so, i'm not opposed to, well, anything, but i am wondering what adding angular into the mix gets us? | 00:18 |
| krotscheck | mordred: The next-more-complex step would be to provide a compiled set of modules which are published independently, and a whitelabel provider can configure features on their app simply by declaring dependencies. | 00:18 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix typo in ircbot's default matrix-notifier. https://review.openstack.org/79328 | 00:18 |
| jeblair | mordred: i mean, are you planning on using jquery and angular? | 00:18 |
| *** openstack has joined #openstack-infra | 00:20 | |
| jeblair | mordred: that's so simple, why do we need another js framework to do that? | 00:20 |
| mordred | which is getting us to a published "whitelabel" version of zuul-status that openstack-zuul-status can consume as a library | 00:20 |
| mordred | jeblair: we may not need angular itself - it may be that it's bower/npm that do the heavy lifting here | 00:20 |
| mordred | jeblair: I'm mainly saying all of the words I know to see which ones hit and which ones fall apart | 00:21 |
| sdague | honestly, the branding should just all be pulled out into a separate style sheet I would think | 00:21 |
| sdague | adding js for it seems super overkill | 00:21 |
| mordred | there is a header and footer menu bar | 00:21 |
| jeblair | mordred: okay, os it's the "consume as library" bit that you're getting at | 00:21 |
| mordred | yes | 00:21 |
| mordred | sdague: it's not adding js - there is a js framework already - this is about build tooling and repeatability | 00:22 |
| sdague | sure, just normally people handing branding on their side basically understand css | 00:23 |
| jeblair | mordred: so the brain thought that makes sense in my thinking ball is that jheskeths work is heading toward feature parity, then we need to make sure the css that is used is sufficiently neutral that we can apply our own (and, in particular, not apply bootstrap) | 00:23 |
| jeblair | mordred: and then we package that thing up and install it and apply our css | 00:23 |
| jeblair | mordred: and wmf installs it, and probably applies little or none of their own css (and uses bootstrap) because they like that | 00:24 |
| mordred | jeblair: yah | 00:24 |
| mordred | sdague: yah - agree - I believe that local css should be able to be applied in exactly that way - sorry, didn't mean to imply that local css changes would take javascript | 00:24 |
| jhesketh | so the other thing with feature parity is infra's page has graphs that are separate from zuul | 00:25 |
| jeblair | mordred: and then when we decide that we really want sdague to be able to log in and promote changes with a mouse click, we think more seriously about making it a real webapp, and maybe using angular...? | 00:25 |
| jhesketh | how to hook those in will also need some consideration | 00:25 |
| sdague | hmmm... that would be a feature :) | 00:25 |
| sdague | then I wouldn't have to disturb fungi before his coffee in the morning | 00:25 |
| jhesketh | jeblair: You wouldn't want to run bootstrap on infra even with your own stylesheet? | 00:25 |
| jeblair | jhesketh: i think we can consider the graphs at the bottom "footer material" in this case... | 00:25 |
| mordred | jhesketh: yah - that would be why I think taking advantage of build framework would be nice | 00:26 |
| jhesketh | well pipeline sparks | 00:26 |
| clarkb | sdague: you would run your own graph set? | 00:26 |
| *** matsuhashi has joined #openstack-infra | 00:26 | |
| jeblair | jhesketh: even though they are a bit complex (they also need the visibility and timeout functions, but i'm okay with that duplication) | 00:26 |
| sdague | clarkb: no, the thing jeblair said | 00:26 |
| *** flaper87 is now known as flaper87|afk | 00:26 | |
| sdague | promote zuul changes from zuul web ui | 00:26 |
| clarkb | sdague: oh interesting, I missed that | 00:26 |
| mordred | jeblair, jhesketh: http://git.openstack.org/cgit/openstack-infra/storyboard-webclient/tree/src/index.html | 00:26 |
| clarkb | I am about to afk now | 00:26 |
| sdague | which is crazy future talk | 00:27 |
| * jhesketh looked at storyboard-webclient last night... much impressive | 00:27 | |
| clarkb | jeblair: any chance you can review the crm114 change tomorrow morning? I can babysit that change and see if it helps the timeouts | 00:27 |
| sdague | but still would be awesome | 00:27 |
| mordred | the main section is a bit more complex than jeblair's etherpad | 00:27 |
| mordred | but if you look at: | 00:27 |
| jeblair | jhesketh: but yeah, i think your suggestion of having it grok sparks and making that an flag seems like a good idea | 00:27 |
| mordred | http://git.openstack.org/cgit/openstack-infra/storyboard-webclient/tree/src/index.html#n49 and http://git.openstack.org/cgit/openstack-infra/storyboard-webclient/tree/src/index.html#n77 | 00:27 |
| mordred | those are build-time substituted header and footer sections | 00:28 |
| jhesketh | jeblair: yeah, I think that's also inline with how zuul optionally reports stats anyway | 00:28 |
| jeblair | jhesketh: ++ | 00:28 |
| mordred | which makes me think that if we made openstack-zuul-status have a local header/footer (including graphs) | 00:28 |
| mordred | then we could make the normal one not have those things - or have different "normal zuul" ones | 00:28 |
| mordred | and it would be quite simple and make sense and fit into existing tools | 00:28 |
| jeblair | mordred, krotscheck: so does "header ng-include" come from angular? | 00:29 |
| krotscheck | jeblair: That’s the ng-include directive, and it’s actually resolved at runtime. | 00:30 |
| jeblair | krotscheck: by what? | 00:30 |
| mordred | oh - it is? | 00:30 |
| mordred | silly me | 00:30 |
| krotscheck | angular. | 00:30 |
| mordred | krotscheck: the bower sections are build time, yeah? | 00:30 |
| krotscheck | There’s an optimization in there that wraps all the templates into a single file for easier downloading/caching. | 00:30 |
| jeblair | jhesketh: yeah, i'm not sure what boostrap would get us? | 00:31 |
| *** bknudson has joined #openstack-infra | 00:31 | |
| jeblair | jhesketh: i mean, i don't want the status page to look like bootstrap, i want it to look like www.openstack.org... | 00:31 |
| jhesketh | jeblair: well the grid layout stuff would need reproducing if you weren't using the framework | 00:32 |
| krotscheck | mordred: anything that’s got a <!— build: directive is compiletime | 00:32 |
| mordred | krotscheck: nod | 00:32 |
| krotscheck | Everything else is, more or less, concatenation. | 00:32 |
| *** CaptTofu has joined #openstack-infra | 00:34 | |
| mordred | zaro: is it safe to run puppet on jenkins-dev.o.o ? | 00:35 |
| jeblair | jhesketh: ok; worth thinking about | 00:35 |
| jhesketh | jeblair: we should be able to use the stylesheet from storyboard + some extra css and get it pretty close to how it currently looks without the need to redo the grid/layout or progress bars etc | 00:35 |
| jhesketh | but it's something if it's going to be an issue we should discuss now so that we don't end up with two different implementations again between zuul and infra | 00:36 |
| mordred | jeblair: I'll see if I can cook up a POC at some point to show mechanism of whitelabel and branding so we can evaluate the mechanism before we spend much effort on it in context of zuul status | 00:36 |
| jeblair | jhesketh: basically where i'm coming from is that i think it should look more or less how it looks now (and not like bootstrap), so whether it's easier to go with bootstrap and layer on top to unbootstrapify that, or go the other way i don't have a strong feeling. | 00:37 |
| *** salv-orlando has joined #openstack-infra | 00:37 | |
| krotscheck | jeblair, jhesketh: We’re already overriding the LESS variables in the storyboard bootstrap to make it less shitty, publishing a common, OpenStackified bootstrap would create a decent whitelabel baseline for all JS projects. | 00:37 |
| mordred | and would make it nice to re-consume across our things | 00:38 |
| * mordred back on the library thing | 00:38 | |
| krotscheck | Blah, less tech talk. Create an OpenStack version of Bootstrap. Publish that as the “Hey, here’s how you whitelabel openstack!" | 00:38 |
| jhesketh | krotscheck: yep, that sounds good.. as a tangent, does angularjs expect you to provide your own css scaffolding (eg bootstrap)? | 00:38 |
| *** mgagne has quit IRC | 00:39 | |
| jhesketh | jeblair: sure, would you care if it was slightly different in terms of padding/spacing but kept the openstack colour theme | 00:39 |
| jhesketh | eg imagine what the zuul version looks like with my changes but with openstack colours + header/footer | 00:40 |
| mordred | jeblair, clarkb: in case zaro is home and not working this instant - do you know if it's safe to run puppet on jenkins-dev? I wanted to investigate it not being in puppetboard - but don't want to screw zaro | 00:40 |
| krotscheck | zaro is on his way home, he just left. | 00:41 |
| jeblair | jhesketh: i think minor changes are okay, it's mostly the big picture stuff -- boxy versus roundy, etc... | 00:41 |
| jhesketh | jeblair: so you'd like to keep the boxy look? | 00:41 |
| jeblair | jhesketh: 1 sec while i load your latest change | 00:41 |
| jhesketh | sure | 00:42 |
| jhesketh | fyi, I think it's all achievable without too much css, but perhaps somebody who is actually good at this stuff should have a look at how we'd reconcile the differences between zuul's version and infra's before we spend too much time on it | 00:42 |
| jhesketh | the whole point here is to reduce duplication etc | 00:42 |
| *** bhuvan has quit IRC | 00:44 | |
| jeblair | jhesketh: that's the thing -- if it's all in css it shouldn't matter | 00:44 |
| jeblair | jhesketh: my original version was pretty classic bare-bones semantic css | 00:44 |
| jeblair | jhesketh: things like class=pipeline, class=change etc... | 00:44 |
| *** EntropyWorks has quit IRC | 00:45 | |
| *** jcooley has quit IRC | 00:45 | |
| jhesketh | yep, that should still be the case (I hope) | 00:45 |
| *** wchrisj has quit IRC | 00:45 | |
| *** dcramer_ has joined #openstack-infra | 00:46 | |
| jeblair | jhesketh: so back to your question about spacing and other changes | 00:46 |
| jeblair | jhesketh: the openstack version fits 3 nova gate changes onto the screen, but the bootstrap version only 1.5 | 00:47 |
| jeblair | so it's a significant change in information density | 00:47 |
| *** adalbas has quit IRC | 00:47 | |
| jeblair | so i don't want to say that the openstack version is perfect; it especially has legibility problems with longer job names, and in those cases, extra leading or even visual cues like leaders might help | 00:48 |
| jhesketh | yes, the vertical sizing has been increased significantly | 00:48 |
| jeblair | but the bootstrap version i think goes a bit too far on both | 00:48 |
| jeblair | jhesketh: i think the best course would be to try to get the openstack overlay on the updated app.js looking as close to what we have now as possible first | 00:52 |
| jeblair | jhesketh: and then evaluate changes to it after that on their own merit | 00:53 |
| jeblair | jhesketh: because the current version looks the way it does more or less on purpose | 00:53 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1295876 https://review.openstack.org/82245 | 00:53 |
| uvirtbot | Launchpad bug 1295876 in nova "libvirtError: internal error unable to add domain xxx to cgroup: No space left on device" [Medium,Confirmed] https://launchpad.net/bugs/1295876 | 00:53 |
| *** yamahata has joined #openstack-infra | 00:55 | |
| jeblair | jhesketh: btw, you might have seen my comment, but just to reiterate how excited i am, i really love the overall graph -- it's a brilliant bit of visualization! :) | 00:55 |
| jhesketh | :-) | 00:55 |
| *** yamahata_ has quit IRC | 00:56 | |
| *** tianst20 has quit IRC | 00:56 | |
| *** wchrisj has joined #openstack-infra | 00:56 | |
| jhesketh | jeblair: so I think the last thing I need to do to get feature parity is to add in the nnfi trees | 00:56 |
| *** yamahata_ has joined #openstack-infra | 00:56 | |
| jhesketh | but my css skills are limited here | 00:57 |
| jeblair | jhesketh: yeah, i spent a lot of time trying to make a nice pure-css way of doing that | 00:57 |
| jeblair | jhesketh: and eventually decided that, in fact, the right answer was tables | 00:57 |
| jeblair | that was not an easy decision | 00:57 |
| *** weshay has quit IRC | 00:59 | |
| jeblair | jhesketh: i have to run make dinner now; thanks, and ttyl | 01:00 |
| jhesketh | no worries, enjoy! | 01:01 |
| notmyname | jeblair: clarkb: just looking back through some old IRC logs...why is the -infra team using swift3? | 01:03 |
| *** jepoy has joined #openstack-infra | 01:04 | |
| mordred | notmyname: I didn't know we were? | 01:05 |
| notmyname | hmm..maybe that was only in relation to putting it on stackforge https://review.openstack.org/#/c/81991/ (also, why that?) | 01:06 |
| *** rhsu has joined #openstack-infra | 01:07 | |
| *** krtaylor has quit IRC | 01:07 | |
| mordred | notmyname: why put it on stackforge? dunno - I'm not a swift3 hacker - but I hear some people really like our systems :) | 01:08 |
| notmyname | I was seeing it temporally associated with -infra logging to swift, so I may have been confused there | 01:10 |
| *** thuc has quit IRC | 01:10 | |
| mordred | ah - no, I doubt we'd use swift3 for that - we are interested in using swift for that - but I dont' think that project is using or intending to use swift3 in any way | 01:10 |
| *** thuc has joined #openstack-infra | 01:10 | |
| morganfainberg | clarkb, quick glance, keystone logs down from ~5M to ~1.7M w/ that patch that just entered gate. | 01:11 |
| *** krtaylor has joined #openstack-infra | 01:11 | |
| notmyname | seems that it was just the IRC log messages getting me confused (although a stackforge project with no-op gate jobs and a -core team all from the same company seems weird to me) | 01:11 |
| morganfainberg | clarkb, py27 that is. | 01:11 |
| mordred | morganfainberg: w00t! | 01:11 |
| morganfainberg | mordred, :) | 01:11 |
| morganfainberg | mordred, and we have full parallel testing capability in keystone now, just need to hammer out a couple transient possible bugs before enabling more than 1 worker | 01:12 |
| mordred | morganfainberg: dood. you're killing it! | 01:14 |
| *** UtahDave has quit IRC | 01:14 | |
| dolphm | mordred: ++! | 01:14 |
| morganfainberg | mordred, so much test cleanup! and we landed it before the next big rush of features :) | 01:15 |
| *** thuc has quit IRC | 01:15 | |
| * mordred hands dolphm and morganfainberg a very large cookie | 01:15 | |
| morganfainberg | yay! | 01:16 |
| *** yjiang has joined #openstack-infra | 01:17 | |
| dolphm | morganfainberg and dstanek deserve all the cookies there! | 01:17 |
| dolphm | and bknudson! | 01:17 |
| dstanek | if there's cookies i'll be there | 01:18 |
| morganfainberg | don't leave bknudson out! | 01:18 |
| fungi | notmyname: from the earlier discussion i got the impression that something (someone's tempest tests maybe?) used swift3 in their requirements/test-requirements and it was suggested on the -infra ml that its devs might be amenable to moving development of it into stackforge and gating its changes on whether or not they broke openstack test uses | 01:20 |
| *** krtaylor has quit IRC | 01:20 | |
| * fungi checks ml archive for proper reference | 01:20 | |
| *** sabari has quit IRC | 01:21 | |
| fungi | notmyname: devstack apparently referenced/references a git url for it... http://lists.openstack.org/pipermail/openstack-infra/2014-March/001041.html | 01:22 |
| notmyname | fungi: thanks | 01:22 |
| *** mwagner_lap has joined #openstack-infra | 01:23 | |
| *** krtaylor has joined #openstack-infra | 01:23 | |
| *** rwsu has quit IRC | 01:26 | |
| openstackgerrit | Nathan Kinder proposed a change to openstack-infra/config: Add the openstack-security-notes project https://review.openstack.org/73157 | 01:29 |
| *** SumitNaiksatam has joined #openstack-infra | 01:29 | |
| *** nosnos has joined #openstack-infra | 01:29 | |
| *** matsuhashi has quit IRC | 01:30 | |
| *** matsuhashi has joined #openstack-infra | 01:30 | |
| *** wchrisj has quit IRC | 01:31 | |
| *** zhiyan_ is now known as zhiyan | 01:32 | |
| *** mbacchi has quit IRC | 01:35 | |
| *** wchrisj has joined #openstack-infra | 01:39 | |
| *** rhsu has quit IRC | 01:39 | |
| *** rhsu1 has joined #openstack-infra | 01:39 | |
| *** wenlock has joined #openstack-infra | 01:41 | |
| *** emagana has quit IRC | 01:41 | |
| *** jasondotstar has joined #openstack-infra | 01:41 | |
| *** rwsu has joined #openstack-infra | 01:42 | |
| *** wchrisj has quit IRC | 01:44 | |
| *** SumitNaiksatam has quit IRC | 01:45 | |
| *** Ryan_Lane has quit IRC | 01:46 | |
| *** sandywalsh has joined #openstack-infra | 01:46 | |
| *** khyati has quit IRC | 01:47 | |
| *** yamahata_ has quit IRC | 01:48 | |
| *** wchrisj has joined #openstack-infra | 01:49 | |
| *** CaptTofu has quit IRC | 01:49 | |
| *** yamahata_ has joined #openstack-infra | 01:50 | |
| *** ArxCruz_ has quit IRC | 01:51 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:52 | |
| *** wchrisj_ has joined #openstack-infra | 01:53 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Task status updates https://review.openstack.org/82690 | 01:53 |
| *** wchrisj has quit IRC | 01:53 | |
| *** yonglihe_ has quit IRC | 01:56 | |
| *** yjiang has quit IRC | 01:56 | |
| *** yjiang|2 has joined #openstack-infra | 01:56 | |
| *** ogelbukh has quit IRC | 01:57 | |
| *** katyafervent has quit IRC | 01:57 | |
| *** katyafervent has joined #openstack-infra | 01:57 | |
| *** jamespage has quit IRC | 01:59 | |
| *** yonglihe_ has joined #openstack-infra | 01:59 | |
| *** wenlock_ has joined #openstack-infra | 01:59 | |
| *** wchrisj_ has quit IRC | 02:00 | |
| *** jamespage has joined #openstack-infra | 02:00 | |
| *** jamespage has joined #openstack-infra | 02:00 | |
| *** ogelbukh has joined #openstack-infra | 02:00 | |
| *** fifieldt_ has joined #openstack-infra | 02:00 | |
| *** bknudson has quit IRC | 02:16 | |
| lifeless | I've a pypi-mirror failure | 02:17 |
| lifeless | http://pastebin.com/8dZAncTD | 02:17 |
| lifeless | how do you add --allow-external pysendfile to the pip invocation it uses ? | 02:18 |
| lifeless | clarkb: ^ | 02:18 |
| *** dcramer_ has quit IRC | 02:19 | |
| lifeless | is it perhaps that the copy on pypi is ancient? Yes, yes it is. | 02:21 |
| lifeless | RELEASES PEOPLE! | 02:21 |
| *** thuc has joined #openstack-infra | 02:21 | |
| *** wenlock__ has joined #openstack-infra | 02:22 | |
| *** CaptTofu has joined #openstack-infra | 02:22 | |
| *** wenlock__ has joined #openstack-infra | 02:23 | |
| *** wenlock_ has quit IRC | 02:24 | |
| *** saschpe has quit IRC | 02:26 | |
| *** thuc has quit IRC | 02:26 | |
| *** amcrn has quit IRC | 02:26 | |
| *** wenlock_ has joined #openstack-infra | 02:28 | |
| *** mgagne has joined #openstack-infra | 02:28 | |
| *** saschpe has joined #openstack-infra | 02:28 | |
| *** HenryG has joined #openstack-infra | 02:29 | |
| openstackgerrit | Benjamin Staffin proposed a change to openstack-infra/jenkins-job-builder: Add support for Slave Utilization plugin https://review.openstack.org/82693 | 02:29 |
| *** jasondotstar has quit IRC | 02:30 | |
| *** wenlock__ has quit IRC | 02:30 | |
| openstackgerrit | Benjamin Staffin proposed a change to openstack-infra/jenkins-job-builder: Clean up a few errant tabs in setup.cfg https://review.openstack.org/82694 | 02:32 |
| *** mestery has joined #openstack-infra | 02:34 | |
| *** jepoy has quit IRC | 02:36 | |
| *** jepoy has joined #openstack-infra | 02:36 | |
| *** Sukhdev has joined #openstack-infra | 02:40 | |
| *** jepoy has quit IRC | 02:40 | |
| *** rfolco has quit IRC | 02:41 | |
| *** mestery has quit IRC | 02:46 | |
| *** thuc has joined #openstack-infra | 02:46 | |
| *** mestery has joined #openstack-infra | 02:46 | |
| *** mriedem has quit IRC | 02:48 | |
| *** salv-orlando_ has joined #openstack-infra | 02:49 | |
| *** salv-orlando has quit IRC | 02:50 | |
| *** salv-orlando_ is now known as salv-orlando | 02:50 | |
| *** thuc_ has joined #openstack-infra | 02:51 | |
| *** jepoy has joined #openstack-infra | 02:52 | |
| *** thuc has quit IRC | 02:54 | |
| *** wchrisj has joined #openstack-infra | 03:02 | |
| *** wchrisj has quit IRC | 03:06 | |
| *** wchrisj has joined #openstack-infra | 03:07 | |
| *** yjiang|2 has quit IRC | 03:07 | |
| *** CaptTofu has quit IRC | 03:09 | |
| *** zhiyan is now known as zhiyan_ | 03:10 | |
| *** zhiyan_ is now known as zhiyan | 03:11 | |
| *** matsuhashi has quit IRC | 03:12 | |
| *** wchrisj has quit IRC | 03:14 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/devstack-gate: Sets DEFAULT_INSTANCE_TYPE in localrc for Ironic https://review.openstack.org/82703 | 03:15 |
| *** wchrisj has joined #openstack-infra | 03:15 | |
| *** david-lyle has joined #openstack-infra | 03:17 | |
| *** wchrisj_ has joined #openstack-infra | 03:19 | |
| *** wchrisj has quit IRC | 03:20 | |
| *** sweston has joined #openstack-infra | 03:21 | |
| *** wchrisj has joined #openstack-infra | 03:24 | |
| *** wchrisj_ has quit IRC | 03:25 | |
| *** gokrokve has joined #openstack-infra | 03:27 | |
| *** wchrisj_ has joined #openstack-infra | 03:28 | |
| *** wchrisj has quit IRC | 03:29 | |
| *** sabari has joined #openstack-infra | 03:29 | |
| *** jhesketh has quit IRC | 03:29 | |
| *** nosnos has quit IRC | 03:29 | |
| *** jhesketh has joined #openstack-infra | 03:30 | |
| *** matsuhashi has joined #openstack-infra | 03:30 | |
| *** thuc has joined #openstack-infra | 03:31 | |
| *** wchrisj has joined #openstack-infra | 03:32 | |
| *** wchrisj_ has quit IRC | 03:32 | |
| *** Ryan_Lane has joined #openstack-infra | 03:32 | |
| *** chandankumar_ has joined #openstack-infra | 03:33 | |
| *** thuc_ has quit IRC | 03:34 | |
| *** sabari has quit IRC | 03:34 | |
| *** sabari has joined #openstack-infra | 03:35 | |
| *** vkozhukalov has joined #openstack-infra | 03:35 | |
| *** thuc has quit IRC | 03:35 | |
| *** matsuhashi has quit IRC | 03:36 | |
| *** wchrisj_ has joined #openstack-infra | 03:39 | |
| *** wchrisj has quit IRC | 03:40 | |
| *** chandankumar_ has quit IRC | 03:40 | |
| *** mspreitz has joined #openstack-infra | 03:40 | |
| *** wchrisj has joined #openstack-infra | 03:43 | |
| *** wchrisj_ has quit IRC | 03:43 | |
| *** Sukhdev has quit IRC | 03:43 | |
| *** wchrisj has quit IRC | 03:47 | |
| *** harlowja_ is now known as harlowja_away | 03:49 | |
| tchaypo | https://review.openstack.org/#/c/80858/ is still waiting for an approval | 03:50 |
| *** gokrokve has quit IRC | 03:50 | |
| tchaypo | since uvirtbot isn't being helpful, I'll fill in the detail that it's just adding an alias for me into gitdm | 03:50 |
| *** yjiang has joined #openstack-infra | 03:50 | |
| *** gokrokve has joined #openstack-infra | 03:52 | |
| *** alex-gone is now known as alex-sick | 03:54 | |
| *** chandankumar_ has joined #openstack-infra | 03:55 | |
| *** gokrokve has quit IRC | 03:57 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/gitdm: Add StevenK to the HP group list https://review.openstack.org/82706 | 03:57 |
| tchaypo | hehehe | 04:00 |
| SergeyLukjanov | evening folks | 04:10 |
| *** nati_ueno has quit IRC | 04:13 | |
| mspreitz | SergeyLukjanov: good evening | 04:15 |
| *** thuc has joined #openstack-infra | 04:18 | |
| SergeyLukjanov | mspreitz, heh, it's just 8am for me ;) | 04:18 |
| *** thuc_ has joined #openstack-infra | 04:18 | |
| *** thuc has quit IRC | 04:22 | |
| *** matsuhashi has joined #openstack-infra | 04:23 | |
| *** chandankumar_ has quit IRC | 04:23 | |
| *** nosnos has joined #openstack-infra | 04:25 | |
| *** fifieldt_ has quit IRC | 04:28 | |
| openstackgerrit | Gregory Haynes proposed a change to openstack-infra/pypi-mirror: Use openstack git urls instead of github https://review.openstack.org/82649 | 04:31 |
| *** Sukhdev has joined #openstack-infra | 04:32 | |
| openstackgerrit | Gregory Haynes proposed a change to openstack-infra/pypi-mirror: Add blurb on configuration and run_mirror in README https://review.openstack.org/80861 | 04:34 |
| *** thuc_ has quit IRC | 04:40 | |
| *** nati_ueno has joined #openstack-infra | 04:40 | |
| *** thuc has joined #openstack-infra | 04:41 | |
| *** thuc_ has joined #openstack-infra | 04:45 | |
| *** thuc has quit IRC | 04:45 | |
| nibalizer | greghaynes: fix it fix it fix it | 04:46 |
| greghaynes | nibalizer: no | 04:46 |
| greghaynes | :p | 04:46 |
| *** wenlock has quit IRC | 04:52 | |
| *** rhsu has joined #openstack-infra | 04:54 | |
| *** rhsu1 has quit IRC | 04:54 | |
| *** khyati has joined #openstack-infra | 04:54 | |
| *** yamahata has quit IRC | 04:58 | |
| *** yjiang has quit IRC | 04:58 | |
| *** yamahata has joined #openstack-infra | 04:58 | |
| *** yamahata has quit IRC | 04:59 | |
| *** yamahata has joined #openstack-infra | 04:59 | |
| *** _nadya_ has joined #openstack-infra | 05:01 | |
| *** thuc has joined #openstack-infra | 05:03 | |
| *** masayukig has joined #openstack-infra | 05:04 | |
| *** vkozhukalov has quit IRC | 05:05 | |
| *** thuc_ has quit IRC | 05:06 | |
| *** thuc has quit IRC | 05:08 | |
| *** CaptTofu has joined #openstack-infra | 05:10 | |
| *** matsuhashi has quit IRC | 05:14 | |
| *** vkozhukalov has joined #openstack-infra | 05:15 | |
| *** CaptTofu has quit IRC | 05:15 | |
| *** yjiang has joined #openstack-infra | 05:16 | |
| *** fifieldt_ has joined #openstack-infra | 05:17 | |
| *** matsuhashi has joined #openstack-infra | 05:18 | |
| clarkb | Alex_Gaynor: ohey you joined the list of non twitter users | 05:18 |
| *** skraynev_afk is now known as skraynev | 05:19 | |
| *** gokrokve has joined #openstack-infra | 05:21 | |
| *** _nadya_ has quit IRC | 05:21 | |
| *** gokrokve has quit IRC | 05:25 | |
| *** Manishanker has joined #openstack-infra | 05:26 | |
| *** emagana has joined #openstack-infra | 05:27 | |
| *** mihgen has joined #openstack-infra | 05:29 | |
| *** talluri has joined #openstack-infra | 05:29 | |
| *** sabari has quit IRC | 05:33 | |
| *** sabari has joined #openstack-infra | 05:35 | |
| *** fifieldt_ has quit IRC | 05:39 | |
| *** vkozhukalov has quit IRC | 05:39 | |
| *** _nadya_ has joined #openstack-infra | 05:41 | |
| openstackgerrit | Gregory Haynes proposed a change to openstack-infra/pypi-mirror: Add blurb on configuration and run_mirror in README https://review.openstack.org/80861 | 05:43 |
| *** _nadya_ has quit IRC | 05:44 | |
| openstackgerrit | Gregory Haynes proposed a change to openstack-infra/pypi-mirror: Use openstack git urls instead of github https://review.openstack.org/82649 | 05:46 |
| *** nati_ueno has quit IRC | 05:47 | |
| *** chandan_kumar has quit IRC | 05:48 | |
| *** e0ne has joined #openstack-infra | 05:49 | |
| *** nati_ueno has joined #openstack-infra | 05:51 | |
| *** nicedice has quit IRC | 05:53 | |
| *** pcrews has quit IRC | 05:55 | |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Add neutron functional job to devstack check queue https://review.openstack.org/82631 | 05:56 |
| tchaypo | clarkb: i technically still have a twitter account, but I only ever look at it when something is on fire. Do I count as a non-twitterer? | 05:57 |
| clarkb | tchaypo: I suppose | 05:57 |
| clarkb | tchaypo: a bunch of us don't twitter at all so find out about software releases or things that should be CVEs via alternate methods which causes interesting problems at times | 05:58 |
| tchaypo | sorry, when i said "on fire" i meant "literally on fire" | 05:59 |
| clarkb | oh right you live in a place that likes to burn | 05:59 |
| tchaypo | the most recent time I used twitter was for http://www.abc.net.au/news/2014-03-12/fire-at-barangaroo-construction-site-in-sydney-creating-large-v/5315696 | 06:00 |
| *** Sukhdev has quit IRC | 06:04 | |
| *** mrda is now known as mrda_away | 06:08 | |
| *** basha has joined #openstack-infra | 06:08 | |
| *** thuc has joined #openstack-infra | 06:14 | |
| *** andreaf has joined #openstack-infra | 06:14 | |
| *** dstanek has quit IRC | 06:15 | |
| *** ildikov_ has quit IRC | 06:15 | |
| *** mestery has quit IRC | 06:18 | |
| *** thuc has quit IRC | 06:19 | |
| *** sabari has quit IRC | 06:20 | |
| *** dstanek has joined #openstack-infra | 06:22 | |
| *** Ryan_Lane has quit IRC | 06:24 | |
| *** mspreitz has quit IRC | 06:26 | |
| *** rhsu has quit IRC | 06:34 | |
| *** imcsk8_lap has joined #openstack-infra | 06:44 | |
| *** ildikov_ has joined #openstack-infra | 06:59 | |
| *** fifieldt_ has joined #openstack-infra | 07:00 | |
| *** vkozhukalov has joined #openstack-infra | 07:03 | |
| *** dstanek has quit IRC | 07:04 | |
| *** bogdando has quit IRC | 07:05 | |
| *** jlibosva has joined #openstack-infra | 07:06 | |
| *** andreaf has quit IRC | 07:08 | |
| *** basha has quit IRC | 07:09 | |
| *** CaptTofu has joined #openstack-infra | 07:11 | |
| *** CaptTofu has quit IRC | 07:16 | |
| *** sandywalsh has quit IRC | 07:16 | |
| *** gokrokve has joined #openstack-infra | 07:21 | |
| *** wenlock_ has quit IRC | 07:23 | |
| *** fifieldt_ has quit IRC | 07:24 | |
| *** gokrokve has quit IRC | 07:26 | |
| *** khyati has quit IRC | 07:27 | |
| *** saju_m has joined #openstack-infra | 07:30 | |
| *** nati_ueno has quit IRC | 07:37 | |
| *** yolanda has joined #openstack-infra | 07:48 | |
| *** flaper87|afk is now known as flaper87 | 07:51 | |
| *** rlandy has joined #openstack-infra | 07:55 | |
| *** bogdando has joined #openstack-infra | 08:03 | |
| *** che-arne has quit IRC | 08:04 | |
| *** e0ne has joined #openstack-infra | 08:10 | |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Add a neutron job for api testing https://review.openstack.org/82226 | 08:19 |
| *** mihgen has quit IRC | 08:19 | |
| *** jasondotstar has joined #openstack-infra | 08:27 | |
| *** jgallard has joined #openstack-infra | 08:35 | |
| *** dizquierdo has joined #openstack-infra | 08:38 | |
| *** jooools has joined #openstack-infra | 08:45 | |
| *** matsuhashi has quit IRC | 08:49 | |
| *** saju_m has quit IRC | 08:50 | |
| marun | rake is missing? | 08:50 |
| marun | http://logs.openstack.org/31/82631/2/check/gate-config-puppet-lint/ce72930/console.html | 08:50 |
| marun | fungi, clarkb, jeblair ^ | 08:50 |
| *** yassine has joined #openstack-infra | 08:52 | |
| *** saju_m has joined #openstack-infra | 08:52 | |
| *** pblaho has joined #openstack-infra | 08:53 | |
| *** andreaf has joined #openstack-infra | 08:53 | |
| *** matsuhashi has joined #openstack-infra | 08:56 | |
| *** jcoufal has joined #openstack-infra | 08:57 | |
| *** yjiang has quit IRC | 09:02 | |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** hashar has joined #openstack-infra | 09:02 | |
| *** afazekas has joined #openstack-infra | 09:04 | |
| *** gokrokve has joined #openstack-infra | 09:05 | |
| *** gokrokve has joined #openstack-infra | 09:05 | |
| *** CaptTofu has joined #openstack-infra | 09:12 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Remove python-savannaclient from globalr req https://review.openstack.org/81084 | 09:12 |
| *** chandan_kumar has joined #openstack-infra | 09:13 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: automatically retry connect with allow_agent=False https://review.openstack.org/82748 | 09:13 |
| *** matsuhashi has quit IRC | 09:14 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Remove obsolete python-savannaclient https://review.openstack.org/81084 | 09:14 |
| *** derekh has joined #openstack-infra | 09:15 | |
| *** CaptTofu has quit IRC | 09:16 | |
| *** matsuhashi has joined #openstack-infra | 09:18 | |
| *** mkoderer has joined #openstack-infra | 09:21 | |
| *** johnthetubaguy has joined #openstack-infra | 09:21 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: automatically retry connect with allow_agent=False https://review.openstack.org/82748 | 09:25 |
| *** ociuhandu has quit IRC | 09:29 | |
| *** ominakov has joined #openstack-infra | 09:30 | |
| SergeyLukjanov | marun, yup, looks like some problems with ruby/rake installation | 09:31 |
| *** saju_m has quit IRC | 09:38 | |
| *** nosnos has quit IRC | 09:39 | |
| *** che-arne has joined #openstack-infra | 09:41 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Remove gantt from nova.json https://review.openstack.org/82754 | 09:43 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Remove gantt from nova.json https://review.openstack.org/82754 | 09:44 |
| *** andreaf has quit IRC | 09:49 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Remove gantt from nova.json https://review.openstack.org/82754 | 09:55 |
| *** saju_m has joined #openstack-infra | 09:56 | |
| *** ociuhandu has joined #openstack-infra | 09:57 | |
| *** ociuhandu has quit IRC | 09:59 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Retry ssh connect a few times https://review.openstack.org/82762 | 10:02 |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Format JSON text https://review.openstack.org/82541 | 10:03 |
| *** morganfainberg is now known as morganfainberg_Z | 10:05 | |
| *** matsuhashi has quit IRC | 10:08 | |
| *** alff has joined #openstack-infra | 10:08 | |
| *** matsuhashi has joined #openstack-infra | 10:10 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Retry ssh connect a few times https://review.openstack.org/82762 | 10:12 |
| *** AJaeger has quit IRC | 10:12 | |
| *** emagana has quit IRC | 10:13 | |
| *** salv-orlando has quit IRC | 10:14 | |
| *** e0ne_ has joined #openstack-infra | 10:16 | |
| *** jp_at_hp has joined #openstack-infra | 10:16 | |
| *** bauzas has joined #openstack-infra | 10:17 | |
| bauzas | morning all | 10:18 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: pbx: don't install asterisk-sounds-extra-en-g722 https://review.openstack.org/82765 | 10:18 |
| bauzas | I'm trying to provide a new patchset for https://review.openstack.org/#/c/80113/ which is a draft | 10:18 |
| bauzas | but gerrit fails to push to master | 10:18 |
| bauzas | ! [remote rejected] HEAD -> refs/drafts/master/bp/scheduler-lib (change state corrupt) | 10:19 |
| bauzas | does someone has any idea on what is the problem ? | 10:19 |
| *** e0ne has quit IRC | 10:19 | |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Added launchpad projects to definition https://review.openstack.org/82767 | 10:21 |
| *** CaptTofu has joined #openstack-infra | 10:25 | |
| *** masayukig has quit IRC | 10:28 | |
| *** ociuhandu has joined #openstack-infra | 10:28 | |
| *** markmcclain has joined #openstack-infra | 10:39 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Use OVERRIDE_ZUUL_BRANCH in devstack-vm-gate.sh https://review.openstack.org/82496 | 10:43 |
| *** emagana has joined #openstack-infra | 10:44 | |
| *** weshay has joined #openstack-infra | 10:47 | |
| *** jasondotstar has quit IRC | 10:47 | |
| *** andreaf has joined #openstack-infra | 10:47 | |
| *** emagana has quit IRC | 10:52 | |
| *** mihgen has joined #openstack-infra | 11:01 | |
| *** saju_m has quit IRC | 11:07 | |
| *** saju_m has joined #openstack-infra | 11:08 | |
| *** hashar has quit IRC | 11:08 | |
| *** saju_m has quit IRC | 11:09 | |
| *** mwagner_lap has quit IRC | 11:12 | |
| *** alff has quit IRC | 11:13 | |
| *** mbacchi has joined #openstack-infra | 11:14 | |
| *** saju_m has joined #openstack-infra | 11:15 | |
| *** hashar has joined #openstack-infra | 11:17 | |
| *** CaptTofu has quit IRC | 11:17 | |
| *** CaptTofu has joined #openstack-infra | 11:21 | |
| *** gokrokve has quit IRC | 11:23 | |
| *** miqui has joined #openstack-infra | 11:26 | |
| *** tianst20 has joined #openstack-infra | 11:31 | |
| *** e0ne_ has quit IRC | 11:33 | |
| *** e0ne has joined #openstack-infra | 11:33 | |
| *** gokrokve has joined #openstack-infra | 11:34 | |
| anteaya | bauzas: I see you have abandoned 80113 | 11:36 |
| anteaya | on the next attempt I do recommend you stay away from drafts | 11:37 |
| anteaya | while you should still be able to submit patchsets to drafts | 11:37 |
| *** dkliban has quit IRC | 11:37 | |
| *** gokrokve_ has joined #openstack-infra | 11:38 | |
| anteaya | gerrit does some interesting things when it comes to rebasing and other operations that we have set up to run automatically when it comes to drafts | 11:38 |
| *** saju_m has quit IRC | 11:38 | |
| *** e0ne has quit IRC | 11:38 | |
| *** saju_m has joined #openstack-infra | 11:38 | |
| anteaya | unfortunately I am light on details but our general recommendation is that developers avoid drafts | 11:38 |
| *** gokrokve has quit IRC | 11:39 | |
| *** hashar has quit IRC | 11:40 | |
| anteaya | regarding the missing rake in http://logs.openstack.org/31/82631/2/check/gate-config-puppet-lint/ce72930/console.html | 11:41 |
| anteaya | check what version of ruby you installed and what package installed it | 11:41 |
| anteaya | some versions of ruby might assume you are using bundler (a ruby gem management library) and have left out rake believeing you would get it if you wanted it by running bundle install after specifying rake in the bundler config file | 11:42 |
| *** gokrokve_ has quit IRC | 11:42 | |
| anteaya | http://bundler.io/bundle_config.html | 11:43 |
| anteaya | for our needs bundler is not the way to go, we just might need to look at sourcing ruby that is packaged with rake | 11:44 |
| *** ArxCruz has joined #openstack-infra | 11:48 | |
| *** basha has joined #openstack-infra | 11:51 | |
| *** jgallard has quit IRC | 11:51 | |
| anteaya | I don't see any changes to /config or /jjb regarding puppet-lint | 11:52 |
| anteaya | what image does puppet-lint run on? | 11:52 |
| anteaya | answered my own question, bare precise: bare-precise-hpcloud-az3 | 11:53 |
| *** eharney has quit IRC | 11:57 | |
| michchap | anteaya: there was a broken gate today when rake bumped its version and deprecated old ruby support | 11:58 |
| michchap | anteaya: We fixed by pinning to old rake, it's probably related. | 11:59 |
| anteaya | michchap: ah | 11:59 |
| anteaya | I don't see a patch to /requirements: http://git.openstack.org/cgit/openstack/requirements/log/ | 12:00 |
| anteaya | where did the pin to old rake take place? | 12:00 |
| michchap | https://review.openstack.org/#/c/82662/1/Gemfile | 12:00 |
| michchap | +8 or so others for other modules | 12:01 |
| *** skraynev is now known as skraynev_afk | 12:01 | |
| anteaya | the gate already uses bundler | 12:01 |
| anteaya | interesting | 12:01 |
| anteaya | michchap: was there a bug for this? | 12:02 |
| *** harlowja_away has quit IRC | 12:03 | |
| anteaya | and does this mean that I can send marun's patch https://review.openstack.org/#/c/82226/ back into check? | 12:03 |
| *** pdmars has joined #openstack-infra | 12:03 | |
| michchap | anteaya: no. I can make one. Chris (from puppetlabs) thought he had fixed it by pinning | 12:03 |
| anteaya | thanks | 12:04 |
| michchap | I have no idea re: marun's patch... I don't know infra very well. | 12:04 |
| anteaya | then I can do recheck bug <bug number> as I should | 12:04 |
| anteaya | k, I'm foggy on how we consume puppet-openstack | 12:05 |
| anteaya | I'm going with we don't track master and we do conscious updates when called for | 12:05 |
| * anteaya stands by to be corrected in about 90 minutes or less | 12:05 | |
| *** pbelanyi has joined #openstack-infra | 12:06 | |
| michchap | I'm not really sure how it's set up to be honest. | 12:06 |
| anteaya | that is okay | 12:06 |
| anteaya | if I have a bug number I will recheck marun's patch | 12:07 |
| anteaya | it will either pass puppet-lint or it won't | 12:07 |
| michchap | https://bugs.launchpad.net/openstack-ci/+bug/1297244 | 12:09 |
| uvirtbot | Launchpad bug 1297244 in openstack-ci "Rake not available to puppet-lint gate job" [Undecided,New] | 12:09 |
| anteaya | thank you | 12:09 |
| michchap | This is what the failures earlier today looked like: http://logs.openstack.org/43/82243/2/check/gate-puppet-ceilometer-puppet-unit-3.1/c35155b/console.html | 12:10 |
| *** gokrokve has joined #openstack-infra | 12:10 | |
| michchap | these were fixed by pinning, then a few hours later the lint job started to die. | 12:11 |
| *** e0ne has joined #openstack-infra | 12:12 | |
| anteaya | why is the lint job dying? | 12:13 |
| *** e0ne has quit IRC | 12:13 | |
| *** julim has joined #openstack-infra | 12:13 | |
| michchap | tmp/hudson4652546853617966169.sh: line 12: rake: command not found | 12:14 |
| *** e0ne has joined #openstack-infra | 12:14 | |
| michchap | so if bundler is running earlier and failing to install rake, it's the same issue essentially | 12:14 |
| anteaya | yes that is the output that marun's patch has in the log | 12:14 |
| anteaya | running earlier | 12:14 |
| anteaya | running earlier than what? | 12:14 |
| michchap | earlier than the job? | 12:15 |
| michchap | I'm not sure. | 12:16 |
| michchap | There is nothing in the lint job that is installing rake, and yet the command is not available, so I'm curious as to where the command is coming from unless it's baked into the image. | 12:16 |
| *** matsuhashi has quit IRC | 12:16 | |
| *** dkranz has joined #openstack-infra | 12:17 | |
| anteaya | the lint job shouldn't be installing rake | 12:17 |
| anteaya | rake should be installed with the dependencies like ruby | 12:17 |
| anteaya | I would complain to rake maintainers except Jim died about a month ago | 12:17 |
| anteaya | *tears* | 12:17 |
| anteaya | and I don't know who is maintaining rake now | 12:18 |
| michchap | :( | 12:18 |
| anteaya | did you know him? | 12:18 |
| michchap | I did not. | 12:18 |
| anteaya | he was such a great guy | 12:18 |
| anteaya | big loss | 12:18 |
| *** e0ne has quit IRC | 12:18 | |
| fungi | anteaya: the puppet projects on stackforge use bundler to install a recent rake. we were relying on ubuntu-provided rake packages on our precise nodes instead for the infra puppet-lint job i think... checking to be sure now | 12:18 |
| fungi | actually, yeah, i think we were using the one which was inside the ruby packages on ubuntu precise | 12:19 |
| *** gokrokve has quit IRC | 12:19 | |
| *** jgallard has joined #openstack-infra | 12:19 | |
| anteaya | fungi: yeah that is consistent with my expectations | 12:20 |
| michchap | Using rake (10.1.1) | 12:20 |
| *** gokrokve has joined #openstack-infra | 12:22 | |
| michchap | the puppet gate jobs were using bundler to install the deps, but the rake was already 10.1.1 so it wasn't being installed. | 12:22 |
| anteaya | right | 12:22 |
| michchap | s/were/are | 12:22 |
| anteaya | the puppet gate jobs | 12:22 |
| michchap | yep, not lint. | 12:22 |
| anteaya | what are you refering to? | 12:23 |
| michchap | eg: http://logs.openstack.org/62/82662/1/gate/gate-puppet-openstack-puppet-unit-3.3/88be31e/console.html | 12:23 |
| *** matsuhashi has joined #openstack-infra | 12:23 | |
| *** david-lyle has quit IRC | 12:23 | |
| anteaya | the gate jobs for stackforge/puppet-openstack? | 12:23 |
| michchap | yes | 12:23 |
| fungi | right, and i'm talking about the puppet-lint jobs for the openstack-infra/config project | 12:23 |
| *** maxbit has joined #openstack-infra | 12:23 | |
| anteaya | okay | 12:23 |
| fungi | which were written previous to those and are different | 12:23 |
| bauzas | anteaya: thanks for the feedback | 12:23 |
| *** markmcclain has quit IRC | 12:24 | |
| anteaya | when I am talking about "the gate jobs" I mean the ones for the openstack/ projects | 12:24 |
| *** markmcclain has joined #openstack-infra | 12:24 | |
| anteaya | bauzas: np | 12:24 |
| bauzas | anteaya: so, gerrit drafts are buggy ? | 12:24 |
| bauzas | anteaya: I never noticed that behaviour before | 12:24 |
| anteaya | bauzas: I wouldn't say buggy, since the behaviour seems to be consistent | 12:25 |
| anteaya | as in we consistently recommend they not be used | 12:25 |
| bauzas | anteaya: but as per googling, seems like my change-id was forgotten by gerrit | 12:25 |
| *** markmcclain1 has joined #openstack-infra | 12:25 | |
| *** alff has joined #openstack-infra | 12:25 | |
| anteaya | but I don't know enough about the intricacies of how drafts are tracked at each stage to say exactly what happens | 12:25 |
| bauzas | anteaya: I often updated drafts, that's the first time I see this behaviour, hence the word "buggy" ;) | 12:26 |
| anteaya | bauzas: if your change id was forgotten by gerrit, that behaviour would be considered a bug, yes | 12:26 |
| anteaya | drafts should be able to be updated yes | 12:26 |
| bauzas | anteaya: that is, because I provided 3 patchsets on that review | 12:26 |
| anteaya | it is farther down the food chain in the land of rebasing and such where we usually see problems | 12:27 |
| bauzas | anteaya: that's only when trying to provide a 4th patchset that gerrit complained | 12:27 |
| *** yamahata has quit IRC | 12:27 | |
| bauzas | anteaya: oh ok got it | 12:27 |
| *** markmcclain2 has joined #openstack-infra | 12:27 | |
| bauzas | anteaya: do you know if there is some way to promote a change as WIP thanks to CLI ? | 12:27 |
| *** markmcclain2 has quit IRC | 12:27 | |
| anteaya | bauzas: which I find interesting, i have not seen a lost change id circumstance in my travels with gerrit before | 12:27 |
| *** gokrokve has quit IRC | 12:28 | |
| anteaya | not using CLI, just the WIP button on the gui | 12:28 |
| bauzas | anteaya: because I made my way up by creating a second patch, but jenkins/zuul was too quick for me | 12:28 |
| *** dkranz has quit IRC | 12:28 | |
| *** markmcclain2 has joined #openstack-infra | 12:28 | |
| bauzas | the patch went into the pipeline before I had the time to click on the button :) | 12:28 |
| *** dkranz has joined #openstack-infra | 12:29 | |
| anteaya | bauzas: I don't understand you second last statement | 12:29 |
| anteaya | oh yes it will | 12:29 |
| *** markmcclain has quit IRC | 12:29 | |
| anteaya | you can still mark it as WIP | 12:29 |
| anteaya | check tests will run on WIP | 12:29 |
| *** AJaeger has joined #openstack-infra | 12:29 | |
| bauzas | anteaya: I was thinking that WIP patchsets don't go into zuul | 12:29 |
| bauzas | I mean the check queue | 12:29 |
| anteaya | they do | 12:29 |
| *** markmcclain1 has quit IRC | 12:29 | |
| anteaya | zuul reads all the gerrit events | 12:30 |
| bauzas | oh that's right | 12:30 |
| bauzas | only drafts don't | 12:30 |
| *** markmcclain2 has quit IRC | 12:30 | |
| bauzas | anteaya: don't you think it would be worth to leave WIP patchsets not going into check queue ? | 12:31 |
| anteaya | I don't use drafts but looking at https://review.openstack.org/#/c/80113/ I would agree with your conclusion | 12:31 |
| *** aysyd has joined #openstack-infra | 12:31 | |
| *** markmcclain has joined #openstack-infra | 12:31 | |
| bauzas | anteaya: a WIP patch could still be checked thanks to the experimental queue | 12:31 |
| *** dachary has quit IRC | 12:31 | |
| bauzas | anteaya: that would be on-demand | 12:31 |
| anteaya | bauzas: the resources we have to provide check tests are such that it is easier to let them go through check | 12:31 |
| *** gokrokve has joined #openstack-infra | 12:32 | |
| anteaya | you can keep working on the patch series and not worry about the check test feedback until you are ready | 12:32 |
| *** gaelL_ has quit IRC | 12:32 | |
| bauzas | anteaya: but that do consume hardware resources on such a thing which is 'in progress' ;) | 12:32 |
| anteaya | but honestly it is helpful to give that feedback to devs | 12:32 |
| *** gaelL has joined #openstack-infra | 12:32 | |
| anteaya | virtual resources | 12:32 |
| anteaya | we run no hardware | 12:33 |
| bauzas | yey, you're right | 12:33 |
| bauzas | but virtual resources tend to use slices of hardware :) | 12:33 |
| *** dachary has joined #openstack-infra | 12:33 | |
| anteaya | they do | 12:33 |
| anteaya | but honestly we do have the resources available to us | 12:33 |
| *** rfolco has joined #openstack-infra | 12:34 | |
| anteaya | and the percentage consumed by WIP patches is worth it | 12:34 |
| anteaya | if it becomes not worth it we will revise | 12:34 |
| fungi | apparently our sudden rake problem is... puppet | 12:34 |
| bauzas | anteaya: that sounds ok | 12:34 |
| fungi | 2014-03-25 02:27:08,347 INFO nodepool.image.build.hpcloud-az2.bare-precise: ^[[1;35merr: /Stage[main]/Jenkins::Slave/Package[puppetlabs_spec_helper]/ensure: change from absent to latest failed: Could not update: Execution of '/usr/bin/gem install --include-dependencies --no-rdoc --no-ri puppetlabs_spec_helper' returned 1: ERROR: Error installing puppetlabs_spec_helper: | 12:34 |
| fungi | 2014-03-25 02:27:08,347 INFO nodepool.image.build.hpcloud-az2.bare-precise: rake requires Ruby version >= 1.9. | 12:34 |
| anteaya | the puppetlabs_spec_helper needs a pin? | 12:35 |
| fungi | i... guess? | 12:35 |
| anteaya | michchap: did you say Chris Hoge works with puppetlabs? | 12:35 |
| michchap | it's the same patch was we made to the puppet-openstack jobs | 12:35 |
| michchap | pin the version of rake to 10.1.1 | 12:35 |
| anteaya | fungi: do we need to grab the latest puppetlabs_spec_helper? | 12:36 |
| fungi | michchap: i don't think we were explicitly installing rake previously, and were just letting gem figure it out as a transitive dep | 12:36 |
| anteaya | michchap: has puppetlabs_spec_helper been pinned? | 12:36 |
| fungi | but i'll see where i can shoehorn it in | 12:36 |
| michchap | fungi: it's probably just pulling in master from the spec_helper gem then. | 12:36 |
| anteaya | fungi: we can't really, since we don't use bundler | 12:36 |
| anteaya | michchap: yes, the spec_helper gem needs a patch with the pin | 12:37 |
| fungi | anteaya: oh, gem install won't let us specify versions of multiple packages on the same command line? | 12:37 |
| anteaya | well it might, I will look it up | 12:38 |
| anteaya | but with bundler being the new hotness, folks tend not to use gem install anymore | 12:38 |
| *** zul has quit IRC | 12:38 | |
| fungi | looks like maybe we use puppet's gem package provider | 12:39 |
| *** ken1ohmichi has joined #openstack-infra | 12:40 | |
| anteaya | fungi: oh, no idea how to work with that | 12:41 |
| *** zul has joined #openstack-infra | 12:41 | |
| *** saju_m has quit IRC | 12:41 | |
| fungi | yep, provider => gem, | 12:41 |
| SergeyLukjanov | fungi, anteaya, morning | 12:42 |
| fungi | so just need to dig into that and see if it can do something other than ensure => latest | 12:42 |
| SergeyLukjanov | fungi, are you planning to explicitly preinstall rake on slaves? | 12:42 |
| *** maxbit has quit IRC | 12:42 | |
| fungi | right now we're using it to install whatever it thinks the "latest" gems are of bundler, puppet-lint and puppetlabs_spec_helper | 12:42 |
| *** gokrokve has quit IRC | 12:43 | |
| anteaya | SergeyLukjanov: right now we are figuring out the best way to address the rake breakage | 12:43 |
| fungi | SergeyLukjanov: not sure. need to find out if there's a ruby 1.8 compatible release train for rake we should be limiting ourselves to so we can still get security fixes | 12:43 |
| *** gokrokve has joined #openstack-infra | 12:43 | |
| anteaya | preinstalling rake won't help since it will be overwritten with "latest" rake later | 12:43 |
| michchap | if you do gem install rake --version 10.1.1, then gem install puppetlabs_spec_helper, your rake won't be overwritten. | 12:44 |
| anteaya | we need to find the bit that is the last item to check for rake and point it at 10.1.1 | 12:44 |
| fungi | without having to, you know, throw away the idea of running stable server releases. i think maybe the ruby community don't run production servers for years at a time? could have to do with being unable to get jobs in that which last longer than a 6 month contract ;) | 12:44 |
| anteaya | and eric hodel is the sole maintainer of rake now | 12:44 |
| anteaya | never could talk to him | 12:45 |
| *** CaptTofu has quit IRC | 12:45 | |
| michchap | so making a new package with provider gem for rake, and specifying the version, and adding it as a dependency to the puppetlabs_spec_helper package might be an option. | 12:45 |
| anteaya | fungi: yeah, not being held back by backward compatibility is a theme song with ruby | 12:45 |
| anteaya | so this is not a surprise | 12:46 |
| fungi | not being held back by stodgy old greybeards who want their servers to keep running | 12:46 |
| anteaya | exactly | 12:46 |
| anteaya | now you get the picture | 12:46 |
| fungi | anyway, i'm edumacating myself up on the gem package provider | 12:46 |
| *** mwagner_lap has joined #openstack-infra | 12:46 | |
| *** hashar has joined #openstack-infra | 12:46 | |
| anteaya | michchap: so in this proposal, who is making a new package with provider gem for rake? | 12:47 |
| anteaya | michchap: us or puppetlabs? | 12:47 |
| anteaya | I'm hoping puppetlabs since I have no idea how to do it | 12:47 |
| anteaya | fungi: what are you reading? | 12:47 |
| *** gokrokve has quit IRC | 12:47 | |
| michchap | anteaya: maybe I'm totally confused, but you pasted a log that indicated this was a puppet manifest above: Stage[main]/Jenkins::Slave/Package[puppetlabs_spec_helper]/ensure: change from absent to latest failed | 12:48 |
| fungi | anteaya: http://docs.puppetlabs.com/puppet/2.7/reference/ | 12:48 |
| fungi | michchap: yes, i think we just need to tell it to ensure something other than latest | 12:48 |
| anteaya | fungi: thanks, joining you | 12:48 |
| anteaya | michchap: fungi pasted that | 12:48 |
| *** adalbas has joined #openstack-infra | 12:48 | |
| michchap | whatever that manifest is, make a package { 'rake': provider => gem, version => '10.1.1', before => Package['puppetlabs_spec_helper'] } | 12:49 |
| fungi | http://docs.puppetlabs.com/references/2.7.latest/type.html#package does suggest that the gem provider is versionable | 12:50 |
| fungi | michchap: excellent--thanks! i'll try that in a dev env | 12:51 |
| fungi | once i get a little more coffee in me | 12:51 |
| michchap | fungi: or maybe ensure => '10.1.1', I will get back to you in a sec, I forget | 12:51 |
| fungi | michchap: figured it would be one of those. we do similar things with other package providers | 12:51 |
| *** mrodden has quit IRC | 12:52 | |
| *** mrodden has joined #openstack-infra | 12:52 | |
| *** dims_ has quit IRC | 12:52 | |
| anteaya | under package > provider > gem Required binaries: gem. Supported features: installable, uninstallable, upgradeable, versionable. | 12:52 |
| anteaya | looks like versionable might come in handy | 12:53 |
| *** jhesketh has quit IRC | 12:53 | |
| *** jasondotstar has joined #openstack-infra | 12:53 | |
| *** jhesketh_ has quit IRC | 12:53 | |
| michchap | fungi: confirmed it's ensure => $version | 12:53 |
| *** gokrokve has joined #openstack-infra | 12:53 | |
| fungi | michchap: thanks! | 12:54 |
| *** saju_m has joined #openstack-infra | 12:55 | |
| *** thuc has joined #openstack-infra | 12:57 | |
| *** jeckersb_gone is now known as jeckersb | 12:58 | |
| *** thuc_ has joined #openstack-infra | 12:58 | |
| anteaya | http://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/manifests/slave.pp#n169 | 12:58 |
| *** saju_m has quit IRC | 12:59 | |
| fungi | anteaya: yep, that's where i'm editing | 12:59 |
| *** dkranz has quit IRC | 12:59 | |
| *** saju_m has joined #openstack-infra | 13:00 | |
| fungi | anteaya: michchap: trying https://review.openstack.org/82817 in a dev env here shortly (after i find out why openstackgerrit has gone on silent running) | 13:00 |
| *** thuc has quit IRC | 13:01 | |
| anteaya | fungi: did you want to toss in fixes bug https://bugs.launchpad.net/openstack-ci/+bug/1297244 | 13:01 |
| uvirtbot | Launchpad bug 1297244 in openstack-ci "Rake not available to puppet-lint gate job" [Undecided,New] | 13:01 |
| anteaya | in the commit message? | 13:01 |
| *** lcostantino has joined #openstack-infra | 13:01 | |
| *** heyongli has joined #openstack-infra | 13:02 | |
| fungi | oh, we had a bug? sure, i'll add that before i'm done testing it | 13:02 |
| *** dcramer_ has joined #openstack-infra | 13:02 | |
| *** dims_ has joined #openstack-infra | 13:02 | |
| fungi | eerily quiet in the gate right now... | 13:03 |
| anteaya | michchap cobbled one together for me | 13:03 |
| anteaya | so i had a number to use for recheck | 13:03 |
| *** matsuhashi has quit IRC | 13:03 | |
| *** dprince has joined #openstack-infra | 13:03 | |
| fungi | too quiet--it's not picked up my change for testing | 13:04 |
| fungi | i think gerrit may be broken | 13:04 |
| anteaya | too quiet | 13:04 |
| anteaya | bigger problem | 13:04 |
| fungi | yep, switching gears to that npw | 13:04 |
| fungi | bnow | 13:04 |
| * fungi gives up trying to type | 13:04 | |
| *** saju_m has quit IRC | 13:04 | |
| * anteaya nods | 13:05 | |
| *** saju_m has joined #openstack-infra | 13:05 | |
| fungi | ossh -p 29418 review gerrit show-queue | 13:06 |
| *** ildikov_ has quit IRC | 13:06 | |
| *** markmc has joined #openstack-infra | 13:06 | |
| fungi | we've got a ton of waiting stream-events sessions again | 13:06 |
| *** heyongli has quit IRC | 13:07 | |
| *** heyongli has joined #openstack-infra | 13:07 | |
| anteaya | the fix for that last time was...? | 13:07 |
| * anteaya forgets | 13:07 | |
| anteaya | restarting gerrit I think | 13:07 |
| fungi | we restarted gerrit and lost all teh pending events | 13:07 |
| anteaya | since we couldn't figure out how to force a disconnection from the listeners | 13:08 |
| fungi | i'm trying to refresh my memory from logs now to see whether there was anything else we decided in the aftermath we could have tried | 13:08 |
| anteaya | it was you and jim mostly | 13:08 |
| anteaya | and a Sunday afternoon I think | 13:08 |
| fungi | i now (since troubleshooting nodepool connectivity issues for tripleo-ci) have a potential way to force-disconnect established tcp sockets | 13:08 |
| fungi | which i was wishing i could figure out how to do at the time | 13:09 |
| fungi | i'll see if i can get that working | 13:09 |
| *** bknudson has joined #openstack-infra | 13:09 | |
| *** saju_m has quit IRC | 13:09 | |
| fungi | this little treasure... http://killcx.sourceforge.net/ | 13:10 |
| anteaya | great name | 13:10 |
| *** saju_m has joined #openstack-infra | 13:10 | |
| anteaya | what quantity of sessions? | 13:11 |
| fungi | i can use it to kill them individually | 13:11 |
| fungi | and see if i can isolate which one was possible "stuck" | 13:11 |
| fungi | possibly | 13:11 |
| anteaya | do you think it is a question of a stuck session | 13:12 |
| anteaya | or too many sessions at the same time? | 13:12 |
| fungi | no clue--we couldn't determine from the javamelody stats last time | 13:12 |
| anteaya | right | 13:12 |
| fungi | here's what my previous adventure in killcx looked like: http://paste.openstack.org/show/72814/ | 13:12 |
| *** ildikov_ has joined #openstack-infra | 13:13 | |
| fungi | now just have to figure out what the prerequisite package names ended up being. probably have those in my shell history on nodepool still | 13:13 |
| fungi | looks like just libnet-rawip-perl, libnet-pcap-perl and libnetpacket-perl | 13:14 |
| anteaya | 50% effective? | 13:15 |
| anteaya | after killcx'ing 138.35.77.16:13774 it was still established | 13:16 |
| anteaya | unless it was reestablished | 13:16 |
| fungi | reestablished--look at the source port number | 13:18 |
| fungi | not the same as it was in the beforetime | 13:18 |
| *** dstanek has joined #openstack-infra | 13:18 | |
| anteaya | yes, I noticed the source port was different | 13:19 |
| *** todd_dsm has joined #openstack-infra | 13:20 | |
| fungi | #status alert the gerrit event stream is currently hung, blocking all testing. troubleshooting is in progress (next update at 14:00 utc) | 13:20 |
| openstackstatus | fungi: sending alert | 13:20 |
| *** wchrisj has joined #openstack-infra | 13:20 | |
| openstackstatus | NOTICE: the gerrit event stream is currently hung, blocking all testing. troubleshooting is in progress (next update at 14:00 utc) | 13:21 |
| *** ChanServ changes topic to "the gerrit event stream is currently hung, blocking all testing. troubleshooting is in progress (next update at 14:00 utc)" | 13:21 | |
| *** dkranz has joined #openstack-infra | 13:21 | |
| openstackstatus | fungi: finished sending alert | 13:22 |
| *** changbl has quit IRC | 13:22 | |
| *** eharney has joined #openstack-infra | 13:23 | |
| *** wchrisj has quit IRC | 13:26 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add check-rally-install-* template and jobs https://review.openstack.org/82617 | 13:26 |
| *** mriedem has joined #openstack-infra | 13:27 | |
| *** dkliban has joined #openstack-infra | 13:27 | |
| fungi | well, false alarm. whatever the problem was, it cleared on its own while i was still carefully choosing which connections to sever first | 13:28 |
| *** mestery has joined #openstack-infra | 13:28 | |
| *** mestery has quit IRC | 13:28 | |
| anteaya | fungi: you scared it | 13:28 |
| fungi | #status ok the issue with gerrit cleared on its own before any corrective action was taken | 13:28 |
| openstackstatus | fungi: sending ok | 13:28 |
| *** mestery has joined #openstack-infra | 13:29 | |
| fungi | though i'm keeping a very close eye on it now | 13:29 |
| anteaya | back to the regularly scheduled brokeness | 13:29 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 13:29 | |
| *** dizquierdo is now known as dizquierdo_afk | 13:30 | |
| fungi | i think i'll take this opportunity to get my second cup of coffee, and then get back to shaking my fist at rake for a few minutes | 13:30 |
| anteaya | sounds good | 13:33 |
| *** markmcclain has quit IRC | 13:34 | |
| *** wchrisj has joined #openstack-infra | 13:35 | |
| *** yamahata has joined #openstack-infra | 13:38 | |
| *** nati_ueno has joined #openstack-infra | 13:40 | |
| *** thuc_ has quit IRC | 13:41 | |
| *** thuc has joined #openstack-infra | 13:42 | |
| *** wenlock has joined #openstack-infra | 13:45 | |
| *** thuc has quit IRC | 13:47 | |
| *** dcramer_ has quit IRC | 13:53 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/git-review: Skip invalid unicode in commit messages https://review.openstack.org/82826 | 13:54 |
| *** dangers is now known as dangers_away | 13:55 | |
| michchap | fungi: so if the lint check fails because the image is bad and can't run the lint job, how does one update the image that is built from the thing that is being linted? | 13:56 |
| *** zns has joined #openstack-infra | 13:57 | |
| mordred | michchap: that sounds like an existential question... | 13:57 |
| fungi | michchap: with mighty sysadmin force | 13:57 |
| *** wchrisj has quit IRC | 13:57 | |
| *** signed8bit has joined #openstack-infra | 13:57 | |
| mordred | once we have public glance endpoints and can rework that part of image creation and upload, I believe we're quite interested in running tests using our images before we upload them | 13:58 |
| mordred | at least I am | 13:58 |
| *** wchrisj_ has joined #openstack-infra | 13:59 | |
| michchap | mordred: I thought this was #philosophy, sorry. | 13:59 |
| mordred | :) | 13:59 |
| *** nati_ueno has quit IRC | 13:59 | |
| * mordred goes back to read the whole scrollback | 13:59 | |
| anteaya | if I want to add a variable that an individual channel can set the value for inorder to allow meetbot to log to eavesdrop but not start meetings in error | 14:00 |
| anteaya | would this be where I introduce the variable: http://git.openstack.org/cgit/openstack-infra/meetbot/tree/MeetBot/config.py?id=5f8bdbeef15f9ab7486e2f2f8e7c622a223cdb4e#n43 | 14:00 |
| anteaya | setting the lurk option doesn't work unless a meeting is currently underway, I do believe since it doesn't work if there is no meeting in progress | 14:02 |
| *** heyongli has quit IRC | 14:02 | |
| mordred | fungi: I didn't read the whole scrollback - but have we considered porting the openstack-infra/config puppetlint job to use the stuff the stackforge puppet stuff uses? | 14:04 |
| *** hashar_ has joined #openstack-infra | 14:04 | |
| *** hashar has quit IRC | 14:05 | |
| *** prad has joined #openstack-infra | 14:05 | |
| *** wenlock has quit IRC | 14:05 | |
| *** pfallenop has quit IRC | 14:05 | |
| *** pcrews has joined #openstack-infra | 14:05 | |
| *** pfallenop has joined #openstack-infra | 14:07 | |
| *** jswarren has quit IRC | 14:08 | |
| *** prad has left #openstack-infra | 14:09 | |
| *** talluri has quit IRC | 14:11 | |
| *** alff has quit IRC | 14:11 | |
| fungi | mordred: sounds like a reasonable enough approach | 14:11 |
| *** ken1ohmichi has quit IRC | 14:13 | |
| fungi | currently trying to update my vm build scripts to just use the nodepool scripts, so that i can test whether the currently proposed patch works | 14:13 |
| *** prad has joined #openstack-infra | 14:16 | |
| *** jasondotstar has quit IRC | 14:21 | |
| *** mestery has quit IRC | 14:24 | |
| *** homeless has joined #openstack-infra | 14:25 | |
| *** CaptTofu has joined #openstack-infra | 14:27 | |
| *** alexpilotti has quit IRC | 14:28 | |
| anteaya | fungi: radware is in the voting category in gerrit: https://review.openstack.org/#/admin/groups/91,members | 14:28 |
| anteaya | fungi: is that a recent change? | 14:29 |
| *** signed8bit has quit IRC | 14:29 | |
| anteaya | I was just going to link them in a reply to http://lists.openstack.org/pipermail/openstack-infra/2014-March/001092.html | 14:29 |
| anteaya | to say they are able to comment | 14:30 |
| *** signed8bit has joined #openstack-infra | 14:30 | |
| fungi | anteaya: no idea... | 14:30 |
| anteaya | k | 14:30 |
| anteaya | sounds like you didn't change anything yesterday with their account | 14:31 |
| *** david-lyle has joined #openstack-infra | 14:31 | |
| fungi | nope | 14:31 |
| fungi | was that the one who was asking in the ml whether they could be reenabled? | 14:32 |
| openstackgerrit | Nathan Kinder proposed a change to openstack-infra/config: Add the openstack-security-notes project https://review.openstack.org/73157 | 14:32 |
| fungi | maybe we disabled their account due to commentspam or something months ago, before there were separate voting and non-voting groups | 14:32 |
| anteaya | fungi: yes | 14:32 |
| anteaya | possibly | 14:32 |
| *** markmcclain has joined #openstack-infra | 14:32 | |
| *** markmcclain has quit IRC | 14:32 | |
| fungi | only recent additions have been going into the non-voting group first | 14:32 |
| anteaya | k | 14:33 |
| *** mestery has joined #openstack-infra | 14:33 | |
| *** markmcclain has joined #openstack-infra | 14:33 | |
| anteaya | so do you want to leave them as voting or move them to non-voting pending markmcclain's approval? | 14:33 |
| anteaya | from his response, he isn't ready to have them voting | 14:33 |
| anteaya | that is my read of it | 14:33 |
| fungi | sure, i can do whatever the neutron community feels is appropriate with that account | 14:34 |
| miqui | hi, where is the pylint env setting for tox? | 14:34 |
| *** luisg has joined #openstack-infra | 14:35 | |
| fungi | miqui: usually in your project's tox.in | 14:36 |
| anteaya | fungi: leave them as is | 14:36 |
| fungi | tox.ini file | 14:36 |
| fungi | anteaya: i already moved them to the non-voting group, but haven't checked to see whether their account is disabled or anything | 14:36 |
| fungi | i can move them back to voting if that's desired after all | 14:36 |
| *** todd_dsm has quit IRC | 14:37 | |
| miqui | fungi: i'll paste bin what i have...i am looking at the tox.ini under /config dir and there is no deps=pylint or env= for it | 14:37 |
| fungi | miqui: in what project? | 14:37 |
| *** todd_dsm has joined #openstack-infra | 14:37 | |
| anteaya | fungi: did you | 14:37 |
| miqui | fungi: openstack-infra/config | 14:37 |
| fungi | miqui: we don't run pylint against the openstack-infra/config project. should we? (it doesn't really have much python in it) | 14:38 |
| anteaya | markmcclain: what do you want fungi to do? | 14:38 |
| anteaya | to be honest I feel kind of useless here | 14:38 |
| fungi | anteaya: i can put them back in the voting group like they were previously. i just don't want to be the decision maker on these sorts of issues because they don't impact me. i'm just the button-masher | 14:39 |
| *** jswarren has joined #openstack-infra | 14:39 | |
| * fungi puts things back the way he found them | 14:40 | |
| *** dprince has quit IRC | 14:40 | |
| *** todd_dsm has quit IRC | 14:40 | |
| miqui | fungi: not even for zuul? (lots of python there) | 14:41 |
| *** wenlock has joined #openstack-infra | 14:41 | |
| miqui | fungi: just took a look there... | 14:41 |
| fungi | miqui: zuul is in a separate project, not openstack-infra/config | 14:41 |
| markmcclain | fungi: I think leaving things as they were makes sense | 14:41 |
| miqui | fungi: i know... am poking around to find a project that has pylint defined in tox.ini | 14:42 |
| fungi | miqui: but yes, we stopped caring about pylint a long time ago. pyflakes is better (and then we started using flake8 which basically implements pep8 and pyflakes as backends) | 14:42 |
| sdague | miqui: nova has a pylint target | 14:42 |
| sdague | however it's mostly ignored, because pylint has too many false positives | 14:42 |
| sdague | it seems to break down on anything reasonably complicated | 14:43 |
| *** Manishanker has left #openstack-infra | 14:43 | |
| *** mfer has joined #openstack-infra | 14:46 | |
| SergeyLukjanov | sdague, it helps a bit, we're using it in the same way as nova and once-per-month it founds something ;) | 14:46 |
| *** jasondotstar has joined #openstack-infra | 14:46 | |
| sdague | SergeyLukjanov: that's a pretty low signal to noise :) | 14:47 |
| fungi | SergeyLukjanov: it does find errors pyflakes misses? | 14:47 |
| fungi | SergeyLukjanov: worth filing bugs against pyflakes for those, if so | 14:47 |
| SergeyLukjanov | fungi, yup, it does several times I think | 14:47 |
| *** basha has quit IRC | 14:48 | |
| SergeyLukjanov | fungi, but I'm not sure that it's covered by pyflakes | 14:48 |
| sdague | yeh, it's a different kind of toolchain | 14:48 |
| fungi | i figured the two of them implemented mostly overlapping static analysis | 14:48 |
| *** vhoward has joined #openstack-infra | 14:49 | |
| *** reed has joined #openstack-infra | 14:51 | |
| *** dprince has joined #openstack-infra | 14:53 | |
| *** CaptTofu has quit IRC | 14:54 | |
| *** dcramer_ has joined #openstack-infra | 14:54 | |
| *** rcleere has joined #openstack-infra | 14:54 | |
| *** thedodd has joined #openstack-infra | 14:54 | |
| jeblair | fungi: morning; bummer about gerrit and the stuck sessions | 14:55 |
| jeblair | fungi: anything you'd like me to look into? | 14:56 |
| *** thuc_ has joined #openstack-infra | 14:57 | |
| sdague | is there a good way to query what was stuck and flush them back out? | 14:57 |
| *** mgagne has quit IRC | 14:58 | |
| sdague | I only realized later that a tempest review which was needed for a nova rc1 bug was in limbo after I'd approved it, and russell brought it up | 14:58 |
| mordred | sdague, miqui I've been arguing fora while that now that we have pyflakes, we shoudl retire all of the pylint stuff | 14:59 |
| fungi | jeblair: i couldn't spot anything out of the ordinary in graphs or anything | 14:59 |
| sdague | mordred: no arguments here | 14:59 |
| mordred | sdague: woto | 14:59 |
| mordred | woot | 14:59 |
| mordred | damn | 14:59 |
| fungi | jeblair: other than it looked like nodepool and zuul stopped getting anything on their event streams for a while, starting around 11:00 utc (rough estimate) | 14:59 |
| mordred | miqui: so yeah, I recommend using flake8 - or hacking which uses flake8 - I think you'll find the experience much more pleasant | 15:00 |
| sdague | is woto the new zomg ? | 15:00 |
| fungi | woto is the new boto replacement | 15:01 |
| *** lcostantino has quit IRC | 15:04 | |
| sdague | if only :) | 15:04 |
| *** primeministerp has joined #openstack-infra | 15:04 | |
| *** eharney has quit IRC | 15:05 | |
| *** adalbas has quit IRC | 15:08 | |
| *** signed8bit has quit IRC | 15:12 | |
| *** saju_m has quit IRC | 15:13 | |
| *** signed8bit has joined #openstack-infra | 15:13 | |
| dhellmann | sdague: have a sec to talk about namespace packages? I'm trying to reproduce and understand the nova issue you mentioned yesterday. | 15:14 |
| openstackgerrit | Chris Hoge proposed a change to openstack-infra/config: Wraps running of puppet-lint in Bundler for version pinning. https://review.openstack.org/82844 | 15:14 |
| *** packet has joined #openstack-infra | 15:14 | |
| *** wchrisj_ has quit IRC | 15:15 | |
| *** johnthetubaguy has quit IRC | 15:15 | |
| *** johnthetubaguy has joined #openstack-infra | 15:16 | |
| *** dangers_away is now known as dangers | 15:16 | |
| sdague | dhellmann: sure | 15:17 |
| fungi | hogepodge: ironically, ^ is going to fail on infra's separate puppet-lint job | 15:17 |
| fungi | hogepodge: i'm trying to set up a way to test whether https://review.openstack.org/82817 will fix our image builds | 15:17 |
| dhellmann | sdague: yesterday you said the nova devs can't run unit tests on a devstack box, but I've never had that issue -- is that an active problem? | 15:18 |
| miqui | mordred, fungi: thanks... | 15:18 |
| sdague | dhellmann: yes | 15:19 |
| sdague | dhellmann: dansmith might have the reproduce | 15:19 |
| anteaya | hogepodge: in 82844 where is the bundle.config file coming from that is used when bundle install is run? | 15:19 |
| *** e0ne has joined #openstack-infra | 15:19 | |
| dansmith | dhellmann: yeah, it's usually because our default unit test environment has site_packages=False, and there's some failure with the oslo.config that devstack installs on the system | 15:20 |
| *** nkinder has joined #openstack-infra | 15:20 | |
| dansmith | sorry, site_packages=True | 15:20 |
| *** changbl has joined #openstack-infra | 15:20 | |
| dansmith | setting to site_packages=False makes it work because it gets its own oslo.config | 15:21 |
| dhellmann | dansmith: when it happens, which oslo lib can't be imported? | 15:21 |
| *** eharney has joined #openstack-infra | 15:21 | |
| dansmith | dhellmann: oslo.config | 15:21 |
| dhellmann | when the failure happens, is there a copy of oslo.config in the venv at all? | 15:21 |
| dansmith | dhellmann: well, of course it's not happening right now :/ | 15:22 |
| dansmith | dhellmann: I've been using our new site=False env for a couple weeks, let me rebuild the py27 one and see if I can get it to happen | 15:22 |
| dhellmann | yesterday I messed around with copying libvirt into the virtualenv and setting sitepackages=False, but had some (I think) unrelated failures | 15:22 |
| dansmith | dhellmann: but no, IIRC, oslo.config is not in the venv when this happens | 15:22 |
| dhellmann | dansmith: ok, there's also an option to pip install to tell it to *force* installation, so we might want to try that | 15:23 |
| dhellmann | that way all of nova's actual dependencies would be in the virtualenv, even if there are copies installed globally | 15:23 |
| dansmith | dhellmann: that seems like a good safety measure anyway, what's the downside? | 15:23 |
| dhellmann | dansmith: pip install -U --force-reinstall | 15:23 |
| dhellmann | dansmith: well, for a new virtualenv, nothing AFAICT | 15:24 |
| dansmith | sdague: ? | 15:24 |
| sdague | dhellmann: so won't that bork us on libvirt again? | 15:24 |
| *** mestery has quit IRC | 15:24 | |
| dhellmann | sdague: no, because you'd have sitepackages=True still | 15:24 |
| sdague | right, but you --force-reinstall | 15:24 |
| dhellmann | is libvirt a pip requirement of nova? | 15:25 |
| sdague | dhellmann: no | 15:26 |
| sdague | so maybe that works | 15:26 |
| sdague | would it be safer to just do that for the olso libs? | 15:26 |
| dhellmann | I'm not sure how we could | 15:26 |
| dhellmann | I'm not even 100% sure we can have tox do it in general | 15:26 |
| *** freyes has joined #openstack-infra | 15:27 | |
| dhellmann | if not, maybe we can add that feature | 15:27 |
| sdague | just put it in commands | 15:27 |
| dhellmann | hmm | 15:27 |
| dhellmann | that would do it every time the tests run, which would be an annoying delay locally | 15:27 |
| *** mgagne has joined #openstack-infra | 15:27 | |
| sdague | even if it was in testenv:venv? | 15:28 |
| sdague | I guess we could be extra clever and try to import, if fail, install | 15:28 |
| sdague | just add couple more layers of duct tape on this one | 15:29 |
| dhellmann | sdague: yeah, we could script some of that -- we would want to force install using the requirements.txt settings anyway | 15:29 |
| dhellmann | yeah | 15:29 |
| dhellmann | I need to experiment some more with non-namespaced packages to see if we have similar issues -- I know the imports don't break, but if we end up with the 'wrong" version of a lib that could also cause issues. | 15:30 |
| dhellmann | for example, we need to make sure that changes to projects don't *require* HEAD of an unreleased lib | 15:30 |
| *** dizquierdo_afk is now known as dizquierdo | 15:31 | |
| *** sabari1 has joined #openstack-infra | 15:32 | |
| sdague | sure | 15:32 |
| dhellmann | ok, I think I have enough info to run some more tests today -- thanks sdague & dansmith | 15:32 |
| openstackgerrit | Chris Hoge proposed a change to openstack-infra/config: Wraps running of puppet-lint in Bundler for version pinning. https://review.openstack.org/82844 | 15:32 |
| sdague | dhellmann: thanks for diving into this | 15:32 |
| dansmith | dhellmann: I just repro'd | 15:33 |
| dansmith | dhellmann: if you want anything | 15:33 |
| *** hashar has joined #openstack-infra | 15:33 | |
| dhellmann | sdague: yeah, if we need to change, I need to know before we start making more libs :-) | 15:33 |
| dhellmann | dansmith: I'm running devstack on a clean box now. Then I just run tox in nova? | 15:33 |
| dansmith | dhellmann: http://paste.openstack.org/show/74262/ | 15:33 |
| dansmith | dhellmann: yep | 15:33 |
| dhellmann | dansmith: ok, I'll be doing that later today | 15:33 |
| dansmith | dhellmann: cool, thanks a bunch! | 15:33 |
| *** dcramer_ has quit IRC | 15:34 | |
| *** wchrisj has joined #openstack-infra | 15:35 | |
| *** sabari1 has quit IRC | 15:35 | |
| *** sabari1 has joined #openstack-infra | 15:35 | |
| *** VijayTripathi has joined #openstack-infra | 15:35 | |
| *** hogepodge_ has joined #openstack-infra | 15:35 | |
| *** sabari1 has joined #openstack-infra | 15:35 | |
| *** wchrisj has quit IRC | 15:35 | |
| *** hashar_ has quit IRC | 15:36 | |
| *** wchrisj has joined #openstack-infra | 15:36 | |
| *** CaptTofu has joined #openstack-infra | 15:37 | |
| *** mgagne1 has joined #openstack-infra | 15:38 | |
| *** hashar_ has joined #openstack-infra | 15:38 | |
| *** emagana has joined #openstack-infra | 15:38 | |
| *** sabari1 is now known as sabari | 15:39 | |
| *** mgagne has quit IRC | 15:40 | |
| hogepodge_ | fungi I guess the rake update was a wider hit. | 15:40 |
| fungi | hogepodge_: seems that way :/ | 15:41 |
| *** hashar has quit IRC | 15:41 | |
| mgagne1 | fungi: can rake be pinned in openstack-infra/config ? | 15:41 |
| fungi | mgagne1: yeah, that's what i'm testing | 15:41 |
| *** krtaylor has quit IRC | 15:42 | |
| mgagne1 | fungi: I see ruby1.9 is installed, can we use that instead? | 15:43 |
| mgagne1 | fungi: https://github.com/openstack-infra/config/blob/master/modules/jenkins/manifests/params.pp#L138 | 15:43 |
| fungi | i finally got my test environment working to the point where it encounters that installation error but continues puppeting like the image builds are doing, so now i should be able to try the pin | 15:43 |
| *** rossella_ has quit IRC | 15:44 | |
| pleia2 | good morning | 15:44 |
| *** CaptTofu has quit IRC | 15:44 | |
| *** alexpilotti has joined #openstack-infra | 15:44 | |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Documented module functions. https://review.openstack.org/82849 | 15:44 |
| *** rossella_ has joined #openstack-infra | 15:45 | |
| mgagne1 | fungi: I suspect something is pulling rake from rubygems, I don't see it explicitly installed | 15:46 |
| fungi | mgagne1: yeah, it's pulled indirectly by the spec helper | 15:46 |
| mgagne1 | fungi: that's what I'm looking at =) | 15:46 |
| *** alexpilotti has quit IRC | 15:46 | |
| *** mgagne1 is now known as mgagne | 15:46 | |
| *** markwash has joined #openstack-infra | 15:46 | |
| fungi | mgagne: that much has been figured out for sure, just working on confirming whether https://review.openstack.org/82817 will have the effect we're hoping for | 15:47 |
| *** SumitNaiksatam has quit IRC | 15:47 | |
| *** hashar_ has quit IRC | 15:49 | |
| fungi | since it's going to have to get crammed through gating forcibly | 15:49 |
| *** krtaylor has joined #openstack-infra | 15:50 | |
| jeblair | fungi: if it's very problematic, should we consider a nodepool image-delete? | 15:50 |
| rcarrillocruz | hey fungi, regarding the initial gerrit group request bug for openstackdroid. I'm a bit confused, do I need to do anything to allow bug reporting on Launchpad? not sure from now on what we have to do for new projects, since it seems we are ditching launchpad for storyboard... | 15:51 |
| mgagne | fungi: right, I didn't see that change yet | 15:51 |
| *** thuc_ has quit IRC | 15:51 | |
| fungi | jeblair: good point--i didn't think about rolling back the nodepool images | 15:51 |
| *** thuc has joined #openstack-infra | 15:51 | |
| fungi | jeblair: i'll do that real quick | 15:51 |
| *** jergerber has joined #openstack-infra | 15:54 | |
| fungi | okay, deleted any bare-precise images built in the past 24 hours, so hopefully the remaining nodes spun up from those will be out of rotation shortly | 15:55 |
| *** ominakov has quit IRC | 15:55 | |
| *** ominakov has joined #openstack-infra | 15:55 | |
| *** thuc has quit IRC | 15:56 | |
| *** amotoki_ has joined #openstack-infra | 15:56 | |
| mgagne | hogepodge: I would like to add the possibility to override default behaviour of puppet-lint job where system gems packages are used. Not sure how yet, but that would be nice. =) | 15:57 |
| *** packet has quit IRC | 15:57 | |
| mgagne | hogepodge: maybe checking if Gemfile exists, if so, use bundler to install gems and run puppet-lint wrapped in bundler, otherwise use system packages and run puppet-lint without wrapping it with bundler. | 15:58 |
| anteaya | hogepodge_: where is the bundle.config file coming from that bundle install uses in https://review.openstack.org/#/c/82844/1/modules/openstack_project/files/jenkins_job_builder/config/puppet-module-jobs.yaml | 15:58 |
| fungi | mgagne: if we're only using rake and friends for the openstack-infra/config puppet-lint job, then maybe once things are back on track we can start using the bundler solution the stackforge-puppet-.* projects are running | 15:58 |
| mgagne | anteaya: from nowhere, from my review: "We should coordinate efforts with them to get this fixed. They don't use Bundler to manage Ruby dependencies like we do." | 15:59 |
| *** lcheng_ has joined #openstack-infra | 15:59 | |
| *** packet has joined #openstack-infra | 15:59 | |
| *** mestery has joined #openstack-infra | 15:59 | |
| mgagne | fungi: right | 15:59 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Finish initial docs https://review.openstack.org/82854 | 16:00 |
| anteaya | okay just as a heads up using bundler is like using rvm | 16:00 |
| mgagne | anteaya: not sure it can be compared to rvm tbh | 16:00 |
| anteaya | which are rubisms to get around the poor relationship with distro package managers | 16:00 |
| anteaya | mgagne: I'm not comparing them | 16:00 |
| fungi | mgagne: which would make me happy because it's several fewer things we have to install system-wide by default on all of the bare-precise job workers without using distro packages | 16:00 |
| mgagne | anteaya: what about requirements.txt? =) | 16:01 |
| anteaya | I am saying they are in the category of things ruby uses as a way of not having to deal with distro packages | 16:01 |
| anteaya | mgagne: what about requirements.txt | 16:01 |
| anteaya | ruby has a long standing bad relationship with distro package managers | 16:01 |
| fungi | anteaya: yeah, it's similar to how we run python jobs in virtualenvs most of the time (solves a related class of problems) | 16:01 |
| anteaya | the ruby community built tools to deal with it | 16:02 |
| mgagne | anteaya: that's what I meant ^ | 16:02 |
| *** Sukhdev has joined #openstack-infra | 16:02 | |
| anteaya | right | 16:02 |
| anteaya | so as long as you are aware that that is the background motivation of bundler | 16:02 |
| anteaya | and that there are other tools like that | 16:02 |
| anteaya | you won't be surprised by what happens should you chose to use them | 16:02 |
| *** dcramer_ has joined #openstack-infra | 16:02 | |
| *** hogepodge_ has quit IRC | 16:03 | |
| mgagne | anteaya: irregardless of the relationship health with distro package manager, python uses the same (similar) virtual env system so. | 16:03 |
| anteaya | as in the tools expect you to use other tools, not distro packages | 16:03 |
| anteaya | great | 16:03 |
| anteaya | so you are fully cognisant | 16:03 |
| anteaya | which was my goal | 16:03 |
| mgagne | anteaya: I learned a new word =) | 16:03 |
| fungi | i wonder if it would make sense to absorb/duplicate the bundler invocation/configuration for these jobs into the projects on which they're being run, so that changes to them become self-testing | 16:03 |
| anteaya | mgagne: I make up new words all the time | 16:04 |
| anteaya | part of what I have to accept being such a poor speller | 16:04 |
| notmyname | clarkb: are we good on the swift/syslog/ceilometer/log level/logstash issue for now? is there anything else you need from us? | 16:04 |
| *** SumitNaiksatam has joined #openstack-infra | 16:04 | |
| fungi | similar to how we use tox (and so changes to the tox.ini in a project get tested directly by the jobs they configure) | 16:04 |
| mgagne | anteaya: haha it wasn't sarcasm | 16:04 |
| anteaya | mgagne: oh okay | 16:04 |
| anteaya | I never know | 16:05 |
| anteaya | been ridiculed for my poor spelling a lot | 16:05 |
| anteaya | just have come to accept it as fact | 16:05 |
| mgagne | anteaya: I'm not a native English speaker, I had to look it up ;) | 16:05 |
| anteaya | oh okay | 16:05 |
| anteaya | glad to share some vocabulary with you | 16:06 |
| anteaya | fungi has a great vocabulary as well | 16:06 |
| fungi | notmyname: very short sample period so far, unfortunately, and we're fresh out of the compression cronjob from the weekend, but it does at least *look* like everyone's efforts on curtailing log volume is helping... http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 16:06 |
| *** Sukhdev has quit IRC | 16:06 | |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Documented module functions. https://review.openstack.org/82849 | 16:07 |
| fungi | growth rate over the past couple days looks less steep than it was previously | 16:07 |
| mgagne | fungi: puppet-lint target will have to be invoked somehow. I don't know how far we can push it back into the project. | 16:07 |
| *** freyes has quit IRC | 16:07 | |
| anteaya | fungi: to my eyes, I see it dip | 16:07 |
| fungi | mgagne: yeah, it was probably wishful thinking on my part, but it's good to strive for self-testing configuration where possible | 16:08 |
| zaro | morning | 16:08 |
| mgagne | fungi: I don't fully understand the self-testing part. Are you referring to infra/config itself? | 16:08 |
| notmyname | fungi: ok. my understanding is that everything is better now, but clarkb had given the impression that reconfiguring syslog to put the right logs in the right places wasn't a good solution in his opinion. (this is after ceilometer reduces the log level) | 16:08 |
| notmyname | fungi: but we also felt that there wasn't really a resolution to the issue of double-logging. so I wanted to check in... | 16:09 |
| fungi | mgagne: i'm referring to any job configuration which can be extracted into the project on which it's being run (openstack-infra/config is a bit of a meta-problem in this regard) | 16:09 |
| *** mestery has quit IRC | 16:09 | |
| *** SumitNaiksatam has quit IRC | 16:10 | |
| *** mestery has joined #openstack-infra | 16:10 | |
| mgagne | fungi: Are you suggesting moving JJB job template into the project? | 16:10 |
| *** SumitNaiksatam has joined #openstack-infra | 16:10 | |
| *** hogepodge_ has joined #openstack-infra | 16:10 | |
| fungi | mgagne: no, just suggesting that, as an analogy, our python jobs rely on the projects to declare what versions of tools they wanted to be tested with and how they should be configured | 16:10 |
| *** rhsu has joined #openstack-infra | 16:11 | |
| mgagne | fungi: oh, gotcha. 100% agree with you | 16:11 |
| hogepodge_ | anteya I’m assuming your q was answered? I dropped offline for a bit | 16:11 |
| fungi | and if we're using tools which can be installed by unprivileged users and run locally, instead of needing root access to get them installed system-wide, that becomes easier to accomplish sometimes | 16:12 |
| zaro | mordred: yes, it's ok to run puppet on jenkins-dev.o.o | 16:12 |
| mordred | zaro: thanks! | 16:12 |
| *** khyati has joined #openstack-infra | 16:12 | |
| anteaya | hogepodge_: yes mgagne points out that there is no bundle.config for bundle install to use | 16:12 |
| anteaya | since -infra/config doesn't use bundler currently | 16:12 |
| *** luisg has quit IRC | 16:12 | |
| anteaya | morning zaro | 16:13 |
| hogepodge_ | Gemfile usually manages that | 16:13 |
| hogepodge_ | but yes, it’s probably not appropriate | 16:13 |
| mordred | zaro: ok. it's reporting now: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Must pass mysql_password to Class[Openstack_project::Nodepool] at /opt/config/production/modules/openstack_project/manifests/nodepool.pp:3 on node jenkins-dev.openstack.org | 16:13 |
| anteaya | hogepodge_: is there a Gemfile? | 16:13 |
| anteaya | and yes, you are correct Gemfile not bundle.config, it has been a while | 16:14 |
| *** skraynev_afk is now known as skraynev | 16:14 | |
| *** nati_ueno has joined #openstack-infra | 16:17 | |
| *** Steap_ is now known as Steap | 16:18 | |
| *** khyati has quit IRC | 16:18 | |
| *** khyati has joined #openstack-infra | 16:19 | |
| *** hashar has joined #openstack-infra | 16:20 | |
| *** atiwari has joined #openstack-infra | 16:20 | |
| *** markwash has quit IRC | 16:20 | |
| *** mihgen has quit IRC | 16:21 | |
| mgagne | hogepodge_: mind if I propose something else? | 16:21 |
| *** dkranz has quit IRC | 16:22 | |
| zaro | mordred: how come i don't see the report on web page? | 16:23 |
| *** vkozhukalov has quit IRC | 16:23 | |
| *** markwash has joined #openstack-infra | 16:23 | |
| mordred | zaro: because it was unsuccessful | 16:23 |
| mordred | zaro: if you want to run it - just become root and then run "puppet agent --test" | 16:24 |
| zaro | mordred: ahh. i'll take a look at fixing that. | 16:24 |
| *** medieval1 has quit IRC | 16:25 | |
| zaro | mordred: did your puppet run wipe out local changes i that i made on jenkins-dev? | 16:25 |
| *** thuc has joined #openstack-infra | 16:25 | |
| *** thuc has quit IRC | 16:26 | |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/config: Add ability to use Bundler to run puppet-lint https://review.openstack.org/82859 | 16:26 |
| *** thuc has joined #openstack-infra | 16:26 | |
| mgagne | hogepodge_: Our conversation got split between 2 channels but here is the change I mentionned | 16:27 |
| hogepodge_ | I like it. ^^ | 16:27 |
| *** medieval1 has joined #openstack-infra | 16:27 | |
| *** andreaf has quit IRC | 16:27 | |
| hogepodge_ | +1’ed it | 16:28 |
| mgagne | hogepodge_: cool thanks | 16:28 |
| pabelanger | If I setup a target.image in nodepool to have a min-ready of 0, should I expect jenkins to tell nodepool to launch one when a new job is published to the queue? | 16:29 |
| *** nati_ueno has quit IRC | 16:30 | |
| pabelanger | My logic for min-ready = 0 is, i have some nodes that I don't expect to be too active. So, rather them sitting around on the cloud (eating up my budget), it only expect them to launch once jenkins needed to test it. | 16:31 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gearman-plugin: change logging level from info to debug https://review.openstack.org/82627 | 16:33 |
| *** luisg has joined #openstack-infra | 16:33 | |
| mordred | zaro: I do not believe it actually ran | 16:34 |
| *** dkranz has joined #openstack-infra | 16:35 | |
| *** wenlock_ has joined #openstack-infra | 16:35 | |
| *** wenlock has quit IRC | 16:35 | |
| *** wenlock_ has quit IRC | 16:35 | |
| mordred | zaro: as in - it failed in the compilation step, so it did not affect any change | 16:35 |
| *** wenlock has joined #openstack-infra | 16:35 | |
| *** hogepodge_ has quit IRC | 16:36 | |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Documented module functions. https://review.openstack.org/82849 | 16:38 |
| zaro | mordred: oops, i don't have ssh access to jenkins-dev so i cant test out my changes. i'll just post on gerrit. | 16:38 |
| *** adalbas has joined #openstack-infra | 16:38 | |
| *** UtahDave has joined #openstack-infra | 16:39 | |
| fungi | okay, after much gnashing of teeth, i've confirmed that https://review.openstack.org/82817 should fix the gem install of rake on our nodepool bare-precise images the next time they're updated | 16:40 |
| fungi | any objections to manually submitting that and bypassing gate jobs? | 16:40 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: disable the jeepyb trivial rebase hook on review.o.o https://review.openstack.org/82416 | 16:40 |
| fungi | oh, wait, shouldn't need to now | 16:40 |
| *** bauzas has quit IRC | 16:40 | |
| fungi | the broken slaves are likely out of rotation and back on old images | 16:41 |
| *** rhsu has quit IRC | 16:41 | |
| mordred | fungi: I approve of the patch | 16:41 |
| fungi | long term, let's join forces with the stackforge/puppet-.* crowd and start running their lint job, if we can | 16:42 |
| *** bhuvan has joined #openstack-infra | 16:44 | |
| anteaya | glad 82817 works | 16:45 |
| *** markwash_ has joined #openstack-infra | 16:48 | |
| mordred | fungi: ++ | 16:48 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add per-test database fixture https://review.openstack.org/82865 | 16:48 |
| *** markwash has quit IRC | 16:49 | |
| *** markwash_ is now known as markwash | 16:49 | |
| jeblair | mordred: can you take a look at that and tell me if that's the thing we keep talking about but no one has written yet ^ ? | 16:49 |
| jeblair | clarkb: when you're around ^ | 16:49 |
| mordred | I will look | 16:49 |
| mordred | jeblair: yes. thank you for writing that. it's possible it may exist in oslo now - or something much more complex | 16:50 |
| dhellmann | dansmith, sdague : I was unable to reproduce the nova unit tests issue on a fresh system. Perhaps there's a step I'm missing? | 16:51 |
| dansmith | dhellmann: I don't have a fresh system handy to try it (could spin something up of course), but is there anything I can look at on my system to debug? | 16:52 |
| *** jpeeler has joined #openstack-infra | 16:52 | |
| *** jpeeler has quit IRC | 16:52 | |
| *** jpeeler has joined #openstack-infra | 16:52 | |
| jeblair | mordred: do you know if we have grant option set on our slaves? | 16:52 |
| dhellmann | dansmith: ".tox/py27/bin/pip freeze | grep oslo" and "ls .tox/py27/lib/python2.7/site-packages/oslo" | 16:52 |
| mordred | jeblair: I believe taht we do - clarkb ? | 16:54 |
| dansmith | dhellmann: http://paste.openstack.org/show/74268/ | 16:54 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add per-test database fixture https://review.openstack.org/82865 | 16:54 |
| fungi | jeblair: recent change gave full administrative rights to the citest account over the entire mysql database i thought. checking the commit log now | 16:55 |
| *** rossella_s has joined #openstack-infra | 16:56 | |
| jeblair | Grant_priv: Y | 16:56 |
| jeblair | assuming that means what i think it means (mordred?) looks like yes... | 16:56 |
| *** talluri has joined #openstack-infra | 16:56 | |
| dhellmann | dansmith: interesting, I don't have rootwrap installed in my virtualenv | 16:57 |
| dhellmann | dansmith: it's installed globally | 16:57 |
| dansmith | dhellmann: I have it installed globally too | 16:57 |
| fungi | jeblair: it was in https://review.openstack.org/76634 about a month ago | 16:57 |
| fungi | privileges => ['all'] | 16:58 |
| jeblair | fungi: yeah, i wasn't quite sure because grant is handled differently | 16:58 |
| fungi | ohh... got it | 16:59 |
| jeblair | fungi: in sql, you have to say "grant all .... with grant option" | 16:59 |
| *** jerryz has joined #openstack-infra | 16:59 | |
| jeblair | (technically that's not sql but you know what i mean) | 16:59 |
| dhellmann | dansmith: you have it installed globally from a package, not from source -- devstack should be installing it from source now | 16:59 |
| *** dkliban is now known as dkliban_afk | 16:59 | |
| dansmith | dhellmann: pip freeze shows it installed from git, per devstack | 16:59 |
| dhellmann | dansmith: line 4 of your pastebin implies otherwise | 17:00 |
| *** jcoufal has quit IRC | 17:00 | |
| dhellmann | ah, wait | 17:00 |
| dhellmann | that's in the virtualenv | 17:00 |
| dhellmann | well that's interesting | 17:00 |
| dansmith | right | 17:00 |
| zaro | mordred: fix for puppet has been on gerrit for a while: realize ( | 17:00 |
| zaro | User::Virtual::Localuser['zaro'], | 17:00 |
| zaro | ) | 17:00 |
| zaro | opps. | 17:00 |
| zaro | mordred: fix for puppet is here: https://review.openstack.org/#/c/61321 | 17:01 |
| dhellmann | dansmith: which version is installed globally? | 17:01 |
| *** markmcclain1 has joined #openstack-infra | 17:01 | |
| *** markmcclain1 has quit IRC | 17:01 | |
| dansmith | dhellmann: http://paste.openstack.org/show/74269/ | 17:01 |
| *** markmcclain1 has joined #openstack-infra | 17:02 | |
| *** markmcclain has quit IRC | 17:02 | |
| *** nati_ueno has joined #openstack-infra | 17:02 | |
| clarkb | good morning | 17:03 |
| *** harlowja has joined #openstack-infra | 17:03 | |
| clarkb | jeblair: mordred: on the single use slaves mysql grants all beause we are throwing away the machine anyways | 17:04 |
| *** rossella_ has quit IRC | 17:04 | |
| clarkb | jeblair: mordred postgres doesn't need that because any schema you create in postgres you own and can do whatever you want with (with the way we have acls setup there) | 17:04 |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Documented module functions. https://review.openstack.org/82849 | 17:04 |
| *** morganfainberg_Z is now known as morganfainberg | 17:04 | |
| dhellmann | dansmith: which os and version of pip? | 17:05 |
| dansmith | dhellmann: http://paste.openstack.org/show/74271/ | 17:05 |
| clarkb | notmyname: I don't think syslog is a good general solution for devstack in the gate | 17:06 |
| clarkb | notmyname: I think for swift it is the only solution aiui | 17:06 |
| dhellmann | dansmith: ok, same here | 17:06 |
| dansmith | dhellmann: hrm | 17:06 |
| dhellmann | yeah | 17:06 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Remove legacy priority column from stories table https://review.openstack.org/82504 | 17:06 |
| *** rhsu has joined #openstack-infra | 17:09 | |
| *** dcramer_ has quit IRC | 17:09 | |
| *** rediskin has joined #openstack-infra | 17:09 | |
| dhellmann | dansmith: I'll keep trying | 17:10 |
| dansmith | dhellmann: thanks, I hate to be a bother, but several people have hit this now and again, I seem to hit it constantly | 17:11 |
| dansmith | dhellmann: I think most folks run devstack and unit tests on different boxes to avoid it | 17:11 |
| dhellmann | dansmith: yeah, they shouldn't have to do that -- I didn't realize this was a constant problem, I thought we had it worked out a while back. | 17:11 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/storyboard: Use MySQL for unit tests https://review.openstack.org/82872 | 17:11 |
| *** Ryan_Lane has joined #openstack-infra | 17:11 | |
| dansmith | dhellmann: well, surely appreciate the help :) | 17:12 |
| jeblair | mordred, krotscheck: a starting point if we want to use mysql for unit tests: https://review.openstack.org/82872 | 17:12 |
| dansmith | dhellmann: an I really hope it's not something I'm doing wrong :D | 17:12 |
| dhellmann | dansmith: there's obviously some difference, but it's not clear what that is, yet -- either way we should have devstack and tox set up to work without special care on your part :-) | 17:13 |
| dansmith | dhellmann: yeah, cool :) | 17:13 |
| dhellmann | dansmith: I don't suppose you have a box where this problem exists that I can get a login on? | 17:13 |
| dhellmann | dansmith: would cut down on the 20-questions style debugging :-) | 17:14 |
| dansmith | dhellmann: well, it's definitely not a public box, but I could arrange it I guess, if you promise to be good :) | 17:14 |
| clarkb | jeblair: reviewed the db test fixture change | 17:14 |
| * dhellmann crosses his heart | 17:14 | |
| *** nicedice has joined #openstack-infra | 17:15 | |
| jeblair | clarkb: on determinism -- isn't the fixture cleanup going to make sure we never have a db to inspect after a failure? | 17:16 |
| clarkb | jeblair: ya, it will so maybe that isn't quite so important | 17:17 |
| clarkb | jeblair: we could possibly check an env flag that doesn't do the cleanup if set | 17:17 |
| clarkb | jeblair: but maybe we optimize for that case if it becomes necessary | 17:17 |
| jeblair | clarkb: so another thing to consider is that db names are limited to 16 chars | 17:18 |
| jeblair | and i'm pretty sure the charset is limited | 17:18 |
| jeblair | so it could be tricky to build the string with the test name (especially with scenarios) | 17:19 |
| clarkb | oh I did not realize that. In that case I think what you have is a good solution | 17:19 |
| clarkb | jeblair: what if you did 8 random chars + pid or similar to avoid the relatively small chance of a name collision? | 17:19 |
| *** khyati has quit IRC | 17:19 | |
| fungi | alternatively, fold the test name into a 128-bit hash and use the hex version of it | 17:19 |
| jeblair | clarkb: if you wanted to keep a db around, maybe the thing to do would be to output the name of the db (the dburi even) so you'd know which one | 17:20 |
| *** mriedem has quit IRC | 17:20 | |
| jeblair | you could add it as a detail too | 17:21 |
| clarkb | jeblair: ++ | 17:21 |
| *** zns has quit IRC | 17:21 | |
| fungi | really you could just truncate an md5 hash or similar to 16 hex digits for that matter--deterministic and unlikely to collide. you wouldn't really need a special 128-bit hash construct | 17:22 |
| openstackgerrit | Joseph Matthew Ching proposed a change to openstack-infra/config: Added pep8 to check and gate jobs https://review.openstack.org/82877 | 17:23 |
| clarkb | fungi: right or you could do something that is garunteed to not collide | 17:23 |
| clarkb | fungi: we run a lot of tests but probably not enough to see random uuid generation collide | 17:23 |
| clarkb | fungi: unless (and I have no idea how the entropy is used) the entropy collection is terrible and two python processes running at the same time have a much higher chance of colliding | 17:24 |
| fungi | i meant as far as getting a 16-character database name which should be unique per test. picking numbers at random is not really any more or less likely to collide than a properly-designed cryptographic hash (distribution is one of the properties they design for, after all) | 17:25 |
| *** andreaf has joined #openstack-infra | 17:25 | |
| *** khyati has joined #openstack-infra | 17:25 | |
| fungi | but yes, if your pid is already guaranteeing uniqueness per test, then that works to discriminate them too | 17:25 |
| jeblair | uh, right, so how do you think we should construct the name? :) | 17:28 |
| jeblair | fungi, clarkb: ^ your final answers? :) | 17:28 |
| clarkb | jeblair: I think if we do randombits + pid we will get non colliding names across test runners | 17:28 |
| clarkb | and it shouldn't be hard to do ^ so probably is a good idea | 17:29 |
| *** amotoki_ has quit IRC | 17:29 | |
| fungi | i am fine with pid+rand | 17:29 |
| *** salv-orlando has joined #openstack-infra | 17:30 | |
| *** skraynev is now known as skraynev_afk | 17:32 | |
| *** gyee has joined #openstack-infra | 17:32 | |
| *** VijayTripathi has quit IRC | 17:33 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add per-test database fixture https://review.openstack.org/82865 | 17:33 |
| jeblair | done | 17:34 |
| jeblair | clarkb: also, see comments in review; interpolation isn't straightforward either | 17:34 |
| *** bhuvan has quit IRC | 17:34 | |
| *** zns has joined #openstack-infra | 17:35 | |
| *** Sukhdev has joined #openstack-infra | 17:35 | |
| *** gothicmindfood has quit IRC | 17:35 | |
| *** e0ne has quit IRC | 17:36 | |
| *** markmcclain1 has quit IRC | 17:36 | |
| jeblair | clarkb: http://logs.openstack.org/72/82872/1/check/gate-storyboard-python27/317eb89/testr_results.html.gz | 17:37 |
| jeblair | clarkb: (storyboard has tests running but nodepool doesn't so that's the only place we can see this in action on the slaves... | 17:37 |
| jeblair | it looks like the db doesn't exist.. i wonder why? | 17:38 |
| *** khyati has quit IRC | 17:38 | |
| clarkb | jeblair: is it because your db connection in the fixture already specifies a DB? | 17:39 |
| *** dizquierdo has quit IRC | 17:39 | |
| clarkb | so create database fails in that context? | 17:39 |
| jeblair | clarkb: i don't follow | 17:39 |
| clarkb | if you connect to a mysqld giving it a schema argument is create database valid in that "scope"? (disclaimer I have no idea what I am talking about) | 17:40 |
| jeblair | clarkb: oh, i think you have to connect to a db | 17:41 |
| *** flaper87 is now known as flaper87|afk | 17:41 | |
| clarkb | jeblair: I see the thing about interpolation, and since we are generarting the name it is safe there | 17:42 |
| jeblair | clarkb: at any rate, the nodepool version works locally, i'm trying to sift through errors in the storyboard version (there are expected errors because storyboards db tests only work with sqlite) | 17:42 |
| clarkb | jeblair: maybe try bumping the sqlalchemy debug level to debug, it should capture more of what it is doing before the test runs | 17:43 |
| clarkb | jeblair: the self.dburi value doesn't include the password could that be it? | 17:44 |
| *** UtahDave has quit IRC | 17:46 | |
| *** mriedem has joined #openstack-infra | 17:46 | |
| *** markmcclain has joined #openstack-infra | 17:46 | |
| *** hashar has quit IRC | 17:47 | |
| *** SumitNaiksatam has quit IRC | 17:47 | |
| clarkb | jeblair: another thing that may be nice to add to the fixture is a "No mysql available throw skiptest exception" feature | 17:48 |
| *** mestery_ has joined #openstack-infra | 17:48 | |
| *** mestery has quit IRC | 17:48 | |
| *** amcrn has joined #openstack-infra | 17:49 | |
| *** dstanek has quit IRC | 17:49 | |
| *** signed8bit has quit IRC | 17:49 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:50 | |
| *** mestery_ is now known as mestery | 17:50 | |
| jeblair | clarkb: i have no idea how to set the sqlalchemy debug logging in the context of these unit tests | 17:51 |
| jeblair | nothing is what it's supposed to be | 17:51 |
| *** dcramer_ has joined #openstack-infra | 17:51 | |
| jeblair | logging.getLogger('sqlalchemy.engine').setLevel(logging.DEBUG) | 17:51 |
| jeblair | AttributeError: 'ContextAdapter' object has no attribute 'setLevel' | 17:51 |
| jeblair | oh! | 17:51 |
| jeblair | they import a different logging module as logging | 17:51 |
| jeblair | ugh :( | 17:51 |
| clarkb | :/ that isn't very friendly | 17:51 |
| *** melwitt has joined #openstack-infra | 17:52 | |
| *** lcheng_ has quit IRC | 17:53 | |
| *** gothicmindfood has joined #openstack-infra | 17:53 | |
| *** signed8bit has joined #openstack-infra | 17:54 | |
| *** signed8bit has quit IRC | 17:54 | |
| *** signed8bit has joined #openstack-infra | 17:55 | |
| *** derekh has quit IRC | 17:55 | |
| *** jgallard has quit IRC | 17:56 | |
| jeblair | clarkb: i think something deep in storyboard's db abstraction (possibly from oslo) is caching the db | 17:56 |
| jeblair | clarkb: the dburi detail (yay!) doesn't have the same db name as the error reported from sqlalchemy | 17:57 |
| clarkb | woo | 17:57 |
| jeblair | clarkb: so good news is it will probably work with nodepool | 17:57 |
| jeblair | bad news is it may take quite a bit of spelunking to find out why it dosn't work for storyboard | 17:58 |
| clarkb | awesome, what do you think about the skip if no mysql feature above? | 17:58 |
| *** markmcclain has quit IRC | 17:58 | |
| jeblair | clarkb: as a toggleable feature, sure; but i'm not sure why we'd want to turn it on | 17:59 |
| clarkb | jeblair: so that I can run tests on my laptop/testvm without configuring MySQL <_< | 17:59 |
| *** pbelanyi has quit IRC | 18:00 | |
| jeblair | clarkb: yeah, but we're talking about projects that _don't work_ with anything but mysql, so there's no point in running tests under that configuration. :) | 18:00 |
| *** changbl has quit IRC | 18:00 | |
| *** thedodd has quit IRC | 18:00 | |
| clarkb | jeblair: fungi mordred SergeyLukjanov I think it would be good to get the nodepool changes up to the introduction of tests merged, then merge the config change to run the tests and we can test the changes that add tests | 18:01 |
| *** dstanek has joined #openstack-infra | 18:01 | |
| *** talluri has quit IRC | 18:02 | |
| jeblair | clarkb: i think we need to go one past that to 'finish docs' because docs don't build until then | 18:03 |
| *** yolanda has quit IRC | 18:03 | |
| jeblair | wow. the oslo db code is an abstraction layer on top of sqlalchemy. | 18:03 |
| clarkb | jeblair: oh right the docs job, ok | 18:04 |
| fungi | clarkb: agreed. i keep starting to review those and then get sidetracked | 18:04 |
| clarkb | jeblair: one more comment on the fixture thing that I think was lost in the interpolation noise | 18:04 |
| davidlenwell | random non-sequitur I find it entertaining that I can -2 and +2 approve in the same review | 18:05 |
| *** dkranz has quit IRC | 18:05 | |
| fungi | +1 approve? | 18:06 |
| fungi | but yes, it's definitely possible | 18:06 |
| davidlenwell | yeah .. miss type | 18:07 |
| jeblair | clarkb: oh, yeah, so about that... it's random, right, so there should never be a conflict, even from a previous session. however, there's a small chance that the current session might have a collison... | 18:07 |
| jeblair | clarkb: i kinda think it would be a good idea to throw an error in that case, rather than risk having two tests accessing the same db | 18:07 |
| davidlenwell | I'm just trying to imagine a situation where I would -2 code .. but then approve it for merge | 18:07 |
| *** packet has quit IRC | 18:07 | |
| clarkb | jeblair: the case I was thinking of was you kill -9 the test runner preventing the cleanup from running on your laptop, then some months later you run the tests again and have a pid collision. Definitely not a problem for the gate so maybe an error is better | 18:08 |
| anteaya | it won't merge | 18:08 |
| jeblair | clarkb: yeah, even in that case, i think it's better to error so you start from a defined state | 18:08 |
| clarkb | ok that wfm | 18:09 |
| anteaya | zuul will see the -2 and not merge it even if it passes the gate tests | 18:09 |
| *** pblaho has quit IRC | 18:09 | |
| openstackgerrit | A change was merged to openstack-infra/config: Pin rake gem to 10.1.1 so it works with ruby 1.8.x https://review.openstack.org/82817 | 18:10 |
| *** _nadya_ has joined #openstack-infra | 18:10 | |
| *** vkozhukalov has joined #openstack-infra | 18:10 | |
| *** bhuvan has joined #openstack-infra | 18:10 | |
| clarkb | jeblair: any chance I can get a review on https://review.openstack.org/#/c/82581/ would like to have that running on the workers today if possible | 18:11 |
| clarkb | and I am going to the optometrist this afternoon so this morning is my best chance | 18:11 |
| *** packet has joined #openstack-infra | 18:11 | |
| *** khyati has joined #openstack-infra | 18:12 | |
| *** emagana has quit IRC | 18:12 | |
| *** emagana has joined #openstack-infra | 18:12 | |
| *** vhoward has left #openstack-infra | 18:14 | |
| sdague | davidlenwell: and through the miracle of prolog you'll fail to acheive sufficiency :) | 18:14 |
| *** jooools has quit IRC | 18:14 | |
| fungi | clarkb: jeblair: was https://review.openstack.org/82583 the approach you had in mind for fetching the specific tag related to a job from the zuul merger? | 18:15 |
| jeblair | clarkb: i'm not a fan of the repetition (i'm having trouble accepting that's a necessary thing in a language _explicitly_ designed for string manipulation)... | 18:16 |
| clarkb | AJaeger: areyou still around? I don't think we should commit .tx files automatically in https://review.openstack.org/#/c/82384/1 | 18:16 |
| clarkb | jeblair: I agree, I could make it a subroutine if I figure out how to write those and make the repitition happen once | 18:16 |
| davidlenwell | anteaya: sdague .. I didn't think zuul would merge it .. its mostly a ui issue .. | 18:17 |
| jeblair | clarkb: i left a +2 because i'm not going to look up how to do it right now and i think it's worth getting in. | 18:17 |
| clarkb | jeblair: but I couldn't find a way to completely get rid of it without scanning the entire string each time | 18:17 |
| jeblair | clarkb: i found the problem in storyboard | 18:18 |
| jeblair | new patch soon | 18:18 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/storyboard: Use MySQL for unit tests https://review.openstack.org/82872 | 18:18 |
| *** jasondotstar has quit IRC | 18:18 | |
| anteaya | davidlenwell: sometimes a patch has been approved and then for some reason after it is in the gate queue the decision is made to not merge the patch | 18:19 |
| *** dkranz has joined #openstack-infra | 18:19 | |
| anteaya | davidlenwell: pushing a new patchset snipes the patch from the gate causing a reset | 18:19 |
| *** changbl has joined #openstack-infra | 18:20 | |
| *** _nadya_ has quit IRC | 18:20 | |
| anteaya | -2'ing the patch keeps the gate queue intact and the patch doesn't merge | 18:20 |
| anteaya | not by design, but that is the behaviour which we have come to use | 18:20 |
| *** jasondotstar has joined #openstack-infra | 18:21 | |
| fungi | clarkb: for 82581 i guess the assumption on masking out 7+ digit numbers is that we won't have error codes or other useful things matching that? | 18:21 |
| fungi | 32-digit hex strings and uuids definitely seem safe | 18:21 |
| clarkb | fungi: specifically it matches the -XXXXXX suffixes we have all over the place | 18:22 |
| clarkb | fungi: slave names, tempest test names, and so on | 18:22 |
| sdague | davidlenwell: it's actually not even zuul, it's gerrit | 18:22 |
| clarkb | jeblair: woot | 18:23 |
| fungi | clarkb: ahh, missed the - part of the pattern. fot it | 18:23 |
| fungi | er, got it | 18:23 |
| *** shakamunyi has joined #openstack-infra | 18:25 | |
| *** shakamunyi has quit IRC | 18:25 | |
| *** shakamunyi has joined #openstack-infra | 18:26 | |
| *** shakamunyi has quit IRC | 18:26 | |
| openstackgerrit | A change was merged to openstack-infra/config: Don't treat IDs as uniquely special in CRM114 https://review.openstack.org/82581 | 18:27 |
| *** ildikov_ has quit IRC | 18:29 | |
| *** ominakov has quit IRC | 18:29 | |
| clarkb | woot thankyou | 18:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Use rolled-up nodepool stats https://review.openstack.org/82388 | 18:31 |
| *** ociuhandu has quit IRC | 18:31 | |
| jeblair | graphite breaths a sigh of relief ^ | 18:33 |
| jeblair | breathes even (i assume it was out of breath) | 18:33 |
| *** yassine has quit IRC | 18:33 | |
| *** johnthetubaguy has quit IRC | 18:36 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Comments UI https://review.openstack.org/82213 | 18:36 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: propose_translation_update: Handle .tx/config in git https://review.openstack.org/82384 | 18:37 |
| *** yassine has joined #openstack-infra | 18:39 | |
| *** e0ne has joined #openstack-infra | 18:41 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Task status updates https://review.openstack.org/82690 | 18:41 |
| clarkb | jeblair: fungi not sure if you saw the latest resposne to the DNSSEC thread. Basically the 206 address was an old default... | 18:47 |
| clarkb | so yay adds to the confusion | 18:47 |
| clarkb | I am going to respond asking for the feature enhancement request for DNSSEC support | 18:48 |
| *** thuc has quit IRC | 18:48 | |
| fungi | clarkb: yeah, and apparently dnssec being broken isn't a bug, merely a missing feature we might want to request they add some day | 18:48 |
| clarkb | (I didn't realize that this particular thread wasn't already implicitly one) | 18:48 |
| clarkb | fungi: right, :? | 18:48 |
| * clarkb makes it explicit | 18:48 | |
| *** thuc has joined #openstack-infra | 18:49 | |
| *** jpich has quit IRC | 18:49 | |
| *** dcramer_ has quit IRC | 18:50 | |
| *** _nadya_ has joined #openstack-infra | 18:50 | |
| *** thuc has quit IRC | 18:51 | |
| *** thuc_ has joined #openstack-infra | 18:51 | |
| *** zhiyan is now known as zhiyan_ | 18:58 | |
| *** che-arne has quit IRC | 18:59 | |
| *** derekh has joined #openstack-infra | 18:59 | |
| openstackgerrit | A change was merged to openstack-infra/config: Added pep8 to check and gate jobs https://review.openstack.org/82877 | 18:59 |
| *** _nadya_ has quit IRC | 18:59 | |
| jeblair | that probably wanted a better subject :/ | 18:59 |
| *** thedodd has joined #openstack-infra | 18:59 | |
| *** jlibosva has quit IRC | 19:00 | |
| fungi | meeting time | 19:00 |
| openstackgerrit | A change was merged to openstack-infra/config: Add kite notifications to the barbican channel https://review.openstack.org/82660 | 19:00 |
| clarkb | jeblair: :/ sorry | 19:00 |
| clarkb | the commit message does capture it properly but not the subject | 19:00 |
| jeblair | clarkb: nbd, just talking out loud. :) | 19:01 |
| *** zhiyan_ is now known as zhiyan | 19:03 | |
| *** rhsu has quit IRC | 19:04 | |
| *** _nadya_ has joined #openstack-infra | 19:05 | |
| *** bhuvan has quit IRC | 19:06 | |
| *** markwash has quit IRC | 19:08 | |
| *** zns has quit IRC | 19:09 | |
| *** dkranz has quit IRC | 19:10 | |
| *** SumitNaiksatam has quit IRC | 19:11 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:13 | |
| *** dcramer_ has joined #openstack-infra | 19:13 | |
| *** openstackgerrit has quit IRC | 19:18 | |
| *** openstackgerrit has joined #openstack-infra | 19:18 | |
| *** denis_makogon_ has joined #openstack-infra | 19:21 | |
| *** denis_makogon_ is now known as denis_makogon | 19:22 | |
| *** dkranz has joined #openstack-infra | 19:22 | |
| *** zul has quit IRC | 19:26 | |
| *** jp_at_hp has quit IRC | 19:27 | |
| dhellmann | sdague: https://review.openstack.org/82896 should fix the namespace package issue for nova | 19:29 |
| *** zzelle has joined #openstack-infra | 19:29 | |
| *** pblaho has joined #openstack-infra | 19:29 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/meetbot: Prevents meetbot from starting meetings in non-meeting channels https://review.openstack.org/82898 | 19:29 |
| sdague | dhellmann: cool | 19:29 |
| sdague | would be interesting to see if that fixes it for dansmith | 19:29 |
| dansmith | sdague: just about to find out | 19:30 |
| * sdague waits for a woohoo or a table flip | 19:30 | |
| *** zul has joined #openstack-infra | 19:30 | |
| dansmith | my machine is doing a live upgrade of a simulated cloud right now, so it's a scotch busy | 19:30 |
| *** pblaho has quit IRC | 19:31 | |
| pleia2 | anteaya: why would channels be using meetbot to log non-meetings? | 19:31 |
| zaro | fungi: tag already created for this https://review.openstack.org/#/c/82559/ | 19:31 |
| zaro | fungi: should i abandone? | 19:31 |
| zzelle | Hi everyone, | 19:32 |
| sdague | hmmm... scotch | 19:32 |
| pleia2 | is openstack the eavesdrop bot? | 19:32 |
| pleia2 | (I should probably know this :)) | 19:32 |
| anteaya | pleia2: we use it for logging | 19:33 |
| zzelle | I got the following error during gate tests "ls: cannot access /var/log/secure-*: No such file or directory" in http://logs.openstack.org/46/82746/1/gate/gate-neutron-python26/e2c71a6/console.html | 19:33 |
| anteaya | pleia2: if I channel wants logging we use meetbot | 19:33 |
| pleia2 | neat, for some reason I thought it was a separate logging bot | 19:33 |
| clarkb | zzelle: sounds like a neutron bug | 19:33 |
| sdague | zzelle: that's not the issue | 19:33 |
| sdague | the issue is the job timed out | 19:33 |
| pleia2 | anteaya: thanks | 19:33 |
| anteaya | The OpenStack Infrastructure team run a slightly modified Meetbot to log IRC channel activity and meeting minutes. | 19:33 |
| anteaya | np | 19:33 |
| dansmith | sdague: dhellmann: +2, works for me, thanks! | 19:33 |
| anteaya | i questioned it myself | 19:34 |
| sdague | http://logs.openstack.org/46/82746/1/gate/gate-neutron-python26/e2c71a6/console.html#_2014-03-25_18_50_04_707 | 19:34 |
| zzelle | clarkb, sdague it's only hiding the real trouble ? | 19:34 |
| sdague | zzelle: Build timed out (after 70 minutes). Marking the build as failed. | 19:34 |
| sdague | it's right there | 19:34 |
| *** Sukhdev has quit IRC | 19:35 | |
| zzelle | but it appends after the var/log error ? | 19:35 |
| *** zhiyan is now known as zhiyan_ | 19:36 | |
| zzelle | it look like a bootstrap/pre-test error ? | 19:36 |
| fungi | zaro: sure, i'm just still trying to fix gerrit-git-prep so that the job can be re-triggered successfully | 19:36 |
| *** vhoward has joined #openstack-infra | 19:37 | |
| *** hashar has joined #openstack-infra | 19:38 | |
| *** mrmartin has joined #openstack-infra | 19:38 | |
| pabelanger | Is -infra mirroring ruby gems that are used by jenkin slaves? | 19:39 |
| anteaya | pabelanger: not that I am aware | 19:39 |
| pabelanger | Hmm | 19:39 |
| pabelanger | anteaya, thanks | 19:39 |
| anteaya | why do you ask | 19:39 |
| anteaya | do you think current mirrors have stability issues? | 19:40 |
| pabelanger | maybe | 19:40 |
| * anteaya listens | 19:40 | |
| sdague | zzelle: no, the test suit hit the timeout | 19:40 |
| sdague | so was failed | 19:40 |
| pabelanger | I'm not sure if I'm having a nodepool issue building my jenkins slaves, or just the ruby mirrors are going up and down. | 19:40 |
| pabelanger | figured I check here to see if you've had issues | 19:40 |
| zzelle | sdague, but the bug /var/log/secure-* appends at the beginning ? | 19:41 |
| anteaya | we build images once per day - usually | 19:41 |
| sdague | zzelle: that's not important | 19:41 |
| pabelanger | right, so not sure if it is a timing issue right now | 19:41 |
| anteaya | and if I understand correctly we reverted to yesterday's images to address the rake issue | 19:41 |
| * anteaya nods | 19:42 | |
| zzelle | sdague, okay fine for me | 19:42 |
| *** mrmartin has quit IRC | 19:42 | |
| sdague | zzelle: this is a working log - http://logs.openstack.org/75/82775/2/check/gate-neutron-python26/e87e224/console.html | 19:42 |
| sdague | for instance | 19:42 |
| sdague | it's in every log | 19:42 |
| *** dkliban_afk is now known as dkliban | 19:43 | |
| zzelle | sdague, oups my apologies | 19:43 |
| sdague | the real issue is python 2.6 neutron is now regularly exceeding an hour to run | 19:44 |
| sdague | so the timeout is getting tripped from time to time | 19:44 |
| pabelanger | anteaya, Ya, I worked around the rake issue too. | 19:44 |
| pabelanger | okay, will continue to test | 19:44 |
| fungi | pabelanger: anteaya: right, we're running nodes from images which were built before the rake update, and i've also merged a patch which pins rake for the next image builds | 19:44 |
| zaro | fungi: thanks. | 19:44 |
| *** _nadya_ has quit IRC | 19:44 | |
| sdague | that should be brought up with the neutron team about the path forward, because we can bump the timeout again, but the unit tests really need a diet | 19:44 |
| pabelanger | fungi, but in general, you don't have issues downloading gems from ruby mirrors? | 19:45 |
| fungi | pabelanger: for infra jobs we preinstall them on the nodepool images, so they're already available in the system context. i think the stackforge/puppet-.* projects may run into that instability for stuff they bundler/rvm/whatever into their jobs | 19:46 |
| pabelanger | fungi, Ya, that's what I am trying to do, preinstall them. But that's where the instability is, seems the gems are not getting downloaded correctly and puppet is reporting everything successful installed. | 19:47 |
| pabelanger | okay, I'll keep digging at it | 19:48 |
| zzelle | sdague, good not to know | 19:49 |
| zzelle | sdague, argh, good to know :) | 19:49 |
| fungi | pabelanger: oh, right, so when our puppet manifest was failing to install rake and the things which depended on it, puppeting continued regardless of the error (and then we had broken jobs because of it) | 19:50 |
| sdague | zzelle: heh | 19:50 |
| fungi | finding a way to make that fail the puppet apply outright would be good, since we don't re-run it | 19:50 |
| *** dcramer_ has quit IRC | 19:52 | |
| *** shakayumi has joined #openstack-infra | 19:54 | |
| *** khyati has quit IRC | 19:55 | |
| *** emagana has quit IRC | 19:55 | |
| *** shakamunyi has joined #openstack-infra | 19:56 | |
| *** rhsu has joined #openstack-infra | 19:57 | |
| *** shakayumi has quit IRC | 19:58 | |
| *** dstanek is now known as dstanekafk | 19:58 | |
| *** _nadya_ has joined #openstack-infra | 19:59 | |
| *** changbl has quit IRC | 20:00 | |
| *** CaptTofu has joined #openstack-infra | 20:02 | |
| *** rhsu has left #openstack-infra | 20:03 | |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Add blurb on configuration and run_mirror in README https://review.openstack.org/80861 | 20:03 |
| *** shakamunyi has quit IRC | 20:03 | |
| pabelanger | fungi, Ya, that's what I think I'm going to try and do. | 20:04 |
| *** ociuhandu has joined #openstack-infra | 20:04 | |
| *** derekh has quit IRC | 20:05 | |
| *** shakamunyi has joined #openstack-infra | 20:06 | |
| *** yassine has quit IRC | 20:12 | |
| *** e0ne has quit IRC | 20:15 | |
| *** e0ne has joined #openstack-infra | 20:16 | |
| *** khyati has joined #openstack-infra | 20:17 | |
| *** dstanekafk is now known as dstanek | 20:17 | |
| *** bhuvan has joined #openstack-infra | 20:20 | |
| *** jasondotstar has quit IRC | 20:20 | |
| *** _nadya_ has quit IRC | 20:20 | |
| *** e0ne has quit IRC | 20:20 | |
| pabelanger | fungi, had another nodepool question, but think people missed it: | 20:21 |
| pabelanger | <pabelanger> If I setup a target.image in nodepool to have a min-ready of 0, should I expect jenkins to tell nodepool to launch one when a new job is published to the queue? | 20:22 |
| pabelanger | <pabelanger> My logic for min-ready = 0 is, i have some nodes that I don't expect to be too active. So, rather them sitting around on the cloud (eating up my budget), it only expect them to launch once jenkins needed to test it. | 20:22 |
| *** dcramer_ has joined #openstack-infra | 20:22 | |
| fungi | pabelanger: yeah, saw it but not sure. i don't think i've tried setting min-ready < 1 before | 20:23 |
| pabelanger | fungi, okay, that is what I suspect. | 20:24 |
| *** rlandy has quit IRC | 20:24 | |
| pabelanger | going to dig around the source and see what I can find | 20:24 |
| jeblair | pabelanger: i think so, though also i don't think nodepool will create the image if it's missing in that case | 20:24 |
| *** signed8bit has quit IRC | 20:24 | |
| jeblair | pabelanger: (it will still create it during the daily update) | 20:24 |
| jeblair | (i'm not positive about that, it actually might still create the image if min-ready is 0 as long as it's still attached to a target) | 20:25 |
| pabelanger | jeblair, from what I see, the template image gets created properly, and snapshoted. However, it seems nodepool doesn't see the job in queue. So, I'm not sure if jenkins is not notifying because there is no builder online | 20:26 |
| jeblair | pabelanger: do you have nodepool connected up to your zuul gearman server? | 20:27 |
| pabelanger | fungi, back to my ruby gem issue, I think it is related to hpcloud and slowness. I just retested using rackspace and everything worked as expected | 20:27 |
| jeblair | pabelanger: (it's optional, so it's possible it's not) | 20:27 |
| pabelanger | jeblair, checking | 20:27 |
| jeblair | pabelanger: if so, then when zuul adds the job to the gearman queue, nodepool should see the demand for it and spin up a node | 20:27 |
| jeblair | pabelanger: however, one thing to note as that zuul itself won't see that job as being registered unless there is a node online that can run it when zuul starts | 20:28 |
| pabelanger | jeblair, no, I don't see a setting in the yaml file. Off to find the setting | 20:28 |
| *** vkozhukalov has quit IRC | 20:28 | |
| jeblair | pabelanger: (once it sees the job once, it will remember it, but until it sees the first one, it doesn't know whether the job is real or a typo) | 20:28 |
| pabelanger | jeblair, okay, added gearman server to nodepool, going to try again | 20:31 |
| clarkb | jeblair: russellb pabelanger this asterisk thing works pretty well | 20:32 |
| pabelanger | fungi, so, I think the download is happening too slow, then gets killed in a bad state | 20:33 |
| *** dprince has quit IRC | 20:33 | |
| russellb | clarkb: oh yeah? cool. :-) | 20:33 |
| pabelanger | clarkb, what you doing with it? | 20:33 |
| clarkb | pabelanger: just a small call | 20:33 |
| clarkb | nothing fancy but it is super useful | 20:33 |
| pabelanger | one of my next tasks is to hackup nodepool to see if I can get a cluster of asterisk boxes going. | 20:34 |
| russellb | neat, i'm still #2 in all time commits for it :) | 20:34 |
| pabelanger | then rotate them in / out of production | 20:34 |
| fungi | pabelanger: can you paste the error you're getting on paste.o.o | 20:34 |
| fungi | pabelanger: i've seen some ruby execution timeouts on nodepool image builds in the past, particularly in hpcloud, wondering if this is the same | 20:35 |
| pabelanger | fungi, http://pastebin.com/zWiUv5Zs | 20:35 |
| pabelanger | ya, I'm on hpcloud-region-b right now for testing | 20:36 |
| fungi | pabelanger: ohh, i thought you meant download errors during a nodepool image build | 20:36 |
| *** signed8bit has joined #openstack-infra | 20:37 | |
| *** adalbas has quit IRC | 20:38 | |
| * clarkb hangs out on the couch for a change of scenery | 20:38 | |
| *** bhuvan has quit IRC | 20:39 | |
| pabelanger | fungi, no, I don't see any actually errors when nodepool build my template on hpcloud, everything works well | 20:39 |
| *** vhoward has left #openstack-infra | 20:39 | |
| pabelanger | however, once the job launches, ruby is clearly corrupt | 20:39 |
| *** bhuvan has joined #openstack-infra | 20:40 | |
| pabelanger | jeblair, okay, progress: here is what nodepool sees: http://pastebin.com/gjEtC6jQ | 20:40 |
| fungi | pabelanger: in the case of rake not installing earlier (due to dependency issues) i was actually able to find the errors in the puppet output recorded in the nodepool image.log | 20:40 |
| pabelanger | fungi, okay, let me confirm | 20:41 |
| jeblair | pabelanger: that output looks good | 20:41 |
| jeblair | pabelanger: you know about the multi-node patches in review? | 20:41 |
| fungi | he chimed in on the warning thread in the -infra ml | 20:41 |
| pabelanger | jeblair, Aware, that's about the extent of it | 20:42 |
| pabelanger | no issue for me blowing this away, just testing right now | 20:42 |
| *** markwash has joined #openstack-infra | 20:49 | |
| pabelanger | jeblair, okay, so you wouldn't expect nodepool to launch and attach the node to jenkins if it seen the job in the queue? | 20:50 |
| jeblair | pabelanger: on the contrary, with that output, i would expect it to do exactly that | 20:51 |
| pabelanger | jeblair, okay, so in that case nothing happened :(. I'm going to clear out everything and try again | 20:51 |
| *** changbl has joined #openstack-infra | 20:51 | |
| *** hogepodge_ has joined #openstack-infra | 20:52 | |
| *** mrda_away is now known as mrda | 20:52 | |
| jeblair | pabelanger: what was the output from the allocator? you should see some log entries related to that just after the bit you copied | 20:52 |
| *** dkliban has quit IRC | 20:53 | |
| *** hogepodge has quit IRC | 20:55 | |
| *** hogepodge_ is now known as hogepodge | 20:55 | |
| pabelanger | jeblair, I believe this is what you are looking for: http://pastebin.com/bHd4ButE | 20:55 |
| *** bhuvan has quit IRC | 20:55 | |
| clarkb | ~. | 20:55 |
| clarkb | wow fail | 20:55 |
| fungi | NO CARRIER | 20:55 |
| *** bhuvan has joined #openstack-infra | 20:56 | |
| jeblair | pabelanger: aha | 20:56 |
| jeblair | pabelanger: the allocator gave your jenkins server a weight of 0 based on the min-ready value there; that should be fixed (inadvertently) in the patch series i have up for review | 20:57 |
| jeblair | pabelanger: (and so therefore it decided there was no place to attach the node) | 20:57 |
| pabelanger | jeblair, okay, cool. I can give that patchset ago | 20:57 |
| pabelanger | and see what happens | 20:57 |
| jeblair | pabelanger: it's the series that has the config file format change though, so be aware of that | 20:58 |
| *** ArxCruz has quit IRC | 20:58 | |
| pabelanger | that is no issue | 20:58 |
| clarkb | fungi: heads up https://review.openstack.org/#/c/81301/ | 21:01 |
| fungi | clarkb: thanks | 21:02 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Allow BRANCH instead of ZUUL_REF in ggp https://review.openstack.org/81301 | 21:03 |
| fungi | clarkb: https://review.openstack.org/82583 is in a similar vein (improvements to ggp), but also fixes an outstanding bug... i'm waiting to get that into a fresh image so i can re-trigger a couple failing publications repo tag jobs | 21:04 |
| *** VijayTripathi has joined #openstack-infra | 21:05 | |
| *** dcramer_ has quit IRC | 21:05 | |
| clarkb | fungi: looking, I think you are set to reapprove 81301 | 21:05 |
| fungi | thanks | 21:06 |
| jeblair | clarkb, fungi: i believe we have images for all node types in region-b; shall i approve https://review.openstack.org/#/c/81874/1 ? | 21:06 |
| *** aysyd has quit IRC | 21:07 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: gerrit plugin builds need to use version-properties.sh https://review.openstack.org/82921 | 21:07 |
| *** pdmars has quit IRC | 21:07 | |
| fungi | jeblair: looks like it. i assume nobody else has heard of any mysterious errors cropping up on region b workers | 21:07 |
| clarkb | jeblair: I think we should go for it; however I am going to afk for the afternoon in a little while so can't help firefight if necessary | 21:08 |
| clarkb | hopefully they don't dilate my eyes | 21:08 |
| *** shakayumi has joined #openstack-infra | 21:08 | |
| jeblair | fungi, clarkb: at least some jobs have passed in region-b. :) | 21:08 |
| fungi | i'm about to head out to dinner, but can keep tabs on things in an hour or two when i get back | 21:09 |
| *** shakayumi has quit IRC | 21:09 | |
| clarkb | fungi: reviewed 82583, suggsetion for making it a bit more clear | 21:10 |
| pabelanger | jeblair, okay, giving the patchset a run now. Let's see what happens | 21:10 |
| *** mbacchi has quit IRC | 21:10 | |
| *** shakamunyi has quit IRC | 21:11 | |
| *** rfolco has quit IRC | 21:12 | |
| *** dizquierdo has joined #openstack-infra | 21:13 | |
| *** ildikov_ has joined #openstack-infra | 21:13 | |
| openstackgerrit | Benjamin Staffin proposed a change to openstack-infra/jenkins-job-builder: Add support for Slave Utilization plugin https://review.openstack.org/82693 | 21:15 |
| mattoliverau | Morning | 21:15 |
| openstackgerrit | A change was merged to openstack-infra/config: Update Murano gate job config https://review.openstack.org/82316 | 21:15 |
| anteaya | morning mattoliverau | 21:15 |
| *** khyati has quit IRC | 21:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: Allow BRANCH instead of ZUUL_REF in ggp https://review.openstack.org/81301 | 21:16 |
| fungi | clarkb: thanks, will have a look | 21:20 |
| * fungi heads out, back in a couple hours | 21:22 | |
| *** signed8bit has quit IRC | 21:23 | |
| *** bhuvan has quit IRC | 21:23 | |
| *** signed8bit has joined #openstack-infra | 21:23 | |
| openstackgerrit | A change was merged to openstack-infra/config: Stop using hpcloud az1-az3 in favor of region-b https://review.openstack.org/81874 | 21:25 |
| *** bhuvan has joined #openstack-infra | 21:26 | |
| *** gokrokve has quit IRC | 21:32 | |
| *** thomasem has joined #openstack-infra | 21:32 | |
| *** hashar has quit IRC | 21:33 | |
| *** jooools has joined #openstack-infra | 21:34 | |
| *** dkranz has quit IRC | 21:36 | |
| *** khyati has joined #openstack-infra | 21:37 | |
| *** Alex_Gaynor has quit IRC | 21:39 | |
| *** dangers is now known as dangers_away | 21:40 | |
| *** melwitt1 has joined #openstack-infra | 21:40 | |
| *** Alex_Gaynor has joined #openstack-infra | 21:41 | |
| *** UtahDave has joined #openstack-infra | 21:41 | |
| *** eharney has quit IRC | 21:42 | |
| *** jeblair_ has joined #openstack-infra | 21:42 | |
| *** jeblair_ is now known as corvus | 21:43 | |
| *** aude_ has joined #openstack-infra | 21:43 | |
| pabelanger | jeblair, okay. Your patch looks to be working as you described with min-ready: 0 | 21:44 |
| corvus | neat! | 21:44 |
| *** alexpilotti has joined #openstack-infra | 21:44 | |
| pabelanger | just testing 1 more thing, then I can help review | 21:44 |
| *** melwitt has quit IRC | 21:45 | |
| *** aude has quit IRC | 21:45 | |
| *** jeblair has quit IRC | 21:45 | |
| *** corvus is now known as jeblair | 21:45 | |
| clarkb | stealth jeblair | 21:45 |
| jeblair | the freenode server i was on died | 21:46 |
| mattoliverau | Yay, I finally have approval to go to the Atlanta Design Summit! So if anyone sees a very jetlagged Aussie there, then it would probably be me :P | 21:46 |
| jeblair | mattoliverau: whew! | 21:46 |
| clarkb | mattoliverau: after doing LCA I feel that pain | 21:46 |
| *** StevenK has quit IRC | 21:47 | |
| *** jbryce has quit IRC | 21:47 | |
| *** russellb has quit IRC | 21:47 | |
| *** zigo has quit IRC | 21:47 | |
| *** wendar has quit IRC | 21:47 | |
| *** jlk has quit IRC | 21:47 | |
| *** jlk has joined #openstack-infra | 21:47 | |
| *** jlk has joined #openstack-infra | 21:47 | |
| *** StevenK has joined #openstack-infra | 21:47 | |
| *** jbryce has joined #openstack-infra | 21:47 | |
| *** dolphm has quit IRC | 21:47 | |
| *** russellb has joined #openstack-infra | 21:48 | |
| *** mwagner_lap has quit IRC | 21:49 | |
| *** wendar has joined #openstack-infra | 21:49 | |
| mattoliverau | clarkb: yeah, I spent 1/2 of feb in san antonio, that messed me around alot.. but what can I say I'm excited, so am a glutten for sleep deprivment :P | 21:49 |
| *** zigo has joined #openstack-infra | 21:49 | |
| *** mfer has quit IRC | 21:50 | |
| *** mriedem has quit IRC | 21:50 | |
| pabelanger | jeblair, okay, so here is something different. when I have min-ready:0 now, and start nodepoold for the first time. The templates for my images are not built right away. Is that expected behaviour? | 21:50 |
| *** dolphm has joined #openstack-infra | 21:50 | |
| *** dizquierdo is now known as dizquierdo_afk | 21:51 | |
| pabelanger | jeblair, and when a new job enters the queue, nodepool detects it, says need to launch 1 bare-precise node for jenkins, but no current image for bare-precise can be found. then loops | 21:52 |
| pabelanger | look like the template task is getting skipped | 21:53 |
| clarkb | pabelanger: yes, I think nodepool will only make sure it has images for images with min ready > 0 | 21:53 |
| jeblair | pabelanger: yeah, that's the issue i was half-expecting earlier | 21:54 |
| pabelanger | okay, thought so | 21:54 |
| jeblair | shouldn't be too hard to fix | 21:54 |
| pabelanger | I can go do the leg work, how do you expect it to work? | 21:55 |
| pabelanger | only create a template if it is missing? | 21:55 |
| pabelanger | and new job requires it? | 21:55 |
| jeblair | pabelanger: definitely that, but how to determine whether it's missing is interesting: | 21:56 |
| jogo | just wanted to confirm: no plans on moving gantt repo around? | 21:56 |
| jeblair | pabelanger: current method is "it has a min-ready > 0" which isn't sufficient | 21:56 |
| jogo | re: https://review.openstack.org/#/c/82858/ | 21:56 |
| jeblair | jogo: i'm going to send an email to confirm this soon, but based on all the information we have, we think the right thing to do is to freeze that repo for now, and ask the folks who want to continue that work elsewhere | 21:57 |
| jeblair | *do so elsewhere* | 21:57 |
| pabelanger | jeblair, okay, let me see if I can find another way to determine it is missing. Going to poke around in the code. | 21:57 |
| jeblair | pabelanger: sorry, wasn't finished | 21:57 |
| *** thomasem has quit IRC | 21:58 | |
| jeblair | pabelanger: we could change that to "has min-ready>0 or is required by a job", or "is associated with a label", or just do it for all images | 21:59 |
| jogo | jeblair: works for me thanks. I'll hold off my vote on that patch until your email is sent | 21:59 |
| jeblair | pabelanger: i think the idea originally was to avoid the image creation loop for images that weren't in use | 21:59 |
| *** markmc has quit IRC | 22:01 | |
| pabelanger | jeblair, okay. Ya, my use case is I want to save some $$$ on public clouds and only launch when a job is needed. I could afford to wait the 5mins for it to spin up. But the change you are suggesting seems to work for that use case. However, not sure if it create more churn on the -infra side | 22:01 |
| clarkb | ok /me AFKs | 22:02 |
| jeblair | pabelanger: i think it would be okay to perform the missing check for all images | 22:02 |
| *** jhesketh_ has joined #openstack-infra | 22:02 | |
| jeblair | probably the simplest solution and i don't think it would cause undue stress | 22:02 |
| jhesketh_ | Morning | 22:03 |
| *** bknudson has quit IRC | 22:03 | |
| pabelanger | jeblair, okay, just so I understand, you're suggestion build the templates for all images regardless of min-ready > 0? | 22:04 |
| pabelanger | or something else | 22:04 |
| jeblair | pabelanger: yes, that suggestion. | 22:04 |
| pabelanger | jeblair, okay, let me see if I can patch nodepool to do that | 22:04 |
| pabelanger | another option would be to set min-ready: -1 if you didn't actually want them to be built | 22:06 |
| pabelanger | okay, found the code and commented out the check. Testing now | 22:07 |
| jeblair | jhesketh_: good morning | 22:07 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 22:08 |
| *** dizquierdo_afk is now known as dizquierdo | 22:11 | |
| *** markwash_ has joined #openstack-infra | 22:12 | |
| *** markwash has quit IRC | 22:14 | |
| *** markwash_ is now known as markwash | 22:14 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 22:16 |
| *** changbl has quit IRC | 22:17 | |
| *** dims_ has quit IRC | 22:18 | |
| *** zns has joined #openstack-infra | 22:19 | |
| jeblair | jhesketh_: you may be interested in this: https://review.openstack.org/#/c/82865/ | 22:21 |
| zaro | clarkb: why is this commented out? http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/jenkins.pp#n87 | 22:21 |
| jeblair | zaro: not sure it's been released | 22:22 |
| *** prad has quit IRC | 22:22 | |
| *** dstanek has quit IRC | 22:22 | |
| jeblair | nibalizer: what would it take to get more than 10 reports out of puppetboard? | 22:23 |
| zaro | jeblair: a few versions have already been release -> http://repo.jenkins-ci.org/repo/org/jenkins-ci/plugins/zmq-event-publisher/ | 22:23 |
| *** alex-sick is now known as Alexandra | 22:24 | |
| jeblair | zaro: then you can probably update that | 22:24 |
| nibalizer | jeblair: why do you want them? | 22:24 |
| nibalizer | you can get them for analysis and stuff straight from the puppetdb api | 22:24 |
| *** signed8bit has quit IRC | 22:24 | |
| nibalizer | clicking through that stuff would be a drag | 22:24 |
| nibalizer | but let me give you the code | 22:24 |
| jeblair | nibalizer: i have some pretty long test-debug cycles; i often want to see something that happened more than 2 hours ago | 22:25 |
| *** zns has quit IRC | 22:25 | |
| nibalizer | https://github.com/nedap/puppetboard/blob/master/puppetboard/app.py#L176 | 22:25 |
| nibalizer | so ten_reports is hard codede | 22:25 |
| jeblair | for instance, i just wanted to check on an event that would have happened about 4 hours ago | 22:25 |
| *** zns has joined #openstack-infra | 22:25 | |
| nibalizer | but we could refactor that function to take a number (default to ten) to back in time with | 22:26 |
| nibalizer | and add that to default_settings.py | 22:26 |
| jeblair | that sounds like a swell implementation | 22:27 |
| mgagne | I was flabbergasted too when I found out about the ten_reports function... | 22:30 |
| nibalizer | part of that is the history of puppetboard/puppetdb | 22:30 |
| nibalizer | in the past, like 2 months ago, puppetdb itself didn's support pagination | 22:30 |
| nibalizer | so asking for too much meant really long requests and load times | 22:30 |
| mgagne | is it fixed now? | 22:30 |
| nibalizer | puppetdb is fixed now | 22:31 |
| nibalizer | so that is a place to fix it | 22:31 |
| nibalizer | there is also a #puppetboard where you can talk with daenney | 22:31 |
| nibalizer | who is the primary author of this thing | 22:31 |
| nibalizer | and is super responsive to user feedback and taking pull requests and stuff | 22:31 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 22:31 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: make puppet install jenkins plugins https://review.openstack.org/77040 | 22:31 |
| mgagne | nibalizer: is there a way to filter by time now? | 22:32 |
| nibalizer | im not sure | 22:32 |
| * nibalizer not familiar with the internals | 22:32 | |
| mgagne | nibalizer: ok, I got my negative answer here: https://tickets.puppetlabs.com/browse/PDB-201 | 22:32 |
| *** dims_ has joined #openstack-infra | 22:34 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added paging to list endpoints https://review.openstack.org/79757 | 22:34 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: let puppet manage jenkins zmq-event-publisher plugin https://review.openstack.org/82936 | 22:36 |
| *** dizquierdo has quit IRC | 22:37 | |
| *** dstanek has joined #openstack-infra | 22:37 | |
| nibalizer | mgagne: jeblair sorry, what can I do? | 22:38 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 22:40 |
| *** nkinder has quit IRC | 22:43 | |
| *** jergerber has quit IRC | 22:44 | |
| *** denis_makogon has quit IRC | 22:46 | |
| pabelanger | jeblair, okay, patch works. Calling it a day, but will look to get it up for review soon | 22:46 |
| *** rcleere has quit IRC | 22:46 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added paging to list endpoints https://review.openstack.org/79757 | 22:52 |
| *** jpeeler has quit IRC | 22:54 | |
| *** nkinder has joined #openstack-infra | 22:55 | |
| clarkb | woo eyes are dilating dont expect me to be super useful | 22:55 |
| fungi | but wow is that ever fun | 22:58 |
| fungi | (i'm assuming... never been through it mysqlf since i still somehow don't need glasses) | 22:58 |
| *** bhuvan has quit IRC | 22:58 | |
| fungi | also glad to see zuul's world hasn't blown up even though we're no longer using old hpcloud | 22:59 |
| fungi | though we are building a lot more nodes than i'd expect, according to the graph... looking into it now | 23:00 |
| fungi | 110 hpcloud-region-b | 23:00 |
| jogo | I apporoved a patch but I don't see it running in the check or gate queues | 23:01 |
| jogo | https://review.openstack.org/#/c/79617/ | 23:01 |
| *** mwagner_lap has joined #openstack-infra | 23:02 | |
| fungi | Queue lengths: 9 events, 13 results. | 23:02 |
| fungi | might be in there still | 23:02 |
| jogo | fungi: approved it a while ago | 23:02 |
| SpamapS | is there some way to make gerrit show me the diff of two diffs, rather than the diff including rebased on changes? | 23:02 |
| jogo | fungi: almost anhour ago | 23:02 |
| fungi | jogo: oh, then no | 23:03 |
| StevenK | SpamapS: I had to reach into my local branch and run git di <rev>..<rev> | 23:03 |
| *** melwitt1 has quit IRC | 23:04 | |
| fungi | SpamapS: closest you can get is to review the files on the later patchset, then switch the base patchset, only click the files with green "reviewed" checkmarks, and ignore chunks you know are from other changes | 23:04 |
| SpamapS | fungi: thats exactly what I do. :( | 23:05 |
| fungi | SpamapS: me too. i feel ur painz | 23:05 |
| jogo | fungi: I re-approved but still don't see it | 23:06 |
| jogo | unless I am doing something wrong | 23:06 |
| jogo | https://review.openstack.org/#/c/79617/ | 23:06 |
| fungi | jogo: i think i see what happened. when gerrit was straining this morning it apparently failed to send some events (or zuul reconnected to it in that timeframe and subsequently missed them). i think 79617,8 was uploaded in that timeframe, while the results from ,7 hadn't been reported yet | 23:07 |
| fungi | so there's no jenkins vote on the change at all | 23:07 |
| *** david-lyle has quit IRC | 23:08 | |
| fungi | jogo: leave a "recheck no bug" comment to kick it into processing the jobs for that change | 23:08 |
| jogo | so do I need to run recheck no bug first | 23:08 |
| jogo | ahh | 23:08 |
| jogo | that kicked it off | 23:08 |
| *** jhesketh__ has joined #openstack-infra | 23:09 | |
| fungi | great. theory validated. now hopefully one of us figures out a good way to prevent that pattern in the future | 23:09 |
| *** markmcclain has joined #openstack-infra | 23:10 | |
| fungi | odd constellation of coincidences needed to cause it | 23:10 |
| *** markmcclain1 has joined #openstack-infra | 23:11 | |
| fungi | i think we need - username: jenkins\n older-than: 24h to also work when there is no comment from that username at all | 23:13 |
| *** ArxCruz has joined #openstack-infra | 23:13 | |
| *** markmcclain has quit IRC | 23:15 | |
| *** thedodd has quit IRC | 23:17 | |
| *** ArxCruz has quit IRC | 23:17 | |
| chmouel | mordred, notmyname: about swift3 and stackforge this is about having it for devstack, since things that's out of openstack infra is not going to be supported in devstack | 23:17 |
| chmouel | and github is down gets annoying after a while | 23:18 |
| *** thuc_ has quit IRC | 23:18 | |
| notmyname | chmouel: I'm not sure I understand that. this migration is to make devstack work? | 23:18 |
| mordred | no. devstack works off of github | 23:18 |
| mordred | it's to make 10000 runs of devstack per day work consistently | 23:18 |
| *** thuc has joined #openstack-infra | 23:18 | |
| chmouel | notmyname: what mordred says | 23:18 |
| chmouel | in devstack the repos that are not on openstack infra will be removed | 23:19 |
| mordred | another option would be for the swift3 guys to release to pypi and for us to cache that via normal mechanisms | 23:19 |
| notmyname | so not to make devstack work. but, really, to make devstack work ;-) | 23:19 |
| mordred | no | 23:20 |
| mordred | not really | 23:20 |
| notmyname | work == work consistently | 23:20 |
| mordred | yes. there is a difference :) | 23:20 |
| chmouel | well if someone could fix github then devstack would not have to be 'fixed' | 23:20 |
| notmyname | I understand that devstack can function by pulling stuff from github. | 23:20 |
| chmouel | and oss it at the same time | 23:20 |
| *** dstanek has quit IRC | 23:21 | |
| *** jhesketh__ has quit IRC | 23:21 | |
| fungi | chmouel: ultimately, less fixing github and more fixing the internet in generall... we just happen to have extra belt-and-braces ways to reduce unstable internet impact on our tests for stuff we host already | 23:21 |
| fungi | though github does seem to be marginally less reliable than the internet as a whole | 23:22 |
| *** aude_ is now known as aude | 23:22 | |
| *** andreaf has quit IRC | 23:22 | |
| *** jhesketh__ has joined #openstack-infra | 23:22 | |
| *** thuc has quit IRC | 23:23 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed a change to openstack-infra/gerritbot: Add support for notification of releases on IRC https://review.openstack.org/79205 | 23:23 |
| *** krtaylor has quit IRC | 23:23 | |
| chmouel | fungi: :) | 23:24 |
| *** jgrimm has quit IRC | 23:25 | |
| *** julim has quit IRC | 23:30 | |
| *** markmcclain1 has quit IRC | 23:30 | |
| fungi | lifeless: fyi, still hitting... "OverLimit: Quota exceeded for ram: Requested 16384, but already used 948224 of 960000 ram (HTTP 413)" | 23:31 |
| fungi | in hpcloud-region-b we're getting a lot of "Exception: Timeout waiting for waiting for server" which explains the high building count there | 23:36 |
| *** wenlock has quit IRC | 23:37 | |
| fungi | clarkb: jeblair: mordred: ^ not sure whether it's a good idea to continue overnight or revert 46bf851 | 23:38 |
| *** dstanek has joined #openstack-infra | 23:38 | |
| *** dkliban has joined #openstack-infra | 23:38 | |
| fungi | i see servers climbing up around 70 minutes in a building state there, then timing out | 23:39 |
| jeblair | fungi: ugh | 23:41 |
| jeblair | fungi: what's the success rate? | 23:42 |
| fungi | jeblair: not sure... i'll try to work out a ratio from the logs | 23:42 |
| jeblair | fungi: hrm, that might be tricky... | 23:43 |
| jeblair | fungi: we can probably get the number of used servers this way: | 23:44 |
| jeblair | grep Received debug.log|grep region-b|wc -l | 23:44 |
| jeblair | 1365 | 23:44 |
| fungi | it seems to be provider-level since the ratio of image types in a building state is roughly what i'd expect | 23:44 |
| *** zzelle has quit IRC | 23:44 | |
| jeblair | fungi: do you have a signature for the failure that associates it with region-b? | 23:45 |
| fungi | jeblair: not without several layers of correlation | 23:45 |
| jeblair | that number above is probably 3x the number of completed jobs | 23:45 |
| jeblair | so 455 nodes | 23:45 |
| fungi | unless we just assume all of "Timeout waiting for waiting for server" is region b (might be within an order of magnitude anyway) | 23:46 |
| fungi | 5482 since the log rolled over | 23:47 |
| jeblair | fungi: grep "Timeout waiting for waiting for server" debug.log |grep -v deletion|wc -l | 23:47 |
| jeblair | gives me 583 today, compared with 464 yesterday | 23:48 |
| fungi | hmm... not too different then, i guess | 23:48 |
| *** Ryan_Lane has quit IRC | 23:48 | |
| jeblair | though we've only been heavily using region-b for a few hours | 23:48 |
| fungi | maybe if these are spending a log time in building state and not turning over quickly, and only in the past few hours | 23:49 |
| fungi | right | 23:49 |
| fungi | may not impact the overall tally that much yet | 23:49 |
| jeblair | fungi: i lean toward leaving the config as-is and trying to work on it from there | 23:49 |
| jeblair | at least, inasmuch as it doesn't seem completely broken | 23:49 |
| fungi | i was sounding more of an early warning alarm, given that the building count roughly doubled on the graph around teh time we switched to region b and hasn't backed off since | 23:50 |
| fungi | also, i think we're starving for rarer node types because so many of their allocation count are tied up in building nodes which never finish | 23:52 |
| fungi | bunch of jobs in check waiting on py3k-precise nodes | 23:52 |
| jeblair | fungi: hrm. you're convicing me that reverting might be better. | 23:53 |
| *** mwagner_ has joined #openstack-infra | 23:54 | |
| fungi | if only we had more detailed failure data for these timing out in building (i'm looking at you nova{,client}) | 23:55 |
| lifeless | fungi: still waiting on info from mikal AFAIK | 23:55 |
| jeblair | heh, we should also add more info to the nodepool logs so we can separate this out from other providers easier | 23:55 |
| fungi | i assume these are where we say 'nova make me a sandwich' and then never see the sandwich nor any excuse for the lack thereof | 23:56 |
| mordred | fungi: or even any bread | 23:56 |
| *** colinmcnamara has joined #openstack-infra | 23:56 | |
| jeblair | fungi: yeh, well, we see it go to the kitchen but it never comes back. | 23:56 |
| StevenK | fungi: Maybe 'sudo nova make me a sandwich' will work better ... | 23:56 |
| fungi | mordred: i'd take a pita at this point | 23:56 |
| mordred | fungi: lifeless say we shouldn't eat bread product | 23:56 |
| jeblair | fungi: i'm pretty sure we have a pita. | 23:56 |
| mordred | jeblair wins | 23:57 |
| fungi | bwahaha | 23:57 |
| lifeless | sodu nova make me a roast | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!