| *** jamesmcarthur has quit IRC | 00:00 | |
| *** matsuhashi has joined #openstack-infra | 00:02 | |
| *** thomasem has quit IRC | 00:06 | |
| *** michchap has quit IRC | 00:06 | |
| *** michchap has joined #openstack-infra | 00:06 | |
| lifeless | clarkb: if around - https://review.openstack.org/#/c/54152/ needs second +2. | 00:06 |
|---|---|---|
| lifeless | mordred: if you have suggestions on how I could have communicated what I wanted better ^ I'd appreciate them - I was asking for the same stauff for the last 3 iterations or so | 00:07 |
| lifeless | :) | 00:07 |
| clarkb | lifeless: any idea why that valueerror exception might be triggered? we already ignore blank lines and comments | 00:09 |
| lifeless | clarkb: urls | 00:11 |
| lifeless | clarkb: requirements.txt can also include index urls, bizarrely enough | 00:11 |
| clarkb | o_O | 00:12 |
| clarkb | approved | 00:12 |
| *** slong has quit IRC | 00:13 | |
| *** matsuhashi has quit IRC | 00:14 | |
| *** matsuhashi has joined #openstack-infra | 00:15 | |
| *** matsuhashi has quit IRC | 00:15 | |
| *** matsuhashi has joined #openstack-infra | 00:15 | |
| *** wenlock has quit IRC | 00:16 | |
| *** herndon has quit IRC | 00:22 | |
| *** matsuhashi has quit IRC | 00:29 | |
| *** matsuhashi has joined #openstack-infra | 00:29 | |
| *** dcramer_ has quit IRC | 00:30 | |
| *** sarob has joined #openstack-infra | 00:32 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: proposal to allow additional people to release jjb https://review.openstack.org/56823 | 00:32 |
| *** matsuhashi has quit IRC | 00:34 | |
| zaro | clarkb: if you got time i would like this to merge.. https://review.openstack.org/#/c/56715 | 00:34 |
| *** senk has joined #openstack-infra | 00:36 | |
| *** boris-42 has joined #openstack-infra | 01:03 | |
| *** markwash has joined #openstack-infra | 01:05 | |
| *** talluri has quit IRC | 01:06 | |
| cyeoh | mikal: sorry about that - I don't know the guy personally. Please feel free to reply to him saying you don't respond to off-list emails. I'll try to chase up what is going on | 01:06 |
| *** senk has quit IRC | 01:07 | |
| *** matsuhashi has joined #openstack-infra | 01:11 | |
| *** boris-42 has quit IRC | 01:17 | |
| *** nati_ueno has joined #openstack-infra | 01:20 | |
| *** ljjjustin has joined #openstack-infra | 01:22 | |
| *** sarob has quit IRC | 01:23 | |
| *** sarob has joined #openstack-infra | 01:23 | |
| *** nosnos has joined #openstack-infra | 01:24 | |
| *** matsuhashi has quit IRC | 01:24 | |
| *** matsuhashi has joined #openstack-infra | 01:25 | |
| *** sarob has quit IRC | 01:29 | |
| *** matsuhas_ has joined #openstack-infra | 01:29 | |
| *** amotoki has joined #openstack-infra | 01:32 | |
| *** matsuhashi has quit IRC | 01:32 | |
| *** xchu has joined #openstack-infra | 01:35 | |
| *** matsuhas_ has quit IRC | 01:37 | |
| *** matsuhashi has joined #openstack-infra | 01:38 | |
| *** bingbu has joined #openstack-infra | 01:40 | |
| *** ljjjustin has quit IRC | 01:43 | |
| *** ljjjustin has joined #openstack-infra | 01:44 | |
| *** matsuhas_ has joined #openstack-infra | 01:45 | |
| *** matsuhashi has quit IRC | 01:48 | |
| *** wenlock has joined #openstack-infra | 01:51 | |
| *** nati_ueno has quit IRC | 01:59 | |
| *** mriedem has joined #openstack-infra | 02:00 | |
| *** yaguang has joined #openstack-infra | 02:05 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add python-solumclient subproject https://review.openstack.org/56810 | 02:06 |
| jhesketh | clarkb: ping | 02:16 |
| clarkb | pong | 02:17 |
| jhesketh | would you mind taking a look at why I can't push changes to gerrit for turbo hipster please? | 02:18 |
| jhesketh | clarkb: https://review.openstack.org/#/c/56849/ | 02:18 |
| jhesketh | they aren't merging somehow | 02:18 |
| clarkb | ugh I know what the problem is, zuul cloned the empty repo again so has a bad local cache | 02:19 |
| clarkb | jhesketh: if you promise not to push more patchsets for a bit I will fix it | 02:19 |
| *** michchap has quit IRC | 02:19 | |
| jhesketh | deal | 02:19 |
| jhesketh | thanks clarkb | 02:19 |
| *** yamahata has joined #openstack-infra | 02:21 | |
| clarkb | I have to ninja fix it under zuul and don't want it interfering with me which it will do if new patchsets show up :) | 02:21 |
| *** michchap has joined #openstack-infra | 02:22 | |
| jhesketh | what is broken? | 02:22 |
| clarkb | jhesketh: it is a race between the repo being populated in gerrit and zuul cloning from gerrit | 02:22 |
| clarkb | jhesketh: what happens is zuul clones an empty repo because the project is created in gerrit without any content | 02:22 |
| clarkb | then we rerun the manage projects script which populates the repo and everything is good | 02:23 |
| *** paul-- has quit IRC | 02:23 | |
| clarkb | but zuul is left in the old state | 02:23 |
| clarkb | jhesketh: ok try it now (recheck should work) | 02:24 |
| jhesketh | ah okay | 02:24 |
| clarkb | turbo-hipster jobs are queued I think that means I fixed it | 02:26 |
| *** bingbu has quit IRC | 02:26 | |
| jhesketh | yep, saw them get queued | 02:27 |
| jhesketh | thanks for that :-) | 02:27 |
| clarkb | not the underlying problem though, we really need to decouple github stuff from populating gerrit in manage-projects | 02:27 |
| clarkb | np | 02:27 |
| *** cody-somerville has joined #openstack-infra | 02:28 | |
| *** sarob has joined #openstack-infra | 02:34 | |
| *** dcramer_ has joined #openstack-infra | 02:35 | |
| clarkb | russellb: wow how many changes did you just updated? | 02:37 |
| russellb | clarkb: a couple | 02:38 |
| russellb | ... dozen | 02:38 |
| russellb | i don't know | 02:38 |
| clarkb | russellb: :) | 02:38 |
| *** sarob has quit IRC | 02:39 | |
| russellb | 28 i guess | 02:39 |
| *** bingbu has joined #openstack-infra | 02:43 | |
| clarkb | russellb: I had just noticed that a bunch of nova changes showed up and were using all of the slaves | 02:43 |
| *** cody-somerville has quit IRC | 02:44 | |
| russellb | my bad :) | 02:46 |
| russellb | i don't feel that bad. | 02:47 |
| * russellb feels powerful | 02:47 | |
| *** sjing has joined #openstack-infra | 02:51 | |
| sjing | Hello -infra | 02:53 |
| sjing | I just wanted to update the message I got from Claire about the slides for the talk presentations in HK summit | 02:53 |
| sjing | in case you have the same question | 02:53 |
| sjing | her reply is : | 02:54 |
| sjing | All of the videos are now live on the website - http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/. | 02:54 |
| sjing | However we only have presentation slides from the speakers who are responsive and willing to give them to us to post on the website. Hopefully in the next week or two we will get more of the slides. | 02:54 |
| *** dcramer_ has quit IRC | 02:57 | |
| amotoki | hi, a patch from Jenkins to update global requirements is something wrong. https://review.openstack.org/#/c/56163/ | 02:57 |
| *** guohliu has joined #openstack-infra | 02:58 | |
| amotoki | it tried to update the requirement file to old ones and gate-neutron-requirements always fails.... | 02:58 |
| clarkb | amotoki: yes, there is a bug for that somewhere, basically the script was proposing changes made to stable requirements to master branches of projects | 02:58 |
| clarkb | we have disabled that job | 02:59 |
| *** paul-- has joined #openstack-infra | 02:59 | |
| *** bingbu has quit IRC | 02:59 | |
| amotoki | clarkb: thanks. I -2'ed the patch. | 03:01 |
| clarkb | amotoki: fwiw jenkins won't let it merge | 03:01 |
| clarkb | amotoki: we should probably abandon all of those changes, fungi and I discussed it but haven't done that yet | 03:02 |
| *** nati_ueno has joined #openstack-infra | 03:09 | |
| *** bingbu has joined #openstack-infra | 03:12 | |
| *** nati_ueno has quit IRC | 03:14 | |
| openstackgerrit | Michael Chapman proposed a change to openstack-infra/config: Add stackforge project: puppet_openstack_builder https://review.openstack.org/56854 | 03:15 |
| *** talluri has joined #openstack-infra | 03:19 | |
| *** markwash has joined #openstack-infra | 03:23 | |
| lifeless | clarkb: you're slacking off http://russellbryant.net/openstack-stats/all-reviewers-30.txt | 03:24 |
| lifeless | :) | 03:24 |
| clarkb | lifeless: ya, I have been distracted by too many other things | 03:26 |
| pleia2 | clarkb: but jeblair comes back tomorrow! :D | 03:26 |
| *** markwash has quit IRC | 03:28 | |
| *** mriedem has quit IRC | 03:28 | |
| *** wenlock has quit IRC | 03:42 | |
| *** chandankumar has quit IRC | 03:45 | |
| *** chandankumar has joined #openstack-infra | 03:45 | |
| *** matsuhas_ has quit IRC | 03:46 | |
| *** matsuhashi has joined #openstack-infra | 03:47 | |
| *** ben_duyujie has joined #openstack-infra | 03:48 | |
| *** boris-42 has joined #openstack-infra | 03:49 | |
| *** matsuhashi has quit IRC | 03:51 | |
| *** rcleere has joined #openstack-infra | 03:52 | |
| *** paul-- has quit IRC | 03:52 | |
| *** boris-42 has quit IRC | 03:54 | |
| *** jamesmcarthur has joined #openstack-infra | 04:02 | |
| *** nati_ueno has joined #openstack-infra | 04:09 | |
| *** senk has joined #openstack-infra | 04:10 | |
| *** senk has quit IRC | 04:11 | |
| *** senk has joined #openstack-infra | 04:12 | |
| *** markwash has joined #openstack-infra | 04:16 | |
| *** jamesmcarthur has quit IRC | 04:16 | |
| *** matsuhashi has joined #openstack-infra | 04:20 | |
| *** CaptTofu has quit IRC | 04:24 | |
| *** CaptTofu has joined #openstack-infra | 04:24 | |
| *** markwash has quit IRC | 04:24 | |
| jhesketh | clarkb: How do you get past the docs gate when you have dependencies like graphviz/dot? | 04:26 |
| clarkb | jhesketh: is the problem the builds fails because we don't have graphviz installed on the slaves? | 04:26 |
| jhesketh | more or less | 04:26 |
| jhesketh | I was also using sphinxcontrib.programoutput but since that failed I just removed it | 04:27 |
| jhesketh | clarkb: eg: http://logs.openstack.org/49/56849/3/check/gate-turbo-hipster-docs/7bf90d0/console.html | 04:27 |
| clarkb | I thought I had a change proposed to add graphviz to the slaves, wonder what happened to it | 04:27 |
| clarkb | jhesketh: I can't find it, you can propose a change to openstack-infra/config/modules/jenkins/manifests/slave.pp and params.pp to add graphviz | 04:29 |
| clarkb | programoutput should work though, you may need to list it in one of your requirements files, JJB uses it iirc | 04:30 |
| jhesketh | I think the problem with programout is that turbo-hipster has dependencies that aren't installed | 04:31 |
| jhesketh | I'll look at adding graphviz :-) | 04:31 |
| clarkb | non python dependencies? | 04:32 |
| clarkb | non python dependencies like graphviz can be installed via puppet like I mentioned. python deps should go in a requirements or test-requirements file | 04:32 |
| jhesketh | nope, but I didn't look closely | 04:32 |
| jhesketh | it might be my requirements file is wrong | 04:32 |
| jhesketh | I'll play with a virtualenv | 04:33 |
| *** wenlock has joined #openstack-infra | 04:34 | |
| clarkb | really weird that I can't find my graphviz change | 04:37 |
| clarkb | maybe I wrote it and never pushed it, and to look I am on the wrong machine | 04:38 |
| *** matsuhashi has quit IRC | 04:42 | |
| *** matsuhashi has joined #openstack-infra | 04:42 | |
| *** paul-- has joined #openstack-infra | 04:45 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Install graphviz on jenkins slaves for docs https://review.openstack.org/56857 | 04:45 |
| jhesketh | clarkb: ^ | 04:45 |
| jhesketh | so I didn't ensure that it's actually installed because 1) I couldn't see other packages doing that and 2) It's not a very important package | 04:46 |
| clarkb | the ensure happens further down in the code, what you have is fine, but can you update the comment in slave.pp? I will leave a comment in gerrit | 04:47 |
| jhesketh | right, I meant ensure that /usr/bin/dot exists | 04:48 |
| jhesketh | but I didn't see the package ensure which is good | 04:48 |
| jhesketh | ah yes, copy and paste fail | 04:48 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Install graphviz on jenkins slaves for docs https://review.openstack.org/56857 | 04:49 |
| jhesketh | clarkb: fixed. Sorry about that. I didn't mention sphinx doc building as it can apply more generically if we have other places that want graphs | 04:49 |
| clarkb | yup wfm | 04:50 |
| clarkb | is dot provided by the graphviz package? | 04:50 |
| jhesketh | yes | 04:51 |
| clarkb | mordred: ^ I think we are safe depending on EPL licensed projects to produce data for us | 04:56 |
| clarkb | mordred: but a second set of eyes would be nice | 04:56 |
| *** wenlock has quit IRC | 05:00 | |
| *** dstanek has quit IRC | 05:01 | |
| *** sjing has quit IRC | 05:04 | |
| *** sjing has joined #openstack-infra | 05:05 | |
| *** dstanek has joined #openstack-infra | 05:08 | |
| zaro | ohh graphviz. clarkb, does that mean graphviz will be on all slaves? | 05:10 |
| clarkb | zaro: if that change merges | 05:11 |
| *** sarob has joined #openstack-infra | 05:11 | |
| *** senk has quit IRC | 05:12 | |
| *** yamahata has quit IRC | 05:12 | |
| zaro | i was thinking of **maybe** building dependency graph of our build jobs. | 05:12 |
| zaro | they are so dang crazy to follow. | 05:12 |
| clarkb | package dependencies? | 05:13 |
| zaro | no build jobs. | 05:13 |
| clarkb | like for the plugins? (I guess I don't know what "build" jobs are and what dependencies we are talking about) | 05:15 |
| clarkb | most of our jenkins jobs don't have any dependencies, they just run | 05:16 |
| clarkb | the jobs related to publishing have a small dependency list as the sdist must be built first then pushed to pypi then mirror updates | 05:16 |
| clarkb | its also late on sunday night and my brain is probably just not working | 05:17 |
| zaro | yeah, i guess what i mean is gate and release jobs in layout.yaml. | 05:18 |
| zaro | also, are there dependencies defined in the jenkins jobs themselves? | 05:18 |
| clarkb | no we don't define dependencies in the jenkins jobs directly | 05:19 |
| clarkb | I think at one time we might have, but now we rely on zuul | 05:19 |
| zaro | ohh. no upstream/downstream jobs? | 05:19 |
| clarkb | not anymore I think, you can grep in jjb to double chek | 05:20 |
| *** talluri has quit IRC | 05:20 | |
| zaro | maybe tommorrow, my brain not cooperating with my eyes. too late. | 05:21 |
| zaro | but that's good to hear. | 05:21 |
| clarkb | the big reason for that is multiple jenkins | 05:22 |
| clarkb | that doens't work too well with downstream jobs | 05:22 |
| zaro | so maybe just a graph of dependent jobs from yaml file might be interesting. not sure if you can generate dependeny graphs from yaml | 05:22 |
| zaro | guess you not feeling that. | 05:25 |
| clarkb | I am tired too :) you should be able to build a graph based on the yaml. may need massaging though | 05:26 |
| zaro | jhesketh: +1 for graphviz. thanks. | 05:28 |
| zaro | clarkb: can't wait to hear about your weekend IRCing.. | 05:29 |
| clarkb | zaro: it was great, I watched football and had an irc client open | 05:29 |
| clarkb | I did that + sleep that is about it | 05:30 |
| zaro | ahhh football.. espn.. those were the days.. | 05:31 |
| * zaro dreams about clarkb days tonight. | 05:32 | |
| *** dstanek has quit IRC | 05:33 | |
| *** talluri has joined #openstack-infra | 05:35 | |
| *** nosnos_ has joined #openstack-infra | 05:36 | |
| *** nosnos has quit IRC | 05:36 | |
| clarkb | zaro: it might be neat to fiddle witht he puppet graphing stuff too | 05:38 |
| clarkb | I think it uses graphviz to show the catalog graph, which probably isn't too interesting as a whole but might be neat in smaller subsets | 05:39 |
| clarkb | but that doesn't show you inter host stuff which is actually interesting | 05:39 |
| *** aardvark has quit IRC | 05:40 | |
| *** aardvark has joined #openstack-infra | 05:40 | |
| *** DennyZhang has joined #openstack-infra | 05:40 | |
| *** afazekas has quit IRC | 05:40 | |
| *** nati_ueno has quit IRC | 05:41 | |
| *** boris-42 has joined #openstack-infra | 05:44 | |
| *** jhesketh has quit IRC | 05:44 | |
| clarkb | zaro: the JJB change lgtm after a quick look, but I need to spend some time to go through all of the new files | 05:46 |
| *** paul-- has quit IRC | 05:48 | |
| *** rcleere has quit IRC | 05:49 | |
| *** SergeyLukjanov has joined #openstack-infra | 05:54 | |
| *** jhesketh has joined #openstack-infra | 05:57 | |
| *** sarob has quit IRC | 05:57 | |
| *** sarob has joined #openstack-infra | 05:57 | |
| *** sarob has quit IRC | 06:02 | |
| *** boris-42 has quit IRC | 06:03 | |
| *** DennyZhang has quit IRC | 06:04 | |
| *** boris-42 has joined #openstack-infra | 06:06 | |
| jog0 | fungi mordred lifeless mikal clarkb: do any of you take weekend off? | 06:06 |
| clarkb | occasionally | 06:07 |
| jog0 | fungi: the risk of doing auto-recheck for things that elastic-recheck is we may get stuck in an infite loop. | 06:07 |
| jog0 | for a bad query | 06:07 |
| clarkb | jog0: I think fungi was thinking that we would require a bug in e-r to recheck | 06:07 |
| jog0 | clarkb: I like that idea | 06:07 |
| clarkb | I mostly didn't work this weekend. I was watching football | 06:07 |
| jog0 | clarkb: football + IRC = ? | 06:08 |
| clarkb | fun | 06:08 |
| jog0 | I do think we could eventually do auto recheck based on e-r if we have more precautions in place | 06:08 |
| jog0 | (long term) | 06:08 |
| clarkb | the next few weekends I won't be working much | 06:10 |
| jog0 | thats what you say now ... | 06:10 |
| clarkb | no really. going to be in portland for the next one | 06:10 |
| clarkb | which means hanging out with people and not owrking | 06:10 |
| jog0 | mikal: nice find for the console error | 06:10 |
| jog0 | want to file a patch to elastic-recheck for it? | 06:11 |
| jog0 | mikal: https://github.com/openstack-infra/elastic-recheck/blob/master/queries.yaml | 06:12 |
| jog0 | clarkb: so I didn't read the entire backlog about rechecks but, it sounds like we all agree its a problem, which is good | 06:12 |
| clarkb | let me try and tl;dr for you | 06:12 |
| clarkb | current problem appears to be we add new problems quicker than we remove them so what do we do? | 06:13 |
| jog0 | clarkb: easy, cry | 06:13 |
| clarkb | do we disable problematic tests? remove ability to recheck? run many tests to have higher chance of catching things? educate our devs? etc | 06:13 |
| clarkb | I don't think we came away with any hard answers but we do need to do something | 06:13 |
| jog0 | clarkb: I think this sounds like a fun ML fodder | 06:14 |
| jog0 | well sdague and I have at least one idea | 06:14 |
| jog0 | but I have to go so that will have to wait for the ML thread that I am hoping someone will start | 06:15 |
| clarkb | I think sdague or mordred may be best to start it | 06:15 |
| jog0 | clarkb: ++ | 06:15 |
| jog0 | sdague mordred: ^ | 06:15 |
| jog0 | anyway back to my weekend | 06:15 |
| clarkb | because TC and all that | 06:16 |
| lifeless | jog0: I totally did | 06:17 |
| lifeless | jog0: I got nothing done | 06:17 |
| *** markwash has joined #openstack-infra | 06:20 | |
| *** marun has joined #openstack-infra | 06:27 | |
| *** sarob has joined #openstack-infra | 06:36 | |
| *** SergeyLukjanov has quit IRC | 06:41 | |
| *** matsuhashi has quit IRC | 06:54 | |
| *** pblaho has joined #openstack-infra | 06:56 | |
| *** matsuhashi has joined #openstack-infra | 06:56 | |
| *** nosnos_ has quit IRC | 06:56 | |
| *** nosnos has joined #openstack-infra | 06:56 | |
| mikal | jog0: sure, I'll take a look at elastic recheck after dinner | 07:01 |
| *** zoresvit has joined #openstack-infra | 07:01 | |
| *** nati_ueno has joined #openstack-infra | 07:06 | |
| *** nati_ueno has quit IRC | 07:08 | |
| *** ljjjusti1 has joined #openstack-infra | 07:14 | |
| *** ljjjustin has quit IRC | 07:18 | |
| *** sarob has quit IRC | 07:19 | |
| *** sarob has joined #openstack-infra | 07:19 | |
| *** sarob has quit IRC | 07:24 | |
| *** jcoufal has joined #openstack-infra | 07:27 | |
| *** yolanda has joined #openstack-infra | 07:32 | |
| *** boris-42 has quit IRC | 07:37 | |
| *** DinaBelova has joined #openstack-infra | 07:42 | |
| *** michchap has quit IRC | 07:43 | |
| *** michchap has joined #openstack-infra | 07:43 | |
| *** DinaBelova has quit IRC | 07:46 | |
| *** flaper87 has quit IRC | 07:48 | |
| *** flaper87 has joined #openstack-infra | 07:48 | |
| *** DinaBelova has joined #openstack-infra | 07:51 | |
| *** alexpilotti has joined #openstack-infra | 07:53 | |
| *** uvirtbot has joined #openstack-infra | 07:53 | |
| *** SergeyLukjanov has joined #openstack-infra | 08:02 | |
| *** matsuhashi has quit IRC | 08:13 | |
| *** matsuhashi has joined #openstack-infra | 08:14 | |
| *** matsuhashi has quit IRC | 08:15 | |
| *** matsuhashi has joined #openstack-infra | 08:15 | |
| *** osanchez has joined #openstack-infra | 08:24 | |
| *** che-arne has quit IRC | 08:29 | |
| *** che-arne has joined #openstack-infra | 08:34 | |
| *** shardy_afk is now known as shardy | 08:36 | |
| *** sjing has quit IRC | 08:36 | |
| *** sjing has joined #openstack-infra | 08:38 | |
| *** fbo_away is now known as fbo | 08:47 | |
| *** zoresvit has quit IRC | 08:48 | |
| *** sjing has quit IRC | 08:48 | |
| *** hashar has joined #openstack-infra | 08:49 | |
| *** ljjjusti1 has quit IRC | 08:49 | |
| *** zoresvit has joined #openstack-infra | 08:52 | |
| *** ilyashakhat has joined #openstack-infra | 08:52 | |
| *** guohliu has quit IRC | 08:54 | |
| openstackgerrit | Michael Still proposed a change to openstack-infra/elastic-recheck: Add bug 1251920 to our list of bugs we suggest for recheck https://review.openstack.org/56883 | 08:54 |
| uvirtbot | Launchpad bug 1251920 in nova "Tempest failures due to failure to return console logs from an instance" [Undecided,New] https://launchpad.net/bugs/1251920 | 08:54 |
| *** nosnos has quit IRC | 08:59 | |
| *** nosnos has joined #openstack-infra | 08:59 | |
| *** andreaf has quit IRC | 09:00 | |
| *** andreaf has joined #openstack-infra | 09:01 | |
| *** nosnos has quit IRC | 09:03 | |
| *** hashar has quit IRC | 09:05 | |
| *** che-arne has quit IRC | 09:06 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Bump sqlalchemy-migrate to 0.7.3 https://review.openstack.org/56888 | 09:07 |
| *** davidhadas has quit IRC | 09:09 | |
| *** hashar has joined #openstack-infra | 09:09 | |
| *** jpich has joined #openstack-infra | 09:10 | |
| *** Ryan_Lane has quit IRC | 09:10 | |
| *** che-arne has joined #openstack-infra | 09:14 | |
| *** yassine has joined #openstack-infra | 09:18 | |
| *** bingbu has quit IRC | 09:27 | |
| *** rpodolyaka has joined #openstack-infra | 09:29 | |
| mikal | Any adult supervision around? | 09:34 |
| mikal | My elastic recheck patch just failed its check, but the links to logs are 404 | 09:34 |
| mikal | Which is a bit... special | 09:34 |
| *** rpodolyaka has left #openstack-infra | 09:39 | |
| openstackgerrit | Yuuichi Fujioka proposed a change to openstack-dev/hacking: Add metaclass for Python3 compatibility https://review.openstack.org/56890 | 09:41 |
| *** Ryan_Lane has joined #openstack-infra | 09:41 | |
| *** dpyzhov has left #openstack-infra | 09:41 | |
| *** markmc has joined #openstack-infra | 09:43 | |
| *** davidhadas has joined #openstack-infra | 09:44 | |
| *** derekh has joined #openstack-infra | 09:45 | |
| *** saschpe_ has joined #openstack-infra | 09:45 | |
| *** saschpe has quit IRC | 09:46 | |
| *** boris-42 has joined #openstack-infra | 09:49 | |
| *** Ryan_Lane has quit IRC | 09:50 | |
| *** soren has joined #openstack-infra | 09:50 | |
| *** che-arne has quit IRC | 09:50 | |
| *** SergeyLukjanov has quit IRC | 09:51 | |
| *** Dafna has joined #openstack-infra | 09:54 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:54 | |
| *** BobBall is now known as BobBallAway | 09:55 | |
| *** matsuhashi has quit IRC | 09:55 | |
| *** adarazs has joined #openstack-infra | 09:55 | |
| *** matsuhashi has joined #openstack-infra | 09:56 | |
| *** matsuhashi has quit IRC | 10:01 | |
| *** matsuhashi has joined #openstack-infra | 10:05 | |
| *** dizquierdo has joined #openstack-infra | 10:17 | |
| *** xchu has quit IRC | 10:21 | |
| *** marun has quit IRC | 10:21 | |
| *** afazekas has joined #openstack-infra | 10:25 | |
| *** amotoki has quit IRC | 10:37 | |
| openstackgerrit | ChangBo Guo proposed a change to openstack-dev/hacking: Clean up how test env variables are parsed https://review.openstack.org/56898 | 10:37 |
| *** jhesketh has quit IRC | 10:38 | |
| *** yassine has quit IRC | 10:39 | |
| *** dizquierdo has quit IRC | 10:41 | |
| *** marun has joined #openstack-infra | 10:42 | |
| *** marun has quit IRC | 10:46 | |
| *** matsuhashi has quit IRC | 10:47 | |
| *** Ryan_Lane has joined #openstack-infra | 10:47 | |
| *** matsuhashi has joined #openstack-infra | 10:47 | |
| *** nsaje has joined #openstack-infra | 10:49 | |
| openstackgerrit | ChangBo Guo proposed a change to openstack-dev/hacking: Clean up how test env variables are parsed https://review.openstack.org/56898 | 10:49 |
| *** Ryan_Lane has quit IRC | 10:51 | |
| *** zoresvit has quit IRC | 10:51 | |
| *** rfolco has joined #openstack-infra | 10:51 | |
| jd__ | I think Jenkins is having some issues :( | 10:52 |
| *** jhesketh has joined #openstack-infra | 10:52 | |
| jd__ | some tests fail but I can't even retrieve the log | 10:52 |
| *** matsuhashi has quit IRC | 10:52 | |
| *** zoresvit has joined #openstack-infra | 10:53 | |
| *** marun has joined #openstack-infra | 10:53 | |
| *** matsuhashi has joined #openstack-infra | 10:54 | |
| *** fifieldt has joined #openstack-infra | 10:57 | |
| *** nsaje_ has joined #openstack-infra | 10:57 | |
| *** nsaje has quit IRC | 10:57 | |
| *** ArxCruz has joined #openstack-infra | 11:01 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added task ordering https://review.openstack.org/56026 | 11:07 |
| *** johnthetubaguy has joined #openstack-infra | 11:11 | |
| *** yaguang has quit IRC | 11:12 | |
| *** michchap has quit IRC | 11:12 | |
| *** matsuhashi has quit IRC | 11:13 | |
| *** michchap has joined #openstack-infra | 11:13 | |
| *** pblaho has quit IRC | 11:13 | |
| *** matsuhashi has joined #openstack-infra | 11:14 | |
| *** jcoufal has quit IRC | 11:14 | |
| *** lcestari has joined #openstack-infra | 11:14 | |
| *** arata has joined #openstack-infra | 11:15 | |
| *** ilyashakhat has quit IRC | 11:17 | |
| *** ilyashakhat has joined #openstack-infra | 11:18 | |
| *** matsuhashi has quit IRC | 11:19 | |
| *** DinaBelova has quit IRC | 11:19 | |
| *** pcm_ has joined #openstack-infra | 11:20 | |
| *** pcm_ has joined #openstack-infra | 11:20 | |
| *** michchap has quit IRC | 11:22 | |
| *** michchap has joined #openstack-infra | 11:23 | |
| arata | There is something wrong with jenkins01 | 11:23 |
| arata | not resonding. | 11:24 |
| *** SergeyLukjanov has quit IRC | 11:24 | |
| *** yamahata_ has joined #openstack-infra | 11:26 | |
| *** SergeyLukjanov has joined #openstack-infra | 11:27 | |
| *** nsaje_ has quit IRC | 11:29 | |
| *** matsuhashi has joined #openstack-infra | 11:32 | |
| *** dizquierdo has joined #openstack-infra | 11:34 | |
| *** fifieldt has quit IRC | 11:34 | |
| *** nsaje has joined #openstack-infra | 11:35 | |
| *** DinaBelova has joined #openstack-infra | 11:36 | |
| *** ben_duyujie has quit IRC | 11:39 | |
| *** dkranz has quit IRC | 11:41 | |
| *** talluri has quit IRC | 11:47 | |
| *** Ryan_Lane has joined #openstack-infra | 11:48 | |
| openstackgerrit | Soren Hansen proposed a change to openstack-infra/config: Add BasicDB to stackforge https://review.openstack.org/55519 | 11:52 |
| *** Ryan_Lane has quit IRC | 11:52 | |
| *** adalbas has joined #openstack-infra | 11:57 | |
| *** nsaje has quit IRC | 11:59 | |
| *** DinaBelova has quit IRC | 12:01 | |
| *** jcoufal has joined #openstack-infra | 12:01 | |
| *** davidhadas_ has joined #openstack-infra | 12:03 | |
| *** davidhadas has quit IRC | 12:06 | |
| *** yassine has joined #openstack-infra | 12:06 | |
| mordred | arata: I'll look in to it in a bit - dealing with my personal network being bonghits first | 12:12 |
| mordred | -bash: fork: Cannot allocate memory | 12:12 |
| mordred | wow | 12:12 |
| mordred | that's exciting | 12:12 |
| *** michchap has quit IRC | 12:14 | |
| *** michchap has joined #openstack-infra | 12:15 | |
| mordred | ok. jenkins01.openstack.org was in a world of hurt - it could not fork processes, I could not even cat things in /oroc | 12:17 |
| mordred | so I nova reboot-ed it | 12:17 |
| mordred | jd__, arata, mikal ^^ | 12:18 |
| jd__ | thanks mordred | 12:18 |
| *** yamahata_ has quit IRC | 12:19 | |
| * mordred wishes he had more information about why it was broken, but given that /proc was broken, diagnosing was a bit rough :) | 12:20 | |
| arata | mordred: although still getting ssl_error_rx_record_too_long, thanks anyway | 12:23 |
| *** hashar has quit IRC | 12:28 | |
| *** arata has left #openstack-infra | 12:29 | |
| mordred | ah. great. jenkins01 is even more borked than I thought | 12:29 |
| mordred | fungi: when you wake up ^^ | 12:30 |
| mordred | fungi: root@jenkins01:~# puppet agent --test | 12:30 |
| mordred | Exiting; no certificate found and waitforcert is disabled | 12:30 |
| mordred | jenkins01 has gotten REALLY unhappy | 12:30 |
| ttx | mordred: when you have 5 minutes I'd like to exchange a bit on your Thunderbird setup | 12:33 |
| mordred | ttx: I always have 5 minutes for you! | 12:33 |
| *** nsaje has joined #openstack-infra | 12:34 | |
| mordred | ttx: it's really simple, I have a folder for openstack-dev, and in that folder, I got to the View menu, sort by, threaded | 12:34 |
| *** mattymo has quit IRC | 12:34 | |
| ttx | mordred: I already used threaded view but was using "N" to jump to next unread, which still makes the sheer number a pain to process. What do you use and how do you go through it ? | 12:34 |
| mordred | my mouse | 12:34 |
| mordred | also \ is helpful for 'collapse all' | 12:35 |
| ttx | so.. you select interesting threads and expand them ? then once done you mark folder as read ? | 12:35 |
| mordred | ttx: yes. or I mark the thread as read | 12:35 |
| mordred | which is r when the thread is highlighted | 12:35 |
| mordred | that's how I ignore threads I don't care about | 12:36 |
| *** sandywalsh has joined #openstack-infra | 12:36 | |
| ttx | but then it still comes back whenever a new message is posted to it. that said "ignore thread" is a bit extreme | 12:37 |
| mordred | well, yeah - there's 'k' for ignore thread | 12:37 |
| mordred | and 'w' for watch thread | 12:37 |
| mordred | (which I've never used, but am going to) | 12:37 |
| ttx | 't' shows promise | 12:38 |
| ttx | "Go to Next Unread Thread (and mark current thread as read)" | 12:38 |
| ttx | not sure what "watch thread" actually does | 12:38 |
| mordred | me either | 12:38 |
| ttx | do you use starring at all ? | 12:39 |
| mordred | ttx: ah - watch thread is probably not useful to us at al | 12:40 |
| mordred | you can watcha thread, and then switch to a view where you only see watched threads | 12:40 |
| ttx | anyway, most of it is about accepting I can't read every message. My old workflow (using "n") was still pretending to read them all | 12:40 |
| mordred | I do not use starring | 12:40 |
| ttx | which made the prospect of parsing 192 messages in the morning not that exciting | 12:40 |
| mordred | and yeah , I think it's more about accepting that than anything | 12:40 |
| ttx | I recently switched my reading of openstack@l.o.o to "read thread subjects only" and it works well | 12:41 |
| mordred | yah. subject filtering by eyeball is pretty effective | 12:42 |
| ttx | mordred: do you manage to get TB to always start in collapsed threads mode, or do you abuse \ regularly to force it to submission ? | 12:42 |
| mordred | abuse \ | 12:43 |
| ttx | ok, same here | 12:43 |
| *** dkliban has quit IRC | 12:45 | |
| *** hashar has joined #openstack-infra | 12:47 | |
| *** weshay has joined #openstack-infra | 12:48 | |
| *** Ryan_Lane has joined #openstack-infra | 12:48 | |
| *** che-arne has joined #openstack-infra | 12:52 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Remove duplicate entries https://review.openstack.org/56920 | 12:53 |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 12:53 | |
| *** Ryan_Lane has quit IRC | 12:53 | |
| *** _SergeyLukjanov has quit IRC | 12:53 | |
| *** hashar has quit IRC | 12:54 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Enable wheel processing in the tests https://review.openstack.org/56817 | 12:55 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Clean up integration script https://review.openstack.org/56816 | 12:55 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation https://review.openstack.org/48803 | 12:55 |
| *** hashar has joined #openstack-infra | 12:58 | |
| *** dkranz has joined #openstack-infra | 13:03 | |
| *** yamahata_ has joined #openstack-infra | 13:05 | |
| *** matsuhashi has quit IRC | 13:05 | |
| *** matsuhashi has joined #openstack-infra | 13:05 | |
| *** CaptTofu has quit IRC | 13:05 | |
| *** CaptTofu has joined #openstack-infra | 13:06 | |
| *** dkliban has joined #openstack-infra | 13:06 | |
| openstackgerrit | Yassine Lamgarchal proposed a change to openstack-infra/config: Add tooz https://review.openstack.org/56927 | 13:18 |
| *** dkliban has quit IRC | 13:21 | |
| openstackgerrit | Yassine Lamgarchal proposed a change to openstack-infra/config: Add tooz https://review.openstack.org/56927 | 13:22 |
| *** dkliban has joined #openstack-infra | 13:26 | |
| *** dkliban has quit IRC | 13:30 | |
| *** thomasem has joined #openstack-infra | 13:33 | |
| *** nati_ueno has joined #openstack-infra | 13:35 | |
| *** dstanek has joined #openstack-infra | 13:36 | |
| *** nati_ueno has quit IRC | 13:37 | |
| *** nati_ueno has joined #openstack-infra | 13:38 | |
| ArxCruz | mordred: what need to have this https://review.openstack.org/#/c/53432/ merged? :) | 13:39 |
| *** nsaje has quit IRC | 13:39 | |
| ArxCruz | what do I need to do* | 13:40 |
| mordred | ArxCruz: nothing. need another review on it | 13:40 |
| mordred | ArxCruz: we've been short staffed last week, so the review queue is a bit long | 13:40 |
| ArxCruz | fungi: https://review.openstack.org/#/c/53432/ :D ? | 13:40 |
| ArxCruz | mordred: okay :) | 13:40 |
| *** nsaje has joined #openstack-infra | 13:40 | |
| *** nsaje has quit IRC | 13:42 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:44 | |
| *** yamahata_ has quit IRC | 13:47 | |
| *** yamahata_ has joined #openstack-infra | 13:49 | |
| *** Ryan_Lane has joined #openstack-infra | 13:49 | |
| *** Ryan_Lane has quit IRC | 13:54 | |
| *** dprince has joined #openstack-infra | 13:54 | |
| *** matsuhashi has quit IRC | 13:56 | |
| *** lascii is now known as alaski | 13:56 | |
| *** matsuhashi has joined #openstack-infra | 13:57 | |
| *** matsuhas_ has joined #openstack-infra | 13:58 | |
| openstackgerrit | Chuck Short proposed a change to openstack/requirements: Add HACKING.rst https://review.openstack.org/55909 | 13:58 |
| *** matsuhashi has quit IRC | 13:59 | |
| *** sdake__ is now known as sdake | 14:01 | |
| hashar | if any Gerrit folk is around, one of our volunteer found out a WIP plugin for Gerrit : https://github.com/davido/gerrit-wip-plugin (by David Ostrovsky ) | 14:03 |
| hashar | mordred: ^^ | 14:03 |
| mordred | hashar: yup! | 14:04 |
| mordred | hashar: _david_ wrote that for us | 14:04 |
| hashar | \O/ | 14:04 |
| *** ryanpetrello has joined #openstack-infra | 14:04 | |
| mordred | hashar: we need to get one patch landed to upstream gerrit, aiui | 14:04 |
| hashar | now that it is a plugin, we will be able to add it on our installation \O/ | 14:05 |
| openstackgerrit | Yuuichi Fujioka proposed a change to openstack-dev/hacking: Add metaclass for Python3 compatibility https://review.openstack.org/56890 | 14:05 |
| hashar | mordred: ^d and qchris from Wikimedia might be able to assist in review / +1. Although both are working on other projects nowadays. | 14:05 |
| mordred | hashar: w00t! | 14:07 |
| mordred | hashar: I find it very funny that we can't get WIP mainlined | 14:07 |
| mordred | but, that's a topic for beer | 14:08 |
| hashar | I guess Google folks are only interested in maintaining features they are actively using | 14:09 |
| hashar | leaving third parties with plugins. | 14:09 |
| *** jpeeler has joined #openstack-infra | 14:09 | |
| *** julim has joined #openstack-infra | 14:12 | |
| *** cody-somerville has joined #openstack-infra | 14:16 | |
| *** mfer has joined #openstack-infra | 14:16 | |
| *** mriedem has joined #openstack-infra | 14:21 | |
| *** davidhadas_ has quit IRC | 14:22 | |
| *** julim has quit IRC | 14:23 | |
| *** dkliban has joined #openstack-infra | 14:24 | |
| *** julim has joined #openstack-infra | 14:26 | |
| anteaya | fungi teh jenkins is sick | 14:27 |
| anteaya | mordred thinks jenkins01 needs some care | 14:27 |
| *** nsaje has joined #openstack-infra | 14:28 | |
| *** Dafna has quit IRC | 14:28 | |
| *** dkranz has quit IRC | 14:30 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 14:32 | |
| *** _SergeyLukjanov has quit IRC | 14:32 | |
| *** nsaje has quit IRC | 14:32 | |
| *** nsaje has joined #openstack-infra | 14:34 | |
| *** afazekas has quit IRC | 14:36 | |
| *** alcabrera has joined #openstack-infra | 14:37 | |
| *** Loquacity has quit IRC | 14:38 | |
| *** Loquacity has joined #openstack-infra | 14:39 | |
| *** nsaje has quit IRC | 14:44 | |
| *** jamesmcarthur has joined #openstack-infra | 14:45 | |
| *** dkranz has joined #openstack-infra | 14:45 | |
| *** Dafna has joined #openstack-infra | 14:48 | |
| *** afazekas has joined #openstack-infra | 14:49 | |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Run tox tests for pecan to gate against WSME, Ceilometer, and Ironic. https://review.openstack.org/54333 | 14:50 |
| *** xeyed4good has joined #openstack-infra | 14:51 | |
| *** ftcjeff has joined #openstack-infra | 14:53 | |
| *** ftcjeff has quit IRC | 14:55 | |
| *** damnsmith is now known as dansmith | 14:56 | |
| *** matsuhas_ has quit IRC | 14:57 | |
| *** matsuhashi has joined #openstack-infra | 14:57 | |
| fungi | anteaya: having a look | 14:59 |
| *** davidhadas has joined #openstack-infra | 14:59 | |
| *** oubiwann has joined #openstack-infra | 15:01 | |
| *** SergeyLukjanov has joined #openstack-infra | 15:02 | |
| fungi | looks like mordred rebooted it several hours ago | 15:03 |
| mordred | fungi: indeed. but now it's in a land of unhappy | 15:03 |
| fungi | yeah, ssl errors on 443 | 15:03 |
| mordred | fungi: the jenkins service is not in a happy place - AND - running puppet agent --test on it | 15:03 |
| mordred | returns missing cert | 15:03 |
| mordred | which makes me think that it's bonghits | 15:03 |
| *** matsuhashi has quit IRC | 15:04 | |
| *** alcabrera is now known as alcabrera|afk | 15:04 | |
| *** thedodd has joined #openstack-infra | 15:04 | |
| *** nsaje has joined #openstack-infra | 15:04 | |
| *** matsuhashi has joined #openstack-infra | 15:04 | |
| *** SergeyLukjanov has quit IRC | 15:04 | |
| fungi | that does seem very un-right | 15:05 |
| fungi | comparing against jenkins02 now | 15:05 |
| *** jgrimm has joined #openstack-infra | 15:05 | |
| fungi | puppet's fine on jenkins02. good | 15:07 |
| fungi | Exiting; no certificate found and waitforcert is disabled | 15:07 |
| *** alcabrera|afk has quit IRC | 15:07 | |
| fungi | that's so great | 15:07 |
| *** Dafna1 has joined #openstack-infra | 15:08 | |
| *** Dafna has quit IRC | 15:08 | |
| *** Loquacity has quit IRC | 15:08 | |
| *** matsuhashi has quit IRC | 15:08 | |
| *** Loquacity has joined #openstack-infra | 15:09 | |
| *** ryanpetrello_ has joined #openstack-infra | 15:10 | |
| mordred | right? | 15:10 |
| mordred | I mean | 15:10 |
| *** SergeyLukjanov has joined #openstack-infra | 15:10 | |
| mordred | I was really excited by it myself | 15:10 |
| *** ryanpetrello has quit IRC | 15:12 | |
| *** ryanpetrello_ is now known as ryanpetrello | 15:12 | |
| fungi | yeah, i'm trying to determine what, if anything, is up with the puppet cert for it now | 15:14 |
| *** Dafna1 has quit IRC | 15:15 | |
| *** CaptainTacoSauce has joined #openstack-infra | 15:15 | |
| *** rwsu-pto is now known as rwsu | 15:16 | |
| *** alcabrera|afk has joined #openstack-infra | 15:16 | |
| *** alcabrera|afk is now known as alcabrera | 15:16 | |
| *** jpeeler1 has joined #openstack-infra | 15:17 | |
| *** jpeeler has quit IRC | 15:17 | |
| *** pblaho has joined #openstack-infra | 15:19 | |
| mordred | fungi: I blame gremlins | 15:19 |
| *** davidhadas has quit IRC | 15:19 | |
| mordred | fungi: btw - not important, but I self approved this: https://review.openstack.org/#/c/56920/ this morning | 15:21 |
| mordred | fungi: my cleanup of the pbr integration test caused it to start throwing errors on duplicates in the PROJECTS list | 15:21 |
| mordred | I figure, fix the dupllicates, then land the pbr change, then no more duplicates can sneak in - profit! | 15:21 |
| *** herndon has joined #openstack-infra | 15:23 | |
| fungi | puppet cert on jenkins01 seems to validate against the ca, and it's the same ca which signed the 02 cert | 15:24 |
| *** thedodd has quit IRC | 15:26 | |
| *** dcramer_ has joined #openstack-infra | 15:26 | |
| *** Dafna has joined #openstack-infra | 15:29 | |
| mordred | fungi: somehow that makes me less happy | 15:30 |
| fungi | yeah, the syslog also has "puppet-agent[1128]: Did not receive certificate" over and over since reboot | 15:31 |
| anteaya | looking at the test nodes graphic in status.openstack.org/zuul, something happened at about 6am test node time | 15:32 |
| *** dcramer_ has quit IRC | 15:32 | |
| anteaya | was that the reboot perhaps? | 15:32 |
| *** dkranz has quit IRC | 15:33 | |
| *** jpeeler1 is now known as jpeeler | 15:33 | |
| *** jpeeler has joined #openstack-infra | 15:33 | |
| fungi | anteaya: roughly | 15:33 |
| *** datsun180b has joined #openstack-infra | 15:36 | |
| *** mgagne has joined #openstack-infra | 15:37 | |
| anteaya | k | 15:37 |
| *** davidhadas has joined #openstack-infra | 15:39 | |
| *** markwash has quit IRC | 15:40 | |
| fungi | ran through everything in http://docs.puppetlabs.com/pe/latest/trouble_comms.html just to be sure, but it all checks out | 15:41 |
| *** wenlock has joined #openstack-infra | 15:41 | |
| jd__ | fungi, mordred got a minute to review https://review.openstack.org/#/c/56927/ ? the devs would like to start pushing patches soon :) | 15:42 |
| fungi | doing a debsums run on it now to see if maybe we've got any obviously corrupted executables | 15:44 |
| fungi | nothing out of the ordinary there | 15:46 |
| anteaya | jd__: fungi will be awhile he is tending to the jenkins | 15:46 |
| *** dkranz has joined #openstack-infra | 15:46 | |
| jd__ | bad bad Jenkins :) | 15:48 |
| anteaya | or something | 15:48 |
| anteaya | so far no clues what is causing the sickness | 15:48 |
| fungi | well, more like too many clues, leading in different directions | 15:48 |
| anteaya | oh | 15:48 |
| anteaya | even worse | 15:48 |
| anteaya | what are you seeing | 15:49 |
| fungi | here's a possible bite... puppet.conf was changed today at 09:23z | 15:50 |
| fungi | i bet unattended upgrades are to blame | 15:51 |
| anteaya | oh | 15:51 |
| anteaya | that would bring us to our knees | 15:51 |
| anteaya | can we revert the puppet.conf change? | 15:51 |
| fungi | nope. today's didn't run on that server until 15:42z | 15:52 |
| fungi | dime to start diffing | 15:52 |
| anteaya | huh | 15:52 |
| fungi | er, time | 15:52 |
| anteaya | k | 15:52 |
| *** dcramer_ has joined #openstack-infra | 15:54 | |
| fungi | somehow it got certname=undef | 15:54 |
| fungi | should have its fqdn in there | 15:54 |
| anteaya | is that an easy correction? | 15:55 |
| fungi | yeah, now i'm trying to figure out how it happened though | 15:56 |
| fungi | and here we go... | 15:56 |
| fungi | Nov 18 09:23:42 jenkins01 puppet-agent[1123]: (/Stage[main]/Openstack_project::Base/File[/etc/puppet/puppet.conf]/content) content changed '{md5}db139fe80a2db5dfe350a9a5eaa0ccae' to '{md5}bf3bd06b3c035a33167619450190f861' | 15:56 |
| * fungi checks for recently approved changes to that file | 15:57 | |
| anteaya | I am not seeing a change to openstack-infra/config for Nov 18 | 16:01 |
| * anteaya isn't sure where else a change to puppet.conf could come from | 16:02 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/reviewstats: Add storyboard project https://review.openstack.org/56972 | 16:03 |
| *** senk has joined #openstack-infra | 16:04 | |
| *** julim has quit IRC | 16:04 | |
| *** julim has joined #openstack-infra | 16:05 | |
| *** jergerber has joined #openstack-infra | 16:05 | |
| *** pblaho1 has joined #openstack-infra | 16:06 | |
| *** dripton has quit IRC | 16:07 | |
| *** nsaje has quit IRC | 16:08 | |
| *** pblaho has quit IRC | 16:08 | |
| zaro | mgagne: jjb release is eminent. | 16:08 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added task ordering https://review.openstack.org/56026 | 16:09 |
| *** mrodden has joined #openstack-infra | 16:09 | |
| zaro | mgagne: https://review.openstack.org/#/c/56823 | 16:09 |
| *** dripton has joined #openstack-infra | 16:10 | |
| *** SergeyLukjanov has quit IRC | 16:13 | |
| *** saschpe_ has quit IRC | 16:13 | |
| *** saschpe has joined #openstack-infra | 16:14 | |
| fungi | anteaya: mordred: doesn't look like anything changed in our puppet config to cause it... seems this was puppet spontaneously breaking itself by falling back on a (presumably very old) cached config... http://paste.openstack.org/show/53536/ | 16:15 |
| fungi | anteaya: mordred: my best guess so far is that jenkins01 lost contact with the puppet master somehow for an extended period between 08:43 and 09:23, but possibly related to a too-many-open-processes issue? (note the cron can't fork error in there) | 16:16 |
| fungi | er, either too many open file descriptors or too many processes | 16:17 |
| * anteaya clicks | 16:17 | |
| fungi | would be my guess | 16:17 |
| anteaya | any way of know how many processes were running on that jenkins at the time? | 16:18 |
| fungi | not unless mordred took notes before he rebooted it | 16:18 |
| fungi | could also have been an oom condition, but i'd have expected to see that reflected in the logs | 16:18 |
| *** afazekas has quit IRC | 16:19 | |
| zaro | horray for multiple masters! | 16:20 |
| anteaya | zaro: I'm wondering if this means we need more | 16:21 |
| anteaya | and a script that sets a maximum of processes per master | 16:21 |
| zaro | memory setting might be problem, always is with java. | 16:21 |
| anteaya | if we don't already have that | 16:21 |
| *** katyafervent has joined #openstack-infra | 16:22 | |
| fungi | anteaya: we have separate issues as a cascading effect from this which is causing us to be starved for d-g slaves (note the purported deleting, building and in use states on the graph many of which i suspect are tied to the dead jenkins master) | 16:23 |
| zaro | fungi: all jobs that were running on that master were probably lost. does logs show what was running? | 16:23 |
| *** dprince has quit IRC | 16:23 | |
| *** boris-42 has quit IRC | 16:23 | |
| fungi | and also causing some jobs to be blocking the gate because they never reported back as finished, i think | 16:23 |
| fungi | zaro: it might, i haven't looked yet | 16:23 |
| ryanpetrello | there was talk at the summit of moving to wheels for pypi uploads | 16:24 |
| ryanpetrello | I'm guessing stackforge projects will be doing this too? | 16:24 |
| anteaya | fungi yes I have been watching that graph with interest | 16:24 |
| ryanpetrello | wondering what steps packages should start taking to prepare, e.g., https://gist.github.com/kennethreitz/7489603 | 16:24 |
| zaro | ohh wait, doesn't zuul keep those jobs or at least tell us which jobs timed out? | 16:24 |
| *** dkranz has quit IRC | 16:25 | |
| *** afazekas has joined #openstack-infra | 16:25 | |
| anteaya | ryanpetrello mordred is working on wheels | 16:25 |
| ryanpetrello | yea, I'll poke him about it | 16:26 |
| ryanpetrello | just want to do what I can to prepare :) | 16:26 |
| fungi | okay, jenkins01 is back up and running jobs again. going to see what i can do to start cleaning up the deleting (in nodepool)/offline (in jenkins) d-g slave mess now | 16:28 |
| *** pabelanger has quit IRC | 16:29 | |
| anteaya | ryanpetrello: good plan | 16:30 |
| *** krotscheck has joined #openstack-infra | 16:30 | |
| *** rnirmal has joined #openstack-infra | 16:31 | |
| *** pabelanger has joined #openstack-infra | 16:33 | |
| *** pabelanger has quit IRC | 16:33 | |
| *** pabelanger has joined #openstack-infra | 16:33 | |
| *** ben_duyujie has joined #openstack-infra | 16:33 | |
| *** ^d has joined #openstack-infra | 16:33 | |
| pabelanger | fungi, anteaya: You can disable puppet from use the cached catalog to stop that exact behaviour. | 16:34 |
| *** afazekas has quit IRC | 16:34 | |
| anteaya | do you have a command for that? | 16:34 |
| anteaya | or is that a config option? | 16:34 |
| pabelanger | usecacheonfailure = false | 16:35 |
| pabelanger | for agents | 16:35 |
| anteaya | thank you | 16:35 |
| *** dstanek has quit IRC | 16:35 | |
| *** ben_duyujie has quit IRC | 16:35 | |
| fungi | pabelanger: awesome! so i guess "puppet agent eats itself" is a known issue with cached configs and keeping your puppet.config in puppet | 16:37 |
| pabelanger | fungi, ya, I hit that same issue many moons ago. My nodes couldn't download the catalog for some reason, puppet basically nuked my production boxes systematically across my network. Not a fun day that was explaining it to customers | 16:38 |
| fungi | of course, if usecacheonfailure = false is an option which goes in the agent config, we won't be able to apply it with puppet without blowing away the certname option on all of them at the same time | 16:38 |
| fungi | so adding that option will take some planning, i suspect | 16:39 |
| *** dstanek has joined #openstack-infra | 16:39 | |
| pabelanger | why do you need to blow away certname? | 16:39 |
| *** reed has joined #openstack-infra | 16:39 | |
| fungi | we don't need to. that's just what i expect puppet will do if we make a change to that file | 16:39 |
| * pabelanger goes to check how puppet.conf is built | 16:39 | |
| fungi | right now the base one in our config repo has certname=undef in it | 16:40 |
| annegentle | jeblair: around? | 16:40 |
| fungi | and i suspect changing any part of that file will set them all to undef when it reapplies the updated file | 16:40 |
| *** dkranz has joined #openstack-infra | 16:41 | |
| pabelanger | fungi, Oh, I see | 16:41 |
| *** mdenny has joined #openstack-infra | 16:41 | |
| annegentle | fungi: just got another reminder email from Holly at O'Reilly, should we just schedule something and hope jeblair can make it? | 16:42 |
| fungi | annegentle: i'm uncomfortable doing that--he seemed particularly displeased that i didn't push them to fully integrate with our existing systems on that first call where i was the only infra representative, so i'd really rather not repeat that without him around to make his desires very clear | 16:43 |
| annegentle | fungi: yep, makes sense. oh jeblair jeblair where are you? | 16:43 |
| *** UtahDave has joined #openstack-infra | 16:44 | |
| *** hashar has quit IRC | 16:46 | |
| *** danger_fo_away is now known as dangers | 16:47 | |
| mgagne | zaro: cool =) | 16:47 |
| fungi | still over 100 nodes showing in a delete status in nodepool after jenkins01 has had a chance to settle, and the graph suggests they all ended up in that state following its nosedive earlier, so i'm going to 'nodepool delete' those | 16:51 |
| *** branen has joined #openstack-infra | 16:51 | |
| zaro | mgagne: if you have time i would really like this to merge.. https://review.openstack.org/#/c/56715 | 16:55 |
| mgagne | zaro: checking | 16:55 |
| zaro | mgagne: what did gf think of the food you ate? | 16:55 |
| *** pblaho1 has quit IRC | 16:57 | |
| *** krotscheck has quit IRC | 16:57 | |
| mgagne | zaro: was ok with it, I guess she was hoping for much more, especially desserts :P | 16:58 |
| *** UtahDave has quit IRC | 16:58 | |
| *** markmc has quit IRC | 16:59 | |
| *** SergeyLukjanov has joined #openstack-infra | 17:01 | |
| *** weshay has quit IRC | 17:02 | |
| *** dkranz has quit IRC | 17:02 | |
| mgagne | zaro: my coworker found a bakery in central hk but I couldn't find it again later. We do have some chinese bakeries here but it's probably not the same =( | 17:04 |
| *** hogepodge has joined #openstack-infra | 17:05 | |
| mgagne | zaro: jenkins disagrees: https://review.openstack.org/#/c/56715/ | 17:07 |
| zaro | mgagne: egg tarts were the thing in HK, but are kinda difficult to find these days. not sure what's up with that :( | 17:07 |
| Alex_Gaynor | mordred: pbr now has more installs from pypi than psycopg2 :] | 17:08 |
| Alex_Gaynor | err, than pymongo | 17:08 |
| *** nsaje has joined #openstack-infra | 17:08 | |
| *** nsaje has quit IRC | 17:11 | |
| *** nsaje has joined #openstack-infra | 17:12 | |
| *** pblaho has joined #openstack-infra | 17:13 | |
| *** dkranz has joined #openstack-infra | 17:14 | |
| *** julim has quit IRC | 17:15 | |
| jlk | it's all the hipster python coders thinking it's a beer library | 17:15 |
| dstufft | I thought all the hipsters used node | 17:16 |
| jlk | was more a joke about "PBR" as in Pabst Blue Ribbon | 17:17 |
| *** hogepodge has quit IRC | 17:18 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/reviewstats: Add storyboard project https://review.openstack.org/56972 | 17:18 |
| dstufft | jlk: :] | 17:18 |
| *** julim has joined #openstack-infra | 17:19 | |
| *** senk has quit IRC | 17:20 | |
| openstackgerrit | Kevin proposed a change to openstack-infra/jenkins-job-builder: publishers: added runAfterFinalised to copy-to-master https://review.openstack.org/56989 | 17:20 |
| openstackgerrit | Kevin proposed a change to openstack-infra/jenkins-job-builder: Added template-test command & run-after-finalized https://review.openstack.org/54993 | 17:25 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: update doc and add new JJB unit tests https://review.openstack.org/56715 | 17:26 |
| *** pblaho has quit IRC | 17:26 | |
| *** pblaho has joined #openstack-infra | 17:27 | |
| fungi | ttx: did you want to push signed folsom-eol tags on the tip of stable/folsom for that list of projects i posted to the openstack-stable-maint ml last week, or should someone else do it (one of the other stable release managers)? | 17:27 |
| *** xeyed4good has quit IRC | 17:27 | |
| *** guohliu has joined #openstack-infra | 17:28 | |
| *** pblaho has quit IRC | 17:31 | |
| *** pblaho has joined #openstack-infra | 17:31 | |
| *** jcoufal has quit IRC | 17:31 | |
| *** pblaho has quit IRC | 17:32 | |
| *** pblaho has joined #openstack-infra | 17:33 | |
| *** dprince has joined #openstack-infra | 17:33 | |
| *** pblaho has quit IRC | 17:33 | |
| *** dizquierdo has quit IRC | 17:36 | |
| *** dizquierdo has joined #openstack-infra | 17:37 | |
| *** flaper87 is now known as flaper87|afk | 17:37 | |
| *** dizquierdo has quit IRC | 17:38 | |
| *** dstanek has quit IRC | 17:39 | |
| *** jpich has quit IRC | 17:40 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Add storyboard project https://review.openstack.org/56972 | 17:42 |
| salv-orlando | I do apologise for disturbing, but if I can get another infra-core reviewing this: https://review.openstack.org/#/c/56722/ we might be able to do some serious progress towards neutron parallel testing (doing that locally now, but it would be useful testing against the gate as well) | 17:42 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add storyboard project https://review.openstack.org/56972 | 17:42 |
| *** yassine has quit IRC | 17:45 | |
| *** hogepodge has joined #openstack-infra | 17:48 | |
| *** dangers is now known as danger_fo | 17:49 | |
| *** osanchez has quit IRC | 17:51 | |
| *** markwash has joined #openstack-infra | 17:51 | |
| *** danger_fo is now known as dangers | 17:51 | |
| salv-orlando | thanks fungi! | 17:52 |
| fungi | salv-orlando: no problem. i definitely don't want to stand in the way of neutron testing/gating improvements | 17:52 |
| *** johnthetubaguy has quit IRC | 17:54 | |
| clarkb | morning | 17:55 |
| *** pmathews has joined #openstack-infra | 17:55 | |
| clarkb | poor jenkins01 | 17:56 |
| olaph | morning clarkb | 17:56 |
| openstackgerrit | A change was merged to openstack-infra/config: Add job for neutron parallel testing w/tenant isolation https://review.openstack.org/56722 | 17:56 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: fix jjb scp publisher example https://review.openstack.org/56998 | 17:57 |
| *** dcramer_ has quit IRC | 17:58 | |
| fungi | clarkb: now that the deleting nodes are mostly cleared, i'm starting to become less and less convinced that the available count is correct either | 17:58 |
| *** harlowja has joined #openstack-infra | 17:59 | |
| pleia2 | good morning | 17:59 |
| clarkb | fungi: interesting | 17:59 |
| *** derekh has quit IRC | 18:00 | |
| clarkb | fungi: is it convinced it has more nodes than actually exist? | 18:00 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: fix jjb scp publisher example https://review.openstack.org/56998 | 18:00 |
| fungi | clarkb: i suspect so--haven't checked yet. also jenkins01 has hundreds of what i assume are old devstack slaves in an offline state | 18:01 |
| fungi | whereas jenkins02 has maybe half a dozen | 18:01 |
| clarkb | fungi: may need to manually GC those. I can help with that if we determine that nodepool has forgotten about them | 18:01 |
| fungi | i wanted to wait until the deletes finished so i could confirm | 18:02 |
| clarkb | wfm | 18:02 |
| fungi | i think those are just abotu wrapped up now | 18:02 |
| *** Dafna has left #openstack-infra | 18:04 | |
| markwash | can I use gerrit to track two major-release series of python-glanceclient? Like, suppose I want to create a 1.0 branch and start making backwards incompatible changes. . . | 18:06 |
| markwash | meanwhile maintaining a 0.0 branch | 18:06 |
| *** sarob has joined #openstack-infra | 18:07 | |
| clarkb | markwash: you can create branches in gerrit like that and tag commits on them with arbitrary tags. So gerrit won't enforce what you describe but would allow it. Doing so would change glanceclient's release process compared to the other clients (not necessarily a bad thing but something to consider) | 18:10 |
| *** gyee has joined #openstack-infra | 18:10 | |
| clarkb | fungi I know you have thought a lot about python client release and compatiblity schemes. any thoughts on ^ | 18:10 |
| markwash | clarkb: so the way to get gerrit to track another branch for python-glanceclient is just to push a branch to the gerrit remote? and who will end up with core powers on that branch? | 18:11 |
| fungi | clarkb: markwash: well, getting a release management opinion on it would help (ttx?) | 18:11 |
| *** alexpilotti has quit IRC | 18:11 | |
| *** melwitt has joined #openstack-infra | 18:12 | |
| fungi | but also, keep in mind that it multiplies what testign we do, and we'd need to decide how the two independent versions of glanceclient would be tested adequately | 18:12 |
| pleia2 | zaro: thanks for the jjb docs updates \o/ | 18:12 |
| markwash | fungi: yeah, it seems there are lot of complicating implications | 18:12 |
| *** nsaje has quit IRC | 18:12 | |
| fungi | markwash: clarkb: and then there's the vulnerability management concern of having to track backports for security fixes which affect both branches | 18:13 |
| *** nsaje has joined #openstack-infra | 18:13 | |
| clarkb | fungi: I knew there was a reason I asked for your thoughts :) | 18:13 |
| clarkb | but yeah changing snowflakes usually comes with all sorts of fun stuff to work through | 18:13 |
| fungi | there are almost certainly more issues which aren't springing to mind just yet | 18:14 |
| clarkb | markwash: gerrit ACLs are weird and in this case you would create the branch directly in gerrit. the release mgmt team would have rights to non master branches iirc | 18:14 |
| clarkb | so more reason to bring this up with ttx | 18:15 |
| zaro | pleia2: np. i've actually found a lot more of those. it was driving me crazy because jjb doesn't tell you that anything is wrong. | 18:15 |
| *** dstanek has joined #openstack-infra | 18:17 | |
| *** nsaje has quit IRC | 18:18 | |
| *** ArxCruz has quit IRC | 18:23 | |
| *** Loquacity has quit IRC | 18:26 | |
| *** Loquacity has joined #openstack-infra | 18:26 | |
| *** guohliu has quit IRC | 18:26 | |
| *** dstanek has quit IRC | 18:30 | |
| *** nsaje has joined #openstack-infra | 18:32 | |
| clarkb | fungi: let me know if I can help with the jenkins situation. also https://review.openstack.org/#/c/56720/ for when jenkins is happy | 18:34 |
| *** oubiwann has quit IRC | 18:35 | |
| *** oubiwann has joined #openstack-infra | 18:35 | |
| *** mrmartin has joined #openstack-infra | 18:36 | |
| fungi | clarkb: deletes seem to have finished, so i'm going to manually gc jenkins01 offline devstack nodes starting from the bottom of the list and working my way up (since nodepool lists no "delete" state nodes on 01 now) | 18:36 |
| *** Ryan_Lane has joined #openstack-infra | 18:37 | |
| clarkb | fungi: I can start at the top | 18:37 |
| fungi | clarkb: awesome--thanks! | 18:37 |
| openstackgerrit | Brant Knudson proposed a change to openstack-infra/reviewstats: Morgan Fainberg is Keystone core https://review.openstack.org/57008 | 18:39 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Morgan Fainberg is Keystone core https://review.openstack.org/57008 | 18:42 |
| openstackgerrit | Vijendar Komalla proposed a change to openstack/requirements: Update python-troveclient version https://review.openstack.org/52137 | 18:42 |
| openstackgerrit | Hunter Haugen proposed a change to openstack-infra/config: Pass $mysql_password through to gerrit class https://review.openstack.org/56306 | 18:43 |
| *** alcabrera is now known as alcabrera|afk | 18:49 | |
| *** pabelanger has quit IRC | 18:54 | |
| clarkb | fungi: this is quite the manual task. I probably should learn to use the jenkins api for things like this | 18:54 |
| *** pabelanger has joined #openstack-infra | 18:54 | |
| mordred | Alex_Gaynor: wow. that's pretty crazy | 18:55 |
| Alex_Gaynor | mordred: 22nd most downloaded package of all time, it'll overtake psycopg2 in a few days | 18:55 |
| *** hogepodge has quit IRC | 18:56 | |
| *** alcabrera|afk is now known as alcabrera | 18:57 | |
| *** Guest79045 is now known as Vivek | 18:57 | |
| *** Vivek has quit IRC | 18:57 | |
| *** Vivek has joined #openstack-infra | 18:57 | |
| *** hogepodge has joined #openstack-infra | 18:57 | |
| *** rnirmal has quit IRC | 18:58 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: fix jjb scp publisher example https://review.openstack.org/56998 | 19:00 |
| *** vipul is now known as vipul-away | 19:01 | |
| *** sarob has quit IRC | 19:01 | |
| *** sarob has joined #openstack-infra | 19:01 | |
| *** weshay has joined #openstack-infra | 19:04 | |
| *** sarob has quit IRC | 19:06 | |
| clarkb | Alex_Gaynor: mordred: that almost makes me wonder if our test slaves are hitting pypi.python.org :) | 19:06 |
| mordred | clarkb: makes me wonder something similar | 19:08 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: update doc and add new JJB unit tests https://review.openstack.org/56715 | 19:08 |
| Alex_Gaynor | I think maybe people just use this openstack thing a lot | 19:08 |
| clarkb | fungi: I left those slaves in the az1 670k series there because those seem new enough taht nodepool should know about them | 19:10 |
| clarkb | fungi: but they aren't going away veryquickly either | 19:10 |
| fungi | clarkb: okay | 19:10 |
| fungi | otherwise, seems we've met in the middle | 19:10 |
| *** vipul-away is now known as vipul | 19:10 | |
| EmilienM | hi, i would like a review on Neutron Metering agent in devstack > https://review.openstack.org/#/c/48042/ | 19:10 |
| *** hogepodge has quit IRC | 19:11 | |
| mordred | Alex_Gaynor: bah. I hear CloudStack is where it's at | 19:11 |
| clarkb | EmilienM: https://review.openstack.org/#/admin/groups/50,members is the group of folks that can give you +2s | 19:11 |
| *** hogepodge has joined #openstack-infra | 19:12 | |
| fungi | EmilienM: also the devstack developers tend to be more numerous on the #openstack-qa channel | 19:12 |
| clarkb | mordred: now that they have a code review system and one apparent way to submit patches we should watch out :) (seriously though good for them changing the code submission process) | 19:12 |
| EmilienM | clarkb: fungi : i already asked on qa channel, this patch takes a while to merge, i don't understand | 19:12 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1251516 https://review.openstack.org/57016 | 19:12 |
| mordred | clarkb: I had no idea that they'd done any of this | 19:12 |
| uvirtbot | Launchpad bug 1251516 in tempest "test_get_console_output fails with "Console output was empty"" [Undecided,New] https://launchpad.net/bugs/1251516 | 19:12 |
| *** senk has joined #openstack-infra | 19:13 | |
| clarkb | mordred: I checked recently because I was complaining to someone about how bad it was (email patch, attach to jira, pull request) and was told that isn't the case anymore and sure enough they use reviewboard iirc | 19:13 |
| EmilienM | clarkb: can i add you as reviewer ? | 19:13 |
| clarkb | EmilienM: sure, but I can't help approve that patch | 19:13 |
| mordred | well good or them | 19:13 |
| mordred | for | 19:13 |
| clarkb | I don't have +2 | 19:13 |
| EmilienM | ok | 19:14 |
| clarkb | EmilienM: really you need another devstack-core to take a look at it | 19:14 |
| EmilienM | clarkb: thx :) | 19:14 |
| EmilienM | i'll ping them | 19:14 |
| clarkb | fungi: it is really weird that jenkins01 has so few slaves now. I guess we need to wait for nodepool to shift pool growth from jenkins02 to jenkins01 | 19:15 |
| fungi | clarkb: perhaps. i think we may also be waiting for nodepool to notice that all those slaves on jenkins01 weren't really available | 19:16 |
| anteaya | jog0 around today? | 19:16 |
| fungi | clarkb: since there are plenty of devstack jobs queued but the test nodes graph claims ~150-200 nodes available | 19:16 |
| *** herndon has quit IRC | 19:17 | |
| *** derekh has joined #openstack-infra | 19:19 | |
| clarkb | fungi: jenkins02 has a large number of offline nodes too, though most of those look recent | 19:19 |
| clarkb | I wonder if we should kick nodepool | 19:19 |
| jog0 | anteaya: yup | 19:20 |
| clarkb | oooohhh you know what | 19:20 |
| clarkb | I wonder if hpcloud is hating us right now | 19:20 |
| *** jamesmcarthur has quit IRC | 19:20 | |
| *** hogepodge has quit IRC | 19:21 | |
| clarkb | the rackspace nodes seem to be coming up and doing work but there aren't as many hpcloud nodes as I would expect | 19:21 |
| *** jamesmcarthur has joined #openstack-infra | 19:21 | |
| clarkb | fungi: mordred: yup there seem to be some fun stacktraces in the nodepool debug log. ssh timeouts and RAM limits exceeded | 19:23 |
| clarkb | I wonder if we have attempted to allocate a bunch of servers that are not coming up within the timeout and as a result are hitting quota limits | 19:23 |
| anteaya | jog0: prepping for the neutron meeting and the tempest topic: https://wiki.openstack.org/wiki/Network/Meetings#Neutron_Tempest_.28anteaya.29 | 19:23 |
| fungi | clarkb: possible | 19:23 |
| anteaya | can we discuss this in -neutron so we have current info, using logstash? | 19:23 |
| fungi | also possible some of those entries are me running nodepool delete too quickly and hitting api rate limit throttles | 19:24 |
| jog0 | anteaya: sure | 19:24 |
| *** dcramer_ has joined #openstack-infra | 19:24 | |
| fungi | clarkb: i'm in favor of giving nodepool a restart after what we've just put it through. should be nondisruptive | 19:25 |
| *** jerdfelt has joined #openstack-infra | 19:25 | |
| clarkb | fungi: it shouldn't hurt but I don't think it will help either | 19:25 |
| mordred | clarkb: wow. that's fantastic | 19:26 |
| clarkb | instead I think we may want to bump the ssh timeout and see if that fixes the quota problems | 19:26 |
| jerdfelt | clarkb: hi. did i remember on Friday you said that rackspace already had a third-party testing account in gerritt? | 19:26 |
| fungi | i think we need to compare the nodes it thinks are available with ones which actually exist in jenkins | 19:26 |
| *** dstanek has joined #openstack-infra | 19:26 | |
| clarkb | jerdfelt: I am not sure if mikal has that configured yet, but I believe it was/is his eventual goal | 19:26 |
| jerdfelt | clarkb: i was trying to track down the account information. but i didn't see anything at https://review.openstack.org/#/admin/groups/91,members that looked like rackspace | 19:27 |
| fungi | clarkb: the "delete" status node count associated with jenkins02 is falling, so i suspect nodepool is working through those okay | 19:27 |
| jerdfelt | i've also sent an email to mikal to find out what he knows and has done | 19:27 |
| *** blamar has quit IRC | 19:28 | |
| *** sarob has joined #openstack-infra | 19:28 | |
| clarkb | fungi: cool. I see more hpcloud nodes on jenkins02 now | 19:28 |
| clarkb | jerdfelt: they may not have gotten that far then | 19:29 |
| clarkb | fungi: still not many on jenkins01, may still be working through the shift of load | 19:29 |
| fungi | clarkb: i wonder if it would be wise to put jenkins01 in shutdown mode, then nodepool delete all nodes associated with it, then bring it back up and do the same on jenkins02 | 19:29 |
| jerdfelt | clarkb: is it possible that it may not be listed in that gerritt group or is that group accurate? | 19:30 |
| *** hogepodge has joined #openstack-infra | 19:31 | |
| clarkb | jerdfelt: the group should be accurate | 19:31 |
| fungi | jerdfelt: that group is how those accounts get permission to do a verify column vote | 19:31 |
| *** dstanek has quit IRC | 19:31 | |
| jerdfelt | clarkb, fungi: cool, thanks. i'll get to the bottom of what mikal has done. thanks! | 19:32 |
| fungi | jerdfelt: jhesketh may also have some awareness of what's going on there, not sure | 19:32 |
| *** jamesmcarthur has quit IRC | 19:33 | |
| jerdfelt | fungi: i'll follow up with him too | 19:34 |
| clarkb | fungi: I think that may be a good idea. There is a large discrepency between ready nodes in nodepool and available slaves in jenkins | 19:35 |
| fungi | clarkb: since jenkins01 isn't pulling its weight yet anyway, let's hit it fist i guess? | 19:36 |
| fungi | er, hit it first | 19:36 |
| clarkb | fungi: yup | 19:36 |
| fungi | okay, putting jenkins01 into shutdown mode. i fear the longer things run at the current limited capacity the bigger the pile-up in zuul will become | 19:38 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: update doc and add new JJB unit tests https://review.openstack.org/56715 | 19:38 |
| clarkb | ya best to start fixing this now | 19:38 |
| * mordred in meeting - only partially useful - ping me if I can be of help | 19:38 | |
| fungi | once jenkins01 finishes quiescing, i'll nodepool delete any of its remaining nodes nodepool knows about and then manually delete any straggler d-g nodes through the jenkins ui, then fire it back up | 19:39 |
| clarkb | sounds good | 19:40 |
| *** ArxCruz has joined #openstack-infra | 19:40 | |
| jog0 | mordred: do you want to start a ML thread about the discussion over the weekend (re: that its bad that we rely on recheck so much) | 19:40 |
| mordred | jog0: kinda. I also kinda want jeblair to tell me I'm dumb a little bit first | 19:40 |
| *** jcooley_ has joined #openstack-infra | 19:41 | |
| jog0 | mordred: when does jeblair return? | 19:41 |
| *** dcramer_ has quit IRC | 19:41 | |
| mordred | jog0: he arrived back in the US a few hours ago | 19:42 |
| mordred | so, hopefully tomorrow? | 19:42 |
| *** senk has quit IRC | 19:43 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add bug 1251920 to our list of bugs we suggest for recheck https://review.openstack.org/56883 | 19:43 |
| uvirtbot | Launchpad bug 1251920 in nova "Tempest failures due to failure to return console logs from an instance" [Medium,Confirmed] https://launchpad.net/bugs/1251920 | 19:43 |
| jog0 | mordred: cool | 19:45 |
| clarkb | man that zuul backlog is impressive | 19:45 |
| jog0 | clarkb: whao just looked what happened? | 19:45 |
| clarkb | jog0: jenkins01 fell over and we are still dealing with the fallout | 19:46 |
| clarkb | fungi: I don't see any hpcloud nodes running jobs on jenkins01, you can probably safely start deleting hpcloud nodes on jenkins01 | 19:46 |
| jog0 | clarkb: ahhh | 19:46 |
| fungi | clarkb: good call. doing now | 19:46 |
| *** senk has joined #openstack-infra | 19:48 | |
| *** sarob has quit IRC | 19:50 | |
| *** sarob has joined #openstack-infra | 19:50 | |
| zaro | mordred: have time for this one? https://review.openstack.org/#/c/56823 | 19:50 |
| *** afazekas has joined #openstack-infra | 19:51 | |
| *** SergeyLukjanov has quit IRC | 19:52 | |
| *** sarob_ has joined #openstack-infra | 19:52 | |
| *** sarob has quit IRC | 19:52 | |
| *** jamesmcarthur has joined #openstack-infra | 19:52 | |
| fungi | clarkb: i split the jenkins01 hpcloud node list 5 ways and have parallel deletes running on them | 19:53 |
| fungi | so should hopefully finish fairly quickly | 19:54 |
| *** dcramer_ has joined #openstack-infra | 19:54 | |
| *** hogepodge has quit IRC | 19:54 | |
| *** nsaje has quit IRC | 19:55 | |
| *** nsaje has joined #openstack-infra | 19:55 | |
| *** dstanek has joined #openstack-infra | 19:58 | |
| *** nsaje has quit IRC | 20:00 | |
| *** mestery_ has joined #openstack-infra | 20:01 | |
| *** sarob_ has quit IRC | 20:02 | |
| jamesmcarthur | fungi: thanks for the feedback on https://review.openstack.org/#/c/53644/ Just an FYI Sebastian is on vacation for the next week, so it might be a bit before we get back to you to address your concerns. I think the Redis version was something Sebastian addressed earlier, but I'll have to discuss with him to get the details. | 20:02 |
| *** sarob has joined #openstack-infra | 20:02 | |
| *** mestery has quit IRC | 20:02 | |
| *** dstanek has quit IRC | 20:03 | |
| clarkb | fungi: down to 25 jenkins01 hpcloud nodes | 20:03 |
| *** mgagne has quit IRC | 20:03 | |
| *** nsaje has joined #openstack-infra | 20:04 | |
| *** nsaje has quit IRC | 20:04 | |
| fungi | jamesmcarthur: right, if memory serves we were looking to switch it to use the ubuntu packages for redis, but the current puppet module in that change seems to want to install redis from source instead i think | 20:04 |
| *** mgagne has joined #openstack-infra | 20:04 | |
| *** mgagne has quit IRC | 20:04 | |
| *** mgagne has joined #openstack-infra | 20:04 | |
| *** nsaje has joined #openstack-infra | 20:05 | |
| jamesmcarthur | fungi: yeah, i believe that's correct | 20:05 |
| *** jcooley_ has quit IRC | 20:05 | |
| fungi | jamesmcarthur: but anyway, mostly i just wanted to give the current state of the change an initial test drive to spot where it's broken, so we can start to iterate through it once he's back at the wheel there | 20:06 |
| clarkb | fungi: the rackspace nodes are done running tests on jenkins01 too | 20:06 |
| jamesmcarthur | fungi: roger that. thank you sir! | 20:06 |
| fungi | clarkb: cool, i'll start popping them as well | 20:06 |
| *** rfolco has quit IRC | 20:06 | |
| *** sarob has quit IRC | 20:07 | |
| *** mestery_ is now known as mestery | 20:08 | |
| *** senk has quit IRC | 20:09 | |
| *** sandywalsh has quit IRC | 20:09 | |
| *** nsaje has quit IRC | 20:09 | |
| clarkb | fungi: there appears to be one stubborn hpcloud node associated with jenkins01 | 20:09 |
| clarkb | oh it is gone now \o/ | 20:09 |
| fungi | clarkb: i saw it and added it to the pile for this round | 20:10 |
| *** dcramer_ has quit IRC | 20:13 | |
| *** senk has joined #openstack-infra | 20:13 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1251512 https://review.openstack.org/57038 | 20:17 |
| uvirtbot | Launchpad bug 1251512 in tempest "test_get_console_output fails in gate with MismatchError" [Undecided,New] https://launchpad.net/bugs/1251512 | 20:17 |
| *** rnirmal has joined #openstack-infra | 20:18 | |
| clarkb | fungi: down to 8 rax slaves | 20:20 |
| fungi | still a couple delete threads going but i think they're almost done | 20:21 |
| *** sandywalsh has joined #openstack-infra | 20:21 | |
| *** jgrimm has quit IRC | 20:21 | |
| *** sdake_ has quit IRC | 20:21 | |
| *** yolanda has quit IRC | 20:22 | |
| *** herndon_ has joined #openstack-infra | 20:24 | |
| fungi | this seems to have done wonders for the available nodes accuracy on the graph now | 20:26 |
| *** jamesmcarthur has quit IRC | 20:27 | |
| *** dcramer_ has joined #openstack-infra | 20:27 | |
| openstackgerrit | Edward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better https://review.openstack.org/51425 | 20:27 |
| *** dprince has quit IRC | 20:30 | |
| *** jamesmcarthur has joined #openstack-infra | 20:31 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Run df after gate https://review.openstack.org/56298 | 20:32 |
| clarkb | fungi: woot | 20:33 |
| clarkb | fungi: there is one extra node on jenkins 01 | 20:33 |
| clarkb | the UI looks fine though so jsut needs to be killed in nodepool | 20:33 |
| *** hogepodge has joined #openstack-infra | 20:35 | |
| *** hashar has joined #openstack-infra | 20:36 | |
| *** hogepodge has quit IRC | 20:36 | |
| clarkb | fungi: woot we are down to 0 now | 20:36 |
| clarkb | fungi: should we reenable jenkins01? | 20:36 |
| *** derekh has quit IRC | 20:36 | |
| *** nsaje has joined #openstack-infra | 20:37 | |
| *** nsaje has quit IRC | 20:37 | |
| *** nsaje has joined #openstack-infra | 20:38 | |
| fungi | yeah, all clear, deletes finished | 20:38 |
| fungi | reenabling now | 20:38 |
| fungi | once nodepool builds it back up to a reasonable level of use, i'll do the same to jenkins02 | 20:39 |
| clarkb | ++ I am going to run out and grab lunch and stuff quickly while we wait for things to shift | 20:40 |
| fungi | k | 20:40 |
| *** nsaje has quit IRC | 20:42 | |
| *** sarob has joined #openstack-infra | 20:44 | |
| *** jcooley_ has joined #openstack-infra | 20:45 | |
| *** jcooley_ has quit IRC | 20:46 | |
| fungi | we've got 70 nodes on jenkins01 now | 20:53 |
| *** mrmartin has quit IRC | 20:53 | |
| fungi | or at least associated with it, probably still all building since i don't see them in its webui just yet | 20:53 |
| *** Ryan_Lane has quit IRC | 20:55 | |
| fungi | looks like they're starting to appear in the webui and run jobs now | 20:55 |
| *** Ryan_Lane has joined #openstack-infra | 20:55 | |
| anteaya | it has been a colourful test node graph day | 20:56 |
| mordred | clarkb, fungi: thank you for fixing that - sorry I've been absentish | 20:56 |
| *** hogepodge has joined #openstack-infra | 20:57 | |
| anteaya | -neutron meeting is about to start | 20:58 |
| *** denis_makogon_ has joined #openstack-infra | 20:59 | |
| *** marun has quit IRC | 20:59 | |
| *** marun has joined #openstack-infra | 21:00 | |
| *** thedodd has joined #openstack-infra | 21:01 | |
| *** amotoki has joined #openstack-infra | 21:02 | |
| *** denis_makogon_ is now known as denis_makogon | 21:03 | |
| mikal | So... I'd really like to try and sneak https://review.openstack.org/#/c/56158/ in soon as it will unblock some stuff we want to do. Who do I need to beg to do jeepyb reviews? | 21:03 |
| clarkb | mikal: I can take a look since you are so important in openstack | 21:06 |
| mikal | clarkb: "kind of big in openstack" you mean | 21:07 |
| mikal | clarkb: and thanks | 21:07 |
| fungi | mikal: now you just need a shirt which says "i'm kind of a big deal in openstack" | 21:08 |
| mikal | fungi: that is already in progress... | 21:08 |
| mikal | I was thinking I should have one for the next summit | 21:09 |
| pabelanger | fungi, , I thought russellb was the only big deal? | 21:14 |
| russellb | o.O | 21:14 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1251512 https://review.openstack.org/57038 | 21:15 |
| uvirtbot | Launchpad bug 1251512 in tempest "test_get_console_output fails in gate with MismatchError" [Undecided,New] https://launchpad.net/bugs/1251512 | 21:15 |
| *** dstanek has joined #openstack-infra | 21:17 | |
| *** julim has quit IRC | 21:19 | |
| lifeless | mordred: jenkins hates you https://review.openstack.org/#/c/54152/5 | 21:19 |
| fungi | i suspect it's mutual | 21:19 |
| mikal | mriedem / jog0: that bug (1251512) looks lik a special case of 1251920 to me | 21:20 |
| mikal | I've been seeing that failure an marking it as 1251920 when I see it | 21:20 |
| mikal | Until we can get better console log debugging in the libvirt driver | 21:20 |
| fungi | i think a good litmus test for jenkins02 shutdown/cleanup will be once demand starts to fall for centos6 slaves, since that'll mean that we're mostly blocking on devstack slave capacity at that point | 21:25 |
| *** hogepodge has quit IRC | 21:25 | |
| *** ilyashakhat has quit IRC | 21:26 | |
| fungi | any earlier and we'll be hurting gate throughput by holding up jobs which could have used unit test slaves associated with jenkins02 | 21:26 |
| *** ilyashakhat has joined #openstack-infra | 21:26 | |
| *** senk has quit IRC | 21:27 | |
| *** jhesketh has quit IRC | 21:27 | |
| fungi | and after that, i'll take a look at the state of alien nodes which look like something nodepool once created. we might have some of those wasting part of our quotas, i suspect | 21:29 |
| *** alcabrera has quit IRC | 21:30 | |
| fungi | yeah, looks like a couple dozen. maybe i'll just go ahead and clean those up now | 21:32 |
| clarkb | fungi: I don't think we need to worry too much about waiting that long | 21:33 |
| clarkb | better to get this done with than drag it out imo | 21:33 |
| lifeless | zuul is a bit sick atm right ? | 21:35 |
| fungi | lifeless: it's mainly better now, just catching up (and there are a few "stuck" jobs i think from earlier when jenkins01 went awol) | 21:36 |
| fungi | er, stuck changes i mean | 21:36 |
| lifeless | kk thanks | 21:36 |
| lifeless | lots of gate-tempest-devstack-vm-full: queued | 21:36 |
| lifeless | that means we've hit our quota right ? | 21:37 |
| clarkb | lifeless: sort of, jenkins01 went sideways and nodepool added a bunch of capacity that the jnekins slaves can't see | 21:37 |
| *** jamesmcarthur has quit IRC | 21:37 | |
| clarkb | so we are at "capacity" according to nodepool but not jenkins. we are slowly fixing that | 21:37 |
| fungi | lifeless: when jenkins01 tanked, it left nodepool thinking we had a lot of available machines even though we didn't. that's cleaned up now and new slaves are being spun up and put to work as fast as we can manage | 21:37 |
| lifeless | gotchya | 21:38 |
| lifeless | is nodepool single threaded in that regard? | 21:38 |
| lifeless | or concurrent? | 21:38 |
| clarkb | concurrent | 21:38 |
| clarkb | well its python threads | 21:39 |
| fungi | however our providers also limit how quickly we can make api calls | 21:39 |
| openstackgerrit | Michael Still proposed a change to openstack-infra/elastic-recheck: Tell people to do a recheck https://review.openstack.org/56118 | 21:39 |
| fungi | and we further throttle that so that we don't get smacked down by their rate limiting, i believe | 21:40 |
| clarkb | fungi: I would say put jenkins02 in shutdown mode now so that load starts shifting, the currently running jobs will take a bit of time to get through anyways | 21:40 |
| fungi | will do | 21:40 |
| *** jcooley_ has joined #openstack-infra | 21:43 | |
| openstackgerrit | Vijendar Komalla proposed a change to openstack/requirements: Update python-troveclient version https://review.openstack.org/52137 | 21:44 |
| *** blamar has joined #openstack-infra | 21:50 | |
| clarkb | mikal: reviewed | 21:52 |
| *** ryanpetrello has quit IRC | 21:53 | |
| *** wenlock has quit IRC | 21:54 | |
| *** wenlock has joined #openstack-infra | 21:55 | |
| *** emagana has joined #openstack-infra | 21:56 | |
| clarkb | fungi: the results queue in zuul is a little scary (we should keep an eye on that and make sure it falls at some point | 21:56 |
| clarkb | I mean it should right? we don't have enough slaves to run all of the tests :) | 21:56 |
| *** jhesketh has joined #openstack-infra | 21:57 | |
| *** melwitt has quit IRC | 21:57 | |
| reed | the core-non-spider discussion gets more and more complicated | 21:57 |
| *** melwitt has joined #openstack-infra | 21:57 | |
| zaro | mordred: hope this works for you.. https://review.openstack.org/#/c/56760 | 21:58 |
| *** vipul is now known as vipul-away | 21:59 | |
| clarkb | fungi: it just fell by about 200 events | 21:59 |
| *** wenlock has quit IRC | 22:00 | |
| *** wenlock has joined #openstack-infra | 22:00 | |
| clarkb | fungi: only rax slaves are running jobs on 02 now. if you want to start deleting the hpcloud nodes | 22:00 |
| *** nati_uen_ has joined #openstack-infra | 22:00 | |
| fungi | i am doing so now | 22:01 |
| mriedem | mikal: yeah, different failures though | 22:01 |
| mriedem | in one case the output is empty, in one it's not the number of lines expected | 22:01 |
| clarkb | reed: which discussion? I am failing at context switching right now | 22:01 |
| reed | the badly titled "Re: [Openstack] [Foundation Board] Resolutions from the Technical Committee" | 22:02 |
| *** mfer has quit IRC | 22:02 | |
| *** dkliban has quit IRC | 22:02 | |
| zaro | clarkb, fungi: there's a major fix to jenkins that reduces the number of threads by 75%. do you think we should upgrade just for that? | 22:03 |
| clarkb | zaro: I think we should, but I don't think I want to deal with that while sorting out the nodepool problems | 22:03 |
| fungi | zaro: i definitely think we should discuss it, and probably test it out on jenkins-dev at least | 22:03 |
| clarkb | zaro: I think we should spend some time this week and attempt to upgrade if everyone is happy with it though | 22:03 |
| * fungi agrees | 22:03 | |
| clarkb | in fact I can upgrade jenkins-dev right now if we want | 22:04 |
| *** melwitt has quit IRC | 22:04 | |
| *** nati_ueno has quit IRC | 22:04 | |
| zaro | is upgrading jenkins-dev something i can help with? | 22:04 |
| clarkb | let me do that so that we have that step out of the way | 22:04 |
| *** melwitt has joined #openstack-infra | 22:04 | |
| clarkb | zaro: you can follow along over my shoulder | 22:04 |
| zaro | clarkb: coolio. | 22:04 |
| clarkb | zaro: pleia2: the basic process isn't too exciting. stop puppet on jenkins-dev, copy jenkins war (it is in /usr/share/jenkins) to a version specific war that we can use to roll back if necessary, apt-get update, apt-get install jenkins and I think that restarts the service | 22:05 |
| clarkb | then restart puppet, really the only thing I do that is special is backup the old war for super easy revert | 22:06 |
| clarkb | I am sure apt magic can make that easy enough as well but apt magic is dark voodoo | 22:06 |
| *** ryanpetrello has joined #openstack-infra | 22:06 | |
| fungi | all hpcloud devstack nodes associated with jenkins02 have been nodepool deleted | 22:10 |
| *** vipul-away is now known as vipul | 22:13 | |
| clarkb | oh stop jenkins on jenkins-dev before updating it because it doesn't always stop cleanly | 22:13 |
| reed | oh, more opinions on the mailing list split! | 22:13 |
| clarkb | I am stoping jenkins on jenkins-dev now | 22:13 |
| clarkb | fungi: ^ | 22:13 |
| fungi | clarkb: sounds good | 22:13 |
| clarkb | fungi: pleia2: zaro and I have noted a couple things from the release notes of jenkins that may bite us. There are new REST API permissions around creating, deleting and updating nodes. and the ssh-slaves and credentials plugins which are bundled in jenkins have been upgraded a couple times | 22:15 |
| fungi | too fun | 22:16 |
| clarkb | trying to sort out how to run the jenkins-test script in the config repo | 22:21 |
| fungi | i've not done it | 22:21 |
| *** sarob_ has joined #openstack-infra | 22:22 | |
| clarkb | it looks simple, just trying to sort out what sort of config it wants | 22:23 |
| *** CaptainTacoSauce has quit IRC | 22:23 | |
| *** sarob has quit IRC | 22:25 | |
| *** sarob_ has quit IRC | 22:27 | |
| fungi | jenkins02 is idle now. nodepool deleting the remaining nodes from it | 22:31 |
| mriedem | are we using different cirros images lately in the full gate jobs? | 22:31 |
| clarkb | mriedem: I think we cache them once a day. If upstream cirros has updated we wuold get that within 24 hours | 22:32 |
| mriedem | clarkb: ok, looks like the latest cirros release was february 8th, so i guess nothing recent there | 22:34 |
| mriedem | how about libvirt? | 22:34 |
| mriedem | do we use 1.1.4? | 22:35 |
| clarkb | fungi: that test script relies on old d-g and zuul which had interaction with jenkins. need to rewrite it to use nodepool libs instead. I am not going to worry about that now as there are more pressing things happening. Starting puppet on jenkins-dev now | 22:35 |
| fungi | k | 22:36 |
| clarkb | mriedem: libvirt we get from cloud archive iirc | 22:37 |
| *** dangers is now known as danger_fo_away | 22:37 | |
| clarkb | fungi: down to two nodes on 02 | 22:37 |
| mriedem | clarkb: ok, trying to figure out what if anything could be recently changing to cause so many test_get_console_output failures lately, it got real flaky in the last week or so | 22:38 |
| fungi | clarkb: yeah, just one delete thread still running and on its last entry | 22:38 |
| *** jcooley_ has quit IRC | 22:38 | |
| clarkb | mriedem: it is possible that libvirt upgraded for the havana release in cloud archive | 22:39 |
| fungi | clarkb: jenkins02 nodepool list is now empty, and the aliens-list for devstack-precise-<providers> has been manually cleaned out as well | 22:39 |
| *** nsaje has joined #openstack-infra | 22:39 | |
| *** mriedem has quit IRC | 22:40 | |
| fungi | ready for me to start jenkins02? | 22:40 |
| clarkb | fungi: woot, go for it | 22:40 |
| fungi | hopefully things will start to pick up again now once it gets some new nodes | 22:40 |
| *** ryanpetrello has quit IRC | 22:44 | |
| *** xeyed4good has joined #openstack-infra | 22:44 | |
| *** nsaje has quit IRC | 22:44 | |
| clarkb | fungi: then if zuul catches up we can restart it to have it forget about those changes that have been around for 10 hours | 22:46 |
| *** lcestari has quit IRC | 22:47 | |
| *** loq_mac has joined #openstack-infra | 22:47 | |
| *** julim has joined #openstack-infra | 22:49 | |
| clarkb | fungi: so uh | 22:50 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: fix jjb job template documenation https://review.openstack.org/57062 | 22:50 |
| clarkb | we seem to have run into the same problem again | 22:50 |
| clarkb | maybe nodepool really does need to be kicked | 22:51 |
| *** rcleere has joined #openstack-infra | 22:52 | |
| clarkb | fungi: I think the event stream from jenkins01 may not be working as expected | 22:54 |
| clarkb | fungi: looking at logstash to see if I can find any jenkins01 jobs | 22:54 |
| clarkb | fungi: I think that is the problem | 22:56 |
| clarkb | netstat shows tcp connections are established though | 22:57 |
| clarkb | fungi: I am getting the events locally | 22:59 |
| *** pcm_ has quit IRC | 23:00 | |
| mordred | zaro: thank you | 23:00 |
| clarkb | so the event stream is fine, I think it must be on the client side | 23:00 |
| mgagne | zaro: which job/script is testing import order in python projects? I suspect JJB isn't tested against it. | 23:00 |
| *** julim has quit IRC | 23:01 | |
| *** thomasem has quit IRC | 23:02 | |
| *** fifieldt has joined #openstack-infra | 23:03 | |
| *** amotoki has quit IRC | 23:04 | |
| clarkb | fungi: I think we should restart nodepoold | 23:05 |
| clarkb | then hit everything with a hammer again (though maybe more aggressively and deal with the fallout) | 23:05 |
| zaro | mgagne: import order? not sure what you mean. | 23:06 |
| * clarkb goes ahead and restarts nodepoold | 23:07 | |
| mgagne | zaro: I think it's a lint/pep8 check | 23:07 |
| zaro | mgagne: while you are here. can you please take another look at https://review.openstack.org/#/c/56715 | 23:07 |
| mgagne | zaro: "import sys\nimport os" would throw an error about import not being alphabetically ordered. | 23:07 |
| fungi | mmm | 23:08 |
| clarkb | fungi: it has been restarted | 23:08 |
| fungi | okay | 23:08 |
| clarkb | fungi: I think what was happening is gearman plugin put the slaves in offline mode | 23:08 |
| zaro | mgagne: hmm.. i think pep8 is run against jjb project. let me check. | 23:08 |
| clarkb | fungi: but nodepool never saw the nodes as being used because the finished events were not coming through | 23:09 |
| clarkb | mgagne: zaro: pep8 is run but not hacking | 23:09 |
| mgagne | zaro: or is it hacking ? | 23:09 |
| mgagne | zaro: right | 23:09 |
| mgagne | clarkb, zaro: how hard would it be to add it? | 23:09 |
| clarkb | mgagne: depends on how unclean jjb is :) | 23:10 |
| fungi | clarkb: makes sense. and explains the giant mass of "available" nodes doing nothing but offline in jenkins | 23:10 |
| clarkb | changing the tox.ini is simple, but all of the failures need to be fixed too | 23:10 |
| zaro | mgagne: not difficult to add, but what clarkb said. | 23:10 |
| clarkb | fungi: so now I think we should put jenkins01 in shutdown mode again, wait for it to finish then kill all of its nodes | 23:10 |
| clarkb | fungi: or just start manually pewpewing nodes on that host | 23:10 |
| fungi | clarkb: we ought to be able to nodepool delete them this time, i think | 23:10 |
| clarkb | fungi: ok | 23:10 |
| mgagne | zaro: would fixing hacking failures worth it? | 23:10 |
| * fungi ponders | 23:11 | |
| clarkb | fungi: it looks like doing hpcloud is safe | 23:11 |
| mgagne | zaro: or we add hacking as a non-voting job until failures are fixed? | 23:11 |
| clarkb | but not rax | 23:11 |
| clarkb | mgagne: zaro: I would just do it in one go | 23:11 |
| zaro | mgagne: what did you want from hacking? | 23:11 |
| clarkb | rather than wait for all failures to be fixed | 23:11 |
| mgagne | zaro: import order is bothering me :D | 23:12 |
| fungi | clarkb: i think all the "ready" nodes in nodepool could just be blown away at this point. checking that assumption now | 23:12 |
| *** dkranz has quit IRC | 23:12 | |
| *** hogepodge has joined #openstack-infra | 23:12 | |
| zaro | mgagne: hmmm, i think i can help with that. but don't want to do it myself since it doesn't bother me ;) | 23:12 |
| fungi | maybe not. 86 in a "building" state, 288 "ready" | 23:13 |
| mgagne | zaro: if someone can explain me how to enable hacking, I don't mind taking a look | 23:14 |
| *** ^d is now known as ^demon|busy | 23:14 | |
| fungi | and a lot more "used" in jenkins than nodepool seems to think | 23:14 |
| *** beekneemech is now known as bnemec | 23:14 | |
| zaro | mgagne: first clone openstack-infra/config | 23:14 |
| blamar | anyone know off the top of their head if it's possible to escape braces ({}) in jjb templates so they don't get substituted? will dive into the code next… | 23:14 |
| mgagne | zaro: next: bootstrap whole CI infra on your laptop :D | 23:14 |
| mgagne | blamar: {{}} | 23:14 |
| clarkb | blamar: yes, use {{}} | 23:14 |
| blamar | ty | 23:15 |
| blamar | ty | 23:15 |
| blamar | all of you! | 23:15 |
| fungi | clarkb: so i think nodepool delete blindly on ready nodes is probably not safe, since i get the impression some of those may actually be doing work | 23:16 |
| fungi | best to go back through putting jenkins01 in shutdown i think | 23:16 |
| zaro | mgagne: haha! it's not that bad. | 23:16 |
| clarkb | fungi: ok | 23:16 |
| fungi | then i can safely delete any nodes associated with it (again) | 23:16 |
| zaro | mgagne: look in modules/opesntack_project/files/zuul/layout.yaml | 23:17 |
| *** julim has joined #openstack-infra | 23:17 | |
| mgagne | zaro: ok, the idea is to look for examples in existing projects =) | 23:17 |
| clarkb | fungi: when it is in shutdown mode and we are waiting you should review the havana jobs change :) | 23:18 |
| zaro | mgagne: yup. that's where all the projects are, there are already many that use hacking. just add it to jjb. | 23:18 |
| fungi | note to self. when a jenkins master goes toes-up, nodepool can lose track of the event stream and needs a restart | 23:18 |
| morganfainberg | did we decide at the summit that 1 core fore transifex is the general policy? | 23:19 |
| clarkb | fungi: fwiw it shouldn't do that, 0mq is supposed to retry on an interval and according to netstat it had an established connection | 23:19 |
| morganfainberg | i remember something along those lines | 23:19 |
| clarkb | possible bug in 0mq or pyzmq | 23:19 |
| clarkb | morganfainberg: that sounds right, you basically just want ot have a sanity check that only translation files were updated instead of code and so on | 23:19 |
| *** hogepodge has quit IRC | 23:20 | |
| morganfainberg | clarkb, figured as much. *pokes jenkins with a stick again* we have one w/ like 33k changes... i personally want to have those go in a bit more frequently :P | 23:20 |
| *** yamahata_ has quit IRC | 23:21 | |
| clarkb | fungi: I think it is safe to start deleting hpcloud jenkins01 slaves | 23:22 |
| *** datsun180b has quit IRC | 23:22 | |
| zaro | mgagne: arrg!!! good catch on that filename. i'm really bad with english. | 23:23 |
| mgagne | zaro: if you think I'm gooder than you ;) | 23:24 |
| clarkb | fungi: then if we finally get this all to settle out, I can restart zuul tonight once things have caught up assuming they catch up | 23:25 |
| *** denis_makogon has quit IRC | 23:26 | |
| *** vipul is now known as vipul-away | 23:28 | |
| fungi | k | 23:29 |
| *** yjiang5 has joined #openstack-infra | 23:29 | |
| fungi | deleting hpcloud nodes from jenkins01 now | 23:29 |
| mgagne | guys, finalised or finalized ? | 23:31 |
| clarkb | use the zed look | 23:32 |
| clarkb | *luke wow I fail hard | 23:32 |
| *** vipul-away is now known as vipul | 23:32 | |
| mgagne | google says 2.5M vs 13.2M results :O | 23:32 |
| clarkb | mordred: fungi: ordering the new hpcloud zone changes | 23:35 |
| clarkb | mordred: fungi: I think the one that adds az support to nodepool needs to go in first? then we can update the nodepool config | 23:35 |
| clarkb | does that sound right? | 23:35 |
| pabelanger | mordred, reading your comment on https://review.openstack.org/#/c/56371/ always wondered why tox -edocs was never created | 23:37 |
| pabelanger | vs people adding a Makefile | 23:37 |
| fungi | mgagne: depends on which side of the atlantic you source your english | 23:38 |
| mgagne | fungi: is JJB british or american? :D | 23:38 |
| clarkb | it is both! I am pretty sure we don't nit pick those things | 23:38 |
| mgagne | clarkb: lets support both configs! :D | 23:39 |
| clarkb | oh is this for the config itself? then I guess this matters in different ways | 23:39 |
| clarkb | for docs I just go meh | 23:39 |
| fungi | mgagne: well, he was written by americans but has a british name, so could go either way | 23:39 |
| clarkb | fungi: LinuxJedi wrote the first pass | 23:39 |
| clarkb | and he speaks the wrong english | 23:39 |
| fungi | d'oh, right! | 23:39 |
| fungi | er, i mean, wrong! | 23:40 |
| fungi | i would tend to gravitate to config options which don't suffer from alternate spellings, personally. pick a different word if possible | 23:40 |
| *** boris-42 has joined #openstack-infra | 23:41 | |
| fungi | though if it's mirroring a config option in jenkins, then do whatever it does i guess | 23:41 |
| mgagne | fungi: jenkins config is finalised, jjb doc/jjb config/commit is finalized ^^' | 23:42 |
| fungi | yeah, in this case i'd do what jenkins does | 23:42 |
| mgagne | alright, thanks! | 23:42 |
| fungi | but i'm just one voice, after all | 23:42 |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/devstack-gate: Bump grenade master to use stable/havana. https://review.openstack.org/57066 | 23:42 |
| *** vipul is now known as vipul-away | 23:44 | |
| *** vipul-away is now known as vipul | 23:44 | |
| *** weshay has quit IRC | 23:44 | |
| fungi | nodepool delete is about halfway through the hpcloud nodes on jenkins01 | 23:45 |
| *** zjdriver has joined #openstack-infra | 23:46 | |
| fungi | as soon as the last couple rax jobs finish i'll start some threads deleting those too | 23:46 |
| *** boris-42 has quit IRC | 23:47 | |
| *** thedodd has quit IRC | 23:51 | |
| openstackgerrit | Edward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better https://review.openstack.org/51425 | 23:51 |
| *** vipul is now known as vipul-away | 23:54 | |
| *** vipul-away is now known as vipul | 23:54 | |
| *** vipul is now known as vipul-away | 23:55 | |
| *** vipul-away is now known as vipul | 23:55 | |
| wenlock | mordred, that commit works behind our firewall .... if you can review it when you get some time | 23:55 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: use jjb tests as the examples https://review.openstack.org/57068 | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!