| jeblair | jhesketh, clarkb: there are two issues: the any/all thing caused changes to show up in gate that did not have jenkins+1. it's fairly harmless | 00:00 |
|---|---|---|
| jeblair | jhesketh, clarkb: the getting stuck is caused by the fix for things getting stuck, and yeah, i'm working on reproing | 00:00 |
| jhesketh | jeblair: right, good find | 00:00 |
| jhesketh | sorry I havne't been able to help more with debugging | 00:01 |
| jeblair | jhesketh: (though note we did revert the any/all thing until we can straighten that out) | 00:01 |
| ianw | clarkb / EmilienM: sure, some generic way to copy logs is fine. i just don't think devstack-gate cleanup is it. for a start, it's got all that grenade $BASE crap in there that i don't think works with the posted review | 00:01 |
| jhesketh | jeblair: yep, saw that. And given the confusion in the scrollback looks like it needs some work anyway | 00:01 |
| jeblair | jhesketh: np | 00:01 |
| *** achanda has quit IRC | 00:01 | |
| dstanek | ianw: yep, that would do it | 00:01 |
| clarkb | ianw: ya my suggestion was to change it to taking a prefix | 00:01 |
| clarkb | ianw: then that could default to the $BASE stuff | 00:01 |
| ianw | clarkb / EmilienM: i think i'd prefer to see that copied out and cleaned up into a much smaller slave_script that doesn't bother with the devstack stuff | 00:03 |
| clarkb | ianw: thats fine too | 00:03 |
| ianw | clarkb / EmilienM : yes it's some duplication ,but it's all pretty trivial stuff, and gives good separation of logic | 00:03 |
| EmilienM | I can do that | 00:03 |
| clarkb | as a third alternative we could use d-g to do the setup | 00:03 |
| *** otter768 has quit IRC | 00:03 | |
| clarkb | and cleanup | 00:03 |
| clarkb | but run puppet in the middle instead of devstack | 00:03 |
| clarkb | but we may have sailed way beyond that ship at this point | 00:04 |
| EmilienM | yeah | 00:04 |
| EmilienM | clarkb: that's a great idea though | 00:04 |
| ianw | clarkb: yeah, i think that a generic log collecting script can be useful in more places, too | 00:04 |
| EmilienM | imagine devstack able to pull our integration script | 00:04 |
| EmilienM | ianw, clarkb, crinkle: so can I go for a slave bash script in project-config that take some piece of code from d-g and do the cleanup? If so I'll make the script flexible so anyone can re-use it maybe | 00:05 |
| ianw | EmilienM: i'd like to see you just copy cleanup_host into a slave script, and go through it and cut out all the devstack/grendade stuff so you're just getting pretty generic openstack logs | 00:06 |
| ianw | remove all the $BASE/old/new switching stuff, etc, that isn't relevant | 00:07 |
| EmilienM | ianw: ok | 00:07 |
| EmilienM | clarkb: does that work for you too? | 00:07 |
| *** dims_ has joined #openstack-infra | 00:08 | |
| *** austin81 has left #openstack-infra | 00:11 | |
| jeblair | clarkb, jhesketh, fungi: reproduced! the trigger is a change in the queue with a non-succeeding build for a job that is removed. | 00:12 |
| jeblair | (in the most recent case, the non-succeeding value was NOT_REGISTERED) | 00:12 |
| jeblair | (but FAILED will do as well) | 00:12 |
| clarkb | EmilienM: ianw ya that seems like a reasonable place to start | 00:13 |
| EmilienM | clarkb: ack | 00:13 |
| EmilienM | thanks infra' | 00:13 |
| fungi | jeblair: wow, i didn't realize we remove jobs from zuul with that great a frequency, but with as many as we have i shouldn't be surprised | 00:16 |
| *** SumitNaiksatam has quit IRC | 00:17 | |
| *** bharath has quit IRC | 00:18 | |
| *** glauco has joined #openstack-infra | 00:18 | |
| *** oomichi has joined #openstack-infra | 00:21 | |
| *** esmute_ has quit IRC | 00:22 | |
| *** nelsnelson has joined #openstack-infra | 00:22 | |
| Shrews | reed: this message is suspicious "that fixed IP already has a floating IP on external network" | 00:23 |
| *** geoffarnold has quit IRC | 00:23 | |
| *** glauco has quit IRC | 00:23 | |
| *** esmute_ has joined #openstack-infra | 00:23 | |
| Shrews | reed: looks like you're trying to associate a floating IP with a port that already has a floating IP. i can't verify that from the output you pasted | 00:23 |
| reed | Shrews, yeah, my brain is starting to melt ... the good thing is that using auto_IP=True creates a machine and assisgns a floating IP | 00:23 |
| *** maurosr has joined #openstack-infra | 00:23 | |
| reed | if I set auto_IP=False and try to assign an IP then it fails... | 00:24 |
| reed | i'll do more tests tomorrow and keep good track of everything | 00:24 |
| Shrews | reed: please do. if you can create a reproducible test case, please forward that to me | 00:24 |
| *** zz_ja has joined #openstack-infra | 00:24 | |
| *** esmute_ has quit IRC | 00:26 | |
| reed | Shrews, try http://paste.openstack.org/show/406521/ on DreamHost | 00:26 |
| reed | this is what I got running that file: http://paste.openstack.org/show/406520/ | 00:26 |
| reed | I have to go now, i'm done for the day | 00:28 |
| Shrews | reed: i've been done. :) will try it tomorrow | 00:28 |
| *** tiswanso has joined #openstack-infra | 00:32 | |
| *** pvaneck has quit IRC | 00:32 | |
| *** jasondotstar has quit IRC | 00:33 | |
| *** abitha has quit IRC | 00:34 | |
| *** aukhan has joined #openstack-infra | 00:39 | |
| *** Sukhdev_ has quit IRC | 00:39 | |
| openstackgerrit | Dmitry Borodaenko proposed openstack-infra/project-config: Add non voting fuel agent python jobs https://review.openstack.org/204711 | 00:41 |
| *** tiswanso has quit IRC | 00:41 | |
| *** mtanino has quit IRC | 00:44 | |
| pleia2 | thingee: for the sake of publicly logged record, I've disabled "StorPool distributed storage CI" as requested on https://etherpad.openstack.org/p/ELGSLX0T9U | 00:45 |
| *** romcheg has quit IRC | 00:46 | |
| pleia2 | ...and I just noticed the mailing list post where anteaya said the same, so apparently someone else got to it first | 00:46 |
| *** wyatt has joined #openstack-infra | 00:47 | |
| pleia2 | anyway, etherpad is updated so hopefully we're all on the same page now :) | 00:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Fix bug with removing a failed job https://review.openstack.org/207675 | 00:47 |
| *** Somay has quit IRC | 00:48 | |
| jeblair | fungi, clarkb, jhesketh: ^ there's a fix | 00:49 |
| jeblair | fungi, clarkb, jhesketh: i've verified that passes both pep8 and py27 locally | 00:50 |
| jhesketh | jeblair: looking | 00:50 |
| *** kfoo has joined #openstack-infra | 00:50 | |
| *** wyatt has quit IRC | 00:50 | |
| *** puranamr has joined #openstack-infra | 00:51 | |
| clarkb | I am reviewing too | 00:52 |
| *** yamamoto has joined #openstack-infra | 00:54 | |
| StevenK | clarkb: | 00:55 |
| jhesketh | jeblair: lgtm | 00:55 |
| StevenK | Conference wifi is conference wifi | 00:55 |
| *** yamamoto has quit IRC | 00:55 | |
| *** thorst has joined #openstack-infra | 00:55 | |
| *** yamamoto has joined #openstack-infra | 00:56 | |
| *** puranamr has quit IRC | 00:56 | |
| jeblair | clarkb, jhesketh: i'll work on applying that manually to zuul.o.o and restarting | 00:56 |
| jhesketh | noted | 00:57 |
| *** boris-42 has joined #openstack-infra | 00:57 | |
| *** thorst has quit IRC | 00:57 | |
| StevenK | clarkb: There's no jeepyb on the proposal slave -- I can move ZanataRestService and the registeration code into jeepyb -- or am I going about this the wrong way? | 00:57 |
| clarkb | StevenK: things that consume projects.yaml are generally expected to by jeepyb | 00:58 |
| clarkb | *to be | 00:58 |
| *** kfoo has quit IRC | 00:58 | |
| jeblair | restarting zuul | 00:59 |
| clarkb | so basically if we remove the job from a project then cancel and remove any exisitng instances of that job from items | 00:59 |
| clarkb | jeblair: shouldn't line 1202 be if build and build.result not if build.job and build.rsult? | 01:00 |
| jeblair | clarkb: yeah, we were canceling, but leaving the builds in the buildset (and therefore report) cause "why not?", but that was before we actually _acted_ on them :) | 01:00 |
| *** SumitNaiksatam has joined #openstack-infra | 01:00 | |
| StevenK | clarkb: Adding translate: true to projects.yaml was also agreed to in an infra meeting, did I see you object to it? | 01:00 |
| jeblair | clarkb: gah, that's from an earlier attempt, i think it can simply not change now. thanks. | 01:00 |
| jeblair | i think that will be harmless in production though, so we don't need to abort the restart | 01:00 |
| clarkb | StevenK: I was informed that its useful because having all projects in zanata will make life hard for translator | 01:01 |
| clarkb | jeblair: ok | 01:01 |
| *** ybathia has quit IRC | 01:01 | |
| StevenK | clarkb: Yes, not every project requires translations | 01:02 |
| clarkb | StevenK: my bigger concern is coming up with a second pattern for which there are a known set of things to do | 01:02 |
| clarkb | jeepyb + projects.yaml + puppet == update projects in gerrit and storyboard | 01:02 |
| clarkb | (and on git mirrors) | 01:02 |
| *** jasondotstar has joined #openstack-infra | 01:03 | |
| jeblair | StevenK: so i think clarkb is suggesting that we should drive this on zanata.o.o itself via jeepyb rather than via zuul | 01:03 |
| clarkb | yup, similra to the examples above | 01:03 |
| *** aukhan has quit IRC | 01:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Fix bug with removing a failed job https://review.openstack.org/207675 | 01:04 |
| *** tiswanso has joined #openstack-infra | 01:04 | |
| clarkb | I am amazed that we were removing jobs from projects with this frequency | 01:04 |
| jeblair | clarkb, jhesketh: ^ reverified locally, and also rebased | 01:04 |
| jeblair | clarkb: the example i looked at was actually a job rename | 01:05 |
| StevenK | I don't think I want to shoehorn it into manage-projects, but I will refactor the code for the project-config into a jeepyb module/CLI | 01:05 |
| StevenK | jeblair, clarkb: ^ | 01:05 |
| StevenK | There's a missing branch in that sentence but oh well | 01:05 |
| clarkb | StevenK: it doesn't have to go in manage-projects, but it should go in jeepyb | 01:06 |
| *** tiswanso has quit IRC | 01:06 | |
| jeblair | StevenK: that sounds good | 01:06 |
| clarkb | (in fact I don't think it should go in manage-projects) | 01:06 |
| jeblair | yeah, it'll be nice to have some modularity in jeepyb; it's another project that hasn't really been 'designed' :) | 01:06 |
| openstackgerrit | Ian Cordasco proposed openstack/requirements: Synchronize bandit's requirements with global-requirements https://review.openstack.org/207678 | 01:06 |
| *** tiswanso has joined #openstack-infra | 01:06 | |
| StevenK | clarkb: I will not Abandon the project-config change at the moment, but I'll Workflow-1 so it doesn't get a +A by accident | 01:07 |
| *** bpokorny has quit IRC | 01:07 | |
| clarkb | StevenK: ok | 01:07 |
| jeblair | omg some of the change records that zuul queries from gerrit are HUGE | 01:07 |
| clarkb | jeblair: should I go ahead an dapprove the zuul change? | 01:07 |
| jeblair | clarkb: yep, thanks | 01:08 |
| *** yamamoto has quit IRC | 01:08 | |
| jeblair | why is nothing running? | 01:09 |
| clarkb | doesnt look like gearman submitted jobs | 01:11 |
| jeblair | clarkb: i think maybe we're waiting on mergers? | 01:11 |
| clarkb | ah | 01:11 |
| jeblair | oh | 01:12 |
| jeblair | it's the gear listening address thing again | 01:13 |
| jeblair | jhesketh: what's the current situation with that? | 01:13 |
| jeblair | i wrote https://review.openstack.org/204682 but it has not merged | 01:14 |
| jeblair | i guess i'll just manually apply that too and restart zuul | 01:14 |
| *** ivar-laz_ has joined #openstack-infra | 01:14 | |
| *** tqtran has quit IRC | 01:15 | |
| *** btully has quit IRC | 01:16 | |
| clarkb | I don't know what that situation is, will read 204682 to catch up | 01:16 |
| clarkb | oh it listens on 127.0.0.1 but mergers are off host | 01:17 |
| jeblair | things are running now | 01:17 |
| *** ivar-lazzaro has quit IRC | 01:17 | |
| *** rfolco has quit IRC | 01:18 | |
| jhesketh | jeblair: ah, you might still be hitting the host issue | 01:18 |
| jhesketh | jeblair: I had been patching zuul.conf to listen on 0.0.0.0 | 01:18 |
| jhesketh | until the real fix merges | 01:18 |
| jeblair | jhesketh: ah ok | 01:19 |
| *** dalgaaf has quit IRC | 01:19 | |
| *** bharath has joined #openstack-infra | 01:19 | |
| *** ivar-laz_ has quit IRC | 01:19 | |
| jhesketh | jeblair: if you remember the first instance of the problem after restoring we had a lot of NOT_REGISTERED errors | 01:19 |
| jeblair | hopefully clarkb will +3 204682 and then the next time we restart zuul it'll be running the two local changes it is now :) | 01:19 |
| clarkb | yup just did it | 01:19 |
| jhesketh | the fix for that was to boot a new nodepool node of that type as when it registers with jenkins that's when jenkins sends a CAN_DO | 01:20 |
| jeblair | jhesketh: we should be okay because the ramp-down while zuul was stuck allowed nodepool to have all node types ready | 01:20 |
| *** Sukhdev_ has joined #openstack-infra | 01:20 | |
| jhesketh | jeblair: right, but I think the registering matters because gearman gets restarted too | 01:21 |
| jhesketh | but so long as nodes are cycling through it'll be fine | 01:21 |
| jeblair | jhesketh: it does, but it waits for the function set to settle, so generally, if you start with a full complement of node types, it should be fine | 01:21 |
| *** ddieterly has joined #openstack-infra | 01:22 | |
| jeblair | (and workers are constantly trying to reconnect to gearman, so they do so immediately on zuul start) | 01:22 |
| jhesketh | jeblair: right, okay | 01:23 |
| jeblair | (though i do think we should add a dequeue command and then drop the registration check; it's sort of getting annoying, and we don't usually mess that up these days) | 01:23 |
| jeblair | (the dequeue command so that we can recover from when we do mess it up) | 01:23 |
| *** abregman|ZzZzZ has quit IRC | 01:23 | |
| jeblair | anyway, i think this should be good now, and i've got the re-enqueues chugging along; i'm going to get some food now | 01:24 |
| openstackgerrit | Merged openstack-infra/zuul: Specify gearman server listen address with separate option https://review.openstack.org/204682 | 01:26 |
| openstackgerrit | Merged openstack-infra/project-config: Do not run keystone-propose-config-updates https://review.openstack.org/206341 | 01:27 |
| *** shashank_hegde has quit IRC | 01:31 | |
| anteaya | pleia2: thank you | 01:32 |
| *** dimtruck is now known as zz_dimtruck | 01:32 | |
| anteaya | pleia2: I should have updated the ehterpad, I didn't do that, sorry | 01:32 |
| *** unicell has quit IRC | 01:34 | |
| pleia2 | anteaya: we got there in the end | 01:34 |
| openstackgerrit | Merged openstack-infra/project-config: Move hook-{name}-rtfd to proposal node https://review.openstack.org/206618 | 01:34 |
| openstackgerrit | Merged openstack-infra/project-config: Create a repository for stackforge/xstatic-mdi https://review.openstack.org/206754 | 01:35 |
| openstackgerrit | Merged openstack-infra/zuul: Set build descriptions less often https://review.openstack.org/195785 | 01:35 |
| anteaya | pleia2: we did, thanks | 01:36 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: jenkins: implement copy-puppet-logs publisher https://review.openstack.org/206693 | 01:37 |
| EmilienM | ianw, clark, crinkle: done ^ | 01:37 |
| *** rick__ has joined #openstack-infra | 01:38 | |
| rick__ | anteaya: Hi Anteaya, Are you here? | 01:38 |
| anteaya | hi rick__ | 01:39 |
| anteaya | your system is disabled because it doesn't work | 01:39 |
| anteaya | so first thing, get it to work | 01:39 |
| anteaya | rick__: http://lists.openstack.org/pipermail/third-party-announce/2015-July/000257.html | 01:40 |
| *** signed8bit has joined #openstack-infra | 01:40 | |
| anteaya | rick__: then after it is working come back here and tell us which of the three gerrit accounts you want associated with the account: http://lists.openstack.org/pipermail/third-party-announce/2015-July/000258.html | 01:40 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: jenkins: implement copy-puppet-logs publisher https://review.openstack.org/206693 | 01:40 |
| anteaya | rick__: there were three, that makes us unhappy | 01:41 |
| anteaya | rick__: and please don't make any more gerrit accounts that share an email address, each gerrit account should have a unique email address for each account | 01:41 |
| *** pahuang has quit IRC | 01:41 | |
| *** camunoz has quit IRC | 01:41 | |
| *** ianw has quit IRC | 01:43 | |
| *** ddieterl_ has joined #openstack-infra | 01:43 | |
| EmilienM | anteaya: you still here? :) | 01:44 |
| anteaya | EmilienM: I am | 01:44 |
| *** ddieterly has quit IRC | 01:44 | |
| *** ianw has joined #openstack-infra | 01:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Create a repository for stackforge/xstatic-roboto-fontface https://review.openstack.org/206767 | 01:45 |
| openstackgerrit | Merged openstack-infra/project-config: Write post job logs to apache and swift https://review.openstack.org/207195 | 01:45 |
| openstackgerrit | Merged openstack-infra/project-config: Add repo for apps-catalog-ui https://review.openstack.org/207253 | 01:45 |
| *** pahuang has joined #openstack-infra | 01:45 | |
| *** camunoz has joined #openstack-infra | 01:45 | |
| anteaya | EmilienM: I am in Brisbane | 01:47 |
| *** zz_dimtruck is now known as dimtruck | 01:48 | |
| EmilienM | anteaya: oh dear ! | 01:48 |
| EmilienM | anteaya: it's very far | 01:48 |
| anteaya | took a while to get here, yeah | 01:49 |
| anteaya | :) | 01:49 |
| anteaya | winter in Brisbane and summer at my house is about the same weather | 01:49 |
| EmilienM | lol | 01:49 |
| jhesketh | anteaya: quite similar for me... one of my brisbane coworkers is wearing a jacket and scarf! | 01:51 |
| *** yamamoto has joined #openstack-infra | 01:51 | |
| rick__ | anteaya: I don't know why has three account. Sorry. | 01:52 |
| anteaya | jhesketh: ha ha ha | 01:52 |
| *** esmute_ has joined #openstack-infra | 01:52 | |
| anteaya | rick__: okay how did you create the account? perhaps there is a step in the account creation that has a problem | 01:53 |
| *** PinkFreud has quit IRC | 01:53 | |
| anteaya | sometimes that happens | 01:53 |
| anteaya | if three accounts with the same email was not intentional | 01:53 |
| anteaya | jhesketh: I love seeing the fur trimmed down jackets | 01:53 |
| anteaya | I hold myself back from pointing and laughing | 01:54 |
| EmilienM | w00t, zuul down to 256 jobs, awesome | 01:54 |
| *** bhunter71 has quit IRC | 01:55 | |
| rick__ | anteaya: In a previous, I follow third-part ci document to send email to request account. And I had send email to request change account name. | 01:55 |
| rick__ | anteaya: But I only use an account in my zuul configuration. | 01:57 |
| *** skylerberg has quit IRC | 01:57 | |
| anteaya | rick__: okay, let's get your jobs working first | 01:59 |
| anteaya | rick__: then we can get thingee to approve | 01:59 |
| anteaya | rick__: then we can figure out which gerrit account id to re-enable | 01:59 |
| anteaya | rick__: sound okay? | 01:59 |
| *** otter768 has joined #openstack-infra | 02:00 | |
| anteaya | rick__: do you understand that your account was disabled because your system continued to post failures for everything | 02:00 |
| rick__ | anteaya: OK, Thanks. Yes, I understand it. | 02:01 |
| *** bhunter71 has joined #openstack-infra | 02:02 | |
| thingee | rick__: what was the reason this time? | 02:02 |
| *** Sukhdev_ has quit IRC | 02:02 | |
| *** esmute_ has quit IRC | 02:03 | |
| *** bharath has quit IRC | 02:03 | |
| *** gildub_ has quit IRC | 02:04 | |
| *** gildub has quit IRC | 02:04 | |
| *** otter768 has quit IRC | 02:05 | |
| *** bharath has joined #openstack-infra | 02:05 | |
| rick__ | anteaya: I don't know which changed to make my ci machine get error "2015-07-29 10:04:46.128013 ImportError: cannot import name deploy" in the log keystone.log of the apache.log. | 02:06 |
| anteaya | thingee: this is http://lists.openstack.org/pipermail/third-party-announce/2015-July/000257.html | 02:06 |
| rick__ | anteaya: I need uninstall "python-pastedeploy" for each ci verifying. | 02:07 |
| *** salv-orlando has joined #openstack-infra | 02:08 | |
| *** esmute_ has joined #openstack-infra | 02:09 | |
| *** bapalm has joined #openstack-infra | 02:09 | |
| *** bharath has quit IRC | 02:10 | |
| rick__ | anteaya: Please re-enable the only one gerrit id "prophetstor-ci" for us, Thanks. | 02:10 |
| *** tjones has joined #openstack-infra | 02:11 | |
| anteaya | rick__: well first we need to hear that thingee is ready for it to be re-enabled | 02:11 |
| anteaya | so can you run some jobs and show thingee the logs of some successful runs? | 02:11 |
| rick__ | anteaya: OK, Thanks. | 02:12 |
| anteaya | thank you | 02:12 |
| *** rick__ has quit IRC | 02:12 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 02:12 | |
| thingee | whoa he just walked on out of here. | 02:12 |
| anteaya | thingee: it makes me sad when they do that | 02:13 |
| anteaya | hang around folks | 02:14 |
| anteaya | someone might offer some help | 02:14 |
| anteaya | thingee: tell us when you want them re-enabled | 02:14 |
| anteaya | we will wait to hear from you | 02:15 |
| *** markvoelker has joined #openstack-infra | 02:15 | |
| *** salv-orlando has quit IRC | 02:15 | |
| *** markvoelker has quit IRC | 02:15 | |
| openstackgerrit | Kun Huang proposed openstack/os-testr: update requirements https://review.openstack.org/207691 | 02:16 |
| *** rick__ has joined #openstack-infra | 02:17 | |
| *** banix has joined #openstack-infra | 02:17 | |
| *** bapalm has quit IRC | 02:19 | |
| rick__ | anteaya: http://download.prophetstor.com/prophetstor_ci/203895/3/check/prophetstor-dsvm-tempest-cinder-ci/5018/console.html, this is my testing now. | 02:19 |
| *** markvoelker has joined #openstack-infra | 02:19 | |
| anteaya | rick__: great, let's see a few more | 02:20 |
| rick__ | thingee: http://download.prophetstor.com/prophetstor_ci/203895/3/check/prophetstor-dsvm-tempest-cinder-ci/5018/console.html, this is my openstack ci testing report now. | 02:20 |
| *** larainema has quit IRC | 02:21 | |
| *** markvoelker_ has joined #openstack-infra | 02:21 | |
| *** esmute_ has quit IRC | 02:22 | |
| *** otter768 has joined #openstack-infra | 02:22 | |
| rick__ | anteaya: I has some problem, sometime I don't know which changed to make failure but our cinder volume driver did not change any code. This problem sometime need more time to fix. | 02:22 |
| *** markvoelker has quit IRC | 02:23 | |
| anteaya | rick__: ah okay, yes that is why we disabled the account, you need to show us the problem is fixed before you are re-enabled | 02:24 |
| anteaya | rick__: so take the time you need to fix the problem | 02:24 |
| rick__ | anteaya: And these failure did not relate on the cinder project. It usually make the devstack unable to work. | 02:24 |
| *** fedexo has joined #openstack-infra | 02:24 | |
| anteaya | right | 02:24 |
| anteaya | so your system has a problem | 02:24 |
| anteaya | that needs to be fixed | 02:24 |
| *** mriedem has quit IRC | 02:25 | |
| *** skylerberg has joined #openstack-infra | 02:25 | |
| *** esmute_ has joined #openstack-infra | 02:26 | |
| *** esmute_ has quit IRC | 02:26 | |
| rick__ | anteaya: Like this, the system is fine some days. I don't know which changed to make us need to uninstall the "python-pastedeploy" to solve this problem. | 02:26 |
| *** larainema has joined #openstack-infra | 02:27 | |
| *** gildub has joined #openstack-infra | 02:28 | |
| *** openstackgerrit has quit IRC | 02:31 | |
| rick__ | anteaya: So, Now, I need wait thingee to check the attached report and re-enable my gerrit account, right. | 02:31 |
| *** shashank_hegde has joined #openstack-infra | 02:31 | |
| *** openstackgerrit has joined #openstack-infra | 02:31 | |
| *** bhuvan has joined #openstack-infra | 02:34 | |
| *** plol has quit IRC | 02:34 | |
| *** plol has joined #openstack-infra | 02:35 | |
| *** otter768 has quit IRC | 02:35 | |
| *** otter768 has joined #openstack-infra | 02:37 | |
| *** achanda has joined #openstack-infra | 02:42 | |
| *** esmute_ has joined #openstack-infra | 02:43 | |
| *** btully has joined #openstack-infra | 02:44 | |
| *** rick__ has quit IRC | 02:44 | |
| *** aukhan has joined #openstack-infra | 02:45 | |
| *** nelsnelson has quit IRC | 02:47 | |
| *** esmute_ has quit IRC | 02:47 | |
| *** dzamboni has quit IRC | 02:47 | |
| *** bhuvan has quit IRC | 02:47 | |
| *** btully has quit IRC | 02:48 | |
| *** shashank_hegde has quit IRC | 02:49 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: jenkins: implement copy-puppet-logs publisher https://review.openstack.org/206693 | 02:51 |
| *** ddieterl_ has quit IRC | 02:52 | |
| *** puranamr has joined #openstack-infra | 02:52 | |
| *** puranamr has quit IRC | 02:56 | |
| *** jasondotstar has quit IRC | 02:58 | |
| *** tjones has quit IRC | 03:00 | |
| *** bharath has joined #openstack-infra | 03:07 | |
| *** dimtruck is now known as zz_dimtruck | 03:08 | |
| *** bharath has quit IRC | 03:11 | |
| *** signed8bit has quit IRC | 03:12 | |
| *** shashank_hegde has joined #openstack-infra | 03:13 | |
| *** bhuvan has joined #openstack-infra | 03:17 | |
| *** glauco_ has joined #openstack-infra | 03:20 | |
| *** rick__ has joined #openstack-infra | 03:21 | |
| *** bhuvan has quit IRC | 03:22 | |
| *** markvoelker_ has quit IRC | 03:23 | |
| *** glauco_ has quit IRC | 03:24 | |
| *** shashank_hegde has quit IRC | 03:25 | |
| *** markvoelker_ has joined #openstack-infra | 03:26 | |
| *** ashleighfarnham has joined #openstack-infra | 03:27 | |
| *** PinkFreud has joined #openstack-infra | 03:28 | |
| *** achanda has quit IRC | 03:28 | |
| *** tiswanso has quit IRC | 03:31 | |
| *** dims_ has quit IRC | 03:33 | |
| *** AzherKhan has joined #openstack-infra | 03:33 | |
| *** esmute_ has joined #openstack-infra | 03:34 | |
| *** rick__ has quit IRC | 03:36 | |
| *** sdake_ has joined #openstack-infra | 03:42 | |
| *** bhunter71 has quit IRC | 03:43 | |
| *** sdake has quit IRC | 03:43 | |
| *** sdake_ has quit IRC | 03:45 | |
| StevenK | Heh | 03:45 |
| *** sdake has joined #openstack-infra | 03:45 | |
| StevenK | Turns out jeepyb is on pypi, using a short git sha as the version, from 2013 | 03:45 |
| *** coolsvap|away is now known as coolsvap | 03:49 | |
| *** esmute_ has quit IRC | 03:50 | |
| *** sdake_ has joined #openstack-infra | 03:51 | |
| *** sdake has quit IRC | 03:52 | |
| *** ddieterly has joined #openstack-infra | 03:53 | |
| *** pal has joined #openstack-infra | 03:54 | |
| *** shashank_hegde has joined #openstack-infra | 03:54 | |
| *** ashleighfarnham has quit IRC | 03:55 | |
| *** fabo has quit IRC | 03:56 | |
| anteaya | rick__: well part of the problem is that your system started failing and you were not aware of it | 03:56 |
| *** ddieterly has quit IRC | 03:57 | |
| anteaya | rick__: I highly recommend you find a way to monitor your system so that if this happens again and your jobs start failing you notice and disable it yourself | 03:57 |
| *** Sukhdev_ has joined #openstack-infra | 03:59 | |
| *** salv-orlando has joined #openstack-infra | 04:00 | |
| *** esmute_ has joined #openstack-infra | 04:00 | |
| *** esmute_ has quit IRC | 04:03 | |
| *** bhuvan has joined #openstack-infra | 04:05 | |
| *** salv-orlando has quit IRC | 04:07 | |
| anteaya | rick__: actually you need to figure out what changed that created the problem, find a way to monitor your system, convey you have completed these steps to thingee so that he can see it for himself and then he will communicate with infra | 04:10 |
| *** davideagnello has quit IRC | 04:16 | |
| *** WyAt has joined #openstack-infra | 04:17 | |
| *** davideagnello has joined #openstack-infra | 04:17 | |
| *** ashleighfarnham has joined #openstack-infra | 04:19 | |
| *** sdake_ has quit IRC | 04:21 | |
| *** sdake has joined #openstack-infra | 04:21 | |
| *** BrandonL08 has joined #openstack-infra | 04:22 | |
| *** Somay has joined #openstack-infra | 04:23 | |
| *** Somay has left #openstack-infra | 04:23 | |
| *** ajmiller has quit IRC | 04:25 | |
| *** harlowja_at_home has joined #openstack-infra | 04:27 | |
| *** sdake has quit IRC | 04:28 | |
| *** sdake_ has joined #openstack-infra | 04:28 | |
| *** BharatK has joined #openstack-infra | 04:28 | |
| *** sdake has joined #openstack-infra | 04:31 | |
| *** sdake_ has quit IRC | 04:31 | |
| *** skylerberg has quit IRC | 04:33 | |
| *** bharath has joined #openstack-infra | 04:33 | |
| *** sdake_ has joined #openstack-infra | 04:35 | |
| *** sdake has quit IRC | 04:36 | |
| *** bharath has quit IRC | 04:38 | |
| *** sdake_ has quit IRC | 04:41 | |
| *** sdake has joined #openstack-infra | 04:42 | |
| *** otter768 has quit IRC | 04:43 | |
| *** sdake has quit IRC | 04:44 | |
| *** fedexo has quit IRC | 04:44 | |
| *** sdake has joined #openstack-infra | 04:45 | |
| *** BrandonL08 has quit IRC | 04:45 | |
| *** shashank_hegde has quit IRC | 04:46 | |
| *** sdake has quit IRC | 04:47 | |
| *** sdake has joined #openstack-infra | 04:47 | |
| Nakato | lifeless: Does the requirements management spec have a blueprint, does it need one? | 04:51 |
| *** pal has quit IRC | 04:51 | |
| *** shashank_hegde has joined #openstack-infra | 04:53 | |
| *** bharath_ has joined #openstack-infra | 04:54 | |
| *** ddieterly has joined #openstack-infra | 04:54 | |
| *** banix has quit IRC | 04:55 | |
| *** fawadkhaliq has joined #openstack-infra | 04:55 | |
| *** bharath_ has quit IRC | 04:56 | |
| *** deepakcs has joined #openstack-infra | 04:58 | |
| *** ddieterly has quit IRC | 04:58 | |
| openstackgerrit | Andrei V. Ostapenko proposed openstack-infra/project-config: Move from stackforge/magnetodb to openstack/magnetodb https://review.openstack.org/203524 | 05:00 |
| openstackgerrit | Andrei V. Ostapenko proposed openstack-infra/project-config: Changes jobs description for MagnetoDB https://review.openstack.org/204478 | 05:01 |
| *** pal has joined #openstack-infra | 05:05 | |
| *** sergmelikyan has joined #openstack-infra | 05:06 | |
| *** coolsvap is now known as coolsvap|away | 05:08 | |
| *** fedexo has joined #openstack-infra | 05:08 | |
| *** rick__ has joined #openstack-infra | 05:11 | |
| *** sergmelikyan has quit IRC | 05:11 | |
| rick__ | thingee: HI Thingee. | 05:11 |
| lifeless | Nakato: enfi | 05:12 |
| *** BharatK has quit IRC | 05:13 | |
| rick__ | thingee: I ready fixed my CI machine, can you re-enable my gerrit account "prophetstor-ci". http://download.prophetstor.com/prophetstor_ci/203895/3/check/prophetstor-dsvm-tempest-cinder-ci/5018/console.html | 05:13 |
| *** salv-orlando has joined #openstack-infra | 05:13 | |
| *** rm_work is now known as rm_work|away | 05:16 | |
| *** coolsvap|away is now known as coolsvap | 05:17 | |
| *** pal has quit IRC | 05:18 | |
| *** esmute_ has joined #openstack-infra | 05:19 | |
| *** salv-orlando has quit IRC | 05:20 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul: Retry change query if change not found https://review.openstack.org/206043 | 05:20 |
| *** fedexo has quit IRC | 05:20 | |
| *** glauco has joined #openstack-infra | 05:21 | |
| *** salv-orlando has joined #openstack-infra | 05:21 | |
| *** amickus has joined #openstack-infra | 05:22 | |
| *** pal has joined #openstack-infra | 05:23 | |
| *** jaosorior has joined #openstack-infra | 05:23 | |
| *** ig0r_ has joined #openstack-infra | 05:24 | |
| *** glauco has quit IRC | 05:25 | |
| *** alexsyip has joined #openstack-infra | 05:26 | |
| *** salv-orlando has quit IRC | 05:27 | |
| *** fedexo has joined #openstack-infra | 05:29 | |
| *** alexsyip has quit IRC | 05:29 | |
| *** esmute_ has quit IRC | 05:29 | |
| *** achanda has joined #openstack-infra | 05:30 | |
| *** mrmartin has joined #openstack-infra | 05:32 | |
| *** esmute_ has joined #openstack-infra | 05:34 | |
| *** mrmartin has quit IRC | 05:34 | |
| *** armax has quit IRC | 05:35 | |
| *** sdake has quit IRC | 05:35 | |
| *** mahito has joined #openstack-infra | 05:36 | |
| *** mrmartin has joined #openstack-infra | 05:36 | |
| *** sergmelikyan has joined #openstack-infra | 05:37 | |
| *** esmute_ has quit IRC | 05:38 | |
| *** sdake has joined #openstack-infra | 05:41 | |
| anteaya | rick__: as I said both in logs and in my rationale for refusing your post to third-party-announce, you have some work to do | 05:42 |
| anteaya | rick__: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-07-31.log.html#t2015-07-31T03:56:48 | 05:42 |
| anteaya | rick__: if you can't stay connected to the channel, please read the channel logs | 05:42 |
| rick__ | anteaya: OK, I got. I already set e-mail alert in my jenkins server. | 05:43 |
| rick__ | anteaya: And I already fixed the devstack unable to start issue.Verify it again based on my ci machine last failure project. | 05:45 |
| *** skylerberg has joined #openstack-infra | 05:47 | |
| *** markvoelker_ has quit IRC | 05:48 | |
| anteaya | rick__: great, thingee is probably unavailable right now, when the sun is in the sky over north america next try to find him in the -cinder channel | 05:49 |
| rick__ | anteaya: Thanks | 05:53 |
| rick__ | anteaya: Can you pass my message to him by email? | 05:53 |
| *** ddieterly has joined #openstack-infra | 05:55 | |
| *** btully has joined #openstack-infra | 05:56 | |
| *** vlaza has joined #openstack-infra | 05:58 | |
| *** camunoz has quit IRC | 05:58 | |
| *** Somay has joined #openstack-infra | 05:58 | |
| *** Sukhdev_ has quit IRC | 05:59 | |
| ianw | EmilienM: thanks, that's on the right path, but big bits of that should be cut and bits don't work. i've got some ideas, i won't get to it today but i might make some edits and send it back to you | 05:59 |
| openstackgerrit | Steve Kowalik proposed openstack-infra/jeepyb: Add a register-zanata-projects script https://review.openstack.org/207729 | 05:59 |
| *** ddieterly has quit IRC | 05:59 | |
| *** fedexo has quit IRC | 06:02 | |
| openstackgerrit | Ryu Ishimoto proposed openstack-infra/project-config: Set DEVSTACK_GATE_SETTINGS for midonet gate job https://review.openstack.org/207731 | 06:03 |
| *** harlowja_at_home has quit IRC | 06:04 | |
| *** coolsvap is now known as coolsvap|away | 06:05 | |
| *** BharatK has joined #openstack-infra | 06:05 | |
| *** BharatK has quit IRC | 06:07 | |
| *** evrardjp_ is now known as evrardjp | 06:07 | |
| *** BharatK has joined #openstack-infra | 06:07 | |
| anteaya | rick__: why don't you join #openstack-cinder and leave the message yourself | 06:08 |
| *** ParsectiX has joined #openstack-infra | 06:08 | |
| anteaya | rick__: and if you can't stay connected to irc, read the cinder channel logs: http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/ | 06:08 |
| anteaya | rick__: it would be good if you learned how to communication effectively using irc | 06:09 |
| anteaya | s/communication/communicate | 06:09 |
| rick__ | anteaya: OK, thanks. | 06:10 |
| anteaya | welcome | 06:10 |
| *** jyuso1 has joined #openstack-infra | 06:11 | |
| *** ig0r__ has joined #openstack-infra | 06:11 | |
| *** ig0r_ has quit IRC | 06:12 | |
| *** coolsvap|away is now known as coolsvap | 06:13 | |
| *** unicell has joined #openstack-infra | 06:13 | |
| *** pal has quit IRC | 06:17 | |
| *** scheuran has joined #openstack-infra | 06:19 | |
| *** pal has joined #openstack-infra | 06:20 | |
| *** BharatK has quit IRC | 06:21 | |
| *** erikwilson_ has quit IRC | 06:22 | |
| *** k4n0 has joined #openstack-infra | 06:23 | |
| *** erikwilson_ has joined #openstack-infra | 06:23 | |
| *** fesp has joined #openstack-infra | 06:23 | |
| *** pal_ has joined #openstack-infra | 06:23 | |
| *** unicell1 has joined #openstack-infra | 06:24 | |
| *** pal has quit IRC | 06:24 | |
| *** unicell has quit IRC | 06:25 | |
| *** salv-orlando has joined #openstack-infra | 06:26 | |
| *** erikwilson_ has quit IRC | 06:27 | |
| *** pal_ has quit IRC | 06:28 | |
| *** armax has joined #openstack-infra | 06:28 | |
| *** BharatK has joined #openstack-infra | 06:34 | |
| *** armax has quit IRC | 06:35 | |
| *** salv-orlando has quit IRC | 06:37 | |
| *** pal has joined #openstack-infra | 06:39 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/207739 | 06:39 |
| *** sergmelikyan has quit IRC | 06:43 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/207741 | 06:44 |
| *** otter768 has joined #openstack-infra | 06:44 | |
| *** salv-orlando has joined #openstack-infra | 06:46 | |
| *** skylerberg has quit IRC | 06:46 | |
| *** matbu has joined #openstack-infra | 06:48 | |
| *** markvoelker has joined #openstack-infra | 06:48 | |
| *** otter768 has quit IRC | 06:49 | |
| *** markvoelker has quit IRC | 06:53 | |
| *** ddieterly has joined #openstack-infra | 06:56 | |
| *** fesp has quit IRC | 06:59 | |
| *** ddieterly has quit IRC | 07:00 | |
| *** coolsvap is now known as coolsvap|away | 07:02 | |
| *** nikeshm has joined #openstack-infra | 07:04 | |
| *** nikeshm has quit IRC | 07:05 | |
| *** nim has joined #openstack-infra | 07:05 | |
| nim | hi | 07:05 |
| *** Ala has joined #openstack-infra | 07:08 | |
| *** boris-42 has quit IRC | 07:10 | |
| *** pal has quit IRC | 07:10 | |
| *** coolsvap|away is now known as coolsvap | 07:10 | |
| *** matbu has left #openstack-infra | 07:10 | |
| openstackgerrit | Tim Rupp proposed openstack-infra/jenkins-job-builder: Adds onlyIfSuccessful to ArtifactArchiver publisher https://review.openstack.org/203730 | 07:11 |
| *** markus_z has joined #openstack-infra | 07:11 | |
| nim | hi is any one facing below issue in FC passthrough for rasselin CI | 07:11 |
| nim | ? | 07:11 |
| *** jlanoux has joined #openstack-infra | 07:11 | |
| nim | http://paste.openstack.org/show/406551/ | 07:12 |
| nim | we are using ubuntu 14.04 | 07:12 |
| nim | server | 07:12 |
| nim | node | 07:12 |
| *** rick__ has quit IRC | 07:13 | |
| *** yamamoto has quit IRC | 07:15 | |
| *** rossella_s has quit IRC | 07:16 | |
| *** achanda has quit IRC | 07:16 | |
| *** rossella_s has joined #openstack-infra | 07:16 | |
| *** arxcruz has joined #openstack-infra | 07:17 | |
| *** dalgaaf has joined #openstack-infra | 07:19 | |
| *** Slashme has left #openstack-infra | 07:21 | |
| *** ifarkas has joined #openstack-infra | 07:26 | |
| *** mpavone has joined #openstack-infra | 07:26 | |
| *** melwitt has quit IRC | 07:27 | |
| openstackgerrit | Merged openstack-infra/shade: Pass socket timeout to swiftclient https://review.openstack.org/168672 | 07:27 |
| openstackgerrit | Merged openstack-infra/shade: Process config options via os-client-config https://review.openstack.org/206130 | 07:28 |
| *** yamamoto has joined #openstack-infra | 07:30 | |
| openstackgerrit | Merged openstack-infra/shade: Remove unused server functions https://review.openstack.org/206730 | 07:30 |
| *** salv-orlando has quit IRC | 07:33 | |
| *** melwitt has joined #openstack-infra | 07:33 | |
| *** mpaolino has joined #openstack-infra | 07:36 | |
| *** devvesa has joined #openstack-infra | 07:38 | |
| *** ParsectiX has quit IRC | 07:42 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-os_client_config: Manage OSCC config file with Puppet https://review.openstack.org/199674 | 07:43 |
| *** ParsectiX has joined #openstack-infra | 07:43 | |
| *** jaosorior has quit IRC | 07:44 | |
| *** jaosorior has joined #openstack-infra | 07:47 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Add openstack-infra/puppet-infra-cookiecutter project https://review.openstack.org/207109 | 07:48 |
| *** salv-orlando has joined #openstack-infra | 07:49 | |
| *** gildub has quit IRC | 07:51 | |
| *** dtantsur|afk is now known as dtantsur | 07:53 | |
| *** bharath has joined #openstack-infra | 07:57 | |
| *** marzif_ has joined #openstack-infra | 07:57 | |
| *** kushal has joined #openstack-infra | 08:00 | |
| *** _nadya_ has joined #openstack-infra | 08:01 | |
| *** abregman has joined #openstack-infra | 08:01 | |
| *** bharath has quit IRC | 08:01 | |
| *** oomichi has quit IRC | 08:02 | |
| *** mpavone has quit IRC | 08:03 | |
| *** shashank_hegde has quit IRC | 08:08 | |
| *** marzif_ has quit IRC | 08:09 | |
| *** afazekas has joined #openstack-infra | 08:10 | |
| *** juggler_ is now known as juggler | 08:10 | |
| openstackgerrit | Doron Chen proposed openstack-infra/project-config: adding a new openstack/storlets project https://review.openstack.org/203418 | 08:12 |
| *** witek has quit IRC | 08:14 | |
| *** jcoufal has joined #openstack-infra | 08:17 | |
| *** ihrachyshka has joined #openstack-infra | 08:18 | |
| *** jasondotstar has joined #openstack-infra | 08:19 | |
| *** yamahata has joined #openstack-infra | 08:23 | |
| *** erikwilson_ has joined #openstack-infra | 08:23 | |
| *** btully has quit IRC | 08:25 | |
| *** mrmartin has quit IRC | 08:26 | |
| *** mrmartin has joined #openstack-infra | 08:26 | |
| *** bhuvan has quit IRC | 08:27 | |
| *** salv-orlando has quit IRC | 08:27 | |
| *** erikwilson_ has quit IRC | 08:28 | |
| *** jistr has joined #openstack-infra | 08:32 | |
| *** yamahata has quit IRC | 08:33 | |
| *** mfmcdonagh has quit IRC | 08:34 | |
| *** mcdonagm has quit IRC | 08:34 | |
| *** mahito has quit IRC | 08:35 | |
| *** e0ne has joined #openstack-infra | 08:35 | |
| *** ZZelle has quit IRC | 08:38 | |
| *** ZZelle has joined #openstack-infra | 08:38 | |
| *** liam__ has joined #openstack-infra | 08:39 | |
| *** e0ne has quit IRC | 08:40 | |
| *** liam__ has quit IRC | 08:40 | |
| *** marzif_ has joined #openstack-infra | 08:40 | |
| *** marzif_ has quit IRC | 08:41 | |
| *** marzif_ has joined #openstack-infra | 08:42 | |
| *** otter768 has joined #openstack-infra | 08:45 | |
| *** ihrachyshka has quit IRC | 08:48 | |
| *** markvoelker has joined #openstack-infra | 08:49 | |
| SergK | hi, My patch (create project on stackforge) was successfully merged https://review.openstack.org/#/c/205963, so, who can help me to become a member of fuel-mirror-core group https://review.openstack.org/#/admin/groups/1016,members | 08:50 |
| *** otter768 has quit IRC | 08:50 | |
| *** abregman is now known as abregman|brb | 08:51 | |
| *** luqas__ has quit IRC | 08:52 | |
| *** mpaolino has quit IRC | 08:54 | |
| *** markvoelker has quit IRC | 08:54 | |
| *** ddieterly has joined #openstack-infra | 08:57 | |
| *** nijaba has quit IRC | 09:01 | |
| *** ddieterly has quit IRC | 09:02 | |
| *** luqas has joined #openstack-infra | 09:06 | |
| *** e0ne has joined #openstack-infra | 09:13 | |
| *** jcoufal has quit IRC | 09:13 | |
| *** amickus has quit IRC | 09:22 | |
| *** glauco has joined #openstack-infra | 09:22 | |
| *** glauco has quit IRC | 09:27 | |
| openstackgerrit | Greg Chapple proposed openstack-infra/git-review: Allow reviewers to be defined in .gitreview file https://review.openstack.org/207795 | 09:29 |
| *** jcoufal has joined #openstack-infra | 09:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove non-voting job from networking-odl https://review.openstack.org/199868 | 09:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: jenkins: implement copy-puppet-logs publisher https://review.openstack.org/206693 | 09:35 |
| *** zeih has joined #openstack-infra | 09:36 | |
| *** BharatK_ has joined #openstack-infra | 09:37 | |
| *** BharatK has quit IRC | 09:39 | |
| *** mase_home has quit IRC | 09:41 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: jenkins: implement copy-puppet-logs publisher https://review.openstack.org/206693 | 09:45 |
| *** salv-orlando has joined #openstack-infra | 09:48 | |
| openstackgerrit | Michael Prokop proposed openstack-infra/jenkins-job-builder: Add ClamAV Plugin support https://review.openstack.org/207239 | 09:55 |
| *** nagyz has joined #openstack-infra | 09:58 | |
| nagyz | I've pushed in a change for devstack, however i noticed jenkins didn't run gate-tempest-dvsm-full-ceph on it. is there a way for me to ask jenkins nicely to please run it on my change as well? | 09:58 |
| *** _nadya_ has quit IRC | 09:59 | |
| *** _nadya_ has joined #openstack-infra | 09:59 | |
| *** yamamoto has quit IRC | 10:02 | |
| *** _nadya_ has quit IRC | 10:04 | |
| *** zeih has quit IRC | 10:05 | |
| *** e0ne has quit IRC | 10:07 | |
| *** pc_m has joined #openstack-infra | 10:11 | |
| *** pc_m_ has joined #openstack-infra | 10:12 | |
| *** pcm_ has joined #openstack-infra | 10:13 | |
| *** scheuran has quit IRC | 10:13 | |
| *** pc_m has quit IRC | 10:15 | |
| *** pc_m_ has quit IRC | 10:16 | |
| *** pcm_ has quit IRC | 10:16 | |
| *** jasondotstar has quit IRC | 10:22 | |
| *** erikwilson_ has joined #openstack-infra | 10:24 | |
| *** erikwilson_ has quit IRC | 10:29 | |
| *** BharatK_ has quit IRC | 10:31 | |
| *** e0ne has joined #openstack-infra | 10:31 | |
| openstackgerrit | Merged openstack-infra/zuul: Fix bug with removing a failed job https://review.openstack.org/207675 | 10:40 |
| *** jasondotstar has joined #openstack-infra | 10:40 | |
| *** pc_m has joined #openstack-infra | 10:43 | |
| *** yamamoto has joined #openstack-infra | 10:43 | |
| *** yamamoto_ has joined #openstack-infra | 10:44 | |
| *** BharatK_ has joined #openstack-infra | 10:45 | |
| *** otter768 has joined #openstack-infra | 10:46 | |
| *** yamamoto has quit IRC | 10:48 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Per IRC meeting owner, this is at 2000 https://review.openstack.org/207614 | 10:49 |
| *** otter768 has quit IRC | 10:51 | |
| *** cdent has joined #openstack-infra | 10:51 | |
| *** neiljerr` has quit IRC | 10:52 | |
| *** BobBall_AWOL is now known as BobBall | 10:53 | |
| *** mpaolino has joined #openstack-infra | 10:53 | |
| *** mpaolino has quit IRC | 10:53 | |
| *** ddieterly has joined #openstack-infra | 10:59 | |
| *** jyuso1 has quit IRC | 11:00 | |
| *** mpaolino has joined #openstack-infra | 11:00 | |
| *** jasondotstar has quit IRC | 11:01 | |
| *** _nadya_ has joined #openstack-infra | 11:02 | |
| *** jasondotstar has joined #openstack-infra | 11:03 | |
| *** ddieterly has quit IRC | 11:03 | |
| *** dims_ has joined #openstack-infra | 11:04 | |
| kashyap | mordred: lol! -- "The Real Slim Shade" | 11:08 |
| kashyap | mordred: I was quietly working in a coffee shop and stumbled on this. While I began to read the first line of the abstract, my brain was humming the tune and I was laughing out loud, to the amusement of people around. | 11:09 |
| *** scheuran has joined #openstack-infra | 11:10 | |
| *** sobersabre has joined #openstack-infra | 11:11 | |
| *** AzherKhan has quit IRC | 11:11 | |
| *** aukhan has quit IRC | 11:11 | |
| kashyap | Crazy. (But, thanks for the laughs!) | 11:12 |
| *** ig0r__ has quit IRC | 11:15 | |
| *** jasondotstar has quit IRC | 11:17 | |
| *** signed8bit has joined #openstack-infra | 11:17 | |
| *** sobersabre has quit IRC | 11:17 | |
| *** ig0r_ has joined #openstack-infra | 11:19 | |
| *** yamamoto_ has quit IRC | 11:20 | |
| *** signed8b_ has joined #openstack-infra | 11:20 | |
| *** signed8bit has quit IRC | 11:23 | |
| *** afazekas has quit IRC | 11:31 | |
| *** yamamoto has joined #openstack-infra | 11:31 | |
| *** mrmartin has quit IRC | 11:34 | |
| *** ldnunes has joined #openstack-infra | 11:35 | |
| *** dtantsur is now known as dtantsur|lunch | 11:36 | |
| dims_ | hmm, "test nodes" graph shows just blue ("in use"), nothing in building/available/deleting for the past 2 hours at least - http://graphite.openstack.org/render/?from=-6hours&height=400&until=now&width=600&bgcolor=ffffff&fgcolor=000000&areaMode=stacked&target=color(alias(sumSeries(stats.gauges.nodepool.target.building),%20%27Building%27),%20%27ffbf52%27)&target=color(alias(sumSeries(stats.gauges.nodepool.target.ready),% | 11:38 |
| dims_ | 20%27Available%27),%20%2700c868%27)&target=color(alias(sumSeries(stats.gauges.nodepool.target.used),%20%27In%20Use%27),%20%276464ff%27)&target=color(alias(sumSeries(stats.gauges.nodepool.target.delete),%20%27Deleting%27),%20%27c864ff%27)&title=Test%20Nodes&_t=0.4210144605021924#1438342568693 | 11:38 |
| dims_ | oops - http://bit.ly/1Iu21qX | 11:39 |
| *** yamamoto has quit IRC | 11:39 | |
| *** coolsvap is now known as coolsvap|away | 11:39 | |
| *** hdd has joined #openstack-infra | 11:46 | |
| *** e0ne has quit IRC | 11:48 | |
| *** e0ne has joined #openstack-infra | 11:50 | |
| *** baoli has joined #openstack-infra | 11:52 | |
| *** teran has joined #openstack-infra | 11:53 | |
| openstackgerrit | Bharat Kumar Kobagana (BharatK) proposed openstack-infra/project-config: WIP: Add experimental glusterfs-manila job https://review.openstack.org/207425 | 11:53 |
| *** rfolco has joined #openstack-infra | 11:54 | |
| *** thorst has joined #openstack-infra | 11:55 | |
| *** weshay has joined #openstack-infra | 11:55 | |
| *** fhubik has joined #openstack-infra | 11:56 | |
| *** baoli has quit IRC | 11:56 | |
| *** baoli has joined #openstack-infra | 11:57 | |
| tobiash | hi, how do you upgrade zuul in the openstack infrastructure without loosing gerrit events? Do you schedule a downtime or is it possible to upgrade zuul seamlessly? | 11:57 |
| openstackgerrit | Steve Kowalik proposed openstack-infra/jeepyb: Add a register-zanata-projects script https://review.openstack.org/207729 | 11:58 |
| *** ddieterly has joined #openstack-infra | 11:59 | |
| *** amotoki has quit IRC | 12:03 | |
| *** ddieterly has quit IRC | 12:04 | |
| *** abregman|brb has quit IRC | 12:04 | |
| *** marzif_ has quit IRC | 12:05 | |
| *** kushal has quit IRC | 12:05 | |
| *** marcusvrn has joined #openstack-infra | 12:06 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Add newlines to the ends of files https://review.openstack.org/207856 | 12:07 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Add newlines to the ends of files https://review.openstack.org/207856 | 12:08 |
| *** deepakcs has quit IRC | 12:08 | |
| *** moravec has quit IRC | 12:09 | |
| *** gordc has joined #openstack-infra | 12:09 | |
| *** yfried__ has joined #openstack-infra | 12:10 | |
| *** olaph has joined #openstack-infra | 12:12 | |
| *** ddieterly has joined #openstack-infra | 12:14 | |
| *** abregman|brb has joined #openstack-infra | 12:18 | |
| *** bswartz has quit IRC | 12:19 | |
| *** jasondotstar has joined #openstack-infra | 12:23 | |
| *** ddieterly has quit IRC | 12:24 | |
| *** BharatK_ has quit IRC | 12:24 | |
| *** baoli has quit IRC | 12:24 | |
| *** tlian has joined #openstack-infra | 12:31 | |
| *** tonytan4ever has quit IRC | 12:33 | |
| *** mpaolino has quit IRC | 12:33 | |
| *** otter768 has joined #openstack-infra | 12:35 | |
| *** cburroughs has joined #openstack-infra | 12:35 | |
| *** coolsvap|away is now known as coolsvap | 12:36 | |
| *** _nadya_ has quit IRC | 12:36 | |
| *** _nadya_ has joined #openstack-infra | 12:36 | |
| *** kushal has joined #openstack-infra | 12:38 | |
| *** yamamoto has joined #openstack-infra | 12:39 | |
| *** _nadya_ has quit IRC | 12:40 | |
| *** yamamoto has quit IRC | 12:41 | |
| *** hdd has quit IRC | 12:41 | |
| *** yamamoto has joined #openstack-infra | 12:41 | |
| *** _nadya_ has joined #openstack-infra | 12:41 | |
| *** yamamoto has quit IRC | 12:41 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Add newlines to the ends of files https://review.openstack.org/207856 | 12:42 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Add newlines to the ends of files https://review.openstack.org/207856 | 12:42 |
| *** bapalm has joined #openstack-infra | 12:42 | |
| *** fhubik is now known as fhubik_afk | 12:43 | |
| *** hdd has joined #openstack-infra | 12:43 | |
| *** coolsvap is now known as coolsvap|away | 12:43 | |
| *** abregman|brb has quit IRC | 12:44 | |
| *** coolsvap|away is now known as coolsvap | 12:45 | |
| *** yamamoto has joined #openstack-infra | 12:47 | |
| *** yfried__ is now known as yfried|afk | 12:49 | |
| *** fhubik_afk is now known as fhubik | 12:50 | |
| *** baoli has joined #openstack-infra | 12:50 | |
| *** cbader has quit IRC | 12:51 | |
| sdague | fungi: so I haven't really been keeping up with why zuul has been so touchy this week. It might be really good to have someone in the know send a post mortem to the list to help people understand. | 12:51 |
| *** alivigni has joined #openstack-infra | 12:51 | |
| *** ddieterly has joined #openstack-infra | 12:52 | |
| *** bhunter71 has joined #openstack-infra | 12:52 | |
| *** ddieterly has quit IRC | 12:53 | |
| openstackgerrit | Greg Chapple proposed openstack-infra/git-review: Allow reviewers to be defined in .gitreview file https://review.openstack.org/207795 | 12:54 |
| *** glauco has joined #openstack-infra | 12:54 | |
| *** marzif_ has joined #openstack-infra | 12:54 | |
| fungi | sdague: agreed. i've been mostly out of touch this week as well | 12:54 |
| *** jasondotstar has quit IRC | 12:55 | |
| *** marzif_ has quit IRC | 12:56 | |
| pabelanger | morning | 12:56 |
| *** marzif_ has joined #openstack-infra | 12:56 | |
| openstackgerrit | will soula proposed openstack-infra/jenkins-job-builder: Add support for whitesource plugin https://review.openstack.org/204615 | 12:57 |
| fungi | sdague: i sort of caught the tail end of the resolution last night, but jhesketh has been the one in the trenches on most of the incidents and jeblair did the research and fixes so it's probably better for one of them to elaborate | 12:57 |
| *** yfried|afk is now known as yfried__ | 12:57 | |
| pabelanger | tobiash: schedule downtime | 12:57 |
| *** glauco has quit IRC | 12:59 | |
| *** jasondotstar has joined #openstack-infra | 12:59 | |
| sdague | fungi: looks like graphite crashed again | 13:03 |
| *** zz_dimtruck is now known as dimtruck | 13:03 | |
| *** dkranz has joined #openstack-infra | 13:04 | |
| *** bswartz has joined #openstack-infra | 13:04 | |
| akuznetsova | hi guys, is there a way to deploy environment the same as on bare-trusty/bare-precise node ? | 13:05 |
| akuznetsova | I found some puppets in system-config, but I am not sure what exactly I need to use | 13:06 |
| *** hashar has joined #openstack-infra | 13:07 | |
| *** PinkFreud has quit IRC | 13:07 | |
| pabelanger | akuznetsova: you can look at the prepare node script in project-config: https://github.com/openstack-infra/project-config/tree/master/nodepool/scripts | 13:07 |
| pabelanger | we have prepare_node_bare.sh for baren nodes | 13:07 |
| pabelanger | bare* | 13:07 |
| *** jsavak has joined #openstack-infra | 13:07 | |
| *** PinkFreud has joined #openstack-infra | 13:08 | |
| *** coolsvap is now known as coolsvap|away | 13:09 | |
| *** yfried__ is now known as yfried|afk | 13:10 | |
| *** mpaolino has joined #openstack-infra | 13:10 | |
| akuznetsova | pabelanger, thanks, I will try | 13:11 |
| *** yfried|afk is now known as yfried__ | 13:12 | |
| *** kdas__ has joined #openstack-infra | 13:13 | |
| *** salv-orlando has quit IRC | 13:13 | |
| *** amitgandhinz has joined #openstack-infra | 13:15 | |
| *** kushal has quit IRC | 13:15 | |
| *** kdas__ is now known as kushal | 13:16 | |
| *** kushal has quit IRC | 13:16 | |
| *** kushal has joined #openstack-infra | 13:16 | |
| *** coolsvap|away is now known as coolsvap | 13:16 | |
| *** bknudson has quit IRC | 13:16 | |
| *** mpaolino has quit IRC | 13:16 | |
| *** marzif_ has quit IRC | 13:17 | |
| *** tiswanso has joined #openstack-infra | 13:17 | |
| *** mriedem has joined #openstack-infra | 13:17 | |
| *** mpaolino has joined #openstack-infra | 13:17 | |
| StevenK | Queue lengths: 1318 events, 1724 results. Hmmm, I don't think Zuul is happy | 13:20 |
| fungi | StevenK: merge or reset in progress? | 13:20 |
| *** banix has joined #openstack-infra | 13:21 | |
| *** eharney has joined #openstack-infra | 13:21 | |
| StevenK | fungi: I don't think so, but my browser is *really* unhappy with zuul's status page | 13:22 |
| fungi | yeah, i expect the queue is just really huge, but i'll look when i'm at a computer with enough ram | 13:22 |
| *** bknudson has joined #openstack-infra | 13:23 | |
| *** kgiusti has joined #openstack-infra | 13:24 | |
| *** glauco_ has joined #openstack-infra | 13:24 | |
| *** sergmelikyan has joined #openstack-infra | 13:25 | |
| *** mriedem has quit IRC | 13:25 | |
| *** markvoelker has joined #openstack-infra | 13:26 | |
| StevenK | Hmmm, the little graphs next to the queue lengths look to have dropped to 0 as well, which makes them look visually odd | 13:27 |
| *** aysyd has joined #openstack-infra | 13:28 | |
| *** pilgrimstack1 has joined #openstack-infra | 13:28 | |
| *** mriedem has joined #openstack-infra | 13:28 | |
| *** kdas__ has joined #openstack-infra | 13:29 | |
| *** zeih has joined #openstack-infra | 13:29 | |
| *** markvoelker_ has joined #openstack-infra | 13:30 | |
| *** coolsvap is now known as coolsvap|away | 13:30 | |
| *** kushal has quit IRC | 13:31 | |
| *** markvoelker has quit IRC | 13:31 | |
| fungi | StevenK: yeah, graphite.o.o's carbon cache has likely oom'd again and needs a restart | 13:31 |
| *** jsavak has quit IRC | 13:31 | |
| *** kdas__ is now known as kushal | 13:31 | |
| *** kushal has joined #openstack-infra | 13:31 | |
| *** jsavak has joined #openstack-infra | 13:32 | |
| *** signed8b_ is now known as signed8bit_ZZZzz | 13:32 | |
| *** alaski is now known as lascii | 13:32 | |
| *** signed8bit_ZZZzz is now known as signed8b_ | 13:32 | |
| *** dansmith is now known as superdan | 13:32 | |
| *** annegentle has joined #openstack-infra | 13:33 | |
| *** dimtruck is now known as zz_dimtruck | 13:34 | |
| fungi | [Fri Jul 31 10:38:21 2015] Out of memory: Kill process 32197 (carbon-cache.py) score 753 or sacrifice child | 13:35 |
| *** moravec has joined #openstack-infra | 13:35 | |
| *** yfried__ is now known as yfried|afk | 13:36 | |
| fungi | restarted now | 13:36 |
| *** ZZelle has quit IRC | 13:37 | |
| StevenK | fungi: Huh, I wonder if that was blocking something inside zuul, since the queues are emptying | 13:38 |
| *** rguillebert has joined #openstack-infra | 13:38 | |
| fungi | StevenK: nah, results queue up if they can't be processed fast enough, and then as they're processed many of those translate into more events | 13:39 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for instance_extra DBDeadlock bug 1480305 https://review.openstack.org/207902 | 13:39 |
| openstack | bug 1480305 in OpenStack Compute (nova) "FloatingIPsTestJSON fails with DBDeadlock inserting into instance_extra" [Medium,Confirmed] https://launchpad.net/bugs/1480305 | 13:39 |
| *** moravec has quit IRC | 13:40 | |
| fungi | it seems to be burning through things sensibly | 13:40 |
| fungi | it just has a lot to do | 13:40 |
| EmilienM | fungi: do you have root permissions on Gerrit? | 13:41 |
| fungi | EmilienM: yeah, why? | 13:41 |
| *** moravec has joined #openstack-infra | 13:41 | |
| EmilienM | fungi: I have a request: https://review.openstack.org/#/c/207596/ - and add me + crinkle to the new group (I can add her myself also) | 13:42 |
| *** moravec has quit IRC | 13:43 | |
| *** ayoung is now known as ayoung-shh-busy | 13:43 | |
| *** zz_dimtruck is now known as dimtruck | 13:46 | |
| fungi | EmilienM: that group doesn't seem to exist yet. something may be broken with our acl update automation | 13:47 |
| EmilienM | fungi: but the patch is not merged | 13:47 |
| EmilienM | maybe I'm missing something | 13:47 |
| EmilienM | I'm trying to create the group actually | 13:47 |
| fungi | EmilienM: oh! the patch isn't approved yet | 13:48 |
| EmilienM | fungi: yeah :-) | 13:48 |
| fungi | EmilienM: sorry, in fix-what's-breaking mode rather than review-some-changes mode. lgtm, approving | 13:48 |
| *** coolsvap|away is now known as coolsvap | 13:48 | |
| EmilienM | fungi: no problem ! infra is having hard time this week :( | 13:48 |
| *** johnbelamaric has joined #openstack-infra | 13:50 | |
| pabelanger | EmilienM: fungi I think the patch will fail. | 13:50 |
| pabelanger | shouldn't it be: create = group puppet-manager-release | 13:51 |
| *** coolsvap is now known as coolsvap|away | 13:51 | |
| EmilienM | oops | 13:52 |
| fungi | oh, yep gerrit's going to fail to apply that acl. good catch | 13:52 |
| EmilienM | fungi: can you -2 before it merges? | 13:52 |
| *** tvanderwiel has joined #openstack-infra | 13:52 | |
| fungi | i unapproved it | 13:52 |
| EmilienM | pabelanger: thanks | 13:52 |
| *** tvanderwiel has left #openstack-infra | 13:52 | |
| fungi | it won't merge without a workflow +1 | 13:52 |
| pabelanger | EmilienM: some issue is in other puppet-*.configs | 13:52 |
| *** pal has joined #openstack-infra | 13:54 | |
| fungi | i assume you meant s/some/same/ | 13:54 |
| fungi | if so, i agree and have marked them just now | 13:54 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: gerrit/puppet-openstack: create puppet-manager-release group https://review.openstack.org/207596 | 13:54 |
| *** jaosorior has quit IRC | 13:54 | |
| fungi | pabelanger: if patchset #2 looks right to you, then i'm happy to reapprove it | 13:56 |
| *** yfried|afk is now known as yfried__ | 13:56 | |
| tobiash | pabelanger: thanks for the information | 13:57 |
| kragniz | how's zuul at the moment? | 13:59 |
| *** sergmelikyan has quit IRC | 14:00 | |
| *** sergmelikyan has joined #openstack-infra | 14:00 | |
| kragniz | https://review.openstack.org/#/c/202793 seems to be a bit stuck | 14:00 |
| *** banix has quit IRC | 14:01 | |
| fungi | kragniz: catching up slowly. i'm giving everything a careful going-over to make sure the system is healthy | 14:01 |
| kragniz | fungi: awesome, thanks! | 14:01 |
| nim | hi | 14:01 |
| fungi | kragniz: i don't see any jobs running for that change | 14:02 |
| *** rguillebert has quit IRC | 14:02 | |
| kragniz | fungi: I didn't, either | 14:02 |
| fungi | kragniz: oh, it needs to be reapproved. unset your workflow +1 and then set it again | 14:02 |
| *** jgriffith has quit IRC | 14:02 | |
| *** dtantsur|lunch is now known as dtantsur | 14:02 | |
| nim | hi fc passthrough is not working with new qemu version installing by devstack | 14:03 |
| fungi | kragniz: recheck/reverify when there's already a jenkins +1 verified won't do any good. current limitation of how we use gerrit, it doesn't emit vote data in its event stream if an existing vote is replaced by a new vote of teh same value | 14:03 |
| nim | in ubuntu 14.04 | 14:03 |
| kragniz | fungi: can I do that myself, or do I need the person who set workflow +1 originally to do it? | 14:03 |
| kragniz | fungi: okay, good to know | 14:03 |
| fungi | kragniz: i believe i heard this is fixed in newer gerrit releases, once we're able to safely upgrade | 14:03 |
| Shrews | reed: I sent you an email about the shade problem we discussed last night | 14:03 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:03 | |
| *** jgriffith has joined #openstack-infra | 14:04 | |
| fungi | kragniz: oh, if you're a core reviewer on that project, just add an additional workflow +1 | 14:04 |
| fungi | kragniz: if you're not, you need to get a core reviewer for glance to add an additional workflow +1 or have flaper87 unset and readd his (if he's around) | 14:04 |
| kragniz | fungi: cool, I just did it myself | 14:05 |
| *** ParsectiX has quit IRC | 14:05 | |
| openstackgerrit | John Trowbridge proposed openstack/diskimage-builder: Add ability to build ironic-python-agent ramdisk from packages https://review.openstack.org/207601 | 14:05 |
| kragniz | looks like a job kicked off, so we're all good :) | 14:05 |
| kragniz | thanks for the help | 14:05 |
| fungi | nim: exciting. i don't think we use fc passthrough. you might want to discuss that with ubuntu and/or nova developers? | 14:05 |
| nim | its part of cinder third party testing | 14:06 |
| nim | https://git.openstack.org/cgit/stackforge/third-party-ci-tools/tree/provisioning_scripts/fibre_channel | 14:06 |
| *** yfried__ is now known as yfried|afk | 14:06 | |
| nim | this is what i am facing http://paste.openstack.org/show/406551/ | 14:07 |
| fungi | nim: neat. maybe there's someone here who knows about it, but it's not really on topic | 14:07 |
| nim | rasselin patrickeast might know | 14:08 |
| *** jsavak has quit IRC | 14:09 | |
| *** jsavak has joined #openstack-infra | 14:09 | |
| *** ashleighfarnham has quit IRC | 14:10 | |
| *** nelsnelson has joined #openstack-infra | 14:10 | |
| *** zeih has quit IRC | 14:11 | |
| pabelanger | annegentle: Do you mind deleting the icehouse-update branch from https://github.com/trystack/trystack_org_webcontents | 14:12 |
| pabelanger | annegentle: we plan on importing it into openstack-infra, and just doing some house cleaning on it | 14:12 |
| *** jgriffith has quit IRC | 14:13 | |
| *** dimtruck is now known as zz_dimtruck | 14:14 | |
| fungi | grar! NotFound: No Image matching {'name': 'Ubuntu Server 12.04.5 LTS (amd64 20150127) - Partner Image'}. (HTTP 404) | 14:16 |
| fungi | i think hpcloud has retired/renamed base images on us again | 14:16 |
| *** rguillebert has joined #openstack-infra | 14:16 | |
| fungi | yep, that's now "Ubuntu Server 12.04.5 LTS (amd64 20150127) - Partner Image (deprecated)" | 14:17 |
| fungi | and looks like we need to switch to "Ubuntu Server 12.04.5 LTS (amd64 20150707) - Partner Image" | 14:18 |
| fungi | patch on the way | 14:18 |
| *** abregman has joined #openstack-infra | 14:19 | |
| *** amitgandhinz has quit IRC | 14:20 | |
| fungi | yeah, they deprecated the 14.04.1 we were using too | 14:21 |
| *** jgriffith has joined #openstack-infra | 14:21 | |
| nim | fungi: passthrogh is working on 14.04.1 but not on 14.04.2 | 14:22 |
| nim | fungi: going with 14.04.1 as of now | 14:22 |
| nim | :) | 14:22 |
| annegentle | pabelanger: done | 14:22 |
| fungi | nim: not really relevant to what i'm working on, but good to know i guess in case we ever end up needing it | 14:23 |
| *** bapalm_ has joined #openstack-infra | 14:23 | |
| pabelanger | annegentle: danke | 14:23 |
| *** mtanino has joined #openstack-infra | 14:24 | |
| EmilienM | fungi: paul +1 https://review.openstack.org/#/c/207596 (puppet/gerrit) | 14:25 |
| *** annegentle has quit IRC | 14:26 | |
| *** prad has joined #openstack-infra | 14:26 | |
| *** bapalm has quit IRC | 14:26 | |
| *** MarkAtwood has joined #openstack-infra | 14:27 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Update Ubuntu base image names for hpcloud https://review.openstack.org/207918 | 14:28 |
| fungi | jeblair: clarkb: mordred: pleia2: ^ | 14:28 |
| greghaynes | oh, new hp images? | 14:28 |
| *** Ala has quit IRC | 14:28 | |
| fungi | greghaynes: that same old fire drill, yes | 14:29 |
| greghaynes | fun fun | 14:29 |
| *** cbader has joined #openstack-infra | 14:30 | |
| *** nelsnelson has quit IRC | 14:30 | |
| *** amitgandhinz has joined #openstack-infra | 14:31 | |
| *** nelsnelson has joined #openstack-infra | 14:31 | |
| *** nijaba has joined #openstack-infra | 14:31 | |
| fungi | hpcloud is throwing a lot of 500 errors when trying to attach floating ips again | 14:31 |
| fungi | also rax-dfw claims we're over quota. presumably nodes stuck in delete state again | 14:32 |
| *** zz_dimtruck is now known as dimtruck | 14:32 | |
| fungi | seems like johnthetubaguy/alaski said that was a recent regression in nova cells | 14:32 |
| *** glauco_ has quit IRC | 14:32 | |
| clarkb | ya | 14:32 |
| clarkb | for 500 errors with fips did we leak them? | 14:33 |
| fungi | clarkb: i don't think so, since i'd just cleaned some up a few minutes prior and there were <100 | 14:34 |
| *** btully has joined #openstack-infra | 14:36 | |
| pabelanger | fungi: you'd think they'd send out an email about the image renames :) | 14:38 |
| bhunter71 | anyone know of something changing with python cron? I have a few places where I disable things by setting cron to Feb 31, and.. these tasks seem to have started running in both zuul and nodepool. Thanks! | 14:39 |
| phschwartz | I have a pretty large jjb setup for creating jobs on multiple jenkins servers. I just added a set of labels that go to hosts based on our cloud regions. Currently for jobs that I want to run on these servers I have a job-template with node: label1 that I want to pass the node to use in from the project block. I.e. label-ord, lable-dfw, etc. I just can't find a clean way to do it without creating the jobs directly and not using a job template | 14:39 |
| phschwartz | (currently used to create 25 jobs) | 14:39 |
| phschwartz | Any thouhgs? | 14:39 |
| phschwartz | *thoughts | 14:39 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: fwaas: Enable python3 job https://review.openstack.org/207927 | 14:40 |
| *** markvoelker_ has quit IRC | 14:40 | |
| fungi | phschwartz: we do that currently. have the template specify node: {node} | 14:40 |
| *** glauco has joined #openstack-infra | 14:40 | |
| *** ajmiller has joined #openstack-infra | 14:40 | |
| fungi | phschwartz: and then set node: somevalue in your project entries | 14:40 |
| *** fawadkhaliq has quit IRC | 14:41 | |
| *** tiswanso has quit IRC | 14:41 | |
| fungi | phschwartz: example http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n61 | 14:42 |
| *** ayoung-shh-busy is now known as ayoung | 14:42 | |
| *** xyang0 has joined #openstack-infra | 14:42 | |
| *** tiswanso has joined #openstack-infra | 14:43 | |
| phschwartz | fungi: interesting. I tried that and get TypeError: cannot serialize {'node': None} (type dict) | 14:43 |
| *** lxsli is now known as lexloofer | 14:43 | |
| *** jgriffith has quit IRC | 14:43 | |
| *** sabeen1 has joined #openstack-infra | 14:44 | |
| fungi | phschwartz: are you making sure to provide it for all your instantiations of that job-template? | 14:44 |
| *** coolsvap|away is now known as coolsvap | 14:44 | |
| *** sabeen2 has joined #openstack-infra | 14:45 | |
| phschwartz | fungi: yes. Just verified that it is set for each | 14:45 |
| fungi | don't know what else to suggest other than look at our configs for anything you might be doing differently, and try to reduce your problem set to the simplest possible test case to make it easier to scrutinize | 14:47 |
| swat30 | hey, not sure if you guys are responsible for the website, but openstack.org is displaying an Nginx 404 | 14:47 |
| fungi | swat30: we're not, but i'll bring it to the attention of those who are | 14:47 |
| swat30 | cool, tks fungi | 14:47 |
| *** hashar has quit IRC | 14:48 | |
| fungi | swat30: though it's reachable just fine for me. may be a problem with one of their third-party proxy pool | 14:48 |
| fungi | swat30: you mean http://www.openstack.org/ specifically, yes? | 14:48 |
| swat30 | actually, just noticed, it's https:// | 14:48 |
| *** sabeen1 has quit IRC | 14:48 | |
| *** jsavak has quit IRC | 14:49 | |
| fungi | hrm, yeah not sure if that exists/existed. checking | 14:49 |
| swat30 | fungi, Google uses https:// | 14:49 |
| fungi | yeah, i have some cached urls that indicate it worked at one time | 14:49 |
| fungi | thanks, that's some good detail | 14:49 |
| swat30 | cool, np | 14:50 |
| pabelanger | fungi: do you know the reason why we lose graphite? or the stats at the bottom on status.o.o/zuul? | 14:50 |
| clarkb | pabelanger carbon-cache OOMs | 14:51 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Add openstack-infra/puppet-infra-cookiecutter project https://review.openstack.org/207109 | 14:51 |
| pabelanger | clarkb: ouch, that sucks | 14:51 |
| clarkb | looks like it may be a newish leak but also possibly tied to our large wueue sizes? | 14:51 |
| clarkb | cacti definitely shows a new uptick in memory use that wasnt there before | 14:52 |
| *** jgriffith has joined #openstack-infra | 14:52 | |
| *** salv-orlando has joined #openstack-infra | 14:53 | |
| *** yfried|afk is now known as yfried__ | 14:53 | |
| phschwartz | fungi: Here is a cut down test that is failing. http://paste.openstack.org/show/406602/ running it with `jenkins-jobs test -o test test.yaml` | 14:53 |
| phschwartz | fungi: here is the return http://paste.openstack.org/show/406603/ | 14:53 |
| phschwartz | fungi: jjb version 1.2.0 which pip is returning as the latest | 14:53 |
| *** jsavak has joined #openstack-infra | 14:54 | |
| *** fawadkhaliq has joined #openstack-infra | 14:54 | |
| *** Somay has quit IRC | 14:54 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/system-config: Remove http.maxWait hack as fixed in Gerrit 2.5 https://review.openstack.org/207941 | 14:55 |
| *** rguillebert has quit IRC | 14:55 | |
| fungi | phschwartz: sorry, i meant node: '{node}' like at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/python-jobs.yaml#n19 | 14:56 |
| fungi | phschwartz: jjb and yaml like to fight over what { and } mean | 14:57 |
| *** dingyichen has quit IRC | 14:57 | |
| *** annegentle has joined #openstack-infra | 14:57 | |
| phschwartz | fungi: ah, that did it. ty | 14:57 |
| fungi | np | 14:57 |
| *** chlong has quit IRC | 14:57 | |
| fungi | pyyaml is interpreting {node} as a literal dict so jjb never gets a chance to replace the node parameter because it expects a string with the parameter name embedded | 14:58 |
| *** jsavak has quit IRC | 14:58 | |
| *** jsavak has joined #openstack-infra | 14:58 | |
| *** fawadkhaliq has quit IRC | 14:59 | |
| *** hdd has quit IRC | 15:00 | |
| *** jasondotstar has quit IRC | 15:03 | |
| *** BrandonL08 has joined #openstack-infra | 15:04 | |
| *** vlaza has left #openstack-infra | 15:05 | |
| *** fawadkhaliq has joined #openstack-infra | 15:06 | |
| *** rguillebert has joined #openstack-infra | 15:07 | |
| *** hdd has joined #openstack-infra | 15:07 | |
| *** glauco has quit IRC | 15:07 | |
| *** tonytan4ever has joined #openstack-infra | 15:10 | |
| *** maurosr has quit IRC | 15:10 | |
| *** zz_ja has quit IRC | 15:10 | |
| *** chlong has joined #openstack-infra | 15:11 | |
| *** zz_ja has joined #openstack-infra | 15:11 | |
| *** maurosr has joined #openstack-infra | 15:12 | |
| pleia2 | happy sysadmin day, my friends :) | 15:13 |
| *** ashleighfarnham has joined #openstack-infra | 15:14 | |
| *** bpokorny has joined #openstack-infra | 15:15 | |
| nigelb | pleia2: here's to a read-only friday on sysadmin appreciation day ;) | 15:16 |
| *** dannywilson has joined #openstack-infra | 15:16 | |
| pleia2 | hah, yeah right | 15:16 |
| *** dims_ has quit IRC | 15:17 | |
| *** dannywilson has quit IRC | 15:17 | |
| pleia2 | it's launch grafana.openstack.org day! (after coffee) | 15:17 |
| *** dims_ has joined #openstack-infra | 15:17 | |
| *** dannywilson has joined #openstack-infra | 15:17 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for instance_extra DBDeadlock bug 1480305 https://review.openstack.org/207902 | 15:18 |
| openstack | bug 1480305 in OpenStack Compute (nova) "FloatingIPsTestJSON fails with DBDeadlock inserting into instance_extra" [Medium,Confirmed] https://launchpad.net/bugs/1480305 | 15:19 |
| *** pal has quit IRC | 15:19 | |
| *** pal has joined #openstack-infra | 15:20 | |
| * fungi would rather have a write-only day | 15:20 | |
| fungi | so much to get done | 15:20 |
| *** dims_ is now known as dimsum__ | 15:21 | |
| *** tonytan4ever has quit IRC | 15:22 | |
| *** tonytan_brb has joined #openstack-infra | 15:22 | |
| *** devvesa has quit IRC | 15:23 | |
| *** pal_ has joined #openstack-infra | 15:23 | |
| *** olaph has quit IRC | 15:24 | |
| *** armax has joined #openstack-infra | 15:24 | |
| *** pal has quit IRC | 15:24 | |
| *** austin81 has joined #openstack-infra | 15:24 | |
| *** olaph has joined #openstack-infra | 15:25 | |
| *** jasondotstar has joined #openstack-infra | 15:26 | |
| *** bharath has joined #openstack-infra | 15:27 | |
| jeblair | i'm signing up for an ovh account. it's all in french, and google translate gives me: | 15:29 |
| jeblair | "A set of guides is not available to explain how not to manage your services:" | 15:30 |
| jeblair | which is, tbh, a huge disappointment. i think we could really use a set of guides to explain how not to use openstack. | 15:30 |
| *** yfried__ is now known as yfried|afk | 15:30 | |
| *** puranamr has joined #openstack-infra | 15:31 | |
| *** sergmelikyan has quit IRC | 15:32 | |
| *** bhuvan has joined #openstack-infra | 15:32 | |
| *** BrandonL08 has quit IRC | 15:34 | |
| *** tonytan_brb is now known as tonytan4ever | 15:35 | |
| *** flepied1 has quit IRC | 15:35 | |
| *** flepied has joined #openstack-infra | 15:35 | |
| *** jcoufal has quit IRC | 15:35 | |
| fungi | that's great | 15:35 |
| fungi | i thought they had an english-language site too, or maybe that was just for runabove specifically | 15:36 |
| *** scheuran has quit IRC | 15:36 | |
| jeblair | fungi: apparently the ovh cloud is only accepting french accounts right now | 15:37 |
| openstackgerrit | Anastasia Kuznetsova proposed openstack-infra/project-config: Add Mistral gate that run unit tests and use postgresql https://review.openstack.org/207961 | 15:37 |
| *** hdd has quit IRC | 15:39 | |
| fungi | ahh | 15:40 |
| clarkb | runabove has english | 15:40 |
| fungi | jeblair: put on a beret and fake an accent. you'll do fine | 15:40 |
| jeblair | fungi: pretty much what i did; the address is "Austin, TX, USA, France" and the phone number is the paris opera W hotel. | 15:41 |
| *** flepied1 has joined #openstack-infra | 15:42 | |
| *** yfried|afk has quit IRC | 15:42 | |
| jeblair | the management interface supports english | 15:42 |
| fungi | granted, that is a pretty nice hotel | 15:42 |
| *** puranamr has quit IRC | 15:43 | |
| jeblair | creds are in the password fire | 15:43 |
| jeblair | er, file | 15:43 |
| fungi | thanks! | 15:43 |
| jeblair | "i'm chilled; throw another password on the fire!" | 15:43 |
| jeblair | those long ones burn the best | 15:44 |
| clarkb | can I get bbq in austin, france? | 15:44 |
| fungi | (...cok me up some bacon and some beans, go out to the car and change the tire, wash my socks and mend my old bluejeans...) | 15:44 |
| jeblair | that particular typo is decidedly a dvorak typo -- you don't get that on qwerty keyboards | 15:44 |
| *** sergmelikyan has joined #openstack-infra | 15:45 | |
| jeblair | clarkb: i bet you can get it in paris, tx | 15:45 |
| *** flepied has quit IRC | 15:45 | |
| * fungi now has the muppet show video of the waylon jennings performance of tompall glaser's musical interpretation of the shel silverstein poem "put another log on the fire" stuck in his head | 15:47 | |
| openstackgerrit | Merged openstack/requirements: Adds python-magnumclient>=0.2.1 https://review.openstack.org/205366 | 15:47 |
| bpokorny | Does anyone know what I need to do to get this review to merge? https://review.openstack.org/#/c/206765/ | 15:48 |
| bpokorny | I thought the reverify would do it, but Jenkins came back with a success on that already. | 15:48 |
| bpokorny | It's no longer in the zuul queue. | 15:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Move puppet-syntax gate to devstack-centos7 https://review.openstack.org/207964 | 15:49 |
| fungi | bpokorny: have TravT remove and replace the workflow +1 vote on it, or add an additional workflow +1 yourself if you're a core reviewer on that repo | 15:49 |
| pabelanger | EmilienM: ^ see commit | 15:49 |
| fungi | bpokorny: or have some horizon core reviewer do the same | 15:49 |
| * jeblair registered for the summit -- AND actually remembered how to apply the discount code this time | 15:50 | |
| bpokorny | Thanks, fungi. I'll ask the horizon channel for that. | 15:50 |
| fungi | jeblair: it's great we make that so easy to figure out | 15:50 |
| fungi | jeblair: now if they can actually have my badge at the check-in this time instead of hunting for where it's stashed and then reprinting it, i'll be thrilled | 15:51 |
| jeblair | fungi: did the proposal bot get an invite? | 15:51 |
| *** shashank_hegde has joined #openstack-infra | 15:51 | |
| fungi | jeblair: sadly, no, i sobered up enough to strip its e-mail address out of the list of invitees | 15:51 |
| fungi | maybe next time | 15:51 |
| jeblair | fungi: no more working sober for you | 15:51 |
| fungi | indeed | 15:51 |
| EmilienM | pabelanger: ack | 15:52 |
| *** pleia2 has left #openstack-infra | 15:52 | |
| *** yamamoto has quit IRC | 15:52 | |
| *** pleia2 has joined #openstack-infra | 15:52 | |
| jeblair | you must be this drunk [jeblair holds his hand 2 feet above the ground] to operate atc machinery | 15:52 |
| fungi | it's the only way to cope while preserving your few remaining san points | 15:53 |
| *** jsavak has quit IRC | 15:53 | |
| jeblair | clarkb, fungi: wow, the io stats on graphite look way different with the latest restart | 15:54 |
| *** katyafervent is now known as katyafervent_awa | 15:55 | |
| jeblair | though, it looks like they were similarly write-heavy a few weeks ago... | 15:55 |
| jeblair | http://cacti.openstack.org/cacti/graph_image.php?action=view&local_graph_id=1927&rra_id=4 | 15:55 |
| jeblair | that doesn't make any sense to me | 15:55 |
| fungi | jeblair: well, this time i restarted not only carbon-cache but also statsd and apache2 | 15:58 |
| jeblair | oh interesting | 15:58 |
| fungi | so there may have been filesystem reads previously cached which i flushed | 15:58 |
| *** markvoelker has joined #openstack-infra | 15:59 | |
| jeblair | fungi: yeah, though it looks like writes are higher than previously | 15:59 |
| *** shashank_hegde has quit IRC | 15:59 | |
| fungi | in good news, with zuul back in a happy place and devs seeming to have taken friday off, we're nearly caught all the way back up again | 16:00 |
| zaro | morning | 16:00 |
| *** kushal has quit IRC | 16:00 | |
| *** yamamoto has joined #openstack-infra | 16:00 | |
| clarkb | fungi woot | 16:00 |
| jeblair | fungi: yeah, maybe 1.5-2 hours out from zero-backlog | 16:00 |
| fungi | also looks like we're burning through 1kjph now | 16:00 |
| clarkb | I hope to get sdagues nidepool devstack plugin working today | 16:01 |
| *** xyang0 has quit IRC | 16:01 | |
| jeblair | oh, if that's the case, then <1h | 16:01 |
| *** larainema has quit IRC | 16:01 | |
| clarkb | and work on turning that into a job. which should aid in zuulv3 work | 16:01 |
| *** jsavak has joined #openstack-infra | 16:01 | |
| *** xyang0 has joined #openstack-infra | 16:01 | |
| *** larainema has joined #openstack-infra | 16:01 | |
| jeblair | oh, hey, has anyone opened a rax-deleted-server ticket recently? | 16:02 |
| fungi | jeblair: no, though i saw we're consistently over-quota again today | 16:02 |
| *** coolsvap is now known as coolsvap|away | 16:02 | |
| *** markvoelker has quit IRC | 16:02 | |
| fungi | jeblair: however my friend they recently hired and then stuck on the hazing task of helping them maintain their cells patches says that regression has a fix being rolled out right now | 16:03 |
| jeblair | fungi: oh, are we? i only see two rax servers stuck in deleted | 16:03 |
| fungi | jeblair: it may have cleared up on its own in that case | 16:03 |
| *** armax has quit IRC | 16:03 | |
| *** puranamr has joined #openstack-infra | 16:03 | |
| fungi | i haven't looked again in the last hour or so | 16:03 |
| jeblair | yeah, and the 2 we have are in iad; i understand they were focusing on dfw, where we now have none | 16:03 |
| fungi | right, dfw was where they all were earlier when i was seeing the quota errors icoming back | 16:04 |
| *** kushal has joined #openstack-infra | 16:04 | |
| *** sergmelikyan has quit IRC | 16:04 | |
| jeblair | fungi: it looks like nodepool ocassionally is off by 4 | 16:05 |
| jeblair | which isn't that bad | 16:05 |
| *** tjones has joined #openstack-infra | 16:05 | |
| fungi | jeblair: oh, also that may in part be courtesy of template instances for snapshot image updates | 16:05 |
| *** yfried|afk has joined #openstack-infra | 16:05 | |
| fungi | since i think nodepool doesn't factor them into the quota | 16:06 |
| jeblair | yep, plus a few "monty-testing-something-for-shade" servers | 16:06 |
| fungi | ahh | 16:06 |
| *** AzherKhan has joined #openstack-infra | 16:06 | |
| *** aukhan has joined #openstack-infra | 16:06 | |
| *** jasondotstar has quit IRC | 16:06 | |
| *** markus_z has quit IRC | 16:06 | |
| jeblair | | 48cf2f7f-be29-434a-94d2-42b31ecabf02 | proposal.slave.openstack.org | SHUTOFF | - | Shutdown | public=23.253.239.202; private=10.208.195.86 | | 16:06 |
| *** flepied has joined #openstack-infra | 16:06 | |
| jeblair | fungi: time to delete that one? | 16:06 |
| fungi | so anyway, we may need to adjust our overhead clearance again, until we can stop building some of those | 16:06 |
| fungi | jeblair: yep, i can but feel free if you're already right there | 16:07 |
| *** johnbelamaric has quit IRC | 16:07 | |
| jeblair | fungi: i am, will do | 16:07 |
| fungi | jeblair: for the record, i put the replacement proposal and release slaves in the ci tenant rather than the jenkins tenant | 16:07 |
| pleia2 | pabelanger: lmk when you're around, sorting out how we want to do sizing on this grafana.o.o system (flavor for the VM and what we need in terms of mysql) | 16:08 |
| *** geoffarnold has joined #openstack-infra | 16:08 | |
| fungi | can rebuild them fairly easily though if people disagree with that choice | 16:08 |
| *** glauco_ has joined #openstack-infra | 16:08 | |
| jeblair | fungi: that works for me | 16:08 |
| fungi | makes the nodepool quota calculations a little easier in my opinion | 16:09 |
| *** flepied1 has quit IRC | 16:09 | |
| *** ildikov has quit IRC | 16:09 | |
| jeblair | fungi: we had 4 old servers in error state plun the proposal; i've deleted all of those | 16:09 |
| jeblair | plus | 16:09 |
| fungi | thanks! | 16:09 |
| jeblair | lascii: hi, we don't have any servers in dfw stuck in 'deleting'! :) | 16:10 |
| lascii | jeblair: awesome! | 16:10 |
| *** geoffarnold has quit IRC | 16:10 | |
| fungi | lascii: i heard from randall there was a fix for that in the process of rolling out today. does that mean it's likely over in dfw now? | 16:11 |
| *** yamamoto has quit IRC | 16:11 | |
| lascii | we moved back to using the sqlalchemy instance delete rather than our homegrown one and that seems to be making a big difference | 16:11 |
| *** geoffarnold has joined #openstack-infra | 16:11 | |
| fungi | ooh, excellent | 16:11 |
| *** pilgrimstack1 has quit IRC | 16:11 | |
| lascii | yes, dfw should be totally updated now | 16:11 |
| jeblair | lascii: the only ones we have now are 2 in iad, and they are from multiple days ago | 16:11 |
| jeblair | i'll just open a ticket to ask for them to be cleaned up | 16:12 |
| *** glauco_ has quit IRC | 16:12 | |
| *** amitgandhinz has quit IRC | 16:13 | |
| lascii | iad should have been deployed on tuesday, so that makes sense | 16:13 |
| *** e0ne has quit IRC | 16:13 | |
| lascii | if it happens again just ping me and I'll dig in, since I added some additional logging around that case. just in case we are missing a case still | 16:13 |
| *** ashuk has joined #openstack-infra | 16:13 | |
| *** aukhan has quit IRC | 16:13 | |
| *** amitgandhinz has joined #openstack-infra | 16:14 | |
| jeblair | lascii: cool, will do, thanks :) | 16:14 |
| *** AzherKhan has quit IRC | 16:14 | |
| *** yamamoto has joined #openstack-infra | 16:14 | |
| openstackgerrit | Tim Hinrichs proposed openstack-infra/irc-meetings: Change Congress meeting day/time https://review.openstack.org/207981 | 16:14 |
| jeblair | Shrews, mordred, clarkb: regarding rax servers showing up without IP information, we got a response to the ticket we opened: | 16:15 |
| jeblair | "This is indicative of a build that failed to network properly. You can either try to "connect" PublicNet and ServiceNet using the control panel or you can delete the servers and retry your builds." | 16:15 |
| clarkb | ok so nodepools behavior of delete and try again is sane | 16:15 |
| jeblair | sounds like it | 16:15 |
| jeblair | also sounds pretty similar to the other (older?) behavior we have seen where we get an error if they don't have an ip address ready | 16:16 |
| openstackgerrit | Tim Hinrichs proposed openstack-infra/irc-meetings: Change Congress meeting day/time https://review.openstack.org/207981 | 16:16 |
| jeblair | ticket opened; hopefully the last! | 16:18 |
| *** jsavak has quit IRC | 16:19 | |
| openstackgerrit | Tim Hinrichs proposed openstack-infra/irc-meetings: Change Congress meeting day/time https://review.openstack.org/207981 | 16:19 |
| *** jsavak has joined #openstack-infra | 16:19 | |
| jroll | jeblair: why can't nodepool just go to the control panel and connect those? :) | 16:20 |
| *** signed8bit has joined #openstack-infra | 16:21 | |
| *** ildikov has joined #openstack-infra | 16:21 | |
| *** unicell has joined #openstack-infra | 16:22 | |
| *** unicell1 has quit IRC | 16:23 | |
| *** yfried|afk has quit IRC | 16:24 | |
| jeblair | jroll: its trackball is broken? | 16:24 |
| *** mriedem has quit IRC | 16:24 | |
| pabelanger | pleia2: sure. I think we can do the default size of a node. is that medium? | 16:24 |
| *** ybathia has joined #openstack-infra | 16:24 | |
| jroll | jeblair: :P | 16:24 |
| Shrews | jeblair: i must have missed the discussion of that problem | 16:24 |
| *** signed8b_ has quit IRC | 16:24 | |
| jeblair | Shrews: it's the thing mordred put more debugging info in for (where we got a server record back without any ip info at all) | 16:25 |
| pleia2 | pabelanger: we typically go with an 8G of ram node unless there's a good reason to do more | 16:25 |
| *** ashuk has quit IRC | 16:25 | |
| pabelanger | pleia2: Ya, I think that is a good place to start | 16:25 |
| pabelanger | pleia2: a lot of the processing of graphs defaults to client side | 16:26 |
| pleia2 | great, I'll log into the rackspace interface to see what our options are database-wise | 16:26 |
| *** sdake has quit IRC | 16:26 | |
| pabelanger | cool | 16:26 |
| fungi | fwiw, we often size general purpose servers smaller than that if we know they won't need that beefy of an instance | 16:26 |
| fungi | but it's a good middle-ground | 16:26 |
| pleia2 | thanks fungi | 16:27 |
| fungi | and if we put the data on a cinder volume and/or in a trove instance, then we can replace the nova instance for it with a more appropriately sized flavor once we collect some cacti stats | 16:27 |
| pleia2 | I was planning on using trove | 16:27 |
| fungi | without having to copy lots of stuff around | 16:27 |
| *** pvaneck has joined #openstack-infra | 16:28 | |
| fungi | yeah, no idea if grafana keeps persistent data on-disk as well or entirely in the db | 16:28 |
| *** dtantsur is now known as dtantsur|afk | 16:30 | |
| *** ddieterly has joined #openstack-infra | 16:31 | |
| pleia2 | pabelanger: any idea what we're looking at data-size for the database? looks like we tend to do 2-4G on the low end, more for things like gerrit and the etherpads | 16:37 |
| *** sdake has joined #openstack-infra | 16:37 | |
| *** baoli has quit IRC | 16:37 | |
| *** ddieterly has quit IRC | 16:37 | |
| *** glauco_ has joined #openstack-infra | 16:37 | |
| *** Guest80 has joined #openstack-infra | 16:39 | |
| *** hemna is now known as chewbacca | 16:40 | |
| clarkb | the database should just be graph configs and the real data is on the graphite host right? | 16:41 |
| clarkb | pleia2: I would expect it to be quite small | 16:41 |
| *** chewbacca is now known as wookie | 16:41 | |
| *** wookie is now known as Guest9155 | 16:42 | |
| *** jistr has quit IRC | 16:42 | |
| pleia2 | ah, good | 16:43 |
| *** briancline has joined #openstack-infra | 16:43 | |
| pleia2 | DFW or ORD? | 16:44 |
| *** krtaylor has quit IRC | 16:44 | |
| clarkb | dfw is our default for services | 16:44 |
| pleia2 | I should do more server spinups, so rusty | 16:44 |
| pleia2 | thanks | 16:44 |
| *** wenlock has joined #openstack-infra | 16:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Rename -translation IRC channel to -i18n https://review.openstack.org/205979 | 16:45 |
| *** fhubik has quit IRC | 16:45 | |
| *** ddieterly has joined #openstack-infra | 16:45 | |
| pabelanger | pleia2: Ya, I think that is fine too | 16:46 |
| pabelanger | pleia2: I don't think we'll end up storing much data | 16:46 |
| pabelanger | since we pull it all in from external sources | 16:46 |
| *** amitgandhinz has quit IRC | 16:46 | |
| *** Guest9155 is now known as pipercolt | 16:47 | |
| *** crc32 has joined #openstack-infra | 16:47 | |
| *** marzif_ has joined #openstack-infra | 16:47 | |
| *** tjones has quit IRC | 16:48 | |
| *** teran has quit IRC | 16:48 | |
| *** ivar-lazzaro has joined #openstack-infra | 16:49 | |
| *** ddieterly has quit IRC | 16:49 | |
| *** ivar-lazzaro has quit IRC | 16:49 | |
| *** amit213 has quit IRC | 16:50 | |
| *** shashank_hegde has joined #openstack-infra | 16:50 | |
| *** amit213 has joined #openstack-infra | 16:50 | |
| *** ivar-lazzaro has joined #openstack-infra | 16:50 | |
| *** glauco_ has quit IRC | 16:51 | |
| *** pal_ has quit IRC | 16:52 | |
| *** _nadya_ has quit IRC | 16:52 | |
| *** shashank_hegde has quit IRC | 16:52 | |
| *** tjones has joined #openstack-infra | 16:52 | |
| *** _nadya_ has joined #openstack-infra | 16:53 | |
| *** pal has joined #openstack-infra | 16:53 | |
| pleia2 | pabelanger: the secret_key is just something we generate like a password, nothing special about it? | 16:53 |
| pabelanger | pleia2: ya, that can be a hash or something | 16:53 |
| pabelanger | nothing special | 16:53 |
| pleia2 | cool | 16:53 |
| *** tonytan4ever has quit IRC | 16:54 | |
| *** glauco has joined #openstack-infra | 16:54 | |
| openstackgerrit | Merged openstack-infra/project-config: gerrit/puppet-openstack: create puppet-manager-release group https://review.openstack.org/207596 | 16:54 |
| openstackgerrit | Merged openstack-infra/system-config: Update Ubuntu base image names for hpcloud https://review.openstack.org/207918 | 16:55 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Add option to ignore ref-updated events emitted by branch deletions https://review.openstack.org/178833 | 16:55 |
| pleia2 | pabelanger: does it use the root mysql password? | 16:56 |
| *** jsavak has quit IRC | 16:56 | |
| *** jsavak has joined #openstack-infra | 16:56 | |
| pabelanger | pleia2: yes | 16:57 |
| *** krtaylor has joined #openstack-infra | 16:57 | |
| *** _nadya_ has quit IRC | 16:57 | |
| pabelanger | pleia2: grafana will create tables as root user | 16:57 |
| pleia2 | fungi, clarkb - I don't see us using root on other trove instances, should I just add it as another user account? | 16:57 |
| *** olaph has quit IRC | 16:57 | |
| *** jyuso1 has joined #openstack-infra | 16:57 | |
| *** sergmelikyan has joined #openstack-infra | 16:59 | |
| *** olaph has joined #openstack-infra | 16:59 | |
| fungi | pleia2: pabelanger: right, you need to precreate the database and tell grafana the host, database and user/password to use to create its tables within that database | 17:00 |
| fungi | so also precreate the user you want grafana to use for that and grant it full access to the database you create for the trove instance | 17:01 |
| zaro | pelix: had a question for you in https://review.openstack.org/#/c/180185/ | 17:03 |
| *** e0ne has joined #openstack-infra | 17:03 | |
| pleia2 | fungi: so using the rackspace database interface, I'm not seeing a whole lot of options | 17:03 |
| fungi | pleia2: yep, you just add a database and then add an initial database user | 17:03 |
| *** markvoelker has joined #openstack-infra | 17:03 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/system-config: Make hiera default values optional https://review.openstack.org/186568 | 17:03 |
| *** erlon has quit IRC | 17:03 | |
| fungi | and pick names for them | 17:03 |
| clarkb | sdague: with enable_plugin it takes a branch/ref or defaults to master. Does that mena all of our tests that enable_plugin are defaulting to master and are not testing the zuul_ref? | 17:04 |
| fungi | pleia2: a trove instance will be created corresponding to the database you ask to have created | 17:04 |
| pleia2 | fungi: right, so I added a regular user to the grafana database | 17:04 |
| fungi | that should be sufficient | 17:04 |
| *** tiswanso has quit IRC | 17:04 | |
| *** skylerberg has joined #openstack-infra | 17:04 | |
| *** olaph has quit IRC | 17:04 | |
| pleia2 | fungi: grafana also wants a root user | 17:04 |
| sdague | clarkb: no, because it goes through the same git_clone code in devstack, which means in a zuul environment, zuul creates all the refs, and it explodes if it actually tries to manipulate a git tree in devstack | 17:05 |
| *** olaph has joined #openstack-infra | 17:05 | |
| pleia2 | https://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n120 | 17:05 |
| sdague | it works for the same reason it works on all other git trees in devstack | 17:05 |
| fungi | pleia2: can you tell grafana that the root user is that same account? | 17:05 |
| *** tiswanso has joined #openstack-infra | 17:05 | |
| pleia2 | pabelanger: ^^ | 17:05 |
| pleia2 | fungi: it would need a change to the puppet config at least | 17:06 |
| *** Guest80 has quit IRC | 17:06 | |
| fungi | pleia2: grafana shouldn't need to do anything as "root" because the account you created for it has full control of the database | 17:06 |
| clarkb | sdague: gotcha | 17:06 |
| pabelanger | pleia2: fungi: we use puppet to provision a mysql::server | 17:06 |
| fungi | pleia2: the only things it can't do with that account are create additional accounts, create additional databases, drop the database you created, change mysql configuration settings... | 17:06 |
| pabelanger | so, in this case we'd have to drop it | 17:06 |
| pabelanger | to change it to be unmanaged | 17:06 |
| fungi | pabelanger: right, i think in other modules we've made local database management/provisioning optional with a toggle | 17:07 |
| pleia2 | ah, I had trove on the brain and missed that we were actually using local mysql here | 17:07 |
| fungi | or abstracted it into something like a grafana::mysql class which isn't directly included but which you can compose with the base ::grafana class | 17:07 |
| clarkb | sdague: I am running your plugin change locally now | 17:08 |
| pabelanger | pleia2: okay, so in this case we don't want a mysql server? | 17:08 |
| fungi | pabelanger: well, if we use trove, that allows us to spin up a replacement grafana server without having to dump/import the database | 17:08 |
| pabelanger | fungi: Ya, I am fine with that. I just expect mysql to be provisioned too | 17:09 |
| pabelanger | I can make the change | 17:09 |
| pleia2 | ok | 17:09 |
| pelix | zaro: just haven't had time to put together the tests needed for testing the recursive behaviour of get_jobs() | 17:09 |
| *** chlong has quit IRC | 17:09 | |
| *** achanda has joined #openstack-infra | 17:09 | |
| *** dalgaaf has quit IRC | 17:09 | |
| *** annegentle is now known as theannegentle | 17:09 | |
| pleia2 | and I can remove grafana_mysql_root_password from hiera since we won't need it | 17:10 |
| *** jsavak has quit IRC | 17:11 | |
| jesusaurus | jeblair: can you revisit your -1 on 186621 now that clarkb has weighed in? | 17:11 |
| *** mjturek1 has joined #openstack-infra | 17:11 | |
| *** mjturek1 has left #openstack-infra | 17:11 | |
| *** jsavak has joined #openstack-infra | 17:12 | |
| *** glauco has quit IRC | 17:12 | |
| *** Sukhdev_ has joined #openstack-infra | 17:12 | |
| *** cdent_ has joined #openstack-infra | 17:13 | |
| *** cdent has quit IRC | 17:14 | |
| *** cdent_ is now known as cdent | 17:14 | |
| clarkb | sdague: think I found a bug in devstack where it assumes the requirements repo is present to use constraints when installing prettytable | 17:14 |
| clarkb | sdague: I will work on a fix | 17:14 |
| openstackgerrit | Tim Hinrichs proposed openstack-infra/irc-meetings: Change Congress meeting day/time https://review.openstack.org/207981 | 17:14 |
| pabelanger | fungi: pleia2: which existing modules in system-config are using trove? | 17:15 |
| *** rguillebert has quit IRC | 17:15 | |
| pleia2 | pabelanger: review, etherpad, paste, translate-dev, storyboard, groups, wiki | 17:16 |
| *** bhuvan has quit IRC | 17:16 | |
| fungi | openstackid | 17:16 |
| pabelanger | thanks | 17:16 |
| fungi | possibly others | 17:17 |
| *** hdd has joined #openstack-infra | 17:19 | |
| *** theannegentle has quit IRC | 17:19 | |
| *** theannegentle has joined #openstack-infra | 17:20 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 17:20 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 17:22 | |
| pabelanger | pleia2: paste cannot be right. We still use root_password for mysql | 17:23 |
| *** mpaolino has quit IRC | 17:23 | |
| *** hdd has quit IRC | 17:23 | |
| *** _nadya_ has joined #openstack-infra | 17:23 | |
| *** fawadkhaliq has quit IRC | 17:23 | |
| clarkb | paste is local db now iirc | 17:23 |
| pabelanger | same with wiki | 17:23 |
| *** fawadkhaliq has joined #openstack-infra | 17:23 | |
| clarkb | it had toruble iwht the remote db disconnecting | 17:23 |
| pleia2 | ah, I just saw it in the list of trove instances we're running | 17:24 |
| *** pal has quit IRC | 17:24 | |
| *** ivar-lazzaro has quit IRC | 17:24 | |
| fungi | right, those have gone back and forth | 17:24 |
| *** signed8b_ has joined #openstack-infra | 17:25 | |
| *** salv-orlando has quit IRC | 17:25 | |
| *** jasondotstar has joined #openstack-infra | 17:25 | |
| fungi | paste could go back to trove again i think, now that i'm pretty sure we've worked out what was causing it to constantly break | 17:25 |
| zaro | clarkb: was jenkins wonky last week sometime? | 17:25 |
| clarkb | zaro: it is still wonky, the slave thread leak is affecting us | 17:26 |
| *** markvoelker has quit IRC | 17:26 | |
| fungi | i moved paste off trove before i realized that rackspace defaults the server to timing out idle connections after 2 minutes (as opposed to mysqld's 8-hour default) | 17:26 |
| zaro | clarkb: so slave threads don't go away after disconnect? | 17:26 |
| fungi | so now we have a custom trove config to set that to a more sane value | 17:27 |
| *** vponomaryov has quit IRC | 17:27 | |
| rcarrillocruz | folks | 17:27 |
| rcarrillocruz | question | 17:27 |
| rcarrillocruz | how does the diskimage-builder elements folder keep up to date with glean? | 17:28 |
| *** signed8bit has quit IRC | 17:28 | |
| rcarrillocruz | i mean | 17:28 |
| rcarrillocruz | there's openstack-infra/glean | 17:28 |
| rcarrillocruz | and openstack/diskimage-builder, which contains as an element glean | 17:28 |
| jeblair | jesusaurus: aprvd | 17:28 |
| rcarrillocruz | i checked but it's not a git submodule within diskimage-builder | 17:28 |
| clarkb | zaro: correct and there seems to be a shared lock around that operation so all of the slave threads pile up behind it | 17:29 |
| *** ivar-lazzaro has joined #openstack-infra | 17:29 | |
| clarkb | zaro: and we end up with 12k threads and a dead jenkins server | 17:29 |
| pabelanger | pleia2: okay, so you are going to setup the trove DB and give the grafana user correct permissions right? | 17:29 |
| clarkb | rcarrillocruz: it installs latest glean from pypi | 17:29 |
| rcarrillocruz | huh | 17:30 |
| rcarrillocruz | o-k | 17:30 |
| pabelanger | pleia2: that should allow me to remove all the mysql::server logic | 17:30 |
| rcarrillocruz | so that means | 17:30 |
| rcarrillocruz | that in order to get latest fixes of glean as an element | 17:30 |
| pleia2 | pabelanger: yes, a user with access to the grafana database | 17:30 |
| rcarrillocruz | a new release must be cut for glean | 17:30 |
| clarkb | rcarrillocruz: yes and I believe a new release was cut a day or two ago with all oustanding changes | 17:30 |
| pleia2 | pabelanger: you don't need to remove it, it's nice to have in the module if other people want it, just don't include it in our instance of the module | 17:30 |
| pabelanger | pleia2: okay, user will need 'all' access. removing code now | 17:30 |
| fungi | rcarrillocruz: http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/simple-init | 17:31 |
| pleia2 | er, our configuration | 17:31 |
| rcarrillocruz | i'm hitting the /root/.ssh folder missing issue, it fails on ubuntu-minimal as that folder is not created by default | 17:31 |
| SpamapS | interesting.. | 17:31 |
| rcarrillocruz | i wonder if it got thru | 17:31 |
| SpamapS | 1TB is not actually the magic size for grub to fail. | 17:31 |
| SpamapS | it must be geometry related | 17:31 |
| fungi | rcarrillocruz: http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/simple-init/install.d/simple-init-source-install/40-glean | 17:31 |
| SpamapS | 1100GB works fine | 17:31 |
| pabelanger | pleia2: well, it looks like I will need to create a puppet-grafana modules moving forward, to toggle that logic now. | 17:31 |
| SpamapS | 1800GB does not | 17:31 |
| fungi | rcarrillocruz: it needs a new release on pypi | 17:31 |
| rcarrillocruz | yep | 17:31 |
| rcarrillocruz | mordred: ^ | 17:31 |
| pleia2 | pabelanger: yeah, likely so | 17:31 |
| rcarrillocruz | TheJulia: ^ | 17:32 |
| pabelanger | since this was a simple local mysql, I just kept it in system-config | 17:32 |
| pleia2 | pabelanger: it just needs all on that one database, right? | 17:32 |
| pabelanger | pleia2: right | 17:32 |
| pleia2 | pabelanger: ok perfect | 17:32 |
| *** _nadya_ has quit IRC | 17:32 | |
| fungi | rcarrillocruz: https://git.openstack.org/cgit/openstack-infra/glean/ shows the /root/.ssh fix landed after 1.4.0 was tagged (most recent release) | 17:32 |
| rcarrillocruz | ++ | 17:32 |
| clarkb | ah there is a newer fix | 17:32 |
| rcarrillocruz | yep :/ | 17:33 |
| *** e0ne has quit IRC | 17:33 | |
| SpamapS | IIRC mordred is in UTC+10 right now | 17:33 |
| fungi | looks like it could stand to be 1.4.1 | 17:33 |
| SpamapS | so, likely awake soon | 17:33 |
| SpamapS | 3 hours or so | 17:33 |
| fungi | i would consider that patch a simple bug fix, from a semver perspective | 17:33 |
| *** e0ne has joined #openstack-infra | 17:33 | |
| *** xyang0 has quit IRC | 17:33 | |
| jesusaurus | jeblair: awesome, thanks | 17:34 |
| rcarrillocruz | i'm not familiar with semver, you mean simple bugfix => 1.4.1 instead of 1.5 right? | 17:34 |
| SpamapS | fungi: concurr, 1.4.1 | 17:34 |
| fungi | and yes, it's nearing the coffee hour for mordred | 17:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Remove local grafana database https://review.openstack.org/208002 | 17:34 |
| fungi | in his current waveform-collapsed observed state | 17:34 |
| pabelanger | pleia2: fungi: ^ that should be what you are looking for | 17:34 |
| fungi | rcarrillocruz: yep, if it included a new feature then 1.5.0 and if it included a backward-incompatible/breaking change then 2.0.0 | 17:35 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Add newlines to the ends of files https://review.openstack.org/207856 | 17:35 |
| rcarrillocruz | nod | 17:35 |
| fungi | anyway, since he'll likely be awake soonish, i'd leave it to mordred to tag because he's tagged at least the last few on that repo. but if it's extremely urgent then i or someone else can tag it sooner | 17:36 |
| *** shashank_hegde has joined #openstack-infra | 17:37 | |
| rcarrillocruz | providing it's tagged today i'm ok | 17:37 |
| rcarrillocruz | i can bug him later | 17:37 |
| rcarrillocruz | have my deploys broken due to that, but they are not in prod so it's not the end of the world whatsoever | 17:37 |
| *** amitgandhinz has joined #openstack-infra | 17:39 | |
| *** sdake has quit IRC | 17:39 | |
| pabelanger | EmilienM: Ya, we need dsvm-centos7 in the name of the job to keep zuul happy | 17:40 |
| *** abregman has quit IRC | 17:40 | |
| *** sdake has joined #openstack-infra | 17:40 | |
| pleia2 | pabelanger: great, thanks | 17:43 |
| *** glauco_ has joined #openstack-infra | 17:43 | |
| *** kushal has quit IRC | 17:43 | |
| *** crc32 has quit IRC | 17:44 | |
| pabelanger | np | 17:44 |
| *** sdake has quit IRC | 17:44 | |
| *** sdake has joined #openstack-infra | 17:45 | |
| *** theannegentle has quit IRC | 17:45 | |
| *** glauco__ has joined #openstack-infra | 17:46 | |
| *** baoli has joined #openstack-infra | 17:46 | |
| *** glauco_ has quit IRC | 17:46 | |
| *** olaph has quit IRC | 17:46 | |
| *** crc32 has joined #openstack-infra | 17:52 | |
| *** crc32 has quit IRC | 17:54 | |
| *** crc32 has joined #openstack-infra | 17:54 | |
| *** bswartz has quit IRC | 17:55 | |
| *** glauco__ has quit IRC | 17:55 | |
| *** SumitNaiksatam has quit IRC | 17:55 | |
| *** glauco_ has joined #openstack-infra | 17:55 | |
| *** Hal has joined #openstack-infra | 17:55 | |
| *** mriedem has joined #openstack-infra | 17:56 | |
| *** crc32 has quit IRC | 17:56 | |
| *** Hal is now known as Guest2926 | 17:56 | |
| *** kushal has joined #openstack-infra | 17:59 | |
| clarkb | wow you can't base compute in 2GB | 18:01 |
| clarkb | * 2GB of ram | 18:01 |
| *** bswartz has joined #openstack-infra | 18:01 | |
| *** ashleighfarnham has quit IRC | 18:01 | |
| *** crc32 has joined #openstack-infra | 18:02 | |
| *** ashleighfarnham has joined #openstack-infra | 18:02 | |
| *** yamamoto has quit IRC | 18:02 | |
| *** jyuso1 has quit IRC | 18:02 | |
| *** jyuso1 has joined #openstack-infra | 18:03 | |
| *** jlanoux has quit IRC | 18:06 | |
| *** ivar-lazzaro has quit IRC | 18:07 | |
| *** patrickeast has joined #openstack-infra | 18:07 | |
| *** jyuso1 has quit IRC | 18:08 | |
| jeblair | clarkb: i'm having trouble parsing that. i'm not sure if it's a drug reference. | 18:08 |
| EmilienM | pabelanger: it's weird a bit | 18:09 |
| clarkb | jeblair: I am trying to test the nodepool devstack plugin and naively did so on a 2GB VM. I run out of memory with mysql + nova + neutron + glance + swift + keystone | 18:09 |
| clarkb | I remember back in the day when 1GB was plenty if you just wanted to run a cirros VM | 18:09 |
| mordred | fungi: what did I do? | 18:10 |
| fungi | mordred: nothing, rcarrillocruz is just asking for a glean 1.4.1 for the /root/.ssh fix | 18:10 |
| fungi | and since you did the prior glean releases, i thought i'd save that for you | 18:10 |
| rcarrillocruz | yeah, turns out that fix came just a bit after the the latest release was cut | 18:10 |
| *** Guest2926 has quit IRC | 18:11 | |
| rcarrillocruz | hitting that on my bifrost/ironic deploys | 18:11 |
| EmilienM | pabelanger: can't we just change the trusty | precise thing? | 18:11 |
| *** theannegentle has joined #openstack-infra | 18:11 | |
| mordred | rcarrillocruz, fungi: done | 18:11 |
| *** jsavak has quit IRC | 18:11 | |
| rcarrillocruz | thx a bunch! | 18:11 |
| rcarrillocruz | TheJulia: ^ | 18:11 |
| fungi | EmilienM: the tricky part is not in the job configuration but in the zuul configuration, specifically at http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/openstack_functions.py#n57 | 18:12 |
| *** jsavak has joined #openstack-infra | 18:13 | |
| *** johnbelamaric has joined #openstack-infra | 18:13 | |
| EmilienM | fungi: ok | 18:13 |
| fungi | EmilienM: so it's matching regular expressions against the job name to figure out where to run things since we still have some jobs that run on different node types conditional on what branch is being tested | 18:14 |
| TheJulia | woot | 18:14 |
| EmilienM | fungi: +1 | 18:14 |
| rcarrillocruz | jeblair: when you get a change, i responded to your comments on https://review.openstack.org/#/c/141016/ , chatted with mordred about it last week iirc | 18:14 |
| EmilienM | fungi: https://review.openstack.org/#/c/207964 | 18:14 |
| rcarrillocruz | i'm out for the weekend now | 18:14 |
| rcarrillocruz | have a good one folks! | 18:14 |
| pleia2 | have a nice weekend, rcarrillocruz | 18:15 |
| *** Sukhdev_ has quit IRC | 18:15 | |
| *** bknudson has quit IRC | 18:15 | |
| *** tonytan4ever has joined #openstack-infra | 18:16 | |
| fungi | EmilienM: yeah, lgtm | 18:16 |
| *** baoli has quit IRC | 18:16 | |
| pabelanger | EmilienM: -infra wants to remove the bare-* nodes in the future. So, the patch is just preparing for that | 18:16 |
| EmilienM | pabelanger: this is super cool | 18:17 |
| EmilienM | pabelanger: I did not have all the bits | 18:17 |
| fungi | pabelanger: though fwiw we want to remove the devstack-* nodes too, and have more generic node names ;) | 18:17 |
| clarkb | jeblair: I am somewhat amazed we are able to run tempest on 8GB nodes iwth all the additional services | 18:17 |
| *** arxcruz has quit IRC | 18:17 | |
| pabelanger | fungi: true | 18:17 |
| *** johnbelamaric has quit IRC | 18:17 | |
| pabelanger | looking forward to zuulv3 stuff | 18:18 |
| fungi | hence ubuntu-trusty instead of bare-trusty/devstack-trusty and centos-7 instead of bare-centos7/devstack-centos7 | 18:18 |
| *** baoli has joined #openstack-infra | 18:18 | |
| *** HeOS has quit IRC | 18:18 | |
| *** jcoufal has joined #openstack-infra | 18:18 | |
| *** unicell has quit IRC | 18:19 | |
| clarkb | greghaynes: / being 13GB for ubuntu-trusty on rax is the resize problem? | 18:19 |
| greghaynes | clarkb: yes | 18:19 |
| greghaynes | clarkb: I suspect its the broken cloud-initramfs-growroot package | 18:19 |
| *** ildikov has quit IRC | 18:19 | |
| *** banix has joined #openstack-infra | 18:19 | |
| clarkb | greghaynes: so we are just waiting for a package fix? | 18:20 |
| pabelanger | fungi: if you have some time, would like your feedback on https://review.openstack.org/#/c/203756/ and the following patch | 18:20 |
| *** bknudson has joined #openstack-infra | 18:21 | |
| *** ivar-lazzaro has joined #openstack-infra | 18:21 | |
| greghaynes | clarkb: that needs to be confirmed as the cause, I know that we are hitting it on debian but havent confirmed for ubuntu. Also would be fixed by https://review.openstack.org/#/c/206636/ if the package is actually the cause | 18:22 |
| clarkb | we hit it on ubuntu-trusty | 18:23 |
| greghaynes | clarkb: you checked initrd logs? | 18:23 |
| clarkb | no, I just checked that / is 13GB and should be 30GB iirc | 18:23 |
| greghaynes | I know it wasnt resizing, but I need to boot a node and confirm its growroot barfing | 18:23 |
| greghaynes | or, someone does :) | 18:24 |
| *** banix has quit IRC | 18:24 | |
| *** marzif_ has quit IRC | 18:25 | |
| *** pipercolt is now known as hemna | 18:26 | |
| pabelanger | pleia2: do you mind reviewing https://review.openstack.org/#/c/207964/1 too? | 18:27 |
| *** baoli has quit IRC | 18:27 | |
| *** salv-orlando has joined #openstack-infra | 18:27 | |
| *** markvoelker has joined #openstack-infra | 18:28 | |
| *** baoli has joined #openstack-infra | 18:28 | |
| *** Yichen has joined #openstack-infra | 18:28 | |
| *** bpokorny_ has joined #openstack-infra | 18:29 | |
| clarkb | greghaynes: where should I be looking? I don't see anything in boot.log | 18:31 |
| greghaynes | clarkb: prints out on the console, I only know because I saw qemu spit it out | 18:31 |
| *** bpokorny has quit IRC | 18:31 | |
| *** baoli has quit IRC | 18:31 | |
| *** baoli has joined #openstack-infra | 18:33 | |
| *** baoli has quit IRC | 18:34 | |
| *** glauco_ has quit IRC | 18:35 | |
| *** baoli has joined #openstack-infra | 18:35 | |
| clarkb | greghaynes: ah, thats harder to check so I probably wont and will go back to findning and 8GB test node to test this thing | 18:35 |
| clarkb | greghaynes: you can test locally right? | 18:35 |
| greghaynes | clarkb: haha | 18:35 |
| *** jcoufal has quit IRC | 18:35 | |
| greghaynes | clarkb: ill just boot up in a cloud and look at the console log | 18:35 |
| greghaynes | clarkb: that isnt something I want to do on vacation though ;) | 18:35 |
| *** salv-orlando has quit IRC | 18:36 | |
| *** ildikov has joined #openstack-infra | 18:36 | |
| clarkb | are you sure? fixing grow root sounds like fun | 18:36 |
| greghaynes | clarkb: oh, im writing the script | 18:36 |
| greghaynes | just not making an image to boot in a cloud to look at a console | 18:36 |
| greghaynes | because that sounds tedious | 18:36 |
| clarkb | fungi: jeblair I am just going to boot a devstack-trusty node in one of the cloud regions and will lcean it up when done | 18:37 |
| *** patrickeast has quit IRC | 18:37 | |
| *** e0ne has quit IRC | 18:37 | |
| greghaynes | ubuntu-trusty? | 18:37 |
| *** markvoelker has quit IRC | 18:37 | |
| fungi | clarkb: k | 18:37 |
| *** jsavak has quit IRC | 18:37 | |
| pleia2 | pabelanger: aprvd | 18:37 |
| clarkb | greghaynes: no because that dones't give me enough disk | 18:38 |
| clarkb | greghaynes: I am trying to test nodepool devstack plugin | 18:38 |
| greghaynes | oh, unrelated problem | 18:38 |
| clarkb | and was chased away from ubuntu0trusty by ENODISK | 18:38 |
| greghaynes | yea | 18:38 |
| clarkb | which I could fix by hand but meh | 18:38 |
| greghaynes | clarkb: oh, if you actually booted it | 18:38 |
| greghaynes | clarkb: just run growpart | 18:38 |
| *** davideagnello has quit IRC | 18:38 | |
| *** bswartz has quit IRC | 18:38 | |
| greghaynes | growpart /dev/vda 1 or w/e your / | 18:39 |
| pabelanger | pleia2: danke | 18:39 |
| *** fawadkhaliq has quit IRC | 18:40 | |
| *** banix has joined #openstack-infra | 18:41 | |
| *** davideagnello has joined #openstack-infra | 18:41 | |
| *** sdake has quit IRC | 18:42 | |
| *** banix has quit IRC | 18:44 | |
| *** ashleighfarnham has quit IRC | 18:44 | |
| *** sdake has joined #openstack-infra | 18:44 | |
| *** banix has joined #openstack-infra | 18:45 | |
| *** e0ne has joined #openstack-infra | 18:45 | |
| *** Sukhdev_ has joined #openstack-infra | 18:45 | |
| *** davideagnello has quit IRC | 18:47 | |
| *** e0ne has quit IRC | 18:47 | |
| *** davideagnello has joined #openstack-infra | 18:48 | |
| *** MarkAtwood has quit IRC | 18:49 | |
| *** glauco_ has joined #openstack-infra | 18:49 | |
| pleia2 | no meetings going on for the rest of the day (2 were scheduled for 1800, but no activity), if someone wants to approve this one we can finish the move to #openstack-i18n I began this morning and get them logging https://review.openstack.org/#/c/205980/ | 18:50 |
| pleia2 | for the peanut gallery: redirect from their old channel is in place, so they're pretty much moved now once they channel cycle | 18:51 |
| *** jsavak has joined #openstack-infra | 18:52 | |
| *** mrmartin has joined #openstack-infra | 18:53 | |
| *** bswartz has joined #openstack-infra | 18:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Move puppet-syntax gate to devstack-centos7 https://review.openstack.org/207964 | 18:55 |
| *** ddieterly has joined #openstack-infra | 18:56 | |
| *** hemna is now known as hemnafk | 18:56 | |
| *** jsavak has quit IRC | 18:57 | |
| *** ddieterly has quit IRC | 18:57 | |
| *** ddieterly has joined #openstack-infra | 18:58 | |
| *** jsavak has joined #openstack-infra | 18:59 | |
| *** olaph has joined #openstack-infra | 19:01 | |
| *** glauco_ has quit IRC | 19:01 | |
| *** salv-orlando has joined #openstack-infra | 19:01 | |
| *** yamamoto has joined #openstack-infra | 19:03 | |
| openstackgerrit | Merged openstack-dev/pbr: Fix docs for environment markers in extras https://review.openstack.org/207600 | 19:05 |
| *** fawadkhaliq has joined #openstack-infra | 19:05 | |
| *** lhcheng_ has joined #openstack-infra | 19:06 | |
| *** lhcheng_ has left #openstack-infra | 19:06 | |
| *** glauco has joined #openstack-infra | 19:07 | |
| *** yamamoto has quit IRC | 19:08 | |
| *** baoli has quit IRC | 19:08 | |
| *** glauco has quit IRC | 19:08 | |
| *** fawadkhaliq has quit IRC | 19:09 | |
| *** baoli has joined #openstack-infra | 19:10 | |
| jeblair | aww, what happened, did people come in to work? | 19:11 |
| fungi | heh | 19:12 |
| *** jasondotstar has quit IRC | 19:12 | |
| fungi | at the moment all my gerritbot spam is hitting #openstack-nova instead | 19:12 |
| fungi | though i'm preparing to knock off early and schlep more junk between houses once i see whether my latest bindep experiment is working out | 19:13 |
| *** tjones has quit IRC | 19:14 | |
| *** mrmartin has quit IRC | 19:15 | |
| *** shashank_hegde has quit IRC | 19:15 | |
| *** patrickeast has joined #openstack-infra | 19:16 | |
| *** rossella_s has quit IRC | 19:16 | |
| *** rossella_s has joined #openstack-infra | 19:16 | |
| clarkb | sdague: ok I have it all running and it is attempting to build an image now, will let yo uknow how that goes | 19:18 |
| sdague | clarkb: cool, did you need fixes? | 19:18 |
| openstackgerrit | Merged openstack-infra/infra-specs: Add Zuul v3 spec. https://review.openstack.org/164371 | 19:18 |
| clarkb | sdague: there is one devstack fix I need to push up, will do that now. Not sure if nodepool plugin needs changes until we see how this image build does | 19:18 |
| sdague | clarkb: ok cool | 19:18 |
| *** afazekas has joined #openstack-infra | 19:19 | |
| clarkb | sdague: but it is attempting to build the image so thats major progress | 19:19 |
| *** sergmelikyan has quit IRC | 19:19 | |
| *** tonytan4ever has quit IRC | 19:19 | |
| *** jasondotstar has joined #openstack-infra | 19:19 | |
| *** _nadya_ has joined #openstack-infra | 19:20 | |
| *** ifarkas has quit IRC | 19:21 | |
| *** banix has quit IRC | 19:21 | |
| *** mrmartin has joined #openstack-infra | 19:22 | |
| *** banix has joined #openstack-infra | 19:22 | |
| sdague | nice | 19:22 |
| clarkb | sdague: image built now it is attempting to boot a slave off of it | 19:24 |
| *** jasondotstar has quit IRC | 19:25 | |
| clarkb | if that works we can merge your change then add a test that runs devstack, sleep 15 minutes then checks the nodepool state | 19:25 |
| openstackgerrit | Merged openstack-infra/system-config: Rename -translation IRC channel and enable logging https://review.openstack.org/205980 | 19:25 |
| *** baoli has quit IRC | 19:25 | |
| *** rlandy has joined #openstack-infra | 19:26 | |
| *** ig0r_ has quit IRC | 19:26 | |
| clarkb | nibalizer: ^ btw dsague was kind enough to write a nodepool plugin given my doc on how to do it manually | 19:26 |
| clarkb | nibalizer: so in theory you can just use that if you are dvestacking | 19:26 |
| *** afazekas has quit IRC | 19:26 | |
| *** baoli has joined #openstack-infra | 19:27 | |
| nibalizer | clarkb: thats neat | 19:27 |
| nibalizer | is that a devstack plugin for nodepool or a nodepool plugin for devstack? | 19:27 |
| fungi | devstack plugin for nodepool, so that we can integration-test a running nodepoold | 19:28 |
| clarkb | nibalizer: its a devstack plugin that installs configures and runs nodepool | 19:28 |
| pleia2 | pabelanger: going to lunch, we can keep an eye on 208002 passing checks so hopefully we can get it in today so we can get grafana done, hopefully the hard parts are done, trove is set up and everything is in hiera, just need to deploy with this database change | 19:28 |
| nibalizer | oh so you get an openstack and a nodepool to poke at the same time | 19:28 |
| nibalizer | neat | 19:28 |
| pabelanger | pleia2: ack | 19:28 |
| clarkb | nibalizer: yes | 19:28 |
| nibalizer | ya that would be realyl useful when I start hacking on beaker-nodepol | 19:28 |
| nibalizer | wow I am a bad typist | 19:28 |
| * nibalizer => run away now | 19:29 | |
| clarkb | nibalizer: well mostly I want you to see that yo ucan just run a nodepool without a zuul and without a jenkins | 19:29 |
| *** ashleighfarnham has joined #openstack-infra | 19:29 | |
| clarkb | nibalizer: and it juts works beause see devstack plugin :) | 19:29 |
| clarkb | sdague: yup node booted | 19:29 |
| * fungi tells nodepool to hurry up and build more ubuntu-trusty already | 19:29 | |
| *** aysyd has quit IRC | 19:30 | |
| clarkb | nibalizer: and very soon now we will test that it continues to do that | 19:30 |
| nibalizer | clarkb: oh i believe you, I use it myself | 19:31 |
| *** banix has quit IRC | 19:31 | |
| nibalizer | no wait Im not actually here | 19:31 |
| *** nibalizer has left #openstack-infra | 19:31 | |
| *** tonytan4ever has joined #openstack-infra | 19:31 | |
| clarkb | sdague: this is awesome, just confirming that using config drive works too (as that appaers to be our default devstack setup) and if it does will suggest one tiny update to the config for the nodepool service | 19:32 |
| fungi | wow, he's right... he's _not_ actually here | 19:32 |
| openstackgerrit | Merged openstack-dev/cookiecutter: Stop specifying requirements.txt in tox.ini https://review.openstack.org/202721 | 19:32 |
| *** e0ne has joined #openstack-infra | 19:32 | |
| *** openstack has joined #openstack-infra | 19:35 | |
| *** jsavak has quit IRC | 19:36 | |
| *** glauco has quit IRC | 19:37 | |
| mtreinish | fungi: ooh I like that unit, we should standarize it as 1 fungi = 1 kjph :) | 19:37 |
| fungi | kjph is 20% less typing | 19:37 |
| *** sdake has quit IRC | 19:38 | |
| *** baoli has quit IRC | 19:38 | |
| *** sergmelikyan has quit IRC | 19:38 | |
| mtreinish | heh, then 1 fung = 1kjph | 19:38 |
| fungi | wfm | 19:38 |
| *** baoli has joined #openstack-infra | 19:38 | |
| mtreinish | or 1 ungi :) | 19:38 |
| *** abregman is now known as abregman|brb | 19:39 | |
| mtreinish | whichever you prefer | 19:39 |
| cody-somerville | fungi: Do you have this charted? | 19:39 |
| *** sdake has joined #openstack-infra | 19:39 | |
| fungi | cody-somerville: at the bottom of http://status.openstack.org/zuul/ | 19:39 |
| fungi | though there's an unsightly gap from when graphite fell over earlier today | 19:40 |
| *** davideagnello has quit IRC | 19:40 | |
| *** xyang0 has joined #openstack-infra | 19:40 | |
| fungi | (just mind the gap) | 19:41 |
| *** jsavak has joined #openstack-infra | 19:41 | |
| *** puranamr has quit IRC | 19:42 | |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Add a JJB quick start guide https://review.openstack.org/208037 | 19:42 |
| *** glauco has joined #openstack-infra | 19:43 | |
| clarkb | sdague: commented about the one update. I can push a new patchset if that is easier for you but this seems to work. I am writing a job now | 19:45 |
| *** mrmartin has quit IRC | 19:46 | |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Improve delete all job performance https://review.openstack.org/199193 | 19:46 |
| *** rm_work|away is now known as rm_work | 19:49 | |
| *** rlandy has quit IRC | 19:51 | |
| *** abregman|brb has quit IRC | 19:51 | |
| zaro | clarkb: do you think the gearman-plugin issue could be due to a WORK_FAIL event?? https://review.openstack.org/#/c/192429/ | 19:52 |
| *** glauco has quit IRC | 19:52 | |
| *** sergmelikyan has joined #openstack-infra | 19:53 | |
| *** rlandy has joined #openstack-infra | 19:55 | |
| EmilienM | pabelanger: re- https://review.openstack.org/#/c/207964/ - BTW, why not also lint & unit ? | 19:58 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run a nodepool devstack test https://review.openstack.org/208043 | 19:59 |
| *** glauco_ has joined #openstack-infra | 19:59 | |
| clarkb | zaro: what gearman-plugin issue? | 19:59 |
| clarkb | sdague: fungi ^ | 19:59 |
| zaro | pelix: meh, not a big deal but probably should remove it. i noticed that send-as is not in any of the hipchat tests so maybe just remove it in your change and add it to one of the existing tests that target plugin version 0.1.8 ? | 20:00 |
| *** glauco_ has quit IRC | 20:01 | |
| zaro | clarkb: I thought the jenkins being wonky was a gearman-plugin issue, no? | 20:01 |
| *** sdake_ has joined #openstack-infra | 20:01 | |
| clarkb | zaro: no it sa jenkins ssh plugin issue | 20:02 |
| *** nibalizer has joined #openstack-infra | 20:02 | |
| clarkb | zaro: the plugin is holding a shared lock and backing up slave threads | 20:02 |
| zaro | clarkb: ohh, sorry i just came back from vaca. | 20:02 |
| clarkb | zaro: this is the same issue from before | 20:02 |
| *** shashank_hegde has joined #openstack-infra | 20:03 | |
| zaro | ahh, i remember there was a bug for it? any progress? | 20:03 |
| pelix | zaro: will do | 20:04 |
| clarkb | there is a bug upsream I haven't seen any movement after I provided the stack dumps | 20:04 |
| *** abregman|brb has joined #openstack-infra | 20:04 | |
| *** alivigni has quit IRC | 20:05 | |
| *** sdake has quit IRC | 20:05 | |
| jeblair | would anyone be terribly upset if i restarted zuul? | 20:06 |
| fungi | not i | 20:06 |
| pelix | zuul might be :p | 20:06 |
| jeblair | heh | 20:06 |
| jeblair | actually, i want to see I7c255937f08aef28bdad72e8f523ca34f2c6262c in action | 20:06 |
| jeblair | "set build descriptions less often" | 20:06 |
| jeblair | it might appreciate having less work | 20:06 |
| *** vnod has joined #openstack-infra | 20:07 | |
| *** abregman|brb is now known as abregman | 20:08 | |
| *** tiswanso has quit IRC | 20:08 | |
| *** cburroughs has quit IRC | 20:09 | |
| *** vnod has quit IRC | 20:11 | |
| *** hamzy has quit IRC | 20:11 | |
| *** cdent has quit IRC | 20:11 | |
| *** vnod has joined #openstack-infra | 20:12 | |
| openstackgerrit | Pete Zaitcev proposed openstack-infra/git-review: Update test requirements for Python 2.x https://review.openstack.org/200860 | 20:13 |
| *** baoli has quit IRC | 20:13 | |
| *** baoli has joined #openstack-infra | 20:14 | |
| openstackgerrit | Sean Dague proposed openstack-infra/nodepool: nodepool devstack plugin https://review.openstack.org/207095 | 20:14 |
| *** sergmelikyan has quit IRC | 20:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Log gerrit an INFO level https://review.openstack.org/208049 | 20:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul: Log gerrit at INFO level https://review.openstack.org/208049 | 20:15 |
| *** abregman has quit IRC | 20:15 | |
| mtreinish | sdague: ^^^ oh cool so that'll enable running nodepool on a local devstack env | 20:15 |
| zaro | waynr: anything i can do to get a +2 on https://review.openstack.org/#/c/151472 ? | 20:15 |
| sdague | mtreinish: yes | 20:15 |
| *** alivigni has joined #openstack-infra | 20:16 | |
| *** davideagnello has joined #openstack-infra | 20:17 | |
| mtreinish | nice, that makes it simple to play around with it. Last time I had to play with nodepool stuff I just submitted blind | 20:17 |
| clarkb | mtreinish: yup and 208043 will test that nodepool changes work with devstack | 20:18 |
| clarkb | shoul dmake nodepool dev a bit easier (which is a good thing with zuulv3 happening) | 20:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Make sure puppet-syntax jobs have beaker installed https://review.openstack.org/208050 | 20:19 |
| jeblair | mtreinish: it has fakes for all of its external interactions, so if you aren't working on that part of it, you can run it locally very easily (just needs a mysql db, which it will create). but yeah, there hasn't been a good story for testing it with a cloud other than "point it at your hpcloud account" until now. so this is very cool. :) | 20:20 |
| pabelanger | EmilienM: clarkb: fungi Do you mind revewing ^ to fix the puppet-gate? | 20:21 |
| EmilienM | pabelanger: I did, thanks for the fix, hopefully it will unblock our CI | 20:21 |
| mtreinish | jeblair: oh, well I didn't know that either. I guess I could have tested things locally then :) | 20:21 |
| jeblair | it's good to cover the bases | 20:21 |
| *** cdent has joined #openstack-infra | 20:22 | |
| *** hamzy has joined #openstack-infra | 20:23 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Make sure puppet-syntax jobs have bundler installed https://review.openstack.org/208050 | 20:24 |
| *** angdraug has joined #openstack-infra | 20:24 | |
| *** ildikov has quit IRC | 20:24 | |
| pabelanger | EmilienM: ^ fixed typo | 20:26 |
| *** baoli has quit IRC | 20:28 | |
| *** kgiusti has left #openstack-infra | 20:28 | |
| *** e0ne has quit IRC | 20:30 | |
| jeblair | Exception: Dependency cycle detected: 187171 in [u'187171', u'185942'] | 20:30 |
| jeblair | ooh neat | 20:30 |
| *** tonytan4ever has quit IRC | 20:30 | |
| *** xyang0 has quit IRC | 20:31 | |
| jeblair | indeed, those 2 changes depend on each other | 20:31 |
| clarkb | oh crap bug in my make nodepool dsvm test go change | 20:31 |
| *** baoli has joined #openstack-infra | 20:31 | |
| jeblair | "oh crap" should be a bug severity | 20:32 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run a nodepool devstack test https://review.openstack.org/208043 | 20:32 |
| clarkb | jeblair: yes! | 20:32 |
| mtreinish | jeblair: heh, yeah I agree | 20:32 |
| clarkb | I think 208043 is ready togo now | 20:32 |
| *** _nadya_ has quit IRC | 20:33 | |
| *** _nadya_ has joined #openstack-infra | 20:33 | |
| clarkb | it is super simple now and just checks that nodepool made an image and booted a node after 15 minutes | 20:34 |
| openstackgerrit | David Lyle proposed openstack-infra/system-config: Add searchlight channel to statusbot list https://review.openstack.org/207519 | 20:34 |
| EmilienM | pabelanger: +1 - we might need fast approval on this one | 20:36 |
| *** vnod has quit IRC | 20:36 | |
| pabelanger | EmilienM: ya, that is my bad. I assume beaker was setup properly, missed it :( | 20:36 |
| *** xyang0 has joined #openstack-infra | 20:36 | |
| EmilienM | pabelanger: no prob, it happens :) | 20:37 |
| pabelanger | EmilienM: I actually plan to look at the puppet stuff closes next week, see how we can avoid duplicated code better | 20:37 |
| *** baoli has quit IRC | 20:37 | |
| EmilienM | pabelanger: cool | 20:37 |
| jeblair | i'm slightly concerned that zuul is wandering down a long road... it last started its queue processor >45 minutes ago | 20:37 |
| *** _nadya_ has quit IRC | 20:38 | |
| *** puranamr has joined #openstack-infra | 20:38 | |
| clarkb | jeblair: it was like this earlier today and got it down to 0 | 20:38 |
| pabelanger | jeblair: Ya, see that too | 20:38 |
| clarkb | but not sure how long that took | 20:38 |
| *** bapalm_ has quit IRC | 20:38 | |
| jeblair | yeah, i'm not panicking; i'm just going to look into whether it's doing unecessary work | 20:39 |
| *** baoli has joined #openstack-infra | 20:39 | |
| *** xyang0 has quit IRC | 20:39 | |
| openstackgerrit | Merged stackforge/python-jenkins: Add socket timeout tests and helpers https://review.openstack.org/199530 | 20:40 |
| fungi | yay! bindep/zuul-cloner variant of nova py27 unit tests on ubuntu-trusty worker with a minimal bindep file in the repo, running the same number of tests as its traditional gerrit-git-prep predecessor on bare-trusty | 20:41 |
| *** ildikov has joined #openstack-infra | 20:41 | |
| jeblair | 2015-07-31 19:46:19,716 DEBUG zuul.DependentPipelineManager: Starting queue processor: gate | 20:41 |
| jeblair | 2015-07-31 20:40:22,161 DEBUG zuul.DependentPipelineManager: Finished queue processor: gate (changed: True) | 20:41 |
| jeblair | it did finish :) | 20:41 |
| *** bpokorny has joined #openstack-infra | 20:41 | |
| pabelanger | jeblair: is that metric exposed to statsd, the queue length? | 20:41 |
| jeblair | it's just, that normally takes about 0.1 seconds | 20:41 |
| jeblair | pabelanger: i don't think so, but could be fairly easily | 20:42 |
| pabelanger | jeblair: Ya, working on some grafana.o.o dashboards. would be neat to add it | 20:42 |
| *** bswartz has quit IRC | 20:43 | |
| *** johnbelamaric has joined #openstack-infra | 20:43 | |
| *** vnod has joined #openstack-infra | 20:43 | |
| *** bpokorny_ has quit IRC | 20:45 | |
| clarkb | jeblair: oh was that time for a single event/rsult to be processed? | 20:45 |
| *** Sukhdev_ has quit IRC | 20:45 | |
| jeblair | clarkb: a single pass of the nnfi algorithm through the current gate pipeline | 20:45 |
| clarkb | wow | 20:46 |
| jeblair | i'm filtering out log lines from it. i'm down to 24961 lines. | 20:46 |
| *** vnod has quit IRC | 20:46 | |
| *** vnod has joined #openstack-infra | 20:46 | |
| *** sdake_ is now known as sdake | 20:46 | |
| *** e0ne has joined #openstack-infra | 20:48 | |
| *** doug-fish has left #openstack-infra | 20:48 | |
| pabelanger | jeblair: I'll play with zuul this weekend to add it. Any suggestions on statsd key name for the trigger_event_queue? | 20:49 |
| *** tjones has joined #openstack-infra | 20:50 | |
| *** baoli has quit IRC | 20:50 | |
| pabelanger | jeblair: zuul.event.trigger? | 20:50 |
| *** thorst has quit IRC | 20:50 | |
| *** baoli has joined #openstack-infra | 20:50 | |
| *** johnbelamaric has quit IRC | 20:51 | |
| jeblair | pabelanger: we already incr 'gerrit.event.<type>'. that's actually wrong now, we should future-proof that a little by moving that into the trigger class, though maybe after jhesketh's refactor lands | 20:51 |
| jeblair | (we don't guage the queue though) | 20:52 |
| pabelanger | okay, let me check out jhesketh patch | 20:52 |
| jeblair | pabelanger: for the queue itself, i'd probably do 'zuul.event_queue.trigger' (and also .management and .result) | 20:53 |
| pabelanger | ya, I was going to add all 3 | 20:53 |
| *** glauco has joined #openstack-infra | 20:56 | |
| fungi | okay, heading out for the evening. back later | 20:56 |
| jeblair | fungi: night! | 20:56 |
| EmilienM | jeblair: do you mind if I ask you an outstanding +A to unblock our CI: https://review.openstack.org/208050 - thank you | 20:57 |
| *** glauco has quit IRC | 20:57 | |
| *** vnod has quit IRC | 20:57 | |
| *** BrandonL08 has joined #openstack-infra | 20:57 | |
| jeblair | EmilienM: sure, let me finish debugging zuul though, i'm keeping a lot of stuff in my head right now | 20:57 |
| EmilienM | jeblair: sorry for interruption, no problem | 20:58 |
| clarkb | EmilienM: I will take a look | 20:58 |
| *** jsavak has quit IRC | 20:58 | |
| EmilienM | clarkb: cool | 20:59 |
| *** HeOS has joined #openstack-infra | 20:59 | |
| EmilienM | clarkb: btw, ianw pushed over my patch about logs | 20:59 |
| *** jsavak has joined #openstack-infra | 20:59 | |
| *** vnod has joined #openstack-infra | 20:59 | |
| EmilienM | clarkb: I'll test it. Is it possible to exec it on a remote dvsm node? | 21:00 |
| clarkb | EmilienM: propose a change that uses it? | 21:00 |
| *** glauco has joined #openstack-infra | 21:00 | |
| EmilienM | clarkb: sounds cool | 21:00 |
| pabelanger | pleia2: gate happy with 208002. I need to step out for a bit, but can be back online in 2 hours | 21:01 |
| *** vnod has quit IRC | 21:01 | |
| *** baoli has quit IRC | 21:02 | |
| pleia2 | pabelanger: I see! needs another +2 | 21:02 |
| *** baoli has joined #openstack-infra | 21:02 | |
| pabelanger | pleia2: okay, if we don't land it today when is your next window? | 21:02 |
| pleia2 | can someone have a look at https://review.openstack.org/#/c/208002/ ? hoping if this lands today, we can for real get grafana going | 21:02 |
| pleia2 | pabelanger: I'm around Monday | 21:02 |
| pabelanger | pleia2: holiday here in Canada :( | 21:03 |
| pleia2 | pabelanger: tuesday too! | 21:03 |
| pleia2 | Wednesday I get on the plane | 21:03 |
| pabelanger | eep | 21:03 |
| pleia2 | so let's try for today :) | 21:03 |
| pabelanger | Ya | 21:03 |
| pabelanger | hopefully we find a +2 :D | 21:03 |
| pabelanger | I | 21:03 |
| *** jsavak has quit IRC | 21:03 | |
| pleia2 | fail over is tuesday | 21:03 |
| pabelanger | I'll check back in 2hr | 21:03 |
| clarkb | I am looking | 21:03 |
| *** jsavak has joined #openstack-infra | 21:03 | |
| pleia2 | pabelanger: see you in a bit | 21:04 |
| *** dimtruck is now known as zz_dimtruck | 21:04 | |
| clarkb | approved | 21:04 |
| *** MarkAtwood has joined #openstack-infra | 21:05 | |
| *** Sukhdev_ has joined #openstack-infra | 21:05 | |
| *** glauco has quit IRC | 21:05 | |
| pleia2 | clarkb: thanks! | 21:06 |
| *** fawadkhaliq has joined #openstack-infra | 21:06 | |
| *** vnod has joined #openstack-infra | 21:06 | |
| *** glauco_ has joined #openstack-infra | 21:07 | |
| *** zz_dimtruck is now known as dimtruck | 21:07 | |
| *** vnod has quit IRC | 21:08 | |
| *** achanda has quit IRC | 21:08 | |
| *** changbl has quit IRC | 21:09 | |
| jeblair | clarkb: okay, i understand what was happening -- sever changes in different repos merged during that pass of the processor. the zuul-trigger that supports the merge-check queue by querying open changes ran for each of them, and it can take a while. eg, it took nova like 15 minutes with a cold cache. | 21:09 |
| *** jsavak has quit IRC | 21:10 | |
| *** davideagnello has quit IRC | 21:10 | |
| *** jsavak has joined #openstack-infra | 21:10 | |
| jeblair | we _do_ have a cache, but we expire things from it at the end of every queue run to keep it from growing indefinitely | 21:10 |
| *** baoli has quit IRC | 21:10 | |
| jeblair | (and that includes changes that go into the merge-check pipelines, so it can actually stay at a substantial size) | 21:11 |
| *** johnbelamaric has joined #openstack-infra | 21:11 | |
| *** alivigni has quit IRC | 21:11 | |
| *** baoli has joined #openstack-infra | 21:11 | |
| *** fawadkhaliq has quit IRC | 21:11 | |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add non voting fuel agent python jobs https://review.openstack.org/204711 | 21:12 |
| jeblair | however, changes that aren't in any pipeline (including closed changes, etc) will still be expired from the cache, and of course, once merge-check works through the backlog, the cache will get pretty small again | 21:12 |
| clarkb | I have deleted my test node for the nodepool devstack plugin | 21:12 |
| jeblair | clarkb: so i'm thinking we may want to let zuul keep a larger change cache, but to avoid it having every change ever, have it discard closed changes, or maybe do a time-based expiry | 21:13 |
| clarkb | jeblair: that sounds reasonable. Can we also make the gerrit queries faster? | 21:13 |
| jeblair | (maybe discard changes untouched in 30 mins or something) | 21:13 |
| clarkb | I am guessing we already do a bulk give me 500 open chages at a time query | 21:14 |
| clarkb | I seem to recall reviewing it that way | 21:14 |
| *** prad has quit IRC | 21:14 | |
| *** davideagnello has joined #openstack-infra | 21:14 | |
| *** hogepodge has quit IRC | 21:14 | |
| jeblair | clarkb: well, we do a fast "give me all the changes" query and ask for very little data, but then we query each individual change to get all the data. otoh, that's a little slower, however, it does let us use the cache. (and of course we're on a persistent ssh connection, so network overhead is minimal) | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| *** ivar-lazzaro has quit IRC | 21:15 | |
| *** cdent has quit IRC | 21:15 | |
| *** ivar-lazzaro has joined #openstack-infra | 21:16 | |
| jeblair | clarkb: we could consider making the open change query include more data, so that we don't have to do the secondary queries. however, i don't know if that would be better. maybe i'll benchmark that. | 21:16 |
| *** baoli has quit IRC | 21:18 | |
| openstackgerrit | Merged openstack-infra/system-config: Remove local grafana database https://review.openstack.org/208002 | 21:19 |
| *** teran has joined #openstack-infra | 21:20 | |
| clarkb | pleia2: can I trade you https://review.openstack.org/#/c/208043/1 ? | 21:20 |
| clarkb | pleia2: also https://review.openstack.org/#/c/207095/ sinc eyou were familiar with my doc on how to do it by hand | 21:21 |
| *** hogepodge has joined #openstack-infra | 21:21 | |
| pleia2 | clarkb: sure, taking a look now | 21:21 |
| pleia2 | ooh neat | 21:21 |
| *** shashank_hegde has quit IRC | 21:22 | |
| *** tlian has quit IRC | 21:23 | |
| *** boris-42 has joined #openstack-infra | 21:23 | |
| *** rfolco has quit IRC | 21:23 | |
| *** shashank_hegde has joined #openstack-infra | 21:24 | |
| waynr | zaro: looking | 21:25 |
| *** cbader has quit IRC | 21:25 | |
| waynr | oh looks like i +1'd it back before attaining +2 powers | 21:27 |
| *** tqtran has joined #openstack-infra | 21:27 | |
| waynr | zaro: what does "workflow +1" mean? | 21:27 |
| jeblair | waynr: that's the button you press to enqueue it into the gate pipeline in zuul to be merged | 21:28 |
| jeblair | waynr: so there's a separation of saying "i think this is ready to be merged (code-review +2)" from "i will cause this to be merged now (workflow +1)" | 21:29 |
| *** e0ne has quit IRC | 21:30 | |
| *** akraminsky has joined #openstack-infra | 21:30 | |
| *** akraminsky_ has joined #openstack-infra | 21:30 | |
| waynr | oh, has JJB's requirement for workflow +1 been reduced to a single +2? for some reason I had through previously that 2 +2's were required | 21:30 |
| *** BrandonL08 has quit IRC | 21:30 | |
| *** MarkAtwood has quit IRC | 21:30 | |
| *** bapalm_ has joined #openstack-infra | 21:31 | |
| waynr | my brain is probably addled by heat and too much work | 21:31 |
| pleia2 | aynr: 2 +2s is a social construct, not technically enforced one | 21:32 |
| pleia2 | waynr: ^^ | 21:32 |
| jeblair | waynr: there's only a _technical_ requirement for one +2, it's a self-observed policy that there be 2x+2s. this way we can exercise some judgement and in some cases (trival rebase of patch with multiple prior +2s; typo fix, etc) bypass the requirement | 21:32 |
| zaro | pleia2: nice description, very concise. | 21:33 |
| jeblair | pleia2 is good at words | 21:34 |
| pleia2 | I've had to explain it many times :) | 21:34 |
| * jeblair is not as good at word-putting-togethering. | 21:34 | |
| *** xyang0 has joined #openstack-infra | 21:34 | |
| *** wenlock has quit IRC | 21:34 | |
| pleia2 | jeblair: you are detailed though! | 21:34 |
| *** crc32 has quit IRC | 21:34 | |
| *** pelix has quit IRC | 21:34 | |
| waynr | thanks both of you for the explanation :D | 21:35 |
| jeblair | clarkb: i'm (still) running that benchmark now. | 21:35 |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Remove q-vpn service enable from devstack based jobs https://review.openstack.org/208062 | 21:36 |
| jeblair | clarkb: ah, it just finished. summary: simple query: 2MB data in 20s. full query: 55MB data in 492s. | 21:36 |
| *** alexsyip has joined #openstack-infra | 21:37 | |
| jeblair | clarkb: that's probably enough data that my dsl line may have an impact on the test time. | 21:37 |
| *** dimtruck is now known as zz_dimtruck | 21:37 | |
| jeblair | clarkb: at any rate, i think i am leaning toward the option of leaving the queries the way they are and keeping a larger cache | 21:38 |
| jeblair | i think that will reduce the load on gerrit more than the option of gathering more data in the query | 21:39 |
| *** teran has quit IRC | 21:40 | |
| *** BrandonL08 has joined #openstack-infra | 21:41 | |
| *** DericHorn-HP has joined #openstack-infra | 21:41 | |
| jeblair | clarkb: oh, i missed something earlier -- we actually only prune the cache when we reconfigure | 21:41 |
| *** pc_m has quit IRC | 21:43 | |
| *** signed8b_ is now known as signed8bit_ZZZzz | 21:44 | |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Improve delete all job performance https://review.openstack.org/199193 | 21:48 |
| *** amitgandhinz has quit IRC | 21:49 | |
| *** xyang0 has quit IRC | 21:49 | |
| *** ivar-lazzaro has quit IRC | 21:50 | |
| *** jsavak has quit IRC | 21:50 | |
| *** ivar-lazzaro has joined #openstack-infra | 21:51 | |
| *** thorst has joined #openstack-infra | 21:51 | |
| *** jsavak has joined #openstack-infra | 21:51 | |
| openstackgerrit | ewitter proposed openstack-infra/git-review: Add topic support for commit messages with 'US'. https://review.openstack.org/208065 | 21:51 |
| *** alexsyip has quit IRC | 21:52 | |
| *** thorst has quit IRC | 21:53 | |
| *** sdake has quit IRC | 21:54 | |
| *** sdake has joined #openstack-infra | 21:54 | |
| *** bapalm_ has quit IRC | 21:55 | |
| *** ildikov has quit IRC | 21:56 | |
| *** tsekiyama has joined #openstack-infra | 21:59 | |
| *** tjones has quit IRC | 22:01 | |
| *** sabeen2 has quit IRC | 22:01 | |
| *** tjones has joined #openstack-infra | 22:01 | |
| *** sdake has quit IRC | 22:02 | |
| *** teran has joined #openstack-infra | 22:02 | |
| *** rlandy has quit IRC | 22:03 | |
| *** gordc has quit IRC | 22:04 | |
| *** ivar-lazzaro has quit IRC | 22:05 | |
| *** ivar-lazzaro has joined #openstack-infra | 22:05 | |
| *** shashank_hegde has quit IRC | 22:06 | |
| *** glauco_ has quit IRC | 22:07 | |
| *** nelsnelson has quit IRC | 22:11 | |
| *** sergmelikyan has joined #openstack-infra | 22:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Make sure puppet-syntax jobs have bundler installed https://review.openstack.org/208050 | 22:14 |
| pleia2 | if any roots are bored (ha ha) and want to help me debug grafana, I welcome the collaboration over on 23.253.76.254 | 22:14 |
| jeblair | pleia2: you got it. | 22:15 |
| jeblair | pleia2: i do not, however, confess to being bored. | 22:15 |
| *** shashank_hegde has joined #openstack-infra | 22:15 | |
| jeblair | ;) | 22:15 |
| pleia2 | jeblair: cool, so there's a weird error in /var/log/grafana/grafana.log about the database | 22:15 |
| pleia2 | it seems to do smart things first, then falls over on the database url that totally resolves | 22:16 |
| pleia2 | apache also has problems in /var/log/apache2/grafana.openstack.org_error.log but that may just be because grafana isn't quite right in the head yet | 22:16 |
| *** olaph has quit IRC | 22:16 | |
| *** olaph has joined #openstack-infra | 22:16 | |
| pleia2 | yeah, it's not running on 127.0.0.1:8080 so apache won't redirect | 22:17 |
| jeblair | yeah, i'd assume that's the case and focus on the grafana db problem first | 22:17 |
| jeblair | which i agree, is weird. | 22:17 |
| nibalizer | so did the grafana patches land? | 22:17 |
| nibalizer | I looked at modules.env but did not see the grafana module | 22:17 |
| pleia2 | nibalizer: it's there | 22:17 |
| jeblair | pleia2: it looks like it succeeded at 21:53, but then maybe restarted at 21:54 and failed? | 22:19 |
| pleia2 | jeblair: yeah, it does look like it's starting over what it was trying to do before | 22:20 |
| pleia2 | this is a brand new machine, only did the first puppet run so far | 22:20 |
| nibalizer | huh okay, I think we discovered internally that the grafana module requires camptocamp-archive and puppet-elasticsearch requires puppetcommunity-archive which do not have the same api | 22:20 |
| pleia2 | (ignore 9:50 AM in puppetboard, that's a different server) | 22:20 |
| nibalizer | so that is probably a problem for upstream as well | 22:21 |
| *** ddieterly has quit IRC | 22:21 | |
| pleia2 | nibalizer: oh, that would be unfortunate | 22:21 |
| nibalizer | however puppetboard seems sorta happy so Idunno... | 22:22 |
| *** Yichen has quit IRC | 22:23 | |
| *** xyang0 has joined #openstack-infra | 22:24 | |
| *** sdake has joined #openstack-infra | 22:25 | |
| jeblair | we should add the mysql client to the host :) | 22:26 |
| clarkb | +1 | 22:26 |
| *** wenlock has joined #openstack-infra | 22:26 | |
| jeblair | tables exist in the db and have data in them | 22:27 |
| pleia2 | for debugging? | 22:27 |
| jeblair | pleia2: yep | 22:27 |
| *** xyang0 has quit IRC | 22:27 | |
| jeblair | i've logged in from another host that does have a client | 22:27 |
| pleia2 | I'll toss up a patch | 22:27 |
| jeblair | so i'm pretty sure when grafana said it ran the migrations, it wasn't lying | 22:27 |
| clarkb | for that you can use the mysql module :) | 22:27 |
| * pleia2 nods | 22:27 | |
| jeblair | erm. i'm kind of at "restart grafana and see if it happens again" | 22:28 |
| pleia2 | wfm, I'll see how that goes | 22:28 |
| clarkb | pleia2: for the nodepool test I was sort of going for minimal "it works" then we can go from there like poll and pass early, delete and check it rebuilds, etc etc | 22:28 |
| jeblair | btw /var/log/grafana/xorm.log is empty. :/ | 22:28 |
| pleia2 | hey, that worked | 22:29 |
| pleia2 | gremlins | 22:29 |
| pleia2 | I'll set up dns and hope it doesn't break again | 22:29 |
| jeblair | that's pretty much my favorite kind of software. | 22:29 |
| *** puranamr has quit IRC | 22:30 | |
| *** yolanda has quit IRC | 22:30 | |
| pleia2 | ok, dns should be resolving by the time pabelanger returns and can do his thing | 22:31 |
| clarkb | there is indeed a thing there | 22:32 |
| clarkb | I don't have any dashboards though | 22:32 |
| pleia2 | clarkb: yeah, that's fine, it should usually work :) | 22:33 |
| pleia2 | and yeah, I don't know what grafana is supposed to have right now | 22:33 |
| *** mwhahaha has joined #openstack-infra | 22:33 | |
| jeblair | i think it's empty until we start defining dashboards with grafyaml | 22:34 |
| pleia2 | but it's running and not spewing errors | 22:34 |
| clarkb | I have to say this UI is far too busy for me | 22:34 |
| clarkb | I just want to graph a thing | 22:34 |
| cody-somerville | clarkb: graphite should still be available to you ol' man ;) | 22:35 |
| pleia2 | haha | 22:35 |
| clarkb | it is, I am just saying that the number of knobs here is ecrazy | 22:35 |
| mwhahaha | hey openstack-infra folks, we have some reviews that were workflowed this morning that still haven't merged (for example https://review.openstack.org/#/c/206493/) what is the best way to proceed? | 22:36 |
| clarkb | looks like I can define new ones but not save them, guessing we can manually creat things ,export then put into the grafyaml? | 22:37 |
| jeblair | clarkb: well, i hope we can just define them in the grafyaml; it should be easy to read | 22:38 |
| *** ashleighfarnham has quit IRC | 22:38 | |
| jeblair | but yeah, can create things ad-hoc in memory to play around | 22:38 |
| pabelanger | pleia2: back online | 22:38 |
| pabelanger | just reading scrollback | 22:38 |
| clarkb | though I can't ad hoc with our graphite server as it is not (yet) a datasource | 22:39 |
| clarkb | mwhahaha: its possible it was lost in the zuul restart earlier, I would try re approving it | 22:40 |
| pabelanger | pleia2: NICE! http://grafana.openstack.org/ | 22:40 |
| pabelanger | pleia2: would be interested to see what DB issue was | 22:41 |
| pleia2 | it lives, eventually | 22:41 |
| pleia2 | pabelanger: the error log said it could resolve the url, after it just did a minute before | 22:41 |
| pabelanger | pleia2: Hmm, I'll have to test with a remote DB locally and see what happens | 22:42 |
| pabelanger | lets see how it holds up | 22:42 |
| * pleia2 nods | 22:42 | |
| pleia2 | we can always fall back to local if there's something wrong with this setup, but I'm hoping it works | 22:42 |
| pabelanger | agreed | 22:43 |
| pabelanger | very cool | 22:43 |
| jeblair | clarkb, mwhahaha: yeah, it was missing a jenkins +1, so it needs a recheck | 22:43 |
| pabelanger | pleia2: now I can get started on grafyaml stuff | 22:43 |
| pabelanger | pleia2: thanks for the help! | 22:43 |
| pleia2 | pabelanger: hooray, let me know if you need anything | 22:43 |
| pabelanger | will do | 22:43 |
| *** shashank_hegde has quit IRC | 22:43 | |
| pleia2 | pabelanger: and enjoy your 3 day weekend :) | 22:44 |
| pabelanger | pleia2: will thanks | 22:44 |
| *** ashleighfarnham has joined #openstack-infra | 22:44 | |
| *** shashank_hegde has joined #openstack-infra | 22:44 | |
| clarkb | jeblair: any chance you can review https://review.openstack.org/#/c/208043/ so that the nodepool plugin can be self testing before it merges? | 22:45 |
| mwhahaha | jeblair: thanks we'll give it a shot | 22:45 |
| mwhahaha | clarkb: thanks | 22:45 |
| *** mwhahaha has left #openstack-infra | 22:45 | |
| timrc | pabelanger: Congrats on grafana :) | 22:46 |
| pabelanger | timrc: thanks, should be fun the coming weeks | 22:46 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: configure git PathRestriction as an extension https://review.openstack.org/201599 | 22:46 |
| *** theannegentle has quit IRC | 22:48 | |
| clarkb | greghaynes: I am looking at your WIP'd dib change for growrooting | 22:49 |
| clarkb | greghaynes: looks like its only a debian/upstart thing? is that because everything else has working growroots? | 22:49 |
| *** sergmelikyan has quit IRC | 22:50 | |
| jeblair | clarkb: +3 with comment | 22:53 |
| jeblair | clarkb, pleia2: i believe this is a request to manually trigger some jobs: http://lists.openstack.org/pipermail/openstack-infra/2015-July/002988.html | 22:53 |
| jeblair | i can do that; but it's a release job, so i wanted to make sure to grab the mutex on that | 22:54 |
| *** wenlock has quit IRC | 22:54 | |
| *** BrandonL08 has quit IRC | 22:55 | |
| clarkb | jeblair: I won't touch it | 22:55 |
| *** earlephilhower has quit IRC | 22:55 | |
| *** apevec has joined #openstack-infra | 22:55 | |
| *** armax has joined #openstack-infra | 22:55 | |
| pleia2 | jeblair: all yours | 22:55 |
| apevec | jeblair, thanks for looking into missing tag events | 22:58 |
| jeblair | apevec: no problem, i'm preparing to run the jobs now | 22:59 |
| *** austin81 has left #openstack-infra | 22:59 | |
| apevec | ttx mention that dhellmann had issues w/ *aas for liberty-2 too, but that was unrelated I guess | 22:59 |
| *** puranamr has joined #openstack-infra | 22:59 | |
| jeblair | yeah, i believe we've worked around those and expect it to succeed | 22:59 |
| timrc | pabelanger: Yeah, I already want to push a dashboard for nodepool :) | 23:00 |
| *** theannegentle has joined #openstack-infra | 23:01 | |
| *** sergmelikyan has joined #openstack-infra | 23:02 | |
| jeblair | honestly, i think this is complex enough that i want to quickly update the enqueue command to support refs | 23:03 |
| *** davideag_ has joined #openstack-infra | 23:03 | |
| *** davideagnello has quit IRC | 23:03 | |
| *** Sukhdev_ has quit IRC | 23:04 | |
| openstackgerrit | Terry Howe proposed openstack/requirements: Add openstacksdk to the global requirements https://review.openstack.org/208079 | 23:05 |
| openstackgerrit | Merged openstack-infra/project-config: Run a nodepool devstack test https://review.openstack.org/208043 | 23:06 |
| clarkb | greghaynes: ok review posted | 23:06 |
| *** theannegentle has quit IRC | 23:06 | |
| mordred | jeblair: bah | 23:06 |
| mordred | jeblair: (oh, I was bah-ing something from scrollback, nm) | 23:07 |
| jeblair | mordred: blaeunaouhnohu! | 23:07 |
| mordred | jeblair: yes. that is the correct response | 23:08 |
| *** jsavak has quit IRC | 23:08 | |
| clarkb | mordred: we are || that close to testing nodepool upstream against devstack | 23:10 |
| clarkb | mordred: it is going to be awesome | 23:10 |
| mordred | clarkb: WOOT | 23:10 |
| mordred | clarkb: I am very excited about that | 23:10 |
| *** armax has quit IRC | 23:11 | |
| mordred | clarkb: that will make landing the patches to use more shade functions less terrifying | 23:11 |
| clarkb | mordred: yup | 23:11 |
| clarkb | mordred: and should make testing the zuulv3 functionality less scary | 23:11 |
| mordred | clarkb: I feel like we're blocked on some of the dib-nodepool things but keep travelling - is there anything I should be looking at? | 23:12 |
| clarkb | mordred: we need to fix rootfs growing | 23:12 |
| clarkb | mordred: https://review.openstack.org/#/c/206636/ thats where greghaynes is trying to do it with dib | 23:12 |
| clarkb | I just reviewed it and its WIP so general ideaing there is probably good | 23:12 |
| mordred | cool | 23:13 |
| mordred | ok. and I think we need to add retries for rackspace 396 and also maybe fail slightly better when instances don't get IPs | 23:13 |
| clarkb | mordred: as is we can't devstack on ubuntu-trusty because the / partition is only 13 GB | 23:13 |
| mordred | I saw in the scrollback that we're doing the right thing on that already | 23:13 |
| mordred | but we could properly throw the right error | 23:13 |
| clarkb | mordred: jeblair got the response for lack of IP and they basically said delete and make a new node | 23:13 |
| mordred | yah | 23:13 |
| clarkb | yup | 23:13 |
| mordred | right now I think we catch that as an exception later in nodepool and delete it - it would be nice to fail in the create server same as if it hit error state | 23:14 |
| mordred | hrm. and maybe a statsd counter for "error" then one for "error.error" and one for "error.noip" | 23:15 |
| mordred | might be nice to be able to show no-ip server error rates | 23:15 |
| clarkb | for error 396 you just want shade to rety or nodepool to retry? Blindly retrying there seems odd because its an obtuse error and its meaning could change at any time | 23:16 |
| *** rossella_s has quit IRC | 23:16 | |
| *** rossella_s has joined #openstack-infra | 23:16 | |
| mordred | clarkb: I want shade to retry until we hit the configured timeout | 23:16 |
| mordred | clarkb: since that's the meaning of the error that has been communicated so far - but I do think logging that we're doing it is important | 23:17 |
| mordred | oh | 23:17 |
| mordred | clarkb: I already added that to shade :) | 23:18 |
| clarkb | also we are all living in an oven in the northwest | 23:18 |
| mordred | if status.status == 'failure': | 23:18 |
| mordred | if status.message == IMAGE_ERROR_396: | 23:18 |
| clarkb | so come monday we may have been eaten by the sasquatches | 23:18 |
| mordred | :) | 23:18 |
| mordred | clarkb: you shudl come to brisbane, it's a lovely 14C here now | 23:18 |
| *** shashank_hegde has quit IRC | 23:18 | |
| mordred | clarkb: just consider - when is the last time someone in australia said "you shoudl come visit here, the weather is more reasonably cool" | 23:19 |
| docaedo | clarkb: I'm concerned, haven't been up here more than a year - does the heat increase sasquatch activity? Should we be worried? | 23:19 |
| clarkb | docaedo: I am pretty sure they use the volcanos as heating elements to cook the humans for a great feast | 23:19 |
| mordred | docaedo: just make sure to rub old garbage on the back of your neck - it helps scare them off | 23:19 |
| clarkb | also I didn't realize it had been years since we had a 100F day and now I feel spoiled | 23:20 |
| docaedo | great advice, thanks! brb | 23:20 |
| *** DericHorn-HP has quit IRC | 23:22 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Move puppet-unit gate to devstack-centos7 https://review.openstack.org/208084 | 23:23 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add query for ceilometer meters faultstring 500 https://review.openstack.org/208085 | 23:23 |
| mordred | clarkb: so - it looks like we're not having 396 errors anymore | 23:24 |
| *** olaph has quit IRC | 23:24 | |
| clarkb | mordred: oh? did the images start uplodaing? | 23:24 |
| mordred | clarkb: at least, Ido not see 396 or "Image creation failed:" in any of the logs | 23:24 |
| clarkb | I saw it when running manually | 23:24 |
| clarkb | so not sure how it shows up in the service logs | 23:25 |
| mordred | yeah? I don't see it in the logs at all | 23:25 |
| pabelanger | EmilienM: ^ for puppet-unit jobs. Going to call it a day, see you Tuesday | 23:25 |
| *** olaph has joined #openstack-infra | 23:25 | |
| *** DericHorn-HP has joined #openstack-infra | 23:25 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:26 | |
| *** pcrews has quit IRC | 23:26 | |
| *** johnbelamaric has quit IRC | 23:28 | |
| *** pcrews has joined #openstack-infra | 23:28 | |
| *** shashank_hegde has joined #openstack-infra | 23:28 | |
| *** dannywilson has quit IRC | 23:30 | |
| *** patrickeast has quit IRC | 23:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Ensure we actually purge all puppet related directories https://review.openstack.org/208088 | 23:31 |
| *** achanda has joined #openstack-infra | 23:31 | |
| *** bharath has quit IRC | 23:33 | |
| clarkb | gah small bug in the nodepool devstack job def | 23:35 |
| *** MarkAtwood has joined #openstack-infra | 23:35 | |
| EmilienM | pabelanger: ack | 23:36 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run gate-dsvm-nodepool-nv on devstack-trusty https://review.openstack.org/208090 | 23:36 |
| clarkb | mordred: jeblair pleia2 ^ sorry to bother but thats the fix (wrong node type was used previously) | 23:36 |
| *** julim has quit IRC | 23:37 | |
| mordred | clarkb: +2 | 23:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Throw an exception when servers don't have IPs https://review.openstack.org/208092 | 23:39 |
| *** yamamoto has joined #openstack-infra | 23:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add enqueue-ref command to client https://review.openstack.org/208093 | 23:40 |
| * clarkb reviews the new zuul command | 23:40 | |
| jeblair | clarkb, mordred: can i trouble you for a review on that so that i can instamerge it, restart zuul, and use it for the neutron releases? | 23:40 |
| clarkb | jeblair: yup trade you for review on 208090 :) | 23:40 |
| mordred | jeblair: on it | 23:40 |
| jeblair | clarkb: deal | 23:40 |
| * jeblair pretends to spend longer thinking about 208090 | 23:41 | |
| *** yamamoto has quit IRC | 23:41 | |
| clarkb | jeblair: refresh my memory, trigger is gerrit or timer? maybe we should help doc that better? | 23:42 |
| clarkb | followup chnage is fine since the older command doesn't help it either | 23:42 |
| *** ajmiller has quit IRC | 23:42 | |
| jeblair | clarkb: trigger=gerrit in our case (timer would be a valid value that won't actually work) | 23:42 |
| mordred | jeblair: fine by me - also, in the world of docs - something indicating difference between enqueue and enqueue-ref would be nice - not necessary this instant though | 23:43 |
| *** yamamoto has joined #openstack-infra | 23:43 | |
| jeblair | clarkb: i think it gets better with the jhesketh stack | 23:43 |
| *** marcusvrn has quit IRC | 23:43 | |
| *** shashank_hegde has quit IRC | 23:44 | |
| *** olaph has quit IRC | 23:51 | |
| clarkb | mordred: I don't know what a hing is and now I feel like I don't understand your talk that I am not at | 23:51 |
| *** BrandonL08 has joined #openstack-infra | 23:51 | |
| *** olaph has joined #openstack-infra | 23:51 | |
| EmilienM | clarkb: are you root on gerrit? | 23:52 |
| clarkb | EmilienM: yes | 23:52 |
| mordred | clarkb: :) | 23:52 |
| EmilienM | clarkb: please add me on https://review.openstack.org/#/admin/groups/1022,members | 23:52 |
| EmilienM | clarkb: this is a new group, that allows me and crinkle to manage branches and tags | 23:52 |
| clarkb | EmilienM: done | 23:52 |
| *** yamamoto has quit IRC | 23:52 | |
| EmilienM | clarkb: thanks | 23:52 |
| *** ybathia has quit IRC | 23:53 | |
| EmilienM | crinkle: welcome to puppet-manager-release team ! we are 2 | 23:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add enqueue-ref command to client https://review.openstack.org/208093 | 23:54 |
| jeblair | pep8 can diaf. | 23:55 |
| jeblair | pyflakes, otoh, thank you very much. | 23:55 |
| clarkb | jeblair: this way your whitespace looks like my whitespace and everyone rejoices \o/ thats how it works right? | 23:56 |
| *** sdake has quit IRC | 23:57 | |
| jeblair | clarkb: most of those were "golly gee your git shas are too long" | 23:57 |
| clarkb | so I have thought about this a bit and I think some enforcement is good ebcaus whenever I have to read jenkins related code vim gets all crnaky | 23:57 |
| *** tsekiyam_ has joined #openstack-infra | 23:57 | |
| clarkb | because they mix tabs and spaces and its all over the place hard to read, but realigning function args when they are aligned already is meh | 23:57 |
| jeblair | clarkb: yeah, there are some good things. figuring out how to fit a git sha on a line in a test isn't one of them. :/ | 23:58 |
| *** theannegentle has joined #openstack-infra | 23:58 | |
| jeblair | the result is actually _less_ readable than before. | 23:58 |
| jeblair | but anyway | 23:59 |
| mordred | yah. line-length is not particularly helpful I don't think | 23:59 |
| jeblair | i'm going to pick up my film and maybe that will be merged when i get back | 23:59 |
| *** BrandonL08 has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!