| *** hashar has joined #openstack-infra | 00:01 | |
| *** jgrimm has quit IRC | 00:02 | |
| *** CaptTofu has joined #openstack-infra | 00:02 | |
| *** markmcclain has joined #openstack-infra | 00:02 | |
| fungi | wow, there's an odd failure to see in the gate... https://jenkins03.openstack.org/job/gate-glance-python27/142/consoleText | 00:02 |
|---|---|---|
| *** vipul-away is now known as vipul | 00:02 | |
| *** slong has joined #openstack-infra | 00:03 | |
| morganfainberg | fungi, py27 failure? | 00:03 |
| *** gokrokve has quit IRC | 00:04 | |
| morganfainberg | fungi, oh oh in gate?! | 00:04 |
| *** slong_ has quit IRC | 00:04 | |
| *** gokrokve has joined #openstack-infra | 00:04 | |
| *** pcrews has quit IRC | 00:05 | |
| fungi | morganfainberg: yeah, with what looks like something that would be very hard to blame on infrastructure... could indicate a bug in glance or the glance unit tests, i suppose | 00:05 |
| *** markmcclain has quit IRC | 00:05 | |
| morganfainberg | fungi, yeah, passed a couple days ago. maybe something merged in that is changing the result of those tests now | 00:06 |
| anteaya | done, those patches are out of the gate | 00:07 |
| fungi | or it's a bug which doesn't crop up on every test run | 00:07 |
| anteaya | let me know if you see any others, I will remove them | 00:07 |
| morganfainberg | fungi, possible as well. | 00:07 |
| *** sarob has joined #openstack-infra | 00:07 | |
| fungi | anteaya: thanks for the info. i'll let you know if/when i spot others | 00:07 |
| *** ok_delta__ has quit IRC | 00:08 | |
| *** ok_delta has quit IRC | 00:08 | |
| morganfainberg | fungi, wonder if that patch needs to be pulled out of the gate. just saw a reset at the top and it's running again | 00:08 |
| anteaya | fungi: thanks | 00:09 |
| *** gokrokve has quit IRC | 00:09 | |
| fungi | morganfainberg: it's a judgement call for the glance devs. if you don't think that change is likely to be tyhe cause, then i'm not sure pulling it out will be uch help | 00:09 |
| morganfainberg | fungi, not a glance dev, so... :P | 00:10 |
| *** sarob has quit IRC | 00:10 | |
| fungi | morganfainberg: th "reset" was the pending promote of the nova fix we've been trying to get in all day | 00:10 |
| morganfainberg | fungi, though as a dev for openstack and waiting for lots of I2 patches, i'd say if it's failing again it might be worth pulling. | 00:10 |
| morganfainberg | fungi, aye, i saw that | 00:10 |
| *** hashar has quit IRC | 00:11 | |
| morganfainberg | not complaining about the reset ;) | 00:11 |
| *** sarob has joined #openstack-infra | 00:11 | |
| lifeless | jeblair: I could hook that in quite easily | 00:11 |
| anteaya | the zuul status page still shows them in the gate queue | 00:11 |
| fungi | when i started the promote, there weren't any changes being tested i the gate, but the ref recalculation lag made it take long enough stuff started testing before it took effect | 00:12 |
| lifeless | jeblair: also I was considering moving to bulk operations - do one list of current floating ips, servers, keypairs, figure out what to delete, then issue the deletes | 00:12 |
| morganfainberg | anteaya, likely waiting on events to process | 00:12 |
| anteaya | morganfainberg: I hope | 00:12 |
| morganfainberg | anteaya, 1555 events in queue | 00:12 |
| morganfainberg | anteaya, my guess is yes | 00:12 |
| lifeless | jeblair: so that we don't keep re-querying the same data - I guess it depends on the rate limit implementation of the provider whether we'd be penalised for that | 00:12 |
| *** pballand has quit IRC | 00:12 | |
| fungi | anteaya: yeah, the new patchsets will be gerrit events. it'll take a while for them to land (may even be after the changes make it to the top of the gate and fall out anyway) | 00:13 |
| anteaya | morganfainberg: ummm, I'm talking about 2 patches I just sniped from the gate queue, I'm still seeing them in the gate queue | 00:13 |
| lifeless | jeblair: but since *we* have a naive seconds-since-last-op implementation, it would increase how many deletes we can send through | 00:13 |
| morganfainberg | anteaya, ^ | 00:13 |
| anteaya | fungi: boo | 00:13 |
| anteaya | is there something I can do that is faster? | 00:13 |
| *** oubiwann_ has quit IRC | 00:13 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add rate limiting to dependent pipeline queues https://review.openstack.org/68219 | 00:13 |
| clarkb | jeblair: ^ I am actually fairly happy with that | 00:13 |
| clarkb | I am now open to opinions on removing the _type and _factor stuff | 00:14 |
| * fungi sees if he can thumb his nose at the internet gods long enough to review that | 00:14 | |
| clarkb | since that isn't strongly tested I think it should probably be on the chopping block | 00:14 |
| jeblair | lifeless: ok, i think with the addition of cleanup-per-provider, i'm sold on your approach; the bulk operations change makes sense and sounds good. | 00:14 |
| *** DennyZhang has joined #openstack-infra | 00:14 | |
| *** sarob has quit IRC | 00:15 | |
| fungi | clarkb: a doc patch corresponding to this new feature/behavior would also make a good todo item for soon, though doesn't need to be part of that change of course | 00:16 |
| *** bauzas has quit IRC | 00:17 | |
| clarkb | fungi: definitely, was hoping to have something concrete before I documented say window_increase_factor | 00:17 |
| fungi | absolutely | 00:17 |
| *** senk has quit IRC | 00:18 | |
| fungi | the test does a great job of exercising all the effects, i think | 00:19 |
| *** wenlock has quit IRC | 00:19 | |
| clarkb | fungi: yeah, I had it go through step by step. the reset and going from 2 to 1 is a nice thing :) | 00:20 |
| clarkb | fungi: once thing that doesn't test is the affect on dependent changes. not sure if thatneeds to be explicitly tested | 00:20 |
| fungi | with window and floor i think we now need walls, a ceiling and door (the fire marshall would probably insist) | 00:20 |
| jeblair | fungi: we have a gate | 00:20 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: make scm test as the example https://review.openstack.org/65186 | 00:20 |
| fungi | we do! | 00:20 |
| *** vipul is now known as vipul-away | 00:21 | |
| *** yassine has quit IRC | 00:21 | |
| *** ryanpetrello has joined #openstack-infra | 00:22 | |
| *** fifieldt has quit IRC | 00:24 | |
| fungi | clarkb: the only other potential misbehavior i worry about is if we have multiple changes failing in reverse order caused by one change ahead of them (through happenstance of varying provider performance impacting relative job ru-time) | 00:25 |
| fungi | run-time | 00:25 |
| anteaya | I have to get some sleep, the good thing is I will probably be awake early in the morning anyway | 00:25 |
| fungi | clarkb: but i think that's likely to be rare enough in practice so as to deal with it when it happens | 00:26 |
| anteaya | leave messages in channel if there is anything I can do when I return | 00:26 |
| *** mrodden has quit IRC | 00:26 | |
| *** sarob has joined #openstack-infra | 00:27 | |
| *** markmcclain has joined #openstack-infra | 00:27 | |
| *** esker has joined #openstack-infra | 00:27 | |
| *** dangers is now known as dangers_away | 00:27 | |
| *** CaptTofu has quit IRC | 00:28 | |
| *** CaptTofu has joined #openstack-infra | 00:28 | |
| *** reed has quit IRC | 00:28 | |
| fungi | gah, 68147,3 failed its py27 unit tests | 00:29 |
| jorisroovers | fungi, I figured it out, thanks for the help :-) | 00:30 |
| jog0 | AssertionError: False is not true. such a descriptive error | 00:30 |
| *** dangers_away has quit IRC | 00:31 | |
| *** jasondotstar has quit IRC | 00:31 | |
| fungi | jorisroovers: sorry i couldn't be more helpful | 00:31 |
| fungi | jorisroovers: how did you end up accomplishing it? | 00:31 |
| *** sandywalsh has quit IRC | 00:31 | |
| jorisroovers | fungi, no worries. It turned out the be a file that was removed in master and that I had moved in my patch | 00:31 |
| *** carl_baldwin has quit IRC | 00:32 | |
| jorisroovers | I just did a rebase, then during conflict removed that file and commited/reviewed | 00:32 |
| *** dangers_away has joined #openstack-infra | 00:32 | |
| fungi | jog0 it passed the same job in the check pipeline too | 00:32 |
| fungi | jorisroovers: yep, that doesn't sound so bad as what i was expecting then. thanks for the report | 00:33 |
| jorisroovers | fungi, yeah. I just took me a while as started over completely after messing up my local copies | 00:33 |
| fungi | okay, dropping offline again for a while in hopes of catching my next flight | 00:33 |
| jorisroovers | fungi, good luck with that. As always, your help was VERY much appreciated! | 00:34 |
| *** ArxCruz has joined #openstack-infra | 00:34 | |
| jog0 | fungi: that was then, this is now maybe nova changed underneath | 00:34 |
| *** zul has joined #openstack-infra | 00:37 | |
| clarkb | fungi: that would harshly shrink the window size, but I am not sure how we could handle that | 00:38 |
| clarkb | fungi: actually no that won't | 00:38 |
| clarkb | we only adjust the window when we report | 00:38 |
| clarkb | fungi: I think I will just add some dependent changes locally and bump the test count up :) | 00:40 |
| clarkb | does anyone know if the stable branches are happy now? | 00:42 |
| *** flaper87 is now known as flaper87|afk | 00:42 | |
| jeblair | clarkb: i think grizzly is but not havana | 00:42 |
| clarkb | jeblair: thanks | 00:42 |
| *** reed has joined #openstack-infra | 00:44 | |
| *** reed has quit IRC | 00:48 | |
| *** kgriffs_afk is now known as kgriffs | 00:50 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add rate limiting to dependent pipeline queues https://review.openstack.org/68219 | 00:50 |
| *** kgriffs has left #openstack-infra | 00:50 | |
| clarkb | jeblair: fungi ^ now with a bit more testing. I am fairly confident in the change now. Looking for feedback on all of the toggles | 00:50 |
| clarkb | I am going to context switch into putting the new SCP plugin build on jenkins04 | 00:51 |
| *** CaptTofu has quit IRC | 00:54 | |
| *** thuc has joined #openstack-infra | 00:56 | |
| *** dcramer__ has joined #openstack-infra | 00:56 | |
| *** morganfainberg is now known as morganfainberg|z | 01:00 | |
| russellb | fungi: saw that >_< unrelated failure | 01:01 |
| clarkb | 68147,3 failed python27 tests | 01:01 |
| russellb | yeah :( | 01:02 |
| clarkb | oh fungi noticed earlier | 01:02 |
| russellb | sorry ... | 01:02 |
| russellb | that fail is https://bugs.launchpad.net/nova/+bug/1270654 | 01:02 |
| clarkb | jenkins04 will be idle shortly, I am going to restart it to pick up the new scp plugin | 01:02 |
| clarkb | oh man that test | 01:03 |
| russellb | i'll approve again after it fails i guess | 01:03 |
| clarkb | russellb: pretty sure that test was really broken before testr'ing and it only sort of got fixed after testr'ing | 01:03 |
| russellb | i'm bummed this failed ... took all day to get it this far | 01:03 |
| russellb | 318 hits in the last 12 hours | 01:03 |
| russellb | may be worth ninja-merging if that's possible (assuming the rest passes) | 01:04 |
| russellb | check passed fine | 01:04 |
| clarkb | russellb: I will let jeblair make a call on that | 01:04 |
| russellb | ok | 01:05 |
| lifeless | fungi: so, about getting ci-overcloud enabled for reals ? | 01:05 |
| locke105 | is that a new project ? | 01:07 |
| fungi | lifeless: we need a nodepool restart for that. it seems not worth delaying the gate with additional resource starvation right now, but maybe when i'm not sitting in an airport and zuul's got new throttling in place | 01:07 |
| fungi | which sounds very soon | 01:07 |
| *** hashar has joined #openstack-infra | 01:08 | |
| fungi | we'll see how exhausted i am when in finally find a hotel room | 01:08 |
| *** blamar has joined #openstack-infra | 01:08 | |
| lifeless | fungi: ok; we're very very very very keen about this, as you can tell | 01:08 |
| *** DennyZhang has quit IRC | 01:09 | |
| locke105 | is ci-overcloud the hyperv CI thing? | 01:10 |
| fungi | locke105: it's the bare metal ci thing | 01:10 |
| locke105 | not related to this https://github.com/cloudbase/ci-overcloud-init-scripts ? | 01:11 |
| fungi | lifeless: i'm very keen on it too, and hope i don't come across otherwise ;) | 01:11 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Include check in fake.yaml. https://review.openstack.org/68295 | 01:12 |
| lifeless | locke105: ci-overcloud.tripleo.org | 01:12 |
| lifeless | locke105: no, not related | 01:12 |
| lifeless | locke105: MS have said they are going to contribute hardware to tripleo's test cloud which would get hyperv check or possibly even gate testing eventually | 01:12 |
| locke105 | i c | 01:13 |
| lifeless | but that looks like cloudbase doing third-party testing | 01:13 |
| locke105 | yeah | 01:13 |
| lifeless | which can't (on current policy anyhow) ever become check-or-gate | 01:13 |
| *** senk1 has joined #openstack-infra | 01:13 | |
| locke105 | ci-overcloud.tripleo.org doesn't seem to resolve to anything? | 01:14 |
| lifeless | locke105: your DNS may be broken | 01:14 |
| locke105 | figures | 01:14 |
| lifeless | locke105: there's no web page there though, if you used a browser to test ;) | 01:15 |
| fungi | ci-overcloud.tripleo.org has address 138.35.77.16 | 01:15 |
| locke105 | comes up on my other machine yeah | 01:15 |
| locke105 | weird | 01:15 |
| fungi | clarkb: i feel silly for asking, but any particular reason why the two tests have the same docstring? just two scenarios to exercise it, or is there something more subtle i'm missing between them besides the function names? | 01:16 |
| fungi | also, boarding. back in a bit once i'm on the plane | 01:16 |
| clarkb | fungi: oh, because I copy pasta'd | 01:16 |
| clarkb | I will fix that | 01:16 |
| *** UtahDave has quit IRC | 01:17 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add rate limiting to dependent pipeline queues https://review.openstack.org/68219 | 01:17 |
| locke105 | mm copy pasta | 01:17 |
| *** mestery has quit IRC | 01:18 | |
| *** oubiwann_ has joined #openstack-infra | 01:19 | |
| jeblair | clarkb, fungi: i need to check out for the day (still sick); i don't feel i'm in a position to do serious code review or make good judgement calls at this point | 01:20 |
| clarkb | jeblair: thats fine, we can let that stew overnight then make big changes tomorrow | 01:20 |
| jeblair | k | 01:20 |
| clarkb | jeblair: I will be restarting jenkins04 shortly though | 01:20 |
| russellb | jeblair: hope you feel better soon | 01:20 |
| russellb | health always more important IMO | 01:20 |
| clarkb | to pick up the newer scp plugin, zaro tested locally and on jenkins-dev so I am fairly confident in it | 01:21 |
| *** hogepodge has quit IRC | 01:23 | |
| sdague | fungi: I support ninja merging russellb's patch | 01:25 |
| sdague | for what it's worth, it should reduce our reset rate | 01:26 |
| clarkb | unless it makes that unittest super unreliable | 01:26 |
| fungi | jeblair: speedy recovery | 01:26 |
| *** pcrews has joined #openstack-infra | 01:26 | |
| *** mrodden has joined #openstack-infra | 01:26 | |
| *** smurugesan has quit IRC | 01:26 | |
| russellb | sdague: should be completely unrelated to the unit test failure (it was a libvirt unit test) | 01:27 |
| sdague | russellb: agreed | 01:27 |
| fungi | russellb: sdague: nova devs are fairly certain that change is unlikely to tickle that test i a way that makes it fail more often? i can cram it in if so | 01:28 |
| *** mrodden1 has joined #openstack-infra | 01:28 | |
| * fungi prepares | 01:28 | |
| sdague | fungi: yes, we're fairly sure | 01:28 |
| russellb | yes | 01:28 |
| * russellb isn't going to bed any time soon | 01:28 | |
| russellb | and will not go to bed until i clean up a mess i made if by chance it blows up | 01:29 |
| russellb | :) | 01:29 |
| sdague | heh | 01:29 |
| * russellb just starting a pot of chili, wooo | 01:29 | |
| sdague | I, on the other hand, am running away from computers for the night. | 01:29 |
| russellb | sdague: enjoy :) | 01:29 |
| sdague | ooo chili :) | 01:29 |
| sdague | night all | 01:29 |
| clarkb | I need to do that soon, but will get jenkins04's scp plugin updated first! jenkins and I love doing battle | 01:30 |
| *** praneshp has quit IRC | 01:30 | |
| clarkb | sdague: tomorrow morning your review on my zuul change would be appreciated | 01:30 |
| clarkb | sdague: you may have an opinion on all o fthe toggles I added | 01:30 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1270654 https://review.openstack.org/68296 | 01:31 |
| russellb | fungi: if you'd rather wait until tomorrow so that *you* don't feel like you have to sit around, that's cool too, totally understand | 01:31 |
| *** mrodden has quit IRC | 01:31 | |
| fungi | russellb: i plan to sit around. i'm waaay too behind on my work anyway | 01:31 |
| fungi | i'm glued to a plane seat for at least another 1.5 hours as well | 01:31 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Storyboard API Interface and basic project management https://review.openstack.org/67582 | 01:31 |
| *** gothicmindfood has quit IRC | 01:32 | |
| russellb | fungi: oh, plane wifi, neat :) | 01:33 |
| fungi | i have no idea how civilization managed to get any work done before in-flight network access | 01:33 |
| *** harlowja has joined #openstack-infra | 01:33 | |
| clarkb | fungi: they made people work in the same office | 01:33 |
| russellb | ha | 01:33 |
| russellb | true story | 01:33 |
| fungi | barbaric | 01:33 |
| fungi | they had to smell one another and everything | 01:34 |
| StevenK | fungi: How is that different from a plane seat, then? | 01:34 |
| *** jorisroovers has quit IRC | 01:34 | |
| fungi | StevenK: airplane seats are so uncomfortable you completely forget about the smell | 01:34 |
| StevenK | Haha | 01:35 |
| fungi | my elbows are hating me | 01:35 |
| *** nosnos has joined #openstack-infra | 01:35 | |
| *** krotscheck has quit IRC | 01:36 | |
| openstackgerrit | A change was merged to openstack-infra/askbot-theme: removed a broken line from the script https://review.openstack.org/68183 | 01:37 |
| StevenK | fungi: I find its easier if you leave your elbows and knees in the overhead locker | 01:37 |
| lifeless | offices are terrible for productivity | 01:38 |
| fungi | i'm surprised my typos don't make it obvious that i type with only my elbows and knees | 01:38 |
| lifeless | and mental health | 01:38 |
| StevenK | lifeless: And actual health during flu season | 01:39 |
| clarkb | 04 is idle, restarting it now | 01:41 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 01:42 |
| *** dcramer__ has quit IRC | 01:43 | |
| clarkb | its up, will watch that scp uploads are happy | 01:43 |
| jhesketh | jeblair: ^ the start of the log storing stuff | 01:43 |
| *** jasondotstar has joined #openstack-infra | 01:44 | |
| *** thuc has quit IRC | 01:44 | |
| *** yaguang has joined #openstack-infra | 01:49 | |
| clarkb | scp on jenkins04 looks happy, a docs job with tons of files uploaded ok | 01:49 |
| clarkb | we can roll that out to the remaining masters over the week and upgrade the unupgraded 2 | 01:50 |
| clarkb | this assumes I will have time >_> | 01:50 |
| *** hashar has quit IRC | 01:55 | |
| dims | nice to hear clarkb | 01:56 |
| *** zhiwei has joined #openstack-infra | 02:00 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Teach periodicCleanup how to do one provider. https://review.openstack.org/68299 | 02:01 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Move cron loading below provider loading. https://review.openstack.org/68300 | 02:01 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Move cron definition out of the inner loop. https://review.openstack.org/68301 | 02:01 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Decouple cron names from config file names. https://review.openstack.org/68302 | 02:01 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Run per-provider cleanup threads. https://review.openstack.org/68303 | 02:01 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Include check in fake.yaml. https://review.openstack.org/68295 | 02:01 |
| lifeless | jeblair: ^ | 02:01 |
| clarkb | fungi: why does the welcome message thing need a service account? | 02:01 |
| lifeless | jeblair: I can move that below the other refactoring if you'd prefer | 02:01 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Add support to run nova-api-metadata as separate binary https://review.openstack.org/68304 | 02:04 |
| fungi | clarkb: so that it can post a comment to a change | 02:06 |
| clarkb | fungi: don't we have accounts that can do that already, or are they named in such a way that would be confusing | 02:06 |
| *** markmcclain has quit IRC | 02:07 | |
| *** sarob has quit IRC | 02:08 | |
| fungi | clarkb: i think the hope was that the display name of the account would serve as a visual clue to reviewers (would appear as something like "Welcome New Contributor!" in bold even in collapsed comment view that way) | 02:08 |
| *** starmer has joined #openstack-infra | 02:08 | |
| clarkb | gotcha | 02:08 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Run tempest-dsvm-postgres-full with nova-api-metadata binary https://review.openstack.org/68305 | 02:08 |
| *** sarob has joined #openstack-infra | 02:08 | |
| fungi | clarkb: gerrit suexec api calls would allow us to do it without authenticating as the account (trivial-rebase does this) but still requires a skeleton account in the db | 02:09 |
| *** dims has quit IRC | 02:11 | |
| *** sarob has quit IRC | 02:13 | |
| *** gyee has quit IRC | 02:13 | |
| *** miqui has quit IRC | 02:16 | |
| jog0 | https://github.com/openstack/openstack/graphs/commit-activity says 35 commits to o/o yesterday (I think thats UTC) | 02:17 |
| jog0 | which is pretty good | 02:17 |
| clarkb | fungi: I am going to walk home and will try to decompress as tomorrow will be a very busy day | 02:18 |
| fungi | clarkb: good call | 02:18 |
| clarkb | I have a thing at hp early (for me) then I plan on digging into my zuul change and the jenkins upgrades | 02:19 |
| *** CaptTofu has joined #openstack-infra | 02:19 | |
| fungi | jog0: put a nickel in the github jar | 02:20 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Teach periodicCleanup how to do one provider. https://review.openstack.org/68299 | 02:21 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Move cron loading below provider loading. https://review.openstack.org/68300 | 02:21 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Move cron definition out of the inner loop. https://review.openstack.org/68301 | 02:21 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Decouple cron names from config file names. https://review.openstack.org/68302 | 02:21 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Run per-provider cleanup threads. https://review.openstack.org/68303 | 02:21 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Use the nonblocking cleanupServer. https://review.openstack.org/68004 | 02:21 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Include check in fake.yaml. https://review.openstack.org/68295 | 02:21 |
| fungi | when that jar is full, we'll use it to buy more git servers | 02:21 |
| jog0 | fungi: heh, they actually have something really nice that we don't yet | 02:21 |
| jog0 | for once | 02:21 |
| lifeless | whats that? | 02:21 |
| jog0 | lifeless: https://github.com/openstack/openstack/graphs/commit-activity pretty pictures | 02:21 |
| *** michchap has quit IRC | 02:24 | |
| *** michchap has joined #openstack-infra | 02:25 | |
| *** senk1 has quit IRC | 02:25 | |
| *** markwash has quit IRC | 02:25 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Run per-provider cleanup threads. https://review.openstack.org/68303 | 02:28 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Include check in fake.yaml. https://review.openstack.org/68295 | 02:28 |
| *** miqui has joined #openstack-infra | 02:28 | |
| *** starmer has quit IRC | 02:29 | |
| *** andrew_plunk has joined #openstack-infra | 02:29 | |
| *** vkozhukalov has joined #openstack-infra | 02:30 | |
| *** dpyzhov has joined #openstack-infra | 02:31 | |
| *** coolsvap_away has quit IRC | 02:33 | |
| *** dcramer__ has joined #openstack-infra | 02:40 | |
| andrew_plunk | hello everyone. I was wondering if anyone could point me to the code used to link launchpad blueprints to gerrit reviews. I am interested in using that information for presenting verbose changelogs between heat builds | 02:40 |
| *** yamahata has quit IRC | 02:42 | |
| clarkb | it is in the openstack-infra/jeepyb project | 02:44 |
| *** AaronGr is now known as aarongr_away | 02:47 | |
| *** CaptTofu has quit IRC | 02:48 | |
| andrew_plunk | awesome clarkb thank you so much | 02:49 |
| andrew_plunk | yeah when I looked at the huge page of openstack-infra repos I did not know where to start | 02:50 |
| *** changbl has quit IRC | 02:50 | |
| *** melwitt1 has quit IRC | 02:51 | |
| andrew_plunk | dang it seems to be using gerrit's database a lot | 02:52 |
| andrew_plunk | I was hoping for rest api or json over ssh | 02:52 |
| andrew_plunk | the launchpad code is very helpful though | 02:53 |
| *** changbl has joined #openstack-infra | 02:54 | |
| *** starmer has joined #openstack-infra | 02:55 | |
| *** andrew_plunk has quit IRC | 03:00 | |
| *** slong_ has joined #openstack-infra | 03:01 | |
| *** slong has quit IRC | 03:01 | |
| *** david-lyle_ has joined #openstack-infra | 03:03 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 03:04 |
| *** changbl has quit IRC | 03:07 | |
| *** changbl has joined #openstack-infra | 03:09 | |
| *** jerryz_ has joined #openstack-infra | 03:13 | |
| *** changbl has quit IRC | 03:15 | |
| *** vipul-away is now known as vipul | 03:17 | |
| *** dstanek has quit IRC | 03:17 | |
| *** changbl has joined #openstack-infra | 03:18 | |
| *** dstanek has joined #openstack-infra | 03:18 | |
| *** thuc has joined #openstack-infra | 03:21 | |
| clarkb | anteaya: the gerrit db use should be limited to adding projects iirc | 03:22 |
| clarkb | oh they left... | 03:22 |
| *** jerryz_ has quit IRC | 03:25 | |
| *** rnirmal has joined #openstack-infra | 03:26 | |
| portante | clarkb: regarding swift gate job timeouts | 03:27 |
| portante | 125 minutes seems kinda long, so perhaps 200s instead? | 03:27 |
| clarkb | 200s is far too short | 03:28 |
| clarkb | it takes longer to install devstack | 03:28 |
| *** ArxCruz has quit IRC | 03:28 | |
| *** vogxn has joined #openstack-infra | 03:29 | |
| portante | the average run time for the functional tests in the last 14 days is always less than 200s, really less than 105s | 03:29 |
| portante | 1050s | 03:29 |
| portante | 150s | 03:29 |
| portante | sorry | 03:29 |
| portante | so add 200s to whatever it typically takes to install devstack for your cap | 03:29 |
| portante | so maybe 25 minutes, 30 minutes at the most? | 03:31 |
| jog0 | so no neutron patch has merged since the 18th https://review.openstack.org/#/q/status:merged+project:openstack/neutron,n,z and neutron patch 53609,10 just caused a massive failure | 03:31 |
| portante | clarkb | 03:31 |
| clarkb | random change sample https://review.openstack.org/#/c/67905/ took over 13 minutes | 03:31 |
| jog0 | should someone snipe it out of gate? | 03:31 |
| clarkb | I would do 25 at the low end 30 is probably safer since the test resources are so variable | 03:31 |
| portante | much better than 125. :) | 03:32 |
| clarkb | ya | 03:32 |
| jog0 | clarkb: ^ | 03:32 |
| clarkb | jog0: fine with me | 03:33 |
| jog0 | clarkb: you want to do it, I am not sure what snipe etiquette is | 03:33 |
| notmyname | portante: I seem to have missed something. why does lowering the timeout make make things better? | 03:34 |
| clarkb | i am dinnering | 03:34 |
| clarkb | i just leave a comment explaining the noop patch | 03:34 |
| jog0 | clarkb: looks like anita already did, but zuul is very far behind | 03:35 |
| jog0 | she uploaded revision 11 at 4:05PM | 03:35 |
| jog0 | doh 1350 events in zuul | 03:35 |
| jog0 | that explains that :/ | 03:36 |
| portante | notmyname: because it lanquished for 125 minutes before the devstack environment killed it | 03:38 |
| *** CaptTofu has joined #openstack-infra | 03:38 | |
| *** CaptTofu has quit IRC | 03:38 | |
| *** CaptTofu has joined #openstack-infra | 03:39 | |
| portante | clarkb: I am done with that Jenkins instance we had for that investigation | 03:39 |
| portante | thanks | 03:39 |
| clarkb | portante: ok, I will try to remember to delete it tomorrow | 03:40 |
| clarkb | though it should cleanup on its own after 24 hours | 03:40 |
| portante | as long as you it is not held up thinking we still need it for investigations, I'm cool | 03:41 |
| clarkb | k | 03:42 |
| *** slong has joined #openstack-infra | 03:43 | |
| *** slong_ has quit IRC | 03:43 | |
| notmyname | portante: did the tests not run? | 03:44 |
| notmyname | portante: sorry, I feel I'm missing some context | 03:44 |
| *** jasondotstar has quit IRC | 03:44 | |
| *** vogxn has left #openstack-infra | 03:45 | |
| clarkb | notmyname: the tests ran then hung for 2 hours because they blocked on something | 03:48 |
| clarkb | it took a long time for them to report back | 03:48 |
| *** hub_cap has joined #openstack-infra | 03:48 | |
| hub_cap | hey krusty krew, if i have a review in the pipeline thats failed for a known reason, can i kill it somehow? ive put reverify bug XXX on it already, is that enough? | 03:49 |
| clarkb | hub_cap: no thats not enough, in this case you can wait or push a new patchset | 03:50 |
| notmyname | clarkb: ok, thanks | 03:50 |
| notmyname | portante: clarkb: any idea what they blocked on? was it an issue with the tests or an issue with the infrastructure? | 03:50 |
| hub_cap | ah its not my patchset (dont want to take over authorship from git's perspective), and its 3/4 done, so maybe i wait... thx clarkb for the fast answer <3 | 03:51 |
| *** praneshp has joined #openstack-infra | 03:51 | |
| *** miqui has quit IRC | 03:52 | |
| *** miqui has joined #openstack-infra | 03:53 | |
| *** harlowja is now known as harlowja_away | 03:54 | |
| *** gokrokve has joined #openstack-infra | 03:55 | |
| *** Hefeweizen has joined #openstack-infra | 03:55 | |
| *** mriedem has quit IRC | 03:56 | |
| *** emagana has quit IRC | 03:57 | |
| notmyname | clarkb: I want to see the console output from a job that doesn't seem to be on jenkins02 anymore. possible? | 04:00 |
| *** jerryz_ has joined #openstack-infra | 04:02 | |
| *** jamielennox is now known as jamielennox|away | 04:05 | |
| *** coolsvap has joined #openstack-infra | 04:08 | |
| *** jerryz_ has quit IRC | 04:12 | |
| portante | notmyname: torgomatic and I discussed it in -swift | 04:13 |
| portante | we don't know exactly what happened | 04:14 |
| notmyname | kk | 04:14 |
| notmyname | looking at the scrollback | 04:14 |
| *** CaptTofu has quit IRC | 04:15 | |
| lifeless | notmyname: the console output wasn't archived? | 04:18 |
| notmyname | lifeless: maybe? it's not on the jenkins box. if it goes somewhere else, I don't know about that | 04:18 |
| lifeless | notmyname: was it a special thing, or a regular gerrit driven test? | 04:19 |
| notmyname | normal thing | 04:19 |
| lifeless | what review #? | 04:19 |
| notmyname | lifeless: https://jenkins02.openstack.org/job/gate-swift-python26/3660/console | 04:19 |
| lifeless | notmyname: do you know the gerrit review # ? | 04:20 |
| notmyname | no, sorry | 04:20 |
| notmyname | lifeless: there is a swift unittest error that very rarely shows up when the test system is under very heavy load. that's the last jenkins job I know that had it, and I wanted to see the error message more clearly and see how hard it would be to fix | 04:21 |
| lifeless | sure | 04:21 |
| lifeless | jenkins is bad at archive though, so we delete everything from it fairly rapidly | 04:22 |
| notmyname | ya, makes sense | 04:22 |
| notmyname | turns out the oldest thing I saw on that box was job 3665, so I just missed it :-) | 04:22 |
| lifeless | so if you look at (say) https://review.openstack.org/#/c/63326/ | 04:22 |
| lifeless | we archive everything from all jobs to http://logs.openstack.org/26/63326/7/ | 04:23 |
| lifeless | the 26 is the last digits of the gerrit id | 04:23 |
| lifeless | 7 is the patch set | 04:23 |
| lifeless | then under http://logs.openstack.org/26/63326/7/check/ we have all the jobs | 04:23 |
| lifeless | and http://logs.openstack.org/26/63326/7/check/gate-swift-python26/ as you'd expect | 04:24 |
| notmyname | hmm..ok. thanks (is that on the wiki anywhere?) | 04:24 |
| lifeless | then http://logs.openstack.org/26/63326/7/check/gate-swift-python26/fb4125a/console.html has the console log from jenkins for that job | 04:24 |
| lifeless | note that there's no way to figure this uot from jenkins job #, because thats transient | 04:24 |
| lifeless | so the primary key, if you will, is gerrit | 04:25 |
| lifeless | no idea if its on the wiki | 04:25 |
| lifeless | I'm fairly sure most of it is described in the CI docs at ci.openstack.org | 04:25 |
| notmyname | lifeless: thanks | 04:25 |
| *** rnirmal has quit IRC | 04:26 | |
| *** gokrokve has quit IRC | 04:28 | |
| *** miqui has quit IRC | 04:29 | |
| *** gokrokve has joined #openstack-infra | 04:29 | |
| *** DennyZhang has joined #openstack-infra | 04:37 | |
| *** vipul is now known as vipul-away | 04:40 | |
| *** dpyzhov has quit IRC | 04:41 | |
| *** vipul-away is now known as vipul | 04:44 | |
| *** nicedice has quit IRC | 04:51 | |
| *** nicedice has joined #openstack-infra | 04:52 | |
| *** gokrokve has quit IRC | 04:53 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove remaining cases of '@message' https://review.openstack.org/67754 | 04:58 |
| *** DinaBelova_ is now known as DinaBelova | 04:58 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Fixed doc build https://review.openstack.org/67376 | 04:59 |
| *** thuc has quit IRC | 05:06 | |
| *** thuc has joined #openstack-infra | 05:06 | |
| hub_cap | is storyboard taking off again? does anyone know whats up w/ that? (mordred?) | 05:08 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 05:08 |
| *** emagana has joined #openstack-infra | 05:08 | |
| hub_cap | i was considering writing an interface to lp's disgusting ui to quell my fits of insanity in dealing w/ it | 05:08 |
| *** chandankumar_ has joined #openstack-infra | 05:08 | |
| *** thuc has quit IRC | 05:11 | |
| *** thuc has joined #openstack-infra | 05:13 | |
| *** yamahata has joined #openstack-infra | 05:13 | |
| *** vipul is now known as vipul-away | 05:15 | |
| *** sarob has joined #openstack-infra | 05:15 | |
| *** emagana has quit IRC | 05:15 | |
| *** nicedice has quit IRC | 05:21 | |
| *** coolsvap is now known as coolsvap_away | 05:23 | |
| *** vogxn has joined #openstack-infra | 05:24 | |
| *** gokrokve has joined #openstack-infra | 05:25 | |
| *** oubiwann_ has quit IRC | 05:27 | |
| *** vipul-away is now known as vipul | 05:32 | |
| *** thuc_ has joined #openstack-infra | 05:35 | |
| *** pcrews has quit IRC | 05:35 | |
| *** thuc has quit IRC | 05:38 | |
| *** vipul is now known as vipul-away | 05:47 | |
| *** thuc_ has quit IRC | 06:01 | |
| *** thuc has joined #openstack-infra | 06:01 | |
| *** DinaBelova is now known as DinaBelova_ | 06:04 | |
| *** starmer_ has joined #openstack-infra | 06:06 | |
| *** thuc has quit IRC | 06:06 | |
| *** starmer has quit IRC | 06:08 | |
| *** blamar has quit IRC | 06:08 | |
| *** vipul-away is now known as vipul | 06:10 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 06:12 |
| *** CaptTofu has joined #openstack-infra | 06:16 | |
| *** coolsvap_away is now known as coolsvap | 06:19 | |
| *** kraman has quit IRC | 06:21 | |
| *** CaptTofu has quit IRC | 06:21 | |
| *** blamar has joined #openstack-infra | 06:23 | |
| *** sarob has quit IRC | 06:30 | |
| *** sarob has joined #openstack-infra | 06:31 | |
| *** sarob has quit IRC | 06:35 | |
| *** mrda has quit IRC | 06:35 | |
| *** afazekas has joined #openstack-infra | 06:35 | |
| *** andreaf has joined #openstack-infra | 06:39 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 06:41 | |
| *** xBsd has joined #openstack-infra | 06:43 | |
| *** nosnos_ has joined #openstack-infra | 06:43 | |
| *** nosnos has quit IRC | 06:43 | |
| *** DennyZhang has quit IRC | 06:44 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 06:50 | |
| *** mrda__ is now known as mrda_away | 06:51 | |
| *** kraman has joined #openstack-infra | 06:52 | |
| *** vkozhukalov has quit IRC | 06:52 | |
| *** kraman has quit IRC | 06:56 | |
| *** sarob has joined #openstack-infra | 07:01 | |
| *** emagana has joined #openstack-infra | 07:07 | |
| *** sarob has quit IRC | 07:09 | |
| *** sarob has joined #openstack-infra | 07:17 | |
| *** yolanda has joined #openstack-infra | 07:18 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 07:18 | |
| *** sarob_ has joined #openstack-infra | 07:19 | |
| *** afazekas has quit IRC | 07:19 | |
| *** vipul is now known as vipul-away | 07:21 | |
| *** sarob has quit IRC | 07:21 | |
| *** kraman has joined #openstack-infra | 07:22 | |
| *** sarob_ has quit IRC | 07:23 | |
| *** vipul-away is now known as vipul | 07:25 | |
| *** odyssey4me has joined #openstack-infra | 07:25 | |
| david-lyle_ | anyone available that can promote this https://review.openstack.org/#/c/68268/ ? licensing issue that needs to make i-2 | 07:27 |
| *** kraman has quit IRC | 07:27 | |
| *** afazekas has joined #openstack-infra | 07:27 | |
| *** vogxn has quit IRC | 07:33 | |
| *** lyxus has joined #openstack-infra | 07:36 | |
| *** yamahata has quit IRC | 07:38 | |
| *** vipul is now known as vipul-away | 07:45 | |
| clarkb | david-lyle: the machine with keys is off. I canpromote in the morning if fungi/jeblair dont brat me to it | 07:50 |
| *** obondarev_ has joined #openstack-infra | 07:53 | |
| *** vipul-away is now known as vipul | 07:54 | |
| *** vogxn has joined #openstack-infra | 07:56 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Configure automatic formatting of README files https://review.openstack.org/60375 | 08:02 |
| *** flaper87|afk is now known as flaper87 | 08:03 | |
| *** mrmartin has joined #openstack-infra | 08:04 | |
| *** mancdaz_away is now known as mancdaz | 08:08 | |
| *** yanghe has joined #openstack-infra | 08:09 | |
| *** xBsd has quit IRC | 08:12 | |
| *** jcoufal has joined #openstack-infra | 08:13 | |
| *** DinaBelova_ is now known as DinaBelova | 08:15 | |
| *** CaptTofu has joined #openstack-infra | 08:17 | |
| *** sarob has joined #openstack-infra | 08:17 | |
| *** yanghe has left #openstack-infra | 08:17 | |
| *** sarob has quit IRC | 08:22 | |
| *** CaptTofu has quit IRC | 08:22 | |
| *** kraman has joined #openstack-infra | 08:23 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 08:23 | |
| *** luqas has joined #openstack-infra | 08:23 | |
| *** kraman has quit IRC | 08:27 | |
| *** vkozhukalov has joined #openstack-infra | 08:30 | |
| *** pblaho has joined #openstack-infra | 08:34 | |
| *** fbo_away is now known as fbo | 08:34 | |
| *** saschpe has quit IRC | 08:41 | |
| *** saschpe has joined #openstack-infra | 08:41 | |
| *** luqas has quit IRC | 08:55 | |
| *** praneshp has quit IRC | 09:00 | |
| *** derekh has joined #openstack-infra | 09:03 | |
| *** yassine has joined #openstack-infra | 09:06 | |
| *** zhiwei has quit IRC | 09:09 | |
| *** jpich has joined #openstack-infra | 09:10 | |
| *** BobBallAWay is now known as BobBall | 09:11 | |
| *** San_D has joined #openstack-infra | 09:11 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Add some dependencies required by toci https://review.openstack.org/67685 | 09:14 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Enable precise-backports on tripleo test nodes https://review.openstack.org/67958 | 09:16 |
| *** San_D has quit IRC | 09:17 | |
| *** sarob has joined #openstack-infra | 09:17 | |
| *** derekh has quit IRC | 09:18 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: API tests for rest https://review.openstack.org/67447 | 09:19 |
| *** sarob has quit IRC | 09:22 | |
| *** kraman has joined #openstack-infra | 09:23 | |
| *** kraman has quit IRC | 09:25 | |
| *** kraman1 has joined #openstack-infra | 09:25 | |
| *** luqas has joined #openstack-infra | 09:27 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_a | 09:27 | |
| *** SergeyLukjanov_a is now known as SergeyLukjanov_ | 09:28 | |
| *** kraman1 has quit IRC | 09:30 | |
| openstackgerrit | Pavel Sedlák proposed a change to openstack-infra/jenkins-job-builder: Add support for Test Stability with Junit https://review.openstack.org/68152 | 09:32 |
| *** bauzas has joined #openstack-infra | 09:36 | |
| *** zhiwei has joined #openstack-infra | 09:36 | |
| *** markmc has joined #openstack-infra | 09:38 | |
| *** jooools has joined #openstack-infra | 09:39 | |
| *** mugsie has quit IRC | 09:39 | |
| openstackgerrit | Zang MingJie proposed a change to openstack-infra/zuul: Use ssh to fetch packs instead of HTTP https://review.openstack.org/67858 | 09:39 |
| *** mugsie has joined #openstack-infra | 09:40 | |
| *** mugsie has quit IRC | 09:40 | |
| *** mugsie has joined #openstack-infra | 09:40 | |
| *** matrohon has quit IRC | 09:40 | |
| *** matrohon has joined #openstack-infra | 09:40 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 09:40 | |
| *** jp_at_hp has joined #openstack-infra | 09:43 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 09:47 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 09:47 | |
| *** mancdaz is now known as mancdaz_away | 09:52 | |
| *** johnthetubaguy has joined #openstack-infra | 09:52 | |
| *** lyxus has quit IRC | 09:53 | |
| *** alexpilotti has joined #openstack-infra | 09:58 | |
| *** DinaBelova is now known as DinaBelova_ | 09:58 | |
| *** jasondotstar has joined #openstack-infra | 10:00 | |
| *** boris-42 has quit IRC | 10:02 | |
| *** boris-42 has joined #openstack-infra | 10:04 | |
| *** vogxn has quit IRC | 10:07 | |
| *** pblaho has quit IRC | 10:12 | |
| anteaya | I have been operating under the belief that reverify had been removed, I just sniped two neutronclient patches https://review.openstack.org/#/c/63986/ and https://review.openstack.org/#/c/63328/ that got back in with reverify bug <bug number> | 10:13 |
| *** sarob has joined #openstack-infra | 10:13 | |
| *** sarob_ has joined #openstack-infra | 10:17 | |
| *** CaptTofu has joined #openstack-infra | 10:18 | |
| *** lyxus has joined #openstack-infra | 10:18 | |
| *** sarob has quit IRC | 10:18 | |
| *** vogxn has joined #openstack-infra | 10:20 | |
| *** sarob_ has quit IRC | 10:22 | |
| *** CaptTofu has quit IRC | 10:22 | |
| *** jhesketh_ has quit IRC | 10:23 | |
| *** kraman has joined #openstack-infra | 10:23 | |
| anteaya | 0 events, 0 results, 112 in the gate, 159 in check | 10:24 |
| anteaya | 44.5 hours for the oldest gate patches | 10:25 |
| *** kraman has quit IRC | 10:27 | |
| AJaeger | anteaya, https://review.openstack.org/#/c/67708/ - this was not approved yet. | 10:34 |
| AJaeger | anteaya, the above is the patch you had in mind with reverify removal | 10:34 |
| *** jesusaurus has quit IRC | 10:35 | |
| *** derekh has joined #openstack-infra | 10:35 | |
| AJaeger | anteaya, interesting way to annotate the commit message with those two ;) | 10:36 |
| *** jesusaurus has joined #openstack-infra | 10:36 | |
| *** jroovers has joined #openstack-infra | 10:38 | |
| anteaya | AJaeger: thanks for pointing me to 67708 | 10:39 |
| anteaya | I had hoped I could retire the big stick - getting tired of doing the cop routine | 10:39 |
| anteaya | not sure what else to do | 10:39 |
| *** max_lobur_afk is now known as max_lobur | 10:39 | |
| AJaeger | Blog about it as reference? Write another email pointing to the blog... | 10:41 |
| AJaeger | Still, too many will not read it, so 67708 seems the best way to do it for now | 10:42 |
| AJaeger | 0 events, 0 results is great - compared to the over 1000 earlier... | 10:43 |
| *** pelix has joined #openstack-infra | 10:56 | |
| *** pblaho has joined #openstack-infra | 10:57 | |
| anteaya | yes | 10:57 |
| anteaya | I am so tired, I haven't blogged in a long time | 10:57 |
| anteaya | I am a few blog posts behind | 10:58 |
| *** yassine_ has joined #openstack-infra | 10:58 | |
| *** yassine has quit IRC | 10:59 | |
| *** jasondotstar has quit IRC | 11:00 | |
| *** mancdaz_away is now known as mancdaz | 11:01 | |
| *** dkranz has quit IRC | 11:01 | |
| pelix | clarkb: wondering if you're happy with the update to https://review.openstack.org/#/c/63579 ? | 11:02 |
| *** dkranz has joined #openstack-infra | 11:02 | |
| AJaeger | anteaya, that's sad, hope you find some time to recover soon | 11:02 |
| *** michchap has quit IRC | 11:06 | |
| anteaya | AJaeger: thanks me too, it is not like I am the only one though | 11:06 |
| *** michchap has joined #openstack-infra | 11:06 | |
| AJaeger | yeah - not sure how much sleep fungi got the last week ;( | 11:07 |
| *** dizquierdo has joined #openstack-infra | 11:08 | |
| *** jroovers has quit IRC | 11:10 | |
| *** gokrokve has quit IRC | 11:11 | |
| *** mrmartin has quit IRC | 11:14 | |
| anteaya | AJaeger: yeah, not much | 11:15 |
| *** sarob has joined #openstack-infra | 11:17 | |
| *** pelix has left #openstack-infra | 11:17 | |
| *** sarob has quit IRC | 11:22 | |
| *** kraman has joined #openstack-infra | 11:23 | |
| *** lcestari has quit IRC | 11:26 | |
| *** kraman has quit IRC | 11:27 | |
| *** derekh has quit IRC | 11:29 | |
| *** lcestari has joined #openstack-infra | 11:29 | |
| *** afazekas_ has joined #openstack-infra | 11:41 | |
| *** jasondotstar has joined #openstack-infra | 11:41 | |
| *** yaguang has quit IRC | 11:41 | |
| *** jroovers has joined #openstack-infra | 11:50 | |
| *** dpyzhov has joined #openstack-infra | 11:51 | |
| *** dstanek has quit IRC | 11:52 | |
| *** boris-42 has quit IRC | 11:53 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack/requirements: Allow use of oslo.messaging 1.3.0a4 from pypi https://review.openstack.org/68040 | 11:54 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1270654 https://review.openstack.org/68296 | 11:54 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1097592 https://review.openstack.org/68282 | 11:54 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1270382 https://review.openstack.org/68280 | 11:54 |
| *** yassine_ has quit IRC | 11:56 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Sort uncategorized fails by time https://review.openstack.org/67761 | 11:56 |
| *** boris-42 has joined #openstack-infra | 11:56 | |
| *** yassine has joined #openstack-infra | 11:57 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1271331 https://review.openstack.org/68270 | 11:57 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1270710 https://review.openstack.org/67764 | 11:57 |
| *** mancdaz is now known as mancdaz_away | 11:58 | |
| *** ArxCruz has joined #openstack-infra | 11:58 | |
| *** mancdaz_away is now known as mancdaz | 11:59 | |
| *** vogxn has quit IRC | 12:00 | |
| *** gokrokve has joined #openstack-infra | 12:06 | |
| *** thuc has joined #openstack-infra | 12:09 | |
| *** b3nt_pin has joined #openstack-infra | 12:11 | |
| *** b3nt_pin is now known as beagles | 12:11 | |
| *** gokrokve has quit IRC | 12:12 | |
| *** thuc has quit IRC | 12:13 | |
| *** jcoufal has quit IRC | 12:14 | |
| *** dims has joined #openstack-infra | 12:14 | |
| *** jcoufal has joined #openstack-infra | 12:15 | |
| *** dpyzhov has quit IRC | 12:16 | |
| *** sarob has joined #openstack-infra | 12:17 | |
| *** CaptTofu has joined #openstack-infra | 12:19 | |
| *** dpyzhov has joined #openstack-infra | 12:19 | |
| *** dstanek has joined #openstack-infra | 12:21 | |
| *** sarob has quit IRC | 12:22 | |
| *** alexpilotti has quit IRC | 12:22 | |
| *** kraman has joined #openstack-infra | 12:23 | |
| *** CaptTofu has quit IRC | 12:23 | |
| *** dpyzhov has quit IRC | 12:26 | |
| *** kraman has quit IRC | 12:27 | |
| *** dstanek has quit IRC | 12:29 | |
| *** julim has joined #openstack-infra | 12:32 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Temporary HACK : Enable UCA https://review.openstack.org/67564 | 12:33 |
| *** dpyzhov has joined #openstack-infra | 12:37 | |
| *** vogxn has joined #openstack-infra | 12:45 | |
| *** smarcet has joined #openstack-infra | 12:46 | |
| *** ociuhandu has quit IRC | 12:49 | |
| *** CaptTofu has joined #openstack-infra | 12:50 | |
| *** emagana has quit IRC | 12:52 | |
| *** CaptTofu has quit IRC | 12:55 | |
| *** mriedem has joined #openstack-infra | 12:57 | |
| *** gokrokve has joined #openstack-infra | 13:01 | |
| *** mancdaz is now known as mancdaz_away | 13:02 | |
| *** coolsvap has quit IRC | 13:03 | |
| *** mancdaz_away is now known as mancdaz | 13:05 | |
| *** dcramer__ has quit IRC | 13:06 | |
| *** gokrokve has quit IRC | 13:06 | |
| *** luqas has quit IRC | 13:08 | |
| *** heyongli has joined #openstack-infra | 13:09 | |
| *** ociuhandu has joined #openstack-infra | 13:10 | |
| *** david-lyle_ has quit IRC | 13:11 | |
| *** ociuhandu has quit IRC | 13:15 | |
| *** sarob has joined #openstack-infra | 13:17 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: update web ui for better sorting https://review.openstack.org/68374 | 13:18 |
| *** xchu has joined #openstack-infra | 13:18 | |
| matel | Ajaeger: Hi, I don't quite understand your comment here: https://review.openstack.org/68363 | 13:20 |
| *** amotoki has joined #openstack-infra | 13:20 | |
| AJaeger | matel, wrong review - I didn't comment on that one | 13:21 |
| *** jasondotstar has quit IRC | 13:21 | |
| matel | Ajaeger: Oh, yeah, I meant this: https://review.openstack.org/68181 | 13:22 |
| AJaeger | matel, do you mean https://review.openstack.org/#/c/68181/ ? | 13:22 |
| AJaeger | Ah, you do ;) | 13:22 |
| *** sarob has quit IRC | 13:22 | |
| *** kraman has joined #openstack-infra | 13:23 | |
| AJaeger | As part of which repository testing do you need this? | 13:23 |
| AJaeger | Is that repo already in projects.txt? | 13:23 |
| matel | So it's a package to be synced from pip, and it's a runtime dependency for nova. | 13:24 |
| matel | We usually install it with devstack: https://github.com/openstack-dev/devstack/blob/master/tools/xen/prepare_guest.sh#L26 | 13:25 |
| AJaeger | matel, ok, then my comment is wrong, wasn't clear to me. | 13:25 |
| *** dizquierdo has quit IRC | 13:26 | |
| *** kraman has quit IRC | 13:27 | |
| matel | Okay, thanks, I will put this info to as a comment. | 13:27 |
| matel | Ah, you already did it, thanks. | 13:28 |
| AJaeger | matel, I've added a comment as well ;) | 13:28 |
| *** rfolco has quit IRC | 13:31 | |
| *** oubiwann_ has joined #openstack-infra | 13:31 | |
| *** rfolco has joined #openstack-infra | 13:33 | |
| *** derekh has joined #openstack-infra | 13:35 | |
| *** jasondotstar has joined #openstack-infra | 13:37 | |
| *** luqas has joined #openstack-infra | 13:38 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: update web ui for better sorting https://review.openstack.org/68374 | 13:38 |
| *** eharney has joined #openstack-infra | 13:38 | |
| *** thomasem has joined #openstack-infra | 13:43 | |
| *** mancdaz is now known as mancdaz_away | 13:44 | |
| *** mancdaz_away is now known as mancdaz | 13:47 | |
| *** zhiwei has quit IRC | 13:49 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: put the fails24 in the right place https://review.openstack.org/68385 | 13:52 |
| *** thuc has joined #openstack-infra | 13:53 | |
| *** prad has joined #openstack-infra | 13:54 | |
| *** yamahata has joined #openstack-infra | 13:55 | |
| *** mestery has joined #openstack-infra | 13:57 | |
| *** mfer has joined #openstack-infra | 13:59 | |
| *** heyongli has quit IRC | 14:01 | |
| *** dprince has joined #openstack-infra | 14:01 | |
| *** gokrokve has joined #openstack-infra | 14:02 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: put the fails24 in the right place https://review.openstack.org/68385 | 14:02 |
| *** yamahata has quit IRC | 14:05 | |
| portante | sdague: should I see a change on the elastic recheck page yet? | 14:05 |
| sdague | I forget how often puppet triggers that update | 14:06 |
| *** boris-42 has quit IRC | 14:06 | |
| *** CaptTofu has joined #openstack-infra | 14:06 | |
| *** gokrokve has quit IRC | 14:06 | |
| portante | k thanks, I assume that fix was to address the "undefined" text on that page? | 14:06 |
| sdague | yep | 14:06 |
| portante | thx | 14:06 |
| sdague | I'm sorting the bugs by fails in last 24 hrs now | 14:07 |
| sdague | and wanted to provide actual numbers | 14:07 |
| sdague | instead of just the graphs | 14:07 |
| portante | that sounds like a good idea | 14:07 |
| portante | we can then play wack-a-mole easier | 14:07 |
| sdague | yep | 14:07 |
| *** dpyzhov has quit IRC | 14:07 | |
| sdague | and not be distracted by things which are mostly fixed | 14:07 |
| sdague | you can see that russellb managed to fully nail - 1270680 | 14:08 |
| sdague | which is great | 14:08 |
| portante | the right data helps one steer the ship away from the icebergs | 14:08 |
| sdague | i like it when stuff flatlines | 14:08 |
| sdague | yep | 14:08 |
| * portante looking | 14:08 | |
| russellb | sdague: yar | 14:09 |
| *** boris-42 has joined #openstack-infra | 14:10 | |
| *** dpyzhov has joined #openstack-infra | 14:10 | |
| portante | nice work russellb | 14:10 |
| anteaya | yes thanks for taking out 1270680 | 14:11 |
| russellb | more to whack though | 14:11 |
| russellb | sdague: if you see another nova one sinking the ship ping me, otherwise i'm playing in zuul | 14:11 |
| *** mriedem has quit IRC | 14:12 | |
| russellb | there's a libvirt unit test we need to stab ... | 14:12 |
| *** matsuhashi has joined #openstack-infra | 14:14 | |
| *** yassine has quit IRC | 14:15 | |
| *** yassine has joined #openstack-infra | 14:15 | |
| sdague | yep, I'm working on the update email right now, it looks like the SSH bug is back on top | 14:16 |
| anteaya | very true | 14:17 |
| anteaya | dang | 14:17 |
| russellb | sdague: ACK thanks | 14:17 |
| *** sarob has joined #openstack-infra | 14:17 | |
| anteaya | I think the ssh bug is the isolated job failure | 14:20 |
| anteaya | I think | 14:20 |
| *** vogxn has quit IRC | 14:20 | |
| *** coolsvap has joined #openstack-infra | 14:22 | |
| *** sarob has quit IRC | 14:22 | |
| *** kraman has joined #openstack-infra | 14:23 | |
| sdague | russellb: Bug 1270654 - test_different_fname_concurrency flakey fail is also something probably worth trying to get someone to look at | 14:24 |
| sdague | it's a nova unit test race | 14:24 |
| russellb | sdague: yeah that's the one i was referring to ... | 14:24 |
| sdague | ok gotach | 14:24 |
| russellb | guess i should jump on it | 14:24 |
| russellb | asking people nicely to work on these bugs didn't really work for a long time :) | 14:25 |
| sdague | it's 5th on the list of non infra fingerprints | 14:25 |
| russellb | OK, worth the time then i thnk | 14:25 |
| sdague | so not huge, but 15 fails in last 24 hrs | 14:25 |
| sdague | across all queues | 14:25 |
| *** jaypipes has quit IRC | 14:27 | |
| *** jaypipes has joined #openstack-infra | 14:28 | |
| *** xchu has quit IRC | 14:29 | |
| *** thuc has quit IRC | 14:30 | |
| *** thuc has joined #openstack-infra | 14:30 | |
| *** alexpilotti has joined #openstack-infra | 14:31 | |
| *** kraman has quit IRC | 14:32 | |
| anteaya | ttx do share when you cut i2 | 14:32 |
| *** mriedem has joined #openstack-infra | 14:33 | |
| *** thuc has quit IRC | 14:35 | |
| *** CaptTofu has quit IRC | 14:35 | |
| *** esker has quit IRC | 14:39 | |
| *** esker has joined #openstack-infra | 14:40 | |
| *** burt1 has joined #openstack-infra | 14:40 | |
| ttx | anteaya: anytime now | 14:41 |
| ttx | but won't do all of them at the same time, so I can do some ordering | 14:41 |
| ttx | based on what's just at the top of the queue | 14:42 |
| *** otherwiseguy has quit IRC | 14:42 | |
| *** dcramer__ has joined #openstack-infra | 14:42 | |
| anteaya | ttx you can cut neutron anytime | 14:43 |
| anteaya | I have removed everything from the gate | 14:43 |
| anteaya | since everything is currently failing isolated jobs | 14:44 |
| anteaya | everything == neutron in the above sentence | 14:44 |
| *** esker has quit IRC | 14:45 | |
| *** dkliban is now known as dkliban_afk | 14:45 | |
| *** matsuhashi has quit IRC | 14:45 | |
| ttx | anteaya: ok, neutron is first then | 14:45 |
| *** jasondotstar has quit IRC | 14:46 | |
| sdague | anteaya: thanks | 14:48 |
| *** yolanda has quit IRC | 14:49 | |
| *** miqui has joined #openstack-infra | 14:49 | |
| anteaya | sure | 14:50 |
| *** sHellUx has joined #openstack-infra | 14:50 | |
| *** ogelbukh has joined #openstack-infra | 14:52 | |
| *** prad has quit IRC | 14:52 | |
| ttx | anteaya: so I should just defer everything not implemented from https://launchpad.net/neutron/+milestone/icehouse-2, right ? | 14:53 |
| anteaya | ah, that is a good question for markmcclain when he awakes | 14:53 |
| *** prad has joined #openstack-infra | 14:53 | |
| anteaya | that is a question I can't answer, I'm just trying to protect the integrity of the gate | 14:54 |
| anteaya | I have no say on the direction of neutron | 14:54 |
| *** yolanda has joined #openstack-infra | 14:55 | |
| ttx | well, deferring -- we can al fix that if that was wrong for some | 14:56 |
| anteaya | ttx okay | 14:56 |
| anteaya | I'll get markmcclain to find you when he arrives unless you find him first | 14:56 |
| *** nosnos_ has quit IRC | 14:57 | |
| *** dkliban_afk is now known as dkliban | 14:58 | |
| derekh | fungi: hi ya, how would we go about enabling ci-overcloud in the production nodepool ? were pretty close to being able to test the tripleo-ci stuff | 14:59 |
| *** oubiwann_ has quit IRC | 14:59 | |
| *** dstanek has joined #openstack-infra | 15:00 | |
| anteaya | derekh: note fungi is in utah this week at a foundation thing, I'm uncertain of his online schedule - I haven't seen him so far this morning | 15:00 |
| derekh | anteaya: ok, thanks | 15:01 |
| derekh | anybody else know how to go about it ?^^ | 15:01 |
| *** alexpilotti has quit IRC | 15:01 | |
| *** gokrokve has joined #openstack-infra | 15:03 | |
| sdague | derekh: right now, with gate situation, a lot of things are blocked up | 15:03 |
| *** odyssey4me has quit IRC | 15:03 | |
| sdague | especially on infra team, so the best way to free up review time for things like that is help on some of the gate reseting bugs | 15:03 |
| derekh | sdague: ok, thanks, will come back when things are a bit calmer, and will see if I can pick on any of the bugs | 15:04 |
| sdague | derekh: thanks! | 15:04 |
| *** CaptTofu has joined #openstack-infra | 15:05 | |
| *** sandywalsh has joined #openstack-infra | 15:06 | |
| *** thuc has joined #openstack-infra | 15:06 | |
| *** dims has quit IRC | 15:08 | |
| *** gokrokve has quit IRC | 15:08 | |
| *** dims has joined #openstack-infra | 15:09 | |
| *** gokrokve has joined #openstack-infra | 15:09 | |
| *** rnirmal has joined #openstack-infra | 15:11 | |
| *** thuc has quit IRC | 15:13 | |
| *** thuc has joined #openstack-infra | 15:14 | |
| *** alexpilotti has joined #openstack-infra | 15:15 | |
| *** sarob has joined #openstack-infra | 15:17 | |
| sdague | well that's a thing to see | 15:18 |
| *** thuc has quit IRC | 15:18 | |
| sdague | currently all the gate fails triggering resets are unit tests | 15:18 |
| sdague | 2 on nova, 1 on swift | 15:19 |
| sdague | 2 on glance | 15:19 |
| *** otherwiseguy has joined #openstack-infra | 15:19 | |
| anteaya | that is odd | 15:20 |
| anteaya | is there some similiarity in the unit test failures, I wonder | 15:21 |
| anteaya | since after isolated jobs, neutron needs to address unit test failures | 15:21 |
| *** kraman has joined #openstack-infra | 15:21 | |
| sdague | not really, russellb is looking at the nova one | 15:21 |
| *** sarob has quit IRC | 15:22 | |
| *** dhellmann is now known as dhellmann_ | 15:22 | |
| anteaya | k | 15:23 |
| *** oubiwann_ has joined #openstack-infra | 15:25 | |
| *** DinaBelova_ is now known as DinaBelova | 15:25 | |
| *** jgrimm has joined #openstack-infra | 15:28 | |
| *** jroovers has quit IRC | 15:28 | |
| anteaya | both glance unit test failures are hitting test_index_with_sort_dir | 15:29 |
| anteaya | and one nova patch is hitting a pep8 which is taking out the nova patch behind it | 15:29 |
| anteaya | but the swift and nova unit test failures appear unique | 15:30 |
| sdague | fungi: 66974 could use promotion when we get a reset | 15:30 |
| sdague | anteaya: yeh, honestly, if we get a reset I might pull all the glance patches out of the queue, that unit test fail looks really regular | 15:30 |
| *** dmsimard has joined #openstack-infra | 15:31 | |
| dmsimard | Woah guys, what happened to jenkins ? It's so fast today :D | 15:31 |
| anteaya | dmsimard: i2 cut off | 15:32 |
| anteaya | the rush to submit patches is off | 15:32 |
| anteaya | sdague: yeah, if the tests are going to prevent them from merging anyway | 15:32 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 15:33 | |
| *** jasond` has joined #openstack-infra | 15:40 | |
| *** luqas has quit IRC | 15:41 | |
| *** senk has joined #openstack-infra | 15:44 | |
| *** starmer_ has quit IRC | 15:45 | |
| *** senk1 has joined #openstack-infra | 15:46 | |
| jasond` | is there anyway to estimate how long it will be before an approved review gets merged? | 15:46 |
| jasond` | is there a queue i can view somewhere? | 15:47 |
| dmsimard | http://status.openstack.org/zuul/ | 15:47 |
| dmsimard | jasond`: ^ | 15:47 |
| jasond` | dmsimard: thank you | 15:48 |
| *** senk has quit IRC | 15:48 | |
| *** emagana has joined #openstack-infra | 15:48 | |
| sdague | just had a reset event about 6 deep, but I don't think we've got anyone around to do a promote after the stuff on top merges | 15:49 |
| *** luqas has joined #openstack-infra | 15:50 | |
| anteaya | mordred? | 15:50 |
| anteaya | he is the only person I think might be around | 15:50 |
| *** emagana has quit IRC | 15:53 | |
| *** senk1 has quit IRC | 15:54 | |
| *** senk has joined #openstack-infra | 15:54 | |
| anteaya | 12 in post, woohoo | 15:57 |
| annegentle | hey infra, just wanted to let you know that some intense Operations Guide updates are happening tomorrow and Friday | 15:57 |
| anteaya | except the swift patch is failing the post job | 15:57 |
| anteaya | awesome, thanks annegentle | 15:57 |
| anteaya | do you expect increased load for zuul? | 15:57 |
| anteaya | as a result of the intense updates? | 15:57 |
| annegentle | I don't think it affects your day-to-day much, nor much for load... but how would I know? | 15:57 |
| notmyname | anteaya: something I need to look at in swift? | 15:58 |
| anteaya | yeah look at the post queue | 15:58 |
| anteaya | the swift patch | 15:58 |
| *** senk has quit IRC | 15:58 | |
| anteaya | 0000000 as an id | 15:58 |
| *** chandankumar_ has quit IRC | 15:58 | |
| anteaya | annegentle: okay thanks for the heads up | 15:58 |
| annegentle | anteaya: it's building in under 3 minutes so I think we're good | 15:59 |
| anteaya | awesome | 16:00 |
| *** chandankumar_ has joined #openstack-infra | 16:01 | |
| sdague | notmyname: there was also another swift unit test fail in the gate | 16:02 |
| sdague | which I think just got reset over | 16:02 |
| *** esker has joined #openstack-infra | 16:02 | |
| *** rcleere has joined #openstack-infra | 16:02 | |
| portante | sdague: can you point me at it? | 16:02 |
| portante | notmyname: I'll review | 16:02 |
| notmyname | portante: thanks | 16:02 |
| sdague | portante: https://jenkins02.openstack.org/job/gate-swift-python27/3268/console | 16:03 |
| portante | sdague: looking ... | 16:03 |
| *** DennyZhang has joined #openstack-infra | 16:04 | |
| *** kmartin has quit IRC | 16:05 | |
| *** gokrokve has quit IRC | 16:05 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_a | 16:07 | |
| *** vkozhukalov has quit IRC | 16:07 | |
| *** CaptTofu has quit IRC | 16:08 | |
| openstackgerrit | Steve Martinelli proposed a change to openstack/requirements: Remove oauth2 requirement https://review.openstack.org/68422 | 16:08 |
| *** SergeyLukjanov_a is now known as SergeyLukjanov_ | 16:08 | |
| *** thouveng has joined #openstack-infra | 16:10 | |
| *** UtahDave has joined #openstack-infra | 16:10 | |
| portante | sdague: how do I see the rest of the logs for the above? | 16:10 |
| *** CaptTofu has joined #openstack-infra | 16:10 | |
| *** dmsimard has left #openstack-infra | 16:10 | |
| sdague | click on the link towards the top | 16:10 |
| sdague | there should be a full log link | 16:11 |
| portante | I meant like syslog and other things, sorry | 16:11 |
| sdague | it's unit tests | 16:11 |
| sdague | I don't think we collect syslog | 16:11 |
| *** pcrews has joined #openstack-infra | 16:11 | |
| portante | k | 16:11 |
| sdague | http://logs.openstack.org/86/66986/3/gate/gate-swift-python27/1e37d7f/ is everything that's artifact collected | 16:12 |
| fungi | derekh: mostly we need to restart nodepool with those new patches and config to test it, which is questionable under recent gate resource starvation. we hope that if we get some changes applied to zuul today we'll be straining the current pool capacity a lot less | 16:12 |
| *** nicedice has joined #openstack-infra | 16:12 | |
| sdague | morning fungi | 16:12 |
| sdague | fungi: 66974 could use promotion when we get a reset | 16:13 |
| sdague | it's the fix horizon needs for i2 on licensing | 16:13 |
| fungi | sdague: ttx also has a release-critical patch which needs to go i at the same time | 16:13 |
| anteaya | how was your flight? | 16:13 |
| sdague | fungi: I think it's the same patch :) | 16:13 |
| portante | sdague: can I hop on that instance to see what else is running? | 16:13 |
| fungi | sdague: oh, he said 68268 | 16:13 |
| ttx | hhm | 16:13 |
| sdague | fungi: oh, listen to ttx | 16:13 |
| sdague | portante: no, the unit tests nodes are all rotated through | 16:13 |
| sdague | portante: this is unit tests, why would it be going to syslog? | 16:14 |
| portante | k, thx | 16:14 |
| anteaya | portante: if the instance is still running fungi is the only one here who can grant access to a running vm | 16:14 |
| ttx | fungi: 68268 confirmed | 16:14 |
| derekh | fungi: ok, so basically we just need to wait for a good time to restart nodepool | 16:14 |
| portante | sdague, I don't know how, just trying to figure out what happened | 16:14 |
| anteaya | if it isn't still running, then the vm has been destroyed, or is being destroyed | 16:14 |
| sdague | portante: ok | 16:14 |
| *** amotoki has quit IRC | 16:14 | |
| fungi | sdague: derekh right, and cross our fingers and hope it's right, since debugging it if it's wrong means project-wide work stoppage | 16:14 |
| fungi | er, not sdague | 16:14 |
| *** senk has joined #openstack-infra | 16:14 | |
| openstackgerrit | ZhiQiang Fan proposed a change to openstack/requirements: Upgrade six to 1.5.2 https://review.openstack.org/68424 | 16:15 |
| *** jcoufal has quit IRC | 16:16 | |
| derekh | fungi: ok thanks | 16:16 |
| fungi | sdague: now that i'm able to get the status page up, i can see we're in a reset anyway | 16:17 |
| fungi | so bumping it now | 16:17 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add multi-project irc support to the bot https://review.openstack.org/67540 | 16:18 |
| openstackgerrit | ZhiQiang Fan proposed a change to openstack/requirements: Ignore egg-info directory https://review.openstack.org/68425 | 16:18 |
| *** senk has quit IRC | 16:19 | |
| *** mrodden1 is now known as mrodden | 16:19 | |
| *** _ruhe is now known as ruhe | 16:20 | |
| *** branen has quit IRC | 16:20 | |
| *** thouveng has quit IRC | 16:21 | |
| *** andreaf has quit IRC | 16:24 | |
| *** wenlock has joined #openstack-infra | 16:24 | |
| *** jasondotstar has joined #openstack-infra | 16:27 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 16:28 | |
| afazekas | Searching for reviewer for these changes: https://review.openstack.org/#/c/65145/ and https://review.openstack.org/#/c/65140/ | 16:29 |
| *** thuc has joined #openstack-infra | 16:29 | |
| *** herndon has joined #openstack-infra | 16:31 | |
| *** dangers_away is now known as dangers | 16:32 | |
| jeblair | fungi, sdague, ttx: morning | 16:32 |
| ttx | jeblair: hi! | 16:33 |
| *** portante has quit IRC | 16:33 | |
| *** gyee has joined #openstack-infra | 16:33 | |
| *** ladquin has joined #openstack-infra | 16:34 | |
| *** emagana has joined #openstack-infra | 16:35 | |
| *** thuc_ has joined #openstack-infra | 16:35 | |
| anteaya | jeblair: morning | 16:38 |
| anteaya | I hope you are feeling a bit better today | 16:38 |
| jeblair | anteaya: not particularly, but thanks. | 16:39 |
| *** thuc has quit IRC | 16:39 | |
| anteaya | :( | 16:39 |
| *** jasondotstar has quit IRC | 16:42 | |
| *** portante has joined #openstack-infra | 16:45 | |
| *** jasondotstar has joined #openstack-infra | 16:46 | |
| *** fifieldt has joined #openstack-infra | 16:47 | |
| *** pballand has joined #openstack-infra | 16:47 | |
| sdague | morning | 16:47 |
| sdague | ok, running to lunch | 16:47 |
| *** gyee_ has joined #openstack-infra | 16:49 | |
| *** mrmartin has joined #openstack-infra | 16:51 | |
| *** gyee has quit IRC | 16:53 | |
| *** portante has quit IRC | 16:53 | |
| *** rakhmerov has joined #openstack-infra | 16:55 | |
| pvo | mordred: you guys doing better on capacity today? | 16:55 |
| *** dpyzhov has quit IRC | 16:58 | |
| mordred | pvo: I'm actually just about to start work on increasing the pool size (We have to add more jenkins masters to be able to handle more slaves - it's a vicious cycle) | 16:59 |
| anteaya | mordred: hello there | 16:59 |
| mordred | mornign anteaya | 16:59 |
| anteaya | don't let me interrupt you | 16:59 |
| *** resker has joined #openstack-infra | 17:00 | |
| jeblair | mordred: steps: write a puppet change; create self-signed certs and put them in hiera; then talk to me about the undocumented process of setting up the initial config | 17:00 |
| anteaya | cacti is showing 5 jenkins masters, do we have more than 5 now? | 17:01 |
| ttx | zuul busy reshuffling right now, most jobs queued | 17:01 |
| anteaya | ttx did you cut neutron? | 17:01 |
| ttx | anteaya: I did | 17:01 |
| anteaya | thanks | 17:01 |
| ttx | page still needs a bit of cleanup | 17:01 |
| *** mrmartin has quit IRC | 17:02 | |
| anteaya | what page? | 17:02 |
| anteaya | sorry I feel I should know and I don't | 17:02 |
| ttx | icehouse-2 milestone page | 17:02 |
| *** esker has quit IRC | 17:02 | |
| ttx | but can't work on it right now | 17:02 |
| ttx | otherwise looks good | 17:02 |
| mordred | jeblair: I'm excited about that | 17:03 |
| *** eharney_ has joined #openstack-infra | 17:03 | |
| mordred | jeblair: the undocumented process part | 17:03 |
| mordred | jeblair: what's our current thinking on jenkins master to slave ratio? | 17:03 |
| jeblair | mordred: 100/1 | 17:03 |
| mordred | ok. so we need 2 more masters ish | 17:04 |
| *** rnirmal has quit IRC | 17:04 | |
| mordred | bumping IAD from 60 to 192 and DFW from 2 to 100 (I believe we leave headroom in nodepool in dfw because of static slaves, yeah?) | 17:04 |
| mordred | so that's potentially 230 new slaves - perhaps I should just do three jenkins masters | 17:05 |
| openstackgerrit | Nadya Privalova proposed a change to openstack/requirements: Fix happybase version https://review.openstack.org/68435 | 17:05 |
| jeblair | mordred: i think our total quota is oto 1000, right? | 17:05 |
| *** eharney__ has joined #openstack-infra | 17:05 | |
| *** senk1 has joined #openstack-infra | 17:05 | |
| *** herndon has quit IRC | 17:05 | |
| anteaya | ttx okay thanks | 17:05 |
| *** pballand has quit IRC | 17:05 | |
| *** jasondotstar has quit IRC | 17:06 | |
| *** eharney has quit IRC | 17:06 | |
| *** portante has joined #openstack-infra | 17:07 | |
| *** pballand has joined #openstack-infra | 17:07 | |
| *** jasondotstar has joined #openstack-infra | 17:07 | |
| *** eharney__ has quit IRC | 17:07 | |
| *** portante has quit IRC | 17:08 | |
| *** eharney_ has quit IRC | 17:08 | |
| mordred | jeblair: I'm not sure | 17:09 |
| mordred | jeblair: also, do we have a "create a self-signed cert" script or a particular way we like to do that? | 17:09 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Increase quota limits in RAX IAD and DFW https://review.openstack.org/68439 | 17:10 |
| jeblair | mordred: oh, no we're at 48 in hpcloud az1 and 3? | 17:10 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 17:10 | |
| jeblair | mordred: so that's 732 as the new total quota, so 3 masters would be good | 17:11 |
| *** eharney has joined #openstack-infra | 17:11 | |
| *** branen has joined #openstack-infra | 17:11 | |
| Alex_Gaynor | To what extent are more servers going to help? It was my impression that the biggest issue was frequent gate resets? | 17:12 |
| anteaya | it can allow for a faster turn around on check tests | 17:13 |
| jeblair | mordred: root@ci-puppetmaster:~/certs might help; it's designed to create csr's but also creates self-signed certs | 17:13 |
| mordred | Alex_Gaynor: we're also seeing resource starvation because of the frequent resets of deep queue - so the check queue is starving | 17:13 |
| Alex_Gaynor | mordred: ah, makes sense | 17:13 |
| *** dizquierdo has joined #openstack-infra | 17:13 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add three new jenkins servers https://review.openstack.org/68442 | 17:13 |
| Alex_Gaynor | mordred: is there anything I could be doing to help ya'll out with this? | 17:13 |
| *** bauzas has quit IRC | 17:14 | |
| mordred | Alex_Gaynor: I think the jenkins setup is on me - jeblair, anything Alex_Gaynor can do to help your end of things? | 17:14 |
| *** eharney has quit IRC | 17:15 | |
| *** ociuhandu has joined #openstack-infra | 17:16 | |
| clarkb | morning, I am not actually here yet | 17:16 |
| *** dpyzhov has joined #openstack-infra | 17:16 | |
| jeblair | Alex_Gaynor: not that i can think of right now, thanks | 17:16 |
| *** pblaho has quit IRC | 17:17 | |
| clarkb | jeblair: re https://review.openstack.org/#/c/68219/7 do you want to address the comments while I am distracted and possibly rip out the extra toggles? I will get to it in about an hour and a half if not | 17:17 |
| *** aarongr_away is now known as AaronGr | 17:17 | |
| *** senk1 has quit IRC | 17:17 | |
| *** senk has joined #openstack-infra | 17:17 | |
| *** rnirmal has joined #openstack-infra | 17:17 | |
| jeblair | pvo: what are the rax api rate limits? i ran 'nova rate-limits' but get an empty response | 17:18 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 17:18 | |
| anteaya | morning clarkb who isn't actually here | 17:18 |
| clarkb | jeblair: also, I realized that the way I set the window in the layout means the value will reset whenever the layout is reloaded which we probably don't want | 17:18 |
| jeblair | lifeless: hpcloud rate limits: http://paste.openstack.org/show/61688/ | 17:19 |
| clarkb | jeblair: not sure if we care about tackling that in the first patch | 17:19 |
| clarkb | as reseting like that may be desireable as we use it initially | 17:19 |
| pvo | jeblair: interesting. You should get something... let me check mine. | 17:19 |
| pvo | jeblair: what region? Mine are listing from dfw. | 17:20 |
| *** rakhmerov has quit IRC | 17:21 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Fix CLI args for welcome-message https://review.openstack.org/66623 | 17:22 |
| fifieldt | fungi, thanks for making the key :) | 17:23 |
| jeblair | pvo: all 3 of dfw/ord/iad | 17:23 |
| jeblair | pvo: let me try a new version of novaclient | 17:23 |
| *** chandankumar_ has quit IRC | 17:24 | |
| *** eharney has joined #openstack-infra | 17:24 | |
| ArxCruz | jeblair: hey, I would like to make a change in o-infra/config to make puppet.conf server configurable, today is hardcoded ci-openstack.openstack.org | 17:24 |
| ArxCruz | it will be necessary a lot of changes | 17:25 |
| ArxCruz | what's the best approach? several patches or one at once ? | 17:25 |
| ArxCruz | basically all puppet recipes that uses openstack_project::base will have to be changed | 17:25 |
| *** jp_at_hp has quit IRC | 17:26 | |
| jeblair | pvo: still no joy with latest novaclient 2.15.0; all 3 regions for 'openstackjenkins' account | 17:27 |
| anteaya | ArxCruz: right now jeblair and mordred are working on spinning up 3 new jenkinses | 17:27 |
| *** pcrews has quit IRC | 17:27 | |
| ArxCruz | anteaya: oh, okay I can wait :) | 17:27 |
| anteaya | so there might be a slight pause in service while that work takes place | 17:27 |
| anteaya | ArxCruz: awesome, thank you | 17:27 |
| jeblair | ArxCruz: probably several changes | 17:27 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 17:28 | |
| clarkb | jeblair: mordred: before yo uget too far along spinning up more jenkinses, I am not sure gaer will be able to handle when they all restart together | 17:29 |
| clarkb | when we moved zuul geard and the gearman plugin were having trouble when all 5 jenkinses attempted to register together | 17:29 |
| clarkb | we had to start geard then start jenkinses individually before it would work | 17:29 |
| jeblair | clarkb: can you elaborate on 'trouble'? | 17:29 |
| clarkb | I submitted a bug to openstack-ci about it | 17:29 |
| *** chandankumar_ has joined #openstack-infra | 17:29 | |
| clarkb | jeblair: geard would throw an exception running status because some job key would be invalid | 17:30 |
| *** marun has joined #openstack-infra | 17:30 | |
| clarkb | and that would kill geard | 17:30 |
| jeblair | clarkb: ah yeah, that's like a one line fix to geard | 17:30 |
| *** yamahata has joined #openstack-infra | 17:31 | |
| clarkb | ok | 17:31 |
| *** gothicmindfood has joined #openstack-infra | 17:32 | |
| *** browne has joined #openstack-infra | 17:32 | |
| *** pballand has quit IRC | 17:32 | |
| jasond` | i keep seeing the gate tests under "openstack/heat 67971,2" go from SUCCESS to queued on the zuul status page. is that supposed to happen? | 17:34 |
| mordred | jeblair, clarkb: you're saying I should launch them one at a time perhaps? | 17:34 |
| jeblair | mordred: no, we'll start them one at a time | 17:34 |
| mordred | jeblair: also, hiera changes are in | 17:34 |
| mordred | jeblair: which I believe means I should be able to land https://review.openstack.org/68442 | 17:36 |
| mordred | and start launching nodes | 17:37 |
| jeblair | mordred: almost there | 17:37 |
| anteaya | jasond`: yes, that means that something above it is causing a reset | 17:37 |
| jasond` | anteaya: oh ok, thanks | 17:37 |
| anteaya | np | 17:37 |
| *** DennyZhang has quit IRC | 17:37 | |
| anteaya | mordred: you have a suggestion on how to expand your patch, do you want to do it, or shall I? | 17:38 |
| mordred | anteaya: aroo? | 17:39 |
| anteaya | to include cacti and nodepool | 17:39 |
| anteaya | I can make the change if you need to focus | 17:39 |
| mordred | anteaya: I appreciate any and all help | 17:39 |
| anteaya | so I will make the change to your 68442 patch | 17:40 |
| mordred | anteaya: thank you | 17:40 |
| *** jpich has quit IRC | 17:40 | |
| *** mancdaz is now known as mancdaz_away | 17:40 | |
| mordred | oh wow. we have to parameterize that down in the manifests and not just in site.pp. | 17:40 |
| *** blamar_ has joined #openstack-infra | 17:40 | |
| mordred | anteaya: actually - perhaps we should add nodepool as a follow on patch | 17:41 |
| jeblair | def | 17:41 |
| anteaya | okay just cacti.pp and jenkins-log-client.yaml added to 68422 | 17:42 |
| anteaya | correct? | 17:42 |
| *** markwash has joined #openstack-infra | 17:43 | |
| mordred | yeah | 17:43 |
| *** blamar has quit IRC | 17:43 | |
| *** blamar_ is now known as blamar | 17:43 | |
| anteaya | k | 17:43 |
| jeblair | clarkb: comments/questions about tests in https://review.openstack.org/#/c/68219/ | 17:43 |
| fungi | mordred: if you're looking for another change to test the manage-projects script, i think https://review.openstack.org/61954 ready to go for this point (once the current scramble is settled, reading scrollback now to see what's broken) | 17:44 |
| mordred | fungi: okie. thanks | 17:44 |
| *** luqas has quit IRC | 17:45 | |
| jeblair | mordred: actually: https://review.openstack.org/#/c/65191/ | 17:46 |
| jeblair | mordred: that lists all the places it's safe to add the new jenkins servers in the first change | 17:46 |
| *** dizquierdo has quit IRC | 17:46 | |
| mordred | anteaya: ^^ if you're updating it | 17:47 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Add three new jenkins servers https://review.openstack.org/68442 | 17:47 |
| anteaya | yeah, that is the one I followed, thanks to Roman for linking it | 17:48 |
| *** jooools has quit IRC | 17:48 | |
| anteaya | jeblair: can I add the nodepool file then as well? | 17:48 |
| anteaya | clarkb ^ | 17:48 |
| jeblair | anteaya: yes | 17:48 |
| anteaya | okay | 17:49 |
| jeblair | anteaya: but just the bits that were in that change | 17:49 |
| mordred | sigh. I seem to have left my power adapter at the office and my battery is now dying | 17:49 |
| mordred | I'm confused | 17:49 |
| mordred | I'll be back online in a little while | 17:49 |
| *** markmcclain has joined #openstack-infra | 17:50 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Add three new jenkins servers https://review.openstack.org/68442 | 17:50 |
| *** rakhmerov has joined #openstack-infra | 17:51 | |
| anteaya | I can't comment on my own patch | 17:51 |
| anteaya | please folks make sure the syntax is consistent, even if you don't know puppet | 17:52 |
| markwash | I'm seeing the whole "externally hosted" problem again and again with pip install / tox, for packages psutils and pysendfile. seemingly related is a complete inability to install oslo.messaging into tox's venv. anybody got any tips for me? | 17:53 |
| *** DennyZhang has joined #openstack-infra | 17:55 | |
| wenlock | markwash: i wonder if this patch would help https://review.openstack.org/#/c/51425/ | 17:55 |
| *** melwitt has joined #openstack-infra | 17:56 | |
| markwash | wenlock: hmm, sorry I'm actually seeing it locally rather than in the gate; it seems like that patch is more geared towards the gate? also maybe I'm in the wrong channel :-) | 17:56 |
| *** hashar has joined #openstack-infra | 17:56 | |
| wenlock | ahhh :D | 17:57 |
| *** mrodden has quit IRC | 17:58 | |
| markwash | it seems like maybe if I could somehow constrain the version of pip that tox is using I could fix things, but googling did not lead me to any conclusions there | 17:59 |
| *** fifieldt has quit IRC | 18:00 | |
| *** mriedem has quit IRC | 18:03 | |
| *** gyee_ has quit IRC | 18:04 | |
| *** vkozhukalov has joined #openstack-infra | 18:05 | |
| *** jasond` has quit IRC | 18:06 | |
| *** pballand has joined #openstack-infra | 18:06 | |
| *** CaptTofu has quit IRC | 18:10 | |
| *** praneshp has joined #openstack-infra | 18:11 | |
| *** mrodden has joined #openstack-infra | 18:11 | |
| *** otherwiseguy has quit IRC | 18:12 | |
| *** morganfainberg|z is now known as morganfainberg | 18:15 | |
| anteaya | I have to jet for a meeting | 18:15 |
| anteaya | go mordred | 18:15 |
| anteaya | back later | 18:15 |
| *** derekh has quit IRC | 18:16 | |
| *** gokrokve has joined #openstack-infra | 18:17 | |
| *** CaptTofu has joined #openstack-infra | 18:17 | |
| *** gsamfira has quit IRC | 18:18 | |
| *** marun has quit IRC | 18:18 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 18:21 | |
| *** hashar has quit IRC | 18:21 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:22 | |
| *** hashar has joined #openstack-infra | 18:23 | |
| pvo | jeblair: re: rate limits... ok. let me see what I can dig up. | 18:23 |
| jeblair | pvo: cool, thx, let me know if you need more info | 18:23 |
| *** dpyzhov has quit IRC | 18:23 | |
| *** rakhmerov has quit IRC | 18:23 | |
| SumitNaiksatam | hi...does any one here know how to reset the #openstack-meeting meet bot? | 18:23 |
| SumitNaiksatam | i am trying to to end a meeting, but the meet bot is not picking up | 18:23 |
| jeblair | SumitNaiksatam: just try to start a meeting | 18:24 |
| SumitNaiksatam | jeblair: ah ok | 18:24 |
| SumitNaiksatam | jeblair: didn't work | 18:25 |
| *** rakhmerov has joined #openstack-infra | 18:26 | |
| jeblair | hrm i'll look | 18:26 |
| SumitNaiksatam | jeblair: it wouldn't let me start the new meeting, but its not letting me end either | 18:26 |
| jeblair | SumitNaiksatam: oh i see, your nick changed. | 18:27 |
| SumitNaiksatam | jeblair: yeah, i just noticed that as well | 18:27 |
| jeblair | SumitNaiksatam: try changing it back to 'SumitNaiksatam_' and then #endmeeting | 18:27 |
| SumitNaiksatam | ok | 18:27 |
| *** senk has quit IRC | 18:27 | |
| *** fbo is now known as fbo_away | 18:28 | |
| SumitNaiksatam | jeblair: done, i think i got dc'ed and that created the problem, did not realize that the nick changed | 18:29 |
| jeblair | SumitNaiksatam: cool, glad that worked; if it didn't i or another infra core could have used super-user privs to end it, but it's good you could do it yourself | 18:29 |
| SumitNaiksatam | jeblair: thanks, was able to end it | 18:30 |
| *** pcrews has joined #openstack-infra | 18:30 | |
| *** gothicmindfood has quit IRC | 18:31 | |
| *** portante has joined #openstack-infra | 18:32 | |
| *** gyee has joined #openstack-infra | 18:32 | |
| *** harlowja_away is now known as harlowja | 18:32 | |
| zaro | jeblair: i'm looking for ideas on how to test gerritbot with review-dev.o.o | 18:33 |
| *** krotscheck has joined #openstack-infra | 18:33 | |
| zaro | jeblair: would i need to run my own gerritbot that sends updates to my own channels? | 18:35 |
| portante | sdague: the swift test failure is very odd, still trying to figure out what caused it, have not been able to reproduce it locally yet | 18:35 |
| jeblair | zaro: yes, it shouldn't be too hard; i usually just have it join a test irc channel that only i have joined | 18:37 |
| krotscheck | Hey everyone. We've got a bit of a patch backlog on storyboard, does anyone have time amidst the crazyness to look at some of these? 67520, 67729, 67731, 65017 | 18:38 |
| krotscheck | (They're all infra patches | 18:38 |
| krotscheck | (I mean config) | 18:38 |
| lifeless | jeblair: ok, so now to see if we use PUT at all; if we do we need to limit to 1/6 seconds otherwise 4/6 seconds | 18:39 |
| *** ruhe is now known as _ruhe | 18:39 | |
| *** praneshp has quit IRC | 18:40 | |
| zaro | krotscheck: i can take a look, but only able to give +1. | 18:40 |
| krotscheck | zaro: More Eyeballs == better | 18:41 |
| jeblair | lifeless: actual rax ratelimits for our account are unknown; pvo is looking into it | 18:41 |
| *** dpyzhov has joined #openstack-infra | 18:41 | |
| lifeless | jeblair: ack, thanks | 18:42 |
| jeblair | lifeless: also, i would like to revert the logging change because i think it is verbose and the default rate change because i think 1/sec is a good default and config is the place for further tuning | 18:42 |
| krotscheck | zaro: Two of those can be summed up with "Hey let's just have tox run our build for us" | 18:42 |
| mgagne | jeblair: I want to approve this change but XML changed: https://review.openstack.org/#/c/64610/ | 18:43 |
| *** senk1 has joined #openstack-infra | 18:43 | |
| lifeless | jeblair: the rate change - sure, but the logging change - I would really like to get diagnostics there at some stage; can you suggest a better mechanism? | 18:43 |
| *** praneshp has joined #openstack-infra | 18:43 | |
| jeblair | lifeless: well, by design we should pretty much always be hitting the rate limit (unless the api call itself takes longer than the interval) | 18:47 |
| mordred | jeblair: back | 18:47 |
| jeblair | lifeless: so i expect it should be emitting a constant stream of log lines, one per provider per interval | 18:47 |
| jeblair | lifeless: and i'm not sure what it tells you, other than (a) whether the program runs faster than the permitted rate, and (b) whether the api calls themselves take longer than the interval | 18:48 |
| *** mriedem has joined #openstack-infra | 18:48 | |
| jeblair | mgagne: that change looks harmless; zaro do you agree? | 18:50 |
| zaro | mgagne, jeblair: yes, i agree. | 18:50 |
| mgagne | jeblair: side effect is that jenkins servers will be "hammered" by update requests | 18:50 |
| jeblair | mgagne: that should be okay. as long as the jobs themselves don't break | 18:51 |
| mgagne | jeblair: although no changes should be made | 18:51 |
| mgagne | jeblair: ok, will approve | 18:52 |
| jeblair | mgagne: cool, thx | 18:52 |
| *** jcoufal has joined #openstack-infra | 18:52 | |
| * mgagne puts on his cowboy hat | 18:52 | |
| jeblair | yeeehaw! | 18:52 |
| *** marun has joined #openstack-infra | 18:53 | |
| *** burt1 has quit IRC | 18:54 | |
| *** asadoughi has joined #openstack-infra | 18:54 | |
| *** julim has quit IRC | 18:54 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations https://review.openstack.org/66414 | 18:55 |
| *** julim has joined #openstack-infra | 18:57 | |
| *** sarob has joined #openstack-infra | 18:57 | |
| *** DinaBelova is now known as DinaBelova_ | 18:57 | |
| *** sarob has quit IRC | 18:58 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add three new jenkins servers https://review.openstack.org/68442 | 18:58 |
| *** nati_ueno has joined #openstack-infra | 19:00 | |
| mordred | jeblair: ok. I'm going to start launching jenkins servers | 19:00 |
| *** markmcclain has quit IRC | 19:01 | |
| *** hashar has quit IRC | 19:01 | |
| *** markmcclain has joined #openstack-infra | 19:01 | |
| *** _ruhe is now known as ruhe | 19:01 | |
| *** rakhmerov has quit IRC | 19:03 | |
| *** markmcclain has quit IRC | 19:03 | |
| *** sarob has joined #openstack-infra | 19:03 | |
| *** markmcclain has joined #openstack-infra | 19:03 | |
| openstackgerrit | Matt Ray proposed a change to openstack-infra/config: Chef style testing enablement and minor speed cleanup starting with checks https://review.openstack.org/67964 | 19:05 |
| *** browne has left #openstack-infra | 19:05 | |
| *** jroovers has joined #openstack-infra | 19:06 | |
| *** rakhmerov has joined #openstack-infra | 19:07 | |
| zaro | clarkb, sdague : disabling gerrit drafts is a pending change.. https://gerrit-review.googlesource.com/#/c/53947 | 19:08 |
| clarkb | zaro: I saw that :( also you don't want to prevent anonymous users from pushing drafts, they can't push drafts anyways | 19:09 |
| clarkb | zaro: you want to prevent registered users | 19:09 |
| *** rakhmerov has quit IRC | 19:10 | |
| clarkb | jeblair: ok I am around properly now | 19:11 |
| clarkb | jeblair: re double releases in the zuul tset, I think I was operating under the assumption that only the first job matching the regex would be released, I see that isn't true. I will fix that | 19:12 |
| *** elasticio has joined #openstack-infra | 19:14 | |
| *** gsamfira has joined #openstack-infra | 19:14 | |
| jeblair | clarkb: cool, thought that might be the case. the *-merge releases are doubled in some cases because of dependent changes, where the merge for change B won't start until the merge for change A finishes (so there's a settle between them to allow that to happen) | 19:15 |
| ttx | fungi, jeblair: would be nice to get https://review.openstack.org/#/c/68135/ in icehouse-2 as well | 19:15 |
| ttx | fungi, jeblair: so if there is a way to bump it at the top at next reset... would be nice | 19:15 |
| ttx | fungi, jeblair: hmm, unless milestone-proposed changes go in a separate queue ? in which case we could propose the change there | 19:17 |
| ttx | but IIRC that's not the case | 19:17 |
| jeblair | ttx: is it a gate-fixing bug or a security fix? | 19:17 |
| jeblair | ttx: no, not the case | 19:17 |
| ttx | jeblair: it's just a milestone-critical fix | 19:17 |
| zaro | clarkb: anonynmous covers all users, so seems ok to me. | 19:17 |
| *** vipul is now known as vipul-away | 19:17 | |
| clarkb | zaro: it covers logged in users too? weird | 19:18 |
| zaro | clarkb: yes, i believe so. | 19:18 |
| *** max_lobur is now known as max_lobur_afk | 19:19 | |
| *** yassine has quit IRC | 19:19 | |
| jeblair | ttx: to prevent our becoming actual human gatekeepers, we adopted a policy of only promiting gate-fixing bugs or security fixes; do you feel strongly enough about this change to ask us to consider widening that policy? | 19:19 |
| ttx | jeblair: we made an exception to that rule already for the licensing issue in horizon (currently at top of queue) | 19:20 |
| jeblair | alas we are human | 19:20 |
| ttx | that said, that should have been tested | 19:20 |
| ttx | so we can keep it in regular queue | 19:21 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix multibyte character problem https://review.openstack.org/64610 | 19:21 |
| ttx | jeblair: I would add legal issues to that policy above | 19:21 |
| fungi | i'll apologize for promoting the licensing patch. it was legal | 19:21 |
| fungi | heh | 19:21 |
| fungi | it felt like a justifiable grey area | 19:22 |
| jeblair | i'll buy legal. :) | 19:22 |
| clarkb | I think legal issues belong in that list | 19:22 |
| jeblair | done. gate-fixing bugs, security fixes, legal issues. :) | 19:22 |
| fungi | since technically horizon was misrepresenting licensing of software in the published repository with out it | 19:22 |
| *** vipul-away is now known as vipul | 19:23 | |
| fungi | yay, we're back to being inhuman again then | 19:23 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Remove mock API interfaces from storyboard. https://review.openstack.org/68464 | 19:27 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Add tests for Alembic migrations https://review.openstack.org/66414 | 19:27 |
| fungi | speaking of the horizon change... it looks like jenkins failed to run a couple of the jobs on it and zuul re-queued them... i don't suppose we have any easy way to get that change moving again until the next gate reset further down causes a bunch of check jobs to get new nodes before zuul will circle back around to getting nodes on those two jobs still lacking them? | 19:27 |
| clarkb | fungi: nope, we have no way of modifying the geard queues iirc | 19:28 |
| jeblair | tis true | 19:28 |
| fungi | figured | 19:28 |
| *** johnthetubaguy has quit IRC | 19:28 | |
| fungi | just wanted to be 100% sure before i told ttx to give up and eat lunch | 19:28 |
| jeblair | what we should do is have zuul re-enqueue those jobs with a high priority, but that is sadly non-trivial | 19:28 |
| jeblair | in the mean time, remind yourself (and ttx) to be happy that it didn't already kick it out because jenkins has in fact already failed once! | 19:29 |
| *** sHellUx has quit IRC | 19:29 | |
| *** pballand has quit IRC | 19:30 | |
| fungi | oh, already mentioned that, yes | 19:30 |
| fungi | cold comfort is better than none at all ;) | 19:30 |
| ttx | I feel so happy | 19:30 |
| clarkb | jeblair: fungi: I think we should decide on two things about my zuul change. First should I remove the _type and _factor flags? and second is it a problem that in the inital patch any config reload of the layout.yaml will resset the window value? | 19:31 |
| ttx | I just feel like the whole queue is going to wait for that now | 19:31 |
| jesusaurus | clarkb: sdague: reading the gate update email got me thinking about the (rare) need for ninja merges. whats the feasibility of modifying zuul's scheduler to use a priority queue: an "everyone" priority, and a "ninja" priority? that would allow certain changesets to be put at the front of the queue when zuul is recalculating the gate after a failure. im just not sure where/how to set the priority | 19:31 |
| fungi | clarkb: i think those are slight overengineering, but i can also see us wanting to experiment with them without needing a zuul restart | 19:32 |
| fungi | so i'm in favor of keeping them | 19:32 |
| clarkb | jesusaurus: jeblair hinted at that not too long ago when fungi asked about promoting specific changes. it is non trivial | 19:32 |
| jeblair | ttx: fungi: you could re-promote it. it would start everything over again but may be faster. | 19:32 |
| clarkb | jesusaurus: we do have gearman priority, but right now can only set that on a pipeline level not a change level | 19:33 |
| ttx | jeblair: yeah, I feel like the whole queue will be blocked for longer if we don't | 19:33 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add multi-project irc support to the bot https://review.openstack.org/67540 | 19:33 |
| fungi | jeblair: well, if we repromote it, everything currently in the check queue will still get all remaining nodes, and we're likely to have a reset behind that change fairly soon anyway (statistically speaking) | 19:34 |
| jeblair | fungi: (and as long as you're doing it you could probably accidentally promote the other m-p change behind it) | 19:34 |
| fungi | that's an option | 19:34 |
| jeblair | fungi: okay, i've not been paying enough attention to guage the odds on that | 19:34 |
| jeblair | fungi: your call :) | 19:34 |
| *** sHellUx has joined #openstack-infra | 19:35 | |
| hub_cap | jeblair: bet on black | 19:35 |
| *** beagles is now known as beagles_brb | 19:35 | |
| *** sHellUx has quit IRC | 19:35 | |
| *** pballand has joined #openstack-infra | 19:35 | |
| jeblair | hub_cap: there are 3 blacks but they're too far down. bad bet i think. | 19:35 |
| sdague | jesusaurus: I don't know, I do think it would be interesting | 19:36 |
| mgagne | zaro: if you aren't too busy, would you mind rebasing your JJB changes against master? | 19:36 |
| sdague | jesusaurus: though in this case a ninja merge is skip the gate entirely, and just merge directly | 19:36 |
| *** marun has quit IRC | 19:36 | |
| jesusaurus | im not familiar enough with zuuls scheduler to know how many moving pieces would be affected | 19:36 |
| *** praneshp has quit IRC | 19:37 | |
| jeblair | jesusaurus: having it wait for a reset was sufficiently hard that i punted on that when writing the manual promote script | 19:38 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 19:38 |
| *** sarob has quit IRC | 19:38 | |
| *** sarob has joined #openstack-infra | 19:38 | |
| jeblair | clarkb: i'm fine keeping the extra knobs and the proviso that the window is reset on reload for now | 19:39 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add multi-project irc support to the bot https://review.openstack.org/67540 | 19:39 |
| *** sHellUx has joined #openstack-infra | 19:39 | |
| *** jcoufal has quit IRC | 19:41 | |
| *** thuc has joined #openstack-infra | 19:41 | |
| clarkb | jeblair: ok, new patchset should arrive shortly | 19:41 |
| *** hashar has joined #openstack-infra | 19:42 | |
| *** sarob has quit IRC | 19:43 | |
| *** sHellUx has joined #openstack-infra | 19:44 | |
| mordred | jeblair: dammit. I made jenkins05 just fine. then I was still in rax-nova and not ci-rax-nova when I made 06. making 07 in the right place, will go back and fix 06 in a second | 19:44 |
| *** thuc_ has quit IRC | 19:44 | |
| jeblair | mordred: ok. i use screen and one window per host and do them all at once when i'm doing multiple hosts | 19:45 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add rate limiting to dependent pipeline queues https://review.openstack.org/68219 | 19:45 |
| clarkb | how does that look? | 19:45 |
| *** vkozhukalov has quit IRC | 19:45 | |
| *** thuc has quit IRC | 19:46 | |
| *** otherwiseguy has joined #openstack-infra | 19:46 | |
| mordred | jeblair: I was actually doing one set so I could be methodical about it :) | 19:47 |
| *** sHellUx has joined #openstack-infra | 19:47 | |
| mordred | jeblair: what's the difference between rdns create and record-create/ | 19:49 |
| mordred | ? | 19:49 |
| jeblair | mordred: reverse and forward dns | 19:49 |
| mordred | oh. duh | 19:49 |
| mordred | nevermind | 19:49 |
| mordred | yup | 19:49 |
| mordred | sometimes asking the question is all you need to do | 19:50 |
| *** gsamfira has quit IRC | 19:51 | |
| *** markmcclain has quit IRC | 19:53 | |
| *** rfolco has quit IRC | 19:53 | |
| *** DennyZha` has joined #openstack-infra | 19:54 | |
| *** sHellUx_ has joined #openstack-infra | 19:55 | |
| *** _david_ has joined #openstack-infra | 19:55 | |
| *** DennyZhang has quit IRC | 19:56 | |
| *** hogepodge has joined #openstack-infra | 19:57 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 19:57 | |
| *** sHellUx has quit IRC | 19:57 | |
| jeblair | clarkb: i approved your change since there were only trivial changes since the last patchset. if someone objects, i assume there's a little bit of time still left before it merges. :) | 19:59 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Introducing basic REST API https://review.openstack.org/63118 | 19:59 |
| jeblair | when it merges, i think we shut down and deploy | 19:59 |
| anteaya | back | 19:59 |
| mordred | jeblair: all three new jenkins servers created | 19:59 |
| clarkb | jeblair: ok, I will plan to be around for that | 19:59 |
| mordred | jeblair: I would like to read your literature on your super secret setup sauce | 20:00 |
| *** DennyZha` has quit IRC | 20:00 | |
| mordred | jeblair: at your convenience | 20:00 |
| *** sHellUx_ has quit IRC | 20:00 | |
| jeblair | clarkb: maybe lunch now then? i'm getting ready to | 20:00 |
| *** ladquin is now known as ladquin_afk | 20:00 | |
| clarkb | jeblair: sure | 20:00 |
| jeblair | mordred: left out a step; need to restart iptables on the hosts that list the new jenkins servers in their firewalls | 20:00 |
| jeblair | mordred: (since they weren't in dns until now) | 20:01 |
| mordred | jeblair: ok. I'll go do that, then ping you back | 20:01 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Load projects from yaml file https://review.openstack.org/66280 | 20:02 |
| *** obondarev_ has quit IRC | 20:02 | |
| _david_ | jeblair, mordred clarkb Can you consider giving a talk on Gerrit UC? https://groups.google.com/forum/#!topic/repo-discuss/5T0E-GG3Pag | 20:02 |
| *** smurugesan has joined #openstack-infra | 20:03 | |
| *** DinaBelova_ is now known as DinaBelova | 20:04 | |
| *** MarkAtwood has joined #openstack-infra | 20:05 | |
| *** yamahata has quit IRC | 20:05 | |
| jeblair | mordred: when you're ready see jenkins04:~corvus/README | 20:05 |
| jeblair | _david_: i bet one of us could manage that | 20:06 |
| openstackgerrit | James Slagle proposed a change to openstack-infra/release-tools: Added ignore to additional egg-info files https://review.openstack.org/68471 | 20:07 |
| _david_ | jeblair, That would be really really, great. We are biggest (public) Gerrit installation site and we should make our voice in the community | 20:07 |
| *** david-lyle_ has joined #openstack-infra | 20:08 | |
| _david_ | jeblair, And you and zaro gave a talk on JUC in the past, though, why not to explain Gerrit maintainer, that having 800 active contributors put some special requirements, | 20:08 |
| anteaya | like 8 jenkinses? | 20:09 |
| jeblair | _david_: yeah, i think we missed the cfp deadline in previous years | 20:09 |
| jeblair | so it's very good of you to remind us | 20:09 |
| _david_ | yes, we should do that, we want to make upstream contribution process a bit easier, what ? ;-) | 20:10 |
| *** derekh has joined #openstack-infra | 20:11 | |
| openstackgerrit | Eric Guo proposed a change to openstack/requirements: Sort global-requirements https://review.openstack.org/64943 | 20:11 |
| _david_ | I am giving a talk about decentralized CI infrastructure with LibreOffice-Gerrit-buildbot-plugin, so another reason to attend to meet you guys ;-) | 20:12 |
| *** senk1 has quit IRC | 20:15 | |
| *** lcestari has quit IRC | 20:17 | |
| *** vipul is now known as vipul-away | 20:19 | |
| *** vipul-away is now known as vipul | 20:19 | |
| lifeless | fungi: so, about getting ci-overcloud enabled ;) | 20:20 |
| *** burt1 has joined #openstack-infra | 20:20 | |
| *** fbo_away is now known as fbo | 20:21 | |
| *** afazekas_ has quit IRC | 20:22 | |
| *** beagles_brb is now known as beagles | 20:24 | |
| *** CaptTofu has quit IRC | 20:26 | |
| *** markmcclain has joined #openstack-infra | 20:26 | |
| *** dizquierdo has joined #openstack-infra | 20:27 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 20:28 | |
| *** elasticio has quit IRC | 20:28 | |
| *** sarob has joined #openstack-infra | 20:30 | |
| *** sarob has quit IRC | 20:31 | |
| *** markmc has quit IRC | 20:32 | |
| *** praneshp has joined #openstack-infra | 20:33 | |
| *** CaptTofu has joined #openstack-infra | 20:33 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add per job classification rate to uncategorized.html https://review.openstack.org/68478 | 20:34 |
| *** vipul is now known as vipul-away | 20:34 | |
| *** mrda_away is now known as mrda | 20:35 | |
| sdague | oh, clarkb https://review.openstack.org/#/c/67591/ | 20:35 |
| sdague | to get the uncategorized hit list out there | 20:36 |
| jog0 | sdague: ^^ shows per job stats | 20:36 |
| jog0 | sdague: 99% for gate-tempest-dsvm-full | 20:36 |
| sdague | jog0: yep, I just saw your new patch, haven't looked at it deeply | 20:36 |
| jog0 | no problem | 20:36 |
| jog0 | I am just impressed with some of the stats | 20:37 |
| *** senk1 has joined #openstack-infra | 20:37 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add rate limiting to dependent pipeline queues https://review.openstack.org/68219 | 20:37 |
| jog0 | the jenkins interrupt a job and mark as failure bug is messing up those numbers | 20:37 |
| *** DinaBelova is now known as DinaBelova_ | 20:37 | |
| sdague | jog0: yeh, we need to get those not to be marked as fails in ES | 20:38 |
| jog0 | sdague: yup | 20:38 |
| jog0 | and grenade jobs need more then console.html | 20:38 |
| sdague | jog0: yeh, can you work up the ES patch for that one? | 20:38 |
| sdague | I know you added some other ES stuff | 20:38 |
| anteaya | yay for 68219 | 20:39 |
| sdague | yep | 20:39 |
| jog0 | sdague: not sure what best way to get the logs/new/screen-n into ES | 20:39 |
| clarkb | sdague: looking | 20:39 |
| jog0 | I think we want it to be mapped to logs/screen-n* | 20:39 |
| lyxus | I already posted this on -dev but might be for infra actually. For tempest, I was wondering 1) Is tempest supposed to be passing 100% of the test from the trunk 2) Does the standard use the openvswitch plugin | 20:39 |
| jog0 | so the queries all work without any changes | 20:39 |
| *** senk1 has quit IRC | 20:40 | |
| *** senk has joined #openstack-infra | 20:40 | |
| sdague | jog0: we might need another piece of metadata then | 20:40 |
| clarkb | lyxus: 1) yes, 2) no the 'standard' if there is one is nova network | 20:40 |
| jog0 | sdague: such as? | 20:40 |
| sdague | so we can tell new vs. old n-cpu | 20:40 |
| sdague | because it might be important | 20:41 |
| sdague | otherwise it will look like all of it is the same file to ES | 20:41 |
| clarkb | sdague: jog0: I think the filename value should have the logs/new/screen-n value | 20:41 |
| clarkb | then you can glob the /new/ out when you do serach | 20:41 |
| sdague | clarkb: so the problem with that is existing matches wont | 20:41 |
| clarkb | sdague: right you would need a glob | 20:41 |
| sdague | clarkb: so what if we just made grenade_branch another piece of metadata | 20:42 |
| sdague | so the filename stayed logs/screen-n | 20:42 |
| clarkb | sdague: we can do that too, it just complicates the mapping from file to indexed data | 20:42 |
| sdague | but we'd be able to facet | 20:42 |
| sdague | clarkb: it does, but I think it simplifies use dramatically | 20:42 |
| clarkb | k | 20:42 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 20:42 | |
| clarkb | or, and this might be crazy | 20:43 |
| clarkb | what if we combine the logs in the jobs and just have a single log file | 20:43 |
| clarkb | then there isn't old vs new just two chunks of data in a file | 20:43 |
| sdague | hmmm... | 20:43 |
| jeblair | clarkb: ready for surgery? | 20:43 |
| clarkb | jeblair: give me a couple minutes | 20:44 |
| sdague | that might work | 20:44 |
| clarkb | sdague: I am not entirely sold on that idea yet (it just occured to me) | 20:44 |
| sdague | it would at least be a start | 20:44 |
| jeblair | clarkb: np, back to my hacking hole | 20:44 |
| sdague | if we decided that we hated it later we could change the indexer | 20:44 |
| *** ociuhandu has quit IRC | 20:44 | |
| sdague | jog0: thoughts? | 20:44 |
| lyxus | clarkb, I want to test the impact of my plugin. So i should just git clone devstack and run tempest and it should be 100% | 20:45 |
| *** rcleere has quit IRC | 20:46 | |
| jog0 | sdague: I like the idea of single log file | 20:47 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 20:47 | |
| clarkb | lyxus: yes tempest is expected to pass at least the tests marked gate or whateve rthe tag is | 20:47 |
| jog0 | it wouldn't be to hard to figure out which is old nad new too | 20:48 |
| jog0 | I think | 20:48 |
| anteaya | mikal please don't recheck any neutron patches | 20:48 |
| anteaya | they won't pass check until isolated jobs are fixed | 20:48 |
| clarkb | jeblair: ok, I have caffeine, ping me when ready | 20:48 |
| anteaya | I am posting comments to neutron patches in check informing people to stop rechecking until isolated jobs are fixed | 20:48 |
| anteaya | and inviting them to help fix the issue | 20:49 |
| jeblair | clarkb: hi | 20:49 |
| jog0 | yeah everytime a linenumber is printed we see the path (new vs old) | 20:49 |
| jog0 | although right now we mainly care about the new logs | 20:50 |
| jog0 | being thats what we run tempest against | 20:50 |
| clarkb | jeblair: ohai | 20:50 |
| *** sarob has joined #openstack-infra | 20:50 | |
| jeblair | zuul is updated. i think it should just be a matter of stopping and starting. probably a one-person job; i'll do it | 20:51 |
| clarkb | jeblair: ok | 20:51 |
| *** vipul-away is now known as vipul | 20:51 | |
| clarkb | I will tail the log and watch the window sie | 20:51 |
| clarkb | *window size | 20:51 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 20:51 | |
| jeblair | #status alert Zuul is about to restart for an upgrade; changes will be re-enqueued | 20:51 |
| openstackstatus | NOTICE: Zuul is about to restart for an upgrade; changes will be re-enqueued | 20:51 |
| *** ChanServ changes topic to "Zuul is about to restart for an upgrade; changes will be re-enqueued" | 20:51 | |
| jeblair | starting zuul | 20:54 |
| *** burt1 has quit IRC | 20:55 | |
| jeblair | clarkb: what's the default window behavior? | 20:55 |
| *** sarob has quit IRC | 20:55 | |
| clarkb | jeblair: 20 incrementing up exponential down to 3 iirc | 20:56 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 20:57 | |
| clarkb | yup | 20:57 |
| clarkb | in scheduler.py | 20:57 |
| mtreinish | clarkb: actually we don't have a tag for the gate tests it just runs the api, scenario, cli, and thirdparty test dirs | 20:57 |
| *** DennyZhang has joined #openstack-infra | 20:59 | |
| *** sarob has joined #openstack-infra | 20:59 | |
| jeblair | clarkb: i'm deleting all nodepool nodes that were marked used during the downtime | 21:00 |
| clarkb | ok | 21:00 |
| *** CaptTofu has quit IRC | 21:01 | |
| jeblair | clarkb: fwiw it looks like #21 is not launching jobs | 21:01 |
| clarkb | cool :) | 21:01 |
| jeblair | and check jobs enqueued ahead of it seem to be | 21:02 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 21:02 | |
| *** yolanda has quit IRC | 21:02 | |
| sdague | oh, right, enqueue times all reset on zuul restart | 21:03 |
| clarkb | sdague: ssshh! :) | 21:04 |
| sdague | :P | 21:04 |
| anteaya | when does the gate queue limiter go into effect? | 21:04 |
| *** smarcet has left #openstack-infra | 21:05 | |
| clarkb | sdague: your puppet change lgtm for the es stuff | 21:05 |
| clarkb | anteaya: right now | 21:05 |
| anteaya | I'm seeing 106 in the gate queue | 21:05 |
| sdague | clarkb: great | 21:05 |
| clarkb | anteaya: note things still queue up and show in status, but won't have jobs started for things outside the window | 21:05 |
| jeblair | anteaya: 67641,2 is the first change outside the window | 21:05 |
| * anteaya looks again | 21:05 | |
| sdague | the sooner we can get it in, the better, then we can farm out categorizing bugs easier | 21:06 |
| jeblair | (a ui indication of the window is a possible future enhancement) | 21:06 |
| *** DennyZhang has quit IRC | 21:06 | |
| anteaya | oh okay, I will wait until jobs start | 21:06 |
| *** DennyZhang has joined #openstack-infra | 21:07 | |
| jeblair | anteaya: wait for what? | 21:08 |
| anteaya | so right now, other than you telling me, I have no way of knowing which changes are inside the window and which aren't | 21:08 |
| *** DennyZhang has quit IRC | 21:08 | |
| anteaya | since some changes inside the window don't have jobs started | 21:08 |
| anteaya | waiting for jobs to start | 21:08 |
| *** DennyZhang has joined #openstack-infra | 21:08 | |
| jeblair | anteaya: all changes inside the window are running jobs | 21:08 |
| *** hashar is now known as hasharMeeting | 21:09 | |
| anteaya | okay, I have expanded them to see that | 21:09 |
| jeblair | anteaya: the first 20 changes are all running some number of jobs. changes 21-106 are running no jobs | 21:09 |
| *** senk has quit IRC | 21:09 | |
| anteaya | yes, expanding them shows me that | 21:10 |
| anteaya | thanks | 21:10 |
| anteaya | well done | 21:10 |
| *** senk has joined #openstack-infra | 21:10 | |
| jeblair | #status ok | 21:11 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 21:11 | |
| *** senk1 has joined #openstack-infra | 21:13 | |
| *** mrmartin has joined #openstack-infra | 21:13 | |
| jeblair | clarkb: check queue seems to be proceeding well past 20 | 21:14 |
| clarkb | jeblair: perfect | 21:14 |
| *** senk has quit IRC | 21:15 | |
| *** kraman has quit IRC | 21:16 | |
| *** CaptTofu has joined #openstack-infra | 21:19 | |
| *** dprince has quit IRC | 21:19 | |
| jeblair | ttx: your license change is 26 min out | 21:20 |
| *** jooools has joined #openstack-infra | 21:21 | |
| *** kraman has joined #openstack-infra | 21:21 | |
| *** whoops has joined #openstack-infra | 21:22 | |
| *** thuc has joined #openstack-infra | 21:23 | |
| *** ruhe is now known as _ruhe | 21:23 | |
| *** thuc_ has joined #openstack-infra | 21:23 | |
| *** thuc has quit IRC | 21:26 | |
| portante | clarkb, sdague: do you guys know how this patch got reentered into the gate: https://review.openstack.org/66986 | 21:27 |
| portante | ? | 21:27 |
| portante | I need to file a bug for the failure mode so we can track it | 21:27 |
| portante | it is really weird | 21:27 |
| *** jhesketh_ has joined #openstack-infra | 21:28 | |
| *** enqae has joined #openstack-infra | 21:28 | |
| *** enqae has quit IRC | 21:28 | |
| anteaya | it was possibly reenqueued on the zuul restart | 21:29 |
| *** jroovers has quit IRC | 21:30 | |
| portante | anteaya: okay, thanks | 21:30 |
| clarkb | portante: yeah probably happeend when jeblair reenqueued things after the zuul restart | 21:33 |
| clarkb | we just got a gate reset | 21:33 |
| clarkb | woot and it was cheap. The js renders the subway graph oddly but I can live with that for now | 21:34 |
| anteaya | yay for a cheap gate reset | 21:34 |
| *** DennyZhang has quit IRC | 21:34 | |
| portante | did the ratelimit the gate jobs land yet? | 21:35 |
| *** jasondotstar has quit IRC | 21:35 | |
| *** DennyZhang has joined #openstack-infra | 21:35 | |
| anteaya | portante: yes | 21:35 |
| lifeless | sdague: so unit tests are dependent on other projects a lot of the time - e.g. client libraries | 21:35 |
| portante | nice | 21:35 |
| lifeless | sdague: I'm worried your slimmed down gate proposal will let lots of needle-threads through | 21:35 |
| anteaya | if you expand by default in the zuul status page, the gate patches in the limited window will have jobs running and those outside will not | 21:36 |
| anteaya | about the first 20 patches right now | 21:36 |
| anteaya | I think the algorithm is flexible | 21:36 |
| *** svarnau has joined #openstack-infra | 21:37 | |
| jhesketh | Morning | 21:37 |
| anteaya | morning jhesketh | 21:37 |
| anteaya | I haven't seen mattoliverau lately | 21:37 |
| anteaya | did we scare him away? | 21:37 |
| jhesketh | He's busy moving homes | 21:37 |
| anteaya | ah | 21:37 |
| jhesketh | I'm sure he'll be back in action soon :-) | 21:38 |
| anteaya | that's okay then | 21:38 |
| anteaya | sure, from a happy new home | 21:38 |
| jhesketh | yes, but also terrible internet ;-) | 21:38 |
| anteaya | noooo | 21:38 |
| anteaya | close to a coffee shop with great internet? | 21:38 |
| jhesketh | I think he's quite central so I'd be surprised if not | 21:38 |
| anteaya | cool | 21:39 |
| lifeless | fungi: ping ? | 21:39 |
| jhesketh | anything I can do to help you guys out? | 21:39 |
| fungi | okay, so refreshing myself on the current state of what i missed whilst abandoning you all morning... we have several additional jenkins masters leveraging the additional rackspace quota, and clarkb's dynamic zuul throttle mechanism is in place now? | 21:39 |
| clarkb | anteaya: you can more clearly see the window break in the UI now because the js isn't rendering it properly :) | 21:39 |
| clarkb | fungi: I don't think the new jenkinses are fully up | 21:39 |
| fungi | lifeless: yes? | 21:39 |
| anteaya | clarkb: awesome | 21:39 |
| lifeless | fungi: hi! so - I understand derekh chatted w/you about enabled ci-overcloud | 21:40 |
| lifeless | s/enabled/enabling/ | 21:40 |
| clarkb | but dynamic throttling is in, you can see the window break in the subway graphs bad rendering | 21:40 |
| anteaya | last I heard mordred and jeblair were still working on configuring those | 21:40 |
| fungi | lifeless: is about to ask me whether we're in a good state to take nodepool offline and play with it to see if the stack of new changes will stuff | 21:40 |
| fungi | er, will break stuff | 21:40 |
| clarkb | I am reasonably happy with were we are right now | 21:41 |
| lifeless | fungi: no, just to enable ci-overcloud | 21:41 |
| anteaya | jhesketh: ummm, nothing right atm, but do celebrate the configuring of 3 new jenkinses for additional rax nodes | 21:41 |
| lifeless | fungi: which the previously landed changes support | 21:41 |
| anteaya | plus rate limiting on the gate queue | 21:41 |
| anteaya | yay | 21:41 |
| jhesketh | anteaya: another 3? Are we up to 8? | 21:42 |
| lifeless | fungi: I would suggest landing your handle-flavor-lookup-errors toot | 21:42 |
| lifeless | s/toot/too/ | 21:42 |
| anteaya | jhesketh: as soon as they are configured we will be up to 8 | 21:42 |
| jhesketh | nice stuff :-) | 21:42 |
| clarkb | jhesketh: ya that is about 100 slaves per master | 21:42 |
| anteaya | it is exciting, yes | 21:42 |
| fungi | lifeless: right. i need to see if i got recommendations on my horrible flavor list try/except patch | 21:43 |
| * fungi checks | 21:43 | |
| lifeless | you did from me | 21:43 |
| fungi | excellent--thank you | 21:43 |
| *** mrmartin has quit IRC | 21:44 | |
| *** DennyZhang has quit IRC | 21:45 | |
| *** DennyZhang has joined #openstack-infra | 21:46 | |
| mordred | clarkb: I'm still working on new jenkinses | 21:47 |
| *** _david_ has quit IRC | 21:47 | |
| fungi | lifeless: we should probably also include https://review.openstack.org/67684 | 21:48 |
| *** rakhmerov has joined #openstack-infra | 21:49 | |
| fungi | that one's just an outright bugfix of copy-paste errors | 21:49 |
| lifeless | yes; if we're including config changes then https://review.openstack.org/#/c/67958/2 and https://review.openstack.org/#/c/67685/ too please (though we can work around those in tripleo-ci) | 21:49 |
| *** sarob has quit IRC | 21:50 | |
| fungi | also, we have a separate issue in nodepool which i haven't even tracked down yet... i think we may need a longer ssh timeout/retry for image building--we're often struggling to build new images in hpcloud, particularly in az2 | 21:50 |
| *** sarob has joined #openstack-infra | 21:50 | |
| jeblair | fungi: oh again? we raised it twice :( | 21:50 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Send swift upload instructions to workers https://review.openstack.org/68297 | 21:51 |
| fungi | we seem to be able to build servers, but maybe image building has a separate timeout? | 21:52 |
| fungi | my brief look through the image.log suggested that it was throwing paramiko exceptions between nova boot step and puppeting | 21:52 |
| mattoliverau | anteaya: I've been moving into a new house :) | 21:53 |
| *** DennyZhang has quit IRC | 21:53 | |
| *** DennyZhang has joined #openstack-infra | 21:53 | |
| *** DennyZhang has quit IRC | 21:54 | |
| mattoliverau | anteaya: Still am really, got boxes everywhere. but back at work today. | 21:54 |
| *** DennyZhang has joined #openstack-infra | 21:54 | |
| fungi | if you look at nodepool image-list, the devstack-precise image for hpcloud-az2 is 547 hours old, for example | 21:55 |
| *** sarob has quit IRC | 21:55 | |
| lifeless | pleia2: were you bringing up a fedora image defn patch? | 21:55 |
| jeblair | clarkb: i think we may have some ui issues to work through. :) | 21:56 |
| clarkb | jeblair: yup :) but it shows you where the window break is | 21:56 |
| jeblair | clarkb: why is 67641,2 out of line? | 21:56 |
| jeblair | clarkb: did the window shrink? | 21:56 |
| clarkb | jeblair: no it did not shrink that was the old one on the outside of the window | 21:57 |
| jeblair | clarkb: oh, ok. so the ui weirdness is just extra weird. :) | 21:57 |
| clarkb | ya I believe so | 21:57 |
| jeblair | clarkb: it might be good for changes outside the window to be disconnected. | 21:58 |
| jeblair | clarkb: (and probably get a different color dot) | 21:58 |
| clarkb | list of things we need to add to this new zuul feature: reporting in status.json, rpc command to set values, then we can prevent layout.yaml updates from changing the window on the fly, UI updates | 21:58 |
| clarkb | jeblair: wfm, I was thinking a pair of two differently shaded backgrounds, but disconnecting the usbway and changing station type fits into the existing ui well | 21:59 |
| zaro | krotscheck: question about 67731. were does the version info go in the packaging? | 22:00 |
| mordred | jeblair: hey! I think that something is weird, because home for jenkins is /home/jenkins, not /var/lib/jenkins | 22:02 |
| mordred | clarkb: ^^ you know anything about that? | 22:02 |
| jeblair | mordred: i think we found that something changed in the jenkins packaging... | 22:02 |
| clarkb | oh jenkins | 22:02 |
| jeblair | like we expected it to create a user/homedir or something and it didn't | 22:02 |
| jeblair | i can't remember the way we fixed it | 22:02 |
| krotscheck | zaro: Nowhere, yet. I suspect that'll vary based on project, since some will be packaged as python modules and others will be packaged as tarballs. | 22:03 |
| jeblair | did we remove the user and let puppet re-create it? | 22:03 |
| jeblair | clarkb: fungi: ^ ? | 22:03 |
| clarkb | jeblair: mordred: I think puppet creates the user for us | 22:03 |
| clarkb | perhaps the package needs to require the user? | 22:03 |
| fungi | jeblair: yes, i believe we did | 22:03 |
| * mordred tries taht | 22:03 | |
| jeblair | mordred: that also reminds me that we probably want to manually install the deb | 22:03 |
| jeblair | the jenkins debs for the same version we use on other hosts | 22:03 |
| *** derekh has quit IRC | 22:03 | |
| clarkb | yes dpkg -i the version you want, also grab the scp plugin from jenkins04 | 22:04 |
| mordred | clarkb: which version do I want? | 22:04 |
| *** ArxCruz has quit IRC | 22:05 | |
| jeblair | lifeless: i've heard back and i think we can assume a very high number for our rax limit for the moment (exact numbers forthcoming); maybe we should just set it to the same as hpcloud for now | 22:05 |
| mordred | fungi: nope. I deleted the user, re-ran puppet, still ended up with user in wrong place | 22:05 |
| lifeless | jeblair: so hpcloud is actually lower than the default rax limit | 22:05 |
| mordred | perhaps delte user then dpkg -i ? | 22:05 |
| lifeless | jeblair: now i'm off of phones I can put up a patch for that, sec | 22:05 |
| fungi | mordred: worth a try | 22:06 |
| jeblair | lifeless: what's the rax default again if you have it offhand? | 22:06 |
| lifeless | http://docs.rackspace.com/loadbalancers/api/v1.0/clb-devguide/content/Rate_Limits-d1e821.html | 22:06 |
| *** svarnau has quit IRC | 22:06 | |
| *** david-lyle has quit IRC | 22:06 | |
| *** wenlock has quit IRC | 22:06 | |
| *** svarnau has joined #openstack-infra | 22:06 | |
| lifeless | jeblair: | 22:06 |
| *** david-lyle has joined #openstack-infra | 22:06 | |
| lifeless | DELETE /v1.0/* ^/1.0/.* 50/minute is the lowest figure | 22:07 |
| *** wenlock has joined #openstack-infra | 22:07 | |
| lifeless | or 5/6 of a second | 22:07 |
| *** sarob has joined #openstack-infra | 22:07 | |
| clarkb | mordred: of jenkins 1.543 | 22:07 |
| clarkb | mordred: of scp plugin the scp.jpi on jenkins04 | 22:08 |
| lifeless | jeblair: compare to | 22:08 |
| lifeless | | PUT | /{suburi} | 10 | 10 | MINUTE | 2014-01-22T17:16:52Z | | 22:08 |
| zaro | krotscheck: i created a script that extracts git version info to use if you like. it would be good to append the git sha to the tarball or put it in a version file or something. here's the script.. http://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/maven-properties.sh | 22:09 |
| lifeless | 1/5th the rate | 22:09 |
| fungi | so i think we're hitting ssh timeouts on hpcloud image builds semi-often, but the stale hpcloud-az2.devstack-precise is something else entirely. the image.log files dating back to the start of the month show that it only tried to build it once in january (on the 9th) | 22:10 |
| krotscheck | zaro: Funny- clarkb JUST pointed me at that. | 22:10 |
| fungi | and died with a ruby timeout | 22:10 |
| *** ArxCruz has joined #openstack-infra | 22:10 | |
| jeblair | lifeless: do we put? | 22:11 |
| zaro | krotscheck: the file is inappropriately named. i should change it.. | 22:11 |
| lifeless | jeblair: I was going to try and figure that out | 22:11 |
| jeblair | lifeless: also, slightly different table: http://docs.rackspace.com/servers/api/v2/cs-devguide/content/Rate_Limits-d1e862.html | 22:11 |
| *** DennyZhang has quit IRC | 22:11 | |
| lifeless | jeblair: I haven't yet; but since we only have one metric | 22:11 |
| jeblair | lifeless: we should be able to assume 1000+/min get/post for rax | 22:12 |
| lifeless | ok | 22:12 |
| clarkb | jeblair: fungi: window was bumped to 22 then cut to 11. I was reading code and I think it may still be running in a loop using the 22 slice. will have to see where the split is. But overall seems to be happy | 22:12 |
| jeblair | lifeless: (i don't necessarily want to abuse that, which is why i was suggesting matching hp, but if if that ends up being _really_ slow, i guess we'll make up something reasonable sounding. :) | 22:13 |
| fungi | clarkb: excellent | 22:13 |
| clarkb | hrm doesn't seem to have discarded builds for things in the queue yet | 22:13 |
| krotscheck | zaro: Cool. As soon as I get this integration suite done I'll add a patch to inject the version into the build env | 22:13 |
| lifeless | jeblair: btw how would you feel about s/rate/interval/ - the code is interval based not rate based | 22:13 |
| *** jooools has quit IRC | 22:13 | |
| lifeless | the unit is seconds, not actions/second | 22:13 |
| lifeless | so inverted | 22:13 |
| jeblair | lifeless: fine by me | 22:13 |
| lifeless | jeblair: I'll do that when we're not in super busy mode | 22:14 |
| clarkb | jeblair: can you look at the gate queue and see if that looks funny to you? | 22:14 |
| jeblair | lifeless: k thx | 22:14 |
| jeblair | clarkb: it looks funny. | 22:14 |
| clarkb | jeblair: the first two changes seem to have done the correct thing | 22:14 |
| *** david-lyle_ has quit IRC | 22:15 | |
| anteaya | mattoliverau: I hope the move went well | 22:15 |
| anteaya | mattoliverau: ah the boxes lifestyle, I know that one | 22:16 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Set appropriate rate-limit for RAX clouds. https://review.openstack.org/68509 | 22:16 |
| anteaya | mattoliverau: just glad we haven't scared you off | 22:16 |
| jeblair | clarkb: what do you mean by the first 2 changes? | 22:16 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Set a ratelimit for tripleo-test-cloud. https://review.openstack.org/68510 | 22:17 |
| lifeless | jeblair: how do you feel about assuming we don't do puts and basing our HPCS rate on 40/minute ? | 22:17 |
| zaro | _david_, jeblair : i've submitted and am on the schedule for a talk at gerrit UC. it's a general talk about gerrit and OS CI. I wouldn't mind submitting another one for multi-master jenkins and let someone else do the general CI talk. | 22:17 |
| mattoliverau | anteaya: not yet :) It takes alot to scare me off! | 22:17 |
| * anteaya makes a note to try harder | 22:17 | |
| anteaya | :D | 22:17 |
| mattoliverau | lol | 22:18 |
| *** sarob has quit IRC | 22:18 | |
| jeblair | lifeless: pretty good since we're assuming 60/min now and it's mostly working. i'm pretty sure if we were hitting a 10/min limit we'd fail completely. | 22:18 |
| mattoliverau | I'm going down stairs to grab a coffee, brb | 22:18 |
| anteaya | mattoliverau: k | 22:18 |
| clarkb | jeblair: 66986 and 67788 | 22:18 |
| clarkb | jeblair: reading the logs I don't think it did the cancel behind failing item properly, I am now looking at code | 22:19 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Set HP cloud rate limits. https://review.openstack.org/68512 | 22:19 |
| jeblair | clarkb: what's unexpected to you? | 22:20 |
| zaro | mgagne: will rebase my jjb changes shortly. | 22:20 |
| clarkb | jeblair: 66258 should have had its jobs that ran removed | 22:20 |
| jeblair | clarkb: https://jenkins02.openstack.org/job/gate-python-heatclient-pep8/1585/ | 22:21 |
| clarkb | jeblair: and 57245 should not be red | 22:21 |
| sdague | lifeless: unit tests don't install client libraries from git | 22:21 |
| jeblair | clarkb: according to jenkins it's tested on the 2 changes ahead of it | 22:21 |
| jeblair | clarkb: possible you missed the reset -- that's running on a static precise node, so it swooped in and ran the jobs quick | 22:21 |
| lifeless | sdague: right, which leads to firedrills when client libraries release and break things (like https://bugs.launchpad.net/heat/+bug/1271367) | 22:21 |
| jeblair | clarkb: (it's not tested based on anything not currently in the queue, so it looks right to me) | 22:22 |
| clarkb | jeblair: ok, I must have missed a reset then | 22:22 |
| jeblair | clarkb: double check, but that seems to hold for the cinderclient and tempest jobs below too | 22:22 |
| *** burt1 has joined #openstack-infra | 22:22 | |
| jeblair | (and the tempest change is not tested with the red cinderclient chaneg) | 22:22 |
| clarkb | how did the heat pythonclient stuff run before the swift jobs? | 22:23 |
| sdague | lifeless: that's fine, but it's currently not a feature we have. And in the move back to check land, that would redline heat in check. We'd have to fix it, but the rest would still flow | 22:23 |
| fungi | and that cinderclient pypy failure looks odd. like the slave did something unexpected or something's screwed up the workspace on it | 22:23 |
| jeblair | clarkb: static precise node vs bare-precise | 22:23 |
| clarkb | jeblair: gotcha, ok I feel much better about what I am reading now thanks | 22:24 |
| *** miqui has quit IRC | 22:24 | |
| lifeless | sdague: where do you see tripleo-ci deployments living? gate or just check ? | 22:25 |
| mgagne | zaro: https://review.openstack.org/#/c/68152/2 This change (introducing Test Stability plugin support) piggybacks the junit plugin publisher. What do you think should be the policy in that regard? Should a plugin be allowed to be configured through an other plugin section or not? | 22:26 |
| *** sarob has joined #openstack-infra | 22:29 | |
| sdague | lifeless: check | 22:30 |
| sdague | I think, honestly, having never seen one, I don't know | 22:30 |
| sdague | I think we figure it out over time | 22:30 |
| *** dizquierdo has quit IRC | 22:30 | |
| zaro | mgagne: so if it's not allowed what would be the alternative? create a seperate jjb target, something like junit_stability? | 22:30 |
| lifeless | sdague: this plan seems to massively increase thread-the-needle events to me | 22:31 |
| sdague | lifeless: I agree | 22:31 |
| lifeless | sdague: *and*, we have a range of tests that are not safe to run in check | 22:31 |
| sdague | lifeless: well that needs to be addressed then | 22:31 |
| lifeless | sdague: specifically anything running on baremetal needs to be vetted before running to avoid run-malicious-code attacks | 22:31 |
| sdague | because we can't run a test for the first time in gate | 22:31 |
| sdague | full stop | 22:31 |
| *** jergerber has joined #openstack-infra | 22:31 | |
| mgagne | zaro: tbh, I don't know. It's the first time (I'm aware of) someone introduces this kind of change | 22:32 |
| pleia2 | lifeless: sorry, got pulled into a call - yeah, planning on writing the fedora def today, need to grab lunch first though | 22:33 |
| lifeless | pleia2: I'll do it now | 22:33 |
| zaro | mgagne: so what do you think? | 22:33 |
| zaro | mgagne: i'm ok with it because i don't see a better alternative. | 22:33 |
| *** alexpilotti has quit IRC | 22:34 | |
| mgagne | zaro: me neither I guess. Can this option be enabled without the Junit plugin or is it a dependency? | 22:34 |
| pleia2 | lifeless: so I was thinking, in the definition will you just have it spin up 0 to start? it will still need a restart to add the appropriate amount when we're ready | 22:34 |
| *** alexpilotti has joined #openstack-infra | 22:34 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add a fedora image definition for tripleo-cloud https://review.openstack.org/68515 | 22:36 |
| lifeless | pleia2: ^ | 22:36 |
| mgagne | zaro: ok, I give in. Test stability history is shown as a sub-option of the junit publisher =) | 22:36 |
| mattoliverau | So looks like alot has happened, honestly guys I take 1 day off and you change everything :P | 22:36 |
| zaro | mgagne: i'm guessing that installing test stability will auto install junit plugin. but i think junit is a core plugin anyway. | 22:37 |
| anteaya | mattoliverau: welcome to our world | 22:37 |
| mgagne | zaro: alright | 22:37 |
| *** sdake is now known as sdake-ooo | 22:37 | |
| lifeless | pleia2: uploading a fedora image to glance now | 22:37 |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Add local-branch option https://review.openstack.org/65369 | 22:37 |
| pleia2 | lifeless: so I think with that change it will load up 4 images total - 2 precies and 2 fedora | 22:38 |
| lifeless | yes | 22:38 |
| pleia2 | will that actually work with fedora? | 22:38 |
| lifeless | pleia2: we'll find out | 22:39 |
| lifeless | pleia2: we don't need to restart nodepool to iterate further though | 22:39 |
| pleia2 | ok :) | 22:39 |
| lifeless | fungi: https://review.openstack.org/68515 too please | 22:39 |
| pleia2 | I haven't tried any of the prepare_node* scripts with fedora | 22:39 |
| pleia2 | see, I was going to test before writing the patch! anyway, I can test after lunch, it's late | 22:40 |
| lifeless | pleia2: yeah, shoo :) | 22:40 |
| lifeless | pleia2: principle of separated concerns thoug | 22:40 |
| pleia2 | hehe | 22:40 |
| *** alexpilotti has quit IRC | 22:41 | |
| *** dangers is now known as dangers_away | 22:42 | |
| openstackgerrit | Eli Klein proposed a change to openstack-infra/jenkins-job-builder: Added rbenv-env wrapper https://review.openstack.org/65352 | 22:43 |
| clarkb | ok I have done more digging in the zuul logs and have most of my confidence back :) | 22:44 |
| clarkb | we are just being starved by the check queue which isn't super horrible because it should clear that massive list out relatively quickly | 22:45 |
| russellb | clarkb: nice work on the rate limiting patch | 22:45 |
| clarkb | russellb: thanks, I keep second guessing it, but it appears to be doing the correct thing | 22:46 |
| lifeless | sdague: so what we need is jobs that run only on +A, before the integrated gate jobs are queued, then ? | 22:46 |
| *** resker has quit IRC | 22:46 | |
| lifeless | sdague: or possibly a check job that runs on +2 ? | 22:46 |
| sdague | lifeless: I think you could modify zuul to run a set of jobs on first +2 | 22:47 |
| fungi | lifeless: while the design requires discussion, we could probably have a separate independent pipeline for +2 events (but it would tend to get rerun on each +2) | 22:47 |
| fungi | right, to only have it run on the first +2 would probably need a zuul patch | 22:48 |
| lifeless | so what we have is virt emulation that can run in regular check | 22:48 |
| *** thomasem has quit IRC | 22:48 | |
| lifeless | and we have baremetal that must run before landing (because it's the actual verification) | 22:48 |
| *** ivar-lazzaro has joined #openstack-infra | 22:49 | |
| lifeless | but as sdague says we don't want to trigger pipeline stalls at least until we get rid of many more bugs | 22:49 |
| sdague | lifeless: I'd say right now what you probably actually want to get going is a sufficiency check in experimental | 22:50 |
| sdague | so check experimental runs different jobs if it has a +2 than if not | 22:50 |
| lifeless | sdague: right now we're working up the testing stack | 22:50 |
| lifeless | sdague: we're about to have experimental actually doing shit; then nonvoting check | 22:51 |
| sdague | that would let you actually see how a job would run in the gate, and protect yuo | 22:51 |
| sdague | lifeless: right, but anyone can trigger check experimental | 22:51 |
| sdague | so that doesn't solve your security problem | 22:51 |
| lifeless | sdague: so experimental must be virt only then | 22:51 |
| sdague | lifeless: which doesn't solve running real tests in any experimental way | 22:51 |
| lifeless | sdague: we are a ways off of having the virt stuff bedded down, and we'll get a lot of reliabilty from just that | 22:52 |
| sdague | lifeless: ok, so then don't overengineer the future :) | 22:52 |
| lifeless | sdague: but yeah, Ironic really wants real baremetal soon | 22:52 |
| lifeless | sdague: so I'm just getting my head around having the design rug pulled out | 22:52 |
| sdague | your road to having real baremetal as part of the equation is a +2 experimental class | 22:52 |
| sdague | clarkb: yeh, nice work on the zuul bits | 22:53 |
| lifeless | pleia2: fedora boots two-nics ok, mtu is wrong, eth1 is down, of course. | 22:53 |
| sdague | it might be nice to put the "runable" part of the queue into the json | 22:53 |
| sdague | so we could highlight the set of jobs that are in the run set | 22:54 |
| lifeless | pleia2: but - image is there in the ci cloud, so you can play with it if you get the nodepool user creds | 22:54 |
| lifeless | sdague: in the sense that experimental is the onramp for any new testing endeavour? | 22:54 |
| sdague | lifeless: correct | 22:54 |
| sdague | at this point the default place for a new test job to go is in experimental | 22:55 |
| sdague | for lots of good reasons | 22:55 |
| fungi | sdague: yeah, adjusting the ui (and needing extra bits in the json to support that) has already come up, so it's presumably in the works | 22:58 |
| fungi | i'm assuming all the changes beyond the window will appear visually disconnected, and probably with a separate colored dot | 22:59 |
| lifeless | https://bugs.launchpad.net/zuul/+bug/1271766 | 23:02 |
| *** dcramer__ has quit IRC | 23:04 | |
| *** CaptTofu has quit IRC | 23:04 | |
| *** dims has quit IRC | 23:04 | |
| *** oubiwann_ has quit IRC | 23:04 | |
| *** senk1 has quit IRC | 23:06 | |
| *** sandywalsh has quit IRC | 23:06 | |
| *** alexpilotti has joined #openstack-infra | 23:08 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add require-approval to Gerrit trigger https://review.openstack.org/68516 | 23:09 |
| jeblair | sdague, clarkb, fungi, mordred: ^ | 23:09 |
| clarkb | jeblair: cool | 23:10 |
| jeblair | sdague, clarkb, fungi, mordred: Um. I'm particularly excited about the "approval with old jenkins vote causes automatic enqueue in check; then positive check result causes automatic enqueue into gate" behavior, which is actually shown in a test there. :) | 23:10 |
| ivar-lazzaro | Hello folks, I need some advice for configuring Jenkins+Gerrit filters... Specifically, I would like to run a Build whenever I get a specific comment from the stream | 23:10 |
| jeblair | sdague, clarkb, fungi, mordred: what could possibly go wrong with Zuul responding to its own events. :) | 23:11 |
| sdague | jeblair: heh | 23:11 |
| sdague | it's turtles all the way down | 23:11 |
| clarkb | ivar-lazzaro: we haven't used the gerrit trigger plugin in jenkins for a very long time. I am not personally aware of how to do that | 23:13 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add require-approval to Gerrit trigger https://review.openstack.org/68516 | 23:13 |
| ivar-lazzaro | clarkb: Thanks for your answer... hopefully someone around ever dealt with this problem | 23:14 |
| *** eharney has quit IRC | 23:15 | |
| anteaya | ivar-lazzaro: if you lurk in #openstack-neutron and look for sukhdev he may be able to help you | 23:15 |
| mikal | anteaya: noted on the neutron rechecks, although my script isn't running t the moment | 23:15 |
| ivar-lazzaro | anteaya: thanks! | 23:16 |
| *** MarkAtwood has quit IRC | 23:16 | |
| anteaya | mikal: thanks | 23:16 |
| anteaya | ivar-lazzaro: np | 23:16 |
| *** burt1 has quit IRC | 23:17 | |
| clarkb | jeblair: in your check example pipeline, check tests will only run at most once every 48 hours? | 23:18 |
| pleia2 | lifeless: the infra creds for nodepool? | 23:19 |
| jeblair | clarkb: yes; and only on changes that are producing events | 23:19 |
| clarkb | right, it doesn't trigger on that interval | 23:19 |
| jeblair | clarkb: so if no one cares about a change, it can sit there and not be updated. if people are commenting on it, etc, it will get updated, and of course, if the only comment is an aprv and it's old, then it goes through the check->gate progression | 23:20 |
| *** dims has joined #openstack-infra | 23:20 | |
| pleia2 | lifeless: and which image did you upload? fedora cloud image? | 23:20 |
| *** jgrimm has quit IRC | 23:20 | |
| jeblair | mikal: i believe what we are discussing will fill the need to have an automated system rechecking old changes | 23:21 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: webapp: set cache-control headers to prevent caching https://review.openstack.org/66583 | 23:21 |
| jeblair | mikal: (so in other words, i believe zuul is about to grow the ability to do this itself) | 23:21 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: make job creation consistent https://review.openstack.org/60633 | 23:21 |
| mikal | jeblair: yeah, I saw sdague's email bout zuul growing thus functionality, which I am fine with | 23:22 |
| lifeless | pleia2: Fedora 20 64-bit | 23:22 |
| *** jergerber has quit IRC | 23:24 | |
| pleia2 | lifeless: I saw that much :) wasn't sure if there was a specific cloud image or something like ubuntu has | 23:24 |
| lifeless | pleia2: there is one | 23:25 |
| *** sarob has quit IRC | 23:25 | |
| *** sarob has joined #openstack-infra | 23:25 | |
| ttx | fungi: where are you hiding ? | 23:27 |
| fungi | ttx: working from my room | 23:27 |
| ttx | fungi: we are by the fire near the breakfast area if you want to join us (Heidi, tom) | 23:28 |
| *** sarob has quit IRC | 23:29 | |
| fungi | cool, be right over | 23:30 |
| lifeless | fungi: hey, so hows nodepool :) | 23:30 |
| *** rockyg has joined #openstack-infra | 23:31 | |
| sdague | clarkb: you watching -qa? we just lost a bunch of console logs | 23:38 |
| clarkb | sdague: ya, I think old jenkins is susceptible to that at a much lower rate than new jenkins with new scp plugin was | 23:39 |
| lifeless | sdague: replied to the gate thread | 23:39 |
| *** mfer has quit IRC | 23:39 | |
| lifeless | sdague: I'm fairly worried about the change now I've had time to think about it :( | 23:39 |
| clarkb | sdague: I am waiting for mordred's jenkinses then will do all of the others | 23:39 |
| fungi | lifeless: not entirely sure what has caused it to decide not to do nightly builds of hpcloud-az2.devstack-precise (the logs don't show it even trying). wondering whether it will persist after we restart it | 23:40 |
| lifeless | fungi: /me starts chanting 'restart', 'restart', 'restart' | 23:40 |
| fungi | well, it's in the middle of building ~150 nodes constantly to churn through the check pipeline | 23:41 |
| lifeless | fungi: does that make restarting it hard? | 23:42 |
| fungi | oh, actually closer to 200 | 23:42 |
| fungi | lifeless: i believe restarting nodepool will abandon all of the currently building vms | 23:42 |
| fungi | so after a restart i will presumably need to manually delete any older than the start time | 23:43 |
| lifeless | fungi: yes; stop, list | grep BUILD | xards nodepool delete | 23:43 |
| lifeless | then start | 23:43 |
| lifeless | or | 23:43 |
| *** sarob has joined #openstack-infra | 23:43 | |
| lifeless | stop; list > file; start; grep building < file | xargs -n1 nodepool delete | 23:43 |
| *** whoops has quit IRC | 23:44 | |
| fungi | mmm, i haven't tried nodepool list/delete when nodepoold isn't running. i guess that works? | 23:44 |
| lifeless | yup | 23:44 |
| jeblair | yeah, that ^; if it's a lot you can parallelize it a bit | 23:44 |
| lifeless | It might be an idea to make that queue things up to happen in the server, but at the moment its entirely client based | 23:44 |
| fungi | right, i'd split the list five ways like i'd been doing and run five delete loops in parallel. that's seemed to work well enough | 23:45 |
| jeblair | (which is occasionally pretty handy) | 23:45 |
| jog0 | are there any plans to prevent the gate queue from getting stuck with the top change in queued mode | 23:46 |
| jeblair | jog0: nothing should ever be stuck. can you elaborate? | 23:46 |
| jog0 | stuck is probably the wrong word, if you look at http://status.openstack.org/zuul/ | 23:47 |
| *** kraman has quit IRC | 23:47 | |
| sdague | jog0: right, we don't have any d-g nodes | 23:47 |
| jeblair | jog0: node starvation due to check load | 23:48 |
| sdague | this is just starvation | 23:48 |
| jog0 | the top gate queue patch 66986,3 is waiting for d-g nodes | 23:48 |
| jeblair | fungi: i also wonder how many of those building nodes are really building; can you look while you're there? | 23:48 |
| fungi | so, jeblair any input on https://review.openstack.org/66958 (if it's okay we should probably approve before a nodepoold restart). i'm pretty comfortable with self-approving https://review.openstack.org/67684 | 23:48 |
| jog0 | jeblair: right, what about propritizing the top n changes in gate queue? | 23:48 |
| jog0 | for some low value of n | 23:48 |
| sdague | jog0: so we did that before at one point, and it starved out the check queue entirely | 23:49 |
| fungi | jeblair: i'm running a watch on what nodepool list reports in what states on what providers and am showing about 200 building across various providers currently | 23:49 |
| jog0 | sdague: that was for prioritizing just the top n? what was n? | 23:49 |
| fungi | actually it's dropped to about 150 noe | 23:49 |
| fungi | now | 23:49 |
| jeblair | fungi: aprvd | 23:49 |
| sdague | jog0: no, but that's more complex logic that doesn't exist | 23:49 |
| jeblair | jog0: this situation is slightly abnormal | 23:50 |
| jog0 | sdague: ahh thats what I thought. | 23:50 |
| sdague | jog0: the real answer is not to be starved by a factor of 6 | 23:50 |
| jeblair | jog0: it's largely the result of an earlier zuul restart where all 100 changes were enqueued into check at once | 23:50 |
| jog0 | jeblair: this may be abnormal now, but abnormal may become the new normal | 23:50 |
| fungi | slammed it pretty hard | 23:50 |
| sdague | which is basically where we stand, given the average number of nodes available | 23:50 |
| jeblair | jog0: i hope restarting zuul and enqueuing 100 changes at once is never normal. | 23:50 |
| fungi | i hope manually reloading the zuul pipelines isn't about to become the new normal | 23:51 |
| jog0 | jeblair: ahh, | 23:51 |
| lifeless | so here's a crazy question | 23:51 |
| jog0 | although i have seen this before with a long check queue and a top of gate reset | 23:51 |
| jeblair | jog0: the point being that the gate queue is currently waiting for _all_ 100 changes to be serviced which is not normall, usually they only have to wait for a handful as they trickle in in real time. | 23:51 |
| lifeless | what about treating the middle nodes in the queue as an optimisation and running the *end* of the queue first | 23:51 |
| lifeless | only if it fails do you need to the results from the ones in the middle | 23:51 |
| jeblair | jog0: the restart was to pick up the change clarkb wrote to only run jobs for the top N changes; once that really gets going we'll have a very different dynamic | 23:52 |
| lifeless | it would give up a current 'guarantee', that each commit merged is independently good | 23:52 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Catch exceptions from nova flavor-list calls https://review.openstack.org/66958 | 23:52 |
| *** markmcclain has quit IRC | 23:53 | |
| jeblair | jog0: so i'd like to let this settle out before we tune again. also, mordred is spinning up new jenkins masters to handle 250 more nodes | 23:53 |
| jog0 | jeblair: ahh I see thanks for explaining | 23:53 |
| jog0 | 250 more nodes, nice | 23:53 |
| jeblair | lifeless: we never get to the end | 23:53 |
| lifeless | jeblair: I know, but I don't think that matters | 23:54 |
| lifeless | jeblair: the main point is not to cancel out a full stack of tests because one failed; it might be a spurious failure | 23:54 |
| * StevenK blinks at the post horizon job | 23:55 | |
| sdague | lifeless: but you had to run the test anyway? or are you saying just squash the whole queue? | 23:55 |
| lifeless | jeblair: put another way, given Change C, Change C', C'', C''' etc, if any of these pass, either the predecessors had a transient bug (e.g. C' is broken but C'' fixes it), or it was a spurious failure (C' failed because of nondeterministic test) or it is a spurious pass | 23:56 |
| jeblair | lifeless: ah, this is similar to the 'batching changes' suggestion. yes, it sacrifices bisectability. | 23:56 |
| lifeless | sdague: I'm saying, don't reset the gate queue, let it run and if a pass happens, land it | 23:56 |
| sdague | lifeless: how do you land a pass? | 23:56 |
| lifeless | sdague: concurrently, build a parallel queue with the head ejected, and try that | 23:56 |
| sdague | that's 50 deep changes in heat requires, in which no heat tests ran? | 23:57 |
| jeblair | lifeless: okay, now that's the 'alternate branch' suggestion. :) | 23:57 |
| jeblair | lifeless: which doesn't lose bisectability. | 23:57 |
| jeblair | but uses extra resources | 23:57 |
| lifeless | sdague: if the original head lands, we discard the alternate; if none of them do, the alternate is the new main | 23:57 |
| lifeless | jeblair: actually they have very different latency characteristics, I think | 23:57 |
| sdague | lifeless: ok, sure, but you did get the point that node starvation is one of our key issues right? | 23:58 |
| lifeless | sdague: I did, but its a key issue because we're assuming that C' failure means C'' must fail and so we throw away and restart all 50 changes | 23:58 |
| clarkb | so the problem with alternate branch that no one considers, is we have no resources :P | 23:58 |
| lifeless | sdague: which is a nontrivial exercise | 23:58 |
| lifeless | anyhow, just putting it out there | 23:59 |
| lifeless | sdague: I'm not sure what you mean by not heat tests ran | 23:59 |
| sdague | so 50 deep in the queue | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!