| *** ericw has quit IRC | 00:00 | |
| *** gyee has quit IRC | 00:03 | |
| openstackgerrit | A change was merged to openstack/requirements: Add tempest to projects list https://review.openstack.org/49225 | 00:03 |
|---|---|---|
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Flow project support https://review.openstack.org/49608 | 00:05 |
| dstufft | mordred: clarkb whoever else, the thrift 503/timeout error should hopefully be fixed over the weekend, upstream thinks they identified the problem | 00:05 |
| clarkb | dstufft: hurray | 00:06 |
| dstufft | I'll let you guys know once they think it's actually solved, so you can tell me if you keep seeing it after that | 00:09 |
| clarkb | fungi: https://review.openstack.org/#/c/49772/ where did we end up with d-g and stable branches? | 00:10 |
| clarkb | jeblair: for nodepool. Is it generally safe to just approve changes? how much coordination should we at least attempt to perform? | 00:17 |
| *** CaptTofu has quit IRC | 00:31 | |
| *** ericw has joined #openstack-infra | 00:32 | |
| *** CaptTofu has joined #openstack-infra | 00:32 | |
| *** dims has quit IRC | 00:53 | |
| *** ericw has quit IRC | 00:55 | |
| *** Ryan_Lane has quit IRC | 00:55 | |
| *** sarob has joined #openstack-infra | 00:57 | |
| jeblair | clarkb: it's about like zuul; it's CD but isn't auto-restarted, so is generally okay; but the config is continuously re-read, so extra care is needed in merging disruptive changes to nodepool.yaml in infra/config | 00:58 |
| jeblair | pabelanger: that sounds good; i'd suggest making it as openstack-sounding as possible | 00:59 |
| *** melwitt has quit IRC | 01:01 | |
| *** senk has joined #openstack-infra | 01:04 | |
| *** senk has quit IRC | 01:05 | |
| *** senk has joined #openstack-infra | 01:05 | |
| *** dims has joined #openstack-infra | 01:07 | |
| *** ericw has joined #openstack-infra | 01:12 | |
| *** ericw has quit IRC | 01:12 | |
| *** changbl has joined #openstack-infra | 01:14 | |
| anteaya | doth my eyes deceive me? is zuul actually sitting idle with no work to do? | 01:16 |
| anteaya | there's one | 01:17 |
| anteaya | one in check | 01:18 |
| anteaya | and done | 01:18 |
| anteaya | so fast | 01:18 |
| *** Ryan_Lane has joined #openstack-infra | 01:26 | |
| *** Ryan_Lane has quit IRC | 01:33 | |
| *** rcleere has joined #openstack-infra | 01:38 | |
| *** rcleere has quit IRC | 01:42 | |
| *** sarob has quit IRC | 01:42 | |
| *** sarob has joined #openstack-infra | 01:43 | |
| *** sarob has quit IRC | 01:47 | |
| fungi | clarkb: oh, bummer on https://review.openstack.org/49429 (guess i didn't realize) | 01:54 |
| *** Ryan_Lane has joined #openstack-infra | 02:01 | |
| fungi | as for d-g checks against stable branches i'll rework those as .*-stable-{branch} templates. great idea | 02:06 |
| *** Ryan_Lane has quit IRC | 02:06 | |
| *** thingee is now known as thingee_zzz | 02:07 | |
| morganfainberg | anteaya, shhhhh, you'll scare Zuul since it's sitting idle, i like to think it's day dreaming. | 02:13 |
| anteaya | I wonder what zuul would daydream about | 02:14 |
| anteaya | since he hasn't had the time to do so in quite a while | 02:14 |
| morganfainberg | anteaya, the keymaster? /oblig. ghostbusters reference | 02:14 |
| anteaya | yeah maybe | 02:14 |
| anteaya | but after that | 02:15 |
| *** sarob has joined #openstack-infra | 02:24 | |
| *** jerryz has quit IRC | 02:33 | |
| *** sarob has quit IRC | 02:51 | |
| *** sarob has joined #openstack-infra | 02:52 | |
| *** sarob has quit IRC | 02:56 | |
| *** vipul has quit IRC | 03:00 | |
| *** vipul has joined #openstack-infra | 03:01 | |
| *** Ryan_Lane has joined #openstack-infra | 03:01 | |
| clarkb | jeblair: thanks, I will do my best to remember that :) | 03:01 |
| clarkb | fungi: thanks. If havana wasn't so close I probably wouldn't care so much | 03:02 |
| *** sarob has joined #openstack-infra | 03:13 | |
| *** svarnau has quit IRC | 03:25 | |
| *** Ryan_Lane has quit IRC | 03:37 | |
| *** senk has quit IRC | 03:48 | |
| *** senk has joined #openstack-infra | 03:51 | |
| *** anteaya has quit IRC | 03:52 | |
| *** ArxCruz has quit IRC | 03:58 | |
| *** ArxCruz has joined #openstack-infra | 03:59 | |
| *** oubiwann-lambda has quit IRC | 04:04 | |
| *** Ryan_Lane has joined #openstack-infra | 04:05 | |
| *** Ryan_Lane has quit IRC | 04:09 | |
| *** Ryan_Lane has joined #openstack-infra | 04:35 | |
| *** ArxCruz has quit IRC | 04:36 | |
| *** blamar has quit IRC | 04:39 | |
| *** blamar has joined #openstack-infra | 04:40 | |
| *** Ryan_Lane has quit IRC | 04:40 | |
| *** mestery_ has joined #openstack-infra | 04:42 | |
| *** zul has quit IRC | 04:43 | |
| *** HenryG has quit IRC | 04:43 | |
| *** zul has joined #openstack-infra | 04:44 | |
| *** beekneemech has quit IRC | 04:45 | |
| *** mestery has quit IRC | 04:46 | |
| *** CaptTofu has quit IRC | 04:47 | |
| *** CaptTofu has joined #openstack-infra | 04:47 | |
| *** salv-orlando has quit IRC | 05:20 | |
| *** senk has quit IRC | 05:20 | |
| *** salv-orlando has joined #openstack-infra | 05:20 | |
| *** senk has joined #openstack-infra | 05:20 | |
| *** dcramer_ has quit IRC | 05:33 | |
| *** dcramer_ has joined #openstack-infra | 05:49 | |
| lifeless | does reviewerstats auto-deploy? | 05:54 |
| *** ryanpetrello has quit IRC | 06:04 | |
| *** sarob has quit IRC | 06:04 | |
| *** sarob has joined #openstack-infra | 06:04 | |
| *** sarob has quit IRC | 06:09 | |
| *** senk has quit IRC | 06:27 | |
| *** dims has quit IRC | 06:38 | |
| *** dims has joined #openstack-infra | 06:39 | |
| *** AlexF has joined #openstack-infra | 06:46 | |
| *** dims has quit IRC | 06:49 | |
| *** dims has joined #openstack-infra | 06:49 | |
| *** dstufft_ has joined #openstack-infra | 06:57 | |
| *** dstufft has quit IRC | 06:58 | |
| *** fifieldt has joined #openstack-infra | 06:59 | |
| *** boris-42 has joined #openstack-infra | 07:04 | |
| *** garyk has quit IRC | 07:07 | |
| *** AlexF has quit IRC | 07:08 | |
| *** changbl has quit IRC | 07:11 | |
| *** sarob has joined #openstack-infra | 07:15 | |
| *** dstufft_ is now known as dstufft | 07:18 | |
| *** sarob has quit IRC | 07:21 | |
| *** garyk has joined #openstack-infra | 07:27 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:28 | |
| *** garyk has quit IRC | 07:29 | |
| *** dims has quit IRC | 07:32 | |
| *** dims has joined #openstack-infra | 07:37 | |
| *** yolanda has joined #openstack-infra | 07:45 | |
| *** tvb has quit IRC | 07:50 | |
| *** tvb has joined #openstack-infra | 07:50 | |
| *** tvb has quit IRC | 08:07 | |
| *** yolanda has quit IRC | 08:11 | |
| *** tvb has joined #openstack-infra | 08:16 | |
| *** sdake_ has quit IRC | 08:22 | |
| *** sdake has quit IRC | 08:23 | |
| *** sdake has joined #openstack-infra | 08:55 | |
| *** sdake_ has joined #openstack-infra | 08:55 | |
| *** sdake_ has quit IRC | 09:06 | |
| *** sdake has quit IRC | 09:08 | |
| *** vipul has quit IRC | 09:09 | |
| *** vipul has joined #openstack-infra | 09:09 | |
| *** sdake_ has joined #openstack-infra | 09:10 | |
| *** sdake has joined #openstack-infra | 09:10 | |
| *** nijaba has joined #openstack-infra | 09:20 | |
| *** nijaba has quit IRC | 09:20 | |
| *** nijaba has joined #openstack-infra | 09:20 | |
| *** sdake has quit IRC | 09:27 | |
| *** sdake_ has quit IRC | 09:28 | |
| *** sdake has joined #openstack-infra | 09:39 | |
| *** sdake_ has joined #openstack-infra | 09:40 | |
| *** sdake has quit IRC | 09:56 | |
| *** sdake_ has quit IRC | 09:57 | |
| *** sdake_ has joined #openstack-infra | 09:58 | |
| *** sdake has joined #openstack-infra | 09:59 | |
| *** sdake has joined #openstack-infra | 09:59 | |
| *** sdake has quit IRC | 10:03 | |
| *** sdake_ has quit IRC | 10:03 | |
| *** SergeyLukjanov has quit IRC | 10:05 | |
| *** sdake_ has joined #openstack-infra | 10:07 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:10 | |
| *** salv-orlando has quit IRC | 10:29 | |
| *** fifieldt has quit IRC | 10:44 | |
| *** SergeyLukjanov has quit IRC | 10:46 | |
| *** SergeyLukjanov has joined #openstack-infra | 11:33 | |
| *** SergeyLukjanov has quit IRC | 11:51 | |
| *** qba73 has joined #openstack-infra | 11:59 | |
| *** SergeyLukjanov has joined #openstack-infra | 12:02 | |
| *** CaptTofu has quit IRC | 12:26 | |
| *** CaptTofu has joined #openstack-infra | 12:26 | |
| *** dcramer_ has quit IRC | 12:32 | |
| *** AlexF has joined #openstack-infra | 12:36 | |
| *** AlexF_ has joined #openstack-infra | 12:40 | |
| *** AlexF has quit IRC | 12:40 | |
| *** AlexF_ has quit IRC | 12:44 | |
| *** enikanorov-w has quit IRC | 12:47 | |
| *** enikanorov-w has joined #openstack-infra | 12:48 | |
| *** AlexF has joined #openstack-infra | 12:48 | |
| *** alexpilotti has quit IRC | 12:53 | |
| *** CaptTofu has quit IRC | 12:55 | |
| *** CaptTofu has joined #openstack-infra | 12:55 | |
| *** ruhe has joined #openstack-infra | 12:56 | |
| *** ruhe has quit IRC | 12:57 | |
| *** alexpilotti has joined #openstack-infra | 12:59 | |
| *** SergeyLukjanov has quit IRC | 13:02 | |
| *** ken1ohmichi has joined #openstack-infra | 13:12 | |
| *** AlexF has quit IRC | 13:12 | |
| *** salv-orlando has joined #openstack-infra | 13:12 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:19 | |
| *** qba73 has quit IRC | 13:40 | |
| sdague | dkranz: wget -q -O- --header="Accept-Encoding: gzip" http://logs.openstack.org/40/49840/1/check/check-tempest-devstack-vm-full/e9defdf/logs/screen-n-cpu.txt.gz does pull it gzip over the wire | 13:44 |
| * sdague dumping answers to all the questions I saw, sorry, was NYC all day friday | 13:45 | |
| sdague | mtreinish: the change I was going to make was just to change the downloaded filename to not end in .gz if you didn't pull gz content, not to magically compress if you didn't tell it you support compression | 13:46 |
| *** AlexF has joined #openstack-infra | 13:53 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: statsd documentation https://review.openstack.org/49885 | 14:09 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: doc: disable intersphinx mapping https://review.openstack.org/49887 | 14:13 |
| *** SergeyLukjanov has quit IRC | 14:13 | |
| *** AlexF has quit IRC | 14:14 | |
| *** HenryG has joined #openstack-infra | 14:16 | |
| *** DennyZhang has joined #openstack-infra | 14:21 | |
| *** dims has quit IRC | 14:24 | |
| *** oubiwann has joined #openstack-infra | 14:25 | |
| *** ken1ohmichi has quit IRC | 14:36 | |
| *** senk has joined #openstack-infra | 14:39 | |
| *** thomasm has joined #openstack-infra | 14:41 | |
| *** thomasm has quit IRC | 14:44 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:47 | |
| *** sdake has joined #openstack-infra | 14:48 | |
| *** changbl has joined #openstack-infra | 14:53 | |
| fungi | jeblair: clarkb: mordred: i'm filling out https://etherpad.openstack.org/infra-maint-2013-10-05 for us to work from | 15:00 |
| jeblair | fungi: cool, thx | 15:01 |
| fungi | i'd consider shutting down jenkins01/02 now, but there's really nothing going on | 15:01 |
| sdague | so I found we broken stable/grizzly on the devstack-gate id issolation, trying to fix it in devstack. Probably not a huge rush with the gerrit shutdown today, but fyi | 15:19 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Inspect the Gearman queue for immediate demand https://review.openstack.org/48517 | 15:20 |
| jeblair | sdague: even beyond the directory chmod that fungi fixed? | 15:20 |
| sdague | so I did a recheck on something this morning and it broke | 15:21 |
| sdague | when did that merge? | 15:21 |
| *** AlexF has joined #openstack-infra | 15:21 | |
| jeblair | sdague: it did not. | 15:21 |
| sdague | https://review.openstack.org/#/c/48316/ still failed on recheck | 15:21 |
| jeblair | sdague: https://review.openstack.org/#/c/49772/ | 15:21 |
| sdague | ok, so that should merge as soon as anyone can push it through | 15:22 |
| jeblair | sdague: we still have time before the maint window, so i aprvd it | 15:22 |
| sdague | does make me realize we should probably test devstack-gate master against stable/* on changes | 15:23 |
| jeblair | sdague: fungi has been working on that too | 15:23 |
| fungi | sdague: heh | 15:23 |
| sdague | nice | 15:24 |
| fungi | sdague: hopefully my next patchset later today will be suitable there | 15:24 |
| sdague | cool | 15:24 |
| jeblair | https://review.openstack.org/#/c/49805/ | 15:24 |
| sdague | I'll have dubious ability to look at things this weekend, we've got a wedding today, and friends coming into town | 15:24 |
| sdague | but I'll try to look late on Sunday | 15:25 |
| *** qba73 has joined #openstack-infra | 15:25 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Start using gearman-based bursting with nodepool https://review.openstack.org/49890 | 15:26 |
| jeblair | fungi: can you take a look at that ^ real quick? i figure we may as well do that while nodepool is idle | 15:27 |
| jeblair | fungi: i stopped puppet on nodepool, so it's safe to aprv | 15:27 |
| fungi | jeblair: sure thing | 15:28 |
| fungi | in it goes | 15:29 |
| sdague | I threw the trivial devstack stable/grizzly change into the merge queue behind the devstack-gate change, so it should prove that we are working again | 15:29 |
| fungi | and we're ready to merge the corresponding nodepool code change as well? | 15:29 |
| jeblair | fungi: i did that a few mins ago | 15:30 |
| fungi | ahh, awesome | 15:30 |
| jeblair | fungi: so as soon as the config change merges, i'll put zuul into graceful shutdown | 15:30 |
| jeblair | (the timing should be about right, assuming the d-g change passes) | 15:30 |
| fungi | sounds good | 15:30 |
| jeblair | fungi: i changed the etherpad a bit | 15:30 |
| fungi | cool. i'm still filling out the later bits, but earlier stuff is fair game | 15:31 |
| openstackgerrit | A change was merged to openstack-infra/config: Start using gearman-based bursting with nodepool https://review.openstack.org/49890 | 15:31 |
| jeblair | #status alert Zuul is shutting down for Gerrit downtime from 1600-1630 UTC | 15:33 |
| jeblair | someday fixing that bot is going to be the most important thing on my list. | 15:33 |
| *** openstackstatus has joined #openstack-infra | 15:34 | |
| jeblair | #status alert Zuul is shutting down for Gerrit downtime from 1600-1630 UTC | 15:34 |
| openstackstatus | NOTICE: Zuul is shutting down for Gerrit downtime from 1600-1630 UTC | 15:34 |
| *** ChanServ changes topic to "Zuul is shutting down for Gerrit downtime from 1600-1630 UTC" | 15:34 | |
| jeblair | fungi: sql lgtm | 15:37 |
| *** senk has quit IRC | 15:39 | |
| *** SergeyLukjanov has quit IRC | 15:41 | |
| fungi | thx | 15:42 |
| jeblair | sdague: i think that change just illustrated a bug in devstack-gate's branch fallback; when it falls back to master for a project that lacks the current branch, it doesn't check to see if zuul prepared a ref for the master branch | 15:47 |
| jeblair | sdague: i think that should be relatively straightforward to fix by extending the approach used in this change: https://review.openstack.org/#/c/41945/9/devstack-vm-gate-wrap.sh | 15:48 |
| jeblair | sdague: but for now, the stable branch change failed because it was not correctly tested with the d-g change in place. | 15:48 |
| sdague | jeblair: yep, no worries | 15:48 |
| clarkb | morning | 15:50 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Make sure tempest directories exist when modifying https://review.openstack.org/49772 | 15:51 |
| jeblair | clarkb: morning; etherpad at https://etherpad.openstack.org/infra-maint-2013-10-05 | 15:51 |
| clarkb | fungi: how does 49772 help stable/* ? Won't tempest still try to run in isolation and fail? | 15:51 |
| *** CaptTofu has quit IRC | 15:51 | |
| *** CaptTofu has joined #openstack-infra | 15:52 | |
| sdague | clarkb: the issue is really just directories aren't there, so it's an early fail based on the state dir chmod not happening | 15:52 |
| clarkb | sdague: but what ahppens when tempest runs? won't tempest need those dirs to exist and be owned by the tempest user? | 15:53 |
| sdague | clarkb: no, that state dir is for locking | 15:54 |
| sdague | which we don't do in stable/tempest | 15:54 |
| sdague | because we aren't running parallel | 15:54 |
| sdague | it's basically just used for a couple of locks for aggregates tests | 15:55 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Improve fallback to master branch https://review.openstack.org/49894 | 15:55 |
| jeblair | (while i was thinking about it) | 15:55 |
| jeblair | current eta for running tests is ~6 mins | 15:56 |
| fungi | perfect | 15:59 |
| jeblair | i made some job assignments in the etherpad | 15:59 |
| clarkb | is there a reason to prefer shutting down zuul over stopping jenkins? (are we upgrading zuul?) | 15:59 |
| fungi | i guess now that there are no more periodic jenkins jobs, it's a single point of control | 16:00 |
| fungi | however it does leave a window where changes to gerrit get missed | 16:00 |
| jeblair | yeah, i suppose it's a bit of a toss-up. we could try shutdown-mode in jenkins in the future. | 16:01 |
| jeblair | #status alert Gerrit will be down for maintenance from 1600-1630 UTC | 16:03 |
| openstackstatus | NOTICE: Gerrit will be down for maintenance from 1600-1630 UTC | 16:03 |
| *** ChanServ changes topic to "Gerrit will be down for maintenance from 1600-1630 UTC" | 16:03 | |
| *** SergeyLukjanov has joined #openstack-infra | 16:03 | |
| jeblair | clarkb, fungi: etherpad look good? | 16:03 |
| fungi | jeblair: lgtm | 16:03 |
| *** dkliban has joined #openstack-infra | 16:04 | |
| clarkb | lgtm | 16:05 |
| fungi | just threw in a couple optional items we noticed last time with cgit-related lag updating the repos list | 16:05 |
| jeblair | fungi: cool | 16:05 |
| fungi | assuming this goes smoothly i'll propose an update to the gerrit.rst later today | 16:05 |
| jeblair | zuul is stopped | 16:06 |
| fungi | so i see. time to start? | 16:06 |
| jeblair | fungi: you're clear to proceed | 16:06 |
| SergeyLukjanov | hi | 16:06 |
| jeblair | SergeyLukjanov: hi; you can follow what we're doing here: https://etherpad.openstack.org/infra-maint-2013-10-05 | 16:07 |
| jeblair | clarkb: with gerrit down, i believe we can proceed with our steps now | 16:07 |
| fungi | gerrit is offline now | 16:08 |
| SergeyLukjanov | jeblair, thx, looking on it no | 16:08 |
| SergeyLukjanov | now* | 16:08 |
| clarkb | jeblair: fungi: ok, updating stackforge repos on github now | 16:08 |
| fungi | working on 4 and 5 now | 16:08 |
| *** kong has joined #openstack-infra | 16:09 | |
| *** DennyZhang has quit IRC | 16:12 | |
| fungi | i'm ready to start gerrit | 16:13 |
| jeblair | fungi: i think we're waiting on clarkb | 16:14 |
| fungi | yep | 16:14 |
| fungi | that looks like the last github edit | 16:14 |
| clarkb | I am done with github | 16:14 |
| clarkb | so much clicking | 16:14 |
| fungi | so we're all set for 9? | 16:14 |
| jeblair | fungi: looks like you're clear to proceed | 16:14 |
| fungi | should be up now | 16:15 |
| jeblair | i'll start zuul | 16:15 |
| sdague | is zuul automatically throwing things back into the merge queue? | 16:17 |
| fungi | zuul looketh busy already | 16:17 |
| sdague | I'm confused why 48316,1 jumped back into the merge queue by itself | 16:18 |
| sdague | not that it's an issue | 16:18 |
| clarkb | sdague: it may have queued while zuul was in shutdown mode | 16:18 |
| sdague | just unexpected | 16:18 |
| fungi | i think those are things which got submitted while zuul was shutting down | 16:18 |
| sdague | fungi: well it was actually in the merge queue and failed | 16:18 |
| jeblair | aproved config changes | 16:18 |
| SergeyLukjanov | guys, is zuul layout changing requires manual zuul restart? just wondering | 16:19 |
| fungi | SergeyLukjanov: not any longer | 16:19 |
| SergeyLukjanov | fungi, cool | 16:19 |
| jeblair | clarkb: zmq plugin has gone silent again :( | 16:19 |
| fungi | it can reload intelligently on the fly as of a couple months ago | 16:20 |
| openstackgerrit | A change was merged to openstack-infra/config: Move tuskar to openstack. https://review.openstack.org/49255 | 16:20 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix fuel repos layout https://review.openstack.org/49407 | 16:20 |
| clarkb | jeblair: :/ | 16:20 |
| clarkb | jeblair: did we stop jenkins at all? | 16:20 |
| jeblair | clarkb: no | 16:20 |
| openstackgerrit | A change was merged to openstack-infra/config: Move savanna under openstack org https://review.openstack.org/48491 | 16:20 |
| fungi | jeblair: did you want to pull on the puppetmaster or shall i? | 16:21 |
| jeblair | fungi: i will | 16:21 |
| jeblair | fungi: done | 16:21 |
| fungi | ahh, yeah i saw you weren't idle | 16:21 |
| clarkb | jeblair: how do we know it went silent? is nodepool no longer receiving notifications? | 16:21 |
| fungi | okay, so ready for #11? | 16:21 |
| jeblair | clarkb: hrm, it might be nodepool that's dead | 16:21 |
| *** michchap_ has quit IRC | 16:22 | |
| clarkb | jeblair: yeah the logstash side of zmq seems to be receiving events | 16:22 |
| jeblair | fungi: yes, go on 11 | 16:22 |
| fungi | puppet updating now. will start puppet once it completes | 16:22 |
| *** openstackgerrit has quit IRC | 16:22 | |
| *** openstackgerrit has joined #openstack-infra | 16:23 | |
| fungi | it's still going. the manage-projects run will take a few minutes to finish | 16:23 |
| jeblair | fungi: i can proceed with #12, right? | 16:24 |
| *** mestery_ has quit IRC | 16:24 | |
| *** michchap has joined #openstack-infra | 16:24 | |
| fungi | yeah, should be safe to update puppetry on the cgit servers at least | 16:24 |
| jeblair | zuul just reloaded its config | 16:24 |
| fungi | and i think triggering replication now is probably also fine | 16:25 |
| *** dkliban has quit IRC | 16:26 | |
| jeblair | #12 done | 16:26 |
| jeblair | i'm production-fixing nodepool. :( | 16:26 |
| fungi | manage-projects is still running | 16:26 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add jobs for extra and dib repos https://review.openstack.org/49167 | 16:26 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove unused savanna rtfd jobs https://review.openstack.org/49168 | 16:26 |
| fungi | oof, so something about the gearman patch broke it? | 16:26 |
| jeblair | fungi: yeah, 2 things -- 1 gear didn't get installed (easy enough fix) | 16:27 |
| jeblair | fungi: and then i think there's a bug | 16:27 |
| fungi | well, we've got plenty of available servers for a while still | 16:27 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove unused savanna rtfd jobs https://review.openstack.org/49168 | 16:27 |
| fungi | yay buffer | 16:28 |
| *** dkliban has joined #openstack-infra | 16:28 | |
| clarkb | I was going to suggest we drop the idle pool number particularly after we get it to be more reactionary, but having the buffer now is handy | 16:28 |
| jeblair | clarkb: i dropped it a little with the config change (so we'de be able to see it), but hopefully still it'll be a good buffer | 16:29 |
| fungi | okay, manage-projects finished running and puppet is started again on review.o.o | 16:30 |
| fungi | i believe this completes the planned maintenance, though i'll go through and propose .gitreview file updates to the 8 projects which moved to openstack | 16:30 |
| fungi | and we might want to not officially conclude the maintenance until nodepool is happy, i suppose | 16:31 |
| *** erfanian has joined #openstack-infra | 16:33 | |
| jeblair | the function list is more complicated in reality than my testing so the parsing was off | 16:36 |
| *** AlexF has quit IRC | 16:38 | |
| *** dkliban has quit IRC | 16:41 | |
| jeblair | i think we should revert the nodepool change | 16:41 |
| jeblair | i'll do that real quick | 16:41 |
| clarkb | wfm | 16:42 |
| jeblair | clarkb: can you double check that zmq events from jenkins01 are working? | 16:43 |
| clarkb | jeblair: yes | 16:43 |
| SpamapS | interesting | 16:43 |
| SpamapS | juju has grown a "null" provider for installing juju onto existing machines | 16:44 |
| clarkb | jeblair: looks like it @tags:"console.html" AND ( @message:"Finished: SUCCESS" OR @message:"Finished: FAILURE" ) AND @fields.build_name:"gate-nova-python27" AND @fields.build_uuid:"de7652e2fe2e4532a23a43ffbfd1138a" ran on jenkins01 | 16:44 |
| SpamapS | jeblair: IIRC you guys asked for that at the San Francisco summit. | 16:44 |
| SpamapS | oh, crisis mode | 16:44 |
| * SpamapS avoids interjecting :) | 16:44 | |
| *** senk has joined #openstack-infra | 16:45 | |
| clarkb | jeblair: @tags:"console.html" AND ( @message:"Finished: SUCCESS" OR @message:"Finished: FAILURE" ) AND @fields.build_uuid:"26de60147e2f4d82863307e370587a88" as well | 16:45 |
| jeblair | #status ok | 16:46 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 16:46 | |
| jeblair | nodepool is running again | 16:46 |
| *** senk has quit IRC | 16:46 | |
| jeblair | i'll try to clean up from events it may have missed | 16:46 |
| jeblair | i also believe there's enough data in the debug log for me to do better offline testing | 16:46 |
| clarkb | jeblair: if we want we can fiddle with making zmq queue up events for nodepool if nodepool goes away | 16:47 |
| jeblair | clarkb: ooh, interesting | 16:47 |
| clarkb | jeblair: when you create the socket on the nodepool side you set an option that indicates to zmq it should queue on the other end | 16:48 |
| jeblair | clarkb: oh that sounds even easier than i thought it would be | 16:48 |
| jeblair | clarkb: now i want a zmq plugin for gerrit | 16:48 |
| clarkb | I haven't used it yet because there doesn't seem to be a good way to tell the other end to stop queueing | 16:49 |
| clarkb | but the max queue size on the jenkins size is relatively small, 1k, so shouldn't be a major problem | 16:49 |
| clarkb | that is also changeable with a plugin udpate if 1k is too small | 16:49 |
| SpamapS | error: Couldn't resolve host 'git.openstack.org' while accessing https://git.openstack.org/openstack/tripleo-incubator/info/refs?service=git-upload-pack | 16:50 |
| SergeyLukjanov | thank you guys! | 16:50 |
| SpamapS | n/m | 16:50 |
| SpamapS | retrying worked | 16:50 |
| * SpamapS blames google's forward resolver | 16:51 | |
| SergeyLukjanov | clarkb, I've updated https://review.openstack.org/#/c/49167/ to publish only image-elements to pypi | 16:51 |
| SergeyLukjanov | like tripleo-image-elements | 16:51 |
| *** erfanian has quit IRC | 16:51 | |
| jeblair | ok, i did 'nodepool delete' for all the jenkins nodes that were marked offline; it should slowly work through that list | 16:52 |
| fungi | .gitreview updates proposed https://review.openstack.org/#/q/status:open+branch:master+topic:gitreview,n,z | 16:53 |
| fungi | though looks like *tuskar* have test issues atm | 16:53 |
| SergeyLukjanov | fungi, thx! | 16:54 |
| fungi | well, except tuskar-ui which isn't tested | 16:54 |
| fungi | SergeyLukjanov: my pleasure | 16:54 |
| jeblair | fungi, SpamapS, lifeless: istr something about working through a requirements issue with tuskar now that it's subject to the mirror? | 16:54 |
| fungi | yeah, that sounds familiar. looks like that's what they're bumping into | 16:55 |
| SpamapS | jeblair: do you have evidence I can take a look at? | 16:55 |
| fungi | SpamapS: sure, http://logs.openstack.org/01/49901/1/check/gate-tuskar-pep8/49532b4/console.html and http://logs.openstack.org/00/49900/1/check/gate-python-tuskarclient-pep8/b99fa9b/console.html | 16:55 |
| fungi | have fun chewing on those | 16:56 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/reviewstats: Savanna has been moved to openstack org https://review.openstack.org/49903 | 16:56 |
| * SpamapS sighs | 16:57 | |
| SpamapS | boto | 16:57 |
| clarkb | SergeyLukjanov: thank you for that (I can never remember which non python projects we actually want to be pypi'd and so on) | 16:58 |
| SpamapS | fungi: ty | 16:58 |
| SergeyLukjanov | clarkb, my bad with savanna-extra | 16:58 |
| jeblair | nodepool seems to be operating normally. puppet is disabled on the host because of the manual revert | 17:00 |
| jeblair | #status log puppet disabled on nodepool due to manually reverting gearman change | 17:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove unused savanna rtfd jobs https://review.openstack.org/49168 | 17:01 |
| SergeyLukjanov | is it ok now to add python-savannaclient to global requirements? | 17:02 |
| jeblair | SergeyLukjanov: i think so. | 17:03 |
| jeblair | I'm going to go saturday now. | 17:03 |
| SergeyLukjanov | thank you, have a good day | 17:04 |
| fungi | i'll be around but mostly quiet and doing projects around the house | 17:06 |
| *** klrmn has left #openstack-infra | 17:07 | |
| *** garyk has joined #openstack-infra | 17:11 | |
| *** senk has joined #openstack-infra | 17:16 | |
| annegentle_ | hey | 17:18 |
| annegentle_ | working on the workshop at Grace Hopper, seeing an ssh error for one person that I can't quite figure out, anyone have a minute? | 17:18 |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/nodepool: Add fixed_network parameter to createServer https://review.openstack.org/49904 | 17:19 |
| *** mestery has joined #openstack-infra | 17:21 | |
| *** klrmn has joined #openstack-infra | 17:22 | |
| *** senk has quit IRC | 17:23 | |
| clarkb | annegentle_: sure | 17:23 |
| clarkb | annegentle_: can you paste output? | 17:23 |
| clarkb | I discovered the tokyogastronomy videos on youtube. I am very hungry right now | 17:24 |
| *** kong has quit IRC | 17:30 | |
| annegentle_ | clarkb: it's Permission denied (publickey) | 17:31 |
| annegentle_ | clarkb: before that, "git remote update gerrit" | 17:33 |
| annegentle_ | clarkb: nom nom nom | 17:33 |
| *** michchap has quit IRC | 17:33 | |
| annegentle_ | clarkb: sorry have to go get lunch now but will back, the username is mmoconner23 | 17:34 |
| clarkb | annegentle_: can the user do `ssh -p29418 mmoconner23@review.openstack.org gerrit ls-projects` if that works we may have a git review config issue and if it doesn't then a gerrit config issue | 17:36 |
| *** michchap has joined #openstack-infra | 17:37 | |
| SergeyLukjanov | clarkb, the same command works ok for me | 17:40 |
| clarkb | SergeyLukjanov: it should. Trying to sort out if keys are configured for mmoconner23 in gerrit | 17:40 |
| *** garyk has quit IRC | 18:07 | |
| *** erfanian has joined #openstack-infra | 18:31 | |
| annegentle_ | clarkb: k, that doesn't work so it's a gerrit config issue. we've looked at her settings and the key matches, what else should I look for? | 18:39 |
| clarkb | annegentle_: double check the username at https://review.openstack.org/#/settings/ | 18:39 |
| annegentle_ | clarkb: oh she doesn't have a username! | 18:39 |
| clarkb | annegentle_: you can fill one in, pick wisely as changing it is well we don't want to change it | 18:40 |
| SpamapS | fungi: | 18:42 |
| SpamapS | err | 18:42 |
| SpamapS | sorry.. so anyway.. .gitreview is broken and thus it is complicated to also remove boto==2.5.1 from tuksar's requirements.txt | 18:42 |
| SpamapS | do we force-merge the .gitreview change, or update .gitreview in the same change as the requirements.txt | 18:43 |
| SpamapS | I think the latter | 18:43 |
| clarkb | SpamapS: combine them if you have to, or just get the requirements.txt change in first | 18:43 |
| clarkb | SpamapS: if you have the .gitreview file locally git review will work, it does not need to be committed to the repo | 18:44 |
| SpamapS | ah doesn't git review whine about that? | 18:45 |
| clarkb | nope, its just looking for the file on disk | 18:45 |
| mordred | and even then it only looks for it if you don't have a gerrit remote configured | 18:46 |
| mordred | it's a completely optional file | 18:46 |
| SpamapS | https://review.openstack.org/49920 | 18:46 |
| SpamapS | fixes both | 18:46 |
| mordred | SpamapS: nicely done | 18:47 |
| lifeless | jeblair: for it to be part of the integrated release yes | 19:01 |
| lifeless | jeblair: oh blah, I failed to read ;). Yes, fixups to do. | 19:02 |
| mordred | lifeless: reading is hard | 19:02 |
| * fungi reads | 19:17 | |
| * fungi agrees... hard | 19:17 | |
| lifeless | are there any other uses of the tuskar-core group now? | 19:18 |
| lifeless | can we gc it? | 19:18 |
| *** CaptTofu has quit IRC | 19:30 | |
| *** CaptTofu has joined #openstack-infra | 19:31 | |
| fungi | lifeless: i can empty and hide it | 19:33 |
| fungi | gerrit doesn't have a "delete group" function (or a delete anything function it seems) | 19:34 |
| fungi | eventually i intend to figure out whether there are any particularly messy tie-ins to other tables, or whether deleting in the db would be trivial | 19:35 |
| lifeless | fungi: if it's not in use, please do that. | 19:38 |
| lifeless | fungi: the patch I put forward should have updated all uses of it to use tripleo-core instead | 19:38 |
| fungi | yup, seems to have no other use in acls or in the db, so i'll make it out-of-sight-out-of-mind | 19:39 |
| lifeless | danke | 19:39 |
| *** DennyZhang has joined #openstack-infra | 19:54 | |
| *** Ryan_Lane has joined #openstack-infra | 20:07 | |
| *** CaptTofu has quit IRC | 20:31 | |
| *** CaptTofu has joined #openstack-infra | 20:31 | |
| *** qba73 has quit IRC | 20:45 | |
| *** SergeyLukjanov has quit IRC | 20:50 | |
| *** qba73 has joined #openstack-infra | 21:01 | |
| *** adalbas has quit IRC | 21:15 | |
| *** AlexF has joined #openstack-infra | 21:37 | |
| *** AlexF has quit IRC | 21:47 | |
| *** sarob has joined #openstack-infra | 21:53 | |
| *** DennyZhang has quit IRC | 21:54 | |
| pabelanger | Excellent! Few more patches for nodepool and my launcher is basically working | 21:56 |
| mordred | pabelanger: neat! | 21:57 |
| *** dcramer has joined #openstack-infra | 22:01 | |
| pabelanger | mordred, I'll start dancing once I bootstrap masters then bootstrap slave, registering with master | 22:07 |
| *** AlexF has joined #openstack-infra | 22:08 | |
| *** AlexF has quit IRC | 22:21 | |
| *** DennyZhang has joined #openstack-infra | 22:29 | |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/nodepool: Allow kwargs to be passed into createServer https://review.openstack.org/49930 | 22:46 |
| *** svarnau has joined #openstack-infra | 23:18 | |
| *** svarnau has quit IRC | 23:19 | |
| *** DennyZhang has quit IRC | 23:27 | |
| *** dkliban has joined #openstack-infra | 23:30 | |
| *** alexpilotti has quit IRC | 23:30 | |
| *** sarob has quit IRC | 23:42 | |
| *** sarob has joined #openstack-infra | 23:42 | |
| *** sarob has quit IRC | 23:47 | |
| *** senk has joined #openstack-infra | 23:52 | |
| *** senk has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!