Tuesday, 2015-12-22

fungizaro: see what is in that file for the dc_local_interfaces value00:00
fungichances are it's set to 'lo' or something00:00
*** salv-orlando has joined #openstack-infra00:00
jeblairzaro: why not run it on the same server?00:01
*** irenab has joined #openstack-infra00:02
zarojeblair: yeah, got it working from my mac (dev machine) but thought it would be nicer to have a dedicated server for that, where i can connect gerrit from other machines.00:02
fungiand yeah, if you just need to configure an smtp relay (smarthost) for a gerrit service, you can do that locally on your gerrit server (that's what we do on ours)00:02
*** piet has quit IRC00:02
*** markvoelker has joined #openstack-infra00:02
jeblairzaro: also, be aware that if you're planning on having the server accept mail from any host, unauthenticated, and then send that mail off to other hosts rather than delivering locally, that particular configuration is called an 'open relay', and is a really bad idea.  so make sure it either only delivers locally or only accepts authenticated connections.00:02
fungioh, you're running gerrit on your mac and don't want to configure the mac as a smart host?00:02
*** salv-orl_ has quit IRC00:03
*** irenab has quit IRC00:03
zarofungi: sorry, i don't know the term 'smart host.00:03
fungizaro: common industry term for an smtp relay00:03
zaroalright i guess jeblair says bad idea so i'll just decomission and use the local one i've setup.00:04
fungiyeah, if you do set it up to receive and forward e-mail, then you should either set it up to only accept authenticated submissions (over tls encryption), or restrict the source ip addresses which it will allow to relay through00:04
*** apuimedo has quit IRC00:04
fungiand of course test it to make sure you can't send through it from random places without authentication, because open relays are a bad scene00:04
*** apuimedo_ has quit IRC00:05
zarogot it, bad idea!00:05
*** apuimedo has joined #openstack-infra00:06
fungiyou can restrict the source ip addresses allowed to relay with the dc_relay_nets in that file, and you can also restrict what destination domains to which you'll allow relaying with the dc_relay_domains option00:06
*** apuimedo_ has joined #openstack-infra00:06
*** admin0 has quit IRC00:07
*** markvoelker has quit IRC00:07
*** sdake has quit IRC00:07
zarofungi: yeah, i see that but might as well just setup local smtp if i have to maintain that.  so i'll just stick with the localhost setup.00:07
fungibut to jeblair's earlier point, the reason it sets itself up by default to only listen on the loopback interface is because of years of the internet struggling to stamp out people cluelessly running open relays without realizing they were doing it00:07
*** piet has joined #openstack-infra00:08
fungione of those "i hope you know what you're getting yourself into" configuration steps00:08
* zaro is really clueless00:08
*** irenab has joined #openstack-infra00:09
zarowell learned my lesson after only a few hours of mucking :)00:09
fungirunning mailservers is pretty esoteric, especially these days, so it's pretty much expected that most people, even many sysadmins, may be clueless about it00:09
lifelesssudo apt-get install clueless-smtpd ?00:10
lifelessshould be a thing00:11
*** irenab has quit IRC00:11
*** rbrndt has quit IRC00:11
*** coolsvap|away has quit IRC00:12
*** xarses has quit IRC00:12
*** otter768 has joined #openstack-infra00:12
*** jswarren_ has joined #openstack-infra00:12
*** sdake has joined #openstack-infra00:12
*** asselin has quit IRC00:13
*** apuimedo_ has quit IRC00:13
*** BobBall_AWOL has quit IRC00:14
*** jswarren has quit IRC00:14
*** BobBall_AWOL has joined #openstack-infra00:14
*** notmorgan has quit IRC00:14
*** coolsvap|away has joined #openstack-infra00:14
*** apuimedo_ has joined #openstack-infra00:15
*** zeih has quit IRC00:15
*** zz_dimtruck is now known as dimtruck00:19
*** apuimedo_ has quit IRC00:19
docaedoDoes a change to jenkins-slave manifest (like https://review.openstack.org/#/c/260230 for instance) take effect relatively soon after merge?00:19
*** apuimedo has quit IRC00:20
*** apuimedo has joined #openstack-infra00:20
*** apuimedo_ has joined #openstack-infra00:21
*** Qiming has joined #openstack-infra00:22
*** weshay_xchat has quit IRC00:23
fungiClint: yep, officially cynical in my old age i guess00:26
*** tzn has joined #openstack-infra00:27
Clintfungi: this is why we can't have nice anti-spam software00:27
*** EricGonczer_ has quit IRC00:27
*** irenab has joined #openstack-infra00:28
*** Jay__ has joined #openstack-infra00:28
*** irenab has quit IRC00:29
* fungi has nice anti-spam software00:29
fungiat least spamassassin is working pretty well for me anyway00:29
*** mudassirlatif has quit IRC00:31
*** apuimedo has quit IRC00:32
*** eliqiao has joined #openstack-infra00:33
*** apuimedo has joined #openstack-infra00:33
*** eliqiao_ has quit IRC00:35
*** gildub has quit IRC00:36
*** apuimedo_ has quit IRC00:37
*** apuimedo has quit IRC00:38
*** apuimedo has joined #openstack-infra00:38
*** apuimedo_ has joined #openstack-infra00:39
*** dims has quit IRC00:42
jeblairdocaedo: should within 30m or so...00:42
*** apuimedo has quit IRC00:43
docaedojeblair: cool, thanks - now mordred's patch just needs a little love :)00:43
*** apuimedo_ has quit IRC00:43
*** notmorgan has joined #openstack-infra00:44
*** notmorgan has quit IRC00:44
*** notmorgan has joined #openstack-infra00:44
*** apuimedo has joined #openstack-infra00:44
*** irenab has joined #openstack-infra00:45
*** apuimedo_ has joined #openstack-infra00:45
*** irenab has quit IRC00:45
*** xarses has joined #openstack-infra00:45
*** asselin has joined #openstack-infra00:47
SpamapSis it just me, or is login.launchpad.net down?00:49
*** EricGonczer_ has joined #openstack-infra00:49
lifelessSpamapS: what?00:49
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure the Gerrit's sendmail.includeDiff option  https://review.openstack.org/26028300:49
lifelessSpamapS: that isn't a host, is it?00:49
SpamapSlifeless: it's the one that gerrit sends me to00:49
*** apuimedo_ has quit IRC00:50
SpamapSfor my https://launchpad.net/~clint-fewbar openid url00:50
lifelesshmm, LP should really update its openid redirects00:50
lifelessits up00:50
SpamapSoh yeah it worked now00:50
*** irenab has joined #openstack-infra00:50
SpamapSlifeless: I think it actually goes to login.ubuntu.com now00:50
*** salv-orlando has quit IRC00:51
SpamapShm still different IPs00:51
lifelessyes, because old ssl stacks00:51
*** apuimedo_ has joined #openstack-infra00:51
*** hdd has quit IRC00:51
*** apuimedo has quit IRC00:51
*** irenab has quit IRC00:51
*** apuimedo has joined #openstack-infra00:52
*** apuimedo_ has quit IRC00:56
*** apuimedo has quit IRC00:57
openstackgerritBoris Pavlovic proposed openstack-infra/project-config: Add experimental job to check Rally installation on Centos 7  https://review.openstack.org/26025800:57
*** apuimedo has joined #openstack-infra00:58
*** eil397 has quit IRC00:58
*** apuimedo_ has joined #openstack-infra00:58
openstackgerritKhai Do proposed openstack-infra/system-config: Include unified diffs in Gerrit notification emails  https://review.openstack.org/26028400:59
*** irenab has joined #openstack-infra00:59
*** tzn has quit IRC01:01
*** irenab has quit IRC01:02
*** apuimedo_ has quit IRC01:03
*** apuimedo_ has joined #openstack-infra01:04
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure the Gerrit's sendmail.includeDiff option  https://review.openstack.org/26028301:06
*** apuimedo has quit IRC01:07
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure the Gerrit's sendmail.includeDiff option  https://review.openstack.org/26028301:07
*** angdraug has quit IRC01:07
openstackgerritKhai Do proposed openstack-infra/system-config: Include unified diffs in Gerrit notification emails  https://review.openstack.org/26028401:08
*** Swami has quit IRC01:08
*** irenab has joined #openstack-infra01:08
*** apuimedo has joined #openstack-infra01:08
*** irenab has quit IRC01:08
*** apuimedo_ has quit IRC01:09
*** sdake has quit IRC01:10
*** apuimedo_ has joined #openstack-infra01:10
*** baoli has joined #openstack-infra01:11
*** irenab has joined #openstack-infra01:12
*** Jay__ has quit IRC01:13
*** Jay__ has joined #openstack-infra01:13
*** apuimedo has quit IRC01:15
*** baoli has quit IRC01:16
*** dims has joined #openstack-infra01:16
*** apuimedo has joined #openstack-infra01:16
*** zeih has joined #openstack-infra01:16
fungiokay, well i'm just going to take a leap of faith in 260213 and then go through the manual jjb updates everywhere01:16
*** apuimedo_ has quit IRC01:17
fungijhesketh: ^ just a heads up this is happening in your daytime, pass it along01:17
*** zhenguo has joined #openstack-infra01:17
*** apuimedo_ has joined #openstack-infra01:17
fungiadded a note in my approval comment about what i'll be running where, in case anyone needs to revert and redo01:19
*** lykinsbd has quit IRC01:20
*** zeih has quit IRC01:21
*** apuimedo has quit IRC01:21
*** lykinsbd has joined #openstack-infra01:22
anteayafungi: sounds good01:22
*** apuimedo_ has quit IRC01:22
anteayaI have to be up in about 6 hours so I'm going to go to bed now01:22
*** apuimedo has joined #openstack-infra01:22
anteayaI'll read backscroll when I'm awake again01:23
*** apuimedo_ has joined #openstack-infra01:23
*** lykinsbd_ has joined #openstack-infra01:23
tonybI saw this come up somewhere but what does01:23
tonyb"NOT_REGISTERED" mean from a Jenkins POV?01:24
openstackgerritMerged openstack-infra/project-config: Get rid of precise||trusty job node declarations  https://review.openstack.org/26021301:25
*** lykinsbd has quit IRC01:27
*** PsionTheory has joined #openstack-infra01:27
fungitonyb: means that zuul couldn't find a registration for the job it wanted run, possibly because the job hadn't been created on the jenkins masters yet (they tend to update more slowly than zuul does) or because it's looking for it in combination with the wrong node type (260213 in combination with its child change seek to address that second complexity)01:27
*** Daisy has joined #openstack-infra01:28
*** gyee has quit IRC01:29
tonybfungi: Thanks.01:29
*** sputnik1_ has quit IRC01:30
funginormally zuul passes along jenkins job states (success, fail, unstable, abort) but in this case it's made up one because jenkins doesn't have a "state" for nonexistent jobs (for obvious reasons)01:32
fungiwell, s/state/result/01:32
*** irenab has joined #openstack-infra01:33
*** irenab has quit IRC01:33
*** apuimedo_ has quit IRC01:34
tonybfungi: okay.  I'll look into when the job.  It's a f23 job so I don't think the work you're doing for precise is related.01:35
*** apuimedo_ has joined #openstack-infra01:35
*** sputnik13 has joined #openstack-infra01:35
tonybAhh I see someone recently switched from f21 -> f2301:35
*** larainema has quit IRC01:36
*** jyuso1 has joined #openstack-infra01:36
*** apuimedo has quit IRC01:36
tonybfungi: Thanks again.01:36
*** irenab has joined #openstack-infra01:36
*** irenab has quit IRC01:36
fungiyeah, and we've been having trouble building fedora 23 workers01:36
fungiianw has some updates to diskimage-builder which should get that going01:37
ianwyep, should be working soon!01:37
* ianw is a bit sick of saying that, but i believe it, sort of, now01:37
*** irenab has joined #openstack-infra01:38
*** irenab has quit IRC01:38
*** apuimedo has joined #openstack-infra01:38
*** larainema has joined #openstack-infra01:39
*** Daisy has quit IRC01:39
*** Daisy has joined #openstack-infra01:40
*** lykinsbd has joined #openstack-infra01:40
*** dims has quit IRC01:40
*** apuimedo_ has quit IRC01:42
*** PsionTheory_ has joined #openstack-infra01:42
*** dimtruck is now known as zz_dimtruck01:43
*** apuimedo_ has joined #openstack-infra01:43
*** asselin__ has joined #openstack-infra01:43
*** lykinsbd_ has quit IRC01:43
*** lykinsbd_ has joined #openstack-infra01:44
*** gongysh_ has joined #openstack-infra01:44
*** Daisy has quit IRC01:44
*** triggerwarning has joined #openstack-infra01:45
*** asselin_ has quit IRC01:45
*** lykinsbd_ has quit IRC01:45
*** triggerwarning has quit IRC01:45
*** triggerwarning has joined #openstack-infra01:45
*** PsionTheory has quit IRC01:45
*** TongMaster is now known as ruagair01:45
*** ruagair is now known as Guest5693101:46
*** Daisy has joined #openstack-infra01:46
*** lykinsbd has quit IRC01:46
*** Qiming has quit IRC01:47
*** EricGonczer_ has quit IRC01:47
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Include query results in graph  https://review.openstack.org/26018801:48
*** Guest56931 is now known as TongMaster01:48
*** jerryz has quit IRC01:49
*** oomichi_away is now known as oomichi01:49
*** TongMaster is now known as craige01:49
*** zz_dimtruck is now known as dimtruck01:50
*** dims has joined #openstack-infra01:50
*** Daisy has quit IRC01:51
*** gongysh_ has quit IRC01:51
*** irenab has joined #openstack-infra01:52
openstackgerritJulia Kreger proposed openstack-infra/project-config: Build a Debian simple-init images  https://review.openstack.org/25489201:52
ianwgreghaynes / SpamapS : appreciate any comments on -> https://www.technovelty.org/openstack/platform-bootstrap-for-openstack-infrastructure.html01:53
*** mriedem_away has quit IRC01:53
*** irenab has joined #openstack-infra01:54
greghaynesianw: oooo01:54
*** apuimedo_ has quit IRC01:54
*** apuimedo has quit IRC01:54
*** irenab has quit IRC01:54
*** apuimedo has joined #openstack-infra01:55
*** apuimedo_ has joined #openstack-infra01:55
*** mriedem has joined #openstack-infra01:58
*** apuimedo_ has quit IRC02:00
*** apuimedo_ has joined #openstack-infra02:00
*** apuimedo has quit IRC02:02
*** apuimedo has joined #openstack-infra02:03
*** markvoelker has joined #openstack-infra02:03
*** dims has quit IRC02:06
*** apoorvad has quit IRC02:07
*** markvoelker has quit IRC02:07
*** apuimedo has quit IRC02:07
*** apuimedo_ has quit IRC02:07
*** Daisy has joined #openstack-infra02:08
fungiokay, looks like it's been long enough for 260213 to be on the jenkins masters according to my grep of the local config copies on them, so i'm starting manual job update runs on all 8 now02:08
*** apuimedo has joined #openstack-infra02:08
fungiwill keep tabs on them over the next few hours and restart any as needed until they all complete02:08
*** apuimedo_ has joined #openstack-infra02:08
*** [1]Thelo has joined #openstack-infra02:11
*** zhurong has joined #openstack-infra02:11
*** apuimedo has quit IRC02:13
*** Thelo has quit IRC02:13
*** [1]Thelo is now known as Thelo02:13
*** apuimedo has joined #openstack-infra02:13
* craige notes it's after midday on his last working day of the year and opens a beer. Happy holidays!02:14
*** sdake has joined #openstack-infra02:16
*** Qiming has joined #openstack-infra02:17
fungicheers, craige!02:17
fungitry not to break anything on your way out02:17
*** zeih has joined #openstack-infra02:17
*** apuimedo has quit IRC02:18
*** apuimedo_ has quit IRC02:18
*** apuimedo has joined #openstack-infra02:19
*** apuimedo_ has joined #openstack-infra02:20
*** zeih has quit IRC02:22
*** apuimedo has quit IRC02:24
*** apuimedo has joined #openstack-infra02:26
*** apuimedo_ has quit IRC02:27
*** apuimedo_ has joined #openstack-infra02:28
*** dims has joined #openstack-infra02:29
*** piet has quit IRC02:30
*** Daisy has quit IRC02:30
*** Daisy has joined #openstack-infra02:31
*** hichihara has quit IRC02:32
*** bardia has quit IRC02:32
*** apuimedo_ has quit IRC02:33
*** apuimedo has quit IRC02:33
*** apuimedo has joined #openstack-infra02:34
*** sdake_ has joined #openstack-infra02:34
*** apuimedo_ has joined #openstack-infra02:34
*** sdake has quit IRC02:34
*** Daisy has quit IRC02:35
*** unicell1 has quit IRC02:37
craigeI'm going to find who wrote launchpadlib and give them the longest, most uncomfortable hug ever.02:38
*** apuimedo has quit IRC02:38
*** harlowja_at_home has joined #openstack-infra02:39
*** apuimedo has joined #openstack-infra02:39
*** zzzeek has quit IRC02:44
*** apuimedo has quit IRC02:44
*** apuimedo_ has quit IRC02:44
*** apuimedo has joined #openstack-infra02:44
*** apuimedo_ has joined #openstack-infra02:45
*** dims has quit IRC02:45
*** zzzeek has joined #openstack-infra02:45
*** dims has joined #openstack-infra02:47
*** apuimedo has quit IRC02:51
*** apuimedo_ has quit IRC02:52
*** apuimedo has joined #openstack-infra02:53
*** apuimedo_ has joined #openstack-infra02:53
*** salv-orlando has joined #openstack-infra02:55
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Update query for nova db deadlock bug 1480305  https://review.openstack.org/26030002:55
openstackbug 1480305 in OpenStack Compute (nova) "DBDeadlock inserting into instance_extra" [Medium,Confirmed] https://launchpad.net/bugs/148030502:55
*** apuimedo_ has quit IRC02:57
*** tzn has joined #openstack-infra02:58
*** apuimedo_ has joined #openstack-infra02:59
*** apuimedo has quit IRC02:59
*** apuimedo has joined #openstack-infra03:01
*** appprod0 has joined #openstack-infra03:01
*** appprod0 has left #openstack-infra03:01
*** appprod0 has joined #openstack-infra03:02
*** tzn has quit IRC03:03
*** apuimedo_ has quit IRC03:03
*** markvoelker has joined #openstack-infra03:04
*** apuimedo_ has joined #openstack-infra03:04
*** baoli has joined #openstack-infra03:05
*** apuimedo has quit IRC03:05
*** rvasilets___ has quit IRC03:05
*** links has joined #openstack-infra03:06
*** apuimedo has joined #openstack-infra03:06
openstackgerritMerged openstack-infra/elastic-recheck: Update query for nova db deadlock bug 1480305  https://review.openstack.org/26030003:07
openstackbug 1480305 in OpenStack Compute (nova) "DBDeadlock inserting into instance_extra" [Medium,Confirmed] https://launchpad.net/bugs/148030503:07
*** baoli has quit IRC03:08
*** dims has quit IRC03:08
*** markvoelker has quit IRC03:08
*** harlowja_at_home has quit IRC03:09
*** harlowja_at_home has joined #openstack-infra03:09
*** apuimedo has quit IRC03:10
*** apuimedo has joined #openstack-infra03:12
*** piet has joined #openstack-infra03:13
*** ramineni has joined #openstack-infra03:13
*** yamahata has quit IRC03:14
*** harlowja_at_home has quit IRC03:14
*** thorst has joined #openstack-infra03:14
*** thorst has quit IRC03:16
*** apuimedo_ has quit IRC03:16
*** greghaynes has quit IRC03:16
*** thorst has joined #openstack-infra03:16
*** apuimedo_ has joined #openstack-infra03:17
*** greghaynes has joined #openstack-infra03:17
*** mriedem has quit IRC03:19
*** thorst has quit IRC03:20
*** gongysh_ has joined #openstack-infra03:21
*** coolsvap|away is now known as coolsvap03:21
*** vilobhmm111 has quit IRC03:21
*** apuimedo_ has quit IRC03:21
*** apuimedo_ has joined #openstack-infra03:22
fungijenkins03-7 took around 1.25 hours to finish updating, 01-2 are still underway03:25
*** rlandy has quit IRC03:26
*** apuimedo_ has quit IRC03:26
*** apuimedo has quit IRC03:27
*** apuimedo has joined #openstack-infra03:27
*** baoli has joined #openstack-infra03:27
*** apuimedo_ has joined #openstack-infra03:27
*** baoli has quit IRC03:28
*** piet has quit IRC03:28
*** Daisy has joined #openstack-infra03:31
*** apuimedo has quit IRC03:31
*** apuimedo has joined #openstack-infra03:33
*** baoli has joined #openstack-infra03:34
*** salv-orlando has quit IRC03:35
*** baoli has quit IRC03:35
*** harlowja_at_home has joined #openstack-infra03:39
*** apuimedo_ has quit IRC03:39
*** apuimedo_ has joined #openstack-infra03:40
*** Daisy has quit IRC03:40
*** apuimedo has quit IRC03:40
janonymousCan someone review a Hacking patch : https://review.openstack.org/#/c/259729/03:40
*** Daisy has joined #openstack-infra03:41
*** apuimedo has joined #openstack-infra03:41
*** apuimedo_ has quit IRC03:45
*** Daisy has quit IRC03:45
*** apuimedo has quit IRC03:45
*** apuimedo has joined #openstack-infra03:46
*** apuimedo_ has joined #openstack-infra03:47
*** gildub has joined #openstack-infra03:48
*** zhurong has quit IRC03:48
fungijenkins01 is done, 02 still going03:50
*** apuimedo_ has quit IRC03:51
*** apuimedo has quit IRC03:53
*** apuimedo has joined #openstack-infra03:53
*** apuimedo_ has joined #openstack-infra03:54
*** gongysh_ has quit IRC03:55
*** apuimedo has quit IRC03:58
*** apuimedo has joined #openstack-infra03:59
*** irenab has joined #openstack-infra03:59
*** tzn has joined #openstack-infra03:59
*** apuimedo_ has quit IRC04:00
*** irenab has quit IRC04:00
*** apuimedo_ has joined #openstack-infra04:01
*** apuimedo has quit IRC04:03
*** tzn has quit IRC04:04
fungijanonymous: you're more likely to find reviewers for the openstack-dev/hacking repo active in #openstack-qa rather than here, since it's under the control of the qa team http://governance.openstack.org/reference/projects/quality-assurance.html04:04
*** apuimedo has joined #openstack-infra04:04
*** irenab has joined #openstack-infra04:06
*** PsionTheory_ has quit IRC04:06
*** irenab has quit IRC04:06
*** apuimedo_ has quit IRC04:06
*** apuimedo_ has joined #openstack-infra04:06
*** irenab has joined #openstack-infra04:10
*** irenab has quit IRC04:10
*** apuimedo_ has quit IRC04:11
fungijenkins02 done (2 hours 5 minutes wall clock time)04:13
sputnik13is there a way to get the demo or admin user credentials in devstack short of sourcing openrc?04:13
sputnik13I mean as part of a plugin04:13
sputnik13not as a user04:13
*** apuimedo_ has joined #openstack-infra04:13
*** irenab has joined #openstack-infra04:14
janonymousfungi: hey thanks, i would ask for review in there04:14
fungiremoved my -2 from 260214 now that the jjb manual updates are all completed04:16
*** irenab has joined #openstack-infra04:17
*** apuimedo_ has quit IRC04:18
*** irenab has quit IRC04:18
*** apuimedo has quit IRC04:19
*** zeih has joined #openstack-infra04:19
*** apuimedo has joined #openstack-infra04:20
*** apuimedo_ has joined #openstack-infra04:20
*** Thelo has quit IRC04:22
*** Ramanjaneya has joined #openstack-infra04:24
clarkbsputnik13 read the vars directly?04:24
*** fawadkhaliq has joined #openstack-infra04:25
*** zeih has quit IRC04:25
sputnik13right, the obvious answer :)04:26
sputnik13except I see ADMIN_PASSWORD, but don't see a DEMO_PASSWORD04:26
clarkbthey are the same iirc04:26
sputnik13must have tunnel vision, don't see it >_<04:26
*** apuimedo has quit IRC04:27
*** apuimedo_ has quit IRC04:27
*** irenab has joined #openstack-infra04:27
sputnik13duh, ok openrc actually has a comment "#   Set ADMIN_PASSWORD to set the password for 'admin' and 'demo'"04:27
*** irenab has quit IRC04:27
*** apuimedo has joined #openstack-infra04:28
*** apuimedo_ has joined #openstack-infra04:28
clarkbfungi still around?04:30
clarkbI can approve 260214 but not really babysit04:30
fungiclarkb: yeah, were 260213/260214 sane?04:30
fungii can babysit for another couple hours04:30
clarkb(tending to real life babies)04:30
clarkb260214 looks sane04:30
fungialso trying to make sure jhesketh is aware, and yolanda once she wakes up04:31
* jhesketh is here04:31
fungijust in case any of this needs undoing because zomgsobroken04:31
jheskethI saw 260213 go in and look okay04:32
fungias opposed to a job here or there which was somehow using the wrong label or something04:32
clarkb+2'd approve at will04:32
jheskethlooking at 214 now04:32
*** irenab has joined #openstack-infra04:32
fungii'll let jhesketh take a peek before i do04:32
fungithat one's short at least04:32
fungithe parent change was freaking huge, but also likely to merge conflict for all eternity if we didn't approve quickly04:33
*** irenab has quit IRC04:33
*** nikhil has joined #openstack-infra04:33
*** fawadkhaliq has quit IRC04:34
*** irenab has joined #openstack-infra04:36
*** vilobhmm11 has joined #openstack-infra04:36
*** nikhil_k has quit IRC04:36
*** apuimedo has quit IRC04:37
*** apuimedo_ has quit IRC04:37
*** irenab has quit IRC04:37
*** apuimedo has joined #openstack-infra04:38
jheskethfungi: with that change I'm confused how devstack gets the right node now04:38
*** apuimedo_ has joined #openstack-infra04:39
fungijhesketh: er, because the nodes that can run dsvm jobs are the ones which register them?04:39
fungior do we have a hole there?04:40
jheskethhmm I haven't looked at the node allocation in ages... I remember the jenkins/gearman/zuul stuff being a little bit magic04:40
raminenihi , can someone do quick review on https://review.openstack.org/#/c/255654/ , it has 2 +2ś already04:40
fungiright now the jjb configs for them should only be listing node: bare-trusty04:40
jheskethin the way that the host type is added to gearman for the jenkins-gearman plugin04:40
fungier, sorry, node: devstack-trusty i mean04:41
jheskethright, it is possible we have some jobs missing that, but we can catch those04:41
*** unicell has joined #openstack-infra04:41
jheskethhow does jenkins know which slaves are which for the jobs though04:41
*** irenab has joined #openstack-infra04:41
fungiand i thought as long as zuul isn't told a specific node label to look for it just went with build:job-name instead of build:job-name:node-label04:41
fungithe parent patch of that one replaced all the "node: devstack-precise || devstack-trusty" with just "node: devstack-trusty" so there should no longer be multiple node types registering the same jobs04:43
jheskethright, zuul should do that04:44
jheskethand yes, the jobs should have their node defined in jenkins04:44
jheskethwhat I'm not sure is how does jenkins know what slave is what...04:44
jheskethis that nodepool registration that does that04:44
* jhesketh checks04:44
*** apuimedo has quit IRC04:45
fungiyeah, nodepool sets a common node label on each one as it connects it to the jenkins master as a slave04:45
*** apuimedo_ has quit IRC04:46
*** apuimedo has joined #openstack-infra04:47
*** apuimedo_ has joined #openstack-infra04:47
*** rguillebert has quit IRC04:47
fungibut definitely thanks for walking back through all this and confirming my recollections. it's a change with a large potential for impact to pretty much all jobs04:47
jheskethokay I can see that the nodes are created with the name given.. does the jenkins-gearman plugin use that failing a ZUUL_NODE from gearman?04:48
* jhesketh now checks jenkins-gearman04:48
fungiif you look at the job registrations for a job, you'll see it gets workers registered with and without the :<node-label> suffix04:49
fungiyou can see them in the gearman status output04:49
*** markvoelker has joined #openstack-infra04:49
jheskethyeah I see the suffix, I'm wondering once zuul stops giving that suffix to jenkins what it'll fall back on04:50
openstackgerritAnusha Ramineni proposed openstack-infra/project-config: congress: disable tempest plugin for stable branches  https://review.openstack.org/25879904:50
*** piet has joined #openstack-infra04:51
*** apuimedo_ has quit IRC04:52
*** apuimedo_ has joined #openstack-infra04:53
*** markvoelker has quit IRC04:54
*** apuimedo has quit IRC04:58
*** eil397 has joined #openstack-infra04:58
*** apuimedo has joined #openstack-infra05:00
*** jamespage has quit IRC05:00
*** gnuoy has quit IRC05:00
*** tzn has joined #openstack-infra05:00
*** gnuoy has joined #openstack-infra05:00
*** armax has quit IRC05:01
*** Daisy has joined #openstack-infra05:02
jheskethfungi: so I think the presence of nodelabelparameter jenkins plugin is what does the magic for us05:03
*** fawadkhaliq has joined #openstack-infra05:04
*** apuimedo has quit IRC05:04
*** tzn has quit IRC05:05
*** apuimedo_ has quit IRC05:05
*** piet has quit IRC05:05
fungiThis plugin adds two new parameter types to job configuration - node and label, this allows to dynamically select the node where a job/project should be executed.05:06
fungiyep, we're definitely using it05:06
fungiso that's what allows us to specify a job should run on a node with a particular label05:06
*** apuimedo has joined #openstack-infra05:06
fungii hadn't realized that was a plugin, just always thought it was a builtin feature05:07
*** apuimedo_ has joined #openstack-infra05:07
jheskethfungi: yep, so I've confirmed nodepool sets a label for each node it attaches to jenkins05:08
jheskeththe documentation on https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin makes it sound like a label is set on a job05:09
jheskethbut it seems like the label is set on the node and a "node" type is set on the job05:09
fungiyeah, that's basically it05:09
jheskethassuming the jjb defintion of "node" sets it correctly for the job and jenkins matches that to a node label we shoudl be all good05:09
* jhesketh is possibly just failing to grep the plugin doc05:10
funginodes are labeled, and you can optionally specify a label or the name of a specific node to which the job should be restricted05:10
jheskethfungi: either way, I'm +2 on the patch, but it certainly needs baby sitting05:10
fungiyeah, i'll stick around and fire some test jobs once it merges05:10
jheskethI was about to say I'm happy to watch it05:11
fungithe earlier in the week we have this done the more likely it is we'll catch any omissions that need fixing before i'm gone for holidays05:11
jheskethgiven it must be silly o'clock for you05:11
*** apuimedo has quit IRC05:11
fungiwell, your eye will be less bleary than mine ;)05:11
fungiapprove at will, and i'm around for an hour or two just in case05:11
*** apuimedo_ has quit IRC05:11
jheskethfungi: will do05:11
* jhesketh +w05:12
fungii also have a job which is currently not_registered which should in theory be fixed by this05:12
fungiwhich was what lit a fire under me to just get it cleaned up finally05:12
*** apuimedo has joined #openstack-infra05:12
*** apuimedo_ has joined #openstack-infra05:13
*** piet has joined #openstack-infra05:13
*** apuimedo has quit IRC05:19
*** apuimedo has joined #openstack-infra05:20
openstackgerritMerged openstack-infra/project-config: Remove precise/trusty label mapping in Zuul config  https://review.openstack.org/26021405:21
fungiand now the fun begins05:21
*** sdake_ has quit IRC05:23
tangchenHi everyone, is anybody familiar with XenProject CI in Nova ?  Please take a look this patch. https://review.openstack.org/#/c/202605/05:23
tangchenThe XenProject CI always fails to merge the patch. But I don't think it is the patch's problem. Would anyone please tell me how to recheck it ?05:24
tangchenAnyway, the gating job started. Not sure if the XenProject CI failure matters.05:26
zarotangchen: i believe 3rd party CI votes do not block approvals.05:29
tangchenzaro: Oh, I see. Thanks a lot. :)05:31
*** apuimedo_ has quit IRC05:32
*** apuimedo_ has joined #openstack-infra05:33
*** apuimedo has quit IRC05:37
*** apuimedo_ has quit IRC05:37
*** apuimedo has joined #openstack-infra05:38
*** abregman has joined #openstack-infra05:38
*** apuimedo_ has joined #openstack-infra05:38
*** salv-orlando has joined #openstack-infra05:41
*** eil397 has quit IRC05:41
*** harlowja_at_home has quit IRC05:42
*** apuimedo_ has quit IRC05:43
*** apuimedo_ has joined #openstack-infra05:44
openstackgerritGal Sagie proposed openstack-infra/project-config: Add rally-plot publisher to OVN, Dragonflow and Kuryr rally jobs  https://review.openstack.org/26033005:46
*** baoli has joined #openstack-infra05:47
*** apuimedo has quit IRC05:47
*** apuimedo_ has quit IRC05:48
*** apuimedo has joined #openstack-infra05:49
fungijhesketh: i see zuul was last reconfigured 27 minutes ago and any changes enqueued for less than that aren't getting workers assigned to their jobs05:49
fungijhesketh: looking at debug logs now05:49
*** apuimedo_ has joined #openstack-infra05:49
*** markvoelker has joined #openstack-infra05:50
*** maishsk has quit IRC05:50
*** baoli has quit IRC05:51
fungiokay, this is strange. gearman status shows build:gate-tempest-dsvm-full:devstack-trusty registered but no build:gate-tempest-dsvm-full05:52
funginevermind. grep fail05:52
jheskethfungi: chances are the full name will stick around for a while05:53
jheskethI can't recall when gearman forgets about them, but I think it might only be on restart05:53
fungibuild:gate-tempest-dsvm-full:devstack-trusty    8       8       11105:53
fungibuild:gate-tempest-dsvm-full    0       0       11105:53
jheskethwhich means we won't get any not-registered05:53
clarkbya I think it remembers forever until you kill the process05:54
fungiso it is claiming to have the same workers05:54
jheskethactually those workers will keep registering as that name05:54
fungiright, no i was expecting to see them05:54
fungii simply had a malformed grep05:54
fungiso that part looks right05:54
*** tlian has quit IRC05:54
*** zhurong has joined #openstack-infra05:55
*** markvoelker has quit IRC05:55
*** apuimedo has quit IRC05:56
*** apuimedo_ has quit IRC05:56
*** apuimedo has joined #openstack-infra05:57
*** apuimedo_ has joined #openstack-infra05:58
*** otter768 has quit IRC05:59
*** yamahata has joined #openstack-infra05:59
*** tzn has joined #openstack-infra06:01
fungiokay, i see the problem, i think06:02
*** _nadya_ has joined #openstack-infra06:04
jheskethfungi: woops, do you have a patch incoming?06:04
*** apuimedo has quit IRC06:04
jheskethotherwise I can put it togheter06:04
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Correct set_node_options argument list  https://review.openstack.org/26033406:05
fungijhesketh: ^ plain brown wrapper06:05
*** apuimedo has joined #openstack-infra06:05
*** tzn has quit IRC06:05
jheskethfungi: +w06:05
fungimuch obliged06:06
jheskethfungi: however I think we'll need to apply manually06:06
fungioh? you think the current state will prevent it from reloading its config?06:06
fungiright, it won't get tests run :/06:06
jheskethyeah because all jobs have that function applied06:06
fungii'll do that now06:06
jheskethactually we could just merge manually, not necessarily apply06:07
*** apuimedo_ has quit IRC06:07
fungii just manually changed the parameter function file. does zuul need a signal to reread it?06:07
*** apuimedo_ has joined #openstack-infra06:08
jheskethI think a reload should do it06:08
fungibut yeah, i'll go ahead and manually submit the change to merge as well06:08
jheskethwon't need a restart I don't think06:08
*** Daisy has quit IRC06:08
*** Daisy has joined #openstack-infra06:09
*** HeOS has joined #openstack-infra06:11
*** sputnik13 has quit IRC06:12
*** apuimedo_ has quit IRC06:12
*** Daisy has quit IRC06:12
*** Daisy has joined #openstack-infra06:13
*** sputnik13 has joined #openstack-infra06:14
*** apuimedo_ has joined #openstack-infra06:14
*** apuimedo has quit IRC06:14
*** apuimedo has joined #openstack-infra06:15
openstackgerritMerged openstack-infra/project-config: Correct set_node_options argument list  https://review.openstack.org/26033406:16
fungithere we go06:16
fungithey made that harder in the new change screen06:16
*** salv-orlando has quit IRC06:16
*** sputnik13 has quit IRC06:16
fungianyway, it also looks like manually applying the change on zuul.o.o and reloading its config got things lubricated06:17
fungiso changes are getting workers assigned to run their jobs again06:17
jheskethyay :-)06:17
*** sabeen has quit IRC06:17
*** sputnik13 has joined #openstack-infra06:18
funginow to try my previously broken job06:18
*** apuimedo_ has quit IRC06:19
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/26034006:19
*** sputnik13 has quit IRC06:20
*** apuimedo_ has joined #openstack-infra06:20
fungithere's a guinea pig, just in time!06:20
*** rossella_s has quit IRC06:21
*** HeOS has quit IRC06:21
*** rossella_s has joined #openstack-infra06:21
*** zeih has joined #openstack-infra06:21
*** sputnik13 has joined #openstack-infra06:22
fungijhesketh: anyway, assuming no major broken across the board, any jobs people report failing because of using a new node type probably just need their job definitions corrected06:23
jheskethyep, I hope so06:23
jheskethI wonder if they'll fail silently though06:24
jheskethin that they'll hang in zuul waiting to be picked up06:24
jheskethsince the job definition may still be there06:24
*** apuimedo has quit IRC06:24
fungii expect at least a few because this may have been hiding some sort of broken job configuration previously06:24
jheskethoh yeah, totally06:25
*** apuimedo has joined #openstack-infra06:25
*** zeih has quit IRC06:26
*** vdrok has quit IRC06:27
*** vilobhmm11 has quit IRC06:28
*** apuimedo has quit IRC06:30
openstackgerritMerged openstack-infra/project-config: Turn bluebox servers off in nodepool.  https://review.openstack.org/26024706:30
*** apuimedo_ has quit IRC06:31
*** apuimedo has joined #openstack-infra06:31
*** _nadya_ has quit IRC06:32
*** apuimedo_ has joined #openstack-infra06:33
*** pasquier-s has quit IRC06:34
*** pasquier-s has joined #openstack-infra06:34
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/26034606:37
*** vgridnev has joined #openstack-infra06:37
*** apuimedo_ has quit IRC06:37
*** apuimedo_ has joined #openstack-infra06:38
*** Jay_ has joined #openstack-infra06:38
*** apuimedo has quit IRC06:38
*** apuimedo has joined #openstack-infra06:39
*** dimtruck is now known as zz_dimtruck06:40
*** Jay__ has quit IRC06:41
*** apuimedo_ has quit IRC06:42
*** irenab has quit IRC06:43
*** hurgleburgler has quit IRC06:43
*** apuimedo_ has joined #openstack-infra06:44
*** apuimedo has quit IRC06:44
*** tangchen has quit IRC06:44
*** apuimedo has joined #openstack-infra06:44
openstackgerritAnton Tcitlionok proposed openstack-infra/jenkins-job-builder: Clarify 'trigger-parametrized-builds' description  https://review.openstack.org/25987006:46
*** apuimedo_ has quit IRC06:48
*** apuimedo_ has joined #openstack-infra06:50
*** apuimedo has quit IRC06:51
*** apuimedo has joined #openstack-infra06:52
*** maishsk has joined #openstack-infra06:52
*** apuimedo_ has quit IRC06:54
*** pahuang has quit IRC06:55
*** apuimedo_ has joined #openstack-infra06:55
*** aeng has quit IRC06:55
*** apuimedo has quit IRC06:56
*** piet has quit IRC06:57
*** apuimedo has joined #openstack-infra06:57
*** sdake has joined #openstack-infra06:57
*** coolsvap is now known as coolsvap|away06:58
openstackgerritAnton Tcitlionok proposed openstack-infra/jenkins-job-builder: Clarify 'trigger-parameterized-builds' description  https://review.openstack.org/25987007:01
*** tzn has joined #openstack-infra07:02
*** apuimedo_ has quit IRC07:02
*** AJaeger has joined #openstack-infra07:03
*** apuimedo_ has joined #openstack-infra07:03
AJaegergood morning, is everything working fine, fungi?07:04
*** coolsvap|away is now known as coolsvap07:04
*** hdd has joined #openstack-infra07:04
*** aopincar has joined #openstack-infra07:04
openstackgerritAndreas Jaeger proposed openstack-infra/jeepyb: Add docimpact tag for bugs that target project queues  https://review.openstack.org/25322907:05
*** markvoelker has joined #openstack-infra07:06
*** tzn has quit IRC07:06
*** apuimedo has quit IRC07:06
AJaegerreading backscroll, you seem to have manged fine - great!07:06
*** apuimedo has joined #openstack-infra07:07
*** markvoelker has quit IRC07:10
*** apuimedo_ has quit IRC07:10
* AJaeger does not see any UNREGISTERED jobs right now - great!07:11
*** irenab has joined #openstack-infra07:12
AJaegerI'll review now all changes for project-config and -1 those with multiple node declarations...07:12
*** apuimedo has quit IRC07:12
*** apuimedo has joined #openstack-infra07:12
*** apuimedo_ has joined #openstack-infra07:13
AJaegerjhesketh, yolanda: could you review the stack at 260202, please? it would unblock us to move to a general linters target - and unblock bandit.07:13
AJaegeryolanda: please ping me once you're around07:13
jheskethAJaeger: taking a look07:15
*** sdake has quit IRC07:15
odyssey4meanyone around to sponsor another vote on https://review.openstack.org/259922 ?07:16
*** chmouel_ is now known as chmouel07:17
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/26034007:17
jheskethAJaeger: looks good to me.. I only just noticed we're validating the jjb here.. Not sure I'm a fan of that. I'd rather see JJB handle it, perhaps with a flag to fail on unknown vars07:18
AJaegerjhesketh: Thanks! https://review.openstack.org/#/c/259480/ would be nice as well07:19
AJaegerjhesketh: we merged that validation recently. I'm happy to remove it once jjb does it.07:19
*** apuimedo has quit IRC07:19
AJaegerthe change for the validation was month up for review - as WIP07:19
AJaegerWe failed to notice typos like builder/builders etc.07:19
jheskethI'm not complaining that I missed the review, just surprised that we are doing it07:20
AJaegerAnd that broke a few jobs...07:20
*** apuimedo has joined #openstack-infra07:20
AJaegerjhesketh: that was the quickest way - if you can teach jjb about it, that would be even better...07:20
jheskethokay fair enough07:21
jheskethIf I remember tomorrow I might poke at it07:21
AJaeger260192 and 260215 are the ones I need it in from the stack - and if I hadn't used description in one, I wouldn't have needed this change ;)07:22
*** Daisy has quit IRC07:22
AJaegerjhesketh: that would be a nice christmas present ;)07:22
*** Daisy has joined #openstack-infra07:23
*** ParsectiX has joined #openstack-infra07:24
*** ParsectiX has quit IRC07:24
*** ParsectiX has joined #openstack-infra07:25
*** ParsectiX has quit IRC07:25
*** shardy has joined #openstack-infra07:25
*** ParsectiX has joined #openstack-infra07:25
*** ParsectiX has quit IRC07:26
*** ParsectiX has joined #openstack-infra07:26
*** ParsectiX has quit IRC07:27
jheskethAJaeger: lgtm07:27
jheskeththanks for always keeping all our jobs nice and tidy :-)07:27
*** Daisy has quit IRC07:27
*** binbincong has joined #openstack-infra07:28
*** ParsectiX has joined #openstack-infra07:28
AJaegerthanks, jhesketh !07:29
*** maishsk has quit IRC07:29
openstackgerritMerged openstack-infra/project-config: Fix constraint jobs branch  https://review.openstack.org/25948007:29
*** maishsk has joined #openstack-infra07:32
*** hdd has quit IRC07:32
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Do the sdist check in a virtualenv  https://review.openstack.org/25546607:33
*** ParsectiX has quit IRC07:35
AJaeger#success The node handling in our CI setup has been greatly simplified, just specify the node you want and no need for the complex logic to figure out precise or trusty. Thanks, fungi and reviewers!07:44
openstackstatusAJaeger: Added success to Success page07:44
*** apuimedo_ has quit IRC07:45
*** apuimedo_ has joined #openstack-infra07:46
AJaegerjhesketh: another cleanup review request if you have time, please: https://review.openstack.org/#/c/259814/07:50
*** apuimedo has quit IRC07:52
*** apuimedo has joined #openstack-infra07:53
*** apuimedo_ has quit IRC07:57
*** apuimedo_ has joined #openstack-infra07:59
*** fawadkhaliq has quit IRC07:59
*** mpavone has joined #openstack-infra07:59
*** otter768 has joined #openstack-infra08:00
*** gildub has quit IRC08:02
*** apuimedo has quit IRC08:02
*** tzn has joined #openstack-infra08:03
*** apuimedo has joined #openstack-infra08:03
*** apuimedo_ has quit IRC08:04
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove proposed section from gerrit ACLs  https://review.openstack.org/26037608:04
*** vilobhmm11 has joined #openstack-infra08:05
*** otter768 has quit IRC08:05
*** apuimedo_ has joined #openstack-infra08:05
*** ifarkas has joined #openstack-infra08:05
*** Hal1 has joined #openstack-infra08:06
*** gildub has joined #openstack-infra08:07
*** Daisy has joined #openstack-infra08:07
*** apuimedo has quit IRC08:07
*** apuimedo has joined #openstack-infra08:08
*** tzn has quit IRC08:08
*** e0ne has joined #openstack-infra08:11
*** apuimedo has quit IRC08:12
*** apuimedo_ has quit IRC08:12
*** apuimedo has joined #openstack-infra08:13
*** apuimedo_ has joined #openstack-infra08:13
*** yaume has joined #openstack-infra08:13
*** zhurong has quit IRC08:14
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove ACL for refs/heads/release- for mistral  https://review.openstack.org/26037908:14
*** irenab has quit IRC08:15
anteayaAJaeger: glad to hear this are moving smoothly08:15
anteayaAJaeger: do you need anything from me or may I return to sleep?08:15
yolandahi AJaeger, sorry i'm late, but i started holiday today08:16
yolandado you need something?08:16
anteayaI skimmed backscroll08:16
AJaegeranteaya: go to sleep!08:16
anteayaAJaeger: very good08:16
AJaegeranteaya: looks fine here...08:16
* anteaya does as told08:16
anteayaAJaeger: so happy to hear that08:16
odyssey4meanyone around to sponsor another vote on https://review.openstack.org/259922 in addition to that provided by AJaeger ?08:16
AJaegeryolanda: If you have holiday, no need to be here. Just wanted to bring you up to speed on an important change that fungi did yesterday: The node handling has been greatly simplified, no need anymore to say "node: precise || trusty" - or to say it differently: Just use the *one* node you need.08:17
AJaegerPlease keep that in mind when reviewing. I -1 all changes that are affected.08:17
lifelessAJaeger: how does that work ?08:18
AJaegerfungi, anteaya, and myself are expecting at least some fallout from this - so if people show up, we know where to direct them.08:18
openstackgerritNicko Glayre proposed openstack-infra/jenkins-job-builder: Removing perModuleEmail paramater management  https://review.openstack.org/25998708:18
yolandaso mostly trusty?08:18
AJaegeryolanda: yes, trusty -  or centos7.08:19
*** coolsvap is now known as coolsvap|away08:19
AJaegerlifeless: we had some logic in zuul/*functions* that set the node to precise if a job was run on stable/icehouse but to precise otherwise.08:19
AJaegerSince icehouse is end-of-life, we do not need this logic anymore and can simplify the job templates in a significant way.08:20
AJaegerMain changes are https://review.openstack.org/#/c/260213/ and https://review.openstack.org/#/c/260214/08:20
*** coolsvap|away is now known as coolsvap08:21
AJaegerlifeless: could you +2A an addition to requirements, please? https://review.openstack.org/#/c/257157/08:21
*** zeih has joined #openstack-infra08:23
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes  https://review.openstack.org/25204108:24
AJaegerthanks, lifeless !08:24
*** apuimedo has quit IRC08:25
*** apuimedo has joined #openstack-infra08:25
*** zeih has quit IRC08:27
*** maishsk has quit IRC08:28
*** e0ne has quit IRC08:28
*** unicell1 has joined #openstack-infra08:31
*** unicell has quit IRC08:31
*** apuimedo_ has quit IRC08:32
openstackgerritMerged openstack-infra/project-config: Gate requirements repo on docs  https://review.openstack.org/25981408:32
*** apuimedo_ has joined #openstack-infra08:32
*** apuimedo has quit IRC08:32
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes  https://review.openstack.org/25204108:33
*** e0ne has joined #openstack-infra08:33
*** yamamoto has joined #openstack-infra08:33
*** apuimedo has joined #openstack-infra08:33
*** yamamoto has quit IRC08:36
*** yamamoto has joined #openstack-infra08:36
AJaegerinfra-root, looking at status.openstack.org/zuul, the post queue seems stuck - and we have lots of available workers. This looks like a fallout from fungi's node change, could anybody investigate, please? I'm not seeing anything obvious directly from here08:36
yolandaAJaeger, i need to leave for a school festival just now, sorry :( but i can take a look when i come back if there is no one available08:37
*** apuimedo_ has quit IRC08:37
*** maishsk has joined #openstack-infra08:37
AJaegeryolanda: enjoy!08:37
AJaegeryolanda: no CritSit here, just the post jobs ;)08:38
yolandaok i feel less guilty then :)08:38
*** apuimedo_ has joined #openstack-infra08:38
AJaegeryolanda: this can even wait for the US to wake up if my assessment is correct - just wanted to inform everybody in case somebody has time...08:39
AJaegeryolanda: so, stop feeling guilty and enjoy yourself!08:39
yolandaleaving now then, bye!08:39
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add instructions on how to troubleshoot bots  https://review.openstack.org/26006808:39
*** salv-orlando has joined #openstack-infra08:40
AJaegerbye, yolanda !08:40
*** yamamoto has quit IRC08:41
*** gildub has quit IRC08:43
*** vgridnev has quit IRC08:43
*** egonzalez has joined #openstack-infra08:44
*** apuimedo has quit IRC08:44
*** christiana_ has quit IRC08:45
*** apuimedo has joined #openstack-infra08:45
*** Daisy has quit IRC08:46
*** Daisy has joined #openstack-infra08:47
*** markus_z has joined #openstack-infra08:47
*** abregman has quit IRC08:48
*** andreaf has quit IRC08:48
*** andreaf has joined #openstack-infra08:49
*** abregman has joined #openstack-infra08:49
*** apuimedo has quit IRC08:49
*** apuimedo has joined #openstack-infra08:50
*** Daisy has quit IRC08:51
*** salv-orl_ has joined #openstack-infra08:51
*** salv-orl_ has quit IRC08:51
*** salv-orlando has quit IRC08:51
*** salv-orlando has joined #openstack-infra08:52
*** apuimedo has quit IRC08:54
*** apuimedo has joined #openstack-infra08:55
*** salv-orl_ has joined #openstack-infra08:55
*** apuimedo_ has quit IRC08:56
*** apuimedo_ has joined #openstack-infra08:57
*** fawadkhaliq has joined #openstack-infra08:57
*** fawadkhaliq has quit IRC08:57
*** christiana_ has joined #openstack-infra08:58
*** fawadkhaliq has joined #openstack-infra08:58
*** salv-orlando has quit IRC08:58
*** apuimedo has quit IRC09:00
*** apuimedo has joined #openstack-infra09:00
*** hashar has joined #openstack-infra09:02
*** tzn has joined #openstack-infra09:04
*** admin0 has joined #openstack-infra09:04
*** apuimedo_ has quit IRC09:04
*** apuimedo_ has joined #openstack-infra09:05
*** apuimedo has quit IRC09:05
*** apuimedo has joined #openstack-infra09:06
*** markvoelker has joined #openstack-infra09:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove redundant node declarations  https://review.openstack.org/26040009:08
*** jamespage has joined #openstack-infra09:08
*** tzn has quit IRC09:09
*** _nadya_ has joined #openstack-infra09:11
*** markvoelker has quit IRC09:11
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add non-voting linters-nv job-template  https://review.openstack.org/26040209:12
jheskethAJaeger: just noticed your message09:12
AJaegerjhesketh: as I said to yolanda: No critsit AFAIU ;)09:13
jheskethcritsit sorry?09:13
*** apuimedo has quit IRC09:13
AJaegerCritical situation09:13
AJaegerSomething that needs our full attention - urgently... - this is not the case here09:14
*** apuimedo has joined #openstack-infra09:14
*** e0ne has quit IRC09:16
openstackgerritAndrey Pavlov proposed openstack-infra/project-config: Revert "Add limited tempest job to swift3 check pipeline"  https://review.openstack.org/25976109:17
openstackgerritEvgeniy L proposed openstack-infra/project-config: Create bareon-api repository  https://review.openstack.org/25935209:17
openstackgerritAndreas Jaeger proposed openstack-infra/system-config: Create linters test target  https://review.openstack.org/26040509:18
*** HeOS has joined #openstack-infra09:18
openstackgerritmathieu bultel proposed openstack-infra/shade: Add normalize stack function for heat stack_list  https://review.openstack.org/25433709:19
*** apuimedo_ has quit IRC09:19
*** apuimedo_ has joined #openstack-infra09:19
*** vilobhmm11 has quit IRC09:20
*** e0ne has joined #openstack-infra09:20
*** dizquierdo has joined #openstack-infra09:22
AJaegerjhesketh: the graph for ready nodes looks interesting - over 40 for each jenkins: http://grafana.openstack.org/dashboard/db/nodepool This is since the last change by fungi merged09:22
*** tzn has joined #openstack-infra09:22
AJaegerjhesketh: and post queue contains jobs that do not move forward at all since the last merged node change09:22
jheskethAJaeger: yeah I think nodepool is responding to the demand but the jobs aren't being assigned09:22
jheskethI think they aren't moving as they run on persistent nodes09:23
jheskethso there's something funky with the job assignment there09:23
AJaegerI don't see why fuel-specs-publish-specs should run on persistent node before the change09:23
*** aopincar has quit IRC09:24
jheskethI could be wrong, that was a theory I'm investigating09:24
*** e0ne has quit IRC09:24
*** apuimedo_ has quit IRC09:24
*** apuimedo has quit IRC09:24
AJaegerjhesketh: you might be right as well ;)09:25
AJaegerhttps://jenkins01.openstack.org/view/All/job/fuel-specs-publish-specs/lastBuild/consoleFull is a run - it run on bare-trusty09:25
*** apuimedo has joined #openstack-infra09:25
AJaegerand the node was taken offline afterwards if I interpret https://jenkins01.openstack.org/view/All/job/fuel-specs-publish-specs/lastBuild/parameters/ correct09:26
*** apuimedo_ has joined #openstack-infra09:26
*** apuimedo has quit IRC09:30
*** apuimedo_ has quit IRC09:30
*** apuimedo has joined #openstack-infra09:31
*** apuimedo_ has joined #openstack-infra09:32
*** derekh has joined #openstack-infra09:35
*** apuimedo has quit IRC09:36
*** apuimedo_ has quit IRC09:36
*** apuimedo has joined #openstack-infra09:37
*** apuimedo_ has joined #openstack-infra09:38
*** aopincar has joined #openstack-infra09:39
*** _nadya_ has quit IRC09:39
*** Hal1 has quit IRC09:39
*** yamamoto has joined #openstack-infra09:42
*** apuimedo_ has quit IRC09:42
*** Hal1 has joined #openstack-infra09:43
*** apuimedo_ has joined #openstack-infra09:43
*** apuimedo has quit IRC09:44
*** sshnaidm has quit IRC09:45
*** apuimedo has joined #openstack-infra09:46
*** triggerwarning has quit IRC09:47
*** dcentral has quit IRC09:47
tdurakovhi folks09:47
*** dcentral has joined #openstack-infra09:47
AJaegerhi tdurakov. Is everything fine?09:47
tdurakovgot question about devstack, is there any reason why when we set USE_SCREEN to false, we don't use services?09:48
*** apuimedo_ has quit IRC09:48
tdurakovAJaeger, ^09:48
AJaegertdurakov: I cannot help with that - for devstack you better ask on #openstack-qa09:48
*** yamamoto has quit IRC09:48
jheskeththere are 47 workers claiming to be able to do fuel-specs-publish-specs, but for some reason jenkins aren't picking it out of the gearman queue09:49
tdurakovAJaeger, ok, thank you09:49
*** tzn has quit IRC09:49
*** vgridnev has joined #openstack-infra09:49
AJaeger47 should be enough ;) This is strange but beyond my debug foo ;(09:50
*** apuimedo_ has joined #openstack-infra09:50
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use linters for system-config  https://review.openstack.org/26042009:51
openstackgerritRamana Raja proposed openstack-infra/project-config: Add Manila with CephFSNative driver CI job  https://review.openstack.org/25932309:52
*** BobBall_AWOL is now known as BobBall09:53
*** Daisy has joined #openstack-infra09:53
*** apuimedo_ has quit IRC09:55
*** apuimedo_ has joined #openstack-infra09:55
*** sshnaidm has joined #openstack-infra09:57
*** apuimedo has quit IRC09:57
*** yamamoto has joined #openstack-infra09:58
*** apuimedo has joined #openstack-infra09:58
*** fawadkhaliq has quit IRC10:00
*** zhenguo has quit IRC10:01
*** otter768 has joined #openstack-infra10:01
*** zhenguo has joined #openstack-infra10:01
*** maishsk has quit IRC10:01
*** apuimedo has quit IRC10:03
*** maishsk has joined #openstack-infra10:03
*** apuimedo has joined #openstack-infra10:03
*** otter768 has quit IRC10:06
*** apuimedo has quit IRC10:08
*** aopincar has quit IRC10:09
*** apuimedo has joined #openstack-infra10:09
*** apuimedo_ has quit IRC10:09
*** apuimedo_ has joined #openstack-infra10:10
*** bexelbie has joined #openstack-infra10:10
*** yamamoto has quit IRC10:10
*** apuimedo has quit IRC10:14
*** apuimedo_ has quit IRC10:14
AJaegerjhesketh: did the last puppet run complete on all jenkins?10:15
*** sshnaidm has quit IRC10:15
AJaegeror did it run into any timeouts and we need to update them first with jjb config?10:15
jheskethAJaeger: haven't checked that in particular, but the job definition has the update10:15
AJaegerjhesketh: http://puppetboard.openstack.org/ gives me an internal server error10:15
*** apuimedo has joined #openstack-infra10:16
jheskethyeah I saw that but figured I'd solve one problem at a time :p10:16
*** Qiming has quit IRC10:16
*** apuimedo_ has joined #openstack-infra10:16
* AJaeger was wondering why gate-requirements-docs is still NOT_REGISTERED (approved a few hours ago) - and a not-running puppet could explain that and might play a role in the post jobs. But since you checked that those are fine...10:17
AJaegerjhesketh: sure10:17
AJaegervponomaryov: glad to hear that your manila-image problem is solved!10:18
*** irenab has joined #openstack-infra10:18
vponomaryovAJaeger: CI side - yes ))10:19
AJaegervponomaryov: one step at a time ;)10:19
vponomaryovAJaeger: do you have link to change that fixed Ci part?10:19
AJaegerMain changes are https://review.openstack.org/#/c/260213/ and https://review.openstack.org/#/c/260214/10:20
AJaegervponomaryov: fungi reworked the node handling completely with the two changes above10:20
*** Daisy has quit IRC10:21
*** apuimedo has quit IRC10:23
vponomaryovAJaeger: thanks for info and fungi for work!10:23
*** yamamoto has joined #openstack-infra10:23
*** apuimedo_ has quit IRC10:23
*** yamamoto has quit IRC10:24
*** apuimedo has joined #openstack-infra10:24
openstackgerritJaume Devesa proposed openstack-infra/project-config: Add puppet-midonet to OpenStack infrastructure  https://review.openstack.org/26009310:24
*** apuimedo_ has joined #openstack-infra10:24
*** zeih has joined #openstack-infra10:24
AJaegerjhesketh: Don't work through the whole night - if you find anything, tell it here and let others take over later...10:25
*** yamahata has quit IRC10:25
jheskethAJaeger: night is young yet ;-)10:25
*** e0ne has joined #openstack-infra10:26
*** Hal1 has quit IRC10:27
*** sshnaidm has joined #openstack-infra10:27
*** zeih has quit IRC10:29
*** apuimedo has quit IRC10:29
*** apuimedo_ has quit IRC10:29
*** apuimedo has joined #openstack-infra10:30
*** apuimedo_ has joined #openstack-infra10:30
*** daemontool has joined #openstack-infra10:31
*** fawadkhaliq has joined #openstack-infra10:34
*** apuimedo has quit IRC10:35
*** fawadkhaliq has quit IRC10:35
*** apuimedo has joined #openstack-infra10:36
*** anshprat has quit IRC10:36
*** vmbrasseur has quit IRC10:36
*** vmb has joined #openstack-infra10:36
*** vmb is now known as vmbrasseur10:37
*** apuimedo_ has quit IRC10:38
openstackgerritmathieu bultel proposed openstack-infra/shade: Add normalize stack function for heat stack_list  https://review.openstack.org/25433710:38
*** apuimedo_ has joined #openstack-infra10:38
*** Hal1 has joined #openstack-infra10:39
*** zhurong has joined #openstack-infra10:39
*** apuimedo has quit IRC10:40
*** apuimedo has joined #openstack-infra10:41
*** apuimedo_ has quit IRC10:43
*** apuimedo_ has joined #openstack-infra10:45
*** apuimedo has quit IRC10:45
*** apuimedo has joined #openstack-infra10:46
*** maishsk has quit IRC10:46
ttxAJaeger: any infra issue I should be aware of ?10:46
* ttx tries to parse the IRC logs10:47
AJaegerttx: We changed the node setup in project-config - greatly simplified it.10:48
AJaegerSeems like some post-jobs are not run currently.10:48
*** _nadya_ has joined #openstack-infra10:48
AJaegerBut everything else looks fine.10:48
ttxAJaeger: I just W+1ed https://review.openstack.org/#/c/230699/ 10 min ago and it does not get picked up10:48
*** rvasilets___ has joined #openstack-infra10:49
*** maishsk has joined #openstack-infra10:49
AJaegerttx, that's strange and not known. We expected some fallout from the node handling change but not this kind ;(10:50
ttxLet me try something else. I tried to use the W+1 1-click gerrit button, maybe that fails10:50
*** apuimedo has quit IRC10:51
AJaegerAh, the one-click button does not +2!10:51
ttxhmm, though I don't really see why/how10:51
ttxthat repo does not require +210:51
ttxit requires CodeRevbiew+1 RollCall+110:51
AJaegerAnd we have both.10:51
*** apuimedo_ has quit IRC10:52
*** apuimedo has joined #openstack-infra10:52
ttxhmm, /me tries again10:52
*** apuimedo_ has joined #openstack-infra10:52
*** markvoelker has joined #openstack-infra10:52
AJaegerstill not there ;(10:53
ttxAJaeger: yeah. And can't see why it would block it10:53
*** hashar has quit IRC10:53
ttxit just parsed a series of events10:54
AJaegerttx, one more issue for infra-root to debug - let me just log it here: 230699 has been approved but Zuul is not putting it into the gate10:54
* ttx will hold on processing releases10:54
AJaegerttx, thanks for telling here. jhesketh is currently looking at the post queue and fungi et all will help later...10:55
ttxAJaeger: it's probably an unrelated issue10:55
jheskethAJaeger, ttx: http://paste.openstack.org/show/482496/10:56
AJaegerttx, agreed10:56
openstackgerritIvan Kolodyazhny proposed openstack-infra/project-config: Add python-brick-cinderclient-ext to OpenStack  https://review.openstack.org/24308010:57
AJaegerjhesketh: thanks for that line. Something is indeed wrong here.10:57
ttxjhesketh: hmm, so for some reason the original check is missing10:57
*** markvoelker has quit IRC10:57
* jhesketh is looking10:57
* ttx tries recheck10:57
*** fawadkhaliq has joined #openstack-infra10:57
ttxcould be it fell in the cracks of earlier issues10:57
ttxok, it's in the check queue now10:58
*** hashar has joined #openstack-infra10:58
ttxOK, I suspect the check failed to trigger earlier, and it does not automatically get retriggered on W+110:58
ttxshould be ok now10:59
*** zeih has joined #openstack-infra10:59
jheskethyeah but it's not clear why the original check never happened10:59
ttxunrelated issue, probably a consequence of an earlier issue10:59
jheskethmind you, that was a week ago10:59
AJaeger14th of December - not sure what hit us at that time ;(11:00
jheskethttx: so with the releases maybe just start with one and see how it goes.. I'm still trying to debug the post queue11:00
AJaegerok, issue solved ;)11:00
ttxjhesketh: yeah, I'll wait a bit anyway11:00
jheskethadmittedly I'm running out of ideas11:01
jheskethit appears unique to the post pipeline and only jobs on non-persistent nodes11:01
*** fawadkhaliq has quit IRC11:02
jheskethI'm tempted to try restarting a jenkins master, but I believe that'll lose/screw up the work that it has in progress11:02
*** fawadkhaliq has joined #openstack-infra11:02
*** yamamoto has joined #openstack-infra11:02
*** yamamoto has quit IRC11:02
*** apuimedo has quit IRC11:06
*** apuimedo_ has quit IRC11:06
*** apuimedo has joined #openstack-infra11:08
*** dtantsur|afk is now known as dtantsur11:08
*** apuimedo_ has joined #openstack-infra11:08
AJaegerjhesketh: you can schedule it for downtime somehow - so that no new jobs get run on that jenkins and then restart it once it's idle...11:09
jheskethoh, that'd be handy11:10
* jhesketh looks for that thing11:10
AJaegerjhesketh: might be this information ? ^11:11
jheskethAJaeger: very helpful, thankyou :-)11:12
*** smarcet has joined #openstack-infra11:12
*** apuimedo has quit IRC11:12
*** apuimedo_ has quit IRC11:13
*** apuimedo has joined #openstack-infra11:13
*** apuimedo_ has joined #openstack-infra11:14
*** shardy has quit IRC11:14
*** lucas-dinner is now known as lucasagomes11:15
*** irenab has quit IRC11:16
*** Qiming has joined #openstack-infra11:17
*** smarcet has quit IRC11:17
*** Jay_ has quit IRC11:18
*** binbincong has quit IRC11:19
*** apuimedo_ has quit IRC11:20
*** apuimedo_ has joined #openstack-infra11:22
*** egonzalez has quit IRC11:23
*** Jay_ has joined #openstack-infra11:24
*** triggerwarning has joined #openstack-infra11:25
*** Jay_ has quit IRC11:25
*** apuimedo has quit IRC11:25
*** triggerwarning has quit IRC11:25
*** triggerwarning has joined #openstack-infra11:25
*** Jay_ has joined #openstack-infra11:25
*** apuimedo has joined #openstack-infra11:26
openstackgerritDougal Matthews proposed openstack-infra/tripleo-ci: Remove the tripleo.sh Flavor creation call  https://review.openstack.org/26044811:26
AJaegerjhesketh: which jenkins are you taking down?11:26
jheskethAJaeger: jenkins0111:27
*** apuimedo_ has quit IRC11:27
jheskethAJaeger: however it looks like things are unstuck11:27
jheskethbut not due to restarting 0111:27
jheskethso I'm not sure what's going on just yet11:27
AJaegerindeed, there's progress!11:27
jhesketh01 is still being restarted though so I have to let that finish11:27
*** apuimedo_ has joined #openstack-infra11:27
AJaegerload is light enough that we can restart it...11:27
openstackgerritAndrey Pavlov proposed openstack-infra/project-config: remove one of ec2-api gating job  https://review.openstack.org/26044911:27
*** tzn has joined #openstack-infra11:30
*** Daisy has joined #openstack-infra11:30
*** fawadkhaliq has quit IRC11:31
*** fawadkhaliq has joined #openstack-infra11:31
*** ramineni has left #openstack-infra11:31
*** apuimedo_ has quit IRC11:32
*** apuimedo_ has joined #openstack-infra11:33
*** apuimedo has quit IRC11:35
*** egonzalez has joined #openstack-infra11:36
*** Daisy has quit IRC11:36
*** apuimedo has joined #openstack-infra11:37
*** Daisy has joined #openstack-infra11:37
*** apuimedo_ has quit IRC11:37
*** apuimedo_ has joined #openstack-infra11:38
*** zeih has quit IRC11:38
*** rvba` is now known as rvba11:39
*** Daisy has quit IRC11:40
*** Daisy has joined #openstack-infra11:41
jheskethAJaeger: so it got through some, and now it is waiting again11:41
jheskethAJaeger: I wonder if it's just due to the low precedence on the post pipeline11:42
jheskethhowever nodepool seems to be over allocating demand at the moment11:42
*** apuimedo_ has quit IRC11:42
jheskethwhich it's doing so we can pick up on all of the queued jobs11:43
*** apuimedo has quit IRC11:43
*** apuimedo has joined #openstack-infra11:44
*** salv-orl_ has quit IRC11:44
AJaegerthe post pipeline has indeed a low precedence - but stuck for several hours is rare, especially if enough nodes are available.11:44
*** apuimedo_ has joined #openstack-infra11:44
jheskethoh yeah, something is still fishy, but that might be why a few snuck through11:46
AJaegerjhesketh: we can wait for the US to wake up as well and see how this evolves...11:46
jheskethyeah I don't think it's bad enough to warrant reverting or anything too drastic11:48
jheskethit can likely wait for people much smarter than me to wake up11:48
AJaegermuch smarter -> more familiar ;)11:48
*** apuimedo has quit IRC11:48
*** jyuso1 has quit IRC11:48
openstackgerritVladimir Kozhukalov proposed openstack-infra/project-config: Create fuel-ui project  https://review.openstack.org/26045511:48
*** lxsli_festive has quit IRC11:48
*** apuimedo has joined #openstack-infra11:49
*** lxsli_festive has joined #openstack-infra11:52
openstackgerritVladimir Kozhukalov proposed openstack-infra/project-config: Create fuel-ui project  https://review.openstack.org/26045511:52
*** apuimedo has quit IRC11:54
*** apuimedo_ has quit IRC11:54
*** apuimedo has joined #openstack-infra11:54
*** isaacb has joined #openstack-infra11:55
*** apuimedo_ has joined #openstack-infra11:56
*** Daisy has quit IRC11:56
*** Daisy has joined #openstack-infra11:56
*** xwizard_ has quit IRC11:57
*** apuimedo has quit IRC11:59
*** apuimedo has joined #openstack-infra12:00
*** apuimedo_ has quit IRC12:00
*** Daisy has quit IRC12:00
*** Jay_ has quit IRC12:01
openstackgerritVladimir Kozhukalov proposed openstack-infra/project-config: Create fuel-ui project  https://review.openstack.org/26045512:01
*** apuimedo_ has joined #openstack-infra12:01
*** otter768 has joined #openstack-infra12:02
*** hashar has quit IRC12:03
*** Hal1 has quit IRC12:05
*** apuimedo_ has quit IRC12:06
*** egonzalez has quit IRC12:06
*** apuimedo_ has joined #openstack-infra12:07
*** apuimedo has quit IRC12:07
*** hashar has joined #openstack-infra12:07
*** otter768 has quit IRC12:07
*** apuimedo has joined #openstack-infra12:08
*** jaypipes has joined #openstack-infra12:09
AJaegerjhesketh: jenkins01 should be idle now...12:10
*** claudiub has joined #openstack-infra12:11
*** apuimedo_ has quit IRC12:11
*** apuimedo has quit IRC12:12
*** apuimedo has joined #openstack-infra12:13
*** _nadya_ has quit IRC12:13
*** apuimedo_ has joined #openstack-infra12:13
jheskethAJaeger: one to go12:13
*** ociuhandu has joined #openstack-infra12:13
*** apuimedo has quit IRC12:17
*** apuimedo_ has quit IRC12:18
*** yamamoto has joined #openstack-infra12:19
*** apuimedo has joined #openstack-infra12:19
*** daemontool has quit IRC12:19
*** apuimedo_ has joined #openstack-infra12:19
AJaegerI see...12:21
*** egonzalez has joined #openstack-infra12:21
jheskethAJaeger: okay restart done12:21
*** daemontool has joined #openstack-infra12:21
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: TEST: this is supposed to trigger a major upgrade  https://review.openstack.org/26046612:23
*** yamamoto has quit IRC12:23
*** Jay_ has joined #openstack-infra12:23
*** apuimedo has quit IRC12:23
*** hashar has quit IRC12:24
*** apuimedo has joined #openstack-infra12:24
openstackgerritRamana Raja proposed openstack-infra/project-config: Add Manila with CephFSNative driver CI job  https://review.openstack.org/25932312:24
*** yamamoto has joined #openstack-infra12:25
*** Jay_ has quit IRC12:25
*** Jay_ has joined #openstack-infra12:26
*** Jay_ has joined #openstack-infra12:27
*** egonzalez has quit IRC12:29
*** gordc has joined #openstack-infra12:30
*** apuimedo_ has quit IRC12:31
*** apuimedo_ has joined #openstack-infra12:32
AJaegerjhesketh: doesn't seem to have changed anything from what I can see12:34
*** dprince has joined #openstack-infra12:34
jheskethno unfortunately :-(12:34
*** apuimedo has quit IRC12:36
*** apuimedo_ has quit IRC12:37
*** apuimedo has joined #openstack-infra12:37
*** apuimedo_ has joined #openstack-infra12:38
*** markvoelker has joined #openstack-infra12:38
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack-infra/elastic-recheck: Correct typos  https://review.openstack.org/26047412:38
*** ZZelle has joined #openstack-infra12:40
*** yamamoto has quit IRC12:40
*** rhallisey has joined #openstack-infra12:41
*** rhallisey has quit IRC12:42
*** apuimedo_ has quit IRC12:42
*** pblaho has joined #openstack-infra12:43
*** markvoelker has quit IRC12:43
*** apuimedo has quit IRC12:44
*** apuimedo has joined #openstack-infra12:44
*** apuimedo_ has joined #openstack-infra12:44
*** salv-orlando has joined #openstack-infra12:45
*** Hal1 has joined #openstack-infra12:48
*** apuimedo has quit IRC12:48
*** apuimedo_ has quit IRC12:49
*** yamamoto has joined #openstack-infra12:50
*** apuimedo has joined #openstack-infra12:50
*** apuimedo_ has joined #openstack-infra12:50
*** sshnaidm has quit IRC12:53
*** amuller has joined #openstack-infra12:54
*** sdake has joined #openstack-infra12:54
*** apuimedo_ has quit IRC12:55
*** apuimedo_ has joined #openstack-infra12:55
*** baoli has joined #openstack-infra12:56
openstackgerritgordon chung proposed openstack-infra/project-config: add python-aodhclient to openstack repo  https://review.openstack.org/25963712:56
*** rguillebert has joined #openstack-infra12:56
*** apuimedo has quit IRC12:57
*** apuimedo has joined #openstack-infra12:57
*** mudassirlatif has joined #openstack-infra12:58
*** _nadya_ has joined #openstack-infra12:59
*** abregman_ has joined #openstack-infra13:01
*** dims has joined #openstack-infra13:01
*** apuimedo has quit IRC13:02
*** apuimedo has joined #openstack-infra13:03
*** Jay_ has quit IRC13:04
*** abregman has quit IRC13:05
*** sshnaidm has joined #openstack-infra13:06
*** apuimedo_ has quit IRC13:06
*** abregman_ has quit IRC13:07
*** apuimedo_ has joined #openstack-infra13:07
*** Jay_ has joined #openstack-infra13:07
*** apuimedo has quit IRC13:08
*** Jay__ has joined #openstack-infra13:08
*** apuimedo has joined #openstack-infra13:08
*** Jay_ has quit IRC13:12
*** apuimedo has quit IRC13:13
*** apuimedo has joined #openstack-infra13:14
*** dizquierdo is now known as dizquierdo_afk13:14
*** apuimedo_ has quit IRC13:14
*** apuimedo_ has joined #openstack-infra13:15
*** sshnaidm has quit IRC13:17
*** ihrachys has joined #openstack-infra13:17
*** salv-orlando has quit IRC13:18
*** salv-orlando has joined #openstack-infra13:18
*** sdake has quit IRC13:19
*** ZZelle has quit IRC13:19
*** apuimedo_ has quit IRC13:25
*** apuimedo_ has joined #openstack-infra13:26
*** weshay_xchat has joined #openstack-infra13:28
*** sshnaidm has joined #openstack-infra13:29
*** tzn has quit IRC13:30
*** shardy has joined #openstack-infra13:34
*** dcentral has quit IRC13:34
*** triggerwarning has quit IRC13:35
*** apuimedo_ has quit IRC13:36
*** apuimedo_ has joined #openstack-infra13:36
*** ZZelle has joined #openstack-infra13:39
*** piet has joined #openstack-infra13:44
*** mudassirlatif has quit IRC13:46
openstackgerritgordon chung proposed openstack-infra/project-config: rename ceilometer-specs to telemetry-specs  https://review.openstack.org/24889713:46
*** triggerwarning has joined #openstack-infra13:46
*** yamamoto has quit IRC13:47
*** dcentral has joined #openstack-infra13:47
*** EricGonczer_ has joined #openstack-infra13:47
*** abregman has joined #openstack-infra13:47
*** yamamoto has joined #openstack-infra13:48
*** EricGonczer_ has quit IRC13:49
*** admin0 has left #openstack-infra13:52
*** abregman has quit IRC13:53
anteayaAJaeger: I'm here for about 30 minutes before I have to depart for errands, what can I do to help?13:55
*** tlian has joined #openstack-infra13:55
jd__anteaya: some outage I guess?13:56
anteayajd__: we changed some configuration for allocated nodes yesterday, AJaeger and jhesketh seem to be tracking an issue13:56
jd__ok, I indeed get 502 error on Gerrit13:56
anteayajd__: I just woke up, anyone following backscroll knows more than me at this point13:56
anteayajd__: okay thank you13:57
* anteaya tries do gerrit13:57
jd__there's no much in the backlog actually :)13:57
anteayaokay thanks13:57
*** thorst_ has joined #openstack-infra13:58
AJaegeranteaya: Zuul/Jenkins is slow, not sure what'S going on.13:59
anteayaI think we have an overabundance of available test nodes13:59
anteayaAJaeger: very good, thanks13:59
AJaegeranteaya: Run your errands and let's wait for fungi and clarkb to figure it out.13:59
*** links has quit IRC13:59
openstackgerritDong Ma proposed openstack-infra/jenkins-job-builder: Add support for OWASP Dependency-Check Plugin  https://review.openstack.org/17960413:59
anteayagerrit seems fine for me so far13:59
*** piet has quit IRC13:59
anteayaAJaeger: very good, thanks13:59
AJaegeranteaya: Yes, far too many test nodes - and Jenkins not using them ;(13:59
anteayaAJaeger: ah, yeah that is an issue13:59
AJaegerjd__: gerrit works fine here as well13:59
* jd__ retries14:00
jd__it works again indeed, though it's quite slow14:01
*** EricGonczer_ has joined #openstack-infra14:01
jd__or not, just got a 50214:01
AJaegeranteaya: but reviewing my linters changes today would be great - see https://review.openstack.org/260192 - but that's part of the stack14:01
AJaegerjd__: yeah, it's slow now ;(14:01
AJaegerbut no 502 yet for me14:02
jd__➔ curl https://review.openstack.org/changes/260168/revisions/72f5156fa29c9d5d4894f99a2b2ff32bbc513b12/comments14:02
jd__<title>502 Proxy Error</title>14:02
*** apuimedo_ has quit IRC14:02
anteayaI confirm 502 here14:02
jd__<address>Apache/2.4.7 (Ubuntu) Server at review.openstack.org Port 443</address>14:02
*** thorst_ has quit IRC14:02
jd__(I truncated the html a bit but you got it)14:02
AJaegerright now zuul is taking again 3 changes from the post queue - after those have been sitting there for ages...14:02
*** otter768 has joined #openstack-infra14:02
anteayajd__: the gerrit 502 may be an artifact of the zuul/jenkins issue14:02
* jd__ nods14:03
* AJaeger got all his requests server but it was slow14:03
*** apuimedo_ has joined #openstack-infra14:03
anteayaI think I will have breakfast14:03
anteayathat isn't very helpful but at least I'm not adding to the problem14:03
*** markvoelker has joined #openstack-infra14:04
AJaegerFound the bottom of my stack for review - once you have time after your errands, anteaya : https://review.openstack.org/#/c/26020214:04
AJaegeranteaya: enjoy!14:04
jd__AJaeger: you're privileged ;)14:05
*** hamzy is now known as hamzy_vacay14:05
*** weshay_xchat has quit IRC14:06
*** peristeri has joined #openstack-infra14:06
*** thorst_ has joined #openstack-infra14:07
*** weshay_xchat has joined #openstack-infra14:07
*** otter768 has quit IRC14:07
*** sdake has joined #openstack-infra14:08
*** sdake has quit IRC14:08
*** abregman has joined #openstack-infra14:08
*** markvoelker has quit IRC14:08
*** sdake has joined #openstack-infra14:09
*** abregman has quit IRC14:09
*** apuimedo_ has quit IRC14:10
*** apuimedo_ has joined #openstack-infra14:11
*** markvoelker has joined #openstack-infra14:12
*** markvoelker has quit IRC14:13
*** markvoelker has joined #openstack-infra14:14
*** thorst_ has quit IRC14:14
*** dkranz has joined #openstack-infra14:15
dhellmannis something going on with gerrit? I'm getting errors loading https://review.openstack.org/#/c/194868/14:17
markus_zdhellmann: I'm experiencing issues too. I don't know the cause.14:17
Shrewsdhellmann: i believe so, looking at scrollback14:17
*** apuimedo_ has quit IRC14:18
dhellmannShrews : thanks14:19
dhellmannmarkus_z : thanks for confirming14:19
*** apuimedo_ has joined #openstack-infra14:19
dulekI can confirm proxy errors too. And yes - I'm behind a corporate proxy.14:20
AJaegerdhellmann: somehow gerrit is slow...14:20
AJaegerinfra-root, is anybody already awake to check what's going on with gerrit?14:21
dhellmannAJaeger : ok. I'm seeing actual page load failures, like jd__ was14:21
* dhellmann wonders if this is a sign we should all take the day off ;-)14:21
*** rossella_s has quit IRC14:21
AJaegerdhellmann: it is a sign ;)14:21
*** rossella_s has joined #openstack-infra14:21
AJaegerdhellmann: a sign to take the rest of the year off ;)14:22
* dhellmann thinks that escalated quickly14:22
*** yamamoto has quit IRC14:22
*** rbrndt has joined #openstack-infra14:22
*** rlandy has joined #openstack-infra14:24
notmorganhm.. getting 503s from review.o.o14:24
notmorganand 502s14:24
*** apuimedo_ has quit IRC14:24
AJaegerjd__: Now the 502s reached me as well ;(14:24
bdemersyup, 502s here too14:24
*** apuimedo_ has joined #openstack-infra14:24
*** admin0 has joined #openstack-infra14:24
notmorgandhellmann: take the day off :P14:24
monesteralso got 502 errors on gerrit14:24
notmorganit is intermittant14:25
notmorgani have had a a few pages loading ok for me14:25
mescanef"The proxy server received an invalid response from an upstream server."14:25
jd__AJaeger: looks like you got unprivileged! :)14:25
notmorganyeah, this is likely the java process having some issues14:26
AJaegerUnfortunately the US admins are not yet here and the Australian ones have left, so let's all take a break...14:26
jd__who got a nice story to tell then? :p14:26
* AJaeger configures his out-of-office replies...14:26
notmorganAJaeger: you're much nicer than I am14:27
*** bgmccollum has joined #openstack-infra14:27
notmorgani don't bother with out-of-office replies14:27
*** vdrok has joined #openstack-infra14:28
* fungi is here, though likely not terribly functional yet since was up late14:28
AJaegermorning fungi14:28
AJaegerLet me give a quick summary:14:29
fungiseeing what i can piece together from scrollback14:29
notmorganfungi: drink moar caffiene14:29
mordredfungi: high load average14:29
notmorganfungi: we can also wait for you to be more functional14:29
mordredgerrit not responding ish14:29
mordredfungi: I was considering bouncing it14:29
AJaegerZuul was kind of slow the whole day, it only looked every two hours at the post queue and we already thought it was stuck14:29
*** EricGonczer_ has quit IRC14:29
AJaegerAnd now gerrit gives 502s ;(14:29
notmorganmordred: is the server status from apache showing ick or just the java ?14:30
AJaegerfungi: No actual reports of problems in jobs due to the node change14:30
mordrednotmorgan: well, there's at least one apache pegged at 95% cpu14:30
notmorganwondering if this is mod_proxy being cranky because something happened with gerrit.14:31
*** piet has joined #openstack-infra14:31
fungioh the post pipeline backup is proably gearman incorrectly offlining one of the multi-use workers. checking14:31
*** markus_z has quit IRC14:31
elmikosounds like others are seeing the 502s from review.os.o?14:31
*** apuimedo_ has quit IRC14:31
AJaegerelmiko: welcome to the 502 party ;)14:32
notmorganelmiko: yep, it's being looked at14:32
elmikoack, thanks =)14:32
*** apuimedo_ has joined #openstack-infra14:32
notmorganmordred: can you take a look at server-status?14:32
notmorganmordred: [since it's 403'd from the outside world]14:32
fungihrm, no offline workers after all14:32
AJaegerfungi: There'S also a very large amound of available nodes14:32
AJaegerright now around 50 per jenkins, see http://grafana.openstack.org/dashboard/db/nodepool14:33
* notmorgan is leaning towards an issue with gerrit itself, but wans to be sure it isn't apache misbehaving14:33
AJaegerAnd http://puppetboard.openstack.org/ is dead14:33
mordrednotmorgan: do I just curl localhost/server-status?14:33
fungioh i see, their jobs ran but haven't reported14:33
AJaeger"Internal Server Error"14:33
notmorganmordred: i think so.14:33
fungii'll check zuul's debug log to see why it's not reporting14:33
*** mattt has joined #openstack-infra14:33
notmorganbecause i'm getting massively slow response for even the 403 trying to hit that externally14:34
dmsimardo/ reporting errors/timeouts on review.o.o as well14:34
notmorganthe whole server seems verrrry cranky atm14:34
*** mattymo has joined #openstack-infra14:34
fungimordred: you need to reach server-status over an ssh tunnel with a browser14:35
mattymoI'm sure everyone is aware of gerrit and 502 errors14:35
anteayamattymo: you are correct14:35
AJaegermattymo: yes14:35
notmorganfungi: curl should be sufficient shouldn't it? i mean it's just text last i saw.14:35
notmorganfungi: or did they wind up other magic?14:35
*** mriedem has joined #openstack-infra14:35
bauzasgerrit is down and I see a convo here, could we please have a topic for all the IRC chans?14:35
funginotmorgan: oh, yeah a text-based browser or curl or something would, probably. it's just bigger than my terminal i think14:36
*** ihrachys has quit IRC14:36
notmorganfungi: ah yeah14:36
*** dizquierdo_afk is now known as dizquierdo14:37
notmorganesp. with a bunch of workers14:37
fungino clue what's going on with puppetboard14:38
notmorganyeah i'm seeing 4-5 seconds to get a 403 trying to hit server-status externally. so the workers are all bound up at the very least. i don't have insight into the java process health so can't say which side it is, likely if a bounce of gerrit is needed, so will apache need to be bounced.14:38
*** sshnaidm has quit IRC14:38
notmorganjust to clear things up more quickly.14:39
anteayamy company has arrived and I have to go do errands now, back in a few hours14:39
*** sshnaidm has joined #openstack-infra14:40
*** apuimedo_ has quit IRC14:41
mordredfungi: any problems with me bouncing our friend mr. gerrit?14:41
*** mohankumar has joined #openstack-infra14:42
*** apuimedo_ has joined #openstack-infra14:42
*** notnownikki has joined #openstack-infra14:42
*** dslev has joined #openstack-infra14:42
fungimordred: go for it. i don't see anything interesting in zuul's logs yet to explain the jobs in the post pipeline waiting to report14:42
fungithough i wouldn't expect them to be blocking on gerrit anyway14:43
*** breton has joined #openstack-infra14:43
dmsimardreview.o.o is back and snappy for me now.14:44
mordredk. restarted14:44
mordredI also restarted apache, fwiw14:44
AJaegerfungi, interesting: zuul seems to have nearly finished the post queue14:44
AJaegerthanks, mordred14:45
notmorganmordred: looks much happier14:45
notmorganjust from a responsive standpoint14:46
notmorganyeah 403 is now taking partial seconds so the apache processes are happier14:46
*** apuimedo_ has quit IRC14:46
AJaegerpost queue is empty, fungi - this is strange...14:46
fungidoesn't look like we had a runaway ci commenting this time14:47
fungiAJaeger: well, they looked like their jobs were all finished but zuul hadn't gotten around to processing the results14:48
*** apuimedo_ has joined #openstack-infra14:48
*** eharney has quit IRC14:49
*** xwizard_ has joined #openstack-infra14:50
notmorganfungi: i'm going to look into tuning down our proxy timeouts a little so apache is quicker to fail/less likely to bind up if gerrit gets unhappy14:51
fungimordred: so before the restart, the server status seemed to indicate that all 6 apache worker threads had 25 connections each14:51
AJaegerfungi: that was only now, for several hours nothing happend. We had this morning them in the queue for 4 hours with only the translation jobs running, then jhesketh checked and the first 3 or 5 changes merged.14:51
funginotmorgan: ^14:51
notmorganfungi: yeah14:51
AJaegerfungi: and then it was stuck again - until you poked and suddenly all of them finished...14:51
*** zhurong has quit IRC14:51
notmorganfungi: but to be fair almost all of them were blocking in "sending reply" could be blocked on getting data from gerrit14:51
notmorganfungi: based on server-status14:52
fungiAJaeger: i have a feeling zuul was stuck getting through to gerrit, even though it didn't need to report post pipeline results to gerrit, not sure since i didn't get a thread dump before things cleared up14:52
AJaegergerrit worked this morning - but the recent slowdown might have started earlier and build up somehow...14:53
pabelanger_nibalizer: asselin: Patches for puppet-nodepool are up, so I think it is just a matter of rotating through puppet modules now.14:53
*** pabelanger_ is now known as pabelanger14:53
AJaegerlet's wait and see whether everything continues fine on that front...14:53
*** links has joined #openstack-infra14:54
asselin__pabelanger, how does that help for https://review.openstack.org/#/c/253619/ ?14:55
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for manila-neutron job timeout bug 1528597  https://review.openstack.org/26057514:55
openstackbug 1528597 in Manila "manila (neutron) gate jobs are timing out" [Undecided,New] https://launchpad.net/bugs/152859714:55
pabelangerasselin__: we'll be using puppetlabs-apache directly, IIRC they don't have the issue14:58
pabelangerasselin__: so, I believe your check is simpler14:58
pabelangerbut you'd need to test14:58
asselin__pabelanger, so it automatically includes the necessary mods?14:58
*** lucasagomes is now known as lucas-hungry14:59
pabelangerYou still still need to include them14:59
pabelangerbut I believe the original reload issue is gone14:59
*** maishsk has quit IRC14:59
pabelangerhonestly, I'd have to test the patch under puppetlabs-apache and see what would be needed14:59
*** apuimedo_ has quit IRC15:00
SpamapSianw: I read your post, and it looks great. Thanks for making it!15:00
*** apuimedo_ has joined #openstack-infra15:00
mordredSpamapS: link?15:00
SpamapSmordred: https://www.technovelty.org/openstack/platform-bootstrap-for-openstack-infrastructure.html15:01
SpamapSmordred: you're more likely to find anything infra-wrong with it than I am. ;)15:01
*** hashar has joined #openstack-infra15:03
mordredianw: awesome. typo: "Now you'll need to setup a cloud.yaml file to give yourself the permissions to upload the image. " s/cloud.yaml/clouds.yaml/15:03
skolekonovjoin #openstack-meeting-415:04
mordredianw: great post!15:04
*** dtantsur is now known as dtantsur|brb15:04
*** maishsk has joined #openstack-infra15:05
*** cdent has joined #openstack-infra15:05
cdentDoes anybody know if this change that gordc has made is the right way to deal with independent release projects used in other jobs: https://review.openstack.org/#/c/258642/15:06
*** weshay_xchat has quit IRC15:08
*** eharney has joined #openstack-infra15:08
*** xyang1 has joined #openstack-infra15:09
*** apuimedo_ has quit IRC15:10
*** sdake has quit IRC15:10
*** apuimedo_ has joined #openstack-infra15:10
mordredinfra-root: heads up - ansible 2.0 is estimated/slated to be released on Jan 6. it SHOULD continue to work for us with no differences, but we should be aware in case something goes weird15:11
krotscheckWhat could possibly go wrong15:11
mordredkrotscheck: :)15:12
openstackgerritRamy Asselin proposed openstack-infra/project-config: Generate puppet-openstack ci docs  https://review.openstack.org/25924615:14
*** thorst_ has joined #openstack-infra15:15
asselin__AJaeger, just a commit message update ^^ so that it can merge15:15
*** maishsk has quit IRC15:16
notmorganmordred: at least it's happening AFTER new years.15:17
notmorganmordred: they could have been really mean and released it on the 28th or 30th15:17
notmorganand then jumped out for the year ...15:17
*** esker has quit IRC15:18
*** mattt has left #openstack-infra15:18
*** Qiming has quit IRC15:19
openstackgerritMerged openstack-infra/elastic-recheck: Add query for manila-neutron job timeout bug 1528597  https://review.openstack.org/26057515:19
*** zigo_ is now known as zigo15:19
openstackbug 1528597 in Manila "manila (neutron) gate jobs are timing out" [Undecided,Confirmed] https://launchpad.net/bugs/152859715:19
*** apuimedo_ has quit IRC15:19
* AJaeger does not understand why the fuel team proposes one repo after the other - but does not want to have them officially as part of the fuel team, see https://review.openstack.org/#/c/260208/ or https://review.openstack.org/#/c/259403/15:20
*** apuimedo_ has joined #openstack-infra15:20
mwhahahabecause it's an official fuel team thing as defined by those instructions15:20
mwhahahathe problem is the plugins aren't really part of fuel15:20
mwhahahaThe question "is this part of the fuel team" is different then "is this part of the fuel project under the big tent"15:21
AJaegermwhahaha: I don't understand why you need to make this difference...15:21
mwhahahaa plugin can be developed by anyone and may not be tied to fuel15:22
mwhahahathe core project15:22
mwhahahabut we want to put it out there for community development rather than assign it to a specific person15:22
mwhahahait does not necessarily track with the fuel project itself15:23
AJaegerso, fuel-plugin-designate: Who will work on that?15:23
*** thorst_ has quit IRC15:23
mwhahahaat the moment it probably has two devs who are responsible for it15:23
mwhahahai have no idea. i can only speak to the etckeeper one which will be myself initially and will probably be handed off over time to different folks15:24
fungiit was a little easier to see in the days of the stackforge git namespace, but official and unofficial projects generally don't share groups in their gerrit acls because they're necessarily developed/maintained by different teams of people15:24
*** esker has joined #openstack-infra15:25
AJaegerfungi: they all have different - separate - acls15:25
pabelangerAJaeger: thanks again for the help landing the tox patches yesterday15:26
AJaegerpabelanger: you're welcome15:26
*** zz_dimtruck is now known as dimtruck15:26
openstackgerritMerged openstack-infra/elastic-recheck: Correct typos  https://review.openstack.org/26047415:27
AJaegermwhahaha, fungi: I'm a bit surprised that we now have 82 fuel repos - and every week we get at least another one - and for each of them I hear: This is not part of the big fuel tent. And most of these repos come from the same person that requests them15:27
*** rbrndt has quit IRC15:27
*** mohankumar has quit IRC15:28
mwhahahayea probably the guy who knows what is required15:28
AJaegermwhahaha: I would have expected something else when every repo starts with fuel...15:28
fungirepos with "fuel" in their names account for 10% of the repos in our gerrit15:29
AJaegermwhahaha: I'm not going to block your change - I'm just trying to understand how you work together on this.15:29
mwhahahaSure and i appreciate that, I'm working through this as best I can because I don't know what is expected15:29
asselin__AJaeger, can you +A it?15:30
AJaegerasselin__: I see yolanda gave already a +2A before...15:30
*** bcourt has joined #openstack-infra15:30
AJaegerin that case, I do ;)15:31
asselin__:) thanks...helps with progress on the other patch...15:31
*** otter768 has joined #openstack-infra15:32
AJaegerfungi, coming back to post queue: Top of queue is now 40 min in the queue, we have lots of available workers (didn't check which type) and post queue only does translation update. Is that ok with our lowered precedence?15:33
fungiAJaeger: mwhahaha: it sounds like these are independently maintained repos then, which act as extensions or plugins to the fuel suite while not being maintained or overseen by the fuel team? we have some parallels in, for example, the neutron plugin ecosystem15:33
openstackgerritKaitlin Farr proposed openstack/requirements: Cap PyKMIP version  https://review.openstack.org/26059915:33
AJaegerfungi: yeah, that's how it looks like to me. Still, strikes me as odd - especially since most of the plugin repos get created by mescanef, but seems he does repo-creation-as-a-service ;)15:34
*** weshay_xchat has joined #openstack-infra15:34
AJaegerhave to run some errands, bye for now!15:35
jeblairfungi, mordred: did you get any info from gerrit?  like from javamelody or show-queues or show-caches?15:35
mwhahahayes plugins themselves follow the neutron plugin model more than they are tied to core fuel15:35
*** HeOS has quit IRC15:36
*** mudassirlatif has joined #openstack-infra15:36
jeblairfungi, mordred: cacti basically just shows a cpu pegged, which could be apache, but we still don't know if the underlying cause was apache or (more likely) gerrit, yeah?15:36
AJaegermwhahaha: the neutron plugins are nearly all part of neutron ...15:36
*** AJaeger has quit IRC15:36
fungijeblair: i did not in the moments before it was restarted. i was looking at zuul logs at the time15:36
*** HeOS has joined #openstack-infra15:36
fungialso i think nodepool is mistaken about how many ready nodes there actually are15:37
* mwhahaha shrugs15:37
mwhahahaI'll bring it up to the ptl and get his take on if they should get moved under the fuel umbrella15:38
mwhahahaI don't care one way or the other to be honest15:38
mwhahahai just need a repo in gerrit to put code15:38
*** hurgleburgler has joined #openstack-infra15:38
fungioh, weird. for some reason nodepool decided to create a ton of bare-precise and devstack-precise nodes15:39
*** sabeen1 has joined #openstack-infra15:40
fungiis this perhaps an upshot of zuul no longer knowing what node labels are associated with the jobs it needs to run? did nodepool's demand calculation grow dependent on our node label double-accounting for jobs?15:41
jeblair[2015-12-22 13:23:52,774] WARN  org.eclipse.jetty.io.SelectorManager : Could not process key for channel java.nio.channels.SocketChannel[connected local=/ remote=/]15:42
jeblairwe started getting *a lot* of those errors in the gerrit log starting then15:42
jeblair[2015-12-22 13:23:52,774] WARN  org.eclipse.jetty.util.thread.QueuedThreadPool : HTTP{STARTED,5<=25<=25,i=17,q=50} rejected org.eclipse.jetty.io.AbstractConnection$2@476f11b15:43
openstackgerritDiana Whitten proposed openstack-infra/project-config: Add openstack-ux repo  https://review.openstack.org/25923915:43
*** weshay_xchat has quit IRC15:43
fungithe bulk of the precise nodes seem to have been built around when the parameter functions were updated last night, though there have been more built since even though almost none of them are needed for running jobs15:44
*** kgiusti has joined #openstack-infra15:44
jeblairfungi, mordred: so i suspect something was stuck in gerrit; those error messages are related to the http connection queue15:45
*** sabeen3 has joined #openstack-infra15:46
*** apuimedo_ has quit IRC15:46
fungijeblair: that sounds likely. and if zuul was waiting for some timeout while reporting i suppose that could account for why it was taking a while to process results even for pipelines that don't report back into gerrit (e.g. post)?15:47
fungias for the nodepool precise demand, it definitely seems to think there is som15:47
fungi2015-12-22 15:37:30,210 DEBUG nodepool.NodePool:   Demand from gearman: bare-precise: 7215:47
fungi2015-12-22 15:37:30,211 DEBUG nodepool.NodePool:   Demand from gearman: devstack-precise: 16315:47
*** apuimedo_ has joined #openstack-infra15:47
openstackgerritMerged openstack-infra/project-config: Generate puppet-openstack ci docs  https://review.openstack.org/25924615:47
pabelangerWow, looking at zuul-status, I've never seen that many nodes available! :D15:48
jeblairfungi, mordred: confirmation the error is related to the incoming request queue: https://groups.google.com/forum/m/#!msg/repo-discuss/-2hnEOqQA7M/BEq1UZhZCgAJ15:49
*** sabeen1 has quit IRC15:49
*** yamamoto has joined #openstack-infra15:49
jeblairfungi, mordred: basically "gerrit was slow doing _something_ and the queue filled"; possible causes include database issues... so a mysql show processlist might be a good idea15:49
fungipabelanger: yeah, nodepool's building node types that zuul doesn't actually need. i think this is an upshot of my change to drop the job->node map from the zuul parameter functions15:49
*** odlD2 has quit IRC15:50
pabelangerfungi: darn! I thought christmas came early in the form of more clouds! hehe15:51
jeblairfungi: what was your change number?15:51
fungijeblair: there were several, it was part of a series to get rid of the old precise defaults for icehouse-era branches. digging it up now15:52
openstackgerritNicko Glayre proposed openstack-infra/jenkins-job-builder: Adding setup of credentialsId in stashNotifier  https://review.openstack.org/26060615:53
clarkb260213 and 26021415:53
fungijeblair: 260213, then 260214, then 26033415:53
*** apuimedo_ has quit IRC15:54
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Add astara to projects  https://review.openstack.org/25731415:54
*** apuimedo_ has joined #openstack-infra15:54
*** baoli has quit IRC15:55
fungioh! i wonder whether zuul needs a restart so that we clear out all the old precise node registrations15:55
*** baoli has joined #openstack-infra15:56
fungimaybe nodepool is assuming it may need precise nodes when zuul queues a label-unqualified work request which is registered by both types?15:56
*** bexelbie_ has joined #openstack-infra15:57
*** davideagnello has quit IRC15:58
*** bexelbie has quit IRC15:59
jeblairfungi: for unqualified jobs, nodepool picks the first node type it sees for that job when assigning demand to nodes15:59
*** davideagnello has joined #openstack-infra15:59
*** apuimedo_ has quit IRC15:59
fungijeblair: makes sense, so i can delete this excess temporarily and then we can do a controlled restart of zuul to clean up the stale registrations16:00
jeblairfungi: that likely means that now that the bulk of our demand is for unqualified jobs, it's picking the wrong node type16:00
fungibecause once the gearman registrations are repopulated from scratch, we should only see jobs registered for one label each16:00
*** apuimedo_ has joined #openstack-infra16:00
jeblairthis is probably something that could be corrected by paying attention to function registration count, but considering this happens once every few years, probably easier to just defer to zuulv316:01
jeblairfungi: you can delete the exccess, but it will immediately create more16:01
fungisounds good. mystery solved16:01
fungioh. hrm. okay i'll plan to do a zuul restart shortly in that case16:02
jeblairi think that's probably best16:02
openstackgerritThierry Carrez proposed openstack-infra/infra-manual: Update release management chapter  https://review.openstack.org/26061016:03
*** HeOS has quit IRC16:04
*** odlD2 has joined #openstack-infra16:04
*** lucas-hungry is now known as lucasagomes16:04
*** lezbar has joined #openstack-infra16:05
*** apuimedo_ has quit IRC16:05
*** apuimedo_ has joined #openstack-infra16:07
ttxlong overdue ^16:08
*** ifarkas has quit IRC16:09
openstackgerritMerged openstack-infra/reviewstats: Update barbican and related projects  https://review.openstack.org/25732116:09
openstackgerritMerged openstack-infra/reviewstats: Update infrastructure projects  https://review.openstack.org/25733716:09
*** HeOS has joined #openstack-infra16:09
openstackgerritMerged openstack-infra/reviewstats: Add cloudkitty to projects  https://review.openstack.org/25732716:10
*** rbrndt has joined #openstack-infra16:11
openstackgerritMerged openstack-infra/reviewstats: Update cinder and related projects  https://review.openstack.org/25731916:11
openstackgerritMerged openstack-infra/reviewstats: Add chef related projects  https://review.openstack.org/25731616:11
openstackgerritMerged openstack-infra/reviewstats: Update ironic and related projects  https://review.openstack.org/25733616:11
*** FallenPegasus has joined #openstack-infra16:14
*** armax has joined #openstack-infra16:14
openstackgerritMerged openstack-infra/reviewstats: Update nova and related projects  https://review.openstack.org/25733316:14
*** dtantsur|brb is now known as dtantsur16:15
*** vgridnev has quit IRC16:15
openstackgerritMerged openstack-infra/reviewstats: Add app-catalog to projects  https://review.openstack.org/25733016:15
openstackgerritMerged openstack-infra/reviewstats: Add additional TripleO repositories  https://review.openstack.org/22261816:15
openstackgerritMerged openstack-infra/reviewstats: Remove projects not in OpenStack repo  https://review.openstack.org/25349016:16
openstackgerritMerged openstack-infra/reviewstats: Update climate to blazar  https://review.openstack.org/25712816:16
openstackgerritMerged openstack-infra/reviewstats: Add cue to projects  https://review.openstack.org/25507916:16
*** apuimedo_ has quit IRC16:16
*** nelsnelson has joined #openstack-infra16:16
*** daemontool has quit IRC16:17
*** apuimedo_ has joined #openstack-infra16:17
*** ZZelle_ has joined #openstack-infra16:17
*** esker has quit IRC16:18
*** ZZelle_ has quit IRC16:19
*** yamamoto has quit IRC16:19
*** esker has joined #openstack-infra16:19
*** pblaho has quit IRC16:20
*** thorst_ has joined #openstack-infra16:21
*** harlowja_at_home has joined #openstack-infra16:23
*** baoli has quit IRC16:24
*** shardy has quit IRC16:24
*** baoli has joined #openstack-infra16:26
*** coolsvap is now known as coolsvap|away16:26
*** _nadya_ has quit IRC16:26
fungiokay, dumping check/gate pipelines and restarting zuul now16:27
fungiunless there are any objections16:27
clarkbgo for it16:28
fungihere goes16:28
fungiit's on its way back up now16:28
*** thorst_ has quit IRC16:28
*** apuimedo_ has quit IRC16:28
*** apuimedo_ has joined #openstack-infra16:29
clarkbjeblair: greghaynes: I need to catch up on where reviews are, but if the image builders changes for nodepool are ready to go I would like to approve them today, restart nodepool, trigger a manual image build, then watch image builds tomorrow morning16:29
jeblairclarkb: sounds good to me16:30
fungilooks like we have all the requisite node types registered already, so i'll reenqueue check and gate now16:31
krotscheckjeblair: Could I trouble you to refresh your +2 on https://review.openstack.org/#/c/253236/ ?16:32
*** fawadkhaliq has quit IRC16:32
*** mpavone has quit IRC16:33
krotscheckOooh, neat, a "diff against patchset" dropdown.16:34
* krotscheck keeps discovering things in new gerrit that he likes.16:34
*** tonytan4ever has joined #openstack-infra16:34
*** gyee has joined #openstack-infra16:34
*** dprince has quit IRC16:35
*** apuimedo_ has quit IRC16:36
jeblairkrotscheck: that's always been there :)16:37
*** smarcet has joined #openstack-infra16:37
smarcetfungi: morning16:37
smarcetwhen u have some time, could u approve https://review.openstack.org/#/c/259598/?16:37
smarcetthx u16:37
fungiokay, all the exported gate and check pipeline changes have been reenqueued. working on cleaning up the unnecessary precise node surplus in nodepool now16:38
jeblairkrotscheck: ps7 seems to be a rebase; was there any other change from ps6?16:38
krotscheckjeblair: Nope.16:38
*** apuimedo_ has joined #openstack-infra16:38
krotscheckjeblair: It just lost your +2 for some reason, maybe because of the merge conflict?16:38
*** maishsk has joined #openstack-infra16:38
jeblairkrotscheck: ah, yeah, merge conflicts don't count as simple rebases so votes don't carry over.  i trust that you resolved it correctly :)16:39
krotscheckjeblair: Urr.... well, I dunno. Past-me and Current-me don't really trust each other.16:39
krotscheck(Let's not talk about future-me. He's crazy)16:39
jeblairkrotscheck: if your different selves could have separate gerrit accounts, that will make things easier.  :)16:40
*** tsg has joined #openstack-infra16:40
krotscheckjeblair: I keep meaning to do that, but past-me keeps changing the password so I can't lock him out :)16:41
*** piet has quit IRC16:41
*** dims has quit IRC16:42
*** apuimedo_ has quit IRC16:42
asselin__clarkb, it would be nice to have this too for new image builds https://review.openstack.org/#/c/217914/16:43
*** resker has joined #openstack-infra16:43
*** piet has joined #openstack-infra16:44
*** resker has quit IRC16:44
*** apuimedo_ has joined #openstack-infra16:44
*** fawadkhaliq has joined #openstack-infra16:44
*** yamamoto has joined #openstack-infra16:44
*** resker has joined #openstack-infra16:44
*** esker has quit IRC16:46
*** apuimedo_ has quit IRC16:49
*** AJaeger has joined #openstack-infra16:49
*** apuimedo_ has joined #openstack-infra16:50
*** admin0 has quit IRC16:50
*** maishsk has quit IRC16:52
flip214The XenProject CI says FAILED, but doesn't provide logs for https://review.openstack.org/#/c/256292/3 ... is anyone interested in that data point?16:53
AJaegerflip214: there's a contact email for XenProject CI, see the comment. They usually react pretty fast if you tell them16:54
*** apuimedo_ has quit IRC16:55
AJaegerfungi, jeblair: Glad to thear that the node mystery is solved!16:55
*** maishsk has joined #openstack-infra16:55
flip214AJaeger: right now it started to work again... strange.16:55
*** kashyap has left #openstack-infra16:55
*** ayoung has quit IRC16:55
*** apuimedo_ has joined #openstack-infra16:56
*** yamamoto has quit IRC16:56
flip214AJaeger: but the failure is due to the setup...
flip2142015-12-22 13:48:36.761 | + sudo ip -6 addr add 2001:db8::2/64 dev br-ex16:56
*** yamamoto has joined #openstack-infra16:56
flip2142015-12-22 13:48:36.776 | RTNETLINK answers: File exists16:56
flip2142015-12-22 13:48:36.778 | + exit_trap16:56
AJaegerflip214: this is the wrong channel to discuss them, I can't help with that...16:57
flip214asselin__: ^^ interested in that?16:57
isaacbflip214: see https://review.openstack.org/#/c/239363/16:57
*** Hal1 has quit IRC16:58
*** yamamoto has quit IRC16:58
flip214isaacb: thanks for the info.16:58
*** Hal1 has joined #openstack-infra16:59
*** abregman has joined #openstack-infra17:00
*** bgmccollum has left #openstack-infra17:00
*** kzaitsev_mb has joined #openstack-infra17:00
*** apuimedo_ has quit IRC17:01
*** yaume has quit IRC17:01
BobBallflip214: That was a known bug... but if you recheck things seem to be working now17:01
AJaegerfungi, clarkb, jeblair: A review of the linters changes would be great - bottom of stack is https://review.openstack.org/#/c/260202/ .17:01
openstackgerritMerged openstack-infra/puppet-openstackid: Fix on bower test file exist condition:  https://review.openstack.org/25959817:01
BobBallflip214: I see on the change you've not used the correct syntax to recheck the XenProject CI; it tells you how to recheck in the message from the CI (click Toggle CI if you don't see it) - type "Xen: recheck" to ask for the CI to run the test again17:02
*** apuimedo_ has joined #openstack-infra17:02
*** amuller is now known as amuller_afk17:03
fungiis anybody interested in looking into why puppetboard is broken? if not, i'll dig into that next17:03
*** zeih has joined #openstack-infra17:03
flip214BobBall: thank you very much, I already wasted a few tries...17:04
flip214Yeah right, "Toggle CI" does help ;\17:04
*** hashar has quit IRC17:05
*** maishsk has quit IRC17:05
*** apuimedo_ has quit IRC17:09
*** apuimedo_ has joined #openstack-infra17:10
*** sshnaidm has quit IRC17:11
*** dims has joined #openstack-infra17:12
*** yamahata has joined #openstack-infra17:12
*** daemontool has joined #openstack-infra17:13
*** boris-42 has quit IRC17:13
openstackgerritYuriy Zveryanskyy proposed openstack/diskimage-builder: [EXPERIMENTAL] Ansible deploy ramdisk for Ironic  https://review.openstack.org/23944717:13
*** HeOS has quit IRC17:13
openstackgerritPaul Belanger proposed openstack-infra/zuul: Bump APScheduler to >=3.0  https://review.openstack.org/26063717:14
*** vgridnev has joined #openstack-infra17:14
pabelangerjeblair: would be interested in your thoughts ^17:14
pabelangerbumping to APScheduler 3.0, but need to dig more to see what configuration changes there are17:15
openstackgerritYuriy Zveryanskyy proposed openstack/diskimage-builder: [EXPERIMENTAL] Ansible deploy ramdisk for Ironic  https://review.openstack.org/23944717:15
pabelangerjeblair: main reason for this, fedora has python-APScheduler 3.0.517:15
*** piet has quit IRC17:15
*** davideagnello has quit IRC17:15
*** apuimedo_ has quit IRC17:17
*** harlowja_at_home has quit IRC17:18
*** apuimedo_ has joined #openstack-infra17:19
*** unicell1 has quit IRC17:19
*** isaacb has quit IRC17:20
AJaegertherve: regarding 259643: Will your change work with stable/kilo and stable/liberty as well? Please comment in the review...17:22
*** apuimedo_ has quit IRC17:23
fungi2015-12-22 17:07:15,356 ERROR [o.a.a.b.BrokerService] Failed to start ActiveMQ JMS Message Broker (localhost, null). Reason: org.apache.activemq.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).17:23
*** davideagnello has joined #openstack-infra17:23
*** kzaitsev_mb has quit IRC17:23
openstackgerritMerged openstack-infra/reviewstats: Add astara to projects  https://review.openstack.org/25731417:23
fungithat seems to be causing puppetdb to shut down immediately when started17:23
openstackgerritPaul Belanger proposed openstack-infra/zuul: Bump APScheduler to >=3.0  https://review.openstack.org/26063717:24
*** apuimedo_ has joined #openstack-infra17:24
openstackgerritMerged openstack-infra/project-config: Add experimental job to check Rally installation on Centos 7  https://review.openstack.org/26025817:24
*** eil397 has joined #openstack-infra17:25
openstackgerritMonty Taylor proposed openstack/requirements: Bump os-client-config constraint and minimum  https://review.openstack.org/26009217:25
funginibalizer: you aware of anything that might have caused puppetdb to start going toes-up recently?17:25
fungii see we also have warnings about postgresql <=9.1 support being deprecated, and another warning about being unable to install optimal indexing17:26
*** thorst_ has joined #openstack-infra17:27
fungi"For maximum index performance, we recommend using PostgreSQL 9.3 or greater."17:27
fungiso much fun stuff in the puppetdb log17:27
SpamapSwow.. every time I click a review link now, the new horrible UI just slaps me in the face17:27
*** dizquierdo has quit IRC17:27
fungiSpamapS: gerritslap!17:27
SpamapSI wonder if the new UI is as hated in other gerrit user communities as it is here?17:28
BobBallIs that the new trout slap?17:28
*** apuimedo_ has quit IRC17:28
crinklefungi: cacti says puppetdb is almost out of space?17:28
fungicrinkle: ooh!17:28
*** apuimedo_ has joined #openstack-infra17:29
*** derekh has quit IRC17:29
fungicrinkle: right you are, that probably explains the corrupt transaction log. i bet it's truncated17:29
*** peristeri has quit IRC17:29
fungii should have looked at disk utilization first17:29
*** woodster_ has joined #openstack-infra17:29
*** zeih has quit IRC17:30
*** devananda has quit IRC17:30
AJaegermestery, dougwig, armax : Could one of you neutron liasons review https://review.openstack.org/#/c/256719/ , please?17:30
nibalizerfungi: crinkle neat17:31
fungiwe have about 8.5gb in /var (more than a third of the rootfs)17:32
fungiinterestingly, we have >3gb in /usr/src17:32
fungiheaders packages for ~70 kernel versions. that seems pretty unnecessary17:33
*** thorst_ has quit IRC17:33
*** apuimedo_ has quit IRC17:33
fungiapt-get autoremove is cleaning up 4gb on disk now17:34
*** dtantsur is now known as dtantsur|afk17:35
*** apuimedo_ has joined #openstack-infra17:36
*** tmcpeak has joined #openstack-infra17:36
fungiand apt-get clean freed up almost another 4gb17:36
tmcpeakAJaeger, fungi: o/17:36
*** jerryz has joined #openstack-infra17:37
*** maishsk has joined #openstack-infra17:37
*** piet has joined #openstack-infra17:38
*** dprince has joined #openstack-infra17:39
*** devananda has joined #openstack-infra17:40
nibalizerfungi: cool17:42
tmcpeakfungi: can you take a look at this bad boy when you get a chance? https://review.openstack.org/#/c/260192/17:42
nibalizerwhat caused apt to keep so many files around?17:42
openstackgerritJames E. Blair proposed openstack-infra/zuul: Support project templates  https://review.openstack.org/26064917:43
*** apuimedo_ has quit IRC17:43
*** zeih has joined #openstack-infra17:43
*** apuimedo_ has joined #openstack-infra17:43
*** resker has quit IRC17:44
*** zns has joined #openstack-infra17:44
*** zns has quit IRC17:44
*** jyuso1 has joined #openstack-infra17:45
funginibalizer: unattended-upgrades doesn't autoremove or clean by default17:45
*** notnownikki has quit IRC17:46
fungithere was a change in review to switch that on recently, not sure if it got approved and isn't taking effect for some reason, or is still under discussion17:46
dougwigAJaeger: looking17:46
*** zeih has quit IRC17:47
fungicrinkle: nibalizer: since the puppetdb service is complaining about corruption in /var/lib/puppetdb/mq/localhost/KahaDB/db-12340.log is there any reason i shouldn't move that transaction log out of the way and retry starting? we'd just lose some history? and we only keep the last 10 reports for a node anyway, right?17:47
*** e0ne has quit IRC17:47
nibalizerfungi: im fine with that17:48
fungigiving it a try17:48
*** abregman is now known as abregman|nb17:49
*** piet has quit IRC17:50
*** cburgess has quit IRC17:50
fungino such luck. still complaining that it can't start the broker, "probably due to KahaDB corruption or version incompatibility after a PuppetDB downgrade"17:50
nibalizer'Unable to start broker in "/var/lib/puppetdb/mq". This is probably due to KahaDB corruption or version incompatibility after a PuppetDB downgrade (see "KahaDB Corruption" in the PuppetDB manual)'17:50
nibalizerhaha jinx17:51
nibalizerhave you issued a service puppetdb restart?17:51
fungiyeah, that's after a restrat17:51
fungitips on an easy way to reinitialize an empty state?17:51
*** jyuso1 has quit IRC17:52
*** cburgess has joined #openstack-infra17:52
*** apuimedo_ has quit IRC17:52
fungilooks like it's complaining about /var/lib/puppetdb/mq/localhost/KahaDB/db-12340.log still even though i moved it out of there entirely17:53
*** apuimedo_ has joined #openstack-infra17:53
fungilooks like you started it again?17:53
*** apoorvad has joined #openstack-infra17:53
nibalizeri haven't started it no17:53
* anteaya is back17:54
anteayaanything I can do to help?17:54
fungioh, or maybe i was reading the uncaught exceptions at shutdown as related to startup17:54
* anteaya goes to read backscroll17:54
nibalizerfungi: maybe its fixed?17:54
fungiseems to be happier now according to the logs17:54
nibalizerthe logs is hapier and puppetboard is functional17:54
fungiyep, web interface is good again17:54
*** admin0 has joined #openstack-infra17:54
fungineed to break for lunch, then spend a couple minutes prepping for our meeting an hour from now17:55
anteayafungi: enjoy lunch17:56
*** Swami has joined #openstack-infra17:56
*** apuimedo_ has quit IRC18:00
*** apuimedo_ has joined #openstack-infra18:02
*** openstackgerrit has quit IRC18:02
*** openstackgerrit has joined #openstack-infra18:03
*** jab is now known as zz_jab18:03
openstackgerritMerged openstack-infra/system-config: Initial commit of codesearch docs  https://review.openstack.org/25762618:04
openstackgerritJames E. Blair proposed openstack-infra/zuul: Support project templates  https://review.openstack.org/26064918:05
*** amuller_afk is now known as amuller18:05
*** rvasilets___ has quit IRC18:05
*** xyang1 has quit IRC18:06
AJaegerthanks, dougwig !18:06
*** apuimedo_ has quit IRC18:06
*** apuimedo_ has joined #openstack-infra18:08
*** cdent has quit IRC18:08
*** thorst_ has joined #openstack-infra18:12
*** apuimedo_ has quit IRC18:13
*** apuimedo_ has joined #openstack-infra18:14
*** piet has joined #openstack-infra18:14
*** rbrndt has quit IRC18:14
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add 'testing' node to site.pp  https://review.openstack.org/24149718:14
*** bexelbie_ has quit IRC18:15
anteayaAJaeger: I agree with your concern, 82 repos in gerrit without any kind of belief that they are all useful18:15
*** links has quit IRC18:15
anteayaI'm sure some are, just not sure all 82+ are18:15
*** kmartin has quit IRC18:16
AJaegeranteaya: even if they some would be usefull, I wonder how they are organized...18:17
AJaegerthis does not scale18:18
*** apuimedo_ has quit IRC18:18
*** thorst_ has quit IRC18:18
anteayaAJaeger: agreed, I think it is time for some overview and accounting here18:19
*** thorst_ has joined #openstack-infra18:19
anteayato share some understanding18:19
openstackgerritMerged openstack-infra/project-config: add more jobs for networking-fortinet  https://review.openstack.org/25671918:19
pabelangerfungi: so, openstackid on ubuntu trusty seems to work, but some issues.  Maybe we can stand up a new dev server and can work on it there.18:19
*** apuimedo_ has joined #openstack-infra18:20
pabelangerthings like enabling php modules (mcrypt) needs to be done18:20
*** zeih has joined #openstack-infra18:20
AJaegeranteaya: if you have time, I'd like the stack starting at 260202 to move forward, please18:21
*** rossella_s has quit IRC18:21
nibalizerSpamapS: re: changescreen you can look at the future future here: http://polygerrit.nibalizer.com/q/status:open18:21
*** rossella_s has joined #openstack-infra18:22
anteayaAJaeger: okay I think I have a sense of things again18:22
anteayaAJaeger: I'll take a look18:22
*** kmartin has joined #openstack-infra18:23
AJaegerthanks, anteaya18:23
SpamapSnibalizer: nice and clean18:24
nibalizerand unable to diff lol18:24
anteayaAJaeger: so when I look here: https://review.openstack.org/#/c/260202 and it says in the top right hand corner that there are 5 related changes, these are the children of this parent patch?18:24
nibalizerbut it used to be able to diff so I am thinking that is an issue with my deployment18:24
AJaegeranteaya: correct18:24
*** unicell has joined #openstack-infra18:25
clarkbjeblair: can you make sure that https://review.openstack.org/#/c/232800/15 looks good to you now? the commit message needs an update I think but the code seems to address your comments18:25
anteayaAJaeger: and when you say the stack is starting at 260202 is that the parent of all the other changes, or the last child?18:25
clarkbjesusaurus: for your comments on that test I can write a follow up change to address those comments to avoid rebases18:25
*** breton has left #openstack-infra18:25
AJaegeranteaya: parent of all, so 260202 will merge first18:25
anteayaAJaeger: thank you18:25
*** bharathm has joined #openstack-infra18:25
AJaeger260202 is a prerequisite for 260192 - 260192 adds a description and the test failed, so I updated the testsuite first18:26
jeblairclarkb: yeah, i agree the commit msg is out of date but didn't think that was worth holding up on so i previously applied +2; i don't think there's anything substatially diffirent since then, just a rebase yeah?18:26
jeblairclarkb: (i have +2d)18:26
openstackgerritPaul Belanger proposed openstack-infra/puppet-openstackid: Enable php5-mcript  https://review.openstack.org/26066018:27
pabelangerfungi: so far, only issue ^18:27
clarkbjeblair: oh I was just diffing against where I least voted where you had comments must've missed a +2 that went away due to rebase18:27
anteayaAJaeger: so what happens when 260202 merges and there is no description?18:27
jeblairclarkb: ah cool18:27
*** thorst_ has quit IRC18:28
fungipabelanger: awesome--thanks!18:28
anteayaAJaeger: is this allowing a space  for their to be descriptions or forcing descriptions?18:28
clarkbok so I think whole stack is ready to go in this case, I will approve after the meeting and get the service restarted18:28
clarkbjesusaurus: and I will push a followup chane ot address your -1 to avoid another rebase on that stack18:28
AJaegeranteaya: description is an optional parameter18:28
anteayaAJaeger: thank you18:28
jesusaurusclarkb: a follow up is fine by me18:28
AJaegerFor builders, see ".Required('builders')" - so that's a required one18:29
*** piet has quit IRC18:29
*** baoli has quit IRC18:29
openstackgerritPhilipp Marek proposed openstack-infra/ciwatch: Remove jQuery filter to improve page load time  https://review.openstack.org/26019318:29
fungipabelanger: is that safe on precise as well? the challenge in this exercise is to be able to upgrade one server (development) without immediately upgrading the other (production)18:30
*** zeih has quit IRC18:30
*** zeih has joined #openstack-infra18:30
*** notnownikki has joined #openstack-infra18:30
greghaynesclarkb: w00t18:30
*** zeih has quit IRC18:30
*** zeih has joined #openstack-infra18:31
*** zeih has quit IRC18:31
*** zeih has joined #openstack-infra18:31
anteayaAJaeger: ah there it is, thank you18:31
*** dimtruck is now known as zz_dimtruck18:31
openstackgerritJames E. Blair proposed openstack-infra/zuul: Refactor pipeline loading  https://review.openstack.org/26066218:31
pabelangerfungi: Ya, I need to test that for sure.18:31
*** zeih_ has joined #openstack-infra18:32
*** xarses has quit IRC18:35
anteayaAJaeger: so this file you take away the filters on the newly created gate-project-config-example-pep8 job, since it is no longer filtered using gate-project-config-pep8: https://review.openstack.org/#/c/260192/6/zuul/layout.yaml18:35
anteayaAJaeger: so that is a difference for the -example job, yes?18:35
*** harlowja has quit IRC18:36
*** zeih has quit IRC18:36
*** apuimedo_ has quit IRC18:36
*** changbl has quit IRC18:36
*** harlowja has joined #openstack-infra18:36
openstackgerritMerged openstack-infra/project-config: Add description label to jenkins checker  https://review.openstack.org/26020218:37
*** piet has joined #openstack-infra18:37
*** e0ne has joined #openstack-infra18:37
*** vilobhmm11 has joined #openstack-infra18:37
*** apuimedo_ has joined #openstack-infra18:37
crinkleclarkb: want to look at 235005, 258193, and 257640 for me today? (mysql)18:38
*** _nadya_ has joined #openstack-infra18:38
*** _nadya_ has quit IRC18:38
AJaegeranteaya: running the job on both projects was bad style18:38
AJaegerI could add the filter back for project-config-example but with the few changes going into that one, we don#t need it.18:39
openstackgerritIan proposed openstack-infra/jenkins-job-builder: Adding Build Keeper plugin.  https://review.openstack.org/26066418:39
anteayaAJaeger: no argument from me18:39
jesusaurusclarkb: is your logstash curator cron working? I just noticed that mine still has unescaped %s in the crontab18:40
anteayaokay fine, just acknowledging that the code change does affect that job18:40
anteayaasselin__: ^^18:40
openstackgerritMorgan Fainberg proposed openstack-infra/shade: No Mutable Defaults  https://review.openstack.org/26066518:40
clarkbjesusaurus: let me check18:41
anteayaAJaeger: mind including that point in the commit message on 260192?18:41
*** piet has quit IRC18:41
AJaegerwill do directly18:42
clarkbjesusaurus: ya looks like it is working now18:42
anteayaAJaeger: thank you18:42
*** angdraug has joined #openstack-infra18:42
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Replace bashate/pep8 in project-config by linters  https://review.openstack.org/26019218:43
*** _nadya_ has joined #openstack-infra18:43
jesusaurusclarkb: ah, looks like maybe a problem with install_modules.sh on my end18:43
AJaegeranteaya: done ^18:43
*** peristeri has joined #openstack-infra18:44
clarkbcrinkle: so looking at these changes we can interrogate the version that is installed to determine how to use the mysql module?18:44
clarkbcrinkle: I didn't realize this was possible, thats a neat trick18:46
crinkleclarkb: yes18:46
crinkleclarkb: yep it's a new feature in stdlib from nibalizer18:46
anteayaI am trying to +2 https://review.openstack.org/#/c/260192/6 and when I click the reply button I have a text box and two buttons post and cancel but zero radio buttons to vote18:47
*** hdd has joined #openstack-infra18:47
clarkbcrinkle: do the paste and wiki use of mysql also need the version checking like nodepool?18:48
*** salv-orlando has quit IRC18:48
AJaegeranteaya: switch to revision 718:48
crinkleclarkb: I didn't use it there since it's all in system-config and can be done all at once18:48
AJaegertop-left corner18:48
anteayasomehow I was on patchset 6/7 yet refreshing the patch did not take me to revision 718:48
*** apuimedo_ has quit IRC18:48
clarkbcrinkle: gotcha18:49
anteayaAJaeger: yeah, very time consuming18:49
AJaegeryour URL contains a 618:49
anteayarefreshing used to just take me to the most recent version18:49
*** EricGonczer_ has joined #openstack-infra18:49
*** apuimedo_ has joined #openstack-infra18:49
clarkbcool so I guess I should just confirm that innodb is the default in puppet-mysql for my own sanity, but otherwise this set of changes lgtm18:50
crinkleclarkb: cool18:50
crinkleclarkb: this will cause mysql to restart but I don't forsee any other issues18:51
*** piet has joined #openstack-infra18:51
anteayaAJaeger: this patch is in a can't merge state: https://review.openstack.org/#/c/260420/118:52
*** _nadya_ has quit IRC18:52
clarkbcrinkle: ok18:52
clarkbmaybe we should stop the services manually then and schedule this18:52
nibalizercrinkle: i think i have reviewed all the relevant patches18:52
nibalizerlet me know if  i missed something18:52
clarkb"wiki and paste and nodepool will be offline for 30 minutes while we update infrastructure management of these services"18:52
clarkbcrinkle: nibalizer ^ what do you think18:53
crinklenibalizer: ty18:53
dhellmannAJaeger : can you join us in #openstack-oslo to discuss a translation question?18:53
nibalizerclarkb: meeting in 10 minutes though right?18:53
nibalizermaybe want to wait until after18:53
AJaegerdhellmann: will do18:53
*** zeih has joined #openstack-infra18:53
clarkbnibalizer: yes after, I am saying we should properly shcedule this since the services will flap18:54
*** hdd has quit IRC18:54
clarkbrather than just do it18:54
*** zeih_ has quit IRC18:55
clarkbI am having a hard time seeing how override_optons does anything but affect hte user used18:55
*** notnownikki has quit IRC18:56
clarkboh because puppet can't update a var in place so the name changes18:56
*** rbrndt has joined #openstack-infra18:56
anteayaclarkb: this is puppet-mysql changes to those services?18:57
clarkbanteaya: we are upgrading the puppet-mysql version which will apparently restart mysql18:58
anteayaokey dokey, thanks18:58
*** apuimedo_ has quit IRC18:58
clarkbanteaya: the services and mysql themselves shouldn't change, but the restart of the db will cause an outage18:58
*** ihrachys has joined #openstack-infra18:58
fungiclarkb: crinkle: if we want to do this after the meeting, can we coordinate a paste outage to also move its db to trove? i'm happy to drive that part, and that way we don't need additional outages for it18:59
anteayaclarkb: cool18:59
clarkbfungi: sure? I thought it was still broken?18:59
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for manila volume create NoValidHost bug 1528665  https://review.openstack.org/26067018:59
openstackbug 1528665 in Manila "Share instance fails to create due to volume creation fails" [Undecided,New] https://launchpad.net/bugs/152866518:59
fungiclarkb: we never retried after we discovered how to fix the timeouts, afaik18:59
fungiit's that (weekly meeting) time once again! find us in #openstack-meeting for the next hour19:00
clarkbthough I am already doing image builders after the meeting19:00
clarkbso don't wantto commit to too much all at once19:00
*** apuimedo_ has joined #openstack-infra19:00
*** lyz_ is now known as lyz19:00
fungiahh, okay19:00
*** zz_dimtruck is now known as dimtruck19:00
fungiwell, if i do the paste migration first, then it shouldn't require an outage when the wiki/nodepool outage happens19:01
*** qb_ has joined #openstack-infra19:04
*** ianw has quit IRC19:06
*** ianw has joined #openstack-infra19:07
*** dimtruck is now known as zz_dimtruck19:07
*** apuimedo_ has quit IRC19:07
*** fawadkhaliq has quit IRC19:07
*** apuimedo_ has joined #openstack-infra19:08
*** piet has quit IRC19:08
*** vilobhmm11 has quit IRC19:09
*** vilobhmm11 has joined #openstack-infra19:09
*** smarcet has quit IRC19:09
*** aopincar has joined #openstack-infra19:10
openstackgerritMerged openstack-infra/system-config: Upgrade the stdlib module to 4.10.0  https://review.openstack.org/23500519:11
*** apuimedo_ has quit IRC19:12
*** FallenPegasus has quit IRC19:13
*** xarses has joined #openstack-infra19:14
*** dmsimard is now known as caffeine19:14
*** apuimedo_ has joined #openstack-infra19:14
*** caffeine is now known as dmsimard19:14
*** xarses has quit IRC19:15
*** xarses has joined #openstack-infra19:15
*** mdenny has joined #openstack-infra19:16
*** piet has joined #openstack-infra19:17
*** ihrachys has quit IRC19:18
openstackgerritMerged openstack-infra/elastic-recheck: Add query for manila volume create NoValidHost bug 1528665  https://review.openstack.org/26067019:20
openstackbug 1528665 in Manila "Share instance fails to create due to volume creation fails due to NoValidHost" [Undecided,Confirmed] https://launchpad.net/bugs/152866519:20
*** baoli has joined #openstack-infra19:21
*** baoli_ has joined #openstack-infra19:22
*** admin0 has left #openstack-infra19:22
openstackgerritPaul Belanger proposed openstack-infra/puppet-openstackid: Enable php5-mcript  https://review.openstack.org/26066019:22
*** thorst_ has joined #openstack-infra19:25
*** baoli has quit IRC19:26
*** apuimedo_ has quit IRC19:26
*** apuimedo_ has joined #openstack-infra19:28
*** claudiub has quit IRC19:28
*** rossella_s has quit IRC19:30
*** thorst_ has quit IRC19:32
*** apuimedo_ has quit IRC19:32
*** piet has quit IRC19:33
*** vgridnev has quit IRC19:33
*** apuimedo_ has joined #openstack-infra19:33
*** apuimedo_ has quit IRC19:38
*** hashar has joined #openstack-infra19:39
*** apuimedo_ has joined #openstack-infra19:39
*** zeih_ has joined #openstack-infra19:40
*** vgridnev has joined #openstack-infra19:44
*** zeih has quit IRC19:44
*** dims has quit IRC19:45
*** apuimedo_ has quit IRC19:46
*** salv-orlando has joined #openstack-infra19:47
*** apuimedo_ has joined #openstack-infra19:47
*** _nadya_ has joined #openstack-infra19:48
*** _nadya_ has quit IRC19:48
*** eharney has quit IRC19:50
*** Ramanjaneya has quit IRC19:51
*** hdd has joined #openstack-infra19:54
*** lucasagomes is now known as lucas-pto19:55
*** apuimedo_ has quit IRC19:56
clarkbI have approved the nodepool image builders change stack19:56
*** apuimedo_ has joined #openstack-infra19:57
*** markvoelker has quit IRC19:57
openstackgerritMonty Taylor proposed openstack/requirements: Bump os-client-config constraint and minimum  https://review.openstack.org/26009219:58
*** aopincar has quit IRC19:59
*** terriyu has joined #openstack-infra19:59
openstackgerritColleen Murphy proposed openstack-infra/puppet-nodepool: Add logic to work with old and new mysql module  https://review.openstack.org/25819320:00
*** xyang1 has joined #openstack-infra20:00
pleia2my turn to be sick, away resting for a bit20:01
*** iraj has joined #openstack-infra20:01
*** ihrachys has joined #openstack-infra20:01
*** iraj has left #openstack-infra20:01
nibalizerclarkb: did you approve the puppet change too?20:01
*** apuimedo_ has quit IRC20:01
fungiget well, pleia2!20:01
clarkbnibalizer: no, it isn't necessary20:01
greghaynesnibalizer: I think the plan is to run default first20:01
anteayapleia2: :(20:01
jeblairfungi: i think the way i have been looking at the scripts situation is that basic things that don't need to change often should be slave scripts.  i figured the 'install stuff using bindep' script would be among those20:01
anteayapleia2: hope you feel better soon20:02
AJaegerpleia2: get well soon!20:02
jeblairfungi: but if it's changing enough to be a concern, maybe we should pull it back into jjb.  and maybe we should rethink that and pull more into jjb too.20:02
clarkbnibalizer: step 2 is to update puppet to start splitting the services apart, but it should run the service as is against zuul geard just fine20:02
jeblairfungi: none of that addresses the testing aspect; only ease of changes20:02
clarkbnibalizer: on the same host and all that so not what we want long term but good for "does this work"20:02
fungijeblair: yeah, agreed but those scripts start somewhere and having them self-tested while they're on their way to stabilizing would help reduce risks20:02
*** ihrachys has quit IRC20:02
*** apuimedo_ has joined #openstack-infra20:03
nibalizerclarkb: we don't need a service definition and an init script?20:03
fungilike, for example, now i want the script to confirm that it did in fact install everything we wanted (because you can't trust yum's exit codes on nonexistent packages)20:03
clarkbnibalizer: nope nodepoold starts one for us by default20:03
*** ybathia has joined #openstack-infra20:03
nibalizeroh ok20:03
*** zeih_ has quit IRC20:03
*** dims has joined #openstack-infra20:04
greghaynesYep, by default it operates the same way as before, you just have the option to run them separately now.20:04
fungiand might be nice to add some loops around things like package list updates and package retrieval to mitigate some network-related breakage20:04
fungibeing able to demonstrate in our ci that these additions work would go a long way to making them easier to review20:04
openstackgerritAndreas Jaeger proposed openstack-infra/infra-specs: Gerrit-2.11 spec is implemented  https://review.openstack.org/26069420:04
AJaegerhere's my action item ^20:05
nibalizergreghaynes: oh okay that makes more sense then20:05
*** bharathm has quit IRC20:05
fungias it stands, i'll probably just continue trickling improvements into it while leaving the jobs using it only experimental for a while still20:05
*** eharney has joined #openstack-infra20:05
openstackgerritColleen Murphy proposed openstack-infra/system-config: Upgrade mysql module to 3.6.1  https://review.openstack.org/25764020:05
anteayaAJaeger: thank you20:06
*** terriyu has quit IRC20:06
AJaegeranteaya: thanks for the linters reviews. Anybody here to give another +2 for https://review.openstack.org/#/c/260192 ? jhesketh gave his +2 already and then I updated the commit message...20:06
clarkbgreghaynes: and unit tests failed :(20:06
crinkleclarkb: I updated the puppet-nodepool and system-config patches to be explicit about the default-storage-engine setting20:06
fungijeblair: and keeping the logic inside the job configuration isn't a ton better. it's still not self-tested, just has a slightly faster turn-around on the proposal->implementation cycle20:07
AJaegeranteaya: I'll rebase 260420 once its dependencies are merged...20:07
clarkbcrinkle: thanks20:07
greghaynesclarkb: pymysql fixture?20:07
clarkbgreghaynes: osclientconfig20:07
greghaynesclarkb: wah20:07
clarkbgreghaynes: that may be a sign20:07
fungijeblair: though in zuul v3 that changes of course20:07
anteayaAJaeger: awesome20:07
greghaynesdid they release?20:07
clarkbRegion fake-region is not a valid region name for cloud envvars. Valid choices are . Please note that region names are case sensitive.20:07
greghaynesclarkb: there was a release ~1 day ago :(20:08
*** ybathia_ has joined #openstack-infra20:08
*** hdd has quit IRC20:08
openstackgerritMerged openstack/diskimage-builder: yum-minimal: do not configure eth0 & eth1 for DHCP automatically  https://review.openstack.org/25717220:09
*** thorst_ has joined #openstack-infra20:09
*** apuimedo_ has quit IRC20:10
*** apuimedo_ has joined #openstack-infra20:10
clarkbIe2f3d2ca3ccbe7e3dd674983136b42c323544997 is the change in oscc that added the error20:10
*** ybathia has quit IRC20:10
*** ybathia_ is now known as ybathia20:10
openstackgerritMonty Taylor proposed openstack-infra/shade: Make server variable expansion optional  https://review.openstack.org/25743720:11
openstackgerritMonty Taylor proposed openstack-infra/shade: Have inventory use os-client-config extra_config  https://review.openstack.org/25743620:11
clarkbgreghaynes: I think I may have a fix20:11
clarkbgive me a few20:11
*** dims_ has joined #openstack-infra20:11
greghaynesclarkb: sweet20:12
mriedemhmm, tox not liking some gate runs? ValueError: No closing quotation20:13
AJaegermriedem: tox 2.x incompatiblity20:13
AJaegerwhich project is this on?20:13
fungimriedem: mostly on stable branches?20:14
mriedemmostly master20:14
openstackgerritSpencer Krum proposed openstack-infra/puppet-nodepool: Nodepool image build workers  https://review.openstack.org/24009320:14
*** dims has quit IRC20:14
nibalizergreghaynes: clarkb i think thats ^^ the way we want the puppet to look20:14
greghaynesnibalizer: cool20:14
*** hdd has joined #openstack-infra20:14
clarkbgreghaynes: basically we need to write a cloud config for all tests now :/20:15
clarkbgreghaynes: and populate it iwth a valid region20:15
*** apuimedo_ has quit IRC20:15
AJaegermriedem: give me one repo name and I'll propose a patch...20:15
*** apuimedo_ has joined #openstack-infra20:15
greghaynesclarkb: wah? What was it doing before for regions by default?20:15
openstackgerritMerged openstack/diskimage-builder: yum-minimal: leave behind dummy /etc/resolv.conf  https://review.openstack.org/25717320:16
clarkbgreghaynes: it would return None I think20:16
openstackgerritMerged openstack/diskimage-builder: yum-minimal : install selinux policy packages  https://review.openstack.org/25986720:16
mriedemAJaeger: https://github.com/openstack/designate/blob/master/devstack/gate/run_tempest_tests.sh#L3220:16
*** tzn has joined #openstack-infra20:16
mriedemAJaeger: ahhhh https://github.com/openstack/designate/commit/0f5fd246c7ecb851aff1ec489bca1904110fe9fa20:16
greghaynesclarkb: oh, it is because we are setting a region_name20:17
AJaegermriedem: exactly, remove the comments20:17
clarkbgreghaynes: ya, maybe we just unset region_name?20:17
greghaynesclarkb: hrm, this is wierd. If we pass in a region_name then why would that force us to have one in the config20:17
clarkbgreghaynes: but also, the best part is we pass validate = False20:17
greghaynesseems backwards20:17
clarkbgreghaynes: because if it doesn't find one in the config oscc raises20:18
mriedemhence the question about stable, we have to backport those tox.ini changes20:18
clarkbgreghaynes: I agree that seems broken in oscc, particularly since we say validate = False20:18
AJaegerso, designate is fixed, mriedem . Let me fix devstack-plugin-ceph...20:18
mriedemironic is fixed too20:18
*** zeih has joined #openstack-infra20:19
*** rfolco has quit IRC20:19
greghaynesclarkb: Ya, so I think occ should not fail here if you pass in a region as an arg20:20
greghaynesclarkb: regardless of validate20:20
mriedemAJaeger: tracking with 152756220:20
mriedembug 152756220:20
openstackbug 1527562 in Ironic "tox fails when run - ValueError: No closing quotation" [High,Fix released] https://launchpad.net/bugs/1527562 - Assigned to Lucas Alvares Gomes (lucasagomes)20:20
greghaynesmordred: ^20:20
clarkbgreghaynes: agreed20:20
clarkbgreghaynes: reading the code it seems pretty redundant20:20
openstackgerritMonty Taylor proposed openstack-infra/shade: Make server variable expansion optional  https://review.openstack.org/25743720:20
clarkbgreghaynes: you pass in a region and only if that region is in the config does it accepti20:20
AJaegermriedem: thanks20:20
greghaynesclarkb: yes, it is effectively sanity checking that the arg you passed in is valid20:21
*** boris-42 has joined #openstack-infra20:21
clarkbgreghaynes: except that has the side effect of requiring that config everywhere20:21
clarkbso we can fix it in nodepool's tests by writing that config everywhere20:21
jesusaurusso that check isn't respecting validate = False?20:21
clarkbor make oscc less picky20:21
clarkbjesusaurus: nope20:21
greghaynesclarkb: I want to fix occ20:22
clarkbgreghaynes: ++20:22
greghaynesclarkb: not sure if we want to do both since nodepool is going to be broken until that releases20:22
*** EricGonczer_ has quit IRC20:22
AJaegermriedem: training-labs is fixed already20:23
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for tox ValueError fail bug 1527562  https://review.openstack.org/26070120:23
openstackbug 1527562 in devstack-plugin-ceph "tox fails when run - ValueError: No closing quotation" [Undecided,In progress] https://launchpad.net/bugs/1527562 - Assigned to Andreas Jaeger (jaegerandi)20:23
clarkbjesusaurus: apparently validate = False means don't make a ksc auth plugin instance20:23
clarkbmordred: are you around?20:23
mordredclarkb: yes20:23
AJaegermriedem: releases is fixes AFAIK as well, is devstack fixed?20:24
mordredclarkb, greghaynes point me at the issue?20:24
clarkbmordred: oscc now requires that the region name be in your clouds.yaml before you can use the region name20:24
greghaynesmordred: http://logs.openstack.org/75/195175/36/gate/gate-nodepool-python27/f01d50c/console.html20:24
clarkbso we pass in region-name = fake-region to get_one_cloud and it raises and exception because we don't have a clouds.yaml with a region name in it20:24
mriedemAJaeger: looks like it20:24
openstackgerritPaul Belanger proposed openstack-infra/zuul: Bump python-daemon to 2.1.0  https://review.openstack.org/26070220:24
*** EricGonczer_ has joined #openstack-infra20:25
mordredso, the _intent_ there is that if your cloud defines a list of regions in the vendors yaml and you request a bad region name, that we give you a "that's an invalid region name" error20:25
mordredthis is, obviously, a bug in that :)20:25
*** hdd has quit IRC20:25
mordredI will see if I can come up with a somewhat sane patch to fix on the occ side20:26
clarkbmordred: I can fix that if that is what the intent is20:26
*** baoli_ has quit IRC20:26
*** e0ne has quit IRC20:28
*** tmcpeak has quit IRC20:28
clarkbmordred: greghaynes https://review.openstack.org/260704 something like that20:29
clarkbI think we also need to handle the case where regions and region_name are nil equivalents20:29
pabelangerif anybody wants to look at python-daemon patch, checkout zuul ^.  Seems they added new logic to setup supplementary groups, which looks to need super user creds20:29
clarkbnot sure if we just return '' as region_name in that case or if that is an error20:30
mordredclarkb: that looks about right20:30
pabelangernot sure why they didn't default False out of the box20:30
clarkbmordred: if regions and region_name are falsey should it also raise and require one or the other?20:30
jesusauruspabelanger: ah, that issue strikes again20:30
greghaynesclarkb: if region_name is '' and regions is [] then we might still want to error there20:31
pabelangerjesusaurus: Oh, where did you see it?20:31
clarkbgreghaynes: ya thats what I am wondering about, is that avalid config or an error?20:31
hasharpabelanger: Joshua had a patch for zuul / python-daeon20:31
* greghaynes looks at mordred20:31
pabelangerhashar: Ya, mine bumps the min requirements to support it :)20:31
pabelangerhashar: fedora packaging20:31
jesusauruspabelanger: in log_processor20:31
hasharpabelanger: got it by hit a few weeks ago ( https://review.openstack.org/#/c/250315/ )20:31
pabelangerjesusaurus: Ah, going to check code20:32
*** tzn has quit IRC20:32
*** e0ne has joined #openstack-infra20:32
pabelangerhashar: Ha ha!20:32
pabelangerwill abandon mine20:32
mordredclarkb: no, region is not required in all places20:32
greghaynesmordred: clarkb ok, LGTM then20:33
*** irenab has joined #openstack-infra20:33
hasharpabelanger: sorry for my crappy english. It is late here :D20:33
pabelangerany zuul-core interested in 250315? :)20:33
*** xarses has quit IRC20:33
pabelangerhashar: np20:33
*** odlD2 has quit IRC20:34
clarkbcool, now just to figure out how to test this20:34
pabelangerhashar: Oh, hit me up tomorrow or next week. Want to chat about your nodepool SIGINT patch you landed a while back20:34
pabelangerhashar: can't get it working as expected20:34
*** xarses has joined #openstack-infra20:34
greghaynesclarkb: so, it is probably a good time for me to foods since were gonig to just be waiting on this to merge/release20:34
clarkbmordred: in the case of no regions and no region_name what should we return? is the expanded region of '' correct?20:34
clarkbgreghaynes: ya20:35
openstackgerritMerged openstack-infra/elastic-recheck: Add query for tox ValueError fail bug 1527562  https://review.openstack.org/26070120:35
openstackbug 1527562 in devstack-plugin-ceph "tox fails when run - ValueError: No closing quotation" [Undecided,In progress] https://launchpad.net/bugs/1527562 - Assigned to Andreas Jaeger (jaegerandi)20:35
hasharhave fun folks, I am off *wave*20:36
*** apuimedo_ has quit IRC20:36
mordredclarkb: dunno. I need to read keystoneauth code ... notmorgan ^^ ? you know ?20:37
* notmorgan reads up20:37
*** hashar has quit IRC20:37
*** apuimedo_ has joined #openstack-infra20:38
notmorganoh weird20:38
notmorgani don't know off the top of my head, but i think ''20:39
*** hdd has joined #openstack-infra20:39
clarkbnotmorgan: { 'region': '', 'values': {} } is what we will return20:40
clarkbwith my change as is20:40
mordredyah - '' looks like it will work20:40
clarkbmordred: notmorgan so not the dict above? just ''?20:40
clarkbI will have ot update my change if so20:41
AJaegerasselin: I'll update project-config-example...20:41
asselin__AJaeger, thanks20:42
*** zeih has quit IRC20:44
mordredclarkb: the dict20:44
mordredthe region value should be ''20:44
openstackgerritAndreas Jaeger proposed openstack-infra/project-config-example: Use linters tox environment  https://review.openstack.org/26071020:45
*** apuimedo_ has quit IRC20:45
AJaegerasselin__: ^20:45
clarkbmordred: cool my change should be correct then, working on a set of tests though20:45
*** apuimedo_ has joined #openstack-infra20:45
*** zeih has joined #openstack-infra20:46
*** odlD2 has joined #openstack-infra20:47
*** maishsk has quit IRC20:48
openstackgerritfumihiko kakuma proposed openstack-infra/devstack-gate: Add the log of package list for pip3  https://review.openstack.org/24601720:49
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use linters job for project-config-example  https://review.openstack.org/26071120:49
AJaegerasselin__: and this one ^20:49
*** salv-orlando has quit IRC20:49
*** baoli has joined #openstack-infra20:49
AJaegerall changes for the linters job now use the linters topic: https://review.openstack.org/#/q/topic:linters+status:open20:50
*** tsg has quit IRC20:50
*** apuimedo_ has quit IRC20:50
*** apuimedo_ has joined #openstack-infra20:50
*** zeih has quit IRC20:51
asselin__anyone know how to you see parent/child relationships in new gerrit ui?20:52
AJaegerasselin__: the "Related changes" on the top left20:52
anteayaasselin__: top right20:52
*** kgiusti has left #openstack-infra20:52
*** bharathm has joined #openstack-infra20:53
asselin__oh, so those are the depends-on...ok20:53
asselin__what about needed-by?20:53
*** baoli has quit IRC20:54
asselin__nevermind...goes up in that case..ok not as intuitive, but good enough20:54
anteayathe patches taht used to be considered dependencies are also listed there20:54
anteayayeah, I'm trying to figure it out as well myself20:55
*** baoli has joined #openstack-infra20:55
* asselin__ goes to lunch20:55
Shrewsasselin__: yeah, i find that entire view totally unintuitive myself. one of the reasons i switched to gertty20:55
anteayaha ha ha20:55
*** apuimedo_ has quit IRC20:55
anteayalunch is never a bad idea20:55
*** apuimedo_ has joined #openstack-infra20:55
*** thorst_ has quit IRC20:57
openstackgerritmathieu bultel proposed openstack-infra/shade: Add normalize stack function for heat stack_list  https://review.openstack.org/25433720:57
*** thorst_ has joined #openstack-infra20:57
*** markvoelker has joined #openstack-infra20:58
openstackgerritDoug Wiegley proposed openstack-infra/puppet-openstackci: Do initial git commit with stock puppet files  https://review.openstack.org/26071420:58
*** nelsnelson has quit IRC20:59
openstackgerritJames E. Blair proposed openstack/gertty: Add notes to see reference-gertty.yaml for more info  https://review.openstack.org/26071521:00
*** bharathm has quit IRC21:00
*** nelsnelson has joined #openstack-infra21:01
openstackgerritVictor Ryzhenkin proposed openstack/requirements: Add openstack/murano-agent to projects.txt  https://review.openstack.org/25655521:01
*** markvoelker has quit IRC21:03
*** maishsk has joined #openstack-infra21:03
*** otter768 has quit IRC21:05
*** apuimedo_ has quit IRC21:05
*** edmondsw has joined #openstack-infra21:06
*** edmondsw has quit IRC21:06
*** thorst_ has quit IRC21:06
*** apuimedo_ has joined #openstack-infra21:06
*** zeih has joined #openstack-infra21:06
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Make install-distro-packages.sh more robust  https://review.openstack.org/26071721:07
clarkbmordred: ok updated with tests now21:07
clarkbmordred: should cover the entirety of that one method21:07
*** eharney has quit IRC21:07
clarkbmordred: if that looks good to you a release would be awesome21:07
clarkboh wait I need to add one more test case21:08
*** eharney has joined #openstack-infra21:08
*** odlD2 has quit IRC21:09
*** bharathm has joined #openstack-infra21:09
*** vgridnev has quit IRC21:11
*** qadaemon has joined #openstack-infra21:11
*** maishsk has quit IRC21:11
*** tmcpeak has joined #openstack-infra21:12
*** maishsk has joined #openstack-infra21:12
*** otter768 has joined #openstack-infra21:13
*** maishsk has quit IRC21:13
clarkbya my change is broken21:14
*** electrofelix has joined #openstack-infra21:15
*** woodster_ has quit IRC21:16
electrofelixWondering if anyone else has encountered issues trying to use the new gerrit with android mobile browsers?21:16
clarkbelectrofelix: it works fine for me so far except for the diff pages21:17
clarkbscrolling is almost impossible on the diffs21:17
openstackgerritMerged openstack-infra/ciwatch: Remove jQuery filter to improve page load time  https://review.openstack.org/26019321:17
electrofelixclarkb: Don't use it too much,but found out useful to be able to reply to comments, but now it keeps preventing me adding inline comments.21:18
clarkbI havne't tried to do inline comments because I can't really see what I am commenting on21:18
electrofelixOk, I'll look to log an issue about it. Presume storyboard?21:21
*** abregman|nb has quit IRC21:21
jeblairelectrofelix: https://code.google.com/p/gerrit/issues/list21:21
*** ajmiller_ is now known as ajmiller21:21
*** tonytan4ever has quit IRC21:21
*** apuimedo_ has quit IRC21:22
*** odlD2 has joined #openstack-infra21:22
*** apuimedo_ has joined #openstack-infra21:23
*** davidlenwell has quit IRC21:24
*** Mmike has quit IRC21:25
clarkbmordred: ok current patchset should be good21:26
clarkbtest case covers nodepool testing situation and additonal tests cover the old behavior21:26
*** Mmike has joined #openstack-infra21:27
*** Mmike has joined #openstack-infra21:27
electrofelixjeblair: so no need to track it locally?21:27
*** thingee has quit IRC21:27
*** adam_g has quit IRC21:27
*** baoli has quit IRC21:27
*** eil397 has quit IRC21:27
*** trown is now known as trown|outttypeww21:28
*** baoli has joined #openstack-infra21:28
clarkbgreghaynes: maybe you want to review https://review.openstack.org/260704 when you have a chance21:28
*** apuimedo_ has quit IRC21:28
greghaynesclarkb: now with 100% more tests?21:28
clarkbgreghaynes: ya21:28
clarkboh bah I broke other tests though21:29
clarkbgrumble grumble /me fixes21:29
*** apuimedo_ has joined #openstack-infra21:29
anteayaelectrofelix: tracking locally doesn't do anything, we can't change the upstream code any faster than gerrit devs21:29
anteayaactually they are faster at it21:29
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Make install-distro-packages.sh more robust  https://review.openstack.org/26071721:29
fungimust appease the bash8 overlords21:29
greghaynesclarkb: I have a nit21:30
greghaynesclarkb: if youre updating it21:30
clarkboh too late but I can reupdate21:30
*** thingee has joined #openstack-infra21:31
clarkbgreghaynes: we should hit that21:31
clarkbgreghaynes: but we just want t oreturn None in that case21:31
clarkbno known regions21:32
clarkbreturn value == None21:32
openstackgerritMerged openstack-infra/tripleo-ci: Add containers job to tripleo.org CI status  https://review.openstack.org/25917721:32
clarkbthe problem is _get_regions was overloaded between list known regions and return valid default fallback21:32
*** tzn has joined #openstack-infra21:32
clarkbso I split it up a bit to make that division of duties21:32
greghaynesAh, ok, I like21:33
clarkbgreghaynes: otherwise we get the valid default when trying to determine if the region name we passed in should just be used and it gets all confused21:33
*** bhunter71 has joined #openstack-infra21:33
*** adam_g has joined #openstack-infra21:33
*** adam_g has quit IRC21:34
*** adam_g has joined #openstack-infra21:34
anteayaAJaeger: what is the purpose of non-voting lint jobs? https://review.openstack.org/#/c/260402/121:34
anteayaAJaeger: to me that just translates as noise21:34
AJaegeranteaya: I was looking at changes like https://review.openstack.org/#/c/251223/2/zuul/layout.yaml21:35
*** davidlenwell has joined #openstack-infra21:36
anteayaAJaeger: I don't understand21:36
anteayaapparently I don't have as much context as I need for that to make sense to me as an argument21:37
AJaegerthat change adds a non-voting lint job (using bandit)21:37
anteayawhat is bandit?21:37
AJaegeranteaya: bandit is a security linter21:37
anteayaI still don't see the argument for wanting a non-voting lint job21:38
*** tzn has quit IRC21:38
anteayasecurity linter or no21:38
electrofelixanteaya: not expecting it to be fixed any faster, but how do you keep an eye on issues affecting the instance/version run for openstack usage?21:38
AJaegerso, is this a single case where people add a linter initially as non-voting and then migrate it to voting once it passes21:38
tmcpeakanteaya: security things can be harder to teach for developers21:38
tmcpeakso a project might want to run non-voting to educate their developers what things will not be done in the future21:38
AJaegeror is that a pattern - the pattern we use for introduction of new changes: First experimental, thne non-voting, then voting.21:38
*** bharathm has quit IRC21:38
anteayaelectrofelix: I ask zaro personally, and he reads the gerrit dev output of bugs and patches21:39
tmcpeakgenerally though I'd expect this to be mostly voting21:39
anteayaso a security linter is in a lint category of its own?21:39
fungiyeah, having more developers from our community get deeply embedded in the developer communities of the tools on which we rely is the best model we have21:39
anteayaAJaeger: if it is a single case why are we adding a template?21:39
AJaegeranteaya: I made this change to have *one* target for all linters: pep8, bashate, bandit...21:40
tmcpeakno, it's pretty much the same.  Let's say a project had a bunch of pep8 violating things.  You might say: "ok guys, we're going to start enforcing this in a month.  Meanwhile please look at the output to see what not to do"21:40
fungiregarding gerrit issue tracking impacting our user base21:40
*** e0ne has quit IRC21:40
anteayaAJaeger: yeah I see that, I just don't get the non-voting part21:40
AJaegeranteaya: I ask the question whether it's a single case, or the usual pattern...21:40
tmcpeakin this particular use of Bandit though, I'd expect people to go straight to voting21:40
anteayalint or don't lint21:40
anteayaI don't see why non-voting is useful21:40
tmcpeakhonestly it probably isn't21:40
AJaegeranteaya: that -nv change is not crucial for my patch series ;) It's not used anywhere, so I'm not pushing for it.21:41
anteayatmcpeak: I don't say okay guys for anything, as I see zero benefit to excluding the women21:41
tmcpeakI'd be ok with voting only for the 'linters' target21:41
AJaegerI see your point, we can come back to this in case somebody has real arguments...21:42
tmcpeakanteaya: hmm, sorry, I didn't mean it offensively, I'm from California and we just use it as a a figure of speech.  I say "ok guys" to groups that include women equally21:42
AJaegerand so far go with tmcpeak'S voting only for linters...21:42
anteayaokay well I have reviewed as much as makes sense to me21:42
AJaegerinteresting gate-sqlalchemy-migrate-pep8 has voting false - as only job21:43
AJaegerwhich supports anteaya's view IMO.21:43
AJaegeranteaya: I'll abandon it, thanks21:43
anteayaclarkb: I know you are busy with other things but this logstash 2.0 change is in merge conflict and jenkins is -1 on it: https://review.openstack.org/#/c/247714/21:43
*** davidlenwell has quit IRC21:43
anteayaclarkb: once you have finished your current thing21:44
anteayaAJaeger: thanks for understanding21:44
AJaegerthanks for reviewing and asking, anteaya !21:44
anteayaAJaeger: glad I could elbow myself enough space where I could hold a thought in my head to do so21:45
*** e0ne has joined #openstack-infra21:45
openstackgerritKaitlin Farr proposed openstack/requirements: Cap PyKMIP version  https://review.openstack.org/26059921:46
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Initial sphynx doc setup for publishing  https://review.openstack.org/25924521:48
openstackgerritMerged openstack-infra/tripleo-ci: Add ci logs helper script  https://review.openstack.org/23799221:50
*** apuimedo_ has quit IRC21:50
*** christiana_ has quit IRC21:51
*** electrofelix has quit IRC21:51
*** apuimedo_ has joined #openstack-infra21:51
*** adam_g has quit IRC21:52
*** bharathm has joined #openstack-infra21:54
*** davidlenwell has joined #openstack-infra21:55
*** adam_g has joined #openstack-infra21:56
*** adam_g has quit IRC21:56
*** adam_g has joined #openstack-infra21:56
*** apuimedo_ has quit IRC21:56
*** apuimedo_ has joined #openstack-infra21:57
*** changbl has joined #openstack-infra21:58
*** qadaemon has quit IRC21:59
*** e0ne has quit IRC21:59
*** markvoelker has joined #openstack-infra21:59
*** e0ne has joined #openstack-infra21:59
*** ociuhandu has quit IRC21:59
*** peristeri has quit IRC22:00
*** nelsnels_ has joined #openstack-infra22:00
*** piet has joined #openstack-infra22:00
*** nelsnelson has quit IRC22:00
openstackgerritMerged openstack/diskimage-builder: Fix growroot device detection  https://review.openstack.org/25974822:01
openstackgerritMerged openstack/diskimage-builder: Package installs defaults to tracing off  https://review.openstack.org/25971722:02
openstackgerritMerged openstack/diskimage-builder: Don't print trace unless trace is on in pkg-map  https://review.openstack.org/25971622:02
*** davidlenwell has quit IRC22:02
*** markvoelker has quit IRC22:03
*** thorst_ has joined #openstack-infra22:04
*** adam_g has quit IRC22:06
*** apuimedo_ has quit IRC22:06
*** apuimedo_ has joined #openstack-infra22:07
tmcpeakAJaeger: how are we looking on the original patch?22:07
tmcpeakthat one seems to have pretty good support22:08
*** xarses has quit IRC22:08
*** xarses has joined #openstack-infra22:08
*** thorst_ has quit IRC22:08
openstackgerritKaitlin Farr proposed openstack/requirements: Cap PyKMIP version  https://review.openstack.org/26059922:09
clarkbanteaya: huh how did that merge conflict22:09
clarkbanteaya: I will check on it22:09
anteayaclarkb: thanks22:09
openstackgerritJames E. Blair proposed openstack-infra/zuul: Alter config format to lists of dictionaries  https://review.openstack.org/26072722:09
AJaegertmcpeak: yeah, 260192 needs another review...22:10
tmcpeakahh ok cool22:10
*** adam_g has joined #openstack-infra22:10
*** adam_g has quit IRC22:11
*** adam_g has joined #openstack-infra22:11
clarkbanteaya: local git says no conflict so maybe just gerrit being gerrit?22:11
openstackgerritClark Boylan proposed openstack-infra/system-config: Logstash 2.0 compat ES output rule  https://review.openstack.org/24771422:11
clarkbrebase pushed to fix it22:11
jeblairif anyone is curious https://review.openstack.org/260727 is most of the way to the zuul v3 configuration syntax, setting aside most of the job configuration for the moment22:11
*** dprince has quit IRC22:12
jeblairit's a good one to look at just to see the config files in the tests22:12
*** davidlenwell has joined #openstack-infra22:12
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Fix shade tests with OCC 1.13.0  https://review.openstack.org/26073022:14
*** piet has quit IRC22:14
Shrewsmordred: jeblair: ^^^  shade tests has a sad22:14
*** AJaeger has quit IRC22:15
clarkbI am looking at https://review.openstack.org/#/c/260704/3 and it doesn't show jenkins' +122:15
clarkbis that a known issue?22:15
*** e0ne has quit IRC22:16
jeblairclarkb: jenkins+1 appears in gertty22:16
clarkbjeblair: yes :) I think this is actually a bug in hideci.js though22:16
clarkbit seems to be replacing the vote table with the test results table22:17
*** david-lyle has joined #openstack-infra22:17
anteayajeblair: thanks22:18
*** Jay_ has joined #openstack-infra22:18
asselin__anyone curious to see 3rd party ci docs gerenated? http://docs-draft.openstack.org/45/259245/2/check/gate-puppet-openstackci-docs/b5086c2/doc/build/html/22:18
jeblairclarkb: there's a vote table?22:19
jeblairi thought it just threw stuff into the reviewers area22:19
jeblairi hesitate to call that area a table22:19
clarkbsure it does something like that22:19
clarkbbut its not there even though jenkins has +1'd22:19
jeblairoh yeah, there's supposed to be a table there22:20
clarkbok its not just me and my browser22:20
anteayaa table would be an improvement22:21
*** apuimedo_ has quit IRC22:21
anteayaclarkb: I also do not see jenkins give a +1 in the spot I would expect it22:21
anteayaregardless of how we are defining that spot22:21
*** Jay__ has quit IRC22:21
openstackgerritMerged openstack/diskimage-builder: Extend root device hints for different types of WWN  https://review.openstack.org/24583922:22
anteayagiven the speed which which tourists are able to trigger their car alarms to go off after they arrive, I expect most suburban areas to be non-stop car alarm activity these days22:22
openstackgerritDoug Wiegley proposed openstack-infra/project-config-example: Fix typo referencing '17', and provide an example randomizer  https://review.openstack.org/26073422:23
*** apuimedo_ has joined #openstack-infra22:23
*** yamamoto has joined #openstack-infra22:24
*** tzn has joined #openstack-infra22:24
jeblairi asked if the gerritforge folks had plans to make their .deb packages use online reindexing for their automatic upgrades...22:24
jeblairthey said "you can propose a patch".22:25
*** tzn has quit IRC22:25
*** yamamoto has quit IRC22:25
clarkbyolo packaging22:25
clarkbits because I had patchset 3 selected not 422:26
clarkbthis is the most confusing /grump22:26
clarkbI really like the old/gertty way of showing you a change not a patchset22:26
clarkbwell that mystery is solved22:26
jeblairif the new way were really consistent, it wouldn't show you comments for ps4.22:26
greghaynesclarkb: this is how I was not able to find the rebase button22:26
clarkbjeblair: ++22:26
openstackgerritKaitlin Farr proposed openstack/requirements: Cap PyKMIP version  https://review.openstack.org/26059922:26
jeblairso it's like, half the screen is from the past, and half is from the future.22:27
*** thingee has quit IRC22:27
anteayathe twilight zone approach to code review22:27
anteayalike we needed something more to distract us22:28
fungialso worth noting, when leaving votes now if you have submit permission, there is no save and submit. the submit button now seems to only be available as a separate action from reviewing22:28
fungidiscovered that last night when we had to bypass running job on a change to fix the parameter functions so that we could start running jobs again22:29
fungiconfused me for a bit since i thought maybe we were missing a permission for the project-bootstrappers group in 2.11 acls22:30
*** mriedem has quit IRC22:32
*** esker has joined #openstack-infra22:33
*** dkranz has quit IRC22:34
*** apuimedo_ has quit IRC22:34
*** apuimedo_ has joined #openstack-infra22:35
*** thingee has joined #openstack-infra22:35
*** xyang1 has quit IRC22:36
openstackgerritJoey D. proposed openstack/diskimage-builder: Fix discoverd bug when dmidecode reports GB  https://review.openstack.org/25875722:37
*** apuimedo_ has quit IRC22:39
openstackgerritDoug Wiegley proposed openstack-infra/project-config-example: Fix typo referencing '17', and provide an example randomizer  https://review.openstack.org/26073422:40
*** apuimedo_ has joined #openstack-infra22:41
*** Qiming has joined #openstack-infra22:45
*** pahuang has joined #openstack-infra22:46
*** bharathm has quit IRC22:46
*** bharathm has joined #openstack-infra22:47
*** xarses has quit IRC22:47
*** Jay_ has quit IRC22:47
*** larainema has quit IRC22:49
*** Qiming has quit IRC22:50
*** apuimedo_ has quit IRC22:51
*** bharathm_ has joined #openstack-infra22:52
*** bharathm has quit IRC22:52
*** apuimedo_ has joined #openstack-infra22:52
*** mudassirlatif has quit IRC22:52
*** amuller has quit IRC22:53
jheskethHey all22:55
*** bharathm_ has quit IRC22:55
*** bharathm has joined #openstack-infra22:55
jheskethlooks like the post pipeline issue was solved22:55
jheskethwhat was the fix?22:55
*** mfisch` is now known as mfisch22:55
*** mfisch is now known as Guest1369622:56
*** apuimedo_ has quit IRC22:57
*** apuimedo_ has joined #openstack-infra22:59
fungijhesketh: zuul needed a restart to clear the defunct precise label registrations22:59
*** apoorvad has quit IRC22:59
jheskethyeah just caught up on that piece of scrollback22:59
jheskethand it looks like the post pipeline grew because gerrit was being slow?23:00
fungisince zuul was requesting unqualified jobs and nodepool was just picking the first label it saw registered for those jobs to determine demand, which ~half of the time was precise23:00
*** markvoelker has joined #openstack-infra23:00
fungioh, yes also that, gerrit was apparently (as best as we can tel after the fact) blocked fulfilling http requests23:00
fungiand that slowed zuul's reporting to a crawl, so there were lots of completed jobs it wasn't clearing from the pipeline23:01
jheskethah right23:01
jheskethnice find23:01
jheskethfungi: has the zuul restart been done yet?23:01
fungias soon as gerrit was restarted, zuul quickly burned down the results backlog23:02
fungijhesketh: yeah, i restarted zuul shortly after we determined where the precise node surplus was originating23:02
fungiand then i deleted the remnants to make room for more productive node types23:02
jheskeththe restart was smooth?23:02
jheskethis this the first restart since connections?23:02
fungiyeah, also even rare node types reregistered in gearman very quickly after the zuul restart23:03
*** bharathm_ has joined #openstack-infra23:03
fungibasically as soon as i checked they were already there23:03
fungiso much faster than i was used to previously... not sure what we've changed that might alter that23:03
*** bharathm has quit IRC23:04
jheskethpossibly just a smaller gearman?23:04
jheskethI don't think gearman cleans things up much, so a restart might have been appreciated by it23:05
*** markvoelker has quit IRC23:05
fungianyway, thanks for keeping tabs on that stuff last night. much less impact that i feared23:05
jheskethany time :-)23:05
jheskethsorry I failed to find the gerrit performance23:06
fungireally the biggest scramble of the morning was whatever mysterious malady gerrit came down with, which should be entirely unrelated to that change23:06
fungihopefully next time it happens we can catch it for a little longer, check database queues, process list within the jvm, et cetera23:07
fungialso puppetboard ran out of disk, so that wasn't helping things (made us slightly more blind to what might be going on network-wide)23:07
clarkbit wasn't the normal garbage collection spins cpu leak fun?23:07
*** apuimedo_ has quit IRC23:08
fungiclarkb: i don't know. i was focused on zuul logs at the time thinking we were looking at two unrelated issues, so mordred was checking on gerrit23:08
fungii briefly saw apache stats from it, which indicated it had a lot of sockets active writing23:09
funginot sure what else he looked at23:09
*** apuimedo_ has joined #openstack-infra23:10
*** bradjones has quit IRC23:10
*** larainema has joined #openstack-infra23:11
*** bradjones has joined #openstack-infra23:13
*** bradjones has quit IRC23:13
*** bradjones has joined #openstack-infra23:13
*** apuimedo_ has quit IRC23:16
*** apuimedo_ has joined #openstack-infra23:18
*** dims has joined #openstack-infra23:18
*** dims_ has quit IRC23:19
*** mudassirlatif has joined #openstack-infra23:21
*** rbrndt has quit IRC23:21
*** apoorvad has joined #openstack-infra23:21
*** apuimedo_ has quit IRC23:22
*** apuimedo_ has joined #openstack-infra23:23
*** harlowja has quit IRC23:25
*** harlowja has joined #openstack-infra23:25
*** zeih has quit IRC23:25
*** apuimedo_ has quit IRC23:28
*** apuimedo_ has joined #openstack-infra23:29
*** triggerwarning has quit IRC23:30
*** triggerwarning has joined #openstack-infra23:30
openstackgerritJames E. Blair proposed openstack-infra/zuul: Remove boilerplate zuul.conf from test fixtures  https://review.openstack.org/26074923:32
*** FallenPegasus has joined #openstack-infra23:33
*** apuimedo_ has quit IRC23:33
*** apuimedo_ has joined #openstack-infra23:33
*** rbrndt_ has joined #openstack-infra23:34
*** sputnik1_ has joined #openstack-infra23:35
*** ayoung has joined #openstack-infra23:35
*** apuimedo_ has quit IRC23:38
*** apuimedo_ has joined #openstack-infra23:40
*** apuimedo_ has quit IRC23:44
*** apuimedo_ has joined #openstack-infra23:46
*** salv-orlando has joined #openstack-infra23:47
*** jyuso1 has joined #openstack-infra23:48
*** ChuckC has joined #openstack-infra23:49
*** apuimedo_ has quit IRC23:50
*** apuimedo_ has joined #openstack-infra23:51
*** jyuso1 has quit IRC23:54
*** Verilium has joined #openstack-infra23:54
*** apuimedo_ has quit IRC23:56
*** apuimedo_ has joined #openstack-infra23:56
*** bharathm_ has quit IRC23:58
*** hdd has quit IRC23:58
*** dahoo has joined #openstack-infra23:58
*** rlandy has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!