Friday, 2016-08-05

*** rossella_s has quit IRC00:00
*** rossella_s has joined #openstack-infra00:00
*** amitgandhinz has joined #openstack-infra00:02
*** mriedem has joined #openstack-infra00:02
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Include subnodes in updateStats function v2  https://review.openstack.org/35143500:04
*** SumitNaiksatam has quit IRC00:04
pabelangerjeblair: ^ should be faster00:04
*** sdake_ has joined #openstack-infra00:06
*** thorst has quit IRC00:06
*** zhurong has quit IRC00:07
*** amitgandhinz has quit IRC00:07
*** sdake has quit IRC00:09
*** _ari_ has joined #openstack-infra00:12
*** markvoelker has joined #openstack-infra00:12
jeblairthe only ready nodes sitting around now (~10-20 mins) are for lesser-used image types.00:13
jeblairi spot checked a couple and they really are idle00:13
fungii just cleaned up 3 more used nodes that crept past the 3 hour mark00:14
jeblairfungi: what do you think about doing the 2-3 hour ones?00:14
*** markvoelker_ has quit IRC00:14
fungithere's >100 of those00:14
*** _ari_ has quit IRC00:14
jeblairwhich if we assume we should be running 450 jobs, is about our deficit (gearman says 332 are running)00:15
fungii guess any really long-running jobs are not that great in number and aren't a huge loss if they get rescheduled00:15
fungiyeah, i buy that math. i'll clean those out now too00:15
jeblairmy feeling is that jobs > 1 hour are 'not officially supported'  :)00:15
fungii can get behind that00:15
*** sputnik13 has quit IRC00:17
ianwhey, is it appropriate to copy stuff to static.openstack.org manually?  particularly a kernel for fedora 24 which we're going to need until a fix gets in and released?  my plan is to add it as a fixup in infra-package-needs until then00:18
fungifeels a little weird, i don't think we've done that in the past00:18
*** pahuang has quit IRC00:18
*** kzaitsev_mb has quit IRC00:19
ianwok, np ... we have copr for fedora to build packages00:19
fungion the other hand, there's all sorts of relatively irreplaceable artifacts on tarballs.o.o00:19
ianwi'm just having issues getting it signing the packages, etc ... just unrelated stuff00:19
fungii'm not particularly opposed if others aren't00:19
fungicertainly wouldn't want that checked into a git repo anyway00:20
ianwcool, well i need to double-check that dib doesn't make this too hard by screwing up the grub config and install the wrong kernel .. but that's the plan00:20
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-log_processor: Make the Gemfile zuul-aware  https://review.openstack.org/35143700:21
fungibasically, i don't have a better suggestion on where to put that at the moment, so unless someone else does...00:21
ianwcopr is made for this ... i've started again with https://copr.fedorainfracloud.org/coprs/iwienand/kernel-4.6.4-301.rhbz1361414/ ... last time it took like 7 hours and still failed00:22
*** pahuang has joined #openstack-infra00:22
jeblairi don't like any part of the ci system requiring manual intervention on the part of roots.  i'm not going to object to anyone else doing it, but i would not do it if asked.00:22
*** cody-somerville has joined #openstack-infra00:22
mgagnepabelanger: so I went with a solution similar to this one for my earlier problem with voluptuous: http://paste.openstack.org/show/549324/00:24
jeblairis f24 gating anything?  can we switch jobs to non-voting until it's fixed?00:24
*** tqtran has quit IRC00:25
ianwjeblair: no it's not gating any, just doesn't work at all because it oops.  i've root caused it & fixed it (http://marc.info/?l=linux-netdev&m=147020530109885&w=2) but it's just going to take a while to make it through the system00:25
*** karthikp_ has quit IRC00:26
*** pvaneck_ has quit IRC00:26
jeblairgotcha.00:28
*** mkoderer__ has quit IRC00:29
jeblairfungi: i think some more nodes crept past the 2 hour mark00:29
jeblairwe're at 429 now, so we probably only have about 30 more leaked00:30
jeblairer, 20 more :)00:31
fungilooks like 3400:31
fungideleting00:31
*** chlong has quit IRC00:32
*** r-mibu has quit IRC00:35
*** mkoderer__ has joined #openstack-infra00:37
*** mtanino has quit IRC00:37
jeblairi'm happy with all the squiggly lines now00:39
*** amotoki has joined #openstack-infra00:39
*** r-mibu has joined #openstack-infra00:40
*** rhallisey has joined #openstack-infra00:43
*** gouthamr_ has joined #openstack-infra00:44
*** xnox has quit IRC00:48
*** xarses has joined #openstack-infra00:48
*** dimtruck is now known as zz_dimtruck00:48
rbergeronjeblair: squiggly lines make all the things better.00:50
*** hichihara has quit IRC00:51
*** hemanthm|afk is now known as hemanthm00:51
*** hichihara has joined #openstack-infra00:51
*** elo has joined #openstack-infra00:54
*** jamielennox is now known as jamielennox|away00:55
pabelangerthat's the first time I can remember have to deal with leaked nodes since switching from jenkins00:56
pabelangerso that is good00:56
*** jamielennox|away is now known as jamielennox00:57
*** markvoelker has quit IRC00:59
*** amitgandhinz has joined #openstack-infra01:03
*** thorst has joined #openstack-infra01:04
*** xnox has joined #openstack-infra01:05
*** gildub has joined #openstack-infra01:05
*** shashank_hegde has quit IRC01:06
*** amitgandhinz has quit IRC01:07
*** ayoung has joined #openstack-infra01:08
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Add dvr/linuxbridge neutron jobs to devstack experimental queue  https://review.openstack.org/35145001:08
*** Sukhdev has quit IRC01:08
*** xnox has quit IRC01:10
*** thorst has quit IRC01:11
*** jcoufal has joined #openstack-infra01:13
*** xnox has joined #openstack-infra01:14
*** Apoorva has quit IRC01:14
*** thorst has joined #openstack-infra01:15
*** zhurong has joined #openstack-infra01:16
*** piet_ has joined #openstack-infra01:16
*** thorst has quit IRC01:16
*** Apoorva has joined #openstack-infra01:17
*** amotoki has quit IRC01:18
*** xnox has quit IRC01:19
*** zhurong has quit IRC01:21
*** zhurong has joined #openstack-infra01:22
*** ddieterly has joined #openstack-infra01:25
*** yanyanhu has joined #openstack-infra01:28
*** piet_ has quit IRC01:30
*** dingyichen has quit IRC01:32
openstackgerritTang Chen proposed openstack-infra/project-config: Fix Ironic functional job by properly setting PROJECTS  https://review.openstack.org/35113801:33
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699501:35
*** cody-somerville has quit IRC01:39
*** esberglu has quit IRC01:43
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Cloud Foundry plugin  https://review.openstack.org/35094001:46
openstackgerritMerged openstack-infra/project-config: Add channel #openstack-dib for diskimage-builder  https://review.openstack.org/35099701:47
prometheanfirewoo01:56
*** mriedem has quit IRC01:58
*** Apoorva_ has joined #openstack-infra01:58
*** Apoorva has quit IRC02:01
*** Apoorva_ has quit IRC02:02
*** sdake has joined #openstack-infra02:03
*** tphummel has quit IRC02:03
*** amitgandhinz has joined #openstack-infra02:04
*** sdake_ has quit IRC02:04
*** rhallisey has quit IRC02:05
*** yamahata has quit IRC02:07
*** amitgandhinz has quit IRC02:08
*** jamielennox is now known as jamielennox|away02:12
*** amotoki has joined #openstack-infra02:13
*** sdake_ has joined #openstack-infra02:14
*** esberglu has joined #openstack-infra02:14
*** sflanigan is now known as sflanigan_afk02:17
*** thorst has joined #openstack-infra02:17
*** sdake has quit IRC02:17
*** cody-somerville has joined #openstack-infra02:20
*** cody-somerville has joined #openstack-infra02:20
*** thorst has quit IRC02:25
*** ddieterly has quit IRC02:26
*** julim has joined #openstack-infra02:28
*** gouthamr_ has quit IRC02:29
*** daemontool_ has quit IRC02:31
*** huanxie has joined #openstack-infra02:32
*** sflanigan_afk is now known as sflanigan02:32
*** esberglu has quit IRC02:35
*** zhurong_ has joined #openstack-infra02:37
*** zhurong has quit IRC02:39
*** itisha has quit IRC02:40
*** thorst has joined #openstack-infra02:43
*** thorst has quit IRC02:43
*** zul has joined #openstack-infra02:44
openstackgerritqinchunhua proposed openstack-infra/project-config: Make py35 voting  https://review.openstack.org/35146402:44
*** dchen has joined #openstack-infra02:44
openstackgerritPaul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node}  https://review.openstack.org/35121902:45
*** pt_15 has quit IRC02:46
openstackgerritqinchunhua proposed openstack-infra/project-config: Make py35 voting for networking-midonet  https://review.openstack.org/35146402:46
*** ddieterly has joined #openstack-infra02:47
*** amotoki has quit IRC02:48
*** yuanying has quit IRC02:49
*** yamahata has joined #openstack-infra02:49
*** xnox has joined #openstack-infra02:54
*** esberglu has joined #openstack-infra02:58
*** esberglu has quit IRC02:59
*** piet_ has joined #openstack-infra02:59
*** xnox has quit IRC02:59
*** amitgandhinz has joined #openstack-infra03:04
*** amitgandhinz has quit IRC03:09
openstackgerritqinchunhua proposed openstack-infra/project-config: Make py35 voting for networking-midonet  https://review.openstack.org/35146403:12
*** amotoki has joined #openstack-infra03:15
openstackgerritPaul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node}  https://review.openstack.org/35121903:16
*** ddieterly has quit IRC03:17
*** yamahata has quit IRC03:18
*** dchen has quit IRC03:19
*** liusheng has joined #openstack-infra03:21
*** xnox has joined #openstack-infra03:26
*** ayoung is now known as ayoung_ZZZzzz03:26
*** xnox has quit IRC03:29
*** julim has quit IRC03:31
*** amotoki has quit IRC03:31
*** Srinu has joined #openstack-infra03:34
*** jamielennox|away is now known as jamielennox03:35
Srinuhi03:36
*** xnox has joined #openstack-infra03:37
Srinuplease help me. this issue is not sloving for me long ago........http://paste.openstack.org/show/549835/03:40
*** sdake_ has quit IRC03:42
SrinuIn nodepool configuration how can i use same provider for 2 slaves but different flavor on the provider03:43
*** thorst has joined #openstack-infra03:44
*** yuanying has joined #openstack-infra03:48
*** chlong has joined #openstack-infra03:49
*** karthikp_ has joined #openstack-infra03:52
*** thorst has quit IRC03:52
*** piet_ has quit IRC03:52
*** esberglu has joined #openstack-infra03:55
*** sdake has joined #openstack-infra03:57
*** esberglu has quit IRC04:02
*** jcoufal has quit IRC04:02
*** karthik__ has joined #openstack-infra04:03
*** karthikp_ has quit IRC04:05
*** amitgandhinz has joined #openstack-infra04:05
*** rlandy has quit IRC04:08
*** amitgandhinz has quit IRC04:10
*** armax has quit IRC04:10
*** sdake has quit IRC04:14
*** amotoki has joined #openstack-infra04:16
*** cody-somerville has quit IRC04:22
*** javeriak has joined #openstack-infra04:22
*** dingyichen has joined #openstack-infra04:24
*** sdake has joined #openstack-infra04:24
*** dingyichen has quit IRC04:27
*** dingyichen has joined #openstack-infra04:32
*** Sukhdev has joined #openstack-infra04:34
*** markvoelker has joined #openstack-infra04:38
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Cloud Foundry plugin  https://review.openstack.org/35094004:40
*** Sukhdev has quit IRC04:43
*** jamielennox is now known as jamielennox|away04:43
*** Sukhdev has joined #openstack-infra04:43
*** markvoelker has quit IRC04:43
*** jaosorior has joined #openstack-infra04:45
*** jaosorior has quit IRC04:51
*** thorst has joined #openstack-infra04:51
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Cloud Foundry plugin  https://review.openstack.org/35094004:51
*** SumitNaiksatam has joined #openstack-infra04:52
*** roxanaghe has joined #openstack-infra04:52
*** SumitNaiksatam has quit IRC04:52
*** jaosorior has joined #openstack-infra04:52
*** psachin has joined #openstack-infra04:54
*** javeriak has quit IRC04:54
*** jaosorior has quit IRC04:54
*** yottatsa has joined #openstack-infra04:55
*** jaosorior has joined #openstack-infra04:55
*** thorst has quit IRC04:58
*** rbuzatu has joined #openstack-infra04:58
*** rbuzatu has quit IRC05:01
*** sarob has joined #openstack-infra05:02
*** pahuang has quit IRC05:03
*** amitgandhinz has joined #openstack-infra05:06
*** yottatsa has quit IRC05:07
*** yottatsa has joined #openstack-infra05:09
*** amitgandhinz has quit IRC05:10
*** Sukhdev has quit IRC05:12
*** jaosorior has quit IRC05:14
*** M-docaedo_vector has quit IRC05:14
*** _nadya_ has joined #openstack-infra05:14
*** jaosorior has joined #openstack-infra05:15
*** yottatsa has quit IRC05:15
*** narayrak has joined #openstack-infra05:15
*** pahuang has joined #openstack-infra05:15
*** jaosorior has quit IRC05:17
*** jaosorior has joined #openstack-infra05:18
*** roxanaghe has quit IRC05:25
*** sarob has quit IRC05:27
*** sarob has joined #openstack-infra05:27
*** rcernin has joined #openstack-infra05:31
*** e0ne has joined #openstack-infra05:31
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node}  https://review.openstack.org/35121905:33
*** sarob has quit IRC05:35
*** kien-ha has quit IRC05:36
*** links has joined #openstack-infra05:36
*** _nadya_ has quit IRC05:37
*** _nadya_ has joined #openstack-infra05:40
*** M-docaedo_vector has joined #openstack-infra05:42
*** nmagnezi has joined #openstack-infra05:44
*** _nadya_ has quit IRC05:46
openstackgerritCraige McWhirter proposed openstack-infra/puppet-phabricator: Patches Required to Deliver Pholio  https://review.openstack.org/34248105:49
*** rbuzatu has joined #openstack-infra05:54
*** rbuzatu has quit IRC05:55
*** rbuzatu has joined #openstack-infra05:56
*** thorst has joined #openstack-infra05:56
*** pgadiya has joined #openstack-infra05:57
*** ekarlso has quit IRC05:58
*** chem has joined #openstack-infra06:00
*** thorst has quit IRC06:03
*** Ravikiran_K has joined #openstack-infra06:06
*** amitgandhinz has joined #openstack-infra06:07
openstackgerritTimothy R. Chavez proposed openstack-infra/jenkins-job-builder: Add support for the random string parameter  https://review.openstack.org/35138406:07
*** chlong has quit IRC06:08
*** amitgandhinz has quit IRC06:11
*** armax has joined #openstack-infra06:12
*** coolsvap has joined #openstack-infra06:14
*** camunoz has quit IRC06:14
*** kaisers has quit IRC06:15
*** kaisers has joined #openstack-infra06:16
*** armax has quit IRC06:17
*** ianychoi has quit IRC06:18
*** jimbaker has quit IRC06:19
*** jianghuaw has quit IRC06:20
*** sandanar has joined #openstack-infra06:20
*** jimbaker has joined #openstack-infra06:23
*** jimbaker has quit IRC06:23
*** jimbaker has joined #openstack-infra06:23
*** javeriak has joined #openstack-infra06:29
*** Hal has joined #openstack-infra06:32
*** rbuzatu_ has joined #openstack-infra06:32
odyssey4medo we have any idea when the new bindep will be available on nodepool images?06:32
*** rbuzatu_ has quit IRC06:33
*** rbuzatu_ has joined #openstack-infra06:35
*** rbuzatu has quit IRC06:36
*** markvoelker has joined #openstack-infra06:40
*** chlong has joined #openstack-infra06:40
*** jtomasek has joined #openstack-infra06:44
*** aeng has joined #openstack-infra06:44
*** markvoelker has quit IRC06:44
*** cody-somerville has joined #openstack-infra06:49
*** e0ne has quit IRC06:51
*** chlong has quit IRC06:52
*** ianychoi has joined #openstack-infra06:56
*** yamamot__ has joined #openstack-infra07:00
*** thorst has joined #openstack-infra07:01
*** Na3iL has joined #openstack-infra07:03
*** gildub has quit IRC07:07
*** amitgandhinz has joined #openstack-infra07:07
*** thorst has quit IRC07:08
*** cody-somerville has quit IRC07:08
*** mikelk has joined #openstack-infra07:09
*** amitgandhinz has quit IRC07:12
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute014.vanilla metadata for better tracking  https://review.openstack.org/35152707:12
*** cody-somerville has joined #openstack-infra07:13
yolandagood morning07:13
AJaegergood morning, yolanda07:14
AJaegerfungi, set topic to constraints - see https://review.openstack.org/#/q/topic:constraints+status:open07:14
* rcarrillocruz waves07:14
AJaegermorning rcarrillocruz07:15
*** jimbaker has quit IRC07:16
*** pahuang has quit IRC07:16
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute012.vanilla metadata for better tracking  https://review.openstack.org/35153107:18
*** jimbaker has joined #openstack-infra07:19
*** jimbaker has quit IRC07:20
*** jimbaker has joined #openstack-infra07:20
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute011.vanilla metadata for better tracking  https://review.openstack.org/35153307:21
*** cody-somerville has quit IRC07:23
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP for compute024.vanilla  https://review.openstack.org/35153507:25
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP for compute030.vanilla  https://review.openstack.org/35153907:29
sdakeAJaeger about?07:29
AJaegersdake: yes07:29
sdakesay do you know anything about the afs mirroring?07:29
openstackgerritMerged openstack-infra/project-config: Add release-notes-jobs to osc-lib  https://review.openstack.org/35134207:30
sdakehttps://review.openstack.org/34927807:30
openstackgerritMerged openstack-infra/project-config: Add dvr/linuxbridge neutron jobs to devstack experimental queue  https://review.openstack.org/35145007:30
sdakenot sure if thats correct or not ^^07:30
AJaegersdake: not really. pabelanger, jeblair, mordred are the experts07:30
openstackgerritMerged openstack-infra/project-config: Add gate-tripleo-quickstart-base-centos-7-nv job  https://review.openstack.org/35030107:30
AJaegerBut let me check your review07:30
sdakei need to do about 3 or 4 like em for ubuntu07:30
sdakewanted to get cracking07:31
openstackgerritMerged openstack-infra/project-config: Fix Ironic functional job by properly setting PROJECTS  https://review.openstack.org/35113807:31
AJaegersdake: I cannot really comment on that, best way for the rest of the team to wake up and ask07:32
openstackgerritqinchunhua proposed openstack-infra/project-config: Make py35 voting for networking-midonet  https://review.openstack.org/35146407:32
sdakei did ask pabelanger last week to take a look but eveyrone is super overloaded atm07:32
sdakei'll try again in the morning07:32
sdakethanks :)07:32
AJaegersdake: get some rest ;)07:32
sdake48 hours or bust!07:33
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute023.vanilla metadata for better tracking  https://review.openstack.org/35154107:35
*** jpavlik has joined #openstack-infra07:35
jpavlikHello all, please can anybody take a look at following review https://review.openstack.org/#/c/343663/07:35
jpavlikit hangs on "needs verify" for more than week without any action07:36
*** flaper87 has joined #openstack-infra07:36
*** flaper87 is now known as Guest7354807:36
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP and rack for compute009.vanilla  https://review.openstack.org/35154207:38
*** jpich has joined #openstack-infra07:39
AJaegerjpavlik: remove the +Workflow and add it again. It might have been approved during an outage - and now we react only to state changes. A recheck will not give a state cahnge07:39
AJaegerjpavlik: I see the problem: You first did a +W, and afterwards a +2.07:40
jpavlikOK, so remove and add +1 on workflow?07:40
AJaegerjpavlik: that's the problem! Your +W is only valid if there's a prior +2 already. Since we only look at state change, the later +2 does is not seen.07:40
AJaegerjpavlik: yes, just toggle it.07:40
AJaegerjpavlik: and it already starts in the gate as expected ;)07:41
jpavlikOK, thanks a lot.07:42
jpavlikso it is same problem here https://review.openstack.org/#/c/346810/07:42
jpavlikbut this must be done by Thierry07:42
AJaegerjpavlik: that change has "Depends-On: I662a011f0fcf2f5694da058df7e3c08a6493e3b6"07:42
*** vincentll has joined #openstack-infra07:42
AJaegerAnd the dependent change has not merged. And indeed, once that dependent change has merged, ttx needs to toggle the +W.07:43
openstackgerritMerged openstack-infra/project-config: Ironic: Make ironic-multitenant-network a voting job  https://review.openstack.org/35070307:43
AJaegeryolanda: care to review https://review.openstack.org/#/c/346674/ as well, please?07:43
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute015.vanilla metadata for better tracking  https://review.openstack.org/35154307:44
jpavlikOK, thanks Andreas07:44
ttxLet me see07:44
yolandasure07:44
ttxyeah, dependent change must merge first07:45
*** karthik__ has quit IRC07:45
openstackgerritYanyan Hu proposed openstack-infra/project-config: Fix senlin integration test job  https://review.openstack.org/35154407:46
AJaegerttx, should be merged in half an hour ;). Thanks, yolanda07:47
jpavlikThanks guys!07:47
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP and rack for baremetal01.vanilla  https://review.openstack.org/35154507:49
*** markmcd has quit IRC07:51
*** aeng has quit IRC07:51
AJaegeryolanda: do you have time for these older ones as well, please? https://review.openstack.org/350616 and https://review.openstack.org/350113 https://review.openstack.org/#/c/346953/ https://review.openstack.org/#/c/317672/07:51
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute033.vanilla metadata for better tracking  https://review.openstack.org/35154707:52
yolandaAJaeger, sure07:52
*** markmcd has joined #openstack-infra07:53
AJaegerthanks!07:55
*** shardy has joined #openstack-infra07:55
*** dtantsur|afk is now known as dtantsur07:56
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update compute032.vanilla metadata for better tracking  https://review.openstack.org/35155107:57
openstackgerritMerged openstack-infra/project-config: Adding OpenStack-Salt formula kubernetes  https://review.openstack.org/34667407:59
*** zzzeek has quit IRC08:00
*** rossella_s has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** rossella_s has joined #openstack-infra08:00
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute025.vanilla metadata for better tracking  https://review.openstack.org/35155308:02
*** migi_ is now known as migi08:02
openstackgerritMerged openstack-infra/project-config: provide a manual override mode  https://review.openstack.org/35061608:03
*** openstackgerrit has quit IRC08:03
*** openstackgerrit has joined #openstack-infra08:03
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/35155508:04
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute036.vanilla metadata for better tracking  https://review.openstack.org/35155808:08
*** thorst has joined #openstack-infra08:08
openstackgerritMerged openstack-infra/project-config: Add nv test job for Django 1.10 in Django OS Auth  https://review.openstack.org/35011308:09
openstackgerritMerged openstack-infra/project-config: grafana: split neutron integrated gate dashboard  https://review.openstack.org/34695308:10
openstackgerritMerged openstack-infra/project-config: Add python jobs to pyeclib project  https://review.openstack.org/31767208:10
*** mhickey has joined #openstack-infra08:10
AJaegerrcarrillocruz: could you review https://review.openstack.org/#/c/350397/ today, please? It adds python jobs to a a docs tool repo...08:10
rcarrillocruz+2+a08:11
*** jaosorior has quit IRC08:13
*** thorst has quit IRC08:13
AJaegerthanks!08:13
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node}  https://review.openstack.org/35121908:13
*** shardy has quit IRC08:14
*** jaosorior has joined #openstack-infra08:14
*** armax has joined #openstack-infra08:14
*** shardy has joined #openstack-infra08:14
*** yaume has joined #openstack-infra08:14
*** nmagnezi has quit IRC08:15
*** tesseract- has joined #openstack-infra08:16
*** Goneri has joined #openstack-infra08:16
*** armax has quit IRC08:18
*** jpavlik has quit IRC08:21
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add controller00.vanilla metadata for better tracking  https://review.openstack.org/35156308:21
*** asettle has joined #openstack-infra08:21
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP and rack for compute035.vanilla  https://review.openstack.org/35156708:24
*** sflanigan has quit IRC08:26
openstackgerritMerged openstack-infra/project-config: Add py27, py35 jobs for openstack-doc-tools  https://review.openstack.org/35039708:26
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP and rack of compute031.vanilla  https://review.openstack.org/35157008:26
*** ggnel_t has joined #openstack-infra08:27
*** lucas|dinner is now known as lucasagomes08:28
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/35155508:28
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP for baremetal00  https://review.openstack.org/35129108:29
*** chem has quit IRC08:30
*** chem has joined #openstack-infra08:30
*** d0ugal has quit IRC08:32
openstackgerritMerged openstack-dev/pbr: Expose deb version to match exposing rpm version  https://review.openstack.org/35086108:32
*** chem has quit IRC08:32
*** d0ugal has joined #openstack-infra08:33
*** d0ugal has quit IRC08:33
*** d0ugal has joined #openstack-infra08:33
*** d0ugal has quit IRC08:33
*** d0ugal has joined #openstack-infra08:33
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP and rack for compute007.vanilla  https://review.openstack.org/35157408:35
*** ams__ has joined #openstack-infra08:35
vsaienkoinfra team, I'm wondered if it is allowed to extend sudo rights for temepst user at the gates job?08:35
*** sdake has quit IRC08:37
*** Guest73548 is now known as flaper8708:39
*** hichihara has quit IRC08:39
*** flaper87 is now known as Guest7864108:39
*** markvoelker has joined #openstack-infra08:40
*** Guest78641 has quit IRC08:44
*** markvoelker has quit IRC08:45
openstackgerritWitold Bedyk proposed openstack-infra/project-config: Add dstat to gate-tempest-dsvm-monasca-log job  https://review.openstack.org/35158008:50
*** acoles_ is now known as acoles08:52
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use zuul-cloner for {name}-branch-tarball  https://review.openstack.org/35158508:56
*** narayrak has quit IRC08:59
*** narayrak has joined #openstack-infra09:00
*** itisha has joined #openstack-infra09:01
*** pt_15 has joined #openstack-infra09:02
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use zuul-cloner for grenade tests  https://review.openstack.org/35158609:02
*** chem has joined #openstack-infra09:02
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use zuul-cloner for gate-horizon-selenium-headless  https://review.openstack.org/35158709:04
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update controller00.chocolate metadata for better tracking  https://review.openstack.org/35158909:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use zuul-cloner for {pipeline}-kolla-kubernetes-{action}-{distro}-{type}  https://review.openstack.org/35159009:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use zuul-cloner for kolla  https://review.openstack.org/35159009:08
*** amitgandhinz has joined #openstack-infra09:09
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute002.chocolate metadata for better tracking  https://review.openstack.org/35159109:10
*** thorst has joined #openstack-infra09:10
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use zuul-cloner for bashate job  https://review.openstack.org/35159709:13
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Position archived items at the bottom of worklists  https://review.openstack.org/35126509:13
*** amitgandhinz has quit IRC09:14
*** e0ne has joined #openstack-infra09:14
yolandahi AJaeger, i have a problem with one project, tripleo-validations09:14
yolandawe need to consume jsonschema09:14
yolandain global-requirements is set to global-requirements.txt:jsonschema>=2.0.0,<3.0.0,!=2.5.0  # MIT09:15
yolandabut upper requirements sets to 2.5.009:15
*** Goneri has quit IRC09:15
*** sambetts|afk is now known as sambetts09:15
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use zuul-cloner for docs publishing  https://review.openstack.org/35159909:16
AJaegeryolanda: let me check...09:17
AJaegeryolanda: saw your comment that you solved it yourself09:17
*** thorst has quit IRC09:18
*** Goneri has joined #openstack-infra09:18
yolandayep, my fault09:18
AJaegerno worries09:18
AJaegerfungi, converted jobs that use tox and might use constraints (so, no infra jobs) to use proper zuul-cloner.09:22
*** rbuzatu_ has quit IRC09:28
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute094.chocolate metadata for better tracking  https://review.openstack.org/35160609:28
*** javeriak has quit IRC09:29
*** claudiub has joined #openstack-infra09:30
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute001.chocolate metadata for better tracking  https://review.openstack.org/35160909:33
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute059.chocolate metadata for better tracking  https://review.openstack.org/35161009:35
*** flaper87 has joined #openstack-infra09:40
*** flaper87 has quit IRC09:40
*** flaper87 has joined #openstack-infra09:40
*** kzaitsev_mb has joined #openstack-infra09:43
*** amoralej is now known as amoralej|brb09:46
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute058.chocolate metadata for better tracking  https://review.openstack.org/35161409:46
*** senk has joined #openstack-infra09:47
senksomeone able to tell me why jenkins is not testing https://review.openstack.org/#/c/350222/ ?09:48
*** markmcd has quit IRC09:48
*** rbuzatu has joined #openstack-infra09:49
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Complex priorities UI in stories  https://review.openstack.org/31266609:50
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute053.chocolate metadata for better tracking  https://review.openstack.org/35161609:51
*** rbuzatu has quit IRC09:53
*** _nadya_ has joined #openstack-infra09:55
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute060.chocolate metadata for better tracking  https://review.openstack.org/35162209:58
*** _nadya_ has quit IRC09:58
AJaegersenk: interesting ;( Jenkins never tested it. Let me dig a bit...10:00
senkthanks AJaeger10:00
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute082.chocolate metadata for better tracking  https://review.openstack.org/35162310:00
*** jcoufal has joined #openstack-infra10:00
*** javeriak has joined #openstack-infra10:03
*** _nadya_ has joined #openstack-infra10:03
*** _nadya_ has quit IRC10:03
AJaegersenk: We need to change infra setup, missed one place. Problem is that for a job to be tested, we need at least one job. And the jobs are setup right now that not a single job runs for your change.10:03
senkAJaeger: ah okay.. but there are existing jobs for the openstack/kolla project in general?10:04
*** amoralej|brb is now known as amoralej10:06
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix kolla docs skip  https://review.openstack.org/35162610:06
AJaegersenk: ^10:06
AJaegersenk: we renamed the job from gate-kolla-docs to gate-kolla-docs-ubuntu-xenial (and ...-trusty) and the skip-if was forgotten.10:06
AJaegeryolanda, rcarrillocruz: Could you review 351626, please?10:06
rcarrillocruzlgtm10:08
AJaegersenk: once that change is in and live (will be live for sure 1h after merge), add a "recheck" comment to your change and then ask a core to either add another +W or toggle it - remove it and add it back again. Then your change hsould merge10:08
AJaegersenk: but please do it all in order ;)10:08
senkAJaeger: of course :) thanks for your help!10:09
*** zhurong_ has quit IRC10:10
*** amitgandhinz has joined #openstack-infra10:10
AJaegersenk: sorry for this, we missed this when we moved jobs from trusty to xenial10:10
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute005.chocolate metadata for better tracking  https://review.openstack.org/35162910:10
senkAJaeger: nevermind, was only a very small change.10:11
*** yanyanhu has quit IRC10:12
AJaegerconsistency counts - git.openstack.org is really our master, github is just a mirror...10:13
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute000.chocolate metadata for better tracking  https://review.openstack.org/35163110:14
*** amitgandhinz has quit IRC10:14
*** asselin has quit IRC10:15
yolandaAJaeger, approved10:15
AJaegerthanks, yolanda10:15
*** armax has joined #openstack-infra10:15
*** thorst has joined #openstack-infra10:16
*** armax has quit IRC10:21
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute039.chocolate metadata for better tracking  https://review.openstack.org/35163410:22
*** thorst has quit IRC10:23
senkAJaeger: there are more occurences in kolla especially in the codebase.. try to fix that later...10:23
*** yamamot__ has quit IRC10:24
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute007.chocolate metadata for better tracking  https://review.openstack.org/35163510:24
openstackgerritThierry Carrez proposed openstack-infra/release-tools: Initial version for ACL manager  https://review.openstack.org/35163710:28
*** Goneri has quit IRC10:29
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute047.chocolate metadata for better tracking  https://review.openstack.org/35163810:30
*** Goneri has joined #openstack-infra10:31
openstackgerritMerged openstack-infra/project-config: Fix kolla docs skip  https://review.openstack.org/35162610:32
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute008.chocolate metadata for better tracking  https://review.openstack.org/35164210:32
dtantsurfolks, how to forcibly reset "This change depends on a change that failed to merge." on a change?10:33
dtantsuris "recheck" enough?10:33
*** _nadya_ has joined #openstack-infra10:34
*** _nadya_ has quit IRC10:35
*** _nadya_ has joined #openstack-infra10:35
openstackgerritAndrey Pavlov proposed openstack-infra/project-config: ec2api experimental job: fix rally job definition  https://review.openstack.org/35164310:39
*** tosky has joined #openstack-infra10:43
jrolldtantsur: it is, that happens when a lower patch in the chain is updated while using depends-on10:43
dtantsursigh10:44
AJaegerdtantsur: yes, recheck should be fine.10:44
*** amotoki has quit IRC10:44
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute083.chocolate metadata for better tracking  https://review.openstack.org/35164510:46
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute018.chocolate metadata for better tracking  https://review.openstack.org/35164710:49
*** javeriak has quit IRC10:50
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute003.chocolate metadata for better tracking  https://review.openstack.org/35164910:51
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP and rack for compute087.chocolate  https://review.openstack.org/35165110:54
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute023.chocolate metadata for better tracking  https://review.openstack.org/35165310:57
*** Ravikiran_K has quit IRC10:58
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute021.chocolate metadata for better tracking  https://review.openstack.org/35165510:59
*** jcoufal has quit IRC11:01
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Use ssh-agent instead of ugly ssh-wrapper scripts  https://review.openstack.org/33114811:01
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Add ability to use SSH instead of HTTP to get refs  https://review.openstack.org/24643311:01
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute049.chocolate metadata for better tracking  https://review.openstack.org/35165811:04
*** rhallisey has joined #openstack-infra11:07
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add compute062.chocolate metadata for better tracking  https://review.openstack.org/35166111:08
*** amitgandhinz has joined #openstack-infra11:11
*** lucasagomes is now known as lucas-hungry11:11
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP and rack for compute015.chocolate  https://review.openstack.org/35166311:11
*** amitgandhinz has quit IRC11:15
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP and rack for compute017.chocolate  https://review.openstack.org/35166411:16
*** armax has joined #openstack-infra11:18
*** yamamoto has joined #openstack-infra11:19
*** thorst has joined #openstack-infra11:20
*** javeriak has joined #openstack-infra11:20
*** rbuzatu has joined #openstack-infra11:21
*** armax has quit IRC11:22
*** tpsilva has joined #openstack-infra11:24
*** ganesan has joined #openstack-infra11:25
*** florianf has joined #openstack-infra11:27
*** thorst has quit IRC11:28
ganesanI am getting following error when nodepool is launching node11:28
ganesanERROR nodepool.NodeLauncher: Timeout launching node id: 23480 in provider: local_02 error: Timeout waiting for ssh access11:28
*** karlos_ has joined #openstack-infra11:28
*** sdague has joined #openstack-infra11:30
*** jkilpatr has joined #openstack-infra11:31
*** zhurong has joined #openstack-infra11:31
karlos_Hello, can someone please help me with following question ? I am runnning one of 3rd party CI's and I am trying to configure additional jobs via jenkins job builder in yaml format. I would like to use and change some variables, like using small shell script. MY_IP_ADDRESS=`ip route get 8.8.8.8 | awk '{print $NF; exit}'`  echo "$MY_IP_ADDRESS  ubuntu" | sudo tee -a /etc/hosts. This is working fine when job is configured in j11:31
*** Srinu has quit IRC11:34
*** javeriak has quit IRC11:35
openstackgerritMonty Taylor proposed openstack/os-client-config: Add test for precedence rules  https://review.openstack.org/35166811:36
*** javeriak has joined #openstack-infra11:37
*** ldnunes has joined #openstack-infra11:39
*** gouthamr has joined #openstack-infra11:43
openstackgerritMonty Taylor proposed openstack-infra/shade: Migrate functional tests to betamax  https://review.openstack.org/34439711:44
openstackgerritMonty Taylor proposed openstack-infra/shade: Stop creating cloud objects in functional tests  https://review.openstack.org/34439611:44
openstackgerritMonty Taylor proposed openstack-infra/shade: Move list_magnum_services to OperatorCloud  https://review.openstack.org/34439511:44
mordredrcarrillocruz: keystoneauth made a release with the new yamljsonserializer - so we don't need to forklift it in anymore (thanks for approving the patch though :) )11:45
mordredI have now abandoned it11:45
*** yamamoto has quit IRC11:45
*** asettle has quit IRC11:45
*** rfolco_ has joined #openstack-infra11:46
*** yamamoto has joined #openstack-infra11:47
*** thorst has joined #openstack-infra11:48
*** yamamoto has quit IRC11:51
openstackgerritMonty Taylor proposed openstack-infra/shade: Lay the groundwork for multi-cloud in unit tests  https://review.openstack.org/34391611:52
openstackgerritMonty Taylor proposed openstack-infra/shade: Add new test with betamax for create flavors  https://review.openstack.org/30219211:52
*** ganesan has quit IRC11:53
*** gordc has joined #openstack-infra11:56
zigomordred: Hi there! Could you please review https://review.openstack.org/#/c/346095/ ?11:56
zigomordred: It's the last infra patch before we can start adding packages and do the actual work! :)11:56
mordredwoot!11:58
*** asettle has joined #openstack-infra11:58
*** psilvad has joined #openstack-infra11:58
mordredzigo: looks good in general - although I think we want echo 'deb http://mirror.dfw.rax.openstack.org/debian-openstack jessie-newton main' >/etc/a11:59
mordredpt/sources.list.d/jessie-newto11:59
openstackgerritThierry Carrez proposed openstack-infra/release-tools: Initial version for ACL manager  https://review.openstack.org/35163711:59
mordredto not be hardcoded to dfw ... one sec while I look for a thing you can use11:59
*** dingyichen has quit IRC11:59
*** weshay has joined #openstack-infra11:59
*** rossella_s has quit IRC12:00
zigomordred: I believe that's not even needed, considering that it should be there by default in the image.12:00
*** rossella_s has joined #openstack-infra12:00
mordredzigo: no, I think it is - I do not believe we'll have that enabled by default unless I'm missing something12:01
mordredzigo: but luckily - it's easy to grab the right one ...12:01
zigomordred: I trust you! :)12:01
zigomordred: I'm also interested to know how.12:01
AJaegermordred: see https://review.openstack.org/341856 - that file should be used12:01
mordredoh!12:02
mordredzigo: I stand corrected. thanks AJaeger12:02
AJaegerSo, just copy or link to it.12:02
AJaegerBut don't create one of your own.12:02
zigoJust remove the thing, IMO.12:02
AJaegerthat only merged on Tuesday, hope we have images with these in now12:02
*** Goneri has quit IRC12:03
mordredzigo: so: cp /etc/apt/sources.list.available.d/debian-openstack-newton.list /etc/apt/sources.list.d12:03
mordredzigo: (left it on the review)12:04
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add automatic backports  https://review.openstack.org/34609512:04
zigoOh...12:04
rcarrillocruzmordred: sweet12:05
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add automatic backports  https://review.openstack.org/34609512:05
zigomordred: There you go! :)12:05
mordredyes!12:05
mordredzigo: looks great to me - maybe AJaeger will give you the other +212:05
zigo:)12:06
* AJaeger checks12:06
*** rodrigods has quit IRC12:07
*** rodrigods has joined #openstack-infra12:07
AJaeger+2A12:08
AJaegerwait - where's the governance change for that?12:09
*** amoralej is now known as amoralej|lunch12:09
AJaegerzigo: Why did you remove the "Needed-By: Ie96af7d7d9717ef3fb634f1b676a05fd4196121a" line fro mthe commit message? Could you add it back again, please?12:10
AJaegerThat was in patchset 912:10
*** baoli has joined #openstack-infra12:11
*** amitgandhinz has joined #openstack-infra12:11
*** alaski is now known as lascii12:12
*** julim has joined #openstack-infra12:12
*** baoli_ has joined #openstack-infra12:12
AJaegerso, changed to -1 and wait for that one to be added back...12:12
openstackgerritJavier Peña proposed openstack-infra/tripleo-ci: Set --dev option for dlrn execution instead of env var  https://review.openstack.org/35168212:13
AJaegerzigo, mordred I'll just do it quickly myself...12:13
*** dprince has joined #openstack-infra12:13
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add automatic backports  https://review.openstack.org/34609512:13
AJaegermordred: now you can +2A ;)12:13
mordred\o/12:14
*** Goneri has joined #openstack-infra12:14
*** mkoderer_ has joined #openstack-infra12:15
*** amitgandhinz has quit IRC12:16
*** baoli has quit IRC12:16
*** cgross has quit IRC12:17
*** asettle has quit IRC12:17
*** lmiccini_ has joined #openstack-infra12:17
*** lmiccini has quit IRC12:17
*** pradk has joined #openstack-infra12:18
*** mkoderer has quit IRC12:18
*** cgross has joined #openstack-infra12:18
*** huanxie has quit IRC12:19
*** ggnel_t has quit IRC12:21
zigogr8 !!! :)12:21
openstackgerritMonty Taylor proposed openstack-infra/shade: Add plumbing for betamax functional tests.  https://review.openstack.org/34439712:23
*** lucas-hungry is now known as lucasagomes12:24
*** rbuzatu has quit IRC12:25
*** amotoki has joined #openstack-infra12:25
*** aviau has quit IRC12:25
*** aviau has joined #openstack-infra12:25
*** piet_ has joined #openstack-infra12:26
*** amotoki has quit IRC12:29
*** ramishra has quit IRC12:31
*** savihou has joined #openstack-infra12:32
senkAJaeger: FYI Jenkins works now fine on kolla docs12:32
*** ramishra has joined #openstack-infra12:32
AJaegersenk: glad to hear! Thanks12:34
jrollhey infra folks, we have some post jobs that publish to tarballs.o.o, and one doesn't seem to be working correctly12:34
jrollhere we haven't seen updates since june 14: http://tarballs.openstack.org/ironic-python-agent/tinyipa/12:34
jrollbut the job seems successful: http://logs.openstack.org/27/277d97ba2b27fdd193b8daf91b375241d4f2c385/post/ironic-python-agent-buildimage-tinyipa/1ce0376/console.html12:35
jrolland appears similar to the job that is publishing okay: http://logs.openstack.org/27/277d97ba2b27fdd193b8daf91b375241d4f2c385/post/ironic-python-agent-buildimage-coreos/2c32993/console.html12:35
AJaegerjroll: did you check the ansible files as well?12:35
jrollAJaeger: ah12:36
AJaegerjroll: http://logs.openstack.org/27/277d97ba2b27fdd193b8daf91b375241d4f2c385/post/ironic-python-agent-buildimage-tinyipa/1ce0376/_zuul_ansible/ansible_log.txt12:36
AJaegerlast line is the one to look at12:36
jrollyeah, I see it12:36
* jroll hrms, will come back if I need help12:36
jrollthanks AJaeger !12:36
AJaegerjroll: best talk with jeblair and pabelanger once you're back ;)12:37
jrollsure :)12:37
*** rlandy has joined #openstack-infra12:43
*** piet_ has quit IRC12:45
*** thorst_ has joined #openstack-infra12:45
*** amotoki has joined #openstack-infra12:47
*** Goneri has quit IRC12:48
*** thorst has quit IRC12:49
*** xyang1 has joined #openstack-infra12:49
*** jcoufal has joined #openstack-infra12:52
*** amotoki has quit IRC12:52
*** ccamacho is now known as ccamacho|lunch12:53
*** jaosorior has quit IRC12:54
openstackgerritEmilien Macchi proposed openstack-infra/project-config: zuul/puppet: enable integration jobs on puppet-barbican  https://review.openstack.org/35173212:55
*** psilvad has quit IRC12:55
*** markvoelker has joined #openstack-infra12:56
*** matt-borland has joined #openstack-infra12:57
*** rbuzatu has joined #openstack-infra12:57
*** psilvad has joined #openstack-infra12:58
pabelangerAJaeger: mordred: zigo: 346095 is going to fail to cp source.list, you need to use sudo12:58
zigoOh !12:58
mordredblerg12:58
zigobad me ...12:58
mordredpabelanger: can't you be awake all the time?12:58
zigo;P12:58
pabelangerzigo: -2 it from merging so you can update it12:59
pabelangermordred: if only12:59
pabelangeralso, morning13:00
zigopabelanger: Is there a /etc/apt/sources.list.available.d/debian-openstack-newton-backports.list file too?13:00
zigoI guess, not yet, since the repo is currently empty, right?13:00
openstackgerritDarragh Bailey proposed openstack-infra/git-review: Limit reviews by branch when specified  https://review.openstack.org/31205613:00
*** asselin has joined #openstack-infra13:00
*** tldr_ has joined #openstack-infra13:01
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add automatic backports  https://review.openstack.org/34609513:01
zigomordred: pabelanger: Should be fine now.13:01
pabelangerzigo: no, we'll likely add jessie-backports into the exist debian-openstack-newton.list, if you are fine with that13:01
zigoFine for me.13:01
AJaegerrcarrillocruz, mordred : Could you +2A pabelangers change for moving tox t oxenial, please? https://review.openstack.org/35121913:01
*** psilvad has quit IRC13:02
AJaegerpabelanger: I fixed the merge conflict on it and the one problem it still had ^. LGTM now - let's merge!13:02
*** Goneri has joined #openstack-infra13:03
mordredAJaeger: lgtm13:03
*** ddieterly has joined #openstack-infra13:04
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add $LSBDISTCODENAME-newton-backports for debian-openstack mirror list  https://review.openstack.org/35173413:04
pabelangerzigo: ^ will add jessie-backports13:04
AJaegerthanks, mordred13:04
pabelangerAJaeger: ++13:04
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add 'ceph status' and tripleo-overcloud-passwords to host_info  https://review.openstack.org/34886213:05
*** _nadya_ has quit IRC13:07
*** yamamoto has joined #openstack-infra13:08
*** ddieterly has quit IRC13:08
*** piet has joined #openstack-infra13:09
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add 'ceph status' and tripleo-overcloud-passwords to host_info  https://review.openstack.org/34886213:09
zigomordred: Can you +workflow https://review.openstack.org/346095 ?13:10
zigoSo we can get going ... :P13:10
*** javeriak has quit IRC13:10
*** kzaitsev_mb has quit IRC13:11
*** sdake has joined #openstack-infra13:11
*** esberglu has joined #openstack-infra13:12
*** amitgandhinz has joined #openstack-infra13:12
*** psilvad has joined #openstack-infra13:12
mordredzigo: done13:12
zigo:)13:12
*** asettle has joined #openstack-infra13:13
pabelangerzigo: have you done any commit to deb-openstack-pkg-tools recently?13:13
zigopabelanger: My colleague did a dumb commit today, yes, and it was published.13:14
pabelangercool, let me check something13:14
*** mdrabe has joined #openstack-infra13:14
zigoI'm not sure why they wanted to do that.13:14
openstackgerritMerged openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node}  https://review.openstack.org/35121913:14
*** mkoderer has joined #openstack-infra13:14
*** lmiccini has joined #openstack-infra13:15
*** amitgandhinz has quit IRC13:17
*** mriedem has joined #openstack-infra13:17
*** mkoderer_ has quit IRC13:17
*** lmiccini_ has quit IRC13:18
vponomaryovsomething wrong with zuul, it does not start jobs anymore13:18
*** cgross has quit IRC13:18
*** rbuzatu has quit IRC13:18
*** cgross has joined #openstack-infra13:18
*** kien-ha has joined #openstack-infra13:19
*** amoralej|lunch is now known as amoralej13:20
sdakepabelanger jeblair i'm told either of you have a solid enough understanding of afs shares in infra to review the patch i have in queu to add percona as a mirror13:21
*** mtanino has joined #openstack-infra13:21
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gpg signing job for reprepro  https://review.openstack.org/34333613:21
pabelangerzigo: mordred: ^ should give us signed commits for the debian-openstack repo13:22
sdakepabelanger jeblair any chance you folks could burn some cycles on that soon - I have a few more to do and want to make sure the work is correct as I am not entirely clear if it is13:22
pabelangersdake: yes, it is on my list. I have to land a few other mirrors before I can get to that.13:22
pabelangersdake: but it does look correct at first glance13:22
sdakepabelanger roger didnt know you had ot land other mirrors13:22
sdakepabelanger cool thats what i was lookinig for13:22
sdakei'll get working on submitting my other patches13:22
sdakeand you can land them when convienent :)13:22
pabelangerya, I need to get our gem mirror online, puppet team has been waiting for a few months on that13:23
sdakedoes it take few months to get mirrors in place ?13:23
sdake(or was base dependency softwre not available)13:23
pabelangersdake: lets see if we can find another infra-root to do it, it should be a good review for people wanting to see how we setup our AFS bit13:23
pabelangersdake: just takes time to schedule the work13:24
sdakepabelanger there is no rush now that i know the patch looks correct13:24
sdakewould be nice for it not to tkae months obviously13:24
sdakebut understand its a big request :)13:24
*** pt_15 has quit IRC13:26
*** senk has quit IRC13:26
sdakepabelanger just a heads up - we have about 5-6 centos mirrors needed and 2-3 ubuntu mirrors13:27
zigopabelanger: So, you're not producing a InRelease file? :/13:27
sdakepabelanger the good news is other teams will probably want  these mirrors as well13:28
zigopabelanger: Example InRelease file: http://ftp.fr.debian.org/debian/dists/testing/InRelease13:29
zigoBest would be to have both.13:29
*** _ari_ has joined #openstack-infra13:29
* zigo comments in the patch13:29
*** itisha has quit IRC13:30
pabelangerzigo: sure, if you tell me how to do it, I can add it13:31
pabelangersdake: sure13:31
openstackgerritJavier Peña proposed openstack-infra/tripleo-ci: Set --use-public option for dlrn execution instead of env var  https://review.openstack.org/35168213:32
zigopabelanger: Done. FYI, the advantage is that InRelease updates are atomic (ie: it is a single file instead of 2 files with Release and Release.gpg)13:32
zigoSo the repo itself becomes atomically updatable.13:32
zigo(and mirrorable)13:32
*** Ravikiran_K has joined #openstack-infra13:33
*** ccamacho|lunch is now known as ccamacho13:33
mordredpabelanger: you wanna wait to update based on zigo's thing?13:33
*** rbuzatu has joined #openstack-infra13:33
zigopabelanger: I commented in.13:34
Shrewspabelanger: I *think* those random zookeeper timeout errors we see in nodepool might be we just aren't waiting long enough for the java stuff13:34
Shrewspabelanger: the kazoo python client waits only 10sec by default13:34
Shrewscccccceviitdrrtdgljjvjfjbrenjgcrckircdrlubgl13:34
Shrewsdoh13:34
zigogate-project-config-compare-xml really takes forever ... :/13:35
openstackgerritMerged openstack-infra/project-config: Add automatic backports  https://review.openstack.org/34609513:35
*** knikolla has joined #openstack-infra13:37
*** mat128 is now known as mat128|away13:37
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gpg signing job for reprepro  https://review.openstack.org/34333613:38
pabelangerupdated for zigo comments and added jessie-backports too13:39
pabelangerShrews: Ah, that would explain it. We likely need to increase that for slower clouds13:39
zigoHow long until I can git clone https://git.openstack.org/openstack/deb-auto-backports and work on it?13:40
openstackgerritSam Betts proposed openstack-infra/project-config: Clean up TinyIPA chroot directories in POST job  https://review.openstack.org/35174213:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: TestSubCommand inherit from UpdateSubCommand.  https://review.openstack.org/31961913:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Don't expose YamlParser methods unnecessarily.  https://review.openstack.org/31961813:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move subcommand logic into subcommand classes.  https://review.openstack.org/31961713:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move 'update' yaml parsing to JenkinsJobs.execute.  https://review.openstack.org/31961613:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Factor XmlJobGenerator out of YamlParser.  https://review.openstack.org/31962313:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Disentangle YamlParser and ModuleRegistry classes  https://review.openstack.org/31962213:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove deprecated 'extended-choice' property.  https://review.openstack.org/31962113:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove YamlParser references from JJB modules.  https://review.openstack.org/31962013:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Be explicit about objects exported by modules.  https://review.openstack.org/31962513:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Cleanup various deprecation warnings.  https://review.openstack.org/31962413:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove unused builder.Builder.update_job method  https://review.openstack.org/31975213:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Rename Builder.delete_job to Builder.delete_jobs.  https://review.openstack.org/31975313:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Rename "parallelize" decorator to "concurrent"  https://review.openstack.org/33378013:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins  https://review.openstack.org/31975413:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Test at more appropriate boundaries for update.  https://review.openstack.org/31961513:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move CacheStorage to its own module.  https://review.openstack.org/31975513:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Make internal builder.Jenkins attrs internal-only.  https://review.openstack.org/31975613:41
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move 'delete' yaml parsing to JenkinsJobs.execute.  https://review.openstack.org/26295413:41
pabelangerzigo: after is merges, should take ~30mins for it to work13:41
*** wewe0901 has quit IRC13:42
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Allow extra time for initial ZooKeeper connection  https://review.openstack.org/35174513:45
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gpg signing job for reprepro  https://review.openstack.org/34333613:45
Shrewspabelanger: hopefully that fixes it ^^^^13:45
pabelangerShrews: ack13:45
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Allow extra time for initial ZooKeeper connection  https://review.openstack.org/35174513:46
*** kgiusti has joined #openstack-infra13:47
Shrewscccccceviitdlrkngfetujiirrclijdkhguekvthitne13:47
Shrewscrap13:47
mordredShrews: you're winning13:47
Shrewsso much13:47
*** karlos_ has quit IRC13:48
*** krtaylor has quit IRC13:48
*** krtaylor has joined #openstack-infra13:50
matt-borlandHello...I would like someone to help review https://review.openstack.org/#/c/349077/ ...Splitting the Horizon integration test job into two jobs...thanks in advance!13:52
*** tonytan4ever has joined #openstack-infra13:53
*** tonytan4ever has quit IRC13:55
*** tonytan4ever has joined #openstack-infra13:55
openstackgerritMerged openstack/os-client-config: Revert "Fix precedence for pass-in options"  https://review.openstack.org/35139413:56
openstackgerritMerged openstack/os-client-config: Pass the argparse data into to validate_auth  https://review.openstack.org/35140513:56
*** mriedem1 has joined #openstack-infra13:56
*** amotoki has joined #openstack-infra13:57
*** amitgandhinz has joined #openstack-infra13:57
*** mriedem has quit IRC13:57
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs  https://review.openstack.org/35127813:57
fungicrinkle: does my answer to your question on https://review.openstack.org/349023 make sense?13:58
openstackgerritMerged openstack/os-client-config: Add test for precedence rules  https://review.openstack.org/35166813:59
openstackgerritAleksey Zvyagintsev proposed openstack-infra/project-config: [wip]  https://review.openstack.org/35175713:59
crinklefungi: yes, thank you14:00
*** _nadya_ has joined #openstack-infra14:00
fungithanks for reviewing that14:00
*** kzaitsev_mb has joined #openstack-infra14:01
fungijust trying to get these flushed through since they're blocking the release team testing tag automation14:01
*** burgerk has joined #openstack-infra14:01
*** mriedem1 is now known as mriedem14:02
*** openstackgerrit has quit IRC14:03
*** openstackgerrit has joined #openstack-infra14:03
*** edmondsw has joined #openstack-infra14:04
*** e0ne has quit IRC14:05
*** e0ne has joined #openstack-infra14:05
*** amitgandhinz has quit IRC14:06
*** amitgandhinz has joined #openstack-infra14:06
*** mkoderer has quit IRC14:08
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Fix requirements for broken os-client-config  https://review.openstack.org/35176214:08
*** cgross has quit IRC14:08
Shrewsmordred: ^^^14:08
*** ddieterly has joined #openstack-infra14:09
*** lmiccini has quit IRC14:09
mordredShrews: thanks!14:10
mordredShrews: I have +Ad14:10
*** mctaylor has joined #openstack-infra14:10
*** PollyZ has joined #openstack-infra14:11
*** coolsvap has quit IRC14:11
*** rbrndt has joined #openstack-infra14:13
azvyagintsevmmedvede fudgi Could you please take a look into "PoC" patches https://review.openstack.org/#/c/351757/ https://review.openstack.org/#/c/344706/ ?14:15
*** mkoderer has joined #openstack-infra14:17
*** tonytan4ever has quit IRC14:17
*** jaosorior has joined #openstack-infra14:17
*** tonytan4ever has joined #openstack-infra14:17
zigopabelanger: Still no repo after 45 minutes, should I be worried?14:18
pabelangermordred: AJaeger: mind helping land https://review.openstack.org/#/c/351734/ that should be the last step for zigo14:18
*** lmiccini has joined #openstack-infra14:19
openstackgerritMerged openstack-infra/puppet-jenkins: Add user.signingkey to .gitconfig  https://review.openstack.org/34902214:19
pabelangerzigo: let me look14:19
*** cgross has joined #openstack-infra14:19
*** armax has joined #openstack-infra14:20
pabelangerzigo: import has been running for 15mins, a lot of projects validate these days14:20
zigoOkay.14:20
*** zz_dimtruck is now known as dimtruck14:20
AJaegermatt-borland: please have some horizon cores review that one as well14:20
matt-borlandAJaeger, robcresswell is the Horizon PTL.14:21
matt-borlandI can get more as well14:21
AJaegermatt-borland: last time I looked robcresswell didn't +1 - so now I'll review ;)14:21
matt-borlandsorry...thanks for checking in on it :)14:21
mtreinishinfra-root: cfriesen_ pointed out on the -qa channel that there is data missing from the subunit2sql db (noticed via o-h) I have a feeling one of the workers isn't working. Can someone take a look at the logs and see if anything is up?14:23
fungilooking14:23
AJaegermatt-borland: so, both jobs should be run on mitaka branch and newer, correct?14:23
fungimtreinish: i rebooted logstash.o.o yesterday after log processor debug logs filled up the rootfs14:23
matt-borlandthat's right AJaeger14:24
*** pcrews has quit IRC14:24
mtreinishfungi: this is going back a few days14:24
fungimtreinish: ahh, got it14:24
mtreinishlike the nova mitaka job doesn't have any results since Aug. 1st14:24
mtreinishs/mitaka job/mitaka py27 job/14:24
AJaegermatt-borland: ok, then LGTM.14:24
*** armax has quit IRC14:25
matt-borlandawseome AJaeger...thanks for that!14:25
*** pcrews has joined #openstack-infra14:25
fungimtreinish: subunit-worker-A is running on worker01 and worker02, checking logs for them now14:25
*** pgadiya has quit IRC14:25
*** senk has joined #openstack-infra14:25
*** mctaylor has quit IRC14:26
*** daemontool has joined #openstack-infra14:26
*** yamamoto has quit IRC14:27
*** zhurong has quit IRC14:27
fungimtreinish: /var/log/subunit2sql/subunit-worker-A-debug.log is empty on worker01, but on worker02 it's full of http://paste.openstack.org/14:27
mtreinishfungi: I think you forgot the numbers on the paste :)14:28
mtreinishfungi: hmm, the log also shouldn't be empty14:28
mtreinishIIRC things are normally kinda verbose14:29
fungioops, http://paste.openstack.org/show/550608/14:29
*** Jeffrey4l has quit IRC14:29
*** Jeffrey4l has joined #openstack-infra14:29
mtreinishfungi: ok, so that means worker 2 probably isn't doing anything. But that shouldn't result in missing data. If it can't connect to the gearman server it just won't process any jobs14:30
fungiworker02 had "receiving packet from" log entries up until 22:15:14 utc yesterday14:30
anteayafungi: you saw this failed in the gate, yeah? https://review.openstack.org/#/c/349023/14:31
mtreinishfungi: that would correspond to our last data in the db14:31
*** karthikp_ has joined #openstack-infra14:31
mtreinishwhich is from: 2016-08-04 22:12:4314:31
fungimtreinish: oh, looks like subunit-worker-A-debug.log on worker01 has been empty since late may14:32
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Keivox KMAP plugin  https://review.openstack.org/35177114:32
fungiso no idea what's up there14:32
mtreinishfungi: http://paste.openstack.org/show/550611/14:32
fungianteaya: yeah, that's some fun bitrot14:32
anteaya:(14:32
mtreinishfungi: yeah that's weird, it's running but not logging. I'm going to assume that worker 1 is the problem child.14:33
mtreinishand our data halted after your logstash.o.o reboot because worker 2 can't connect anymore14:33
AJaegersambetts: thanks for solving that puzzle!14:34
anteayafungi: bit rot in puppet-openstackid?14:34
prometheanfireso, do we have a problem with log size? http://logs.openstack.org/71/349371/5/check/gate-cross-keystone-python27-db-ubuntu-xenial/e4eeaf6/console.html#_2016-08-05_07_15_27_87980014:34
fungianteaya: i think it's another broken git repo on our git backend actually14:34
anteayaah14:35
fungianteaya: that's the same error we had last time when there was a repo that spontaneously lost its master branch14:35
anteayaI missed that there are some broken git repos on our git backend14:35
fungianteaya: me too14:35
anteayaI must have missed that error the other time14:35
fungii need to look into this once i'm done looking at subunit2sql14:35
anteayaI don't know how it was fixed14:35
* anteaya waits14:36
*** karthikp_ has quit IRC14:36
fungianteaya: i found which git server had the broken repo content, blew it away, reran the createcgitrepos script and then retriggered a mirror sync from gerrit for that repo14:36
fungilast time14:36
fungiso this time will likely be similar14:36
*** Goneri has quit IRC14:36
anteayaoh I can check which git server14:36
AJaegervoting time again - thanks anteaya for supervising again!14:36
mtreinishfungi: I really wish we had something to go on worker-1. It feels like it was accepting jobs from gearman but never doing anything with it (which is a scary bug)14:37
anteayajust cloning that repo should tell me, yeah?14:37
mtreinishbecause that's the only thing I could think would cause the missing data14:37
fungianteaya: finding which git server is quick, but i want to audit all the git repos on all the git servers again to make sure this is the only one14:37
anteayaAJaeger: thanks for caring, glad you also appreciate elections :)14:37
anteayafungi: ah okay14:37
fungimtreinish: i can strace the running process tehre14:37
mtreinishfungi: yeah, I lets see if that shows anything14:38
mtreinishif it is doing that it means we've likely lost all the data since the logstash reboot :(14:38
prometheanfireare logs capped at 50MB?14:38
AJaegerfungi, we discussed yesterday that stackalytics.o.o and stackalytics.com had different numbers. stackalaytics.o.o shows me "The data was last updated on 01 Aug 2016 23:48:34 UTC", so that could explain differences in stats14:38
*** pgadiya has joined #openstack-infra14:39
fungimtreinish: looks like it was last restarted on may 20 at 14:39 and stopped logging after 02:24:08 on may 2714:39
fungimtreinish: it's just sitting at a "futex(0x4092d40, FUTEX_WAIT_PRIVATE, 0, NULL"14:40
fungideadlocked maybe?14:40
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Keivox KMAP plugin  https://review.openstack.org/35177114:40
mtreinishfungi: it could have been misbehaving all this time. It would have been very difficult to tell, since worker 2 would fill in half the data14:40
fungimtreinish: claims to have an established gearman socket: tcp        0      0 23.253.73.200:40801     23.253.230.235:4730     ESTABLISHED14:41
mtreinishfungi: hmm, the only explict lock file in the worker script is the pid lockfile which I doubt is that14:42
sambettsAJaeger: yeah, that was a weird one, is there a way to get a notification if a post job like the IPA one fails?14:42
*** e0ne has quit IRC14:42
mtreinishfungi: yeah, that's the bit that's probably all wrong14:42
*** ramishra_ has joined #openstack-infra14:42
*** ayoung_ZZZzzz is now known as ayoung14:42
mtreinishif it does that, but never actually writes anything to the db14:42
*** jdennis has joined #openstack-infra14:42
fungimtreinish: i think that's a dead socket it's listening for14:42
fungii find no evidence it exists on the logstash.o.o side14:43
mtreinishah, ok. that would make sense14:43
fungiso probably some sort of network interruption killed the logstash end of that open connection but failed to signal the worker end14:43
mtreinishbut it does kinda give some evidence that it was probably talking to gearman before the reboot14:43
fungialso after rebooting the logstash server i see no indication anything is listening on 4730/tcp so we probably have a service that isn't set to start at boot there14:44
fungiwhat normally listens on that port on the logstash server?14:44
*** yamamoto has joined #openstack-infra14:45
*** ramishra has quit IRC14:45
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Keivox KMAP plugin  https://review.openstack.org/35177114:45
AJaegersambetts: not really, you could only check for it. For example via health.openstack.org14:46
mtreinishfungi: that's how we have the workers configured: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash/jenkins-subunit-worker.yaml14:46
mtreinishit's the same on the logstash side: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash/jenkins-log-worker.yaml14:46
fungimtreinish: like, it's a gearman server of some sort, just trying to figure out what the initscript for it is14:46
mtreinishfungi: it would be the gearman server14:46
jeblairit's a geard14:46
*** slaweq has quit IRC14:47
fungiyep, i just don't see any obvious gearman or geard initscript so it's presumably embedded in something... jenkins-log-client?14:47
mtreinishfungi: yeah it's the gearman client14:47
fungilooks like that script runs /usr/local/bin/log-gearman-client.py so maybe the client is a server14:47
mtreinishthe server is baked into the script iirc14:47
fungiconfusingly named ;)14:48
mtreinishfungi: http://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/log-gearman-client.py#n15014:48
jeblairyeah, it embeds a gear server14:48
fungiokay, after starting jenkins-log-client manually, it's listening on 4730/tcp again14:49
*** eharney has joined #openstack-infra14:49
jeblair(but is also both a zmq and gearman client)14:49
fungiso presumably worker02 will reconnect automatically. i have a feeling the process on worker01 needs a restart to clear that hung socket14:49
mtreinishfungi: yeah, that will solve the lack of data since yesterday. I'm just a bit worried about the lost data14:49
fungilooks like we have /etc/rc2.d/S20jenkins-log-client symlinked to that initscript, so maybe we have a startup race that prevented it from starting correctly at boot14:50
sambettsAJaeger: what project do the post jobs show up under? I don't see the buildimage-* jobs under http://status.openstack.org/openstack-health/#/g/project/openstack~2Fironic-python-agent14:50
*** mkoderer_ has joined #openstack-infra14:51
*** lmiccini_ has joined #openstack-infra14:51
AJaegersambetts: they need to provide subunit data as well - and not all do, mtreinish can tell14:52
pabelangerAJaeger: have a minute to help me trace down why https://review.openstack.org/#/c/336642/ isn't launching jobs? I suspect the skip-if logic is preventing it, but I cannot seem to find that14:52
AJaegerpabelanger: A minute? Ok ;)14:52
*** roxanaghe has joined #openstack-infra14:52
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Keivox KMAP plugin  https://review.openstack.org/35177114:52
mtreinishAJaeger: right, they need to put subunit data in the log output for it to show up there14:52
pabelangerAJaeger: sure, take your time14:53
AJaegerpabelanger: I see "Queue lengths: 155 events, 120 results. " - let's wait until that is down to zero...14:53
AJaegerbut I'll check zuul output as well14:53
pabelangerAJaeger: sure14:53
mtreinishfungi: it looks like we're getting data again: http://paste.openstack.org/show/550615/14:53
*** cgross has quit IRC14:54
*** mkoderer has quit IRC14:54
*** igorn has quit IRC14:54
*** lmiccini has quit IRC14:54
*** clenimar has quit IRC14:54
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for Keivox KMAP plugin  https://review.openstack.org/35177114:54
prometheanfirecan we talk about the 50MB log size limit?  is there any way to change it?14:55
*** mdrabe has quit IRC14:55
*** igorn has joined #openstack-infra14:55
*** clenimar has joined #openstack-infra14:55
*** cgross has joined #openstack-infra14:55
AJaegerpabelanger: skip-ifs look fine, at least the docs,pep8 and python27 jobs should always run14:55
sambettsmtreinish: This is the job https://review.openstack.org/#/c/351742/1/jenkins/jobs/ironic-python-agent-jobs.yaml@16 that I'd like to get the status of, its a post job that builds an image, and then uploads it to tarballs.openstack.org, what would we need to add to this to make it appear in the health.openstack.org tracker14:56
sambetts?14:56
pabelangerAJaeger: ya, I would expect that too14:56
openstackgerritMerged openstack-infra/jenkins-job-builder: Test at more appropriate boundaries for update.  https://review.openstack.org/31961514:56
mtreinishfungi: I'm thinking we need some sort of feedback loop here. There is no way to tell we're missing stuff short of manual inspection right now, and everything has proven to be pretty unreliable over time14:56
openstackgerritMerged openstack-infra/jenkins-job-builder: Move 'update' yaml parsing to JenkinsJobs.execute.  https://review.openstack.org/31961614:56
mtreinishfungi: it was easier to detect with a single worker, just look at the most recent run in the db. But with >1 worker that doesnt really work14:56
*** roxanaghe has quit IRC14:57
AJaegerpabelanger: but looks strange indeed, i would have expected some more jenkins entries in the history for that change14:57
*** yamamoto has quit IRC14:57
mtreinishsambetts: you need to add subunit output to the job. Here is an example AJaeger did for the translation jobs: https://review.openstack.org/#/c/311459/14:59
*** karthikp_ has joined #openstack-infra15:00
pabelangerAJaeger: okay, 351785 works as expected15:00
mmedvedeazvyagintsev: reviewed15:00
*** vincentll has quit IRC15:01
pabelangerAJaeger: for some reason I think https://review.openstack.org/#/c/336524 needs a new commit, so fix the problem15:01
pabelangertesting that now15:01
AJaegerpabelanger: I suggest to check zuul log files on the server.15:01
AJaegerpabelanger: let's figure out the real problem15:02
azvyagintsevmmedvede: thanks!15:02
*** mdrabe has joined #openstack-infra15:02
fungittx: https://review.openstack.org/#/admin/groups/1525,members15:03
pabelangerAJaeger: yes, checking that now15:03
wznoinskmtreinish: (just looking at your feedback loop suggestion out of curiosity), do you think something like reconciliation against jenkins build history here?15:03
ttxfungi: great, thx!15:03
fungimtreinish: i've restarted jenkins-subunit-worker-A on worker01 now and there's an established gearman connection from it again on logstash.o.o15:04
fungiso i think that must have been disruption back in mat15:04
fungimay15:04
funginetwork disruption15:04
*** d0ugal has quit IRC15:05
fungimtreinish: what do you think about statsd reporting for the number of work requests processed from each worker, and a grafana dashboard trending those?15:05
mtreinishfungi: something had to have been popping gearman jobs off the queue and not populating the db for things to go missing. I'd really like to know what caused that15:05
*** clenimar_ has joined #openstack-infra15:05
mtreinishfungi: so graph the number of incoming requests and then have the actual number of things done ontop or next to it?15:06
AJaegerfungi, I've used topic constraints for all changes to use zuul-cloner with constraints - did a few this morning: https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:constraints I would appreciate review of at least the two oldest ones.15:06
mtreinishwznoinsk: that was something I was thinking about. I just thin we should have a way to make sure we don't drop something on the floor and never realize it.15:06
fungiAJaeger: yep, saw that. it's queued up behind the broken things i need to look into this morning15:06
AJaegerfungi, sure. Let's first fix before we potentially break more ;)15:07
*** dansmith is now known as superdan15:07
*** shardy is now known as shardy_afk15:07
openstackgerritAleksey Zvyagintsev proposed openstack-infra/project-config: [jenkinss]Add {murano-app}-unittest job  https://review.openstack.org/35175715:07
*** gomarivera has joined #openstack-infra15:07
mtreinishwznoinsk: the issue with that though is that there isn't always subunit output from the run (even if it normally has it)15:08
pabelangerAJaeger: working: http://paste.openstack.org/show/550620/15:08
wznoinskmtreinish: know exactly that feeling, I'm paranoid on that bit15:08
*** ddieterly is now known as ddieterly[away]15:08
*** ddieterly[away] is now known as ddieterly15:08
*** ddieterly is now known as ddieterly[away]15:08
pabelangerAJaeger: not-working: http://paste.openstack.org/show/550621/15:09
*** xarses has quit IRC15:09
wznoinskmtreinish: if that to be something you can rely on has to be from the outside of the job so post_test_hook is not an option15:09
openstackgerritAleksey Zvyagintsev proposed openstack-infra/project-config: [jenkinss]Add {murano-app}-unittest job  https://review.openstack.org/35175715:10
fungianteaya: looks like this time the git problem is a missing master branch in /var/lib/git/openstack-infra/puppet-openstackid.git on git08.o.o (and nowhere else)15:10
fungiwhat's strange is that it's the same repo that broke last time, except last time it broke this way on git0415:11
fungigiven we have >1k repos, this can't be a coincidence15:11
*** yamamoto has joined #openstack-infra15:11
fungii'm at a loss as to what's different about that repo though15:12
*** olaph is now known as beefboymattface15:13
*** jistr is now known as jistr|mtg15:14
sambettsmtreinish: Uploading the subunit information won't catch if the problem occurs in the publisher right?15:14
*** ajmiller has joined #openstack-infra15:14
*** beefboymattface is now known as beefboymf15:14
openstackgerritMerged openstack-infra/tripleo-ci: Enable net-iso for ovb jobs  https://review.openstack.org/34301715:15
sambettsmtreinish: ie. the recent error we saw was caused by the scp publishers rsync command trying to read a file that it didn't have permissions for, so the job failed to upload the files we wanted15:15
*** beefboymf is now known as beefboymattface15:15
AJaegerpabelanger: could there be a cycle in these depends-on from https://review.openstack.org/#/c/337676/ ?15:15
mtreinishsambetts: no they won't. There has been a plan to use a zuul mysql reporter for the top level info for a while. (which would catch those and other failures we miss) But, I don't think the patch adding that has landed15:16
openstackgerritTimothy R. Chavez proposed openstack-infra/jenkins-job-builder: Add support for the random string parameter  https://review.openstack.org/35138415:16
pabelangerAJaeger: oh, maybe15:17
*** e0ne has joined #openstack-infra15:17
Shrewspabelanger: looks like the timeout bump might be good: https://review.openstack.org/35174515:17
pabelangerAJaeger: going to have them remove https://review.openstack.org/#/c/337669/ depends-on as it is circular15:17
*** d0ugal has joined #openstack-infra15:18
*** mhickey has quit IRC15:18
mtreinishsambetts: nope it's still open: https://review.openstack.org/#/c/223333/15:18
*** xnox has quit IRC15:18
*** e0ne has quit IRC15:19
pabelangerShrews: great15:19
AJaegerpabelanger: yep, 337676 -> 337669 -> 333612 -> 337676. Just don't understand how 336642 plays into it. But fixing that cycle is important15:19
pabelangerAJaeger: Agreed, lets start with that15:20
AJaegerpabelanger: ah, 336312 depends-on 336551 - and your change 642 is stacked on top of 551. So, that shows the relation.15:20
*** javeriak has joined #openstack-infra15:20
AJaegerpabelanger: Let them break the cycle and be happy ;)15:20
*** tesseract- has quit IRC15:21
anteayafungi: that is really odd that that is the repo that has broken twice15:22
*** ganesan has joined #openstack-infra15:23
anteayafungi: how does a repo even lose its master branch15:23
fungianteaya: some sort of interruption/corruption/truncation during a mirror push from gerrit is the only way i can think of15:24
anteayathat is really odd15:25
*** pt_15 has joined #openstack-infra15:25
*** armax has joined #openstack-infra15:26
*** rcernin has quit IRC15:26
*** itisha has joined #openstack-infra15:26
ganesanI am setting up 3rd party ci and struck with node pool. nodepool spins vm everytime due to ssh issue " ERROR nodepool.NodeLauncher: Timeout launching node id: 23480 in provider: local_02 error: Timeout waiting for ssh access"15:27
wznoinskganesan: can you spawn a VM manually on local_02 cloud and connect to it?15:28
wznoinskganesan: we can discuss it here so others can learn too, if you/nodepool can reach the machine over network, it may indicate a problem with logging in to the machine (user/pass/ssh key)15:30
ganesansure15:30
pabelangerAJaeger: nice work, that appears to have fixed it15:30
ganesanmanually it works15:31
*** ddieterly[away] is now known as ddieterly15:31
ganesanI do have 2 network, 1 - private, 2- public (floating IP)15:31
*** Teresita-Warrior has joined #openstack-infra15:31
wznoinskganesan: have you tried to log in to the VM spawned by nodepool while it is waiting to be timedout? (it's probably 1-2 mins before it does it)15:31
AJaegergreat, pabelanger15:32
AJaegerpabelanger: could you review sambetts's change https://review.openstack.org/351742 , please?15:33
*** hurgleburgler has joined #openstack-infra15:33
ganesanyes, I tried, but couldn't due private network is assigned to vm , which is not accessible from nodepool machine15:33
*** links has quit IRC15:33
wznoinskfungi: maybe something like iwatch or audit on the directory would be good if the problem persists? ... I guess the branches may be dropped/recreated on sync to avoid potential problem... storage may not be syncing quick enough or something's still using the files and branch can't be deleted15:34
ganesanI do have floating IP and trying to assign to those vm15:34
*** gouthamr has quit IRC15:34
wznoinsk... leaving it in a dirty state15:34
*** gouthamr has joined #openstack-infra15:34
fungiwznoinsk: that would be strange for sure, but doesn't explain that out of the only two incidences in recent history it happened to the same repo (out of more than a thousand) on two different git servers15:35
*** yamamoto has quit IRC15:35
openstackgerritSam Yaple proposed openstack-infra/shade: [WIP] Add update_endpoint()  https://review.openstack.org/28905215:36
wznoinskare there other (infra) scripts using that repo that may collide at the time of sync?15:36
wznoinskganesan: I don't use private network in my cloud I give to nodepool, never tested what you have15:36
*** burgerk has quit IRC15:36
ganesanis it possible to specify floating IP to vm's in addition to private network IP15:37
*** _nadya_ has quit IRC15:37
wznoinskwznoinsk: but it sounds like a floating ip issue (networks setting in nodepool.yaml)15:37
wznoinskunder 'providers' section15:38
wznoinsks/wznoinsk/ganesan - talking to myself again15:38
ganesanI am trying to config nodepool to set floating ip for vm's. But I could not find how to specify15:39
openstackgerritRobin Naundorf proposed openstack-infra/project-config: Changed comment which is no longer valid  https://review.openstack.org/35180415:39
wznoinskhttps://github.com/openstack-infra/project-config/blob/master/nodepool/nodepool.yaml#L633, use public instead15:40
wznoinskor whatever the name your public network has15:40
openstackgerritSam Betts proposed openstack-infra/project-config: Get IPA buildimage POST job into healthcheck  https://review.openstack.org/35180615:42
*** javeriak has quit IRC15:42
sambettsmtreinish, AJaeger ^^15:42
*** sdague has quit IRC15:42
*** javeriak has joined #openstack-infra15:43
sambettsmtreinish, AJaeger: based on the example patch, I think this looks more or less the right way to do it for our job15:43
*** javeriak has quit IRC15:43
*** HenryG has quit IRC15:43
fungiwznoinsk: the only other possibility i can think of is that something is going wrong during git repack, but i' still at a loss as to why it would be just this one repo15:43
*** HenryG has joined #openstack-infra15:43
anteayafungi: has this just started happening since zaro's patch merged?15:45
openstackgerritMerged openstack-infra/project-config: Clean up TinyIPA chroot directories in POST job  https://review.openstack.org/35174215:45
AJaegersambetts: yeah, looks good.15:46
AJaegersambetts: I regularly check for translations jobs: http://status.openstack.org/openstack-health/#/g/build_queue/post?groupKey=build_queue&searchJob=translation15:46
openstackgerritMerged openstack-infra/storyboard: Position archived items at the bottom of worklists  https://review.openstack.org/35126515:46
zaromorning15:46
*** rfolco_ has quit IRC15:47
AJaegerdhellmann: do you know why reno fails to generate translations in the nova post job? See http://logs.openstack.org/4c/4cbfed44f4b1dc4af21e9a09ed830617cb3caa05/post/nova-upstream-translation-update/2fd7f7b/console.html#_2016-08-05_15_02_44_41776015:47
*** pgadiya has quit IRC15:48
anteayamorning zaro15:48
*** florianf has quit IRC15:48
fungianteaya: yeah, that's what i'm trying to determine. the timestamps are suspicious... git fsck says my archived copies of the broken repo on git04 and git08 have bad packfiles. the broken packed-refs file on git04 was modified at 04:07 utc on july 27, and on git08 at 04:07 utc on august 515:48
fungi04:07 is when the git gc runs every night on our git server backends15:49
*** florianf has joined #openstack-infra15:49
sambettsAJaeger: yeah it would be super useful for us to be able to do that with the buildimage jobs, one of them has been failing for >1 month without us realising15:49
*** xarses has joined #openstack-infra15:49
sambettsAJaeger: and if that data appears in health.openstack.org, I have an idea to build a bot that watches for it too15:50
fungianteaya: zaro: oh, this could be the problem... :(15:50
AJaegersambetts: integrate it with harlowja's new irc bot and have it complain to your channel whenever something fails ;)15:50
fungianteaya: zaro: we have two cronjobs running on our git servers at 04:07. one that does git pack-ref and another than does git gc15:50
anteayaoh my15:50
AJaegerfungi, in parallel?15:51
sambettsAJaeger: ooo interesting, is there a link to that somewhere?15:51
zarothat's not good15:51
anteayathe point was to change the cronjob: https://review.openstack.org/#/c/342388/15:51
fungianteaya: zaro: i bet the puppet change to add the mirror_gitgc cronjob on them didn't ensure=>absent on the mirror_repack cronjob15:51
AJaegersambetts: openstack-dev mailing list discussion15:51
anteayafungi: ah15:51
anteayagood hunting15:51
zaroi thought i set puppet to remove the repack cron job?15:51
fungiso we have a repack and a gc happening at the same time, stepping on one another15:52
anteayazaro: this is the patch to change the cronjob, right? https://review.openstack.org/#/c/342388/15:52
anteayafungi: I'm surprised we didn't have more races15:52
fungianteaya: that's on gerrit, not on our git backends15:52
sambettsAJaeger: ah sweet, I'll try to dig that up15:52
anteayafungi: ah15:52
AJaegersambetts: http://lists.openstack.org/pipermail/openstack-dev/2016-July/100409.html - repo is openstack-infra/err2d215:52
AJaegersambetts: http://git.openstack.org/cgit/openstack-infra/err2d215:52
sambettsAJaeger: :D thanks!15:52
*** jlanoux has joined #openstack-infra15:52
krotscheckAJaeger: I changed the job names as you requested, is this what you were looking for? https://review.openstack.org/#/c/346130/15:52
krotscheckOh, right.15:53
krotscheckSorry sick brain.15:53
* krotscheck remembers the conversation about bindep15:53
*** florianf has quit IRC15:53
fungianteaya: zaro: i see the problem :/15:53
anteaya:(15:53
AJaegerkrotscheck: yes, that's what I was looking for regarding names. Did you see my comment on installing a package of sphinx instead of globally pip installing?15:53
AJaegerkrotscheck: on revision 815:54
*** narayrak has quit IRC15:54
krotscheckAJaeger: Ah, I did not. Let me check that for you.15:55
zarofungi: i think it's set https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/git_backend.pp#n12015:55
krotscheckAJaeger: I suspect the debian package may be out of date.15:55
krotscheckBut I'll check15:55
fungizaro: you might think that, but check the coming patch15:55
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Ensure git cronjob absent for correct user  https://review.openstack.org/35181615:55
fungizaro: anteaya: ^15:55
*** sdake has quit IRC15:56
*** sdake has joined #openstack-infra15:56
anteayatricky15:56
zarothanks fungi.15:56
fungianteaya: zaro: it was ensuring teh mirror_repack cronjob was absent from root's crontab, which it was, because it never existed15:56
anteayathank you fungi15:56
fungii'll check the gerrit manifest to make sure it's not broken similarly15:57
anteayaeasy to ensure something is absent if it never existed15:57
zaroi think there were a couple other spots that need this as well?15:57
*** mikelk has quit IRC15:57
AJaegerdhellmann: normal release notes for nova are broken as well - how could they merged something that broke the releasenotes? See http://logs.openstack.org/e9/e99c3665431462fc32877b71ba876da81cb61f3a/post/nova-releasenotes/0d85028/console.html15:57
zaroohh yeah, looks like your on it already15:57
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Fix /v1/worklists/:id/items endpoint  https://review.openstack.org/35181915:58
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Ensure git cronjob absent for correct user  https://review.openstack.org/35181615:58
fungithere was indeed a second one in system-config15:58
anteayathank you15:59
zarofungi: there's another one in puppet-gerrit project in  manifests/cron.pp15:59
*** gyee has joined #openstack-infra16:00
fungizaro: i'm aware--in the middle of editing that now16:00
*** jlanoux has quit IRC16:01
openstackgerritJeremy Stanley proposed openstack-infra/puppet-gerrit: Ensure git cronjob absent for correct user  https://review.openstack.org/35182016:01
fungizaro: anteaya: ^16:01
fungii think those two changes cover everything16:02
*** psachin has quit IRC16:02
*** signed8bit has joined #openstack-infra16:03
pabelangerzigo: So, you didn't update gerrit/projects.yaml to include deb-auto-backports, that's why the repo isn't created16:03
zarome too16:03
*** PollyZ has quit IRC16:04
krotscheckAJaeger: Current version of xenial python-sphinx is 1.3.6 - will that do?16:04
anteayathanks fungi and zaro16:04
*** davideagnello has joined #openstack-infra16:04
pabelangerfungi: anteaya: do you have a moment to help review https://review.openstack.org/#/c/351251/ adds a grafana dashboard for tripleo-test-cloud-rh116:04
*** pcaruana has quit IRC16:05
anteayapabelanger: swap you for https://review.openstack.org/#/c/351816/2 and https://review.openstack.org/#/c/351820/1 which fixes a too many cronjobs issue with the git repos16:05
pabelangeranteaya: sure, looking16:06
*** roxanaghe has joined #openstack-infra16:08
anteayapabelanger: thank you, +3 on yours16:08
*** dtantsur is now known as dtantsur|afk16:08
pabelangeranteaya: thanks, +2 on both patches16:08
AJaegerkrotscheck: try it ;) For python jobs, we have in global requiremnents capped to < 1.316:08
anteayapabelanger: thank you16:08
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-tripleo-quickstart-docs-ubuntu-xenial job  https://review.openstack.org/34908716:08
*** senk has quit IRC16:09
pabelangerAJaeger: ^decided not to use openstack-server-publish-jobs as ansible-role-jobs is our intended template16:09
*** jimbaker has quit IRC16:11
*** shardy_afk is now known as shardy16:12
*** davideagnello has quit IRC16:12
*** roxanaghe has quit IRC16:12
*** PollyZ has joined #openstack-infra16:12
AJaegerpabelanger: please keep consistency in mind. Let's see how that looks later...16:12
*** davideagnello has joined #openstack-infra16:13
AJaegernote that openstack-server-publish-jobs would publish, your change just builds but does not publish16:13
*** amoralej is now known as amoralej|off16:13
pabelangerAJaeger: right, I am trying to figure out the difference between {name}-docs-{node} and gate-{name}-docs-{node}16:13
pabelangerAJaeger: right now, we use gate-*16:13
*** roxanaghe has joined #openstack-infra16:13
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: TestSubCommand inherit from UpdateSubCommand.  https://review.openstack.org/31961916:13
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Don't expose YamlParser methods unnecessarily.  https://review.openstack.org/31961816:13
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move subcommand logic into subcommand classes.  https://review.openstack.org/31961716:13
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Factor XmlJobGenerator out of YamlParser.  https://review.openstack.org/31962316:13
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Disentangle YamlParser and ModuleRegistry classes  https://review.openstack.org/31962216:13
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove deprecated 'extended-choice' property.  https://review.openstack.org/31962116:13
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove YamlParser references from JJB modules.  https://review.openstack.org/31962016:13
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Be explicit about objects exported by modules.  https://review.openstack.org/31962516:13
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Convert multiline strings to concatenated single-quote.  https://review.openstack.org/35174316:14
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Cleanup various deprecation warnings.  https://review.openstack.org/31962416:14
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Remove unused builder.Builder.update_job method  https://review.openstack.org/31975216:14
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Rename Builder.delete_job to Builder.delete_jobs.  https://review.openstack.org/31975316:14
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Rename "parallelize" decorator to "concurrent"  https://review.openstack.org/33378016:14
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins  https://review.openstack.org/31975416:14
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move CacheStorage to its own module.  https://review.openstack.org/31975516:14
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Make internal builder.Jenkins attrs internal-only.  https://review.openstack.org/31975616:14
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move 'delete' yaml parsing to JenkinsJobs.execute.  https://review.openstack.org/26295416:14
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move macro expansion into YamlParser.  https://review.openstack.org/33307616:14
pabelangerAJaeger: okay, I see16:14
pabelangerAJaeger: one is post job16:14
openstackgerritMerged openstack-infra/system-config: Set the gitpgpkey for signing nodes  https://review.openstack.org/34902316:14
pabelangerother is gate16:14
AJaegerpabelanger: exactly16:14
pabelangerAJaeger: in that case, we don't need post jobs right now. But will in the future16:15
*** jimbaker has joined #openstack-infra16:15
*** jimbaker has quit IRC16:15
*** jimbaker has joined #openstack-infra16:15
pabelangerAJaeger: actually, I can add them now16:15
openstackgerritMerged openstack-infra/project-config: Add tripleo-test-cloud grafana dashboard  https://review.openstack.org/35125116:15
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Rename "parallelize" decorator to "concurrent"  https://review.openstack.org/33378016:17
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Convert multiline strings to concatenated single-quote.  https://review.openstack.org/35174316:17
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-tripleo-quickstart-docs-ubuntu-xenial job  https://review.openstack.org/34908716:17
pabelangerAJaeger: update^16:17
*** Swami has joined #openstack-infra16:19
*** PollyZ has quit IRC16:19
*** mriedem is now known as mriedem_lunch16:20
pabelangercloudnull: how did you make out adding ipv6 to osic?16:20
cloudnullpabelanger: not yet. im doing it next week16:20
pabelangercloudnull: nice16:20
krotscheckAJaeger: Alright, so, just to clarify your comment - should I be using the install-sphinx macro I created, wait for the bindep changes to land, or...16:21
pabelangerwas just looking at nodepool logs and ipv6 address scrolled by, reminded me about the upgrade16:21
cloudnullpabelanger:  i wanted to do it this week but its been a little hectic16:21
*** vvalderrv has joined #openstack-infra16:21
openstackgerritMerged openstack-infra/shade: Fix requirements for broken os-client-config  https://review.openstack.org/35176216:22
*** asettle has quit IRC16:22
AJaegerkrotscheck: my personal preference would be for not installing in project-config - and that means solving the bindep issue first. And that needs some discussion between a not ill krotscheck, jeblair and perhaps others on naming first...16:23
*** asettle has joined #openstack-infra16:23
AJaegerkrotscheck: and then you could just install python-sphinx via the bindep file.16:23
AJaegerYou could also for now change the install-sphinx macro to use the package instead of pip - to make it clearer what you want and ease the migration to bindep.16:24
*** roxanaghe has quit IRC16:24
AJaegerkrotscheck: sorry, I'm out for today. Feel free to discuss without me on how to move forward.16:26
krotscheckAJaeger: No worries16:26
* AJaeger wishes a great weekend to everybody16:26
krotscheckAJaeger: Well, once we do bindep we can remove all the install-xvfb macros etc.16:26
krotscheckSo I'll assume cleanup will happen and do that.16:27
AJaegerkrotscheck: yeah, I would love that ;)16:27
jeblairi'm available for bindep bikeshedding if others are16:27
*** asettle has quit IRC16:27
*** burgerk has joined #openstack-infra16:27
AJaegerjeblair: I might stay a bit longer for a nice bikeshed but might need to leave rather quickly then...16:27
*** baoli_ has quit IRC16:27
*** baoli has joined #openstack-infra16:28
jeblairfungi: around?16:28
fungijeblair: yep16:28
fungii vote fuscha16:28
*** rwsu has quit IRC16:28
AJaegerjeblair: this change triggered the discussion: https://review.openstack.org/33487016:28
jeblairso i think the important thing is whether bindep reqs files are meant to be user facing16:29
*** coolsvap has joined #openstack-infra16:29
fungiin my mind, their primary importance is so that developers know what distro packages they will need to install before they can test changes they make to the repo they've checked out16:29
jeblairmy understanding was that they were, and that we wanted other-requirements to be not only consumed by ci, but information provided to users as well16:29
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: fix its_rules parameterization in template  https://review.openstack.org/35132316:30
fungithe goal with the ci integration, was to hook into that user-facing documentation and reuse it to avoid duplicating information about dependencies16:30
mordredyes. I agree with both of those goals16:30
AJaegerI agree with you - I just want to point out that this will never be a perfect information. We test it in the gate, so this is the minimal package set. We might miss a package or two - but that can be easily changed16:30
AJaegerkrotscheck, still around and reading the above?16:31
jeblairright, so my question for krotscheck is -- was the objection to 'other-requirements.txt' something about the filename (like "use a word other than requirements"), or that the filename was visible to developers when they look at the repo?16:31
*** gomarivera has quit IRC16:31
fungiin addition to the list serving as documentation, it's formatted in a standardized machine-parsable way, and there is an available tool (bindep) to check whether you have them installed on your system16:31
*** rwsu has joined #openstack-infra16:31
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add 'ceph status' and tripleo-overcloud-passwords to host_info  https://review.openstack.org/34886216:32
*** sambetts is now known as sambetts|afk16:32
pabelangerrcarrillocruz: just approved a bunch of omfracloud patches of yours.  Does that get us to run bifrost now?16:32
*** baoli has quit IRC16:33
fungibut to divine bindep's purpose, it was originally conceived because someone was locally testing patches to different openstack projects, and got tired of having to keep track in ad-hoc prose documentation of what distro packages they needed to install to do that16:33
*** Teresita-Warrior has quit IRC16:33
fungibecause it differed from one project to the next, and some documented it better than others, or in different places than others, or often not at all16:34
*** shashank_hegde has joined #openstack-infra16:34
mordredI could see just calling it "bindep.txt" - the filename "other-requirements.txt" does not inspire me to think that the bindep tool might be involved somehow16:35
fungileveraging a standardized descriptive format in our ci was seen also as a means of encouraging projects to adopt a standard around it16:35
fungier, to adopt that standard16:35
*** Apoorva has joined #openstack-infra16:35
jeblair'system-requirements.txt' if the word 'other' is problematic.  but bindep.txt wfm.16:36
*** tphummel has joined #openstack-infra16:36
*** lucasagomes is now known as lucas|dinner16:36
fungiand yeah, i'm not married to the other-requirements.txt filename. that was simply chosen by the initial author early on, and was unfortunately inspired by contemporary work in tracking python package requirements in "other" requirements.txt files16:36
*** jpich has quit IRC16:36
openstackgerritMerged openstack-infra/system-config: Add controller01.vanilla metadata for better tracking  https://review.openstack.org/35133116:37
fungiwe should keep it working for backwards compatibility, but i'm fine with a different filename16:37
mordredI could imagine 'requirements.txt' having python connotations and folks from other languages not wanting to have 'python' files in their source repos16:37
mordredbut yes to keeping it working for backwards compat16:37
openstackgerritMerged openstack-infra/system-config: Add compute040.vanilla metadata for better tracking  https://review.openstack.org/35133216:37
jeblairyeah, the previous alternate suggestion was .bindeprc as implemented here: https://review.openstack.org/#/c/350184/16:38
jeblairand i feel that counteracts the goal of developer visibility16:38
krotscheckjeblair: I have three objections.16:38
jeblair(so just want to try to get us on the same page)16:38
krotscheckjeblair: Firstly, 'other-requirements.txt' is fairly clearly a python thing, as it's very similar to requirements.txt, test-requirements.txt, etc.16:38
krotscheckjeblair:  Secondly, it depends on those other files to inform its context - If I see those three files in a repository, they provide meaning to each other's purpose. If I only see other-requirements.txt (as I would in a non-python project like java or javascript), then the prefix other-... means what? What's going on? I don't know!16:38
krotscheckjeblair:  The third objection is that it in no way informs what tool it is related to. JS developer comes along, sees that file, opens it and sees package names, they can infer what the purpose is, but would they know how to properly use it? There's nothing that says "this is a bidep thing".16:39
AJaegerkrotscheck: So, system-requirements.txt would work?16:39
openstackgerritMerged openstack-infra/system-config: Add compute042.vanilla metadata for better tracking  https://review.openstack.org/35133416:39
mordredkrotscheck: or bindep.txt ?16:39
jeblairya, both words in "other" and "requirements" are problematic then.  :)16:39
*** tonytan4ever has quit IRC16:39
krotscheckmordred: yes, AJaeger: yes, but less so.16:40
fungikrotscheck: the file supports comments. in the changes where i've proposed it to other projects i've included a very descriptive comment preamble explaining what it is, why it's there, and where to find further documentation on it16:40
jeblairsystem-requirements.txt arguably addresses objections 1 and 2.  probably not 3.  maybe that could be addressed by a boilerplate comment in the file.  bindep.txt addresses all 3, i think?16:40
AJaegerkrotscheck: https://review.openstack.org/#/c/334342/10/other-requirements.txt has in the first lines a comment what the file is about16:40
krotscheckI feel that if we're going to change the name to begin with, we should go with something that makes sense cross languages, like .bindeprc or /.config/bindep.??16:40
mordredwell, we would like to not have it be hidden16:40
AJaegerkrotscheck: but then it's not visible16:41
mordredand I think there's good precedent for that in other languages - bower.json is not a hidden file, for  instance16:41
krotscheckmordred: Yes, but .eslintrc is, as is .travis.yml16:41
krotscheckmordred: The precedent that I'm familiar with is that files directly related to the code are visible, files related to the meta tooling are not.16:42
jeblairthe difference is that bindep is meant to be consumed by humans, not only machines16:42
openstackgerritMerged openstack-infra/system-config: Add compute021.vanilla metadata for better tracking  https://review.openstack.org/35134816:42
openstackgerritMerged openstack-infra/system-config: Add compute018.vanilla metadata for better tracking  https://review.openstack.org/35135016:42
mordredkrotscheck: yah - in this case the file seems related to the code to me, although I can see how a different pov could exist16:42
openstackgerritMerged openstack-infra/system-config: Add compute027.vanilla metadata for better tracking  https://review.openstack.org/35135216:42
openstackgerritMerged openstack-infra/system-config: Add compute028.vanilla metadata for better tracking  https://review.openstack.org/35135416:42
fungikrotscheck: is documentation related to the code, or the tooling?16:43
mordredkrotscheck: as in "this code is not going to work if you don't install these packages on your host os"16:43
fungior "you won't be able to test changes to this code if you don't install the following tools first"16:43
*** senk has joined #openstack-infra16:44
krotscheckRight, I don't actually care about the visibility of the file. I would like a name where there's enough established cross-project patterns so there's a reasonable chance someone can infer the file's context from its name.16:44
krotscheckI'm using .bindeprc as an example. The .<name>rc pattern is pretty well established, but others also exist.16:44
*** ddieterly is now known as ddieterly[away]16:44
*** d0ugal has quit IRC16:44
*** tonytan4ever has joined #openstack-infra16:45
fungiand if people want to know what packages your project needs, you tell them to look in this hidden file16:45
jeblairthen i think the best compromise here is going to be either bindep.txt or system-requirements.txt.16:45
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update properties.slack to use convert_mapping_to_xml  https://review.openstack.org/35184116:46
* mordred votes for bindep.txt16:46
mordredlargely since system-requirements.txt doesn't tie to tool and also has a python implication16:46
krotscheckmordred: Well, system-requirements I feel carries its own context.16:47
krotscheckmordred: Anyone who's ever played a videogame knows what system-requirements imply.16:48
jeblair4mb of ram!16:48
AJaegerWhat should the order be? Read other-requirements.txt if it exists; if not try system-requirements/bindep (or whatever we choose)?16:48
*** Swami_ has joined #openstack-infra16:48
*** ddieterly[away] is now known as ddieterly16:48
krotscheckjeblair: DirectX5!16:48
krotscheckSo I'm fine with either16:48
mordredkrotscheck: ah. good point16:48
* krotscheck wonders if system-requirements.txt will include a line called '3.5" floppy'16:49
*** Swami__ has joined #openstack-infra16:49
*** Swami has quit IRC16:49
pabelangerI like bindep.txt, as it relates to mordred comments about bower.json16:49
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update properties.slack to use convert xml  https://review.openstack.org/35184116:49
jeblair2 votes for bindep.txt!16:49
fungii have no preference, i just think having it as a dotfile makes it a bit nonobvious when it's meant to be obvious16:50
openstackgerritMerged openstack-infra/system-config: Add compute029.vanilla metadata for better tracking  https://review.openstack.org/35135716:50
krotscheckIs the extension correct? Is it just a text file, or a .conf style file, or a .yaml?16:50
*** senk has quit IRC16:50
openstackgerritMerged openstack-infra/system-config: Add compute020.vanilla metadata for better tracking  https://review.openstack.org/35135916:50
openstackgerritMerged openstack-infra/system-config: Add compute013.vanilla metadata for better tracking  https://review.openstack.org/35136916:50
openstackgerritMerged openstack-infra/system-config: Add compute010.vanilla metadata for better tracking  https://review.openstack.org/35137116:50
AJaegerkrotscheck: just text file that allows comments16:50
fungikrotscheck: it's its own text-based format16:50
mordredkrotscheck: it's not .conf or .yaml16:50
openstackgerritMerged openstack-infra/system-config: Add compute019.vanilla metadata for better tracking  https://review.openstack.org/35137216:50
openstackgerritMerged openstack-infra/system-config: Add compute006.vanilla metadata for better tracking  https://review.openstack.org/35137416:50
openstackgerritMerged openstack-infra/system-config: Add compute001.vanilla metadata for better tracking  https://review.openstack.org/35137816:51
openstackgerritMerged openstack-infra/system-config: Add compute000.vanilla metadata for better tracking  https://review.openstack.org/35137716:51
openstackgerritMerged openstack-infra/system-config: Add compute034.vanilla metadata for better tracking  https://review.openstack.org/35137916:51
jeblairi think bindep.txt is the winner then16:51
openstackgerritMerged openstack-infra/system-config: Add compute017.vanilla metadata for better tracking  https://review.openstack.org/35138016:51
*** tphummel has quit IRC16:51
jeblairalso, does anyone else suddenly want ice cream?16:51
openstackgerritMerged openstack-infra/system-config: Add compute039.vanilla metadata for better tracking  https://review.openstack.org/35138316:51
fungiwe could make up a vanity extension to denote that it's in "bindep format" but that seems unnecessary16:51
pabelangerjeblair: yes!16:51
openstackgerritMerged openstack-infra/system-config: Update iLO IP for compute016.vanilla  https://review.openstack.org/35139216:51
fungijeblair: i suddenly have a craving for vanilla16:52
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Add .bindeprc as additional filename  https://review.openstack.org/35018416:52
* AJaeger forgot to update commit message, try again...16:52
jeblair:)16:52
*** Swami_ has quit IRC16:52
krotscheckjeblair: I like vanilla :)16:52
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Add bindep.txt as additional filename  https://review.openstack.org/35018416:53
pabelangeraiming for maple walnut for a 3rd zone16:53
jeblairkrotscheck, AJaeger, fungi, mordred, pabelanger: thanks for sticking through a bikeshed convo :)16:53
* krotscheck prefers red.16:53
AJaegerplease review whether the logic is fine in 350184 - I tested the previous version and that worked fine.16:53
AJaegerjeblair: was a pleasure ;)16:54
jeblairpabelanger: i thought we were going neapolitan -- maybe zone 4?16:54
*** jbernard has quit IRC16:54
*** kzaitsev_mb has quit IRC16:55
pabelangerjeblair: actually, I think we should name the cloud Neapolitan and have 3 zones of f vanilla, chocolate, and strawberry16:55
* fungi wants a cosmopolitan now16:55
fungiit _is_ friday after all16:55
*** ajmiller_ has joined #openstack-infra16:55
krotscheckThe next project I'm a member of we're going to use ice cream flavors as release names.16:57
mordredkrotscheck: what about types of parsley?16:57
jeblairAJaeger: left a comment16:58
*** Swami__ has quit IRC16:58
krotscheckmordred: I don't know, parsley doesn't seem like a great theme for a release party.16:58
jeblairkrotscheck: "drinks made with parsley", however...16:58
*** mat128|away is now known as mat12816:58
krotscheckmordred: Ah Yes.16:58
mordredkrotscheck: come on ... giant tubs of parsely? you could also have a parsley bowl next to a cilantro bowl and play a guessing game ...16:59
krotscheckjeblair: I shall put that on my list-of-playa-cocktails-to-design.16:59
fungiat least everyone would have fresh breath16:59
AJaegerjeblair: right now it uses specified file - and uses other-requirements as default for that.16:59
krotscheckmordred: The first party? Sure.... but in a CI/CD project, I feel like that might get old.16:59
AJaegerjeblair: I can implement what you suggested. Could others comment as well on what the behaviour should be ,pl easE?16:59
mordredkrotscheck: :)16:59
krotscheckmordred: "We released 3 times today, here's your tub of greens"17:00
mordredkrotscheck: the gimmick could be that all of the parties are always identically boring17:00
*** ajmiller has quit IRC17:00
mordredwhich is actually what CI/CD should really be - as boring as possible, right?17:00
krotscheckmordred: Yeah, if CI/CD is interesting, that's a problem.17:00
mordredAJaeger, jeblair: I think it should look for both files, work if either are found, and error if both are found - but that's just me17:01
openstackgerritPaul Belanger proposed openstack-infra/project-config: Fix incorrect labels for RH1 in grafyaml  https://review.openstack.org/35184617:02
krotscheckianw: If you're around, we've got a new version of the NPM-devstack job https://review.openstack.org/#/c/348056/17:02
AJaegermordred, jeblair : IF a command line arg is given, use that file. Otherwise check for bindep, check for other-requirements - and fail if both don't exist?17:03
fungikrotscheck: i expect ianw's asleep for a while still. we usually don't see him around in here until 20:00 utc or later (but also it's his saturday already)17:04
jeblairAJaeger: yes, though mordred also suggested failing if _both_ exist17:04
krotscheckfungi: Ah, wasn't aware that he's down under.17:05
krotscheckTimezones are hard on IRC17:05
AJaegerjeblair: indeed. Let's go with that...17:05
AJaegerthanks17:07
*** d0ugal has joined #openstack-infra17:11
*** gomarivera has joined #openstack-infra17:12
dhellmannfungi : how do I get the launchpad libraries and their dependencies installed on the signing node? http://logs.openstack.org/30/3076a6c65b3f1e93da77fe61881f60f0b10d3759/release-post/tag-releases/2645a3a/console.html17:13
*** daemontool_ has joined #openstack-infra17:13
*** tphummel has joined #openstack-infra17:13
fungidhellmann: oh, we can puppet the distro packages for those, but the bigger concern is how to puppet authentication for that17:14
fungii can look at what we do to update bugs from gerrit hooks, though i think there may be a manual bootstrapping step in there we never fully automated17:15
*** signed8bit is now known as signed8bit_Zzz17:15
dhellmannok17:15
fungisince last i looked you have to use their openid provider to authenticate your api connection17:15
dhellmannyes, that's true17:16
*** sdague has joined #openstack-infra17:16
*** daemontool has quit IRC17:16
fungimordred: jeblair: ^ do you remember if we ever found a way around that, to avoid having to ssh into the server and use a text-based web browser to step through the openid authentication and save the token?17:17
jeblairfungi: i recall doing it on my workstation and copying the resulting files17:17
jeblairthis was 5 years ago?  :)17:17
fungibut we're not using private hiera for that at the moment i guess17:18
jeblairfungi: we can probably stick it in hiera and use it on both servers?17:18
fungii can probably put the token in hiera17:18
fungiyeah17:18
fungithat17:18
*** HeOS has quit IRC17:18
fungidhellmann: lemme noodle on it a little, but i'm pretty sure i can just reuse some of what we have in puppet for gerrit/jeepyb17:19
dhellmannfungi : ok, sounds good17:19
*** davideagnello has quit IRC17:20
*** dizquierdo has joined #openstack-infra17:20
dhellmannhmm, the tarball signing job for that test release failed http://logs.openstack.org/c4/c4ef0ccb90a827366b71e935994fb974c8f8950e/release/release-test-tarball-signing/1182ccd/console.html17:20
*** electrofelix has quit IRC17:21
*** gomarivera has quit IRC17:22
openstackgerritMerged openstack-infra/nodepool: Allow extra time for initial ZooKeeper connection  https://review.openstack.org/35174517:22
*** signed8bit_Zzz is now known as signed8bit17:22
fungidhellmann: interesting, it's erroring on a script that we also use for pypi uploads, though looks like maybe the logic in it has a bit of a hole17:22
*** Sukhdev has joined #openstack-infra17:22
fungiit's trying to use len(sys.argv) as a bool, but i don't think that will ever be false17:23
dhellmannand I'm getting a "relay not permitted" error from the announce email script17:23
openstackgerritAndreas Jaeger proposed openstack-infra/bindep: Add bindep.txt as default filename  https://review.openstack.org/35018417:24
dhellmannso all of the steps are being run, and we have a few pieces missing to make them work17:24
dhellmannthis is definitely progress17:24
AJaegerkrotscheck, jeblair, mordred, fungi: Updated 350184 ^ - the bindep.txt chnage17:24
dhellmannannounce log: http://logs.openstack.org/c4/c4ef0ccb90a827366b71e935994fb974c8f8950e/release/release-test-announce-release/47980fb/console.html17:24
fungishould probably be "if len(sys.argv) != 1" instead of "if not len(sys.argv)"17:24
dhellmannfungi : that's right, it should do "if sys.argv[1:]"17:24
*** karthikp_ has quit IRC17:24
krotscheckAJaeger: woot17:24
krotscheckAJaeger: You should go have a weekend.17:24
dhellmannfungi : or what you said, sure17:24
fungisomething like that, yeah17:24
dhellmannoh, the relay issue is probably because I'm sending to myself and not the real mailing list17:25
dhellmannso that's not an issue17:25
AJaegerkrotscheck: let me try again ;)17:25
dhellmannI didn't want to spam the dev list with test announcements17:25
fungiindeed17:25
* AJaeger waves good-bye17:25
fungihave a good weekend, AJaeger17:25
jeblairShrews, mordred, pabelanger: re 351745 -- i the zk fixture should only exit once the server is actually running (it watches the logfile); in practice, that usually takes only 1 or 2 seconds.17:26
AJaegerthanks, fungi17:26
*** tosky has quit IRC17:27
jeblairdhellmann: we can make a test list if you want17:27
*** davideagnello has joined #openstack-infra17:27
dhellmannjeblair : nah, that's ok, the email sending part of this is already working17:28
jeblairk17:28
dhellmannjeblair : the only new parts are doing the bug updates and tags on this node17:28
dhellmannthe tag push worked great17:28
dhellmannthe bug update step is missing some software dependencies (the launchpad stuff)17:28
krotscheckmordred: Side question. DOes the auth_url feature care about keystone version path suffix? i.e. will both http://example.com:35357/ and http://example.com:35357/v2.0 work?17:28
mordredkrotscheck: it depends on the keystone - but any modern keystone should properly handle the unversioned auth_url17:29
krotscheckmordred: Ok. I'm doing design by documentation, trying to teach JS developers how to set up their clouds.yaml file.17:29
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Correct command line logic in pypi-extract-name.py  https://review.openstack.org/35185717:29
fungidhellmann: ^17:29
mordredkrotscheck: cool. people should use the unversioned url unless their keystone is too old/broken17:30
Shrewsjeblair: hrm, then perhaps the log string the fixture is looking for is incorrect in a newer version? either way, the change in 351745 can't hurt17:30
dhellmannfungi : you know, argparse is really really easy to set up17:30
mordredkrotscheck: although it's safe to blindly put in whatever the OS_AUTH_URL is from the openrc that horizon provides17:30
*** pradk has quit IRC17:30
fungidhellmann: yes, it is. now that we're not on python 2.6 anyway17:30
dhellmannfair17:31
mordredkrotscheck: so if they are using a cloud that is not in the set of known vendors, and they don't know how else to find the information, that is a source that is safe to consume from17:31
jeblairShrews: yes, i agree it's harmless.  just wanted to mention that in case the situation doesn't improve.17:31
mordredShrews: I rechecked a bunch of shade changes now that the occ pin landed17:31
fungidhellmann: i think in general most of the scripts in there don't end up using argparse unless they have >1 non-exclusive option to handle, but i wouldn't reject patches converting them to use it consistently17:32
fungi(for reasons such as this bug)17:32
pabelangerjeblair: Shrews: it I understand, ZookeeperServerFixture will wait up to 30 seconds to start?17:32
dhellmannfungi : ok, I'll put that on my "spare time" list17:32
*** piet has quit IRC17:32
Shrewsjeblair: noted. might need to hold a failed node to figure out what's happening17:32
Shrewsmordred: \o/17:33
jeblairpabelanger: yep; it usually either starts or fails within a few seconds.17:33
fungidhellmann: but now i'm back to figuring out why we don't normally trip that failure, and how we tripped it here17:33
*** Swami has joined #openstack-infra17:33
jeblairShrews: want me to set a trap now?17:33
*** pradk has joined #openstack-infra17:33
dhellmannfungi : keep in mind that this test project may be missing pieces that "real" projects have17:33
SpamapShm17:34
SpamapSour jjb just deleted all of our jenkins jobs17:34
SpamapS(running from master)17:34
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add gate-tripleo-quickstart-docs-ubuntu-xenial job  https://review.openstack.org/34908717:34
SpamapSI believe --delete-old is broken17:34
Shrewsjeblair: let's wait to see if my change has any impact first17:34
* SpamapS is investigating17:34
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add config support for ZooKeeper servers  https://review.openstack.org/35062317:34
*** jed56 has quit IRC17:35
*** amitgandhinz has quit IRC17:35
Shrewsjeblair: pabelanger: just rebased 623 ^^^ on master to see if it passes now17:35
*** amitgandhinz has joined #openstack-infra17:35
*** yamahata has joined #openstack-infra17:36
*** shashank_hegde has quit IRC17:37
fungiSpamapS: what other command-line options were you passing along with --delete-old?17:38
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: if we fail to send mail, the job should fail  https://review.openstack.org/35186017:38
*** tldr_ has quit IRC17:38
fungiSpamapS: wondering if there's an untested interaction between options17:38
pabelangerjeblair: is 351435 what you were thinking to fix our database issue with subnodes and updateStats function?17:39
ShrewsSpamapS: sounds like it needs an '--i-am-a-dummy' option, ala mysql  :)17:40
SpamapSjenkins-jobs -l debug --conf "/etc/jenkins_jobs/jenkins_jobs.ini" update --delete-old "/etc/jenkins_jobs/config"17:40
SpamapSthat's the command we use17:41
SpamapSand what it does is, it creates all the jobs, then deletes them all17:41
SpamapSwhich is.. really.. really awesome17:41
mordredSpamapS: you want it to create them and then not delete them?17:41
fungiSpamapS: sounds like it's probably a very recent regression. had you been using --delete-old previously?17:41
fungi"these are so old. some are from several minutes ago!"17:42
mordredis it possible that a file permission issue on the jjb cache could be causing jjb to think the jobs in question are not ones that it wants?17:42
openstackgerritMerged openstack-infra/system-config: Update iLO IP for compute004.vanilla  https://review.openstack.org/35139517:42
openstackgerritMerged openstack-infra/system-config: Add compute002.vanilla metadata for better tracking  https://review.openstack.org/35133517:42
openstackgerritMerged openstack-infra/system-config: Add docs for firehose.o.o  https://review.openstack.org/35076417:43
SpamapSmordred: I was thinking that would be the best way17:43
*** dizquierdo has quit IRC17:43
SpamapSthis morning when our job ran, it deleted the job that runs jjb, which killed the job17:43
*** e0ne has joined #openstack-infra17:44
SpamapSso I have a pretty clear time frame when this happened.. some small things seem to have merged in that time frame17:44
mordredSpamapS: just for giggles, can you check permissions on your cache location and make sure the user running jjb can write to it?17:44
fungiSpamapS: there were only a couple changes merged in the past week, both just a few hours ago17:44
jeblairpabelanger: yep17:45
SpamapSIt happened some time just before 05/Aug/2016:15:23:14 +000017:46
mordredoh - so ... https://review.openstack.org/#/c/319616/917:46
mordreddarragh's comment seems related: "If someone was to call delete_old_managed() without any arguments, after calling update_jobs() they might have a reasonable expectation that it doesn't go a delete the jobs they just updated. It seems a bit of an inconvenience to have to supply the information on what jobs were updated."17:46
SpamapSI have some meatspace urgent issues to attend to.. back in 1017:46
mordredSpamapS: also, I take it that you are running jjb from master?17:47
fungiwaynr: zxiiro: zaro: looks like https://review.openstack.org/319616 might warrant an emergency revert17:47
mordredSpamapS: I think they're working towards a 2.0 ... it might be safer to run the latest release17:47
openstackgerritMonty Taylor proposed openstack-infra/jenkins-job-builder: Revert "Move 'update' yaml parsing to JenkinsJobs.execute."  https://review.openstack.org/35186417:48
* mordred submitted the revert patch for consideration ^^17:48
fungiwaynr: zxiiro: zaro: ^17:48
fungielectrofelix seems to have knocked off for the weekend17:48
*** rbrndt has quit IRC17:49
fungikien-ha: ^ you reviewed the original too17:49
mordredI put in SpamapS command into the change for reference17:49
*** eharney has quit IRC17:49
*** ganesan has quit IRC17:49
*** claudiub has quit IRC17:50
*** baoli has joined #openstack-infra17:50
fungiwould have bitten us as well if we hadn't already moved off jenkins17:50
fungiour puppet exec looked similar to that17:50
fungiquite possible others running a similar ci are hitting it now as well17:51
openstackgerritMonty Taylor proposed openstack-infra/puppet-openstackci: Pin jjb to 1.6.1  https://review.openstack.org/35186517:51
jeblairasselin, fungi: maybe it's worth... heh, that^17:51
mordredI recommend we recommend people pin17:52
jeblairi recommend we follow mordred's recommendation17:52
fungithat looks like a programmatic way to recommend that17:52
mordredthat would also give the 2.0 effort a bit of space to get 2.0 up and happy17:52
*** sarob has joined #openstack-infra17:53
*** mriedem_lunch is now known as mriedem17:53
* krotscheck reccomends that we recommend people follow jeblair's reccomendation.17:53
kien-hafungi: I believe zxiiro and electrofelix will be back soon17:53
mordred\o/17:54
openstackgerritThomas Goirand proposed openstack-infra/project-config: Fix gerrit/projects.yaml for deb-auto-backports  https://review.openstack.org/35186617:54
mordredI just got a spam with the subject: "Tactical Lasers - 65% Off"17:54
fungimordred: are you sure it's spam?17:54
mordredfungi: NO17:54
zigopabelanger: mordred: That should fix the not created git repo.17:54
mordredfungi: I clearly need some disount tactical lasers17:54
Shrewsmordred: with shark-approved head straps?17:55
mordredShrews: definitely17:55
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Ansible launcher: Pin JJB to 1.6.1  https://review.openstack.org/35186717:56
jeblairnot urgent, but while we're thinking about it ^17:56
SpamapSmordred: indeed, I think we'll just pin to 1.6.117:57
mordredSpamapS: cool17:57
greghayneshaha, I just proposed that same patch :)17:57
*** xnox has joined #openstack-infra17:58
*** weshay has quit IRC17:58
*** sarob has quit IRC17:59
*** sarob has joined #openstack-infra17:59
greghaynes(internally)18:00
*** ddieterly is now known as ddieterly[away]18:00
waynryeah i missed that comment, too much going on18:01
*** weshay has joined #openstack-infra18:01
waynr(not in the JJB 2.0 api review process, but between $dayjob and 2.0 api review)18:01
SpamapSIMO the thing that is missing is functional test coverage in the gate18:02
waynrand people talking loudly about pimped out cars next to my desk18:02
SpamapSWe're humans, we miss things.18:02
*** shardy is now known as shardy_afk18:03
*** sandanar has quit IRC18:03
*** tonytan4ever has quit IRC18:03
*** eharney has joined #openstack-infra18:03
openstackgerritMiles Gould proposed openstack/diskimage-builder: Allow installing proliant tools from an OS package  https://review.openstack.org/34421418:04
*** Na3iL has quit IRC18:04
asselinjeblair, fungi mordred, yeah that was proposed for 3rd party ci folks here: https://review.openstack.org/#/c/348035/4/contrib/single_node_ci_data.yaml18:05
waynrzaro: zxiiro I think we should merge this: https://review.openstack.org/#/c/35186418:05
*** e0ne_ has joined #openstack-infra18:07
*** e0ne has quit IRC18:07
fungiasselin: so i guess 351865 is unlikely to make any difference for anyone using single_node_ci_data.yaml?18:08
waynrwhile i'm glad someone caught this bug before release, i don't think it's a very good idea to run from a development head of a repo in a production system18:09
timrc++18:09
*** piet has joined #openstack-infra18:09
timrcPin all the things (in production) is core to my belief system... but that would imply you have some way of testing "latest" (e.g. you have a staging env)18:10
jeblairwaynr: that is a change from the practice we have recommended with jjb so far.  that's fine with me if you want to change it -- i just want to make it clear it is a change18:10
*** gouthamr has quit IRC18:10
jeblairtimrc: i don't agree with that, but that's okay.  :)18:10
asselinfungi, it might still if they didn't 'manually' pin in their own hiera config.18:10
timrcjeblair: I mean... if you care about uptime...18:11
jeblairtimrc: we will absolutely continue to run zuul and nodepool as tip of master in openstack's production env.18:11
waynrjeblair: ah, i didn't realize that was being recommended18:11
SpamapSwaynr: we run from the development head of all of openstack.. so...........18:11
phschwartzwaynr: I am more concerned that things like this are making it into the repo. They should be caught/fixed during the review process.18:11
jeblairtimrc: i care about continuous deployment18:11
*** acoles is now known as acoles_18:11
*** karthik__ has joined #openstack-infra18:11
timrcjeblair: You can achieve that by deploying to staging first in a continuous way and then once validated, deploy to production.18:11
asselinfungi, I didn't send any e-mail about it to the ML. I guess there's enough need to warrant that now.18:11
jeblairtimrc: yes you can.  i won't be doing that.18:11
waynrphschwartz: bugs are a risk associated with software development in general18:11
timrcSure there's the "push button" on staging/production boundry18:12
waynrbut yes this was a pretty stupid bug if that's the point you're trying to make18:12
timrcjeblair: :) Oh, I know18:12
phschwartzwaynr: My point is maybe there hasn't been enough oversite lately with jjb and things are being rushed to merge.18:13
fungiwaynr: i don't personally think that it's a stupid bug, just that it's a case worth testing which wasn't previously tested18:13
jeblairfungi: ++18:13
notmorganfungi: as are most bugs18:13
*** eharney_ has joined #openstack-infra18:13
fungii can totally see how that could have broken (granted, electrofelix also spotted it in review and maybe should have pressed for adding a test for that scenario)18:14
jeblairanyway, if the jjb folks want to have room to make master unstable, i think that's fine and folks can pin to the release (or, v2 work could happen in a feature branch)18:14
timrcphschwartz: I dunno, hopefully we can create systems that support rapid change.18:15
*** eharney has quit IRC18:15
SpamapSI don't thing bugs are stupid. I do think we were a little silly for not running our own tests for something we run from master. ;)18:15
waynri also don't think these patches are necessarily being rushed to merge so much as we were unaware of the impact that merging questionable code into master would have on others using JJB18:15
*** shashank_hegde has joined #openstack-infra18:16
fungiall code is questionable18:16
SpamapSyep, it's our fault for not having tests for our use case18:17
SpamapSso don't think anybody here is blaming jjb for this :)18:17
timrcwaynr: FWIW I really appreciate all the activity around jjb.  Stuff happens, we learn from it, we fix it, we move on :)18:17
SpamapSAnd, we're pinning to 1.6.1 now18:17
SpamapSbecause we don't actually want to write those tests. ;)18:17
waynrha18:17
SpamapSThough I do kind of wonder...18:18
SpamapSwould we have exploded when we unpinned to 2.0?18:18
SpamapSnot sure how one gets their use case tested18:18
waynrpatches are definitely welcome but I am making a note in the 2.0 API etherpad to add a test for that particular case18:18
*** gouthamr has joined #openstack-infra18:18
waynr(when we fix and re-submit the original patch)18:18
*** eharney_ has quit IRC18:19
phschwartzwaynr: which reminds me, have you revisted that review I put up fixing the config issue faced by people not passing --conf and expecting to get the standard conf files used.18:19
pabelangerzigo: do you need to import an existing repo?18:20
pabelangerzigo: if so, you need to use upstream:18:20
zigopabelanger: Yes.18:20
zigoFixing.18:20
pabelangerjeblair: great, thanks18:20
waynrphschwartz: it's in my browser now, looks like it causes XML output to change: http://logs.openstack.org/84/347984/3/check/jenkins-job-builder-compare-xml/f6f72c2/18:20
*** davideagnello has quit IRC18:21
openstackgerritThomas Goirand proposed openstack-infra/project-config: Fix gerrit/projects.yaml for deb-auto-backports  https://review.openstack.org/35186618:22
*** _nadya_ has joined #openstack-infra18:22
jlkQuick poll, does anybody care that the Blue Box cloud you use is named "MontyTaylor" ?18:23
*** daemontool_ has quit IRC18:23
pabelangeris OverloadMontyTaylor an option?18:24
timrcpabelanger: You may get -EMONTY...18:25
jeblairjlk: not really -- we mask that and call it 'bluebox-sjc1' in anything public facing18:25
jeblairjlk: if you wanted to change it tho, it would not be a problem18:26
jlkI don't want to change it18:26
phschwartzwaynr: hmm, not sure why that is failing. The compare-xml-new never returns a response.18:26
*** mctaylor has joined #openstack-infra18:26
jlkbecause changing endpoints and such is lame.18:26
pabelangerI'm fine with status quo, makes bringing the cloud online easier IMO18:27
jeblairjlk: then likely not worth it :)18:27
*** senk has joined #openstack-infra18:27
fungijlk: i think it will lead to much amusement. let's not rename ;)18:28
fungi"has anybody noticed that MontyTaylor is full of failing nodes this morning?" "did you see how fast that devstack job ran in MontyTaylor?"18:29
jlkhaha18:29
fungithe possibilities are endless18:29
pabelanger:)18:29
jlkmontytaylor has fallen over again.18:29
jlkCompletely exhausted.18:29
fungithankfully that doesn't happen like it used to18:30
openstackgerritMerged openstack-infra/infra-specs: Initial commit of the Pholio spec  https://review.openstack.org/34064118:30
jlkToo bad we didn't name it mordred so that his IRC gets pinged every time we talk about it.18:30
*** davideagnello has joined #openstack-infra18:30
fungii already get amused whenever i update a ticket in one of our rackspace accounts and see support address us as "monty" (mordred) or "jonathan" (jbryce) or "paul" (pvo) depending on the account18:31
waynrphschwartz: i'm working on an update to your patch right now18:32
*** eharney_ has joined #openstack-infra18:33
jlkoh pvo.18:34
*** _ari_ has quit IRC18:34
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Correct configuration file usage.  https://review.openstack.org/34798418:35
*** kzaitsev_mb has joined #openstack-infra18:35
waynrphschwartz: does that address your problem? ^^18:37
*** roxanaghe has joined #openstack-infra18:38
waynri can try to remember to take another look tonight if not, but i really can't split my attention between work i am being paid for (and have to stay later to make progress on) and JJB anymore today18:39
phschwartzwaynr: that looks good.18:39
*** gomarivera has joined #openstack-infra18:40
*** sputnik13 has joined #openstack-infra18:41
*** senk has quit IRC18:44
*** gomarivera has quit IRC18:44
*** rbrndt has joined #openstack-infra18:45
*** roxanaghe has quit IRC18:46
*** e0ne_ is now known as e0ne18:46
*** xnox has quit IRC18:47
*** PollyZ has joined #openstack-infra18:49
*** tongli has quit IRC18:51
*** electrofelix has joined #openstack-infra18:52
*** xnox has joined #openstack-infra18:52
*** _nadya_ has quit IRC18:52
*** amitgandhinz has quit IRC18:53
*** esberglu has quit IRC18:53
*** amitgandhinz has joined #openstack-infra18:54
*** PollyZ_ has joined #openstack-infra18:54
openstackgerritKevin Daniels proposed openstack-infra/jenkins-job-builder: Add support for xcode plugin  https://review.openstack.org/35188318:55
jrollfungi: curious if the pvo accounts get a faster response :)18:55
fungijroll: hard to tell, honestly18:56
fungiwe don't open as many tickets under that account18:56
jrollah18:56
*** PollyZ has quit IRC18:57
*** cody-somerville has joined #openstack-infra18:58
*** cody-somerville has joined #openstack-infra18:58
zxiirosorry was out for lunch, catching up on chat now18:58
*** jimbaker has quit IRC18:58
*** ddieterly[away] has quit IRC19:00
*** jimbaker has joined #openstack-infra19:02
*** jimbaker has joined #openstack-infra19:02
*** fifieldt has quit IRC19:02
*** xnox has quit IRC19:03
*** roxanaghe has joined #openstack-infra19:03
*** karthik__ has quit IRC19:04
*** electrofelix has quit IRC19:04
*** jaosorior has quit IRC19:04
*** ddieterly has joined #openstack-infra19:05
*** baoli has quit IRC19:07
zxiiroouch that's a bad bug, sorry to hear we caused someone to lose their jenkins jobs19:08
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add labels panel to nodepool dashboard  https://review.openstack.org/35188619:08
*** xnox has joined #openstack-infra19:09
*** esberglu has joined #openstack-infra19:10
pabelangerfungi: I think you were talking about node type in grafana this week. ^should address that. cc jeblair19:12
pabelangeradds another panel to display that info19:12
*** sarob has quit IRC19:13
jeblairpabelanger: ++19:14
openstackgerritZane Bitter proposed openstack-infra/git-review: Clarify that submitting multiple commits is OK  https://review.openstack.org/35188819:14
*** eharney_ is now known as eharney19:16
*** roxanaghe has quit IRC19:18
*** fifieldt has joined #openstack-infra19:19
*** xnox has quit IRC19:19
*** piet has quit IRC19:20
*** karthik__ has joined #openstack-infra19:20
*** rbuzatu has quit IRC19:21
*** rbuzatu has joined #openstack-infra19:21
*** thorst__ has joined #openstack-infra19:23
openstackgerritMerged openstack-infra/tripleo-ci: Don't run undercloud idempotent on tempest jobs  https://review.openstack.org/35106619:23
openstackgerritMathieu Gagné proposed openstack/os-client-config: Update Internap information  https://review.openstack.org/35188919:24
*** burgerk_ has joined #openstack-infra19:25
*** thorst_ has quit IRC19:26
*** thorst__ is now known as thorst_19:26
*** xnox has joined #openstack-infra19:26
*** sdake has quit IRC19:26
zaromtreinish: would you have time to take another look at the gerrit plugin to save test metadata?19:27
*** sdake has joined #openstack-infra19:28
*** burgerk has quit IRC19:29
*** e0ne has quit IRC19:29
*** roxanaghe has joined #openstack-infra19:30
*** cody-somerville has quit IRC19:31
*** davideagnello has quit IRC19:33
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update cobertura to use convert xml  https://review.openstack.org/35189119:33
openstackgerritJeremy Stanley proposed openstack-infra/zuul: Cloner: Update heads and tags from the cache  https://review.openstack.org/35063719:34
*** davideagnello has joined #openstack-infra19:34
*** shashank_hegde has quit IRC19:36
*** xnox has quit IRC19:37
mtreinishzaro: do you have a link?19:41
*** baoli has joined #openstack-infra19:42
jeblairfungi: is there a reason not to use the same format for the fetch calls we just added here?  http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/merger/merger.py#n18119:43
jeblair(side question, i don't immediately see why we couldn't actually use the Repo() class there even....19:44
fungijeblair: oh, i missed those19:44
*** xnox has joined #openstack-infra19:44
jeblairbut that's of secondary importance)19:44
fungiyep, now i remember when we added those19:45
jeblairfungi: so, specifically, i'm wondering if: 1) the '+refs:refs' bit is unecessary (my guess is yes?) 2) i dunno about update_head_ok, maybe that is a difference we need?.  3) since that bit of code is getting complicated, is it possible to use the Repo class instead.19:46
zaropleia2: weekend is upon us.  can i interst you to review these gerrit backports? https://review.openstack.org/#/q/status:open+project:openstack-infra/gerrit+branch:openstack/2.11.4+topic:gerrit-upgrade19:46
fungijeblair: yeah, i think we can reuse some of this. really i just want to make sure that we fetch heads and tags from the cache, so we're only subsequently fetching new heads and tags from the remote when they weren't present in the cache19:47
fungiit dawned on me that the fact that we jump through hoops to pre-cache all teh heads and tags is pointless with zuul-cloner just doing a straight clone from the cache19:48
*** thiagolib has quit IRC19:48
jeblairfungi: yep, i'm down with the improvement.19:49
fungijeblair: we could probably reuse merger.Repo, i agree19:49
openstackgerritMonty Taylor proposed openstack-infra/shade: Add ability to upload duplicate images  https://review.openstack.org/34971019:50
openstackgerritMonty Taylor proposed openstack-infra/shade: Add a 'meta' passthrough parameter for glance images  https://review.openstack.org/35011419:50
fungithough will take a bit more refactoring than i expect to find time for this afternoon, so i'll wip this and make a note19:50
asselinCan I get a quick review on this (simple) change: https://review.openstack.org/#/c/344336/19:53
asselinfor openstack-infra/ansible-role-puppet19:54
*** roxanaghe has quit IRC19:55
openstackgerritJeremy Stanley proposed openstack-infra/puppet-gerrit: Ensure git cronjob absent for correct user  https://review.openstack.org/35182019:56
fungipabelanger: ^ that one had angry rocketships19:56
fungipuppet-lint now also seems to object when you have extra spaces before your rockets, even if they're aligned19:57
fungithough it doesn't complain if there's only one rocketshipped line with extra space19:57
pabelangerfungi: +219:57
anteayathank goodness for lint19:57
fungidunno how i'd ever survive without it looking after my spacebar19:58
anteayaha ha ha19:58
*** clenimar_ has quit IRC19:58
fungiend spacebar abuse now!19:58
anteayaha ha ha19:58
mordredfungi: you never want the wrong amount of space aroud your rocketship19:58
jeblairmordred: truer words have never been said19:58
anteayahurry you can make the weekend protest schedule19:58
fungimordred: unless you only have one19:58
fungiand then it's apparently okay19:58
fungi(more likely an oversight the whitespace fascists will address as soon as they find out)19:59
*** megm has quit IRC19:59
*** xnox has quit IRC19:59
*** xnox has joined #openstack-infra19:59
anteayaasselin: I only have +1 on ansible-role-puppet, but you have it19:59
*** rossella_s has quit IRC20:00
asselinanteaya, thanks20:00
anteayawelcome, your patch makes sense to me20:00
*** rossella_s has joined #openstack-infra20:00
fungimordred: jeblair: you may be interested in 351816 and 351820 which i'm surprised haven't caused more corruption already but we should fix before 04:07 utc20:01
*** megm has joined #openstack-infra20:01
*** ddieterly is now known as ddieterly[away]20:02
fungiit seems to probably be the culprit for the puppet-openstackid repo breaking on git04 a week ago and now git08 today20:02
openstackgerritMerged openstack-infra/ansible-role-puppet: Honor show-diff options with puppet apply  https://review.openstack.org/34433620:03
fungisomehow that repo has won the race twice and been killed for its trouble20:03
anteayahopefully nevermore20:03
anteayathanks jeblair20:04
openstackgerritMerged openstack-infra/jenkins-job-builder: Move 'delete' yaml parsing to JenkinsJobs.execute.  https://review.openstack.org/26295420:04
openstackgerritJames E. Blair proposed openstack-infra/zuul: Correct default value for hold-following-changes  https://review.openstack.org/35072520:04
openstackgerritJames E. Blair proposed openstack-infra/zuul: Tests: Improve support in fake builds  https://review.openstack.org/35072620:04
openstackgerritJames E. Blair proposed openstack-infra/zuul: Restore test_parallel_changes  https://review.openstack.org/35072720:04
openstackgerritJames E. Blair proposed openstack-infra/zuul: Remove unused Inheritable class  https://review.openstack.org/35072820:04
openstackgerritJames E. Blair proposed openstack-infra/zuul: Tests: remove FakeWorker  https://review.openstack.org/35073020:04
openstackgerritJames E. Blair proposed openstack-infra/zuul: Tests: ensure fake builds recorded in order  https://review.openstack.org/35189720:04
openstackgerritMerged openstack-infra/jenkins-job-builder: Move subcommand logic into subcommand classes.  https://review.openstack.org/31961720:05
openstackgerritMerged openstack-infra/jenkins-job-builder: Don't expose YamlParser methods unnecessarily.  https://review.openstack.org/31961820:05
openstackgerritMerged openstack-infra/jenkins-job-builder: TestSubCommand inherit from UpdateSubCommand.  https://review.openstack.org/31961920:05
jeblairi inserted a new change in that stack that should address the random test failures20:05
openstackgerritMerged openstack-infra/jenkins-job-builder: Remove YamlParser references from JJB modules.  https://review.openstack.org/31962020:05
openstackgerritMerged openstack-infra/jenkins-job-builder: Remove deprecated 'extended-choice' property.  https://review.openstack.org/31962120:06
*** amitgandhinz has quit IRC20:07
*** Apoorva has quit IRC20:07
*** ddieterly[away] is now known as ddieterly20:07
*** ddieterly is now known as ddieterly[away]20:08
*** _nadya_ has joined #openstack-infra20:08
*** gomarivera has joined #openstack-infra20:09
*** xnox has quit IRC20:10
*** _nadya_ has quit IRC20:10
*** cody-somerville has joined #openstack-infra20:10
krotscheckHey, anyone around who can de-queue/kill a job for us?20:11
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update Build Name Setter plugin to use convert xml  https://review.openstack.org/35189920:12
anteayakrotscheck: which patch/job?20:12
krotscheckanteaya: The release job for the 4.0.0 tag on eslint-config-openstack20:13
* krotscheck is trying to find it.20:13
*** rbuzatu has quit IRC20:14
krotscheckanteaya: I don't see it in zuul.20:14
*** gomarivera has quit IRC20:14
Shrewsjeblair: can you re-approve https://review.openstack.org/350623 for us?20:14
Shrewsjeblair: i rebased it to get the timeout change20:14
anteayaI don't see anything in the tag pipeline in zuul20:15
*** Apoorva has joined #openstack-infra20:15
jeblairShrews: done20:15
krotscheckanteaya: I don't either.20:15
anteayais this job running on a specific patch?20:15
krotscheckanteaya: I do see the remote tag in gerrit though20:15
anteayacan you share what you are seeing?20:16
anteayaso far I don't have a reference for what you are asking help for20:16
*** xnox has joined #openstack-infra20:16
krotscheckanteaya: http://paste.openstack.org/show/550675/20:16
fungianteaya: krotscheck: http://logs.openstack.org/13/13a4e1f9ab9a0d5be3abece054ce3eaab96a396b/20:16
*** thorst__ has joined #openstack-infra20:16
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Add documentation on howto consume events from firehose.o.o  https://review.openstack.org/35190020:16
mtreinishfungi: ^^^ I had a spare sec so I wrote up the client side docs20:17
krotscheckfungi, anteaya: Thanks! I'm walking betherly through the release process, and we accidentally incremented once too often.20:17
fungimtreinish: oh, thank you! my last 48 hours have been less conducive than i expected20:17
betherlythanks for your help sorting this krotscheck, fungi, anteaya :) sorry!!!20:17
anteayano no worries looks like from the timestamp the job finished on its own20:18
anteayakudos for taking the time to mentor someone krotscheck :)20:18
*** thorst_ has quit IRC20:18
anteayagood learning betherly20:18
mtreinishfungi: no worries, I figured that was the case :)20:18
openstackgerritMerged openstack-infra/system-config: Ensure git cronjob absent for correct user  https://review.openstack.org/35181620:18
krotscheckanteaya: Yeah, that job finishing is not a good thing. We skipped a version we didn't want to skip.20:19
fungikrotscheck: betherly: that 13a4e1f9ab9a0d5be3abece054ce3eaab96a396b is the result of `git show-ref 4.0.0` in that repo, or looking at the parenthetical sha shown at http://git.openstack.org/cgit/openstack/eslint-config-openstack/tag/?h=4.0.020:19
krotscheckIt's not a huge problem.20:19
anteayakrotscheck: ah well I think we missed the boat on being able to stop it20:19
krotscheckRight, so do we care enough to unpublish the package from npm.20:20
krotscheckHrm - it's not published to npm yet.20:20
anteayaI personally do not20:20
anteayahowever this is the first I have heard of this repo20:20
anteayaso I have zero emotional investment in its tagging history20:20
jeblairit skipped the upload because the tarball build failed20:21
anteayaso I might be the wrong person to ask20:21
openstackgerritMerged openstack-infra/nodepool: Add config support for ZooKeeper servers  https://review.openstack.org/35062320:21
fungiyeah, it seems it tried to checkout the tag from the zuul-merger, which didn't have it on hand20:21
*** davideagnello has quit IRC20:22
*** baoli has quit IRC20:23
*** _nadya_ has joined #openstack-infra20:23
openstackgerritMerged openstack-infra/shade: Remove discover from test-requirements  https://review.openstack.org/34580620:23
krotscheckfungi: Was that in response to my thread?20:23
fungikrotscheck: yes20:23
openstackgerritMerged openstack-infra/shade: Move list_magnum_services to OperatorCloud  https://review.openstack.org/34439520:23
krotscheckfungi: Hrm. So the actual publish failed? That's a little alarming.20:23
*** xnox has quit IRC20:23
fungikrotscheck: yes, the gerrit-git-prep builder failed to find the tag, and so the job bailed out at that point20:24
*** _nadya_ has quit IRC20:24
krotscheckfungi: Ok, so help me here. When we want to do a release, we push a tag, but that tag wasn't available, so... is there an order-of-operations problem here?20:25
fungikrotscheck: for reasons i don't yet know, http://zm08.openstack.org/p/openstack/eslint-config-openstack was missing a copy of the tag which should have been retrieved from gerrit before running the job (i think)20:25
krotscheckfungi: Got it. Sounds like we can't do anything right now until there's more information?20:26
krotscheck(We being myself and betherly)20:26
fungikrotscheck: yes, did you want that to get published to npm? i can reenqueue that ref20:27
betherlyfungi: krotscheck: so i ran 'npm version major' not the push tags part or git review etc etc20:27
krotscheckfungi: Well, could you just delete the ref for us in gerrit so we can try again?20:27
krotscheck(or can we do that?)20:27
openstackgerritMerged openstack-infra/shade: Stop creating cloud objects in functional tests  https://review.openstack.org/34439620:27
* krotscheck tries to push a tag delete.20:28
* krotscheck expects the world to explode20:28
*** diltram has joined #openstack-infra20:28
diltramhey20:28
betherlyfungi: because i didnt have gpg set up. so then once gpg was sorted i did the process again this time with the push tag part20:28
fungikrotscheck: no, deleting and recreating that tag would only make things worse. deletion or replacement of existing tags don't propagate via pull/fetch20:28
diltramhow is configured sending updates from review to launchpad?20:28
betherlycould that have been why v3.0.0 missed the tag?20:28
krotscheckbetherly: Naah, v3.0.0 failed because npm requires that you sign tags.20:29
krotscheck(as do we)20:29
fungidiltram: http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/update_bug.py20:29
krotscheckAnd you didn't have gpg set up20:29
betherlyok20:29
krotscheckfungi: In that case, just re-enqueue that ref.20:29
fungidiltram: you want to know how it works, or where to configure it in our system?20:29
diltramwhere to configure20:29
diltrambecause for octavia this is not working20:29
diltramI just uploaded a review and there is no info on launchpad about that20:30
fungidiltram: there are a number of potential reasons for it not to work. rather than trying to rule them all out, let's start with an example of the change you feel should have updated a bug20:30
rm_workI think sometimes it has to do with placement of the Closes-Bug: message20:30
rm_workhttps://review.openstack.org/#/c/265065/20:30
rm_work^^ that one20:30
*** davideagnello has joined #openstack-infra20:31
diltramexactly20:31
rm_workI see it happen a lot with changes in Octavia though... maybe it's just because we are bad at Gerrit :P20:31
diltramhahaha :P20:31
fungidiltram: rm_work: https://bugs.launchpad.net/octavia shows your bug supervisor is "michael johnson"20:32
fungithat's the problem20:32
rm_workmichael johnson is the problem? :P20:32
diltramthe problem is with our PTL? ok changing :P20:32
*** dimtruck is now known as zz_dimtruck20:32
fungiwell, the fact that it's not a group containing the OpenStack Infra (hudson-openstack) lp account is the problem20:33
anteayahttp://docs.openstack.org/infra/manual/creators.html#set-up-launchpad20:33
*** ddieterly[away] is now known as ddieterly20:33
rm_workcan't find a user named "octavia-bugs"20:34
fungiif our automation account isn't in your bug supervisor group, it lacks sufficient permissions to reassign a bug, so if the bug update would also involve reassignment (for example, initial change upload and the bug is not yet assigned to the change uploader) then it will fail to reassign the bug due to insufficient permissions and bail out before commenting20:34
rm_workmaybe I skipped a step20:34
anteayarm_work: which is why I found the set up launchpad docs: http://docs.openstack.org/infra/manual/creators.html#set-up-launchpad20:34
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection listener  https://review.openstack.org/35191020:34
diltramok, I'm gonna send that document to Michael to update that stuff, thx fungi :)20:34
rm_workah, the note20:34
rm_workanteaya: yeah that is what I was reading20:34
anteayagreat20:35
anteayarm_work: if you don't find the solution in there, please send a patch to the doc20:35
*** Jeffrey4l has quit IRC20:35
*** Jeffrey4l has joined #openstack-infra20:36
fungikrotscheck: confirmed, zm08 still seems to not have the 4.0.0 tag in its copy of the repo. checking logs to see if i can figure out why20:36
anteayafungi: also I'm not sure we can do anything about it but bknudson is finding spam where he is expecting keystone bug reports20:36
krotscheckfungi: Thanks. Does the review have to land for that to get propagated?20:37
fungianteaya: i think lp has a "report as spam" link for bug comments20:37
anteayafungi: ah thank you20:37
fungikrotscheck: what review?20:37
fungikrotscheck: did you push a tag for a commit which isn't in the branch yet?20:37
krotscheckfungi: I believe that may have happened, because communication breakdown.20:38
diltramthx fungi and CU people20:38
fungikrotscheck: oh... if so, that would completely explain this. i'm confirming now20:38
*** diltram has left #openstack-infra20:38
krotscheckfungi: We're working on getting the correct patch up20:38
*** shardy_afk is now known as shardy20:38
fungikrotscheck: betherly: for future reference, http://docs.openstack.org/infra/manual/drivers.html#tagging-a-release20:39
openstackgerritMonty Taylor proposed openstack-infra/shade: Add ability to upload duplicate images  https://review.openstack.org/34971020:39
openstackgerritMonty Taylor proposed openstack-infra/shade: Add a 'meta' passthrough parameter for glance images  https://review.openstack.org/35011420:39
fungikrotscheck: betherly: yep, mystery solved. i don't see the tagged commit showing up in the history of master (your only branch)20:40
*** r-daneel has joined #openstack-infra20:41
fungikrotscheck: betherly: in fact, i don't see the tagged commit showing up anywhere in gerrit, even proposed20:41
*** HeOS has joined #openstack-infra20:41
*** gouthamr has quit IRC20:41
krotscheckfungi: betherly is doing an interactive rebase right now, give her a sec.20:42
fungias a result, i recommend not using 4.0.0 as a version number at all. probably need 4.0.1 when you do push a tag for an actual merged commit20:42
fungithough it's _possible_ this is one of those rare cases where deleting and replacing the tag isn't a problem, since nobody could have fetched it20:43
betherlyfungi: so ive just done the rebase and review20:43
betherlyshould i amend the commit to 4.0.1?20:43
fungiit's probably safest to do that, yes20:43
fungikrotscheck: ^ ? 4.0.1 okay?20:43
krotscheckNo problems here.20:44
*** kien-ha has quit IRC20:44
openstackgerritMerged openstack-infra/puppet-openstackci: Pin jjb to 1.6.1  https://review.openstack.org/35186520:44
betherlydo i do that just by literally amending the file or do i need to do something in terminal too?20:44
betherlyfungi: ^20:44
*** gomarivera has joined #openstack-infra20:44
betherlykrotscheck: fungi: and im guessing change the commit message to 4.0.1 too?20:44
fungibetherly: i'm not a javascript expert, but it sounds like you need to edit package.json to say 4.0.1, propose that with git-review, wait for it to get approved and merge, then checkout the master branch tip and tag that20:45
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement TripleO centos-7-nonha-multinode-updates  https://review.openstack.org/35191420:46
fungibecause it's always possible in gerrit that another change races yours to merging, and then the head of master is no longer the same commit which was the local parent of your commit, so gerrit ends up needing to create a merge commit to stitch it into the branch20:46
*** thorst__ has quit IRC20:46
fungiso it's best to avoid tagging your commit until you know the branch state is updated and then tag the branch tip20:46
fungifrom the remote repo20:47
fungithat's why our instructions mention `git pull --ff-only`20:48
fungiyou never want to tag a possibly out of sync local copy of the branch20:48
*** burgerk_ has quit IRC20:49
betherlyfungi: ok that all makes sense. thanks for helping with this20:49
anteayaseems reporting launchpad spam is done here: https://answers.launchpad.net/launchpad by posting a question, like can you remove this spam please20:50
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Reinstall python-requests in bootstrap script  https://review.openstack.org/35052120:50
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Make $TRIPLEO_ROOT overrideable  https://review.openstack.org/35053920:50
*** thorst_ has joined #openstack-infra20:50
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Raise ping test count after multinode setup  https://review.openstack.org/35191620:50
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Fix resetting /etc/hosts  https://review.openstack.org/35191720:50
*** zz_dimtruck is now known as dimtruck20:52
*** csomerville has joined #openstack-infra20:53
*** shashank_hegde has joined #openstack-infra20:55
*** thorst_ has quit IRC20:55
*** vvalderrv has left #openstack-infra20:55
*** netsinn has joined #openstack-infra20:56
*** cody-somerville has quit IRC20:56
*** shardy has quit IRC20:58
*** netsin has quit IRC20:58
*** ddieterly is now known as ddieterly[away]20:59
*** jkilpatr has quit IRC20:59
*** kgiusti has left #openstack-infra21:00
*** gomarivera has quit IRC21:00
*** netsin has joined #openstack-infra21:01
*** mat128 is now known as mat128|weekend21:01
*** pradk has quit IRC21:03
*** netsinn has quit IRC21:04
*** psilvad has quit IRC21:04
*** gouthamr has joined #openstack-infra21:05
*** gordc has quit IRC21:07
*** amitgandhinz has joined #openstack-infra21:07
*** yamamoto has joined #openstack-infra21:09
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: WIP: Fix the integrated gate filtering on uncategorized  https://review.openstack.org/35192421:10
*** karthik__ has quit IRC21:12
*** karthik__ has joined #openstack-infra21:12
*** sdake has quit IRC21:12
*** ddieterly[away] is now known as ddieterly21:12
*** ldnunes has quit IRC21:12
*** dprince has quit IRC21:12
*** amitgandhinz has quit IRC21:12
*** mriedem has quit IRC21:13
*** geguileo has quit IRC21:13
*** rbuzatu has joined #openstack-infra21:14
*** yamamoto has quit IRC21:16
*** rbuzatu has quit IRC21:16
*** rbuzatu has joined #openstack-infra21:16
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Correct command wrapping in tarball signing script  https://review.openstack.org/35192721:17
fungidhellmann: ^ finally circled back around to the signing error from earlier and tracked down the issue21:17
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Correct command wrapping in tarball signing script  https://review.openstack.org/35192721:18
*** geguileo has joined #openstack-infra21:19
fungidhellmann: apparently the brokenness in the script was unnoticed because it still worked for projects whose package name matched their repo name, but in the case of the release-test repo/openstack-release-test package this was not the case21:19
*** rfolco_ has joined #openstack-infra21:21
*** Ravikiran_K has quit IRC21:21
*** rfolco_ has quit IRC21:21
*** rfolco_ has joined #openstack-infra21:21
*** rfolco_ has quit IRC21:21
rm_workfungi: weird, I didn't add it -- must have happened automatically when I set it as the owner21:22
*** shashank_hegde has quit IRC21:23
*** jkilpatr has joined #openstack-infra21:25
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add some documentation on functional testing tools  https://review.openstack.org/35193121:28
jeblairmordred, notmorgan, Shrews: ^ that's not complete by a longshot, but it's perhaps a start.  :)21:28
openstackgerritMonty Taylor proposed openstack-infra/shade: Add plumbing for betamax functional tests.  https://review.openstack.org/34439721:29
*** matt-borland has quit IRC21:30
openstackgerritMonty Taylor proposed openstack-infra/shade: Add plumbing for betamax functional tests.  https://review.openstack.org/34439721:31
mordredjeblair: woot!21:31
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add some documentation on functional testing tools  https://review.openstack.org/35193121:31
jeblairself-review :)21:31
*** rbuzatu has quit IRC21:35
*** bnemec is now known as beekneemech21:36
zigomordred: Hey, can you +2 workflow the last patch ? https://review.openstack.org/#/c/351866/21:38
openstackgerritRoopak Parikh proposed openstack-infra/project-config:  Adding first version of project mors  https://review.openstack.org/35193321:38
*** rbuzatu has joined #openstack-infra21:38
zigoThat way, I can get to build the backports over the week-end...21:39
*** davideagnello has quit IRC21:40
fungirm_work: yeah, i think lp auto-adds the group owner as an administrator member21:40
*** xnox has joined #openstack-infra21:41
*** edmondsw has quit IRC21:42
*** karthik__ has quit IRC21:44
*** mixpix3ls has joined #openstack-infra21:44
mixpix3lsHey folks, JJB question here. I'd like to reference the name of the job within the job definition.21:45
mixpix3lsIs that possible?21:45
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699521:46
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement non-ovb overcloud update job - Newton -> Newton  https://review.openstack.org/35133021:46
*** mctaylor has quit IRC21:48
zaromixpix3ls: you need to have configure a project example here: http://docs.openstack.org/infra/jenkins-job-builder/definition.html#project21:49
mixpix3lsWoah, I was just looking at that :)21:50
mixpix3lsThanks zaro:21:50
*** gomarivera has joined #openstack-infra21:52
*** davideagnello has joined #openstack-infra21:52
*** sdake has joined #openstack-infra21:54
*** karthik__ has joined #openstack-infra21:56
*** gomarivera has quit IRC21:56
*** yamamoto has joined #openstack-infra21:59
*** tdasilva has quit IRC21:59
*** rlandy has quit IRC22:00
*** mixpix3ls has quit IRC22:00
*** jcoufal_ has joined #openstack-infra22:00
*** timl0101 has quit IRC22:02
*** weshay has quit IRC22:02
mtreinishanteaya: if you get a sec can you take a look at: https://review.openstack.org/351900 it's the firehose doc patch for users22:03
*** jcoufal has quit IRC22:04
fungiooh, cool the draft is posted now22:04
fungias of a couple minutes ago22:04
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add Major upgrade function to the tripleo ci script  https://review.openstack.org/32375022:04
*** mdrabe has quit IRC22:05
fungior not... i thought we had a gate-system-config-docs job22:05
EmilienMhello there, tripleo folks is preparing some work to run > 2 nodes jobs. I was wondering what infra folks thinks about it, what would be the challenges, etc22:05
mtreinishfungi: it's there: http://docs-draft.openstack.org/00/351900/1/check/gate-system-config-docs-ubuntu-xenial/34213ac//doc/build/html/firehose.html22:06
fungimtreinish: thanks--i'm blind22:06
mtreinishfungi: looking at that I need to add syntax highlighting to the script22:06
mtreinishthat's the first thing that stands out :)22:06
*** signed8b_ has joined #openstack-infra22:06
fungimtreinish: yeah, you can specify the highlighting method22:07
fungias "python"22:07
fungishould do the trick22:07
fungi.. code-block:: python22:07
*** tpsilva has quit IRC22:08
openstackgerritMerged openstack-infra/project-config: Correct command line logic in pypi-extract-name.py  https://review.openstack.org/35185722:08
*** signed8bit has quit IRC22:09
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement TripleO centos-7-nonha-multinode-updates  https://review.openstack.org/35191422:09
jeblairfungi: http://docs-draft.openstack.org/00/351900/1/check/gate-system-config-docs-ubuntu-xenial/34213ac//doc/build/html/22:10
jeblairfungi: it's called gate-system-config-docs-ubuntu-xenial now22:10
*** msuriar has left #openstack-infra22:10
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Fix the integrated gate filtering on uncategorized  https://review.openstack.org/35192422:11
mtreinishfungi: ^^^ now with highlights22:11
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: upgrades job is renamed to updates  https://review.openstack.org/35193722:12
*** karthik__ has quit IRC22:13
*** karthik__ has joined #openstack-infra22:13
*** dimtruck is now known as zz_dimtruck22:14
*** zz_dimtruck is now known as dimtruck22:14
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699522:14
zigofungi: Can you help me finalize the backport repo stuff ? https://review.openstack.org/#/c/351866/22:14
*** jcoufal_ has quit IRC22:15
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton  https://review.openstack.org/35133022:17
*** larainema has quit IRC22:19
*** jcoufal has joined #openstack-infra22:20
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton  https://review.openstack.org/35133022:20
*** yamahata has quit IRC22:21
mtreinishfungi: heh, you should of called me out on hitting git review on the wrong terminal :)22:21
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Add documentation on howto consume events from firehose.o.o  https://review.openstack.org/35190022:23
*** dimtruck is now known as zz_dimtruck22:23
*** ddieterly has quit IRC22:23
*** _nadya_ has joined #openstack-infra22:25
*** karthik__ has quit IRC22:26
*** yamamoto has quit IRC22:26
*** tdasilva has joined #openstack-infra22:27
*** yamahata has joined #openstack-infra22:27
openstackgerritMerged openstack-infra/puppet-gerrit: Ensure git cronjob absent for correct user  https://review.openstack.org/35182022:28
*** gomarivera has joined #openstack-infra22:29
*** _nadya_ has quit IRC22:29
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699522:30
*** csomerville has quit IRC22:30
*** PollyZ_ has quit IRC22:32
*** kzaitsev_mb has quit IRC22:33
*** ljjjustin has quit IRC22:33
*** gomarivera has quit IRC22:33
*** rbuzatu has quit IRC22:33
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375022:34
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: Add conditional check for tripleo-test-cloud-rh1  https://review.openstack.org/34698522:35
*** jcoufal has quit IRC22:35
openstackgerritCorentin Ardeois proposed openstack-infra/project-config: Created DSVM Job for NPM Projects  https://review.openstack.org/34805622:37
*** ljjjustin has joined #openstack-infra22:37
*** jcoufal has joined #openstack-infra22:40
*** roxanaghe has joined #openstack-infra22:41
*** kzaitsev_mb has joined #openstack-infra22:43
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Fix the integrated gate filtering on uncategorized  https://review.openstack.org/35192422:44
*** hurgleburgler1 has joined #openstack-infra22:46
*** hurgleburgler has quit IRC22:47
*** gouthamr has quit IRC22:49
*** kzaitsev_mb has quit IRC22:51
EmilienMclarkb: hola, what are your thoughts if tripleo folks would propose a 4-nodes CI job?22:51
clarkbEmilienM: what are you trying to accomplish?22:52
EmilienMclarkb: deploying TripleO in multinode > 2 nodes22:52
EmilienMlike having an undercloud (1 node), an overcloud (1 controller, 1 compute, 1 storage)22:53
fungiare there common failure modes where an aio controller/compute/storage would be blind?22:53
clarkbah yes that was my next question, will this cover things in a way all in one cant?22:54
EmilienMwe are testing network isolation22:54
EmilienMand also pacemaker, with HA services22:54
EmilienMhaving such a scenario is really close to reality22:54
EmilienMmost of the bugs we find in production are multinode things, like networking, clustering, etc22:55
clarkbis the 2 node job working?22:55
EmilienMyes, very good22:55
clarkband does it not cover those bitd?22:55
EmilienMwe're super happy with it22:55
openstackgerritMerged openstack-infra/project-config: zuul/puppet: enable integration jobs on puppet-barbican  https://review.openstack.org/35173222:55
*** kzaitsev_mb has joined #openstack-infra22:56
clarkbbecausr you get network isolation and can test ha?22:56
EmilienMright now, the 2-nodes job deploys an undercloud and an overcloud (aio)22:56
EmilienMbut installing an overcloud on one node is like a single node devstack22:56
EmilienMyou don't test live migration22:56
EmilienMnetwork isolation, neutron tunneling, etc22:56
*** signed8b_ is now known as signed8bit_Zzz22:56
*** yamamoto has joined #openstack-infra22:57
clarkbneutron tunneling is actually the samr in aio as mulyinide if you dont dvr22:57
clarkbjust fyi22:57
*** rbrndt has quit IRC22:57
openstackgerritMerged openstack-infra/project-config: Fix gerrit/projects.yaml for deb-auto-backports  https://review.openstack.org/35186622:57
fungiplaying devil's advocate, nova already tests live migration across separate job nodes. why would their testing not be sufficient to keep live migration working in tripleo deployments?22:57
*** davideagnello has quit IRC22:58
EmilienMfungi: they test it in devstack22:58
clarkbmy only concern is that peoplr seem to think more nodes == more coverage22:58
EmilienMand tripleo/puppet CI have found a lot of interesting bugs that devstack didn't find.22:58
clarkbif you actually have a scenatio that requires more nodes then thats great we can probably do it22:58
fungii think the testing framework they use is irrelevant. devstack is just a means to and end. they're not testing live migration in devstack. they're testing live migration22:58
EmilienMright but the config is not the same22:59
fungithat's sort of like saying you need a job to run nova unit tests under nose because right now they're only tested under testr22:59
EmilienMtripleo CI deploys SSL and IPv6 by default22:59
EmilienMPuppet CI and TripleO CI deploy SSL and IPv6 in fact22:59
mtreinishfungi: they likely wouldn't work at all under nose... :)22:59
EmilienMit helped a lot over the last months22:59
fungimtreinish: shush ;)22:59
EmilienMyou can ask mtreinish how useful was Puppet CI over the last 6 months22:59
*** hurgleburgler1 has quit IRC23:00
EmilienMAFIK devstack doesn't deploy SSL & Ipv6 in gate23:00
*** itisha has quit IRC23:00
fungii'm sure testing the same thing in different ways does often expose different bugs, but might also be a sign that the existing tests need more work to cover some code paths23:00
EmilienMand this is a big problem because in reality, most of people deploy at least SSL, etc. That's an example23:00
jeblair"well, there's the problem" :)23:00
fungiwe've suggested in the past ipv6 and ssl should be the default test scenario, fwiw23:01
EmilienMI'm sure you did23:01
EmilienMbut we (puppet/tripleo) made it23:01
*** ajmiller_ has quit IRC23:02
EmilienManyway, just asking around about feedback on multinodes23:02
jeblairi guess it works now?  there shouldn't be any objection to adding it to devstack then23:02
*** asettle has joined #openstack-infra23:02
jeblairit used to "not work"23:02
mtreinishclarkb, fungi: any idea why this job has 2 statuses listed: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_uuid:%20%5C%22f656a52474d9441ebc124008f8cb65bc%5C%22%20AND%20error_pr:%5B%5C%22-1000.0%5C%22%20TO%20%5C%22-10.0%5C%22%5D%20&from=864000s23:02
EmilienMjeblair: ssl/ipv6 was an example23:03
mtreinishjeblair: the issue with turning it on in devstack wasn't that people didn't want it or it didn't work, but that the devstack support for those isn't really there. The neutron config in particular was in a very bad place23:03
mtreinishwe're only now getting to a place where lib/neutron is usable23:04
*** ams__ has quit IRC23:04
EmilienMjeblair: I (and some other folks) am trying to help TripleO project to move their jobs from Red Hat private cloud to use more upstream resources23:04
*** yamahata has quit IRC23:04
mtreinishclarkb, fungi: oh, there are 2 jobs listed there23:04
fungiluckily defining worker types with different numbers of nodes shouldn't require new images and the current devstack-gate implementation is generalized to support additional nodes beyond 2 (though i gather tripleo-ci doesn't use devstack-gate), but it does mean that we have n x min-ready nodes occupied waiting around for jobs that require that node count23:04
mtreinishI missed that, I just saw the double status23:05
*** PollyZ has joined #openstack-infra23:05
jeblairEmilienM: are we really all on board with that?23:05
EmilienMjeblair: and the second thing is IMHO tripleo jobs bring super useful feedback as we test things with production environment, with HA, SSL IPV623:05
jeblairEmilienM: is the tripleo cloud ready to accept jobs from other projects?23:05
*** pt_15 has quit IRC23:06
EmilienMjeblair: I'm not sure about the future of tripleo cloud right now.23:06
*** davideagnello has joined #openstack-infra23:06
fungiEmilienM: is tripleo prepared to integrate their scenarios into devstack-gate or are they stuck on using a separate framework?23:06
EmilienMpabelanger: if you around, you can certainly help here :)23:06
*** asettle has quit IRC23:06
pabelangerI think tripleo-test-cloud-rh1 could run other jobs today.23:07
pabelangerwe have AFS setup23:07
pabelangerand using DIBs now23:07
jeblairsee there was this mailing list thread a while ago: http://lists.openstack.org/pipermail/openstack-dev/2016-March/088988.html23:07
jeblairand it kind of went to a bad place with some folks getting really defensive23:07
jeblairwhich is unfortunate, because in my mind, we're now in a kind of awkward limbo23:07
EmilienMjeblair: you asked me if 'we' were all onboard with that. Ask pabelanger, he's working hard on it.23:08
pabelangerI still think tripleo-test-cloud-rh1 should be moved to third party23:08
fungiooh, i recognize that e-mail!23:08
pabelangerhowever, if they do exist in nodepool.o.o, there is nothing stopping the cloud from excepting more jobs today. We have everything in place now23:08
pabelangerexcept people agreeing to it23:09
*** amitgandhinz has joined #openstack-infra23:09
jeblairi like what you are doing with tripleo testing -- afaict, it's getting better and better which is great :)23:09
EmilienMthanks23:10
EmilienM(if it was addressed to us :))23:10
jeblairyep23:10
jeblairwe took on the tripleo cloud expecting to have multiple regions for stability, and, at the time, the tripleo project was the "official" deployment project23:10
jeblairbut *many* things have changed since then23:10
EmilienMso back to the request. I'm going to dress a list of benefits to have > 2 nodes jobs23:11
jeblairand based on the way it's currently operated, it better fits the third-party ci model (if tripleo were to stand this up today, we'd say "3rd party ci")23:11
betherlyhi all23:11
EmilienMand we'll make sure it's really worth it23:11
fungibetherly: welcome back23:12
betherlyready to do the final tagging for the release of eslint-config-openstack 4.0.123:12
fungibetherly: what commit are you tagging?23:12
betherlyfungi: https://review.openstack.org/#/c/351913/ has merged23:12
jeblairEmilienM, pabelanger: i feel it would be better as a third-party ci, but if there are plans for multi-region support and a willingness to make the platform available for all projects, that seems like it might be a good fit for upstream openstack infra23:12
pabelangerjeblair: the objections from moving tripleo-test-cloud-rh1 to 3rd party CI was it would result in the loss of voting (check) rights in the gate.23:13
pabelangerwhich, in fact, they do not have today23:13
betherlyso is it just a matter of checking out the master branch,  git tag 4.0.1, git push gerrit --tags23:13
fungipabelanger: it doesn't have those now, no23:13
*** rhallisey has quit IRC23:13
*** csomerville has joined #openstack-infra23:13
betherlyfungi: ^23:13
jeblairpabelanger: indeed, it will have the exact same privileges23:13
*** xarses has quit IRC23:13
jeblairit will just be an acknowledgement that it has a different set of operators than the rest of the system, which is also actually the case today :)23:14
pabelangerjeblair: right, this is what I was explaining as well. There was not technical reason they could not vote, but the lack of multi-region cloud was the problem23:14
*** amitgandhinz has quit IRC23:14
pabelangerBut, with that said. There is actually tripleo-test-cloud-rh2 online at the moment, which _could_ be the 2nd region23:14
pabelangerright now it is disabled23:15
fungibetherly: yeah, `git show-ref HEAD` should indicate it's 4549ae61e2ed19659c3d641ebd37ebdf12eb4d7f after you checkout master and pull --ff-only like in the example in our docs23:15
betherlyyup23:15
fungibetherly: `git tag -s 4.0.1` and `git push gerrit 4.0.1`23:16
pabelangerjeblair: but I agree, now that tripleo-jobs are starting to work across other clouds in the gate. There is no technical reasons why we couldn't enable more jobs in tripleo-test-cloud-rh1, even if its 10 nodes23:16
betherlyfungi: should i do that from the master branch?23:16
EmilienMpabelanger: exactly23:16
jeblairpabelanger: would it be okay for fuel to use it?23:17
betherlyfungi: apologies to double check everything. brain not entirely functional at this time so making sure i dont  make any more screw ups today lol23:17
fungibetherly: you want to make sure you use -s (which you didn't say above) and you need to push a specific tag name since --tags could push all sorts of other local tags you might have made in addition to the one you mean23:17
fungibetherly: the example at http://docs.openstack.org/infra/manual/drivers.html#tagging-a-release is the one to follow23:17
pabelangerjeblair: I personally don't have an objection.23:17
pabelangerjeblair: but that is likely a question for somebody at Red Hat23:17
betherlyfungi: ok so not in master then. create another branch which is a clone of master?23:18
jeblairpabelanger: heh, somebody else at red hat :)23:18
pabelanger:D23:18
fungibetherly: the example means "master" unless you're making a stable branch tag, and then "stable/something" instead23:18
betherlyok23:18
EmilienMI haven't seen efforts from Fuel to run their multinode jobs in osinfra afik23:18
fungibetherly: you don't need to make a separate branch for this23:19
pabelangerEmilienM: I have, there was people in Austin wanting to do it23:19
betherlyfungi: should the tag message just be the version?23:20
fungiEmilienM: fuel team members have expressed interest in running their jobs upstream, but right now a lot of their testing relies on bare-metal access and much larger multinode counts23:20
*** davideagnello has quit IRC23:20
fungibetherly: i usually look at earlier tags for the same repo, or if there are none then for other related repos, and try to keep the new tag message consistent with them23:20
fungibetherly: it could for example be something like "Release eslint-config-openstack version 4.0.1"23:21
openstackgerritMorgan Fainberg proposed openstack-infra/zuul: WIP: Re-enable test_connection tests  https://review.openstack.org/35194923:21
*** claudiub has joined #openstack-infra23:22
jeblairit feels like the tripleo cloud has been grandfathered in its current state for a while.  i'd just like to make sure we're being fair to everyone.  so if tripleo wants to run tripleo jobs, then i think we should move it to 3rd party ci.  i think that's a fine choice and we can continue to work together (please!) but with better division of reponsibilities.  or, if we want to revise the idea of a multi-provider hardware platform that's ...23:22
jeblair... available for all openstack projects, i'm game for that.  it would be great, but more work.23:22
betherlyfungi: trying to figure out how to view previous tags to keep it consistent. are they  viewable as just previous commits at some point?23:22
fungibetherly: `git tag` will list the names of existing tags, and `git show 2.0.0` will show that tag including its message23:23
fungibetherly: or you can click on them to browse them on https://git.openstack.org/23:24
fungibetherly: for example http://git.openstack.org/cgit/openstack/eslint-config-openstack/tag/?h=2.0.0 shows the tag message was just "2.0.0"23:24
betherlyfungi: i was literally about to type just that lol23:24
fungibetherly: whereas http://git.openstack.org/cgit/openstack-infra/bindep/tag/?h=2.0.0 shows a tag message of "Release bindep 2.0.0."23:25
betherlyfungi: would you say best practise would be to keep that consistent or make the next one more descriptive with you suggestion?23:25
betherlyie "Release eslint-config-openstack version 4.0.1"23:25
fungii tend to just keep it consistent23:25
*** Swami has quit IRC23:26
fungiif you switch to release management automation later you'll end up with all sorts of fun stuff injected into the tag message, for example http://git.openstack.org/cgit/openstack/python-novaclient/tag/?h=5.0.023:26
openstackgerritRoopak Parikh proposed openstack-infra/project-config:  Adding first version of project mors  https://review.openstack.org/35193323:26
pabelangerEmilienM: jeblair: I would agree with that. Let me compose an email summarizing when we talked about today, and move it to a thread on the ML.  That will get more eyes on the topic.23:27
EmilienMpabelanger: thanks23:27
fungii'm going to do some non-openstack things with the rest of my friday night, but will be around this weekend some23:27
fungihave a good night, all23:27
*** karthik__ has joined #openstack-infra23:28
*** karthik__ has quit IRC23:28
* bkero waves23:28
bkeroHave a good weekend23:28
*** karthik__ has joined #openstack-infra23:29
betherlyughhhhh for some reason i am now getting that the repo does not appear to be a gerrit repo. it seems to have lost its link to gerrit somehow23:30
*** jcoufal has quit IRC23:30
betherlygoodness only knows. friday evenings23:30
* betherly facepalms23:30
*** sdake has quit IRC23:31
clarkbbetherly: git review -s configures the repo to talk yo gerrit with git review23:32
clarkbnot sure what you mean by repo does nit appear to be a gerrit repo23:32
betherlyclarkb: thanks for sorting me out there23:33
betherlycan someone confirm for me that eslint-config-openstack v4.0.1 has now tagged and released correctly?23:34
*** roxanaghe has quit IRC23:34
*** xyang1 has quit IRC23:35
*** PollyZ has quit IRC23:37
*** sdague has quit IRC23:38
*** yamamoto has quit IRC23:41
*** jcoufal has joined #openstack-infra23:41
*** yamamoto has joined #openstack-infra23:41
*** yamamoto has quit IRC23:41
*** kzaitsev_mb has quit IRC23:41
*** zz_dimtruck is now known as dimtruck23:43
*** karthik__ has quit IRC23:45
*** ljjjustin has quit IRC23:45
fungii'm not really here but looks okay... http://git.openstack.org/cgit/openstack/eslint-config-openstack http://logs.openstack.org/1d/1d8ecdb54568df15d94ee826bd36c6b33ca61a90/release/23:47
fungibetherly: ^23:47
* fungi is really gone now23:47
zarojeblair: thought you might be interested, gerrit talking about zuul CRD https://groups.google.com/d/msg/repo-discuss/32oU6IaulW4/Ze-MTnNxCgAJ23:48
*** ljjjustin has joined #openstack-infra23:49
betherlythanks fungi :D23:50
*** xarses has joined #openstack-infra23:51
*** esberglu has quit IRC23:52
*** lascii is now known as alaski23:56
openstackgerritRoopak Parikh proposed openstack-infra/project-config: Adding first version of project mors  https://review.openstack.org/35195623:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!