Wednesday, 2017-07-12

*** ihrachys has quit IRC00:01
*** gmann has quit IRC00:11
*** gmann has joined #openstack-infra00:11
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add docs for web server component  https://review.openstack.org/48263000:13
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add link to Zuul v3 docs to the README  https://review.openstack.org/48234900:14
*** thorst has quit IRC00:17
*** jbadiapa_ has quit IRC00:17
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Fix ZuulWeb() invocation  https://review.openstack.org/48232100:19
*** dingyichen has joined #openstack-infra00:19
*** Swami has quit IRC00:22
*** sree has joined #openstack-infra00:23
*** Julien-zte has joined #openstack-infra00:25
*** vhosakot has quit IRC00:26
*** makowals has quit IRC00:27
*** sree has quit IRC00:27
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Use package module to install bindep packages  https://review.openstack.org/48258400:29
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Simplify bindep logic removing fallback support  https://review.openstack.org/48265000:29
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Move ssh-agent cleanup into existing exception handler  https://review.openstack.org/48273500:30
*** jbadiapa_ has joined #openstack-infra00:32
*** rloo has left #openstack-infra00:34
*** rloo has quit IRC00:35
*** xyang1 has quit IRC00:35
*** MatthewLi has joined #openstack-infra00:37
*** EricGonczer_ has quit IRC00:42
*** Julien-zte has quit IRC00:42
*** gildub has joined #openstack-infra00:43
*** zhurong has joined #openstack-infra00:43
*** slaweq has joined #openstack-infra00:43
*** Julien-zte has joined #openstack-infra00:43
*** Apoorva has quit IRC00:48
*** slaweq has quit IRC00:48
*** Julien-zte has quit IRC00:48
*** MatthewLi has quit IRC00:49
*** Julien-zte has joined #openstack-infra00:49
*** baoli has joined #openstack-infra00:49
*** Julien-zte has quit IRC00:50
*** MatthewLi has joined #openstack-infra00:51
*** cuongnv has joined #openstack-infra00:53
*** LindaWang has joined #openstack-infra00:54
*** bobh has joined #openstack-infra00:55
*** liujiong has joined #openstack-infra00:56
*** gcb has quit IRC00:57
openstackgerritJamie Lennox proposed openstack-infra/zuul feature/zuulv3: Remove isJobRegistered  https://review.openstack.org/48275501:00
*** gouthamr has quit IRC01:06
*** bobh has quit IRC01:10
*** masuberu has joined #openstack-infra01:14
*** cuongnv has quit IRC01:16
*** bobh has joined #openstack-infra01:17
*** masber has quit IRC01:18
*** pahuang has joined #openstack-infra01:20
*** cshastri has joined #openstack-infra01:21
*** rossella_s has quit IRC01:26
*** dfflanders has quit IRC01:26
*** ihrachys has joined #openstack-infra01:31
*** bobh has quit IRC01:33
*** mriedem has quit IRC01:33
*** aviau has quit IRC01:34
*** aviau has joined #openstack-infra01:34
*** s-shiono has joined #openstack-infra01:42
*** makowals has joined #openstack-infra01:44
*** slaweq has joined #openstack-infra01:45
*** vhosakot has joined #openstack-infra01:46
openstackgerritMatthew Treinish proposed openstack-infra/project-config master: Add gerribot to #openstack-tc for governance changes  https://review.openstack.org/48276101:46
openstackgerritMatthew Treinish proposed openstack-infra/project-config master: Add gerribot to #openstack-tc  https://review.openstack.org/48276101:48
*** slaweq has quit IRC01:49
*** mwarad has quit IRC01:55
*** masber has joined #openstack-infra01:57
*** annegentle has joined #openstack-infra01:58
*** masuberu has quit IRC02:00
*** yamahata has quit IRC02:00
*** iyamahat has quit IRC02:00
*** bobh has joined #openstack-infra02:03
*** dargains has joined #openstack-infra02:05
*** cuongnv has joined #openstack-infra02:07
*** hongbin has joined #openstack-infra02:09
*** dargains has quit IRC02:11
*** dtroyer has quit IRC02:11
*** dtroyer has joined #openstack-infra02:12
*** gildub has quit IRC02:15
*** mwarad has joined #openstack-infra02:19
fungiinc0: i've locked out the kolla images again. tarballs.o.o started getting flooded at about 01:45 and sites became unresponsive again02:22
*** gouthamr has joined #openstack-infra02:26
*** annegentle has quit IRC02:29
vhosakotfungi: do you know which kolla gate job caused the flood by any chance?02:29
fungivhosakot: i couldn't begin to guess. one of the ones that downloads registry tarballs from tarballs.o.o02:30
*** GregHouse has joined #openstack-infra02:31
fungithey're between 2 to 4 gigabytes apiece and that site is capped a 800 megabits per second combined inbound plus outbound02:31
fungiso it only takes a few dozen downloads that size at a time to overload the server for a while02:32
openstackgerritwes hayutin proposed openstack-infra/project-config master: add tripleo container periodic job  https://review.openstack.org/48277102:32
*** makowals has quit IRC02:33
vhosakotfungi: all gates that build from source pull tarballs from tarballs.openstack.org.  I think running such gates is parellal is causing the flood.02:37
fungii agree02:37
*** thorst has joined #openstack-infra02:39
fungitesting more than a couple of kolla changes at a time is probably our limit until we can finish getting the caching capacity upfit on the mirror servers02:39
*** cuongnv has quit IRC02:40
vhosakotok02:40
fungii had hoped i'd have a chance to finish that today, but i was busy with meetings all day so pabelanger has generously offered to do the two server rebuilds it's blocked on02:44
*** thorst has quit IRC02:44
*** slaweq has joined #openstack-infra02:45
*** ramishra has joined #openstack-infra02:45
clarkbfungi: thats ti increase disk available to the cache?02:45
clarkbfungi: was it only infracloud we need to do that in?02:46
fungiclarkb: yeah, i already did cinder volumes everywhere else and have the stack of changes to move the cache location02:46
*** makowals has joined #openstack-infra02:46
clarkbneat02:46
fungiexcept ovh where the rootfs is huge so didn't really need anythinfg02:46
fungicurrent bottom of the stack is https://review.openstack.org/482228 which is wip until those two instances are rebuilt with the updated flavor02:47
*** yamahata has joined #openstack-infra02:50
*** slaweq has quit IRC02:50
clarkbwe'll also likely need to test that apache2s default file serving serves files such that they are cacheable, but thats much easier once this gets in place02:51
*** zzzeek_ has quit IRC02:52
clarkbwhen did tarballs get https? thats nice02:52
clarkbfungi: comment on https://review.openstack.org/#/c/482203/4 I +2'd the others02:52
fungibeen https-able for a year or more, when i started doing a multi-domain cert on static.o.o02:56
fungian thanks for the reviews!02:56
*** makowals has quit IRC03:01
*** zhurong has quit IRC03:08
*** gildub has joined #openstack-infra03:08
*** bobh has quit IRC03:09
*** ramishra has quit IRC03:10
openstackgerritJeremy Stanley proposed openstack-infra/system-config master: Caching proxies for tarballs.openstack.org  https://review.openstack.org/48220303:13
fungiclarkb: ^ fixed03:14
*** cuongnv has joined #openstack-infra03:16
*** dave-mccowan has joined #openstack-infra03:19
*** dingyichen has quit IRC03:21
*** baoli has quit IRC03:22
*** rlandy has quit IRC03:23
*** baoli has joined #openstack-infra03:26
*** baoli has quit IRC03:31
*** gcb has joined #openstack-infra03:36
fungiinc0: i reset the kolla images to readable again a while ago and the spike didn't return so maybe we're okay again for the time being03:38
fungihttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all03:38
*** makowals has joined #openstack-infra03:39
*** dave-mccowan has quit IRC03:40
openstackgerritGoutham Pacha Ravi proposed openstack-infra/project-config master: Limit jobs in manila driverfixes branches  https://review.openstack.org/48278503:41
*** rodrigopaiva has joined #openstack-infra03:41
*** zhurong has joined #openstack-infra03:44
*** gouthamr has quit IRC03:46
*** slaweq has joined #openstack-infra03:46
fungiinfra-root: if we get another traffic spike on static.o.o, i've been doing `sudo chmod o-r /srv/static/tarballs/kolla/images/*.tar.gz` and then restarting apache2 to kill the in-progress downloads03:49
*** slaweq has quit IRC03:51
openstackgerritAmrith Kumar proposed openstack/diskimage-builder master: the current 08-debootstrap doesn't allow cross platform development  https://review.openstack.org/48279204:00
*** dingyichen has joined #openstack-infra04:03
*** tonyb has quit IRC04:05
*** sshnaidm|afk has joined #openstack-infra04:08
*** hughhalf has quit IRC04:12
*** ykarel has joined #openstack-infra04:12
*** adisky__ has joined #openstack-infra04:21
*** rodrigopaiva has quit IRC04:27
*** cuongnv has quit IRC04:28
*** tonyb has joined #openstack-infra04:28
*** psachin has joined #openstack-infra04:30
*** makowals has quit IRC04:32
*** yamamoto has joined #openstack-infra04:33
*** hongbin has quit IRC04:35
*** gongysh has joined #openstack-infra04:35
*** thorst has joined #openstack-infra04:40
*** pcaruana has joined #openstack-infra04:41
*** thorst has quit IRC04:45
*** cuongnv has joined #openstack-infra04:45
*** ramishra has joined #openstack-infra04:47
*** coolsvap has joined #openstack-infra04:56
*** pcaruana has quit IRC04:58
*** makowals has joined #openstack-infra05:01
*** slaweq has joined #openstack-infra05:05
*** sree has joined #openstack-infra05:05
*** slaweq has quit IRC05:05
openstackgerritCaleb Boylan proposed openstack-infra/system-config master: (WIP) Get install_puppet.sh to install puppet 4  https://review.openstack.org/48195505:05
*** vhosakot has quit IRC05:06
openstackgerritMatthew Thode proposed openstack-infra/irc-meetings master: update time, tony's back  https://review.openstack.org/48280305:09
*** MatthewLi has quit IRC05:11
*** armax has quit IRC05:14
*** armax has joined #openstack-infra05:14
*** armax has quit IRC05:15
*** armax has joined #openstack-infra05:15
*** armax has quit IRC05:16
*** armax has joined #openstack-infra05:16
*** armax has quit IRC05:16
*** armax has joined #openstack-infra05:17
*** armax has quit IRC05:17
*** krtaylor has quit IRC05:17
*** armax has joined #openstack-infra05:17
*** armax has quit IRC05:18
*** armax has joined #openstack-infra05:18
openstackgerritMatthew Thode proposed openstack-infra/irc-meetings master: update time, tony's back  https://review.openstack.org/48280305:18
*** armax has quit IRC05:19
*** armax has joined #openstack-infra05:19
*** armax has quit IRC05:20
*** baoli has joined #openstack-infra05:22
*** baoli has quit IRC05:27
*** cuongnv has quit IRC05:29
*** rcernin has joined #openstack-infra05:33
*** sanket_ has joined #openstack-infra05:36
*** gcb has quit IRC05:37
*** nicolasbock has joined #openstack-infra05:38
*** pgadiya has joined #openstack-infra05:39
*** gildub has quit IRC05:41
*** gyee has quit IRC05:52
*** prateek has joined #openstack-infra05:58
*** MatthewLi has joined #openstack-infra06:02
*** slaweq has joined #openstack-infra06:02
*** ramishra has quit IRC06:04
*** cuongnv has joined #openstack-infra06:04
*** dargains has joined #openstack-infra06:06
*** ramishra has joined #openstack-infra06:06
*** slaweq_ has joined #openstack-infra06:06
*** slaweq has quit IRC06:06
*** slaweq has joined #openstack-infra06:07
*** dargains_ has joined #openstack-infra06:07
*** tjones has quit IRC06:09
*** dargains has quit IRC06:11
*** florianf has joined #openstack-infra06:11
*** dargains_ has quit IRC06:11
*** slaweq_ has quit IRC06:12
AJaegerinfra-root, I get 502s from review.o.o ;(. Anybody around to check and potentially restart, please?06:13
AJaegerAfter a reload it works but is slowish.06:13
*** gtrxcb has joined #openstack-infra06:13
*** gtrxcb has quit IRC06:13
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: send graphite stats from ARA  https://review.openstack.org/48012106:14
*** liujiong has quit IRC06:19
*** liujiong has joined #openstack-infra06:19
sshnaidm|afkseems like gerrit is kind of stuck..06:20
openstackgerritMerged openstack-infra/project-config master: Add gerribot to #openstack-tc  https://review.openstack.org/48276106:21
*** isaacb has joined #openstack-infra06:25
AJaegersshnaidm|afk: read my comment from 7 minutes before yours06:26
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Use ara generation from collect-logs role  https://review.openstack.org/47854906:26
*** wolverineav has joined #openstack-infra06:31
*** isaacb_ has joined #openstack-infra06:32
*** isaacb has quit IRC06:33
*** gongysh has quit IRC06:34
calebbhttp://ask.openstack.org/ also looks down from here06:39
*** smcginnis has quit IRC06:39
*** thorst has joined #openstack-infra06:41
rakhmerovAJaeger: hi, I'd like to develop a new gate. Not really experienced in that yet. Let's say I sent a patch to add a new gate into CI. Is there a way to activate it per request even if it wasn't merged yet into project-config?06:41
rakhmerovin other words, how to debug it06:41
AJaegerrakhmerov: not with current zuul - but zuul v3 will do that (coming in September/October)06:42
rakhmerovaah06:42
rakhmerovok06:42
AJaegerSo, what I advise: Add to experimental queue and trigger with "run experimental" - and iterate on that.06:42
rakhmerovyep, got it06:42
AJaegerAnd once it works reliable, move to check queue06:42
rakhmerovthanks06:42
AJaeger"check experimental" is the key I think06:42
AJaegercalebb: indeed - thanks for telling06:43
*** cuongnv has quit IRC06:43
calebbnp06:43
*** smcginnis has joined #openstack-infra06:43
*** kjackal_ has quit IRC06:43
rakhmerovAJaeger: ok06:44
AJaegerinfra-root, ask.o.o is down06:44
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove journal binary log  https://review.openstack.org/48281806:44
*** andreas_s has joined #openstack-infra06:44
*** cuongnv has joined #openstack-infra06:44
*** d0ugal has quit IRC06:44
*** thorst has quit IRC06:46
*** openstackgerrit has quit IRC06:48
*** liujiong_lj has joined #openstack-infra06:50
AJaeger#status notice FYI, ask.openstack.org is down, review.o.o is slow - please have patience until this is fixed06:50
openstackstatusAJaeger: sending notice06:50
*** openstackgerrit has joined #openstack-infra06:50
openstackgerritAkihiro Motoki proposed openstack-infra/shade master: Improve doc formatting a bit  https://review.openstack.org/48281906:50
*** dhajare has joined #openstack-infra06:51
-openstackstatus- NOTICE: FYI, ask.openstack.org is down, review.o.o is slow - please have patience until this is fixed06:51
*** cuongnv has quit IRC06:53
openstackstatusAJaeger: finished sending notice06:53
*** liujiong has quit IRC06:54
*** cuongnv has joined #openstack-infra06:54
*** zhurong has quit IRC06:55
*** cuongnv has quit IRC07:03
*** ykarel_ has joined #openstack-infra07:04
berendtlooks like review.o.o is working normal now07:04
*** cuongnv has joined #openstack-infra07:05
*** ykarel has quit IRC07:07
*** kjackal_ has joined #openstack-infra07:07
*** slaweq_ has joined #openstack-infra07:07
cmurphyit's off and on07:09
AJaegerberendt: if it's in the usual garbage collection mess, you might be lucky if I scared enough people away ;)07:11
*** tesseract has joined #openstack-infra07:11
*** slaweq_ has quit IRC07:12
*** shardy has joined #openstack-infra07:13
openstackgerritMerged openstack-infra/project-config master: Updating tripleo tempest periodic job  https://review.openstack.org/47787907:19
openstackgerritArx Cruz proposed openstack-infra/tripleo-ci master: Fix stackviz collect data  https://review.openstack.org/47455907:20
openstackgerritMerged openstack-infra/project-config master: Add gating jobs to quickstart extras repo  https://review.openstack.org/48258807:20
*** cuongnv has quit IRC07:22
openstackgerritMerged openstack-infra/project-config master: Skip non-doc jobs for doc-only python-ironic-inspector-client changes  https://review.openstack.org/48245107:23
AJaegeryolanda, here're a few small cleanups of mine that I'd like to be reviewed https://review.openstack.org/#/c/481806/ https://review.openstack.org/#/c/481892/ https://review.openstack.org/#/c/481951/ https://review.openstack.org/#/c/482117/ - and could you check review.o.o whether it's in Garbage collect mode again, please?07:24
*** alexchadin has joined #openstack-infra07:30
*** pcaruana has joined #openstack-infra07:31
*** isaacb_ has quit IRC07:31
*** zhurong has joined #openstack-infra07:32
*** aarefiev_afk is now known as aarefiev07:32
*** egonzalez has joined #openstack-infra07:34
*** tuanluong has joined #openstack-infra07:36
*** mwarad has quit IRC07:37
*** ccamacho has joined #openstack-infra07:38
*** claudiub has joined #openstack-infra07:39
openstackgerritSergey Kulanov proposed openstack-infra/jeepyb master: Ensure acl_config has absolute path  https://review.openstack.org/48261207:39
*** pcaruana has quit IRC07:41
yolandahi AJaeger ... yep, review.o.o is very slow today07:44
yolandai'll check it07:44
*** amotoki_away is now known as amotoki07:45
*** pcaruana has joined #openstack-infra07:45
yolanda#status notice Gerrit is going to be restarted, due to low performance07:48
openstackstatusyolanda: sending notice07:48
-openstackstatus- NOTICE: Gerrit is going to be restarted, due to low performance07:49
AJaegeryolanda: thanks07:49
openstackstatusyolanda: finished sending notice07:51
yolanda#status ok Gerrit has been successfully restarted07:53
openstackstatusyolanda: sending ok07:53
*** sree has quit IRC07:54
*** jpich has joined #openstack-infra07:54
-openstackstatus- NOTICE: Gerrit has been successfully restarted07:56
*** sree_ has joined #openstack-infra07:58
*** sree_ is now known as Guest5897607:58
openstackstatusyolanda: finished sending ok07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** gouthamr has joined #openstack-infra08:02
openstackgerritMerged openstack-infra/project-config master: layout.yaml: Move mutex list last  https://review.openstack.org/48180608:02
*** liujiong_lj is now known as liujiong08:02
*** Guest58976 has quit IRC08:03
*** sshnaidm|afk has quit IRC08:03
*** rossella_s has joined #openstack-infra08:04
*** sshnaidm|afk has joined #openstack-infra08:05
openstackgerritMerged openstack-infra/project-config master: Export IRONIC_USE_RESOURCE_CLASSES for Mogan  https://review.openstack.org/48248408:06
openstackgerritMerged openstack-infra/project-config master: Set Tempest concurrency for non-multinode Patrole jobs  https://review.openstack.org/48269308:06
*** rkukura has quit IRC08:08
*** slaweq_ has joined #openstack-infra08:08
*** cuongnv has joined #openstack-infra08:09
*** rkukura has joined #openstack-infra08:12
*** slaweq_ has quit IRC08:13
openstackgerritMerged openstack-infra/project-config master: Add grafana devstack failure rate graph for openSUSE  https://review.openstack.org/48270608:14
openstackgerritMerged openstack-infra/project-config master: Update zuul-status rendering for the Tempest/Grenade graphs  https://review.openstack.org/48271108:15
openstackgerritMerged openstack-infra/project-config master: Add neutron-dynamic-routing periodic job with ryu master  https://review.openstack.org/48234008:16
openstackgerritMerged openstack-infra/project-config master: Add fedora-26 support for nodepool dsvm jobs  https://review.openstack.org/48256608:17
*** ralonsoh has joined #openstack-infra08:17
*** ykarel__ has joined #openstack-infra08:17
*** efoley has joined #openstack-infra08:17
*** makowals_ has joined #openstack-infra08:17
*** makowals has quit IRC08:20
*** ykarel_ has quit IRC08:20
openstackgerritMerged openstack-infra/project-config master: OSA: Add periodic test for translations scenario  https://review.openstack.org/48212008:21
*** gouthamr has quit IRC08:21
openstackgerritMerged openstack-infra/project-config master: Use python-jobs template for python-heatclient  https://review.openstack.org/48243408:22
openstackgerritMerged openstack-infra/project-config master: jenkins: jobs: ansible-role-jobs: Check if yum-config-manager is present  https://review.openstack.org/48246508:22
*** makowals has joined #openstack-infra08:22
openstackgerritMerged openstack-infra/project-config master: Remove oslo.* from the oslo group  https://review.openstack.org/48066708:23
openstackgerritMerged openstack-infra/project-config master: Set up DLM in n-g-s tempest job  https://review.openstack.org/48215508:24
*** makowals_ has quit IRC08:24
openstackgerritMerged openstack-infra/project-config master: Add functional job for watcherclient  https://review.openstack.org/48049508:24
openstackgerritMerged openstack-infra/project-config master: Add diskimage-builder-ci group  https://review.openstack.org/47905708:24
*** makowals has quit IRC08:30
*** lucas-afk is now known as lucasagomes08:32
*** pcaruana has quit IRC08:33
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Add settings for ovb in rdocloud  https://review.openstack.org/48095008:36
*** makowals has joined #openstack-infra08:41
*** makowals has quit IRC08:41
*** thorst has joined #openstack-infra08:42
*** s-shiono has quit IRC08:43
*** makowals has joined #openstack-infra08:44
*** jascott1 has quit IRC08:44
*** dklyle has joined #openstack-infra08:45
*** jascott1 has joined #openstack-infra08:45
*** david-lyle has quit IRC08:45
*** d0ugal has joined #openstack-infra08:45
*** pcaruana has joined #openstack-infra08:46
*** Douhet has quit IRC08:46
*** thorst has quit IRC08:47
*** liujiong has quit IRC08:47
*** liujiong has joined #openstack-infra08:47
*** electrofelix has joined #openstack-infra08:48
*** raissa_ has joined #openstack-infra08:49
*** jascott1 has quit IRC08:49
*** jbadiapa_ has quit IRC08:50
*** katkapilatova has joined #openstack-infra08:51
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Add settings for ovb in rdocloud  https://review.openstack.org/48095008:51
*** jbadiapa_ has joined #openstack-infra08:52
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Case sensitive label matching  https://review.openstack.org/48285608:53
*** d0ugal has quit IRC08:54
*** d0ugal has joined #openstack-infra08:55
*** d0ugal has quit IRC08:55
*** d0ugal has joined #openstack-infra08:55
*** makowals has quit IRC08:58
*** makowals has joined #openstack-infra09:04
*** Douhet has joined #openstack-infra09:05
*** iyamahat has joined #openstack-infra09:06
*** slaweq_ has joined #openstack-infra09:09
*** jpich has quit IRC09:09
*** jpich has joined #openstack-infra09:11
*** claudiub|2 has joined #openstack-infra09:11
*** claudiub has quit IRC09:14
*** slaweq_ has quit IRC09:14
*** liujiong has quit IRC09:15
*** iyamahat has quit IRC09:15
*** liujiong has joined #openstack-infra09:15
*** claudiub has joined #openstack-infra09:15
*** isaacb has joined #openstack-infra09:17
*** claudiub|2 has quit IRC09:18
*** tosky has joined #openstack-infra09:19
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Add support for custom ssh port  https://review.openstack.org/46875209:22
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: EOL ubuntu-precise for dsvm job  https://review.openstack.org/48261309:24
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove isJobRegistered  https://review.openstack.org/48275509:25
*** e0ne has joined #openstack-infra09:30
*** pcaruana has quit IRC09:32
*** askb has quit IRC09:32
*** pcaruana has joined #openstack-infra09:32
*** yamahata has quit IRC09:34
*** kjackal__ has joined #openstack-infra09:34
openstackgerritAndy McCrae proposed openstack-infra/project-config master: Add os_tacker role to OpenStack-Ansible  https://review.openstack.org/48287309:36
*** kjackal___ has joined #openstack-infra09:36
*** dargains has joined #openstack-infra09:37
*** kjackal_ has quit IRC09:37
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add job dependencies to status.json  https://review.openstack.org/48206109:37
openstackgerritAmrith Kumar proposed openstack/diskimage-builder master: the current 08-debootstrap doesn't allow cross platform development  https://review.openstack.org/48279209:39
*** sambetts|afk is now known as sambetts09:40
*** kjackal__ has quit IRC09:40
sshnaidm|afksorry for dumb question, but why do I see in repo commit like "Something changed" and then 1 day after commit from Jenkins "Merged Something changed"?09:42
*** thorst has joined #openstack-infra09:43
openstackgerritKien Nguyen proposed openstack-infra/project-config master: Setup multinode CI for Zun  https://review.openstack.org/48287709:46
*** thorst has quit IRC09:47
openstackgerritKien Nguyen proposed openstack-infra/project-config master: Setup multinode CI for Zun  https://review.openstack.org/48287709:47
*** dargains has quit IRC09:48
openstackgerritMerged openstack-infra/shade master: Add Neutron QoS bandwidth limit rule commands  https://review.openstack.org/48232509:55
openstackgerritAmrith Kumar proposed openstack/diskimage-builder master: the current 08-debootstrap doesn't allow cross platform development  https://review.openstack.org/48279209:59
AJaegersshnaidm|afk: which repo?10:04
AJaegerbest to explain on specific example10:04
sshnaidm|afkAJaeger, any, for example: https://github.com/openstack/tripleo-heat-templates/commits/master10:05
*** isaacb has quit IRC10:06
sshnaidm|afkAJaeger, "Switch from oslosphinx to openstackdocstheme" 15 days ago, and by Jenkins "Merge "Switch from oslosphinx to openstackdocstheme"" today10:06
AJaegersshnaidm|afk: that'S the order those are merged in - the order is different from when they were initially submitted.10:06
*** dizquierdo has joined #openstack-infra10:07
AJaegerThe Somethign changed is when you run "git commit"10:07
AJaegerThe "Merged something" is when gerrit merged it after approval10:07
sshnaidm|afkoh, I see..10:07
sshnaidm|afkand is this commit merged? "https://github.com/openstack/tripleo-heat-templates/commit/366825ec09ac44fe0440ed31685f5870032680a1"10:08
sshnaidm|afkbecause I don't see merge commit by Jenkins10:08
*** isaacb has joined #openstack-infra10:09
openstackgerritMarkos Chandras (hwoarang) proposed openstack-infra/project-config master: zuul: layout: Promote the openSUSE bifrost tinyipa job to gate  https://review.openstack.org/48288510:10
*** slaweq_ has joined #openstack-infra10:10
*** isaacb has quit IRC10:10
*** wolverineav has quit IRC10:11
*** isaacb has joined #openstack-infra10:12
cmurphysshnaidm|afk: it's a fast-forward merge, it was a simple enough history that it didn't need to create a merge commit10:13
*** slaweq_ has quit IRC10:14
*** slaweq_ has joined #openstack-infra10:14
*** tuanluong has quit IRC10:14
*** cuongnv has quit IRC10:14
sshnaidm|afkcmurphy, thanks, so it's merged10:15
openstackgerritChandan Kumar proposed openstack-infra/project-config master: Switch to tox for running tempest plugin sanity check  https://review.openstack.org/47416810:18
*** isaacb has quit IRC10:18
openstackgerritMonty Taylor proposed openstack-infra/puppet-zuul master: Add support for zuul-web and scheduler connections  https://review.openstack.org/48288910:19
openstackgerritJamie Lennox proposed openstack-infra/zuul feature/zuulv3: Remove function_cache variables from executor client  https://review.openstack.org/48289010:22
openstackgerritTomasz Trębski proposed openstack-infra/project-config master: Make Py35 gate votable for monasca-notification  https://review.openstack.org/48289210:22
*** cshastri has quit IRC10:23
*** liujiong has quit IRC10:23
*** stakeda has quit IRC10:24
*** rossella_s has quit IRC10:26
openstackgerritMonty Taylor proposed openstack-infra/puppet-zuul master: Add support for zuul-web and scheduler connections  https://review.openstack.org/48288910:26
*** xarses_ has quit IRC10:27
*** dklyle has quit IRC10:28
*** rossella_s has joined #openstack-infra10:28
*** david-lyle has joined #openstack-infra10:28
*** alexchadin has quit IRC10:28
*** dargains has joined #openstack-infra10:29
*** xarses has joined #openstack-infra10:29
*** dargains has quit IRC10:34
*** dargains has joined #openstack-infra10:35
*** rossella_s has quit IRC10:36
*** zhurong has quit IRC10:41
pabelangerfungi: clarkb: going to work in infracloud mirrors this morning10:42
*** dargains has quit IRC10:43
*** dargains has joined #openstack-infra10:45
*** dargains_ has joined #openstack-infra10:47
*** xinliang has quit IRC10:48
*** rossella_s has joined #openstack-infra10:48
*** d0ugal has quit IRC10:49
*** dargains has quit IRC10:50
*** dargains_ has quit IRC10:57
AJaegermorning pabelanger , you're up early10:57
pabelangerAJaeger: indeed, kids still jet lagged10:58
AJaeger;(10:58
openstackgerritMerged openstack-infra/shade master: Add flag to include all images in image list  https://review.openstack.org/48170210:59
*** xinliang has joined #openstack-infra11:01
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Avoid using apt-add-repository  https://review.openstack.org/48268311:03
*** jkilpatr has quit IRC11:06
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Add support to test fedora-26  https://review.openstack.org/48256811:07
*** wolverineav has joined #openstack-infra11:07
*** rhallisey has joined #openstack-infra11:07
*** markvoelker has quit IRC11:09
*** lucasagomes is now known as lucas-hungry11:10
*** markvoelker has joined #openstack-infra11:10
*** dargains has joined #openstack-infra11:11
openstackgerritMarkos Chandras (hwoarang) proposed openstack-infra/project-config master: zuul: layout: Promote the openSUSE bifrost tinyipa job to gate  https://review.openstack.org/48288511:11
AJaegermordred, pabelanger, here're a few small cleanups of mine that I'd like to be reviewed to cleanup the queue https://review.openstack.org/#/c/481892/ https://review.openstack.org/#/c/481951/ https://review.openstack.org/#/c/482117/ - could you push this on your review list, please?11:13
pabelanger+311:14
mordred++11:14
*** markvoelker has quit IRC11:14
AJaegerthanks to both of you!11:14
*** slaweq_ has joined #openstack-infra11:15
pabelangerclarkb: where did we end up with ipv6 on citycloud? Did we ever do your network test to see if it would work properly?11:16
*** dargains has quit IRC11:19
*** dave-mccowan has joined #openstack-infra11:19
*** slaweq_ has quit IRC11:20
openstackgerritMerged openstack-infra/project-config master: Remove unused jobs for paunch  https://review.openstack.org/48189211:20
openstackgerritMerged openstack-infra/project-config master: Remove api-ref-src from api-ref job-group  https://review.openstack.org/48195111:21
openstackgerritMerged openstack-infra/project-config master: Stop translation the cisco horizon plug-in  https://review.openstack.org/48211711:21
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci master: DNM - hijack 3nodes job to test docker+HA  https://review.openstack.org/48291311:23
*** isaacb has joined #openstack-infra11:25
openstackgerritTomasz Trębski proposed openstack-infra/project-config master: Make Py35 gate votable for monasca-notification  https://review.openstack.org/48289211:26
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Remove default value for mirror_host  https://review.openstack.org/48291511:26
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Replace hand-written slug with ansible_managed  https://review.openstack.org/48291611:26
*** LindaWang has quit IRC11:28
*** isaacb has quit IRC11:28
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Remove openstack-doc-build from zuul-jobs  https://review.openstack.org/48292111:29
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add openstack-docs-build to zuul-jobs  https://review.openstack.org/48292211:30
mordredpabelanger: ^^ those last two move openstack-docs-build from zuul-jobs to project-config11:31
openstackgerritMerged openstack-infra/zuul-jobs master: Enable verify_host for synchronize  https://review.openstack.org/48265511:32
*** shardy is now known as shardy_lunch11:32
pabelangermordred: wait, how do we have 2 project definitions for openstack-infra/zuul-jobs?11:35
pabelangerI never knew that was valid11:35
pabelangeris this the new shadow jobs configuration?11:35
pabelangerOh wait, I think I see11:35
pabelangerokay, was confused for a second11:36
mordredpabelanger: we can define things centrally in project-config - then a project can additinally add jobs to itself in its own zuul.yaml11:36
mordredpabelanger: it's a whole new thing :)11:36
openstackgerritMerged openstack-infra/zuul-jobs master: Remove openstack-doc-build from zuul-jobs  https://review.openstack.org/48292111:36
*** xarses_ has joined #openstack-infra11:36
pabelangermordred: Ya, remember now. We had to do that to define the default pipelines also11:36
mordredpabelanger: this is how we can centrally mandate that all projects run pep8 and py27, but still let them add their own jobs locally :)11:36
*** gordc has joined #openstack-infra11:36
pabelangerlittle rusty on zuulv3 :D11:37
mordredyah11:37
mordredpabelanger: :)11:37
*** ldnunes has joined #openstack-infra11:37
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Run zuul-web on zuulv3.o.o  https://review.openstack.org/48292611:37
mordredpabelanger: speaking of - https://review.openstack.org/#/c/482889/ and ^^11:37
mordredpabelanger: should get us the websocket streaming running11:38
*** thorst has joined #openstack-infra11:38
pabelangermordred: excellent11:39
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove legacy and unused code from tripleo-ci  https://review.openstack.org/48292911:39
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: DNM: Test spec helper changes  https://review.openstack.org/48194811:40
*** xarses has quit IRC11:40
*** jkilpatr has joined #openstack-infra11:43
openstackgerritMerged openstack-infra/project-config master: Add openstack-docs-build to zuul-jobs  https://review.openstack.org/48292211:44
*** florianf has quit IRC11:45
*** isaacb has joined #openstack-infra11:51
*** markvoelker has joined #openstack-infra11:55
*** pradk has joined #openstack-infra11:55
*** florianf has joined #openstack-infra11:59
chandankumarAJaeger: yolanda please have a look on this https://review.openstack.org/#/c/474168/ when you are free thanks :-)11:59
*** alexchadin has joined #openstack-infra12:00
*** isaacb has quit IRC12:00
*** isaacb has joined #openstack-infra12:01
*** dprince has joined #openstack-infra12:05
openstackgerritColleen Murphy proposed openstack-infra/project-config master: Make puppet-openstackci beaker jobs voting  https://review.openstack.org/48293812:06
*** jtomasek_ has joined #openstack-infra12:07
*** lucas-hungry is now known as lucasagomes12:09
*** shardy_lunch is now known as shardy12:10
openstackgerritMonty Taylor proposed openstack-infra/puppet-zuul master: Add support for a parallel set of secrets for connections  https://review.openstack.org/48294112:14
openstackgerritPaul Belanger proposed openstack-infra/nodepool master: Support fedora-26 for nodepool dsvm job  https://review.openstack.org/48294212:14
openstackgerritPaul Belanger proposed openstack/diskimage-builder master: Bump fedora-minimal DIB_RELEASE 26  https://review.openstack.org/48257012:15
*** slaweq_ has joined #openstack-infra12:16
*** kjackal_ has joined #openstack-infra12:18
*** kjackal__ has joined #openstack-infra12:20
*** rlandy has joined #openstack-infra12:20
*** kjackal___ has quit IRC12:20
*** slaweq_ has quit IRC12:21
*** jamesden_ has joined #openstack-infra12:21
*** kjackal_ has quit IRC12:22
*** krtaylor has joined #openstack-infra12:23
*** jamesdenton has quit IRC12:24
*** jtomasek_ has quit IRC12:25
pabelangerfungi: I think we might need to restore mysql.qcow2 for a day or so. Dealing with job failured on older branched of devstack currently12:27
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Add MySQL and Github connection information for Zuul v3  https://review.openstack.org/48294912:27
pabelangerfungi: gate-grenade-dsvm-multinode seems tobe failing on newton12:28
fungipabelanger: okay, is there a revert?12:28
pabelangerfungi: no revert yet, I was thinking we could just add the qcow2 image back to tarballs.o.o for now do we can get DIBs building again12:29
mordredcmurphy: if you don't mind- would you look at https://review.openstack.org/482941 ? I did more templating in that than I normally do12:29
pabelangerat 3 days of failures because we are still attempting to download it12:29
cmurphymordred: sure12:29
mordredcmurphy: there is an example of the data in https://review.openstack.org/482949 - the data in zuul_connections_secret is the same format12:30
fungipabelanger: oh, now i get what you mean12:30
cmurphymordred: the syntax job is already helping you12:30
openstackgerritThierry Carrez proposed openstack-infra/project-config master: Update ACLs for stable/pike in prep for release  https://review.openstack.org/48295112:31
fungipabelanger: i believe the changes will merge fine if you depends-on in sequence all the way back to the one for the oldest branch12:31
*** zzzeek_ has joined #openstack-infra12:32
mordredcmurphy: woot!12:32
fungipabelanger: but i can also just restore it for now12:32
openstackgerritMonty Taylor proposed openstack-infra/puppet-zuul master: Add support for a parallel set of secrets for connections  https://review.openstack.org/48294112:33
pabelangerfungi: I'm not sure I follow on your depends-on comment12:33
pabelangeralso: http://logs.openstack.org/03/482603/2/check/gate-grenade-dsvm-multinode/76487d1/logs/grenade.sh.txt.gz#_2017-07-12_11_02_41_587 is the failure atm12:33
pabelangermtreinish: ^ does that sound familiar for devstack failure?12:34
pabelangerthis is on newton branch12:34
fungipabelanger: i've temporarily restored that one image to it's original location12:35
*** pgadiya has quit IRC12:35
pabelangerfungi: great, ty12:35
openstackgerritThierry Carrez proposed openstack-infra/project-config master: Update ACLs for stable/pike in prep for release  https://review.openstack.org/48295112:35
fungias for depends-on between branches, this is common for sequencing upgrade fixes for grenade jobs12:35
openstackgerritAkihiro Motoki proposed openstack-dev/hacking master: rearrange existing documentation to fit the new standard layout  https://review.openstack.org/48253712:35
openstackgerritAkihiro Motoki proposed openstack-dev/hacking master: Switch from oslosphinx to openstackdocstheme  https://review.openstack.org/48253612:35
openstackgerritAkihiro Motoki proposed openstack-dev/hacking master: Update the doc URL in the documents  https://review.openstack.org/48253812:35
fungipabelanger: make the stable/newton change depends-on the stable/mitaka change (or maybe it's time to drop grenade jobs for newton?)12:36
amotokiis there any way to know which repository is specified in Depends-On?12:37
pabelangerfungi: right, I'll have to see what openstack-qa would like to do12:37
amotokiI am wondering how we can send a patch to neutron which depends on libraries like neutron-lib.12:37
fungisince the issue is that the newton grenade job starts by installing mitaka devstack which still wants to download that file, unless you depends-on the change which makes that no longer happen12:38
pabelangerfungi: okay, I think I see what you are saying12:38
mordredamotoki: you can just put the change-id in the depends-on12:39
fungiamotoki: not sure what you mean. a change can declare multiple depends-on change-ids, but also a single change-id can refer to multiple changes in different repositories12:39
mordredamotoki: it totally works across repos12:39
*** gtmanfred has quit IRC12:40
*** zhenguo_ has joined #openstack-infra12:40
*** v1k0d3n_ has joined #openstack-infra12:40
*** persia__ has joined #openstack-infra12:40
*** hrybacki_ has joined #openstack-infra12:40
*** fmccrthy_ has joined #openstack-infra12:40
*** szaher has quit IRC12:40
*** squid has joined #openstack-infra12:40
fungiamotoki: are you asking how to switch behavior between installing a dependent change from source vs installing the same library from a release?12:40
*** zerick has quit IRC12:41
*** chandankumar has quit IRC12:41
amotokifungi: mordred: if we send a patch which depends on a library patch, upper-constraints enforces to use a release version. I woudl like to run edit-constraints if we have a lib patch in Depends-On .12:41
*** yamamoto has quit IRC12:41
*** tbarron_ has joined #openstack-infra12:41
pabelangermordred: fungi: do you mind adding https://review.openstack.org/#/c/482592/ to your review pipeline. It brings fedora-26 mirrors online. I've confirmed we have the HDD space in afs12:41
*** zbitter has joined #openstack-infra12:41
*** fmccrthy has quit IRC12:41
*** v1k0d3n has quit IRC12:41
*** DinaBelova has quit IRC12:41
*** rfolco has quit IRC12:41
*** vipul has quit IRC12:41
*** hrybacki has quit IRC12:41
*** zhenguo has quit IRC12:41
*** persia has quit IRC12:41
*** calebb has quit IRC12:41
*** fbouliane has quit IRC12:41
*** dprince has quit IRC12:41
*** zaneb has quit IRC12:41
*** numans has quit IRC12:41
*** eglute_s has quit IRC12:41
*** tbarron has quit IRC12:41
*** vaidy has quit IRC12:41
*** kuromagi has quit IRC12:41
*** loquacities has quit IRC12:41
*** melwitt has quit IRC12:41
*** rhagarty has quit IRC12:41
*** vkmc has quit IRC12:41
*** fmccrthy_ is now known as fmccrthy12:41
*** v1k0d3n_ is now known as v1k0d3n12:41
*** hrybacki_ is now known as hrybacki12:41
*** zhenguo_ is now known as zhenguo12:41
*** zerick has joined #openstack-infra12:41
*** chandankumar has joined #openstack-infra12:41
*** melwitt has joined #openstack-infra12:41
fungiif so, you don't do that. you have different jobs for from source or from release packages and run both12:41
*** vipul has joined #openstack-infra12:41
*** mrunge has quit IRC12:41
*** tbarron_ is now known as tbarron12:41
pabelangermordred: fungi: once online and things migrated to fedora-26, I'll remove the fedora-25 mirroring12:41
mordredamotoki: ah - that'sa whole different thing. you'll need to have a job that ^^ yah, what fungi  said12:41
*** numans has joined #openstack-infra12:41
*** melwitt is now known as Guest8620812:41
*** eglute has joined #openstack-infra12:42
*** mrunge has joined #openstack-infra12:42
*** kuromagi has joined #openstack-infra12:42
*** DinaBelova has joined #openstack-infra12:42
*** gtmanfred has joined #openstack-infra12:42
*** fbouliane has joined #openstack-infra12:42
*** rfolco has joined #openstack-infra12:42
*** vkmc has joined #openstack-infra12:42
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove legacy and unused code from tripleo-ci  https://review.openstack.org/48292912:42
amotokifungi: mordred: that solves the issue, but it wastes our infra resources in most cases.12:42
*** rhagarty has joined #openstack-infra12:42
*** loquacities has joined #openstack-infra12:43
*** vaidy has joined #openstack-infra12:44
*** notmyname has quit IRC12:44
*** timburke has quit IRC12:44
fungiamotoki: having a job selectively install proposed changes on branch tip sometimes but release versions other times can deadlock your repository12:44
*** szaher has joined #openstack-infra12:44
*** dgonzalez has quit IRC12:44
fungiamotoki: also you would require some fairly intractable logic to determine recursive dependencies in a series of changes12:45
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Adjust MTUs in multinode workflow on RDO Cloud  https://review.openstack.org/48122212:45
andreas_spabelanger, clarkb Seems like the issue with the trove image still exists. The cache-devstack element code iterates over all branches https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos#L12412:45
*** dgonzalez has joined #openstack-infra12:46
*** timburke has joined #openstack-infra12:46
* amrith tunes in when he hears the word trove12:46
amrithwhat did we do now?12:46
amotokifungi: ah I see. it seems more complicated than I thought.12:46
fungiamotoki: zuul precalculates all that for the job and presents a consistently-named git revision, but to make use of that the job would need to install each repository you're concerned about through devstack-gate or zuul-cloner12:46
pabelangerandreas_s: yes, still working on patches to devstack, for now the image is back online so we can build images12:47
*** notmyname has joined #openstack-infra12:47
* amrith wanders away ... that trove ...12:47
pabelangerandreas_s: once I can confirm we can build DIBs with the image still on tarballs.o.o, I'll ask fungi to remove it again12:47
*** dprince has joined #openstack-infra12:47
*** eharney has joined #openstack-infra12:47
fungiamrith: stable branches of devstack were still insisting on downloading trove images12:47
amrithuh oh, sorry about that12:48
andreas_spabelanger, ah great - I still was seeing the issue in my thirdparty ci - but let me check again12:48
amrithdo you want to revert them in a different place?12:48
pabelangerandreas_s: Ya, we just restored in about 20mins go12:48
pabelangeramrith: we currently need to land: https://review.openstack.org/#/c/482603/12:48
fungiamrith: i temporarily moved the one it uses back where devstack is looking for it until we can get devstack reviewers to approve the stable branch changes to rip it out12:49
pabelangeramrith: trying to determine the best path forward on grenade job12:49
amrithpabelanger fungi just restore the file on tarballs.o.o12:49
andreas_spabelanger, yep, seems to work again...12:49
amrithand land the patch12:49
amrithwe now know what falls apart when we remove t.o.o12:49
amrithwe an remove it at a later date12:49
amrithsorry for the confusion, I didn't know that things outside trove used our turds12:50
* amrith grins ... sorry, images12:50
fungiamrith: i think devstack used to unconditionally download it in case trove got enabled12:50
andreas_spabelanger, I guess we also need to fix that for the liberty kilo and mitaka branches (not sure why we still have them...)12:51
fungii'll be surprised if those can even still run jobs12:52
pabelangerYa, noticed devstack-cache was still hitting those branches12:52
pabelangerneed to see if they actually add images12:53
*** jaypipes has joined #openstack-infra12:53
pabelangeror just EOL them?12:53
*** dougs2 has left #openstack-infra12:54
*** sanket_ has quit IRC12:54
openstackgerritAttila Darazs proposed openstack-infra/puppet-openstackci master: Add a static footer for TripleO Quickstart logs  https://review.openstack.org/48221012:55
pabelangerfungi: https://review.openstack.org/#/c/450818/ seems to be related12:56
*** xinliang has quit IRC12:56
*** d0ugal has joined #openstack-infra12:56
pabelangerfungi: looking to see if it applies to stable branches12:56
*** xinliang has joined #openstack-infra12:57
*** kgiusti has joined #openstack-infra12:57
toskyuhm, are there connection issues for tarballs.o.o? ( Failed to connect to tarballs.openstack.org port 80: Connection timed out)12:57
toskyoh, no, sorry12:57
toskyignore me12:57
toskyold logs12:57
*** tushar has joined #openstack-infra12:58
pabelangertosky: we've been having issues recently but fungi is working to fix them12:58
toskypabelanger: yes, and the logs tricked me12:58
toskysorry for the confusion12:58
*** tushar has quit IRC12:58
*** tushar has joined #openstack-infra12:58
*** ralonsoh has quit IRC12:58
*** ralonsoh_ has joined #openstack-infra12:58
*** tushar has quit IRC12:58
*** tushar has joined #openstack-infra12:59
*** LindaWang has joined #openstack-infra12:59
*** tushar has quit IRC12:59
*** Goneri has joined #openstack-infra12:59
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Cleanup components doc  https://review.openstack.org/48263612:59
*** ralonsoh_ is now known as ralonsoh13:00
fungifwiw, i'm still keeping an eye on http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all until we can cut jobs over to using the local tarballs proxy caches13:01
pabelangerfungi: question, how does devstack install stable/mitake of keystone, when I don't see any branch on git.o.o? http://logs.openstack.org/03/482603/2/check/gate-grenade-dsvm-multinode/76487d1/logs/old/devstacklog.txt.gz#_2017-07-12_11_01_25_73713:01
fungipabelanger: i don't think it can, unless someone has been updating devstack to switch from branch names to eol tags (or unless dib isn't deleting cached branches when they disappear from the remote?)13:03
pabelangerfungi: http://logs.openstack.org/03/482603/2/check/gate-grenade-dsvm-multinode/76487d1/logs/pip2-freeze.txt.gz seems to imply to used the master branch13:04
fungipabelanger: yep! that's the fallback behavior when it can't find the requested branch13:05
*** Julien-zte has joined #openstack-infra13:05
pabelangerfungi: okay, ya so it looks like grenade job is indeed broken on newton. I guess it is time to remove or update devstack to use eol tags?13:05
fungibasically there's no point in keeping stable/mitaka around for devstack once the branch is eol on the services it's using13:05
pabelangerfungi: okay, let me remove job and have mtreinish review13:06
fungiwhile it may run, it won't be testing what you expect13:06
pabelangerya13:06
*** kjackal__ has quit IRC13:06
pabelangerseems to be that13:06
pabelangerinfact, I think gate-grenade-dsvm-multinode can now be deleted, seems to only run on newton13:08
*** yamamoto has joined #openstack-infra13:08
*** mdrabe has joined #openstack-infra13:10
*** zzzeek_ has quit IRC13:12
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Remove gate-grenade-dsvm-multinode job  https://review.openstack.org/48296213:13
*** zzzeek_ has joined #openstack-infra13:13
pabelangerfungi: mtreinish: ^ When you have time to review13:13
*** yamamoto has quit IRC13:13
*** zhurong has joined #openstack-infra13:14
pabelangerandreaf: ^ might be interested as well13:14
*** slaweq_ has joined #openstack-infra13:17
pabelangerfungi: clarkb: 15.184.66.172 is our new mirror in infracloud-vanilla13:17
pabelangerfungi: clarkb: I'm going to start the process of updating DNS records to point to it13:18
*** d0ugal has quit IRC13:19
*** isaacb has quit IRC13:19
pabelanger#status log DNS for mirror.regionone.infracloud-vanilla.openstack.org changed to 15.184.66.172, 60min TTL13:22
openstackstatuspabelanger: finished logging13:22
*** slaweq_ has quit IRC13:22
fungipabelanger: awesome, thanks! disk space lgtm13:23
*** rossella_s has quit IRC13:23
pabelangerfungi: i'll wait until I see some traffic to new server before working on infracloud-chocolate13:23
fungisounds great13:23
*** michaelxin has joined #openstack-infra13:25
pabelangerstarting to see traffic to the new server now13:27
pabelangernc 15.184.66.182 1988513:27
*** isaacb has joined #openstack-infra13:28
*** baoli has joined #openstack-infra13:28
pabelangerk, switching to infracloud-chocolate now13:28
*** zzzeek_ has quit IRC13:29
*** frickler has quit IRC13:33
*** zzzeek_ has joined #openstack-infra13:34
*** bobh has joined #openstack-infra13:37
*** rossella_s has joined #openstack-infra13:38
*** armax has joined #openstack-infra13:39
*** tushar has joined #openstack-infra13:40
tusharHi All13:40
*** frickler has joined #openstack-infra13:41
tusharI have setup a new third party CI13:41
tusharfacing some issue in jenkins slave13:41
tusharI am not able to see any jenkins slave on jenkins UI13:41
tusharcan anybody tell what I am missing here ?13:41
pabelangertushar: nodepool debug.log should help explain what is going on.13:42
pabelangeris your nodes being launched in your cloud properly?13:42
*** trown is now known as trown|brb13:42
tusharPabelanger : node is launched and is in ready  state13:42
pabelangertushar: okay, so you should see nodepool handing off the node to jenkins13:43
pabelangertushar: then, jenkins will attempt to SSH into said node, using SSH key that has been setup in jenkins13:43
pabelangerbut logs in nodepool will show this process and help debug where the failure is13:44
tusharok..checking the nodepool logs13:44
*** coolsvap has quit IRC13:45
fungiworth noting, this sounds a lot like bkero's issue where he needed python-jenkins master branch to use nodepool with a recent (in the last few months) jenkins release13:46
*** mriedem has joined #openstack-infra13:46
fungijenkins changed some api calls in backward-incompatible ways recently13:46
pabelangeragree, that is what I am thinking too13:46
pabelangerIIRC, is was due to a security issue13:47
*** trown|brb is now known as trown13:47
*** jascott1 has joined #openstack-infra13:47
clarkbfungi: I think the issue ended up being fixed by using latest nodepool which uses python-jenkins rather than the old nodepool vode wrapped around python-jenkins that eventually got into python-jenkins and was improves there13:51
*** yamamoto has joined #openstack-infra13:51
clarkbtldr use latest nodepool13:51
*** mat128 has joined #openstack-infra13:51
*** jascott1 has quit IRC13:52
*** mat128 has quit IRC13:54
*** adarazs has joined #openstack-infra13:55
*** xyang1 has joined #openstack-infra13:56
*** frickler has quit IRC13:57
*** ZZelle has quit IRC13:57
adarazshi folks, I'm trying to get this change to pass the gates: https://review.openstack.org/482210 (openstack-infra/puppet-openstackci project) and it seems "gate-openstackci-beaker-centos-7" is failing with something not relevant to my change, can somebody help me here?13:58
adarazs"Gem::RemoteFetcher::UnknownHostError: no such name" -> http://logs.openstack.org/10/482210/2/check/gate-openstackci-beaker-centos-7/6f445f7/console.html#_2017-07-12_13_30_21_57024613:58
openstackgerritRabi Mishra proposed openstack-infra/project-config master: Modify some heat related projects to use standard templates  https://review.openstack.org/48297314:01
*** jcoufal has joined #openstack-infra14:02
*** frickler has joined #openstack-infra14:03
adarazshm, it passed before, so it's probably some temporary issue, I'll recheck and hope for the best.14:04
*** gcb has joined #openstack-infra14:04
*** ykarel__ is now known as ykarel14:05
*** hongbin has joined #openstack-infra14:06
fungiadarazs: yes, sometimes (like anywhere else on the internet) dns lookups fail14:08
fungiso it was probably temporary14:08
*** rbrndt has joined #openstack-infra14:09
*** ykarel is now known as ykarel|afk14:10
tusharpabelanger : nothing much I found in nodepool and jenkins log14:12
tusharin nodepool logs I can see "  node 1234 is ready "14:12
*** mat128 has joined #openstack-infra14:13
pabelangertushar: do you see nodepool telling jenkins to add the node as a slave?14:13
tusharafter this  i should see " Node id: 12345 added to jenkins "14:13
tusharbut its missing14:13
tusharand there is no error message as well14:14
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Adjust MTUs in multinode workflow on RDO Cloud  https://review.openstack.org/48122214:14
pabelangertushar: okay, so this sounds like the issue clarkb and fungi were just talking about. Likely need to update nodepool per clarkb suggestion14:14
*** ykarel|afk has quit IRC14:14
tusharPabelanger : you mean need to upgrade nodepool version?14:15
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Correct run job parameter documentation  https://review.openstack.org/48297714:15
pabelangertushar: ya14:15
tusharPabelnger :  oky14:16
tusharbut I am using old versions of nodepool as well as jenkins14:16
*** makowals has quit IRC14:16
tusharanyway I will try by upgrading nodepool14:16
*** arxcruz has quit IRC14:17
clarkbya new nodepool uses python-jenkins a bit more whovh handles jenkins behavior better14:18
*** arxcruz has joined #openstack-infra14:18
*** slaweq_ has joined #openstack-infra14:18
*** makowals has joined #openstack-infra14:19
tusharclarkb : new nodepool means 0.3.1 version?14:20
*** tjones has joined #openstack-infra14:20
openstackgerritzhurong proposed openstack-infra/project-config master: Remove the outdate murano-ci Third-Party CI  https://review.openstack.org/48298014:20
clarkbtushar: 0.4.1 I think14:20
tusharclarkb: with this version of nodepool ...which version of jnkins is compatible?14:21
clarkbtushar: whatever versions python-jenkins supports14:22
tusharok14:22
tusharclarkb : will try this...thanks14:22
*** wolverineav has quit IRC14:22
*** slaweq_ has quit IRC14:23
*** zzzeek has quit IRC14:24
cmurphyif anyone's interested i did some beaker refactoring that i'm hoping will make overall puppet reviews easier https://review.openstack.org/#/q/topic:refactor-infra-beaker14:25
fungiclarkb: though nodepool 0.4.1 _does_ require also installing zookeeper, right?14:26
fungicmurphy: thanks!14:26
*** camunoz has joined #openstack-infra14:26
clarkbfungi: yes it does14:28
*** dizquierdo has quit IRC14:29
mnaserreally throwing this out here but would zuulv3 be ready enough to use for simple periodic jobs? (we are puppet-izing all the things and automation for jenkins has been hell)14:33
mnaserthe jobs are pretty much just poll a url to check if new images are released by upstream (ex: ubuntu cloud images) and then download them, test them and publish them14:33
*** Julien-zte has quit IRC14:34
fungimnaser: maybe, but asking in #zuul is probably better. biggest issues are the syntax is still shifting sands to some extent and security isolation for workers is still being hammered out14:34
mnaseroh, didnt know there was a channel for it now, thought that was still in here14:34
*** gouthamr has joined #openstack-infra14:35
fungihere is good for general operational discussions but zuul v3 development state is probably more on topic in #zuul until it's released14:35
*** Julien-zte has joined #openstack-infra14:35
*** Julien-zte has quit IRC14:35
*** wolverineav has joined #openstack-infra14:36
fungiyou're likely to get more complete info in there about what's still missing/broken/changing14:36
pabelangermnaser: I think it is ready for experimenting, but not sure about production just yet.  We've been getting good fixes for zuulv3 from people running it locally for testing so far14:36
*** Julien-zte has joined #openstack-infra14:36
*** Julien-zte has quit IRC14:36
fungiwe have it running today in an advisory capacity self-testing changes proposed to the zuul repo and some zuulv3-specific configuration repos so it _does_ run, but we're a couple months out from where we expect to be able to reasonably switch our load and use cases to it14:37
*** Julien-zte has joined #openstack-infra14:37
*** Julien-zte has quit IRC14:37
*** zbitter is now known as zaneb14:37
*** Julien-zte has joined #openstack-infra14:38
mnaserfungi i've figured that if it can self-test at this point, it's probably more than enough to do what we're looking to do out of it14:38
*** Julien-zte has quit IRC14:38
* mnaser will do some more reading14:38
*** ykarel has joined #openstack-infra14:38
*** Julien-zte has joined #openstack-infra14:38
*** Julien-zte has quit IRC14:39
openstackgerritTomasz Trębski proposed openstack-infra/project-config master: Make Py35 gate votable for monasca-notification  https://review.openstack.org/48289214:39
fungifor example, i couldn't tell you if periodic triggers are working yet but the people who are paying close attention to #zuul likely know14:39
*** Julien-zte has joined #openstack-infra14:39
*** Julien-zte has quit IRC14:39
pabelangerYa, we don't actually have a periodic job running on zuulv3.o.o yet14:40
*** Julien-zte has joined #openstack-infra14:40
*** Julien-zte has quit IRC14:40
*** Julien-zte has joined #openstack-infra14:41
*** Julien-zte has quit IRC14:41
*** dingyichen has quit IRC14:42
*** Julien-zte has joined #openstack-infra14:42
*** Julien-zte has quit IRC14:42
*** dizquierdo has joined #openstack-infra14:44
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Updated Eventbrite permission model  https://review.openstack.org/48273014:45
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Updated Eventbrite permission model  https://review.openstack.org/48273014:46
pabelanger#status log DNS for mirror.regionone.infracloud-chocolate.openstack.org changed to 15.184.69.112, 60min TTL14:46
openstackstatuspabelanger: finished logging14:46
pabelangerfungi: clarkb: new infracloud-chocolate mirror online, just waiting for traffic to start hitting it14:46
ianychoiclarkb, fungi: Hello! I wanted to attend to last infra team meeting but it was too difficult to wake up at 4am unfortunately. If you have time, would you see http://lists.openstack.org/pipermail/openstack-infra/2017-July/005494.html and support with higher priority?14:49
*** nicolasbock has quit IRC14:50
pabelangernc 15.184.69.142 1988514:52
pabelangeris now hitting new mirror14:52
*** zhurong has quit IRC14:53
*** isaacb has quit IRC14:54
*** psachin has quit IRC14:54
fungipabelanger: disk space looks good there now too. i'll get the new apache cache locations set up and warmed on them and then un-wip the bottom of the change stack14:54
pabelangerfungi: great, I think we can now delete the original vanilla mirror, I haven't see any traffic for a bit now14:55
pabelangerchocolate will need another 60mins or so14:55
*** dtantsur is now known as dtantsur|afk14:55
pabelangerand deleted14:57
inc0good morning14:57
*** david-lyle has quit IRC14:58
*** david-lyle has joined #openstack-infra14:58
*** prateek has quit IRC14:59
*** marst has joined #openstack-infra15:00
fungipabelanger: i've approved https://review.openstack.org/482228 and https://review.openstack.org/482229 now that the caches are all ready15:01
fungirather, cache locations15:01
pabelangerfungi: Oh, I still need to SSH into new servers from puppetmaster15:01
*** rcernin has quit IRC15:01
pabelangerlet me do that now15:01
openstackgerritKien Nguyen proposed openstack-infra/project-config master: Setup multinode CI for Zun  https://review.openstack.org/48287715:01
fungipabelanger: thanks, the changes definitely won't take effect until that happens15:02
*** zzzeek_ has quit IRC15:02
pabelangerfungi: done, new host keys accepted15:02
fungiapproved https://review.openstack.org/482203 just now too, so once those are confirmed working inc0 should be able to recheck his change to use the proxy for tarballs access15:03
inc0fungi: just let me know15:04
*** zzzeek has joined #openstack-infra15:04
fungiinc0: will do. we should be very close now15:04
*** makowals has quit IRC15:04
inc0cool, thanks!15:04
*** felipemonteiro has joined #openstack-infra15:07
*** felipemonteiro_ has joined #openstack-infra15:08
pabelangerfungi: did we update /etc/mirrors.sh with new proxy info?15:08
*** squid is now known as calebb15:09
fungipabelanger: oh, i guess we could. i didn't think to add anything to it15:09
pabelangerfungi: ya, we have both docker / rdo proxies listed today15:09
*** nicolasbock has joined #openstack-infra15:10
pabelangereg: NODEPOOL_RDO_PROXY15:10
pabelangerwe should likely add NODEPOOL_TARBALLS_PROXY15:10
pabelangerI can propose that now15:10
*** makowals has joined #openstack-infra15:10
fungiyeah, i just found where we're adding that15:10
openstackgerritMerged openstack-infra/system-config master: Move the per-region proxy caches into /var/cache  https://review.openstack.org/48222815:10
pabelangerfungi: cool15:10
fungigo for it, though inc0's patch is just hardcoding the reference to http://$NODEPOOL_MIRROR_HOST:8080/tarballs15:11
fungiso that kolla won't need to wait for new images15:11
fungi(new nodepool images, i should clarify)15:11
pabelangerright, they should be able to update once we get images online in a day or so15:11
openstackgerritMerged openstack-infra/system-config master: Increase per-region caching proxy capacity  https://review.openstack.org/48222915:12
*** felipemonteiro has quit IRC15:12
openstackgerritMerged openstack-infra/system-config master: Caching proxies for tarballs.openstack.org  https://review.openstack.org/48220315:13
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add NODEPOOL_TARBALLS_PROXY to configure_mirror.sh  https://review.openstack.org/48300615:14
*** isaacb has joined #openstack-infra15:16
*** slaweq_ has joined #openstack-infra15:19
*** prateek has joined #openstack-infra15:20
*** slaweq_ has quit IRC15:24
*** alexchadin has quit IRC15:25
*** makowals has quit IRC15:32
*** mdrabe has quit IRC15:33
*** ldnunes has quit IRC15:33
calebbpabelanger: hey, I was taking a look at updating the puppet install script to know how to install puppet 4, and was trying to update the fedora stuff but wanted to ask if the fedora stuff is still used/needed15:34
*** coolsvap has joined #openstack-infra15:34
*** Sukhdev has joined #openstack-infra15:34
*** yamahata has joined #openstack-infra15:35
pabelangercalebb: ya, we can likely drop that now, since we don't use fedora on control plane servers. The funny thing, is that is already setup to use puppet 415:36
pabelangercalebb: ubuntu and centos-7 are the operating systems we have15:36
calebbahh, heh, ok thanks!15:37
*** makowals has joined #openstack-infra15:37
*** LindaWang has quit IRC15:37
*** andreas_s has quit IRC15:38
*** mdrabe has joined #openstack-infra15:38
smcginnisA normal cinder stable core pointed out to me that they don't have +2 on Ocata branches.15:39
smcginnisI think I remember we do a limited group during RC. Was something missed after final cut?15:39
fungismcginnis: looks possible http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack/cinder.config#n1915:41
*** ccamacho has left #openstack-infra15:41
fungismcginnis: oh, though that's consistent across other projects' acls15:41
*** liusheng has quit IRC15:42
fungii think it has more to do with the release team's scripts to update group memberships on cinder-release-branch15:42
smcginnisfungi: Hmm. Does the */ocata* section just need to be removed?15:42
openstackgerritCaleb Boylan proposed openstack-infra/system-config master: (WIP) Get install_puppet.sh to install puppet 4  https://review.openstack.org/48195515:42
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Adjust MTUs in multinode workflow on RDO Cloud  https://review.openstack.org/48122215:43
smcginnisShould now fall under the normal ref/heads/stable/* where the cinder-stable-maint group is.15:43
*** liusheng has joined #openstack-infra15:43
fungismcginnis: i think that happens when the stable/pike section is created, but maybe ask in #openstack-release once the release team meeting concludes15:43
fungihttps://review.openstack.org/#/admin/groups/cinder-release-branch looks similar to https://review.openstack.org/#/admin/groups/nova-release-branch anyway15:44
smcginnisfungi: OK, will do. Thanks for the pointers.15:44
*** ldnunes has joined #openstack-infra15:45
fungismcginnis: oh! this showed up in my review queue today... https://review.openstack.org/48295115:46
*** jtomasek_ has joined #openstack-infra15:47
*** yamamoto has quit IRC15:48
*** yamamoto has joined #openstack-infra15:49
*** jcoufal has quit IRC15:49
*** yamamoto has quit IRC15:51
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix race in test_shadow  https://review.openstack.org/48301615:51
*** iyamahat has joined #openstack-infra15:51
*** lucasagomes is now known as lucas-afk15:52
*** jtomasek_ has quit IRC15:53
*** ihrachys has quit IRC15:54
*** ihrachys has joined #openstack-infra15:55
*** amotoki is now known as amotoki_away15:56
*** egonzalez has quit IRC15:57
*** amotoki_away is now known as amotoki15:59
*** amotoki is now known as amotoki_away15:59
*** Rockyg has joined #openstack-infra15:59
*** jamesden_ has quit IRC15:59
*** jascott1 has joined #openstack-infra16:00
*** jamesdenton has joined #openstack-infra16:01
*** esberglu has quit IRC16:04
*** jcoufal has joined #openstack-infra16:04
*** tesseract has quit IRC16:05
*** isaacb has quit IRC16:07
*** gcb has quit IRC16:09
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-roles master: Create executor-info role  https://review.openstack.org/48302216:09
fungiany available project-config-core reviewers for a second +2 and approval on https://review.openstack.org/482951 ? it fixes a bit of a stable/ocata branch review emergency since a couple steps in the release management process were performed too close together16:11
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add debug info about which executor a job is run from  https://review.openstack.org/48302316:11
*** vhosakot has joined #openstack-infra16:12
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-roles master: Create executor-info role  https://review.openstack.org/48302216:13
*** trown is now known as trown|lunch16:13
*** annegentle has joined #openstack-infra16:14
*** esberglu has joined #openstack-infra16:18
openstackgerritThierry Carrez proposed openstack-infra/project-config master: Remove stable/ocata ACL for os-apply-config  https://review.openstack.org/48302416:18
*** slaweq_ has joined #openstack-infra16:21
*** mat128 has quit IRC16:21
*** esberglu has quit IRC16:22
*** shardy has quit IRC16:25
*** slaweq_ has quit IRC16:25
*** rossella_s has quit IRC16:26
bkeroclarkb: so it looks like the SSHLauncher jenkins plugin added a default behavior of deny while connecting to hosts without known ssh host keys. They added a toggle for that, and I think I can add that as a launcher param, but it doesn't seem to be working. http://paset.openstack.org/show/615185/16:26
*** esberglu has joined #openstack-infra16:27
*** jpich has quit IRC16:27
bkerojeblair: would you still happen to remember enough jenkins_manager.py to know how to reference a non-string/int param?16:27
pabelanger#status log both mirrors in infracloud-chocolate and infracloud-vanilla replaced with 250GB HDD mirror flavors now.16:28
openstackstatuspabelanger: finished logging16:28
pabelangerfungi: clarkb: finished with mirror migration16:28
*** ihrachys has quit IRC16:28
pabelangerold server deleted too16:28
*** ihrachys has joined #openstack-infra16:29
fungipabelanger: thanks! i'm checking the proxy now16:29
pabelangerfungi: great16:30
fungiclarkb: is mirror.la1.citycloud in active use? i don't see it in cacti16:30
fungii should probably double-check our ansible inventory for mirrors i missed before we go further16:30
clarkbfungi: yes it should be16:31
clarkbla1 kna1 sto2 and lon116:31
*** ralonsoh has quit IRC16:32
jeblairbkero: no, but you might be able to use web developer tools to look at the web page to see what it does16:33
*** annegentle has quit IRC16:33
*** dizquierdo has quit IRC16:33
bkerojeblair: ah, fair point. I'll look at the outbound json packet16:34
*** annegentle has joined #openstack-infra16:34
*** Guest86208 is now known as melwitt16:36
bkero+"sshHostKeyVerificationStrategy":+{"stapler-class":+"hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy"16:37
bkero+"$class":+"hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy"}16:37
bkeroI'm guessing that's it. I need to make it a dict with a stapler class key16:37
fungianybody happen to know why `sudo ansible '*' --list-hosts` on puppetmaster.o.o doesn't list the infra-cloud mirrors?16:38
fungido we need to flush the dynamic inventory cache after replacing those instances?16:38
*** mdrabe has quit IRC16:39
*** raissa_ has quit IRC16:39
*** mdrabe_ has joined #openstack-infra16:39
*** Sukhdev has quit IRC16:41
*** claudiub has quit IRC16:43
fungideleting /var/cache/ansible-inventory/ansible-inventory.cache seems to have solved that16:44
*** mdrabe_ has quit IRC16:46
*** mdrabe_ has joined #openstack-infra16:46
*** jamesden_ has joined #openstack-infra16:48
*** jamesdenton has quit IRC16:50
*** yamamoto has joined #openstack-infra16:51
fungilooks like mirror.sto2.citycloud wasn't getting puppet updates due to an invalid cached hostkey on the puppetmaster16:53
fungifixing16:53
*** tosky_ has joined #openstack-infra16:53
*** jcoufal has quit IRC16:53
*** tosky is now known as Guest4517716:53
*** tosky_ is now known as tosky16:53
fungiokay, so luckily the only ones i missed were the citycloud mirrors and they all have a sufficiently large rootfs as well16:55
*** Guest45177 has quit IRC16:55
*** pcaruana has quit IRC16:56
*** jtomasek_ has joined #openstack-infra16:58
*** yamamoto has quit IRC17:02
fungipabelanger: i realized i should probably add space on mirror.regionone.tripleo-test-cloud-rh1 too, but openstack volume create is leaving me with a cinder volume in an error state17:02
fungipabelanger: any idea whether (1) this is supposed to work or (2) we should even care to make ample space for an apache proxy cache there?17:03
dhellmannfungi : do you remember how the git config is set up for the signing node? I'm seeing an error with the release jobs because gerrit doesn't like the email address being used by the proposal step http://logs.openstack.org/74/74b983ec535eb2939edf772a2237e16466cf7367/release-post/tag-releases/aa9abad/console.html17:06
dhellmannI don't remember if that's system data somewhere or if the script does it17:06
pabelangerfungi: ya, I don't think they have the space currently. However, I don't think we should care as I am helping tripleo deprecate that cloud in favor of thirdpary CI in rdocloud17:07
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job's project as implicit role project  https://review.openstack.org/48272617:07
funginormally the job would set the git configuration if memory serves... checking now17:07
fungidhellmann: ^17:07
fungipabelanger: okay, i'll just ignore it for now17:07
*** e0ne has quit IRC17:07
pabelangerfungi: ++17:07
*** iyamahat has quit IRC17:09
openstackgerritMerged openstack-infra/project-config master: Add networking-ovn job to ovsdbapp gate  https://review.openstack.org/48165017:11
fungidhellmann: nevermind. it's preconfigured on that node by puppet: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/signing_node.pp17:11
dhellmannok17:12
dhellmannso, I guess I need to change the email address used there?17:12
*** eharney has quit IRC17:12
fungidhellmann: yeah, well the job needs to _not_ change the e-mail address used to create the tag being pushed17:12
dhellmannoh, well, the error message says infra-root@openstack.org is registered and that's the email used17:12
*** eharney has joined #openstack-infra17:13
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Pass result data back from ansible  https://review.openstack.org/47944217:13
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Support relative urls in success-url  https://review.openstack.org/48134217:13
dhellmannhrm17:13
*** mat128 has joined #openstack-infra17:13
*** ramishra has quit IRC17:13
dhellmannfungi : I see no signs in the log for that job that it did try to change the email address17:13
*** mat128 has quit IRC17:14
dmsimardHey, asking on behalf of another user -- anyone know if it's possible to send a review over http (as opposed to over ssh) ?17:15
fungidhellmann: aha! the script is trying to push this commit to gerrit: https://git.openstack.org/cgit/openstack/manila/commit/?id=ee0e5d71ee385d2a6ecc26bc42c2958d427b650c17:15
dmsimardNetwork has egress ssh on lockdown17:15
fungidmsimard: yes, it should work. i think we even have directions for configuring git-review to use the https interface17:16
dmsimardfungi: okay I'll try and find them in the docs, thanks17:16
dhellmannfungi : oh, weird17:16
fungidmsimard: i'm trying to find it17:16
fungidhellmann: i wonder if the script is incorrectly trying to push the branch as a review?17:17
clarkbdmsimard: fungi there is old ml thread on how to do it to bypass great firewall of china17:17
dhellmannfungi : it does this: git push gerrit driverfixes/mitaka17:17
dhellmannafter creating the branch17:17
dhellmannthat has been working, but maybe not in this case?17:18
fungiclarkb: dmsimard: i think we incorporated a synopsis of it into one of our documents. thought it might be in the git-review docs bit not finding it17:18
*** trown|lunch is now known as trown17:18
*** eranrom has quit IRC17:19
dhellmannfungi : ee0e5d71ee385d2a6ecc26bc42c2958d427b650c does not exist in gerrit. I wonder how it got into the repo.17:19
fungiclarkb: dmsimard: aha! https://docs.openstack.org/infra/manual/developers.html#code-review17:19
dhellmannit's a merge commit, could that be why?17:19
*** weshay is now known as weshay|lunch17:20
openstackgerritMerged openstack-infra/project-config master: Update ACLs for stable/pike in prep for release  https://review.openstack.org/48295117:20
*** weshay|lunch is now known as weshay17:20
dmsimardfungi: that html anchor is bad but I found what you meant to link to :)17:21
dmsimardhttps://docs.openstack.org/infra/manual/developers.html#accessing-gerrit-over-https17:21
*** slaweq_ has joined #openstack-infra17:21
fungidmsimard: yep, sorry, i copied the wrong anchor17:21
fungidhellmann: i think i might know what's going on here17:22
*** claudiub has joined #openstack-infra17:22
dhellmannfungi : good; I'm stumped.17:22
*** Apoorva has joined #openstack-infra17:25
*** yamahata has quit IRC17:25
*** kjackal_ has joined #openstack-infra17:25
*** slaweq_ has quit IRC17:26
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacing lines and formatting to result dicts  https://review.openstack.org/48304217:27
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Add play recap back in to the end of plays  https://review.openstack.org/48304317:27
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Log execution phase and include information  https://review.openstack.org/48304417:27
dhellmannfungi : I'll revert the patch to create the branches, so I can re-revert it and run the jobs again after we've fixed whatever it is. https://review.openstack.org/#/c/48304017:30
*** ykarel has quit IRC17:30
fungidhellmann: still trying to run this down... when i git remote update in openstack/manila i don't see a mitaka-eol tag. something's screwy here17:30
dhellmannoh, weird17:31
dhellmannI have that tag on 211f691d5cc82b32c5d83b5ca4646523167a4e86 fwiw17:31
dhellmanndo we have a git repo sync issue?17:31
AJaegerfungi, http://git.openstack.org/cgit/openstack/manila/tag/?h=mitaka-eol17:31
*** mat128 has joined #openstack-infra17:31
fungiokay, once i did a git pull --tags i got it, but git remote update wasn't for some strange reason17:31
dhellmannodd17:32
funginow i have the object in question17:32
*** sambetts is now known as sambetts|afk17:32
fungidhellmann: okay, so i _think_ the issue at hand is that it's trying to create a new branch which includes commits not in any existing branch17:33
fungithus gerrit believes the merge commits in there are new commits17:33
dhellmannhmm17:33
dhellmannit creates the branch using 211f691d5cc82b32c5d83b5ca4646523167a4e8617:34
dhellmannand git branch -a --contains 211f691d5cc82b32c5d83b5ca4646523167a4e86 shows me origin/stable/mitaka17:34
dhellmannis there an origin/stable/mitaka still?17:34
fungidhellmann: only because you haven't pruned old (now deleted) branches17:34
dhellmannaha17:35
fungianyway, continuing my explanation, the release managers group doesn't have the required forgeCommitter permission to be able to push commits it wasn't the committer on17:35
dhellmanndoes that mean I couldn't even do this through the web UI?17:35
fungiyou probably could because that's not a git push operation17:35
dhellmannwe could change the eol process to ensure the driverfixes branch comes before the stable branch deletion (and we could validate that in the releases repo check job)17:36
dhellmannok, let me try to do it by hand and see if that works better17:36
fungias for forgeCommitter, normally that's only needed by users importing commits from outside gerrit (i.e. those in the project bootstrappers group like our new repo import automation)17:36
dhellmannyeah, if this is a sequencing error, I'm fine with addressing that through docs17:37
dhellmannand not changing the acls17:37
fungigerrit has special handling when you're pushing commits so that if your history contains commits you're not a committer on but they're already part of a change in gerrit or in the history of an existing branch it won't reject them17:38
dhellmannthe UI did let me create the branch17:38
fungihowever it doesn't seem to do the same when the commits are only in the history of tags17:38
fungiand yeah, i expected the webui would work here17:38
*** tosky has quit IRC17:38
fungiso if we can switch to api calls for this step later (maybe after the gerrit upgrade? can't remember if you can do that yet in 2.11) then the sequencing issue goes away17:39
*** adam_g has quit IRC17:39
dhellmannI think I didn't switch to API calls because it had to be able to submit a patch anyway, so it seemed like we'd be dealing with extra credentials for no real reason. it sounds like that was not a fully informed decision.17:40
dhellmannI will add that to the future list for queens17:40
*** adam_g has joined #openstack-infra17:41
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add new line between zuul_stream messages  https://review.openstack.org/48304817:41
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Add play recap back in to the end of plays  https://review.openstack.org/48304317:42
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacing lines and formatting to result dicts  https://review.openstack.org/48304217:42
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Log execution phase and include information  https://review.openstack.org/48304417:42
*** iyamahat has joined #openstack-infra17:43
gouthamrinteresting scrollback. learnt a thing about these scripts. thanks for resolving this dhellmann fungi , i was able to cherry-pick a change to the new branch17:44
dhellmanngouthamr : you'll want to update the .gitreview file in those branches, too17:45
dhellmannI may be able to run the script to do that, let me see17:45
gouthamroh, sure thing..17:46
*** annegentle has quit IRC17:47
dhellmanngouthamr : you'll want to update https://review.openstack.org/#/c/483052/ with a valid url for the constraints file17:49
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Fix race in test_shadow  https://review.openstack.org/48301617:49
dhellmanngouthamr : and https://review.openstack.org/483054 may need to be updated, too17:50
gouthamrdhellmann: yep thank you.. i'll update these.. I think jenkins will attempt to run all the jobs, i proposed a project-config change to limit those: https://review.openstack.org/#/c/482785/17:51
dhellmanngouthamr : sounds good. I'll leave it to you to take those over, but let me know if you run into trouble and I can help.17:51
gouthamrdhellmann: will do, thanks! :)17:52
*** gyee has joined #openstack-infra17:53
*** jaosorior has joined #openstack-infra17:54
*** yamahata has joined #openstack-infra17:54
openstackgerritMerged openstack-infra/project-config master: Remove stable/ocata ACL for os-apply-config  https://review.openstack.org/48302417:55
*** jascott1 has quit IRC17:57
*** electrofelix has quit IRC17:57
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Adjust MTUs in multinode workflow on RDO Cloud  https://review.openstack.org/48122217:58
*** rbrndt has quit IRC17:59
*** jamesden_ has quit IRC18:00
*** tosky has joined #openstack-infra18:01
*** jcoufal has joined #openstack-infra18:01
openstackgerritMerged openstack-infra/zuul-jobs master: Create configure-mirrors role  https://review.openstack.org/48259318:01
openstackgerritMerged openstack-infra/zuul-jobs master: Remove default value for mirror_host  https://review.openstack.org/48291518:01
openstackgerritMerged openstack-infra/zuul-jobs master: Replace hand-written slug with ansible_managed  https://review.openstack.org/48291618:01
*** jamesdenton has joined #openstack-infra18:01
dhellmannfungi : thanks for your help with that branching issue; I think we're all set now.18:02
*** jascott1 has joined #openstack-infra18:04
*** e0ne has joined #openstack-infra18:04
*** yamamoto has joined #openstack-infra18:04
*** jascott1 has quit IRC18:04
*** ykarel has joined #openstack-infra18:06
fungidhellmann: excellent!18:07
*** annegentle has joined #openstack-infra18:08
fungiinc0: i'm testing retrieving /tarballs/kolla/images/centos-binary-registry-master.tar.gz from http://mirror.iad.rax.openstack.org:8080 right now to make sure it retrieves and caches the entire file (it's your largest at the moment, weighing in at ~4.2g)18:09
*** eranrom has joined #openstack-infra18:10
fungiinc0: if that works, then you're probably safe to try rechecking your change18:10
fungii'll let you know shortly18:10
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config master: ironic: Add DEVSTACK_GATE_IRONIC_INSPECTOR=1 for inspector jobs  https://review.openstack.org/48305818:12
*** xarses has joined #openstack-infra18:13
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate master: Add ironic-inspector services to features.yaml  https://review.openstack.org/48305918:14
jlvillal^^ I am hoping that those two patches will fix the ironic-inspector grenade gate. Reviews appreciated.18:16
*** xarses_ has quit IRC18:16
jlvillalError we are seeing now: http://logs.openstack.org/85/480185/4/check/gate-grenade-dsvm-ironic-inspector-ubuntu-xenial/9790ec4/logs/grenade.sh.txt.gz#_2017-07-11_23_55_12_30118:16
*** ykarel has quit IRC18:19
jeblairpabelanger, fungi: i've seen several instances of chocolate nodes being very slow18:20
*** yamamoto has quit IRC18:20
jeblairpabelanger, fungi: a bunch of them were on zuulv3 jobs, so we wrote it off to our ongoing work to use mirrors in v3 jobs, but i just saw a zuulv2 job fail18:20
*** yamamoto has joined #openstack-infra18:21
jeblairhttp://logs.openstack.org/36/482636/3/gate/gate-zuul-python35/f4102c2/console.html18:21
*** slaweq_ has joined #openstack-infra18:22
*** Rockyg has quit IRC18:23
*** rlandy is now known as rlandy|brb18:24
*** yamamoto has quit IRC18:26
*** slaweq_ has quit IRC18:27
pabelangerjeblair: we did replace mirrors this morning, I wonder if that is impacting jobs18:30
pabelangeror AFS cache is not warmed up yet?18:30
jeblairpabelanger: that would mean the v3 jobs are slow because we're using upstream ubuntu, and the v2 jobs are slow because our mirror caches are cold.  that's a significant coincidence.18:31
pabelangerokay, infracloud-chocolate mirror is under a large amount of load18:32
pabelangerjeblair: agree, let me see if I can determine if we have a bad compute node18:32
jeblairpabelanger: want me to dig up some instance uuids?18:34
pabelangerhttp://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=553&page=2 seems a little spotty18:34
pabelangerjeblair: sure, that would be helpful18:34
*** kjackal_ has quit IRC18:35
jeblairpabelanger: some of the v3 timeouts were in vanilla18:36
*** jascott1 has joined #openstack-infra18:37
*** ldnunes has quit IRC18:37
pabelangerchocolate mirror is under more load then vanilla, that is for sure18:38
jeblairpabelanger: hrm, i wonder if we're just oversubscribed?18:39
jeblairzuul is running at full capacity18:40
*** efoley has quit IRC18:40
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Pass result data back from ansible  https://review.openstack.org/47944218:40
pabelangerjeblair: I think you might be correct, http://grafana.openstack.org/dashboard/db/nodepool-infra-cloud18:40
pabelangerwe are also getting a large amount of launch failures too18:41
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: TEST: Use v2 net-iso templates in ci  https://review.openstack.org/48306418:41
pabelangerjeblair: looks like we are up to 4+ hours on all pipeline now.  We look to be a full capacity on all our clouds18:42
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate master: WIP: Test patch to see if fixes ironic-inspector grenade  https://review.openstack.org/48306518:43
*** Sukhdev has joined #openstack-infra18:45
*** makowals has quit IRC18:46
*** kjackal_ has joined #openstack-infra18:46
*** kjackal_ has quit IRC18:47
fungiinc0: my download test through mirror.rax.iad resulted in the correct size and checksum, and seems to have increased the cache there by roughly the expected amount, so i thing you should be all set to move forward18:47
inc0fungi: recheck is triggered https://review.openstack.org/#/c/482217/518:49
jeblairpabelanger: i looked into why zuul's job queue is >15k.  it looks like it's mostly set_description jobs, which just aren't getting picked up because the launchers are so busy18:49
jeblair(that should be pretty harmless and we can ignore it)18:49
*** Swami has joined #openstack-infra18:50
pabelangerYa, guess everybody is trying land patches for pike18:50
pabelangerits been a while since I've see zuul this busy18:50
jeblairthat accounts for 12187 out of 16890 jobs, leaving 4703 actual jobs18:50
*** ldnunes has joined #openstack-infra18:50
fungiyikes, what happened around 16:00 utc to spike job activity so high?18:51
fungilooks like it just came up out of nowhere18:51
pabelangerthink we had a gate reset, I can see a large amount of nodes deleted just before it18:51
pabelangerbut ya, we launched about 650 nodes18:52
fungiyeah, there are several resets in the integrated queue18:52
jeblaira gate reset doesn't change the overall number of jobs in the system (or even nodes, since deleted==building)18:52
fungithat top change failed on a job timeout in infracloud-chocolate18:53
jeblairwe may want to dial back max-servers for those two clouds18:53
pabelangerya, I think that is a good idea18:53
fungiagreed, wondering if we have a cascade/chain reaction sort of situation going on18:53
pabelangerwell, we have been losing nodes over time in infracloud, I don't think we ever adjusted our max-server when we turned some of the servers off18:54
*** mdrabe_ has quit IRC18:54
pabelangerso, we likely need to re-calculate our capacity again18:54
fungiwell, the nova change causing the more recent reset in the integrated gate looks like it timed out on an ansible task after the main job completed18:55
fungiand happened in ovh-bhs118:55
jeblairabout 1000 jobs were queued between 15:30 and 15:4518:56
*** mdrabe has joined #openstack-infra18:56
*** annegentle has quit IRC18:57
*** rlandy|brb is now known as rlandy18:57
*** rbrndt has joined #openstack-infra18:57
pabelangerare we okay with reducing max-servers by 10 nodes for each infracloud region?18:58
*** baoli has quit IRC18:59
fungisounds like a fine place to start19:00
*** baoli has joined #openstack-infra19:00
jeblairpabelanger: how many online compute hosts do we have?19:00
*** baoli has quit IRC19:00
pabelangerjeblair: let me check19:01
fungilooking at http://logs.openstack.org/97/482497/1/gate/gate-nova-pep8-ubuntu-xenial/c8c6067/_zuul_ansible/ansible_log.txt all i can guess is the launcher was so congested it took ~35 minutes to start the post playbook after the job playbook was done19:01
fungiload average on the launchers is pretty high, though that's not unusual19:01
*** baoli has joined #openstack-infra19:01
*** annegentle has joined #openstack-infra19:02
pabelangerjeblair: infracloud-vanilla is reporting 40 compute nodes ATM19:03
*** coolsvap has quit IRC19:03
fungiactually, some of the launcher load averages look like they're starting to come down (05 and 06 in particular)19:04
pabelangerjeblair: of that, 38 are functional19:05
pabelangerlet me see if I can find out why that is19:05
fungisimilarly, looks like the zuul job queue and nodes in use are starting to come down as well19:06
*** annegentle has quit IRC19:06
jeblairoh, so it looks like we're neither ram or disk constrained there.  only cpu constrained?19:06
jeblairdropping by 10 servers (approx 10% of our current) sounds good19:07
fungiso guessing we had a major job spike around 15:30 and 15:45 as jeblair indicates, and by the time we started looking into it whatever caused it was mostly complete and so we were looking at the aftermath/backlog?19:07
pabelangerYa, 5 compute nodes in infracloud-vanilla appear to be unreachable via networking. I'll make a note to try to access them via ilo and see if they can be recovered19:08
pabelangerotherwise, we likely need feet on the ground to help troubleshoot19:08
jeblairfungi: yes.  the set_description jobs make it hard to tell what the actual queue level is at this point, but looks like they should dissipate soon.19:09
AJaegermriedem: could you review https://review.openstack.org/#/c/482962 , please?19:10
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Reduce infracloud choclate / vanilla max-servers by 10  https://review.openstack.org/48307119:11
pabelangerjeblair: fungi: ^19:11
mriedemok19:11
jeblairfungi, pabelanger: spot checking jobs launched in that timeframe, i don't see any smoking guns19:11
fungijeblair: maybe we had a lag in zuul acting on new events which was long enough that the event queue pileup resulted in the utilization spike we see on the graph?19:12
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add some indexes to allow efficient dashboard sql  https://review.openstack.org/48161419:14
*** e0ne has quit IRC19:14
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove function_cache variables from executor client  https://review.openstack.org/48289019:14
openstackgerritPatrick East proposed openstack-infra/devstack-gate master: Set cinder image cache limits  https://review.openstack.org/48307319:15
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Cleanup components doc  https://review.openstack.org/48263619:15
AJaegerteam, would appreciate a review on https://review.openstack.org/475300 (project-config changes for omni) - the change should work but it's sooo unusual that I'd like some of you review it19:15
jeblairfungi: maybe -- zuul was still active in the preceding 30 minutes, but not as active as normal.19:16
*** yamamoto has joined #openstack-infra19:16
*** nicolasbock has quit IRC19:16
pabelangerTIL: infracloud.io19:17
*** harlowja has quit IRC19:17
*** david-lyle has quit IRC19:17
jeblairAJaeger: is that an official project?19:17
*** david-lyle has joined #openstack-infra19:17
AJaegerjeblair: no19:18
jeblairpabelanger, fungi: we're now under capacity19:18
fungiyeah, it sloped off very quickly19:19
AJaegerjeblair: omni = "Multi and hybrid cloud management with OpenStack API." - they allow using Amazon with OpenStack API19:19
jeblairfungi: and yeah, i think we may have overloaded the launchers a bit, based on the playbook delay you found19:19
*** adisky__ has quit IRC19:20
jeblairfungi: the math says we might run up to 187 jobs on a launcher now, which may be a bit too much.  we may want to throw another 2-4 launchers at it.19:20
pabelangerYa, I think we could stand to use atleast another 2 for sure19:21
fungijeblair: bumping up to 10 dynamic launchers seems prudent19:21
fungior thereabouts19:22
jeblairAJaeger: okay.  i think most of the stuff we'd be concerned about (zuul-cloner) has been handled in earlier patchsets.  if they ever wanted to become an official project, this is something to be aware of.19:22
openstackgerritEric Kao proposed openstack-infra/project-config master: enable py35 gate for congress-dashboard  https://review.openstack.org/48307619:22
inc0fungi: cache seems to be working19:23
inc0what's invalidation period for this?19:23
fungijeblair: what's our target peak? not still 100 jobs per launcher i guess?19:23
*** slaweq_ has joined #openstack-infra19:23
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: WIP Testing changes required to move 3nodes job to quickstart  https://review.openstack.org/48307719:23
inc0technically we could invalidate when file changes on source19:23
jeblairfungi: probably more.  drat.  we really need a statsd metric for currently running jobs on each launcher.  :(19:23
AJaegerthanks, jeblair19:23
clarkbinc0: the cache itself is 24 hours.19:25
fungiinc0: whatever the default in apache 2.4.7 default is; vhost config here: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb19:25
clarkbfungi: 24 houra iic19:25
fungicool. i woulda had to look that up in the apache docs ;)19:25
inc0yeah that's totally fine19:25
pabelangerjeblair: I think I had a patch up for that a while back, but think it got -2. I can see if I can find it again19:25
inc0since we'll be moving to 24h updates of files anyway19:26
fungijeblair: zuul v4 will have nodepool elastically scaling nodepool launchers, right? ;)19:26
*** slaweq_ has quit IRC19:26
pabelangerfungi: yes please :D19:26
jeblairfungi: quite possibly.  :)  something something containers something.19:26
*** slaweq_ has joined #openstack-infra19:26
fungier, nodepool elastically scaling zuul launchers i mean19:26
fungiwhat, you want another build when there are already 1000 going? you're queued until an 11th launcher node is built and brought into service19:28
openstackgerritMerged openstack-infra/project-config master: Reduce infracloud choclate / vanilla max-servers by 10  https://review.openstack.org/48307119:28
openstackgerritEric Kao proposed openstack-infra/project-config master: enable py35 gate for congress-dashboard  https://review.openstack.org/48307619:29
*** e0ne has joined #openstack-infra19:29
jeblairfungi, pabelanger: so if we add 2 more, our ceiling will be 150/launcher.  that seems like a good nominal target for now.19:29
*** annegentle has joined #openstack-infra19:30
fungia fine place to start and then see if similar lags occur between playbooks or not19:30
fungiat least we have a fingerprint of sorts for misbehavior under excess load19:30
pabelangeragree, 150 seems fine to me19:31
fungiso we should be able to tell if it happens again19:31
jeblairthe ptg conflicts with The Linux Foundation's Open Source Summit North America which is being keynoted by Joseph Gordon-Levitt.19:31
fungiyep, unfortunate. they announced their dates after we'd nailed down our contracts aiui19:32
pabelangerfungi: jeblair: do either of you want to add the new launchers? I don't might turning them up if needed19:32
jeblairpabelanger: i was about to try to find some food, so if you're up for it, that'd be great19:32
fungisimilar to oscon announcing their date on top of our boston summit after we'd already booked the venue19:32
fungipabelanger: go for it!19:32
pabelangerk, let me top up my coffee first19:34
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Correct run job parameter documentation  https://review.openstack.org/48297719:34
openstackgerritEric Kao proposed openstack-infra/project-config master: enable py35 gate for congress-dashboard  https://review.openstack.org/48307619:34
zara_the_lemur__(heh, just saw that open source summit email)19:34
*** camunoz has quit IRC19:35
AJaegerteam, change 482067 passes http proxy variables in tox.ini for project-config - is that something we should take?19:37
jeblairyeah, i mean, we're going to need to book a real a-lister for the ptg keynote if we want to compete.  :)19:37
zara_the_lemur__I actually had no idea who he was but could tell the email thought I did.19:38
jeblairAJaeger: i think it's ok.  hopefully will help some folks run tox before submitted project-config changes.  probably not many, but hey.19:38
jeblairzara_the_lemur__: third rock from the sun?19:39
fungiapparently he played snowden in the 2016 movie of the same name19:40
fungiwhich is presumably the draw there19:40
zara_the_lemur__I think I have actually seen none of the films on his imdb19:41
fungiwe got a keynote from the real edward snowden, they probably had to settle for the fake one19:41
jeblairyeah.  not a bad likeness.19:41
zara_the_lemur__(I guess I see about 1 film a year or something...)19:41
zara_the_lemur__haha19:41
zara_the_lemur__well the email had a big photo of him19:41
zara_the_lemur__so now I know what he looks like19:42
jeblairzara_the_lemur__: emails have photos?19:42
fungizara_the_lemur__: that's about how often i go out to watch a talkie anyway19:42
jeblairi go to the local movie palace whenever they show a 70mm print.  which, actually, is twice this month.  so kind of awesome.19:42
*** slaweq_ has quit IRC19:43
zara_the_lemur__I can take a screenshot of this email photo for those who have html such things firmly turned off19:43
zara_the_lemur__it's quite something19:43
*** wolverineav has quit IRC19:43
*** slaweq_ has joined #openstack-infra19:43
jeblairokay, not always.  i passed on batman vs superman.19:43
*** wolverineav has joined #openstack-infra19:44
zara_the_lemur__from what I've heard, that was the right decision19:44
zara_the_lemur__(and, well, also from the title)19:44
fungii figured it was just a godzilla vs mothra remake at heart19:44
zara_the_lemur__http://imgur.com/a/3cshz19:45
jeblairfungi: that i would see19:45
jeblairzara_the_lemur__, fungi: no glasses, so he must not be doing his snowden impression19:46
fungii think i have it on dvd, but yes a picture show like that really needs a bigger screen than my living room can hold19:46
zara_the_lemur__:D19:46
fungisorry, mothra vs godzilla (the '64 film). g vs m was the '92 one19:48
fungiit's unfortunately all too easy to confuse the two19:48
*** slaweq_ has quit IRC19:48
*** slaweq_ has joined #openstack-infra19:49
zara_the_lemur__I found a poster: http://vignette4.wikia.nocookie.net/godzilla/images/d/da/Mothra_vs._Godzilla_Poster_A.png/revision/latest?cb=2014051400454419:49
funginice!19:50
*** wolverineav has quit IRC19:50
*** mdrabe_ has joined #openstack-infra19:53
*** mdrabe_ has left #openstack-infra19:53
*** yamamoto has quit IRC19:54
*** florianf has quit IRC19:55
*** baoli has quit IRC19:56
pabelangerokay, starting on zuul-launchers19:56
fungii'm rolling forward with ptgbot changes now that things seem to have calmed down for a few minutes19:57
fungihopefully with a couple hours to focus i can get this squared away19:57
openstackgerritMerged openstack-infra/puppet-ptgbot master: Add instructions related to the PTGbot IRC bot  https://review.openstack.org/48093820:00
*** baoli has joined #openstack-infra20:00
*** baoli has quit IRC20:00
*** harlowja has joined #openstack-infra20:02
dmsimardpabelanger: hey, having problems bootstrapping a new nodepool DIB image. It looks like the image is taking slightly more than 30 minutes to build and it doesn't get the time to finish before nodepool starts another image build :/20:03
dmsimardpabelanger: can this be configured somewhere ?20:04
mtreinishpabelanger: that patch lgtm20:05
pabelangerdmsimard: do you have a build log? a new DIB only will get created if previous failed. How often are you setup to rebuild images?20:06
openstackgerritMerged openstack-infra/puppet-ptgbot master: Serve Web site content  https://review.openstack.org/48123020:07
*** baoli has joined #openstack-infra20:07
*** makowals has joined #openstack-infra20:08
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Add zl08.o.o and zl09.o.o  https://review.openstack.org/48308820:12
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add environment file for multinode tests on RDO Cloud  https://review.openstack.org/48122220:18
dhellmannfungi : checking in to see if you have any feedback on that script to move the old content around on the AFS servers?20:19
pabelangerdmsimard: which version of nodepool are you using again?20:22
openstackgerritMerged openstack-infra/system-config master: Install ptgbot on eavesdrop.o.o  https://review.openstack.org/48120120:22
dmsimardpabelanger: I'm trying something first, let you know if I figure out the issue20:22
fungidhellmann: that's https://review.openstack.org/481710 right? did you ever manage to get any feedback from the docs team on it?20:23
fungion the end result i mean20:23
*** jerryz has joined #openstack-infra20:24
*** jkilpatr has quit IRC20:25
pabelangerdmsimard: the old way was setting image-update value for crontab to trigger builds, it is possible you have it setup incorrectly20:25
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate master: Add ironic-inspector services to features.yaml  https://review.openstack.org/48305920:25
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate master: WIP: Test patch to see if fixes ironic-inspector grenade  https://review.openstack.org/48306520:25
jerryzclarkb: Hi, could you please add me in fuel-plugin-fortinet-core group? thanks!20:27
fungidhellmann: ignore my earlier question about docs team reviewers. for some reason i'd cached that change at a time prior to when they'd left comments/votes on it20:27
jerryzclarkb: my gerrit id is 6671 xyzjerry@gmail.com20:28
dhellmannfungi : I can adjust the path thing in the script, though I didn't necessarily intend for us to actually merge that change -- it was just an easy way to share the script20:28
fungidhellmann: oh, in that case it's good enough20:29
dhellmannfungi : ok, cool20:29
dhellmannfungi: what would be a good time to sit down and run that?20:29
sbezverkfungi: ping20:29
fungii can take a stab at it tomorrow... i'm in the middle of ptgbot things right at the moment20:29
dhellmannand then to run the script it produces, of course20:29
clarkbjerryz: another infra-root will need to do it. I am currently in transit20:29
*** e0ne has quit IRC20:29
dhellmannoh, sure, I didn't mean right now, just looking to set a rough time20:30
jerryzclarkb: ok. thanks.20:30
dhellmannfungi : perhaps after lunch tomorrow? unless earlier is better for you20:30
fungiafter lunch is fine20:31
sbezverkfungi: we have kolla-kubernetes gates broken and it seems related to constraints gate-kolla-kubernetes-python27-ubuntu-xenial/upper-constraints.txt20:31
dhellmannok. AJaeger wanted me to remove some of the older releases, so I'll do that and post a new version20:31
dhellmannfungi : thanks20:31
fungii'll want to remember to incorporate jeblair's suggestion of disabling the vos release for it and then vetting the read-write volume prior to accepting the results20:31
pabelangerfungi: jeblair: both zl08.o.o and zl09.o.o are online now. Do you mind reviewing https://review.openstack.org/#/c/483088/ to add them into system-config (firewall, cacti)20:32
jeblairpabelanger: we didn't have a zl08 before?20:32
pabelangerno, just zl01-zl0720:33
fungijeblair: we had zlstatic01 and zl01-zl07 i thought20:33
jeblairwow my math was all wrong then.20:33
jeblairwe were at 214/zl20:33
*** e0ne has joined #openstack-infra20:33
jeblairgoing down to 166 with this20:33
jeblairso neat.  better than we thought.  :)20:33
sbezverkall jobs are failing here http://logs.openstack.org/82/483082/1/check/gate-kolla-kubernetes-python27-ubuntu-xenial/32616ff/console.html#_2017-07-12_20_00_25_252101, appreciate if somebody could check this..20:33
*** makowals has quit IRC20:34
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: WIP Testing changes required to move 3nodes job to quickstart  https://review.openstack.org/48307720:34
fungisbezverk: we're not the requirements team. however i agree that looks like it _could_ be the result of a recent cliff release20:35
*** wolverineav has joined #openstack-infra20:35
jeblairsbezverk: http://git.openstack.org/cgit/openstack/cliff/ says there was one 2 days ago20:35
pabelangerYa, I see a new version on pypi from 2 days ago20:35
sbezverkfungi: any suggestion for my next step?20:35
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Add play recap back in to the end of plays  https://review.openstack.org/48304320:36
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacing lines and formatting to result dicts  https://review.openstack.org/48304220:36
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Log execution phase and include information  https://review.openstack.org/48304420:36
sbezverkright it started a couple of days20:36
sbezverkago20:36
*** wolverineav has quit IRC20:37
*** wolverineav has joined #openstack-infra20:37
fungisbezverk: maybe dtroyer has a suggestion since he was the one to propose that release20:38
fungihttps://git.openstack.org/cgit/openstack/cliff/tag/?h=2.8.020:38
*** kgiusti has quit IRC20:38
dhellmannsbezverk : I'll try to help you. Let's move to #openstack-sdks20:39
*** jkilpatr has joined #openstack-infra20:41
sbezverkdhellmann : thanks a lot20:43
inc0fungi: change merged20:44
inc0let me know if you get more issues with tarballs20:45
*** Goneri has quit IRC20:50
*** e0ne has quit IRC20:53
*** trown is now known as trown|outtypewww20:54
fungiinc0: thanks, i'll keep a close eye on it20:54
*** yamamoto has joined #openstack-infra20:54
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: WIP Testing changes required to move 3nodes job to quickstart  https://review.openstack.org/48307720:56
*** yamamoto has quit IRC21:01
*** vhosakot has quit IRC21:01
*** jtomasek_ has quit IRC21:02
*** vhosakot has joined #openstack-infra21:02
*** jcoufal has quit IRC21:03
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Log execution phase and include information  https://review.openstack.org/48304421:03
*** iyamahat_ has joined #openstack-infra21:03
*** iyamahat has quit IRC21:04
*** Sukhdev has quit IRC21:04
*** iyamahat_ has quit IRC21:07
*** iyamahat has joined #openstack-infra21:07
openstackgerritMonty Taylor proposed openstack-infra/puppet-openstackci master: Add support for zuul v3 job-output files  https://review.openstack.org/48310121:08
*** dave-mccowan has quit IRC21:09
*** bramwelt has quit IRC21:13
*** lennyb has quit IRC21:15
*** mat128 has quit IRC21:17
*** dprince has quit IRC21:20
*** thorst has quit IRC21:23
*** yland has joined #openstack-infra21:25
*** lennyb has joined #openstack-infra21:27
*** iyamahat_ has joined #openstack-infra21:31
*** iyamahat has quit IRC21:32
*** mat128 has joined #openstack-infra21:33
*** wolverineav has quit IRC21:36
*** wolverineav has joined #openstack-infra21:37
openstackgerritMerged openstack-infra/shade master: Improve doc formatting a bit  https://review.openstack.org/48281921:38
*** zzzeek_ has joined #openstack-infra21:42
*** yland has quit IRC21:44
*** Goneri has joined #openstack-infra21:46
openstackgerritJeremy Stanley proposed openstack-infra/puppet-ptgbot master: Pass vhost_name into the vhost resource  https://review.openstack.org/48310721:49
fungiany infra-puppet-core up for a quick fix? ^21:49
*** armax has quit IRC21:51
*** felipemonteiro_ has quit IRC21:52
*** armax has joined #openstack-infra21:52
*** mattmceuen has joined #openstack-infra21:53
mordredfungi: looking21:54
*** eharney has quit IRC21:54
*** ihrachys has quit IRC21:54
mordred+321:54
fungithanks cmurphy and mordred!21:54
mordredspekaing of small fixes ... https://review.openstack.org/483101 is easy if anyone wants it :)21:54
fungilgtm21:55
*** armax has quit IRC21:56
pabelangerfungi: mordred: jeblair: clarkb: sorry for ping again, but 483088 is last step for new zuul-launchers. If you want to review today great, otherwise, likely hold off until morning to start new zuul-launchers21:58
*** Apoorva has quit IRC21:58
jeblairpabelanger: oh sorry, thanks for the reping21:59
fungiyep, same. juggling too many torches21:59
fungiapproved now21:59
jeblairmordred: why don't we gzip job-output.txt?21:59
* fungi needs to go juggle the dinner torch for a bit now21:59
jeblairmordred: maybe we can do that in the post-playbook?22:00
mordredjeblair: I thought about that too - we run in to that lovely chicken-and-egg of not being able to log our actions of gzipping the log file :)22:00
mordredjeblair: also - last week one of the files DID get gzipped - whichconfuses me22:01
jeblairmordred: cron gzipped it probably22:02
mordredjeblair: ah. that's why22:02
mordredhttp://logs.openstack.org/14/481014/5/check/9ce8405/22:02
mordredjeblair: that's the one with a gzipped log22:02
jeblairmordred: still, there's no escaping the fact that at some point, we have to stop logging.  i'm okay with the gzip being on the other side of that line.22:03
mordredkk22:03
jeblairmordred: (i think an advantage of zuulv3 is that we can actually put the gzip steps in the base post playbooks and maybe even get rid of the cron)22:03
*** jaosorior has quit IRC22:03
*** Apoorva has joined #openstack-infra22:04
jeblair(we can also enforce a max log size)22:04
openstackgerritMerged openstack-infra/puppet-ptgbot master: Pass vhost_name into the vhost resource  https://review.openstack.org/48310722:04
pabelangernice, 481014 is multi-node!22:04
*** Apoorva has quit IRC22:04
*** ldnunes has quit IRC22:05
openstackgerritMerged openstack-infra/system-config master: Add zl08.o.o and zl09.o.o  https://review.openstack.org/48308822:09
*** Apoorva has joined #openstack-infra22:10
*** wolverineav has quit IRC22:10
*** esberglu has quit IRC22:11
*** wolverineav has joined #openstack-infra22:13
*** mat128 has joined #openstack-infra22:13
openstackgerritMerged openstack-infra/project-config master: Remove gate-grenade-dsvm-multinode job  https://review.openstack.org/48296222:13
*** jamesdenton has quit IRC22:14
*** makowals has joined #openstack-infra22:14
*** jamesdenton has joined #openstack-infra22:14
*** zzzeek_ has quit IRC22:15
*** dprince has joined #openstack-infra22:16
*** zzzeek_ has joined #openstack-infra22:17
*** zzzeek has quit IRC22:18
*** jamesdenton has quit IRC22:19
*** wolverineav has quit IRC22:21
*** dprince has quit IRC22:22
*** slaweq_ has quit IRC22:23
*** wolverineav has joined #openstack-infra22:24
*** makowals has quit IRC22:28
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Correct run job parameter documentation  https://review.openstack.org/48297722:28
*** Goneri has quit IRC22:32
*** d34dh0r53 has quit IRC22:32
*** puiterwijk has quit IRC22:32
*** puiterwi1 has joined #openstack-infra22:32
*** tosky has quit IRC22:32
*** watersoul has quit IRC22:32
*** d34dh0r53 has joined #openstack-infra22:32
*** rm_work has quit IRC22:32
*** puiterwi1 is now known as puiterwijk22:32
*** tosky has joined #openstack-infra22:32
*** bobh has quit IRC22:32
*** rm_work has joined #openstack-infra22:32
openstackgerritMonty Taylor proposed openstack-infra/os-loganalyze master: Add processing for zuul phases  https://review.openstack.org/48311722:34
*** rhagarty has quit IRC22:35
*** mpjetta has quit IRC22:35
*** raginbajin has quit IRC22:35
*** yuval has quit IRC22:35
*** slagle has quit IRC22:35
mordredjeblair, pabelanger: ^^ there's a stab at the first part of consuming those phases from the zuul_stream output22:35
*** slagle has joined #openstack-infra22:35
*** sambetts|afk has quit IRC22:36
*** mpjetta has joined #openstack-infra22:37
*** szaher has quit IRC22:37
*** szaher has joined #openstack-infra22:37
*** yuval has joined #openstack-infra22:37
*** rhagarty has joined #openstack-infra22:37
*** sambetts_ has joined #openstack-infra22:37
*** watersoul has joined #openstack-infra22:38
*** claudiub has quit IRC22:38
*** mat128 has quit IRC22:38
*** mat128 has joined #openstack-infra22:39
*** raginbajin has joined #openstack-infra22:42
*** mat128 has quit IRC22:43
*** makowals has joined #openstack-infra22:44
*** thorst has joined #openstack-infra22:45
*** thorst has quit IRC22:45
*** rbrndt has quit IRC22:45
openstackgerritBrian Haley proposed openstack-infra/project-config master: Split Neutron check queue grafana dashboard  https://review.openstack.org/48311922:46
*** rlandy is now known as rlandy|bbl22:46
*** ihrachys has joined #openstack-infra22:46
jeblairmordred: couple of things inline on that.22:47
jeblairmordred: and flake8 caught a real thing too22:48
openstackgerritJeremy Stanley proposed openstack/ptgbot master: List requests as a requirement  https://review.openstack.org/48312022:48
*** makowals has quit IRC22:50
*** yamamoto has joined #openstack-infra22:52
*** wolverineav has quit IRC22:55
openstackgerritJeremy Stanley proposed openstack-infra/puppet-ptgbot master: Correct the db_filename path  https://review.openstack.org/48312422:55
*** tosky has quit IRC22:57
*** armax has joined #openstack-infra22:58
mordredjeblair: I like it when it does that122:59
*** mattmceuen has quit IRC23:00
*** annegentle has quit IRC23:01
*** wolverineav has joined #openstack-infra23:02
*** gordc has quit IRC23:02
openstackgerritJeremy Stanley proposed openstack-infra/puppet-ptgbot master: Make HTML and Javascript files world readable  https://review.openstack.org/48312523:10
*** d0ugal has joined #openstack-infra23:11
*** baoli_ has joined #openstack-infra23:16
*** baoli has quit IRC23:18
openstackgerritMonty Taylor proposed openstack-infra/os-loganalyze master: Add processing for zuul phases  https://review.openstack.org/48311723:20
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate master: Add ironic-inspector services to features.yaml  https://review.openstack.org/48305923:22
openstackgerritJohn L. Villalovos proposed openstack-infra/devstack-gate master: WIP: Test patch to see if fixes ironic-inspector grenade  https://review.openstack.org/48306523:22
*** wolverineav has quit IRC23:23
*** slaweq_ has joined #openstack-infra23:23
*** wolverineav has joined #openstack-infra23:23
*** mat128 has joined #openstack-infra23:26
*** baoli_ has quit IRC23:26
pabelangerwow, something odd with nodepool23:27
pabelanger300 ready nodes23:27
pabelangerodd23:28
pabelangerOh, I think I know23:28
pabelangerbecause zl08 and zl09 haven't started23:28
pabelangeroh, maybe not23:28
pabelangerHmm23:28
*** slaweq_ has quit IRC23:28
pabelangerugh, I forgot to add them to nodepool.yaml23:30
pabelangerk, playing them in shutdown mode23:30
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add zl08 and zl09 to nodepool.yaml  https://review.openstack.org/48312823:31
pabelangerinfra-root: does anybody mind +3 on ^, I forgot to update nodepool with the new zuul-launchers23:32
mordredpabelanger: done23:34
pabelangerty23:35
*** dhajare has quit IRC23:37
*** stakeda has joined #openstack-infra23:38
*** mattmceuen has joined #openstack-infra23:39
*** hongbin has quit IRC23:40
*** marst has quit IRC23:41
*** wolverineav has quit IRC23:44
*** wolverineav has joined #openstack-infra23:46
pabelangerokay, both zl08 and zl09 stopped. zuul rescheduled the jobs on those launchers, I'm cleaning up nodepool now23:48
jeblairpabelanger: what's to clean up?23:51
openstackgerritMerged openstack-infra/project-config master: Add zl08 and zl09 to nodepool.yaml  https://review.openstack.org/48312823:51
pabelangerjeblair: between both new launchers, we had approx 300 nodes in ready state. Which was resulting in no other launchers getting nodes assigned to them23:52
pabelangerand wasn't able to get new nodes assigned to a good launcher to land ^23:52
jeblairpabelanger: how did you stop zuul-launcher?23:53
pabelangerI used zuul-launcher stop23:53
jeblairoh :(  that's like the one time you don't want to do that.23:53
jeblairpabelanger: either 'zuul-launcher graceful' or 'zuul-launcher release' would have returned all the nodes to nodepool23:54
*** Swami has quit IRC23:54
*** s-shiono has joined #openstack-infra23:54
pabelangerAh23:54
jeblairperhaps we could have added release to stop as well23:54
pabelangerI was going to do graceful, but wasn't sure how long it would have taken23:54
jeblairpabelanger: no time at all if it wasn't running any jobs23:55
pabelangerjeblair: true23:55
pabelangeris there a downside not to release for stop?23:55
jeblairpabelanger: oh... i misunderstood23:55
jeblairpabelanger: they were running jobs but not getting marked used23:55
pabelangerright, some nodes were used, but didn't update nodepool properly23:56
jeblairnevermind then :)23:56
*** bobh has joined #openstack-infra23:56
pabelangerk23:56
*** mat128 has quit IRC23:57
*** Julien-zte has joined #openstack-infra23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!