| *** thorst has joined #openstack-infra | 00:02 | |
| *** thorst has quit IRC | 00:08 | |
| *** gmann has quit IRC | 00:11 | |
| *** gmann has joined #openstack-infra | 00:11 | |
| *** dingyichen has joined #openstack-infra | 00:15 | |
| *** sflanigan has joined #openstack-infra | 00:15 | |
| *** sflanigan has joined #openstack-infra | 00:15 | |
| *** thorst has joined #openstack-infra | 00:36 | |
| *** thorst has quit IRC | 00:36 | |
| *** mat128 has joined #openstack-infra | 00:37 | |
| *** funzo_ has quit IRC | 00:39 | |
| *** s-shiono has joined #openstack-infra | 00:40 | |
| *** mriedem has quit IRC | 00:42 | |
| *** mat128 has quit IRC | 00:45 | |
| *** zhurong has joined #openstack-infra | 00:48 | |
| *** cuongnv has joined #openstack-infra | 00:53 | |
| *** ijw has joined #openstack-infra | 00:59 | |
| *** liusheng has quit IRC | 01:04 | |
| *** liusheng has joined #openstack-infra | 01:09 | |
| *** gildub has joined #openstack-infra | 01:12 | |
| *** gouthamr has quit IRC | 01:13 | |
| *** gildub has quit IRC | 01:17 | |
| *** LindaWang has joined #openstack-infra | 01:24 | |
| *** cshastri has joined #openstack-infra | 01:26 | |
| *** ijw_ has joined #openstack-infra | 01:28 | |
| *** ijw has quit IRC | 01:31 | |
| *** ijw has joined #openstack-infra | 01:32 | |
| *** dave-mccowan has joined #openstack-infra | 01:33 | |
| *** gildub has joined #openstack-infra | 01:35 | |
| *** xarses_ has joined #openstack-infra | 01:36 | |
| *** ijw_ has quit IRC | 01:36 | |
| *** hongbin has joined #openstack-infra | 01:36 | |
| *** xarses_ has quit IRC | 01:36 | |
| *** xarses_ has joined #openstack-infra | 01:37 | |
| *** xarses has quit IRC | 01:38 | |
| *** thorst has joined #openstack-infra | 01:52 | |
| *** thorst has quit IRC | 01:56 | |
| *** bobh has joined #openstack-infra | 02:10 | |
| *** lbragstad has joined #openstack-infra | 02:10 | |
| *** ijw has quit IRC | 02:12 | |
| *** baoli has joined #openstack-infra | 02:21 | |
| *** baoli has quit IRC | 02:28 | |
| *** zzzeek_ has joined #openstack-infra | 02:30 | |
| *** MatthewLi has joined #openstack-infra | 02:33 | |
| *** ijw has joined #openstack-infra | 02:36 | |
| *** ijw_ has joined #openstack-infra | 02:37 | |
| openstackgerrit | chenzhiying proposed openstack-infra/project-config master: enable service q-fwaas-v2 to test firewall scenrio https://review.openstack.org/481986 | 02:39 |
|---|---|---|
| *** ijw has quit IRC | 02:41 | |
| *** ijw__ has joined #openstack-infra | 02:41 | |
| *** zhurong has quit IRC | 02:43 | |
| *** jamesmcarthur has joined #openstack-infra | 02:44 | |
| *** ijw_ has quit IRC | 02:44 | |
| *** gildub has quit IRC | 02:45 | |
| *** ijw__ has quit IRC | 02:46 | |
| openstackgerrit | chenzhiying proposed openstack-infra/project-config master: enable service q-fwaas-v2 to test firewall scenrio https://review.openstack.org/481986 | 02:48 |
| *** jamesmcarthur has quit IRC | 02:49 | |
| *** ijw has joined #openstack-infra | 02:56 | |
| openstackgerrit | chenzhiying proposed openstack-infra/project-config master: enable service q-fwaas-v2 to test firewall scenrio https://review.openstack.org/481986 | 02:59 |
| *** ykarel has joined #openstack-infra | 03:01 | |
| *** ijw has quit IRC | 03:01 | |
| openstackgerrit | chenzhiying proposed openstack-infra/project-config master: enable service q-fwaas-v2 to test firewall scenrio https://review.openstack.org/481986 | 03:05 |
| *** gcb has joined #openstack-infra | 03:12 | |
| *** bobh has quit IRC | 03:13 | |
| *** bobh has joined #openstack-infra | 03:17 | |
| *** bobh has quit IRC | 03:21 | |
| *** burgerk has joined #openstack-infra | 03:33 | |
| *** ramishra has joined #openstack-infra | 03:38 | |
| *** ijw has joined #openstack-infra | 03:41 | |
| *** yamamoto has joined #openstack-infra | 03:47 | |
| *** slaweq has joined #openstack-infra | 03:47 | |
| *** Dinesh_Bhor has joined #openstack-infra | 03:52 | |
| *** slaweq has quit IRC | 03:52 | |
| *** thorst has joined #openstack-infra | 03:53 | |
| *** gildub has joined #openstack-infra | 03:55 | |
| *** psachin has joined #openstack-infra | 03:56 | |
| *** thorst has quit IRC | 03:57 | |
| *** hongbin has quit IRC | 03:58 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Cleanup on ssh-agent failure https://review.openstack.org/481344 | 03:59 |
| *** armax has joined #openstack-infra | 04:01 | |
| *** armax has quit IRC | 04:03 | |
| *** lbragstad has quit IRC | 04:04 | |
| *** dave-mccowan has quit IRC | 04:05 | |
| *** adisky__ has joined #openstack-infra | 04:07 | |
| *** bobh has joined #openstack-infra | 04:17 | |
| *** ijw has quit IRC | 04:20 | |
| *** jascott1 has quit IRC | 04:28 | |
| *** gcb has quit IRC | 04:30 | |
| *** burgerk has quit IRC | 04:36 | |
| *** jascott1 has joined #openstack-infra | 04:43 | |
| *** stakeda has joined #openstack-infra | 04:46 | |
| *** zhurong has joined #openstack-infra | 04:46 | |
| *** ianychoi has joined #openstack-infra | 04:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add support for zuul.d configuration split https://review.openstack.org/473764 | 04:59 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add support for zuul.d configuration split https://review.openstack.org/473764 | 05:13 |
| *** pgadiya has joined #openstack-infra | 05:28 | |
| *** pratik has joined #openstack-infra | 05:30 | |
| pratik | AJaeger: Can you please review https://review.openstack.org/#/c/475300/? | 05:31 |
| *** prateek has joined #openstack-infra | 05:36 | |
| *** jbadiapa has joined #openstack-infra | 05:43 | |
| *** thorst has joined #openstack-infra | 05:54 | |
| *** thorst has quit IRC | 05:59 | |
| *** AJaeger has quit IRC | 06:03 | |
| *** AJaeger has joined #openstack-infra | 06:09 | |
| *** dfflanders has joined #openstack-infra | 06:09 | |
| AJaeger | prateek: what you're doing is really unusual, it would help if you could update the commit message to explain why you do stuff. What your right now say is obvious and misses all the important bits. Since it'S unusual, I'd like to see clarkb comment as well. | 06:12 |
| prateek | AJaeger, can you please let me know the patch we are talking about here ? | 06:13 |
| prateek | pratik, what AJaeger was saying is meant for you | 06:15 |
| *** gildub has quit IRC | 06:17 | |
| pratik | AJaeger: will update commit message | 06:18 |
| *** andreas_s has joined #openstack-infra | 06:20 | |
| *** isaacb has joined #openstack-infra | 06:22 | |
| *** MatthewLi has quit IRC | 06:24 | |
| *** jtomasek has joined #openstack-infra | 06:26 | |
| *** ociuhandu has quit IRC | 06:26 | |
| *** ociuhandu has joined #openstack-infra | 06:27 | |
| *** alexchadin has joined #openstack-infra | 06:27 | |
| *** cshen has joined #openstack-infra | 06:27 | |
| *** rvba has joined #openstack-infra | 06:28 | |
| *** rvba has quit IRC | 06:28 | |
| *** rvba has joined #openstack-infra | 06:28 | |
| *** dfflanders has quit IRC | 06:29 | |
| AJaeger | prateek, pratik sorry, for mistyping the nick | 06:32 |
| prateek | AJaeger, no worries | 06:32 |
| AJaeger | dmsimard: could you get me the information about packstack retirement, please? See http://lists.openstack.org/pipermail/openstack-dev/2017-July/119434.html | 06:34 |
| *** dhajare has joined #openstack-infra | 06:35 | |
| *** s-shiono has quit IRC | 06:38 | |
| openstackgerrit | Pratik Shah proposed openstack-infra/project-config master: Added changes to run unit tests for Omni project using Zuul https://review.openstack.org/475300 | 06:39 |
| *** claudiub has joined #openstack-infra | 06:39 | |
| *** MatthewLi has joined #openstack-infra | 06:43 | |
| AJaeger | fungi, any idea why translation proposal uses a py34 environment here: http://logs.openstack.org/periodic/karbor-propose-translation-update/c661ea8/console.html ? | 06:44 |
| *** coolsvap has joined #openstack-infra | 06:47 | |
| AJaeger | this is since 4 or 5 days the case - I don't see the failure in other repos. | 06:47 |
| *** rcernin has joined #openstack-infra | 06:47 | |
| *** e0ne has joined #openstack-infra | 06:48 | |
| chandankumar | AJaeger: hello | 06:49 |
| chandankumar | AJaeger: please have a look https://review.openstack.org/#/c/474168/ and i am confused where to add tox job? | 06:49 |
| AJaeger | chandankumar: search for tox in jenkins/jobs/projects.yaml and check the name of the tox job and how others do it. | 06:52 |
| chandankumar | AJaeger: so i will adding two patches one for tox job and another patch to remove existing job? | 06:55 |
| chandankumar | let me do it | 06:55 |
| *** isaacb has quit IRC | 07:00 | |
| *** ccamacho has joined #openstack-infra | 07:02 | |
| *** maciejjozefczyk has quit IRC | 07:05 | |
| *** amotoki_away is now known as amotoki | 07:11 | |
| *** harlowja has quit IRC | 07:11 | |
| AJaeger | chandankumar: you can do that in two steps - or in one. IF you do it in tow, stack them on top of each other | 07:13 |
| *** shardy has joined #openstack-infra | 07:14 | |
| *** jascott1 has quit IRC | 07:14 | |
| *** jascott1 has joined #openstack-infra | 07:24 | |
| *** LindaWang has quit IRC | 07:34 | |
| *** LindaWang has joined #openstack-infra | 07:34 | |
| *** tesseract has joined #openstack-infra | 07:36 | |
| *** nicolasbock has joined #openstack-infra | 07:37 | |
| openstackgerrit | Omer Anson proposed openstack-infra/irc-meetings master: Dragonflow meetings to alternate bi-weekly https://review.openstack.org/482034 | 07:40 |
| *** masber has quit IRC | 07:40 | |
| openstackgerrit | chenzhiying proposed openstack-infra/project-config master: enable service q-fwaas-v2 to test create-and-list-firewallgroup scenrio https://review.openstack.org/482037 | 07:46 |
| *** e0ne has quit IRC | 07:47 | |
| amotoki | AJaeger: I would like to have your help on https://review.openstack.org/#/c/481850/ (it tries to add language index to individual pages of relnotes) | 07:47 |
| *** jpich has joined #openstack-infra | 07:47 | |
| amotoki | AJaeger: it is marked as WIP as I am not sure Babel is (or can be) installed. | 07:47 |
| *** sambetts is now known as sambetts|afk | 07:47 | |
| amotoki | copied from the horizon channel. (I post this in the wrong channel) | 07:47 |
| *** markmcd has quit IRC | 07:48 | |
| AJaeger | amotoki: we install Babel already in venv... | 07:51 |
| *** markmcd has joined #openstack-infra | 07:52 | |
| AJaeger | amotoki: ask ianychoi and dhellmann to review as well, please | 07:52 |
| AJaeger | amotoki: I'll try to review the next days... | 07:52 |
| *** masber has joined #openstack-infra | 07:53 | |
| *** namnh has joined #openstack-infra | 07:54 | |
| amotoki | AJaeger: thanks. will update some and add them to the reviewers. | 07:54 |
| *** thorst has joined #openstack-infra | 07:54 | |
| amotoki | AJaeger: there is no need to rush | 07:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/tripleo-ci master: Delete overcloud after collecting logs https://review.openstack.org/482039 | 07:56 |
| *** markmcd has quit IRC | 07:56 | |
| *** e0ne has joined #openstack-infra | 07:56 | |
| *** sambetts|afk is now known as sambetts | 07:57 | |
| *** markmcd has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** thorst has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:02 | |
| *** shardy has quit IRC | 08:06 | |
| *** shardy has joined #openstack-infra | 08:07 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Dragonflow meetings to alternate bi-weekly https://review.openstack.org/482034 | 08:10 |
| *** dizquierdo has joined #openstack-infra | 08:13 | |
| *** e0ne has quit IRC | 08:13 | |
| *** e0ne has joined #openstack-infra | 08:15 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 08:16 |
| dtantsur | hi infra! is there someone with devstack-gate core? could you please check https://review.openstack.org/#/c/480624/ ? | 08:17 |
| *** dingyichen has quit IRC | 08:19 | |
| *** jistr|off is now known as jistr | 08:25 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Add Heat grenade mutlinode gate job https://review.openstack.org/482048 | 08:27 |
| *** efoley has joined #openstack-infra | 08:31 | |
| *** afazekas|away is now known as afazekas | 08:34 | |
| *** gildub has joined #openstack-infra | 08:35 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 08:36 |
| *** jgrassler has quit IRC | 08:39 | |
| *** namnh has quit IRC | 08:39 | |
| *** gildub has quit IRC | 08:39 | |
| openstackgerrit | chenzhiying proposed openstack-infra/project-config master: enable service q-fwaas-v2 to test create-and-list-firewall-group scenrio https://review.openstack.org/481986 | 08:40 |
| *** cuongnv has quit IRC | 08:40 | |
| openstackgerrit | chenzhiying proposed openstack-infra/project-config master: enable service q-fwaas-v2 to test create-and-list-firewall-group scenrio https://review.openstack.org/481986 | 08:43 |
| *** dizquierdo has quit IRC | 08:43 | |
| *** spiffxp has joined #openstack-infra | 08:46 | |
| *** vipul has joined #openstack-infra | 08:47 | |
| *** SergK has joined #openstack-infra | 08:50 | |
| *** ralonsoh has joined #openstack-infra | 08:51 | |
| *** tosky has joined #openstack-infra | 08:52 | |
| *** pratik has quit IRC | 08:54 | |
| *** felipemonteiro has joined #openstack-infra | 08:56 | |
| apuimedo | yolanda: ping | 08:59 |
| *** felipemonteiro has quit IRC | 09:01 | |
| *** LindaWang has quit IRC | 09:01 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add job dependencies to status.json https://review.openstack.org/482061 | 09:03 |
| *** pblaho has joined #openstack-infra | 09:05 | |
| *** eranrom has quit IRC | 09:05 | |
| *** eranrom has joined #openstack-infra | 09:06 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Add settings for ovb in rdocloud https://review.openstack.org/480950 | 09:18 |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Pass environment variables of proxy to tox https://review.openstack.org/482067 | 09:21 |
| *** thorst has joined #openstack-infra | 09:23 | |
| *** ociuhandu has quit IRC | 09:27 | |
| *** thorst has quit IRC | 09:28 | |
| *** dizquierdo has joined #openstack-infra | 09:28 | |
| arturb | hi all. I would like to create a new repository on the review.openstack.org. According to the documentation https://docs.openstack.org/infra/manual/creators.html I need to create a change in project-infra. And I have a question. Do I have to create first new repository on PyPi or maybe the gates will create it automatically? | 09:33 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: neutron-fwaas: Add a non-voting job to run networking-midonet unit tests https://review.openstack.org/482074 | 09:37 |
| *** vipul has quit IRC | 09:38 | |
| *** spiffxp has quit IRC | 09:38 | |
| frickler | arturb: you need to create it yourself, see https://docs.openstack.org/infra/manual/creators.html#give-openstack-permission-to-publish-releases for details | 09:38 |
| *** slaweq has joined #openstack-infra | 09:39 | |
| arturb | frickler: thanks.currently I am struggling with pypi. I want to push project to testpypi but I receive only errors about authorization | 09:39 |
| *** spiffxp has joined #openstack-infra | 09:40 | |
| *** electrofelix has joined #openstack-infra | 09:41 | |
| *** electrofelix has quit IRC | 09:45 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: neutron-fwaas: Add a non-voting job to run networking-midonet unit tests https://review.openstack.org/482074 | 09:46 |
| *** isaacb has joined #openstack-infra | 09:46 | |
| *** vipul has joined #openstack-infra | 09:47 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: neutron-fwaas: Add a non-voting job to run networking-midonet unit tests https://review.openstack.org/482074 | 09:53 |
| *** electrofelix has joined #openstack-infra | 09:55 | |
| *** mat128 has joined #openstack-infra | 09:59 | |
| *** dtantsur is now known as dtantsur|bbl | 09:59 | |
| arxcruz | Hi all, I would like to know if any devstack-gate core can review https://review.openstack.org/#/c/475788/ | 10:00 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 10:03 |
| *** LindaWang has joined #openstack-infra | 10:09 | |
| *** SergK has quit IRC | 10:12 | |
| *** cshastri has quit IRC | 10:13 | |
| *** SergK has joined #openstack-infra | 10:16 | |
| *** LindaWang has quit IRC | 10:17 | |
| *** sanket_ has joined #openstack-infra | 10:19 | |
| *** sdague has joined #openstack-infra | 10:21 | |
| *** stakeda has quit IRC | 10:29 | |
| *** isaacb has quit IRC | 10:41 | |
| *** xinliang has quit IRC | 10:53 | |
| *** dtantsur|bbl is now known as dtantsur | 10:57 | |
| *** jkilpatr has joined #openstack-infra | 11:01 | |
| *** lucasagomes is now known as lucas-hungry | 11:03 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Adding tempest .testrepository directory in collect-logs https://review.openstack.org/482096 | 11:04 |
| *** xinliang has joined #openstack-infra | 11:05 | |
| *** alexchadin has quit IRC | 11:10 | |
| AJaeger | arturb: read the infra manual, it explains how to set up pypi | 11:12 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 11:13 |
| *** rhallisey has joined #openstack-infra | 11:16 | |
| *** wolverineav has joined #openstack-infra | 11:18 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 11:36 |
| *** Guest75308 is now known as rook | 11:37 | |
| *** ldnunes has joined #openstack-infra | 11:38 | |
| *** isaacb has joined #openstack-infra | 11:43 | |
| *** gordc has joined #openstack-infra | 11:45 | |
| *** danpawlik has quit IRC | 11:48 | |
| sanket_ | AJaeger, clarkb : Can you please review ? Resolved all review comments. Added changes to run unit tests for Omni project using Zuul https://review.openstack.org/#/c/475300/ | 11:53 |
| *** thorst has joined #openstack-infra | 11:53 | |
| *** danpawlik has joined #openstack-infra | 11:54 | |
| *** dave-mccowan has joined #openstack-infra | 11:55 | |
| *** jkilpatr has quit IRC | 11:57 | |
| *** pradk has joined #openstack-infra | 12:02 | |
| *** d0ugal has joined #openstack-infra | 12:04 | |
| *** d0ugal has quit IRC | 12:04 | |
| *** d0ugal has joined #openstack-infra | 12:04 | |
| *** jcoufal has joined #openstack-infra | 12:06 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 12:07 | |
| *** dprince has joined #openstack-infra | 12:07 | |
| *** lucas-hungry is now known as lucasagomes | 12:08 | |
| *** d0ugal has quit IRC | 12:12 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Stop translation the cisco horizon plug-in https://review.openstack.org/482117 | 12:13 |
| AJaeger | robcresswell, ianychoi ^ | 12:13 |
| robcresswell | AJaeger: +1 | 12:13 |
| robcresswell | Again, sorry for the time waste. I had not considered the translation aspect. | 12:13 |
| AJaeger | thanks, robcresswell | 12:13 |
| AJaeger | ianychoi: I'll make the repo read-only now | 12:13 |
| *** krtaylor_ has joined #openstack-infra | 12:18 | |
| *** krtaylor has quit IRC | 12:20 | |
| *** rlandy has joined #openstack-infra | 12:22 | |
| *** mwarad has joined #openstack-infra | 12:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-roles master: Add debug-info role from zuul-jobs as validate-host https://review.openstack.org/482119 | 12:24 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: OSA: Add periodic test for translations scenario https://review.openstack.org/482120 | 12:24 |
| *** baoli has joined #openstack-infra | 12:25 | |
| *** alexchadin has joined #openstack-infra | 12:25 | |
| *** baoli has quit IRC | 12:27 | |
| *** baoli has joined #openstack-infra | 12:28 | |
| *** bobh has quit IRC | 12:28 | |
| *** bobh has joined #openstack-infra | 12:28 | |
| *** jkilpatr has joined #openstack-infra | 12:29 | |
| *** mdrabe has joined #openstack-infra | 12:29 | |
| *** cshen has quit IRC | 12:30 | |
| mwarad | Folks, I'm not if this is correct forum to ask this... I would like to contribute as openstack tester, can anyone of you guide me which is the correct forum to discuss about this? Pretty much gone through the openstack.org launchpad links and wikis, if there are further info shared elsewhere, please share... thanks | 12:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use the validate-host role instead of the openstack-info role https://review.openstack.org/482123 | 12:32 |
| *** cshen has joined #openstack-infra | 12:33 | |
| *** zhurong has quit IRC | 12:36 | |
| mordred | mwarad: the #openstack-qa channel is probably a good place to start | 12:37 |
| *** sambetts has quit IRC | 12:38 | |
| *** bobh has quit IRC | 12:41 | |
| *** slaweq has quit IRC | 12:41 | |
| *** slaweq has joined #openstack-infra | 12:42 | |
| *** sambetts has joined #openstack-infra | 12:42 | |
| *** kgiusti has joined #openstack-infra | 12:44 | |
| *** camunoz has joined #openstack-infra | 12:44 | |
| *** jaypipes has joined #openstack-infra | 12:46 | |
| *** Guest9640 is now known as beagles | 12:47 | |
| *** LindaWang has joined #openstack-infra | 12:47 | |
| *** LindaWang has quit IRC | 12:48 | |
| *** trown|outtypewww is now known as trown | 12:52 | |
| *** d0ugal has joined #openstack-infra | 12:54 | |
| *** d0ugal has quit IRC | 12:54 | |
| *** d0ugal has joined #openstack-infra | 12:54 | |
| *** electrofelix has quit IRC | 12:58 | |
| *** d0ugal has quit IRC | 12:59 | |
| *** baoli has quit IRC | 13:00 | |
| *** funzo has joined #openstack-infra | 13:01 | |
| *** dhajare has quit IRC | 13:03 | |
| mwarad | mordred: Thanks, will follow up in that forum | 13:04 |
| *** masber has quit IRC | 13:06 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add web-based console log streaming https://review.openstack.org/463353 | 13:07 |
| *** d0ugal has joined #openstack-infra | 13:07 | |
| *** lbragstad has joined #openstack-infra | 13:08 | |
| *** sanket_ has quit IRC | 13:09 | |
| *** esberglu has joined #openstack-infra | 13:09 | |
| *** electrofelix has joined #openstack-infra | 13:10 | |
| *** eharney has joined #openstack-infra | 13:15 | |
| *** bobh has joined #openstack-infra | 13:22 | |
| *** sdague has quit IRC | 13:30 | |
| *** Goneri has joined #openstack-infra | 13:30 | |
| *** sshnaidm is now known as sshnaidm|mtg | 13:30 | |
| *** gcb has joined #openstack-infra | 13:31 | |
| *** sdague has joined #openstack-infra | 13:33 | |
| *** felipemonteiro has joined #openstack-infra | 13:34 | |
| *** zzzeek_ has quit IRC | 13:34 | |
| *** xyang1 has joined #openstack-infra | 13:34 | |
| *** askb has quit IRC | 13:35 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:36 | |
| *** felipemonteiro has quit IRC | 13:39 | |
| dmsimard | AJaeger: re: packstack. I will, sorry. | 13:44 |
| *** burgerk has joined #openstack-infra | 13:51 | |
| openstackgerrit | Lee Yarwood proposed openstack-infra/shade master: router: Ignore L3 HA ports when listing interfaces https://review.openstack.org/482153 | 14:00 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 14:00 |
| *** esberglu has quit IRC | 14:02 | |
| *** superdan is now known as dansmith | 14:02 | |
| *** esberglu has joined #openstack-infra | 14:02 | |
| openstackgerrit | Lee Yarwood proposed openstack-infra/shade master: Add debug to tox environment https://review.openstack.org/482154 | 14:03 |
| *** d0ugal has quit IRC | 14:03 | |
| *** baoli has joined #openstack-infra | 14:05 | |
| *** jkilpatr has quit IRC | 14:06 | |
| *** slaweq has quit IRC | 14:07 | |
| *** gyee has joined #openstack-infra | 14:07 | |
| *** rbrndt has joined #openstack-infra | 14:08 | |
| *** gouthamr has joined #openstack-infra | 14:10 | |
| stephenfin | Anyone have any idea why I can't access docs builds for pbr changes? For example https://review.openstack.org/#/c/481674 -> http://docs-draft.openstack.org/74/481674/2/check/gate-pbr-docs-ubuntu-xenial/7588747//doc/build/html/ | 14:11 |
| stephenfin | It just hangs | 14:11 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Set up DLM in n-g-s tempest job https://review.openstack.org/482155 | 14:11 |
| stephenfin | Oh, it's not just that project https://review.openstack.org/#/c/478473/ http://docs-draft.openstack.org/73/478473/6/check/gate-nova-docs-ubuntu-xenial/8fad85f//doc/build/html/ | 14:13 |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Adding tempest .testrepository directory in collect-logs https://review.openstack.org/482096 | 14:13 |
| *** hongbin has joined #openstack-infra | 14:14 | |
| dmsimard | stephenfin: aye, there seems to be an issue with the log server. | 14:14 |
| dmsimard | I can't access logs in general either right now | 14:14 |
| fungi | stephenfin: dmsimard: i just noticed the same when trying to pull up a log for something else. i'm troubleshooting now | 14:14 |
| dmsimard | fungi, clarkb ^ | 14:14 |
| fungi | load average is pretty high on static.o.o (the server serving that site) | 14:15 |
| fungi | tons of iowait | 14:15 |
| *** mriedem has joined #openstack-infra | 14:15 | |
| fungi | something is reading a ton of data from the server. it's flatlined at ~800mbps which is the bw cap for that flavor | 14:17 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all | 14:18 |
| *** sshnaidm|mtg is now known as sshnaidm | 14:18 | |
| fungi | going to be fun to try to sort out the responsible site and client(s) | 14:20 |
| *** beekneemech is now known as bnemec | 14:22 | |
| *** isaacb_ has joined #openstack-infra | 14:22 | |
| *** isaacb has quit IRC | 14:24 | |
| *** chlong_ has joined #openstack-infra | 14:24 | |
| *** alexchadin has quit IRC | 14:25 | |
| *** pgadiya has quit IRC | 14:25 | |
| *** isaacb_ has quit IRC | 14:26 | |
| *** zzzeek_ has joined #openstack-infra | 14:28 | |
| *** isaacb has joined #openstack-infra | 14:28 | |
| *** psachin has quit IRC | 14:28 | |
| *** zhurong has joined #openstack-infra | 14:30 | |
| dmsimard | fungi: fwiw, also unable to reach releases.openstack.org | 14:30 |
| *** ykarel has quit IRC | 14:30 | |
| fungi | dmsimard: yep, same server | 14:30 |
| dmsimard | I guess it is served out of static too. | 14:30 |
| dmsimard | okay | 14:30 |
| *** armax has joined #openstack-infra | 14:31 | |
| dmsimard | fungi: worth a notice from the bot ? | 14:31 |
| fungi | looks like the bulk of the network traffic leaving that server via http(s) is for tarballs.openstack.org | 14:31 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add web-based console log streaming https://review.openstack.org/463353 | 14:32 |
| fungi | #static notice Sites served from static.openstack.org (logs, governance, releases, specs, tarballs, docs-draft) are currently unavailable due to a large volume of requests; the situation is currently under investigation | 14:32 |
| fungi | i don't think this is necessarily web traffic. i need to get another tool on here to profile comparative utilization of different sockets since the apache logs aren't adding up to nearly enough to saturate the pipe | 14:33 |
| *** dizquierdo has quit IRC | 14:34 | |
| cmurphy | oh good i just came here to ask about it | 14:35 |
| *** cshen has quit IRC | 14:39 | |
| *** dhajare has joined #openstack-infra | 14:40 | |
| *** cshen has joined #openstack-infra | 14:41 | |
| *** Apoorva has joined #openstack-infra | 14:43 | |
| *** trown is now known as trown|brb | 14:43 | |
| *** ianychoi has quit IRC | 14:44 | |
| *** annegentle has joined #openstack-infra | 14:45 | |
| *** xarses has joined #openstack-infra | 14:46 | |
| fungi | aha, iftop is the utility i was trying to remember | 14:46 |
| *** trown|brb is now known as trown | 14:47 | |
| dmsimard | iftop and iptraf and both good at what they do :) | 14:48 |
| *** priteau has joined #openstack-infra | 14:48 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: Add a static README for debugging quickstart logs https://review.openstack.org/482168 | 14:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Add per-build ssh key to base job https://review.openstack.org/481773 | 14:53 |
| *** zhurong has quit IRC | 14:56 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use the validate-host role instead of the openstack-info role https://review.openstack.org/482123 | 14:57 |
| *** rcernin has quit IRC | 14:57 | |
| *** armax has quit IRC | 14:58 | |
| *** jamesmcarthur has joined #openstack-infra | 14:58 | |
| *** isaacb has quit IRC | 14:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-roles master: Remove unused openstack-info roles https://review.openstack.org/482119 | 14:59 |
| *** yamahata has quit IRC | 14:59 | |
| clarkb | fungi: you might be able to isolate it looking at disk reads too | 14:59 |
| clarkb | though that might end up washing out as network is smaller straw | 15:00 |
| fungi | we are getting a lot of requests to the tarballs site for kolla registry tarballs in the 3.5gb a pop neighborhood | 15:00 |
| *** armax has joined #openstack-infra | 15:00 | |
| *** yamahata has joined #openstack-infra | 15:00 | |
| clarkb | fungi: my understanind was that all of those requests were supposed to go through the docker hub proxy? I guess if we don't have the publish to dockerhub bit done yet then that may be a problem | 15:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Pass result data back from ansible https://review.openstack.org/479442 | 15:02 |
| clarkb | the idea being images would be cached locally in clouds and not hit docker/tarballs often | 15:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Support relative urls in success-url https://review.openstack.org/481342 | 15:02 |
| mordred | clarkb: maybe the images are stale and so more jobs are fetching the content than normal? | 15:03 |
| mordred | fungi, clarkb: let me know what, if anything, I can do to help- I'm currently assuming that fungi is looking and more eyeballs will not necessarily directly help | 15:03 |
| fungi | well, the traffic has sloped off below the bandwidth cap for the moment and so things look back to normal | 15:05 |
| *** jkilpatr has joined #openstack-infra | 15:06 | |
| AJaeger | jhesketh: I've put up the collected late repo retire requests in http://lists.openstack.org/pipermail/openstack-infra/2017-July/005487.html . Will you retire these - or should I ask somebody else? | 15:06 |
| *** LindaWang has joined #openstack-infra | 15:06 | |
| fungi | argh, bandwidth has dropped because load average is up to ~135 now and the cpus are spending 90% of their time in iowait states | 15:08 |
| fungi | so no, not back to normal | 15:08 |
| fungi | i'm going to take apache offline there for a few minutes and let the dust settle while i try blocking access to the kolla tarballs | 15:09 |
| mordred | fungi: ++ | 15:09 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add image-id and image-name options to cloud-images https://review.openstack.org/474369 | 15:11 |
| fungi | infra-root: inc0: in have temporarily done this on the static.o.o server: `sudo chmod o-r /srv/static/tarballs/kolla/images/*.tar.gz` | 15:12 |
| fungi | 5-minute load on the server is back down under 5.00 now so i'm starting apache back up and we'll see if this cures the bandwidth consumption issue | 15:13 |
| *** chlong_ has quit IRC | 15:16 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Explicitly set REVIEW_RELEASE on master https://review.openstack.org/451445 | 15:18 |
| fungi | iowait is still nonzero but it's back to normal (~10%) levels for this system | 15:19 |
| *** vhosakot has joined #openstack-infra | 15:20 | |
| fungi | load average hovering around 2ish now | 15:20 |
| *** d0ugal has joined #openstack-infra | 15:20 | |
| *** d0ugal has quit IRC | 15:20 | |
| *** d0ugal has joined #openstack-infra | 15:20 | |
| fungi | bandwidth utilization has fallen back to normal as well | 15:20 |
| *** liusheng has quit IRC | 15:21 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Add Heat grenade mutlinode gate job https://review.openstack.org/482048 | 15:22 |
| *** liusheng has joined #openstack-infra | 15:22 | |
| *** dizquierdo has joined #openstack-infra | 15:23 | |
| *** jamesmcarthur has quit IRC | 15:25 | |
| AJaeger | team, would be nice to have https://review.openstack.org/482117 (stop translations for to be retired repo) and my last round of mitaka-eol changes reviewed today, please: https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:mitaka-eol | 15:25 |
| inc0 | fungi: good morning, let me get some caffeine, but our images causes problems? | 15:26 |
| *** jamesmcarthur has joined #openstack-infra | 15:26 | |
| *** ccamacho has quit IRC | 15:27 | |
| *** cshen has quit IRC | 15:28 | |
| fungi | inc0: looks like maybe an uptick in either image sizes or the number of jobs trying to use them, but basically the static site server was flooded with outbound traffic the majority of which was serving kolla registry tarballs to jobs | 15:28 |
| inc0 | well we did increase number of jobs significantly lately | 15:28 |
| fungi | i've temporarily made them all non-world-readable on the filesystem so apache can't serve them (which means you likely have job failures) | 15:28 |
| inc0 | yeah, that effectively turns off our gates | 15:29 |
| clarkb | iirc the plan was to serve the images via dockerhub through our caching proxies | 15:29 |
| clarkb | doing that would likely address this problem | 15:29 |
| inc0 | that was the plan, yes | 15:29 |
| fungi | but that has gotten the traffic volume back down from 800mbps to around 40mbps | 15:29 |
| *** ccamacho has joined #openstack-infra | 15:29 | |
| inc0 | yeah, well every job downloads this | 15:30 |
| inc0 | and we have a lot of jobs per patch | 15:30 |
| fungi | could we cache requests to tarballs.o.o on the per-provider proxies? | 15:30 |
| *** chlong_ has joined #openstack-infra | 15:30 | |
| fungi | i want to say we talked about it as a possibility before and there were no objections to adding that | 15:30 |
| inc0 | well I can't think of any issue | 15:31 |
| inc0 | maybe besides stale images every now and then but that's not a big problem, especially when we plan to upload them once per day | 15:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Fix syntax error in post-ssh playbook https://review.openstack.org/482184 | 15:32 |
| openstackgerrit | Andy McCrae proposed openstack-infra/project-config master: Propose nova-lxd job using OpenStack-Ansible https://review.openstack.org/482185 | 15:33 |
| jeblair | mordred, fungi: i broke things; can you approve https://review.openstack.org/482184 right quick like? | 15:33 |
| clarkb | fungi: yes we should be able to do that, similar to how we could cache pypi mirror requests | 15:33 |
| jeblair | clarkb: ^? :) | 15:33 |
| clarkb | fungi: I'm just wondering why we are in this situation in the first place if the plan was to do something different/ | 15:34 |
| fungi | jeblair: wow, significant whitespace strikes again! | 15:35 |
| jeblair | yeah, we should probably ansible-lint project-config | 15:36 |
| fungi | or even have some sort of schema validation | 15:37 |
| inc0 | clarkb: reason is we don't publish stuff to dockerhub yet | 15:37 |
| *** ramishra has quit IRC | 15:38 | |
| *** isaacb has joined #openstack-infra | 15:38 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Make sure bindep is on the node https://review.openstack.org/481817 | 15:39 |
| *** darvon has quit IRC | 15:40 | |
| *** isaacb has quit IRC | 15:40 | |
| *** isaacb has joined #openstack-infra | 15:41 | |
| *** e0ne has quit IRC | 15:41 | |
| jeblair | fungi: i think the best we can do is ansible-lint; i'm not sure there's a way to do schema validation of ansible without actually running it. but ansible-lint should catch some errors. also, it's not to hard to test changes to the base job by applying them to another job (i did that); it was just the reorg of that code into the base job that failed here. so if we follow that procedure, we're only likely to be bitten by errors ... | 15:42 |
| jeblair | ... similar to this, which should be caught by linting. | 15:42 |
| *** ccamacho has left #openstack-infra | 15:42 | |
| *** darvon has joined #openstack-infra | 15:42 | |
| fungi | ahh, yeah the self-testing jobs scenario means we can do sandbox jobs to vet a lot of the stuff which can't safely allow direct pre-merge testing | 15:43 |
| fungi | great point | 15:43 |
| inc0 | fungi: how can I help with this per provider cache? | 15:43 |
| fungi | inc0: you could try adding it to the proxy configuration we have for the mirror servers, but i should be able to get to it myself shortly (i'm just juggling several things at once unfortunately) | 15:44 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix syntax error in post-ssh playbook https://review.openstack.org/482184 | 15:45 |
| inc0 | np, I'm not sure how to do it so I'll leave it to you. I don't think anything has to change on kolla gates side right? It should just work | 15:46 |
| clarkb | you'll hve to point jobs at the proxy instead of tarballs.openstack.orf | 15:47 |
| fungi | inc0: your jobs will probably need to change to retrieve the images from the proxy | 15:47 |
| fungi | yeah, what clarkb just said | 15:47 |
| inc0 | and this address will be different per region right? and what will populate cache initially? | 15:48 |
| *** jkilpatr has quit IRC | 15:48 | |
| clarkb | yes and your jobs | 15:48 |
| fungi | requests to the cache will populate it since it's passthrough | 15:49 |
| jeblair | does anyone have $100,000 so we can register .orf? | 15:49 |
| fungi | inc0: there's a file in /etc/nodepool/ on the nodes you can source with variable definitions including one for the mirror/proxy server in its region | 15:49 |
| inc0 | ok, I'll start working on it | 15:50 |
| *** gongysh has joined #openstack-infra | 15:51 | |
| fungi | jeblair: i can try to convince the nearest regional airport here that they should register it and let us use it (it's the faa/iata code for the nORFolk airport) | 15:52 |
| fungi | i fly in and out of there often enough i should have some clout with the tech dept | 15:52 |
| *** LindaWang has quit IRC | 15:53 | |
| *** jascott1 has quit IRC | 15:53 | |
| jeblair | fungi: tell them they will have *at least* one customer at $10/year! | 15:53 |
| *** mika has quit IRC | 15:55 | |
| *** Apoorva has quit IRC | 15:55 | |
| *** baoli has quit IRC | 15:55 | |
| *** d0ugal has quit IRC | 15:56 | |
| *** Apoorva has joined #openstack-infra | 15:56 | |
| *** baoli has joined #openstack-infra | 15:56 | |
| dhellmann | fungi : I was holding off on approving any releases until the issue with the tarballs server was resolved. It seems like things have settled down for now and it is ok to proceed? | 15:57 |
| *** mika has joined #openstack-infra | 15:57 | |
| *** mika has joined #openstack-infra | 15:57 | |
| *** jamesmcarthur has quit IRC | 15:57 | |
| *** rhallisey is now known as rhallisey-biab | 15:58 | |
| *** rhallisey-biab has quit IRC | 15:59 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Service Create https://review.openstack.org/482194 | 16:00 |
| *** Apoorva has quit IRC | 16:01 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Service List https://review.openstack.org/482195 | 16:01 |
| *** ldnunes has quit IRC | 16:01 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Service Update https://review.openstack.org/482196 | 16:01 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Service Delete https://review.openstack.org/482197 | 16:01 |
| *** Julien-zte has quit IRC | 16:04 | |
| *** Julien-zte has joined #openstack-infra | 16:05 | |
| *** Julien-zte has quit IRC | 16:05 | |
| fungi | dhellmann: yes, it should be safe to resume normal operations there. traffic levels are back under control now that we've blocked access to kolla registry tarballs: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all | 16:05 |
| *** Julien-zte has joined #openstack-infra | 16:06 | |
| *** chlong_ has quit IRC | 16:06 | |
| *** Julien-zte has joined #openstack-infra | 16:06 | |
| *** Julien-zte has quit IRC | 16:07 | |
| *** Julien-zte has joined #openstack-infra | 16:07 | |
| *** Julien-zte has quit IRC | 16:07 | |
| *** Julien-zte has joined #openstack-infra | 16:08 | |
| *** Julien-zte has quit IRC | 16:08 | |
| *** Apoorva has joined #openstack-infra | 16:08 | |
| *** Julien-zte has joined #openstack-infra | 16:09 | |
| *** Julien-zte has quit IRC | 16:09 | |
| *** Apoorva has quit IRC | 16:10 | |
| *** Apoorva has joined #openstack-infra | 16:10 | |
| *** ijw has joined #openstack-infra | 16:10 | |
| *** Apoorva_ has joined #openstack-infra | 16:11 | |
| *** annegentle has quit IRC | 16:12 | |
| *** jkilpatr has joined #openstack-infra | 16:13 | |
| *** ldnunes has joined #openstack-infra | 16:13 | |
| *** Apoorva has quit IRC | 16:14 | |
| *** gongysh has quit IRC | 16:15 | |
| *** ijw has quit IRC | 16:15 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Caching proxies for tarballs.openstack.org https://review.openstack.org/482203 | 16:15 |
| fungi | clarkb: mordred: inc0: ^ | 16:15 |
| *** isaacb has quit IRC | 16:16 | |
| *** coolsvap has quit IRC | 16:16 | |
| inc0 | fungi: so it'll be available in https://github.com/openstack/kolla-ansible/blob/master/tools/setup_gate.sh#L12 NODEPOOL_MIRROR_HOST/tarballs? | 16:18 |
| openstackgerrit | Merged openstack-infra/shade master: Remove hard-coding of timeout from API https://review.openstack.org/481958 | 16:18 |
| *** jascott1 has joined #openstack-infra | 16:19 | |
| fungi | inc0: yes, e.g. http://$NODEPOOL_MIRROR_HOST:8080/tarballs/kolla/images/centos-source-registry-master.tar.gz | 16:20 |
| *** chlong_ has joined #openstack-infra | 16:20 | |
| inc0 | :8080? allright | 16:21 |
| fungi | yeah, the caching proxy is on 8080/tcp (and 8081/tcp for a couple things, perhaps only temporarily) | 16:21 |
| *** jpich has quit IRC | 16:22 | |
| fungi | we could probably collapse all those into the normal http(s) ports later. i assume it was done this way to avoid namespace collisions with things we might also mirror | 16:23 |
| fungi | since the proxy rules would shadow the filesystem paths | 16:23 |
| *** baoli has quit IRC | 16:23 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Rework bindep role to be more ansible and less shell https://review.openstack.org/481818 | 16:23 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Clean up debug-ansible role https://review.openstack.org/481819 | 16:24 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fail if traceroute doesn't work https://review.openstack.org/481949 | 16:24 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Move node debug information into a directory https://review.openstack.org/481944 | 16:24 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Rename debug-ansible to validate-host https://review.openstack.org/481826 | 16:24 |
| *** camunoz has quit IRC | 16:24 | |
| fungi | oh, looks like i should probably bump the CacheMaxFileSize value too | 16:25 |
| fungi | and the htcacheclean threshold | 16:29 |
| openstackgerrit | Attila Darazs proposed openstack-infra/puppet-openstackci master: Add a static footer for TripleO Quickstart logs https://review.openstack.org/482210 | 16:29 |
| *** jascott1 has quit IRC | 16:33 | |
| *** annegentle has joined #openstack-infra | 16:34 | |
| fungi | inventorying the mirror servers to see if we actually have space to do this | 16:34 |
| fungi | ouch, /srv/static/tarballs/kolla/images is currently 57G of content | 16:35 |
| *** shardy is now known as shardy_afk | 16:37 | |
| fungi | our dfw.rax and iad.rax mirrors, the worst off of the bunch, only have 15G of available space | 16:37 |
| *** harlowja has joined #openstack-infra | 16:37 | |
| fungi | but the only mirrors which actually have a large enough filesystem to accommodate caching that much data are the ovh ones | 16:38 |
| *** chlong_ has quit IRC | 16:38 | |
| fungi | the contents of /srv/static/tarballs/kolla/images/tmp are unlikely to need caching, but leaving that out still only drops the current need down to around 49G | 16:39 |
| fungi | which is too much for us to cache anywhere besides ovh without adding cinder volumes and/or moving the caches around on the filesystems | 16:40 |
| fungi | on top of that, we probably need to do something about the filesystem utilization for the tarballs site itself... it nearly tripled around the beginning of the year and is still steadily ratcheting upward, nearly 80% used now | 16:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Use the validate-host role instead of the openstack-info role https://review.openstack.org/482123 | 16:46 |
| clarkb | fungi :/ | 16:46 |
| *** slaweq has joined #openstack-infra | 16:48 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for keystone v3 tokens 401 bug 1702211 https://review.openstack.org/482214 | 16:53 |
| openstack | bug 1702211 in OpenStack Identity (keystone) "test_password_history_not_enforced_in_admin_reset failed in tempest test" [Undecided,Confirmed] https://launchpad.net/bugs/1702211 | 16:53 |
| *** slaweq has quit IRC | 16:54 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Fix website style per openstackdocstheme changes https://review.openstack.org/482215 | 16:55 |
| *** jascott1 has joined #openstack-infra | 16:55 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-roles master: Remove unused openstack-info roles https://review.openstack.org/482119 | 16:56 |
| fungi | looks like i can free up the 150gb lv on static.o.o for the old mirror.o.o site since it's not been in use for something like 1.5 years. if i extend tarballs into that it'll get our utilization there back down to around 40% | 16:57 |
| fungi | doing that now | 16:57 |
| *** armax has quit IRC | 16:59 | |
| *** xyang1 has quit IRC | 16:59 | |
| fungi | #status log old mirror lv on static.o.o reclaimed to extend the tarballs lv by 150g | 17:01 |
| openstackstatus | fungi: finished logging | 17:01 |
| *** tesseract has quit IRC | 17:01 | |
| *** dhajare has quit IRC | 17:02 | |
| inc0 | fungi: https://review.openstack.org/#/c/482217/ | 17:02 |
| *** iyamahat has joined #openstack-infra | 17:05 | |
| *** lucasagomes is now known as lucas-afk | 17:06 | |
| fungi | inc0: that looks like it should work once i sort out the infrastructure capacity issues this raises | 17:08 |
| *** xyang1 has joined #openstack-infra | 17:09 | |
| inc0 | yeah, just let me know when proxies are up and running and we'll try this | 17:09 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Fix website style per openstackdocstheme changes https://review.openstack.org/482215 | 17:11 |
| fungi | infra-root: i was considering moving /opt/apache_cache to a dedicated cinder volume on all our mirror servers besides the two in ovh like we do for /var/cache/openafs, except this leaves infra-cloud unsolved still (the afs caches are in the rootfs there, do we have working cinder in infra-cloud?) | 17:12 |
| jeblair | fungi: no, but we can adjust the flavor as needed | 17:13 |
| fungi | okay, so this would imply a mirror rebuild in both infra-cloud regions i guess | 17:13 |
| jeblair | ya | 17:13 |
| *** sshnaidm has quit IRC | 17:17 | |
| *** jkilpatr has quit IRC | 17:19 | |
| *** yamamoto has quit IRC | 17:19 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for keystone v3 tokens 401 bug 1702211 https://review.openstack.org/482214 | 17:19 |
| openstack | bug 1702211 in OpenStack Identity (keystone) "test_password_history_not_enforced_in_admin_reset failed in tempest test" [Undecided,Confirmed] https://launchpad.net/bugs/1702211 | 17:20 |
| *** slaweq has joined #openstack-infra | 17:20 | |
| *** trown is now known as trown|lunch | 17:24 | |
| *** slaweq has quit IRC | 17:24 | |
| *** efoley has quit IRC | 17:30 | |
| *** Kevin_Zheng has quit IRC | 17:33 | |
| *** tosky has quit IRC | 17:37 | |
| *** rhallisey has joined #openstack-infra | 17:38 | |
| *** EricGonczer_ has joined #openstack-infra | 17:40 | |
| *** armax has joined #openstack-infra | 17:41 | |
| bkero | Does anyone know where the clouds.yaml profiles are defined? I'm attempting to set up a nodepool and looking to reference infra | 17:41 |
| *** EricGonczer_ has quit IRC | 17:41 | |
| Shrews | bkero: ~/.config/openstack or /etc/openstack or $HOME, iirc | 17:42 |
| bkero | Shrews: What does a profiles file look like though? | 17:42 |
| Shrews | bkero: oh, not $HOME, but $CWD: https://docs.openstack.org/os-client-config/latest/user/configuration.html#config-files | 17:43 |
| Shrews | bkero: that doc should answer that, too | 17:43 |
| bkero | Shrews: Read that, sadly it doesn't really answer what a 'profile: $X' reference does | 17:44 |
| bkero | It seems to be workarounds or quirks tables for specific clouds, but I can't see where it's defined | 17:44 |
| Shrews | bkero: 'profile' refers to one of the standard profile files in that repo. e.g., https://github.com/openstack/os-client-config/tree/master/os_client_config/vendors | 17:44 |
| *** ralonsoh has quit IRC | 17:44 | |
| bkero | Shrews: Great, that's what I was looking for. Thanks. | 17:45 |
| Shrews | bkero: profile == vendor | 17:45 |
| bkero | Got it, so it's splitting out some cruft from clouds.yaml | 17:46 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Caching proxies for tarballs.openstack.org https://review.openstack.org/482203 | 17:46 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Move the per-region proxy caches into /var/cache https://review.openstack.org/482228 | 17:46 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Increase per-region caching proxy capacity https://review.openstack.org/482229 | 17:46 |
| *** jamesmcarthur has joined #openstack-infra | 17:46 | |
| *** dprince has quit IRC | 17:48 | |
| bkero | I'm guessing my problem of nodepool-builder/launcher not uploading images is due to the lack of a 'region' entry for a cloud | 17:48 |
| *** dizquierdo has quit IRC | 17:48 | |
| *** yamahata has quit IRC | 17:49 | |
| *** iyamahat has quit IRC | 17:49 | |
| Shrews | bkero: what's reported in the builder log? | 17:50 |
| *** jamesmcarthur has quit IRC | 17:50 | |
| *** jamesmcarthur has joined #openstack-infra | 17:50 | |
| bkero | Shrews: The images are built and ready | 17:51 |
| *** slaweq has joined #openstack-infra | 17:51 | |
| fungi | bkero: if it helps, the default region name is usually "RegionOne" | 17:51 |
| bkero | fungi: That might help. 'openstack endpoint list' is hanging. Thank you. | 17:51 |
| *** annegentle has quit IRC | 17:56 | |
| bkero | Well, I set the region name, but it's still not uploading. :/ | 17:56 |
| *** jamesmcarthur has quit IRC | 17:57 | |
| *** annegentle has joined #openstack-infra | 17:59 | |
| bkero | See anything particularly wrong with this clouds.yaml or nodepool.yaml? http://paste.openstack.org/show/614955/ | 18:00 |
| *** dtantsur is now known as dtantsur|afk | 18:00 | |
| *** tosky has joined #openstack-infra | 18:01 | |
| *** jamesdenton has quit IRC | 18:01 | |
| bkero | $ OS_CLOUD=mycloud openstack image list # succeeds, so I don't think it's a creds problem. Nothing in /var/log/nodepool/*log | 18:02 |
| *** dmellado has quit IRC | 18:02 | |
| *** dmellado has joined #openstack-infra | 18:02 | |
| *** jamesdenton has joined #openstack-infra | 18:03 | |
| fungi | bkero: if it helps, the template for our upstream clouds.yaml is http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/clouds.yaml.erb | 18:03 |
| sdake | fungi i saw some mention of kolla sucking up resources with gating? | 18:04 |
| fungi | sdake: kolla ci jobs overran available bandwidth for tarballs.openstack.org trying to download registry tarballs | 18:04 |
| sdake | not sure what i can do to help - any suggestions are do you have it sorted out | 18:05 |
| fungi | i'm working on an alternative solution to be able to cache those closer to the job nodes, but we need to increase the cache capacity on them which involves a fair amount of manual work | 18:05 |
| *** iyamahat has joined #openstack-infra | 18:05 | |
| *** sambetts is now known as sambetts|afk | 18:05 | |
| fungi | the stack of patches is 481230, 482229, 482203, 482217 currently | 18:06 |
| sdake | so then - notohign I can really do to help? | 18:06 |
| fungi | and i'm digging into the manual sysadmin work for it now | 18:06 |
| fungi | sdake: nope, nothing i can think of at this stage | 18:06 |
| sdake | fungi you now wher eto find me :) | 18:06 |
| sdake | now/know | 18:06 |
| *** makowals has quit IRC | 18:07 | |
| *** sshnaidm has joined #openstack-infra | 18:08 | |
| *** electrofelix has quit IRC | 18:09 | |
| *** rlandy is now known as rlandy|brb | 18:10 | |
| *** Swami has joined #openstack-infra | 18:14 | |
| inc0 | sdake: I'm on it | 18:15 |
| sdake | inc0 roger | 18:15 |
| inc0 | you can review this https://review.openstack.org/#/c/482217/ | 18:15 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Caching proxies for tarballs.openstack.org https://review.openstack.org/482203 | 18:16 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Move the per-region proxy caches into /var/cache https://review.openstack.org/482228 | 18:16 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Increase per-region caching proxy capacity https://review.openstack.org/482229 | 18:16 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Increase Infra-cloud mirror flavor disk to 250GiB https://review.openstack.org/482236 | 18:16 |
| fungi | update for that stack, now starts with the infra-cloud flavor disk increase ^ | 18:16 |
| AJaeger | sdake: unless you shrink the images by an order of magnitude ;) | 18:17 |
| inc0 | AJaeger: doable - we just need to drop bunch of projects from big tent;) | 18:18 |
| *** dprince has joined #openstack-infra | 18:19 | |
| *** yamamoto has joined #openstack-infra | 18:19 | |
| AJaeger | ;) | 18:20 |
| inc0 | our tent is big, all I'm saying | 18:20 |
| fungi | infra-root: for 482236 is there anything i need to check to make sure we have sufficient capacity in infra-cloud to safely bump that flavor's disk size? | 18:21 |
| *** makowals has joined #openstack-infra | 18:22 | |
| * fungi works on earning his openstack operator wings | 18:22 | |
| *** Sukhdev has joined #openstack-infra | 18:24 | |
| *** yamahata has joined #openstack-infra | 18:24 | |
| *** yamamoto has quit IRC | 18:27 | |
| *** baoli has joined #openstack-infra | 18:31 | |
| *** makowals has quit IRC | 18:39 | |
| sdake | AJaeger ya - i'll tell you what - you join the BOD - you write the new governance for that - and kolla will drop the projects :) | 18:40 |
| sdake | inc0 big tent = varying poitical and tehcnical viewpoints | 18:40 |
| openstackgerrit | Sanket Sudake proposed openstack-infra/project-config master: Added changes to run unit tests for Omni project using Zuul https://review.openstack.org/475300 | 18:41 |
| sdake | inc0 big tent hs nothing to do with a circus :) | 18:41 |
| inc0 | I know, just sayin | 18:41 |
| inc0 | we build a lot of images | 18:41 |
| inc0 | north of 200 as of late | 18:41 |
| sdake | 216 i think :) | 18:41 |
| fungi | actually, the tc defines what it governs, the bod does not. the tc also tags a subset of that to identify things the bod is allowed to include in trademark programs | 18:41 |
| sdake | fungi to do what AJaeger suggeted would require a change to the foundation governance docs | 18:42 |
| sdake | (from my limited understandnig) | 18:42 |
| fungi | nah, the tc drops projects from governance (the "big tent") all the time | 18:42 |
| fungi | just dropped fuel last week | 18:42 |
| sdake | poor fuel | 18:42 |
| fungi | dropped app-catalog only a few months back | 18:43 |
| fungi | and openstack-ux | 18:43 |
| fungi | and akanda or whatever they renamed themselves to | 18:43 |
| sdake | fungi ya - I think i am miscommunicating but i'm in a bad mood | 18:43 |
| inc0 | I don't think we ever had image for app catalog | 18:43 |
| fungi | no need to alter foundational documents to allow the tc to drop projects from the "big tent" | 18:43 |
| inc0 | that being said, I'm pretty sure we never had images for fuel :P | 18:43 |
| sdake | fungi i think what i was getting at is a order of magnitutde change would reuqire a change in foundation bylaws | 18:44 |
| *** rlandy|brb is now known as rlandy | 18:44 | |
| sdake | fungi although IANAL | 18:44 |
| fungi | perhaps. the tc couldn't drop anything currently covered by a trademark program/refstack anyway without collaboration with the bod ;) | 18:44 |
| sdake | and not really keen to ever make a change to the foundation bylaws | 18:44 |
| sdake | fungi ya - there is a whole bunch of interlocked stuff in the bylaws | 18:44 |
| *** trown|lunch is now known as trown | 18:45 | |
| sdake | i've only witnesssed bylaws changed once | 18:45 |
| sdake | and it took 1 year | 18:45 |
| fungi | so for example the current -dev ml thread about "replacing" glance with glare... that would need discussing with the board | 18:45 |
| sdake | and it was like 10 sentences | 18:45 |
| fungi | because glance is covered by refstack and trademark program(s) | 18:46 |
| sdake | yup | 18:46 |
| sdake | i've read the bylaws several times | 18:46 |
| sdake | although standing on shoulders of giants - hard to see any improvements | 18:47 |
| fungi | i find i have to reread them pretty often as well | 18:47 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role Create https://review.openstack.org/482245 | 18:52 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role List https://review.openstack.org/482246 | 18:52 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: DONT REVIEW: Test commit to check periodic multinode jobs https://review.openstack.org/481222 | 18:53 |
| *** shardy_afk is now known as shardy | 18:59 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role List https://review.openstack.org/482246 | 19:01 |
| fungi | AJaeger: i finally found a moment to track down the issue with the translation updates using python3.4 instead of python2.7 | 19:02 |
| fungi | AJaeger: reinstalling tox with pip under python 2.7 seems to have solved it | 19:03 |
| * AJaeger hands fungi the OpenStack operator wings | 19:03 | |
| AJaeger | thanks, fungi! | 19:03 |
| fungi | AJaeger: keep an eye out for the next run and see if that's better | 19:03 |
| *** annegentle has quit IRC | 19:03 | |
| AJaeger | fungi: will try to remember tomorrow | 19:03 |
| fungi | #status log reinstalled tox on proposal.slave.o.o using python 2.7, as it had defaulted to 3.4 at some point in the past (possibly related to the pip vs pip3 mixup last month) | 19:04 |
| openstackstatus | fungi: finished logging | 19:04 |
| *** mat128 has quit IRC | 19:06 | |
| fungi | eep, signing01.ci seems to be in the same boat | 19:06 |
| *** raissa has quit IRC | 19:07 | |
| fungi | as does release.slave | 19:07 |
| fungi | betting mirror-update.slave is too | 19:07 |
| fungi | AJaeger: when did you say you saw it start? | 19:08 |
| AJaeger | fungi, 5 days ok | 19:08 |
| * AJaeger checks | 19:08 | |
| openstackgerrit | Yih Leong Sun proposed openstack-infra/system-config master: Enterprise WG is using UC mailling list now. The enterprise-wg maillist is no longer needed. https://review.openstack.org/482251 | 19:08 |
| AJaeger | fungi, http://status.openstack.org/openstack-health/#/job/karbor-propose-translation-update | 19:08 |
| AJaeger | starting on the 6th of July | 19:08 |
| AJaeger | and it's only karbor! | 19:08 |
| fungi | this may be unrelated to the python version default then | 19:09 |
| AJaeger | http://status.openstack.org/openstack-health/#/job/karbor-upstream-translation-update | 19:09 |
| AJaeger | also starting on the 6th | 19:09 |
| openstackgerrit | Yih Leong Sun proposed openstack-infra/system-config master: Remove Enterprise WG maillist. https://review.openstack.org/482251 | 19:09 |
| AJaeger | so, is karbor doing something strange? | 19:09 |
| fungi | because it looks like the failure message is "Could not import extension openstackdocsthemereno.sphinxext (exception: No module named 'openstackdocsthemereno') | 19:10 |
| fungi | " | 19:10 |
| AJaeger | ahh... | 19:10 |
| AJaeger | I can fix that ;) | 19:10 |
| fungi | i will work on cleaning up the default pythons on all our static nodes | 19:11 |
| AJaeger | fungi: I missed scrolling down to that line ;( | 19:11 |
| AJaeger | fungi: https://review.openstack.org/482252 - missing "," as separator ;( | 19:13 |
| *** jbernard has quit IRC | 19:13 | |
| *** jbernard has joined #openstack-infra | 19:13 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role Create https://review.openstack.org/482245 | 19:14 |
| *** makowals has joined #openstack-infra | 19:14 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Role List https://review.openstack.org/482246 | 19:15 |
| *** rhallisey is now known as rhallisey-biab | 19:16 | |
| * AJaeger signs off now | 19:17 | |
| fungi | have a good night AJaeger! | 19:17 |
| AJaeger | thanks, fungi | 19:20 |
| fungi | #status log similarly reinstalled tox on all other ubuntu-based zuul_nodes tracked in hiera (centos nodes seem to have been unaffected) | 19:22 |
| openstackstatus | fungi: finished logging | 19:22 |
| fungi | hopefully that's the last thing we have to clean up from that mess | 19:23 |
| *** prateek has quit IRC | 19:38 | |
| *** eharney has quit IRC | 19:39 | |
| *** mpjetta has joined #openstack-infra | 19:40 | |
| *** annegentle has joined #openstack-infra | 19:42 | |
| openstackgerrit | Monty Taylor proposed openstack-dev/cookiecutter master: Fix errors found from making a project https://review.openstack.org/482261 | 19:46 |
| mordred | AJaeger, fungi: I just created a repo with cookiecutter and found those errors ^^ | 19:46 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 19:48 |
| *** e0ne has joined #openstack-infra | 19:49 | |
| *** eharney has joined #openstack-infra | 19:54 | |
| *** d0ugal has joined #openstack-infra | 19:57 | |
| openstackgerrit | Monty Taylor proposed openstack-dev/cookiecutter master: Add support for storyboard https://review.openstack.org/482262 | 19:57 |
| bkero | Hm, for nodepool now I have image uploading, but it doesn't seem to be launching nodes to meet min-ready | 20:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make sure we always log the exit line https://review.openstack.org/473966 | 20:01 |
| bkero | There are also no logs in /var/log/nodepool referencing launching or gearman :/ | 20:02 |
| openstackgerrit | Rajath Agasthya proposed openstack-dev/cookiecutter master: Remove pbr version from setup.py https://review.openstack.org/482264 | 20:02 |
| mordred | fungi: uhm - so, I know you were cleaning up after a python3 related thing earlier ... | 20:05 |
| fungi | oh? | 20:05 |
| mordred | fungi: but I just created a brand new thing locally and ran tox -epy35 on it | 20:05 |
| mordred | and got this: | 20:05 |
| mordred | /home/mordred/src/git.openstack.org/openstack/os-service-types/.tox/py35/lib/python3.5/site.py:165: DeprecationWarning: 'U' mode is deprecated | 20:05 |
| mordred | dhellmann: ^^ it seems like PYTHONWARNINGS=default::DeprecationWarning in tox.ini may break py35 at the moment :( | 20:06 |
| fungi | looks very familiar | 20:07 |
| mordred | or - maybe I have an old version of tox/virtualenv locally - checking | 20:07 |
| dhellmann | is that in our cookiecutter? | 20:07 |
| mordred | it is | 20:07 |
| fungi | not actually related to what i was cleaning up as far as i'm aware though | 20:07 |
| mordred | ok. cool | 20:07 |
| mordred | mostly wanted to make sure I wasn't re-examining a gate failure you'd just examined :) | 20:07 |
| *** dizquierdo has joined #openstack-infra | 20:08 | |
| fungi | no, the failure which sent me down that rabbit hole actually turned out to be a missing comma in a config in karbor | 20:08 |
| *** jkilpatr has joined #openstack-infra | 20:08 | |
| mordred | fungi: oh awesome | 20:08 |
| fungi | which was breaking their translation updates by trying to import a nonexistent module as a sphinx extension | 20:08 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add web-based console log streaming https://review.openstack.org/463353 | 20:09 |
| fungi | but when looking at the logs it became apparent that tox was defaulting to using python3.4 on proposal.slave | 20:09 |
| mordred | fungi, dhellmann: ah - there was another thing that was causing that to be an error | 20:09 |
| mordred | or, causing tox to fail, that is | 20:09 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add support for zuul.d configuration split https://review.openstack.org/473764 | 20:09 |
| mordred | it's still a warning that gets printed, but not a fatal one | 20:09 |
| *** Goneri has quit IRC | 20:09 | |
| mordred | dhellmann: I'm guessing that should be a patch to virtualenv? | 20:10 |
| fungi | so totally unrelated to the error in karbor, i ended up discovering that all our static nodes (except the centor-7 mirror update one) had tox misinstalled defaulting to python3 instead of python2.7 | 20:10 |
| fungi | which i think was fallout from the pip3 symlink oopsie back in late may | 20:10 |
| dhellmann | mordred : I'm not sure what "U" mode is or what is producing that. | 20:10 |
| fungi | and just went unnoticed until now | 20:11 |
| dhellmann | you might be right, since it's coming from site.py | 20:11 |
| mordred | dhellmann: https://github.com/pypa/virtualenv/blob/263283be88061b965ead9ebff47c37021d7d3f62/virtualenv_embedded/site.py | 20:11 |
| fungi | U is a prefix for <type 'unicode'> in python 2.7 | 20:11 |
| fungi | type(U"foo") | 20:12 |
| mordred | it's for "universal newlines" in the open call | 20:12 |
| fungi | may also be a file mode | 20:12 |
| fungi | oh, right-o, that | 20:12 |
| dhellmann | aha | 20:12 |
| fungi | file mode for universal newlines | 20:12 |
| dhellmann | yeah, virtualenv bug | 20:12 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add networking-ovn job to ovsdbapp gate https://review.openstack.org/481650 | 20:13 |
| *** jamesmcarthur has joined #openstack-infra | 20:13 | |
| *** e0ne has quit IRC | 20:15 | |
| mordred | https://github.com/pypa/virtualenv/pull/1064 | 20:17 |
| mordred | I'm sure it won't be quite that easy | 20:17 |
| *** jkilpatr has quit IRC | 20:21 | |
| *** dizquierdo has quit IRC | 20:22 | |
| *** rhallisey has joined #openstack-infra | 20:30 | |
| odyssey4me | hmm, has zuul disappeared or something | 20:30 |
| odyssey4me | http://status.openstack.org/zuul/ is not getting updates | 20:31 |
| fungi | server's still running, but yeah http://zuul.openstack.org/status.json is returning a proxy error for me after a lengthy wait | 20:33 |
| fungi | i get no content back when trying a get /status.json on localhost:8001 either | 20:34 |
| *** nicolasbock has quit IRC | 20:34 | |
| fungi | checking service logs | 20:34 |
| *** rderose has joined #openstack-infra | 20:34 | |
| fungi | last entry in the debug log is from about 20 minutes ago | 20:35 |
| fungi | jeblair: ^ you might be interested | 20:35 |
| fungi | infra-root: ^ | 20:35 |
| *** vhosakot has quit IRC | 20:37 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make sure we always log the exit line https://review.openstack.org/473966 | 20:37 |
| *** vhosakot has joined #openstack-infra | 20:37 | |
| fungi | now i'm getting status.json responses again | 20:37 |
| *** makowals has quit IRC | 20:38 | |
| fungi | and we have recent debug logs once more | 20:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Actually run the unittests post playbooks https://review.openstack.org/482282 | 20:38 |
| fungi | there's a gap in the log from 2017-07-10 20:16:09,961 until 2017-07-10 20:36:45,652 | 20:39 |
| *** kgiusti has left #openstack-infra | 20:41 | |
| mordred | fungi: 20 minute gaps in logs are not awesome | 20:41 |
| *** thorst has quit IRC | 20:42 | |
| *** thorst has joined #openstack-infra | 20:45 | |
| fungi | lots of tracebacks for "AttributeError: 'str' object has no attribute '_ZuulGearmanClient__gearman_job'" but there seem to be plenty of those elsewhere in the log not just around the dead zone | 20:47 |
| *** jkilpatr has joined #openstack-infra | 20:47 | |
| jeblair | fungi: zuul is swapping | 20:47 |
| jeblair | i guess we still have a memory leak :( | 20:48 |
| *** jkilpatr has quit IRC | 20:48 | |
| *** jkilpatr has joined #openstack-infra | 20:48 | |
| jeblair | i will restart it | 20:48 |
| fungi | oh, yep, i glanced at free and only noticed the relatively large amount of free swap, didn't key in on the used swap value there | 20:48 |
| *** thorst has quit IRC | 20:49 | |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=393&rra_id=all | 20:49 |
| jeblair | that shows the 20 minute gap is due to swap activity | 20:49 |
| jeblair | i'll take that as an 'okay to restart' :) | 20:49 |
| fungi | yes, thanks! | 20:50 |
| fungi | i find it odd though that swap utilization seems to have started up back in early june and we didn't really notice any impact before now | 20:50 |
| fungi | we've been at/over 5gb swap used for a couple weeks solid now | 20:51 |
| jeblair | i think zuul pausing for a short enough time is probably not as noticeable | 20:51 |
| *** d0ugal has quit IRC | 20:52 | |
| fungi | though i guess it also crept up by a goodly bump today, topping out at nearly 10gb | 20:53 |
| *** rockyg has joined #openstack-infra | 20:54 | |
| jeblair | we're currently in the register 50 bazillion gear functions stage of zuul startup | 20:55 |
| *** hichihara has joined #openstack-infra | 20:58 | |
| *** thorst has joined #openstack-infra | 21:00 | |
| *** annegentle has quit IRC | 21:01 | |
| *** annegentle has joined #openstack-infra | 21:01 | |
| fungi | service log zuul service on zuul.openstack.org restarted to clear memory utilization from slow leak | 21:01 |
| fungi | #status log zuul service on zuul.openstack.org restarted to clear memory utilization from slow leak | 21:01 |
| openstackstatus | fungi: finished logging | 21:01 |
| jeblair | huh, why are we getting all these can_do packets? i thought we switched to mass_do | 21:03 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Include gate jobs in the cistatus page https://review.openstack.org/482316 | 21:03 |
| fungi | #status log 100gb cinder volume added and corresponding proxycache logical volume mounted at /var/cache/apache2 on mirrors for ca-ymq-1.vexxhost, dfw.rax, iad.rax, mtl01.internap, ord.rax, regionone.osic-cloud1 | 21:03 |
| openstackstatus | fungi: finished logging | 21:03 |
| vhosakot | the zuul is down? | 21:06 |
| inc0 | fungi: early june would correspond with us increasing gate coverage | 21:06 |
| fungi | vhosakot: it's on its way back up now | 21:07 |
| fungi | undergoing a controlled restart | 21:07 |
| vhosakot | fungi: cool, thanks for the info :) | 21:07 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Caching proxies for tarballs.openstack.org https://review.openstack.org/482203 | 21:09 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Move the per-region proxy caches into /var/cache https://review.openstack.org/482228 | 21:09 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Increase per-region caching proxy capacity https://review.openstack.org/482229 | 21:09 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/irc-meetings master: Add OpenStack Jenkins Job Builder team meeting https://review.openstack.org/481664 | 21:09 |
| *** trown is now known as trown|outtypewww | 21:11 | |
| *** askb has joined #openstack-infra | 21:12 | |
| *** eharney has quit IRC | 21:13 | |
| dhellmann | fungi : asettle and AJaeger have looked over the script in https://review.openstack.org/481710 to move content out of /developer into the new structure. I would like to try to run that this week, if you're going to have time to do it. Can you take a look at the script in the next day or two, and then we can talk about it and scheduling a run (assuming the script looks sane to you)? | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| dhellmann | the script to build the real move script takes a few minutes to run. I have no idea how long the actual moving will take. | 21:15 |
| fungi | dhellmann: you got it | 21:16 |
| jeblair | dhellmann, fungi: you can suspend the automatic release of the docs afs volume while running that, and verify everything looks good before releasing it. | 21:16 |
| fungi | jeblair: excellent idea | 21:16 |
| jeblair | dhellmann, fungi: i'd say you could roll back, but we'll also have docs publishing jobs running at the same time, so a roll back would lose that content. (though, of course, it can be regenerated with further docs jobs) | 21:16 |
| vhosakot | fungi: should we recheck jobs manually? or they'll get auto-rechecked? | 21:17 |
| jeblair | vhosakot: at this point if you don't see your change on the status page, you should recheck it | 21:17 |
| vhosakot | jeblair: cool, thanks | 21:17 |
| *** jkilpatr has quit IRC | 21:17 | |
| *** dklyle has joined #openstack-infra | 21:18 | |
| dhellmann | fungi : thanks | 21:18 |
| dhellmann | jeblair : transactional filesystems? what will they think of next?! :-) | 21:19 |
| *** david-lyle has quit IRC | 21:19 | |
| *** makowals has joined #openstack-infra | 21:19 | |
| *** tonytan4ever has joined #openstack-infra | 21:19 | |
| fungi | dhellmann: they'll go on to improve on the gopher protocol, and doom us all in the process | 21:19 |
| * dhellmann used to run a gopher server | 21:20 | |
| jeblair | infra is about to run a really big finger server. but we're going to hide it behind a websocket server so people think we're cooler than we are. | 21:21 |
| fungi | infra will make finger hip for the new generation | 21:21 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Fix ZuulWeb() invocation https://review.openstack.org/482321 | 21:22 |
| jeblair | "what's a finger hip?" "it's what we used to call a knuckle." | 21:22 |
| * fungi waxes the handlebar mustache on the finger protocol | 21:23 | |
| fungi | infra-root: for 482236 is there anything i need to check to make sure we have sufficient capacity in infra-cloud to safely bump that flavor's disk size? | 21:24 |
| *** calebb has quit IRC | 21:25 | |
| *** oanson has quit IRC | 21:28 | |
| *** priteau has quit IRC | 21:28 | |
| *** oanson has joined #openstack-infra | 21:29 | |
| *** jamesmcarthur has quit IRC | 21:29 | |
| jeblair | fungi: i think the scheduler should just deal with it when we create new instances? | 21:30 |
| *** jamesmcarthur has joined #openstack-infra | 21:30 | |
| *** calebb has joined #openstack-infra | 21:31 | |
| fungi | okay, but should be able to assume generally that we've got plenty of disk space in aggregate that taking up an extra 150gb (or 350 temporarily while i'm building the replacement) shouldn't cause us to stop being able to boot up to quota for test instances | 21:31 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Add Neutron QoS bandwidth limit rule commands https://review.openstack.org/482325 | 21:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fail execution if a role repo is a bare collection of roles https://review.openstack.org/482327 | 21:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Actually run the unittests post playbooks https://review.openstack.org/482282 | 21:35 |
| mordred | jeblair: also - it turns out we have the html bit already - it just wasnt' running :) | 21:35 |
| mordred | jeblair: ^^ | 21:35 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul-jobs master: Fix a playbook typo https://review.openstack.org/482329 | 21:36 |
| *** baoli has quit IRC | 21:36 | |
| *** mpjetta has quit IRC | 21:37 | |
| *** rderose has quit IRC | 21:37 | |
| *** zzzeek_ has quit IRC | 21:37 | |
| *** baoli has joined #openstack-infra | 21:37 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix a playbook typo https://review.openstack.org/482329 | 21:37 |
| jeblair | fungi: i'm not sure what our headroom is right now. but since space is partitioned by compute host, it's worth doing the math to see how many instances the host holding the mirror can hold, and add that to the number of instances the other hosts can hold to find out our max | 21:37 |
| fungi | jeblair: thanks, i'll see what i can figure out there | 21:38 |
| fungi | at this point i have all the other mirror hosts squared away with sufficient space to be able to be able to support up to 100gib apache proxy cache | 21:40 |
| *** zzzeek_ has joined #openstack-infra | 21:40 | |
| fungi | so just a matter of merging 482236 and building replacements for the two in infra-cloud | 21:41 |
| *** priteau has joined #openstack-infra | 21:41 | |
| *** hichihara has quit IRC | 21:42 | |
| *** camunoz has joined #openstack-infra | 21:46 | |
| *** priteau has quit IRC | 21:47 | |
| *** gongysh has joined #openstack-infra | 21:47 | |
| *** esberglu has quit IRC | 21:49 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Actually run the unittests post playbooks https://review.openstack.org/482282 | 21:51 |
| *** armax has quit IRC | 21:52 | |
| *** mpjetta has joined #openstack-infra | 21:53 | |
| *** slaweq has quit IRC | 21:55 | |
| *** slaweq has joined #openstack-infra | 21:55 | |
| *** mpjetta has quit IRC | 21:55 | |
| *** jascott1 has quit IRC | 21:56 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Actually run the unittests post playbooks https://review.openstack.org/482282 | 21:56 |
| *** jascott1 has joined #openstack-infra | 21:56 | |
| *** makowals has quit IRC | 21:57 | |
| *** mpjetta has joined #openstack-infra | 21:57 | |
| *** rderose has joined #openstack-infra | 21:57 | |
| *** jascott1 has quit IRC | 21:58 | |
| *** rderose has quit IRC | 21:58 | |
| *** jascott1 has joined #openstack-infra | 21:58 | |
| *** rderose has joined #openstack-infra | 21:58 | |
| *** slaweq has quit IRC | 21:59 | |
| *** slaweq has joined #openstack-infra | 22:00 | |
| jeblair | it's zuul meeting time in #openstack-meeting-alt | 22:00 |
| *** jascott1 has quit IRC | 22:00 | |
| fungi | so it is! | 22:01 |
| *** jascott1 has joined #openstack-infra | 22:01 | |
| *** dprince has quit IRC | 22:01 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Add neutron-dynamic-routing periodic job with ryu master https://review.openstack.org/482340 | 22:02 |
| *** thorst has quit IRC | 22:02 | |
| *** jascott1 has quit IRC | 22:03 | |
| *** jascott1_ has joined #openstack-infra | 22:03 | |
| *** xyang1 has quit IRC | 22:07 | |
| *** jascott1_ has quit IRC | 22:08 | |
| *** makowals has joined #openstack-infra | 22:09 | |
| *** burgerk has quit IRC | 22:11 | |
| amrith | hi infra-root ... any chance I can get your help cleaning up some old stuff in tarballs.openstack.org/trove/images | 22:12 |
| amrith | the stuff there is old, and causing more confusion than good | 22:12 |
| amrith | it is no longer maintained | 22:12 |
| amrith | fungi jeblair mordred ^^ | 22:12 |
| fungi | amrith: have a list of what needs removing? i'll take care of it now | 22:12 |
| *** deep-book-gk_ has joined #openstack-infra | 22:12 | |
| amrith | fungi , everything in the trove/images directory | 22:13 |
| amrith | a couple of days back I'd proposed (can't find the etherpad) a short note saying the stuff is no longer maintained, hence deleted. | 22:13 |
| amrith | thanks fungi | 22:13 |
| *** deep-book-gk_ has left #openstack-infra | 22:13 | |
| fungi | amrith: done! (moved into my homedir temporarily in case anyone decides they needed something from it after all) | 22:14 |
| fungi | freed up ~5gib too, so that's nice | 22:15 |
| *** slaweq has quit IRC | 22:16 | |
| amrith | fungi we can rebuild anytime we need | 22:18 |
| amrith | there's an experimental job which can do that :) | 22:19 |
| fungi | amrith: you want https://etherpad.openstack.org/p/Nnihrtxs4X stuffed into a text file at https://tarballs.openstack.org/trove/images/README or something? | 22:20 |
| * amrith looks | 22:20 | |
| fungi | that's the etherpad you linked last week | 22:20 |
| amrith | yes that one please | 22:20 |
| fungi | according to a grep of my channel logs | 22:20 |
| *** felipemonteiro_ has quit IRC | 22:22 | |
| openstackgerrit | Honza Pokorny proposed openstack-infra/project-config master: Create tempest-tripleo project https://review.openstack.org/481158 | 22:22 |
| *** shardy has quit IRC | 22:22 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add link to Zuul v3 docs to the README https://review.openstack.org/482349 | 22:23 |
| fungi | amrith: now displayed at https://tarballs.openstack.org/trove/images/README | 22:24 |
| *** makowals has quit IRC | 22:24 | |
| amrith | thanks fungi | 22:24 |
| fungi | any time! | 22:25 |
| * amrith bows | 22:25 | |
| *** wolverineav has quit IRC | 22:26 | |
| *** bobh has quit IRC | 22:27 | |
| *** zzzeek_ has quit IRC | 22:32 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 22:37 |
| *** slaweq has joined #openstack-infra | 22:37 | |
| *** jascott1 has joined #openstack-infra | 22:38 | |
| *** slaweq has quit IRC | 22:41 | |
| *** Julien-zte has joined #openstack-infra | 22:43 | |
| *** armax has joined #openstack-infra | 22:44 | |
| *** jamesmcarthur has quit IRC | 22:44 | |
| *** makowals has joined #openstack-infra | 22:48 | |
| *** zzzeek_ has joined #openstack-infra | 22:50 | |
| *** makowals has quit IRC | 22:52 | |
| *** gordc has quit IRC | 22:53 | |
| *** priteau has joined #openstack-infra | 22:54 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Instructions for registering up a new IRC bot nick https://review.openstack.org/482358 | 22:57 |
| *** priteau has quit IRC | 22:58 | |
| *** xarses has quit IRC | 23:00 | |
| *** rbrndt has quit IRC | 23:02 | |
| *** aeng has quit IRC | 23:03 | |
| *** Julien-zte has quit IRC | 23:04 | |
| *** Julien-zte has joined #openstack-infra | 23:06 | |
| *** rockyg has quit IRC | 23:08 | |
| *** jcoufal has quit IRC | 23:09 | |
| *** claudiub has quit IRC | 23:11 | |
| rm_work | err, is there a channel for nova? #openstack-compute is invite only? | 23:16 |
| rm_work | ah nm it's just -nova | 23:16 |
| *** thorst has joined #openstack-infra | 23:17 | |
| rm_work | and I have so many channels I didn't even see I was in it already >_< | 23:17 |
| *** makowals has joined #openstack-infra | 23:18 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 23:19 |
| *** aeng has joined #openstack-infra | 23:19 | |
| clarkb | it looks like we largely stopped running jobs a few hours ago? | 23:20 |
| *** rhallisey has quit IRC | 23:20 | |
| * clarkb has been afk all day but any idea on what changed that might cause that? | 23:20 | |
| clarkb | infra-root: ^ looks to possibly be nodepool related. I am going to go look at launcher logs | 23:20 |
| mordred | clarkb: there was a zuul issue | 23:21 |
| *** thorst has quit IRC | 23:21 | |
| *** zaneb has joined #openstack-infra | 23:21 | |
| mordred | clarkb: I don't have the full details - jeblair or fungi can fill you in | 23:22 |
| *** rderose has quit IRC | 23:22 | |
| clarkb | looks like it isn't corrected | 23:22 |
| clarkb | nodepool launcher is just reloading its config in a loop, maybe it has no gearman connection? | 23:23 |
| *** bobh has joined #openstack-infra | 23:24 | |
| *** jkilpatr has joined #openstack-infra | 23:24 | |
| mordred | clarkb: maybe its gearman didn't properly reconnect after the zuul restart? | 23:24 |
| clarkb | ya nodepool side seems to think it is connected, going to check zuul side now | 23:27 |
| *** bobh has quit IRC | 23:27 | |
| mordred | clarkb: did you restart nodepool again? | 23:27 |
| mordred | s/again// | 23:27 |
| jeblair | mordred, clarkb: i'm here | 23:28 |
| clarkb | mordred: no I haven't touched anything, just looking | 23:29 |
| mordred | clarkb: I basically only see activity in the log from citycloud-sto2 and rax-ord right now | 23:29 |
| clarkb | there is a discrpenecny in gearman connections between the two | 23:29 |
| clarkb | zuul sees 4 connections, nodepool 5, but only 3 appear to match up | 23:29 |
| mordred | raise NoConnectedServersError("No connected Gearman servers") | 23:30 |
| mordred | just got that | 23:30 |
| clarkb | tcp 0 0 23.253.73.160:39334 104.130.246.207:4730 ESTABLISHED 29874/python on nodepool is the launcher daemon and that connection doesn't show up on zuul | 23:30 |
| clarkb | mordred: maybe tcp is timing out then? | 23:31 |
| mordred | clarkb: weirdness is definitely happening - but I do not fully understand it | 23:33 |
| jeblair | File "/usr/local/lib/python2.7/dist-packages/gear/__init__.py", line 355, in sendAdminRequest | 23:33 |
| jeblair | raise TimeoutError() | 23:33 |
| clarkb | ya me either, but seems the gearman situation does seem fishy | 23:33 |
| jeblair | that is probably the cause | 23:33 |
| jeblair | that's probably kicking the gearman connection offline and causing the exception mordred saw | 23:33 |
| jeblair | (it will reconnect, and try again, and fail again) | 23:33 |
| *** bobh has joined #openstack-infra | 23:34 | |
| clarkb | jeblair: why would it fail again? | 23:34 |
| jeblair | clarkb: if it's continually timing out on the function list | 23:34 |
| *** Swami has quit IRC | 23:34 | |
| clarkb | ah | 23:35 |
| *** bobh has quit IRC | 23:35 | |
| clarkb | ya netstat confirms we seem to be cycling those | 23:38 |
| jeblair | time echo "status"|nc zuul.openstack.org 4730 >status | 23:38 |
| jeblair | real0m1.850s | 23:38 |
| jeblair | that doesn't seem exceptional. | 23:38 |
| clarkb | jeblair: possible regression in gear? | 23:38 |
| jeblair | clarkb: maybe, but i'm not finding a smoking gun yet. that makes me think the server is okay -- and the server is the thing that we just restarted.... | 23:39 |
| clarkb | jeblair: is status the admin requiest that is failing? what is the other one? | 23:39 |
| clarkb | but ya other clients being able to run commands would indicate server which is what restarted is functional | 23:39 |
| *** bobh has joined #openstack-infra | 23:39 | |
| jeblair | clarkb: yes, it's the one that lists 27378 functions. the other is workers, but nodpool doesn't call that. | 23:39 |
| clarkb | ah | 23:41 |
| *** hongbin has quit IRC | 23:42 | |
| *** bobh has quit IRC | 23:43 | |
| *** amotoki is now known as amotoki_away | 23:43 | |
| *** annegentle has quit IRC | 23:45 | |
| jeblair | i'm running a tcpdump on nodepool | 23:45 |
| clarkb | ok | 23:45 |
| openstackgerrit | Mike Perez proposed openstack-infra/statusbot master: Implement #thanks command in statusbot https://review.openstack.org/473534 | 23:46 |
| jeblair | okay, some job assignments are completing | 23:47 |
| jeblair | er node assignments | 23:47 |
| clarkb | I didn't change anything fwiw so not sure why | 23:48 |
| jeblair | i think some may still be failing though? | 23:48 |
| jeblair | and i don't know what the main thread is doing | 23:48 |
| mordred | jeblair: yah - I see some successes and some failures in the launcher debug log | 23:49 |
| mordred | like - some successes and osme raise NoConnectedServersError("No connected Gearman servers") | 23:49 |
| pabelanger | o/ | 23:49 |
| clarkb | and it continues to be rax ord and citycloud sto2 ? | 23:49 |
| mordred | look it's a pabelanger | 23:49 |
| clarkb | thats what I seem to see at least | 23:49 |
| jeblair | hrm, i wonder if something is fubar with their gear connection, as well as the mainloop | 23:50 |
| jeblair | i'm going to do a thread dump | 23:50 |
| pabelanger | indeed | 23:50 |
| *** bobh has joined #openstack-infra | 23:50 | |
| jeblair | pabelanger: welcome back! i tried to take good #info notes in the zuul meeting if you want to catch up on the meeting log | 23:51 |
| *** aeng has quit IRC | 23:51 | |
| pabelanger | jeblair: great! I was planning on looking that over this evening | 23:52 |
| jeblair | mordred, clarkb: i didn't see any significant traffic on my tcpdump from the admin request, so i suspect it's more like we have 3 gear connections in a permanently bad state, rather than too much traffic. iow, the timeout is a red herring. | 23:53 |
| clarkb | jeblair: gotcha. Should we attempt a service restart to force a reset on the gearman connection state? | 23:53 |
| jeblair | looking over the thread dump to see if there's anything useful we can get out of it. suspect we will want to restart in a minute. | 23:53 |
| jeblair | wait, we only have one gearman connection | 23:55 |
| clarkb | I see 5, with netstat, 2 of which are part of the launcher | 23:56 |
| clarkb | zuul sees 4 | 23:56 |
| *** thorst has joined #openstack-infra | 23:57 | |
| jeblair | clarkb: so just focusing on the launcher, you see 2 in netstat -- are both of those seen by zuul? | 23:57 |
| clarkb | jeblair: yes, but one of them is the one that is flapping in a loop so it changes over time. The other tcp 0 0 23.253.73.160:35832 104.130.246.207:4730 ESTABLISHED 29874/python seems fixed | 23:57 |
| *** dingyichen has joined #openstack-infra | 23:58 | |
| *** oanson has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!