| *** dkranz has quit IRC | 00:00 | |
| *** dmellado has quit IRC | 00:06 | |
| *** sdake has joined #openstack-infra | 00:07 | |
| *** mriedem is now known as mriedem_away | 00:10 | |
| *** sdake_ has joined #openstack-infra | 00:15 | |
| *** sdake has quit IRC | 00:15 | |
| sdake_ | hey foks, can someone add me to kolla-mesos group | 00:24 |
|---|---|---|
| sdake_ | its a new group with no members | 00:24 |
| sdake_ | (ptl of kolla here) | 00:25 |
| sdake_ | kolla-mesos-core i think | 00:25 |
| jhesketh | sdake_: yep, give me a second | 00:29 |
| sdake_ | thanks jhesketh | 00:29 |
| *** yfried|afk has quit IRC | 00:32 | |
| *** oomichi has joined #openstack-infra | 00:32 | |
| jhesketh | sdake_: done | 00:32 |
| sdake_ | thanks | 00:33 |
| jhesketh | sdake_: should the governance repo be updated with that project? | 00:33 |
| sdake_ | there isa review pending - 1 week timer lazy consensus | 00:33 |
| sdake_ | should hit the repo early next week | 00:33 |
| jhesketh | ah cool, sounds good | 00:33 |
| sdake_ | but the answer is yes | 00:33 |
| sdake_ | ;) | 00:33 |
| openstackgerrit | Merged openstack-infra/shade: Remove test reference to api_versions https://review.openstack.org/242769 | 00:42 |
| *** yfried|afk has joined #openstack-infra | 00:44 | |
| *** Piet has joined #openstack-infra | 00:47 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add in extra connections tests https://review.openstack.org/169300 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Refactor sources out of triggers https://review.openstack.org/118993 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Move Item formatting into Reporters https://review.openstack.org/223298 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Document the new connections in zuul https://review.openstack.org/169270 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for sqlalchemy reporter https://review.openstack.org/223333 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Pass into reporters the trigger type https://review.openstack.org/239565 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Tidy up loggers https://review.openstack.org/224336 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Configure triggers dynamically https://review.openstack.org/119534 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support for 'connection' concept https://review.openstack.org/121528 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for triggers https://review.openstack.org/119532 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for sources https://review.openstack.org/119531 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add base class for reporters https://review.openstack.org/119530 | 00:51 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Add support to reject changes from approvals https://review.openstack.org/211108 | 00:51 |
| openstackgerrit | Merged openstack-infra/system-config: Update plugins and use files from project_config repo https://review.openstack.org/240393 | 00:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: Logrotate even if file is empty https://review.openstack.org/242890 | 00:55 |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Builds and publishes Compute API Guide https://review.openstack.org/231000 | 00:56 |
| *** tiswanso has joined #openstack-infra | 01:00 | |
| *** davideagnello has joined #openstack-infra | 01:00 | |
| *** yamamoto has joined #openstack-infra | 01:03 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: Use /opt for git repos https://review.openstack.org/242892 | 01:09 |
| *** tiswanso has quit IRC | 01:10 | |
| *** sdake_ has quit IRC | 01:13 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: Run crontab for processor every hour https://review.openstack.org/242893 | 01:13 |
| *** zhenguo has joined #openstack-infra | 01:15 | |
| *** August1914 has joined #openstack-infra | 01:20 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Builds and publishes Compute API Guide https://review.openstack.org/231000 | 01:20 |
| *** jyuso1 has joined #openstack-infra | 01:28 | |
| *** gildub has joined #openstack-infra | 01:34 | |
| *** btully has quit IRC | 01:37 | |
| *** amotoki has joined #openstack-infra | 01:37 | |
| *** zhurong has joined #openstack-infra | 01:50 | |
| *** shashank_hegde has joined #openstack-infra | 01:51 | |
| *** zhurong has quit IRC | 02:00 | |
| *** zhurong has joined #openstack-infra | 02:01 | |
| *** otter768 has quit IRC | 02:02 | |
| *** pleia2 has quit IRC | 02:03 | |
| *** pleia2 has joined #openstack-infra | 02:03 | |
| *** ildikov has quit IRC | 02:09 | |
| *** kengo_sakai has joined #openstack-infra | 02:12 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Builds and publishes Compute API Guide https://review.openstack.org/231000 | 02:19 |
| *** wenlock has joined #openstack-infra | 02:22 | |
| *** wenlock has quit IRC | 02:26 | |
| *** Liuqing has joined #openstack-infra | 02:27 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Builds and publishes Compute API Guide https://review.openstack.org/231000 | 02:29 |
| *** tjones has joined #openstack-infra | 02:44 | |
| *** kengo_sa_ has joined #openstack-infra | 02:44 | |
| *** kengo_sakai has quit IRC | 02:45 | |
| *** wenlock has joined #openstack-infra | 02:52 | |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config: Add mistral-specs https://review.openstack.org/242747 | 02:54 |
| *** wenlock has quit IRC | 02:56 | |
| *** julim_ has quit IRC | 03:00 | |
| *** zhurong has quit IRC | 03:00 | |
| *** zhurong has joined #openstack-infra | 03:01 | |
| eliqiao | hello, guys, how can I overwrite a parameter in localrc on gate ? | 03:01 |
| *** wenlock has joined #openstack-infra | 03:01 | |
| eliqiao | I meant unset a specify parameter. | 03:01 |
| *** cloudtrainme has joined #openstack-infra | 03:03 | |
| *** fedexo has joined #openstack-infra | 03:04 | |
| *** sdake has joined #openstack-infra | 03:06 | |
| *** tiswanso has joined #openstack-infra | 03:06 | |
| *** amotoki has quit IRC | 03:09 | |
| *** thorst has joined #openstack-infra | 03:14 | |
| *** tiswanso has quit IRC | 03:18 | |
| *** thorst has quit IRC | 03:19 | |
| *** gomarivera__ has joined #openstack-infra | 03:20 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Magnum: set SERVICE_HOST to eth0's ip address https://review.openstack.org/242910 | 03:21 |
| *** zhurong has quit IRC | 03:21 | |
| *** cloudtrainme has quit IRC | 03:22 | |
| *** zhurong has joined #openstack-infra | 03:22 | |
| *** yuanying has quit IRC | 03:22 | |
| *** tjones has quit IRC | 03:22 | |
| *** davideagnello has quit IRC | 03:30 | |
| *** davideagnello has joined #openstack-infra | 03:40 | |
| *** Sree has joined #openstack-infra | 03:43 | |
| *** harlowja_at_home has joined #openstack-infra | 03:44 | |
| *** davideagnello has quit IRC | 03:45 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Magnum: set SERVICE_HOST to eth0's ip address https://review.openstack.org/242910 | 03:50 |
| *** dolpher has joined #openstack-infra | 03:51 | |
| *** gomarivera__ has left #openstack-infra | 03:58 | |
| SamYaple | can I get an ack on https://review.openstack.org/#/c/242772/ ? One of our gates is stuck NOT_REGISTERED without that patch | 04:02 |
| *** otter768 has joined #openstack-infra | 04:02 | |
| *** dims has quit IRC | 04:06 | |
| *** otter768 has quit IRC | 04:07 | |
| *** yuanying has joined #openstack-infra | 04:07 | |
| *** Sukhdev has joined #openstack-infra | 04:13 | |
| *** tiswanso has joined #openstack-infra | 04:17 | |
| *** harlowja_at_home has quit IRC | 04:20 | |
| *** tiswanso has quit IRC | 04:24 | |
| *** unicell1 has joined #openstack-infra | 04:36 | |
| *** unicell has quit IRC | 04:37 | |
| *** Ramanjaneya has joined #openstack-infra | 04:39 | |
| *** otter768 has joined #openstack-infra | 04:45 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Enable tests from tempest plugins for tempest-dsvm-neutron-full jobs https://review.openstack.org/242925 | 04:49 |
| *** jamielennox is now known as jamielennox|away | 04:57 | |
| *** _denisra has quit IRC | 05:04 | |
| *** amotoki has joined #openstack-infra | 05:08 | |
| *** otter768 has quit IRC | 05:10 | |
| *** Piet has quit IRC | 05:15 | |
| *** rguillebert has quit IRC | 05:18 | |
| *** krtaylor has joined #openstack-infra | 05:19 | |
| *** tiswanso has joined #openstack-infra | 05:22 | |
| *** jamielennox|away is now known as jamielennox | 05:24 | |
| eliqiao | hello, any infra team core arond to help merge https://review.openstack.org/242341 (+2 already and magnum ptl has +1 on it) | 05:26 |
| *** links has joined #openstack-infra | 05:29 | |
| *** tiswanso has quit IRC | 05:31 | |
| *** kushal has joined #openstack-infra | 05:33 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: Use single uwsgi process for now https://review.openstack.org/242935 | 05:34 |
| pabelanger | yolanda: do you mind help landing that ^? | 05:36 |
| pabelanger | currently fine tuning stackalytics.o.o and need that merged | 05:37 |
| pabelanger | we're running out of memory :( | 05:37 |
| *** achanda has quit IRC | 05:38 | |
| pabelanger | Will be trying nginx in the morning to see if there is any difference | 05:39 |
| *** davideagnello has joined #openstack-infra | 05:41 | |
| *** davideagnello has quit IRC | 05:42 | |
| *** davideagnello has joined #openstack-infra | 05:42 | |
| *** davideagnello has quit IRC | 05:43 | |
| *** tlian has quit IRC | 05:44 | |
| *** nadya has joined #openstack-infra | 05:44 | |
| greghaynes | pabelanger: that shouldn't make a difference. Usually that means you need to lower max-requests | 05:46 |
| *** mgrosz has joined #openstack-infra | 05:46 | |
| mgrosz | Hello, can someone please review https://review.openstack.org/#/c/242817/? Added a new project vitrage-dashboard. Thanks | 05:47 |
| *** salv-orlando has joined #openstack-infra | 05:47 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Add a neutron-full variant which runs tests from tempest plugins https://review.openstack.org/242925 | 05:47 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Add a neutron-full variant which runs tests from tempest plugins https://review.openstack.org/242925 | 05:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: Use single uwsgi process for now https://review.openstack.org/242935 | 05:55 |
| *** Sukhdev has quit IRC | 06:02 | |
| *** links has quit IRC | 06:02 | |
| *** Sukhdev has joined #openstack-infra | 06:03 | |
| *** yfried|afk has quit IRC | 06:05 | |
| greghaynes | pabelanger: so running with one worker is pretty bad, I'd highly recommend trying to find a lower max-requests value which works for us and bumping workers up to something like the number of cores | 06:06 |
| lifeless | workers == python processes == no parallelism | 06:07 |
| *** tobiash has quit IRC | 06:07 | |
| *** bardia has joined #openstack-infra | 06:09 | |
| *** tobiash has joined #openstack-infra | 06:09 | |
| bardia | anyone know a good guide to setup a private openstack provider for my openstack-ci env? | 06:10 |
| bardia | tried to setup devstack on a single VM but failed miserably with the networking setup | 06:10 |
| *** eliqiao is now known as da_la_la | 06:11 | |
| *** da_la_la is now known as eliqiao | 06:11 | |
| *** shashank_hegde has quit IRC | 06:11 | |
| bardia | nodepool was communicating with the devstack VM just fine and spawned instances but the instance was unreachable | 06:11 |
| *** sridhar_ram has joined #openstack-infra | 06:13 | |
| *** alex_xu_ is now known as alex_xu | 06:14 | |
| lifeless | mordred: http://jenkins-ci.org/content/mitigating-unauthenticated-remote-code-execution-0-day-jenkins-cli followup | 06:14 |
| lifeless | (h/t ruagair ) | 06:14 |
| *** jyuso1 has quit IRC | 06:15 | |
| *** jyuso1 has joined #openstack-infra | 06:16 | |
| *** mgrosz has quit IRC | 06:16 | |
| *** mgrosz has joined #openstack-infra | 06:20 | |
| *** hdd has joined #openstack-infra | 06:21 | |
| *** fedexo has quit IRC | 06:22 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/237894 | 06:27 |
| *** sridhar_ram1 has joined #openstack-infra | 06:27 | |
| *** claudiub has joined #openstack-infra | 06:28 | |
| *** abregman has joined #openstack-infra | 06:28 | |
| *** sridhar_ram has quit IRC | 06:28 | |
| *** tiswanso has joined #openstack-infra | 06:29 | |
| *** aopincar has joined #openstack-infra | 06:32 | |
| mgrosz | Hello, can someone please review https://review.openstack.org/#/c/242817/? Added a new project vitrage-dashboard. Thanks | 06:34 |
| ruagair | Taking a look mgrosz | 06:36 |
| *** achanda has joined #openstack-infra | 06:38 | |
| mgrosz | ruagair: Thanks. | 06:39 |
| *** salv-orlando has quit IRC | 06:39 | |
| *** tiswanso has quit IRC | 06:40 | |
| *** links has joined #openstack-infra | 06:41 | |
| *** oomichi has quit IRC | 06:42 | |
| *** mahatic has joined #openstack-infra | 06:42 | |
| *** oomichi has joined #openstack-infra | 06:44 | |
| *** achanda has quit IRC | 06:46 | |
| openstackgerrit | Alexey Weyl proposed openstack-infra/irc-meetings: Adding vitrage team meeting https://review.openstack.org/242816 | 06:47 |
| openstackgerrit | Fathi Boudra proposed openstack-infra/jenkins-job-builder: Add support of 'somewhereElse' value to Copy to slave plugin https://review.openstack.org/239250 | 06:47 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add PyPI publishing for networking-bagpipe-l2 https://review.openstack.org/240890 | 06:47 |
| openstackgerrit | Fathi Boudra proposed openstack-infra/jenkins-job-builder: Add support for matrices' subset in multijobs. https://review.openstack.org/237082 | 06:48 |
| openstackgerrit | Merged openstack-infra/project-config: Add devstack-logs publisher for Kolla https://review.openstack.org/241878 | 06:48 |
| *** shashank_hegde has joined #openstack-infra | 06:50 | |
| *** mgrosz has quit IRC | 06:51 | |
| *** aspiers has joined #openstack-infra | 06:52 | |
| *** sridhar_ram1 has quit IRC | 06:52 | |
| *** amotoki_ has joined #openstack-infra | 06:53 | |
| *** baoli has quit IRC | 06:53 | |
| *** amotoki has quit IRC | 06:54 | |
| StevenK | ruagair: There was a large amount of chatter in this channel over the weekend about the jenkins thing, FWIW | 06:54 |
| FallenPegasus | just dont let Jenkins deserialize used provided serialized java objects | 06:55 |
| *** gildub has quit IRC | 06:57 | |
| tchaypo | Oh | 06:57 |
| *** pahuang_ has joined #openstack-infra | 06:57 | |
| tchaypo | This is why people were talking about Java and deserialisation all over Facebook? | 06:57 |
| FallenPegasus | yes | 06:58 |
| FallenPegasus | its not as bad as python or js deserialization, in java you can only deserialize a object you already have | 06:58 |
| FallenPegasus | but... you can make really good guesses what objects a java instance has | 06:58 |
| FallenPegasus | and you can deserialize EVERY OBJECT in the soup | 06:59 |
| FallenPegasus | java is a terrible language | 06:59 |
| ruagair | Ah, cool. Thanks StevenK. I always miss weekend chatter. | 06:59 |
| *** garyk has joined #openstack-infra | 07:00 | |
| ruagair | Probably a good use case to mute part/joins | 07:00 |
| *** kushal has quit IRC | 07:04 | |
| *** mgrosz has joined #openstack-infra | 07:08 | |
| *** otter768 has joined #openstack-infra | 07:10 | |
| *** bardia has quit IRC | 07:11 | |
| *** garyk has quit IRC | 07:11 | |
| *** garyk has joined #openstack-infra | 07:11 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add --force to grub-install https://review.openstack.org/236661 | 07:14 |
| *** otter768 has quit IRC | 07:15 | |
| *** Sukhdev has quit IRC | 07:15 | |
| *** k4n0 has joined #openstack-infra | 07:18 | |
| openstackgerrit | Thomas Bechtold proposed openstack-infra/project-config: add release notes jobs for manila https://review.openstack.org/242961 | 07:20 |
| *** ParsectiX has joined #openstack-infra | 07:26 | |
| tangchen_ | Hi guys | 07:29 |
| tangchen_ | I got this error in openstack client CI: | 07:29 |
| tangchen_ | Traceback (most recent call last): | 07:29 |
| tangchen_ | File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner | 07:29 |
| tangchen_ | return func(*args, **kwargs) | 07:29 |
| tangchen_ | File "/opt/stack/new/nova/nova/scheduler/manager.py", line 84, in select_destinations | 07:29 |
| tangchen_ | filter_properties) | 07:29 |
| tangchen_ | File "/opt/stack/new/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations | 07:29 |
| tangchen_ | raise exception.NoValidHost(reason=reason) | 07:29 |
| tangchen_ | NoValidHost: No valid host was found. There are not enough hosts available. | 07:29 |
| tangchen_ | Looks like the CI has not enough host. | 07:29 |
| tangchen_ | The patch is here. https://review.openstack.org/#/c/238409/ | 07:30 |
| tangchen_ | Please help to look at it. | 07:30 |
| *** oomichi_ has joined #openstack-infra | 07:33 | |
| *** oomichi has quit IRC | 07:35 | |
| *** mgrosz has quit IRC | 07:36 | |
| *** tiswanso has joined #openstack-infra | 07:37 | |
| *** sdake has quit IRC | 07:37 | |
| *** sdake has joined #openstack-infra | 07:38 | |
| *** scheuran has joined #openstack-infra | 07:38 | |
| *** mgrosz has joined #openstack-infra | 07:39 | |
| *** salv-orlando has joined #openstack-infra | 07:39 | |
| *** ifarkas has joined #openstack-infra | 07:42 | |
| *** nmagnezi has joined #openstack-infra | 07:43 | |
| *** devvesa has joined #openstack-infra | 07:43 | |
| *** davideagnello has joined #openstack-infra | 07:44 | |
| *** salv-orlando has quit IRC | 07:44 | |
| *** ihrachys has joined #openstack-infra | 07:45 | |
| *** yfried|afk has joined #openstack-infra | 07:45 | |
| *** tiswanso has quit IRC | 07:48 | |
| *** yfried has joined #openstack-infra | 07:48 | |
| *** davideagnello has quit IRC | 07:49 | |
| *** yfried|afk has quit IRC | 07:49 | |
| *** sridhar_ram has joined #openstack-infra | 07:51 | |
| *** mgrosz has quit IRC | 07:54 | |
| *** Ala has joined #openstack-infra | 07:56 | |
| *** ParsectiX has quit IRC | 07:57 | |
| *** amotoki_ has quit IRC | 07:59 | |
| openstackgerrit | Ramakrishnan G proposed openstack/diskimage-builder: Add proliant-tools element https://review.openstack.org/163754 | 08:00 |
| *** afazekas is now known as afazekas|sick | 08:01 | |
| *** ihrachys has quit IRC | 08:02 | |
| *** mgrosz has joined #openstack-infra | 08:02 | |
| *** AJaeger has joined #openstack-infra | 08:02 | |
| *** jed56 has joined #openstack-infra | 08:02 | |
| *** sdake_ has joined #openstack-infra | 08:03 | |
| *** sridhar_ram1 has joined #openstack-infra | 08:03 | |
| yolanda | morning | 08:03 |
| *** sridhar_ram has quit IRC | 08:04 | |
| *** achanda has joined #openstack-infra | 08:05 | |
| *** ParsectiX has joined #openstack-infra | 08:06 | |
| *** ParsectiX has quit IRC | 08:06 | |
| *** sdake has quit IRC | 08:06 | |
| *** sridhar_ram1 has quit IRC | 08:06 | |
| AJaeger | morning, yolanda ! | 08:06 |
| *** ParsectiX has joined #openstack-infra | 08:07 | |
| *** kozhukalov_ has joined #openstack-infra | 08:08 | |
| *** sridhar_ram1 has joined #openstack-infra | 08:09 | |
| *** mpavone has joined #openstack-infra | 08:10 | |
| *** ttx has joined #openstack-infra | 08:10 | |
| *** ParsectiX has quit IRC | 08:11 | |
| *** amotoki has joined #openstack-infra | 08:11 | |
| *** ParsectiX has joined #openstack-infra | 08:12 | |
| *** amotoki has quit IRC | 08:12 | |
| *** amotoki has joined #openstack-infra | 08:12 | |
| *** sridhar_ram1 has quit IRC | 08:13 | |
| *** baoli has joined #openstack-infra | 08:13 | |
| *** sridhar_ram has joined #openstack-infra | 08:15 | |
| *** arxcruz has joined #openstack-infra | 08:16 | |
| *** zigo has quit IRC | 08:16 | |
| *** mpavone has quit IRC | 08:17 | |
| *** baoli has quit IRC | 08:18 | |
| *** zigo has joined #openstack-infra | 08:19 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: add release notes jobs for keystone https://review.openstack.org/242972 | 08:19 |
| *** sridhar_ram1 has joined #openstack-infra | 08:20 | |
| *** ihrachys has joined #openstack-infra | 08:20 | |
| *** sridhar_ram has quit IRC | 08:21 | |
| *** kushal has joined #openstack-infra | 08:22 | |
| *** dtantsur|afk is now known as dtantsur | 08:24 | |
| *** binbincong has joined #openstack-infra | 08:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Run releasenotes jobs on Liberty+ https://review.openstack.org/242975 | 08:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Only run releasenotes jobs if needed https://review.openstack.org/242976 | 08:29 |
| *** flepied has joined #openstack-infra | 08:29 | |
| AJaeger | dhellmann: please review the two patches above, I've added you as well ^ | 08:31 |
| *** kaisers has quit IRC | 08:31 | |
| *** sridhar_ram has joined #openstack-infra | 08:31 | |
| AJaeger | dhellmann: on glance liberty is broken since no releasenotes jobs exist, the first change will fix pre-liberty everywhere. The second is an optimization but will help as well with liberty... | 08:32 |
| *** zhurong_ has joined #openstack-infra | 08:32 | |
| *** kaisers has joined #openstack-infra | 08:32 | |
| *** Liuqing_ has joined #openstack-infra | 08:32 | |
| *** dingyichen has quit IRC | 08:33 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Skip unnecessary tests for horizon docs/unittests changes https://review.openstack.org/242879 | 08:33 |
| openstackgerrit | Isaac Beckman proposed openstack-infra/devstack-gate: Multinode i-faces on singlenode n-net create once https://review.openstack.org/239363 | 08:34 |
| *** shuquan has joined #openstack-infra | 08:34 | |
| *** sridhar_ram1 has quit IRC | 08:34 | |
| *** sshnaidm has joined #openstack-infra | 08:35 | |
| *** Liuqing has quit IRC | 08:35 | |
| *** zhurong has quit IRC | 08:35 | |
| *** yuanying has quit IRC | 08:36 | |
| *** gildub has joined #openstack-infra | 08:36 | |
| *** yuanying has joined #openstack-infra | 08:36 | |
| *** fhubik has joined #openstack-infra | 08:37 | |
| *** eantyshev has left #openstack-infra | 08:37 | |
| *** jcoufal has joined #openstack-infra | 08:38 | |
| *** sridhar_ram1 has joined #openstack-infra | 08:38 | |
| *** shuquan has quit IRC | 08:38 | |
| *** ZZelle has quit IRC | 08:38 | |
| *** kaisers has quit IRC | 08:39 | |
| *** sridhar_ram has quit IRC | 08:39 | |
| *** ZZelle has joined #openstack-infra | 08:39 | |
| *** dguitarbite has joined #openstack-infra | 08:39 | |
| *** yfried is now known as yfried|afk | 08:39 | |
| *** shuquan has joined #openstack-infra | 08:41 | |
| *** kaisers has joined #openstack-infra | 08:41 | |
| *** jamielennox is now known as jamielennox|away | 08:42 | |
| *** salv-orlando has joined #openstack-infra | 08:45 | |
| *** tiswanso has joined #openstack-infra | 08:45 | |
| *** e0ne has joined #openstack-infra | 08:46 | |
| *** luqas has joined #openstack-infra | 08:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Only run releasenotes jobs if needed https://review.openstack.org/242976 | 08:46 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Run releasenotes jobs on Liberty+ https://review.openstack.org/242975 | 08:46 |
| *** sridhar_ram1 has quit IRC | 08:47 | |
| *** jlanoux has joined #openstack-infra | 08:47 | |
| *** cdent has joined #openstack-infra | 08:48 | |
| *** jordanP has joined #openstack-infra | 08:53 | |
| *** salv-orlando has quit IRC | 08:53 | |
| *** Hal has joined #openstack-infra | 08:55 | |
| *** e0ne has quit IRC | 08:55 | |
| *** Hal is now known as Guest46496 | 08:55 | |
| *** tiswanso has quit IRC | 08:56 | |
| *** zeih has joined #openstack-infra | 08:56 | |
| *** e0ne has joined #openstack-infra | 08:57 | |
| *** AJaeger has quit IRC | 08:58 | |
| *** mkoderer has quit IRC | 08:59 | |
| *** shardy has joined #openstack-infra | 08:59 | |
| *** yfried|afk is now known as yfried | 08:59 | |
| *** matrohon has joined #openstack-infra | 09:00 | |
| *** mkoderer has joined #openstack-infra | 09:02 | |
| *** bexelbie has joined #openstack-infra | 09:03 | |
| *** Sree has quit IRC | 09:04 | |
| *** mgrosz has quit IRC | 09:05 | |
| *** Sree has joined #openstack-infra | 09:05 | |
| *** mgrosz has joined #openstack-infra | 09:06 | |
| openstackgerrit | Merged openstack-infra/system-config: Add #openstack-performance channel to the channels logging https://review.openstack.org/240243 | 09:07 |
| *** rossella_s has quit IRC | 09:07 | |
| *** rossella_s has joined #openstack-infra | 09:07 | |
| *** sshnaidm has quit IRC | 09:08 | |
| openstackgerrit | Daniel Watkins proposed openstack-infra/jenkins-job-builder: useMatrixChild support in parameterized trigger https://review.openstack.org/235452 | 09:10 |
| *** achanda has quit IRC | 09:10 | |
| *** otter768 has joined #openstack-infra | 09:11 | |
| *** kushal has quit IRC | 09:12 | |
| *** peterlisak has joined #openstack-infra | 09:12 | |
| *** yfried is now known as yfried|afk | 09:12 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Adding vitrage team meeting https://review.openstack.org/242816 | 09:12 |
| *** zhurong_ has quit IRC | 09:13 | |
| *** kushal has joined #openstack-infra | 09:14 | |
| *** mgrosz has quit IRC | 09:15 | |
| *** otter768 has quit IRC | 09:16 | |
| *** openstack has joined #openstack-infra | 09:21 | |
| *** sshnaidm has joined #openstack-infra | 09:22 | |
| *** ParsectiX has quit IRC | 09:23 | |
| *** sorantis has joined #openstack-infra | 09:25 | |
| *** hdd has quit IRC | 09:26 | |
| *** Sree has quit IRC | 09:30 | |
| openstackgerrit | Christian Berendt proposed openstack-infra/irc-meetings: Add documentation install team meeting https://review.openstack.org/243006 | 09:30 |
| *** pahuang_ has quit IRC | 09:30 | |
| *** hdd has joined #openstack-infra | 09:31 | |
| *** yfried|afk is now known as yfried | 09:32 | |
| *** ParsectiX has joined #openstack-infra | 09:35 | |
| *** ParsectiX has left #openstack-infra | 09:35 | |
| *** sdake_ has quit IRC | 09:36 | |
| *** ParsectiX has joined #openstack-infra | 09:37 | |
| *** k4n0 has quit IRC | 09:39 | |
| *** shuquan has quit IRC | 09:39 | |
| *** Liuqing_ has quit IRC | 09:40 | |
| *** jistr has joined #openstack-infra | 09:41 | |
| *** mgrosz has joined #openstack-infra | 09:44 | |
| *** davideagnello has joined #openstack-infra | 09:45 | |
| *** kushal has quit IRC | 09:46 | |
| *** rguillebert has joined #openstack-infra | 09:46 | |
| *** davideagnello has quit IRC | 09:49 | |
| *** shuquan has joined #openstack-infra | 09:51 | |
| openstackgerrit | Timofey Durakov proposed openstack-infra/project-config: Initial job for live-migrations https://review.openstack.org/243013 | 09:52 |
| *** tiswanso has joined #openstack-infra | 09:53 | |
| *** fhubik is now known as fhubik_brb | 09:54 | |
| *** ilyashakhat has joined #openstack-infra | 09:55 | |
| *** FallenPegasus has quit IRC | 09:55 | |
| *** Ala has quit IRC | 10:00 | |
| *** zhenguo has quit IRC | 10:02 | |
| *** derekh has joined #openstack-infra | 10:02 | |
| *** tiswanso has quit IRC | 10:02 | |
| *** Ala has joined #openstack-infra | 10:03 | |
| *** hashar has joined #openstack-infra | 10:07 | |
| *** mgrosz has quit IRC | 10:08 | |
| *** electrofelix has joined #openstack-infra | 10:08 | |
| *** mgrosz has joined #openstack-infra | 10:09 | |
| mgrosz | ruagair: Thanks. | 10:09 |
| mgrosz | Hello, can someone please review https://review.openstack.org/#/c/242817/? Added a new project vitrage-dashboard. Thanks | 10:09 |
| *** abregman is now known as abregman|afk | 10:09 | |
| *** vponomaryov has joined #openstack-infra | 10:10 | |
| *** Liuqing has joined #openstack-infra | 10:10 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/system-config: Rename Ceilometer to Telemetry https://review.openstack.org/243020 | 10:12 |
| *** fhubik_brb is now known as fhubik | 10:12 | |
| *** sshnaidm has quit IRC | 10:12 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Add magnum support to functional tests on shade https://review.openstack.org/242742 | 10:17 |
| aspiers | Hi all, is it recommended to hold IRC meetings in one of the #openstack-meeting-* channels? or does the meetbot work in any channel it's already eavesdropping without extra configuration? | 10:17 |
| *** ParsectiX has quit IRC | 10:17 | |
| *** BobBall_AWOL is now known as BobBall | 10:20 | |
| *** shuquan has quit IRC | 10:21 | |
| *** sshnaidm has joined #openstack-infra | 10:25 | |
| *** AJaeger has joined #openstack-infra | 10:25 | |
| AJaeger | yolanda: please be carefull with Depends-On! | 10:25 |
| aspiers | it seems from https://wiki.openstack.org/wiki/Meetings/CreateaMeeting that it is indeed recommended to use one of those | 10:25 |
| aspiers | but it would be nice if the rationale was documented | 10:26 |
| yolanda | oh, did i miss something? | 10:26 |
| AJaeger | yolanda: the way it's implemented in Zuul is that if you approve it and the dependend merge is not approved, it will not automatically merge | 10:26 |
| yolanda | ah yes, i know | 10:26 |
| AJaeger | yolanda: so, you still need to +A the change again | 10:26 |
| yolanda | AJaeger, so you prefer to don't +A until the parent lands? | 10:27 |
| yolanda | i'm normally setting a +A even if i have a depends on | 10:27 |
| AJaeger | Approving those changes does not make a change - but if now somebody runs recheck (my dependent change is in) it will not make a difference | 10:27 |
| AJaeger | yolanda: I normally don't +A and just say: Waiting for change to merge. | 10:27 |
| AJaeger | yolanda: not sure whether there's a "best practice" yet. | 10:27 |
| *** ldnunes has joined #openstack-infra | 10:27 | |
| yolanda | AJaeger, ok, let me revisit those | 10:28 |
| aspiers | AJaeger: any ideas? ^^^ | 10:28 |
| AJaeger | yolanda: If you're aware that you need to toggle your +A once the dependent change is in, then I'm fine... | 10:28 |
| *** yfried is now known as yfried|afk | 10:28 | |
| AJaeger | aspiers: let me read scrollback first... | 10:28 |
| *** HeOS has quit IRC | 10:28 | |
| yolanda | AJaeger, is difficult to track this | 10:28 |
| hrubi | jhesketh: hi, going to check & address your comments in the zuul/connection_tests. are you ok with me adding patchsets to your changes and putting some new changes inbetween the existing ones? | 10:29 |
| AJaeger | yolanda: yeah, it's all tricky - so feel free to ask others for best practice | 10:29 |
| *** mgrosz has quit IRC | 10:29 | |
| yolanda | i removed the +a | 10:29 |
| aspiers | I am guessing rationale for using #openstack-meeting-* is something like 1) reduces number of simultaneous meetings and hence conflicts for individuals, and 2) prevents IRC users from joining normal channel during meeting and disrupting it | 10:29 |
| aspiers | but those are wild guesses | 10:30 |
| aspiers | and it's also not clear where meetbot is configured to supervise meeting channels | 10:30 |
| AJaeger | aspiers: I would guess the same | 10:30 |
| AJaeger | aspiers: meetbot is setup in system-config repo | 10:31 |
| aspiers | AJaeger: but that repo does not mention #openstack-meeting-* anywhere where it doesn't also mention normal non-meeting channels | 10:32 |
| aspiers | AJaeger: which suggests that you could do #startmeeting in any channel | 10:32 |
| AJaeger | aspiers: yeah, might work... | 10:32 |
| AJaeger | aspiers: better ask one of the US admins later today... | 10:32 |
| AJaeger | sorry, have to leave again... | 10:32 |
| *** AJaeger has quit IRC | 10:32 | |
| *** yfried|afk is now known as yfried | 10:34 | |
| *** ldnunes_ has joined #openstack-infra | 10:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Skip unnecessary tests for horizon docs/unittests changes https://review.openstack.org/242879 | 10:35 |
| openstackgerrit | Merged openstack-infra/publications: Typo in doc about "git co {branch_name}" https://review.openstack.org/241057 | 10:36 |
| *** bradjones has joined #openstack-infra | 10:37 | |
| *** bradjones has quit IRC | 10:37 | |
| *** bradjones has joined #openstack-infra | 10:37 | |
| *** ldnunes_ has quit IRC | 10:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Add dsvm into job names for Kolla https://review.openstack.org/242772 | 10:41 |
| *** eliqiao has quit IRC | 10:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Add new pipeline for magnum https://review.openstack.org/242341 | 10:48 |
| openstackgerrit | Merged openstack-infra/project-config-example: Add tox -e zuul prerequisites https://review.openstack.org/242252 | 10:48 |
| *** mgrosz has joined #openstack-infra | 10:49 | |
| openstackgerrit | Merged openstack-infra/project-config: Set ZUUL_BRANCH and ZUUL_REF https://review.openstack.org/242449 | 10:50 |
| openstackgerrit | Merged openstack-infra/project-config: heat: remove legacy functional jobs https://review.openstack.org/237685 | 10:53 |
| openstackgerrit | Merged openstack-infra/infra-manual: Document adding hudson-openstack to project-bugs https://review.openstack.org/232140 | 10:56 |
| *** cschwede has joined #openstack-infra | 10:57 | |
| *** pc_m has joined #openstack-infra | 10:58 | |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: Update cli-shutdown.groovy for license grant https://review.openstack.org/242713 | 10:59 |
| *** tiswanso has joined #openstack-infra | 10:59 | |
| sambetts | yolanda: Thanks for +Wing my networking-cisco patch :) | 11:00 |
| yolanda | sounds it was really needed :) | 11:00 |
| *** Sree has joined #openstack-infra | 11:01 | |
| sambetts | Yeah, I hadn't even thought about the stable branches being affected when I enabled py3 | 11:01 |
| sambetts | :( | 11:01 |
| openstackgerrit | Merged openstack-infra/project-config: Disable py34 for networking-cisco kilo and juno https://review.openstack.org/241994 | 11:02 |
| *** cschwede has quit IRC | 11:02 | |
| *** bunting has quit IRC | 11:04 | |
| *** ihrachys has quit IRC | 11:05 | |
| *** maishsk has joined #openstack-infra | 11:05 | |
| *** eliqiao1 has joined #openstack-infra | 11:06 | |
| mgrosz | Hello, can someone please review https://review.openstack.org/#/c/242817/? Added a new project vitrage-dashboard. Thanks | 11:09 |
| *** tiswanso has quit IRC | 11:09 | |
| mgrosz | yolanda: Can you please review https://review.openstack.org/#/c/242817/? Added a new project vitrage-dashboard. Thanks.... | 11:10 |
| yolanda | hi mgrosz, sorry, in urgent things now, but i'll take a look when i have som slot | 11:10 |
| *** aysyd has joined #openstack-infra | 11:10 | |
| mgrosz | yolanda: thanks | 11:11 |
| *** cschwede has joined #openstack-infra | 11:11 | |
| *** larainema has quit IRC | 11:12 | |
| *** amotoki has quit IRC | 11:12 | |
| *** otter768 has joined #openstack-infra | 11:12 | |
| eliqiao1 | hello, yolanda: | 11:13 |
| *** wenlock has quit IRC | 11:13 | |
| *** abregman|afk is now known as abregman | 11:13 | |
| *** eliqiao1 is now known as eliqiao | 11:13 | |
| yolanda | hi eliqiao1 | 11:14 |
| eliqiao | yolanda: thanks for helping merge https://review.openstack.org/242341 | 11:14 |
| yolanda | you are welcome | 11:14 |
| *** aysyd has quit IRC | 11:15 | |
| eliqiao | yolanda: after it get merged, our project get gate failed with  NOT_REGISTERED , is that normal ? see https://review.openstack.org/#/c/243011/ | 11:15 |
| eliqiao | yolanda: or, did I missing something ? | 11:15 |
| yolanda | that's because it cannot find relevant jobs in jenkins | 11:15 |
| *** aysyd has joined #openstack-infra | 11:15 | |
| *** dewsday has joined #openstack-infra | 11:15 | |
| *** dewsday has quit IRC | 11:15 | |
| *** dims has joined #openstack-infra | 11:15 | |
| eliqiao | yolanda: how can I fix that ? | 11:16 |
| yolanda | when it is a new job, it can take some time for jenkins to refresh | 11:16 |
| *** pbourke has quit IRC | 11:16 | |
| yolanda | it looks fine to me, so i suggest that you wait for a bit, and then try a recheck | 11:16 |
| *** pbourke has joined #openstack-infra | 11:16 | |
| eliqiao | yolanda: thanks, get it :). hope every thing works fine. | 11:17 |
| *** otter768 has quit IRC | 11:17 | |
| yolanda | i can see that you create the jobs in jenkins correctly, so it should be a matter of time for jjb to create them and be available on gearman | 11:17 |
| eliqiao | cool, thank you | 11:19 |
| *** larainema has joined #openstack-infra | 11:19 | |
| *** eliqiao is now known as eliqiao_ | 11:20 | |
| *** e0ne has quit IRC | 11:21 | |
| *** eliqiao has joined #openstack-infra | 11:23 | |
| *** eliqiao_ has quit IRC | 11:26 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/237894 | 11:26 |
| *** e0ne has joined #openstack-infra | 11:30 | |
| *** yfried is now known as yfried|afk | 11:30 | |
| *** sergmelikyan has joined #openstack-infra | 11:32 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Ironic: Create and enable pxe_ipa-ipxe-nv in ironic https://review.openstack.org/233626 | 11:34 |
| *** Sree has quit IRC | 11:34 | |
| *** yfried|afk is now known as yfried | 11:35 | |
| *** eliqiao has quit IRC | 11:35 | |
| jhesketh | hrubi: sure, what did you have in mind? | 11:36 |
| *** eliqiao has joined #openstack-infra | 11:37 | |
| *** sergmelikyan has quit IRC | 11:37 | |
| *** ParsectiX has joined #openstack-infra | 11:38 | |
| *** ihrachys has joined #openstack-infra | 11:39 | |
| *** jaypipes has joined #openstack-infra | 11:41 | |
| *** weihan has joined #openstack-infra | 11:43 | |
| *** sergmelikyan has joined #openstack-infra | 11:43 | |
| *** sergmelikyan has quit IRC | 11:44 | |
| *** yfried is now known as yfried|afk | 11:45 | |
| *** davideagnello has joined #openstack-infra | 11:46 | |
| *** eliqiao_ has joined #openstack-infra | 11:48 | |
| *** jlanoux has quit IRC | 11:48 | |
| *** maiteb has joined #openstack-infra | 11:50 | |
| *** davideagnello has quit IRC | 11:50 | |
| *** eliqiao- has joined #openstack-infra | 11:52 | |
| *** cdent has quit IRC | 11:54 | |
| *** eliqiao has quit IRC | 11:54 | |
| *** eliqiao- has quit IRC | 11:54 | |
| *** eliqiao_ has quit IRC | 11:54 | |
| *** eliqiao1 has joined #openstack-infra | 11:54 | |
| *** eliqiao1 is now known as eliqiao | 11:54 | |
| *** nijaba has joined #openstack-infra | 11:56 | |
| *** nijaba has joined #openstack-infra | 11:56 | |
| *** jlanoux has joined #openstack-infra | 11:58 | |
| *** yfried|afk is now known as yfried | 11:59 | |
| *** rfolco has joined #openstack-infra | 12:00 | |
| *** maishsk has quit IRC | 12:01 | |
| *** FallenPegasus has joined #openstack-infra | 12:02 | |
| jhesketh | hrubi: I'm off sorry, but feel free to patch it however you see fit :-) | 12:02 |
| *** glauco has joined #openstack-infra | 12:03 | |
| *** maishsk has joined #openstack-infra | 12:03 | |
| *** smarcet has joined #openstack-infra | 12:04 | |
| *** eliqiao has quit IRC | 12:04 | |
| yolanda | jhesketh so we are having a problem with zuul downstream, not processing periodic jobs, but i still don't know the problem | 12:05 |
| yolanda | i had to revert the resync, and i will be applying patch by patch, we didn't sync in a long time | 12:05 |
| yolanda | it seems something related with NullChange | 12:05 |
| yolanda | but i see that works fine on upstream, so we may have something different | 12:05 |
| *** tiswanso has joined #openstack-infra | 12:05 | |
| *** eliqiao has joined #openstack-infra | 12:06 | |
| *** eliqiao_ has joined #openstack-infra | 12:06 | |
| *** eliqiao- has joined #openstack-infra | 12:07 | |
| *** FallenPegasus has quit IRC | 12:07 | |
| *** eliqiao- has quit IRC | 12:07 | |
| *** eliqiao has quit IRC | 12:07 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Ironic: Create and enable pxe_ipa-ipxe-nv in ironic https://review.openstack.org/233626 | 12:08 |
| *** eliqiao_ is now known as eliqiao | 12:08 | |
| *** eliqiao has left #openstack-infra | 12:08 | |
| *** yfried has quit IRC | 12:08 | |
| *** Ramanjaneya has quit IRC | 12:10 | |
| *** baoli has joined #openstack-infra | 12:15 | |
| *** yfried has joined #openstack-infra | 12:15 | |
| *** ig0r__ has joined #openstack-infra | 12:17 | |
| *** gildub has quit IRC | 12:19 | |
| *** tiswanso has quit IRC | 12:19 | |
| nibalizer | aa | 12:20 |
| *** baoli has quit IRC | 12:20 | |
| *** fhubik is now known as fhubik_brb | 12:21 | |
| *** ociuhandu has joined #openstack-infra | 12:22 | |
| *** zeih_ has joined #openstack-infra | 12:23 | |
| *** eliqiao has joined #openstack-infra | 12:23 | |
| hrubi | jhesketh: I'll stick to your suggestions in comments of your changes, it will be some partial cherry-picking of changes in my github-integration topic branch | 12:23 |
| *** yamamoto has quit IRC | 12:26 | |
| *** placko has joined #openstack-infra | 12:26 | |
| *** zeih has quit IRC | 12:26 | |
| openstackgerrit | Ian Y. Choi proposed openstack-dev/pbr: More support Sphinx >=1.3b1 and <1.3.1 https://review.openstack.org/243065 | 12:26 |
| openstackgerrit | Merged openstack/requirements: Deal with underscores in blacklist during generate-constraints https://review.openstack.org/242798 | 12:26 |
| *** daemontool has quit IRC | 12:27 | |
| *** kengo_sa_ has quit IRC | 12:30 | |
| *** amotoki has joined #openstack-infra | 12:34 | |
| *** eliqiao has left #openstack-infra | 12:38 | |
| *** cloudtrainme has joined #openstack-infra | 12:39 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack-infra/project-config: Add python-brickclient to OpenStack https://review.openstack.org/243080 | 12:41 |
| openstackgerrit | Evgeny Antyshev proposed openstack-infra/system-config: Make hieraedit.py to not shuffle data items https://review.openstack.org/239479 | 12:43 |
| *** eliqiao1 has joined #openstack-infra | 12:44 | |
| *** eliqiao has joined #openstack-infra | 12:46 | |
| *** cloudtrainme has quit IRC | 12:49 | |
| *** thorst has joined #openstack-infra | 12:51 | |
| *** gordc has joined #openstack-infra | 12:51 | |
| *** dtardivel has joined #openstack-infra | 12:52 | |
| *** salv-orlando has joined #openstack-infra | 12:52 | |
| *** fhubik_brb is now known as fhubik | 12:53 | |
| *** shuquan has joined #openstack-infra | 12:54 | |
| *** zhurong has joined #openstack-infra | 12:55 | |
| *** salv-orlando has quit IRC | 12:56 | |
| *** maiteb has quit IRC | 12:56 | |
| openstackgerrit | Sergey Lukjanov proposed openstack-infra/project-config: Add reno jobs for sahara https://review.openstack.org/243085 | 12:57 |
| *** weihan has quit IRC | 12:57 | |
| *** maiteb has joined #openstack-infra | 12:57 | |
| *** jordanP has quit IRC | 12:59 | |
| *** dims has quit IRC | 12:59 | |
| *** aopincar has quit IRC | 13:00 | |
| *** maishsk has quit IRC | 13:02 | |
| *** samuelBartel has joined #openstack-infra | 13:04 | |
| *** marcusvrn_ has quit IRC | 13:05 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: also report on stable/liberty changes in openstack-qa https://review.openstack.org/243089 | 13:06 |
| *** daemontool has joined #openstack-infra | 13:06 | |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Bumps os-win version to 0.0.5 https://review.openstack.org/243091 | 13:07 |
| *** cloudtrainme has joined #openstack-infra | 13:07 | |
| *** weshay_xchat has joined #openstack-infra | 13:08 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** maishsk has joined #openstack-infra | 13:08 | |
| *** yamamoto has quit IRC | 13:09 | |
| *** yamamoto has joined #openstack-infra | 13:09 | |
| *** tlian has joined #openstack-infra | 13:09 | |
| *** yamamoto_ has joined #openstack-infra | 13:10 | |
| openstackgerrit | KATO Tomoyuki proposed openstack-infra/infra-manual: Add previewing a change https://review.openstack.org/241982 | 13:12 |
| *** otter768 has joined #openstack-infra | 13:13 | |
| *** yamamoto has quit IRC | 13:13 | |
| *** amuller has joined #openstack-infra | 13:14 | |
| pabelanger | greghaynes: that is fine for now. I am just debugging things right now. The site is not production yet. | 13:14 |
| *** shuquan has quit IRC | 13:15 | |
| pabelanger | greghaynes: also, each process is consuming about 3.5GB of RAM right now | 13:15 |
| *** dkranz has joined #openstack-infra | 13:15 | |
| pabelanger | between that, apache, and stackalytics-processor we are blowing through 15GB of memory | 13:15 |
| pabelanger | moving to nginix this morning to see what we can shave off | 13:15 |
| ilyashakhat | pabelanger, how many worker processes are there? | 13:16 |
| pabelanger | ilyashakhat: for which part? | 13:17 |
| *** tiswanso has joined #openstack-infra | 13:17 | |
| ilyashakhat | front-end, which hold web app | 13:18 |
| pabelanger | currently 2 processes in uwsgi | 13:18 |
| pabelanger | and apache is lauching 2 also | 13:18 |
| *** otter768 has quit IRC | 13:18 | |
| ilyashakhat | and does apache consume memory too? | 13:18 |
| *** shuquan has joined #openstack-infra | 13:18 | |
| pabelanger | in this case, ya | 13:19 |
| ilyashakhat | ooh :( | 13:19 |
| pabelanger | seems to be just doubling things | 13:19 |
| pabelanger | the reason for my email | 13:19 |
| ilyashakhat | in my case all memory is in uwsgi, nginx consumes very little | 13:19 |
| *** tlian has quit IRC | 13:19 | |
| pabelanger | ya | 13:19 |
| *** Sree has joined #openstack-infra | 13:19 | |
| pabelanger | That is what I am going to do this morning | 13:19 |
| pabelanger | won't be hard to test | 13:20 |
| *** _elmiko is now known as elmiko | 13:20 | |
| pabelanger | full dump will be ready in a few hours | 13:20 |
| pabelanger | then load up my local instance and compare | 13:20 |
| *** cloudtrainme has quit IRC | 13:20 | |
| mgrosz | Hello, can someone please review https://review.openstack.org/#/c/242817/? Added a new project vitrage-dashboard. Thanks | 13:22 |
| *** amotoki has quit IRC | 13:22 | |
| *** yfried is now known as yfried|afk | 13:22 | |
| *** e0ne has quit IRC | 13:23 | |
| *** tiswanso has quit IRC | 13:23 | |
| *** mrmartin has joined #openstack-infra | 13:24 | |
| mrmartin | hi | 13:24 |
| *** Sree has quit IRC | 13:24 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add puppet-nginx support https://review.openstack.org/243102 | 13:25 |
| *** xyang1 has joined #openstack-infra | 13:25 | |
| mrmartin | I just deployed the python-openstackclient from pip, on an Ubuntu 14.04LTS, how can I disable the verbose output? I get something like that: http://paste.openstack.org/show/478304/ | 13:26 |
| mrmartin | it is a python-openstackclient (1.8.0) | 13:26 |
| *** tlian has joined #openstack-infra | 13:26 | |
| *** dprince has joined #openstack-infra | 13:26 | |
| *** amotoki has joined #openstack-infra | 13:27 | |
| *** Sree has joined #openstack-infra | 13:28 | |
| *** boris-42 has quit IRC | 13:28 | |
| *** cloudtrainme has joined #openstack-infra | 13:29 | |
| *** edmondsw has joined #openstack-infra | 13:30 | |
| *** zeih_ has quit IRC | 13:30 | |
| mordred | mrmartin: hrm. interesting | 13:30 |
| *** Sree has quit IRC | 13:30 | |
| mrmartin | maybe I deployed something, but it seems to be a default debug on. | 13:31 |
| mordred | no - I agree with you - it definitely seems to be defaulting to debug on | 13:31 |
| mrmartin | ok, so it is not for me alone | 13:31 |
| mrmartin | I see no option as --no-debug or something like that | 13:32 |
| *** Sree has joined #openstack-infra | 13:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: [WIP] Replace apache with nginx https://review.openstack.org/243106 | 13:34 |
| *** yfried|afk is now known as yfried | 13:34 | |
| *** trown|outttypeww is now known as trown | 13:37 | |
| *** zeih has joined #openstack-infra | 13:37 | |
| *** e0ne has joined #openstack-infra | 13:37 | |
| *** jordanP has joined #openstack-infra | 13:37 | |
| mrmartin | it seems to be false on default: https://github.com/openstack/python-openstackclient/blob/678e69064854e3a3d3499171b0a29f30f2771840/examples/common.py#L138 | 13:38 |
| *** baoli has joined #openstack-infra | 13:39 | |
| *** cloudtrainme has quit IRC | 13:39 | |
| *** mgrosz has quit IRC | 13:40 | |
| *** mwagner has quit IRC | 13:44 | |
| *** baoli has quit IRC | 13:44 | |
| *** davideagnello has joined #openstack-infra | 13:47 | |
| *** amotoki has quit IRC | 13:47 | |
| mrmartin | mordred: I found the bug ticket for the python-openstackclient default debug output: https://bugs.launchpad.net/os-client-config/+bug/1513919 this seems to be related to os-client-config | 13:48 |
| openstack | Launchpad bug 1513919 in os-client-config "os-client-config 1.10.1 output debug information by default" [Undecided,New] | 13:48 |
| *** weshay_xchat is now known as weshay | 13:48 | |
| mordred | mrmartin: ok. I'll figure it out | 13:48 |
| *** FallenPegasus has joined #openstack-infra | 13:50 | |
| *** davideagnello has quit IRC | 13:51 | |
| *** freyes has quit IRC | 13:52 | |
| *** regXboi has joined #openstack-infra | 13:53 | |
| *** cdent has joined #openstack-infra | 13:54 | |
| *** FallenPegasus has quit IRC | 13:55 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Add magnum support to functional tests on shade https://review.openstack.org/242742 | 13:55 |
| *** amitgandhinz has joined #openstack-infra | 13:57 | |
| *** salv-orlando has joined #openstack-infra | 13:58 | |
| *** bswartz has joined #openstack-infra | 13:59 | |
| *** ddieterly has quit IRC | 13:59 | |
| *** boris-42 has joined #openstack-infra | 13:59 | |
| *** amotoki has joined #openstack-infra | 14:00 | |
| *** jyuso1 has quit IRC | 14:03 | |
| *** burgerk has joined #openstack-infra | 14:03 | |
| *** rlandy has joined #openstack-infra | 14:03 | |
| *** nadya has quit IRC | 14:04 | |
| *** nadya_ has joined #openstack-infra | 14:04 | |
| *** barnaby has joined #openstack-infra | 14:04 | |
| *** fhubik is now known as fhubik_brb | 14:04 | |
| *** rossella_ has joined #openstack-infra | 14:05 | |
| *** doug-fish has joined #openstack-infra | 14:06 | |
| *** rossella_s has quit IRC | 14:06 | |
| fungi | mrmartin: i noticed late last week that puppet agent is still disabled on groups.openstack.org. is that intentional? | 14:08 |
| mrmartin | fungi: no, feel free to enable. I'll be around if something goes wrong :) | 14:08 |
| fungi | mrmartin: great, switching it on now and checking to see what the outstanding changeset is for it | 14:09 |
| *** freyes has joined #openstack-infra | 14:10 | |
| fungi | mrmartin: drush will upgrade pear from 1.9.5 to 1.10.1, and there's an outstanding content change to the memcached_max_memory.rb file | 14:11 |
| *** wenlock has joined #openstack-infra | 14:14 | |
| *** kgiusti has joined #openstack-infra | 14:14 | |
| *** shuquan has quit IRC | 14:15 | |
| openstackgerrit | Michael Jeanson proposed openstack-infra/jenkins-job-builder: Add 'do-not-fingerprint' to 'copyartifact' builder https://review.openstack.org/219312 | 14:15 |
| *** mwagner has joined #openstack-infra | 14:15 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack-infra/project-config: Add python-brickclient to OpenStack https://review.openstack.org/243080 | 14:16 |
| *** tiswanso has joined #openstack-infra | 14:17 | |
| *** akshai has joined #openstack-infra | 14:17 | |
| *** julim has joined #openstack-infra | 14:17 | |
| *** wenlock has quit IRC | 14:18 | |
| *** Ala has quit IRC | 14:18 | |
| *** fhubik_brb is now known as fhubik | 14:19 | |
| *** zeih has quit IRC | 14:20 | |
| *** weshay has quit IRC | 14:20 | |
| *** zeih has joined #openstack-infra | 14:20 | |
| openstackgerrit | Victor Stinner proposed openstack/requirements: Bump PyECLib version to 1.1.1 https://review.openstack.org/238771 | 14:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: [WIP] Replace apache with nginx https://review.openstack.org/243106 | 14:21 |
| *** zeih_ has joined #openstack-infra | 14:21 | |
| *** zeih__ has joined #openstack-infra | 14:23 | |
| *** zeih_ has quit IRC | 14:24 | |
| *** zeih_ has joined #openstack-infra | 14:24 | |
| *** zeih has quit IRC | 14:25 | |
| *** doug-fish has quit IRC | 14:25 | |
| *** kozhukalov_ has quit IRC | 14:25 | |
| *** superdan is now known as dansmith | 14:27 | |
| *** baoli has joined #openstack-infra | 14:27 | |
| *** zeih__ has quit IRC | 14:28 | |
| *** haypo has joined #openstack-infra | 14:28 | |
| *** alivigni has joined #openstack-infra | 14:29 | |
| *** Guest47571 has quit IRC | 14:29 | |
| haypo | hi. is there someone available to review this simple upgrade? https://review.openstack.org/#/c/238771/ "Bump PyECLib version to 1.1.1". the change is written by one of the two authors of PyECLib, the change was validated by the Swift PTL, and only Swift uses PyEClib | 14:29 |
| *** fhubik has quit IRC | 14:30 | |
| *** andersonvom has joined #openstack-infra | 14:32 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add stable/liberty tripleo job https://review.openstack.org/240938 | 14:32 |
| *** fhubik has joined #openstack-infra | 14:33 | |
| *** doug-fish has joined #openstack-infra | 14:34 | |
| *** ddieterly has joined #openstack-infra | 14:34 | |
| *** mriedem_away is now known as mriedem | 14:35 | |
| *** sdake has joined #openstack-infra | 14:35 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/project-config: Add stable/liberty tripleo job https://review.openstack.org/240938 | 14:35 |
| *** ddieterl_ has joined #openstack-infra | 14:36 | |
| *** dims has joined #openstack-infra | 14:36 | |
| *** tongli has joined #openstack-infra | 14:37 | |
| *** weshay_xchat has joined #openstack-infra | 14:37 | |
| *** sdake_ has joined #openstack-infra | 14:37 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Correct ACLs for Neutron sub-projects https://review.openstack.org/242196 | 14:37 |
| *** baoli has quit IRC | 14:38 | |
| *** kozhukalov_ has joined #openstack-infra | 14:38 | |
| *** mestery has joined #openstack-infra | 14:38 | |
| *** baoli has joined #openstack-infra | 14:38 | |
| mordred | mrmartin: I have found your python-openstackclient issue and have a patch up to fix it | 14:39 |
| mordred | mrmartin: thank you for pointing it out | 14:39 |
| *** ddieterly has quit IRC | 14:39 | |
| mrmartin | mordred: thanks. | 14:39 |
| openstackgerrit | Merged openstack/requirements: Bump upper-constraint for os-client-config https://review.openstack.org/242160 | 14:40 |
| *** sdake has quit IRC | 14:40 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack-infra/project-config: Add python-brickclient to OpenStack https://review.openstack.org/243080 | 14:42 |
| openstackgerrit | Alexander Maretskiy proposed openstack/requirements: Increase minimal Jinja2 version from 2.6 to 2.8 https://review.openstack.org/240556 | 14:44 |
| *** signed8bit has joined #openstack-infra | 14:44 | |
| *** marekd has quit IRC | 14:44 | |
| *** zeih_ has quit IRC | 14:44 | |
| *** antonym has quit IRC | 14:46 | |
| *** antonym has joined #openstack-infra | 14:47 | |
| *** antonym has quit IRC | 14:47 | |
| *** antonym has joined #openstack-infra | 14:47 | |
| sdague | anteaya / fungi / mordred - quick approve on an irc channel tweak for -qa? - https://review.openstack.org/#/c/243089/ | 14:47 |
| mordred | sdague: done | 14:48 |
| sdague | thanks | 14:48 |
| *** sdake_ is now known as sdake | 14:48 | |
| *** pradk has joined #openstack-infra | 14:49 | |
| *** annegentle has joined #openstack-infra | 14:49 | |
| *** sergmelikyan has joined #openstack-infra | 14:49 | |
| openstackgerrit | Christian Berendt proposed openstack-infra/irc-meetings: Add documentation install team meeting https://review.openstack.org/243006 | 14:50 |
| *** sdake_ has joined #openstack-infra | 14:50 | |
| openstackgerrit | Merged openstack-infra/project-config: also report on stable/liberty changes in openstack-qa https://review.openstack.org/243089 | 14:52 |
| *** fhubik is now known as fhubik_brb | 14:53 | |
| *** sdake has quit IRC | 14:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: [WIP] Replace apache with nginx https://review.openstack.org/243106 | 14:54 |
| *** marekd has joined #openstack-infra | 14:54 | |
| *** eliqiao1 has quit IRC | 14:54 | |
| *** antonym has quit IRC | 14:55 | |
| *** sdake_ is now known as sdake | 14:56 | |
| *** ddieterl_ has quit IRC | 14:57 | |
| *** asselin__ has joined #openstack-infra | 14:57 | |
| *** fhubik_brb is now known as fhubik | 15:00 | |
| *** baoli has quit IRC | 15:01 | |
| *** asselin__ has quit IRC | 15:01 | |
| *** salv-orlando has quit IRC | 15:02 | |
| *** antonym has joined #openstack-infra | 15:02 | |
| *** ddieterly has joined #openstack-infra | 15:02 | |
| *** mission712 has joined #openstack-infra | 15:03 | |
| *** asselin__ has joined #openstack-infra | 15:03 | |
| *** zhurong has quit IRC | 15:04 | |
| *** sridhar_ram has joined #openstack-infra | 15:05 | |
| *** amotoki has quit IRC | 15:06 | |
| *** zeih has joined #openstack-infra | 15:06 | |
| *** maiteb has quit IRC | 15:07 | |
| *** baoli has joined #openstack-infra | 15:07 | |
| *** yamamoto_ has quit IRC | 15:07 | |
| *** amotoki has joined #openstack-infra | 15:08 | |
| *** ParsectiX has quit IRC | 15:09 | |
| openstackgerrit | Timofey Durakov proposed openstack-infra/project-config: Initial job for live-migrations https://review.openstack.org/243013 | 15:09 |
| openstackgerrit | Christian Berendt proposed openstack-infra/irc-meetings: Add documentation install team meeting https://review.openstack.org/243006 | 15:09 |
| *** achanda has joined #openstack-infra | 15:10 | |
| *** zeih has quit IRC | 15:11 | |
| mrmartin | mordred: shade-1.0.0 and ansible 2.0.x seems to be broken also due to an attach_ip_to_server call: https://github.com/ansible/ansible-modules-core/blob/devel/cloud/openstack/os_floating_ip.py#L169 | 15:12 |
| *** marekd has quit IRC | 15:12 | |
| *** tdurakov has joined #openstack-infra | 15:13 | |
| tdurakov | sdague, hi, are you around? | 15:13 |
| sdague | tdurakov: yeh, what's up? | 15:13 |
| *** stevemar_ has joined #openstack-infra | 15:13 | |
| *** otter768 has joined #openstack-infra | 15:14 | |
| tdurakov | sdague, https://review.openstack.org/#/c/243013/ - could you review? | 15:14 |
| sdague | sure | 15:15 |
| *** inteq has quit IRC | 15:15 | |
| tdurakov | sdague, need further help with live-migration job for nova, tried to set up local env, but it seems to be dead end( | 15:15 |
| *** wenlock has joined #openstack-infra | 15:16 | |
| *** eharney has joined #openstack-infra | 15:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Only run releasenotes jobs if needed https://review.openstack.org/242976 | 15:16 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Run releasenotes jobs only on master https://review.openstack.org/242975 | 15:16 |
| sdague | tdurakov: how far do you get if you use devstack-vagrant to build a 2 node devstack | 15:17 |
| *** openstackgerrit has quit IRC | 15:17 | |
| sdague | I think that's probably the local dev environment we want to extend for this | 15:17 |
| *** openstackgerrit has joined #openstack-infra | 15:17 | |
| *** nmagnezi has quit IRC | 15:17 | |
| tdurakov | sdague, devstack-vagrant? I thought we need to use devstack-gate? | 15:17 |
| mordred | Shrews: see mrmartin comment above | 15:18 |
| *** smarcet has quit IRC | 15:18 | |
| *** spzala has joined #openstack-infra | 15:18 | |
| tdurakov | sdague, it's not a problem to setup local multinode env for me, the problem was with setting all env like project-config puppet-master, etc.. | 15:19 |
| mordred | Shrews: I think we missed os_floating_ip when we were workign on that | 15:19 |
| *** otter768 has quit IRC | 15:19 | |
| *** kevinbenton_ has quit IRC | 15:20 | |
| Shrews | mordred: ugh | 15:20 |
| mrmartin | mordred, Shrews: previous version 0.16.0 seems to be ok, I'm testing now. | 15:20 |
| *** kevinbenton has joined #openstack-infra | 15:20 | |
| mordred | mrmartin: yah. it's definite broken | 15:20 |
| mordred | patch coming | 15:20 |
| *** wenlock_ has joined #openstack-infra | 15:21 | |
| tdurakov | sdague, should I use devstack-gate for new job, right? | 15:21 |
| sdague | tdurakov: right, I was mostly wondering if the issue was local reproduce | 15:21 |
| sdague | tdurakov: yes, we use devstack-gate | 15:21 |
| mrmartin | mordred, I think it is not late, as we don't have a stable release from ansible 2.0.0 yet. | 15:21 |
| mordred | mrmartin: I agree | 15:21 |
| mordred | Shrews: I think we can actually remove most of the code in that module at this point | 15:21 |
| sdague | devstack-vagrant started down the path of making it easy to replicate devstack-gate locally | 15:21 |
| mrmartin | I just wanted to launch an instance :) | 15:22 |
| sdague | it's not fully there though | 15:22 |
| *** abregman has quit IRC | 15:22 | |
| mordred | mrmartin: so - thanks for the report :) | 15:24 |
| tdurakov | sdague, what project should be right place for scripts, that sets NFS and ceph and orchestrating config switching? | 15:24 |
| Shrews | mordred: so, we made attach_ip_to_server private, but not detach_ip_from_server | 15:24 |
| mordred | mrmartin: you can just use auto_ip = True on os_server | 15:24 |
| tdurakov | sdague, devstack-gatse? | 15:25 |
| mordred | mrmartin: and not need to use the os_floating_ip module at all if you just want to launch an instance | 15:25 |
| *** peristeri has joined #openstack-infra | 15:25 | |
| mordred | Shrews: that's because we win at life | 15:25 |
| *** trown is now known as trown|brb | 15:26 | |
| Shrews | mordred: do we need to undo that for os_floating_ip to work? | 15:26 |
| mordred | Shrews: no - I think we can fix os_floating_ip | 15:26 |
| *** tonytan4ever has joined #openstack-infra | 15:26 | |
| mordred | Shrews: to use add_ips_to_server instead | 15:26 |
| mrmartin | mordred, ok I can imagine a use-case where I need to use an os_floating_ip, anyway it works well with the downgraded 0.16.0 shade. | 15:26 |
| mordred | mrmartin: yah - Imean, the module shold work | 15:26 |
| mordred | just wanted to let you konw if you were blocked on your current task :) | 15:27 |
| *** ddieterly has quit IRC | 15:27 | |
| *** salv-orlando has joined #openstack-infra | 15:27 | |
| *** chrall_ has joined #openstack-infra | 15:28 | |
| sdague | tdurakov: I think devstack-gate is probably not the right answer, this is too specific to a test payload | 15:28 |
| greghaynes | pabelanger: yes, like I said. Apache is almost certainly not the issue. Max-requests is usually how you make the processes use less RAM. | 15:28 |
| *** amotoki has quit IRC | 15:28 | |
| *** salv-orlando has quit IRC | 15:28 | |
| tdurakov | sdague, so we need to set up env using devstack-gatse, then run some external script for configuration and tests, right? | 15:29 |
| *** dangers_away is now known as dangers | 15:29 | |
| pabelanger | greghaynes: any docs on how that works? | 15:29 |
| sdague | tdurakov: lets start with this in the nova tree | 15:29 |
| sdague | tdurakov: yes | 15:29 |
| *** cbader has quit IRC | 15:29 | |
| pabelanger | greghaynes: my understanding is the process will respawn after x requests | 15:30 |
| tdurakov | sdague, nova/devstack is ok for that? | 15:30 |
| greghaynes | In uwsgi docs. It is just memory fragmentation | 15:30 |
| greghaynes | pabelanger: yes | 15:30 |
| *** trown|brb is now known as trown | 15:30 | |
| pabelanger | greghaynes: Right, I'll see about adding it. However each process from start up consumes 3.5GB of RAM | 15:31 |
| pabelanger | per process | 15:31 |
| sdague | no, that's going to be a little odd I think | 15:31 |
| sdague | how about nova/tests/live_migration | 15:31 |
| pabelanger | I suspect the issue is apache also consuming 3GB per process too | 15:31 |
| sdague | and we'll bikeshed on location later | 15:31 |
| pabelanger | I am testing nginx right now, but it currently looks way better | 15:31 |
| sdague | tdurakov: do you already have a working etherpad for all of this? | 15:32 |
| *** krtaylor has quit IRC | 15:32 | |
| *** someara2 has joined #openstack-infra | 15:32 | |
| *** burgerk has quit IRC | 15:33 | |
| tdurakov | sdague, could make a final script for initial test and nfs setup in 30 minutes | 15:33 |
| tdurakov | sdague, it would be ok imho to start with 2 of 3 config | 15:33 |
| mordred | Shrews: something like this: http://paste.openstack.org/show/478331 | 15:33 |
| sdague | tdurakov: right, I'm just saying an etherpad would be handy to figuring out some assumptions up front instead of waiting for gerrit feedback | 15:33 |
| *** links has quit IRC | 15:34 | |
| *** zeih has joined #openstack-infra | 15:34 | |
| *** FallenPegasus has joined #openstack-infra | 15:34 | |
| greghaynes | pabelanger: so apache really should not. 3.5g isn't that much, so it'd be significantly less than 7gb total | 15:34 |
| tdurakov | sdague, ok, give me 30 minutes, I'll share etherpad link with you then | 15:34 |
| sdague | tdurakov: sounds great, thanks | 15:34 |
| tdurakov | sdague, thank you:) | 15:35 |
| pabelanger | greghaynes: we also have memcached that takes up 4GB, plus 2GB for stackalytics-processor, then another 2Gb for system processes. So right now, including apache, we are maxing our 15GB | 15:35 |
| pabelanger | I am testing with 2 processes for uwsgi right now | 15:36 |
| *** Ala has joined #openstack-infra | 15:36 | |
| pabelanger | with ngnix and it is night and day with apache | 15:36 |
| pabelanger | so, looks like I'll need to fine a way to tune apache better | 15:36 |
| greghaynes | Sure, so if we can't fit the python process from the start in how much RAM we have then we don't have enough. | 15:36 |
| greghaynes | Is it possible you're using a different set of uwsgi settings? | 15:37 |
| *** placko has quit IRC | 15:37 | |
| pabelanger | Well, right now stackalytics.com uses nginx and stackalytics.o.o is apache (default settings) | 15:37 |
| pabelanger | uwsgi is configured the same for both | 15:37 |
| pabelanger | now it is | 15:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Plumb fixed_address through add_ips_to_server https://review.openstack.org/243169 | 15:38 |
| mordred | Shrews: we'll need that ^^ | 15:38 |
| *** zeih has quit IRC | 15:39 | |
| pabelanger | 2024 www-data 20 0 3661116 3.403g 2424 S 0.0 43.6 10:57.90 uwsgi | 15:39 |
| *** sergmelikyan has quit IRC | 15:39 | |
| pabelanger | so, 10mins to launch wsgi process and currently 3.4GB | 15:39 |
| pabelanger | that is not a full process | 15:39 |
| Clint | whee | 15:39 |
| *** ifarkas has quit IRC | 15:40 | |
| *** someara2 has quit IRC | 15:40 | |
| greghaynes | Yea, that is just the python program it is running though? | 15:41 |
| pabelanger | ya | 15:41 |
| pabelanger | python app loads everything my memcached at start | 15:41 |
| greghaynes | So, any server would be running the same thing. | 15:42 |
| pabelanger | Right | 15:42 |
| mordred | mrmartin: are you in a position to test a fix to the ansible module? | 15:42 |
| *** someara2 has joined #openstack-infra | 15:42 | |
| mordred | mrmartin: if so, https://github.com/emonty/ansible-modules-core/tree/fix-floating-ip-api | 15:42 |
| *** marekd has joined #openstack-infra | 15:42 | |
| mordred | mrmartin: oh, wait -nevermind, you'll need a shade patch too | 15:42 |
| pabelanger | but in our current case, apache seems to dupe memory usage. Don't know why | 15:43 |
| mrmartin | mordred, if you provide me the shade patch, I can do the tests here | 15:43 |
| greghaynes | pabelanger: because it probably is doing the prefork optimization | 15:43 |
| fungi | pabelanger: greghaynes: this is almost certaining courtesy of the in-memory database "design" | 15:43 |
| fungi | s/certaining/certainly/ | 15:43 |
| mordred | mrmartin: https://review.openstack.org/243169 | 15:43 |
| greghaynes | Yea, I bet it's those two combined | 15:44 |
| mrmartin | looking now. | 15:44 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Drop tag pushing rights from ironic-inspector-release https://review.openstack.org/243171 | 15:44 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-jenkins: SECURITY-218: Update to latest cli-shutdown.groovy https://review.openstack.org/242787 | 15:44 |
| *** mtanino has joined #openstack-infra | 15:45 | |
| pabelanger | fungi: Ya, I suspect there will be some updates to stackalytics to be more forgiving with RAM | 15:45 |
| *** krtaylor has joined #openstack-infra | 15:45 | |
| greghaynes | pabelanger: if something else doesn't do that, then it isn't storing a preforked process in memory which means restarting workers will take the full 10mins | 15:45 |
| asselin__ | fungi, ^^ updates to latest script | 15:45 |
| pabelanger | greghaynes: right, that make sense | 15:45 |
| greghaynes | It's possible the nginx one is just doing it in an extra uwsgi process... | 15:45 |
| pabelanger | makes* | 15:45 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-jenkins: SECURITY-218: Update to latest cli-shutdown.groovy https://review.openstack.org/242787 | 15:46 |
| BobBall | How do I find a specific gerrit user's user ID? | 15:46 |
| *** dansmith is now known as thedan | 15:46 | |
| greghaynes | Since that is the only real difference - in mod_wsgi I think uwsgi runs in-process | 15:46 |
| *** kun_huang has joined #openstack-infra | 15:46 | |
| *** chrall__ has joined #openstack-infra | 15:46 | |
| *** thedan is now known as dansmith | 15:46 | |
| *** sergmelikyan has joined #openstack-infra | 15:46 | |
| fungi | BobBall: you can use the gerrit rest api to perform a lookup | 15:47 |
| kun_huang | dhellmann: hi, do you know what's the EOF date for juno? | 15:47 |
| mordred | yah. I believe you want to use mod_uwsgi for apache | 15:47 |
| mordred | not mod_wsgi | 15:47 |
| *** davideagnello has joined #openstack-infra | 15:47 | |
| greghaynes | Er, that | 15:47 |
| mordred | so that uwsgi is a thing that just runs and apache connects to it | 15:48 |
| mordred | rather than the other thing | 15:48 |
| greghaynes | I think that is mod_wsgi_proxy? | 15:48 |
| openstackgerrit | Adam Spiers proposed openstack-infra/project-config: Make #openstack-ha receive HA guide change alerts https://review.openstack.org/243173 | 15:48 |
| * mordred waves hands in the air | 15:48 | |
| *** Liuqing has quit IRC | 15:48 | |
| *** achanda has quit IRC | 15:48 | |
| greghaynes | Mod_proxy_uwsgi | 15:48 |
| *** cbader has joined #openstack-infra | 15:49 | |
| greghaynes | But yea, that would be more similar to how nginx does it, but I think should have similar usage | 15:49 |
| fungi | mod_for_mod_proxy_thing_mod | 15:49 |
| pabelanger | mordred: currently using proxy_uwsgi in our config | 15:49 |
| BobBall | fungi: Which port? Seems that 8080 is the default for gerrit rest, but review.openstack.org doesn't respond on that port for me? | 15:49 |
| *** chrall_ has quit IRC | 15:50 | |
| fungi | BobBall: it's just a gerrit url. i'll dig up an example query for you, just a sec | 15:50 |
| pabelanger | giving mod_uwsgi a go now | 15:50 |
| fungi | trying to wrap up a meeting right now | 15:50 |
| BobBall | fungi: I see - sorry - that works. | 15:51 |
| *** garyk has quit IRC | 15:51 | |
| greghaynes | pabelanger: so a better question - if you run with proxy_uwsqi with two workers and make one req to the site, how much free RAM does free -m show | 15:51 |
| greghaynes | A lot more useful than adding up used | 15:51 |
| *** davideagnello has quit IRC | 15:52 | |
| *** dolpher has quit IRC | 15:53 | |
| *** peterlisak has quit IRC | 15:53 | |
| pabelanger | greghaynes: here is the current memory with proxy_uswgi and 2 uwsgi processes (plus other things listed above) | 15:53 |
| pabelanger | http://paste.openstack.org/show/478332/ | 15:53 |
| *** kun_huang has left #openstack-infra | 15:53 | |
| * greghaynes waits for paste.o.o... | 15:54 | |
| *** someara2 has quit IRC | 15:54 | |
| *** Sukhdev has joined #openstack-infra | 15:54 | |
| greghaynes | Yea so 2.5gigish | 15:54 |
| greghaynes | Sounds like the app itself just uses more than we have | 15:55 |
| *** burgerk has joined #openstack-infra | 15:55 | |
| *** maishsk has quit IRC | 15:56 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Prepopulate log server for single-node-ci https://review.openstack.org/242800 | 15:57 |
| clayton | Does nodepool now require configuring things via os-cloud-config? Our internal instance upgraded to master and now none of the authentication works (after updating secure.conf) | 15:57 |
| *** achanda has joined #openstack-infra | 15:57 | |
| *** morgabra has quit IRC | 15:58 | |
| pabelanger | clayton: Been a few weeks, but I could only get it working by using os-cloud-config | 15:58 |
| pabelanger | clayton: didn't get a chance to debug why | 15:59 |
| clayton | seems like the docs haven't been updated to reflect that :/ | 15:59 |
| pabelanger | clayton: Ya, noticed that too :( | 15:59 |
| clayton | do you remember how the sections of the nodepool.yaml correspond to the cloud config? | 15:59 |
| *** someara2 has joined #openstack-infra | 15:59 | |
| greghaynes | I don't remember us removing support for nodepool.yaml provider with config | 16:00 |
| rcarrillocruz | clayton: yeah, your 'nodepool user should have .config/openstack/clouds.yaml | 16:00 |
| *** tiswanso has quit IRC | 16:00 | |
| rcarrillocruz | 'nodepool' | 16:00 |
| *** sfinucan has joined #openstack-infra | 16:00 | |
| greghaynes | But I might have missed it | 16:00 |
| *** sergmelikyan has quit IRC | 16:00 | |
| rcarrillocruz | check it out puppet-openstackci, chances are they already use clouds.yaml so you can adapt to your environment | 16:00 |
| rcarrillocruz | asselin ^ | 16:00 |
| clayton | rcarrillocruz do you know how providers map to clouds.yaml? | 16:00 |
| rcarrillocruz | i can check | 16:01 |
| rcarrillocruz | sec | 16:01 |
| *** tiswanso has joined #openstack-infra | 16:01 | |
| *** jordanP has quit IRC | 16:01 | |
| pabelanger | clayton: not off the top of my head, sorry | 16:01 |
| *** chrall__ has quit IRC | 16:01 | |
| *** morgabra has joined #openstack-infra | 16:01 | |
| *** chrall_ has joined #openstack-infra | 16:02 | |
| asselin__ | clayton, see here for patch (not yet merged) https://review.openstack.org/#/c/237365/ | 16:02 |
| *** eharney has quit IRC | 16:02 | |
| *** morgabra has quit IRC | 16:02 | |
| *** morgabra has joined #openstack-infra | 16:02 | |
| clayton | ah, thanks :) | 16:02 |
| rcarrillocruz | clayton: you'd have something like: | 16:02 |
| rcarrillocruz | providers: | 16:02 |
| rcarrillocruz | name: <cloud name> | 16:02 |
| clayton | I'd be happy to just have my existing config work though, and do the os-cloud-config migration later, since this is blocking internal CI | 16:02 |
| rcarrillocruz | region-name: <region name> | 16:02 |
| rcarrillocruz | cloud: <your OSCC cloud name from clouds.yaml> | 16:03 |
| rcarrillocruz | that's how you do the mapping | 16:03 |
| *** chrall__ has joined #openstack-infra | 16:03 | |
| mordred | rcarrillocruz: no - we removed that | 16:03 |
| *** ddieterly has joined #openstack-infra | 16:03 | |
| mordred | it shold just be in /etc/openstack | 16:03 |
| rcarrillocruz | k, we have that in prod, probably we haven't resynced yet | 16:04 |
| rcarrillocruz | yolanda ^ | 16:04 |
| *** Ala has quit IRC | 16:04 | |
| mordred | oh - wait sorry | 16:04 |
| mordred | don't listen to me | 16:04 |
| *** nadya has joined #openstack-infra | 16:04 | |
| mordred | I thought you were talking about devstack jobs | 16:04 |
| mordred | both should work | 16:04 |
| rcarrillocruz | no, just nodepool oscc | 16:04 |
| mordred | (bot locations) | 16:04 |
| mordred | yah | 16:04 |
| mordred | both | 16:04 |
| * rcarrillocruz is relieved for having one less thing to care about in prod | 16:05 | |
| mordred | also - direct config in nodepool.yaml _should_ work | 16:05 |
| *** nadya_ has quit IRC | 16:05 | |
| greghaynes | Yea, I didn't remember that getting removed... | 16:05 |
| pabelanger | mordred: ansible role naming question, what is your preferred way to name role? For example: does ansible-role-foobar make sense to you, if foobar was the app you were configuring | 16:05 |
| mordred | but I don't think anyone has spent any time debugging the issues non-clouds.yaml folks have seen | 16:05 |
| greghaynes | It is possible our software has some bugs in it ;) | 16:05 |
| mordred | pabelanger: well, we just did ansible-puppet for the git repo for the puppet role | 16:06 |
| mordred | pabelanger: and the role itself is called "puppet" | 16:06 |
| *** chrall_ has quit IRC | 16:06 | |
| mordred | http://git.openstack.org/cgit/openstack-infra/ansible-puppet | 16:06 |
| mordred | for reference | 16:06 |
| *** maishsk has joined #openstack-infra | 16:07 | |
| *** woodster_ has joined #openstack-infra | 16:07 | |
| *** krotscheck has joined #openstack-infra | 16:07 | |
| *** fhubik has quit IRC | 16:07 | |
| *** yamamoto has joined #openstack-infra | 16:08 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Prepopulate log server for single-node-ci https://review.openstack.org/242800 | 16:08 |
| mrmartin | mordred: I got this invalid syntax error: http://paste.openstack.org/show/478333/ at ansible side: https://github.com/ansible/ansible-modules-core/compare/devel...emonty:fix-floating-ip-api#diff-a0a94c4d24f3d51fb0d3692ad2818623L173 | 16:08 |
| *** thedodd has joined #openstack-infra | 16:08 | |
| *** salv-orlando has joined #openstack-infra | 16:09 | |
| mordred | mrmartin: HEADDESK | 16:09 |
| TheJulia | win 9 | 16:09 |
| TheJulia | doh | 16:09 |
| mordred | mrmartin: I pushed afix for that - sorry for being dumb | 16:10 |
| mrmartin | ok, I'm checking now | 16:10 |
| openstackgerrit | Ivan Kolodyazhny proposed openstack-infra/project-config: Add python-brickclient to OpenStack https://review.openstack.org/243080 | 16:10 |
| yolanda | mordred, rcarrillocruz, so we haven't still resynced that bits | 16:11 |
| yolanda | about nodepool | 16:11 |
| *** arxcruz has quit IRC | 16:11 | |
| * yolanda is a bit traumatized with resyncs | 16:11 | |
| rcarrillocruz | lol | 16:11 |
| rcarrillocruz | yeah | 16:11 |
| openstackgerrit | Timofey Durakov proposed openstack-infra/project-config: Initial job for live-migrations https://review.openstack.org/243013 | 16:11 |
| rcarrillocruz | enough crap today | 16:11 |
| openstackgerrit | Darragh Bailey proposed openstack/python-jenkins: Use metaprogramming to add additional url tests https://review.openstack.org/231109 | 16:12 |
| *** thedodd has quit IRC | 16:12 | |
| *** sergmelikyan has joined #openstack-infra | 16:12 | |
| *** thedodd has joined #openstack-infra | 16:12 | |
| *** sergmelikyan has quit IRC | 16:12 | |
| fungi | jeblair: clarkb: asselin__ has proposed 242787 to update the groovy script for the recent jenkins cli vulnerability to the latest version from github, which should in theory fix the startup race we observed | 16:12 |
| *** yamamoto has quit IRC | 16:12 | |
| asselin__ | fungi, I have this setup here: http://15.125.105.66:8080/cli/ | 16:14 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/irc-meetings: update the releaseteam meeting for mitaka https://review.openstack.org/243188 | 16:14 |
| *** chrall__ has quit IRC | 16:14 | |
| fungi | asselin__: awesome. and that's after a restart too? | 16:14 |
| asselin__ | fungi, not sure how to test 'startup races' though. sudo service jenkins restart seems to work fine. | 16:14 |
| *** maishsk has quit IRC | 16:14 | |
| asselin__ | fungi, yes | 16:14 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/irc-meetings: update the releaseteam meeting for mitaka https://review.openstack.org/243188 | 16:14 |
| * asselin__ restarts it again | 16:14 | |
| fungi | yeah, the implication was that we were hitting the startup race for that due to "activity volume" on our masters | 16:15 |
| *** maishsk has joined #openstack-infra | 16:15 | |
| fungi | so it may not be consistently reproducable | 16:15 |
| asselin__ | fungi, ok, well that's a test ci setup, so if you need something done on that, let me know | 16:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config: WIP: Add job to test ansible modules against shade https://review.openstack.org/243189 | 16:16 |
| fungi | asselin__: thanks, i don't really know what else to check besides getting a 404 from /cli | 16:16 |
| fungi | lgtm | 16:16 |
| *** scheuran has quit IRC | 16:16 | |
| *** tonytan4ever has quit IRC | 16:16 | |
| jeblair | fungi: wow. | 16:16 |
| jeblair | +3 | 16:16 |
| asselin__ | fungi, this (from the readme) also fails correctly after the change: To see if the mitigation is successfully applied, try running java -jar cli.jar -s $JENKINS_URL with cli.jar and make sure you get EOFException like the following: | 16:17 |
| fungi | BobBall: did you still need a gerrit rest api user query example, or did you manage to find what you need? | 16:17 |
| *** eharney has joined #openstack-infra | 16:17 | |
| BobBall | I got it, thanks fungi | 16:17 |
| *** Sree has quit IRC | 16:17 | |
| fungi | asselin__: yep, i more meant not sure what we could do to attempt to exercise the old race condition, but it's good to check the cli calls too agreed | 16:18 |
| fungi | thanks for updating that for us! | 16:18 |
| *** chrall__ has joined #openstack-infra | 16:19 | |
| electrical | asselin__: fungi i just bind jenkisn to localhost and only proxy 8080 since cli is on a random port. | 16:19 |
| *** sorantis has quit IRC | 16:19 | |
| mrmartin | mordred: it is almost good, but "os_floating_ip\", line 159, in main\n server=server, ips=f_ip, reuse=reuse,\nUnboundLocalError: local variable 'f_ip' referenced before assignment" the f_ip gets an assignment later, anyway. | 16:20 |
| fungi | electrical: except the jenkins rest api over the web interface is also vulnerable | 16:20 |
| fungi | (in theory) | 16:20 |
| openstackgerrit | Adam Spiers proposed openstack-infra/irc-meetings: Add a weekly HA meeting https://review.openstack.org/243193 | 16:20 |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder: Fix diskimage-builder image size https://review.openstack.org/242273 | 16:20 |
| electrical | fungi: ah okay. i missed that part.. | 16:20 |
| fungi | electrical: they implemented a "fallback" for the cli tool to use the http(s) interface | 16:21 |
| electrical | ahh i see. | 16:21 |
| fungi | and since they implement the same serialization, and that's where the vulnerability resides, they should both (tcp socket and http interface) be similarly exploitable | 16:21 |
| electrical | Hmm, very true. | 16:22 |
| mordred | mrmartin: cool. I just pushed one more patch that should fix that for you | 16:22 |
| fungi | electrical: another workaround we toyed with is using apache to reject requests for /cli but i don't know for sure that's a 100% effective solution either | 16:23 |
| electrical | fungi: could it be that it mainly has to do with the cli-tool jar thing that handles the serialization between the server and client version? | 16:23 |
| fungi | or what other nuanced ways there might be to reach the cli interface through the web service | 16:23 |
| electrical | but yeah.. disabling the /cli path might be better yeah | 16:23 |
| electrical | since it should connect to that part | 16:24 |
| *** eharney has quit IRC | 16:24 | |
| fungi | electrical: well, the vulnerability can't be in the client. after all, if they "fixed" it client side then you could just exploit it by writing your own client | 16:24 |
| electrical | hmm true | 16:24 |
| fungi | (or even easier, running a version of the client form before the fix) | 16:25 |
| *** sergmelikyan has joined #openstack-infra | 16:25 | |
| fungi | s/form/from/ | 16:25 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: WIP: Add Ansible testing infrastructure https://review.openstack.org/243196 | 16:25 |
| *** hdd has quit IRC | 16:25 | |
| *** e0ne has quit IRC | 16:26 | |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: SECURITY-218: Update to latest cli-shutdown.groovy https://review.openstack.org/242787 | 16:26 |
| openstackgerrit | John Warren proposed openstack-infra/puppet-openstackci: Add generic graphite manifest https://review.openstack.org/243199 | 16:26 |
| *** rkukura_ has joined #openstack-infra | 16:27 | |
| *** flepied has quit IRC | 16:27 | |
| *** rkukura has quit IRC | 16:27 | |
| *** rkukura_ is now known as rkukura | 16:27 | |
| electrical | fungi: https://jenkins-ci.org/content/mitigating-unauthenticated-remote-code-execution-0-day-jenkins-cli with the groovy script it disables the whole cli thing as well | 16:28 |
| *** peterlisak has joined #openstack-infra | 16:29 | |
| fungi | electrical: yep, anyone doing continuous deployment from our puppet-jenkins repository already has it http://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/files/cli-shutdown.groovy | 16:30 |
| asselin__ | fungi, your welcome...I as setting up public jenkins and I saw the discussion here on friday, so it was easy to test. | 16:30 |
| fungi | just needs a jenkins service restart | 16:30 |
| electrical | ahh okay. cool | 16:30 |
| openstackgerrit | Adam Spiers proposed openstack-infra/project-config: Make #openstack-ha receive HA guide change alerts https://review.openstack.org/243173 | 16:30 |
| tdurakov | sdague, https://etherpad.openstack.org/p/live-migration-multinode | 16:30 |
| fungi | electrical: we added it friday (well, early saturday utc) but it had some bugs which have only just been fixed and synched over in the past few minutes | 16:31 |
| electrical | fungi: ah i see. I just heard about it a hour ago :-) | 16:31 |
| electrical | to busy partying as usual. lol | 16:32 |
| fungi | heh | 16:32 |
| fungi | this is how openstack infra parties ;) | 16:32 |
| electrical | :p | 16:32 |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror https://review.openstack.org/165240 | 16:32 |
| electrical | Went to a concert of Jamie Cullum yesterday with my new date :-) the things i do for a girl. lol | 16:33 |
| *** salv-orlando has quit IRC | 16:33 | |
| tdurakov | sdague, updated patch to run tempest explicitly | 16:33 |
| clarkb | fungi: jeblair asselin__ we should attempt restarting jenkins.o.o once the latest patch version is in | 16:33 |
| clarkb | since that was how the fail was previously found | 16:33 |
| *** Piet has joined #openstack-infra | 16:34 | |
| *** e0ne has joined #openstack-infra | 16:34 | |
| openstackgerrit | Darragh Bailey proposed openstack/python-jenkins: Use TestWithScenarios to add additional url tests https://review.openstack.org/231109 | 16:35 |
| *** eharney has joined #openstack-infra | 16:36 | |
| openstackgerrit | Sean Dague proposed openstack-infra/system-config: fix error multiline folding for indexing https://review.openstack.org/243211 | 16:36 |
| mrmartin | mordred: ok one more step, and it will be splendid: you forget to add 'cloud' as a first parameter to _get_floating_ip call, so you need this one: f_ip = _get_floating_ip(cloud, fip_address) | 16:37 |
| sdague | clarkb / mriedem - https://review.openstack.org/243211 | 16:37 |
| mordred | mrmartin: hahaha. thanks! | 16:37 |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror https://review.openstack.org/165240 | 16:37 |
| clarkb | nibalizer: I am ready to start elasticsearch things whenever you are | 16:37 |
| sdague | that explains the mystery of no more loglevel:ERROR lines in logstash | 16:37 |
| *** peterlisak has quit IRC | 16:37 | |
| mrmartin | mordred: with that fix, works well for me | 16:38 |
| clarkb | something something I hate logging | 16:38 |
| *** peristeri has quit IRC | 16:38 | |
| greghaynes | someone who knows anything about puppet should review https://review.openstack.org/#/c/165240/ for the setting up wheel mirrrors | 16:39 |
| *** baoli has quit IRC | 16:39 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add heat lbaas specific test for v1 and v2 https://review.openstack.org/237156 | 16:39 |
| *** tiswanso has quit IRC | 16:39 | |
| mordred | mrmartin, Shrews: https://github.com/ansible/ansible-modules-core/pull/2436 | 16:39 |
| *** cdent has quit IRC | 16:39 | |
| mordred | mrmartin: I squashed the commits into one | 16:39 |
| mordred | mrmartin: and re-pushed to that branch - also added a co-authored-by footer | 16:39 |
| mordred | :) | 16:39 |
| mrmartin | thnx | 16:40 |
| *** tiswanso has joined #openstack-infra | 16:40 | |
| *** harlowja_at_home has joined #openstack-infra | 16:40 | |
| *** maishsk has quit IRC | 16:41 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config: Add job to test ansible modules against shade https://review.openstack.org/243189 | 16:44 |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Bump os-client-config upper constraint https://review.openstack.org/243217 | 16:44 |
| *** maiteb has joined #openstack-infra | 16:44 | |
| *** rhallisey has joined #openstack-infra | 16:44 | |
| *** sdake has quit IRC | 16:45 | |
| *** thiagop has joined #openstack-infra | 16:45 | |
| rhallisey | hello. Looking for a second review for tripleo-docker job. https://review.openstack.org/#/c/238212/ | 16:46 |
| *** tonytan4ever has joined #openstack-infra | 16:46 | |
| mordred | Shrews: we should probably add those as non-voting in check first so that we can make sure it works | 16:46 |
| mordred | Shrews: then follow up with a patch to add to gate | 16:47 |
| tdurakov | sdague, seen your comments | 16:47 |
| mordred | Shrews: if that doens't make sense to you I can update your change | 16:47 |
| sdague | tdurakov: yeh, I think probably the next step is the nova patch for the test hook that can be called from the devstack gate job | 16:47 |
| sdague | but, otherwise, I think this all looks pretty on track | 16:48 |
| *** gyee has joined #openstack-infra | 16:48 | |
| tdurakov | sdague, ok, I'll update current infra patch soon and add one for nova. | 16:48 |
| *** sergmelikyan has quit IRC | 16:49 | |
| *** jlanoux has quit IRC | 16:49 | |
| tdurakov | sdague, I'll ping you when it's ready to review. what about devstack plugin? | 16:50 |
| sdague | tdurakov: thanks | 16:50 |
| Shrews | mordred: please update accordingly | 16:50 |
| *** nadya has quit IRC | 16:50 | |
| Shrews | mordred: i understand and agree, just not sure where to change that :) | 16:51 |
| *** peristeri has joined #openstack-infra | 16:52 | |
| sdague | tdurakov: put a couple of phases of success in there | 16:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Add job to test ansible modules against shade https://review.openstack.org/243189 | 16:52 |
| mordred | Shrews: voila | 16:52 |
| sdague | so lets just get the existing stuff working in a new job for phase 1, and call that the first checkpoint | 16:52 |
| *** mfalatic has joined #openstack-infra | 16:52 | |
| sdague | and once that looks stable, we can add the reconfiguring of services, which might need a devstack plugin | 16:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Make ansible job for shade voting https://review.openstack.org/243220 | 16:53 |
| mordred | and there is the follow up change for once we like the results | 16:53 |
| mordred | I will now procedural -2 that one | 16:53 |
| tdurakov | sdague, ok | 16:54 |
| Shrews | mordred: Sokath, his eyes opened | 16:54 |
| *** tdurakov has quit IRC | 16:54 | |
| mordred | Shrews: Darmok and Jalad at Tanagra | 16:55 |
| TheJulia | heh | 16:55 |
| fungi | most... annoying... episode... EVAR | 16:56 |
| Shrews | fungi: you hush! | 16:56 |
| * fungi hushes | 16:56 | |
| openstackgerrit | matthew wagoner proposed openstack-infra/shade: Add new context manager for shade exceptions, cont. again https://review.openstack.org/243221 | 16:56 |
| *** ihrachys has quit IRC | 16:56 | |
| *** cody-somerville has quit IRC | 16:57 | |
| *** kushal has joined #openstack-infra | 16:57 | |
| *** bpokorny has joined #openstack-infra | 16:58 | |
| *** baoli has joined #openstack-infra | 16:58 | |
| mordred | olaph: I like patches that are +100/-259 | 16:58 |
| *** haypo has left #openstack-infra | 16:58 | |
| mordred | Shrews: clarkb when the ELK fell? | 16:59 |
| clarkb | mordred: ? | 16:59 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: Move Item formatting into Reporters https://review.openstack.org/223298 | 16:59 |
| clarkb | if nibalizer will wake up we can get started on upgrade nowish | 16:59 |
| clarkb | I might start without nibalizer in a few | 16:59 |
| mordred | clarkb: I'm making star trek communication metaphors based on your exploits | 17:00 |
| mordred | clarkb: it's better to just ignore | 17:00 |
| *** sabeen has joined #openstack-infra | 17:00 | |
| sdague | mordred: ++ for clarkb when the ELK fell? | 17:01 |
| *** tlian2 has joined #openstack-infra | 17:01 | |
| fungi | clarkb: you could just drive 'round to his place and start ringing the doorbell until he stirs? | 17:01 |
| clarkb | fungi: I could, but that would involve going out in the cold rainy outside | 17:02 |
| mordred | fungi: we could install a robot on his place that rings his doorbell in response to us pinging him on IRC | 17:02 |
| fungi | sounds like a plan | 17:02 |
| mordred | fungi: it's possible he might hate us - but I think we'd all find it amusing | 17:02 |
| greghaynes | clarkb: Maybe we can send some amazon now to his place | 17:03 |
| greghaynes | clarkb: and write in the notes to bang really loudly | 17:03 |
| clarkb | hahaha | 17:03 |
| sdague | that's it, order him a pizza | 17:03 |
| *** ilyashak_ has joined #openstack-infra | 17:03 | |
| jeblair | or a bottle of water | 17:03 |
| sdague | pizza delivery for "Time to upgrade Elastic Search" | 17:03 |
| clarkb | (I really don't want to be the only person elasticsearching though so am inclined to wait a bit) | 17:04 |
| *** daemontool has quit IRC | 17:04 | |
| *** tlian has quit IRC | 17:04 | |
| mordred | hahahaha | 17:04 |
| *** flepied has joined #openstack-infra | 17:04 | |
| * mordred really wants someone to have a kid and name it "Time to upgrade Elastic Search" | 17:04 | |
| crinkle | greghaynes: what happened to the idea of distributing host keys with ansible instead of puppet (re 165240) | 17:04 |
| *** daemontool has joined #openstack-infra | 17:04 | |
| greghaynes | crinkle: nothing, fixing the host key thing was something that came to my mind when writing that but I dont think we should require that beforehand | 17:05 |
| greghaynes | crinkle: I think it is going to be a bit of work | 17:05 |
| openstackgerrit | Merged openstack-infra/shade: Plumb fixed_address through add_ips_to_server https://review.openstack.org/243169 | 17:05 |
| *** matrohon has quit IRC | 17:05 | |
| crinkle | greghaynes: heh ok | 17:05 |
| mordred | WOOT! | 17:05 |
| mordred | my neutronclient patch to add support for clouds.yaml landed | 17:06 |
| crinkle | greghaynes: help me understand why jenkins-pypi-mirror.yaml was removed in that patch? the commit message doesn't really say | 17:06 |
| mordred | the world domination is now starting | 17:06 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: [Rally] add new job for verification https://review.openstack.org/240580 | 17:06 |
| *** Sree has joined #openstack-infra | 17:07 | |
| greghaynes | crinkle: oh, that file was there from back when we ran pypi-mirror. I originally updated it but then realized we dont need to use pypi-mirror... so just removing the stale file | 17:07 |
| *** rossella_ has quit IRC | 17:07 | |
| greghaynes | crinkle: should probably put that in msg | 17:07 |
| clarkb | fungi: jeblair has anyone tried restarting jenkins.o.o with the latest groovy patch in place? | 17:07 |
| *** rossella_ has joined #openstack-infra | 17:07 | |
| TheJulia | win 12 | 17:08 |
| TheJulia | ddoh | 17:08 |
| clarkb | if it does work I Think we should consider running the master restart playbook as 02 and 04 are both thread leaking | 17:08 |
| *** achanda has quit IRC | 17:09 | |
| *** e0ne has quit IRC | 17:09 | |
| *** sputnik13 has joined #openstack-infra | 17:10 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: update the releaseteam meeting for mitaka https://review.openstack.org/243188 | 17:11 |
| *** krotscheck has quit IRC | 17:12 | |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Add requestsexceptions to global requirements https://review.openstack.org/241324 | 17:12 |
| *** sputnik13 has quit IRC | 17:12 | |
| *** jed56 has quit IRC | 17:12 | |
| fungi | clarkb: not yet | 17:13 |
| fungi | but i can give it a shot in a moment | 17:14 |
| *** samuelBartel has quit IRC | 17:14 | |
| clarkb | cool | 17:14 |
| *** armax has joined #openstack-infra | 17:14 | |
| *** otter768 has joined #openstack-infra | 17:15 | |
| *** AJaeger has joined #openstack-infra | 17:15 | |
| clayton | anyone seen an error like this from nodepool: OpenStackCloudException: Error getting compute endpoint on :NCE: The request you have made requires authentication. (HTTP 401) (Request-ID: req-df5f123b-5397-4cb1-a7d8-f1415afba2c7) (Inner Exception: The request you have made requires authentication.) | 17:15 |
| *** sabeen3 has joined #openstack-infra | 17:16 | |
| clayton | found a mention here from june-ish, but the resolution wasn't clear | 17:16 |
| mordred | clayton: that looks like your cloud config settings are potentially unhappy | 17:16 |
| nibalizer | clarkb: ohai | 17:16 |
| *** sergmelikyan has joined #openstack-infra | 17:16 | |
| mordred | clayton: can you paste your clouds.yaml (redacting your password of course) | 17:16 |
| clayton | not using /etc/openstack/clouds.yaml yet | 17:16 |
| mordred | ah | 17:16 |
| mordred | ok | 17:17 |
| clarkb | nibalizer: good morning | 17:17 |
| Clint | quick, cancel the pizza | 17:17 |
| mordred | clayton: so, it's possible there is a bug in the non-clouds.yaml case. can you paste your nodepool.yaml without passwords and I'll take a look and see what I can figure out? | 17:17 |
| *** tonytan4ever has quit IRC | 17:17 | |
| clarkb | nibalizer: I think what we want to do is run each hosts block in that playbook as its own playbook | 17:18 |
| clayton | sure, give me a few min | 17:18 |
| clarkb | nibalizer: as that will give us alot of control over each step and we can make sure things are going well | 17:18 |
| *** kushal has quit IRC | 17:18 | |
| mordred | clayton: your cloud does not have a region either I'm taking it? | 17:18 |
| clarkb | nibalizer: also I use elasticsearch head and bigdesk to monitor the cluster | 17:18 |
| *** harlowja_at_home has quit IRC | 17:18 | |
| *** sabeen has quit IRC | 17:18 | |
| nibalizer | with ssh -L or so? | 17:18 |
| mordred | clayton: oh - or, rather, you don't have a named cloud, so NCE is your region | 17:18 |
| clayton | we're provisioning in 3 different clouds, 2 have regions, one does not, all started failing this morning with the same error when we upgraded to nodepool master | 17:18 |
| clayton | right | 17:18 |
| mordred | cool | 17:19 |
| clarkb | nibalizer: yup ssh -L elasticsearch02 -L 9200:localhost:9200 | 17:19 |
| mordred | yeah - post me your thing and I'll see what I can figure out | 17:19 |
| greghaynes | clayton: You mentioned that this is after you rebased to head, it would also be nice to know how far back ago it did work | 17:19 |
| clarkb | nibalizer: so first thing I would do is set those up as the will be handy for visualizing progress | 17:19 |
| clayton | greghaynes: if I knew that I'd have already reverted to that commit :) | 17:19 |
| clarkb | nibalizer: I can give you sha1's that work with our old cluster version | 17:19 |
| greghaynes | clayton: hah, ok | 17:19 |
| mordred | clayton: :) | 17:19 |
| hrubi | jhesketh: please take a look at the 8th patchset in https://review.openstack.org/223298 | 17:19 |
| nibalizer | clarkb: ok | 17:19 |
| *** btully has joined #openstack-infra | 17:19 | |
| *** otter768 has quit IRC | 17:19 | |
| clarkb | nibalizer: bigdesk v2.2.0 and elasticsearch head 3706e79d501bbca457c7a36fa3350131cb76b29e | 17:20 |
| clarkb | git checkout - | 17:20 |
| clarkb | its amazing | 17:20 |
| clarkb | also reflog | 17:20 |
| *** kushal has joined #openstack-infra | 17:20 | |
| *** sturivnyi has joined #openstack-infra | 17:21 | |
| clayton | clarkb oh, hadn't thought of reflog, thanks | 17:21 |
| sturivnyi | Hi guys! | 17:22 |
| sturivnyi | I have issue with `gate-ironicclient-dsvm-functional`. `fatal error: libxml/xmlversion.h: No such file or directory` in devstacklog | 17:23 |
| sturivnyi | Here link -> http://logs.openstack.org/77/238577/11/check/gate-ironicclient-dsvm-functional/e79d0f2/ | 17:23 |
| sturivnyi | Could anybody help with this? | 17:23 |
| nibalizer | clarkb: ok i've got the dashboards up | 17:23 |
| *** btully has quit IRC | 17:24 | |
| clayton | greghaynes it appears that we were on a commit from oct 9th before this broke 95d0e6b678c983a1d662d35a870d0b66a31844ab | 17:25 |
| clayton | I suspect the nodepool code change didn't break things, I suspect newer libraries got pulled in when it ran pip install | 17:25 |
| AJaeger | hi project-config cores, could I get some reviews on https://review.openstack.org/#/c/242975/ to limit running of releasenotes jobs? This will unbreak stable glance branches... | 17:26 |
| pleia2 | good morning | 17:26 |
| greghaynes | clayton: ok, there also have been some recent nodepool changes relating to config processing and clouds.yaml so I wouldnt rule it out ;) | 17:26 |
| clarkb | nibalizer: cool I am recovering for battery death on laptop but I think next step is pull that playbook onto puppet master in your homedir (or wherever) and split it up into a handful of discrete playbooks we can run | 17:27 |
| nibalizer | ok | 17:27 |
| clarkb | nibalizer: I think each hosts block wants to be its own thing | 17:27 |
| nibalizer | do you want to post a new rev to 238185 or just manual? | 17:27 |
| clarkb | either way is fine | 17:27 |
| clarkb | mostly I juts want to have a little control over things so ansible doesn't go crazy if I got something wrong | 17:28 |
| nibalizer | agree | 17:28 |
| *** someara2 has quit IRC | 17:28 | |
| *** krotscheck has joined #openstack-infra | 17:29 | |
| *** tsg has joined #openstack-infra | 17:30 | |
| mordred | greghaynes: I have pasted a patch to clayton to add in some crappy debugging so we can try to figure out where things are going south there | 17:30 |
| clarkb | let me know when you are ready but I think step after that is to run the first 3 playbooks which shutsdown the cluster and restarts the 6 cluster nodes | 17:31 |
| *** unicell1 has quit IRC | 17:34 | |
| BobBall | patrickeast: Scoreboard down? | 17:34 |
| *** amuller is now known as amuller_afk | 17:34 | |
| *** yfried has quit IRC | 17:35 | |
| *** sshnaidm has quit IRC | 17:35 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add ansible playbook to upgrade elasticsearch https://review.openstack.org/238185 | 17:35 |
| nibalizer | clarkb: ^^ | 17:35 |
| clarkb | looking | 17:36 |
| clarkb | lgtm, do you want me to dig up ansible-playbook invocation syntax? | 17:36 |
| nibalizer | nah I've been doing it a lot on my test setup | 17:37 |
| openstackgerrit | Paul Carver proposed openstack-infra/irc-meetings: Update meeting time and day for Kuryr https://review.openstack.org/243228 | 17:37 |
| clarkb | in this case sudo -H ansible-playbook $yaml_file should work fine | 17:37 |
| clarkb | I don't think we need to override the -f default and there are no extra vars | 17:37 |
| nibalizer | cool | 17:37 |
| clarkb | (and system default inventory will find the hosts properly) | 17:37 |
| nibalizer | yes | 17:38 |
| clarkb | nibalizer: and I guess go ahead and start when you are ready | 17:40 |
| nibalizer | want to screen -x it? | 17:41 |
| *** tqtran has joined #openstack-infra | 17:41 | |
| clarkb | sure | 17:41 |
| clarkb | I don't see a screen yet, let me know when it is started | 17:42 |
| nibalizer | okay there is a screen running as nibz | 17:42 |
| nibalizer | or we can root | 17:42 |
| clarkb | you're going to make me try and remember how to attach to $user owned screen | 17:42 |
| fungi | it's annoying to connect to a non-root screen other than your own (you can work around it by doing `script /dev/null` before calling screen -x) | 17:42 |
| nibalizer | we can just root then :P | 17:42 |
| nibalizer | clarkb: screen created | 17:43 |
| clarkb | nibalizer: ansible is already installed on that host fwiw | 17:44 |
| *** rkukura_ has joined #openstack-infra | 17:44 | |
| clarkb | but venving won't hurt | 17:44 |
| clarkb | arst | 17:45 |
| *** rkukura has quit IRC | 17:45 | |
| *** rkukura_ is now known as rkukura | 17:45 | |
| fungi | frightening... https://github.com/rackspace/gophercloud | 17:45 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Update ES to 1.7.3 https://review.openstack.org/243232 | 17:45 |
| clarkb | nibalizer: I am ready for you to run that command if you are, watch bigdesk and all the logstash-workers should go away | 17:46 |
| nibalizer | kk here goes | 17:46 |
| *** hodos has joined #openstack-infra | 17:46 | |
| *** regXboi has quit IRC | 17:47 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/gear: inline helps for interactive commands https://review.openstack.org/66591 | 17:47 |
| *** sdake has joined #openstack-infra | 17:47 | |
| nibalizer | yep they are gone in bigdesk | 17:47 |
| clarkb | nibalizer: and it looks like all 20 were affected | 17:47 |
| clarkb | so we didn't somehow miss one because derped inventory | 17:48 |
| *** stevemar_ has quit IRC | 17:48 | |
| fungi | clarkb: jeblair: asselin__: i've restarted jenkins.o.o with the latest puppet-jenkins change confirmed in place, and https://jenkins.openstack.org/cli/ is still 404 so i _think_ we should be good to start cycling the others if desired | 17:48 |
| nibalizer | looks kinda like worker08 and worker11 already had es stopped but :shrug: | 17:49 |
| clarkb | nibalizer: so if yo uare happy with the ansible output I say do the next one (I am happy with it based on counting the nodes and seeing the changed counts) | 17:49 |
| clarkb | nibalizer: Ithink it must've crashed there probably OOM | 17:49 |
| nibalizer | kk here goes | 17:49 |
| clarkb | fungi: thanks, do we also want my mod rewrite change for extra layers or meh? | 17:49 |
| *** wenlock has quit IRC | 17:49 | |
| clarkb | nibalizer: still looking good, next one is the scary one | 17:50 |
| clarkb | ready when you are | 17:50 |
| fungi | clarkb: maybe just hold it in reserve for now. i worry that's further complication | 17:50 |
| clarkb | fungi: kk | 17:50 |
| clarkb | I can WIP it | 17:50 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/irc-meetings: Rename irc meeting: akanda->astara https://review.openstack.org/243235 | 17:50 |
| *** sergmelikyan has quit IRC | 17:50 | |
| *** wenlock has joined #openstack-infra | 17:51 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: WIP: Add Ansible testing infrastructure https://review.openstack.org/243196 | 17:51 |
| nibalizer | clarkb: does the star in es-head indicate the 'leader' or whatever es calls it? | 17:51 |
| *** wenlock_ has quit IRC | 17:51 | |
| clarkb | nibalizer: yes it is the master | 17:51 |
| nibalizer | definitely feels like es02 is how the scripts are written and es04 is the master | 17:51 |
| *** tiswanso has quit IRC | 17:51 | |
| clarkb | nibalizer: it doesn't matter | 17:51 |
| fungi | who runs elk-town? master blaster runs elk-town | 17:52 |
| nibalizer | kk | 17:52 |
| clarkb | master is an elected thing iwthin es itself | 17:52 |
| clarkb | but they are all data nodes capable of servicing requests (so es02 is my default typically) | 17:52 |
| *** derekh has quit IRC | 17:52 | |
| clayton | just for the benefit of anyone else, if you're getting "OpenStackCloudException: Error getting compute endpoint on :NCE: The request you have made requires authentication. (HTTP 401) (Request-ID: req-df5f123b-5397-4cb1-a7d8-f1415afba2c7) (Inner Exception: The request you have made requires authentication.)" from nodepool | 17:52 |
| clayton | the issue for us was that we had a project name in the project-id field, which now has to be a uuid, not a name. the fix is to use project-name | 17:52 |
| fungi | clayton: ahh, yep i think someone else ran into that as well (might have even been us) | 17:53 |
| *** tonytan4ever has joined #openstack-infra | 17:53 | |
| nibalizer | clarkb: ah yea lets split that last 'hosts' entry | 17:53 |
| *** weshay_xchat is now known as weshay | 17:53 | |
| mordred | I'm going to try to make a patch for nodepool that does the more righter thing there | 17:53 |
| clarkb | nibalizer: kk | 17:53 |
| nibalizer | thats just an oversight on my part | 17:53 |
| nibalizer | clarkb: you good for me to run this? | 17:54 |
| *** wenlock has quit IRC | 17:55 | |
| clarkb | ya lets get it over with :) | 17:55 |
| mordred | fungi: or maybe we should just all remember that when people hit this iit's the first thing to try | 17:55 |
| nibalizer | this probalby will take a bit | 17:55 |
| nibalizer | since reboots | 17:55 |
| clarkb | yes | 17:55 |
| *** sergmelikyan has joined #openstack-infra | 17:56 | |
| *** bpokorny has quit IRC | 17:56 | |
| *** hashar has quit IRC | 17:57 | |
| clayton | so, thanks to monty for the help with that, and also thanks to whoever realized fractional hours in the nodepool list output might not be optimal :) | 17:57 |
| fungi | heh | 17:57 |
| clarkb | nibalizer: I suppose its good that that failed :/ | 17:58 |
| greghaynes | mordred: or maybe catch and reraise a more helpful error message? | 17:58 |
| clarkb | nibalizer: I am going to manually run that check on 02 | 17:58 |
| nibalizer | clayton: eh.. thats a bit contentious | 17:58 |
| mordred | greghaynes: well, we do that as best we can | 17:58 |
| fungi | clayton: if only i could purchase a clock with tenths and hundredths of an hour instead of minutes and seconds | 17:58 |
| nibalizer | I got really used to fractional hours | 17:58 |
| *** devvesa has quit IRC | 17:58 | |
| mordred | greghaynes: but unfortunately there is no way to differentiate "that's not a valid project" from "your password is wrong" to "you do not have an account on this cloud" | 17:58 |
| clarkb | fractions are easier than base 60 maths | 17:59 |
| nibalizer | clarkb: yikes | 17:59 |
| greghaynes | mordred: totally, I am just thinking "Maybe you did X" | 17:59 |
| fungi | actually i wish i could update the linux system clock to operate on decimal fractions of a day | 17:59 |
| clayton | nibalizer I think the new output is easier to grep though, for example if I want to find all nodes that have been around for over a day | 17:59 |
| clarkb | nibalizer: so on 02 they match I think my test command is just bonkers | 17:59 |
| greghaynes | mordred: it doesnt have to be always correct, but that is an easy way to make sure that is the first thing someone checks | 17:59 |
| clayton | which for us usually means it never got connected to jenkins | 17:59 |
| fungi | clarkb: not entirely true. base 12/60 math was much easier in the days of purely rational mathematics | 18:00 |
| patrickeast | BobBall: yea looks like the ec2 one went down, i just kicked it... should be back up now | 18:00 |
| clarkb | nibalizer: oh I am stupid | 18:00 |
| clarkb | nibalizer: the problem is the filename is in the output | 18:00 |
| clarkb | nibalizer: and they are two different files | 18:00 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add initial setup for magnum in shade https://review.openstack.org/243239 | 18:00 |
| *** shashank_hegde has quit IRC | 18:00 | |
| clarkb | nibalizer: I think I can use cut to fix this real quick | 18:01 |
| nibalizer | clarkb: we need | 18:01 |
| nibalizer | [ "`sha1sum /tmp/elasticsearch-1.7.3.deb | cut -d ' ' -f 1`" = "`cat /tmp/elasticsearch-1.7.3.deb.sha1.txt| cut -d ' ' -f 1 `" ] | 18:01 |
| *** dtantsur is now known as dtantsur|afk | 18:01 | |
| nibalizer | yay hax | 18:01 |
| clarkb | ya that | 18:02 |
| clarkb | nibalizer: so I guess trim the old playbook to remove all the stuff that worked and update the sha1 check | 18:02 |
| *** jistr has quit IRC | 18:03 | |
| nibalizer | hrm... bu thte single quoting already present in the yaml file will probably screw with my single quoting | 18:03 |
| clarkb | sorry for that, I should've realize | 18:03 |
| clarkb | hahaha | 18:03 |
| BobBall | thanks patrickeast - is there a different instance that I should reference instead? or was the other instance internal? | 18:03 |
| *** ilyashak_ has quit IRC | 18:03 | |
| clarkb | nibalizer: the way around that is use the > quoting like curl did | 18:03 |
| patrickeast | BobBall: the one i use primarily sits behind our firewall :( | 18:03 |
| BobBall | understood | 18:03 |
| clarkb | nibalizer: so shell: >\n | 18:03 |
| clarkb | nibalizer: then indent your ne wstring without the outer ''s on the newline | 18:04 |
| patrickeast | BobBall: at some point i should really just figure out why the ui process throws up with http://paste.openstack.org/show/478366/ | 18:04 |
| patrickeast | something in the flask server i think... not sure how to catch it | 18:04 |
| clarkb | nibalizer: one more space indent | 18:04 |
| *** baoli has quit IRC | 18:04 | |
| *** nadya has joined #openstack-infra | 18:05 | |
| clarkb | nibalizer: the [ goes under e in shell | 18:05 |
| nibalizer | kk | 18:05 |
| notmyname | I have an issue with dependencies and global requirements, and I was hoping for some advice | 18:05 |
| clarkb | nibalizer: and remove all the stuff above it | 18:05 |
| BobBall | Yeah, that sucks... | 18:05 |
| clarkb | nibalizer: so we don't reboot again and all that | 18:05 |
| nibalizer | ya | 18:05 |
| nibalizer | good to run? | 18:06 |
| *** baoli has joined #openstack-infra | 18:06 | |
| clarkb | I think so, it will redownload but thats fine | 18:06 |
| notmyname | current dependency in swift/GR is pyeclib==1.0.7. we want to bump this to >=1.1.1. but we can't because swift-side can't land without the GR change and GR can't land since it breaks all the tests that are looking for ==1.0.7 | 18:06 |
| *** alexz has quit IRC | 18:06 | |
| notmyname | we've got open patches for both, and I think that if we can get the patches to land right next to each other it should work | 18:06 |
| greghaynes | crinkle: any tips on debugging http://logs.openstack.org/40/165240/12/check/gate-infra-puppet-apply-bare-precise/0ed3dcc/console.html#_2015-11-09_16_52_59_065 | 18:07 |
| greghaynes | crinkle: seems pretty unrelated to the change... | 18:07 |
| clarkb | ok es is running now | 18:07 |
| *** unicell has joined #openstack-infra | 18:07 | |
| AJaeger | notmyname: can't you ask for >=1.0.7 instead? It will get the latest one | 18:07 |
| clarkb | nibalizer: you will want to reconnect with bigdesk to see that we didn't split brain or anything | 18:07 |
| clarkb | nibalizer: and elasticsearch head should show all the indexes | 18:07 |
| AJaeger | so, two changes, first asking for >= 1.0.7, second >= 1.1.1 | 18:07 |
| *** haypo has joined #openstack-infra | 18:07 | |
| nibalizer | oh haha | 18:07 |
| AJaeger | notmyname: you need overlapping ranges... | 18:08 |
| nibalizer | I'm sitting here spamming reconnect and nothing is happening... well my ssh tunnel died because the box rebooted | 18:08 |
| nibalizer | obviously | 18:08 |
| crinkle | greghaynes: don't worry about that, it ended in 'exit 0' so it's okay | 18:08 |
| crinkle | greghaynes: you have to worry about this one http://logs.openstack.org/40/165240/12/check/gate-infra-puppet-apply-bare-precise/0ed3dcc/console.html#_2015-11-09_16_51_12_624 | 18:08 |
| greghaynes | crinkle: hrm, the test failed, so something else I guess | 18:08 |
| clarkb | nibalizer: ya you need to redo ssh | 18:08 |
| greghaynes | crinkle: aha! | 18:08 |
| notmyname | AJaeger: there's some versions between 1.0.7 and 1.1.1 that I don't want. I'm wondering if there's any way to avoid the extra code churn | 18:09 |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: add stable/liberty branch to gerritbot for project openstack-chef https://review.openstack.org/243245 | 18:09 |
| clarkb | nibalizer: the logs (/var/log/elasticsearch/elasticsearch.log) seem good so far, queries are fialing because we are in state RECOVERING | 18:09 |
| *** sdake has quit IRC | 18:09 | |
| openstackgerrit | Merged openstack-infra/nodepool: Add dib image command tests https://review.openstack.org/229652 | 18:09 |
| clarkb | nibalizer: but it seems to see the nodes | 18:09 |
| clarkb | nibalizer: also we may need to update bigdesk and elasticsearch head to latest versions now to support newer api | 18:10 |
| nibalizer | I'm not getting my dashboards to reconnect :( | 18:10 |
| nibalizer | but curl localhost:9200/_stats worked so thats good | 18:10 |
| clarkb | nibalizer: curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' is about what I expect | 18:10 |
| crinkle | greghaynes: the template from https://review.openstack.org/#/c/165240/12/modules/openstack_project/manifests/proposal_slave.pp isn't in scope to retrieve those variables, you should probably just add that file resource to wheel_mirror_slave.pp instead | 18:10 |
| clarkb | nibalizer: I think need to update versions of those tools to get them to work? | 18:11 |
| nibalizer | ok | 18:11 |
| clarkb | nibalizer: anyways I am using the above curl and it says it is initializing shards and all 6 data nodes are present | 18:11 |
| *** baoli has quit IRC | 18:11 | |
| mordred | notmyname: it might be ugly ... but you could do >=1.0.7,!=1.0.8,!=1.0.9 etc. | 18:11 |
| *** baoli has joined #openstack-infra | 18:11 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Skip long-running jobs for release note changes https://review.openstack.org/243246 | 18:12 |
| *** kozhukalov_ has quit IRC | 18:12 | |
| AJaeger | notmyname: in the gates we install the latest version anyway, so make this a small window between those two patches... | 18:12 |
| fungi | notmyname: the jobs are engineered to require that we don't require atomically merging changes to two git repositories. it's preferred that the way forward be a graceful non-atomic series of changes instead (so that they can be paused for indefinite periods of time within that sequence and not leave things broken) | 18:12 |
| *** wenlock has joined #openstack-infra | 18:12 | |
| openstackgerrit | Mark T. Voelker proposed openstack-infra/irc-meetings: Change DefCore from 15:00 to 16:30 UTC https://review.openstack.org/241695 | 18:12 |
| *** Sukhdev has quit IRC | 18:14 | |
| greghaynes | crinkle: So, there is something I am missing about templates I thinkg... | 18:14 |
| greghaynes | crinkle: Do I need to be explicitly passing in context? | 18:14 |
| clarkb | nibalizer: oh newer es may have proper CORS stuff now | 18:14 |
| clarkb | nibalizer: so that explains why bigdesk and head aren't working :( | 18:15 |
| greghaynes | haha | 18:15 |
| nibalizer | 'yay' | 18:15 |
| fungi | which means that you need to properly co-host bigdesk and head on the server now? | 18:15 |
| *** sdake has joined #openstack-infra | 18:15 | |
| *** ZZelle_ has joined #openstack-infra | 18:15 | |
| greghaynes | You can punch headers in if its behind apache or somesuch | 18:15 |
| fungi | if so, that's unfortunate | 18:15 |
| crinkle | greghaynes: the template gets context from the class in which the template() function is called | 18:16 |
| greghaynes | or usualy you can configure allowed-hosts | 18:16 |
| clarkb | nibalizer: once the initializing shards falls to 0 I think that means all our primary shards are recovered | 18:16 |
| fungi | oh, manually tweaking allowed-hosts is probably an acceptable nuisance | 18:16 |
| clarkb | nibalizer: at that point cluster won't do anything until we run the next step and reenable allocation | 18:16 |
| clarkb | nibalizer: so once we get there I think we reenable allocation | 18:16 |
| clarkb | greghaynes: there is no apache | 18:16 |
| nibalizer | kk | 18:16 |
| clarkb | greghaynes: this is in es itself | 18:16 |
| greghaynes | clarkb: http.cors.allow-origin | 18:16 |
| notmyname | AJaeger: mordred: fungi: unfortunately, existing tests won't pass with newest pyeclib because newest pyeclib has dropped bundled jerasure (and replaced it with it's own implementation). that's a very good thing and what we want, but it means that upping the GR version will cause stuff to break unless the corresponding swift patch lands too | 18:17 |
| greghaynes | https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html | 18:17 |
| crinkle | greghaynes: you called template() in openstack::proposal_slave which only knows about variables in that class and in the proposal.slave.o.o node | 18:17 |
| greghaynes | clarkb: not sure if that applies to your version... | 18:17 |
| *** sridhar_ram1 has joined #openstack-infra | 18:17 | |
| clarkb | thats 2.0 but I am sure 1.7.3 is similar | 18:17 |
| openstackgerrit | Merged openstack-infra/nodepool: Add node command tests https://review.openstack.org/229653 | 18:17 |
| *** sridhar_ram has quit IRC | 18:17 | |
| clarkb | I am happy to use curl for now | 18:18 |
| greghaynes | crinkle: allow me to try and grok.... | 18:18 |
| mordred | notmyname: hrm | 18:18 |
| clarkb | nibalizer: ok run that status command and you will see cluster is yellow and basically holding still | 18:18 |
| notmyname | mordred: indeed. | 18:18 |
| nibalizer | clarkb: intializing shards is 0 | 18:18 |
| clarkb | notmyname: initializing shards is 0 ya | 18:18 |
| clarkb | nibalizer: er ^ | 18:18 |
| clarkb | nibalizer: so next step is to reenable shard allocation so that those 100 shards get initialized as secondary shards | 18:19 |
| greghaynes | crinkle: ah, yes | 18:19 |
| nibalizer | kk | 18:19 |
| greghaynes | crinkle: I added the vars to the wrong class | 18:19 |
| crinkle | greghaynes: :) | 18:19 |
| mordred | notmyname: my brain is trying to compute a solution | 18:20 |
| notmyname | mordred: so we could respin distro packages that also include the dependency. and update the library to support both. but that seems like a large amount of work simply to work around our own CI system | 18:20 |
| nibalizer | clarkb: looking good so far | 18:20 |
| *** Ryan_Lane has joined #openstack-infra | 18:20 | |
| Ryan_Lane | howdy. is openstack still doing its CLAs in gerrit? | 18:21 |
| *** Swami has joined #openstack-infra | 18:21 | |
| clarkb | Ryan_Lane: we are (though recent board meeting seemed to give go ahead to DCO for individual contributors) | 18:21 |
| clarkb | nibalizer: ya and logstash works if yo uexpand the time range out from 15 minutes to an hour | 18:21 |
| Ryan_Lane | what's DC0? | 18:21 |
| Ryan_Lane | DCO | 18:21 |
| Ryan_Lane | I'm trying to find a reasonable way of hosting a CLA | 18:22 |
| notmyname | mordred: also, I haven't yet considered how this is going to affect any stable branch testing. I kinda dodge that by staying in swift-land since our patches work with the dependencies they specify | 18:22 |
| clarkb | Ryan_Lane: https://wiki.openstack.org/wiki/OpenStackAndItsCLA#The_Proposal | 18:22 |
| Ryan_Lane | thought I'd ask for advice here :) | 18:22 |
| fungi | notmyname: swift can't support both the old and new jerasure versions? it sounds like you're saying they need to upgrade in lock-step? | 18:22 |
| dims | Ryan_Lane https://wiki.openstack.org/wiki/OpenStackAndItsCLA#The_Developer_Certificate_of_Origin | 18:22 |
| dims | oops clarkb already posted | 18:22 |
| *** wenlock has quit IRC | 18:22 | |
| Ryan_Lane | :) | 18:22 |
| *** wenlock1 has joined #openstack-infra | 18:22 | |
| *** thedodd has quit IRC | 18:23 | |
| clarkb | nibalizer: so ya now we wait for those shards to init and cluster to go green | 18:23 |
| *** thedodd has joined #openstack-infra | 18:23 | |
| clarkb | nibalizer: once that happens we can run the playbook to turn stuff back on the workers | 18:23 |
| clarkb | nibalizer: then we can get my puppet change merged and reenable puppet everywhere and finally figure out how we want ot do CORS | 18:23 |
| fungi | Ryan_Lane: also http://docs.openstack.org/infra/manual/developers.html#using-signed-off-by | 18:24 |
| *** davideagnello has joined #openstack-infra | 18:24 | |
| notmyname | fungi: close. the truth is somewhat more complicated | 18:24 |
| *** asselin__ has quit IRC | 18:24 | |
| Ryan_Lane | hm. interesting. | 18:24 |
| mordred | notmyname: I agree in general about it being a lot of work and whatnot for your case - the system in general is there to handle the more complex case (and more common one, tbh) where 2 different openstack projects that commonly get installed on the same machine want to upgrade to a new version of a lib on a different cadence | 18:25 |
| *** baoli has quit IRC | 18:25 | |
| mordred | which means there needs to be a somewhat reasonable way to be in a middle-ground position where each of the combos works | 18:25 |
| Ryan_Lane | fungi: if openstack continued to use a CLA and didn't want to host in gerrit, what would you consider? | 18:26 |
| mordred | notmyname: otoh - it does prevent lockstep upgrades within a single project | 18:26 |
| Ryan_Lane | clarkb: ^^ | 18:26 |
| notmyname | we don't actually directly use jerasure. current master uses pyeclib 1.0.7 which bundles jerasure. current swift unit tests use a config that requests a jerasure backend for pyeclib. pyeclib 1.1.1 has removed the jerasure bundling and provides it's own simple ec implementation. that's what we're wanting for testing. | 18:26 |
| Ryan_Lane | DCO is cool and hopefully I can use that instead. it looks a lot easier to deal with | 18:26 |
| *** baoli has joined #openstack-infra | 18:27 | |
| mordred | Ryan_Lane: I recomend that | 18:27 |
| *** mgrosz has joined #openstack-infra | 18:27 | |
| mordred | Ryan_Lane: CLAs are a nightmare and should be avoided unless you're at lawyer-gunpoint | 18:27 |
| fungi | Ryan_Lane: apache projects use a manual process to verify a cla is in place, but they also only require it of their "contributors" (their equivalent of our core reviewers) | 18:28 |
| notmyname | mordred: yeah. we're the only openstack consumer of pyeclib and the commits proposed to master all work. what's challenging is getting the CI system to approve use using a different library | 18:28 |
| notmyname | *different version of the library | 18:28 |
| *** kozhukalov_ has joined #openstack-infra | 18:28 | |
| *** shashank_hegde has joined #openstack-infra | 18:28 | |
| *** asselin_ has joined #openstack-infra | 18:28 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Skip long-running jobs for release note changes https://review.openstack.org/243246 | 18:28 |
| *** sridhar_ram has joined #openstack-infra | 18:29 | |
| notmyname | which is why I wanted to ask in here if there is any other way beyond doing more respins of dependencies just for the sake of openstack CI | 18:29 |
| notmyname | note that pyeclib is not currently in the openstack namespace (however, we're working on changing that after we get this version bump, actually) | 18:29 |
| mordred | notmyname: other choice might be a conditional in your unittests that try to attach the bundled ec backend and if that's not there try the othe rthing? | 18:29 |
| mordred | notmyname: like, you have the patch to move between the two in swift - can it be conditionaled such that you don'tneed to actuall respin depends? | 18:30 |
| clarkb | nibalizer: it is slowly making the unassigned shards number smaller | 18:30 |
| clarkb | nibalizer: now is probably a good time for breakfast | 18:30 |
| notmyname | mordred: I feel like we're slaves to our own tools instead of the other way around :-/ | 18:30 |
| mordred | notmyname: in this case, yes | 18:31 |
| tsg | notmyname, mordred: as a side-note, (and a lack of my detailed understanding of upper-constraints) but Jenkins seems to be happy if I up global-requirements on master to >=1.1.1 and leave upper-constraints on master at ==1.0.7. | 18:31 |
| mordred | notmyname: but that's because the tools are there to solve a much more complex case and this isa n outlier | 18:31 |
| clarkb | I am trying to figure out if we need to worry about CORS at all because of how we set up the proxying | 18:31 |
| mordred | notmyname: but it is absolutely making this task harder than it needs to be | 18:31 |
| notmyname | mordred: we're in agreement :-) | 18:31 |
| clarkb | basically nothing can hit the api directly so if we tell the api itself that * is ok for CORS then I don't think we are really an less secure | 18:31 |
| clarkb | fungi: ^ that make sense? | 18:31 |
| *** sridhar_ram1 has quit IRC | 18:32 | |
| *** otherwiseguy has quit IRC | 18:32 | |
| notmyname | mordred: btw tsg is the maintainer for pyeclib | 18:32 |
| nibalizer | clarkb: kk imma grab a burrito | 18:32 |
| *** sputnik13 has joined #openstack-infra | 18:32 | |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Support GitHub PR webhooks https://review.openstack.org/163117 | 18:33 |
| *** davideagnello has quit IRC | 18:33 | |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Make the string representation of change transparent https://review.openstack.org/238948 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Fix job hierarchy bug. https://review.openstack.org/192457 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Add 'push' and 'tag' github webhook events. https://review.openstack.org/191207 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Add 'pr-comment' github webhook event https://review.openstack.org/239203 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Configurable SSH access to GitHub https://review.openstack.org/239138 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Assign fake connections to test class dynamically https://review.openstack.org/238820 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Allow using webapp from connections https://review.openstack.org/215642 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Don't pass fake_gerrit to FakeURLOpener https://review.openstack.org/238818 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Pass upstream_root to fake gerrit explicitly https://review.openstack.org/238819 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Support for github commit status https://review.openstack.org/239303 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Generalize event queue handling in tests https://review.openstack.org/238817 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Allow list values in template parameters. https://review.openstack.org/191208 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Add basic Github Zuul Reporter. https://review.openstack.org/191312 | 18:33 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Merge pull requests from github reporter https://review.openstack.org/243250 | 18:33 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets https://review.openstack.org/242363 | 18:33 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add unit tests for subunit2sql.shell.main https://review.openstack.org/242587 | 18:33 |
| mordred | hi tsg ! | 18:33 |
| *** davideagnello has joined #openstack-infra | 18:33 | |
| fungi | clarkb: yep, seems safe | 18:33 |
| *** otherwiseguy has joined #openstack-infra | 18:34 | |
| mordred | tsg: hrm. what if you set >=1.0.7 and upper-constraints to ===1.0.7 ? | 18:34 |
| mordred | tsg: becuase the problem where this fails is swifts _unit_ tests, right? | 18:34 |
| *** pvaneck has joined #openstack-infra | 18:34 | |
| tsg | hi! mordred, https://review.openstack.org/#/c/238771/ | 18:35 |
| tsg | http://logs.openstack.org/71/238771/3/check/gate-tempest-dsvm-full/00b928d/logs/devstacklog.txt.gz | 18:35 |
| tsg | + swift-init --run-dir=/opt/stack/data/swift/run all stop" | 18:35 |
| tsg | (...) | 18:35 |
| tsg | pkg_resources.DistributionNotFound: The 'PyECLib==1.0.7' distribution was not found and is required by swift | 18:35 |
| notmyname | right. there aren't any functional tests in openstack's CI that test against an EC storage policy (there are in 3rd party CI) | 18:35 |
| mordred | notmyname: so - upper-constraints only affects devstack jobs at the moment | 18:36 |
| *** mgrosz has quit IRC | 18:36 | |
| fungi | notmyname: tsg: so to rephrase, there will be a (hopefully extremely brief) period of time where swift master can't work with the jerasure declared in our list of global requirements... an obvious solution is that for that period of time swift simply shouldn't be tested against global requirements (or vice versa) since they're mutually incompatible | 18:36 |
| mordred | which means you should be able to land >=1.0.7, ===1.0.7 in g-r without anything breaking | 18:36 |
| *** sridhar_ram has quit IRC | 18:37 | |
| fungi | i don't see that as fighting against our tools, it's our tools enforcing the things we ask them to | 18:37 |
| *** mgrosz has joined #openstack-infra | 18:37 | |
| fungi | if we don't want them to enforce those things, then we can of course turn them off | 18:37 |
| *** ilyashakhat_mobi has joined #openstack-infra | 18:37 | |
| greghaynes | clarkb: re: cors - its really a matter of whether we would potentially store anything 'secure' in ES | 18:37 |
| tsg | mordred: that's true - but won't CI pick up 1.1.1? | 18:37 |
| greghaynes | clarkb: which we dont | 18:37 |
| greghaynes | clarkb: so * is fine | 18:37 |
| mordred | which then means swift could change its requirements.txt to be >=1.0.7 - which it could do in the patch that changes the unit tests | 18:37 |
| clarkb | greghaynes: sort of | 18:37 |
| clarkb | greghaynes: we don't want arbitrary things to store anything but they can't because of the proxy | 18:38 |
| mordred | tsg: CI will pick up 1.1.1 in unittests not in devstack | 18:38 |
| mordred | which is where g-r changes are tested | 18:38 |
| mordred | so you'll be able to land the g-r change | 18:38 |
| clarkb | greghaynes: so ya I think * is ok for when we talk directly to the api via ssh port forward | 18:38 |
| *** daemontool has quit IRC | 18:38 | |
| tsg | mordred: got it - I see that it looks at upper-constraints | 18:38 |
| greghaynes | clarkb: Yea, what the CORS is protecting is whether someone would have special access circumvented by a third party site | 18:38 |
| mordred | then once you do, the g-r system will allow you to land the swift change to swift's requirements | 18:38 |
| clarkb | greghaynes: right and I think we already have stronger controls in place with the proxy? | 18:38 |
| greghaynes | clarkb: Yep | 18:39 |
| *** trown is now known as trown|lunch | 18:39 | |
| mordred | and then once you've done that, you can land an upper-constraints patch to set it to >=1.1.1 | 18:39 |
| *** daemontool has joined #openstack-infra | 18:39 | |
| mordred | tsg: can you try that on your g-r patch and see if my theory holds? | 18:39 |
| greghaynes | clarkb: also, not doing * would mean we prevent people from hosting their own kibana I think | 18:39 |
| clarkb | greghaynes: I don't know that that is the case since they would talk to logstash.o.o apache proxy anyways | 18:40 |
| tsg | mordred: sure, to recap - 1) submit >= 1.0.7 for global-requirements, leave upper-constraints at ==1.0.7, 2) up swift requirement to >=1.1.1, 3) up upper-constraints.txt on master to >=1.1.1 | 18:40 |
| clarkb | but maybe it does, I haven't looked at the headres closely yet | 18:40 |
| clarkb | but ya I think proxy controls are actually what we need/want and we don't care about CORS | 18:40 |
| mordred | tsg: cloes | 18:40 |
| mordred | tsg: 2 should be "up swift requirement to >= 1.0.7" | 18:41 |
| greghaynes | clarkb: agreed | 18:41 |
| mordred | since that will, at the moment, install 1.1.1 since it's always going to get the latest | 18:41 |
| tsg | mordred: yeah - makes sense | 18:41 |
| notmyname | tsg: mordred: and then step 4 is to up swift and g-r both to be >=1.1.1? | 18:41 |
| mordred | yup | 18:41 |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:41 | |
| mordred | well, step 3 can update g-r and upper-constrints to 1.1.1 | 18:41 |
| lifeless | o/ | 18:42 |
| notmyname | ok | 18:42 |
| mordred | oh good | 18:42 |
| mordred | lifeless is here | 18:42 |
| mordred | lifeless: please vet my suggestion above | 18:42 |
| *** kushal has quit IRC | 18:42 | |
| *** bardia has joined #openstack-infra | 18:42 | |
| Ryan_Lane | fungi: yeah, I have a feeling I'm going to need to require every contributor to sign one | 18:42 |
| lifeless | mordred: nearly | 18:42 |
| lifeless | 1) >=1.0.7 in global-requirements | 18:42 |
| Ryan_Lane | which is why I'm trying to avoid a manual process | 18:43 |
| lifeless | 2) >=1.0.7 in swift | 18:43 |
| lifeless | 3) up to >=1.1.1 in g-r | 18:43 |
| *** sdake_ has joined #openstack-infra | 18:43 | |
| *** Hunner has quit IRC | 18:43 | |
| lifeless | 2's step has to match g-r because of the consistency checks | 18:43 |
| *** thiagop has quit IRC | 18:43 | |
| lifeless | and 4) swift to >= 1.1.1 | 18:43 |
| lifeless | so yes, mordred you have it | 18:44 |
| mordred | yay! | 18:44 |
| * mordred was usefulish | 18:44 | |
| tsg | mordred, lifeless: thx! .. also update u-r in (3) to ==1.1.1? | 18:44 |
| lifeless | [and this is why pins are bad] | 18:44 |
| *** sabeen3 has quit IRC | 18:44 | |
| notmyname | lifeless: and upper-constraints? | 18:44 |
| *** signed8bit_ZZZzz is now known as signed8bit | 18:44 | |
| tsg | or can u-r be at >=1.1.1? | 18:44 |
| lifeless | notmyname: it moves at 3 | 18:44 |
| lifeless | notmyname: if we move it at 1, swift's dependency cross-check will error [btw this is why most openstack projects disabled that check] | 18:44 |
| lifeless | tsg: constraints are always ===, since they are documenting known-good's | 18:45 |
| *** sdake has quit IRC | 18:45 | |
| *** achanda has joined #openstack-infra | 18:45 | |
| tsg | lifeless: makes sense :) | 18:45 |
| lifeless | tsg: that way we have no variation in CI other than the code being tested, | 18:45 |
| lifeless | s/,/./ | 18:45 |
| *** Hunner has joined #openstack-infra | 18:45 | |
| *** Hunner has quit IRC | 18:45 | |
| *** Hunner has joined #openstack-infra | 18:45 | |
| *** kozhukalov_ has quit IRC | 18:45 | |
| *** amuller_afk is now known as amuller | 18:45 | |
| *** achanda has quit IRC | 18:45 | |
| mordred | s/\./ :)/ | 18:46 |
| *** yfried has joined #openstack-infra | 18:46 | |
| *** thiagop has joined #openstack-infra | 18:47 | |
| notmyname | lifeless: mordred: thanks | 18:47 |
| *** tonytan4ever has quit IRC | 18:48 | |
| *** tiswanso has joined #openstack-infra | 18:49 | |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror https://review.openstack.org/165240 | 18:50 |
| *** kozhukalov_ has joined #openstack-infra | 18:50 | |
| *** tdurakov has joined #openstack-infra | 18:50 | |
| lifeless | notmyname: anytime | 18:50 |
| shardy | Hey folks, we're trying to add CI for some new stable/liberty tripleo branches | 18:51 |
| shardy | I think we have the project-config stuff OK now, any reviews much appreciated: | 18:51 |
| shardy | https://review.openstack.org/#/c/240938/ | 18:51 |
| notmyname | lifeless: even before 8am! ;-) | 18:52 |
| lifeless | notmyname: even before 8am :) | 18:52 |
| greghaynes | hrm, on https://review.openstack.org/#/c/233852/2 gerrit reports merge failure that zuul doesnt know about - does that mean jgit and python-git disagree on mergability? | 18:52 |
| *** ilyashakhat_mobi has quit IRC | 18:53 | |
| *** tdurakov has quit IRC | 18:53 | |
| clarkb | greghaynes: is the comment from the gerrit bird? | 18:53 |
| clarkb | greghaynes: if sothen yes | 18:53 |
| greghaynes | Yep, it has the birdie | 18:53 |
| clarkb | woo we need to restart cluster to pick up an http module settings :/ | 18:54 |
| clarkb | might save that for the weekend then | 18:54 |
| openstackgerrit | matthew wagoner proposed openstack-infra/shade: Add new context manager for shade exceptions, cont. again https://review.openstack.org/243221 | 18:54 |
| *** daemontool has quit IRC | 18:54 | |
| *** mgrosz has quit IRC | 18:54 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Explicitly block in nodepool upload commands https://review.openstack.org/233852 | 18:54 |
| fungi | greghaynes: more specifically, it's jgit and cgit | 18:55 |
| greghaynes | fungi: ah, is cgit what we use in zuul? | 18:55 |
| openstackgerrit | Tushar Gohad proposed openstack/requirements: Bump PyECLib version to >=1.0.7 https://review.openstack.org/238771 | 18:55 |
| fungi | greghaynes: the jgit merge conflict resolution is primitive compared to the variety of resolution models cgit tries | 18:55 |
| *** tdurakov has joined #openstack-infra | 18:55 | |
| *** sergmelikyan has quit IRC | 18:55 | |
| *** alex_xu has quit IRC | 18:56 | |
| greghaynes | fun | 18:56 |
| fungi | greghaynes: well, cgit is what's getting called into eventually, yes | 18:56 |
| clarkb | https://www.elastic.co/guide/en/elasticsearch/reference/1.7/modules-http.html is the docs for the version we are running | 18:56 |
| clarkb | basically the same which is nice | 18:56 |
| greghaynes | clarkb: sweet | 18:56 |
| fungi | greghaynes: we intentionally have zuul neuter/cripple the resolution models it lets cgit use in at attempt to make it as simplistic as the jgit gerrit uses, but they still occasionally disagree | 18:57 |
| nibalizer | so enabled = true and allow-origin = * ? | 18:57 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Remove wait_for_threads in test_commands https://review.openstack.org/237219 | 18:57 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image builds using gearman https://review.openstack.org/195175 | 18:57 |
| *** alex_xu has joined #openstack-infra | 18:57 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image uploads using gearman https://review.openstack.org/197722 | 18:57 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Add dib deletion test https://review.openstack.org/215275 | 18:57 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image delete using gearman https://review.openstack.org/198114 | 18:57 |
| *** signed8bit is now known as signed8bit_ZZZzz | 18:57 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Move nodepool config loading into module https://review.openstack.org/231592 | 18:58 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Make secure.conf optional and backwards-compat https://review.openstack.org/232800 | 18:58 |
| Ryan_Lane | thanks for the help everyone. nice to be able to ask you folks for advice :) | 18:58 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Decouple nodepool db and config from builders https://review.openstack.org/222398 | 18:58 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Builders distinguish between failure and exception https://review.openstack.org/242131 | 18:58 |
| fungi | Ryan_Lane: nice to be able to do something for you for a change! | 18:58 |
| Ryan_Lane | :) | 18:59 |
| clarkb | 19 shards to go, probably gonna take the longest | 18:59 |
| nibalizer | clarkb: ya its been at 181/19 for the past 10 minutes | 18:59 |
| *** e0ne has joined #openstack-infra | 19:00 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Builds and publishes Compute API Guide https://review.openstack.org/231000 | 19:00 |
| clarkb | nibalizer: ya Ithink enabled = true and allow-origin = * | 19:00 |
| *** signed8bit_ZZZzz is now known as signed8bit | 19:03 | |
| *** asselin_ has quit IRC | 19:03 | |
| *** austin81 has joined #openstack-infra | 19:03 | |
| AJaeger | clarkb, mordred,fungi, could I get another +2 review on https://review.openstack.org/#/c/242975/ to limit running of releasenotes jobs, please? This will unbreak stable glance branches... | 19:04 |
| clarkb | nibalizer: I am going to clean up some of these 0015.X.Y indexes which are creating errors in the logs (no timestamp found) | 19:04 |
| *** stevemar_ has joined #openstack-infra | 19:05 | |
| fungi | AJaeger: is it that gate-{name}-tox-releasenotes can only ever run on master, or simply that it's implemented for mitaka and later? | 19:06 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/project-config: Add create ACL for Octavia https://review.openstack.org/243264 | 19:06 |
| clarkb | nibalizer: I have previously closed a couple of the "bad" indexes so that we retain the data on how possibly they got created | 19:06 |
| clarkb | nibalizer: if you run curl -XGET 'http://localhost:9200/_cat/indices' you will see them all | 19:07 |
| fungi | AJaeger: based on the commit message, it seems like there's an assumption that we'll get post-merge testing of release notes collection for stable branches indirectly by checking master branch changes | 19:07 |
| *** sridhar_ram has joined #openstack-infra | 19:07 | |
| AJaeger | fungi: my understanding is that it really will only run on master - and collect information from other branches | 19:07 |
| fungi | AJaeger: so it is in theory possible to merge a change to stable/liberty which will then leave this job broken on all master branch changes? | 19:07 |
| AJaeger | fungi: mmh - yeah ;( | 19:08 |
| fungi | AJaeger: or does it sanely ignore failures for stable branch commits to prevent that | 19:08 |
| fungi | ? | 19:08 |
| AJaeger | fungi: seems we should ask dhellmann for this | 19:08 |
| AJaeger | fungi: very good question! | 19:09 |
| *** e0ne has quit IRC | 19:10 | |
| fungi | i summarized my concern in a review comment. we can continue there | 19:10 |
| AJaeger | fungi: thanks for adding this as comment, let's see what dhellmann answers! | 19:10 |
| *** tdurakov has quit IRC | 19:10 | |
| *** e0ne has joined #openstack-infra | 19:10 | |
| *** tnarg has quit IRC | 19:10 | |
| tsg | lifeless: https://review.openstack.org/#/c/238771/ | 19:10 |
| *** someara2 has joined #openstack-infra | 19:11 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: add release notes jobs for keystone https://review.openstack.org/242972 | 19:12 |
| stevemar_ | AJaeger: should be all fixed up now ^ | 19:12 |
| *** Sree has quit IRC | 19:13 | |
| *** jamesmcarthur has joined #openstack-infra | 19:13 | |
| lifeless | tsg: +2A hammer applied | 19:13 |
| *** sabeen1 has joined #openstack-infra | 19:13 | |
| *** chrall__ has quit IRC | 19:13 | |
| clarkb | nibalizer: curl -XGET 'http://localhost:9200/logstash-2015.11.06/_recovery?pretty&human' shows the replica shard for shard id 1 is in progress | 19:13 |
| AJaeger | stevemar_: cool | 19:14 |
| clarkb | nibalizer: and if you run it a couple times the percentages get bigger so it is in progress | 19:14 |
| nibalizer | clarkb: 'human' | 19:14 |
| *** chrall__ has joined #openstack-infra | 19:14 | |
| tsg | lifeless: gracias! | 19:14 |
| clarkb | nibalizer: human readable | 19:14 |
| clarkb | nibalizer: don't worry the machines don't think they are one of us yet | 19:15 |
| *** sdake_ is now known as sdake | 19:15 | |
| nibalizer | ya my point is thats quite a bit json blob so 'human' isn't quite right | 19:15 |
| nibalizer | much less readable than the table thing that comes out of indicies | 19:15 |
| clarkb | haha ya I dunno its from their docs | 19:15 |
| clarkb | https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-recovery.html | 19:15 |
| nibalizer | mostly just grumping don't pay attention to me | 19:15 |
| clarkb | maybe thats a thing in 2.0 and noop here | 19:15 |
| *** otter768 has joined #openstack-infra | 19:16 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: add stable/liberty branch to gerritbot for project openstack-chef https://review.openstack.org/243245 | 19:17 |
| tsg | lifeless: to confirm - in step (2) after we update swift requirements to 1.0.7 - do we expect devstack in CI to still use 1.0.7, correct? essentially, any changes to swift related to 1.1.1 update should be postponed until the last step (swift update to >=1.1.1) - correct? | 19:17 |
| *** dprince has quit IRC | 19:18 | |
| *** electrofelix has quit IRC | 19:18 | |
| lifeless | tsg: yes (still 1.0.7), no( you should make sure swift *works* with 1.1.1 before we can bump the version to 1.1.1 in g-r) | 19:18 |
| tsg | lifeless: good, so I will split my existing 1.1.1 patch for swift into step (2) and step (4) | 19:18 |
| *** ociuhandu has quit IRC | 19:19 | |
| *** sdake has quit IRC | 19:20 | |
| *** otter768 has quit IRC | 19:21 | |
| *** sdake has joined #openstack-infra | 19:21 | |
| *** smarcet has joined #openstack-infra | 19:21 | |
| *** trown|lunch is now known as trown | 19:22 | |
| *** signed8bit has quit IRC | 19:23 | |
| *** sputnik13 has quit IRC | 19:25 | |
| *** eliqiao1 has joined #openstack-infra | 19:26 | |
| *** sabeen2 has joined #openstack-infra | 19:26 | |
| *** e0ne has quit IRC | 19:27 | |
| *** daemontool has joined #openstack-infra | 19:29 | |
| *** sabeen1 has quit IRC | 19:29 | |
| *** sputnik13 has joined #openstack-infra | 19:30 | |
| *** asselin_ has joined #openstack-infra | 19:32 | |
| *** asselin__ has joined #openstack-infra | 19:32 | |
| clarkb | nibalizer: I am trying to decide if we should fix CORS when we upgrade to trusty | 19:34 |
| clarkb | nibalizer: but I think upgrade to trusty will be rolling upgrade one host at a time which likely isn't enough to fix CORS since that requires full restart | 19:34 |
| *** asselin__ has quit IRC | 19:34 | |
| nibalizer | not having es-head does kinda bite | 19:34 |
| clarkb | ya | 19:35 |
| nibalizer | especially if one of the goals is to get more people up to speed | 19:35 |
| nibalizer | I've had the benefit of doing ES at a couple old jobs so I can limp along but it is really nice | 19:35 |
| clarkb | so probably best to just update config and eat another cluster restart once we are happy with the state of things (that way we can bundle up any other config changes if we find new fun) | 19:35 |
| *** bpokorny has joined #openstack-infra | 19:35 | |
| *** ihrachys has joined #openstack-infra | 19:35 | |
| *** tonytan4ever has joined #openstack-infra | 19:35 | |
| *** sdake has quit IRC | 19:35 | |
| nibalizer | what about just adding the plugin and serving it to the world? is there anything in es-head that is bad? | 19:35 |
| nibalizer | well now that I think about it yes yes there is | 19:36 |
| clarkb | yes, delete | 19:36 |
| nibalizer | like 'delete stuff' | 19:36 |
| *** August1914 has quit IRC | 19:36 | |
| *** sdake has joined #openstack-infra | 19:36 | |
| nibalizer | thats a good point, with a new version this might not be the only thing we need to change config-wise | 19:36 |
| nibalizer | so giving some time for those issues to bubble up is a good idea | 19:36 |
| clarkb | yup | 19:37 |
| *** packet has joined #openstack-infra | 19:37 | |
| *** packet has quit IRC | 19:37 | |
| nibalizer | so index 3 has been at 74.8 for a long time | 19:38 |
| nibalizer | if i'm reading this right | 19:38 |
| clarkb | I think that means its part of the uninitialized set | 19:39 |
| clarkb | ES appears to scan all the indexes to figure out what work needs to be done so it knows its 74% done to start | 19:39 |
| clarkb | but then it starts work elsewhere | 19:39 |
| clarkb | now it seems to be moving | 19:39 |
| nibalizer | ah yes | 19:40 |
| *** Piet has quit IRC | 19:40 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 19:41 | |
| openstackgerrit | Merged openstack/diskimage-builder: Fix diskimage-builder image size https://review.openstack.org/242273 | 19:41 |
| *** nadya has quit IRC | 19:43 | |
| *** tdurakov has joined #openstack-infra | 19:43 | |
| *** angdraug has joined #openstack-infra | 19:44 | |
| *** annegentle has quit IRC | 19:45 | |
| *** ilyashakhat_mobi has quit IRC | 19:46 | |
| *** flepied has quit IRC | 19:46 | |
| dhellmann | fungi , AJaeger : regarding reno on stable branches, the plan was never to run that build on the stable branch itself. | 19:49 |
| dhellmann | fungi , AJaeger : after this initial setup period, we should only be adding notes based on backports, so we shouldn't have a high risk of breaking master | 19:50 |
| *** hashar has joined #openstack-infra | 19:50 | |
| *** dprince has joined #openstack-infra | 19:51 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/gear: inline helps for interactive commands https://review.openstack.org/66591 | 19:52 |
| AJaeger | dhellmann: so, are you asking fungi to approve - and consider the risk "accpetable"? | 19:52 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Merge pull requests from github reporter https://review.openstack.org/243250 | 19:52 |
| openstackgerrit | Jan Hruban proposed openstack-infra/zuul: (WIP) Support for github commit status https://review.openstack.org/239303 | 19:52 |
| dhellmann | AJaeger, fungi : yes, for now? I'm open to discussing ways to mitigate the risk, too, but let's get unjammed. | 19:52 |
| *** someara2 has quit IRC | 19:53 | |
| AJaeger | dhellmann, fungi: In this context, is https://review.openstack.org/242976 and over-optimization - or something we should do? | 19:54 |
| dhellmann | AJaeger : I'm comfortable with that optimization, now that I understand how the post job is configured | 19:55 |
| AJaeger | dhellmann: post jobs will be run for all changes and on all branches. I might need to make the publish job more robust to ignore branches (separate step) | 19:55 |
| dhellmann | AJaeger : SURE | 19:56 |
| dhellmann | oops, sorry | 19:56 |
| *** sarob_ has joined #openstack-infra | 19:56 | |
| *** someara2 has joined #openstack-infra | 19:56 | |
| *** achanda has joined #openstack-infra | 19:56 | |
| *** chrall__ has quit IRC | 19:58 | |
| *** wenlock has joined #openstack-infra | 19:59 | |
| *** baoli has quit IRC | 19:59 | |
| *** nadya has joined #openstack-infra | 19:59 | |
| fungi | AJaeger: remember that post jobs don't have branch awareness in zuul | 19:59 |
| *** chrall__ has joined #openstack-infra | 19:59 | |
| fungi | AJaeger: so you have to do some magic in the job itself to try and figure out if this is a change for master or a stable branch and then noop appropriately | 20:00 |
| dhellmann | AJaeger , fungi : considering testing, we probably need a release notes scan page set without a branch, to test notes on the "current" branch for any given patch coming in. I suppose we could then backport the reno job definition in each project and have a check job but no post job | 20:00 |
| nibalizer | clarkb: fyi i have an appointment at 1:30 so I will be leaving this to you before too long here | 20:00 |
| AJaeger | fungi: yeah, currently trying to find a good way for doing this.... | 20:01 |
| clarkb | nibalizer: ok | 20:01 |
| nibalizer | not like, right now | 20:01 |
| clarkb | things are moving along and really the last step is to turn on the workers | 20:01 |
| *** cfarquhar has quit IRC | 20:01 | |
| fungi | AJaeger: or really s/change/changeish/ (it's an arbitrary git ref from zuul's perspective) | 20:01 |
| *** matrohon has joined #openstack-infra | 20:01 | |
| dhellmann | fungi , AJaeger : how do we do this for the normal developer doc builds? | 20:02 |
| dhellmann | that is, how do we ensure the post jobs only run on master? | 20:02 |
| *** sputnik13 has quit IRC | 20:02 | |
| fungi | dhellmann: we don't. as i said, we ensure that post jobs check what branch the ref appears in, and if it's only in a non-master branch then we have the job do nothing successfully and exit | 20:03 |
| *** wenlock has quit IRC | 20:03 | |
| AJaeger | fungi: I'll see what I can reuse from jenkins/scripts/run-docs.sh... | 20:03 |
| clarkb | 8 shards to go, so close | 20:03 |
| fungi | dhellmann: basically, zuul is what decides what jobs run, and zuul doesn't have information on what branch a ref merged to when it gets the ref-updated event from gerrit | 20:04 |
| dhellmann | fungi : ah, ok, so I guess we'll need to do what AJaeger just said. I didn't realize that's how those worked, I thought we'd be able to control for branch in zuul. | 20:04 |
| *** sputnik13 has joined #openstack-infra | 20:04 | |
| fungi | dhellmann: tag-based pipelines also rely on ref-updated, and so are similarly non-branch-aware. we do similar workarounds on jobs which run in those pipelines and need to do certain things depending on the branch(es) in which a tag's commit appears | 20:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/grafyaml: Refactor Panel class https://review.openstack.org/241711 | 20:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/grafyaml: Additional fields for singlestat panel https://review.openstack.org/241678 | 20:06 |
| dhellmann | fungi : this explains why the doc build isn't a simple call to tox. | 20:06 |
| pabelanger | timrc: good eye. I've updated the regex to patch on 10's now. | 20:06 |
| *** jamesmcarthur has quit IRC | 20:06 | |
| *** cfarquhar has joined #openstack-infra | 20:07 | |
| *** cfarquhar has quit IRC | 20:07 | |
| *** cfarquhar has joined #openstack-infra | 20:07 | |
| *** asselin_ has quit IRC | 20:07 | |
| clarkb | nibalizer: down to 5, may be done before you have to leave | 20:07 |
| nibalizer | cool | 20:07 |
| *** Sukhdev has joined #openstack-infra | 20:10 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack-infra/project-config: Add doc8 job for sahara https://review.openstack.org/243290 | 20:10 |
| *** asselin_ has joined #openstack-infra | 20:11 | |
| *** hodos has quit IRC | 20:11 | |
| *** hodos has joined #openstack-infra | 20:12 | |
| *** sarob_ has quit IRC | 20:12 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Publish releasenotes https://review.openstack.org/242765 | 20:13 |
| AJaeger | fungi, what do you think of the above? ^ | 20:13 |
| *** annegentle has joined #openstack-infra | 20:13 | |
| dhellmann | fungi , AJaeger : would it make sense to make the reno jobs on master non-voting until we sort some of this out? | 20:13 |
| fungi | dhellmann: i would make them non-voting and only run in check until the finer details are ironed out | 20:14 |
| *** sputnik13 has quit IRC | 20:14 | |
| fungi | or even switch them to the experimental pipeline if their output is not meant to be advisory for reviewers yet | 20:14 |
| jesusaurus | clarkb: hows elasticsearch going? looks mostly done | 20:14 |
| dhellmann | fungi : ack, I'll work on making them non-voting for now | 20:14 |
| AJaeger | fungi: non-voting but still publishing - or disable publishing for now? | 20:14 |
| clarkb | jesusaurus: 3 shards to recover then we should be green and will turn workers back on | 20:14 |
| *** chrall__ has quit IRC | 20:15 | |
| AJaeger | dhellmann: non-voting on master, so enhance my change... | 20:15 |
| clarkb | jesusaurus: only major issue so far is we ran into CORS stuff with elasticsearch head and bigdesk | 20:15 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Filter non-voting jobs from the uncategorized bugs page https://review.openstack.org/242207 | 20:15 |
| clarkb | jesusaurus: looks like will require a config update and cluster retsrat to fix that | 20:15 |
| jesusaurus | hm, ok | 20:15 |
| dhellmann | AJaeger : https://review.openstack.org/#/c/242765? | 20:15 |
| fungi | AJaeger: well, publishing happens in post right? so voting status is irrelevant there | 20:15 |
| nibalizer | clarkb: 2 shards! | 20:15 |
| AJaeger | dhellmann: https://review.openstack.org/#/c/242975/ | 20:15 |
| *** chrall__ has joined #openstack-infra | 20:16 | |
| clarkb | jesusaurus: so will likely wait on that a bit to make sure that we don't need any other config changes first | 20:16 |
| *** ldnunes has quit IRC | 20:16 | |
| *** sputnik13 has joined #openstack-infra | 20:16 | |
| AJaeger | fungi: yes. Just wonder whether it's proper to publish based on non-voting tests. But I think publishing will help us to see that everything works... | 20:16 |
| jesusaurus | clarkb: smart | 20:16 |
| dhellmann | AJaeger : the more I think about it, the more I'm convinced fungi is right that we should run the check job on stable/liberty, too | 20:16 |
| dhellmann | but we actually need a thing in each project to make the test do something useful | 20:17 |
| AJaeger | so, non-voting on master and liberty. | 20:17 |
| AJaeger | Let me update 242975, ok? | 20:17 |
| *** mrmartin has quit IRC | 20:17 | |
| dhellmann | AJaeger : sure | 20:17 |
| dhellmann | AJaeger : yes, thanks | 20:17 |
| fungi | AJaeger: minor concern after skimming 242765, you're adding a job which relies on a slave script, and starting to run it right away even though the job runs on nodepool-managed workers which won't have that script on them until the next image update | 20:18 |
| *** sputnik13 has quit IRC | 20:19 | |
| *** tdurakov has quit IRC | 20:19 | |
| *** andersonvom has quit IRC | 20:20 | |
| *** sputnik13 has joined #openstack-infra | 20:20 | |
| *** maiteb has quit IRC | 20:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Run releasenotes jobs only on liberty and newer https://review.openstack.org/242975 | 20:21 |
| AJaeger | fungi: so, split that change in two parts? | 20:21 |
| *** binbincong has quit IRC | 20:22 | |
| fungi | AJaeger: ideally yes, i would add the script and job in one change, then start running the job in a second change (this also gives it time to appear on the jenkins masters, though we don't normally worry as much about that race) | 20:22 |
| *** binbincong has joined #openstack-infra | 20:22 | |
| AJaeger | fungi: ok, will do... | 20:22 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Publish releasenotes: Add infrastructre https://review.openstack.org/242765 | 20:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Publish releasenotes: Run job https://review.openstack.org/243299 | 20:26 |
| AJaeger | dhellmann, fungi: Done ^ | 20:26 |
| *** Sukhdev has quit IRC | 20:26 | |
| *** mestery has quit IRC | 20:27 | |
| clarkb | nibalizer: down to 1 now | 20:28 |
| nibalizer | woo | 20:29 |
| sfinucan | Can anyone review a doc series for pbr for me, please? | 20:29 |
| sfinucan | https://review.openstack.org/#/q/status:open+project:openstack-dev/pbr+branch:master+topic:bug/1509376,n,z | 20:29 |
| sfinucan | The (broken) gate was -1ing it for over a week, but it | 20:29 |
| mordred | infra-root: it's time to cut a new shade. I do not expect any issues with nodepool given that we have testing and whatnot, nor have we made any changes that should have any effects - but | 20:29 |
| sfinucan | *it's finally fixed :) | 20:29 |
| *** sputnik13 has quit IRC | 20:31 | |
| *** chrall__ has quit IRC | 20:32 | |
| *** abregman has joined #openstack-infra | 20:32 | |
| *** chrall__ has joined #openstack-infra | 20:32 | |
| *** sputnik13 has joined #openstack-infra | 20:32 | |
| clarkb | nibalizer: we are green | 20:33 |
| clarkb | nibalizer: I say lets run that last playbook to turn workers back on | 20:33 |
| openstackgerrit | devdatta-kulkarni proposed openstack-infra/project-config: add release notes jobs for solum https://review.openstack.org/243301 | 20:33 |
| *** devkulkarni has joined #openstack-infra | 20:33 | |
| *** maiteb has joined #openstack-infra | 20:34 | |
| *** sputnik13 has quit IRC | 20:34 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add ansible playbook to upgrade elasticsearch https://review.openstack.org/238185 | 20:35 |
| nibalizer | ok | 20:35 |
| nibalizer | review updated | 20:35 |
| nibalizer | im not sure we want to land it but at least what's in gerrit lines up with what we actually did | 20:35 |
| clarkb | nibalizer: we need to fix the sha1sum check in 04 too | 20:35 |
| clarkb | nibalizer: ++ | 20:36 |
| *** abregman is now known as abregman|nearby | 20:37 | |
| *** sputnik13 has joined #openstack-infra | 20:37 | |
| *** tdurakov has joined #openstack-infra | 20:37 | |
| clarkb | nibalizer: you gonna get the sha1sum fix in or should I do it? | 20:37 |
| fungi | dhellmann: AJaeger: one other question on the release notes publication... only publishing release notes when changes merge to the master branch means that if someone adds something noteworthy on a stable branch then it won't actually appear in stable branch release notes until a subsequent change lands in master. is that irrelevant (we don't publish branch-specific release note information)> | 20:38 |
| fungi | ? | 20:38 |
| openstackgerrit | matthew wagoner proposed openstack-infra/shade: Add new context manager for shade exceptions, cont. again https://review.openstack.org/243221 | 20:38 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Add ansible playbook to upgrade elasticsearch https://review.openstack.org/238185 | 20:38 |
| dhellmann | fungi : I thought about that, but I think master changes often enough that it won't be an issue | 20:38 |
| nibalizer | clarkb: ^^ | 20:38 |
| dhellmann | fungi : otherwise we need separate builds and publishing locations for each branch, which just seems excessive | 20:38 |
| clarkb | nibalizer: lgtm, time to run it? | 20:39 |
| *** tsg is now known as tsg_away | 20:39 | |
| nibalizer | kk | 20:39 |
| fungi | dhellmann: AJaeger: well, by default the job uses the master branch anyway because it's not a branch-aware pipeline. so any reason _not_ to run it (reusing the current state of master and any other branch tips) any time a change merges on a stable branch too? | 20:39 |
| AJaeger | dhellmann: or we would need to checkout master instead of the branch and publish from there | 20:39 |
| dhellmann | fungi : ok, maybe this is just my ignorance again. If something merges to stable/liberty, the code checked out for the post job isn't set to the commit that just merged? | 20:40 |
| fungi | dhellmann: it depends on what the job wants to do. jobs don't necessarily check out anything | 20:41 |
| nibalizer | clarkb: evidently cut can take a filename | 20:41 |
| dhellmann | AJaeger , fungi : it would be fine to publish on a merge to a stable branch as long as the *build* happens from the master branch, to get all of the notes. I assumed that wouldn't be easy/possible. It sounds like I'm wrong? | 20:41 |
| *** sputnik1_ has joined #openstack-infra | 20:41 | |
| *** rlandy has quit IRC | 20:41 | |
| nibalizer | so at least on the second half we can just do 'cut -d " " -f 1 /tmp/shasum.txt' | 20:41 |
| clarkb | nibalizer: rather than catting the file? | 20:41 |
| clarkb | ya | 20:41 |
| fungi | dhellmann: zuul will tell it (via an environment variable) the git ref which merged, but the job can ignore that and run whatever it wants | 20:41 |
| *** mahatic has quit IRC | 20:41 | |
| dhellmann | fungi : ok, what sets up the workspace for the job? the job? zuul? something else? | 20:42 |
| fungi | dhellmann: whatever macro you define in the job to do the workspace setup. in this case the gerrit-git-prep macro | 20:42 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/project-config: Identity API v3 only jobs non-voting in QA repos https://review.openstack.org/241452 | 20:42 |
| clarkb | nibalizer: we should see total nodes climb to 26 on the health json | 20:42 |
| *** rlandy has joined #openstack-infra | 20:42 | |
| *** sputnik13 has quit IRC | 20:43 | |
| nibalizer | its at 26 now | 20:43 |
| clarkb | so now we can just kibana to see if pushing in new things | 20:43 |
| fungi | dhellmann: AJaeger: but there's a branch prep version of that where you can just pass it a branch name such as master (we use that on our stable bitrot jobs in the periodic pipeline, which is similarly branchless) | 20:43 |
| dhellmann | fungi : ok, I guess I didn't realize this was so configurable | 20:43 |
| fungi | dhellmann: not only configurable, but already mostly written | 20:44 |
| fungi | we have the pieces to do that part, just need to plug in the right ones | 20:44 |
| clarkb | nibalizer: looks like ther eare new events | 20:45 |
| AJaeger | fungi: THANKS! Yes, this looks like a great suggestion. Thanks for the carefull review! | 20:45 |
| * AJaeger updates... | 20:45 | |
| clarkb | nibalizer: so now that will run and catch up, I think we are mostly done, just need my puppet patch for 1.7.3 to go in, reenable puppet, keep an eye out for other issues we need to address | 20:45 |
| *** arxcruz has joined #openstack-infra | 20:45 | |
| nibalizer | kk | 20:45 |
| *** sputnik1_ has quit IRC | 20:46 | |
| nibalizer | i +2'd your puppet patch | 20:46 |
| clarkb | mtreinish: mriedem if you all are around, now would be a reasonbale time to merge the e-r pyelasticsearch upgrade change | 20:46 |
| mtreinish | clarkb: whenever works for me I think | 20:47 |
| *** haypo has left #openstack-infra | 20:47 | |
| *** arxcruz has quit IRC | 20:47 | |
| *** chrall__ has quit IRC | 20:47 | |
| AJaeger | fungi: in that case I don't need the script and can use tox for publishing again ;) | 20:48 |
| mriedem | clarkb: did the unit tests get sorted out | 20:48 |
| fungi | AJaeger: yep, much simpler actually, sorry i didn't connect earlier on the concept that it was safe for this jobs to run as often as we like | 20:49 |
| mriedem | i've been side tracked with this cells bug for the last week | 20:49 |
| *** andersonvom has joined #openstack-infra | 20:49 | |
| clarkb | mriedem: I didn't realize there were unittest issues | 20:49 |
| *** someara2 has quit IRC | 20:50 | |
| AJaeger | fungi: no worries. I learn a bit this way ;) | 20:50 |
| clarkb | mriedem: do those tests connect to the real cluster? | 20:50 |
| *** achanda has quit IRC | 20:50 | |
| clarkb | I can look into it | 20:51 |
| *** cody-somerville has joined #openstack-infra | 20:51 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pull server list cache setting via API https://review.openstack.org/243311 | 20:52 |
| mordred | Shrews: ^^ I lied about being ready for release | 20:52 |
| Shrews | mordred: i place the cone of shame around your neck | 20:52 |
| mordred | Shrews: I tried to not pee in the corner | 20:53 |
| *** maiteb has quit IRC | 20:53 | |
| ihrachys | fungi: ping re periodic jobs for stable and branch enforcement | 20:53 |
| *** andersonvom has quit IRC | 20:54 | |
| *** yamamoto has joined #openstack-infra | 20:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Publish releasenotes https://review.openstack.org/242765 | 20:54 |
| nibalizer | clarkb: okay im taking off | 20:54 |
| nibalizer | good luck! | 20:54 |
| *** nikhil_k has quit IRC | 20:54 | |
| clarkb | nibalizer: ty | 20:54 |
| clarkb | mriedem: now we get ImportError: No module named backports.functools_lru_cache | 20:54 |
| *** hichihara has joined #openstack-infra | 20:54 | |
| fungi | ihrachys: yeah, the stable bitrot jobs already specify a branch explicitly, i believe. if other jobs got added to the periodic pipeline they probably need to be similarly templated and adjusted to have a periodic variant | 20:54 |
| *** nadya has quit IRC | 20:54 | |
| fungi | so that you can indicate with which branch you want them to run | 20:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Publish releasenotes https://review.openstack.org/242765 | 20:56 |
| AJaeger | dhellmann, fungi: rebased on top of the other change - hope that's it. | 20:56 |
| ihrachys | fungi: so I am looking at https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L388 and I wonder whether I can somehow set branch there. I tried to add branch: stable/liberty but it's probably not working for what I see in tox -e jjb output. | 20:56 |
| clarkb | https://bitbucket.org/jaraco/backports.functools_lru_cache appears to be the bug? | 20:56 |
| AJaeger | I'm calling it a day now, if something is not correct, add comments and I reworked tomorrow again... | 20:56 |
| AJaeger | fungi, thanks a lot for your help! | 20:56 |
| ihrachys | fungi: [note that I am bad at project-config patches, I usually just copy paste and hope for the best, so I may miss smth] | 20:57 |
| *** jamesmcarthur has joined #openstack-infra | 20:57 | |
| *** sridhar_ram1 has joined #openstack-infra | 21:00 | |
| *** sridhar_ram has quit IRC | 21:01 | |
| docaedo | HI! I have a question for infra folks regarding hosting for apps.openstack.org | 21:01 |
| docaedo | We are working on implementing an API, and as part of this we will rely on access to a MySQL DB. | 21:01 |
| docaedo | Would I be writing a puppet manifest to stand up the DB, or do we rely on a hosted DB instance on RAX? | 21:01 |
| *** AJaeger has quit IRC | 21:01 | |
| *** Swami has quit IRC | 21:02 | |
| docaedo | *or something else, if not one of those choices | 21:02 |
| *** tdurakov has quit IRC | 21:02 | |
| clarkb | mriedem: so I think we need a dep cap to prevent that broken package from installing and the rest appears to be a mock fail | 21:03 |
| mordred | docaedo: the puppet manifest should assume that a mysql db exists outside of itself | 21:03 |
| mordred | docaedo: and should take host/user/db/password as parameters | 21:04 |
| *** ddecapit has joined #openstack-infra | 21:04 | |
| *** ddecapit is now known as DuaneDeC7 | 21:04 | |
| docaedo | mordred: thanks, thats easy :) | 21:04 |
| clarkb | mriedem: the new pyelasticsearch Elasticsearch object attempts to connect to the remote in __init__ now | 21:04 |
| mordred | docaedo: then, calling context can decide whether it'll also install a db or use a trove | 21:04 |
| mriedem | clarkb: yeah, the __init__ thing is what i remember seeing failing | 21:04 |
| mriedem | and something about ports, but i don't remember | 21:04 |
| mriedem | like they changed the default port | 21:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pull server list cache setting via API https://review.openstack.org/243311 | 21:05 |
| clarkb | mriedem: ya default port is 9200 now so you have to explicitly state 80 for logstash | 21:05 |
| mordred | Shrews: now with a test | 21:05 |
| clarkb | which I thought we did but that may have just been our config and not sufficient for the tests | 21:05 |
| mordred | Shrews: or, rather, a tweak to a test that existed | 21:05 |
| openstackgerrit | greghaynes proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror https://review.openstack.org/165240 | 21:06 |
| *** rossella_ has quit IRC | 21:07 | |
| *** rossella_ has joined #openstack-infra | 21:07 | |
| *** Piet has joined #openstack-infra | 21:08 | |
| *** Sukhdev has joined #openstack-infra | 21:09 | |
| *** pradk has quit IRC | 21:09 | |
| fungi | ihrachys: those jobs are from templates defined in jenkins/jobs/python-bitrot-jobs.yaml and have a job-group defined as python-liberty-bitrot-jobs at the end of it. that job-group is added to several projects so far in jenkins/jobs/projects.yaml and then you seem to have similar associations of your periodic-liberty project-template to projects in zuul/layout.yaml | 21:11 |
| *** cody-somerville has quit IRC | 21:11 | |
| *** pradk has joined #openstack-infra | 21:11 | |
| fungi | ihrachys: so i think this should work. do you have an example periodic-{name}-docs-liberty or periodic-{name}-python27-liberty job log for one of those projects which seems to use the wrong branch? | 21:11 |
| ihrachys | fungi: there was one in my email | 21:12 |
| *** sridhar_ram1 is now known as sridhar_ram | 21:12 | |
| ihrachys | fungi: http://logs.openstack.org/periodic-stable/periodic-neutron-lbaas-python27-liberty/3da452c/ | 21:12 |
| *** sdake has quit IRC | 21:12 | |
| *** sdake_ has joined #openstack-infra | 21:13 | |
| fungi | ihrachys: looks fine http://logs.openstack.org/periodic-stable/periodic-neutron-lbaas-python27-liberty/3da452c/console.html#_2015-11-09_06_10_27_612 | 21:14 |
| fungi | "No ZUUL_REF so using requested branch stable/liberty from origin." | 21:14 |
| ihrachys | fungi: not when it comes to fetching neutron though: http://logs.openstack.org/periodic-stable/periodic-neutron-lbaas-python27-liberty/3da452c/tox/py27-1.log | 21:15 |
| *** craigbr has joined #openstack-infra | 21:15 | |
| ihrachys | fungi: note that the job is for lbaas that depends on neutron (and it fetches the dep using zuul-cloner) | 21:15 |
| greghaynes | https://review.openstack.org/#/c/237219/2 and the one below it should be a pretty easy +A if someone gets a free minute - trying to shrink down my nodepool patch stack :) | 21:15 |
| fungi | ihrachys: oh, neutron-lbaas is doing broken things in their project | 21:15 |
| ihrachys | fungi: but I had all details in the email | 21:15 |
| fungi | ihrachys: well, broken from the perspective of the periodic job anyway | 21:16 |
| ihrachys | fungi: it's ... unusual, yes, but that's the way it is (neutron is not released as a library, so we fetch from git) | 21:16 |
| fungi | ihrachys: zuul-cloner expects a ZUUL_REF or ZUUL_BRANCH. the job probably needs to export that if neutron-lbaas's tox env expects to consume it in that job | 21:16 |
| ihrachys | fungi: we need to tell to zuul-cloner somehow that it should fetch stable/* branch for neutron. one way is argument to the tool, another is envvar | 21:17 |
| *** otter768 has joined #openstack-infra | 21:17 | |
| ihrachys | fungi: right, that's the idea - to update the jobs to set ZUUL_BRANCH | 21:17 |
| ihrachys | fungi: though I am bad at jjb-fu so I don't follow where to do it, and how | 21:17 |
| mordred | Clint: there are times when I worry that we have too much governance structure in openstack | 21:20 |
| mordred | Clint: but then there are times like the debian thread you tweeted | 21:20 |
| craigbr | Hi. We're trying to get our tempest jobs set up for monasca-api and we've made the changes suggested by jhesketh. Could we get further review of https://review.openstack.org/#/c/240600/? Thanks | 21:20 |
| fungi | ihrachys: it's _probably_ safe to do it at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n89 alongside the export BRANCH={branch} since jobs with no ZUUL_BRANCH are generally the only place we use that macro anyway so i don't expect any to care what the "original" ZUUL_BRANCH value would have been | 21:20 |
| mordred | (https://lists.debian.org/debian-devel/2015/11/msg00110.html) for anyone following along | 21:21 |
| *** rbrndt has joined #openstack-infra | 21:21 | |
| mordred | Clint: and it makes me realize that I'm glad we have the process we have | 21:21 |
| *** otter768 has quit IRC | 21:21 | |
| ihrachys | fungi: so having another line with 'export ZUUL_BRANCH={branch}' there? | 21:22 |
| *** jhesketh has quit IRC | 21:23 | |
| fungi | ihrachys: yep | 21:23 |
| fungi | ihrachys: it might also be useful to call your tools/tox_install.sh script with xtrace so we can see the logic exercised there in the logs | 21:24 |
| fungi | right now i can only infer that zuul-cloner failed because the hard-coded neutron master branch git url is showing up in the pip output | 21:24 |
| *** jhesketh has joined #openstack-infra | 21:25 | |
| ihrachys | fungi: you can see zuul-cloner output and env in tox log. isn't it enough? | 21:28 |
| clarkb | the prefix on the log at http://logs.openstack.org/40/165240/14/check/gate-infra-puppet-apply-bare-trusty/4f219b2/console.html is polluting our indexing | 21:28 |
| clarkb | but thats the only oddity I have seen so far digging in kibana and thats unrelated to the upgrade | 21:29 |
| *** thiagop has quit IRC | 21:29 | |
| *** erikwilson has joined #openstack-infra | 21:29 | |
| *** DuaneDeC7 has quit IRC | 21:29 | |
| *** HeOS has joined #openstack-infra | 21:30 | |
| fungi | ihrachys: oh there is is in py27-1.log | 21:30 |
| fungi | ihrachys: yeah, so that confirms that it's looking for ZUUL_BRANCH | 21:31 |
| ihrachys | fungi: yes, I linked to it above | 21:31 |
| timrc | pabelanger: Your regex still looks broken to me... I can't match on say, "25%" or "2%" | 21:31 |
| timrc | pabelanger: Oh by 10% | 21:31 |
| *** nmagnezi has joined #openstack-infra | 21:32 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/elastic-recheck: Update pyelasticsearch to >=1.4 https://review.openstack.org/238633 | 21:32 |
| openstackgerrit | Clark Boylan proposed openstack-infra/elastic-recheck: Fix unittests https://review.openstack.org/243324 | 21:32 |
| clarkb | mriedem: asselin_ mtreinish ^ I think that may do it | 21:32 |
| asselin_ | clarkb, thanks, never got a chance to go back to that | 21:33 |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Refactor VPN functional tests to use common template https://review.openstack.org/240990 | 21:34 |
| fungi | ihrachys: an easy solution may be to just export something like ZUUL_BRANCH=${ZUUL_BRANCH-$BRANCH} in tools/tox_install.sh | 21:34 |
| clarkb | looks like the log weirdness may be related to swift | 21:34 |
| clarkb | er rather swift uploading logs | 21:34 |
| clarkb | I don't think swift itself is the problem | 21:34 |
| fungi | ihrachys: and then test what happens if you run it with export BRANCH=stable/liberty in the calling environment | 21:34 |
| ihrachys | fungi: that seems similar to adding an argument to zuul-cloner. note that lbaas is not the only job that uses that approach, that's why I wanted to avoid changes per project | 21:34 |
| mriedem | clarkb: can we get a comment with the cap in requirements.txt? then i'm +2 | 21:34 |
| clarkb | jesusaurus: any chance you were planning to update the partial get support in os-loganalyze today? | 21:35 |
| clarkb | mriedem: the commit message isn't enough? | 21:35 |
| mriedem | clarkb: posterity | 21:35 |
| clarkb | mriedem: git log :P | 21:35 |
| clarkb | but sure | 21:35 |
| mriedem | i'll forget in a year and i'm afraid git blame won't help me | 21:35 |
| ihrachys | fungi: btw testing the change for periodic is a bit unclear since I can't post a patch to trigger it | 21:35 |
| mriedem | don't we require comments in g-r for the same reason? | 21:35 |
| mriedem | for caps | 21:35 |
| fungi | ihrachys: agreed, i just meant try that locally to confirm a similar conditional fallback export in the branch-git-prep builder macro is sane | 21:35 |
| clarkb | mriedem: i don't know but its a silly rule if so | 21:35 |
| clarkb | mriedem: the info should go in the commit message imo | 21:36 |
| *** tsg_away is now known as tsg | 21:36 | |
| clarkb | git log -p is your friend | 21:36 |
| fungi | ihrachys: that avoids overwriting ZUUL_BRANCH with BRANCH except in situations where it's not already set (such as in the periodic pipeline) | 21:36 |
| fungi | ihrachys: oh, wait, exporting it in branch-git-prep isn't going to cause it to end up in the environment for the python27 builder | 21:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/elastic-recheck: Update pyelasticsearch to >=1.4 https://review.openstack.org/238633 | 21:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/elastic-recheck: Fix unittests https://review.openstack.org/243324 | 21:38 |
| fungi | ihrachys: so within the tools/tox_install.sh or similar script of projects using this pattern is probably the sanest place for it to reside | 21:38 |
| *** cschwede has quit IRC | 21:38 | |
| ihrachys | fungi: ack. I believe we are safe to do the same in master right? I don't want to introduce stable only changes on every new release | 21:40 |
| *** hdd has joined #openstack-infra | 21:40 | |
| *** achanda has joined #openstack-infra | 21:40 | |
| *** cschwede has joined #openstack-infra | 21:41 | |
| *** achanda has quit IRC | 21:41 | |
| fungi | ihrachys: yes, should be perfectly safe on master. basically if BRANCH is set and ZUUL_BRANCH is not set then the script calling zuul-cloner can assume ZUUL_BRANCH should be set to BRANCH. that way if neither of them is set then it still falls back to master (because that's zuul-cloner's default drop back and punt behavior) | 21:41 |
| *** kozhukalov_ has quit IRC | 21:41 | |
| fungi | ihrachys: also, as i said, you can test it locally by just setting those variables independently, together, or not, and seeing what tox tests | 21:42 |
| *** mwagner has quit IRC | 21:42 | |
| *** hdd has quit IRC | 21:42 | |
| ihrachys | fungi: I need to install zuul though | 21:43 |
| fungi | ihrachys: you can install it into a virtualenv and put the path to the executable in the ZUUL_CLONER definition at the top of tools/tox_install.sh | 21:44 |
| ihrachys | fungi: yeah, sure. will do. | 21:44 |
| fungi | zuul-cloner doesn't have to be installed into tox's virtualenv, the script just needs to know where to find teh zuul-cloner entrypoint script | 21:44 |
| ihrachys | fungi: roger. thanks for holding my hand on it. | 21:45 |
| fungi | ihrachys: no problem, and sorry for having you reexplain it because i didn't read closely enough when skimming dev ml threads this morning! | 21:45 |
| *** alivigni has quit IRC | 21:49 | |
| jesusaurus | clarkb: yeah, I can work on that today. the general plan is to change the list appends to yields, right? | 21:49 |
| anteaya | clarkb: I'm thinking about what to post for this edition of learning to debug the gate, is there anything helpful about new logstash that I can point at for today's edition? | 21:49 |
| *** SumitNaiksatam has joined #openstack-infra | 21:49 | |
| *** austin81 has left #openstack-infra | 21:49 | |
| *** jaypipes has quit IRC | 21:49 | |
| clarkb | jesusaurus: ya | 21:50 |
| *** devl_ has joined #openstack-infra | 21:51 | |
| *** salv-orlando has joined #openstack-infra | 21:51 | |
| anteaya | clarkb: I'll link to the 1.7 docs: https://www.elastic.co/guide/en/elasticsearch/reference/1.7/index.html | 21:51 |
| *** trown is now known as trown|outttypeww | 21:53 | |
| *** bswartz has quit IRC | 21:56 | |
| *** Sukhdev has quit IRC | 21:57 | |
| *** matrohon has quit IRC | 21:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Pull server list cache setting via API https://review.openstack.org/243311 | 21:59 |
| mordred | Shrews: ^^ sigh. had to fix pep8 | 22:00 |
| *** baoli has joined #openstack-infra | 22:00 | |
| mordred | clarkb, fungi, pleia2, jeblair: if you have a sec, simple patch I'd like to get landed before releasing | 22:01 |
| *** dprince has quit IRC | 22:01 | |
| *** hdd has joined #openstack-infra | 22:02 | |
| *** thorst has quit IRC | 22:03 | |
| Shrews | mordred: it's simple enough i just +3'd it | 22:03 |
| jhesketh | Morning | 22:03 |
| clarkb | I have to say I have no idea what that fixes given the commit message and diff | 22:03 |
| clarkb | seems like a bug in cloud config | 22:04 |
| mordred | clarkb: sorry bout that - it was a int vs. string issue - internal config storage in occ is all normalized to strings | 22:04 |
| clarkb | I would expect the dict data to match what the api retunrs too | 22:04 |
| mordred | clarkb: becuase openstack blows up all over the place | 22:04 |
| jeblair | infra-root, zaro, sdague, anteaya: i wrote a summary report from the gerrit user summit: http://lists.openstack.org/pipermail/openstack-dev/2015-November/078945.html | 22:04 |
| mordred | when you pass ints | 22:04 |
| clarkb | mordred: oh its a case of flatten openstack sillyness? | 22:05 |
| mordred | clarkb: yah | 22:05 |
| *** julim has quit IRC | 22:05 | |
| *** salv-orlando has quit IRC | 22:05 | |
| mordred | clarkb: so config[] is flattened silliness - but then on a case-by-case basis some things require other types because YAY | 22:05 |
| jeblair | also ttx: http://lists.openstack.org/pipermail/openstack-dev/2015-November/078945.html | 22:05 |
| *** Sukhdev has joined #openstack-infra | 22:05 | |
| mordred | jeblair: woot! | 22:06 |
| *** Swami has joined #openstack-infra | 22:07 | |
| clarkb | jeblair: neat, the reserach on code quality is interesting as is the polymer stuff | 22:08 |
| jeblair | clarkb: yeah, i've seen shane at a few open sourcey conferences, he seems interested in our community and enjoys the amount of data we provide to him and his grad students :) | 22:09 |
| jeblair | and polymer looks fairly cool, and comprehensible (you make new html tags and use them!) | 22:10 |
| fungi | jeblair: thanks for posting that to the ml! | 22:10 |
| *** rhallisey has quit IRC | 22:11 | |
| jeblair | np | 22:11 |
| *** pc_m has quit IRC | 22:11 | |
| *** yamamoto has quit IRC | 22:11 | |
| jeblair | it was an intense two days -- i was about as active as at an openstack summit, so hopefully captured the important bits | 22:11 |
| *** achanda has joined #openstack-infra | 22:12 | |
| *** peristeri has quit IRC | 22:12 | |
| *** angdraug has quit IRC | 22:12 | |
| jeblair | (except instead of going to parties, i was driving up 880) | 22:12 |
| *** gordc has quit IRC | 22:13 | |
| *** wenlock has joined #openstack-infra | 22:14 | |
| *** tqtran is now known as tqtran-afk | 22:14 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Fix unittests https://review.openstack.org/243324 | 22:15 |
| *** barnaby has quit IRC | 22:15 | |
| pabelanger | timrc: Ya, that is what the current UI for grafana uses | 22:17 |
| *** dkranz has quit IRC | 22:18 | |
| anteaya | jeblair: thanks for the summary, many interesting points | 22:19 |
| *** wenlock has quit IRC | 22:19 | |
| *** amuller is now known as amuller_afk | 22:19 | |
| *** tiswanso has quit IRC | 22:20 | |
| *** jamesmcarthur has quit IRC | 22:20 | |
| ihrachys | fungi: I would appreciate if you review it: https://review.openstack.org/#/c/243333/1 | 22:21 |
| clarkb | infra-root one more review on https://review.openstack.org/#/c/243232/1 will allow me to turn puppet back on in the ES cluster | 22:22 |
| clarkb | I haven't seen any other issues. Just the CORS thing which is only really exposed to admins | 22:22 |
| *** hichihara has quit IRC | 22:23 | |
| jeblair | done | 22:23 |
| *** Sukhdev has quit IRC | 22:24 | |
| *** erikwilson has quit IRC | 22:24 | |
| *** devl_ has quit IRC | 22:25 | |
| clarkb | ty | 22:26 |
| *** FallenPegasus has quit IRC | 22:26 | |
| openstackgerrit | Merged openstack-infra/system-config: Update ES to 1.7.3 https://review.openstack.org/243232 | 22:26 |
| *** nikhil has joined #openstack-infra | 22:27 | |
| *** hdd has quit IRC | 22:27 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Update pyelasticsearch to >=1.4 https://review.openstack.org/238633 | 22:29 |
| jeblair | fungi, mordred: i believe the ethercalc site ate my spreadsheet -- maybe they drop data periodically? at any rate, i guess that answers the question of why we might want to run our own if we like it. :) (i like it) | 22:29 |
| *** rfolco has quit IRC | 22:30 | |
| *** thorst has joined #openstack-infra | 22:30 | |
| *** barnaby has joined #openstack-infra | 22:30 | |
| anteaya | jeblair: so I believe there are gerrit sprints this week, if I remember correctly zaro is taking part, I can't remember if you said you were or not, are you? | 22:31 |
| jeblair | anteaya: there are, he is, i am not | 22:31 |
| anteaya | thanks | 22:31 |
| *** antonym has quit IRC | 22:32 | |
| anteaya | shane is from mcgill the paper is on a queens site | 22:32 |
| * anteaya is curious | 22:32 | |
| *** antonym has joined #openstack-infra | 22:32 | |
| *** kgiusti has quit IRC | 22:33 | |
| anteaya | the lab is at queens | 22:33 |
| *** August1914 has joined #openstack-infra | 22:34 | |
| *** sdake_ is now known as sdake | 22:34 | |
| *** dims_ has joined #openstack-infra | 22:34 | |
| *** thorst has quit IRC | 22:34 | |
| *** angdraug has joined #openstack-infra | 22:35 | |
| greghaynes | Speaking of hosting random web services - AFAICT the robots.txt did not fix paste.o.o being super slow | 22:35 |
| *** dims has quit IRC | 22:35 | |
| greghaynes | Someone should probably look at the logs and confirm though... | 22:35 |
| jeblair | anteaya: i believe he did his phd work at queens and was recently hired into a professorship at mcgill | 22:36 |
| *** jamielennox|away is now known as jamielennox | 22:38 | |
| jeblair | anteaya: it's also possible some of what he was relaying is from not-yet-published papers. it was a pretty high-level talk. | 22:39 |
| anteaya | jeblair: awesome | 22:39 |
| anteaya | yes the queens lab has him listed as alumni | 22:39 |
| anteaya | he sounds like a great person to spend time with | 22:40 |
| clarkb | anteaya: latest tree style tabs is working with ff42 now | 22:40 |
| anteaya | also the sail lab looked at realease notes: http://sail.cs.queensu.ca/publications/pubs/EMSE-%20Abebe-2015.pdf dhellmann ttx you might be interested, | 22:40 |
| anteaya | clarkb: woooot | 22:40 |
| clarkb | my browser is useable again | 22:40 |
| anteaya | clarkb: yay! | 22:40 |
| jeblair | oh, maybe i should have mentioned this in the summary email -- he also made what he described as "the worst sales pitch" -- leave your high-paying industry tech jobs and come join him as a grad student! | 22:41 |
| anteaya | ha ha ha | 22:41 |
| anteaya | it sounds like a really interesting area | 22:41 |
| anteaya | I'm sitting in a room full of comp sci students, I'll share that info | 22:41 |
| clarkb | I have already had at least one tenured professor try that sales pitch on me | 22:41 |
| jeblair | clarkb: do babies love ramen? | 22:42 |
| *** devl_ has joined #openstack-infra | 22:42 | |
| clarkb | jeblair: no they have mcuh more expensive tastes | 22:42 |
| clarkb | in fact I really need to look into blending my own baby food | 22:43 |
| *** docaedo2 has joined #openstack-infra | 22:43 | |
| *** harlowja_ has quit IRC | 22:43 | |
| anteaya | pitched the room | 22:44 |
| anteaya | my sister used to blend her own baby food | 22:44 |
| anteaya | she would mash the peas and then freeze in in ice cube trays | 22:45 |
| mordred | clarkb: the humans who were living in my apartment with a baby blended their own food | 22:45 |
| *** bexelbie has quit IRC | 22:45 | |
| clarkb | the problem is with 2 we don't relaly have any extra time to do extra work around the house | 22:45 |
| mordred | clarkb: turns out, you can totally blend basically whatever you're eating already and it works great - and helps develop the palette | 22:45 |
| *** sdake_ has joined #openstack-infra | 22:45 | |
| clarkb | same reason we gave up on trying reuseable diapers | 22:45 |
| clarkb | mordred: ya | 22:45 |
| *** rbrndt has left #openstack-infra | 22:45 | |
| clarkb | you can also just make small enough chunks and make sure its not too hot and have them go to town | 22:46 |
| anteaya | clarkb: yeah, makes it tough to do the extra work | 22:46 |
| clarkb | its just so much extra work | 22:46 |
| mordred | yup | 22:46 |
| mordred | I helped out with that some | 22:46 |
| * anteaya is giggling at that image | 22:46 | |
| clarkb | and as greghaynes will tell you when you are in the poop you are in the poop | 22:46 |
| mordred | that baby thinks that normal baby food is very boring now - wants real people food that tastes like real people food | 22:46 |
| pabelanger | So, stackalytics-processor finally finished: http://stackalytics.openstack.org | 22:46 |
| greghaynes | haha, yes | 22:46 |
| pabelanger | however still debugging apache performance | 22:46 |
| greghaynes | There is a feedback loop with two babies | 22:47 |
| *** sdake has quit IRC | 22:47 | |
| *** baoli has quit IRC | 22:47 | |
| jeblair | clarkb: what's the dsvm-resolver job? and should i be worried it failed for https://review.openstack.org/241324 ? | 22:47 |
| mordred | pabelanger: there must be a bug - that shows me in the top 10 of reviews which can't possibly be correct | 22:47 |
| clarkb | jeblair: I think it does some fancy try to resolve the deps directly rathre than pip freezing and no its expected to fail | 22:47 |
| clarkb | jeblair: because our current 300 something dep list is not directly resolvable | 22:47 |
| clarkb | \o/ | 22:47 |
| jeblair | clarkb: okay, i will +2 then! | 22:48 |
| *** FallenPegasus has joined #openstack-infra | 22:48 | |
| *** dewsday has joined #openstack-infra | 22:48 | |
| pabelanger | mordred: crazy, but true :) | 22:48 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: [Rally] add new job for verification https://review.openstack.org/240580 | 22:48 |
| clarkb | jeblair: it uses lifeless' experimental pip resolver iirc | 22:48 |
| mtreinish | jeblair: yeah it builds pip from lifeless's branch with a real dep solver: http://logs.openstack.org/24/241324/4/check/gate-requirements-integration-dsvm-resolver/54518b5/console.html#_2015-11-09_18_19_36_786 | 22:48 |
| jeblair | pabelanger: woot! andreas is #1 so the data are probably correct :) | 22:48 |
| *** amuller_afk is now known as amuller | 22:49 | |
| mtreinish | and then tries to install all of global reqs in one shot | 22:49 |
| pabelanger | mordred: you're 9 on stackaltyics.com. I haven't rerun the processor today | 22:49 |
| mtreinish | which never works | 22:49 |
| clarkb | fixing that would actually be a goo dtask for this cycle | 22:49 |
| clarkb | probably require owrking with $upstreams though | 22:49 |
| pabelanger | full import took 36 hours or something | 22:49 |
| mordred | pabelanger: it's snappy to use | 22:49 |
| mordred | pabelanger: which it was NOT the last time I ran it | 22:49 |
| *** Sukhdev has joined #openstack-infra | 22:49 | |
| *** stevemar_ has quit IRC | 22:49 | |
| *** baoli has joined #openstack-infra | 22:49 | |
| *** baoli has quit IRC | 22:49 | |
| *** baoli has joined #openstack-infra | 22:50 | |
| timrc | mordred: Snappy is trademarked by Canonical. A cease and desist letter has been sent. | 22:50 |
| pabelanger | mordred: Still some tuning to do, right now we are pushing 12.5GB of RAM for apache, uwsgi and memcached | 22:50 |
| *** amitgandhinz has quit IRC | 22:50 | |
| timrc | pabelanger: Super cool re: stackalytics | 22:50 |
| pabelanger | testing with nginx locally, seems to be more forgiving | 22:50 |
| * mordred hands timrc a mostly chewed potato | 22:50 | |
| nibalizer | jeblair: thanks for the aummary | 22:53 |
| *** eharney has quit IRC | 22:53 | |
| *** nmagnezi has quit IRC | 22:55 | |
| *** tiswanso has joined #openstack-infra | 22:55 | |
| fungi | jeblair: i'm in favor of someone working out how to get an ethercalc (spreadsheet.openstack.org?) service up and running | 22:55 |
| *** nmagnezi has joined #openstack-infra | 22:55 | |
| *** tiswanso has quit IRC | 22:55 | |
| *** tonytan4ever has quit IRC | 22:55 | |
| *** tonytan4ever has joined #openstack-infra | 22:56 | |
| *** tiswanso has joined #openstack-infra | 22:56 | |
| *** docaedo2 has quit IRC | 22:56 | |
| jeblair | fungi: cool | 22:56 |
| *** FallenPegasus has quit IRC | 22:56 | |
| *** burgerk has quit IRC | 22:57 | |
| *** barnaby has quit IRC | 22:57 | |
| *** jamesmcarthur has joined #openstack-infra | 22:57 | |
| fungi | greghaynes: i think there may be multiple factors slowing paste.o.o... i also noticed the other day drizzled was running and eating 100% of the cpu there. i stopped it and i think configured it to no longer start by default since our puppet manifest hasn't used it for some ages now | 22:57 |
| *** tonytan4ever has quit IRC | 22:57 | |
| greghaynes | fungi: :) that sounds a lot like the DB issue I have been saying for a while | 22:58 |
| timrc | mordred: Yum. | 22:58 |
| *** mwagner has joined #openstack-infra | 22:58 | |
| *** bexelbie has joined #openstack-infra | 22:58 | |
| fungi | greghaynes: well, i think it was just wasting cpu cycles, but still not helpful | 22:58 |
| greghaynes | fungi: Yea, I have run into this issue with lodgeit before where if you paste an enormous one-line paste then it forever makes all loads slow | 22:59 |
| mordred | fungi: we should maybe uninstall it? | 23:00 |
| *** nmagnezi has quit IRC | 23:00 | |
| openstackgerrit | matthew wagoner proposed openstack-infra/shade: Add new context manager for shade exceptions, cont. again https://review.openstack.org/243221 | 23:00 |
| *** ihrachys has quit IRC | 23:01 | |
| fungi | mordred: oh, i think i did maybe. just checked now and only drizzle-client, libdrizzle1 and libdrizzledmessage0 remained, but i removed those just now as well | 23:01 |
| clarkb | nibalizer: puppet is reenabled on all those hosts now | 23:01 |
| timrc | greghaynes: I exposed a similar problem in Jira... apparently adding a 100MB comment really screws things up :) Glad they put some constraints checking around their API. | 23:01 |
| *** krtaylor has quit IRC | 23:02 | |
| clarkb | timrc: was it not even an attachment? | 23:02 |
| greghaynes | fungi: oh, you mean to say that our lodgeit doesnt use drizzle and yet there was a drizzle doing that | 23:02 |
| greghaynes | fungi: hahaah | 23:02 |
| timrc | clarkb: Not an attachment a comment. Total accident on my part. | 23:02 |
| fungi | greghaynes: yep, that was basically the point i was attempting to convey | 23:02 |
| clarkb | timrc: was it an ISO? | 23:03 |
| timrc | clarkb: It happily accepted the comment and because the wiki and bug tracker were integrated, DoS'ed the wiki too. | 23:03 |
| * clarkb remembers when 700MB ISOs broke gerrit | 23:03 | |
| timrc | clarkb: Nah it was a bunch of unfiltered concated log files lol | 23:03 |
| *** someara2 has joined #openstack-infra | 23:03 | |
| *** sabeen2 has quit IRC | 23:04 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/os-loganalyze: Add Range header support https://review.openstack.org/241058 | 23:04 |
| *** ybathia has joined #openstack-infra | 23:04 | |
| jesusaurus | clarkb: here's a new patchset with yields ^^ | 23:04 |
| *** aysyd has quit IRC | 23:04 | |
| *** achanda has quit IRC | 23:05 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Decouple builders from nodepool instance https://review.openstack.org/232135 | 23:06 |
| *** wenlock has joined #openstack-infra | 23:07 | |
| dhellmann | hey, did something about gerrit change today? maybe a minor upgrade? some of my css overrides are no longer working | 23:07 |
| adam_g | is there a specific -infra person who can help me register a new irc channel? | 23:08 |
| *** markmcclain has quit IRC | 23:09 | |
| *** markmcclain has joined #openstack-infra | 23:10 | |
| dhellmann | adam_g : pleia2 pointed me to http://docs.openstack.org/infra/system-config/irc.html when we renamed the release management channel recently | 23:10 |
| pabelanger | fungi: greghaynes: results from testing apache and nginx: http://pastebin.com/3V7XEmZL | 23:10 |
| adam_g | dhellmann, ah, cool! thanks | 23:10 |
| pleia2 | adam_g: once you complete the link from dhellmann, I can leave comments in the review if anything different needs to be done | 23:10 |
| pabelanger | nginx way leaner | 23:11 |
| pabelanger | per process | 23:11 |
| dhellmann | adam_g : http://docs.openstack.org/infra/manual/creators.html#configure-gerritbot-to-announce-changes may also be useful | 23:11 |
| clarkb | it looks like 10kb difference | 23:11 |
| clarkb | thats not huge | 23:11 |
| *** sputnik13 has joined #openstack-infra | 23:11 | |
| adam_g | pleia2, was just hoping to rename and move irc bots /w https://review.openstack.org/#/c/242642/ but its failing on channel not being registered with chanserv | 23:12 |
| adam_g | ill do that now | 23:12 |
| *** yamamoto has joined #openstack-infra | 23:12 | |
| greghaynes | pabelanger: So for the nginx one, are there other uwsgi processes? | 23:12 |
| greghaynes | pabelanger: Seems like if no then that is the difference - it isnt doing the prefork trick | 23:12 |
| *** FallenPegasus has joined #openstack-infra | 23:13 | |
| pabelanger | greghaynes: that is cropped data but yes still 2 uwsgi processes | 23:13 |
| pleia2 | adam_g: ah, renaming is trickier, I tend to use these instructions for the technical/freenode/chanserv bits: https://wiki.ubuntu.com/IRC/MovingChannels#Forwarding | 23:14 |
| greghaynes | pabelanger: ok, so seems like they are pretty similar | 23:14 |
| *** krtaylor has joined #openstack-infra | 23:14 | |
| greghaynes | or am I missing something? | 23:14 |
| adam_g | pleia2, ah, i was planning on manually herding people over | 23:14 |
| pleia2 | adam_g: unless it's a very small channel, that rarely works :) | 23:15 |
| *** sputnik13 has quit IRC | 23:15 | |
| pabelanger | greghaynes: Ya, I think nginx is just forwarding the request straight through to uwsgi. Where apache might be caching stuff | 23:15 |
| pleia2 | and mentions of the old channel remain in logs, notes, etc | 23:15 |
| Jokke_ | pabelanger: am I missing something? For me it looks like the diff is ~100megs out of ~6gigs | 23:15 |
| clarkb | Jokke_: I think its kb not megs | 23:16 |
| clarkb | 12 vs 2 | 23:16 |
| clarkb | resident, unless I am missreading | 23:16 |
| *** wenlock has quit IRC | 23:16 | |
| Jokke_ | clarkb: there is around 100meg diff in the memcached and uwsgi, dunno if related or not | 23:16 |
| greghaynes | Yea, IMO I wouldnt even be adding up numbers, I would just run them identically, hit the page exactly once, then record free -m | 23:16 |
| clarkb | Jokke_: I think we are just comparing nginx vs apache | 23:16 |
| *** sdake_ is now known as sdake | 23:17 | |
| pabelanger | My bad, I was looking at virtual memory | 23:17 |
| pabelanger | ya, res memory only a few k different | 23:17 |
| greghaynes | but regardless, the http server isnt really doing anything here | 23:17 |
| *** ZZelle_ has quit IRC | 23:17 | |
| Jokke_ | clarkb: yeah, that looks like 10k diff ;) | 23:17 |
| *** yamamoto has quit IRC | 23:17 | |
| *** otter768 has joined #openstack-infra | 23:18 | |
| greghaynes | pabelanger: so, if after responding to 1req we dont have enough mem to run 2 workers (and I would hope something more like 4) then we just dont have enough mem to run the app in that configuration. | 23:18 |
| clarkb | you can likely trim the virtual count by disabling unused mods | 23:18 |
| clarkb | but thats likely just not necessary in the grand scheme of things | 23:18 |
| greghaynes | Yea, it is pretty small compared to where all the mem is going I am willing to bet | 23:18 |
| pabelanger | greghaynes: so, today we have a 15GB instance with apache and 2 uwsgi process. We are just about maxing 14.5GB | 23:19 |
| pabelanger | stackalytics.com, is using 16GB but I don't know the stats for it. It runs nginx and uwsgi | 23:20 |
| pabelanger | next step for us is 30GB | 23:20 |
| greghaynes | pabelanger: from what I saw before there was 2g or so of cached? | 23:20 |
| mordred | or multiple machines | 23:20 |
| clarkb | we can use 16GB flavor | 23:20 |
| jeblair | pabelanger: where are you getting 14.5GB from? | 23:20 |
| *** pradk has quit IRC | 23:21 | |
| pabelanger | jeblair: apache + uwsgi + memcached + stackaltyics-processor | 23:21 |
| greghaynes | I think we really need at least 2 workers per machine even if we did multiple boxes | 23:21 |
| mordred | we could totally run a memcached machine and one or more apache/nginx machines | 23:21 |
| pabelanger | mordred: ya, if we want to do that | 23:21 |
| greghaynes | oh, yea, if you can offload some of the memory to other hosts then win | 23:21 |
| jeblair | pabelanger: i see ~10G used currently | 23:22 |
| pabelanger | jeblair: ya, stackaltyics-processor is not running | 23:22 |
| jeblair | gotcha | 23:22 |
| pabelanger | jeblair: that import takes about 7 hours | 23:22 |
| pabelanger | and consumes 3gb or so | 23:22 |
| mordred | pabelanger: that's a process that starts up, consumes memory, writes the data into memcached and then goes away, yeah? | 23:22 |
| jeblair | what's uwsgi doing? | 23:22 |
| *** otter768 has quit IRC | 23:22 | |
| jeblair | i mean, right now, there are 2 uwsgi procs running at 100% cpu with 3.5G ram each... | 23:23 |
| pabelanger | mordred: on stackup we import dump into memcached, then every 8 hours process new data (process goes away at end) | 23:23 |
| pabelanger | jeblair: yes, that apache trying to load stackalytics | 23:24 |
| *** tiswanso has quit IRC | 23:24 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Revert pinning of astara to stable/liberty https://review.openstack.org/243349 | 23:24 |
| jeblair | pabelanger: i don't understand that | 23:24 |
| pabelanger | currently, from cold start take 10mins to process the data | 23:24 |
| clarkb | jeblair: basically everything is in memory | 23:24 |
| clarkb | so it is high cost to start | 23:24 |
| mordred | so - sake of argument (not saying this is a good idea) we could have a memcached server, a stackalytics-processor server, N web servers and a load balancer | 23:24 |
| *** devkulkarni has quit IRC | 23:24 | |
| pabelanger | apache seems to reload data every so often (30mins it seems). nginx loads once from what I see | 23:25 |
| greghaynes | There is both a max lifetime and max requests deal in uwsgi | 23:25 |
| greghaynes | which is something we will definitely want to tune | 23:25 |
| jeblair | we're using proxy_uwsgi, right? | 23:25 |
| pabelanger | yes | 23:25 |
| jeblair | that means apache isn't doing anything except proxying requests, right? | 23:25 |
| pabelanger | that is my understanding | 23:26 |
| jeblair | ok, so *uwsgi* is loading everything into memory and taking a long time and lots of resources to do it... | 23:26 |
| jeblair | apache is just waiting for that to happen | 23:26 |
| pabelanger | right, thats what I see | 23:27 |
| nibalizer | clarkb: sweet re: puppet enabled | 23:27 |
| pabelanger | jeblair: it looks like the data is only valid for so long, because uwsgi will pin to 100% every few minutes. | 23:27 |
| jeblair | pabelanger: does apache launch the uwsgi processes, or are they started separately? | 23:27 |
| pabelanger | jeblair: separately | 23:28 |
| pabelanger | uwsgi server launches them | 23:28 |
| pabelanger | service* | 23:28 |
| greghaynes | pabelanger: that is probably the max-lifetime | 23:28 |
| pabelanger | greghaynes: in uwsgi? | 23:28 |
| greghaynes | yes | 23:28 |
| greghaynes | which is scary - because that means it doesnt prefork well | 23:29 |
| pabelanger | I wonder if that is because I am using a tcp socket for apache and a unix file for nginx | 23:29 |
| pabelanger | *shurgs* | 23:29 |
| *** sdake_ has joined #openstack-infra | 23:29 | |
| *** sdake has quit IRC | 23:29 | |
| *** abregman|nearby has quit IRC | 23:30 | |
| jeblair | greghaynes: where is max-lifetime? | 23:30 |
| * greghaynes googles | 23:30 | |
| greghaynes | jeblair: looks like its called max-worker-lifetime | 23:31 |
| *** pahuang_ has joined #openstack-infra | 23:31 | |
| greghaynes | jeblair: in uwsgi config | 23:31 |
| mtreinish | clarkb: did we ever restart puppet on the logstash.o.o? | 23:31 |
| jeblair | greghaynes: that's disabled by default; are we setting that? | 23:32 |
| clarkb | mtreinish: yes | 23:32 |
| pabelanger | no | 23:32 |
| greghaynes | jeblair: I have no idea | 23:32 |
| jeblair | i don't see any evidence to that effect | 23:32 |
| greghaynes | huh. So the app could have something else out of band | 23:32 |
| pabelanger | our current settings: https://github.com/openstack-infra/puppet-stackalytics/blob/master/files/uwsgi.ini | 23:32 |
| greghaynes | that is also wierd to do though | 23:32 |
| *** sputnik13 has joined #openstack-infra | 23:32 | |
| mtreinish | clarkb: hmm, I'm wondering why I'm not seeing anything in the db from: https://review.openstack.org/#/c/240377/ | 23:32 |
| mtreinish | oh I know why, because I'm dumb | 23:33 |
| greghaynes | since usually youre just waiting on an accept() in the framework | 23:33 |
| mtreinish | that regex doesn't match periodic jobs | 23:33 |
| clarkb | mtreinish: ok, let me know if you need any other insights :) | 23:33 |
| *** sputnik13 has quit IRC | 23:36 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Fix regex for periodic job subunit collection https://review.openstack.org/243351 | 23:36 |
| mtreinish | clarkb: ^^^ | 23:36 |
| jeblair | pabelanger: /var/log/uwsgi/app/stackalytics.log says things like [Mon Nov 9 23:18:32 2015] GET / => generated 0 bytes in 682178 msecs (HTTP/1.1 200) 2 headers in 81 bytes (0 switches on core 0) | 23:37 |
| pabelanger | greghaynes: jeblair: looks like stackaltyics will force a reload of data ever hour. Going to see if that is the issue | 23:37 |
| jeblair | pabelanger: right after Mon Nov 9 23:29:54 2015 - uwsgi_response_write_body_do(): Broken pipe [core/writer.c line 296] | 23:37 |
| jeblair | IOError: write error | 23:37 |
| clarkb | mtreinish: needs to be .* | 23:37 |
| mtreinish | ugh, one sec | 23:37 |
| *** FallenPegasus has quit IRC | 23:37 | |
| jeblair | pabelanger: there are no timestamps in that log, so i can't tell if those line up with your 1hour hypothesis | 23:37 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Fix regex for periodic job subunit collection https://review.openstack.org/243351 | 23:37 |
| mtreinish | clarkb: ^^^ there we go | 23:37 |
| *** achanda has joined #openstack-infra | 23:37 | |
| *** sdake_ has quit IRC | 23:37 | |
| greghaynes | If it does that, well want to see about how much mem that uses and do some math to set max-lifetime | 23:38 |
| greghaynes | after making sure that it actually does that, obviously ;) | 23:38 |
| clarkb | mtreinish: +2 | 23:38 |
| *** achanda has quit IRC | 23:38 | |
| mtreinish | clarkb: cool, thanks | 23:38 |
| *** sdake has joined #openstack-infra | 23:38 | |
| pabelanger | jeblair: Ya, I don't know how to add timestamps. Will google | 23:38 |
| pabelanger | checking my local instance too | 23:39 |
| pabelanger | jeblair: broken pipe might be apache closing connection to uwsgi after some default time out | 23:40 |
| jeblair | pabelanger: that would make sense, making that a symptom of the delayed response and not the cause | 23:41 |
| pabelanger | ya | 23:41 |
| pabelanger | eventually apache returns 503 when uwsgi is pinned | 23:41 |
| greghaynes | Its possible that first request is just extremely slow, maybe there is some preloading things we need to do in uwsgi? | 23:42 |
| jeblair | greghaynes: i think pabelanger's theory is that first request and one request every hour thereafter is slow | 23:42 |
| pabelanger | Ya, that seems to make sense right now | 23:42 |
| *** dewsday has quit IRC | 23:43 | |
| pabelanger | would be interested to see how uwsgi handles the max-worker-lifetime and if it avoids any downtime to the user | 23:43 |
| greghaynes | So there is a preload deal in all these wsgi frameworks to deal with the first request issue | 23:44 |
| greghaynes | not sure of the exact way in uwsgi | 23:44 |
| *** hashar_ has joined #openstack-infra | 23:44 | |
| greghaynes | but you also need to know what to preload | 23:44 |
| lifeless | jeblair: as answered, but no you should not worry about it failing | 23:44 |
| *** otter768 has joined #openstack-infra | 23:45 | |
| *** claudiub has quit IRC | 23:45 | |
| greghaynes | pabelanger: that is a big reason you have to run more than one worker | 23:45 |
| *** sabeen1 has joined #openstack-infra | 23:46 | |
| pabelanger | greghaynes: we are not defining any workers, so its the default right now | 23:46 |
| pabelanger | not sure what that is | 23:46 |
| pabelanger | googling | 23:46 |
| jeblair | workers = 2 | 23:47 |
| jeblair | /usr/share/uwsgi/conf/default.ini | 23:47 |
| pabelanger | Yup, just seen that | 23:47 |
| *** hashar has quit IRC | 23:47 | |
| *** rfolco has joined #openstack-infra | 23:48 | |
| pabelanger | Oh, workers and processes are the same thing? | 23:48 |
| *** ddieterly has quit IRC | 23:49 | |
| greghaynes | I actually am not entirely sure in uwsgi land, I think no but need to read | 23:50 |
| jeblair | perhaps workers=threads*processes? | 23:50 |
| pabelanger | Ya, some confusing stuff on google :( | 23:50 |
| *** sabeen1 has quit IRC | 23:50 | |
| pabelanger | maybe | 23:50 |
| *** sfinucan has quit IRC | 23:50 | |
| *** sabeen1 has joined #openstack-infra | 23:50 | |
| greghaynes | jeblair: That is what I would think - worker is a uwsgi worker which can spawn N processes | 23:51 |
| * greghaynes comes from gunicorn land which is similar but not identical | 23:51 | |
| *** dingyichen has joined #openstack-infra | 23:51 | |
| *** rfolco has quit IRC | 23:52 | |
| greghaynes | actually | 23:52 |
| greghaynes | I think they are synonyms | 23:52 |
| greghaynes | http://uwsgi-docs.readthedocs.org/en/latest/Options.html#processes | 23:52 |
| *** mestery has joined #openstack-infra | 23:52 | |
| *** jamesmcarthur has quit IRC | 23:52 | |
| *** tnarg has joined #openstack-infra | 23:52 | |
| *** mriedem is now known as mriedem_away | 23:54 | |
| pabelanger | okay, uwsgi on local nginx just reloaded data from memory. That was about 1 hour, but will try again | 23:54 |
| pabelanger | then timed out | 23:54 |
| pabelanger | jeblair: stackalytics-processor should be starting in 5mins | 23:55 |
| *** sputnik13 has joined #openstack-infra | 23:55 | |
| greghaynes | pabelanger: where did you read about the once per hour reloading? | 23:56 |
| openstackgerrit | vinay yadhav proposed openstack-infra/irc-meetings: Adding Tap as a Service meeting details https://review.openstack.org/243355 | 23:56 |
| pabelanger | greghaynes: If I am reading correctly: https://github.com/openstack/stackalytics/blob/master/stackalytics/dashboard/vault.py#L93 | 23:56 |
| *** akshai has quit IRC | 23:57 | |
| pabelanger | dashboard_update_interval = 3600 by default | 23:57 |
| *** otter768 has quit IRC | 23:57 | |
| *** Sukhdev has quit IRC | 23:58 | |
| *** boris-42 has quit IRC | 23:58 | |
| *** tiswanso has joined #openstack-infra | 23:59 | |
| greghaynes | wow | 23:59 |
| *** tnarg has quit IRC | 23:59 | |
| greghaynes | so, about once an hour someone is always going to get unlucky | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!