| *** nelsnels_ has joined #openstack-infra | 00:06 | |
| *** Qiming has quit IRC | 00:07 | |
| *** nelsnelson has quit IRC | 00:07 | |
| *** _amrith_ is now known as amrith | 00:18 | |
| openstackgerrit | Jamie Lennox proposed openstack/requirements: Allow maintaining extras from project https://review.openstack.org/303271 | 00:20 |
|---|---|---|
| *** sflanigan has joined #openstack-infra | 00:22 | |
| *** chlong has quit IRC | 00:27 | |
| *** dchen has joined #openstack-infra | 00:28 | |
| *** ayoung has joined #openstack-infra | 00:32 | |
| *** dchen has quit IRC | 00:33 | |
| *** chlong has joined #openstack-infra | 00:34 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Remove DB existence check from open session termination https://review.openstack.org/303615 | 00:36 |
| *** tiswanso has joined #openstack-infra | 00:45 | |
| *** gongysh has quit IRC | 00:47 | |
| *** tiswanso has quit IRC | 00:49 | |
| *** zz_dimtruck is now known as dimtruck | 00:50 | |
| *** mriedem has quit IRC | 00:50 | |
| *** nelsnelson has joined #openstack-infra | 00:51 | |
| *** nelsnels_ has quit IRC | 00:54 | |
| Nakato | lifeless: Do you have any context on propose-reqs being disabled on stable branches? https://review.openstack.org/#/c/297225/ | 01:03 |
| *** Qiming has joined #openstack-infra | 01:06 | |
| *** julim has quit IRC | 01:09 | |
| *** julim has joined #openstack-infra | 01:11 | |
| *** tiswanso has joined #openstack-infra | 01:15 | |
| *** mylu has quit IRC | 01:15 | |
| lifeless | Nakato: yes; it pulls in new releases of openstack components | 01:15 |
| lifeless | Nakato: and without an actual backwards compat policy, this is a problem | 01:16 |
| lifeless | Nakato: all the non-openstack components tend to be more careful about backwards compat | 01:16 |
| *** tiswanso has quit IRC | 01:16 | |
| lifeless | Nakato: but | 01:16 |
| lifeless | Nakato: unless we state that e.g. mitaka libs should work to run a liberty cloud, its clearly wrong to do that | 01:16 |
| *** tiswanso has joined #openstack-infra | 01:17 | |
| *** ramishra has joined #openstack-infra | 01:17 | |
| *** mylu has joined #openstack-infra | 01:17 | |
| Nakato | lifeless: So, should we aim for a backwards compat policy, or aim to enhance that job to keep stable versions pinned? | 01:19 |
| *** gongysh has joined #openstack-infra | 01:19 | |
| lifeless | Nakato: I have a backwards compat policy proposal drafted | 01:20 |
| lifeless | Nakato: mriedm and I plan to go through it face to face in atlanta | 01:20 |
| lifeless | since we're failing to get consensus via deuelling review comments | 01:20 |
| lifeless | Nakato: in terms of stability, not modifying anything is actually fairly good for released code | 01:20 |
| *** dimtruck is now known as zz_dimtruck | 01:23 | |
| *** krtaylor has quit IRC | 01:32 | |
| *** thorst has joined #openstack-infra | 01:35 | |
| *** amotoki has joined #openstack-infra | 01:38 | |
| Nakato | lifeless: Thanks. | 01:39 |
| *** thorst has quit IRC | 01:39 | |
| *** claudiub has quit IRC | 01:40 | |
| *** gildub has joined #openstack-infra | 01:41 | |
| lifeless | Nakato: collaborating on that discussion would be great | 01:44 |
| *** achanda has joined #openstack-infra | 01:45 | |
| Nakato | lifeless: Could you fling me the links? | 01:45 |
| lifeless | https://review.openstack.org/#/c/226157/ | 01:46 |
| *** amotoki has quit IRC | 02:00 | |
| *** mylu has quit IRC | 02:01 | |
| *** Jeffrey4l_ has joined #openstack-infra | 02:02 | |
| *** thorst has joined #openstack-infra | 02:04 | |
| *** Jeffrey4l_ has quit IRC | 02:08 | |
| *** kengo_sakai has joined #openstack-infra | 02:10 | |
| *** amotoki has joined #openstack-infra | 02:10 | |
| *** kengo_sakai has quit IRC | 02:11 | |
| *** shu-mutou-AFK is now known as shu-mutou | 02:11 | |
| *** kengo_sakai has joined #openstack-infra | 02:11 | |
| *** nhandler has quit IRC | 02:14 | |
| *** tiswanso has quit IRC | 02:16 | |
| openstackgerrit | Sachi King proposed openstack-infra/project-config: Update stable constraints url https://review.openstack.org/267941 | 02:17 |
| *** ramishra has quit IRC | 02:21 | |
| *** ramishra has joined #openstack-infra | 02:21 | |
| *** Jeffrey4l_ has joined #openstack-infra | 02:22 | |
| *** sdake has joined #openstack-infra | 02:24 | |
| *** knikolla has quit IRC | 02:24 | |
| *** mylu has joined #openstack-infra | 02:24 | |
| *** thorst has quit IRC | 02:24 | |
| *** nhandler has joined #openstack-infra | 02:27 | |
| *** sdake_ has joined #openstack-infra | 02:27 | |
| *** sdake has quit IRC | 02:28 | |
| *** beisner has quit IRC | 02:30 | |
| *** beisner has joined #openstack-infra | 02:31 | |
| *** asettle has quit IRC | 02:33 | |
| *** kengo_sa_ has joined #openstack-infra | 02:34 | |
| *** kengo_sakai has quit IRC | 02:35 | |
| *** sflanigan has quit IRC | 02:37 | |
| *** sdake has joined #openstack-infra | 02:39 | |
| *** sdake_ has quit IRC | 02:43 | |
| *** salv-orlando has joined #openstack-infra | 02:45 | |
| *** rkukura has joined #openstack-infra | 02:45 | |
| *** yuanying has quit IRC | 02:46 | |
| *** akshai has quit IRC | 02:46 | |
| *** salv-orl_ has quit IRC | 02:47 | |
| *** FallenPegasus has joined #openstack-infra | 02:59 | |
| *** achanda has quit IRC | 02:59 | |
| *** mylu has quit IRC | 03:09 | |
| *** bkeller` has joined #openstack-infra | 03:10 | |
| *** Sukhdev has joined #openstack-infra | 03:10 | |
| *** mfisch has quit IRC | 03:11 | |
| *** mylu has joined #openstack-infra | 03:12 | |
| *** Guest89657 has joined #openstack-infra | 03:12 | |
| *** bkeller` has quit IRC | 03:12 | |
| *** yamahata has joined #openstack-infra | 03:13 | |
| *** Guest89657 has quit IRC | 03:13 | |
| *** Guest89657 has joined #openstack-infra | 03:13 | |
| *** Guest89657 is now known as mfisch | 03:14 | |
| *** rkukura has quit IRC | 03:14 | |
| *** sputnik13 has joined #openstack-infra | 03:19 | |
| lifeless | fungi: could you comment on https://review.openstack.org/#/c/303054/1 please | 03:21 |
| *** zhurong has joined #openstack-infra | 03:25 | |
| *** harlowja_at_home has joined #openstack-infra | 03:32 | |
| ianw | hmm, not sure if it's local, but git in my local image builds is hanging a lot | 03:33 |
| ianw | TCP 192.168.0.14:47098->104.130.246.128:git (ESTABLISHED) | 03:34 |
| *** mylu has quit IRC | 03:35 | |
| *** sputnik13 has quit IRC | 03:36 | |
| *** rkukura has joined #openstack-infra | 03:37 | |
| *** mylu has joined #openstack-infra | 03:37 | |
| *** rkukura has quit IRC | 03:41 | |
| *** abregman has quit IRC | 03:41 | |
| *** achanda has joined #openstack-infra | 03:44 | |
| *** zz_dimtruck is now known as dimtruck | 03:45 | |
| *** links has joined #openstack-infra | 03:45 | |
| *** achanda has quit IRC | 03:46 | |
| *** rkukura has joined #openstack-infra | 03:47 | |
| *** yuanying has joined #openstack-infra | 03:49 | |
| *** Nikolay_St has quit IRC | 03:57 | |
| *** FallenPegasus has quit IRC | 03:58 | |
| *** treaki_ has joined #openstack-infra | 03:59 | |
| prometheanfire | ianw: hi :P | 04:01 |
| *** harlowja_at_home has quit IRC | 04:02 | |
| *** treaki has quit IRC | 04:02 | |
| *** jamesmcarthur has joined #openstack-infra | 04:04 | |
| *** jamesmcarthur has quit IRC | 04:08 | |
| *** oanson has joined #openstack-infra | 04:10 | |
| *** zhurong has quit IRC | 04:15 | |
| ianw | ok, maybe it was local | 04:16 |
| *** achanda has joined #openstack-infra | 04:18 | |
| *** kengo_sakai_mba has joined #openstack-infra | 04:21 | |
| *** zhurong has joined #openstack-infra | 04:21 | |
| *** amotoki has quit IRC | 04:22 | |
| *** amotoki has joined #openstack-infra | 04:24 | |
| *** thorst has joined #openstack-infra | 04:25 | |
| *** thorst has quit IRC | 04:29 | |
| *** mylu has quit IRC | 04:30 | |
| *** kengo_sakai_mba has quit IRC | 04:31 | |
| *** mylu has joined #openstack-infra | 04:32 | |
| *** zhurong has quit IRC | 04:33 | |
| *** Nikolay_St has joined #openstack-infra | 04:38 | |
| *** gongysh has quit IRC | 04:38 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Upload the ironic-python-agent images to cache https://review.openstack.org/301698 | 04:39 |
| *** jaosorior has joined #openstack-infra | 04:46 | |
| *** kengo_sakai_mba has joined #openstack-infra | 04:58 | |
| *** sflanigan has joined #openstack-infra | 05:00 | |
| *** sflanigan has joined #openstack-infra | 05:00 | |
| *** ramishra has quit IRC | 05:04 | |
| *** ramishra has joined #openstack-infra | 05:04 | |
| *** vmbrasseur has left #openstack-infra | 05:05 | |
| *** abregman has joined #openstack-infra | 05:08 | |
| *** grue_pm has joined #openstack-infra | 05:15 | |
| *** grue_pm has quit IRC | 05:16 | |
| *** sdake_ has joined #openstack-infra | 05:23 | |
| *** sdake has quit IRC | 05:24 | |
| *** akaszuba has joined #openstack-infra | 05:27 | |
| *** asettle has joined #openstack-infra | 05:28 | |
| *** claudiub has joined #openstack-infra | 05:29 | |
| *** hdd has joined #openstack-infra | 05:31 | |
| *** Nikolay_St has quit IRC | 05:36 | |
| *** mylu has quit IRC | 05:36 | |
| *** kushal has joined #openstack-infra | 05:39 | |
| *** pcaruana has joined #openstack-infra | 05:39 | |
| *** gongysh has joined #openstack-infra | 05:40 | |
| *** dimsum__ has quit IRC | 05:40 | |
| *** achanda has quit IRC | 05:43 | |
| *** sdake_ is now known as sdake | 05:45 | |
| *** dims has joined #openstack-infra | 05:47 | |
| *** kengo_sakai_mba has quit IRC | 05:49 | |
| openstackgerrit | Merged openstack-infra/shade: Use event_utils.poll_for_events for stack polling https://review.openstack.org/292591 | 05:52 |
| *** dimtruck is now known as zz_dimtruck | 05:54 | |
| openstackgerrit | Merged openstack-infra/shade: Wrap stack operations in a heat_exceptions https://review.openstack.org/292592 | 05:54 |
| *** tricksters has joined #openstack-infra | 05:56 | |
| openstackgerrit | Merged openstack-infra/shade: Delete image objects after failed upload https://review.openstack.org/303201 | 05:57 |
| openstackgerrit | Merged openstack-infra/shade: Add release note about the swift Large Object changes https://review.openstack.org/303712 | 05:57 |
| *** eric_lopez has quit IRC | 05:58 | |
| *** oanson has quit IRC | 06:01 | |
| *** zeih has joined #openstack-infra | 06:03 | |
| *** tphummel has joined #openstack-infra | 06:11 | |
| *** hrubi has joined #openstack-infra | 06:12 | |
| openstackgerrit | Merged openstack/diskimage-builder: Turn down tracing for source-repo cache https://review.openstack.org/295569 | 06:12 |
| *** chlong has quit IRC | 06:13 | |
| *** achanda has joined #openstack-infra | 06:15 | |
| *** oanson has joined #openstack-infra | 06:15 | |
| *** rcernin has joined #openstack-infra | 06:15 | |
| *** achanda has quit IRC | 06:20 | |
| yolanda | good morning | 06:22 |
| openstackgerrit | Steve Kowalik proposed openstack-infra/system-config: Switch to new release of puppet-mysql https://review.openstack.org/303916 | 06:25 |
| *** abregman has quit IRC | 06:28 | |
| AJaeger | morning, yolanda ! | 06:29 |
| *** abregman has joined #openstack-infra | 06:30 | |
| *** scheuran has joined #openstack-infra | 06:30 | |
| *** dims has quit IRC | 06:31 | |
| *** tesseract has joined #openstack-infra | 06:35 | |
| *** tphummel has quit IRC | 06:35 | |
| *** tesseract is now known as Guest8989 | 06:35 | |
| *** roxanaghe has joined #openstack-infra | 06:37 | |
| openstackgerrit | Ryota MIBU proposed openstack-infra/project-config: [ceilometer] enable ceilo tempest plugin tests https://review.openstack.org/303921 | 06:38 |
| *** jtomasek has joined #openstack-infra | 06:39 | |
| *** roxanaghe has quit IRC | 06:41 | |
| *** achanda has joined #openstack-infra | 06:44 | |
| *** fedexo has quit IRC | 06:44 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Create Merger Git repos using connection names https://review.openstack.org/282660 | 06:45 |
| *** Hal has joined #openstack-infra | 06:45 | |
| *** Hal is now known as Guest19614 | 06:46 | |
| *** gildub has quit IRC | 06:48 | |
| *** sputnik13 has joined #openstack-infra | 06:49 | |
| *** sputnik13 has joined #openstack-infra | 06:49 | |
| *** daemontool has joined #openstack-infra | 06:53 | |
| *** dims has joined #openstack-infra | 06:54 | |
| *** esikachev has joined #openstack-infra | 06:55 | |
| *** kengo_sakai_mba has joined #openstack-infra | 06:56 | |
| *** sputnik13 has quit IRC | 06:56 | |
| *** kushal has quit IRC | 06:56 | |
| *** dims has quit IRC | 06:58 | |
| *** arxcruz has joined #openstack-infra | 07:01 | |
| *** markvoelker has joined #openstack-infra | 07:01 | |
| *** ihrachys has joined #openstack-infra | 07:03 | |
| *** jaosorior has quit IRC | 07:03 | |
| *** jaosorior has joined #openstack-infra | 07:04 | |
| *** kengo_sakai_mba has quit IRC | 07:04 | |
| *** markvoelker has quit IRC | 07:06 | |
| *** dtardivel has joined #openstack-infra | 07:07 | |
| *** shashank_hegde has quit IRC | 07:08 | |
| *** mikelk has joined #openstack-infra | 07:11 | |
| *** jamielennox is now known as jamielennox|away | 07:15 | |
| *** vincentll has joined #openstack-infra | 07:17 | |
| openstackgerrit | Dong Ma proposed openstack-infra/subunit2sql: Fix get_test_metadata to use test_id https://review.openstack.org/303928 | 07:17 |
| *** jed56 has joined #openstack-infra | 07:21 | |
| *** fhubik has joined #openstack-infra | 07:24 | |
| *** placko_ has joined #openstack-infra | 07:26 | |
| *** Sukhdev has quit IRC | 07:28 | |
| *** chlong has joined #openstack-infra | 07:31 | |
| *** nmagnezi has joined #openstack-infra | 07:31 | |
| *** fhubik has quit IRC | 07:34 | |
| *** shardy has joined #openstack-infra | 07:34 | |
| *** jlanoux has joined #openstack-infra | 07:35 | |
| yolanda | AJaeger,thx for the change to increase heap size on gerrit, i missed the conversation yesterday | 07:41 |
| *** dims has joined #openstack-infra | 07:45 | |
| *** kushal has joined #openstack-infra | 07:45 | |
| *** jaosorior has quit IRC | 07:47 | |
| *** jaosorior has joined #openstack-infra | 07:48 | |
| openstackgerrit | Dong Ma proposed openstack-infra/subunit2sql: Fix get_test_metadata to use test_id https://review.openstack.org/303928 | 07:49 |
| *** jaosorior has joined #openstack-infra | 07:49 | |
| *** kairat has joined #openstack-infra | 07:50 | |
| *** achanda has quit IRC | 07:56 | |
| kairat | AJaeger, Hi, do you know if reno cache release notes somewhere? | 07:57 |
| *** pblaho has joined #openstack-infra | 07:57 | |
| *** chem has joined #openstack-infra | 07:57 | |
| *** e0ne has joined #openstack-infra | 08:02 | |
| AJaeger | kairat: what do you mean with cache? | 08:04 |
| AJaeger | yolanda: you're welcome! Just was briefly online when that discussion happened... | 08:04 |
| kairat | I am trying to understand why note broke translation but it seems that changes in release note doesn't affect tests | 08:04 |
| kairat | so i am trying fix the test | 08:04 |
| AJaeger | kairat: the glance_store problem? | 08:04 |
| kairat | but when I execute tox -e releasenotes again | 08:04 |
| kairat | yep | 08:04 |
| kairat | it somehow uses old release note definition | 08:05 |
| kairat | You can try to reproduce it on your local env | 08:05 |
| kairat | just change the release note that breaks the gate | 08:06 |
| AJaeger | thanks for digging into this. Sphinx does some caching, so thta might be involved here. | 08:06 |
| kairat | and you will see that error is at the same place | 08:06 |
| AJaeger | But in the gates we always start with clean workspace, so there is no caching involved | 08:06 |
| kairat | Yeah | 08:06 |
| AJaeger | I reproduced it locally - that's why I send the email. | 08:06 |
| kairat | I am trying to find the root cause | 08:07 |
| kairat | So i am interesting in cleaning that cache | 08:07 |
| AJaeger | give me a sec, I need to double check something | 08:08 |
| kairat | Perhaps, you know how to do that cause I am not familiar with sphinx | 08:08 |
| kairat | ok | 08:08 |
| openstackgerrit | Ryota MIBU proposed openstack-infra/project-config: [telemetry] enable ceilo/aodh tempest plugin tests https://review.openstack.org/303921 | 08:08 |
| *** asettle has quit IRC | 08:09 | |
| AJaeger | kairat: releasenotes/build/.doctrees is the cache, just rm -rf it | 08:09 |
| *** yamahata has quit IRC | 08:10 | |
| *** BobBall_AWOL is now known as BobBall | 08:11 | |
| AJaeger | kairat: to me it looks like the mitaka releasenotes are broken - and I wonder how that could have passed the gates. You might need to discuss with dhellmann how to avoid this in the future... | 08:11 |
| *** BobBall is now known as Guest80100 | 08:11 | |
| kairat | AJaeger, I tried that already | 08:11 |
| kairat | With no effect | 08:12 |
| kairat | I also cleaned releasenotes env in .tox | 08:12 |
| AJaeger | kairat: for me it fails everytime reruning tox - no need to do anything | 08:12 |
| *** jordanP has joined #openstack-infra | 08:12 | |
| kairat | Can you update release not | 08:13 |
| kairat | will error message be changed after that? | 08:13 |
| kairat | for example you can insert several additional strings | 08:13 |
| AJaeger | Yes, you should be able to update the release notes but since reno parses the git history I'm not sure whether testing will get this really. Best talk to dhellmann ... | 08:15 |
| kairat | Ok, thanks | 08:16 |
| kairat | Actuallym that was my change | 08:16 |
| kairat | so I am interesting to fix that ASAP | 08:16 |
| kairat | And understand what was the trouble | 08:16 |
| AJaeger | Ah ;) | 08:17 |
| AJaeger | kairat: I'm interested mainly to learn why the gates did not complain earlier - why it was possible to merge this at all... | 08:18 |
| *** dingyichen has quit IRC | 08:22 | |
| *** achanda has joined #openstack-infra | 08:22 | |
| *** Guest80100 is now known as BobBall^ | 08:22 | |
| *** yamamoto has joined #openstack-infra | 08:24 | |
| *** roxanaghe has joined #openstack-infra | 08:25 | |
| *** vgridnev has joined #openstack-infra | 08:25 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 08:25 |
| *** bauwser is now known as bauzas | 08:29 | |
| *** roxanaghe has quit IRC | 08:29 | |
| AJaeger | kairat: Ah, I see - glance_store has no releasenotes jobs defined at all. Let me add them now... | 08:31 |
| openstackgerrit | Merged openstack/diskimage-builder: Debian: dont set always the hostname to debian https://review.openstack.org/205047 | 08:31 |
| *** jistr has joined #openstack-infra | 08:32 | |
| *** vikram_ has joined #openstack-infra | 08:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add release note publishing to glance_store https://review.openstack.org/303962 | 08:33 |
| vikram_ | can someone plz help in approving https://review.openstack.org/#/c/302120/ | 08:33 |
| AJaeger | kairat, nikhil_k, flaper87: See 303962, please | 08:33 |
| *** derekh has joined #openstack-infra | 08:33 | |
| AJaeger | vikram_: please describe your change so that the right people can review it. | 08:34 |
| vikram_ | AJaeger: The change is about creating a new project for Huawei's ML2 plugin code for interacting with neutron | 08:35 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 08:36 |
| AJaeger | vikram_: LGTM, approved | 08:37 |
| vikram_ | AJaeger: Thanks ;) | 08:37 |
| *** jlanoux has quit IRC | 08:38 | |
| vikram_ | AJaeger: One query.. when the new repo will be up | 08:38 |
| *** jlanoux has joined #openstack-infra | 08:38 | |
| AJaeger | vikram_: this is all driven via puppet, expect it to be up 1h after the change is merged | 08:39 |
| *** ramishra has quit IRC | 08:39 | |
| vikram_ | AJaeger: Ok | 08:39 |
| AJaeger | vikram_: and then ask here for an admin to add you to ACLs - note I'm not an admin and cannot do it. | 08:39 |
| vikram_ | AJaeger: Ok.. After the project is created will do that.. | 08:40 |
| AJaeger | kairat: since there're no gate jobs for releasenotes yet, it's no wonder that this could go in as is. | 08:40 |
| AJaeger | kairat: But that also means you can easily fix it - and then we enable the gate jobs ;) | 08:40 |
| *** mrmartin has joined #openstack-infra | 08:43 | |
| *** salv-orl_ has joined #openstack-infra | 08:44 | |
| *** salv-orlando has quit IRC | 08:47 | |
| *** dims has quit IRC | 08:48 | |
| openstackgerrit | Lenny Verkhovsky proposed openstack-infra/project-config-example: First commit. README update https://review.openstack.org/303970 | 08:51 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors https://review.openstack.org/302192 | 08:52 |
| openstackgerrit | Lenny Verkhovsky proposed openstack-infra/project-config-example: First commit. README update https://review.openstack.org/303970 | 08:53 |
| *** skolekonov|away is now known as skolekonov | 08:55 | |
| *** dims has joined #openstack-infra | 08:55 | |
| *** pcaruana has quit IRC | 09:00 | |
| openstackgerrit | Thomas Herve proposed openstack-dev/pbr: Fix wsgiref script use with oslo.config https://review.openstack.org/296061 | 09:01 |
| *** zhurong has joined #openstack-infra | 09:02 | |
| *** markvoelker has joined #openstack-infra | 09:02 | |
| *** nadya has joined #openstack-infra | 09:02 | |
| *** rossella_s has quit IRC | 09:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Adding networking-huawei project. https://review.openstack.org/302120 | 09:03 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 09:03 |
| *** rossella_s has joined #openstack-infra | 09:03 | |
| *** ociuhandu has quit IRC | 09:04 | |
| *** abregman has quit IRC | 09:05 | |
| *** markvoelker has quit IRC | 09:07 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Added nodepool::mysql class https://review.openstack.org/299661 | 09:09 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors https://review.openstack.org/302192 | 09:10 |
| *** sambetts|afk is now known as sambetts | 09:12 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Added nodepool::mysql class https://review.openstack.org/299661 | 09:14 |
| *** ramishra has joined #openstack-infra | 09:16 | |
| *** ramishra has quit IRC | 09:16 | |
| *** abregman has joined #openstack-infra | 09:18 | |
| *** zhurong_ has joined #openstack-infra | 09:18 | |
| *** Nikolay_St has joined #openstack-infra | 09:21 | |
| *** ramishra has joined #openstack-infra | 09:21 | |
| *** zhurong has quit IRC | 09:22 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors https://review.openstack.org/302192 | 09:23 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 09:23 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Update delete flavor test to work with betamax https://review.openstack.org/303459 | 09:27 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 09:27 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/project-config: Add os-{refresh,collect,net}-config projects https://review.openstack.org/303984 | 09:29 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add new test for deleting non-existing flavors https://review.openstack.org/303515 | 09:36 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax https://review.openstack.org/303529 | 09:37 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax https://review.openstack.org/303529 | 09:39 |
| *** briancline has quit IRC | 09:40 | |
| *** briancline has joined #openstack-infra | 09:42 | |
| openstackgerrit | Giulio Fidente proposed openstack/requirements: Bump os-net-config to 0.2.4 https://review.openstack.org/303987 | 09:42 |
| *** ociuhandu has joined #openstack-infra | 09:43 | |
| esikachev | yolanda: hi! take a look on https://review.openstack.org/#/c/303475/ what did you think? | 09:43 |
| yolanda | ah, as I saw the -1 from Khai i didn't review sorry. Taking a look now | 09:44 |
| yolanda | +2 | 09:46 |
| esikachev | ok, thanks | 09:46 |
| *** gfidente has joined #openstack-infra | 09:46 | |
| *** ociuhandu has quit IRC | 09:47 | |
| gfidente | hi, do you guys know the nick for Andreas Jaeger ? | 09:48 |
| *** kairat has left #openstack-infra | 09:48 | |
| gfidente | ping AJaeger | 09:49 |
| AJaeger | gfidente: You guessed correctly ;) | 09:52 |
| AJaeger | gfidente: how can I help? | 09:52 |
| gfidente | AJaeger sorry but it was my first attempt | 09:53 |
| gfidente | thanks for reviewing | 09:53 |
| gfidente | so I think what I wanted is to map os-net-config to its separate ACL file which I am adding | 09:53 |
| gfidente | and to update the mappings for a couple of those projects, let me try another update | 09:54 |
| AJaeger | gfidente: *why* do you want to do this? | 09:55 |
| gfidente | because we wanted to split up permissions for os-net-config | 09:55 |
| *** tosky has joined #openstack-infra | 09:55 | |
| AJaeger | gfidente: so, have a separate group for os-net-config? Ok, then add that information to the commit message, please | 09:56 |
| gfidente | I think initially though it'll be tripleo-core anyway | 09:56 |
| BobBall^ | So it seems that libvirt-python now depends on having libvirt installed in order to install itself as of a few days ago... libvirt-python is in global requirements, but possibly should not be since it is not requried for non-libvirt environments. How do we normally handle this? | 09:57 |
| AJaeger | gfidente: all of you three new files have the same content. | 09:57 |
| gfidente | AJaeger yeah I removed those for os-collect and os-refresh | 09:57 |
| AJaeger | That's something to avoid, just add it once and reference it. | 09:57 |
| AJaeger | gfidente: ok | 09:57 |
| gfidente | AJaeger yeah thanks | 09:57 |
| AJaeger | gfidente: looking forward to the next iteration ;) | 09:57 |
| gfidente | also I noticed os-collect-config is missing the reference to os-apply-config.config ? | 09:58 |
| AJaeger | gfidente: indeed, that looks wrong ;( | 09:58 |
| openstackgerrit | Merged openstack/requirements: Add Windows as supported operating system https://review.openstack.org/302945 | 09:58 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/project-config: Add ACLs for os-net-config https://review.openstack.org/303984 | 10:00 |
| gfidente | AJaeger thanks for helping, hope it looks better now | 10:01 |
| AJaeger | gfidente: gerrit/acls/openstack/tripleo.config is the same as your file, isn't it? | 10:01 |
| AJaeger | gfidente: Please don't duplicate ;) | 10:01 |
| *** kengo_sakai_mba has joined #openstack-infra | 10:02 | |
| gfidente | AJaeger yes it is but that's the purpose of the split | 10:02 |
| gfidente | though until we have a split to do it's not worth it | 10:02 |
| gfidente | so I'll step back on that for now | 10:02 |
| AJaeger | gfidente: You can either create a new group - or reference the acl file... | 10:02 |
| AJaeger | gfidente: but send the fix for os-collect-config, please | 10:03 |
| AJaeger | bbl | 10:03 |
| *** cdent has joined #openstack-infra | 10:03 | |
| *** thorst has joined #openstack-infra | 10:04 | |
| *** ociuhandu has joined #openstack-infra | 10:06 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/project-config: Add missing acl-config reference for os-collect-config to os-apply-config.config https://review.openstack.org/303984 | 10:06 |
| *** roxanaghe has joined #openstack-infra | 10:13 | |
| *** kengo_sakai_mba has quit IRC | 10:14 | |
| *** ams_ has joined #openstack-infra | 10:16 | |
| *** roxanaghe has quit IRC | 10:18 | |
| *** Guest19614 has quit IRC | 10:19 | |
| *** ramishra has quit IRC | 10:19 | |
| *** kengo_sakai_mba has joined #openstack-infra | 10:20 | |
| *** Qiming has quit IRC | 10:20 | |
| *** ramishra has joined #openstack-infra | 10:24 | |
| *** briancline has quit IRC | 10:25 | |
| *** briancline has joined #openstack-infra | 10:25 | |
| openstackgerrit | Alexey Stepanov proposed openstack-infra/project-config: fuel-qa: Add vote to pylint checks https://review.openstack.org/304017 | 10:25 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in tripleo.sh for cleanup_pingtest method https://review.openstack.org/304020 | 10:26 |
| *** thorst has quit IRC | 10:27 | |
| *** thorst has joined #openstack-infra | 10:27 | |
| *** briancline has quit IRC | 10:30 | |
| *** briancline has joined #openstack-infra | 10:31 | |
| *** sdague has joined #openstack-infra | 10:31 | |
| *** thorst has quit IRC | 10:32 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in tripleo.sh for cleanup_pingtest method https://review.openstack.org/304020 | 10:32 |
| *** penguinolog has joined #openstack-infra | 10:36 | |
| penguinolog | Hello! Review, plz: https://review.openstack.org/#/c/304017/ | 10:37 |
| *** vikram_ has quit IRC | 10:45 | |
| *** sdake_ has joined #openstack-infra | 10:45 | |
| *** thorst has joined #openstack-infra | 10:46 | |
| *** jcoufal has joined #openstack-infra | 10:48 | |
| *** sdake has quit IRC | 10:48 | |
| *** kushal has quit IRC | 10:50 | |
| *** ifarkas has joined #openstack-infra | 10:50 | |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Make gate-trove-python34-db voting; add gate job https://review.openstack.org/304030 | 10:56 |
| *** jaypipes has joined #openstack-infra | 10:57 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax https://review.openstack.org/303529 | 10:59 |
| *** markvoelker has joined #openstack-infra | 11:03 | |
| *** maishsk has joined #openstack-infra | 11:03 | |
| *** zhurong_ has quit IRC | 11:04 | |
| *** thorst has quit IRC | 11:04 | |
| *** thorst has joined #openstack-infra | 11:05 | |
| *** bkeller` has joined #openstack-infra | 11:05 | |
| *** mat128_ is now known as mat128 | 11:06 | |
| *** markvoelker has quit IRC | 11:07 | |
| *** nadya has quit IRC | 11:09 | |
| *** thorst has quit IRC | 11:09 | |
| *** ramishra has quit IRC | 11:16 | |
| *** tpsilva has joined #openstack-infra | 11:18 | |
| *** kushal has joined #openstack-infra | 11:18 | |
| *** esikachev has quit IRC | 11:20 | |
| *** vtapia_ has quit IRC | 11:20 | |
| *** vtapia_ has joined #openstack-infra | 11:21 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-os_swift_sync repository https://review.openstack.org/304049 | 11:21 |
| *** bkeller` has quit IRC | 11:22 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 11:24 |
| *** Goneri has joined #openstack-infra | 11:24 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors https://review.openstack.org/302192 | 11:27 |
| *** ramishra has joined #openstack-infra | 11:27 | |
| Zara | fungi: a while ago you mentioned that we had a limited time to improve the storyboard email setup before it stopped working; I can't remember the details (I think to do with how we were handling bounces?) do you have a moment to go through it with me again? worried since bugsquash is coming up... | 11:28 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Update delete flavor test to work with betamax https://review.openstack.org/303459 | 11:29 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add new test for deleting non-existing flavors https://review.openstack.org/303515 | 11:30 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax https://review.openstack.org/303529 | 11:31 |
| *** esikachev has joined #openstack-infra | 11:31 | |
| *** weshay has quit IRC | 11:36 | |
| *** ldnunes has joined #openstack-infra | 11:37 | |
| *** cdent has quit IRC | 11:40 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Update test for setting flavor specs to use betamax https://review.openstack.org/304064 | 11:41 |
| *** gordc has joined #openstack-infra | 11:42 | |
| *** Qiming has joined #openstack-infra | 11:43 | |
| *** binbincong has quit IRC | 11:43 | |
| *** ramishra has quit IRC | 11:43 | |
| *** yamamoto has quit IRC | 11:46 | |
| *** vtapia_ has quit IRC | 11:46 | |
| *** nadya has joined #openstack-infra | 11:47 | |
| *** esikachev has quit IRC | 11:47 | |
| *** vtapia_ has joined #openstack-infra | 11:48 | |
| *** camunoz_ has joined #openstack-infra | 11:49 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Update test for setting flavor specs to use betamax https://review.openstack.org/304064 | 11:49 |
| *** baoli_ has joined #openstack-infra | 11:50 | |
| *** flaper87 has quit IRC | 11:51 | |
| *** flaper87 has joined #openstack-infra | 11:51 | |
| *** sshnaidm has quit IRC | 11:51 | |
| *** flepied1 has joined #openstack-infra | 11:51 | |
| *** baoli has quit IRC | 11:53 | |
| *** esikachev has joined #openstack-infra | 11:53 | |
| *** flepied has quit IRC | 11:54 | |
| *** binbincong has joined #openstack-infra | 11:56 | |
| *** sshnaidm has joined #openstack-infra | 11:57 | |
| openstackgerrit | Aparna proposed openstack/diskimage-builder: Support to add certificate in ironic-agent https://review.openstack.org/295203 | 11:58 |
| *** amotoki has quit IRC | 11:58 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Update test_unset_flavor_specs to use betamax https://review.openstack.org/304073 | 11:59 |
| *** sdake_ is now known as sdake | 11:59 | |
| *** roxanaghe has joined #openstack-infra | 12:01 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in tripleo.sh (Old heat * calls) https://review.openstack.org/301087 | 12:03 |
| *** thorst has joined #openstack-infra | 12:04 | |
| *** icey has joined #openstack-infra | 12:04 | |
| *** xiangxinyong has quit IRC | 12:04 | |
| *** eliqiao has quit IRC | 12:05 | |
| AJaeger | odyssey4me: are you around for your retirement change? | 12:06 |
| *** eliqiao has joined #openstack-infra | 12:06 | |
| *** roxanaghe has quit IRC | 12:06 | |
| *** xiangxinyong has joined #openstack-infra | 12:06 | |
| *** icey has quit IRC | 12:06 | |
| *** esikachev has quit IRC | 12:07 | |
| *** cdent has joined #openstack-infra | 12:07 | |
| AJaeger | odyssey4me: commented on review | 12:07 |
| *** aysyd has joined #openstack-infra | 12:09 | |
| *** amotoki has joined #openstack-infra | 12:09 | |
| *** gongysh has quit IRC | 12:09 | |
| *** sdake_ has joined #openstack-infra | 12:10 | |
| openstackgerrit | Aparna proposed openstack/diskimage-builder: Support to add certificate in ironic-agent https://review.openstack.org/295203 | 12:10 |
| *** icey has joined #openstack-infra | 12:11 | |
| *** icey has quit IRC | 12:11 | |
| *** sdake has quit IRC | 12:12 | |
| *** rodrigods has quit IRC | 12:13 | |
| *** aysyd has quit IRC | 12:13 | |
| *** rodrigods has joined #openstack-infra | 12:13 | |
| *** aysyd has joined #openstack-infra | 12:14 | |
| *** icey has joined #openstack-infra | 12:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add release note publishing to glance_store https://review.openstack.org/303962 | 12:16 |
| *** pradk has joined #openstack-infra | 12:17 | |
| *** sdake_ has quit IRC | 12:18 | |
| *** achanda has quit IRC | 12:18 | |
| *** markvoelker has joined #openstack-infra | 12:19 | |
| *** dprince has joined #openstack-infra | 12:21 | |
| *** yamamoto has joined #openstack-infra | 12:22 | |
| *** dtantsur is now known as dtantsur|mtg | 12:25 | |
| *** weshay has joined #openstack-infra | 12:25 | |
| *** edmondsw has joined #openstack-infra | 12:26 | |
| *** camunoz_ has quit IRC | 12:26 | |
| *** amotoki has quit IRC | 12:27 | |
| *** rlandy has joined #openstack-infra | 12:30 | |
| *** Hal has joined #openstack-infra | 12:30 | |
| *** Hal is now known as Guest5559 | 12:30 | |
| *** zumbi has joined #openstack-infra | 12:32 | |
| odyssey4me | AJaeger I saw the thing about making it read-only, but if I do that it'll apply to all of our repositories I think. Is there a way to do that for only one of the repositories as they all share the same ACL file? | 12:34 |
| *** achanda has joined #openstack-infra | 12:35 | |
| *** achanda has quit IRC | 12:35 | |
| *** zhurong has joined #openstack-infra | 12:35 | |
| AJaeger | odyssey4me: sure.;) | 12:36 |
| *** zhurong has quit IRC | 12:36 | |
| AJaeger | odyssey4me: let me check your setup | 12:36 |
| *** kengo_sakai_mba has quit IRC | 12:36 | |
| *** salv-orl_ has quit IRC | 12:36 | |
| AJaeger | odyssey4me: you have in gerrit/projects.yaml "acl-config: /home/gerrit2/acls/openstack/openstack-ansible.config" | 12:36 |
| *** zhurong has joined #openstack-infra | 12:37 | |
| *** zhurong has quit IRC | 12:37 | |
| AJaeger | odyssey4me: create a new acl file gerrit/acls/openstack/openstack-ansible_swift_sync.config with the read only content from the ifnra manual and remove the acl-config line above. | 12:37 |
| odyssey4me | ah ok, that makes sense - let me revise that patch now with that advice - thanks! | 12:38 |
| AJaeger | yw | 12:38 |
| *** burgerk has joined #openstack-infra | 12:38 | |
| *** eharney has quit IRC | 12:39 | |
| AJaeger | odyssey4me: also, do you have a governance change? Please reference it | 12:39 |
| *** BobBall^ is now known as BobBall | 12:39 | |
| odyssey4me | sure, will do | 12:40 |
| *** kengo_sa_ has quit IRC | 12:40 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-os_swift_sync repository https://review.openstack.org/304049 | 12:41 |
| AJaeger | odyssey4me: no need to add information about defaults, just remove the acl-config line... | 12:42 |
| odyssey4me | ah ok, so the default acl file is the repo name? | 12:43 |
| mrmartin | hi | 12:44 |
| *** amrith is now known as _amrith_ | 12:44 | |
| *** achanda has joined #openstack-infra | 12:44 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-os_swift_sync repository https://review.openstack.org/304049 | 12:44 |
| *** gongysh has joined #openstack-infra | 12:46 | |
| mrmartin | the groups.o.o build seems to be broken, because the bundle package is missing, and compass requires that to compile the sass files. What is the proper way to add the missing package? Am I need to add it as a bindep dependency? | 12:46 |
| AJaeger | odyssey4me: yes | 12:46 |
| odyssey4me | nifty, and sensible :) | 12:46 |
| AJaeger | mrmartin: you could add an other-requirements.txt file that lists all the requirements your repo has. | 12:47 |
| AJaeger | mrmartin: we don't want to enhance bindep-fallback in general | 12:47 |
| pabelanger | morning | 12:47 |
| AJaeger | mrmartin: package name is bundle? | 12:47 |
| AJaeger | morning, pabelanger. Could you read the last 6 mins of backscroll and help mrmartin, please? | 12:48 |
| pabelanger | AJaeger: Sure, mrmartin do you mind posting a log file | 12:48 |
| mrmartin | AJaeger, pabelanger : yeap, the package name is "bundler" http://packages.ubuntu.com/trusty/bundler | 12:48 |
| mrmartin | http://paste.openstack.org/show/493614/ | 12:48 |
| mrmartin | this one | 12:48 |
| mrmartin | and the log: http://logs.openstack.org/12/303812/1/gate/gate-groups-unittests/a80a8f3/console.html | 12:48 |
| pabelanger | mrmartin: Oh, okay. I would suggest doing what puppet jobs do and sandbox your bundler install | 12:50 |
| pabelanger | https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/puppet-module-jobs.yaml#L16 | 12:50 |
| AJaeger | mrmartin: the log file says "SUCCESS" - that looks strange | 12:50 |
| krotscheck | Is the openstack-infra namespace still something I should use for infra-specific projects, or should I use the openstack namespace? | 12:50 |
| mrmartin | yeah, because bash not stopped with a proper exit code | 12:50 |
| pabelanger | mrmartin: I can get a patch for JJB to do it | 12:50 |
| *** akshai has joined #openstack-infra | 12:51 | |
| pabelanger | actually, where is scripts/dev-build.sh located? | 12:51 |
| pabelanger | should be either to add logic to add bundler there | 12:51 |
| mrmartin | pabelanger: ok we need to add this one only: gem install bundler --no-rdoc --no-ri --verbose | 12:51 |
| mat128 | pabelanger: https://github.com/openstack-infra/groups/blob/master/scripts/dev-build.sh | 12:51 |
| mrmartin | pabelanger: https://github.com/openstack-infra/groups/blob/master/scripts/dev-build.sh | 12:52 |
| pabelanger | mrmartin: oh, it is a path issue in your script | 12:52 |
| pabelanger | 1 sec for patch | 12:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/groups: Use path name for bundle https://review.openstack.org/304104 | 12:54 |
| pabelanger | mrmartin: that should help | 12:54 |
| pabelanger | lets see what the gate does | 12:54 |
| pabelanger | ugh | 12:55 |
| pabelanger | 1 sec | 12:55 |
| *** xyang1 has joined #openstack-infra | 12:55 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] disable releasenotes jobs for liberty and kilo branch https://review.openstack.org/304105 | 12:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/groups: Install bundler with gem https://review.openstack.org/304104 | 12:56 |
| vgridnev | folks please take a look on the https://review.openstack.org/304105 ; it should unlock our gate for stable/liberty | 12:57 |
| pabelanger | okay, that should be better | 12:57 |
| *** jsavak has joined #openstack-infra | 12:58 | |
| pabelanger | looks like jenkins02 is down again. 3 mondays in a row, I guess something is wrong with our ansible playbook | 12:58 |
| pabelanger | I'll fix it up shortly | 12:58 |
| AJaeger | pabelanger: it's time to move jenkins02 to a new server... | 13:01 |
| AJaeger | pabelanger: thanks for noticing and fixing it! | 13:01 |
| AJaeger | vgridnev: looking | 13:02 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to opentack infra https://review.openstack.org/304108 | 13:03 |
| pabelanger | AJaeger: Ya, clarkb mentioned that last week too. Going to bring it up at tomorrows meetings and see if people are on board with upgrade | 13:03 |
| krotscheck | opentack? | 13:03 |
| krotscheck | oops | 13:03 |
| pabelanger | mrmartin: mat128: Okay, should be fixed with 304104 | 13:04 |
| ZZelle | Hi everyone! | 13:04 |
| AJaeger | pabelanger: I think all are on board - it was just not high enough on anybody's priority list yet | 13:04 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra https://review.openstack.org/304108 | 13:04 |
| krotscheck | There we go | 13:04 |
| pabelanger | AJaeger: Ya, I also think we have a quota issue too | 13:04 |
| pabelanger | okay, starting on jenkins02 | 13:05 |
| AJaeger | krotscheck: that change needs a governance change as well... | 13:05 |
| openstackgerrit | Merged openstack-infra/groups: Install bundler with gem https://review.openstack.org/304104 | 13:06 |
| *** maishsk has quit IRC | 13:07 | |
| *** amotoki has joined #openstack-infra | 13:08 | |
| krotscheck | AJaeger: yep, workin on it. | 13:09 |
| *** kzaitsev_mb has joined #openstack-infra | 13:09 | |
| AJaeger | krotscheck: once you have it, add Needed-By to your commit message, please | 13:09 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Refactor releasenote branch condition https://review.openstack.org/304116 | 13:10 |
| krotscheck | Needed by? Is that new? | 13:10 |
| ZZelle | The redirection from review.o.o to launchpad for oauth fails, am i the only one impacted? | 13:11 |
| *** maishsk has joined #openstack-infra | 13:11 | |
| pabelanger | okay, jenkins02 back online | 13:12 |
| AJaeger | krotscheck: introduced together with depends-on - and documented in Infra Manual governance change section http://docs.openstack.org/infra/manual/creators.html#add-new-repository-to-the-governance-repository | 13:12 |
| *** trown|outtypewww is now known as trown | 13:12 | |
| krotscheck | AJaeger: Odd. In the past you've told me to put depends-on in the governance patch, not needed-by in the project_config patch | 13:12 |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] configure jobs for sahara-tests https://review.openstack.org/304121 | 13:13 |
| AJaeger | krotscheck: you should do *both* | 13:13 |
| *** zhurong has joined #openstack-infra | 13:14 | |
| AJaeger | needed-by is the backwards pointer to depends-on | 13:14 |
| AJaeger | krotscheck: the depends-on is essential, the needed-by very good practice ;) | 13:14 |
| mrmartin | pabelanger: looks better now, thnx: http://logs.openstack.org/04/304104/2/gate/gate-groups-unittests/f13efa9/console.html | 13:14 |
| krotscheck | This seems like an awful lot of red tape to jump through. | 13:14 |
| fungi | Zara: it was more of a concern that in the very distant future, the /var/spool/mail/storyboard mbox file might get so full of bounces and unanticipated replies that it consumes all of the available space on the root filesystem. since those messages are small bits of text and we have 34gb available space on the server, it's not especially urgent and can probably wait (even years) until storyboard has | 13:14 |
| fungi | inbound e-mail processing | 13:15 |
| pabelanger | mrmartin: np | 13:15 |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] configure publist-to-pypi jobs for sahara-tests https://review.openstack.org/304121 | 13:15 |
| *** _ari_ has joined #openstack-infra | 13:16 | |
| AJaeger | krotscheck: I need to governance review to +2 your change. So, putting it in the commit message is the best place for it. | 13:17 |
| Zara | fungi: ahhh, okay. :D are there any circumstances under which it would fill up faster? eg: if messages could contain pictures, etc? (not that we have plans to do this, but that way my brain will flag it if it comes up in the future) | 13:17 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra https://review.openstack.org/304108 | 13:17 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 13:17 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Temporarily disable mistral on the undercloud https://review.openstack.org/302832 | 13:17 |
| fungi | mrmartin: pabelanger: it does appear that we `gem install bundler` in our puppet jobs... is bundler a deployment requirement for the groups webapp too, or just used in testing it? | 13:17 |
| mrmartin | fungi: it is using to compile SaSS to CSS, so we are using it for packaging actually | 13:18 |
| mrmartin | it is / we are | 13:19 |
| penguinolog | review please: https://review.openstack.org/#/c/304017/ | 13:20 |
| pabelanger | fungi: mrmartin: Ya, for the most part, puppet jobs will just sandbox bundler, since they have boostrap script in the gate. Looks like groups maybe have just overlooked it from the start | 13:20 |
| pabelanger | fungi: moving to ubuntu-trusty exposed the missing dependency | 13:20 |
| AJaeger | fungi, https://review.openstack.org/#/c/304108 needs infra PTL+1, please put it on your queue - this is krotscheck's change | 13:20 |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: clening up python3 jobs for sahara-tests https://review.openstack.org/304130 | 13:21 |
| *** links has quit IRC | 13:21 | |
| *** piet has joined #openstack-infra | 13:21 | |
| *** tiswanso has joined #openstack-infra | 13:21 | |
| AJaeger | penguinolog: if that's critical for you, please explain what it is and why you urgently need it... | 13:21 |
| krotscheck | AJaeger Except it's broken. | 13:21 |
| krotscheck | Just a minute | 13:21 |
| AJaeger | krotscheck: do you mind capitalizing OpenStack in gerrit/projects.yaml then as well? | 13:22 |
| fungi | pabelanger: AJaeger: clarkb thought that https://review.openstack.org/303756 might fix it. merged yesterday at 07:14 utc, about 7 hours after the playbook ran | 13:23 |
| penguinolog | AJaeger, it's not urgent, but now we have >1 week positive status on this check and after voting in team we selected to enable voting | 13:23 |
| krotscheck | AJaeger- That's why it's broken | 13:23 |
| AJaeger | ;) | 13:23 |
| krotscheck | Silly infra, whyfore do you explicitly reference an executable by path? /bin/mktemp: No such file or directory | 13:24 |
| *** resker has quit IRC | 13:25 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra https://review.openstack.org/304108 | 13:25 |
| *** flip214 has quit IRC | 13:25 | |
| fungi | Zara: i suppose if storyboard started sending very large messages with image attachments and people's mtas were bouncing them with all attachments intact that could conceivably fill up a little quicker, but not enough i'd be worried about it making a huge difference | 13:25 |
| pabelanger | fungi: clarkb: maybe we should default master executors to 0 with a grovvy script: https://git.openstack.org/cgit/openstack/windmill/tree/playbooks/files/jenkins/var/lib/jenkins/init.groovy.d/executors.groovy | 13:25 |
| pabelanger | vs offlining them | 13:26 |
| *** flip214 has joined #openstack-infra | 13:26 | |
| *** pkoniszewski has joined #openstack-infra | 13:26 | |
| Zara | fungi: cool, I'll file it under 'not gonna worry about this' (I'd like to get inbound email processing working for other reasons, but yeah). thank you. :) | 13:27 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Use the new launch client in zuul server https://review.openstack.org/304132 | 13:27 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Don't launch jobs in threads https://review.openstack.org/304133 | 13:27 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Use the seeded repos when launching a job https://review.openstack.org/304134 | 13:27 |
| anteaya | I think for the longest time (I think this is a recent change) we had a word for the people who work in this channel and we had a separate word for the tooling we support | 13:27 |
| *** mtanino has joined #openstack-infra | 13:27 | |
| AJaeger | morning, anteaya ! | 13:27 |
| anteaya | having two separate words enabled folks to respect and support the people while communicating frustration and disappointment with the tooling | 13:28 |
| anteaya | AJaeger: morning | 13:28 |
| anteaya | that has changed | 13:28 |
| anteaya | and now folks use one word (not sure why) | 13:28 |
| anteaya | to express frustration and disapointment | 13:28 |
| anteaya | which now extends to the people | 13:28 |
| anteaya | which, I have to say, becomes tiring fast | 13:28 |
| * AJaeger agrees with anteaya ... | 13:29 | |
| *** krtaylor has joined #openstack-infra | 13:29 | |
| anteaya | AJaeger: thanks, glad you have seen it happen as well | 13:29 |
| AJaeger | anteaya: doyou want to advise also to people what to use instead of just "infra" for both? | 13:31 |
| *** oanson has quit IRC | 13:31 | |
| *** Guest5559 has quit IRC | 13:31 | |
| * mordred calls himself mordred | 13:31 | |
| anteaya | AJaeger: no | 13:32 |
| *** eharney has joined #openstack-infra | 13:32 | |
| anteaya | I'm carrying the standard of using the word folks people and group, as well as friends and collegues | 13:32 |
| anteaya | I can't take this on as well | 13:32 |
| anteaya | mostly I'm saying that word choice makes a big difference in group health | 13:32 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Correct error message when domain is required https://review.openstack.org/304137 | 13:32 |
| anteaya | and I'm observing this | 13:32 |
| anteaya | mordred: morning mordred | 13:33 |
| * AJaeger welcomes mordred called mordred | 13:33 | |
| mordred | yah. I've got floating ips, and "openstack is not the place for cloud native workloads" - so I'm also a bit full on conceptual flags | 13:33 |
| mordred | morning anteaya and AJaeger ! | 13:33 |
| anteaya | :) | 13:33 |
| AJaeger | anteaya: I agree, word choice is important | 13:34 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax https://review.openstack.org/303529 | 13:34 |
| anteaya | AJaeger: but in this instance I don't have a suggestion for a solution | 13:34 |
| *** e0ne has quit IRC | 13:34 | |
| trown | hola infra folks, I do not see anything in the project creator's guide about creating groups in gerrit, could someone point me in the right direction? | 13:34 |
| fungi | trown: gerrit groups are automatically created if gerrit acls refer to them | 13:34 |
| trown | fungi: ah, cool, who gets edit permissions for group membership? | 13:35 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra https://review.openstack.org/304108 | 13:35 |
| *** mylu has joined #openstack-infra | 13:35 | |
| AJaeger | trown: if the content is not clear, please send a patch to enlighten the next one reading it... | 13:35 |
| trown | AJaeger: will do :) | 13:35 |
| *** mriedem has joined #openstack-infra | 13:35 | |
| trown | in general it is very good | 13:35 |
| AJaeger | trown: did yo uread http://docs.openstack.org/infra/manual/creators.html#update-the-gerrit-group-members ? | 13:36 |
| fungi | trown: normally they're self-managed, and (upon request) a gerrit admin will add the gerrit account of the contributor who submitted the acl addition as the initial member | 13:36 |
| openstackgerrit | Alexis Lee proposed openstack-infra/shade: Add test for delete flavor exception with betamax https://review.openstack.org/303529 | 13:36 |
| *** thorst has quit IRC | 13:36 | |
| fungi | trown: or the gerrit account of the ptl in the event that the acl is for repos under an official team in openstack governance | 13:36 |
| trown | AJaeger: that is precisely what I was looking for thanks! | 13:37 |
| *** salv-orlando has joined #openstack-infra | 13:37 | |
| AJaeger | fungi, thanks for approving the translation script change yesterday! No need for any rerun of post jobs... | 13:37 |
| AJaeger | fungi, New Babel release had broken handling of multiple keywords on command line ;( | 13:38 |
| AJaeger | upstream patch to fix it exists, so I hope to be able to revert my change soon... | 13:38 |
| *** e0ne has joined #openstack-infra | 13:39 | |
| *** amitgandhinz has joined #openstack-infra | 13:40 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Add NPM mirror hosting to unified mirrors https://review.openstack.org/303650 | 13:40 |
| mordred | derekh: morning! | 13:40 |
| openstackgerrit | Merged openstack-infra/project-config: fuel-qa: Add vote to pylint checks https://review.openstack.org/304017 | 13:40 |
| mordred | derekh: so - when you get a sec, I encountered an issue with the tripleo cloud over the weekend. we have a workaround for it, but I thought you should know | 13:40 |
| derekh | mordred: howdy, whats up? | 13:40 |
| *** jtomasek has quit IRC | 13:41 | |
| mordred | derekh: basically, if you create a floating ip via the neutron api handing it the port id of the server in question - the cloud happily makes and attaches the floating ip to the server but seems to not inform nova about it | 13:41 |
| krotscheck | oookay. | 13:42 |
| mordred | derekh: I've verified that the same calls work on the rax/intel osic, on dreamhost and on our bluebox cloud (which all require floating ips) so I _think_ it's something wonky on the tripleo cloud | 13:42 |
| krotscheck | I think I've got all the npm things ready. | 13:42 |
| *** tonytan4ever has joined #openstack-infra | 13:42 | |
| anteaya | morning yolanda, I'm taking a look at the etherpad for the gerrit migration today: https://etherpad.openstack.org/p/gerrit_server_replacement | 13:42 |
| mordred | derekh: but like I said - not a big deal, we have a config flag we can use to change our behavior to use the nova proxy api - mostly wanted to heads-up you | 13:42 |
| yolanda | hi anteaya | 13:42 |
| mordred | krotscheck: zomg. that's super awesome | 13:43 |
| *** salv-orlando has quit IRC | 13:43 | |
| anteaya | yolanda: did you have an opportunity to read the suggestions jeblair left on it? | 13:43 |
| mriedem | are the openstack irc channel topics stored in an infra repo somewhere? | 13:43 |
| yolanda | yes, i took a look today | 13:43 |
| anteaya | yolanda: the line above step 1 and the line below step 1 | 13:43 |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] configure publist-to-pypi jobs for sahara-tests https://review.openstack.org/304121 | 13:43 |
| krotscheck | I don't suppose I can get someone with an AFS keytab to go in and clear out the /mirror/npm directory? It's got some old things in it. | 13:44 |
| AJaeger | mriedem: not currentlyy | 13:44 |
| krotscheck | mordred ^^ | 13:44 |
| yolanda | yep, i actually wanted to check with infra-roots today about that. As we use trove, wanted to check what's the better way for creating backups | 13:44 |
| anteaya | yolanda: and I was wondering if you wanted to paste in the commands you expect to run for each step | 13:44 |
| anteaya | yolanda: like we do with the rename workflow | 13:44 |
| mordred | krotscheck: sure. one sec | 13:44 |
| anteaya | yolanda: great | 13:44 |
| derekh | mordred: ok, are the calls your making in nodepool somewhere and I'll see if I can reproduce? Also we are hoping to redeploy that cloud shortly after the summit so if its not easily fixable the new cloud should do the right thing | 13:45 |
| AJaeger | mriedem: but would be a neat idea, jhesketh was considering it last week as well when he restarted the bot and had to resurrect all channel messages - the bot had changed the topics temporraily and didn't know to set it back ;( | 13:45 |
| mriedem | AJaeger: yeah i noticed the nova channel topic changed | 13:45 |
| yolanda | about the dns, i also have the question if we change it just when we start the maintenance window, or after the updates have been done | 13:45 |
| fungi | yolanda: i'd do it at the start of the maintenance window (think i said that in the meeting as well) | 13:46 |
| anteaya | yolanda: can you add your questions to the etherpad? | 13:46 |
| fungi | that way it gets more time to propagate | 13:46 |
| AJaeger | mriedem: jhesketh grepped the topic from eavesdrop and set it back again | 13:46 |
| yolanda | i added my question to the ehterpad | 13:46 |
| jhesketh | mriedem: the statusbot stores it in memory | 13:46 |
| anteaya | yolanda: that way when you have answers everyone can see | 13:46 |
| anteaya | yolanda: thanks | 13:46 |
| jhesketh | mriedem: topics also land in the logs which is where I extracted all the ones from to restore them | 13:46 |
| yolanda | ok, updated the document with dns change | 13:47 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Correct error message when domain is required https://review.openstack.org/304137 | 13:47 |
| jhesketh | a tool to pull them from logs would be useful, or otherwise a data store of sorts for the bot to manage | 13:47 |
| anteaya | jhesketh: hey there! | 13:47 |
| fungi | jhesketh: yep, that's how we've restored it in the past if statusbot gets restarted in the middle of an alert condition | 13:47 |
| yolanda | fungi, also how about the backups? how do we do it on trove? | 13:47 |
| anteaya | yolanda: thanks | 13:47 |
| *** kgiusti has joined #openstack-infra | 13:47 | |
| jhesketh | I did it by hand because I figured it might be easier than writing a bot, but I think given the time to do it I may have been wrong | 13:47 |
| jhesketh | anteaya: howdy | 13:47 |
| fungi | yolanda: there's a daily cronjob in root's crontab (our puppet mysql backup module creates it) | 13:47 |
| yolanda | so we already have the backups? | 13:47 |
| mordred | krotscheck: you said you want all of the things in mirror/npm deleted? | 13:47 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Correct error message when domain is required https://review.openstack.org/304137 | 13:47 |
| yolanda | or do we launch one just before? | 13:47 |
| fungi | yolanda: might be a good idea to do one more just before we start | 13:48 |
| fungi | yolanda: but can simply reuse the syntax from the crontab | 13:48 |
| *** jtomasek has joined #openstack-infra | 13:48 | |
| yolanda | ok i will add it | 13:48 |
| *** ddieterly has joined #openstack-infra | 13:49 | |
| *** jtomasek has quit IRC | 13:50 | |
| *** achanda has quit IRC | 13:50 | |
| *** jtomasek has joined #openstack-infra | 13:50 | |
| yolanda | anteaya, i'm adding a bit more of details to the etherpad, although i cannot create an step by step instruction. But will be enough to get a clear idea | 13:51 |
| *** mtanino has quit IRC | 13:51 | |
| krotscheck | mordred: yep. | 13:51 |
| krotscheck | mordred: And while you're in there, can you delete my access rights? | 13:51 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Correct error message when domain is required https://review.openstack.org/304137 | 13:52 |
| openstackgerrit | Merged openstack-infra/project-config: Update regex for tempest-dsvm-ironic-inspector job https://review.openstack.org/303310 | 13:52 |
| AJaeger | dhellmann: what's the best way to fix broken releasenotes? See https://review.openstack.org/303989 - does it need a backport to mitaka like in https://review.openstack.org/304142 ? | 13:53 |
| openstackgerrit | John Trowbridge proposed openstack-infra/project-config: Create tripleo-quickstart subteam https://review.openstack.org/304145 | 13:53 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Fix filtering of automatic worklist contents https://review.openstack.org/304146 | 13:54 |
| *** _amrith_ is now known as amrith | 13:54 | |
| mordred | krotscheck: done | 13:54 |
| krotscheck | mordred: Awesome, thanks | 13:54 |
| yolanda | fungi, can you double check the etherpad again, to ensure that all steps are valid? https://etherpad.openstack.org/p/gerrit_server_replacement | 13:55 |
| amrith | yolanda, trove does backups for you ... | 13:55 |
| *** esker has joined #openstack-infra | 13:56 | |
| *** esker has quit IRC | 13:56 | |
| *** esker has joined #openstack-infra | 13:57 | |
| yolanda | amrith, thanks. I got the commands to do a local backup before the maintenance | 13:57 |
| *** nmagnezi has quit IRC | 13:57 | |
| *** avarner has joined #openstack-infra | 13:57 | |
| *** achanda has joined #openstack-infra | 13:58 | |
| *** xarses has quit IRC | 13:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Consume config values for NAT destination https://review.openstack.org/300592 | 13:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Search subnets for gateway_ip to discover NAT dest https://review.openstack.org/300598 | 13:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: WIP Start stamping the has_service debug messages https://review.openstack.org/303853 | 13:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Refactor guts of _find_interesting_networks https://review.openstack.org/301738 | 13:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Refactor the port search logic https://review.openstack.org/303852 | 13:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add nat_destination filter to floating IP creation https://review.openstack.org/302104 | 13:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Allow passing nat_destination to get_active_server https://review.openstack.org/303851 | 13:58 |
| AJaeger | amrith: what's up with trove-dashboard? The translation imports do not pass your tests: https://review.openstack.org/303250 and https://review.openstack.org/303690 . Would be great to merge these, please | 13:58 |
| openstackgerrit | Merged openstack-infra/shade: Make delete_project to call get_project https://review.openstack.org/291940 | 13:59 |
| *** rkukura has quit IRC | 13:59 | |
| *** doug-fish has joined #openstack-infra | 14:00 | |
| *** jsavak has quit IRC | 14:01 | |
| *** e0ne has quit IRC | 14:01 | |
| *** jsavak has joined #openstack-infra | 14:01 | |
| *** woodster_ has joined #openstack-infra | 14:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add simple script to help debug shade image uploads https://review.openstack.org/287296 | 14:03 |
| *** e0ne has joined #openstack-infra | 14:04 | |
| *** yamamoto has quit IRC | 14:06 | |
| *** achanda has quit IRC | 14:07 | |
| *** nadya has quit IRC | 14:07 | |
| *** burgerk has quit IRC | 14:07 | |
| *** bkeller` has joined #openstack-infra | 14:08 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/project-config: Create tripleo-quickstart subteam https://review.openstack.org/304145 | 14:09 |
| *** yamamoto has joined #openstack-infra | 14:10 | |
| *** jtomasek has quit IRC | 14:11 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Update delete flavor test to work with betamax https://review.openstack.org/303459 | 14:13 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add new test for deleting non-existing flavors https://review.openstack.org/303515 | 14:13 |
| *** vgridnev has quit IRC | 14:14 | |
| mrmartin | pabelanger: I'm checking the last puppet run on groups-dev.o.o and it was at 9:08: http://puppetboard.openstack.org/node/groups-dev.openstack.org | 14:15 |
| *** roxanaghe has joined #openstack-infra | 14:18 | |
| pabelanger | mrmartin: Hmm, looks like ansible is blocked again | 14:18 |
| *** dizquierdo has joined #openstack-infra | 14:18 | |
| jeblair | good morning | 14:19 |
| jhesketh | o/ | 14:19 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Update test for setting flavor specs to use betamax https://review.openstack.org/304064 | 14:19 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Return boolean from delete_project https://review.openstack.org/304155 | 14:19 |
| pabelanger | Hmm, looks to be hung on the past facts task since 07:00 UTC | 14:20 |
| jhesketh | jeblair: I'm about to head off, but if you have time this week I think we should go over a few of the open zuul changes and mark some for consideration to merge before v3.. there are a few that we may want in v3 and some we may just want sooner etc | 14:20 |
| *** zz_dimtruck is now known as dimtruck | 14:20 | |
| *** ajmiller has joined #openstack-infra | 14:20 | |
| *** rvasilets has left #openstack-infra | 14:20 | |
| mrmartin | pabelanger: does it affect groups.o.o only, or everything else? | 14:21 |
| jhesketh | jeblair: I /think/ this one should help with the memory leak https://review.openstack.org/#/c/281166/ | 14:21 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Update test_unset_flavor_specs to use betamax https://review.openstack.org/304073 | 14:21 |
| pabelanger | mrmartin: everything | 14:21 |
| *** ddieterly is now known as ddieterly[away] | 14:21 | |
| mrmartin | cool | 14:21 |
| jhesketh | jeblair, fungi: which, btw, it looks like zuul might run out of memory any minute now :-s (http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all) | 14:21 |
| jeblair | jhesketh: okay -- it sounds like you've started -- you want to send me a list of ones you think i should look at? | 14:22 |
| *** roxanaghe has quit IRC | 14:22 | |
| pabelanger | mrmartin: okay, looks to be unblocked, should start running again in 8mins | 14:22 |
| jhesketh | jeblair: yeah I went over the zuul reviews yesterday but didn't mark any particulars down... I'll do that tomorrow :-) | 14:22 |
| *** zhurong has quit IRC | 14:22 | |
| jeblair | jhesketh, fungi: that suggests that the foreign project change is probably not responsible for the leak -- we've been running with that manually reverted since the last restart | 14:22 |
| mrmartin | pabelanger: thnx again | 14:23 |
| fungi | jhesketh: jeblair: okay, so the foreign project for pipeline revert we have running on zuul.o.o right now doesn't seem to have helped the leak i guess? does that place the ball more squarely in the connections change court? | 14:23 |
| fungi | er, what jeblair just said | 14:23 |
| jhesketh | jeblair: oh I didn't know that was applied.. interesting | 14:23 |
| *** ddieterly[away] is now known as ddieterly | 14:23 | |
| jhesketh | fungi, jeblair: I wonder if it's worth doing the same experiment with 281166 | 14:23 |
| *** jsavak has quit IRC | 14:24 | |
| jeblair | jhesketh: i think we can experiment with it -- but i don't want to land that patch because it is a feature regression; it's definitely intended that everything should be reloaded | 14:24 |
| *** jsavak has joined #openstack-infra | 14:24 | |
| pabelanger | I wonder if we should pass --timeout=SECONDS to the ansible-playbook command to help improving our connection issues | 14:24 |
| jhesketh | jeblair: well the .conf was reloaded previously, but the connection to gerrit (for example) was never re-established | 14:25 |
| *** vgridnev has joined #openstack-infra | 14:25 | |
| jhesketh | so updated credentials were never picked up | 14:25 |
| jhesketh | and I think it owuld have been the same for smtp connection (since the class is spawned at the start) | 14:25 |
| jeblair | jhesketh: ok. we may need to handle connections separately (we don't want to needlessly restart them) -- but i also don't want to lose updates to the logging config, or log urls, or other options. | 14:26 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Return boolean from delete_project https://review.openstack.org/304155 | 14:26 |
| *** avarner has quit IRC | 14:27 | |
| jhesketh | jeblair: hmm, I wonder if just not stopping and registering connections is enough... | 14:28 |
| *** tongli has joined #openstack-infra | 14:28 | |
| jeblair | jhesketh: maybe so? | 14:28 |
| pabelanger | #status log ansible-playbook blocked since 0700 UTC, manually killed ansible-playbook process to unblock puppet_run_all.lock | 14:28 |
| jhesketh | yeah I'm thinking so... I'm just taking a look | 14:28 |
| *** avarner has joined #openstack-infra | 14:28 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors https://review.openstack.org/302192 | 14:28 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 14:28 |
| mordred | yolanda: we still waiting on betamax release? | 14:29 |
| *** mtanino has joined #openstack-infra | 14:29 | |
| yolanda | mordred yes, i have pinged sigmavirus24, but i see him offline since 7th april | 14:29 |
| mordred | kk. | 14:29 |
| jeblair | yolanda: where's the gerrit upgrade etherpad? | 14:30 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Only reload layout on HUP https://review.openstack.org/281166 | 14:30 |
| yolanda | https://etherpad.openstack.org/p/gerrit_server_replacementç | 14:31 |
| jhesketh | jeblair: so nothing obvious in terms of using the connections from the config... so hopefully ^ is safe | 14:31 |
| yolanda | https://etherpad.openstack.org/p/gerrit_server_replacement | 14:31 |
| jhesketh | (note, untested) | 14:31 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Return boolean from delete_project https://review.openstack.org/304155 | 14:31 |
| jeblair | jhesketh: cool; we can try restarting with that next time | 14:31 |
| jhesketh | jeblair: sounds good... I wonder if it's worth preempting the memory running out and doing it in a quiet period (eg tomorrow night) | 14:32 |
| jhesketh | assuming it hasn't ran out in the mean time | 14:32 |
| *** regXboi has joined #openstack-infra | 14:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add checks for gerrit ACLs https://review.openstack.org/304164 | 14:33 |
| *** sc68cal has joined #openstack-infra | 14:33 | |
| jhesketh | jeblair: or maybe during the gerrit switchover | 14:33 |
| jeblair | jhesketh: that would be a good one | 14:33 |
| AJaeger | the change above will fail until https://review.openstack.org/303984 is in - reviews for both are welcome, please | 14:33 |
| *** rkukura has joined #openstack-infra | 14:33 | |
| AJaeger | project-config cores, 304164 checks for all problems I noticed in reviews today - and a few more ;) | 14:34 |
| *** jamesmcarthur has joined #openstack-infra | 14:35 | |
| *** yamahata has joined #openstack-infra | 14:36 | |
| jeblair | AJaeger: somehow 304164 conflicts with 303984 | 14:38 |
| AJaeger | jeblair, interesting... | 14:38 |
| *** burgerk has joined #openstack-infra | 14:38 | |
| AJaeger | I'll rebase/recheck once 303984 is in | 14:38 |
| *** knikolla has joined #openstack-infra | 14:38 | |
| jeblair | git does not think so, but gerrit does | 14:39 |
| *** greghaynes has quit IRC | 14:39 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra https://review.openstack.org/304108 | 14:39 |
| AJaeger | jeblair, jhesketh : We currently have nearly 6 MB of log files for zuul/layout.yaml, is there a good way to "compress" it? I'm speaking here about the project-config job. What about removing DEBUG values? | 14:39 |
| *** haypo has joined #openstack-infra | 14:40 | |
| AJaeger | jeblair: ok, let me rebase now... | 14:40 |
| *** tpeoplesvacation is now known as tpeoples | 14:40 | |
| haypo | hi. "recheck" asks to run again all checks, but is it possible to only ask to retry a single job? on my job https://review.openstack.org/#/c/280732/ gate-grenade-dsvm-neutron failed on apt-get: "Cannot initiate the connection to mirror.dfw.rax.openstack.org:80" | 14:41 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add checks for gerrit ACLs https://review.openstack.org/304164 | 14:41 |
| anteaya | haypo: no | 14:42 |
| anteaya | haypo: all jobs run on purpose | 14:42 |
| *** jaosorior is now known as jaosorior_away | 14:42 | |
| anteaya | as rechecking jobs individually will allow patches that have races to be merged | 14:42 |
| *** vikram_ has joined #openstack-infra | 14:42 | |
| anteaya | check your patch, you might be triggering or introducing a nondeterministic failure | 14:43 |
| anteaya | just rechecking until your patch merges introduces races in the code | 14:43 |
| haypo | anteaya: what? | 14:43 |
| jeblair | krotscheck, fungi: if we felt like bikeshedding -- afs-blob-store isn't actually afs-specific, is it? it could as well be 'posix-blob-store' or 'fs-blob-store', yeah? having said that, i don't mind having more things in the world that say afs, so i'm just going to +2 304108 unless one of you are like 'aha!' | 14:43 |
| anteaya | and makes the code base worse for everyone | 14:43 |
| anteaya | haypo: what part of my reply did you not understand? | 14:43 |
| haypo | anteaya: apt-get failed. my patch only changes the neutron code, not the code to create/setup the VM | 14:44 |
| *** maishsk has quit IRC | 14:44 | |
| anteaya | haypo: right, I'm giving you the reason we have all the jobs run every time | 14:44 |
| anteaya | haypo: you asked can you trigger one job at a time, I said no and told you why not | 14:45 |
| haypo | anteaya: my question is specific to random failures which are unrelated to my change, when i checked and i know that it's really unrelated to my change and "random" | 14:45 |
| krotscheck | jeblair: Well, fs-blob-store is an interface in the node world. | 14:45 |
| krotscheck | so xxx-blob-store is the usual naming convention for thing-that-implements-blob-store-for-thing | 14:45 |
| anteaya | haypo: right, we have no provision for checking on job for 'random' failures | 14:46 |
| vikram_ | Hi Admin | 14:46 |
| *** jtomasek has joined #openstack-infra | 14:46 | |
| anteaya | s/on/one | 14:46 |
| jeblair | krotscheck: oh, 'fs-blob-store' is the one that implements an unhierarchical blob store on the fs? | 14:46 |
| krotscheck | jeblair: Yep. | 14:46 |
| vikram_ | Can someone please add me to ACL's for the newly created project https://github.com/openstack/networking-huawei | 14:46 |
| krotscheck | jeblair: If you look at the implementation that I did, the afs thing actually sits on top of fs-blob-store | 14:46 |
| jeblair | krotscheck: then i guess i'm left with suggesting 'uninsane-blob-store' so 'afs-blob-store' is probably the better name. | 14:46 |
| haypo | anteaya: last weeks, i started to check each time why a test failed, and it's very common that a job fails before running tests, it fails to setup the VM :-/ | 14:46 |
| krotscheck | And, well, it's not afs specific, no, but it's at that odd intersection of a-little-too-precise-use-case. | 14:47 |
| vikram_ | fungi: Can you plz help me out | 14:47 |
| *** rbrndt has joined #openstack-infra | 14:47 | |
| anteaya | haypo: did you file a bug? | 14:47 |
| anteaya | haypo: or investigate bugs for that failure? | 14:47 |
| vikram_ | fungi: Adding me to ACL's for the newly created project https://github.com/openstack/networking-huawei | 14:47 |
| krotscheck | jeblair: Right, but if you consider what the name says to the rest of the world, one could assume that "Oh, this is a thing that'll run vos-release automatically" | 14:47 |
| krotscheck | Which it doesn't. | 14:47 |
| rcarrillocruz | mordred: i was chatting the other day with Shrews about https://review.openstack.org/#/c/302770/ (yeah, revisiting designate again :D ). We agreed to add the designate plugins on all the dsvm jobs, rather than adding a new specific designate-dsvm job | 14:48 |
| jeblair | krotscheck: yeah, but leaves the door open. | 14:48 |
| rcarrillocruz | all good? | 14:48 |
| mordred | krotscheck: so - fwiw - it looks to me like maybe you didn't git add in your repo | 14:48 |
| mordred | krotscheck: all I see is tests and code to handle arguments | 14:48 |
| rcarrillocruz | me thinks we may eventually want to have a non-core services dsvm job | 14:48 |
| *** placko_ has quit IRC | 14:48 | |
| rcarrillocruz | and leave neutron/nova/keystone to just test that | 14:48 |
| mordred | but it's also probably my js foo is not as awesome as I'd like and I'm missing something | 14:48 |
| jeblair | mordred: javascript is just that amazing. | 14:48 |
| haypo | anteaya: oh... i'm not sure that a bug report would help... today i noticed two failures (1) gpg failed to retrieve a key. i tried manually: it works. it looks like a network issue or timeout issue (2) apt-get failed to download Releases.gpg -- network issue or issue with the package server | 14:48 |
| jeblair | mordred: you *only* have to write tests. | 14:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add checks for gerrit ACLs https://review.openstack.org/304164 | 14:49 |
| rcarrillocruz | where we put designate, heat, $new_service | 14:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add missing acl-config reference for os-collect-config to os-apply-config.config https://review.openstack.org/303984 | 14:49 |
| mordred | jeblair: dude. we should totally write that framework | 14:49 |
| haypo | anteaya: for (1), the resource is external, it's the public GPG server hkp://keys.gnupg.net | 14:49 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: WIP: Add shade call to get flavor access list https://review.openstack.org/304171 | 14:49 |
| *** GheRivero_vaca is now known as GheRivero | 14:49 | |
| mordred | jeblair: write the tests, and we'll infer and generate the code that satisfies the tests | 14:49 |
| *** piet has quit IRC | 14:49 | |
| AJaeger | jhesketh, jeblair: Thanks for review, fixed the issues and removed basing on unrelated change | 14:49 |
| haypo | anteaya: the (2), i don't know who handles http://mirror.dfw.rax.openstack.org/ubuntu/dists/trusty/Release.gpg and it expects a reply "issue already fixed" or "network issue, retry" | 14:49 |
| haypo | anteaya: for* (2) | 14:49 |
| anteaya | haypo: okay well if you think a bug report won't help, then me thinking it will won't result in you filing a bug report | 14:50 |
| haypo | anteaya: i will open a bug if i start to see regulary the same failure multiple times | 14:50 |
| anteaya | haypo: okay | 14:50 |
| haypo | anteaya: sadly, the last errors i saw were all unique! | 14:50 |
| haypo | (and very different) | 14:50 |
| anteaya | okay | 14:50 |
| vikram_ | AJaeger: The project is created now.. Whom I should contact now | 14:51 |
| *** Jeffrey4l_ has quit IRC | 14:51 | |
| *** yamamoto has quit IRC | 14:51 | |
| Zara | clarkb: saw your feature-request in the logs, have put it in this story: https://storyboard.openstack.org/#!/story/2000545 ; not sure when I'll get to improving subscriptions but I'd like to look at it at some point. | 14:51 |
| krotscheck | jeblair: Anything since 'uninsane-blob-store'? I had a bouncer hiccup | 14:51 |
| jhesketh | AJaeger: I think that'll fail Jenkins sorry | 14:51 |
| jhesketh | (see comment) | 14:51 |
| vikram_ | AJaeger: Can you plz show me the way :) | 14:52 |
| mordred | krotscheck: yes ... | 14:52 |
| AJaeger | jhesketh: I thought I fixed id ;( Will push again.. | 14:52 |
| jeblair | krotscheck: from me: general agreement with your forward looking statements about potentially adding more afs to afs-blob-store | 14:52 |
| AJaeger | vikram_: ask here for "infra-root" help, and reference the commit | 14:52 |
| mordred | krotscheck: I see nothing in your project other than tests and args.js ... maybe missing git add ? | 14:52 |
| *** greghaynes has joined #openstack-infra | 14:52 | |
| krotscheck | Hrm. IRC job is failing. | 14:52 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add checks for gerrit ACLs https://review.openstack.org/304164 | 14:52 |
| AJaeger | jhesketh: better? | 14:52 |
| vikram_ | AJaeger: Ok | 14:53 |
| AJaeger | krotscheck: fungi fixed it for you, didn't he? | 14:53 |
| krotscheck | mordred: index.json | 14:53 |
| krotscheck | sorry | 14:53 |
| krotscheck | mordred: index.js | 14:53 |
| jhesketh | AJaeger: lgtm, thanks | 14:53 |
| mordred | krotscheck: ah - in the root and not in the lib dir | 14:53 |
| AJaeger | thanks, jhesketh ! | 14:53 |
| vikram_ | Need infra-root for https://review.openstack.org/#/c/302120/ . Can someone plz help | 14:54 |
| krotscheck | AJaeger: Oh, cool. Didn't see that | 14:54 |
| fungi | krotscheck: yeah, gate-project-config-irc-access looks like it's passing (according to zuul status) since my tweak, but it hasn't reported yet because gate-project-config-compare-xml is still running | 14:54 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for phantomjs / horizon lint job bug 1568325 https://review.openstack.org/304173 | 14:54 |
| openstack | bug 1568325 in OpenStack Dashboard (Horizon) "Phantom is causing JS tests to fail constantly" [High,In progress] https://launchpad.net/bugs/1568325 - Assigned to Rob Cresswell (robcresswell) | 14:54 |
| *** esikachev has joined #openstack-infra | 14:54 | |
| krotscheck | fungi: cool, thanks for stepping in :) | 14:55 |
| krotscheck | robcresswell: For ^^, what solution are y'all going after? | 14:56 |
| fungi | krotscheck: any last-minute alterations/additions you needed t make to that repo before i approve it for import? | 14:57 |
| krotscheck | fungi: Nope. I know there's some work to be done, i.e. add license headers, but I can do those afterwards. | 14:58 |
| krotscheck | Unless mordred, AJaeger, or jeblair have comments. | 14:58 |
| mordred | I think it's awesome | 14:58 |
| mordred | of course, I was in favor of it the instant it had afs in the name | 14:58 |
| fungi | afs is the reviewer trump card | 14:58 |
| fungi | [approved because: afs] | 14:59 |
| *** jtomasek has quit IRC | 14:59 | |
| krotscheck | mordred: So, if I created an app called afs-rm-dash-r-star, you'd approve it? | 14:59 |
| *** cdent has quit IRC | 14:59 | |
| mordred | krotscheck: totally. and run it on my laptop | 14:59 |
| krotscheck | ....ooookay. I'll be right back. | 14:59 |
| krotscheck | ;) | 15:00 |
| mordred | krotscheck: of course, it didn't say afs-rm-dash-r-star-then-vos-release ... so it's safe ;) | 15:00 |
| krotscheck | mordred: Well, we can address that issue in v2 | 15:00 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit initial json https://review.openstack.org/304177 | 15:01 |
| *** bkeller` has quit IRC | 15:01 | |
| *** fawadkhaliq has joined #openstack-infra | 15:01 | |
| fungi | vikram_: done | 15:01 |
| mordred | krotscheck: I'd say we could talk about design for it at #cubaconference - but it conflicts with the summit | 15:01 |
| *** bkeller` has joined #openstack-infra | 15:02 | |
| vikram_ | fungi : Thanks | 15:02 |
| *** amitgandhinz has quit IRC | 15:02 | |
| krotscheck | mordred: whaaa.... apparently that's a thing. | 15:02 |
| *** amitgandhinz has joined #openstack-infra | 15:02 | |
| krotscheck | mordred: Have I told you recently about my conversation with the phantomjs people vis-a-vis having a mirror more reliable than bitbucket? | 15:03 |
| krotscheck | mordred: Their response was: We have a mirror, here it is *points at china* | 15:03 |
| *** salv-orlando has joined #openstack-infra | 15:03 | |
| *** salv-orlando has quit IRC | 15:03 | |
| *** jtomasek has joined #openstack-infra | 15:04 | |
| *** salv-orlando has joined #openstack-infra | 15:04 | |
| robcresswell | krotscheck: Using firefox rather than phantom for now. Its only used for the UTs, so we can remove it from the dependencies quite easily. | 15:04 |
| mordred | krotscheck: wow | 15:05 |
| krotscheck | robcresswell: Alrightey. | 15:05 |
| mordred | krotscheck: so, is our npm mirror about to become the world's best npm mirror? | 15:05 |
| krotscheck | robcresswell: I'll keep an eye on it to see if anything ever progresses. | 15:05 |
| *** apoorvad has joined #openstack-infra | 15:05 | |
| krotscheck | mordred: I dunno about that. | 15:05 |
| krotscheck | mordred: But it's about to become the world's only public npm mirror. | 15:05 |
| mordred | oh. well | 15:06 |
| mordred | if it's te world's only public npm mirror | 15:06 |
| krotscheck | npm-inc asked all the other mirrors to shut down because CDN | 15:06 |
| mordred | then I think we win 'best' by default :) | 15:06 |
| mordred | ah | 15:06 |
| mordred | yeah | 15:06 |
| openstackgerrit | Merged openstack-infra/project-config: Add missing acl-config reference for os-collect-config to os-apply-config.config https://review.openstack.org/303984 | 15:06 |
| mordred | the old "cdn solves everything" talke | 15:06 |
| mordred | tale | 15:06 |
| krotscheck | Which, of course, really hasn't impacted their reliability at all. | 15:06 |
| krotscheck | Plus or minus | 15:06 |
| mordred | totally | 15:06 |
| AJaeger | jeblair: could you review https://review.openstack.org/#/c/304164/ again, please? I updated based on jhesketh's review... | 15:06 |
| *** abregman has quit IRC | 15:08 | |
| krotscheck | mordred: Speaking of which, go bleed on this with a red pen -> https://review.openstack.org/#/c/235617/ | 15:08 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Summit initial json https://review.openstack.org/304177 | 15:08 |
| *** jsavak has quit IRC | 15:09 | |
| * krotscheck is switching internets. His home network is terribad right now. | 15:10 | |
| *** jsavak has joined #openstack-infra | 15:10 | |
| fungi | the internets are all terribad | 15:10 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 15:11 |
| *** sputnik13 has joined #openstack-infra | 15:11 | |
| fungi | these are not the bits you're looking for | 15:11 |
| mordred | krotscheck: registry-static errors on incomplete mirroring runs I'm assuming? | 15:11 |
| openstackgerrit | Merged openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra https://review.openstack.org/304108 | 15:12 |
| vincentll | when I attempt to set my contact information I get a "Server Error - Cannot store contact information". Is it a known error? | 15:12 |
| *** hdd has quit IRC | 15:13 | |
| *** xarses has joined #openstack-infra | 15:13 | |
| *** apoorvad has quit IRC | 15:14 | |
| *** yamahata has quit IRC | 15:15 | |
| *** arxcruz has quit IRC | 15:15 | |
| *** Jeffrey4l_ has joined #openstack-infra | 15:15 | |
| *** achanda has joined #openstack-infra | 15:15 | |
| *** sputnik13 has quit IRC | 15:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/devstack-gate: Test commit https://review.openstack.org/303032 | 15:16 |
| *** akaszuba has quit IRC | 15:17 | |
| *** piet has joined #openstack-infra | 15:17 | |
| *** openstackgerrit has quit IRC | 15:18 | |
| *** openstackgerrit has joined #openstack-infra | 15:18 | |
| jhesketh | jeblair: decided to create a list for v2 now: https://etherpad.openstack.org/p/zuul_changes_to_consider | 15:18 |
| *** fawadkhaliq has quit IRC | 15:18 | |
| jhesketh | (clearly not urgent, but please put your thoughts in there when you can) | 15:19 |
| *** hdd has joined #openstack-infra | 15:19 | |
| jhesketh | jeblair: I've been leaving zuulv2 patches for you to merge.. but maybe we should come up with a rule as to what is okay (if anything) to merge into master while we're working on v3 | 15:19 |
| jhesketh | jeblair: and on the topic of zuul, I've put up a couple of small changes to the v3 branch fyi | 15:19 |
| *** thorst has joined #openstack-infra | 15:20 | |
| jeblair | jhesketh: cool, thanks! | 15:20 |
| *** fawadkhaliq has joined #openstack-infra | 15:21 | |
| *** yamamoto has joined #openstack-infra | 15:21 | |
| anteaya | vincentll: yes, look at https://ask.openstack.org/en/question/56720/cannot-store-contact-information-when-updating-info-in-openstack-gerrit/ | 15:21 |
| *** e0ne has quit IRC | 15:21 | |
| anteaya | vincentll: and see if it helps you | 15:21 |
| amrith | AJaeger, I've logged it in LP https://bugs.launchpad.net/trove/+bug/1568876 and have pinged the folks working on dashboard to take a look | 15:22 |
| openstack | Launchpad bug 1568876 in OpenStack DBaaS (Trove) "trove-dashboard python27 gate job fails" [High,New] | 15:22 |
| fungi | jpmaxman: jamesmcarthur: i don't see smarcet in here at the moment, but i approved https://review.openstack.org/293491 just now. please take note of the conditions mentioned in my approval comment (i'll remind smarcet the next time i see him in here as well) | 15:22 |
| vincentll | anteaya: thanks i'll have a look | 15:22 |
| *** e0ne has joined #openstack-infra | 15:23 | |
| anteaya | vincentll: may I ask what documentation you were following to set up your account? | 15:23 |
| *** matt-borland has joined #openstack-infra | 15:23 | |
| *** haypo has left #openstack-infra | 15:23 | |
| *** zeih has quit IRC | 15:23 | |
| *** njohnston has joined #openstack-infra | 15:24 | |
| vincentll | anteaya: I was actually trying to do my first "git review" on a new stackforge project, the prompt asked me first to accept the Contributor Agreement, and then the contact information | 15:24 |
| anteaya | vincentll: ah, so no documentation, might I suggest some? | 15:24 |
| vincentll | anteaya: yes please | 15:24 |
| anteaya | vincentll: http://docs.openstack.org/infra/manual/developers.html#getting-started | 15:25 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Don't reload connections on HUP https://review.openstack.org/281166 | 15:25 |
| anteaya | vincentll: read everything about setting up accounts very closely | 15:25 |
| anteaya | there are no extraneous steps in that documenation | 15:25 |
| anteaya | vincentll: and welcome to openstack | 15:25 |
| *** thorst has quit IRC | 15:25 | |
| vincentll | anteaya: thanks! | 15:26 |
| *** daemontool_ has joined #openstack-infra | 15:26 | |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Make gate-trove-python34-db voting; add gate job https://review.openstack.org/304030 | 15:26 |
| anteaya | vincentll: welcome | 15:26 |
| *** daemontool_ has quit IRC | 15:26 | |
| *** daemontool_ has joined #openstack-infra | 15:27 | |
| *** yamamoto has quit IRC | 15:27 | |
| *** daemontool has quit IRC | 15:28 | |
| clarkb | morning | 15:28 |
| vincentll | oh, one more question, when importing my project into the newly created stackforge project, is it ok to keep the git history from my project? Or should I squash all the commits into one (as the git review prompt suggested)? | 15:28 |
| anteaya | morning clarkb | 15:28 |
| *** hdd has quit IRC | 15:28 | |
| anteaya | vincentll: what do you mean importing your project into the newly created stackforge project? | 15:29 |
| anteaya | vincentll: which stackforge project and which is your project? | 15:29 |
| anteaya | we have a step where a project is imported when a project is created on our gerrit | 15:29 |
| anteaya | did you know about the project import step during project creation? | 15:29 |
| *** mylu has quit IRC | 15:30 | |
| clarkb | is the gerrit lucene index on the volume device(s)? | 15:30 |
| clarkb | wondering ifwe will need to reindex | 15:30 |
| anteaya | clarkb: that is a good question | 15:31 |
| *** exploreshaifali has joined #openstack-infra | 15:31 | |
| anteaya | I don't think anyone has asked that yet | 15:31 |
| mordred | clarkb: hrm. I have no idea where that index is stored :) | 15:32 |
| mordred | Shrews, infra-root: woot! https://review.openstack.org/#/q/status:open+project:openstack-infra/shade+branch:master+topic:shade-nodepool is green and ready for review | 15:32 |
| pabelanger | ack | 15:33 |
| *** ajmiller has quit IRC | 15:33 | |
| clarkb | it is in review_site/index so will move with the volumes | 15:33 |
| vincentll | anteaya: yes, I did know about the import step, but at that time our git repo was not ready to be imported (the stackforge project is http://git.openstack.org/cgit/openstack/synergy-service/) | 15:33 |
| anteaya | clarkb: are you purple on the etherpad? | 15:33 |
| *** lakshmiS has joined #openstack-infra | 15:34 | |
| clarkb | anteaya: yes | 15:34 |
| anteaya | clarkb: wonderful | 15:34 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Archive /etc/openstack/clouds.yaml in gate runs https://review.openstack.org/303838 | 15:34 |
| fungi | vincentll: you can propose all your commits as individual changes for review and approve them one by one, or you can squash them all into one change. it's really up to you and your contributors/reviewers preference-wise | 15:34 |
| openstackgerrit | Merged openstack-infra/project-config: Add checks for gerrit ACLs https://review.openstack.org/304164 | 15:34 |
| openstackgerrit | Merged openstack-infra/system-config: Add user smarcet to openstackid production server https://review.openstack.org/293491 | 15:34 |
| vincentll | fungi: ok, thanks for the clarification | 15:35 |
| *** Guest8989 has quit IRC | 15:35 | |
| krotscheck | mordred: I believe so. I will admit that I haven't run a full mirror, and therefore don't know if there's any annoying edge cases in the guts of the nm registry | 15:35 |
| mordred | krotscheck: kk | 15:36 |
| mordred | krotscheck: we'll learn about them | 15:36 |
| jhesketh | okay, I'm off... good luck with the gerrit switchover, sorry I'm not around to help | 15:36 |
| mordred | jhesketh: woot! enjoy your night! | 15:36 |
| anteaya | jhesketh: enjoy sleep! | 15:36 |
| krotscheck | mordred: I've got the personal email of the maintainer over at yahoo, so I can ask him for help if we run into anything. | 15:36 |
| jpmaxman | Cool fungi thank you. Comments noted. | 15:36 |
| *** kengo_sakai_mba has joined #openstack-infra | 15:36 | |
| jhesketh | o/ | 15:36 |
| *** jsavak has quit IRC | 15:38 | |
| *** rbrndt__ has joined #openstack-infra | 15:38 | |
| *** harlowja_at_home has joined #openstack-infra | 15:38 | |
| *** mylu has joined #openstack-infra | 15:38 | |
| *** rbrndt has quit IRC | 15:39 | |
| fungi | yolanda: i added reverse dns entries on the replacement gerrit server just now. that would have caused a lot of e-mail to not get delivered | 15:40 |
| fungi | but this is far enough in advance the negative cache ttl should expire | 15:41 |
| anteaya | fungi: yolanda how long will it take to create the db backup do you think? | 15:41 |
| clarkb | anteaya: about 5 minutes | 15:41 |
| anteaya | awesome, thank you | 15:41 |
| krotscheck | infra-core: js-afs-blob-store just merged and was replicated, could I get someone to update the ACL's to add infra-core as approvers, and add myself to the js-afs-blob-store +2 group? | 15:41 |
| *** Swami has joined #openstack-infra | 15:41 | |
| *** kengo_sakai_mba has quit IRC | 15:41 | |
| *** yamamoto has joined #openstack-infra | 15:42 | |
| fungi | yolanda: any reason not to do step 4 before the maintenance (like preferably 30 minutes to an hour before)? | 15:42 |
| *** mylu has quit IRC | 15:43 | |
| yolanda | fungi yes, can be done before | 15:44 |
| *** chem has quit IRC | 15:44 | |
| *** dtantsur|mtg is now known as dtantsur|afk | 15:44 | |
| *** gyee has joined #openstack-infra | 15:45 | |
| yolanda | i will connect an hour before the maintenance to start with prev steps | 15:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Return boolean from delete_project https://review.openstack.org/304155 | 15:45 |
| *** Qiming has quit IRC | 15:45 | |
| fungi | yolanda: or one of us can do it so you don't need to be around that soon. just mostly concerned we'll end up waiting for an in-progress puppet run | 15:46 |
| yolanda | fungi, i'm reading about the volume changes, how do you have normally done in the past? using some script for that, or just using the ui? | 15:46 |
| *** vincentll has quit IRC | 15:46 | |
| *** chem has joined #openstack-infra | 15:46 | |
| *** bpokorny has joined #openstack-infra | 15:46 | |
| fungi | yolanda: usually use nova volume detach/attach | 15:46 |
| fungi | from the puppetmaster | 15:47 |
| yolanda | ++ | 15:47 |
| *** fawadkhaliq has quit IRC | 15:47 | |
| *** jaosorior_away has quit IRC | 15:47 | |
| fungi | yolanda: there are some examples at http://docs.openstack.org/infra/system-config/sysadmin.html#cinder-volume-management | 15:47 |
| *** ddieterly is now known as ddieterly[away] | 15:48 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] configure publist-to-pypi jobs for sahara-tests https://review.openstack.org/304121 | 15:48 |
| *** jaosorior_away has joined #openstack-infra | 15:48 | |
| *** mylu has joined #openstack-infra | 15:48 | |
| *** ifarkas has quit IRC | 15:49 | |
| *** jlanoux has quit IRC | 15:49 | |
| *** mrmartin has quit IRC | 15:49 | |
| yolanda | fungi thx | 15:49 |
| fungi | yolanda: in this case, we'll also need to do some lvm setup on the new server i think, since this isn't just a straight filesystem on a cinder block device but rather an logical volume. though vgscan will _probably) pick it up once attached i think | 15:49 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for phantomjs / horizon lint job bug 1568325 https://review.openstack.org/304173 | 15:50 |
| openstack | bug 1568325 in OpenStack Dashboard (Horizon) "Phantom is causing JS tests to fail constantly" [High,In progress] https://launchpad.net/bugs/1568325 - Assigned to Rob Cresswell (robcresswell) | 15:50 |
| yolanda | ok going to add some of these bits to the etherpad | 15:50 |
| *** ddieterly[away] is now known as ddieterly | 15:51 | |
| *** Sukhdev has joined #openstack-infra | 15:51 | |
| *** bpokorny has quit IRC | 15:51 | |
| fungi | also we had an old gerrit homedir (lv name was "gerritold") mounted at /mnt but i'm in the process of getting rid of that now so it doesn't confuse us later today | 15:52 |
| *** yamamoto has quit IRC | 15:52 | |
| *** bpokorny has joined #openstack-infra | 15:52 | |
| *** dtroyer_zz has quit IRC | 15:52 | |
| clarkb | fungi: thats what the second volume on the etherpad was, I was about to start recording info about that but you were ahead of me :) | 15:52 |
| *** dtroyer has joined #openstack-infra | 15:52 | |
| *** thorst has joined #openstack-infra | 15:56 | |
| clarkb | fungi: you should update the volume attach and detach commands if one of those volumes is that old fs | 15:56 |
| fungi | clarkb: oh, yep, this needs further cleanup. apparently we added the old and new cinder devices into one vg | 15:56 |
| clarkb | ah | 15:56 |
| fungi | still detangling | 15:56 |
| clarkb | in that case probably easier to clean up after the fact? | 15:56 |
| * clarkb defers to you as you are working it | 15:57 | |
| *** dprince has quit IRC | 15:57 | |
| fungi | well, i'm removing the old logvol, and then will see what's involved in a vgreduce (whether we have blocks for one lv on both pvs, which we likely don't) | 15:57 |
| *** dprince has joined #openstack-infra | 15:58 | |
| fungi | clarkb: chances are we want a larger cinder volume anyway as we're at 83% full on one 100gb volume for production already | 15:58 |
| *** mikelk has quit IRC | 15:58 | |
| *** jsavak has joined #openstack-infra | 15:59 | |
| *** Nikolay_St has quit IRC | 15:59 | |
| fungi | we could just lvextend this onto the second cinder volume, or pvmove onto a new larger one | 15:59 |
| fungi | also both are things which can be done online after the maintenance | 15:59 |
| yolanda | fungi clarkb, so we just need the main-gerrit volume, not the gerritold? i was documenting everything assuming two volumes, but looking at review, fstab, etc... and your comments, seems that one is old and trash? | 15:59 |
| fungi | yolanda: yes, i'm still working on cleaning that up so it should simplify teh maintenance steps | 16:00 |
| yolanda | thanks | 16:00 |
| *** dizquierdo has quit IRC | 16:01 | |
| *** dizquierdo has joined #openstack-infra | 16:02 | |
| anteaya | so 4 hours until gerrit goes offline | 16:02 |
| fungi | yep, after an lvremove of main-gerritold, /dev/xvdb1 is 100% free so i can vgreduce it out of main | 16:02 |
| anteaya | how do folks feel about issuing a channel status alert to folks as a reminder? | 16:02 |
| fungi | anteaya: i was going to suggest a #status notice (not alert) at around now since it's the 4-hour mark | 16:02 |
| yolanda | i'll step out for a while and will come back later | 16:02 |
| fungi | yolanda: thanks for working on this! | 16:03 |
| anteaya | fungi: ah, sorry to get in your way | 16:03 |
| anteaya | fungi: yes, I used the wrong word, notice | 16:03 |
| *** zeih has joined #openstack-infra | 16:03 | |
| *** armax has joined #openstack-infra | 16:03 | |
| anteaya | fungi: and I agree with your suggestion of a notice | 16:03 |
| clarkb | image builds failed this morning on a new git repo (I think there is a small race between creating the repo on git.o.o backends and on review.o.o and te replication to the backends | 16:04 |
| *** jaosorior_away has quit IRC | 16:04 | |
| clarkb | not the end of the world | 16:04 |
| fungi | yes, there is a race there | 16:04 |
| pabelanger | clarkb: Ya, ansible was hung for about 12 hours | 16:05 |
| *** jaosorior has joined #openstack-infra | 16:05 | |
| anteaya | clarkb: yes, I think that was what wznoinsk was trying to draw attention to last week when he was talking about dib breakages | 16:05 |
| clarkb | pabelanger: similar thing as to when noepool went away? | 16:05 |
| clarkb | anteaya: those were related to the nodepool server going away and ansible getting unhappy about it | 16:05 |
| fungi | did we decide on a duration for the maintenance window? one hour? more? | 16:05 |
| clarkb | anteaya: and were corrected same day | 16:05 |
| anteaya | clarkb: ah, thank you | 16:05 |
| pabelanger | clarkb: yes, looking into the --timeout flag for ansible-playbook to see if that helps resolve the blockage | 16:05 |
| anteaya | fungi: I think we had decided one hour but I'm not sure that was communicated | 16:06 |
| clarkb | fungi: considering we shouldn't need to reindenx or any of that an hour should be plenty | 16:06 |
| fungi | agreed | 16:06 |
| anteaya | fungi: just the start time was communicated in the initial email | 16:06 |
| fungi | something like #status notice Reminder, Gerrit will be offline from 20:00 to 21:00 UTC for a server replacement http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 16:06 |
| AJaeger | thanks, amrith for the bug! | 16:06 |
| anteaya | the followup email also doesn't appear to communicate duration | 16:06 |
| pleia2 | good morning | 16:07 |
| anteaya | fungi: works for me, thank you | 16:07 |
| anteaya | pleia2: good morning | 16:07 |
| *** jsavak has quit IRC | 16:07 | |
| *** jsavak has joined #openstack-infra | 16:07 | |
| fungi | okay, after a vgreduce and pvremove of /dev/xvdb1 on review.o.o i'm ready to detach that cinder volume | 16:08 |
| anteaya | yay! | 16:08 |
| fungi | #status notice Reminder, Gerrit will be offline from 20:00 to 21:00 UTC for a server replacement http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 16:08 |
| anteaya | wherefore art thou statusbot? | 16:08 |
| fungi | oh, i forgot statusbot was sleeping in. moments while i wake it up | 16:09 |
| anteaya | yup | 16:09 |
| anteaya | it is Monday after all | 16:09 |
| anteaya | meetbot seems to be online | 16:09 |
| *** openstackstatus has joined #openstack-infra | 16:10 | |
| *** ChanServ sets mode: +v openstackstatus | 16:10 | |
| *** tphummel has joined #openstack-infra | 16:11 | |
| *** jordanP has quit IRC | 16:11 | |
| fungi | i'm starting to think that we should configure statusbot not to hang out in the meeting channels. we tend to cause fights between it and meetbot over channel topics, and it's arguably disruptive to have it chiming in during meetings at all | 16:11 |
| pleia2 | +1 | 16:11 |
| fungi | on the other hand, there's the chance that a meeting channel is the only place some users are lurking | 16:11 |
| fungi | though i think that's probably a corner case not worth concerning ourselves with | 16:11 |
| pleia2 | yeah | 16:11 |
| *** Hal has joined #openstack-infra | 16:12 | |
| *** Hal is now known as Guest61323 | 16:13 | |
| anteaya | can statusbot still report in meeting channels but defer to meetbot on the topics? | 16:13 |
| *** jcoufal has quit IRC | 16:14 | |
| *** julim has quit IRC | 16:14 | |
| pleia2 | it doesn't currently have that capability | 16:14 |
| *** david-lyle_ is now known as david-lyle | 16:15 | |
| anteaya | ah okay, well then I agree with the corner case not worth concerning ourselves with | 16:15 |
| fungi | some infra-root want to check behind me before i `nova volume-detach review.openstack.org 4ae4bfbf-c90e-4bad-a9d8-4c1f70604ac1` ? | 16:15 |
| *** fawadkhaliq has joined #openstack-infra | 16:16 | |
| *** armax has quit IRC | 16:16 | |
| fungi | cinder show says it's /dev/xvdb on 87741fa9-92cb-4d1e-8195-6841e2468fb2 (the old, current production review.o.o) | 16:16 |
| pleia2 | I'll have a look, what should it be? | 16:16 |
| *** fawadkhaliq has quit IRC | 16:16 | |
| clarkb | thats unfortunate /dev/by-id doesn't have the cinder info for these devices (guessing due to lvm?) | 16:16 |
| fungi | i'm just trying to make doubly-sure i'm not about to detach the in use cinder volume for /home/gerrit2 | 16:17 |
| fungi | #status notice Reminder, Gerrit will be offline from 20:00 to 21:00 UTC for a server replacement http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 16:17 |
| openstackstatus | fungi: sending notice | 16:17 |
| fungi | the name and creation date on that volume are also earlier than the presumed active one, so i think it should be fine | 16:18 |
| clarkb | fungi: sudo pvs says xvdc1 is the only device backing lvm right now. I am assuming xvdb is the one you are trying to remove, I think a nova volume-show will confirm that | 16:18 |
| *** gongysh has quit IRC | 16:18 | |
| -openstackstatus- NOTICE: Reminder, Gerrit will be offline from 20:00 to 21:00 UTC for a server replacement http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 16:18 | |
| * clarkb runs thta command to check | 16:18 | |
| *** e0ne has quit IRC | 16:18 | |
| robcresswell | Hey all. What's the process for adding to horizon-stable-maint? Is it a dev mailer process, or patched etc? | 16:19 |
| robcresswell | Sorry if this isn't the correct place to ask. | 16:19 |
| anteaya | robcresswell: adding what to horizon-stable-maint? | 16:19 |
| anteaya | a person? | 16:19 |
| fungi | clarkb: hrm, it complains i don't have a cinder v2 endopoint in the catalog | 16:20 |
| AJaeger | robcresswell: the ACL is self-maintained, ask one of the people in it to add you if you're not in there - and if you're there, do it yourself ;) | 16:20 |
| openstackstatus | fungi: finished sending notice | 16:20 |
| openstackgerrit | Merged openstack-infra/project-config: [sahara] disable releasenotes jobs for liberty and kilo branch https://review.openstack.org/304105 | 16:20 |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: cleaning up python3 jobs for sahara-tests https://review.openstack.org/304130 | 16:20 |
| clarkb | fungi: | 4ae4bfbf-c90e-4bad-a9d8-4c1f70604ac1 | review.openstack.org/main01 | in-use | 100 | Attached to review.openstack.org on /dev/xvdb | | 16:20 |
| clarkb | fungi: I had to use openstack client | 16:20 |
| robcresswell | anteaya: Yes | 16:20 |
| zaro | morning | 16:20 |
| anteaya | robcresswell: great, I think AJaeger's response is probably the best one | 16:20 |
| anteaya | zaro: morning zaro | 16:21 |
| anteaya | zaro: we are here: https://etherpad.openstack.org/p/gerrit_server_replacement | 16:21 |
| robcresswell | AJaeger: None of the existing members are able to add; the group itself is owned by stable-maint-core so I assumed it was managaed by those. | 16:21 |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: cleaning up python3 jobs for sahara-tests https://review.openstack.org/304130 | 16:21 |
| *** vgridnev has quit IRC | 16:21 | |
| AJaeger | robcresswell: then ask the stable team for it | 16:21 |
| robcresswell | AJaeger: Will do, thanks. | 16:21 |
| fungi | clarkb: which osc verb do i want? is it under server? | 16:21 |
| clarkb | fungi: openstack --os-cloud openstackci-rax --os-region-name DFW volume list | 16:22 |
| fungi | clarkb: that's calling into volume not server though, so equivalent to cinderclient not novaclient? | 16:22 |
| clarkb | yes | 16:22 |
| clarkb | the message I got from nova client was use cinderclient or osc | 16:22 |
| fungi | i got the same info out of cinder show already | 16:22 |
| *** cdent has joined #openstack-infra | 16:23 | |
| *** jogo has quit IRC | 16:24 | |
| fungi | so i think i'll amend the instructions to only move the volume we're 99+% sure is the one in use, and then delete the other one when we delete the old server | 16:24 |
| clarkb | 2fm | 16:24 |
| clarkb | er | 16:24 |
| clarkb | wfm | 16:25 |
| fungi | if we make the db backup into /home/gerrit2 then we can also use that as a canary to let us know we've got the right volume | 16:25 |
| anteaya | do we say in the instructions how long we are going to wait before we delete the old server? | 16:25 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config: Remove statusbot from meeting channels https://review.openstack.org/304213 | 16:25 |
| *** jogo has joined #openstack-infra | 16:25 | |
| fungi | anteaya: we don't say, but i'd give it a couple days | 16:25 |
| AJaeger | team , here's the suggested statusbot removal on meeting channels ^ | 16:25 |
| anteaya | fungi: great | 16:25 |
| anteaya | AJaeger: oh I had thought we had said we weren't going to bother | 16:26 |
| *** esker has quit IRC | 16:26 | |
| anteaya | AJaeger: as there are some cases where having it report in meeting channels can be useful | 16:26 |
| pabelanger | clarkb: have we ever talked about using async in our playbooks for puppet before? | 16:26 |
| pleia2 | anteaya: I think AJaeger is right, keeping it *in* the channel was a corner case | 16:26 |
| fungi | yes, that's what i attempted to say (perhaps not clearly) | 16:27 |
| *** scheuran has quit IRC | 16:27 | |
| clarkb | pabelanger: I don't think so, not sure we would need to with only ~10 connections at any time | 16:27 |
| AJaeger | anteaya: let me reread the dsicussion, seems I misunderstood it. | 16:27 |
| anteaya | AJaeger: or perhaps I did | 16:27 |
| *** yamahata has joined #openstack-infra | 16:27 | |
| fungi | AJaeger: nope, your change is what i was suggesting. i'm in favor, i just think we should collect some input on it before we approve | 16:28 |
| esikachev | guys, we need review https://review.openstack.org/#/c/304121/ this needs for release | 16:28 |
| *** fawadkhaliq has joined #openstack-infra | 16:28 | |
| *** harlowja_at_home has quit IRC | 16:29 | |
| AJaeger | anteaya: I interpreted your answer the way that you're in favor of the change I did ("I agree with the corner case not worth concerning ourselves with") | 16:29 |
| pabelanger | clarkb: Ya, my thought ansible gets less flaky when the far end disappears when using async, since it reconnects ever x seconds. | 16:29 |
| *** armax has joined #openstack-infra | 16:29 | |
| AJaeger | esikachev: please explain what the change is, which repo it touches and why it blocks you. | 16:29 |
| pabelanger | but not sure that is the best solution either | 16:29 |
| AJaeger | esikachev: not everybody here reviews everything... | 16:29 |
| fungi | "people only lurking in meeting channels and nowhere else" was what i was saying i felt was a corner case we could reasonably ignore | 16:30 |
| *** abregman has joined #openstack-infra | 16:30 | |
| *** bkeller` has quit IRC | 16:30 | |
| anteaya | fungi: ah I mis-understood you | 16:30 |
| anteaya | I've commented on the patch | 16:30 |
| anteaya | it is possible I am in the minority | 16:30 |
| fungi | anteaya: not at all, that's why i encouraged further discussion even though i'm in favor | 16:31 |
| anteaya | yup, thanks | 16:31 |
| fungi | feel free to make your comment a -1 too if you have a strong enough opinion on it | 16:31 |
| *** bkeller` has joined #openstack-infra | 16:31 | |
| anteaya | no not strong enough for a -1 | 16:32 |
| anteaya | just registering what I see as a usecase | 16:32 |
| *** apoorvad has joined #openstack-infra | 16:33 | |
| *** hdd has joined #openstack-infra | 16:33 | |
| mfisch | can we get openstack-security as a tag that I can subscibe to? | 16:33 |
| mfisch | since my mail to openstack-sec..@lists was redirected to -dev | 16:33 |
| *** jaosorior has quit IRC | 16:33 | |
| clarkb | mfisch: you need to contact the list admins for the list in question | 16:33 |
| AJaeger | btw. if my change merges, let's merge https://review.openstack.org/299009 - kolla to #openstack-kolla rename - and https://review.openstack.org/264307 (powervm) as well ,please | 16:33 |
| clarkb | mfisch: they should be listed on the lists.openstack.org page for that specific list | 16:34 |
| fungi | mfisch: official subject tags need to be set by the list admin, so ttx in this case according to the details at the bottom of http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev | 16:34 |
| zaro | fungi: what do you mean 'all content' from /home/gerrit2? | 16:34 |
| mfisch | looks like thieerry is admin | 16:34 |
| fungi | zaro: anything inside /home/gerrit2 on the root filesystem, which will get shadowed when we mount the logical volume there | 16:35 |
| zaro | fungi: what else is in there besides review_site? | 16:36 |
| fungi | zaro: dotfiles/dotdirs from /etc/skel at a minimum | 16:37 |
| zaro | fungi: ohh, i see, you mean clean it out in prep for existing gerrit2 content | 16:38 |
| zaro | yeah, good idea i guess. | 16:38 |
| *** Guest61323 has quit IRC | 16:38 | |
| *** rcernin has quit IRC | 16:40 | |
| fungi | right, it's hard to delete files off the parent filesystem after you mount a new filesystem on top of them | 16:43 |
| fungi | so easier to move or delete them out of the way before anything is mounted there | 16:43 |
| *** jaosorior has joined #openstack-infra | 16:44 | |
| fungi | clarkb: thinking more about the reboot idea, there's a bit of a chicken and egg with turning puppet back on | 16:45 |
| clarkb | fungi: is it? | 16:45 |
| *** sputnik13 has joined #openstack-infra | 16:45 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again https://review.openstack.org/288915 | 16:45 |
| fungi | clarkb: we want to reenable puppet and test that it's working before we reboot i think? since services (gerrit/gerritbot) are likely to start automatically at boot | 16:45 |
| fungi | or maybe we've already got them well disabled. checking | 16:46 |
| clarkb | in that case I think we can just run puppet first, then reboot? | 16:46 |
| clarkb | basically we should reboto at some point during the time window | 16:46 |
| fungi | i agree we should reboot to test that it's safe | 16:47 |
| fungi | i'm just wondering to what extent we can rely on the reboot itself to do things for us (like finding, activating and mounting the logvol) | 16:47 |
| anteaya | did we discuss at all if we are going to address the zuul memory usage issue during the gerrit maintenance window? | 16:47 |
| clarkb | fungi: you mean the /home/gerrit2 volume? | 16:48 |
| fungi | clarkb: yep | 16:48 |
| *** thiagop has joined #openstack-infra | 16:48 | |
| clarkb | ya may need to just mount -a after editing fstab, run puppet, then reboot later | 16:48 |
| *** fawadkhaliq has quit IRC | 16:48 | |
| fungi | anteaya: it's probably a good time to do a zuul restart (in fact, i was thinking that we probably need to stop zuul before this maintenance anyway and start it again after), though no idea if there were patches jhesketh and jeblair wanted to try getting in place in advance of that restart | 16:49 |
| *** fawadkhaliq has joined #openstack-infra | 16:49 | |
| anteaya | fungi: can you add some zuul steps in the etherpad, then I can see if I can track down patch urls if any apply? | 16:49 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Temporarily disable mistral on the undercloud https://review.openstack.org/302832 | 16:51 |
| *** infra-red has joined #openstack-infra | 16:51 | |
| fungi | anteaya: just added, yes | 16:51 |
| anteaya | fungi: thank you | 16:51 |
| anteaya | jeblair: I have lost track, I saw you and jhesketh discussing https://review.openstack.org/#/c/281166/ earlier, did you want that patch to merge prior to restarting zuul today? | 16:51 |
| *** yamamoto has joined #openstack-infra | 16:52 | |
| *** esikachev has quit IRC | 16:53 | |
| *** bkeller` has quit IRC | 16:53 | |
| *** dutsmoc is now known as comstud | 16:54 | |
| *** dtantsur|afk is now known as dtantsur | 16:55 | |
| *** derekh has quit IRC | 16:57 | |
| *** dprince has quit IRC | 16:57 | |
| *** daemontool__ has joined #openstack-infra | 16:57 | |
| *** dprince has joined #openstack-infra | 16:58 | |
| *** julim has joined #openstack-infra | 16:59 | |
| *** yamamoto has quit IRC | 17:00 | |
| *** daemontool_ has quit IRC | 17:00 | |
| jeblair | anteaya, fungi: yeah, i will restart zuul with that patch manually applied | 17:02 |
| jeblair | anteaya, fungi: (and only that patch -- so the foreign project support will be back in) | 17:03 |
| *** vgridnev has joined #openstack-infra | 17:04 | |
| fungi | i've cleaned up/expanded on the maintenance steps further to include the steps for mounting the homedir and suggested reboot | 17:04 |
| fungi | and moved the puppet disablement to pre-maintenance | 17:05 |
| *** flepied has joined #openstack-infra | 17:05 | |
| jeblair | fungi: oh, we should remove the init script links on old gerrit | 17:05 |
| *** fawadkhaliq has quit IRC | 17:06 | |
| jeblair | ('reboot' made me think of that) | 17:06 |
| fungi | good call | 17:06 |
| anteaya | jeblair: thank you | 17:06 |
| *** jamesmcarthur has quit IRC | 17:07 | |
| *** Hal has joined #openstack-infra | 17:07 | |
| *** Hal is now known as Guest5162 | 17:07 | |
| *** hdd has quit IRC | 17:07 | |
| fungi | jeblair: added in step 1.1 | 17:08 |
| *** edmondsw has quit IRC | 17:08 | |
| fungi | right after we stop the services | 17:08 |
| *** flepied1 has quit IRC | 17:08 | |
| jeblair | ++ | 17:08 |
| anteaya | I have edited step 7.5 to added manually applying the contents of 281166 to zuul prior to restart | 17:08 |
| fungi | noted explicitly to stop gerritbot on the old server too, that was missed | 17:08 |
| fungi | thanks anteaya | 17:09 |
| anteaya | thank you | 17:09 |
| fungi | at this point all the suggestions i can think of are addressed in the etherpad | 17:09 |
| anteaya | fungi: yay | 17:09 |
| *** fawadkhaliq has joined #openstack-infra | 17:09 | |
| fungi | but we still have a few hours to go if anyone realizes something's missing/wrong | 17:09 |
| *** dizquierdo has quit IRC | 17:10 | |
| AJaeger | anteaya, fungi: is applying https://review.openstack.org/303846 on the agenda? | 17:10 |
| anteaya | AJaeger: yes, line 64 | 17:11 |
| AJaeger | That's the heap size change | 17:11 |
| AJaeger | thanks, anteaya ! | 17:11 |
| AJaeger | should I remove my WIP? | 17:11 |
| anteaya | well it isn't a step but the patch is noted | 17:11 |
| *** bpokorny has quit IRC | 17:11 | |
| anteaya | AJaeger: I thinking removing the WIP at this point should be okay | 17:11 |
| anteaya | s/thinking/think | 17:11 |
| AJaeger | done | 17:11 |
| AJaeger | thanks, anteaya | 17:11 |
| anteaya | thanks for the patch! | 17:12 |
| *** ihrachys has quit IRC | 17:13 | |
| *** jsavak has quit IRC | 17:13 | |
| fungi | wondering whether that should happen after we disable puppet for the old gerrit server but before the maintenance? or if we should plan to add that after gerrit's back up and then do a separate gerrit restart to pick it up | 17:13 |
| *** jsavak has joined #openstack-infra | 17:13 | |
| *** e0ne has joined #openstack-infra | 17:14 | |
| anteaya | would there be any benefit to adding it after gerrit is back up? | 17:14 |
| anteaya | I'm leaning towards one gerrit stop/start myself | 17:15 |
| fungi | it's one more thing to have to revert if we roll back to the old server | 17:15 |
| anteaya | true | 17:15 |
| anteaya | okay well I guess I quick gerrit restart afterwards isn't a huge inconvience | 17:15 |
| clarkb | considering it is the reason for this whole swith we should try to get it in soon | 17:16 |
| anteaya | and having a clean roll back path is important | 17:16 |
| *** tonytan4ever has quit IRC | 17:16 | |
| anteaya | clarkb: oh I think we are agreeing the change happens right after the migration | 17:16 |
| anteaya | just not the same as the migration | 17:16 |
| anteaya | at least that I what I'm agreeing to | 17:17 |
| fungi | mainly a question of whether we do only one gerrit restart and so squeeze it in before we take zuul and old gerrit offline, or whether we do a second gerrit restart with that chance in place after we've verified it's up and working | 17:18 |
| fungi | s/chance/change/ | 17:18 |
| *** nadya has joined #openstack-infra | 17:18 | |
| anteaya | vote option two | 17:18 |
| jeblair | vote option one | 17:19 |
| fungi | i could go either way, though option will make the maintenance slightly shorter (assuming we don't need to roll back) | 17:20 |
| *** bpokorny has joined #openstack-infra | 17:20 | |
| fungi | er, option one will | 17:20 |
| *** skolekonov is now known as skolekonov_away | 17:20 | |
| dims | fungi : is it easy to tickle the "propose-requirements-constraints-master" periodic job? the run this morning failed and i wanted to see if it happens again. there's no rush at all, just curiousity | 17:22 |
| *** tonytan4ever has joined #openstack-infra | 17:22 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks https://review.openstack.org/291393 | 17:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add common bash functions to help track metrics. https://review.openstack.org/291392 | 17:24 |
| nibalizer | I think option one as well | 17:25 |
| clarkb | ya I think we should just do it | 17:26 |
| clarkb | it is a really easy revert | 17:26 |
| krotscheck | infra-root: I'd like to grab the attention of someone this week to help land https://review.openstack.org/#/c/235617/ - does anyone have a few hours, say, wednesday? | 17:26 |
| nibalizer | safe change i think | 17:26 |
| pabelanger | agree | 17:26 |
| krotscheck | Work would involve copy/pasting logs so i can debug them. | 17:26 |
| pleia2 | wednesday is our bug day | 17:27 |
| *** e0ne has quit IRC | 17:27 | |
| krotscheck | Hrm. | 17:27 |
| pabelanger | pleia2: nice, forgot about that | 17:27 |
| krotscheck | Tomorrow's my meeting day | 17:27 |
| *** lakshmiS has quit IRC | 17:27 | |
| pabelanger | krotscheck: if you haven't found anybody else, feel free to ping me | 17:27 |
| krotscheck | pabelanger: Awesome, thanks :) | 17:28 |
| fungi | dims: not sure if we can reenqueue periodic jobs, but i can give it a shot | 17:28 |
| *** trown is now known as trown|lunch | 17:28 | |
| nibalizer | krotscheck: i have cycles this week :) | 17:28 |
| krotscheck | nibalizer: Do you have cycles... right now? ;) | 17:28 |
| *** lakshmiS has joined #openstack-infra | 17:29 | |
| *** jsavak has quit IRC | 17:29 | |
| dims | thanks fungi | 17:29 |
| *** Rockyg has joined #openstack-infra | 17:29 | |
| *** e0ne has joined #openstack-infra | 17:30 | |
| nibalizer | krotscheck: no, today is break gerrit day | 17:30 |
| krotscheck | nibalizer: Wednesday is bug day. Tomorrow I've got meetings all other the place, so I can't pay 100% attention to what's going on. | 17:31 |
| jeblair | krotscheck: 617 needs an update | 17:31 |
| krotscheck | jeblair: Checking | 17:31 |
| *** jaosorior has quit IRC | 17:31 | |
| nibalizer | krotscheck: thursday? | 17:31 |
| fungi | dims: looks like zuul enqueue-ref was able to do it | 17:31 |
| fungi | sudo zuul enqueue-ref --trigger timer --pipeline periodic --project openstack/requirements --ref refs/heads/master --newrev d4326d4ced84677ed37f5709cf0637b092044e9e | 17:31 |
| *** ddieterly is now known as ddieterly[away] | 17:31 | |
| fungi | dims: i see it running on the zuul status page now | 17:31 |
| fungi | insta-failed | 17:32 |
| dims | ah nice! thanks fungi | 17:32 |
| *** jsavak has joined #openstack-infra | 17:32 | |
| fungi | Command '['/tmp/tmpcJLYgc/bin/pip', 'install', '-r', 'global-requirements.txt']' returned non-zero exit status 2 | 17:32 |
| krotscheck | nibalizer: ...thursday I have a thing at the Google offices here in kirkland, but if you're available around 7AM, I'm sure we can get it landed :) | 17:32 |
| dims | saved it in case ... https://gist.github.com/dims/3f0d83cb6d2f0cd6751b9f92896b4fea | 17:32 |
| jeblair | krotscheck: does the principal exist? | 17:32 |
| dims | y that was the same problem from before | 17:32 |
| dims | fungi : thanks, will try to repro locally | 17:33 |
| krotscheck | jeblair: I don't think so. | 17:33 |
| fungi | dims: so this may be related to the "need to install libvirt to pip install python-libvert latest release? | 17:33 |
| fungi | dims: we probably also collect the tox logs from this. checking now | 17:33 |
| jeblair | krotscheck: ok; that should be done before landing it | 17:33 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added NPM Mirror to mirror_update https://review.openstack.org/235617 | 17:33 |
| fungi | dims: ahh, nope, we just collect the console log for that job | 17:34 |
| dims | fungi : ack | 17:34 |
| nibalizer | so is this the only review for today's activity ? https://review.openstack.org/#/q/topic:heap30g | 17:34 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added NPM Mirror to mirror_update https://review.openstack.org/235617 | 17:34 |
| krotscheck | jeblair: Added a note to the commit message, and addressed your issues ^^ | 17:34 |
| jeblair | krotscheck: w00t; +2 | 17:35 |
| anteaya | nibalizer: you can also review: https://review.openstack.org/#/c/281166/ which will be manually applied to zuul prior to restart as outlined in step 7.5 | 17:35 |
| anteaya | nibalizer: but yes 303846 is the only patch we need to merge today | 17:35 |
| fungi | dims: anyway, i saw someone say that global requirements containing libvirt-python now means you have to install the libvirt (library? headers?) from your distro since the 1.3.3 release late last week | 17:36 |
| dims | fungi : thanks let me see if i can find anything | 17:37 |
| *** vikram_ has quit IRC | 17:38 | |
| AJaeger | team, seems proposal slave is out of space see https://jenkins.openstack.org/job/neutron-upstream-translation-update/2832/console | 17:38 |
| AJaeger | "IOError: [Errno 28] No space left on device" | 17:38 |
| fungi | AJaeger: oh, that could explain dims's issue too | 17:38 |
| anteaya | AJaeger: :/ | 17:38 |
| fungi | looking into it now | 17:38 |
| AJaeger | thanks, fungi! | 17:38 |
| fungi | says /dev/xvda1 40G 37G 544M 99% / (so almost full anyway) | 17:39 |
| dims | AJaeger : yay nice find | 17:39 |
| AJaeger | dims: I just saw some translation jobs failing - and those shouldn't fail ;) | 17:39 |
| dims | haha | 17:40 |
| AJaeger | dims: but wouldn't have checked without your inquiery | 17:40 |
| AJaeger | dims, thanks for asking! | 17:40 |
| fungi | chances are we just don't have enough space in the homedir for all the workspaces which build up over time, but i should know the culprit in a bit | 17:40 |
| fungi | 28gb in /home/jenkins | 17:40 |
| fungi | (this is one of the reasons we moved most jobs to single-use workers) | 17:41 |
| fungi | yeah, by far the bulk of the rootfs utilization is from /home | 17:41 |
| AJaeger | we create now a venv for the releasenotes and don't clean that one up ;( | 17:42 |
| fungi | pleia2: you have nearly 2gb of data in your homedir on proposal.slave.o.o, not sure what it is or whether you still need it | 17:42 |
| *** dangers_away has quit IRC | 17:42 | |
| AJaeger | I could remove that one... | 17:42 |
| *** oanson has joined #openstack-infra | 17:42 | |
| *** bkeller` has joined #openstack-infra | 17:42 | |
| *** sc68cal has quit IRC | 17:43 | |
| pleia2 | fungi: thanks, probably some translations stuff, I'll get it sorted | 17:43 |
| odyssey4me | pabelanger are we in a position to do experimental xenial checks on any jobs yet? | 17:43 |
| pabelanger | odyssey4me: Yup, we are asking projects not to move them from experimental to check / gate until after the conference. | 17:44 |
| fungi | propose-requirements-updates is by far the largest workspace at 1.8gb | 17:44 |
| pleia2 | fungi: ok, better now | 17:44 |
| clarkb | fungi: it clones all of the repos iirc | 17:44 |
| *** mrmartin has joined #openstack-infra | 17:44 | |
| clarkb | fungi: in order to generate the transitive dep list | 17:44 |
| pabelanger | odyssey4me: so, we could likely add experimental job for ansible | 17:44 |
| odyssey4me | pabelanger ah ok, cool - that'd be nice :) | 17:45 |
| fungi | though a lot of the translation-update jobs are big too | 17:45 |
| fungi | hundreds of mb each | 17:45 |
| odyssey4me | pabelanger another thing I needed to discuss - I'd like to add release announcement and release note jobs to the roles - should we add that to the template, or to individual jobs? | 17:45 |
| pabelanger | odyssey4me: should be easy to add it into ansible-role-jobs.yaml | 17:45 |
| anteaya | odyssey4me: just to give you an update on the rename of your repo: https://review.openstack.org/#/c/299192/ we haven't scheduled it yet | 17:46 |
| pabelanger | odyssey4me: Ya, we should have a template for sure. | 17:46 |
| fungi | for now, i'm going to take proposal.slave.o.o offline in jenkins for a few minutes and as soon as it's not running any jobs, clear out its workspace cache. separately i'll start working on adding a cinder volume for /home/jenkins on it | 17:46 |
| anteaya | odyssey4me: currently renames require gerrit to be offline for 4 hours | 17:46 |
| odyssey4me | anteaya thanks for the update - any idea on when we'll be able to set a date? | 17:46 |
| pabelanger | odyssey4me: I haven't looked into reno jobs as of yet but if you get a patch up, I can help review | 17:46 |
| anteaya | odyssey4me: so timing renames is more onourous that it used to be | 17:46 |
| AJaeger | fungi, we delete some of the translations every day. Is there something else I should delete for translations? | 17:46 |
| anteaya | odyssey4me: well it won't happen prior to summit | 17:47 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/irc-meetings: Use a boostrap grid to list all the meetings https://review.openstack.org/241522 | 17:47 |
| odyssey4me | anteaya yeah, I'm aware of that - but we wanted to get it into the queue to happen as soon as possible, whenever a time can be scheduled for it | 17:47 |
| anteaya | odyssey4me: and we like to group them if we can since it is 4 hours per rename, regardless of how many we do | 17:47 |
| *** Rockyg has quit IRC | 17:47 | |
| anteaya | odyssey4me: okay great, just thought I'd let you know | 17:48 |
| odyssey4me | 4 hours *per rename* - ouch! | 17:48 |
| odyssey4me | I thought it was 4 hours overall. :/ | 17:48 |
| fungi | AJaeger: the openstack-manuals translations seem to be the largest at anywhere from 0.5-1gb in size (and there are 7 that i see, so possibly cruft from multiple job renames?) | 17:48 |
| anteaya | odyssey4me: sorry, no that is what I mean | 17:48 |
| anteaya | 4 hours for every rename maintenance | 17:48 |
| AJaeger | fungi, do you have some names for me? | 17:48 |
| jlvillal | ttx: I removed the WIP message from: https://review.openstack.org/#/c/241522/ No other changes. | 17:48 |
| anteaya | odyssey4me: when do you expect a tag will be used for the repo? milestone 1? | 17:48 |
| *** infra-red has quit IRC | 17:48 | |
| openstackgerrit | Henrique Truta proposed openstack-infra/project-config: Run identity v3 only job in nova https://review.openstack.org/304243 | 17:49 |
| *** shardy has quit IRC | 17:49 | |
| fungi | AJaeger: sizes in mb for the top 19 job workspaces on that server http://paste.openstack.org/show/493680/ | 17:49 |
| *** shashank_hegde has joined #openstack-infra | 17:50 | |
| AJaeger | fungi, anything "*-zanata" can be deleted completely | 17:50 |
| AJaeger | fungi, and we did rename the jobs - and they are all recreated, aren't they? So get rid of all translation jobs... | 17:51 |
| *** hdd has joined #openstack-infra | 17:51 | |
| *** jistr has quit IRC | 17:51 | |
| fungi | AJaeger: removing those seems to have freed up about 3.3gb | 17:51 |
| clarkb | I think we should start with just remove all workspaces | 17:51 |
| clarkb | let it steady state again and if we need to then add a volume | 17:51 |
| fungi | yeah, can do | 17:51 |
| *** tonytan4ever has quit IRC | 17:51 | |
| clarkb | its possible we are well under the threshold we just have old cruft | 17:51 |
| clarkb | (jenkins doesn't delete workspaces when jobs are deleted?) | 17:51 |
| fungi | i don't think it does, no | 17:52 |
| *** tonytan4ever has joined #openstack-infra | 17:52 | |
| odyssey4me | anteaya for the moment we'll be referring to the github URL for the repo in all docs and code, as the rename will cause a redirect when accessing github so it'll work before and after | 17:52 |
| reed | who manages the Corporate CLA nowadays? this page should have a contact point: https://wiki.openstack.org/wiki/HowToUpdateCorporateCLA | 17:52 |
| odyssey4me | anteaya once the rename is done, we'll switch to using git.o.o in all branches | 17:52 |
| fungi | okay, i'm blowing away all cached workspaces there and then we can look at it again later in the week to see what's going on there | 17:52 |
| clarkb | reed: danny does iirc | 17:52 |
| anteaya | odyssey4me: your rename commit message mentions tags breaking | 17:52 |
| fungi | reed: danny careno | 17:52 |
| clarkb | reed: I will let him know to update the wiki | 17:53 |
| reed | clarkb, fungi, k, thanks | 17:53 |
| fungi | reed: thanks for spotting that | 17:53 |
| anteaya | odyssey4me: when do you expect applying a tag to that repo? | 17:53 |
| *** ddieterly[away] is now known as ddieterly | 17:53 | |
| reed | i'd update it but Launchpad is taking a break | 17:53 |
| odyssey4me | anteaya we'll be doing the first tag before the summit, so we had to make a plan and work with what we have | 17:53 |
| *** shardy has joined #openstack-infra | 17:53 | |
| *** achanda has quit IRC | 17:53 | |
| fungi | okay, cached workspaces have been cleaned up on proposal.slave.o.o and it's reenabled on jenkins.o.o now | 17:54 |
| anteaya | odyssey4me: oh well the commit message just mentions a tag, I didn't know you would be tagging prior to summit | 17:54 |
| anteaya | odyssey4me: when are you taggin? | 17:54 |
| anteaya | tagging? | 17:54 |
| *** pvaneck has joined #openstack-infra | 17:54 | |
| odyssey4me | anteaya hopefully at the end of this week - but failing that it'll be by the end of next week | 17:54 |
| anteaya | odyssey4me: can you say so in the commit message? | 17:55 |
| odyssey4me | anteaya sure, I'll fix that up | 17:55 |
| anteaya | odyssey4me: we can discuss it again at tomorrow's meeting, if someone feels like doing a rename prior to you tagging then it saves a lot of trouble | 17:55 |
| anteaya | odyssey4me: I don't think anyone knew your tagging plans so it was hard to keep them in mind when discussing maintenance | 17:56 |
| *** lakshmiS has quit IRC | 17:56 | |
| *** achanda has joined #openstack-infra | 17:56 | |
| *** lakshmiS has joined #openstack-infra | 17:56 | |
| *** tonytan_brb has joined #openstack-infra | 17:57 | |
| anteaya | myself I was assuming milestone 1 | 17:57 |
| fungi | dims: i've retriggered it again, console log is streaming now at https://jenkins.openstack.org/job/propose-requirements-constraints-master/290/console | 17:57 |
| dims | cool fungi : will watch that | 17:57 |
| openstackgerrit | Tim Burke proposed openstack-infra/git-review: Remove worthless print https://review.openstack.org/304246 | 17:58 |
| fungi | dims: or, you know, do something productive and check it again in a little while ;) | 17:58 |
| fungi | it's likely not a fast job | 17:58 |
| dims | fungi : LOL | 17:58 |
| dims | yep | 17:58 |
| *** amotoki has quit IRC | 17:59 | |
| *** tonytan4ever has quit IRC | 17:59 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Rename openstack-ansible-ironic to openstack-ansible-os_ironic https://review.openstack.org/299192 | 17:59 |
| odyssey4me | anteaya ^ is that suitable? | 18:00 |
| *** exploreshaifali has quit IRC | 18:00 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/304249 | 18:00 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again https://review.openstack.org/288915 | 18:00 |
| fungi | dims: oh, it is a fast job after all, and has already succeeded and produced https://review.openstack.org/304249 | 18:01 |
| anteaya | well you said end of this week in irc, so I'll add that as a comment on the patch | 18:01 |
| dims | wasn't bad at all! 4 mins | 18:01 |
| clarkb | fungi: how big is the workspace now? | 18:01 |
| *** sambetts is now known as sambetts|afk | 18:01 | |
| dims | thanks. review it generated looks good fungi | 18:01 |
| *** doug-fis_ has joined #openstack-infra | 18:01 | |
| clarkb | I wonder if that job needs more git clean action | 18:01 |
| fungi | clarkb: 6mb | 18:01 |
| anteaya | odyssey4me: we are just all really tired and would like to avoid more maintenance if we can, but we also don't want to break things if we can avoid it | 18:01 |
| *** dtantsur is now known as dtantsur|afk | 18:01 | |
| fungi | clarkb: though that wasn't one of the big workspaces before | 18:02 |
| clarkb | oh I thought it was for some reason | 18:02 |
| fungi | clarkb: propose-requirements-updates was the big one | 18:02 |
| *** doug-fi__ has joined #openstack-infra | 18:02 | |
| odyssey4me | anteaya yep, completely understood - we're happy to roll with it and will fit in however we can - let me know if there's anything I can do to help | 18:03 |
| *** jsavak has quit IRC | 18:03 | |
| anteaya | odyssey4me: well know I know your tagging intentions that helps me understand, thank you | 18:03 |
| fungi | clarkb: my console history says it was 23mb before cleanup | 18:03 |
| *** jsavak has joined #openstack-infra | 18:03 | |
| anteaya | odyssey4me: if you are available tomorrow for the infra meeting at 19:00 utc that can't hurt as others may have questions | 18:03 |
| *** dimtruck is now known as zz_dimtruck | 18:03 | |
| *** achanda_ has joined #openstack-infra | 18:04 | |
| *** achanda has quit IRC | 18:04 | |
| odyssey4me | anteaya sure, let me add a reminder | 18:04 |
| anteaya | odyssey4me: great, thank you | 18:04 |
| fungi | clarkb: so crufty, but not one of the huge "problem" workspaces | 18:04 |
| clarkb | ya | 18:04 |
| *** doug-f___ has joined #openstack-infra | 18:04 | |
| *** doug-fish has quit IRC | 18:04 | |
| odyssey4me | anteaya which room is the meeting in? | 18:05 |
| anteaya | odyssey4me: in #openstack-meeting | 18:05 |
| *** maishsk has joined #openstack-infra | 18:05 | |
| anteaya | odyssey4me: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Weekly_Project_Infrastructure_team_meeting | 18:05 |
| odyssey4me | anteaya cool, reminder set | 18:05 |
| anteaya | odyssey4me: great, thank you | 18:05 |
| *** doug-f___ is now known as doug-fish | 18:06 | |
| *** doug-fis_ has quit IRC | 18:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: remove release note building venv https://review.openstack.org/304250 | 18:06 |
| AJaeger | fungi, that venv was 230 MB just for nova on my local system... | 18:07 |
| *** doug-fi__ has quit IRC | 18:07 | |
| fungi | AJaeger: wow, pretty big | 18:07 |
| *** Sukhdev has quit IRC | 18:07 | |
| *** denisra has joined #openstack-infra | 18:07 | |
| AJaeger | but venv is used for tarballs as well, so is it good to remove it? | 18:08 |
| AJaeger | wait, tarballs run on a different node, don't they? | 18:08 |
| openstackgerrit | Henrique Truta proposed openstack-infra/project-config: Run identity v3 only job in cinder https://review.openstack.org/304253 | 18:09 |
| *** jcooley_ has joined #openstack-infra | 18:09 | |
| *** daemontool__ has quit IRC | 18:10 | |
| *** avarner has quit IRC | 18:11 | |
| fungi | AJaeger: tarball generation happens on single-use workers because it runs arbitrary code from the projects | 18:12 |
| fungi | courtesy of the python ecosystem relying on setup.py as its package building entrypoint | 18:13 |
| *** jcooley_ has quit IRC | 18:13 | |
| *** jcooley has joined #openstack-infra | 18:13 | |
| fungi | this is why we split out package uploading to pypi as a separate job | 18:13 |
| *** trown|lunch is now known as trown | 18:14 | |
| fungi | so that we don't compromise our upload credentials | 18:14 |
| AJaeger | fungi, ok. In Austin, I'd like to sit with somebody together to review the translation jobs setup and figure out whether there's any risk... | 18:14 |
| *** tiswanso has quit IRC | 18:14 | |
| *** tricksters is now known as elopez | 18:14 | |
| anteaya | AJaeger: do you want to add that to the sessions etherpad? | 18:14 |
| *** tiswanso has joined #openstack-infra | 18:15 | |
| *** elopez is now known as eric_lopez | 18:15 | |
| AJaeger | anteaya: good idea. Do you have the link handy? | 18:15 |
| fungi | AJaeger: well, we can't _rely_ on cleanup steps in jobs to fire (for example if the job breaks and terminates early), but a git clean at the end probably can't hurt | 18:15 |
| anteaya | AJaeger: https://etherpad.openstack.org/p/infra-newton-summit-planning | 18:15 |
| AJaeger | fungi, will think about more about cleanup... | 18:15 |
| AJaeger | anteaya: thanks | 18:15 |
| *** ddieterly is now known as ddieterly[away] | 18:16 | |
| AJaeger | added | 18:18 |
| anteaya | AJaeger: thank you | 18:18 |
| *** sdake has joined #openstack-infra | 18:18 | |
| openstackgerrit | Henrique Truta proposed openstack-infra/project-config: Run identity v3 only job in swift https://review.openstack.org/304263 | 18:18 |
| crinkle | if an infra-core has the time it would be neat to have feedback on https://review.openstack.org/#/q/topic:infra-cloud+status:open+project:openstack-infra/puppet-infracloud , since it'll be easier to start work on liberty->mitaka after we have functional tests | 18:19 |
| *** kzaitsev_mb has quit IRC | 18:20 | |
| *** fawadkhaliq has quit IRC | 18:20 | |
| *** shardy is now known as shardy_afk | 18:20 | |
| *** fawadkhaliq has joined #openstack-infra | 18:21 | |
| anteaya | crinkle: I've added that link as a reference for clarkb's proposed sprint session on line 116: https://etherpad.openstack.org/p/infra-newton-summit-planning | 18:21 |
| *** achanda_ has quit IRC | 18:21 | |
| *** achanda has joined #openstack-infra | 18:21 | |
| AJaeger | fungi, we currently have jenkins/scripts/slave-cleanup.sh that deletes once a day the .venv directory | 18:22 |
| AJaeger | should I extend this to run "git clean"? | 18:22 |
| AJaeger | Or remove after each run? | 18:23 |
| *** cody-somerville has quit IRC | 18:23 | |
| openstackgerrit | Merged openstack/requirements: Bump os-net-config to 0.2.4 https://review.openstack.org/303987 | 18:23 |
| *** dangers_away has joined #openstack-infra | 18:24 | |
| *** mrmartin has quit IRC | 18:24 | |
| *** avarner has joined #openstack-infra | 18:25 | |
| crinkle | anteaya: thanks, though it's only four patches and most of them are reasonably small so i was hoping they'd get attention a little sooner than the summit | 18:25 |
| fungi | AJaeger: oh, we have jobs creating a ~jenkins/.venv too? git clean won't help with that (and it's also not contributing to workspace sizes) | 18:26 |
| anteaya | crinkle: oh yes, I agree, but I think that same query will apply for new infra-cloud patches, will it not? | 18:26 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/303416 | 18:26 |
| *** mrmartin has joined #openstack-infra | 18:26 | |
| fungi | AJaeger: also that ~jenkins/.venv is basically empty (just contains the CREATED marker file) | 18:27 |
| *** achanda has quit IRC | 18:27 | |
| crinkle | anteaya: actually a better query would be https://review.openstack.org/#/q/status:open+topic:infra-cloud (topic rather than project), I'll go ahead and change it | 18:27 |
| AJaeger | fungi, I wonder whether it's the wrong directory that gets cleaned... | 18:27 |
| anteaya | crinkle: awesome, thank you | 18:27 |
| *** tiswanso has quit IRC | 18:28 | |
| fungi | AJaeger: what jobs are creating this that you're trying to clean up after? | 18:28 |
| * AJaeger currently tries to remember and digs into git logs... | 18:28 | |
| *** tiswanso has joined #openstack-infra | 18:28 | |
| *** ociuhandu has quit IRC | 18:29 | |
| *** infra-red has joined #openstack-infra | 18:29 | |
| AJaeger | fungi, do you have a ~/.m2 file on the slave? How large is that? | 18:29 |
| AJaeger | directory I mean... | 18:29 |
| *** achanda has joined #openstack-infra | 18:30 | |
| fungi | AJaeger: looks like the cleanup is only called from these jobs running the proposal-slave-cleanup macro http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/translation-jobs.yaml | 18:30 |
| sdake | fungi https://review.openstack.org/#/c/304080/2 | 18:31 |
| AJaeger | fungi, yes, correct. | 18:31 |
| fungi | AJaeger: there is a ~jenkins/.m2 but all it contains is a 1kb settings.xml file | 18:31 |
| *** esikachev has joined #openstack-infra | 18:31 | |
| sdake | there is speculation that bluebox runs multiple kolla jobs at once and the multicast is enabled on their networks causing keepalived interference | 18:31 |
| AJaeger | fungi: good ;) So, do not collect lots of java files there | 18:31 |
| fungi | sdake: aha! races | 18:31 |
| sdake | fungi not relaly races, kolla defaults to 224.0.0.18 multicat address | 18:32 |
| fungi | sdake: it's possible bluebox is _not_ filtering multipcast your jobs are leaking onto the provider network but other providers are? | 18:32 |
| sdake | our keepalive config is password protected | 18:32 |
| sdake | so it may be a bug in keepalived | 18:32 |
| sdake | fungi roger thats the speculation | 18:32 |
| fungi | sdake: so yeah, lack of isolation from the outer environment in this case i guess | 18:32 |
| sdake | fungi we dont know for certain | 18:32 |
| *** maishsk has quit IRC | 18:33 | |
| sdake | the other gate problem we had has to do with erlang crashing when compield with ipv6 | 18:33 |
| sdake | which was just done for centos about 2 weeks ago | 18:33 |
| fungi | sdake: but i did see invalid password errors in your keepalived logs on multiple providers, not just bluebox | 18:33 |
| * AJaeger thinks the slave script is just broken ;( | 18:33 | |
| sdake | which coincides with major gate flakiness | 18:33 |
| *** fawadkhaliq has quit IRC | 18:33 | |
| sdake | fungi oh really? | 18:33 |
| fungi | sdake: yeah, your jobs archive the keepalived log | 18:33 |
| clarkb | sdake: fungi our iptables rules should prevent that multicast | 18:34 |
| *** fawadkhaliq has joined #openstack-infra | 18:34 | |
| sdake | clarkb unless we are using a 2x8 setup hopefully ;) | 18:34 |
| fungi | clarkb: well, unless that job is modifying iptables on the worker | 18:34 |
| clarkb | fungi: ya good point | 18:34 |
| sdake | we do not modify ptables I am prety sure | 18:34 |
| anteaya | AJaeger: yeah, that might be the case | 18:34 |
| *** nadya has quit IRC | 18:34 | |
| sdake | fungi anyway just speculation at this point | 18:35 |
| sdake | I thought I'd share since the password thing indicates the lack of isolation | 18:35 |
| sdake | which is maybe not what infra is after as clarkb indicates | 18:35 |
| sdake | the password error is real though | 18:35 |
| clarkb | sdake: basically our tests should be isolated from each oter due to our firewall rules | 18:35 |
| sdake | so something is either getting through or keepalived has gone buggy | 18:35 |
| clarkb | sdake: regardless of provider | 18:35 |
| AJaeger | fungi, anteaya : The idea was to delete the generated environments once a day (as optimization). I'll change this now to a git clean with some debugging output... | 18:36 |
| fungi | sdake: well, assuming your jobs are testing correctly, you should just be able to recheck 304080 until it hits a bluebox worker to see whether that helps, but i agree better identification of the underlying cause is warranted | 18:36 |
| sdake | fungi yup thats what the plan is | 18:36 |
| anteaya | AJaeger: ah debugging, that sounds like the ticket | 18:36 |
| anteaya | AJaeger: let's have it tell us what it is doing | 18:36 |
| *** achanda has quit IRC | 18:36 | |
| sdake | fungi if you saw it on othre cloud providers, I am curious why only bluebox fails | 18:36 |
| fungi | sdake: right, that may point to this being unrelated | 18:37 |
| sdake | maybe the password lekaage is different from the keepalive faillign to go to master issue | 18:37 |
| sdake | rather multicast leakage | 18:37 |
| fungi | but anyway, should be able to tell by looking at the logs from job runs on this change whether the errors continue to appear in the keepalived logs | 18:37 |
| sdake | its pretty consistent on bluebox | 18:38 |
| sdake | but typically need lots of kolla patches running in the gate at the same time | 18:38 |
| sdake | i guess i'll find a slection of patches to recheck :) | 18:38 |
| fungi | sdake: the reason i remember is that i at first thought it might be related to the issue we were discussing, so i compared to a keepalived from a passing job run and it had the same invalid password errors | 18:38 |
| sdake | this gate bug is not nearly as onerous as the ipv6 problem | 18:38 |
| sdake | fungi that is relaly intereting - I didn't think of that at the time | 18:39 |
| sdake | fungi i'll report back when we find out the underlyign issue | 18:39 |
| sdake | fungi (if it has effect for infra) | 18:39 |
| *** piet has quit IRC | 18:40 | |
| *** jaypipes has quit IRC | 18:40 | |
| *** LaT0rtue has joined #openstack-infra | 18:41 | |
| *** julim has quit IRC | 18:42 | |
| fungi | cool, thanks! | 18:42 |
| *** infra-re_ has joined #openstack-infra | 18:43 | |
| *** ddieterly[away] is now known as ddieterly | 18:44 | |
| *** daemontool has joined #openstack-infra | 18:46 | |
| *** med_ has quit IRC | 18:46 | |
| anteaya | it is an hour and 14 minutes before we begin our maintenance window, does everyone who is playing along know what they will be doing? https://etherpad.openstack.org/p/gerrit_server_replacement | 18:47 |
| pabelanger | sdake: please take a moment to review https://review.openstack.org/#/c/301320/ as you are the kolla PTL | 18:47 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rework slave-cleanup.sh https://review.openstack.org/304285 | 18:47 |
| *** infra-red has quit IRC | 18:47 | |
| AJaeger | anteaya: I'll disappear before ;) | 18:47 |
| anteaya | AJaeger: figured you would have a plan | 18:47 |
| AJaeger | fungi, anteaya: Here's my rework ^ | 18:47 |
| *** mylu has quit IRC | 18:47 | |
| sdake | pabelanger will do moment | 18:48 |
| nibalizer | anteaya: i'm planning on being in channel, I don't have any tasks right now | 18:48 |
| nibalizer | which is fine | 18:48 |
| *** infra-re_ has quit IRC | 18:49 | |
| anteaya | nibalizer: great, thank you | 18:49 |
| anteaya | nibalizer: thanks for reviewing the patch(es) | 18:49 |
| *** tonytan_brb has quit IRC | 18:49 | |
| *** infra-red has joined #openstack-infra | 18:49 | |
| fungi | i'm planning to add the unpuppeting for old gerrit in a few minutes | 18:49 |
| *** cdent has quit IRC | 18:49 | |
| sdake | pabelanger ugh image changes t-5 days until release | 18:49 |
| anteaya | fungi: awesome | 18:50 |
| sdake | pabelanger our release is the 15th, so this is our final go to week to fix bugs | 18:50 |
| sdake | we need a functioanl gate, and from what I saw of the experimental jobs, bindep was failing | 18:50 |
| *** mrmartin has quit IRC | 18:50 | |
| sdake | i am happy to temporarily tear apart the gate after the 15th | 18:50 |
| *** mylu has joined #openstack-infra | 18:50 | |
| *** tonytan4ever has joined #openstack-infra | 18:51 | |
| sdake | I hope that can fit in with your schedule? | 18:51 |
| pabelanger | sdake: I am confident they are passing. they are only failing because of hardcoded tox settings. see: https://review.openstack.org/#/c/303391/ | 18:51 |
| fungi | but it does seem prudent to wait another 4 days until their release work is done | 18:52 |
| fungi | i'll be around all next week too and am happy to help get thise final image changes put through and debugged as needed | 18:53 |
| sdake | pabelanger nice | 18:53 |
| sdake | pabelanger ok that wfm then | 18:53 |
| sdake | still a little frightening :) | 18:53 |
| pabelanger | fair enough we can hold off until kolla releases | 18:53 |
| pabelanger | they are the last project on devstack-centos7 | 18:53 |
| clarkb | we aren't pressured for disk space any longer | 18:54 |
| sdake | pabelanger yes we are asking the tc for a general one week extension for all deployment projects in release_with_milestones | 18:54 |
| sdake | although we re not in this release model yet we intend to be shortly | 18:54 |
| *** med_ has joined #openstack-infra | 18:54 | |
| *** med_ has quit IRC | 18:54 | |
| *** med_ has joined #openstack-infra | 18:54 | |
| pabelanger | fungi: okay, -1 workflow until you +A commit | 18:55 |
| *** rbrndt has joined #openstack-infra | 18:55 | |
| pabelanger | clarkb: Ya, that is true | 18:55 |
| *** rbrndt__ has quit IRC | 18:55 | |
| anteaya | AJaeger: reading https://www.kernel.org/pub/software/scm/git/docs/git-clean.html the -d flag, do you want to use the -f flag twice in your command? | 18:55 |
| sdake | see: https://review.openstack.org/#/c/298278/ | 18:56 |
| sdake | pabelanger ^ | 18:56 |
| sdake | pabelanger i will add data to review about our deadlines | 18:56 |
| *** tiswanso has quit IRC | 18:56 | |
| clarkb | I should be around next week as well | 18:57 |
| *** tonytan4ever has quit IRC | 18:57 | |
| clarkb | its the best time to get work done | 18:57 |
| *** doug-fish has quit IRC | 18:57 | |
| *** infra-red has quit IRC | 18:57 | |
| fungi | since preference seemed to be around merging the jgit heap size bump prior to the maintenance, i've squeezed that in to happen soon after i disable puppet for the old server | 18:57 |
| *** tonytan4ever has joined #openstack-infra | 18:58 | |
| *** doug-fish has joined #openstack-infra | 18:58 | |
| *** tiswanso has joined #openstack-infra | 18:58 | |
| *** infra-red has joined #openstack-infra | 18:58 | |
| AJaeger | anteaya: using it twice is only needed "if an untracked directory is managed by a different Git repository" - so, we should be fine with doing it once. And the du output will show is if we really need it twice in case I missed anything... | 18:58 |
| anteaya | AJaeger: awesome, thank you | 18:58 |
| *** sputnik13 has quit IRC | 18:59 | |
| AJaeger | anteaya: thanks for double checking | 19:00 |
| anteaya | I do try | 19:00 |
| *** yarkot_ has joined #openstack-infra | 19:00 | |
| anteaya | and you're welcome | 19:00 |
| *** julim has joined #openstack-infra | 19:01 | |
| yolanda | hi, i am around now. fungi, did you have progress with volumes? | 19:01 |
| *** doug-fis_ has joined #openstack-infra | 19:01 | |
| fungi | yolanda: yep, instructions updated | 19:01 |
| fungi | yolanda: i've also disabled puppet for the old uuid | 19:01 |
| fungi | and am tailing logs to make sure it stuck | 19:02 |
| *** edmondsw has joined #openstack-infra | 19:02 | |
| *** sputnik13 has joined #openstack-infra | 19:02 | |
| yolanda | ok so everything looks ready for the replacement in one hour? | 19:02 |
| fungi | yep | 19:02 |
| *** doug-fish has quit IRC | 19:03 | |
| anteaya | fungi: so you have done step 0.2? | 19:03 |
| fungi | i'll approve the heap size bump change in a few minutes if i don't see any evidence that the puppet disablement didn't take | 19:03 |
| anteaya | yes you crossed it out | 19:03 |
| fungi | anteaya: yep | 19:03 |
| *** amitgandhinz has quit IRC | 19:03 | |
| anteaya | thank you | 19:03 |
| fungi | anteaya: semi-done. i'm watching logs to make sure it worked as intended | 19:03 |
| *** amitgandhinz has joined #openstack-infra | 19:03 | |
| anteaya | fungi: ah yes, saw you say that above, I just missed it was step 0.2 under discussion, sorry | 19:04 |
| fungi | other than approving https://review.openstack.org/303846 there's not much else to do until moments before 20:00 | 19:04 |
| *** tonytan4ever has quit IRC | 19:04 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Reconcile compute and controller classes https://review.openstack.org/293620 | 19:04 |
| anteaya | wonderful | 19:04 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Parameterize libvirt virt type https://review.openstack.org/293621 | 19:04 |
| AJaeger | fungi, do you want to review my slave cleanup change while waiting? https://review.openstack.org/#/c/304285/ I'm still a bit around to discuss it if needed - but it's not urgent... | 19:05 |
| *** doug-fis_ has quit IRC | 19:06 | |
| AJaeger | we could add this one also to all the propose jobs once we're sure it does the right thing | 19:06 |
| openstackgerrit | Hongbin Lu proposed openstack/requirements: Downgrade docker-py from 1.8.0 to 1.7.2 https://review.openstack.org/304296 | 19:07 |
| *** tonytan4ever has joined #openstack-infra | 19:07 | |
| fungi | AJaeger: does `du -h --max-depth=1 .` have any benefit over the simpler `du -hs .`? | 19:07 |
| fungi | AJaeger: oh, i see you're getting the next level trees under it | 19:08 |
| fungi | -s is more like --max-depth=0 | 19:08 |
| AJaeger | fungi, correct | 19:11 |
| * AJaeger wanted more details | 19:11 | |
| *** Sukhdev has joined #openstack-infra | 19:12 | |
| fungi | seems fine. any reason to keep the daily limit/marker hassle? | 19:12 |
| fungi | rather than just run this on each job invocation? | 19:12 |
| AJaeger | fungi, I can remove it. Idea was initially to reduce tox setup time... | 19:13 |
| *** krtaylor has quit IRC | 19:13 | |
| fungi | AJaeger: now that pip caches wheels, i doubt tox setup time is an issue any longer, but we can keep it around for now and you can compare i guess | 19:13 |
| fungi | first run of the day will reflect the "longer" tox setup time | 19:14 |
| AJaeger | fungi, yep... | 19:14 |
| fungi | 2016-04-11 19:11:11,218 p=4541 u=root | ERROR! Unexpected Exception: [Errno 12] Cannot allocate memory | 19:15 |
| fungi | just saw that show up in the ansible run-all log | 19:15 |
| fungi | not sure which host that was for | 19:15 |
| fungi | oh, local! | 19:16 |
| fungi | [Mon Apr 11 19:11:09 2016] Killed process 18885 (ansible-playboo) total-vm:1188152kB, anon-rss:153304kB, file-rss:4kB | 19:16 |
| fungi | nibalizer: mordred: ^ if you want to dig into that juiciness... | 19:16 |
| bkero | oh boy oom | 19:17 |
| fungi | system has 4gb virtual memory split between ram/swap | 19:18 |
| fungi | i don't see a ton of resident memory on current processes according to top/free | 19:18 |
| fungi | wonder if we're trying to parallelize too much? | 19:19 |
| *** jamesmcarthur has joined #openstack-infra | 19:19 | |
| clarkb | fungi: or if shade is pulling too much data from clouds? | 19:20 |
| AJaeger | fungi, clarkb, do you consider updating of 40 jobs fine or might it involve manual JJB runs? I didn't approve https://review.openstack.org/#/c/303508/ yet... | 19:20 |
| *** salv-orlando has quit IRC | 19:21 | |
| clarkb | 40 jobs should be fine | 19:21 |
| fungi | AJaeger: should be fine | 19:21 |
| AJaeger | so, approve now before downtime - or better tomorrow? | 19:21 |
| fungi | now's probably fine | 19:22 |
| fungi | gate seems pretty quiet | 19:22 |
| AJaeger | anteaya: I guess you like https://review.openstack.org/#/c/303622/ - adds voting job to gate that was only in check... | 19:23 |
| clarkb | I am about to eat a quick lunch and will be back fro the fun | 19:23 |
| AJaeger | fungi, clarkb : Thanks, approved | 19:23 |
| openstackgerrit | Merged openstack-infra/project-config: Rework slave-cleanup.sh https://review.openstack.org/304285 | 19:24 |
| fungi | i've approved 303846... hoping that puppet is really disabled for the old review.openstack.org and it's not simply getting skipped because of the oom on the puppetmaster | 19:24 |
| *** baoli_ has quit IRC | 19:25 | |
| *** xiangxinyong has quit IRC | 19:25 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Dump end of logs to console.log on failure https://review.openstack.org/302999 | 19:25 |
| *** x00350071 has joined #openstack-infra | 19:26 | |
| anteaya | AJaeger: thank you | 19:26 |
| nibalizer | fungi: ok | 19:26 |
| anteaya | fungi: ack | 19:26 |
| *** piet has joined #openstack-infra | 19:27 | |
| *** ddieterly is now known as ddieterly[away] | 19:27 | |
| *** stevemar has quit IRC | 19:27 | |
| AJaeger | thanks, anteaya | 19:28 |
| *** krtaylor has joined #openstack-infra | 19:28 | |
| nibalizer | looks like we oom on ansible pretty frequently | 19:28 |
| anteaya | welcome, thanks for tracking that change | 19:28 |
| *** stevemar has joined #openstack-infra | 19:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Move gate-tempest-dsvm-neutron-dvr to gate queue https://review.openstack.org/303622 | 19:30 |
| *** mrmartin has joined #openstack-infra | 19:30 | |
| *** nadya has joined #openstack-infra | 19:30 | |
| openstackgerrit | Merged openstack-infra/system-config: Raise gerrit heaplimit to 30 GB https://review.openstack.org/303846 | 19:31 |
| *** Jeffrey4l__ has joined #openstack-infra | 19:32 | |
| mordred | fungi: oh yay | 19:32 |
| *** e0ne has quit IRC | 19:32 | |
| *** Jeffrey4l_ has quit IRC | 19:34 | |
| fungi | and now that's merged, we're clear for another ~20 minutes before jeblair works on stopping zuul | 19:35 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add more db api tests part 2 https://review.openstack.org/303265 | 19:36 |
| *** ihrachys has joined #openstack-infra | 19:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Do not run npm shrinkwrap if npm-shrinkwrap.json already exists https://review.openstack.org/303508 | 19:37 |
| *** jamesmcarthur has quit IRC | 19:37 | |
| *** cdent has joined #openstack-infra | 19:37 | |
| *** mixos has joined #openstack-infra | 19:38 | |
| *** jamesmcarthur has joined #openstack-infra | 19:38 | |
| *** fawadkhaliq has quit IRC | 19:39 | |
| *** fawadkhaliq has joined #openstack-infra | 19:40 | |
| *** jamesmcarthur has quit IRC | 19:40 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add documentation about openstack-health deployment https://review.openstack.org/237268 | 19:40 |
| *** jamesmcarthur has joined #openstack-infra | 19:41 | |
| mtreinish | anteaya, yolanda: sry for the multiple month turnaround on ^^^ | 19:42 |
| *** jamesmcarthur has quit IRC | 19:42 | |
| mtreinish | I forgot it was there | 19:42 |
| anteaya | mtreinish: ha ha ha | 19:43 |
| anteaya | mtreinish: I had forgotten what I had said | 19:43 |
| anteaya | like you say in comments, by the time it is merged your original may be correct | 19:43 |
| * AJaeger waves good bye and wishes a successful gerrit move | 19:43 | |
| mtreinish | well on the rename comment it definitely was | 19:44 |
| mtreinish | I don't think anything else has really changed | 19:44 |
| fungi | thanks AJaeger! have a good evening | 19:44 |
| anteaya | AJaeger: thank you, enjoy | 19:45 |
| anteaya | mtreinish: :) | 19:45 |
| jeblair | back from lunch | 19:45 |
| clarkb | I am back from lunch as well | 19:45 |
| fungi | jeblair: we're on track for stopping zuul in 10-15 minutes from now. nothing else pending at this point | 19:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add nodepool request framework https://review.openstack.org/304302 | 19:46 |
| anteaya | jeblair: clarkb welcome back | 19:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-subunit2sql: Ensure all jobs send a completion packet https://review.openstack.org/304303 | 19:46 |
| jeblair | mtreinish: i noticed this while reviewing your patch on the plane | 19:46 |
| *** hdd has quit IRC | 19:47 | |
| *** jamesmcarthur has joined #openstack-infra | 19:48 | |
| fungi | #status notice Gerrit will be offline from 20:00 to 21:00 UTC (starting 10 minutes from now) for a server replacement http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 19:49 |
| openstackstatus | fungi: sending notice | 19:49 |
| mtreinish | jeblair: hmm, sending a jobException on gearman doesn't clear it from server? | 19:49 |
| * mtreinish isn't super familiar with the protocol | 19:49 | |
| -openstackstatus- NOTICE: Gerrit will be offline from 20:00 to 21:00 UTC (starting 10 minutes from now) for a server replacement http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 19:50 | |
| *** ddieterly[away] is now known as ddieterly | 19:50 | |
| jeblair | mtreinish: it does -- the patch fixes the case where aborted jobs don't send anything | 19:50 |
| *** fawadkhaliq has quit IRC | 19:51 | |
| jeblair | mtreinish: also, i just -1'd it :) | 19:51 |
| *** fawadkhaliq has joined #openstack-infra | 19:51 | |
| *** maishsk has joined #openstack-infra | 19:51 | |
| openstackstatus | fungi: finished sending notice | 19:51 |
| *** ajmiller has joined #openstack-infra | 19:52 | |
| *** ddieterly is now known as ddieterly[away] | 19:52 | |
| mtreinish | jeblair: heh, ok | 19:52 |
| jeblair | mtreinish: so i should say, *ideally* the patch fixes that case :) | 19:52 |
| *** cdent has quit IRC | 19:52 | |
| *** mylu has quit IRC | 19:53 | |
| *** salv-orlando has joined #openstack-infra | 19:54 | |
| jeblair | fungi: i'll prepare to stop zuul and stage 281166 | 19:54 |
| *** fawadkhaliq has quit IRC | 19:54 | |
| fungi | jeblair: thanks! | 19:54 |
| *** zz_dimtruck is now known as dimtruck | 19:54 | |
| *** ZZelle_ has joined #openstack-infra | 19:54 | |
| jeblair | (if i can log in [ram crunch]) | 19:54 |
| *** fawadkhaliq has joined #openstack-infra | 19:55 | |
| fungi | as soon as you confirm zuul is dead (one way or another!) we'll press forward with step 1 | 19:55 |
| *** ddieterly[away] is now known as ddieterly | 19:55 | |
| fungi | yolanda: did you want to drive the gerrit server bits starting in 5 minutes? | 19:55 |
| mtreinish | jeblair: ah, ok with that comment I understand better now. I was just wondering about the L124 changing the sendWorkException() to a raise | 19:55 |
| fungi | i'll get the maintenance alert queued up | 19:55 |
| * clarkb is here to offer a helping hand where necessary | 19:55 | |
| * mordred is here to occasionally offer humorous interjections | 19:56 | |
| mtreinish | jeblair: I was thinking that would have a similar effect by not clearing it from the server when we encounter a missing file | 19:56 |
| jeblair | mtreinish: ah, that's just to make the flow simpler so we don't have to separately handle that case | 19:56 |
| *** maishsk has quit IRC | 19:56 | |
| *** ddutta has joined #openstack-infra | 19:56 | |
| mtreinish | because it'll break out of the function before anything is sent | 19:56 |
| yolanda | fungi, yes, i'm fine with driving that, with guidance of other infra-root just in case | 19:56 |
| fungi | yolanda: at least several of us are here to support you | 19:56 |
| *** maishsk has joined #openstack-infra | 19:56 | |
| * mordred hands infra-root a not entirely eaten cake | 19:56 | |
| jeblair | mtreinish: with the raise, it'll be caught by the outer execption handler that will send the work exception | 19:56 |
| yolanda | less pressure then :) | 19:56 |
| jeblair | anyone happen to be logged into zuul.o.o? | 19:57 |
| yolanda | i am already logged to old and new gerrit | 19:57 |
| * nibalizer on deck | 19:57 | |
| clarkb | jeblair: not me | 19:57 |
| *** Nikolay_St has joined #openstack-infra | 19:57 | |
| nibalizer | jeblair: not me | 19:57 |
| fungi | jeblair: i was maybe an hour ago | 19:57 |
| mtreinish | jeblair: ah, so it will. I should have looked a bit lower down in the file :) | 19:57 |
| fungi | but i logged out | 19:57 |
| * jeblair is looking for someone with an active shell there :| | 19:57 | |
| *** jamesmcarthur has quit IRC | 19:57 | |
| fungi | yeah, i figured | 19:57 |
| fungi | looks like we picked the wrong moment to try to log in http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=389&rra_id=all | 19:58 |
| jeblair | aup | 19:58 |
| anteaya | oh my | 19:59 |
| *** kzaitsev_mb has joined #openstack-infra | 19:59 | |
| *** achanda has joined #openstack-infra | 19:59 | |
| fungi | high system and iowait, load average of 30+ | 19:59 |
| *** tonytan4ever has quit IRC | 19:59 | |
| jeblair | it's also not serving status.json, so i can't remotely save it either | 19:59 |
| fungi | oddly paging hasn't jumped up | 19:59 |
| jeblair | i say we declare it a lost cause, and i'll reboot it externally after gerrit is stopped | 19:59 |
| *** nadya has quit IRC | 20:00 | |
| fungi | no increase in swap utilization or spike in paging activity | 20:00 |
| anteaya | sounds accurate | 20:00 |
| yolanda | ok, are we good to start? | 20:00 |
| fungi | so this is slammed filesystem from something else presumably | 20:00 |
| *** jsavak has quit IRC | 20:00 | |
| jeblair | fungi: i suspect it's still ram related | 20:00 |
| fungi | yolanda: well, we were trying to get zuul pipelines saved and zuul stopped before taking gerrit down | 20:00 |
| jeblair | fungi: (could be related to oom killer or something) | 20:00 |
| *** jsavak has joined #openstack-infra | 20:00 | |
| * pleia2 stands by | 20:01 | |
| yolanda | ok i'll wait until that | 20:01 |
| *** trown is now known as trown|outtypewww | 20:01 | |
| openstackgerrit | Henrique Truta proposed openstack-infra/project-config: Adds ironic v3 only job https://review.openstack.org/304306 | 20:01 |
| jeblair | fungi: i don't recommend waiting for zuul; i recommend proceeding without it. but i will continue to try if you like. | 20:01 |
| fungi | i/o activity on xvda spiked, which is the rootfs | 20:01 |
| fungi | jeblair: well, you can keep trying to get into it while we take gerrit down i guess | 20:02 |
| fungi | i doubt it's doing much else at the moment | 20:02 |
| fungi | the spike on xvda has started to trail off (i think) | 20:02 |
| *** Sukhdev has quit IRC | 20:02 | |
| fungi | so maybe cpu/load/iowait will fall her ein a few | 20:02 |
| fungi | i'll go ahead with the alert | 20:03 |
| fungi | #status alert Gerrit is offline until 21:00 UTC for a server replacement http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 20:03 |
| openstackstatus | fungi: sending alert | 20:03 |
| openstackgerrit | Henrique Truta proposed openstack-infra/project-config: Run identity v3 only job in ironic https://review.openstack.org/304307 | 20:04 |
| fungi | yolanda: go ahead and stop gerrit following step 1 in the plan | 20:04 |
| *** dimtruck is now known as zz_dimtruck | 20:04 | |
| yolanda | ok | 20:04 |
| *** openstackgerrit has quit IRC | 20:05 | |
| anteaya | we are following this etherpad for those watching at home: https://etherpad.openstack.org/p/gerrit_server_replacement | 20:05 |
| yolanda | ok first step done | 20:05 |
| *** maishsk has quit IRC | 20:05 | |
| *** shamail has joined #openstack-infra | 20:05 | |
| -openstackstatus- NOTICE: Gerrit is offline until 21:00 UTC for a server replacement http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 20:05 | |
| *** ChanServ changes topic to "Gerrit is offline until 21:00 UTC for a server replacement http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html" | 20:05 | |
| fungi | yolanda: and per jeblair's suggestio, delete the initscripts there at step 1.1 | 20:06 |
| yolanda | done | 20:06 |
| fungi | yolanda: then feel free to proceed with the db backup at step 2 | 20:06 |
| fungi | note i updated the path to be on the cinder volume, so we can also use that to make sure we've got the right volume at step 5.4 | 20:07 |
| sdake | you probalaby know this but getting 503 on review.oo | 20:07 |
| clarkb | fungi: yolanda should we keep a local copy too so that part of the backup isn't invovled in the move? | 20:07 |
| yolanda | ah good thought | 20:07 |
| anteaya | sdake: yes: http://lists.openstack.org/pipermail/openstack-dev/2016-March/088985.html | 20:07 |
| *** bkeller` has quit IRC | 20:07 | |
| yolanda | backup is processing | 20:07 |
| clarkb | (I don't expect there to be problems moving volumes but I am also paranoid) | 20:07 |
| anteaya | sdake: we are replacing the server | 20:07 |
| sdake | groan | 20:07 |
| *** jamesmcarthur has joined #openstack-infra | 20:07 | |
| mordred | (see irc channel status) | 20:07 |
| sdake | ok thanks ;) | 20:07 |
| sdake | i see now | 20:08 |
| anteaya | welcome | 20:08 |
| mordred | :) | 20:08 |
| sdake | apologies for noise | 20:08 |
| fungi | sdake: and mailing list announcements going back a month ;) | 20:08 |
| * mordred hands anteaya an extra cuddly bunny | 20:08 | |
| sdake | fungi yup i remember those actually :) | 20:08 |
| *** tosky has quit IRC | 20:08 | |
| anteaya | mordred: awe, thank you | 20:08 |
| jeblair | sdake: we should add #openstack-kolla to the statusbot config so you'll see the announcements there | 20:08 |
| fungi | clarkb: yolanda: yep, lets cp that dump file into ~root/ too | 20:08 |
| openstackstatus | fungi: finished sending alert | 20:08 |
| *** tosky has joined #openstack-infra | 20:09 | |
| fungi | i'll add a step 2.1 for that | 20:09 |
| *** shamail has left #openstack-infra | 20:09 | |
| yolanda | ++ | 20:09 |
| anteaya | jeblair: the patch is up to system-config to do that | 20:09 |
| *** yarkot_ has quit IRC | 20:09 | |
| *** dirk has joined #openstack-infra | 20:09 | |
| *** amrith is now known as _amrith_ | 20:09 | |
| *** bkeller` has joined #openstack-infra | 20:09 | |
| yolanda | backup is taking its time | 20:10 |
| sdake | jeblair i'll do that | 20:10 |
| *** maishsk has joined #openstack-infra | 20:10 | |
| dirk | hi, does anyone know whether http://docs.openstack.org/infra/manual/drivers.html#create-stable-branch is still current? | 20:10 |
| sdake | jeblair it was atually done already i think | 20:10 |
| sdake | waiting for no meetings to go on for its final ack I think | 20:10 |
| dirk | when I follow the instructions, I get a gerrit error "503 service unavailable" | 20:10 |
| *** mylu has joined #openstack-infra | 20:10 | |
| sdake | jeblair i'd point out review - but gerrit is in maintenance ;) | 20:10 |
| jeblair | sdake, anteaya: yep, 299009; i just approved it. | 20:10 |
| *** oomichi has joined #openstack-infra | 20:10 | |
| clarkb | dirk: gerrit is offline for a srver upgrade (see channel topic) | 20:10 |
| anteaya | jeblair: thank you | 20:11 |
| fungi | dirk: see channel topic in here and in most official channels | 20:11 |
| fungi | dirk: and prior announcements to the -dev and -infra mailing lists | 20:11 |
| dirk | clarkb: fungi : d'oh. sorry for the noise | 20:11 |
| jeblair | sdake, anteaya: (i don't let little details like gerrit being offline stop me from approving patches ;) | 20:11 |
| anteaya | jeblair: I see that | 20:11 |
| yolanda | ok backup done | 20:11 |
| anteaya | you are prepared | 20:11 |
| fungi | three cheers for asynchronous reviewing | 20:11 |
| yolanda | going to proceed with the volume umount | 20:11 |
| anteaya | hip hip | 20:11 |
| jeblair | hooray | 20:12 |
| fungi | yolanda: you copied the backup file to root's homedir too (new step 2.1)? | 20:12 |
| yolanda | yes | 20:12 |
| fungi | ahh, i see you marked it out. perfect | 20:12 |
| yolanda | /home/gerrit2 device is busy | 20:12 |
| sdake | best wishes for success on the upgrade folks | 20:12 |
| sdake | i guess i'll go do something else - not sure what :) | 20:12 |
| fungi | yolanda: can use lsof to see if someone/something has an open file handle in there | 20:12 |
| yolanda | someone is tailing logs | 20:12 |
| anteaya | sdake: some folks like bug triage | 20:12 |
| yolanda | ok to kill it? | 20:12 |
| anteaya | I hear it is a popular choice during gerrit outages | 20:13 |
| sdake | anteaya ya that is what i was doing with gerrit at the time :) | 20:13 |
| anteaya | cool | 20:13 |
| mordred | fungi, yolanda: next step is dns - should one of us do that asynchronously while the backups and stuff are happening? | 20:13 |
| fungi | yolanda: yeah, kill any processes you see with an open fd in there | 20:13 |
| sdake | to make our plate more accurately measured for htis week ;) | 20:13 |
| fungi | mordred: sure, you want to go ahead and do that? | 20:13 |
| fungi | details are all there in the pad | 20:13 |
| mordred | fungi: on it | 20:13 |
| fungi | thanks! | 20:14 |
| yolanda | umount /home/gerrit2 | 20:14 |
| yolanda | root@review:/home/yolanda# vgchange -a n | 20:14 |
| yolanda | 0 logical volume(s) in volume group "main" now active | 20:14 |
| yolanda | done | 20:14 |
| fungi | awesome | 20:14 |
| fungi | yolanda: step 5 next i guess, since mordred's working on 3 and 4 got moved to pre-maintenance | 20:15 |
| yolanda | ok to proceed with volume changes? | 20:15 |
| *** esikachev has quit IRC | 20:15 | |
| nibalizer | niec | 20:15 |
| fungi | yolanda: yep, should be all set | 20:15 |
| yolanda | fungi, shall root on puppetmaster have the right creds? getting an error on nova, missing username | 20:16 |
| jeblair | zuul is still awol, i'm just going to reboot the server | 20:16 |
| anteaya | jeblair: makes sense | 20:16 |
| yolanda | i shall export OS_CLOUD ? | 20:16 |
| mordred | DNS done | 20:16 |
| fungi | jeblair: yep, i see i/o still falling (and it's started responding to snmp again, at least briefly) but no guarantees this will subside in any way other than the server falling over anyway | 20:17 |
| mordred | yolanda: OS_CLOUD only works with openstack client | 20:17 |
| jeblair | fungi: or take >1hr to do so | 20:17 |
| mordred | yolanda: you either need to source the variable or use osc | 20:17 |
| fungi | yolanda: . ~root/ci-launch/openstackci-rs-nova.sh | 20:17 |
| yolanda | ah thankx | 20:17 |
| fungi | yolanda: . ~root/ci-launch/cinder.sh | 20:17 |
| fungi | you want both of those | 20:17 |
| fungi | the latter makes cinderclient work | 20:17 |
| *** yamahata has quit IRC | 20:17 | |
| fungi | though i guess you only need novaclient for this part | 20:18 |
| mordred | when we're done with this - I would like the overhaul the way we do those things | 20:18 |
| *** shardy_afk has quit IRC | 20:18 | |
| fungi | so the first one is i guess sufficient after all unless you want to use cinderclient to diagnose anything | 20:18 |
| yolanda | ok, volumes switched | 20:18 |
| anteaya | mordred: those things == volume commands? | 20:18 |
| fungi | mordred: agreed, we can update to use `export OS_CLOUD=openstackci-rax` i guess | 20:19 |
| *** maishsk has quit IRC | 20:19 | |
| fungi | anteaya: server actually | 20:19 |
| mordred | well, any of them | 20:19 |
| jeblair | mordred: yeah, the files are in place now, we just need to update the docs i think | 20:19 |
| mordred | jeblair: yup | 20:19 |
| clarkb | ya I use them almost exclusively now | 20:19 |
| anteaya | fungi: thank you | 20:19 |
| *** jamesmcarthur has quit IRC | 20:19 | |
| fungi | oh, i thought anteaya was asking about the specific novaclient commands translating to openstackclient | 20:19 |
| clarkb | there are a lot of minor issues that can be dangerous though | 20:19 |
| mordred | I'd actually like for us to be able to write little ansible playbooks for these things | 20:19 |
| clarkb | the most recent one I found was the paging by default so you don't actually operate on a complete set of data | 20:20 |
| mordred | so that we can review and the manual step is "run the playbook for the maint" | 20:20 |
| fungi | yolanda: should also check dmesg on the new server and the dev fs | 20:20 |
| mordred | clarkb: *headdesk* | 20:20 |
| clarkb | mordred: having done that for the elasticsearch upgrade it works slightly well | 20:20 |
| fungi | yolanda: i see [Mon Apr 11 20:18:47 2016] xvdb: xvdb1 | 20:20 |
| clarkb | mordred: but then we bikeshed the details after the upgrade and never merge the playbook | 20:20 |
| mordred | yah | 20:20 |
| mordred | I mean | 20:20 |
| clarkb | mordred: I think if we go that route we have to be more "this is what we did" and less "this is ZOMG perfect" | 20:20 |
| mordred | it's not a perfect process yet - merely an aspirational desire | 20:20 |
| fungi | yolanda: also i see a /dev/xvdb1 on the new server | 20:21 |
| jeblair | let's discuss this later? | 20:21 |
| mordred | jeblair: ++ | 20:21 |
| jeblair | i'm going to tell nodepool to delete all | 20:21 |
| *** zz_dimtruck is now known as dimtruck | 20:21 | |
| anteaya | ack | 20:21 |
| fungi | yolanda: pvs, vgs and lvs also show what i'd expect on the new server now | 20:21 |
| yolanda | fungi yes, it looks good | 20:21 |
| fungi | yolanda: note the step we added at 5.3 per clarkb's suggestion to clear out the mountpoint before we mount there | 20:22 |
| jeblair | done; nodepool is purging | 20:22 |
| yolanda | yes, doing step by step | 20:22 |
| *** achanda has quit IRC | 20:22 | |
| yolanda | it's nice to have everything documented like that | 20:22 |
| fungi | so we've found ;) | 20:22 |
| yolanda | ok 5.3 done | 20:23 |
| fungi | makes it easier to remember what we did months down the road too | 20:23 |
| fungi | yolanda: yep, looks good! | 20:23 |
| fungi | i guess go ahead and mount it | 20:23 |
| yolanda | we have volume with the backup on place | 20:23 |
| fungi | and we'll make sure we see the db backup you created earlier in there | 20:23 |
| *** jsavak has quit IRC | 20:23 | |
| fungi | there is is. perfect | 20:23 |
| *** jamesmcarthur has joined #openstack-infra | 20:23 | |
| fungi | er, there it is | 20:23 |
| yolanda | ok, the volume steps are done | 20:24 |
| yolanda | are we ok to start with puppetmaster? | 20:24 |
| fungi | should be fine to start puppeting them (undisable the new server, make sure we can sudo ssh to it by name, run ansible) | 20:24 |
| yolanda | ++ | 20:24 |
| *** trukise has joined #openstack-infra | 20:25 | |
| fungi | mordred: do we need to clear the ansible host cache at this point too? | 20:25 |
| fungi | er, ansible inventory cache | 20:25 |
| mordred | fungi: might as well | 20:25 |
| *** maishsk has joined #openstack-infra | 20:25 | |
| jeblair | zuul change 281166 staged on zuul.o.o | 20:25 |
| mordred | yh, waiut | 20:25 |
| mordred | no | 20:25 |
| mordred | it will not do anything | 20:25 |
| fungi | okay | 20:25 |
| mordred | the old review,o.o still exists | 20:25 |
| *** sridhar_ram has joined #openstack-infra | 20:25 | |
| fungi | oh, right, the inventory is independent of enabled/disabled status in groups | 20:25 |
| *** trukise has quit IRC | 20:26 | |
| anteaya | jeblair: yay | 20:26 |
| yolanda | ok running puppet | 20:26 |
| *** openstackgerrit has joined #openstack-infra | 20:27 | |
| yolanda | fatal: [afb0cdb2-4800-4268-af83-0b2d5f68a635]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_args": {"environment": null, "facter_basename": "ansible", "facts": {"project_config_ref": "5c6ee77f47d05ae730f7335dc6a90d01e209eb99"}, "logdest": "syslog", "manifest": "/opt/system-config/production/manifests/site.pp", "puppetmaster": null, "show_diff": false, "timeout": "30m"}, "module_name": "puppet"}, "msg": "/usr/bin/timeout - | 20:27 |
| yolanda | s 9 30m /usr/bin/puppet apply --detailed-exitcodes --logdest syslog --no-noop /opt/system-config/production/manifests/site.pp failed with return code: 6", "rc": 6, "stderr": "", "stdout": "", "stdout_lines": []} | 20:27 |
| yolanda | so mordred, what do we do with that? we run puppet apply inside the vm? | 20:28 |
| mordred | error code 6 ... | 20:28 |
| clarkb | look at the puppet logs in syslog | 20:28 |
| mordred | yah | 20:28 |
| clarkb | that means puppet itself failed | 20:28 |
| *** julim has quit IRC | 20:28 | |
| jeblair | Apr 11 20:27:28 review puppet-user[19602]: (/Stage[main]/Openstack_project::Gerrit/Exec[manage_projects]) Failed to call refresh: /usr/local/bin/manage-projects -v -l /var/log/manage_projects.log returned 1 instead of one of [0] | 20:28 |
| jeblair | Apr 11 20:27:28 review puppet-user[19602]: (/Stage[main]/Openstack_project::Gerrit/Exec[manage_projects]/returns) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 29418 on or 104.130.246.91 | 20:28 |
| jeblair | it tried to run m-p | 20:28 |
| *** jsavak has joined #openstack-infra | 20:29 | |
| nibalizer | its a traceback in manage-projects | 20:29 |
| mordred | ah. yes. it's not going to be able to do that | 20:29 |
| yolanda | gerrit is stopped of course.. | 20:29 |
| fungi | ohh, we have uid/gid mismatch | 20:29 |
| nibalizer | paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 29418 on or 104.130.246.91 | 20:29 |
| fungi | yolanda: need to `sudo chown -R gerrit2:gerrit2 /home/gerrit2` | 20:29 |
| clarkb | oh fun | 20:29 |
| fungi | on old review, gerrit2 was 1001/1001, on new review it's 3000/3000 | 20:30 |
| nibalizer | heh, nice | 20:30 |
| fungi | so puppet is trying to start things as gerrit2 and getting permissions issues | 20:30 |
| yolanda | changing ownership | 20:30 |
| nibalizer | fungi: it figured out some stuff on its own :P (/Stage[main]/Openstack_project::Gerrit/File[/home/gerrit2/acls/openstack/fuel-ostf.config]/owner) owner changed '1001' to 'gerrit2' | 20:31 |
| clarkb | dont't forget to chown the top level dir as well | 20:31 |
| fungi | nibalizer: indeed it did, just... not enough stuff | 20:31 |
| clarkb | er that command will do it nevermind | 20:31 |
| fungi | clarkb: it should get that, but worth double-checking | 20:31 |
| *** jamesmcarthur has quit IRC | 20:32 | |
| anteaya | I added the chown command to the etherpad on line 32 if someone wants to knit that into step 6 after we confirm it works for archival purposes | 20:32 |
| *** amphiprions has joined #openstack-infra | 20:32 | |
| jeblair | are we counting on the puppet-apply to do anything important for us? | 20:33 |
| anteaya | works, for archival purposes | 20:33 |
| clarkb | jeblair: I think just to make sure puppet is happy | 20:33 |
| *** jsavak has quit IRC | 20:33 | |
| jeblair | in other words, do we need to engineer how to run it before the window is over, or wait until then? | 20:33 |
| clarkb | and remove any delta between old and new servers if that exists | 20:33 |
| fungi | well, it's had puppet disabled on it for a while | 20:33 |
| reed | jeblair, your comment about being able to apt-get install git-review on windows is quite premature :) | 20:33 |
| jeblair | reed: but funny, right? :) | 20:33 |
| fungi | and also it's applying the heap limit bump change for us | 20:33 |
| reed | totally :) | 20:34 |
| *** jsavak has joined #openstack-infra | 20:34 | |
| jeblair | reed: that's what i was going for :) | 20:34 |
| jeblair | reed: (but the other bits were serious ;) | 20:34 |
| mordred | (I'm not sure why it's premature - you're supposed to be able to apt-get install anything in the ubuntu archive - I would expect git-review to work) | 20:34 |
| reed | i admit that I had to pause and think about it... | 20:34 |
| fungi | so yeah, i think it's necessary | 20:34 |
| yolanda | changing the ownership is taking long | 20:34 |
| reed | mordred, it's not released yet | 20:34 |
| mordred | reed: ah | 20:34 |
| fungi | yolanda: those git repos have... a lot of loose objects which are all separate files | 20:34 |
| reed | it will be in the next windows version | 20:34 |
| mordred | yolanda: yah. it' sgot to recurse the whole tree | 20:34 |
| mordred | with the gits | 20:35 |
| *** infra-re_ has joined #openstack-infra | 20:35 | |
| mordred | fungi: are we managing the uid/gid of gerrit2 now? | 20:35 |
| *** ddieterly is now known as ddieterly[away] | 20:35 | |
| clarkb | mordred: no, but we shifted the base number | 20:35 |
| fungi | mordred: we're managing the range of uids/gids for accounts created by puppet | 20:35 |
| clarkb | mordred: to give us 1k uids/gids for our users | 20:35 |
| mordred | k | 20:35 |
| nibalizer | we can se the manage projects exec to be notrun. or add '1' to its acceptable output codes | 20:35 |
| nibalizer | i do think we want a clear puppet run before we attempt to start gerrit | 20:36 |
| *** skolekonov has joined #openstack-infra | 20:36 | |
| clarkb | nibalizer: it runs on refresh | 20:36 |
| clarkb | which it shouldn't refresh after the chown I hope | 20:36 |
| *** kgiusti has left #openstack-infra | 20:36 | |
| *** dprince has quit IRC | 20:36 | |
| fungi | clarkb: well, 1k is the former conflict zone. we manually pin uids/gids in the 2k range and puppet is creating non-pinned users/groups in the 3k and up range | 20:36 |
| jeblair | well that's simple enough :) | 20:36 |
| *** angdraug has joined #openstack-infra | 20:37 | |
| fungi | i guess we could have changed uid/gid in /etc/{group,gshadow,passwd,shadow} instead of the chown but this makes us redeployable more easily later | 20:37 |
| *** skolekonov has quit IRC | 20:37 | |
| nibalizer | clarkb: i agree | 20:37 |
| jeblair | are 'disable db backup', 'disable server backup', and 'disable github close pull requests cronjob' all on the old server things we should add to the list and do now? | 20:38 |
| *** infra-red has quit IRC | 20:38 | |
| jeblair | and openstackwatch | 20:38 |
| yolanda | ok ownership change finished, going to run again | 20:39 |
| *** baoli has joined #openstack-infra | 20:39 | |
| clarkb | jeblair: with puppet disabled and volume moved I think we can go ahead iwt hthose | 20:39 |
| clarkb | will let fungi and yolanda confirm though | 20:39 |
| fungi | jeblair: yes, i think they should be safe to do now on the old server | 20:39 |
| yolanda | yes, i think is safe | 20:39 |
| fungi | mostly it's just more stuff to undo if we discover we need to roll this back, but i think the risk is minimal | 20:40 |
| jeblair | okay, all cron jobs deleted | 20:40 |
| fungi | thanks jeblair | 20:40 |
| jeblair | fungi: and puppet will put them back for us | 20:40 |
| jeblair | if we want | 20:40 |
| yolanda | ok puppet finished successfully now | 20:40 |
| fungi | exactly | 20:40 |
| *** doug-fis_ has joined #openstack-infra | 20:40 | |
| fungi | yolanda: excellent. that was just the chown we needed then? | 20:41 |
| yolanda | yes, i didn't do anything else | 20:41 |
| clarkb | ya what was happening was puppet was changing ownership on the files which triggered th eexec refresh | 20:41 |
| yolanda | so shall i restart the server now? | 20:41 |
| fungi | okay, so now we want to reboot and make sure everything comes back up correctly | 20:41 |
| fungi | yes | 20:41 |
| yolanda | ++ | 20:41 |
| clarkb | since we did the chown out of band there was no exec refresh | 20:41 |
| *** openstackgerrit has quit IRC | 20:41 | |
| jeblair | puppet did change some things back to root, which is good. i don't see anything scary in the puppet log. | 20:42 |
| jeblair | (not everything in ~gerrit2 should be root) | 20:42 |
| jeblair | er should be gerrit2 | 20:42 |
| fungi | same (or i didn't, before i got diconnected for the reboot) | 20:42 |
| anteaya | yolanda: can you add the chown command you ran to step 6 so the history is accurate? | 20:42 |
| *** ddieterly[away] is now known as ddieterly | 20:42 | |
| *** hdd has joined #openstack-infra | 20:43 | |
| jeblair | back up | 20:43 |
| fungi | mainly i think we make sure the puppet-managed files are root-owned and gerrit2-readable so that it can't alter them for us, but we need to make sure that's encoded in puppet anywhere we find it not happening | 20:43 |
| nibalizer | ++ | 20:43 |
| yolanda | anteaya, added | 20:43 |
| *** jamesmcarthur has joined #openstack-infra | 20:43 | |
| anteaya | yolanda: thank you | 20:44 |
| fungi | gerrit seems to be running | 20:44 |
| yolanda | ok, gerrit and gerritbot are running | 20:44 |
| anteaya | gerrit renders for me | 20:44 |
| fungi | yay, almost done! | 20:44 |
| jeblair | shall i restart zuul? | 20:44 |
| fungi | yep, now's as good a time as any | 20:44 |
| yolanda | so there is a manual apply of a change pending, jeblair, are you dealing with that? | 20:45 |
| jeblair | yolanda: its done | 20:45 |
| fungi | he already staged it earlier | 20:45 |
| *** doug-fis_ has quit IRC | 20:45 | |
| yolanda | cool | 20:45 |
| jeblair | my gertty seems happy | 20:45 |
| yolanda | so what's pending ? a matter of checking keys in zuul, git mirrors, etc? | 20:45 |
| anteaya | woooo | 20:45 |
| jeblair | zuul is starting | 20:46 |
| fungi | gerritbot looks unhappy | 20:46 |
| anteaya | :( | 20:46 |
| *** maishsk has quit IRC | 20:46 | |
| anteaya | yolanda: yes I guess that is step 8 | 20:46 |
| fungi | oh, it's getting data now | 20:46 |
| yolanda | yep, gerritbot log shows fine | 20:46 |
| craige | o/ | 20:46 |
| mordred | I have rechecked an orphaned o/r change | 20:47 |
| mordred | and it has been picked up | 20:47 |
| fungi | i was seeing it spam tracebacks up until a moment ago | 20:47 |
| *** abregman has quit IRC | 20:47 | |
| clarkb | https://review.openstack.org/monitoring reports the heap size is set correctly | 20:47 |
| fungi | AttributeError: 'GerritBot' object has no attribute 'reconnect' | 20:47 |
| clarkb | you have t oexpand the details area | 20:47 |
| fungi | wondering if we're on a different ircpython version | 20:47 |
| mordred | o good | 20:47 |
| mordred | what's the old review.o.o ip? | 20:48 |
| *** hdd has quit IRC | 20:48 | |
| anteaya | I logged into review.openstack.org via chrome (which I was not logged in with before) and logging in worked for me | 20:48 |
| jeblair | mordred: it's in the etherpad https://etherpad.openstack.org/p/gerrit_server_replacement | 20:48 |
| fungi | yeah, old review.o.o has irc (13.1) | 20:48 |
| pleia2 | mordred: listed at http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 20:48 |
| anteaya | clarkb: yay | 20:48 |
| fungi | new review has irc (14.1) | 20:48 |
| pleia2 | 104.130.159.134 | 20:48 |
| fungi | wondering if something's different there | 20:48 |
| *** maishsk has joined #openstack-infra | 20:48 | |
| nibalizer | mordred: 104.130.159.134 | 20:48 |
| mordred | I think we should downgrade old-review to 13.1 for now | 20:48 |
| yolanda | i tried a gerrit show-queue, shows 7762 tasks now | 20:48 |
| nibalizer | o pleia2 beat me to it | 20:48 |
| mordred | till we figure it out | 20:49 |
| yolanda | going to check logs as well | 20:49 |
| clarkb | mordred: you mean new review to 13.1? | 20:49 |
| mordred | yes | 20:49 |
| mordred | sorry | 20:49 |
| clarkb | wfm | 20:49 |
| yolanda | replication tasks are shrinking so it looks ok | 20:49 |
| fungi | yolanda: i expect most of the gerrit task queue is replicating into /var/lib/git | 20:49 |
| fungi | or /opt/git or wherever we ended up with that | 20:50 |
| fungi | i expect it should go quickly since it's local | 20:50 |
| yolanda | yep, there are tasks for local replication, and some others for mirrors | 20:50 |
| mordred | http://paste.openstack.org/show/493690 is the pip installed differences between the two | 20:51 |
| *** esikachev has joined #openstack-infra | 20:51 | |
| *** jsavak has quit IRC | 20:51 | |
| fungi | it seems to be listening to the event stream fine, but not connecting to freenode | 20:51 |
| fungi | so i strongly suspect it's something different in the irc module | 20:51 |
| *** jsavak has joined #openstack-infra | 20:51 | |
| fungi | oh | 20:52 |
| *** jamesmcarthur has quit IRC | 20:52 | |
| fungi | we were running a non-release version of gerritbot too | 20:52 |
| mordred | wow. something caused +d2to1==0.2.12.post1 | 20:52 |
| fungi | on the old server | 20:52 |
| *** doug-fish has joined #openstack-infra | 20:52 | |
| mordred | to be installed | 20:52 |
| * mordred needs to track that down | 20:52 | |
| fungi | mordred: i bet it's gerritbot 0.2.0 being ancient | 20:53 |
| mordred | yah | 20:53 |
| mordred | I agree | 20:53 |
| fungi | how about i upgrade gerritbot to master for the short term and get ready to tag a 0.3.0 | 20:53 |
| anteaya | fungi: I can live with that | 20:53 |
| *** jamesmcarthur has joined #openstack-infra | 20:53 | |
| mordred | fungi: ++ | 20:53 |
| nibalizer | wfm | 20:54 |
| anteaya | https://pypi.python.org/pypi/d2to1 | 20:54 |
| fungi | yeah, last commit on master is jhesketh's "Fix reconnection method" | 20:54 |
| *** openstackgerrit has joined #openstack-infra | 20:55 | |
| fungi | okay, gerritbot 0.2.1.dev11 is installed on new review, same as was on old, and i've restarted it | 20:55 |
| fungi | and it's joining channels now | 20:55 |
| fungi | so that seems to have been it | 20:55 |
| anteaya | d2to1 looks like it is a packageing tool | 20:55 |
| clarkb | anteaya: ya its the precursor to pbr | 20:56 |
| anteaya | ah | 20:56 |
| mordred | yup | 20:56 |
| mordred | it sohuldn't really show up anywhere we do anything | 20:56 |
| *** bpokorny_ has joined #openstack-infra | 20:56 | |
| mordred | anymore | 20:56 |
| clarkb | the java melody data looks good so far | 20:56 |
| anteaya | makes sense | 20:56 |
| clarkb | we have spiked to 9GB heap used | 20:56 |
| anteaya | clarkb: yay! | 20:56 |
| fungi | openstackgerrit is in here now so i think we're good on that front | 20:57 |
| yolanda | gerrit log shows some connection reset by peer errors | 20:57 |
| clarkb | I expect that to steadily climb past where we were at 12GB earlier | 20:57 |
| *** doug-fish has quit IRC | 20:57 | |
| mordred | jeblair: you did the nodepooling earlier, yeah? | 20:57 |
| *** doug-fish has joined #openstack-infra | 20:57 | |
| mordred | jeblair: we seem to not have new nodes yet | 20:57 |
| fungi | sounded like he cleared all old nodes from it. may need a restart to find gearman again | 20:58 |
| fungi | ? | 20:58 |
| mordred | ah. maybe that's it? | 20:58 |
| yolanda | mm | 20:58 |
| yolanda | [2016-04-11 20:58:14,532] WARN com.google.gerrit.sshd.GerritServerSession : Exception caught | 20:58 |
| yolanda | java.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 / server: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1) | 20:58 |
| fungi | yolanda: i'm willing to bet that's benign. looks like someone tried to connect with a key exchange not supported by mina-sshd | 20:59 |
| clarkb | fungi: mordred you should be able to check the nodepool log to see if it is connected to gearman or not | 20:59 |
| *** bpokorny has quit IRC | 20:59 | |
| yolanda | fungi yes, i tried clones, reviews, etc... and everything looks ok | 20:59 |
| clarkb | hrm ya it seems ot be allocated not much | 21:00 |
| yolanda | however the same error repeats for several times, it may be same one retrying | 21:00 |
| clarkb | which should indicate no load from gearman | 21:00 |
| *** Sukhdev has joined #openstack-infra | 21:00 | |
| jeblair | looking | 21:00 |
| *** trukise has joined #openstack-infra | 21:00 | |
| *** trukise has quit IRC | 21:01 | |
| anteaya | are we out of maintenance yet? | 21:01 |
| jeblair | yeah, it probably died with zuul | 21:01 |
| jeblair | the connection that is | 21:01 |
| jeblair | i'll restart nodepool | 21:01 |
| anteaya | or do we have more steps to complete? | 21:01 |
| clarkb | anteaya: need to get nodepool making nodes again but after that I think we are mostly good? | 21:02 |
| mordred | yah | 21:02 |
| anteaya | cool | 21:02 |
| fungi | it's probably safe to clear the alert and let people start assuming gerrit's working (though nodepool's still on the blink for a bit maybe) | 21:02 |
| clarkb | ya, everything should queue up and get served as able | 21:02 |
| anteaya | yolanda: nice work on this | 21:02 |
| *** ssancheztrujillo has joined #openstack-infra | 21:02 | |
| anteaya | yolanda: great prep | 21:02 |
| *** jsavak has quit IRC | 21:03 | |
| yolanda | anteaya, kudos to all team, the etherpad was greatly documented | 21:03 |
| fungi | #alert ok Gerrit move maintenance completed successfully; note that DNS has been updated to new IP addresses as indicated in http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 21:03 |
| *** rossella_s has quit IRC | 21:03 | |
| anteaya | yolanda: agreed | 21:03 |
| fungi | #status ok Gerrit move maintenance completed successfully; note that DNS has been updated to new IP addresses as indicated in http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 21:03 |
| openstackstatus | fungi: sending ok | 21:03 |
| *** eharney has quit IRC | 21:03 | |
| *** weshay has quit IRC | 21:03 | |
| * fungi grumbles at his inept fingers | 21:03 | |
| *** rossella_s has joined #openstack-infra | 21:03 | |
| yolanda | is great when things go so smoothly | 21:03 |
| anteaya | indeed it is | 21:03 |
| anteaya | yolanda: you want to draft an update to the mailing list? | 21:04 |
| fungi | thanks yolanda (and everyone who helped)! excellently planned and executed | 21:04 |
| yolanda | anteaya, shall we send that now or we wait for zuul/nodepool to complete? | 21:04 |
| anteaya | yolanda: I think you could create a draft in an etherpad | 21:04 |
| *** icey has quit IRC | 21:04 | |
| anteaya | and then we can send once zuul/nodepool are happy again | 21:05 |
| jeblair | i think the mergers may be stuck | 21:05 |
| fungi | 2016-04-11 21:02:02,494 DEBUG nodepool.NodePool: Adding gearman server zuul.openstack.org_4730 | 21:05 |
| jeblair | i think nodepool was probably fine | 21:05 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 21:05 | |
| -openstackstatus- NOTICE: Gerrit move maintenance completed successfully; note that DNS has been updated to new IP addresses as indicated in http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 21:05 | |
| yolanda | i'll create the etherpad and leave for some other infra-root to send | 21:05 |
| anteaya | yolanda: sounds good | 21:05 |
| clarkb | jeblair: oh! that would explain why gearman jobs aren't ending up getting queued | 21:06 |
| clarkb | jeblair: need help restarting those? | 21:06 |
| fungi | jeblair: oh, yep, no job work requests means nodepool sees no demand | 21:06 |
| jeblair | clarkb: i'm going to ansible them to double check and if so, restart them | 21:06 |
| fungi | looking at logs on the mergers | 21:06 |
| clarkb | jeblair: ok, question about nodepool restart, did you make sure the older version with working tripleo was installed? | 21:06 |
| *** AJaeger has quit IRC | 21:07 | |
| fungi | last log entry on zm01 is from 19:44 | 21:07 |
| jeblair | clarkb: no | 21:07 |
| clarkb | we will probably need to restart it again :( mordred jeblair can we revert that change and merge again when it is expected to work? | 21:07 |
| jeblair | fungi: yeah, they all say that; i will restart them all, but only after we ensure we're running the right nodepool | 21:07 |
| jeblair | clarkb: which change should we be running? | 21:07 |
| *** AJaeger has joined #openstack-infra | 21:08 | |
| clarkb | jeblair: afdd58c10a8b85ea1466d32c3a29c0d7a1b0aea9 | 21:08 |
| openstackstatus | fungi: finished sending ok | 21:08 |
| clarkb | jeblair: the one after that commit is the use shade for all interactions change | 21:09 |
| *** regXboi has quit IRC | 21:09 | |
| jeblair | clarkb: oh that, i thought we had another thing on top or something | 21:09 |
| jeblair | restarting nodepool with that | 21:09 |
| clarkb | jeblair: it iddn't work because the assumption about using the first nic was not valid | 21:09 |
| mordred | clarkb: well, I'm hoping to be able to merge the fixes today. however, I'd also be fine with merging a revert and then doing a re-revert | 21:09 |
| clarkb | you have to pick the nic that has an external gateway | 21:09 |
| yolanda | infra-root, anteaya, https://etherpad.openstack.org/p/gerrit_server_replacement_ok | 21:09 |
| mordred | the shade fixes did take longer than I was originally anticipating | 21:09 |
| jeblair | clarkb, mordred: yeah, let's either merge the fixes or a revert today | 21:09 |
| *** fawadkhaliq has quit IRC | 21:10 | |
| mordred | clarkb: you have to pick a nic on a network that has a subnet that has an external gateway | 21:10 |
| mordred | jeblair: ++ | 21:10 |
| *** fawadkhaliq has joined #openstack-infra | 21:10 | |
| jeblair | restarting mergers now | 21:10 |
| jeblair | and so many jobs running now | 21:10 |
| mordred | jeblair: fixes are ready - need to push a response to a Shrews review to the stack as soon as the o/r change lands | 21:10 |
| anteaya | yolanda: looks great to me | 21:10 |
| jeblair | and so many nodes launching now | 21:10 |
| anteaya | yolanda: thank you | 21:10 |
| *** ddutta has quit IRC | 21:11 | |
| yolanda | jeblair, mordred, so if everything is ok, i can just send the email now while i'm connected | 21:11 |
| jeblair | 291 nodes launching | 21:11 |
| *** jamesmcarthur has quit IRC | 21:11 | |
| *** fawadkhaliq has quit IRC | 21:11 | |
| anteaya | yolanda: let's not tempt sending an okay before we are sure | 21:11 |
| *** fawadkhaliq has joined #openstack-infra | 21:12 | |
| anteaya | yolanda: but thanks for all your work, and I appreciate it is late for you | 21:12 |
| clarkb | melody continues to look good | 21:12 |
| anteaya | yay! | 21:12 |
| clarkb | really low http mean times | 21:12 |
| anteaya | wooooo | 21:13 |
| yolanda | anteaya, thanks. Yep, a bit late , but no problem, it's good to have flexible hours | 21:13 |
| anteaya | yolanda: indeed | 21:13 |
| yolanda | ok leaving now, it has been nice, thanks for all the help | 21:13 |
| fungi | have a good evening yolanda! | 21:13 |
| mordred | thanks yolanda ! | 21:13 |
| fungi | mergers definitely seem active again | 21:14 |
| mordred | clarkb, jeblair: however, I don't expect substantial changes to the stack between now and then - so if y'all want to start slogging through https://review.openstack.org/#/q/status:open+project:openstack-infra/shade+branch:master+topic:shade-nodepool | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| mordred | and also https://review.openstack.org/#/c/303850/ | 21:15 |
| *** maishsk has quit IRC | 21:15 | |
| anteaya | ttx: on the newton release schedule: http://releases.openstack.org/newton/schedule.html it appears there is a one week hiatus between ptl self-nomination and the ptl election, is that intentional? | 21:15 |
| mordred | which should be safe to land now | 21:15 |
| *** ajmiller has quit IRC | 21:16 | |
| *** ajmiller has joined #openstack-infra | 21:16 | |
| clarkb | mordred: https://review.openstack.org/#/c/303853/ has WIP in title | 21:19 |
| clarkb | ignore for now? | 21:19 |
| *** maishsk has joined #openstack-infra | 21:19 | |
| mordred | clarkb: yes | 21:20 |
| openstackgerrit | Merged openstack-infra/system-config: Move statusbot and eavesdrop bot for kolla https://review.openstack.org/299009 | 21:20 |
| mordred | clarkb: it's there to remind me to actually do that | 21:20 |
| mordred | it's purely a debug log message noise issue - no substantive things | 21:20 |
| clarkb | yolanda: the etherpad message lgtm | 21:21 |
| mordred | basically the debug logging gets REALLY happy to tell you that it's skipping network investigation because you told it to | 21:21 |
| *** ssancheztrujillo has quit IRC | 21:21 | |
| fungi | looks like all rare node types are registered in zuul's gearman, and not seeing any NOT_REGISTERED failures on the zuul status page | 21:22 |
| fungi | gonna take a break to make dinner, then work up a gerritbot release tag | 21:23 |
| clarkb | mordred: comments in https://review.openstack.org/#/c/300592/4 say done but that is the latest patchset | 21:24 |
| clarkb | where are the things being addressed? | 21:24 |
| anteaya | fungi: enjoy dinner | 21:24 |
| clarkb | we haven't sent the all clear email yet have we? I haven't seen it | 21:25 |
| clarkb | was someone going to do that? | 21:25 |
| anteaya | clarkb: that was yolanda's email | 21:25 |
| anteaya | and no it hasnt' been sent | 21:25 |
| mordred | clarkb: I haven't pushed the next patchset yet because it'll fail the nodepool-shade-src test until the openstack requirements patch lands (it's in gate queue) | 21:25 |
| anteaya | she left it for someone to send once zuul/nodepool was happy | 21:25 |
| anteaya | https://etherpad.openstack.org/p/gerrit_server_replacement_ok for others to review | 21:26 |
| mordred | clarkb: but I have that fixed locally- will push as soon as we won't be wasting the 100 nodes | 21:26 |
| jeblair | oh, i think it's been happy for a while | 21:26 |
| *** fawadkhaliq has quit IRC | 21:26 | |
| clarkb | mordred: ok | 21:26 |
| *** fawadkhaliq has joined #openstack-infra | 21:26 | |
| mordred | clarkb: (also, it's possible you'll find something and I can bundle that in one set of updates) | 21:26 |
| anteaya | who would like to send yolanda's all clear email? | 21:26 |
| tsufiev | Folks, how I can draw zuul attention again to patches that were in gate queue when it was shut off? | 21:26 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 21:26 |
| *** jamesmcarthur has joined #openstack-infra | 21:27 | |
| *** icey has joined #openstack-infra | 21:27 | |
| tsufiev | Namely https://review.openstack.org/#/c/303769/2 and https://review.openstack.org/#/c/299934/ | 21:27 |
| *** |-paul-| has joined #openstack-infra | 21:28 | |
| *** ddutta has joined #openstack-infra | 21:28 | |
| mordred | tsufiev: just leave a recheck comment | 21:28 |
| fungi | tsufiev: leave a comment starting with the word "recheck" (no quotes) since they don't currently have any verified score, and as soon as they get passed by check jobs they'll be automatically enqueued into the gate pipeline | 21:28 |
| *** kzaitsev_mb has quit IRC | 21:28 | |
| openstackgerrit | Merged openstack-infra/system-config: Tell os-client-config that tripleo doesn't have neutron https://review.openstack.org/303850 | 21:29 |
| tsufiev | Thanks, it helped! | 21:29 |
| fungi | unfortunately zuul was in a sad load situation just as we were starting our maintenance, and were unable to ssh into it to save its queue states | 21:29 |
| fungi | anteaya: clarkb: jeblair: yep, i didn't know if anyone was still making sure things seemed stable, but happy to send the follow-up maintenance e-mail in a bit unless someone else is already queuing up what yolanda put in that pad | 21:30 |
| anteaya | fungi: I was not, its all yours | 21:30 |
| *** thiagop has quit IRC | 21:30 | |
| anteaya | fungi: thank you | 21:30 |
| fungi | it could probably stand an extra paragraph about the fact that we lost changes from the check/gate pipelines | 21:30 |
| anteaya | good point | 21:31 |
| fungi | so that people with tsufiev's problem don't all have to come ask us about it | 21:31 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Run less parallelism in ansible https://review.openstack.org/304318 | 21:31 |
| *** kzaitsev_mb has joined #openstack-infra | 21:31 | |
| anteaya | yup | 21:31 |
| *** burgerk_ has joined #openstack-infra | 21:31 | |
| *** infra-re_ has quit IRC | 21:31 | |
| nibalizer | so if we just run puppet apply in launch_node.py now, we can turn off all the apache/puppetmaster stuff ya? | 21:31 |
| anteaya | do you think it should be a reply to either one of these posts? http://lists.openstack.org/pipermail/openstack-dev/2016-March/088985.html or http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html | 21:31 |
| nibalizer | should give us back some ram on the puppetmaster | 21:32 |
| *** oanson has quit IRC | 21:32 | |
| clarkb | nibalizer: yes, except that we still need the boot with volume + run puppet apply stuff to ahppen | 21:32 |
| mordred | nibalizer: yes. I htink we're safe to stop running puppetmaster on puppetmaster | 21:32 |
| mordred | clarkb: yah. but puppetmaster itself doesn't help with that | 21:32 |
| clarkb | because right now the only way to boot and attach a volume is with the master | 21:32 |
| clarkb | mordred: it does in that its the only way it works today | 21:32 |
| mordred | wait - what? | 21:32 |
| clarkb | mordred: because I wrote a change to make it work | 21:32 |
| nibalizer | oh we have two launch node paths right now? | 21:32 |
| clarkb | then it got ignored while we merge conflicted it very badly | 21:32 |
| mordred | clarkb: I thought I'd updated launch-node.py to include your change | 21:33 |
| clarkb | mordred: oh? I never saw that change | 21:33 |
| mordred | yah. I'm pretty sure old launch-node.py no longer exists | 21:33 |
| nibalizer | i dont see 'apply' appearing anywhere in launch_node | 21:33 |
| clarkb | mordred: it doens't I check out my change from gerrit | 21:33 |
| clarkb | which was unmerged last time I looked | 21:33 |
| mordred | sigh | 21:33 |
| mordred | ok | 21:33 |
| mordred | can you point me to your change? | 21:33 |
| clarkb | mordred: https://review.openstack.org/#/c/285477/ | 21:34 |
| *** burgerk has quit IRC | 21:34 | |
| mordred | thanks | 21:34 |
| mordred | nibalizer: also: https://review.openstack.org/#/c/249815/ | 21:35 |
| *** burgerk_ has quit IRC | 21:35 | |
| mordred | clarkb: I'll get that ported in to shade-launch-node | 21:36 |
| clarkb | mordred: ok | 21:36 |
| mordred | and then make patches to make the move final. I could have _sworn_ I already had that patchup, but I do not see it in gerrit, so clearly I'm wrong | 21:36 |
| clarkb | mordred: it does need to lose the ability to set the volume device because apparently that is going away in nova, except its the only way to get vexxhost to work | 21:38 |
| clarkb | so both options are bad | 21:38 |
| *** mylu has quit IRC | 21:38 | |
| *** njohnston has left #openstack-infra | 21:39 | |
| *** ZZelle_ has quit IRC | 21:39 | |
| fungi | anteaya: i probably won't go to the trouble of digging out the message-id header for one of those but i'll follow up with an re: subject line for one of them | 21:39 |
| anteaya | fungi: wonderful, thanks | 21:39 |
| anteaya | helps me when I go looking for it in archives later | 21:39 |
| *** pradk has quit IRC | 21:40 | |
| clarkb | actually its already gone away in nova I think, which is why it iddn't work on osic | 21:40 |
| *** _ari_ has quit IRC | 21:40 | |
| *** vgridnev has quit IRC | 21:41 | |
| *** mylu has joined #openstack-infra | 21:43 | |
| *** icey has quit IRC | 21:43 | |
| *** doug-fish has quit IRC | 21:45 | |
| *** mrmartin has quit IRC | 21:45 | |
| *** bkeller` has quit IRC | 21:46 | |
| *** kzaitsev_mb has quit IRC | 21:46 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:47 | |
| *** ddieterly is now known as ddieterly[away] | 21:48 | |
| *** matt-borland has quit IRC | 21:48 | |
| *** rockyg has joined #openstack-infra | 21:48 | |
| *** icey has joined #openstack-infra | 21:48 | |
| *** mriedem is now known as mriedem_away | 21:48 | |
| *** edmondsw has quit IRC | 21:48 | |
| *** tiswanso has quit IRC | 21:49 | |
| *** bkeller` has joined #openstack-infra | 21:50 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Switch gate-neutron-dsvm-api to be invoked via tempest plugin framework https://review.openstack.org/304321 | 21:51 |
| *** bpokorny_ has quit IRC | 21:51 | |
| *** amitgandhinz has quit IRC | 21:52 | |
| *** bpokorny has joined #openstack-infra | 21:52 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Switch gate-neutron-dsvm-api to be invoked via tempest plugin framework https://review.openstack.org/304321 | 21:53 |
| mordred | clarkb: so, current shade uses block_device_mapping_v2 to do the attach at boot time | 21:56 |
| *** amitgandhinz has joined #openstack-infra | 21:56 | |
| mordred | clarkb: which I'm updating your patch to do | 21:56 |
| clarkb | you can't reliably map them though | 21:56 |
| clarkb | thats the problem, on vexxhost it is the only way to avoid conflicting with the config drive, on osic it doens't work at all | 21:57 |
| clarkb | and apparently that is because nova decided to stop supporting that | 21:57 |
| *** amitgandhinz has quit IRC | 21:58 | |
| * morgan blinks. | 21:58 | |
| morgan | ick. | 21:59 |
| *** piet has quit IRC | 21:59 | |
| *** bkeller` has quit IRC | 22:00 | |
| morgan | clarkb: it makes me sad we have more implementation details leaking through | 22:00 |
| mordred | clarkb: so - lemme cover the non-vexxhost case first | 22:02 |
| *** dimtruck is now known as zz_dimtruck | 22:02 | |
| mordred | and then figure out the vexxhost case | 22:02 |
| clarkb | mordred: ok the non vexxhost case is just attach the volume without telling nova/cinder where to attach it | 22:02 |
| mordred | yah | 22:02 |
| clarkb | mordred: also did find one thing about the nat networks change | 22:02 |
| clarkb | will post that momentarily | 22:03 |
| mordred | woot! | 22:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add support to shade-launch-node for cinder attach https://review.openstack.org/285477 | 22:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add support to launch-node for cinder attach https://review.openstack.org/285477 | 22:03 |
| mordred | there is no shade-launch-node anymore :) | 22:03 |
| clarkb | mordred: comments posted | 22:04 |
| mordred | clarkb: thanks! | 22:04 |
| *** ociuhandu has joined #openstack-infra | 22:04 | |
| *** knikolla has quit IRC | 22:04 | |
| *** lakshmiS has quit IRC | 22:05 | |
| mordred | clarkb: so - that issue you comment in gets fixed in a follow on patch | 22:05 |
| mordred | clarkb: the "if len(ports) AND not self._nat_destination" was really terrible logic | 22:05 |
| mordred | lemme point to it | 22:05 |
| mordred | clarkb: https://review.openstack.org/#/c/303852/2/shade/openstackcloud.py | 22:06 |
| *** jamesmcarthur has quit IRC | 22:07 | |
| *** ddutta has quit IRC | 22:07 | |
| *** dtardivel has quit IRC | 22:08 | |
| *** markus_z has quit IRC | 22:08 | |
| * mordred is starting to think it might be easier to review this as one squashed change | 22:08 | |
| morgan | mordred: in rare cases... | 22:08 |
| *** ihrachys has quit IRC | 22:08 | |
| *** aysyd has quit IRC | 22:09 | |
| *** mixos has quit IRC | 22:09 | |
| mordred | clarkb: oh - you know what else? shade may not have a workaround for the vexxhost case because vexxhost didnt' have cinder back when I wrote most of that | 22:09 |
| clarkb | mordred: one thing on https://review.openstack.org/#/c/285477/8 that got through rebase that shouldn't | 22:10 |
| *** claudiub has quit IRC | 22:10 | |
| clarkb | mordred: ya I think the fix for vexxhost is to figure out why libvirt and nova and cinder are not cooperating | 22:10 |
| mordred | clarkb: I'll take a todo list item to make sure shade works with vexxhost | 22:10 |
| clarkb | mordred: I filed a bug with nova and that seemed to dead end with "what version of libvirt are you running" | 22:10 |
| mordred | clarkb: yah, I mean- ultimately having the cloud not derp is important | 22:11 |
| clarkb | asked mnaser to update that but I don't think that ever happened | 22:11 |
| morgan | #TODO: Why...why...why can't we have interoperable clouds. | 22:11 |
| * mordred hands morgan a potbellied pig | 22:11 | |
| morgan | not that it's wrong to make a selection *if* the software supports it | 22:11 |
| clarkb | mordred: I think the possible alternative here is to use the /dev/by-id path | 22:11 |
| morgan | just.. | 22:11 |
| morgan | mordred: oooh for roasting?! | 22:11 |
| clarkb | mordred: it uses some prefix of the cinder voilume uuid there iirc | 22:11 |
| mordred | clarkb: yes. although we need to see if that works at rax yet | 22:11 |
| clarkb | ah yup | 22:11 |
| clarkb | it wasn't working on review.o.o earlier today :( | 22:11 |
| mordred | clarkb: you can actually do /dev/by-name | 22:11 |
| mordred | clarkb: on not-rax | 22:12 |
| clarkb | or to have vexxhost switch to using iso config drives | 22:12 |
| clarkb | so they show up under a different /dev prefix | 22:12 |
| *** esikachev has quit IRC | 22:13 | |
| fungi | maintenance conclusion e-mail has been sent | 22:13 |
| clarkb | mordred: I do think that https://review.openstack.org/#/c/303852/2 can be squashed into the base change | 22:14 |
| clarkb | thats basically a new patchset | 22:14 |
| clarkb | not a new logical change | 22:14 |
| anteaya | fungi: thank you | 22:16 |
| *** tpsilva has quit IRC | 22:17 | |
| *** maishsk has quit IRC | 22:17 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add support to launch-node for cinder attach https://review.openstack.org/285477 | 22:18 |
| mordred | clarkb: all fixed | 22:18 |
| *** Guest5162 has quit IRC | 22:18 | |
| mordred | clarkb: I'd also dropped the keep argument | 22:18 |
| mordred | clarkb: (this is what I get from trying to save typing by cutting and pasting between files) | 22:18 |
| *** kzaitsev_mb has quit IRC | 22:19 | |
| *** maishsk has joined #openstack-infra | 22:20 | |
| mordred | clarkb: k. just tried on the rebasing. unfortunately stuff moves in between - I could do it, but it might take a few hours | 22:21 |
| mordred | otoh - o/r change moved from check to gate - so only one more hour until we can recheck all of these | 22:21 |
| clarkb | just changing the order and s/pick/squash/ is too conflicty? | 22:22 |
| mordred | yah | 22:22 |
| *** kzaitsev_mb has joined #openstack-infra | 22:22 | |
| mordred | sadly | 22:22 |
| mordred | I tried a few different options for doing tat | 22:22 |
| mordred | just out of curiosity - I'm going to see what the whole thing as one patch looks like | 22:22 |
| clarkb | mordred: also it might be nice to have 3 functions. One find external networks, one find internal networks, and one find nat networks | 22:23 |
| *** jamesmcarthur has joined #openstack-infra | 22:23 | |
| clarkb | to avoid the multipage function def | 22:23 |
| mordred | yah | 22:23 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Switch gate-neutron-dsvm-api to be invoked via tempest plugin framework https://review.openstack.org/304321 | 22:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: SAKE OF ARGUMENT - Fix NAT processing https://review.openstack.org/304326 | 22:25 |
| mordred | clarkb: I'm not sure whether that ^^ is more readable | 22:25 |
| mordred | clarkb: but that's the whole thing squashed | 22:25 |
| mordred | zuul seems to be having a fun time processing its queue | 22:26 |
| mordred | Queue lengths: 3037 events, 138 results. | 22:27 |
| *** gfidente has quit IRC | 22:27 | |
| clarkb | mordred: queues have fallen, I think it just takes time to load up all the merge events and process them when we merge | 22:30 |
| mordred | ah - nod | 22:31 |
| *** mixos has joined #openstack-infra | 22:31 | |
| *** mtanino_ has quit IRC | 22:32 | |
| *** mixos has quit IRC | 22:32 | |
| *** thorst has quit IRC | 22:33 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Revert "Make gate-neutron-dsvm-api job non-voting" https://review.openstack.org/303621 | 22:33 |
| *** thorst has joined #openstack-infra | 22:33 | |
| *** markvoelker has quit IRC | 22:33 | |
| openstackgerrit | Merged openstack/os-testr: Remove openstack-common.conf https://review.openstack.org/300943 | 22:34 |
| *** mixos has joined #openstack-infra | 22:34 | |
| *** mixos has quit IRC | 22:34 | |
| armax | AJaeger: what did I miss on https://review.openstack.org/#/c/303622/? | 22:35 |
| armax | anteaya: ^ | 22:36 |
| *** asettle has joined #openstack-infra | 22:36 | |
| *** ddieterly[away] is now known as ddieterly | 22:37 | |
| *** thorst has quit IRC | 22:38 | |
| *** kzaitsev_mb has quit IRC | 22:38 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:40 | |
| *** jamesmcarthur has quit IRC | 22:41 | |
| *** mixos has joined #openstack-infra | 22:41 | |
| *** mixos has quit IRC | 22:44 | |
| clarkb | cacti and melody continue to look good | 22:46 |
| *** jamesmcarthur has joined #openstack-infra | 22:46 | |
| *** mylu has quit IRC | 22:47 | |
| *** |-paul-| has quit IRC | 22:47 | |
| jhesketh | Morning | 22:48 |
| clarkb | jhesketh: good morning | 22:49 |
| *** mylu has joined #openstack-infra | 22:49 | |
| *** thorst has joined #openstack-infra | 22:50 | |
| *** fawadkhaliq has quit IRC | 22:51 | |
| jhesketh | How'd the gerrit change go? | 22:51 |
| clarkb | pretty well, only had one hickup where the gerrit users uid changed between hosts, but a recursive chown sorted that out | 22:51 |
| *** fawadkhaliq has joined #openstack-infra | 22:51 | |
| anteaya | jhesketh: morning | 22:52 |
| mordred | clarkb: do you have an opinion on one patch or patch series? | 22:52 |
| mordred | morning jhesketh ! | 22:52 |
| clarkb | mordred: I haven't gotten far enough in the stack to know yet | 22:52 |
| jhesketh | Nice stuff :-) | 22:53 |
| *** sdake_ has joined #openstack-infra | 22:53 | |
| *** mylu has quit IRC | 22:53 | |
| anteaya | armax: what do you mean, what did you miss? | 22:54 |
| armax | on the review | 22:55 |
| anteaya | armax: what do you expect to see that you are not seeing? | 22:55 |
| *** mylu has joined #openstack-infra | 22:55 | |
| clarkb | mordred: I htink I do like having utility methods in first, then add in gateway subnet detection for auto port assignment | 22:55 |
| anteaya | armax: yes, what about the review? | 22:55 |
| clarkb | mordred: mostly because then we can test the utiltiies independent of the new features | 22:55 |
| *** sdake has quit IRC | 22:55 | |
| armax | AJaeger says “As discussed on IRC: Now is fine to merge" | 22:55 |
| *** ddieterly is now known as ddieterly[away] | 22:55 | |
| mordred | kk | 22:55 |
| armax | I didn’t recall having conversation on review https://review.openstack.org/#/c/303622/ | 22:55 |
| *** ddieterly[away] is now known as ddieterly | 22:55 | |
| anteaya | armax: he was reviewing two patches at the same time | 22:56 |
| armax | anteaya: I mean, what was the discussion about? I didn’t join the discussion | 22:56 |
| armax | anteaya: ok, I figured something like that happened | 22:56 |
| anteaya | armax: he +A'd that one and the reverted his +A | 22:56 |
| *** Nikolay_St has quit IRC | 22:56 | |
| anteaya | as he +A'd the wrong patch | 22:56 |
| fungi | jhesketh: other (minor) hiccup was that we were running a newer non-release install of gerritbot with your reconnect fix. i need to tag a proper new release of gerritbot | 22:56 |
| anteaya | so in irc he was discussing a different patch | 22:56 |
| anteaya | armax: thanks for checking though | 22:57 |
| armax | anteaya: ok, all good I guess | 22:57 |
| anteaya | all good | 22:57 |
| anteaya | thanks for the follow up :) | 22:57 |
| *** gordc has quit IRC | 22:57 | |
| armax | anteaya: sure | 22:57 |
| armax | thanks | 22:57 |
| anteaya | :) | 22:57 |
| jhesketh | fungi: which change was that? Did it cause problems ? | 22:58 |
| *** sputnik13 has quit IRC | 22:58 | |
| *** rbrndt has quit IRC | 22:59 | |
| fungi | jhesketh: it's tip of master. your change was fine, we just haven't released gerritbot in a couple years | 23:00 |
| openstackgerrit | Zhipeng Huang proposed openstack-infra/project-config: Add new project Coupler to openstack repo https://review.openstack.org/302791 | 23:01 |
| jhesketh | fungi: ah cool. | 23:03 |
| jhesketh | And I believe the zuul connection change was applied too? Has that been going okay? | 23:03 |
| *** kzaitsev_mb has quit IRC | 23:04 | |
| clarkb | I haven't seen zuul anything odd, but not sure how to monitor that closely | 23:04 |
| *** kzaitsev_mb has joined #openstack-infra | 23:04 | |
| *** maishsk has quit IRC | 23:04 | |
| *** maishsk_ has joined #openstack-infra | 23:04 | |
| anteaya | jhesketh: 281166 was manually applied to zuul before zuul was started | 23:05 |
| jhesketh | Okay. Guess we just keep an eye on it then | 23:05 |
| *** ddieterly is now known as ddieterly[away] | 23:05 | |
| *** ddieterly[away] has quit IRC | 23:06 | |
| anteaya | yeah, pretty early to tell | 23:06 |
| fungi | the memory leak has been taking days/weeks to surface, so only time will tell | 23:06 |
| *** Sukhdev has quit IRC | 23:07 | |
| *** dingyichen has joined #openstack-infra | 23:08 | |
| asselin | dstufft, you around? We're seeing errrors on missing mosquitto 1.2.3 https://pypi.python.org/pypi/mosquitto | 23:08 |
| asselin | pulled in from https://github.com/python-beaver/python-beaver/blob/master/requirements/base.txt | 23:08 |
| anteaya | asselin: I just realized I competely forgot about the third party meeting today | 23:08 |
| dstufft | what kind of errors | 23:08 |
| anteaya | asselin: I was focused on the gerrit maintenance | 23:08 |
| anteaya | and got up from my chair and realized what I had done | 23:09 |
| clarkb | anteaya: google says http://mosquitto.org/documentation/python/ | 23:09 |
| anteaya | asselin: sorry about that | 23:09 |
| anteaya | clarkb: thanks | 23:09 |
| asselin | anteaya, I missed it too | 23:10 |
| *** kzaitsev_mb has quit IRC | 23:10 | |
| clarkb | er asselin google says ^ | 23:10 |
| dstufft | oh, yea it looks like it got deleted today | 23:10 |
| asselin | dstufft, ok...seems the package was deleted/renamed as clarkb said | 23:10 |
| *** keedya has joined #openstack-infra | 23:10 | |
| anteaya | asselin: great minds think alike apparently | 23:10 |
| dstufft | mosquitto | | remove | 2016-04-11 20:34:22 | 23:10 |
| dstufft | I need to implement the thing that doesn't allow re-registering a name | 23:12 |
| dstufft | :/ | 23:12 |
| anteaya | oh my | 23:13 |
| anteaya | that would be painful | 23:13 |
| asselin | dstufft, i'm surprised deleting is allowed... | 23:13 |
| *** jamesmcarthur has quit IRC | 23:16 | |
| dstufft | asselin: PyPI is a balancing act between giving power to authors to manage their projects as best they see fit, and taking power away from authors to provide more consistency to end users | 23:17 |
| anteaya | sounds familiar | 23:17 |
| *** maishsk_ has quit IRC | 23:19 | |
| asselin | yeah....cause issue like this: https://github.com/python-beaver/python-beaver/issues/398 | 23:19 |
| asselin | anyway, understood. I found it in a snapshot, so we'll be ok. | 23:19 |
| *** maishsk has joined #openstack-infra | 23:21 | |
| *** salv-orlando has quit IRC | 23:21 | |
| *** furlongm has joined #openstack-infra | 23:21 | |
| *** tongli_ has joined #openstack-infra | 23:22 | |
| *** tongli_ has quit IRC | 23:22 | |
| *** jamielennox|away is now known as jamielennox | 23:22 | |
| *** tongli_ has joined #openstack-infra | 23:22 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 23:24 |
| *** yamahata has joined #openstack-infra | 23:24 | |
| asselin | dstufft, but I got your hint finally: I can take over the namespace and re-upload it :p | 23:24 |
| dstufft | asselin: :) | 23:24 |
| * anteaya wonders how long it will be before we see a commit fly by with the title "What are you looking at?" | 23:25 | |
| dstufft | that really shouldn't be allowed, but it is currently | 23:25 |
| *** tongli has quit IRC | 23:25 | |
| clarkb | it is an odd switch they did here, I wonder why they felt they should do it that way rather than handing over contrl of the existing package | 23:26 |
| clarkb | thats pretty simple on pypi iirc, you just add a new owner user | 23:26 |
| clarkb | then they remove the old owner | 23:26 |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/304249 | 23:26 |
| *** jamesmcarthur has joined #openstack-infra | 23:26 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:27 | |
| *** tongli_ has quit IRC | 23:27 | |
| dstufft | clarkb: yea it's not super hard, on the other hand PyPI has a UI straight from the depths of the 90's, so I don't blame anyone for getting confused about what is possible with it :] | 23:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Consume config values for NAT destination https://review.openstack.org/300592 | 23:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Search subnets for gateway_ip to discover NAT dest https://review.openstack.org/300598 | 23:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: WIP Start stamping the has_service debug messages https://review.openstack.org/303853 | 23:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Refactor guts of _find_interesting_networks https://review.openstack.org/301738 | 23:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Refactor the port search logic https://review.openstack.org/303852 | 23:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add nat_destination filter to floating IP creation https://review.openstack.org/302104 | 23:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Allow passing nat_destination to get_active_server https://review.openstack.org/303851 | 23:29 |
| *** baoli has quit IRC | 23:30 | |
| *** baoli has joined #openstack-infra | 23:31 | |
| *** sdague has quit IRC | 23:31 | |
| *** ddutta has joined #openstack-infra | 23:32 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: copy_puppet_logs: copy audit.log if it does exist https://review.openstack.org/304334 | 23:32 |
| *** markvoelker has joined #openstack-infra | 23:34 | |
| *** tosky has quit IRC | 23:35 | |
| *** kzaitsev_mb has quit IRC | 23:36 | |
| *** erlon has quit IRC | 23:37 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:38 | |
| *** xarses has quit IRC | 23:38 | |
| *** markvoelker has quit IRC | 23:39 | |
| *** jamesmcarthur has quit IRC | 23:40 | |
| *** keedya has quit IRC | 23:41 | |
| *** markusz has joined #openstack-infra | 23:41 | |
| *** mylu has quit IRC | 23:42 | |
| *** Jeffrey4l__ has quit IRC | 23:42 | |
| *** maishsk has quit IRC | 23:43 | |
| *** Qiming has joined #openstack-infra | 23:44 | |
| *** mylu has joined #openstack-infra | 23:44 | |
| *** kzaitsev_mb has quit IRC | 23:45 | |
| clarkb | did anyone #success the gerrit upgrade? | 23:46 |
| clarkb | wiki says now | 23:46 |
| clarkb | #success infra team redeployed Gerrit on a new larger server. Should serve reviews with fewer 500 errors. | 23:46 |
| openstackstatus | clarkb: Added success to Success page | 23:46 |
| *** tongli has joined #openstack-infra | 23:47 | |
| jlvillal | tonyb: FYI: I removed the WIP comment for: https://review.openstack.org/241522 Use a boostrap grid to list all the meetings | 23:48 |
| *** mylu has quit IRC | 23:48 | |
| *** mylu has joined #openstack-infra | 23:49 | |
| *** akshai has quit IRC | 23:49 | |
| *** mixos has joined #openstack-infra | 23:49 | |
| *** baoli has quit IRC | 23:50 | |
| JayF | To a wiki admin; https://wiki.openstack.org/wiki/Main_Page has a dead link to a "How to use this wiki" page, which seems to only have been ever populated by spammers | 23:50 |
| *** baoli has joined #openstack-infra | 23:51 | |
| *** tongli has quit IRC | 23:51 | |
| fungi | JayF: good find. i'll clean that up | 23:52 |
| *** sdake has joined #openstack-infra | 23:52 | |
| * JayF trying hard and failing hard to find any nova documentation for virt driver writers | 23:52 | |
| openstackgerrit | Sean McGinnis proposed openstack/requirements: Update to os-brick 1.3.0 https://review.openstack.org/302786 | 23:53 |
| *** sdake_ has quit IRC | 23:53 | |
| mordred | JayF: I think the docs are "copy one of the existing drivers and make changes until it works" | 23:54 |
| JayF | mordred: that's what I was afraid of | 23:54 |
| *** angdraug has quit IRC | 23:54 | |
| fungi | looks like we had some "how to use this wiki" contents when we were still using moin http://old-wiki.openstack.org/HowToUseTheWiki | 23:55 |
| *** keedya has joined #openstack-infra | 23:56 | |
| clarkb | fungi: the joys of wiki migrations | 23:56 |
| clarkb | I am not getting anything done, I should stop starting at a computer screen | 23:56 |
| fungi | but yeah, apparently the page didn't exist (according to history) when spammers attempted to redirect it to a spam page back in february | 23:57 |
| *** baoli has quit IRC | 23:57 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!