| *** robcresswell has quit IRC | 00:04 | |
| *** ijw has joined #openstack-infra | 00:04 | |
| *** zhurong has quit IRC | 00:04 | |
| harlowja | damn, everyone gonna be on the TC | 00:07 |
|---|---|---|
| harlowja | lol | 00:07 |
| *** pahuang has quit IRC | 00:08 | |
| clarkb | pabelanger: out of curiousity do you all make it a thing to watch the debate too north of the border? | 00:08 |
| *** esberglu has joined #openstack-infra | 00:08 | |
| *** esberglu has quit IRC | 00:08 | |
| jeblair | afs is installed | 00:08 |
| *** amotoki has joined #openstack-infra | 00:08 | |
| pabelanger | clarkb: I'm actually following along pretty closely. Mostly because, we don't have anything like that in Canada | 00:09 |
| jeblair | fungi, clarkb, pabelanger: can you aprv https://review.openstack.org/375779 if it looks good to you? | 00:09 |
| pabelanger | our election window is like 58 days or something | 00:09 |
| *** pahuang has joined #openstack-infra | 00:09 | |
| mordred | we should have a much shorter window | 00:10 |
| clarkb | mordred: WRONG | 00:11 |
| clarkb | :P | 00:11 |
| mordred | clarkb: maybe SpamapS will build us a wall :) | 00:11 |
| *** kaisers2 has joined #openstack-infra | 00:13 | |
| *** kaisers3 has joined #openstack-infra | 00:13 | |
| fungi | jeblair: lgtm, though i will admit i didn't test the ruby example | 00:13 |
| *** kaisers1 has quit IRC | 00:13 | |
| jeblair | fungi: then it worked | 00:14 |
| fungi | harlowja: we're gonna need a bigger boat | 00:14 |
| harlowja | def | 00:14 |
| *** gildub has joined #openstack-infra | 00:14 | |
| openstackgerrit | Merged openstack-infra/system-config: Add docs on using SSL/TLS MQTT with firehose https://review.openstack.org/375779 | 00:15 |
| *** kaisers has quit IRC | 00:15 | |
| *** ijw has quit IRC | 00:16 | |
| *** tphummel has quit IRC | 00:16 | |
| anteaya | pabelanger: really? | 00:17 |
| anteaya | I've stopped reading the news because I can't get away from it | 00:17 |
| anteaya | I can't get any canadian news anymore | 00:18 |
| mtreinish | fungi: I have been testing the ruby example :) | 00:18 |
| anteaya | every headline is us politics | 00:18 |
| anteaya | has been since the spring | 00:18 |
| anteaya | pabelanger: I have to say I'm impressed | 00:18 |
| *** ijw has joined #openstack-infra | 00:18 | |
| anteaya | I can't manage it myself | 00:18 |
| mtreinish | fungi: I was using it to verify the logstash plugin call path wasn't broken at one point | 00:18 |
| *** baoli has joined #openstack-infra | 00:19 | |
| jeblair | gah https://review.openstack.org/379616 didn't land | 00:20 |
| *** lin_yang has joined #openstack-infra | 00:20 | |
| fungi | anteaya: one of the only ways i manage to survive living in this country is to just not watch any "news" programs (or network television drivel at all really) | 00:20 |
| anteaya | fungi: I like your cartoons approach | 00:20 |
| fungi | it's a level of reality even i can deal with | 00:20 |
| anteaya | and yeah at one point we had a measured approach to the news | 00:20 |
| anteaya | we seem to have lost that | 00:21 |
| anteaya | if is a good choice | 00:21 |
| anteaya | it is a good choice* | 00:21 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add a note to docs for websockets being disabled https://review.openstack.org/379536 | 00:21 |
| anteaya | the other night I held space for a friend that was having a difficult time | 00:21 |
| anteaya | I watched ok go videos | 00:21 |
| pabelanger | Ya, same. Most of my following is online, but I did tune in the the recent debate | 00:21 |
| anteaya | I laughed for 2 straight hours | 00:21 |
| anteaya | pabelanger: awesome | 00:22 |
| pabelanger | likely follow the next 2, along with VP | 00:22 |
| *** vsaienko has quit IRC | 00:22 | |
| anteaya | points for that | 00:22 |
| *** pvaneck has quit IRC | 00:22 | |
| *** yamahata has joined #openstack-infra | 00:22 | |
| *** rfolco_ has joined #openstack-infra | 00:23 | |
| *** mtanino has quit IRC | 00:23 | |
| *** ijw has quit IRC | 00:24 | |
| fungi | jeblair: is that our random post_failure still lingering? | 00:24 |
| fungi | looks like we didn't get some log files, including the ansible ones | 00:24 |
| jeblair | fungi: yeah, it was a post failure, but i didn't see when it ran | 00:24 |
| anteaya | I think this became one of my favourites: https://www.youtube.com/watch?v=MejbOFk7H6c#t=0m2s | 00:25 |
| *** baoli has quit IRC | 00:25 | |
| fungi | jeblair: looks like it was well after the log volume cleanup began anyway, so wouldn't have been from that | 00:25 |
| pabelanger | had another record today, 2.212 kilojobs/hr :) | 00:26 |
| *** markvoelker has joined #openstack-infra | 00:28 | |
| openstackgerrit | Merged openstack-infra/system-config: Add service name to afs zuul-launcher sites https://review.openstack.org/379616 | 00:28 |
| *** amotoki_ has joined #openstack-infra | 00:30 | |
| *** amitgandhinz has joined #openstack-infra | 00:32 | |
| *** annegent_ has quit IRC | 00:32 | |
| *** amotoki has quit IRC | 00:34 | |
| *** tonytan4ever has joined #openstack-infra | 00:35 | |
| *** xarses has joined #openstack-infra | 00:36 | |
| *** yamahata has quit IRC | 00:37 | |
| mtreinish | hmm, we're only getting about 17-20 msgs/sec on firehose (at least to my desktop) using http://paste.openstack.org/show/583569/ | 00:39 |
| mtreinish | unless I screwed something up measuring it | 00:39 |
| mtreinish | fwiw, that script should work from anywhere | 00:40 |
| pabelanger | clarkb: cloudnull: nodepool-builder restarted to pickup removal of template-name | 00:40 |
| pabelanger | and the removal of osic-cloud1 provider | 00:40 |
| *** tonytan4ever has quit IRC | 00:41 | |
| clarkb | cool | 00:41 |
| pabelanger | also tested the latest glean release on infracloud-vanilla both with xenial / trusty and things look real good | 00:42 |
| pabelanger | expecting a big drop in launch failures for tomorrow | 00:42 |
| pabelanger | once all the images have been uploaded | 00:42 |
| *** jamesdenton has quit IRC | 00:44 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Pull get-pip.py from jenkins cache https://review.openstack.org/378999 | 00:50 |
| *** john-davidge has joined #openstack-infra | 00:50 | |
| anteaya | clarkb: should it try pypa if it isn't in cache? | 00:51 |
| *** yamamoto has joined #openstack-infra | 00:52 | |
| clarkb | anteaya: ya thats why there is the test in that change | 00:53 |
| clarkb | basically only set it if the file exists | 00:53 |
| anteaya | okay thank you | 00:54 |
| *** john-davidge has quit IRC | 00:55 | |
| cloudnull | pabelanger: cool | 00:55 |
| *** nadya has joined #openstack-infra | 00:57 | |
| anteaya | goodness zuul is quiet right now | 00:57 |
| *** lezbar has quit IRC | 00:58 | |
| *** lezbar has joined #openstack-infra | 00:59 | |
| *** gyee has quit IRC | 00:59 | |
| *** gouthamr has quit IRC | 01:01 | |
| *** nadya has quit IRC | 01:02 | |
| *** thcipriani is now known as thcipriani|afk | 01:03 | |
| *** Julien-zte has joined #openstack-infra | 01:04 | |
| *** ddieterly has joined #openstack-infra | 01:06 | |
| *** amitgandhinz has quit IRC | 01:07 | |
| *** thorst has joined #openstack-infra | 01:08 | |
| *** thorst has quit IRC | 01:10 | |
| *** baoli has joined #openstack-infra | 01:10 | |
| *** aeng has quit IRC | 01:15 | |
| *** zhurong has joined #openstack-infra | 01:20 | |
| *** gouthamr has joined #openstack-infra | 01:21 | |
| openstackgerrit | Merged openstack-infra/system-config: Add a note to docs for websockets being disabled https://review.openstack.org/379536 | 01:23 |
| *** tiswanso has quit IRC | 01:23 | |
| *** tiswanso has joined #openstack-infra | 01:23 | |
| *** chem has quit IRC | 01:26 | |
| jeblair | oh right, we need to restart zuul launchers again after a zuul.conf change like that | 01:26 |
| jeblair | i will hard-restart them again | 01:28 |
| *** Julien-zte has quit IRC | 01:29 | |
| *** Julien-z_ has joined #openstack-infra | 01:29 | |
| fungi | it's a ghost town in there anyway | 01:34 |
| fungi | i doubt anyone will notice | 01:34 |
| jeblair | done | 01:34 |
| anteaya | yes my comment was more along the line of ghost town | 01:35 |
| anteaya | though if it sparked some maintenance that needed addressing, my happy too | 01:35 |
| anteaya | thank you | 01:35 |
| jeblair | zuul is running 200 jobs. i love that we consider that a ghost town. | 01:35 |
| anteaya | s/my/I'm | 01:35 |
| anteaya | ha ha ha | 01:36 |
| anteaya | I can't even typo correctly anymore | 01:36 |
| *** yanyanhu_ has joined #openstack-infra | 01:36 | |
| anteaya | I started listening to a movie soundtrack on youtube, found my way to patrick watson and ended up listening to https://en.wikipedia.org/wiki/Gregory_Alan_Isakov | 01:39 |
| anteaya | it is so beautiful | 01:39 |
| *** dfflanders has joined #openstack-infra | 01:40 | |
| openstackgerrit | Merged openstack-infra/system-config: Remove unused log_archive_maintenance.sh script https://review.openstack.org/379559 | 01:40 |
| *** ddieterly has quit IRC | 01:41 | |
| anteaya | the logstash logs change seems to be holding steady: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=422&rra_id=all | 01:42 |
| jeblair | fungi, pabelanger, mordred: we has afs docs: http://files.openstack.org/docs/infra/system-config/ | 01:43 |
| anteaya | yay! | 01:44 |
| *** salv-orlando has joined #openstack-infra | 01:44 | |
| *** bswartz has quit IRC | 01:44 | |
| jeblair | here's the log, etc: http://logs.openstack.org/51/511a4cf86eec7c88f1409b42c785f45b9ef33d8e/post/system-config-infra-docs/ea63b06/_zuul_ansible/ | 01:44 |
| *** salv-orl_ has quit IRC | 01:46 | |
| anteaya | in the ansible log if it ends with TASK [copy] then it has reached the end, yes? | 01:46 |
| fungi | jeblair: #success! | 01:47 |
| anteaya | I have never seen any output for the TASK [copy] task | 01:47 |
| *** thorst has joined #openstack-infra | 01:47 | |
| *** thorst has quit IRC | 01:49 | |
| *** coolsvap has joined #openstack-infra | 01:54 | |
| *** lezbar has quit IRC | 01:55 | |
| *** lezbar has joined #openstack-infra | 01:55 | |
| *** tuanluong has joined #openstack-infra | 01:55 | |
| anteaya | the logstash job queue has taken a leap up | 01:56 |
| *** Sukhdev has quit IRC | 01:58 | |
| *** tuanluong has quit IRC | 01:59 | |
| *** tonytan4ever has joined #openstack-infra | 02:00 | |
| *** amitgandhinz has joined #openstack-infra | 02:02 | |
| *** jamesdenton has joined #openstack-infra | 02:04 | |
| *** jamesden_ has joined #openstack-infra | 02:08 | |
| pabelanger | jeblair: great! I'll finish up the vhosts first thing tomorrow | 02:08 |
| *** jamesdenton has quit IRC | 02:09 | |
| *** thorst has joined #openstack-infra | 02:10 | |
| *** thorst has quit IRC | 02:10 | |
| *** lezbar has quit IRC | 02:11 | |
| *** caowei has joined #openstack-infra | 02:11 | |
| *** lezbar has joined #openstack-infra | 02:11 | |
| *** rhallisey has quit IRC | 02:12 | |
| *** bswartz has joined #openstack-infra | 02:15 | |
| *** Apoorva_ has quit IRC | 02:15 | |
| *** EricGonczer_ has quit IRC | 02:15 | |
| *** katyafervent_awa has quit IRC | 02:17 | |
| *** katyafervent_awa has joined #openstack-infra | 02:21 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Delete ping test environment in periodic jobs https://review.openstack.org/346134 | 02:21 |
| *** gouthamr has quit IRC | 02:22 | |
| sridhar_ram | infra-team: we intermittently hitting cirros image download failure in out gate test .. http://logs.openstack.org/68/375468/3/check/gate-tacker-dsvm-functional/fe23cbd/logs/screen-h-eng.txt.gz?level=ERROR .. any clue what might cause this ? | 02:24 |
| *** baoli has quit IRC | 02:25 | |
| *** baoli has joined #openstack-infra | 02:25 | |
| *** lezbar has quit IRC | 02:28 | |
| *** lezbar has joined #openstack-infra | 02:28 | |
| *** rfolco_ has quit IRC | 02:30 | |
| *** rfolco_ has joined #openstack-infra | 02:31 | |
| clarkb | you shpuldnt be downloaded that from cirros directly it is already cached on the job host in devstack's files dir | 02:32 |
| *** sflanigan_ has quit IRC | 02:33 | |
| *** sflanigan_ has joined #openstack-infra | 02:33 | |
| *** amitgandhinz has quit IRC | 02:36 | |
| sridhar_ram | clarkb: it is getting pulled as part of a tacker scenario test that exercises a heat feature that allows an glance image to be downloaded off an URL. What baffles is the intermittent nature of this | 02:36 |
| clarkb | I would put an http server in front of the local file if you need to test that specifically. this is why we cache these things | 02:40 |
| *** lin_yang has quit IRC | 02:40 | |
| clarkb | internets are never reliable when you need them to be | 02:40 |
| *** tonytan4ever has quit IRC | 02:41 | |
| *** rfolco_ has quit IRC | 02:43 | |
| fungi | presumably apache is already running, would just need a few lines in its config | 02:44 |
| *** yuanying has quit IRC | 02:47 | |
| *** aeng has joined #openstack-infra | 02:55 | |
| *** yuanying has joined #openstack-infra | 02:55 | |
| *** binbincong has joined #openstack-infra | 02:59 | |
| *** yamahata has joined #openstack-infra | 02:59 | |
| *** baoli has quit IRC | 03:00 | |
| *** david-lyle has quit IRC | 03:04 | |
| *** tongli_ has joined #openstack-infra | 03:08 | |
| *** Guest92_ has joined #openstack-infra | 03:08 | |
| *** tongli has quit IRC | 03:10 | |
| *** thorst has joined #openstack-infra | 03:11 | |
| *** Guest92__ has joined #openstack-infra | 03:11 | |
| *** Guest92 has quit IRC | 03:12 | |
| *** Guest92_ has quit IRC | 03:12 | |
| *** Julien-z_ has quit IRC | 03:15 | |
| *** Mmike has quit IRC | 03:15 | |
| *** Mmike has joined #openstack-infra | 03:15 | |
| *** amotoki_ has quit IRC | 03:15 | |
| *** Guest92__ has quit IRC | 03:16 | |
| *** priteau has joined #openstack-infra | 03:16 | |
| *** thorst has quit IRC | 03:20 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Expand the valid characters in chair names https://review.openstack.org/379912 | 03:20 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: check_chair: Actually fail if there is a problem. https://review.openstack.org/379913 | 03:20 |
| *** priteau has quit IRC | 03:21 | |
| *** sflanigan_ is now known as sflanigan_afk | 03:23 | |
| *** bswartz has quit IRC | 03:27 | |
| *** dizquierdo has joined #openstack-infra | 03:27 | |
| *** amitgandhinz has joined #openstack-infra | 03:33 | |
| *** vikrant has joined #openstack-infra | 03:34 | |
| *** yuanying has quit IRC | 03:38 | |
| *** sflanigan_afk is now known as sflanigan | 03:40 | |
| *** ddieterly has joined #openstack-infra | 03:44 | |
| *** ddieterly has quit IRC | 03:45 | |
| *** jamesden_ has quit IRC | 03:47 | |
| *** yuanying has joined #openstack-infra | 03:49 | |
| *** tiswanso has quit IRC | 03:51 | |
| *** dizquierdo has quit IRC | 03:52 | |
| *** amotoki has joined #openstack-infra | 03:53 | |
| *** Rockyg has quit IRC | 03:55 | |
| *** amotoki has quit IRC | 03:57 | |
| *** tonytan4ever has joined #openstack-infra | 03:57 | |
| *** tuannguyen has joined #openstack-infra | 03:58 | |
| *** Julien-zte has joined #openstack-infra | 03:58 | |
| *** links has joined #openstack-infra | 03:59 | |
| *** markvoelker has quit IRC | 04:01 | |
| *** Sukhdev has joined #openstack-infra | 04:02 | |
| *** tonytan4ever has quit IRC | 04:02 | |
| *** tuannguyen has quit IRC | 04:03 | |
| *** amitgandhinz has quit IRC | 04:07 | |
| *** amotoki has joined #openstack-infra | 04:10 | |
| *** thorst has joined #openstack-infra | 04:18 | |
| sridhar_ram | clarkb: fungi: I'll explore that option, to serve locally cached image off http... however, this test was reliably working for many months now. | 04:19 |
| *** lezbar has quit IRC | 04:20 | |
| *** lezbar has joined #openstack-infra | 04:20 | |
| *** thorst has quit IRC | 04:25 | |
| *** ramishra has quit IRC | 04:26 | |
| *** ramishra has joined #openstack-infra | 04:28 | |
| *** lezbar has quit IRC | 04:37 | |
| *** lezbar has joined #openstack-infra | 04:37 | |
| *** sdake_ has joined #openstack-infra | 04:42 | |
| *** tnovacik has joined #openstack-infra | 04:46 | |
| *** sdake has quit IRC | 04:46 | |
| *** ddieterly has joined #openstack-infra | 04:47 | |
| *** ddieterly has quit IRC | 04:52 | |
| *** pgadiya has joined #openstack-infra | 04:52 | |
| *** tongli has joined #openstack-infra | 04:52 | |
| *** adriant has quit IRC | 04:53 | |
| *** tongli_ has quit IRC | 04:55 | |
| *** psachin has joined #openstack-infra | 04:57 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Expand the valid characters in chair names https://review.openstack.org/379912 | 04:57 |
| *** tonytan4ever has joined #openstack-infra | 04:58 | |
| *** nadya has joined #openstack-infra | 04:58 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: check_chair: Actually fail if there is a problem. https://review.openstack.org/379913 | 04:59 |
| *** markvoelker has joined #openstack-infra | 05:02 | |
| *** nadya has quit IRC | 05:03 | |
| *** tonytan4ever has quit IRC | 05:03 | |
| *** amitgandhinz has joined #openstack-infra | 05:03 | |
| openstackgerrit | Merged openstack-infra/release-tools: make the announce script verbose https://review.openstack.org/379611 | 05:06 |
| *** markvoelker has quit IRC | 05:07 | |
| AJaeger | jeblair: great! | 05:11 |
| *** sdake_ has quit IRC | 05:19 | |
| *** thorst has joined #openstack-infra | 05:22 | |
| *** kaisers has joined #openstack-infra | 05:29 | |
| *** thorst has quit IRC | 05:30 | |
| *** amitgandhinz has quit IRC | 05:38 | |
| *** jaosorior has joined #openstack-infra | 05:40 | |
| *** nadya has joined #openstack-infra | 05:46 | |
| *** nadya has quit IRC | 05:47 | |
| *** lezbar has quit IRC | 05:52 | |
| *** flepied has quit IRC | 05:52 | |
| *** lezbar has joined #openstack-infra | 05:52 | |
| *** andreas_s has joined #openstack-infra | 05:53 | |
| *** sandanar has joined #openstack-infra | 05:54 | |
| *** salv-orlando has quit IRC | 05:56 | |
| *** sdake has joined #openstack-infra | 05:59 | |
| *** tuannguyen has joined #openstack-infra | 06:00 | |
| *** amotoki has quit IRC | 06:01 | |
| *** tuannguyen has quit IRC | 06:04 | |
| *** rcernin has joined #openstack-infra | 06:07 | |
| *** hichihara has joined #openstack-infra | 06:19 | |
| *** pradiprwt has left #openstack-infra | 06:20 | |
| *** salv-orlando has joined #openstack-infra | 06:20 | |
| *** vsaienk0 has joined #openstack-infra | 06:26 | |
| vsaienk0 | Infra team please review: https://review.openstack.org/#/c/379952/. This revert for commit that break setting fixed_network_name tempest option. As result ironic-grenade job become broken | 06:28 |
| *** thorst has joined #openstack-infra | 06:28 | |
| *** aeng has quit IRC | 06:29 | |
| *** tonytan4ever has joined #openstack-infra | 06:29 | |
| *** Ravikiran_K has joined #openstack-infra | 06:29 | |
| *** vinaypotluri has quit IRC | 06:32 | |
| *** amitgandhinz has joined #openstack-infra | 06:34 | |
| *** tonytan4ever has quit IRC | 06:34 | |
| *** sdake has quit IRC | 06:35 | |
| *** rlu has joined #openstack-infra | 06:35 | |
| *** sdake has joined #openstack-infra | 06:35 | |
| *** thorst has quit IRC | 06:35 | |
| AJaeger | vsaienk0: which repository is that for? | 06:36 |
| *** Julien-zte has quit IRC | 06:38 | |
| *** pcaruana has joined #openstack-infra | 06:39 | |
| yolanda | good morning | 06:40 |
| AJaeger | morning yolanda ! | 06:40 |
| *** jaosorior has quit IRC | 06:40 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add network config for puppet-infracloud on hiera https://review.openstack.org/371607 | 06:44 |
| *** nadya has joined #openstack-infra | 06:45 | |
| *** nadya has quit IRC | 06:48 | |
| *** nherciu_ has joined #openstack-infra | 06:49 | |
| *** rlu has quit IRC | 06:50 | |
| *** Julien-zte has joined #openstack-infra | 06:51 | |
| *** amotoki has joined #openstack-infra | 06:51 | |
| *** apetrich has quit IRC | 06:52 | |
| *** watersoul has quit IRC | 06:52 | |
| *** apetrich has joined #openstack-infra | 06:53 | |
| *** flepied has joined #openstack-infra | 06:53 | |
| *** Julien-zte has quit IRC | 06:53 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add network config for puppet-infracloud on hiera https://review.openstack.org/371607 | 06:54 |
| *** Julien-zte has joined #openstack-infra | 06:54 | |
| *** rlu has joined #openstack-infra | 06:59 | |
| *** Sukhdev has quit IRC | 07:02 | |
| *** markvoelker has joined #openstack-infra | 07:03 | |
| *** matrohon has joined #openstack-infra | 07:03 | |
| *** amotoki has quit IRC | 07:04 | |
| *** rossella_ has joined #openstack-infra | 07:06 | |
| *** markvoelker has quit IRC | 07:07 | |
| *** amitgandhinz has quit IRC | 07:08 | |
| *** Ravikiran_K has quit IRC | 07:11 | |
| *** tkelsey has joined #openstack-infra | 07:11 | |
| *** e0ne has joined #openstack-infra | 07:13 | |
| *** ihrachys has joined #openstack-infra | 07:16 | |
| *** pahuang_ has joined #openstack-infra | 07:18 | |
| *** pahuang has quit IRC | 07:18 | |
| *** esikachev has joined #openstack-infra | 07:19 | |
| openstackgerrit | Merged openstack-dev/cookiecutter: Enable Python DeprecationWarning in test environments https://review.openstack.org/353197 | 07:20 |
| *** hashar has joined #openstack-infra | 07:22 | |
| *** yanyanhu_ has quit IRC | 07:22 | |
| *** jamielennox is now known as jamielennox|away | 07:23 | |
| *** drifterza has joined #openstack-infra | 07:24 | |
| *** martinkopec has joined #openstack-infra | 07:24 | |
| *** jpena|off is now known as jpena | 07:24 | |
| *** lezbar has quit IRC | 07:26 | |
| *** lezbar has joined #openstack-infra | 07:27 | |
| *** rwsu has quit IRC | 07:27 | |
| *** panda|zZ is now known as panda | 07:27 | |
| *** e0ne has quit IRC | 07:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Add code coverage job for kuryr and kuryr-libnetwork https://review.openstack.org/379566 | 07:33 |
| *** sdake_ has joined #openstack-infra | 07:34 | |
| *** thorst has joined #openstack-infra | 07:34 | |
| *** pahuang_ has quit IRC | 07:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Run gate-tempest-dsvm-full-devstack-plugin-nfs-nv on xenial https://review.openstack.org/379770 | 07:36 |
| openstackgerrit | Merged openstack-infra/project-config: Mark murano-pkg-check ready for translations https://review.openstack.org/366420 | 07:36 |
| *** sdake has quit IRC | 07:37 | |
| *** gildub has quit IRC | 07:37 | |
| *** jlanoux has joined #openstack-infra | 07:39 | |
| *** thorst has quit IRC | 07:40 | |
| *** kaisers has quit IRC | 07:41 | |
| *** drifterza has quit IRC | 07:41 | |
| *** drifterza has joined #openstack-infra | 07:42 | |
| *** ssancheztrujillo has joined #openstack-infra | 07:43 | |
| *** salv-orl_ has joined #openstack-infra | 07:44 | |
| *** lezbar has quit IRC | 07:46 | |
| *** amotoki has joined #openstack-infra | 07:46 | |
| *** kaisers has joined #openstack-infra | 07:46 | |
| *** lezbar has joined #openstack-infra | 07:46 | |
| *** salv-orlando has quit IRC | 07:46 | |
| openstackgerrit | Merged openstack-infra/project-config: [Manila] Add lost job to master and newton branches pipelines https://review.openstack.org/379282 | 07:49 |
| openstackgerrit | Merged openstack-infra/project-config: Added .git suffix for github.com repos https://review.openstack.org/379272 | 07:49 |
| openstackgerrit | Merged openstack-infra/project-config: Add access to set tags for k8s-docker-suite-app-murano project https://review.openstack.org/379362 | 07:50 |
| *** jpich has joined #openstack-infra | 07:53 | |
| *** lezbar has quit IRC | 07:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Make tacker python3 jobs voting https://review.openstack.org/370388 | 07:54 |
| *** ihrachys has quit IRC | 07:54 | |
| *** rwsu has joined #openstack-infra | 07:54 | |
| *** lezbar has joined #openstack-infra | 07:55 | |
| *** yamahata has quit IRC | 07:56 | |
| *** Ravikiran_K has joined #openstack-infra | 07:56 | |
| *** jordanP has joined #openstack-infra | 07:58 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** watersoul has joined #openstack-infra | 08:00 | |
| *** tuannguyen has joined #openstack-infra | 08:01 | |
| *** andrewbogott has quit IRC | 08:02 | |
| *** andrewbogott has joined #openstack-infra | 08:02 | |
| openstackgerrit | Merged openstack-infra/project-config: gnocchi: test upgrade from newton branch https://review.openstack.org/378305 | 08:03 |
| AJaeger | jamespage: could you review https://review.openstack.org/378534 and the governance change? There seems to be some confusion about OpenStack project teams. They apply as separate team - but their change puts it under OpenStack charms. | 08:04 |
| jamespage | AJaeger, morning | 08:05 |
| *** amitgandhinz has joined #openstack-infra | 08:05 | |
| AJaeger | morning, jamespage | 08:05 |
| jamespage | AJaeger, I've been chatting with bilal about this on irc | 08:05 |
| *** lezbar has quit IRC | 08:05 | |
| jamespage | AJaeger, I don't have it quite set in my mind about team structure yet | 08:05 |
| *** lezbar has joined #openstack-infra | 08:06 | |
| jamespage | AJaeger, I anticipate that we'll end up with alot of SDN/Storage vendor integration like charms wanting to take this route | 08:06 |
| *** tuannguyen has quit IRC | 08:06 | |
| jamespage | and I want to enable the current pool of contributors to those charms to keep control/ownership | 08:06 |
| AJaeger | the way it's currently, I can only -1. The changes are fine on a technical level and with a different commit message - and your (as OpenStack Charms PTL) blessing, it could merge. | 08:07 |
| jamespage | AJaeger, I guess I'm aiming for something similar to what neutron has done with neutron core, and then stadium | 08:07 |
| AJaeger | jamespage: yeah, those are interesting challenges. Neutron had those all initially in the big stadium but pushed all propietary HW enablement out. Fuel has some as plugin but outside the project. | 08:08 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add deb-zaqar-ui https://review.openstack.org/379989 | 08:09 |
| jamespage | AJaeger, as PTL, I need charms that are part of OpenStack Charms to keep aligned to the 3 month release cycle we have - I think that's part of the inclusion policy (although that's currently undocumented - I'll get that fixed) | 08:09 |
| AJaeger | jamespage: I would say: If the underlying project is part of OpenSTack, like networking-midonet, include it - if the driver is outside - like networking-plumgrid -, leave them out. | 08:09 |
| *** Jeffrey4l_ has joined #openstack-infra | 08:09 | |
| AJaeger | They can still use the OpenStack infrastructure - like networking-plumgrid does. | 08:09 |
| jamespage | AJaeger, OK - I'll chat with bilal and comment on the review | 08:10 |
| *** Jeffrey4l has quit IRC | 08:10 | |
| AJaeger | jamespage: you could also ask them to just add the repos and have the discussion later. | 08:10 |
| jamespage | will do | 08:10 |
| AJaeger | jamespage: lot's of options ;) | 08:10 |
| jamespage | AJaeger, agreed :-) | 08:10 |
| *** dtantsur has joined #openstack-infra | 08:11 | |
| AJaeger | I suggest to talk with armax from Neutron and a few others facing similiar challenges. Ansible ( odyssey4me ), Puppet ( EmilienM ), etc are facing the same challenge. | 08:12 |
| *** Ravikiran_K has quit IRC | 08:13 | |
| *** hichihara has quit IRC | 08:14 | |
| *** shardy has joined #openstack-infra | 08:16 | |
| jamespage | AJaeger, I'll hijack their time when things get a little quieter... | 08:17 |
| *** Ravikiran_K has joined #openstack-infra | 08:17 | |
| *** binbincong has quit IRC | 08:19 | |
| *** priteau has joined #openstack-infra | 08:20 | |
| *** derekh has joined #openstack-infra | 08:20 | |
| *** sflanigan is now known as sflanigan_afk | 08:23 | |
| *** sdake_ has quit IRC | 08:23 | |
| AJaeger | jamespage: ;) | 08:26 |
| *** vsaienko has joined #openstack-infra | 08:26 | |
| *** amotoki_ has joined #openstack-infra | 08:27 | |
| *** amotoki has quit IRC | 08:29 | |
| openstackgerrit | Michal Dulko proposed openstack-infra/project-config: Make Cinder multinode grenade job voting https://review.openstack.org/379346 | 08:31 |
| *** shu-mutou-AWAY has quit IRC | 08:33 | |
| *** berendt has joined #openstack-infra | 08:36 | |
| *** thorst has joined #openstack-infra | 08:37 | |
| *** ihrachys has joined #openstack-infra | 08:38 | |
| *** amitgandhinz has quit IRC | 08:39 | |
| *** zhurong has quit IRC | 08:39 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/380013 | 08:41 |
| openstackgerrit | Merged openstack-infra/system-config: Create Member role on infracloud clouds https://review.openstack.org/378976 | 08:41 |
| *** Naeil has joined #openstack-infra | 08:44 | |
| *** mhickey has joined #openstack-infra | 08:45 | |
| *** thorst has quit IRC | 08:45 | |
| *** john-davidge has joined #openstack-infra | 08:45 | |
| *** john-davidge has quit IRC | 08:46 | |
| *** john-davidge has joined #openstack-infra | 08:46 | |
| *** e0ne has joined #openstack-infra | 08:48 | |
| *** vsaienko has quit IRC | 08:52 | |
| *** dtardivel has joined #openstack-infra | 08:54 | |
| *** vsaienko has joined #openstack-infra | 08:55 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/devstack-gate: Set SUBNETPOOL_PREFIX_V4 to match FIXED_RANGE https://review.openstack.org/379521 | 08:58 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/380013 | 08:59 |
| *** _degorenko is now known as degorenko | 09:01 | |
| *** vsaienko has quit IRC | 09:01 | |
| *** markvoelker has joined #openstack-infra | 09:04 | |
| *** chem has joined #openstack-infra | 09:05 | |
| *** salv-orl_ has quit IRC | 09:05 | |
| *** dragonmaster has joined #openstack-infra | 09:06 | |
| *** nadya has joined #openstack-infra | 09:08 | |
| *** markvoelker has quit IRC | 09:08 | |
| openstackgerrit | Chris Dent proposed openstack-infra/devstack-gate: DNM Test using the optional placement db in nova https://review.openstack.org/379797 | 09:08 |
| *** tosky has joined #openstack-infra | 09:11 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add roles assignment for openstackci and openstackzuul user/projects https://review.openstack.org/380041 | 09:13 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add network config for puppet-infracloud on hiera https://review.openstack.org/371607 | 09:16 |
| openstackgerrit | Ondřej Nový proposed openstack-infra/project-config: Set track-upstream for Swift related Debian packaging projects https://review.openstack.org/379400 | 09:20 |
| *** rossella_ has quit IRC | 09:21 | |
| openstackgerrit | Merged openstack-infra/system-config: Add roles assignment for openstackci and openstackzuul user/projects https://review.openstack.org/380041 | 09:25 |
| dulek | AJaeger: Hi! Not all voting cinder jobs are explicitly listed in gate queue, what am I missing in regards to your "voting jobs should be in both check and gate queue" comment? | 09:25 |
| *** robcresswell has joined #openstack-infra | 09:25 | |
| *** rlu has quit IRC | 09:25 | |
| *** joehuang has joined #openstack-infra | 09:25 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Rename infracloud-roles to admin-roles https://review.openstack.org/380051 | 09:28 |
| joehuang | would someone help to add initial members to trio2o project core and release group? The initial member list are: joehuang@huawei.com , shinobu@linux.com, shinobu.kj@gmail.com, luckyvega.g@gmail.com | 09:28 |
| joehuang | also send mail: https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg93364.html | 09:28 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Switch remaining ironic-inspector tempest jobs to Xenial https://review.openstack.org/380052 | 09:29 |
| dtantsur | AJaeger, next step in the Xenial story ^^^ | 09:29 |
| rcarrillocruz | joehuang: i add you as bootstrap user, you can then add whoever you see fit | 09:29 |
| *** john-davidge has quit IRC | 09:29 | |
| *** rlu has joined #openstack-infra | 09:29 | |
| rcarrillocruz | joehuang: you are now a member of both groups, you should be able to add the other folks | 09:30 |
| *** rossella_ has joined #openstack-infra | 09:30 | |
| *** salv-orlando has joined #openstack-infra | 09:31 | |
| *** ralonsoh has joined #openstack-infra | 09:33 | |
| *** Julien-zte has quit IRC | 09:33 | |
| *** mkarpin has joined #openstack-infra | 09:33 | |
| *** amitgandhinz has joined #openstack-infra | 09:35 | |
| mkarpin | Hello, all, have a question: I have third party CI system with Jenkins Nodepool Zuul, and have such annoying behavior - often Jenkins schedules builds on slaves which already have finished jobs and will be soon removed, and thus this new scheduled build will definitely fail, so the question did anyone have such behavior? and how it can be fixed? | 09:37 |
| *** ramishra has quit IRC | 09:38 | |
| *** Naeil has quit IRC | 09:39 | |
| *** ramishra has joined #openstack-infra | 09:39 | |
| *** vsaienko has joined #openstack-infra | 09:42 | |
| openstackgerrit | Merged openstack-infra/system-config: Rename infracloud-roles to admin-roles https://review.openstack.org/380051 | 09:42 |
| *** thorst has joined #openstack-infra | 09:42 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Fix openstackci/openstackzuul user roles profiles https://review.openstack.org/380059 | 09:43 |
| openstackgerrit | Bilal Baqar proposed openstack-infra/project-config: Add PLUMgrid Juju Charms https://review.openstack.org/378534 | 09:44 |
| *** joehuang has quit IRC | 09:46 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Declare openstackci-user-roles and openstackzuul-user-roles on Infracloud clouds https://review.openstack.org/380064 | 09:47 |
| *** thorst has quit IRC | 09:50 | |
| *** vsaienko has quit IRC | 09:50 | |
| *** yolanda has quit IRC | 09:54 | |
| *** vsaienko has joined #openstack-infra | 09:54 | |
| *** chem has quit IRC | 09:55 | |
| *** chem has joined #openstack-infra | 09:55 | |
| *** yolanda has joined #openstack-infra | 09:55 | |
| openstackgerrit | Merged openstack-infra/system-config: Fix openstackci/openstackzuul user roles profiles https://review.openstack.org/380059 | 09:55 |
| *** electrofelix has joined #openstack-infra | 09:55 | |
| openstackgerrit | Merged openstack-infra/system-config: Declare openstackci-user-roles and openstackzuul-user-roles on Infracloud clouds https://review.openstack.org/380064 | 09:59 |
| AJaeger | dulek: which change are you talking about? | 09:59 |
| AJaeger | dulek, your change looks fine. Let's solve one problem at a time ;) | 10:01 |
| *** vsaienko has quit IRC | 10:01 | |
| *** tuannguyen has joined #openstack-infra | 10:03 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Move declaration of openstackci/openstackzuul user roles to admin clouds https://review.openstack.org/380084 | 10:06 |
| *** tuannguyen has quit IRC | 10:08 | |
| *** amitgandhinz has quit IRC | 10:10 | |
| AJaeger | rcarrillocruz: could you review https://review.openstack.org/379631 and https://review.openstack.org/377968 , please? | 10:10 |
| *** Naeil has joined #openstack-infra | 10:11 | |
| rcarrillocruz | done | 10:11 |
| *** rossella_ has quit IRC | 10:12 | |
| AJaeger | thanks, rcarrillocruz | 10:15 |
| *** rtheis has joined #openstack-infra | 10:17 | |
| *** yamamoto has quit IRC | 10:19 | |
| *** john-davidge has joined #openstack-infra | 10:22 | |
| *** rossella_ has joined #openstack-infra | 10:27 | |
| *** tonytan4ever has joined #openstack-infra | 10:31 | |
| *** tonytan4ever has quit IRC | 10:35 | |
| AJaeger | mmh, openstackgerrit is not reporting anything anymore. | 10:40 |
| *** salv-orl_ has joined #openstack-infra | 10:41 | |
| *** salv-orlando has quit IRC | 10:41 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Get rid of ipv4- prefix when looking for sys-interfaces https://review.openstack.org/380142 | 10:42 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Update gerritbot branches monitored for OpenStack-Ansible https://review.openstack.org/380150 | 10:47 |
| *** thorst has joined #openstack-infra | 10:48 | |
| *** gildub has joined #openstack-infra | 10:50 | |
| odyssey4me | yolanda AJaeger I'd appreciate a quick review of https://review.openstack.org/380150 | 10:52 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 10:52 |
| *** denisra_ has quit IRC | 10:53 | |
| openstackgerrit | Merged openstack-infra/system-config: Move declaration of openstackci/openstackzuul user roles to admin clouds https://review.openstack.org/380084 | 10:54 |
| *** denisra has joined #openstack-infra | 10:54 | |
| *** thorst has quit IRC | 10:55 | |
| *** jkilpatr has quit IRC | 10:56 | |
| *** shardy is now known as shardy_lunch | 10:59 | |
| *** Naeil has quit IRC | 11:03 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Add Horizon Failure Rate dashboard to Grafana https://review.openstack.org/380167 | 11:05 |
| *** amitgandhinz has joined #openstack-infra | 11:06 | |
| dulek | AJaeger: I was just curious and wondered if other jobs are tied to gate queue somewhere else. | 11:10 |
| *** jkilpatr has joined #openstack-infra | 11:17 | |
| *** rhallisey has joined #openstack-infra | 11:19 | |
| *** gildub has quit IRC | 11:26 | |
| *** liusheng has quit IRC | 11:27 | |
| *** lucas-afk is now known as lucasagomes | 11:31 | |
| *** lezbar has quit IRC | 11:33 | |
| *** lezbar has joined #openstack-infra | 11:34 | |
| *** yamamoto has joined #openstack-infra | 11:34 | |
| *** ldnunes has joined #openstack-infra | 11:35 | |
| *** yamamoto_ has joined #openstack-infra | 11:36 | |
| *** andreas_s has quit IRC | 11:36 | |
| *** dizquierdo has joined #openstack-infra | 11:36 | |
| *** artmr has joined #openstack-infra | 11:37 | |
| *** dizquierdo has quit IRC | 11:39 | |
| *** amitgandhinz has quit IRC | 11:40 | |
| *** dprince has joined #openstack-infra | 11:40 | |
| *** ccamacho has quit IRC | 11:40 | |
| *** rossella_ has quit IRC | 11:40 | |
| *** yamamoto has quit IRC | 11:40 | |
| openstackgerrit | Beth Elwell proposed openstack-infra/storyboard-webclient: WIP - Converting less to scss https://review.openstack.org/379595 | 11:40 |
| *** shardy_lunch is now known as shardy | 11:42 | |
| openstackgerrit | Chris Dent proposed openstack-infra/devstack-gate: DNM Test using the optional placement db in nova https://review.openstack.org/379797 | 11:43 |
| *** tlbr has quit IRC | 11:43 | |
| *** jpena is now known as jpena|lunch | 11:46 | |
| *** tonytan4ever has joined #openstack-infra | 11:46 | |
| *** tpsilva has joined #openstack-infra | 11:46 | |
| strigazi | hi mordred, is 378999 working ok? I'm trying to find the right place to put the same file in our project. | 11:48 |
| *** Naeil has joined #openstack-infra | 11:49 | |
| *** _ari_ has quit IRC | 11:50 | |
| *** thorst has joined #openstack-infra | 11:50 | |
| *** tlbr has joined #openstack-infra | 11:50 | |
| *** ccamacho has joined #openstack-infra | 11:51 | |
| *** baoli has joined #openstack-infra | 11:51 | |
| *** lezbar has quit IRC | 11:51 | |
| *** tonytan4ever has quit IRC | 11:51 | |
| *** baoli_ has joined #openstack-infra | 11:51 | |
| *** lezbar has joined #openstack-infra | 11:52 | |
| *** Kiall has joined #openstack-infra | 11:54 | |
| *** EricGonczer_ has joined #openstack-infra | 11:55 | |
| *** baoli has quit IRC | 11:55 | |
| *** caowei has quit IRC | 11:58 | |
| AJaeger | dulek: some of them might be non-voting. | 11:58 |
| *** _ari_ has joined #openstack-infra | 11:59 | |
| *** rfolco has joined #openstack-infra | 12:02 | |
| *** _ari_ has quit IRC | 12:04 | |
| *** tuannguyen has joined #openstack-infra | 12:04 | |
| *** raildo has joined #openstack-infra | 12:05 | |
| *** tuannguyen has quit IRC | 12:09 | |
| *** _ari_ has joined #openstack-infra | 12:09 | |
| *** tuannguyen has joined #openstack-infra | 12:10 | |
| *** tiswanso has joined #openstack-infra | 12:10 | |
| *** tiswanso has quit IRC | 12:10 | |
| *** tiswanso has joined #openstack-infra | 12:11 | |
| *** gordc has joined #openstack-infra | 12:12 | |
| *** trown|outtypewww is now known as trown | 12:12 | |
| *** amoralej is now known as amoralej|lunch | 12:14 | |
| *** panda is now known as panda|afk | 12:15 | |
| *** baoli_ has quit IRC | 12:17 | |
| *** baoli has joined #openstack-infra | 12:18 | |
| *** ssancheztrujillo has quit IRC | 12:18 | |
| *** edmondsw has joined #openstack-infra | 12:21 | |
| *** dfflanders has quit IRC | 12:23 | |
| mordred | strigazi: yes - it is working ... which project are you working on? | 12:24 |
| mordred | (I knew yesterday but have slept since then) | 12:24 |
| *** ccamacho is now known as ccamacho|lunch | 12:25 | |
| AJaeger | infra-root, logs.openstack.org/periodic/propose-requirements-constraints-master/825dd06/console.html#_2016-09-30_07_23_55_926434a shows a failure on the proposal slave. We need to install libnss3-dev there. | 12:25 |
| *** tiswanso has quit IRC | 12:25 | |
| AJaeger | could anybody give me a pointer how best to do this? adding to system-config/modules/openstack_project/manifests/thick_slave.pp ? | 12:25 |
| *** yamamoto_ has quit IRC | 12:26 | |
| mordred | AJaeger: why do we need to install that there all of a suddent do you think? | 12:27 |
| mordred | oh - a requirement has a new build-depend does it? | 12:27 |
| *** tuannguyen has quit IRC | 12:28 | |
| *** pgadiya has quit IRC | 12:28 | |
| *** lezbar has quit IRC | 12:29 | |
| *** lezbar has joined #openstack-infra | 12:30 | |
| *** kgiusti has joined #openstack-infra | 12:30 | |
| mordred | AJaeger: something seems odd to me - I'd like to get fungi's take | 12:30 |
| *** tuannguyen has joined #openstack-infra | 12:32 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config: Install nss-devel on proposal slave https://review.openstack.org/380222 | 12:33 |
| AJaeger | mordred: exactly | 12:33 |
| AJaeger | fungi, mordred ^ that's my proposal | 12:34 |
| AJaeger | dirk: ^ | 12:34 |
| dirk | AJaeger: looks like black magic :) | 12:35 |
| AJaeger | dirk: yeah ;( | 12:36 |
| strigazi | mordred, I work in magnum but in we get timeouts because of octavia | 12:36 |
| mordred | strigazi: so somewhere in the devstack plugin for octavia it's doing a disk-image-builder run I take it? | 12:37 |
| strigazi | mordred, yes | 12:37 |
| *** amitgandhinz has joined #openstack-infra | 12:37 | |
| strigazi | mordred, https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L28 | 12:38 |
| openstackgerrit | Beth Elwell proposed openstack-infra/storyboard-webclient: Converting less to scss https://review.openstack.org/379595 | 12:38 |
| mordred | strigazi: one sec ... | 12:38 |
| *** mdrabe has joined #openstack-infra | 12:38 | |
| *** gouthamr has joined #openstack-infra | 12:40 | |
| mordred | strigazi: https://review.openstack.org/380255 | 12:42 |
| *** amitgandhinz has quit IRC | 12:42 | |
| openstackgerrit | Beth Elwell proposed openstack-infra/storyboard-webclient: Converting less to scss https://review.openstack.org/379595 | 12:43 |
| *** amitgandhinz has joined #openstack-infra | 12:43 | |
| strigazi | mordred, thanks I was trying to do it in magnum but no lack https://review.openstack.org/#/c/379997/ | 12:43 |
| strigazi | *luck | 12:43 |
| *** Jeffrey4l_ has quit IRC | 12:47 | |
| *** jpena|lunch is now known as jpena | 12:47 | |
| EmilienM | gerrit-admins: please add puppet-manager-core into https://review.openstack.org/#/admin/groups/1592,members | 12:47 |
| *** Jeffrey4l_ has joined #openstack-infra | 12:48 | |
| *** psachin has quit IRC | 12:50 | |
| mordred | strigazi: I just pushed an update to your patch to try something | 12:50 |
| strigazi | mordred, ok | 12:50 |
| *** jheroux has joined #openstack-infra | 12:51 | |
| *** markvoelker has joined #openstack-infra | 12:52 | |
| *** matt-borland has joined #openstack-infra | 12:52 | |
| rm_work | mordred: Ah, thought we handled that with our last patch (cached get-pip.py) but I guess not :P | 12:53 |
| rm_work | seems pretty straightforward | 12:53 |
| mordred | rm_work: it's possible you did and I'm not following the right things :) | 12:54 |
| rm_work | nah we were looking at a different location | 12:55 |
| rm_work | in /opt/stack/new | 12:55 |
| *** yamamoto has joined #openstack-infra | 12:56 | |
| *** links has quit IRC | 12:56 | |
| *** david-lyle has joined #openstack-infra | 12:57 | |
| *** rkukura has quit IRC | 12:58 | |
| *** vikrant has quit IRC | 13:01 | |
| *** yamamoto has quit IRC | 13:01 | |
| *** yamamoto has joined #openstack-infra | 13:03 | |
| *** yamamoto has quit IRC | 13:03 | |
| *** drifterza has quit IRC | 13:03 | |
| *** dragonmaster has quit IRC | 13:03 | |
| *** sdake has joined #openstack-infra | 13:04 | |
| *** cardeois has joined #openstack-infra | 13:05 | |
| openstackgerrit | Chris Dent proposed openstack-infra/devstack-gate: DNM Test using the optional placement db in nova https://review.openstack.org/379797 | 13:05 |
| openstackgerrit | Merged openstack-infra/nodepool: Pull get-pip.py from jenkins cache https://review.openstack.org/378999 | 13:06 |
| openstackgerrit | Miguel Angel Ajo proposed openstack-infra/project-config: Create experimental Octavia multinode job https://review.openstack.org/380272 | 13:07 |
| *** mtanino has joined #openstack-infra | 13:08 | |
| *** andymaier has joined #openstack-infra | 13:08 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute053.chocolate provisioning and public IP addresses https://review.openstack.org/380274 | 13:08 |
| *** Goneri has joined #openstack-infra | 13:08 | |
| *** caowei has joined #openstack-infra | 13:08 | |
| *** panda|afk is now known as panda | 13:09 | |
| *** lezbar has quit IRC | 13:09 | |
| *** lezbar has joined #openstack-infra | 13:09 | |
| *** yamamoto has joined #openstack-infra | 13:10 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute057.chococolate provisioning and public IP addresses https://review.openstack.org/380276 | 13:11 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute058.chococolate provisioning and public IP addresses https://review.openstack.org/380276 | 13:12 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute059.chocolate provisioning and public IP addresses https://review.openstack.org/380279 | 13:13 |
| *** cardeois_ has joined #openstack-infra | 13:14 | |
| *** tiswanso has joined #openstack-infra | 13:14 | |
| *** yamamoto has quit IRC | 13:15 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute060.chocolate public and provisioning IP addresses https://review.openstack.org/380281 | 13:15 |
| openstackgerrit | Travis Truman (automagically) proposed openstack-infra/project-config: Tune OSA functional gate triggers https://review.openstack.org/380282 | 13:15 |
| *** cardeois has quit IRC | 13:16 | |
| *** rkukura has joined #openstack-infra | 13:17 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute061.chocolate public and provisioning IP addresses https://review.openstack.org/380283 | 13:17 |
| *** esberglu has joined #openstack-infra | 13:18 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute062.chocolate provisioning and public IP addresses https://review.openstack.org/380285 | 13:19 |
| *** jamesdenton has joined #openstack-infra | 13:19 | |
| *** jamesdenton has quit IRC | 13:19 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute063.chocolate public and provisioning IP addresses https://review.openstack.org/380287 | 13:20 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute064.chocolate public and provisioning IP addresses https://review.openstack.org/380288 | 13:21 |
| *** ccamacho|lunch is now known as ccamacho | 13:21 | |
| *** jamesdenton has joined #openstack-infra | 13:21 | |
| *** amoralej|lunch is now known as amoralej | 13:22 | |
| *** vsaienko has joined #openstack-infra | 13:22 | |
| *** yamamoto has joined #openstack-infra | 13:22 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute065.chocolate public and provisioning IP addresses https://review.openstack.org/380289 | 13:22 |
| *** amitgandhinz has quit IRC | 13:23 | |
| *** amitgandhinz has joined #openstack-infra | 13:23 | |
| *** yamamoto has quit IRC | 13:24 | |
| openstackgerrit | Merged openstack-infra/system-config: Update compute053.chocolate provisioning and public IP addresses https://review.openstack.org/380274 | 13:25 |
| openstackgerrit | Merged openstack-infra/system-config: Update compute058.chococolate provisioning and public IP addresses https://review.openstack.org/380276 | 13:26 |
| openstackgerrit | Merged openstack-infra/system-config: Update compute059.chocolate provisioning and public IP addresses https://review.openstack.org/380279 | 13:27 |
| *** alaski is now known as lascii | 13:29 | |
| openstackgerrit | Merged openstack-infra/system-config: Update compute060.chocolate public and provisioning IP addresses https://review.openstack.org/380281 | 13:29 |
| openstackgerrit | Merged openstack-infra/system-config: Update compute061.chocolate public and provisioning IP addresses https://review.openstack.org/380283 | 13:30 |
| *** tonytan4ever has joined #openstack-infra | 13:34 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute067.chocolate public and provisioning IP addresses https://review.openstack.org/380295 | 13:34 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute068.chocolate public and provisioning IP addresses https://review.openstack.org/380296 | 13:35 |
| openstackgerrit | Merged openstack-infra/system-config: Update compute062.chocolate provisioning and public IP addresses https://review.openstack.org/380285 | 13:36 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update compute069.chocolate public and provisioning IP addresses https://review.openstack.org/380297 | 13:36 |
| openstackgerrit | Merged openstack-infra/system-config: Update compute063.chocolate public and provisioning IP addresses https://review.openstack.org/380287 | 13:37 |
| openstackgerrit | Merged openstack-infra/system-config: Update compute064.chocolate public and provisioning IP addresses https://review.openstack.org/380288 | 13:39 |
| *** kaisers has quit IRC | 13:42 | |
| *** amitgandhinz has quit IRC | 13:42 | |
| *** salv-orlando has joined #openstack-infra | 13:43 | |
| openstackgerrit | Merged openstack-infra/system-config: Update compute065.chocolate public and provisioning IP addresses https://review.openstack.org/380289 | 13:44 |
| fungi | AJaeger: mordred: yeah, it's unfortunate... this is what our bindep implementation would normally fix but the constraints generation (unlike most of our proposal jobs) installs a bunch of things into a virtualenv and so as things get added to global requirements or those packages get new upstream releases over time with new build dependencies, we end up having to wrangle it in puppet because we don't | 13:44 |
| fungi | run the install-distro-packages builder in jobs that are assigned to static job nodes | 13:44 |
| *** salv-orl_ has quit IRC | 13:46 | |
| AJaeger | fungi, thanks for confirming. Could you review 380222 to unblock the requirements proposal jobs, please? | 13:46 |
| openstackgerrit | Merged openstack-infra/system-config: Update compute067.chocolate public and provisioning IP addresses https://review.openstack.org/380295 | 13:47 |
| fungi | maybe if we built up a chroot as part of the job or something, we could use bindep there, but i agree this is a more pragmatic (and less complex) solution | 13:47 |
| fungi | as long as it stays relegated to the thick_slave class i think it should be fine | 13:48 |
| *** adrian_otto has joined #openstack-infra | 13:48 | |
| mordred | cool | 13:48 |
| AJaeger | We could have an extra class for proposal slave if really needed... | 13:49 |
| *** dansmith is now known as superdan | 13:49 | |
| openstackgerrit | Merged openstack-infra/system-config: Update compute068.chocolate public and provisioning IP addresses https://review.openstack.org/380296 | 13:49 |
| AJaeger | thanks, fungi. Anybody else to review 380222, please? | 13:49 |
| * AJaeger needs to run some errands now... | 13:49 | |
| fungi | mordred: did that cover your concerns? | 13:49 |
| fungi | trying to figure out what about it seemed weird | 13:50 |
| *** Naeil has quit IRC | 13:50 | |
| *** akshai has joined #openstack-infra | 13:51 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 13:52 |
| EmilienM | gerrit-admins: please add puppet-manager-core into https://review.openstack.org/#/admin/groups/1592,members - thanks! | 13:52 |
| *** vsaienko has quit IRC | 13:52 | |
| mordred | fungi: yah - just wanted to make sure that we were solving the issue with explicit puppet depends | 13:52 |
| fungi | oh, i may see an issue there | 13:54 |
| fungi | double-checking | 13:54 |
| openstackgerrit | Beth Elwell proposed openstack-infra/storyboard-webclient: Converting less to scss https://review.openstack.org/379595 | 13:56 |
| *** eharney has joined #openstack-infra | 13:56 | |
| fungi | yep, the ubuntu package name is wrong in there. i'll upload a fix | 13:56 |
| rcarrillocruz | EmilienM: done | 13:56 |
| openstackgerrit | Mike Fedosin proposed openstack-infra/project-config: Create glare-specs project https://review.openstack.org/370749 | 13:56 |
| EmilienM | rcarrillocruz: ++ | 13:56 |
| openstackgerrit | Merged openstack-infra/system-config: Update compute069.chocolate public and provisioning IP addresses https://review.openstack.org/380297 | 13:58 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Install nss-devel on proposal slave https://review.openstack.org/380222 | 13:58 |
| *** kaisers_ has joined #openstack-infra | 13:59 | |
| fungi | that should be better | 13:59 |
| *** e0ne has quit IRC | 13:59 | |
| *** rodrigods has quit IRC | 14:00 | |
| *** e0ne has joined #openstack-infra | 14:00 | |
| *** rodrigods has joined #openstack-infra | 14:00 | |
| *** ggnel_t has quit IRC | 14:01 | |
| *** vsaienko has joined #openstack-infra | 14:01 | |
| *** mriedem has joined #openstack-infra | 14:02 | |
| sdake | ok heres a super noob question :) | 14:02 |
| sdake | is there a way to mail the core team directly without having to type all their names in | 14:02 |
| sdake | some kind of gerrit refelctor? | 14:02 |
| sdake | or whatever | 14:03 |
| anteaya | in mail the core team to you mean add them to a review so they get email alerts? | 14:03 |
| *** rbrndt has joined #openstack-infra | 14:03 | |
| anteaya | you can add a group to a review | 14:04 |
| *** tuannguyen has quit IRC | 14:04 | |
| anteaya | I just tried it, if you go to add and start typing the name of the group auto complete offers it up | 14:04 |
| *** amotoki_ has quit IRC | 14:04 | |
| *** adrian_otto has quit IRC | 14:04 | |
| *** yamahata has joined #openstack-infra | 14:05 | |
| *** tuannguyen has joined #openstack-infra | 14:05 | |
| mordred | yah - that said - it's not going to mail anyone if they have their email preferences for reviews set to "dear god never email me" like I do | 14:05 |
| mordred | nor does it do anytihng to cause that review to show up in any of my lists of things I should be looking at - it's a real low point in gerrit IMO | 14:05 |
| *** tiswanso has quit IRC | 14:06 | |
| sdake | anteaya sorry for lag - its a personal mail about personal business | 14:07 |
| fungi | we've discussed some sort of auto-alias in the past for turning gerrit groups into distribution lists, but ultimately that fosters closed discussions. the best way to reach the core reviewers for some project _should_ be the openstack-dev ml | 14:07 |
| *** sjmc7 has joined #openstack-infra | 14:07 | |
| sdake | fungi this is personal business - related to my personal budgets | 14:07 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate compute051 up to compute070 on Puppetmaster Ansible inventory https://review.openstack.org/380324 | 14:07 |
| sdake | fungi not something the whole world needs to knwo about ;) | 14:07 |
| anteaya | sdake: then i have no idea why gerrit would be concerned about emailing a group about a personal email about personal business | 14:07 |
| mordred | sdake: yah - for that typing them in is the best way | 14:08 |
| fungi | so for the rare case where you do actually need a private discussion like that, no we haven't added anything to make that easy | 14:08 |
| anteaya | that doesn't strike me as a use case for gerrit | 14:08 |
| sdake | mordred roger thanks | 14:08 |
| mordred | yah - what fungi said | 14:08 |
| sdake | anteaya agreed | 14:08 |
| sdake | anteaya i jusst for some reason thought it was a feature | 14:08 |
| * sdake lazy - rather make use of existing stuff | 14:08 | |
| *** tiswanso has joined #openstack-infra | 14:08 | |
| mordred | lazy is good | 14:08 |
| anteaya | well you did scope your question accurately | 14:08 |
| mordred | I so perfer when people use things that exist than just write something new | 14:08 |
| AJaeger | fungi, thanks for fixing it. This is indeed correct. I shouldn't have copied & pasted ;( | 14:11 |
| *** andymaier has quit IRC | 14:11 | |
| *** john-davidge has quit IRC | 14:11 | |
| anteaya | AJaeger: you copy/pasted? I don't believe it | 14:12 |
| anteaya | :) | 14:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Revert "Revert "Revert "Revert "Disable rax-iad due to launch failure rate""""" https://review.openstack.org/380330 | 14:12 |
| AJaeger | yeah, duplicated a line - and forgot to change the second part ;( | 14:12 |
| pabelanger | \o/ | 14:12 |
| anteaya | AJaeger: it happens | 14:12 |
| pabelanger | ready to bring rax-iad back online | 14:12 |
| *** vsaienko has quit IRC | 14:12 | |
| anteaya | AJaeger: thank goodness for fungi | 14:12 |
| AJaeger | paul_: Let's rewrite to Revert*5 ;) | 14:12 |
| pabelanger | no doubt | 14:13 |
| AJaeger | anteaya: indeed, he's great! | 14:13 |
| AJaeger | now I need another system-config reviewer to +2A https://review.openstack.org/#/c/380222/, please | 14:13 |
| anteaya | I looked at it but I'm not a system-config reviewer, sorry | 14:14 |
| fungi | pester mordred some more. he's already looked at it some | 14:14 |
| AJaeger | mordred: PLEASE ;) | 14:14 |
| mordred | AJaeger: done | 14:14 |
| *** Naeil has joined #openstack-infra | 14:14 | |
| AJaeger | btw. there're infra-manual changes that I'd like to see in - Zara's https://review.openstack.org/371515 , dhellmann's https://review.openstack.org/#/c/363941/ and mine https://review.openstack.org/376211 . Can anybody give a second +2 for each, please? | 14:15 |
| AJaeger | thanks, mordred | 14:15 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add mosquitto_tls_ca_file to public hiera https://review.openstack.org/375787 | 14:17 |
| *** eharney has quit IRC | 14:18 | |
| openstackgerrit | Merged openstack-infra/system-config: Add chocolate compute051 up to compute070 on Puppetmaster Ansible inventory https://review.openstack.org/380324 | 14:21 |
| openstackgerrit | Merged openstack-infra/system-config: Install nss-devel on proposal slave https://review.openstack.org/380222 | 14:22 |
| *** Jeffrey4l_ has quit IRC | 14:22 | |
| *** Jeffrey4l_ has joined #openstack-infra | 14:23 | |
| *** rlu has quit IRC | 14:23 | |
| *** amotoki has joined #openstack-infra | 14:24 | |
| fungi | AJaeger: ^ should be working again in 15-30 minutes, hopefully | 14:24 |
| *** rlu has joined #openstack-infra | 14:26 | |
| *** yamahata has quit IRC | 14:27 | |
| *** bauzas is now known as bauwser | 14:28 | |
| *** rlu has quit IRC | 14:28 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add mosquitto_tls_ca_file to public hiera https://review.openstack.org/375787 | 14:29 |
| *** yamamoto has joined #openstack-infra | 14:29 | |
| jroll | are there known problems with osic-cloud1 (or in general) with external network connectivity? | 14:30 |
| anteaya | jroll: what are you seeing? | 14:30 |
| jroll | seeing issues connecting to index.docker.io:443 and repo.tinycorelinux.net:80 | 14:30 |
| rcarrillocruz | jroll: i've been bitten quite a bit these days on osic, things like timeouts on connecting to github for examples | 14:30 |
| rcarrillocruz | *example* | 14:30 |
| jroll | the two I've looked at so far are osic-cloud1, looking at the rest | 14:30 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Tune OSA functional gate triggers https://review.openstack.org/380282 | 14:31 |
| *** matt-borland has quit IRC | 14:31 | |
| *** lucasagomes is now known as lucas-afk | 14:31 | |
| anteaya | we were having issues yesterday getting get-pip.py from bootstrap.pypa.io, we didn't consider it a osic cloud1 issue though | 14:32 |
| jroll | anteaya: yep, this is all cloud1, have seven identical failures connecting to one of those two endpoints | 14:33 |
| anteaya | did the job run on any other clouds? | 14:33 |
| anteaya | just we have so much cloud1 these days, it can be hard to tell | 14:34 |
| jroll | looking | 14:34 |
| anteaya | thanks | 14:34 |
| jroll | anteaya: I saw one on infracloud that did fine until it failed later for other reasons | 14:34 |
| anteaya | okay did all the jobs running on cloud1 fail, or just all the failures ran on cloud1 | 14:35 |
| *** oanson has joined #openstack-infra | 14:35 | |
| jroll | all jobs on this patch that ran on cloud1 failed for the same reason https://review.openstack.org/#/c/377446/ | 14:35 |
| jroll | only one did not run on cloud1 | 14:35 |
| anteaya | oh so not just one job, swaths of jobs | 14:36 |
| jroll | other patches to that project see similar failure rates, but haven't yet checked why/which clouds | 14:36 |
| jroll | yes | 14:36 |
| jroll | 11/13 jobs landed on cloud1 there, all failed | 14:36 |
| anteaya | hmmm | 14:37 |
| fungi | jroll: could this be related to neutron's recent address pool change that's conflicting with addresses in osic? | 14:37 |
| fungi | i'm unsure if the fixes for that have landed yet | 14:37 |
| *** david-lyle has quit IRC | 14:37 | |
| jroll | fungi: I'm not aware of that problem | 14:37 |
| *** david-lyle has joined #openstack-infra | 14:38 | |
| fungi | it's been going on for a few days now i think? | 14:38 |
| jroll | been fighting grenade issues and internal things this week | 14:38 |
| sambetts | jroll: https://review.openstack.org/#/c/380023/ | 14:38 |
| *** nherciu_ has quit IRC | 14:38 | |
| jroll | fungi: I heard something about 10./8 being weird, would that affect external connectivity? | 14:38 |
| jroll | "uses 10/8 as the external network" | 14:38 |
| jroll | whee. | 14:38 |
| AJaeger | fungi, great, thanks | 14:38 |
| fungi | jroll: sure, if you're trying to reach an outside ipv4 address since osic is rfc1918 on its default route | 14:39 |
| sambetts | jroll: I ran into the same thing in my third party CI | 14:39 |
| fungi | jroll: we only have global ipv6 there, v4 is rfc1918 to pat | 14:39 |
| jroll | fungi: yep, makes sense. thanks | 14:39 |
| EmilienM | AJaeger: sorry what was context of the ping earlier? | 14:39 |
| * jroll tests with depends-on | 14:39 | |
| *** amotoki has quit IRC | 14:39 | |
| *** burgerk has joined #openstack-infra | 14:40 | |
| jroll | btw, depends-on is still the most awesome feature I've seen since I worked on openstack, thank you whoever built that | 14:40 |
| jroll | (I think jeblair) | 14:40 |
| sambetts | ++ | 14:41 |
| *** drawsmcgraw has joined #openstack-infra | 14:41 | |
| *** yamamoto has quit IRC | 14:41 | |
| AJaeger | EmilienM: discussion with jamespage on policies about support for vendor drivers. No need to follwup today - just suggested that the two of you chat. Question is whether those repos for propietary drivers should be part of the project or only part of OpenStack CI. | 14:41 |
| EmilienM | AJaeger: ok thanks | 14:42 |
| fungi | jroll: the basic design was hashed out among several (infra and qa) people in a studio in manhattan, but jeblair did most of the refinement and implementation | 14:43 |
| jroll | fungi: well then thank you all :D | 14:43 |
| fungi | it was a fun space. mordred gets credit for finding that place | 14:44 |
| anteaya | it was a nice place | 14:44 |
| anteaya | folks rent out their homes there for meetings | 14:45 |
| *** burgerk has quit IRC | 14:45 | |
| *** SamYaple has quit IRC | 14:46 | |
| *** SamYaple has joined #openstack-infra | 14:46 | |
| *** amotoki has joined #openstack-infra | 14:46 | |
| *** salv-orlando has quit IRC | 14:47 | |
| *** xyang1 has joined #openstack-infra | 14:47 | |
| *** adrian_otto has joined #openstack-infra | 14:48 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Remove osic-cloud1 from grafana https://review.openstack.org/380359 | 14:50 |
| *** annegentle has joined #openstack-infra | 14:50 | |
| *** adrian_otto has quit IRC | 14:50 | |
| sjmc7 | hi fungi , clarkb , i’m told you might be able to help me. i had an ACL group (searchlight-ui-core, https://review.openstack.org/#/admin/groups/1584,members) created in https://review.openstack.org/#/c/375148/ but have had trouble tracking someone down to add me as the initial member so i can populate it | 14:52 |
| *** hongbin has joined #openstack-infra | 14:52 | |
| anteaya | pabelanger: so on the index page for grafana we no longer have our nodepool page in the dashboard list http://grafana.openstack.org/ | 14:54 |
| mordred | fungi: yah - I liked that space a lot | 14:54 |
| mordred | fungi: also, I think we might have been the first mid-cycle | 14:55 |
| anteaya | pabelanger: any thoughts about reworking the index page so that it appears again? | 14:55 |
| anteaya | yes it was | 14:55 |
| anteaya | before the word mid-cycle was used | 14:55 |
| anteaya | I think we called it boot camp | 14:55 |
| pabelanger | we should be able to do that with grafyaml, I haven't tested it. That is the default view right now | 14:55 |
| anteaya | cool | 14:56 |
| ajo_ | AJaeger, thanks for looking at https://review.openstack.org/#/c/380272/1 (octavia), I need project-config first, then I can iterate on the other patch with check experimental | 14:56 |
| anteaya | right now it looks like we are hosting a tool for evaluating drivers | 14:57 |
| ajo_ | the other patch is likely to be buggy, or I will go a fill a lottery ticket :) | 14:57 |
| AJaeger | ajo_: that's fine - it's just always good to know how you want to do it;) | 14:57 |
| *** yamamoto has joined #openstack-infra | 14:57 | |
| ajo_ | :) AJaeger | 14:57 |
| *** yamamoto has quit IRC | 14:58 | |
| AJaeger | ajo_: should be ready in 90 mins for you to "check experimental" - it takes around an hour after merge to have our systems configured | 14:58 |
| ajo_ | AJaeger, thank you, and thanks for the heads up too :) | 14:59 |
| *** ijw has joined #openstack-infra | 15:00 | |
| *** ijw has quit IRC | 15:01 | |
| *** ijw has joined #openstack-infra | 15:01 | |
| HeOS | Folks, could you please add me to the following group: https://review.openstack.org/#/admin/groups/1589,members, because that group was created by my request: https://review.openstack.org/#/c/377639/ ? | 15:02 |
| *** cindy has joined #openstack-infra | 15:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Create experimental Octavia multinode job https://review.openstack.org/380272 | 15:03 |
| *** r-mibu has quit IRC | 15:03 | |
| *** r-mibu has joined #openstack-infra | 15:04 | |
| *** rkukura has quit IRC | 15:04 | |
| *** eharney has joined #openstack-infra | 15:06 | |
| anteaya | pabelanger: a quick skim of the grafana docs indicate they expect folks to want to use their modular design but not necessarily customize the look of the index page | 15:07 |
| *** amotoki has quit IRC | 15:07 | |
| anteaya | many references to the dashboard panel not much on how to organize the dashboard panel index | 15:08 |
| anteaya | that I have found in any case | 15:08 |
| clarkb | OH! the subnet pools break more than mulyinode because ipv4 nat | 15:08 |
| clarkb | woo fun maybe we can impress upon devstack/neutron that this needs fixing asap | 15:09 |
| * clarkb goes back to transiting | 15:09 | |
| *** sandanar has quit IRC | 15:09 | |
| *** thcipriani|afk is now known as thcipriani | 15:11 | |
| *** cindy has quit IRC | 15:11 | |
| *** amitgandhinz has joined #openstack-infra | 15:13 | |
| rcarrillocruz | pabelanger: we have now roles and roles assignment on the launcher now | 15:14 |
| rcarrillocruz | s/launcher now/launcher | 15:14 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: aodh: run MySQL and PostgreSQL jobs on Xenial starting with Ocata https://review.openstack.org/373227 | 15:15 |
| *** yamamoto has joined #openstack-infra | 15:15 | |
| *** amitgandhinz has quit IRC | 15:15 | |
| *** amitgandhinz has joined #openstack-infra | 15:15 | |
| pabelanger | anteaya: a quick look shows we should be able to create home.yaml in project-config/grafana, with updated setting. Grafyaml should then override the default dashboard already created | 15:17 |
| pabelanger | but, I've never tested that actually | 15:17 |
| *** akshai has quit IRC | 15:17 | |
| pabelanger | it does look in v3.0, you can redirect the home page to a different dashboard by default too | 15:17 |
| *** dizquierdo has joined #openstack-infra | 15:17 | |
| *** Naeil has quit IRC | 15:17 | |
| anteaya | awesome, thank you for looking | 15:18 |
| *** bnemec is now known as beekneemech | 15:20 | |
| anteaya | pabelanger: also thanks for letting me know I was looking for the home dashboard, I had been looking for the index dashboard | 15:21 |
| pabelanger | np | 15:21 |
| *** esikachev has quit IRC | 15:21 | |
| pabelanger | we should be able to create: http://grafana.openstack.org/dashboard/db/home | 15:22 |
| pabelanger | which will be used by the default landing page | 15:22 |
| anteaya | awesome | 15:22 |
| *** martinkopec has quit IRC | 15:22 | |
| *** berendt has quit IRC | 15:24 | |
| *** amotoki has joined #openstack-infra | 15:24 | |
| *** jordanP has quit IRC | 15:24 | |
| rcarrillocruz | pretty productive day, i've added like 10-15 more computes in chocolate | 15:24 |
| rcarrillocruz | i'm off now, have a good weekend folks | 15:25 |
| anteaya | pabelanger: we are running grafana v3? | 15:25 |
| * rcarrillocruz waves | 15:25 | |
| mordred | \o/ | 15:25 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Raise exception on unrecognized project type https://review.openstack.org/314136 | 15:25 |
| pabelanger | anteaya: sadly no | 15:26 |
| AJaeger | rcarrillocruz: have a great weekend! | 15:26 |
| anteaya | pabelanger: oh, what version are we running? | 15:27 |
| *** adrian_otto has joined #openstack-infra | 15:27 | |
| *** Naeil has joined #openstack-infra | 15:29 | |
| *** ccamacho has left #openstack-infra | 15:29 | |
| *** mhickey has quit IRC | 15:29 | |
| *** rkukura has joined #openstack-infra | 15:29 | |
| *** burgerk has joined #openstack-infra | 15:32 | |
| pabelanger | 2.6.0 | 15:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add AFS publishers to all docs jobs https://review.openstack.org/380381 | 15:32 |
| jeblair | AJaeger, mordred: ^ please review that carefully -- I haven't had breakfast yet :) | 15:32 |
| *** vinaypotluri has joined #openstack-infra | 15:32 | |
| * jeblair eats breakfast | 15:32 | |
| mordred | jeblair: bah. I assume all code you wrote is perfect. I plan on blindly clicking +A | 15:34 |
| mordred | jeblair: (did you emacs that or python that?) | 15:34 |
| anteaya | pabelanger: thanks | 15:35 |
| anteaya | pabelanger: got as far as finding in system-config that we use puppet-grafana 2.1.0 and that that module installs whatever current version of grafana | 15:35 |
| mordred | jeblair: (it actually does lgtm) | 15:36 |
| notmyname | fungi: can you join me in #openstack-swift, please? | 15:37 |
| pabelanger | anteaya: I think we've pinned to 2.6.0, atleast how I remember it. I've been wanting to try 3.0 for a while, but haven't found the time. | 15:37 |
| *** bhavik1 has joined #openstack-infra | 15:40 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Refactor base test classes inheritance for reuse https://review.openstack.org/336090 | 15:42 |
| *** zhenguo has quit IRC | 15:43 | |
| *** krtaylor has joined #openstack-infra | 15:44 | |
| *** krtaylor has quit IRC | 15:44 | |
| jeblair | mordred: nope, has an error :) | 15:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add AFS publishers to all docs jobs https://review.openstack.org/380381 | 15:47 |
| jeblair | mordred, AJaeger: ^ | 15:48 |
| *** hashar is now known as hasharAway | 15:48 | |
| *** sambetts is now known as sambetts|afk | 15:49 | |
| mordred | jeblair: bah | 15:49 |
| *** dtantsur is now known as dtantsur|afk | 15:50 | |
| mordred | jeblair: good catch - I was looking more at the dict bodies - didn't catch the key name | 15:50 |
| *** esikachev has joined #openstack-infra | 15:50 | |
| *** dizquierdo has quit IRC | 15:51 | |
| *** matrohon has quit IRC | 15:52 | |
| *** woodster_ has joined #openstack-infra | 15:53 | |
| *** e0ne has quit IRC | 15:55 | |
| *** caowei has quit IRC | 15:56 | |
| *** tuannguyen has quit IRC | 15:57 | |
| *** tuannguyen has joined #openstack-infra | 15:57 | |
| *** xarses has quit IRC | 15:58 | |
| *** zhenguo has joined #openstack-infra | 15:59 | |
| zaro | mordred: just found this, https://github.com/chungwu/gmail-gerrit | 16:00 |
| zaro | your probably not using gmail though | 16:00 |
| *** tkelsey has quit IRC | 16:01 | |
| *** akshai has joined #openstack-infra | 16:01 | |
| mordred | zaro: interesting. yah - that would not help my life any (I'm actually happy not getting emails) - but I bet there are people whom that would make happy | 16:01 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: make the script for proposing final releases more flexible https://review.openstack.org/380418 | 16:02 |
| *** tuannguyen has quit IRC | 16:02 | |
| *** Hal1 has quit IRC | 16:02 | |
| *** akshai_ has joined #openstack-infra | 16:02 | |
| *** lezbar has quit IRC | 16:03 | |
| zaro | mordred: for some reason my memory says that you wanted that feature at some point in time. | 16:03 |
| *** lezbar has joined #openstack-infra | 16:03 | |
| *** ccamacho has joined #openstack-infra | 16:03 | |
| mordred | zaro: oh - well, yeah - I thought a general email interface would be nice - so that people could submit patches by sending emails to an address, and could review patches by responding to emails | 16:04 |
| mordred | zaro: I don't want it myself - but it's a thing I've heard people desire | 16:05 |
| *** krtaylor has joined #openstack-infra | 16:06 | |
| *** akshai has quit IRC | 16:06 | |
| *** nherciu has joined #openstack-infra | 16:06 | |
| *** itisha has joined #openstack-infra | 16:06 | |
| *** EricGonczer_ has quit IRC | 16:07 | |
| *** rcernin has quit IRC | 16:08 | |
| *** EricGonczer_ has joined #openstack-infra | 16:08 | |
| *** rkukura has quit IRC | 16:09 | |
| *** EricGonc_ has joined #openstack-infra | 16:11 | |
| anteaya | pabelanger: understood thank you | 16:11 |
| *** yamamoto has quit IRC | 16:12 | |
| *** EricGonczer_ has quit IRC | 16:13 | |
| *** Douhet has quit IRC | 16:14 | |
| *** jlanoux has quit IRC | 16:14 | |
| *** Douhet has joined #openstack-infra | 16:14 | |
| jeblair | pabelanger, anteaya: i think AJaeger may be doing friday evening things... can you aprv 380381 ? | 16:17 |
| *** tuannguyen has joined #openstack-infra | 16:17 | |
| *** tphummel has joined #openstack-infra | 16:17 | |
| *** lezbar has quit IRC | 16:19 | |
| pabelanger | sure, let me review | 16:19 |
| *** lezbar has joined #openstack-infra | 16:19 | |
| zxiiro | electrofelix: waynr I'm at the airport are we sprinting today? Anything I should look at? | 16:20 |
| anteaya | jeblair: I was just in the process of doing that, yes | 16:20 |
| anteaya | the only whoops was the one you found first | 16:20 |
| pabelanger | cool | 16:20 |
| jeblair | thanks! | 16:20 |
| anteaya | welcome, thanks for doing all the work | 16:21 |
| *** tosky has quit IRC | 16:21 | |
| jeblair | several infra projects have published into afs overnight | 16:21 |
| pabelanger | I've started work on the vhost patch, will be good to see some directory layout to compare | 16:21 |
| anteaya | yay | 16:21 |
| pabelanger | nice | 16:21 |
| jeblair | these are all separate builds: http://files.openstack.org/docs/infra/ http://files.openstack.org/docs/infra/ http://files.openstack.org/docs/infra/jenkins-job-builder/ http://files.openstack.org/docs/infra/nodepool/ | 16:22 |
| clarkb | jeblair: does jjb just ignore the afs publiaher when talking to jenkins? | 16:22 |
| electrofelix | zxiiro: I hadn't kept track, though I think we only have a few patches to land? | 16:22 |
| * clarkb wonders how that was handled | 16:22 | |
| jeblair | oops, that second one was supposed to be http://files.openstack.org/docs/infra/system-config/ | 16:22 |
| jeblair | clarkb: jjb doesn't know anything about the afs publisher | 16:22 |
| *** Swami has joined #openstack-infra | 16:23 | |
| clarkb | right wondering what gappens if someone tries to reuse our job defs but with jenkins | 16:23 |
| clarkb | does it juat ignore it? | 16:23 |
| jeblair | clarkb: the only implementation is in zuul-launcher; project-config has a stub jjb extension library to make validation work | 16:23 |
| jeblair | clarkb: it will fail | 16:23 |
| zxiiro | electrofelix: yeah last I checked we're almost done. I would love to have review on the views patch though. It's a very important feature for me. | 16:23 |
| waynr | zxiiro: i've been reviewing some changes | 16:24 |
| *** tuannguyen has quit IRC | 16:25 | |
| *** priteau has quit IRC | 16:26 | |
| *** tuannguyen has joined #openstack-infra | 16:26 | |
| *** priteau has joined #openstack-infra | 16:26 | |
| *** shardy has quit IRC | 16:28 | |
| electrofelix | zxiiro: I have some comments on that one | 16:28 |
| clarkb | jeblair gotcha | 16:28 |
| openstackgerrit | Merged openstack-infra/project-config: Add AFS publishers to all docs jobs https://review.openstack.org/380381 | 16:28 |
| *** ihrachys has quit IRC | 16:28 | |
| *** ralonsoh has quit IRC | 16:30 | |
| *** tuannguyen has quit IRC | 16:31 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix confirm() util true state https://review.openstack.org/372014 | 16:32 |
| *** pilgrimstack has quit IRC | 16:34 | |
| waynr | electrofelix zxiiro, i don't think view management should necessarily block a 2.0.0 release... since it's a feature addition we can always release a new minor version later | 16:34 |
| *** jpena is now known as jpena|off | 16:35 | |
| *** bin_ has quit IRC | 16:37 | |
| *** Apoorva has joined #openstack-infra | 16:40 | |
| *** jpich has quit IRC | 16:40 | |
| openstackgerrit | Merged openstack-dev/pbr: Updated from global requirements https://review.openstack.org/377422 | 16:41 |
| zxiiro | waynr: electrofelix agreed. I'd really like to see it made in though as it's a major pain point for us at the moment. | 16:43 |
| *** jlanoux has joined #openstack-infra | 16:44 | |
| *** panda is now known as panda|bbl | 16:45 | |
| *** electrofelix has quit IRC | 16:47 | |
| *** lin_yang has joined #openstack-infra | 16:47 | |
| *** esberglu has quit IRC | 16:49 | |
| *** jlanoux has quit IRC | 16:52 | |
| *** amitgandhinz has quit IRC | 16:53 | |
| openstackgerrit | Merged openstack-infra/nodepool: No need to make sure ZK lock path exists https://review.openstack.org/378961 | 16:53 |
| *** rakhmerov__ is now known as rakhmerov | 16:55 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Move macro expansion into YamlParser. https://review.openstack.org/333076 | 16:56 |
| *** ijw has quit IRC | 16:56 | |
| *** kaisers_ has quit IRC | 16:57 | |
| *** kaisers has joined #openstack-infra | 16:58 | |
| *** derekh has quit IRC | 16:59 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 16:59 |
| *** gyee has joined #openstack-infra | 17:02 | |
| *** ijw has joined #openstack-infra | 17:02 | |
| *** e0ne has joined #openstack-infra | 17:05 | |
| *** Hal has joined #openstack-infra | 17:05 | |
| *** nadya has quit IRC | 17:06 | |
| *** bethwhite has quit IRC | 17:10 | |
| *** nherciu has quit IRC | 17:12 | |
| *** yamamoto has joined #openstack-infra | 17:13 | |
| *** denisra has quit IRC | 17:13 | |
| *** Naeil has quit IRC | 17:14 | |
| *** Naeil has joined #openstack-infra | 17:15 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Bump rh1 back up to 60 max-servers" https://review.openstack.org/380454 | 17:16 |
| *** ijw has quit IRC | 17:16 | |
| *** krtaylor has quit IRC | 17:17 | |
| *** Ravikiran_K has quit IRC | 17:18 | |
| pabelanger | EmilienM: ^ heads up, would like to reduce triple-test-cloud-rh1 back to 50 to reduce the amount of launch failures we are seeing in nodepool. http://grafana.openstack.org/dashboard/db/nodepool-tripleo-test-cloud | 17:18 |
| EmilienM | pabelanger: +1 | 17:19 |
| EmilienM | thanks | 17:19 |
| *** salv-orlando has joined #openstack-infra | 17:19 | |
| *** yamamoto has quit IRC | 17:19 | |
| *** flepied has quit IRC | 17:20 | |
| *** jheroux has quit IRC | 17:21 | |
| *** salv-orlando has quit IRC | 17:21 | |
| *** HeOS has quit IRC | 17:21 | |
| *** salv-orlando has joined #openstack-infra | 17:22 | |
| *** claudiub|2 has quit IRC | 17:22 | |
| *** amitgandhinz has joined #openstack-infra | 17:22 | |
| *** jamesden_ has joined #openstack-infra | 17:26 | |
| *** jamesdenton has quit IRC | 17:28 | |
| *** akshai_ has quit IRC | 17:29 | |
| *** esikachev has quit IRC | 17:30 | |
| *** amoralej is now known as amoralej|off | 17:31 | |
| *** tqtran has joined #openstack-infra | 17:32 | |
| *** dtardivel has quit IRC | 17:37 | |
| clarkb | pabelanger: I am going to be distracted today (feels like I have been all week) but I'd like to make sure I help you with getting the ssh issues sorted out. Have a quick list of changes outstanding for that? or maybe a topic I can use? | 17:37 |
| *** sflanigan_afk has quit IRC | 17:38 | |
| clarkb | portland has been infected by a plague and now I am in seattle to ensure I do my part ot spread it around | 17:39 |
| *** rbrndt has quit IRC | 17:39 | |
| *** pfallenop has quit IRC | 17:40 | |
| anteaya | thanks for doing your part | 17:40 |
| zaro | clarkb: you in SEA? | 17:40 |
| clarkb | zaro: ya at openstackday seattle | 17:40 |
| clarkb | zaro: if you are downtown maybe we can grab pho or something for lunch (its at the W, 4th and seneca) | 17:41 |
| zaro | ohh man, i should have registered for that :( | 17:41 |
| *** pfallenop has joined #openstack-infra | 17:41 | |
| pabelanger | clarkb: actually,, I think we have everything worked out! I've upload ubuntu-trusty / ubuntu-xenail to all regions and SSH timeouts have dropped bigtime | 17:41 |
| clarkb | not sure what our lunch plans are | 17:41 |
| clarkb | zaro: I really want henry's taiwan actually | 17:41 |
| clarkb | might be worth an expedition for that :) | 17:41 |
| zaro | clarkb: sorry, i already have a lunch meeting today :( henry's is good though | 17:42 |
| clarkb | zaro: no problem | 17:43 |
| *** degorenko is now known as _degorenko|afk | 17:43 | |
| *** trown is now known as trown|lunch | 17:43 | |
| *** akshai has joined #openstack-infra | 17:43 | |
| *** Hal has quit IRC | 17:45 | |
| *** Hal has joined #openstack-infra | 17:45 | |
| *** pfallenop has quit IRC | 17:46 | |
| oomichi | pleia2: hello, can you take a look at https://review.openstack.org/#/c/358149/ and https://review.openstack.org/#/c/372581/ ? | 17:46 |
| *** tongli has quit IRC | 17:47 | |
| *** oanson has quit IRC | 17:47 | |
| *** nadya has joined #openstack-infra | 17:48 | |
| *** pfallenop has joined #openstack-infra | 17:49 | |
| mordred | automagically: it took me a second to realize that the automagically in your gerrit name was your irc nick and not an indication that you were a bot :) | 17:53 |
| automagically | mordred: I got that yesterday too | 17:53 |
| automagically | You infra folks must really be on the bot defensive | 17:53 |
| * fungi is now known as fungibot | 17:53 | |
| *** pfallenop has quit IRC | 17:53 | |
| fungi | this bot needs to go refuel. out to lunch, back in a while | 17:54 |
| pabelanger | I, for one, welcome our new overlord fungibot | 17:54 |
| fungi | (like all robots, i need beer to power my circuits) | 17:54 |
| *** pfallenop has joined #openstack-infra | 17:55 | |
| pabelanger | clarkb: I could use a review on: https://review.openstack.org/#/c/380330/ brings rax-iad back online | 17:55 |
| clarkb | 5 reverts! | 17:56 |
| *** dkehn has quit IRC | 17:56 | |
| fungi | ha, ha, ha</thecount> | 17:56 |
| *** krtaylor has joined #openstack-infra | 17:57 | |
| * morgan sighs and is getting frustrated with freenode. | 17:59 | |
| anteaya | automagically: we have had a lot of them | 17:59 |
| anteaya | automagically: and sometimes they take out our systems | 17:59 |
| anteaya | makes us twitchy | 17:59 |
| anteaya | me anyway | 17:59 |
| automagically | anteaya: Understood | 18:00 |
| anteaya | thanks | 18:00 |
| *** pfalleno1 has joined #openstack-infra | 18:00 | |
| *** pfallenop has quit IRC | 18:01 | |
| mordred | automagically: yah - although at the time I was more amused that a bot might have had an opinion on the patch in question | 18:01 |
| anteaya | ha ha ha | 18:01 |
| fungi | veterans of the bot wars | 18:01 |
| mordred | it felt like growing fans or something | 18:01 |
| *** rkukura has joined #openstack-infra | 18:02 | |
| anteaya | ha ha ha | 18:02 |
| anteaya | the fan gardens you must keep | 18:02 |
| anteaya | rows and rows of sprouting fans | 18:02 |
| anteaya | the texas weather must be good for that | 18:03 |
| *** john-davidge has joined #openstack-infra | 18:03 | |
| *** psilvad has joined #openstack-infra | 18:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Revert "Revert "Revert "Revert "Disable rax-iad due to launch failure rate""""" https://review.openstack.org/380330 | 18:03 |
| anteaya | 5 reverts, not a record | 18:03 |
| mordred | I thnk we should revert that | 18:03 |
| anteaya | I think I have seen 6 of them | 18:03 |
| anteaya | there we go then it would tie with what I've seen | 18:04 |
| *** pfalleno1 has quit IRC | 18:06 | |
| fungi | okay, really heading out to lunch now. bbiaw | 18:06 |
| anteaya | enjoy lunch | 18:07 |
| anteaya | and beer | 18:07 |
| *** ccamacho has left #openstack-infra | 18:08 | |
| *** adrian_otto has quit IRC | 18:11 | |
| *** adrian_otto has joined #openstack-infra | 18:14 | |
| *** jamesden_ has quit IRC | 18:15 | |
| *** john-davidge has quit IRC | 18:15 | |
| *** john-davidge has joined #openstack-infra | 18:16 | |
| *** Naeil has quit IRC | 18:16 | |
| *** EricGonc_ has quit IRC | 18:17 | |
| clarkb | sjmc7: any luck with acls yet? | 18:17 |
| *** rossella_ has joined #openstack-infra | 18:17 | |
| *** EricGonczer_ has joined #openstack-infra | 18:17 | |
| clarkb | sjmc7: looks like no /me will poke at the change and see what he can do | 18:18 |
| clarkb | done | 18:19 |
| *** jamesdenton has joined #openstack-infra | 18:19 | |
| *** itisha has quit IRC | 18:21 | |
| *** kaisers has quit IRC | 18:22 | |
| *** rbrndt has joined #openstack-infra | 18:22 | |
| pabelanger | http://files.openstack.org/docs/developer/ | 18:22 |
| pabelanger | yay | 18:22 |
| *** zhenguo has quit IRC | 18:23 | |
| clarkb | do we know where we get the index.html from? | 18:23 |
| clarkb | I assume that that is something built in a docs repo we just need to also update publishing for? | 18:23 |
| pabelanger | not sure | 18:23 |
| pabelanger | maybe AJaeger knows | 18:24 |
| pabelanger | I'll finish up vhosts now | 18:24 |
| cloudnull | afternoons | 18:25 |
| *** aviau has quit IRC | 18:26 | |
| *** aviau has joined #openstack-infra | 18:26 | |
| *** nadya has quit IRC | 18:27 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Bug on remove from attendee schedule https://review.openstack.org/380480 | 18:27 |
| jeblair | pabelanger, clarkb: it may just not have run yet | 18:28 |
| jeblair | pabelanger, clarkb: (we're just filling in content as jobs happen to run when changes land) | 18:29 |
| *** caphrim007 has quit IRC | 18:29 | |
| clarkb | cloudnull: hello there. Out of curiousity what sort of data collection are you doing on your end with the split provider setup in cloud1? I think initial data on our end shows that we are mostly cpu bound so wondering if the data you are getting is useful for you (runtimes for tempest tests are fairly consistent across the flavors) | 18:29 |
| clarkb | jeblair: gotcha | 18:29 |
| cloudnull | ohai | 18:29 |
| *** e0ne has quit IRC | 18:29 | |
| cloudnull | we have https://cloud1.osic.org/grafana/dashboard/db/openstack-all-compute-aggregates | 18:30 |
| clarkb | we probably do have a subset of jobs that are more IO bound | 18:30 |
| clarkb | but I would have to do more digging to find those | 18:30 |
| cloudnull | with various dashboards based on the profiles. | 18:30 |
| cloudnull | and then we have the system dashboard which can drill down into a specific host. | 18:30 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Bug on remove from attendee schedule https://review.openstack.org/380480 | 18:31 |
| cloudnull | I also have appformix running which is giving more detauls into the compute nodes | 18:31 |
| *** EricGonc_ has joined #openstack-infra | 18:31 | |
| clarkb | cloudnull: basically any of the jobs that swap I think will end up giving you more interesting IO data | 18:31 |
| clarkb | cloudnull: I think the trove integration tests and the magnum integration tests do this | 18:32 |
| *** EricGonczer_ has quit IRC | 18:32 | |
| *** nadya has joined #openstack-infra | 18:33 | |
| mgagne | cloudnull: I like those graphs =) | 18:34 |
| cloudnull | since balancing things out we're getting a lot more usage on the s3(5|7)00 nodes which has produced some interesting aggregate metics | 18:34 |
| cloudnull | we're still crunching numbers (we're as in irtermite is) but the data coming out is nice. | 18:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add docs.o.o vhost for files.o.o https://review.openstack.org/380483 | 18:35 |
| pabelanger | jeblair: clarkb: first attempt for docs.o.o vhost ^ | 18:35 |
| cloudnull | mgagne: thanks :) | 18:36 |
| pabelanger | I wonder if we could get access to the current one | 18:36 |
| *** pfallenop has joined #openstack-infra | 18:36 | |
| *** pfallenop has joined #openstack-infra | 18:36 | |
| clarkb | pabelanger: I don't think so as its hosted by cloud sites and we confiure via htpassword | 18:37 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Allow to sign deb-openstack-pkg-tools https://review.openstack.org/380485 | 18:37 |
| clarkb | but I may be wrong | 18:37 |
| openstackgerrit | Merged openstack/os-client-config: Update homepage with developer documentation page https://review.openstack.org/376605 | 18:37 |
| *** bswartz has joined #openstack-infra | 18:38 | |
| jeblair | pabelanger: thanks, one -1 | 18:39 |
| jeblair | pabelanger: (i had a robots.txt in the vhost for files.o.o because it's serving things from a place that isn't normally a web root, but i think in /docs we expect it to be a web root and for there to be a robots.txt file in there -- presumably it will show up when the index.html shows up) | 18:40 |
| clarkb | pabelanger: jeblair also is now the time to think about https? | 18:40 |
| jeblair | pabelanger: i think right after we finish this is the time for that | 18:40 |
| jeblair | clarkb: ^ sorry | 18:40 |
| clarkb | kk | 18:40 |
| clarkb | I don't see anythingwrong with what is there | 18:41 |
| openstackgerrit | Merged openstack/os-client-config: Using assertIsNone() instead of assertEqual(None, ...) https://review.openstack.org/378205 | 18:41 |
| clarkb | another thing to kepe in mind is that if we end up getting the sort of load that the docs people think we may then we might need to bump apaches workers and threads and connections count | 18:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add docs.o.o vhost for files.o.o https://review.openstack.org/380483 | 18:42 |
| jeblair | clarkb: with afs we can bump apache server count :) | 18:42 |
| clarkb | oh right we can do that too | 18:42 |
| clarkb | in different clouds and regions even | 18:42 |
| jeblair | ayup | 18:42 |
| pabelanger | we have nodes again in rax-iad | 18:45 |
| mordred | clarkb: it be almost cloud-like | 18:46 |
| openstackgerrit | Merged openstack/os-client-config: List py35 in the default tox env list https://review.openstack.org/376600 | 18:47 |
| *** trown|lunch is now known as trown | 18:47 | |
| clarkb | pabelanger: what does httpd::params::apache_name resolve to there? | 18:48 |
| clarkb | after saying I see nothing wrong with that I wonder if we are going to end up with two vhosts logging to the same files | 18:48 |
| pabelanger | clarkb: should be apache2 | 18:49 |
| *** salv-orlando has quit IRC | 18:50 | |
| clarkb | oh neat ok | 18:50 |
| clarkb | because centos/rhel use httpd | 18:50 |
| *** Sukhdev has joined #openstack-infra | 18:53 | |
| odyssey4me | project-config cores, I'd appreciate another review of https://review.openstack.org/380150 | 18:53 |
| *** ijw has joined #openstack-infra | 18:54 | |
| odyssey4me | it's a quick and simple one, I promise :) | 18:54 |
| *** ijw has quit IRC | 18:56 | |
| anteaya | odyssey4me: done | 18:57 |
| *** ijw_ has joined #openstack-infra | 18:57 | |
| *** ijw has joined #openstack-infra | 18:58 | |
| *** john-davidge has quit IRC | 18:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Revert "Revert per-resource dogpile.cache work" https://review.openstack.org/366143 | 18:58 |
| odyssey4me | thanks anteaya | 18:59 |
| anteaya | in the category of keeping our monitoring tools in top shape, anyone care to take a look at logstash? the logstash job queue graph is sawtoothing and the most recent elastic search index update was yesterday | 18:59 |
| anteaya | odyssey4me: welcome, thanks for the cleanup | 18:59 |
| *** hamzy has quit IRC | 19:02 | |
| *** nadya has quit IRC | 19:02 | |
| *** tnovacik has quit IRC | 19:03 | |
| *** jkilpatr has quit IRC | 19:04 | |
| *** thorst has quit IRC | 19:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Update gerritbot branches monitored for OpenStack-Ansible https://review.openstack.org/380150 | 19:09 |
| *** annegentle has quit IRC | 19:10 | |
| *** esikachev has joined #openstack-infra | 19:10 | |
| *** annegentle has joined #openstack-infra | 19:10 | |
| *** nadya has joined #openstack-infra | 19:11 | |
| *** lezbar has quit IRC | 19:11 | |
| *** nadya has quit IRC | 19:11 | |
| *** Hal has quit IRC | 19:11 | |
| *** lezbar has joined #openstack-infra | 19:11 | |
| *** panda|bbl is now known as panda | 19:12 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add docs.o.o vhost for files.o.o https://review.openstack.org/380483 | 19:13 |
| *** openstackgerrit has quit IRC | 19:18 | |
| *** openstackgerrit has joined #openstack-infra | 19:18 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Make the ovb-updates job work again https://review.openstack.org/374406 | 19:21 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Force rebuild of images https://review.openstack.org/380521 | 19:21 |
| JayF | project-config changes still will not be in effect if depends-on'd, right? | 19:21 |
| JayF | I know everything is zuul now, but that's zuul-next, correct? | 19:21 |
| *** Sukhdev has quit IRC | 19:22 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test with scheduler hints https://review.openstack.org/378040 | 19:23 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add support for testing predictable placement https://review.openstack.org/378014 | 19:23 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test hostname map https://review.openstack.org/378017 | 19:23 |
| *** bhavik1 has quit IRC | 19:25 | |
| *** thorst has joined #openstack-infra | 19:26 | |
| anteaya | JayF: I'm not sure what you are asking | 19:28 |
| anteaya | can you try again? | 19:28 |
| JayF | anteaya: If I make a project-config change, then a "dummy" ironic change that depends-on it, the ironic change *will not* have the new project-config changes currently, right? | 19:29 |
| anteaya | correct | 19:31 |
| anteaya | since the project config change won't be deployed out to zuul to change the job | 19:31 |
| JayF | zuul v3 will hcange that, right? | 19:31 |
| *** thorst has quit IRC | 19:31 | |
| anteaya | I don't see how | 19:31 |
| JayF | Hm, okay | 19:31 |
| anteaya | has someone said that zuulv3 will employ a yet to be merged project config change? | 19:32 |
| anteaya | I haven't kept up on everything to do with zuulv3 | 19:32 |
| anteaya | so i might be wrong | 19:32 |
| *** Sukhdev has joined #openstack-infra | 19:35 | |
| *** nadya has joined #openstack-infra | 19:36 | |
| *** hamzy has joined #openstack-infra | 19:39 | |
| jeblair | anteaya, JayF: yes (in some (most?) circumstances). :) | 19:41 |
| morgan | anteaya: v3 will support verifying changes to the .zuul.yml files within a repo (what replaces the large monolithic layout) and subsequent tests will be executed with this config | 19:41 |
| morgan | anteaya: think like how changes to .travis.yml works and that tests will verify they work with those changes. | 19:41 |
| *** jkilpatr has joined #openstack-infra | 19:41 | |
| morgan | the .zuul.yml will be in the project's tree vs project-config. there will (afaik) still be common config centrally located | 19:42 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Upgrade to LV 5.2 https://review.openstack.org/305521 | 19:43 |
| anteaya | jeblair: yes, zuulv3 will be able to test with a project config patch? | 19:43 |
| anteaya | morgan: oh cool, thank you | 19:43 |
| anteaya | so not a project config patch a patch to the repos zuul.yml file, nice | 19:44 |
| anteaya | JayF: thanks for your question, I learned something new today | 19:45 |
| JayF | no problem, ty both for your help | 19:45 |
| jeblair | morgan, anteaya: the zuulv3 dynamic config obeys depends-on as well, so a job can execute with the unmerged config in a different repo. | 19:47 |
| *** nadya has quit IRC | 19:47 | |
| anteaya | nice | 19:48 |
| JayF | When is this new shiny awesomeness going to exist? Weeks? Months? "Eventually"? | 19:48 |
| *** lezbar has quit IRC | 19:49 | |
| anteaya | JayF: I think we are somewhere in between months and eventually | 19:49 |
| jeblair | i think that fairly characterizes it | 19:49 |
| *** lezbar has joined #openstack-infra | 19:49 | |
| anteaya | my understanding of the problem is that jeblair is just too popular | 19:49 |
| anteaya | and can't get the monk's environment to write the code | 19:49 |
| jeblair | well, i'm finishing up this afs thing and then i go back in the hole | 19:49 |
| anteaya | yay docs on afs | 19:50 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/377431 | 19:58 |
| fungi | quiet lunch break, even for a friday. it's like people think this release is a foregone conclusion | 19:59 |
| anteaya | ha ha ha | 20:00 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Create (md5|sha256) checksum files for images https://review.openstack.org/380577 | 20:01 |
| pabelanger | clarkb: mordred: ^checksum files generated by diskimage-builder now | 20:02 |
| jeblair | fungi: isn't it? i heard dhellmann built an army of robots that does the release regardless of any human failings? :) | 20:03 |
| fungi | jeblair: clarkb: pabelanger: when https comes up, give me a heads up since i want to make sure we're configuring our stronger options at the start. there's no risk of regression on a site that never had https ;) | 20:03 |
| clarkb | ++ | 20:04 |
| anteaya | dhellmann: does have a mission to automate all the things | 20:05 |
| jeblair | fungi: i think https may be a great opportunity for someone not-jeblair :) | 20:05 |
| *** artmr has quit IRC | 20:05 | |
| fungi | jeblair: yep, i think it can totally come afterward too | 20:05 |
| fungi | just mentioning it because clarkb was asking about it | 20:05 |
| pabelanger | I had to fix https://review.openstack.org/#/c/380483/ because of puppet-lint, but that is our vhost for docs.o.o now | 20:06 |
| fungi | i vote for starting with just http because anything extra we don't need to stir in now is just unnecessary complication | 20:06 |
| *** tdasilva has quit IRC | 20:06 | |
| fungi | jeblair: oh, also, is this what your workstation looks like? https://mobile.twitter.com/filler/status/781175066742534144 | 20:06 |
| * fungi can't pass up a good emacs joke | 20:07 | |
| jeblair | pabelanger: the 'require file' bit is confusing -- we don't actually have that as a file resource, do we? i'm not sure it should be there.... | 20:07 |
| pabelanger | jeblair: Ya, I think openafs-client takes care of that. So, we shouldn't have it in puppet I believe | 20:08 |
| pabelanger | my patch removed requires field | 20:08 |
| *** tiswanso has quit IRC | 20:08 | |
| jeblair | pabelanger: ah right, cool | 20:08 |
| *** salv-orlando has joined #openstack-infra | 20:09 | |
| clarkb | fungi: looks like we killed all our gearman workers again on the logstash workers... | 20:10 |
| *** kaisers has joined #openstack-infra | 20:10 | |
| jeblair | fungi: my chinese isn't that good, you insensitive clod! :) | 20:11 |
| clarkb | did we restart the logstash jvm again? | 20:11 |
| clarkb | I am going to start them on 02 and watch it as best I can | 20:11 |
| clarkb | fungi: maybe we can bounce the others? | 20:12 |
| *** tiswanso has joined #openstack-infra | 20:12 | |
| fungi | clarkb: i didn't restart the java worker, no | 20:13 |
| *** dprince has quit IRC | 20:13 | |
| *** annegentle has quit IRC | 20:13 | |
| clarkb | fungi: I wonder if we merged a patch to do it? | 20:13 |
| *** Apoorva has quit IRC | 20:13 | |
| fungi | should i restart all the java workers (sans 02) and then all the gearman workers again? | 20:13 |
| *** annegentle has joined #openstack-infra | 20:14 | |
| clarkb | fungi: I would leave the java ones alone | 20:14 |
| clarkb | and just do the python gearman workers | 20:14 |
| fungi | oh, you're saying maybe we already _did_ restart the jvms. that makes more sense | 20:14 |
| *** salv-orlando has quit IRC | 20:14 | |
| clarkb | ya | 20:14 |
| clarkb | and we need to bump up the timeout in the python workers to maybe 2-3 minutes instead of just 90 seconds | 20:14 |
| *** burgerk has quit IRC | 20:14 | |
| fungi | i still have my earlier loop queued up and can re-fire it. want me to skip 02 this time? | 20:14 |
| clarkb | fungi: please | 20:14 |
| clarkb | 90 seconds used to be plenty for the jvm to bind to a socket | 20:15 |
| *** lcastell has joined #openstack-infra | 20:15 | |
| fungi | it's underway now, should be finished in a minute then i'll double-check them | 20:15 |
| clarkb | tyty | 20:15 |
| lcastell | anyone around who can help me add some people in here:https://review.openstack.org/#/admin/projects/openstack/craton-dashboard,access | 20:15 |
| lcastell | ? | 20:16 |
| *** kaisers has quit IRC | 20:16 | |
| clarkb | lcastell: have a link to the change that added craton-dashboard to gerrit? | 20:16 |
| *** tiswanso has quit IRC | 20:17 | |
| pabelanger | clarkb: so, we just need to upload trusty / xenial images to rax-iad, happening now. And after that, our known SSH issues will have been solved. | 20:17 |
| lcastell | not sure about that clarkb | 20:17 |
| pabelanger | we basically when 1 hour today with out a single launch failure | 20:17 |
| lcastell | where can I find it? | 20:17 |
| lcastell | :P | 20:17 |
| clarkb | lcastell: in gerrit under the openstack-infra/project-config project | 20:17 |
| fungi | `git log -p` in project-config followed by a keyword search for craton-dashboard is one way | 20:18 |
| lcastell | K | 20:18 |
| lcastell | 1 sec | 20:18 |
| *** salv-orlando has joined #openstack-infra | 20:19 | |
| fungi | clarkb: after a mass restart, number of gearman client processes on the workers is all over the map | 20:19 |
| clarkb | fungi: hrm | 20:19 |
| clarkb | oh wow | 20:20 |
| lcastell | this one clarkb https://review.openstack.org/#/c/354274/ ? | 20:20 |
| fungi | clarkb: like they're already in the process of dying again, maybe | 20:20 |
| clarkb | fungi: 02 has had more recent connections refused | 20:20 |
| clarkb | fungi: which is really really weird | 20:20 |
| clarkb | and the logstash process is gone | 20:20 |
| *** e0ne has joined #openstack-infra | 20:20 | |
| clarkb | no recent OOMs | 20:21 |
| clarkb | I think that maybe we are killing the logstash service somehow | 20:21 |
| *** nadya has joined #openstack-infra | 20:21 | |
| fungi | related to the mqtt integration you think, or separate? | 20:21 |
| clarkb | maybe? perhaps something in that is causing it to crash? | 20:22 |
| clarkb | like the mqtt plugin | 20:22 |
| clarkb | lcastell: I have added david-lyle to the group. The other requested account actually has two gerrit accounts with the same email addr so its not letting me add via name + email | 20:23 |
| *** nadya has quit IRC | 20:24 | |
| lcastell | is that Eddie Ramirez clarkb? | 20:24 |
| clarkb | lcastell: yes | 20:24 |
| lcastell | K | 20:24 |
| clarkb | will need to work with someone here to dedup that (I am in conference talk so not relaly in a space where I can help with that) | 20:24 |
| lcastell | no problem | 20:25 |
| clarkb | fungi: maybe we should put 01 in the emergency file then just quickly remove the mqtt stuff from it and restart things see if it dies quickly | 20:25 |
| fungi | clarkb: sure, doing now | 20:25 |
| lcastell | Thanks for the help clarkb | 20:26 |
| fungi | clarkb: it's in the emergency file now, waiting for the current run_all.sh that started at 20:15 to finish before i go restart processes | 20:26 |
| clarkb | fungi: ok thanks | 20:26 |
| *** xarses has joined #openstack-infra | 20:28 | |
| *** xarses has quit IRC | 20:29 | |
| *** xarses has joined #openstack-infra | 20:29 | |
| odyssey4me | if any project-config cores are available to give this workflow (it already has two votes) then that'd be appreciated https://review.openstack.org/379642 | 20:30 |
| *** ijw has quit IRC | 20:30 | |
| *** Apoorva has joined #openstack-infra | 20:33 | |
| anteaya | odyssey4me: done | 20:33 |
| anteaya | yolanda didn't approve it since the dependency hadn't merged when she reviewed | 20:34 |
| odyssey4me | thanks anteaya | 20:34 |
| anteaya | welcome | 20:34 |
| odyssey4me | yes, I figure as much | 20:34 |
| *** Jeffrey4l_ has quit IRC | 20:35 | |
| *** tiswanso has joined #openstack-infra | 20:36 | |
| anteaya | cool | 20:37 |
| *** baoli has quit IRC | 20:37 | |
| *** baoli has joined #openstack-infra | 20:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Add OSA ceilometer gnocchi functional test https://review.openstack.org/379642 | 20:39 |
| *** ijw has joined #openstack-infra | 20:40 | |
| *** tdasilva has joined #openstack-infra | 20:40 | |
| *** esikachev has quit IRC | 20:40 | |
| *** tiswanso has quit IRC | 20:41 | |
| fungi | clarkb: okay, we're clear to hack on worker01 | 20:44 |
| *** Goneri has quit IRC | 20:44 | |
| mtreinish | fungi, clarkb: did I break everything again? | 20:47 |
| clarkb | mtreinish: somethin is causing logstash jvm to crash | 20:47 |
| fungi | mtreinish: no clue, just trying to figure that out now | 20:47 |
| clarkb | which causes all the logstash python gearman workers to die | 20:47 |
| fungi | i'm reverse-engineering the logstash::indexer class now to see what the manual equivalent of enable_mqtt=>false us | 20:49 |
| fungi | er, is | 20:49 |
| njohnston | Hi! Kind of a basic question, but I can't find it in the docs... I have a periodic job that I'd like to debug, but I don't know how to get the logs for it. Where should I look for periodic job logs? | 20:49 |
| clarkb | njohnston: http://logs.openstack.org/periodic | 20:49 |
| mtreinish | fwiw, that mqtt plugin is stupid simple it just takes in a logstash event and passes that straight to the ruby mqtt publish() call | 20:49 |
| mtreinish | fungi: it would be to drop the mqtt config in /etc/logstash/conf.d/99-output.conf | 20:50 |
| fungi | mtreinish: yep, and the troubleshooting we're doind now is similarly stupid simple, just disabling it on one worker to see if the gearman processes survive | 20:50 |
| mtreinish | that's all enable_mqtt does after the plugin is installed | 20:50 |
| njohnston | Thanks clarkb! | 20:50 |
| clarkb | {:timestamp=>"2016-09-30T20:17:07.452000+0000", :message=>"Error Broken pipe while publishing to MQTT server. Will retry in 10 seconds.", :level=>:error} | 20:50 |
| clarkb | that is in the logstash log | 20:50 |
| openstackgerrit | Merged openstack-infra/system-config: Add docs.o.o vhost for files.o.o https://review.openstack.org/380483 | 20:52 |
| fungi | mtreinish: clarkb: on worker01 i've moved /etc/logstash/conf.d/99-output.conf out of the way and restarted the logstash service. in another minute i'll restart the four gearman services | 20:52 |
| clarkb | fungi: sounds good thanks | 20:53 |
| mtreinish | clarkb: that error, doesn't sound like it's fatal though | 20:53 |
| clarkb | mtreinish: ya it happened a bunch | 20:53 |
| clarkb | so I don't think its fatal but maybe leads up to something? | 20:53 |
| *** rtheis has quit IRC | 20:53 | |
| *** EricGonc_ has quit IRC | 20:55 | |
| *** raildo has quit IRC | 20:55 | |
| fungi | mtreinish: clarkb: jenkins-log-worker-A through D restarted on worker01 now | 20:57 |
| *** EricGonczer_ has joined #openstack-infra | 20:57 | |
| fungi | a quick check indicates worker01 and worker12 are the only places any of them are running at this point (no clue what's up with worker12) | 20:58 |
| mtreinish | clarkb: I am curious what's going on with the connection to firehose. I wouldn't have expected it to be that unreliable | 20:58 |
| *** gtmanfred has quit IRC | 20:58 | |
| mtreinish | fungi: I'm only seeing logstash messages from worker 12 on firehose right now | 20:58 |
| fungi | makes sense | 20:59 |
| fungi | it's the only one that somehow survived | 20:59 |
| fungi | 01 has lpmqtt disabled for now | 20:59 |
| fungi | i can try restarting the log worker processes on 02 through 20 again to see if them re-dying after the last restart was a fluke | 21:00 |
| *** tiswanso has joined #openstack-infra | 21:01 | |
| *** psilvad has quit IRC | 21:01 | |
| * anteaya puts money on 12 at the roulette wheel | 21:01 | |
| *** amitgandhinz has quit IRC | 21:03 | |
| *** gtmanfred has joined #openstack-infra | 21:04 | |
| fungi | in craps that's "boxcars" or "midnight" | 21:04 |
| anteaya | is it? | 21:05 |
| * anteaya learns another new thing today | 21:05 | |
| fungi | some tables apparently pay triple on 12 | 21:05 |
| * fungi stops looking at wikipedia again | 21:05 | |
| *** tiswanso has quit IRC | 21:05 | |
| anteaya | do they? | 21:05 |
| anteaya | ha ha ha | 21:05 |
| mgagne | is there any reasons for a bot/ci other than Jenkins to vote on specs? | 21:06 |
| anteaya | the one time I was in a casino it struck me as one of the most depressing places to ever be | 21:06 |
| anteaya | mgagne: only if the project wants it to | 21:06 |
| fungi | mgagne: depends on whether you count "misconfiguration" as a reason | 21:06 |
| anteaya | mgagne: have you a patch where this is happening? | 21:07 |
| mgagne | fungi: legitimate reasons then =) | 21:07 |
| mgagne | https://review.openstack.org/#/c/312626/ | 21:07 |
| *** akshai has quit IRC | 21:07 | |
| fungi | none that i can think of, but as anteaya says if the nova team has requested that "Open-E JovianDSS CI" report on their specs, i suppose that's up to them | 21:08 |
| mtreinish | mgagne: maybe someone is working on an auto translation bot. Like pipe specs repos into google translate :) | 21:08 |
| *** tuannguyen has joined #openstack-infra | 21:08 | |
| fungi | or maybe the jovian planet is really interested in novas, so that it has other options than becoming a brown dwarf | 21:09 |
| anteaya | they are commenting on random things | 21:09 |
| anteaya | someone fired it up about 10 minutes ago looks like | 21:09 |
| mgagne | awesome =) | 21:09 |
| *** john-davidge has joined #openstack-infra | 21:10 | |
| anteaya | I'd say misconfigured, they probably brought in the current project config files and pressed go | 21:10 |
| mgagne | and it's commenting with "Starting gate jobs." | 21:10 |
| anteaya | again a newbie mistake | 21:10 |
| * fungi expects that his vague reference to 2010 went unnoticed, and sighs | 21:11 | |
| *** gouthamr has quit IRC | 21:11 | |
| clarkb | I love clarke but I have never been able to read that series of books | 21:12 |
| clarkb | maybe I should try again | 21:12 |
| *** thorst has joined #openstack-infra | 21:12 | |
| *** trown is now known as trown|outtypewww | 21:12 | |
| anteaya | I don't see an irc nick: https://wiki.openstack.org/wiki/ThirdPartySystems/Open-E_JovianDSS_CI | 21:12 |
| odyssey4me | anteaya mgagne oh yeah, I'm seeing results from them too on openstack-ansible repositories | 21:12 |
| anteaya | fungi: I'm sorry I missed that | 21:12 |
| anteaya | odyssey4me: yeah it is all over | 21:12 |
| mgagne | -> https://wiki.openstack.org/wiki/ThirdPartySystems/Open-E_JovianDSS_CI Current Status: disabled | 21:12 |
| anteaya | yeah | 21:12 |
| anteaya | soon to be true | 21:12 |
| fungi | granted, my attempt to draw similarities between the greek and roman names for a similar god were probably a stretch | 21:12 |
| amrith | clarkb, ping | 21:13 |
| clarkb | amrith: pong | 21:13 |
| *** tuannguyen has quit IRC | 21:13 | |
| amrith | hiya | 21:13 |
| amrith | I think email was not quite working for what I was trying to say | 21:13 |
| amrith | do you have a minute? | 21:13 |
| clarkb | sure | 21:13 |
| amrith | about the PYTHONBYTECODE thing | 21:13 |
| amrith | currently, trove has this in tox.ini | 21:14 |
| amrith | find ./trove -type f -name "*.pyc" -delete | 21:14 |
| *** ldnunes has quit IRC | 21:14 | |
| anteaya | gerrit id 22381: http://paste.openstack.org/show/583683/ | 21:14 |
| amrith | I saw the mail about PYTHONDONTWRITEBYTECODE = 1 | 21:14 |
| anteaya | fungi: if you could be so kind | 21:14 |
| amrith | and figured this may be a better solution than the find/delete | 21:14 |
| amrith | but it doesn't seem to NOT make .pyc's (sorry for double negative) | 21:14 |
| amrith | it still makes .pyc's | 21:14 |
| clarkb | fungi: looks like 01 is still running the workers | 21:14 |
| clarkb | amrith: yes | 21:14 |
| amrith | and so I think we still need the find/rm | 21:14 |
| *** john-davidge has quit IRC | 21:14 | |
| amrith | ok | 21:14 |
| clarkb | amrith: my guess is the var is not being set properly | 21:14 |
| fungi | anteaya: 22381 is inactive now | 21:15 |
| clarkb | my suggestion was if that matters to you just rebuild your venv | 21:15 |
| anteaya | fungi: thank you | 21:15 |
| clarkb | as that is far better than running a find or telling the python interpreter to not do things that speed up execution time | 21:15 |
| fungi | anteaya: the two connections that account had open have now been manually closed as well | 21:15 |
| amrith | clarkb, please see https://gist.github.com/anonymous/fdf7cdaeee32405af53d9ad2e84d1056 | 21:15 |
| clarkb | amrith: remove the whitespace around the = | 21:16 |
| fungi | heh, yeah, shell assignments are tricky like that | 21:17 |
| amrith | so, clarkb, not like this https://github.com/openstack/ironic/blob/c445c19285c2f3ee6a099f9f7473dd6fe087116b/tox.ini#L10 | 21:17 |
| amrith | the sample code from ironic has the whitespace | 21:17 |
| amrith | I'll try without | 21:17 |
| clarkb | ya thats not how your shell works typically | 21:17 |
| *** edmondsw has quit IRC | 21:17 | |
| clarkb | fwiw current pip maintains a wheel cache when you install stuff | 21:18 |
| clarkb | which means reinstalls should be very very fast | 21:18 |
| clarkb | which in turn means that tox -r and not worrying should be relatively quick | 21:18 |
| clarkb | (though there are a bunch of things that factor into that) | 21:18 |
| fungi | learned just now from a thread on debian-devel, setting RES_OPTIONS=attempts:0 in the environment effectively disables any name resolution through glibc! | 21:18 |
| fungi | that's a potentially useful trick | 21:19 |
| clarkb | fungi: is that so you can avoid using sytemd dns over dbus? | 21:19 |
| jeblair | clarkb: lol | 21:19 |
| fungi | clarkb: itym systemdbus | 21:19 |
| jeblair | fungi: should we set that in the zuul parameter function? :) | 21:19 |
| clarkb | jeblair: you laugh but I learned a couple days ago its a thing | 21:19 |
| anteaya | fungi: awesome thank you | 21:19 |
| *** sdake has quit IRC | 21:19 | |
| anteaya | http://lists.openstack.org/pipermail/third-party-announce/2016-September/000308.html | 21:20 |
| *** rfolco has quit IRC | 21:20 | |
| fungi | thanks for handling the communication on that anteaya! | 21:20 |
| anteaya | welcome, thanks for disabling :) | 21:20 |
| fungi | clarkb: for whatever reason, worker01 and worker12 are still going strong. 01 has lpmqtt disabled, 12 has it enabled | 21:21 |
| anteaya | c'mon 12 | 21:21 |
| clarkb | fungi: perhaps load related? | 21:21 |
| fungi | i'm tempted to restart 02 through 20 again and see what happens | 21:21 |
| *** lascii is now known as alaski | 21:22 | |
| anteaya | did you restart at 21:00? | 21:22 |
| anteaya | 02 through 20 that is | 21:22 |
| fungi | anteaya: no, not yet | 21:23 |
| fungi | curious to see whether 12 surviving is a fluke | 21:23 |
| *** e0ne has quit IRC | 21:23 | |
| anteaya | ah | 21:23 |
| fungi | anteaya: last mass restart i did was at 20:15 utc | 21:23 |
| anteaya | thanks | 21:23 |
| fungi | so more than an hour ago | 21:24 |
| *** e0ne has joined #openstack-infra | 21:24 | |
| fungi | all but 12 died pretty quickly after that (within 10-15 minutes) | 21:24 |
| anteaya | was attempting to figure out which timestamp the again should point to | 21:24 |
| *** e0ne has quit IRC | 21:24 | |
| fungi | i'll give it a shot and see if we have any consistency | 21:24 |
| *** e0ne has joined #openstack-infra | 21:25 | |
| *** eharney has quit IRC | 21:25 | |
| *** e0ne has quit IRC | 21:25 | |
| *** sdake has joined #openstack-infra | 21:25 | |
| * anteaya puts another $5 on 12 | 21:25 | |
| fungi | running now. i've omitted 01 since we're waiting to see if it survives. 12 is in the restart batch to see if it's still unaffected next go round | 21:25 |
| *** e0ne has joined #openstack-infra | 21:25 | |
| *** e0ne has quit IRC | 21:26 | |
| *** e0ne has joined #openstack-infra | 21:26 | |
| *** rlandy has quit IRC | 21:26 | |
| *** e0ne has quit IRC | 21:27 | |
| fungi | right now there are 4 running on each of all 20 workers | 21:27 |
| *** e0ne has joined #openstack-infra | 21:27 | |
| fungi | i'll check back in on them in a few minutes | 21:27 |
| *** e0ne has quit IRC | 21:27 | |
| mtreinish | fungi: the floodgates opened on firehose again | 21:28 |
| *** e0ne has joined #openstack-infra | 21:28 | |
| fungi | i expect so | 21:28 |
| amrith | clarkb, no joy. Please see: https://gist.github.com/anonymous/da3bb0fb30a7f47bf1c6e7a7b4b02339 | 21:28 |
| fungi | just a question of how long the deluge persists | 21:28 |
| amrith | I removed the whitespace | 21:28 |
| amrith | cleaned (git clean -f -x -d) | 21:28 |
| amrith | checked that there are no .pyc's | 21:28 |
| *** e0ne has quit IRC | 21:28 | |
| amrith | then ran tox -e pep8 | 21:28 |
| fungi | mtreinish: clarkb: one has already died on worker03 | 21:28 |
| amrith | and there are now 5423 pyc's :( | 21:28 |
| anteaya | 3 minutes, nice | 21:29 |
| mtreinish | fungi: any indication why? | 21:29 |
| clarkb | amrith: the bash invocation there may be killing it if the env is not being exported | 21:30 |
| fungi | mtreinish: two more on 03 have died in the time it took me to log in | 21:30 |
| clarkb | amrith: I do not know if tox exports properly | 21:30 |
| amrith | ok, thx. I'll check with the ironic folks who started the thread | 21:30 |
| mtreinish | heh now I'm only seeing messages from worker 19 | 21:30 |
| amrith | clearly something appears to be working for them. | 21:30 |
| amrith | thanks clarkb | 21:30 |
| fungi | mtreinish: error: [Errno 111] Connection refused | 21:30 |
| amrith | have a good weekend! | 21:30 |
| mtreinish | fungi: to logstash? | 21:30 |
| mtreinish | now it's 04 and 19 | 21:31 |
| fungi | mtreinish: i expect so, raised in processor.handle_log_event() -> _connect_socket() | 21:31 |
| mtreinish | I guess that's jsut how things are getting split out | 21:31 |
| anteaya | there is a voting mechanism with logstash isn't there | 21:31 |
| mtreinish | fungi: hmm, yeah that makes sense | 21:32 |
| anteaya | the group of them select on who the leader is | 21:32 |
| anteaya | would they be taking everyone down and just leaving one | 21:32 |
| anteaya | are they fighting? | 21:32 |
| fungi | clarkb: mtreinish: this time we have log-workers still running on 01, 04 and 19. all the rest have died now | 21:32 |
| fungi | anteaya: that's for the elasticsearch cluster master i believe | 21:32 |
| anteaya | ah sorry | 21:33 |
| fungi | so unrelated to logstash-worker servers | 21:33 |
| *** Goneri has joined #openstack-infra | 21:33 | |
| mtreinish | fungi: just for a sanity check is there anything on firehose01.o.o (either dmesg or the mosquitto logs) | 21:33 |
| mtreinish | maybe it's dropping connections because we're hitting some kinda limit or something | 21:33 |
| mtreinish | and if that happens enough logstash dies | 21:33 |
| mtreinish | like that connection ulimit we were hitting in the local testing a few months ago (but that was 1k concurrent connections) | 21:34 |
| fungi | mtreinish: nothing "recent" in dmesg output (latest entry was an apparmor audit entry from some 42 hours ago) | 21:35 |
| *** dkehn has joined #openstack-infra | 21:36 | |
| fungi | mosquitto logs are extremely verbose and in epoch time so no promises, but skimming i'm not seeing obvious issues logged in there | 21:36 |
| mtreinish | I'm trying to remember where I saw the ulimit errors before | 21:36 |
| mtreinish | fungi: oh, hah I probably should revert: https://review.openstack.org/#/c/374154/ | 21:37 |
| fungi | oh, here's something. this corresponds to a logstash worker... | 21:37 |
| fungi | 1475271100: OpenSSL Error: error:140F3042:SSL routines:ssl_undefined_const_function:called a function you should not call | 21:37 |
| fungi | 1475271100: Socket error on client <unknown>, disconnecting. | 21:37 |
| mtreinish | yeah, that would explain a broken connection | 21:38 |
| mtreinish | not sure why that's happening though | 21:38 |
| fungi | the lines around that indicate the connection was from "104.130.138.60" (worker19) | 21:38 |
| jeblair | ...called a function you should not call and caused an error you should not cause. | 21:38 |
| JayF | https://bz.apache.org/bugzilla/show_bug.cgi?id=58956 related, perhaps? | 21:39 |
| openstack | bz.apache.org bug 58956 in mod_ssl "SSL Library Error ssl_undefined_function:called a function you should not call after updating to OpenSSL 1.0.2f" [Normal,Resolved: wontfix] - Assigned to bugs | 21:39 |
| fungi | mtreinish: interestingly, all the workers on 19 are still running | 21:39 |
| JayF | it sounds like that's a symptom, not the cause though, looking at the commit | 21:39 |
| fungi | though they04 | 21:40 |
| fungi | grrr | 21:40 |
| fungi | mtreinish: though they've all died on 04 now | 21:40 |
| JayF | seems like if you shutdown the ssl socket mid-handshake openssl can throw that error erroneously | 21:40 |
| *** roaet_ has quit IRC | 21:40 | |
| anteaya | fungi: what worker are you on currently as you look at that log? | 21:40 |
| fungi | JayF: yeah, i saw lines after it indicating the client reconnected successfully | 21:40 |
| fungi | anteaya: no worker, those log entries were on firehose01.o.o | 21:40 |
| anteaya | ah | 21:40 |
| fungi | anteaya: in the mosquitto service log | 21:41 |
| *** roaet has joined #openstack-infra | 21:41 | |
| anteaya | thanks | 21:41 |
| mtreinish | fungi: that probably corresponds to the errors clarkb was talking about before then | 21:41 |
| mtreinish | which don't seem fatal | 21:41 |
| fungi | so anyway, it looks like worker01 which has been manually reconfigured to not use lpmqtt is still running strong, but with the rest we reach a steady state where only one has any workers still running (at least for a while) | 21:42 |
| mtreinish | fungi: we can revert the revert and turn mqtt reporting off again | 21:42 |
| clarkb | we might try inverting it and see if one works with mqtt without dying | 21:42 |
| fungi | could do that, sure. let's revert lpmqtt enablement, but leave worker01 disabled in puppet | 21:43 |
| fungi | then i can manually put the lpmqtt plugin config back into place on 01 | 21:44 |
| *** gordc has quit IRC | 21:44 | |
| fungi | and restart the java daemon and the gearman clients on it | 21:44 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Revert "Revert "Disable mqtt output plugin on logstash workers"" https://review.openstack.org/380662 | 21:44 |
| mtreinish | fungi: ^^^ | 21:44 |
| mtreinish | fungi: heh, also lpmqtt is my lp imap -> mqtt daemon :) | 21:45 |
| *** sdake has quit IRC | 21:45 | |
| *** rockyg has joined #openstack-infra | 21:46 | |
| *** sdake has joined #openstack-infra | 21:46 | |
| *** thorst has quit IRC | 21:46 | |
| fungi | er, lsmqtt ;) | 21:47 |
| *** thorst has joined #openstack-infra | 21:47 | |
| fungi | sorry about that | 21:47 |
| mtreinish | fungi: heh, no worries. I figured it was jsut a typo | 21:48 |
| fungi | it was a fingers wishing we had an easier way to type mqtt plugin for logstash workers | 21:48 |
| *** ijw has quit IRC | 21:49 | |
| mtreinish | lswmqttp? | 21:49 |
| *** jtomasek has quit IRC | 21:50 | |
| fungi | wwmqttd? | 21:51 |
| *** priteau has quit IRC | 21:51 | |
| *** HeOS has joined #openstack-infra | 21:52 | |
| clarkb | oh man | 21:52 |
| clarkb | ITIL | 21:52 |
| fungi | that's sort of a random interjection | 21:53 |
| fungi | do you interject itil often? | 21:53 |
| *** Swami has quit IRC | 21:53 | |
| clarkb | I just walked into a room | 21:53 |
| clarkb | and they are talking abouy iyil | 21:53 |
| fungi | WALK BACK OUT | 21:53 |
| *** jamesdenton has quit IRC | 21:53 | |
| clarkb | I hoped to never care about this again | 21:53 |
| clarkb | oh I know | 21:53 |
| clarkb | I am ITIL certified for somethibg or other | 21:53 |
| fungi | i think working on itil anything qualifies you as certifiable for institutionalization | 21:54 |
| clarkb | now I am curious how this maps to opemstack | 21:54 |
| anteaya | I love clarkb phone typing | 21:54 |
| fungi | in some process manager's obscene dreams, probably | 21:54 |
| anteaya | its my favourite | 21:55 |
| *** thorst has quit IRC | 21:55 | |
| clarkb | I did like a week of ITIL training in a windowless basement in a galaxy far far away | 21:56 |
| anteaya | sounds memorable | 21:57 |
| fungi | clarkb: looks like sc68cal took over 379543 a few hours ago... still lgty? | 21:58 |
| *** jamielennox|away is now known as jamielennox | 21:59 | |
| clarkb | ya should make our end happy | 21:59 |
| fungi | seems he just expanded FIXED_RANGE_V6 to a /56 | 21:59 |
| fungi | otherwise it's still what you had | 21:59 |
| clarkb | not sure if those default defaults make sense for devstack though | 21:59 |
| jeblair | clarkb: where are you? :) | 21:59 |
| *** sdake has quit IRC | 21:59 | |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Revert "Disable mqtt output plugin on logstash workers"" https://review.openstack.org/380662 | 22:00 |
| *** kaisers has joined #openstack-infra | 22:00 | |
| clarkb | ya the /56 allows several /64s to be created which is I guess the need | 22:00 |
| clarkb | jeblair: openstack day seattle | 22:00 |
| fungi | chief seattle was always a big fan of free software | 22:00 |
| clarkb | looks like cloudify + servicenow + openstack == ITIL | 22:02 |
| *** flepied has joined #openstack-infra | 22:04 | |
| *** annegentle has quit IRC | 22:05 | |
| *** kaisers has quit IRC | 22:05 | |
| zigo | fungi: https://review.openstack.org/380485 <--- Can I be allowed to tag deb-openstack-pkg-tools please? | 22:06 |
| *** devananda is now known as devananda|away | 22:08 | |
| *** ijw has joined #openstack-infra | 22:09 | |
| fungi | zigo: that group is starting to grow (it's up to 5 members now). you have a deb-packaging-release team that's the restricted subset able to create branches... wouldn't that make more sense for the list of accounts allowed to push tags? | 22:09 |
| fungi | we've just observed that the infrequency with which tags are created and pushed, and relative danger/disruption from that process, is better limited to fewer contributors than general change approval | 22:10 |
| zigo | fungi: In fact, I was thinking that probably only people with upload rights in Debian should be allowed to sign, so maybe, yeah. | 22:11 |
| *** Apoorva_ has joined #openstack-infra | 22:11 | |
| fungi | zigo: if you want to update it now, i'm happy to approve | 22:12 |
| zigo | fungi: Where is defined the deb-packaging-release team members? | 22:12 |
| fungi | https://review.openstack.org/#/admin/groups/1137,members | 22:12 |
| zigo | Ah, cool, it's easy to edit, just like cores ! :) | 22:13 |
| fungi | you can either browse to group member lists from the project acl views in gerrit, or search the groups index by name | 22:13 |
| *** ijw has quit IRC | 22:14 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Allow to sign deb-openstack-pkg-tools https://review.openstack.org/380485 | 22:15 |
| *** Apoorva has quit IRC | 22:15 | |
| zigo | fungi: Done ! | 22:15 |
| fungi | and approved | 22:15 |
| zigo | Cheers. | 22:15 |
| zigo | fungi: FYI, besides a few bugs I'm currently addressing, and testing all of everything, we're done with the Newton release packaging. :) | 22:16 |
| zigo | All in Infra ! :) | 22:16 |
| fungi | clarkb: mtreinish: i suspect i disabled the mqtt plugin on worker01 incorrectly, as i moved all of 99-output.conf out of the way rather than removing the mqtt section from it, so it's not been doing elasticsearch output either | 22:16 |
| zigo | It wouldn't have happened without you and many others. | 22:16 |
| zigo | So, a big thanks ! | 22:16 |
| mordred | zigo: woot! congrats! | 22:16 |
| *** tiswanso has joined #openstack-infra | 22:17 | |
| fungi | zigo: that's awesome news! i know we fought back and forth on design and planning for this, but i'm glad the end result was a satisfying first compromise for you | 22:17 |
| mtreinish | fungi: oh, heh yeah that means it's not outputting anythign | 22:17 |
| fungi | and there's certainly room to improve further | 22:17 |
| zigo | fungi: That's the list of things that are already available. A start means already uploaded to Debian Experimental. | 22:17 |
| zigo | http://paste.openstack.org/show/583693/ | 22:17 |
| fungi | mtreinish: right, so that could also explain why worker01 is super stable | 22:17 |
| fungi | zigo: amazing for basically all happening over the course of a single dev cycle | 22:18 |
| zigo | fungi: It really happened between Newton b2 and now. So that's 1/3rd of a dev cycle, really ! :) | 22:19 |
| mtreinish | fungi, clarkb: which version of logstash are we running on the workers? | 22:19 |
| mtreinish | I was just looking at the dep list on the gems page: https://rubygems.org/gems/logstash-output-mqtt/ | 22:19 |
| fungi | zigo: well, you tackled the learning curve from basically knowing nothing about our ci to being one of the most complex use cases for it in 6 months | 22:19 |
| mriedem | clarkb: any idea what package projects used to use to generate nose_results.html output? | 22:19 |
| mtreinish | and I'm wondering if it changed a version in a bad way | 22:19 |
| zigo | fungi: Now, the current biggest pain point is not having a "real" signed debian mirror, I hope you guys can help with that soonish. | 22:19 |
| mtreinish | mriedem: probably horizon | 22:19 |
| mriedem | clarkb: looking at https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/run-tox.sh#L97 | 22:19 |
| mtreinish | I think they're the only using nose still (for django tests) | 22:20 |
| zigo | fungi: Oh, there's still lots of magic that are going on in the background which I don't understand at all ! :) | 22:20 |
| zigo | fungi: As I hope to reprocude that CI for Debian itself, hopefully, I'll learn. | 22:20 |
| fungi | zigo: me either! | 22:20 |
| zigo | fungi: Do you think it'd be hard for me to setup zuul, nodepool and such on a PoC VM ? | 22:21 |
| mriedem | mtreinish: yeah but some nose plugin generates the html file | 22:21 |
| fungi | mtreinish: looks like 2.0.0 distro package | 22:21 |
| mtreinish | mriedem: oh I misread the question. Is it this thing: https://github.com/openstack-dev/openstack-nose | 22:21 |
| mtreinish | fungi: ok, that matches the logstash-core dep on the plugin | 22:21 |
| mriedem | mtreinish: ah ok sweet | 22:21 |
| *** tiswanso has quit IRC | 22:22 | |
| fungi | zigo: if you feel like starting from a puppet manifest that installs them from source, we already have it automated. if you want to work from scratch then i don't know how easy that'll be, but will involve a much deeper reading of the per-service documentation we provide | 22:22 |
| zigo | fungi: The DSAs (Debian System Administrators) team very much like puppet, so that would IMO be the way to go. My only concern is if it uses packages from Debian itself or not. | 22:23 |
| jeblair | zigo: also, note that we expect zuulv3 to be a lot easier for folks to be able to install and run | 22:23 |
| zigo | fungi: It'd had to run on Debian Stable if possible. | 22:23 |
| fungi | zigo: if you can, i'd recommend setting them up from our "openstackci" puppet module on an ubuntu 14.04 lts vm, and then using that as a known-working example to reference while doing the package-oriented version | 22:24 |
| *** tpsilva has quit IRC | 22:24 | |
| zigo | fungi: Thanks for the tip. | 22:24 |
| fungi | trying to get it all working together blind from scratch will be a much greater challenge than if you have one that works (even if it's only running sort-of-debian and source/pip installed bits) | 22:24 |
| zigo | fungi: I also would like to start helping the infra team by taking some packaging work for the tooling you guys are using. | 22:25 |
| fungi | which you can use to compare your configuraiton and behavior against | 22:25 |
| zigo | fungi: Right after the release will be the perfect moment. | 22:25 |
| zigo | (maybe a few days after, so I can polish the packages and upload them to Unstable + Jessie backports official) | 22:25 |
| mriedem | mtreinish: that doesn't generate the nose_results.html file though | 22:25 |
| fungi | zigo: i'm sure that's most welcome. we have some intermittently-packaged services/utilities which the current maintainers would probably be happy to move into collab-maint or maybe the dpmt/papt or openstack teams | 22:26 |
| zigo | fungi: I want to also package them the same way as the rest of OpenStack ! :) | 22:26 |
| zigo | fungi: Though maybe with a "package from trunk" way... | 22:27 |
| zigo | fungi: For Ocata, that's also one thing we want to start doing: package from trunk. | 22:27 |
| zigo | Also building for Xenial. | 22:27 |
| zigo | (Ubuntu + Debian builds from the same sources) | 22:27 |
| openstackgerrit | Merged openstack-infra/project-config: Allow to sign deb-openstack-pkg-tools https://review.openstack.org/380485 | 22:28 |
| fungi | zigo: that would probably be fine as well. i think hasharAway, pabelanger and SpamapS are maybe the people to confirm with | 22:28 |
| mtreinish | mriedem: dunno, it's been long enough I don't remember where that comes from | 22:28 |
| mtreinish | mriedem: just switch it to run ostestr and call it a day | 22:29 |
| mriedem | oh i think i see the issue | 22:29 |
| mriedem | there are like 10 nose*html* forks | 22:29 |
| mriedem | i was using the wrong fork | 22:29 |
| hasharAway | zigo: I have yet to follow up on the debian/ci mail you sent on one of the list :/ I guess I get too many mails | 22:29 |
| *** gyee has quit IRC | 22:30 | |
| *** ijw has joined #openstack-infra | 22:31 | |
| hasharAway | zigo: I got Nodepool && Zuul packaged and get them running on Jessie box in production. But the packages are hacky/not so nice :/ | 22:31 |
| hasharAway | probably easier to start from scratch | 22:31 |
| hasharAway | anyway it is too late for me :/ gotta sleep sorry | 22:32 |
| *** cardeois_ has quit IRC | 22:33 | |
| *** sjmc7 has quit IRC | 22:34 | |
| *** hasharAway has quit IRC | 22:34 | |
| *** amitgandhinz has joined #openstack-infra | 22:34 | |
| *** ijw has quit IRC | 22:35 | |
| fungi | clarkb: mtreinish: okay, now worker01 is the only one configured to report into mqtt and the other 19 are just doing normal elasticsearch output. for the moment we seem to have our full compliment of 80 worker processes still running | 22:38 |
| *** Sukhdev has quit IRC | 22:39 | |
| *** amitgandhinz has quit IRC | 22:39 | |
| mtreinish | fungi: oh, I wonder if it's something with the client_id. I think by default the mqtt plugin will use the same client_id | 22:42 |
| mtreinish | I was just thinking about what could be breaking things when there is >1 plugin running | 22:42 |
| mtreinish | err, maybe not the code says it's generated automaticall | 22:43 |
| mtreinish | y | 22:43 |
| *** xyang1 has quit IRC | 22:43 | |
| mtreinish | that's easy enough to see in the mosquitto logs anyway | 22:43 |
| fungi | also we can scale this up a few workers at a time and see what happens once we have a steady state confirmed | 22:43 |
| fungi | but for the moment, none have died yet | 22:43 |
| mtreinish | I also don't know why reusing the client id would break anything, it's just an identifer and only really means anything for subcriptions | 22:43 |
| anteaya | mtreinish: maybe your new nick should be chaos monkey? | 22:48 |
| *** thiagolib has quit IRC | 22:48 | |
| *** ijw has joined #openstack-infra | 22:52 | |
| anteaya | the logstash job queue is now a gently rounded curve | 22:54 |
| *** rbrndt has quit IRC | 22:55 | |
| fungi | clarkb: mtreinish: datapoint... we still have all our worker processes _except_ those on worker01, which have all died | 22:55 |
| fungi | that was the only server with the mqtt plugin still enabled | 22:56 |
| fungi | i restarted them all at the same time | 22:56 |
| clarkb | interesting | 22:56 |
| *** ijw has quit IRC | 22:56 | |
| fungi | so something about the mqtt plugin is almost definitely destabilizing the workers | 22:57 |
| fungi | i'll restart them on 01 again and see if they die again | 22:57 |
| *** salv-orlando has quit IRC | 22:58 | |
| anteaya | funny it would keep on alive out of a cluster | 23:01 |
| anteaya | yet die when it is alone | 23:01 |
| anteaya | s/on/one | 23:01 |
| anteaya | maybe it needs two, one to get rid of? | 23:02 |
| fungi | yep, completely dead on worker01 again, already. the other 19 servers are still all fine | 23:02 |
| *** lezbar has quit IRC | 23:03 | |
| *** lezbar has joined #openstack-infra | 23:03 | |
| *** rockyg has quit IRC | 23:05 | |
| anteaya | I'm interested in seeing what happens if you give the plugin two nodes, if you feel like trying it | 23:05 |
| anteaya | but given it is getting into Friday evening I'm guessing you might select the "no" option | 23:06 |
| fungi | yeah, i think i'm going to try to transition into a friday night mode here | 23:06 |
| anteaya | understandable | 23:06 |
| anteaya | have a good evening | 23:08 |
| *** mriedem has quit IRC | 23:10 | |
| *** hongbin has quit IRC | 23:10 | |
| mtreinish | fungi: sigh, I guess this means I'll have to setup a local logstash to test this more and figure out how it breaks things | 23:11 |
| mtreinish | or just save it for a future project :) | 23:11 |
| *** Apoorva_ has quit IRC | 23:11 | |
| *** Apoorva has joined #openstack-infra | 23:12 | |
| *** ijw has joined #openstack-infra | 23:12 | |
| asselin_ | mtreinish, I need some python help. I'm trying to load a class dynamically, e.g. a class name defined in a conf file. Anyone have pointers to existing code that does that? (This if for e-r) | 23:13 |
| *** fkautz_ is now known as fkautz | 23:13 | |
| *** fkautz has joined #openstack-infra | 23:13 | |
| mgagne | asselin_: I think this can help: http://docs.openstack.org/developer/oslo.utils/api/importutils.html | 23:14 |
| asselin_ | mgagne, cools this...will try it out | 23:14 |
| asselin_ | s/this/thanks/ | 23:14 |
| pabelanger | So, it looks like http://git.openstack.org/cgit/openstack/openstack-manuals/tree/www/index.html is the index page of docs.o.o | 23:15 |
| mgagne | asselin_: there is an example in Nova: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L339 | 23:15 |
| pabelanger | guess we need to figure out how to make that the default for afsdocs now | 23:15 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Make size a graph https://review.openstack.org/380670 | 23:15 |
| jeblair | masayukig: ^ what do you think of that? it's a little ascii-art graph instead of numbers for the size column. | 23:16 |
| jeblair | fungi: ^ friday night fun | 23:16 |
| *** salv-orlando has joined #openstack-infra | 23:16 | |
| jeblair | masayukig: (we could make that a config-file option if some folks prefer numbers, or even not having the size column at all) | 23:17 |
| pabelanger | jeblair: it looks like we might need to vcsrepo openstack-manuals to files.o.o and update apache to use http://git.openstack.org/cgit/openstack/openstack-manuals/tree/www as the base directory | 23:19 |
| pabelanger | if I am understanding the current setup correctly | 23:19 |
| *** ijw has quit IRC | 23:19 | |
| jeblair | pabelanger: i am fairly certain that is not happening on the current site. | 23:19 |
| clarkb | we should be able to have a job against openstack-manuals that published that properly if we don't already | 23:20 |
| jeblair | pretty sure we do | 23:20 |
| *** Goneri has quit IRC | 23:20 | |
| pabelanger | sure, let me check JJB | 23:20 |
| jeblair | pabelanger: http://git.openstack.org/cgit/openstack/openstack-manuals/tree/tools/publishdocs.sh looks relevant | 23:25 |
| jeblair | pabelanger: but also, there is a readme in that dir saying to use openstack-doc-tools | 23:25 |
| jeblair | but i think the publishdocs script in openstack-manuals may still be being used | 23:26 |
| pabelanger | Ya, I'm just cloning openstack-manuals now | 23:26 |
| *** sdake has joined #openstack-infra | 23:29 | |
| masayukig | jeblair : oh, yeah, I like this :) And I agree with the config option. | 23:30 |
| *** kaisers has joined #openstack-infra | 23:30 | |
| *** kaisers3 has quit IRC | 23:31 | |
| *** kaisers2 has quit IRC | 23:31 | |
| *** kaisers1 has joined #openstack-infra | 23:32 | |
| *** ijw has joined #openstack-infra | 23:35 | |
| *** amitgandhinz has joined #openstack-infra | 23:35 | |
| pabelanger | okay, figure it out | 23:37 |
| pabelanger | openstack-manuals-tox-doc-publishdocs should be the job | 23:37 |
| pabelanger | there are 3 in post right now | 23:37 |
| pabelanger | will see what happens | 23:37 |
| mgagne | I found a curious behavior with Gerrit (with a private instance). I cannot push some tags (from nova repository), Gerrit rejects with "prohibited by Gerrit". I push a second time and it goes through just fine. =) | 23:43 |
| jeblair | mgagne: gerrit beer goggles? :) | 23:44 |
| mgagne | I'm not sure what to think about it =) | 23:44 |
| mgagne | implementing a retry mechanism around git push --tags would be silly | 23:45 |
| jeblair | mgagne: i wonder if there's anything interesting in the gerrit error log | 23:46 |
| jeblair | pabelanger: why are they queued for so long? | 23:46 |
| pabelanger | jeblair: waiting for xenial nodes, booting from rackspace | 23:47 |
| pabelanger | we also just uploaded images to rax | 23:47 |
| mgagne | that's the first place I looked and nothing interesting in it. At first, I thought it was because they were plain or annotted tag so I added the pushTag and pushAnnottedTag permissions to "refs/tags/*". Not much luck. | 23:47 |
| jeblair | pabelanger: this job seems to run all the time; one finished just a few minutes ago | 23:47 |
| *** salv-orlando has quit IRC | 23:48 | |
| *** yamamoto has joined #openstack-infra | 23:48 | |
| pabelanger | Oh, I didn't see that | 23:48 |
| pabelanger | currently following b53b76d | 23:48 |
| jeblair | pabelanger: oh, wait, i think they may be in a restart loop | 23:48 |
| pabelanger | Oh | 23:48 |
| pabelanger | neat | 23:48 |
| *** kaisers_ has joined #openstack-infra | 23:49 | |
| *** claudiub|2 has joined #openstack-infra | 23:49 | |
| mgagne | or maybe there is some silly size limit in Gerrit I'm not aware of | 23:49 |
| jeblair | (next time we update the zuul launcher, we should really have it fill in the Worker information so we can track things in the main zuul log easier) | 23:49 |
| *** xarses has quit IRC | 23:49 | |
| jeblair | mgagne: i think there may be size limits | 23:49 |
| mgagne | but that would also impact initial upstream import with jeepby, unless it implements a different logic :-/ | 23:50 |
| *** furlongm has quit IRC | 23:50 | |
| mgagne | will take a look | 23:50 |
| jeblair | mgagne: yeah, i wouldn't expect them on pushes | 23:50 |
| *** mriedem has joined #openstack-infra | 23:50 | |
| *** claudiub has joined #openstack-infra | 23:52 | |
| *** furlongm has joined #openstack-infra | 23:52 | |
| jeblair | pabelanger: File "/usr/local/lib/python2.7/dist-packages/zuul/launcher/ansiblelaunchserver.py", line 1140, in _makeAFSTask | 23:52 |
| jeblair | (afstarget,)) | 23:52 |
| jeblair | Exception: Target path / is not below site root | 23:52 |
| jeblair | that's what's causing the loop | 23:52 |
| pabelanger | Ha | 23:52 |
| pabelanger | we had that issue before | 23:53 |
| *** kaisers_ has quit IRC | 23:53 | |
| jeblair | we did? | 23:53 |
| pabelanger | Ya, for ftp I think | 23:53 |
| *** incog has joined #openstack-infra | 23:53 | |
| pabelanger | looking for commit with fix | 23:54 |
| jeblair | pabelanger: hrm we should have the same logic with the same inputs for ftp vs afs | 23:54 |
| pabelanger | Hmm | 23:54 |
| *** claudiub|2 has quit IRC | 23:55 | |
| pabelanger | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-06-15.log.html#t2016-06-15T16:59:16 | 23:55 |
| pabelanger | let me see how we fixed it | 23:55 |
| *** dkehn has quit IRC | 23:55 | |
| pabelanger | https://review.openstack.org/#/c/330090/ | 23:56 |
| jeblair | pabelanger: oh we may have missed that? | 23:56 |
| pabelanger | ya | 23:56 |
| pabelanger | checking now | 23:56 |
| jeblair | yeah i don't see an lstrip there | 23:56 |
| pabelanger | cool, easy fix | 23:57 |
| jeblair | pabelanger: you want to do it? | 23:57 |
| pabelanger | sure | 23:57 |
| pabelanger | 1 sec | 23:57 |
| *** thorst has joined #openstack-infra | 23:58 | |
| *** baoli has quit IRC | 23:58 | |
| jeblair | pabelanger: indeed, it looks like you were right, i made the same mistake again :) that's a neat trick to have copied that code and managed to remove such an important bit. | 23:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Strip leading / from afs targets https://review.openstack.org/380673 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!