| *** mtanino has quit IRC | 00:00 | |
| *** sarob has joined #openstack-infra | 00:01 | |
| *** Julien-zte has quit IRC | 00:02 | |
| *** zhurong has quit IRC | 00:02 | |
| *** ddieterly has joined #openstack-infra | 00:08 | |
| *** david-lyle has quit IRC | 00:08 | |
| *** sarob has quit IRC | 00:11 | |
| openstackgerrit | Merged openstack-infra/shade: Ensure per-resource caches work without global cache https://review.openstack.org/358776 | 00:11 |
|---|---|---|
| openstackgerrit | Merged openstack-infra/shade: Move list_server cache to dogpile https://review.openstack.org/358871 | 00:11 |
| *** sarob has joined #openstack-infra | 00:11 | |
| openstackgerrit | Merged openstack-infra/shade: Allow image and flavor by name for create_server https://review.openstack.org/355251 | 00:11 |
| *** dcwangmit01_ has quit IRC | 00:12 | |
| *** dcwangmit01 has joined #openstack-infra | 00:12 | |
| *** chlong has quit IRC | 00:13 | |
| *** yolanda has joined #openstack-infra | 00:13 | |
| *** gouthamr_ has joined #openstack-infra | 00:13 | |
| *** sarob has quit IRC | 00:16 | |
| *** gouthamr has quit IRC | 00:16 | |
| *** roxanagh_ has joined #openstack-infra | 00:19 | |
| *** sdague has joined #openstack-infra | 00:22 | |
| sdague | hey, so the generated at time for http://status.openstack.org/elastic-recheck/data/integrated_gate.html still seems off, I think that's about 4 hours ago | 00:24 |
| sdague | I wonder if the cron job is still failing for other reasons | 00:26 |
| *** chlong has joined #openstack-infra | 00:26 | |
| *** sputnik13 has quit IRC | 00:30 | |
| pabelanger | sdague: I think pleia2 and mtreinish might have been working on it today. Not sure the outcome | 00:30 |
| sdague | pabelanger: they were, and we got at least one manual run | 00:30 |
| pleia2 | I'll have a look again | 00:31 |
| sdague | pleia2: awesome, thank you | 00:31 |
| sdague | I wasn't sure who was still around | 00:31 |
| clarkb | I have seen older generators get stuck holding the log file | 00:32 |
| sdague | http://status.openstack.org/elastic-recheck/data/others.html is still on the 18th | 00:34 |
| sdague | which means at least something is not updating that bit correctly as well | 00:34 |
| pleia2 | Generated at: 2016-08-24T20:59 | 00:34 |
| pleia2 | where do you see 18th? | 00:34 |
| sdague | pleia2: the other pge | 00:35 |
| sdague | page | 00:35 |
| sdague | Other Projects Uncategorized | 00:35 |
| pleia2 | yeah, that's what I'm looking at | 00:35 |
| sdague | which, honestly, I don't really care about that much, it just seems to indicate some issues in running behind | 00:35 |
| sdague | oh... gah, this caches really weirdly | 00:36 |
| sdague | I had to force flush my cache | 00:36 |
| pleia2 | weird | 00:36 |
| sdague | ok, 20:59, yep | 00:36 |
| pleia2 | still, that's old | 00:36 |
| pleia2 | running it manually again to see if there are any errors.. | 00:36 |
| sdague | yep | 00:36 |
| sdague | there is also definitely a template change we landed, which is not showing up | 00:36 |
| sdague | that, ironicly, would highlight the date when it was more than an hour behind :) | 00:37 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Remove q-lbaas from tempest pre-installed stuff. https://review.openstack.org/358258 | 00:37 |
| pleia2 | oh yeah, big traceback | 00:38 |
| sdague | ok, what now? | 00:38 |
| pleia2 | http://paste.openstack.org/show/563160/ | 00:39 |
| pleia2 | the last line goes on and on for pages, just included a sample | 00:39 |
| sdague | oh... that's interesting | 00:41 |
| sdague | the query killed it | 00:41 |
| *** Julien-zte has joined #openstack-infra | 00:42 | |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: drop parens on query, they kill ES https://review.openstack.org/360139 | 00:43 |
| sdague | pleia2: ok, so I think that will fix the stack trace | 00:43 |
| pleia2 | I'll give it a try | 00:43 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add os_monasca repository to OpenStack-Ansible https://review.openstack.org/360074 | 00:43 |
| *** raunak has quit IRC | 00:44 | |
| sdague | orthoginal to that, I'm not sure why puppet isn't applying the template changes - https://review.openstack.org/#/c/359932/ | 00:44 |
| sdague | which merged a number of hours ago | 00:44 |
| pleia2 | sdague: that patch didn't seem to go far enough, still dying | 00:48 |
| sdague | hmmm | 00:48 |
| *** markvoelker has joined #openstack-infra | 00:55 | |
| *** Julien-zte has quit IRC | 00:59 | |
| *** gyee has quit IRC | 00:59 | |
| sdague | pleia2: I wonder if I need to quote the cinder part, I'm poking locally to see if I can get a clean run | 00:59 |
| pleia2 | ok | 00:59 |
| sdague | unrelated, can you see if the git tree for CDing updated after https://review.openstack.org/#/c/359932/ hit? | 01:00 |
| *** yanyanhu has joined #openstack-infra | 01:00 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: drop parens on query, they kill ES https://review.openstack.org/360139 | 01:00 |
| pleia2 | yep, let's see | 01:01 |
| pleia2 | sdague: it's definitely on disk, but the timestamp is 21:01 and since the integrated_gate.html template hasn't been applied since then it hasn't gone live | 01:02 |
| pleia2 | darn 2 minutes | 01:03 |
| sdague | ah... ok | 01:03 |
| sdague | well, that is at least reasuring | 01:03 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: projects must be quoted https://review.openstack.org/360148 | 01:03 |
| sdague | that has not *yet* failed for me locally | 01:04 |
| *** pvaneck has quit IRC | 01:04 | |
| sdague | but... it doesn't really give much of a progress report | 01:04 |
| pleia2 | heh, yeah | 01:04 |
| sdague | I'll have to yell at mtreinish about that :) | 01:04 |
| *** Julien-zte has joined #openstack-infra | 01:04 | |
| *** shashank_hegde has quit IRC | 01:05 | |
| * pleia2 patpat ES servers | 01:05 | |
| *** sdake has quit IRC | 01:06 | |
| sdague | I honestly think that patch might merge before I get a local return here | 01:07 |
| pleia2 | it is quite a query | 01:08 |
| *** thorst has quit IRC | 01:09 | |
| *** thorst has joined #openstack-infra | 01:10 | |
| anteaya | there is currently nothing in the check queue older than 3 hours | 01:10 |
| *** Sukhdev has quit IRC | 01:10 | |
| anteaya | so yay working ipv6 | 01:10 |
| *** jamielennox is now known as jamielennox|away | 01:11 | |
| *** jamielennox|away is now known as jamielennox | 01:11 | |
| asselin__ | anyone know why 'openstack server list' would be very very slow recently? http://paste.openstack.org/show/563163/ | 01:12 |
| sdague | yeh, the gate is burning down nicely now | 01:13 |
| *** raunak has joined #openstack-infra | 01:13 | |
| *** caowei has joined #openstack-infra | 01:14 | |
| anteaya | yay! | 01:14 |
| asselin__ | I guess it's something to do between python-openstackclient==2.6.0 and python-openstackclient==3.0.1 | 01:16 |
| *** zhurong has joined #openstack-infra | 01:16 | |
| *** yolanda has quit IRC | 01:18 | |
| *** ddieterly has quit IRC | 01:18 | |
| *** thorst has quit IRC | 01:19 | |
| asselin__ | so much for backwards compatibility: Unable to establish connection to https://10.245.78.7:5000/v2.0/tokens | 01:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 01:21 |
| anteaya | asselin__: mordred might know more about what is happening with openstackclient | 01:21 |
| asselin__ | anteaya, thanks...I find it odd that there's a deprecation notice coupled with it failing to work altogether on my cloud. :( | 01:23 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: WIP: enable more reasonable logging https://review.openstack.org/360164 | 01:24 |
| sdague | first pass on some logging tweaks that might make things better later, I'll sync with mtreinish in the morning | 01:24 |
| sdague | have a good night folks | 01:24 |
| pleia2 | night sdague | 01:24 |
| sdague | pleia2: thanks again for looking into the fails | 01:24 |
| pleia2 | sure thing | 01:24 |
| anteaya | asselin__: :( | 01:24 |
| anteaya | sdague: night | 01:25 |
| asselin__ | mordred, if you care, here's the error output with pip freeze on a box that doesn't work: http://paste.openstack.org/show/563164/ and another box setup a week or 2 ago with it working: http://paste.openstack.org/show/563165/ | 01:25 |
| *** Apoorva has quit IRC | 01:29 | |
| *** yolanda has joined #openstack-infra | 01:31 | |
| *** tphummel has quit IRC | 01:32 | |
| *** baoli has joined #openstack-infra | 01:32 | |
| *** fguillot has quit IRC | 01:33 | |
| *** roxanagh_ has quit IRC | 01:34 | |
| *** raunak has quit IRC | 01:35 | |
| *** markvoelker has quit IRC | 01:36 | |
| *** sdake has joined #openstack-infra | 01:36 | |
| *** yolanda has quit IRC | 01:37 | |
| *** sdague has quit IRC | 01:38 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Split multinode.yaml template https://review.openstack.org/360168 | 01:38 |
| *** sdake_ has joined #openstack-infra | 01:39 | |
| *** sdake has quit IRC | 01:42 | |
| *** kgiusti is now known as kgiusti_afk | 01:44 | |
| *** fguillot has joined #openstack-infra | 01:45 | |
| amrith | good evening folks, is anyone else reporting issues logging in to gerrit; launchpad's auth system has been timing out for the past hour or so | 01:46 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Split multinode.yaml template https://review.openstack.org/360168 | 01:46 |
| amrith | login.launchpad.net | 01:46 |
| *** yolanda has joined #openstack-infra | 01:47 | |
| * amrith cracks open a bottle of wine and waits for login.launchpad.net ... anyone want to come join me? | 01:48 | |
| *** kgiusti_afk has left #openstack-infra | 01:48 | |
| *** ddieterly has joined #openstack-infra | 01:49 | |
| *** shashank_hegde has joined #openstack-infra | 01:49 | |
| *** sdake_ has quit IRC | 01:54 | |
| *** mtanino has joined #openstack-infra | 01:58 | |
| fungi | it's beer over here, but yes | 01:58 |
| *** thorst has joined #openstack-infra | 01:59 | |
| fungi | at casa del fungi we've reached the beer 'n' video games portion of this evening's program, already in progress | 01:59 |
| * fungi gets back to it | 01:59 | |
| *** thorst has quit IRC | 01:59 | |
| *** thorst has joined #openstack-infra | 02:00 | |
| clarkb | I have a wrird ginger beer | 02:00 |
| clarkb | its not a proper ginger beer. It is real beer with ginger in it | 02:00 |
| *** ddieterly has quit IRC | 02:02 | |
| *** yamahata has quit IRC | 02:03 | |
| *** sdake has joined #openstack-infra | 02:03 | |
| openstackgerrit | Xiang Chen proposed openstack/diskimage-builder: Explain difference between two envvar https://review.openstack.org/345935 | 02:04 |
| *** ddieterly has joined #openstack-infra | 02:04 | |
| *** gouthamr_ is now known as gouthamr | 02:05 | |
| fungi | clarkb: still sounds marvellous | 02:05 |
| openstackgerrit | Xiang Chen proposed openstack/diskimage-builder: Explain difference between two envvars https://review.openstack.org/345935 | 02:06 |
| *** thorst has quit IRC | 02:08 | |
| *** zhurong_ has joined #openstack-infra | 02:09 | |
| *** zhurong has quit IRC | 02:11 | |
| *** vinaypotluri has quit IRC | 02:11 | |
| amrith | Hey clarkb, looks like it works again | 02:14 |
| amrith | hi fungi | 02:14 |
| amrith | managed to push a review https://review.openstack.org/360175 | 02:15 |
| amrith | so all is good for now | 02:15 |
| amrith | time to call it a night | 02:15 |
| *** shashank_hegde has quit IRC | 02:15 | |
| *** yolanda has quit IRC | 02:17 | |
| clarkb | amrith: fwiw puahing code shouldnt require openid | 02:18 |
| clarkb | its either your ssh key or your http password key thing that is sued depending on protocol | 02:19 |
| *** raunak has joined #openstack-infra | 02:20 | |
| amrith | clarkb, sorry. i needed to look at reviews (open reviews) before I could push a patch set. | 02:20 |
| *** yolanda has joined #openstack-infra | 02:21 | |
| *** jcook has quit IRC | 02:22 | |
| *** jcook has joined #openstack-infra | 02:22 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Tidy up absent rule https://review.openstack.org/333082 | 02:23 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Tidy up zuul-env-reqs https://review.openstack.org/199833 | 02:23 |
| *** thorst has joined #openstack-infra | 02:25 | |
| *** rackertom has quit IRC | 02:25 | |
| amrith | wanted to make sure that a couple of patches had landed before I marked this interim release for the client. | 02:25 |
| amrith | OK, now, really ... g'night | 02:26 |
| *** simonmcc has quit IRC | 02:26 | |
| jhesketh | craige: I was just about to update the infra agenda to discuss vagrant files next meeting.. but perhaps we should just kick another thread off on openstack-infra? | 02:26 |
| craige | Sounds plan jhesketh | 02:27 |
| *** rackertom has joined #openstack-infra | 02:27 | |
| craige | s/sounds/sound/ | 02:27 |
| *** thiagolib has quit IRC | 02:27 | |
| jhesketh | craige: want me to kick that off? | 02:27 |
| craige | Why not goo sir. | 02:27 |
| *** simonmcc has joined #openstack-infra | 02:28 | |
| craige | i'm about to knock off for lunch (which really just means working on the farm instead of the computer) | 02:28 |
| *** thiagolib has joined #openstack-infra | 02:28 | |
| *** thorst has quit IRC | 02:28 | |
| jhesketh | craige: no worries, have fun :-0 | 02:28 |
| jhesketh | *:-) | 02:28 |
| *** serverascode has quit IRC | 02:28 | |
| craige | Thanks jhesketh | 02:29 |
| *** thcipriani is now known as thcipriani|afk | 02:29 | |
| *** mtanino has quit IRC | 02:31 | |
| *** serverascode has joined #openstack-infra | 02:31 | |
| *** ddieterly has quit IRC | 02:31 | |
| *** sdake has quit IRC | 02:34 | |
| *** sdake has joined #openstack-infra | 02:35 | |
| *** julim has quit IRC | 02:35 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-gerrit: Create index on fresh install https://review.openstack.org/359683 | 02:37 |
| *** ddieterly has joined #openstack-infra | 02:37 | |
| ianw | zaro: ^ as discussed | 02:37 |
| *** ddieterly has quit IRC | 02:37 | |
| *** bswartz has quit IRC | 02:38 | |
| *** Wei_Liu has quit IRC | 02:43 | |
| *** fguillot has quit IRC | 02:43 | |
| *** Wei_Liu has joined #openstack-infra | 02:43 | |
| *** jamesdenton has joined #openstack-infra | 02:44 | |
| *** sdake has quit IRC | 02:45 | |
| *** sdake has joined #openstack-infra | 02:47 | |
| *** dimtruck is now known as zz_dimtruck | 02:48 | |
| *** yuanying has quit IRC | 02:49 | |
| mordred | asselin__: I do care, and thanks! today kind of fell apart, I'll look tomorrow | 02:50 |
| *** raunak has quit IRC | 02:51 | |
| *** yuanying has joined #openstack-infra | 02:53 | |
| *** changzhi has joined #openstack-infra | 02:55 | |
| *** gildub has quit IRC | 02:59 | |
| *** sarob has joined #openstack-infra | 02:59 | |
| *** gildub has joined #openstack-infra | 03:01 | |
| *** sarob has quit IRC | 03:02 | |
| *** shashank_hegde has joined #openstack-infra | 03:02 | |
| *** sarob has joined #openstack-infra | 03:02 | |
| *** sflanigan has quit IRC | 03:04 | |
| *** raunak has joined #openstack-infra | 03:09 | |
| *** jamielennox is now known as jamielennox|away | 03:09 | |
| *** woodster_ has quit IRC | 03:09 | |
| *** sflanigan has joined #openstack-infra | 03:10 | |
| *** sflanigan has joined #openstack-infra | 03:10 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add osic-cloud8 to grafana https://review.openstack.org/360198 | 03:11 |
| *** sdake has quit IRC | 03:12 | |
| *** tonytan4ever has quit IRC | 03:13 | |
| *** hughhalf has quit IRC | 03:16 | |
| *** hughhalf has joined #openstack-infra | 03:16 | |
| *** salv-orlando has joined #openstack-infra | 03:17 | |
| asselin__ | mordred, sorry...after more investigation, I can't even ping that IP....which is strange since its the same cloud & tenant hosting the VM and the previous box. I'll look at that tomorrow. Just want to let you know there's likely some env issue on my env. I will try launching a new node and see if I can reproduce. | 03:20 |
| *** salv-orl_ has quit IRC | 03:20 | |
| *** chlong has quit IRC | 03:23 | |
| *** jamielennox|away is now known as jamielennox | 03:23 | |
| *** nwkarsten has joined #openstack-infra | 03:24 | |
| *** senk has joined #openstack-infra | 03:25 | |
| *** xcGlQHdZGT has quit IRC | 03:28 | |
| *** thorst has joined #openstack-infra | 03:29 | |
| *** shashank_hegde has quit IRC | 03:29 | |
| *** dkehn_ has quit IRC | 03:31 | |
| *** dkehn has quit IRC | 03:31 | |
| *** Julien-zte has quit IRC | 03:32 | |
| *** chlong has joined #openstack-infra | 03:35 | |
| *** thorst has quit IRC | 03:37 | |
| *** vikrant has joined #openstack-infra | 03:38 | |
| *** dkehn has joined #openstack-infra | 03:38 | |
| *** bhunter71 has quit IRC | 03:38 | |
| *** chlong has quit IRC | 03:38 | |
| *** chlong has joined #openstack-infra | 03:38 | |
| *** rlandy|bbl is now known as rlandy | 03:39 | |
| *** rlandy has quit IRC | 03:40 | |
| *** Julien-zte has joined #openstack-infra | 03:41 | |
| *** changzhi has quit IRC | 03:41 | |
| *** vinaypotluri has joined #openstack-infra | 03:42 | |
| *** raunak has quit IRC | 03:42 | |
| *** dkehn_ has joined #openstack-infra | 03:43 | |
| *** gongysh has joined #openstack-infra | 03:43 | |
| *** senk has quit IRC | 03:43 | |
| *** baoli has quit IRC | 03:44 | |
| *** yuanying has quit IRC | 03:45 | |
| *** raunak has joined #openstack-infra | 03:46 | |
| *** sarob has quit IRC | 03:47 | |
| mordred | asselin__: cool. well, I certainly don't want new code to break you | 03:47 |
| *** armax has quit IRC | 03:47 | |
| *** yuanying has joined #openstack-infra | 03:48 | |
| *** coolsvap_ has joined #openstack-infra | 03:50 | |
| *** bhunter71 has joined #openstack-infra | 03:51 | |
| *** raunak has quit IRC | 03:57 | |
| *** raunak has joined #openstack-infra | 03:58 | |
| *** hqkLChMUKc has joined #openstack-infra | 04:02 | |
| *** jogo has quit IRC | 04:06 | |
| *** shashank_hegde has joined #openstack-infra | 04:06 | |
| *** amotoki has joined #openstack-infra | 04:06 | |
| *** mtanino has joined #openstack-infra | 04:08 | |
| *** raunak has quit IRC | 04:12 | |
| *** bswartz has joined #openstack-infra | 04:12 | |
| *** tonytan4ever has joined #openstack-infra | 04:13 | |
| stevemar | clarkb: not sure what i should do about this bug: https://bugs.launchpad.net/keystone/+bug/1602407 | 04:18 |
| openstack | Launchpad bug 1602407 in OpenStack Identity (keystone) "MySQL tests significantly slower than PostgreSQL tests" [Undecided,New] | 04:18 |
| *** tonytan4ever has quit IRC | 04:18 | |
| stevemar | clarkb: apparently mysql tests are always slower than postgres tests :P | 04:18 |
| *** senk has joined #openstack-infra | 04:18 | |
| asselin__ | mordred, ok, I setup a new env using the same setup script. Here's the error AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult' Sanitized debug output for openstack, ansible & pip output http://paste.openstack.org/show/563229/ | 04:22 |
| *** gongysh has quit IRC | 04:25 | |
| asselin__ | ping works this time. for the ansible openstack script I'm using https://github.com/ansible/ansible commit acc8a13eff73d27977525e04454fe7e904484e6f | 04:26 |
| asselin__ | . Basically following http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/puppetmaster.pp, but using ansible==2.1.0.0 | 04:26 |
| *** raunak has joined #openstack-infra | 04:29 | |
| *** Ravikiran_K has joined #openstack-infra | 04:29 | |
| *** thorst has joined #openstack-infra | 04:35 | |
| *** links has joined #openstack-infra | 04:37 | |
| *** nwkarsten has quit IRC | 04:38 | |
| *** gouthamr has quit IRC | 04:39 | |
| *** caowei has quit IRC | 04:39 | |
| *** thorst has quit IRC | 04:42 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Revert "Store ssh_host_key of remote node" https://review.openstack.org/360221 | 04:43 |
| pabelanger | ianw: jhesketh: do you mind a quick +A^ | 04:43 |
| pabelanger | reverts zuul patch does is broken | 04:43 |
| jhesketh | pabelanger: looking | 04:43 |
| pabelanger | just tested it on zlstatic01.o.o | 04:43 |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: congress: remove new_arch jobs https://review.openstack.org/360222 | 04:44 |
| pabelanger | http://paste.openstack.org/show/563235/ is the failure | 04:44 |
| pabelanger | looks like and issue with the args passed to ssh-keygen | 04:45 |
| jhesketh | pabelanger: +w | 04:45 |
| pabelanger | thanks | 04:46 |
| *** psachin has joined #openstack-infra | 04:48 | |
| AJaeger | good morning! | 04:48 |
| AJaeger | pabelanger: you're still here? What miracles are you doing? ;) | 04:49 |
| *** psachin has quit IRC | 04:49 | |
| AJaeger | jhesketh: good you review https://review.openstack.org/#/c/358462/ , please? | 04:49 |
| jhesketh | AJaeger: sure, looking | 04:49 |
| jhesketh | +w | 04:50 |
| *** yamahata has joined #openstack-infra | 04:50 | |
| pabelanger | AJaeger: Ya, about to call it a night. Have sick kids tonight, offered to say up a little later in case they wake up with issues. | 04:50 |
| pabelanger | okay, figure out ssh-keygen issue. Will upload a patch in the morning. | 04:52 |
| pabelanger | night | 04:52 |
| *** mtanino has quit IRC | 04:53 | |
| *** senk has quit IRC | 04:54 | |
| AJaeger | pabelanger: hope your kids get well soon! | 04:58 |
| AJaeger | thanks, jhesketh | 04:58 |
| AJaeger | pabelanger: thanks for looking at hte ssh-keygen | 04:59 |
| *** raunak has quit IRC | 04:59 | |
| *** raunak has joined #openstack-infra | 05:00 | |
| openstackgerrit | Merged openstack-infra/zuul: Revert "Store ssh_host_key of remote node" https://review.openstack.org/360221 | 05:02 |
| *** scottda has quit IRC | 05:06 | |
| *** sarob has joined #openstack-infra | 05:09 | |
| *** jaosorior has joined #openstack-infra | 05:10 | |
| AJaeger | project-config cores, could you review https://review.openstack.org/359656 , please? This removes an unneeded post job | 05:13 |
| *** sarob has quit IRC | 05:14 | |
| *** psachin has joined #openstack-infra | 05:14 | |
| AJaeger | therve: is https://review.openstack.org/#/c/317817/ ready now? | 05:14 |
| *** raunak has quit IRC | 05:17 | |
| *** Sukhdev has joined #openstack-infra | 05:20 | |
| *** baoli has joined #openstack-infra | 05:21 | |
| AJaeger | #success Devstack runs fine on IPv6 only nodes | 05:22 |
| openstackstatus | AJaeger: Added success to Success page | 05:22 |
| *** senk has joined #openstack-infra | 05:23 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config: Adding check-requirements for log-api https://review.openstack.org/360248 | 05:23 |
| *** cody-somerville has joined #openstack-infra | 05:25 | |
| *** harlowja_at_home has joined #openstack-infra | 05:25 | |
| *** ilyashakhat has joined #openstack-infra | 05:26 | |
| *** baoli has quit IRC | 05:27 | |
| ianw | i don't know why, but fedora-24 hasn't updated in rax-iad for quite a few days | 05:27 |
| ianw | uploading image 921 now ... see what happens ... | 05:30 |
| openstackgerrit | Aleksandr Dobdin proposed openstack-infra/project-config: add: ansible-like tool https://review.openstack.org/359831 | 05:31 |
| *** _nadya_ has joined #openstack-infra | 05:31 | |
| *** kaisers_ has joined #openstack-infra | 05:32 | |
| *** ilyashakhat has quit IRC | 05:33 | |
| *** _nadya_ has quit IRC | 05:36 | |
| *** cody-somerville has quit IRC | 05:38 | |
| *** thorst has joined #openstack-infra | 05:40 | |
| *** david-lyle has joined #openstack-infra | 05:43 | |
| *** harlowja_at_home has quit IRC | 05:45 | |
| *** thorst has quit IRC | 05:48 | |
| *** ilyashakhat has joined #openstack-infra | 05:48 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Make Undercloud OpenStack service workers configurable https://review.openstack.org/360255 | 05:48 |
| *** simonmcc has quit IRC | 05:51 | |
| *** claudiub has joined #openstack-infra | 05:51 | |
| *** jogo has joined #openstack-infra | 05:52 | |
| *** jogo has quit IRC | 05:52 | |
| *** jogo has joined #openstack-infra | 05:52 | |
| *** simonmcc has joined #openstack-infra | 05:53 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/shade: Delete objname in image_delete https://review.openstack.org/360257 | 05:53 |
| *** aeng has quit IRC | 05:55 | |
| ramishra | AJaeger: hi, therve is on leave this week, yeah, the experimental job is now green http://logs.openstack.org/22/360122/1/experimental/gate-heat-dsvm-dsplugin/775cd01/ | 05:56 |
| ianw | mordred: ^^ this may be wrong. i'm more interested in why it doesn't fail CI. if you have any thoughts to drop on that in the review, i can look at it | 05:57 |
| *** e0ne has joined #openstack-infra | 05:57 | |
| *** sandanar has joined #openstack-infra | 05:58 | |
| *** Sukhdev has quit IRC | 05:59 | |
| *** edtubill has quit IRC | 06:03 | |
| *** adriant has quit IRC | 06:03 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Remove Fedora 24 work-around kernel https://review.openstack.org/353093 | 06:04 |
| *** ccamacho has joined #openstack-infra | 06:04 | |
| *** rajinir has quit IRC | 06:05 | |
| *** david-lyle has quit IRC | 06:09 | |
| *** caowei has joined #openstack-infra | 06:10 | |
| *** Genek has joined #openstack-infra | 06:14 | |
| Genek | Hi, I'm now working as a intern in Foundation. I'm currently working on improving ask.openstack.org . Now we have 4800+ tags on it where most of than have less than 1 question associate to it. | 06:16 |
| Genek | We're currently working on cleaning those tags. However it'll take a lot of time deleting them using the admin panel. | 06:17 |
| Genek | Is it possible to delete those tags directly on the server? Thanks a lot :) | 06:17 |
| AJaeger | ramishra: thanks for confirming and answering! Would appreciate one or more heat cores to +1 the change. | 06:19 |
| ramishra | AJaeger: I'm one of them and have +1ed it:) | 06:22 |
| *** baoli has joined #openstack-infra | 06:23 | |
| AJaeger | Welcome, Genek! I doubt anybody is around right now that can help you, hope others read it later and answer - but you might need to reach out to some via email. | 06:23 |
| AJaeger | ramishra: understood, just saying I'd like some broader support for the change - so if you could ask one ore more to speak up, I'll +2 if it's fine. | 06:24 |
| AJaeger | ramishra: I remember some discussion about that change, so like to be careful | 06:25 |
| ramishra | AJaeger: np. Have asked a few to have a look. | 06:25 |
| clarkb | Genek I think there is a postgresql database behind ask.openstack.org. Itay be possibly to directly modify that but it depends on the application and db schema | 06:26 |
| clarkb | Genek: I dont know that anyone here knows if it is possible without reading docs and testing it first | 06:26 |
| *** baoli has quit IRC | 06:28 | |
| *** rcernin has joined #openstack-infra | 06:28 | |
| *** VnrycepuxO has quit IRC | 06:31 | |
| *** r-mibu has quit IRC | 06:32 | |
| *** claudiub has quit IRC | 06:33 | |
| Genek | clarkb I think there's a command called "delete_tags", haven't tried it yet. I think I'll set up a test environment for testing. | 06:34 |
| Genek | clarkb: I think there's a command called "delete_tags", haven't tried it yet. I think I'll set up a test environment for testing. | 06:34 |
| *** nikhil has quit IRC | 06:36 | |
| *** nikhil has joined #openstack-infra | 06:38 | |
| *** kushal has quit IRC | 06:39 | |
| *** hqkLChMUKc has quit IRC | 06:39 | |
| *** andreas_s has joined #openstack-infra | 06:40 | |
| *** watersoul has quit IRC | 06:42 | |
| *** watersoul has joined #openstack-infra | 06:42 | |
| *** thorst has joined #openstack-infra | 06:44 | |
| *** watersoul has joined #openstack-infra | 06:47 | |
| *** ihrachys has joined #openstack-infra | 06:50 | |
| *** salv-orlando has quit IRC | 06:51 | |
| *** thorst has quit IRC | 06:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:52 | |
| *** Na3iL has joined #openstack-infra | 06:53 | |
| yolanda | good morning | 06:56 |
| *** ihrachys has quit IRC | 06:56 | |
| *** ilyashakhat has quit IRC | 06:59 | |
| *** senk has quit IRC | 06:59 | |
| *** yaume has joined #openstack-infra | 07:00 | |
| AJaeger | good morning, yolanda ! | 07:00 |
| Genek | good morning! | 07:01 |
| *** florianf has joined #openstack-infra | 07:04 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize certificates in infracloud https://review.openstack.org/359757 | 07:06 |
| *** senk has joined #openstack-infra | 07:08 | |
| *** tesseract- has joined #openstack-infra | 07:08 | |
| *** ifarkas_afk is now known as ifarkas | 07:09 | |
| *** DrifterZA has joined #openstack-infra | 07:11 | |
| *** mikelk has joined #openstack-infra | 07:14 | |
| *** sarob has joined #openstack-infra | 07:14 | |
| *** Ravikiran_K has quit IRC | 07:14 | |
| *** tonytan4ever has joined #openstack-infra | 07:15 | |
| *** kzaitsev_mb has quit IRC | 07:16 | |
| *** sarob has quit IRC | 07:19 | |
| *** tonytan4ever has quit IRC | 07:20 | |
| *** dtardivel has joined #openstack-infra | 07:20 | |
| *** vinaypotluri has quit IRC | 07:21 | |
| *** Genek has quit IRC | 07:23 | |
| *** baoli has joined #openstack-infra | 07:23 | |
| *** Genek has joined #openstack-infra | 07:24 | |
| *** Genek has joined #openstack-infra | 07:25 | |
| *** martinkopec has joined #openstack-infra | 07:25 | |
| *** r-mibu has joined #openstack-infra | 07:27 | |
| *** baoli has quit IRC | 07:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix translation setup for networking-odl https://review.openstack.org/360296 | 07:29 |
| AJaeger | project-config cores, the change above should go in soonish. Please review ^. | 07:30 |
| *** e0ne has quit IRC | 07:32 | |
| *** e0ne has joined #openstack-infra | 07:33 | |
| *** matthewbodkin has joined #openstack-infra | 07:36 | |
| *** yarkot1 has quit IRC | 07:40 | |
| *** jpich has joined #openstack-infra | 07:41 | |
| *** yaume has quit IRC | 07:43 | |
| *** yaume has joined #openstack-infra | 07:44 | |
| *** abregman has joined #openstack-infra | 07:44 | |
| *** shardy has joined #openstack-infra | 07:47 | |
| *** thorst has joined #openstack-infra | 07:50 | |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Add docker-suite-app-murano project to openstack https://review.openstack.org/357745 | 07:50 |
| *** hashar has joined #openstack-infra | 07:52 | |
| *** sflanigan has quit IRC | 07:57 | |
| *** sflanigan has joined #openstack-infra | 07:58 | |
| *** sflanigan has joined #openstack-infra | 07:58 | |
| *** thorst has quit IRC | 07:58 | |
| *** shashank_hegde has quit IRC | 07:58 | |
| *** salv-orl_ has joined #openstack-infra | 07:59 | |
| *** salv-orl_ has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** salv-orl_ has joined #openstack-infra | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Parameterize certificates in infracloud https://review.openstack.org/359757 | 08:00 |
| *** berendt has joined #openstack-infra | 08:00 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Revert "networking-midonet: Consume gate_hook if exists" https://review.openstack.org/360312 | 08:03 |
| *** dizquierdo has joined #openstack-infra | 08:04 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add launcher profiles to infracloud openstackci/openstackzuul oscc clouds https://review.openstack.org/360313 | 08:07 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Revert "networking-midonet: Consume gate_hook if exists" https://review.openstack.org/360312 | 08:08 |
| *** yamamoto has quit IRC | 08:11 | |
| *** flepied1 is now known as flepied | 08:11 | |
| openstackgerrit | Merged openstack-infra/project-config: congress: remove new_arch jobs https://review.openstack.org/360222 | 08:14 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add infracloud openstackci oscc cloud to ansible-clouds.yaml https://review.openstack.org/360319 | 08:16 |
| *** sflanigan has quit IRC | 08:17 | |
| *** skath has quit IRC | 08:18 | |
| *** skath has joined #openstack-infra | 08:19 | |
| *** esikachev has joined #openstack-infra | 08:20 | |
| *** asettle has joined #openstack-infra | 08:20 | |
| *** dingyichen has quit IRC | 08:22 | |
| *** Genek has quit IRC | 08:22 | |
| *** Genek has joined #openstack-infra | 08:22 | |
| *** chem has joined #openstack-infra | 08:22 | |
| *** pgadiya has joined #openstack-infra | 08:23 | |
| *** newcomer25 has joined #openstack-infra | 08:24 | |
| openstackgerrit | Merged openstack-infra/project-config: Update api-site zuul config https://review.openstack.org/359656 | 08:24 |
| *** claudiub has joined #openstack-infra | 08:25 | |
| openstackgerrit | Merged openstack-infra/system-config: Remove rcarrillocruz from infracloud users https://review.openstack.org/360063 | 08:25 |
| *** newcomer25 has left #openstack-infra | 08:26 | |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Add k8s-docker-suite-app-murano project to openstack https://review.openstack.org/357745 | 08:27 |
| *** lucas-dinner is now known as lucasagomes | 08:28 | |
| *** senk has quit IRC | 08:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Adding check-requirements for log-api https://review.openstack.org/360248 | 08:29 |
| *** _nadya_ has joined #openstack-infra | 08:30 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add infracloud oscc to nodepool clouds https://review.openstack.org/360324 | 08:30 |
| *** _nadya_ has quit IRC | 08:30 | |
| *** _nadya_ has joined #openstack-infra | 08:31 | |
| *** jordanP has joined #openstack-infra | 08:32 | |
| *** dtantsur|afk is now known as dtantsur | 08:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Add os_monasca repository to OpenStack-Ansible https://review.openstack.org/360074 | 08:32 |
| openstackgerrit | Merged openstack-infra/project-config: Remove whitespaces and add parenthesis to print statements https://review.openstack.org/360080 | 08:32 |
| openstackgerrit | Merged openstack-infra/project-config: Fix legend for 'Time to Ready' for nodepool dashboard https://review.openstack.org/360113 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config: Fix ready node launch attemps panel for nodepool https://review.openstack.org/360118 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config: Switch ironic-multinode job to wholedisk agent_ssh https://review.openstack.org/359431 | 08:33 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add infracloud openstackci oscc cloud to ansible-clouds.yaml https://review.openstack.org/360319 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config: Add osic-cloud8 to grafana https://review.openstack.org/360198 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config: remove vitrage-release tag rights for vitrage repos https://review.openstack.org/359965 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config: sahara/tempest: run also client tests and a pre script https://review.openstack.org/359920 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config: Move cinder multinode grenade job to check https://review.openstack.org/359275 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config: kuryr-libnetwork-core addition https://review.openstack.org/357945 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config: kuryr-kubernetes-core addition https://review.openstack.org/357946 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config: kuryr: Add fuxi subproject to gerritbot https://review.openstack.org/359899 | 08:34 |
| openstackgerrit | Merged openstack-infra/project-config: Few changes to the kolla-kubernetes job https://review.openstack.org/355199 | 08:34 |
| *** andymaier has joined #openstack-infra | 08:34 | |
| *** Genek has quit IRC | 08:35 | |
| *** psachin has quit IRC | 08:35 | |
| *** ccamacho is now known as ccamacho|afk | 08:37 | |
| *** senk has joined #openstack-infra | 08:38 | |
| *** Julien-zte has quit IRC | 08:41 | |
| *** aarefiev_ has joined #openstack-infra | 08:43 | |
| *** azvyagintsev has joined #openstack-infra | 08:44 | |
| *** zul has joined #openstack-infra | 08:46 | |
| *** dizquierdo has quit IRC | 08:47 | |
| *** alexz__ has quit IRC | 08:47 | |
| *** aarefiev has quit IRC | 08:47 | |
| *** yamamoto has joined #openstack-infra | 08:48 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** openstackgerrit has joined #openstack-infra | 08:48 | |
| *** aarefiev_ is now known as aarefiev | 08:50 | |
| *** bethwhite_ has joined #openstack-infra | 08:51 | |
| *** ihrachys has joined #openstack-infra | 08:52 | |
| *** rkukura has quit IRC | 08:53 | |
| openstackgerrit | Merged openstack-infra/system-config: Add infracloud openstackci oscc cloud to ansible-clouds.yaml https://review.openstack.org/360319 | 08:55 |
| *** Na3iL has quit IRC | 08:55 | |
| *** jordanP has quit IRC | 08:56 | |
| *** thorst has joined #openstack-infra | 08:56 | |
| *** jordanP has joined #openstack-infra | 08:57 | |
| *** yamamoto has quit IRC | 08:57 | |
| *** andreas_s has quit IRC | 08:57 | |
| *** yamamoto has joined #openstack-infra | 08:57 | |
| *** andreas_s has joined #openstack-infra | 08:57 | |
| *** yamamoto has quit IRC | 08:58 | |
| *** yamamoto has joined #openstack-infra | 08:58 | |
| openstackgerrit | Merged openstack-infra/system-config: Add launcher profiles to infracloud openstackci/openstackzuul oscc clouds https://review.openstack.org/360313 | 08:58 |
| openstackgerrit | Merged openstack-infra/system-config: Add infracloud oscc to nodepool clouds https://review.openstack.org/360324 | 08:58 |
| openstackgerrit | Merged openstack-infra/project-config: Fix translation setup for networking-odl https://review.openstack.org/360296 | 08:59 |
| *** yaume has quit IRC | 09:00 | |
| *** yaume_ has joined #openstack-infra | 09:00 | |
| *** derekh has joined #openstack-infra | 09:00 | |
| *** sambetts|afk is now known as sambetts | 09:01 | |
| *** thorst has quit IRC | 09:03 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/360358 | 09:08 |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config: Switch all heat jobs to use devstack plugin https://review.openstack.org/317817 | 09:09 |
| *** ccamacho|afk is now known as ccamacho | 09:14 | |
| ramishra | AJaeger, yolanda , ^^^ rebased the patch as if failed to merge. | 09:15 |
| *** sarob has joined #openstack-infra | 09:15 | |
| *** tonytan4ever has joined #openstack-infra | 09:16 | |
| *** salv-orlando has joined #openstack-infra | 09:18 | |
| *** e0ne has quit IRC | 09:18 | |
| *** sarob has quit IRC | 09:20 | |
| *** salv-orl_ has quit IRC | 09:20 | |
| *** tonytan4ever has quit IRC | 09:20 | |
| *** gildub has quit IRC | 09:21 | |
| *** dkehn has quit IRC | 09:22 | |
| *** dkehn_ has quit IRC | 09:22 | |
| *** dizquierdo has joined #openstack-infra | 09:22 | |
| *** kaisers_ has quit IRC | 09:23 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:26 | |
| *** yamamoto has quit IRC | 09:29 | |
| *** dkehn has joined #openstack-infra | 09:29 | |
| *** e0ne has joined #openstack-infra | 09:31 | |
| *** yamamoto has joined #openstack-infra | 09:32 | |
| *** zul has quit IRC | 09:33 | |
| AJaeger | fungi, thanks for your zuul-env change. Today's translation import looks fine, no zuul-cloner failures! Still waiting for some post jobs to check that the problem is really fixed | 09:34 |
| AJaeger | hashar: could you review https://review.openstack.org/355012 , please? | 09:36 |
| *** kaisers_ has joined #openstack-infra | 09:37 | |
| hashar | AJaeger: don't you need to raise bindep requirement to >=2.1.0 ? | 09:37 |
| hashar | if one runs an older version, it would expect other-requirements.txt which is no more around | 09:38 |
| hashar | but I am nitpicking really | 09:38 |
| *** zul has joined #openstack-infra | 09:38 | |
| hashar | AJaeger: if my comment does not make sense, I will happily CR+2 / W+1 the change | 09:39 |
| *** dkehn_ has joined #openstack-infra | 09:39 | |
| AJaeger | tox will install the latest version. Using >= 2.1.0 wouldn't harm but only few will have it. | 09:40 |
| AJaeger | hashar: good thinking. | 09:40 |
| *** baoli has joined #openstack-infra | 09:40 | |
| AJaeger | yolanda: https://github.com/openstack/deb-python-fixtures is not imported, could you do the usual magic, please? | 09:41 |
| hashar | AJaeger: I am probably overthinking anyway | 09:41 |
| hashar | for openstack CI it is going to be just fine | 09:41 |
| hashar | since the tox env is build fresh on each run | 09:41 |
| hashar | the edge case is when running using system packages (unlikely) | 09:41 |
| AJaeger | hashar: the tox env is never used | 09:41 |
| AJaeger | the tox env is for developers only | 09:41 |
| hashar | or for folks having outdated tox venv in this case they would just have to recreate them with tox -r | 09:42 |
| *** tosky has joined #openstack-infra | 09:42 | |
| AJaeger | the tox env is a one time think when you setup your system. So, nobody has done this already ;) | 09:42 |
| AJaeger | hashar: so, yes, let's merge it, please | 09:43 |
| hashar | I am sometime abusing: apt-get install <list of python modules> ; http_proxy=. https_proxy=. tox --sitepackages | 09:43 |
| hashar | ;D | 09:43 |
| hashar | AJaeger: W+1 ed it :} | 09:44 |
| *** baoli has quit IRC | 09:44 | |
| AJaeger | thanks, hashar | 09:45 |
| *** claudiub|2 has joined #openstack-infra | 09:45 | |
| openstackgerrit | Merged openstack/python-jenkins: Move other-requirements.txt to bindep.txt https://review.openstack.org/355012 | 09:46 |
| *** claudiub has quit IRC | 09:47 | |
| *** mkoderer__ has joined #openstack-infra | 09:48 | |
| hashar | \O/ | 09:48 |
| yolanda | AJaeger, let me take a look | 09:49 |
| hashar | AJaeger: also I poked you about some quota issue you had with OVH a month or so ago. pabelanger got me the answer (OVH fixed the tenant quota manually) | 09:49 |
| yolanda | AJaeger, fixed | 09:51 |
| *** yamamoto has quit IRC | 09:58 | |
| *** yamamoto has joined #openstack-infra | 09:58 | |
| *** yamamoto has quit IRC | 09:58 | |
| *** claudiub|2 has quit IRC | 10:00 | |
| *** thorst has joined #openstack-infra | 10:00 | |
| *** zhurong_ has quit IRC | 10:01 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:01 | |
| *** yanyanhu has quit IRC | 10:02 | |
| *** xarses_ has joined #openstack-infra | 10:02 | |
| *** yamamoto has joined #openstack-infra | 10:02 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 10:03 |
| *** xarses has quit IRC | 10:05 | |
| *** thorst has quit IRC | 10:07 | |
| *** yamamoto has quit IRC | 10:07 | |
| *** yamamoto has joined #openstack-infra | 10:08 | |
| openstackgerrit | Anton Arefiev proposed openstack-infra/project-config: Don't run tempest-dsvm-ironic-inspector on stable/mitaka https://review.openstack.org/360396 | 10:08 |
| *** e0ne has quit IRC | 10:09 | |
| *** Na3iL has joined #openstack-infra | 10:15 | |
| *** electrofelix has joined #openstack-infra | 10:17 | |
| openstackgerrit | Anton Arefiev proposed openstack-infra/project-config: Don't run tempest-dsvm-ironic-inspector on stable/mitaka https://review.openstack.org/360396 | 10:19 |
| *** psachin has joined #openstack-infra | 10:19 | |
| *** chem has quit IRC | 10:22 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Refactor infra-cloud-bridge element to support CentOS/RH https://review.openstack.org/359909 | 10:22 |
| *** chem has joined #openstack-infra | 10:22 | |
| *** hockeynut has quit IRC | 10:23 | |
| *** lucasagomes is now known as lucas-relocate | 10:25 | |
| *** Genek has joined #openstack-infra | 10:27 | |
| *** Genek_ has joined #openstack-infra | 10:29 | |
| *** Genek has quit IRC | 10:29 | |
| mordred | ianw: I think that patch is correct. | 10:29 |
| mordred | ianw: as for CI - it's possible we have poor test coverage for that I'm guessing (I know we don't have any functional tests for that code path, becaue it's not possible to get a devstack installed that actually works in that configuration, so we have to rely on unit tests for that) | 10:30 |
| openstackgerrit | Dominic Cleal proposed openstack-infra/jenkins-job-builder: Add cancel-builds-on-update to GHPRB trigger https://review.openstack.org/360407 | 10:31 |
| *** jed56 has quit IRC | 10:35 | |
| *** gildub has joined #openstack-infra | 10:42 | |
| yolanda | hi mordred, care reviewing a pair of glean patches? https://review.openstack.org/359961, https://review.openstack.org/359982 | 10:47 |
| *** sdague has joined #openstack-infra | 10:51 | |
| *** ccamacho is now known as ccamacho|afk | 10:53 | |
| rcarrillocruz | pabelanger: when you get around, i was getting failurs on the cloud launcher runs, by using /etc/openstack/all-clouds.yaml | 10:53 |
| rcarrillocruz | there was no creds for openstackzuul-bluebox | 10:53 |
| rcarrillocruz | added creds in hiera | 10:53 |
| rcarrillocruz | and now getting failures due to openstackci-tripleo-test-cloud-rh1 not existing, it appears you call it openstack-nodepool-tripleo | 10:54 |
| rcarrillocruz | should I change to that name on clouds_launcher/clouds_layouts.yaml or you want to change it the other way around? | 10:54 |
| odyssey4me | Hi everyone - now that https://review.openstack.org/360074 has merged, please add me to https://review.openstack.org/#/admin/groups/1543,members | 10:55 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: Add query for broken apt mirrors https://review.openstack.org/360432 | 10:55 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: Add query for broken apt mirrors https://review.openstack.org/360432 | 10:56 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 10:56 |
| rcarrillocruz | odyssey4me: done | 10:56 |
| odyssey4me | thanks rcarrillocruz | 10:56 |
| Genek_ | hi rcarrillocruz, I'm currently testing on shade. I've heard from David Flanders that you've implemented the Magnum support. Do you know any public clouds that supports Magnum? Thanks | 10:58 |
| rcarrillocruz | oh yeah, sorry, totally forgot to reply | 10:58 |
| rcarrillocruz | i don't know of any clouds with magnum i'm afraid | 10:58 |
| rcarrillocruz | mordred may know? | 10:58 |
| rcarrillocruz | other than that, you better ask in the openstack magnum channel | 10:58 |
| Genek_ | Thanks for the suggestion | 10:59 |
| *** YorikSar has quit IRC | 11:00 | |
| *** asettle has quit IRC | 11:01 | |
| *** asettle has joined #openstack-infra | 11:02 | |
| *** YorikSar has joined #openstack-infra | 11:02 | |
| *** thorst has joined #openstack-infra | 11:05 | |
| *** ccamacho|afk is now known as ccamacho | 11:05 | |
| AJaeger | thanks, yolanda. | 11:07 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Skip OSA Xenial integrated build jobs for Liberty & Mitaka https://review.openstack.org/360440 | 11:07 |
| *** jkilpatr has quit IRC | 11:07 | |
| *** sandanar_ has joined #openstack-infra | 11:07 | |
| AJaeger | hashar: I saw that - thanks. | 11:07 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Fix rabbitmq install for RHEL/CentOS https://review.openstack.org/360441 | 11:08 |
| * rcarrillocruz is uploading ubuntu-trusty dib image from nodepool to infracloud glance | 11:09 | |
| *** sandanar_ has quit IRC | 11:09 | |
| *** sandanar__ has joined #openstack-infra | 11:09 | |
| *** sdague has quit IRC | 11:10 | |
| yolanda | rcarrillocruz, woot! | 11:10 |
| yolanda | really looking forward to see all that job we've done applied finally | 11:10 |
| *** sandanar has quit IRC | 11:10 | |
| rcarrillocruz | indeed | 11:12 |
| rcarrillocruz | long trip | 11:12 |
| *** thorst has quit IRC | 11:12 | |
| *** lucas-relocate is now known as lucas-hungry | 11:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Plumb ipv6-preferred into force-ipv4 setting https://review.openstack.org/360445 | 11:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic https://review.openstack.org/360446 | 11:14 |
| mordred | rcarrillocruz, Genek_: I am not aware of any public clouds with magnum | 11:15 |
| *** tonytan4ever has joined #openstack-infra | 11:17 | |
| mordred | yolanda: done. I want clarkb to look at https://review.openstack.org/#/c/359982/ | 11:17 |
| yolanda | ok no rush | 11:18 |
| mordred | yolanda: I mean, I'm pretty sure it's fine - but he's mr. bridge man :) | 11:19 |
| yolanda | mordred, so we were not hitting the problem with glean dealing with bridges, because we had it configured under the same vlan config file. But for CentOS, it needs to be on an indepdendent file, and along with Ricardo we decided that should be cleaner to have it on an independent file as well for Debian | 11:19 |
| yolanda | but then glean comes and tries to deal with it :) | 11:20 |
| *** Genek_ has quit IRC | 11:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/360358 | 11:20 |
| *** ramishra has quit IRC | 11:20 | |
| *** rodrigods has quit IRC | 11:21 | |
| *** rodrigods has joined #openstack-infra | 11:21 | |
| openstackgerrit | Merged openstack-infra/glean: Improve the support for checking vlan interfaces https://review.openstack.org/359961 | 11:21 |
| *** tonytan4ever has quit IRC | 11:22 | |
| *** ramishra has joined #openstack-infra | 11:22 | |
| *** Na3iL has quit IRC | 11:22 | |
| *** kencjohnston_ has quit IRC | 11:23 | |
| *** sdague has joined #openstack-infra | 11:24 | |
| *** kencjohnston has joined #openstack-infra | 11:24 | |
| *** _degorenko|afk is now known as degorenko | 11:24 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for broken apt mirrors https://review.openstack.org/360432 | 11:25 |
| rcarrillocruz | regardless of CentOS needing one interface per file, i really prefer having one interface per file myself | 11:26 |
| rcarrillocruz | cos doing ls /etc/network/interfaces.d you can get with a glance what interfaces are configured on that machine | 11:27 |
| *** YorikSar has quit IRC | 11:27 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Hide timeline events and comments of private stories https://review.openstack.org/359895 | 11:28 |
| yolanda | rcarrillocruz, also for opnfv, i moved bridge to hiera. I'm thinking on that, or either giving the bridge a static name | 11:28 |
| yolanda | because on debian/centos, the interface naming is different | 11:28 |
| yolanda | do you see value of having br-vlanxxx name or br-ethxxx name? or we could just go with br-infracloud? | 11:28 |
| rcarrillocruz | there are many things we should move off puppet-infracloud to either system-config or hiera | 11:29 |
| rcarrillocruz | cos any hit of br-vlan2551 on the puppet-infracloud repo is a bug | 11:29 |
| yolanda | ip ranges for example... | 11:29 |
| *** YorikSar has joined #openstack-infra | 11:29 | |
| rcarrillocruz | br-vlan2551 is env spcific, config | 11:29 |
| rcarrillocruz | thus , it should be out of the puppet module | 11:29 |
| *** matbu is now known as matbu|lch | 11:29 | |
| yolanda | rcarrillocruz, my point is... we could even name it with a fixed, non meaningful name, such as br-infracloud and that will work everywhere | 11:30 |
| *** sdague has quit IRC | 11:31 | |
| *** jkilpatr has joined #openstack-infra | 11:35 | |
| rcarrillocruz | sure | 11:35 |
| rcarrillocruz | so cute | 11:35 |
| rcarrillocruz | nodepool@nodepool:~$ OS_CLOUD=infracloud openstack server list | 11:35 |
| rcarrillocruz | WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils | 11:35 |
| *** sandanar__ has quit IRC | 11:35 | |
| rcarrillocruz | +--------------------------------------+-------------+--------+--------------------+---------------+ | 11:35 |
| rcarrillocruz | | ID | Name | Status | Networks | Image Name | | 11:35 |
| rcarrillocruz | +--------------------------------------+-------------+--------+--------------------+---------------+ | 11:35 |
| rcarrillocruz | | 1e50978e-1f15-4203-887c-e507afdd0933 | test-trusty | ACTIVE | public=15.184.65.4 | ubuntu-trusty | | 11:35 |
| rcarrillocruz | +--------------------------------------+-------------+--------+--------------------+---------------+ | 11:35 |
| yolanda | woot! | 11:36 |
| *** sandanar__ has joined #openstack-infra | 11:36 | |
| *** jcoufal has joined #openstack-infra | 11:37 | |
| *** deadnull has joined #openstack-infra | 11:39 | |
| *** jcoufal has quit IRC | 11:39 | |
| *** gildub has quit IRC | 11:39 | |
| *** thorst has joined #openstack-infra | 11:44 | |
| asselin__ | good morning. Is it just me or is http://paste.openstack.org/ very slow? | 11:44 |
| *** rfolco has joined #openstack-infra | 11:47 | |
| *** dansmith has quit IRC | 11:49 | |
| asselin__ | nevermind...working now... | 11:49 |
| *** dansmith has joined #openstack-infra | 11:49 | |
| *** rfolco_ has joined #openstack-infra | 11:50 | |
| *** pt_15 has joined #openstack-infra | 11:50 | |
| *** rfolco has quit IRC | 11:51 | |
| *** jaosorior has quit IRC | 11:52 | |
| *** jaosorior has joined #openstack-infra | 11:52 | |
| *** dprince has joined #openstack-infra | 11:55 | |
| *** ldnunes has joined #openstack-infra | 11:55 | |
| *** asettle has quit IRC | 11:55 | |
| *** asettle has joined #openstack-infra | 11:56 | |
| *** esberglu has joined #openstack-infra | 11:58 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Hide timeline events and comments of private stories https://review.openstack.org/359895 | 12:03 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add new ZK method for sending cluster heartbeat https://review.openstack.org/358868 | 12:04 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Stop using NodePoolBuilder class https://review.openstack.org/359896 | 12:04 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add new ZK method for registering a watch. https://review.openstack.org/358837 | 12:04 |
| *** pradk has joined #openstack-infra | 12:09 | |
| *** tonytan4ever has joined #openstack-infra | 12:10 | |
| *** gouthamr has joined #openstack-infra | 12:11 | |
| *** degorenko has quit IRC | 12:12 | |
| *** degorenko has joined #openstack-infra | 12:13 | |
| *** degorenko has quit IRC | 12:14 | |
| *** degorenko has joined #openstack-infra | 12:14 | |
| asselin__ | mordred, fyi I submitted a bug for the issue I'm running into: https://bugs.launchpad.net/python-novaclient/+bug/1616891 | 12:14 |
| openstack | Launchpad bug 1616891 in python-openstackclient "openstack server list fails with AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult'" [Undecided,New] | 12:14 |
| *** sandanar_ has joined #openstack-infra | 12:15 | |
| *** zul has quit IRC | 12:15 | |
| *** lucas-hungry is now known as lucasagomes | 12:15 | |
| *** salv-orlando has quit IRC | 12:15 | |
| *** kgiusti has joined #openstack-infra | 12:16 | |
| mordred | asselin__: oh wow. lovely | 12:16 |
| *** tonytan_brb has joined #openstack-infra | 12:17 | |
| *** kgiusti has left #openstack-infra | 12:17 | |
| mordred | asselin__: if you only downgrade python-novaclient and oslo-utils, does it work? | 12:18 |
| *** tpsilva has joined #openstack-infra | 12:18 | |
| asselin__ | I will try | 12:18 |
| rcarrillocruz | hmm, in the new zuul v2.5/v3 , is there a recipe to quickly reproduce the test on a vm? i see when we run a job, we keep on folder _zuul_ansible the playbooks, inventory and scripts used by the job execution. However, that's run in the context of a zuul launcher. I assume I could make my VM a zuul launcher itself | 12:18 |
| *** csatari has quit IRC | 12:18 | |
| *** sandanar__ has quit IRC | 12:18 | |
| rcarrillocruz | wondering if there's some doc or someone has a gist for ^ | 12:18 |
| *** jcook has quit IRC | 12:18 | |
| *** baoli has joined #openstack-infra | 12:18 | |
| *** tonytan4ever has quit IRC | 12:19 | |
| *** baoli has quit IRC | 12:19 | |
| *** tchaypo_ has joined #openstack-infra | 12:19 | |
| *** tchaypo has quit IRC | 12:19 | |
| *** prometheanfire has quit IRC | 12:19 | |
| *** tchaypo_ is now known as tchaypo | 12:19 | |
| *** wendar_ has joined #openstack-infra | 12:19 | |
| *** rpodolyaka has quit IRC | 12:19 | |
| *** sc68cal has quit IRC | 12:20 | |
| *** jswarren has quit IRC | 12:20 | |
| *** arif-ali has quit IRC | 12:20 | |
| *** wendar has quit IRC | 12:20 | |
| *** matbu|lch is now known as matbu | 12:20 | |
| mordred | asselin__: also you havea really old six installed | 12:20 |
| mordred | asselin__: I have 1.10.0 on my system | 12:20 |
| mordred | and oslo.utils says: six>=1.9.0 # MIT | 12:20 |
| *** shardy has quit IRC | 12:21 | |
| *** sc68cal has joined #openstack-infra | 12:21 | |
| asselin__ | mordred, I noticed that too...but why? This is what gets installed on a fresh system | 12:21 |
| mordred | in its requirements.txt | 12:21 |
| *** prometheanfire has joined #openstack-infra | 12:21 | |
| *** sandanar_ has quit IRC | 12:21 | |
| *** csatari has joined #openstack-infra | 12:21 | |
| mordred | asselin__: do you happen to know exactly what you are pip installing to get that? | 12:21 |
| *** bradjones has quit IRC | 12:21 | |
| *** eeiden has quit IRC | 12:21 | |
| mordred | asselin__: I just installed python-openstackclient with pip into a new venv and I got six 1.10.0 | 12:22 |
| asselin__ | not exactly..it's all through puppet. I can try to trace it | 12:22 |
| mordred | asselin__: is it possible you have the python-six distro package installed? | 12:22 |
| *** shardy has joined #openstack-infra | 12:22 | |
| *** mwhahaha_ has joined #openstack-infra | 12:23 | |
| asselin__ | python-six install | 12:23 |
| *** timsim_ has joined #openstack-infra | 12:23 | |
| *** timsim has quit IRC | 12:23 | |
| *** electrical has quit IRC | 12:23 | |
| *** timsim_ is now known as timsim | 12:23 | |
| *** arif-ali has joined #openstack-infra | 12:23 | |
| *** edwarnicke has quit IRC | 12:23 | |
| *** briancurtin has quit IRC | 12:23 | |
| *** PalTale has quit IRC | 12:23 | |
| *** vern has quit IRC | 12:23 | |
| *** mwhahaha has quit IRC | 12:23 | |
| mordred | is that maybe at version 1.5 ? | 12:24 |
| *** mwhahaha_ is now known as mwhahaha | 12:24 | |
| *** asaprykin has quit IRC | 12:24 | |
| asselin__ | python-six/trusty-updates,now 1.5.2-1ubuntu1 all [installed] | 12:24 |
| mordred | there's your problem | 12:24 |
| *** sigmavirus has quit IRC | 12:24 | |
| mordred | so, now, the question is - how to fix it | 12:24 |
| *** rpodolyaka has joined #openstack-infra | 12:24 | |
| *** vern has joined #openstack-infra | 12:24 | |
| *** ansmith has joined #openstack-infra | 12:24 | |
| *** baoli has joined #openstack-infra | 12:24 | |
| asselin__ | right...this is starting on clean ubuntu 14.04 image | 12:24 |
| *** baoli has quit IRC | 12:24 | |
| *** PalTale has joined #openstack-infra | 12:25 | |
| *** asaprykin has joined #openstack-infra | 12:25 | |
| mordred | yah - so my curiosity is ... is python-six on the clean ubuntu image and it needs to be uninstalled before the pip stuff gets installed? | 12:25 |
| mordred | or is something in our puppet installing python-six explcitly | 12:25 |
| *** jaosorior is now known as jaosorior_away | 12:25 | |
| *** _sigmavirus24 has joined #openstack-infra | 12:25 | |
| asselin__ | i will check that | 12:25 |
| *** electrical has joined #openstack-infra | 12:25 | |
| *** briancurtin has joined #openstack-infra | 12:25 | |
| *** jcook has joined #openstack-infra | 12:26 | |
| mordred | I don't see any references to python-six in our puppet on codesearch.o.o | 12:26 |
| *** edwarnicke has joined #openstack-infra | 12:26 | |
| *** bradjones has joined #openstack-infra | 12:27 | |
| *** bradjones has quit IRC | 12:27 | |
| *** bradjones has joined #openstack-infra | 12:27 | |
| *** mtreinish has quit IRC | 12:27 | |
| mordred | so I'm thinking that perhaps putting in (somewhere) an apt-get remove of python-six | 12:27 |
| *** rkukura has joined #openstack-infra | 12:27 | |
| mordred | or, maybe an explicit pip install -U six | 12:27 |
| mordred | not sure whta the 'best' way is to make this resilient ... this, btw, is the reason we ditched cloud-init :) | 12:27 |
| mordred | well, one of them | 12:27 |
| *** uberjay has quit IRC | 12:28 | |
| mordred | because cloud-init brings in additional distro python packages, but we have to install some stuff via pip, and the two do not play well together | 12:28 |
| asselin__ | git grep python-six | 12:28 |
| asselin__ | doesn't return anything, but could be in a puppet-module or hidden somewhere else. clean image is booting now.... | 12:28 |
| mordred | cool | 12:29 |
| *** _sigmavirus24 is now known as sigmavirus | 12:30 | |
| *** sigmavirus has joined #openstack-infra | 12:30 | |
| asselin__ | installed in the image: python-six/trusty-updates,now 1.5.2-1ubuntu1 all [installed] | 12:30 |
| *** mtreinish has joined #openstack-infra | 12:30 | |
| mordred | yay | 12:30 |
| *** uberjay has joined #openstack-infra | 12:30 | |
| *** nwkarsten has joined #openstack-infra | 12:30 | |
| mordred | so, try apt-get remove python-six before you install other things | 12:30 |
| *** weshay_bbiab is now known as weshay | 12:30 | |
| mordred | and see if that fixes it | 12:30 |
| *** gordc has joined #openstack-infra | 12:31 | |
| *** ccamacho is now known as ccamacho|lunch | 12:33 | |
| *** sarob has joined #openstack-infra | 12:34 | |
| *** edmondsw has joined #openstack-infra | 12:34 | |
| AJaeger | team, today's problem might be cinder-python27-db-ubuntu-xenial jobs OOMing - I got a couple of those. | 12:35 |
| *** kgiusti has joined #openstack-infra | 12:36 | |
| *** scottda has joined #openstack-infra | 12:36 | |
| *** martinkopec_ has joined #openstack-infra | 12:36 | |
| *** rlandy has joined #openstack-infra | 12:37 | |
| AJaeger | ah, already classified as https://bugs.launchpad.net/cinder/+bug/1616561 | 12:37 |
| openstack | Launchpad bug 1616561 in Cinder "Unit tests periodically oom " [Medium,Confirmed] | 12:37 |
| *** sbezverk_ has joined #openstack-infra | 12:37 | |
| *** sarob has quit IRC | 12:38 | |
| *** sbezverk has quit IRC | 12:39 | |
| *** mdrabe has joined #openstack-infra | 12:39 | |
| *** martinkopec has quit IRC | 12:39 | |
| *** jcoufal has joined #openstack-infra | 12:40 | |
| EmilienM | review.openstack.org is unreachable for me | 12:40 |
| mordred | EmilienM: I can get to it | 12:41 |
| EmilienM | mhh now it is, nevermind. | 12:41 |
| mordred | :) | 12:41 |
| *** degorenko has quit IRC | 12:41 | |
| *** rhallisey has joined #openstack-infra | 12:41 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Refactor TaskManager to be more generic https://review.openstack.org/358646 | 12:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add submit_function method to TaskManager https://review.openstack.org/358647 | 12:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Change naming style of submitTask https://review.openstack.org/358684 | 12:42 |
| anteaya | ummmm, gerrit memory usage is quite high: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all | 12:44 |
| *** derekh has quit IRC | 12:44 | |
| *** degorenko has joined #openstack-infra | 12:44 | |
| *** degorenko_ has joined #openstack-infra | 12:44 | |
| anteaya | we seem to have a pattern of restarting it on a weekly basis | 12:45 |
| anteaya | and appears that according to that cadence we are due anytime now | 12:45 |
| *** ddieterly has joined #openstack-infra | 12:45 | |
| *** xyang1 has joined #openstack-infra | 12:45 | |
| *** jcoufal has quit IRC | 12:47 | |
| *** derekh has joined #openstack-infra | 12:47 | |
| mordred | oh lovely | 12:47 |
| anteaya | yeah | 12:48 |
| *** apetrich has quit IRC | 12:48 | |
| *** markvoelker has joined #openstack-infra | 12:48 | |
| anteaya | mordred: are you in a position to be able to restart it? | 12:48 |
| anteaya | or are you in nyc at a live event and on a mobile device? | 12:48 |
| * anteaya is guessing the later | 12:48 | |
| mordred | nope. I'm at home ... | 12:49 |
| mordred | but | 12:49 |
| anteaya | oh okay | 12:49 |
| mordred | I have not been involved in the previous restarts - so I don't know if I have all the context and would love it if someone else who did was also around | 12:49 |
| anteaya | fair enough | 12:49 |
| openstackgerrit | Yuval Brik proposed openstack-infra/zuul: webapp: Support get change status by hash id https://review.openstack.org/360514 | 12:49 |
| anteaya | yolanda: you have performed some gerrit restarts recently, have you not? | 12:49 |
| *** markusry has joined #openstack-infra | 12:50 | |
| *** thiagop has joined #openstack-infra | 12:51 | |
| mgagne_ | pabelanger: re credentials: they stay the same | 12:52 |
| *** apetrich has joined #openstack-infra | 12:52 | |
| cloudnull | mornings | 12:52 |
| anteaya | morning cloudnull | 12:52 |
| cloudnull | o/ anteaya | 12:52 |
| anteaya | cloudnull: how does your cloud look today? | 12:52 |
| *** Na3iL has joined #openstack-infra | 12:52 | |
| anteaya | all our queues look much happier today | 12:52 |
| sshnaidm | AJaeger, yolanda, pabelanger hi, do you know where can I download template-centos-7 built image from web? if it's possible at all | 12:52 |
| cloudnull | i've not looked at it yet. | 12:53 |
| anteaya | cloudnull: fair enough | 12:53 |
| mgagne_ | pabelanger: keystone is global at internap. We pre-provisioned the networks for both accounts (one for the account with the mirror, and the other for test nodes) | 12:53 |
| *** eharney has quit IRC | 12:53 | |
| AJaeger | sshnaidm: template-centos-7? What is that? | 12:53 |
| AJaeger | sshnaidm: if you mean the images we use in our clouds: Those are not uploaded but you can build them yourself | 12:54 |
| cloudnull | but if the queues in infra are happier I'm thrilled. | 12:54 |
| cloudnull | so i just wont look at it, because no news is good news. | 12:54 |
| cloudnull | :) | 12:54 |
| AJaeger | sshnaidm: there'S tools/build-image.sh in project-config to built it yourself | 12:54 |
| *** mgagne_ is now known as mgagne | 12:54 | |
| *** ddieterly has quit IRC | 12:55 | |
| sshnaidm | AJaeger, yeah, I mean the images from nodepool | 12:56 |
| sshnaidm | AJaeger, but can't I get it ready from somewhere..? | 12:56 |
| asselin__ | mordred, ok, passed the six error. Now ssl cert errors even though verify: false is in clouds.yaml. | 12:57 |
| asselin__ | https://bugs.launchpad.net/python-openstackclient/+bug/1616891/comments/2 | 12:57 |
| openstack | Launchpad bug 1616891 in python-openstackclient "openstack server list fails with AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult'" [Undecided,New] | 12:57 |
| *** woodster_ has joined #openstack-infra | 12:57 | |
| AJaeger | sshnaidm: no, you cannot - those have keys build in and are really large (5GB+) for sharing | 12:58 |
| *** zhurong has joined #openstack-infra | 12:58 | |
| sshnaidm | AJaeger, ok, thanks | 12:58 |
| AJaeger | sshnaidm: what problem are you trying to solve? | 12:58 |
| *** edtubill has joined #openstack-infra | 12:58 | |
| sshnaidm | AJaeger, running job on it, just wanted to save a time | 12:58 |
| *** salv-orlando has joined #openstack-infra | 12:58 | |
| tosky | sshnaidm: related: http://lists.openstack.org/pipermail/openstack-dev/2016-July/100451.html | 12:59 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Update job skip condition for horizon doc-only changes https://review.openstack.org/360517 | 12:59 |
| mordred | asselin__: hrm. paste? | 12:59 |
| tosky | oh, AJaeger already answered | 12:59 |
| anteaya | cloudnull: well there is that approach as well | 12:59 |
| mordred | asselin__: oh - I don't thnk openstackclient honors verify: False? | 12:59 |
| mordred | (I'd have to go check) | 12:59 |
| *** ccamacho|lunch is now known as ccamacho | 12:59 | |
| dtroyer_zz | mordred: it should | 12:59 |
| sshnaidm | tosky, right, it is, thanks | 13:00 |
| asselin__ | hmmm...this worked fine before...and still does on my other machine. | 13:00 |
| * dtroyer_zz says in a drive-by as he closes his laptop to go to the airport… | 13:00 | |
| anteaya | dtroyer_zz: safe travels | 13:00 |
| mordred | dtroyer_zz: cool | 13:00 |
| mordred | asselin__: can you try openstack --insecure server list | 13:00 |
| mordred | asselin__: and just see if that works? | 13:00 |
| mordred | (will help track down the issue) | 13:00 |
| asselin__ | yes that works | 13:01 |
| mordred | asselin__: k | 13:01 |
| mordred | asselin__: can you paste the output of "env | grep OS_" | 13:01 |
| asselin__ | OS_CLOUD=gpc | 13:01 |
| anteaya | <-- breakfast | 13:01 |
| *** psilvad has joined #openstack-infra | 13:01 | |
| mordred | asselin__: just out of curiosity, can you try putting insecure: True in your clouds.yaml ? | 13:02 |
| asselin__ | yup, that works | 13:03 |
| *** vikrant has quit IRC | 13:03 | |
| *** fguillot has joined #openstack-infra | 13:03 | |
| *** yarkot1 has joined #openstack-infra | 13:03 | |
| mordred | ok. thanks. I think there is a bug somewhere in the occ/osc handoff that dtroyer_zz and I will need to go look in to - but maybe for now just doing insecure: True will keep you moving | 13:03 |
| asselin__ | mordred, yes that works for me. Let me know if you need a new bug for that and how to update the current bug status | 13:04 |
| *** yamamoto has quit IRC | 13:05 | |
| *** yamamoto has joined #openstack-infra | 13:06 | |
| mordred | asselin__: nah - I think the existing one is fine | 13:06 |
| *** martinkopec_ has quit IRC | 13:06 | |
| *** baoli has joined #openstack-infra | 13:07 | |
| *** martinkopec has joined #openstack-infra | 13:07 | |
| *** baoli_ has joined #openstack-infra | 13:07 | |
| *** dizquierdo has quit IRC | 13:08 | |
| *** notmyname has joined #openstack-infra | 13:09 | |
| *** julim_ has joined #openstack-infra | 13:09 | |
| *** fguillot_ has joined #openstack-infra | 13:11 | |
| *** psilvad has quit IRC | 13:11 | |
| *** baoli has quit IRC | 13:11 | |
| *** fguillot_ has quit IRC | 13:12 | |
| *** fguillot_ has joined #openstack-infra | 13:12 | |
| *** Wei_Liu has quit IRC | 13:13 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add os_trove to OpenStack-Ansible https://review.openstack.org/360530 | 13:14 |
| *** julim_ has quit IRC | 13:15 | |
| *** dizquierdo has joined #openstack-infra | 13:16 | |
| *** yamamoto has quit IRC | 13:17 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks https://review.openstack.org/357517 | 13:17 |
| *** yamamoto has joined #openstack-infra | 13:17 | |
| *** julim_ has joined #openstack-infra | 13:17 | |
| sc68cal | gate looks a lot healthier today | 13:17 |
| anteaya | indeed | 13:18 |
| AJaeger | sc68cal: indeed - with exception of cinder python27 tests - lots of OOMs | 13:18 |
| anteaya | thanks for your help yesterday, sc68cal | 13:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic https://review.openstack.org/360446 | 13:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Plumb ipv6-preferred into force-ipv4 setting https://review.openstack.org/360445 | 13:19 |
| *** jordanP has quit IRC | 13:20 | |
| anteaya | sc68cal: there was someone asking for help setting up devstack with linuxbridge in -dev yesterday, I told them today might be better | 13:20 |
| anteaya | sc68cal: if you feel like taking a peek | 13:20 |
| sc68cal | ok let me check scrollback | 13:21 |
| anteaya | thank you | 13:21 |
| anteaya | it was in the -dev channel | 13:22 |
| *** links has quit IRC | 13:22 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Upgrade to LV 5.2 https://review.openstack.org/305521 | 13:24 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework https://review.openstack.org/322307 | 13:24 |
| *** Wei_Liu has joined #openstack-infra | 13:25 | |
| AJaeger | Is there any infra-core to review https://review.openstack.org/334870 for openstack-infra/js-afs-blob-store, please? It's adding a bindep.txt file. krotscheck already gave a +2. | 13:27 |
| *** eranrom has quit IRC | 13:27 | |
| *** fguillot has quit IRC | 13:28 | |
| *** claudiub|2 has joined #openstack-infra | 13:29 | |
| AJaeger | thanks, rcarrillocruz ! | 13:29 |
| *** zul has joined #openstack-infra | 13:31 | |
| *** mat128 is now known as mat128|away | 13:31 | |
| *** eharney has joined #openstack-infra | 13:31 | |
| *** roxanagh_ has joined #openstack-infra | 13:32 | |
| anteaya | I'm out for most of the morning, I'm still looking for a gerrit restart due to memory usage | 13:32 |
| anteaya | thank you | 13:32 |
| *** matt-borland has joined #openstack-infra | 13:32 | |
| openstackgerrit | Mathieu Mitchell proposed openstack/diskimage-builder: Generate and use upper-constraints for ironic-agent https://review.openstack.org/360547 | 13:35 |
| *** asettle has quit IRC | 13:35 | |
| *** asettle has joined #openstack-infra | 13:35 | |
| *** salv-orlando has quit IRC | 13:35 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Reduce job log retention to 90 days https://review.openstack.org/360551 | 13:36 |
| *** roxanagh_ has quit IRC | 13:36 | |
| openstackgerrit | Merged openstack-infra/nodepool: Add new ZK method for registering a watch. https://review.openstack.org/358837 | 13:36 |
| *** edtubill has quit IRC | 13:36 | |
| openstackgerrit | Merged openstack-infra/nodepool: Add new ZK method for sending cluster heartbeat https://review.openstack.org/358868 | 13:36 |
| openstackgerrit | Merged openstack-infra/nodepool: Stop using NodePoolBuilder class https://review.openstack.org/359896 | 13:37 |
| openstackgerrit | Frédéric Guillot proposed openstack-infra/project-config: Add Almanach IRC channel https://review.openstack.org/360049 | 13:37 |
| *** jlvillal has left #openstack-infra | 13:38 | |
| *** senk has quit IRC | 13:38 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add os_trove to OpenStack-Ansible https://review.openstack.org/360530 | 13:38 |
| *** sdague has joined #openstack-infra | 13:41 | |
| *** DrifterZA has quit IRC | 13:43 | |
| sc68cal | anteaya: lol, I read scrollback | 13:46 |
| sc68cal | anteaya: oh god he uses slack somewhere and knows how to use @all to ping everyone at once. :-\ | 13:46 |
| sc68cal | the worst idea ever | 13:46 |
| *** eharney_ has joined #openstack-infra | 13:46 | |
| *** eharney has quit IRC | 13:47 | |
| *** jaosorior_away is now known as jaosorior | 13:48 | |
| *** haaj has joined #openstack-infra | 13:49 | |
| *** haaj has quit IRC | 13:50 | |
| dims | pabelanger : fungi : is there a sunset date for switching off the mirantis stackalytics in favor of http://stackalytics.openstack.org/ ? | 13:50 |
| *** yamamoto has quit IRC | 13:50 | |
| *** eharney_ is now known as eharney | 13:51 | |
| *** yamamoto has joined #openstack-infra | 13:51 | |
| *** melwitt_ has joined #openstack-infra | 13:51 | |
| *** kaisers_ has quit IRC | 13:52 | |
| fungi | dims: pabelanger was having trouble getting up with the stackalytics maintainers to help us solve a couple of lingering manageability issues. for example right now the only configuration update possible results in a multi-hour outage for the service, but much of its correlation/override data is configuration-driven so it needs frequent reloads | 13:52 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add iLO, private and public IP addresses for compute003.vanilla https://review.openstack.org/360562 | 13:53 |
| *** hongbin has joined #openstack-infra | 13:53 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate: Revert "Add q-vpn back to base services but exclude for liberty runs" https://review.openstack.org/310900 | 13:53 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: remove grenade jobs from horizon https://review.openstack.org/360563 | 13:53 |
| fungi | dims: also we need to ascertain whether the dead gerrit ssh connection slash-and-burn stackalytics.com has been experiencing is a possible problem for stackalytics.o.o as well | 13:54 |
| *** ddieterly has joined #openstack-infra | 13:54 | |
| *** rhallisey has quit IRC | 13:54 | |
| dims | fungi : ack thanks! | 13:55 |
| *** jheroux has joined #openstack-infra | 13:56 | |
| fungi | for the reconfiguration issue, we boiled it down to the fact that the service is mostly stateless and seems to reprocess all its data to regenerate stats from scratch, then stores them all in an in-memory database. so any restart means hours of it redoing all that statistical analysis | 13:56 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: don't start horizon in grenade https://review.openstack.org/360565 | 13:56 |
| *** ddieterly has quit IRC | 13:56 | |
| *** bauzas has joined #openstack-infra | 13:56 | |
| dims | fungi : right | 13:56 |
| *** ddieterly has joined #openstack-infra | 13:57 | |
| *** yamamoto has quit IRC | 13:57 | |
| sdague | fungi et al - https://review.openstack.org/360563 would be really nice as it would decouple horizon from the rest of the integrated gate | 13:57 |
| *** rhallisey has joined #openstack-infra | 13:57 | |
| sdague | especially as that job doesn't do anything any more | 13:57 |
| sdague | at least from a horizon perspective | 13:57 |
| fungi | dims: so solving that probably means some significant rework of its backend data handling design | 13:57 |
| *** ociuhandu has joined #openstack-infra | 13:57 | |
| * fungi is unsure whether he should be flattered or frightened that he's the only one called out before "et al" | 13:58 | |
| anteaya | sc68cal: indeed | 13:58 |
| sdague | heh | 13:58 |
| dims | fungi : ack. some folks were asking for better kubernetes stats in that tool and i am going to push back on that. hence checking | 13:58 |
| sdague | fungi: well, you are an active nic that I saw once I connected :) | 13:58 |
| anteaya | sc68cal: it amazes me that people enter and ask for advice and then clearly demonstrate they are unwilling to take advice | 13:58 |
| anteaya | sc68cal: and then still expect people to spend time on thm | 13:59 |
| fungi | sdague: heh | 13:59 |
| anteaya | sc68cal: I gave four options which the person then proceeded to reject, demonstrating their cleverness | 13:59 |
| fungi | robcresswell: you seem to be subscribed as a requested reviewer on https://review.openstack.org/360563 | 14:00 |
| sdague | fungi: thank you | 14:00 |
| sdague | anteaya would you also like to look at - https://review.openstack.org/#/c/360563/ ? | 14:00 |
| *** e0ne has joined #openstack-infra | 14:00 | |
| sc68cal | anteaya: yeah. I mean we do have a section in the devstack docs, in the neutron guide about linuxbridge | 14:00 |
| *** devananda|OSE is now known as devananda|OPS | 14:01 | |
| sc68cal | not sure about that create_ovs_taps call he's talking about | 14:02 |
| anteaya | sdague: I tried | 14:02 |
| *** aviau has quit IRC | 14:02 | |
| anteaya | sc68cal: I had no idea | 14:02 |
| *** aviau has joined #openstack-infra | 14:02 | |
| anteaya | fungi: what do you think about a gerrit restart? | 14:02 |
| robcresswell | fungi: It appears so | 14:02 |
| anteaya | fungi: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=27&rra_id=all | 14:03 |
| EmilienM | can someone approve https://review.openstack.org/#/c/359539/ please? | 14:03 |
| fungi | anteaya: i need to switch computers to one where i can view the javamelody graphs for it more readily | 14:03 |
| EmilienM | fungi: if you have time, could you add https://review.openstack.org/#/c/353662/ do your review list please? my group is waiting for this new repo, thanks a lot | 14:04 |
| *** zul has quit IRC | 14:04 | |
| * fungi failed at his policy of not breaking radio silence while still trying to catch up on e-mail | 14:05 | |
| *** zul has joined #openstack-infra | 14:05 | |
| sdague | rcarrillocruz: thank you for the review | 14:05 |
| *** xarses_ has quit IRC | 14:06 | |
| anteaya | fungi: fair enough | 14:06 |
| rcarrillocruz | np | 14:06 |
| anteaya | fungi: ah | 14:06 |
| anteaya | yeah you gotta keep your own rules | 14:07 |
| *** Julien-zte has joined #openstack-infra | 14:08 | |
| *** jaosorior has quit IRC | 14:08 | |
| fungi | yeah, once people realize that i'm around, the rest of my day is driven by random requests | 14:08 |
| AJaeger | ;) | 14:09 |
| *** sarob has joined #openstack-infra | 14:10 | |
| *** hockeynut has joined #openstack-infra | 14:11 | |
| *** _ari_ has joined #openstack-infra | 14:11 | |
| fungi | anteaya: gerrit's not really spending much/any time on java garbage collection yet, which is usually what drives us to restart it | 14:12 |
| EmilienM | AJaeger: can you also look https://review.openstack.org/#/c/359539/ please? | 14:12 |
| *** zz_dimtruck is now known as dimtruck | 14:12 | |
| *** rbrndt has joined #openstack-infra | 14:12 | |
| AJaeger | EmilienM: is that project-config? | 14:12 |
| EmilienM | AJaeger: no | 14:12 |
| EmilienM | AJaeger: aren't you core on puppet infra modules? | 14:13 |
| AJaeger | EmilienM: then I don't have +2 powers - and little knowledge... | 14:13 |
| fungi | anteaya: were you getting timeouts from the webui or something? | 14:13 |
| AJaeger | EmilienM: no, I'm not | 14:13 |
| asselin__ | mordred, thank you very much for all your help getting passed those issues ^^. I'm not back to where I was and able to launch a node! cc/ rcarrillocruz https://review.openstack.org/#/c/320466/ is now working again (with some local fixes that I'll push up) | 14:13 |
| EmilienM | AJaeger: dang ! | 14:13 |
| EmilienM | who can approve this puppet infra change? https://review.openstack.org/#/c/359539/ | 14:13 |
| AJaeger | EmilienM: project-config is enough for me ;) | 14:13 |
| robcresswell | fungi: Its just because it affects Horizon I assume (I dont know who tagged me) but the change is fine. | 14:13 |
| rcarrillocruz | 'o/ | 14:13 |
| rcarrillocruz | \o/ too | 14:13 |
| *** esker has joined #openstack-infra | 14:13 | |
| fungi | robcresswell: yeah, i haven't looked but assumed you were added as the horizon infra liaison or something | 14:13 |
| AJaeger | robcresswell: I tagged you | 14:14 |
| fungi | robcresswell: er, ptl! i forgot you're the horizon ptl now | 14:14 |
| fungi | so de-facto infra liaison for your project ;) | 14:14 |
| robcresswell | haha | 14:14 |
| anteaya | fungi: some folks were reporting they couldn't reach gerrit, so I looked at memory usage in cacti | 14:14 |
| AJaeger | fungi: yeah, wanted to make the PTL aware ;) | 14:14 |
| *** deadnull is now known as _deadnull | 14:15 | |
| anteaya | fungi: I personally had no gerrit issues | 14:15 |
| robcresswell | AJaeger: Ah, cool, yeah its fine with me. | 14:15 |
| anteaya | fungi: also I'm out for the rest of the morning, thanks for looking at it, I leave the decision to you | 14:15 |
| *** fguillot_ has quit IRC | 14:16 | |
| fungi | anteaya: yeah, system memory utilization on review.o.o is misleading. about the only thing that server runs is a java jvm, which is configured to be able to use up to the majority of the memory available | 14:16 |
| *** kushal has joined #openstack-infra | 14:16 | |
| *** mtanino has joined #openstack-infra | 14:16 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Revert "Remove infracloud-west" https://review.openstack.org/360579 | 14:16 |
| fungi | it in turn performs its own memory management within that jvm | 14:16 |
| *** pgadiya has quit IRC | 14:17 | |
| rcarrillocruz | i will wip ^ btw | 14:17 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Revert "Remove infracloud" https://review.openstack.org/360579 | 14:17 |
| *** dtardivel has quit IRC | 14:17 | |
| fungi | so while system memory utilization does drop when we restart the jvm (because its cache and uncollected garbage are suddenly dumped into the bitbucket), that's not the reason we restart it | 14:17 |
| *** fguillot has joined #openstack-infra | 14:19 | |
| fungi | anteaya: as far as i can see the server is behaving in a healthy manner. if people were complaining about access from the conference, it could have been a local network issue there | 14:19 |
| openstackgerrit | Marc Aubry proposed openstack-infra/gerritbot: Pin several dependencies https://review.openstack.org/253238 | 14:19 |
| *** sarob has quit IRC | 14:20 | |
| *** sarob has joined #openstack-infra | 14:21 | |
| *** claudiub|2 has quit IRC | 14:26 | |
| pleia2 | fungi: prepping the gerrit rename email, we're loosely confirmed for 18:00-22:00 UTC, yeah? (that's what we decided in the meeting) | 14:28 |
| *** berendt has quit IRC | 14:28 | |
| slagle | hi, i had to recreate the flavor used by nodepool to launch instnaces in the tripleo test cloud and the id changed | 14:28 |
| slagle | pabelanger: ^ | 14:28 |
| slagle | i'm seeing some instances fail to launch now. i'm thinking nodepool might be using the old id | 14:29 |
| slagle | actually i can recreate the new flavor with the old id... | 14:29 |
| *** e0ne has quit IRC | 14:29 | |
| fungi | pleia2: confirmation is slightly firmer now that we've heard back from teh release team | 14:29 |
| fungi | pleia2: basically, they expect to have all milestone 3 release activities out of the way well in advance of our start time, but they'll let us know if there's any work running over (and we should try to remember to check in with them regardless, just to be safe) | 14:30 |
| pleia2 | ok yeah, I saw dhellmann had talked to ttx | 14:30 |
| * pleia2 nods | 14:30 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute003.vanilla metadata for better tracking https://review.openstack.org/360589 | 14:30 |
| fungi | pleia2: so let's treat it as a firm start time, and if we need to delay the start a little or postpone and reschedule then we'll do that | 14:31 |
| pleia2 | and we don't expect to use the whole window, but just in case we need to do offline reindex (I won't mention that, let's just make everyone happy with early gerrit return ;)) | 14:31 |
| pleia2 | sounds good | 14:31 |
| fungi | agreed | 14:31 |
| *** esker has quit IRC | 14:32 | |
| *** rajinir has joined #openstack-infra | 14:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Create puppet-panko repository https://review.openstack.org/353662 | 14:33 |
| *** e0ne has joined #openstack-infra | 14:34 | |
| *** edtubill has joined #openstack-infra | 14:35 | |
| openstackgerrit | Merged openstack-infra/system-config: Document use of its-storyboard in Gerrit config https://review.openstack.org/358250 | 14:35 |
| *** esker has joined #openstack-infra | 14:35 | |
| *** dtardivel has joined #openstack-infra | 14:35 | |
| *** esker has quit IRC | 14:36 | |
| fungi | ouch, just saw a gate-cinder-python34-db job oom in the gate pipeline... http://logs.openstack.org/37/355837/3/gate/gate-cinder-python34-db/7b49f24/console.html#_2016-08-25_13_42_12_352115 | 14:36 |
| AJaeger | fungi, there are several ;( | 14:36 |
| AJaeger | fungi already classified as https://bugs.launchpad.net/cinder/+bug/1616561 | 14:36 |
| openstack | Launchpad bug 1616561 in Cinder "Unit tests periodically oom " [Medium,Confirmed] | 14:36 |
| fungi | well that's not good :/ | 14:37 |
| AJaeger | indeed - currently no #1 problem | 14:37 |
| *** ociuhandu has quit IRC | 14:38 | |
| *** esker has joined #openstack-infra | 14:38 | |
| *** esker has quit IRC | 14:38 | |
| fungi | i wonder if py35 is any better? | 14:38 |
| AJaeger | py35 is non-voting for cinder | 14:39 |
| dtroyer_zz | asselin: https://review.openstack.org/360600 should fix your verify problem in OSC. | 14:39 |
| *** melwitt_ has quit IRC | 14:40 | |
| AJaeger | fungi, didn't hear anything there for py34/py35 - but didn't dig deeper. | 14:40 |
| fungi | yeah, looks like qa/cinder are on it, so good enough for me | 14:40 |
| *** rkukura has quit IRC | 14:40 | |
| fungi | if it becomes too painful i guess they can stop gating on it temporarily | 14:41 |
| *** sdake has joined #openstack-infra | 14:42 | |
| *** zul has quit IRC | 14:42 | |
| *** DrifterZA has joined #openstack-infra | 14:43 | |
| sshnaidm | AJaeger, can you please help with build image? I'm trying to build it using "DISTRO=centos7 IMAGE_NAME=template_centos ./project-config/tools/build-image.sh" but I have an error: Element 'openstack-repos' not found in 'nodepool/elements:/usr/bin/../share/diskimage-builder/elements' How can I get all elements I need? | 14:43 |
| AJaeger | sshnaidm: no, I cannot any further - never used the script. Hope somebody else from the team can... | 14:44 |
| sshnaidm | pabelanger, maybe you know? ^^ | 14:44 |
| *** flepied has quit IRC | 14:44 | |
| AJaeger | sshnaidm: I would call it from project-config directory, so try "cd project-config" first | 14:44 |
| fungi | sshnaidm: i think the most common cause of that error is that your working directory is not the root of the project-config repo. you need to do `DISTRO=centos7 IMAGE_NAME=template_centos ./tools/build-image.sh` | 14:45 |
| sshnaidm | AJaeger, I checked, I really don't have these elements | 14:45 |
| *** e0ne has quit IRC | 14:45 | |
| fungi | sshnaidm: you don't have http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/openstack-repos ? | 14:45 |
| AJaeger | They are in ./nodepool/elements/openstack-repos | 14:46 |
| *** senk has joined #openstack-infra | 14:46 | |
| sshnaidm | AJaeger, fungi aah, I was looking in /usr/share/.. | 14:46 |
| AJaeger | sshnaidm: see the path, I really think you should cd project-config | 14:46 |
| fungi | sshnaidm: the script is looking for elements in ./nodepool/elements which is why your cwd needs to be the root of the project-config repo | 14:46 |
| fungi | they are in that repo | 14:46 |
| sshnaidm | AJaeger, fungi thanks, it works! | 14:47 |
| AJaeger | :) | 14:47 |
| fungi | awesome | 14:47 |
| *** openstackgerrit has quit IRC | 14:48 | |
| *** openstackgerrit has joined #openstack-infra | 14:49 | |
| *** hockeynut has quit IRC | 14:51 | |
| *** david-lyle has joined #openstack-infra | 14:53 | |
| *** yamamoto has joined #openstack-infra | 14:54 | |
| *** caowei has quit IRC | 14:56 | |
| *** xarses_ has joined #openstack-infra | 14:56 | |
| *** david-lyle has quit IRC | 14:59 | |
| *** yamamoto has quit IRC | 15:00 | |
| *** zul has joined #openstack-infra | 15:01 | |
| *** thcipriani|afk is now known as thcipriani | 15:02 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Parameterize certificates in infracloud https://review.openstack.org/359757 | 15:03 |
| openstackgerrit | Merged openstack-infra/system-config: Add iLO, private and public IP addresses for compute003.vanilla https://review.openstack.org/360562 | 15:03 |
| *** tonytan4ever has joined #openstack-infra | 15:03 | |
| *** jpich has quit IRC | 15:04 | |
| *** flepied has joined #openstack-infra | 15:04 | |
| *** tonytan_brb has quit IRC | 15:06 | |
| clarkb | We have a really odd shared queue in the gate pipeline. It includes puppet-swift, puppet-nova, puppet-trove but also python-neutronclient, python-ironic-inspector-client, designate-tempest-plugin, and tripleo-heat-templates | 15:07 |
| *** bnemec has quit IRC | 15:08 | |
| *** dprince has quit IRC | 15:09 | |
| *** jamesden_ has joined #openstack-infra | 15:10 | |
| clarkb | the common test thread putting them in the same queue is not immediately obvious to me. The tripleo things have a shared job with tripleo things and the puppet things have a shared job with puppet things but why the overlap and inclusion of neutronclient and ironic-inspector-client and the designate tempest plugin | 15:10 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 15:11 |
| *** zul has quit IRC | 15:11 | |
| *** vinaypotluri has joined #openstack-infra | 15:11 | |
| *** jamesdenton is now known as Guest1506 | 15:11 | |
| mordred | clarkb: sooooooo | 15:11 |
| mordred | clarkb: https://review.openstack.org/#/c/360445/ | 15:11 |
| mordred | clarkb: the dual-network stack detection support uncovered that devstack is providing a dual network stack :) | 15:12 |
| *** dizquierdo has quit IRC | 15:12 | |
| *** salv-orlando has joined #openstack-infra | 15:12 | |
| mordred | Shrews: ^^ (you'll love that just as much as clarkb I think) | 15:12 |
| *** ociuhandu has joined #openstack-infra | 15:12 | |
| Shrews | neat | 15:13 |
| *** bnemec has joined #openstack-infra | 15:13 | |
| mordred | I'm on a call ...so I'll chat with sc68cal about his review on the related shade patch when I get off the phone | 15:13 |
| clarkb | and the shade force ipv4 means give me a public ip thats ipv4 and do floating ips etc even if ipv6 is an option? | 15:13 |
| mordred | yah | 15:14 |
| mordred | basically "I know you detected that I can do ipv6, but srrsly, I wanna do ipv4" | 15:14 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Remove EPEL usage https://review.openstack.org/347499 | 15:15 |
| mordred | the nodepool thing makes me wnat to bang on the shade patch a little more | 15:15 |
| *** WJMUqqJZRD has joined #openstack-infra | 15:15 | |
| mordred | to make sure there isn't another niggly that got missed | 15:15 |
| clarkb | couldn't shade just always do its best to do both by default? | 15:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add mtl01 region to internap on puppetmaster https://review.openstack.org/360622 | 15:15 |
| pabelanger | mgagne: thanks, creds appear to be working | 15:15 |
| clarkb | rather than needing to explicitly flag things? | 15:15 |
| *** mat128|away is now known as mat128 | 15:16 | |
| pabelanger | mordred: clarkb: fungi: ^ first changes to add internap mtl01 to puppetmaster | 15:16 |
| pabelanger | on to cloud_launcher next | 15:16 |
| mordred | mgagne: should we add mtl01 to os-client-config vendors? | 15:16 |
| mordred | clarkb: it could ... although we'd need a way to turn that off for osic in that case | 15:17 |
| mgagne | mordred: the region is not publicly available yet and won't be for a couple of months. | 15:17 |
| mordred | mgagne: ok. cool | 15:17 |
| mgagne | pabelanger: each region has a set of networks too. Will it be considered in the config? | 15:17 |
| mordred | clarkb: because with osic we could make a floating ip but we don't want to | 15:17 |
| mordred | clarkb: but with devstack we do want to make a floating ip | 15:17 |
| *** salv-orl_ has joined #openstack-infra | 15:18 | |
| mordred | clarkb: (this is why I think it's a good nudge that the nodepool test gave us and I think it's worth digging in to further when I'm not on the phone... I agree with you on the ick factor of the config here) | 15:18 |
| pabelanger | mgagne: Ya, I was wondering about that. We might need to make changes to our layout | 15:18 |
| clarkb | mordred: but osic should only make a floating ip if you are using GATEWAY_NET? | 15:18 |
| clarkb | I guess I don't fully understand the path to "can/should I floating IP" | 15:18 |
| mordred | clarkb: right. it's possible that I've overtought that part | 15:18 |
| *** vhosakot has joined #openstack-infra | 15:19 | |
| mordred | clarkb: and we're safe on osic because of lack of router connected to GATEWAY_NET_IPV6 | 15:19 |
| *** furlongm_ has joined #openstack-infra | 15:19 | |
| clarkb | ya | 15:19 |
| mordred | anywho - tl;dr - I think there is more work to be done here before we're happy | 15:19 |
| clarkb | at least that is what I was expecting in my "do as much as I can by default" idea | 15:19 |
| mordred | yah | 15:19 |
| *** furlongm has quit IRC | 15:20 | |
| *** salv-orlando has quit IRC | 15:20 | |
| anteaya | fungi: thanks for explaining the gerrit memory utilization situation, it was EmilienM who mentioned not being able to reach gerrit this morning, I don't know if he is at a conference or not | 15:20 |
| clarkb | mordred: and http://logs.openstack.org/45/360445/2/check/gate-dsvm-nodepool-src-shade/460dc39/logs/nodepool-list.txt.gz shows it to be preferring ipv4 with devstack | 15:21 |
| mgagne | pabelanger: and AFAIK, you don't use the LAN network right? | 15:21 |
| *** links has joined #openstack-infra | 15:21 | |
| *** weshay is now known as weshay_afk | 15:21 | |
| *** sarob has quit IRC | 15:21 | |
| EmilienM | anteaya: no, I'm in old Québec :) | 15:22 |
| pabelanger | mgagne: let me check, unless clarkb know of the top of his head | 15:22 |
| *** sarob has joined #openstack-infra | 15:22 | |
| mgagne | pabelanger: ok, just want to make sure I made the right assumption. Because we didn't provision a LAN network in mtl01. | 15:22 |
| *** sdague has quit IRC | 15:22 | |
| clarkb | pabelanger: mgagne meaning do we rely on l2 between the instances? no because enough clouds don't provide us that so we end up just making our own l2 overlays if they are necessary | 15:22 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute003.vanilla metadata for better tracking https://review.openstack.org/360589 | 15:23 |
| mgagne | clarkb: just the L2/L3 LAN networks. (whatever connectivity type you need) | 15:23 |
| pabelanger | mordred: So, with clouds.yaml for internap. Should we remove the current network fields (since nyj01 and mtl01 are different) or create a new entry per region? I feel removing them is right way, but wanted to confirm | 15:23 |
| fungi | EmilienM: ahs it continued to be a problem? are you stuck having to reach gerrit through the oqc municipal aqueducts? | 15:23 |
| pabelanger | clarkb: Ya, we have 2 networks defined in clouds.yaml today (WAN / LAN). Why is that? | 15:23 |
| openstackgerrit | Maksim Malchuk proposed openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5 https://review.openstack.org/357335 | 15:24 |
| mgagne | pabelanger: I think the issue was: How can nodepool detect the right network with public connectivity. | 15:24 |
| pabelanger | mgagne: maybe | 15:24 |
| pabelanger | checking git history | 15:24 |
| EmilienM | fungi: no, it was during ~5 min but now everything is fine I can continue to reach gerrit through canadian cables :) | 15:24 |
| clarkb | mordred: ok reviewed the nodepool chnage | 15:24 |
| rajinir | Hi, is there automatic way or script of some sort to recheck all the failed builds in the past week? | 15:25 |
| mordred | clarkb: thx | 15:25 |
| clarkb | pabelanger: that I do not know. I think it was jeblair who set up the first internap cloud so may know | 15:25 |
| EmilienM | fungi: my only hope is that cable doesn't disturb the mooses | 15:25 |
| fungi | EmilienM: okay, good. if the problem comes back and persists, we can try to troubleshoot | 15:25 |
| mordred | pabelanger: because we cannot detect their qualities | 15:25 |
| mordred | pabelanger: you need to put them into clouds.yaml to tell shade which network is public and which is private | 15:25 |
| clarkb | rajinir: I think our preference would be that there is a little critical thought into whether or not a failure is due to the change or needs a recheck | 15:25 |
| mordred | pabelanger: it's easy for a human - the public network has "WAN" in the name | 15:26 |
| *** zul has joined #openstack-infra | 15:26 | |
| mordred | pabelanger: those settings can be set on a per-region basis | 15:26 |
| rajinir | clarb: ok, I take that as a no. In our case we had hardware issues which is solved now | 15:27 |
| pabelanger | mordred: okay, cool. let me read the docs | 15:27 |
| fungi | rajinir: oh, you mean for your third-party ci results? | 15:27 |
| *** hockeynut has joined #openstack-infra | 15:27 | |
| fungi | rajinir: you can write a shell script that loops over the `zuul enqueue` subcomannd | 15:27 |
| *** rhallisey_ has joined #openstack-infra | 15:28 | |
| fungi | rajinir: that's one of the primary reasons that command-line tool exists | 15:28 |
| pabelanger | haha | 15:28 |
| pabelanger | there is an internap example in the os-client-config readme | 15:28 |
| mordred | pabelanger: http://docs.openstack.org/developer/os-client-config/#per-region-settings | 15:28 |
| anteaya | EmilienM: figured you were in old quebec | 15:28 |
| *** sarob has quit IRC | 15:28 | |
| mordred | pabelanger: up | 15:28 |
| mordred | pabelanger: we wrote the feature because of internap :) | 15:28 |
| anteaya | yeah don't disturb the mooses | 15:28 |
| *** nmagnezi has joined #openstack-infra | 15:28 | |
| pabelanger | mordred: excellent! Using that now | 15:28 |
| * anteaya has a story of being chased by a moose | 15:28 | |
| rajinir | fungi: exactly. that's cool. will give it a try | 15:29 |
| fungi | rajinir: you'd likely need to perform a gerrit query first to look up the change,patchset numbers for every change where your ci account has a verify -1 vote | 15:29 |
| fungi | rajinir: and then turn that into a series of zuul enqueue commands | 15:29 |
| *** jlvillal has joined #openstack-infra | 15:29 | |
| *** _deadnull is now known as deadnull | 15:29 | |
| *** deadnull has quit IRC | 15:30 | |
| *** sdague has joined #openstack-infra | 15:30 | |
| fungi | rajinir: gerrit has a very flexible and well-documented rest api: https://review.openstack.org/Documentation/rest-api-changes.html#list-changes | 15:30 |
| *** derekh has quit IRC | 15:31 | |
| rajinir | fungi: This is very helpful. Thank you | 15:31 |
| *** fguillot has quit IRC | 15:32 | |
| *** rhallisey has quit IRC | 15:32 | |
| *** sandanar has joined #openstack-infra | 15:32 | |
| fungi | you're welcome | 15:32 |
| fungi | rajinir: and zuul enqueue is documented at http://docs.openstack.org/infra/zuul/client.html#usage | 15:33 |
| openstackgerrit | Maksim Malchuk proposed openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5 https://review.openstack.org/357335 | 15:34 |
| clarkb | pabelanger: mordred mgagne then in nodepool we can specify that we only need the WAN network? | 15:34 |
| *** derekh has joined #openstack-infra | 15:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add mtl01 region to internap on puppetmaster https://review.openstack.org/360622 | 15:35 |
| mordred | clarkb: whatever we did for the other internap in nodepool | 15:35 |
| * mordred can't remember | 15:35 | |
| *** andreas_s has quit IRC | 15:35 | |
| *** psachin has quit IRC | 15:36 | |
| *** xarses_ is now known as xarses | 15:37 | |
| *** hockeynu_ has joined #openstack-infra | 15:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add mtl01 region to internap on puppetmaster https://review.openstack.org/360622 | 15:38 |
| *** gouthamr is now known as gouthamr|afk | 15:38 | |
| *** sarob has joined #openstack-infra | 15:39 | |
| *** Swami has joined #openstack-infra | 15:39 | |
| *** martinkopec has quit IRC | 15:40 | |
| *** kaisers_ has joined #openstack-infra | 15:40 | |
| *** armax has joined #openstack-infra | 15:41 | |
| *** hockeynut has quit IRC | 15:41 | |
| *** yaume_ has quit IRC | 15:41 | |
| *** roxanagh_ has joined #openstack-infra | 15:42 | |
| *** kaisers_ has quit IRC | 15:44 | |
| mgagne | pabelanger: openstackci-internap vs openstackjenkins-internap ? | 15:44 |
| *** rhallisey_ is now known as rhallisey | 15:45 | |
| pabelanger | mgagne: 1 for mirror, other is nodepool | 15:45 |
| mgagne | pabelanger: ok, both don't have the same network name fyi | 15:46 |
| pabelanger | mgagne: also, just did a test: http://paste.openstack.org/show/563407/ any thoughts on the error? | 15:46 |
| *** andymaier has quit IRC | 15:46 | |
| clarkb | hrm now we have an integrated gate and a horizon/tempest-horizon queue except the integrated gate has nothingin it and the horizon one has not horizon in it | 15:46 |
| pabelanger | mgagne: okay, will confirm that now | 15:46 |
| clarkb | I wonder if this is just a transitive state while zuul shuffles things around. It looks really funny on the status page | 15:46 |
| *** roxanagh_ has quit IRC | 15:46 | |
| clarkb | can someone thati s not me confirm they see the same thing ^ | 15:47 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 15:47 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make sidebar submenu the same length as sidebar https://review.openstack.org/355554 | 15:47 |
| mordred | clarkb: I see horizon and not-horizon in the horizon pipeline | 15:47 |
| openstackgerrit | Merged openstack-infra/system-config: Add compute004 to compute007 vanilla machines to static infracloud inventory https://review.openstack.org/360611 | 15:47 |
| mgagne | pabelanger: will be looking into it shortly | 15:48 |
| *** _nadya_ has quit IRC | 15:48 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Limit Heat engine workers to 4 in rh1 https://review.openstack.org/360650 | 15:48 |
| mgagne | pabelanger: make sure you use the right tenant. | 15:48 |
| mgagne | pabelanger: the one you used is for nodepool and don't have capacity for B series, only A series | 15:48 |
| pabelanger | mgagne: okay, that could be it | 15:49 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 15:49 |
| clarkb | jeblair: the current state of the zuul status page may be of interest to you | 15:49 |
| clarkb | jeblair: it seems to have confused the integreated gate and horizon | 15:50 |
| pabelanger | mgagne: thanks, that was the issue | 15:50 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 15:50 |
| sdague | clarkb: we just split that | 15:52 |
| clarkb | hrm and there is a straggler neutron change that could potentially merge without cross testing with the rest of the integreated gate | 15:52 |
| clarkb | sdague: I think we may have broken split it out then. Look at hte zuul status page | 15:52 |
| sdague | ok, well all that done was to stop 1 job on horizon | 15:53 |
| *** shashank_hegde has joined #openstack-infra | 15:53 | |
| sdague | are you sure it's not just a visualization bug | 15:53 |
| fungi | clarkb: i want to say in the past we've tracked that down to a status rendering issue | 15:54 |
| fungi | but i could certainly be wrong | 15:54 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 15:54 |
| clarkb | sdague: no I am not. I guess I could try to wrap my head around all the json | 15:54 |
| * clarkb tries | 15:54 | |
| *** abashmak has joined #openstack-infra | 15:54 | |
| fungi | or just double-check by grabbing zuul refs from the merger | 15:54 |
| clarkb | fungi: well they are all still tested together | 15:55 |
| sdague | clarkb: for reference https://review.openstack.org/#/c/360563/ is the change that probably triggered this | 15:55 |
| clarkb | just in the horizon queue not the integrated gate queue | 15:55 |
| fungi | yeah, it was purely a job removal | 15:55 |
| jeblair | my guess based on the timestamps is that the previous contents of the integrated queue ended up in the horizon queue, but new changes are going into the correct ones | 15:56 |
| sdague | it should make horizon have an independent gate | 15:56 |
| sdague | but there are enough changes in the pipeline that I can imagine it being confused until everything is fresh in there | 15:56 |
| abashmak | Good morning folks. have a question about using OpenStack with mysql 5.6+ (MariaDB 10.0+). I'm seeing an "foregin key too large" error when accessing DB on Ubuntu 16.04, which uses MariaDB 10.0 | 15:57 |
| jeblair | we should probably do a full restart, otherwise we're going to have 2 integrated gates fighting it out for the next several hours | 15:57 |
| clarkb | jeblair: ok | 15:57 |
| *** kushal has quit IRC | 15:57 | |
| clarkb | another thing I have noticed is that puppet openstack things, tripleo, and a bunch of the python clients all share a queue | 15:57 |
| fungi | oh, so it's not that the horizon change there is actually still running jobs in the integrated queue but zuul thinks it should display it as being in its own queue? | 15:57 |
| clarkb | not sure if that is also related but it means that python-*clients will take up to 3 ish hours to get tested if a puppet/tripleo change is ahead of them | 15:57 |
| *** matbu is now known as matbu|afk | 15:57 | |
| clarkb | which is :( | 15:58 |
| abashmak | Was wondering if someone's run into that. I think I heard OS just moved the gate jobs to 16.04. Wondering how that issue was solved? | 15:58 |
| pabelanger | clarkb: I haven't checked, but this could be a result of tripleo starting to run more jobs in public clouds and other projects | 15:58 |
| pabelanger | clarkb: we likely need to optimize it | 15:58 |
| *** kushal has joined #openstack-infra | 15:58 | |
| fungi | abashmak: openstack deployed from master branch tip of all projects? | 15:58 |
| clarkb | pabelanger: it definitely need to be cleaned up. I don't think we want tripleo bogging down the python clients. And not sure if the puppet folks want that either but I will let them decide | 15:58 |
| pabelanger | clarkb: yup | 15:59 |
| fungi | abashmak: if you're running mitaka, you're probably better off checking with the ubuntu package maintainers since they shipped mitaka packages with ubuntu 16.04 lts | 15:59 |
| fungi | upstream, we didn't test mitaka on 16.04 | 15:59 |
| abashmak | fungi: I'm just deploying individual services one at a time, i.e. keystone, I also built Glance from source | 15:59 |
| *** matthewbodkin has quit IRC | 16:00 | |
| clarkb | also designate is in that queue too | 16:00 |
| jeblair | it looks liek all of the "real" release jobs have happened, it's just the wheel build stuff that's stuck in the queue | 16:00 |
| jeblair | so i think we're safe to restart now | 16:00 |
| *** links has quit IRC | 16:00 | |
| fungi | abashmak: and we're still in the process of getting master branch jobs working on 16.04 actually. i think a lot of our jobs for newton are still running on 14.04 | 16:00 |
| jeblair | i'm thinking: save queue; stop zuul; hard-stop all launchers; start zuul; start all launchers; restore queue | 16:01 |
| jeblair | that should do a full nodepool recycle as well | 16:01 |
| SpamapS | jeblair: how does one save/restore the queue btw? We've been looking at doing that. | 16:02 |
| fungi | abashmak: the qa team in #openstack-qa is likely to know more about whether there are lingering issues with mariadb on ubuntu 16.04 and/or whether they did anything to work around that | 16:02 |
| * SpamapS closes eyes and crosses fingers and mumbles "There's no place like documentation..." | 16:02 | |
| jeblair | SpamapS: http://docs.openstack.org/infra/system-config/zuul.html#restarts | 16:02 |
| abashmak | fungi: thanks much, I'll ping them | 16:02 |
| jeblair | SpamapS: wish granted :) | 16:02 |
| fungi | SpamapS: http://git.openstack.org/cgit/openstack-infra/zuul/tree/tools/zuul-changes.py | 16:02 |
| *** derekh has quit IRC | 16:02 | |
| pabelanger | clarkb: is it possible it is the releasenote jobs? | 16:03 |
| pabelanger | clarkb: that's causing the shared queue | 16:03 |
| jeblair | i didn't see any objection with my plan, so i'm going to proceed... | 16:03 |
| fungi | jeblair: yeah, seems like a reasonable precaution | 16:03 |
| clarkb | pabelanger: the release notes jobs are project specific not a single shared job so I don't think so | 16:03 |
| clarkb | jeblair: yup go for it | 16:03 |
| jeblair | bombs away | 16:04 |
| *** jianghuaw-m has joined #openstack-infra | 16:04 | |
| * SpamapS opens eyes, and hugs the python | 16:04 | |
| sdague | jeblair / clarkb - I think as soon as 349997,1 merges, the visualization probably fixes itself | 16:04 |
| zaro | morning | 16:04 |
| anteaya | morning zaro | 16:05 |
| *** zul has quit IRC | 16:07 | |
| sdague | clarkb: yeh, why is check-osc-clients joining the world? | 16:07 |
| clarkb | sdague: oh is that the job thats doing it? | 16:08 |
| sdague | I think so | 16:08 |
| clarkb | sdague: any shared (transitive) job will do that | 16:08 |
| clarkb | the idea being that its an integration test that should not be broken by not cross testing | 16:08 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 16:08 |
| *** sarob has quit IRC | 16:09 | |
| sdague | right, but we know doing that arbitrarily just means the giant pile of mud | 16:10 |
| fungi | doing it selectively is even more accident-prone though, yeah? | 16:10 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 16:11 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 16:11 |
| fungi | for a lot of these integration test jobs, trying to keep track of all the transitively related projects which should be serialized to confirm the integration test actually integrates changes in the pipeline would be madness | 16:11 |
| mugsie | i seem to remember there was a place to add filters to make sure files from devstack jobs got sent to logstash - but I cannot find it now ... does anyone have any pointers? | 16:12 |
| jeblair | clarkb: re tripleo+puppet -- since the big slow tripleo jobs happen in their own pipeline, if they keep the regulare check/gate jobs to reasonable lengths, it shouldn't necessarily adversely impact other projects sharing their queue. | 16:12 |
| sc68cal | mordred: I'm back | 16:13 |
| EmilienM | fungi: gerrit is super slow for me now | 16:13 |
| frickler | EmilienM: +1, was about to write the same | 16:13 |
| jlvillal | Not responding for me | 16:13 |
| mugsie | fungi: EmilienM: met too - getting 502 proxy Errors | 16:13 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 16:14 |
| clarkb | jeblair: there was a 2.5 hour long job in the shared pipelinr so I dont think they have limitrd the long tests to their pipeline | 16:14 |
| sdague | clarkb: https://github.com/openstack-infra/project-config/blob/a10f839a4b2ad3e60fb920ef9801a96b7087fa13/zuul/layout.yaml#L14112 | 16:14 |
| sdague | that I think is the link | 16:14 |
| clarkb | jeblair: it was running against puppet swift and holding up everything behind it including python-neutronclient | 16:15 |
| *** tonytan_brb has joined #openstack-infra | 16:15 | |
| sdague | which was fine when it wasn't in the gate | 16:15 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 16:15 |
| fungi | i'm seeing 1019 sockets for 443/tcp on review.openstack.org | 16:15 |
| fungi | it's falling now | 16:15 |
| fungi | EmilienM: frickler: mugsie: clarkb surmised yesterday we're maxing out the open sockets on review.o.o | 16:15 |
| sdague | clarkb: removing that lynch pin would probably let the python clients not sit behind giant puppet testing | 16:16 |
| mugsie | ah | 16:16 |
| sdague | clarkb: anyway, dropping again here. Lunch at ops summit, but if you want to decouple, that's the thing I'd remove | 16:16 |
| clarkb | sdague: ok | 16:16 |
| sdague | the osc team I think got way to agressive on gate the world against them to prevent their breaks, which caused this | 16:17 |
| pabelanger | sdague: clarkb: Ya, I think once check-tripleo-jobs landed, the queues merged | 16:17 |
| jeblair | maybe we could look into it a little more before we just remove it -- python clients testing with osc does not sound necessarily like a bad thing | 16:17 |
| pabelanger | since those jobs now run on public clouds | 16:17 |
| fungi | EmilienM: frickler: mugsie: well, file descriptors ultimately i guess? so apache is proxying each connection and when we get to ~1024 that's 2048 file descriptors apache has open | 16:17 |
| sdague | jeblair: it's fine, but when you classifiy the tripleo-client as one of those clients | 16:17 |
| sdague | then it gets weird | 16:17 |
| *** tonytan4ever has quit IRC | 16:17 | |
| sdague | because that cross checks with puppet + tripleo | 16:17 |
| jeblair | sdague: oh, you're saying specifically remove it from tripleoclient | 16:17 |
| sdague | yes | 16:18 |
| jeblair | sdague: and yeah, that does sound like the place to focus :) | 16:18 |
| jeblair | sorry i misparsed | 16:18 |
| sdague | yep, no worries | 16:18 |
| *** dprince has joined #openstack-infra | 16:19 | |
| sdague | ok, dropping, have fun folks :) | 16:19 |
| redrobot | Hi Infra friends! | 16:20 |
| redrobot | I was wondering where I can report an account that is vandalizing the wiki | 16:20 |
| *** abashmak has left #openstack-infra | 16:20 | |
| *** zhurong has quit IRC | 16:20 | |
| redrobot | The user https://wiki.openstack.org/w/index.php?title=User:Donaldsmith2600&redirect=no | 16:20 |
| redrobot | just deleted the blueprint page to my project to redirect it to a spam page about Dell OpenStack support | 16:20 |
| jeblair | it's probably worth considering the proximity of the gerrit slowness to the zuul restart (where zuul will have been issueing many queries) it won't have been using http sockets, but it may have slowed gerrit down enough to generate a backlog | 16:21 |
| fungi | redrobot: i check for them several times a day and clean them up | 16:21 |
| fungi | looking again now | 16:21 |
| redrobot | fungi thanks! | 16:21 |
| fungi | redrobot: i've blocked and deleted new pages created by the two spam users who just showed up in the last hour. one of them moves some existing pages, so i'll unroll that | 16:22 |
| fungi | er, moved | 16:22 |
| redrobot | fungi good deal, thanks! | 16:22 |
| *** sdague has quit IRC | 16:23 | |
| *** raunak has joined #openstack-infra | 16:24 | |
| *** zul has joined #openstack-infra | 16:25 | |
| *** shashank_hegde has quit IRC | 16:25 | |
| rcarrillocruz | pabelanger, clarkb : i created an instance on the infracloud, based off Nodepoo ubuntu trusty dib image | 16:25 |
| rcarrillocruz | 15.184.65.4 | 16:25 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 16:26 |
| rcarrillocruz | now i'm not sure how to test drive a given job on it. In zuul v2.5/3, there's _zuul_ansible folder containing the playbook, inventory and bash scripts for a given test run, however that is executed in a Zuul runner context | 16:26 |
| rcarrillocruz | how you folks tackle that? | 16:26 |
| rcarrillocruz | do you make your test vms a zuul runner and run everything from it? | 16:26 |
| clarkb | rcarrillocruz: I wpuod use xenial and run the reproduce.sh script for a devstack gate run | 16:27 |
| *** fguillot has joined #openstack-infra | 16:27 | |
| rcarrillocruz | so a dsvm | 16:27 |
| rcarrillocruz | ok | 16:27 |
| clarkb | no I just run reproduce.sh and or tox | 16:27 |
| clarkb | clone nova and tox -epy27 | 16:27 |
| rcarrillocruz | and fwiw | 16:27 |
| rcarrillocruz | https://review.openstack.org/#/c/360579/ | 16:27 |
| clarkb | wget reproduce.sh and run it | 16:27 |
| rcarrillocruz | i wip'd it | 16:27 |
| *** gyee has joined #openstack-infra | 16:27 | |
| fungi | redrobot: okay, all unrolled and cleaned up now. a third spammer showed up while i was doing that, but i've now blocked and cleaned up after that one as well | 16:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add mtl01 region to internap on puppetmaster https://review.openstack.org/360622 | 16:28 |
| clarkb | since these are our puvlished run it lcally steps I like to use them | 16:28 |
| *** hockeynu_ has quit IRC | 16:28 | |
| rcarrillocruz | i find potentially the new zuul stuff could be easier for folks to reproduce test runs locally, but didnt' want to go the rabbit hole of figuring zuul runner puppet deps and all | 16:28 |
| pabelanger | mgagne: thanks! Update network information^ | 16:28 |
| rcarrillocruz | just in case anyone had a paste/gist to do it | 16:28 |
| *** tphummel has joined #openstack-infra | 16:28 | |
| rcarrillocruz | https://review.openstack.org/#/c/360579/ is the revert of the infracloud nodepool.yaml, but tweaked a bit for what we have now | 16:29 |
| *** hockeynut has joined #openstack-infra | 16:29 | |
| clarkb | rcarrillocruz: you would need to edit the playbooks too to remove the file copies and probably other items | 16:29 |
| rcarrillocruz | yeah, and the inventory to put the relevant IP | 16:29 |
| rcarrillocruz | but i think it may be doable-ish | 16:29 |
| rcarrillocruz | i'll get back to it when i have sorted all the infracloud bring up | 16:30 |
| openstackgerrit | Zara proposed openstack-infra/infra-manual: Document commit message syntax for StoryBoard https://review.openstack.org/360668 | 16:32 |
| fungi | clarkb: so are you thinking we should be doing something similar to https://git.openstack.org/cgit/openstack-infra/puppet-etherpad_lite/tree/files/apache-connection-tuning in puppet-gerrit? | 16:34 |
| jeblair | rcarrillocruz: we won't be doing the same thing in zuulv3, so i would rather avoid recommending people use the zuul ansible folder to reproduce builds | 16:35 |
| jeblair | rcarrillocruz: i don't want to introduce a feature that we're going to remove | 16:35 |
| pabelanger | clarkb: mordred: we haven't restarted nodepool today right? | 16:36 |
| *** dprince has quit IRC | 16:37 | |
| zigo | Weird, I'm not having any ready nodes, they're not building either, and there's 4 jobs waiting for a VM. Has zuul gone mad? | 16:37 |
| zigo | Or are we out of capacity? | 16:37 |
| clarkb | fungi: ya I think so | 16:37 |
| clarkb | pabelanger: not yet | 16:37 |
| zigo | Ah, building now! :) | 16:38 |
| pabelanger | clarkb: okay, tripleo-test-cloud-rh1 is basically down because is cannot get FIPs | 16:38 |
| rcarrillocruz | sure | 16:38 |
| jeblair | rcarrillocruz: also, it's important that our tests are simple -- as clarkb says, all you really need to do is run 'tox' or 'reproduce.sh' | 16:38 |
| clarkb | jeblair: ++ thats why I try to stick to currently published methods. reproduce.sh and tox work well | 16:38 |
| fungi | clarkb: fwiw, that last burst of problem reports seems to coincide with another spike of similar magnitude to yesterday http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=30&rra_id=all | 16:38 |
| pabelanger | clarkb: which I think mordred has fixed | 16:38 |
| *** e0ne has joined #openstack-infra | 16:39 | |
| clarkb | pabelanger: we are waiting on a few more fixes aiui including one to nodepool | 16:39 |
| clarkb | pabelanger: to handle the dual ip stack of osic | 16:39 |
| fungi | clarkb: my guess is it's topping out at ~1024 open connections but not for a full 5-minute sample so it's only averaging out to ~800 | 16:39 |
| clarkb | fungi: gotcha | 16:39 |
| *** Apoorva has joined #openstack-infra | 16:40 | |
| fungi | that would basically be a 4-minute spike to 1024 | 16:40 |
| pabelanger | clarkb: no problems, I'm going to decrease capacity in tripleo-test-cloud-rh1 to make it lest bursty. We only started having problems when we bumped it | 16:40 |
| *** sshnaidm is now known as sshnaidm|afk | 16:40 | |
| *** Apoorva has quit IRC | 16:40 | |
| * zigo has built 97 packages so far using OpenStack infra | 16:40 | |
| *** jpich has joined #openstack-infra | 16:40 | |
| zigo | pabelanger: It's working very well now! :P | 16:40 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Hide timeline events and comments of private stories https://review.openstack.org/359895 | 16:40 |
| openstackgerrit | Kiall Mac Innes proposed openstack-infra/system-config: Add Designate logs to Logstash https://review.openstack.org/360672 | 16:41 |
| *** hashar has quit IRC | 16:41 | |
| *** Apoorva has joined #openstack-infra | 16:42 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Increase rh1 max-servers to 75" https://review.openstack.org/360673 | 16:42 |
| *** ifarkas is now known as ifarkas_afk | 16:42 | |
| pabelanger | clarkb: ^ | 16:42 |
| pabelanger | zigo: excellent | 16:42 |
| clarkb | mordred: maybe you can tldr the current backlog for nodepool restart? | 16:42 |
| *** dimtruck is now known as zz_dimtruck | 16:42 | |
| *** e0ne has quit IRC | 16:43 | |
| *** roxanagh_ has joined #openstack-infra | 16:43 | |
| pabelanger | slagle: ^ revert tripleo-test-cloud-rh1 to 50nodes, until we restart nodepool | 16:43 |
| slagle | pabelanger: ok, worth trying i guess | 16:44 |
| slagle | i still don't see any erorrs with requesting or associating fip's in the cloud | 16:44 |
| slagle | it happens very quickly | 16:44 |
| pabelanger | another 5mins and nodes will start deleting again | 16:45 |
| *** rcernin has quit IRC | 16:45 | |
| slagle | yes, and last time i saw a bunch of requests come in to create fip's, all were successful | 16:46 |
| slagle | but nothing assigned them to the instances | 16:46 |
| *** gouthamr|afk is now known as gouthamr | 16:46 | |
| jeblair | pabelanger: my process to get fingerprints from all ansible runs on the log server is still running. it's collected 950k fingerprints so far, but has not found the one we're looking for. | 16:46 |
| jeblair | i'll check again tomorrow :) | 16:47 |
| *** roxanagh_ has quit IRC | 16:47 | |
| *** jamesdenton has joined #openstack-infra | 16:48 | |
| pabelanger | jeblair: Oh, nice. I had to revert our ssh-keygen patch in zuul, because ssh-keygen is different between fedora and trusty. I have a fix, will push it up now | 16:48 |
| openstackgerrit | Chris Krelle proposed openstack/diskimage-builder: WIP: A hardware burn-in element. https://review.openstack.org/355675 | 16:48 |
| *** jamesden_ has quit IRC | 16:48 | |
| *** florianf has quit IRC | 16:50 | |
| *** david-lyle has joined #openstack-infra | 16:50 | |
| *** Kiall has quit IRC | 16:51 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Revert "Revert "Store ssh_host_key of remote node"" https://review.openstack.org/360674 | 16:51 |
| pabelanger | jeblair: ^ correct syntax now | 16:51 |
| anteaya | fungi: are the current wiki spammers experiencing all our latest captchas and still able to spam? | 16:51 |
| * anteaya boggles at the fantastic condition of the check, gate and post queues | 16:52 | |
| pabelanger | indeed, looks to be in good shape now | 16:53 |
| *** jamesdenton has quit IRC | 16:53 | |
| pabelanger | minus cinder's OOMs | 16:53 |
| *** dprince has joined #openstack-infra | 16:53 | |
| anteaya | best I've ever seen at this point in the release | 16:53 |
| *** _nadya_ has joined #openstack-infra | 16:53 | |
| anteaya | including cinder's OOMs | 16:53 |
| *** esikachev has quit IRC | 16:54 | |
| *** jamesdenton has joined #openstack-infra | 16:54 | |
| pabelanger | slagle: should have see an attempt that time | 16:55 |
| *** DrifterZA has quit IRC | 16:55 | |
| *** _nadya_ has quit IRC | 16:55 | |
| *** tesseract- has quit IRC | 16:56 | |
| stevemar | AJaeger: regarding the translation sync job break ... do we need to revert the osc change? | 16:57 |
| *** kushal has quit IRC | 16:57 | |
| *** mikelk has quit IRC | 16:58 | |
| *** fguillot_ has joined #openstack-infra | 16:58 | |
| *** tkelsey has joined #openstack-infra | 16:58 | |
| *** fguillot_ has quit IRC | 16:58 | |
| *** fguillot has quit IRC | 16:58 | |
| slagle | pabelanger: yea, 6 new instnaces got spawned and they got fip's | 16:59 |
| *** edtubill has quit IRC | 17:01 | |
| pabelanger | slagle: should see a few more happening now | 17:01 |
| *** edtubill has joined #openstack-infra | 17:01 | |
| pabelanger | slagle: like I said, problem happens a lot more when we burst a lot of nodes a once | 17:02 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: Enable Neutron for the Tempest/Sahara tests https://review.openstack.org/360680 | 17:03 |
| *** weshay_afk is now known as weshay | 17:03 | |
| slagle | pabelanger: maybe we should enable debug logs for nova/neutron in the cloud to see what might be happening | 17:03 |
| *** zul has quit IRC | 17:04 | |
| AJaeger | stevemar: no need to revert the osc change - it should be fine as is with current zuul-cloner. I checked logs and it is fine now. | 17:04 |
| AJaeger | stevemar: you can if you want - should work either way. | 17:04 |
| pabelanger | slagle: if we still have the issue after the latest release of shade and nodepool retart we could do that | 17:04 |
| *** jerryz has joined #openstack-infra | 17:05 | |
| *** hockeynut has quit IRC | 17:05 | |
| *** jkilpatr has quit IRC | 17:05 | |
| slagle | pabelanger: ok. b/c we just assigned 40 fip's almost all at once, and it went fine | 17:05 |
| dougwig | are the gerrit issues a known problem? | 17:05 |
| *** e0ne has joined #openstack-infra | 17:07 | |
| anteaya | dougwig: what are you experiencing? | 17:07 |
| *** sarob has joined #openstack-infra | 17:07 | |
| dougwig | anteaya: overall slow, 502 errors. whole room here at midcycle, across several networks. | 17:07 |
| * AJaeger experiences right now a timeout on every second change in gerrit ;( | 17:07 | |
| *** zz_dimtruck is now known as dimtruck | 17:07 | |
| mordred | pabelanger, clarkb: tl;dr - I'm finally off the phone and now I need to go do some poking at the issue exposed by the nodepool test | 17:07 |
| *** hockeynut has joined #openstack-infra | 17:08 | |
| *** asettle has quit IRC | 17:08 | |
| mordred | I'll give a better tl;dr once I've context switched from the call into the troubleshooting | 17:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add internap-mtl01 to grafana dashboard https://review.openstack.org/360683 | 17:08 |
| *** asettle has joined #openstack-infra | 17:09 | |
| anteaya | dougwig: thank you, yes we are having a problem with how apache is configured regarding its max connections | 17:09 |
| dougwig | anteaya: k, thanks. | 17:09 |
| anteaya | dougwig: it can take more connecitons but the current configuration is telling it it is at its max | 17:09 |
| anteaya | dougwig: thanks for sharing your experience | 17:10 |
| *** kushal has joined #openstack-infra | 17:10 | |
| anteaya | dougwig: I don't think we had hit it before but we had a spike yesterday, then again today | 17:10 |
| stevemar | AJaeger yeah, gerrit needs a restart :P | 17:11 |
| dhellmann | I have a patch for which jenkins claims to have started the gate jobs but that isn't showing up in the zuul status page: https://review.openstack.org/#/c/360612/ | 17:11 |
| anteaya | dougwig: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=30&rra_id=all | 17:11 |
| dhellmann | jenkins posted ~15 minutes ago | 17:11 |
| anteaya | dhellmann: queue lengths just reached zero | 17:11 |
| anteaya | is it there now? | 17:12 |
| dhellmann | anteaya : I don't see it; I thought I saw the event queue lengths at 0 a while back but maybe I had stale data in my browser | 17:12 |
| *** lucasagomes is now known as lucas-dinner | 17:12 | |
| anteaya | hmmm, no I also don't see it | 17:13 |
| *** chem is now known as chem|off | 17:13 | |
| anteaya | and I confirm that starting gate jobs was posted on the patch almost 20 minutes ago | 17:13 |
| dhellmann | anteaya : it seemed weird that jenkins was reporting already if zuul wasn't showing it, but I may have a bad mental model of how those things are related | 17:13 |
| *** asettle has quit IRC | 17:13 | |
| AJaeger | did zuul get a timeout accessing gerrit - like us humans? | 17:13 |
| anteaya | AJaeger: maybe? | 17:14 |
| *** sarob has quit IRC | 17:14 | |
| dhellmann | doesn't zuul tell jenkins when to start the jobs? | 17:14 |
| AJaeger | jeblair: do you have any ideas? ^ or should dhellmann toggle the +W (remove/add it again) ? | 17:14 |
| fungi | anteaya: yes, some spammers are bypassing the newer captcha implementation. i think it's preventing more than the old one did | 17:16 |
| *** flepied1 has joined #openstack-infra | 17:16 | |
| fungi | anteaya: also i reenabled image uploads, which i've confirmed is still not covered by the confirmedit captchas, so even with that the volume has been relatively manageable | 17:16 |
| EmilienM | dhellmann: there is no jenkins anymore | 17:17 |
| anteaya | fungi: okay thank you | 17:17 |
| dhellmann | EmilienM : that makes it even more mysterious :-) | 17:17 |
| *** amotoki has quit IRC | 17:17 | |
| fungi | stevemar: i don't think gerrit needs a restart from what i can see. instead it's getting an increased number of connections lately. we need to tune apache better to handle that volume | 17:17 |
| EmilienM | dhellmann: zuul now runs the jobs with ansible (zuul-launcher) | 17:17 |
| anteaya | dougwig: ^^ | 17:18 |
| EmilienM | dhellmann: http://lists.openstack.org/pipermail/openstack-dev/2016-June/097584.html | 17:18 |
| anteaya | dougwig: fungi's answer to stevemar | 17:18 |
| *** abregman has quit IRC | 17:18 | |
| stevemar | fungi: i just assume gerrit needs a restart whenever it's slow :P | 17:18 |
| dhellmann | EmilienM : I assume it's still just using the old jenkins user in gerrit to announce when the jobs start. Since it did that, what happened to the info about those jobs in the status page? | 17:18 |
| fungi | stevemar: unfortunately, computer software is more subtly complex than that :/ | 17:19 |
| stevemar | fungi: https://media.giphy.com/media/F7yLXA5fJ5sLC/giphy.gif | 17:19 |
| dhellmann | fungi : bah | 17:19 |
| fungi | i'm putting together an apache tuning change now | 17:19 |
| *** hockeynut has quit IRC | 17:19 | |
| stevemar | fungi: just teasing :) | 17:19 |
| *** flepied has quit IRC | 17:19 | |
| fungi | stevemar: great show though ;) | 17:19 |
| EmilienM | dhellmann: jobs in status page ( status.openstack.org/zuul/ ) are still updated | 17:19 |
| jeblair | dhellmann: looking | 17:19 |
| dhellmann | EmilienM : do you see 360612 on your page? | 17:20 |
| *** jkilpatr has joined #openstack-infra | 17:20 | |
| *** dizquierdo has joined #openstack-infra | 17:20 | |
| dhellmann | jeblair : thanks | 17:20 |
| EmilienM | dhellmann: no, but I think it was squeezed during a service interruption, maybe networking or zuul was restarted? | 17:21 |
| *** fguillot has joined #openstack-infra | 17:21 | |
| EmilienM | dhellmann: best thing to do is to do is to reapprove it | 17:21 |
| dhellmann | EmilienM : ok, I'll wait to see what jeblair says first since he's looking at it | 17:21 |
| EmilienM | dhellmann: remove -A and +A againt | 17:22 |
| EmilienM | it will trigger jobs in gate | 17:22 |
| EmilienM | you can also run "recheck" but it will also run check pipeline jobs again, but you don't want that. | 17:22 |
| AJaeger | EmilienM: like I suggested - but let's first jeblair check quickly whether there is a bug, he might have a better idea. I'm sure dhellmann can wait a few minutes... | 17:22 |
| *** senk has quit IRC | 17:23 | |
| dhellmann | yeah, I always like to make sure it's not a symptom of a broader problem. if it's just a hiccup, I can re-approve, but if there's an issue that might affect other folks and this is just an early sign keeping the state of the system as it is may help with debugging. | 17:23 |
| anteaya | dhellmann: thank you | 17:23 |
| *** zul has joined #openstack-infra | 17:24 | |
| pabelanger | rcarrillocruz: okay, have region_name working with ansible-role-cloud-launcher. Tested with creating a security group | 17:24 |
| pabelanger | rcarrillocruz: incoming patch | 17:24 |
| *** ilyashakhat has joined #openstack-infra | 17:25 | |
| *** _nadya_ has joined #openstack-infra | 17:26 | |
| *** hashar has joined #openstack-infra | 17:26 | |
| *** hashar is now known as hasharAway | 17:26 | |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config: Add new worker-model job, and remove 2 legacy dvsm jobs https://review.openstack.org/360695 | 17:26 |
| *** AnarchyAo has joined #openstack-infra | 17:26 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack-infra/bugdaystats: Add "daily" argument to update_stats() https://review.openstack.org/358149 | 17:27 |
| oomichi | pleia2: hi, I drop https://review.openstack.org/#/c/358148/ on ^^^ | 17:28 |
| slagle | pabelanger: so with this latest batch, we ended up with just 8 stuck in building | 17:29 |
| slagle | pabelanger: i was going to try deleting those | 17:29 |
| slagle | pabelanger: will that cause nodepool to try again? | 17:29 |
| *** kaisers_ has joined #openstack-infra | 17:29 | |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config: Add new worker-model job, and remove 2 legacy dvsm jobs https://review.openstack.org/360695 | 17:30 |
| pabelanger | slagle: should be able just to leave it, need to wait 30mins for nodepool to time out | 17:30 |
| *** sarob has joined #openstack-infra | 17:30 | |
| pabelanger | 30min more* | 17:30 |
| *** sarob has quit IRC | 17:31 | |
| slagle | pabelanger: yea, was just wondering if i could short circuit that by deleting them | 17:31 |
| *** sarob has joined #openstack-infra | 17:31 | |
| jeblair | dhellmann, fungi, AJaeger: zuul did not log sending 'Staring gate jobs' to gerrit. this is mysterious. | 17:31 |
| pabelanger | rcarrillocruz: let me know when you have cycles to talk about https://review.openstack.org/360698 | 17:31 |
| dhellmann | jeblair : you're saying that the message that appears on the review isn't in the log file for zuul? | 17:32 |
| dhellmann | or at least a record of having left that message | 17:32 |
| pabelanger | rcarrillocruz: maybe it is better just do use a region_name setting, and drop the with_items | 17:32 |
| *** zul has quit IRC | 17:32 | |
| jeblair | dhellmann: exactly. though zuul separately recorded having received a comment-added event around that time. | 17:32 |
| dhellmann | it sounds like there's a leak in a tube somewhere | 17:33 |
| dhellmann | jeblair : it sounds like there's not much info to debug with, and that this is probably not a wide-spread problem. is it ok to re-approve that patch? | 17:33 |
| jeblair | dhellmann: yep | 17:34 |
| dhellmann | thanks! | 17:34 |
| *** dizquierdo has quit IRC | 17:34 | |
| *** kaisers_ has quit IRC | 17:34 | |
| jeblair | dhellmann: oh, let me enqueue that for you | 17:34 |
| jeblair | it will need a recheck otherwise | 17:34 |
| dhellmann | oh, so just removing the approval and reapplying it isn't enough? | 17:35 |
| jeblair | dhellmann: (zuul did see the workflow +1 but did not act on it because there was no verified+1 from jenkins -- because mystery zuul removed it) | 17:35 |
| dhellmann | aha | 17:35 |
| fungi | [2016-08-25 16:55:49,093 +0000] 521da478 jenkins a/3 gerrit.review.--project.openstack/releases.--message.Starting gate jobs. | 17:36 |
| jeblair | dhellmann: should be in now | 17:36 |
| fungi | that's from the gerrit sshd_log | 17:36 |
| jeblair | fungi: i was just about to go look for that log line | 17:36 |
| *** Kiall has joined #openstack-infra | 17:36 | |
| *** dtantsur is now known as dtantsur|afk | 17:36 | |
| jeblair | fungi: i don't suppose we can trace that back to an originating ip address? | 17:36 |
| *** sambetts is now known as sambetts|afk | 17:36 | |
| fungi | jeblair: checking | 17:36 |
| dhellmann | jeblair : thank you | 17:37 |
| *** sputnik13 has joined #openstack-infra | 17:37 | |
| *** degorenko is now known as _degorenko|afk | 17:37 | |
| *** ddieterly is now known as ddieterly[away] | 17:37 | |
| jeblair | fungi: [2016-08-25 16:06:35,011 +0000] 521da478 jenkins a/3 LOGIN FROM 2001:4800:7815:101:3bc3:d7f6:ff04:e07f | 17:38 |
| fungi | jeblair: according to show-connections... 521da478 16:06:34 00:00:00 a/3 2001:4800:7815:101:3bc3:d7f6:ff04:e07f | 17:38 |
| fungi | jeblair: so yes, confirmed two ways | 17:38 |
| jeblair | reverse dns calls that zuul.openstack.org | 17:38 |
| jeblair | forward dns does not | 17:38 |
| jeblair | nor is that thi ip address on the zuul server. | 17:38 |
| fungi | i wonder if someone rebooted our old zuul server we shut down and never deleted | 17:39 |
| jeblair | so i think we have a rogue zuul.openstack.org wandering around here | 17:39 |
| anteaya | dhellmann: so thanks for reporting | 17:39 |
| fungi | precise server, up 135 days | 17:39 |
| dhellmann | oh, wow | 17:39 |
| fungi | i just now did a `sudo service zuul stop` on it | 17:40 |
| anteaya | did puppet start it up? | 17:40 |
| fungi | i'm checking syslog to find out | 17:40 |
| * dhellmann almost refrains from making ghost in the machine references | 17:40 | |
| fungi | puppet is definitely still active on that server | 17:40 |
| anteaya | dhellmann: thanks for your almost restraint | 17:40 |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config: Add new worker-model job, and remove 2 legacy dvsm jobs https://review.openstack.org/360695 | 17:40 |
| dhellmann | anteaya :-) | 17:40 |
| fungi | courtesy of our ansible inventory considering anything in our openstack tenants a viable target | 17:40 |
| jeblair | aha | 17:41 |
| jeblair | so i "started" it | 17:41 |
| *** ilyashakhat has quit IRC | 17:41 | |
| anteaya | ha ha ha | 17:41 |
| jeblair | when i told ansible to start zuul | 17:41 |
| fungi | gah | 17:41 |
| anteaya | it did what you asked | 17:41 |
| anteaya | you just didn't specify how many | 17:41 |
| *** Jeffrey4l__ has quit IRC | 17:41 | |
| fungi | anyone mind if i poweroff this instance, take a snapshot and then delete it? | 17:41 |
| *** StevenK has quit IRC | 17:41 | |
| jeblair | which, i did to avoid having two gates fighting each other. | 17:41 |
| anteaya | I do not mind | 17:41 |
| jeblair | fungi: how about just deleting it? | 17:41 |
| fungi | jeblair: i'm fine with that if we don't think anybody needs any old files from it | 17:42 |
| *** StevenK has joined #openstack-infra | 17:42 | |
| *** tosky has quit IRC | 17:42 | |
| jeblair | i think it's been quite long enough | 17:42 |
| fungi | infra-root: heads up, i'm deleting the old zuul.openstack.org from before we upgraded to trusty | 17:42 |
| mordred | fungi: ++ | 17:43 |
| *** ilyashakhat has joined #openstack-infra | 17:43 | |
| fungi | also the old eavesdrop and the hacked wiki | 17:44 |
| anteaya | fungi: ack | 17:44 |
| jeblair | presumably, this is something that would be addressed by having unique hostnames -- however, i think we would have a nearly symmetrical problem in that case -- we will need to remember to update playbooks to, eg, reference zuul02.openstack.org. otherwise we will restart the wrong hosts because of a lingering reference to zuul01. | 17:44 |
| dasm | hmm... is it me, or gerrit has problems? | 17:44 |
| anteaya | dasm: we need to tune the apache config, fungi is working on it | 17:44 |
| mordred | jeblair: yah. | 17:44 |
| anteaya | after he finishes removing one too many zuuls | 17:45 |
| dasm | anteaya: thanks. will pass down the info. | 17:45 |
| anteaya | dasm: thank you | 17:45 |
| mordred | jeblair: I think the current thing is clearly deficient. I do not yet have a good idea as to the way to make it less deficient | 17:45 |
| jeblair | turns out you *can* have too many zuuls. | 17:45 |
| mordred | jeblair: sometimes you have all the zuuls | 17:45 |
| jeblair | mordred: yeah, this is maybe where another layer would need to come in (like terraform or etcd or something) that has metadata like "this is the current zuul"? | 17:46 |
| mordred | jeblair: yah | 17:46 |
| *** eharney_ has joined #openstack-infra | 17:47 | |
| pabelanger | rcarrillocruz: mordred: lets try https://review.openstack.org/#/c/360698 for region_names in cloud-launcher | 17:47 |
| *** sdague has joined #openstack-infra | 17:48 | |
| fungi | we've still got duplicate ci-backup-rs-ord servers, puppetdb servers (3 if you include puppetdb01), review-dev servers (i think ianw is working on that though), and static servers | 17:48 |
| jeblair | anteaya: do you have a few mins to help me clean this up? | 17:48 |
| fungi | i'm pretty comfortable deleting the old static.o.o now, but i don't know the state for the backup and puppetdb servers | 17:49 |
| anteaya | jeblair: how can I be of assistance? | 17:49 |
| jeblair | anteaya: these are the changes that may have similarly been affected: https://etherpad.openstack.org/p/hStqJuyi3o | 17:49 |
| anteaya | jeblair: and yes, I have time if you tell me what I can do that you would consider helpful | 17:49 |
| anteaya | only 19 of them | 17:49 |
| *** eharney has quit IRC | 17:49 | |
| jeblair | anteaya: we need to go through them manually and look for the same circumstance -- the last comment from 'jenkins' being 'starting gate jobs' | 17:49 |
| *** eharney_ is now known as eharney | 17:50 | |
| jeblair | anteaya: then i can enqueue those | 17:50 |
| fungi | okay, old static.o.o at 192.237.219.31 has been deleted now too | 17:50 |
| jeblair | anteaya: can you work down from the top, and i'll work up from the bottom? | 17:50 |
| anteaya | so from that list find the ones that have a starting gate jobs comment on them | 17:50 |
| anteaya | yes | 17:50 |
| *** hockeynut has joined #openstack-infra | 17:50 | |
| *** hockeynut has quit IRC | 17:51 | |
| * anteaya notes gerrit is slow :) | 17:51 | |
| *** ilyashakhat has quit IRC | 17:51 | |
| * clarkb returns from proper breakfast | 17:51 | |
| clarkb | mordred: what issue was exposed? is this the thing your nodepool change is intended to fix? | 17:52 |
| jeblair | anteaya: let's make the ones that need re-enquing bold, and the ones that don't strikthrough | 17:52 |
| mordred | clarkb: yes. ish. | 17:52 |
| mordred | clarkb: can you look at this: http://paste.openstack.org/show/563437/ | 17:52 |
| mordred | clarkb: and tell me if you agree with the english? | 17:52 |
| anteaya | jeblair: very good | 17:53 |
| clarkb | mordred: I agree with your english | 17:53 |
| anteaya | jeblair: to make sure I'm doing this right, this patch: https://review.openstack.org/#/c/360654 | 17:53 |
| mordred | clarkb: ok. good | 17:53 |
| *** sandanar has quit IRC | 17:53 | |
| anteaya | jeblair: does not have the required comment, correct? | 17:53 |
| mordred | clarkb: I need to make one more change to make that english true | 17:53 |
| mordred | and then we'll be good | 17:53 |
| *** hockeynut has joined #openstack-infra | 17:53 | |
| * anteaya strikes through https://review.openstack.org/#/c/360654 | 17:54 | |
| *** rcernin has joined #openstack-infra | 17:54 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 17:54 |
| jeblair | anteaya: i will tell you in just a few moments once it loads :) | 17:54 |
| jeblair | anteaya: correct | 17:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add internap to cloud_launcher https://review.openstack.org/360707 | 17:55 |
| jeblair | anteaya: (and be sure you have toggle ci set to show you the ci comments) | 17:55 |
| pabelanger | mordred: rcarrillocruz: example how regions work in cloud-launcher ^ | 17:55 |
| *** tkelsey has quit IRC | 17:55 | |
| *** pt_15 has quit IRC | 17:56 | |
| anteaya | jeblair: thank you, and yes toggle ci is set | 17:56 |
| jeblair | anteaya: oh i just remembered one more thing i'll need, sorry | 17:57 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Tune review.o.o for more connections https://review.openstack.org/360710 | 17:57 |
| fungi | clarkb: ^ a first stab | 17:57 |
| jeblair | anteaya: can you add the project name for each change to the etherpad? | 17:57 |
| jeblair | anteaya: well, at least the bold ones | 17:57 |
| clarkb | the cinder python jobs are the ones OOMing right? I am skimming the ansible ssh connection errors and cinder python jobs show up | 17:57 |
| clarkb | gonna guess related to OOM if that is the test that was mentioned way back ago in scrollback | 17:58 |
| *** nwkarsten has quit IRC | 17:58 | |
| fungi | clarkb: gate-cinder-python34-db | 17:58 |
| fungi | on trusty, obviously | 17:58 |
| fungi | not sure if others are as well | 17:58 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 17:59 |
| fungi | but i saw that one pop this morning in the gate | 17:59 |
| clarkb | I am seeing the python35 job on xenial error with the ssh errors | 17:59 |
| fungi | ahh, py35 is nonvoting for cinder still | 17:59 |
| clarkb | yup | 17:59 |
| clarkb | I am going to attribute that to the OOMing | 17:59 |
| AJaeger | and also gate-cross-cinder-python27-db-ubuntu-xenial | 17:59 |
| *** jcoufal has joined #openstack-infra | 17:59 | |
| AJaeger | that is run on requirements changes | 17:59 |
| clarkb | pabelanger: so maybe we do want working cgroups | 17:59 |
| fungi | okay, so 2.7 (in some variant) is seeing it as well | 17:59 |
| clarkb | pabelanger: to constrain the tests just enough that ssh can continue working | 17:59 |
| *** david-lyle has quit IRC | 18:00 | |
| * clarkb skims the list to see if there are any other jobs causing problems | 18:00 | |
| pabelanger | clarkb: just readying backscroll | 18:00 |
| AJaeger | fungi, I only remember cinder-python27-db-ubuntu-xenial - not seeing 34 myself. | 18:01 |
| AJaeger | might happen less often on 3.4 | 18:01 |
| rcarrillocruz | Sorry, away from home now | 18:01 |
| *** dstufft has quit IRC | 18:01 | |
| pabelanger | clarkb: however, we should be using cgroups in xenial. Last patch was to add systemd support | 18:01 |
| *** dstufft has joined #openstack-infra | 18:01 | |
| anteaya | jeblair: project name, ack, will start now and will go back once finished | 18:01 |
| pabelanger | slagle: should be online now | 18:02 |
| *** senk has joined #openstack-infra | 18:02 | |
| fungi | AJaeger: i linked the logs for one in here earlier. finding it again | 18:02 |
| *** ihrachys has quit IRC | 18:02 | |
| openstackgerrit | Varun Gadiraju proposed openstack-infra/project-config: added first block of code, tox still works https://review.openstack.org/360713 | 18:03 |
| openstackgerrit | Varun Gadiraju proposed openstack-infra/project-config: added ironic_build_deploy_ramdisk if statements https://review.openstack.org/360714 | 18:03 |
| *** rhallisey has quit IRC | 18:03 | |
| openstackgerrit | Varun Gadiraju proposed openstack-infra/project-config: works without %% agent https://review.openstack.org/360715 | 18:03 |
| openstackgerrit | Varun Gadiraju proposed openstack-infra/project-config: agent condition working https://review.openstack.org/360716 | 18:03 |
| openstackgerrit | Varun Gadiraju proposed openstack-infra/project-config: added code for BUILD_TIMEOUT=${DEVSTACK_GATE_TEMPEST_BAREMETAL_BUILD_TIMEOUT:-600} https://review.openstack.org/360717 | 18:03 |
| openstackgerrit | Varun Gadiraju proposed openstack-infra/project-config: ironic deploy driver set https://review.openstack.org/360718 | 18:03 |
| openstackgerrit | Varun Gadiraju proposed openstack-infra/project-config: fixed weird equals sign inclusion https://review.openstack.org/360719 | 18:03 |
| *** sdake has quit IRC | 18:03 | |
| clarkb | pabelanger: ok maybe its not applying the memory limits to the test processes in a way that protects the ssh process | 18:03 |
| fungi | AJaeger: http://logs.openstack.org/37/355837/3/gate/gate-cinder-python34-db/7b49f24/console.html#_2016-08-25_13_42_12_352115 | 18:03 |
| clarkb | prometheanfire: that could happen because they all share the same user maybe | 18:03 |
| anteaya | jeblair: I'll go back and add project names now | 18:03 |
| *** gyee has quit IRC | 18:03 | |
| clarkb | http://paste.openstack.org/show/563439/ this is the list off of zl01's current launcher-debug log | 18:03 |
| clarkb | most of it is cinder | 18:04 |
| *** david-lyle has joined #openstack-infra | 18:04 | |
| anteaya | jeblair: on line two the patch hasn't been seen by zuul at all, but it hadn't gotten a comment about gate jobs, so i struckthrough it | 18:04 |
| anteaya | many of the patches I saw had merged | 18:04 |
| clarkb | we have ovn and neutron and fuel jobs too. Would not surprise me if those three just happen to break networkign on the host for some reason | 18:04 |
| pabelanger | clarkb: yes, that is possible. I didn't actually test cgroups on xenial | 18:04 |
| *** sdake has joined #openstack-infra | 18:04 | |
| prometheanfire | clarkb: mistab? | 18:04 |
| AJaeger | fungi, I believe you - and glad for the info. I was concentrating more on the 27 ones but looking at clarkb's list, it seems spread evenly... | 18:05 |
| clarkb | prometheanfire: yes meant to get pabelanger | 18:05 |
| *** tqtran has joined #openstack-infra | 18:05 | |
| jeblair | anteaya: i think i won't dig into that one. | 18:05 |
| anteaya | jeblair: very good | 18:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Remove 'running' as a public method from builder https://review.openstack.org/359950 | 18:06 |
| pabelanger | project-config-core: when you have cycles, if you could help land 360673. We are decreasing capacity in tripleo-test-cloud-rh1 until we schedule a nodepool restart | 18:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Simplify builder start/stop methods https://review.openstack.org/359951 | 18:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Remove unecessecary builder scheduler thread https://review.openstack.org/359949 | 18:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Rename BuilderScheduler to NodePoolBuilder https://review.openstack.org/359952 | 18:06 |
| *** jkilpatr has quit IRC | 18:06 | |
| clarkb | fungi: did you forget to git add the tuning file? | 18:06 |
| *** yamahata has quit IRC | 18:06 | |
| *** tqtran_ has joined #openstack-infra | 18:07 | |
| *** vgadiraj has joined #openstack-infra | 18:07 | |
| mordred | clarkb: if you have to ask to see the tuning file, you're not ready for the tuning file | 18:07 |
| pabelanger | clarkb: still trying to catch up to the issue you are debugging. Can you summarize please? | 18:07 |
| vgadiraj | Hiwas multitasking, messed up, and a bunch of local commits on my machine got pushed to gerrit as individual commit | 18:07 |
| vgadiraj | sorry all | 18:07 |
| clarkb | pabelanger: which one? the continuation of the ansible rc 3? that appears to be related in part to cinder python tests OOMing which breaks ssh | 18:07 |
| fungi | clarkb: indeed i did. just a sec | 18:08 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Tune review.o.o for more connections https://review.openstack.org/360710 | 18:08 |
| fungi | clarkb: ^ now with more git add | 18:08 |
| *** ociuhandu has quit IRC | 18:09 | |
| anteaya | jeblair: I think I am done | 18:09 |
| jeblair | anteaya: thanks! | 18:09 |
| anteaya | jeblair: is there anything else I can do to help | 18:09 |
| anteaya | jeblair: thanks for asking me | 18:09 |
| fungi | clarkb: though i just spotted a typo. another patchset incoming | 18:10 |
| *** tqtran has quit IRC | 18:10 | |
| jeblair | anteaya: i think that's it, i just need to transform these into commands now and run them | 18:10 |
| anteaya | jeblair: very good | 18:10 |
| anteaya | thank you | 18:10 |
| clarkb | fungi: ok I was really confused and checking the current master wondering if the file existed already :) | 18:10 |
| *** shashank_hegde has joined #openstack-infra | 18:10 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Tune review.o.o for more connections https://review.openstack.org/360710 | 18:10 |
| fungi | clarkb: ^ there we go | 18:10 |
| *** david-lyle has quit IRC | 18:11 | |
| clarkb | pabelanger: I don't think fixing that is hugely urgent. The scope and frequency of it is small and localized to cinder mostly | 18:11 |
| clarkb | pabelanger: just something to keep in mind if we want to make ssh reliability better | 18:12 |
| pabelanger | clarkb: okay, let me finish getting internap working, and I then see how to debug OOMs | 18:12 |
| fungi | clarkb: we'll need to manually reload apache with that version... i wasn't sure how to subscribe an apache reload to a file being added in the calling class there. we could get fancier with puppeting it through into the gerrit module as an optional/templated thing if you think that's warranted | 18:12 |
| *** david-lyle has joined #openstack-infra | 18:12 | |
| clarkb | fungi: I think those turning parameters require a full restart anyways | 18:12 |
| fungi | clarkb: oh, yep you're right | 18:13 |
| fungi | we likely also want to consider adding the stats module like we're using on etherpad (or have we done that already? i didn't even look) | 18:13 |
| pabelanger | rcarrillocruz: crinkle: looks like infracloud is on the alternate list for Barcelona | 18:13 |
| *** hockeynut has quit IRC | 18:13 | |
| clarkb | I don't think we have added the stats module anywhere else | 18:13 |
| *** zul has joined #openstack-infra | 18:13 | |
| fungi | yeah, without adding it in, we're sort of blind to the results of adjusting these values | 18:14 |
| clarkb | want to add that in a parent change and we can do both together? | 18:14 |
| * clarkb reviews the tuning params now | 18:14 | |
| *** david-lyle_ has joined #openstack-infra | 18:15 | |
| *** julim_ has quit IRC | 18:15 | |
| fungi | i wonder if we should also tune the jvm memory allocation back down a bit to avoid additional apache threads increasing memory pressure? | 18:15 |
| *** david-lyle_ has quit IRC | 18:15 | |
| fungi | clarkb: i need to step away for about 20 minutes to help chris pick up her car from the mechanic, and then i can work on the server-status module change | 18:16 |
| clarkb | top says apache is using so little memory | 18:16 |
| *** WJMUqqJZRD has quit IRC | 18:16 | |
| fungi | yeah, we likely don't need any tweaks to the jvm for this then | 18:16 |
| jeblair | fungi, clarkb: remember that gerrit has tuning parameters too regarding how many requests it services at once | 18:16 |
| clarkb | and we have half a gig of free memory apache can grow into with about 10GB of cache/buffers | 18:16 |
| clarkb | jeblair: I see 100 threads for sshd and a maxqueued value of 200 for gerrit | 18:17 |
| clarkb | sshd doesn't go through apache so I think we can mostly ignore that | 18:18 |
| * clarkb reads docs on maxqueued for httpd | 18:18 | |
| jeblair | there's something about the fact that ssh+http access methods actually sharing workers though -- i put comments in the gerrit module about it | 18:18 |
| clarkb | ok will look there too | 18:19 |
| *** jkilpatr has joined #openstack-infra | 18:19 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Change default for FORCE_CONFIG_DRIVE to False https://review.openstack.org/357443 | 18:19 |
| jeblair | is anyone aware of problems with the deb-* repos on github? | 18:20 |
| openstackgerrit | Chris Krelle proposed openstack-infra/glean: Adjust the way we wait for interfaces to become available https://review.openstack.org/359471 | 18:20 |
| jeblair | it looks like many (all?) of them lack team acls | 18:20 |
| jeblair | so gerrit is unable to push there | 18:20 |
| jeblair | which is why i'm seeing 0f0ac405 waiting .... 18:11:24.780 (retry 2819) [681463c0] push git@github.com:openstack/deb-os-client-config.git | 18:21 |
| jeblair | in the gerrit queue | 18:21 |
| *** tphummel has quit IRC | 18:21 | |
| *** NomNKDJIvx has joined #openstack-infra | 18:21 | |
| *** senk has quit IRC | 18:21 | |
| clarkb | looks like there is an httpd.minthreads and httpd.maxthreads. Defaults for both are 5 and 25 respectively and we don't set either | 18:21 |
| *** tphummel has joined #openstack-infra | 18:21 | |
| jeblair | anyone who cares about github want to fix that, or should we just turn it off? :) | 18:21 |
| clarkb | jeblair: I want to say thats a problem with the github api throttling | 18:22 |
| *** dizquierdo has joined #openstack-infra | 18:22 | |
| clarkb | jeblair: users only get like 5k requests per hour and we easily cross that threshold when creating that many projects at once | 18:22 |
| jeblair | clarkb: yeah, likely we were throttled when they were all created, so now they are broken | 18:22 |
| AJaeger | when I review the "Normalize projects.yaml" changes, I always check that github is uptodate | 18:22 |
| *** esikachev has joined #openstack-infra | 18:22 | |
| AJaeger | Seems that this check was not done with https://review.openstack.org/357032 ;( | 18:23 |
| jeblair | AJaeger: i'm not sure that would have helped | 18:23 |
| jeblair | AJaeger: i think the issue is that the projects were created in github, but adding acls failed | 18:23 |
| jeblair | AJaeger: and i don't think jeepyb is smart enough to detect that and correct it in a subsequent run | 18:24 |
| AJaeger | jeblair: What I mean: I only approve those changes when there's content - and if not, I speak up here. | 18:24 |
| clarkb | jeblair: looks like the relationship between httpd thraeds and sshd threads is that we need to have more db threads than the max of sshd and httpd | 18:24 |
| *** e0ne has quit IRC | 18:24 | |
| AJaeger | both yesterday and today I had to ping yolanda to fix new repos... | 18:24 |
| clarkb | fungi: I think we should also consider increasing the gerrit parameters, maybe even start there | 18:25 |
| AJaeger | jeblair: 279 repos were created at that day ;( | 18:25 |
| clarkb | fungi: increase httpd.maxthreads to ~100, minthreads to ~20, then increase db threads to whatever the math comes out to | 18:25 |
| clarkb | then keep a really close eye on memory use | 18:26 |
| clarkb | jeblair: does gertty use a persistent http connection? I wonder if as the number of gertty users goes up the free slots fall drastically | 18:26 |
| apuimedo | AJaeger: I'd need to have myself added to https://review.openstack.org/#/admin/groups/1542,members kuryr-libnetwork-core and also to kuryr-kubernetes-core | 18:27 |
| apuimedo | otherwise I can't manage to add other people | 18:27 |
| apuimedo | nor to add kuryr-core as a subgroup of the other two groups | 18:27 |
| pabelanger | jeblair: clarkb: fungi: When you have time to spare, 360622 adds internap-mtl01 to puppetmaster.o.o. | 18:27 |
| apuimedo | these two groups and the acls are very recent | 18:27 |
| *** tqtran_ has quit IRC | 18:28 | |
| AJaeger | apuimedo: I cannot help you with those. Please give the link to the review as reference and somebody else with admin nights will most likely pick it up. | 18:28 |
| *** tqtran has joined #openstack-infra | 18:28 | |
| apuimedo | AJaeger: do you know who? | 18:28 |
| AJaeger | It's nearly night time - I meant admin rights ;9 | 18:28 |
| *** esikachev has quit IRC | 18:28 | |
| clarkb | ya I think httpd.maxthreads to 100 to match sshd threads amkes sense. Then we can support roughly the same number of incoming requests over either protocol | 18:28 |
| apuimedo | this is the other group | 18:28 |
| apuimedo | https://review.openstack.org/#/admin/groups/1542,members | 18:28 |
| AJaeger | apuimedo: just ask here in the round, the admins are around. And please link to the reviews! | 18:29 |
| anteaya | AJaeger: admin nights, I don't want any of them | 18:29 |
| *** nwkarsten has joined #openstack-infra | 18:29 | |
| *** pvaneck has joined #openstack-infra | 18:29 | |
| jeblair | clarkb: there is a suggestion in http://docs.python-requests.org/en/master/user/advanced/ that using the session object gets you a persistent connection. | 18:29 |
| apuimedo | https://review.openstack.org/357946 | 18:29 |
| clarkb | jeblair: zaro I think you all have way more experience tuning gerrits though. What do you think? | 18:29 |
| anteaya | AJaeger: but admin knights might be useful | 18:29 |
| jeblair | clarkb: so maybeso. | 18:29 |
| apuimedo | https://review.openstack.org/357945 | 18:29 |
| apuimedo | these are the reviews | 18:29 |
| apuimedo | :-) | 18:29 |
| jeblair | clarkb: though those connections will still timeout quickly | 18:29 |
| AJaeger | anteaya: yeah, we should ask for admin knights instead of infra admins ;) | 18:29 |
| anteaya | we could see what kind of response we get | 18:30 |
| *** electrofelix has quit IRC | 18:30 | |
| *** roxanaghe has joined #openstack-infra | 18:31 | |
| anteaya | new high for me: Queue lengths: 6783 events, 2790 results. it is a gate reset, just high numbers | 18:31 |
| zaro | clarkb: yeah, that's a good idea. | 18:32 |
| clarkb | jeblair: and the trade off there is when gertty needs to make a bunch of requests it uses a single connection rather than many so probably better overall | 18:32 |
| jeblair | i can not believe we broke a url across lines in puppet just becaues the line was too long | 18:32 |
| AJaeger | anteaya: I've seen over 11k some time ago ;( | 18:32 |
| jeblair | please never ever ever ever never ever do that | 18:32 |
| *** rhallisey has joined #openstack-infra | 18:33 | |
| anteaya | AJaeger: ah | 18:33 |
| *** Sukhdev has joined #openstack-infra | 18:33 | |
| AJaeger | anteaya: could you review https://review.openstack.org/360673 to help with tripleo cloud, please? | 18:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Fix URL in gerrit tuning comments https://review.openstack.org/360726 | 18:33 |
| AJaeger | anteaya: but 6k is already high and not a normal value | 18:33 |
| anteaya | AJaeger: once gerrit lets me | 18:34 |
| AJaeger | ;/ | 18:34 |
| anteaya | lots moved on in the gate, so yay! | 18:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Fix URL in gerrit tuning comments https://review.openstack.org/360726 | 18:34 |
| *** zul has quit IRC | 18:34 | |
| zaro | clarkb: and maybe set the others to match, acceptorThreads and maxQueued? | 18:34 |
| clarkb | zaro: maxqueued is already 200 I think we can leave it that way | 18:34 |
| jeblair | due to the ssh slowness, i'm more inclined to say the issue is closer to gerrit than apache. | 18:34 |
| *** tqtran has quit IRC | 18:35 | |
| clarkb | jeblair: ya I think you may be right that we start seeing the apache issues because gerrit is not keeping up forcing apache to queue | 18:35 |
| jeblair | clarkb: did you read https://groups.google.com/forum/#!topic/repo-discuss/uRSRwYUpWnE | 18:35 |
| clarkb | jeblair: not yet just the comments in the puppet | 18:35 |
| jeblair | sshd.threads - number of Git fetch or push threads for the entire | 18:35 |
| jeblair | server. These threads are shared between the SSHD and HTTPD. | 18:35 |
| *** roxanaghe has quit IRC | 18:35 | |
| jeblair | that's what i was remembering | 18:35 |
| clarkb | jeblair: reading current docs I don't knwo if thats true anymore | 18:35 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Revert "set check-osc-job to voting" https://review.openstack.org/360728 | 18:35 |
| jeblair | ok | 18:35 |
| clarkb | jeblair: the httpd section has its own min and max threads. 5 and 25 by default | 18:35 |
| jeblair | we should probably update the comments then :) | 18:36 |
| jeblair | once we figure out how things work again | 18:36 |
| clarkb | ++ | 18:36 |
| sdague | jeblair / clarkb ok, I looked into the client pipeline issue, and proposed the revert https://review.openstack.org/360728 | 18:36 |
| AJaeger | stevemar: please see ^ | 18:37 |
| sdague | because I think that test architecture for collision detection is going to grow forever | 18:37 |
| *** salv-orl_ has quit IRC | 18:37 | |
| sdague | AJaeger: yeh, I've been talking with stevemar, I did ask him to comment on it | 18:37 |
| AJaeger | sdague: thanks | 18:37 |
| clarkb | jeblair: specifically see https://review.openstack.org/Documentation/config-gerrit.html#httpd compared to https://review.openstack.org/Documentation/config-gerrit.html#sshd | 18:37 |
| clarkb | jeblair: granted this assumes the docs are correct :) | 18:38 |
| *** ddieterly[away] has quit IRC | 18:38 | |
| jeblair | sdague: okay, well, that comment i said earlier now applies :) | 18:38 |
| sdague | jeblair: it does | 18:38 |
| clarkb | but it sounds like our intent was to have 100 httpd threads in the past so upping to that value now makes even more sense to me | 18:38 |
| *** asettle has joined #openstack-infra | 18:38 | |
| *** asettle has quit IRC | 18:38 | |
| *** asettle has joined #openstack-infra | 18:39 | |
| stevemar | sdague: ah this only causes an issue cause the gates queues things together | 18:39 |
| jeblair | sdague: what was wrong with just removing it from tripleo? | 18:39 |
| *** ddieterly has joined #openstack-infra | 18:39 | |
| sdague | jeblair: because the intent of the job is to do that with everything with osc plugin | 18:39 |
| sdague | so we can start saying "you can do this, unless you also do X" | 18:39 |
| sdague | but that gets pretty complicated | 18:39 |
| sdague | and doesn't actually give them the coverage they are looking for | 18:40 |
| sdague | stevemar: right, that's the point of the gate model :) | 18:40 |
| zaro | clarkb: https://groups.google.com/forum/#!topic/repo-discuss/-2hnEOqQA7M | 18:40 |
| zaro | suggestions on those settings | 18:40 |
| jeblair | sdague: oh, so this is not testing that these things work together, they're just testing that they don't have colliding commands? | 18:41 |
| sdague | jeblair: yes | 18:41 |
| *** marcusvrn_ has joined #openstack-infra | 18:41 | |
| *** dizquierdo has quit IRC | 18:41 | |
| stevemar | sdague: i think i'd prefer de-coupling (and risking the chance of a wedge), rather than making it check and non-voting (no one was reading why non-voting jobs fail) | 18:41 |
| jeblair | i agree that throwing a shared queue at that problem may be heavyweight | 18:41 |
| fungi | okay, i'm back now and catching up | 18:42 |
| stevemar | jeblair: i think they only share a queue because the jobs has the same name, i didn't know that :\ | 18:42 |
| sdague | stevemar: but that's the point | 18:42 |
| jeblair | stevemar: well, the job hase the same name because it's the same job :) | 18:43 |
| *** kaisers_ has joined #openstack-infra | 18:43 | |
| sdague | I think there is enough confusion here about what is wanted we should revert | 18:43 |
| jeblair | stevemar: so it is correct that they share a queue | 18:43 |
| sdague | and approach again next cycle | 18:43 |
| *** yamahata has joined #openstack-infra | 18:44 | |
| jeblair | stevemar: (the idea is that if a job uses both projects A and B, then those projects both influence the outcome of the job, so they should always be tested together and should share a queue) | 18:44 |
| jeblair | this is quite likely exactly the case here | 18:44 |
| sdague | jeblair: it's definitely the case here, but as you said, the mechanism being used is super heavy weight for this | 18:45 |
| jeblair | i think sdague is just arguing that doesn't scale to all projects that might possibly have an osc plugin, so some other approach might be warranted | 18:45 |
| stevemar | jeblair: so it sounds like its correct :P | 18:45 |
| sdague | jeblair: right, exactly | 18:45 |
| AJaeger | yeah, having a shared queue avoids the race where two projects both define the same command and merge at the same time... | 18:45 |
| sdague | this is solving a social problem with a massive CI solution | 18:45 |
| sdague | that has a lot of negative fallout | 18:45 |
| AJaeger | sdague: yeah, the cost is high. | 18:45 |
| *** tqtran has joined #openstack-infra | 18:45 | |
| openstackgerrit | Chris Krelle proposed openstack/diskimage-builder: WIP: A hardware burn-in element. https://review.openstack.org/355675 | 18:46 |
| *** hasharAway is now known as hashar | 18:46 | |
| sdague | so, lets roll it back now. Make it one of the osc discussion points in barcelona, try to get a better solution there | 18:46 |
| stevemar | and i guess we can't make the check job voting if there is no gate job? | 18:46 |
| krotscheck | infra-core: I'm going to cease my activities supporting the various javascript efforts here in infra, in preparation of joining a new non-openstack employer. As it's become pretty clear that after pushing this effort for two years, there's little-to-no community support for advanced UI development, I'm saving myself the continued frustration and am throwing in the towel. I wish all of you the best of luck, it's been a blast. | 18:46 |
| AJaeger | stevemar: you could - but that would allow two cchanges to merge with same commands in it. | 18:47 |
| sdague | stevemar: and wedge | 18:47 |
| sdague | and block everyone until you could chase down fixes in one of those projects | 18:47 |
| AJaeger | stevemar: imagine both test in the check and then both go through gate - and both go in, and then nobody will be able to merge change until this is reverted | 18:47 |
| AJaeger | what sdague says ;) | 18:47 |
| sdague | anyway, the week before freeze is probably not a time to figure out a novel solution for osc here | 18:47 |
| marcusvrn_ | asselin: ping | 18:48 |
| jeblair | krotscheck: thank you for the excellent work you've done, and for letting us know. i particularly enjoyed working with you on the mirror project. :) | 18:48 |
| sdague | and the world was ok before 7 weeks ago | 18:48 |
| sdague | so reverting to that state seems not terrible | 18:48 |
| stevemar | eh | 18:48 |
| sdague | and make this part of barcelona discussions | 18:48 |
| * AJaeger hugs krotscheck and wishes him all the best. Sad to see you move on | 18:48 | |
| stevemar | it gave me piece of mind knowing they couldn't stomp | 18:48 |
| sdague | stevemar: at a huge cost | 18:48 |
| stevemar | yeah, alright | 18:49 |
| stevemar | i will write to the ML and let the plugins know | 18:49 |
| *** YqwHkAMaYX has joined #openstack-infra | 18:49 | |
| stevemar | hopefully they read why things are failing and actually check | 18:49 |
| jeblair | is the problem workable if we drop tripleo from that? | 18:49 |
| pabelanger | krotscheck: yes, thanks for the work you've done. Like jeblair, the mirror projects was a fantastic base | 18:49 |
| sdague | jeblair: it slices it down a bit, it's still gluing quite a bit together | 18:49 |
| fungi | clarkb: jeblair: okay, caught up on the scrollback now and i agree we should look at gerrit performance tuning before jumping to apache tuning | 18:50 |
| sdague | but, I really think that someone is going to readd tripleoclient later because they see it missing | 18:50 |
| sdague | and add some other new client | 18:50 |
| fungi | i'm going to see if i can figure out anything interesting from javamelody too | 18:50 |
| melwitt | hi, gerrit is extra slow right now and returning 502s sometimes. just reporting it in case it hasn't been reported yet | 18:50 |
| sdague | which joins in 6 other projects | 18:50 |
| *** krotscheck has left #openstack-infra | 18:50 | |
| jeblair | melwitt: it has but we keep getting distracted | 18:50 |
| anteaya | melwitt: thanks, we are working on a fix | 18:51 |
| pabelanger | AJaeger: So, I'm think maybe we should revert back to 20 min-ready for both ubuntu-trusty and ubuntu-xenail? Now that we have the capacity again | 18:51 |
| *** salv-orlando has joined #openstack-infra | 18:51 | |
| anteaya | melwitt: right now we are looking at options to tune apache and gerrit | 18:51 |
| fungi | melwitt: usage seems to have increased drastically this week | 18:51 |
| *** jcoufal has quit IRC | 18:51 | |
| melwitt | okay, thanks all | 18:51 |
| anteaya | melwitt: it is being told it can only handle so many connections, and we are trying to figure out how to tell it it can handle more than that | 18:52 |
| anteaya | melwitt: thanks for reporting | 18:52 |
| *** ilyashakhat has joined #openstack-infra | 18:52 | |
| AJaeger | pabelanger: looking at the constant demand we have for these, it seems reasanable. | 18:52 |
| jeblair | sdague: i'm also wondering how much of an issue this is in practice -- i mean, the report was basically clarkb looking at the status page and saying 'huh that's weird' | 18:52 |
| *** sarob has quit IRC | 18:52 | |
| melwitt | anteaya: a-ha, cool | 18:52 |
| * dtroyer_zz just caught up on the gate question | 18:52 | |
| sdague | jeblair: my experience is that these things continue to grow until the whole universe is joined together | 18:53 |
| dtroyer_zz | I'm good with dropping that job for now, we do need a better solution to the command problem | 18:53 |
| anteaya | dtroyer_zz: hello, welcome, I take it your travel was successful | 18:53 |
| jeblair | sdague: so i'm wondering whether a scaled-down approach would be okay till bcn | 18:53 |
| fungi | jeblair: clarkb: worth noting, javamelody reports the gc activity picked up mostly out of nowhere just around 3 hours ago, so another gerrit restart will likely relieve some of this now too | 18:53 |
| dtroyer_zz | anteaya: layover, one more flight | 18:53 |
| anteaya | dtroyer_zz: ah, happy layover to you | 18:53 |
| jeblair | fungi: around when i restarted zuuls | 18:53 |
| sdague | dtroyer_zz: you want to +1 the review? | 18:54 |
| fungi | but if we want to implement some tuning changes, we need a restart anyway, so that seems like it might be well-timed | 18:54 |
| fungi | jeblair: right at 16:00 utc | 18:54 |
| jeblair | yep | 18:54 |
| fungi | so maybe we creamed it | 18:54 |
| dtroyer_zz | sdague: trying, gerrit not appy as already noted | 18:54 |
| dtroyer_zz | why I wanted to mention it here ;) | 18:55 |
| stevemar | sdague: go ahead and revert it, dtroyer_zz and i both know about it | 18:56 |
| jeblair | dtroyer_zz, sdague, stevemar: if we want to get creative, we could consider a new pipeline+gerrit category | 18:56 |
| stevemar | you're clear in our books | 18:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Bump ubuntu-(trusty|xenial) to 20 min-ready https://review.openstack.org/360734 | 18:56 |
| *** ilyashakhat has quit IRC | 18:57 | |
| pabelanger | AJaeger: anteaya: If you have a moment to review:^ | 18:57 |
| stevemar | jeblair: i don't have the know how to come up with a more clever solution, i thought my osc-check-job was already pretty clever! :P | 18:57 |
| fungi | clarkb: jeblair: so should i hand-patch httpd.maxThreads=100 into the gerrit.config immediately before restarting? | 18:57 |
| dtroyer_zz | jeblair: I would actually prefer a way for plugins to do this outside the gate so if anything does happen again inside it is just a backstop | 18:57 |
| fungi | clarkb: jeblair: and then write up the puppet change for that? | 18:57 |
| dtroyer_zz | this isn't the sort of thing you should be finding out that late in the development | 18:57 |
| jeblair | dtroyer_zz, stevemar: yeah, i think the problem is that this *is* the correct solution | 18:57 |
| anteaya | pabelanger: I'm on board, waiting for gerrit to let me say so | 18:58 |
| jeblair | (lacking a registry like sdague was suggesting) | 18:58 |
| *** sdague has quit IRC | 18:58 | |
| dtroyer_zz | I think the registry is where we are headed. we already track plugins existance and the list of resources, that's just another thing ans what will drive us to formalize it rather than just keeping a doc | 18:59 |
| jeblair | anyway, i guess just pay attention to it in check for now. :) | 18:59 |
| AJaeger | pabelanger: +2A | 18:59 |
| fungi | javamelody has total thread counts but doesn't seem to break out httpd and sshd threads into their own counts | 19:00 |
| anteaya | fungi: I'm fine with the hand patch and restart followed by the puppet change, I have no comment on the question about whether that is the correct value though | 19:00 |
| jeblair | ssh review gerrit show-caches -w 120 --show-threads | 19:00 |
| jeblair | fungi: that may help | 19:00 |
| fungi | jeblair: aha, i can grep that, yes | 19:00 |
| jeblair | fungi: well you may want to skip the "-w 120" :) | 19:00 |
| dtroyer_zz | boarding time! | 19:00 |
| fungi | jeblair: yep, getting a count now | 19:01 |
| * dtroyer_zz puts mic gentrly back into the mic stand | 19:01 | |
| fungi | or at least i will when that command finishes | 19:01 |
| anteaya | dtroyer_zz: safe flight | 19:01 |
| *** Ahharu_ has joined #openstack-infra | 19:01 | |
| Ahharu_ | Hello, is the mitaka.puppet-openstack.fuel-library.pkgs.ubuntu.neutron_vlan_ha gate working properly? | 19:01 |
| fungi | gerrit: show-threads: not found | 19:01 |
| * fungi reads documentation | 19:01 | |
| jeblair | fungi, clarkb: can we move this to incident? | 19:02 |
| fungi | jeblair: yep | 19:02 |
| *** bhunter71 has quit IRC | 19:02 | |
| Ahharu_ | It is failing all the time I recheck even on a cherry-picked commit | 19:02 |
| Ahharu_ | https://review.openstack.org/#/c/360584/ | 19:02 |
| *** ilyashakhat has joined #openstack-infra | 19:02 | |
| AJaeger | Ahharu_: that's a job executed by the Fuel CI - a third party CI. | 19:03 |
| pabelanger | AJaeger: anteaya: great, thanks | 19:03 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Increase rh1 max-servers to 75" https://review.openstack.org/360673 | 19:03 |
| AJaeger | Ahharu_: nothing we can help you with - best talk with the fuel folks on their channel | 19:03 |
| Ahharu_ | ok, will do :) | 19:06 |
| *** sarob has joined #openstack-infra | 19:08 | |
| *** david-lyle has quit IRC | 19:10 | |
| *** zul has joined #openstack-infra | 19:11 | |
| *** sarob has quit IRC | 19:12 | |
| *** guilherme has joined #openstack-infra | 19:12 | |
| pabelanger | clarkb: okay, added an auto-hold for gate-cross-cinder-python27-db-ubuntu-xenial. So we can check cgroups | 19:15 |
| pabelanger | also going to clean up our held nodes for ipv6 failures | 19:15 |
| clarkb | pabelanger: ok ty for doing that. Was on my list to do today just haven't gotten to it | 19:16 |
| clarkb | I have rechecked 358105 since our images are mostly up to date in all cloud regions now | 19:16 |
| pabelanger | k | 19:16 |
| *** raunak has quit IRC | 19:18 | |
| *** markusry has quit IRC | 19:19 | |
| *** YqwHkAMaYX has quit IRC | 19:19 | |
| *** markusry has joined #openstack-infra | 19:19 | |
| *** Apoorva has quit IRC | 19:20 | |
| clarkb | if ntp looks happy on those I am going to self approve (it already ahs the +2s) this should make our d-g jobs run a little quicker | 19:20 |
| *** e0ne has joined #openstack-infra | 19:21 | |
| *** JasonF has joined #openstack-infra | 19:21 | |
| *** salv-orlando has quit IRC | 19:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Bump ubuntu-(trusty|xenial) to 20 min-ready https://review.openstack.org/360734 | 19:21 |
| *** aeng has joined #openstack-infra | 19:23 | |
| *** e0ne has quit IRC | 19:23 | |
| *** sarob has joined #openstack-infra | 19:24 | |
| *** rhallisey has quit IRC | 19:25 | |
| *** rhallisey has joined #openstack-infra | 19:26 | |
| *** sdague has joined #openstack-infra | 19:26 | |
| clarkb | one thing I have noticed is that we are not uploading images to ovh bhs1 reliably and an image listing there shows we have leaked a bunch of images | 19:27 |
| clarkb | I am going to clear the leaked images out so we can see if that is related to image upload reliability | 19:27 |
| clarkb | mordred: ^ that may be something shade/glance need to sort out though | 19:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Increase httpd threads for review.o.o https://review.openstack.org/360744 | 19:28 |
| *** tphummel has quit IRC | 19:33 | |
| *** Na3iL has quit IRC | 19:34 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Clean up vendor support list https://review.openstack.org/360747 | 19:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add timestamp.txt for rsync mirrors https://review.openstack.org/360748 | 19:34 |
| *** edtubill has quit IRC | 19:35 | |
| pabelanger | cloudnull: odyssey4me: adds timestamp.txt to rsync repos first ^ | 19:35 |
| mordred | clarkb: oh goodie! | 19:35 |
| pabelanger | clarkb: maybe issues with qcow2? | 19:35 |
| clarkb | pabelanger: maybe? like if they have to do a conversion on the backend that could be less reliable | 19:36 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Increase httpd threads for review.o.o https://review.openstack.org/360744 | 19:36 |
| clarkb | also its really hard to filter this list out and be careful not to delete any images we are using at the same time | 19:36 |
| pabelanger | clarkb: Right, my thinking too | 19:36 |
| * clarkb tries to be extra careful | 19:37 | |
| clarkb | (the "building images don't have uuids in nodepool db yet so trying to make sure I don't accidentally delete any of those | 19:38 |
| *** sdague has quit IRC | 19:39 | |
| *** sdague has joined #openstack-infra | 19:39 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add timestamp.txt for rsync mirrors https://review.openstack.org/360748 | 19:39 |
| sdague | clarkb: ok, given that you were the first to notice - https://review.openstack.org/#/c/360728/ - dtroyer_zz is +1 | 19:40 |
| *** zul has quit IRC | 19:41 | |
| *** cody-somerville has joined #openstack-infra | 19:41 | |
| *** cody-somerville has quit IRC | 19:41 | |
| *** cody-somerville has joined #openstack-infra | 19:41 | |
| clarkb | I estimate we have leaked 290 images in ovh-bhs1 | 19:41 |
| *** njohnston has joined #openstack-infra | 19:45 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add timestamp.txt to debian-openstack repo https://review.openstack.org/360750 | 19:45 |
| *** njohnston has left #openstack-infra | 19:46 | |
| pabelanger | clarkb: Wow | 19:46 |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is restarting to implement some performance tuning adjustments, and should return to working order momentarily. | 19:46 | |
| pabelanger | zigo: ^ packaging job updates to add timestamp.txt to the repo | 19:46 |
| pabelanger | zigo: some extra info to know when we last changed it | 19:46 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: don't start horizon in grenade https://review.openstack.org/360565 | 19:46 |
| zigo | Well, we last changed it *today*, and this will be turth every day! :) | 19:47 |
| clarkb | pabelanger: mordred I have started the deletes (doing them serially to avoid api limits so not very quick) | 19:47 |
| zigo | Looking at the CR. | 19:47 |
| pabelanger | zigo: right, mostly helpful if we have have a problem and it becomes stale for some reason | 19:47 |
| clarkb | we should probably do a check of the other clouds for similar situations | 19:47 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 19:47 |
| clarkb | (I can do that but likely not until much later as I have to run a bunch of errands in about an hour | 19:47 |
| zigo | pabelanger: What I also plan on implementing once everything is built, is a tempest validation job which we can use to validate the whole repo. | 19:48 |
| *** Ahharu_ has quit IRC | 19:48 | |
| cloudnull | pabelanger: awesome! | 19:48 |
| mordred | clarkb: I'm almost done with the FIP follow up patch | 19:48 |
| clarkb | mordred: cool | 19:48 |
| *** jheroux has quit IRC | 19:48 | |
| mordred | clarkb: I'll start looking at images after that | 19:48 |
| pabelanger | zigo: Ya, that would be good | 19:49 |
| zigo | pabelanger: BASE=/afs/.openstack.org/mirror/deb-openstack <--- Wasn't it debian-openstack ? | 19:49 |
| clarkb | today is remove cast from kids leg day so its a "fun" errand | 19:49 |
| zigo | pabelanger: FYI, I have that already on my Jenkins. | 19:49 |
| pabelanger | zigo: it is deb-openstack in AFS, we have char limts | 19:49 |
| *** tphummel has joined #openstack-infra | 19:50 | |
| zigo | ok | 19:50 |
| mordred | zigo: fwiw, if you "apt-get install openafs-client" and then "ls /afs/.openstack.org/mirror/deb-openstack" you can verify it yourself locally | 19:50 |
| fungi | zigo: also, the openafs-client package in debian works great. i use it to check stuff from home, no authentication needed for the read-only volumes http://docs.openstack.org/infra/system-config/afs.html#client-configuration | 19:51 |
| mordred | heh | 19:51 |
| fungi | hah | 19:51 |
| mordred | fungi: jinx | 19:51 |
| * fungi goes back to breaking things | 19:51 | |
| *** jheroux has joined #openstack-infra | 19:51 | |
| clarkb | mordred: http://paste.openstack.org/show/563453/ is a list of the image uuids I am deleting | 19:51 |
| pabelanger | zigo: so, wouldn't it be neat is in the check / gate we somehow merged the package being build with the packages already built, then ran tempest. | 19:51 |
| clarkb | if you need them to compare to logs | 19:52 |
| pabelanger | zigo: so we validate everything before pushing in the post pipeline | 19:52 |
| zigo | pabelanger: Unfortunately, the full tempest test takes currently more than 1 hour, and we can't afford to wait that long. | 19:53 |
| clarkb | sdague: stevemar https://review.openstack.org/#/c/360728/1/zuul/layout.yaml will run that job non voting against mitaka. The old code was excluding mitaka. Do we need to edit the revert to continue excluding mitaka? | 19:53 |
| clarkb | sdague: stevemar in general though I Think we should do this | 19:53 |
| zigo | pabelanger: I'd have to think about ways to make it faster. | 19:53 |
| pabelanger | clarkb: do we have the ability to resize puppetmaster.o.o? I haven't check in rackspace | 19:54 |
| *** raunak has joined #openstack-infra | 19:54 | |
| clarkb | pabelanger: I don't think so | 19:54 |
| *** zul has joined #openstack-infra | 19:54 | |
| clarkb | pabelanger: not on the performance flavors iirc | 19:54 |
| pabelanger | zigo: Ya, some sort of validation before doing vos release would be neat. | 19:54 |
| sdague | clarkb: I suspect excluding mitaka was because they couldn't vote with it | 19:55 |
| clarkb | sdague: gotcha | 19:55 |
| pabelanger | zigo: I'm sure we'll talk more about it in Barcelona | 19:55 |
| sdague | we can always clean up that separately | 19:55 |
| zigo | pabelanger: I already plan on adding piuparts tests. | 19:55 |
| clarkb | and since this is non voting its fine | 19:55 |
| zigo | Yup. | 19:55 |
| clarkb | will approve | 19:55 |
| sdague | clarkb: coolio | 19:55 |
| *** sarob has quit IRC | 19:55 | |
| *** Apoorva has joined #openstack-infra | 19:56 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Update undercloud flavor stats https://review.openstack.org/360756 | 19:57 |
| clarkb | mordred: in the past I thought that leakage was happening beacuse the glance image upload is a two step process of make image, then upload and if the upload fails we aren't deleting the image. But I thought shade addressed that as I recall reviewing changes to properly delete the newly created image if an upload fails | 19:58 |
| mordred | yah. it's supposed to do that for sure | 19:58 |
| mordred | however ... | 19:58 |
| mordred | I betcha if a proxy throws a 500 timeout like HP used to ... and the image is still uploading | 19:59 |
| apuimedo | anybody with gerrit adming rights? | 19:59 |
| mordred | and/or converting | 19:59 |
| *** jheroux has quit IRC | 19:59 | |
| apuimedo | s/adming/admin/ | 19:59 |
| mordred | and then you try to delete and the cloud says "nope, image isn't active" | 19:59 |
| mordred | I mean ... it's a pattern we've seen before | 19:59 |
| apuimedo | we need to populate kuryr-libnetwork-core and kuryr-kubernetes-core | 20:00 |
| *** JayF has quit IRC | 20:00 | |
| mordred | clarkb: we should start naming these patterns | 20:00 |
| *** jpich has quit IRC | 20:00 | |
| *** JasonF is now known as JayF | 20:00 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: remove old tests https://review.openstack.org/360761 | 20:00 |
| clarkb | mordred: we can call that one "cloud is too busy for you user no deletions" | 20:01 |
| *** sambetts|afk has quit IRC | 20:02 | |
| mordred | clarkb: yah. I feel like I should get that tattooed | 20:02 |
| *** tonytan_brb is now known as tonytan4ever | 20:02 | |
| *** csomerville has joined #openstack-infra | 20:02 | |
| pleia2 | heh | 20:02 |
| *** sdake has quit IRC | 20:03 | |
| openstackgerrit | Alex Kavanagh proposed openstack-infra/project-config: Add charm-manila https://review.openstack.org/360762 | 20:03 |
| *** jheroux has joined #openstack-infra | 20:03 | |
| *** sdake has joined #openstack-infra | 20:04 | |
| marcusvrn_ | Hi folks, I'm configuring a nodepool CI and I got stuck in the following step: I added a keypair into openstack, configured common.yaml, adding "keypair" label, same name as openstack keypair....but noodepool service fails when tries to create a server (http://paste.openstack.org/show/563454/). Any ideas? | 20:04 |
| clarkb | apuimedo: have a link to the change that created those groups? | 20:05 |
| clarkb | apuimedo: I can go ahead and get things sorted out but it helps to have the change | 20:05 |
| *** sambetts has joined #openstack-infra | 20:05 | |
| apuimedo | sure! | 20:05 |
| marcusvrn_ | detail: Metadata Key name is NONE | 20:05 |
| apuimedo | getting it to you right now | 20:05 |
| apuimedo | https://review.openstack.org/357946 | 20:05 |
| clarkb | also looks like gal is ptl? we will likely add gal to the groups | 20:05 |
| apuimedo | https://review.openstack.org/357945 | 20:05 |
| *** ilyashakhat has quit IRC | 20:06 | |
| *** cody-somerville has quit IRC | 20:06 | |
| * clarkb double checks the governance stuff | 20:06 | |
| anteaya | marcusvrn_: so it created the server, and then it deleted the server | 20:06 |
| apuimedo | clarkb: well, it would be very helpful if you add gal and kuryr-core as a subgroup | 20:06 |
| apuimedo | to both of them | 20:06 |
| anteaya | marcusvrn_: do you have a configuration file for nodepool yet? | 20:06 |
| clarkb | apuimedo: the process we use is the add the ptl to the groups then the ptl can elect to manage it as they see fit | 20:07 |
| *** dprince has quit IRC | 20:07 | |
| anteaya | marcusvrn_: if you have a maximum server quota set to 0 that might be what is telling nodepool to delete the server | 20:07 |
| *** sarob has joined #openstack-infra | 20:07 | |
| *** markusry has quit IRC | 20:07 | |
| apuimedo | clarkb: the problem with that is that Friday is non-working in israel, so tomorrow the other cores won't be able to +2 anything | 20:07 |
| *** markusry has joined #openstack-infra | 20:07 | |
| *** markusry has quit IRC | 20:08 | |
| anteaya | marcusvrn_: timestamps are 19 seconds apart so the server wouldn't even have been available yet | 20:08 |
| clarkb | apuimedo: I am not sure I feel comfortable going behind a PTL's back to delegate for them | 20:08 |
| clarkb | apuimedo: this system is set up this way so that infra does not have to be involved beyond handing over control to the ptl | 20:08 |
| marcusvrn_ | anteaya: nodepool.yaml -> http://paste.openstack.org/show/563458/ | 20:08 |
| marcusvrn_ | it's set to 2 | 20:09 |
| clarkb | and both those projects live under kuryr according to the governance repo | 20:09 |
| apuimedo | clarkb: I understand. I'll try to nag him on his weekend | 20:09 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "set check-osc-job to voting" https://review.openstack.org/360728 | 20:09 |
| *** rhallisey has quit IRC | 20:09 | |
| apuimedo | clarkb: right, that's why kuryr-core should be a sub group to both :P | 20:09 |
| pabelanger | clarkb: so we don't setup cgroups for single_use_slaves | 20:09 |
| clarkb | apuimedo: that is for the ptl to decide | 20:09 |
| clarkb | apuimedo: not for infra to decide | 20:09 |
| clarkb | pabelanger: aha! ok we probably want to add it there then | 20:09 |
| apuimedo | clarkb: as I said, I understand | 20:09 |
| anteaya | marcusvrn_: yes, good, is that the first server your nodepool is trying to create? | 20:09 |
| apuimedo | did you add him already? | 20:10 |
| marcusvrn_ | anteaya: hmmm...but the keypair need to be added to the server, right? the metadata "key name" should be "jenkins2" | 20:10 |
| marcusvrn_ | yes | 20:10 |
| clarkb | apuimedo: yes its done | 20:10 |
| apuimedo | thank you clarkb! Have a good rest of day | 20:10 |
| anteaya | in 19 seconds I doubt nodepool got as far as worrying about keys at all | 20:10 |
| anteaya | I'm not sure what happens in the first 19 seconds of server creation that would tell nodepool to delete that server | 20:11 |
| anteaya | but I don't think that is time for the server to go live at all | 20:11 |
| *** tqtran has quit IRC | 20:12 | |
| anteaya | and yes, keys should be added to the server, but I dont' think keys came into play here | 20:12 |
| *** sarob has quit IRC | 20:12 | |
| clarkb | marcusvrn_: nodepool doesn't add keys to insatnces like that. It expects them to exist already | 20:12 |
| clarkb | marcusvrn_: eg your image build process should take care of that not nova | 20:12 |
| anteaya | ah thank you, clarkb | 20:12 |
| openstackgerrit | Alex Kavanagh proposed openstack-infra/project-config: Add charm-manila https://review.openstack.org/360762 | 20:13 |
| marcusvrn_ | clarkb: ah, ok. I'll try to generate again the image, maybe keys are not matching | 20:14 |
| clarkb | marcusvrn_: you can always boot one manually and fiddle with it to see | 20:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add jenkins::cgroups to single_use_slaves https://review.openstack.org/360770 | 20:14 |
| pabelanger | clarkb: ^ | 20:14 |
| clarkb | marcusvrn_: running a nova show on the instances then booting one of my own is my typical process to debug | 20:14 |
| clarkb | pabelanger: I am good with that but with the release coming up maybe we should WIP that until after feature freeze? Slight worry that it will start causing jobs to fail that were right up against the memory limit | 20:15 |
| rajinir | Restarted zuul with a new job in jenkins. I see an error job is not registered with Gearman. Have restarted zuul-merger and nodepool as well. Any solution? | 20:15 |
| clarkb | pabelanger: since I know some tests for projects like magnum run deep into swap | 20:15 |
| pabelanger | clarkb: sure, I am fine with that | 20:15 |
| *** kzaitsev_mb has quit IRC | 20:15 | |
| clarkb | rajinir: the jobs are registered by the gearman works you need to make sure whatever is acting as your gearman worker is conencted to gearman and registering jobs | 20:15 |
| clarkb | rajinir: if using jenkins you can double check it is connected and if so make sure it has instances capable of running that job | 20:16 |
| rajinir | clarkb: its running the old jobs, only the new jobs have this problem. Will restarting jenkins help? | 20:17 |
| marcusvrn_ | clarkb: nice, I'll do that | 20:17 |
| *** dtardivel has quit IRC | 20:17 | |
| marcusvrn_ | clarkb: anteaya thanks! | 20:17 |
| clarkb | rajinir: no don't restart jenkins... just make sure jenkins is connected to gearman and thati t has slaves capable of running the new job. Maybe your new job isn't in jenkins yet? or none of your slave labels match? | 20:17 |
| rajinir | clarkb: good point, will check | 20:17 |
| rajinir | thanks | 20:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks https://review.openstack.org/357517 | 20:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Detect the need for FIPs better in auto_ip https://review.openstack.org/360771 | 20:18 |
| mordred | clarkb: ok ^^ that's the gist of it - I need one more patch to add unittests for the first patch | 20:19 |
| *** roxanaghe has joined #openstack-infra | 20:19 | |
| mordred | clarkb: but the second patch changes the way we get networks, so I thought it would be harder to read combined | 20:19 |
| clarkb | mordred: ok I am going to disappear in a little bit so don't wait around for me unless you think you need to | 20:19 |
| mordred | clarkb: I won't - I wanna see those run in the gate and I want a patch with unittests | 20:19 |
| clarkb | kk | 20:19 |
| * clarkb hopes that they don't have to do xrays again | 20:20 | |
| clarkb | convincing a 14 month old to sit still for xrays is unfun | 20:20 |
| mordred | clarkb: I currently have in my house a 2 month old and a 4 year old (and their parents) | 20:21 |
| mordred | clarkb: if the 4 year old is any indication ... you have some fun awaiting you in 2 years and 10 months | 20:21 |
| clarkb | mordred: yes... I had to figure out health benefits again last week and it basically came down to lets just pay for the better one because they are constantly hurting themselves | 20:21 |
| *** nmagnezi has quit IRC | 20:22 | |
| apuimedo | mine already got stitches when he was two | 20:22 |
| *** kgiusti has left #openstack-infra | 20:22 | |
| apuimedo | (3.5yo now) | 20:22 |
| *** roxanaghe has quit IRC | 20:24 | |
| clarkb | apuimedo: my 14 month old has been in a cast for the last 4 weeks after breaking her leg falling off a chair | 20:24 |
| apuimedo | clarkb: I'm a very lucky dad | 20:24 |
| apuimedo | my little kamikaze even rolled down the tiled stairs (16 of them) | 20:24 |
| apuimedo | backwards | 20:24 |
| apuimedo | and not even a scratch | 20:24 |
| apuimedo | I was astounded | 20:24 |
| openstackgerrit | Ken Giusti proposed openstack-infra/project-config: Run the AMQP 1.0 oslo.messaging functional tests on Python 3.4 https://review.openstack.org/358745 | 20:25 |
| fungi | clarkb: falling off chairs already, and not even old enough to drink? that's not setting a good precedent | 20:26 |
| apuimedo | more of less like https://www.youtube.com/watch?v=wi-UOV0HKbY | 20:26 |
| apuimedo | but backwards | 20:26 |
| *** kzaitsev_mb has joined #openstack-infra | 20:27 | |
| *** tqtran has joined #openstack-infra | 20:28 | |
| *** rhallisey has joined #openstack-infra | 20:28 | |
| *** tphummel has quit IRC | 20:28 | |
| mordred | fungi: I did not hear anything about not being old enough to drink | 20:28 |
| anteaya | mordred: does the 4 year old know how to swim? | 20:28 |
| mordred | fungi: my assumption is that all liquid at clarkb's house is beer | 20:28 |
| mordred | anteaya: very well | 20:28 |
| anteaya | oh good | 20:29 |
| bkero | mordred: you're forgetting about the gin | 20:29 |
| mordred | bkero: that's an excellent point you make | 20:29 |
| anteaya | I assume the 2 month old can't make it to the pool yet | 20:29 |
| *** sdake has quit IRC | 20:29 | |
| mordred | anteaya: well, the 2 month old can make it near the pool if a larger human carries it there | 20:30 |
| anteaya | right | 20:31 |
| clarkb | apuimedo: ya normally they bounce no idea why this one time it was a problem | 20:31 |
| anteaya | but hopefully the larger human sticks around | 20:31 |
| clarkb | bkero: the botanist is pretty good | 20:31 |
| apuimedo | clarkb: yes. It's usually the smallest hits that break | 20:31 |
| apuimedo | anteaya: the trick is to have a very very cold swimming pool. They usually don't get in it | 20:32 |
| *** asettle has quit IRC | 20:32 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: update release dashboard output for newton https://review.openstack.org/360776 | 20:33 |
| anteaya | apuimedo: good call | 20:34 |
| anteaya | apuimedo: you also must have been a life guard | 20:34 |
| *** sdake has joined #openstack-infra | 20:35 | |
| apuimedo | anteaya: the best way to bay watch is handing a couple of euros to little kids and have them shout "jellyfish!" every half an hour | 20:35 |
| apuimedo | nobody ever gets deep into the water like that | 20:36 |
| *** asettle has joined #openstack-infra | 20:36 | |
| *** kzaitsev_mb has quit IRC | 20:36 | |
| anteaya | I can't even share the stories of what they used to do at the pool to accomplish the same goal | 20:37 |
| *** zul has quit IRC | 20:38 | |
| *** tphummel has joined #openstack-infra | 20:39 | |
| *** sarob has joined #openstack-infra | 20:39 | |
| *** csomerville has quit IRC | 20:39 | |
| fungi | i am totally unable to figure out why the trusty apply job failed on https://review.openstack.org/360744 | 20:39 |
| fungi | every test seems to have exited 0 | 20:40 |
| pabelanger | fungi: timeout? took 33mins to run | 20:40 |
| fungi | hrm... maybe | 20:40 |
| fungi | pabelanger: oh, yep! | 20:40 |
| fungi | there's a 25 minute lag before the job complete line at the end of the log | 20:41 |
| * fungi is blind | 20:41 | |
| fungi | thanks! | 20:41 |
| pabelanger | Ya, we need to restart zuul-launcher still, to pick up the timeout fix in ansible | 20:41 |
| *** ansmith has quit IRC | 20:42 | |
| *** gyee has joined #openstack-infra | 20:42 | |
| *** jkilpatr has quit IRC | 20:42 | |
| *** sarob has quit IRC | 20:44 | |
| fungi | okay, i'm going to disappear to grab some dinner, then i'll come back and continue trying to make sure gerrit still looks sane | 20:45 |
| *** asettle has quit IRC | 20:45 | |
| anteaya | enjoy dinner | 20:45 |
| fungi | thanks! | 20:46 |
| *** salv-orlando has joined #openstack-infra | 20:46 | |
| *** sarob has joined #openstack-infra | 20:47 | |
| *** senk has joined #openstack-infra | 20:47 | |
| *** eharney has quit IRC | 20:49 | |
| asselin__ | marcusvrn_, hi | 20:50 |
| oomichi | hi, are there any relationship between nodes with/without some suffix like "-181073"? | 20:52 |
| oomichi | eg: ubuntu-xenial-2-node-rax-ord-3874987 | 20:52 |
| oomichi | and ubuntu-xenial-2-node-rax-ord-3874987-181073 | 20:53 |
| *** tqtran has quit IRC | 20:53 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/irc-meetings: add link to the requirements team agenda https://review.openstack.org/360781 | 20:54 |
| oomichi | I saw a little bit for live-migration test failure, and the host was different from expected host. | 20:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Clean up tripleo-test-cloud-rh1 os-client-config https://review.openstack.org/360782 | 20:55 |
| *** sarob has quit IRC | 20:56 | |
| *** tqtran has joined #openstack-infra | 20:57 | |
| *** eharney has joined #openstack-infra | 20:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Handle a project-template with no jobs https://review.openstack.org/360783 | 20:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add an OpenStack test https://review.openstack.org/360784 | 20:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Pass assigned nodes to the launcher https://review.openstack.org/360785 | 20:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add a python27 job variant to the OpenStack test https://review.openstack.org/360786 | 20:57 |
| *** tphummel has quit IRC | 20:57 | |
| ianw | fungi: yes, working on the review-dev one (from 4 hours ago :) | 20:59 |
| *** senk has quit IRC | 21:00 | |
| *** csomerville has joined #openstack-infra | 21:01 | |
| *** rfolco_ has quit IRC | 21:02 | |
| *** rfolco_ has joined #openstack-infra | 21:03 | |
| *** rfolco_ has quit IRC | 21:03 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use delorean's exit code to decide on retry https://review.openstack.org/348790 | 21:04 |
| *** sdake_ has joined #openstack-infra | 21:05 | |
| *** sarob has joined #openstack-infra | 21:07 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: update release dashboard output for newton https://review.openstack.org/360776 | 21:07 |
| *** sdake has quit IRC | 21:08 | |
| jroll | clarkb | convincing a 14 month old to sit still for xrays is unfun <- fiancee is a rad tech, she is thankful for people like you that actually help with that instead of just watching her struggle | 21:09 |
| *** thorst has quit IRC | 21:09 | |
| *** tphummel has joined #openstack-infra | 21:10 | |
| asselin__ | dtroyer_zz, any idea when new osc-lib will get released after https://review.openstack.org/360600 merges? | 21:10 |
| jeblair | jroll: you're assuming clarkb was relating his own experience and not saying "wow it looks like the rad tech didn't have fun doing that!" :) | 21:10 |
| jroll | heh | 21:11 |
| *** thorst has joined #openstack-infra | 21:12 | |
| *** sarob has quit IRC | 21:12 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** thorst has quit IRC | 21:16 | |
| *** salv-orl_ has joined #openstack-infra | 21:17 | |
| *** sdague has quit IRC | 21:18 | |
| *** thiagop has quit IRC | 21:18 | |
| *** hongbin_ has joined #openstack-infra | 21:18 | |
| *** salv-orlando has quit IRC | 21:19 | |
| *** sdake_ has quit IRC | 21:20 | |
| clarkb | oh no I was on the tabke in a lead syit trying ti hold her down | 21:20 |
| *** hongbin has quit IRC | 21:21 | |
| pabelanger | clarkb: ianw: any chance we can land https://review.openstack.org/#/c/360622/ today? Adding mtl01 region for internap on puppetmaster.o.o | 21:22 |
| *** sdake has joined #openstack-infra | 21:22 | |
| *** guilherme has quit IRC | 21:22 | |
| *** hongbin_ has quit IRC | 21:26 | |
| anteaya | it is moments like this I think of https://www.youtube.com/watch?v=RJf41HZ76So#t=6m | 21:26 |
| *** david-lyle has joined #openstack-infra | 21:26 | |
| *** hongbin has joined #openstack-infra | 21:26 | |
| ianw | when i thought my son at a similar age had swallowed loose change (he hadn't) ISTR they stuck him in a jig. nobody enjoyed that experience. The dr's helpful advice was "don't let children play with small coins" ... thanks for that | 21:27 |
| anteaya | the dr obviously has no children | 21:27 |
| *** _ari_ has quit IRC | 21:27 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:29 | |
| *** matt-borland has quit IRC | 21:30 | |
| pleia2 | my little sister stuck a penny in an outlet once | 21:31 |
| pleia2 | zap, emergency room | 21:31 |
| pleia2 | kids :) | 21:31 |
| *** david-lyle has quit IRC | 21:31 | |
| *** eharney has quit IRC | 21:32 | |
| irtermite | nephew, fingers, lamp socket... ever overcooked a hot dog on the grill and had it split open? | 21:32 |
| anteaya | oh | 21:32 |
| anteaya | can't do this | 21:32 |
| pleia2 | x_x | 21:32 |
| jeblair | yeah, i think irtermite "wins" and the game is over now :) | 21:33 |
| irtermite | hahaha | 21:34 |
| irtermite | we were playing a game? what did I win? | 21:35 |
| anteaya | the opportunity to select the next topic of conversation | 21:37 |
| *** jamesdenton has quit IRC | 21:40 | |
| *** dizquierdo has joined #openstack-infra | 21:40 | |
| *** kzaitsev_mb has quit IRC | 21:41 | |
| *** pradk has quit IRC | 21:41 | |
| openstackgerrit | Merged openstack-infra/system-config: Add mtl01 region to internap on puppetmaster https://review.openstack.org/360622 | 21:41 |
| irtermite | anteaya: ummm puppy dogs and unicorns? | 21:42 |
| *** rlandy is now known as rlandy|bbl | 21:43 | |
| marcusvrn_ | asselin__: hi I was talking to clarkb and anteaya about nodepool CI. I created again the image used by nodepool, uploaded it to openstack and when nodepool try to setup a server, it fails... here is the console log of the VM that nodepool tried to setup: http://paste.openstack.org/show/563686/ | 21:44 |
| *** dimtruck is now known as zz_dimtruck | 21:44 | |
| *** Hal1 has joined #openstack-infra | 21:44 | |
| asselin__ | marcusvrn_, I have this documented somewhere.... | 21:44 |
| asselin__ | marcusvrn_, did you follow these instructions? http://docs.openstack.org/infra/openstackci/third_party_ci.html#start-nodepool | 21:46 |
| asselin__ | marcusvrn_, this part in particular: Ensure the NODEPOOL_SSH_KEY variable is in the environment | 21:46 |
| marcusvrn_ | asselin__: yes, that's what I did | 21:48 |
| asselin__ | marcusvrn_, I don't think you'll find anything in the VM console log. Instead, if you did this correctly and need to debug, you need to mount your DIB created image and inspect the authorized keys | 21:48 |
| marcusvrn_ | yes, I just create an instance with a keypair generated by me to go into the VM and check the authorized keys | 21:49 |
| asselin__ | marcusvrn_, you can mount the image has the correct keys by mounting it. Instructions are here: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/README.rst | 21:49 |
| *** jerryz has quit IRC | 21:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove Fedora 24 work-around kernel https://review.openstack.org/353093 | 21:53 |
| *** kushal has quit IRC | 21:54 | |
| *** _nadya_ has quit IRC | 21:54 | |
| marcusvrn_ | asselin__: hmm it shows a different key into authorized_keys... | 21:55 |
| *** kushal has joined #openstack-infra | 21:56 | |
| asselin__ | marcusvrn_, I bet it's -infra's key....you might find it here: http://codesearch.openstack.org/ | 21:57 |
| *** shardy has quit IRC | 21:57 | |
| *** gouthamr has quit IRC | 22:00 | |
| marcusvrn_ | asselin__: no, it was my old key...but I grep the whole system to check if I forgot some place to change, before generate the new image...and that key does not exist anymore | 22:00 |
| *** gouthamr has joined #openstack-infra | 22:00 | |
| *** gouthamr has quit IRC | 22:00 | |
| asselin__ | marcusvrn_, perhaps saved in an env variable in some shell? try stopping and restarting services | 22:01 |
| marcusvrn_ | ok, restarting all services | 22:02 |
| marcusvrn_ | asselin__: just a question, is it normal to take an hour to generate a image? | 22:02 |
| asselin__ | marcusvrn_, first time yes, subsequent times should be much faster since there's a lot cached | 22:02 |
| asselin__ | marcusvrn_, http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/README.rst#n41 you can make it faster for debug purposes | 22:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Detect the need for FIPs better in auto_ip https://review.openstack.org/360771 | 22:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks https://review.openstack.org/357517 | 22:04 |
| *** esberglu has quit IRC | 22:04 | |
| *** gordc has quit IRC | 22:05 | |
| *** ddieterly is now known as ddieterly[away] | 22:05 | |
| *** roxanaghe has joined #openstack-infra | 22:07 | |
| marcusvrn_ | asselin__: hmm nice! | 22:08 |
| *** dizquierdo has quit IRC | 22:08 | |
| *** fguillot has quit IRC | 22:10 | |
| *** ddieterly[away] is now known as ddieterly | 22:10 | |
| *** csomerville has quit IRC | 22:10 | |
| openstackgerrit | Merged openstack-infra/puppet-gerrit: Create index on fresh install https://review.openstack.org/359683 | 22:11 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement scenario001 CI job https://review.openstack.org/360039 | 22:11 |
| *** roxanagh_ has joined #openstack-infra | 22:12 | |
| *** roxanaghe has quit IRC | 22:12 | |
| *** hashar has quit IRC | 22:13 | |
| *** dizquierdo has joined #openstack-infra | 22:14 | |
| *** kaisers_ has quit IRC | 22:15 | |
| *** chem|off has quit IRC | 22:16 | |
| *** csomerville has joined #openstack-infra | 22:16 | |
| *** roxanagh_ has quit IRC | 22:16 | |
| *** chem|off has joined #openstack-infra | 22:18 | |
| *** dizquierdo has quit IRC | 22:18 | |
| *** tpsilva has quit IRC | 22:18 | |
| *** Goneri has joined #openstack-infra | 22:22 | |
| *** vhosakot has quit IRC | 22:23 | |
| zigo | How much RAM does the jessie flavor have? | 22:24 |
| *** sdake has quit IRC | 22:24 | |
| *** dizquierdo has joined #openstack-infra | 22:24 | |
| *** gouthamr has joined #openstack-infra | 22:25 | |
| openstackgerrit | Merged openstack/os-client-config: Clean up vendor support list https://review.openstack.org/360747 | 22:25 |
| *** jheroux has quit IRC | 22:26 | |
| cloudnull | zigo: i dont work on infra specifically, but last i recall all of the flavors are 8GB ram, 4cpus, ~80gb disk | 22:28 |
| cloudnull | zigo: this is the flavor used within the osic http://cdn.pasteraw.com/gx3kl22a2okwqx9lj7fm70qwk5udcxe | 22:30 |
| zigo | cloudnull: Ok, then now I understand why one of my packages failed. | 22:30 |
| zigo | I mean, failed to build. | 22:30 |
| zigo | 8 GB is not enough RAM to build webkit2gtk ... :( | 22:31 |
| zigo | At least, not enough if no swap. | 22:31 |
| zigo | FYI, webkit2gtk is needed to build sphinx. | 22:31 |
| zigo | (yes, that's weird, but such are dependencies...) | 22:32 |
| *** dizquierdo has quit IRC | 22:32 | |
| cloudnull | our flavor is 2GB of swap, however other providers may not have that set. | 22:32 |
| zigo | Ok. | 22:32 |
| zigo | The package maintainer told me that webkit2gtk needs at least 4500 MB of RAM + same of swap. | 22:32 |
| cloudnull | make a swap file? | 22:34 |
| *** salv-orl_ has quit IRC | 22:34 | |
| pabelanger | Ya, we can add that to the job | 22:35 |
| zigo | pabelanger: Then the package will use more time than the timeout, I guess. | 22:35 |
| pabelanger | zigo: have a log? | 22:36 |
| zigo | pabelanger: So, I believe just downloading the package from official jessie-backports was the right thing to do... | 22:36 |
| zigo | pabelanger: Not anymore, since I cheated and did a download from jessie-backports. | 22:36 |
| pabelanger | ok | 22:36 |
| zigo | It was the correct thing to do IMO. | 22:36 |
| zigo | No need to bother so much for just an indirect dependency for sphinx ... | 22:37 |
| *** sdake has joined #openstack-infra | 22:37 | |
| zigo | pabelanger: FYI, I also tried rebuilding on a machine with 4GB of swap, and it failed at the exact same point: when linking hundreds of objects together. | 22:37 |
| zigo | One day, hopefully, we'll have a Build-Memory field in debian/control ... | 22:38 |
| zigo | Another beast is pypy. | 22:38 |
| *** Goneri has quit IRC | 22:39 | |
| zigo | It wouldn't even START building if you have less than 4GB of RAM. | 22:39 |
| zigo | Anyway, moving on ... ;P | 22:39 |
| *** thorst has joined #openstack-infra | 22:42 | |
| *** Guest1506 is now known as jamesdenton | 22:45 | |
| *** chem|off` has joined #openstack-infra | 22:46 | |
| *** thorst has quit IRC | 22:47 | |
| *** thorst has joined #openstack-infra | 22:47 | |
| *** chem|off has quit IRC | 22:48 | |
| pabelanger | clarkb: jeblair: So, I've been going back and forth in my mind about using nodepool.o.o to create images for our control plan servers. Since today we are just using stock ubuntu images from cloud-images.u.c, I'd love to switch them out to a basic ubuntu-minimal image | 22:48 |
| *** yamamoto has joined #openstack-infra | 22:49 | |
| pabelanger | another option is creating a periodic job to do it, but we still have to upload them | 22:49 |
| *** Apoorva has quit IRC | 22:50 | |
| *** kzaitsev_mb has quit IRC | 22:54 | |
| persia | Are you sure you want -minimal? -standard is usually small enough for most things, and -minimal lacks many things that are usually taken for granted. | 22:54 |
| *** tqtran has quit IRC | 22:55 | |
| persia | For Xenial, http://people.canonical.com/~ubuntu-archive/seeds/platform.xenial/standard is standard, whereas minimal is only http://people.canonical.com/~ubuntu-archive/seeds/platform.xenial/minimal (and dependencies, in both cases) | 22:55 |
| *** tqtran has joined #openstack-infra | 22:55 | |
| *** adriant has joined #openstack-infra | 22:55 | |
| pabelanger | Ya, we use -minimal today for test jobs plus an infra specific element | 22:55 |
| *** ddieterly has quit IRC | 22:56 | |
| *** thorst has quit IRC | 22:56 | |
| pabelanger | I wonder if that minimal seed matches the DIB element -minimal | 22:56 |
| persia | Wow. I'm surprised (but pleased) that is useful. | 22:57 |
| cloudnull | pabelanger: what would it take to build lxc images using DIB? | 22:57 |
| *** eset has quit IRC | 22:57 | |
| cloudnull | i see there is a docker option | 22:57 |
| pabelanger | cloudnull: not sure, I haven't tried before | 22:57 |
| *** rhallisey has quit IRC | 22:57 | |
| cloudnull | i guess i could try and build off of that? | 22:57 |
| *** chlong has quit IRC | 22:58 | |
| pabelanger | cloudnull: greghaynes or ianw might be better to answer that | 22:58 |
| *** Apoorva has joined #openstack-infra | 22:59 | |
| *** hockeynut has joined #openstack-infra | 22:59 | |
| *** eset has joined #openstack-infra | 22:59 | |
| *** tqtran has quit IRC | 22:59 | |
| cloudnull | -^- ping greghaynes ianw | 22:59 |
| *** gouthamr_ has joined #openstack-infra | 22:59 | |
| cloudnull | i'm +/- curious . | 23:00 |
| *** tqtran has joined #openstack-infra | 23:00 | |
| jeblair | pabelanger: you may want to include mordred in that conversation as he has been inching us toward being ready for control-plane images for a while, and greghaynes because he's interested in nodepool as a sort of dib scheduler like you are suggesting. | 23:00 |
| *** hongbin has quit IRC | 23:01 | |
| jeblair | pabelanger: if you used nodepool.o.o, we'd have to stick an openstackci cred on a system that only knows openstackjenkins. i've always been worried about what happens when something there goes wrong (ie, automatically delete entire control plane). | 23:02 |
| jeblair | pabelanger: maybe worth considering running a copy of nodepool on another server just for this purpose. | 23:02 |
| *** gouthamr has quit IRC | 23:03 | |
| pabelanger | jeblair: Ya, I was thinking about that too. Adding a second nodepool server might be in the cards | 23:03 |
| pabelanger | Doesn't have to happen today, but I think we've nailed the DIB process down pretty good now. Would be nice to start expanding them | 23:04 |
| pabelanger | I'll add a topic for our next meeting | 23:04 |
| cloudnull | my interest in DIB is related to OpenStack-Ansible, we use LXC base images which we get from https://images.linuxcontainers.org/ and while they work great I'd love to be able to pull the image build pipeline into the our OpenStack build pipeline using DIB (if it at all possible) | 23:04 |
| *** tqtran has quit IRC | 23:04 | |
| cloudnull | from a million miles away, which is my perspective of DIB only having hacked on it once, it looks like DIB could/should be able to do the needful. | 23:06 |
| jeblair | cloudnull: one thing you might be able to do soonish is create a periodic job to use dib to build an image and stash it on tarballs.o.o, then fetch it from there in your jobs | 23:06 |
| persia | pabelanger: As far as I can tell with a quick browse of the code, DIB will use apt to install the ubuntu-minimal task, which is generated from the minimal seed, so yes. | 23:07 |
| cloudnull | jeblair: ++ i'll look into that . | 23:07 |
| pabelanger | cloudnull: Ya, I did that recently with python-tripleoclient. It is experimental right now, but will give you an idea of everything works together. | 23:08 |
| cloudnull | do you have a review handy i can go steal from ? | 23:08 |
| cloudnull | i can look it up. | 23:09 |
| cloudnull | too so no worries. | 23:09 |
| *** tqtran has joined #openstack-infra | 23:09 | |
| pabelanger | cloudnull: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/tripleo.yaml#n64 | 23:09 |
| cloudnull | tyvm | 23:09 |
| *** hockeynut has quit IRC | 23:10 | |
| pabelanger | cloudnull: so, it doesn't call disk-image-create directly, python-tripleclient does that, but it shows a good example of setting up zuul-cloner for external elements and preparing the virtualenv | 23:10 |
| pabelanger | cloudnull: there are a few other projects doing it today, grep buildimage in job names | 23:11 |
| cloudnull | ok. that's great. gives me a starting point . | 23:11 |
| cloudnull | much appreciated. | 23:11 |
| *** chlong has joined #openstack-infra | 23:12 | |
| *** tqtran_ has joined #openstack-infra | 23:13 | |
| *** tqtran has quit IRC | 23:13 | |
| *** cody-somerville has joined #openstack-infra | 23:14 | |
| *** csomerville has quit IRC | 23:17 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Zuul v3: update branches job keyword https://review.openstack.org/360817 | 23:19 |
| pabelanger | #status log mirror.mtl01.internap.openstack.org now online | 23:20 |
| openstackstatus | pabelanger: finished logging | 23:20 |
| jeblair | \o/ | 23:20 |
| pabelanger | time to update nodepool :) | 23:21 |
| ianw | zaro: relaunching .... | 23:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add internap-mtl01 to nodepool clouds.yaml https://review.openstack.org/360818 | 23:25 |
| *** yamahata has quit IRC | 23:25 | |
| *** chem|off` has quit IRC | 23:25 | |
| *** chem|off` has joined #openstack-infra | 23:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Zuul v3: don't implicitly define job variables https://review.openstack.org/360819 | 23:28 |
| *** xarses has quit IRC | 23:29 | |
| *** rbrndt has quit IRC | 23:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add internap-mtl01 to nodepool.o.o https://review.openstack.org/360820 | 23:31 |
| *** Hal1 has quit IRC | 23:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Update nodepool zookeeper spec with branch https://review.openstack.org/360821 | 23:31 |
| pabelanger | mgagne: ^ I believe you said there is 120 nodes in mtl01, right? | 23:31 |
| pabelanger | we'll need to restart nodepool-builder too | 23:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add internap-mtl01 to nodepool.o.o https://review.openstack.org/360820 | 23:32 |
| *** sputnik13 has quit IRC | 23:32 | |
| *** chlong has quit IRC | 23:35 | |
| jeblair | pabelanger: are we still running with 4 upload workers instead of 16? | 23:37 |
| *** yamahata has joined #openstack-infra | 23:37 | |
| pabelanger | jeblair: no, we changed to 16 a while back. | 23:38 |
| jeblair | pabelanger: oh ok | 23:39 |
| jeblair | sheesh it's slow | 23:39 |
| jeblair | pabelanger: we don't have any critical updates needed in today's image do we? maybe we should land the nodepool changes now and restart the builder? | 23:40 |
| *** jianghuaw-m has quit IRC | 23:41 | |
| *** AnarchyAo has quit IRC | 23:41 | |
| pabelanger | jeblair: ya, I think we can do the restart tonight. I only know of the ntp change clarkb did, and pretty sure we've updated everywhere now | 23:41 |
| jeblair | pabelanger: +2s on both from me | 23:42 |
| ianw | happy to babysit if it helps | 23:42 |
| jeblair | ianw: thanks, i'm very close to eod | 23:43 |
| pabelanger | ianw: sure, trying to launch internap-mtl01, have a few reviews up | 23:43 |
| jeblair | https://review.openstack.org/360820 and https://review.openstack.org/360818 | 23:44 |
| jeblair | any others? | 23:44 |
| pabelanger | that should be all for now | 23:44 |
| jeblair | oh, the less urgent https://review.openstack.org/360683 | 23:44 |
| pabelanger | Ya, a nice to have | 23:45 |
| ianw | pabelanger: what's up with http://logs.openstack.org/18/360818/1/check/gate-openstackci-beaker-centos-7/660e0bc/console.html ? is that a complete log? | 23:45 |
| pabelanger | could not find job | 23:45 |
| pabelanger | that's what ansible said | 23:45 |
| jeblair | i assumed it was a zuul-ansible snafu and rechecked | 23:45 |
| pabelanger | I don't think I've see that one before | 23:46 |
| jeblair | pabelanger: any idea what that means? | 23:46 |
| pabelanger | maybe async_runner on node died? | 23:46 |
| pabelanger | would need to dig more into it | 23:46 |
| jeblair | i added topic:internap to those 3 | 23:47 |
| pabelanger | https://github.com/ansible/ansible-modules-core/blob/devel/utilities/logic/async_status.py#L69 | 23:48 |
| pabelanger | looks like ~/.ansible_async was 404 | 23:48 |
| pabelanger | for the job id | 23:49 |
| pabelanger | will keep an eye on it | 23:49 |
| openstackgerrit | Merged openstack-infra/irc-meetings: add link to the requirements team agenda https://review.openstack.org/360781 | 23:49 |
| *** david-lyle has joined #openstack-infra | 23:52 | |
| *** xyang1 has quit IRC | 23:52 | |
| *** rcernin has quit IRC | 23:53 | |
| *** thorst has joined #openstack-infra | 23:54 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:54 | |
| *** zhurong has joined #openstack-infra | 23:54 | |
| *** xarses has joined #openstack-infra | 23:56 | |
| ianw | great ... now puppet just times out building review-dev. it's probably lucky we're not trying to rebuild review.openstack.org in a crisis situation .... | 23:57 |
| *** roxanaghe has joined #openstack-infra | 23:57 | |
| *** nwkarsten has quit IRC | 23:58 | |
| openstackgerrit | Merged openstack-infra/project-config: Add internap-mtl01 to grafana dashboard https://review.openstack.org/360683 | 23:58 |
| *** nwkarsten has joined #openstack-infra | 23:58 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/devstack-gate: Use local AFS mirror for network_sanity_check https://review.openstack.org/318714 | 23:58 |
| *** woodster_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
