| armax | clarkb: did you end up checking the size of the other projects logs? | 00:00 |
|---|---|---|
| clarkb | armax: yes I provided all that data in the paste I gave you today | 00:00 |
| armax | clarkb: um, I must have missed it | 00:00 |
| dstufft | it's amazing what a good CDN and aggressively killing things that don't force people onto that CDN can do :D | 00:00 |
| clarkb | armax: http://paste.openstack.org/show/567259/ | 00:00 |
| armax | let me scroll back | 00:01 |
| armax | ah, thanks | 00:01 |
| dstufft | also caching | 00:01 |
| dstufft | ALL THE CACHING | 00:01 |
| fungi | cache all the things, in multiple places | 00:01 |
| dstufft | cache your caches | 00:01 |
| *** Swami has quit IRC | 00:01 | |
| dstufft | if one cache is good, surely two is better | 00:01 |
| fungi | we're still dealing with fallout from using caches of things we didn't think about causing a lack of test coverage that we could build the things we've cached | 00:02 |
| armax | clarkb: cool, thanks | 00:02 |
| fungi | new cffi release today exposed lots of projects that don't declare they need libffi-dev system packages, because they normally just use our cached cffi wheels | 00:02 |
| fungi | because, you know, that latter scenario is way faster | 00:03 |
| dstufft | fungi: breakage that exposes flawed assumptions is the best breakage | 00:03 |
| fungi | i call it "testing" | 00:03 |
| *** AnarchyAo has quit IRC | 00:05 | |
| *** tqtran has quit IRC | 00:06 | |
| sdake | hey folks re bindep | 00:07 |
| *** adrian_otto has quit IRC | 00:07 | |
| sdake | i made an empty bindep file | 00:07 |
| sdake | and see following | 00:07 |
| sdake | http://logs.openstack.org/28/366328/1/check/gate-kolla-dsvm-deploy-centos-source-centos-7-nv/41a9197/console.html#_2016-09-06_20_32_32_711831 | 00:07 |
| *** adrian_otto has joined #openstack-infra | 00:08 | |
| sdake | oh nm | 00:08 |
| sdake | apparently its bindep.txt | 00:08 |
| sdake | im a dummy :( | 00:08 |
| *** thorst has quit IRC | 00:08 | |
| *** adrian_otto has quit IRC | 00:09 | |
| *** thorst has joined #openstack-infra | 00:09 | |
| fungi | sdake: you're hardly a dummy... if you can spot places where our documentation is unclear, suggestions on improving it are most welcome | 00:10 |
| sdake | nah - i just typoed | 00:10 |
| sdake | or didn't read the code | 00:10 |
| sdake | or something went wrong :) | 00:10 |
| sdake | anyway emergency averted :) | 00:10 |
| fungi | fair enough. we're probably not going to document "type carefully" | 00:10 |
| openstackgerrit | Kirill Zaitsev proposed openstack-infra/project-config: Mark murano-pkg-check ready for translations https://review.openstack.org/366420 | 00:11 |
| fungi | it's already printed on all our beer mats anyway | 00:11 |
| sdake | fungi fwiw i rrely reada docs unlelss its a aproject i ork 80% or more on | 00:11 |
| fungi | nice ;) | 00:11 |
| sdake | i spend alot of time on infra work - but its mostly rltd to kolla at this point | 00:11 |
| fungi | *idea for summit swag: openstack infra beer mats) | 00:12 |
| fungi | deathwish coffee sent me some really simple blank-on-cork coasters. i bet i could find a place that prints those in bulk | 00:13 |
| fungi | er, black-on-cork | 00:14 |
| pleia2 | or just paper ones https://www.evermine.com/beer_coasters/ | 00:14 |
| pleia2 | need to wait for our ant though ;) | 00:14 |
| sdake | oh did you get the ant? | 00:15 |
| *** Benj_ has quit IRC | 00:15 | |
| fungi | yeah, the cardboard ones i'm sure can do multi-tone process | 00:15 |
| sdake | iirc gluster has an ant as their logo as well | 00:15 |
| fungi | also, i guess cork is endangered now or something | 00:15 |
| sdake | or mascot | 00:15 |
| sdake | or whatever | 00:15 |
| sdake | fungi ya very disturbing to see all the plastic corks in wine these days | 00:15 |
| fungi | sdake: i suggested we claim ours is a velvet ant instead, which is actually a kind of wasp | 00:15 |
| pleia2 | sneaky | 00:16 |
| * fungi hates wading through diffs of batches of code contributors to weed out duplicates resulting from account merges and transitions from extra-atc to regular atc | 00:17 | |
| *** Apoorva has quit IRC | 00:17 | |
| *** thorst has quit IRC | 00:17 | |
| fungi | that whole "attend ptgs to get free admission to summits" plan can't come soon enough | 00:18 |
| *** mmedvede has quit IRC | 00:19 | |
| sdake | fungi since i've got you | 00:19 |
| sdake | it looks like we need libffi-devel | 00:19 |
| sdake | but this is clled something else on ubuntu | 00:19 |
| sdake | does bindep automatically translate that? | 00:19 |
| sdake | or do we specify both? | 00:19 |
| sdake | or how to tckle that? | 00:20 |
| clarkb | you have to specify both. The bindep docs should cover it in a lot of detail | 00:20 |
| sdake | oh docs | 00:20 |
| clarkb | you basically tag package names with the platofrm that they exist on | 00:20 |
| fungi | sdake: yeah, we have several distro-specific lines for it in the openstack-infra/project-config:jenkins/data/bindep-fallback.txt file you can copy | 00:20 |
| sdake | fungi cool i'll look there thanks :) | 00:20 |
| sdake | thanksclarkb | 00:20 |
| sdake | thanks clarkb | 00:20 |
| fungi | there's one for rpm-based distros, one for dpkg-based distros, and one for gentoo | 00:21 |
| fungi | they're all clustered on adjacent lines... i know because i linked someone in a similar situation to it earlier today | 00:21 |
| *** AnarchyAo has joined #openstack-infra | 00:23 | |
| *** roxanaghe has joined #openstack-infra | 00:24 | |
| sdake | whats the one for gentoo - for coreos? | 00:24 |
| *** mriedem has quit IRC | 00:24 | |
| sdake | we have a kollian cat who got olla running on gentoo | 00:24 |
| sdake | to my amazement | 00:24 |
| *** mriedem has joined #openstack-infra | 00:24 | |
| clarkb | just gentoo | 00:24 |
| clarkb | iirc | 00:25 |
| fungi | we have gentoo users who want to run things like unit tests | 00:25 |
| *** ddieterly has quit IRC | 00:25 | |
| clarkb | not sure about coreos | 00:25 |
| sdake | fair enough | 00:25 |
| sdake | the gentoo cat in our channel said 3% of distros wer egentoo | 00:25 |
| fungi | bindep.txt entries are not purely for ci, in fact they're intended more as documentation/convenience to devs so they can know what they need installed in their local system to have a working dev environment | 00:25 |
| sdake | is binddep.txt a python standard thing? | 00:26 |
| *** sdague has quit IRC | 00:26 | |
| fungi | it's a thing we made up because there was no standard | 00:26 |
| fungi | but we made it up in a non-openstack-community-specific way such that it could become a standard | 00:26 |
| mordred | yup. I can't believe we made it to 2016 without there being a thing like bindep | 00:26 |
| clarkb | we were all so happy with our single distro caves | 00:28 |
| *** roxanaghe has quit IRC | 00:28 | |
| fungi | the success of free software has made people increasingly need to emerge from their distro-specific caves and interact with one another | 00:29 |
| tonyb | Do I need to be authed to use the (gerrit) /accounts REST endpoint? | 00:29 |
| fungi | tonyb: i don't believe so... checking now | 00:29 |
| *** jamesdenton has joined #openstack-infra | 00:29 | |
| tonyb | fungi: Thanks. | 00:29 |
| fungi | tonyb: teh tools/owners.py script in openstack-infra/system-config is an example of using the accounts methods without authentication | 00:30 |
| *** Apoorva has joined #openstack-infra | 00:30 | |
| fungi | so, no it's not needed | 00:30 |
| *** sarob has joined #openstack-infra | 00:30 | |
| tonyb | fungi: \o/ Thanks. | 00:32 |
| *** sarob has quit IRC | 00:33 | |
| armax | clarkb: btw https://bugs.launchpad.net/neutron/+bug/1620864 | 00:33 |
| openstack | Launchpad bug 1620864 in neutron "control size of neutron logs" [High,Confirmed] - Assigned to Armando Migliaccio (armando-migliaccio) | 00:33 |
| fungi | tonyb: worth noting, gerrit basically has two api endpoints... an authenticated one (prefixed by "a/") and an anonymous one (no prefix), so if you try to use the authenticated endpoint without authenticating, you will get permission errors even though the same methods may be available via the anonymous endpoint | 00:34 |
| * fungi learned this the usual, painful way one learns things about apis | 00:34 | |
| *** Apoorva has quit IRC | 00:35 | |
| *** ddieterly has joined #openstack-infra | 00:37 | |
| *** esberglu has joined #openstack-infra | 00:38 | |
| tonyb | fungi: Thanks. It turned out I could just add DETAILED_ACCOUNTS to the options I passed when getting the reviews and save the second query, which I didn't get from reading the docs | 00:38 |
| *** dave4925_ has joined #openstack-infra | 00:39 | |
| *** dave4925_ has joined #openstack-infra | 00:40 | |
| *** dave4925_ has quit IRC | 00:40 | |
| fungi | tonyb: yep, i've found a number of optimizations like that | 00:40 |
| *** dave4925_ has joined #openstack-infra | 00:40 | |
| *** dave4925_ has joined #openstack-infra | 00:40 | |
| fungi | their api is getting increasingly good about having options to return related objects with your query results so you can avoid chaining multiple queries | 00:40 |
| *** dave4925_ has joined #openstack-infra | 00:41 | |
| *** dave4925_ has joined #openstack-infra | 00:41 | |
| tonyb | fungi: yeah it actually works pretty well for me | 00:41 |
| *** Illumitardi has quit IRC | 00:42 | |
| *** kaisers has quit IRC | 00:42 | |
| *** kaisers has joined #openstack-infra | 00:43 | |
| *** jamesdenton has quit IRC | 00:43 | |
| fungi | tonyb: for the record, that owners.py script i mentioned is what i've been using to generate electoral rolls (among other things) and with our latest gerrit backports from upstream it's able to run entirely anonymously | 00:43 |
| fungi | so there's a ton of data you can get at in the gerrit rest api now without authenticating at all | 00:44 |
| tonyb | fungi: Oh cool. I thought there was an aspect of it that needed to gerrit admin perms | 00:44 |
| fungi | up until recently, yes | 00:44 |
| tonyb | fungi: does that mean in theory one of the lecetion officals could generate the rolls this time round? | 00:44 |
| fungi | tonyb: yeah, i'm planning to work with them so that they generate the rolls, but i'll generate a copy too and we can compare results just to be sure | 00:45 |
| tonyb | fungi: I like that plan | 00:45 |
| fungi | but actually, it means _anybody_ can generate electoral rolls, not just election officials | 00:45 |
| tonyb | fungi: :) | 00:45 |
| *** esberglu has quit IRC | 00:45 | |
| fungi | we are openstack | 00:46 |
| clarkb | also our release is a hurricane | 00:46 |
| fungi | in the pacific, of all places | 00:46 |
| tonyb | fungi: we might want to consider moving the tool to a different repo. I can see it being helpful for a bunch of things | 00:46 |
| *** Jeffrey4l has joined #openstack-infra | 00:47 | |
| fungi | so pacific tropical cyclone forming in the western hemisphere, which is a bit unusual to make landfall aiui | 00:47 |
| fungi | hence it being a hurricane and not a typhoon | 00:47 |
| fungi | tonyb: yep, system-config has, among other things, been a dumping ground for random scripts infra root sysadmins ran | 00:47 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Run bashate test over devstack-gate too https://review.openstack.org/236819 | 00:47 |
| fungi | but there are several in there that could go elsewhere | 00:48 |
| clarkb | fungi: hawaii gets hit by them occasionally | 00:48 |
| clarkb | but ya for them to no NE I think is less common | 00:48 |
| tonyb | fungi: the trick will be finding a good spot and then turning things into a helpful library | 00:48 |
| fungi | maybe someone will turn my teribly hackish code into good software | 00:49 |
| fungi | (every developer's dream) | 00:49 |
| *** vhosakot has joined #openstack-infra | 00:50 | |
| tonyb | fungi: :) | 00:50 |
| *** mmedvede has joined #openstack-infra | 00:54 | |
| *** Sukhdev has joined #openstack-infra | 00:56 | |
| *** spzala has joined #openstack-infra | 00:57 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tripleo: enable voting for scenarios https://review.openstack.org/366428 | 00:58 |
| *** Sukhdev has quit IRC | 01:01 | |
| *** aviau has quit IRC | 01:03 | |
| *** aviau has joined #openstack-infra | 01:03 | |
| *** vhosakot has quit IRC | 01:05 | |
| *** vhosakot has joined #openstack-infra | 01:06 | |
| *** caowei has joined #openstack-infra | 01:08 | |
| *** kzaitsev_mb has quit IRC | 01:10 | |
| *** fguillot_ has quit IRC | 01:10 | |
| *** gongysh has joined #openstack-infra | 01:12 | |
| *** fguillot_ has joined #openstack-infra | 01:13 | |
| *** thorst has joined #openstack-infra | 01:14 | |
| pabelanger | clarkb: so, nodepool image-list just changes all images currently as building to ready, even though uploads are not finished | 01:15 |
| *** thorst has quit IRC | 01:16 | |
| *** thorst has joined #openstack-infra | 01:16 | |
| *** rossella_s has quit IRC | 01:18 | |
| *** rossella_s has joined #openstack-infra | 01:19 | |
| *** JerryOpenix has quit IRC | 01:19 | |
| *** vhosakot has quit IRC | 01:22 | |
| *** tphummel has quit IRC | 01:23 | |
| fungi | that sounds like not what we want | 01:24 |
| *** isviridov_away has quit IRC | 01:25 | |
| clarkb | pabelanger: I don't see timestamps lining up with that in the image-lsit output | 01:26 |
| *** Julien-zte has joined #openstack-infra | 01:26 | |
| clarkb | eg no ready with age all ebing the same in the last 10 minutes | 01:27 |
| clarkb | I see some staggering | 01:27 |
| *** isviridov_away has joined #openstack-infra | 01:28 | |
| clarkb | | 51137 | rax-ord | centos-7 | centos-7 | 1473181364 | 269d13ec-4c45-4a4a-840c-eb7e434bd28f | None | ready | 00:00:25:12 | looks like one of the more recent ones | 01:28 |
| *** esberglu has joined #openstack-infra | 01:29 | |
| *** jamesdenton has joined #openstack-infra | 01:29 | |
| *** amotoki has joined #openstack-infra | 01:29 | |
| clarkb | and it shows up in rax-ord with image show (though the disk size seems small for it) | 01:30 |
| clarkb | pabelanger: I think what may have happened is we hit a timeout for the builds so they were all deleted not transitioend from building to ready | 01:30 |
| clarkb | pabelanger: except that we can't delete on the builder side until the uploads are done so there is a weird coordination issue with the two daemons? | 01:31 |
| *** thorst has quit IRC | 01:32 | |
| *** zhurong has joined #openstack-infra | 01:32 | |
| *** Julien-zte has quit IRC | 01:32 | |
| *** amotoki has quit IRC | 01:33 | |
| clarkb | and I thought my patch fixed this. Even wrote a test :( | 01:35 |
| clarkb | bah I see the issue :( | 01:35 |
| *** zshuo has joined #openstack-infra | 01:36 | |
| *** ddieterly has quit IRC | 01:36 | |
| pabelanger | clarkb: ya, ubuntu-trusty just finished uploading in rax-dfw, but nodepool didn't register it | 01:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Don't delete images in a building state https://review.openstack.org/366434 | 01:38 |
| clarkb | pabelanger: ^ thats the fix | 01:39 |
| clarkb | you can see the all got deleted if you grep for that log line | 01:39 |
| clarkb | so pretty sure thats what happened here | 01:39 |
| clarkb | pabelanger: they did not transition from building to ready, they transitioned from building to delete, and were removed from the table | 01:39 |
| clarkb | which is probably a big part of why we leak these images in the first place | 01:39 |
| *** roxanaghe has joined #openstack-infra | 01:39 | |
| clarkb | (yay for finally sorting that out so that things can be cleaned up) | 01:39 |
| pabelanger | clarkb: Ah, cool | 01:40 |
| *** hongbin has joined #openstack-infra | 01:40 | |
| pabelanger | Right, that would explain the leaked images for sure | 01:40 |
| *** ddieterly has joined #openstack-infra | 01:40 | |
| clarkb | I should add a test for that too | 01:41 |
| pabelanger | k | 01:41 |
| clarkb | my previous change was completely fine it just didn't realize there were the two tables | 01:41 |
| pabelanger | I am going to stop nodepool-builder so we don't leak more images | 01:41 |
| pabelanger | then manually upload | 01:41 |
| pabelanger | okay, starting rackspace uploads for ubuntu-xenial | 01:43 |
| *** baoli has joined #openstack-infra | 01:43 | |
| *** roxanaghe has quit IRC | 01:44 | |
| pabelanger | uploading now | 01:44 |
| *** yamahata has quit IRC | 01:44 | |
| *** yanyanhu has joined #openstack-infra | 01:44 | |
| *** amotoki has joined #openstack-infra | 01:46 | |
| clarkb | pabelanger: you have to have a running builder for uploads to work. Or do you mean you are using mordred's shade hack and not nodepool image-upload? | 01:46 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Don't delete images in a building state https://review.openstack.org/366434 | 01:46 |
| clarkb | there is the test | 01:46 |
| clarkb | I need to disappear for dinner now | 01:47 |
| *** esberglu has quit IRC | 01:47 | |
| clarkb | but that should work jsut fine as its based on the earlier code | 01:47 |
| lifeless | clarkb: python 2.6? No, noone has done the work to do that inside the openstack infra. | 01:48 |
| pabelanger | clarkb: no, I just stopped / started nodepool-builder to clear out uploads in process. Since nodepool won't see the images and register them, then manually did image-upload | 01:48 |
| clarkb | well we'll still leak those images | 01:50 |
| clarkb | since the db no longer knows about them | 01:50 |
| clarkb | I think once the above change gets in and we confirm we aren't doing the delete of building images after 8 hours we can redo the leaked iamge cleanup and that should get us really close to happyness | 01:50 |
| *** pvaneck has quit IRC | 01:52 | |
| pabelanger | ok | 01:53 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on summit event description https://review.openstack.org/366441 | 01:54 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on summit event description https://review.openstack.org/366441 | 01:55 |
| *** spzala has quit IRC | 01:57 | |
| *** spzala has joined #openstack-infra | 01:57 | |
| *** ijw has joined #openstack-infra | 01:58 | |
| *** esberglu has joined #openstack-infra | 01:58 | |
| *** ijw has quit IRC | 01:58 | |
| *** r1chardj0n3s has joined #openstack-infra | 01:58 | |
| openstackgerrit | Richard Jones proposed openstack-infra/project-config: Add missing xstatic projects to Horizon channel https://review.openstack.org/366442 | 01:59 |
| *** ijw has joined #openstack-infra | 01:59 | |
| *** spzala has quit IRC | 02:02 | |
| *** amotoki has quit IRC | 02:05 | |
| *** ddieterly has quit IRC | 02:06 | |
| *** ijw has quit IRC | 02:09 | |
| mordred | clarkb: just glanced in - is there a shade-side issue casing that? or kjust bad logic | 02:10 |
| *** r1chardj0n3s has left #openstack-infra | 02:11 | |
| clarkb | just bad nodepool logic | 02:14 |
| *** asettle has joined #openstack-infra | 02:15 | |
| *** inc0 has joined #openstack-infra | 02:16 | |
| *** mriedem has quit IRC | 02:17 | |
| *** esberglu_ has joined #openstack-infra | 02:18 | |
| *** esberglu has quit IRC | 02:19 | |
| *** hongbin has quit IRC | 02:20 | |
| *** amotoki has joined #openstack-infra | 02:20 | |
| *** asettle has quit IRC | 02:20 | |
| *** hongbin has joined #openstack-infra | 02:20 | |
| *** baoli has quit IRC | 02:21 | |
| *** esberglu_ has quit IRC | 02:22 | |
| *** priteau has joined #openstack-infra | 02:22 | |
| *** akshai has joined #openstack-infra | 02:23 | |
| *** sarob has joined #openstack-infra | 02:24 | |
| *** thcipriani is now known as thcipriani|afk | 02:24 | |
| *** thorst has joined #openstack-infra | 02:24 | |
| *** sarob has quit IRC | 02:25 | |
| *** baoli has joined #openstack-infra | 02:26 | |
| *** priteau has quit IRC | 02:27 | |
| *** amotoki has quit IRC | 02:28 | |
| *** dcwangmit01 has quit IRC | 02:28 | |
| *** dcwangmit01 has joined #openstack-infra | 02:29 | |
| *** thorst has quit IRC | 02:29 | |
| *** salv-orlando has joined #openstack-infra | 02:30 | |
| *** akshai has quit IRC | 02:30 | |
| *** salv-orl_ has quit IRC | 02:33 | |
| *** spzala has joined #openstack-infra | 02:35 | |
| *** spzala has quit IRC | 02:35 | |
| *** baoli has quit IRC | 02:35 | |
| *** spzala has joined #openstack-infra | 02:36 | |
| *** spzala has quit IRC | 02:36 | |
| *** spzala has joined #openstack-infra | 02:36 | |
| *** spzala has quit IRC | 02:37 | |
| *** spzala has joined #openstack-infra | 02:37 | |
| *** spzala has quit IRC | 02:37 | |
| *** spzala has joined #openstack-infra | 02:38 | |
| *** vhosakot has joined #openstack-infra | 02:41 | |
| *** amotoki has joined #openstack-infra | 02:41 | |
| *** spzala has quit IRC | 02:42 | |
| *** akshai has joined #openstack-infra | 02:42 | |
| *** vhosakot has quit IRC | 02:44 | |
| *** thorst has joined #openstack-infra | 02:48 | |
| *** thorst has quit IRC | 02:48 | |
| *** amotoki has quit IRC | 02:48 | |
| *** thorst has joined #openstack-infra | 02:48 | |
| *** dimtruck is now known as zz_dimtruck | 02:50 | |
| *** jamesdenton has quit IRC | 02:58 | |
| *** spzala has joined #openstack-infra | 02:58 | |
| *** waynr has quit IRC | 03:00 | |
| *** jogo has joined #openstack-infra | 03:03 | |
| *** tongli has quit IRC | 03:03 | |
| *** fguillot_ has quit IRC | 03:03 | |
| *** hongbin has quit IRC | 03:08 | |
| *** spzala has quit IRC | 03:13 | |
| *** xarses has quit IRC | 03:18 | |
| *** akscram has quit IRC | 03:19 | |
| *** akscram has joined #openstack-infra | 03:20 | |
| *** psachin has joined #openstack-infra | 03:26 | |
| *** roxanaghe has joined #openstack-infra | 03:28 | |
| *** akscram has quit IRC | 03:29 | |
| *** waynr has joined #openstack-infra | 03:29 | |
| *** akscram has joined #openstack-infra | 03:30 | |
| *** vikrant has joined #openstack-infra | 03:31 | |
| *** Apoorva has joined #openstack-infra | 03:32 | |
| *** roxanaghe has quit IRC | 03:32 | |
| *** sdake_ has joined #openstack-infra | 03:32 | |
| *** sdake has quit IRC | 03:35 | |
| *** gouthamr has quit IRC | 03:37 | |
| *** Apoorva has quit IRC | 03:37 | |
| *** kambiz has quit IRC | 03:40 | |
| *** inc0 has quit IRC | 03:41 | |
| *** gouthamr has joined #openstack-infra | 03:45 | |
| *** links has joined #openstack-infra | 03:45 | |
| *** coolsvap_ has joined #openstack-infra | 03:50 | |
| *** markvoelker has quit IRC | 03:54 | |
| *** jascott1_ has joined #openstack-infra | 04:00 | |
| *** jascott1 has quit IRC | 04:01 | |
| *** dotplus has quit IRC | 04:13 | |
| *** roxanaghe has joined #openstack-infra | 04:19 | |
| tristanC | ttx: Could you please tag the governance repo with the sept-2016-elections tag ? | 04:19 |
| tristanC | ttx: also note that we change a little bit the proposed timeline to match with the newton release schedule, it's available here: http://governance.openstack.org/election/ | 04:20 |
| *** amotoki has joined #openstack-infra | 04:20 | |
| *** gouthamr has quit IRC | 04:21 | |
| *** roxanaghe has quit IRC | 04:22 | |
| *** roxanaghe has joined #openstack-infra | 04:25 | |
| *** senk has joined #openstack-infra | 04:26 | |
| *** akshai has quit IRC | 04:27 | |
| *** stevemar has quit IRC | 04:28 | |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: Congress - Add replicated PE job https://review.openstack.org/365862 | 04:31 |
| *** thorst has quit IRC | 04:43 | |
| *** thorst has joined #openstack-infra | 04:45 | |
| *** akshai has joined #openstack-infra | 04:48 | |
| *** thorst has quit IRC | 04:50 | |
| *** markvoelker has joined #openstack-infra | 04:55 | |
| *** senk has quit IRC | 04:55 | |
| *** thorst has joined #openstack-infra | 04:56 | |
| *** abregman has joined #openstack-infra | 04:57 | |
| *** markvoelker has quit IRC | 04:59 | |
| *** thorst has quit IRC | 05:00 | |
| *** Jeffrey4l_ has joined #openstack-infra | 05:00 | |
| *** claudiub has joined #openstack-infra | 05:03 | |
| openstackgerrit | Shu Muto proposed openstack-infra/project-config: Add new project Zun-UI to openstack repo https://review.openstack.org/366489 | 05:04 |
| *** Jeffrey4l has quit IRC | 05:04 | |
| *** claudiub|2 has joined #openstack-infra | 05:05 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 05:05 | |
| *** xarses has joined #openstack-infra | 05:06 | |
| *** asselin_ has quit IRC | 05:08 | |
| *** honza has quit IRC | 05:08 | |
| *** claudiub has quit IRC | 05:09 | |
| *** tonytan_brb has quit IRC | 05:09 | |
| *** yamahata has joined #openstack-infra | 05:11 | |
| *** jaosorior has joined #openstack-infra | 05:12 | |
| *** ilyashakhat_mobi has quit IRC | 05:14 | |
| openstackgerrit | Merged openstack-infra/project-config: Add missing deb repo 1/5 https://review.openstack.org/366318 | 05:14 |
| *** harlowja_at_home has joined #openstack-infra | 05:18 | |
| *** openstackgerrit has quit IRC | 05:18 | |
| *** rossella_s has quit IRC | 05:18 | |
| *** openstackgerrit has joined #openstack-infra | 05:18 | |
| *** rossella_s has joined #openstack-infra | 05:19 | |
| toabctl | fungi, tarballs.openstack.org has a nova-master tarball containing nova-14.0.0.0b3.dev842 but yesterday there was already nova-14.0.0.0b4.dev6 . is that problem known? | 05:21 |
| AJaeger | toabctl: b3 vs b4 | 05:24 |
| AJaeger | that looks fine to me | 05:24 |
| *** akshai has quit IRC | 05:24 | |
| toabctl | AJaeger, isn't b4 higher than b3? | 05:25 |
| AJaeger | yes, it is. And thuse dev starts at zeroagain. | 05:25 |
| AJaeger | Ah, now I get what you mean... | 05:25 |
| AJaeger | sorry, too early for me ;/ | 05:25 |
| toabctl | :-) | 05:26 |
| *** roxanaghe has quit IRC | 05:27 | |
| * AJaeger gets some fresh air now to wake up fully. bbl | 05:27 | |
| *** thorst has joined #openstack-infra | 05:28 | |
| *** akshai has joined #openstack-infra | 05:29 | |
| *** senk has joined #openstack-infra | 05:29 | |
| *** Sukhdev has joined #openstack-infra | 05:29 | |
| *** thorst has quit IRC | 05:29 | |
| *** thorst has joined #openstack-infra | 05:32 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 05:35 |
| *** shu-mutou has joined #openstack-infra | 05:36 | |
| *** akshai has quit IRC | 05:39 | |
| *** matbu|bbl is now known as matbu | 05:42 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 05:44 | |
| *** sdake_ is now known as sdake | 05:46 | |
| *** ilyashakhat_mobi has quit IRC | 05:48 | |
| *** adriant has quit IRC | 05:51 | |
| *** Sukhdev has quit IRC | 05:51 | |
| *** Apoorva has joined #openstack-infra | 05:52 | |
| *** e0ne has joined #openstack-infra | 05:52 | |
| *** harlowja_at_home has quit IRC | 05:54 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on summit event description https://review.openstack.org/366441 | 05:54 |
| *** Apoorva has quit IRC | 05:56 | |
| *** rajinir has quit IRC | 06:05 | |
| *** AnarchyAo has quit IRC | 06:05 | |
| *** stevemar has joined #openstack-infra | 06:06 | |
| *** tonytan4ever has joined #openstack-infra | 06:10 | |
| *** stevemar has quit IRC | 06:11 | |
| *** pgadiya has joined #openstack-infra | 06:13 | |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config: add #openstack-ec2api to the list of OpenStack channels https://review.openstack.org/366510 | 06:14 |
| *** tonytan4ever has quit IRC | 06:15 | |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/system-config: add openstack-ec2api irc channel to list of logged channels https://review.openstack.org/357636 | 06:15 |
| *** sdake has quit IRC | 06:18 | |
| *** oanson has joined #openstack-infra | 06:18 | |
| *** shardy has joined #openstack-infra | 06:19 | |
| *** _nadya_ has joined #openstack-infra | 06:20 | |
| oanson | Hello | 06:21 |
| oanson | I have sent patch https://review.openstack.org/#/c/366033/ to add a CI test to Dragonflow. | 06:22 |
| *** eranrom_ is now known as eranrom | 06:22 | |
| oanson | I have sent patch https://review.openstack.org/#/c/366035/ to test this change (with a matching Depends-On: ... field in the commit message). | 06:22 |
| oanson | But I can't see the new test in the second patch. | 06:22 |
| oanson | Could someone please assist? | 06:22 |
| frickler | oanson: the Depends-On doesn't work for project-config changes, so you need the first change to be merged before you can see results for that job | 06:25 |
| *** thorst has quit IRC | 06:26 | |
| oanson | frickler: I see. Thank you. | 06:26 |
| *** thorst has joined #openstack-infra | 06:28 | |
| *** thorst has quit IRC | 06:28 | |
| odyssey4me | clarkb ok, I'll ask mhayden to get on reducing https://bugs.launchpad.net/openstack-ansible/+bug/1620849 from the gate jobs... and we'll see what we can do about compressing all our logs | 06:31 |
| openstack | Launchpad bug 1620849 in openstack-ansible "openstack ansible test jobs are logging excessive amounts of data" [Undecided,New] | 06:31 |
| *** dahoo has quit IRC | 06:33 | |
| odyssey4me | ianw thanks for digging into it | 06:33 |
| *** pcaruana has joined #openstack-infra | 06:34 | |
| sshnaidm | do AFS pypi mirrors have a problem? I see jobs fail to connect to it sometimes: http://logs.openstack.org/59/364859/23/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/7c45f3c/console.html#_2016-09-07_06_05_48_920830 | 06:36 |
| *** ilyashakhat_mobi has joined #openstack-infra | 06:40 | |
| *** ggnel_t has quit IRC | 06:41 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Save console logs from all vms to files https://review.openstack.org/365369 | 06:43 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Save console logs from all vms to files https://review.openstack.org/365369 | 06:43 |
| *** Ravikiran_K has joined #openstack-infra | 06:44 | |
| *** stevemar has joined #openstack-infra | 06:44 | |
| *** mminesh has joined #openstack-infra | 06:44 | |
| AJaeger | oanson: http://docs.openstack.org/infra/manual/developers.html#limitations-and-caveats | 06:44 |
| yolanda | good morning | 06:45 |
| *** martinkopec has joined #openstack-infra | 06:46 | |
| AJaeger | sshnaidm: that does not look like AFS mirror problem - and it could as well be a tripleo cloud problem | 06:46 |
| AJaeger | morning, yolanda | 06:46 |
| *** pgadiya has quit IRC | 06:47 | |
| sshnaidm | AJaeger, I see pip fails to download pbr module from AFS pypi mirror. | 06:48 |
| *** isaacb has joined #openstack-infra | 06:48 | |
| sshnaidm | AJaeger, it may be a problem between AFS and tripleo cloud, right, but no other networks problem were detected | 06:49 |
| sshnaidm | AJaeger, do you have a good idea how to check it? I can run a few pip installs in a row, just to see if something fails.. | 06:49 |
| *** vinaypotluri has quit IRC | 06:52 | |
| ianw | odyssey4me: np ... zipping is easy ... but i wonder how useful all those logs are .... | 06:53 |
| odyssey4me | ianw yeah, we don't really need the audit logs for CI, but we do need to exercise the security role to ensure that it is working properly | 06:53 |
| *** andreas_s has joined #openstack-infra | 06:54 | |
| odyssey4me | I can put in a temporary patch to disable that part of it and figure something out with mhayden when he comes online. | 06:54 |
| odyssey4me | meanwhile, I'll also look at zipping up our logs | 06:54 |
| AJaeger | sshnaidm: not really. ianw or one of the other infra-roots might be able to help. | 06:56 |
| *** markvoelker has joined #openstack-infra | 06:56 | |
| *** asettle has joined #openstack-infra | 07:00 | |
| *** devananda|travel has quit IRC | 07:00 | |
| *** thorst has joined #openstack-infra | 07:00 | |
| *** Na3iL has joined #openstack-infra | 07:01 | |
| *** markvoelker has quit IRC | 07:01 | |
| *** e0ne has quit IRC | 07:01 | |
| *** tesseract- has joined #openstack-infra | 07:02 | |
| *** pgadiya has joined #openstack-infra | 07:03 | |
| *** sdake has joined #openstack-infra | 07:03 | |
| *** jascott1_ has quit IRC | 07:04 | |
| *** jascott1 has joined #openstack-infra | 07:05 | |
| *** mptacekx has joined #openstack-infra | 07:05 | |
| *** berendt has joined #openstack-infra | 07:05 | |
| *** thorst has quit IRC | 07:06 | |
| odyssey4me | ianw clarkb first patch is up: https://review.openstack.org/366534 | 07:06 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add ironic-dsvm-multitest experimental job. https://review.openstack.org/340295 | 07:07 |
| *** nmagnezi_ has joined #openstack-infra | 07:08 | |
| *** shardy is now known as shardy_afk | 07:08 | |
| *** Julien-zte has joined #openstack-infra | 07:11 | |
| openstackgerrit | Zhenyu Zheng proposed openstack-infra/project-config: Add api-ref and releasenote jobs for Nimble https://review.openstack.org/366540 | 07:13 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 2/5 https://review.openstack.org/366337 | 07:15 |
| *** matrohon has joined #openstack-infra | 07:16 | |
| *** mikelk has joined #openstack-infra | 07:16 | |
| AJaeger | odyssey4me: thanks for that! | 07:21 |
| oanson | AJaeger, thank you. | 07:22 |
| openstackgerrit | Zhenyu Zheng proposed openstack-infra/project-config: Add api-ref and releasenote jobs for Nimble https://review.openstack.org/366540 | 07:22 |
| *** ilyashakhat_mobi has quit IRC | 07:22 | |
| *** hashar has joined #openstack-infra | 07:23 | |
| *** drifterza has joined #openstack-infra | 07:23 | |
| *** jpich has joined #openstack-infra | 07:24 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: zun: fix post script dir https://review.openstack.org/366545 | 07:27 |
| *** abregman_ has joined #openstack-infra | 07:28 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add ironic-dsvm-multitest experimental job. https://review.openstack.org/340295 | 07:29 |
| *** abregman has quit IRC | 07:31 | |
| *** slaweq__ has quit IRC | 07:31 | |
| *** rlandy|bbl is now known as rlandy | 07:31 | |
| *** rlandy has quit IRC | 07:31 | |
| *** armax has quit IRC | 07:31 | |
| *** florianf has joined #openstack-infra | 07:31 | |
| *** slaweq_ has joined #openstack-infra | 07:32 | |
| *** jascott1 has quit IRC | 07:33 | |
| *** _nadya_ has quit IRC | 07:34 | |
| openstackgerrit | Zhenyu Zheng proposed openstack-infra/project-config: Add api-ref and releasenote jobs for Nimble https://review.openstack.org/366540 | 07:35 |
| ttx | tristanC: tagging now | 07:36 |
| *** thorst has joined #openstack-infra | 07:36 | |
| *** thorst has quit IRC | 07:36 | |
| AJaeger | ttx, when will stable branches for projects like nova etc get created? With RC1? | 07:37 |
| *** yaume has joined #openstack-infra | 07:38 | |
| ttx | AJaeger: yes. We create them as release branches at RC1, they turn into stable branches at release time | 07:39 |
| ttx | tristanC: done | 07:39 |
| *** woodster_ has quit IRC | 07:39 | |
| *** ifarkas_afk is now known as ifarkas | 07:42 | |
| *** Rockyg has joined #openstack-infra | 07:43 | |
| yolanda | AJaeger, about https://review.openstack.org/#/c/366442/... i thought we were not allowing new channels in gerritbot? or is that for some other bot? | 07:45 |
| *** tonytan4ever has joined #openstack-infra | 07:47 | |
| AJaeger | ttx, thanks. | 07:47 |
| AJaeger | yolanda: taht's not adding a new channel: Just new notifications to existing channel - isn't it? | 07:48 |
| yolanda | ah so is ok to add projects into existing channels, but not add new channels, right? | 07:49 |
| yolanda | capacity problems come on some limit per channel then? | 07:49 |
| openstackgerrit | Merged openstack-infra/project-config: [Manila] Add services to minimal-dsvm job https://review.openstack.org/347180 | 07:51 |
| *** _nadya_ has joined #openstack-infra | 07:51 | |
| *** tonytan4ever has quit IRC | 07:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Make py35 voting for Horizon https://review.openstack.org/359123 | 07:52 |
| *** miniman has joined #openstack-infra | 07:52 | |
| *** ihrachys has joined #openstack-infra | 07:52 | |
| *** abregman_ has quit IRC | 07:52 | |
| *** matthewbodkin has joined #openstack-infra | 07:55 | |
| *** asettle has quit IRC | 07:56 | |
| *** shardy_afk is now known as shardy | 07:57 | |
| AJaeger | yolanda: correct, we cannot add gerritbot to new channels. But the testsuite will fail for that case, so no worries. If you have a +1 from the irc check, the change is fine on that part ;) | 07:57 |
| openstackgerrit | Merged openstack-infra/project-config: Run bashate test over devstack-gate too https://review.openstack.org/236819 | 07:57 |
| AJaeger | yolanda: we can only handle 126 (?) or so channels for a single bot. | 07:58 |
| AJaeger | that's a freenode limit | 07:58 |
| yolanda | AJaeger, thanks for clarifying, wanted to be sure before reviewing | 07:58 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** thorst has joined #openstack-infra | 08:01 | |
| AJaeger | sure ;) | 08:02 |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: zun: fix post script path https://review.openstack.org/366545 | 08:02 |
| *** asettle has joined #openstack-infra | 08:02 | |
| *** gildub has quit IRC | 08:03 | |
| *** csmart has quit IRC | 08:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove heat-templates-buildimages job https://review.openstack.org/365370 | 08:05 |
| *** abregman_ has joined #openstack-infra | 08:07 | |
| *** ggnel_t has joined #openstack-infra | 08:07 | |
| *** stevemar_ has joined #openstack-infra | 08:07 | |
| *** csmart has joined #openstack-infra | 08:07 | |
| *** thorst_ has joined #openstack-infra | 08:10 | |
| *** thorst__ has joined #openstack-infra | 08:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Add missing xstatic projects to Horizon channel https://review.openstack.org/366442 | 08:11 |
| openstackgerrit | Merged openstack-infra/project-config: Enable a missing Magnum job https://review.openstack.org/366155 | 08:11 |
| *** stevemar_ has quit IRC | 08:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Ignore openstack-ansible for ansible-func-ubuntu-trusty https://review.openstack.org/366225 | 08:12 |
| *** thorst has quit IRC | 08:12 | |
| *** dingyichen has quit IRC | 08:12 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Gnocchi functional jobs to Xenial https://review.openstack.org/366573 | 08:12 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Telemetry jobs to Xenial https://review.openstack.org/366574 | 08:12 |
| *** thorst_ has quit IRC | 08:14 | |
| odyssey4me | ianw AJaeger clarkb I expect that https://review.openstack.org/366570 should also compress the logs well | 08:15 |
| *** dizquierdo has joined #openstack-infra | 08:17 | |
| *** lucas-dinner is now known as lucasagomes | 08:22 | |
| *** aspiers has joined #openstack-infra | 08:23 | |
| *** yuval has joined #openstack-infra | 08:25 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/366578 | 08:26 |
| openstackgerrit | Joe Hakim Rahme proposed openstack-infra/project-config: Adds a new project: nova-tempest-staging https://review.openstack.org/366091 | 08:26 |
| *** priteau has joined #openstack-infra | 08:27 | |
| *** salv-orl_ has joined #openstack-infra | 08:29 | |
| *** mhickey__ has joined #openstack-infra | 08:31 | |
| *** thorst__ has quit IRC | 08:32 | |
| *** flepied has quit IRC | 08:32 | |
| *** nmagnezi_ is now known as nmagnezi | 08:32 | |
| *** salv-orlando has quit IRC | 08:32 | |
| *** claudiub has joined #openstack-infra | 08:32 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 08:34 | |
| *** openstackgerrit has quit IRC | 08:34 | |
| *** openstackgerrit has joined #openstack-infra | 08:34 | |
| *** thorst has joined #openstack-infra | 08:34 | |
| *** thorst has quit IRC | 08:35 | |
| *** claudiub|2 has quit IRC | 08:35 | |
| *** ihrachys has quit IRC | 08:36 | |
| *** derekh has joined #openstack-infra | 08:40 | |
| *** e0ne has joined #openstack-infra | 08:41 | |
| sshnaidm | AJaeger, yolanda, ianw, hi again, could we please check AFS pypi mirrors performance anyhow? I don't know on which side it's a problem, but it's critically slow | 08:45 |
| *** dtantsur|afk is now known as dtantsur | 08:46 | |
| sshnaidm | if it's only us that suffer from this problem, that's ok | 08:46 |
| *** shu-mutou is now known as shu-mutou-AFK | 08:46 | |
| *** thorst has joined #openstack-infra | 08:46 | |
| *** thorst has quit IRC | 08:49 | |
| yolanda | sshnaidm, is that giving you some errors? not sure if i can help with afs | 08:49 |
| yolanda | do you have any logs? | 08:50 |
| sshnaidm | yolanda, sometimes socket timeout, but mostly pip installs a few modules for about 45 minutes | 08:50 |
| sshnaidm | yolanda, http://logs.openstack.org/59/364859/23/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/7c45f3c/console.html#_2016-09-07_06_05_48_920830 | 08:51 |
| openstackgerrit | Waldemar Znoinski proposed openstack-infra/irc-meetings: add agenda link for Kolla meetings https://review.openstack.org/366597 | 08:51 |
| *** snarwade has quit IRC | 08:53 | |
| *** links has quit IRC | 08:53 | |
| *** vikrant has quit IRC | 08:53 | |
| *** vikrant has joined #openstack-infra | 08:54 | |
| *** psachin has quit IRC | 08:54 | |
| *** pgadiya has quit IRC | 08:54 | |
| *** snarwade has joined #openstack-infra | 08:54 | |
| *** priteau has quit IRC | 08:55 | |
| *** pgadiya has joined #openstack-infra | 08:55 | |
| *** links has joined #openstack-infra | 08:55 | |
| *** psachin has joined #openstack-infra | 08:55 | |
| *** markvoelker has joined #openstack-infra | 08:57 | |
| AJaeger | sshnaidm: since that only happens in tripleo, can you have somebody look at that from the tripleo side? | 09:00 |
| *** jimbaker has quit IRC | 09:01 | |
| *** acoles_ is now known as acoles | 09:01 | |
| *** markvoelker has quit IRC | 09:01 | |
| sshnaidm | AJaeger, sure | 09:01 |
| AJaeger | sshnaidm: I hope pabelanger can help later as well | 09:02 |
| wznoinsk | hi infra, could I have someone to look at this nit? (irc-meetings change): https://review.openstack.org/#/c/366597/ | 09:02 |
| AJaeger | sshnaidm: sorry, I'm not an admin, I can just advise ;) | 09:02 |
| sshnaidm | AJaeger, sure, thanks :) | 09:02 |
| AJaeger | wznoinsk: irc-meetings are handled by it's own core team - tonyb is doing that mainly AFAIK | 09:02 |
| wznoinsk | given there are only 2 cores in irc-meetings group and os-infra group is also a member of it I thought no harm to shout it here | 09:03 |
| *** dizquierdo has quit IRC | 09:04 | |
| wznoinsk | AJaeger: will catch him then, cheers | 09:04 |
| openstackgerrit | Merged openstack-infra/project-config: Correct the directory path for Zun https://review.openstack.org/366231 | 09:04 |
| *** jimbaker has joined #openstack-infra | 09:05 | |
| *** jimbaker has quit IRC | 09:05 | |
| *** jimbaker has joined #openstack-infra | 09:05 | |
| *** zshuo has quit IRC | 09:06 | |
| *** stevemar_ has joined #openstack-infra | 09:08 | |
| AJaeger | wznoinsk: infra team is not regular reviewer of it | 09:10 |
| wznoinsk | AJaeger: roger | 09:10 |
| *** stevemar_ has quit IRC | 09:13 | |
| *** flepied has joined #openstack-infra | 09:13 | |
| *** flepied has quit IRC | 09:16 | |
| *** ihrachys has joined #openstack-infra | 09:19 | |
| *** yamahata has quit IRC | 09:21 | |
| *** flepied has joined #openstack-infra | 09:21 | |
| tonyb | wznoinsk: 'sup? with irc-meetings? | 09:22 |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config: Monasca Gate Setup for ORM+PostgreSQL https://review.openstack.org/366617 | 09:23 |
| *** ilyashakhat_mobi has quit IRC | 09:23 | |
| yolanda | sshnaidm, i'm seeing some afs errors in dmesg , but i don't have the knowledge to dig further. If you can come until other infra-roots come, will be better | 09:24 |
| yolanda | basically there are several errors about lost contact with file server | 09:25 |
| wznoinsk | tonyb: yes | 09:25 |
| *** kzaitsev_mb has joined #openstack-infra | 09:26 | |
| *** ihrachys has quit IRC | 09:26 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 09:31 |
| sshnaidm | yolanda, can you paste them please? | 09:32 |
| *** ilyashakhat_mobi has joined #openstack-infra | 09:32 | |
| yolanda | sshnaidm, http://paste.openstack.org/show/567386/ | 09:33 |
| *** r-mibu has quit IRC | 09:33 | |
| sshnaidm | derekh, ^^ | 09:33 |
| sshnaidm | yolanda, thanks | 09:33 |
| *** r-mibu has joined #openstack-infra | 09:33 | |
| *** wznoinsk has quit IRC | 09:35 | |
| derekh | sshnaidm: ack, we just gotta wait for somebody from infra-root | 09:35 |
| *** ilyashakhat_mobi has quit IRC | 09:36 | |
| *** yanyanhu has quit IRC | 09:40 | |
| *** rossella_s has quit IRC | 09:44 | |
| *** rossella_s has joined #openstack-infra | 09:44 | |
| *** apetrich has quit IRC | 09:44 | |
| *** sdake has quit IRC | 09:47 | |
| *** _degorenko|afk is now known as degorenko | 09:48 | |
| *** kzaitsev_mb has quit IRC | 09:53 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Remove default security group rules on Infra clouds https://review.openstack.org/366636 | 09:53 |
| rcarrillocruz | pabelanger, clarkb , mordred : ^ | 09:54 |
| rcarrillocruz | the change to get rid of the default 'default' security group rules with the cloud launcher | 09:54 |
| *** markvoelker has joined #openstack-infra | 09:57 | |
| *** thorst has joined #openstack-infra | 10:00 | |
| jaosorior | coolsvap_: https://review.openstack.org/#/c/366631/ is needed ot get barbican to work again | 10:00 |
| *** Hal has joined #openstack-infra | 10:01 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Fix count_slot_usage.py meeting links https://review.openstack.org/366123 | 10:02 |
| *** markvoelker has quit IRC | 10:02 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: add agenda link for Kolla meetings https://review.openstack.org/366597 | 10:03 |
| *** thorst has quit IRC | 10:04 | |
| *** isaacb has quit IRC | 10:05 | |
| *** ihrachys has joined #openstack-infra | 10:05 | |
| *** zhurong has quit IRC | 10:05 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 2/5 https://review.openstack.org/366337 | 10:06 |
| coolsvap_ | jaosorior: yeah, its just part of the process, we have requirements meeting in couple of hours. We can finalize on this by then | 10:07 |
| *** Ravikiran_K has quit IRC | 10:08 | |
| *** Julien-zte has quit IRC | 10:09 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Create Architecture Working Group meeting https://review.openstack.org/366127 | 10:10 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Change Glare meeting time https://review.openstack.org/365744 | 10:10 |
| jaosorior | coolsvap_: sure, anyway, just to let you know. a barbican release would be meaningless if it's broken by that requirement. | 10:11 |
| openstackgerrit | Aleksandr Dobdin proposed openstack-infra/project-config: New Project timmy - ansible-like tool https://review.openstack.org/359831 | 10:12 |
| odyssey4me | AJaeger are there any options for us with regards to https://bugs.launchpad.net/openstack-ansible/+bug/1589842 | 10:14 |
| openstack | Launchpad bug 1589842 in openstack-ansible "Published web content still contains old files which were deleted" [Low,Confirmed] - Assigned to Jesse Pretorius (jesse-pretorius) | 10:14 |
| *** mhickey__ has quit IRC | 10:20 | |
| *** isaacb has joined #openstack-infra | 10:22 | |
| *** Hal has quit IRC | 10:26 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 10:27 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE TESTING https://review.openstack.org/316436 | 10:27 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 10:29 |
| *** ramishra has quit IRC | 10:31 | |
| *** ramishra has joined #openstack-infra | 10:32 | |
| *** psachin has quit IRC | 10:33 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 10:35 |
| *** sdague has joined #openstack-infra | 10:37 | |
| *** Julien-zte has joined #openstack-infra | 10:40 | |
| *** sdague has quit IRC | 10:41 | |
| *** isaacb has quit IRC | 10:41 | |
| *** dgonzalez has quit IRC | 10:45 | |
| *** Julien-zte has quit IRC | 10:46 | |
| *** Julien-zte has joined #openstack-infra | 10:46 | |
| *** sdague has joined #openstack-infra | 10:46 | |
| *** caowei has quit IRC | 10:48 | |
| *** thorst has joined #openstack-infra | 10:48 | |
| *** thorst has quit IRC | 10:48 | |
| *** notnownikki has joined #openstack-infra | 10:50 | |
| *** isaacb has joined #openstack-infra | 10:53 | |
| *** dgonzalez has joined #openstack-infra | 10:57 | |
| *** rtheis has joined #openstack-infra | 10:57 | |
| *** bookwar_ has quit IRC | 11:00 | |
| *** vkramskikh has quit IRC | 11:00 | |
| *** psachin has joined #openstack-infra | 11:02 | |
| *** asselin_ has joined #openstack-infra | 11:04 | |
| *** ddieterly has joined #openstack-infra | 11:06 | |
| *** slagle has joined #openstack-infra | 11:06 | |
| *** amotoki has quit IRC | 11:06 | |
| *** mptacekx has quit IRC | 11:08 | |
| *** asselin_ has quit IRC | 11:08 | |
| *** mptacekx has joined #openstack-infra | 11:08 | |
| *** stevemar_ has joined #openstack-infra | 11:09 | |
| *** Julien-zte has quit IRC | 11:10 | |
| *** Julien-zte has joined #openstack-infra | 11:11 | |
| *** gmann has quit IRC | 11:13 | |
| *** stevemar_ has quit IRC | 11:13 | |
| jroll | hi folks, does anyone mind looking at the infra side of this chain to allow ironic CI to test partition-style images with our agent driver? pretty straightforward and moves some logic out of d-s-g: https://review.openstack.org/#/q/topic:ipa-partition-images+status:open | 11:14 |
| jroll | thanks :) | 11:14 |
| *** baoli has joined #openstack-infra | 11:17 | |
| *** jkilpatr has joined #openstack-infra | 11:18 | |
| *** lucasagomes is now known as lucas-hungry | 11:20 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Try again to install modules from AFS PyPi mirror https://review.openstack.org/366687 | 11:26 |
| dulek | Is there a way to check out a non-voting job using openstack-health? I wanted to find recent failing ones so I can point this out in the patches before they get merged to RC. | 11:26 |
| *** gongysh has quit IRC | 11:26 | |
| *** hashar has quit IRC | 11:28 | |
| *** ddieterly has quit IRC | 11:28 | |
| *** hashar has joined #openstack-infra | 11:28 | |
| *** dkehn__ has quit IRC | 11:29 | |
| *** dkehn has quit IRC | 11:29 | |
| *** rtheis_ has joined #openstack-infra | 11:29 | |
| *** dkehn_ has quit IRC | 11:29 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:30 | |
| *** rtheis has quit IRC | 11:30 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Rework the task list layout https://review.openstack.org/357306 | 11:32 |
| *** sambetts|afk is now known as sambetts | 11:32 | |
| *** Hal has joined #openstack-infra | 11:32 | |
| *** hashar has quit IRC | 11:33 | |
| *** dkehn has joined #openstack-infra | 11:36 | |
| *** amotoki has joined #openstack-infra | 11:37 | |
| *** Hal has quit IRC | 11:37 | |
| *** _nadya_ has quit IRC | 11:38 | |
| *** _nadya_ has joined #openstack-infra | 11:39 | |
| *** dkehn__ has joined #openstack-infra | 11:42 | |
| *** dkehn_ has joined #openstack-infra | 11:42 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Rework the task list layout https://review.openstack.org/357306 | 11:48 |
| *** tonytan4ever has joined #openstack-infra | 11:49 | |
| *** amotoki has quit IRC | 11:50 | |
| *** flepied has quit IRC | 11:51 | |
| *** ddieterly has joined #openstack-infra | 11:51 | |
| *** flepied has joined #openstack-infra | 11:51 | |
| *** flepied has quit IRC | 11:51 | |
| *** flepied has joined #openstack-infra | 11:51 | |
| vponomaryov | dulek: you can try this -> http://ci-watch.tintri.com/project?project=cinder&time=7+days | 11:52 |
| *** salv-orl_ has quit IRC | 11:52 | |
| *** salv-orlando has joined #openstack-infra | 11:52 | |
| *** hashar has joined #openstack-infra | 11:52 | |
| *** shardy is now known as shardy_lunch | 11:53 | |
| *** tonytan4ever has quit IRC | 11:53 | |
| *** baoli_ has joined #openstack-infra | 11:54 | |
| *** zhurong has joined #openstack-infra | 11:55 | |
| dulek | vponomaryov: Awesome, thanks! | 11:57 |
| *** baoli has quit IRC | 11:57 | |
| *** amotoki has joined #openstack-infra | 11:58 | |
| *** markvoelker has joined #openstack-infra | 11:58 | |
| *** Na3iL has quit IRC | 11:59 | |
| *** zeus has quit IRC | 12:00 | |
| *** coolsvap_ is now known as coolsvap | 12:01 | |
| *** trown|outtypewww is now known as trown | 12:02 | |
| *** markvoelker has quit IRC | 12:03 | |
| rcarrillocruz | pabelanger: i've put the controller of the infracloud on emergency hosts | 12:04 |
| rcarrillocruz | bumped dhcp agent log to debug | 12:04 |
| rcarrillocruz | and i'm tailing logs for next ERROR | 12:04 |
| coolsvap | jaosorior: we need a mail on list regarding all FFEs | 12:04 |
| *** links has quit IRC | 12:04 | |
| rcarrillocruz | have not been lucky for the past 3h tho | 12:04 |
| *** asselin_ has joined #openstack-infra | 12:04 | |
| pabelanger | rcarrillocruz: okay | 12:04 |
| rcarrillocruz | just in case you see it on that file, nothing wrong, just to avoid ansible-puppet to revert the debug logging | 12:05 |
| pabelanger | rcarrillocruz: did we have a launch failure today? | 12:05 |
| rcarrillocruz | last one from today: 2016-09-07 10:31:13.535 16724 ERROR | 12:05 |
| *** chlong has joined #openstack-infra | 12:05 | |
| *** acabot has quit IRC | 12:05 | |
| AJaeger | odyssey4me: best discuss with clarkb | 12:06 |
| AJaeger | odyssey4me: sorry, wrong bug - I can take care of it... | 12:06 |
| *** devananda has joined #openstack-infra | 12:06 | |
| *** devananda is now known as devananda|travel | 12:06 | |
| *** amotoki has quit IRC | 12:07 | |
| *** acabot has joined #openstack-infra | 12:07 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Revert "Revert "Disable rax-iad due to launch failure rate""" https://review.openstack.org/366708 | 12:07 |
| pabelanger | we should be ready to start using rax-iad again, now that images have been updated | 12:08 |
| *** stevemar_ has joined #openstack-infra | 12:10 | |
| AJaeger | odyssey4me: fixed, all old content deleted | 12:11 |
| odyssey4me | AJaeger ? you mean for https://bugs.launchpad.net/openstack-ansible/+bug/1589842 ? | 12:11 |
| openstack | Launchpad bug 1589842 in openstack-ansible "Published web content still contains old files which were deleted" [Low,Fix released] - Assigned to Andreas Jaeger (jaegerandi) | 12:11 |
| AJaeger | odyssey4me: yes | 12:11 |
| AJaeger | odyssey4me: yes, that bug is "fix released" now ;) | 12:11 |
| odyssey4me | oh great - is that something that needs doing once in a while? isn't there some way to automate it so that it happens on ever publish? | 12:12 |
| AJaeger | odyssey4me: we will swithc docs publishing and as part of that it should be done directly. | 12:12 |
| AJaeger | odyssey4me: for now, this is a manual step ;( | 12:12 |
| odyssey4me | AJaeger ok great, good to know - any idea when the timeline is for the switch to a better mechanism? | 12:13 |
| AJaeger | sdague: could you comment on https://review.openstack.org/366091 ? That'S a new nova-tempest repo which strikes me as odd | 12:13 |
| asettle | AJaeger: yes win! | 12:13 |
| AJaeger | odyssey4me: as soon as we manage ;) | 12:13 |
| AJaeger | odyssey4me: sorry, can't be more concrete but this is a priority now for infra | 12:14 |
| zigo | AJaeger: What does this mean? http://logs.openstack.org/52/366352/1/check/gate-project-config-gerrit/84cc403/console.html#_2016-09-07_10_12_16_374882 | 12:14 |
| odyssey4me | AJaeger well, thank you very much :) good to know that this is known, a fix is in the works, and that we have a workaround for now | 12:14 |
| *** stevemar_ has quit IRC | 12:14 | |
| *** askb has quit IRC | 12:15 | |
| *** links has joined #openstack-infra | 12:16 | |
| *** pradk has joined #openstack-infra | 12:18 | |
| AJaeger | odyssey4me: yeah, a non-scaling workaround ;) | 12:19 |
| AJaeger | zigo, an overzealous spell checker ;( | 12:20 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 12:20 |
| zigo | AJaeger: Is there something I can do to fix? | 12:20 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Gnocchi functional jobs to Xenial https://review.openstack.org/366573 | 12:20 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Telemetry jobs to Xenial https://review.openstack.org/366574 | 12:20 |
| AJaeger | Fix your text or the script, see it's invocation earlier in the log file | 12:21 |
| *** mhickey__ has joined #openstack-infra | 12:21 | |
| AJaeger | It complaints about using openstack instead of OpenStack in the description | 12:21 |
| zigo | AJaeger: That's the description of the repo that is problematic, right? | 12:21 |
| AJaeger | yes | 12:21 |
| *** Na3iL has joined #openstack-infra | 12:22 | |
| *** tongli has joined #openstack-infra | 12:22 | |
| zigo | Got you, thanks. | 12:22 |
| *** ddieterly has quit IRC | 12:22 | |
| *** lucas-hungry is now known as lucasagomes | 12:23 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 3/5 https://review.openstack.org/366352 | 12:23 |
| *** gongysh has joined #openstack-infra | 12:23 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:24 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/system-config: Give gate-infra-puppet-apply-fedora-24 some work https://review.openstack.org/366718 | 12:26 |
| *** dprince has joined #openstack-infra | 12:27 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack-infra/project-config: Enable QoS on neutron for python-openstackclient. https://review.openstack.org/366719 | 12:27 |
| *** rlandy has joined #openstack-infra | 12:28 | |
| *** szaher_ has quit IRC | 12:30 | |
| *** szaher has joined #openstack-infra | 12:31 | |
| *** edmondsw has joined #openstack-infra | 12:31 | |
| *** markvoelker has joined #openstack-infra | 12:32 | |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make tags reusable https://review.openstack.org/366188 | 12:33 |
| *** amotoki has joined #openstack-infra | 12:33 | |
| sdague | AJaeger: so those are redhat qa team tests? | 12:34 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Telemetry jobs to Xenial https://review.openstack.org/366574 | 12:35 |
| AJaeger | sdague: I have no idea ;( But I wanted to make you aware and ask for your feedback, the naming sounded really odd to me. | 12:35 |
| *** mmedvede has quit IRC | 12:36 | |
| *** sarob has joined #openstack-infra | 12:36 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Gnocchi functional jobs to Xenial https://review.openstack.org/366573 | 12:38 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Telemetry jobs to Xenial https://review.openstack.org/366574 | 12:38 |
| *** apetrich has joined #openstack-infra | 12:40 | |
| *** sarob has quit IRC | 12:40 | |
| *** yaume has quit IRC | 12:41 | |
| *** ggnel_t has quit IRC | 12:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Give better error description in check_valid_gerrit_projects.py https://review.openstack.org/366727 | 12:41 |
| AJaeger | zigo: I couldn't fix the regex easily but I fixed the error message ^ | 12:41 |
| *** yaume has joined #openstack-infra | 12:41 | |
| openstackgerrit | Maciej Relewicz proposed openstack-infra/project-config: Added new repos for Murano chef cookbook https://review.openstack.org/366729 | 12:42 |
| *** woodster_ has joined #openstack-infra | 12:42 | |
| zigo | Much better indeed. | 12:42 |
| AJaeger | zigo, if you manage to fix the regex that it handles your case as well, it would be fine as well - but that'S a corner case, so I've given up on that for now. | 12:43 |
| *** kzaitsev_mb has quit IRC | 12:44 | |
| zigo | AJaeger: I don't think we should care so much about these repo long description, the repo names themselves are self explanatory anyway. | 12:44 |
| *** rhallisey has joined #openstack-infra | 12:45 | |
| AJaeger | ;) | 12:45 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 4/5 https://review.openstack.org/366734 | 12:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix incorrect flavor for osic-cloud8 https://review.openstack.org/366735 | 12:45 |
| AJaeger | zigo, you can steck your changes after each other instead of making them independent... | 12:45 |
| pabelanger | AJaeger: rcarrillocruz: ^ fixes in correct flavor for osic-cloud8 | 12:45 |
| rcarrillocruz | +2 | 12:46 |
| AJaeger | pabelanger: I have to trust you on those names - it looks consistent, thus +2a | 12:46 |
| zigo | AJaeger: Right... | 12:46 |
| AJaeger | zigo: in case they conflict... | 12:46 |
| zigo | AJaeger: This would have avoided merge conflict. | 12:47 |
| zigo | And there WILL be. | 12:47 |
| *** vikrant has quit IRC | 12:47 | |
| *** pradk has quit IRC | 12:47 | |
| zigo | AJaeger: Doesn't mater, just make sure to approve them in order, and wait for one to be merged before approving the next, I'll manage with merge conflicts. | 12:47 |
| AJaeger | zigo: once they pass ask mordred to +2 - and then I'll +2A and wait 2hours between them to not overwhelm github | 12:47 |
| zigo | Ok. | 12:47 |
| AJaeger | zigo: Exactly | 12:47 |
| AJaeger | could you give them the same topic, please? | 12:48 |
| *** kzaitsev_mb has joined #openstack-infra | 12:48 | |
| zigo | AJaeger: No worries for the wait time, there's no urgency for having it all of them today. I needed deb-migrate, and that one was in 1/5, the others can wait later today or tomorrow. | 12:48 |
| flip214 | I saved me a query in json format, from logstash.openstack.org. but I can't find a way to upload that file again, to restore my query. | 12:48 |
| AJaeger | zigo: ok | 12:49 |
| *** zeus has joined #openstack-infra | 12:49 | |
| zigo | AJaeger: What I need to move on though is this one: https://review.openstack.org/365477 | 12:49 |
| *** jaosorior has quit IRC | 12:49 | |
| *** zeus is now known as Guest49408 | 12:50 | |
| *** jaosorior has joined #openstack-infra | 12:50 | |
| *** amitgandhinz has quit IRC | 12:50 | |
| zigo | AJaeger: As discussed with fungi, it's ok to have signed tag push, I don't really mind if non-signed tag push is not possible. | 12:50 |
| zigo | I'll re-tag with my signature if needed. | 12:51 |
| *** amitgandhinz has joined #openstack-infra | 12:51 | |
| *** Goneri has joined #openstack-infra | 12:51 | |
| AJaeger | zigo, +2A | 12:51 |
| zigo | Though for most OpenStack stuff, tags will be already signed by someone else, so it will be easy to push. | 12:51 |
| zigo | Cheers ! :) | 12:51 |
| *** Guest49408 is now known as zeus` | 12:51 | |
| AJaeger | bbl | 12:52 |
| *** ddieterly has joined #openstack-infra | 12:52 | |
| *** zeus` is now known as zeus | 12:52 | |
| *** zeus has quit IRC | 12:52 | |
| *** zeus has joined #openstack-infra | 12:52 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Gnocchi functional jobs to Xenial https://review.openstack.org/366573 | 12:56 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Telemetry jobs to Xenial https://review.openstack.org/366574 | 12:56 |
| *** ddieterly has quit IRC | 12:57 | |
| *** thorst has joined #openstack-infra | 12:57 | |
| *** rods has joined #openstack-infra | 12:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Allow deb-packaging-core to push tags https://review.openstack.org/365477 | 12:58 |
| *** mdrabe has joined #openstack-infra | 12:58 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:59 | |
| *** claudiub|2 has joined #openstack-infra | 13:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix incorrect flavor for osic-cloud8 https://review.openstack.org/366735 | 13:01 |
| *** mmedvede has joined #openstack-infra | 13:02 | |
| *** claudiub has quit IRC | 13:03 | |
| *** kgiusti has joined #openstack-infra | 13:05 | |
| *** pilgrimstack1 has joined #openstack-infra | 13:06 | |
| *** gongysh has quit IRC | 13:07 | |
| *** chem has quit IRC | 13:07 | |
| *** pilgrimstack has quit IRC | 13:07 | |
| *** chem has joined #openstack-infra | 13:07 | |
| *** shardy_lunch is now known as shardy | 13:08 | |
| *** florianf has quit IRC | 13:10 | |
| *** stevemar_ has joined #openstack-infra | 13:10 | |
| rcarrillocruz | pabelanger: with all the CI resources we have now, i suspect we are going to wait quite a bit for another infracloud ERROR hit :/ | 13:11 |
| *** drawsmcgraw has quit IRC | 13:11 | |
| *** links has quit IRC | 13:12 | |
| *** asettle has quit IRC | 13:12 | |
| *** asettle has joined #openstack-infra | 13:13 | |
| *** xyang1 has joined #openstack-infra | 13:13 | |
| AJaeger | rcarrillocruz: we need to add "priorities" in nodepool so that we can prioritize infracloud over other resources to stress test it ;) | 13:13 |
| rcarrillocruz | heh | 13:13 |
| rcarrillocruz | i think it would be good to have the zuul+nodepool dev env | 13:13 |
| rcarrillocruz | to have a stage-prod promotion kind of workflow for clouds | 13:14 |
| *** stevemar_ has quit IRC | 13:15 | |
| *** drifterza has quit IRC | 13:16 | |
| *** matt-borland has joined #openstack-infra | 13:17 | |
| *** trown is now known as trown|brb | 13:17 | |
| *** rossella_s has quit IRC | 13:18 | |
| openstackgerrit | Maciej Relewicz proposed openstack-infra/project-config: Added new repos for Murano chef cookbook https://review.openstack.org/366729 | 13:18 |
| *** rossella_s has joined #openstack-infra | 13:19 | |
| *** oanson has quit IRC | 13:21 | |
| *** pgadiya has quit IRC | 13:23 | |
| *** cardeois has joined #openstack-infra | 13:23 | |
| *** florianf has joined #openstack-infra | 13:24 | |
| pabelanger | AJaeger: rcarrillocruz: maybe a new staging pipeline, start new clouds there while work out the issues. | 13:24 |
| pabelanger | _shrugs_ | 13:24 |
| pabelanger | I'm on the fence about a new nodepool instance for staging, only because the amount of overhead needed to bring a cloud online. EG: image uploads for example | 13:25 |
| fungi | zigo: i'm curious to hear how your tag pushing goes... at least in most cases gerrit requires that the account authenticating to its git service have a validated e-mail address matching the address of the committer (i haven't tested whether this also extends to pushing tags created by others) | 13:26 |
| *** tonytan4ever has joined #openstack-infra | 13:26 | |
| zigo | fungi: Indeed, it doesn't seem to work to push someone else's tag. | 13:26 |
| pabelanger | rcarrillocruz: do you mind reviewing https://review.openstack.org/#/c/366708/ | 13:27 |
| zigo | fungi: But that should be ok though, I can "git fetch" from upstream before build, for those tags I don't need to re-tag. | 13:27 |
| *** snarwade has quit IRC | 13:27 | |
| *** trown|brb is now known as trown | 13:28 | |
| zigo | fungi: Is there a way to automate "git pull" in the master branch of all deb-* stuff | 13:28 |
| zigo | ? | 13:28 |
| *** jamesdenton has joined #openstack-infra | 13:28 | |
| rcarrillocruz | sure | 13:28 |
| *** jcoufal has joined #openstack-infra | 13:28 | |
| rcarrillocruz | approved | 13:29 |
| rcarrillocruz | pabelanger: https://review.openstack.org/#/c/366636/ , the default rules secgroup thing that was discussed yesterday | 13:29 |
| zigo | fungi: Otherwise, I'll just do this: https://review.openstack.org/365367 | 13:29 |
| rcarrillocruz | i've run it on the infracloud, it removes them | 13:29 |
| *** caowei has joined #openstack-infra | 13:29 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 13:29 |
| pabelanger | rcarrillocruz: great | 13:30 |
| zigo | paste.openstack.org seems down | 13:30 |
| fungi | zigo: yeah, that's what i was suggesting with the upstream tracking option we have, for example we use it to maintain our fork of the gerrit source code http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/projects.yaml#n110 | 13:31 |
| zigo | ah, no paste.o.o works, it's just super slow. | 13:31 |
| *** isaacb has quit IRC | 13:32 | |
| *** dtardivel has joined #openstack-infra | 13:32 | |
| mordred | morning pabelanger | 13:32 |
| *** weshay is now known as weshay_mtg | 13:32 | |
| mordred | pabelanger: did we ever get the ipv6 addresses fixed on rax nodes? | 13:32 |
| mordred | pabelanger: (I remember we were looking in to that last week or the week before, but then I got distracted) | 13:33 |
| fungi | zigo: in that example, we pull copies of all tags from https://gerrit.googlesource.com/gerrit into the local openstack-infra/gerrit repo, and also update copies of all upstream branches but prepend "upstream/" to their branch names | 13:33 |
| pabelanger | mordred: no, we are still ipv4. But cannot remember why, I can look again this morning | 13:33 |
| pabelanger | was eth1 missing? | 13:33 |
| mordred | pabelanger: yah - I think it's missing eth1 | 13:34 |
| mordred | OR | 13:34 |
| mordred | actually, I think it's that we're writing out static network config for the interface | 13:34 |
| mordred | but nothing that says "oh, hey, please get an IPv6 address too" | 13:34 |
| pabelanger | ya | 13:34 |
| fungi | mordred: talking about the job nodes in rackspace? remember there is no ipv6 address data in the configdrive (but maybe adding inet6 auto would pick up assignments?) | 13:34 |
| zigo | fungi: http://paste.openstack.org/show/567466/ <--- Here's an example | 13:34 |
| *** hamzy has quit IRC | 13:34 | |
| mordred | fungi: exactly | 13:34 |
| mordred | it's inet6 auto we're missing | 13:35 |
| pabelanger | okay, I can test that quickly | 13:35 |
| zigo | fungi: If I just do "get fetch upstream" before build, will it automatically use a cached version of the git repo? | 13:35 |
| fungi | mordred: just noticing that on our control plane servers, we have static v6 configuration so... no guarantees auto works with rax's network implementation | 13:36 |
| mordred | fungi: oh. weird. | 13:36 |
| mordred | fungi: is that true of new control plane servers? (I wonder if there was a behavior change?) | 13:37 |
| mordred | fungi: oh - also, it's nova-agent still on our control plane servers, not glean+config-drive | 13:37 |
| mordred | so maybe they have not exposed the ipv6 info in config-drive yet | 13:37 |
| mordred | and our control plane servers get it from xenstore | 13:37 |
| fungi | mordred: i assume that's so that they can directly map v6 addrewses in nova. likely they don't have a good wau of integrating that with auto assignments since those will either be chosen by the guest instance as an eui-64 address or a privacy address | 13:37 |
| *** kzaitsev_mb has quit IRC | 13:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Revert "Revert "Disable rax-iad due to launch failure rate""" https://review.openstack.org/366708 | 13:38 |
| fungi | and as they don't do dhcp for v5 they're unlikely to have set up dhcp6 to dole out reservations mapped to nova/neutron either | 13:39 |
| fungi | s/v5/v4/ | 13:39 |
| fungi | which leaves only static addressing | 13:39 |
| fungi | though our static assignment there _does_ look like it's a proper eui-64 based on the vnic's mac, so auto might just work fine as long as we're getting router advertisements | 13:40 |
| *** mriedem has joined #openstack-infra | 13:40 | |
| mordred | yah - I have a hunch RAs work for some reason ... | 13:41 |
| mordred | fungi: now, follow up question... | 13:41 |
| fungi | i mean, they have the mac assigned for each vnic so in theory they could just recreate the eui-64 mapping in their backend database | 13:41 |
| mordred | if RAs work there, adding inet6 auto would take care of it | 13:41 |
| fungi | yup | 13:41 |
| fungi | pretty sire | 13:41 |
| fungi | sure | 13:41 |
| mordred | but is that a safe thing to just add to all static configs that are missing ipv6? | 13:41 |
| mordred | like, inet6 auto doesn't hang waiting for an RA iirc | 13:42 |
| *** annegentle has joined #openstack-infra | 13:42 | |
| *** krtaylor has joined #openstack-infra | 13:42 | |
| fungi | zigo: just plain "git fetch" won't magically use our local git caches. you either need the job to use zuul-cloner (installed on all our workers as /usr/zuul-env/bin/zuul-cloner) or manually clone from /opt/git/openstack/nova and then edit your origin remote to be git.openstack.org or something along those lines | 13:42 |
| openstackgerrit | Merged openstack-infra/system-config: Remove default security group rules on Infra clouds https://review.openstack.org/366636 | 13:43 |
| fungi | mordred: router advertisements usually come with great frequency, so it shouldn't wait long at all for the v6 address to become reachable after the interface is brought up under normal circumstances. since it's just broadcast from the routers there's no back-and-forth like you have with dhcp6 | 13:44 |
| mordred | fungi: no - I mean - if we have glean just always add "auto inet6" to network configs if it doesn't find static ipv6 | 13:45 |
| fungi | mordred: also, looking back over our control plane server network configs, we must be getting ra because we don't have static routing configuration in there | 13:46 |
| mordred | will that break clouds that have no ipv6 | 13:46 |
| mordred | ? | 13:46 |
| *** eharney has joined #openstack-infra | 13:46 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:46 | |
| fungi | mordred: inet6 auto should be perfectly safe even when there's no v6 infrastructure, afaik, but that's probably relatively trivial to test too? | 13:46 |
| mordred | fungi: yah, I think you're right | 13:47 |
| fungi | basically, all interfaces already have (working!) linklocal addresses. inet6 auto simply causes the kernel to act on router advertisements and tack on additional global and site-local (sometimes) addresses as well as routes | 13:48 |
| fungi | the only times where it would be problematic is if there are router advertisements broadcast in the lan but handing out global network prefixes and routes which are not actually working | 13:49 |
| *** ddieterly has joined #openstack-infra | 13:49 | |
| mordred | fungi: I believe I would physically kill an operator who did that to us dead | 13:50 |
| fungi | then the server might try to reach stuff over v6 and fail, perhaps delaying before falling back to v4 or perhaps failing outright and retrying indefinitely depending on the application/protocol | 13:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use interface_ip instead of internal logic https://review.openstack.org/360446 | 13:50 |
| *** ddieterly has quit IRC | 13:51 | |
| *** ddieterly has joined #openstack-infra | 13:51 | |
| fungi | worth noting, though, inet6 auto may not always be entirely sufficient, for example if the provider expects you to use dhcp6 to get a tracked reservation from their ipam or something | 13:52 |
| AJaeger | zigo, going one step back: With Debian Packaging you currently build a complete Packaging Build Infrastructure in OpenStack CI. Doesn't Debian have one? I still don't get why it's needed - and looking at the problems you run into, I wonder whether it's really worth continuing down the current approach or whether there might be different approaches. The RDO and SUSE folks approach this differently for | 13:52 |
| AJaeger | example. | 13:52 |
| *** annegent_ has joined #openstack-infra | 13:53 | |
| fungi | AJaeger: up-side here is that we provide a code review system with an integrated ci, while most distros (debian included) only have autobuilders and maybe some simple autopkgtest framework | 13:53 |
| *** pradk has joined #openstack-infra | 13:53 | |
| dhellmann | mordred: seeing job failures for the release-tools repo related to that ffi issue we discussed yesterday. Is that a wide-spread thing, or do I need to do something special for that repo? | 13:53 |
| fungi | so integration-testing proposed changes to packages would be hard if not impossible to make work there | 13:54 |
| dhellmann | mordred : https://review.openstack.org/#/c/364644/ when you're free | 13:54 |
| fungi | aiui, the goal here is for debian package maintainers to push a proposed change to, say, pkg-nova which will not only make sure tat it can still build teh package with that, but then stand up an openstack deployment integrating that test build and run maybe tempest smoke tests against it (and with depends-on you could be testing a series of changes for nova, neutron and cinder, maybe, before | 13:55 |
| fungi | approving all of them) | 13:55 |
| AJaeger | fungi, The RPM packaging project uses this CI and then imports in their build systems. And at least the SUSE build system allows review and has testing included - on a packaging level, so unit test - but also whole system testing. | 13:55 |
| AJaeger | fungi, I know too little about how Debian does it - just seeing all the problems that zigo runs in, I wonder whether there are alternatives | 13:57 |
| *** rcernin has joined #openstack-infra | 13:57 | |
| *** annegentle has quit IRC | 13:57 | |
| fungi | AJaeger: got it. what i know of debian's package build automation is that it only has the capability of doing things like unit testing (you could sort of try to hack up some integration tests with it, but sequencing dependent changes to packages would get tough) | 13:57 |
| fungi | AJaeger: i think most of the problems zigo is facing stem from his expected workflow not easily mapping to our toolchain, which i expect is ultimately going to require adjustments to that workflow | 13:58 |
| AJaeger | fungi, this works with SUSE's system. But if Debian does not support some of these, I understand zigo's challenge. You really want integration testing - and only if it passes, publish packages... | 13:58 |
| fungi | right. automated testing in debian is per-package uploaded, so if your nova change depends on a neutron change, you have to keep adjusting and reuploading the neutron change until the package successfully "builds" before your nova uploads would be able to make use of that | 14:00 |
| AJaeger | fungi, So, you say: GEneral approach looks fine - but zigo needs to adapot his workflow so that it's fine with gerrit. | 14:00 |
| *** abregman_ is now known as abregman | 14:00 | |
| AJaeger | fungi, yeah, such single package testing is indeed not sufficient. | 14:01 |
| *** drawsmcgraw has joined #openstack-infra | 14:01 | |
| fungi | AJaeger: right, the track-upstream option i mentioned is one such tool we provide for handling repos which are forks of other (usually external) repos and need to receive updates from them | 14:01 |
| fungi | and as we've discovered, teh pkg-.* repos are actually forks of the repos they're packaging | 14:01 |
| AJaeger | yeah ;( | 14:01 |
| mordred | dhellmann: it's related to libffi-dev being missing from bindep.txt | 14:02 |
| AJaeger | mordred: do we need to push libffi-dev into all bindep.txt files? | 14:02 |
| mordred | dhellmann: we were ok before because there was a wheel we were using that meant we didn't need the dev headers | 14:02 |
| openstackgerrit | Alexey Ovchinnikov proposed openstack-infra/project-config: [Manila] Fix for LVM job template https://review.openstack.org/366771 | 14:02 |
| mordred | AJaeger: I think only the ones who hit the cffi build issue | 14:02 |
| *** zhurong has quit IRC | 14:02 | |
| fungi | AJaeger: pabelanger is working on altering the gate-{name}-pep8 jobs to not use wheels so we catch this early when bindep.txt files are proposed/updated | 14:03 |
| AJaeger | fungi, is there anything I should take care of while reviewing zigo's changes? Right now, I can only check for "looks sane" | 14:03 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 14:03 |
| zigo | AJaeger: fungi: I've discussed with Debian people about using Gerrit within the Debian infrastructure, and they've told me that I should first show a proof of concept. Here we are, we have it within OpenStack infra. Though that's not the point of what I've been doing. | 14:03 |
| AJaeger | fungi, pabelanger : Great | 14:03 |
| AJaeger | mordred: ok. | 14:03 |
| pabelanger | indeed! I am just trying to figure out how to pass --no-binary to tox | 14:03 |
| pabelanger | or if we need to update tox.ini | 14:03 |
| zigo | AJaeger: fungi: One of the very interesting use case is puppet-openstack, which will be able to re-use packages and gate on it. | 14:03 |
| *** rbrndt has joined #openstack-infra | 14:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/release-tools: Add libffi-dev to bindep.txt https://review.openstack.org/366774 | 14:04 |
| AJaeger | zigo, yeah... | 14:04 |
| zigo | AJaeger: fungi: And opposite way as well: packages will be able to use puppet (possibly in a multi-node scenario) to gate on a deployment. | 14:04 |
| mordred | dhellmann, AJaeger https://review.openstack.org/366774 | 14:04 |
| zigo | AJaeger: fungi: I am also foreseeing automation, like packaging proposal bots whenever a new upstream release of Oslo happens. | 14:05 |
| zigo | One thing at a time though, now I believe we have everything working, finally ... | 14:05 |
| AJaeger | mordred: which package pulls in cffi? | 14:05 |
| zigo | :) | 14:05 |
| *** zul_ has joined #openstack-infra | 14:05 | |
| AJaeger | zigo ;) | 14:05 |
| zigo | AJaeger: I confirm that OpenStack needs cffi. | 14:06 |
| AJaeger | fungi, zigo Thanks for explanataion | 14:06 |
| zigo | I can't remember which package though, but I can find out. | 14:06 |
| AJaeger | zigo: Which package exactly? | 14:06 |
| AJaeger | ;) | 14:06 |
| zigo | Ok, hang on... | 14:06 |
| *** rodrigods has quit IRC | 14:06 | |
| *** rodrigods has joined #openstack-infra | 14:06 | |
| zigo | Reverse Depends: python-barbican (1:2.0.0-5) | 14:07 |
| zigo | Reverse Depends: python-nacl (1.0.1-2) | 14:07 |
| zigo | Reverse Depends: python-oslo.privsep (1.3.0-2) | 14:07 |
| zigo | Reverse Depends: python-pygit2 (>= 0.24.1-1) | 14:07 |
| zigo | At least these ... | 14:07 |
| mordred | I know I had to add it for shade ... one sec | 14:08 |
| zigo | Then there's indirect dependencies. | 14:08 |
| *** rbrndt_ has joined #openstack-infra | 14:08 | |
| zigo | Like: zaqar -> python-autobahn -> python-nacl | 14:08 |
| *** rbrndt has quit IRC | 14:09 | |
| zynzel | fungi: online? :) | 14:09 |
| mordred | Collecting cffi>=1.4.1 (from cryptography!=1.3.0,>=1.0->python-magnumclient>=2.1.0->shade) | 14:09 |
| mordred | AJaeger: I think crypography might be a common way it's getting in | 14:10 |
| *** andymccr_ has joined #openstack-infra | 14:10 | |
| *** hongbin has joined #openstack-infra | 14:10 | |
| *** adrian_otto has joined #openstack-infra | 14:11 | |
| AJaeger | ah - and some repos include that one already in bindep ;) | 14:11 |
| * AJaeger patches swift now | 14:12 | |
| *** mptacekx has quit IRC | 14:12 | |
| *** ajo_ has joined #openstack-infra | 14:12 | |
| AJaeger | ah, swift and keystone have it already - so they were not hit ;) | 14:13 |
| openstackgerrit | Dmitry Ukov proposed openstack-infra/project-config: Create fuel-nailgun-extension-day2 project https://review.openstack.org/366779 | 14:14 |
| *** xarses has quit IRC | 14:14 | |
| *** dkehn__ has quit IRC | 14:15 | |
| *** SamYaple_ has joined #openstack-infra | 14:15 | |
| openstackgerrit | Dmitry Ukov proposed openstack-infra/project-config: Create fuel-nailgun-extension-day2 project https://review.openstack.org/366779 | 14:15 |
| *** aviau_ has joined #openstack-infra | 14:15 | |
| mtreinish | infra-root: I'm getting 503 responses for manual elasticrecheck queries I'm running locally (the health.o.o/status page is also returning a 500 but that might just be because of the error from elastic search) was that brought down? | 14:17 |
| *** dmsimard is now known as dmsimard|afk | 14:19 | |
| *** adrian_otto has quit IRC | 14:20 | |
| *** zul_ has quit IRC | 14:20 | |
| *** pradk has quit IRC | 14:20 | |
| *** caowei has quit IRC | 14:20 | |
| *** rossella_s has quit IRC | 14:20 | |
| *** asettle has quit IRC | 14:20 | |
| *** salv-orlando has quit IRC | 14:20 | |
| *** aviau has quit IRC | 14:20 | |
| *** andymccr has quit IRC | 14:20 | |
| *** SamYaple has quit IRC | 14:20 | |
| *** cinerama has quit IRC | 14:20 | |
| *** jgrassler has quit IRC | 14:20 | |
| *** gbraad has quit IRC | 14:20 | |
| *** dulek has quit IRC | 14:20 | |
| *** degorenko has quit IRC | 14:20 | |
| *** lennyb has quit IRC | 14:20 | |
| *** spiffxp has quit IRC | 14:20 | |
| *** wfoster has quit IRC | 14:20 | |
| *** darrenc has quit IRC | 14:20 | |
| *** SotK has quit IRC | 14:20 | |
| *** anteaya has quit IRC | 14:20 | |
| *** sambetts has quit IRC | 14:20 | |
| *** stevebaker has quit IRC | 14:20 | |
| *** eliqiao has quit IRC | 14:20 | |
| mordred | AJaeger: yah - I thnk it's a good thing to fix - it's an actual bug in the bindep files | 14:20 |
| mordred | mtreinish: looking | 14:20 |
| mordred | mtreinish: logstash.o.o certainly doesn't seem happy | 14:21 |
| *** jascott1 has joined #openstack-infra | 14:21 | |
| *** nmagnezi has quit IRC | 14:21 | |
| AJaeger | sdague: https://review.openstack.org/#/c/360994/ should fix gerrit-dash-creator bindep file for cffi. See https://review.openstack.org/366764 for failure | 14:22 |
| sdague | AJaeger: that first link is tripleo? | 14:23 |
| mtreinish | mordred: :/ need to restart it or something? | 14:23 |
| AJaeger | sdague: argh ;( | 14:24 |
| AJaeger | sdague: https://review.openstack.org/366783 should be better | 14:24 |
| sdague | AJaeger: any idea why we need cffi? | 14:24 |
| AJaeger | sdague: http://logs.openstack.org/64/366764/1/check/gate-gerrit-dash-creator-pep8-ubuntu-xenial/ce2b9cc/console.html#_2016-09-07_13_54_21_455570 | 14:25 |
| *** pradk has joined #openstack-infra | 14:25 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 14:26 |
| AJaeger | sdague: I suspect launchpadlib | 14:26 |
| sdague | ok, I'm just going to delete that instead | 14:26 |
| clarkb | mordred: on elasticsearch02 you can check /var/log/elasticsearch.log it may complain there about why it is unhappy | 14:27 |
| *** yamahata has joined #openstack-infra | 14:27 | |
| rcarrillocruz | pabelanger: human-readable output for the launcher https://review.openstack.org/#/c/366723/3 when you get a chance pls | 14:28 |
| *** andymccr_ is now known as andymccr | 14:28 | |
| rcarrillocruz | excerpt http://logs.openstack.org/23/366723/3/check/gate-ansible-role-cloud-launcher-dsvm-ansible-func-ubuntu-trusty/85278b8/console.html | 14:28 |
| AJaeger | sdague: abandoned my change then | 14:28 |
| *** caowei has joined #openstack-infra | 14:28 | |
| zigo | mordred: https://review.openstack.org/366337 <--- Add missing deb repo 2/5 | 14:29 |
| mordred | clarkb: I see no unhappies | 14:29 |
| *** lucasagomes is now known as lucas-afk | 14:29 | |
| clarkb | mordred: nothing about garbage collecting or running out of heap memory? | 14:30 |
| *** salv-orlando has joined #openstack-infra | 14:30 | |
| mordred | clarkb: nope | 14:30 |
| mordred | clarkb: it's 11 lines long | 14:30 |
| *** mdrabe has quit IRC | 14:30 | |
| cloudnull | good mornings | 14:32 |
| *** weshay_mtg is now known as weshay | 14:33 | |
| *** adrian_otto has joined #openstack-infra | 14:33 | |
| pabelanger | clarkb: happen to have your shade script handy that you used to upload new-glean.vhd to rackspace? I wanted to upload the image to rax-iad to see what is going on there | 14:33 |
| *** annegent_ has quit IRC | 14:33 | |
| pabelanger | clarkb: for some reason, we have zero networking still | 14:33 |
| clarkb | pabelanger: I just put what mordred had in irc into an interactive python interpeter | 14:34 |
| pabelanger | k | 14:35 |
| pabelanger | let me find the syntax | 14:35 |
| mordred | pabelanger: I can make you a new one | 14:35 |
| *** reed_ has joined #openstack-infra | 14:35 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 4/5 https://review.openstack.org/366734 | 14:35 |
| mordred | pabelanger: import shade ; shade.simple_logging(debug=True) ; cloud=shade.openstack_cloud(cloud='rax', region_name='IAD') ; cloud.create_image(name='new-glean', filename='new-glean.vhd', wait=True) | 14:36 |
| mordred | pabelanger: I believe should do it | 14:36 |
| mordred | pabelanger: (I just typed that, so it may have typos) | 14:36 |
| *** adrian_otto has quit IRC | 14:36 | |
| pabelanger | mordred: thanks | 14:36 |
| mordred | pabelanger: ok. syntax should be good - cloud name may be off | 14:36 |
| pabelanger | mordred: looks good, thanks again | 14:37 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002 https://review.openstack.org/365026 | 14:39 |
| zigo | fungi: I specifically choose non-upstream-conflicting branch names, so in my case, I can simply declare options: - track-upstream, without adding a upstream-prefix: option, right? | 14:39 |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002 https://review.openstack.org/365026 | 14:40 |
| *** rfolco has joined #openstack-infra | 14:40 | |
| zigo | fungi: So then, master and stable/* branches will be pulled? | 14:40 |
| *** salv-orlando has quit IRC | 14:40 | |
| *** xarses has joined #openstack-infra | 14:41 | |
| *** amitgandhinz has quit IRC | 14:41 | |
| *** zul_ has joined #openstack-infra | 14:42 | |
| *** rossella_s has joined #openstack-infra | 14:42 | |
| *** asettle has joined #openstack-infra | 14:42 | |
| *** cinerama has joined #openstack-infra | 14:42 | |
| *** gbraad has joined #openstack-infra | 14:42 | |
| *** dulek has joined #openstack-infra | 14:42 | |
| *** degorenko has joined #openstack-infra | 14:42 | |
| *** lennyb has joined #openstack-infra | 14:42 | |
| *** spiffxp has joined #openstack-infra | 14:42 | |
| *** wfoster has joined #openstack-infra | 14:42 | |
| *** darrenc has joined #openstack-infra | 14:42 | |
| *** SotK has joined #openstack-infra | 14:42 | |
| *** anteaya has joined #openstack-infra | 14:42 | |
| *** sambetts has joined #openstack-infra | 14:42 | |
| *** stevebaker has joined #openstack-infra | 14:42 | |
| *** eliqiao has joined #openstack-infra | 14:42 | |
| *** amitgandhinz has joined #openstack-infra | 14:42 | |
| *** zz_dimtruck is now known as dimtruck | 14:42 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Make builder recognize manual build requests https://review.openstack.org/365112 | 14:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Revert "Revert "Revert "Disable rax-iad due to launch failure rate"""" https://review.openstack.org/366792 | 14:43 |
| *** mwhahaha has quit IRC | 14:44 | |
| *** jgrassler has joined #openstack-infra | 14:44 | |
| *** Swami has joined #openstack-infra | 14:45 | |
| *** mdrabe has joined #openstack-infra | 14:46 | |
| *** hamzy has joined #openstack-infra | 14:46 | |
| *** Swami_ has joined #openstack-infra | 14:46 | |
| *** mwhahaha has joined #openstack-infra | 14:47 | |
| *** senk has quit IRC | 14:49 | |
| *** adrian_otto has joined #openstack-infra | 14:49 | |
| mtreinish | mordred, clarkb: did we figure out what was up with ES? | 14:50 |
| *** Swami has quit IRC | 14:50 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Gnocchi functional jobs to Xenial https://review.openstack.org/366573 | 14:50 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Switch Telemetry jobs to Xenial https://review.openstack.org/366574 | 14:50 |
| clarkb | mtreinish: no gonna have to look more | 14:50 |
| mtreinish | ok | 14:50 |
| mtreinish | if there's anything i can do to help let me know | 14:51 |
| mtreinish | clarkb: the only thing that looked suspect to me was the disk usage on logstash.o.o: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=422&rra_id=all | 14:51 |
| mtreinish | but it's been like that for a while now | 14:52 |
| mtreinish | so I doubt that's it | 14:52 |
| *** vinaypotluri has joined #openstack-infra | 14:52 | |
| clarkb | and its just a prpxy there so.shouldnt really care | 14:52 |
| clarkb | its likely something broken with es itself | 14:52 |
| *** adrian_otto1 has joined #openstack-infra | 14:52 | |
| *** _nadya_ has quit IRC | 14:53 | |
| *** diltram has joined #openstack-infra | 14:55 | |
| diltram | morning | 14:55 |
| diltram | I have a question | 14:55 |
| diltram | about kvm on gates | 14:55 |
| *** adrian_otto has quit IRC | 14:55 | |
| *** amotoki has quit IRC | 14:56 | |
| diltram | by default devstack should use kvm when available and fallback to qemu | 14:56 |
| fungi | zigo: i believe so, yes | 14:56 |
| diltram | but we know that when we're running tests it by default is using qemu | 14:56 |
| clarkb | dlahn: yes because not all of our test clouds have nested virt and in those that do we have found it to be unreliable amd results in crashed VMs that we cant debug | 14:57 |
| diltram | https://review.openstack.org/#/c/366472/1 - here is our test patch set where we manually enabling kvm | 14:57 |
| zigo | fungi: Ok, thanks. I've added a ./debian/rules fetch-upstream-remote (which fetches from github) as a temporary workaround, I'll do that as a better fix. | 14:57 |
| clarkb | you cannot and should not rely on nested virt | 14:57 |
| diltram | clarkb: why? | 14:57 |
| clarkb | er dlahn that was for diltram sorry | 14:57 |
| *** spzala has joined #openstack-infra | 14:58 | |
| diltram | clarkb: but devstack has fallback mechanizm | 14:58 |
| clarkb | diltram: because not all of our clouds support and those that do we have found it to be unreliable resulting in crashed test VMs that are unreachable and cannot be debugged | 14:58 |
| diltram | if there is no kvm then fallback to qemu | 14:58 |
| clarkb | also it didnt represent aignificantly faster test runs (likely because we moatly use cirros) | 14:59 |
| clarkb | but thats why its unreliable and not availalbe in many placea | 14:59 |
| diltram | no, take a look into patchset which I provided | 14:59 |
| openstackgerrit | Simon McCartney proposed openstack-infra/puppet-jenkins: Enable a specific version of jenkins to be installed https://review.openstack.org/366803 | 14:59 |
| *** gouthamr has joined #openstack-infra | 14:59 | |
| diltram | all service vm projects are gaining great performance boost by access to kvm | 15:00 |
| diltram | enabling kvm in this patch resulted with reduction testing time from 1:25h to 33 minutes | 15:00 |
| clarkb | diltram: they probably dont run cirros? | 15:00 |
| diltram | nooo | 15:00 |
| clarkb | yes thats why then | 15:00 |
| diltram | we're building ubuntu image with load balancer | 15:00 |
| clarkb | with cirros the difference is minimal | 15:00 |
| diltram | but all projects which use service vm are not using cirros | 15:01 |
| diltram | becase we can't use this image | 15:01 |
| clarkb | no but the vast majority of our testing does | 15:01 |
| *** gouthamr_ has joined #openstack-infra | 15:01 | |
| fungi | diltram: there's no control on whether your jobs run in providers that support nested virt either, so you'd just wind up with some runs being fast and some being slow (in addition to outright crashing for undiagnosable reasons) | 15:01 |
| diltram | I know, but I'm wondering why we're not using default fallback mechanizm in devstack | 15:01 |
| clarkb | diltram: I have explained the reason twice now | 15:02 |
| fungi | consistency | 15:02 |
| diltram | ok, I understand right now | 15:02 |
| *** yuval has quit IRC | 15:02 | |
| diltram | but it something weird that using kvm is making cloud unstable | 15:02 |
| *** matrohon has quit IRC | 15:03 | |
| diltram | anyone on the world who is using kvm has that problem | 15:03 |
| *** abregman has quit IRC | 15:03 | |
| *** sdake has joined #openstack-infra | 15:03 | |
| *** abregman has joined #openstack-infra | 15:03 | |
| *** johnsom has joined #openstack-infra | 15:04 | |
| *** gongysh has joined #openstack-infra | 15:04 | |
| *** gouthamr has quit IRC | 15:05 | |
| *** thcipriani|afk is now known as thcipriani | 15:06 | |
| mordred | diltram: nested virt is more unstable | 15:06 |
| diltram | ok, thx fungi, clarkb for all info about that :) | 15:06 |
| mordred | diltram: so even if it is available sometimes, the increase in test failures makes it not worth having on in the general case | 15:07 |
| diltram | I'm using the same cloud which is provided from OSIC and I never had any problems with nested virt using kvm | 15:07 |
| johnsom | Hi, I just missed that conversation, but I expect it was related to my excitement that nested virt is now available. | 15:07 |
| diltram | so I need to take a look into that why it's not working | 15:07 |
| mordred | diltram: right. but openstack infra runs across 7 different clouds | 15:07 |
| johnsom | HUGE improvement. Takes our test runs from over 1:30 to 34 minutes. | 15:07 |
| mordred | nested virt is NOT generally available | 15:08 |
| *** sdake has quit IRC | 15:08 | |
| mordred | you should not count on it | 15:08 |
| diltram | I know, but again devstack supports fallback - so +kvm use nested virt, -kvm use qemu | 15:08 |
| mordred | if you enable it, and it works, it's an accident | 15:08 |
| johnsom | Yeah, how can we support getting it widely available? | 15:08 |
| mordred | we are not going to | 15:08 |
| mordred | it is not possible | 15:08 |
| johnsom | I see that a few clouds have it now. | 15:08 |
| diltram | mordred: if egrep --quiet '(vmx|svm)' /proc/cpuinfo; then | 15:08 |
| diltram | export DEVSTACK_GATE_LIBVIRT_TYPE=kvm | 15:08 |
| diltram | fi | 15:08 |
| diltram | this code is verifying do we have access to kvm | 15:08 |
| mordred | so - as clarkb said - there are TWO issues | 15:09 |
| fungi | all clouds donating resources to us would need to support it, or we would need to stop accepting donations from providers who don't support it | 15:09 |
| mordred | one is "it's not everywhere" | 15:09 |
| mordred | the other one is "it's not stable" | 15:09 |
| mordred | so, even if it was everywhere, we'd STILL not recommend using it in the gate | 15:09 |
| mordred | bceause it's not stable | 15:09 |
| mordred | because it's nested virt | 15:09 |
| fungi | for example, making kvm nested virt work on a xen domu in rackspace is highly unlikely to function | 15:09 |
| *** caowei has quit IRC | 15:09 | |
| mordred | not _actual_ normal kvm | 15:09 |
| johnsom | mordred can you point me to some bugs that address the instability? | 15:09 |
| BobBall | fungi: As I understand it, Rackspace do not enable nested virt. | 15:10 |
| mordred | johnsom: not really, it was never worth chasing, so basically no work has been done on it | 15:10 |
| diltram | yeah, it will be nice to see something, I would like to test that on our cloud | 15:10 |
| clarkb | johnsom: there are no bugs because the only data we have is i stanve goes away | 15:10 |
| diltram | BobBall: but then we can still use qemu | 15:10 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenario001: deploy Ceph https://review.openstack.org/366810 | 15:10 |
| fungi | BobBall: right. so what i'm saying is we'd either have to reject donations from rackspace and others who lack nested virt, or continue assuming it isn't available consistently | 15:10 |
| BobBall | qemu absolutely, just not with kvm | 15:10 |
| clarkb | johnsom: as soon as we turn off nested virt that syops | 15:10 |
| johnsom | I'm worried we are shooting ourselves in the foot with FUD | 15:10 |
| mordred | johnsom: nope | 15:10 |
| *** annegentle has joined #openstack-infra | 15:10 | |
| mordred | johnsom: we cannot start relying on it | 15:11 |
| fungi | johnsom: being unable to do nested kvm on xen is fud? | 15:11 |
| mordred | because we have at least one cloud provider that will NEVER have it | 15:11 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Revert "Revert "Revert "Disable rax-iad due to launch failure rate"""" https://review.openstack.org/366792 | 15:11 |
| BobBall | fungi: I would recommend Rackspace against enabling nested virt; it exposes too much risk to the level 0 hypervisor of the guest VM being able to crash the host. | 15:11 |
| mordred | BobBall: yah | 15:11 |
| diltram | guys, you're completely not listening us | 15:11 |
| mordred | diltram: you are completely not listening to us | 15:12 |
| fungi | diltram: the feeling is mutual | 15:12 |
| mordred | we ACTIVELY do not want to use it | 15:12 |
| mordred | it's an active choice | 15:12 |
| johnsom | So here is the follow up question if we can't point to bugs on the issues. Can we setup a pool for our scenario gates that does allow it? The qemu performance issue is a serious problem for us to be able to have complete scenario tests. | 15:12 |
| mordred | johnsom: no, we cannot | 15:12 |
| mordred | johnsom: we cannot run jobs on only a subset of our providers | 15:13 |
| mordred | it's a very important feature of the system that we span multiple clouds | 15:13 |
| mordred | because clouds go away all the time | 15:13 |
| mordred | and if the one cloud that provides that goes away | 15:13 |
| mordred | all of a sudden your jobs stop working | 15:13 |
| mordred | like, completely stop working | 15:13 |
| mordred | so depending on a feature that only exists in one cloud is not a thing that is supported at all - no matter what the feature is | 15:13 |
| johnsom | Yeah, I understand that. We have at least two providers with it enabled. Hmmmm. | 15:13 |
| fungi | diltram: please explain how having some percentage of your jobs run faster if they happen to end up on a provider supporting nested virt (and deadlocking your project when your jobs failing if they happen to not work with or without nested virt because some earlier change that only worked with one and not the other merged to your repo by sheer luck) is beneficial to you | 15:14 |
| *** mdrabe has quit IRC | 15:14 | |
| mordred | I mean - I get the desire for sure | 15:14 |
| *** mdrabe has joined #openstack-infra | 15:14 | |
| *** hieulq_ has joined #openstack-infra | 15:14 | |
| fungi | testing platforms need consistency. we either need to run jobs _with_ nested virt everywhere (and stop accepting resource donations from providers that don't support it) or not use nested virt acceleration anywhere | 15:14 |
| mordred | there are jobs we could run much better if they were faster- we just don't have consistent enough access to such resources - and we know that at least one of our providers (the one that has been the most consistent over time, fwiw) will never provide it | 15:15 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tripleo: run scenarios jobs on tripleo-ci project https://review.openstack.org/366814 | 15:15 |
| diltram | fungi: because when our jobs are landing on that gates we have results faster, and we're not blocking gates with out jobs | 15:15 |
| fungi | diltram: except you don't. say 66% of the jobs for one change complete sooner because nested virt acceleration was available on the providers where they ran but the other 33% ran in providers where it was not and so were slow | 15:16 |
| diltram | plus we currently need to split all testing scenarions on smaller chunks to even be able to test our code | 15:16 |
| fungi | the change itself still takes just as long to complete testing | 15:16 |
| openstackgerrit | Merged openstack-infra/python-storyboardclient: Give user option to bypass verification https://review.openstack.org/366099 | 15:16 |
| johnsom | The issue for us is two fold. It burns up a ton of host time to run some simple tests and we are limited on the number of tests we can run because with qemu tcg they exceed the maximum run time the gates allow. | 15:16 |
| mordred | in any case- we can't stop you from turning it on inside of your jobs - but the question was "why don't we use the default kvm/fallback code in devstack" more generally | 15:17 |
| mordred | the answer to that is the consistency answer | 15:17 |
| mordred | and that's why it's not going to be a normal config for normal devstack jobs | 15:17 |
| mordred | we would counsel, based on past experience | 15:17 |
| mordred | that turning it on in your repo will not have as much happy as you might want | 15:17 |
| mordred | largely due to the fact that it's not available everywhere | 15:17 |
| mordred | so you'll still have to plan for worst case | 15:18 |
| mordred | which means even if you do get _some_ fast runs, you will also get some slow runs | 15:18 |
| fungi | and lack a mechanism to allow you to conveniently avoid environments where you're unable to use nested kvm acceleration, and are unlikely to complicate our automation to add such support for a small fraction of projects that are doing heavy vm-in-vm workloads | 15:18 |
| mordred | so you'll still have to split things | 15:18 |
| *** dizquierdo has joined #openstack-infra | 15:18 | |
| *** sarob has joined #openstack-infra | 15:18 | |
| fungi | as for teh random undiagnosable failures we've seen in the past, they stem from part of the virtualization context for your nested vm belonging to the underlying hypervisor, for which we as tenants in the cloud have no log access so can't actually tell you why things broke | 15:20 |
| clarkb | odyssey4me: thank you for addressing those logs. That should help us increase log retention as things roll off the backend of the circular buffer | 15:20 |
| clarkb | mordred: corvus pabelanger fungi (and anyone else interested in nodepool things) https://review.openstack.org/#/c/366434/ is needed to help us with our long upload cycles. Also it may fix the source of our image leaks in smoe of the clouds | 15:21 |
| *** ociuhandu has joined #openstack-infra | 15:21 | |
| clarkb | fungi: right. We know based on testing that toggling kvm vs qemu seems to highly correlate to the behavior but we can't actually debug further than that as we lose all access to the instance | 15:21 |
| johnsom | fungi That just seems like we aren't working together as best as we could. | 15:21 |
| clarkb | I have left the instances up for days hoping they would come back up again and nothing | 15:22 |
| clarkb | they are truly dead and no longer available | 15:22 |
| *** martinkopec has quit IRC | 15:22 | |
| *** sarob has quit IRC | 15:23 | |
| fungi | johnsom: perhaps, but the infra team also has fairly limited bandwidth for working with various resource providers to determine why things are breaking. generally we attempt to limit using technologies which eat into what what time we have for tat | 15:24 |
| mgagne | in grafana, what's "Time to Ready" ? Is it time to SSH? does it also include time to "active" ? | 15:24 |
| mordred | yah. also, some of our providers are SUPER respondive - mgagne, mnaser and cloudnull being great examples of that | 15:24 |
| diltram | fungi: but to help you with that we're asking for some logs/bug smth to help with that | 15:24 |
| mordred | others of our providers ... not so much | 15:24 |
| *** beagles is now known as beagles_brb | 15:24 | |
| *** lucas-afk is now known as lucasagomes | 15:24 | |
| clarkb | mgagne: I believe that includes all of the time from startign to boot to flipping the db state to ready. This includes ssh and running our ready script on the instance then adding it to jenkins/zuul-launcher depending on when the data is from | 15:24 |
| *** sdake has joined #openstack-infra | 15:24 | |
| *** andreas_s has quit IRC | 15:25 | |
| clarkb | mgagne: its more of a time until we can run a job on the instance measure | 15:25 |
| fungi | diltram: sure, we end up with no logs. best we can tell you is the uuid of the dead job node | 15:25 |
| mordred | and since that's on a public cloud, we do not have access to their production logs | 15:25 |
| mgagne | clarkb: right. I would be interested in more granular metrics. Are those available in graphite? | 15:25 |
| johnsom | I understand that. I just see the huge performance improvements it provides and think about the impact it would have on our gate backlogs. | 15:25 |
| clarkb | mgagne: I am not sure. Let me check the nodepool statsd usage really quick | 15:26 |
| fungi | diltram: if we have available time, and if it's a regular occurrence, we engage with the service provider where that instance ran to try and figure out why the instance became unresponsive | 15:26 |
| mgagne | johnsom: I found enabling kvm challenging, even in our own private cloud where we found the CPU model had an impact on the stability of the VM running the nested VMs. | 15:26 |
| *** jkilpatr has quit IRC | 15:26 | |
| *** ifarkas is now known as ifarkas_afk | 15:27 | |
| diltram | would be nice fungi | 15:27 |
| diltram | mgagne: on intel or xen? | 15:27 |
| mgagne | diltram: intel | 15:27 |
| *** burgerk has joined #openstack-infra | 15:27 | |
| diltram | what cpu model | 15:27 |
| diltram | ? | 15:27 |
| * fungi wonders how intel and xen are an either/or question | 15:27 | |
| *** armax has joined #openstack-infra | 15:27 | |
| johnsom | mgagne Do you have a bug #? I find that really interesting as the nested virt is emulated CPU instructions, it's not lot it's implemented as a pure pass through. | 15:27 |
| mgagne | no bug #, we just changed the cpu model | 15:28 |
| diltram | hahahaha | 15:28 |
| diltram | nice | 15:28 |
| mgagne | for instance, E5-2650 v1 is unstable in our case, we had to upgrade to at least E5-2650 v2 | 15:28 |
| diltram | mgagne: how old that v1 cpus are | 15:28 |
| mgagne | you mean how long ago Intel released them? | 15:29 |
| clarkb | mgagne: actually rereading the source the time to ready is the time to first ssh we then bubble that further along | 15:29 |
| diltram | yeah | 15:29 |
| diltram | we have right now v4 in production | 15:29 |
| BobBall | fungi: We know that there are cases with nested virt for Xen where the support just isn't solid enough from the hypervisor side. I can't speak for KVM, but on Xen if you use nested virt today, then here be dragons and there are no garuntees... Even with the logs | 15:29 |
| diltram | we really changed almost everything in processors from time of v1 release | 15:30 |
| mgagne | diltram: right, this is for our private cloud, not public. But a public cloud provider *could* have this CPU model without you knowing and now you are stuck debugging something that could be totally outside your control | 15:30 |
| mgagne | Q1'12 =) | 15:30 |
| pabelanger | mordred: So... rax-iad is trying to setup ipv6 with glean | 15:30 |
| diltram | mgagne: I don't believe that someone is still using v1 cpus | 15:30 |
| pabelanger | mordred: http://paste.openstack.org/show/567498/ | 15:30 |
| clarkb | mgagne: however thats still a best effort by nodepool with all of the background threading goign on. Its likely not the most accurate number but is probably fairly reliable when looked at in aggregate over time | 15:31 |
| diltram | it's after 4 years of using them so companies are changing them | 15:31 |
| pabelanger | mordred: glean log: http://paste.openstack.org/show/567523/ | 15:31 |
| *** Benj_ has joined #openstack-infra | 15:31 | |
| *** Benj_ has quit IRC | 15:31 | |
| mgagne | clarkb: I'm looking at 50m difference, I think it's far outside the margin of error =) | 15:31 |
| *** dkehn__ has joined #openstack-infra | 15:32 | |
| clarkb | we also know that many clouds have heterogenous cpus | 15:33 |
| mgagne | clarkb: thanks for the info, I will try to find some time to look more into it this week. Just want to make sure it's not something on our side | 15:33 |
| clarkb | and not all of them lie to the VMs about what the cpu flags that are supported are | 15:33 |
| *** stevemar_ has joined #openstack-infra | 15:33 | |
| clarkb | anyways this is a tricky thing, we don't control the clouds which gives us limited options in many cases for debugging or actually fixing issues. Historically "tough" was a very common answer | 15:33 |
| diltram | all cpus starting from v2 should work without any problems because there was a huge change | 15:33 |
| *** jascott1_ has joined #openstack-infra | 15:34 | |
| diltram | but ok, thx for all info | 15:34 |
| clarkb | diltram: right but I can't even tell you what cpus are used in most of the clouds | 15:34 |
| mgagne | what's this "huge change" ? | 15:34 |
| clarkb | or that there aren't 100 different cpus in use | 15:34 |
| clarkb | thats the point | 15:34 |
| *** stevemar_ has quit IRC | 15:34 | |
| *** baoli_ has quit IRC | 15:34 | |
| clarkb | we have found a least common denominator that works reliably | 15:34 |
| clarkb | so we use it | 15:34 |
| *** jascott1_ has quit IRC | 15:35 | |
| *** ddieterly is now known as ddieterly[away] | 15:35 | |
| *** jpeeler has joined #openstack-infra | 15:35 | |
| *** jpeeler has quit IRC | 15:35 | |
| *** jpeeler has joined #openstack-infra | 15:35 | |
| clarkb | even testing live migration is ridiculously complicated because of this. We have our own libvirt cpu model now as a result | 15:35 |
| *** krtaylor has quit IRC | 15:35 | |
| clarkb | or did, I think libvirt broke making that work | 15:35 |
| clarkb | sdague: ^ did that ever get resolved in a way that nova was happy about? | 15:36 |
| johnsom | I just worry that we aren't accounting for advances in the technology, hardware but mostly software, that makes this a viable technology. Pretty much everyone on the development team runs with nested virt without issue. It takes VM boot times from 8-10 minutes down to 30 seconds on our dev boxes and we see the same benefit on the gate hosts that have it. | 15:37 |
| johnsom | If there is anyway we can do some experimental jobs or testing as a path to enabling it, we would support that effort. | 15:37 |
| *** david-lyle_ has joined #openstack-infra | 15:38 | |
| *** ddieterly[away] is now known as ddieterly | 15:38 | |
| clarkb | johnsom: as mordred said you can make your jobs run with it | 15:38 |
| *** david-lyle_ has quit IRC | 15:38 | |
| clarkb | but its not going to be our default | 15:38 |
| *** Sukhdev has joined #openstack-infra | 15:38 | |
| clarkb | which is what the question was | 15:38 |
| johnsom | Yeah, I'm not pushing for default. That doesn't bother me. It's encouraging the providers to enable the capability that is the challenge. | 15:39 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenario001: deploy Ceph https://review.openstack.org/366810 | 15:39 |
| mordred | johnsom: totally hear that - and honestly, working with providers to push things forward is a worthwhile thing for sure | 15:41 |
| *** chlong has quit IRC | 15:41 | |
| clarkb | mordred: mtreinish the reason ES was not working is es02 (which is the one we proxy to) was not running an es process | 15:41 |
| *** salv-orlando has joined #openstack-infra | 15:41 | |
| clarkb | mordred: that explains why the logs were so light :) | 15:41 |
| mordred | clarkb: oh. piddle | 15:41 |
| clarkb | I have restarted the process and the cluster is recovering | 15:41 |
| mtreinish | clarkb: yeah I was just looking at the cacti for es02 and it stopped at like 2am on the graphs | 15:41 |
| *** Benj_ has joined #openstack-infra | 15:41 | |
| clarkb | (we can run a local es process on logstash.o.o and have it load balance for us) | 15:41 |
| clarkb | but that requires more memory on that server | 15:41 |
| pabelanger | clarkb: should have pinged you too, glean is configuring rax-iad eth0 to be only ipv6 | 15:43 |
| *** adrian_otto1 has quit IRC | 15:43 | |
| *** matthewbodkin has quit IRC | 15:43 | |
| pabelanger | clarkb: however, ipv4 is also passed: http://paste.openstack.org/show/567498/ | 15:43 |
| clarkb | pabelanger: fun, does it work as configured? might need to tell nodepool to use ipv6 only in rax? | 15:43 |
| pabelanger | clarkb: that is what I am testing now | 15:44 |
| fungi | pabelanger: interesting. do you have an example /etc/network/interfaces from that? | 15:44 |
| *** adrian_otto has joined #openstack-infra | 15:44 | |
| pabelanger | fungi: http://paste.openstack.org/show/567523/ | 15:44 |
| fungi | i'm curious to see what it decided to configure | 15:44 |
| pabelanger | that is what glean does | 15:44 |
| clarkb | pabelanger: fungi and probably want the config drive network_info too | 15:44 |
| openstackgerrit | Chris Holcombe proposed openstack-infra/project-config: Add Ceph Charm Shared Lib https://review.openstack.org/366842 | 15:45 |
| pabelanger | that is the first pastebin | 15:45 |
| fungi | right, in the past it was missing in at least some rackspace regions | 15:45 |
| *** senk has joined #openstack-infra | 15:45 | |
| pabelanger | So, I think glean should be setting up both ipv4 / ipv6 for eth0 | 15:45 |
| *** gongysh has quit IRC | 15:45 | |
| pabelanger | but it is only doing ipv6 | 15:45 |
| clarkb | yes this was never a problem before because config drive had no ipv6 info | 15:45 |
| clarkb | pabelanger: ya its likely a bug since its never been done before | 15:45 |
| pabelanger | clarkb: yes | 15:45 |
| pabelanger | I think rackspace changed this back in Aug. 8th | 15:46 |
| fungi | probably glean needs some tweaking to grok finding both v4 and v6 addresses in the configdrive metadata file | 15:46 |
| clarkb | mtreinish: do things work for you now? maybe a little slow but should msotly function | 15:46 |
| pabelanger | time to update glean test cases | 15:46 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 15:46 |
| sdague | clarkb: we deleted all that custom cpu stuff, because we're running under qemu | 15:46 |
| sdague | and we don't need it | 15:46 |
| dhellmann | mordred : thanks for that. is it worth a ML thread in case others see the same issue? (maybe there is one and I haven't caught up yet) | 15:46 |
| sdague | there was a nova bug that made us use it for qemu, which was fixed and backported | 15:46 |
| clarkb | sdague: oh interesting, I always did think it a little odd that we would care about that with qemu | 15:47 |
| *** salv-orlando has quit IRC | 15:48 | |
| mtreinish | clarkb: I'm not getting 503s anymore hitting 300sec timeouts on my requests but that's likely just while the cluster is recovering | 15:49 |
| *** jkilpatr has joined #openstack-infra | 15:49 | |
| openstackgerrit | Merged openstack-infra/release-tools: Add libffi-dev to bindep.txt https://review.openstack.org/366774 | 15:49 |
| clarkb | mtreinish: ya its redistributing indexes now which is likely slowing the cluster down for queries | 15:49 |
| odyssey4me | clarkb AJaeger ianw we're working on compressing the logs from the OSA jobs - I wonder if it'd be possible to add the appropriate changes to the web server to make the log files viewable even if compressed (like the compressed html files)? for example: http://logs.openstack.org/09/366709/3/gate/gate-openstack-ansible-security-ansible-func-ubuntu-xenial/aa86cfd/logs/ | 15:50 |
| clarkb | odyssey4me: yes that should all be there already. Its how the devstack-gate logs work | 15:50 |
| odyssey4me | clarkb hmm, then I wonder why it's not working for those :/ | 15:51 |
| clarkb | odyssey4me: you may need to rename the files first to have a .txt extension | 15:51 |
| clarkb | odyssey4me: check the logs from a tempest job run | 15:51 |
| odyssey4me | ah ok, let me give that a go | 15:51 |
| clarkb | odyssey4me: then when you compress you end up with .txt.gz | 15:51 |
| *** rajinir has joined #openstack-infra | 15:52 | |
| *** ociuhandu has quit IRC | 15:52 | |
| *** yolanda has quit IRC | 15:53 | |
| *** psachin has quit IRC | 15:54 | |
| anteaya | for logging purposes I am not an ant, I don't care what marketers say | 15:55 |
| *** yolanda has joined #openstack-infra | 15:55 | |
| *** annegentle has quit IRC | 15:55 | |
| anteaya | nor am I a wasp nor any other insect | 15:55 |
| *** shamail has joined #openstack-infra | 15:56 | |
| anteaya | if you really want to know I formed a bond with a red squirrel yesterday | 15:57 |
| anteaya | was a lovely feeling | 15:57 |
| *** salv-orlando has joined #openstack-infra | 15:57 | |
| *** dtantsur is now known as dtantsur|afk | 15:57 | |
| dougwig | wow, i missed the nested virt fun. | 15:57 |
| anteaya | he is busy planting trees around me to help me | 15:57 |
| anteaya | dougwig: you did so, welcome back to today's irc | 15:57 |
| *** sandanar has joined #openstack-infra | 15:58 | |
| *** annegentle has joined #openstack-infra | 15:59 | |
| jamesmcarthur | hi! Wondering if one of you kind folks could help me. Got this message on Twitter "@OpenStack your IP 185.110.132.87 is trying to ssh brute-force my server." | 16:00 |
| mordred | pabelanger: oh - weird | 16:00 |
| anteaya | jamesmcarthur: thanks for reporting | 16:00 |
| *** vhosakot has joined #openstack-infra | 16:00 | |
| mordred | jamesmcarthur: I don't have a login to anything on 185.110.132.87 | 16:00 |
| jamesmcarthur | I checked and it does appear to be registered to OpenStack, but it looks like the server is in Russia. | 16:01 |
| mgagne | @openstack.net.ua | 16:01 |
| mordred | jamesmcarthur: I'm confused about openstack having any ips registered at all | 16:01 |
| *** pcaruana has quit IRC | 16:01 | |
| mgagne | looks unrelated to openstack itself | 16:01 |
| mordred | yah. this sounds like a thing to get jbryce involved in asap | 16:01 |
| jamesmcarthur | mordred: yeah... could be bogus | 16:01 |
| jamesmcarthur | http://whatismyipaddress.com/ip/185.110.132.87 | 16:01 |
| jamesmcarthur | I'm assuming that's where the guy got the info though | 16:01 |
| mordred | yah | 16:02 |
| jamesmcarthur | Initially thought it might be part of o.o block at Rackspace | 16:02 |
| mordred | fungi: ^^ in case you didn't see | 16:02 |
| greghaynes | huh, and that site has a gerrit on it | 16:02 |
| fungi | i'm digging into it now | 16:03 |
| jamesmcarthur | It also lists us as the ISP, which is nice | 16:03 |
| clarkb | whois says it belongs to descr: Union Financial Service Ltd | 16:03 |
| openstackgerrit | Alex Kavanagh proposed openstack-infra/project-config: Add charm-manila https://review.openstack.org/360762 | 16:03 |
| jamesmcarthur | "My internet is broken" | 16:03 |
| mordred | the server itself seems to be running windows | 16:03 |
| mordred | http://185.110.132.87/ brings up an IIS banner :) | 16:03 |
| fungi | Union Financial Service Ltd, Karamurzov Barasbi <abusemail@openstack.net.ua> | 16:03 |
| fungi | per whois | 16:04 |
| * clarkb wonders why people wouldn't use whois and report things on twitter | 16:04 | |
| *** hieulq_ has quit IRC | 16:04 | |
| clarkb | the world has gone upside down and inside out | 16:04 |
| mordred | well, I'm glad they did | 16:04 |
| mordred | because otherwise we would not have learned about this human | 16:04 |
| pabelanger | mordred: fungi: clarkb: mind helping land: https://review.openstack.org/#/c/355644/ afs mirror in tripleo-test-cloud-rh1 might be having some networking issues, would be great to see if cacti.o.o confirms that | 16:04 |
| mordred | pabelanger: wfm | 16:05 |
| *** beagles_brb is now known as beagles | 16:05 | |
| fungi | openstack.net.ua is registered to "Domen zaregistrirovan v interesah klienta" with nameservers in yandex.net | 16:05 |
| greghaynes | heh, so that site looks like a non ssl gerrit that proxies to ours :/ | 16:05 |
| jamesmcarthur | clarkb: it's more fun to yell at people on Twitter ;) | 16:06 |
| *** abregman has quit IRC | 16:06 | |
| anteaya | I've learned that that has become the main purpose of twitter, to yell at others | 16:06 |
| anteaya | hence my desire to no longer twitter | 16:06 |
| jamesmcarthur | anteaya: i use it to yell at our mayor, but I feel justified since he killed Uber. | 16:07 |
| anteaya | jamesmcarthur: sounds like you are using it for its new purpose | 16:08 |
| clarkb | fungi: and the A record for it points at 127.0.0.1 | 16:08 |
| fungi | AS40965 seems to be an infium customer, "Rise-v, Ltd." in the ukraine | 16:08 |
| anteaya | and I personally don't like uber, but the point of the discussion is clarifying identities so I am willing to stick to the point | 16:08 |
| greghaynes | oh im a derp I just missed the a points at us | 16:09 |
| clarkb | greghaynes: which A? | 16:09 |
| *** tesseract- has quit IRC | 16:09 | |
| fungi | traceroute goes through ua.retn.net and then maxnet.ua so clearly ukranian | 16:09 |
| greghaynes | er, sorry, it points at me | 16:10 |
| *** akshai has joined #openstack-infra | 16:10 | |
| greghaynes | becaues of what you said | 16:10 |
| clarkb | greghaynes: ha they got you! | 16:10 |
| greghaynes | and I have a funky setup from hacking on gerrit | 16:10 |
| greghaynes | ya | 16:10 |
| greghaynes | :) | 16:10 |
| greghaynes | way to mitm yourself, greg | 16:10 |
| mordred | greghaynes: nicely done! | 16:11 |
| openstackgerrit | Shamail Tahir proposed openstack-infra/irc-meetings: Add Interop Challenge Weekly IRC Meeting https://review.openstack.org/366851 | 16:11 |
| *** tqtran has joined #openstack-infra | 16:11 | |
| *** Julien-zte has quit IRC | 16:12 | |
| clarkb | anyways not sure there is much that can be done other than to say "please learn to use whois and traceroute" and maybe try to get that dns domain sorted out (not sure if legit or not) | 16:12 |
| *** roxanaghe has joined #openstack-infra | 16:12 | |
| mordred | I can't imagine the domain is legit | 16:13 |
| dougwig | mordred, fungi: re: nested virt, it totally makes sense that it can't be everywhere, and that it shouldn't be relied upon. would we entertain a change where it's possible to note what extra features a cloud has, and then another to be able to specify job preferences, with a fallback to standard if no nodes available? something along the lines of | 16:14 |
| dougwig | octavia/trove/tacker adding a flag to their jobs that says, "hey, i'd work better on nested virt, if available". or too complicated to maintain? | 16:14 |
| jamesmcarthur | clarkb: mordred: fungi: thanks for info | 16:14 |
| fungi | i doubt the sort of people who are inclined to complain about network attacks through twitter are likely to figure out whois and traceroute unless those have a proxy through twitter | 16:14 |
| jamesmcarthur | lol | 16:14 |
| *** wznoinsk has joined #openstack-infra | 16:15 | |
| *** ajo_ has quit IRC | 16:15 | |
| mordred | dougwig: certainly not before zuul v3 is done - and I think we'd need to think long and hard about what effect that would have on scheduling | 16:16 |
| *** ajo_ has joined #openstack-infra | 16:16 | |
| mordred | dougwig: the interaction between "I want node type X" and a system that creates nodes based on demand | 16:16 |
| mordred | dougwig: means that I think getting 'fallback' would be extra hard | 16:16 |
| *** gyee_ has joined #openstack-infra | 16:17 | |
| mordred | dougwig: since "if you don't have one" isn't really a concept that fits in | 16:17 |
| greghaynes | dougwig: have you all given thought to trying to container? Since dib abstracts away the output format it might be not a lot of work beyond -t docker... | 16:17 |
| fungi | ooh! `whois AS21141` | 16:17 |
| dougwig | mordred: i'd think we'd want to it be more, "i'd like node X, if you already have it or have to spawn a new one", rather than have it be a direct trigger. might as well just give osic nodes their own label if it was a hard requirement on the job, which i think we'd all oppose. | 16:17 |
| fungi | "Openstack Ltd" | 16:17 |
| greghaynes | dougwig: and just use the same elements you all use already | 16:17 |
| *** florianf has quit IRC | 16:17 | |
| * amrith tries to follow what's being said | 16:18 | |
| fungi | so it looks like there may be a service provider in the ukraine doing business as "openstack ltd" | 16:18 |
| mordred | dougwig: I hear that - but the flow is actually always "please spawn me a X node" | 16:18 |
| mordred | dougwig: with pre-spawning of nodes being an optimization that we are able to use a lot | 16:18 |
| dougwig | greghaynes: aye, containers is also being investigated as a solution to the time issue, via nova-lxd i believe. | 16:18 |
| mordred | ++ | 16:19 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 16:19 |
| anteaya | fungi: that sounds like a clear breach of trademark | 16:19 |
| mordred | nova-lxc/nova-lxd is likely a GREAT option that'll be more generally applicable | 16:19 |
| clarkb | fungi: nice | 16:19 |
| jamesmcarthur | fungi: agreed and good find. I can pass along to lawyer town. | 16:19 |
| greghaynes | dougwig: good deal. I've been hoping someone would make use of dib in this way (using it as an abstraction layer between vm images and containers) so let me know if theres dib hacks I can help with | 16:19 |
| *** tqtran has quit IRC | 16:20 | |
| dougwig | mordred: aye, i'm floating ideas on the nested virt thing. i was warring with thinking it was too messy as i typed it, too. feels like there's something there, but the expense may not be worth it. | 16:20 |
| fungi | jamesmcarthur: awesome--thanks! | 16:20 |
| mordred | dougwig: ++ | 16:20 |
| mordred | dougwig: I think that's the best way to put it | 16:20 |
| fungi | anteaya: well, "trademark" law in the ukraine may not exist at all, or might at a minimum completely ignore trademarks registered in most other countries | 16:21 |
| fungi | jurisdiction for intellectual property laws is often nebulous | 16:21 |
| *** akshai_ has joined #openstack-infra | 16:21 | |
| *** akshai has quit IRC | 16:21 | |
| fungi | or there could actually be trademark laws they're in breach of which are generally just not enforced ever so they don't care | 16:22 |
| mordred | dougwig: we certainly _could_ do a thing ... it's only code, after all - but at the end of the day I think the results would not be desirable - and getting nova-lxc as a workable host for things that need to test guest agents seems lovely | 16:22 |
| clarkb | mordred: dougwig related to that thats the entire benefit to supporting multiple "hypervisors" behind nova in the first place | 16:23 |
| clarkb | so we should take advantage | 16:23 |
| anteaya | fungi: hmmmm | 16:23 |
| *** thorst has quit IRC | 16:24 | |
| *** thorst has joined #openstack-infra | 16:24 | |
| clarkb | (with consistent apis I can leverage the benefits of $hypervisor where more appropriate without completely rewriting all of my code) | 16:24 |
| *** krtaylor has joined #openstack-infra | 16:26 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Change Kolla jobs to 90 minutes for short term https://review.openstack.org/366859 | 16:27 |
| dougwig | clarkb, mordred: the counter-argument to that (devil's advocate) is that NIC plugging is so different that we will also also have to test VMs anyway. and the counter-counter to that (devil's devil?) is that most jobs could use containers, with one sanity job using VMs with a minimal feature set, and hey, we can enabled nested virt in that one if it happens | 16:27 |
| dougwig | to be available. | 16:27 |
| dougwig | i can really just argue with myself. | 16:28 |
| *** matt-borland has quit IRC | 16:28 | |
| *** dmsimard|afk is now known as dmsimard | 16:28 | |
| *** notnownikki has quit IRC | 16:28 | |
| clarkb | dougwig: ya I still don't understand the whole nic plugging thing. AIUI its because you want to require shared L2 between front and backends which doesn't amke sense to me | 16:29 |
| *** baoli has joined #openstack-infra | 16:29 | |
| clarkb | dougwig: just rely on your route table and then the abstraction holds regardless of VM and container and it means I have a more flexible tool to use in whatever cloud I choose | 16:29 |
| *** sshnaidm is now known as sshnaidm|afk | 16:30 | |
| anteaya | dougwig: be sure to take a pause so the audience knows when to applaud | 16:30 |
| dougwig | clarkb: a very common use case of LBs is to not ever put the backend servers on a routable network, so the LB needs to be multi-homed in that case and act as a proxy for the VIP. | 16:30 |
| clarkb | dougwig: right you can still do that is my point. But rely on the route table not doing it directly | 16:31 |
| clarkb | dougwig: locally attached networks get route table entries | 16:31 |
| *** jpich has quit IRC | 16:31 | |
| *** ddieterly is now known as ddieterly[away] | 16:32 | |
| *** thorst has quit IRC | 16:32 | |
| *** jaosorior has quit IRC | 16:33 | |
| *** gus has quit IRC | 16:33 | |
| dougwig | clarkb: that works on a single node devstack, but unless i'm misunderstanding, that's not testing how the service VM is going to use neutron ports in the wild. | 16:33 |
| dougwig | clarkb: because in the wild, a route may not be possible. | 16:33 |
| clarkb | dougwig: my point is its not your test's job to test that. Its neutrons | 16:34 |
| clarkb | dougwig: if the load balacner's abstractions is nova + neutron + route table then you just assume it works then make sure neutron + nova test nic plugging | 16:34 |
| clarkb | by separating the concerns there you make testing a lot saner | 16:34 |
| dougwig | clarkb: that's a fair point, though i've been burned in the past when testing an application single-homed and then running it multi-homed later. | 16:35 |
| *** ddieterly[away] is now known as ddieterly | 16:35 | |
| *** yamahata has quit IRC | 16:35 | |
| clarkb | dougwig: sort of in the same way we assume libvirt's apis are going to work and ovs's and so on. Occasionally they do break on us and we address it then | 16:36 |
| clarkb | but most of the time we don't worry and trust the api | 16:36 |
| mordred | pabelanger: I think I see the glean bug | 16:37 |
| clarkb | and I think thats the right mechanism here and the one you should take for eg service instance testing | 16:37 |
| *** asselin_ has quit IRC | 16:37 | |
| pabelanger | mordred: Oh, nice. I'm still working on it. So far, it looks like a problem in get_config_drive_interfaces? | 16:37 |
| mordred | pabelanger: it has sections of ipv4 and ipv6 - but it treats interfaces as a thing with a single entry | 16:38 |
| pabelanger | ah | 16:38 |
| mordred | pabelanger: so depending on load order, it'll either put the ipv4 or the ipv6 for the interface into the interfaces dicct | 16:38 |
| pabelanger | right | 16:38 |
| sdake | hey folks | 16:39 |
| sdake | see alot of cores around | 16:39 |
| sdake | could I get an emergency merge on https://review.openstack.org/366859 | 16:39 |
| pabelanger | mordred: so, we likely want both ipv4 and ipv6 if provided | 16:39 |
| mordred | yah. | 16:39 |
| mordred | pabelanger: check this out ... | 16:39 |
| sdake | short term - we plan to fix properly asap | 16:39 |
| *** irtermite is now known as irjroll | 16:39 | |
| *** sigmavirus is now known as irvirus | 16:40 | |
| *** irvirus is now known as sigmavirus | 16:40 | |
| *** irjroll is now known as irtermite | 16:40 | |
| zaro | morning | 16:41 |
| *** gus has joined #openstack-infra | 16:42 | |
| *** shamail has left #openstack-infra | 16:42 | |
| anteaya | morning zaro | 16:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/glean: Append to interface definition for multi-address https://review.openstack.org/366862 | 16:43 |
| anteaya | sdake: why is this an emergency? | 16:43 |
| mordred | pabelanger: ^^ not tested. needs test cases | 16:43 |
| *** asselin_ has joined #openstack-infra | 16:43 | |
| sdake | anteaya because our gates are fialing | 16:43 |
| sdake | anteaya and blocking merges | 16:43 |
| sdake | resulting in 3-4 recheecks | 16:43 |
| *** akshai_ has quit IRC | 16:43 | |
| sdake | before they finally go green | 16:43 |
| mordred | sdake: lgtm | 16:43 |
| *** fguillot has quit IRC | 16:43 | |
| sdake | thanks mordred :) | 16:43 |
| pabelanger | mordred: okay, let me rebase on that. I have tests going locally already | 16:44 |
| sdake | failing because of timeout not becauset hey are broken | 16:44 |
| sdake | they actually make forwrd progress on allgate fialures i've seen | 16:44 |
| sdake | just time out | 16:44 |
| mordred | pabelanger: neat! | 16:44 |
| sdake | i do see some random gate faillures here and there that we are debugging | 16:44 |
| sdake | but this one is persistent | 16:44 |
| sdake | i'd say 70-80% off our reviews get stuck with recheck madness | 16:44 |
| mordred | pabelanger: we should _also_ probably figure out the case of static ipv4 and no entry for ipv6 and we want to tell ipv6 to just respond to RAs - but if that's not the immediate concern we can do it later | 16:45 |
| sdake | anteaya before the core team will even look at a review, they want the gate to be green | 16:45 |
| anteaya | sdake: what is your definition of short term? | 16:45 |
| *** lucasagomes is now known as lucas-dinner | 16:45 | |
| sdake | anteaya i plan to submit a stack of mirroring patches to solve the problem in early o1 | 16:45 |
| sdake | 1-2 weeks in | 16:45 |
| sdake | then its on infra to make the mirrors :) | 16:46 |
| sdake | by stack - I mean 8-10 patches | 16:46 |
| clarkb | I wouldn't count on mirrors saving 30 minutes per job | 16:46 |
| clarkb | devsatck gained like 5-8 minutes from wheel mirros whcih you already use aiui | 16:46 |
| anteaya | sdake: do you have someone in infra who agreed to create mirrors? | 16:47 |
| mordred | I wouldn't either - but I don't have a problem with this as a thing until we get more mirrors made | 16:47 |
| sdake | anteaya i do not | 16:47 |
| anteaya | sdake: might be best to get that person to approve the patch | 16:47 |
| clarkb | mordred: ya bumping the timeouts is fine. Just think we should identify the slow bits and specifically target them rather than hope mirros are a huge performance boost | 16:47 |
| sdake | clarkb mirrors save 30 minutes on ubuntu jobs.. | 16:47 |
| sdake | (for ubuntu) | 16:47 |
| clarkb | (we have the mirrors more for reliability than performance) | 16:47 |
| sdake | we use the packaging mirrors | 16:48 |
| clarkb | sdake: are you not using the package cache on the images? | 16:48 |
| mordred | sdake: wow, do they really? | 16:48 |
| anteaya | sdake: well if you are saying it is on infra to do something, I'd like to see someone in infra saying they are willing to do the work | 16:48 |
| sdake | clarkb we do | 16:48 |
| sdake | mordred indeed | 16:48 |
| clarkb | sdake: then the mirrors really shouldn't save that much time | 16:48 |
| mordred | oh - it's because you make a ton of containers | 16:48 |
| mordred | clarkb: he's doing apt installs _multiple_ times - not just once | 16:48 |
| sdake | ya we make about 50-60 containers | 16:48 |
| sdake | nah actually more like 130 or so | 16:48 |
| clarkb | mordred: ya but with the local package cache those are really fast | 16:48 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: [Rally] enable panko project in rally jobs https://review.openstack.org/366865 | 16:48 |
| clarkb | oh wat | 16:48 |
| mordred | clarkb: apparently they aren't | 16:48 |
| sdake | th local package cache heps | 16:49 |
| mordred | I think it's purely mutiplicative math in sdake's case | 16:49 |
| sdake | turnign off bindep helped | 16:49 |
| mordred | and I have faith that sdake is motivated to continue to work to make the situation better over time | 16:49 |
| sdake | anteaya pabelanger is the only cat i know who knows how to create the mirrors | 16:49 |
| anteaya | me too | 16:49 |
| sdake | pabelanger any chance ou can ack that change and get some training going on the mirrors? | 16:49 |
| clarkb | sdake: have a link to a timing out build log? | 16:49 |
| anteaya | but it is not up to me to tell him what to do | 16:49 |
| *** Apoorva has joined #openstack-infra | 16:49 | |
| sdake | clarkb moment | 16:49 |
| anteaya | and it isn't obliged to create mirrors just because someone else expects him to | 16:50 |
| *** trown is now known as trown|lunch | 16:50 | |
| sdake | clarkb https://review.openstack.org/#/c/366430/ | 16:50 |
| sdake | anteaya totaly agree | 16:50 |
| anteaya | sdake: so this is where I'd recommend a more diplomatic approach, rather than 'then it is on infra to create the mirrors' | 16:50 |
| sdake | anteaya in his last review he indicated soem training was needed internally | 16:50 |
| clarkb | sdake: all of those succeeded | 16:50 |
| sdake | anteaya apologies if that came off as brusque | 16:51 |
| anteaya | sdake: then I suggest you continue your discussion with him | 16:51 |
| sdake | it wasn't my intent | 16:51 |
| clarkb | sdake: do you have a link to the logs of a job that timed out? | 16:51 |
| anteaya | sdake: well I'd suggest you clarify your intent with someone in infra willing to talk to you about creating mirrors to help you | 16:51 |
| sdake | clarkb moment | 16:51 |
| sdake | clarkb https://review.openstack.org/#/c/352515/ | 16:52 |
| clarkb | sdake: I see them tahnks | 16:52 |
| clarkb | also have a centos mirror now too | 16:53 |
| clarkb | (looks like its centos jobs that timed out) | 16:53 |
| sdake | epel and centos itself - i think we are using thosee | 16:53 |
| sdake | let me check | 16:53 |
| *** ddieterly is now known as ddieterly[away] | 16:53 | |
| anteaya | s/it/he timestamp 16:50 | 16:54 |
| *** yaume has quit IRC | 16:55 | |
| *** Swami_ has quit IRC | 16:55 | |
| *** hashar has quit IRC | 16:55 | |
| clarkb | sdake: http://logs.openstack.org/15/352515/20/check/gate-kolla-dsvm-deploy-centos-source-centos-7-nv/96b0812/console.html#_2016-09-07_15_49_14_984219 I don't think you are (I will admit to not being an expert at reading yum output and may misinterpret that | 16:56 |
| *** mikelk has quit IRC | 16:56 | |
| sdake | clarkb we are not - just confirmd | 16:56 |
| pabelanger | mordred: I see the issue, it is because both the ipv4 / ipv6 share the same link id: tap8056a42d-1c in this case. Which is causing an issue here: http://git.openstack.org/cgit/openstack-infra/glean/tree/glean/cmd.py#n524 | 16:57 |
| sdake | clarkb - ok we will do that right now | 16:57 |
| clarkb | sdake: cool | 16:57 |
| sdake | as in today | 16:57 |
| *** vhosakot has quit IRC | 16:57 | |
| pabelanger | mordred: that is the first issue | 16:57 |
| sdake | if that solves the problem - i'd be happy to revert the 90 minute time back to 60 | 16:57 |
| pabelanger | clarkb: sdake: yes, that means you are using mirrorlist to resolve base / epel | 16:58 |
| *** Benj_ has quit IRC | 16:58 | |
| *** dahoo has joined #openstack-infra | 16:58 | |
| clarkb | you should also see if you are using the local yum package cache | 16:58 |
| *** watanabe_isao has joined #openstack-infra | 16:58 | |
| sdake | clarkb we are not - we can't do that unfortunately | 16:59 |
| sdake | clarkb we delete the cache after every major build | 16:59 |
| *** annegentle has quit IRC | 16:59 | |
| sdake | or every yum command rather | 16:59 |
| *** _nadya_ has joined #openstack-infra | 17:00 | |
| clarkb | sdake: what is the reason for that? | 17:00 |
| sdake | yum cache sucks up disk space which annoys people ;( | 17:00 |
| sdake | we already have complaints our images are oto large | 17:00 |
| clarkb | also http://logs.openstack.org/15/352515/20/check/gate-kolla-dsvm-deploy-centos-source-centos-7-nv/96b0812/console.html#_2016-09-07_15_59_38_789124 is where your image builds end. Takes about 17 minutes. The actual tests then take all the time until the timeout | 17:00 |
| sdake | it doens't affect me I am on gige to the intenet | 17:00 |
| clarkb | so even fi you werent doing an image build you'd only save 17 minutes | 17:00 |
| sdake | clarkb interesting | 17:00 |
| sdake | clarkb I hadn't picked upon that | 17:00 |
| clarkb | sdake: bind mount the cache in from the host and make it an option? just pointing out we cache things to speed stuff up | 17:01 |
| *** Sukhdev has quit IRC | 17:01 | |
| clarkb | so actively undermining that is a bit :( | 17:01 |
| sdake | clarkb i'll have to give that a think from the seecurity and usability pov | 17:01 |
| sdake | clarkb the above log where build takes 17 minutes on centos source | 17:02 |
| sdake | clarkb which review is that based off/ I wanted to see thebuild time for ubuntu source | 17:02 |
| *** jkilpatr has quit IRC | 17:02 | |
| *** ddieterly[away] is now known as ddieterly | 17:02 | |
| clarkb | sdake: the change number is in the url 352515 | 17:03 |
| *** gyee_ has quit IRC | 17:03 | |
| mordred | pabelanger: yes! | 17:03 |
| *** dkehn_ has quit IRC | 17:05 | |
| *** derekh is now known as derekh_ark | 17:05 | |
| *** derekh_ark is now known as derekh_afk | 17:05 | |
| *** annegentle has joined #openstack-infra | 17:05 | |
| *** dkehn has quit IRC | 17:06 | |
| *** ddieterly is now known as ddieterly[away] | 17:06 | |
| *** _nadya_ has quit IRC | 17:06 | |
| *** dkehn__ has quit IRC | 17:06 | |
| sdake | clarkb 9 minutes to bulid on ubuntu source | 17:07 |
| sdake | clarkb we have a whole slew more mirros for ubuntu to use | 17:07 |
| sdake | in the deploy gate | 17:07 |
| clarkb | sdake: sure definitely use the mirros in the centos job thatd what they are there for.yum is slower than apt in my experience so youay not see the samespeedup. and best caseyou shorten the jb by 17 minutes meaning if you actually want to save half anhour you need to look.in the testsinstead | 17:08 |
| sdake | clarkb i think 10 minuts would get the job done tbh | 17:10 |
| sdake | i just wnted to give us a little headroom | 17:10 |
| *** Swami has joined #openstack-infra | 17:11 | |
| *** vhosakot has joined #openstack-infra | 17:11 | |
| *** sshnaidm|afk is now known as sshnaidm | 17:11 | |
| *** flepied has quit IRC | 17:12 | |
| *** gyee_ has joined #openstack-infra | 17:12 | |
| *** gyee_ has quit IRC | 17:12 | |
| *** jcoufal has quit IRC | 17:13 | |
| *** gyee_ has joined #openstack-infra | 17:13 | |
| *** dkehn has joined #openstack-infra | 17:13 | |
| *** e0ne has quit IRC | 17:14 | |
| *** Sukhdev has joined #openstack-infra | 17:14 | |
| *** ajo_ has quit IRC | 17:14 | |
| *** ajo_ has joined #openstack-infra | 17:14 | |
| *** links has joined #openstack-infra | 17:15 | |
| *** krtaylor has quit IRC | 17:16 | |
| *** krtaylor has joined #openstack-infra | 17:16 | |
| *** annegentle has quit IRC | 17:17 | |
| *** ilyashakhat_mobi has joined #openstack-infra | 17:17 | |
| *** dkehn_ has joined #openstack-infra | 17:17 | |
| *** matt-borland has joined #openstack-infra | 17:18 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote working OSA CentOS nv jobs to voting https://review.openstack.org/366880 | 17:18 |
| *** AnarchyAo has joined #openstack-infra | 17:18 | |
| *** shardy has quit IRC | 17:19 | |
| *** dkehn__ has joined #openstack-infra | 17:21 | |
| *** Na3iL has quit IRC | 17:22 | |
| *** inc0 has joined #openstack-infra | 17:24 | |
| *** jkilpatr has joined #openstack-infra | 17:25 | |
| *** mrjk has quit IRC | 17:25 | |
| *** dizquierdo has quit IRC | 17:26 | |
| mriedem | AJaeger: it looks like elastic-recheck is missing something from bindep.txt | 17:28 |
| mriedem | AJaeger: http://logs.openstack.org/77/366277/1/check/gate-elastic-recheck-pep8-ubuntu-xenial/e191393/console.html#_2016-09-07_16_43_08_557732 | 17:28 |
| *** asettle has quit IRC | 17:28 | |
| mriedem | AJaeger: this is the current list for that repo https://github.com/openstack-infra/elastic-recheck/blob/master/bindep.txt | 17:28 |
| mriedem | is there a tool to find what's missing? | 17:28 |
| *** links has quit IRC | 17:29 | |
| *** HeOS has quit IRC | 17:29 | |
| *** gouthamr_ is now known as gouthamr | 17:29 | |
| *** thorst has joined #openstack-infra | 17:29 | |
| *** ihrachys has quit IRC | 17:30 | |
| mriedem | looks like it's missing libffi | 17:32 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add libffi-dev(el) to bindep https://review.openstack.org/366885 | 17:33 |
| *** _nadya_ has joined #openstack-infra | 17:35 | |
| openstackgerrit | Merged openstack-infra/project-config: add #openstack-ec2api to the list of OpenStack channels https://review.openstack.org/366510 | 17:35 |
| *** Sukhdev has quit IRC | 17:36 | |
| fungi | mriedem: i think the new cffi release caught a bunch of projects off guard who added bindep.txt and didn't realize that they didn't include enough in it to build sdists of everything because they were testing with prebuilt wheels | 17:37 |
| mriedem | fungi: yup | 17:37 |
| *** thorst has quit IRC | 17:38 | |
| *** yamahata has joined #openstack-infra | 17:38 | |
| mriedem | fungi: sdague: https://review.openstack.org/#/c/366885/ | 17:38 |
| *** mrjk has joined #openstack-infra | 17:38 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add libffi-dev(el) to bindep https://review.openstack.org/366885 | 17:41 |
| *** Sukhdev has joined #openstack-infra | 17:41 | |
| *** rbrndt_ has quit IRC | 17:42 | |
| *** harlowja has quit IRC | 17:46 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for cells v1 snapshot race bug 1620761 https://review.openstack.org/366277 | 17:47 |
| openstack | bug 1620761 in OpenStack Compute (nova) "test_create_second_image_when_first_image_is_being_saved intermittently times out in teardown in cells v1 job" [Undecided,Confirmed] https://launchpad.net/bugs/1620761 | 17:47 |
| *** csomerville has joined #openstack-infra | 17:48 | |
| *** harlowja has joined #openstack-infra | 17:49 | |
| *** vhosakot has quit IRC | 17:49 | |
| anteaya | fungi: 87%, I had no idea | 17:49 |
| *** piet has joined #openstack-infra | 17:50 | |
| *** kzaitsev_mb has quit IRC | 17:50 | |
| *** sandanar_ has joined #openstack-infra | 17:50 | |
| *** dahoo has quit IRC | 17:50 | |
| anteaya | I too thought the release cadence represented the majority | 17:50 |
| *** vhosakot has joined #openstack-infra | 17:50 | |
| *** sridhar_ram is now known as sridhar_ram_afk | 17:52 | |
| *** flepied has joined #openstack-infra | 17:52 | |
| *** ajo_ has quit IRC | 17:53 | |
| *** eranrom has quit IRC | 17:53 | |
| *** ajo_ has joined #openstack-infra | 17:53 | |
| *** sandanar has quit IRC | 17:53 | |
| *** eranrom has joined #openstack-infra | 17:54 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for cells v1 snapshot race bug 1620761 https://review.openstack.org/366277 | 17:55 |
| openstack | bug 1620761 in OpenStack Compute (nova) "test_create_second_image_when_first_image_is_being_saved intermittently times out in teardown in cells v1 job" [Undecided,Confirmed] https://launchpad.net/bugs/1620761 | 17:55 |
| *** sambetts is now known as sambetts|afk | 17:55 | |
| *** trown|lunch is now known as trown | 17:56 | |
| *** mandre has quit IRC | 17:57 | |
| *** _nadya_ has quit IRC | 17:57 | |
| fungi | anteaya: it's probably a misleading statistic, but "release:cycle" appears ~23% as often in the projects.yaml as "repos:" | 17:58 |
| anteaya | fungi: thanks for quantifying your metrics | 17:59 |
| *** Sukhdev has quit IRC | 17:59 | |
| *** inc0 has quit IRC | 17:59 | |
| *** inc0 has joined #openstack-infra | 18:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Change Kolla jobs to 90 minutes for short term https://review.openstack.org/366859 | 18:04 |
| AJaeger | mriedem: your change looks fine. | 18:05 |
| *** sarob has joined #openstack-infra | 18:05 | |
| *** mandre has joined #openstack-infra | 18:05 | |
| *** sarob has quit IRC | 18:05 | |
| *** dahoo has joined #openstack-infra | 18:05 | |
| *** sarob has joined #openstack-infra | 18:06 | |
| *** sarob has quit IRC | 18:09 | |
| *** vhosakot has quit IRC | 18:09 | |
| openstackgerrit | Dong Ma proposed openstack-infra/jenkins-job-builder: Update clone-workspace plugin https://review.openstack.org/366894 | 18:10 |
| *** matt-borland has quit IRC | 18:10 | |
| *** jcoufal has joined #openstack-infra | 18:10 | |
| openstackgerrit | Eran Rom proposed openstack-infra/irc-meetings: Requested slot for storlet dev meetings on IRC https://review.openstack.org/366895 | 18:12 |
| *** rbrndt has joined #openstack-infra | 18:12 | |
| *** mhickey__ has quit IRC | 18:12 | |
| *** ociuhandu has joined #openstack-infra | 18:13 | |
| *** akshai has joined #openstack-infra | 18:13 | |
| *** csomerville has quit IRC | 18:14 | |
| *** acoles is now known as acoles_ | 18:14 | |
| *** sandanar_ has quit IRC | 18:15 | |
| *** watanabe_isao has quit IRC | 18:16 | |
| *** sarob has joined #openstack-infra | 18:18 | |
| *** matt-borland has joined #openstack-infra | 18:18 | |
| *** e0ne has joined #openstack-infra | 18:19 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenarios: set Debug to True https://review.openstack.org/366896 | 18:19 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenario001: deploy Ceph https://review.openstack.org/366810 | 18:21 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Rework the task list layout https://review.openstack.org/357306 | 18:23 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add a way to add tasks to worklists from the story view https://review.openstack.org/366899 | 18:23 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 18:26 |
| *** ddieterly[away] is now known as ddieterly | 18:26 | |
| *** zul has quit IRC | 18:27 | |
| patrickeast | hey, do we use multi-nic xenial vm's in the gate anywhere? a quick check of the upstream nodepool.yaml didn't show any, but figured i'd ask too | 18:30 |
| clarkb | we do in rax | 18:30 |
| clarkb | thats the only cloud iirc | 18:31 |
| patrickeast | oh cool, have you guys run into any problems with the order that the nic's are brought up in? | 18:31 |
| *** dkehn__ has quit IRC | 18:31 | |
| *** dkehn_ has quit IRC | 18:31 | |
| *** dkehn has quit IRC | 18:31 | |
| clarkb | no we dont actually care and xenial usesbiosname now I think | 18:31 |
| patrickeast | im setting up jobs on my 3rd party ci, but getting random ordering and the default gateway is a crapshoot :/ | 18:31 |
| clarkb | ens3 etc | 18:31 |
| patrickeast | gotcha | 18:31 |
| AJaeger | fungi, what do you think of https://review.openstack.org/352575 for zuul-cloner in post jobs? Ready to merge now? | 18:32 |
| clarkb | patrickeast: use dhcp and dont worry? | 18:32 |
| *** pfallenop has quit IRC | 18:32 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Complex priorities UI in stories https://review.openstack.org/312666 | 18:32 |
| patrickeast | clarkb: it is, i'm not really sure why neutron/dnsmasq are doing this | 18:32 |
| patrickeast | clarkb: whichever nic comes up first gets the default gateway | 18:32 |
| patrickeast | clarkb: and the metadata server route | 18:33 |
| AJaeger | team, pabelanger has a first change to replace jenkins user with zuul: https://review.openstack.org/#/c/337821/ . Is it safe to merge this now? | 18:33 |
| clarkb | patrickeast: huh thst seems like a neutron issue maybe.that should be basedon the network not the nic | 18:33 |
| patrickeast | clarkb: yea, either that or just some weirdness in how i've set things up | 18:34 |
| patrickeast | going to try setting a 'supersede routers' entry in the nodepool image's dhclient.conf to see if that will work around it for now | 18:34 |
| patrickeast | clarkb: good to know that its not a normal thing though, gives some direction on looking for a fix | 18:35 |
| AJaeger | is there any infra-root that can force-push a repo? See https://review.openstack.org/365047 . deb-websockify repo should be fixed and I'd like to see the the broken setup for that repo fixed, please | 18:35 |
| *** diltram has left #openstack-infra | 18:35 | |
| *** ddieterly is now known as ddieterly[away] | 18:35 | |
| *** sridhar_ram_afk is now known as sridhar_ram | 18:37 | |
| AJaeger | Here's one review request for system-config: https://review.openstack.org/#/c/365615/ - for publishing user committee content on governance.openstack.org. team, please review | 18:37 |
| anteaya | patrickeast: hello, can I ask you to use team, group, people or folks instead of guys please? | 18:37 |
| anteaya | patrickeast: i know you mean to include me when you address the channel | 18:37 |
| patrickeast | anteaya: of course, didn't mean to exclude, just a bad habbit | 18:37 |
| patrickeast | anteaya: thanks for the reminder :) | 18:38 |
| *** thorst has joined #openstack-infra | 18:39 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: Enable QoS on neutron for python-openstackclient. https://review.openstack.org/366719 | 18:39 |
| *** goneri_ has joined #openstack-infra | 18:39 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: Enable QoS on neutron for python-openstackclient. https://review.openstack.org/366719 | 18:40 |
| fungi | AJaeger: sorry, trying to finish up this puppet module for mediawiki right now and not get distracted | 18:40 |
| anteaya | patrickeast: thank you :) | 18:40 |
| *** aviau_ has quit IRC | 18:40 | |
| *** aviau has joined #openstack-infra | 18:40 | |
| *** goneri_ has quit IRC | 18:41 | |
| AJaeger | fungi sure. I can ask again later... | 18:41 |
| stevemar | AJaeger: i think that one makes sense now ^ | 18:41 |
| *** ddieterly[away] is now known as ddieterly | 18:41 | |
| * AJaeger checks | 18:41 | |
| *** wajdi has joined #openstack-infra | 18:41 | |
| *** abregman has joined #openstack-infra | 18:41 | |
| fungi | any puppet gurus want to tell me why not passing a "type" parameter to the mediawiki::extension type is causing $type to resolve to "Other" rather than the declared default there? http://git.openstack.org/cgit/openstack-infra/puppet-mediawiki/tree/manifests/extension.pp#n2 the reserved words list in the puppet language reference does say type is a reserved word but not a reserved parameter/variable | 18:42 |
| fungi | name | 18:42 |
| AJaeger | stevemar: use ENABLED_SERVICES | 18:42 |
| *** amitgandhinz has quit IRC | 18:42 | |
| stevemar | AJaeger: cool | 18:42 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/project-config: Ensure that gerrit projects have zuul pipelines https://review.openstack.org/363969 | 18:42 |
| *** amitgandhinz has joined #openstack-infra | 18:43 | |
| AJaeger | stevemar: let me double check with sdague first | 18:43 |
| *** pvaneck has joined #openstack-infra | 18:43 | |
| *** thorst has quit IRC | 18:43 | |
| *** jcoufal_ has joined #openstack-infra | 18:44 | |
| AJaeger | sdague: could your review https://review.openstack.org/366719, please? I think they should use ENABLED_SERVICE - but I see the same line in jenkins/jobs/sdk.yaml as well and like to see your comment on it, please | 18:44 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 18:44 |
| clarkb | fungi: i am guessing you cant use type there then | 18:44 |
| clarkb | fungi: but thats a guess | 18:44 |
| *** dkehn has joined #openstack-infra | 18:44 | |
| fungi | it's also my guess, but *sigh* | 18:44 |
| fungi | i'll do some local testing | 18:44 |
| mgagne | does $type fact exists for some reason? | 18:44 |
| *** dkehn__ has joined #openstack-infra | 18:45 | |
| mtreinish | mordred, fungi: do you have any pointers on the exim config for lpmqtt? | 18:45 |
| *** jcoufal has quit IRC | 18:45 | |
| mtreinish | I started looking at what storyboard does as an example, but it was all basically the exim config syntax which I don't really know | 18:45 |
| fungi | mgagne: oh, that's possible... i wonder if this just needs tighter scoping | 18:45 |
| mtreinish | so I'm not sure how to best add that to the firehose.pp to add the account for lpmqtt | 18:45 |
| fungi | mtreinish: also the storyboard use of an mta is a bit inverted from firehose's | 18:45 |
| mtreinish | heh, yeah there is that too | 18:46 |
| fungi | sb wants to be able to send e-mail notifications from a local user/process to the internet, fh wants to receive messages coming from a host/site on the internet and deliver them locally | 18:46 |
| *** abregman is now known as abregman|nb | 18:46 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 18:47 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: Enable QoS on neutron for python-openstackclient functional tests https://review.openstack.org/366719 | 18:47 |
| mtreinish | fungi: if I read the config in sb correctly it does accept messages that come in over lo | 18:47 |
| stevemar | AJaeger: done, and cleaned up the commit message | 18:47 |
| mtreinish | but that's irrelevant to your point | 18:47 |
| *** dkehn_ has joined #openstack-infra | 18:48 | |
| fungi | mtreinish: most of my exim configuring is done via debian's templated config parameters for their exim package, so i'd have to spend some time digging into how our puppet module works with configuration to know what to suggest | 18:49 |
| *** ilyashakhat_mobi has quit IRC | 18:49 | |
| *** Benj_ has joined #openstack-infra | 18:49 | |
| zigo | mordred: Please, I need your approval as PTL, otherwise AJaeger will not +2A... https://review.openstack.org/366337 | 18:49 |
| mtreinish | fungi: heh, we should just use the test gmail account I setup (which is already subscribed to lp openstack) and call it a day :) | 18:50 |
| *** wajdi has quit IRC | 18:50 | |
| *** degorenko is now known as _degorenko|afk | 18:50 | |
| AJaeger | stevemar: looks fine now - could you change sdk.yaml the same way, please? | 18:51 |
| *** Apoorva_ has joined #openstack-infra | 18:51 | |
| * AJaeger checked devstack and it should be fine indeed | 18:51 | |
| fungi | clarkb: mgagne: oh, i bet part of the issue is that $type is being used in the default value for $source which won't work as they're both resource parameters. probably need to default source to undef and then test for that in the resource definition and replace it with our actual desired default? | 18:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add support for jmespath filter expressions https://review.openstack.org/366906 | 18:53 |
| fungi | because the errors i'm seeing, $type is resolving correctly where it's used in the local path name for the vcsrepo destination | 18:53 |
| fungi | just not where it's being substituted in the $source default value | 18:54 |
| mgagne | how about a smaller manifest with $type and you output that variable? | 18:54 |
| mgagne | which version of Puppet? | 18:54 |
| fungi | mgagne: i'm a little fuzzy on what a smaller manifest for type would look like | 18:55 |
| clarkb | fungi: oh ya you cant rely on interpolation of defaults in other defaults or even parameters used in other parameter defaults iirc | 18:55 |
| fungi | puppet 3.8.7 | 18:55 |
| *** Apoorva has quit IRC | 18:55 | |
| *** zul_ has quit IRC | 18:55 | |
| mgagne | define foorbar($type) { fail("type=${type}") } | 18:56 |
| mgagne | something like that | 18:56 |
| mgagne | what clarkb said ^ | 18:56 |
| *** tonytan4ever has quit IRC | 18:56 | |
| openstackgerrit | Dmitry Ukov proposed openstack-infra/project-config: Create fuel-nailgun-extension-day2 project https://review.openstack.org/366779 | 18:57 |
| fungi | i think i'll stick with my test for undef logic... seems a little less like opaque voodoo | 18:57 |
| *** jeet has joined #openstack-infra | 18:57 | |
| *** akshai has quit IRC | 18:59 | |
| *** hashar has joined #openstack-infra | 19:00 | |
| *** piet has quit IRC | 19:01 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Conditionally define the default extension source https://review.openstack.org/366911 | 19:01 |
| fungi | clarkb: mgagne: ^ i think that's it? | 19:02 |
| *** sdake has quit IRC | 19:02 | |
| *** pfallenop has joined #openstack-infra | 19:02 | |
| *** salv-orlando has quit IRC | 19:02 | |
| fungi | oh, minor typo in there | 19:04 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Conditionally define the default extension source https://review.openstack.org/366911 | 19:04 |
| *** senk has quit IRC | 19:05 | |
| *** Mary has joined #openstack-infra | 19:06 | |
| *** Mary is now known as Guest10847 | 19:06 | |
| *** _nadya_ has joined #openstack-infra | 19:07 | |
| *** _nadya_ has quit IRC | 19:08 | |
| fungi | oh, actually my local testing says i had it right the first time | 19:09 |
| *** annegentle has joined #openstack-infra | 19:09 | |
| *** clenimar has quit IRC | 19:10 | |
| *** sdake has joined #openstack-infra | 19:10 | |
| *** zul has joined #openstack-infra | 19:12 | |
| fungi | Cannot reassign variable source at /etc/puppet/modules/mediawiki/manifests/extension.pp:11 | 19:13 |
| fungi | grr | 19:13 |
| *** salv-orlando has joined #openstack-infra | 19:13 | |
| fungi | two variable names it is | 19:13 |
| *** derekh_afk is now known as derekh | 19:13 | |
| *** lucas-dinner has quit IRC | 19:13 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: cleanup sdk functional test definition https://review.openstack.org/366914 | 19:15 |
| AJaeger | thanks, stevemar ^ | 19:15 |
| *** lucasagomes has joined #openstack-infra | 19:15 | |
| *** Apoorva_ has quit IRC | 19:16 | |
| stevemar | AJaeger: np, theres a ceilometer one there too i think can be cleaned up... | 19:16 |
| *** Apoorva has joined #openstack-infra | 19:16 | |
| *** Sukhdev_ has joined #openstack-infra | 19:17 | |
| AJaeger | stevemar: where? I'm not seeing that - but I see some others ;( | 19:18 |
| stevemar | AJaeger: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/sdk.yaml#L28 | 19:18 |
| AJaeger | that ceilometer plugin might be fine | 19:19 |
| stevemar | AJaeger: yeah, wasn't sure, that's why i didn't touch :) | 19:20 |
| *** vhosakot has joined #openstack-infra | 19:20 | |
| AJaeger | ;) | 19:20 |
| *** rcernin has quit IRC | 19:21 | |
| *** ddieterly is now known as ddieterly[away] | 19:21 | |
| AJaeger | sdague, or any other devstack experts: Is the enable_service here really needed? http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n1937 | 19:24 |
| mordred | mtreinish: sorry - jeblair is out this week - he's the exim/cyrus expert | 19:25 |
| mordred | mtreinish: I honestly think we should loop him in before we get too far down the road | 19:25 |
| mordred | zigo: on it | 19:25 |
| mtreinish | mordred: sure, that's fine. I defer to corvus for all things email too | 19:26 |
| mordred | mtreinish: :) | 19:26 |
| mtreinish | mordred: I'm fine waiting for him. If we were doing postfix + dovecot I'd be in much better shape to have real input on how we do this | 19:27 |
| mtreinish | but even in that case waiting for corvus would still make sense | 19:28 |
| *** kzaitsev_mb has joined #openstack-infra | 19:28 | |
| mordred | yah | 19:29 |
| *** zul_ has joined #openstack-infra | 19:31 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Conditionally define the default extension source https://review.openstack.org/366911 | 19:31 |
| fungi | clarkb: mgagne: Krenair: okay, ^ that is tested and seems to be further along to working for me (no additional puppet failures when bootstrapping now) | 19:31 |
| fungi | i'm not sure if it's idempotent yet, but will test that next | 19:32 |
| *** tqtran has joined #openstack-infra | 19:32 | |
| mordred | fungi: lgtm | 19:33 |
| fungi | i do have one minor concern, which i think we ran into with etherpad, where managing additional untracked content in a vcsrepo clone causes vcsrepo to fail to update it | 19:34 |
| mordred | fungi: yah. I could see that being weird | 19:34 |
| fungi | though it may be that was only modifying tracked files which caused grief there | 19:34 |
| fungi | as untracked files shouldn't conflict with anything | 19:35 |
| Krenair | fungi, certainly could if a new commit introduced a file with the same name as an untracked one | 19:36 |
| openstackgerrit | Merged openstack-infra/project-config: Add missing deb repo 2/5 https://review.openstack.org/366337 | 19:37 |
| fungi | well, reapplying the (fixed per above) manifest at least doesn't fail. this is looking good so far | 19:37 |
| AJaeger | devstack-gate experts, sdague : Please review export ENABLED_SERVICES+=,g-glare | 19:37 |
| AJaeger | devstack-gate experts, sdague : Please review https://review.openstack.org/366920 - and confirm that this is the proper way | 19:38 |
| * AJaeger hates two ways of copy&paste ;( | 19:38 | |
| fungi | Krenair: it takes 65 seconds to reapply, but that's probably mostly due to performing origin remote updates to see whether there are any new commits to pull. i don't see any jumps of >2-3 seconds in the puppet apply log anyway | 19:39 |
| *** thorst has joined #openstack-infra | 19:39 | |
| Krenair | yes | 19:39 |
| Krenair | I'd check, but no access, so | 19:40 |
| Krenair | probably | 19:40 |
| clarkb | fungi: I think the issue with untracked files we had there was more to do with file permissions and such. So things churn a little but vcsrepo doesn't really care too much | 19:41 |
| fungi | well, once we have 366911 passing ci and merged (hopefully in a few minutes) i'll blow away this wiki-dev01 and launch another, make sure it completely deploys from scratch with no new issues, and then can proceed to adding dns and testing functionality | 19:41 |
| *** abregman|nb has quit IRC | 19:41 | |
| anteaya | yay | 19:41 |
| *** Hal has joined #openstack-infra | 19:42 | |
| openstackgerrit | Merged openstack-infra/release-tools: Change to checked-out directory https://review.openstack.org/364644 | 19:43 |
| *** ddieterly[away] is now known as ddieterly | 19:44 | |
| openstackgerrit | Augustina Ragwitz proposed openstack-infra/project-config: Don't run openstack/nova dsvm tests for .gitignore changes https://review.openstack.org/366933 | 19:45 |
| fungi | auggy: i'm curious what's going on in nova that it sees that much churn in .gitignore ;) | 19:46 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Make builder recognize scheduled builds https://review.openstack.org/366934 | 19:46 |
| *** Hal has quit IRC | 19:46 | |
| openstackgerrit | Augustina Ragwitz proposed openstack-infra/project-config: Don't run openstack/nova dsvm tests for .gitignore changes https://review.openstack.org/366933 | 19:46 |
| *** Hal has joined #openstack-infra | 19:46 | |
| *** dtardivel has quit IRC | 19:47 | |
| mtreinish | fungi: it doesn't look like that much: http://git.openstack.org/cgit/openstack/nova/log/.gitignore | 19:47 |
| auggy | fungi: we just had a dsvm failure on a patch that added something to gitignore and thought thoses resources should be wasted on that kind of change | 19:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean: Index networks using id, not links https://review.openstack.org/366937 | 19:47 |
| auggy | fungi: s/should/shouldn't/ | 19:48 |
| mtreinish | just adding new auto generated things to the list mostly | 19:48 |
| *** thorst has quit IRC | 19:48 | |
| fungi | fair enough. didn't know how frequent .gitignore changes in the nova repo were | 19:48 |
| auggy | fungi: yeah not that frequent, but like if we're at FF and trying to get things through the gate, better not to waste resources | 19:48 |
| pabelanger | mordred: clarkb: So, ^ is my first stab at using network ID as key for interfaces in glean, but for some reason /etc/hosts unitest is failing. Still trying to figure that out | 19:48 |
| anteaya | the -meeting-alt channel could use a topic refresh when someone has a moment | 19:49 |
| fungi | auggy: also it looks like that will filter changes for the .gitreview config file as well | 19:49 |
| pabelanger | I also plan on adding a test cause for rax-iad | 19:49 |
| fungi | anteaya: on it | 19:49 |
| anteaya | fungi: thank you | 19:49 |
| auggy | fungi: yeah that's the idea | 19:49 |
| pabelanger | using http://paste.openstack.org/show/567498/ | 19:49 |
| AJaeger | fungi, auggy: I don't think it's worth that. | 19:49 |
| mordred | pabelanger: looking | 19:49 |
| auggy | fungi: i should ammend the commit msg | 19:49 |
| AJaeger | auggy: I suggest to rather abandon it. It happens so infrequent that this is over-optimization IMHO. | 19:50 |
| AJaeger | auggy: I'll refuse to +2 that ;) | 19:50 |
| pabelanger | mordred: I am likely wrong in my approach, not familiar with the interworkings of glean yet | 19:50 |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Conditionally define the default extension source https://review.openstack.org/366911 | 19:50 |
| mordred | pabelanger: nobody is | 19:50 |
| *** jeet has left #openstack-infra | 19:50 | |
| rbergeron | (nobody except mordred :D) | 19:51 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Make builder recognize scheduled builds https://review.openstack.org/366934 | 19:51 |
| clarkb | mordred: pabelanger thats why every time I touch it I end up writing 3 or 4 "comment this section of code that really makes no sense at all so that future me doesn't hate past me" | 19:51 |
| *** annegentle has quit IRC | 19:51 | |
| auggy | AJaeger: what's the cost of having it? not having it increases gate load during critical release periods potetially | 19:51 |
| auggy | AJaeger: and could potentially impact a critical release not getting through | 19:52 |
| clarkb | really I think glean neesd to be like 75% comments 25% sadness | 19:52 |
| anteaya | clarkb: ha ha ha | 19:52 |
| openstackgerrit | Augustina Ragwitz proposed openstack-infra/project-config: Don't run nova dvsm tests for .git repo changes https://review.openstack.org/366933 | 19:52 |
| rcarrillocruz | mordred, pabelanger : mind checking https://review.openstack.org/#/c/366723/ ? look at the dsvm job logs, USEFUL output now | 19:53 |
| *** tonytan4ever has joined #openstack-infra | 19:53 | |
| AJaeger | auggy: I cannot see that .gitignore changes are release critical. | 19:53 |
| auggy | AJaeger: i'm not referring to the .gitignore changes not getting through | 19:53 |
| auggy | AJaeger: i'm referring to other patches in queue behind it | 19:53 |
| auggy | AJaeger: sure, feel free to leave your comments on the patch and we can discuss it there | 19:53 |
| AJaeger | auggy: seeing that you had *three* commits for .gitignore in 2016, this review costs us more than any infra churn. | 19:54 |
| fungi | AJaeger: actually, pbr does i think rely on .gitignore having a sane list of things it shouldn't package up into the sdist/wheel | 19:54 |
| pabelanger | mordred: clarkb: Ya, if I can figure out why the hosts files fail on mock, next patch will be a bunch of comments and rax-iad test | 19:54 |
| * AJaeger has seen to many broken skip-ifs that needed hunting down that he considers this change more risky than the next .gitignore change that most likely will come in three months time... | 19:55 | |
| AJaeger | auggy: I'm not blocking it, I'm just considering it a waste and refuse to review it further. | 19:55 |
| mat128 | +1 ^ | 19:55 |
| mat128 | skip-ifs are evil :) | 19:55 |
| auggy | AJaeger: great, those are valid reasons why you don't like it, so lets have that discussion in the review | 19:55 |
| fungi | yeah, i'm still of the opinion skip-if is an anti-feature, but it's a feature we accepted into zuul and it's in use | 19:56 |
| AJaeger | auggy: I don't see why I need to repeat that discussion there. I suggest to leave it out and will stay out of further discussion on this. | 19:56 |
| clarkb | pabelanger: so basically in that change you are associating every network with its correct interface then alter when we write out the network configs we should get the correct bits? | 19:57 |
| *** Hal has quit IRC | 19:57 | |
| auggy | AJaeger: generally that's where patch related discussion happens so people who care about the change get the opportunity to participate | 19:57 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 19:57 |
| auggy | AJaeger: on irc, especially in a busy channel like infra, it's easy to only hear one side of the story | 19:57 |
| AJaeger | auggy: I left a comment and as I said: I'm not blocking it. | 19:57 |
| clarkb | pabelanger: making sure I grok why this helps. Not entirely sure how network[id] differs from i at that point. oh i is just an index onto the networks array got it | 19:58 |
| auggy | AJaeger: thanks :) | 19:58 |
| *** Apoorva_ has joined #openstack-infra | 19:58 | |
| AJaeger | auggy: I left the comment 10 minutes ago;) | 19:58 |
| fungi | auggy: there is a general sentiment that a number of project-config core reviewers just skip any skip-if changes in layout.yaml because they're generally concerned about becoming responsible for approving a change that causes important jobs to be skipped for some project. it will likely eventually get reviews from those who don't mind | 19:58 |
| *** Hal has joined #openstack-infra | 19:58 | |
| auggy | fungi: sure, i definitely understand the hesitancy | 19:59 |
| pabelanger | clarkb: the motivation is http://paste.openstack.org/show/567498/, by default we are using link as our index for networks. Which in rax-iad, does work because we have 2 different networks (ipv4/ipv6) using the same link: tap8056a42d-1c | 19:59 |
| auggy | fungi: i think folks brought up some good points and i just want to make sure it's captured in the review comments rather than potentially lost in discussion here | 19:59 |
| pabelanger | clarkb: which causes rax-iad to only provision ipv6 for eth0.cfg, when it should be doing both ipv4 / ipv6 | 19:59 |
| clarkb | pabelanger: re the hosts file not passing tests I think thats because its probably setting the hosts entry IP addr to the ipv4 or ipv6 address now (and properly I think) | 20:01 |
| *** sshnaidm is now known as sshnaidm|afk | 20:01 | |
| clarkb | pabelanger: if you look at ~ line 892 of cmd.py you can see where it sets that based on the interfaces data | 20:01 |
| clarkb | pabelanger: it runs a sort there and since your keys are changing the sort order may change | 20:01 |
| *** dprince has quit IRC | 20:01 | |
| *** Apoorva has quit IRC | 20:01 | |
| clarkb | pabelanger: gotcha and so code in this case is changing to the network id. makes sense | 20:02 |
| clarkb | as for that test I think you just need to update the test to match the new sort order | 20:02 |
| pabelanger | okay, let me dig into that more | 20:02 |
| clarkb | pabelanger: but also add a comment explaining why we index that way "multiple networks may map onto the same link so we use the network as the key instead of the link" | 20:04 |
| *** gyee_ has quit IRC | 20:04 | |
| *** _nadya_ has joined #openstack-infra | 20:04 | |
| clarkb | otherwise the data there is way to obtuse and how to use it properly is like magic | 20:04 |
| *** openstackgerrit has quit IRC | 20:04 | |
| *** openstackgerrit has joined #openstack-infra | 20:04 | |
| *** spzala has quit IRC | 20:05 | |
| *** spzala has joined #openstack-infra | 20:06 | |
| *** notnownikki has joined #openstack-infra | 20:06 | |
| clarkb | pabelanger: then when you have that sorted maybe you acn review https://review.openstack.org/#/c/366434/ to address the other image building issue | 20:06 |
| pabelanger | clarkb: sure, give me a min | 20:07 |
| *** esberglu has joined #openstack-infra | 20:09 | |
| openstackgerrit | Dmitry Ukov proposed openstack-infra/project-config: Create fuel-nailgun-extension-day2 project https://review.openstack.org/366779 | 20:09 |
| *** piet has joined #openstack-infra | 20:10 | |
| *** _nadya_ has quit IRC | 20:10 | |
| *** notnownikki has quit IRC | 20:10 | |
| *** piet has quit IRC | 20:10 | |
| *** spzala has quit IRC | 20:10 | |
| *** salv-orlando has quit IRC | 20:10 | |
| pabelanger | clarkb: Yup, that is the issue. Removing the sort causes the test to bounce. Making updates now | 20:10 |
| *** piet has joined #openstack-infra | 20:12 | |
| *** rfolco has quit IRC | 20:15 | |
| * AJaeger waves good night | 20:16 | |
| anteaya | AJaeger: good night | 20:16 |
| rcarrillocruz | infra-root, please leave the infracloud controller on emergency hosts. I've put debug logging on neutron server, linuxbridge plugin and dhcp agent to see if we get more info about those connect timeouts | 20:16 |
| clarkb | I am going to boot a xenial test machien in osic to test some more neutron job updates | 20:16 |
| anteaya | AJaeger: thanks for the day, enjoy your evening | 20:16 |
| fungi | rcarrillocruz: i recommend you #status log that too | 20:16 |
| fungi | if you haven't already | 20:17 |
| rcarrillocruz | i have not | 20:17 |
| *** salv-orlando has joined #openstack-infra | 20:18 | |
| rcarrillocruz | #status controller is put on emergency hosts, as neutron debugging has been tweaked to investigate sporadic connect timeouts, please leave as is till we get more errors on logs | 20:19 |
| openstackstatus | rcarrillocruz: unknown command | 20:19 |
| rcarrillocruz | lol | 20:19 |
| rcarrillocruz | notice ? | 20:19 |
| rcarrillocruz | i knew i'd fail , hahaha | 20:19 |
| fungi | it's #status log | 20:20 |
| rcarrillocruz | or hash statuslog | 20:20 |
| rcarrillocruz | ok | 20:20 |
| fungi | followed by your content | 20:20 |
| rcarrillocruz | #statuslog controller00 of infracloud is put on emergency hosts, as neutron debugging has been tweaked to investigate sporadic connect timeouts, please leave as is till we get more errors on logs | 20:20 |
| openstackstatus | rcarrillocruz: unknown command | 20:20 |
| *** kzaitsev_mb has quit IRC | 20:21 | |
| rcarrillocruz | third attempt | 20:21 |
| fungi | you left out the word "log" again | 20:21 |
| rcarrillocruz | #status log controller00 of infracloud is put on emergency hosts, as neutron debugging has been tweaked to investigate sporadic connect timeouts, please leave as is till we get more errors on logs | 20:21 |
| openstackstatus | rcarrillocruz: finished logging | 20:21 |
| rcarrillocruz | \o/ | 20:21 |
| fungi | perfect | 20:21 |
| * rcarrillocruz feels less dumber now | 20:21 | |
| rcarrillocruz | thx fungi | 20:22 |
| *** e0ne has quit IRC | 20:22 | |
| fungi | any time! and don't feel dumb, but you probably do want to check out (maybe even improve) our statusbot documentation | 20:22 |
| rcarrillocruz | ++ | 20:22 |
| *** pblaho has quit IRC | 20:22 | |
| rcarrillocruz | clarkb: in case you didn't see with channel traffic: https://review.openstack.org/#/c/366636/ | 20:23 |
| rcarrillocruz | we merged it | 20:23 |
| rcarrillocruz | that fixes the default 'default' sec groups, removes them | 20:23 |
| *** tphummel has joined #openstack-infra | 20:24 | |
| openstackgerrit | Dmitry Ukov proposed openstack-infra/project-config: Create fuel-nailgun-extension-iac project https://review.openstack.org/366779 | 20:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean: Index networks using id, not links https://review.openstack.org/366937 | 20:25 |
| *** vhosakot has quit IRC | 20:25 | |
| *** kgiusti has left #openstack-infra | 20:25 | |
| *** derekh has quit IRC | 20:25 | |
| pabelanger | clarkb: mordred: okay, that updates glean to use network id for the key index. I'll do a follow up patch to deal with rax-iad issue next | 20:26 |
| clarkb | rcarrillocruz: ty | 20:26 |
| pabelanger | clarkb: 366434 +3 | 20:26 |
| clarkb | pabelanger: thanks, I will try to get the nodepool daemon restarted today after it kills todays uploads | 20:27 |
| *** ilyashakhat_mobi has joined #openstack-infra | 20:27 | |
| *** megm_ has quit IRC | 20:28 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 4/5 https://review.openstack.org/366734 | 20:28 |
| *** salv-orl_ has joined #openstack-infra | 20:29 | |
| pabelanger | clarkb: nice | 20:30 |
| *** Goneri has quit IRC | 20:30 | |
| *** inc0 has quit IRC | 20:30 | |
| *** megm has joined #openstack-infra | 20:31 | |
| *** piet has quit IRC | 20:32 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:32 | |
| *** salv-orlando has quit IRC | 20:33 | |
| *** sarob has quit IRC | 20:33 | |
| *** piet has joined #openstack-infra | 20:34 | |
| *** sarob has joined #openstack-infra | 20:35 | |
| *** thorst has joined #openstack-infra | 20:35 | |
| openstackgerrit | Merged openstack-infra/nodepool: Don't delete images in a building state https://review.openstack.org/366434 | 20:36 |
| *** tphummel has quit IRC | 20:36 | |
| *** sarob has quit IRC | 20:36 | |
| *** sarob has joined #openstack-infra | 20:37 | |
| cloudnull | ping, infra folks, how would i go about removing this test from a https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L11885-L11886 specific branch? | 20:38 |
| *** piet has quit IRC | 20:38 | |
| cloudnull | IE: I dont want that run on stable/mitaka ? | 20:38 |
| *** askb has joined #openstack-infra | 20:39 | |
| clarkb | cloudnull: earlier on in the layout file is a big list of job specific flags like voting and not voting and which branches to run things on | 20:39 |
| clarkb | cloudnull: you can add a rule for your jobs there that remove them from mitaka using a negative lookahead regex (should be plenty of examples in the config) | 20:39 |
| cloudnull | great thanks clarkb | 20:40 |
| *** asettle has joined #openstack-infra | 20:40 | |
| *** thorst has quit IRC | 20:42 | |
| *** ilyashakhat_mobi has quit IRC | 20:42 | |
| *** ociuhandu has quit IRC | 20:43 | |
| vgridnev | hello infra team, is there some kind of example of post job that will be working only after gate passes? | 20:44 |
| *** Guest10847 has quit IRC | 20:45 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 5/5 https://review.openstack.org/366960 | 20:46 |
| clarkb | vgridnev: yes there are many jobs like that that publish documentation, build sdist tarballs and so on | 20:46 |
| clarkb | vgridnev: by definition the post jobs only run after merging happens | 20:46 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: fix logic for adding new release notes pages https://review.openstack.org/366962 | 20:46 |
| *** jcoufal_ has quit IRC | 20:48 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Support disallowing robots https://review.openstack.org/366964 | 20:48 |
| *** vhosakot has joined #openstack-infra | 20:48 | |
| *** tphummel has joined #openstack-infra | 20:48 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Disallow robots for wiki-dev site https://review.openstack.org/366965 | 20:48 |
| *** Apoorva_ has quit IRC | 20:49 | |
| clarkb | vgridnev: is there something specific you are looking for? | 20:49 |
| *** gongysh has joined #openstack-infra | 20:49 | |
| *** Apoorva has joined #openstack-infra | 20:50 | |
| *** rtheis has joined #openstack-infra | 20:50 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack-infra/project-config: Remove LXD tests from unsupported branches https://review.openstack.org/366968 | 20:50 |
| *** jkilpatr has quit IRC | 20:51 | |
| *** piet has joined #openstack-infra | 20:51 | |
| vgridnev | clarkb, I'd like to setup a job which well be running in check + gate + post pipelines, but only in post that will publish stuff to tarballs.openstack.org | 20:51 |
| *** ociuhandu has joined #openstack-infra | 20:51 | |
| *** tphummel has quit IRC | 20:51 | |
| *** piet has quit IRC | 20:51 | |
| clarkb | vgridnev: you will need two jobs then, one for check and gate without a publish step and one with a publish step for post. The doc jobs work this way if you want a concrete example | 20:52 |
| *** rtheis_ has quit IRC | 20:52 | |
| vgridnev | I got you. thanks. Is there requirements on files that can stored in tarballs.openstack.org ? | 20:53 |
| *** jcoufal has joined #openstack-infra | 20:54 | |
| fungi | vgridnev: we mostly like it to be actual tarballs, or other installable release artifacts | 20:54 |
| clarkb | vgridnev: also we ask that if you upload an artifact post merge that we don't keep the artifact for every merge and instead overwrite then possibly keep them for each release tag | 20:54 |
| *** tphummel has joined #openstack-infra | 20:55 | |
| fungi | Krenair: i ran a couple more deployment tests. one failed around 13 minutes in (likely choked on a network issue cloning an extension, i didn't have it set to keep the instance so i don't know), second try successfully deployed in just over 16 minutes wall clock time | 20:56 |
| nikhil | AJaeger: heya, any clue why the python-glanceclient release page isn't showing up http://docs.openstack.org/releasenotes/python-glanceclient ? we've got a note :) https://github.com/openstack/python-glanceclient/tree/master/releasenotes/notes | 20:56 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 20:57 |
| *** trown is now known as trown|outtypewww | 20:57 | |
| vgridnev | fungi, actually I'd like to setup storing of few jars files builds at current moment (for sahara-extra repository). For future Plus I was thinking about storing of images for few important plugins in sahara | 20:57 |
| *** _sarob has joined #openstack-infra | 20:57 | |
| fungi | vgridnev: yeah, that sounds like the sort of thing we expect to end up on tarballs.o.o | 20:58 |
| vgridnev | so, images also can be stored right there? | 20:58 |
| *** acoles_ is now known as acoles | 20:58 | |
| vgridnev | few of them a heavy | 20:58 |
| fungi | vgridnev: we have war and jpi files we upload there for releases of some java-based projects | 20:58 |
| vgridnev | few of them are heavy | 20:58 |
| *** cardeois has quit IRC | 20:58 | |
| fungi | images are a grey area. i think some of the service vm projects generate and upload versions of lightweight images there which they then reuse in other jobs | 20:59 |
| clarkb | "lightweight" :) | 21:00 |
| fungi | it would probably depend a lot on how large the images are, how often you update them, how many copies you expect to keep there, and how often they're retrieved by other jobs | 21:00 |
| *** piet has joined #openstack-infra | 21:00 | |
| fungi | as that impacts disk space and network bandwidth | 21:00 |
| *** sarob has quit IRC | 21:01 | |
| mordred | clarkb, pabelanger: https://review.openstack.org/#/c/366294/ - just bumps the shade requirement for nodepool to be accurate | 21:01 |
| clarkb | mordred: didn't you make a 1.12.1 afterwards? | 21:03 |
| clarkb | or maybe that was glean | 21:03 |
| * clarkb can't keep up | 21:03 | |
| mordred | clarkb: yah - I did - but the requirement of 1.12.0 is fine - it'll just always get 1.12.1 | 21:04 |
| mordred | unless you want it updated | 21:04 |
| mordred | (1.12 wasn't broken - the release job was for a second) | 21:04 |
| vgridnev | we have 3 plugins that have image size around 1G for centos7 and ubuntu. 2 plugins with image size 3-4GB, also centos7 + ubuntu. So, in total 6 * 1 + 4 * 4 = 22GB in total | 21:04 |
| clarkb | ah thats what it was ok | 21:04 |
| *** _sarob has quit IRC | 21:05 | |
| openstackgerrit | Merged openstack-infra/nodepool: Remove image-update cron https://review.openstack.org/363837 | 21:06 |
| *** kien-ha has joined #openstack-infra | 21:07 | |
| *** kien-ha has quit IRC | 21:08 | |
| *** ociuhandu has quit IRC | 21:08 | |
| openstackgerrit | Merged openstack-infra/nodepool: Replace watch thread with periodic worker threads https://review.openstack.org/363217 | 21:09 |
| *** krtaylor has quit IRC | 21:11 | |
| openstackgerrit | Dean Troyer proposed openstack-infra/project-config: Create python-openstackclient-stable-maint group https://review.openstack.org/366989 | 21:11 |
| openstackgerrit | Merged openstack-infra/nodepool: Make builder workers proper threads https://review.openstack.org/364607 | 21:13 |
| openstackgerrit | Merged openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers https://review.openstack.org/364617 | 21:13 |
| openstackgerrit | Merged openstack-infra/nodepool: Use time.sleep() when waiting for workers to run https://review.openstack.org/365075 | 21:13 |
| *** mdrabe has quit IRC | 21:14 | |
| *** mdrabe has joined #openstack-infra | 21:14 | |
| *** rtheis_ has joined #openstack-infra | 21:14 | |
| *** mmedvede has quit IRC | 21:15 | |
| *** sarob has joined #openstack-infra | 21:15 | |
| *** rhallisey has quit IRC | 21:15 | |
| *** rtheis has quit IRC | 21:16 | |
| *** jkilpatr has joined #openstack-infra | 21:17 | |
| *** matt-borland has quit IRC | 21:17 | |
| *** rossella_s has quit IRC | 21:18 | |
| *** rossella_s has joined #openstack-infra | 21:19 | |
| *** edmondsw has quit IRC | 21:19 | |
| *** adrian_otto has quit IRC | 21:21 | |
| *** vhosakot has quit IRC | 21:24 | |
| *** jheroux has quit IRC | 21:27 | |
| *** gyee_ has joined #openstack-infra | 21:29 | |
| *** sarob has quit IRC | 21:30 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add missing deb repo 5/5 https://review.openstack.org/366960 | 21:31 |
| openstackgerrit | Chris Holcombe proposed openstack-infra/project-config: Add Ceph Charm Shared Lib https://review.openstack.org/366842 | 21:34 |
| *** piet has quit IRC | 21:35 | |
| *** rtheis_ has quit IRC | 21:37 | |
| pabelanger | mordred: clarkb: okay, have rax-iad working locally in glean for ubuntu / debian. Still need to get syntax for other OSes working, so expect something for tomorrow | 21:39 |
| mordred | pabelanger: sweet! that's exciting | 21:39 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 21:39 |
| *** tphummel has quit IRC | 21:39 | |
| mordred | pabelanger: when that's good, we can also land the nodepool patch that will cause it to prefer ipv6 instead of needing to have ipv6 enabled per provider | 21:39 |
| pabelanger | cool | 21:40 |
| *** thorst has joined #openstack-infra | 21:40 | |
| *** tphummel has joined #openstack-infra | 21:40 | |
| pabelanger | okay, have to run out for some errands | 21:40 |
| *** tphummel has quit IRC | 21:40 | |
| *** Mary has joined #openstack-infra | 21:40 | |
| *** Mary is now known as Guest46455 | 21:41 | |
| *** mmedvede has joined #openstack-infra | 21:41 | |
| *** mriedem has quit IRC | 21:41 | |
| *** claudiub|2 has quit IRC | 21:42 | |
| *** asettle has quit IRC | 21:43 | |
| *** Hal has quit IRC | 21:44 | |
| openstackgerrit | Merged openstack-infra/nodepool: Update shade depend to 1.12.0 https://review.openstack.org/366294 | 21:44 |
| *** Guest46455 has quit IRC | 21:45 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Disallow robots for wiki-dev site https://review.openstack.org/366965 | 21:45 |
| *** Hal has joined #openstack-infra | 21:45 | |
| *** thorst has quit IRC | 21:48 | |
| *** Swami has quit IRC | 21:48 | |
| *** sarob has joined #openstack-infra | 21:49 | |
| gongysh | hi, who knows where is the source code maintained to generate the status.openstack.org? | 21:49 |
| clarkb | gongysh: its several code bases. Which one are you interested in? | 21:50 |
| clarkb | s/which one/which section or tab/ | 21:50 |
| gongysh | clarkb, zuul one, please | 21:50 |
| clarkb | gongysh: its in openstack-infra/system-config/modules/openstack_project/templates/zuul | 21:51 |
| gongysh | clarkb, it seems to be good stuff since it is hid so deeply. :) | 21:52 |
| clarkb | gongysh: zuul has its own status stuff in tree as well but we don't use it yet | 21:52 |
| zigo | pabelanger: FYI, I've switch from httpredir.debian.org to ftp.fr.debian.org, as I get too many failures to fetch packages otherwise. | 21:52 |
| clarkb | it is still a work in progress I think? not sure why we haven't switched to the in tree code | 21:52 |
| zigo | pabelanger: I had probably 20 of them today, that's really annoying. | 21:52 |
| *** jmccrory has quit IRC | 21:53 | |
| *** krtaylor has joined #openstack-infra | 21:53 | |
| *** jmccrory has joined #openstack-infra | 21:53 | |
| *** sarob has quit IRC | 21:54 | |
| *** spzala has joined #openstack-infra | 21:54 | |
| zigo | mordred: Can you have a look into this one now? It should be enough time to avoid flooding github ... https://review.openstack.org/366352 | 21:54 |
| *** tphummel has joined #openstack-infra | 21:54 | |
| *** jmccrory has quit IRC | 21:55 | |
| *** adrian_otto has joined #openstack-infra | 21:55 | |
| *** jmccrory has joined #openstack-infra | 21:55 | |
| *** hashar has quit IRC | 21:56 | |
| *** jmccrory has quit IRC | 21:57 | |
| *** jmccrory has joined #openstack-infra | 21:57 | |
| *** aeng has quit IRC | 21:58 | |
| gongysh | clarkb, where is the zuul in place code to generate the status web page? | 22:00 |
| gongysh | clarkb, https://github.com/openstack-infra/zuul/tree/master/zuul, unlucky to find it | 22:01 |
| *** ianychoi has joined #openstack-infra | 22:02 | |
| fungi | okay, this is a tale most strange... wiki-dev01 when rebooted comes up with only ipv4 global addressing on its eth0 interface. the /etc/network/interfaces file looks just like for our other servers, i haven't touched it. if i ifdown and ifup eth0 without rebooting, the ipv6 address magically appears on its interface | 22:03 |
| *** gongysh has quit IRC | 22:03 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow https://review.openstack.org/355666 | 22:03 |
| fungi | gongysh http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/webapp.py | 22:03 |
| craige | Did get to finish those patches yesterday, mtreinish | 22:04 |
| craige | ugh s/Did/Didn't/ | 22:04 |
| mtreinish | craige: ok, no worries. I figured that was the case, I looked at your open patch list this morning and didn't see anything :) | 22:05 |
| *** sarob has joined #openstack-infra | 22:05 | |
| craige | Yep, still WIP :-) | 22:05 |
| *** Goneri has joined #openstack-infra | 22:05 | |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/system-config: Add #openstack-101 to meetbot-channels https://review.openstack.org/367005 | 22:06 |
| *** _sarob has joined #openstack-infra | 22:06 | |
| fungi | i'm sort of at a loss for troubleshooting why the global v6 address isn't binding at boot time | 22:07 |
| fungi | since i can't recreate the issue once booted | 22:07 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 22:09 |
| *** tonytan4ever has quit IRC | 22:10 | |
| *** fguillot has joined #openstack-infra | 22:10 | |
| *** sarob has quit IRC | 22:10 | |
| clarkb | fungi: you aren't using glean with one of our images right? this is a rax image with nova agent? | 22:11 |
| fungi | clarkb: rax and cloud-init yes, and their nova-agent is running as usual | 22:11 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: SUPER WIP: Use new tempest run workflow https://review.openstack.org/355666 | 22:11 |
| fungi | clarkb: though the /etc/network/interfaces uses a static v6 config anyway | 22:12 |
| fungi | it's in dns so you can `ssh -4 wiki-dev01.openstack.org` to see | 22:12 |
| *** chlong has joined #openstack-infra | 22:12 | |
| fungi | it's freshly rebooted now and has no v6 global address on eth0 | 22:13 |
| *** jamesdenton has quit IRC | 22:13 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: Remove check-requirements for the independent sahara-test https://review.openstack.org/367013 | 22:13 |
| fungi | if you `sudo ifdown eth0 ; sudo ifup eth0` it suddenly grows its configured v6 global address | 22:13 |
| *** roxanaghe has quit IRC | 22:13 | |
| *** tphummel has quit IRC | 22:13 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run neutron linuxbridge and dvr jobs on xenial https://review.openstack.org/367014 | 22:14 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Add neutron api test on xenial to experimental https://review.openstack.org/367015 | 22:14 |
| fungi | and if you reboot again, it goes back to only global v4 on eth0 | 22:14 |
| *** berendt has quit IRC | 22:14 | |
| clarkb | weird | 22:14 |
| fungi | i'm finding nothing of note in its logs | 22:15 |
| *** aeng has joined #openstack-infra | 22:15 | |
| *** vhosakot has joined #openstack-infra | 22:16 | |
| *** baoli_ has joined #openstack-infra | 22:17 | |
| *** baoli has quit IRC | 22:18 | |
| *** roxanaghe has joined #openstack-infra | 22:19 | |
| *** burgerk_ has joined #openstack-infra | 22:19 | |
| TheJulia | random thought: privacy extensions? | 22:20 |
| clarkb | or maybe the two eth0 clauses in /etc/network/interfaces is making it mad? | 22:21 |
| clarkb | should those be folded together? | 22:21 |
| *** esberglu has quit IRC | 22:21 | |
| clarkb | oh no its explicitly inet6 nevermind | 22:21 |
| *** burgerk has quit IRC | 22:22 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/project-config: Ensure that gerrit projects have zuul pipelines https://review.openstack.org/363969 | 22:24 |
| *** acoles is now known as acoles_ | 22:24 | |
| *** burgerk_ has quit IRC | 22:24 | |
| *** sdake has quit IRC | 22:24 | |
| clarkb | there is something funny about setting ipv4 nameservers on an ipv6 interface | 22:24 |
| clarkb | fungi: ^ I have no idea if that could maybe be related? but I think those nameservers should be listed on the ipv4 block | 22:25 |
| *** dimtruck is now known as zz_dimtruck | 22:27 | |
| *** vhosakot has quit IRC | 22:27 | |
| *** gongysh has joined #openstack-infra | 22:27 | |
| jhesketh | Morning | 22:29 |
| clarkb | TheJulia: use_temp_addr is set to 0 for default and eth0 | 22:29 |
| clarkb | er use_tempaddr | 22:29 |
| *** akshai has joined #openstack-infra | 22:30 | |
| *** tphummel has joined #openstack-infra | 22:31 | |
| *** aeng has quit IRC | 22:32 | |
| fungi | clarkb: yeah, i was just pondering that myself, as resolvconf complains about /etc/resolv.conf not being a symlink to /var/run/resolvconf/resolv.conf or whatever when you bounce the interface. i think we simply don't need a nameservers line in there at all | 22:34 |
| clarkb | fungi: that might be how nova agent is writing them out though | 22:34 |
| clarkb | fungi: not sure if there is some other mechanism at play | 22:34 |
| fungi | clarkb: yeah, i have a feeling nova-agent is appending to the eth0 block | 22:34 |
| fungi | or maybe cloud-init? | 22:34 |
| clarkb | I doubt cloud init would since it doesn't have enough information to | 22:35 |
| clarkb | unless, does rax have meta data server? | 22:35 |
| fungi | unless it calls out to nova-agent through a hook or something | 22:35 |
| fungi | well, removing the dns-nameservers line from /etc/network/interfaces and rebooting made no difference, so there goes that theory | 22:36 |
| *** rlandy has quit IRC | 22:36 | |
| fungi | i'm 'a test switching to inet6 auto | 22:36 |
| clarkb | that won't work in rax because they don't ra | 22:37 |
| fungi | they don't? how are we getting routes from them then? | 22:37 |
| fungi | i guess you missed this morning's discussion about switching to inet6 auto across the board with glean | 22:38 |
| fungi | or else i missed subsequent discoveries | 22:38 |
| *** mdrabe has quit IRC | 22:39 | |
| *** gordc has quit IRC | 22:39 | |
| *** Hal has quit IRC | 22:39 | |
| clarkb | the config drive or nova agent supply all of that data | 22:39 |
| clarkb | whcih is why its staticly configured there | 22:39 |
| clarkb | I don't think switching glean to inet6 auto is going to fix anything in rax | 22:40 |
| clarkb | glean had an issue with rax where it wasn't handling multiple networks on one interface properly | 22:40 |
| clarkb | I must've missed anything ipv6 specific | 22:40 |
| *** Mary has joined #openstack-infra | 22:41 | |
| *** Mary is now known as Guest74959 | 22:41 | |
| fungi | ahh, yeah in our early debugging of that it was suggested that since these are correct eui-64 addresses corresponding to the vnic macs and because there seems to be no static route statements in the interfaces file or any hooks called from it, they must be doing ra | 22:41 |
| fungi | but apparently they aren't | 22:42 |
| fungi | so you think nova-agent is going behind our backs and setting network configuration? | 22:42 |
| clarkb | yup | 22:42 |
| clarkb | it might have cloud init involved somehow too but pretty sure that nova agent is how they get around not using dhcp | 22:43 |
| *** mdrabe has joined #openstack-infra | 22:43 | |
| clarkb | or ras for v6 | 22:43 |
| fungi | indeed. it looks like /etc/network/interfaces is likely completely unused during boot? :/ | 22:43 |
| fungi | ls /etc/rc?.d/???networking | 22:43 |
| *** asettle has joined #openstack-infra | 22:44 | |
| fungi | ls: cannot access /etc/rc?.d/???networking: No such file or directory | 22:44 |
| clarkb | oh I thought it was writing eni like glean but its just doing an end around that? | 22:44 |
| fungi | they've completely removed the standard /etc/rcS.d/S12networking symlink to /etc/init.d/networking | 22:44 |
| clarkb | at one time I figured out how all this worked bceause they were overwriting our nameservers too | 22:44 |
| *** adriant has joined #openstack-infra | 22:44 | |
| clarkb | ooh fun | 22:45 |
| fungi | the initscript is still there, it's just not used | 22:45 |
| fungi | so ifupdown is never called (at least not in the usual way) | 22:45 |
| *** spzala has quit IRC | 22:45 | |
| *** aeng has joined #openstack-infra | 22:45 | |
| *** Guest74959 has quit IRC | 22:46 | |
| clarkb | in other news we are 2.5 hours away from nodepoold restart time | 22:46 |
| fungi | awesome | 22:46 |
| *** spzala has joined #openstack-infra | 22:46 | |
| fungi | i'm going to boot another wiki-dev01 and see if i get consistently broken behavior here | 22:46 |
| *** spzala has quit IRC | 22:46 | |
| clarkb | fungi: ok | 22:46 |
| clarkb | fungi: maybe try using an older trusty image if it exists? | 22:46 |
| clarkb | if you show an older insatnce it will give you thei mage id and maybe you can reuse it | 22:46 |
| *** spzala has joined #openstack-infra | 22:47 | |
| *** spzala has quit IRC | 22:47 | |
| clarkb | then tomorrow I will try to clean up any of the currently leaked images in our clouds then hopefully we stop leaking them all tog ether | 22:47 |
| *** spzala has joined #openstack-infra | 22:47 | |
| *** spzala has quit IRC | 22:47 | |
| *** Julien-zte has joined #openstack-infra | 22:47 | |
| *** spzala has joined #openstack-infra | 22:48 | |
| *** spzala has quit IRC | 22:48 | |
| *** spzala has joined #openstack-infra | 22:48 | |
| *** thorst has joined #openstack-infra | 22:48 | |
| *** asettle has quit IRC | 22:49 | |
| *** spzala has joined #openstack-infra | 22:49 | |
| *** spzala has quit IRC | 22:49 | |
| fungi | i see only one ubuntu 14.04 image for pvhvm in the `openstack image list --public` output | 22:49 |
| fungi | oh, show an older instance. good idea | 22:49 |
| *** spzala has joined #openstack-infra | 22:49 | |
| *** spzala has quit IRC | 22:50 | |
| *** ddieterly is now known as ddieterly[away] | 22:50 | |
| *** ddieterly[away] has quit IRC | 22:50 | |
| *** spzala has joined #openstack-infra | 22:50 | |
| fungi | i don't see it list the image uuid from which an instance was booted | 22:50 |
| *** spzala has quit IRC | 22:50 | |
| fungi | no, wait, there it is | 22:51 |
| fungi | giving that one a shot | 22:51 |
| *** spzala has joined #openstack-infra | 22:51 | |
| *** mdrabe has quit IRC | 22:51 | |
| *** spzala has quit IRC | 22:51 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Add neutron api test on xenial to experimental https://review.openstack.org/367015 | 22:51 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Run neutron linuxbridge and dvr jobs on xenial https://review.openstack.org/367014 | 22:51 |
| *** spzala has joined #openstack-infra | 22:52 | |
| *** spzala has quit IRC | 22:52 | |
| *** spzala_ has joined #openstack-infra | 22:53 | |
| *** spzala has joined #openstack-infra | 22:53 | |
| *** spzala_ has quit IRC | 22:53 | |
| *** thorst has quit IRC | 22:53 | |
| *** spzala has joined #openstack-infra | 22:53 | |
| *** spzala has quit IRC | 22:53 | |
| *** baoli_ has quit IRC | 22:54 | |
| *** zz_dimtruck is now known as dimtruck | 22:54 | |
| *** spzala has joined #openstack-infra | 22:54 | |
| *** spzala has quit IRC | 22:54 | |
| *** senk has joined #openstack-infra | 22:54 | |
| *** spzala has joined #openstack-infra | 22:55 | |
| *** roxanaghe has quit IRC | 22:55 | |
| *** spzala has quit IRC | 22:55 | |
| *** xyang1 has quit IRC | 22:55 | |
| *** spzala has joined #openstack-infra | 22:56 | |
| *** spzala has quit IRC | 22:56 | |
| *** pradk has quit IRC | 22:56 | |
| *** spzala_ has joined #openstack-infra | 22:56 | |
| *** spzala has joined #openstack-infra | 22:56 | |
| *** spzala_ has quit IRC | 22:56 | |
| *** spzala has joined #openstack-infra | 22:57 | |
| *** spzala has quit IRC | 22:57 | |
| *** spzala has joined #openstack-infra | 22:57 | |
| *** spzala has quit IRC | 22:58 | |
| *** armax has quit IRC | 22:58 | |
| *** spzala has joined #openstack-infra | 22:58 | |
| *** gouthamr has quit IRC | 22:58 | |
| *** spzala has quit IRC | 22:58 | |
| fungi | cloud.create_server() doesn't like image="some-uuid"? | 22:59 |
| fungi | shade docs here i come | 22:59 |
| *** spzala has joined #openstack-infra | 22:59 | |
| *** spzala has quit IRC | 22:59 | |
| clarkb | its probably something like image_id= | 23:00 |
| *** spzala has joined #openstack-infra | 23:00 | |
| fungi | i expect so | 23:00 |
| *** hongbin has quit IRC | 23:00 | |
| *** spzala has quit IRC | 23:00 | |
| fungi | just looking to see which think like image_id= it is | 23:00 |
| fungi | s/think.thing/ | 23:00 |
| *** spzala has joined #openstack-infra | 23:01 | |
| *** roxanaghe has joined #openstack-infra | 23:01 | |
| *** spzala has quit IRC | 23:01 | |
| fungi | "image – Image dict, name or id to boot with." | 23:01 |
| *** spzala has joined #openstack-infra | 23:02 | |
| fungi | http://docs.openstack.org/infra/shade/usage.html#shade.OpenStackCloud.create_server | 23:02 |
| fungi | so maybe i need to make a dict? | 23:02 |
| *** spzala has quit IRC | 23:02 | |
| *** senk has quit IRC | 23:02 | |
| *** rbrndt has quit IRC | 23:02 | |
| *** spzala has joined #openstack-infra | 23:03 | |
| fungi | i'm not finding any explanation of how that dict should be constructed to represent the image i want to boot | 23:03 |
| *** spzala has quit IRC | 23:03 | |
| *** harlowja has quit IRC | 23:03 | |
| fungi | do i need to query it with one of the image methods and then pass the return value of that in? | 23:03 |
| fungi | seems a bit onerous when i already know the uuid i want to pass to server boot | 23:04 |
| *** spzala has joined #openstack-infra | 23:04 | |
| clarkb | oh ya I bet that may be what it wants | 23:04 |
| clarkb | get image first then pass the image object | 23:04 |
| *** spzala has joined #openstack-infra | 23:04 | |
| *** spzala has quit IRC | 23:05 | |
| *** spzala has joined #openstack-infra | 23:05 | |
| *** spzala has quit IRC | 23:05 | |
| *** spzala has joined #openstack-infra | 23:06 | |
| *** spzala has quit IRC | 23:06 | |
| *** spzala has joined #openstack-infra | 23:07 | |
| *** spzala has quit IRC | 23:07 | |
| *** spzala has joined #openstack-infra | 23:08 | |
| fungi | i love exceptions like this: | 23:08 |
| fungi | shade.exc.OpenStackCloudException: Error in creating instance (Inner Exception: create() takes at least 4 arguments (5 given)) | 23:08 |
| *** spzala has quit IRC | 23:08 | |
| *** spzala has joined #openstack-infra | 23:08 | |
| *** gongysh has quit IRC | 23:08 | |
| *** spzala has quit IRC | 23:08 | |
| *** Goneri has quit IRC | 23:09 | |
| *** spzala has joined #openstack-infra | 23:09 | |
| *** spzala has quit IRC | 23:09 | |
| *** spzala has joined #openstack-infra | 23:10 | |
| pabelanger | okay, going to need some help again with glean | 23:10 |
| *** spzala has quit IRC | 23:10 | |
| pabelanger | this time bonds | 23:10 |
| *** tonytan4ever has joined #openstack-infra | 23:10 | |
| *** AnarchyAo has quit IRC | 23:10 | |
| *** yamahata has quit IRC | 23:11 | |
| *** spzala has joined #openstack-infra | 23:11 | |
| *** spzala has quit IRC | 23:11 | |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/glean/tree/glean/tests/fixtures/liberty/mnt/config/openstack/latest/network_info.json#n82 | 23:11 |
| *** _sarob has quit IRC | 23:11 | |
| *** spzala has joined #openstack-infra | 23:11 | |
| *** spzala has quit IRC | 23:12 | |
| pabelanger | according to an existing test case, that would setup bond0 as dhcp I think | 23:12 |
| *** sdague has quit IRC | 23:12 | |
| *** spzala has joined #openstack-infra | 23:12 | |
| *** spzala has quit IRC | 23:12 | |
| fungi | cloud.get_image('6d833af9-9e31-4d4e-a4ea-1f7f3a4d4406') is returning None... seriously? | 23:12 |
| fungi | when `openstack image show 6d833af9-9e31-4d4e-a4ea-1f7f3a4d4406` returns data | 23:13 |
| *** spzala has joined #openstack-infra | 23:13 | |
| *** sarob has joined #openstack-infra | 23:13 | |
| fungi | i must be doing something terribly wrong here | 23:13 |
| *** spzala has quit IRC | 23:13 | |
| *** spzala has joined #openstack-infra | 23:14 | |
| clarkb | are you using the right region/cloud for the cloud setup? | 23:14 |
| *** spzala has quit IRC | 23:14 | |
| fungi | yeah, i mean create_server() was creating the instance in the right region | 23:14 |
| *** spzala has joined #openstack-infra | 23:15 | |
| *** spzala has quit IRC | 23:15 | |
| *** rajinir has quit IRC | 23:15 | |
| fungi | and this is the image uuid i found by doing a server show of review.openstack.org | 23:15 |
| fungi | so same region | 23:15 |
| *** reed_ has quit IRC | 23:15 | |
| *** tonytan4ever has quit IRC | 23:15 | |
| *** spzala has joined #openstack-infra | 23:15 | |
| *** spzala has quit IRC | 23:15 | |
| clarkb | huh I dunno then | 23:16 |
| fungi | yeah, i'm pretty stumped | 23:16 |
| *** salv-orl_ has quit IRC | 23:16 | |
| *** spzala has joined #openstack-infra | 23:16 | |
| *** spzala has quit IRC | 23:16 | |
| *** salv-orlando has joined #openstack-infra | 23:17 | |
| *** spzala has joined #openstack-infra | 23:17 | |
| *** spzala has quit IRC | 23:17 | |
| jesusaur | maybe cloud.get_image needs a name and not a uuid? | 23:18 |
| *** spzala has joined #openstack-infra | 23:18 | |
| *** spzala has quit IRC | 23:18 | |
| *** beagles has left #openstack-infra | 23:18 | |
| fungi | jesusaur: maybe, but 1. that contradicts the api usage documentation for shade, and 2. that would make it useless for this particular scenario since there are multiple images with the same name and i'm trying to select a deprecated one by id | 23:18 |
| *** spzala has joined #openstack-infra | 23:19 | |
| *** spzala has quit IRC | 23:19 | |
| *** tphummel has quit IRC | 23:19 | |
| *** spzala has joined #openstack-infra | 23:20 | |
| *** spzala has joined #openstack-infra | 23:20 | |
| *** spzala_ has joined #openstack-infra | 23:20 | |
| *** fguillot has quit IRC | 23:21 | |
| *** sarob has quit IRC | 23:21 | |
| *** spzala has joined #openstack-infra | 23:21 | |
| *** spzala has quit IRC | 23:21 | |
| *** salv-orlando has quit IRC | 23:21 | |
| *** spzala has joined #openstack-infra | 23:22 | |
| *** spzala has quit IRC | 23:22 | |
| *** akshai has quit IRC | 23:23 | |
| *** spzala has joined #openstack-infra | 23:23 | |
| *** spzala has quit IRC | 23:23 | |
| *** spzala has joined #openstack-infra | 23:24 | |
| *** akshai has joined #openstack-infra | 23:24 | |
| *** spzala has quit IRC | 23:24 | |
| *** spzala has joined #openstack-infra | 23:24 | |
| *** akshai has quit IRC | 23:25 | |
| *** spzala has joined #openstack-infra | 23:25 | |
| *** spzala has joined #openstack-infra | 23:26 | |
| *** spzala has quit IRC | 23:26 | |
| *** tphummel has joined #openstack-infra | 23:26 | |
| *** spzala has joined #openstack-infra | 23:26 | |
| *** spzala has quit IRC | 23:27 | |
| *** spzala has joined #openstack-infra | 23:27 | |
| *** claudiub|2 has joined #openstack-infra | 23:28 | |
| *** spzala has joined #openstack-infra | 23:28 | |
| *** spzala has quit IRC | 23:28 | |
| *** spzala has joined #openstack-infra | 23:29 | |
| *** spzala has quit IRC | 23:29 | |
| *** sarob has joined #openstack-infra | 23:29 | |
| *** Julien-zte has quit IRC | 23:29 | |
| *** spzala has joined #openstack-infra | 23:30 | |
| *** spzala has joined #openstack-infra | 23:30 | |
| *** spzala has quit IRC | 23:30 | |
| *** sarob has quit IRC | 23:31 | |
| *** spzala has joined #openstack-infra | 23:31 | |
| *** esberglu has joined #openstack-infra | 23:31 | |
| fungi | pabelanger: looking at that example, it seems like it's creating bond0 from an 802.3ad lag of two tap interfaces and configuring (among other things) dhclient to listen on bond0 for a dhcp lease | 23:31 |
| *** spzala has quit IRC | 23:31 | |
| fungi | pabelanger: the additional bond0 configuration is further down on line 194 | 23:32 |
| *** spzala has joined #openstack-infra | 23:32 | |
| *** spzala has quit IRC | 23:32 | |
| *** spzala has joined #openstack-infra | 23:32 | |
| *** spzala has joined #openstack-infra | 23:33 | |
| *** gouthamr has joined #openstack-infra | 23:34 | |
| *** sarob has joined #openstack-infra | 23:34 | |
| *** spzala has joined #openstack-infra | 23:34 | |
| *** spzala has quit IRC | 23:34 | |
| *** gouthamr_ has joined #openstack-infra | 23:34 | |
| *** spzala has joined #openstack-infra | 23:35 | |
| *** spzala has quit IRC | 23:35 | |
| *** fguillot has joined #openstack-infra | 23:35 | |
| fungi | clarkb: as (maddeningly) suspected, blowing away the old wiki-dev01 and booting a replacement from the same image i used the last time, ipv6 is working just fine now | 23:35 |
| *** spzala has joined #openstack-infra | 23:36 | |
| openstackgerrit | Dean Troyer proposed openstack-infra/project-config: Create python-openstackclient-stable-maint group https://review.openstack.org/366989 | 23:36 |
| *** spzala has joined #openstack-infra | 23:36 | |
| *** spzala has quit IRC | 23:37 | |
| clarkb | wow | 23:37 |
| *** spzala has joined #openstack-infra | 23:37 | |
| *** dingyichen has joined #openstack-infra | 23:38 | |
| *** yamahata has joined #openstack-infra | 23:38 | |
| fungi | maybe something about their metadata for the other instance was incomplete and nova-agent was failing to deal | 23:38 |
| *** gouthamr has quit IRC | 23:38 | |
| *** spzala has joined #openstack-infra | 23:38 | |
| *** gmann has joined #openstack-infra | 23:38 | |
| *** spzala has quit IRC | 23:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean: Append to interface definition for multi-address https://review.openstack.org/366862 | 23:39 |
| *** spzala has joined #openstack-infra | 23:39 | |
| pabelanger | fungi: clarkb: okay, some comments are lacking^. Going to push some up soon, but I think there is an issue with gentoo, prometheanfire do you mind helping with the gentoo syntax? | 23:39 |
| *** spzala has joined #openstack-infra | 23:40 | |
| *** spzala has quit IRC | 23:40 | |
| *** Swami has joined #openstack-infra | 23:40 | |
| *** spzala has joined #openstack-infra | 23:41 | |
| *** spzala has joined #openstack-infra | 23:41 | |
| *** Mary has joined #openstack-infra | 23:42 | |
| *** spzala has joined #openstack-infra | 23:42 | |
| *** spzala has quit IRC | 23:42 | |
| *** Mary is now known as Guest9940 | 23:42 | |
| *** spzala has joined #openstack-infra | 23:43 | |
| *** spzala has quit IRC | 23:43 | |
| *** spzala has joined #openstack-infra | 23:44 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/glean: Append to interface definition for multi-address https://review.openstack.org/366862 | 23:44 |
| *** spzala has joined #openstack-infra | 23:44 | |
| *** spzala has joined #openstack-infra | 23:45 | |
| *** asettle has joined #openstack-infra | 23:45 | |
| *** spzala has quit IRC | 23:45 | |
| *** spzala has joined #openstack-infra | 23:45 | |
| *** tonytan4ever has joined #openstack-infra | 23:45 | |
| *** spzala has quit IRC | 23:46 | |
| *** edmondsw has joined #openstack-infra | 23:46 | |
| *** spzala has joined #openstack-infra | 23:46 | |
| *** Guest9940 has quit IRC | 23:46 | |
| *** spzala has quit IRC | 23:46 | |
| *** _sarob has joined #openstack-infra | 23:47 | |
| *** jkilpatr has quit IRC | 23:47 | |
| *** spzala has joined #openstack-infra | 23:47 | |
| clarkb | we are 1.5 hours away from nodepool image uploads getting timed out. Down to 22 images though | 23:47 |
| clarkb | so it gets a good chunk of them done | 23:47 |
| *** spzala has joined #openstack-infra | 23:48 | |
| *** gildub has joined #openstack-infra | 23:48 | |
| *** zhurong has joined #openstack-infra | 23:49 | |
| *** akshai has joined #openstack-infra | 23:49 | |
| *** spzala has joined #openstack-infra | 23:49 | |
| *** spzala has quit IRC | 23:49 | |
| *** sarob has quit IRC | 23:49 | |
| *** edmondsw has quit IRC | 23:49 | |
| *** spzala has joined #openstack-infra | 23:50 | |
| *** spzala_ has joined #openstack-infra | 23:50 | |
| *** armax has joined #openstack-infra | 23:50 | |
| *** thorst has joined #openstack-infra | 23:50 | |
| *** spzala has joined #openstack-infra | 23:50 | |
| *** spzala has quit IRC | 23:50 | |
| *** asettle has quit IRC | 23:51 | |
| *** spzala has joined #openstack-infra | 23:51 | |
| *** spzala has quit IRC | 23:51 | |
| *** spzala has joined #openstack-infra | 23:52 | |
| *** spzala has quit IRC | 23:52 | |
| *** spzala has joined #openstack-infra | 23:53 | |
| *** spzala has joined #openstack-infra | 23:53 | |
| *** spzala has joined #openstack-infra | 23:55 | |
| *** spzala has quit IRC | 23:55 | |
| *** spzala has joined #openstack-infra | 23:55 | |
| *** spzala has quit IRC | 23:56 | |
| *** spzala has joined #openstack-infra | 23:56 | |
| *** spzala has quit IRC | 23:57 | |
| *** rfolco has joined #openstack-infra | 23:57 | |
| tonyb | anteaya: Is the "right" way to reach out to a 3rd party CI to email them and Cc the infra list? | 23:57 |
| *** spzala has joined #openstack-infra | 23:57 | |
| clarkb | is anyone else going to be around in ~1.5 hours to help should anyhting crazy happen with nodepoold restart? (I expect it will be fine since no shade releases since last restart) | 23:57 |
| *** spzala has quit IRC | 23:57 | |
| *** akshai has quit IRC | 23:57 | |
| *** spzala has joined #openstack-infra | 23:58 | |
| *** rfolco has quit IRC | 23:58 | |
| *** spzala has quit IRC | 23:58 | |
| *** thorst has quit IRC | 23:58 | |
| tonyb | clarkb: jhesketh shoudl be awake ... just say'n' ;P | 23:58 |
| *** spzala has joined #openstack-infra | 23:59 | |
| *** rfolco has joined #openstack-infra | 23:59 | |
| clarkb | tonyb: and ianw. I just never know who is busy or not :) | 23:59 |
| *** _sarob has quit IRC | 23:59 | |
| *** esberglu has quit IRC | 23:59 | |
| *** spzala has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!