| *** FallenPegasus has quit IRC | 00:10 | |
| *** FallenPegasus has joined #openstack-infra | 00:11 | |
| *** sdake has quit IRC | 00:12 | |
| *** darrenc_afk is now known as darrenc | 00:13 | |
| *** hichihara has joined #openstack-infra | 00:13 | |
| *** amotoki has joined #openstack-infra | 00:16 | |
| *** davideagnello has joined #openstack-infra | 00:18 | |
| *** salv-orl_ has quit IRC | 00:19 | |
| *** gildub has joined #openstack-infra | 00:19 | |
| *** jsavak has joined #openstack-infra | 00:23 | |
| *** davideagnello has quit IRC | 00:23 | |
| *** achanda has joined #openstack-infra | 00:25 | |
| *** chlong has joined #openstack-infra | 00:26 | |
| *** hamzy has joined #openstack-infra | 00:26 | |
| *** peterlisak_ has joined #openstack-infra | 00:29 | |
| *** amaretskiy1 has joined #openstack-infra | 00:29 | |
| *** clif has joined #openstack-infra | 00:30 | |
| *** FallenPe_ has joined #openstack-infra | 00:33 | |
| *** amaretskiy has quit IRC | 00:33 | |
| *** peterlisak has quit IRC | 00:33 | |
| *** clif_h has quit IRC | 00:33 | |
| *** FallenPegasus has quit IRC | 00:33 | |
| *** peterlisak_ is now known as peterlisak | 00:33 | |
| *** gildub has quit IRC | 00:33 | |
| *** tiswanso has quit IRC | 00:37 | |
| *** FallenPe_ has quit IRC | 00:37 | |
| *** mikal is now known as chair_occupier_t | 00:41 | |
| *** chair_occupier_t is now known as chair_occupier_3 | 00:41 | |
| *** FallenPegasus has joined #openstack-infra | 00:44 | |
| *** davideagnello has joined #openstack-infra | 00:45 | |
| SamYaple | clarkb: re docker image per commit: It would need to be every image every commit (rather than a subset). If we don't do it per commit than what would happen is master would never have working images in dockerhub, but releases would | 00:48 |
|---|---|---|
| SamYaple | that may work too, but just so you are aware. | 00:48 |
| SamYaple | A nightly push is reasonable though. so images may be out of date for a small portion of a day _sometimes_ | 00:48 |
| SamYaple | for stable branches a per tag push is fine | 00:49 |
| *** jsavak has quit IRC | 00:50 | |
| *** davideagnello has quit IRC | 00:50 | |
| *** jsavak has joined #openstack-infra | 00:50 | |
| *** gildub has joined #openstack-infra | 00:50 | |
| *** fedexo has quit IRC | 00:50 | |
| *** gildub has quit IRC | 00:58 | |
| *** dims has quit IRC | 01:01 | |
| *** chlong has quit IRC | 01:03 | |
| *** FallenPegasus has quit IRC | 01:03 | |
| *** amrith is now known as _amrith_ | 01:09 | |
| *** davideagnello has joined #openstack-infra | 01:12 | |
| *** tzn has quit IRC | 01:14 | |
| *** jsavak has quit IRC | 01:15 | |
| *** gildub has joined #openstack-infra | 01:15 | |
| *** jsavak has joined #openstack-infra | 01:17 | |
| *** davideagnello has quit IRC | 01:17 | |
| *** salv-orlando has joined #openstack-infra | 01:19 | |
| *** dims has joined #openstack-infra | 01:24 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: run integration jobs for puppet-gnocchi https://review.openstack.org/277219 | 01:28 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: add tripleo job to test puppet-ceph https://review.openstack.org/277221 | 01:31 |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config: Adds a devstack test job using Congress new architecture https://review.openstack.org/275514 | 01:37 |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config: Adds a tox test job for Congress new architecture https://review.openstack.org/275515 | 01:37 |
| *** tzn has joined #openstack-infra | 01:39 | |
| *** davideagnello has joined #openstack-infra | 01:40 | |
| *** davideagnello has quit IRC | 01:44 | |
| *** tzn has quit IRC | 01:44 | |
| *** dims has quit IRC | 01:46 | |
| *** yamamoto_ has joined #openstack-infra | 01:48 | |
| *** katyafervent_awa has quit IRC | 01:50 | |
| *** salv-orlando has quit IRC | 01:50 | |
| *** katyafervent_awa has joined #openstack-infra | 01:53 | |
| *** jsavak has quit IRC | 02:02 | |
| *** chlong has joined #openstack-infra | 02:06 | |
| *** davideagnello has joined #openstack-infra | 02:07 | |
| *** yamamoto_ has quit IRC | 02:07 | |
| *** yamamoto_ has joined #openstack-infra | 02:08 | |
| *** yamamoto_ has quit IRC | 02:10 | |
| *** yamamoto_ has joined #openstack-infra | 02:10 | |
| *** yamamoto_ has quit IRC | 02:10 | |
| *** davideagnello has quit IRC | 02:11 | |
| *** daemontool has quit IRC | 02:12 | |
| *** daemontool has joined #openstack-infra | 02:13 | |
| *** tiswanso has joined #openstack-infra | 02:16 | |
| *** tiswanso has quit IRC | 02:21 | |
| fungi | nibalizer: yeah, i'm planning to try and set up an account with startssl in the next couple weeks, and i'll get a new cert for it there. namecheap worked well enough when we only had a handful of certs, but it's getting cumbersome now and it looks like startssl will let us get an unlimited number of certs generated whenever we want for a flat rate (less than i'm paying for all our current certs too) | 02:32 |
| *** davideagnello has joined #openstack-infra | 02:34 | |
| clarkb | fungi death wish? | 02:34 |
| clarkb | their website stopped responding, I assume the ad worked | 02:35 |
| fungi | hah | 02:35 |
| fungi | yes, and not surprised :/ | 02:35 |
| mordred | clarkb: they had a sportsball ad? | 02:36 |
| openstackgerrit | Merged openstack/requirements: Bump minimum tempest-lib version https://review.openstack.org/276571 | 02:37 |
| fungi | they won a contest intuit was running to sponsor a small business's national superbowl ad | 02:37 |
| mordred | clarkb: (I did not watch the first half - I was shopping for furniture instead - turns out during superbowl is a GREAT time to go to a very large furniture store :) ) | 02:37 |
| mordred | fungi: I thought that was your coffee ... | 02:37 |
| mordred | oh | 02:37 |
| mordred | hah | 02:37 |
| mordred | I mixed the convo about startssl and the convo about the coffee | 02:37 |
| mordred | I thought you were saying the startssl site was down due to a superbowl ad | 02:38 |
| mordred | and I was like "dude - startssl did a superbowl ad?" | 02:38 |
| * mordred is caught up now | 02:38 | |
| fungi | lack of threading in irc ;) | 02:38 |
| clarkb | mordred: I recommend watching sportball during sportball | 02:38 |
| mordred | clarkb: indeed | 02:38 |
| clarkb | its been a decent game too | 02:38 |
| *** davideagnello has quit IRC | 02:38 | |
| mordred | clarkb: yah - second half is nice | 02:38 |
| mordred | clarkb: however, we just moved into this place that is not the size of a new york apartment, so basically feel like we have no furniture at all | 02:39 |
| clarkb | just amazon 2day some stuff | 02:39 |
| mordred | heh | 02:39 |
| clarkb | we did this for all baby furniture | 02:39 |
| mordred | I think amazoning a sectional sofa might be a bit much | 02:40 |
| mordred | (want to sit/lay on things) | 02:40 |
| pleia2 | mordred: congrats on the new place (and furniture!) | 02:40 |
| mordred | pleia2: thanks! | 02:40 |
| *** FallenPegasus has joined #openstack-infra | 02:40 | |
| * mordred is torn between wanting to invite everyone over for an afterstack ... and imagining having EVERYONE over for an afterstack | 02:40 | |
| clarkb | mordred: people do it, beds too | 02:40 |
| *** tzn has joined #openstack-infra | 02:41 | |
| fungi | luckily when we bought, the place came completely furnished with pretty nice stuff (yay vacation town), so the fact that we were moving from a tiny place and were bringing almost no furniture actually worked out very well | 02:41 |
| * greghaynes got his bed off Amazon | 02:41 | |
| mordred | fungi: nice! | 02:41 |
| mordred | fungi: we did get a table with the place | 02:41 |
| greghaynes | But I have an Amazon problem | 02:41 |
| mordred | the previous owners had a custom table made from the same granite as the countertops | 02:41 |
| clarkb | I should buy a place | 02:41 |
| fungi | greghaynes: admitting it is the first step | 02:41 |
| mordred | so taking it with them didn't really make any sense | 02:41 |
| clarkb | it turns out thisnis difficult | 02:41 |
| clarkb | portland world problems | 02:42 |
| greghaynes | fungi: its a long process | 02:42 |
| mordred | clarkb: there is a house for sale 2 down from me | 02:42 |
| pleia2 | clarkb: heh, yeah, I was going to say you should do it soon since house prices rising up there | 02:42 |
| *** changbl has quit IRC | 02:42 | |
| mordred | clarkb: probably half the price of all of the houses in portland, fwiw | 02:42 |
| clarkb | I actually had an offer accepted | 02:42 |
| clarkb | then sellers got crazy and weird | 02:42 |
| greghaynes | We both did... | 02:42 |
| clarkb | and we noped ouy | 02:42 |
| mordred | *headdesk* | 02:42 |
| greghaynes | Portland housing is not fun | 02:42 |
| *** changbl has joined #openstack-infra | 02:42 | |
| mordred | sellers suck | 02:42 |
| pleia2 | :\ | 02:42 |
| *** oomichi has quit IRC | 02:43 | |
| clarkb | it was really bad | 02:43 |
| clarkb | they started demanding more money and not communicatimg | 02:43 |
| mordred | oh neat | 02:43 |
| mordred | that's exactly what you want | 02:43 |
| clarkb | so I called and they can find anew buyer | 02:43 |
| clarkb | (we apparently pissed them off when we asked them to correct the siding which was also their first item on all advertising with "new siding!!!?") | 02:44 |
| clarkb | but was installed completely wrong | 02:45 |
| clarkb | and it all went downhill from there | 02:45 |
| pleia2 | ugh, that's awful | 02:45 |
| clarkb | at this rate I might find a cardboard shack under a bridge | 02:46 |
| clarkb | at least that will come with river views | 02:46 |
| pleia2 | hah | 02:46 |
| *** daemontool has quit IRC | 02:47 | |
| pleia2 | we bought in 2009, which is probably the smartest thing we've ever done | 02:47 |
| *** akshai has joined #openstack-infra | 02:48 | |
| *** [1]Thelo has joined #openstack-infra | 02:51 | |
| *** Thelo has quit IRC | 02:53 | |
| *** [1]Thelo is now known as Thelo | 02:53 | |
| *** akshai has quit IRC | 02:54 | |
| *** yamamoto_ has joined #openstack-infra | 02:58 | |
| *** davideagnello has joined #openstack-infra | 03:01 | |
| *** davideagnello has quit IRC | 03:06 | |
| *** salv-orlando has joined #openstack-infra | 03:10 | |
| mordred | clarkb: there was that time we were in portland and walked by that nice mattress on the bridge underpass - I bet you could take whoever owns that | 03:11 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Use dnf to cleanup old kernels https://review.openstack.org/249539 | 03:12 |
| *** salv-orlando has quit IRC | 03:13 | |
| *** dims has joined #openstack-infra | 03:13 | |
| *** tzn has quit IRC | 03:14 | |
| *** rguillebert has quit IRC | 03:17 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Use dnf to cleanup old kernels https://review.openstack.org/249539 | 03:21 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Add an experiment job for Neutron FWaaS tempest tests https://review.openstack.org/237376 | 03:21 |
| *** unicell has quit IRC | 03:22 | |
| *** unicell has joined #openstack-infra | 03:22 | |
| openstackgerrit | Angus Lees proposed openstack-infra/project-config: Create new tempest-dsvm-full-head job-template https://review.openstack.org/270143 | 03:25 |
| *** gildub has quit IRC | 03:25 | |
| *** mingyu has joined #openstack-infra | 03:27 | |
| *** mingyu has quit IRC | 03:28 | |
| *** chair_occupier_3 is now known as mikal | 03:34 | |
| *** dims has quit IRC | 03:35 | |
| *** tiswanso has joined #openstack-infra | 03:36 | |
| *** daemontool has joined #openstack-infra | 03:38 | |
| *** dims has joined #openstack-infra | 03:42 | |
| *** davideagnello has joined #openstack-infra | 03:42 | |
| *** dkranz has quit IRC | 03:44 | |
| *** davideagnello has quit IRC | 03:46 | |
| SamYaple | oh.... i guess the superbowl was tonight | 03:48 |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Run all flake8 checks https://review.openstack.org/276500 | 03:57 |
| *** gildub has joined #openstack-infra | 03:58 | |
| *** baoli has joined #openstack-infra | 04:05 | |
| *** hichihara has quit IRC | 04:06 | |
| *** dingyichen has quit IRC | 04:08 | |
| *** dims has quit IRC | 04:08 | |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Add H203 to check that assertIs(Not)None is used https://review.openstack.org/276517 | 04:11 |
| *** tzn has joined #openstack-infra | 04:11 | |
| *** dingyichen has joined #openstack-infra | 04:13 | |
| *** amotoki has quit IRC | 04:13 | |
| *** hichihara has joined #openstack-infra | 04:13 | |
| *** links has joined #openstack-infra | 04:14 | |
| *** tzn has quit IRC | 04:16 | |
| tonyb | lifless: What's the correct workflow for sugesting an enhancement in fixtures? is it all github based? | 04:18 |
| tonyb | lifeless: note I'm not 100% certain it is a fixtures issue but if it is I'm looking at how to fix it ... | 04:19 |
| *** mingyu has joined #openstack-infra | 04:21 | |
| *** davideagnello has joined #openstack-infra | 04:23 | |
| *** amotoki has joined #openstack-infra | 04:27 | |
| *** davideagnello has quit IRC | 04:27 | |
| *** david-lyle has quit IRC | 04:30 | |
| *** dstanek has quit IRC | 04:31 | |
| *** dstanek has joined #openstack-infra | 04:33 | |
| *** tiswanso has quit IRC | 04:33 | |
| *** hdd has joined #openstack-infra | 04:34 | |
| *** unicell has quit IRC | 04:37 | |
| *** unicell has joined #openstack-infra | 04:37 | |
| *** eliqiao_ has quit IRC | 04:38 | |
| *** andrearosa has quit IRC | 04:38 | |
| *** andrearosa has joined #openstack-infra | 04:38 | |
| *** lennyb_ has quit IRC | 04:38 | |
| *** lennyb_ has joined #openstack-infra | 04:38 | |
| *** amotoki has quit IRC | 04:39 | |
| *** eliqiao_ has joined #openstack-infra | 04:40 | |
| *** amotoki has joined #openstack-infra | 04:40 | |
| *** salv-orlando has joined #openstack-infra | 04:44 | |
| *** achanda has quit IRC | 04:45 | |
| *** david-lyle has joined #openstack-infra | 04:45 | |
| *** baoli has quit IRC | 04:48 | |
| *** baoli has joined #openstack-infra | 04:50 | |
| *** dimtruck is now known as zz_dimtruck | 04:50 | |
| *** davideagnello has joined #openstack-infra | 04:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:50 | |
| *** baoli has quit IRC | 04:51 | |
| *** davideagnello has quit IRC | 04:54 | |
| *** chaitu has quit IRC | 04:55 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Revert "Skip centos functional testing" https://review.openstack.org/274938 | 04:55 |
| *** amotoki has quit IRC | 04:56 | |
| *** salv-orlando has quit IRC | 04:56 | |
| *** shashank_hegde has quit IRC | 04:57 | |
| *** hdd has quit IRC | 05:03 | |
| *** eliqiao_ has quit IRC | 05:06 | |
| *** eliqiao_ has joined #openstack-infra | 05:08 | |
| *** amotoki has joined #openstack-infra | 05:11 | |
| *** tzn has joined #openstack-infra | 05:12 | |
| *** hdd has joined #openstack-infra | 05:17 | |
| *** Sukhdev has joined #openstack-infra | 05:17 | |
| *** davideagnello has joined #openstack-infra | 05:17 | |
| *** tzn has quit IRC | 05:17 | |
| *** nikil89_ has joined #openstack-infra | 05:19 | |
| *** achanda has joined #openstack-infra | 05:21 | |
| *** davideagnello has quit IRC | 05:22 | |
| *** hdd has quit IRC | 05:22 | |
| *** hdd has joined #openstack-infra | 05:34 | |
| *** hdd has quit IRC | 05:37 | |
| lifeless | tonyb: issues in launchpad, ocde in github | 05:39 |
| tonyb | lifeless: okay thanks. | 05:39 |
| * tonyb just wasted have a day on a related issue only to find the test "works" on MacOS but fails on Linux *sigh* | 05:40 | |
| *** abregman has joined #openstack-infra | 05:42 | |
| lifeless | tonyb: :/ | 05:43 |
| lifeless | tonyb: what was the issue? | 05:43 |
| tonyb | lifeless: I'm trying to reproduce a failure seen in the gate post testtoosl 2.0.0 release. | 05:44 |
| *** davideagnello has joined #openstack-infra | 05:44 | |
| tonyb | lifeless: the fact the mac worked has somethign to do with it trying to use berkleyDB support. | 05:45 |
| tonyb | lifeless: AFAICT that failing altered the code that was being run so I never got to the actual code I wanted to test :( | 05:45 |
| tonyb | lifeless: copying the same source and reproducing the venv in linux and I trigger the bug right away. | 05:46 |
| tonyb | lifeless: so now to the testtools/fixtures issue | 05:46 |
| lifeless | tonyb: is there a bug report? | 05:46 |
| tonyb | lifeless: on the testtools/fixtures issue? | 05:46 |
| lifeless | tonyb: either :) | 05:46 |
| tonyb | lifeless: I'm not going to bother the the db thing, and I'll do a bug report for the otehr once I confirm it :) | 05:47 |
| *** davideagnello has quit IRC | 05:49 | |
| lifeless | k | 05:54 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Switch simple-init to pip-and-virtualenv element https://review.openstack.org/239562 | 05:58 |
| tonyb | lifeless: https://bugs.launchpad.net/python-fixtures/+bug/1542984 | 06:01 |
| openstack | Launchpad bug 1542984 in Python Fixtures "fixtures 1.2.0 isn't compatible with testtools 2.0.0" [Undecided,New] | 06:01 |
| lifeless | tonyb: thats going to be closed not a bug | 06:01 |
| lifeless | tonyb: 2.0.0 depends on 1.3.0 or greater | 06:01 |
| tonyb | lifeless: It's probably a little rambly as I created it wheil watching the kids in the bath. | 06:01 |
| lifeless | tonyb: and our metadata expresses that | 06:01 |
| tonyb | right but fixtures 1.2.0 allows 2.0.0 to be installed | 06:01 |
| lifeless | tonyb: not as such | 06:02 |
| lifeless | tonyb: you're running into a known bug in pip I suspect; I'll look once Cynthia is in bed | 06:02 |
| tonyb | lifeless: okay. | 06:02 |
| tonyb | lifeless: I inclued links to the repro in tha gate | 06:02 |
| tonyb | lifeless: I'd be happy for any solution. | 06:03 |
| tonyb | lifeless: I'll be back once my kids are in bed | 06:03 |
| *** amotoki_ has joined #openstack-infra | 06:08 | |
| *** amotoki has quit IRC | 06:11 | |
| *** davideagnello has joined #openstack-infra | 06:11 | |
| *** tzn has joined #openstack-infra | 06:14 | |
| *** davideagnello has quit IRC | 06:16 | |
| lifeless | tonyb: k; I have replied in brief there | 06:16 |
| lifeless | tonyb: happy to discuss more, and perhaps do a pragmatic if ick thing, but we'll see | 06:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-nodepool: Fix cleanup of rotated files https://review.openstack.org/277291 | 06:18 |
| *** tzn has quit IRC | 06:19 | |
| *** naresht has joined #openstack-infra | 06:20 | |
| naresht | Hi all. I developed an application for openstack/murano-apps. I want to submit for code review. openstack/murano-apps uses Gerrit for code review. How can I do this? | 06:23 |
| tonyb | naresht: https://wiki.openstack.org/wiki/How_To_Contribute#How_can_I_help | 06:25 |
| *** chaitu has joined #openstack-infra | 06:28 | |
| *** baoli has joined #openstack-infra | 06:28 | |
| chaitu | while starting nodepool services for the first time by default it is starting using option as --no-builder. how could i start nodepool service with builder http://paste.openstack.org/show/486250/ | 06:31 |
| *** baoli has quit IRC | 06:33 | |
| lifeless | tonyb: wb | 06:37 |
| *** davideagnello has joined #openstack-infra | 06:39 | |
| *** davideagnello has quit IRC | 06:43 | |
| *** Daisy has joined #openstack-infra | 06:45 | |
| *** unicell has quit IRC | 06:45 | |
| *** salv-orlando has joined #openstack-infra | 06:48 | |
| *** unicell has joined #openstack-infra | 06:49 | |
| *** Daisy has quit IRC | 06:49 | |
| *** savihou has joined #openstack-infra | 06:50 | |
| *** Daisy has joined #openstack-infra | 06:50 | |
| *** daemontool has quit IRC | 06:53 | |
| *** daemontool has joined #openstack-infra | 06:54 | |
| *** Daisy has quit IRC | 06:55 | |
| *** salv-orlando has quit IRC | 06:55 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/277295 | 06:55 |
| *** gildub has quit IRC | 06:56 | |
| *** daemontool has quit IRC | 06:58 | |
| *** chlong has quit IRC | 07:03 | |
| *** davideagnello has joined #openstack-infra | 07:06 | |
| *** kzaitsev_mb has quit IRC | 07:08 | |
| *** davideagnello has quit IRC | 07:10 | |
| *** oomichi has joined #openstack-infra | 07:12 | |
| openstackgerrit | Fredrik Medley proposed openstack-infra/zuul: Improve job dependencies using graph instead of tree https://review.openstack.org/231124 | 07:13 |
| *** tzn has joined #openstack-infra | 07:14 | |
| *** jtomasek__ has joined #openstack-infra | 07:15 | |
| *** tzn has quit IRC | 07:19 | |
| *** Jeffrey4l has joined #openstack-infra | 07:19 | |
| *** infra-red has joined #openstack-infra | 07:20 | |
| *** jtomasek_ has joined #openstack-infra | 07:25 | |
| *** Daisy has joined #openstack-infra | 07:26 | |
| openstackgerrit | Merged openstack/diskimage-builder: Switch simple-init to pip-and-virtualenv element https://review.openstack.org/239562 | 07:28 |
| *** davideagnello has joined #openstack-infra | 07:33 | |
| *** kushal has joined #openstack-infra | 07:34 | |
| *** davideagnello has quit IRC | 07:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Change shade keystone v2 job for stable/liberty https://review.openstack.org/274877 | 07:40 |
| *** Daisy has quit IRC | 07:42 | |
| *** Daisy has joined #openstack-infra | 07:43 | |
| *** Daisy has quit IRC | 07:46 | |
| *** hichihara has quit IRC | 07:48 | |
| *** zeih_ has joined #openstack-infra | 07:48 | |
| *** flepied has quit IRC | 07:49 | |
| *** nmagnezi has joined #openstack-infra | 07:49 | |
| yolanda | morning | 07:54 |
| *** jaosorior has joined #openstack-infra | 07:57 | |
| SamYaple | morning yolanda | 07:59 |
| *** davideagnello has joined #openstack-infra | 08:00 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 08:01 |
| *** jlanoux has joined #openstack-infra | 08:02 | |
| *** davideagnello has quit IRC | 08:05 | |
| *** jaosorior has quit IRC | 08:05 | |
| *** |-paul-| has joined #openstack-infra | 08:05 | |
| *** salv-orlando has joined #openstack-infra | 08:07 | |
| *** rcernin has joined #openstack-infra | 08:09 | |
| *** scheuran has joined #openstack-infra | 08:09 | |
| *** salv-orlando has quit IRC | 08:12 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add new bi-weekly APAC meeting for user committee https://review.openstack.org/269427 | 08:13 |
| *** samuelBartel has joined #openstack-infra | 08:13 | |
| *** Sukhdev has quit IRC | 08:13 | |
| *** jaosorior has joined #openstack-infra | 08:13 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Script to generate dns for infracloud https://review.openstack.org/277150 | 08:15 |
| *** arxcruz has joined #openstack-infra | 08:16 | |
| *** tzn has joined #openstack-infra | 08:16 | |
| *** flepied has joined #openstack-infra | 08:16 | |
| openstackgerrit | Merged openstack-infra/project-config: run integration jobs for puppet-gnocchi https://review.openstack.org/277219 | 08:18 |
| openstackgerrit | Merged openstack-infra/project-config: add tripleo job to test puppet-ceph https://review.openstack.org/277221 | 08:18 |
| openstackgerrit | Merged openstack-infra/project-config: Adds a devstack test job using Congress new architecture https://review.openstack.org/275514 | 08:18 |
| openstackgerrit | Merged openstack-infra/project-config: Create new tempest-dsvm-full-head job-template https://review.openstack.org/270143 | 08:19 |
| *** rubasov has joined #openstack-infra | 08:19 | |
| *** sdake has joined #openstack-infra | 08:20 | |
| *** tzn has quit IRC | 08:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Adds a tox test job for Congress new architecture https://review.openstack.org/275515 | 08:21 |
| openstackgerrit | Frank Kloeker proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite https://review.openstack.org/274701 | 08:23 |
| *** ifarkas has joined #openstack-infra | 08:24 | |
| *** davideagnello has joined #openstack-infra | 08:27 | |
| *** Daisy has joined #openstack-infra | 08:29 | |
| *** achanda has quit IRC | 08:31 | |
| openstackgerrit | Alistair Coles proposed openstack-infra/project-config: Add job gate-swift-tox-func-in-process-fast-post https://review.openstack.org/276823 | 08:32 |
| *** davideagnello has quit IRC | 08:32 | |
| *** jistr has joined #openstack-infra | 08:33 | |
| *** jistr is now known as jistr|sick | 08:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Don't run gate-sahara-tox-bandit on stable/kilo https://review.openstack.org/277096 | 08:35 |
| openstackgerrit | Merged openstack-infra/project-config: Remove Nova pip-check-reqs job https://review.openstack.org/277076 | 08:37 |
| openstackgerrit | Merged openstack-infra/project-config: Add grenade testing for oslo.log https://review.openstack.org/276476 | 08:38 |
| openstackgerrit | Merged openstack-infra/infra-manual: Creator's Guide: Add some pointer https://review.openstack.org/276813 | 08:39 |
| *** yaume has joined #openstack-infra | 08:40 | |
| *** fhubik has joined #openstack-infra | 08:42 | |
| openstackgerrit | Vasyl Saienko proposed openstack/requirements: Add netmiko to requirements https://review.openstack.org/276200 | 08:42 |
| openstackgerrit | Merged openstack-infra/project-config: Import ansible-roles-jenkins(-plugins) projects https://review.openstack.org/276910 | 08:42 |
| *** fhubik is now known as fhubik_brb | 08:43 | |
| openstackgerrit | Merged openstack-infra/puppet-project_config: Add nodepool.yaml to the layout https://review.openstack.org/276963 | 08:44 |
| *** achanda has joined #openstack-infra | 08:44 | |
| *** mrmartin has joined #openstack-infra | 08:44 | |
| *** dingyichen has quit IRC | 08:46 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul: Allow configuration of X-Account-Meta-Temp-Url-Key https://review.openstack.org/276871 | 08:47 |
| *** chlong has joined #openstack-infra | 08:48 | |
| *** Hal has joined #openstack-infra | 08:49 | |
| *** Hal is now known as Guest48074 | 08:50 | |
| *** fhubik_brb is now known as fhubik | 08:50 | |
| *** davideagnello has joined #openstack-infra | 08:55 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:55 | |
| *** amotoki_ has quit IRC | 08:55 | |
| *** jaosorior has quit IRC | 08:57 | |
| *** davideagnello has quit IRC | 08:59 | |
| *** openstackgerrit has quit IRC | 09:02 | |
| *** openstackgerrit has joined #openstack-infra | 09:02 | |
| *** jaosorior has joined #openstack-infra | 09:03 | |
| openstackgerrit | Vasyl Saienko proposed openstack/requirements: Add netmiko to requirements https://review.openstack.org/276200 | 09:05 |
| *** fhubik is now known as fhubik_brb | 09:06 | |
| *** lezbar has joined #openstack-infra | 09:07 | |
| *** adrienverge has joined #openstack-infra | 09:07 | |
| *** achanda has quit IRC | 09:08 | |
| *** lucas-dinner is now known as lucasagomes | 09:09 | |
| *** fhubik_brb is now known as fhubik | 09:12 | |
| *** zeih_ has quit IRC | 09:13 | |
| *** amotoki has joined #openstack-infra | 09:14 | |
| *** matrohon has joined #openstack-infra | 09:17 | |
| *** tzn has joined #openstack-infra | 09:17 | |
| *** rossella_s has joined #openstack-infra | 09:19 | |
| *** tzn has quit IRC | 09:21 | |
| *** davideagnello has joined #openstack-infra | 09:21 | |
| *** yuanying is now known as yuanying_influen | 09:25 | |
| *** yuanying_influen is now known as yuanying_flu | 09:25 | |
| *** davideagnello has quit IRC | 09:26 | |
| *** hashar has joined #openstack-infra | 09:27 | |
| *** zeih_ has joined #openstack-infra | 09:27 | |
| *** isaacb has joined #openstack-infra | 09:28 | |
| *** fhubik is now known as fhubik_brb | 09:29 | |
| *** salv-orlando has joined #openstack-infra | 09:31 | |
| *** aarefiev_ has quit IRC | 09:31 | |
| *** amotoki has quit IRC | 09:32 | |
| *** aarefiev has joined #openstack-infra | 09:32 | |
| *** ildikov has quit IRC | 09:37 | |
| *** Jeffrey4l has quit IRC | 09:37 | |
| *** dguitarbite has quit IRC | 09:45 | |
| *** mrmartin has quit IRC | 09:48 | |
| *** fhubik_brb is now known as fhubik | 09:49 | |
| *** tzn has joined #openstack-infra | 09:50 | |
| *** kzaitsev_mb has quit IRC | 09:50 | |
| *** tzn has quit IRC | 09:51 | |
| *** ihrachys has joined #openstack-infra | 09:51 | |
| *** tzn has joined #openstack-infra | 09:52 | |
| *** haypo has joined #openstack-infra | 09:55 | |
| haypo | ianw: hello. i tried to use env markers in requirements of the gnocchi project, but tox -e bashate failed because bashate 0.3.2 (latest release) still requires pbr <1.0 | 09:56 |
| openstackgerrit | Julien Danjou proposed openstack/requirements: Add overtest to global-req https://review.openstack.org/274348 | 09:56 |
| haypo | ianw: would you be ok to release a new version of bashate? | 09:56 |
| haypo | jd__, ianw : ah, it looks like bashate doesn't use https://github.com/openstack/releases.git ? | 09:57 |
| jd__ | haypo: which team does it belong to? | 09:58 |
| *** dtantsur|afk is now known as dtantsur | 09:58 | |
| *** pedroalvarez_ is now known as pedroalvarez | 09:58 | |
| jd__ | QA it seems | 09:58 |
| jd__ | and it's marked release:independent | 09:59 |
| jd__ | haypo: so you have to ask mtreinish probably to do a release, or someone in that team | 09:59 |
| haypo | mtreinish: hey! i need a release for bashate :) it's blocking me to "fix" requirements of gnocchi, https://review.openstack.org/#/c/276742/ | 09:59 |
| *** rossella_s has quit IRC | 10:03 | |
| *** rossella_s has joined #openstack-infra | 10:03 | |
| *** flepied has quit IRC | 10:04 | |
| openstackgerrit | Frank Kloeker proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite https://review.openstack.org/274701 | 10:06 |
| *** kzaitsev_mb has joined #openstack-infra | 10:06 | |
| *** placko has joined #openstack-infra | 10:07 | |
| *** ildikov has joined #openstack-infra | 10:10 | |
| *** rguillebert has joined #openstack-infra | 10:14 | |
| *** sfinucan has joined #openstack-infra | 10:15 | |
| *** _nadya_ has joined #openstack-infra | 10:15 | |
| *** pavel_bondar has quit IRC | 10:17 | |
| *** electrofelix has joined #openstack-infra | 10:18 | |
| *** Daisy has quit IRC | 10:20 | |
| *** flepied has joined #openstack-infra | 10:21 | |
| *** flepied1 has joined #openstack-infra | 10:24 | |
| *** isaacb has quit IRC | 10:24 | |
| *** flepied has quit IRC | 10:27 | |
| *** ildikov has quit IRC | 10:28 | |
| *** gildub has joined #openstack-infra | 10:29 | |
| *** Daisy has joined #openstack-infra | 10:29 | |
| *** isq_ has quit IRC | 10:30 | |
| *** ociuhandu has quit IRC | 10:30 | |
| *** isq_ has joined #openstack-infra | 10:31 | |
| *** ildikov has joined #openstack-infra | 10:36 | |
| *** mingyu has quit IRC | 10:39 | |
| *** salv-orl_ has joined #openstack-infra | 10:40 | |
| *** kzaitsev_mb has quit IRC | 10:41 | |
| *** salv-orlando has quit IRC | 10:43 | |
| *** tpsilva has joined #openstack-infra | 10:49 | |
| *** e0ne has joined #openstack-infra | 10:53 | |
| *** aysyd has joined #openstack-infra | 10:56 | |
| *** davideagnello has joined #openstack-infra | 10:57 | |
| AJaeger | haypo: https://review.openstack.org/#/admin/groups/333,members is the list of cores to ask | 10:57 |
| *** chaitu has quit IRC | 10:59 | |
| *** aysyd has quit IRC | 10:59 | |
| *** fhubik is now known as fhubik_brb | 11:00 | |
| *** aysyd has joined #openstack-infra | 11:00 | |
| *** sdake has quit IRC | 11:00 | |
| *** davideagnello has quit IRC | 11:02 | |
| haypo | AJaeger: thanks. email sent | 11:08 |
| *** electrofelix has quit IRC | 11:11 | |
| *** electrofelix has joined #openstack-infra | 11:12 | |
| *** FallenPegasus has quit IRC | 11:15 | |
| *** mingyu has joined #openstack-infra | 11:17 | |
| *** FallenPegasus has joined #openstack-infra | 11:18 | |
| *** esikachev has joined #openstack-infra | 11:19 | |
| *** rcernin has quit IRC | 11:19 | |
| *** derekh has joined #openstack-infra | 11:20 | |
| *** rcernin has joined #openstack-infra | 11:22 | |
| *** rcernin has quit IRC | 11:25 | |
| *** rcernin has joined #openstack-infra | 11:26 | |
| *** yamamoto_ has quit IRC | 11:27 | |
| *** Jeffrey4l has joined #openstack-infra | 11:29 | |
| *** zeih_ has quit IRC | 11:36 | |
| *** esikachev has quit IRC | 11:36 | |
| *** jistr|sick has quit IRC | 11:37 | |
| *** davideagnello has joined #openstack-infra | 11:37 | |
| *** dguitarbite has joined #openstack-infra | 11:38 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:38 | |
| *** dguitarbite has quit IRC | 11:40 | |
| *** rhallisey has joined #openstack-infra | 11:40 | |
| *** fhubik_brb is now known as fhubik | 11:40 | |
| *** jaosorior has quit IRC | 11:41 | |
| *** esikachev has joined #openstack-infra | 11:41 | |
| *** zeih_ has joined #openstack-infra | 11:42 | |
| *** davideagnello has quit IRC | 11:42 | |
| *** jaosorior has joined #openstack-infra | 11:43 | |
| *** kzaitsev_mb has quit IRC | 11:43 | |
| *** baoli has joined #openstack-infra | 11:48 | |
| *** Jeffrey4l has quit IRC | 11:48 | |
| *** baoli has quit IRC | 11:49 | |
| *** |-paul-| has quit IRC | 11:49 | |
| *** vgridnev has joined #openstack-infra | 11:55 | |
| *** jaypipes has joined #openstack-infra | 11:56 | |
| *** dguitarbite has joined #openstack-infra | 11:56 | |
| *** dguitarbite has quit IRC | 11:56 | |
| *** ganso has joined #openstack-infra | 11:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:00 | |
| AJaeger | sdague: could you approve nova translation import, please? https://review.openstack.org/275530 | 12:00 |
| *** erlon has joined #openstack-infra | 12:01 | |
| *** dguitarbite has joined #openstack-infra | 12:02 | |
| *** davideagnello has joined #openstack-infra | 12:05 | |
| *** vgridnev has quit IRC | 12:07 | |
| *** lucasagomes is now known as lucas-hungry | 12:07 | |
| *** dims has joined #openstack-infra | 12:07 | |
| *** davideagnello has quit IRC | 12:10 | |
| *** weshay_xchat has quit IRC | 12:10 | |
| *** amotoki has joined #openstack-infra | 12:13 | |
| *** vgridnev has joined #openstack-infra | 12:13 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing https://review.openstack.org/252814 | 12:16 |
| *** jaosorior has quit IRC | 12:17 | |
| *** salv-orl_ has quit IRC | 12:18 | |
| *** isaacb has joined #openstack-infra | 12:18 | |
| *** tcammann has joined #openstack-infra | 12:19 | |
| *** claudiub has joined #openstack-infra | 12:19 | |
| *** dims has quit IRC | 12:20 | |
| *** gildub has quit IRC | 12:21 | |
| *** yamamoto has joined #openstack-infra | 12:21 | |
| *** dims has joined #openstack-infra | 12:22 | |
| * dims is back. sorry flaky network | 12:22 | |
| *** ociuhandu has joined #openstack-infra | 12:25 | |
| *** gordc has joined #openstack-infra | 12:30 | |
| *** markus_z has joined #openstack-infra | 12:31 | |
| *** davideagnello has joined #openstack-infra | 12:32 | |
| *** amotoki has quit IRC | 12:35 | |
| krotscheck_dcm | I | 12:36 |
| *** davideagnello has quit IRC | 12:37 | |
| krotscheck_dcm | I'm trying to figure out what the job runtimes are for our devstack builds, now that we have the new wheel mirrors in place. Graphite doesn't seem to have ther untimes in there though | 12:37 |
| krotscheck_dcm | ANyone familiar with how we've set up graphite, and can tell me if that data's there? | 12:38 |
| sdague | AJaeger: done | 12:38 |
| sdague | krotscheck_dcm: the data is not in graphite | 12:38 |
| *** miqui has quit IRC | 12:39 | |
| krotscheck_dcm | sdague: Hrm. | 12:39 |
| *** krotscheck_dcm is now known as krotscheck | 12:39 | |
| krotscheck | Well, to the logs I go then :) | 12:39 |
| *** thorst has joined #openstack-infra | 12:40 | |
| sdague | grabbing a random devstack log, it does look like it's working. We were running about 600s before on pip_install, now it's 180 | 12:41 |
| sdague | though, looks like a few things don't yet have wheels - http://logs.openstack.org/10/249110/10/gate/gate-tempest-dsvm-full/e9b7e3b//logs/devstacklog.txt.gz#_2016-02-08_11_36_03_777 | 12:42 |
| *** salv-orlando has joined #openstack-infra | 12:42 | |
| *** yamamoto has quit IRC | 12:42 | |
| *** yamamoto has joined #openstack-infra | 12:43 | |
| *** weshay_xchat has joined #openstack-infra | 12:43 | |
| *** jed56 has joined #openstack-infra | 12:44 | |
| *** gsagie has joined #openstack-infra | 12:44 | |
| *** jaosorior has joined #openstack-infra | 12:46 | |
| krotscheck | sdague: Yeah, we encountered an interesting situation on friday where a wheel was built for cpy27, and thus wasn't downloaded. | 12:47 |
| *** yamamoto has quit IRC | 12:48 | |
| krotscheck | sdague: MarkupSafe is one of them. http://mirror.bhs1.ovh.openstack.org/wheel/ubuntu-14.04-x86_64/MarkupSafe/ | 12:48 |
| krotscheck | I suspect that might be by design? I'ma ssuming that the "c" stands for "I compiled C things", which would be architecture dependent. | 12:50 |
| *** tcammann has quit IRC | 12:50 | |
| *** [1]Thelo has joined #openstack-infra | 12:51 | |
| *** jaosorior has quit IRC | 12:51 | |
| krotscheck | Now that doesn't matter to us, but it could matter to everyone-else-on-the-internet. | 12:51 |
| krotscheck | dstufft - what does the cpy27 in this wheel indicate? http://mirror.bhs1.ovh.openstack.org/wheel/ubuntu-14.04-x86_64/MarkupSafe/ | 12:52 |
| *** sdake has joined #openstack-infra | 12:52 | |
| dstufft | krotscheck: that wheel is specific to CPython 2.7 | 12:52 |
| *** Thelo has quit IRC | 12:53 | |
| *** [1]Thelo is now known as Thelo | 12:53 | |
| krotscheck | dstufft: I don't actually understand how that relates to, say, regular python. It was built automatically by our wheel mirror from a list of requirements. | 12:55 |
| krotscheck | dstufft: Should I be looking in setup.cfg? Our build script? Is it something we actually can't control at all? | 12:56 |
| dstufft | krotscheck: CPython is what most people mean by "Python" | 12:56 |
| dstufft | it's the standard implementation | 12:56 |
| dstufft | alternative implementations are PyPy, Jython, IronPython, etc. | 12:56 |
| krotscheck | Got it. So, I'm guessing, that when pip tries to resolve the wheel, either it's noticing that it's not runing on cpython, or it's not matching the name properly. | 12:57 |
| krotscheck | Given that we have a repeatable environment here, I suspect it's the latter? | 12:58 |
| *** amotoki has joined #openstack-infra | 12:58 | |
| *** yamamoto has joined #openstack-infra | 12:58 | |
| dstufft | is this running on CPython 2.7? | 12:58 |
| *** sdake has quit IRC | 12:58 | |
| krotscheck | dstufft: The wheel build or the jenkins node? | 12:59 |
| dstufft | jenkins | 12:59 |
| * krotscheck goes to build a jenknins slave to check | 12:59 | |
| *** abregman has quit IRC | 12:59 | |
| *** FallenPegasus has quit IRC | 12:59 | |
| dstufft | krotscheck: if you can reproduce, -vvv should give more information about why it's not using that wheel too | 13:00 |
| krotscheck | dstufft: I won't be able to fully confirm until this slave builds, but the jenkins slaves and the wheel slaves are cut from the same proveribal cloth. The thing that built it is the thing that consumes it. | 13:00 |
| dstufft | should be CPython 2.7 then | 13:01 |
| dstufft | I'd run it with -vvv to see what that says | 13:01 |
| krotscheck | Roger wilco | 13:01 |
| *** daemontool has joined #openstack-infra | 13:01 | |
| *** baoli has joined #openstack-infra | 13:02 | |
| *** dkranz has joined #openstack-infra | 13:05 | |
| *** krotscheck has quit IRC | 13:06 | |
| *** dtardivel has joined #openstack-infra | 13:06 | |
| *** dprince has joined #openstack-infra | 13:06 | |
| *** gal_ has joined #openstack-infra | 13:06 | |
| *** dizquierdo has joined #openstack-infra | 13:06 | |
| *** EricGonczer_ has joined #openstack-infra | 13:07 | |
| *** gsagie has quit IRC | 13:10 | |
| *** _nadya_ has quit IRC | 13:10 | |
| *** EricGonczer_ has quit IRC | 13:10 | |
| *** krotscheck1 has joined #openstack-infra | 13:11 | |
| * krotscheck1 peers at krotscheck. | 13:12 | |
| *** lucas-hungry is now known as lucasagomes | 13:12 | |
| *** gal_ is now known as gsagie | 13:12 | |
| *** davideagnello has joined #openstack-infra | 13:13 | |
| *** baoli has quit IRC | 13:14 | |
| Clint | krotscheck1: get him | 13:14 |
| *** nadya has joined #openstack-infra | 13:14 | |
| *** nadya is now known as Guest13826 | 13:15 | |
| *** amotoki has quit IRC | 13:15 | |
| *** xyang1 has joined #openstack-infra | 13:16 | |
| *** davideagnello has quit IRC | 13:18 | |
| dhellmann | AJaeger: now that we have content on releases.openstack.org it would be good to get the redirect in place so folks don't find the stale content at the old location: https://review.openstack.org/#/c/269831/ | 13:18 |
| *** krotscheck has joined #openstack-infra | 13:18 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Fix Bugs in Timeline Events https://review.openstack.org/277386 | 13:19 |
| *** links has quit IRC | 13:20 | |
| *** sdake has joined #openstack-infra | 13:20 | |
| jpr | question: if a jenkins job has a node restriction on it, does that require ZUUL_NODE to be set by parameter-function to the matching value so that jenkins can match the job description correctly? | 13:21 |
| *** claudiub has quit IRC | 13:21 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 13:22 |
| *** rfolco has joined #openstack-infra | 13:23 | |
| *** hashar has quit IRC | 13:24 | |
| krotscheck1 | sdague, dstufft, clarkb: Could not fetch URL http://mirror.bhs1.ovh.openstack.org/wheel/ubuntu-14.04-x86_64/markupsafe/: 404 Client Error: Not Found - skipping | 13:24 |
| krotscheck1 | Case sensitive. | 13:24 |
| dstufft | oh | 13:24 |
| dstufft | you need to normalize | 13:24 |
| krotscheck1 | Yep | 13:25 |
| dstufft | the full normalization rules are in PEP 503 fwiw | 13:25 |
| * krotscheck1 ponders the best place to do that. | 13:25 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Fix Bugs in Timeline Events https://review.openstack.org/277386 | 13:25 |
| *** yamamoto has quit IRC | 13:27 | |
| *** abregman has joined #openstack-infra | 13:29 | |
| *** Jeffrey4l has joined #openstack-infra | 13:31 | |
| *** edmondsw has joined #openstack-infra | 13:32 | |
| *** sdake_ has joined #openstack-infra | 13:33 | |
| *** flepied1 has quit IRC | 13:33 | |
| AJaeger | thanks, sdague | 13:35 |
| AJaeger | dhellmann: Let me review... | 13:35 |
| *** dtantsur is now known as dtantsur|brb | 13:35 | |
| *** sdake has quit IRC | 13:36 | |
| *** jcoufal has joined #openstack-infra | 13:36 | |
| AJaeger | krotscheck1: you can get runtimes out of grafana like done here http://grafana.openstack.org/dashboard/db/nodepool-blue-box - scroll to the bottom | 13:37 |
| *** tongli has joined #openstack-infra | 13:38 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Don't add non-existent items when resolving lists https://review.openstack.org/276303 | 13:38 |
| krotscheck1 | AJaeger: Thanks :) | 13:38 |
| *** trown has joined #openstack-infra | 13:40 | |
| krotscheck1 | AJaeger: Hrm. Good chunk of reduction, but not as impressive as I'd hoped. | 13:41 |
| mordred | krotscheck1: it's still totally solid | 13:41 |
| krotscheck1 | mordred: True. There's a noticable drop in runtime. | 13:42 |
| mordred | yah- and spread across all the builds, it should add up to additional daily node capacity | 13:43 |
| anteaya | mordred: morning, I find antique stores to be much more fun that large box stores but yay for having space | 13:43 |
| mordred | anteaya: yes indeed | 13:43 |
| *** jcoufal has quit IRC | 13:44 | |
| mordred | anteaya: I typically do not enjoy large box stores - but if one must large box store, then empty is better than full :) | 13:44 |
| anteaya | agreed | 13:44 |
| anteaya | so nice work on the timing of that | 13:44 |
| *** flepied has joined #openstack-infra | 13:46 | |
| *** trown has quit IRC | 13:47 | |
| *** trown has joined #openstack-infra | 13:48 | |
| *** krotscheck1 has quit IRC | 13:48 | |
| AJaeger | yes, thanks to everybody for that! | 13:50 |
| anteaya | morning AJaeger | 13:50 |
| anteaya | AJaeger: have a list? | 13:50 |
| anteaya | s/have a list/have you a list | 13:51 |
| AJaeger | anteaya: a long one for jeblair ;) to discuss the renaming ones. | 13:51 |
| AJaeger | anteaya: thanks for asking. Could you look at https://review.openstack.org/273436 and https://review.openstack.org/277063, please? | 13:51 |
| AJaeger | and this one, please https://review.openstack.org/276722 | 13:52 |
| anteaya | okey dokey | 13:52 |
| *** weshay_xchat has quit IRC | 13:52 | |
| *** nikil89_ has quit IRC | 13:52 | |
| AJaeger | mordred: should I +2 your multiverse removal from project-config? Change https://review.openstack.org/277174 | 13:53 |
| *** baoli has joined #openstack-infra | 13:53 | |
| *** baoli has quit IRC | 13:53 | |
| AJaeger | we didn't +A yet in case you wanted to babysit it... | 13:53 |
| *** davideagnello has joined #openstack-infra | 13:54 | |
| AJaeger | mordred: should we push the apt mirror update into today's nodepool build? Then let's ask anteaya for a second +2A.. https://review.openstack.org/#/c/277114/ | 13:54 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Normalize wheel folder structure https://review.openstack.org/277398 | 13:55 |
| *** baoli has joined #openstack-infra | 13:55 | |
| krotscheck | mordred: ^^ don't merge that just yet, I have to get a legacy script ready to go. | 13:56 |
| AJaeger | krotscheck: mark it as WIP for now so that it doesn't merge... | 13:56 |
| *** devananda has joined #openstack-infra | 13:56 | |
| dims | AJaeger : anteaya : please add this to your review queue - https://review.openstack.org/#/c/277086/ - the nova py27/py34 with oslo.* master works well. want to run other projects against oslo.* master now. | 13:57 |
| *** sdake_ has quit IRC | 13:57 | |
| *** sdake has joined #openstack-infra | 13:58 | |
| *** weshay_xchat has joined #openstack-infra | 13:58 | |
| *** weshay_xchat is now known as weshay | 13:58 | |
| *** davideagnello has quit IRC | 13:58 | |
| *** yamamoto has joined #openstack-infra | 13:58 | |
| AJaeger | dims: I'd like to have sdague for nova and one of the neutron liasons like dougwig, mestery, or armax review it as well first - and best also other folks for their projects. This is going into the integrated gate at one point, so let's get consensus early | 13:59 |
| *** sdake has quit IRC | 13:59 | |
| *** sdake has joined #openstack-infra | 14:00 | |
| *** baoli has quit IRC | 14:00 | |
| *** kushal has quit IRC | 14:00 | |
| anteaya | AJaeger: mordred I have +2'd 277114 but am not comfortable +A'ing it as I feel like a mirror tourist rather than driver, would prefer if a mirror driver +A'd | 14:00 |
| *** annegentle has joined #openstack-infra | 14:01 | |
| anteaya | dims: what AJaeger said | 14:01 |
| *** baoli has joined #openstack-infra | 14:01 | |
| AJaeger | anteaya: yeah, let's a mirror driver give his ok - and if mordred self-approves it's fine with me... | 14:01 |
| *** jsavak has joined #openstack-infra | 14:01 | |
| *** abregman has quit IRC | 14:01 | |
| AJaeger | dims: reviewing again, I have one concern: Once you put this into the gate, all the repos will be in the integrated gate - and thus you increased the number of repos in it ;( | 14:02 |
| mordred | krotscheck: you have shown me shell things that are above my meager understanding | 14:02 |
| AJaeger | dims: Discuss this with integrated gate experts like sdague and mtreinish as well... | 14:02 |
| mordred | krotscheck: what does <<< do? | 14:02 |
| *** tiswanso has joined #openstack-infra | 14:02 | |
| *** rossella_s has quit IRC | 14:02 | |
| AJaeger | anteaya: thanks for all the reviews! | 14:03 |
| openstackgerrit | Merged openstack-infra/project-config: Rename stackforge-specs-jobs to build-only-specs-jobs https://review.openstack.org/273436 | 14:03 |
| anteaya | AJaeger: welcome | 14:03 |
| *** rossella_s has joined #openstack-infra | 14:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Re-enable ironic-ui translation https://review.openstack.org/277063 | 14:04 |
| * anteaya goes back to skimming the email list | 14:04 | |
| *** gsagie has quit IRC | 14:05 | |
| *** tiswanso has quit IRC | 14:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Do not publish openstackdocstheme documents https://review.openstack.org/276722 | 14:05 |
| *** tiswanso has joined #openstack-infra | 14:06 | |
| mordred | AJaeger: let's give the mailing list post a little more time to uncover any hostility | 14:06 |
| *** jsavak has quit IRC | 14:07 | |
| *** kushal has joined #openstack-infra | 14:08 | |
| *** Jeffrey4l has quit IRC | 14:08 | |
| anteaya | jaypipes: nice work crafting a lovely gender neutral mailing list post, I appreciate it | 14:08 |
| krotscheck | mordred: It works? | 14:09 |
| * krotscheck may have stackoverflowed that one. | 14:09 | |
| AJaeger | mordred: works for me... | 14:09 |
| anteaya | jaypipes: I refer to http://lists.openstack.org/pipermail/openstack-dev/2016-February/086007.html | 14:09 |
| prometheanfire | ianw: does the finalize stage happen before the sytem is deployed to the target (raw/qcow)? | 14:09 |
| *** YorikSar has quit IRC | 14:09 | |
| *** notnownikki has joined #openstack-infra | 14:09 | |
| mordred | krotscheck: :) | 14:10 |
| *** jsavak has joined #openstack-infra | 14:10 | |
| *** YorikSar has joined #openstack-infra | 14:10 | |
| *** jaosorior has joined #openstack-infra | 14:11 | |
| dims | anteaya : AJaeger : how do i set it up so it's not in integrated-gate? all i need is periodic jobs that run every day | 14:11 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Normalize wheel folder structure https://review.openstack.org/277398 | 14:11 |
| *** naresht has quit IRC | 14:11 | |
| *** rlandy has joined #openstack-infra | 14:11 | |
| *** dtantsur|brb is now known as dtantsur | 14:12 | |
| AJaeger | dims: if it will be periodic, no worries... | 14:12 |
| AJaeger | only if they share the gate | 14:13 |
| dims | AJaeger : yep. i don't need this to run on check or gate...or vote | 14:13 |
| anteaya | AJaeger: my sense from the jobs that dims is creating is that they are designed to be periodic only | 14:13 |
| anteaya | AJaeger: it is a good concern to raise and I'm glad we are talking about it | 14:14 |
| dims | anteaya : yes, the only reason i added them to experimental was to make sure i can test it | 14:14 |
| dims | anteaya : AJaeger : once i kick the tires, we can drop that | 14:14 |
| anteaya | but as you point out the integrated bits affect the gate queue only | 14:14 |
| *** jsavak has quit IRC | 14:14 | |
| anteaya | dims: drop experimental you mean | 14:14 |
| AJaeger | yes, I stand corrected - so should be fine and probably won't need buy-in - but let's still give a heads up to nova and neutron teams - especially neutron asks for review by their liasons | 14:14 |
| *** jsavak has joined #openstack-infra | 14:14 | |
| dims | anteaya : yes | 14:14 |
| anteaya | dims: great | 14:14 |
| *** abregman has joined #openstack-infra | 14:14 | |
| anteaya | AJaeger: yup, buy in is good, glad we had a moment to discuss it | 14:15 |
| anteaya | dims: can you find folks in the other projects to review? | 14:15 |
| AJaeger | dims: I'll gave a +2 but added a comment as well... | 14:16 |
| *** Guest13826 has quit IRC | 14:16 | |
| mordred | jeblair, fungi: when you awaken, I could use a +2 on https://review.openstack.org/#/c/277172 and its immediate ancestor | 14:16 |
| dims | anteaya : this is mainly for oslo team to make sure our code does not break other projects. i am running stuff on travis to do that right now (https://travis-ci.org/dims/) and would like to get rid of my travis setup | 14:17 |
| mordred | I have performed the file moves locally | 14:17 |
| dims | AJaeger : who is the neutron liaison? i can ping | 14:17 |
| *** nadya has joined #openstack-infra | 14:18 | |
| anteaya | dims: armax or dougwig are good | 14:18 |
| dims | thanks anteaya | 14:18 |
| anteaya | thank you | 14:18 |
| *** nadya is now known as Guest56910 | 14:18 | |
| anteaya | dims: I'm not saying i think they will have an issue with what you propose | 14:19 |
| anteaya | but I think we are so large that many folks have things pop up that affect them that then never even knew about | 14:19 |
| *** annegentle has quit IRC | 14:19 | |
| anteaya | so a courtesty heads up goes a long way to reducing stress, I find | 14:19 |
| *** annegentle has joined #openstack-infra | 14:20 | |
| *** jaosorior has quit IRC | 14:21 | |
| *** davideagnello has joined #openstack-infra | 14:21 | |
| *** jaosorior has joined #openstack-infra | 14:21 | |
| *** vsaienko has quit IRC | 14:21 | |
| dims | anteaya : good point | 14:22 |
| mordred | dims: also, I support getting your stuff off of travis :) | 14:22 |
| dims | thanks mordred :) | 14:22 |
| *** hashar has joined #openstack-infra | 14:23 | |
| anteaya | yeah that too | 14:23 |
| *** kgiusti has joined #openstack-infra | 14:24 | |
| mordred | jeblair: also - when you awaken, I'm confused by some permissions - putting the lists and db files into AFS and then running reprepro I get Error 13 creating lock file '/afs/.openstack.org/mirror/apt/db/lockfile': Permission denied! | 14:24 |
| mordred | jeblair: however, /afs/.openstack.org/mirror/apt/db has the same permissions as the other dirs | 14:25 |
| mordred | jeblair: (using fs la of course) | 14:25 |
| *** davideagnello has quit IRC | 14:26 | |
| AJaeger | I blogged about adding jobs to our CI at http://jaegerandi.blogspot.de/2016/02/creating-new-test-jobs-in-openstack-ci.html and will do a followup about templates. If you find anything there that is a good idea to add to the infra-manual, feel free to propose a change for it or tell me exactly what to propose... | 14:26 |
| *** sabeen has joined #openstack-infra | 14:26 | |
| dims | anteaya : AJaeger : i think i am missing some bit as i don't see entries for periodic-nova-py27-with-oslo-master in http://logs.openstack.org/periodic/ | 14:27 |
| dims | anteaya : AJaeger : i was able to run that as experimental on a nova review, just don't see the daily run that was supposed to happen | 14:27 |
| *** yamamoto has quit IRC | 14:28 | |
| AJaeger | dims: found th eproblem: branch: master is a no-go | 14:28 |
| AJaeger | periodic jobs cannot have a branch selection, send a patch to remove that please | 14:29 |
| AJaeger | You have to tell your job that it's on master... | 14:29 |
| AJaeger | time for home made waffels, I'll be back later... | 14:29 |
| dims | AJaeger : ack | 14:30 |
| *** sabeen2 has joined #openstack-infra | 14:30 | |
| *** vincentfrancoise has joined #openstack-infra | 14:31 | |
| *** adrienverge has quit IRC | 14:31 | |
| *** esker has quit IRC | 14:31 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/project-config: Rename tripleo ceph job into upgrades https://review.openstack.org/277419 | 14:32 |
| *** regXboi has joined #openstack-infra | 14:33 | |
| *** sabeen has quit IRC | 14:33 | |
| anteaya | AJaeger: mmmmm waffels | 14:35 |
| *** jsavak has quit IRC | 14:35 | |
| vincentfrancoise | Hi everyone! I have a patchset (see https://review.openstack.org/#/c/274068/) that is waiting for reviews since last week. So if you could review it, that would be much appreciated | 14:35 |
| *** yamahata has quit IRC | 14:35 | |
| *** jsavak has joined #openstack-infra | 14:36 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 14:37 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Drop branch selection on oslo-master periodic job https://review.openstack.org/277421 | 14:37 |
| *** sc68cal has joined #openstack-infra | 14:38 | |
| *** salv-orlando has quit IRC | 14:41 | |
| anteaya | vincentfrancoise: one way of helping us reduce our review queue and get to your patch is to start reviewing patches yourself, reading this may be of interest to you: http://anteaya.info/blog/2013/03/21/reviewing-an-openstack-patch/ | 14:44 |
| *** ccrouch has joined #openstack-infra | 14:44 | |
| fungi | mordred: a little searching suggests that some kinds of file locking get weird on afs (as they do on nfs and other network filesystems), so i wonder if it's some other problem with the lock surfacing as a permissions issue? | 14:45 |
| *** superdan is now known as dansmith | 14:45 | |
| *** adrienverge has joined #openstack-infra | 14:47 | |
| krotscheck | Well, piffle. | 14:47 |
| krotscheck | fungi, mordred: It looks like my naïve "Get distribution name by dash-split" method resulted in some of our packages being misfiled in the wheel mirror. | 14:47 |
| openstackgerrit | Merged openstack-infra/system-config: Update reprepro options file with valid options https://review.openstack.org/277117 | 14:47 |
| mordred | fungi: ah. duh | 14:47 |
| mordred | fungi: so maybe keeping the db files in AFS is a bad idea and we should just store them locally on the mirror-update host | 14:48 |
| krotscheck | UPside: We can squeeze more performance out of wheel! | 14:48 |
| krotscheck | Downside: We kinda have to resort everything. | 14:48 |
| pabelanger | clarkb: greghaynes: Ya, I plan to look into fedora / centos mirrors this week. Mostly to see what AFS is all about. I don't think anybody else is working on it, but correct me if I am wrong | 14:48 |
| *** davideagnello has joined #openstack-infra | 14:48 | |
| mordred | pabelanger: I was going to work on it if nobody else did - so I'm happy to work with you on it when you get there | 14:49 |
| fungi | mordred: or maybe keeping the lockfile on the builder is sufficient (we don't expect to have additional mirror builders attempting concurrent access to the reprepro db, right?) | 14:49 |
| fungi | but still, occam says we're probably just missing something permissions wise | 14:49 |
| mordred | fungi: there does not seem to be a way to configure lockfile location separate from db location | 14:50 |
| *** nelsnelson has quit IRC | 14:50 | |
| vincentfrancoise | anteaya: ok, I'll do as you suggest | 14:50 |
| pabelanger | mordred: Sure. Give me a few hours to clean up some other stuff. I still need to try out the afs client | 14:51 |
| mordred | pabelanger: no rush - still finishing off apt mirrors | 14:51 |
| *** EricGonczer_ has joined #openstack-infra | 14:51 | |
| fungi | mordred: https://lists.openafs.org/pipermail/openafs-info/2010-January/032715.html looks possibly related | 14:51 |
| jeblair | what does it try to do with the lockfile? | 14:52 |
| fungi | no, nevermind that's error 11 not 13 | 14:52 |
| pabelanger | mordred: great. Ya, I'll be sure to see how that all works first | 14:52 |
| *** sridhar_ram has joined #openstack-infra | 14:52 | |
| *** sdake has quit IRC | 14:52 | |
| *** annegentle has quit IRC | 14:53 | |
| jeblair | mordred: can you help me run it in screen to get the error? | 14:53 |
| *** davideagnello has quit IRC | 14:53 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use --overcloud-update for the upgrades job https://review.openstack.org/260466 | 14:53 |
| jeblair | mordred: i have rejoined your screen session | 14:53 |
| mordred | jeblair: neat! | 14:54 |
| mordred | jeblair: one sec | 14:54 |
| jeblair | (oh wow, that still hasn't landed :) | 14:54 |
| *** salv-orlando has joined #openstack-infra | 14:55 | |
| mordred | k. seems to just be a chmod needed on the dir :) | 14:55 |
| mordred | jeblair: so let's land those changes real quick | 14:55 |
| *** amitgandhinz has joined #openstack-infra | 14:55 | |
| fungi | which afs-related changes did i miss? | 14:55 |
| krotscheck | jeblair: THere's a thing in backscroll that's a little disconcerting. | 14:55 |
| anteaya | vincentfrancoise: thank you, do ask if you have any questions as you learn how to review | 14:55 |
| fungi | thought i had reviewed most of them | 14:55 |
| *** abregman has quit IRC | 14:55 | |
| mordred | jeblair. fungi: first one just landed 7 minutes ago | 14:56 |
| fungi | k | 14:56 |
| mordred | jeblair: https://review.openstack.org/#/c/277172/ | 14:56 |
| fungi | a few of them in the series you mentioned earlier still need an additional +2 besides me | 14:56 |
| mordred | yah | 14:56 |
| fungi | so those i guess | 14:56 |
| *** esikachev has quit IRC | 14:57 | |
| fungi | i can switch my -1 on 277189 to +2 if you want to tackle my comments in a separate change | 14:57 |
| fungi | what you have will technically work, just could be better for the long-term | 14:57 |
| vincentfrancoise | anteaya: ok will do | 14:58 |
| mordred | fungi: I could go either way - whatever you prefer | 14:58 |
| mordred | fungi: however, I'm wondering if maybe I should not import any keys for debian until we decide to run a debian mirror | 14:58 |
| mordred | since debian has many keys | 14:58 |
| krotscheck | Wait! I am wrong. | 14:58 |
| anteaya | vincentfrancoise: thank you | 14:58 |
| krotscheck | It's a little odd though. Python package names have to be valid identifiers, meaning they can't contain dashes, but the normalized pep503 repository folder name requires the use of dashes. | 14:59 |
| fungi | mordred: i agree, how about strip that down to just the ubuntu key and switch it to a full fingerprint and i'll +2 | 14:59 |
| *** vgridnev has quit IRC | 14:59 | |
| *** esikachev has joined #openstack-infra | 14:59 | |
| jeblair | krotscheck, sdague: job runtimes are _definitely_ in graphite | 15:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Import gpg keys for mirroring into local keyring https://review.openstack.org/277189 | 15:00 |
| mordred | fungi: how about that | 15:00 |
| * anteaya is very depressed as she has to set a mouse trap :( | 15:00 | |
| jeblair | krotscheck, sdague: also, i have seen devstack-reported pip_install times go from ~20 minutes to 2. | 15:00 |
| *** krtaylor has joined #openstack-infra | 15:01 | |
| fungi | mordred: lgtm | 15:01 |
| sdague | jeblair: yeh, there is definitely an impact when I'm looking at spot timings | 15:01 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:01 | |
| fungi | anteaya: they make non-lethal mouse traps if you'd rather catch and release, but you have to take them pretty far if you don't want them finding their way back | 15:02 |
| *** nmagnezi has quit IRC | 15:02 | |
| jeblair | sdague: when SpamapS finishes his work, we may be able to easily dump pip_install, etc, timings into graphite | 15:02 |
| sdague | jeblair: yeh, I think that was an intent by mtreinish / SpamapS | 15:02 |
| *** egonzalez has joined #openstack-infra | 15:03 | |
| jeblair | sdague: ah cool; i knew about the actual openstack metrics, didn't know the devstack phases had been thought of | 15:03 |
| *** piet has joined #openstack-infra | 15:03 | |
| jeblair | mordred: i guess the debian key id was wrong in earlier versions of 2B90D010 ? | 15:04 |
| openstackgerrit | Merged openstack-infra/storyboard: Don't add non-existent items when resolving lists https://review.openstack.org/276303 | 15:05 |
| eantyshev | anteaya, jeblair: May I ask you again to review https://review.openstack.org/272123 , please? This fixes some broken Zuul functionality | 15:05 |
| jeblair | mordred: why move the db and lists to afs? | 15:05 |
| *** tdurakov has joined #openstack-infra | 15:06 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use --overcloud-update in the upgrades job and ceph in the nonha https://review.openstack.org/260466 | 15:09 |
| anteaya | fungi: thanks, yeah it would just end up in my basement again :( | 15:09 |
| *** jaypipes has quit IRC | 15:09 | |
| *** arxcruz has quit IRC | 15:09 | |
| anteaya | fungi: mostly I'm at odds with my cat who is wonderful at catching mice outside, apparently inside doesn't interest her | 15:09 |
| jeblair | krotscheck: https://review.openstack.org/277398 lgtm | 15:09 |
| krotscheck | jeblair: It's broken | 15:09 |
| zxiiro | electrofelix: | 15:10 |
| jeblair | krotscheck: still lgtm. :) | 15:10 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha and upgrdes job https://review.openstack.org/273424 | 15:10 |
| zxiiro | electrofelix: Have you seen my latest comments regarding the mapping patch? | 15:10 |
| krotscheck | jeblair: I'm replacing all special characters in package names _before_ splitting on '-' | 15:10 |
| krotscheck | jeblair: WHich means that xstatic_foo_bar turns into xstatic-foo-bar and the script things the name is xstatic. | 15:10 |
| krotscheck | Mind you, having all xstatic packages be undiscoverable is something that I actually would condone. | 15:11 |
| krotscheck | But, well, the sideffect isnt' so hot | 15:11 |
| krotscheck | Anyway | 15:11 |
| krotscheck | I'm testing a fix right now. | 15:11 |
| zxiiro | electrofelix: actually I think I'll make a mailing list post as this is an important topic in my opinion. We are being hit buy this use case recently. | 15:11 |
| jeblair | krotscheck: gotcha | 15:11 |
| jeblair | mordred, fungi: are we ready for https://review.openstack.org/277114 ? | 15:12 |
| *** ajmiller has joined #openstack-infra | 15:13 | |
| *** eharney has joined #openstack-infra | 15:13 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Normalize wheel folder structure https://review.openstack.org/277398 | 15:13 |
| krotscheck | jeblair: ^^ is the correct version. | 15:13 |
| openstackgerrit | Merged openstack-infra/system-config: Update paths for AFS mirror https://review.openstack.org/277172 | 15:14 |
| krotscheck | jeblair: That'll require some cleanup for the mirrors. Do you think it'll be safe to flat-out delete any wheels that don't have normalized folder names? | 15:14 |
| krotscheck | Or should I copy them to their new location. | 15:14 |
| zigo | fungi: Hi. Could I ask you to review https://review.openstack.org/#/c/264726/ (debian image patch) ? It's stalled with nobody reviewing... :( | 15:15 |
| jeblair | krotscheck: delete should be fine -- the first mirror run with that change will put them in place. | 15:15 |
| *** annegentle has joined #openstack-infra | 15:15 | |
| *** ildikov has quit IRC | 15:16 | |
| *** zeih_ has quit IRC | 15:17 | |
| jeblair | eantyshev: lgtm | 15:18 |
| *** rbrndt_ has joined #openstack-infra | 15:18 | |
| eantyshev | jeblair: any further steps from me expected? | 15:18 |
| jeblair | eantyshev: no, thanks! it's approved (but let me know if it doesn't merge) | 15:19 |
| eantyshev | jeblair: thanks! | 15:19 |
| mordred | jeblair: I thnk I'd like for the cron to be running and us happy with it before throwing the configure-mirror switch | 15:21 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for lbaas v2 dashboard https://review.openstack.org/275394 | 15:21 |
| jeblair | mordred: ok | 15:21 |
| mordred | jeblair: as for why move them to AFS - it's either AFS or local /var ... but if we move them to local var and then mirror-update goes away, we'll have to rebuild the mirror from scratch | 15:22 |
| jeblair | mordred: got it | 15:22 |
| *** arxcruz has joined #openstack-infra | 15:22 | |
| *** esikachev has quit IRC | 15:22 | |
| *** vgridnev has joined #openstack-infra | 15:23 | |
| *** nelsnelson has joined #openstack-infra | 15:24 | |
| mordred | krotscheck: it's safe to land that script patch yeah? or do we need to coordinate the cleanup? | 15:24 |
| *** amotoki has joined #openstack-infra | 15:24 | |
| krotscheck | mordred: Landing it won't break anything. | 15:24 |
| mordred | ossum | 15:24 |
| mordred | +a then | 15:24 |
| krotscheck | mordred: woot | 15:25 |
| openstackgerrit | Igor Belikov proposed openstack/diskimage-builder: Install additional packages in debian-minimal https://review.openstack.org/277445 | 15:26 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add ZFSonLinux experimental Tempest job for Manila https://review.openstack.org/277446 | 15:26 |
| *** rossella_ has joined #openstack-infra | 15:26 | |
| mordred | fungi: 2016-02-08 15:14:01.516 | Error: Parameter key_id failed on Gnupg_key[Ubuntu Archive]: Invalid key id 0x630239CC130E1A7FD81A27B140976EAF437D05B5 at /home/jenkins/workspace/gate-infra-puppet-apply-bare-trusty/openstack-infra/system-config/modules/openstack_project/manifests/mirror_update.pp:101 | 15:26 |
| AJaeger | jeblair: I reworked the timeout changes as suggested: First upgrade timout to 10 mins for merged change: 277155, then update d-g to 10mins: 277154 - and updated the three remaining timeout changes to include the increase: 276859, 277061, 277069. Could you review these laster, please? | 15:27 |
| *** haypo has left #openstack-infra | 15:27 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add ZFSonLinux experimental Tempest job for Manila https://review.openstack.org/277446 | 15:27 |
| jeblair | AJaeger: will do asap | 15:27 |
| *** rossella_s has quit IRC | 15:28 | |
| fungi | mordred: try dropping the 0x i guess | 15:28 |
| mordred | fungi: seems it only accepts 8 or 16 character IDs | 15:28 |
| fungi | huhwha? | 15:28 |
| mordred | fungi: https://github.com/n1tr0g/golja-gnupg#key_id | 15:28 |
| *** egonzalez has quit IRC | 15:28 | |
| * mordred reading source | 15:28 | |
| fungi | wow, that's terrible | 15:28 |
| fungi | also i got to thinking, it's sort of silly having a puppet module retrieve keys from a keyserver when you already know the key you want. does that module allow us to just supply the public key? | 15:29 |
| jeblair | if not terrible, it's extremely unlikely to be great. | 15:29 |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Add debian-jessie image to nodepool https://review.openstack.org/264726 | 15:29 |
| *** davideagnello has joined #openstack-infra | 15:29 | |
| mordred | fungi: k. it seems like it's just doing passthrough: gpg --fingerprint --with-colons #{resource[:key_id]} | 15:29 |
| mordred | fungi: so removing the 0x may work | 15:29 |
| mordred | fungi: also, yes, it does allow passing in content | 15:30 |
| fungi | yeah, try dropping the 0x for now i guess. i tend to use it to be explicit, but that may be a gpg2-onlyism | 15:30 |
| krotscheck | jeblair, mordred, fungi: http://paste.openstack.org/show/486293/ will remove all the packages that don't match the normalized names. The impact of this is minimal, since the jenkins slaves can't find those packages anyway. | 15:30 |
| mordred | fungi: https://github.com/n1tr0g/golja-gnupg#add-public-key-d50582e6-from-puppet-fileserver-to-user-bar-via-a-string-value | 15:30 |
| krotscheck | We'll want to run that after https://review.openstack.org/#/c/277398/2 merges | 15:31 |
| jeblair | krotscheck: ++ fungi do you want to exercise your afs perms, or should i run that? | 15:31 |
| *** egonzalez has joined #openstack-infra | 15:31 | |
| fungi | jeblair: i'm in a meeting for the next half hour, but i can try it after i'm done | 15:31 |
| jeblair | fungi: we need to wait a bit anyway | 15:31 |
| *** zz_dimtruck is now known as dimtruck | 15:31 | |
| fungi | okay, i'll take a shot at it in ~30 | 15:32 |
| krotscheck | note, it won't delete summary directories that have lowercase names. But _shrug_. | 15:32 |
| krotscheck | Those'll come back as soon as those packages are rebuilt | 15:32 |
| krotscheck | So I didn't see much point to deleting them. | 15:32 |
| *** hdd has joined #openstack-infra | 15:33 | |
| *** achanda has joined #openstack-infra | 15:33 | |
| *** davideagnello has quit IRC | 15:34 | |
| mordred | fungi: it's a bug in the puppet module: https://github.com/n1tr0g/golja-gnupg/blob/master/lib/puppet/type/gnupg_key.rb#L136 | 15:34 |
| *** jaypipes has joined #openstack-infra | 15:34 | |
| mordred | fungi: overeager validation code | 15:34 |
| fungi | ouch | 15:35 |
| *** annegentle has quit IRC | 15:35 | |
| *** annegentle has joined #openstack-infra | 15:36 | |
| mordred | fungi: we've actually already imported the key on the server, so this is brace and suspenders anyway - how about I put in the 16-char version, then submit a patch upstream to accept the longer id | 15:36 |
| *** claudiub has joined #openstack-infra | 15:36 | |
| fungi | mordred: yeah, i guess for now put in whatever works, and later we can consider just puppeting in the key material instead (so that our puppet isn't dependent on being able to connect to the keyserver) | 15:37 |
| *** mtanino has joined #openstack-infra | 15:37 | |
| fungi | that also solves the hkps:// dilemma, because we can retrieve the key manually via an authenticated protocol and then just puppet the blob | 15:38 |
| *** esker has joined #openstack-infra | 15:38 | |
| *** esker has quit IRC | 15:39 | |
| jeblair | yeah, in my experience, depending on keyservers is :( | 15:39 |
| fungi | (public service announcement: normal pgp keyserver protocol does not validate the connection, which makes it hard to bootstrap a web of trust) | 15:39 |
| *** esker has joined #openstack-infra | 15:39 | |
| openstackgerrit | Igor Belikov proposed openstack-infra/project-config: Add debian-jessie image to nodepool https://review.openstack.org/264726 | 15:40 |
| krotscheck | Wait, we already have afs mirros? | 15:40 |
| *** jsavak has quit IRC | 15:40 | |
| krotscheck | Sorry, I meant APT mirrors. | 15:40 |
| krotscheck | That was fast. | 15:40 |
| jeblair | krotscheck: yep! | 15:40 |
| fungi | krotscheck: in fairness, mordred had most of it already written in various states of decay | 15:41 |
| jeblair | krotscheck: http://mirror.nyj01.internap.openstack.org/apt/ | 15:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Import gpg keys for mirroring into local keyring https://review.openstack.org/277189 | 15:41 |
| mordred | fungi, jeblair: updated with 16-char for now | 15:41 |
| jeblair | fungi: i think he only started writing it when we started on the afs pypi mirrors, but yeah, not *strictly* linear | 15:41 |
| *** vincentfrancoise has left #openstack-infra | 15:41 | |
| krotscheck | jeblair: Neat! How much of a job improvement have we seen? | 15:41 |
| jeblair | krotscheck: oh, not in prod yet | 15:42 |
| *** rcernin has quit IRC | 15:42 | |
| mordred | it was easy once krotscheck and jeblair had done all the hard bits | 15:42 |
| anteaya | if an infra-root has a moment, jenkins06 doesn't seem to be part of the team effort currently: http://grafana.openstack.org/dashboard/db/nodepool | 15:42 |
| *** ddecapit has joined #openstack-infra | 15:42 | |
| jeblair | krotscheck: they exist, but we haven't flipped the configure-mirror switch yet | 15:42 |
| *** ddecapit is now known as DuaneDeC7 | 15:42 | |
| *** doug-fish has joined #openstack-infra | 15:42 | |
| fungi | mordred: oh, i thought you already had some earlier patches for setting up deb mirrors | 15:42 |
| mordred | krotscheck: you can look at the run https://review.openstack.org/#/c/272215/ | 15:42 |
| openstackgerrit | Merged openstack-infra/project-config: Enable multinode testing in Watcher https://review.openstack.org/274068 | 15:42 |
| mordred | krotscheck: to see devstack running against them | 15:42 |
| krotscheck | I was just doing that | 15:43 |
| jeblair | krotscheck: i'm not expecting a noticeably improvement in runtime (maybe i'll be surprised?), but we are expecting improvements in reliability. | 15:43 |
| *** achanda has quit IRC | 15:43 | |
| mordred | yah | 15:43 |
| mordred | same here | 15:43 |
| fungi | but yeah, it just seems especially silly to have configuration management retrieve a key from a keyserver via key id (if you have the key id you already know what key you need), especially when you don't need any other metadata about they key either (reprepro isn't validating other signatures of that key for example) | 15:43 |
| *** tzn has quit IRC | 15:43 | |
| mordred | fungi: now - there is another discussion we shoudl maybe have | 15:43 |
| mordred | fungi: which is "do we want to sign the repo we're publishing" | 15:43 |
| AJaeger | dhellmann: could you reveiw solum's reno addition again, please? https://review.openstack.org/243295 | 15:44 |
| mordred | fungi: my current stance is "no" because that makes these mirrors less desirable for people who are not our gate | 15:44 |
| AJaeger | dhellmann: the redirect to release.openstack.org works... | 15:44 |
| mordred | fungi: however, once we have them, there's no good reason to not also use them for our long-lived servers ... at which point I think having a signed repo wouldn't be terrible | 15:44 |
| fungi | mordred: i guess that depends on how much we want to trust the jobs we're running on workers relying on them, and whether we want to use those mirrors for our other servers too | 15:45 |
| krotscheck | So. Many. Mirrors | 15:45 |
| fungi | yeah, that | 15:45 |
| fungi | mordred: i was honestly surprised you could turn secure-apt off and ignore the lack of signing | 15:46 |
| *** amotoki has quit IRC | 15:46 | |
| fungi | it's been default for so long i just assumed the old mode had been disabled years ago | 15:46 |
| jeblair | fungi, mordred: as they stand -- i don't think we need to use these for our own servers. when we start dropping our own packages in them, that changes my answer. :) | 15:46 |
| mordred | jeblair: yah. and those would be more interesting signatures too | 15:47 |
| *** Sukhdev has joined #openstack-infra | 15:47 | |
| jeblair | mordred: we could make that a separate archive though, right? | 15:47 |
| mordred | jeblair: yes. we could make that a separate archive | 15:47 |
| mordred | jeblair: although we could also make it just a separate component in this archive | 15:48 |
| mordred | jeblair: so we'd have trusty trusty-updates trusty-backports trusty-infra for instance | 15:48 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Revert "Enable voting for Puppet OpenStack Integration jobs" https://review.openstack.org/277457 | 15:48 |
| *** jsavak has joined #openstack-infra | 15:48 | |
| EmilienM | dmsimard: ^ | 15:48 |
| mordred | jeblair: haven't thought through that fully yet since it's not really an urgent for us | 15:48 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize wheel folder structure https://review.openstack.org/277398 | 15:49 |
| *** ildikov has joined #openstack-infra | 15:49 | |
| EmilienM | I have an outstanding request for infra, puppet group would like to disable voting for some jobs, just for today & maybe tomorrow. Please review https://review.openstack.org/277457 | 15:49 |
| *** links has joined #openstack-infra | 15:49 | |
| jeblair | mordred: ya | 15:49 |
| *** mikelk has joined #openstack-infra | 15:50 | |
| dhellmann | AJaeger : thanks! | 15:50 |
| dhellmann | AJaeger : looking at the solum change now | 15:50 |
| *** tzn has joined #openstack-infra | 15:52 | |
| *** egonzalez has quit IRC | 15:52 | |
| *** egonzalez has joined #openstack-infra | 15:53 | |
| mordred | jeblair: hrm. I got the lockfile permission error again | 15:53 |
| *** tzn has quit IRC | 15:53 | |
| *** aspiers has quit IRC | 15:53 | |
| *** tzn has joined #openstack-infra | 15:53 | |
| *** pradk has joined #openstack-infra | 15:54 | |
| *** jsavak has quit IRC | 15:54 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 15:55 |
| *** Daisy has quit IRC | 15:56 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Fix for stable/liberty job https://review.openstack.org/277462 | 15:56 |
| *** davideagnello has joined #openstack-infra | 15:56 | |
| fungi | mordred: and you've confirmed there's no existing file there, right? | 15:57 |
| mordred | Shrews: ^^ you're going to want to move not copy | 15:57 |
| anteaya | EmilienM: I commented, you need to have some job running for your gate queue | 15:57 |
| mordred | Shrews: since ~ overrides /etc | 15:57 |
| *** links has quit IRC | 15:57 | |
| Shrews | mordred: if it had found ~, it wouldn't even be a problem :) | 15:57 |
| Shrews | mordred: different user (stack vs. jenkins) | 15:57 |
| EmilienM | anteaya: that's a revert, it was like this before | 15:57 |
| mordred | Shrews: isn't it the sed that's the issue though? | 15:57 |
| *** esikachev has joined #openstack-infra | 15:58 | |
| Shrews | mordred: yes, b/c it couldn't find the file in /etc | 15:58 |
| EmilienM | anteaya: see https://review.openstack.org/#/c/273768/ | 15:58 |
| *** aysyd has quit IRC | 15:58 | |
| mordred | Shrews: right. but if you copy a file to etc and then sed it - the tests are going to still read the one in ~ and then they'll be running against the wrong keystone api version | 15:58 |
| *** jsavak has joined #openstack-infra | 15:58 | |
| *** sridhar_ram has quit IRC | 15:58 | |
| Shrews | mordred: the tests can't SEE the one in ~ | 15:59 |
| *** aysyd has joined #openstack-infra | 15:59 | |
| mordred | Shrews: 0_o | 15:59 |
| anteaya | sc68cal: go go go | 15:59 |
| EmilienM | anteaya: we have some jobs in gate | 15:59 |
| *** vgridnev has quit IRC | 15:59 | |
| Shrews | mordred: b/c the tests run as the jenkins user, but the file is written to the stack user's ~ | 15:59 |
| EmilienM | anteaya: see https://review.openstack.org/#/c/272306/ for example, we have lint & syntax | 15:59 |
| sc68cal | anteaya: hmm? | 15:59 |
| openstackgerrit | Michael Johnson proposed openstack/requirements: Octavia requires fixes in taskflow 1.26.0 to function https://review.openstack.org/276549 | 15:59 |
| mordred | Shrews: k. | 15:59 |
| anteaya | sc68cal: the mailing list post, go you | 15:59 |
| sc68cal | anteaya: ah. I thought so. Thanks :) | 16:00 |
| *** jecarey has joined #openstack-infra | 16:00 | |
| *** EricGonc_ has joined #openstack-infra | 16:00 | |
| anteaya | sc68cal: :) | 16:00 |
| *** harlowja_at_home has joined #openstack-infra | 16:00 | |
| anteaya | EmilienM: I'm missing how lint and syntax are running on that repo, however yes, those jobs would be sufficient to have something running in the gate to merge patches | 16:01 |
| *** davideagnello has quit IRC | 16:01 | |
| Shrews | mordred: $BASE/new is not ~jenkins ... if that makes sense | 16:01 |
| anteaya | you must have some meta template or something | 16:01 |
| EmilienM | anteaya: let me show you the code, a sec | 16:01 |
| *** esikachev has quit IRC | 16:02 | |
| AJaeger | anteaya: the jobs use more than one template | 16:02 |
| *** infra-red has quit IRC | 16:02 | |
| EmilienM | and anyway, it's a revert | 16:03 |
| *** vgridnev has joined #openstack-infra | 16:03 | |
| *** EricGonczer_ has quit IRC | 16:03 | |
| EmilienM | https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/projects.yaml#L5514 | 16:03 |
| *** markmcd has quit IRC | 16:03 | |
| *** markmcd has joined #openstack-infra | 16:04 | |
| anteaya | EmilienM: no I get it, you are changing the template | 16:04 |
| *** trown|brb has joined #openstack-infra | 16:04 | |
| *** markmcd1 has joined #openstack-infra | 16:04 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds swagger migrate and copy job for api-site repo https://review.openstack.org/269809 | 16:04 |
| anteaya | puppet has a lot of templates | 16:04 |
| anteaya | I thought you were removing the jobs from a repo | 16:04 |
| *** ajmiller_ has joined #openstack-infra | 16:05 | |
| EmilienM | anteaya: thanks! no worries | 16:05 |
| anteaya | sorry I was confused | 16:06 |
| *** placko has quit IRC | 16:06 | |
| *** esikachev has joined #openstack-infra | 16:07 | |
| EmilienM | anteaya: it's monday :-) | 16:07 |
| anteaya | and I'm just confused these days | 16:08 |
| anteaya | more like it is me | 16:08 |
| anteaya | anyway, hope you get things sorted out for yourself | 16:08 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add debug-sync option https://review.openstack.org/277473 | 16:08 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add support for last_seen https://review.openstack.org/277474 | 16:08 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add support for customized dashboard sorting https://review.openstack.org/277475 | 16:08 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Make sorting commands use two two keys https://review.openstack.org/277476 | 16:08 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Fix return values in handleCommands in project_list https://review.openstack.org/277477 | 16:08 |
| pabelanger | AJaeger: looking at the debian jessie patch, do we actually use the Restricted repo from Ubuntu? Would be interested what happends if that was removed | 16:09 |
| jeblair | fungi, mordred: ^ the tip of this stack has support for a new dashboard that shows you the changes you have most-recently seen | 16:09 |
| * Shrews wonders what jeblair means by "two two keys" | 16:09 | |
| *** ajmiller has quit IRC | 16:09 | |
| jeblair | Shrews: ha! | 16:09 |
| AJaeger | pabelanger: mordred proposed a patch to remove it from Ubuntu - see also openstack-dev mailing list. Tests look very good! | 16:09 |
| pabelanger | AJaeger: Oh, nice | 16:09 |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds swagger migrate and copy job for api-site repo https://review.openstack.org/269809 | 16:09 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch non-infra Ansible roles to using 'linters' tox target https://review.openstack.org/277479 | 16:10 |
| AJaeger | pabelanger: https://review.openstack.org/277174 | 16:10 |
| odyssey4me | pabelanger note: https://review.openstack.org/277479 | 16:10 |
| *** trown has quit IRC | 16:10 | |
| jeblair | fungi, mordred, Shrews: so when you saw earlier but need to come back to, it's easy to find it. | 16:10 |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds swagger migrate and copy job for api-site repo https://review.openstack.org/269809 | 16:10 |
| jeblair | blah | 16:10 |
| jeblair | fungi, mordred, Shrews: so when you have a patch you saw earlier but need to come back to, it's easy to find it. | 16:11 |
| fungi | that sounds very useful | 16:11 |
| Shrews | ++ | 16:11 |
| fungi | will try to find time to test it out | 16:11 |
| jeblair | fungi, mordred, Shrews: it has a schema change, so making a db backup may be a good idea | 16:12 |
| *** trown|brb has quit IRC | 16:12 | |
| *** trown has joined #openstack-infra | 16:13 | |
| jeblair | fungi, mordred, Shrews: that series also includes conflicts-with support (which is also a schema change) | 16:14 |
| Shrews | ooh! now that i _really_ like | 16:14 |
| *** isaacb has quit IRC | 16:15 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Add column for task links https://review.openstack.org/277481 | 16:15 |
| fungi | jeblair: krotscheck: for http://paste.openstack.org/show/486293/ i need to do that on the r/w volume obviously, should i be trying to set my workstation up as an openafs (i haven't gotten around to that yet) client and do the deletes with my admin credentials, or just punt and do it on one of our systems that are already set up? | 16:15 |
| jeblair | fungi: shouldn't be much of a difference (the operations are lightweight, so being in the same DC won't be too much of an advantage). so whichever you want. :) | 16:16 |
| *** Guest56910 has quit IRC | 16:16 | |
| jeblair | fungi: but you can use your admin creds either way | 16:16 |
| openstackgerrit | Merged openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 2] https://review.openstack.org/276859 | 16:16 |
| fungi | for the sake of personal interest, i'll give the client setup a shot. are the steps at http://docs.openstack.org/infra/system-config/afs.html#client-configuration still accurate? | 16:17 |
| *** DuaneDeC7 has quit IRC | 16:18 | |
| mordred | fungi: yes | 16:19 |
| *** DuaneDeC7 has joined #openstack-infra | 16:19 | |
| Shrews | jeblair: if running from source, do i need to 'pip install -e .' again to get the schema change? | 16:19 |
| *** armax has joined #openstack-infra | 16:20 | |
| jeblair | Shrews: if you last installed with 'pip install -e .' you don't need to do it again; if you did something else previously, then yes. | 16:20 |
| *** aspiers has joined #openstack-infra | 16:21 | |
| *** [HeOS] has joined #openstack-infra | 16:21 | |
| *** HeOS has quit IRC | 16:22 | |
| mordred | jeblair: oh - did you see my earlier comment about having the lockfile problem again? | 16:22 |
| mordred | jeblair: it's in teh scrollback in teh screen | 16:23 |
| *** vsaienko has joined #openstack-infra | 16:24 | |
| *** sbalukoff has quit IRC | 16:24 | |
| vsaienko | Hello infra team, could you please have look https://review.openstack.org/#/c/276200/ | 16:25 |
| *** hashar has quit IRC | 16:25 | |
| *** tdurakov has quit IRC | 16:25 | |
| devananda | ^ is a small g-r update to add netmiko, which is a pre-requirement for testing neutron/ironic integration in in our gate | 16:26 |
| *** tdurakov has joined #openstack-infra | 16:26 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch non-infra Ansible roles to using 'linters' tox target https://review.openstack.org/277479 | 16:28 |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Add column for task links https://review.openstack.org/277481 | 16:28 |
| *** Swami has joined #openstack-infra | 16:29 | |
| *** tcammann has joined #openstack-infra | 16:30 | |
| *** esp has joined #openstack-infra | 16:30 | |
| *** vgridnev has quit IRC | 16:31 | |
| jeblair | mordred: oh, hrm, let me dig that up... | 16:31 |
| *** vgridnev has joined #openstack-infra | 16:31 | |
| *** tcammann_ has joined #openstack-infra | 16:31 | |
| jeblair | mordred: it looks like you ran it without tokens | 16:32 |
| *** NobodyCa1 has joined #openstack-infra | 16:32 | |
| jeblair | mordred: sometimes when i'm doing stuff like this, i 'k5start -- bash' | 16:32 |
| mordred | jeblair: I just ran the script though | 16:32 |
| mordred | jeblair: expecting the scripts k5starts to do the right thing | 16:33 |
| jeblair | mordred: oh, reprepro-mirror-update is the script, sorry | 16:33 |
| mordred | *phew* | 16:33 |
| mordred | I see it | 16:33 |
| mordred | jeblair: thanks | 16:33 |
| *** deva_ has joined #openstack-infra | 16:33 | |
| mordred | jeblair: dumpunreferenced | 16:34 |
| mordred | isn't k5started | 16:34 |
| jeblair | ah and that takes a lockfile? | 16:34 |
| mordred | yah | 16:34 |
| mordred | because it does things with the db | 16:34 |
| jeblair | mordred: what about the next two? | 16:34 |
| mordred | probably yes | 16:34 |
| mordred | patch coming | 16:34 |
| *** adrienverge has quit IRC | 16:35 | |
| *** adrienve1 has joined #openstack-infra | 16:35 | |
| *** tcammann has quit IRC | 16:36 | |
| *** tdurakov has quit IRC | 16:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use k5start even for the readonly reprepro actions https://review.openstack.org/277487 | 16:36 |
| mordred | jeblair: ^^ | 16:36 |
| *** tcammann_ has quit IRC | 16:36 | |
| *** tdurakov has joined #openstack-infra | 16:37 | |
| *** NobodyCa1 has quit IRC | 16:37 | |
| *** esp has quit IRC | 16:37 | |
| *** deva_ has quit IRC | 16:38 | |
| *** savihou has quit IRC | 16:38 | |
| fungi | mordred: any reason not to just fold k5start -t -f /etc/reprepro.keytab service/reprepro -- into $REPREPRO? | 16:38 |
| fungi | mordred: also why is it --$REPREPRO now (should it be -- $REPREPRO there instead)? | 16:39 |
| mordred | fungi: good call on both | 16:39 |
| *** abregman has joined #openstack-infra | 16:39 | |
| EmilienM | outstanding request, can we land https://review.openstack.org/#/c/277457/ ? it's blocking puppet CI, thanks | 16:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use k5start even for the readonly reprepro actions https://review.openstack.org/277487 | 16:40 |
| *** salv-orl_ has joined #openstack-infra | 16:40 | |
| mordred | fungi: how's that? | 16:40 |
| *** kushal has quit IRC | 16:42 | |
| *** tdurakov has quit IRC | 16:42 | |
| *** salv-orlando has quit IRC | 16:43 | |
| *** markmcd has quit IRC | 16:43 | |
| *** markmcd1 has quit IRC | 16:43 | |
| *** egonzalez has quit IRC | 16:44 | |
| *** jecarey has quit IRC | 16:45 | |
| *** jgriffith_away is now known as jgriffith | 16:45 | |
| daemontool | fungi, jeblair here http://logs.openstack.org/71/277371/7/check/gate-freezer-python27/fabc3d5/console.html | 16:45 |
| daemontool | we see the error> | 16:45 |
| daemontool | 2016-02-08 14:50:04.603 | + find .tox -type f -name '*.log' -exec mv '{}' '{}.txt' | 16:45 |
| daemontool | 2016-02-08 14:50:04.603 | find: missing argument to `-exec' | 16:45 |
| daemontool | 2016-02-08 14:50:04.603 | + exit 1 | 16:45 |
| *** markmcd has joined #openstack-infra | 16:45 | |
| daemontool | I think \; is missing there? | 16:45 |
| *** esikachev has quit IRC | 16:46 | |
| *** armax has quit IRC | 16:46 | |
| fungi | daemontool: yes, it looks like it | 16:46 |
| daemontool | is that something that belongs to infra? | 16:46 |
| daemontool | or us? | 16:46 |
| *** tdurakov has joined #openstack-infra | 16:46 | |
| fungi | daemontool: yep, i even think it's a bug introduced in a recentish patch of mine | 16:46 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Clarify Munch object usage in documentation https://review.openstack.org/276878 | 16:46 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add docs tox target https://review.openstack.org/276870 | 16:46 |
| daemontool | fungi, ok ty let me know if there's anything we can do | 16:46 |
| fungi | daemontool: is the job failing, or is that just noise in the log? | 16:46 |
| daemontool | the job is failing | 16:46 |
| fungi | oh, then we need to delete images | 16:47 |
| Shrews | mordred: w00t. job fixed: https://review.openstack.org/277462 | 16:47 |
| *** infra-red has joined #openstack-infra | 16:47 | |
| mordred | jeblair, fungi: the version of the script in that patch works. it's running a vos release now, which I'm hoping will not be terribly long since we've done a few here already and there shouldn't be THAT many changes | 16:48 |
| fungi | daemontool: um, that job for the log you linked failed a unit test | 16:48 |
| daemontool | fungi, ah ok | 16:48 |
| *** cloudtrainme has joined #openstack-infra | 16:48 | |
| *** gyee has joined #openstack-infra | 16:48 | |
| fungi | daemontool: http://logs.openstack.org/71/277371/7/check/gate-freezer-python27/fabc3d5/console.html#_2016-02-08_14_50_03_018 | 16:48 |
| *** vivekd has joined #openstack-infra | 16:48 | |
| fungi | are you sure that's not why the job is failing? | 16:48 |
| daemontool | yes sorry FAIL: tests.test_config.TestConfig.test_ini | 16:48 |
| daemontool | nope I'm not sure, checking it | 16:49 |
| daemontool | sorry | 16:49 |
| *** kzaitsev_mb has quit IRC | 16:49 | |
| fungi | daemontool: basically the find error you quoted would cause any job run via our run-tox.sh script to fail | 16:49 |
| fungi | if it's not causing all jobs to fail, then we can treat the fix for the script as slightly less urgent | 16:49 |
| fungi | just embarrassing (on my part) | 16:49 |
| *** infra-re_ has joined #openstack-infra | 16:49 | |
| clarkb | fungi what is the purpose of moving tox.ini? | 16:50 |
| *** hongbin has joined #openstack-infra | 16:50 | |
| fungi | clarkb: where is it moving tox.ini? | 16:50 |
| clarkb | oh I misread it is tox logs | 16:50 |
| clarkb | fungi my early morning brain misparsed the find command that is failing | 16:50 |
| *** fhubik is now known as fhubik_brb | 16:51 | |
| *** hongbin has left #openstack-infra | 16:52 | |
| *** infra-red has quit IRC | 16:52 | |
| *** DuaneDeC7 has quit IRC | 16:52 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Fix Bugs in Timeline Events https://review.openstack.org/277386 | 16:53 |
| *** shardy has joined #openstack-infra | 16:53 | |
| *** matrohon has quit IRC | 16:53 | |
| *** nadya has joined #openstack-infra | 16:54 | |
| *** nadya is now known as Guest55447 | 16:54 | |
| *** ashleighfarnham has joined #openstack-infra | 16:54 | |
| *** mikelk has quit IRC | 16:54 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add missing semicolon in run-tox.sh https://review.openstack.org/277492 | 16:55 |
| anteaya | clarkb: morning | 16:55 |
| fungi | daemontool: clarkb: AJaeger: sdague: ^ | 16:55 |
| mordred | jeblair: ooh | 16:55 |
| mordred | jeblair: new fun error for you | 16:55 |
| mordred | jeblair: I ran the script once, it succeeded | 16:55 |
| mordred | jeblair: I ran it again to double check | 16:55 |
| greghaynes | https://review.openstack.org/#/c/245297/ might be of interest to you - we can now install pip using get-pip without ever installing it as a package first | 16:56 |
| mordred | jeblair: and got | 16:56 |
| mordred | jeblair: /afs/.openstack.org/mirror/apt/Kerberos initialization for service/reprepro@OPENSTACK.ORG not found, forgetting anyway | 16:56 |
| mordred | Unable to forget unknown filekey 'Kerberos initialization for service/reprepro@OPENSTACK.ORG'. | 16:56 |
| *** harlowja_at_home has quit IRC | 16:56 | |
| *** jpr has quit IRC | 16:56 | |
| *** Daisy has joined #openstack-infra | 16:56 | |
| fungi | daemontool: fwiw, that script doesn't run under errexit so that find syntax error is basically benign (it's no worse than things were before the patch which introduced it) | 16:57 |
| *** xyang1 has quit IRC | 16:57 | |
| daemontool | fungi, ok thanks for the follow up | 16:57 |
| electrofelix | zxiiro: few days behind email, so prehaps haven't seen it yet | 16:57 |
| *** DuaneDeC7 has joined #openstack-infra | 16:58 | |
| *** tzn has quit IRC | 16:59 | |
| *** annegentle has quit IRC | 16:59 | |
| *** fhubik_brb is now known as fhubik | 16:59 | |
| *** annegentle has joined #openstack-infra | 17:00 | |
| *** reed_ has joined #openstack-infra | 17:00 | |
| *** jgriffith is now known as jgriffith_away | 17:00 | |
| *** jsavak has quit IRC | 17:01 | |
| mordred | fungi: could I get you to +2 https://review.openstack.org/#/c/277189/ again? | 17:01 |
| *** Daisy has quit IRC | 17:01 | |
| *** DuaneDeC7 has quit IRC | 17:02 | |
| notmorgan | sdague, dtroyer: https://review.openstack.org/#/c/275327/ for the health of ELK, dropping microseconds is important. see the comment on the review why we can't just use asctime from oslo_log [or tell me we don't care about apache 2.2 :) ] | 17:03 |
| notmorgan | clarkb: ^ cc | 17:03 |
| krotscheck | fungi, jeblair, mordred: Are we ready to get rid of the old pypi mirrors? https://review.openstack.org/#/c/273518/ | 17:03 |
| *** markmcd has quit IRC | 17:03 | |
| *** kushal has joined #openstack-infra | 17:04 | |
| *** jgriffith_away is now known as jgriffith | 17:04 | |
| *** apoorvad has joined #openstack-infra | 17:04 | |
| sdague | notmorgan: will that give us msec still? | 17:04 |
| notmorgan | sdague: nope | 17:04 |
| *** annegentle has quit IRC | 17:04 | |
| notmorgan | sdague: we get microseconds or seconds | 17:04 |
| *** jsavak has joined #openstack-infra | 17:05 | |
| sdague | we really need msec | 17:05 |
| *** salv-orl_ has quit IRC | 17:05 | |
| notmorgan | sdague: because apache 2.2 has no ErrorLogFormat, so it always adds it's own timestamp | 17:05 |
| notmorgan | this means CentOS would be <timestamp> <timestamp> <log> | 17:05 |
| *** salv-orlando has joined #openstack-infra | 17:05 | |
| sdague | oh, this just exposes on centos? | 17:05 |
| notmorgan | rather than <timestamp> <log> | 17:05 |
| sdague | so, if this is just an ELK fix, we can fix it in os-loganalyze instead | 17:06 |
| *** hdd has quit IRC | 17:06 | |
| *** markmcd1 has joined #openstack-infra | 17:06 | |
| *** markmcd has joined #openstack-infra | 17:06 | |
| *** hdd has joined #openstack-infra | 17:06 | |
| clarkb | we can fix it in logstash too | 17:06 |
| clarkb | but microseconds provide zero valur | 17:06 |
| notmorgan | so apache can do "seconds" or "microseconds" | 17:06 |
| sdague | clarkb: sure, but miliseconds are kind of important in debugging | 17:06 |
| clarkb | and will be negative value if we have to fix them | 17:06 |
| notmorgan | in ubuntu we can omit apache's timestamp | 17:07 |
| clarkb | the alternative is to have keystone log timestamps right? | 17:07 |
| notmorgan | on centos we can't | 17:07 |
| *** baoli has quit IRC | 17:07 | |
| clarkb | oh | 17:07 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Rename board and worklist controllers https://review.openstack.org/272211 | 17:07 |
| notmorgan | centos uses apache 2.2 so we have no control on errorlog format | 17:07 |
| sdague | ok, lets narrow the scope of things | 17:07 |
| *** reed_ has quit IRC | 17:07 | |
| sdague | what is the bug, and which platforms does it expose on? | 17:07 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Set PS4 to provide additional debug info https://review.openstack.org/271434 | 17:08 |
| notmorgan | sdague: no bug. just microseconds is a lot of uniques to index | 17:08 |
| notmorgan | sdague: centos uses apache 2.2, always includes the aapche timestamp in the errorlog, where keystone's log is output | 17:08 |
| sdague | centos 7? | 17:08 |
| notmorgan | yep | 17:08 |
| *** ashleighfarnham has quit IRC | 17:08 | |
| fungi | i'm surprised centos 7 didn't go with apache 2.4 | 17:09 |
| notmorgan | i couldn't find a standard apache 2.4 rpm | 17:09 |
| notmorgan | it looks like EPEL might have it | 17:09 |
| BobBall | Anyone seen zuul not starting but not logging anything as to why it's not started either? | 17:09 |
| jlvillal | sdague: ^^^ A PS4 patch to devstack-gate too. I used your updated PS4 | 17:09 |
| gtmanfred | centos7 has 2.4.9 | 17:09 |
| notmorgan | gtmanfred: really? i was always getting 2.2 | 17:09 |
| BobBall | service zuul start just completes and a zuul-server process exists for ~10 seconds then it's gone | 17:09 |
| notmorgan | in centos7 | 17:09 |
| *** jsavak has quit IRC | 17:09 | |
| gtmanfred | one second | 17:09 |
| *** cgalan_ has joined #openstack-infra | 17:09 | |
| notmorgan | if that is the case, easy we go back to oslo_log | 17:09 |
| fungi | apache 2.4 well predated centos/rhel 7 i'm pretty sure | 17:10 |
| gtmanfred | yeah, it should be 2.4 something | 17:10 |
| gtmanfred | http://mirror.rackspace.com/CentOS/7/os/x86_64/Packages/httpd-2.4.6-40.el7.centos.x86_64.rpm | 17:10 |
| gtmanfred | 2.4.6 | 17:10 |
| anteaya | BobBall: I haven't had that as a report, no | 17:10 |
| anteaya | BobBall: what version of zuul? | 17:10 |
| *** jsavak has joined #openstack-infra | 17:10 | |
| gtmanfred | notmorgan: sorry, was 2.4.6 | 17:10 |
| notmorgan | gtmanfred: ok then something was horked on my test | 17:10 |
| gtmanfred | :) | 17:10 |
| notmorgan | because i kept getting apache 2.2 | 17:10 |
| gtmanfred | yeah, i would have had a fit if it was still 2.2 | 17:10 |
| sdague | notmorgan: http://logs.openstack.org/06/276506/1/check/gate-tempest-dsvm-centos7/c0f9354/logs/devstacklog.txt.gz#_2016-02-05_22_42_26_917 | 17:10 |
| gtmanfred | well maybe not, cause i have moved all my stuff to nginx | 17:10 |
| sdague | we're definitely installing 2.4 in the gate | 17:10 |
| notmorgan | sdague: ok i'll rework this to be asctime again in keystone instead | 17:11 |
| AJaeger | fungi, oops - thanks! | 17:11 |
| *** infra-red has joined #openstack-infra | 17:11 | |
| notmorgan | and just assume devstack doesn't handle apache 2.2 | 17:11 |
| BobBall | anteaya: Not the most up to date, but we haven't updated recently and zuul has been working just fine until 3rd February after which it hasn't restarted and we can't restart it manually. | 17:11 |
| fungi | AJaeger: the original typo was entirely mine | 17:11 |
| sdague | notmorgan: I am totally ok with that assumption | 17:11 |
| notmorgan | can i drop the 2.2 ifblock as well around ErrorLogFormat? | 17:11 |
| notmorgan | sdague: ok will clean a couple things up | 17:11 |
| notmorgan | sdague: expect an update a little later today | 17:11 |
| anteaya | BobBall: :( | 17:11 |
| notmorgan | :) | 17:11 |
| sdague | notmorgan: great | 17:11 |
| *** esker has quit IRC | 17:11 | |
| anteaya | BobBall: yeah, you are the first I have seen to report this experience | 17:12 |
| fungi | notmorgan: i can't think of a platform we still strive to support which has apache 2.4 any longer | 17:12 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: rename tag for making stable branches https://review.openstack.org/277499 | 17:12 |
| notmorgan | fungi: wfm | 17:12 |
| fungi | so should be entirely safe to clean that up | 17:12 |
| *** scheuran has quit IRC | 17:12 | |
| *** jlanoux has quit IRC | 17:12 | |
| anteaya | BobBall: I have to eat soup but if you have more, I'll read it upon my return | 17:12 |
| *** adrienve1 has quit IRC | 17:13 | |
| sdague | I think the only question is if there is anything running on stable that's borking logstash | 17:13 |
| gtmanfred | notmorgan: if you want a newer version of 2.4, IUS packages the newest version for cent6 and 7 https://github.com/iuscommunity-pkg/httpd24u | 17:13 |
| gtmanfred | but it sounds like you are just using stock packages | 17:13 |
| *** dizquierdo has quit IRC | 17:13 | |
| *** fhubik has quit IRC | 17:13 | |
| sdague | but, again, we're all trusty + centos7, so I think that seems fine to backport even | 17:13 |
| *** samuelBartel has quit IRC | 17:13 | |
| sdague | gtmanfred: stock or what's in epel | 17:13 |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/elastic-recheck: Add query for "Security group is still in use" bug 1543106 https://review.openstack.org/277501 | 17:13 |
| openstack | bug 1543106 in tempest "TestRebuildInstanceWithVolume.test_rebuild_instance_with_volume failes with "Security group is still in use"" [Undecided,New] https://launchpad.net/bugs/1543106 | 17:13 |
| gtmanfred | kk | 17:13 |
| *** dims has quit IRC | 17:14 | |
| sdague | but, we seem to be getting 2.4.7 from what I can see, which is fine | 17:14 |
| *** infra-re_ has quit IRC | 17:14 | |
| gtmanfred | yar | 17:14 |
| notmorgan | gtmanfred: nope any 2.4 works | 17:14 |
| gtmanfred | :) | 17:14 |
| clarkb | greghaynes: for your comment on https://review.openstack.org/#/c/273703/9 do you mean it isn't related to this change? | 17:14 |
| *** asselin has joined #openstack-infra | 17:14 | |
| *** asselin__ has joined #openstack-infra | 17:15 | |
| * clarkb rechecks to see if it is super consistent | 17:15 | |
| greghaynes | clarkb: I think there is a fail which caused a timeout, the fail in that comment I think is a second fail which is the known pymysql fixture issue that happens during cleanup | 17:15 |
| *** mingyu has quit IRC | 17:15 | |
| jlvillal | I have a minor patch to devstack-gate: https://review.openstack.org/#/c/271463/ Has been 11 days without a review. So I thought I would ask for one :) | 17:15 |
| *** ihrachys has quit IRC | 17:16 | |
| greghaynes | clarkb: I have no idea why it timed out though | 17:16 |
| *** tphummel has joined #openstack-infra | 17:16 | |
| *** mingyu has joined #openstack-infra | 17:16 | |
| *** mingyu has quit IRC | 17:16 | |
| *** mingyu has joined #openstack-infra | 17:16 | |
| *** xyang1 has joined #openstack-infra | 17:16 | |
| clarkb | greghaynes: gotcha | 17:18 |
| clarkb | also no gearman timeouts that I can see in the server log, but we don't have images building this morning | 17:18 |
| openstackgerrit | Merged openstack-infra/system-config: Import gpg keys for mirroring into local keyring https://review.openstack.org/277189 | 17:18 |
| greghaynes | clarkb: :( | 17:18 |
| *** ccrouch has quit IRC | 17:18 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Add column for task links https://review.openstack.org/277481 | 17:19 |
| jeblair | krotscheck: yes, i'm ready; fungi, clarkb: you okay dropping old pypi mirrors now? | 17:20 |
| clarkb | greghaynes: doesn't look like the cron fired, so I think that implies the cron is still running | 17:21 |
| fungi | krotscheck: yeah, seems like it's time. i've seen/heard no failures related to the new mirrors | 17:21 |
| *** cgalan_ has quit IRC | 17:21 | |
| clarkb | I can do a thread dump and check | 17:21 |
| clarkb | jeblair: sure | 17:21 |
| jeblair | mordred: i got lost in scrollback, still seeing that error? | 17:21 |
| *** yamamoto has joined #openstack-infra | 17:21 | |
| mordred | fungi, jeblair: after comments from fungi and follow ups from cody-somerville and SpamapS - I believe I have made a mistake in the current organization of the AFS volume for APT | 17:22 |
| clarkb | have we confirmed new images across the board using the new mirror? | 17:22 |
| zaro | morning | 17:22 |
| odyssey4me | krotscheck I'm told by dstufft that you've been looking into the normalising of names in the pip wheels? do you have a review up for a fix perhaps? | 17:22 |
| mordred | which means we need to rename and remount the current volume | 17:22 |
| *** yamamoto has quit IRC | 17:22 | |
| jeblair | odyssey4me: https://review.openstack.org/277398 | 17:22 |
| mordred | tl;dr - fungi is right, they cannot share a pool - so we want an apt.ubuntu and an apt.debian volume - each with their own reprepro setup | 17:22 |
| greghaynes | clarkb: huh, I wonder if it is hung up in waitForBuiltImages | 17:23 |
| mordred | should be easy enough to change- but we should not roll the current thing live until we do | 17:23 |
| jeblair | clarkb: i think krotscheck did so a long time ago | 17:23 |
| mordred | jeblair: it's in the screen session | 17:23 |
| krotscheck | odyssey4me: Yes, it's merged. | 17:23 |
| mordred | jeblair: and I did not move forward on it because I do not understand the problem | 17:23 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Enable voting for Puppet OpenStack Integration jobs" https://review.openstack.org/277457 | 17:23 |
| greghaynes | clarkb: Do you know about how long ago the last build was? | 17:23 |
| fungi | that is to say not roll the ubuntu mirror live and then try to rename things later when it comes time to add a debian mirror if we can avoid an outage by introducing some namespacing up front | 17:24 |
| odyssey4me | jeblair ah thanks, and thanks krotscheck for digging into it - any idea when a fresh wheel build will be done with that implemented? | 17:24 |
| krotscheck | odyssey4me: There'll be a bit of a delay as it shakes out and we have to deal with some of the cleanup. | 17:24 |
| openstackgerrit | Gábor Antal proposed openstack-infra/gearman-plugin: Removed unnecessary null check in StartJobWorker https://review.openstack.org/277502 | 17:24 |
| jeblair | mordred: re: move -- sounds good to me -- i guess we'll just stop manually running it, merge path changes, move volume, then start up again? | 17:24 |
| jeblair | mordred: re: error -- i'll dig | 17:24 |
| mordred | jeblair: yes | 17:24 |
| fungi | krotscheck: i'm working on the cleanup, though the sid machine i wanted to try it from was behind more than 1k package updates so i'm letting that finish first | 17:24 |
| krotscheck | odyssey4me: Either the next requrements merge, or the next periodic run, assuming that the script has already been synced to the wheel mirror. | 17:25 |
| krotscheck | odyssey4me: ^^ note fungi's comments on removing packages. | 17:25 |
| krotscheck | fungi: THanks :) | 17:25 |
| *** tdurakov has quit IRC | 17:25 | |
| clarkb | greghaynes: http://paste.openstack.org/show/486313/ | 17:25 |
| krotscheck | odyssey4me: Out of curiosity, what's your interest? | 17:25 |
| jeblair | mordred: ha, i understand :) | 17:26 |
| clarkb | greghaynes: 2016-02-07 15:06:57,122 ERROR nodepool.builder.BuildWorker: Exception while building image was last item logged by builder | 17:26 |
| jeblair | mordred: k5start output is being redirected to the cleanup list | 17:26 |
| mordred | hahahahaha | 17:26 |
| clarkb | greghaynes: I wonder if wait for built images expects that job to be restarted | 17:26 |
| odyssey4me | ah, thanks fungi & krotscheck ... I'm working on a patch to make use of the wheels in the gate for OpenStack-Ansible: https://review.openstack.org/265867 | 17:26 |
| clarkb | greghaynes: but since it isn't sits there waiting forever for an image build that failed? | 17:26 |
| greghaynes | clarkb: yep, so I need to fully page in how all it can get stuck there, but basically that waits on all the build jobs completing so if a disconnect happens things can be bad | 17:26 |
| krotscheck | odyssey4me: the script was moved over at 7:52AM PST | 17:26 |
| jeblair | mordred: (so that's an error from reprepro saying "i can't delete the package <k5start gobbledygook>" | 17:26 |
| mordred | yup | 17:26 |
| mordred | makes total sense | 17:26 |
| clarkb | greghaynes: no disconnect happened from what I can see, but there was a failed build | 17:27 |
| *** tdurakov has joined #openstack-infra | 17:27 | |
| odyssey4me | krotscheck until now we've been using pypi and a Rackspace wheel repo - I'm trying to change things up in the gate | 17:27 |
| greghaynes | clarkb: hrm, I thought I have failed handled fine there, let me check | 17:27 |
| openstackgerrit | Merged openstack-infra/shade: Fix for stable/liberty job https://review.openstack.org/277462 | 17:27 |
| odyssey4me | krotscheck in the hope of reducing build time | 17:27 |
| krotscheck | odyssey4me: Ur.... are you running all that on our trusty jenkins slaves? Because if so, the mirror update has been added to the global pip.conf | 17:28 |
| krotscheck | In other words, you should already have it. | 17:28 |
| clarkb | Failed to connect to ca.downloads.xensource.com port 80: Connection refused is what caused the build to fail | 17:28 |
| fungi | odyssey4me: worth noting, the current wheel mirror is only building the transitive dependency set defined by our openstack global requirements list | 17:28 |
| krotscheck | That too | 17:28 |
| *** jsavak has quit IRC | 17:28 | |
| odyssey4me | krotscheck fungi Ah ok. and how complete is the pypi mirror? | 17:29 |
| clarkb | greghaynes: I will look into the build failure if you can look into the state machine around wait for built images | 17:29 |
| mordred | odyssey4me: the pypi mirror is complete | 17:29 |
| pleia2 | so much email piled up, almost forgot to say hello, good morning! | 17:29 |
| nibalizer | you too pleia2 | 17:29 |
| nibalizer | :) | 17:29 |
| mordred | odyssey4me: you should always use the pypi mirror that's configured for your nodes - it's local and complete and consistent and whatnot | 17:29 |
| greghaynes | clarkb: yea, fwiw https://review.openstack.org/#/c/272384/9 improves that state machine a lot | 17:29 |
| greghaynes | so my brain is conflating that a bit | 17:30 |
| *** mingyu has quit IRC | 17:30 | |
| clarkb | greghaynes: ya I think we should just go ahead and continue to get that in today | 17:30 |
| mordred | odyssey4me: you should also in that config also get the wheel mirror, which will have wheels prebuilt for all of openstack/reuqirements | 17:30 |
| clarkb | greghaynes: since it should help debugging and straggler issues | 17:30 |
| *** davideagnello has joined #openstack-infra | 17:30 | |
| odyssey4me | krotscheck it came down to wondering why it was that http://mirror.ord.rax.openstack.org/wheel/ubuntu-14.04-x86_64/MySQL_python/MySQL_python-1.2.5-cp27-none-linux_x86_64.whl was not selected above http://mirror.ord.rax.openstack.org/pypi/packages/source/M/MySQL-python/MySQL-python-1.2.5.zip on a pip wheel repo build which the wheel repo being in the extra index | 17:30 |
| *** mingyu has joined #openstack-infra | 17:30 | |
| krotscheck | odyssey4me: because the directory name wasn't normalized. | 17:31 |
| clarkb | krotscheck: we have to rewrite -'s to _'s right? | 17:31 |
| greghaynes | clarkb: So it looks like workFailed is handled but workException is not | 17:31 |
| clarkb | odyssey4me: also fwiw we are using PyMySQL pretty effectively now | 17:31 |
| fungi | yeah, and lowercase all uppercase characters | 17:31 |
| greghaynes | clarkb: I want to say that we should have sent a workFailed in that scenario, though | 17:31 |
| krotscheck | clarkb: [-_.]+/- | 17:31 |
| krotscheck | Sorry | 17:31 |
| krotscheck | clarkb: / [-_.]+/-/ | 17:31 |
| *** dims has joined #openstack-infra | 17:32 | |
| fungi | normalization includes lower-case transformation, right? | 17:32 |
| BobBall | clarkb: Sorry - which build is this? | 17:32 |
| krotscheck | yep | 17:32 |
| *** esikachev has joined #openstack-infra | 17:32 | |
| krotscheck | It's all in there. | 17:32 |
| clarkb | BobBall: our image builds | 17:32 |
| greghaynes | clarkb: yep, we should have sent workFailed | 17:32 |
| greghaynes | clarkb: so its possible that is a red herring | 17:32 |
| jeblair | clarkb, fungi: this is the normalization patch: https://review.openstack.org/277398 | 17:32 |
| BobBall | *confused* What are you pulling from ca.downloads.xensource.com ? | 17:32 |
| odyssey4me | mordred I'd love to use the image configured /etc/pip.conf, but we're building many containers and dropping configs there... so my initial adventure is to read the configs and to use them with our current build paradigm. If that doesn't work then I'll change out pip config paradigm to be able to slurp in the whole pip.conf from the host | 17:33 |
| greghaynes | clarkb: another thing we should do regardless IMO is put a timeout on these jobs completing | 17:33 |
| clarkb | BobBall: the cirros image for your jobs I think | 17:33 |
| fungi | jeblair: thanks, looks complete to me | 17:33 |
| clarkb | BobBall: they are listed in devstack so get cached | 17:33 |
| odyssey4me | clarkb yep, as are we for openstack services - but fun fact, Ansible requires MySQL-python, as does the MariaDB client | 17:33 |
| jeblair | odyssey4me: we really need you to use the mirrors we have configured | 17:33 |
| clarkb | greghaynes: 2016-02-07 15:06:57,124 DEBUG nodepool.NodePool: DIB Image image-build (id 176259) failed to build. Deleting. | 17:33 |
| BobBall | clarkb: Ah - right. Makes sense. I assumed they were only pulled down if the driver was xenapi | 17:33 |
| clarkb | BobBall: we don't do per driver image builds | 17:33 |
| jeblair | odyssey4me: we can't really manage a system of this complexity with individual projects doing their own things | 17:34 |
| clarkb | greghaynes: so the nodepool daemon did reqcord it as failing to build | 17:34 |
| odyssey4me | clarkb to clarify, Ansible requires that lib only when you use the mysql module | 17:34 |
| clarkb | greghaynes: I think the wait for built image may just not realize it is done waiting when that happens | 17:34 |
| jeblair | odyssey4me: so if you need to run pip in a container, please use the pip.conf on the host | 17:34 |
| BobBall | clarkb: but none of infra's images actually use xenapi so it doesn't need to be cached at all :) | 17:34 |
| BobBall | clarkb: that's where I got confused | 17:34 |
| clarkb | BobBall: then please remove it from devstack | 17:34 |
| clarkb | BobBall: we cache what devstack tells us to cache | 17:34 |
| BobBall | clarkb: the XenServer third party CI tests it within devstack | 17:34 |
| mordred | yah. because we can't know at cache build time what runtime jobs are going to want to select | 17:35 |
| odyssey4me | jeblair Yep, I'm working on converging there as quickly as possible. | 17:35 |
| jeblair | odyssey4me: cool, thanks | 17:35 |
| greghaynes | clarkb: hrm, that is handled by almost the same code path - when the WatchableJob gets onWorkFailed it sets it's event which cleans it out of the queue being waited on | 17:35 |
| BobBall | so it is used, just not by -infra jobs | 17:35 |
| *** vgridnev has joined #openstack-infra | 17:35 | |
| greghaynes | clarkb: We can add some debug logging too | 17:35 |
| fungi | BobBall: clarkb: devstack plugins can define additional files to retrieve right? and we'll properly skip those since they're not in the vanilla devstack file list output | 17:35 |
| mordred | BobBall: is there a devstack plugin for xenserver? | 17:35 |
| mordred | yah. what fungi was getting at | 17:35 |
| BobBall | mordred: No; it's part of stack.sh (predated plugins). I presume that converting to a plugin would help avoid you caching it | 17:36 |
| mordred | BobBall: yah. honestly even if there as just a plugin that the only point of it was downloading this image | 17:37 |
| mordred | BobBall: (I know full devstack plugin migation would be likely non-trivial) | 17:37 |
| *** ccrouch has joined #openstack-infra | 17:37 | |
| clarkb | greghaynes: I don't see where _image_build_jobs has the job remvoed on failure | 17:37 |
| clarkb | greghaynes: we remove it from the db | 17:37 |
| greghaynes | clarkb: JobTracker | 17:38 |
| *** claudiub has quit IRC | 17:38 | |
| greghaynes | clarkb: that class auto sets up the cleanup whenever a job is added to it | 17:38 |
| clarkb | ah | 17:38 |
| BobBall | mordred: I have a feeling that the vmware disk is also cached and that's a lot bigger - now that we're talking about it I think I've seen that before | 17:38 |
| jeblair | fungi, AJaeger: looks like jenkins-jobs is exceeding its timeout. i will merge all of the devstack timeout related changes then run in screen. | 17:39 |
| clarkb | greghaynes: I think I see it. failure doens't imply completion in watchable job | 17:39 |
| mordred | BobBall: yes, I believe you are correct | 17:39 |
| fungi | jeblair: thanks, and i'm not at all surprised | 17:39 |
| clarkb | greghaynes: so that'll do it | 17:39 |
| krotscheck | jeblair, clarkb: Alright, i'll need a +A on https://review.openstack.org/#/c/273518/ and someone to go shut down the old mirrors. | 17:39 |
| fungi | jeblair: did i review that one? | 17:39 |
| greghaynes | clarkb: it does? it calls self._event.set | 17:39 |
| * krotscheck can try to shut them down, but without access i'd have to go the blackhat route. | 17:40 | |
| jeblair | fungi: i think it was yolanda and i; topic:timeout | 17:40 |
| clarkb | greghaynes: thta just means the job is done running on the wait | 17:40 |
| BobBall | mordred: Actually - as a proposal - how about we just dynamically set the IMAGE_URL in the existing nova plugin? Does your caching script just scrape for the URLs from stackrc? | 17:40 |
| clarkb | greghaynes: but it doesn't call the completion handler | 17:40 |
| jeblair | fungi: they were in stages, so i was hoping if i merged 1 at a time it would be ok | 17:40 |
| fungi | krotscheck: sorry, i reviewed it and forgot to vote. approved now | 17:40 |
| krotscheck | fungi: Sweeeeet | 17:40 |
| clarkb | greghaynes: it iterates over distinct lists on failure and completion for the handlers | 17:40 |
| greghaynes | clarkb: ah, yes. So, that should still not cause this, but that is also a bug | 17:41 |
| clarkb | greghaynes: and JobTracker uses completion to remove from the list | 17:41 |
| fungi | krotscheck: i'll delete the servers themselves later today | 17:41 |
| greghaynes | clarkb: since once you set the event it should return immediately, so we wont block on it | 17:41 |
| clarkb | greghaynes: it would cause this if it caused waitForBuiltImages to never return | 17:41 |
| krotscheck | fungi: \o/ | 17:41 |
| clarkb | greghaynes: which it will | 17:41 |
| krotscheck | greghaynes: We have finished wheels! | 17:41 |
| clarkb | greghaynes: since the list that wait waits on never goes to zero | 17:41 |
| krotscheck | (ish) | 17:41 |
| fungi | krotscheck: zoom zoom! | 17:42 |
| clarkb | greghaynes: while len(self._image_build_jobs.running_jobs) > 0: | 17:42 |
| greghaynes | krotscheck: nice! | 17:42 |
| greghaynes | clarkb: oh! so its busy waiting | 17:42 |
| greghaynes | clarkb: is it spinning in a loop? | 17:42 |
| *** cgalan has joined #openstack-infra | 17:42 | |
| fungi | krotscheck: what shape are the npm mirror patches in? | 17:42 |
| greghaynes | clarkb: I bet that is also the cause of our gearman disconnects if soe | 17:42 |
| rm_work | I'm trying to get requirements back into shape for Octavia so we can get back in the checked projects list... we've got a patch to remove ndg-httpsclient up, but can we get this merged in Requirements since we really shouldn't roll back the other reqs change? https://review.openstack.org/#/c/276549/ | 17:42 |
| clarkb | greghaynes: yes thats what my paste showed, that is our cron thread from a day ago | 17:42 |
| *** flepied has quit IRC | 17:43 | |
| krotscheck | fungi: Not AFS-able | 17:43 |
| greghaynes | clarkb: yea, I mean 100% cpu | 17:43 |
| clarkb | greghaynes: it has a half second sleep so shouldn't pin cpu | 17:43 |
| *** cgalan has quit IRC | 17:43 | |
| krotscheck | fungi: So, nooooo touchey | 17:43 |
| *** yamamoto has joined #openstack-infra | 17:43 | |
| jeblair | krotscheck: like, yet, or ever? | 17:43 |
| krotscheck | Also, they've bitrotted. | 17:43 |
| greghaynes | ah | 17:43 |
| *** baoli has joined #openstack-infra | 17:43 | |
| *** baoli has quit IRC | 17:43 | |
| krotscheck | jeblair: yet. | 17:43 |
| fungi | krotscheck: right, i figured they'd bitrotted and been left behind by the afs switch, just didn't know if you'd started updating any yet | 17:43 |
| *** boris-42 has quit IRC | 17:43 | |
| greghaynes | clarkb: oh, I was conflating in my brain how it is now in https://review.openstack.org/#/c/272384/9/nodepool/nodepool.py | 17:44 |
| *** pvaneck has joined #openstack-infra | 17:44 | |
| krotscheck | jeblair: The mirror project provides hooks which I can use to have it automatically write to an AFS directory, I just need to write the plugin. | 17:44 |
| greghaynes | clarkb: (notice the change of while to a for) | 17:44 |
| zxiiro | electrofelix: no worries. i made a mailing list post since I think more folks should be aware of it. | 17:45 |
| jeblair | krotscheck: oh that sounds nice | 17:45 |
| clarkb | greghaynes: ah yup it should be fixed in that change | 17:45 |
| *** kzaitsev_mb has joined #openstack-infra | 17:45 | |
| clarkb | greghaynes: I will just get that merged and restart service then | 17:45 |
| greghaynes | clarkb: ++ | 17:46 |
| openstackgerrit | Merged openstack-infra/system-config: Use k5start even for the readonly reprepro actions https://review.openstack.org/277487 | 17:46 |
| clarkb | greghaynes: how important is it to get the child of that change in too? | 17:46 |
| *** baoli has joined #openstack-infra | 17:46 | |
| *** e0ne has quit IRC | 17:46 | |
| clarkb | greghaynes: should we worry about those fialing tests or just do the parent for now? | 17:46 |
| *** dahoo has joined #openstack-infra | 17:46 | |
| *** baoli_ has joined #openstack-infra | 17:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Add missing semicolon in run-tox.sh https://review.openstack.org/277492 | 17:47 |
| greghaynes | clarkb: the child only handles a case that comes up during gear disconnect | 17:47 |
| greghaynes | clarkb: so it isnt super important | 17:47 |
| *** yaume has quit IRC | 17:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 3] https://review.openstack.org/277061 | 17:47 |
| greghaynes | it handles the case where we are disconnected and submit a job before the client reconnects | 17:48 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-nodepool: Fix periodic log cleanup https://review.openstack.org/277099 | 17:48 |
| clarkb | jeblair: https://review.openstack.org/#/c/272384/9 fixes the current issue with nodepool image builds. ps8 addressed your comments nad ps9 was a rebase to handle conflicts from bumping the gearman timeouts | 17:48 |
| jeblair | clarkb: thanks, i started reviewing it about 2 minutes ago :) | 17:49 |
| clarkb | BobBall: devstack provides a script that you can run to get all of the images to cache | 17:49 |
| clarkb | BobBall: so any changes should likely go there | 17:49 |
| AJaeger | thanks, jeblair ! | 17:49 |
| krotscheck | jeblair: But, frankly, I've been so focused on mirror things that _everyting_ else on my plate has rotted | 17:50 |
| *** Guest55447 has quit IRC | 17:51 | |
| *** baoli has quit IRC | 17:51 | |
| fungi | krotscheck: well, you've done an awesome job on mirror things | 17:51 |
| krotscheck | fungi: Thanks :) | 17:51 |
| anteaya | hear hear | 17:51 |
| *** kzaitsev_mb has quit IRC | 17:51 | |
| jeblair | krotscheck: fermented, not rotted. :) | 17:51 |
| fungi | krotscheck: thanks right back--this is already having a huge positive impact on stability and performance | 17:51 |
| openstackgerrit | Merged openstack-infra/system-config: Removed pypi.<region> mirror manifests https://review.openstack.org/273518 | 17:53 |
| krotscheck | fungi: I can see the performance graphs, is there a similar graph that maps gate stability? | 17:53 |
| *** sridhar_ram has joined #openstack-infra | 17:53 | |
| fungi | krotscheck: just the elastic-recheck trending for known bugs about connecting to pypi probably | 17:53 |
| openstackgerrit | Merged openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 4] https://review.openstack.org/277069 | 17:54 |
| *** cgalan has joined #openstack-infra | 17:54 | |
| krotscheck | fungi: Coooooolio. Checking | 17:54 |
| fungi | krotscheck: though i guess that was partly solved when we implemented our previous pypi mirrors | 17:54 |
| fungi | krotscheck: so if there are stability improvements directly from the wheel mirrors, it'll likely be around jobs that used to time out trying to build wheels at install time | 17:55 |
| openstackgerrit | Merged openstack-infra/project-config: Increase devstack timeout https://review.openstack.org/277155 | 17:55 |
| krotscheck | fungi: Yeah, I'm not seeing much in grafana | 17:55 |
| krotscheck | I mean logstash | 17:55 |
| krotscheck | Either way, 20 minute slice off of devstack is aesome enough. | 17:55 |
| * krotscheck wonders if anyone outside of infra will notice. | 17:56 | |
| *** armax has joined #openstack-infra | 17:56 | |
| *** shashank_hegde has joined #openstack-infra | 17:56 | |
| fungi | krotscheck: the big stability win will be now that we've got an extensible mirror structure and are adding distro package mirrors, we should soon see a marked drop in job failures due to inconsistent or unreachable provider/upstream mirrors | 17:56 |
| *** fawadkhaliq has joined #openstack-infra | 17:56 | |
| *** asselin__ has left #openstack-infra | 17:56 | |
| *** tzn has joined #openstack-infra | 17:57 | |
| clarkb | however we shouldn't expect much of a speedbump from those because we should already cache the majority of the packages on the images themselves | 17:57 |
| krotscheck | fungi: I can't claim credit for that one, jeblair did most of the work. | 17:57 |
| clarkb | wheels give us performance, apt/yum/etc mirrors give us stability | 17:57 |
| *** daemontool has quit IRC | 17:58 | |
| *** daemontool has joined #openstack-infra | 17:58 | |
| clarkb | rm_work: the requirements project has a set of core reviewers, infra tends to be hands off on that repo unless it is gate breaking | 17:59 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:59 | |
| rm_work | Ah, ok | 17:59 |
| rm_work | I thought those people were here | 17:59 |
| rm_work | Do they have their own channel? | 17:59 |
| clarkb | I think many of them do hang out here and in qa and in stable | 17:59 |
| AJaeger | jeblair: could you toggle your +A on https://review.openstack.org/#/c/277154/ ,please? The dependent job merged... | 17:59 |
| jeblair | rm_work, clarkb: #openstack-dev would be a good place to reach them | 17:59 |
| clarkb | I do not know of a dedicated channel, just want to point out that the infra cores probably won't approve that change | 18:00 |
| *** trown is now known as trown|lunch | 18:00 | |
| jeblair | AJaeger: ack | 18:00 |
| rm_work | clarkb: ah yeah no worries | 18:00 |
| *** annegentle has joined #openstack-infra | 18:00 | |
| rm_work | I just remember asking in here last time for global-reqs and it being effective | 18:00 |
| AJaeger | regarding stability and performance: I noticed that we have a long *post* queue and short *gate* according to zuul status. That's quite a difference. And also 78 jobs in the post queue - older 4hours - sounds a lot jobs merged | 18:01 |
| AJaeger | krotscheck, mordred, jeblair: Great job, thanks! | 18:01 |
| *** kzaitsev_mb has joined #openstack-infra | 18:02 | |
| *** rossella_ has quit IRC | 18:02 | |
| AJaeger | regarding timouts: Should I abandon my ironic change to decrease the timeouts now - or rebase it? https://review.openstack.org/272481 | 18:02 |
| *** rossella_ has joined #openstack-infra | 18:03 | |
| *** vivekd_ has joined #openstack-infra | 18:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu https://review.openstack.org/277516 | 18:03 |
| mordred | jeblair: how does that look? ^ | 18:03 |
| *** vivekd has quit IRC | 18:05 | |
| pleia2 | ooh debian, the things I miss with my globetrotting | 18:05 |
| *** vivekd_ is now known as vivekd | 18:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Remove unecessary session.commit() https://review.openstack.org/277518 | 18:06 |
| AJaeger | jeblair, mtreinish: how shall we continue with the pipeline merges (remove -qa/-neutron)? https://review.openstack.org/#/c/276317 and https://review.openstack.org/#/c/276318 ? | 18:06 |
| AJaeger | welcome back, pleia2 ! | 18:06 |
| jeblair | clarkb, greghaynes: https://review.openstack.org/272384 lgtm; i noted ^ but just fixed it in a followup rather than send it around again. | 18:06 |
| anteaya | yes hello pleia2 | 18:06 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/277295 | 18:07 |
| *** armax has quit IRC | 18:07 | |
| clarkb | jeblair: kk will look at that in a few | 18:07 |
| *** derekh is now known as derekh_afk | 18:07 | |
| AJaeger | pleia2: regarding globetrotting: The first major part of the translations work is in and documented in https://review.openstack.org/273759 . Could you put this on your review list for the week, please? | 18:08 |
| *** imcsk8 is now known as imcsk8|afk | 18:08 | |
| *** krtaylor has quit IRC | 18:08 | |
| *** doug-fish has quit IRC | 18:08 | |
| pleia2 | AJaeger: woo! and added to the list, thanks :) | 18:08 |
| *** doug-fish has joined #openstack-infra | 18:09 | |
| *** daemontool has quit IRC | 18:09 | |
| jeblair | mordred: i think i see 1 thing in 277516 | 18:09 |
| *** daemontool_ has joined #openstack-infra | 18:09 | |
| AJaeger | pleia2: amotoki deserves most of the credit for the implementation | 18:09 |
| openstackgerrit | Merged openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support https://review.openstack.org/272384 | 18:09 |
| pleia2 | AJaeger: they were the one making a lot of comments in your reviews, yeah? | 18:11 |
| *** armax has joined #openstack-infra | 18:12 | |
| *** jsavak has joined #openstack-infra | 18:12 | |
| fungi | jeblair: do you have a sec to help me dig into mailman logs? the release announcement job actually delivers the message to lists.o.o and i can in turn see exim there successfully handing off to the mailman router, but find no corresponding log entries from mailman itself: 2016-02-06 16:03:44 1aS5Kq-0007AT-MO => openstack-dev <openstack-dev@lists.openstack.org> R=mailman_router T=mailman_transport | 18:12 |
| *** Sukhdev has quit IRC | 18:12 | |
| jeblair | fungi: yes | 18:12 |
| *** doug-fish has quit IRC | 18:13 | |
| fungi | jeblair: the corresponding job log is http://logs.openstack.org/b7/b7329402f759da3981e34258f67fe3b2dfdf93da/release/osprofiler-announce-release/065bfc2/console.html | 18:13 |
| fungi | but after mailman gets the message it seems to me to go into a black hole | 18:13 |
| jeblair | fungi: i don't see a message-id in the exim logs, which is okay -- exim will add one based on the queue id, so we should be able to search on 1aS5Kq-0007AT-MO in the mailman logs | 18:14 |
| *** shardy has quit IRC | 18:15 | |
| AJaeger | pleia2: amotoki reviewed heavily as well, yes. | 18:16 |
| fungi | jeblair: i tried that, to no avail | 18:16 |
| *** jpr has joined #openstack-infra | 18:16 | |
| fungi | jeblair: i also grepped through them by sender address | 18:16 |
| *** jsavak has quit IRC | 18:16 | |
| greghaynes | jeblair: awesome, ty | 18:16 |
| *** ociuhandu has quit IRC | 18:16 | |
| *** ociuhandu has joined #openstack-infra | 18:17 | |
| *** [HeOS] is now known as HeOS | 18:18 | |
| harlowja | mordred yt jeblair and i were talking about said supposed talk-to-yahoo-about-openstack-infra sometime in the future last week, but u were traveling or something | 18:18 |
| harlowja | perhaps can somehow through the talk (which is still ongoing internally) about y! donating resources to the infra-team in that same meeting/talk/blah blah | 18:19 |
| *** krtaylor has joined #openstack-infra | 18:20 | |
| harlowja | so ya we should do it (said talk) :-P | 18:21 |
| *** annegentle has quit IRC | 18:22 | |
| mgagne | I found oauth token to work fine for creating repositories on GitHub contrary to what this comment states: https://github.com/openstack-infra/jeepyb/blob/master/jeepyb/cmd/manage_projects.py#L279-L281 | 18:22 |
| *** bpokorny has joined #openstack-infra | 18:22 | |
| mgagne | how can I update it so I don't break the world? | 18:23 |
| mgagne | mordred ^ since your name shows in git blame | 18:23 |
| jeblair | fungi: i think we have an extremely high number of files in /var/lib/mailman/qfiles/bounces | 18:23 |
| *** markmcclain has quit IRC | 18:24 | |
| fungi | oh, ew. i had looked in the bounces logs, but not the qfiles | 18:25 |
| *** markmcclain has joined #openstack-infra | 18:25 | |
| jeblair | fungi: don't know if that's related yet... | 18:25 |
| mordred | mgagne: neat - I think it maybe did not used to work | 18:25 |
| jeblair | don't really know anything without ls returning | 18:25 |
| mgagne | mordred works with github enterprise 2.1.3 with following scopes: repo, user, write:org | 18:25 |
| *** yamamoto has quit IRC | 18:26 | |
| fungi | jeblair: i have interrupted my ls|wc -l in that case ;) | 18:26 |
| jeblair | fungi: i'll let you know what mine reports :) | 18:26 |
| fungi | jeblair: thanks! | 18:26 |
| *** annegentle has joined #openstack-infra | 18:26 | |
| *** lucasagomes is now known as lucas-dinner | 18:28 | |
| *** dtantsur is now known as dtantsur|afk | 18:28 | |
| *** nadya has joined #openstack-infra | 18:28 | |
| *** nadya is now known as Guest44901 | 18:28 | |
| *** matrohon has joined #openstack-infra | 18:29 | |
| *** eil397 has joined #openstack-infra | 18:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Remove restricted and multiverse https://review.openstack.org/277173 | 18:29 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Remove tooz testing of etcd/consul on liberty or kilo https://review.openstack.org/277526 | 18:30 |
| fungi | mordred: rebase? | 18:31 |
| jeblair | mordred: oops, i closed your screen session | 18:31 |
| fungi | of 277173 | 18:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu https://review.openstack.org/277516 | 18:31 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Remove tooz testing of etcd/consul on liberty or kilo https://review.openstack.org/277526 | 18:31 |
| jeblair | mordred: sorry | 18:31 |
| mordred | fungi: yes. it had merge conflict | 18:31 |
| *** armax has quit IRC | 18:31 | |
| *** flepied has joined #openstack-infra | 18:31 | |
| *** exploreshaifali has joined #openstack-infra | 18:32 | |
| jeblair | fungi: 868560 | 18:32 |
| mordred | mgagne: any idea if it works with public github? | 18:32 |
| *** Guest44901 has quit IRC | 18:32 | |
| jeblair | fungi: list in ~root/bounces | 18:32 |
| fungi | jeblair: that's a very large number, yes | 18:32 |
| *** jsavak has joined #openstack-infra | 18:33 | |
| fungi | jeblair: those are message-ids? | 18:33 |
| mgagne | mordred I can test. after further test, it only needs "repo or public_repo" and write:org scopes. | 18:33 |
| mgagne | mordred testing with github pub now | 18:33 |
| mordred | mgagne: I think the best way to fix this would be to add an option to the config file for adding an oauth $something | 18:33 |
| mordred | mgagne: (not sure what the oauth python workflow for this would look like) | 18:34 |
| *** baoli_ has quit IRC | 18:34 | |
| mordred | mgagne: and update manage-projects to use oauth creds if they're there, and username/password otherwise | 18:34 |
| mgagne | mordred there is a way to make it work in a backward compat way but would need lot of logic | 18:34 |
| *** baoli has joined #openstack-infra | 18:34 | |
| *** savihou has joined #openstack-infra | 18:34 | |
| mgagne | mordred like if your oauth token doesn't have right scopes... | 18:34 |
| *** kdas_ has joined #openstack-infra | 18:35 | |
| jeblair | fungi: i don't think so -- i think mailman names its pickle files through some kind of timestamp/uid thing... i don't know whether/how to associate those with anything else | 18:35 |
| mgagne | mordred I found jeepyb not to be that much verbose when it comes to error handling | 18:35 |
| *** kushal has quit IRC | 18:35 | |
| mordred | mgagne: yah - it's not one of my finest pieces of code | 18:35 |
| fungi | jeblair: the prefix looks like unix time, so i'm looking for things around 1454774640 | 18:35 |
| jeblair | fungi: i do not see a bounce runner running | 18:35 |
| fungi | (2016-02-06T16:04) | 18:35 |
| *** eil397 has quit IRC | 18:35 | |
| mordred | mgagne: the logic difference really should just be in the constructor, right? | 18:36 |
| *** eil397 has joined #openstack-infra | 18:36 | |
| mordred | mgagne: so you should be able to do "if oauth in config: github(oauth) else: github(username,password)" right? | 18:36 |
| mgagne | mordred yes | 18:36 |
| *** baoli has quit IRC | 18:36 | |
| mordred | woot! | 18:36 |
| mgagne | https://github.com/mgagne/jeepyb/commit/d211abec1ef82d04aa4f00d4ebb2aa7a146bf3e8 | 18:36 |
| mordred | oh - that looks great | 18:37 |
| *** matrohon has quit IRC | 18:37 | |
| mordred | I see no problem landing that as it is upstream | 18:37 |
| jeblair | fungi: i think we should remove (or move) those files and declare bankruptcy on bounces and restart mm | 18:37 |
| mgagne | mordred will do now | 18:37 |
| fungi | jeblair: i absolutely agree there | 18:37 |
| *** slogan621 has joined #openstack-infra | 18:38 | |
| jeblair | fungi: i await the results of your search for any clues regarding our lost message | 18:38 |
| fungi | jeblair: there are a few files in the right time range to be candidates so i'll let you know in a sec | 18:38 |
| *** arxcruz has quit IRC | 18:38 | |
| *** matrohon has joined #openstack-infra | 18:39 | |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/jeepyb: Use oauth token to create GitHub project if available https://review.openstack.org/277527 | 18:39 |
| fungi | jeblair: /var/lib/mailman/qfiles/bounces/1454774626.386724+22e4b301acc4b97bf2c1d61245aaa5650134fb89.pck | 18:41 |
| *** trown|lunch is now known as trown | 18:42 | |
| *** tzn has quit IRC | 18:42 | |
| *** abregman is now known as abregman|afk | 18:43 | |
| fungi | jeblair: "You have to be a subscriber to post to this mailing list, so your\nmessage has been automatically rejected." | 18:44 |
| fungi | jeblair: thanks, now i know where to dig for this stuff | 18:44 |
| *** e0ne has joined #openstack-infra | 18:44 | |
| jeblair | that'll do it | 18:44 |
| *** armax has joined #openstack-infra | 18:44 | |
| jeblair | fungi: and it got stuck there since the bounce runner isn't running | 18:44 |
| * AJaeger just blogged about templates in Zuul - http://jaegerandi.blogspot.de/2016/02/templates-in-openstacks-zuul.html . If there's anything that should be added to infra manual, feel free to propose a change or tell me | 18:45 | |
| fungi | jeblair: right, which is why i didn't find it in the bounces log | 18:46 |
| fungi | jeblair: okay, so i should move /var/lib/mailman/qfiles/bounces elsewhere and create an empty directory there, then restart the mailman initscript? | 18:47 |
| *** piet has quit IRC | 18:47 | |
| *** _amrith_ is now known as amrith | 18:48 | |
| jeblair | fungi: oh, i think it may be slightly more complex -- i think announce -> list as queue id 1aS5Kq-0007AT-MO; list rejects due to sender not subscribed; rejection goes to exim as queue id R=1aS5Ks-0007GV-3e; exim can not deliver it because no-reply@o.o does not exist; exim sends bounce to mailman as queue id 1aS5Ks-0007Gy-8U; then *that* gets stuck in the bounce queue because there is no runner | 18:48 |
| odyssey4me | nice work there AJaeger :) | 18:50 |
| jeblair | fungi, mordred: maybe we can come up with an ansible play to run jjb on all the jenkins masters? | 18:52 |
| AJaeger | thanks, odyssey4me | 18:52 |
| *** jsavak has quit IRC | 18:52 | |
| fungi | jeblair: that sounds like a thrilling convenience | 18:52 |
| odyssey4me | AJaeger now you can just tell people to rtfm :p | 18:52 |
| fungi | (compared to starting screen sessions on 8 servers and having to track them all) | 18:52 |
| *** jsavak has joined #openstack-infra | 18:53 | |
| mordred | jeblair: yes. I think that sounds completely sane | 18:53 |
| jeblair | i'll see if i can whip something up | 18:53 |
| fungi | jeblair: any objection to my plan for moving/recreatign the bounces dir? | 18:54 |
| jeblair | fungi: oh missed that, sorry. lgtm. | 18:54 |
| *** esikachev has quit IRC | 18:54 | |
| *** esikachev has joined #openstack-infra | 18:54 | |
| fungi | looks like it's mode 2770 owned list:list | 18:56 |
| AJaeger | odyssey4me: we hae the infra manual for rtfm but too often people still ask - that's why I'm aksing here for feedback what's missing there and should be added... | 18:58 |
| fungi | jeblair: /var/lib/mailman/qfiles/bounces.old is the old one | 18:58 |
| fungi | can recursively delete once we're comfortable | 18:58 |
| AJaeger | odyssey4me: but yes, sometimes a different way to present content might help as well... | 18:58 |
| odyssey4me | AJaeger hahaha, I made a comment the other day to someone that people don't ready manuals, but they do read blog posts! | 18:58 |
| AJaeger | ;) | 18:58 |
| *** salv-orl_ has joined #openstack-infra | 18:58 | |
| nibalizer | jeblair: ++ | 18:59 |
| odyssey4me | I do think that perhaps it has something to do with the search ranking of the manuals though. Blogs often rank higher. | 18:59 |
| *** jsavak has quit IRC | 18:59 | |
| jeblair | nibalizer, fungi, mordred: http://paste.openstack.org/show/486322/ is that all we should do? | 18:59 |
| *** jsavak has joined #openstack-infra | 19:00 | |
| *** matrohon has quit IRC | 19:00 | |
| mordred | jeblair: yes, that looks good | 19:00 |
| pleia2 | odyssey4me: I think you're right | 19:00 |
| nibalizer | maybe update project-config in that playbook? | 19:00 |
| nibalizer | or just use the version puppet laid down I guess? | 19:00 |
| pleia2 | odyssey4me: maybe we should write blog posts that point to docs :D | 19:01 |
| jeblair | nibalizer: yeah, i'd rather use the latter -- this is mostly for when we know that jjb has timed out on its normal run | 19:01 |
| mordred | jeblair: might be worth looking at http://docs.ansible.com/ansible/playbooks_async.html | 19:01 |
| jeblair | mordred: oh, is ansible going to have a default timeout or anything? | 19:01 |
| jeblair | mordred: heh | 19:01 |
| *** salv-orlando has quit IRC | 19:01 | |
| *** amrith is now known as _amrith_ | 19:02 | |
| mgagne | mordred works fine with public github for me | 19:02 |
| nibalizer | jeblair: oh so you don't see this running out of cron on the bastion and taking over for that puppet exec that runs it now | 19:02 |
| nibalizer | that wfm | 19:02 |
| mordred | jeblair: so async: 3600 poll: 30 or something - then you can run ansible-playbook -f 8 your-playbook.yaml and it _should_ work | 19:02 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:02 | |
| jeblair | nibalizer: right, this is corrective action for exceptional circumstances | 19:02 |
| fungi | jeblair: the --conf is unnecessary but i suppose being explicit is fine | 19:02 |
| nibalizer | mordred: sync is probbaly ok ? | 19:02 |
| AJaeger | pleia2: should I add more links next time? ;) | 19:02 |
| nibalizer | maybe sync for the first one then async for the rest | 19:03 |
| mordred | nibalizer: the issue is the timeout | 19:03 |
| nibalizer | if something is wrong we don't want to break them all at the same time | 19:03 |
| fungi | jeblair: it would also help to git pull --ff-only or git reset --hard origin/master in /etc/project-config first | 19:03 |
| nibalizer | same rough reasoning is in clarks restart script | 19:03 |
| mordred | right - but this is re-running jjb | 19:03 |
| mordred | whichis normally done just by opening 8 screen sessions | 19:03 |
| jeblair | fungi: that's what nibalizer was suggesting, but i was thinking it should already be updated usually... | 19:03 |
| pleia2 | AJaeger: haha, maybe! | 19:04 |
| mordred | and letting them all go and periodically eyeball checking | 19:04 |
| jeblair | fungi: is 1 hour long enough? | 19:04 |
| fungi | jeblair: aside from that, as long as it's running as root and with /root as $HOME it should be equivalent with what i've been doing | 19:04 |
| jeblair | fungi, mordred, nibalizer: https://etherpad.openstack.org/p/Uw4VKo4ZjB | 19:05 |
| fungi | jeblair: it really depends on how excessive the job config update is. i've had jenkins01 and 02 take upwards of 90 minutes to complete (or to finally bomb out on a jenkins api error, more freuquently) | 19:05 |
| jeblair | 7200 it is | 19:05 |
| *** tdurakov has quit IRC | 19:05 | |
| nibalizer | jeblair: lgtm | 19:05 |
| fungi | yeah, that should be fine. the only times it's necessary to make sure project-config updates are in place is if puppet is being separately laggy, or in dire/urgent situations where you don't want to wait for puppet to get around to firing it once | 19:06 |
| jeblair | okay, it's off and running; proposing patch now | 19:07 |
| fungi | or perhaps if you've disabled puppet on the servers, but in those cases you may also want to separately override project-config to a specific commit anyway | 19:07 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Lint YAML files https://review.openstack.org/271845 | 19:08 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix linting errors in YAML files https://review.openstack.org/271844 | 19:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add playbook to run JJB on all jenkins masters https://review.openstack.org/277539 | 19:09 |
| AJaeger | jeblair, mordred: I rebased those yamllint changes. jeblair I looked yesterday closer at them, I think the stylistic issues are all disabled now and we should give this is a try ^ | 19:10 |
| AJaeger | mordred: you still have a -2 on 271844 | 19:10 |
| jeblair | AJaeger: why? | 19:10 |
| nibalizer | crinkle: mordred after lunch would you like to work on moving infracloud-west into management? | 19:10 |
| nibalizer | I believe we are nearly there | 19:10 |
| nibalizer | close enough to JDI | 19:10 |
| AJaeger | jeblair: it will fix issues like duplicate keys | 19:10 |
| jeblair | AJaeger: when has that happened? | 19:11 |
| AJaeger | jeblair: seldom ;) | 19:11 |
| jeblair | AJaeger: i'm not convinced it's useful. | 19:12 |
| AJaeger | jeblair: I'm fine with asking to abandon as well - or merge and revert if it causes problem, I just think we should flip the coin and not let him rebase too often | 19:12 |
| jeblair | AJaeger: most of our yaml is structured to avoid the temptation to have duplicate keys. | 19:12 |
| AJaeger | jeblair: indeed | 19:12 |
| fungi | also i still feel like duplicate key checking should be the realm of the yaml parser itself, if the pyyaml devs ever get around to making an option for that | 19:13 |
| crinkle | nibalizer: I added an agenda item for tomorrow's meeting, maybe we should wait for more discussion to happen? | 19:13 |
| jeblair | fungi: indeed. | 19:13 |
| fungi | i occasionally check back on that bug, and then get depressed that it's still going nowhere | 19:14 |
| AJaeger | fungi: I agree, that would be the best place. | 19:14 |
| fungi | as in there still seems to be no consensus from pyyaml upstream that it's desirable to check for duplicate keys | 19:14 |
| AJaeger | I'll add a comment asking to abandon, saying that the cases it covers are rarely hit... | 19:14 |
| nibalizer | crinkle: i mostly just mean putting the infracloud controller and infracloud compute hosts under bastion control | 19:14 |
| fungi | and now there's a fork of pyyaml which does error on duplicate keys, but it also comes with tons of other baggage dependencies | 19:14 |
| nibalizer | so having ansible run puppet on them | 19:14 |
| nibalizer | not turning on nodepool | 19:15 |
| crinkle | nibalizer: okay | 19:15 |
| nibalizer | that work is going to be a bit touchy | 19:15 |
| *** doug-fish has joined #openstack-infra | 19:16 | |
| *** sbalukoff has joined #openstack-infra | 19:16 | |
| *** jgriffith is now known as jgriffith_away | 19:16 | |
| jeblair | mordred: legit failure at http://logs.openstack.org/16/277516/2/check/gate-infra-puppet-apply-bare-trusty/30cf579/console.html#_2016-02-08_19_03_26_367 | 19:16 |
| openstackgerrit | Spencer Krum proposed openstack-infra/publications: ConfigMgmtConf 2016 by Spencer Krum (nibalizer) https://review.openstack.org/277541 | 19:17 |
| mordred | jeblair: GAH | 19:17 |
| jeblair | mordred: left comment on what i think the fix is | 19:17 |
| nibalizer | how do i git review to a target branch? | 19:18 |
| jeblair | nibalizer: git review <branch> | 19:18 |
| nibalizer | oh just branch | 19:18 |
| nibalizer | ty | 19:18 |
| mordred | jeblair: you are correct | 19:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu https://review.openstack.org/277516 | 19:18 |
| jeblair | (and git restack follows the same pattern) | 19:18 |
| fungi | is this what the openstack health dashboard is supposed to look like? http://health.openstack.org/ | 19:18 |
| *** rbrndt_ has quit IRC | 19:18 | |
| nibalizer | fungi: i think thats the api server | 19:19 |
| AJaeger | fungi: the URL looks correct, the JSON not | 19:19 |
| *** adrienverge has joined #openstack-infra | 19:19 | |
| clarkb | http://status.openstack.org/openstack-health/# is the dashboard | 19:19 |
| clarkb | health.o.o is just the rest api | 19:19 |
| fungi | aha, thanks clarkb | 19:19 |
| pleia2 | nibalizer: referring to handling publication changes? | 19:19 |
| mordred | jeblair: oh - you know - we should maybe make the lockfile in the cron different too | 19:19 |
| adrienverge | Hi | 19:20 |
| nibalizer | pleia2: hahaa yep | 19:20 |
| pleia2 | nibalizer: https://git.openstack.org/cgit/openstack-infra/publications/tree/README.rst?h=template | 19:20 |
| pleia2 | I wrote this handy thing :) | 19:20 |
| nibalizer | haha | 19:20 |
| mordred | jeblair: I'll need to spin up another flock screen for it | 19:20 |
| nibalizer | i think you link this to me every time i start flailing | 19:20 |
| pleia2 | nibalizer: I mess it up every time, so docs++ | 19:21 |
| adrienverge | AJaeger: About #271845, could you explain the reason? Sorry I wasn't on irc | 19:21 |
| nibalizer | woot | 19:21 |
| *** doug-fish has quit IRC | 19:21 | |
| *** pradk has quit IRC | 19:21 | |
| nibalizer | woot https://review.openstack.org/#/c/277542/ | 19:22 |
| AJaeger | adrienverge: read backscroll at eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-02-08.log | 19:22 |
| adrienverge | OK, just read on eavesdrop.openstack.prg | 19:22 |
| nibalizer | also minor announcement, my talk ^^ that talk, got noted in the DevOps Weekly newsletter | 19:22 |
| nibalizer | so thats pretty cool | 19:22 |
| pleia2 | nibalizer: nice, congrats | 19:22 |
| *** doug-fish has joined #openstack-infra | 19:22 | |
| pleia2 | nibalizer: is there video? | 19:22 |
| AJaeger | cool, nibalizer ! | 19:22 |
| adrienverge | the key duplication actually happens (for instance, there are currently some in heat-templates, etc.) | 19:22 |
| * pleia2 sees http://spencerkrum.com/talks/better-together-cfgmgmtcamp2016/slides/#1 \o/ | 19:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu https://review.openstack.org/277516 | 19:23 |
| mordred | jeblair: ^^ ok - I'm now holding both locks | 19:23 |
| AJaeger | adrienverge: but not in project-config... | 19:24 |
| adrienverge | And upstream (pyyaml) wont provide such a "key duplication" detection feature | 19:24 |
| *** ifarkas has quit IRC | 19:24 | |
| jeblair | mordred: if you head down that road, we need to make unref distro-specific | 19:24 |
| adrienverge | AJaeger: That's true | 19:24 |
| *** bardia has joined #openstack-infra | 19:25 | |
| adrienverge | AJaeger: but enabling this linter has little cost, and makes sure it won't happen like in other projects | 19:25 |
| bardia | anyone which channel to go to for tempest issues? | 19:25 |
| jeblair | mordred: anything else that might be shared between them? i'm thinking most of it has moved into afs now... | 19:25 |
| jeblair | bardia: #openstack-qa | 19:25 |
| clarkb | pyyaml actually says you should modify the yaml parser class to detect it if you need it | 19:25 |
| clarkb | salt does this | 19:25 |
| *** Sukhdev has joined #openstack-infra | 19:25 | |
| mordred | jeblair: I cannot parse your sentence | 19:26 |
| mordred | jeblair: unref? | 19:26 |
| bardia | jeblair: thanks | 19:26 |
| AJaeger | adrienverge: as jeblair said: This does happen so rarely that we don't need to check for it | 19:26 |
| jeblair | mordred: see comment | 19:26 |
| clarkb | it is fairly simple code iirc, you keep a hash table of keys and check if any key already exists as you go | 19:26 |
| jeblair | clarkb: i would not be opposed to our tools growing that, or even us releasing a little lib to automate that. | 19:26 |
| *** armax has quit IRC | 19:26 | |
| mordred | jeblair: AH | 19:26 |
| *** electrofelix has quit IRC | 19:26 | |
| adrienverge | AJaeger: alright, got it | 19:27 |
| * mordred has idea | 19:27 | |
| adrienverge | clarkb: Sure, but we don't want to do it in oopenstack-infra when tools exists | 19:27 |
| jeblair | clarkb, adrienverge: because a program should be able to detect its own syntax errors | 19:27 |
| fungi | clarkb: i think you'd need to subclass part of pyyaml if you want to deep check that on load (nested associative arrays). i looked into it briefly at one point | 19:27 |
| clarkb | fungi: yup, that is what salt does | 19:27 |
| clarkb | and jjb does it for other reasons | 19:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu https://review.openstack.org/277516 | 19:28 |
| clarkb | let me find the salt code | 19:28 |
| mordred | jeblair: how about that? | 19:28 |
| mordred | jeblair: (and nice catch) | 19:28 |
| jeblair | mordred: wfm | 19:28 |
| mordred | jeblair: it will be an ugly file name - but in the realm of things I care about - that's not one of them | 19:29 |
| jeblair | changed: [jenkins.openstack.org] | 19:29 |
| jeblair | 1/8 complete :) | 19:29 |
| clarkb | https://github.com/saltstack/salt/blob/develop/salt/utils/yamlloader.py#L80 | 19:29 |
| clarkb | jeblair: greghaynes you ok with me restarting nodepool now to pick up that latest disconnect handling patchset? | 19:31 |
| greghaynes | clarkb: go for it | 19:31 |
| *** bpokorny_ has joined #openstack-infra | 19:31 | |
| jeblair | clarkb: ++ | 19:31 |
| *** annegentle has quit IRC | 19:32 | |
| *** akshai has joined #openstack-infra | 19:32 | |
| clarkb | done | 19:32 |
| *** bpokorny_ has quit IRC | 19:34 | |
| *** bpokorny has quit IRC | 19:34 | |
| clarkb | bluebox may have leaked floating IPs | 19:34 |
| *** bardia has left #openstack-infra | 19:34 | |
| *** armax has joined #openstack-infra | 19:34 | |
| *** bpokorny has joined #openstack-infra | 19:34 | |
| clarkb | mordred: ^ maybe we should make a test case for this with devstack/tempest then see if we can get it fixed | 19:35 |
| clarkb | hrm I wonder if restarting nodepool causes the leak. Oustanding requests may not be associated iwth an instance so when the instance gets deleted the fip remains? | 19:36 |
| *** f1ller- is now known as filler | 19:36 | |
| *** armax has quit IRC | 19:36 | |
| *** esker has joined #openstack-infra | 19:37 | |
| *** esker has quit IRC | 19:37 | |
| mordred | clarkb: yah. that's likely | 19:37 |
| *** esker has joined #openstack-infra | 19:38 | |
| mordred | clarkb: once image workers is stable and I can re-focus on shade-nodepool, the change in fip allocation model in shade should allow us to stop leaking fips that way | 19:38 |
| clarkb | ok | 19:38 |
| clarkb | also I think we are really really close to that stable image builders | 19:38 |
| mordred | clarkb: since the fip create call is done with a server port in the first place, so we'll always have the tracking info | 19:39 |
| mordred | clarkb: awesome | 19:39 |
| clarkb | I am working on doing a 5 minute fip list diff and will cleanup the unused fips | 19:39 |
| mordred | clarkb: I'm almost done with apt mirrors, so will have brainspace to do shade-nodepool next | 19:39 |
| prometheanfire | mordred: who else is core in glean? | 19:39 |
| mordred | prometheanfire: greghaynes | 19:39 |
| greghaynes | I think SpamapS is now too | 19:39 |
| prometheanfire | https://review.openstack.org/#/c/276610/ | 19:40 |
| prometheanfire | :D | 19:40 |
| *** fawadkhaliq has quit IRC | 19:40 | |
| SpamapS | I am | 19:41 |
| *** dprince has quit IRC | 19:41 | |
| SpamapS | I need to hit the review queue today actually. | 19:41 |
| prometheanfire | there are a couple of DIB fixes too, but I imagine it's in people's queues | 19:41 |
| *** piet has joined #openstack-infra | 19:42 | |
| *** cloudtrainme has quit IRC | 19:42 | |
| openstackgerrit | Merged openstack-infra/system-config: Remove restricted and multiverse https://review.openstack.org/277173 | 19:42 |
| *** tdurakov has joined #openstack-infra | 19:43 | |
| *** fitoduarte has joined #openstack-infra | 19:43 | |
| yolanda | hi SpamapS, i have several changes in glean and dib that need review | 19:45 |
| yolanda | i heard about you checking the review queue... | 19:45 |
| *** ociuhandu has quit IRC | 19:46 | |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Support stable requirements https://review.openstack.org/277550 | 19:48 |
| prometheanfire | SpamapS: you failed to be seen sir | 19:48 |
| prometheanfire | or more correctly failed to not be seen | 19:48 |
| * SpamapS [boom]<wwaahhhhhh> | 19:48 | |
| clarkb | bluebox floating IPs cleaned up | 19:49 |
| SpamapS | and now for something completely different (lunch) | 19:49 |
| prometheanfire | lol | 19:49 |
| prometheanfire | ya, 10m til meeting here | 19:49 |
| ianw | prometheanfire: re your question some 6 hours ago ... finalise.d does run in the build chroot ... see http://docs.openstack.org/developer/diskimage-builder/developer/developing_elements.html | 19:50 |
| *** kzaitsev_mb has quit IRC | 19:50 | |
| *** jsavak has quit IRC | 19:50 | |
| prometheanfire | ianw: ah, ok, I'll move it there then | 19:50 |
| prometheanfire | do I need to include the finalize element then? | 19:50 |
| prometheanfire | if there is one | 19:51 |
| prometheanfire | haven't looked | 19:51 |
| clarkb | greghaynes: those test fails on the cild change are pretty conssitent | 19:51 |
| AJaeger | jeblair, fungi: What do you want to do with removal of periodic-qa and periodic-neutron queues, now? mtreinish commented on https://review.openstack.org/#/c/276318 and this might need another reply. Some of these periodic jobs are currently broken, the whole patch series fix them - so I'd like to know how to continue and would appreciate your help here, please | 19:51 |
| *** jsavak has joined #openstack-infra | 19:51 | |
| clarkb | greghaynes: do you have time to look closer at them or should I do that | 19:51 |
| *** rbrndt_ has joined #openstack-infra | 19:51 | |
| greghaynes | clarkb: I cant right now... | 19:51 |
| *** Sukhdev has quit IRC | 19:52 | |
| jeblair | AJaeger: i replied on 318 | 19:52 |
| greghaynes | Although I would really prefer that over debugging this pptp issue | 19:52 |
| prometheanfire | ianw: I specificly don't want finalize.d because it happens after the copy | 19:52 |
| ianw | clarkb: you talking about the image deletion (https://review.openstack.org/#/c/273703/) ? i was having a look yesterday | 19:52 |
| *** salv-orl_ has quit IRC | 19:53 | |
| clarkb | ianw: yup | 19:54 |
| AJaeger | jeblair: yes, you did. So, ok to +A your changes now? | 19:54 |
| clarkb | ianw: did you see that I got fedora-23 updated everywhere late my friday? | 19:55 |
| prometheanfire | ianw: so, should stay in post-install.d | 19:55 |
| greghaynes | ianw: I left a reply to your comment on there btw | 19:55 |
| clarkb | ianw: the root / was grown to 15GB which seemed small but I wasn't sure if that was making use of the entire available disk | 19:55 |
| greghaynes | ianw: the fail you mentioned is a fail, but thats a known one I think that happens during cleanup. The real fail there is that we timed out (which kicked off the cleanup) | 19:56 |
| *** nadya has joined #openstack-infra | 19:56 | |
| *** daemontool_ has quit IRC | 19:56 | |
| ianw | clarkb: hmm, ok, well growroot should be filling up the disk. i'll kick something off and check it out | 19:56 |
| *** nadya is now known as Guest69865 | 19:56 | |
| *** jsavak has quit IRC | 19:57 | |
| *** Sukhdev has joined #openstack-infra | 19:57 | |
| ianw | greghaynes: hmm, ok, it did look rather suspicious :) | 19:57 |
| *** sfinucan has quit IRC | 19:57 | |
| *** tongli has quit IRC | 19:57 | |
| *** akshai has quit IRC | 19:57 | |
| *** jsavak has joined #openstack-infra | 19:58 | |
| *** Guest69865 has quit IRC | 19:59 | |
| ianw | prometheanfire: well, there's some precedent for doing things like cleanup in finalise.d -- see existing redhat-common & ironic-agent elements. another option maybe package-installs should move earlier (98) so it's possible to put another thing after it. get some input from greghaynes/SpamapS | 20:00 |
| prometheanfire | if they respond to me :( | 20:00 |
| *** jtomasek_ has quit IRC | 20:00 | |
| greghaynes | If you leave comments in the reviews i'll reploy when I get to it | 20:01 |
| *** woodster_ has joined #openstack-infra | 20:02 | |
| *** akshai has joined #openstack-infra | 20:02 | |
| *** bpokorny has quit IRC | 20:03 | |
| *** arxcruz has joined #openstack-infra | 20:04 | |
| *** bpokorny has joined #openstack-infra | 20:04 | |
| ianw | clarkb: http://nodepool.openstack.org/dib.fedora-23.log -- 2016-02-07 14:48 : is that a day behind? failed to build as timed out downloading a cirros image from citrix | 20:04 |
| *** annegentle has joined #openstack-infra | 20:04 | |
| greghaynes | ianw: nodepool was wedged and not doign builds yesterday | 20:05 |
| *** dprince has joined #openstack-infra | 20:05 | |
| *** vgridnev has quit IRC | 20:05 | |
| ianw | greghaynes: ahh, so situation normal ;) | 20:05 |
| greghaynes | ianw: I think as a result of that | 20:05 |
| *** ociuhandu has joined #openstack-infra | 20:05 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 20:05 |
| clarkb | ianw: yes we don't hae them as of today, but they were built late friday | 20:05 |
| *** eil397 has quit IRC | 20:06 | |
| AJaeger | dims, dhellmann: Could you review https://review.openstack.org/#/c/277550 for a requirments related change in project-config, please? | 20:07 |
| ianw | clarkb: could you hold/put my key on fedora-23-rax-iad-7793913 so i can poke? | 20:07 |
| clarkb | ianw: sure | 20:07 |
| mordred | jeblair: I believe once https://review.openstack.org/#/c/277516/ lands - I should remove the crufty files in /etc/reprepro, then we should unmount the volume in afs, rename it, make new mount point and mount it, yeah? | 20:08 |
| dims | AJaeger : looking | 20:08 |
| clarkb | ianw: can you point me to your public key? | 20:08 |
| ianw | clarkb: https://github.com/ianw.keys | 20:08 |
| jeblair | mordred: yep | 20:08 |
| *** cgalan has quit IRC | 20:08 | |
| ianw | clarkb: one ending in UcMr is really only one i need | 20:09 |
| clarkb | ianw: I added all of them :) jenkins@162.209.111.210 | 20:09 |
| mrodden | ianw you working on bashate release? | 20:09 |
| mordred | jeblair: cool. | 20:09 |
| ianw | mrodden: i have a whole heap of stuff queued up for ages unreviewed | 20:10 |
| mordred | clarkb: you wanna +A 277516 for me? :) | 20:10 |
| AJaeger | thanks, dims | 20:10 |
| mrodden | ianw, for bashate...? | 20:10 |
| mrodden | we probably need to get out a release just for the pbr > 1.0 fix | 20:10 |
| *** daemontool has joined #openstack-infra | 20:10 | |
| ianw | mrodden: yes. it fixes a whole lot of stuff https://review.openstack.org/#/q/status:open+project:openstack-dev/bashate+branch:master+topic:continuation | 20:11 |
| ianw | mrodden: but i guess it needs a rebase at this point, i'll try to get to that asap | 20:11 |
| clarkb | mordred: its not urgent right? we don't debian yet | 20:11 |
| mrodden | well we have plenty of delta already between 0.3.2 and the next release including a drop of py26 support | 20:11 |
| clarkb | oh! you want the paths to be correct before using the mirror | 20:11 |
| clarkb | I see | 20:12 |
| mrodden | ianw, we should just do it and then 0.4.1 when we get that fixed... | 20:12 |
| *** cloudtrainme has joined #openstack-infra | 20:12 | |
| mordred | clarkb: yah - it's less about debian and more about moving ubuntu | 20:12 |
| clarkb | mordred: https://review.openstack.org/#/c/277516/5/modules/openstack_project/manifests/reprepro.pp should become a defined type otherwise you can only instantiate it once | 20:12 |
| mordred | before go-live | 20:12 |
| clarkb | mordred: you can do that in a follow up if you just want to ge tthe /ubuntu path in | 20:13 |
| *** tdurakov has quit IRC | 20:13 | |
| *** jgriffith_away is now known as jgriffith | 20:13 | |
| mordred | clarkb: let's do that as a followup - but good call | 20:13 |
| *** armax has joined #openstack-infra | 20:13 | |
| ianw | clarkb: this looks sad :( growroot[419]: find: missing argument to `-name' | 20:13 |
| *** tdurakov has joined #openstack-infra | 20:13 | |
| *** rockyg has joined #openstack-infra | 20:13 | |
| *** tdurakov has quit IRC | 20:13 | |
| *** gildub has joined #openstack-infra | 20:14 | |
| *** esp has joined #openstack-infra | 20:14 | |
| clarkb | mordred: http://logs.openstack.org/16/277516/5/check/gate-infra-puppet-apply-bare-trusty/123ca90/console.html#_2016-02-08_19_48_48_535 | 20:15 |
| clarkb | so I will -1 based on ^ | 20:15 |
| openstackgerrit | Ben Kero proposed openstack/diskimage-builder: Replace sfdisk partitioning with parted https://review.openstack.org/258721 | 20:15 |
| mordred | clarkb: wow. I totally thought I'd fixed that already | 20:16 |
| *** tcammann has joined #openstack-infra | 20:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu https://review.openstack.org/277516 | 20:16 |
| mordred | clarkb: sorry about that | 20:17 |
| *** ccrouch has quit IRC | 20:17 | |
| *** NobodyCa1 has joined #openstack-infra | 20:17 | |
| openstackgerrit | Merged openstack/diskimage-builder: Make dkms element depend on dkms package https://review.openstack.org/261772 | 20:17 |
| *** armax has quit IRC | 20:17 | |
| *** armax has joined #openstack-infra | 20:18 | |
| *** adrienverge has quit IRC | 20:18 | |
| clarkb | ianw: once hte job on that host finishes you should be able to manually do a grow root to figure out what the invocation needs to be (maybe during job too) | 20:19 |
| *** rockyg has quit IRC | 20:19 | |
| clarkb | ianw: then I guess it is patch dib and rebuild images? | 20:19 |
| fungi | jeblair: mordred: so after i kinit and aklog as fungi i'm getting a permission denied trying to delete things under /afs/.openstack.org/mirror/wheel/ubuntu-14.04-x86_64/ | 20:19 |
| fungi | jeblair: mordred: should i be using fungi.admin instead? | 20:19 |
| ianw | clarkb: yeah, seems it's not that easy ... the script works *outside* systemd. | 20:19 |
| jeblair | fungi: yes, your normal account doesn't have write perms | 20:20 |
| jeblair | fungi: only fungi.admin is in system:administrators | 20:20 |
| ianw | clarkb: maybe ordering ... udevadm might need something else up? | 20:20 |
| fungi | okay, thanks, thought that might be it but wanted to be sure before i possibly made a mess | 20:20 |
| jeblair | fungi: you can use "fs la /afs/.openstack.org/mirror/wheel/ubuntu-14.04-x86_64/" to check perms; and "pts membership system:administrators" to check membership | 20:20 |
| mordred | jeblair: one more tiny change in https://review.openstack.org/#/c/277516/ since your last +2 | 20:20 |
| *** tcammann has quit IRC | 20:21 | |
| jeblair | mordred: +3 | 20:21 |
| *** esp has quit IRC | 20:21 | |
| mordred | woot | 20:21 |
| mordred | when it lands I will do the manual steps | 20:21 |
| mordred | jeblair: I may have to ask you some questions about the volume rename stuff | 20:21 |
| mordred | as I've never done that before | 20:21 |
| jeblair | ++ | 20:21 |
| fungi | jeblair: manpage suggests klog -pr fungi.admin should work but i'm getting an rpc error on that... | 20:21 |
| jeblair | fungi: aklog | 20:22 |
| fungi | grrr | 20:22 |
| jeblair | fungi: (no args needed) | 20:22 |
| mordred | jeblair: how does k5start work if there's no aklog involved? | 20:22 |
| jeblair | mordred: it gets tokens | 20:22 |
| jeblair | mordred: it's automagic | 20:22 |
| *** NobodyCa1 has quit IRC | 20:23 | |
| fungi | jeblair: thanks! that seems to have probably worked... though now things are hanging when i try to rm a file. there are some nat workarounds needed too, yeah? | 20:23 |
| ianw | clrakb: yeah, i think i probably just got lucky in the ordering with my testing; i think i need to add some bits to the service file to make sure it starts after udevd sockets are up | 20:23 |
| mordred | fungi: you need the udp | 20:23 |
| fungi | realizing now why i probably didn't try setting this up on my workstation back when we first stood up afs | 20:24 |
| clarkb | ianw: ok | 20:24 |
| mordred | fungi: but once you get it set up it's great | 20:24 |
| ianw | clarkb: thanks, done with fedora-23-rax-iad-7793913 | 20:24 |
| fungi | mordred: thanks, i'll hunt around and see how restrictive i should be able to make the portmap for this | 20:24 |
| *** akshai has quit IRC | 20:24 | |
| jeblair | fungi: should be okay with a connection tracking firewall but you may need to increase the udp conntrack timeout | 20:25 |
| fungi | oh, actually it eventually came back and prompted "rm: remove write-protected regular file ..." but when i said y it errored with "Permission denied" | 20:25 |
| *** |-paul-| has joined #openstack-infra | 20:25 | |
| fungi | jeblair: default timeout for udp sessions on openbsd is crazy long but i'll check that | 20:26 |
| *** dolpher has quit IRC | 20:26 | |
| jeblair | fungi: output of klist? | 20:26 |
| fungi | this is on .openstack.org too so it's not the ro volume | 20:26 |
| *** dolpher has joined #openstack-infra | 20:26 | |
| *** salv-orlando has joined #openstack-infra | 20:27 | |
| fungi | jeblair: oh! it's using fungi as my default principal | 20:27 |
| fungi | do i need to kdestroy that one? | 20:27 |
| jeblair | fungi: yeah; maybe 'kdestroy' | 20:27 |
| jeblair | yep | 20:27 |
| *** tiswanso has quit IRC | 20:27 | |
| jeblair | then 'kinit fungi/admin && aklog' should do it | 20:27 |
| *** tiswanso has joined #openstack-infra | 20:28 | |
| fungi | Default principal: fungi/admin@OPENSTACK.ORG | 20:28 |
| fungi | indeed! | 20:28 |
| fungi | and that will let me remove a file | 20:28 |
| fungi | okay, deleting the old non-normalized wheel trees now | 20:29 |
| mordred | yay! | 20:29 |
| clarkb | I have successfully built an image post service restart and have queued the uploads | 20:29 |
| *** jlk has quit IRC | 20:29 | |
| clarkb | just making sure that things look good on demand and that image builds in general don't break again tomorrow morning | 20:29 |
| *** jlk has joined #openstack-infra | 20:30 | |
| *** jlk has quit IRC | 20:30 | |
| *** jlk has joined #openstack-infra | 20:30 | |
| *** armax has quit IRC | 20:30 | |
| *** esikachev has quit IRC | 20:30 | |
| jeblair | clarkb: i thought you made an imgur with "build an image post service" | 20:31 |
| *** annegentle has quit IRC | 20:31 | |
| jeblair | er built | 20:31 |
| *** dolpher has quit IRC | 20:31 | |
| fungi | better than the old snapshot-based snapchat we used to have | 20:31 |
| fungi | krotscheck: jeblair: mordred: cleanup from http://paste.openstack.org/show/486293/ is now done, if anyone wants to spot-check behind me | 20:31 |
| *** annegentle has joined #openstack-infra | 20:32 | |
| *** cloudtrainme has quit IRC | 20:32 | |
| *** dolpher has joined #openstack-infra | 20:32 | |
| jeblair | fungi: it deletes 2 messages ago. and gets stuck for weeks at a time? | 20:32 |
| clarkb | jeblair: that happens after nodepool does images properly | 20:32 |
| clarkb | that is actually the sort of application that makes perfect sense to host on swift | 20:33 |
| clarkb | there is no real index | 20:33 |
| clarkb | upload, view, delete | 20:33 |
| fungi | jeblair: heh, sounds about right | 20:35 |
| *** ccrouch has joined #openstack-infra | 20:35 | |
| *** salv-orlando has quit IRC | 20:36 | |
| *** akshai has joined #openstack-infra | 20:36 | |
| *** vivekd_ has joined #openstack-infra | 20:37 | |
| *** baoli_ has joined #openstack-infra | 20:38 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: rename script for making stable branches https://review.openstack.org/277499 | 20:40 |
| *** vivekd has quit IRC | 20:40 | |
| *** cloudtrainme has joined #openstack-infra | 20:40 | |
| *** vivekd_ is now known as vivekd | 20:40 | |
| *** ZZelle_ has joined #openstack-infra | 20:41 | |
| *** cloudtrainme has quit IRC | 20:44 | |
| anteaya | time for me to get a walk in | 20:45 |
| dims | ah was just about to ping you :) | 20:46 |
| dims | will do when you get back anteaya :) | 20:46 |
| clarkb | fungi: I think the only outstanding branch delete requests are those for murano which I called out on that thread | 20:46 |
| clarkb | fungi: between your handling of manuals and my grabbing of the others I think we are good | 20:46 |
| fungi | clarkb: thanks for taking those! | 20:47 |
| anteaya | dims: okay thanks | 20:47 |
| *** nadya has joined #openstack-infra | 20:47 | |
| anteaya | dims: be about an hour | 20:47 |
| *** nadya is now known as Guest17540 | 20:47 | |
| *** dizquierdo has joined #openstack-infra | 20:47 | |
| *** hashar has joined #openstack-infra | 20:48 | |
| *** rossella_ has quit IRC | 20:50 | |
| *** cloudtrainme has joined #openstack-infra | 20:50 | |
| *** dims_ has joined #openstack-infra | 20:51 | |
| *** dims has quit IRC | 20:52 | |
| *** mrmartin has joined #openstack-infra | 20:52 | |
| *** Sukhdev has quit IRC | 20:54 | |
| *** mrmartin has quit IRC | 20:54 | |
| *** hichihara has joined #openstack-infra | 20:56 | |
| *** ihrachys has joined #openstack-infra | 20:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/shade: Add environment_files to stack_create https://review.openstack.org/277559 | 20:57 |
| *** cloudtrainme has quit IRC | 20:57 | |
| *** dizquierdo has quit IRC | 20:57 | |
| *** annegentle has quit IRC | 20:58 | |
| openstackgerrit | Merged openstack-infra/system-config: Add mirror-update to cacti https://review.openstack.org/276949 | 20:58 |
| *** annegentle has joined #openstack-infra | 20:59 | |
| *** armax has joined #openstack-infra | 20:59 | |
| openstackgerrit | Merged openstack-infra/puppet-gerrit: Tell puppet to check for cgid instead of cgi https://review.openstack.org/272769 | 21:00 |
| *** salv-orlando has joined #openstack-infra | 21:01 | |
| *** cloudtrainme has joined #openstack-infra | 21:01 | |
| *** annegentle has quit IRC | 21:04 | |
| *** reed_ has joined #openstack-infra | 21:05 | |
| *** cloudtrainme has quit IRC | 21:06 | |
| *** dtardivel has quit IRC | 21:08 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:08 | |
| *** savihou has quit IRC | 21:09 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Set DEVSTACK_GATE_TIMEOUT_BUFFER to 10 https://review.openstack.org/277154 | 21:09 |
| *** eil397 has joined #openstack-infra | 21:10 | |
| *** cloudtrainme has joined #openstack-infra | 21:10 | |
| *** cloudtrainme has quit IRC | 21:10 | |
| *** ddecapit has joined #openstack-infra | 21:10 | |
| *** ddecapit is now known as DuaneDeC7 | 21:10 | |
| openstackgerrit | Merged openstack-infra/puppet-lodgeit: Use modern template syntax https://review.openstack.org/271860 | 21:11 |
| *** fitoduarte has quit IRC | 21:12 | |
| *** eil397 has quit IRC | 21:15 | |
| *** kgiusti has left #openstack-infra | 21:16 | |
| *** cloudtrainme has joined #openstack-infra | 21:16 | |
| *** eil397 has joined #openstack-infra | 21:17 | |
| krotscheck | fungi: Sure, just a sec | 21:18 |
| *** Guest17540 has quit IRC | 21:19 | |
| onovy | dhellmann: hi, can we talk about https://review.openstack.org/#/c/275181/2 ? | 21:19 |
| fungi | moving | 21:20 |
| fungi | er, wrong window | 21:20 |
| dhellmann | onovy : sure, let's move to #openstack-release | 21:20 |
| *** jpeeler has quit IRC | 21:20 | |
| krotscheck | fungi: Looks empty to mee! | 21:21 |
| *** jamespd has left #openstack-infra | 21:23 | |
| clarkb | fungi: is there a tl;dr on where we are for using non bare images for unittests? is it a matter of adding a test to all unittest jobs to install the global bindep list and switch the image label at the same time? | 21:23 |
| clarkb | fungi: then next step would be to use in repo bindep lists? | 21:23 |
| clarkb | fungi: I wnt to try helping with that next since rax is the only place able to run unittests ucrrently | 21:23 |
| *** derekh_afk is now known as derekh | 21:24 | |
| *** dims has joined #openstack-infra | 21:26 | |
| clarkb | also we should take the user survey | 21:26 |
| mrodden | AJaeger, fyi bashate 0.4.0 out now... hopefully helps with the pbr issues | 21:27 |
| mrodden | i dont see haypo on, he was asking for that earlier today | 21:27 |
| *** bpokorny_ has joined #openstack-infra | 21:28 | |
| openstackgerrit | Merged openstack-infra/system-config: Split reprepro config to allow for debian and ubuntu https://review.openstack.org/277516 | 21:28 |
| clarkb | ovh gra image uploads take the longest, which makes sense considering the trans atlantic flight involved | 21:28 |
| *** infra-red has quit IRC | 21:28 | |
| clarkb | we can probably work around that by running a builder in ovh gra and removing that location from the config on the current nodepool host | 21:29 |
| *** dims_ has quit IRC | 21:29 | |
| *** jsavak has quit IRC | 21:30 | |
| clarkb | but then we would not be using the same image across the board, also not a huge issue with the current number of images | 21:31 |
| *** bpokorny has quit IRC | 21:31 | |
| *** jsavak has joined #openstack-infra | 21:31 | |
| *** cloudtrainme has quit IRC | 21:32 | |
| fungi | clarkb: yeah, if dib-based ubuntu-trusty images are uploading to rax consistently now, we can proceed with deciding how to cut over jobs from bare-trusty to it | 21:33 |
| clarkb | oh right that | 21:33 |
| clarkb | I do not think we are consistently uploading to rax yet because the token expiration problem is still a problem | 21:34 |
| clarkb | I guess I should work on that some more | 21:34 |
| *** mtanino has quit IRC | 21:34 | |
| fungi | clarkb: alternatively, we can redo the tox-based jobs to run through all the steps to install missing packages and perform database configuration (those should all be no-ops on bare-trusty anyway) and then switch those jobs to use node: bare-trusty || ubuntu-trusty | 21:35 |
| clarkb | well we should fix the swift upload token thing regardless so I can context switch that back in | 21:35 |
| fungi | but it would be simpler if we just had ubuntu-trusty images on rax first | 21:35 |
| openstackgerrit | Volodymyr Samotiy proposed openstack-infra/project-config: Adding broadview-lib project and jobs. https://review.openstack.org/275873 | 21:36 |
| *** jsavak has quit IRC | 21:36 | |
| *** cloudtrainme has joined #openstack-infra | 21:36 | |
| *** jsavak has joined #openstack-infra | 21:37 | |
| *** annegentle has joined #openstack-infra | 21:37 | |
| fungi | clarkb: problem is so far i've really only tested the proposed job configuration change on gate-nova-python27 so either we need to broaden that experimental significantly _or_ just expect there to be possibly outlying jobs which break for various unforeseen reasons | 21:38 |
| *** jaosorior has quit IRC | 21:38 | |
| clarkb | it shouldn't be too hard to sanity check a wider range of projects | 21:39 |
| fungi | yep, can turn the nova job into a template and add it to a bunch of projects | 21:39 |
| clarkb | and that can happen while we figure out uploads (I will upload new ubuntu-trusty images now so we have something recent to test on) | 21:40 |
| fungi | i'll see if i can do that today or tomorrow and then upload empty dnm changes to the same set of projects | 21:40 |
| fungi | from a transition perspective, i guess our two options are to either directly modify the existing tox-based templates or come up with some parallel versions of them | 21:41 |
| fungi | the former is easier and less churn, though the latter allows for a (messy) incremental transition project-by-project | 21:41 |
| clarkb | I think we go with updating the existing jobs, and just do it when projects aren't in the middle of feature freeze or similar | 21:42 |
| fungi | sgtm | 21:42 |
| fungi | my preference as well | 21:42 |
| clarkb | and start advertising real soon now what the jobs will expect for repo local bindep dep lists | 21:43 |
| clarkb | so that projects can work on getting those in | 21:43 |
| clarkb | (ubuntu-trusty is building now and rax uploads should work because I restarted services not that long ago | 21:43 |
| *** cloudtrainme has quit IRC | 21:45 | |
| mgagne | is there a way to restart the import process of an upstream project in jeepyb? Project is created fine in Gerrit and github but it didn't clone the repo | 21:46 |
| *** cloudtrainme has joined #openstack-infra | 21:47 | |
| clarkb | mgagne: typically no, since imports are destructive to any existing projects we don't run them if the project exists | 21:48 |
| clarkb | mgagne: what you can do is manually force push into gerritthough | 21:48 |
| mgagne | clarkb so it's just a matter of recloing the project myself at the location expected by gerrit? | 21:49 |
| *** Sukhdev has joined #openstack-infra | 21:49 | |
| fungi | clarkb: well, project-specific bindep lists aren't mandatory in my current design, however they are self-testing and will lead to a (slight) increase in efficiency and also we'll likely early on (if not immediately) want to stop accepting additions to the fallback bindep list | 21:49 |
| *** sshnaidm has quit IRC | 21:49 | |
| clarkb | mgagne: sort of, gerrit creates extra stuff that you don't want to just delete so what we do is force push into the gerrit remote | 21:49 |
| *** |-paul-| has quit IRC | 21:49 | |
| jhesketh | Morning | 21:50 |
| clarkb | mgagne: you can see how jeepyb does it, its a simple git push --force +refs/heads/*:refs/heads type deal iirc | 21:50 |
| fungi | right, that's basically what manage-projects does as well | 21:50 |
| fungi | yeah, that | 21:50 |
| clarkb | if you don't do that then you lose the refs/meta/config that gerrit creates by dfeault | 21:50 |
| *** exploreshaifali has quit IRC | 21:50 | |
| mgagne | oh, almost forgot about that one | 21:51 |
| *** ihrachys has quit IRC | 21:51 | |
| mgagne | will read sources | 21:51 |
| *** ihrachys has joined #openstack-infra | 21:51 | |
| *** kzaitsev_mb has quit IRC | 21:52 | |
| clarkb | we could probably do something in jeepyb that detected a half setup project and repushed | 21:52 |
| clarkb | basically check if HEAD is 0000000 or the add .gitreview commit | 21:53 |
| *** kzaitsev_mb has joined #openstack-infra | 21:55 | |
| openstackgerrit | Dina Belova proposed openstack/requirements: Make osprofiler requirements >1.0.0 https://review.openstack.org/277583 | 21:55 |
| *** krtaylor has quit IRC | 21:55 | |
| Sam-I-Am | is there a problem with git.openstack.org? | 21:56 |
| Sam-I-Am | getting clone timeoutes | 21:56 |
| clarkb | Sam-I-Am: which protocol? | 21:57 |
| anteaya | dims: back | 21:57 |
| Sam-I-Am | clarkb: whatever devstack uses... looks like git:// | 21:58 |
| clarkb | grafana at least doesn't report errors | 21:58 |
| *** boris-42 has joined #openstack-infra | 21:58 | |
| clarkb | and the total number of connections should be manageable | 21:58 |
| Sam-I-Am | hmm :/ | 21:59 |
| clarkb | Sam-I-Am: if you have a repo and or source IP I can try to look at specific logs | 21:59 |
| Sam-I-Am | clarkb: seems to be fairly random as to what repo fails. source ip is probably nat'd somewhere. :/ | 22:01 |
| *** DuaneDeC7 has quit IRC | 22:01 | |
| *** regXboi has quit IRC | 22:01 | |
| *** jsavak has quit IRC | 22:01 | |
| greghaynes | clarkb: were you able to poke at that test fail? I have a few mins now | 22:03 |
| Sam-I-Am | clarkb: http://paste.openstack.org/show/486336/ | 22:03 |
| Sam-I-Am | thats all i see | 22:03 |
| *** trown has quit IRC | 22:03 | |
| *** jsavak has joined #openstack-infra | 22:03 | |
| clarkb | greghaynes: I havne't trying to catch up on image builds of various things | 22:03 |
| greghaynes | clarkb: ok, I'll have a look | 22:03 |
| *** e0ne has quit IRC | 22:03 | |
| *** daemontool_ has joined #openstack-infra | 22:04 | |
| clarkb | Sam-I-Am: that implies tcp is failing | 22:04 |
| *** akshai has quit IRC | 22:05 | |
| *** aysyd has quit IRC | 22:05 | |
| Sam-I-Am | clarkb: yeah | 22:05 |
| *** tzn has joined #openstack-infra | 22:05 | |
| mordred | jeblair: ok. I'm doing a vos release on mirror.apt before doing anything else | 22:06 |
| *** daemontool has quit IRC | 22:06 | |
| *** derekh has quit IRC | 22:07 | |
| *** krtaylor has joined #openstack-infra | 22:07 | |
| *** rockyg has joined #openstack-infra | 22:07 | |
| jeblair | mordred: ok; has much changed since the last? | 22:07 |
| mordred | jeblair: don't _think_ so - but I have no idea what the volume rename actually does, so I figured I'd do one just to be sure | 22:08 |
| mordred | (I think it will wind up not have any bearing on this process) | 22:08 |
| jeblair | mordred: ok, was asking in case you wanted to run it with localauth in screen, but if not too much has changed you can probably just run locally | 22:08 |
| clarkb | Sam-I-Am: possible route problems between you and git.o.o? or other connectivity trouble? | 22:08 |
| mordred | oh - nah, I ran one earlier today already | 22:08 |
| jeblair | or, remotely as it wore :) | 22:08 |
| jeblair | cool | 22:08 |
| Sam-I-Am | clarkb: other stuff seems fine. who knows :/ | 22:09 |
| Sam-I-Am | clarkb: maybe its telling me to get away from the computer for a while | 22:09 |
| mordred | jeblair: mordred@mirror-update:~$ vos rename mirror.apt mirror.ubuntu | 22:10 |
| Sam-I-Am | which is a great idea | 22:10 |
| mordred | Renamed volume mirror.apt to mirror.ubuntu | 22:10 |
| mordred | jeblair: so that worked | 22:10 |
| mordred | the next step is fs rmmount and fs mkmount | 22:10 |
| *** jgriffith is now known as jgriffith_away | 22:10 | |
| jeblair | ++ | 22:10 |
| *** jsavak has quit IRC | 22:10 | |
| mordred | jeblair: do I do fs mkmount /afs/.openstack.org/mirror/ubuntu -rw ? | 22:11 |
| jeblair | mordred: omit -rw | 22:11 |
| *** kzaitsev_mb has quit IRC | 22:12 | |
| mordred | jeblair: k | 22:12 |
| mordred | jeblair: the readonly location does not exist now | 22:12 |
| *** jaosorior has joined #openstack-infra | 22:12 | |
| mordred | mordred@mirror-update:~$ fs mkmount /afs/.openstack.org/mirror/ubuntu mirror.ubuntu | 22:12 |
| mordred | mordred@mirror-update:~$ ls /afs/.openstack.org/mirror/ubuntu | 22:12 |
| mordred | db dists lists pool | 22:12 |
| mordred | mordred@mirror-update:~$ ls /afs/openstack.org/mirror/ubuntu | 22:13 |
| mordred | ls: cannot access /afs/openstack.org/mirror/ubuntu: No such file or directory | 22:13 |
| jeblair | mordred: i believe if you included -rw, it would specify that in the mount and we'd never hit the ro version | 22:13 |
| *** thorst has quit IRC | 22:13 | |
| jeblair | mordred: vos release mirror | 22:13 |
| jeblair | mordred: the mount itself is on a volume with a read only replica too | 22:13 |
| mordred | ah | 22:13 |
| mordred | yay | 22:13 |
| mordred | that worked | 22:13 |
| *** akshai has joined #openstack-infra | 22:13 | |
| mordred | I'm going to try running the mirror script now | 22:14 |
| *** weshay has quit IRC | 22:14 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Expose br_name in controller and compute classes https://review.openstack.org/275951 | 22:14 |
| *** thorst has joined #openstack-infra | 22:15 | |
| *** ihrachys has quit IRC | 22:16 | |
| *** hichihara has quit IRC | 22:16 | |
| *** jsavak has joined #openstack-infra | 22:16 | |
| mordred | jeblair: oh - I need to not direct k5start output to the file | 22:16 |
| *** jgriffith_away is now known as jgriffith | 22:17 | |
| clarkb | gra upload is still going almost 2 hours later :/ | 22:18 |
| *** imcsk8|afk is now known as imcsk8 | 22:18 | |
| *** jsavak has quit IRC | 22:19 | |
| greghaynes | clarkb: looks like its at about 20mbit | 22:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Suppress k5start banner when running https://review.openstack.org/277591 | 22:20 |
| mordred | jeblair: ^^ | 22:20 |
| *** thorst has quit IRC | 22:20 | |
| *** bpokorny_ has quit IRC | 22:20 | |
| *** bpokorny has joined #openstack-infra | 22:21 | |
| *** amitgandhinz has quit IRC | 22:22 | |
| *** sshnaidm has joined #openstack-infra | 22:22 | |
| *** esker has quit IRC | 22:22 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:22 | |
| *** kzaitsev_mb has quit IRC | 22:22 | |
| *** kzaitsev1mb has joined #openstack-infra | 22:22 | |
| *** esker has joined #openstack-infra | 22:23 | |
| mordred | jeblair: I'm currently doing a vos release on the mirror after having cleaned out multiverse and restricted | 22:23 |
| kzaitsev1mb | clarkb: fungi: about that murano branch delete request — I saw your email, hadn't had my hands on the client yet. Will certainly tag the branch this week and ping you on the ML about that | 22:23 |
| mordred | jeblair: but everything else seems to be working | 22:24 |
| clarkb | kzaitsev1mb: ok | 22:24 |
| *** kzaitsev1mb is now known as kzaitsev_mb | 22:24 | |
| *** krtaylor has quit IRC | 22:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Update doc publishing spec for AFS https://review.openstack.org/276482 | 22:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Update docs-publishing spec to catch up to reality https://review.openstack.org/276481 | 22:24 |
| *** jsavak has joined #openstack-infra | 22:25 | |
| jeblair | mordred: can dumpunreferenced take a filename argument? | 22:25 |
| *** esp has joined #openstack-infra | 22:25 | |
| *** NobodyCa1 has joined #openstack-infra | 22:25 | |
| jeblair | mordred: if not, maybe we can have k5start run that in a shell so we still get the k5start output | 22:25 |
| *** sshnaidm has quit IRC | 22:26 | |
| *** jsavak has quit IRC | 22:26 | |
| mordred | jeblair: -q just suppresses the banner - but let me check | 22:26 |
| mordred | I do not htin kso | 22:26 |
| jeblair | mordred: like "k5start -- bash -c 'reprepro --confdir $REPREPRO_CONFIG dumpunfereferenced $UNREF_FILE'" | 22:26 |
| *** weshay has joined #openstack-infra | 22:26 | |
| *** tcammann_ has joined #openstack-infra | 22:27 | |
| mordred | jeblair: nope | 22:27 |
| mordred | jeblair: just dumps to stdout | 22:27 |
| mordred | jeblair: we could maybe "k5start -- bash -c 'reprepro --confdir $REPREPRO_CONFIG dumpunfereferenced >$UNREF_FILE'" | 22:27 |
| jeblair | yeah, i meant > :) | 22:27 |
| mordred | heh | 22:28 |
| *** weshay has quit IRC | 22:28 | |
| jeblair | mordred: i'd like to avoid having the k5start output go into a blackhole for troubleshooting purposes | 22:28 |
| *** weshay has joined #openstack-infra | 22:28 | |
| *** weshay has quit IRC | 22:29 | |
| *** kzaitsev_mb has quit IRC | 22:29 | |
| *** annegentle has quit IRC | 22:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Suppress k5start banner when running https://review.openstack.org/277591 | 22:29 |
| mordred | jeblair: ^^ how about that? | 22:30 |
| jeblair | mordred: commit message is wrong :) | 22:30 |
| clarkb | fungi: ubuntu-trusty is uploading across the board, should work against rax this first time as tokens haven't had a chance to expire but we shall see | 22:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate: DNM - test out afs mirrors https://review.openstack.org/272215 | 22:31 |
| ianw | mrodden : i rebased https://review.openstack.org/#/q/topic:continuation if you want to take a look | 22:31 |
| *** tcammann_ has quit IRC | 22:31 | |
| *** tzn has quit IRC | 22:31 | |
| *** dims has quit IRC | 22:31 | |
| *** esp has quit IRC | 22:32 | |
| mrodden | ianw, will do | 22:32 |
| *** tzn has joined #openstack-infra | 22:33 | |
| *** weshay_xchat has joined #openstack-infra | 22:34 | |
| *** NobodyCa1 has quit IRC | 22:35 | |
| *** nelsnelson has quit IRC | 22:36 | |
| *** krtaylor has joined #openstack-infra | 22:36 | |
| *** tzn has quit IRC | 22:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Don't write k5start banner to file list https://review.openstack.org/277591 | 22:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Update configure_mirror to use per-region APT mirrors https://review.openstack.org/277114 | 22:38 |
| mordred | jeblair: ok. I think that's the stack | 22:38 |
| mordred | once the system-config patch lands, we can release the locks | 22:38 |
| mordred | then watch the d-g change to make sure it's all good | 22:38 |
| mordred | and when we're happy land the project-config patch and then rebuild images | 22:39 |
| mordred | yah? | 22:39 |
| jeblair | mordred: yep | 22:39 |
| *** thorst has joined #openstack-infra | 22:40 | |
| jeblair | fungi: here's a puzzle... | 22:41 |
| jeblair | i haven't gotten any foundation@lists.o.o mail since december... | 22:41 |
| *** doug-fish has quit IRC | 22:41 | |
| *** sshnaidm has joined #openstack-infra | 22:43 | |
| jeblair | i am apparently not subscribed | 22:43 |
| mordred | jeblair: maybe you're not really in the foundation | 22:44 |
| nibalizer | crinkle: mordred so omfracloud | 22:44 |
| *** doug-fish has joined #openstack-infra | 22:44 | |
| *** thorst has quit IRC | 22:44 | |
| nibalizer | we have the inventory in place on the bastion host (puppetmaster) | 22:45 |
| ianw | do any ruby people know why i'd be getting "/home/iwienand/.gem/ruby/gems/puppet-3.8.5/lib/puppet/vendor/safe_yaml/lib/safe_yaml/syck_node_monkeypatch.rb:42:in `<top (required)>': uninitialized constant Syck (NameError)" ? | 22:45 |
| nibalizer | and the ssh keys have been accepted | 22:45 |
| fungi | jeblair: did you maybe get bounce disabled somehow? | 22:45 |
| nibalizer | and all the infracloud groups are children of disabled | 22:45 |
| ianw | it seems i've hosed my ruby installs, and it's broken puppet, rspec testing, everything :( | 22:45 |
| nibalizer | so in theory we can just land the patches to add the node definitions to site.pp, and un-disable the groups | 22:46 |
| nibalizer | possibly running ansible-puppet manually between those two steps | 22:46 |
| *** doug-fis_ has joined #openstack-infra | 22:46 | |
| fungi | jeblair: i can't think of any changes we've made that would impact the subscription list for that | 22:46 |
| fungi | jeblair: closest i'm aware of was when we renamed the private foundation-board ml and created a new public one | 22:47 |
| fungi | but we didn't touch foundation@ | 22:47 |
| fungi | (certainly not intentionally anyway) | 22:47 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: run Puppet OpenStack integration jobs on puppet-ceph https://review.openstack.org/277598 | 22:47 |
| crinkle | nibalizer: so https://review.openstack.org/#/c/209698 now depends on https://review.openstack.org/#/c/276375/ , so i don't know if i should make an intermediate patch so 209698 can go in right away or if you want to make 276375 go | 22:48 |
| *** jpr has quit IRC | 22:48 | |
| *** doug-fish has quit IRC | 22:48 | |
| clarkb | did it auto unsubscribe you for some reason? | 22:49 |
| clarkb | this is why I left gmail behind for our lists I kept getting unsubed due to fialed deliveries | 22:49 |
| clarkb | but foundation@ is low traffic I would expect that from -dev | 22:49 |
| *** annegentle has joined #openstack-infra | 22:49 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Use SSL for rabbitmq traffic https://review.openstack.org/276375 | 22:49 |
| jeblair | clarkb: i don't see anything in the logs about me being unsubscribed... | 22:49 |
| *** doug-fis_ has quit IRC | 22:50 | |
| *** [1]Thelo has joined #openstack-infra | 22:51 | |
| *** dims_ has joined #openstack-infra | 22:51 | |
| jeblair | indeed, mailman says i am not subscribed... | 22:52 |
| fungi | i wonder if a moderator slipped up | 22:53 |
| *** ayoung has quit IRC | 22:53 | |
| jeblair | i wonder if i was subscribed as jeblair@o.o? | 22:53 |
| fungi | er, list admin | 22:53 |
| fungi | oh, perhaps | 22:53 |
| *** dimtruck is now known as zz_dimtruck | 22:53 | |
| *** annegentle has quit IRC | 22:53 | |
| *** Thelo has quit IRC | 22:53 | |
| *** [1]Thelo is now known as Thelo | 22:53 | |
| jeblair | if i ask for a pw reminder to that address, it does not show up in the mischief log, suggesting it really is subscribed | 22:53 |
| jeblair | but i have not received the password reminder | 22:54 |
| nibalizer | crinkle: my sense is that 276375 is in the 'nice to have' and not 'critical path', is that accurate? | 22:55 |
| crinkle | nibalizer: yes | 22:55 |
| *** weshay_xchat has quit IRC | 22:55 | |
| nibalizer | also that 276375 is a state that is not currently configured | 22:55 |
| crinkle | nibalizer: right | 22:55 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-diskimage_builder: Add a use_git flag https://review.openstack.org/273832 | 22:56 |
| nibalizer | okay so my big goal is that if we can get the infracloud hosts under CD from ansible-puppet then we can use our more traditional workflow to roll changes like 276375 out | 22:56 |
| nibalizer | so hopefully if we put it off for now, it will be straightforward to do later, if that makes any sense | 22:56 |
| crinkle | nibalizer: okay i'm gonna split up 209698 so it can be merged whenever | 22:56 |
| *** annegentle has joined #openstack-infra | 22:57 | |
| nibalizer | ok sweet | 22:57 |
| fungi | jeblair: that's got me wondering if whoever manages the openstack.org mailboxes has disabled non-staff addresses recently | 22:58 |
| *** cloudtrainme has quit IRC | 22:58 | |
| fungi | i guess jbryce is managing those? | 22:58 |
| *** dprince has quit IRC | 22:59 | |
| *** doug-fish has joined #openstack-infra | 22:59 | |
| *** cloudtrainme has joined #openstack-infra | 23:00 | |
| jeblair | wow | 23:00 |
| fungi | no clue, just trying to think of plausible explanations | 23:00 |
| jeblair | jeblair@openstack.org | 23:00 |
| jeblair | SMTP error from remote mail server after RCPT TO:<jeblair@openstack.org>: | 23:00 |
| jeblair | host mx1.emailsrvr.com [173.203.2.36]: 550 5.7.1 <jeblair@openstack.org>: | 23:00 |
| jeblair | Relay access denied. | 23:00 |
| jeblair | wow | 23:00 |
| jeblair | was anyone going to tell me? | 23:01 |
| jeblair | i mean, i _regularly_ got mail at that address | 23:01 |
| jeblair | on account of the fact that i'm still part of the community | 23:01 |
| *** tcammann_ has joined #openstack-infra | 23:01 | |
| jeblair | fungi: you event sent a registration code to that address :) | 23:02 |
| jeblair | our voip.ms account is tied to that address | 23:03 |
| jeblair | (because they required identity verification to set it up) | 23:03 |
| *** esp has joined #openstack-infra | 23:03 | |
| slogan621 | Any other comments on https://review.openstack.org/#/c/275873/4 ?? Best I can tell, it looks good and I gave it a +1. Again, the understanding is we can modify this at a later time to add pypi jobs (which are not important right now and there is no pypi account yet created) | 23:03 |
| jeblair | rackspace notifications went to that address | 23:03 |
| *** doug-fish has quit IRC | 23:03 | |
| *** eharney has quit IRC | 23:04 | |
| *** doug-fish has joined #openstack-infra | 23:04 | |
| jeblair | which explains why i didn't see anything during the trouble we had recently | 23:04 |
| *** jsavak has joined #openstack-infra | 23:04 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:04 | |
| *** mtanino has joined #openstack-infra | 23:06 | |
| nibalizer | infra-root taking the hiera lock | 23:06 |
| *** jaosorior has quit IRC | 23:06 | |
| *** tcammann_ has quit IRC | 23:07 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition https://review.openstack.org/213980 | 23:07 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node https://review.openstack.org/209698 | 23:07 |
| crinkle | nibalizer: ^ also stacked them so they don't conflict | 23:07 |
| *** dingyichen has joined #openstack-infra | 23:07 | |
| *** daemontool_ has quit IRC | 23:07 | |
| *** esp has quit IRC | 23:07 | |
| nibalizer | infra-root lock released | 23:08 |
| nibalizer | crinkle: thanks | 23:08 |
| *** ayoung has joined #openstack-infra | 23:09 | |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/277295 | 23:09 |
| nibalizer | crinkle: do you want to checkout the tip of that locally on the controller and make sure it applies correctly? | 23:09 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Use SSL for rabbitmq https://review.openstack.org/277605 | 23:10 |
| *** annegentle has quit IRC | 23:10 | |
| crinkle | nibalizer: okay | 23:10 |
| *** jsavak has quit IRC | 23:10 | |
| *** annegentle has joined #openstack-infra | 23:11 | |
| fungi | jeblair: yeah, i would have sent a registration code to every address you have in gerrit | 23:12 |
| *** vivekd has quit IRC | 23:12 | |
| *** doug-fish has quit IRC | 23:14 | |
| *** doug-fish has joined #openstack-infra | 23:14 | |
| *** dims_ has quit IRC | 23:14 | |
| *** salv-orlando has quit IRC | 23:14 | |
| *** doug-fish has quit IRC | 23:15 | |
| *** doug-fish has joined #openstack-infra | 23:15 | |
| *** salv-orlando has joined #openstack-infra | 23:15 | |
| jeblair | fungi: i'm having trouble communicating via irc how distressing i find the idea that the foundation would do something like that without any prior notice, or any of its technical staff knowing about it. | 23:16 |
| *** annegent_ has joined #openstack-infra | 23:16 | |
| fungi | jeblair: yep, i've just tried to raise some people about this as well to get clarification | 23:16 |
| fungi | as far as i know i don't have access to the interface which manages those addresses to check the status of them | 23:17 |
| fungi | that i can find anyway | 23:18 |
| jeblair | fungi: i do. | 23:18 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition https://review.openstack.org/213980 | 23:18 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node https://review.openstack.org/209698 | 23:18 |
| jeblair | fungi: i do not have an account. | 23:18 |
| jeblair | fungi: nor do i have an alias. | 23:18 |
| clarkb | https://jenkins07.openstack.org/job/gate-requirements-tox-validate/636/console may also be similra to the thing Sam-I-Am mentioned earlier | 23:18 |
| clarkb | git remote update took so long the job timed out | 23:18 |
| nibalizer | crinkle: is that the diff between ssl-rabbit and not-ssl-rabbit? | 23:18 |
| jeblair | fungi: (of course, i only use it to set up infra-root aliases for us) | 23:19 |
| jeblair | fungi: i don't feel like i have permission to create an alias for myself, so i will not do that. | 23:19 |
| nibalizer | i'll have you know, i read some of my rootermail | 23:19 |
| *** annegentle has quit IRC | 23:19 | |
| crinkle | nibalizer: which that? | 23:19 |
| nibalizer | the port change? | 23:19 |
| crinkle | nibalizer: yes | 23:19 |
| nibalizer | cool | 23:19 |
| nibalizer | im glad you're on that because it greek to me | 23:19 |
| *** dims_ has joined #openstack-infra | 23:19 | |
| nibalizer | would you say we are 'ready' ? | 23:19 |
| fungi | stepping out to grab a bite to eat, but will be back shortly | 23:20 |
| nibalizer | clarkb: jeblair fungi could you review 209698 and 213980 please | 23:20 |
| *** ajmiller_ has quit IRC | 23:20 | |
| crinkle | nibalizer: i'm analyizng --noop output | 23:21 |
| *** annegent_ has quit IRC | 23:21 | |
| nibalizer | kk | 23:21 |
| *** bpokorny has quit IRC | 23:21 | |
| crinkle | nibalizer: http://paste.openstack.org/show/486342/ i think this is good | 23:22 |
| clarkb | nibalizer: crinkle any reason to not rm https://review.openstack.org/#/c/209698/64/modules/openstack_project/manifests/infracloud/controller.pp and put that class directly in site.pp? | 23:22 |
| *** bpokorny has joined #openstack-infra | 23:22 | |
| nibalizer | clarkb: i need that for a follow on patch | 23:23 |
| nibalizer | which I should be rebasing | 23:23 |
| *** nelsnelson has joined #openstack-infra | 23:23 | |
| crinkle | clarkb: it started out that way, putting it in openstack_project adds more room to do things like realize admin users and other site specific things | 23:23 |
| clarkb | nibalizer: you need the distinct class and .pp file? | 23:23 |
| clarkb | crinkle: gotcha | 23:23 |
| nibalizer | to add crinkle greghaynes and rcarrillocruz as infra-cloud rooters, which may not land | 23:23 |
| nibalizer | but these people are allready rooting on these servers with the 'baremetal' user so that would just be a formaility | 23:23 |
| *** doug-fish has quit IRC | 23:23 | |
| clarkb | I think you need to edit the groups.txt file too | 23:24 |
| clarkb | but otherwise that lgtm | 23:24 |
| clarkb | mordred: ^ you clarified that for jeblair recently iirc | 23:24 |
| *** gordc has quit IRC | 23:24 | |
| nibalizer | clarkb: no the groups.txt only serves to groupify otherwise unstructured openstack hosts | 23:24 |
| clarkb | nibalizer: jeblair had to do it for afsadmin hosts | 23:24 |
| nibalizer | yes those are in openstack | 23:24 |
| clarkb | and since you have it set to group infracloud I believe you need to double account in groups.txt | 23:24 |
| nibalizer | i have this | 23:25 |
| nibalizer | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/puppetmaster/infracloud | 23:25 |
| nibalizer | i think i need a patch there though | 23:25 |
| clarkb | right that doesn't specify infracloud anywhere | 23:25 |
| nibalizer | since I got fancy | 23:25 |
| nibalizer | yes we/me let the groups not line up | 23:25 |
| nibalizer | sec I will patch | 23:25 |
| mgagne | looks like if dib build fails in nodepool, you can't delete it, CLI jams :-/ | 23:26 |
| *** doug-fish has joined #openstack-infra | 23:26 | |
| clarkb | mgagne: you shouldn't need to delete them | 23:26 |
| clarkb | if build fails builder cleans up after itself and the record in the db should be removed | 23:27 |
| *** doug-fish has quit IRC | 23:27 | |
| mgagne | clarkb mine got stucked for 6 days and nodepool didn't build a new one since | 23:27 |
| clarkb | however there were some bugs around that so you may need very recent nodepool for that to work properly | 23:27 |
| clarkb | mgagne: yup that bug was fixed a few hours ago | 23:27 |
| mgagne | merged? | 23:27 |
| clarkb | yes | 23:27 |
| mgagne | will update now and test | 23:27 |
| *** zz_dimtruck is now known as dimtruck | 23:27 | |
| clarkb | the issue was failures were not accounted by the code that checked if all the jobs were done running | 23:27 |
| mgagne | you're the man | 23:27 |
| clarkb | so the cron never fired again | 23:27 |
| *** doug-fish has joined #openstack-infra | 23:28 | |
| *** dimtruck is now known as zz_dimtruck | 23:28 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Collapse groups into one group https://review.openstack.org/277609 | 23:28 |
| nibalizer | crinkle: clarkb sorry about that | 23:28 |
| *** ajmiller has joined #openstack-infra | 23:29 | |
| nibalizer | with that patch and crinkles patches: the group in ansible is 'infracloud' the file in hiera is 'infracloud.yaml' and variable in puppet is '$group=infracloud' | 23:29 |
| mgagne | can't wait to migrate our first project to gerrit infra =) | 23:30 |
| nibalizer | mgagne: *thumbsup* | 23:30 |
| clarkb | mgagne: you may need to manually delete from dib_image where state = 1; in the db to correct that broken build | 23:30 |
| mgagne | yea, I learned about that trick =) | 23:31 |
| crinkle | nibalizer: i ran without --noop on the controller, seems to have survived | 23:31 |
| nibalizer | crinkle: awesome | 23:32 |
| clarkb | but once these bugs are worked out a failed build like that should clean itself up then the next day get retried | 23:32 |
| mgagne | restarting nodepool auto deleted the failed build | 23:32 |
| mgagne | and I'm able to delete old images too | 23:32 |
| clarkb | nibalizer: crinkle should https://review.openstack.org/#/c/209698/64 depend on 277609? | 23:33 |
| clarkb | nibalizer: I am happy to +2 both if you want to make sure they go in the right order without another patchset | 23:33 |
| crinkle | clarkb: i can add depends-on | 23:34 |
| *** tzn has joined #openstack-infra | 23:34 | |
| nibalizer | it actually doesn't really matter since once we land all this no automated processes kick off | 23:34 |
| nibalizer | so my thinking was to get it landed | 23:34 |
| nibalizer | then do ansible runs manually from the puppetmaster and see what explodes | 23:34 |
| clarkb | nibalizer: doesn't that group file make automated process work? | 23:34 |
| clarkb | or do we hae to pass an epxlicit -i inventory? | 23:34 |
| nibalizer | no beacuse infracloud is under [disabled:children] | 23:34 |
| nibalizer | so it gets caught by the !disabled guards on all our actual do_it scripts | 23:35 |
| nibalizer | which is actuall run_all.sh | 23:35 |
| *** piet has quit IRC | 23:35 | |
| clarkb | and thats true even with the mismatched names because that happens before any hieraing happens | 23:35 |
| clarkb | nibalizer: crinkle I will approve 209698 then | 23:36 |
| nibalizer | ya | 23:36 |
| nibalizer | clarkb: ty | 23:36 |
| rm_work | err, anyone know what happened here? https://review.openstack.org/#/c/276549/ looks like a timeout?? | 23:36 |
| rm_work | is it possible to just retry a gate check? or need to -A and +A? | 23:37 |
| clarkb | rm_work: yes I linked htat earlier, git remote update hung and the job timed out | 23:37 |
| rm_work | :( | 23:37 |
| clarkb | rm_work: you can just recheck it | 23:37 |
| rm_work | k | 23:37 |
| *** ccrouch has quit IRC | 23:38 | |
| *** doug-fish has quit IRC | 23:38 | |
| *** piet has joined #openstack-infra | 23:38 | |
| clarkb | nibalizer: crinkle thta one and its child for the compute hosts has been approved | 23:38 |
| clarkb | and +2 on the group rename change | 23:39 |
| nibalizer | awesome thanks | 23:39 |
| nibalizer | so we need one more on 277609 | 23:39 |
| nibalizer | maybe when fungi gets back | 23:39 |
| nibalizer | or jeblair maybe ^? | 23:39 |
| *** ccrouch has joined #openstack-infra | 23:39 | |
| nibalizer | dims_: seems to have gotten 'e' https://review.openstack.org/#/c/271828 | 23:40 |
| nibalizer | oh no | 23:40 |
| nibalizer | dims_: is a reviewer, openstack proposal bot strikes again | 23:40 |
| *** tzn has quit IRC | 23:40 | |
| *** doug-fish has joined #openstack-infra | 23:40 | |
| *** esker has quit IRC | 23:43 | |
| *** piet has quit IRC | 23:43 | |
| *** denisra has quit IRC | 23:44 | |
| dims_ | nibalizer what's up? | 23:44 |
| nibalizer | dims_: not much. I just mis-read a thing. Sorry to ping you | 23:44 |
| nibalizer | that review is the numebr most closely matching the constant 'e' | 23:44 |
| *** denisra has joined #openstack-infra | 23:44 | |
| dims_ | no worries is the eventlet bump breaking something? | 23:45 |
| nibalizer | no, this is entirely social chatter :) | 23:46 |
| *** doug-fish has quit IRC | 23:46 | |
| nibalizer | or casual, or whatever | 23:46 |
| *** ybathia has joined #openstack-infra | 23:46 | |
| dims_ | nibalizer :) | 23:47 |
| *** dkranz has quit IRC | 23:47 | |
| openstackgerrit | Merged openstack/requirements: Add manila-image-elements to project list https://review.openstack.org/274135 | 23:49 |
| *** notnownikki has quit IRC | 23:49 | |
| openstackgerrit | Merged openstack/requirements: Bump ryu to 3.30 https://review.openstack.org/264026 | 23:49 |
| openstackgerrit | Merged openstack/requirements: Add sahara-scenario to projects list https://review.openstack.org/270676 | 23:49 |
| *** doug-fish has joined #openstack-infra | 23:50 | |
| *** doug-fish has quit IRC | 23:50 | |
| *** doug-fish has joined #openstack-infra | 23:50 | |
| *** cloudtrainme has quit IRC | 23:51 | |
| openstackgerrit | Merged openstack/requirements: Add kuryr to projects.txt https://review.openstack.org/274446 | 23:53 |
| openstackgerrit | Merged openstack/requirements: Add neutron-lbaas-dashboard to projects.txt https://review.openstack.org/275291 | 23:53 |
| *** doug-fish has quit IRC | 23:54 | |
| *** bpokorny_ has joined #openstack-infra | 23:55 | |
| openstackgerrit | Merged openstack/requirements: For API reference doc builds, fairy-slipper 0.1.0 https://review.openstack.org/270915 | 23:56 |
| *** pahuang_ has quit IRC | 23:56 | |
| *** ZZelle_ has quit IRC | 23:57 | |
| *** bpokorny has quit IRC | 23:58 | |
| *** akshai has quit IRC | 23:58 | |
| dims_ | anteaya : got +1's from both armax and dougwig in https://review.openstack.org/#/c/277086/ | 23:58 |
| dims_ | anteaya : so can you please bless that when you get a chance | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!