| *** saschpe has quit IRC | 00:02 | |
| *** wchrisj has joined #openstack-infra | 00:05 | |
| *** CaptTofu has joined #openstack-infra | 00:05 | |
| *** wchrisj has quit IRC | 00:09 | |
| *** UtahDave1 has joined #openstack-infra | 00:09 | |
| *** saschpe has joined #openstack-infra | 00:10 | |
| *** UtahDave has quit IRC | 00:11 | |
| *** UtahDave1 is now known as UtahDave | 00:11 | |
| zaro | clarkb: gerrit fails when building on HP VMs. i'm wordering if it might be difference in java version? | 00:12 |
|---|---|---|
| zaro | java version "1.7.0_51" | 00:12 |
| zaro | OpenJDK Runtime Environment (IcedTea 2.4.4) (7u51-2.4.4-0ubuntu0.12.04.2) | 00:12 |
| zaro | OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode) | 00:12 |
| *** melwitt has quit IRC | 00:12 | |
| zaro | clarkb: anyways you can check on the rax VMs? | 00:12 |
| clarkb | not right now I am not on laptop anymore | 00:13 |
| zaro | wha, huh? you must be joking :) | 00:14 |
| clarkb | they should all be up to date precise though | 00:16 |
| jeblair | zaro: i'll check | 00:18 |
| zaro | jeblair: ohh, cool. my VM is.. | 00:18 |
| jeblair | zaro: which rax region specifically did it fail in? dfw, ord or iad? | 00:18 |
| jeblair | zaro: the info you supplied exactly matches the output on a bare-precise node in rax-dfw | 00:20 |
| zaro | sorry had some snaffu with my keyboard. | 00:20 |
| jeblair | oh, sorry i got the success/fail backwards (i see you said it fails on hp, so rax must have succeeded) but that doesn't change the results | 00:21 |
| zaro | yes, that is correct. tests pass on rax. | 00:21 |
| *** sweston has quit IRC | 00:21 | |
| *** asselin2 has joined #openstack-infra | 00:21 | |
| zaro | hmm. do you have any ideas? | 00:21 |
| jeblair | zaro: nope. those 4 changes are identical to the ones we already had, right? | 00:23 |
| zaro | yes. | 00:23 |
| jeblair | zaro: i say we bash them in and figure it out later | 00:24 |
| zaro | jeblair: ohh wait, looks like it's passing on my 2nd attempt at running. | 00:24 |
| zaro | jeblair: yep, must be some timing thing. but does seem pretty reproducable on hp VMs only as 3 out of the 4 patches failed on review.o.o | 00:25 |
| jeblair | zaro: i've approved them all; feel free to recheck as needed to get them in | 00:25 |
| *** _nadya_ has quit IRC | 00:25 | |
| zaro | jeblair: ok. i'll try rerunning a few more times to see if something obvious rears it's head. | 00:27 |
| *** sacharya has quit IRC | 00:27 | |
| tchaypo | dstufft: do you remember my pip patch to be case-insensitive when using a file:/// index-url? | 00:29 |
| *** matsuhashi has joined #openstack-infra | 00:29 | |
| tchaypo | dstufft: I've been looking at writing a test for my patch... and I've realised that there's a test that seems like it should exercise the right code and demonstrate the bug - except that the test uses "--no-index -f file:///blah" rather than "--index-url file:///blah" | 00:30 |
| dstufft | tchaypo: --find-links and --index-url are two slightly different code paths | 00:31 |
| dstufft | though if I recall | 00:31 |
| dstufft | it shouldn't matter for what you're doing | 00:31 |
| tchaypo | so if I start writing a test that exercise "--index-url file:///" instead of "--no-index -f file:///", that's likely to be useful and not just be met with "But you should just use --no-index -f file:///"? | 00:33 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update gerrit core branch https://review.openstack.org/88038 | 00:34 |
| dstufft | tchaypo: yea | 00:35 |
| *** derekh has quit IRC | 00:37 | |
| tchaypo | cool, I'll try to get that done this morning then. | 00:39 |
| dstufft | tchaypo: if I don't notice things just keep poking me, I had like 30 threads going at any one time and lately i've been having a hard time keeping track of them all | 00:40 |
| *** maxbit has quit IRC | 00:40 | |
| *** zns has quit IRC | 00:43 | |
| *** derekh has joined #openstack-infra | 00:43 | |
| tchaypo | dstufft: fwiw, I was busy on other things last week, then i had a 4-day weekend - I haven't been waiting for you :) | 00:46 |
| anteaya | tchaypo: did you vote? | 00:49 |
| *** maxbit has joined #openstack-infra | 00:49 | |
| anteaya | some people who were away for 4 days might not have voted yet | 00:49 |
| anteaya | vote | 00:49 |
| tchaypo | anteaya: ptl yes, tc no | 00:49 |
| StevenK | I took ~ 30 minutes yesterday and did my tc vote | 00:49 |
| anteaya | find thou thee tc ballot and be sure to vote before friday at 1300 utc | 00:49 |
| StevenK | Re-read all platforms, etc | 00:49 |
| anteaya | StevenK: woohoo | 00:49 |
| anteaya | well done | 00:50 |
| anteaya | it is a time commitment | 00:50 |
| tchaypo | I don't know I feel like I don't know enough of the people to make a useful vote | 00:50 |
| anteaya | tchaypo: they have candidate platforms | 00:50 |
| StevenK | tchaypo: So do what I did, and read the platforms | 00:50 |
| tchaypo | but maybe reading the platforms would help fill the void.. | 00:50 |
| anteaya | https://wiki.openstack.org/wiki/TC_Elections_April_2014#Candidates | 00:51 |
| anteaya | tchaypo: and all candidates are available for questions | 00:51 |
| anteaya | so find them on the ml or irc and ask them about their platform | 00:51 |
| anteaya | and please vote | 00:52 |
| * tchaypo opens a tab | 00:53 | |
| clarkb | I did my best to read platforms as people declared their candidacy | 00:53 |
| clarkb | spread the work out | 00:53 |
| * anteaya nods | 00:54 | |
| anteaya | that helps | 00:54 |
| *** alexpilotti has quit IRC | 00:54 | |
| dstufft | tchaypo: that's good :) Just wanted to let you know that if you get blocked on me just keep botering me | 00:55 |
| *** mfer has joined #openstack-infra | 00:58 | |
| *** matsuhashi has quit IRC | 00:59 | |
| *** matsuhashi has joined #openstack-infra | 01:01 | |
| tchaypo | dstufft: one question - I'm going to be writing some new tests that fail without my commit but pass with it. Should I do anything to show the output before my patch? | 01:01 |
| dstufft | tchaypo: nah | 01:01 |
| dstufft | we mostly trust that the tests fail without the patch | 01:01 |
| dstufft | unless it looks wonky | 01:01 |
| dstufft | then maybe we'll pull it down and test it | 01:01 |
| tchaypo | cool | 01:02 |
| *** lakshmi has quit IRC | 01:04 | |
| *** wchrisj has joined #openstack-infra | 01:06 | |
| *** mburned_out is now known as mburned | 01:07 | |
| *** SumitNaiksatam has quit IRC | 01:10 | |
| *** wchrisj has quit IRC | 01:10 | |
| *** Ryan_Lane has quit IRC | 01:11 | |
| *** derekh has quit IRC | 01:14 | |
| *** mestery has quit IRC | 01:15 | |
| *** unicell has joined #openstack-infra | 01:16 | |
| anteaya | zaro: my patch merged:https://gerrit-review.googlesource.com/#/c/56290/ | 01:24 |
| anteaya | wow | 01:24 |
| *** mburned is now known as mburned_out | 01:25 | |
| anteaya | apparently I submitted it to stable 2.8 and the patch was abandoned and then applied to stable 2.9 | 01:25 |
| *** msabramo has joined #openstack-infra | 01:25 | |
| anteaya | so I guess I can add Gerrit upstream committer to my bio now | 01:26 |
| *** mestery has joined #openstack-infra | 01:26 | |
| *** marcoemorais has quit IRC | 01:29 | |
| *** saschpe has quit IRC | 01:38 | |
| morganfainberg | anteaya, gratz! | 01:39 |
| *** yjiang has joined #openstack-infra | 01:41 | |
| *** nosnos has joined #openstack-infra | 01:41 | |
| *** zehicle has joined #openstack-infra | 01:43 | |
| *** saschpe has joined #openstack-infra | 01:43 | |
| *** wchrisj has joined #openstack-infra | 01:43 | |
| *** gokrokve_ has quit IRC | 01:45 | |
| *** zehicle_at_dell has quit IRC | 01:46 | |
| *** gokrokve has joined #openstack-infra | 01:47 | |
| *** praneshp has quit IRC | 01:47 | |
| *** wchrisj has quit IRC | 01:48 | |
| *** gokrokve has quit IRC | 01:51 | |
| *** yaguang has joined #openstack-infra | 01:54 | |
| *** wchrisj has joined #openstack-infra | 01:54 | |
| *** wchrisj has quit IRC | 01:59 | |
| *** yjiang|2 has joined #openstack-infra | 02:03 | |
| *** yjiang has quit IRC | 02:03 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:04 | |
| *** adalbas has quit IRC | 02:04 | |
| *** zns has joined #openstack-infra | 02:07 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Create groups-static-pages project https://review.openstack.org/89734 | 02:07 |
| SergeyLukjanov | morning folks | 02:08 |
| clarkb | morning | 02:10 |
| clarkb | I am pretty sure SergeyLukjanov is another North Carolinian robot that doesn't need sleep | 02:11 |
| SergeyLukjanov | clarkb, nope, I need some | 02:12 |
| clarkb | a lot less than I need :) | 02:12 |
| SergeyLukjanov | clarkb, just need to finish something before the trip :) | 02:13 |
| *** jpeeler has quit IRC | 02:13 | |
| SergeyLukjanov | clarkb, I'm feeling like /me will have afternoon sleeping time today :) | 02:13 |
| clarkb | ah yes the fabled nap | 02:13 |
| SergeyLukjanov | yeah | 02:14 |
| *** gokrokve has joined #openstack-infra | 02:16 | |
| *** gokrokve has quit IRC | 02:21 | |
| *** marun has quit IRC | 02:26 | |
| *** mfer has quit IRC | 02:32 | |
| *** coolsvap|afk is now known as coolsvap | 02:33 | |
| *** zz_gondoi is now known as gondoi | 02:34 | |
| *** asalkeld_ has joined #openstack-infra | 02:39 | |
| *** jhesketh has quit IRC | 02:39 | |
| asalkeld_ | can anyone have a moment to look at this? https://review.openstack.org/#/c/87448/ | 02:40 |
| *** jhesketh has joined #openstack-infra | 02:40 | |
| *** unicell has quit IRC | 02:40 | |
| asalkeld_ | s/can/does | 02:40 |
| *** miqui has joined #openstack-infra | 02:40 | |
| *** sweston has joined #openstack-infra | 02:41 | |
| *** miqui has quit IRC | 02:41 | |
| *** harlowja is now known as harlowja_away | 02:43 | |
| *** mfer has joined #openstack-infra | 02:47 | |
| *** gyee has quit IRC | 02:47 | |
| *** mfer has quit IRC | 02:48 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add solum-dashboard (horizon plugin for Solum) https://review.openstack.org/87448 | 02:52 |
| *** mfer has joined #openstack-infra | 02:53 | |
| *** mfer has quit IRC | 02:53 | |
| *** praneshp has joined #openstack-infra | 02:55 | |
| *** wchrisj has joined #openstack-infra | 02:55 | |
| *** wchrisj has quit IRC | 02:57 | |
| *** wchrisj_ has joined #openstack-infra | 02:57 | |
| *** krtaylor has joined #openstack-infra | 02:58 | |
| *** praneshp_ has joined #openstack-infra | 03:00 | |
| *** wchrisj_ has quit IRC | 03:01 | |
| *** praneshp has quit IRC | 03:01 | |
| *** praneshp_ is now known as praneshp | 03:01 | |
| *** cody-somerville has quit IRC | 03:02 | |
| clarkb | jhesketh: I just approved https://review.openstack.org/#/c/71189/ but I did leave some comments inline for things that may need improving | 03:04 |
| clarkb | jhesketh: but at this point I think it is more important to actually start doing the thing than nitpick over little details | 03:04 |
| *** openstackgerrit has quit IRC | 03:04 | |
| clarkb | jhesketh: curious about what you think of those comments though | 03:04 |
| jhesketh | clarkb: thanks | 03:04 |
| *** openstackgerrit has joined #openstack-infra | 03:05 | |
| jhesketh | I'll take a look | 03:05 |
| *** yjiang|2 has quit IRC | 03:07 | |
| *** rfolco has quit IRC | 03:08 | |
| jhesketh | clarkb: responded | 03:08 |
| openstackgerrit | A change was merged to openstack-infra/config: Add a slave-script to Jenkins for pushing to swift https://review.openstack.org/71189 | 03:10 |
| clarkb | jhesketh: doesn't file_list = file_list[:payload['max_file_count']] take the first half of the list? | 03:11 |
| jhesketh | ah, yes :-( | 03:11 |
| jhesketh | okay, well, ignore those comments then :-p | 03:12 |
| clarkb | in any case I think that is a small problem and it is more important to start using swift | 03:12 |
| jhesketh | sure | 03:12 |
| clarkb | we can sort out these little things as we use it | 03:12 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add puppet-storyboard as a unique project https://review.openstack.org/87816 | 03:12 |
| *** mriedem has quit IRC | 03:14 | |
| *** matsuhashi has quit IRC | 03:14 | |
| *** matsuhashi has joined #openstack-infra | 03:15 | |
| *** cody-somerville has joined #openstack-infra | 03:18 | |
| *** aysyd has quit IRC | 03:19 | |
| *** gokrokve has joined #openstack-infra | 03:19 | |
| *** david-lyle has joined #openstack-infra | 03:19 | |
| *** Longgeek has joined #openstack-infra | 03:19 | |
| *** matsuhashi has quit IRC | 03:19 | |
| nibalizer | /win 90 | 03:21 |
| *** dkliban is now known as dkliban_afk | 03:22 | |
| *** gokrokve has quit IRC | 03:24 | |
| *** nosnos has quit IRC | 03:29 | |
| *** sandywalsh has quit IRC | 03:32 | |
| *** changbl has joined #openstack-infra | 03:32 | |
| *** sandywalsh has joined #openstack-infra | 03:34 | |
| *** timrc is now known as timrc-afk | 03:36 | |
| Longgeek | - -~ | 03:39 |
| Longgeek | https://bugs.launchpad.net/openstack-ci/+bug/1306967 | 03:39 |
| uvirtbot | Launchpad bug 1306967 in openstack-ci "thstack-ci service account" [Undecided,New] | 03:39 |
| Longgeek | For ninth days, without any feedback。 | 03:40 |
| clarkb | Longgeek: we have been pretty swamped with heartbleed and gerrit upgrades | 03:41 |
| Longgeek | Well, so can I know the schedule, what's time "thstack-ci" can be integrated in third party testing? | 03:46 |
| clarkb | we don't have one | 03:46 |
| clarkb | I can certainly try to do it tomorrow if more important things don't come up | 03:46 |
| clarkb | but I am still trying to dig out of the review hole | 03:47 |
| *** zhiyan_ is now known as zhiyan | 03:48 | |
| Longgeek | Thanks so much, if any good news you have, let me know quickly, please. thanks agian. | 03:49 |
| *** unicell has joined #openstack-infra | 03:49 | |
| *** rwsu has quit IRC | 03:50 | |
| *** oomichi has joined #openstack-infra | 03:50 | |
| *** marcoemorais has joined #openstack-infra | 03:52 | |
| *** etoews has joined #openstack-infra | 03:54 | |
| *** david-lyle has quit IRC | 03:54 | |
| *** wenlock has joined #openstack-infra | 03:56 | |
| *** marcoemorais1 has joined #openstack-infra | 03:56 | |
| nibalizer | hi, what am i doing wrong here: https://review.openstack.org/#/c/87816/ | 03:56 |
| *** marcoemorais has quit IRC | 03:57 | |
| clarkb | nibalizer: looks like it wored | 03:57 |
| clarkb | nibalizer: was there a particular thing that is wrong? | 03:58 |
| nibalizer | oh maybe i wasn't patient enough | 03:58 |
| nibalizer | oh haha! | 03:58 |
| nibalizer | ya i thought jenkins was ignoring me | 03:58 |
| nibalizer | woot clarkb++ | 03:58 |
| *** Longgeek has quit IRC | 04:05 | |
| *** Longgeek has joined #openstack-infra | 04:05 | |
| *** ilives has joined #openstack-infra | 04:10 | |
| *** gondoi is now known as zz_gondoi | 04:15 | |
| *** matsuhashi has joined #openstack-infra | 04:19 | |
| *** gokrokve has joined #openstack-infra | 04:20 | |
| *** nosnos has joined #openstack-infra | 04:22 | |
| *** miqui has joined #openstack-infra | 04:23 | |
| *** miqui has quit IRC | 04:23 | |
| *** gokrokve has quit IRC | 04:24 | |
| *** UtahDave has quit IRC | 04:41 | |
| *** yfried has quit IRC | 04:41 | |
| *** coolsvap is now known as coolsvap|afk | 04:45 | |
| *** vhoward has joined #openstack-infra | 04:46 | |
| *** andreaf has joined #openstack-infra | 04:48 | |
| *** andreaf_ has joined #openstack-infra | 04:49 | |
| *** andreaf has quit IRC | 04:53 | |
| *** coolsvap|afk is now known as coolsvap | 04:57 | |
| *** ildikov has quit IRC | 05:01 | |
| *** miqui has joined #openstack-infra | 05:05 | |
| *** bookwar has quit IRC | 05:05 | |
| *** pfallenop has quit IRC | 05:05 | |
| *** pfallenop has joined #openstack-infra | 05:06 | |
| *** asselin2 has quit IRC | 05:06 | |
| *** bookwar has joined #openstack-infra | 05:07 | |
| *** etoews has quit IRC | 05:08 | |
| *** etoews has joined #openstack-infra | 05:10 | |
| *** shakamunyi has quit IRC | 05:14 | |
| *** etoews has quit IRC | 05:15 | |
| *** ilyashakhat has quit IRC | 05:15 | |
| *** ilyashakhat has joined #openstack-infra | 05:16 | |
| *** nati_ueno has quit IRC | 05:16 | |
| *** markmcclain has joined #openstack-infra | 05:20 | |
| *** gokrokve has joined #openstack-infra | 05:20 | |
| *** gokrokve_ has joined #openstack-infra | 05:22 | |
| *** ildikov has joined #openstack-infra | 05:23 | |
| *** gokrokve has quit IRC | 05:25 | |
| *** yjiang has joined #openstack-infra | 05:25 | |
| *** gokrokve_ has quit IRC | 05:27 | |
| *** coolsvap is now known as coolsvap|afk | 05:28 | |
| *** markwash has quit IRC | 05:29 | |
| *** markmcclain has quit IRC | 05:30 | |
| *** markwash has joined #openstack-infra | 05:33 | |
| *** asalkeld_ has quit IRC | 05:34 | |
| *** coolsvap|afk is now known as coolsvap | 05:36 | |
| ryanpetrello | anybody active in here this time of night/morning? | 05:37 |
| *** matsuhashi has quit IRC | 05:37 | |
| openstackgerrit | Mathew Odden proposed a change to openstack-infra/config: Create bash8 project in openstack-dev https://review.openstack.org/88747 | 05:37 |
| *** matsuhashi has joined #openstack-infra | 05:37 | |
| clarkb | ryanpetrello: I think it is morning for SergeyLukjanov | 05:38 |
| ryanpetrello | clarkb: think I actually figured it out | 05:38 |
| *** ominakov has joined #openstack-infra | 05:42 | |
| *** matsuhashi has quit IRC | 05:43 | |
| *** matsuhashi has joined #openstack-infra | 05:44 | |
| *** asalkeld_ has joined #openstack-infra | 05:48 | |
| *** yfried has joined #openstack-infra | 05:48 | |
| *** matsuhas_ has joined #openstack-infra | 05:50 | |
| *** matsuhashi has quit IRC | 05:53 | |
| *** chandan_kumar has joined #openstack-infra | 05:54 | |
| *** matsuhas_ has quit IRC | 05:58 | |
| *** matsuhashi has joined #openstack-infra | 05:58 | |
| *** yonglihe_ has quit IRC | 05:58 | |
| *** rfolco has joined #openstack-infra | 06:02 | |
| *** asalkeld_ has quit IRC | 06:06 | |
| *** ilives has quit IRC | 06:07 | |
| *** ilives has joined #openstack-infra | 06:07 | |
| *** _nadya_ has joined #openstack-infra | 06:09 | |
| *** zns has quit IRC | 06:14 | |
| *** talluri has joined #openstack-infra | 06:18 | |
| *** nati_ueno has joined #openstack-infra | 06:19 | |
| *** jlibosva has joined #openstack-infra | 06:21 | |
| *** gokrokve has joined #openstack-infra | 06:23 | |
| *** talluri has quit IRC | 06:25 | |
| *** _nadya_ has quit IRC | 06:26 | |
| *** gokrokve has quit IRC | 06:27 | |
| *** Ryan_Lane has joined #openstack-infra | 06:28 | |
| *** Longgeek has quit IRC | 06:30 | |
| *** _nadya_ has joined #openstack-infra | 06:31 | |
| *** Longgeek has joined #openstack-infra | 06:31 | |
| *** Longgeek has quit IRC | 06:33 | |
| *** Longgeek has joined #openstack-infra | 06:34 | |
| *** pcrews has quit IRC | 06:41 | |
| *** gokrokve has joined #openstack-infra | 06:42 | |
| *** gokrokve has joined #openstack-infra | 06:42 | |
| *** pcrews has joined #openstack-infra | 06:45 | |
| *** _nadya_ has quit IRC | 06:46 | |
| *** talluri has joined #openstack-infra | 06:47 | |
| *** wenlock has quit IRC | 06:48 | |
| *** gokrokve has quit IRC | 06:48 | |
| *** pcrews has quit IRC | 06:50 | |
| *** _nadya_ has joined #openstack-infra | 06:51 | |
| *** achuprin has quit IRC | 06:55 | |
| *** jlibosva has quit IRC | 06:59 | |
| *** mrmartin has joined #openstack-infra | 07:00 | |
| *** flaper87|afk is now known as flaper87 | 07:01 | |
| *** jlibosva has joined #openstack-infra | 07:03 | |
| *** jcoufal has joined #openstack-infra | 07:08 | |
| *** achuprin has joined #openstack-infra | 07:10 | |
| *** pblaho has joined #openstack-infra | 07:15 | |
| *** nati_ueno has quit IRC | 07:16 | |
| *** lukego has joined #openstack-infra | 07:20 | |
| *** _nadya_ has quit IRC | 07:20 | |
| *** sdake_ has quit IRC | 07:25 | |
| *** _nadya_ has joined #openstack-infra | 07:26 | |
| *** ilives has quit IRC | 07:29 | |
| *** ilives has joined #openstack-infra | 07:30 | |
| *** morganfainberg is now known as morganfainberg_Z | 07:38 | |
| *** ildikov has quit IRC | 07:38 | |
| *** ildikov has joined #openstack-infra | 07:43 | |
| *** _nadya_ has quit IRC | 07:45 | |
| *** Longgeek has quit IRC | 07:45 | |
| *** Longgeek has joined #openstack-infra | 07:45 | |
| *** ilives has quit IRC | 07:47 | |
| *** ilives has joined #openstack-infra | 07:53 | |
| *** nati_ueno has joined #openstack-infra | 07:55 | |
| *** praneshp has quit IRC | 08:01 | |
| *** derekh has joined #openstack-infra | 08:02 | |
| *** jpich has joined #openstack-infra | 08:09 | |
| *** reed has joined #openstack-infra | 08:12 | |
| *** jlibosva has quit IRC | 08:16 | |
| *** achuprin has quit IRC | 08:16 | |
| *** achuprin has joined #openstack-infra | 08:16 | |
| *** jlibosva has joined #openstack-infra | 08:17 | |
| *** amotoki has joined #openstack-infra | 08:18 | |
| *** ihrachyshka has joined #openstack-infra | 08:18 | |
| *** dizquierdo has joined #openstack-infra | 08:19 | |
| *** ilives has quit IRC | 08:20 | |
| *** ilives has joined #openstack-infra | 08:22 | |
| *** Alexei_987 has quit IRC | 08:27 | |
| *** matsuhashi has quit IRC | 08:27 | |
| *** rediskin has joined #openstack-infra | 08:28 | |
| *** e0ne has joined #openstack-infra | 08:30 | |
| openstackgerrit | Pavel Sedlák proposed a change to openstack-infra/jenkins-job-builder: Fix recursion depth error cause by Git-remotes https://review.openstack.org/89773 | 08:32 |
| openstackgerrit | Pavel Sedlák proposed a change to openstack-infra/jenkins-job-builder: Fix recursion depth error caused by Git-remotes https://review.openstack.org/89773 | 08:32 |
| *** matsuhashi has joined #openstack-infra | 08:34 | |
| *** marcoemorais1 has quit IRC | 08:35 | |
| *** jooools has joined #openstack-infra | 08:37 | |
| *** ociuhandu has quit IRC | 08:38 | |
| *** BadCub has quit IRC | 08:41 | |
| *** ilives has quit IRC | 08:42 | |
| *** ilives has joined #openstack-infra | 08:43 | |
| *** ilives has quit IRC | 08:50 | |
| *** ilives has joined #openstack-infra | 08:52 | |
| *** etoews has joined #openstack-infra | 08:59 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add rally-cli job https://review.openstack.org/86331 | 09:03 |
| *** marcoemorais has joined #openstack-infra | 09:03 | |
| *** rcarrillocruz has joined #openstack-infra | 09:03 | |
| *** etoews has quit IRC | 09:04 | |
| *** marcoemorais1 has joined #openstack-infra | 09:05 | |
| *** rcarrillocruz1 has quit IRC | 09:05 | |
| *** jamespage_ has joined #openstack-infra | 09:08 | |
| *** rcarrillocruz has quit IRC | 09:08 | |
| *** marcoemorais has quit IRC | 09:08 | |
| *** marcoemorais1 has quit IRC | 09:09 | |
| *** e0ne_ has joined #openstack-infra | 09:12 | |
| *** jcoufal has quit IRC | 09:14 | |
| *** e0ne has quit IRC | 09:15 | |
| *** ociuhandu has joined #openstack-infra | 09:17 | |
| *** arosen1 has quit IRC | 09:26 | |
| *** ildikov_ has joined #openstack-infra | 09:28 | |
| *** ildikov has quit IRC | 09:31 | |
| *** jamespage_ has quit IRC | 09:31 | |
| *** ilives has quit IRC | 09:34 | |
| *** jamespage_ has joined #openstack-infra | 09:34 | |
| *** aglarendil|nb has quit IRC | 09:34 | |
| *** aglarendil|nb has joined #openstack-infra | 09:37 | |
| *** hashar has joined #openstack-infra | 09:37 | |
| *** jamespage_ has quit IRC | 09:38 | |
| *** zehicle_at_dell has joined #openstack-infra | 09:40 | |
| *** yaguang has quit IRC | 09:40 | |
| *** yjiang has quit IRC | 09:42 | |
| *** jamespage_ has joined #openstack-infra | 09:42 | |
| *** zhiyan is now known as zhiyan_ | 09:42 | |
| *** zehicle has quit IRC | 09:44 | |
| *** arosen1 has joined #openstack-infra | 09:44 | |
| *** alexpilotti has joined #openstack-infra | 09:46 | |
| *** zehicle_at_dell has quit IRC | 09:47 | |
| *** Ryan_Lane has quit IRC | 09:48 | |
| *** Ryan_Lane has joined #openstack-infra | 09:50 | |
| *** marcoemorais has joined #openstack-infra | 10:06 | |
| *** marcoemorais has quit IRC | 10:10 | |
| *** matsuhashi has quit IRC | 10:16 | |
| openstackgerrit | Ilya Sviridov proposed a change to openstack-infra/reviewstats: Added MagnetoDB project https://review.openstack.org/89579 | 10:17 |
| *** jcoufal has joined #openstack-infra | 10:18 | |
| *** Ryan_Lane has joined #openstack-infra | 10:19 | |
| *** aglarendil|nb has quit IRC | 10:21 | |
| SergeyLukjanov | ryanpetrello, yep, it's 2:22 pm for me | 10:22 |
| *** matsuhashi has joined #openstack-infra | 10:23 | |
| *** Ryan_Lane has quit IRC | 10:23 | |
| SergeyLukjanov | sdague, could you please take a look at https://review.openstack.org/#/c/89594/ when you have some time | 10:25 |
| *** jamespage_ has quit IRC | 10:28 | |
| *** ociuhandu has quit IRC | 10:32 | |
| *** e0ne_ has quit IRC | 10:32 | |
| *** ociuhandu has joined #openstack-infra | 10:33 | |
| *** _nadya_ has joined #openstack-infra | 10:35 | |
| *** coolsvap is now known as coolsvap|afk | 10:38 | |
| *** nithya_g has joined #openstack-infra | 10:39 | |
| *** _nadya_ has quit IRC | 10:39 | |
| openstackgerrit | Alexei Kornienko proposed a change to openstack/requirements: Added argcomplete requirement https://review.openstack.org/65863 | 10:41 |
| sdague | SergeyLukjanov: looking | 10:42 |
| sdague | SergeyLukjanov: is the backwards compat stuff still needed in master? | 10:42 |
| SergeyLukjanov | sdague, yup, patch is on review for enabling common auth configs | 10:43 |
| sdague | ok | 10:43 |
| sdague | +A | 10:43 |
| SergeyLukjanov | sdague, so, after merging it, I'll propose removal of backward compat | 10:43 |
| SergeyLukjanov | sdague, thx! | 10:43 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Introducing the event timeline https://review.openstack.org/87959 | 10:43 |
| SergeyLukjanov | (I'll propose removal right now to not forget to do it :) ) | 10:43 |
| *** _nadya_ has joined #openstack-infra | 10:44 | |
| sdague | SergeyLukjanov: any idea what's up with nodepool? | 10:52 |
| sdague | we go stupid again with allocating nodes? | 10:53 |
| *** Longgeek has quit IRC | 10:53 | |
| *** Longgeek has joined #openstack-infra | 10:53 | |
| SergeyLukjanov | sdague, /me looking | 10:55 |
| SergeyLukjanov | sdague, what's wrong? | 10:56 |
| sdague | top of gate is waiting on nodes, all of them | 10:56 |
| sdague | check queue is huge for this hour of the day | 10:56 |
| SergeyLukjanov | heh, my internet is too slow to download status.json, still waiting :) | 10:57 |
| sdague | including top of check being a qa-specs job which has been waiting for 4 hrs to get a docs node | 10:57 |
| SergeyLukjanov | oh, yeah... I see | 10:58 |
| sdague | you can see in the zuul events queue something happened at 6am UTC | 10:59 |
| SergeyLukjanov | sdague, mmm, periodic jobs? | 11:00 |
| sdague | maybe | 11:00 |
| sdague | we've never recovered since | 11:00 |
| SergeyLukjanov | sdague, or terrorists... | 11:00 |
| sdague | also tripleo cloud looks offline again | 11:00 |
| sdague | but currently we've got 10 changes in the gate that could merge, except top of gate is busted | 11:01 |
| *** jamielennox is now known as jamielennox|away | 11:02 | |
| *** miqui_ has joined #openstack-infra | 11:02 | |
| SergeyLukjanov | sdague, yup, triplo cloud looks dead | 11:02 |
| *** miqui has quit IRC | 11:02 | |
| SergeyLukjanov | sdague, on the other hand we have about 300-400 slaves in use due to the charts | 11:02 |
| *** e0ne has joined #openstack-infra | 11:03 | |
| SergeyLukjanov | sdague, jenkins looks good - jobs are running https://jenkins05.openstack.org | 11:03 |
| sdague | yeh, I get concerned when the check queue isn't getting processed in order though | 11:04 |
| *** e0ne has quit IRC | 11:04 | |
| *** matsuhashi has quit IRC | 11:04 | |
| *** maxbit has quit IRC | 11:05 | |
| *** e0ne has joined #openstack-infra | 11:05 | |
| SergeyLukjanov | sdague, both check and gate have the same priority IIRC | 11:06 |
| sdague | yeh, that's not my issue :) | 11:06 |
| sdague | my issue is really old things in check queue getting ignored | 11:06 |
| sdague | i.e. it not acting like a queue | 11:06 |
| *** e0ne has quit IRC | 11:06 | |
| *** e0ne has joined #openstack-infra | 11:06 | |
| *** marcoemorais has joined #openstack-infra | 11:07 | |
| SergeyLukjanov | sdague, yeah, I understand you, just noticing aloud | 11:07 |
| sdague | yep | 11:07 |
| SergeyLukjanov | that's really strange that it's not action like a queue | 11:07 |
| sdague | I think it just is zuul being broken | 11:09 |
| sdague | my experience is that if it loses track of a change, it's done | 11:09 |
| sdague | I'm kicking out the taskflow change now for that reason | 11:09 |
| *** marcoemorais has quit IRC | 11:11 | |
| *** e0ne has quit IRC | 11:12 | |
| *** mburned_out is now known as mburned | 11:13 | |
| *** arosen1 has quit IRC | 11:18 | |
| *** e0ne has joined #openstack-infra | 11:18 | |
| *** aglarendil|nb has joined #openstack-infra | 11:19 | |
| *** mwagner_lap has quit IRC | 11:19 | |
| *** mbacchi has joined #openstack-infra | 11:23 | |
| *** ihrachyshka has quit IRC | 11:27 | |
| *** ihrachyshka has joined #openstack-infra | 11:30 | |
| *** maxbit has joined #openstack-infra | 11:31 | |
| *** adalbas has joined #openstack-infra | 11:35 | |
| *** aglarendil|nb has quit IRC | 11:36 | |
| *** zehicle_at_dell has joined #openstack-infra | 11:37 | |
| rediskin | SergeyLukjanov sdague fungi I added some README about gate stuff. could you pls take a look? https://review.openstack.org/#/c/89798/ | 11:38 |
| *** maxbit has quit IRC | 11:42 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add check-rally-dsvm-glance job https://review.openstack.org/89801 | 11:43 |
| *** ociuhandu has quit IRC | 11:46 | |
| *** olaph has quit IRC | 11:47 | |
| *** aglarendil|works has joined #openstack-infra | 11:47 | |
| *** aglarendil|works has quit IRC | 11:47 | |
| *** olaph has joined #openstack-infra | 11:47 | |
| *** aglarendil|works has joined #openstack-infra | 11:49 | |
| *** aglarendil|works has quit IRC | 11:49 | |
| *** Longgeek has quit IRC | 11:59 | |
| *** ociuhandu has joined #openstack-infra | 12:00 | |
| *** unicell has quit IRC | 12:00 | |
| *** coolsvap|afk is now known as coolsvap | 12:01 | |
| *** zehicle_at_dell has quit IRC | 12:02 | |
| *** ociuhandu has quit IRC | 12:04 | |
| *** lukego has quit IRC | 12:06 | |
| *** marcoemorais has joined #openstack-infra | 12:07 | |
| *** pdmars has joined #openstack-infra | 12:10 | |
| *** nosnos has quit IRC | 12:10 | |
| *** pdmars has quit IRC | 12:10 | |
| *** pdmars has joined #openstack-infra | 12:11 | |
| *** marcoemorais has quit IRC | 12:11 | |
| SergeyLukjanov | XenServer CI posting -1s again - https://review.openstack.org/#/c/89801/1 | 12:12 |
| *** dprince has joined #openstack-infra | 12:15 | |
| *** etoews has joined #openstack-infra | 12:15 | |
| *** Longgeek has joined #openstack-infra | 12:17 | |
| *** AJaeger has joined #openstack-infra | 12:20 | |
| AJaeger | HI infra team. Are you aware that the gate is not working at all? | 12:22 |
| *** talluri has quit IRC | 12:23 | |
| *** pblaho has quit IRC | 12:23 | |
| *** miqui_ is now known as miqui | 12:25 | |
| *** ildikov has joined #openstack-infra | 12:26 | |
| AJaeger | reading eavesdrop, you seem to be aware of the issue ;) | 12:26 |
| *** _nadya_ has quit IRC | 12:27 | |
| openstackgerrit | Vladislav Kuzmin proposed a change to openstack-infra/nodepool: Correction logic for work with new KVM/QEMU provider manager https://review.openstack.org/88555 | 12:28 |
| openstackgerrit | Vladislav Kuzmin proposed a change to openstack-infra/nodepool: Add initial layer for work with KVM/QEMU hosts https://review.openstack.org/88554 | 12:28 |
| *** ildikov_ has quit IRC | 12:28 | |
| *** ociuhandu has joined #openstack-infra | 12:29 | |
| *** dkliban_afk is now known as dkliban | 12:31 | |
| *** weshay has joined #openstack-infra | 12:32 | |
| *** jgallard has joined #openstack-infra | 12:32 | |
| *** maxbit has joined #openstack-infra | 12:32 | |
| *** smarcet has joined #openstack-infra | 12:34 | |
| *** aysyd has joined #openstack-infra | 12:34 | |
| *** w_ has joined #openstack-infra | 12:35 | |
| *** _nadya_ has joined #openstack-infra | 12:35 | |
| *** alexpilotti has quit IRC | 12:35 | |
| *** olaph has quit IRC | 12:37 | |
| *** Longgeek has quit IRC | 12:38 | |
| *** rlandy has joined #openstack-infra | 12:39 | |
| *** Longgeek has joined #openstack-infra | 12:39 | |
| *** alexpilotti has joined #openstack-infra | 12:40 | |
| *** _nadya_ has quit IRC | 12:40 | |
| *** _nadya_ has joined #openstack-infra | 12:44 | |
| *** _nadya_ has quit IRC | 12:44 | |
| *** jgrimm has joined #openstack-infra | 12:44 | |
| *** andreaf_ has quit IRC | 12:48 | |
| *** andreaf has joined #openstack-infra | 12:48 | |
| YorikSar | sdague, SergeyLukjanov: mb it's a good time to upgrade Zuul? :) | 12:50 |
| anteaya | YorikSar: zuul and nodepool are having issues right now | 12:53 |
| anteaya | we are waiting for jeblair to come online to hear his thoughts | 12:53 |
| YorikSar | anteaya: I get it. That's why I think noone will notice Zuul restart | 12:54 |
| anteaya | it is possible he might decide to restart zuul, but i don't know | 12:54 |
| * anteaya nods | 12:54 | |
| anteaya | we will have to wait to hear jim's thoughts | 12:54 |
| *** w_ is now known as olaph | 12:57 | |
| *** mwagner_lap has joined #openstack-infra | 12:59 | |
| *** Longgeek_ has joined #openstack-infra | 13:00 | |
| *** dkranz has joined #openstack-infra | 13:00 | |
| *** mfer has joined #openstack-infra | 13:02 | |
| *** lukego has joined #openstack-infra | 13:02 | |
| *** timrc-afk is now known as timrc | 13:02 | |
| *** markmc has joined #openstack-infra | 13:02 | |
| *** Longgeek has quit IRC | 13:04 | |
| *** oomichi has quit IRC | 13:06 | |
| *** marcoemorais has joined #openstack-infra | 13:08 | |
| *** prad has joined #openstack-infra | 13:08 | |
| *** oomichi has joined #openstack-infra | 13:09 | |
| *** oomichi has quit IRC | 13:10 | |
| *** marcoemorais1 has joined #openstack-infra | 13:10 | |
| *** marcoemorais has quit IRC | 13:12 | |
| *** marcoemorais1 has quit IRC | 13:14 | |
| *** nati_ueno has quit IRC | 13:21 | |
| anteaya | I'm guessing we dropped from 800 to 600 nodes because we ran out of quota in a region | 13:30 |
| SergeyLukjanov | anteaya, yup, it's possible I think | 13:30 |
| *** Ryan_Lane has joined #openstack-infra | 13:31 | |
| *** mriedem has joined #openstack-infra | 13:34 | |
| *** thiagop has quit IRC | 13:34 | |
| *** Ryan_Lane has quit IRC | 13:35 | |
| *** devoid has joined #openstack-infra | 13:39 | |
| *** zz_gondoi is now known as gondoi | 13:39 | |
| *** thiagop has joined #openstack-infra | 13:43 | |
| *** mrodden1 has quit IRC | 13:47 | |
| *** jpeeler has joined #openstack-infra | 13:51 | |
| *** jpeeler has quit IRC | 13:51 | |
| *** jpeeler has joined #openstack-infra | 13:51 | |
| *** signed8bit has joined #openstack-infra | 13:53 | |
| anteaya | thiagop: you got your issue from yesterday resolved? | 13:53 |
| thiagop | anteaya: yeah, suddenly the tests passed on a (third) recheck | 13:54 |
| *** chandan_kumar is now known as chandankumar | 13:54 | |
| anteaya | thiagop: odd | 13:55 |
| anteaya | but I am glad they passed | 13:55 |
| thiagop | anteaya: isn't it?! | 13:55 |
| *** wchrisj has joined #openstack-infra | 13:56 | |
| anteaya | thiagop: do you know how to search for or file a bug if it happens again? | 13:57 |
| thiagop | anteaya: no | 13:58 |
| anteaya | let's remedy that | 13:58 |
| anteaya | what project to you contribute to mostly? | 13:58 |
| thiagop | anteaya: please! :) | 13:58 |
| thiagop | mostly Horizon and Keystone | 13:59 |
| anteaya | great | 13:59 |
| anteaya | log into launchpad.net | 13:59 |
| *** darraghb has joined #openstack-infra | 13:59 | |
| *** homeless has joined #openstack-infra | 13:59 | |
| anteaya | search for either openstack horizon or openstack keystone | 13:59 |
| *** mrodden has joined #openstack-infra | 13:59 | |
| anteaya | at the top of the project page, find the bugs button | 14:00 |
| anteaya | which takes you to the bugs filed against that project | 14:00 |
| *** thomasem has joined #openstack-infra | 14:00 | |
| anteaya | tell me when you are there | 14:00 |
| *** zns has joined #openstack-infra | 14:01 | |
| *** gokrokve has joined #openstack-infra | 14:02 | |
| *** doug-fish has joined #openstack-infra | 14:02 | |
| mrodden | even xenserver ci likes the idea of a new bash8 project... https://review.openstack.org/#/c/88747/ | 14:03 |
| anteaya | mrodden: awesome | 14:03 |
| mrodden | wondering why it picked up that patchset | 14:03 |
| thiagop | anteaya: Ok, found it | 14:04 |
| mrodden | they must want to test -infra/config for some reason? | 14:04 |
| anteaya | I guess, BobBall would have more to say about that | 14:04 |
| anteaya | thiagop: well done | 14:04 |
| anteaya | next find your logs from your failures yesterday | 14:04 |
| anteaya | find the error message, or a keyword from it and search the bugs for that keyword | 14:05 |
| *** ominakov has quit IRC | 14:06 | |
| *** yfried has quit IRC | 14:07 | |
| thiagop | anteaya: search for "volume not attached" | 14:08 |
| thiagop | several results | 14:08 |
| thiagop | I'll see if something is related | 14:08 |
| *** marcoemorais has joined #openstack-infra | 14:11 | |
| *** signed8bit has quit IRC | 14:12 | |
| *** weshay has quit IRC | 14:12 | |
| *** signed8bit has joined #openstack-infra | 14:12 | |
| *** markmcclain has joined #openstack-infra | 14:14 | |
| *** rwsu has joined #openstack-infra | 14:14 | |
| *** marcoemorais has quit IRC | 14:15 | |
| *** msabramo has quit IRC | 14:18 | |
| *** cody-somerville has quit IRC | 14:20 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/config: Setup jobs and hooks for Murano docs https://review.openstack.org/87974 | 14:20 |
| *** thedodd has joined #openstack-infra | 14:22 | |
| *** pcrews has joined #openstack-infra | 14:22 | |
| jeblair | hi there | 14:24 |
| *** freyes has joined #openstack-infra | 14:24 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Added MagnetoDB project https://review.openstack.org/89579 | 14:24 |
| anteaya | hey jeblair | 14:25 |
| jeblair | the flat lines on the nodepool graph don't look good | 14:25 |
| anteaya | no | 14:25 |
| anteaya | everything appears stuck | 14:26 |
| jeblair | nodepool is running and jobs are completing though | 14:26 |
| anteaya | hmmm | 14:26 |
| jeblair | there is a problem with hpcloud region b | 14:26 |
| *** julim has joined #openstack-infra | 14:26 | |
| anteaya | oh | 14:26 |
| jeblair | i'm not sure that explains the flat lines though | 14:26 |
| BobBallAway | mrodden, anteaya: it's because I made a mistake :D | 14:27 |
| anteaya | BobBallAway: okay | 14:27 |
| anteaya | glad your system liked the patch though | 14:27 |
| BobBallAway | we love it. | 14:27 |
| mrodden | heh | 14:27 |
| *** markmcclain has quit IRC | 14:27 | |
| anteaya | BobBallAway: you have voted in the tc election, right? | 14:28 |
| *** Alexei_987 has joined #openstack-infra | 14:28 | |
| anteaya | BobBallAway: can you convince some fellow contributors to vote too? | 14:28 |
| BobBallAway | Of course I have :) And spread the word. | 14:28 |
| BobBallAway | I guess voting rates are low? | 14:28 |
| anteaya | thank you so much | 14:28 |
| anteaya | we have expanded significantly | 14:29 |
| mrodden | its open to all ATCs? | 14:29 |
| anteaya | we would like to get the percentage up | 14:29 |
| anteaya | yes 1500+ | 14:29 |
| mrodden | wow | 14:29 |
| anteaya | so lots of votes but we need the percentage | 14:29 |
| BobBallAway | Am I allowed to ask how many have voted so far? just for my curiosity | 14:29 |
| anteaya | so talk it up, and thank you | 14:29 |
| *** markmcclain has joined #openstack-infra | 14:29 | |
| anteaya | 27.5% | 14:30 |
| jeblair | i'm starting to think the flat lines are graphite related | 14:30 |
| anteaya | we would like to be over 30%, so we are getting there | 14:30 |
| BobBallAway | Ah... That's not ideal. | 14:30 |
| anteaya | jeblair: that would be interesting | 14:30 |
| BobBallAway | Indeed. I'll prompt a few people just to make sure they have voted | 14:30 |
| anteaya | BobBallAway: yeah, we need more participation | 14:30 |
| anteaya | BobBallAway: thank you | 14:30 |
| anteaya | BobBallAway: if anyone you talk to has questions or concerns tell them to find me | 14:31 |
| anteaya | I want to help them | 14:31 |
| *** weshay has joined #openstack-infra | 14:31 | |
| BobBallAway | OK | 14:32 |
| anteaya | thanks | 14:32 |
| jeblair | okay, i think nodepool was rebooted because of the memory leak/swap issue, and for some reason didn't start sending data to graphite | 14:32 |
| jeblair | i will restart nodepool | 14:32 |
| jeblair | that should be fixed now | 14:32 |
| BobBallAway | how do you guys manage nodepool btw jeblair? I run it as a daemon but each time I restart it I have to remove the pid file which is frustrating... | 14:32 |
| jeblair | now, for the hpcloud region b problem, which anteaya correctly surmised is responsible for the 200 count node drop... | 14:33 |
| jeblair | BobBallAway: if you can fix that, that would be great. :) | 14:33 |
| jeblair | OverLimit: Maximum number of ports exceeded (HTTP 413) (Request-ID: req-1915fd0f-6d38-4e7e-9713-edc68bfbd3f8) | 14:33 |
| jeblair | that ^ is what we're getting from hpcloud region b. that one is new to me. | 14:34 |
| BobBallAway | hahah ;) | 14:34 |
| *** hashar has quit IRC | 14:34 | |
| *** hashar has joined #openstack-infra | 14:35 | |
| *** msabramo has joined #openstack-infra | 14:35 | |
| anteaya | jeblair: wonderful | 14:38 |
| jeblair | i think we're going to have to open a ticket with hpcloud about that | 14:38 |
| *** zns has quit IRC | 14:40 | |
| *** thiagop has left #openstack-infra | 14:40 | |
| *** zns has joined #openstack-infra | 14:41 | |
| *** zns has quit IRC | 14:41 | |
| *** zns has joined #openstack-infra | 14:41 | |
| *** dkliban is now known as dkliban_afk | 14:44 | |
| *** markwash has quit IRC | 14:46 | |
| *** wenlock has joined #openstack-infra | 14:47 | |
| SergeyLukjanov | jeblair, morning | 14:48 |
| AaronGr | Can someone tell me what raid level infra uses for it's permanent servers? (static.o.o in particular) | 14:50 |
| *** zns has quit IRC | 14:50 | |
| mrodden | over 9000 | 14:50 |
| BobBallAway | raid 9000 wow | 14:51 |
| mrodden | sorry, couldn't resist, i dont actually know | 14:51 |
| AaronGr | :) | 14:51 |
| *** zns has joined #openstack-infra | 14:51 | |
| jeblair | AaronGr: we use rackspace cinder on static | 14:51 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Pass pngcrush on status png files https://review.openstack.org/89865 | 14:51 |
| AaronGr | jeblair: as standalone volumes, or part of an mdraid? | 14:52 |
| jeblair | AaronGr: lvm concat. i believe rax backs them with some kind of raid, but am unsure as to level. | 14:53 |
| jeblair | AaronGr: they use raid10 | 14:54 |
| AaronGr | ok, nice. thank you :) | 14:54 |
| *** david-lyle has joined #openstack-infra | 14:55 | |
| jeblair | http://www.rackspace.com/knowledge_center/product-faq/cloud-block-storage | 14:55 |
| *** ominakov has joined #openstack-infra | 14:56 | |
| jeblair | they have a faq now, which actually recommends using software raid1 in case their controllers fail | 14:56 |
| mordred | oh - wow | 14:56 |
| jeblair | refreshingly realistic advice :) | 14:56 |
| jeblair | but anyway, if we lose all the log data because of a catastrophic event: meh. | 14:57 |
| mordred | do they have anti-affinity rules for provisioning cinder volumes? | 14:57 |
| pleia2 | good morning | 14:57 |
| jeblair | mordred: very good question | 14:57 |
| mordred | because that raid1 isn't going to help if both volumes are from the same phyiscal raid controller :) | 14:57 |
| jeblair | pleia2: morning | 14:58 |
| mrodden | mordred: darn you and your logic | 14:58 |
| mordred | AaronGr: other than static, we're just using the normal filesystems on the cloud servers | 14:58 |
| AaronGr | mordred: thanks, perfect. | 14:58 |
| mrodden | if they use cinder, it would be there with the schedule hints/anti collocation scheduler | 14:58 |
| *** jcoufal has quit IRC | 14:58 | |
| mrodden | filter... thing | 14:59 |
| mordred | mrodden: cool | 14:59 |
| jeblair | mordred: we did start using cinder on graphite. we have 0.95TB of data there | 14:59 |
| *** lxsli has joined #openstack-infra | 15:00 | |
| *** jcoufal has joined #openstack-infra | 15:01 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revert "Reduce IAD usage by 50%" https://review.openstack.org/89871 | 15:01 |
| jeblair | mordred: ^ self approved | 15:01 |
| mordred | jeblair: you know, when we make the new review.o.o - maybe we shoudl think about a raid1 cinder volume pair for /home/gerrit2 - not because we cant' recover from a failure - but because recovering from a failure would be very boring | 15:02 |
| mordred | jeblair: ++ | 15:02 |
| jeblair | mordred: we do back it up | 15:02 |
| *** dkranz has quit IRC | 15:03 | |
| *** UtahDave has joined #openstack-infra | 15:03 | |
| *** ihrachyshka has quit IRC | 15:04 | |
| lxsli | Hiya, the gate seems stuck - anything happening about that please? | 15:04 |
| jeblair | mordred: we're getting interesting errors from hpcloud region b; how should we communicate that to the hpcloud region b folks? | 15:05 |
| mordred | jeblair: oh - well, that looks like a neutron quota issue | 15:06 |
| mordred | I wonder if we're leaking something | 15:06 |
| mordred | or if they are | 15:07 |
| jeblair | mordred: i don't think we do anything with neutron explicitly | 15:07 |
| mordred | jeblair: let me poke for 3 seconds, and then I can file a ticket | 15:08 |
| jeblair | mordred: cool | 15:08 |
| *** devoid has quit IRC | 15:09 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Filter non public fields for user list https://review.openstack.org/89511 | 15:10 |
| *** julim has quit IRC | 15:11 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Delete endpoint removed for Projects https://review.openstack.org/89344 | 15:11 |
| anteaya | morning pleia2 | 15:12 |
| anteaya | lxsli: much is happening about that | 15:13 |
| anteaya | I need to read backscroll to know the current status | 15:13 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Added support for Exclusion plugin https://review.openstack.org/77940 | 15:14 |
| jeblair | it's clearly not actually stuck ^. | 15:15 |
| lxsli | anteaya: OK thanks, I'll butt out! | 15:15 |
| anteaya | true | 15:16 |
| anteaya | lxsli: no, no worries | 15:16 |
| jeblair | just slow since we are running at 65% of capacity | 15:16 |
| anteaya | so jeblair restarted nodepool, we lost 200 nodes due to some error in hp region b, which mordred is looking at, and jeblair just increased our capacity with rax iad | 15:16 |
| anteaya | lxsli: ^ | 15:17 |
| *** dkranz has joined #openstack-infra | 15:17 | |
| hashar | could the OpenStack foundation maintain its own cloud of machines instead of relying on third parties? (sorry if that sounds stupid) | 15:18 |
| lxsli | that's great thanks | 15:18 |
| anteaya | lxsli: np | 15:18 |
| jeblair | hashar: the foundation isn't really staffed at a level to do that, however, the project, in the form of the tripleo program, is doing something similar | 15:18 |
| hashar | we could run a banner to get you some donations so you hire moaar people :] | 15:19 |
| jeblair | hashar: the tripleo program operates a CD openstack cloud with two regions which provides the nodes for the 'tripleo' pipelines in zuul | 15:19 |
| mordred | hashar: we'd rather the foundation not hire more people and instead have folks pony up regions into the tripleo cloud I think | 15:20 |
| jeblair | it should be able to provide nodes for regular jobs eventually, but it's not there yet (i believe it's offline right now) | 15:20 |
| jeblair | mordred: ++ | 15:20 |
| hashar | the tripleO cloud being a cloud you administer but build on top of 3rd parties clouds is it? | 15:20 |
| jeblair | hashar: no it's on real hardware | 15:20 |
| jeblair | (but 3rd parties hardware, yes) | 15:21 |
| openstackgerrit | Thomas Herve proposed a change to openstack-infra/config: Fix gate-heat-templates-dsvm shell script https://review.openstack.org/89875 | 15:22 |
| hashar | so tripleO would be unifying all the hardware material provided to you ? | 15:22 |
| jeblair | hashar: i believe they allocate a bunch of machines and give remote admin capability to the tripleo admin team, then they run tripleo on it to bring up openstack | 15:22 |
| therve | jeblair, SergeyLukjanov : As you were kind enough to review my previous attempt ^^ | 15:22 |
| *** mrmartin has quit IRC | 15:22 | |
| jeblair | hashar: basically, yes -- they take raw hardware in random places and turn it into a region of a cloud that the ci system can consume | 15:22 |
| hashar | does it let you administrate the triple0 over cloud without requiring additional rights from third parties? | 15:23 |
| hashar | I should get in touch with tripleo folks :D | 15:24 |
| jeblair | hashar: they admin the overcloud and the undercloud | 15:24 |
| *** amrith is now known as amrith-is-awol | 15:25 | |
| jeblair | hashar: they are in #tripleo also many of them are here | 15:25 |
| hashar | yeah i joined a while back. I should follow up there maybe. Thanks! | 15:25 |
| jeblair | np | 15:25 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Navigate to story details after create complete https://review.openstack.org/86571 | 15:27 |
| *** jlibosva has quit IRC | 15:28 | |
| *** prad has quit IRC | 15:30 | |
| *** gokrokve_ has joined #openstack-infra | 15:31 | |
| *** prad has joined #openstack-infra | 15:32 | |
| *** asselin2 has joined #openstack-infra | 15:33 | |
| ruhe | is it expected to have XenServer comments on changes in openstack-infra/config? | 15:33 |
| jeblair | ruhe: link? | 15:33 |
| ruhe | jeblair: https://review.openstack.org/#/c/87974/ | 15:33 |
| *** e0ne has quit IRC | 15:34 | |
| *** gokrokve has quit IRC | 15:34 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Adds metadata to story detail page https://review.openstack.org/87030 | 15:34 |
| *** aysyd has quit IRC | 15:34 | |
| jeblair | anteaya: do you know who runs that ^? | 15:34 |
| *** e0ne has joined #openstack-infra | 15:35 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Reduce IAD usage by 50%" https://review.openstack.org/89871 | 15:35 |
| jeblair | BobBallAway: do you run the xenserver third-party ci? | 15:35 |
| BobBallAway | yes | 15:36 |
| mordred | jeblair: (neutronclient)root@ci-puppetmaster:~# neutron quota-show | 15:36 |
| BobBallAway | or I mean uhhh depends who's asking and why... | 15:36 |
| mordred | | floatingip | 600 | | 15:36 |
| mordred | | port | 70 | | 15:36 |
| BobBallAway | I was too hasty in accepting responsibility there, wasn't I :D | 15:37 |
| mordred | so - although we have 600 floating ip's of quota, we only have 70 ports of quota :) | 15:37 |
| jeblair | BobBallAway: heh. it seems to be leaving comments on infra/config patches | 15:37 |
| BobBallAway | I think I've fixed that - I added the wrong project by mistake, sorry :) | 15:37 |
| BobBallAway | But hopefully it didn't fail any of them? :P | 15:37 |
| jeblair | BobBallAway: strangly no! | 15:37 |
| BobBallAway | ah, yes it did... | 15:38 |
| BobBallAway | https://review.openstack.org/#/c/86331/ but there was a second patch so all's good | 15:38 |
| BobBallAway | ah, no, there was a recheck issued | 15:38 |
| jeblair | mordred: how many of those are used? | 15:38 |
| BobBallAway | But hopefully it won't do it any more :) | 15:38 |
| mordred | jeblair: 70 | 15:39 |
| mordred | jeblair: a port is used for each automatic private 10. address | 15:39 |
| mordred | | f9d6a528-58f3-43f7-b51c-8db05dcfc255 | | fa:16:3e:0a:a5:24 | {"subnet_id": "790e5c02-abe8-4bc5-b880-40e6b5061460", "ip_address": "10.0.0.129"} | | 15:39 |
| mordred | is an example line from "neutron port-list" | 15:39 |
| *** gyee has joined #openstack-infra | 15:40 | |
| jeblair | mordred: nodepool knows about 57 nodes, which seems like more than a margin of error away from 70 :/ | 15:40 |
| jeblair | mordred: but there are 10 aliens, so that probably does put it within the margin of error of 70 | 15:41 |
| *** freyes has quit IRC | 15:42 | |
| *** aysyd has joined #openstack-infra | 15:42 | |
| *** msabramo has quit IRC | 15:43 | |
| *** dkliban_afk is now known as dkliban | 15:44 | |
| mordred | jeblair: and actually, it would be 69 - I didn't subtract enough header lines | 15:44 |
| mordred | jeblair: I'm requesting that the port count be increased appropriately | 15:44 |
| anteaya | sorry, my mother is here to help me deal with water, so I am here then gone for a bit | 15:44 |
| *** Sukhdev has joined #openstack-infra | 15:44 | |
| ttx | jeblair: weird status for https://review.openstack.org/#/c/86719/ | 15:45 |
| ttx | "Change cannot be merged due to unsatisfiable dependencies." | 15:45 |
| jeblair | that looks like the special spamaps method of uploading a change with dependencies that aren't in gerrit | 15:45 |
| * ttx blames SpamapS | 15:45 | |
| jeblair | mordred: do you recall the process for that? | 15:46 |
| * SpamapS pwns gerrit | 15:46 | |
| *** cody-somerville has joined #openstack-infra | 15:46 | |
| *** cody-somerville has quit IRC | 15:46 | |
| *** cody-somerville has joined #openstack-infra | 15:46 | |
| * ttx used to blame everything on SpamapS | 15:46 | |
| ttx | now I must be selective | 15:46 |
| mordred | jeblair: I don't- just that SpamapS has special powers | 15:46 |
| SpamapS | ttx: join the club, my whole family plays that game. ;) | 15:46 |
| ttx | I blame this one on SpamapS | 15:46 |
| jeblair | or maybe SpamapS remembers | 15:46 |
| jeblair | i know you can do it if you push to gerrit without using git-review; i can't recall if there's a way to do it when using git review | 15:48 |
| SpamapS | jeblair: I do, IIRC we had to manually push the deps into gerrit. | 15:48 |
| jeblair | (eg, if you push to refs/changes/foo, it's quite easy) | 15:48 |
| jeblair | but git-review pushes to refs/for/branch | 15:49 |
| SpamapS | btw I flipped to this window just before seeing my name because I heard that there are connectivity issues in hpcloud and wondering if they are documented anywhere | 15:49 |
| *** dizquierdo has quit IRC | 15:49 | |
| SpamapS | (since our HP tripleo test rack is also in an hpcloud DC) | 15:49 |
| jeblair | i'm unaware of connectivity issues | 15:49 |
| *** devoid has joined #openstack-infra | 15:50 | |
| *** msabramo has joined #openstack-infra | 15:51 | |
| *** achuprin has quit IRC | 15:51 | |
| anteaya | SpamapS: this was the error jeblair was getting: OverLimit: Maximum number of ports exceeded (HTTP 413) (Request-ID: req-1915fd0f-6d38-4e7e-9713-edc68bfbd3f8) | 15:52 |
| jeblair | krotscheck: did you use git-review to upload patch set 3 of https://review.openstack.org/#/c/86719/ | 15:53 |
| jeblair | krotscheck, ttx: at any rate, the solution is to rebase that patch on patches that are actually in gerrit and upload again | 15:54 |
| krotscheck | jeblair: Yup. | 15:55 |
| krotscheck | I can fix it. | 15:55 |
| *** cody-somerville has quit IRC | 15:55 | |
| ttx | jeblair: yes, I assumed that much -- just wanted to highlight another occurence, as I know you like to understand what they come from | 15:55 |
| *** e0ne has quit IRC | 15:55 | |
| SpamapS | ahh max ports is different, ok | 15:55 |
| krotscheck | Geez, it’s just HTML too :) | 15:56 |
| jeblair | ttx: yeah; i'm guessing krotscheck doesn't have the exact command sequence used to upload ps3 still laying around :/ | 15:57 |
| jeblair | krotscheck: ? | 15:57 |
| krotscheck | jeblair: I did have some issues where I accidentally committed the same “Id: dfjkdsafjkladsf” twice | 15:57 |
| sdague | krotscheck: if you have some time, I clearly got lost in the woods on - https://review.openstack.org/#/c/87699/ and can't figure out how to get it to run and work. Any thoughts you have would be great. | 16:00 |
| krotscheck | jeblair: http://paste.openstack.org/show/76797/ | 16:00 |
| krotscheck | That seems od | 16:00 |
| *** amotoki_ has joined #openstack-infra | 16:00 | |
| krotscheck | sdague: Alright, I’ll look at that next. | 16:00 |
| *** maxbit has quit IRC | 16:01 | |
| *** maxbit has joined #openstack-infra | 16:01 | |
| anteaya | jeblair: not sure what is in your top 5 of things todo right now, I assume figuring out hp region b is somewhere in the top 3 | 16:02 |
| jeblair | anteaya: nope, mordred is opening a ticket, it's out of my hands | 16:03 |
| anteaya | jeblair: whenever you get a chance to let me know which option you would like to take on infra-manuals job templates: https://review.openstack.org/#/c/89421/ | 16:03 |
| anteaya | ah okay | 16:03 |
| jeblair | anteaya: my priority, through next monday is the gerrit upgrade | 16:03 |
| * anteaya nods | 16:03 | |
| *** ildikov has quit IRC | 16:03 | |
| anteaya | waiting for sweston to come back online to see where we are with his docs | 16:03 |
| *** dkranz has quit IRC | 16:04 | |
| anteaya | he couldn't get to them last night for internal reasons I dont' fathom | 16:04 |
| jeblair | cool. i'm trying to figure out what flavor of server to use | 16:04 |
| anteaya | great | 16:04 |
| *** marcoemorais has joined #openstack-infra | 16:04 | |
| anteaya | what are the options? | 16:05 |
| jeblair | anteaya: http://www.rackspace.com/cloud/servers/pricing/ | 16:05 |
| jeblair | anteaya: then check http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=9 to see what we're actually using | 16:05 |
| *** sweston has quit IRC | 16:05 | |
| mordred | ticket opened | 16:05 |
| jeblair | i'm thinking we should go with a 30g performance 2 node | 16:06 |
| anteaya | ha ha ha | 16:06 |
| anteaya | I was guessing that would be where you wanted it | 16:06 |
| jeblair | it gives us the same vcpu count we have now (though they are faster), so it should be a modest performance gain | 16:06 |
| *** cody-somerville has joined #openstack-infra | 16:07 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added story status to project story list. https://review.openstack.org/89887 | 16:07 |
| jeblair | but mostly it's the same amount of ram. we're only using about 10g, but the host seems to make ample use of the remainder for caching | 16:07 |
| jeblair | i think 15g would be enough of a reduction that we would see a performance hit | 16:07 |
| krotscheck | jeblair: I generated a new change ID. Hopefully that’ll work | 16:08 |
| jeblair | krotscheck: er now you lost the review history | 16:08 |
| krotscheck | Harumph | 16:08 |
| krotscheck | Isn’t that tied to the Change ID? | 16:08 |
| krotscheck | My hypothesis was that it’s the change ID that’s the problem. | 16:08 |
| jeblair | krotscheck: yes -- which is why you shouldn't change it. you just needed to rebase it on a change that is in gerrit | 16:08 |
| *** blamar has quit IRC | 16:08 | |
| krotscheck | jeblair: Alright, let’s try that too | 16:09 |
| anteaya | yes, I agree we should be looking at the 30GB flavour at least | 16:09 |
| anteaya | jeblair: any case for the 60GB vm? | 16:09 |
| nibalizer | hey guys i'm famous now! https://puppetlabs.com/blog/puppet-without-root-real-life-example | 16:10 |
| anteaya | nibalizer: w00t! | 16:11 |
| anteaya | such a great picture | 16:12 |
| *** aysyd has quit IRC | 16:12 | |
| jeblair | krotscheck: if you start a new branch from master, then do: | 16:13 |
| jeblair | git cherry-pick f7161106f34498506694595be2d5399f27c764e1 | 16:13 |
| jeblair | git cherry-pick f2b28db72b67d36ba794d8e36e118c8588f1d131 | 16:13 |
| jeblair | git cherry-pick d81c02e29ce1adf1644968f1bd808b7d1cc2ee50 | 16:13 |
| jeblair | git review | 16:13 |
| jeblair | actually... | 16:14 |
| krotscheck | jeblair: Ya know, I think I’m just going to wait for the other patches to collapse. | 16:14 |
| nibalizer | anteaya: thats my srsbsns face | 16:14 |
| jeblair | krotscheck: start a new branch from master | 16:14 |
| *** gokrokve has joined #openstack-infra | 16:14 | |
| jeblair | krotscheck: git reset --hard f2b28db72b67d36ba794d8e36e118c8588f1d131 | 16:14 |
| jeblair | krotscheck: git cherry-pick d81c02e29ce1adf1644968f1bd808b7d1cc2ee50 | 16:14 |
| jeblair | krotscheck: git review | 16:14 |
| jeblair | krotscheck: please do that; it's very confusing to reviewers to have multiple versions of patches outstanding in gerrit | 16:15 |
| *** dkranz has joined #openstack-infra | 16:15 | |
| jeblair | krotscheck: and we also like to keep review history intact -- starting a new change for a previously reviewed one hides reviews | 16:16 |
| *** gokrokv__ has joined #openstack-infra | 16:16 | |
| *** mkerrin has quit IRC | 16:16 | |
| krotscheck | jeblair: I’m aware of that. | 16:16 |
| krotscheck | jeblair: I misunderstood your original suggestion on how to fix it. | 16:16 |
| *** etoews has quit IRC | 16:17 | |
| jeblair | the most recent suggestion should fix it and not touch the other changes already in gerrit | 16:17 |
| jeblair | anteaya: i don't think we need the extra ram at this point | 16:17 |
| *** gokrokve_ has quit IRC | 16:18 | |
| wenlock | nibalizer, grats, im already sharing the article with others :P | 16:18 |
| jeblair | mordred: so we will need a cinder volume to put ~gerrit2/review_site and the apache mirror | 16:19 |
| mordred | jeblair: oh- well, I guess that solves the conversation from earlier | 16:20 |
| *** ominakov has quit IRC | 16:20 | |
| jeblair | mordred: (i assume we'll need to either bind mount or symlink one or both of those) | 16:20 |
| jeblair | mordred: so do you want to use software raid for it? | 16:20 |
| *** gokrokve has quit IRC | 16:20 | |
| zaro | anteaya: congrats on your patch. but it won't get into our gerrit until you backport that patch to our forked gerrit. | 16:21 |
| jeblair | mordred: i lean toward no -- it's additional complexity and without knowing the anti-affinity answer, i'm not sure it would be worth it; i think i'd rather rely on backups in that case | 16:21 |
| zaro | morning | 16:21 |
| mordred | jeblair: ++ | 16:21 |
| mordred | jeblair: why bindmount or symlink? | 16:21 |
| *** lakshmi has joined #openstack-infra | 16:21 | |
| jeblair | mordred: /home/gerrit2/review_site and /var/lib/git don't share a path root | 16:21 |
| mordred | jeblair: why not just make /home/gerrit2 mount /dev/xvda5 or whatever the volume will be? | 16:21 |
| mordred | ah | 16:21 |
| * mordred groks the words jeblair is now saying | 16:22 | |
| anteaya | jeblair: okay | 16:22 |
| jeblair | mordred: heh, we could use two volumes. :) | 16:22 |
| anteaya | zaro: thanks | 16:22 |
| mordred | jeblair: ooh. we could | 16:22 |
| anteaya | zaro: alright, I will work on backporting that patch to our forked gerrit, which fork? | 16:23 |
| anteaya | nibalizer: it is a nice face | 16:23 |
| jeblair | mordred: also, i'm pretty sure that _all_ of the replicas are full git replicas, right? so we already have 6 live off-site copies of the git repos | 16:23 |
| jeblair | mordred: (the ones on the git mirrors get their refs packed, but in a disaster, that's probably not a big deal) | 16:23 |
| mordred | jeblair: sake of argument - do we need /var/lib/git on gerrit? why don't we just serve ~gerrit2/review_site/git with apache directly? | 16:24 |
| *** changbl has quit IRC | 16:24 | |
| zaro | anteaya: clone the git.openstack.org/openstack-infra/gerrit then checkout "openstack/2.8.4" branch | 16:24 |
| *** markmcclain has quit IRC | 16:24 | |
| anteaya | k | 16:24 |
| zaro | anteaya: ohh remember to remote update and pull new changes first | 16:25 |
| zaro | anteaya: then you'll want to cherry-pick your merged changed (using the commit id). | 16:25 |
| jeblair | zaro: did those changes land? are you blocked on anything i can help with? | 16:25 |
| *** mkerrin has joined #openstack-infra | 16:26 | |
| openstackgerrit | Mohammed Naser proposed a change to openstack-infra/elastic-recheck: Add a fingerprint for bug 1311778 https://review.openstack.org/89891 | 16:26 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Fixed text display on Task Status dropdown. https://review.openstack.org/88679 | 16:26 |
| uvirtbot | Launchpad bug 1311778 in nova "Unit tests fail with MessagingTimeout errors" [Undecided,New] https://launchpad.net/bugs/1311778 | 16:26 |
| mordred | jeblair: btw - speaking with the folks at hp support, they suggest that perhaps we might not want all 600 nodes behind one single router | 16:26 |
| mordred | jeblair: on one network | 16:26 |
| mordred | I'm going to find a room and talk to a guy with a whiteboard to get more info | 16:26 |
| zaro | jeblair: nope. the first change didn't land, tried to recheck about 4 times yesterday always get an HP slave. | 16:27 |
| jeblair | zaro: good news! your changes of getting an hp slave are lower right now! :) | 16:27 |
| zaro | jeblair: just tried again a few minutes ao. | 16:27 |
| zaro | ao/ ago | 16:27 |
| anteaya | ha ha ha | 16:27 |
| jeblair | chances even | 16:27 |
| zaro | jeblair: well see.. | 16:28 |
| mordred | jeblair: our neutron port limit has been increased to 610 | 16:29 |
| *** freyes has joined #openstack-infra | 16:29 | |
| jeblair | mordred: yay! | 16:29 |
| mordred | jeblair: turns out ports are also used for the network and the two connections in and out of the router | 16:29 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added task status display to story list view https://review.openstack.org/86712 | 16:29 |
| mordred | jeblair: so we fudged the quota slightly higher than needed | 16:29 |
| krotscheck | sdague: Looking at your thing while I wait for all the patches to land. | 16:30 |
| jeblair | krotscheck: are you going to upload a new version of 86719 ? | 16:30 |
| anteaya | zaro: what am I doing wrong? http://paste.openstack.org/show/76802/ | 16:31 |
| krotscheck | jeblair: Yes, once all the other patches land. | 16:31 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added story status label filter https://review.openstack.org/86718 | 16:31 |
| *** jpich has quit IRC | 16:31 | |
| anteaya | zaro do I have to have the remote branch local? | 16:31 |
| *** hashar has quit IRC | 16:31 | |
| zaro | anteaya: i think it's just 'git checkout openstack/2.8.4', no branch in that command | 16:32 |
| anteaya | k | 16:33 |
| anteaya | zaro you are correct | 16:34 |
| anteaya | I was overthinking | 16:34 |
| *** jcoufal has quit IRC | 16:35 | |
| mnaser | https://review.openstack.org/#/c/87775/ < IBM PowerKVM went crazy here | 16:37 |
| openstackgerrit | Doug Hellmann proposed a change to openstack/requirements: Work around atexit bug in python < 2.7.4 https://review.openstack.org/89382 | 16:37 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Switch review.o.o to testmode https://review.openstack.org/89894 | 16:42 |
| jeblair | mordred, clarkb: can you review https://review.openstack.org/89894 so i can safely spin up a replacement review server? | 16:42 |
| mordred | jeblair: +2 | 16:43 |
| *** darraghb has quit IRC | 16:44 | |
| *** darraghb has joined #openstack-infra | 16:44 | |
| jeblair | i've lowered the ttls for review.o.o to 300 | 16:45 |
| *** etoews has joined #openstack-infra | 16:46 | |
| *** amcrn has joined #openstack-infra | 16:47 | |
| *** Longgeek_ has quit IRC | 16:47 | |
| krotscheck | sdague: You need to add chosen to the injector context. See http://paste.openstack.org/show/76805/ the line that reads ‘localiytics.directives’ | 16:47 |
| krotscheck | After you do that, there seems to be some kind of an initialization error that you should be able to trace. | 16:48 |
| *** mfer has quit IRC | 16:48 | |
| jeblair | mordred: i'm looking at the trove dbs in the ci account | 16:49 |
| jeblair | mordred: there seems to be gerrit-dev and gerrit entries, but i don't think they are used. right? | 16:50 |
| *** dprince has quit IRC | 16:50 | |
| *** blamar has joined #openstack-infra | 16:50 | |
| jeblair | mordred: etherpad says it's using 1.9 of 2GB. i suppose we should increase the disk space there. | 16:50 |
| jeblair | mordred: there are also a bunch of monitoring checks like cpu, load, memory, "mysql" (whatever that is). none of them have any data tohugh | 16:51 |
| openstackgerrit | A change was merged to openstack/requirements: Added argcomplete requirement https://review.openstack.org/65863 | 16:51 |
| jeblair | the one for storyboard does, so i wonder if it's a generational thing | 16:51 |
| jeblair | mordred: i propose to delete the gerrit-*-MySQL trove instances, and create a new one for production. what ram/disk values do you think we should use? | 16:53 |
| *** chandankumar has quit IRC | 16:53 | |
| jeblair | /var/lib/mysql on review.o.o is 2.1G | 16:54 |
| jeblair | so maybe 10G of disk? | 16:55 |
| jeblair | mordred: vmem of the mysql process is 3g with an rss of about 200m. 4g or 8g of ram? | 16:55 |
| nibalizer | krotscheck: i'm still struggling with getting my local storyboard up | 16:56 |
| nibalizer | but I'm at work now | 16:56 |
| jeblair | mordred: somehow the Gerrit-MySQL trove db is using 500MB | 16:56 |
| nibalizer | could we arrange a time to put our heads together on this, after 5pm pacifict time? | 16:56 |
| mordred | jeblair: hey - yeah - I do not believe that we're using them | 16:56 |
| mordred | jeblair: it's possible that we did an import of an old copy of the db as a test a while back | 16:57 |
| nibalizer | not sure what your timezone is, im flexible except for a giant block in the middle of the day I can't do openstack work | 16:57 |
| *** jooools has quit IRC | 16:57 | |
| mordred | jeblair: i _think_ 4G should be fine - and I agree on etherpad | 16:58 |
| *** yfried has joined #openstack-infra | 16:59 | |
| jeblair | etherpad is resizing now. i guess we'll see what that entails. :) | 16:59 |
| *** mriedem has quit IRC | 16:59 | |
| krotscheck | nibalizer: Pacifist? | 16:59 |
| krotscheck | Nice | 16:59 |
| jeblair | not a big deal, apparently, it's done and working. | 16:59 |
| mordred | woot | 16:59 |
| *** derekh has quit IRC | 17:02 | |
| nibalizer | krotscheck: pacific, sorry | 17:03 |
| *** amotoki_ has quit IRC | 17:03 | |
| mordred | jeblair, clarkb:fyi - support cases can be filed with hp cloud now at https://horizon.hpcloud.com/settings/cases/ | 17:03 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Move import checks into their own file https://review.openstack.org/89454 | 17:03 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Compile python 2-> regular expressions https://review.openstack.org/88441 | 17:03 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Move python 2->3 checks into their own module https://review.openstack.org/88440 | 17:03 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Speed up license checks https://review.openstack.org/86726 | 17:03 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Only compile regex for hacking_except_format once https://review.openstack.org/86695 | 17:03 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Move except checks into their own module https://review.openstack.org/86694 | 17:03 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Move docstring checks into their own file https://review.openstack.org/89713 | 17:03 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Move comment checks into their own module https://review.openstack.org/86693 | 17:03 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Make test_doctest use entry points https://review.openstack.org/86692 | 17:03 |
| * mordred throws a wet cat at jogo | 17:03 | |
| * jogo fixed code at the bottom of his branch | 17:03 | |
| jeblair | mordred: cool, thx | 17:04 |
| * jogo ducks | 17:04 | |
| anteaya | mnaser: thanks I will email them, have you seen any other instances of them doing this? I can't view their project page for some reason | 17:04 |
| * anteaya grabs her hat again | 17:04 | |
| mnaser | anteaya: i just saw that as i was glancing over reviews, first tiem i think | 17:04 |
| anteaya | mnaser: thanks | 17:04 |
| *** sacharya has joined #openstack-infra | 17:04 | |
| mtreinish | jeblair: do you know what's going on with the jobs stuck on the top of the check pipeline? | 17:05 |
| *** nati_ueno has joined #openstack-infra | 17:06 | |
| mtreinish | it looks like a handful of jobs have been sitting there as queued for hours | 17:06 |
| *** harlowja_away is now known as harlowja | 17:06 | |
| *** markmcclain has joined #openstack-infra | 17:07 | |
| *** SumitNaiksatam has quit IRC | 17:07 | |
| mordred | I agree with mtreinish | 17:07 |
| mordred | it does look like that | 17:07 |
| anteaya | mnaser: http://lists.openstack.org/pipermail/openstack-infra/2014-April/001191.html | 17:08 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add a fingerprint for bug 1311778 https://review.openstack.org/89891 | 17:08 |
| uvirtbot | Launchpad bug 1311778 in nova "Unit tests fail with MessagingTimeout errors" [Undecided,New] https://launchpad.net/bugs/1311778 | 17:08 |
| *** mriedem has joined #openstack-infra | 17:08 | |
| anteaya | mnaser: I hope they at least acknowledge the email within 4 hours | 17:09 |
| mtreinish | anteaya: I'll bug rfolco about it | 17:09 |
| mtreinish | but I just deployed a larger cloud behind the firewall and they were migrating the infra to the new one | 17:09 |
| mtreinish | so I'm sure it's just migration pains | 17:10 |
| clarkb | morning | 17:11 |
| jeblair | i think zuul lost its connection to its gearman server about 10 hours ago | 17:11 |
| *** Ajaeger1 has joined #openstack-infra | 17:11 | |
| anteaya | mtreinish: great thanks | 17:11 |
| anteaya | morning clarkb | 17:11 |
| zaro | drat got hp slave again!! | 17:11 |
| anteaya | zaro: you have the best luck | 17:11 |
| *** talluri has joined #openstack-infra | 17:11 | |
| anteaya | zaro: http://paste.openstack.org/show/76814/ | 17:11 |
| * mordred hands zaro a wet cat | 17:11 | |
| clarkb | jeblair: reviewing change for replacement gerrit server now | 17:12 |
| *** nati_uen_ has joined #openstack-infra | 17:12 | |
| clarkb | jeblair: I am going to try and address the comments in my stack of zuul changes today so that hopefully they can go in either before or during my absence | 17:12 |
| zaro | anteaya: maybe you should trigger to see if it makes a difference. | 17:12 |
| anteaya | zaro: so do I just delete lines 15 and 34, save the file and git add .; git commit? | 17:12 |
| *** amrith-is-awol is now known as amrith | 17:12 | |
| jeblair | clarkb: afaic, everything is secondary to the gerrit upgrade at this point. we still have a lot to do. | 17:13 |
| jeblair | so i might not get around to reviewing them before you leave | 17:13 |
| anteaya | zaro: what do you mean trigger it? | 17:13 |
| clarkb | jeblair: ok | 17:13 |
| *** markmcclain has quit IRC | 17:14 | |
| clarkb | jeblair: thats fine. Mostly I want to make sure that I don't let it idle on my side for so long that picking it up again is hard | 17:14 |
| zaro | anteaya: yep, that looks right. when you push do 'git review openstack/2.8.4' | 17:14 |
| mordred | wait | 17:14 |
| mordred | don't just delete lines 15 and 34 | 17:14 |
| clarkb | jeblair: https://review.openstack.org/#/c/89894/ has two +2's but I didn't approve because zaro has a comment that should at least be confirmed | 17:14 |
| zaro | anteaya: retrigger='recheck no bug' | 17:14 |
| mordred | you need to resolve the merge conflict | 17:14 |
| clarkb | jeblair: I say go ahead and approve it if my comment is correct | 17:15 |
| anteaya | zaro k I'll do that | 17:15 |
| *** nati_ueno has quit IRC | 17:15 | |
| jeblair | mtreinish: they won't run; the particular way in which zuul lost that job is a bug, i believe. you may be able to snipe them out with a new patchset. | 17:15 |
| anteaya | I have nothing on gerrit yet to recheck | 17:15 |
| mordred | anteaya: I _think_ you're talking about wanting to delete lines 15-20 and line 34 | 17:15 |
| *** gokrokv__ has quit IRC | 17:15 | |
| mordred | anteaya: so that the relevant section from your patch is the one used, and not the one from HEAD | 17:15 |
| jeblair | clarkb: yes, i forgot to mention in commit message | 17:15 |
| jeblair | ammending | 17:16 |
| mordred | anteaya: the ======== on line 20 is the separator between teh two versions | 17:16 |
| mtreinish | jeblair: ok I'll ask the author to push a new revision to retrigger the jobs then | 17:16 |
| mtreinish | thanks | 17:16 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Switch review.o.o to testmode https://review.openstack.org/89894 | 17:16 |
| jeblair | mordred, clarkb: ^ commit message changed | 17:16 |
| *** gokrokve has joined #openstack-infra | 17:17 | |
| mordred | jeblair: +2 | 17:17 |
| *** Ryan_Lane has joined #openstack-infra | 17:18 | |
| clarkb | approved | 17:18 |
| Ajaeger1 | infra team, is there anything I can do about the stuck openstack-manual jobs in the gate? | 17:18 |
| * Ajaeger1 already unapproved https://review.openstack.org/#/c/89683/ but zuul does not see it | 17:19 | |
| clarkb | unapproving doesn't removing things from the zuul pipelines | 17:19 |
| jeblair | it should; it's a bug that it doesn't. | 17:19 |
| clarkb | jeblair: is that related to the geard connection loss? | 17:19 |
| anteaya | mordred: no, I just need to add in the "the" on line 21 | 17:20 |
| anteaya | mordred: very small change | 17:20 |
| zaro | anteaya: i think you want to keep lines 16-19, just replace line 20 with an empty line | 17:20 |
| jeblair | Ajaeger1: i think the only option at this point is either a zuul restart, or pushing new patchsets to kick them out of the queue | 17:20 |
| jeblair | clarkb, mordred: do you think we should take the hit and restart zuul? | 17:20 |
| zaro | anteaya: then push it up for review and i'll double check. | 17:20 |
| clarkb | jeblair: I'm not sure I fully grok the issue yet | 17:21 |
| Ajaeger1 | jeblair: Ok, I'll push new push set | 17:21 |
| clarkb | it looks like most changes are moving | 17:21 |
| clarkb | but those are all newer | 17:21 |
| clarkb | things between 6 and 10 hours old are stuck? | 17:21 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Enabled functional tests https://review.openstack.org/88617 | 17:21 |
| jeblair | clarkb: the changes that have been in the queue for 10+ hours with queued jobs will never run because of the gear disconnection | 17:21 |
| clarkb | gotcha | 17:21 |
| clarkb | in that case maybe we should take the hit | 17:21 |
| clarkb | and restart zuul | 17:21 |
| jeblair | ok will do. i'll mark all the nodepool nodes for deletion while it's down so hopefully we'll recover faster. | 17:22 |
| Ajaeger1 | jeblair, clarkb there're only two for openstack-manuals - I can push updates for both of them... | 17:22 |
| jeblair | Ajaeger1, mtreinish: don't worry about uploading new patchsets | 17:22 |
| jeblair | oh | 17:22 |
| Ajaeger1 | jeblair: done - and the queue is moving forward! | 17:22 |
| jeblair | ok nevermind then. :) | 17:22 |
| Ajaeger1 | jeblair: sorry, i was too fast | 17:22 |
| *** gokrokve has quit IRC | 17:23 | |
| jeblair | mtreinish: cancel my last. :) | 17:23 |
| anteaya | zaro: that looks wrong to me: https://review.openstack.org/#/c/89899/1 | 17:24 |
| mtreinish | jeblair: heh, ok :) | 17:24 |
| anteaya | mordred might be correct | 17:24 |
| SpamapS | mordred: so elements/source-repositories/extra-data.d/98-source-repositories is the script you probably want to inspect | 17:24 |
| SpamapS | doh | 17:24 |
| anteaya | and I have too many change ids | 17:24 |
| anteaya | I'll get rid of the middle one | 17:24 |
| *** praneshp has joined #openstack-infra | 17:25 | |
| zaro | anteaya: remove the conflicts infro as well. make sure the change id matches the one from your upstream change. | 17:25 |
| anteaya | oh really | 17:26 |
| anteaya | okay | 17:26 |
| zaro | that's odd, no msg to openstack-infra channel on your new patchset | 17:26 |
| *** marktraceur has quit IRC | 17:26 | |
| zaro | anteaya: you think you want to fix that? | 17:27 |
| anteaya | https://review.openstack.org/#/c/89900/ | 17:27 |
| anteaya | I'll abandon the other | 17:27 |
| anteaya | it might not be tracking this branch | 17:28 |
| rfolco | anteaya, just turned CI off until we resolve the issue. Sorry for the inconvenience. | 17:29 |
| *** etoews has quit IRC | 17:29 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Support configurable mysql host in gerrit https://review.openstack.org/89901 | 17:29 |
| anteaya | rfolco: thanks for the quick response | 17:29 |
| *** marktraceur has joined #openstack-infra | 17:29 | |
| anteaya | rfolco: was it just that one patch? I can't see the project page for that account for some reason | 17:29 |
| anteaya | mnaser: ^ | 17:29 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Support configurable mysql host in gerrit https://review.openstack.org/89901 | 17:30 |
| mnaser | anteaya / rfolco: i haven't really done a lot of reviews today... but spotted that one | 17:30 |
| *** sweston has joined #openstack-infra | 17:30 | |
| zaro | anteaya: you could have just uploaded a new patch to 89900 instead of abandoning it. but what you did is ok as well | 17:30 |
| *** SumitNaiksatam has joined #openstack-infra | 17:31 | |
| anteaya | zaro: the change id created a new patch | 17:31 |
| jeblair | clarkb, mordred, zaro: when https://review.openstack.org/89901 lands i'll go ahead and move review-dev to trove | 17:31 |
| anteaya | the system wanted a change id at the bottom, I didn't know I needed to keep the same change id as upstream | 17:31 |
| clarkb | jeblair: ok | 17:31 |
| mordred | jeblair: +2 | 17:31 |
| *** Sukhdev has quit IRC | 17:32 | |
| anteaya | zaro: so I did just upload a new patchset, but gerrit saw the new change id as the bottom one and created a new patch | 17:32 |
| zaro | anteaya: ohh ok, that makes sense. | 17:32 |
| anteaya | sub optimal but now I know | 17:32 |
| rfolco | anteaya, not sure, hopefully didn't cause much damage. I'll fix the problem with SSL certs first and make sure everything is good before reporting again. | 17:33 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added story status to project story list. https://review.openstack.org/86719 | 17:33 |
| *** marun has joined #openstack-infra | 17:34 | |
| anteaya | zaro: so if we compare https://review.openstack.org/#/c/89699/1/Documentation/access-control.txt to my patch, mordred is correct and I should get rid of lines 13 - 17 | 17:34 |
| anteaya | rfolco: /nod | 17:34 |
| *** darraghb has quit IRC | 17:34 | |
| mgagne | could someone help me find the root cause of this error? http://logs.openstack.org/10/87810/4/check/gate-jenkins-job-builder-python33/56bf1af/console.html#_2014-04-23_03_40_46_013 | 17:34 |
| anteaya | mgagne: hmmm, yes not a lot to go on | 17:36 |
| krotscheck | jeblair: ^ | 17:36 |
| *** marktraceur has quit IRC | 17:36 | |
| anteaya | mgagne: found this: https://bugs.launchpad.net/testtools/+bug/1271133 | 17:38 |
| uvirtbot | Launchpad bug 1271133 in testrepository "Cryptic error from subunit when an import fails" [High,Fix committed] | 17:38 |
| mgagne | anteaya: all I found when I got this error in an other project of mine is a bad combination of python-subunit and testtools... | 17:38 |
| mgagne | anteaya: not sure how other projects are successfully testing against py33... | 17:38 |
| *** chandan_kumar has joined #openstack-infra | 17:39 | |
| anteaya | mgagne: my trail lead me here: http://bugs.python.org/issue19746 | 17:40 |
| anteaya | yes I have no status on py33 testing across projects | 17:40 |
| anteaya | hopefully that info and a place to track it will be discussed at the py3 summit talk | 17:40 |
| mgagne | anteaya: Marc and I are about to start with JJB | 17:41 |
| mgagne | anteaya: but having the test failing to initialize itself is kind of a blocker =) | 17:42 |
| *** ominakov has joined #openstack-infra | 17:42 | |
| zaro | jeblair: trove, cool! | 17:42 |
| anteaya | mgagne: makes sense to me | 17:42 |
| anteaya | mgagne: all roads point to lifeless as far as I can see | 17:43 |
| anteaya | any chance you can overlap timezones with him for a chat? | 17:43 |
| zaro | mgagne: are you able to repro on any machine? | 17:43 |
| mgagne | zaro: will test on mine | 17:43 |
| *** markmc has quit IRC | 17:44 | |
| anteaya | zaro: so what do you think? remove lines 13 - 17 on my patch? | 17:44 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Moved main navigation into fixed sidebar. https://review.openstack.org/88460 | 17:44 |
| *** jgallard has quit IRC | 17:44 | |
| *** marktraceur has joined #openstack-infra | 17:44 | |
| zaro | anteaya: i've reviewed already i vote -2 | 17:44 |
| zaro | anteaya: sorry :( | 17:45 |
| anteaya | yeah | 17:45 |
| anteaya | well I learned a ton | 17:45 |
| anteaya | will abandon | 17:45 |
| *** asselin2 has quit IRC | 17:46 | |
| zaro | anteaya: at least we got something out of that :) | 17:46 |
| anteaya | we did | 17:46 |
| mgagne | zaro: looks like ConfigParser can't be imported. I used tox -epy33 to run the tests. AFAIK, jenkins uses testr instead. | 17:46 |
| jbryce | i have a random question for anyone who’s interested in providing a random answer | 17:46 |
| anteaya | I got a Gerrit upstream patch in, never thought that was possible | 17:46 |
| anteaya | jbryce: 42 | 17:46 |
| jbryce | anteaya: i figured someone would throw that out = ) | 17:47 |
| anteaya | wet cats | 17:47 |
| anteaya | I've got more | 17:47 |
| anteaya | :D | 17:47 |
| mgagne | http://xkcd.com/221/ =) | 17:47 |
| anteaya | mgagne: ha ha ha | 17:47 |
| zaro | mgagne: not supported on py33? | 17:47 |
| mgagne | zaro: got renamed to configparser I think | 17:47 |
| mgagne | zaro: I trashed my venv and started the tests again | 17:48 |
| mgagne | zaro: I now have the same output as Jenkins | 17:48 |
| anteaya | jbryce: go ahead ask all the random questions you want, didn't mean to cut you off | 17:48 |
| mgagne | zaro: I probably downgraded subunit and testtools earlier | 17:48 |
| jbryce | in addition to official tc-accepted projects, is there a list of other projects that make use of openstack ci anywhere? | 17:48 |
| *** morganfainberg_Z is now known as morganfainberg | 17:48 | |
| anteaya | that make use of openstack ci, as in goes through our testing pipelines? | 17:49 |
| zaro | mgagne: does the test even run? | 17:49 |
| mgagne | zaro: no | 17:50 |
| anteaya | jbryce: I think that would be covered here: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml | 17:50 |
| zaro | jbryce: i think wikimedia uses openstack ci | 17:50 |
| *** marktraceur has quit IRC | 17:50 | |
| mgagne | zaro: marc already fixed this import issue by introducing a compat helper: https://review.openstack.org/#/c/87810/4 | 17:51 |
| mgagne | zaro: tests are however still failing | 17:51 |
| anteaya | jbryce: the projects start here: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n574 | 17:51 |
| mgagne | zaro: so it's not just it | 17:51 |
| jeblair | jbryce: the full list of projects in gerrit is visible here: https://review.openstack.org/#/admin/projects/ or here https://git.openstack.org/cgit | 17:51 |
| zaro | mgagne: run but failing or not running at all? | 17:51 |
| mgagne | zaro: not running | 17:51 |
| mordred | jbryce: do you mean use the openstack instance of infra? or the tools? | 17:51 |
| *** markmcclain has joined #openstack-infra | 17:52 | |
| mordred | jbryce: because as zaro mentions, we have other projects, like wikimedia, running copies of many of them - and then jeblair is pointing out an excellent way to see the list of things that are in our copy | 17:52 |
| zaro | mgagne: like is it failing to compile (syntax check)? | 17:52 |
| mgagne | zaro: it's the error I pasted above | 17:53 |
| *** marktraceur has joined #openstack-infra | 17:53 | |
| mgagne | zaro: http://logs.openstack.org/10/87810/4/check/gate-jenkins-job-builder-python33/56bf1af/console.html#_2014-04-23_03_40_46_013 | 17:53 |
| mgagne | zaro: I retesting Marc's change on my machine with previous versions of subunit/testtools | 17:54 |
| jbryce | mordred: both | 17:54 |
| *** asselin2 has joined #openstack-infra | 17:54 | |
| mordred | jbryce: j. well, jim's list is for the second - I'm not sure we have a comprehensive list for the first | 17:54 |
| jeblair | the usual open source problem of not knowing who's using your software | 17:54 |
| jbryce | mordred: anecdotal is totally fine | 17:55 |
| jeblair | if random support requests are any indication, many openstack related companies are running much of this infra internally. several non-openstack projects/organizations are as well. | 17:55 |
| jeblair | and some of those are completely proprietary | 17:55 |
| *** vhoward has left #openstack-infra | 17:56 | |
| *** marktraceur has quit IRC | 17:56 | |
| jbryce | thanks all | 17:59 |
| *** asselin2 has quit IRC | 17:59 | |
| pleia2 | Utopic Unicorn \o/ | 18:00 |
| anteaya | \o/ | 18:01 |
| anteaya | the graphics team will have been waiting for a shot at this wallpaper for some time | 18:02 |
| *** ildikov has joined #openstack-infra | 18:02 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Containers now left-align https://review.openstack.org/88630 | 18:03 |
| pleia2 | anteaya: HEHE | 18:03 |
| pleia2 | er | 18:03 |
| pleia2 | hehe | 18:03 |
| pleia2 | woo capslock | 18:03 |
| *** mugsie has quit IRC | 18:03 | |
| *** dprince has joined #openstack-infra | 18:03 | |
| *** mugsie has joined #openstack-infra | 18:04 | |
| jeblair | clarkb: when you have a sec, i'm ready for https://review.openstack.org/#/c/89901/ | 18:05 |
| mordred | pleia2: oh wow | 18:05 |
| pleia2 | best name ever | 18:05 |
| pleia2 | especially since a pink dragon with a tutu sits on my desk, I call her zuul, she can now hang out with a unicorn | 18:07 |
| anteaya | ha ha ha | 18:07 |
| *** e0ne has joined #openstack-infra | 18:07 | |
| clarkb | jeblair: looking | 18:09 |
| anteaya | jeblair: https://review.openstack.org/#/c/89901/2/modules/openstack_project/manifests/gerrit.pp line 8 mysql_password stays? | 18:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch review.o.o to testmode https://review.openstack.org/89894 | 18:09 |
| jeblair | anteaya: yep, it just becomes the password to the trove db instead of the local mysql db. | 18:10 |
| anteaya | ah okay | 18:10 |
| jeblair | oh | 18:10 |
| jeblair | well, that's true for review-dev, and will be true for review on monday. | 18:10 |
| mordred | jeblair: ^^ see pleia2's comment- I believe we've just learned the the mascot for zuul is a pink dragon | 18:10 |
| jeblair | anteaya: until then it remains the password for the local db on review. | 18:10 |
| jbryce | so i count 265 projects in gerrit…does that sound about right? | 18:11 |
| anteaya | okay | 18:11 |
| anteaya | jbryce: it does to me | 18:11 |
| mordred | yup | 18:11 |
| mordred | mordred@camelot:~/src/openstack-infra/config$ ssh -p 29418 review.openstack.org gerrit ls-projects | wc -l | 18:11 |
| mordred | 265 | 18:11 |
| clarkb | jeblair: do you want to control when 89901 is approved? | 18:11 |
| *** Sukhdev has joined #openstack-infra | 18:11 | |
| jeblair | pleia2: that's wonderful! | 18:12 |
| jeblair | clarkb: go for it | 18:12 |
| adalbas | hello! One question regarding the tempest full run in gate: Is there a reason to not stop the run at the first failure, instead of running all the testcases everytime. | 18:12 |
| clarkb | adalbas: yes, to catch more failures | 18:13 |
| mordred | adalbas: if it fails in several places, it might be useful information for the developer | 18:13 |
| clarkb | jeblair: done | 18:13 |
| *** e0ne has quit IRC | 18:14 | |
| adalbas | clarkb, mordred : alright, i want to confirm this was already been discussed. I see the good point of doing so is to have less timing consuming jobs, but i agree on getting more info. | 18:15 |
| clarkb | adalbas: it is more time consuming to fail fast if you consider that several runs may be needed to shake out all fails | 18:15 |
| *** jaypipes has quit IRC | 18:15 | |
| sweston | I have the first round of documentation on for Gerrit 2.8 upgrade, and new features on Etherpad, if anyone would like to have a look? https://etherpad.openstack.org/p/Gerrit2.8Docs https://etherpad.openstack.org/p/Gerrit2.8Docs1 | 18:16 |
| adalbas | clarkb, yes, good point! | 18:16 |
| *** gokrokve has joined #openstack-infra | 18:16 | |
| zaro | jbryce: yes | 18:16 |
| anteaya | sweston split them into two etherpads due to the size, over 1500 lines over the two files | 18:17 |
| anteaya | I have to run errands and will be back in a few hours | 18:17 |
| anteaya | thanks sweston! | 18:17 |
| sweston | anteaya: sure, thanks for yours! | 18:18 |
| *** sdake_ has joined #openstack-infra | 18:18 | |
| openstackgerrit | Mathew Odden proposed a change to openstack-infra/config: Create bash8 project in openstack-dev https://review.openstack.org/88747 | 18:18 |
| *** jerryz_ has joined #openstack-infra | 18:19 | |
| jeblair | mordred: gear was downloaded from pypi 2x as much as jjb last month | 18:20 |
| mordred | jeblair: nice! | 18:20 |
| mordred | jbryce: ^^ | 18:20 |
| *** jerryz has quit IRC | 18:20 | |
| mordred | jeblair: do we auto pip install either of them in any of our things? I don't think we do - so those are probably real downloads, right? | 18:20 |
| *** gokrokve has quit IRC | 18:21 | |
| jeblair | mordred: i think zuul may; but i still don't think that should affect the numbers | 18:22 |
| jeblair | oh, zuul unit tests might | 18:22 |
| jeblair | so maybe knock a few hundred off for that | 18:23 |
| mordred | oh right - because we don't mirror for infra | 18:23 |
| zaro | sweston: so i don't understand the intent of the doc. can you xplain? | 18:25 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/jenkins-job-builder: Update authorship information https://review.openstack.org/89912 | 18:26 |
| sweston | zaro: so the purpose was to introduce the most important features that folks would be concerned about/interested in. There is still a lot of work to be done, it is very rough right now .. but everything after the feature list was copied and pasted -- it is meant to be removed ... | 18:27 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Collate story metadata for status https://review.openstack.org/86452 | 18:27 |
| *** e0ne has joined #openstack-infra | 18:28 | |
| *** blamar has quit IRC | 18:28 | |
| *** blamar has joined #openstack-infra | 18:29 | |
| *** ominakov has quit IRC | 18:29 | |
| jeblair | zaro, sweston: i think the main things we need to focus on are the parts of existing workflows that are changing (eg, the workflow label, important changes, ssh keys) | 18:30 |
| jeblair | zaro, sweston: then i think there's some room to highlight some particularly interesting new features; but probably not too many | 18:30 |
| *** mdenny has quit IRC | 18:31 | |
| sweston | jeblair: ok, how long would you like it to be, then? | 18:31 |
| jeblair | zaro, sweston: basically -- we're about to change how 1500+ people interact with this system. they are busy -- most of them just need to know how to do what they were doing. a few will be interested in exploring new things. | 18:31 |
| *** mdenny has joined #openstack-infra | 18:32 | |
| zaro | +1 that | 18:32 |
| openstackgerrit | A change was merged to openstack-infra/config: Support configurable mysql host in gerrit https://review.openstack.org/89901 | 18:33 |
| jeblair | sweston: the parts about workflow changes should be as long as they need to be to explain the changes (i would expect that's on the order of a small number of paragraphs each, or less) | 18:33 |
| sweston | jeblair: ok, that will be easy enough :-) | 18:33 |
| *** blamar has quit IRC | 18:33 | |
| jeblair | sweston: for new features we should highlight just a handful in a bullet list | 18:33 |
| zaro | sweston: well anyways i ignored the entire feature section. best to provide a link there i think. just point out the items that jeblair wrote in his upgrade reminder msg. | 18:34 |
| *** etoews has joined #openstack-infra | 18:34 | |
| zaro | sweston: i think one thing we need to add is that the 'important changes' dashboard link has moved into the "Projects" menu | 18:35 |
| *** e0ne has quit IRC | 18:35 | |
| sdague | jeblair: is there any way to kick the "lost" jobs in zuul to run again? | 18:35 |
| sdague | or do we need to just upload over them | 18:35 |
| jeblair | sdague: the ones stuck for 10+ hours? | 18:35 |
| sdague | yep | 18:35 |
| jeblair | sdague: upload over | 18:35 |
| sweston | zaro, jeblair: ok, will do! | 18:37 |
| *** pblaho has joined #openstack-infra | 18:37 | |
| sdague | is this an edge case in zuul that it's possible to recover from in the future? post summit I was planning to work on the Depends-On: so could sling around here some as well | 18:37 |
| jeblair | sdague: i believe there's a bug in that zuul should have known that the job was not submitted to gearman and marked it as such | 18:38 |
| jeblair | (i mean, it recorded the traceback indicating that, it just didn't give the job a status) | 18:39 |
| *** chandan_kumar has quit IRC | 18:41 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added story status to project story list. https://review.openstack.org/86719 | 18:42 |
| *** hashar has joined #openstack-infra | 18:42 | |
| sdague | yeh, it's times like this when gerrit detecting that no change was made is less fun :) | 18:42 |
| *** gokrokve has joined #openstack-infra | 18:45 | |
| *** lukego has quit IRC | 18:46 | |
| *** _nadya_ has joined #openstack-infra | 18:47 | |
| *** pdmars has quit IRC | 18:47 | |
| *** lukego has joined #openstack-infra | 18:47 | |
| *** lukego has quit IRC | 18:49 | |
| *** talluri has quit IRC | 18:49 | |
| *** markmcclain1 has joined #openstack-infra | 18:49 | |
| *** markmcclain has quit IRC | 18:50 | |
| *** mfer has joined #openstack-infra | 18:51 | |
| jeblair | mordred: /usr/bin/mysqldump --defaults-file=/etc/mysql/debian.cnf --opt reviewdb > reviewdb.sql | 18:53 |
| jeblair | mordred: lgty? | 18:53 |
| jeblair | (for dump/import into trove) | 18:53 |
| *** lukego has joined #openstack-infra | 18:54 | |
| *** lukego has quit IRC | 18:54 | |
| *** lukego has joined #openstack-infra | 18:55 | |
| morganfainberg | oooh mysqldump | 18:57 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Moved main navigation into fixed sidebar. https://review.openstack.org/88460 | 18:57 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Containers now left-align https://review.openstack.org/88630 | 18:57 |
| morganfainberg | jeblair, i _think_ --opt is the default (though doesn't hurt to specify) | 18:59 |
| jeblair | ah good | 18:59 |
| adam_g | 2014-04-23 01:23:26.994 | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 264: bc: command not found | 19:00 |
| adam_g | ^ is this a known issue ATM? | 19:00 |
| jeblair | sdague: ^ whoops | 19:00 |
| jeblair | adam_g: first i've heard of it, but i believe i know the recent change that caused it | 19:01 |
| *** lukego has joined #openstack-infra | 19:01 | |
| *** amrith has left #openstack-infra | 19:01 | |
| jeblair | i suspect this is a result of different providers using different images (i really can't wait to use dib) | 19:02 |
| adam_g | jeblair, yeah https://review.openstack.org/#/c/89381/. i see lots of occurences in longstash | 19:02 |
| *** lukego has quit IRC | 19:02 | |
| *** mrodden has quit IRC | 19:03 | |
| *** changbl has joined #openstack-infra | 19:03 | |
| *** gothicmindfood has quit IRC | 19:03 | |
| *** mrodden has joined #openstack-infra | 19:04 | |
| *** markmcclain1 has quit IRC | 19:04 | |
| sdague | oh, man, we don't have bc :) | 19:05 |
| *** lukego has joined #openstack-infra | 19:05 | |
| sdague | ok, there is a way to do math in bash right? | 19:05 |
| *** lukego has quit IRC | 19:05 | |
| sdague | yeh, that's the slow devstack detection code | 19:05 |
| adam_g | $[x + y] :) | 19:06 |
| jeblair | yeah, looks like the rax images don't have it | 19:07 |
| *** nati_uen_ has quit IRC | 19:08 | |
| sdague | ok, I'll get a fix shortly | 19:08 |
| adam_g | jeblair, the only packages that get added to nodepool images are those pulled from devstack's apt files, right? | 19:09 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: change time calculation to not require bc https://review.openstack.org/89920 | 19:11 |
| jeblair | adam_g: those are cached and then devstack installs them; some other packages are installed on the image before that though -- the ones listed in our puppet manifest for slave nodes | 19:12 |
| adam_g | jeblair, ah, right. thats what i was looking for | 19:12 |
| jeblair | hypothetically if we wanted to fix this that way, puppet would be the place to add bc i think | 19:13 |
| sdague | adam_g: that should fix it | 19:13 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove queries for fixed bugs https://review.openstack.org/88465 | 19:13 |
| jeblair | though i like the non-bc way better | 19:13 |
| sdague | yeh, I forget that bash can do real math | 19:13 |
| adam_g | okay | 19:13 |
| sdague | so used to using bc for that stuff | 19:13 |
| *** nati_ueno has joined #openstack-infra | 19:14 | |
| *** pdmars has joined #openstack-infra | 19:14 | |
| jeblair | sdague: a computer without bc feels like a computer that can't do math. it's weird. :) | 19:14 |
| *** lukego has joined #openstack-infra | 19:14 | |
| *** pdmars has quit IRC | 19:15 | |
| sdague | yeh :) | 19:15 |
| *** pdmars has joined #openstack-infra | 19:15 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Collate story metadata for status https://review.openstack.org/86452 | 19:15 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Support email lists in require-approval https://review.openstack.org/88616 | 19:16 |
| *** talluri has joined #openstack-infra | 19:20 | |
| jeblair | restarting review-dev | 19:23 |
| *** talluri has quit IRC | 19:24 | |
| jeblair | zaro, clarkb, mordred: review-dev is running against a trove db now | 19:24 |
| *** e0ne has joined #openstack-infra | 19:26 | |
| Ajaeger1 | infra team, the propose-requirements-update job failed - see https://jenkins.openstack.org/job/propose-requirements-updates/lastBuild/consoleText | 19:27 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Remove require mysql::db from gerrit init https://review.openstack.org/89921 | 19:28 |
| Ajaeger1 | Seems there's a problem pushing to cliff | 19:28 |
| jeblair | clarkb, mordred: https://review.openstack.org/89921 | 19:28 |
| jeblair | Ajaeger1: wow. how do you find the error in there? | 19:29 |
| Ajaeger1 | jeblair: searched for "error" ;) | 19:29 |
| jeblair | ha | 19:30 |
| *** jaypipes has joined #openstack-infra | 19:30 | |
| * Ajaeger1 checked for failures in post-jobs on jenkins.openstack.org - wanted to check that translations work still fine... | 19:30 | |
| *** SumitNaiksatam has quit IRC | 19:30 | |
| jeblair | ! [remote rejected] HEAD -> refs/publish/master/openstack/requirements (no changes made) | 19:30 |
| *** nati_ueno has quit IRC | 19:30 | |
| Ajaeger1 | ok, that can be ignored. | 19:31 |
| clarkb | yup iirc we handle that properly now but still report failure because it did fail | 19:31 |
| Ajaeger1 | ah, ok... | 19:32 |
| jeblair | clarkb: so failure == everything's okay? | 19:32 |
| Ajaeger1 | in that case everything looks fine to me - just confusing ;( | 19:32 |
| clarkb | jeblair: not entirely | 19:32 |
| clarkb | jeblair: ideally we would be able to check that particular "failure" and ignore it | 19:32 |
| clarkb | jeblair: iirc git + gerrit make this hard | 19:33 |
| *** ominakov has joined #openstack-infra | 19:33 | |
| clarkb | actually I want to say that at one time we grepped for no changes made | 19:33 |
| clarkb | did that get ripped out? /me looks at git log | 19:33 |
| clarkb | or maybe that change was proposed and died | 19:33 |
| *** Ryan_Lane1 has joined #openstack-infra | 19:34 | |
| clarkb | "$OUTPUT" =~ "no new changes" is what we check | 19:34 |
| clarkb | was that just the wrong string or does gerrit do multiple strings? | 19:34 |
| mrodden | i feel like the config-compare-xml should be saying there are changes since the new project should have new python jobs? https://review.openstack.org/#/c/88747/4 | 19:34 |
| *** markmcclain has joined #openstack-infra | 19:35 | |
| clarkb | mrodden: it doesn't diff new files only changes to existing files | 19:35 |
| jeblair | clarkb: multiple | 19:35 |
| mrodden | oh ok | 19:35 |
| clarkb | jeblair: gah ok, I will update that script to handle this othe message too | 19:35 |
| jeblair | clarkb: "no new changes" is "you pushed the same commit (sha) already" | 19:35 |
| *** e0ne has quit IRC | 19:35 | |
| jeblair | clarkb: "no changes made" is "you pushed an identical commit" | 19:35 |
| *** Ryan_Lane has quit IRC | 19:35 | |
| jeblair | https://review.openstack.org/Documentation/error-no-new-changes.html https://review.openstack.org/Documentation/error-no-changes-made.html | 19:36 |
| jeblair | really good docs, actually :) | 19:36 |
| Ajaeger1 | clarkb: will you fix it? Otherwise I'll give it a try. | 19:39 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Handle additional reqs push failure as non failing https://review.openstack.org/89924 | 19:39 |
| clarkb | Ajaeger1: jeblair ^ that should fix it | 19:39 |
| Ajaeger1 | clarkb: you're too quick for me ;) - will review it. Thanks! | 19:40 |
| *** julim has joined #openstack-infra | 19:41 | |
| *** ominakov has quit IRC | 19:41 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added status filter to story API https://review.openstack.org/87410 | 19:43 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Expose assignee_id to API https://review.openstack.org/88676 | 19:46 |
| jeblair | clarkb: did you see the conversation about a cinder volume for review.o.o earlier? | 19:47 |
| jeblair | (the new review, that is) | 19:47 |
| jeblair | clarkb: also, specs for new review? | 19:48 |
| jeblair | clarkb: was thinking 30g performance2 node to maintain the cpu/ram levels we currently have (but should see a modest performance improvement due to the cpu upgrade) | 19:49 |
| *** _nadya_ has quit IRC | 19:49 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove require mysql::db from gerrit init https://review.openstack.org/89921 | 19:50 |
| *** timrc is now known as timrc-afk | 19:50 | |
| *** timrc-afk is now known as timrc | 19:52 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add launchpadlib timeout https://review.openstack.org/89672 | 19:52 |
| *** mrmartin has joined #openstack-infra | 19:54 | |
| jeblair | has anyone tried running install_puppet.sh on trusty yet? | 19:54 |
| jeblair | i get E: Package 'rubygems' has no installation candidate | 19:55 |
| sdague | jeblair: gem was baked into ruby in 1.9 | 19:56 |
| sdague | and 1.8 is no longer on trusty | 19:56 |
| jeblair | yeah, that's the conclusion i was coming to; so we probably just drop that package on trusty.... | 19:57 |
| jeblair | i'm a little worried the next thing i'll find is that puppet2 doesn't work with 1.9 or something | 19:57 |
| sdague | yeh, that was the only change I need to make to my base puppet policy on all my personal machines | 19:57 |
| sdague | that's entirely possible | 19:57 |
| sdague | I flipped to puppet 3 with 13.10 iirc | 19:57 |
| jeblair | sdague: yeah, nibalizer has just started taking the first step to upgrade us to puppet3 | 19:58 |
| sdague | the trusty shipped puppet is 3.4.3 | 19:58 |
| jeblair | i anticipate it will take most of the juno cycle to upgrade | 19:58 |
| *** melwitt has joined #openstack-infra | 19:59 | |
| sdague | is this a nodepool concern, or a wider concern? | 19:59 |
| jeblair | sdague: immediate concern is for the new gerrit server | 20:00 |
| sdague | gotcha | 20:00 |
| jeblair | fallback is we keep using precise there, but upgrading it is a _chore_ so i'd like to try to avoid it. | 20:00 |
| nibalizer | jeblair: sdague yea we should probably cook up an etherpad to collect the info/plans for puppet 3 | 20:01 |
| sdague | I only tripped over a couple of things when I did my policy conversion. That was much smaller than infra's, but it wasn't actually that bad. | 20:03 |
| sdague | nibalizer: what's the prescribed method you recommend for going through | 20:03 |
| jeblair | hrm, for some reason i've been lead to believe infra will be complicated | 20:04 |
| mordred | I dont think it's going to be that bad, based on poking at nodepool recently | 20:04 |
| jeblair | sdague: https://review.openstack.org/#/c/83678/ | 20:04 |
| mordred | the biggest challege is actually more related to supporting trusty - puppet3 wasn't really a problem | 20:04 |
| sdague | I think the real question is if some of the puppetforge modules that are really old need an upgrade as well in the process | 20:04 |
| lifeless | morning | 20:04 |
| jeblair | sdague: i assume the process might involve host-by-host fixes and testing them against the alternate master | 20:04 |
| nibalizer | ya we need to get a p3 master up | 20:05 |
| nibalizer | then go through fixing individual nodes | 20:05 |
| nibalizer | so that means a fair ammount of work from the infra rooters to test against the p3 master | 20:05 |
| *** homeless has quit IRC | 20:05 | |
| nibalizer | it also may be needed to have a long-lived and often-rebased branch with all the noncompatible puppet3 changes | 20:05 |
| jeblair | mordred: can you elaborate on the 'supporting trusty' bit? | 20:05 |
| lifeless | jeblair: what are your thoughts on trusty jenkins slaves ? | 20:06 |
| lifeless | hah, jinx | 20:06 |
| mordred | jeblair: package renames | 20:06 |
| jeblair | lifeless: ask mordred | 20:06 |
| *** nati_ueno has joined #openstack-infra | 20:06 | |
| mordred | jeblair: things that work different | 20:06 |
| mordred | lemme point at something... | 20:06 |
| lifeless | so my interest is the tripleo slaves specifically | 20:06 |
| mordred | hrm. my patches are on a server - I have lke 3 lines of patch it takes | 20:07 |
| mordred | to run puppet3 on a trusty node with our slave stuff | 20:07 |
| sdague | yeh, I would expect it's just a couple of package removes | 20:08 |
| jeblair | mordred: do you expect our server manifests to work on trusty? if not, what do you think needs to be changed? | 20:08 |
| mordred | jeblair: I do, acually | 20:08 |
| jeblair | mordred: if running new review.o.o on trusty is a huge project, i want to avoid it, but if it's just a matter of fixing a few things we should do it | 20:08 |
| mordred | I expect there to be unforseen issues - such as the name of a java package being renamed | 20:08 |
| *** mrmartin has quit IRC | 20:09 | |
| jeblair | lifeless: morderd is working on adding dib support to nodepool, then we can build and use consistent images everywhere | 20:09 |
| *** pblaho has quit IRC | 20:09 | |
| *** _nadya_ has joined #openstack-infra | 20:09 | |
| jeblair | puppet agent --environment production --server ci-puppetmaster.openstack.org --no-daemonize --verbose --onetime --pluginsync true --certname review.openstack.org | 20:10 |
| jeblair | Notice: Skipping run of Puppet configuration client; administratively disabled (Reason: 'Disabled by default on new installations'); | 20:10 |
| jeblair | Use 'puppet agent --enable' to re-enable. | 20:10 |
| jeblair | that's exciting. | 20:10 |
| *** andreaf has quit IRC | 20:11 | |
| jeblair | so, um, i guess i'll do what it says? | 20:11 |
| *** nati_ueno has quit IRC | 20:11 | |
| *** Sukhdev has quit IRC | 20:11 | |
| mordred | jeblair: https://review.openstack.org/#/c/88477/ | 20:11 |
| mordred | jeblair: yes. I'd do that | 20:12 |
| *** marun has quit IRC | 20:12 | |
| *** marun has joined #openstack-infra | 20:12 | |
| jeblair | mordred: your change looks like it does trusty and puppet3 at once | 20:13 |
| mordred | jeblair: yeah - the focus of that patch was getting nodepool able to launch trusty slaves | 20:14 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Changes to get nodepool working with trusty https://review.openstack.org/88477 | 20:15 |
| mordred | (that fixes the puppet lint change) | 20:15 |
| jeblair | mordred: okay, but why the puppet3 change at the same time? | 20:15 |
| *** Ajaeger1 has quit IRC | 20:15 | |
| mordred | because trusty has puppet3 baked in already and I don't think there was a puppet2 apt repo from puppetlabs for trusty when I tried it? | 20:15 |
| jeblair | mordred: oh, there seems to be now, at least, it's getting well past the install puppet phase | 20:16 |
| mordred | gotcha | 20:16 |
| mordred | well - can certainly split it into two patches | 20:16 |
| jeblair | mordred: i think we should separate the concerns, otherwise if we merge your patch, we'll have to move new-review to puppet3 if we put it on trusty) | 20:16 |
| mordred | ++ | 20:16 |
| mordred | if there's a trusty puppet2 now, that makes it much easier | 20:17 |
| jeblair | sdague, mordred: $java_home = $::lsbdistcodename ? { | 20:17 |
| jeblair | 'precise' => '/usr/lib/jvm/java-7-openjdk-amd64/jre', | 20:17 |
| jeblair | sdague, mordred: have a guess as to what that should be on trusty? | 20:17 |
| jeblair | (tbh, i'm not sure why that is there -- it's the only value) | 20:18 |
| sdague | jeblair: that dir is on my trusty boxes | 20:18 |
| *** primeministerp has quit IRC | 20:18 | |
| jeblair | okay, i'll just dup it for now | 20:18 |
| sdague | I actually just finished upgrading all my boxes to trusty about 15 minutes ago | 20:19 |
| sdague | so ask away on questions :) | 20:19 |
| jeblair | woo! | 20:19 |
| mordred | sdague: did you do puppet3 too? | 20:19 |
| *** primeministerp has joined #openstack-infra | 20:19 | |
| sdague | yeh | 20:20 |
| sdague | but I think I took that update during the 13.10 upgrade | 20:20 |
| mordred | jeblair: nibalizer's suggested process was "spin up new puppetmaster running puppet3, run them in paralle, point one host at a time at new puppetmaster" right? | 20:20 |
| sdague | because I'm using puppet from the distro | 20:20 |
| jeblair | mordred: i believe so | 20:21 |
| jeblair | puppet agent --environment corvus --server ci-puppetmaster.openstack.org --no-daemonize --verbose --onetime --pluginsync true --certname review.openstack.org | 20:21 |
| *** sweston has quit IRC | 20:21 | |
| jeblair | Warning: Local environment: "corvus" doesn't match server specified node environment "production", switching agent to "production". | 20:21 |
| jeblair | what is that about? ^ | 20:21 |
| mordred | uhm | 20:21 |
| jeblair | i mean, it did a bunch of stuff between those 2 lines | 20:21 |
| mordred | that seems like a terrible thing to do | 20:21 |
| *** _nadya_ has quit IRC | 20:21 | |
| jeblair | looks like it synced the plugins, then errored out on that | 20:21 |
| mordred | Hunner: ^^ ? | 20:21 |
| mordred | nibalizer: ^^ (in case you happen to be aroudn) | 20:22 |
| mordred | jeblair: http://projects.puppetlabs.com/issues/16789 | 20:23 |
| jeblair | oh and actually somehow we did end up with puppet3 | 20:23 |
| sdague | is there a way that we could loop on the puppet3 server with certname and dryrun to expose any syntax issues? | 20:23 |
| *** nati_ueno has joined #openstack-infra | 20:24 | |
| *** khyati_ has joined #openstack-infra | 20:26 | |
| mordred | jeblair: it's entirely possible that issue is puppet3 client with puppet3 server | 20:26 |
| mordred | jeblair: it's entirely possible that issue is puppet3 client with puppet2 server | 20:26 |
| jeblair | mordred: i think you are right | 20:26 |
| sdague | yes, that's what the bug seems to state is root cause | 20:26 |
| clarkb | jeblair: I think we should 1:1 new review | 20:26 |
| clarkb | jeblair: so 30G nodesounds good | 20:26 |
| clarkb | and I did not see the conversation for a cinder volume | 20:27 |
| jeblair | i'm a little disappointed that the error condition for the package pin on puppet is "install puppet3" | 20:27 |
| clarkb | do we need one? | 20:27 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Story detail view cleanup https://review.openstack.org/88707 | 20:27 |
| jeblair | clarkb: yes, for the git repos | 20:27 |
| asselin_ | Hi, I have a question about the single-use-slaves. They seem to be getting reused...where/how are they restricted to single-use? | 20:27 |
| jeblair | asselin_: link to a job where it was reused? | 20:27 |
| asselin_ | jeblair, in my own environment | 20:28 |
| jeblair | asselin_: that might be a good thing to mention in your question next time. :) | 20:28 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Initial commit of home brewed apt_mirror module https://review.openstack.org/89928 | 20:28 |
| asselin_ | jeblair, yes...sorry...didn't mean to start a fire or anything of the sort :) | 20:28 |
| jeblair | asselin_: zuul sets a job parameter that the gearman plugin for jenkins sees and marks the slave offline after the job completes (OFFLINE_WHEN_COMPLETE i think); then nodepool deletes slaves when jobs are done | 20:29 |
| jeblair | asselin_: setting that parameter is done in the openstack_functions.py part of our zuul config | 20:29 |
| jeblair | mordred, clarkb, sdague: so aiui, we can't upgrade to trusty without upgrading to puppet3 | 20:30 |
| mordred | jeblair: ok - so the above trust+puppet3 patch may not need to be split then | 20:30 |
| asselin_ | jeblair, ok I see...will look into that thanks | 20:30 |
| mordred | jeblair: do you think it's worth spinning up a puppet3 puppetmaster so we can do this with trusty? or leave it for a step 2 later on down the road? | 20:31 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Expose assignee_id to API https://review.openstack.org/88676 | 20:31 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Added status filter to story API https://review.openstack.org/87410 | 20:31 |
| sdague | the theory is that once you deal with any incompatibilities with puppet3 syntax on the puppet master you can just stand up the server, right? | 20:32 |
| sdague | because what's fed to the clients isn't version specific | 20:32 |
| sdague | so puppet3 server, all the clients as is | 20:33 |
| *** julim has quit IRC | 20:33 | |
| sdague | the one thing I did find was the cert path defaults changed | 20:33 |
| sdague | for bootstrapping purposes | 20:33 |
| sdague | that was the one thing I remember running around in circles on for a couple hours | 20:34 |
| jeblair | mordred: i don't want review.o.o to be the odd one out until we actually get around to doing this seriously | 20:34 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Story list views now filter with status https://review.openstack.org/88066 | 20:35 |
| mordred | jeblair: nod | 20:36 |
| mgagne | pleia2: ping | 20:36 |
| jeblair | mordred: so i think we need to move to puppet3 across the board while on precise, then we can upgrade to trusty | 20:36 |
| *** ociuhandu has quit IRC | 20:36 | |
| clarkb | jeblair: yes, we knew that | 20:37 |
| sdague | jeblair: that seems sane | 20:37 |
| jeblair | someone might have mentioned that instead of saying "++ new review on trusty" :) | 20:37 |
| pleia2 | mgagne: pong | 20:37 |
| sdague | jeblair: is there an attempt yet at bringing up infra / config on puppet3 ? | 20:38 |
| mgagne | pleia2: FYI, we are using reprepro to manage our internal mirrors (not the public ones we are hosting here at iWeb) | 20:38 |
| clarkb | sdague: yes, the slaves work under it | 20:38 |
| clarkb | I think I figured that things would mostly work | 20:38 |
| mgagne | pleia2: imo, it's much more flexible than apt_mirror | 20:38 |
| pleia2 | mgagne: ah, good to know, I looked at ubumirror and apt-cacher | 20:38 |
| jeblair | clarkb: install_puppet doesn't work -- mordred has a patch for that | 20:38 |
| mordred | clarkb: so far, the do seem to mostly work -other than puppet3 client/ puppet2 server communication - one you have my patch | 20:38 |
| pleia2 | mgagne: I have a bug outstanding where I'm trying to track ideas, can you share some thoughts? https://bugs.launchpad.net/openstack-ci/+bug/1311855 | 20:38 |
| uvirtbot | Launchpad bug 1311855 in openstack-ci "Create puppet module for Ubuntu repository mirror" [Undecided,In progress] | 20:39 |
| mgagne | pleia2: with apt-mirror, you end up with a structure of folders matching the URL of the mirror you are mirroring which wasn't what we needed | 20:39 |
| * pleia2 nods | 20:39 | |
| *** prad_ has joined #openstack-infra | 20:39 | |
| jeblair | mordred: do the single use slaves talk to the puppetmaster? | 20:39 |
| clarkb | jeblair: no, that is something I didn't consider | 20:39 |
| clarkb | so puppet apply would work but not puppetmaster arg | 20:39 |
| clarkb | or mostly work at least | 20:39 |
| jeblair | well, puppet apply is how we spin up the slaves, right, so they are in a better position | 20:40 |
| clarkb | ya | 20:40 |
| sdague | well that alt least means the pieces of policy they touch work | 20:40 |
| mordred | that's right. we can support trusty for the slaves with puppet3 before we do the other thigns | 20:40 |
| pleia2 | mgagne: reprepro looks cool, I'll dig in | 20:40 |
| clarkb | that slipped my consideration of puppet3 | 20:40 |
| mgagne | pleia2: I can copy/paste a configuration we use | 20:40 |
| mordred | pleia2: AaronGr will want to talk to you about ... | 20:40 |
| mordred | AaronGr: what' the name of the thing you prefer over reprepro ? | 20:41 |
| pleia2 | mgagne: that would be great | 20:41 |
| mgagne | pleia2: we also use debmirror with one of our other mirrors (yes, we do have a bunch internally) | 20:41 |
| *** prad has quit IRC | 20:41 | |
| *** prad_ is now known as prad | 20:41 | |
| *** marun has quit IRC | 20:41 | |
| pleia2 | mordred: heh, I figured we'd all have opinions on this :) | 20:41 |
| clarkb | so I looked at doing a mirror for ourselves and I don't think we can solve the package index out of sync issue | 20:42 |
| mordred | pleia2: I missed the ping - what are you looking to do? | 20:42 |
| clarkb | you have to accept pushes from the upstream to make that work correctly | 20:42 |
| clarkb | and unless we get pushed to from ubuntu how would that work? | 20:42 |
| *** zns_ has joined #openstack-infra | 20:43 | |
| dstufft | mirrors are a cool way to attack packaging infrastructure | 20:43 |
| *** harlowja is now known as harlowja_away | 20:44 | |
| pleia2 | mordred: tripleo wants their own ubuntu (and fedora) and pypi mirrors, tossing it in -infra since we might want to do one someday too | 20:44 |
| mgagne | clarkb: as an official ubuntu mirror, AFAIK, we do rely on upstream to trigger the sync | 20:44 |
| clarkb | mgagne: but we aren't official | 20:44 |
| clarkb | unless I am missing something | 20:44 |
| *** zns has quit IRC | 20:44 | |
| mgagne | clarkb: I know =) | 20:44 |
| clarkb | is this not a thing to fix the problem of rax mirrors derping? | 20:44 |
| mordred | pleia2: well, aaron has some puppet for this already but I don't hink has submitted it upstream yet because it wasn't something we wanted to do in infra yet | 20:45 |
| clarkb | s/rax/rax and hpcloud/ | 20:45 |
| clarkb | mordred: I am not sure we ever want to do it | 20:45 |
| mordred | clarkb: right | 20:45 |
| mgagne | clarkb: just stating that if you are official, you do get the benefit of receiving a trigger instead of relying on a cron job | 20:45 |
| clarkb | mgagne: right and that is the only way to avoid the errors we hae with the existing mirrors aiui | 20:45 |
| clarkb | otherwise there will alwas be index races | 20:45 |
| pleia2 | mordred: ah ok, I'll follow up with AaronGr | 20:46 |
| *** mwagner_lap has quit IRC | 20:47 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Expose assignee_id to API https://review.openstack.org/88676 | 20:47 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add an entry for a new puppetmaster for puppet3 https://review.openstack.org/89931 | 20:48 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Changes to get nodepool working with trusty https://review.openstack.org/88477 | 20:48 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Story detail view cleanup https://review.openstack.org/88707 | 20:48 |
| mordred | clarkb, jeblair: for whenver we want to get aroudn to it ^^ I made a change for adding a new puppetmaster too - with a flag to turn off the remote-ssh execution on the new puppetmaster | 20:48 |
| *** mrodden1 has joined #openstack-infra | 20:49 | |
| jeblair | mordred: how is it different from nibalizer's? https://review.openstack.org/#/c/83678/ | 20:49 |
| *** mrodden has quit IRC | 20:50 | |
| mordred | ah - I did not find that by searching the list for "puppetmaster" - my bad | 20:50 |
| jeblair | mordred, clarkb: new review is at 23.253.232.87 2001:4800:7815:104:3bc3:d7f6:ff03:bf5d | 20:51 |
| clarkb | jeblair: that is 30G precise node with cinder volume? | 20:52 |
| mordred | well, main difference is that mine assumes no pinning of puppet at all on trusty, and also has the flag for turning off the remote puppet execution - but I'll work with nibalizer on combining the two | 20:52 |
| jeblair | clarkb: well, it doesn't have a cinder volume yet | 20:52 |
| clarkb | jeblair: ok. I am really not a fan of using cinder volumes if we can avoid it | 20:52 |
| clarkb | jeblair: becaues there doesn't seem to be a good way to deal with config managemtn + cinder volumes | 20:52 |
| mordred | clarkb: what do you mean? | 20:53 |
| jeblair | clarkb: erm, enotenoughspace | 20:53 |
| clarkb | jeblair: I know | 20:53 |
| clarkb | I think we have to here | 20:53 |
| clarkb | I am just pointing out the suck that is cinder volumes | 20:53 |
| mordred | clarkb: I just still don't know what you mean | 20:53 |
| clarkb | mordred: there are so many manual steps | 20:53 |
| clarkb | and they are racy with config management | 20:53 |
| mordred | how? | 20:53 |
| clarkb | and package installs | 20:53 |
| mordred | you create a node, you add a volume, you mount it, then you run config management | 20:54 |
| mordred | what does that have to do with package installs? | 20:54 |
| jeblair | mordred: except that's not what we do | 20:54 |
| mordred | no? | 20:54 |
| mordred | ok | 20:54 |
| jeblair | mordred: we create a node then run config management, then add a volume | 20:54 |
| clarkb | ya | 20:54 |
| clarkb | and since every package ever wants to start services when installed you end up with races | 20:54 |
| jeblair | mordred: so anything that config management touches on that path needs to be manually moved to the volume | 20:54 |
| *** markmcclain has quit IRC | 20:55 | |
| mordred | why do we run config management before mounting the disks? | 20:55 |
| clarkb | because mounting disks is a manual process | 20:55 |
| clarkb | booting and configuring a node is a one stop shop automagic process | 20:55 |
| mordred | because we wrote a script for it | 20:55 |
| clarkb | right which would need a significant rewrite/addition to do volumes | 20:56 |
| Hunner | Lovely. 2.7 agents *should* function with puppet 3 masters though... | 20:56 |
| *** msabramo has left #openstack-infra | 20:56 | |
| jeblair | mordred: even if our magic script did deal with volumes, you'd still have the race conditions -- | 20:56 |
| Hunner | But not the other way around | 20:56 |
| clarkb | Hunner: thats good to know | 20:56 |
| clarkb | so we may be able to get away with a single puppet3 master | 20:57 |
| jeblair | mordred: it couldn't mount something in /var/lib/path/to/some/service without creating the directory, and we would normally expect the directory to be created by config management | 20:57 |
| mordred | right. I think that's all just about design - but I'm perfectly willing to just defer to the two of you because you've actually touched the problem and I haven't | 20:58 |
| clarkb | mordred: well this leads to my next idea | 20:58 |
| clarkb | I think nova should just do this for us | 20:58 |
| clarkb | but I am lazy | 20:58 |
| jeblair | mordred: AFS, otoh, does not have this problem. | 20:58 |
| clarkb | and really nova shouldn't do all that stuff for us | 20:58 |
| clarkb | it would be pretty cool to nova boot with volume already attached though | 20:58 |
| mordred | I think that clouds give me apis to do things I'd do with my hands if I was in a server room | 20:59 |
| jeblair | natch | 20:59 |
| *** lukego has quit IRC | 20:59 | |
| mordred | so I think that if I had a server, and I wanted to add a new disk, I'd add a new disk | 20:59 |
| mordred | then I'd mount it | 20:59 |
| clarkb | mordred: no no | 20:59 |
| mordred | then I'd run stuff to put stuff in that location | 20:59 |
| clarkb | mordred: if you had a server it would come up workable | 20:59 |
| clarkb | you don't randomly add disks to servers usually. You may swap disks out though | 20:59 |
| mordred | this is a pointless argument | 21:00 |
| *** etoews_ has joined #openstack-infra | 21:00 | |
| clarkb | mordred: well not entirely | 21:00 |
| clarkb | mordred: I think it comes back to that nova partitioning feature | 21:00 |
| *** markmcclain has joined #openstack-infra | 21:00 | |
| clarkb | mordred: if we can have that + preattached volumes we are golden | 21:01 |
| clarkb | but we don't have that yet | 21:01 |
| jeblair | wanting to put /var/log on a cinder volume is particularly problematic, and i think clarkb's suggestion may address that | 21:01 |
| mordred | I still think this is just a deficiency in our launch script - you can do _everything_ you want to do - we just don't do it | 21:01 |
| mordred | now, is it worthwhile to fix the launchs cript? probably not | 21:02 |
| zaro | jeblair: curious, did you puppetize review-dev to setup trove? | 21:02 |
| mordred | zaro: puppet doesn't setup trove | 21:02 |
| clarkb | zaro: no trove is via the apis | 21:02 |
| jeblair | mordred: sure, but the launch script would be duplicating config management if it did things like mount something at /home/gerrit2 | 21:02 |
| *** etoews has quit IRC | 21:03 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add base test class to make subclassing possible https://review.openstack.org/85204 | 21:03 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Fix races in timer trigger tests. https://review.openstack.org/85230 | 21:03 |
| jeblair | zaro: i also did not run puppet there because i didn't know the state, so i changed the config files manually. if it's safe to run puppet though, it should be identical. | 21:03 |
| jeblair | (eg, the files on disk related to database reflect the current state in puppet) | 21:04 |
| jeblair | clarkb, mordred: so anyway, how do we want to handle the extra space needed by /home/gerrit2/review_site and /var/lib/git? | 21:04 |
| *** harlowja_away is now known as harlowja | 21:05 | |
| sdague | clarkb: is there a reason we can't pass --block-device-mapping on our nova boot? | 21:05 |
| sdague | or am I missing something | 21:05 |
| mordred | jeblair: | 21:05 |
| mordred | gah | 21:05 |
| clarkb | sdague: there are two steps. First is attach device, second is partition and format device | 21:05 |
| clarkb | sdague: without both steps we lose | 21:05 |
| clarkb | sdague: I think current upstream nova can do this for us | 21:05 |
| clarkb | our cloud providers don't | 21:05 |
| zaro | jeblair: i think it it's up to date. I would like to run the agent so we see it on puppetdb.o.o. | 21:06 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Handle NullChange "merges" gracefully. https://review.openstack.org/85217 | 21:06 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 21:06 |
| clarkb | jeblair: ok I have zuul out of my system now | 21:06 |
| clarkb | jhesketh: ^ btw | 21:06 |
| jeblair | zaro: run "puppet agent --test" and check out the output first; then /etc/init.d/puppet start | 21:06 |
| sdague | clarkb: because the volume is presented as an unformatted block originally? | 21:06 |
| clarkb | sdague: yup | 21:06 |
| clarkb | sdague: and isn't mounted | 21:06 |
| *** homeless has joined #openstack-infra | 21:06 | |
| zaro | jeblair: ok. will git that a go. | 21:07 |
| sdague | and we can't shove that down cloud-init? | 21:07 |
| mordred | sdague: we don't use cloud-init for things | 21:07 |
| clarkb | sdague: possibly, but that isn't race free | 21:07 |
| sdague | ok | 21:07 |
| jeblair | okay i'm about to just do stuff and tell everyone what i did later. | 21:07 |
| mordred | because it's hard to debug and often doesn't work | 21:07 |
| clarkb | you need it to be done before the OS boots | 21:07 |
| clarkb | jeblair: ++ | 21:07 |
| mordred | jeblair: I support you | 21:07 |
| clarkb | jeblair: I think that is really the best we can do. I would do the lvm stuff | 21:07 |
| clarkb | jeblair: since that is consistent with our other volumes | 21:07 |
| mordred | sdague: I do agree with clarkb that this is one of those places where as a user of a cloud I feel like we're not getting the thing we actually want here | 21:08 |
| sdague | clarkb: ok, I thought cloud init happened early enough that it could handle a case like this, but I can see where it wouldn't be | 21:08 |
| jeblair | one lv or two? if one: bind mount or symlink? | 21:08 |
| clarkb | sdague: it has to happen before / remounts rw | 21:08 |
| *** david-lyle has quit IRC | 21:09 | |
| sdague | clarkb: gotcha | 21:09 |
| jeblair | or disable the local mirror? | 21:09 |
| *** markwash has joined #openstack-infra | 21:09 | |
| clarkb | oh arg | 21:09 |
| sdague | so you could use a sacrificial vm to prep volumes | 21:09 |
| sdague | but getting the mount point that early, yeh, that's a problem | 21:09 |
| clarkb | jeblair: why don't we go with two and mount | 21:10 |
| clarkb | jeblair: they should be approximately the same size right? | 21:10 |
| clarkb | jeblair: so we can grow them together if we ever need to and it shouldn't be confusing | 21:10 |
| *** dkranz has quit IRC | 21:10 | |
| mordred | any thoughts on serving apache git from ~gerrit2 ? | 21:10 |
| jeblair | mordred: why didn't you configure it that way to start with? | 21:11 |
| mordred | jeblair: I honestly am having trouble coming up with a reason | 21:11 |
| clarkb | its because you can restrict what you replicate | 21:11 |
| clarkb | if you serve direction out of gerrit2 you get everything | 21:11 |
| clarkb | s/direction/directly/ | 21:11 |
| jeblair | ah | 21:12 |
| clarkb | at least that seems like a good reason for it :) | 21:12 |
| mordred | but we don't actually restrict what we replicate at this point? | 21:12 |
| jeblair | the "bypass gerrit acl" reason | 21:12 |
| clarkb | mordred: we do iirc | 21:12 |
| mordred | ok. sold | 21:12 |
| *** thomasem has quit IRC | 21:13 | |
| jeblair | GAAAH: https://review.openstack.org/#/c/63190/ | 21:13 |
| clarkb | :( I feel bad about that | 21:14 |
| *** sweston has joined #openstack-infra | 21:14 | |
| jeblair | clarkb: let me know when you have made a swap partition and mounted it on the new review.o.o server | 21:14 |
| jeblair | i'm going afk for a while. | 21:14 |
| clarkb | what is xvde on these macines? is that the ephemeral disk? | 21:17 |
| *** hashar has quit IRC | 21:17 | |
| *** andreaf has joined #openstack-infra | 21:21 | |
| clarkb | also do we want a full 30GB of swap or just 2GB like on the old node? | 21:22 |
| *** ociuhandu has joined #openstack-infra | 21:24 | |
| *** yamahata has joined #openstack-infra | 21:26 | |
| *** smarcet has quit IRC | 21:26 | |
| bnemec | Howdy, infra. I'm looking to resolve the dependency issue with diskimage-builder discussed here: https://bugzilla.redhat.com/show_bug.cgi?id=1086494 | 21:27 |
| uvirtbot | bnemec: Error: Could not parse XML returned by bugzilla.redhat.com: HTTP Error 404: Not Found | 21:27 |
| bnemec | After some discussion in tripleo, we think the right thing to do is split dib-run-parts into its own thing. | 21:28 |
| bnemec | (and possibly some other pieces of diskimage-builder) | 21:28 |
| bnemec | What I'm wondering is whether we need a completely new project for that, or if there's some other way to handle it. | 21:28 |
| *** etoews has joined #openstack-infra | 21:29 | |
| mordred | new project | 21:29 |
| clarkb | jeblair: mordred see newreview:~clarkb/fix_swap.sh | 21:29 |
| clarkb | I have notes | 21:29 |
| mordred | bnemec: everything in infra assumes one repo == one output | 21:29 |
| bnemec | mordred: Okay thanks, that's what I needed to know. Are there docs about setting up a new repo? | 21:31 |
| *** gondoi is now known as zz_gondoi | 21:31 | |
| *** etoews_ has quit IRC | 21:32 | |
| *** fifieldt has quit IRC | 21:32 | |
| zaro | mordred: should manage_projects run for review-dev? puppet is giving me an error http://paste.openstack.org/show/76851 | 21:32 |
| *** fifieldt has joined #openstack-infra | 21:33 | |
| clarkb | if we are happy with the stuff I have commented on in fix_swap.sh I can run that | 21:34 |
| mordred | zaro: it should | 21:34 |
| mordred | clarkb: looking | 21:34 |
| mordred | clarkb: seems reasonble - I dont' see any reason to not mount the excess on /opt | 21:35 |
| *** UtahDave has quit IRC | 21:35 | |
| clarkb | mordred: I think my only concern is it is an ephemeral disk and maybe we want /opt to not be ephemeral? | 21:36 |
| clarkb | mordred: but currently as used it should be fine | 21:36 |
| clarkb | mordred: what about the size of the swap partition? | 21:36 |
| mordred | clarkb: well, we could also leave it unmounted - but I don't think that we put any information we care about in /opt | 21:36 |
| mordred | clarkb: it's worked for us on our last machine, so I'm fine with it | 21:37 |
| clarkb | last machine didn't have ephemeral disk | 21:37 |
| clarkb | it was one giant / | 21:37 |
| clarkb | oh you mean 2GB works for last machine | 21:37 |
| mordred | I thought you said the last machien had 2G ... yeah | 21:37 |
| clarkb | context switches | 21:37 |
| mordred | :) | 21:37 |
| clarkb | why is "* os-refresh-config depends on diskimage-builder package (yuck!)" yuck? | 21:38 |
| clarkb | that is the actual dependency | 21:38 |
| mordred | clarkb: orc is somethign that's expected to be installed on every machine as a guest agent | 21:38 |
| *** zns_ has quit IRC | 21:39 | |
| mordred | clarkb: there is a script inside of dib that orc is using | 21:39 |
| mordred | clarkb: which is the reason taht orc would need the depend - but it doesn't actually depend on dib itself | 21:39 |
| clarkb | sure, its just that dib run parts is part of dib so you depend on dib | 21:39 |
| mordred | or, you make dib run parts its own thing and depend on that | 21:39 |
| clarkb | ya thats definitely a better option. I just don't understand the problem with depending on your dependecny :) | 21:40 |
| *** jamielennox|away is now known as jamielennox | 21:40 | |
| bnemec | Yeah, apparently they are quite opposed to pulling in another dep because o-r-c is going to be included on cloud images, and they want those to be as small as possible. | 21:40 |
| mordred | yeah - it's a footprint size issue | 21:40 |
| *** talluri has joined #openstack-infra | 21:41 | |
| clarkb | jeblair: any thoughts on size of swap (2GB vs 30GB)? and the /opt mount? | 21:41 |
| zaro | jenkins-dev is no longer reporting to puppetdb.o.o, anybody looking into that? | 21:42 |
| clarkb | zaro: puppet doesn't work on it | 21:42 |
| clarkb | bceause of the nodepool thing | 21:42 |
| clarkb | which is another change that got abandoned | 21:42 |
| *** mdenny has quit IRC | 21:42 | |
| zaro | it got replaced with the one jeblair uploaded | 21:43 |
| mordred | clarkb: I believe, based on his change, jeblair is in favor of swap that is MEM=`grep MemTotal /proc/meminfo | awk '{print $2; }'` | 21:43 |
| mordred | clarkb: adn the rest in opt | 21:43 |
| clarkb | mordred: ya | 21:43 |
| clarkb | but now that we are applying it to something like review.o.o I am curious if that still holds | 21:43 |
| zaro | clarkb: here's the new nodepool fix https://review.openstack.org/#/c/88026 | 21:43 |
| mordred | clarkb: I mean, I think his intent was that it apply to everything we run puppet on | 21:44 |
| *** talluri has quit IRC | 21:45 | |
| zaro | mordred: do you have any idea why managed_project script fails when run on review-dev? http://paste.openstack.org/show/76851 | 21:46 |
| clarkb | mordred: ok /me updates script to match the install_puppet proposal | 21:46 |
| *** changbl has quit IRC | 21:47 | |
| *** adalbas has quit IRC | 21:47 | |
| clarkb | I did the math on 2048 wrong anyways | 21:47 |
| clarkb | thats 2MB :/ | 21:47 |
| *** mdenny has joined #openstack-infra | 21:48 | |
| mordred | haha | 21:48 |
| mordred | that's plenty of swap! | 21:49 |
| mordred | who needs more than 2M of anything? | 21:49 |
| mordred | zaro: it looks like an ssh key mismatch | 21:49 |
| clarkb | mordred: ok how does it look now? | 21:49 |
| mordred | clarkb: looks great | 21:49 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Create a swap partition if needed for new servers https://review.openstack.org/63190 | 21:50 |
| clarkb | mordred: do you think I should wait for jeblair to look or just do it? | 21:50 |
| zaro | mordred: what does that mean? i'm running as root on review-dev, so should have the right key correct? | 21:50 |
| mordred | jeblair: ^^ I restored your change and split it in to two scripts as per clarkb's review - if you are opposed to that, I'll revert | 21:50 |
| mordred | clarkb: I think you should just do it | 21:51 |
| *** mriedem has quit IRC | 21:51 | |
| *** dprince has quit IRC | 21:52 | |
| clarkb | ok just doing it now | 21:53 |
| jeblair | clarkb: wfm; if you do mount /opt, be sure to rm what's there already | 21:53 |
| jeblair | to avoid confusion | 21:53 |
| clarkb | jeblair: ok | 21:54 |
| clarkb | I am doing it by hand rather thanrunning the script | 21:55 |
| *** dizquierdo has joined #openstack-infra | 21:55 | |
| clarkb | and will do the rm after the rsync | 21:55 |
| clarkb | Warning: The resulting partition is not properly aligned for best performance. | 21:56 |
| clarkb | I can redo the parted commands with a different $MEM value to fix that | 21:56 |
| * clarkb figures out how to do that | 21:56 | |
| jeblair | clarkb: check devstack-gate; i think sdague fixed it there | 21:57 |
| *** maxbit has quit IRC | 21:57 | |
| *** Sukhdev has joined #openstack-infra | 21:57 | |
| clarkb | thansk | 21:58 |
| jogo | so looping back to an old item: how do I get 'http://specs.openstack.org/nova' to have nova-specs? | 21:59 |
| sdague | clarkb: yeh, just put 1 as the start instead of 0 | 21:59 |
| *** nati_ueno has quit IRC | 21:59 | |
| *** hogepodge_ has joined #openstack-infra | 21:59 | |
| *** etoews has quit IRC | 22:00 | |
| clarkb | sdague: does that work when the end of the range is kB? | 22:00 |
| clarkb | sdague: or should I convert both to megabyte? | 22:01 |
| clarkb | would be '1 30720' for MB | 22:01 |
| *** hogepodge has quit IRC | 22:01 | |
| *** hogepodge_ is now known as hogepodge | 22:01 | |
| * clarkb does that | 22:01 | |
| sdague | yeh, I don't know, that was in MB | 22:01 |
| *** pdmars has quit IRC | 22:02 | |
| anteaya | zaro: I was getting key errors all the time when I tried to run m-p on my gerrit vm | 22:02 |
| anteaya | I don't know the fix | 22:02 |
| jogo | clarkb jeblair: ^ | 22:02 |
| *** zns has joined #openstack-infra | 22:04 | |
| clarkb | jogo: specs.openstack.org isn't a thing | 22:04 |
| clarkb | at least according to DNS | 22:04 |
| clarkb | so we would need to update that | 22:04 |
| clarkb | however, is this another thing that should be hosted as docs? | 22:04 |
| jeblair | jogo: can we work on that after the gerrit upgrade? | 22:05 |
| anteaya | bnemec: if you haven't found anything better, here is a template patch for starting a new repo | 22:05 |
| jogo | clarkb: its not a think yet, this was a old ML thread. about where to make *-specs docs live | 22:05 |
| anteaya | bnemec: note the new-project topic | 22:06 |
| jogo | and everyone agreed it would not be in docs directly | 22:06 |
| jogo | jeblair: sure. this isn't a priority. just wanted to see what would be involved to get the ball rolling | 22:06 |
| anteaya | bnemec: https://review.openstack.org/#/c/89457/ | 22:06 |
| * jogo will revist after gerrit upgrade | 22:06 | |
| anteaya | sorry | 22:06 |
| jeblair | jogo: mostly manual steps requiring infra root, which is mostly why i was suggesting deferring it; i don't think there's much to be done outside of that at this point. thanks | 22:07 |
| bnemec | anteaya: Thanks! I was looking through https://wiki.openstack.org/wiki/Oslo/CreatingANewLibrary#Creating_an_Initial_Repository_for_Import | 22:07 |
| clarkb | jeblair: all done | 22:07 |
| clarkb | jeblair: I will update your change with what I learned about alignment warnings | 22:08 |
| *** mburned is now known as mburned_out | 22:09 | |
| *** mburned_out is now known as mburned | 22:09 | |
| jeblair | clarkb: cool, thanks | 22:09 |
| *** mfer has quit IRC | 22:10 | |
| anteaya | bnemec: you can go the cookiecutter route if you like | 22:11 |
| anteaya | bnemec: it is up to you, if you have source you want to bring in ensure the source repo has trimmed down branches and tags | 22:12 |
| *** melwitt has quit IRC | 22:12 | |
| anteaya | bnemec: this might be helpful too: http://ci.openstack.org/stackforge.html#add-a-project-to-stackforge | 22:12 |
| bnemec | anteaya: Yeah, it's actually a little weird for this project since right now there's no Python code, which confuses cookiecutter a bit. | 22:13 |
| *** marcoemorais has quit IRC | 22:13 | |
| anteaya | I haven't used cookiecutter before, so I will take your word for it | 22:13 |
| bnemec | anteaya: I'll have to play around with it a bit and figure out what's going to work best. I'm sure I'll be back with questions. :-) | 22:13 |
| clarkb | bnemec: anteaya I don't think you want cookiecutter | 22:13 |
| anteaya | the patch deals with jobs, acl, bot (if you want it) | 22:13 |
| clarkb | instead you want git filter branch if talking about dib run parts | 22:13 |
| anteaya | great | 22:14 |
| anteaya | what clarkb said | 22:14 |
| *** melwitt has joined #openstack-infra | 22:14 | |
| *** harlowja has quit IRC | 22:14 | |
| bnemec | clarkb: Okay, thanks. | 22:14 |
| *** marcoemorais has joined #openstack-infra | 22:14 | |
| *** mfer has joined #openstack-infra | 22:14 | |
| bnemec | I guess this is more like graduation than a completely new project, so that makes sense. | 22:14 |
| *** nati_ueno has joined #openstack-infra | 22:15 | |
| clarkb | you want to preserve all of the history of the dib run parts | 22:15 |
| clarkb | which means doing surgery on the existing repo | 22:15 |
| *** SumitNaiksatam has joined #openstack-infra | 22:15 | |
| anteaya | bnemec: cobble together an initial patch and we can groom it through the comments | 22:16 |
| bnemec | anteaya: Will do. I have to take off shortly, but I'll revisit this tomorrow. | 22:17 |
| bnemec | anteaya: clarkb mordred: Thanks for all your help. | 22:17 |
| anteaya | cheers | 22:18 |
| clarkb | jeblair: both sides need to be MB rounded. so 1 8096 on an 8GB machine for example | 22:18 |
| * clarkb is trying to do maths on MemTotal to make that generic | 22:18 | |
| *** marcoemorais has quit IRC | 22:18 | |
| *** markmcclain has quit IRC | 22:18 | |
| zaro | mordred: took a deeper look, looks like managed projects fails when calling listProjects. listProjects fails due to the mismatched keys. The key in gerrit2 looks ok to me so not sure why it's not matching up. | 22:22 |
| zaro | clarkb: ^ did we replace the gerrit key? | 22:23 |
| clarkb | oh probably | 22:24 |
| *** marcoemorais has joined #openstack-infra | 22:24 | |
| zaro | clarkb: i think we need to update this key? /home/gerrit2/review_site/etc/ssh_project_rsa_key | 22:26 |
| clarkb | possibly | 22:26 |
| clarkb | if that isn't puppeted | 22:26 |
| clarkb | actually I bet that key is correct and puppetted | 22:27 |
| clarkb | I think the review-dev DB needs updating | 22:27 |
| zaro | clarkb: it does looks like the keys are puppeted. | 22:30 |
| zaro | clarkb: are you referring to hiera when you say review-dev db? | 22:30 |
| *** sacharya has quit IRC | 22:31 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Creator field added to a Task model https://review.openstack.org/87626 | 22:32 |
| clarkb | zaro: no, I am referring to the key for the project creator user in the mysql DB for review-dev | 22:33 |
| *** mrodden1 has quit IRC | 22:33 | |
| *** amcrn has quit IRC | 22:34 | |
| *** dizquierdo has quit IRC | 22:35 | |
| zaro | clarkb: is that in puppet? i don't see it. | 22:35 |
| *** thedodd has quit IRC | 22:36 | |
| clarkb | it isn't | 22:36 |
| *** sweston has quit IRC | 22:37 | |
| *** sweston has joined #openstack-infra | 22:39 | |
| *** reed has quit IRC | 22:42 | |
| *** talluri has joined #openstack-infra | 22:42 | |
| *** rcarrillocruz has joined #openstack-infra | 22:42 | |
| *** gothicmindfood has joined #openstack-infra | 22:42 | |
| *** jhesketh has quit IRC | 22:42 | |
| *** jhesketh__ has joined #openstack-infra | 22:43 | |
| *** fbo_away has quit IRC | 22:44 | |
| *** jhesketh__ is now known as jhesketh | 22:44 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Create a swap partition if needed for new servers https://review.openstack.org/63190 | 22:44 |
| clarkb | jeblair: mordred ^ I caved and used python for the roudning. Hope that is acceptable | 22:44 |
| jhesketh | Morning | 22:45 |
| clarkb | jhesketh: hi there | 22:45 |
| *** talluri has quit IRC | 22:46 | |
| *** yamahata has quit IRC | 22:47 | |
| *** HenryG has quit IRC | 22:51 | |
| *** matrohon has quit IRC | 22:53 | |
| wenlock | hi, im hoping someone might have an idea on a problem im having with Package['jenkins']. we are getting this error when we pre-create the jenkins user like in jenkinsuser.pp http://paste.openstack.org/show/76856/ i'm wondering if anyone recently encountered that issue on ubuntu jenkins package? | 22:54 |
| wenlock | i do get teh same error when trying : bash -x /var/lib/dpkg/info/jenkins.postinst | 22:55 |
| wenlock | adduser command fails, because i already have the jenkins user pre-setup | 22:55 |
| *** HenryG has joined #openstack-infra | 22:55 | |
| *** devoid has quit IRC | 22:56 | |
| Alex_Gaynor | What's up with the unstarted qa-specs at the head of the check pipeline? They aren't disturbing anything (AFAIK), but I'm curious :-) | 23:00 |
| sdague | Alex_Gaynor: scroll.... way back :) | 23:00 |
| Alex_Gaynor | sdague: uhh, how far, my logs go back 8 hours and no one said "qa-specs" :-) | 23:01 |
| *** pmathews has joined #openstack-infra | 23:02 | |
| sdague | "jeblair: is there any way to kick the "lost" jobs in zuul to run again? " - it wasn't just qa-specs | 23:02 |
| sdague | about 5 hrs back | 23:02 |
| *** flaper87 is now known as flaper87|afk | 23:05 | |
| *** bnemec has quit IRC | 23:09 | |
| *** bnemec has joined #openstack-infra | 23:10 | |
| *** gothicmindfood has quit IRC | 23:16 | |
| *** andreaf has quit IRC | 23:18 | |
| sdague | jeblair: https://review.openstack.org/#/c/89920/ probably should +A it to get it in | 23:18 |
| *** melwitt has quit IRC | 23:19 | |
| sdague | or clarkb :) | 23:19 |
| *** mfer has quit IRC | 23:19 | |
| *** pmathews has quit IRC | 23:19 | |
| jeblair | sdague: done | 23:20 |
| *** mfer has joined #openstack-infra | 23:20 | |
| *** melwitt has joined #openstack-infra | 23:20 | |
| *** homeless has quit IRC | 23:21 | |
| *** marcoemorais has quit IRC | 23:21 | |
| openstackgerrit | Matthieu Baerts proposed a change to openstack-infra/git-review: Python2: fixed UnicodeEncodeError https://review.openstack.org/88828 | 23:21 |
| *** marcoemorais has joined #openstack-infra | 23:22 | |
| *** gokrokve has quit IRC | 23:23 | |
| *** mrodden has joined #openstack-infra | 23:23 | |
| *** che-arne has quit IRC | 23:24 | |
| *** mfer has quit IRC | 23:30 | |
| anteaya | wenlock: I haven't tried to do what you are doing, I have no experience with that error | 23:30 |
| sdague | jeblair: thanks | 23:30 |
| openstackgerrit | Matthieu Baerts proposed a change to openstack-infra/git-review: Python2: fixed UnicodeEncodeError https://review.openstack.org/88828 | 23:30 |
| *** mfer has joined #openstack-infra | 23:30 | |
| clarkb | wenlock: couple of things. the package you are installing is for a very new version of jenkins which won't work with gearman plugin | 23:31 |
| *** markwash has quit IRC | 23:31 | |
| wenlock | anteaya, yeah, i think what im doing is trying to avoid installing another server for a slave, and the jenkins account setup on master , is not compatible with what slave.pp does | 23:31 |
| jeblair | clarkb, mordred: back and looking at the gerrit filesystem... | 23:31 |
| clarkb | wenlock: is it possible that newer package fails differently than older once on the user? | 23:31 |
| clarkb | s/once/ones/ | 23:32 |
| wenlock | clarkb, yeah, i'm thinking the older one usto fail differently | 23:32 |
| wenlock | you guys probably don't mix slave + master, right? | 23:32 |
| clarkb | wenlock: nope | 23:33 |
| wenlock | ok, bingo... if you did, you'd see it | 23:33 |
| clarkb | wenlock: now for a fix. I would make sure the package is installed before the user | 23:33 |
| clarkb | also why would you do that? | 23:33 |
| wenlock | yep, that makes alot of sense | 23:33 |
| wenlock | :d resources arn't cheap | 23:33 |
| clarkb | ya but jenkins master is a beast | 23:33 |
| jeblair | clarkb: https://etherpad.openstack.org/p/00SfC9xKl7 | 23:34 |
| wenlock | ok, thanks for the clues guys, much appreciated! | 23:34 |
| wenlock | as always | 23:34 |
| *** jhesketh has quit IRC | 23:34 | |
| *** wenlock has quit IRC | 23:36 | |
| jeblair | ah the "jeepyb" copy of the git repos are also in /var/lib | 23:37 |
| clarkb | oh right because it caches that now | 23:37 |
| jeblair | most of /root/ is a restore from backup | 23:38 |
| *** rediskin has left #openstack-infra | 23:38 | |
| jeblair | so we can disregard that | 23:38 |
| *** CaptTofu has quit IRC | 23:39 | |
| *** CaptTofu has joined #openstack-infra | 23:39 | |
| *** zns has quit IRC | 23:39 | |
| *** CaptTof__ has joined #openstack-infra | 23:41 | |
| jeblair | i'm striking out things we can disregard for the purposes of figuring out what the new server will look like | 23:41 |
| *** talluri has joined #openstack-infra | 23:42 | |
| jeblair | huh. "nova flavor-list" says we get "40" under "Disk" | 23:43 |
| jeblair | /dev/xvda1 20G 1.8G 17G 10% / | 23:43 |
| *** CaptTofu has quit IRC | 23:43 | |
| openstackgerrit | Matthieu Baerts proposed a change to openstack-infra/git-review: Python2: fixed UnicodeEncodeError https://review.openstack.org/88828 | 23:43 |
| jeblair | that doesn't seem te be what we actually get? | 23:43 |
| mordred | jeblair: maybe we're supposed to divide by 2 | 23:44 |
| mordred | it's telling us sizes in terms of 1000s of 512k blocks | 23:44 |
| * mordred has worked with mysql-cluster before - has flashbacks to units of measure being insane | 23:45 | |
| clarkb | nope | 23:47 |
| clarkb | 17G free was definitely what I saw when I looked | 23:47 |
| jeblair | mikal: rackspace is being weird: flavor-list says disk=40 for a performance2-30 but we actually get 20G. | 23:47 |
| *** mfer has quit IRC | 23:47 | |
| *** talluri has quit IRC | 23:47 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add support for disk-image-builder in nodepool https://review.openstack.org/88479 | 23:47 |
| *** arosen1 has joined #openstack-infra | 23:47 | |
| mordred | ok. that still probably needs more work - but I think it's got all of the big pieces, is structurally there | 23:47 |
| *** jamespage has quit IRC | 23:48 | |
| jeblair | mordred: i mean, i can't argue with that -- it does, indeed, appear to be reporting in units of 1000s of 512k blocks. | 23:48 |
| arosen1 | Hi, just checking if you guys were aware of this: https://bugs.launchpad.net/openstack-ci/+bug/1311835 | 23:48 |
| uvirtbot | Launchpad bug 1311835 in openstack-ci "devstack-vm-gate.sh: line 264: bc: command not found" [Undecided,New] | 23:48 |
| arosen1 | I was about to file that bug as well but noticed that. | 23:48 |
| mordred | jeblair: that makes me want to die inside a little bit | 23:48 |
| jeblair | arosen1: yeah, the fix for that has been approved | 23:48 |
| jeblair | (recently) | 23:49 |
| arosen1 | jeblair: awesome thanks. | 23:49 |
| mordred | lifeless, greghaynes ^^ next time you get 5 minutes, I wouldn't mind some dib expert review on https://review.openstack.org/88479 ... no rush, I know you have things on your plates | 23:49 |
| jeblair | clarkb, mordred: do we actually want to reduce the mysqldumps there? we currently have 1 month's worth of daily dumps | 23:50 |
| jeblair | i kind of like having a bunch there so that if there's a glitch with bup, we can still catch up | 23:50 |
| mordred | jeblair: I can go along with that reasoning | 23:50 |
| *** homeless has joined #openstack-infra | 23:51 | |
| *** jamespage has joined #openstack-infra | 23:52 | |
| *** jamespage has joined #openstack-infra | 23:52 | |
| *** jhesketh has joined #openstack-infra | 23:54 | |
| *** jaypipes has quit IRC | 23:54 | |
| *** mrodden has quit IRC | 23:55 | |
| *** sacharya has joined #openstack-infra | 23:56 | |
| sweston | jeblair: zaro: anteaya: so I can get the docs we discussed out to you this evening. Also, who should I work with in order to complete the upgrade timing? I would like to get started on that tomorrow, if that works for everyone? | 23:57 |
| *** Sukhdev has quit IRC | 23:57 | |
| jeblair | sweston: cool, thanks; probably zaro on upgrade timing | 23:58 |
| mikal | jeblair: huh | 23:58 |
| mikal | jeblair: is that true in all regions? | 23:58 |
| *** gokrokve has joined #openstack-infra | 23:58 | |
| *** mbacchi has quit IRC | 23:58 | |
| sweston | jeblair: always glad to help ;-) | 23:58 |
| jeblair | clarkb, mordred: see etherpad -- if we put /var/lib and /home(/gerrit?) on cinder, then we use 11.7G on / | 23:58 |
| *** sacharya1 has joined #openstack-infra | 23:59 | |
| jeblair | mikal: i don't know; i can check some performance1-8 nodepool nodes real quick | 23:59 |
| clarkb | jeblair: I think that sounds good | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!