| *** Daisy has quit IRC | 00:00 | |
| *** Daisy has joined #openstack-infra | 00:04 | |
| openstackgerrit | JJ Asghar proposed openstack-infra/project-config: [WIP]: Added Standard Checks for Coda https://review.openstack.org/262085 | 00:05 |
|---|---|---|
| *** Daisy has quit IRC | 00:08 | |
| *** masuberu has joined #openstack-infra | 00:09 | |
| *** rlandy has quit IRC | 00:10 | |
| *** masber has quit IRC | 00:12 | |
| *** Daisy has joined #openstack-infra | 00:16 | |
| *** Qiming has quit IRC | 00:16 | |
| *** lxsli_festive has quit IRC | 00:25 | |
| *** lxsli_festive has joined #openstack-infra | 00:25 | |
| *** FallenPegasus has quit IRC | 00:25 | |
| *** woodster_ has joined #openstack-infra | 00:26 | |
| *** Daisy has quit IRC | 00:33 | |
| *** _amrith_ is now known as amrith | 00:38 | |
| *** david-lyle has quit IRC | 00:40 | |
| *** sabeen3 has quit IRC | 00:41 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Replace the Gerrit WIP icon https://review.openstack.org/165228 | 00:44 |
| clarkb | pleia2 zaro we should have plenty memory its just a matter of giving it to the jvm right? | 00:45 |
| clarkb | and if we need more than that a bigger host is an option (but may require new IP depending on how we do that (can size up but thats dowtime)) | 00:46 |
| pleia2 | clarkb: yeah, only using 15G or so on the system | 00:47 |
| pleia2 | so it's fine to bump it to 12G if we want | 00:47 |
| pleia2 | which is what https://review.openstack.org/#/c/262068/3 proposes | 00:48 |
| zaro | yeah, i think 12 G is fine. i think the recomendation is to set heaplimit to about 50% of total RAM. | 00:49 |
| clarkb | 50% would be 15g but +2 as is | 00:50 |
| clarkb | cant babysit or anyrhing today too many errands | 00:50 |
| zaro | or i mean no more than 50%. | 00:50 |
| clarkb | ah ok | 00:50 |
| pleia2 | I will be around, will this auto restart gerrit? | 00:51 |
| clarkb | probably to leave room for the jernel to cache | 00:51 |
| clarkb | pleia2 shouldnt will require manual restart iirc | 00:51 |
| pleia2 | already, well, approving either way | 00:51 |
| pleia2 | alrighty too | 00:51 |
| *** Guest73518 has quit IRC | 00:53 | |
| clarkb | today is actually a nice day for errands, less traffic etc | 00:54 |
| pleia2 | my family fled to go to sonoma for wine tasting | 00:54 |
| pleia2 | I just went a couple weeks ago and I did have a lot of work to catch up on today :\ boooring | 00:55 |
| pleia2 | hehe | 00:55 |
| *** dimtruck is now known as zz_dimtruck | 00:55 | |
| clarkb | and it snowed yesterday | 00:55 |
| pleia2 | neat | 00:55 |
| *** eil397 has quit IRC | 00:57 | |
| openstackgerrit | Merged openstack-infra/system-config: Allow java to allocate more memory for Gerrit https://review.openstack.org/262068 | 00:57 |
| *** _nadya_ has joined #openstack-infra | 01:08 | |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Magnum: Add new job to do mesos functional testing https://review.openstack.org/261894 | 01:13 |
| *** _nadya_ has quit IRC | 01:13 | |
| *** markvoelker has joined #openstack-infra | 01:14 | |
| *** treaki__ has joined #openstack-infra | 01:20 | |
| *** Qiming has joined #openstack-infra | 01:21 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 01:23 | |
| *** angdraug has quit IRC | 01:24 | |
| *** treaki_ has quit IRC | 01:24 | |
| *** FallenPegasus has joined #openstack-infra | 01:27 | |
| *** kun_huang has joined #openstack-infra | 01:35 | |
| *** kevinbenton has quit IRC | 01:37 | |
| *** kevinbenton has joined #openstack-infra | 01:38 | |
| kun_huang | I'm building my jenkins+gerrit, and now my account replied "Build Successful ...". however the text is not colored one and not linked | 01:39 |
| kun_huang | how could I make my auto-reply better? | 01:40 |
| *** bharathm has joined #openstack-infra | 01:44 | |
| *** bharathm has quit IRC | 01:48 | |
| *** david-lyle has joined #openstack-infra | 01:49 | |
| *** rhallisey has quit IRC | 01:49 | |
| yonglihe | dstepanenko_: i got you, which patch are you metion? we will help you investigate if it's a valid result. thanks | 01:56 |
| *** FallenPegasus has quit IRC | 01:57 | |
| *** Qiming_ has joined #openstack-infra | 02:00 | |
| *** Qiming has quit IRC | 02:00 | |
| *** zhurong has joined #openstack-infra | 02:06 | |
| *** FallenPegasus has joined #openstack-infra | 02:07 | |
| *** yamahata has joined #openstack-infra | 02:09 | |
| *** tzn has joined #openstack-infra | 02:14 | |
| *** FallenPegasus has quit IRC | 02:16 | |
| *** zz_dimtruck is now known as dimtruck | 02:16 | |
| *** bhunter71 has joined #openstack-infra | 02:20 | |
| *** tzn has quit IRC | 02:20 | |
| yonglihe | dstepanenko_: i review your change to host manager, there is some pitfall i think, refer my inline comments about that, hope it could help you. | 02:25 |
| *** FallenPegasus has joined #openstack-infra | 02:31 | |
| *** FallenPegasus has quit IRC | 02:31 | |
| *** FallenPegasus has joined #openstack-infra | 02:35 | |
| *** zul has joined #openstack-infra | 02:35 | |
| *** woodster_ has quit IRC | 02:36 | |
| *** unicell has quit IRC | 02:42 | |
| *** dcentral has quit IRC | 02:43 | |
| *** FallenPegasus has quit IRC | 02:47 | |
| *** fawadkhaliq has joined #openstack-infra | 02:54 | |
| *** esker has joined #openstack-infra | 02:56 | |
| *** david-lyle has quit IRC | 02:57 | |
| *** zhurong has quit IRC | 02:59 | |
| *** zhurong has joined #openstack-infra | 02:59 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Update build params before getting swift instruct https://review.openstack.org/262111 | 03:02 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Allow overwriting destination_prefix for swift https://review.openstack.org/262112 | 03:02 |
| *** esker has quit IRC | 03:02 | |
| *** zhurong has quit IRC | 03:10 | |
| *** zhurong has joined #openstack-infra | 03:10 | |
| *** rvasilets___ has quit IRC | 03:15 | |
| *** [1]Thelo has joined #openstack-infra | 03:24 | |
| *** masuberu has quit IRC | 03:24 | |
| *** david-lyle has joined #openstack-infra | 03:26 | |
| *** Thelo has quit IRC | 03:26 | |
| *** [1]Thelo is now known as Thelo | 03:26 | |
| *** sabeen1 has joined #openstack-infra | 03:28 | |
| *** zhurong has quit IRC | 03:31 | |
| *** zhurong has joined #openstack-infra | 03:31 | |
| *** kevinbenton has quit IRC | 03:49 | |
| *** gongysh_ has joined #openstack-infra | 03:52 | |
| *** david-lyle has quit IRC | 03:52 | |
| *** itzdilip_ has joined #openstack-infra | 03:57 | |
| *** markvoelker has quit IRC | 03:58 | |
| *** zhurong has quit IRC | 04:14 | |
| *** ashishb has joined #openstack-infra | 04:15 | |
| *** zhurong has joined #openstack-infra | 04:15 | |
| *** kevinbenton has joined #openstack-infra | 04:18 | |
| *** kevinbenton has quit IRC | 04:18 | |
| *** kevinbenton has joined #openstack-infra | 04:18 | |
| *** harlowja_at_home has joined #openstack-infra | 04:23 | |
| *** harlowja_at_home has quit IRC | 04:24 | |
| *** dims has quit IRC | 04:25 | |
| *** dims has joined #openstack-infra | 04:34 | |
| *** Qiming_ has quit IRC | 04:38 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Update pip-missing-reqs to pip-check-reqs https://review.openstack.org/164974 | 04:41 |
| *** dims has quit IRC | 04:44 | |
| *** dims has joined #openstack-infra | 04:52 | |
| *** fawadkhaliq has quit IRC | 04:56 | |
| *** unicell has joined #openstack-infra | 04:58 | |
| *** markvoelker has joined #openstack-infra | 04:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:58 | |
| *** _et_ has joined #openstack-infra | 04:59 | |
| *** esker has joined #openstack-infra | 05:00 | |
| *** markvoelker has quit IRC | 05:03 | |
| *** unicell has quit IRC | 05:04 | |
| *** esker has quit IRC | 05:05 | |
| *** coolsvap|away is now known as coolsvap | 05:06 | |
| *** unicell has joined #openstack-infra | 05:06 | |
| *** dimtruck is now known as zz_dimtruck | 05:06 | |
| *** Ramanjaneya has joined #openstack-infra | 05:07 | |
| *** Douhet has quit IRC | 05:08 | |
| *** maishsk has quit IRC | 05:09 | |
| *** Douhet has joined #openstack-infra | 05:09 | |
| *** _nadya_ has joined #openstack-infra | 05:10 | |
| *** bhunter71 has quit IRC | 05:10 | |
| *** zz_dimtruck is now known as dimtruck | 05:10 | |
| *** dims has quit IRC | 05:13 | |
| *** _nadya_ has quit IRC | 05:14 | |
| *** gsagie has quit IRC | 05:14 | |
| *** sdake has quit IRC | 05:14 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Add support for appending footers https://review.openstack.org/166596 | 05:18 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/puppet-zuul: Use the latest jquery on zuul https://review.openstack.org/181793 | 05:19 |
| *** zhurong has quit IRC | 05:20 | |
| *** zhurong has joined #openstack-infra | 05:21 | |
| *** fawadkhaliq has joined #openstack-infra | 05:21 | |
| *** dims has joined #openstack-infra | 05:23 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Test osloganalyze in devstack https://review.openstack.org/198728 | 05:23 |
| *** coolsvap is now known as coolsvap|away | 05:25 | |
| *** maishsk has joined #openstack-infra | 05:25 | |
| *** david-lyle has joined #openstack-infra | 05:27 | |
| *** gsagie has joined #openstack-infra | 05:27 | |
| *** maishsk has quit IRC | 05:27 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/infra-specs: Expand on swift logs spec https://review.openstack.org/254718 | 05:31 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Bump pbr https://review.openstack.org/205862 | 05:32 |
| *** maishsk has joined #openstack-infra | 05:32 | |
| *** maishsk has quit IRC | 05:35 | |
| *** maishsk has joined #openstack-infra | 05:37 | |
| *** ashishb has quit IRC | 05:45 | |
| *** changbl has quit IRC | 05:46 | |
| *** maishsk has quit IRC | 05:48 | |
| *** abregman has joined #openstack-infra | 05:48 | |
| *** tzn has joined #openstack-infra | 05:50 | |
| *** rvasilets___ has joined #openstack-infra | 05:53 | |
| *** tzn has quit IRC | 05:57 | |
| *** david-lyle has quit IRC | 05:58 | |
| *** markvoelker has joined #openstack-infra | 06:00 | |
| *** markvoelker has quit IRC | 06:04 | |
| *** otter768 has quit IRC | 06:07 | |
| *** Qiming has joined #openstack-infra | 06:18 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Schedule Gerrit's git garbage collection https://review.openstack.org/262075 | 06:18 |
| *** masuberu has joined #openstack-infra | 06:19 | |
| *** coolsvap|away is now known as coolsvap | 06:19 | |
| *** dcentral has joined #openstack-infra | 06:24 | |
| *** dham1 has joined #openstack-infra | 06:24 | |
| dham1 | hello | 06:26 |
| *** rguillebert has quit IRC | 06:27 | |
| dham1 | I need some assistance please. I need to reset the email account associated with my user account and get my usergroup added to the list of openstack user groups | 06:28 |
| *** david-lyle has joined #openstack-infra | 06:29 | |
| *** irenab has joined #openstack-infra | 06:31 | |
| *** nikil89_ has joined #openstack-infra | 06:34 | |
| *** nikil89_ is now known as naggappan | 06:34 | |
| *** coolsvap is now known as coolsvap|away | 06:34 | |
| *** sabeen1 has quit IRC | 06:34 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/260895 | 06:37 |
| *** _nadya_ has joined #openstack-infra | 06:39 | |
| *** itzdilip_ has quit IRC | 06:40 | |
| *** _nadya_ has quit IRC | 06:41 | |
| *** _denisra has quit IRC | 06:43 | |
| *** FallenPegasus has joined #openstack-infra | 06:44 | |
| *** isaacb has joined #openstack-infra | 06:49 | |
| *** k4n0 has joined #openstack-infra | 06:54 | |
| *** k4n0 has quit IRC | 06:55 | |
| *** masuberu has quit IRC | 06:56 | |
| *** isaacb has quit IRC | 06:59 | |
| *** gampel has joined #openstack-infra | 07:00 | |
| *** markvoelker has joined #openstack-infra | 07:00 | |
| *** pcrews has quit IRC | 07:02 | |
| *** esker has joined #openstack-infra | 07:03 | |
| *** Qiming_ has joined #openstack-infra | 07:03 | |
| *** markvoelker has quit IRC | 07:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use python-jobs template where possible https://review.openstack.org/262141 | 07:05 |
| *** Qiming has quit IRC | 07:06 | |
| *** esker has quit IRC | 07:08 | |
| *** khappone_ has joined #openstack-infra | 07:09 | |
| *** HeOS has quit IRC | 07:11 | |
| khappone_ | Hello. I'm trying to make my first commit to puppet-openstack, and I'm setting up gerrit. review.openstack.org just won't accept my ssh key. I only get "Permission denied (publickey)." I've double checked the pubkey and the username, and tested creating a new key too. Are there any known issues with this? | 07:12 |
| *** sabeen has joined #openstack-infra | 07:17 | |
| *** isaacb has joined #openstack-infra | 07:18 | |
| *** salv-orl_ has joined #openstack-infra | 07:23 | |
| *** gongysh_ has quit IRC | 07:32 | |
| *** _nadya_ has joined #openstack-infra | 07:33 | |
| *** fawadk has joined #openstack-infra | 07:33 | |
| *** maishsk has joined #openstack-infra | 07:33 | |
| *** fawadkhaliq has quit IRC | 07:36 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Replace the Gerrit icon for Workflow -1 vote. https://review.openstack.org/165228 | 07:38 |
| *** kzaitsev_mb has quit IRC | 07:42 | |
| *** bexelbie_ has joined #openstack-infra | 07:48 | |
| *** bexelbie_ is now known as bexelbie | 07:49 | |
| *** fawadk has quit IRC | 07:50 | |
| *** maishsk has quit IRC | 07:52 | |
| *** aopincar has joined #openstack-infra | 07:53 | |
| zaro | jhesketh: ^ answered your questions | 07:55 |
| EmilienM | 503 on gerrit :( | 07:55 |
| *** gampel has quit IRC | 08:00 | |
| *** gampel has joined #openstack-infra | 08:00 | |
| zaro | EmilienM: it seems ok to me. | 08:01 |
| zaro | khappone_: can you login to gerrit and look at settings->agrements? | 08:03 |
| *** boris-42 has quit IRC | 08:03 | |
| khappone_ | zaro: yes, I have this "Verified ICLA OpenStack Individual Contributor License Agreement" | 08:03 |
| zaro | khappone_: you also need the 'new contributor agreement' | 08:04 |
| *** maishsk has joined #openstack-infra | 08:04 | |
| khappone_ | zaro: under taht there's the ICLA, that says "Agreement already submitted." , and for the other ones, it says "DON'T SIGN THIS" | 08:05 |
| *** esikachev has joined #openstack-infra | 08:07 | |
| esikachev | hey guys! | 08:07 |
| EmilienM | zaro: it's random | 08:07 |
| esikachev | can you review https://review.openstack.org/#/c/262051/ ? | 08:07 |
| khappone_ | However, I can't add contact info, when I try to submit it, I get "Server Error, Cannot store contact information" | 08:07 |
| *** fawadkhaliq has joined #openstack-infra | 08:08 | |
| *** otter768 has joined #openstack-infra | 08:08 | |
| anteaya | khappone_: can you read this and follow the suggestions? https://ask.openstack.org/en/question/56720/cannot-store-contact-information-when-updating-info-in-openstack-gerrit/ | 08:09 |
| *** kzaitsev_mb has joined #openstack-infra | 08:10 | |
| *** fawadkhaliq has quit IRC | 08:10 | |
| *** e0ne has joined #openstack-infra | 08:10 | |
| *** fawadkhaliq has joined #openstack-infra | 08:10 | |
| *** HeOS has joined #openstack-infra | 08:11 | |
| khappone_ | anteaya: thanks, that's probably it, I'll look into it | 08:12 |
| anteaya | khappone_: good luck | 08:12 |
| *** otter768 has quit IRC | 08:13 | |
| *** _nadya_ has quit IRC | 08:13 | |
| *** irenab has quit IRC | 08:13 | |
| * anteaya returns to bed | 08:14 | |
| *** lennyb__ has joined #openstack-infra | 08:14 | |
| khappone_ | Now I can save the info, thanks | 08:14 |
| *** masuberu has joined #openstack-infra | 08:14 | |
| *** lennyb_ has quit IRC | 08:15 | |
| *** dimtruck is now known as zz_dimtruck | 08:16 | |
| lennyb__ | zaro: Hi, regarding new gerrit api changes. I've noticed that currently there is only _account_id in the "owner" field instead of "name" that was int he old Gerrit | 08:17 |
| *** salv-orl_ has quit IRC | 08:17 | |
| *** zz_dimtruck is now known as dimtruck | 08:17 | |
| *** maishsk_ has joined #openstack-infra | 08:17 | |
| *** maishsk has quit IRC | 08:18 | |
| *** maishsk_ is now known as maishsk | 08:18 | |
| lennyb__ | zaro: I've posted #link https://review.openstack.org/#/c/259083/ to fix lastcomment monitoring tool, but I prefer to see if there is a change to get "name" in the responce somehow | 08:18 |
| *** FallenPegasus has quit IRC | 08:18 | |
| *** salv-orlando has joined #openstack-infra | 08:23 | |
| *** sshnaidm has joined #openstack-infra | 08:25 | |
| *** dimtruck is now known as zz_dimtruck | 08:27 | |
| *** kzaitsev_mb has quit IRC | 08:28 | |
| *** maishsk has quit IRC | 08:31 | |
| *** maishsk has joined #openstack-infra | 08:32 | |
| *** xek_ is now known as xek | 08:33 | |
| *** ildikov has joined #openstack-infra | 08:35 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:37 | |
| *** rvasilets___ has quit IRC | 08:45 | |
| *** zz_dimtruck is now known as dimtruck | 08:46 | |
| *** e0ne has quit IRC | 08:52 | |
| *** dimtruck is now known as zz_dimtruck | 08:56 | |
| *** markvoelker has joined #openstack-infra | 09:01 | |
| *** salv-orl_ has joined #openstack-infra | 09:01 | |
| *** zhurong has quit IRC | 09:03 | |
| *** salv-orlando has quit IRC | 09:04 | |
| *** _et_ has quit IRC | 09:05 | |
| *** esker has joined #openstack-infra | 09:06 | |
| *** markvoelker has quit IRC | 09:06 | |
| *** Ramanjaneya has quit IRC | 09:06 | |
| *** Ramanjaneya has joined #openstack-infra | 09:07 | |
| *** ildikov has quit IRC | 09:08 | |
| *** kzaitsev_mb has quit IRC | 09:10 | |
| *** vgridnev has joined #openstack-infra | 09:10 | |
| *** esker has quit IRC | 09:11 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:12 | |
| *** kzaitsev_mb has quit IRC | 09:26 | |
| *** tzn has joined #openstack-infra | 09:26 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:28 | |
| *** yamahata has quit IRC | 09:29 | |
| *** irenab has joined #openstack-infra | 09:30 | |
| *** tzn has quit IRC | 09:32 | |
| dulek | There are proxy-errors in Gerrit again in my setup. | 09:34 |
| dulek | "Reason: Error reading from remote server" | 09:34 |
| *** kzaitsev_mb has quit IRC | 09:34 | |
| *** irenab has quit IRC | 09:34 | |
| *** tlian has quit IRC | 09:35 | |
| *** FallenPegasus has joined #openstack-infra | 09:35 | |
| dulek | And now it's 503… | 09:36 |
| tsufiev | same thing | 09:36 |
| mescanef | yep:-( | 09:37 |
| mescanef | 502,503 | 09:37 |
| *** kzaitsev_mb has joined #openstack-infra | 09:38 | |
| tsufiev | even worse than it was yesterday :( | 09:38 |
| *** fawadkhaliq has quit IRC | 09:39 | |
| *** fawadkhaliq has joined #openstack-infra | 09:39 | |
| *** irenab has joined #openstack-infra | 09:39 | |
| *** FallenPegasus has quit IRC | 09:40 | |
| *** claudiub has joined #openstack-infra | 09:42 | |
| *** irenab has quit IRC | 09:42 | |
| *** jschwarz has joined #openstack-infra | 09:44 | |
| jschwarz | review.openstack-org is oh-so-slow today... :< | 09:44 |
| esikachev | guys, please, need review(merge) https://review.openstack.org/#/c/262051/ | 09:45 |
| *** kun_huang has left #openstack-infra | 09:45 | |
| *** kzaitsev_mb has quit IRC | 09:47 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:47 | |
| *** lennyb has joined #openstack-infra | 09:52 | |
| *** kzaitsev_mb has quit IRC | 09:52 | |
| *** ihrachys has joined #openstack-infra | 09:53 | |
| *** tzn has joined #openstack-infra | 09:53 | |
| *** Qiming_ has quit IRC | 09:55 | |
| *** Qiming has joined #openstack-infra | 09:55 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:57 | |
| HeOS | Guys, do you have any estimates to restore your gerrit? | 10:02 |
| *** kzaitsev_mb has quit IRC | 10:03 | |
| *** esker has joined #openstack-infra | 10:07 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:08 | |
| *** jcoufal has joined #openstack-infra | 10:09 | |
| *** otter768 has joined #openstack-infra | 10:09 | |
| *** esker has quit IRC | 10:12 | |
| *** Qiming has quit IRC | 10:13 | |
| *** otter768 has quit IRC | 10:14 | |
| *** kzaitsev_mb has quit IRC | 10:14 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:19 | |
| *** kzaitsev_mb has quit IRC | 10:25 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:26 | |
| *** _nadya_ has joined #openstack-infra | 10:31 | |
| *** kzaitsev_mb has quit IRC | 10:35 | |
| *** sabeen has quit IRC | 10:37 | |
| *** e0ne has joined #openstack-infra | 10:38 | |
| *** jcoufal has quit IRC | 10:39 | |
| *** _nadya_ has quit IRC | 10:41 | |
| *** markvoelker has joined #openstack-infra | 10:47 | |
| *** zz_dimtruck is now known as dimtruck | 10:47 | |
| *** markvoelker has quit IRC | 10:51 | |
| *** fawadkhaliq has quit IRC | 10:53 | |
| *** fawadkhaliq has joined #openstack-infra | 10:53 | |
| *** salv-orl_ has quit IRC | 10:54 | |
| *** salv-orlando has joined #openstack-infra | 10:55 | |
| *** zhurong has joined #openstack-infra | 10:56 | |
| *** salv-orlando has quit IRC | 10:56 | |
| *** salv-orlando has joined #openstack-infra | 10:56 | |
| *** salv-orlando has quit IRC | 10:57 | |
| *** dimtruck is now known as zz_dimtruck | 10:57 | |
| *** salv-orlando has joined #openstack-infra | 10:57 | |
| *** _nadya_ has joined #openstack-infra | 10:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:03 | |
| *** ociuhandu has quit IRC | 11:03 | |
| *** Qiming has joined #openstack-infra | 11:04 | |
| *** ihrachys has quit IRC | 11:04 | |
| *** kzaitsev_mb has quit IRC | 11:08 | |
| *** _tzn has joined #openstack-infra | 11:11 | |
| *** tzn has quit IRC | 11:12 | |
| *** krtaylor has quit IRC | 11:16 | |
| *** masber has joined #openstack-infra | 11:20 | |
| *** apoorvad has joined #openstack-infra | 11:20 | |
| *** masuberu has quit IRC | 11:22 | |
| Thelo | Hi does anyone know how to ask for a "under review version of an os_XXX library in a | 11:26 |
| Thelo | commit | 11:26 |
| *** zz_dimtruck is now known as dimtruck | 11:26 | |
| Thelo | typically I changed os_brick and nova | 11:26 |
| Thelo | and I want the CI to take he version of os_brick under review in my nova patch | 11:27 |
| *** masuberu has joined #openstack-infra | 11:27 | |
| *** Liuqing has joined #openstack-infra | 11:27 | |
| *** mase_x200 has joined #openstack-infra | 11:28 | |
| *** ociuhandu has joined #openstack-infra | 11:29 | |
| *** masber has quit IRC | 11:30 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:30 | |
| amrith | jschwarz, tsufiev, mescanef ... yes, same here. | 11:34 |
| mkoderer | is gerrit completly down now? Get a proxy error | 11:35 |
| dims | mkoderer : ack, me too. | 11:36 |
| *** dimtruck is now known as zz_dimtruck | 11:36 | |
| mkoderer | dims: maybe you have an idea... the tempest kilo gate jobs are broken | 11:37 |
| mkoderer | dims: http://logs.openstack.org/53/223953/5/check/gate-tempest-dsvm-full-kilo/773c1a6/logs/devstacklog.txt.gz#_2015-12-28_13_48_08_754 | 11:37 |
| mkoderer | dims: it seems to be realted to the pbr version that is used. If I update it from 0.11 to 1.1.0 keystone install works fine | 11:38 |
| dims | mkoderer : looks like a new one, i don't see this in the stable-tracker https://etherpad.openstack.org/p/stable-tracker | 11:40 |
| dims | mkoderer : will have to look later as gerrit is totally busted for me now | 11:41 |
| mkoderer | dims: yep thx | 11:41 |
| *** pigmej has joined #openstack-infra | 11:47 | |
| *** kzaitsev_mb has quit IRC | 11:47 | |
| *** rfolco has joined #openstack-infra | 11:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:52 | |
| amrith | gerrit isn't totally down, I managed to get a patch set and a review through. I get very high rate of proxy errors and server unavailable (502, 503) | 11:55 |
| *** apoorvad has quit IRC | 11:55 | |
| *** apoorvad has joined #openstack-infra | 11:58 | |
| pigmej | yeah same there amrith it "sometimes" works | 11:58 |
| *** zhurong has quit IRC | 12:01 | |
| *** Liuqing has quit IRC | 12:01 | |
| *** apoorvad has quit IRC | 12:02 | |
| *** mkj_wk has quit IRC | 12:02 | |
| *** FallenPegasus has joined #openstack-infra | 12:02 | |
| *** mkj_wk has joined #openstack-infra | 12:03 | |
| *** e0ne has quit IRC | 12:04 | |
| *** e0ne_ has joined #openstack-infra | 12:04 | |
| *** xek has quit IRC | 12:04 | |
| *** xek has joined #openstack-infra | 12:05 | |
| amrith | for some suitable value of "sometimes". ymmv. | 12:05 |
| *** zhurong has joined #openstack-infra | 12:05 | |
| *** gampel has quit IRC | 12:06 | |
| *** david-lyle has quit IRC | 12:06 | |
| *** FallenPegasus has quit IRC | 12:07 | |
| *** ildikov has joined #openstack-infra | 12:09 | |
| *** otter768 has joined #openstack-infra | 12:10 | |
| *** esker has joined #openstack-infra | 12:10 | |
| *** otter768 has quit IRC | 12:14 | |
| *** esker has quit IRC | 12:15 | |
| *** jaypipes has joined #openstack-infra | 12:16 | |
| *** masuberu has quit IRC | 12:20 | |
| *** dcentral has quit IRC | 12:21 | |
| *** gsagie has quit IRC | 12:22 | |
| *** Daisy has joined #openstack-infra | 12:27 | |
| *** zhurong has quit IRC | 12:29 | |
| *** zhurong has joined #openstack-infra | 12:30 | |
| *** _tzn has quit IRC | 12:38 | |
| *** ganso has joined #openstack-infra | 12:41 | |
| *** ildikov has quit IRC | 12:41 | |
| *** gsagie has joined #openstack-infra | 12:43 | |
| *** rhallisey has joined #openstack-infra | 12:44 | |
| *** markvoelker has joined #openstack-infra | 12:48 | |
| *** Hal has joined #openstack-infra | 12:48 | |
| *** Hal is now known as Guest56882 | 12:49 | |
| *** Liuqing has joined #openstack-infra | 12:49 | |
| *** isaacb has quit IRC | 12:50 | |
| *** ildikov has joined #openstack-infra | 12:51 | |
| *** markvoelker has quit IRC | 12:52 | |
| *** masuberu has joined #openstack-infra | 12:53 | |
| smcginnis | I had a lot of pain with mod_proxy in a past life. Anyone know if that is what's causing our current issues? | 12:54 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Add check for horizon in sahara-dashboard https://review.openstack.org/262210 | 12:59 |
| *** masuberu has quit IRC | 13:01 | |
| *** zhurong has quit IRC | 13:01 | |
| *** zhurong has joined #openstack-infra | 13:02 | |
| *** fawadkhaliq has quit IRC | 13:03 | |
| *** gampel has joined #openstack-infra | 13:03 | |
| *** Guest56882 has quit IRC | 13:03 | |
| *** Daisy has quit IRC | 13:05 | |
| *** Daisy has joined #openstack-infra | 13:05 | |
| *** sdake has joined #openstack-infra | 13:06 | |
| *** david-lyle has joined #openstack-infra | 13:09 | |
| *** Daisy has quit IRC | 13:10 | |
| *** esker has joined #openstack-infra | 13:11 | |
| *** apoorvad has joined #openstack-infra | 13:12 | |
| *** irenab has joined #openstack-infra | 13:14 | |
| *** esikachev has quit IRC | 13:15 | |
| *** esker has quit IRC | 13:17 | |
| *** apoorvad has quit IRC | 13:17 | |
| *** maishsk has quit IRC | 13:18 | |
| dulek | mkoderer, dims: This is also observable on stable/liberty's grenade check. | 13:19 |
| *** maishsk has joined #openstack-infra | 13:25 | |
| *** [1]Thelo has joined #openstack-infra | 13:25 | |
| *** zz_dimtruck is now known as dimtruck | 13:27 | |
| *** Thelo has quit IRC | 13:27 | |
| *** [1]Thelo is now known as Thelo | 13:27 | |
| *** aopincar has quit IRC | 13:35 | |
| *** Ramanjaneya has quit IRC | 13:35 | |
| *** dimtruck is now known as zz_dimtruck | 13:37 | |
| *** kambiz has quit IRC | 13:37 | |
| *** pleia2 has quit IRC | 13:38 | |
| *** cburgess has quit IRC | 13:38 | |
| *** Clint has quit IRC | 13:38 | |
| *** trey has quit IRC | 13:38 | |
| *** lyz has quit IRC | 13:38 | |
| *** kuromagi has quit IRC | 13:39 | |
| *** waynr has quit IRC | 13:39 | |
| *** kgalanov has joined #openstack-infra | 13:39 | |
| kgalanov | Hello fellows. Maybe I am not the one to ask today, but what is going on with gerrit review ? | 13:42 |
| *** lennyb has quit IRC | 13:44 | |
| *** kzaitsev1ws is now known as kzaitsev_ws | 13:45 | |
| *** maishsk_ has joined #openstack-infra | 13:47 | |
| pigmej | kgalanov: it "just" lags :D | 13:47 |
| *** maishsk has quit IRC | 13:48 | |
| *** maishsk_ is now known as maishsk | 13:48 | |
| *** markvoelker has joined #openstack-infra | 13:48 | |
| *** FallenPegasus has joined #openstack-infra | 13:50 | |
| *** markvoelker has quit IRC | 13:53 | |
| *** FallenPegasus has quit IRC | 13:55 | |
| *** moravec has joined #openstack-infra | 13:57 | |
| *** erlon has joined #openstack-infra | 13:57 | |
| *** Clint has joined #openstack-infra | 13:57 | |
| *** Clint has quit IRC | 13:57 | |
| *** Clint has joined #openstack-infra | 13:57 | |
| *** Daisy has joined #openstack-infra | 13:57 | |
| *** trey has joined #openstack-infra | 13:58 | |
| *** kuromagi has joined #openstack-infra | 13:58 | |
| *** mase_x200 has quit IRC | 14:00 | |
| *** moravec has quit IRC | 14:00 | |
| *** pleia2 has joined #openstack-infra | 14:00 | |
| *** lyz_ has joined #openstack-infra | 14:01 | |
| *** Daisy_ has joined #openstack-infra | 14:01 | |
| *** kambiz has joined #openstack-infra | 14:01 | |
| *** cburgess has joined #openstack-infra | 14:01 | |
| *** amotoki has joined #openstack-infra | 14:02 | |
| *** waynr has joined #openstack-infra | 14:03 | |
| *** Daisy has quit IRC | 14:05 | |
| *** pleia2 has quit IRC | 14:09 | |
| *** pleia2 has joined #openstack-infra | 14:09 | |
| *** tzn has joined #openstack-infra | 14:10 | |
| *** otter768 has joined #openstack-infra | 14:11 | |
| *** ildikov has quit IRC | 14:11 | |
| *** trey has quit IRC | 14:13 | |
| *** trey has joined #openstack-infra | 14:14 | |
| *** otter768 has quit IRC | 14:15 | |
| *** pleia2 has quit IRC | 14:16 | |
| *** pleia2 has joined #openstack-infra | 14:16 | |
| *** fawadkhaliq has joined #openstack-infra | 14:20 | |
| *** pleia2 has quit IRC | 14:24 | |
| *** waynr has quit IRC | 14:25 | |
| *** pleia2 has joined #openstack-infra | 14:33 | |
| *** skolekonov has quit IRC | 14:33 | |
| *** zhurong has quit IRC | 14:34 | |
| *** Liuqing has quit IRC | 14:43 | |
| *** bhunter71 has joined #openstack-infra | 14:44 | |
| *** naggappan has quit IRC | 14:45 | |
| *** Daisy_ has quit IRC | 14:46 | |
| *** rlandy has joined #openstack-infra | 14:46 | |
| *** yamahata has joined #openstack-infra | 14:48 | |
| fungi | notmorgan: pleia2: jhesketh: zaro: looks like you were discussing that ^ in scrollback while i was travelling. might be good to check whatever you wanted to check next once someone is around who isn't still trying to catch up (me) | 14:48 |
| *** bknudson_ has left #openstack-infra | 14:48 | |
| *** bknudson_ has joined #openstack-infra | 14:48 | |
| *** markvoelker has joined #openstack-infra | 14:49 | |
| *** kambiz has quit IRC | 14:50 | |
| *** bhunter71 has quit IRC | 14:52 | |
| *** trey has quit IRC | 14:52 | |
| *** kambiz has joined #openstack-infra | 14:52 | |
| *** waynr has joined #openstack-infra | 14:52 | |
| *** baoli has joined #openstack-infra | 14:52 | |
| *** samyewell has quit IRC | 14:54 | |
| *** markvoelker has quit IRC | 14:54 | |
| *** samyewell has joined #openstack-infra | 14:56 | |
| *** waynr has quit IRC | 14:56 | |
| *** waynr has joined #openstack-infra | 14:56 | |
| *** persia has quit IRC | 14:56 | |
| *** yamahata has quit IRC | 14:59 | |
| *** persia has joined #openstack-infra | 15:01 | |
| *** ashtokolov has quit IRC | 15:01 | |
| *** ams_ has quit IRC | 15:01 | |
| *** salv-orl_ has joined #openstack-infra | 15:02 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:02 | |
| *** trey has joined #openstack-infra | 15:02 | |
| *** Liuqing has joined #openstack-infra | 15:02 | |
| *** ashtokolov has joined #openstack-infra | 15:02 | |
| *** pleia2 has quit IRC | 15:03 | |
| *** unicell1 has joined #openstack-infra | 15:03 | |
| *** pleia2 has joined #openstack-infra | 15:03 | |
| *** salv-orlando has quit IRC | 15:05 | |
| *** ams_ has joined #openstack-infra | 15:05 | |
| *** changbl has joined #openstack-infra | 15:05 | |
| *** unicell has quit IRC | 15:06 | |
| *** dkranz has quit IRC | 15:06 | |
| *** dkranz has joined #openstack-infra | 15:07 | |
| zaro | pleia2, clarkb : did gerrit get restarted after upping memmory? | 15:07 |
| *** vgridnev has quit IRC | 15:11 | |
| *** AJaeger has joined #openstack-infra | 15:11 | |
| AJaeger | fungi, welcome back, hope you had a great christmas vacation. | 15:12 |
| AJaeger | My understanding: gerrit needs more memory, a change has merged for it - now we need to restart gerrit. | 15:12 |
| AJaeger | zaro, correct? | 15:12 |
| fungi | thanks AJaeger. it was a lot of long-distance driving for obligatory family visits. glad to finally be home again | 15:13 |
| zaro | AJaeger: i believe so, not sure if gerrit was already restarted and we are still seeing the slowness or it never got restarted. i'm guessing it's the latter though. | 15:14 |
| mestery | fungi: I think you just summed up christmas vacation quite well there. :) | 15:14 |
| AJaeger | zaro: nobody mentioned that a restart was done in backlog... | 15:14 |
| fungi | zaro: AJaeger: the currently running lvm was last started on december 22 | 15:15 |
| fungi | s/lvm/jvm/ | 15:15 |
| zaro | then it needs a restart | 15:15 |
| fungi | okay, so was there any additional data we needed to collect about the current situation (java melody stats, apache proxy stats, et cetera) before doing that? | 15:15 |
| zaro | i believe clarkb said javamelody was showing abnormally high cpu was going to garbage collection. | 15:17 |
| zaro | not sure if he or pleia2 captured that data though. | 15:17 |
| *** kzaitsev_mb has quit IRC | 15:17 | |
| fungi | interesting, 2.11 no longer presents a monitoring link to take you to the melody interface? | 15:19 |
| * zaro afk, bb in 2hrs. | 15:19 | |
| *** sslypushenko has joined #openstack-infra | 15:20 | |
| zaro | fungi: review.o.o/monitoring ? | 15:20 |
| *** ociuhandu has quit IRC | 15:20 | |
| *** aarefiev has quit IRC | 15:20 | |
| fungi | cool, that works. i wonder why it's no longer linked to anywhere findable when you have permission to access it | 15:21 |
| fungi | memory usage within the jvm is averaging 7gb and reaching 8gb often | 15:23 |
| *** kushal has quit IRC | 15:23 | |
| *** Daisy has joined #openstack-infra | 15:23 | |
| *** kushal has joined #openstack-infra | 15:23 | |
| fungi | cpu utilization within the jvm (no idea how it calculates that?) is at or over 90% most of the time, reaching 100% for some samples | 15:24 |
| fungi | active threads jumped up to 23 at roughly 09:00 utc and has been maxing out there a lot of the time since | 15:25 |
| *** zz_dimtruck is now known as dimtruck | 15:27 | |
| fungi | open file count in the last few hours has climbed to ~2000 | 15:28 |
| *** boris-42 has joined #openstack-infra | 15:28 | |
| fungi | garbage collector time is between 10 and 12 the last few hours... no idea what those units are | 15:29 |
| *** holser_ has joined #openstack-infra | 15:30 | |
| *** ChuckC has quit IRC | 15:31 | |
| *** vgridnev has joined #openstack-infra | 15:32 | |
| *** Qiming has quit IRC | 15:33 | |
| *** baoli has quit IRC | 15:35 | |
| fungi | from the underlying linux system perspective, the jvm is using 15gb of virtual memory, 10gb resident (though there is currently no swap utilization), and about 14gb is in use for buffers and cache | 15:35 |
| *** baoli has joined #openstack-infra | 15:37 | |
| *** ChuckC has joined #openstack-infra | 15:37 | |
| *** FallenPegasus has joined #openstack-infra | 15:39 | |
| fungi | interestingly, fuel-ci has been roughly twice as actively commenting in gerrit as our upstream ci today, according to logs, though the quantity is not high enough that i would expect any relationship with the gerrit load | 15:39 |
| AJaeger | fungi, zaro created also some patches yesterday that might help going forward - https://review.openstack.org/262075 to schedule the git garbage collection, https://review.openstack.org/262058 - put interactive/non-interactive ssh users in separate queues, and from notmorgan https://review.openstack.org/#/c/262065/ to show proxystatus | 15:41 |
| AJaeger | expectation was that the memory increase is the most beneficial | 15:42 |
| *** baoli has quit IRC | 15:42 | |
| fungi | which change added the memory increase? | 15:42 |
| AJaeger | fungi: I67adb239bcb689c2b8764047fd1afefe38911c70 | 15:43 |
| AJaeger | in system-config, last change to merge | 15:43 |
| notmorgan | I want to also add in proxy timeouts | 15:43 |
| notmorgan | But was waiting for zaro's tuning | 15:43 |
| *** FallenPegasus has quit IRC | 15:43 | |
| fungi | thanks, 262068 seems to have merged at 00:57:36 utc today | 15:43 |
| fungi | i'll double-check that it's present on review.o.o to be picked up in the next restart | 15:44 |
| *** davideagnello has quit IRC | 15:44 | |
| *** salv-orl_ has quit IRC | 15:45 | |
| fungi | i see container.heapLimit = 12g in ~/gerrit2/review_site/etc/gerrit.config so that seems right | 15:45 |
| *** salv-orlando has joined #openstack-infra | 15:46 | |
| fungi | so should we go ahead with a gerrit restart now or are there more changes we need to get in? i guess we can do the mod_proxy addition to the server-status page separately since that doesn't need a gerrit restart, just an apache restart | 15:47 |
| *** baoli has joined #openstack-infra | 15:49 | |
| AJaeger | fungi, the two changes from zaro that I references - 262075 and 262058 might need a gerrit restart. But those can be done later after careful review... | 15:50 |
| fungi | yeah, i'm reviewing them already but we need more reviews on them before i feel comfortable approving | 15:51 |
| fungi | currently reading about the sshd batch threads config option in the gerrit docs | 15:51 |
| AJaeger | fungi, I suggest to restart now, review and then restart again in a couple of days. That way we can figure out also whether the memory is really the problem... | 15:52 |
| fungi | sure, sounds reasonable. i'll do that now before my 16:00 utc meeting starts | 15:53 |
| *** baoli has quit IRC | 15:53 | |
| fungi | it's restarting now | 15:54 |
| AJaeger | do you want to send out a #notice? | 15:55 |
| AJaeger | thanks, fungi! | 15:55 |
| *** claudiub|2 has joined #openstack-infra | 15:56 | |
| * AJaeger will answer the emails on openstack-dev and tell that we restarted | 15:56 | |
| fungi | nah, it looks like it's already back up anyway | 15:56 |
| *** sdake has quit IRC | 15:56 | |
| fungi | oh, er maybe that was cached | 15:56 |
| anteaya | garbage collector time units are in percent if I recall correctly from clarkb's comment yesterday | 15:56 |
| fungi | i'm getting a 503 from it now | 15:56 |
| anteaya | the garbage collector time units should be <1% optimally, again going from memory | 15:57 |
| pigmej | fungi: I'm not | 15:57 |
| pigmej | it's pretty random | 15:57 |
| AJaeger | fungi: it's back | 15:57 |
| fungi | weird, yep it's back again | 15:57 |
| pigmej | AJaeger: almost, I was reviewing some stuff, and I have 503 now | 15:57 |
| AJaeger | pigmej: reload - is it ok now? | 15:57 |
| pigmej | it seems so | 15:58 |
| *** claudiub has quit IRC | 15:58 | |
| anteaya | fungi: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-12-28.log.html#t2015-12-28T22:41:54 | 15:58 |
| fungi | anteaya: oh, so that's a percentage of the jvm's cpu capacity or something? | 15:58 |
| *** treaki__ has quit IRC | 15:58 | |
| fungi | it doesn't actually say what units it's in on the melody graph | 15:59 |
| *** baoli has joined #openstack-infra | 16:01 | |
| anteaya | fungi: I don't know what it is a percentage of | 16:02 |
| anteaya | fungi: I just remembered clarkb's comment from yesterday and am sharing | 16:02 |
| anteaya | wish I did know, as that would be more helpful | 16:02 |
| *** jschwarz has quit IRC | 16:02 | |
| anteaya | and welcome back | 16:03 |
| *** lennyb__ has quit IRC | 16:03 | |
| *** Clint has quit IRC | 16:04 | |
| fungi | thanks! | 16:04 |
| *** Clint has joined #openstack-infra | 16:05 | |
| *** Clint has quit IRC | 16:05 | |
| *** Clint has joined #openstack-infra | 16:05 | |
| *** PsionTheory has joined #openstack-infra | 16:05 | |
| *** baoli has quit IRC | 16:05 | |
| *** fawadkhaliq has quit IRC | 16:08 | |
| *** maishsk_ has joined #openstack-infra | 16:09 | |
| *** maishsk has quit IRC | 16:09 | |
| *** maishsk_ is now known as maishsk | 16:09 | |
| *** otter768 has joined #openstack-infra | 16:12 | |
| *** vgridnev has quit IRC | 16:12 | |
| *** kambiz has quit IRC | 16:13 | |
| *** tzn has quit IRC | 16:13 | |
| AJaeger | anteaya: there's a broken non-voting job for sahara-dashboard, do you want to review it, please? https://review.openstack.org/#/c/262051/ | 16:14 |
| *** hdd has joined #openstack-infra | 16:14 | |
| *** aopincar has joined #openstack-infra | 16:14 | |
| *** kambiz has joined #openstack-infra | 16:15 | |
| *** Daisy has quit IRC | 16:16 | |
| *** otter768 has quit IRC | 16:16 | |
| *** gampel has quit IRC | 16:19 | |
| *** sshnaidm has quit IRC | 16:19 | |
| *** baoli has joined #openstack-infra | 16:19 | |
| *** david-lyle has quit IRC | 16:21 | |
| *** aopincar has quit IRC | 16:23 | |
| anteaya | thanks AJaeger | 16:23 |
| *** baoli has quit IRC | 16:23 | |
| anteaya | AJaeger: I still can't recall the correct approach there { or {{ and grepping pre_test_hook in project-config shows me that we have plenty of both | 16:28 |
| stevemar | dhellmann: AJaeger i have something for the osc duplicates, i'm not proud of it, but it works :P | 16:29 |
| anteaya | hey stevemar | 16:29 |
| anteaya | stevemar: any snow for you today? | 16:29 |
| * anteaya looks out the window at the snow | 16:29 | |
| *** EricGonczer_ has joined #openstack-infra | 16:29 | |
| stevemar | anteaya: there was a lot yesterday | 16:29 |
| stevemar | it's slowly becoming a slushy mess | 16:29 |
| dhellmann | stevemar : link? | 16:29 |
| *** tzn has joined #openstack-infra | 16:30 | |
| anteaya | stevemar: nice | 16:30 |
| anteaya | hey dhellmann, did you have nice holidays? | 16:30 |
| *** otter768 has joined #openstack-infra | 16:30 | |
| *** baoli has joined #openstack-infra | 16:30 | |
| dhellmann | hi, anteaya, yes our trip was fun and it was nice to disconnect for a while. you? | 16:30 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: check that osc plugins do not break openstackclient https://review.openstack.org/261828 | 16:31 |
| stevemar | dhellmann: ^ | 16:31 |
| stevemar | anteaya: i had to drive home while it was coming down yesterday | 16:32 |
| stevemar | anteaya: the roads were not plowed at all, saw a bunch of folks in ditches | 16:32 |
| anteaya | dhellmann: nice, very relaxed holidays, quiet, not much over eating, played cards | 16:32 |
| anteaya | stevemar: :( | 16:32 |
| anteaya | stevemar: where were you driving from and to? | 16:32 |
| stevemar | anteaya: aurora to back downtown, not exactly the shortest of drives | 16:33 |
| anteaya | well not that long either | 16:33 |
| anteaya | but all on the 404 | 16:33 |
| stevemar | yep | 16:33 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Use the fallback list in the bindep fallback job https://review.openstack.org/262266 | 16:33 |
| stevemar | i thought once i got into toronto proper it would be alright, but nope | 16:33 |
| anteaya | folks think somehow they are immune when they are on the 400, 404 and 401 | 16:33 |
| anteaya | stevemar: nope | 16:34 |
| anteaya | I don't get people who consider weather a surprise | 16:34 |
| anteaya | they can see it | 16:34 |
| *** FallenPegasus has joined #openstack-infra | 16:34 | |
| fungi | i don't get people | 16:34 |
| fungi | period | 16:34 |
| anteaya | fungi: ha ha ha | 16:34 |
| anteaya | fungi: good family time then? | 16:34 |
| stevemar | anteaya: ¯\_(ツ)_/¯ | 16:34 |
| anteaya | stevemar: glad you are home safe | 16:35 |
| fungi | yes, always good when family interaction can be kept contained to specific spaces and timeframes | 16:35 |
| stevemar | anteaya: me too :) | 16:35 |
| *** baoli has quit IRC | 16:35 | |
| anteaya | fungi: congratulations | 16:35 |
| anteaya | stevemar: :) | 16:35 |
| *** abregman has quit IRC | 16:36 | |
| dhellmann | stevemar : that doesn't look so bad. it avoids reproducing osc logic in the test script, which is good. | 16:37 |
| dhellmann | stevemar : are there any existing duplicate commands? | 16:37 |
| stevemar | dhellmann: nope | 16:37 |
| stevemar | i mangled a setup.cfg in a local heatclient, and run the script, it found the duplicate | 16:38 |
| *** kambiz has quit IRC | 16:38 | |
| pabelanger | timrc: sounds good! I need to finish up template support still for grafyaml. But mostly, just adding missing features people use | 16:39 |
| clarkb | fungi anteaya I believe it to be percentage of cpu time used | 16:39 |
| dhellmann | stevemar : ok, that's good then. | 16:39 |
| anteaya | clarkb: thank you | 16:40 |
| andreykurilin | hi all! Does anyone know what happend with coverage? An example: http://logs.openstack.org/81/243281/8/check/rally-coverage/5a03fef/cover/rally_exceptions_py.html | 16:40 |
| *** kambiz has joined #openstack-infra | 16:41 | |
| boris-42 | andreykurilin: I know | 16:41 |
| andreykurilin | boris-42: new release of coverage package? | 16:41 |
| andreykurilin | :) | 16:41 |
| boris-42 | andreykurilin: the problem is that servers is returning wrong mime types | 16:41 |
| boris-42 | server* | 16:42 |
| boris-42 | andreykurilin: so if you just save this report and open from your local disk they will work | 16:42 |
| stevemar | dhellmann: i didn't like having to split it between python and bash :P | 16:42 |
| *** david-lyle has joined #openstack-infra | 16:42 | |
| andreykurilin | boris-42: oh... | 16:42 |
| dhellmann | stevemar : ah | 16:42 |
| stevemar | dhellmann: at least someone is finally making use of the `openstack command list` command :P | 16:43 |
| *** baoli has joined #openstack-infra | 16:43 | |
| stevemar | dhellmann: if you're wondering how that is done... https://github.com/openstack/python-openstackclient/blob/master/openstackclient/common/commandmanager.py | 16:43 |
| *** ildikov has joined #openstack-infra | 16:43 | |
| anteaya | clarkb: total used cpu time, or user used cpu time? | 16:44 |
| *** EricGonczer_ has quit IRC | 16:44 | |
| *** salv-orlando has quit IRC | 16:45 | |
| mescanef | btw, regarding gerrit, guys, at review.openstack.org do you use 'UseParallerGC'? | 16:45 |
| mescanef | for gerrit | 16:45 |
| dhellmann | stevemar : I think I missed the period when you and dtroyer_zz (and others?) split commands into multiple groups like that | 16:45 |
| *** salv-orlando has joined #openstack-infra | 16:45 | |
| *** maishsk has quit IRC | 16:45 | |
| *** amotoki has quit IRC | 16:45 | |
| clarkb | anteaya its the percentage of time that gerrit spends garbage collecting. so if gerrit uses 50%9% cpu and gc time is 10% it is 10% of 50% | 16:46 |
| *** maishsk has joined #openstack-infra | 16:46 | |
| anteaya | clarkb: thanks | 16:47 |
| *** baoli has quit IRC | 16:47 | |
| smcginnis | Nice work infra folks. Much better response time now. For now... :) | 16:48 |
| anteaya | smcginnis: thanks for the feedback | 16:48 |
| anteaya | smcginnis: yeah we are working on the longer term vison | 16:48 |
| *** david-lyle has quit IRC | 16:48 | |
| *** Liuqing has quit IRC | 16:48 | |
| smcginnis | anteaya: Yeah, things like this usually seem to take a while to work through. But things are good for now, so I'm happy. :) | 16:48 |
| bookwar | hi, fungi, do you have a log snippet for today's Fuel CI activity? we enabled couple of deployment tests, but i don't expect this to be too high | 16:48 |
| anteaya | smcginnis: yay! | 16:49 |
| bookwar | fungi: would be nice if you send me the data via mail for example | 16:49 |
| *** _nadya_ has quit IRC | 16:49 | |
| *** markvoelker has joined #openstack-infra | 16:50 | |
| fungi | bookwar: there's no sensitive data on it. i can for example stick a one-hour sample of the log entries on paste.o.o | 16:50 |
| fungi | bookwar: http://paste.openstack.org/show/482823 it looks like maybe you're adding "build started" messages for every job you're running? | 16:52 |
| *** ChuckC has quit IRC | 16:52 | |
| *** e0ne_ is now known as e0ne | 16:52 | |
| *** Clint has quit IRC | 16:53 | |
| *** pleia2 has quit IRC | 16:53 | |
| fungi | hope that helps | 16:53 |
| *** pleia2 has joined #openstack-infra | 16:53 | |
| *** Clint has joined #openstack-infra | 16:53 | |
| *** baoli has joined #openstack-infra | 16:54 | |
| *** Hal has joined #openstack-infra | 16:54 | |
| *** Hal is now known as Guest77960 | 16:55 | |
| pabelanger | So, -infra back to 100% this week or should I hold off until next week to ask for zuul patch merging :D | 16:55 |
| *** markvoelker has quit IRC | 16:55 | |
| anteaya | mescanef: not that I can see, I have checked out gerrit repo, our system-config repo and our puppet-gerrit module and come up with zero results when grepping for UseParallelGC | 16:55 |
| pabelanger | no rush, FWIW | 16:55 |
| anteaya | mescanef: and I'm not a guy | 16:55 |
| anteaya | mescanef: are you familiar with the use of UseParallelGC? | 16:55 |
| *** FallenPegasus has quit IRC | 16:56 | |
| *** maishsk has quit IRC | 16:56 | |
| anteaya | pabelanger: I have no idea what other people's schedule's are | 16:56 |
| *** david-lyle has joined #openstack-infra | 16:58 | |
| anteaya | I also don't see UseParallelGC in gerritlib, not that I would expect it to be there, but I checked anyway | 16:58 |
| *** baoli has quit IRC | 16:59 | |
| *** david-lyle has quit IRC | 16:59 | |
| fungi | pabelanger: i don't expect infra team activity to ramp back up until next week (and i'll be gone next monday, myself) | 16:59 |
| fungi | likely middle of next week we'll be back in the swing of things again | 17:00 |
| pabelanger | fungi: Great, wfm! | 17:00 |
| *** esker has joined #openstack-infra | 17:00 | |
| mescanef | anteaya: ive heard that by default java uses 'UseSerialGC' and that with 'UseParallerGC' it can perform better along with tuned 'ratio' param. http://www.cubrid.org/blog/dev-platform/how-to-tune-java-garbage-collection/ | 17:00 |
| *** HeOS has quit IRC | 17:00 | |
| * nibalizer yawns; good morning -infra | 17:00 | |
| anteaya | morning nibalizer | 17:01 |
| fungi | mescanef: sounds like something we should definitely look into if zaro hasn't already ruled it out for some reason | 17:01 |
| anteaya | mescanef: thank you | 17:01 |
| anteaya | mescanef: are you familiar with java garbage collection yourself? | 17:01 |
| *** FallenPegasus has joined #openstack-infra | 17:02 | |
| pleia2 | good morning | 17:04 |
| anteaya | morning pleia2 | 17:06 |
| pleia2 | fungi: thanks for handling the gerrit restart | 17:07 |
| fungi | sure thing, just didn't want to step on anyone's toes who had been troubleshooting it up to this point | 17:07 |
| zaro | morning | 17:08 |
| anteaya | morning zaro | 17:08 |
| openstackgerrit | Merged openstack-infra/infra-specs: Move code search spec to Implemented https://review.openstack.org/257628 | 17:08 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: check that osc plugins do not break openstackclient https://review.openstack.org/261828 | 17:09 |
| *** e0ne has quit IRC | 17:09 | |
| *** esker has quit IRC | 17:09 | |
| *** baoli has joined #openstack-infra | 17:10 | |
| pleia2 | ooh yay, code search spec implemented \o/ | 17:10 |
| zaro | mescanef: thanks for the info. we are still looking into ways to tune. | 17:10 |
| pleia2 | I can remove from the agenda | 17:11 |
| fungi | pleia2: feel free. i spotted it when cleaning up the agenda for today | 17:11 |
| fungi | only a couple things on the agenda, one of which will need to be extended/deferred to next week anyway | 17:12 |
| * pleia2 nods | 17:12 | |
| pleia2 | if yesterday was any indication, it will be a very slow week in general dev-wise | 17:12 |
| fungi | mainly because i'm not going to announce a spec objections deadline for new year's eve ;) | 17:13 |
| *** pcrews has joined #openstack-infra | 17:13 | |
| fungi | good week for gerrit tuning | 17:13 |
| openstackgerrit | Merged openstack-infra/infra-specs: Gerrit-2.11 spec is implemented https://review.openstack.org/260694 | 17:13 |
| *** FallenPegasus has quit IRC | 17:14 | |
| * zaro looks forward to that. | 17:14 | |
| anteaya | zaro: have you considered UseParallelGC so far? | 17:15 |
| *** baoli has quit IRC | 17:15 | |
| zaro | anteaya: will investigate it soon. | 17:15 |
| anteaya | zaro: okay great | 17:16 |
| anteaya | fungi: not sure where you stand on this but if you recall cathy's request to rebuilt their fuel plugin repo from last week, I spotted this yesterday: https://review.openstack.org/#/c/261966/ | 17:16 |
| anteaya | fungi: I might be reading this wrong, but it is possible that they have decided to just spin up a new repo rather than address their current concern | 17:17 |
| anteaya | s/rebuilt/rebuild | 17:17 |
| nibalizer | zaro: are we doing any tuning of the permgen/oldgen size? | 17:17 |
| nibalizer | and do we have jmx enabled? | 17:18 |
| anteaya | fungi: not sure if that is worth a meeting agenda item or not | 17:18 |
| *** bhunter71 has joined #openstack-infra | 17:19 | |
| zaro | nibalizer: no, permgen is not tuned. i'm unaware of 'oldgen'. | 17:19 |
| anteaya | my understanding from last week was that cathy was going to post to the infra mailing list regarding her request and I haven't seen that happen | 17:19 |
| fungi | i don't recall whether i was around for a discussion about rebuilding a repo | 17:19 |
| anteaya | fungi: actually you were, yes | 17:20 |
| zaro | nibalizer: the javamelody plugin provies jmx monitoring | 17:20 |
| fungi | doesn't surprise me ;) | 17:20 |
| zaro | #provies/provides | 17:20 |
| anteaya | fungi: that was tuesday night I believe | 17:20 |
| clarkb | nibalizer we appear to be heap bound | 17:20 |
| nibalizer | zaro: okay, i'll look after I get un-lazy and load my ssh key | 17:20 |
| anteaya | fungi: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-12-23.log.html#t2015-12-23T02:06:40 | 17:21 |
| clarkb | permgen/oldgen is not heap | 17:21 |
| anteaya | fungi: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-12-23.log.html#t2015-12-23T03:01:51 | 17:21 |
| zaro | clarkb: yeah, it's not tuned. | 17:21 |
| clarkb | ya but I dont think that is the problem | 17:21 |
| zaro | ohh, i see what you mean | 17:22 |
| clarkb | maybe we need more but that doesnt address the time spent in gc | 17:22 |
| anteaya | fungi: you were there from 2:19 to 2:44 | 17:22 |
| *** unicell1 has quit IRC | 17:22 | |
| fungi | anteaya: so to summarize, the content of some repo needs to get reset to a new state, and then the repo needs to be renamed to openstack/fuel-plugin-onosfw? | 17:22 |
| anteaya | fungi: but if you missed the requests to rebuild the repo I can understand that, but my takeaway is that this is what she was asking for | 17:22 |
| anteaya | fungi: I don't know | 17:23 |
| zaro | i've already pushed a few reviews for gerrit tuning, all with topic 'gerrit-tuning' | 17:23 |
| anteaya | fungi: I don't have enough detail myself, but my concern is that they will just keep creating new things and abandoning things they don't want | 17:23 |
| anteaya | fungi: as that seems to be their current workflow on the onos repo | 17:23 |
| docaedo | fungi: the issue from what I gathered was that they had accidentally pushed a 130mb tar file into the repo, and that seemed to break replication to github | 17:24 |
| fungi | oh | 17:24 |
| anteaya | docaedo: in this situation yes, I'm wondering about their current approach and understanding of our workflow | 17:24 |
| fungi | now i remember | 17:24 |
| fungi | thanks docaedo | 17:24 |
| anteaya | docaedo: as they seem to like the approach of abandoning things | 17:24 |
| docaedo | fungi: glad to help :) | 17:24 |
| docaedo | yeah the workflow issues .. I have nothing constructive to add there | 17:25 |
| anteaya | docaedo: yeah, me either, but I'm not a fan of what I am seeing so far | 17:25 |
| fungi | so anyway, the urgency for them was that they seem to put some priority on github and didn't want to use git.o.o, even though the github mirror was the one which was really the issue | 17:25 |
| docaedo | but I could understand at least the concern of having that one relatively giant lump in their repo :) | 17:25 |
| anteaya | fungi: that was my understanding of their definition of emergency, yes | 17:26 |
| fungi | docaedo: the openstack-manuals repo has more deleted content than that one | 17:26 |
| docaedo | fungi: oh yeah, I know there are lots of other repos with heaps more old stuff in them. I'm not saying it's in any way an emergency, or even necessarily has to be resolved | 17:27 |
| docaedo | TBH since there was no followup to the mailing list, I think it's an issue that does not need to be addressed until someone brings it up again | 17:27 |
| anteaya | docaedo: well they are wanting to create a new repo | 17:27 |
| anteaya | to my mind, that is bringing it up again | 17:27 |
| docaedo | Oh ok I didn't know that | 17:27 |
| fungi | this makes me want to finally get rid of our github copies of stuff in general. it's an attractive nuisance what with people constantly opening new issues and pull requests there not realizing they're read-only replicas nobody's looking at | 17:27 |
| anteaya | docaedo: https://review.openstack.org/#/c/261966/ | 17:28 |
| fungi | docaedo: yeah, they've decided to abandon it and add a new one apparently | 17:28 |
| docaedo | anteaya: thanks | 17:28 |
| anteaya | docaedo: yeah, that was my quesetion to fungi | 17:28 |
| anteaya | if they are content to let it lie, I don't care, but it seems that they don't want to take our suggestions but do their own thing | 17:28 |
| docaedo | fungi: even though I said on the ML that I *like* the rendering of the README when looking at a repo on github, I would be in support of dropping those read-only copies as well | 17:28 |
| *** otter768 has quit IRC | 17:29 | |
| anteaya | I would support not mirroring to github, if that was the decision | 17:29 |
| fungi | i'm not opposed to setting a max commit size limit in gerrit going forward either, but i wouldn't want github's choice of commit filters/limits to determine ours as a policy | 17:33 |
| *** baoli has joined #openstack-infra | 17:35 | |
| notmorgan | fungi: a maxcommit size would be a nice thing to prevent <random binary> from entering the repos | 17:35 |
| notmorgan | fungi: but ++ on not using github's without evaluating if that is the right answer | 17:35 |
| anteaya | fungi: agreed | 17:36 |
| notmorgan | the issue with dropping the R/O mirrors is that a lot of folks still use github. it would worry me to generally drop the external r/o mirrors since there is a lot of history there that people use it. | 17:37 |
| pabelanger | fungi: downside to now mirroring to github, is lack of exposure IMO. Even thought we don't develop a top of it, still a great tool to find projects IMO | 17:37 |
| pabelanger | notmorgan: ++ | 17:37 |
| *** izaakk has joined #openstack-infra | 17:37 | |
| *** sdake has joined #openstack-infra | 17:37 | |
| *** bhunter71 has quit IRC | 17:38 | |
| notmorgan | unless you can make github 301 to the right place on git.openstack.org [don't think this is possible] and have git obey said 301 | 17:38 |
| AJaeger | anteaya: sorry, took me a bit longer to come back | 17:38 |
| *** bhunter71 has joined #openstack-infra | 17:39 | |
| *** maishsk has joined #openstack-infra | 17:39 | |
| anteaya | AJaeger: no worries, you are allowed to have a life | 17:39 |
| AJaeger | anteaya: we use {{ if there's a variable to substitute and { if not. So, job-templates use {{ but jobs not. The change is a job and thus needs { | 17:39 |
| anteaya | well openstack.org is the top hit for me when searching for openstack nova source so perhaps linking to git.o.o from there might help the search engines | 17:39 |
| AJaeger | fungi: Regarding the spec, I'm fine with postponing the voting, I put it on the agenda to get early feedback... | 17:40 |
| fungi | notmorgan: pabelanger: yeah, i expect it would be contentious enough that it wouldn't happen without some more compelling reason than "attractive nuisance" unfortunately | 17:40 |
| anteaya | AJaeger: ah thank you, hoping your explaination sticks | 17:40 |
| fungi | AJaeger: sure, i plan to mention it in the meeting, just will set feedback deadline to late next week rather than this week | 17:40 |
| *** baoli has quit IRC | 17:41 | |
| fungi | so anyway, for the current situation with the "broken" (not replicating to github) repo, we need to determine whether there is a safe way to filter a commit out of gerrit when it's associated with an approved change | 17:41 |
| fungi | would appreciate input from zaro on that | 17:41 |
| fungi | we have a similar request from amrith yesterday to redact a commit from trove which was uploaded to gerrit (though in that case not approved/merged) | 17:42 |
| anteaya | fungi: guess that request was conveyed in a pm | 17:43 |
| fungi | anteaya: it was in scrollback in here too | 17:43 |
| fungi | oh, 19:07 utc but details were in a pm i guess | 17:44 |
| anteaya | yeah, I see reference to a pm | 17:45 |
| anteaya | which is fine | 17:45 |
| anteaya | that tells me enough | 17:45 |
| *** yamahata has joined #openstack-infra | 17:45 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:45 | |
| *** baoli has joined #openstack-infra | 17:46 | |
| clarkb | fungi: I am not sure that such a thing exists with git | 17:47 |
| fungi | anyway, i've responded to his pm now saying that's not a feature of gerrit (review deletion) and asking whether he's comfortable providing more details in public | 17:47 |
| clarkb | there are unsafe methods of rewriting history | 17:47 |
| fungi | clarkb: yep, it would have to be a git push of a revised repo with no named refs carrying the commit in question in their history so that gc removes it | 17:48 |
| openstackgerrit | Merged openstack-infra/project-config: Fix to recover gate-sahara-dashboard-dsvm-integration https://review.openstack.org/262051 | 17:48 |
| fungi | but i'm wondering more what that does to gerrit, since it then has references in its database to a commit which is not in the backing repo | 17:48 |
| anteaya | if redacting something from gerrit were possible, we also woul dhave to redact from our git farm and github as the uploaded patch is mirrored out to them too as a ref | 17:48 |
| clarkb | fungi: I imagine its very similar to hte behavior we saw when the packfiles would be treated as corrupt | 17:49 |
| *** FallenPegasus has joined #openstack-infra | 17:49 | |
| clarkb | fungi: the change will fail to load because gerrit is unable to get diff details from git | 17:49 |
| fungi | yep, a force push of a state of the repo with no references to the "problem" commit would cause it to eventually disappear from those replica repositories as well since it would be a candidate for gc | 17:50 |
| fungi | but yeah, gerrit database surgery is what i'm more concerned about with this | 17:50 |
| *** AJaeger has quit IRC | 17:51 | |
| *** AJaeger has joined #openstack-infra | 17:51 | |
| anteaya | oh okay | 17:51 |
| *** baoli has quit IRC | 17:51 | |
| anteaya | fair enough | 17:52 |
| *** yolanda has quit IRC | 17:53 | |
| fungi | so anyway, we have 3 issues in play i think: 1. there's no way to prevent people from accidentally pushing changes to gerrit when they don't intend to, 2. there are some commits which downstream replication may refuse to allow, 3. removing all traces of a commit from git/gerrit is complex enough that we probably don't want to offer to support that as a "fix" | 17:54 |
| anteaya | yes | 17:56 |
| anteaya | I would agree with that | 17:56 |
| zaro | fungi: does the commit completely break replication or is it only broken for that one commit? | 17:56 |
| fungi | zaro: that commit can't be replicated, and so any commits which descend from it can't either | 17:56 |
| fungi | github's replica of the repo in question basically has the branch at the last state it was in prior to that commit merging | 17:57 |
| fungi | and can't update beyond there | 17:57 |
| *** yolanda has joined #openstack-infra | 17:58 | |
| zaro | so if we ask replication to ignore that commit you think it will continue to work on descendants? | 17:58 |
| *** aopincar has joined #openstack-infra | 17:59 | |
| anteaya | I'm considering requesting that openstack.org have a page with links to the projects on git.openstack.org, like nova source: http://git.openstack.org/cgit/openstack/nova/tree/ | 17:59 |
| AJaeger | anteaya: We can easily add something to docs.openstack.org as well ;) | 18:00 |
| anteaya | as finding the link to the nova source via the current setup is nigh impossible | 18:00 |
| anteaya | AJaeger: okay but my search for openstack nova source came up with openstack.org as the top hit | 18:00 |
| zaro | anteaya: might as well stop replicating to github then | 18:00 |
| anteaya | zaro: I'm in favour of that as a solution | 18:00 |
| fungi | zaro: i don't think so, because it then the branch couldn't be pulled from ever (it would have holes in its history) | 18:01 |
| fungi | zaro: pretty sure that if we don't try to replicate a commit which is part of the branch history, it will either be rejected as incomplete or else will result in an unusable branch state | 18:02 |
| *** sabeen has joined #openstack-infra | 18:02 | |
| AJaeger | anteaya: Let's add a "Source Code" to the footer... | 18:02 |
| AJaeger | ... which is shared by most pages | 18:03 |
| AJaeger | anteaya: I wanted to suggest to add links to both domains | 18:03 |
| fungi | workaround i guess would be to instead push --force a revised branch state which doesn't include the large commit in its history, but not bother deleting it from the repo entirely | 18:03 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: use send-announcements-to metadata for deliverables https://review.openstack.org/262281 | 18:04 |
| AJaeger | anteaya: there's a Software->Source code link on the top on www.openstack.org that goes to http://docs.openstack.org/releases/releases/liberty.html | 18:04 |
| fungi | gerrit will probably continue trying to push the review ref to the downstream replica and it would be repeatedly rejected, though the branch would be able to update via replication again at least | 18:04 |
| *** baoli has joined #openstack-infra | 18:04 | |
| zaro | fungi: that workaround seems better than trying to delete. | 18:06 |
| anteaya | https://bugs.launchpad.net/openstack-org/+bug/1529921 | 18:07 |
| openstack | Launchpad bug 1529921 in openstack-org "include links to source code on git.openstack.org" [Undecided,New] | 18:07 |
| zaro | also the replication plugin only allows filtering at project level, https://gerrit.googlesource.com/plugins/replication/+/master/src/main/resources/Documentation/config.md | 18:07 |
| anteaya | AJaeger: I like your suggestions too | 18:08 |
| zaro | anteaya: having both redirect and replication is slightly confusing. i vote for one or the other. | 18:08 |
| *** baoli has quit IRC | 18:09 | |
| nibalizer | zaro: I don't see any jmx listeners on review.o.o, can you point me in the right direction? | 18:10 |
| AJaeger | I fear a few "stackforge" like projects will object if we stop replicating to github - as well as lots of documents in our repos that use github as reference like "Anvil's official repository is located on GitHub at: https://github.com/openstack/an" - just the first hit on codesearch... | 18:10 |
| anteaya | AJaeger: many folks will object | 18:10 |
| anteaya | AJaeger: however we mirror as a courtesy | 18:11 |
| anteaya | AJaeger: the supported repos are at git.openstack.org | 18:11 |
| anteaya | folks forgetting that doesn't change that | 18:11 |
| zaro | nibalizer: review.o.o/monitoring | 18:12 |
| AJaeger | anteaya: I agree with you and review accordingly ;) I'm just saying that enforcing it will break stuff and assumptions | 18:12 |
| nibalizer | zaro: thats just a web page? | 18:12 |
| anteaya | AJaeger: well perhaps we should clarify the eroneous assumptions | 18:13 |
| * AJaeger fixes the only problem in openstack-manuals now ;) | 18:13 | |
| AJaeger | anteaya: YES, we should | 18:13 |
| *** unicell has joined #openstack-infra | 18:13 | |
| anteaya | AJaeger: anvil is free to do as they like but if they want to be an openstack project, their official repo is at git.o.o | 18:13 |
| *** FallenPegasus has quit IRC | 18:14 | |
| *** david-lyle has joined #openstack-infra | 18:14 | |
| pabelanger | I must have missed it, but what issue does removing the replication to github.com solve? | 18:14 |
| *** jsavak has joined #openstack-infra | 18:14 | |
| anteaya | github limits the size of a given commit | 18:15 |
| anteaya | there is a repo that merged a commit with archived files in it, it is very large | 18:15 |
| anteaya | the history between the repo on git.openstack.org and github has diverged | 18:15 |
| anteaya | the repo on github can no longer accept merged commits as they are decendents of the blocked commit | 18:16 |
| AJaeger | how can we replace https://github.com/openstack/nova/archive/master.zip with git.openstack.org? | 18:16 |
| AJaeger | Is there a way to download a zip archive? | 18:16 |
| fungi | AJaeger: https://tarballs.openstack.org/nova/nova-master.tar.gz | 18:17 |
| *** mrmartin has joined #openstack-infra | 18:18 | |
| fungi | we actively discourage consuming github tarballs/zipfiles since they're not directly installable (not an sdist/wheel) and drop the git metadata necessary to turn them into one | 18:18 |
| *** baoli has joined #openstack-infra | 18:18 | |
| fungi | if we could turn that github feature off, we would | 18:19 |
| zaro | nibalizer: i believe you get the same thing with javamelody. that is the page for javamelody monitoring | 18:19 |
| nibalizer | i think replicating to github is good, I think we should preserve that feature if we can | 18:19 |
| fungi | same with github pull requests. the only reason we have a pull request auto-closer is because github doesn't let you disable pull request submission | 18:19 |
| anteaya | nibalizer: supporting that feature is becoming more and more work | 18:20 |
| AJaeger | fungi, this is part of windows instructions that use unzip ;( | 18:20 |
| fungi | consider that pretty much all the github "features" besides browsing git commits and cloning/pulling are things we'd prefer to disable | 18:20 |
| *** sdake has quit IRC | 18:21 | |
| *** e0ne has joined #openstack-infra | 18:21 | |
| fungi | AJaeger: that sounds like the windows instructions were written by someone who was in a hurry to put together something they could make work from stuff they were aware of, rather than figure out how to use the resources our community supports | 18:22 |
| *** sdake has joined #openstack-infra | 18:22 | |
| anteaya | AJaeger: well that's the thing, isn't it, the longer we use something we don't support the more attached folks get to it making decisions that are contrary to what we do support | 18:22 |
| AJaeger | exactly - and looking atgain, I'm wrong, it's not Windows - so I can change it;) | 18:22 |
| anteaya | and then we are stuck with their decisions | 18:22 |
| *** moravec has joined #openstack-infra | 18:22 | |
| *** PsionTheory has quit IRC | 18:23 | |
| *** baoli has quit IRC | 18:23 | |
| *** david-lyle has quit IRC | 18:24 | |
| nibalizer | fungi: i know a lot of people who prefer to browse our code via the github interface | 18:25 |
| nibalizer | everyone kindof gets that gerrit is the code review system but a lot of people use github as the clone/browse | 18:26 |
| fungi | nibalizer: yep, i mentioned those as the features we're not seeking a way to turn off | 18:26 |
| fungi | however, not being able to block github's other "features" means that the ability to browse and clone/pull from github has to be such a convenience that it outweighs the attractive nuisances which come along with it | 18:27 |
| pleia2 | nibalizer: indeed, I've worked with several folks who grumble a lot when I try to get them to use git.o.o | 18:27 |
| *** Daisy has joined #openstack-infra | 18:27 | |
| greghaynes | I know a couple deployment tools also point at gh by default to help prevent folks DOSing our git service | 18:28 |
| pleia2 | I dutifully grumble back, but I understand, they use it for everything else so the interface is familiar | 18:28 |
| crinkle | stopping replication to github seems like a bit of an overreaction to me | 18:28 |
| crinkle | github didn't accept a problematically huge commit, that seems like a feature | 18:28 |
| crinkle | now it's stuck in git.o.o and hindering its developers | 18:28 |
| fungi | crinkle: well, we've arguably already stopped replicating one repo to github because they're filtering a merged commit | 18:28 |
| AJaeger | anteaya, fungi: FYI, Patch for openstack-manuals https://git.openstack.org/cgit/openstack/diskimage-builder/elements | 18:29 |
| fungi | the hinderance to that repos developers is that they want to use github and not git.o.o | 18:29 |
| fungi | it was already pointed out to them that they could clone successfully from git.o.o | 18:29 |
| crinkle | it's a hinderence to clone that now-huge repo from anywhere | 18:29 |
| pleia2 | crinkle: heh, point | 18:29 |
| fungi | and the commit in their repo is still smaller than the deadweight deleted files in the openstack-manuals repo which all the docs developers live with | 18:29 |
| AJaeger | crinkle: that repo is not huge - only 170 MB ;) | 18:30 |
| anteaya | the point is that they didn't bother to follow our workflow, but having a large enough pool of reviews to catch the issue | 18:30 |
| crinkle | hmm okay, i overestimated how big it was | 18:30 |
| anteaya | and now they want to abandon the repo and create a new one (I believe) | 18:31 |
| anteaya | s/but/by | 18:31 |
| anteaya | s/reviews/reviewers | 18:31 |
| anteaya | and folks grumble about new screen contstantly yet we still moved to gerrit 2.11 | 18:31 |
| *** baoli has joined #openstack-infra | 18:32 | |
| *** jpeeler has quit IRC | 18:32 | |
| *** Daisy has quit IRC | 18:32 | |
| fungi | i don't want to encourage slash-and-burn abandonment/creation of repos, so was thinking through whether a rewritten master branch of that repo omitting the problem file would be likely to generate new breakage | 18:32 |
| fungi | unless they're doing a clone --mirror they're not going to retrieve that commit so long as it's no longer in the branch history | 18:33 |
| clarkb | they will need to reset --hard though | 18:33 |
| *** kdas_ has joined #openstack-infra | 18:34 | |
| fungi | right. but that's not necessary more disruptive than abandoning the repo and starting a new project | 18:34 |
| *** kushal has quit IRC | 18:35 | |
| nibalizer | I think a force push is the better way to fix this than a new repo | 18:35 |
| greghaynes | AJaeger: What is the context for that openstack-manuals patch WRT DIB? | 18:36 |
| pabelanger | nibalizer: +1, if we are voting | 18:36 |
| nibalizer | I saw earlier that someone (zaro?) found a gerrit setting to prevent objects above a size from being submitted, did we do that? | 18:36 |
| *** ewilson has joined #openstack-infra | 18:36 | |
| *** izaakk has quit IRC | 18:36 | |
| *** baoli has quit IRC | 18:36 | |
| anteaya | nibalizer: it is an acl setting | 18:36 |
| pabelanger | anteaya: do we have any docs that say not to use github? | 18:37 |
| anteaya | nibalizer: we would have to have a discussion about what size the limit should be and then change the acl files, or allow repos to opt in | 18:37 |
| pabelanger | I only ask, cause I default all my upstream to github for my repos | 18:37 |
| pabelanger | my remote upstream | 18:37 |
| fungi | we haven't ever told people not to use <whatever> since that's not within our control. we try to lead by example instead, and avoid mentioning services in our documentation if we don't officially support their use | 18:38 |
| anteaya | pabelanger: http://docs.openstack.org/infra/system-config/gerrit.html#github-integration | 18:38 |
| fungi | sticking to git.openstack.org in all our examples and published recommendations | 18:38 |
| AJaeger | greghaynes: that openstack-manuals should have nothing to do with DIB - I'm confused by your questio | 18:38 |
| anteaya | AJaeger: your link in scrollback mentions diskimagebuilder | 18:39 |
| greghaynes | AJaeger: Ah, I just saw your comment of "Patch for openstack-manuals https://git.openstack.org/cgit/openstack/diskimage-builder/elements" | 18:39 |
| greghaynes | AJaeger: Wondering if theres something I should help with ;) | 18:39 |
| AJaeger | anteaya: ah - | 18:39 |
| anteaya | AJaeger: which is broken by I was trying to eat my lunch and was going to mention it once I had finished | 18:39 |
| anteaya | s/by/but | 18:39 |
| nibalizer | "GitHub will warn you when pushing files larger than 50 MB. You will not be allowed to push files larger than 100 MB." | 18:39 |
| AJaeger | greghaynes: I pasted wrong URL, I wanted to paste https://review.openstack.org/262286 | 18:39 |
| nibalizer | I think setting a 100MB file limit is reasonable | 18:40 |
| AJaeger | thanks anteaya for reading what I wrote - I obviously didn't ;( | 18:40 |
| greghaynes | AJaeger: gotcha | 18:40 |
| clarkb | fwiw gerrit does have a limit but its closer to 800mb iirc | 18:40 |
| AJaeger | greghaynes: thanks for offering to help. That openstack-manuals patch indeed touches a link to dib if - that's all. Feel free to review ;) | 18:40 |
| anteaya | AJaeger: no worries | 18:41 |
| pabelanger | fungi: Right. So, educating seems like a good solution too. TIL: a reason not to do `git remote add upstream github.com...` | 18:41 |
| AJaeger | anteaya: enjoy lunch! | 18:41 |
| anteaya | AJaeger: trying to | 18:41 |
| fungi | as i said earlier, i don't mind us choosing a sensible commit size (i don't think we can directly limit by file size?) but wouldn't want to define "sensible" as "one of our downstream mirrors thinks this is sensible for them" | 18:41 |
| clarkb | hrm actually docs say default is inifinit? | 18:41 |
| zaro | nibalizer: i think you would like this https://review.openstack.org/#/c/260324/ | 18:41 |
| anteaya | nibalizer: I think fungi made an excellent point in scrollback that github doesn't set our policies | 18:41 |
| nibalizer | oh neat | 18:42 |
| fungi | i especially don't want to get into a habit of playing chanse with whatever new limits github decides to set in the future. when we discover ways in which the features of github which our community does find marginally beneficial break for us, it's a good opportunity to reassess whether the features we're using there are as convenient as they seem | 18:43 |
| fungi | s/chanse/chase/ | 18:43 |
| nibalizer | fungi: thats why I was trying to speak for the pepole I know who use github | 18:44 |
| nibalizer | I think it provides value | 18:44 |
| *** ociuhandu has joined #openstack-infra | 18:44 | |
| *** baoli has joined #openstack-infra | 18:44 | |
| *** aks has joined #openstack-infra | 18:45 | |
| fungi | i'm sure it does provide some value for some people, but it also apparently requires some compromises | 18:45 |
| pabelanger | what are the files that are >100MB anyways? Some sort of tarball? | 18:46 |
| anteaya | pabelanger: yes | 18:46 |
| fungi | such as making sure that commits we allow are at most a subset of commits they allow. and if we decide out of fairness to start replicating to bitbucket and gitlab too, then what we accept ends up needing to be the intersection of what all our downstream replicas will allow | 18:46 |
| anteaya | pabelanger: the project has two contributors I don't know what level of understanding of our tools they posess | 18:46 |
| anteaya | pabelanger: one person submits, the other one merges | 18:47 |
| anteaya | they have more abandoned patches that any other status | 18:47 |
| anteaya | s/that/than | 18:47 |
| anteaya | fungi: I think that puts us in a poor place to serve the needs of our developers | 18:48 |
| pabelanger | github does provide Git LFS, specifically for large files. However, I'm onboard with educating people that sourcing >100MB tarballs isn't the best use case for git | 18:48 |
| nibalizer | I believe bitbucket - the public git service - died | 18:48 |
| greghaynes | nibalizer: wah? | 18:49 |
| *** baoli has quit IRC | 18:49 | |
| nibalizer | greghaynes: eh https://bitbucket.org/ ? | 18:49 |
| nibalizer | or am I thinking of gitorious? | 18:49 |
| greghaynes | nibalizer: Ya, it is still a thing | 18:49 |
| greghaynes | nibalizer: yes, that died | 18:49 |
| greghaynes | nibalizer: they got bought by gitlab | 18:49 |
| nibalizer | this thing ? https://gitorious.org/ | 18:49 |
| nibalizer | was bitbucket ever a public git service or am I just crazy? | 18:50 |
| greghaynes | nibalizer: it always was a public code hosting service | 18:50 |
| anteaya | pabelanger: I'm going with the belief the majority of our developers already know that | 18:50 |
| greghaynes | nibalizer: They didnt support git for a while | 18:50 |
| clarkb | bitbucket hosts tox | 18:50 |
| clarkb | its a thing | 18:50 |
| fungi | nibalizer: bitbucket provides public hg and git hosting | 18:50 |
| anteaya | pabelanger: the problem is that these folks are off in a corner doing whatever they please and infra has to support their development | 18:50 |
| anteaya | pabelanger: so now we have to discuss what that means | 18:51 |
| *** markvoelker has joined #openstack-infra | 18:51 | |
| nibalizer | ahh ok | 18:51 |
| anteaya | pabelanger: as they currently have the expectation we will rebuild a repo for them whenever they make a mistack | 18:51 |
| anteaya | mistake* | 18:51 |
| *** angdraug has joined #openstack-infra | 18:52 | |
| pabelanger | anteaya: who is they in this context? | 18:52 |
| anteaya | fuel-plugin-onos | 18:52 |
| *** waynr has quit IRC | 18:52 | |
| anteaya | https://review.openstack.org/#/c/261966/ | 18:53 |
| anteaya | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-12-23.log.html#t2015-12-23T01:47:15 | 18:53 |
| anteaya | https://review.openstack.org/#/c/255719/ | 18:54 |
| anteaya | note the number of abandoned patches: https://review.openstack.org/#/q/project:openstack/fuel-plugin-onos | 18:54 |
| nibalizer | fungi: ah okay I was mistaken | 18:54 |
| *** Guest77960 has quit IRC | 18:55 | |
| nibalizer | setting a file limit of under 100MB seems reasonable for us to do, though | 18:56 |
| *** jsavak has quit IRC | 18:56 | |
| *** baoli has joined #openstack-infra | 18:56 | |
| *** markvoelker has quit IRC | 18:56 | |
| fungi | i think we're stuck setting a commit size limit instead, if memory serves | 18:56 |
| fungi | zaro: ^? | 18:56 |
| clarkb | its object file size | 18:56 |
| angdraug | this raises some good generic concerns that still need to be addressed, but our plan with Fuel plugins is indeed to move most of them to review.fuel-infra.org | 18:57 |
| *** ewilson has quit IRC | 18:57 | |
| fungi | ahh, okay, so similar to what github's setting then | 18:57 |
| angdraug | mescanef: ^ | 18:57 |
| *** bhunter71 has quit IRC | 18:58 | |
| AJaeger | angdraug: do you want to comment on https://review.openstack.org/261966 ? | 18:58 |
| *** kzaitsev_mb has joined #openstack-infra | 18:58 | |
| zaro | fungi: don't understand the question | 18:58 |
| fungi | zaro: looking for teh setting to limit file object size. thought it might have been commit size instead | 18:58 |
| pabelanger | nibalizer: I agree. Also, I don't want to do a code review on a 100MB python file either! | 18:59 |
| zaro | i believe you do that with a patchset-commit hook | 18:59 |
| pabelanger | again, if you need more then 100MB, git likely isn't the place for the file | 18:59 |
| *** bhunter71 has joined #openstack-infra | 18:59 | |
| anteaya | angdraug: so you don't want to consider the fuel plugins part of openstack then? | 18:59 |
| pabelanger | we have static.o.o for example | 18:59 |
| fungi | angdraug: it still sounds to me like fuel (and its associated ecosystem) has different needs from the openstack community and would rather have direct control over those than compromise. it's not clear to me why fuel applied to become an openstack project-team | 18:59 |
| fungi | particularly given that they operate a separate code review system | 19:00 |
| clarkb | zaro fungi there is a config setting | 19:00 |
| clarkb | maxobjectsize or similar | 19:00 |
| pleia2 | meeting time | 19:00 |
| fungi | thanks pleia2 | 19:01 |
| *** yamahata has quit IRC | 19:01 | |
| *** baoli has quit IRC | 19:01 | |
| *** yamahata has joined #openstack-infra | 19:01 | |
| *** ybathia has joined #openstack-infra | 19:02 | |
| angdraug | anteaya: I don't want to consider _most_ plugins part of Fuel, only those that are ready to follow OpenStack development process and have active development team (e.g. LMA) | 19:02 |
| zaro | here it ishttps://gerrit-review.googlesource.com/#/c/30450/ | 19:02 |
| anteaya | angdraug: and how are you determining that? | 19:02 |
| anteaya | angdraug: how do you decide which are ready? | 19:03 |
| *** ybathia_ has joined #openstack-infra | 19:03 | |
| angdraug | anteaya: http://governance.openstack.org/reference/new-projects-requirements.html | 19:03 |
| anteaya | angdraug: and how are you applying that? | 19:04 |
| angdraug | if adding a repository would make Fuel fail these criteria, it can't be a part of Fuel | 19:04 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:04 | |
| angdraug | I'm not going to supplant TC and make judgements on other projects | 19:05 |
| angdraug | I'm just trying to make sure that Fuel stays in line with what we've committed to | 19:05 |
| anteaya | yet someone is still adding repos using the fuel name, and at a great rate: https://review.openstack.org/#/c/261966/ | 19:05 |
| angdraug | fungi: ^ | 19:05 |
| *** eil397 has joined #openstack-infra | 19:06 | |
| anteaya | angdraug: how are you doing that? making sure that fuel stays in line with what you've commited to? | 19:06 |
| angdraug | hold on, I get a feeling of a "damned if you do, damned if you don't" situation | 19:06 |
| anteaya | I'd like to know what you are doing | 19:06 |
| *** elo1 has joined #openstack-infra | 19:06 | |
| *** elo1 has left #openstack-infra | 19:06 | |
| anteaya | you are telling me you are doing things, and I believe you | 19:07 |
| anteaya | I would like to hear how you are doing them | 19:07 |
| *** ybathia has quit IRC | 19:07 | |
| *** ybathia_ is now known as ybathia | 19:07 | |
| anteaya | if you have discussions with plugin owners during your meetings, that is great | 19:07 |
| anteaya | link to the logs | 19:07 |
| anteaya | if you have discussions with plugin owners in email, that is great, link to the emails | 19:08 |
| anteaya | if these discussions take place in your channel, that is wonderful, please link to the discussions in your channel logs | 19:08 |
| anteaya | so far all I see is someone making repos for anyone who asks for one | 19:08 |
| anteaya | you are saying you are doing other things and that is wonderful, I'd like to see the evidence | 19:09 |
| anteaya | having the ability to share archives of discussions and conversations is part of the four opens | 19:10 |
| *** baoli has joined #openstack-infra | 19:12 | |
| *** otter768 has joined #openstack-infra | 19:13 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-grafyaml: Use git.openstack.org everywhere https://review.openstack.org/262299 | 19:17 |
| fungi | angdraug: similarly, i'm just trying to figure out where the openstack ecosystem boundaries are, and what makes fuel part of openstack itself but fuel's ecosystem not part of the openstack ecosystem (hosting in a separate code review system entirely) | 19:17 |
| *** baoli has quit IRC | 19:18 | |
| angdraug | very good question that is unfortunately contrary to what we discussed here last week, and that confuses me | 19:18 |
| anteaya | why do you feel it is contrary? | 19:19 |
| *** waynr has joined #openstack-infra | 19:19 | |
| angdraug | my understanding was that your team is unhappy with having to deal with a large number of git repositories that contain code of uncertain quality and ownership | 19:20 |
| angdraug | I proposed to move that code out of openstack infra to help you with that problem | 19:20 |
| anteaya | we are curious | 19:20 |
| anteaya | we would really like to know what is happening | 19:20 |
| anteaya | we like to play nice with others | 19:20 |
| anteaya | that starts with sharing | 19:20 |
| angdraug | now you're telling me that you're unhappy with this solution because it dilutes the boundaries of openstack ecosystem? | 19:20 |
| *** moravec has quit IRC | 19:21 | |
| fungi | angdraug: i think you're reading too much into my (and others') questions | 19:21 |
| angdraug | I was holding back on moving the plugins until after New Year so as not to spring a major change like that onto people who are likely on Christmas holidays | 19:21 |
| *** waynr has quit IRC | 19:21 | |
| *** waynr has joined #openstack-infra | 19:21 | |
| angdraug | fungi: maybe, but all I want is figure out what would be the best way forward with this problem | 19:22 |
| fungi | and would also love for more people to be involved in the discussion. we're on a skeleton crew for much of december | 19:22 |
| angdraug | exactly my concern, too | 19:22 |
| angdraug | on one hand, I don't want Fuel team to own and excert control over each and every Fuel plugin | 19:23 |
| *** isaacb has joined #openstack-infra | 19:23 | |
| angdraug | and without such control, we can't expect all plugins to have active teams, conform to openstack development process, etc | 19:24 |
| fungi | when it got discussed before, my only concern was over an openstack project spending time and effort on hosting and review of poorly-maintained or unmaintained code, which was the reason you (someone anyway? i don't remember who) gave for not adopting them as part of fuel | 19:25 |
| angdraug | if we reach a consensus that because of this, most plugins can't be counted as part of fuel and shouldn't be hosted on openstack infra, they're going to need a new home | 19:25 |
| *** sdake has quit IRC | 19:25 | |
| *** baoli has joined #openstack-infra | 19:25 | |
| fungi | we host all sorts of things in the openstack infra which aren't part of openstack, but merely part of the openstack ecosystem | 19:25 |
| anteaya | how about we start with a sense of the vision fuel has for itself? | 19:25 |
| angdraug | fungi: a valid concern that I think corresponds to the way I represented it above, correct me if I'm wrong | 19:26 |
| anteaya | and how that benefits openstack? | 19:26 |
| anteaya | I'd like to start there rather than skip over that bit | 19:26 |
| *** Hal has joined #openstack-infra | 19:26 | |
| fungi | curious who consensus was there. i was barely around to comment, but get the impression the discussion moved on to a decision after i provided some minimal input | 19:26 |
| *** Hal is now known as Guest60620 | 19:26 | |
| angdraug | to get a proper consensus we're going to need a thread on openstack-dev | 19:27 |
| fungi | so don't want this "you must move your plugins out of openstack's infrastructure if fuel won't adopt them" to be implied as the opinion of the infrastructure team as a whole | 19:27 |
| *** izaakk has joined #openstack-infra | 19:27 | |
| angdraug | fungi: thanks for clarifying that | 19:28 |
| *** david-lyle has joined #openstack-infra | 19:28 | |
| angdraug | I wasn't sure if this implication was valid | 19:28 |
| angdraug | now I see that it's not | 19:28 |
| anteaya | yeah that was not my take away from last week's discussion at all | 19:28 |
| anteaya | my take away was we would like to hear what fuel's vision is | 19:28 |
| anteaya | to better understand it | 19:28 |
| anteaya | as currently I don't | 19:28 |
| *** moravec has joined #openstack-infra | 19:29 | |
| *** FallenPegasus has joined #openstack-infra | 19:29 | |
| fungi | i'd like to see putting parts of the extended openstack ecosystem in our hosting as an opportunity for them to be and stay well maintained and useful for people | 19:30 |
| angdraug | fungi: still, the concern of what to do with the burden of hosting poorly maintained code stands, right? | 19:30 |
| *** baoli has quit IRC | 19:30 | |
| fungi | so i'll certainly discourage using it as a dumping ground for poorly-maintained or unmaintained software | 19:30 |
| fungi | but that doesn't mean they have to meet the tc requirements for being an official part of openstack | 19:31 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-infra-cookiecutter: Use git.openstack.org everywhere https://review.openstack.org/262301 | 19:31 |
| angdraug | fungi: if new-project-requirements is not the right criteria to make that distinction, what is? | 19:31 |
| fungi | they just need someone looking after them, reviewing and approving or providing feedback on changes which are proposed to them | 19:31 |
| *** isaacb has quit IRC | 19:31 | |
| anteaya | teaching them our workflow | 19:32 |
| AJaeger | arg, all our puppet repos mention github for contributors: http://git.openstack.org/cgit/openstack-infra/puppet-infracloud/tree/README.md#n69 . Should we remove this everywhere? 262301 removes it from cookiecutter... | 19:32 |
| *** salv-orlando has quit IRC | 19:33 | |
| fungi | angdraug: if you mean http://governance.openstack.org/reference/new-projects-requirements.html then that's what you need for fuel, as an official part of openstack. not all software we host is held to that level of scrutiny | 19:33 |
| anteaya | AJaeger: yes, we don't have to do it all today, but yes I favour that | 19:33 |
| angdraug | yes, that's exactly what I'm talking about | 19:33 |
| *** david-lyle has quit IRC | 19:33 | |
| fungi | AJaeger: i had a batch of changes proposed ages ago to fix that in all repos. not sure what happened to most of them | 19:33 |
| angdraug | we need a document that defines a reduced level of scrutiny | 19:33 |
| *** david-lyle has joined #openstack-infra | 19:33 | |
| AJaeger | anteaya: let's first merge 262301 and 262299 and then do these... | 19:34 |
| AJaeger | fungi: let me check... | 19:34 |
| angdraug | something that's not enough to be a part of openstack, but enough to be allowed onto openstack-infra | 19:34 |
| fungi | AJaeger: it's possible those repos postdate my mass change | 19:34 |
| fungi | angdraug: i really just want to dissuade people from proposing repos that they expect to abandon | 19:34 |
| anteaya | angdraug: and seeing you provide guidance in the form of reviews is far more valuable in my mind than having a document you can point to but which folks may or may not read | 19:35 |
| fungi | to me, that's what "unmaintained" means (completely abandoned) | 19:35 |
| anteaya | angdraug: you and the fuel team | 19:35 |
| angdraug | anteaya: the important part of having a document is reassuring people that we're applying the same objective requirements to everyone | 19:36 |
| *** aspiers has joined #openstack-infra | 19:36 | |
| anteaya | having a document doesn't reassure me of anything | 19:36 |
| anteaya | seeing you answer questions (referenceing the document if you choose) is what assures me you are providing guidance | 19:36 |
| AJaeger | fungi, I'm not seeing anything open from you on this | 19:36 |
| anteaya | seeing reviews providing guidance, seeing emails providing guidance, seeing meeting logs providing guidance, that reassures me | 19:37 |
| fungi | AJaeger: it was a year or two ago when git.openstack.org went into production | 19:37 |
| AJaeger | fungi: this all looks more recent... | 19:37 |
| fungi | at that time i proposed changes to every repo in gerrit to replace uses of github.com | 19:37 |
| fungi | in hopes that they would merge quickly and then the references wouldn't be cargo-cult copied into new repos | 19:38 |
| fungi | but i underestimated the length of time those sorts of changes can sit ignored | 19:38 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/release-tools: Use git.openstack.org https://review.openstack.org/262304 | 19:39 |
| *** e0ne has quit IRC | 19:39 | |
| anteaya | angdraug: and being consistent in your application is what assures me you are being consistent in your application | 19:39 |
| angdraug | anteaya: as far as the outcome, that's fair expectation, although I still believe that without a document, there'd be too much reliance on tribal knowledge | 19:41 |
| *** baoli has joined #openstack-infra | 19:41 | |
| anteaya | that's fine | 19:41 |
| anteaya | documentation is helpful to consistent support, it just doesn't take the place of it | 19:42 |
| anteaya | it is the consistent support that I'm looking for | 19:42 |
| angdraug | now that we are in agreement on the meta-discussion, I'd still like to receive guidance on what kind of guidance we should be giving to plugin developers ) | 19:42 |
| angdraug | anteaya: noted and agreed | 19:42 |
| *** david-lyle has quit IRC | 19:42 | |
| anteaya | well you and I and fungi have discussed this, others may have their opinions, note the infra meeting is in progress as we are talking | 19:43 |
| *** david-lyle has joined #openstack-infra | 19:43 | |
| anteaya | angdraug: why do you start by giving some guidance | 19:43 |
| anteaya | and see how that works for you | 19:43 |
| anteaya | what do you consider to be a will maintained repo | 19:43 |
| anteaya | and share that vision with others | 19:43 |
| fungi | yeah, i expect we'll end up with a set of repo hosting requirements at some point, because there will always be "rules lawyers" who prefer to spend time arguing every minute detail of this sort of thing and it ends up being easier to have a document to just point them at instead of rehashing conversations around what make up reasonable and unreasonable expectations | 19:44 |
| angdraug | can do, was going to do that in January to let more people come back from vacations for a healthy disucssion | 19:44 |
| angdraug | yup | 19:44 |
| anteaya | angdraug: that is reasonable, thanks for sharing that | 19:44 |
| anteaya | that makes sense to be | 19:44 |
| angdraug | hopefully we'll come to a reasonable conclusion on what fits and what doesn't | 19:44 |
| fungi | really, my request is "please don't waste our time asking us to host something nobody finds useful" | 19:45 |
| angdraug | and apologies about intersecting with your meeting | 19:45 |
| anteaya | angdraug: well more than conclusion, I am hoping for a process | 19:45 |
| fungi | if people find these useful, that's good enough for me | 19:45 |
| anteaya | angdraug: as a working relationship tends to do best with constant interaction | 19:46 |
| angdraug | right | 19:46 |
| fungi | and so long as there are people who will stick around to help keep it useful | 19:46 |
| anteaya | thanks for understanding | 19:46 |
| *** baoli has quit IRC | 19:46 | |
| *** abhi has joined #openstack-infra | 19:46 | |
| *** abhi has left #openstack-infra | 19:46 | |
| angdraug | whatever process we end up with, there will be repositories that don't fit | 19:46 |
| angdraug | what do we do with those? | 19:46 |
| anteaya | angdraug: how about start with a vision of first figuring out what you do have | 19:47 |
| anteaya | you might find if you start there others may migrate to that style as it is effective for you | 19:47 |
| *** aopincar has quit IRC | 19:48 | |
| angdraug | may be. but being able to tell people what would happen to their repo if they don't would be nice | 19:48 |
| angdraug | it will motivate them to pay attention to the discussion, for one | 19:49 |
| angdraug | and at the same time will reassure them that it won't mean wiping their code off the internet | 19:49 |
| anteaya | angdraug: then you have to make a decision | 19:49 |
| *** elo has quit IRC | 19:50 | |
| anteaya | that is part of being an openstack project, you have to make decisions for your project | 19:50 |
| angdraug | going back to the beginning of this conversation, I'm still confused about whether offering to host them on fuel-infra is a good or a bad thing | 19:50 |
| *** salv-orlando has joined #openstack-infra | 19:50 | |
| anteaya | it is neither good nor bad | 19:50 |
| anteaya | it simplying communicates the extent to which you consider yourself part of openstack | 19:50 |
| anteaya | I'm fine with any decision you make | 19:50 |
| anteaya | it is up to you | 19:50 |
| *** aks has left #openstack-infra | 19:50 | |
| anteaya | but decisions have consequences | 19:50 |
| anteaya | and if you chose to not be part of openstack then I will acknowledge that you aren't | 19:51 |
| anteaya | it is your choice | 19:51 |
| angdraug | in my book, not being a part of openstack would certainly fall under "bad" | 19:51 |
| *** markvoelker has joined #openstack-infra | 19:51 | |
| * persia opines that encouraging folk to use a specific non-o.o repo hosting solution for stuff may decrease the sense of developers in those repos that they are "openstack developers", which may lead to confusion about policies and practices later. | 19:52 | |
| angdraug | going back to the question of Fuel's mission, deploying and operating OpenStack is its whole point | 19:52 |
| fungi | to clarify, part of the openstack ecosystem (more than half of the repos we host aren't officially part of "openstack" last time i looked) | 19:52 |
| fungi | we provide our infrastructure to teh benefit of the openstack ecosystem, not only to the benefit of projects which are "an official part of openstack" | 19:53 |
| *** isq has joined #openstack-infra | 19:55 | |
| AJaeger | FYI, my changes for github in docs and infra are at https://review.openstack.org/#/q/status:open+topic:github | 19:55 |
| *** baoli has joined #openstack-infra | 19:56 | |
| *** markvoelker has quit IRC | 19:56 | |
| angdraug | fungi: persia: I may still be reading too much into what you're saying, but what I'm reading is "no, don't move anything to fuel-infra, push people to commit to maintaining their code, and reject plugins that can't make that commitment" | 19:57 |
| persia | That would be consistent with my opinion. I am not authoritative: so my opinion only has value if you agree. | 19:58 |
| fungi | angdraug: that does sound preferable, yes. the burden of maintaining something shouldn't be insurmountable. and honestly i can't imagine how anyone could find unmaintained plugins especially useful (or even trust them at all) | 19:58 |
| angdraug | persia: I'm trying to find a consenus, the option that everyone is comfortable with, that makes your opinion as authoritative as anyone's | 19:59 |
| fungi | if it's unmaintained, then how do i report the problem to the authors or send them a fix? i tend to just avoid using unmaintained software instead | 19:59 |
| angdraug | anteaya: do you agree with fungi and persia on ^? | 20:00 |
| fungi | or adopt it myself | 20:00 |
| fungi | and become the maintainer | 20:00 |
| *** kzaitsev_mb has quit IRC | 20:00 | |
| angdraug | in my experience so far, unmaintained plugin means "we used this to deploy once, it worked, if you want to reuse it again feel free to take over" | 20:00 |
| anteaya | angdraug: I guess I am missing the part of the fuel vision that states "We want to work with folks who don't maintain their code. This is helpful for fuel because..." | 20:01 |
| *** baoli has quit IRC | 20:01 | |
| anteaya | angdraug: and I think someone suggested in that use case to have that code all in one repo | 20:01 |
| anteaya | angdraug: rather than as separate repos | 20:01 |
| angdraug | hm, that's not a bad option | 20:02 |
| *** armax has joined #openstack-infra | 20:02 | |
| anteaya | angdraug: the fuel-one-off-and-then-abandoned repo | 20:02 |
| anteaya | angdraug: then whoever is maintaining that repo can decide what patches they want in that repo | 20:02 |
| openstackgerrit | Bulat Gaifullin proposed openstack-infra/project-config: Create packetary project https://review.openstack.org/261540 | 20:02 |
| fungi | sounds like what the osops team is doing, putting code which may have worked at some point but isn't tested and doesn't fit their quality standards in a "contrib" subtree of their repo | 20:02 |
| anteaya | much easier for all concerned | 20:02 |
| angdraug | then instead of saying "you abandoned this plugin, get off openstack infra" we can say "this plugin is abandoned, moving the code to contrib so we can drop the repo" | 20:03 |
| *** salv-orlando has quit IRC | 20:03 | |
| *** salv-orlando has joined #openstack-infra | 20:03 | |
| fungi | however, keeping multiple deliverable pieces in one repo poses problems of those pieces are expected to have different release lifecycles since tags apply to a particular repo state and not to individual file trees within a repo | 20:04 |
| anteaya | angdraug: or better yet, you plan on not supporting this, rather than start a separate repo offer it as a patch to this repo | 20:04 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Replace the Gerrit icon for Workflow -1 vote. https://review.openstack.org/165228 | 20:04 |
| angdraug | fungi: being abandoned implies that it doesn't _have_ a release cycle | 20:04 |
| angdraug | if a plugin in contrib grows an active maintainer it will need to be extracted into its own repo | 20:05 |
| angdraug | the process should work both ways | 20:05 |
| fungi | angdraug: sure, speaking more to the point of comingling multiple maintained things in a common repo, if the different maintained things want different release lifecycles | 20:05 |
| zaro | fungi: to limit object size on commit: https://gerrit-documentation.storage.googleapis.com/Documentation/2.11/config-gerrit.html#receive.maxObjectSizeLimit | 20:05 |
| anteaya | angdraug: this is where discussing intent with the code creator comes in | 20:05 |
| anteaya | angdraug: if you start off with a discussion rather than straight to repo creation you at least have an opportunity for the guidance we discussed | 20:06 |
| *** bhunter71 has quit IRC | 20:06 | |
| anteaya | angdraug: just asking them their vision can help a lot | 20:06 |
| anteaya | "what do you want to do with this? | 20:06 |
| anteaya | " is a great question I find | 20:07 |
| zaro | fungi: should we set to 100MB? | 20:07 |
| *** baoli has joined #openstack-infra | 20:07 | |
| fungi | zaro: thanks. so if i'm misreading, that's the max size of a commit right? not of a file within a commit? | 20:08 |
| fungi | or am i misunderstanding what "git object" means? (i always thought that was a commit, a tag, et cetera) | 20:08 |
| zaro | looks like it | 20:08 |
| *** bhunter71 has joined #openstack-infra | 20:09 | |
| clarkb | object is more generic than that | 20:09 |
| clarkb | commits and tags are objects but so are the files in git iirc | 20:10 |
| fungi | seems to correspond to hash objects and blobs too | 20:10 |
| clarkb | all a commit is is a message and pointers to other objects belonging to that snapshot in time | 20:10 |
| *** david-lyle has quit IRC | 20:10 | |
| clarkb | each of those are individual objects | 20:10 |
| fungi | reading https://git-scm.com/book/en/v2/Git-Internals-Git-Objects to understand what's implied here | 20:10 |
| fungi | ahh, yep, so maxObjectSizeLimit is going to reject individual files over that size it seems | 20:12 |
| *** baoli has quit IRC | 20:12 | |
| *** dcentral has joined #openstack-infra | 20:12 | |
| fungi | as well as commits over that size in aggregate? | 20:13 |
| clarkb | I don't think aggregate commits will be affected | 20:15 |
| *** angdraug has quit IRC | 20:17 | |
| *** baoli has joined #openstack-infra | 20:20 | |
| crinkle | clarkb: think we can schedule the paste/wiki/nodepool downtime? | 20:21 |
| *** vilobhmm11 has joined #openstack-infra | 20:21 | |
| clarkb | crinkle: ya we should then send email to -dev for it. I know fungi wants to unlocal db paste so would ask him when is good time for that | 20:21 |
| clarkb | crinkle: for wiki/nodepool we can do nodepool at any time leaving wiki the odd one out | 20:22 |
| clarkb | maybe say sometime next week? let me look at calendar | 20:22 |
| greghaynes | Not sure if relavent to doing downtime - but nodepool is still broken on os-c-c releasing | 20:22 |
| greghaynes | broken as in cant merge anything | 20:22 |
| clarkb | greghaynes: shouldn't be we will just flap mysql out from under nodepool | 20:23 |
| clarkb | crinkle: monday or tuesday? | 20:23 |
| greghaynes | ok | 20:23 |
| *** rguillebert has joined #openstack-infra | 20:23 | |
| clarkb | I have doctors wed and thurs | 20:23 |
| crinkle | clarkb: wfm | 20:23 |
| clarkb | and this week is crazyness | 20:23 |
| crinkle | clarkb: fungi if paste moves first ping me so i can update https://review.openstack.org/#/c/257640/5/modules/openstack_project/manifests/paste.pp | 20:23 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/gearman-plugin: Use git.openstack.org https://review.openstack.org/262308 | 20:24 |
| clarkb | crinkle: lets plan for late morning monday then? maybe you want to send email to list about it? I can do anything after 10am probably | 20:24 |
| clarkb | ya after 10am is good | 20:24 |
| pleia2 | I'll be around too if needed | 20:24 |
| *** baoli has quit IRC | 20:25 | |
| crinkle | clarkb: sure, we'll need to decide whether paste is going down or not before i can email | 20:25 |
| clarkb | crinkle: good point | 20:25 |
| crinkle | also could i get some re-reviews on https://review.openstack.org/#/c/258193/ ? i've responded to yolanda's -1 but i'm afraid the -1 is keeping other people from reviewing | 20:25 |
| crinkle | nibalizer: ^ | 20:25 |
| fungi | on the topic of max file size, the largest files in master branches of any of our git repos right now are the ops guide epub/mobi renderings in openstack-manuals (21mb and 38mb respectively). the largest file outside of openstack-manuals is a ttf font file in clouddocs-maven-plugin (coming in at 15mb) | 20:25 |
| *** baoli has joined #openstack-infra | 20:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/statusbot: Use git.openstack.org everywhere https://review.openstack.org/262309 | 20:26 |
| fungi | so even a 50mb limit seems reasonable, if we want to get into the limit-picking business | 20:27 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/groups: Use git.openstack.org everywhere https://review.openstack.org/262310 | 20:28 |
| *** kzaitsev_mb has joined #openstack-infra | 20:29 | |
| AJaeger | and the erroneous import in fuel-plugins-onos is deployment_scripts/puppet/modules/onos/files/jdk-8u51-linux-x64.tar.gz with a size of 166 MB | 20:30 |
| nibalizer | crinkle: lgtm | 20:30 |
| *** baoli has quit IRC | 20:31 | |
| *** Guest60620 has quit IRC | 20:32 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-specs: Use git.openstack.org everywhere https://review.openstack.org/262311 | 20:32 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/nose-html-output: Use git.openstack.org everywhere https://review.openstack.org/262312 | 20:34 |
| *** kzaitsev_mb has quit IRC | 20:34 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstackid: Use git.openstack.org everywhere https://review.openstack.org/262313 | 20:35 |
| *** baoli has joined #openstack-infra | 20:37 | |
| *** Rockyg has joined #openstack-infra | 20:37 | |
| *** moravec has quit IRC | 20:38 | |
| persia | On git object sizes: depending on the pack, it is possible to construct a file that exceeds the max object size, so long as git understands it as a base plus several deltas, rather than a single file. | 20:38 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-grafyaml: Use git.openstack.org everywhere https://review.openstack.org/262299 | 20:38 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-infracloud: Use git.openstack.org everywhere https://review.openstack.org/262314 | 20:39 |
| fungi | persia: interesting. will keep that in mind | 20:39 |
| *** ganso has quit IRC | 20:39 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-odsreg: Use git.openstack.org everywhere https://review.openstack.org/262315 | 20:40 |
| *** baoli has quit IRC | 20:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-refstack: Use git.openstack.org everywhere https://review.openstack.org/262316 | 20:42 |
| *** david-lyle has joined #openstack-infra | 20:43 | |
| pleia2 | AJaeger: maybe use a common topic? | 20:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-storyboard: Use git.openstack.org everywhere https://review.openstack.org/262317 | 20:43 |
| pleia2 | ah, later ones do | 20:43 |
| *** e0ne has joined #openstack-infra | 20:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/tripleo-ci: Use git.openstack.org everywhere https://review.openstack.org/262318 | 20:46 |
| crinkle | clarkb: this is what i see in bifrost, http://paste.openstack.org/show/482830/ is that familiar? | 20:46 |
| *** kzaitsev_mb has quit IRC | 20:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul: Use git.openstack.org everywhere https://review.openstack.org/262319 | 20:47 |
| AJaeger | pleia2: I fixed the one where I didn't - github is the topic | 20:47 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Revert "MySQL database cleanup for paste.o.o" https://review.openstack.org/262320 | 20:48 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Revert "Add a local MySQL database to paste.o.o" https://review.openstack.org/262321 | 20:48 |
| fungi | crinkle: those ^ are the reverts | 20:48 |
| anteaya | AJaeger: care to read my comment here? https://review.openstack.org/#/c/259265/1 | 20:48 |
| fungi | crinkle: i'm going to mark the second wip for now, and stop puppet on paste.o.o temporarily while i prepare to do teh dump/import | 20:49 |
| anteaya | AJaeger: new contributor is frustrated with our workflow and tweeted about it, pabelanger offered a patch which didn't suit him, I did try to explain | 20:49 |
| crinkle | fungi: kk | 20:49 |
| AJaeger | LGTM, anteaya, pabelanger - thanks! | 20:50 |
| *** baoli has joined #openstack-infra | 20:51 | |
| *** markvoelker has joined #openstack-infra | 20:52 | |
| openstackgerrit | Merged openstack-infra/release-tools: Use git.openstack.org https://review.openstack.org/262304 | 20:53 |
| AJaeger | pleia2: I've fixed all topics now, thanks for getting me to double check. | 20:53 |
| AJaeger | All changes are done now for github in openstack-infra with exception of jenkins-job-builder, I leave that to others | 20:54 |
| pleia2 | AJaeger: thanks, I'm not sure about https://review.openstack.org/#/c/262308/1/pom.xml - will that still work? | 20:54 |
| openstackgerrit | Merged openstack-infra/jeepyb: Be more specific for new users https://review.openstack.org/259265 | 20:54 |
| openstackgerrit | Merged openstack-infra/openstackid: Use git.openstack.org everywhere https://review.openstack.org/262313 | 20:55 |
| pleia2 | I know github has ssh things you can pull against, but I don't really know how this works for git.o.o (or what this file is trying to do exactly) | 20:55 |
| AJaeger | pleia2: I'm not sure about that either, I expect so - but somebody needs to test that | 20:55 |
| clarkb | crinkle I think that is what nibalizer ran into | 20:56 |
| *** baoli has quit IRC | 20:56 | |
| *** moravec has joined #openstack-infra | 20:57 | |
| pleia2 | AJaeger: ok, all the rest lgtm :) | 20:57 |
| *** markvoelker has quit IRC | 20:57 | |
| clarkb | crinkle format of yaml was wrong | 20:57 |
| AJaeger | pleia2: thanks for the reviews! | 20:57 |
| openstackgerrit | Merged openstack-infra/puppet-storyboard: Use git.openstack.org everywhere https://review.openstack.org/262317 | 20:57 |
| e0ne | hi team! I didn't find anything im ML. is any update about http://logstash.openstack.org/? it doesn't work for two days | 20:58 |
| pleia2 | e0ne: was someone looking into it for you? (I haven't seen any discussion about it) | 20:59 |
| pleia2 | slim crew this week, so it may not have been noticed yet otherwise | 20:59 |
| *** esker has joined #openstack-infra | 21:00 | |
| anteaya | I confirm logstash.o.o doesn't render the javascript graphs for me | 21:00 |
| e0ne | pleia2: no, I know that infra team is extremly busy this week, so I try didn't ask about it before | 21:00 |
| anteaya | e0ne: I didn't know until you said something a moment ago | 21:00 |
| pleia2 | e0ne: ok, thanks for reporting it | 21:01 |
| e0ne | pleia2, anteya: oops. I will send message to ML next time | 21:01 |
| *** ociuhandu has quit IRC | 21:01 | |
| *** izaakk has quit IRC | 21:01 | |
| pleia2 | e0ne: here is fine :) | 21:01 |
| e0ne | pleia2, anteya: btw, it's not javascript error, there is 503 server error | 21:02 |
| greghaynes | http://logstash.openstack.org/elasticsearch/_nodes | 21:02 |
| greghaynes | the cluster is not happy | 21:02 |
| e0ne | :( | 21:03 |
| AJaeger | I'd like to ask an infra-manual core to review https://review.openstack.org/#/c/261639/ - to follow recent changes in project-config | 21:03 |
| anteaya | e0ne: okay I didn't end up with a 503 | 21:03 |
| *** ociuhandu has joined #openstack-infra | 21:03 | |
| greghaynes | clarkb: I am guessing the es nodes have split | 21:03 |
| *** baoli has joined #openstack-infra | 21:04 | |
| pleia2 | AJaeger: oh good, an easy one :) done | 21:04 |
| AJaeger | pleia2: could you have a look at https://review.openstack.org/261857 as well if you have time, please? | 21:04 |
| AJaeger | Thanks! | 21:04 |
| clarkb | the smaller partition will need to stopped then restarted on at a time to find bigger psrtition | 21:04 |
| pleia2 | AJaeger: and a not easy one! but yeah, looking now | 21:04 |
| e0ne | one more question, please. is logstash right place to find all failures for given job? | 21:05 |
| clarkb | if someone else wants to unsplit them | 21:05 |
| *** rlandy has quit IRC | 21:05 | |
| clarkb | e0ne: no only the last 10 days is logged there | 21:05 |
| pleia2 | e0ne: have you seen this? http://status.openstack.org/elastic-recheck/ | 21:06 |
| AJaeger | pleia2: It should be copy & paste - I was tired of seeing the same code 4 times. But generalizing it is not that easy... | 21:06 |
| e0ne | clarkb: thanks. can I get some stats e.g. for the last 6 months? | 21:06 |
| openstackgerrit | Merged openstack-infra/infra-manual: Adopt node declaration in example https://review.openstack.org/261639 | 21:06 |
| greghaynes | clarkb: Do you just hit http://elasticsearch01.openstack.org/_nodes/stats and so on to see what nodes are part of what partition? | 21:06 |
| pleia2 | e0ne: we can't really go back that far | 21:06 |
| clarkb | greghaynes usually I use bigdesk but ya | 21:07 |
| greghaynes | TIL about bigdesk | 21:07 |
| clarkb | you hit one partition whoch you can use to infer the other(s) | 21:07 |
| e0ne | pleia2: I understand, it's reasonable | 21:07 |
| *** mrmartin has quit IRC | 21:08 | |
| *** jsavak has joined #openstack-infra | 21:08 | |
| openstackgerrit | Merged openstack-infra/project-config: fully retire keystoneauth-saml2 from openstack-ci https://review.openstack.org/261820 | 21:08 |
| clarkb | if you just need pass fail stats graphite.openstack.org tracks that | 21:08 |
| e0ne | I can see failures count using graphite.openstack.org, but I can't see reasons of them | 21:08 |
| clarkb | e0ne: correct because all that records is counts | 21:09 |
| *** baoli has quit IRC | 21:09 | |
| greghaynes | clarkb: hrm, what is the public endpoint to use for a given es node? http://elasticsearch01.openstack.org:9200 resets for me | 21:09 |
| *** moravec has quit IRC | 21:09 | |
| clarkb | greghaynes: oh there isn't one | 21:09 |
| greghaynes | :( | 21:09 |
| fungi | greghaynes: ssh tunnel to localhost | 21:09 |
| clarkb | greghaynes: so ya you can only hit 01 via the proxy | 21:09 |
| e0ne | clarkb: can I subscribe to failures somehow? | 21:09 |
| greghaynes | we could totally plumb through stats to the public I bet ;) | 21:09 |
| greghaynes | but ok | 21:09 |
| clarkb | greghaynes: ya we can open it up more if there are other read only details people want | 21:10 |
| e0ne | clarkb: I would like to monitor 2 cinder jobs to increase stability of them | 21:10 |
| clarkb | e0ne: graphite is good for that | 21:10 |
| e0ne | clarkb: does it support any notifications? | 21:10 |
| fungi | clarkb: if you're cool with https://review.openstack.org/262320 i have time to do the database export/import now | 21:11 |
| anteaya | e0ne: also do you know about openstack-health: http://status.openstack.org/openstack-health/#/ | 21:11 |
| anteaya | e0ne: and I don't believe any of our job log tools support notifications | 21:11 |
| clarkb | e0ne: no you would poll it | 21:11 |
| e0ne | anteaya: I didn't know about it, thanks! | 21:12 |
| fungi | oh, looks like we dropped the paste_db_host key from hiera during the fqdn split. i'll put it back in there | 21:12 |
| anteaya | e0ne: you are welcome, it might not suit your needs, if you want 6 months of data I do believe graphite is what you want | 21:12 |
| anteaya | e0ne: but it is another tool we have | 21:13 |
| anteaya | e0ne: it takes stats from the gate queue only, not the check queue | 21:13 |
| e0ne | anteaya: gate queue is enouph for me | 21:14 |
| *** sdake has joined #openstack-infra | 21:15 | |
| crinkle | clarkb: installing from pypi i get that error i gave earlier, installing from master i get http://paste.openstack.org/show/482831/ | 21:15 |
| anteaya | e0ne: great | 21:16 |
| Rockyg | anteaya, isn't there also a page with the elastic recheck statistics? with bugs related to gate failures? | 21:16 |
| anteaya | Rockyg: http://status.openstack.org/elastic-recheck/ | 21:16 |
| *** erlon has quit IRC | 21:16 | |
| Rockyg | e0ne, might want to look at that, too | 21:16 |
| e0ne | Rockyg: thanks | 21:17 |
| Rockyg | anteaya, thanks! | 21:17 |
| *** openstackgerrit has quit IRC | 21:17 | |
| clarkb | crinkle: I think that may still be invalid yaml, region-name needs to be at top level of the config dict | 21:17 |
| *** openstackgerrit has joined #openstack-infra | 21:17 | |
| anteaya | Rockyg: welcome | 21:18 |
| anteaya | Rockyg: thanks for suggesting it | 21:18 |
| anteaya | AJaeger: in this patch you missed fixing Openstack -> OpenStack as you had caught in other patches: https://review.openstack.org/#/c/262301/1 | 21:19 |
| *** baoli has joined #openstack-infra | 21:19 | |
| AJaeger | thanks for catching that, anteaya. Will update... | 21:19 |
| crinkle | clarkb: are you saying this is pebkac or a bifrost bug? this is new since occ 1.12.0 | 21:19 |
| anteaya | AJaeger: thanks | 21:19 |
| crinkle | so i don't think i should be needing to fix yaml | 21:20 |
| * greghaynes shakes fist at paste.o.o | 21:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-infra-cookiecutter: Use git.openstack.org everywhere https://review.openstack.org/262301 | 21:20 |
| AJaeger | anteaya: I fixed but forgot to upload ;( | 21:20 |
| clarkb | crinkle: I think it may be a broken openstack client config cloud yaml file and oscc has gotten stricter aout that since the recent release | 21:20 |
| *** maishsk has quit IRC | 21:20 | |
| clarkb | crinkle: before it ignored the config it didn't understand but now it complains | 21:20 |
| *** maishsk has joined #openstack-infra | 21:20 | |
| crinkle | clarkb: that sounds backwards incompatible to me | 21:20 |
| greghaynes | crinkle: tis a bug | 21:20 |
| crinkle | okie | 21:20 |
| clarkb | crinkle: yes agreed, similar issue to what nodepool ran into | 21:21 |
| greghaynes | crinkle: There was a release right before everyone went on holiday :( | 21:21 |
| fungi | greghaynes: what's your concern with paste.o.o now? | 21:21 |
| greghaynes | super slow to load | 21:21 |
| greghaynes | spins for about 30secs | 21:21 |
| fungi | greghaynes: i was doing a test mysqldump so that was probably my bad | 21:21 |
| greghaynes | ah, Is that not still the norm? | 21:21 |
| fungi | getting ready to move it back to trove | 21:21 |
| crinkle | greghaynes: yeah i just misinterpreted clarkb, sounded like it was my problem | 21:21 |
| greghaynes | paste.o.o has always done that for me | 21:22 |
| * AJaeger waves good-bye | 21:22 | |
| *** AJaeger has quit IRC | 21:22 | |
| *** baoli has quit IRC | 21:24 | |
| *** dimtruck is now known as zz_dimtruck | 21:25 | |
| e0ne | clarkb, anteya: thanks for the help | 21:26 |
| clarkb | crinkle: well I think you can update the yaml to fix it with current release | 21:26 |
| clarkb | which is what oscc wnts you to do | 21:26 |
| anteaya | e0ne: welcome, thanks for alerting us to logstash's condition | 21:26 |
| docaedo | Would anyone have a moment to take a look at this tiny patch to add requestsexceptions to proposal slave? https://review.openstack.org/#/c/260230/ | 21:28 |
| docaedo | I need it to finish testing a dead-link check script for the app catalog :) | 21:28 |
| *** vilobhmm11 has quit IRC | 21:30 | |
| *** vilobhmm11 has joined #openstack-infra | 21:31 | |
| *** jsavak has quit IRC | 21:32 | |
| *** esker has quit IRC | 21:32 | |
| *** masber has joined #openstack-infra | 21:32 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:32 | |
| *** baoli has joined #openstack-infra | 21:36 | |
| *** masuberu has joined #openstack-infra | 21:37 | |
| *** kzaitsev_mb has quit IRC | 21:40 | |
| *** baoli has quit IRC | 21:41 | |
| *** masber has quit IRC | 21:41 | |
| *** davideagnello has joined #openstack-infra | 21:42 | |
| jhesketh | Morning | 21:43 |
| *** kzaitsev_mb has joined #openstack-infra | 21:45 | |
| anteaya | morning jhesketh | 21:47 |
| *** baoli has joined #openstack-infra | 21:49 | |
| *** kzaitsev_mb has quit IRC | 21:51 | |
| *** markvoelker has joined #openstack-infra | 21:53 | |
| *** claudiub|2 has quit IRC | 21:55 | |
| *** baoli has quit IRC | 21:55 | |
| *** markvoelker has quit IRC | 21:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:03 | |
| *** baoli has joined #openstack-infra | 22:04 | |
| *** maishsk has quit IRC | 22:04 | |
| *** baoli has quit IRC | 22:09 | |
| anteaya | jhesketh: if you are casting about for entertainment at all today AJaeger put together a raft of changes for any occurances of github that was lying about, pointing to git.o.o instead: https://review.openstack.org/#/q/status:open+topic:github | 22:09 |
| *** kzaitsev_mb has quit IRC | 22:14 | |
| *** baoli has joined #openstack-infra | 22:15 | |
| clarkb | is anyone else dealing with ES cluster? if not I will go ahead and see what I can do now | 22:15 |
| anteaya | I am not addressing the es cluster | 22:16 |
| clarkb | [699121.920460] INFO: task java:97129 blocked for more than 120 seconds. appears to be a possible underlying issue | 22:16 |
| anteaya | I would be interested in hearing what you are doing as you do it though, so I can learn | 22:16 |
| *** kzaitsev_mb has joined #openstack-infra | 22:16 | |
| clarkb | so far I have just looked at dmesg to see why elasticsearch is unhappy on elasticsearch02 | 22:17 |
| clarkb | and found the above error | 22:17 |
| clarkb | looks like it hung on an ext4 write, df reports we have disk | 22:17 |
| clarkb | initial guess is that cinder volume went away for a while | 22:18 |
| clarkb | I am going to start elasticsearch on 02 | 22:18 |
| *** tzn has quit IRC | 22:18 | |
| *** jaypipes has quit IRC | 22:19 | |
| *** tzn has joined #openstack-infra | 22:19 | |
| jgriffith | anteaya: hello | 22:19 |
| *** Swanson has joined #openstack-infra | 22:19 | |
| *** baoli has quit IRC | 22:20 | |
| anteaya | jgriffith: hi, thanks | 22:20 |
| anteaya | jgriffith: so short backscroll thus far, clarkb is just getting started | 22:20 |
| anteaya | jgriffith: seems our elastic search cluster has been on holiday for a day or two and we just noticed | 22:20 |
| anteaya | it may or may not have something to do with the cinder volume, clarkb is still digging | 22:21 |
| anteaya | clarkb: thought I may as well as a cinder expert | 22:21 |
| *** dcentral has quit IRC | 22:21 | |
| jgriffith | anteaya: yeah, reading back.... not sure I"m following. But I'll just play along at home and offer anything up if somebody asks | 22:21 |
| jgriffith | anteaya: ie the "cinder went away" part | 22:22 |
| *** kzaitsev_mb has quit IRC | 22:22 | |
| anteaya | jgriffith: fair enough, thanks | 22:22 |
| anteaya | yeah right now we are just guessing, not much to go on yet | 22:23 |
| anteaya | thanks for standing by | 22:23 |
| clarkb | jgriffith: I am guessing something operational that prevented my iscsi backed volume from being writable for >120 seconds | 22:23 |
| clarkb | jgriffith: process blocked on a write and kernel shot it in the head for being unresponsive for over 120 seconds | 22:23 |
| jgriffith | clarkb: ahh... and then did it go R-Only by chance? | 22:23 |
| clarkb | jgriffith: no its rw now | 22:24 |
| pleia2 | stepping out for a while, last afternoon with some cousins in town | 22:24 |
| smcginnis | anteaya: Is it using LVM as the backing storage? Or something else? | 22:24 |
| jgriffith | clarkb: I've seen iSCSI volumes go read only for stalled IO in Nova sometimes. Had to bump my disconnect timeouts | 22:24 |
| clarkb | its remount ro though | 22:24 |
| anteaya | pleia2: have fun | 22:24 |
| pleia2 | anteaya: thanks :) | 22:24 |
| *** tzn has quit IRC | 22:24 | |
| clarkb | smcginnis: it is a 1TB cinder volume with lvm and ext4 as the filesystem | 22:24 |
| clarkb | I do not know what rackspace uses on the back end | 22:25 |
| jgriffith | clarkb: Lunar | 22:25 |
| clarkb | (we use lvm so that we can consolidate vlumes into larger filesystems on the "client" end) | 22:25 |
| smcginnis | jgriffith: Well.. lunr is actually used in place of Cinder, right? | 22:25 |
| jgriffith | CBS == Lunar, their own LVM storage product | 22:25 |
| jgriffith | smcginnis: nope, it's pretty much a device and driver by itself | 22:26 |
| clarkb | anyways we seem to be recovering now that I have restarted elasticsearch on 02, 05 and 06 | 22:26 |
| jgriffith | clarkb: gimmie a shout if you see more problems, happy to try and help | 22:26 |
| jgriffith | but I'm sure you're more than capable yourself | 22:26 |
| clarkb | ya everything seems happy now, I am guessing a transient serivce issue | 22:27 |
| anteaya | jgriffith: thanks for dropping by | 22:27 |
| clarkb | maybe they were trying to take advantage of quiet time to do work | 22:27 |
| * anteaya offers eggnog and cookies | 22:27 | |
| jgriffith | anteaya: sure | 22:27 |
| jgriffith | clarkb: haha :) | 22:27 |
| crinkle | clarkb: it wasn't a yaml issue https://review.openstack.org/#/c/262328/ | 22:27 |
| *** david-lyle has quit IRC | 22:28 | |
| clarkb | crinkle: I don't know that that change is correct | 22:28 |
| clarkb | its intentional that no region would be returned by get_known_regions | 22:29 |
| anteaya | crinkle: you do write good commit messages | 22:29 |
| clarkb | crinkle: I made that new method just for that change and wrote a bunch of tests to make sure it worked properly | 22:29 |
| crinkle | clarkb: it breaks bifrost because get_one_cloud tries to get config['region_name'] which isn't set anymore | 22:30 |
| clarkb | crinkle: aiui according to mordred if you don't have a region set then the region name of '' is what you should use | 22:30 |
| clarkb | which is what should be returned there | 22:30 |
| clarkb | (due to the default parameter value to thta function) | 22:30 |
| clarkb | notmorgan helped figure that out too | 22:31 |
| clarkb | crinkle: it is possible that my fix was incomplete and trips over some other thing in occ | 22:32 |
| clarkb | crinkle: also thats still unreleased code so can't be what is breaking bifrost right? | 22:33 |
| *** baoli has joined #openstack-infra | 22:33 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:34 | |
| crinkle | clarkb: no, https://review.openstack.org/#/c/256979/ is what is breaking bifrost but the region thing is what happens when i try to use occ from source | 22:34 |
| *** salv-orl_ has joined #openstack-infra | 22:34 | |
| crinkle | clarkb: '' is supposed to be returned from which function? | 22:34 |
| clarkb | crinkle: from the one you have updated | 22:35 |
| clarkb | _get_known_regions should return None if it knows of no regions | 22:35 |
| clarkb | when that happens _get_region expands the value for the region_name parameter passed to it | 22:36 |
| *** david-lyle has joined #openstack-infra | 22:36 | |
| clarkb | which by default is '' | 22:36 |
| crinkle | clarkb: okay i'll try to see why it's not working for me | 22:37 |
| *** salv-orlando has quit IRC | 22:37 | |
| clarkb | the idea is callers are saying "make me a region dict with the info available" and it best cases when it has no passed region and no regions specified in clouds.yaml | 22:37 |
| clarkb | if region is not specified as param but known in clouds.yaml that is used | 22:38 |
| *** FallenPegasus has quit IRC | 22:38 | |
| *** baoli has quit IRC | 22:38 | |
| *** bhunter71 has quit IRC | 22:38 | |
| clarkb | crinkle: it does work with your change in place? I can't see how the return values would be different if using region name of '' | 22:39 |
| clarkb | oh because of the next check | 22:39 |
| clarkb | so old code returns expand('') but your change would return regions[0] which should also be expand('') if no region is known | 22:41 |
| *** FallenPegasus has joined #openstack-infra | 22:43 | |
| *** baoli has joined #openstack-infra | 22:45 | |
| crinkle | clarkb: it turns out not to work, not sure what i did that made it seem like it was working | 22:48 |
| *** baoli has quit IRC | 22:49 | |
| *** david-lyle has quit IRC | 22:50 | |
| crinkle | wait, it does work | 22:50 |
| *** jpr has joined #openstack-infra | 22:50 | |
| clarkb | crinkle: and you are sure that no region name is set anywhere? I am reading the big get_one_cloud() function and it looks like it even says if no region_name arg then args['region_name'] = '' | 22:51 |
| clarkb | then it condenses args[] into the config object which lacks that key | 22:52 |
| clarkb | so I am confused how it wouldn't have that key | 22:52 |
| *** dcentral has joined #openstack-infra | 22:53 | |
| clarkb | hrm it does haev a null check but thats explicitly is not None, not a falsey check | 22:54 |
| *** kzaitsev_mb has quit IRC | 22:56 | |
| *** moravec has joined #openstack-infra | 22:57 | |
| clarkb | crinkle: so you do have a cloud config if you get to that line | 22:57 |
| clarkb | otherwise the check above it would retunr | 22:57 |
| *** hdd has quit IRC | 22:59 | |
| crinkle | clarkb: so _get_region returns {'values': {}, 'name': None} - args['region_name'] gets set to None - if val is not None: means it never gets added to config | 23:00 |
| clarkb | how is None getting in there? | 23:00 |
| *** baoli has joined #openstack-infra | 23:00 | |
| clarkb | get_one_cloud explicitly defaults it to '' and its also a parameter default of '' | 23:00 |
| crinkle | working on it | 23:01 |
| *** Rockyg has quit IRC | 23:01 | |
| *** FallenPegasus has quit IRC | 23:01 | |
| clarkb | crinkle: but that would explain how the behavior is different since the old code uses region_name to construct a region whereas yours is going to return '' regardless of the passed name | 23:02 |
| clarkb | (thats basically the change I made to fix nodepool since nodepool knows that the region name is and that should be respected if the cloud config has no idea what the regions are) | 23:02 |
| clarkb | crinkle: aha! | 23:03 |
| mordred | clarkb: did you find it? | 23:03 |
| clarkb | crinkle: because if 'region_name' not in args: is not checking if it is None just if it doesn't exist | 23:03 |
| clarkb | in get_one_cloud | 23:04 |
| clarkb | mordred: maybe, haven't completely tracked it down but I am guessing argparse defaults that value to None | 23:04 |
| clarkb | which means its always in the list so that check doesn't actually fire and replace it with '' | 23:04 |
| mordred | clarkb: ah. that's entire plausible | 23:05 |
| clarkb | crinkle: so I think the most correct fix is likely in _get_region() making region_name == '' if it is None | 23:05 |
| clarkb | since that is what ksa wants apparently | 23:05 |
| *** baoli has quit IRC | 23:05 | |
| mordred | clarkb: btw - bb is going to replace our cpus today | 23:05 |
| crinkle | clarkb: confirmed, _get_region getting None passed in for region_name instead of '' | 23:06 |
| mordred | clarkb: we're still at quota 0 right? | 23:06 |
| *** tzn has joined #openstack-infra | 23:06 | |
| clarkb | mordred: let me check | 23:06 |
| clarkb | crinkle: so ya in the first part of _get_region() you want an if region_name is None: region_name = '' | 23:06 |
| clarkb | crinkle: then add a unittests for region_name == None | 23:07 |
| clarkb | crinkle: I made a bunch for that function so should be easy copy pasta | 23:07 |
| zaro | clarkb: the world of java GC: http://blog.takipi.com/garbage-collectors-serial-vs-parallel-vs-cms-vs-the-g1-and-whats-new-in-java-8/ | 23:07 |
| *** marcusvrn_ has quit IRC | 23:07 | |
| *** alexpilotti_ has joined #openstack-infra | 23:07 | |
| zaro | nibalizer, clarkb : it looks like ParallelGC is already the default GC for java 7 | 23:07 |
| crinkle | clarkb: cool, testing | 23:07 |
| clarkb | mordred: yes max-servers: 0 | 23:07 |
| mordred | clarkb: cool | 23:08 |
| nibalizer | sweet | 23:08 |
| crinkle | works \o/ | 23:08 |
| zaro | it looks like there might be a better option for larger heaps, like our situation. the G1 GC. | 23:08 |
| *** moravec has quit IRC | 23:08 | |
| clarkb | zaro: I looked into that GCer for elasticsearch and it seemed consensus was you really shouldn't need it | 23:08 |
| *** kzaitsev_mb has joined #openstack-infra | 23:09 | |
| zaro | you mean you shouldn't need the G1? | 23:09 |
| anteaya | zaro: wonderful | 23:09 |
| clarkb | zaro: https://www.elastic.co/guide/en/elasticsearch/guide/current/_don_8217_t_touch_these_settings.html | 23:09 |
| clarkb | zaro: yes at least for elasticsearch they say do not use G1 | 23:09 |
| anteaya | mordred: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n925 | 23:10 |
| mordred | anteaya: woot. thanks! | 23:10 |
| anteaya | welcome, just confirming what clarkb said above | 23:11 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:11 | |
| *** baoli has joined #openstack-infra | 23:11 | |
| *** angdraug has joined #openstack-infra | 23:12 | |
| clarkb | zaro: it might be worth trying on review-dev but I worry that if lucene causes it to segfault then it will cause problems for gerrit (since gerrit uses lucene) | 23:12 |
| *** e0ne has quit IRC | 23:12 | |
| *** moravec has joined #openstack-infra | 23:12 | |
| zaro | that was my concern regarding java 7 whether the G1 was solid enough in that release. | 23:14 |
| zaro | that elasticsearch post doesn't mention the java version they are talking about. it says CMS is the default but i am certain that it's parallel | 23:15 |
| zaro | so i'm guessing they might be referring to java 8. | 23:15 |
| clarkb | zaro: CMS runs parallel threads | 23:16 |
| *** alexpilotti_ has quit IRC | 23:17 | |
| *** baoli has quit IRC | 23:17 | |
| clarkb | oh I see parallel is the name of another collector | 23:18 |
| clarkb | zaro: es may mean that they default to that in their startup scripts/binaries | 23:19 |
| jpr | i've been off improving my puppet-fu and have a few questions about adapting the review.openstack.org node in site.pp to a local install | 23:19 |
| clarkb | zaro: ya they explicitly use CMS | 23:19 |
| anteaya | jpr: best to just launch into the asking | 23:20 |
| jpr | specifically in the http://docs.openstack.org/infra/system-config/sysadmin.html#making-a-change-in-puppet section there's a note that says not to use the hiera functionality for the site.pp definitions | 23:20 |
| anteaya | jpr: if someone can offer a response they will | 23:21 |
| clarkb | zaro: gerrit does not explicitly state so the jvm is free to pick the best one based on its metrics | 23:22 |
| clarkb | zaro: likely parallel collector from my reading | 23:22 |
| *** zz_dimtruck is now known as dimtruck | 23:22 | |
| clarkb | zaro: we may want to consider CMS since the problem we have are related to responsiveness, though I worry that we will get less good collection when things get constrained | 23:22 |
| jpr | i get that heira wouldn't necessarily be exposed to the public, given some of the values that are defined there like hiera('github_project_password'), but is there any info in how the project structures the heira db so I can emulate that as well | 23:22 |
| *** tzn has quit IRC | 23:23 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-nodepool: Nodepool image build workers https://review.openstack.org/240093 | 23:23 |
| jpr | should, i just assume i need to provide answers that will satisfy the heira queries i | 23:23 |
| *** baoli has joined #openstack-infra | 23:24 | |
| clarkb | jpr: we don't do anything super special in the base case | 23:24 |
| clarkb | so upstream hiera docs should be good | 23:24 |
| zaro | clarkb: that's probably not a bad idea. | 23:24 |
| *** [1]Thelo has joined #openstack-infra | 23:25 | |
| clarkb | jpr: there is also work to make public the public portions of our hiera data | 23:25 |
| openstackgerrit | Merged openstack-infra/puppet-kibana: Encode query uri component properly https://review.openstack.org/258029 | 23:25 |
| clarkb | nibalizer: ^ where is that hiding? | 23:25 |
| zaro | i guess we can keep that in mind if problem re-occurs | 23:25 |
| jpr | do you use git on the heira backend? | 23:25 |
| clarkb | jpr: we store the yaml in a git repo, but thats an implementation detail | 23:25 |
| jpr | and I assume that git repo has restricted access given some of the sensitive info in it | 23:26 |
| clarkb | yes only people with root have access to that | 23:27 |
| fungi | it's stored locally on our ansible-puppet bastion host (formerly called a "puppetmaster") | 23:27 |
| *** Thelo has quit IRC | 23:28 | |
| *** [1]Thelo is now known as Thelo | 23:28 | |
| fungi | we really just use it for versioning and change tracking the hiera files | 23:28 |
| jpr | ok. thanks | 23:28 |
| *** baoli has quit IRC | 23:29 | |
| *** alexpilotti_ has joined #openstack-infra | 23:30 | |
| nibalizer | jpr: this is sortof what our file looks like | 23:30 |
| nibalizer | https://review.openstack.org/#/c/239783/7 | 23:30 |
| nibalizer | it doesn't actually look like that right now, but thats not really important | 23:31 |
| jpr | fungi: does the ansible-puppet bastion host ref imply a shift from puppet to ansible or just adding it to the mix | 23:31 |
| nibalizer | moving from puppet agent to puppet apply | 23:32 |
| nibalizer | with ansible orchestrating | 23:32 |
| *** dimtruck is now known as zz_dimtruck | 23:32 | |
| *** Qiming has joined #openstack-infra | 23:35 | |
| clarkb | ES cluster is yellow now (I sped up recovery by running index cleanup since that had been failing due to cluste rbeing down) | 23:35 |
| clarkb | we didn't need to recover all those indexes that would be deleted in a few hours | 23:36 |
| *** kzaitsev_mb has quit IRC | 23:38 | |
| anteaya | yay yellow! | 23:39 |
| crinkle | clarkb: https://review.openstack.org/#/c/262328/ | 23:40 |
| *** sdake has quit IRC | 23:40 | |
| clarkb | crinkle: +2 | 23:41 |
| crinkle | clarkb: woot thanks for your help | 23:42 |
| *** kzaitsev_mb has joined #openstack-infra | 23:48 | |
| mordred | crinkle, clarkb: lgtm. thanks! | 23:54 |
| *** markvoelker has joined #openstack-infra | 23:54 | |
| *** david-lyle has joined #openstack-infra | 23:55 | |
| clarkb | zaro: so far the memory pressure hasn't built up on review.o.o | 23:55 |
| clarkb | zaro: its well under 8gb still, may need to wait a bit to see if it is happy at 12gb | 23:55 |
| *** markvoelker has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!