| jeblair | clarkb: so i think that having a local ref that shares commits with a remote speeds up fetch operations... | 00:00 |
|---|---|---|
| *** rgrober has joined #openstack-infra | 00:00 | |
| *** thomasm has quit IRC | 00:01 | |
| jeblair | clarkb: i suspect that having such a local ref means that git keeps packs and objects around that it otherwise wouldn't... | 00:01 |
| clarkb | jeblair: I think you are correct. Having the refs means git can quickly lookup shas and compare to see what is needed | 00:01 |
| *** rfolco has joined #openstack-infra | 00:02 | |
| jeblair | clarkb: so we should consider whether the change i just made would cause zuul to get slower over time by not having good remote tracking branches anymore... | 00:02 |
| jeblair | clarkb: but hand in hand with that... | 00:02 |
| jeblair | clarkb: i think that may be part of why it's so slow to have lots of local refs | 00:02 |
| jeblair | clarkb: it might have to check a lot of possible starting points when it performs a fetch | 00:02 |
| jeblair | clarkb: just fetching from a local filesystem takes 5 seconds with the 36.5k zuul refs | 00:03 |
| jeblair | clarkb: versus .4 seconds if i rm them | 00:03 |
| HenryG | Is there a way to go straight to my draft comments? | 00:03 |
| clarkb | HenryG: if you click the review button you can edit them all from that page | 00:04 |
| *** adalbas has quit IRC | 00:04 | |
| lifeless | jeblair: yes, git sends all refs in both directions all the time | 00:04 |
| HenryG | I can search for reviews with has:drafts but some of them have dozens of patch sets and my drafts are buried in old patch sets. | 00:04 |
| lifeless | jeblair: modulo, perhaps, reducing by reachability | 00:04 |
| *** rgrober has quit IRC | 00:06 | |
| clarkb | HenryG: I am only aware of being able to do it on a per patchset basis using the revuew button | 00:06 |
| clarkb | HenryG: I don't think there is a way to pull all of them up across all patchsets | 00:06 |
| *** UtahDave has quit IRC | 00:07 | |
| clarkb | I try to make individual reviews discrete things for this reason | 00:07 |
| *** vipul-away is now known as vipul | 00:08 | |
| Alex_Gaynor | sigh, the gate is very flaky today | 00:11 |
| clarkb | Alex_Gaynor: ya the neutron, grenade, and tempest testr failures are hurting us | 00:11 |
| Alex_Gaynor | clarkb: oh well, I think it's mostly in the service of getting them fixed sooner :) | 00:15 |
| *** dims has quit IRC | 00:15 | |
| * clarkb walks home | 00:16 | |
| clarkb | jhesketh_: let me know if you have any questions about my zuul review | 00:16 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Revert bogus merger optimizations https://review.openstack.org/43977 | 00:17 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Fix preprocessing for ref-updated events https://review.openstack.org/43978 | 00:17 |
| jhesketh_ | clarkb: nope, it was all pretty sensible. I have draft responses but I just need to fix up a bug before I push and comment | 00:17 |
| clarkb | jeblair: should I review those before packing up the laptop? | 00:17 |
| jeblair | clarkb: they can wait; i don't think i want to restart again tonight | 00:17 |
| clarkb | ok | 00:17 |
| jhesketh_ | clarkb: incidently, read back for discussion over the backwards incompatibility that might answer one of your questions | 00:18 |
| jeblair | clarkb: i think i was heading down the wrong path there; that puts us back where we were (but fixes that ref-updated thing) | 00:18 |
| clarkb | jhesketh_: ya I think that covers the voluptuous stuff | 00:18 |
| *** jergerber has joined #openstack-infra | 00:18 | |
| jeblair | clarkb: i'm thinking tomorrow we merge that, and during the downtime, delete the local repos. that will speed things up because the zuul refs will be gone. | 00:18 |
| clarkb | is that something that we will need to do periodically? | 00:19 |
| jeblair | clarkb: it's kind of looking that way | 00:19 |
| clarkb | or will we add a cron to cleanup refs? | 00:19 |
| clarkb | :/ | 00:19 |
| jeblair | clarkb: i mean, we'll be good for months, so we have a while to figure out a solution | 00:20 |
| *** alexpilotti has joined #openstack-infra | 00:26 | |
| *** jamiem has quit IRC | 00:26 | |
| *** nati_ueno has joined #openstack-infra | 00:28 | |
| *** dims has joined #openstack-infra | 00:31 | |
| *** portante|afk is now known as portante | 00:38 | |
| *** dina_belova has joined #openstack-infra | 00:41 | |
| *** mikal has joined #openstack-infra | 00:45 | |
| *** dina_belova has quit IRC | 00:46 | |
| *** rfolco has quit IRC | 00:49 | |
| *** erfanian has quit IRC | 00:51 | |
| *** erfanian has joined #openstack-infra | 00:51 | |
| yjiang5 | clarkb: hi, I have patch 39890 which was approved before, but after I upload a unchanged one, seems it's not changed to be approved, should I have to ask core to re-approve it? | 00:52 |
| yjiang5 | clarkb: the url is https://review.openstack.org/#/c/39890/ | 00:53 |
| clarkb | yjiang5: yes, if it isupdated you will need a new +A vote | 00:54 |
| *** dkliban has joined #openstack-infra | 00:55 | |
| *** changbl has joined #openstack-infra | 00:56 | |
| *** carlp has joined #openstack-infra | 00:58 | |
| *** alexpilotti has quit IRC | 00:59 | |
| *** carlp-away has quit IRC | 00:59 | |
| *** cyeoh__ has joined #openstack-infra | 00:59 | |
| *** redbeard2 has joined #openstack-infra | 00:59 | |
| yjiang5 | clarkb: no changes at all | 00:59 |
| yjiang5 | clarkb: I just re-submit the patch considering the messy dependency. | 01:00 |
| *** dkliban has quit IRC | 01:00 | |
| *** jhesketh has joined #openstack-infra | 01:00 | |
| yjiang5 | clarkb: it has two +2 added automatically but no +A :( | 01:00 |
| clarkb | yjiang5: right the trivial rebase detection script readded the code review votes, but it will not readd the +A vote. You just need to poke one of the people that voted +2 to readd their +A | 01:00 |
| *** cyeoh_ has quit IRC | 01:01 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support for emailing results via SMTP https://review.openstack.org/42645 | 01:01 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Separate reporters from triggers https://review.openstack.org/42643 | 01:01 |
| yjiang5 | clarkb: got it and thanks. | 01:01 |
| jhesketh | clarkb, jeblair: ^ | 01:01 |
| clarkb | jhesketh: thanks, I probably won't get to a second round of reviewing until tomorrow | 01:02 |
| jhesketh | no pressure, just letting you know :-) | 01:02 |
| clarkb | TF2 has been started which is a semi official end to the work day :) | 01:02 |
| *** xchu has joined #openstack-infra | 01:06 | |
| jhesketh | shiny :-) | 01:06 |
| *** yongli has joined #openstack-infra | 01:17 | |
| *** pcrews has joined #openstack-infra | 01:17 | |
| *** sarob_ has quit IRC | 01:19 | |
| *** gyee is now known as gyee_food | 01:22 | |
| *** nati_ueno has quit IRC | 01:25 | |
| *** nati_ueno has joined #openstack-infra | 01:26 | |
| *** nati_ueno has quit IRC | 01:30 | |
| *** sdake_ has quit IRC | 01:34 | |
| *** rfolco has joined #openstack-infra | 01:35 | |
| *** svarnau has quit IRC | 01:40 | |
| *** dina_belova has joined #openstack-infra | 01:42 | |
| *** dina_belova has quit IRC | 01:46 | |
| *** redbeard2 has quit IRC | 01:47 | |
| *** cyeoh__ is now known as cyeoh | 01:51 | |
| *** dkliban has joined #openstack-infra | 01:52 | |
| *** dkliban has quit IRC | 01:56 | |
| *** rfolco has quit IRC | 01:58 | |
| clarkb | jeblair: heads up. New logstash release is imminent. It changes a bunch of stuff in non backward compatible ways and upgrades elasticsearch to 0.90.3. This means that upgrading will potentially be painful, but a lot of the new features and bug fixes look awesome so I am super excited | 01:58 |
| *** jergerber has quit IRC | 01:59 | |
| clarkb | jeblair: I have a feeling that upgrading will suck up a bunch of my post H3 time, but hope it puts us in a better position | 01:59 |
| HenryG | According to my googles, "Couldn't find index page for 'pbr' (maybe misspelled?)" is supposed to be fixed. So why is it happening to me? | 02:01 |
| clarkb | HenryG: can you be more specific? | 02:01 |
| clarkb | what are you doing that is causing the error? | 02:01 |
| HenryG | I am trying to run "tox -e pep8" in horizon. | 02:02 |
| clarkb | HenryG: can you paste the output of that command? | 02:02 |
| HenryG | It gets to "Running setup.py egg_info for package python-cinderclient" | 02:02 |
| HenryG | clarkb: http://paste.openstack.org/show/45257/ | 02:04 |
| clarkb | HenryG: that is an error talking to pypi.python.org. Are you behind a proxy? | 02:05 |
| HenryG | clarkb: sigh, yes | 02:05 |
| clarkb | HenryG: can you try using pypi.openstack.org/python? Give me a minute and I will have a paste showing you how to do that | 02:06 |
| clarkb | HenryG: I think the problem is speaking https to pypi.python.org | 02:06 |
| clarkb | HenryG: http://paste.openstack.org/show/45258/ | 02:07 |
| clarkb | HenryG: the alternative is figuring out why your proxy and pypi are not friendly | 02:08 |
| HenryG | clarkb: yah, the proxies are run by "the IT department" | 02:10 |
| *** sdake_ has joined #openstack-infra | 02:11 | |
| *** senk has joined #openstack-infra | 02:16 | |
| HenryG | clarkb: it's still trying pypi.python.org | 02:16 |
| *** dims has quit IRC | 02:17 | |
| clarkb | HenryG: if you created those files under the same user account as the one you are using to run tox it should work | 02:22 |
| HenryG | clarkb: the only user on this host | 02:23 |
| *** vipul has quit IRC | 02:35 | |
| *** vipul has joined #openstack-infra | 02:36 | |
| *** dina_belova has joined #openstack-infra | 02:42 | |
| *** dina_belova has quit IRC | 02:47 | |
| *** pcrews has quit IRC | 02:50 | |
| *** gyee_food has quit IRC | 02:50 | |
| *** nicedice_ has quit IRC | 02:55 | |
| *** pcrews has joined #openstack-infra | 02:55 | |
| *** nicedice_ has joined #openstack-infra | 02:56 | |
| *** fifieldt has joined #openstack-infra | 03:01 | |
| *** rcleere has joined #openstack-infra | 03:04 | |
| *** pcrews has quit IRC | 03:14 | |
| *** pcrews has joined #openstack-infra | 03:21 | |
| *** melwitt has quit IRC | 03:26 | |
| *** erfanian has quit IRC | 03:41 | |
| *** dina_belova has joined #openstack-infra | 03:43 | |
| *** dina_belova has quit IRC | 03:48 | |
| *** x05h has joined #openstack-infra | 03:57 | |
| *** vogxn has joined #openstack-infra | 03:57 | |
| *** jpeeler has quit IRC | 04:10 | |
| *** jpeeler has joined #openstack-infra | 04:10 | |
| *** jpeeler has joined #openstack-infra | 04:10 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Just run the install_puppet/modules scripts. https://review.openstack.org/43987 | 04:12 |
| pabelanger | woah, I got salt-cloud actually working | 04:21 |
| pabelanger | \o/ | 04:21 |
| morganfainberg | pabelanger: gratz | 04:24 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add robertc user. https://review.openstack.org/43989 | 04:25 |
| lifeless | where do puppet logs go? | 04:29 |
| lifeless | I'm trying to get the etherpad demo from http://ci.openstack.org/sysadmin.html to work... | 04:29 |
| clarkb | lifeless: how are you running puppet? | 04:29 |
| clarkb | lifeless: if using puppet apply I think you need to give it a log file on the command line otherwise it goes to stdout or stderr | 04:30 |
| pabelanger | lifeless, I think they are piped into //tmp/manifest.log | 04:30 |
| *** tian has quit IRC | 04:30 | |
| *** yjiang5 is now known as yjiang5_away | 04:34 | |
| *** x05h has quit IRC | 04:36 | |
| *** rcleere has quit IRC | 04:38 | |
| *** boris-42 has joined #openstack-infra | 04:39 | |
| *** dina_belova has joined #openstack-infra | 04:43 | |
| *** asalkeld has joined #openstack-infra | 04:44 | |
| asalkeld | what the procedure for importing a project into transifex? | 04:44 |
| asalkeld | (Heat) | 04:45 |
| clarkb | asalkeld: I thought we had this documetned but https://wiki.openstack.org/wiki/Translations is missing it | 04:46 |
| asalkeld | I have logged into transifex | 04:47 |
| asalkeld | but can't find the "add project" | 04:47 |
| asalkeld | openstack/heat | 04:47 |
| clarkb | asalkeld: the project needs to be created on transifex then added to the openstack project hub. Then we need to make sure the openstackjenkins account has permissions to heat after which we can add the jenkins jobs | 04:47 |
| *** dina_belova has quit IRC | 04:47 | |
| asalkeld | I have a patch for "config" ready | 04:48 |
| asalkeld | was just wanting to sort the transifex out | 04:48 |
| clarkb | asalkeld: this always trips me up and they have done a UI update | 04:50 |
| asalkeld | I only see "create organization" | 04:50 |
| asalkeld | also I have only logged in with my twitter account | 04:51 |
| asalkeld | ga, red tape | 04:51 |
| clarkb | asalkeld: I think the switch to organizations has borked us | 04:58 |
| clarkb | asalkeld: projects belong to organizations and not individual users now | 04:58 |
| clarkb | asalkeld: I don't appear to have openstack organization powers | 04:59 |
| clarkb | asalkeld: can should probably try getting in touch with reed, daisy, and or fifieldt | 04:59 |
| fifieldt | hi | 04:59 |
| clarkb | and possibly jeblair or gabrielhurley to see if one of them has organziation super powers | 04:59 |
| clarkb | fifieldt: transifex added this new organization feature. There is an OpenStack organization but I don't seem to have the ability to add projects to it | 05:00 |
| clarkb | fifieldt: any idea of who might have that ability? | 05:00 |
| fifieldt | I think it was gabrielcw | 05:00 |
| fifieldt | who registered it initialy | 05:00 |
| fifieldt | though | 05:00 |
| fifieldt | keep in mind the difference between transifex 'organisations' and transifex 'projects' | 05:01 |
| asalkeld | well, I am happy if someone can register the Heat project | 05:01 |
| clarkb | fifieldt: right but you cannot add projects new projects as an individual anymore | 05:01 |
| clarkb | fifieldt: you must add it to an organization and heat belongs under openstack | 05:01 |
| fifieldt | indeed | 05:01 |
| asalkeld | yip | 05:01 |
| fifieldt | well, looking at it it seems like you should have the access | 05:02 |
| fifieldt | as listed as a maintainer | 05:02 |
| fifieldt | but daisy was saying that there are some odd permissions that are only available to the original registrant | 05:02 |
| fifieldt | this might be one | 05:02 |
| clarkb | fifieldt: I am maintainer of https://www.transifex.com/projects/p/openstack/ but not the organization | 05:02 |
| fifieldt | I'm trying to find the organisation | 05:03 |
| clarkb | fifieldt: previously this worked fine because $person added a project then we added it to the hub | 05:03 |
| fifieldt | got a link handy? | 05:03 |
| clarkb | fifieldt: https://www.transifex.com/organization/openstack | 05:03 |
| *** branen has joined #openstack-infra | 05:03 | |
| *** KurtMartin has joined #openstack-infra | 05:03 | |
| fifieldt | that's a "bad link" for me | 05:03 |
| asalkeld | me too | 05:03 |
| clarkb | hmm maybe I do have super powers | 05:04 |
| asalkeld | :) | 05:04 |
| clarkb | or am I just assoicated with the org | 05:04 |
| fifieldt | fyi - primer on organisations here: http://blog.transifex.com/post/57712611874/organizations | 05:04 |
| *** kmartin has quit IRC | 05:04 | |
| *** branen_ has quit IRC | 05:04 | |
| clarkb | but I cannot find the add project button anywhere on that page | 05:04 |
| fifieldt | "Project Ownership section of that project's settings" is my best guess | 05:05 |
| clarkb | fifieldt: it isn't a project though | 05:05 |
| clarkb | the project and the org are distinct | 05:05 |
| fifieldt | yes | 05:05 |
| *** Dr01d has joined #openstack-infra | 05:05 | |
| fifieldt | yes | 05:05 |
| fifieldt | but what I'm reading here leaves me to guess that maybe the toggle is under the project | 05:05 |
| fifieldt | rather than an organisation 'adding' a project | 05:05 |
| fifieldt | the project has to change some setting | 05:06 |
| clarkb | fifieldt: but there is no project yet | 05:06 |
| clarkb | fifieldt: we want to add a new project | 05:06 |
| * fifieldt reads more | 05:07 | |
| fifieldt | ok, I can see the button is missing that used to be there | 05:07 |
| fifieldt | fun times | 05:07 |
| fifieldt | ah - when logged in I can see the openstack organisation - that's an interesting one | 05:09 |
| fifieldt | ok | 05:09 |
| fifieldt | so | 05:09 |
| *** UtahDave has joined #openstack-infra | 05:09 | |
| fifieldt | when I made a new organisation | 05:09 |
| fifieldt | I go to that organisations dashboard | 05:09 |
| fifieldt | then i get a "+ NEW" button on the left menu | 05:09 |
| fifieldt | to make a new project | 05:09 |
| *** Dr01d has quit IRC | 05:09 | |
| clarkb | fifieldt: but that isn't present for the openstack org? | 05:10 |
| fifieldt | I am now going to try adding you as an admin to my test org | 05:10 |
| fifieldt | to see if you can see it | 05:10 |
| fifieldt | if you can't | 05:10 |
| fifieldt | we know we just need to find the original creator | 05:10 |
| fifieldt | you should now be able to see https://www.transifex.com/organization/test-deleteme | 05:11 |
| fifieldt | can you see the "+ NEW" button in its dashboard? | 05:11 |
| clarkb | I can see the button | 05:11 |
| fifieldt | ok, well that is whacked | 05:11 |
| clarkb | so I think we need to find out who the admin of the openstack org is | 05:11 |
| clarkb | and get them to add some more admins | 05:11 |
| *** senk has quit IRC | 05:11 | |
| fifieldt | indeed | 05:12 |
| clarkb | fifieldt: test-deleteme is an excellent org name btw :) | 05:12 |
| fifieldt | thank you :) | 05:12 |
| fifieldt | daisy is awake and at work right now | 05:12 |
| *** senk has joined #openstack-infra | 05:12 | |
| fifieldt | so I'd expect reasonably rapid response to her IBM email | 05:12 |
| *** senk has quit IRC | 05:12 | |
| fifieldt | you have those coordinates? | 05:12 |
| clarkb | fifieldt: I do somewhere | 05:12 |
| clarkb | I can write her an email and CC you guys but it is almost best time for me | 05:13 |
| fifieldt | indeed | 05:13 |
| asalkeld | thanks guys | 05:13 |
| fifieldt | ok, I need to run to a meeting, on +61 403 900 880 if there's anything urgent | 05:14 |
| *** SergeyLukjanov has joined #openstack-infra | 05:16 | |
| openstackgerrit | Angus Salkeld proposed a change to openstack-infra/config: Add translation updates for Heat https://review.openstack.org/43991 | 05:16 |
| clarkb | mail sent | 05:18 |
| *** michchap_ has joined #openstack-infra | 05:19 | |
| *** michchap has quit IRC | 05:22 | |
| *** nicedice_ has quit IRC | 05:24 | |
| *** vogxn has quit IRC | 05:31 | |
| *** vogxn has joined #openstack-infra | 05:34 | |
| *** dina_belova has joined #openstack-infra | 05:44 | |
| *** dina_belova has quit IRC | 05:48 | |
| *** sgviking has quit IRC | 05:52 | |
| *** sgviking has joined #openstack-infra | 05:52 | |
| lifeless | jeblair: thanks | 05:55 |
| lifeless | jeblair: https://review.openstack.org/#/c/43989/ - I have questions for you there :) | 05:55 |
| jeblair | lifeless: what are you trying to do? | 05:56 |
| lifeless | pabelanger: thanks | 05:56 |
| jeblair | lifeless: it suggests that, perhaps, if you are adding a new server, you might be doing something that would require root access to that server, and if so, that's how you go about it. | 05:56 |
| lifeless | pabelanger: now I need to figure out why it's full of errors :( | 05:56 |
| lifeless | jeblair: big picture : I'm trying to setup the openstack infrastructure for the testcabal projects - subunit etc etc; we can't use the openstack-ci instance of the openstack-ci infra because we don't have CLAs | 05:57 |
| jeblair | lifeless: people can and have added entire new servers to our system without root access, and that is the preferred method | 05:57 |
| lifeless | jeblair: medium picture, I'm trying to get familiar with the tooling and setup before I go head-on to the main thing, by following the docs. | 05:58 |
| *** odyssey4me has joined #openstack-infra | 05:59 | |
| lifeless | jeblair: my experimentation environment is a rackspace cloud instance | 05:59 |
| jeblair | lifeless: i'll note (for completeness) that CLA is an option in gerrit and there are projects in our system without the CLA enabled. but setting up your own is great. we want that to work too. | 06:00 |
| lifeless | jeblair: so it is a new server... | 06:00 |
| lifeless | jeblair: AFAICT there isn't a 'setting up your own HOWTO' at the moment; I intend to write one | 06:00 |
| lifeless | jeblair: (as I'm *also* presenting on that topic at PyCon NZ... sucker for punishment...) | 06:00 |
| jeblair | lifeless: that would be very welcome. i wish monty had done that when he set up his own. | 06:00 |
| *** vogxn has quit IRC | 06:01 | |
| jeblair | lifeless: so for the simple case of "i'm setting up a foo server and testing with a local.pp"... | 06:01 |
| jeblair | lifeless: i think the expectation is that you'll already have an account on the system in the appropriate sudo group, and the puppet config will not break that (or really even know about it) | 06:02 |
| lifeless | ok | 06:03 |
| lifeless | jeblair: would it make sense / be possible for site.pp to have the domain parameterised ? | 06:03 |
| jeblair | lifeless: for the "i'm setting up an alternate infrastructure" case, you'll want wholesale replacement of site.pp, and you probably want to have your own local equivalents of the openstack_project/manifest files | 06:03 |
| jeblair | lifeless: possibly.. | 06:04 |
| jeblair | lifeless: http://ci.openstack.org/sysadmin.html#making-a-change-in-puppet | 06:04 |
| jeblair | lifeless: that section lays out the current api boundaries, so to speak... | 06:04 |
| clarkb | jeblair: I looked very briefly at using r10k (similar to puppet librarian) to start facilitating that split | 06:04 |
| lifeless | jeblair: yeah | 06:05 |
| jeblair | lifeless: at least in theory. that theory was settled upon _well_ after we started | 06:05 |
| lifeless | jeblair: I am expecting dragons :) | 06:05 |
| clarkb | jeblair: if we used that then openstack-infra/config would become manifests/site.pp, the supporting scripts, a puppetfile, and modules/openstack_project. Everything else would be an independent repo independently consumable | 06:05 |
| jeblair | lifeless: and the boundaries are very much not clearly observed | 06:05 |
| lifeless | jeblair: how do you do puppet syntax checks ? | 06:05 |
| jeblair | lifeless: i think the whole system needs a little feedback on whether those boundaries are good, or what alternates would be useful. | 06:06 |
| lifeless | jeblair: Ok; I am taking copious notes ;) | 06:06 |
| jeblair | lifeless, clarkb: i think we are all well agreed that pretty much all the stuff in modules should be individual puppetforge modules | 06:07 |
| jeblair | forking those out is a slow process | 06:07 |
| lifeless | modules/openstack_project/manifests/users.pp seems like something that clearly should be openstack specific | 06:08 |
| lifeless | not how-openstack-runs-things specific | 06:08 |
| lifeless | jeblair: or possibly even in hiera? | 06:08 |
| *** vogxn has joined #openstack-infra | 06:08 | |
| jeblair | lifeless: possibly. | 06:09 |
| lifeless | jeblair: so, the sense I'm getting is that I should setup a fork, maintained via rebasing, to track the delta. | 06:09 |
| lifeless | jeblair: e.g. my infra won't want existing openstack folk as root; will want me and jml and jelmer etc. | 06:10 |
| jeblair | lifeless: things like, well, "if you stand this up, should you have 4 load-balanced git servers?" become interesting questions. | 06:10 |
| lifeless | jeblair: being able to run it all on one node for pathologically small setups would be fun too. | 06:10 |
| clarkb | all in one is hard because apache | 06:10 |
| lifeless | jeblair: [without setting up a nested lxc cloud .. :)] | 06:10 |
| clarkb | doable but that would probably involve a fair amount of puppet mangling | 06:11 |
| lifeless | so sysadmin.rst says 'can do syntax checks' | 06:11 |
| lifeless | how does one do that? | 06:11 |
| jeblair | lifeless: yeah; basically the individual modules/ should clearly be reusable and failure there is an immediate bug; openstack_project you will probably need your own variants of those manifests except for very simple servers. | 06:11 |
| clarkb | lifeless: `puppet parser validate foo.pp` and puppet lint cover our syntax checking | 06:12 |
| clarkb | rake is configured to run puppet lint for you iirc | 06:12 |
| lifeless | so you run 'rake' at the top ? | 06:12 |
| clarkb | ya | 06:13 |
| lifeless | and it doesn't apply? | 06:13 |
| clarkb | nope | 06:14 |
| lifeless | http://paste.ubuntu.com/6035521/ | 06:14 |
| lifeless | http://paste.ubuntu.com/6035522/ | 06:14 |
| lifeless | this is me trying the etherpad example from the docs | 06:14 |
| lifeless | any hints appreciated (including 'dammit those docs are stale') | 06:14 |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/jenkins-job-builder: Add Plot plugin support https://review.openstack.org/43685 | 06:14 |
| lifeless | bbs, bath time for C | 06:16 |
| jeblair | lifeless: i think the etherpad module does not describe its dependencies completely | 06:17 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Fix semantic error in example config. https://review.openstack.org/44000 | 06:17 |
| jeblair | lifeless: and things were being run in the wrong order | 06:17 |
| clarkb | jeblair: that is what it looks like to me | 06:17 |
| jeblair | lifeless: re-running should converge on a working system | 06:17 |
| jeblair | lifeless: fixing involves adding things like "require => User['eplite']" to the Vcsrepo definition, etc | 06:18 |
| clarkb | http://git.openstack.org/cgit/openstack-infra/config/tree/modules/etherpad_lite/manifests/init.pp#n97 need an additional require | 06:18 |
| clarkb | line 105 too | 06:18 |
| jeblair | lifeless: (basically analysing those errors for dependency problems) | 06:18 |
| *** thomasbiege has joined #openstack-infra | 06:18 | |
| jeblair | lifeless: it's past my bedtime; good luck, ttyl. | 06:19 |
| clarkb | mine too | 06:19 |
| clarkb | jeblair: I did just forward you an email that you will probably be interested in but it can wait for tomorrow | 06:19 |
| jeblair | whee | 06:20 |
| *** thomasbiege has quit IRC | 06:22 | |
| lifeless | clarkb: jeblair: thank you! | 06:31 |
| *** hashar has joined #openstack-infra | 06:38 | |
| *** dkranz has quit IRC | 06:39 | |
| lifeless | clarkb: rake blows up, should I file a bug ? | 06:40 |
| *** dkranz has joined #openstack-infra | 06:41 | |
| clarkb | lifeless: sure | 06:41 |
| clarkb | lifeless: I would expect it to run locally for devs | 06:42 |
| *** dina_belova has joined #openstack-infra | 06:44 | |
| lifeless | :!rake | 06:45 |
| lifeless | [No write since last change] | 06:45 |
| lifeless | rake aborted! | 06:45 |
| lifeless | cannot load such file -- puppetlabs_spec_helper/rake_tasks | 06:45 |
| clarkb | lifeless: `rake lint 2>&1` is what we run in the jenkins job | 06:47 |
| clarkb | maybe it needs a target? | 06:47 |
| lifeless | nope | 06:48 |
| lifeless | https://bugs.launchpad.net/openstack-ci/+bug/1217722 | 06:49 |
| uvirtbot | Launchpad bug 1217722 in openstack-ci "'rake' or 'rake lint' blows up" [Undecided,New] | 06:49 |
| *** dina_belova has quit IRC | 06:49 | |
| *** jpich has joined #openstack-infra | 06:50 | |
| lifeless | clarkb: so you were saying init.pp in etherpad_lite needed User requires, or something else ? | 06:52 |
| lifeless | clarkb: also I suspect that not having ssl certs might be part of the problem; they dont' look particularly optional to me | 06:53 |
| *** afazekas has joined #openstack-infra | 06:58 | |
| *** dina_belova has joined #openstack-infra | 07:01 | |
| *** vogxn has quit IRC | 07:12 | |
| *** jpich has quit IRC | 07:15 | |
| *** jpich has joined #openstack-infra | 07:17 | |
| jhesketh | jeblair: ping | 07:18 |
| *** boris-42 has quit IRC | 07:24 | |
| *** jfriedly has quit IRC | 07:26 | |
| *** Bada has joined #openstack-infra | 07:27 | |
| *** shardy_afk is now known as shardy | 07:27 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support for emailing results via SMTP https://review.openstack.org/42645 | 07:40 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Separate reporters from triggers https://review.openstack.org/42643 | 07:40 |
| *** UtahDave has quit IRC | 07:42 | |
| *** jhesketh has quit IRC | 07:48 | |
| *** jhesketh has joined #openstack-infra | 07:49 | |
| *** ruhe has joined #openstack-infra | 08:03 | |
| *** afazekas has quit IRC | 08:03 | |
| *** sgviking has quit IRC | 08:10 | |
| *** Bada has quit IRC | 08:10 | |
| *** thomasbiege has joined #openstack-infra | 08:16 | |
| *** derekh has joined #openstack-infra | 08:21 | |
| *** thomasbiege has quit IRC | 08:23 | |
| *** SergeyLukjanov has quit IRC | 08:24 | |
| *** hashar has quit IRC | 08:37 | |
| openstackgerrit | A change was merged to openstack/requirements: Add jsonpath-rw https://review.openstack.org/43503 | 08:39 |
| *** dina_belova has quit IRC | 08:45 | |
| *** dina_belova has joined #openstack-infra | 08:46 | |
| *** vogxn has joined #openstack-infra | 08:47 | |
| *** Bada has joined #openstack-infra | 08:49 | |
| *** dina_belova has quit IRC | 08:50 | |
| *** dina_belova has joined #openstack-infra | 08:53 | |
| *** Bada has quit IRC | 08:56 | |
| openstackgerrit | daisy-ycguo proposed a change to openstack-infra/config: Update Japanese Ops Guide build job. https://review.openstack.org/43734 | 08:58 |
| *** SergeyLukjanov has joined #openstack-infra | 09:04 | |
| *** afazekas has joined #openstack-infra | 09:06 | |
| *** Bada has joined #openstack-infra | 09:07 | |
| *** rpodolyaka has joined #openstack-infra | 09:11 | |
| *** rpodolyaka has left #openstack-infra | 09:11 | |
| *** rpodolyaka has joined #openstack-infra | 09:11 | |
| *** rpodolyaka has left #openstack-infra | 09:11 | |
| *** rpodolyaka has joined #openstack-infra | 09:11 | |
| *** rpodolyaka has left #openstack-infra | 09:11 | |
| *** dina_belova has quit IRC | 09:12 | |
| *** dina_belova has joined #openstack-infra | 09:13 | |
| *** dina_belova has quit IRC | 09:15 | |
| *** dina_belova has joined #openstack-infra | 09:15 | |
| *** yaguang has joined #openstack-infra | 09:15 | |
| vogxn | hi all. i'm using go 1.1 on darwin and find that my build of a go pkg is failing with "permission denied" since today morning | 09:33 |
| vogxn | i haven't changed anything in my go env. it is very confusing why this is happening | 09:33 |
| vogxn | anyone encounter this before | 09:33 |
| vogxn | ? | 09:33 |
| vogxn | oops | 09:34 |
| vogxn | wrong channel | 09:34 |
| *** dina_belova has quit IRC | 09:36 | |
| *** dina_belova has joined #openstack-infra | 09:36 | |
| *** boris-42 has joined #openstack-infra | 09:37 | |
| *** dina_belova has quit IRC | 09:40 | |
| *** morganfainberg is now known as morganfainberg|a | 09:45 | |
| *** saper has quit IRC | 09:45 | |
| *** saper has joined #openstack-infra | 09:48 | |
| *** SergeyLukjanov has quit IRC | 09:54 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:59 | |
| *** odyssey4me has quit IRC | 10:04 | |
| *** pcm_ has joined #openstack-infra | 10:05 | |
| *** yaguang has quit IRC | 10:05 | |
| *** pcm_ has quit IRC | 10:06 | |
| *** pcm_ has joined #openstack-infra | 10:06 | |
| *** odyssey4me has joined #openstack-infra | 10:13 | |
| openstackgerrit | Roman Podolyaka proposed a change to openstack-infra/config: Fix new sqlalchemy-migrate jobs https://review.openstack.org/44021 | 10:13 |
| *** odyssey4me has quit IRC | 10:17 | |
| lifeless | russellb: how hard do you think it would be to add to http://russellbryant.net/openstack-stats/nova-openreviews.txt a compare-and-contrast of nova core reviewer vs non-core reviewers ? | 10:19 |
| *** ruhe has quit IRC | 10:20 | |
| *** odyssey4me has joined #openstack-infra | 10:24 | |
| *** saper has quit IRC | 10:27 | |
| *** senk has joined #openstack-infra | 10:28 | |
| *** saper has joined #openstack-infra | 10:31 | |
| EmilienM | mgagne: hi, why did not you approved https://review.openstack.org/#/c/43852/ since I already have a code-review from Khai Do ? | 10:36 |
| *** senk has quit IRC | 10:37 | |
| *** chuckieb has quit IRC | 10:41 | |
| *** dina_belova has joined #openstack-infra | 10:47 | |
| *** chuckieb has joined #openstack-infra | 10:49 | |
| *** johnthetubaguy has joined #openstack-infra | 10:49 | |
| *** dina_belova has quit IRC | 10:51 | |
| *** vogxn has quit IRC | 10:53 | |
| *** ruhe has joined #openstack-infra | 11:00 | |
| *** weshay has joined #openstack-infra | 11:01 | |
| *** senk has joined #openstack-infra | 11:04 | |
| *** dina_belova has joined #openstack-infra | 11:04 | |
| *** alexpilotti has joined #openstack-infra | 11:04 | |
| *** senk has quit IRC | 11:08 | |
| *** alexpilotti has quit IRC | 11:09 | |
| *** jamiem has joined #openstack-infra | 11:11 | |
| *** jamiem has quit IRC | 11:16 | |
| openstackgerrit | Zane Bitter proposed a change to openstack-infra/reviewstats: Add Liang to heat-core https://review.openstack.org/44028 | 11:16 |
| *** senk has joined #openstack-infra | 11:24 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/jenkins-job-builder: Add IRCbot plugin support https://review.openstack.org/44032 | 11:26 |
| *** adalbas has joined #openstack-infra | 11:28 | |
| *** lcestari has joined #openstack-infra | 11:31 | |
| *** hashar has joined #openstack-infra | 11:35 | |
| *** dims has joined #openstack-infra | 11:41 | |
| openstackgerrit | will soula proposed a change to openstack-infra/jenkins-job-builder: Adding validation to the ansicolor plugin https://review.openstack.org/43926 | 11:54 |
| *** zul has quit IRC | 11:55 | |
| *** ArxCruz has joined #openstack-infra | 11:57 | |
| *** ruhe has quit IRC | 11:59 | |
| *** afazekas has quit IRC | 12:02 | |
| *** jhesketh_ has quit IRC | 12:06 | |
| *** afazekas has joined #openstack-infra | 12:14 | |
| *** dims has quit IRC | 12:16 | |
| *** dims has joined #openstack-infra | 12:23 | |
| *** dprince has joined #openstack-infra | 12:23 | |
| *** rfolco has joined #openstack-infra | 12:31 | |
| openstackgerrit | Steven Hardy proposed a change to openstack/requirements: bump keystoneclient to 0.3.2 https://review.openstack.org/43919 | 12:38 |
| *** anteaya has joined #openstack-infra | 12:39 | |
| *** ruhe has joined #openstack-infra | 12:44 | |
| anteaya | lifeless: can you run the rake and rake lint tasks with the --trace flag and add the output to https://bugs.launchpad.net/openstack-ci/+bug/1217722 , please? | 12:45 |
| uvirtbot | Launchpad bug 1217722 in openstack-ci "'rake' or 'rake lint' blows up" [Undecided,New] | 12:45 |
| *** afazekas has quit IRC | 12:46 | |
| anteaya | also knowing what project you are in when rake aborts is helpful | 12:49 |
| anteaya | rake is the equivalent of make for ruby | 12:49 |
| *** thomasm has joined #openstack-infra | 12:49 | |
| anteaya | and derives its behaviour from a RAKEFILE or rakefile which is individually configured in the project | 12:50 |
| *** woodspa has joined #openstack-infra | 12:50 | |
| *** gaelL has quit IRC | 12:50 | |
| anteaya | so running rake in one project can result in different behaviour than running rake in another project | 12:50 |
| *** dina_belova has quit IRC | 12:50 | |
| *** dina_belova has joined #openstack-infra | 12:51 | |
| anteaya | often rake takes an argument - which is a command correlating to a command defined in the rakefile, there is also some configuration for rake with no arguments to tell it what to run | 12:51 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Add groups-dev.openstack.org node https://review.openstack.org/42608 | 12:51 |
| *** xchu has quit IRC | 12:51 | |
| anteaya | usually rake with no arguments means run the tests - but that can very project to project as well | 12:52 |
| anteaya | s/very/vary | 12:52 |
| *** vogxn has joined #openstack-infra | 12:52 | |
| thomasm | Hello all! | 12:54 |
| *** dina_belova has quit IRC | 12:55 | |
| *** jergerber has joined #openstack-infra | 12:55 | |
| *** SergeyLukjanov has quit IRC | 12:56 | |
| *** jhesketh_ has joined #openstack-infra | 12:56 | |
| *** jhesketh_ has quit IRC | 12:56 | |
| *** maurosr has quit IRC | 13:00 | |
| *** sgviking has joined #openstack-infra | 13:00 | |
| soren | Do we not have test coverage information anymore? | 13:01 |
| * soren can't seem to find it | 13:01 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:02 | |
| *** dina_belova has joined #openstack-infra | 13:02 | |
| anteaya | hello thomasm | 13:04 |
| anteaya | soren: what kind of test coverage information are you looking for that you can't find? | 13:04 |
| soren | anteaya: nova's. | 13:05 |
| soren | anteaya: I can see from the Jenkins log that the coverage report is being uploaded to static.openstack.org, but I don't know how to reach it. | 13:06 |
| anteaya | hey ttx, what do you want to do with https://review.openstack.org/#/c/43002/? It was announced yesterday and no additional reviews were added, do you want it in or do you want to wait? | 13:06 |
| soren | I'm not familiar with the fs path -> url translation. | 13:06 |
| anteaya | soren: do you have the url for a patch? | 13:06 |
| soren | anteaya: I'm just interested in the newest coverage report. | 13:06 |
| soren | anteaya: I can find the most recently accepted patch, if that helps. | 13:07 |
| ttx | anteaya: I +1ed it, meaning I'm fine with it being merged | 13:07 |
| anteaya | so far, I have no idea what you are talking about | 13:07 |
| soren | I just see stuff like this in the Jenkins log: 2013-08-28 07:09:32.486 | [SCP] uploading file: '/srv/static/logs/a6/a6af830748f0a76fd40b3cc609aeabe8954189f7/post/nova-coverage/35e84ce/cover/style.css' | 13:07 |
| *** afazekas has joined #openstack-infra | 13:07 | |
| *** jamiem has joined #openstack-infra | 13:07 | |
| soren | ...but I don't know how to reach that over http. | 13:07 |
| anteaya | so yes, soren, if I could see a patch I might get a sense of context | 13:07 |
| anteaya | ttx, okay thanks | 13:07 |
| soren | That log snippet is from https://jenkins02.openstack.org/job/nova-coverage/102/consoleFull | 13:07 |
| anteaya | thanks, looking | 13:08 |
| soren | The top-most patch there is a6af830748f0a76fd40b3cc609aeabe8954189f7 | 13:08 |
| *** markmc has joined #openstack-infra | 13:08 | |
| anteaya | hey markmc | 13:08 |
| markmc | hi anteaya | 13:09 |
| soren | Change ID: I8da1708bc45ea6a07e24f4025c2a0ebf13fd21d0 | 13:09 |
| soren | https://review.openstack.org/#/c/43117/ | 13:09 |
| anteaya | yes found it on the change id, thanks | 13:10 |
| soren | I don't care about the patch at all. I just want the most up-to-date coverage report for Nova. | 13:10 |
| anteaya | well this is a new question for me soren, so I don't know the answer | 13:10 |
| *** weshay has quit IRC | 13:10 | |
| soren | anteaya: Never mind. Look at the Apache config for static.openstack.org and work it out that way. | 13:11 |
| anteaya | the only suggestion I have is to wait for jeblair to wake up, now that zuul and friends are purring nicely, he has been showing up around 9am Pacific time | 13:12 |
| *** cppcabrera has joined #openstack-infra | 13:12 | |
| *** mriedem has joined #openstack-infra | 13:13 | |
| soren | Never mind. Found it. | 13:13 |
| soren | http://logs.openstack.org/a6/a6af830748f0a76fd40b3cc609aeabe8954189f7/post/nova-coverage/35e84ce/cover/ | 13:13 |
| soren | anteaya: Thanks. | 13:14 |
| anteaya | ah okay, thanks soren I have never seen that before and didn't know it existed | 13:14 |
| anteaya | glad you found it | 13:14 |
| anteaya | you are being kind, I wasn't any help | 13:14 |
| anteaya | but thanks | 13:14 |
| *** zul has joined #openstack-infra | 13:15 | |
| *** dguitarbite has joined #openstack-infra | 13:15 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add Liang to heat-core https://review.openstack.org/44028 | 13:16 |
| *** sandywalsh has quit IRC | 13:18 | |
| *** dkranz has quit IRC | 13:19 | |
| *** ruhe has quit IRC | 13:23 | |
| *** weshay has joined #openstack-infra | 13:24 | |
| *** dkliban has joined #openstack-infra | 13:31 | |
| *** sandywalsh has joined #openstack-infra | 13:31 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-dev/pbr: Add pypy to tox.ini https://review.openstack.org/44051 | 13:36 |
| *** fifieldt has quit IRC | 13:36 | |
| *** michchap_ has quit IRC | 13:45 | |
| *** michchap has joined #openstack-infra | 13:45 | |
| *** burt has joined #openstack-infra | 13:46 | |
| *** dina_belova has quit IRC | 13:47 | |
| *** dina_belova has joined #openstack-infra | 13:47 | |
| *** changbl has quit IRC | 13:50 | |
| *** ruhe has joined #openstack-infra | 13:51 | |
| *** dina_belova has quit IRC | 13:52 | |
| *** maurosr has joined #openstack-infra | 13:52 | |
| *** arezadr has quit IRC | 13:56 | |
| *** arezadr has joined #openstack-infra | 13:56 | |
| *** ruhe has quit IRC | 13:57 | |
| *** markmc has quit IRC | 13:57 | |
| *** ruhe has joined #openstack-infra | 14:01 | |
| *** arezadr has quit IRC | 14:03 | |
| *** ruhe has quit IRC | 14:04 | |
| *** amotoki_ has quit IRC | 14:07 | |
| *** _TheDodd_ has joined #openstack-infra | 14:09 | |
| *** rnirmal has joined #openstack-infra | 14:09 | |
| *** _TheDodd_ has quit IRC | 14:13 | |
| *** thomasbiege has joined #openstack-infra | 14:13 | |
| *** annegentle has quit IRC | 14:15 | |
| *** dina_belova has joined #openstack-infra | 14:18 | |
| *** thedodd has joined #openstack-infra | 14:20 | |
| openstackgerrit | Daniel Izquierdo proposed a change to openstack-infra/config: Addition of the openstack-dash project to the StackForge environment https://review.openstack.org/44057 | 14:20 |
| *** dina_belova has quit IRC | 14:21 | |
| *** dina_belova has joined #openstack-infra | 14:21 | |
| *** annegentle_ has joined #openstack-infra | 14:25 | |
| *** rcleere has joined #openstack-infra | 14:26 | |
| jeblair | mtreinish: it looks like the gate is moving very slowly, maybe one or two changes at a time? | 14:34 |
| *** michchap has quit IRC | 14:36 | |
| *** thedodd has quit IRC | 14:36 | |
| *** jjmb has quit IRC | 14:37 | |
| *** thedodd has joined #openstack-infra | 14:37 | |
| *** yaguang has joined #openstack-infra | 14:39 | |
| *** thedodd has quit IRC | 14:39 | |
| mtreinish | jeblair: yeah there was a surge last night and I think it's the flaky tests getting in the way | 14:39 |
| *** thedodd has joined #openstack-infra | 14:40 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add SLOCCount plugin support https://review.openstack.org/43852 | 14:40 |
| *** portante is now known as portante|afk | 14:43 | |
| *** dkranz has joined #openstack-infra | 14:44 | |
| *** datsun180b has joined #openstack-infra | 14:46 | |
| *** changbl has joined #openstack-infra | 14:51 | |
| johnthetubaguy | hi all, it seems the cron job for http://status.openstack.org/reviews/ is not working again | 14:54 |
| johnthetubaguy | would be cool if someone could fix that us, or if anyone could let me know how I can help debug the issue, I could take a look | 14:54 |
| johnthetubaguy | BobBall: I guess there smokestack changes recently, did that break the review page cron job? | 14:55 |
| BobBall | which bit? | 14:56 |
| BobBall | oh | 14:56 |
| BobBall | hmmm | 14:56 |
| BobBall | I'm only aware of one change going in for reviewday | 14:56 |
| BobBall | and that wouldn't have broken it | 14:56 |
| BobBall | I hope | 14:56 |
| BobBall | lemme look | 14:57 |
| johnthetubaguy | BobBall: I have not looked into it, just wondered about the timing, it broke on 25th August | 14:57 |
| BobBall | the change was only yesterday | 14:57 |
| BobBall | 2 secs | 14:57 |
| BobBall | *blink* I think it's unrelated to reviewday | 14:58 |
| *** ruhe has joined #openstack-infra | 14:58 | |
| BobBall | Changes were on 15th and 27th | 14:58 |
| BobBall | so if it broke on the 25th then it might be smt else | 14:58 |
| BobBall | I'll just verify master works | 14:58 |
| EmilienM | mgagne: thx for approve ;) | 14:58 |
| mgagne | EmilienM: np | 14:59 |
| BobBall | Could be another BP bug like broke it last time... :) | 14:59 |
| jeblair | might be because of the server move | 14:59 |
| EmilienM | mgagne: could you approve the BP also ? | 14:59 |
| EmilienM | https://blueprints.launchpad.net/openstack-ci/+spec/jenkins-job-builder-sloccount | 14:59 |
| mgagne | checking | 14:59 |
| *** thomasbiege has quit IRC | 14:59 | |
| mgagne | mordred is the approver | 14:59 |
| EmilienM | mgagne: yeah, it's me | 15:00 |
| EmilienM | but you can change | 15:00 |
| EmilienM | I did not know who assign | 15:00 |
| jeblair | BobBall, johnthetubaguy:-rw-rw-r-- 1 reviewday reviewday 0 Aug 23 19:00 update.lock | 15:00 |
| mgagne | EmilienM: mordred is the approver, I cannot change it or update status | 15:00 |
| EmilienM | mgagne: can I do it ? | 15:00 |
| jeblair | BobBall, johnthetubaguy: looks like it doesn't deal well with stuck lockfiles | 15:00 |
| BobBall | where does that lock come from ? | 15:01 |
| EmilienM | mgagne: should be good now | 15:01 |
| BobBall | review day doesn't seem to use it directly... | 15:01 |
| jeblair | BobBall: cron job | 15:01 |
| BobBall | so outside of the reviewday repo? | 15:01 |
| jeblair | however, it seems to be running right now | 15:01 |
| EmilienM | mgagne: same for https://blueprints.launchpad.net/openstack-ci/+spec/jenkins-job-builder-irc and https://blueprints.launchpad.net/openstack-ci/+spec/jenkins-job-builder-plot | 15:02 |
| jeblair | so maybe the presence of the lockfile is fine | 15:02 |
| *** thomasbiege has joined #openstack-infra | 15:02 | |
| BobBall | Don't suppose you got mailed with an error? | 15:02 |
| johnthetubaguy | jeblair: still shows last updated as 25th | 15:02 |
| BobBall | Takes ages to run for me :) | 15:02 |
| *** rcleere_ has joined #openstack-infra | 15:02 | |
| BobBall | johnthetubaguy: Meeting time! | 15:03 |
| *** thomasbiege has quit IRC | 15:03 | |
| jeblair | the ownership of index.html may have been wrong; i've reset it to reviewday.reviewday | 15:03 |
| *** thomasbiege has joined #openstack-infra | 15:03 | |
| BobBall | That'd do it if it couldn't write there | 15:03 |
| johnthetubaguy | BobBall: yep, was waiting for other to finish | 15:03 |
| jeblair | i would expect that kind of error to result in output though, and i don't have any cronspam | 15:03 |
| *** jhesketh has quit IRC | 15:03 | |
| *** rcleere has quit IRC | 15:03 | |
| *** rcleere_ is now known as rcleere | 15:03 | |
| johnthetubaguy | jeblair: worth a shot though, thanks for looking | 15:04 |
| *** svarnau has joined #openstack-infra | 15:05 | |
| BobBall | It runs for me | 15:05 |
| BobBall | so there isn't an obvious bug in the code | 15:06 |
| BobBall | if it doesn't work there must be a setup issue | 15:06 |
| BobBall | jeblair: what were the perms before you changed it? | 15:06 |
| *** dina_belova has quit IRC | 15:06 | |
| jeblair | -rw-rw-r-- 1 1007 1008 766871 Aug 25 15:34 index.html | 15:07 |
| *** dina_belova has joined #openstack-infra | 15:07 | |
| jeblair | now -rw-rw-r-- 1 reviewday reviewday 743283 Aug 28 15:04 index.html | 15:07 |
| jeblair | pretty sure it's from the server move, uids got shuffled. | 15:07 |
| BobBall | ah fun! | 15:10 |
| BobBall | that sounds like it was the problem | 15:10 |
| *** dina_belova has quit IRC | 15:11 | |
| jeblair | it may mean a bug in the puppet code that does the install | 15:11 |
| *** thomasbiege has quit IRC | 15:11 | |
| BobBall | There should have been an error if it can't write to index.html... | 15:14 |
| BobBall | I got one | 15:14 |
| BobBall | I don't suppose the cronjob is canning stderr? ;) | 15:14 |
| *** branen_ has joined #openstack-infra | 15:15 | |
| *** vogxn has quit IRC | 15:15 | |
| jeblair | BobBall: ah, mail is delivered locally for that account. :( | 15:15 |
| EmilienM | mgagne: do you know why can't I see the review link on the blueprint https://blueprints.launchpad.net/openstack-ci/+spec/jenkins-job-builder-sloccount (the patch has been merged...) | 15:15 |
| EmilienM | is it a matter of approval ? | 15:16 |
| *** ftcjeff has joined #openstack-infra | 15:18 | |
| *** branen has quit IRC | 15:18 | |
| BobBall | EmilienM: Where is the patch? | 15:21 |
| EmilienM | BobBall: https://review.openstack.org/#/c/43852/ | 15:21 |
| EmilienM | did I something wrong ? | 15:21 |
| BobBall | Very interesting | 15:21 |
| BobBall | no idea | 15:21 |
| *** Bada has quit IRC | 15:21 | |
| BobBall | doesn't look like it | 15:21 |
| EmilienM | :) | 15:21 |
| BobBall | both topic and implements lines are there | 15:21 |
| EmilienM | yup | 15:22 |
| EmilienM | again, maybe it's because the blueprint has not been approved | 15:22 |
| EmilienM | but I'm not sure... | 15:22 |
| *** pentameter has joined #openstack-infra | 15:23 | |
| *** dguitarbite has quit IRC | 15:23 | |
| jeblair | EmilienM: we don't really use blueprints in -infra, so we don't have the blueprint script configured to map to the launchpad project | 15:23 |
| *** rnirmal has quit IRC | 15:23 | |
| *** rnirmal has joined #openstack-infra | 15:24 | |
| jeblair | EmilienM: (it technically failed because there is no mapping from jenkins-job-builder -> openstack-ci) | 15:24 |
| EmilienM | jeblair: I see, I won't write BP anymore for my infra contributions :) | 15:25 |
| EmilienM | jeblair: do you want to delete them ? or... | 15:25 |
| *** gaelL has joined #openstack-infra | 15:26 | |
| jeblair | EmilienM: no, don't worry about it. | 15:26 |
| EmilienM | cool :) | 15:26 |
| *** hashar has quit IRC | 15:26 | |
| *** dkranz has quit IRC | 15:29 | |
| *** pycabrera has joined #openstack-infra | 15:34 | |
| *** vogxn has joined #openstack-infra | 15:37 | |
| *** cppcabrera has quit IRC | 15:37 | |
| *** zul has quit IRC | 15:41 | |
| *** UtahDave has joined #openstack-infra | 15:42 | |
| *** dkranz has joined #openstack-infra | 15:43 | |
| johnthetubaguy | jeblair: thanks for fixing that review status stuff :) | 15:43 |
| *** jpich has quit IRC | 15:44 | |
| jeblair | johnthetubaguy: no prob | 15:45 |
| *** ruhe has quit IRC | 15:52 | |
| *** olaph has quit IRC | 15:53 | |
| *** burt has quit IRC | 15:56 | |
| *** KurtMartin is now known as kmartin | 15:57 | |
| clarkb | morning | 15:57 |
| *** reed has joined #openstack-infra | 15:57 | |
| *** gyee has joined #openstack-infra | 15:58 | |
| openstackgerrit | Daniel Izquierdo proposed a change to openstack-infra/config: Addition of the community-metrics-dash project to the StackForge environment https://review.openstack.org/44057 | 15:58 |
| jeblair | clarkb: good morning | 15:58 |
| jeblair | clarkb: when you have a minute, can you take a look at my 2 zuul changes? | 15:59 |
| *** changbl has quit IRC | 15:59 | |
| dtroyer | Something changed in the Gerrit UI recently. On the review page the Download command text box now behaves rather badly…in Chrome a second click makes the text box disappear completely and I can't see how to get it back. In Safari, the first click highlights the entire text box but Cmd-C doesn't copy and the highlight is removed. Fortunately Edit -> Copy menu works. | 16:01 |
| dtroyer | https://bugs.launchpad.net/openstack-ci/+bug/1217954 | 16:01 |
| uvirtbot | Launchpad bug 1217954 in openstack-ci "Gerrit UI: Download command text box misbehaves" [Undecided,New] | 16:01 |
| *** pycabrera is now known as cppcabrera | 16:01 | |
| dtroyer | Are there any special tags that should go on that bug? | 16:01 |
| clarkb | dkranz we still run the normal netron tempest job | 16:01 |
| clarkb | jeblair sure | 16:01 |
| jeblair | dtroyer: i believe that bug has been there for a very long time | 16:02 |
| dtroyer | jeblair: I've been copying that box for a while in Chrone but now it totally disappears | 16:02 |
| jeblair | dtroyer: maybe something changed in chrome? because i've dealt with that more or less from the start with firefox | 16:02 |
| jeblair | dtroyer: also, the flash clipboard widget may be involved too... | 16:03 |
| jeblair | dtroyer: (i don't use it) | 16:03 |
| dtroyer | jeblair: could be, I don't think I've updated Chrome recently. | 16:04 |
| jeblair | dtroyer: it looks like it's fixed in 2.7 | 16:05 |
| jeblair | dtroyer: just tried it out on https://gerrit-review.googlesource.com/#/c/49274/ | 16:05 |
| *** changbl has joined #openstack-infra | 16:05 | |
| jeblair | dtroyer: hopefully we'll be able to upgrade soon (zaro is working on getting a change in upstream that we need) | 16:05 |
| dtroyer | jeblair: up, that works for me too. | 16:06 |
| dtroyer | looks like I'm using Safari for a while... | 16:06 |
| dtroyer | thanks | 16:07 |
| jeblair | dtroyer: know about 'git review -d' and 'git review -x' ? | 16:07 |
| *** nati_ueno has joined #openstack-infra | 16:08 | |
| dtroyer | jeblair: I thnk I'm about to! | 16:08 |
| jeblair | dtroyer: 'git review -d ####' downloads and checks-out into a new branch change no #### | 16:08 |
| jeblair | dtroyer: -x #### cherry-picks change #### onto the current branch | 16:08 |
| dtroyer | jeblair: -x, where have you been all my life??? | 16:09 |
| dtroyer | jeblair: I owe you a beer in HK for this | 16:09 |
| jeblair | and i didn't even write it. :) | 16:09 |
| dtroyer | I'm rewarding the messenger | 16:09 |
| jeblair | thanks, that's nice, that's not usually what happens to messengers. :) | 16:10 |
| *** portante|afk is now known as portante | 16:12 | |
| russellb | can someone restore this change for me so I can update it? Thanks! https://review.openstack.org/#/c/32187 | 16:14 |
| jeblair | russellb: done | 16:15 |
| russellb | thanks! | 16:15 |
| *** hemnafk is now known as hemna | 16:17 | |
| *** dina_belova has joined #openstack-infra | 16:17 | |
| *** boris-42 has quit IRC | 16:21 | |
| *** jfriedly has joined #openstack-infra | 16:21 | |
| *** dina_belova has quit IRC | 16:22 | |
| *** nati_ueno has quit IRC | 16:22 | |
| clarkb | jeblair: reviewed | 16:22 |
| clarkb | jeblair: they did what I expected, preserved the bug fix but reverted the optimizations | 16:22 |
| *** nati_ueno has joined #openstack-infra | 16:23 | |
| *** markmcclain has joined #openstack-infra | 16:23 | |
| *** Ryan_Lane has quit IRC | 16:26 | |
| *** pcrews has quit IRC | 16:26 | |
| *** nati_ueno has quit IRC | 16:27 | |
| *** zul has joined #openstack-infra | 16:28 | |
| *** svarnau_ has joined #openstack-infra | 16:29 | |
| clarkb | jeblair: the rackspace network stuff begins in ~1.5 hours? | 16:31 |
| jeblair | clarkb: i think it's over; i think it was midnight-4 central | 16:31 |
| *** svarnau has quit IRC | 16:31 | |
| clarkb | I must've completely misread the timestamps | 16:32 |
| clarkb | oh I bet I read them as PM not AM. I can timezone honest | 16:32 |
| clarkb | jeblair: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=862&rra_id=all is a neat graph showing how many bits git.o.o pushes | 16:33 |
| *** ruhe has joined #openstack-infra | 16:33 | |
| jeblair | clarkb: it pushes some bits! | 16:34 |
| *** olaph has joined #openstack-infra | 16:38 | |
| *** SergeyLukjanov has quit IRC | 16:40 | |
| dkranz | clarkb: If you are referring to my message to dev, the job we are running are "smoke tests" only. | 16:42 |
| *** ruhe has quit IRC | 16:42 | |
| *** ruhe has joined #openstack-infra | 16:42 | |
| dkranz | clarkb: I should have made that more clear and will send a followup. | 16:42 |
| *** cppcabrera is now known as cppcabrera_afk | 16:44 | |
| *** markmcclain has quit IRC | 16:47 | |
| *** thomasbiege has joined #openstack-infra | 16:47 | |
| *** briancline has quit IRC | 16:48 | |
| *** afazekas has quit IRC | 16:49 | |
| *** olaph has quit IRC | 16:49 | |
| *** odyssey4me has quit IRC | 16:53 | |
| *** dina_belova has joined #openstack-infra | 16:54 | |
| openstackgerrit | A change was merged to openstack-infra/config: Just run the install_puppet/modules scripts. https://review.openstack.org/43987 | 16:58 |
| *** markmcclain has joined #openstack-infra | 16:58 | |
| *** ruhe has quit IRC | 16:59 | |
| dkranz | clarkb: BTW, my :( was about the fact that the full neutron job is not being fixed rather than it moving to experimental. | 16:59 |
| *** alexpilotti has joined #openstack-infra | 17:00 | |
| dkranz | clarkb: But I suppose "out-of-sight, out-of-mind" is possible | 17:00 |
| clarkb | dkranz: ya that is a concern | 17:00 |
| dkranz | clarkb: We (tempest) have tried to get folks to fix the issues for months but no luck. | 17:01 |
| clarkb | :( | 17:01 |
| dkranz | clarkb: We are kind of out of ideas. | 17:01 |
| dkranz | clarkb: Maybe we should propose a neutron summit session (half-smiley) | 17:02 |
| jeblair | dkranz: oh, i thought they got involved after we stopped running tests on them. | 17:02 |
| jeblair | dkranz: have they dropped out again? | 17:02 |
| *** nicedice_ has joined #openstack-infra | 17:02 | |
| dkranz | jeblair: This comes up every week at the tempest meeting | 17:03 |
| *** markmcclain has quit IRC | 17:03 | |
| jeblair | dkranz: sorry, i've been distracted and haven't been paying attention; can you catch me up real quick? | 17:03 |
| dkranz | jeblair: The neutron full tempest tests have never passed and were set to nonvoting a while back | 17:04 |
| jeblair | yeah, that i know | 17:04 |
| dkranz | jeblair: No one has been willing to fix them as far as we know | 17:04 |
| *** SergeyLukjanov has joined #openstack-infra | 17:05 | |
| dkranz | jeblair: THey were recently moved to experimental and I was worried this would not even be noticed anymore | 17:05 |
| *** burt has joined #openstack-infra | 17:05 | |
| dkranz | jeblair: Putting us even a step further away from ever getting fixed | 17:05 |
| *** svarnau has joined #openstack-infra | 17:05 | |
| dkranz | jeblair: experimental should be for experimental new features, not things that are supposed to work already. | 17:05 |
| jeblair | dkranz: what does markmcclain have to say? | 17:06 |
| jeblair | dkranz: well, neutron never worked and is not expected to work, is the problem | 17:06 |
| dkranz | jeblair: Yes | 17:06 |
| jeblair | dkranz: experimental is there so that the people working on fixing it can still do what they need | 17:06 |
| *** svarnau_ has quit IRC | 17:06 | |
| dkranz | jeblair: But the tempest team cannot fix it. | 17:06 |
| jeblair | dkranz: just running tests that people ignore is not helpful | 17:06 |
| dkranz | jeblair: I understand. I was not concerned about a technical issue. | 17:07 |
| *** derekh has quit IRC | 17:07 | |
| jeblair | dkranz: ok; this is a people issue | 17:07 |
| dkranz | jeblair: exactly | 17:07 |
| jeblair | dkranz: and i think it's markmcclain's responsibility to tell us what is happening to fix it | 17:07 |
| dkranz | jeblair: I know it is a waste of resourcds | 17:07 |
| jeblair | dkranz: what has he said on the subject? | 17:07 |
| dkranz | jeblair: I don't remember but it was a while ago | 17:07 |
| jeblair | dkranz: i don't care about the resources; it was moved to experimental because running it literally does not help anyone do anything | 17:08 |
| dkranz | jeblair: Perhaps my message to the dev list will prompt a response | 17:08 |
| dkranz | jeblair: If not I will ask it directly. | 17:08 |
| jeblair | dkranz: if people are ignoring it, it's only training them to ignore failing tests | 17:08 |
| *** pabelanger has quit IRC | 17:08 | |
| dkranz | jeblair: I don't disagree. | 17:08 |
| jeblair | dkranz: i'm happy to ask directly. and if i don't receive a satisfactory response, i have no problem submitting a motion to the tc to remove them as an integrated project | 17:08 |
| dkranz | jeblair: That would be interesting to say the least. | 17:09 |
| *** markmcclain has joined #openstack-infra | 17:09 | |
| jeblair | i believe it is not only our only leverage, but it is actually the right thing to do | 17:09 |
| *** olaph has joined #openstack-infra | 17:09 | |
| dkranz | jeblair: Above my pay grade :) | 17:10 |
| jeblair | if they are not an integrated part of openstack, then what are they doing? | 17:10 |
| jeblair | dkranz: hey, the project belongs to all of us. | 17:10 |
| *** ruhe has joined #openstack-infra | 17:10 | |
| dkranz | jeblair: I know. That's not what I meant. | 17:10 |
| dkranz | jeblair: Perhaps you could reply to the message I sent to the dev list a few hours ago. | 17:11 |
| dkranz | jeblair: I guess you think I was too gentle. | 17:11 |
| *** thomasbiege has quit IRC | 17:13 | |
| jeblair | dkranz: i'll send a private email and include you | 17:13 |
| dkranz | jeblair: OK. It is also possible that some of the issues actually require nova to be fixed. | 17:15 |
| *** fbo is now known as fbo_away | 17:18 | |
| *** pcrews has joined #openstack-infra | 17:21 | |
| *** Ryan_Lane has joined #openstack-infra | 17:21 | |
| *** pabelanger has joined #openstack-infra | 17:23 | |
| jeblair | dkranz: the plain (non-full) neutron test is also still non-parallel, right? | 17:24 |
| jeblair | mtreinish: ^? | 17:24 |
| dkranz | jeblair: Yes, because there were still some races/issues etc. | 17:24 |
| dkranz | jeblair: https://bugs.launchpad.net/tempest/+bug/1216076 | 17:26 |
| uvirtbot | Launchpad bug 1216076 in tempest "Neutron jobs won't pass tempest running in parallel" [Critical,Confirmed] | 17:26 |
| *** yaguang has quit IRC | 17:26 | |
| mtreinish | jeblair: actually it probably is running in parallel | 17:27 |
| mtreinish | because it just runs the tox full job | 17:27 |
| jeblair | mtreinish: i mean the non-full one | 17:28 |
| jeblair | mtreinish: the one we're actually gating on | 17:28 |
| mtreinish | the non full one is serial | 17:28 |
| mtreinish | I left the smoke tox job serial because of the bug dkranz mentioned | 17:28 |
| Alex_Gaynor | jeblair: do we have the ability to scale up the number of workers for the non-devstack jobs? the higher than average number of gate restarts is starting the check queue of those workers | 17:30 |
| jeblair | Alex_Gaynor: we can manually add workers, it takes a little bit of work but isn't too difficult | 17:30 |
| jeblair | Alex_Gaynor: i think the shorter runtimes (combined with high number of resets) exacerbates the situtation. | 17:32 |
| Alex_Gaynor | jeblair: yeah, devstack builds are keeping up nicely | 17:32 |
| *** portante is now known as portante|afk | 17:35 | |
| *** ruhe has quit IRC | 17:37 | |
| *** olaph has quit IRC | 17:40 | |
| pleia2 | updated https://wiki.openstack.org/wiki/Gerrit_Workflow and moved the details page to https://wiki.openstack.org/wiki/GerritJenkinsGit (it automatically sets up a redirect, so GerritJenkinsGithub still works) | 17:41 |
| *** boris-42 has joined #openstack-infra | 17:42 | |
| *** morganfainberg|a is now known as morganfainberg | 17:42 | |
| *** thomasbiege has joined #openstack-infra | 17:43 | |
| jeblair | mtreinish, dkranz: http://tinyurl.com/mbpm2uo | 17:43 |
| jeblair | mtreinish, dkranz: in case that's of any use | 17:43 |
| jog0 | jeblair: I have a very dumb zuul question | 17:43 |
| jeblair | jog0: i'll try to come up with a dumb answer :) | 17:44 |
| notmyname | wasn't there a setuptools->distribute->setuptools dependency chain issue at some point? did you guys fix it? | 17:44 |
| jog0 | status/zuul seems to show that in the gate pipeline jobs don't get retried until all the jobs in the bad patch finished | 17:45 |
| jog0 | jeblair: that correct? if so why? | 17:45 |
| jog0 | jeblair: another dumb question, why is savanah in the pipeline | 17:46 |
| jeblair | jog0: at the first failure of a job for a change at the _head_ of the queue, the head is popped, and jobs restarted | 17:46 |
| jeblair | jog0: so right now | 17:47 |
| jog0 | jeblair: why only for the head? | 17:47 |
| *** vogxn has quit IRC | 17:47 | |
| *** yjiang5_away is now known as yjiang5 | 17:47 | |
| jeblair | change C had failed, so every change behind it is canceled because we know that the series ABC is not going to merge | 17:47 |
| *** johnthetubaguy has quit IRC | 17:48 | |
| jeblair | but A and B may or may not fail, so we have to wait to see what happens to them | 17:48 |
| jeblair | if A and B merge, then C was correctly tested and will not merge, so the next head will be D | 17:48 |
| jeblair | if A or B fail to merge, then C was incorrectly tested and the next head is C | 17:48 |
| *** markmcclain has quit IRC | 17:49 | |
| jog0 | I was asking the the case of : ABC and jobs are running to test all of them. A is at the head, and there is a failure at B, but C appears to keep running | 17:49 |
| jog0 | even though we know C was tested incorrectly | 17:49 |
| jeblair | ah, C jobs should be canceled; perhaps it's a timing issue; the status display is actually more up to date than zuul's scheduler is sometimes. | 17:50 |
| jeblair | (also, if a job has already finished, that status will remain on the display) | 17:51 |
| *** sarob has joined #openstack-infra | 17:51 | |
| Alex_Gaynor | jeblair: an interesting problem I've observed, say you have masetr<-A<-B<-C; C fails some flaky test, but doesn't immediately get removed because A and B are still running. A then fails a thing, causing it to be popped, and B<-C to restart. The failure from C is never recorded, so it doesn't contribute the recehck counts | 17:51 |
| jeblair | jog0: as for savanah -- it's in a different change queue; you'll note that there is/was no arrow between it and the openstack queue. | 17:51 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Revert bogus merger optimizations https://review.openstack.org/43977 | 17:51 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Fix preprocessing for ref-updated events https://review.openstack.org/43978 | 17:51 |
| jeblair | jog0: ^ just like zuul also does not share that change queue | 17:52 |
| *** thomasbiege has quit IRC | 17:52 | |
| jeblair | Alex_Gaynor: correct. those data are available in logstash, but are not presented to developers, and actually don't even end up in graphite. | 17:52 |
| Alex_Gaynor | jeblair: we need some machine learning applied to rechecks to automatically tag them :) | 17:53 |
| Alex_Gaynor | (I'm told google has a systme like this for errort tracking from logs) | 17:53 |
| clarkb | back from a call and all this scrollback | 17:53 |
| clarkb | wouldn't it be nice to be google and have the worlds largest indexing infrastructure >_> | 17:54 |
| jog0 | jeblair: thanks. I aks because requeuing faster would help the gate I think | 17:55 |
| Alex_Gaynor | clarkb: and spare PHds to work on the relevant machine learning :P | 17:55 |
| clarkb | Alex_Gaynor: one of the first things I would like to do with elasticsearch indexed logs after the milestone, is a simple daily job that searches for ERROR level logs | 17:55 |
| jeblair | Alex_Gaynor: yep, i'm hoping i'll have time to work on it | 17:55 |
| jeblair | jog0: doing what faster? | 17:55 |
| clarkb | Alex_Gaynor: and track the diffs and try and give a picture of change | 17:55 |
| Alex_Gaynor | Awesome | 17:55 |
| clarkb | Alex_Gaynor: I think doing that will be relatively simple and have a high return on investment | 17:56 |
| jeblair | jog0: (having tests that don't fail randomly is what the gate needs most at the moment) | 17:56 |
| jog0 | jeblair: that too | 17:56 |
| Alex_Gaynor | clarkb: yup, also means we can requeue automatically if it matches a known recheck | 17:56 |
| jog0 | jeblair: woops I missed a step, so when a job is in canceled state is taht equivalent to requeue? | 17:56 |
| jeblair | Alex_Gaynor: i'm not convinced putting zuul/jenkins into a busy loop is the best use of that system | 17:56 |
| Alex_Gaynor | jeblair: I guess it depends on whether it's flaky or just broken | 17:57 |
| jeblair | Alex_Gaynor: providing feedback to developers is useful | 17:57 |
| jeblair | jog0: no, it waits for the head to finish | 17:57 |
| jeblair | jog0: doing anything else is a form of branch prediction which zuul doesn't handle | 17:57 |
| jog0 | jeblair: handle yet? | 17:58 |
| jeblair | jog0: it would be welcome but is not on my roadmap | 17:58 |
| jog0 | jeblair: I don't have numbers but I think that would help speed up gate | 17:58 |
| jeblair | jog0: keep in mind we can already fairly easily use _all_ of our computing resources testing a single branch. | 17:58 |
| jeblair | jog0: yes it would, at the cost of exponentially more computing resources. :) | 17:58 |
| *** cppcabrera_afk is now known as cppcabrera | 17:59 | |
| jeblair | (depending on how many branches you want to test) | 17:59 |
| jog0 | jeblair: why would it be more? once you cancel a job what is the issue with respinning sooner then when the failure hits head | 17:59 |
| jog0 | (I am sure I am missing things) | 17:59 |
| jeblair | jog0: perhaps you're right, that the overall number of computers may not change, though the churn would, as we would be starting and stopping far more jobs. | 18:00 |
| jog0 | jeblair: right, and I don't know if that would be an issue or not | 18:01 |
| jog0 | jeblair: anyway thanks, that ansered my question | 18:01 |
| jeblair | jog0: i think mordred was suggesting something similar a while ago. it's basically a single alternate branch: run the current scenario up to the first failure, and then an alternate scenario without the first failure. | 18:02 |
| clarkb | that would be a neat feature to have, but our gate tests should never fail right :) | 18:03 |
| jeblair | jog0: (though, i think that's a generalized algorithm, and you could do exponential branching by just repeating that) | 18:03 |
| jeblair | clarkb: yeah, i kind of think that optimizing to help get changes in despite tests that fail isn't really the point of the system | 18:04 |
| *** hashar has joined #openstack-infra | 18:04 | |
| jeblair | clarkb: if tests are failing, _that's_ the problem, not our ability to merge changes despite a non-functioning test-suite or software system | 18:04 |
| clarkb | overall in the gate queue our failure rate is actually pretty good | 18:05 |
| clarkb | ~6% over the last 12 hours | 18:05 |
| jog0 | jeblair: yeah, mordeds idea sounds right, and won't result in a significant number of extra nodes needed, just a lot more churn | 18:05 |
| clarkb | jeblair: I agree | 18:05 |
| jeblair | and we're seeing more failures now because we're trying to make a significant change in the test suite | 18:05 |
| Alex_Gaynor | clarkb: If I'm able to do math (I'm not), I'm pretty sure that the badness is exponential, so 6% is 64x worse than 1% | 18:06 |
| jeblair | that's a good thing; we need to get those bugs out | 18:06 |
| jog0 | jeblair: fair enough. althoughI would argue we will *never* have a good enough test ssuite taht has no transient failures | 18:06 |
| clarkb | Alex_Gaynor: I don't think it is exponential | 18:06 |
| jog0 | its not feasable at this scale | 18:06 |
| clarkb | Alex_Gaynor: we are O(n) worst case and O(1) best case | 18:06 |
| jeblair | jog0: yep, but we were pretty darn close the last few weeks | 18:06 |
| Alex_Gaynor | clarkb: it's possible I'm mathing wrong :) | 18:06 |
| clarkb | so each failure drives use to O(n) | 18:06 |
| jog0 | jeblair: \o/ | 18:07 |
| *** morganfainberg is now known as morganfainberg|a | 18:08 | |
| Alex_Gaynor | clarkb: O(n) time taken, but O(n^2) tests run? So not exponential, just quadratic in test runs | 18:08 |
| jog0 | btw on a unrelated tagent, our ci / review system has some interesting data about review process etc that would be cool to see some analysis of, perhaps by an academic | 18:08 |
| jog0 | being we are one of the largest open source projects and all | 18:08 |
| jeblair | jog0: yep, know any in the field? | 18:09 |
| jog0 | jeblair: look at http://status.openstack.org/zuul/ right now | 18:09 |
| Alex_Gaynor | jog0: I know an ex-academic who now does other stuff who'd probably be interested. Greg Wilson, he eited the book "Architecture of Open Source APplications" and runs a thing called Software Carpentry Workshop | 18:09 |
| jeblair | jog0: looking | 18:09 |
| jog0 | 38140,18 failed | 18:09 |
| Alex_Gaynor | I'd be more than happy to do an intro with someone, he probably either is or knows someone who'd be interested | 18:09 |
| clarkb | Alex_Gaynor: that sounds right because last change would be tested n*n times | 18:09 |
| Alex_Gaynor | His work is all about practical software engineering. | 18:09 |
| jog0 | and there are a bunch of cancels that just wait it out | 18:09 |
| jog0 | jeblair: this is the case I am refering to | 18:09 |
| jog0 | Alex_Gaynor: that would be cool, we have tons of data about tests and code review etc | 18:10 |
| Alex_Gaynor | jog0: who should I send this email to :) | 18:10 |
| jog0 | that we could do something smart with | 18:10 |
| Alex_Gaynor | clarkb: is that 6% as a %age of jobs run, or {test sets} (not sure what thye're called | 18:11 |
| jeblair | Alex_Gaynor: i'd be happy to talk to him | 18:11 |
| Alex_Gaynor | jeblair: what's your email? | 18:11 |
| clarkb | Alex_Gaynor: jobs run so not the best data for looking at gate resets | 18:11 |
| jeblair | Alex_Gaynor: jeblair@openstack.org | 18:11 |
| Alex_Gaynor | jeblair: sending something | 18:11 |
| jog0 | Alex_Gaynor: jeblair neato thanks | 18:11 |
| jeblair | jog0: yep, it was waiting on everything ahead of it to find out if 38140,18 will be the new head or not. so the branching comes in in deciding to run tests despite not knowing that. | 18:12 |
| jog0 | ahh because something ahead can fail too | 18:12 |
| * jog0 has a moment of clarity | 18:12 | |
| jeblair | jog0: yep, and if it did do that, it would then possibly repeat the process if something between that change and the head failed | 18:12 |
| jeblair | jog0: (that's hypothetical behavior, not current ^ ) | 18:13 |
| jog0 | jeblair: right hence the hella churn | 18:13 |
| jeblair | jog0: if you look at the bottom real quick | 18:13 |
| jeblair | jog0: you'll see the severed head | 18:13 |
| jog0 | 43444,5 ? | 18:14 |
| jeblair | jog0: yep | 18:14 |
| clarkb | http://tinyurl.com/pkq2grk fwiw | 18:14 |
| clarkb | despite hella churn we are still merging a ton of stuff | 18:14 |
| jeblair | jog0: so that was the head, but it failed, so it was immediately dequed and tests restarted | 18:14 |
| jeblair | jog0: it sits there and finishes its tests and then reports | 18:14 |
| jog0 | jeblair: ahh | 18:15 |
| jeblair | jog0: (zuul is probably too busy relaunching jobs to get around to reporting it right now) | 18:15 |
| jeblair | and there it goes | 18:15 |
| *** thedodd has quit IRC | 18:15 | |
| *** thedodd has joined #openstack-infra | 18:16 | |
| jeblair | clarkb: that's not terrible | 18:16 |
| jeblair | jog0: good example of 2 change queues now | 18:16 |
| *** adalbas has quit IRC | 18:16 | |
| clarkb | jeblair: I think it is incredible :) I don't know of any other project capable of merging 1k commits a week that are gated | 18:16 |
| clarkb | *individually gated | 18:16 |
| jog0 | yeah its amazing | 18:17 |
| jeblair | jog0: if you mouse-over the "Change queue: openstack-dev/openstack-nose," parts you can see the list of projects that share a queue. | 18:17 |
| jeblair | jog0: the gate-noop thing is ridiculous of course; it just hasn't been a priority to optimize for the case of a project with no tests whatsoever. | 18:17 |
| jog0 | the queue list is cool | 18:18 |
| jeblair | jog0: it combines projects that share at least one job | 18:19 |
| clarkb | <3 z/tempest | 18:19 |
| jeblair | (so it's fully automatic) | 18:19 |
| jeblair | clarkb: heh, that can probably go away now | 18:19 |
| Alex_Gaynor | jeblair: you has some (more) email | 18:19 |
| zul | mongodb should be taking another week we are just dealing with security concerns/license issues | 18:19 |
| jeblair | Alex_Gaynor: thx | 18:20 |
| jeblair | zul: just a week to deal with security concerns and license issues? :) | 18:20 |
| zul | jeblair: upstream is slow :( | 18:20 |
| clarkb | jeblair: there is a new IRC channel request on the mail list. Not sure if we want to get ahead of those or deal with them when incubation happens or? | 18:21 |
| clarkb | jeblair: you have been the active Op | 18:21 |
| clarkb | pleia2: https://review.openstack.org/#/c/43222/1 want to review that before I approve? It adds commit messages formating to cgit for things like bug links and change links | 18:26 |
| pleia2 | clarkb: sure, looking now | 18:27 |
| jeblair | i'm going to hard-restart zuul after the next reset | 18:30 |
| jeblair | i will stop it, remove the local git repos, and then start it again | 18:30 |
| clarkb | jeblair: I don't see any meetings going on right now. I am going to approve the two changes that add the meetbot to additional channels | 18:30 |
| clarkb | jeblair: ok | 18:30 |
| jeblair | clarkb: sounds good | 18:30 |
| *** arezadr has joined #openstack-infra | 18:31 | |
| *** danger_fo_away is now known as danger_fo | 18:32 | |
| *** gyee has quit IRC | 18:32 | |
| jeblair | clarkb: how many static jenkins slaves do you think we should add? | 18:33 |
| jeblair | Alex_Gaynor: ^? | 18:33 |
| Alex_Gaynor | How many are we running right now? | 18:33 |
| clarkb | right now we have 32 precise and 12 centos | 18:33 |
| clarkb | 8 precise and 4 centos? | 18:33 |
| Alex_Gaynor | sounds reasonable | 18:33 |
| pleia2 | clarkb: do you know if this regex in commit-filter.sh is lifted from somewhere? | 18:34 |
| clarkb | pleia2: I don't | 18:34 |
| mgagne | pleia2: adapted from the one used in gerrit | 18:34 |
| pleia2 | mgagne: ok cool | 18:35 |
| *** acabrera has joined #openstack-infra | 18:35 | |
| pleia2 | it looks ok to me, but regexes and eyeballs :) | 18:35 |
| clarkb | worst case the commit messages look funny right? | 18:35 |
| mgagne | pleia2: there is an example here: http://centos01.calavera.ca/cgit/nova.git/commit/?id=071d918460b6328f989ec8d8188aaa3a2c7473fa | 18:35 |
| pleia2 | yeah, I installed it on my test server and refreshed some repos, seems fine | 18:35 |
| pleia2 | oh great, mgagne already tested! | 18:36 |
| mgagne | pleia2: blueprint, Change-Id, sha1 | 18:36 |
| mgagne | pleia2: I test my stuff =) | 18:36 |
| pleia2 | awesome, +1ed | 18:36 |
| pleia2 | mgagne: thanks, this is cool :) | 18:36 |
| mgagne | =) | 18:36 |
| *** cppcabrera has quit IRC | 18:37 | |
| clarkb | the change id links will be very useful | 18:37 |
| mgagne | =) | 18:37 |
| *** nati_ueno has joined #openstack-infra | 18:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add the openstack bot to #openstack-dns https://review.openstack.org/42336 | 18:41 |
| openstackgerrit | A change was merged to openstack-infra/config: feat: add marconi channel to eavesdrop https://review.openstack.org/42956 | 18:41 |
| jeblair | clarkb, Alex_Gaynor: how about 2 centos and 8 precise? it looks like the 26 tests aren't nearly as far behind as the other jobs | 18:43 |
| clarkb | good point. That works | 18:44 |
| Alex_Gaynor | +1 from me | 18:44 |
| clarkb | mgagne: I think the recent cgit puppet work caused your change to fail to merge | 18:45 |
| *** openstack has joined #openstack-infra | 18:46 | |
| jeblair | restarting zuul | 18:46 |
| clarkb | goodbye zuul we will miss you | 18:47 |
| jeblair | it's recloning everything | 18:47 |
| clarkb | jeblair: when restarting zuul does it cancel all of its jobs now? | 18:47 |
| clarkb | jeblair: that might be a good thing to do if it doesn't so that slaves are available for use after starting again | 18:48 |
| jeblair | clarkb: nope, they'll just trail off | 18:48 |
| mtreinish | jeblair: the neutron job and the parallel ones are a lot closer than I thought they would be | 18:48 |
| jeblair | clarkb: i'm doing a hard restart anyway. | 18:48 |
| mtreinish | I expected the parallel jobs to have a much higher failure rate | 18:48 |
| *** wenlock has joined #openstack-infra | 18:49 | |
| mtreinish | jeblair: oh, nm I was ignoring the time scale I see the spike at the end | 18:49 |
| anteaya | <jeblair> jog0: good example of 2 change queues now - so I missed how we have two change queues now, but awesome | 18:49 |
| jeblair | mtreinish: keep in mind this is only for changes that zuul reports, so doesn't include a lot of runs (runs that didn't end up being used, but still) | 18:50 |
| clarkb | jeblair: it is only a problem in the hard restart because jobs will stop running in the soft restart. Would it be useful to have a hard restart script that checks gearman for jobs, kills zuul, then submits a bunch of cancelations to geraman? | 18:50 |
| jeblair | clarkb: yeah, i think so, along with something other than this script i use that leaves lots of recheck comments | 18:51 |
| jeblair | wow we have a lot of repos | 18:51 |
| jeblair | clarkb: yay, it's now taking something like 5 seconds to process a nova change | 18:52 |
| clarkb | wow that is significant | 18:53 |
| clarkb | *significantly better | 18:53 |
| * Shrews thanks jeblair for his script to leave recheck comments, stabs clarkb for making everything slow | 18:54 | |
| clarkb | jeblair: does gear(d|man) support dumping a list of running jobs? we need the job uuid to properly cancel them iirc | 18:54 |
| clarkb | jeblair: or will we need zuul to dump that info for us? | 18:55 |
| jeblair | clarkb: it doesn't; best bet is probably to add it to zuul | 18:55 |
| clarkb | or maybe have gearman-plugin kill jobs when the worker thread server connections go down | 18:56 |
| jeblair | clarkb: that would work too | 18:56 |
| *** sarob has quit IRC | 18:57 | |
| *** sarob has joined #openstack-infra | 18:57 | |
| *** melwitt has joined #openstack-infra | 18:57 | |
| *** NobodyCam has quit IRC | 18:58 | |
| jeblair | clarkb: it finished processing my rechecks; it enqueued 137 changes (in both pipelines) in about 5 mins, so it's a little over 2 seconds for each change. | 18:58 |
| *** SergeyLukjanov has quit IRC | 18:58 | |
| clarkb | much better | 18:58 |
| *** SergeyLukjanov has joined #openstack-infra | 18:59 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Backup review-dev mysql server. https://review.openstack.org/44128 | 19:09 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Backup review and wiki MySQL DBs. https://review.openstack.org/44129 | 19:09 |
| anteaya | nice | 19:09 |
| anteaya | pleia2: I read the first wiki page and skimmed the second, nice job | 19:09 |
| clarkb | jeblair: ^ two changes to backup mysql DBs. The first adds the backup stuff to only review-dev so that we can check it there and be happy with it before adding it to review.o.o and wiki.o.o | 19:09 |
| anteaya | thedodd: ping | 19:10 |
| thedodd | Hello | 19:10 |
| thedodd | what's up? | 19:11 |
| clarkb | I am going to grab lunch now. | 19:11 |
| clarkb | jeblair: let me know what you think about splitting the backups into two changes like that | 19:11 |
| thedodd | clarkb, have a nice lunch! | 19:11 |
| clarkb | jeblair: and you can look in etherpad.o.o:/var/backup/mysql_backups to see what the backups look like | 19:11 |
| anteaya | I don't see any mention of your loveliness with bug settings in commit messages in these wiki pages: https://wiki.openstack.org/wiki/Gerrit_Workflow and https://wiki.openstack.org/wiki/GitCommitMessages | 19:11 |
| thedodd | I know. Tell me about it! | 19:12 |
| anteaya | I can add it if you can help me locate what docs you do have on it, was it a post to the mailing list? | 19:12 |
| thedodd | I should file a bug report. | 19:12 |
| anteaya | ha ha ha | 19:12 |
| pleia2 | anteaya: thanks | 19:12 |
| anteaya | :D | 19:12 |
| thedodd | anteaya, might this be what you are looking for ( http://lists.openstack.org/pipermail/openstack-dev/2013-August/012945.html ) | 19:13 |
| anteaya | thanks | 19:13 |
| anteaya | it is | 19:13 |
| *** nati_ueno has quit IRC | 19:14 | |
| anteaya | let me edit those pages | 19:14 |
| *** nati_ueno has joined #openstack-infra | 19:14 | |
| thedodd | anteaya: Thanks for taking care of that. | 19:15 |
| anteaya | np | 19:15 |
| clarkb | jeblair: also I quickly looked at the gearman plugin and it almost looks like we may stop jobs when connections go away | 19:16 |
| anteaya | sorry I am wrong, it was in GitCommitMessages, I missed it the first time through | 19:17 |
| *** nati_ueno has quit IRC | 19:18 | |
| *** gyee has joined #openstack-infra | 19:20 | |
| *** alexpilotti has quit IRC | 19:24 | |
| anteaya | I sprinkled Closes-Bug in the examples | 19:25 |
| *** alexpilotti has joined #openstack-infra | 19:25 | |
| *** Bada has joined #openstack-infra | 19:26 | |
| *** adalbas has joined #openstack-infra | 19:27 | |
| *** ArxCruz has quit IRC | 19:28 | |
| thedodd | anteaya: You rock! Definitely a good idea to have the docs up-to-date for all of the new comers. | 19:34 |
| anteaya | thedodd: thanks, it was there, just needed to be included in some examples | 19:35 |
| anteaya | :D | 19:35 |
| anteaya | good habits start early | 19:35 |
| *** davidhadas has joined #openstack-infra | 19:36 | |
| *** sarob has quit IRC | 19:37 | |
| *** dkranz has quit IRC | 19:41 | |
| *** dkranz has joined #openstack-infra | 19:42 | |
| openstackgerrit | Dan Radez proposed a change to openstack-infra/config: adding puppet-pacemaker to stackforge https://review.openstack.org/44136 | 19:42 |
| *** dolphm has joined #openstack-infra | 19:44 | |
| *** dprince has quit IRC | 19:44 | |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/config: Add commit-filter for cgit https://review.openstack.org/43222 | 19:47 |
| *** Ryan_Lane has quit IRC | 19:48 | |
| *** zul has quit IRC | 19:49 | |
| openstackgerrit | will soula proposed a change to openstack-infra/jenkins-job-builder: Adding validation to the ansicolor plugin https://review.openstack.org/43926 | 19:54 |
| *** nati_ueno has joined #openstack-infra | 19:57 | |
| *** dina_belova has quit IRC | 20:00 | |
| *** dina_belova has joined #openstack-infra | 20:01 | |
| *** acabrera is now known as cppcabrera | 20:02 | |
| *** SergeyLukjanov has quit IRC | 20:05 | |
| *** dina_belova has quit IRC | 20:05 | |
| *** cppcabrera has left #openstack-infra | 20:07 | |
| *** thomasbiege has joined #openstack-infra | 20:10 | |
| *** thomasbiege has quit IRC | 20:11 | |
| *** fbo_away is now known as fbo | 20:13 | |
| *** fbo is now known as fbo_away | 20:13 | |
| jeblair | clarkb, Alex_Gaynor: i added the new nodes; but i also noted that almost as many were stuck in the scp plugin doing native io; i was unable to kill the related threads | 20:15 |
| jeblair | clarkb, Alex_Gaynor: i deleted those nodes from jenkins (twice, because that's what was required for them to really be deleted) and re-added them | 20:15 |
| jeblair | clarkb, Alex_Gaynor: the stuck jenkins threads still exist but are probably harmless | 20:15 |
| Alex_Gaynor | jeblair: are the stuck in scp things a recurring problem? | 20:16 |
| jeblair | Alex_Gaynor: not sure; it could be load related, or it could be due to the network maint rax did last night; most had been stuck for 12 hours, which is about right. some were stuck for 3.5, which is not. | 20:18 |
| jeblair | Alex_Gaynor: (by load related, i mean the load we are placing on jenkins, particularly aborting jobs, which causes interrupt exceptions in all sorts of weird places in java code) | 20:19 |
| jeblair | Alex_Gaynor: I think it's a thing to keep an eye on | 20:19 |
| *** vipul has quit IRC | 20:22 | |
| *** vipul has joined #openstack-infra | 20:23 | |
| jeblair | russellb, pabelanger, clarkb: i'm shutting down asterisk and puppet on pbx to prevent duplicate registrations while i spin up other servers | 20:27 |
| russellb | cool | 20:28 |
| clarkb | Alex_Gaynor: good to know | 20:28 |
| clarkb | er jeblair ^ | 20:28 |
| *** codenamekt_ has joined #openstack-infra | 20:29 | |
| jeblair | clarkb: and i'm destroying the test git servers to make room | 20:30 |
| clarkb | jeblair: thats fine. I don't need anything on them | 20:30 |
| *** portante|afk is now known as portante | 20:30 | |
| *** xBsd has joined #openstack-infra | 20:32 | |
| *** adalbas has quit IRC | 20:34 | |
| *** nicedice_ has quit IRC | 20:39 | |
| jeblair | err: /Stage[main]/Asterisk/Package[asterisk-sounds-extra-en-g722]/ensure: change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install asterisk-sounds-extra-en-g722' returned 1: Error: Nothing to do | 20:39 |
| jeblair | probably not a big deal; maybe it's a package pulled in by another? | 20:40 |
| clarkb | or possibly in a repo that hasn't been added yet? | 20:40 |
| clarkb | jeblair: try running it twice to see if it has a dependency that isn't strictly defined | 20:40 |
| jeblair | asterisk11 and asteriskcurrent repos have been added; it's still running | 20:41 |
| *** nicedice_ has joined #openstack-infra | 20:41 | |
| jeblair | oh, and it's trying to do a bunch of module reloads which fail too | 20:43 |
| *** ryanpetrello has joined #openstack-infra | 20:43 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Support for unofficial projects and add savanna https://review.openstack.org/40953 | 20:47 |
| *** boris-42 has quit IRC | 20:51 | |
| *** Ryan_Lane has joined #openstack-infra | 20:53 | |
| *** sandywalsh has quit IRC | 20:55 | |
| *** thomasbiege has joined #openstack-infra | 20:59 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/jenkins-job-builder: Fix typo in publishers documentation https://review.openstack.org/44149 | 20:59 |
| pabelanger | jeblair, I think that the package does not exist | 20:59 |
| *** sandywalsh has joined #openstack-infra | 20:59 | |
| pabelanger | and likely can be removed from asterisk's puppet manifest | 21:00 |
| *** lcestari has quit IRC | 21:01 | |
| *** dina_belova has joined #openstack-infra | 21:01 | |
| *** xBsd has quit IRC | 21:02 | |
| *** dkranz has quit IRC | 21:02 | |
| *** chuckieb has quit IRC | 21:03 | |
| jeblair | clarkb: did you add the new hpcloud creds to a shell file on puppetmaster? | 21:05 |
| *** rfolco has quit IRC | 21:05 | |
| *** dolphm has quit IRC | 21:05 | |
| jeblair | clarkb: for the non-jenkins account | 21:05 |
| *** dina_belova has quit IRC | 21:05 | |
| jeblair | clarkb: i'll do it | 21:07 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Fix semantic error in example config. https://review.openstack.org/44000 | 21:08 |
| clarkb | jeblair: I did not sorry | 21:08 |
| jeblair | clarkb: np | 21:08 |
| *** boris-42 has joined #openstack-infra | 21:08 | |
| clarkb | jeblair: sorting out how to stop job on gearman disconnection in the gearman plugin is fun :) | 21:09 |
| jeblair | clarkb: i'm not entirely sure i want to do that, as i think about it more | 21:10 |
| jeblair | clarkb: it's very helpful that the post/release/etc jobs continue | 21:10 |
| clarkb | good point | 21:11 |
| jeblair | clarkb: if we saved the contents of the pipelines and re-enqueued them, it might be ok | 21:11 |
| clarkb | and it seems that untangling it in the gearman plugin may be more trouble than it is worth. Particularly if we want to make it pipeline specific | 21:11 |
| *** dina_belova has joined #openstack-infra | 21:11 | |
| jeblair | clarkb: http://paste.openstack.org/show/45320/ | 21:12 |
| jeblair | clarkb: i don't quite know what to make of that | 21:12 |
| jeblair | the only centos 6.4 images are 'partner images' from 'rightimage'. | 21:13 |
| clarkb | jeblair: is that hpcloud? | 21:13 |
| jeblair | clarkb: yes | 21:13 |
| clarkb | I think we want CentOS 6.3 Server 64-bit 20130116 (b) and we can upgrade from there | 21:14 |
| clarkb | puppet should upgrade us then reboot | 21:14 |
| *** dina_belova has quit IRC | 21:15 | |
| *** boris-42 has quit IRC | 21:17 | |
| jeblair | clarkb, Alex_Gaynor: looks like it's making a dent in the check queue | 21:18 |
| clarkb | jeblair: can you look at the change series that starts with https://review.openstack.org/#/c/44128/. I think getting in that first change today will get the backup cycle going on review-dev today | 21:20 |
| *** jmeridth has joined #openstack-infra | 21:20 | |
| jeblair | clarkb: is there anything else running on the gerrit servers that's dumping the db? i know the user sync script used to | 21:21 |
| clarkb | jeblair: not that I am aware of. I can check crontabs | 21:21 |
| clarkb | jeblair: there is a cron to remove ~gerrit2/dbupdates/*.sql.gz files more than 30 days old | 21:22 |
| clarkb | jeblair: I think those files were created by the update script. That dir is empty now on review.o.o | 21:23 |
| openstackgerrit | A change was merged to openstack-infra/config: Backup review-dev mysql server. https://review.openstack.org/44128 | 21:23 |
| jeblair | clarkb: where is az1? | 21:24 |
| jeblair | clarkb: geographically | 21:25 |
| clarkb | west coast | 21:26 |
| *** nati_uen_ has joined #openstack-infra | 21:26 | |
| lifeless | how do we do dns? e.g. own dns servers, or rackspace/hp DnsAAS ? | 21:29 |
| lifeless | or <other> ? | 21:29 |
| clarkb | rackspace DNSaaS | 21:29 |
| clarkb | jeblair: mgagne's rebase of https://review.openstack.org/#/c/43222/ lgtm. I am comfortable merging that without your +2 but will hold off in case you want another look at it | 21:30 |
| *** nati_uen_ has quit IRC | 21:30 | |
| jeblair | clarkb: go for it | 21:30 |
| *** nati_uen_ has joined #openstack-infra | 21:30 | |
| *** pabelanger_ has joined #openstack-infra | 21:30 | |
| *** thomasbiege has quit IRC | 21:31 | |
| jeblair | anyone have a tool handy that takes a list of hosts and makes a sorted summary report of ping RTT for each of them? | 21:31 |
| *** pabelanger has quit IRC | 21:32 | |
| clarkb | jeblair: nmap? | 21:32 |
| *** pabelanger_ has quit IRC | 21:34 | |
| *** pabelanger has joined #openstack-infra | 21:34 | |
| clarkb | nmap --max-hostgroup 2 -sP foo.o.o bar.o.o | sort ? I didn't actually run that | 21:35 |
| lifeless | clarkb: so when you spawn a server do you give it a hostname of 'foo.openstack.org' on the nova boot line, or do you enroll it into puppet some other way? And is there automatic glue for dns updates in dnsaas ? | 21:37 |
| jeblair | az1 is closest to losangeles.voip.ms (12.5ms rtt) | 21:37 |
| *** mriedem has quit IRC | 21:38 | |
| clarkb | lifeless: in openstack-infra/config/launch/ we have scripts to launch nodes and spit out what DNS should be | 21:38 |
| clarkb | lifeless: the launch node script builds a node as foo.openstack.org and sets up the puppet certs so that it can talk to the master | 21:39 |
| lifeless | clarkb: *blink* - puppet && salt? | 21:39 |
| clarkb | lifeless: sure why not. | 21:39 |
| lifeless | clarkb: don't they fight w/ each other? | 21:39 |
| clarkb | lifeless: that script spits out what the DNS commands are but does not run them so that we can have control over DNS eg when replacing a server | 21:39 |
| clarkb | lifeless: no, we use salt as a remote command execution bus and puppet for config management | 21:40 |
| clarkb | though right now we have salt disabled until we can sort out why it keeps OOMing on us | 21:40 |
| lifeless | clarkb: ah | 21:40 |
| lifeless | clarkb: so - when it says 'member of X groups' - does it mean 'run this on a machine thats in the cluster' ? | 21:40 |
| openstackgerrit | A change was merged to openstack-infra/config: Add commit-filter for cgit https://review.openstack.org/43222 | 21:40 |
| clarkb | lifeless: if you just run dns.py it spits out DNS commands for a particular host | 21:40 |
| lifeless | clarkb: as opposed to *my local machine* ? | 21:40 |
| *** danger_fo is now known as danger_fo_away | 21:41 | |
| *** jjmb has joined #openstack-infra | 21:41 | |
| *** thomasbiege has joined #openstack-infra | 21:41 | |
| clarkb | lifeless: in the README? our puppet master is configured to let those groups do a few extra things so joining those groups allows you to manage puppet and salt certs/keys | 21:41 |
| lifeless | clarkb: in the README | 21:41 |
| lifeless | clarkb: it says 'Note that these instructions assume you're working from this | 21:41 |
| lifeless | directory on an updated local clone of the repository, and that | 21:41 |
| lifeless | ' | 21:41 |
| lifeless | 'local clone' to me means 'on my laptop' | 21:41 |
| *** jjmb has quit IRC | 21:41 | |
| clarkb | it should read local clone on the puppetmaster | 21:41 |
| lifeless | but 'sudo adduser YOURUSER puppet' | 21:42 |
| lifeless | clarkb: patch incoming :P | 21:42 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Correct launch README. https://review.openstack.org/44154 | 21:43 |
| jeblair | russellb, pabelanger, clarkb: okay, i have 12 pbx servers: http://paste.openstack.org/show/45322/ | 21:44 |
| lifeless | clarkb: what size flavor is needed for the puppetmaster? (I presume the puppetmaster is the first thing to bring up, and that heira is on it?) | 21:44 |
| jeblair | lifeless: it is an overloaded 2g vm | 21:45 |
| lifeless | jeblair: thanks. | 21:45 |
| jeblair | (though ours has something like 100 servers) | 21:45 |
| clarkb | lifeless: it really depends on the number of servers and whether or not salt is leaking | 21:46 |
| lifeless | clarkb: sure | 21:46 |
| clarkb | lifeless: but 2g is probably plenty for starting | 21:46 |
| lifeless | clarkb: but 'USE THE BIGGEST' isn't the answer is what I needed ;) | 21:46 |
| jeblair | russellb, pabelanger, clarkb: i think when we test them out, we'll need to probably group tests by data center and change the did pop in the voip.ms web ui when switching data centers | 21:47 |
| *** pentameter has quit IRC | 21:47 | |
| lifeless | clarkb: jeblair: so you have the rackspace cli clients installed on the puppetmaster I presume ? | 21:49 |
| clarkb | lifeless: correct | 21:49 |
| clarkb | lifeless: I think it is vanilla novaclient + the rackspace auth plugin | 21:49 |
| jeblair | lifeless: we have the _openstack_ clients installed on the puppetmaster | 21:49 |
| clarkb | lifeless: and it works with hpcloud too | 21:49 |
| jeblair | and the rax auth plugin is not required | 21:49 |
| lifeless | jeblair: for dns? | 21:50 |
| jeblair | lifeless: (the exception is the proprietary rackspace dns; but not much we can do about that atm) | 21:50 |
| lifeless | hah! :) | 21:50 |
| lifeless | jeblair: I was reading launch/ :) | 21:50 |
| jeblair | s/proprietary/not-openstack/ | 21:50 |
| jeblair | but nova and cinder are straight-up. | 21:51 |
| lifeless | jeblair: so, this is my top level understanding of spinning up this infra: | 21:51 |
| lifeless | - fork the config repo | 21:51 |
| clarkb | jog0: I have updated reviews on your large ops test changes. Let me know if you have questions | 21:51 |
| lifeless | - make a new foo_project directory, which gets altered 'run server' definitions | 21:51 |
| lifeless | - replace site.pp with something appropriate referencing foo_project only | 21:52 |
| jog0 | clarkb: just saw will have to fix tomorrow, thanks | 21:52 |
| clarkb | jog0: but I would like to try and merge things in a way that allows us to test the d-g change before merging that particular change | 21:52 |
| jog0 | clarkb: I like that idea too | 21:52 |
| lifeless | jeblair: anything obviously wrong with this? | 21:52 |
| jeblair | lifeless: not that i can see | 21:52 |
| lifeless | jeblair: [I appreciate the 'and give feedback on the split and how well it works', I'm just setting up my torture plan atm :)) | 21:52 |
| anteaya | atm the salt guys feel that the zmq version 2.something is the cause of the memory leaks, after ff they want to come back and help us, including a zmq upgrade to 3.+ | 21:54 |
| clarkb | cool | 21:54 |
| pabelanger | jeblair, 12 server, jebus | 21:55 |
| anteaya | I think we are on zmq 2.2.0 and they suggest 3.2.x | 21:55 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Change use of error numbers to errno https://review.openstack.org/42356 | 21:56 |
| pabelanger | jeblair, you're not planning on load balancing across them are you? | 21:56 |
| clarkb | jeblair: ^ I just updated mordred's change because it is a good one and was simple to fix the merge conflict | 21:56 |
| pabelanger | or are they individual silos | 21:56 |
| lifeless | clarkb: jeblair: so I am still unclear; how do you set the correct hostname for puppet in new servers? | 21:57 |
| jeblair | pabelanger: no, they're so that we can test them one at a time and see if the choppyness is affected by vm resources or data centers | 21:57 |
| lifeless | just in the nova boot line ? | 21:57 |
| pabelanger | jeblair, Ah, cool | 21:57 |
| clarkb | lifeless: no it is more complicated than that | 21:58 |
| lifeless | clarkb: interested people want to know. | 21:58 |
| lifeless | clarkb: e.g. me. | 21:58 |
| clarkb | lifeless: puppet uses client SSL certs to identify with the master. So you set the certname in puppet to tell it who to identify as. We then copy the cert for that certname over to the host in the launch script | 21:59 |
| lifeless | oh; so all the stuff in http://ci.openstack.org/puppet.html#at-a-glance is manual ? | 21:59 |
| clarkb | lifeless: then when puppet runs it uses the certname and cert and identifies itself as whatever that cert is | 21:59 |
| jeblair | zuul launched 720 jobs in an hour earlier | 21:59 |
| clarkb | lifeless: yes | 22:00 |
| * lifeless peels eyeballs | 22:00 | |
| *** dhellmann is now known as dhellmann_ | 22:00 | |
| lifeless | clarkb: is hiera installed on the puppetmaster? | 22:00 |
| clarkb | lifeless: it is | 22:00 |
| *** fifieldt_ has joined #openstack-infra | 22:00 | |
| lifeless | ok | 22:00 |
| lifeless | so I think my first task is getting a puppet master up | 22:00 |
| jeblair | lifeless: i'm pretty sure the answer to your question is 'no' | 22:00 |
| lifeless | jeblair: I have asked several questions; which one is answered via 'no' ? | 22:01 |
| jeblair | not "all of the stuff in http://ci.openstack.org/puppet.html#at-a-glance is manual" | 22:01 |
| *** thomasm has quit IRC | 22:02 | |
| lifeless | jeblair: ok. I am now confused; clarkb said yes! Discuss :) | 22:02 |
| *** thomasm has joined #openstack-infra | 22:02 | |
| jeblair | lifeless: we may need more specific questions | 22:03 |
| *** thomasm has quit IRC | 22:03 | |
| *** markmcclain has joined #openstack-infra | 22:03 | |
| *** thomasbiege has quit IRC | 22:03 | |
| jeblair | lifeless: all of the steps in that doc can be performed manually, and to follow the directions in that doc, you will be performing them manually | 22:03 |
| lifeless | jeblair: to enroll a new server, there is a process documented on the page I referenced; Are there steps from that process that should be -not invoked by hand- ? | 22:04 |
| lifeless | jeblair: or, is that doc not what you actually do? | 22:04 |
| clarkb | by yes I meant they are manual as described there but our launch script does that magic behind the scenes | 22:04 |
| *** nati_uen_ has quit IRC | 22:04 | |
| clarkb | so we don't have to run puppet, sign the cert then run puppet again. Instead the launch script automatically does that behind the scenes whereas the steps are manual when using the doc | 22:05 |
| lifeless | there is no reference to the launch script on that page; I haven't encountered it in the docs so far. Could I get a pointer to it ? | 22:05 |
| lifeless | Pointer to where it is documented, or failing that, to the script itself | 22:05 |
| jeblair | lifeless: you were just reading it a minute ago | 22:05 |
| clarkb | lifeless: openstack-infra/config/launch/README | 22:05 |
| lifeless | jeblair: ah, same dir - ok. | 22:05 |
| lifeless | I got shiny'd at the dns thing | 22:05 |
| lifeless | sorry ;) | 22:06 |
| *** nati_uen_ has joined #openstack-infra | 22:06 | |
| *** nati_uen_ has quit IRC | 22:06 | |
| lifeless | it is not documented in the docs; would the sysadmin page be the right place to add it ? | 22:07 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Change use of error numbers to errno https://review.openstack.org/42356 | 22:07 |
| jeblair | lifeless: the launch script is a bit of an implementation detail; it's how we spin up servers in the openstack accounts using openstack credentials; it's not really necessary, and it's overkill for people who want to contribute to openstack-ci | 22:07 |
| clarkb | mgagne: https://git.openstack.org/cgit/openstack-infra/config/commit/?id=6bd28882555adaa30981a4a5b8d56b5cd1b220ce so shiny | 22:07 |
| *** nati_uen_ has joined #openstack-infra | 22:07 | |
| lifeless | jeblair: right, but it's something you have to communicate to everyone that graduates to root access; or anyone setting up a clone of the infra | 22:08 |
| mgagne | clarkb: the commit message itself isn't a good example of what can be done by this change :P | 22:08 |
| lifeless | jeblair: So I'm assuming the README is sufficient, but I want to add a pointer to that somewhere. | 22:08 |
| jeblair | lifeless: yeah, the sysadmin page would probably be the right place; but we may need to start splitting it up into contributing sysadmin vs root sysadmin | 22:09 |
| jeblair | lifeless: the backup docs are similarly root-only | 22:09 |
| jeblair | and on that page | 22:09 |
| mgagne | clarkb: I like this one better https://git.openstack.org/cgit/openstack/nova/commit/?id=071d918460b6328f989ec8d8188aaa3a2c7473fa =) | 22:10 |
| clarkb | mgagne: no, but it does have a link back to the change via the change Id which is what I really wanted :) | 22:10 |
| mgagne | clarkb: oh cool =) | 22:10 |
| clarkb | having git notes on cgit would be useful too | 22:11 |
| mgagne | clarkb: time to work on syntax highlight =) | 22:11 |
| clarkb | mgagne: pleia2 any idea if that is possible? | 22:11 |
| mgagne | clarkb: and notes | 22:11 |
| mgagne | clarkb: should already be possible I think, I saw the option somewhere | 22:11 |
| clarkb | great, I think that is handy since gerrit puts review info in the notes | 22:12 |
| *** dina_belova has joined #openstack-infra | 22:12 | |
| *** ryanpetrello has quit IRC | 22:13 | |
| pleia2 | yeah, syntax highlight is an option, I forget whether this version is one that requires us to supply our own syntax highlighting engine or if it ships with one (much woe on the mailing list when they stopped shipping one and everyone lost highlighting) | 22:13 |
| * pleia2 checks | 22:13 | |
| mgagne | pleia2: it uses highlight by default and it sucks IMO | 22:13 |
| pleia2 | mgagne: hah, that's probably why they stopped shipping it by default | 22:14 |
| mgagne | pleia2: would like to use pygments instead and add additional languages | 22:14 |
| *** pcrews has quit IRC | 22:14 | |
| *** dkliban has quit IRC | 22:14 | |
| pleia2 | ok, so it's version 0.9.2 where they changed this looks like http://lists.zx2c4.com/pipermail/cgit/2013-August/thread.html#1451 | 22:15 |
| pleia2 | and we're using 0.9.2 | 22:16 |
| *** dina_belova has quit IRC | 22:16 | |
| mgagne | pleia2: with pygments, you have to provide your own stylesheet which can be generated by pygments itself | 22:16 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Make root-only information clear in sysadmin.rst. https://review.openstack.org/44162 | 22:16 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add a pointer to launch/README for root operators. https://review.openstack.org/44163 | 22:17 |
| pleia2 | mgagne: seems reasonable to go that route | 22:18 |
| clarkb | I have found the release notes for 0.9 saying that notes hsould be displayed if present | 22:18 |
| clarkb | but ours doesn't seem to do that | 22:18 |
| mgagne | pleia2: I alright hacked gitweb to use pygments, shouldn't be hard to do the same with cgit | 22:18 |
| pleia2 | mgagne: cool | 22:18 |
| mgagne | clarkb: checking, thinking about notes not being replicated to cgit | 22:18 |
| clarkb | mgagne: they are replicated to github and we use the same replication rules I think | 22:19 |
| clarkb | I am digging more | 22:19 |
| jeblair | i'll check the local repos and see if they are there | 22:19 |
| clarkb | compare http://git.openstack.org/cgit/openstack/nova/commit/?id=4f702880d56fcb4d0e54506c193210b589ce00c9 and https://github.com/openstack/nova/commit/4f702880d56fcb4d0e54506c193210b589ce00c9 | 22:19 |
| mgagne | unless I missed something, I can't consult git notes after cloning | 22:23 |
| mgagne | I used git clone --mirror | 22:23 |
| *** hashar has left #openstack-infra | 22:23 | |
| svarnau | lifeless: you are asking about the areas that I've just been wading thru. You are setting up your own infra to host a non-openstack project? | 22:24 |
| jeblair | mgagne: git fetch gerrit +refs/notes/*:refs/notes/* | 22:25 |
| mgagne | jeblair: for test purposes? | 22:25 |
| lifeless | svarnau: yes | 22:26 |
| jeblair | mgagne: yeah; not sure why it didn't show up in the mirror | 22:26 |
| lifeless | svarnau: I'm going to give a talk on it in 1.5 weeks too :) | 22:26 |
| svarnau | lifeless: yea! sign me up | 22:26 |
| *** changbl has quit IRC | 22:26 | |
| *** codenamekt_ has quit IRC | 22:28 | |
| *** markmcclain has quit IRC | 22:29 | |
| *** markmcclain has joined #openstack-infra | 22:30 | |
| svarnau | lifeless: I also started with setting up a puppet master node, and going from there. I have not gotten much farther yet. | 22:30 |
| *** jamiem_ has joined #openstack-infra | 22:30 | |
| *** jamiem has quit IRC | 22:31 | |
| *** jamiem_ is now known as jamiem | 22:31 | |
| *** dkranz has joined #openstack-infra | 22:31 | |
| *** woodspa has quit IRC | 22:31 | |
| mgagne | jeblair: not sure how I'm supported to consult them, I only see one note: refs/notes/review | 22:32 |
| jeblair | mgagne: i'm puzzled as well | 22:32 |
| *** ftcjeff__ has joined #openstack-infra | 22:32 | |
| mgagne | jeblair: see that github shows "review" above the note | 22:32 |
| *** jhesketh has joined #openstack-infra | 22:32 | |
| *** ftcjeff_ has quit IRC | 22:33 | |
| lifeless | jeblair: I'd like to include some meta docs: 'if you are setting up a clone of the infra, you need to replace <x> with <y> all over, I'm thinking a new rst file for 'running your own instance' ? | 22:33 |
| jeblair | lifeless: sounds good | 22:34 |
| lifeless | jeblair: care to pick a filename ? | 22:34 |
| lifeless | running-your-own.rst? | 22:34 |
| jeblair | lifeless: wfm | 22:35 |
| mgagne | git config --add remote.gerrit.fetch +refs/notes/*:refs/notes/* | 22:36 |
| mgagne | git config notes.displayRef refs/notes/* | 22:36 |
| mgagne | git remote update | 22:36 |
| mgagne | And now git show shows the notes | 22:36 |
| lifeless | jeblair: it occurs to me it would be nice if the jenkins runs for openstack-ci spat out a built copy of docs for eyeballing | 22:37 |
| lifeless | jeblair: perhaps they do already, but if not, I'm happy to file a bug | 22:37 |
| jeblair | mgagne: git log --show-notes=review | 22:37 |
| jeblair | mgagne, clarkb: that works in the local repos on the git servers | 22:38 |
| jeblair | mgagne, clarkb: so, i have no idea where they're actually stored, but they are there. :) | 22:38 |
| mgagne | jeblair: I think they are in refs/notes/review | 22:38 |
| *** dims has quit IRC | 22:38 | |
| mgagne | jeblair: I don't know if we need additional config for cgit to handle them | 22:38 |
| jeblair | lifeless: follow the 'gate-ci-docs' link in the report from jenkins | 22:39 |
| lifeless | jeblair: ahh! thanks ;) | 22:39 |
| jeblair | lifeless: (that used to be more obvious when the url was visible, we may need to make that more discoverable now that it's a text link) | 22:39 |
| jeblair | mgagne: yeah, i'm just puzzled by why that is a single file with a single sha in it, but i don't have to understand that now; the main thing is that the data seem to be on the cgit servers in the usual place. | 22:40 |
| mgagne | jeblair: could be some form of "branch" and not the note itself | 22:40 |
| *** jhesketh__ has joined #openstack-infra | 22:40 | |
| jeblair | mgagne: yeah, it seems to be the latest, so maybe it acts like a branch head | 22:41 |
| *** jhesketh has quit IRC | 22:41 | |
| *** jhesketh has joined #openstack-infra | 22:42 | |
| *** olaph has joined #openstack-infra | 22:42 | |
| *** ftcjeff has quit IRC | 22:43 | |
| jeblair | mgagne: is there an option where we have to indicate to cgit that it should examine the 'review' notes namespace? | 22:43 |
| jeblair | mgagne, clarkb: maybe it only looks at refs/notes/commit by default | 22:43 |
| mgagne | jeblair: hold on ^^' | 22:43 |
| *** thedodd has quit IRC | 22:44 | |
| mgagne | jeblair: was thinking: is the feature still there? | 22:44 |
| jeblair | mgagne: don't understand? | 22:45 |
| clarkb | mgagne: oh maybe they removed it? | 22:46 |
| jeblair | oh, cgit? | 22:46 |
| clarkb | jeblair: feature was added to 0.9 but we run 0.9.2 | 22:46 |
| clarkb | ya | 22:46 |
| jeblair | wow, do they add and remove every feature? :) | 22:46 |
| mgagne | jeblair: code changed meanwhile but | 22:46 |
| mgagne | jeblair: GIT_NOTES_DISPLAY_REF_ENVIRONMENT | 22:46 |
| pleia2 | ooh, side-by-side-diffs might be nice to use (browsing the cgit man page) | 22:46 |
| mgagne | GIT_NOTES_DISPLAY_REF in fact | 22:47 |
| clarkb | pleia2: I think you can select side by side with a drop down in the top right of a commit | 22:47 |
| clarkb | pleia2: http://git.openstack.org/cgit/openstack/nova/commit/?id=4f702880d56fcb4d0e54506c193210b589ce00c9&context=3&ignorews=0&ss=1 | 22:47 |
| mgagne | jeblair: https://www.kernel.org/pub/software/scm/git/docs/git-notes.html notes.displayRef | 22:47 |
| pleia2 | clarkb: yeah, ssdiff | 22:48 |
| mgagne | jeblair: could it be like you said, this variable needs to be overriden to refs/notes/review ? | 22:48 |
| mgagne | jeblair: or GIT_NOTES_REF | 22:48 |
| pleia2 | side-by-side-diffs=1 makes it default view | 22:48 |
| *** rcleere has quit IRC | 22:50 | |
| *** rcleere has joined #openstack-infra | 22:50 | |
| clarkb | mgagne: I think we want DISPLAY_REF as it is in addition to the default | 22:50 |
| mgagne | jeblair: http://centos01.calavera.ca/cgit/nova.git/commit/?id=f953e2062207c0b39e48d8d297eebc48f8d1b13d | 22:51 |
| jeblair | mgagne: lovely! | 22:51 |
| mgagne | jeblair: git config notes.displayRef refs/notes/* | 22:51 |
| clarkb | dangit I think I left the haproxy rsyslog stuff in the cgit module | 22:51 |
| *** olaph has quit IRC | 22:51 | |
| jeblair | mgagne: so that's in the git repos | 22:51 |
| clarkb | jeblair: mgagne: Can we just set the env variable in the apache vhost? | 22:52 |
| mgagne | clarkb: I can test | 22:52 |
| jeblair | other choices would be: get jeepyb to do it when setting up the repo, or convince puppet to do it for all repos it finds... | 22:52 |
| jeblair | i think i like them in that order -- env variable, jeepyb, and puppet just sounds weird i don't like that. | 22:53 |
| *** dims has joined #openstack-infra | 22:53 | |
| jeblair | pleia2: side-by-side by default might be a good idea. | 22:54 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Move location of haproxy rsyslog conf file. https://review.openstack.org/44173 | 22:54 |
| clarkb | jeblair: pleia2 ^ that cleans up the haproxy rsyslog | 22:54 |
| jeblair | i have to run for a bit | 22:55 |
| pleia2 | clarkb: good catch | 22:55 |
| mgagne | clarkb: yes | 22:56 |
| mgagne | clarkb: SetEnv GIT_NOTES_DISPLAY_REF refs/notes/* | 22:56 |
| clarkb | mgagne: I think that is the best way to deal with it then | 22:56 |
| mgagne | clarkb: please look that way | 22:56 |
| mgagne | clarkb: *type* *type* | 22:56 |
| *** Bada has quit IRC | 22:57 | |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/config: Configure cgit to show git notes https://review.openstack.org/44174 | 22:59 |
| *** pcrews has joined #openstack-infra | 23:00 | |
| clarkb | magic | 23:00 |
| clarkb | straight to the gate queue with that one | 23:00 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Make cgit side-by-side diffs the default https://review.openstack.org/44175 | 23:01 |
| *** wenlock has quit IRC | 23:01 | |
| *** nati_ue__ has joined #openstack-infra | 23:01 | |
| *** markmcclain1 has joined #openstack-infra | 23:01 | |
| *** fifieldt_ is now known as fifieldt | 23:03 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Start documenting running your own CI infra. https://review.openstack.org/44177 | 23:03 |
| *** markmcclain has quit IRC | 23:03 | |
| *** nati_uen_ has quit IRC | 23:05 | |
| *** olaph has joined #openstack-infra | 23:05 | |
| *** morganfainberg|a is now known as morganfainberg | 23:06 | |
| openstackgerrit | A change was merged to openstack-infra/config: Configure cgit to show git notes https://review.openstack.org/44174 | 23:08 |
| openstackgerrit | A change was merged to openstack-infra/config: Make cgit side-by-side diffs the default https://review.openstack.org/44175 | 23:09 |
| *** datsun180b has quit IRC | 23:12 | |
| mgagne | *refresh* *refresh* | 23:12 |
| *** dina_belova has joined #openstack-infra | 23:12 | |
| clarkb | mgagne: puppet will update in ~15 minute | 23:12 |
| pleia2 | hehe | 23:12 |
| clarkb | jeblair: looking at the zuul status page it almost appears that gearman isn't running jobs on jenkins in the order they are queued | 23:13 |
| clarkb | jeblair: https://jenkins02.openstack.org/job/gate-grenade-devstack-vm/5586/ started well before https://jenkins01.openstack.org/job/gate-grenade-devstack-vm/5409/ but the change for the second job was in the queue first | 23:13 |
| clarkb | jeblair: is that possibly due to async job submissions? | 23:14 |
| *** sarob has joined #openstack-infra | 23:14 | |
| *** dina_belova has quit IRC | 23:16 | |
| *** nati_ue__ has quit IRC | 23:17 | |
| *** cyeoh is now known as cyeoh_ | 23:19 | |
| *** jaypipes has quit IRC | 23:21 | |
| *** marun has quit IRC | 23:22 | |
| *** dims has quit IRC | 23:22 | |
| *** dims has joined #openstack-infra | 23:24 | |
| *** jergerber has quit IRC | 23:24 | |
| *** xBsd has joined #openstack-infra | 23:26 | |
| *** olaph has quit IRC | 23:28 | |
| *** dims has quit IRC | 23:28 | |
| *** UtahDave has quit IRC | 23:28 | |
| *** morganfainberg is now known as morganfainberg|a | 23:29 | |
| pleia2 | mgagne: looks good :) | 23:29 |
| *** dims has joined #openstack-infra | 23:30 | |
| mgagne | pleia2: some entries seem to be cached (without the notes) | 23:30 |
| mgagne | pleia2: ssdiff =) | 23:30 |
| clarkb | yeah the cache for commit objects is longer because they typically don't change :) | 23:31 |
| mgagne | clarkb: now, how do I make those links clickable in notes =) | 23:31 |
| clarkb | mgagne: I approved https://review.openstack.org/#/c/43924/ which may potentially cause merge conflicts for other JJB changes | 23:31 |
| clarkb | I figured just getting it in would be useful so hopefully not too painful | 23:32 |
| clarkb | mgagne: what do youthink about also trying to start requiring unittests for each new publisher? | 23:32 |
| clarkb | s/publisher/thing/ | 23:32 |
| mgagne | clarkb: I have yet to see how it works | 23:32 |
| mgagne | clarkb: I'm not against it | 23:32 |
| clarkb | mgagne: the tests are pretty simple. You drop an xml and a yaml file for the thing you are testing. JJB compiles yaml to xml and compares | 23:33 |
| *** zul has joined #openstack-infra | 23:33 | |
| mgagne | clarkb: we can inform others about the need to sort it before merging | 23:33 |
| clarkb | mostly useful for making sure that modifying JJB internals does not break any of the entrypoints | 23:33 |
| mgagne | clarkb: looks simple. Have yet to try it myself and see how it can handle multiple scenarios/use cases for the same builder/publisher | 23:33 |
| clarkb | mgagne: I think you can just add separate pairs of yaml+xml files for each scenario | 23:34 |
| clarkb | it is also handy for developing new things. You can mock up the yaml and copy the xml from jenkins and bang on it until the test passes | 23:34 |
| mgagne | clarkb: I wonder about the scalability of such setup ;) | 23:35 |
| mgagne | clarkb: true | 23:35 |
| clarkb | yeah scalability may be a problem. It isn't quite unitesting | 23:35 |
| clarkb | but I imagine the first scale issue will be one of organization for humans and not anything the test suite has trouble with | 23:36 |
| mgagne | clarkb: yes, that was my real concern | 23:36 |
| *** x05h has joined #openstack-infra | 23:37 | |
| clarkb | mgagne: the sorting of the entrpoints has run into a conflict itself :) | 23:38 |
| mgagne | clarkb: fun stuff, I can check | 23:38 |
| *** changbl has joined #openstack-infra | 23:38 | |
| *** markmcclain1 has quit IRC | 23:39 | |
| *** rcleere has quit IRC | 23:39 | |
| *** prad_ has joined #openstack-infra | 23:39 | |
| *** x05h has quit IRC | 23:40 | |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/jenkins-job-builder: Sort entry_points in setup.py for maintainability https://review.openstack.org/43924 | 23:42 |
| *** morganfainberg|a is now known as morganfainberg | 23:42 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix typo in publishers documentation https://review.openstack.org/44149 | 23:42 |
| mgagne | clarkb: fixed. hurry before someone merges something! ;) | 23:43 |
| clarkb | mgagne: if the xml diff change comes back clean I will approve | 23:45 |
| clarkb | (I am paranoid and that is a non voting job not run in the gate so it requires a manual check) | 23:46 |
| *** michchap has joined #openstack-infra | 23:46 | |
| mgagne | clarkb: damn, you can't be fooled easily ;) | 23:46 |
| notmyname | maybe I missed the answer earlier. have there been any issues that you know about (or have resolved) between setuptools and distribute? | 23:48 |
| *** michchap has quit IRC | 23:48 | |
| jeblair | clarkb: gate jobs for the same build set should run in approximately the order submitted | 23:48 |
| *** michchap has joined #openstack-infra | 23:49 | |
| clarkb | notmyname: yes there are issues around the merger | 23:49 |
| jeblair | clarkb: if those ran in a significantly unexpected order, i don't know the reason. we might be able to surmise something with about 30 minutes worth of log spelunking | 23:49 |
| clarkb | notmyname: typically manifesting as pkg_resources or distribute getting wedged or removed | 23:49 |
| notmyname | clarkb: this is our current issue https://bugs.launchpad.net/swift/+bug/1217288 | 23:50 |
| uvirtbot | Launchpad bug 1217288 in swift "Swift installation crashes" [High,Confirmed] | 23:50 |
| clarkb | jeblair: probably not super important right now but worth looking at if it becomes a problem | 23:50 |
| clarkb | notmyname: I think that may be caused by old setuptools/distribute | 23:53 |
| clarkb | notmyname: pip install -U setuptools should fix that or us a newer virtualenv/tox | 23:54 |
| clarkb | https://bugs.launchpad.net/swift/+bug/1217288/comments/8 in particular looks like it needs newer tools | 23:55 |
| uvirtbot | Launchpad bug 1217288 in swift "Swift installation crashes" [High,Confirmed] | 23:55 |
| *** alexpilotti has quit IRC | 23:56 | |
| clarkb | clayg seems to be on top of it | 23:56 |
| *** rnirmal has quit IRC | 23:57 | |
| zul | is there something wrong with the novaclient gate, it seems to be stuck https://review.openstack.org/#/c/42789/ | 23:57 |
| notmyname | clarkb: ya, but he doesn't have a resolution | 23:57 |
| clarkb | zul: it is still queued up | 23:58 |
| zul | clarkb: k | 23:58 |
| clarkb | notmyname: it almost looks like clayg is indicating that very new setuptools is broken :/ I have pinged clayg and am trying to see if that is the case | 23:58 |
| notmyname | clarkb: ya, he just left for the day. (he sits about 5 feet from me) | 23:59 |
| clarkb | I see | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!