| openstackgerrit | A change was merged to openstack-infra/config: Gate Gnocchi against MySQL too https://review.openstack.org/97820 | 00:01 |
|---|---|---|
| clarkb | jesusaurus: http://logs.openstack.org/19/96319/3/check/gate-config-puppet-apply-precise/b81795f/console.html.gz#_2014-06-03_06_35_47_752 is that a known thing? | 00:04 |
| jesusaurus | clarkb: for https://review.openstack.org/#/c/98561/ how strongly do you feel about using a single script rather than copying the prepare_node* architecture for the ready_* scripts? | 00:04 |
| clarkb | jesusaurus: pretty strongly... I think that if we end up needing additional scripts we can split them at that point | 00:05 |
| clarkb | jesusaurus: but starting this way will only lead to drift that we don't intend | 00:05 |
| *** alexpilotti has quit IRC | 00:05 | |
| jesusaurus | clarkb: yeah, that error you linked to will be resolved by https://review.openstack.org/#/c/97659/ | 00:05 |
| clarkb | jesusaurus: cool, I have been trying to get those through so that we can eventually gate on that test | 00:05 |
| clarkb | jesusaurus: I just approved the selinux cleanup | 00:06 |
| clarkb | and approved the apache name change earlier | 00:06 |
| jesusaurus | clarkb: sweet | 00:06 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Allow modules to be installed from source https://review.openstack.org/96336 | 00:07 |
| jesusaurus | clarkb: and okay, i'll change 98561 to be a single ready script | 00:07 |
| morganfainberg | sdague, so we don't avoid logging the token, but i'll get that fix in for the next KSC release. | 00:07 |
| *** gokrokve has quit IRC | 00:08 | |
| *** bknudson has joined #openstack-infra | 00:08 | |
| jesusaurus | bodepd: how familiar are you with r10k? how much effort would it be to turn install_modules.sh into a wrapper for r10k? | 00:09 |
| bodepd | jesusaurus: I'm not sure how much sense it makes to use r10k if you are not using environments | 00:10 |
| bodepd | jesusaurus: it's pretty opinionated around an environmnets based workflow | 00:10 |
| bodepd | jesusaurus: the problem with environments is they kind of don't work | 00:10 |
| bodepd | jesusaurus: that being said, it should be able tp replicate what you guys are doing | 00:11 |
| bodepd | jesusaurus: this might be a better conversation for #puppet the author is usually hanging out in there | 00:13 |
| jesusaurus | oh, i didnt realize r10k was strongly opinionated about environments | 00:13 |
| *** reed_ has quit IRC | 00:13 | |
| *** UtahDave has quit IRC | 00:13 | |
| bodepd | jesusaurus: yeah, it assumes it is populating source into dynamic environments | 00:13 |
| bodepd | jesusaurus: it has some kind of librarian-puppet-ish support | 00:14 |
| bodepd | jesusaurus: but I found it to work in surprising ways and not be that well documented | 00:14 |
| *** mmaglana has joined #openstack-infra | 00:14 | |
| *** lakshmiS has quit IRC | 00:14 | |
| bodepd | jesusaurus: that being said, I should probably be using it so that I don't have to maintain my own tool for the task (like what you are obviously thinking) | 00:14 |
| AaronGr | jesusaurus: i just started using it personally, against a single environment, works well, and easily | 00:14 |
| jesusaurus | bodepd: good to hear, im still at the 'this looks useful' stage of learning about it | 00:15 |
| clarkb | mordred_phone: mordred: you should read https://etherpad.openstack.org/p/nodepool-dib-integration when you have a free moment | 00:17 |
| bodepd | clarkb: that's funny, I've been thinking about the same thing today | 00:18 |
| bodepd | clarkb: but I'm using packer | 00:18 |
| clarkb | bodepd: packer is in and odd place because it doesn't really add anything | 00:18 |
| clarkb | bodepd: it can do the proper VM image builds but those have all the same issues | 00:19 |
| bodepd | clarkb: what do you mean? | 00:19 |
| clarkb | at least for our need | 00:19 |
| bodepd | clarkb: it calls out to a list of shell scripts | 00:19 |
| clarkb | bodepd: as far as the goals go we do something today that is very packer like and it doesn't quite meet what we want | 00:19 |
| clarkb | bodepd: right its basically what we do today | 00:19 |
| clarkb | and we would like to be better | 00:19 |
| bodepd | clarkb: what's an example of 'better'? | 00:20 |
| clarkb | bodepd: the goals on that ethperad | 00:20 |
| bodepd | coo | 00:20 |
| clarkb | the first paragraph | 00:20 |
| clarkb | bodepd: the first item is the tricky one | 00:21 |
| clarkb | bodepd: we basically require people today to hae an hpcloud or rax account if they want to reproduce exactly on their own | 00:21 |
| clarkb | bodepd: packer does have the various backends going for it, but in practice I think your scripts will only work on one | 00:24 |
| clarkb | at least that has been my experience trying to make this stuff work in a chroot | 00:24 |
| openstackgerrit | A change was merged to openstack-infra/config: Only include selinux bits on rhel-based systems https://review.openstack.org/96319 | 00:24 |
| bodepd | clarkb: why do you need different images per provider? | 00:26 |
| bodepd | clarkb: couldn't you just generate a single img file and use it anywhere? | 00:26 |
| clarkb | bodepd: in theory yes, until recently we could not provide a base image and had to rely on what the provider gives you | 00:26 |
| clarkb | bodepd: that is still true for most customers of hpcloud aiui | 00:26 |
| clarkb | so to get a 1:1 you have to start with that | 00:27 |
| *** sarob has joined #openstack-infra | 00:27 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Fix nodepool instance hostnames https://review.openstack.org/98561 | 00:27 |
| bodepd | my hope is that I can reliably build images for vbox and openstack using the same script sets and a single defintion (which should be possible using a single image specification) | 00:28 |
| *** matsuhashi has joined #openstack-infra | 00:28 | |
| clarkb | bodepd: assuming you control the base image for both yes | 00:28 |
| *** Sukhdev has quit IRC | 00:29 | |
| bodepd | yeah, that's my plan, but I wouldn't start services. I guess you need that b/c the services seed data? | 00:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Increase timeout for chef-related jobs, 30min is not enough time https://review.openstack.org/98084 | 00:30 |
| clarkb | bodepd: ya we set up our mysql and postgres DBs for the test suites and that requires modifying user permissions and creating a database | 00:30 |
| fungi | i take a break to pack things and come back to so many pings | 00:30 |
| clarkb | fungi: well you disappeared right as I was trying to do the infra-specs :) | 00:30 |
| mattoliverau | anteaya: thanks for the meeting link, ACL stuff helped :) | 00:31 |
| clarkb | fungi: which is done, please review and I think it is ready to shove through | 00:31 |
| *** weshay has quit IRC | 00:31 | |
| clarkb | fungi: note that I did not do your suggestion because sphinx complains when there is nothing in the implemented/* dir | 00:31 |
| *** nati_ueno has joined #openstack-infra | 00:31 | |
| bodepd | clarkb: http://www.packer.io/docs/provisioners/puppet-masterless.html | 00:31 |
| clarkb | bodepd: yup we do that today | 00:32 |
| bodepd | clarkb: ah, but Puppet itself may dedend on services | 00:32 |
| clarkb | bodepd: we do image builds today | 00:32 |
| clarkb | we just do them like packer | 00:32 |
| openstackgerrit | A change was merged to openstack-infra/config: Allow choice of GIT protocol used. https://review.openstack.org/96539 | 00:32 |
| *** sdague has quit IRC | 00:33 | |
| sarob | can i get a look at https://review.openstack.org/#/c/96334/ | 00:33 |
| *** sdague has joined #openstack-infra | 00:33 | |
| *** gokrokve has joined #openstack-infra | 00:34 | |
| clarkb | sarob: can you make jenkins +1 it | 00:34 |
| lifeless | harlowja: braaaaaains | 00:34 |
| lifeless | that should totally be a thing | 00:35 |
| clarkb | sarob: it cannot merge in that state which is probably why it is being ignored | 00:35 |
| harlowja | lifeless someday brains will be a thing, haha | 00:35 |
| sarob | meaning pass alphebetized test? | 00:35 |
| clarkb | sarob: yes | 00:35 |
| sarob | clarkb: i can reorg all of the projects if that will make it pass | 00:35 |
| bodepd | clarkb: yep, now I understand the issue. You guys should just move to docker if you have to refactor the whole thing | 00:35 |
| *** melwitt has quit IRC | 00:35 | |
| clarkb | sarob: you just need to insert the new project in the correct spot | 00:36 |
| clarkb | bodepd: well docker doesn't solve the problem either | 00:37 |
| lifeless | hmm, gertty seems to be going backwards | 00:37 |
| nibalizer | haha bodepd | 00:37 |
| lifeless | 1610 pending syncs | 00:37 |
| sarob | clarkb: umm, the file has no organization i can tell | 00:37 |
| clarkb | sarob: https://review.openstack.org/#/c/96334/9/modules/openstack_project/files/review.projects.yaml that is the file | 00:37 |
| sarob | clarkb: the projects are kind of grouped similar | 00:37 |
| clarkb | bodepd: we have a deployment mechansim. `nova boot` | 00:37 |
| bodepd | clarkb: no, but if you have to move away from services, it would at least be worth looking into | 00:38 |
| sarob | clarkb: the test is looking for every project name in the file to go from a -z? | 00:38 |
| clarkb | sarob: yes, we did that to avoid merge conflicts. Prior to that everyone added to the end of the file and merge conflicted everywhere | 00:39 |
| sarob | clarkb: ill take care of it | 00:39 |
| sarob | clarkb: patch soon | 00:39 |
| clarkb | sarob: it should only be those 3 lines in that file that need moving | 00:39 |
| sarob | clarkb: there is no project name alpha sort or pattern i can see | 00:40 |
| clarkb | sarob: there should be in https://review.openstack.org/#/c/96334/9/modules/openstack_project/files/review.projects.yaml | 00:41 |
| sarob | clarkb: ah, okay sorry i was looking at projects.yaml | 00:42 |
| sarob | clarkb: bit confusing | 00:42 |
| asselin | clarkb, thanks for the approval! One less workaround :) | 00:42 |
| clarkb | lifeless: yes it is related to the number of projects you are watching and the number of changes in those projects | 00:42 |
| clarkb | lifeless: I think gertty may need to learn to sync more smartly | 00:42 |
| *** nati_ueno has quit IRC | 00:43 | |
| *** Ryan_Lane has quit IRC | 00:44 | |
| openstackgerrit | Sean Roberts proposed a change to openstack-infra/config: Add separate project for training-guides https://review.openstack.org/96334 | 00:46 |
| *** nati_ueno has joined #openstack-infra | 00:47 | |
| clarkb | lifeless: though update if you haven't. latest gertty should have more smarts and only sync if there are changes | 00:47 |
| lifeless | clarkb: last day or so? | 00:48 |
| clarkb | lifeless: no before jeblair left | 00:48 |
| *** markmcclain has quit IRC | 00:48 | |
| lifeless | he's gone? | 00:50 |
| * lifeless is confused | 00:50 | |
| *** nati_ueno has quit IRC | 00:50 | |
| clarkb | he is walking around in the wilderness with no computer apparently | 00:50 |
| clarkb | should be back this week | 00:50 |
| *** devkulkarni has joined #openstack-infra | 00:50 | |
| clarkb | also 97753 adds ^O help | 00:50 |
| lifeless | cool | 00:52 |
| lifeless | so yeah I branched this only a couple days back | 00:52 |
| *** nati_ueno has joined #openstack-infra | 00:54 | |
| sarob | hate to bug, buuut https://review.openstack.org/#/c/96334/ | 00:56 |
| sarob | training project needs its repo | 00:56 |
| clarkb | fungi: what do you think about doing the gerrit ACL normalization and cleanup this week before you move? | 00:57 |
| sarob | clarkb: take a look? | 00:57 |
| lifeless | speaking of acls | 00:57 |
| clarkb | fungi: we can pick an arbitrary day (say thursday) and just do it | 00:57 |
| lifeless | the acl on dib-utils is odd; we can't +2 there | 00:57 |
| *** belliott has left #openstack-infra | 00:57 | |
| clarkb | sarob: do we know why the diff on https://review.openstack.org/#/c/96334/10/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml is so large? | 00:58 |
| sarob | hmm, these changes i dont recognize | 01:00 |
| clarkb | yeah its a bit weird | 01:00 |
| clarkb | looks likethey were added in ps9 | 01:00 |
| sarob | tfifield isnt online | 01:01 |
| harlowja | sarob do u want me to help u in person? | 01:01 |
| harlowja | *if u around* | 01:01 |
| sarob | clarkb: its from the first patch | 01:01 |
| *** salv-orlando has quit IRC | 01:02 | |
| clarkb | sarob: https://review.openstack.org/#/c/96334/1/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml is the first patch | 01:02 |
| clarkb | sarob: but patchset 9 and 10 seem to add things for aviator and friends | 01:02 |
| clarkb | almost like tom rebased weirdly on proposed changes | 01:03 |
| fungi | clarkb: yeah, let me get that patch series back in shape first thing tomorrow and we can power through | 01:03 |
| sarob | clarkb: ahh, right. patch 9 | 01:03 |
| sarob | clarkb: i bet it was an attempt to pass the alphabetsoup test | 01:03 |
| clarkb | fungi: great | 01:04 |
| fungi | clarkb: i'll put that and new git-review release at the top of my list for tomorrow | 01:04 |
| fungi | since the day of neverending meetings is now behind me | 01:04 |
| sarob | clarkb: ill push another patch with projects.yaml from patch 8 | 01:05 |
| sarob | clarkb: good catch | 01:05 |
| sarob | harlowja thx for offer i think we got it | 01:05 |
| *** thuc has quit IRC | 01:05 | |
| harlowja | sarob np | 01:05 |
| *** gema has quit IRC | 01:05 | |
| clarkb | fungi: neverending meetings? surely you know better :P | 01:06 |
| *** thuc has joined #openstack-infra | 01:06 | |
| *** arnaud has quit IRC | 01:06 | |
| *** gema has joined #openstack-infra | 01:07 | |
| fungi | in the meantime, i have an irc server, a mailserver and several websites to move into rackspace | 01:08 |
| *** gyee has quit IRC | 01:08 | |
| * fungi gets to it | 01:08 | |
| *** marun has quit IRC | 01:08 | |
| *** nati_ueno has quit IRC | 01:09 | |
| fungi | at least this is the last of it. all the rest of my systems are migrated now | 01:09 |
| *** thuc has quit IRC | 01:10 | |
| *** thuc has joined #openstack-infra | 01:17 | |
| clarkb | jhesketh: I commented on 98029. I really have no idea how puppet lint didn't break on other things but breaks there | 01:20 |
| clarkb | jhesketh: also https please and if we can we should only download that file if it changes | 01:20 |
| *** camunoz has quit IRC | 01:20 | |
| *** zns has joined #openstack-infra | 01:21 | |
| *** sarob has quit IRC | 01:21 | |
| jhesketh | clarkb: thanks. I have a few patches to reroll so I'll get to that later :-) | 01:22 |
| *** CaptTofu_ has quit IRC | 01:22 | |
| *** CaptTofu_ has joined #openstack-infra | 01:23 | |
| clarkb | jhesketh: sounds good | 01:23 |
| clarkb | jhesketh: in my comment I noted that I think the storyboard manifests do a similar thing but only download the file if it changes | 01:23 |
| clarkb | so you can look there for inspiration | 01:23 |
| jhesketh | okay cheers | 01:24 |
| clarkb | since they have js deps too | 01:25 |
| openstackgerrit | Nikhil Manchanda proposed a change to openstack-infra/config: Added new experimental job for trove functional tests https://review.openstack.org/98517 | 01:27 |
| *** CaptTofu_ has quit IRC | 01:27 | |
| *** thuc_ has joined #openstack-infra | 01:29 | |
| *** thuc has quit IRC | 01:29 | |
| openstackgerrit | A change was merged to openstack/requirements: Update hacking version to align with what projects should use https://review.openstack.org/99172 | 01:30 |
| clarkb | fungi: are you comfortable with me +2 +A'ing 94440 given that I edited it? | 01:32 |
| clarkb | I noticed your comment saying jhesketh mordred_phone and SergeyLukjanov should +2 which means I think you don't want me "self approving" | 01:32 |
| clarkb | jhesketh: SergeyLukjanov mordred_phone mordred: on that note can you review 94440? | 01:33 |
| clarkb | would be excellent to have that in | 01:33 |
| jhesketh | clarkb: Is this a different initial commit, or am I confusing this with a different repository | 01:35 |
| jhesketh | ah yes, I'm confusing it with infra-manuals | 01:35 |
| fungi | clarkb: yeah, i'm cool with it, though the more +2s from cores the better | 01:35 |
| *** dims__ has quit IRC | 01:35 | |
| jhesketh | I'll take a look shortly | 01:35 |
| fungi | since this is defining our planning process | 01:35 |
| fungi | i'd like to show some additional core solidarity on it, but i also don't want it to rot any longer than it already has | 01:36 |
| fungi | and it's not immutable so i guess we can still fix later disagreements if needed | 01:37 |
| clarkb | ++ I think if jhesketh is happy with it then we can go ahead and merge | 01:38 |
| *** sarob has joined #openstack-infra | 01:38 | |
| fungi | wfm | 01:38 |
| * fungi disappears back into the aether again | 01:39 | |
| *** sarob has quit IRC | 01:42 | |
| clarkb | jhesketh: I am looking at 98306, are we assuming that the USED flag in the DB will be updated without zmq? | 01:43 |
| clarkb | jhesketh: I believe today that that flag is set when zmq emits the onbuildstarted event | 01:43 |
| jhesketh | clarkb: No, the USED flag won't be set | 01:43 |
| *** aysyd has quit IRC | 01:44 | |
| jogo | jhesketh: turbo hipster seems to be reporting fails in nova on docstring changes | 01:44 |
| clarkb | jhesketh: that change sets n_used from the DB though | 01:44 |
| jhesketh | jogo: link please? | 01:44 |
| jogo | clarkb: when are folks arriving on Germany? | 01:44 |
| clarkb | jogo: sunday I think | 01:45 |
| clarkb | I arrive at ~9am | 01:45 |
| jhesketh | clarkb: that's counting the number currently used | 01:45 |
| clarkb | have an overnight flight | 01:45 |
| clarkb | jhesketh: right which comes from the DB USED flag | 01:45 |
| clarkb | which comes from zmq | 01:45 |
| jogo | jhesketh: https://review.openstack.org/#/c/98376/ | 01:45 |
| clarkb | jhesketh: I think your math is sound. I am just making sure I grok the separation from zmq | 01:45 |
| jhesketh | clarkb: right, so if you aren't running zmq there will be no used nodes | 01:46 |
| *** nosnos has joined #openstack-infra | 01:46 | |
| *** sarob has joined #openstack-infra | 01:46 | |
| jhesketh | jogo: when you say reporting fails on 'changes', ie plural, have you seen other cases | 01:47 |
| jogo | clarkb: cool, and whats the cool hotel | 01:47 |
| jogo | jhesketh: oh I exaggerated a little ;) | 01:47 |
| jhesketh | sometimes the timing stuff in turbo-hipster is wrong because nodes are hard to predict | 01:47 |
| clarkb | jogo: I booked at the maritim konferenz hotel in darmstadt | 01:47 |
| jogo | clarkb: thanks | 01:47 |
| * jogo books his travel | 01:47 | |
| jhesketh | jogo: as of 30min ago turbo-hipster now uses nodepool rather than rerunning on the same nodes. This has actually helped with less false positives (in our testing) probably due to the way the nodes are scheduled by nova | 01:48 |
| *** miqui has joined #openstack-infra | 01:48 | |
| jhesketh | so hopefully false positives will see a reduction from today | 01:48 |
| clarkb | jhesketh: nice | 01:48 |
| jogo | jhesketh: woot | 01:48 |
| jhesketh | "hopefully"... | 01:48 |
| clarkb | jhesketh: it is prboably too late in the day for me to review this nodepool change. I do not understand how this helps if n_used is always 0 in your calculations | 01:48 |
| jhesketh | clarkb: removing running_jobs from start_demand is what helps us | 01:50 |
| clarkb | oh I see I was focusing on the wrong half | 01:50 |
| jhesketh | because our "running" amount will be higher when not using zmq | 01:50 |
| jhesketh | even though "running" may be "used" for us | 01:51 |
| clarkb | jhesketh: oh running won't be equal? | 01:51 |
| jhesketh | equal to which? | 01:51 |
| *** thuc_ has quit IRC | 01:51 | |
| clarkb | jhesketh: the non zmq number | 01:52 |
| *** thuc has joined #openstack-infra | 01:52 | |
| clarkb | jhesketh: running is a gearman value and should be the same if you use jenkins + zmq or turbo hipster | 01:52 |
| jogo | wow flights from SFO to FRA are $$$ | 01:52 |
| *** signed8bit is now known as signed8bit_ZZZzz | 01:52 | |
| clarkb | jogo: ya I think because summer and all that | 01:53 |
| clarkb | jhesketh: but I can see how removing that subtraction forces your needed nodes higher | 01:54 |
| *** thomasbiege1 has joined #openstack-infra | 01:54 | |
| openstackgerrit | Sean Roberts proposed a change to openstack-infra/config: Add separate project for training-guides https://review.openstack.org/96334 | 01:54 |
| *** thuc has quit IRC | 01:57 | |
| *** adalbas has joined #openstack-infra | 01:57 | |
| clarkb | sarob: out of curiousity what text editor do you use? it stripped off the newline at the end of one of those files. this appears to be a common occurence now | 01:57 |
| *** thomasbiege has quit IRC | 01:57 | |
| sarob | pycharm | 01:58 |
| clarkb | huh I wonder if that is the culprit | 01:58 |
| clarkb | sarob: that patchset looks much better +2 | 01:58 |
| *** changbl has joined #openstack-infra | 01:58 | |
| sarob | projects.yaml, hmm | 01:58 |
| *** HenryG has joined #openstack-infra | 01:58 | |
| boris-42 | clarkb hi there | 01:58 |
| clarkb | boris-42: hello | 01:58 |
| sarob | clarkb: i gathered a bit of cruff | 01:59 |
| sarob | clarkb: it | 01:59 |
| boris-42 | clarkb maybe you can help me with fixing ACL in osprofiler project | 01:59 |
| sarob | clarkb: that is | 01:59 |
| boris-42 | clarkb for some reason they were net accepted | 01:59 |
| *** yaguang has joined #openstack-infra | 01:59 | |
| boris-42 | clarkb and I am not core in project =( | 01:59 |
| clarkb | boris-42: there was a bug in the script | 01:59 |
| harlowja | boris-42 show us credentials first, license and registration | 01:59 |
| clarkb | mordred fixed it today but apparently did not rerun it | 01:59 |
| boris-42 | clarkb so I should just wait a bit? | 02:00 |
| clarkb | boris-42: so I think we just need someone to babysit it at some point | 02:00 |
| clarkb | boris-42: but once that happens your group should be created and we can add you to that | 02:01 |
| clarkb | I would run it now but I am fading fast and need to head home | 02:01 |
| boris-42 | clarkb group is already created | 02:01 |
| clarkb | boris-42: oh right that happens before we push teh acl | 02:01 |
| clarkb | so I can add you to the group now | 02:01 |
| boris-42 | clarkb I am already in | 02:01 |
| clarkb | even better | 02:01 |
| *** camunoz has joined #openstack-infra | 02:01 | |
| boris-42 | clarkb it's rally-core group | 02:01 |
| boris-42 | clarkb but project osprofiler is not related to it | 02:01 |
| clarkb | in that case we just need someone with enough time to babysit it to make sure that it actually fixes the problem | 02:02 |
| clarkb | and I am not that person right now | 02:02 |
| boris-42 | clarkb ok+) | 02:02 |
| boris-42 | clarkb thanks | 02:02 |
| clarkb | SergeyLukjanov may be able to help or I can look into it in the morning as I am sure you are not the only person that has hit this | 02:02 |
| boris-42 | clarkb he is quite busy as well recently=) | 02:02 |
| *** yjiang51 has quit IRC | 02:04 | |
| jhesketh | clarkb, fungi: I've reviewed 64440 | 02:04 |
| jhesketh | *94440 | 02:04 |
| fungi | jhesketh: approve that noise | 02:05 |
| *** arnaud has joined #openstack-infra | 02:05 | |
| fungi | clarkb: it looks like manage-projects may need re-running now that mordred_phone's fix is merged, before boris-42's nre project will have its acl applied | 02:06 |
| *** adalbas has quit IRC | 02:06 | |
| fungi | s/nre/new/ | 02:06 |
| *** trinaths has joined #openstack-infra | 02:06 | |
| *** thuc has joined #openstack-infra | 02:07 | |
| fungi | oh, you already said that ;) | 02:07 |
| clarkb | ya | 02:08 |
| clarkb | but I need beer and basketball now | 02:08 |
| fungi | clarkb: mordred_phone: boris-42: rerunning 'manage-projects -v stackforge/osprofiler' seems to have worked now | 02:09 |
| boris-42 | fungi works1 | 02:09 |
| fungi | so we should be in good shape now | 02:09 |
| boris-42 | fungi clarkb mordred thank you guys! | 02:09 |
| * fungi meant to check it earlier, but only so many hours in the day | 02:10 | |
| clarkb | fungi: thanks | 02:10 |
| fungi | no worries. back to pretending i'm not here ;) | 02:10 |
| *** signed8bit_ZZZzz has quit IRC | 02:16 | |
| *** devkulkarni has quit IRC | 02:18 | |
| *** devkulkarni has joined #openstack-infra | 02:19 | |
| sarob | fungi if i may bother you for a review sir https://review.openstack.org/#/c/96334/ | 02:21 |
| sarob | fungi: training project is bound up until this patch gets our repo up | 02:22 |
| *** ramashri_ has quit IRC | 02:22 | |
| *** thuc has quit IRC | 02:23 | |
| *** bnemec is now known as bnemec_away | 02:24 | |
| *** thuc has joined #openstack-infra | 02:25 | |
| lifeless | clarkb: isthere a fix for | 02:25 |
| lifeless | File "/home/robertc/.virtualenvs/openstack/local/lib/python2.7/site-packages/gertty/view/diff.py", line 340, in handleUndisplayedComments | 02:25 |
| lifeless | diff = self.file_diffs[oldnew][path] | 02:25 |
| lifeless | KeyError: '/COMMIT_MSG' | 02:25 |
| lifeless | which I get on review 95625 | 02:26 |
| clarkb | is that commit msg comments? | 02:26 |
| clarkb | I think no fix yet | 02:26 |
| *** thuc_ has joined #openstack-infra | 02:26 | |
| lifeless | I just hit enter on Diff | 02:26 |
| lifeless | and it barfs | 02:26 |
| clarkb | ya because there are comments on the commit msg | 02:28 |
| lifeless | ok | 02:28 |
| * sarob fungi, wind beneath my wings, yo | 02:28 | |
| lifeless | so, if I had a little more time I'd fix it properly | 02:29 |
| openstackgerrit | A change was merged to openstack-infra/infra-specs: Initial commit https://review.openstack.org/94440 | 02:29 |
| lifeless | sems straight forward | 02:29 |
| lifeless | add the commit to the things to diff | 02:29 |
| lifeless | standard sequence diff rules | 02:29 |
| *** thuc has quit IRC | 02:29 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add separate project for training-guides https://review.openstack.org/96334 | 02:29 |
| lifeless | and call it /COMMIT_MSG | 02:29 |
| *** signed8bit has joined #openstack-infra | 02:30 | |
| lifeless | or perhaps add it to gitrepo | 02:30 |
| *** signed8bit has quit IRC | 02:32 | |
| clarkb | yeah I think if it ends up in gitrepo then it will just work (TM) | 02:35 |
| clarkb | actually maybe no | 02:35 |
| clarkb | for i, diff in enumerate(repo.diff(self.base_commit, self.commit)) <- that is where you want it to show up | 02:36 |
| clarkb | I maybe add this | 02:36 |
| clarkb | which of course is a gitrepo.Repo | 02:37 |
| *** trinaths has quit IRC | 02:39 | |
| lifeless | yes | 02:40 |
| *** thuc_ has quit IRC | 02:40 | |
| lifeless | repo.diff is gitrepo.diff not git.diff | 02:40 |
| clarkb | however it may not be correct to enumerate it there as diffing against the base will result in weirness with commit messages | 02:41 |
| *** thuc has joined #openstack-infra | 02:41 | |
| lifeless | I'll take weirdness over crashing :) | 02:41 |
| *** CaptTofu_ has joined #openstack-infra | 02:41 | |
| lifeless | right crash fixed, now for diffs | 02:43 |
| clarkb | oh good you are working on it. I can go back to beer and basketball :) | 02:43 |
| lifeless | its a vcs | 02:43 |
| lifeless | I spent years doing vcs stuff | 02:43 |
| lifeless | did you really think I could resist ? | 02:43 |
| clarkb | :P | 02:44 |
| clarkb | ideally if diffing against the base the new commit message will be all + lines. and diffing against other patchsets will show a proper diff | 02:44 |
| clarkb | but I will take weirdness over crashing as well | 02:44 |
| *** thuc has quit IRC | 02:45 | |
| *** CaptTofu_ has quit IRC | 02:46 | |
| *** CaptTofu_ has joined #openstack-infra | 02:55 | |
| *** sarob has quit IRC | 02:56 | |
| *** sarob has joined #openstack-infra | 02:56 | |
| *** asettle has quit IRC | 02:58 | |
| *** markwash_ has joined #openstack-infra | 02:58 | |
| *** thuc has joined #openstack-infra | 02:59 | |
| *** yjiang51 has joined #openstack-infra | 03:00 | |
| *** markwash has quit IRC | 03:00 | |
| *** markwash_ is now known as markwash | 03:00 | |
| *** sarob has quit IRC | 03:01 | |
| *** hdd_ has joined #openstack-infra | 03:05 | |
| clarkb | jhesketh: do we know why https://jenkins03.openstack.org/job/experimental-swift-logs-config-pep8/3/console that fails? | 03:07 |
| clarkb | jhesketh: it looks like maybe your fix hasn't made it onto our slave images (the fix that removes $) | 03:08 |
| jhesketh | clarkb: yeah, that's why I kept re-running the experiments but I haven't had time to dig further | 03:08 |
| * clarkb goes to poke nodepool | 03:09 | |
| clarkb | everything is old | 03:10 |
| clarkb | trying a manual build now | 03:11 |
| clarkb | I am guessing we broke it at some point | 03:11 |
| jhesketh | clarkb: all the slaves are old? | 03:11 |
| clarkb | all the images are. about a week old | 03:11 |
| *** thuc has quit IRC | 03:12 | |
| *** thuc has joined #openstack-infra | 03:13 | |
| clarkb | lol I see the problem | 03:13 |
| *** thuc_ has joined #openstack-infra | 03:14 | |
| *** yamahata has joined #openstack-infra | 03:15 | |
| *** hdd_ has quit IRC | 03:15 | |
| *** yamahata has quit IRC | 03:16 | |
| *** otherwiseguy has joined #openstack-infra | 03:16 | |
| *** thuc has quit IRC | 03:17 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make sure we have a hostname when we grep for it https://review.openstack.org/99263 | 03:17 |
| clarkb | jhesketh: ^ that should fix it | 03:17 |
| clarkb | if it looks good to you I will probably self approve | 03:17 |
| clarkb | jhesketh: it evaluates to `grep -q /etc/hosts` when that variable is empty :/ | 03:19 |
| jhesketh | clarkb: why is it empty? | 03:19 |
| clarkb | jhesketh: because nodepool doesn't actually pass that data into the scripts... jesusaurus fixes that in a nodepool change that I think you reviewed | 03:20 |
| jhesketh | right, yeah I did see that | 03:20 |
| clarkb | I believe it was a thing that may have carried over from the pre nodepool days that wasn't properly implemented in nodepool | 03:20 |
| jhesketh | okay, lgtm | 03:20 |
| clarkb | alright I have just approved that change and will rebuild images once that makes it onto things | 03:23 |
| clarkb | I am going to shift back to swift logs now that dib is giving me problems. I need dib to ferment a bit | 03:23 |
| *** CaptTofu_ has quit IRC | 03:24 | |
| *** devkulkarni has quit IRC | 03:24 | |
| *** harlowja is now known as harlowja_away | 03:24 | |
| *** CaptTofu_ has joined #openstack-infra | 03:25 | |
| *** sarob has joined #openstack-infra | 03:27 | |
| fungi | today is the day of one-liner fixes to hilarious errors | 03:27 |
| clarkb | indeed | 03:27 |
| *** thuc has joined #openstack-infra | 03:28 | |
| *** sarob_ has joined #openstack-infra | 03:29 | |
| *** CaptTofu_ has quit IRC | 03:29 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack/requirements: Fix sphinx requirement to add overlap https://review.openstack.org/98947 | 03:30 |
| *** gokrokve has quit IRC | 03:30 | |
| *** lcheng has joined #openstack-infra | 03:30 | |
| *** sarob__ has joined #openstack-infra | 03:30 | |
| fungi | for some reason i always want to use !version instead of !=version pip version specs | 03:30 |
| *** penguinRaider has quit IRC | 03:31 | |
| openstackgerrit | A change was merged to openstack-infra/config: Make sure we have a hostname when we grep for it https://review.openstack.org/99263 | 03:31 |
| *** sarob has quit IRC | 03:31 | |
| *** thuc_ has quit IRC | 03:32 | |
| *** thuc has quit IRC | 03:32 | |
| *** nosnos has quit IRC | 03:33 | |
| *** sarob_ has quit IRC | 03:33 | |
| *** sarob__ has quit IRC | 03:35 | |
| *** yamahata has joined #openstack-infra | 03:35 | |
| *** thuc has joined #openstack-infra | 03:39 | |
| *** gokrokve has joined #openstack-infra | 03:42 | |
| *** asettle has joined #openstack-infra | 03:42 | |
| *** sweston has joined #openstack-infra | 03:42 | |
| *** asettle has left #openstack-infra | 03:42 | |
| *** asettle has joined #openstack-infra | 03:43 | |
| clarkb | civ V is now available on linux I may be slightly more distracted than I expected | 03:43 |
| *** gokrokve has quit IRC | 03:43 | |
| StevenK | clarkb: One of my friends said much the same thing this morning. :-) | 03:44 |
| clarkb | StevenK: I had no idea until about 10 minutes ago | 03:44 |
| clarkb | but this just made my day | 03:45 |
| *** sarob has joined #openstack-infra | 03:50 | |
| *** thuc_ has joined #openstack-infra | 03:54 | |
| *** thuc has quit IRC | 03:57 | |
| kashyap | Hi, tomorrow is M1 per release schedule - https://wiki.openstack.org/wiki/Juno_Release_Schedule, will we be able to make it? | 03:58 |
| clarkb | kashyap: I don't see why not | 03:58 |
| *** thuc_ has quit IRC | 03:58 | |
| clarkb | ttx would be the person to ask thoug | 03:58 |
| kashyap | clarkb, Alright. Yes, since it's time-based release model, I think of course it should be possible. | 03:59 |
| kashyap | I only asked because, I don't see any discussion of release on the dev mailing list :) | 03:59 |
| clarkb | well its not really a release, it is a milestone | 03:59 |
| kashyap | Right, poor wording | 03:59 |
| clarkb | things will be tagged, testing will happen, bugs will be fixed | 03:59 |
| *** arnaud has quit IRC | 04:00 | |
| kashyap | Great. Thanks for your quick response | 04:00 |
| *** zns has quit IRC | 04:01 | |
| *** zns has joined #openstack-infra | 04:02 | |
| *** llu has joined #openstack-infra | 04:06 | |
| *** sweston has quit IRC | 04:06 | |
| *** llu has left #openstack-infra | 04:07 | |
| *** Longgeek has joined #openstack-infra | 04:10 | |
| *** lcheng has quit IRC | 04:14 | |
| *** fifieldt has joined #openstack-infra | 04:15 | |
| *** nosnos has joined #openstack-infra | 04:15 | |
| fifieldt | can someone add sarob to training-manuals-ptl and training-manual-core ? | 04:15 |
| fifieldt | Sean Roberts <seanrob@yahoo-inc.com> | 04:17 |
| fifieldt | s/manuals/guides | 04:17 |
| fifieldt | :) | 04:17 |
| sarob | sarob: fifieldt easy mistake | 04:18 |
| *** matsuhashi has quit IRC | 04:19 | |
| sarob | fifieldt: hard to switch the meeting name, prob forget a few times | 04:19 |
| *** matsuhashi has joined #openstack-infra | 04:19 | |
| fifieldt | made a bug: https://bugs.launchpad.net/openstack-ci/+bug/1328769 | 04:20 |
| uvirtbot | Launchpad bug 1328769 in openstack-ci "Add Sean Roberts to training-guides-ptl, training-guides-core" [Undecided,New] | 04:20 |
| sarob | fifieldt: okay, good idea | 04:21 |
| *** matsuhashi has quit IRC | 04:23 | |
| *** trinaths has joined #openstack-infra | 04:24 | |
| *** praneshp has quit IRC | 04:25 | |
| *** timrc is now known as timrc-afk | 04:27 | |
| *** matsuhashi has joined #openstack-infra | 04:27 | |
| *** reed_ has joined #openstack-infra | 04:34 | |
| *** otherwiseguy has quit IRC | 04:34 | |
| *** gokrokve has joined #openstack-infra | 04:34 | |
| *** otherwiseguy has joined #openstack-infra | 04:34 | |
| *** reed_ has quit IRC | 04:34 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1313655 https://review.openstack.org/99267 | 04:35 |
| uvirtbot | Launchpad bug 1313655 in tempest "tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest.test_run_idempotent_instances[gate,smoke] fails" [Undecided,New] https://launchpad.net/bugs/1313655 | 04:36 |
| *** fifieldt has quit IRC | 04:41 | |
| *** gokrokve has quit IRC | 04:42 | |
| *** CaptTofu_ has joined #openstack-infra | 04:46 | |
| *** maxbit has quit IRC | 04:48 | |
| *** praneshp has joined #openstack-infra | 04:51 | |
| *** CaptTofu_ has quit IRC | 04:51 | |
| *** sarob has quit IRC | 04:55 | |
| *** sarob has joined #openstack-infra | 04:55 | |
| *** praneshp_ has joined #openstack-infra | 04:56 | |
| *** praneshp has quit IRC | 04:58 | |
| *** praneshp_ is now known as praneshp | 04:58 | |
| *** sarob has quit IRC | 04:59 | |
| *** yfried has quit IRC | 05:00 | |
| *** ildikov has quit IRC | 05:00 | |
| *** basha has joined #openstack-infra | 05:02 | |
| *** otherwiseguy has quit IRC | 05:05 | |
| *** gokrokve has joined #openstack-infra | 05:12 | |
| *** denis_makogon has joined #openstack-infra | 05:14 | |
| *** gokrokve_ has joined #openstack-infra | 05:15 | |
| *** gokrokve has quit IRC | 05:17 | |
| *** om has joined #openstack-infra | 05:19 | |
| jogo | hrm why is https://review.openstack.org/#/c/98864/ still failing the global reqs check | 05:20 |
| *** alkari has joined #openstack-infra | 05:21 | |
| jogo | and https://review.openstack.org/#/c/97893/2 landed so not sure if there is another related issue | 05:21 |
| jogo | clarkb fungi: ^ | 05:21 |
| *** talluri has joined #openstack-infra | 05:22 | |
| StevenK | jogo: https://review.openstack.org/#/c/93253/ is also failing in the same way | 05:24 |
| *** marun has joined #openstack-infra | 05:24 | |
| *** om has quit IRC | 05:27 | |
| *** sarob has joined #openstack-infra | 05:30 | |
| *** arnaud has joined #openstack-infra | 05:30 | |
| *** e0ne has joined #openstack-infra | 05:31 | |
| *** talluri has quit IRC | 05:33 | |
| *** sarob has quit IRC | 05:34 | |
| *** arnaud has quit IRC | 05:36 | |
| *** zns has quit IRC | 05:38 | |
| *** ildikov has joined #openstack-infra | 05:39 | |
| *** arnaud has joined #openstack-infra | 05:42 | |
| *** e0ne has quit IRC | 05:44 | |
| openstackgerrit | lifeless proposed a change to stackforge/gertty: Support commits in commits https://review.openstack.org/99272 | 05:45 |
| lifeless | clarkb: is there a bug for the COMMIT_MSG thing ? | 05:45 |
| *** talluri has joined #openstack-infra | 05:45 | |
| *** om has joined #openstack-infra | 05:50 | |
| *** arnaud has quit IRC | 05:51 | |
| openstackgerrit | lifeless proposed a change to stackforge/gertty: Support commits in commits https://review.openstack.org/99272 | 05:51 |
| *** arnaud has joined #openstack-infra | 05:54 | |
| *** gokrokve_ has quit IRC | 05:55 | |
| *** Longgeek has quit IRC | 05:59 | |
| *** sarob has joined #openstack-infra | 06:00 | |
| *** rlandy has joined #openstack-infra | 06:04 | |
| *** alkari has quit IRC | 06:05 | |
| *** sarob has quit IRC | 06:05 | |
| *** arnaud has quit IRC | 06:07 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/infra-specs: Make use of IP per slave optional. https://review.openstack.org/95625 | 06:09 |
| *** praneshp has quit IRC | 06:20 | |
| *** asettle has quit IRC | 06:21 | |
| *** denis_makogon has quit IRC | 06:22 | |
| *** praneshp has joined #openstack-infra | 06:23 | |
| *** cp16net_ has joined #openstack-infra | 06:26 | |
| *** lcheng has joined #openstack-infra | 06:26 | |
| *** rdopieralski has joined #openstack-infra | 06:26 | |
| mattoliverau | I'm off for the day, have a great night all. | 06:30 |
| *** Longgeek has joined #openstack-infra | 06:30 | |
| *** cp16net_ has quit IRC | 06:30 | |
| *** sarob has joined #openstack-infra | 06:30 | |
| *** cp16net_ has joined #openstack-infra | 06:30 | |
| openstackgerrit | lifeless proposed a change to stackforge/gertty: Support comments in commits https://review.openstack.org/99272 | 06:31 |
| *** mmaglana has quit IRC | 06:31 | |
| *** mmaglana has joined #openstack-infra | 06:31 | |
| *** CaptTofu_ has joined #openstack-infra | 06:34 | |
| *** arnaud has joined #openstack-infra | 06:34 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack-dev/pbr: Register testr as a distutil entry point https://review.openstack.org/99277 | 06:34 |
| *** Longgeek has quit IRC | 06:35 | |
| *** sarob has quit IRC | 06:35 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack-infra/config: Add XStatic-* projects with packaged static files for Horizon https://review.openstack.org/95716 | 06:35 |
| *** mmaglana has quit IRC | 06:36 | |
| *** gokrokve has joined #openstack-infra | 06:36 | |
| *** zhiyan_ is now known as zhiyan | 06:37 | |
| *** yfried has joined #openstack-infra | 06:38 | |
| *** CaptTofu_ has quit IRC | 06:38 | |
| *** cp16net__ has joined #openstack-infra | 06:40 | |
| *** cp16net__ has quit IRC | 06:40 | |
| *** cp16net_ has quit IRC | 06:40 | |
| *** zzelle has joined #openstack-infra | 06:41 | |
| *** _nadya_ has joined #openstack-infra | 06:41 | |
| *** gokrokve has quit IRC | 06:41 | |
| *** cp16net_ has joined #openstack-infra | 06:41 | |
| *** matsuhashi has quit IRC | 06:42 | |
| *** matsuhashi has joined #openstack-infra | 06:43 | |
| *** yfried has quit IRC | 06:44 | |
| *** doude has joined #openstack-infra | 06:46 | |
| *** cp16net_ has quit IRC | 06:46 | |
| *** fbo is now known as fbo_away | 06:47 | |
| *** om has quit IRC | 06:47 | |
| *** matsuhashi has quit IRC | 06:47 | |
| *** zzelle has quit IRC | 06:49 | |
| *** matsuhashi has joined #openstack-infra | 06:50 | |
| *** arnaud has quit IRC | 06:50 | |
| *** rdopieralski has quit IRC | 06:50 | |
| *** rdopieralski has joined #openstack-infra | 06:51 | |
| *** jcoufal has joined #openstack-infra | 06:51 | |
| openstackgerrit | Dzmitry Horbach proposed a change to openstack-infra/jenkins-job-builder: support Managed Script Plugin https://wiki.jenkins-ci.org/display/JENKINS/Managed+Script+Plugin https://review.openstack.org/99109 | 06:54 |
| *** salv-orlando has joined #openstack-infra | 06:54 | |
| *** cody-somerville has quit IRC | 06:56 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482 | 06:56 |
| *** camunoz has quit IRC | 06:59 | |
| *** talluri has quit IRC | 07:00 | |
| *** jistr has joined #openstack-infra | 07:00 | |
| *** talluri has joined #openstack-infra | 07:00 | |
| *** dizquierdo has joined #openstack-infra | 07:01 | |
| *** jpich has joined #openstack-infra | 07:01 | |
| *** yfried has joined #openstack-infra | 07:01 | |
| *** Longgeek has joined #openstack-infra | 07:01 | |
| *** derekh_ has joined #openstack-infra | 07:01 | |
| *** matsuhashi has quit IRC | 07:02 | |
| *** matsuhashi has joined #openstack-infra | 07:03 | |
| *** viktors|afk is now known as viktors | 07:03 | |
| openstackgerrit | Isaku Yamahata proposed a change to openstack-infra/config: Add tacker project on StackForge https://review.openstack.org/97435 | 07:03 |
| *** matsuhashi has quit IRC | 07:04 | |
| *** matsuhashi has joined #openstack-infra | 07:04 | |
| *** pblaho has joined #openstack-infra | 07:05 | |
| *** talluri has quit IRC | 07:05 | |
| *** Longgeek has quit IRC | 07:05 | |
| *** basha has quit IRC | 07:06 | |
| *** cody-somerville has joined #openstack-infra | 07:08 | |
| *** _nadya_ has quit IRC | 07:09 | |
| *** yfried has quit IRC | 07:12 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/nodepool: Remove the need for zmq in proper demand calcs. https://review.openstack.org/98306 | 07:12 |
| *** _nadya_ has joined #openstack-infra | 07:13 | |
| *** yfried has joined #openstack-infra | 07:17 | |
| *** Longgeek has joined #openstack-infra | 07:17 | |
| *** jgallard has joined #openstack-infra | 07:19 | |
| *** yfried has quit IRC | 07:21 | |
| *** yfried has joined #openstack-infra | 07:21 | |
| *** tkelsey has joined #openstack-infra | 07:21 | |
| *** Longgeek has quit IRC | 07:22 | |
| *** Hal_ has joined #openstack-infra | 07:22 | |
| *** jlibosva has joined #openstack-infra | 07:22 | |
| *** praneshp has quit IRC | 07:24 | |
| *** flaper87|afk is now known as flaper87 | 07:25 | |
| *** isviridov|away is now known as isviridov | 07:27 | |
| *** sarob has joined #openstack-infra | 07:29 | |
| *** markmc has joined #openstack-infra | 07:32 | |
| *** sarob has quit IRC | 07:33 | |
| *** andreykurilin_ has joined #openstack-infra | 07:34 | |
| *** gokrokve has joined #openstack-infra | 07:35 | |
| *** gokrokve_ has joined #openstack-infra | 07:37 | |
| *** ominakov has quit IRC | 07:38 | |
| *** gokrokve has quit IRC | 07:40 | |
| *** om has joined #openstack-infra | 07:40 | |
| *** gokrokve_ has quit IRC | 07:41 | |
| *** mrda is now known as mrda-away | 07:42 | |
| openstackgerrit | Dzmitry Horbach proposed a change to openstack-infra/jenkins-job-builder: support Managed Script Plugin https://wiki.jenkins-ci.org/display/JENKINS/Managed+Script+Plugin https://review.openstack.org/99109 | 07:47 |
| lifeless | clarkb: btw did you see my q? | 07:50 |
| lifeless | clarkb: oh bah I had paged up in IRC and thought you were around. nvm | 07:51 |
| *** thuc has joined #openstack-infra | 07:54 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Retry command on "hash mismatch" error in tests https://review.openstack.org/98354 | 07:58 |
| openstackgerrit | Dzmitry Horbach proposed a change to openstack-infra/jenkins-job-builder: support Managed Script Plugin https://wiki.jenkins-ci.org/display/JENKINS/Managed+Script+Plugin https://review.openstack.org/99109 | 07:58 |
| *** pblaho has quit IRC | 07:58 | |
| *** dizquierdo has quit IRC | 07:58 | |
| *** thuc has quit IRC | 07:59 | |
| *** talluri has joined #openstack-infra | 08:01 | |
| *** hashar has joined #openstack-infra | 08:02 | |
| *** ociuhandu has quit IRC | 08:03 | |
| *** roeyc has joined #openstack-infra | 08:04 | |
| *** pcrews has quit IRC | 08:05 | |
| *** talluri has quit IRC | 08:06 | |
| *** srenatus has quit IRC | 08:07 | |
| *** srenatus has joined #openstack-infra | 08:07 | |
| *** IvanBerezovskiy has joined #openstack-infra | 08:08 | |
| *** yfried_ has joined #openstack-infra | 08:12 | |
| *** yfried has quit IRC | 08:13 | |
| *** yfried_ has quit IRC | 08:18 | |
| *** pelix has joined #openstack-infra | 08:19 | |
| *** e0ne has joined #openstack-infra | 08:21 | |
| *** yfried has joined #openstack-infra | 08:21 | |
| *** ihrachyshka has joined #openstack-infra | 08:22 | |
| *** CaptTofu_ has joined #openstack-infra | 08:22 | |
| *** talluri has joined #openstack-infra | 08:23 | |
| *** fbo_away is now known as fbo | 08:26 | |
| *** basha has joined #openstack-infra | 08:26 | |
| *** CaptTofu_ has quit IRC | 08:27 | |
| *** ihrachyshka has quit IRC | 08:27 | |
| *** ihrachyshka has joined #openstack-infra | 08:27 | |
| *** talluri has quit IRC | 08:28 | |
| *** sarob has joined #openstack-infra | 08:28 | |
| *** sarob_ has joined #openstack-infra | 08:30 | |
| *** cody-somerville has quit IRC | 08:31 | |
| *** mmaglana has joined #openstack-infra | 08:32 | |
| *** sarob has quit IRC | 08:33 | |
| *** yjiang51 has quit IRC | 08:33 | |
| lifeless | oops | 08:34 |
| lifeless | File "/home/robertc/.virtualenvs/openstack/local/lib/python2.7/site-packages/gertty/mywid.py", line 148, in focusNextItem | 08:34 |
| lifeless | item = min(len(self.selectable_items)-1, self.focused_index+1) | 08:34 |
| lifeless | TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' | 08:34 |
| *** sarob_ has quit IRC | 08:35 | |
| *** mmaglana has quit IRC | 08:36 | |
| *** gokrokve has joined #openstack-infra | 08:36 | |
| *** _nadya_ has quit IRC | 08:37 | |
| *** yfried has quit IRC | 08:39 | |
| *** yfried has joined #openstack-infra | 08:39 | |
| *** gokrokve has quit IRC | 08:41 | |
| *** yfried_ has joined #openstack-infra | 08:42 | |
| *** _nadya_ has joined #openstack-infra | 08:43 | |
| *** yfried has quit IRC | 08:44 | |
| *** amotoki has joined #openstack-infra | 08:46 | |
| *** freyes has joined #openstack-infra | 08:46 | |
| *** yfried_ has quit IRC | 08:48 | |
| *** ominakov has joined #openstack-infra | 08:50 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: ssh publisher: add always-publish-from-master support https://review.openstack.org/98190 | 08:51 |
| *** yfried_ has joined #openstack-infra | 08:55 | |
| *** om has quit IRC | 08:59 | |
| *** Longgeek has joined #openstack-infra | 09:00 | |
| *** matsuhashi has quit IRC | 09:00 | |
| *** yfried_ has quit IRC | 09:00 | |
| *** matsuhashi has joined #openstack-infra | 09:01 | |
| *** yfried_ has joined #openstack-infra | 09:01 | |
| *** jamespage has quit IRC | 09:05 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: matrix project: add support for dynamic axis https://review.openstack.org/97111 | 09:05 |
| *** jamespage has joined #openstack-infra | 09:07 | |
| *** jamespage has joined #openstack-infra | 09:07 | |
| *** jamespage has quit IRC | 09:09 | |
| *** SpamapS has quit IRC | 09:10 | |
| *** om has joined #openstack-infra | 09:10 | |
| *** yfried__ has joined #openstack-infra | 09:11 | |
| *** SpamapS has joined #openstack-infra | 09:11 | |
| *** SpamapS has quit IRC | 09:11 | |
| *** SpamapS has joined #openstack-infra | 09:11 | |
| *** yfried_ has quit IRC | 09:12 | |
| *** rcarrillocruz has quit IRC | 09:12 | |
| *** rcarrillocruz has joined #openstack-infra | 09:14 | |
| *** yfried__ has quit IRC | 09:15 | |
| *** yfried__ has joined #openstack-infra | 09:17 | |
| *** rlandy has quit IRC | 09:17 | |
| *** andreykurilin_ has quit IRC | 09:21 | |
| *** yfried__ has quit IRC | 09:22 | |
| *** Longgeek_ has joined #openstack-infra | 09:25 | |
| *** thomasbiege1 has quit IRC | 09:26 | |
| *** om has quit IRC | 09:26 | |
| *** om has joined #openstack-infra | 09:27 | |
| *** yfried has joined #openstack-infra | 09:27 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Support variable-specific substitutions in templates. https://review.openstack.org/85383 | 09:27 |
| *** thomasbiege has joined #openstack-infra | 09:28 | |
| *** Longgeek has quit IRC | 09:29 | |
| *** sarob has joined #openstack-infra | 09:29 | |
| *** yfried_ has joined #openstack-infra | 09:29 | |
| *** yfried has quit IRC | 09:33 | |
| *** yfried__ has joined #openstack-infra | 09:33 | |
| *** habib has joined #openstack-infra | 09:33 | |
| *** sarob has quit IRC | 09:33 | |
| *** pblaho has joined #openstack-infra | 09:34 | |
| *** yfried_ has quit IRC | 09:36 | |
| *** gokrokve has joined #openstack-infra | 09:36 | |
| *** jgallard has quit IRC | 09:39 | |
| *** gokrokve has quit IRC | 09:41 | |
| *** om has quit IRC | 09:45 | |
| *** dizquierdo has joined #openstack-infra | 09:47 | |
| *** jistr has quit IRC | 09:52 | |
| *** habib has quit IRC | 09:53 | |
| *** habib has joined #openstack-infra | 09:53 | |
| *** jamespage has joined #openstack-infra | 09:57 | |
| *** zhiyan is now known as zhiyan_ | 10:00 | |
| *** om has joined #openstack-infra | 10:01 | |
| *** zhiyan_ is now known as zhiyan | 10:03 | |
| *** matsuhashi has quit IRC | 10:05 | |
| *** matsuhashi has joined #openstack-infra | 10:06 | |
| *** matsuhashi has quit IRC | 10:06 | |
| *** matsuhashi has joined #openstack-infra | 10:06 | |
| openstackgerrit | João Cravo proposed a change to openstack-infra/jenkins-job-builder: Add support for reverse build trigger https://review.openstack.org/95734 | 10:07 |
| openstackgerrit | Isaku Yamahata proposed a change to openstack-infra/config: Add tacker project on StackForge https://review.openstack.org/97435 | 10:07 |
| *** jistr has joined #openstack-infra | 10:08 | |
| *** Hal_ has quit IRC | 10:11 | |
| *** CaptTofu_ has joined #openstack-infra | 10:11 | |
| *** penguinRaider has joined #openstack-infra | 10:14 | |
| *** lcheng has quit IRC | 10:15 | |
| *** CaptTofu_ has quit IRC | 10:15 | |
| *** habib has quit IRC | 10:17 | |
| *** mdenny has quit IRC | 10:22 | |
| *** yamahata has quit IRC | 10:23 | |
| *** trinaths has quit IRC | 10:25 | |
| *** HenryG has quit IRC | 10:25 | |
| *** asselin has quit IRC | 10:26 | |
| *** zhiyan is now known as zhiyan_ | 10:28 | |
| *** rohitk has joined #openstack-infra | 10:28 | |
| *** sarob has joined #openstack-infra | 10:29 | |
| *** Longgeek_ has quit IRC | 10:30 | |
| *** Longgeek has joined #openstack-infra | 10:31 | |
| *** mmaglana has joined #openstack-infra | 10:31 | |
| *** sarob has quit IRC | 10:33 | |
| *** rfolco has quit IRC | 10:34 | |
| *** gokrokve has joined #openstack-infra | 10:36 | |
| *** mmaglana has quit IRC | 10:36 | |
| *** gokrokve has quit IRC | 10:41 | |
| *** matsuhashi has quit IRC | 10:45 | |
| sdague | SergeyLukjanov: if you are up, landing this is fine - https://review.openstack.org/#/c/98280/ | 10:53 |
| sdague | would nice to see fedora show up in the devstack list by default | 10:54 |
| basha | hi, is the jenkins gate bad ryt now?? I've been getting jenkins failures for a while. Does it make sense to recheck now?? Or should I wait till it gets stable?? | 10:55 |
| *** srenatus has quit IRC | 10:56 | |
| *** srenatus has joined #openstack-infra | 10:56 | |
| *** julim has joined #openstack-infra | 10:57 | |
| *** pblaho has quit IRC | 10:59 | |
| *** salv-orlando has quit IRC | 10:59 | |
| *** ociuhandu has joined #openstack-infra | 11:00 | |
| *** dims__ has joined #openstack-infra | 11:01 | |
| *** ociuhandu has quit IRC | 11:02 | |
| *** Guest83169 has joined #openstack-infra | 11:03 | |
| *** hashar is now known as hasharLunch | 11:04 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: display grenade summary instead of all of it https://review.openstack.org/99333 | 11:04 |
| *** rcarrill` has joined #openstack-infra | 11:05 | |
| *** rcarrillocruz has quit IRC | 11:06 | |
| SergeyLukjanov | sdague, morning, I'm here now | 11:07 |
| SergeyLukjanov | sdague, looking on the patch | 11:07 |
| *** gfa has quit IRC | 11:08 | |
| *** gfa has joined #openstack-infra | 11:08 | |
| *** enikanorov__ has joined #openstack-infra | 11:10 | |
| *** IvanBerezovskiy1 has joined #openstack-infra | 11:10 | |
| SergeyLukjanov | sdague, approved | 11:10 |
| *** dtroyer_zz has joined #openstack-infra | 11:10 | |
| *** afazekas is now known as __afazekas | 11:10 | |
| *** rwsu_ has joined #openstack-infra | 11:13 | |
| *** kashyap_lunch has joined #openstack-infra | 11:14 | |
| sdague | SergeyLukjanov: thanks | 11:14 |
| *** IvanBerezovskiy has quit IRC | 11:14 | |
| *** Guest83169 has quit IRC | 11:14 | |
| *** kashyap has quit IRC | 11:14 | |
| *** dtroyer has quit IRC | 11:14 | |
| *** enikanorov_ has quit IRC | 11:14 | |
| *** rwsu has quit IRC | 11:14 | |
| *** nikhil___ has quit IRC | 11:14 | |
| *** johnthetubaguy has quit IRC | 11:14 | |
| *** nikhil__ has joined #openstack-infra | 11:14 | |
| *** johnthetubaguy has joined #openstack-infra | 11:14 | |
| *** jasond` has quit IRC | 11:14 | |
| *** jasond` has joined #openstack-infra | 11:14 | |
| *** ilyashakhat has quit IRC | 11:14 | |
| *** ilyashakhat has joined #openstack-infra | 11:15 | |
| *** tristanC has quit IRC | 11:15 | |
| *** cyli has quit IRC | 11:15 | |
| *** tristanC has joined #openstack-infra | 11:15 | |
| *** devananda has quit IRC | 11:17 | |
| *** cyli has joined #openstack-infra | 11:17 | |
| *** devananda has joined #openstack-infra | 11:17 | |
| *** maurosr has quit IRC | 11:18 | |
| *** ogelbukh has quit IRC | 11:18 | |
| *** JayF has quit IRC | 11:19 | |
| openstackgerrit | A change was merged to openstack-infra/config: Promote devstack/f20 job to check job https://review.openstack.org/98280 | 11:19 |
| *** krotscheck has quit IRC | 11:19 | |
| *** slamont has quit IRC | 11:19 | |
| *** maurosr has joined #openstack-infra | 11:19 | |
| *** JayF has joined #openstack-infra | 11:19 | |
| *** JayF has quit IRC | 11:19 | |
| *** JayF has joined #openstack-infra | 11:19 | |
| *** doude has quit IRC | 11:20 | |
| *** slamont has joined #openstack-infra | 11:20 | |
| *** kashyap_lunch is now known as kashyap | 11:21 | |
| *** kashyap is now known as kashyap_lunch | 11:21 | |
| *** kashyap_lunch is now known as kashyap | 11:21 | |
| *** krotscheck has joined #openstack-infra | 11:21 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: display grenade summary instead of all of it https://review.openstack.org/99333 | 11:22 |
| *** srenatus has quit IRC | 11:24 | |
| *** srenatus has joined #openstack-infra | 11:24 | |
| *** doude has joined #openstack-infra | 11:26 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add f20 job to devstack-gate https://review.openstack.org/98281 | 11:27 |
| *** mancdaz has quit IRC | 11:27 | |
| *** weshay has joined #openstack-infra | 11:28 | |
| *** mattymo has quit IRC | 11:28 | |
| *** mattymo has joined #openstack-infra | 11:28 | |
| *** nikhil___ has joined #openstack-infra | 11:29 | |
| *** gilliard has joined #openstack-infra | 11:29 | |
| *** tkelsey has quit IRC | 11:30 | |
| *** mancdaz has joined #openstack-infra | 11:31 | |
| *** nikhil__ has quit IRC | 11:31 | |
| *** gilliard_ has quit IRC | 11:31 | |
| ttx | fungi: ping me when around so that I can start tagging | 11:33 |
| sdague | ttx: SergeyLukjanov is infra all powerful as well | 11:33 |
| SergeyLukjanov | sdague, ttx, ack | 11:34 |
| ttx | Ok, then I'll start soon :) | 11:34 |
| *** thomasbiege has quit IRC | 11:35 | |
| *** gokrokve has joined #openstack-infra | 11:36 | |
| *** cyli has quit IRC | 11:37 | |
| openstackgerrit | Peter Belanyi proposed a change to openstack-infra/config: Add jshint job for tuskar-ui https://review.openstack.org/96473 | 11:38 |
| *** cyli has joined #openstack-infra | 11:39 | |
| *** ihrachyshka has quit IRC | 11:40 | |
| SergeyLukjanov | ttx, I'll be here for at least several hours, so, I think I'll be able to help | 11:40 |
| *** gokrokve has quit IRC | 11:41 | |
| *** HenryG has joined #openstack-infra | 11:43 | |
| *** om has quit IRC | 11:46 | |
| *** yamahata has joined #openstack-infra | 11:47 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482 | 11:47 |
| *** ihrachyshka has joined #openstack-infra | 11:48 | |
| *** fbo is now known as fbo_away | 11:48 | |
| *** basha has quit IRC | 11:49 | |
| *** lcostantino has joined #openstack-infra | 11:51 | |
| *** dkliban_afk is now known as dkliban | 11:52 | |
| *** reaper has joined #openstack-infra | 11:53 | |
| *** rossella has joined #openstack-infra | 11:55 | |
| *** jgrimm has quit IRC | 11:55 | |
| *** rossella_s has quit IRC | 11:55 | |
| *** rossella is now known as rossella_s | 11:55 | |
| *** fbo_away is now known as fbo | 11:56 | |
| *** med_ has joined #openstack-infra | 11:57 | |
| *** med_ has quit IRC | 11:57 | |
| *** med_ has joined #openstack-infra | 11:57 | |
| *** med_ is now known as Guest22673 | 11:57 | |
| *** spiffxp has quit IRC | 11:58 | |
| *** anteaya has quit IRC | 11:58 | |
| *** CaptTofu_ has joined #openstack-infra | 11:59 | |
| *** spiffxp has joined #openstack-infra | 12:00 | |
| *** thomasbiege has joined #openstack-infra | 12:02 | |
| *** CaptTofu_ has quit IRC | 12:03 | |
| *** thomasbiege has quit IRC | 12:03 | |
| *** thomasbiege has joined #openstack-infra | 12:04 | |
| *** hasharLunch is now known as hashar | 12:04 | |
| *** basha has joined #openstack-infra | 12:04 | |
| *** dprince has joined #openstack-infra | 12:05 | |
| *** mancdaz has quit IRC | 12:06 | |
| *** adalbas has joined #openstack-infra | 12:06 | |
| *** afazekas has joined #openstack-infra | 12:07 | |
| ttx | SergeyLukjanov: ok, let's do this | 12:08 |
| *** mancdaz has joined #openstack-infra | 12:08 | |
| *** eharney has quit IRC | 12:09 | |
| *** rfolco has joined #openstack-infra | 12:11 | |
| *** pfallenop has quit IRC | 12:12 | |
| *** pfallenop has joined #openstack-infra | 12:13 | |
| *** Shrews has quit IRC | 12:16 | |
| *** Shrews has joined #openstack-infra | 12:16 | |
| *** Hal has joined #openstack-infra | 12:17 | |
| *** Hal is now known as Guest83434 | 12:17 | |
| *** Guest83434 is now known as Hal_ | 12:18 | |
| *** ogelbukh has joined #openstack-infra | 12:18 | |
| *** rfolco has quit IRC | 12:19 | |
| *** yaguang has quit IRC | 12:19 | |
| *** YorikSar has quit IRC | 12:20 | |
| *** YorikSar has joined #openstack-infra | 12:20 | |
| ttx | SergeyLukjanov, fungi: looks like it worked fine | 12:26 |
| ttx | checking the tarball for issues | 12:26 |
| *** basha has quit IRC | 12:27 | |
| *** Shrews has quit IRC | 12:28 | |
| *** sarob has joined #openstack-infra | 12:29 | |
| *** dims__ has quit IRC | 12:29 | |
| *** Shrews has joined #openstack-infra | 12:29 | |
| *** weshay has quit IRC | 12:30 | |
| *** sballe has quit IRC | 12:31 | |
| *** rlandy has joined #openstack-infra | 12:33 | |
| *** weshay has joined #openstack-infra | 12:33 | |
| *** sarob has quit IRC | 12:33 | |
| *** gokrokve has joined #openstack-infra | 12:36 | |
| *** dims__ has joined #openstack-infra | 12:36 | |
| *** nosnos has quit IRC | 12:36 | |
| ttx | SergeyLukjanov: post-mirror-python* jobs fail on the sahara pre-release job (https://jenkins.openstack.org/job/post-mirror-python26/117/) | 12:39 |
| ttx | not sure how important that is, the other projects don't seem to have such jobs | 12:39 |
| *** anteaya has joined #openstack-infra | 12:39 | |
| ttx | tarball generation was ok | 12:39 |
| *** oomichi_ has joined #openstack-infra | 12:40 | |
| *** gokrokve has quit IRC | 12:41 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482 | 12:41 |
| *** oomichi has quit IRC | 12:41 | |
| anteaya | mattoliverau: glad to hear it, yeah we were a bit all over the place on reviews, which was becoming unfair to contributors | 12:42 |
| *** nosnos has joined #openstack-infra | 12:43 | |
| *** habib has joined #openstack-infra | 12:43 | |
| *** pdmars has joined #openstack-infra | 12:44 | |
| *** sulochan_ has joined #openstack-infra | 12:45 | |
| *** lbragstad has joined #openstack-infra | 12:46 | |
| *** Alex_Gaynor has quit IRC | 12:46 | |
| *** DinaBelova2 has joined #openstack-infra | 12:46 | |
| *** nosnos has quit IRC | 12:47 | |
| *** oomichi_ is now known as oomichi_sleeping | 12:47 | |
| *** DinaBelova has quit IRC | 12:48 | |
| *** DinaBelova2 is now known as DinaBelova | 12:48 | |
| *** sulochan has quit IRC | 12:48 | |
| *** aysyd has joined #openstack-infra | 12:48 | |
| *** mancdaz has quit IRC | 12:48 | |
| *** mancdaz has joined #openstack-infra | 12:48 | |
| *** dkranz has joined #openstack-infra | 12:48 | |
| *** sandywalsh has joined #openstack-infra | 12:48 | |
| *** roeyc has quit IRC | 12:49 | |
| *** thuc has joined #openstack-infra | 12:50 | |
| *** thuc_ has joined #openstack-infra | 12:51 | |
| *** ArxCruz has joined #openstack-infra | 12:51 | |
| *** Alex_Gaynor has joined #openstack-infra | 12:51 | |
| *** adam_g has quit IRC | 12:51 | |
| *** roaet_ has quit IRC | 12:51 | |
| *** YorikSar has quit IRC | 12:52 | |
| *** YorikSar has joined #openstack-infra | 12:52 | |
| *** adam_g has joined #openstack-infra | 12:52 | |
| *** adam_g has quit IRC | 12:52 | |
| *** adam_g has joined #openstack-infra | 12:52 | |
| *** mwagner_lap has quit IRC | 12:52 | |
| *** roaet_ has joined #openstack-infra | 12:54 | |
| *** thuc has quit IRC | 12:55 | |
| *** prad has quit IRC | 12:55 | |
| *** Guest22673 has quit IRC | 12:56 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Support groovy plugin build steps https://review.openstack.org/99357 | 12:56 |
| *** tkelsey has joined #openstack-infra | 12:57 | |
| *** mancdaz has quit IRC | 12:57 | |
| *** hashar has quit IRC | 12:57 | |
| *** hashar has joined #openstack-infra | 12:57 | |
| *** mancdaz has joined #openstack-infra | 12:57 | |
| *** yolanda has joined #openstack-infra | 12:58 | |
| *** jraim has quit IRC | 13:00 | |
| *** mbacchi has joined #openstack-infra | 13:00 | |
| *** simonmcc has quit IRC | 13:00 | |
| *** zhiyan_ is now known as zhiyan | 13:00 | |
| *** yaguang has joined #openstack-infra | 13:01 | |
| *** simonmcc has joined #openstack-infra | 13:01 | |
| *** jraim has joined #openstack-infra | 13:01 | |
| *** jcoufal has quit IRC | 13:03 | |
| *** Alex_Gaynor has quit IRC | 13:05 | |
| *** mriedem has joined #openstack-infra | 13:05 | |
| *** Alex_Gaynor has joined #openstack-infra | 13:06 | |
| *** CaptTofu_ has joined #openstack-infra | 13:06 | |
| *** markmc has quit IRC | 13:08 | |
| *** wenlock_ has joined #openstack-infra | 13:08 | |
| *** radez_g0n3 is now known as radez | 13:09 | |
| *** markmc has joined #openstack-infra | 13:09 | |
| *** ramashri has joined #openstack-infra | 13:10 | |
| *** wenlock_ has quit IRC | 13:10 | |
| *** katyafervent is now known as katyafervent_awa | 13:10 | |
| *** wenlock_ has joined #openstack-infra | 13:10 | |
| *** atarakt has quit IRC | 13:12 | |
| *** smarcet has joined #openstack-infra | 13:12 | |
| *** jraim has quit IRC | 13:15 | |
| *** _nadya_ has quit IRC | 13:15 | |
| *** eharney has joined #openstack-infra | 13:15 | |
| *** maxbit has joined #openstack-infra | 13:16 | |
| *** trinaths has joined #openstack-infra | 13:16 | |
| *** jraim has joined #openstack-infra | 13:17 | |
| *** ramashri_ has joined #openstack-infra | 13:20 | |
| *** ramashri has quit IRC | 13:21 | |
| *** _nadya_ has joined #openstack-infra | 13:23 | |
| *** yaguang has quit IRC | 13:23 | |
| *** basha has joined #openstack-infra | 13:23 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/git-review: Use OpenSSL to generate test host server keys https://review.openstack.org/99364 | 13:24 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/git-review: Ensure username is set correctly for all tests https://review.openstack.org/99365 | 13:24 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/git-review: Provide nicer user message for missing remote ref https://review.openstack.org/97751 | 13:24 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Support groovy plugin build steps https://review.openstack.org/99357 | 13:28 |
| *** basha has quit IRC | 13:28 | |
| *** sarob has joined #openstack-infra | 13:29 | |
| *** spiffxp has quit IRC | 13:30 | |
| *** spiffxp has joined #openstack-infra | 13:30 | |
| *** jcoufal has joined #openstack-infra | 13:31 | |
| SergeyLukjanov | ttx, I think it's not really important, but I'm not sure | 13:33 |
| SergeyLukjanov | fungi, ^^ | 13:33 |
| *** sarob has quit IRC | 13:33 | |
| SergeyLukjanov | ttx, I'm going to push tags to the rest subprojects | 13:34 |
| *** rohitk has quit IRC | 13:35 | |
| *** yaguang has joined #openstack-infra | 13:35 | |
| *** reaper has quit IRC | 13:36 | |
| *** gokrokve has joined #openstack-infra | 13:36 | |
| *** yaguang has quit IRC | 13:38 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Validate Project and Project Groups names https://review.openstack.org/98728 | 13:38 |
| *** rcarrillocruz has joined #openstack-infra | 13:38 | |
| *** bknudson has quit IRC | 13:39 | |
| *** rcarrill` has quit IRC | 13:40 | |
| *** gokrokve has quit IRC | 13:41 | |
| *** thuc_ has quit IRC | 13:41 | |
| *** prad has joined #openstack-infra | 13:41 | |
| *** thuc has joined #openstack-infra | 13:41 | |
| *** habib_ has joined #openstack-infra | 13:42 | |
| *** prad has quit IRC | 13:43 | |
| fungi | ttx: here as well | 13:43 |
| *** prad has joined #openstack-infra | 13:44 | |
| ttx | fungi: worked well | 13:44 |
| fungi | ttx: awesome | 13:44 |
| *** habib has quit IRC | 13:45 | |
| *** homeless has joined #openstack-infra | 13:45 | |
| *** thuc_ has joined #openstack-infra | 13:45 | |
| fungi | sahara-2014.2.b1.tar.gz and sahara-2014.2.b1-py2-none-any.whl look right to me | 13:46 |
| *** thuc_ has quit IRC | 13:46 | |
| *** thuc has quit IRC | 13:46 | |
| *** briancurtin has left #openstack-infra | 13:46 | |
| *** thuc has joined #openstack-infra | 13:46 | |
| *** yamahata has quit IRC | 13:47 | |
| *** habib_ has quit IRC | 13:47 | |
| *** yamahata has joined #openstack-infra | 13:48 | |
| *** habib_ has joined #openstack-infra | 13:48 | |
| *** prad_ has joined #openstack-infra | 13:48 | |
| *** wenlock_ has quit IRC | 13:48 | |
| *** jaypipes has joined #openstack-infra | 13:49 | |
| EmilienM | hi there, could I have a review on enabling grenade job for ceilometer project ? > https://review.openstack.org/#/c/97430/ | 13:51 |
| SergeyLukjanov | oh, we have an issue... | 13:51 |
| *** basha has joined #openstack-infra | 13:51 | |
| SergeyLukjanov | fungi, ttx ^^ | 13:51 |
| *** thuc has quit IRC | 13:51 | |
| fungi | SergeyLukjanov: details? | 13:51 |
| SergeyLukjanov | wheels published to pypi.python.org | 13:51 |
| *** habib_ has quit IRC | 13:51 | |
| fungi | SergeyLukjanov: that's intentional | 13:51 |
| SergeyLukjanov | I mean 2014.2.b1 | 13:51 |
| fungi | are they broken? | 13:51 |
| *** prad has quit IRC | 13:51 | |
| SergeyLukjanov | fungi, oh, yeah... I forgot about it, that's ok | 13:52 |
| SergeyLukjanov | fungi, nope ,I think everything is ok | 13:52 |
| *** habib_ has joined #openstack-infra | 13:52 | |
| SergeyLukjanov | fungi, just don't expect to see new version in pypi :) | 13:52 |
| SergeyLukjanov | fungi, ttx, sorry, everything is ok | 13:52 |
| fungi | SergeyLukjanov: the versions of pip new enough to notice and install wheels from pypi are also new enough to regognize release and non-release version numbers, and won't pull in the latter by default even if they're newer than the most recent release | 13:52 |
| fungi | SergeyLukjanov: so the plan has been to publish releases to pypi as both wheels and tarballs, while only publishing wheels to pypi for non-release tags | 13:53 |
| fungi | SergeyLukjanov: though this may be the first time that's happened (i wasn't expecting sahara to be on pypi since most of our server projects don't publish to pypi, just clients/libraries/tools/data) | 13:54 |
| fungi | SergeyLukjanov: this actually points out a kink in that plan, which is that we probably don't want pypi displaying only the beta release for a project | 13:55 |
| *** pcrews has joined #openstack-infra | 13:55 | |
| fungi | mordred_phone: ^ ideas? | 13:55 |
| SergeyLukjanov | fungi, yup, thanks for details | 13:55 |
| *** habib_ has quit IRC | 13:55 | |
| SergeyLukjanov | fungi, I just forget about it :( | 13:55 |
| fungi | we might want to set wheel-only beta uploads to "hidden"? | 13:56 |
| fungi | dstufft: ^ you may also have suggestions in this regard | 13:56 |
| *** bknudson has joined #openstack-infra | 13:56 | |
| SergeyLukjanov | fungi, is it possible? | 13:56 |
| fungi | SergeyLukjanov: i believe so, but we haven't used that feature in the past | 13:56 |
| mordred | fungi: I'm not sure the ramifications | 13:57 |
| SergeyLukjanov | fungi, there is hide:yes/no dropdown for each version on pypi (I'm looking now on sahara package) | 13:57 |
| *** jistr has quit IRC | 13:59 | |
| fungi | that may be something we could set at upload | 14:00 |
| fungi | via an api call | 14:00 |
| *** yjiang51 has joined #openstack-infra | 14:01 | |
| fungi | mordred: the main concern is that the next time someone does a prerelease of, say, novaclient, the way we've been setting up these entries people browsing pypi are only going to see the beta listed rather than the most recent release | 14:01 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/git-review: Retry starting gerrit server for tests https://review.openstack.org/99380 | 14:02 |
| fungi | mordred: https://pypi.python.org/pypi/sahara | 14:02 |
| fungi | basically just an unanticipated side-effect of the new plan, i think | 14:03 |
| fungi | somewhat cosmetic, but affecting usability/discoverability for downstream consumers | 14:03 |
| *** jistr has joined #openstack-infra | 14:03 | |
| *** thuc has joined #openstack-infra | 14:04 | |
| mordred | yeah ... otoh it's not too different than a lot of other pypi things ... otoh | 14:04 |
| mordred | I've seen some packages that the main page shows several versions | 14:05 |
| mordred | https://pypi.python.org/pypi/Django | 14:05 |
| mordred | I think this is done by not marking the previous release as hidden or something | 14:05 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for glance API bug 1328738 https://review.openstack.org/99383 | 14:07 |
| uvirtbot | Launchpad bug 1328738 in glance "tempest.api.image.v1.test_images.ListImagesTest fails in setup with BadRequest - The HTTP URL is invalid" [Undecided,New] https://launchpad.net/bugs/1328738 | 14:07 |
| *** rcarrill` has joined #openstack-infra | 14:08 | |
| mordred | fungi: yes, that's how that works | 14:09 |
| *** thuc has quit IRC | 14:09 | |
| mordred | there is also a config option for a project called "auto-hide old releases" | 14:09 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/releasestatus: Include 'Low' blueprints in release status board https://review.openstack.org/99386 | 14:09 |
| *** rcarrillocruz has quit IRC | 14:10 | |
| *** gargola has quit IRC | 14:14 | |
| *** mwagner_lap has joined #openstack-infra | 14:14 | |
| *** dkehn_ has joined #openstack-infra | 14:14 | |
| *** dkehn_ is now known as dkehnx | 14:15 | |
| * devananda wakes up, notes the gate is backed up again | 14:15 | |
| *** zns has joined #openstack-infra | 14:15 | |
| mordred | devananda: it's probably your fault | 14:16 |
| devananda | mordred: could be | 14:16 |
| mordred | devananda: or also maybe not | 14:16 |
| mordred | sdague: I actually had a dream that you sent out an email pointing out that we'd landed a fix that was responsible for half the gate fails | 14:17 |
| mordred | sdague: because that's what I should be dreaming about | 14:18 |
| mordred | also - I got word that jeblair made it out of the wilderness safely | 14:19 |
| devananda | mordred: fwiw, a fix landed that was responsible for half of ironic's random gate fails | 14:19 |
| devananda | mordred: after which we landed about a dozen bug fixes | 14:19 |
| fungi | EmilienM: see my comment on 97430 but i suspect you added the wrong job there | 14:19 |
| *** vhoward has left #openstack-infra | 14:20 | |
| EmilienM | fungi: oops, ok I'm looking | 14:20 |
| devananda | mordred: granted, i think that fix hurt your brain | 14:20 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/releasestatus: Prettified all HTML files https://review.openstack.org/98476 | 14:20 |
| openstackgerrit | A change was merged to openstack-infra/releasestatus: Prettified all HTML files https://review.openstack.org/98476 | 14:21 |
| *** pdmars has quit IRC | 14:21 | |
| sdague | mordred: heh | 14:22 |
| *** pdmars has joined #openstack-infra | 14:22 | |
| sdague | I'll be honest, I need to recharge after log diving for 4 days | 14:22 |
| openstackgerrit | Monty Taylor proposed a change to stackforge/gertty: Move contributing section to its own file https://review.openstack.org/98363 | 14:22 |
| openstackgerrit | Monty Taylor proposed a change to stackforge/gertty: Remove unneeded bit from setup.cfg https://review.openstack.org/98364 | 14:22 |
| sdague | so I'm trying to work up some basic readability fixes in grenade to make it easier to understand what's going on for people | 14:22 |
| sdague | mordred: realistically the gate remains on fire | 14:23 |
| sdague | it took us 4 days to get it to just on fire, from zomg on fire | 14:23 |
| fungi | towering inferno of gate | 14:24 |
| sdague | but we need a bigger firebrigade, or some better tooling, or something to different to get us back on track | 14:24 |
| *** otherwiseguy has joined #openstack-infra | 14:24 | |
| *** gokrokve has joined #openstack-infra | 14:24 | |
| sdague | as someone with 25 outstanding patches, I know exactly how fire we are :) I have about a 50% chance of getting through a test run without a -1 | 14:25 |
| fungi | more people could be throwing water on it instead of fuel, for a start | 14:25 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Run stackalytics in infra https://review.openstack.org/98656 | 14:25 |
| sdague | fungi: right, that's a) "bigger firebrigade" | 14:25 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/releasestatus: Include 'Low' blueprints in release status board https://review.openstack.org/99386 | 14:25 |
| *** gokrokve_ has joined #openstack-infra | 14:25 | |
| fungi | well, it's repurposing the arsonists into the fire brigade, more specifically | 14:26 |
| *** zhiyan is now known as zhiyan_ | 14:26 | |
| mordred | fungi: but setting things on fire is more fun | 14:26 |
| sdague | fungi: did I remember that we landed some change to start purging grenade logs? | 14:27 |
| sdague | because if so, I need us to reverse that | 14:27 |
| * fungi now has the ink spots "i don't want to set the world on fire" stuck in his brain hole | 14:27 | |
| * mordred goes to find coffee and other useful things | 14:27 | |
| fungi | sdague: purging? i don't believe so. there as a change added to start collecting additional logs from grenade runs | 14:28 |
| *** gargola has joined #openstack-infra | 14:28 | |
| EmilienM | fungi: does that make sense then if I add "integrated-gate" in check list of ceilometer? | 14:28 |
| fungi | s/ as / was / | 14:28 |
| sdague | fungi: so this change doesn't include grenade log - http://logs.openstack.org/33/99333/2/check/check-grenade-dsvm/4773a87/console.html | 14:28 |
| sdague | EmilienM: yes, it was an oversite that it was missed | 14:28 |
| sdague | EmilienM: just make sure to pull the duplicate jobs out of the ceilo definition | 14:28 |
| *** gokrokve has quit IRC | 14:28 | |
| fungi | EmilienM: if you add the integrated-gate project template, then you will start being gated on grenade rather than being able to add it as an experimental job | 14:29 |
| *** rcarrillocruz has joined #openstack-infra | 14:29 | |
| sdague | fungi: I think that's fine | 14:29 |
| EmilienM | sdague: ok got it | 14:29 |
| sdague | everyone else is being gated on it | 14:29 |
| fungi | sdague: EmilienM: wfm | 14:29 |
| *** sarob has joined #openstack-infra | 14:29 | |
| *** rcarrill` has quit IRC | 14:31 | |
| *** sarob has quit IRC | 14:33 | |
| *** mdenny has joined #openstack-infra | 14:34 | |
| dhellmann | fungi: just saw your not on the ML about https://review.openstack.org/#/c/98947/2 -- are you ready for +2s on that or are we waiting for nodes to build with the right version of the test script? | 14:35 |
| fungi | dhellmann: if 98947 merges then it buys us more time for the updated nodes | 14:35 |
| dhellmann | fungi: ok | 14:35 |
| ttx | SergeyLukjanov: if you want to review https://review.openstack.org/#/c/99386/ before I self-approve myself | 14:36 |
| fungi | dhellmann: i'm monitoring the image updates currently running for the relevant node types for that job now and restarting any which fail | 14:36 |
| *** doude has quit IRC | 14:36 | |
| dhellmann | fungi: and then should we have another patch to restore the versions we're using now? or do we want the broader range? | 14:36 |
| fungi | dhellmann: but 98947 will trigger a new proposal which will actually pass the misplaced job even on an older node | 14:36 |
| sdague | fungi: yeh, so - https://review.openstack.org/#/c/94753/3/functions.sh | 14:36 |
| sdague | deletes the grenade logs | 14:37 |
| *** cp16net_ has joined #openstack-infra | 14:37 | |
| fungi | dhellmann: i was meaning to ask mordred about that, whether it was intentional that he didn't overlap with the previous entry in the original change | 14:37 |
| sdague | fungi: if I just remove that rm bit, we going to collect those? | 14:38 |
| dhellmann | fungi: ok, I'll go ahead and +2 for now and we can bump up the min version again later if needed | 14:38 |
| *** cp16net_ has quit IRC | 14:38 | |
| *** zz_gondoi is now known as gondoi | 14:39 | |
| *** thomasbiege has left #openstack-infra | 14:39 | |
| *** CaptTofu_ has quit IRC | 14:40 | |
| *** CaptTofu_ has joined #openstack-infra | 14:41 | |
| *** reaper has joined #openstack-infra | 14:41 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: display grenade summary instead of all of it https://review.openstack.org/99333 | 14:41 |
| SergeyLukjanov | ttx, looking | 14:41 |
| sdague | fungi: thoughts on that? I probably need my grenade change in for it to be sane | 14:41 |
| *** cp16net_ has joined #openstack-infra | 14:42 | |
| fungi | sdague: ahh, yep | 14:42 |
| fungi | sdague: that wasn't the only problem with that change in fact | 14:42 |
| fungi | after it merged, i spotted and fixed one issue via https://review.openstack.org/97251 | 14:42 |
| sdague | right, I remember the other one | 14:42 |
| SergeyLukjanov | ttx, lgtm | 14:43 |
| sdague | so actually | 14:43 |
| sdague | I should split that into 2 patches | 14:43 |
| fungi | (my long silence was me trying to make sure i hadn't already submitted a fix for the missing grenade log) | 14:43 |
| ttx | SergeyLukjanov: cool thx | 14:43 |
| fungi | ugh, image updates in rax-iad are failing complaining that we're over our ram quota | 14:43 |
| fungi | checking for leaks/cruft now | 14:44 |
| *** eharney_ has joined #openstack-infra | 14:44 | |
| *** prad_ is now known as prad | 14:44 | |
| *** Guest61298 has quit IRC | 14:44 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: display grenade summary instead of all of it https://review.openstack.org/99333 | 14:45 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: don't delete grenade logs https://review.openstack.org/99396 | 14:45 |
| openstackgerrit | A change was merged to openstack-infra/releasestatus: Include 'Low' blueprints in release status board https://review.openstack.org/99386 | 14:45 |
| *** CaptTofu_ has quit IRC | 14:45 | |
| sdague | if we could get fast approve on the don't delete patch, I can actually work out that my grenade filter patch is doing the right thing | 14:45 |
| *** cp16net has quit IRC | 14:45 | |
| *** cp16net_ is now known as cp16net | 14:47 | |
| sdague | SergeyLukjanov: https://review.openstack.org/#/c/99396 - fast approve appreciated :) Then I can actually make the grenade parts sane in the gate | 14:47 |
| *** eharney has quit IRC | 14:47 | |
| *** CaptTofu_ has joined #openstack-infra | 14:48 | |
| SergeyLukjanov | sdague, approved | 14:48 |
| sdague | I'm doing something crazy, developing a better feature in grenade first, then backporting to devstack - https://review.openstack.org/#/c/99390/ :) | 14:49 |
| fungi | only one alien node in rax-iad, so perhaps we're overcommitted on our quota | 14:50 |
| *** yaguang has joined #openstack-infra | 14:50 | |
| *** talluri has joined #openstack-infra | 14:50 | |
| *** sballe has joined #openstack-infra | 14:51 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for glance API bug 1328738 https://review.openstack.org/99383 | 14:51 |
| uvirtbot | Launchpad bug 1328738 in glance "tempest.api.image.v1.test_images.ListImagesTest fails in setup with BadRequest - The HTTP URL is invalid" [Undecided,New] https://launchpad.net/bugs/1328738 | 14:51 |
| fungi | ahh, yep, nodepool's configured for max-servers=192 and that's our quota | 14:54 |
| *** W00die has joined #openstack-infra | 14:54 | |
| *** yaguang has quit IRC | 14:54 | |
| fungi | so if images try to update when we're node starved, they'll just immediately fail there | 14:55 |
| *** yaguang has joined #openstack-infra | 14:55 | |
| fungi | ideally nodepool would be smart enough to deduct template nodes from its max-servers and wait for an available slot before beginning an image update, but for now i'll just lower max-servers to suit | 14:56 |
| *** zns has quit IRC | 14:57 | |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Some tweaks to get closer to Python 3 compat https://review.openstack.org/87810 | 14:58 |
| *** Sukhdev has joined #openstack-infra | 14:59 | |
| *** yolanda has quit IRC | 14:59 | |
| *** yjiang51 has quit IRC | 15:01 | |
| *** cdub has joined #openstack-infra | 15:03 | |
| fungi | oh crappity | 15:03 |
| *** basha has quit IRC | 15:03 | |
| fungi | maxTotalRAMSize in iad is 768000, which is enough for only 93 9gb nodes | 15:03 |
| fungi | er, 92 8gb instances | 15:03 |
| *** signed8bit has joined #openstack-infra | 15:04 | |
| fungi | phschwartz: ^ any idea why our ram quota in iad would suddenly be chopped by more than half? | 15:04 |
| *** dkehn_ has joined #openstack-infra | 15:04 | |
| cdub | for registering a new IRC channel, is there anything I need to do aside of setting openstackinfra as Founder? | 15:04 |
| fungi | cdub: update http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/accessbot/channels.yaml at a minimum | 15:05 |
| * cdub looks | 15:05 | |
| cdub | ok, and should i manually update wiki page (IRC page)? | 15:06 |
| fungi | cdub: see http://ci.openstack.org/irc.html for details on other bots you may want to join | 15:06 |
| fungi | cdub: yeah, the wiki page is updated manually, so you'll want to do that as well | 15:06 |
| cdub | fungi: thank you | 15:06 |
| *** signed8bit is now known as signed8bit_ZZZzz | 15:06 | |
| fungi | cdub: you're welcome, of course | 15:07 |
| *** dkehnx has quit IRC | 15:07 | |
| *** ramashri_ has quit IRC | 15:07 | |
| *** rickerc has quit IRC | 15:08 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 15:09 | |
| *** mfer has joined #openstack-infra | 15:11 | |
| *** jistr has quit IRC | 15:12 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Adjust nodepool limits for rax providers https://review.openstack.org/99401 | 15:13 |
| fungi | mordred: SergeyLukjanov: clarkb: ^ i'm self-approving that for the moment to get nodepool a little more sane | 15:14 |
| cdub | hmm, can someone verify hostkey for review.openstack.org? | 15:14 |
| SergeyLukjanov | fungi, ack | 15:15 |
| fungi | cdub: see https://review.openstack.org/#/settings/ssh-keys | 15:15 |
| *** lcostantino has quit IRC | 15:15 | |
| cdub | fungi: thanks (again ;) | 15:15 |
| fungi | cdub: the section which says "Server Host Key" has the host key fingerprint | 15:15 |
| *** dkehn_ is now known as dkehnx | 15:17 | |
| *** rdopieralski has quit IRC | 15:18 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/config: ceilometer: enable gate-grenade-dsvm-forward https://review.openstack.org/97430 | 15:19 |
| EmilienM | fungi: sdague ok, I hope I'm good with that ^ | 15:19 |
| *** zns has joined #openstack-infra | 15:20 | |
| EmilienM | was not sure about the experimental | 15:20 |
| *** mmaglana has joined #openstack-infra | 15:20 | |
| sdague | EmilienM: yeh, that looks right | 15:21 |
| *** zns_ has joined #openstack-infra | 15:21 | |
| *** thuc has joined #openstack-infra | 15:21 | |
| *** Longgeek has quit IRC | 15:22 | |
| fungi | EmilienM: lgtm too | 15:22 |
| EmilienM | ok cool, thanks | 15:22 |
| *** dkranz has quit IRC | 15:22 | |
| *** basha has joined #openstack-infra | 15:22 | |
| *** thuc has quit IRC | 15:23 | |
| *** ramashri has joined #openstack-infra | 15:23 | |
| *** jpich has quit IRC | 15:23 | |
| *** jistr has joined #openstack-infra | 15:23 | |
| *** zns__ has joined #openstack-infra | 15:24 | |
| *** zns has quit IRC | 15:24 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: tracking for ceilometer failing to start on grenade new side https://review.openstack.org/99404 | 15:24 |
| *** alkari has joined #openstack-infra | 15:24 | |
| *** alkari has quit IRC | 15:25 | |
| *** shivharis has joined #openstack-infra | 15:25 | |
| shivharis | anteaya: hi | 15:25 |
| *** zns_ has quit IRC | 15:27 | |
| *** UtahDave has joined #openstack-infra | 15:28 | |
| *** shivharis has left #openstack-infra | 15:28 | |
| *** sarob has joined #openstack-infra | 15:29 | |
| *** thuc has joined #openstack-infra | 15:30 | |
| *** virmitio has joined #openstack-infra | 15:31 | |
| *** ihrachyshka_ has joined #openstack-infra | 15:31 | |
| *** james_li has joined #openstack-infra | 15:32 | |
| *** ihrachyshka_ has quit IRC | 15:32 | |
| openstackgerrit | A change was merged to openstack-infra/config: Adjust nodepool limits for rax providers https://review.openstack.org/99401 | 15:33 |
| *** sarob has quit IRC | 15:33 | |
| *** yfried__ has quit IRC | 15:34 | |
| *** dkranz has joined #openstack-infra | 15:34 | |
| *** lcheng has joined #openstack-infra | 15:34 | |
| *** gyee has joined #openstack-infra | 15:35 | |
| *** e0ne has quit IRC | 15:35 | |
| *** ihrachyshka has quit IRC | 15:35 | |
| *** jcoufal has quit IRC | 15:38 | |
| *** jcoufal has joined #openstack-infra | 15:38 | |
| *** alkari has joined #openstack-infra | 15:38 | |
| *** YorikSar has quit IRC | 15:38 | |
| *** sballe_ has joined #openstack-infra | 15:40 | |
| openstackgerrit | David Johansen proposed a change to openstack-infra/jenkins-job-builder: Adding second failure option to the email-ext module https://review.openstack.org/98803 | 15:42 |
| *** sballe has quit IRC | 15:43 | |
| *** habib_ has joined #openstack-infra | 15:46 | |
| *** dangers_away is now known as dangers | 15:46 | |
| *** habib_ has quit IRC | 15:46 | |
| *** habib_ has joined #openstack-infra | 15:47 | |
| trinaths | fungi: hi | 15:48 |
| trinaths | fungi: How to generate console.html.gz with testr run? | 15:49 |
| *** hashar has quit IRC | 15:50 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: tracking for ceilometer failing to start on grenade new side https://review.openstack.org/99404 | 15:50 |
| *** yaguang has quit IRC | 15:50 | |
| *** lcostantino has joined #openstack-infra | 15:51 | |
| *** markwash has quit IRC | 15:53 | |
| *** yjiang51 has joined #openstack-infra | 15:53 | |
| *** eharney_ is now known as eharney | 15:54 | |
| fungi | trinaths: jenkins actually makes that file itself, and it accumulates in memory on the jenkins master. we use the jenkins scp publisher plugin to copy that to our log archive | 15:55 |
| *** dkranz has quit IRC | 15:55 | |
| *** yjiang51 has quit IRC | 15:56 | |
| pleia2 | good morning | 15:56 |
| *** e0ne_ has joined #openstack-infra | 15:57 | |
| trinaths | fungi: okay. But I'm not using any scp to my FTP server. Can you guide me with the path where can I manually get it | 15:57 |
| *** signed8bit is now known as signed8bit_ZZZzz | 15:58 | |
| fungi | trinaths: it doesn't exist as a file as far as i know, which is why we have to use a feature of a jenkins plugin to extract it from a data structure in memory on the jenkins master. zaro might be able to provide more details once he's around for the day | 15:59 |
| *** asselin has joined #openstack-infra | 15:59 | |
| fungi | or clarkb (i think he may have written the original patches to the scp publisher plugin to make that work) | 15:59 |
| *** dizquierdo has quit IRC | 15:59 | |
| trinaths | fungi: Are there any plugins other than SCP to cpature this information | 15:59 |
| fungi | trinaths: the ftp publisher in jenkins might also have a | 16:00 |
| trinaths | fungi: s/cpature/capture | 16:00 |
| fungi | "copy console log" check box | 16:00 |
| fungi | not sure | 16:00 |
| *** signed8bit_ZZZzz is now known as signed8bit | 16:01 | |
| trinaths | fungi: doubt, publisher plugins get invoked after the jenkins build is completed right? | 16:01 |
| fungi | trinaths: correct | 16:01 |
| trinaths | fungi: thanks . will try FTP publisher plugin | 16:03 |
| *** hemna_ is now known as hemna | 16:04 | |
| *** mmaglana has quit IRC | 16:05 | |
| fungi | trinaths: i just checked, and the version of the ftp publisher plugin we're using (1.7) doesn't seem to have an option for the console log | 16:05 |
| *** mmaglana has joined #openstack-infra | 16:05 | |
| fungi | it would probably need a similar patch | 16:05 |
| *** signed8bit has quit IRC | 16:08 | |
| *** dkranz has joined #openstack-infra | 16:08 | |
| *** homeless has quit IRC | 16:08 | |
| trinaths | fungi: an comments on https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+FTP+Plugin | 16:08 |
| *** IvanBerezovskiy1 has left #openstack-infra | 16:09 | |
| trinaths | fungi: s/an/any | 16:09 |
| *** yolanda has joined #openstack-infra | 16:10 | |
| *** mmaglana has quit IRC | 16:10 | |
| *** afazekas has quit IRC | 16:10 | |
| fungi | trinaths: looks slightly newer than the one we're using, but i don't see their changelog mention adding a feature for it | 16:10 |
| yolanda | clarkb , mordred, or anyone who knows... where are jeepyb errors logged? trying to debug an issue but i cannot find any jeepyb logs | 16:11 |
| fungi | yolanda: to stderr of the script loading it | 16:11 |
| *** thomasbiege has joined #openstack-infra | 16:11 | |
| trinaths | fungi: then, how to get hold of these console.html.gz logs... | 16:12 |
| fungi | yolanda: i believe they're just streamed, so they would ultimately be logged wherever that's redirected to | 16:12 |
| zaro | morning | 16:12 |
| fungi | trinaths: you probably can't from the current version of the ftp publisher plugin... it would likely need a patch to add a new feature allowing it to extract the console log from memory on the master | 16:12 |
| *** sweston has joined #openstack-infra | 16:13 | |
| *** homeless has joined #openstack-infra | 16:13 | |
| *** mriedem has quit IRC | 16:13 | |
| zaro | yeah, i think scp plugin gets data from the jenkins logger. | 16:13 |
| fungi | trinaths: or you could separately identify the url to the raw copy of the console log served from the master's web interface and use something like wget or curl to pull it | 16:13 |
| *** reed has joined #openstack-infra | 16:14 | |
| *** basha has quit IRC | 16:14 | |
| trinaths | fungi: okay. then, in jenkins, how to get the ZUUL-CHANGE and ZUUL-PATCHSET env vars. | 16:14 |
| trinaths | fungi: to publish these to FTP server, I need to give a generic path to a specific build. Will Jenkins know about the ZUUL variables. | 16:15 |
| yolanda | fungi, so if that's triggered from puppet agent, it should be logged into puppet? | 16:17 |
| fungi | trinaths: ZUUL_CHANGE and ZUUL_PATCHSET are provided from zuul to jenkins as job parameters and also reflected in the job's shell environment as variables | 16:17 |
| fungi | yolanda: yes, probably logged to syslog | 16:17 |
| yolanda | thx fungi | 16:18 |
| fungi | yolanda: or wherever your puppet agent is logging detailed output (which may require an additional exec option in the manifest to turn that on) | 16:18 |
| yolanda | i think that is just outputting to syslog | 16:19 |
| *** jistr has quit IRC | 16:19 | |
| zaro | trinaths: here's how the scp gets the full jenkins log https://github.com/jenkinsci/scp-plugin/blob/master/src/main/java/be/certipost/hudson/plugin/SCPRepositoryPublisher.java#L144 | 16:19 |
| *** fbo is now known as fbo_away | 16:20 | |
| zaro | trinaths: only available on master, not in the released version. | 16:20 |
| *** chianingwang has joined #openstack-infra | 16:21 | |
| *** derekh_ has quit IRC | 16:21 | |
| *** mriedem has joined #openstack-infra | 16:21 | |
| *** sweston has quit IRC | 16:21 | |
| *** esker has joined #openstack-infra | 16:22 | |
| openstackgerrit | Chris Wright proposed a change to openstack-infra/config: Adding #openstack-nfv channel https://review.openstack.org/99421 | 16:22 |
| *** jcoufal has quit IRC | 16:23 | |
| trinaths | zaro: "not in the released version" ? | 16:24 |
| fungi | yolanda: you may need to add "logoutput=>true" in the relevant exec in your puppet manifest too if it's not there already | 16:25 |
| zaro | trinaths: that is correct. maintainer has not released. we are using a build from master. | 16:25 |
| *** alkari has quit IRC | 16:25 | |
| trinaths | zaro: Is this functionality available with FTP plugin too? | 16:26 |
| zaro | trinaths: i don't know. fungi says he just checked and according to him it's not. | 16:28 |
| yolanda | fungi, thx, we may need to add that | 16:28 |
| *** e0ne has joined #openstack-infra | 16:28 | |
| fungi | zaro: trinaths: i couldn't find an option for it in the version of the ftp publisher plugin we have installed, and there's no mention of it getting added later in their changelog either | 16:28 |
| trinaths | fungi: ok | 16:29 |
| *** habib_ has quit IRC | 16:29 | |
| trinaths | zaro: can you kindly guide me on how to install this SCP plugin to jenkins. | 16:29 |
| *** sarob has joined #openstack-infra | 16:29 | |
| *** e0ne_ has quit IRC | 16:32 | |
| *** atiwari has joined #openstack-infra | 16:33 | |
| zaro | trinaths: you can just download the release and install. let me get you a link. | 16:33 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/git-review: Set StrictHostKeyChecking=no in integration tests https://review.openstack.org/99425 | 16:33 |
| *** sarob has quit IRC | 16:33 | |
| *** markmcclain has joined #openstack-infra | 16:35 | |
| *** mmaglana has joined #openstack-infra | 16:35 | |
| trinaths | zaro: thanks. please give me the help link. will try it | 16:36 |
| *** e0ne has quit IRC | 16:37 | |
| fungi | zaro: though it sounds like trinaths is going to want the jpi for tip of master, since console log publishing still isn't supported yet in the most recent release | 16:37 |
| *** sarob has joined #openstack-infra | 16:37 | |
| zaro | trinaths: i couldn't find a build of it on the jenkins-ci.org repo. do you know how to build jenkins plugins? | 16:38 |
| *** sarob_ has joined #openstack-infra | 16:40 | |
| *** yfried__ has joined #openstack-infra | 16:41 | |
| *** sarob has quit IRC | 16:41 | |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Added a puppet module for livegrep. https://review.openstack.org/96877 | 16:41 |
| *** talluri has quit IRC | 16:41 | |
| *** thomasbiege has quit IRC | 16:41 | |
| *** _nadya_ has quit IRC | 16:42 | |
| *** james_li has quit IRC | 16:42 | |
| trinaths | zaro: I'm new to jenkins. CI introduced me to Jenkins. need help on this. | 16:44 |
| *** marun has quit IRC | 16:45 | |
| zaro | trinaths: do you know java? | 16:45 |
| *** alkari has joined #openstack-infra | 16:45 | |
| trinaths | zaro: no. | 16:46 |
| *** Hal_ has quit IRC | 16:47 | |
| trinaths | zaro: give me some guidelines, I can do with help from google. | 16:47 |
| mordred | fungi: how's things? | 16:48 |
| *** gokrokve_ has quit IRC | 16:48 | |
| fungi | mordred: opening rackspace ticket now to get to the bottom of the quota issue | 16:48 |
| mordred | fungi: phschwartz can't just wave a magic wand? | 16:49 |
| fungi | mordred: i don't think he's around at the moment, so i'm trying not to abuse him unduly ;) | 16:49 |
| fungi | at least i'm able to get images to update in iad now, with the nodepool config change in place | 16:50 |
| *** jlibosva has quit IRC | 16:50 | |
| mordred | fungi: excellent! | 16:50 |
| *** thomasbiege has joined #openstack-infra | 16:50 | |
| *** james_li has joined #openstack-infra | 16:51 | |
| *** rcarrillocruz has left #openstack-infra | 16:51 | |
| *** arnaud has joined #openstack-infra | 16:51 | |
| *** thomasbiege has left #openstack-infra | 16:51 | |
| trinaths | zaro: any links for me.. | 16:51 |
| *** _nadya_ has joined #openstack-infra | 16:52 | |
| *** freyes has quit IRC | 16:52 | |
| fungi | trinaths: https://wiki.jenkins-ci.org/display/JENKINS/Plugin+tutorial#Plugintutorial-BuildingaPlugin | 16:53 |
| *** _nadya_ has quit IRC | 16:53 | |
| zaro | trinaths: ok, finally found a build from master https://jenkins.ci.cloudbees.com/job/plugins/job/scp-plugin/11/org.jenkins-ci.plugins$scp/artifact/org.jenkins-ci.plugins/scp/1.9-SNAPSHOT/scp-1.9-SNAPSHOT.hpi | 16:53 |
| fungi | oh, that was lucky | 16:53 |
| *** harlowja_away is now known as harlowja | 16:53 | |
| trinaths | zaro, fungi: thanks for the link. | 16:54 |
| zaro | just download that and install per directions from google. | 16:54 |
| fungi | i didn't even know they autobuilt those | 16:54 |
| *** yfried__ has quit IRC | 16:54 | |
| zaro | yeah, hard as heck to find though. | 16:55 |
| *** arnaud has quit IRC | 16:55 | |
| *** arnaud has joined #openstack-infra | 16:55 | |
| fungi | so it seems | 16:55 |
| clarkb | o/ | 16:56 |
| *** praneshp has joined #openstack-infra | 16:57 | |
| clarkb | trinaths: fungi: you can grab it like the swift upload script grabs it | 16:57 |
| clarkb | but the file won't be complete | 16:57 |
| *** praneshp_ has joined #openstack-infra | 16:57 | |
| *** viktors is now known as viktors|afk | 16:58 | |
| *** basha has joined #openstack-infra | 16:58 | |
| *** basha has quit IRC | 16:58 | |
| trinaths | clarkb: "file won't be complete" ? | 16:58 |
| *** doude has joined #openstack-infra | 16:59 | |
| *** Sukhdev has quit IRC | 16:59 | |
| clarkb | trinaths: yes it won't contain the entire log as it is collected in the job itself | 16:59 |
| fungi | trinaths: if you retrieve the console log as part of the job, you'll be grabbing a log which doesn't include the completion of the job because the job is still running | 16:59 |
| clarkb | nodepool limit change lgtm assuming those numbers were calculated based on feedback from quotas and or phschwartz ? | 17:00 |
| *** Sukhdev has joined #openstack-infra | 17:00 | |
| openstackgerrit | Christian Berendt proposed a change to openstack/requirements: Add tox as global requirement https://review.openstack.org/98575 | 17:00 |
| fungi | clarkb: nova absolute-limits | 17:00 |
| *** asadough1 is now known as asadoughi | 17:01 | |
| trinaths | fungi: true, but can this is used as console.html still? | 17:01 |
| openstackgerrit | Dirk Mueller proposed a change to openstack/requirements: Require kombu 2.5.0 or newer for switch to py-amqp https://review.openstack.org/92095 | 17:01 |
| *** thomasbiege has joined #openstack-infra | 17:01 | |
| fungi | trinaths: yes | 17:01 |
| bodepd | could I get some review love on: https://review.openstack.org/#/c/97357/ and https://review.openstack.org/#/c/97647/ | 17:01 |
| fungi | trinaths: though it won't have the actual jenkins success or failure result in it | 17:01 |
| *** esker has quit IRC | 17:01 | |
| *** praneshp has quit IRC | 17:01 | |
| *** praneshp_ is now known as praneshp | 17:01 | |
| fungi | clarkb: for iad it reports maxTotalInstances=192 but maxTotalRAMSize=768000 | 17:02 |
| trinaths | fungi: oh!, is there any other way for capturing console.html | 17:02 |
| *** esker has joined #openstack-infra | 17:02 | |
| *** yfried__ has joined #openstack-infra | 17:02 | |
| clarkb | fungi: gotcha. It is a bit annoying that quotas are so independent | 17:02 |
| clarkb | I suppose it is more flexible that way | 17:02 |
| fungi | trinaths: yes, having a jenkins publisher plugin retrieve it from memory as the job is completing. this is why we wrote a patch to the scp publisher plugin to do that | 17:03 |
| fungi | clarkb: i've opened ticket 140611-ord-0003741 under our openstackjenkins account to ask for it to be restored | 17:04 |
| clarkb | fungi: sdague: I haven't booted enough of my brain yet to know for sure but I don't think the ES cluster is under strain. We might getter slightly better performance out of it by adding more nodes but I wouldn't expect that ot be our miracle | 17:05 |
| clarkb | I really think it is the logstash workers that are struggling and disabling crm114 may help | 17:05 |
| *** atiwari has quit IRC | 17:05 | |
| fungi | clarkb: sounds reasonable to me | 17:05 |
| clarkb | adding worker nodes did seem to get us in a "stable yet still backlogged" state | 17:05 |
| clarkb | whereas before we were swinging wildly | 17:06 |
| pleia2 | if anyone has thoughts for other updates to /overview I'd aprpeciate it (I'm giving it for the first time next Wednesday - as a keynote!): https://review.openstack.org/#/c/99173/ | 17:06 |
| fungi | yay! i just got a bare-precise image to update in rax-iad finally | 17:06 |
| clarkb | pleia2: adding it to the list | 17:06 |
| *** esker has quit IRC | 17:06 | |
| pleia2 | thanks | 17:06 |
| clarkb | also binary diffing of images makes reviewing those changes so much better | 17:06 |
| clarkb | fungi: woot | 17:06 |
| pleia2 | yeah :) I love the new image diffs | 17:06 |
| pleia2 | (thanks zaro!) | 17:07 |
| clarkb | lgtm | 17:08 |
| fungi | pleia2: oh, btw i just saw your name on the speakers list for ato. congrats! | 17:08 |
| *** markwash has joined #openstack-infra | 17:09 | |
| *** esker has joined #openstack-infra | 17:09 | |
| pleia2 | fungi: thanks :) | 17:09 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for heat db transaction error bug 1328983 https://review.openstack.org/99429 | 17:09 |
| uvirtbot | Launchpad bug 1328983 in heat "heat engine db transaction error: InvalidRequestError: This session is in 'committed' state; no further SQL can be emitted within this transaction." [Undecided,New] https://launchpad.net/bugs/1328983 | 17:09 |
| *** zzelle_ has joined #openstack-infra | 17:09 | |
| *** thedodd has joined #openstack-infra | 17:10 | |
| *** dprince has quit IRC | 17:10 | |
| *** Ryan_Lane has joined #openstack-infra | 17:10 | |
| clarkb | fungi: sdague: I am going to disable crm114 on logstash-worker20 and see how that affects its throughput | 17:11 |
| clarkb | the cacti graphs should show us if it helps | 17:11 |
| mordred | ilyashakhat: https://review.openstack.org/99430 <-- this stops stackalytics from talking to github and instead gets the project list from gerrit ... I need to add a config value to it for git repo prefix | 17:11 |
| fungi | clarkb: worth a try, agreed | 17:11 |
| *** ildikov has quit IRC | 17:11 | |
| mordred | clarkb: ++ | 17:12 |
| *** chianingwang has quit IRC | 17:12 | |
| *** basha has joined #openstack-infra | 17:12 | |
| *** chianingwang has joined #openstack-infra | 17:13 | |
| *** dguerri has quit IRC | 17:14 | |
| clarkb | I chose that node because it seemed to have hung on 6 hour old crm114 processes :( | 17:14 |
| *** amcrn has joined #openstack-infra | 17:14 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1313655 https://review.openstack.org/99267 | 17:14 |
| uvirtbot | Launchpad bug 1313655 in nova "tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest.test_run_idempotent_instances[gate,smoke] fails" [High,Confirmed] https://launchpad.net/bugs/1313655 | 17:14 |
| *** SumitNaiksatam has joined #openstack-infra | 17:15 | |
| mordred | ilyashakhat: ok. I added the config. that was easy actually | 17:16 |
| fungi | clarkb: eek | 17:17 |
| *** zns__ has quit IRC | 17:18 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Run stackalytics in infra https://review.openstack.org/98656 | 17:18 |
| morganfainberg | for those paying attention to the token discussions yesterday w/ sdague and logging: https://review.openstack.org/#/c/99432/ the goal being don't expose secure data in logs... ever (password or otherwise) | 17:18 |
| *** basha has quit IRC | 17:19 | |
| *** salv-orlando has joined #openstack-infra | 17:19 | |
| mordred | morganfainberg: I'm in favor of that | 17:19 |
| zaro | pleia2: np i guees, i didn't even know that was a feature :) see you on fri @ TLXF | 17:19 |
| *** gokrokve has joined #openstack-infra | 17:19 | |
| morganfainberg | mordred, the more i think about it the more i dislike any form of priviledged auth data (allowing actions to be taken) from hitting logs. | 17:20 |
| pleia2 | zaro: yes! see you then | 17:20 |
| fungi | morganfainberg: yes, i couldn't agree more | 17:20 |
| morganfainberg | mordred, i believe we will provide a hash(sha224?) of token used for actions in CADF notifications as well in the future | 17:20 |
| pleia2 | I fly out tomorrow morning, need to pack | 17:20 |
| morganfainberg | mordred, fungi, that patch will address things in the keystoneclient sessions - and we are working to move everyone to using the ksc session object instead of their own impl | 17:22 |
| mordred | morganfainberg: I'm also in favor of that | 17:23 |
| *** _nadya_ has joined #openstack-infra | 17:23 | |
| fungi | that would be great | 17:23 |
| clarkb | ++ | 17:23 |
| morganfainberg | mordred, next is to coordinate w/ jamielennox and start forcing other teams to use the session object :) | 17:23 |
| morganfainberg | i think we already have patches out there, but it's a ton of code. | 17:24 |
| morganfainberg | jamielennox has been kicking serious ass at getting the code proposed. | 17:24 |
| *** salv-orlando has quit IRC | 17:24 | |
| morganfainberg | pleia2, last minute packing eh? | 17:25 |
| *** rcarrillocruz has joined #openstack-infra | 17:26 | |
| clarkb | fungi: sdague: 33GB of disk left on elasticsearch05's volume | 17:26 |
| *** salv-orlando has joined #openstack-infra | 17:26 | |
| rcarrillocruz | guys, what do you think about adding to jeepyb integration to other issue trackers? | 17:27 |
| rcarrillocruz | as is now, it's just launchpad | 17:27 |
| rcarrillocruz | i think we could refactor the code for storyboard in the future or other issue trackers | 17:28 |
| rcarrillocruz | thoughts? | 17:28 |
| jogo | so I think https://review.openstack.org/#/c/99212/ may help get the log files down | 17:28 |
| jogo | and help elasticsearch | 17:28 |
| jogo | clarkb: ^ | 17:28 |
| *** radez is now known as radez_g0n3 | 17:28 | |
| fungi | rcarrillocruz: i'm quite positive we do want to ti start updating storyboard, so that seems necessary | 17:29 |
| mordred | ++ | 17:29 |
| rcarrillocruz | good, i started to look at how to do it for jira, but the idea is doing it issue tracker aware | 17:29 |
| rcarrillocruz | i will work on it | 17:29 |
| rcarrillocruz | thx | 17:29 |
| * morganfainberg is _really_ _really_ looking forward to storyboard. | 17:30 | |
| clarkb | fungi: sdague: so I think that is impacting our ability to index things | 17:30 |
| *** praneshp has quit IRC | 17:30 | |
| clarkb | fungi: sdague: the quick solution is to kill backlog | 17:30 |
| clarkb | fungi: sdague: and mumble about how we somehow seem to have doubled our log output while not running any additional tests | 17:30 |
| rcarrillocruz | do you guys know if storyboard support oauth token authentication? or at least if there are plans to do so... | 17:30 |
| morganfainberg | as a thought, have we considered the ability to tag (maybe) 3rd Party bug systems (e.g. 3rd party CI testing) w/ results? (cc rcarrillocruz) | 17:31 |
| fungi | clarkb: long term solutions? larger es cluster, larger local volumes on existing cluster members, or less retention? | 17:31 |
| morganfainberg | rcarrillocruz or is that far outside the scope of what you're looking to do at the moment? | 17:31 |
| rcarrillocruz | morganfainberg: i am looking at the moment to integrate it with jira | 17:31 |
| fungi | or hit people over the head to trim unneeded log volume, true | 17:31 |
| morganfainberg | rcarrillocruz, ah very cool | 17:31 |
| jogo | clarkb fungishort term bump https://review.openstack.org/#/c/99212/ up? | 17:31 |
| rcarrillocruz | so i'm open to be agnostic about issue trackers | 17:31 |
| clarkb | fungi: I think larger volumes will get us past this. I am a bit concerned that we don't seem to fully grok that anyone deploying these clouds will run into the same problems | 17:31 |
| jogo | fungi: ^ | 17:31 |
| morganfainberg | rcarrillocruz, like it! | 17:31 |
| clarkb | fungi: if I have to double my storage cluster capacity every time I upgrade openstack I am not a happy openstack user | 17:32 |
| jesusaurus | clarkb: ++ | 17:32 |
| clarkb | fungi: any chance you know off the top of your head where we are on volume quota for dfw? | 17:32 |
| clarkb | fungi: if we add capcity there we should probably add a second 1TB volume to each node which is 6TB | 17:33 |
| *** salv-orlando has quit IRC | 17:33 | |
| trinaths | fungi: how to reference, zuul params in jenkins. I have $BUILD_NUMBER but nothing with ZUUL | 17:33 |
| phschwartz | mordred: fungi: clarkb: Are you hitting a large number of errors in one of our regions again? | 17:33 |
| fungi | jogo: approved several hours ago, but yeah it's probably 10 hours out in the gate | 17:33 |
| fungi | clarkb: i'll have to look, but i think we still have a couple tb open | 17:33 |
| clarkb | rcarrillocruz: fungi mordred my only concern with say jira would be there is no way for us to test that | 17:34 |
| fungi | phschwartz: somehow our ram quota got lowered in iad. i've opened ticket 140611-ord-0003741 about it | 17:34 |
| jogo | fungi: wow gate is 13 hours | 17:34 |
| phschwartz | fungi: Let me take a look at the tenant and see if there is an issue, weird that it would have been cut | 17:35 |
| fungi | clarkb: rcarrillocruz: it would need to be pluggable i think. ideally the bug/bp update hook scripts we have probably need a complete rewrite anyway | 17:35 |
| jesusaurus | clarkb: so it would require 3rd party testing of the jira bits | 17:35 |
| mordred_phone | clarkb: yah. if we can sort out proper APIs and unit tests, we should be able to treat that similar to jjb | 17:35 |
| jogo | ahh we are at quota | 17:36 |
| clarkb | mordred_phone: possibly, but jjb isn't really mocking APIs just data | 17:36 |
| mordred_phone | we don't explicitly test the contributed modules there functionally ... but we do test interface | 17:36 |
| clarkb | which may be good enough | 17:36 |
| *** salv-orlando has joined #openstack-infra | 17:36 | |
| fungi | phschwartz: i believe it was previously at 1.5tb (enough to boot all 192 instances of our quota on an 8gb flavor) and now it's set to just under 0.75tb | 17:37 |
| *** Sukhdev has quit IRC | 17:37 | |
| *** salv-orlando has quit IRC | 17:37 | |
| *** atiwari has joined #openstack-infra | 17:38 | |
| *** UtahDave has quit IRC | 17:38 | |
| *** zehicle_at_dell has joined #openstack-infra | 17:38 | |
| jerryz | trinaths:http://paste.openstack.org/show/83674/ | 17:38 |
| *** salv-orlando has joined #openstack-infra | 17:38 | |
| clarkb | fungi: sdague: what do you think about reducing index retention by a couple days to give us some breathing room while we sort this out? | 17:38 |
| clarkb | jogo: ^ | 17:39 |
| trinaths | jerryz: thanks. it helps. | 17:39 |
| *** salv-orlando has quit IRC | 17:39 | |
| *** andreykurilin_ has joined #openstack-infra | 17:39 | |
| fungi | clarkb: i'm okay with that, but i'm not a primary consumer of the data so i'll defer to the qa peeps | 17:39 |
| *** dprince has joined #openstack-infra | 17:40 | |
| Alex_Gaynor | For hacking, are H404 and H405 mutually exclusive, am I supposed to pick one and disable the other? | 17:41 |
| fungi | trinaths: yes, those parameters are passed from your zuul server via gearman as part of the job request, and should appear as parameters for that job run in jenkins (and as environment variables on the worker for the running job session) | 17:41 |
| *** radez_g0n3 is now known as radez | 17:41 | |
| *** HenryG has quit IRC | 17:42 | |
| morganfainberg | Alex_Gaynor, that seems silly if that is the case | 17:42 |
| fungi | Alex_Gaynor: i think jogo's recommended solution is to bifurcate your reality into parallel universes, one where you chose to enforce H404 and the other for H405 | 17:42 |
| Alex_Gaynor | morganfainberg: I agree, but I want to know if I'm understanding correctly :-) | 17:42 |
| Alex_Gaynor | fungi: All I'm saying is, I've got a very long blog post I need to publish about how dumb fork(2) is | 17:42 |
| fungi | Alex_Gaynor: well, you're not going to get any disagreement from me on that | 17:43 |
| Alex_Gaynor | :P | 17:43 |
| fungi | Alex_Gaynor: it translates well from c, but that's so unpythonic | 17:44 |
| clarkb | wait what | 17:44 |
| clarkb | fork is most excellent thing ever | 17:44 |
| zaro | trinaths: here's an example http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/openstack-publish-jobs.yaml#n30 | 17:44 |
| *** _nadya_ has quit IRC | 17:44 | |
| clarkb | seriously what do you do on windows? | 17:44 |
| Alex_Gaynor | posix_spawn! | 17:44 |
| clarkb | fork it is simple and relatively cheap (today) | 17:44 |
| fungi | clarkb: reformat and replace | 17:44 |
| *** gary-smith has joined #openstack-infra | 17:44 | |
| *** _nadya_ has joined #openstack-infra | 17:45 | |
| clarkb | and forkbombs are fun | 17:45 |
| fungi | now *that* is definitely true | 17:45 |
| *** salv-orlando has joined #openstack-infra | 17:45 | |
| morganfainberg | Alex_Gaynor an no h404 means don't do """<newline>summary | 17:46 |
| morganfainberg | Alex_Gaynor, h405 means """<summary><empty line><more string>""" | 17:46 |
| *** salv-orlando has quit IRC | 17:46 | |
| clarkb | seriously though I want new process, I want to have communication to new process and hand off sockets and not loose all of my cache | 17:46 |
| clarkb | what do I do? fork! | 17:47 |
| Alex_Gaynor | morganfainberg: So, what's the right way to write: | 17:47 |
| Alex_Gaynor | https://www.irccloud.com/pastebin/NcXqCwv1 | 17:47 |
| morganfainberg | Alex_Gaynor, Alex_Gaynor , http://pasteraw.com/70lqbzw0hqbrezwfzljdt2cwpafs1lx | 17:47 |
| fungi | Alex_Gaynor: """<summary>""" i would assume | 17:48 |
| morganfainberg | Alex_Gaynor, it's not > 79 chars that way, and it's not multi-line | 17:48 |
| Alex_Gaynor | ok, that makes sense, after a fashion, how about: | 17:48 |
| Alex_Gaynor | https://www.irccloud.com/pastebin/M1i7VvxV | 17:48 |
| morganfainberg | move the first line onto the same line as the opening """ | 17:48 |
| *** salv-orlando has joined #openstack-infra | 17:48 | |
| clarkb | Alex_Gaynor: fwiw we ignore H* in infra land | 17:48 |
| morganfainberg | http://pasteraw.com/sv8cybmo1odxjidg7ecsory76ch1p0q | 17:48 |
| * Alex_Gaynor is missing "go fmt" really badly now | 17:49 | |
| Alex_Gaynor | morganfainberg: thanks | 17:49 |
| morganfainberg | Alex_Gaynor, no problem :) | 17:49 |
| *** thuc has quit IRC | 17:49 | |
| *** salv-orlando has quit IRC | 17:49 | |
| *** thuc has joined #openstack-infra | 17:49 | |
| gary-smith | I've noticed Jenkins kicking off jobs in horizon in response to comments being added to a change (e.g. no new patchset or rebase). Is this a known issue? I didn't see any open bug for it | 17:50 |
| trinaths | fungi: thanks. checking them now. | 17:50 |
| *** _nadya_ has quit IRC | 17:50 | |
| *** ildikov has joined #openstack-infra | 17:50 | |
| sdague | clarkb: so pull down to 10 days instead of 14? | 17:50 |
| morganfainberg | gary-smith, intentional, it is checking that the Verified isn't stale | 17:50 |
| sdague | I'm game with that | 17:50 |
| clarkb | sdague: sure | 17:50 |
| morganfainberg | gary-smith, so if the last run was >24hrs (i think) re-run the check job. | 17:50 |
| sdague | if you think it will get us to recover | 17:50 |
| clarkb | sdague: ok let me start there and work on a plan to bump volume totals | 17:50 |
| gary-smith | morganfainberg: thanks for the clarification. that helps | 17:50 |
| fungi | gary-smith: it does that if the last time it checked has been more than 73 hours | 17:50 |
| clarkb | sdague: I think it will releive some of the pressure if not all of it | 17:51 |
| sdague | does dropping crm114 give us anything? | 17:51 |
| sdague | because we're not using that data at all | 17:51 |
| morganfainberg | gary-smith, ^ what fungi said, 73hrs | 17:51 |
| clarkb | sdague: so far no | 17:51 |
| fungi | er, i meant 72 but fingerfail | 17:51 |
| Alex_Gaynor | morganfainberg: is http://bpaste.net/show/j79qlavtON0uys3m24M6/ a bug? running that code totally gives me a module object | 17:51 |
| morganfainberg | fungi, i learn something every day | 17:51 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1328997 https://review.openstack.org/99438 | 17:51 |
| uvirtbot | Launchpad bug 1328997 in nova "Unit test failure: openstack_citest" is being accessed by other users\nDETAIL: There are 1 other session(s) using the database." [Critical,Confirmed] https://launchpad.net/bugs/1328997 | 17:51 |
| clarkb | sdague: I think the issue is further downstream. Either fs capacity or logstash itself | 17:51 |
| sdague | clarkb: ok, cool | 17:51 |
| clarkb | let me kill a few indexes by hand right now and that will remove one of the variables | 17:51 |
| morganfainberg | Alex_Gaynor, huh | 17:51 |
| fungi | gary-smith: so that reviewers will have relatively up-to-date test results about the code they're reviewing | 17:51 |
| sdague | clarkb: cool, works for me | 17:52 |
| clarkb | sdague: see http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1754&rra_id=all for crm114 numbers | 17:52 |
| morganfainberg | Alex_Gaynor, that might need a #noqa after it | 17:52 |
| Alex_Gaynor | morganfainberg: I guess I'll file a bug, launchpad I assume? | 17:52 |
| clarkb | there is a gap inbetween ignore that, but throughput remains the same pre and post crm114 disablement | 17:52 |
| morganfainberg | Alex_Gaynor, in theory you could just from six import moves then use urlllib.parse.<method> | 17:52 |
| gary-smith | fungi: but if nothing has changed, why does it need to reverify? Isn't it just doing the same build over again? | 17:52 |
| morganfainberg | Alex_Gaynor, It may not _actually_ be a module, six.moves is weird. | 17:52 |
| Alex_Gaynor | morganfainberg: it's totally a module! or at least acts like one :P | 17:52 |
| fungi | gary-smith: you know for certain that nothing in horizon or in the rest of openstack has changed? | 17:52 |
| jogo | clarkb: dropping index retention concerns me a little but, but if its just a few days ok. | 17:52 |
| clarkb | jogo: what is the concern? | 17:53 |
| fungi | gary-smith: openstack is constantly changing. just because that patch worked three days ago doesn't mean it works now | 17:53 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup a private gerrit instance for security reviews https://review.openstack.org/47937 | 17:53 |
| jogo | Alex_Gaynor: H405, H404 are mutually exclusive? | 17:53 |
| *** salv-orlando has joined #openstack-infra | 17:53 | |
| *** salv-orlando has quit IRC | 17:53 | |
| *** thuc has quit IRC | 17:53 | |
| Alex_Gaynor | jogo: morganfainberg explained that to me, it's super confusing though (IMO) | 17:54 |
| jogo | clarkb: having graphs going back in time by more then just a few days helps us see trends better especially if there are spikes | 17:54 |
| jogo | but I rarely look back past 10 days or so | 17:54 |
| clarkb | jogo: ok so 10 days is safe | 17:54 |
| * clarkb does this | 17:55 | |
| jogo | clarkb: so it sounds like a resonable step. just a little disappointing | 17:55 |
| fungi | jogo: it might be better to start enforcing the pep252 docstring checker instead | 17:55 |
| jogo | fungi: AFAIK I looked into that but and forgot why we didn't go with it, think it too much | 17:55 |
| jogo | morganfainberg: ? | 17:55 |
| morganfainberg | jogo, H404 is don't do """<newline>summary | 17:56 |
| gary-smith | fungi: for horizon, the unit tests are generally performed entirely without the rest of openstack, so it doesn't matter if the rest of openstack changes | 17:56 |
| clarkb | jogo: I think one of the things we should keep in mind is that we aren't really running more tests (we are capped by provider quotas) so this increase is purely in what we are logging | 17:56 |
| morganfainberg | jogo, instead do """<summary> | 17:56 |
| *** thuc has joined #openstack-infra | 17:56 | |
| clarkb | jogo: now logging and indexing more stuff is beneficial in many cases | 17:56 |
| morganfainberg | jogo, h405 is making sure there is a newline after the summary before the rest of the docstring | 17:56 |
| *** thedodd has quit IRC | 17:56 | |
| clarkb | but if this leaks to deployments there will be sad people | 17:56 |
| gary-smith | fungi: but otherwise, I do agree with the rationale | 17:56 |
| jogo | clarkb: sigh | 17:56 |
| fungi | gary-smith: true, but horizon may have changed. also the integration tests do test it with the rest of openstack. separating out which jobs to re-run and which not to re-run is a non-trivial optimization | 17:56 |
| jogo | morganfainberg: and? | 17:57 |
| *** thuc has quit IRC | 17:57 | |
| gary-smith | fungi: yah, makes sense | 17:57 |
| *** praneshp has joined #openstack-infra | 17:57 | |
| jogo | morganfainberg: I think I am missing something | 17:57 |
| *** thuc has joined #openstack-infra | 17:57 | |
| *** markwash has quit IRC | 17:58 | |
| *** thuc has quit IRC | 17:58 | |
| *** thedodd has joined #openstack-infra | 17:58 | |
| fungi | gary-smith: before we started doing this, core reviewers for projects were often approving several-months-old patchsets which had no way of ever even hoping to work in the current state of the project | 17:58 |
| *** thuc has joined #openstack-infra | 17:58 | |
| *** denis_makogon has joined #openstack-infra | 17:58 | |
| *** markwash has joined #openstack-infra | 17:59 | |
| jogo | morganfainberg: I don't see how H404 and H405 break eachother | 17:59 |
| morganfainberg | jogo, http://pasteraw.com/d77zhuog0ykvv9fcdrzfqdnwlrb3371 | 17:59 |
| morganfainberg | jogo, they don't | 18:00 |
| *** ramashri has quit IRC | 18:00 | |
| morganfainberg | jogo, i was explaining how they actually work | 18:00 |
| morganfainberg | it wasn't clear to Alex_Gaynor. | 18:00 |
| *** ramashri has joined #openstack-infra | 18:01 | |
| *** markmcclain has quit IRC | 18:01 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1328997 https://review.openstack.org/99438 | 18:01 |
| uvirtbot | Launchpad bug 1328997 in nova "Unit test failure: openstack_citest" is being accessed by other users\nDETAIL: There are 1 other session(s) using the database." [Critical,Confirmed] https://launchpad.net/bugs/1328997 | 18:01 |
| *** markwash_ has joined #openstack-infra | 18:01 | |
| *** ominakov has quit IRC | 18:01 | |
| *** markmcclain has joined #openstack-infra | 18:02 | |
| jogo | morganfainberg: ohhhh, so false alarm | 18:02 |
| morganfainberg | jogo, yep! :) | 18:02 |
| jogo | morganfainberg: I freak out if I smell a false positive in hacking | 18:02 |
| jogo | err false negative | 18:02 |
| morganfainberg | jogo, nah, no false <positive/netagive/brokeness> here we're good. | 18:03 |
| *** radez is now known as radez_g0n3 | 18:03 | |
| jogo | so based \o/ | 18:03 |
| jogo | \o/ | 18:03 |
| clarkb | jogo: in related news is hacking still taking post commit checks? | 18:03 |
| phschwartz | fungi: looks like iad and dfw changed iad is now .75 and dfw is .5. I am looking for notes as to why they might have changed. Will let you know | 18:03 |
| clarkb | jogo: because I really think it should stop | 18:03 |
| Alex_Gaynor | jogo: There is a false negative on the six thing though, I just filed a bug for that | 18:04 |
| jogo | clarkb: sadly yes. I agree with you | 18:04 |
| *** markwash has quit IRC | 18:04 | |
| *** markwash_ is now known as markwash | 18:04 | |
| jogo | Alex_Gaynor: link? | 18:04 |
| Alex_Gaynor | jogo: https://bugs.launchpad.net/hacking/+bug/1329002 | 18:04 |
| uvirtbot | Launchpad bug 1329002 in hacking "H302 fails to recognize something as a module" [Undecided,New] | 18:05 |
| clarkb | ok indexes removed. It is going to rebalance itself a bit now | 18:05 |
| *** richm has joined #openstack-infra | 18:06 | |
| *** signed8bit has joined #openstack-infra | 18:06 | |
| *** signed8bit has quit IRC | 18:07 | |
| *** nati_ueno has joined #openstack-infra | 18:07 | |
| rcarrillocruz | fungi: agreed. my idea is create a base class that exposes the contract to which all issue trackers should adhere to, and then have classes deriving from it implementing the issue tracker details (jira, storyboard, etc). i'd leave updatebug/bp scripts to handle argument parsing from gerrit and reading the conf files. In those conf files we could put which issue tracker we are using and load the pertinent class to handle it | 18:07 |
| *** signed8bit has joined #openstack-infra | 18:07 | |
| *** lcheng has quit IRC | 18:07 | |
| fungi | phschwartz: i think dfw has been that low for a while, but iad was reduced fairly recently and i don't recall getting a heads up | 18:07 |
| clarkb | rcarrillocruz: ++ | 18:07 |
| *** shardy is now known as shardy_afk | 18:08 | |
| *** trinaths has quit IRC | 18:09 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for heat db transaction error bug 1328983 https://review.openstack.org/99429 | 18:09 |
| uvirtbot | Launchpad bug 1328983 in heat "heat engine db transaction error: InvalidRequestError: This session is in 'committed' state; no further SQL can be emitted within this transaction." [Undecided,New] https://launchpad.net/bugs/1328983 | 18:09 |
| fungi | phschwartz: er, no, ord was lowish for ages, but dfw was 1.5tb when i looked a few minutes ago | 18:09 |
| * fungi checks again | 18:09 | |
| *** rcarrill` has joined #openstack-infra | 18:10 | |
| fungi | phschwartz: yeah, openstackjenkins in dfw still reports maxTotalRAMSize=1572864 | 18:10 |
| phschwartz | sorry, meant ord not dfw | 18:10 |
| jogo | Alex_Gaynor: I don't think its a bug | 18:10 |
| jogo | Alex_Gaynor: try from urllib import parse | 18:10 |
| *** salv-orlando has joined #openstack-infra | 18:11 | |
| *** salv-orlando has quit IRC | 18:11 | |
| Alex_Gaynor | jogo: That doesn't even exist on py2 | 18:11 |
| jogo | that is what from six.moves.urllib import parse, translates to | 18:11 |
| clarkb | of course the rebalancing seems to have killed our ability to index :( | 18:11 |
| jogo | Alex_Gaynor: ohh | 18:11 |
| fungi | phschwartz: then yeah, i think ord has been low because they didn't have sufficient capacity there, but dfw and iad were high enough we could boot 192*8gb | 18:11 |
| *** rcarrillocruz has quit IRC | 18:11 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1313655 https://review.openstack.org/99267 | 18:11 |
| uvirtbot | Launchpad bug 1313655 in nova "tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest.test_run_idempotent_instances[gate,smoke] fails" [High,Confirmed] https://launchpad.net/bugs/1313655 | 18:11 |
| clarkb | which may be part of the problem if killing old indexes makes things go crazy in the evenings | 18:11 |
| jogo | Alex_Gaynor: <module 'urllib' from '/usr/lib/python2.7/urllib.pyc'> | 18:11 |
| jogo | Alex_Gaynor: ahh | 18:11 |
| sdague | clarkb: yeh | 18:11 |
| *** zns has joined #openstack-infra | 18:12 | |
| jogo | Alex_Gaynor: what does the import translate too in six | 18:12 |
| jogo | err 2 | 18:12 |
| *** lcheng has joined #openstack-infra | 18:12 | |
| sdague | clarkb: right, I don't now how to make it better. I just know that if we don't, we're going to need to rethink using it as a backend. Because it's gotten to the point were we aren't reporting on reviews any more. | 18:13 |
| *** markmc has quit IRC | 18:13 | |
| clarkb | sdague: ya | 18:13 |
| clarkb | I can't help but think very often this is the one place where real hardware would help tremendously | 18:14 |
| *** esker has quit IRC | 18:15 | |
| *** tkelsey has quit IRC | 18:16 | |
| *** reed has quit IRC | 18:16 | |
| *** sarob_ has quit IRC | 18:17 | |
| jogo | Alex_Gaynor: are you familier with six? | 18:18 |
| *** sarob_ has joined #openstack-infra | 18:18 | |
| sdague | clarkb: yeh, do we have the ability to get any of that from rax? | 18:18 |
| clarkb | sdague: I don't think so | 18:18 |
| sdague | also, I think that a big part of our current fail rate is actually network blips | 18:19 |
| sdague | %http://status.openstack.org/elastic-recheck/gate.html top 2 are the same issue with networking | 18:19 |
| *** tkelsey has joined #openstack-infra | 18:20 | |
| sdague | and there are some pip networking errors buried | 18:20 |
| *** HenryG has joined #openstack-infra | 18:20 | |
| clarkb | sdague: the second one isn't a network blip | 18:20 |
| clarkb | sdague: or at least adam_g seemed to think it was due to weirdness in apt repos | 18:20 |
| sdague | well ES is silly right now | 18:21 |
| sdague | otherwise I'd look | 18:21 |
| *** andreykurilin_ has quit IRC | 18:21 | |
| sdague | my recollection is it was largely the same error | 18:21 |
| *** andreykurilin_ has joined #openstack-infra | 18:21 | |
| *** tkelsey has quit IRC | 18:21 | |
| adam_g | clarkb, maybe not the apt mirrors but the state of the image's apt cache on first boot. adding that 'apt-get update' seems to have helped | 18:22 |
| *** arnaud has quit IRC | 18:22 | |
| *** sarob_ has quit IRC | 18:23 | |
| jerryz | clarkb: are apt mirrors in o.o managed by puppet too? | 18:24 |
| clarkb | jerryz: no | 18:24 |
| clarkb | today we rely on our cloud providers | 18:24 |
| *** melwitt has joined #openstack-infra | 18:26 | |
| *** markmcclain has quit IRC | 18:27 | |
| *** tkelsey has joined #openstack-infra | 18:27 | |
| pleia2 | jerryz: there's some discussion here about creating one (and I have a WIP patch) but it's not moving much right now https://bugs.launchpad.net/openstack-ci/+bug/1311855 | 18:28 |
| uvirtbot | Launchpad bug 1311855 in openstack-ci "Create puppet module for Ubuntu repository mirror" [Undecided,In progress] | 18:28 |
| *** cody-somerville has joined #openstack-infra | 18:28 | |
| *** rcarrillocruz has joined #openstack-infra | 18:28 | |
| *** hashar has joined #openstack-infra | 18:28 | |
| pleia2 | turns out running a mirror that doesn't get out of sync at the velocity we run at is hard :) | 18:28 |
| sdague | pleia2: if we don't do --delete in rsync, we're good :) | 18:29 |
| sdague | just have lots of disk space | 18:29 |
| clarkb | sdague: :P | 18:29 |
| pleia2 | haha | 18:29 |
| *** rcarrill` has quit IRC | 18:29 | |
| *** shayneburgess has joined #openstack-infra | 18:30 | |
| sdague | it's the cloud! | 18:30 |
| *** gondoi is now known as zz_gondoi | 18:30 | |
| *** yolanda has quit IRC | 18:31 | |
| *** markmcclain has joined #openstack-infra | 18:31 | |
| morganfainberg | sdague, there are... limits to the cloud? :P | 18:32 |
| sdague | I built a mirror at home to speed up my NUC build. It's 800GB | 18:32 |
| sdague | but that's all the supported ubuntu versions | 18:32 |
| morganfainberg | sdague, and when the roll releases i heard the sync is brutal | 18:33 |
| sdague | yeh, probably | 18:33 |
| morganfainberg | sdague, because things get moved to "achive" or something | 18:33 |
| *** zns has quit IRC | 18:33 | |
| jogo | Alex_Gaynor: so hacking uses imp.find_module | 18:33 |
| jogo | Alex_Gaynor: and it looks like six doesn't work with that or some reason | 18:33 |
| morganfainberg | maybe thats debian specific, but something like that was brutal for a company a friend worked for | 18:33 |
| sdague | morganfainberg: well ubuntu package churn during alphas is immense | 18:34 |
| *** penguinRaider has quit IRC | 18:34 | |
| morganfainberg | sdague, ah yeah there we go. | 18:34 |
| sdague | but we could probably exclude that | 18:34 |
| morganfainberg | *nod* | 18:34 |
| sdague | because we definitely don't care | 18:34 |
| clarkb | I have never really felt the need for a personal mirror. But my internets at home have a reasonably good connection to psu | 18:34 |
| morganfainberg | hehe | 18:34 |
| morganfainberg | clarkb, if i had a server @home i'd mirror | 18:35 |
| sdague | clarkb: it speeds up the nuc lights out pxe install by about 10 minutes | 18:35 |
| morganfainberg | clarkb but that is because i get tired of waiting (even waiting on a 30-50+megabit connection get tiresome compared to gigabit) | 18:35 |
| *** pelix has quit IRC | 18:36 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add swift details to experimental-swift-logs.* https://review.openstack.org/99445 | 18:36 |
| *** sweston has joined #openstack-infra | 18:36 | |
| sdague | so it makes a functional system in 15 minutes instead of 25, which is kind of useful to me | 18:36 |
| *** otherwiseguy has quit IRC | 18:36 | |
| *** zns has joined #openstack-infra | 18:36 | |
| clarkb | fungi: ^ can haz review? that should get https://jenkins04.openstack.org/job/experimental-swift-logs-config-pep8/3/console past the failure there | 18:36 |
| clarkb | sdague: oh you redeploy often? | 18:37 |
| *** zns has quit IRC | 18:37 | |
| clarkb | that is probably what I don't need it. The number of times I start from scratch is few and far between | 18:37 |
| sdague | I've got it set up so if I pxe boot it, it reinstalls clean with puppet policy | 18:37 |
| *** zns has joined #openstack-infra | 18:37 | |
| morganfainberg | sdague, ooh. thats kinda fun. | 18:37 |
| sdague | I do then when I've crudded things up too much with grenade / devstack runs to figure out what's wrong | 18:38 |
| sdague | morganfainberg: more fun that it's vpro, so I can trigger a pxe boot remotely :) | 18:38 |
| sdague | so it's almost like having a big boy server at home | 18:38 |
| *** jraim has quit IRC | 18:39 | |
| *** torgomatic has quit IRC | 18:39 | |
| morganfainberg | ok thats really cool! | 18:39 |
| clarkb | sdague: ok ES is not rebalancing. we should see where it ends up in a few hours | 18:39 |
| sdague | clarkb: ok | 18:39 |
| * morganfainberg wants to setup some server-type machines @home... but i don't want to pay electricity for more than my laptop :P | 18:39 | |
| sdague | well, the nuc is about 10w | 18:39 |
| morganfainberg | or... well 2 laptops now. | 18:39 |
| morganfainberg | thats not bad at all | 18:39 |
| morganfainberg | i might need to do that | 18:40 |
| clarkb | yeah I have the htpc which is low powe rbut I use it for watching things :) | 18:40 |
| *** StevenK has quit IRC | 18:40 | |
| *** StevenK has joined #openstack-infra | 18:40 | |
| morganfainberg | wait... did that box play the intel jingle when you opened it? :P | 18:40 |
| *** jraim has joined #openstack-infra | 18:40 | |
| morganfainberg | (aka retail packaging) | 18:40 |
| sdague | honestly, even my big boxes run at about 60w tops. So they are a couple of lightbulbs | 18:40 |
| sdague | yes it did | 18:40 |
| sdague | it was kind of annoying | 18:40 |
| morganfainberg | we laughed pretty hard at metacloud when we opened the box. | 18:40 |
| clarkb | sdague: wow. I am nowhere near that on my big machine | 18:41 |
| clarkb | but I can use it to play civ V now so win | 18:41 |
| morganfainberg | clarkb, hehe | 18:41 |
| clarkb | 60W is pretty impressive | 18:41 |
| sdague | clarkb: that's what the ups tells me on the home server. It's an i7 ivy bridge | 18:41 |
| *** torgomatic has joined #openstack-infra | 18:41 | |
| clarkb | mine is sandy bridge but with a big gpu | 18:42 |
| morganfainberg | ah. | 18:42 |
| sdague | ah, yeh, no gpus | 18:42 |
| clarkb | I did go with a very efficient PSU though. It runs with the fan off most of the time | 18:42 |
| *** thuc has quit IRC | 18:44 | |
| *** thuc has joined #openstack-infra | 18:44 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for 1328500 https://review.openstack.org/99020 | 18:45 |
| clarkb | another thing I notice is we hit es01 much harder than the othe nodes because all of the queries go through it | 18:45 |
| clarkb | I wonder if I can make that better without much effort | 18:45 |
| notmyname | jeblair: here? | 18:47 |
| *** tkelsey has quit IRC | 18:47 | |
| clarkb | notmyname: I think he is still AFK | 18:47 |
| notmyname | ack | 18:47 |
| clarkb | should be back by next week aiui | 18:47 |
| sdague | clarkb: does the query node need to be a shard? | 18:47 |
| notmyname | FYI, since I am having surgery tomorrow morning, I have temporarily added Clay Gerrard (clayg) and Sam Merrirr (torgomatic) to the swift-ptl group in gerrit | 18:48 |
| notmyname | clarkb: mordred ^^ | 18:48 |
| clarkb | notmyname: noted thanks | 18:48 |
| notmyname | *Merritt | 18:48 |
| notmyname | also ttx, please note ^^ | 18:48 |
| clarkb | sdague: I think a simple round robin in the proxy would work | 18:49 |
| ttx | ack | 18:49 |
| clarkb | sdague: to distribute the load of dealing with queries | 18:49 |
| *** Sukhdev has joined #openstack-infra | 18:49 | |
| *** thuc has quit IRC | 18:49 | |
| clarkb | sdague: the actual backend query stuff is load balanced by ES itself, but es01 ends up being the load balancer for that aiui | 18:49 |
| sdague | clarkb: do we need to put a smart proxy in front of it? or would something like dns round robin be sufficient? | 18:50 |
| clarkb | sdague: smart proxy would be better. it may make sense to have an haproxy in there | 18:50 |
| clarkb | apache feels clunky for this, btu does a reasonable job of letting safe API requests though | 18:51 |
| clarkb | fungi: danke for ninja approve | 18:54 |
| *** yfried__ has quit IRC | 18:56 | |
| *** yfried__ has joined #openstack-infra | 18:56 | |
| *** otherwiseguy has joined #openstack-infra | 18:57 | |
| *** zz_gondoi is now known as gondoi | 18:57 | |
| fungi | hashar: does https://bugzilla.wikimedia.org/show_bug.cgi?id=53895 look familiar to you at all? | 18:58 |
| fungi | harlowja: nevermind--i guess it does. i see you commenting in it! ;) | 18:58 |
| fungi | er, hashar ^ | 18:58 |
| fungi | sorry harlowja, tab complete fail | 18:58 |
| harlowja | :) | 18:58 |
| hashar | fungi: yup. We never managed to properly investigate it | 18:58 |
| fungi | hashar: apparently we're able to replicate it fairly reliably, but unable to pin down the cause | 18:59 |
| hashar | fungi: we deemed it an annoyance and instructed folks to retry. A side effect is that Zuul would consider the change unmergeable and thus report back to Gerrit saying the change need to be rebased. | 18:59 |
| clarkb | fungi: hashar: I thought we fixed that in the git review tests | 18:59 |
| fungi | clarkb: nope | 18:59 |
| *** e0ne has joined #openstack-infra | 19:00 | |
| clarkb | fungi: hashar: iirc it was a combination of gerrit servers being slow to stop so subsequent connections got a key they did not expect | 19:00 |
| fungi | clarkb: the concurrency changes didn't help | 19:00 |
| clarkb | fungi: oh I haven't seen a git review test failure for ^ in months | 19:00 |
| hashar | iirc Gerrit has an embedded ssh server | 19:00 |
| hashar | that one might funky from time to time | 19:00 |
| fungi | clarkb: really? they've all been failing with them, we just keep rechecking them until they finally all pass | 19:00 |
| hashar | or maybe the ssh key it offers back is some error message | 19:00 |
| *** sarob_ has joined #openstack-infra | 19:00 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1328997 https://review.openstack.org/99438 | 19:01 |
| uvirtbot | Launchpad bug 1328997 in nova "Unit test failure: openstack_citest" is being accessed by other users\nDETAIL: There are 1 other session(s) using the database." [Critical,Confirmed] https://launchpad.net/bugs/1328997 | 19:01 |
| hashar | another possibility pointed by our Gerrit expert is that we could have some old entry in the known_hosts file | 19:01 |
| fungi | hashar: yes, current suspicions are that it's something bad going on in gerrit's java ssh implementation, but not certain what | 19:01 |
| *** yfried__ has quit IRC | 19:01 | |
| clarkb | hashar: ya thats what I thought was biting us | 19:01 |
| clarkb | restarting gerrit is expensive and it isn't closing the socket properly so you hit the old server with a different known host | 19:01 |
| clarkb | that was the theory at least | 19:02 |
| *** yfried__ has joined #openstack-infra | 19:02 | |
| fungi | clarkb: even running with known hosts set to use /dev/null and strict host key checking disabled, it still sometimes returns a hash mismatch | 19:02 |
| jogo | Alex_Gaynor: ping | 19:02 |
| *** gondoi is now known as zz_gondoi | 19:02 | |
| jogo | https://bugs.launchpad.net/hacking/+bug/1329002 looks like a six related change triggered it | 19:02 |
| uvirtbot | Launchpad bug 1329002 in hacking "H302 fails to recognize something as a module" [Undecided,Confirmed] | 19:02 |
| fungi | clarkb: poking around for potential causes of that error suggest that you can get it when the ssh server returns a broken/invalid host key | 19:03 |
| clarkb | fungi: huh | 19:03 |
| fungi | in which case it's only a key verification failure as a side effect of not getting an actual key at all | 19:03 |
| hashar | fungi: that is a good test case (empty known host and strict host key disabled) | 19:03 |
| hashar | maybe the embedded Ssh server spurt out some exceptions or has some debug logs that could help | 19:04 |
| *** sarob__ has joined #openstack-infra | 19:04 | |
| zaro | hashar: could you pls take a look? https://review.openstack.org/#/c/96309 | 19:05 |
| fungi | hashar: looking, but we archive that as part of the job... http://logs.openstack.org/25/99425/1/check/gate-git-review-python27/1586ca3/console.html#_2014-06-11_17_34_27_068 | 19:05 |
| *** sarob_ has quit IRC | 19:05 | |
| hashar | fungi: in Zuul I get a couple entries per days: zuul.log.2014-06-04.gz:GitCommandError: 'git remote update origin' returned exit status 1: hash mismatch | 19:06 |
| hashar | zaro: sure! | 19:06 |
| *** sarob__ has quit IRC | 19:06 | |
| *** sarob has joined #openstack-infra | 19:06 | |
| fungi | clarkb: this was what i used to test... https://review.openstack.org/#/c/99425/1/git_review/tests/__init__.py | 19:08 |
| clarkb | fungi: wow ok. so I know another thing I saw when reading up on it was that the bouncycastle vs no bouncycastle may make a difference | 19:08 |
| *** zehicle_at_dell is now known as zehicle_in_sfo | 19:09 | |
| fungi | clarkb: it may be worth revisiting. i don't think we've seen this in production, unlike hashar, and we do use bc in production (because it's required by gerrit's contactstore feature) | 19:09 |
| clarkb | yup | 19:10 |
| hashar | zaro: excellent :) Approve +1 | 19:10 |
| clarkb | hashar: do you know if you use bouncycastle? | 19:10 |
| *** nati_ueno has quit IRC | 19:10 | |
| hashar | zaro: I guess that was the last patch pending for the new release of JJB? | 19:10 |
| hashar | clarkb: have no clue but can ask. | 19:11 |
| *** ramashri has quit IRC | 19:11 | |
| *** arnaud has joined #openstack-infra | 19:12 | |
| *** thuc has joined #openstack-infra | 19:12 | |
| *** thuc has quit IRC | 19:13 | |
| clarkb | Kiall: did we still need to speak about unbound? | 19:13 |
| *** ominakov has joined #openstack-infra | 19:13 | |
| *** thuc has joined #openstack-infra | 19:13 | |
| clarkb | Kiall: you *should* be able to put your DNS server on 127.1.1.1 or whatever | 19:13 |
| hashar | clarkb: we are using https://mina.apache.org/sshd-project/index.html | 19:14 |
| zaro | hashar: thanks, i believe that's the only one left i wanted. | 19:14 |
| *** ramashri has joined #openstack-infra | 19:14 | |
| clarkb | hashar: ya that is the ssh server we use but I thought bouncy castle could be used to read/write the key files | 19:14 |
| *** shayneburgess has quit IRC | 19:15 | |
| clarkb | hashar: so bouncy castle isn't doing the ssh server it is just dealing with keys | 19:15 |
| clarkb | fungi: ^ that sound right to you? | 19:15 |
| hashar | clarkb: yeah my contact told me bouncycastle is basically a crypto library | 19:15 |
| *** ominakov has quit IRC | 19:16 | |
| fungi | clarkb: hashar: yes. i believe the ssh server comes with its own internal implementations of the cryptographic primitives it needs, but can use better implementations from bc if it's installed and available | 19:16 |
| *** _nadya_ has joined #openstack-infra | 19:19 | |
| *** crc32 has joined #openstack-infra | 19:21 | |
| hashar | so there is another ssh backend NIO2 but that apparently cause even more troubles | 19:21 |
| *** e0ne has quit IRC | 19:23 | |
| lifeless | sdague: around? | 19:23 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: remove misleading jjb example https://review.openstack.org/96309 | 19:24 |
| Alex_Gaynor | jogo: sorry, just got back from lunch. pong | 19:24 |
| jogo | Alex_Gaynor: been digging into that bug you sent me https://bugs.launchpad.net/hacking/+bug/1329002 | 19:25 |
| uvirtbot | Launchpad bug 1329002 in hacking "H302 fails to recognize something as a module" [Undecided,New] | 19:25 |
| sdague | lifeless: yep | 19:25 |
| Alex_Gaynor | jogo: leaving a comment now | 19:25 |
| *** arnaud has quit IRC | 19:25 | |
| Alex_Gaynor | jogo: I've already re-implemented Python's import machinery in python once, I'd kind of like to not do it again, I hope I can find an easier workaround | 19:26 |
| jogo | Alex_Gaynor: heh, so there is a one easy work around. add six to the import ignore list | 19:26 |
| Alex_Gaynor | jogo: where's your sense of advernture! | 19:27 |
| *** sarob has quit IRC | 19:27 | |
| *** sarob has joined #openstack-infra | 19:27 | |
| *** markmcclain has quit IRC | 19:27 | |
| jogo | it never came back from the last adventure ;) | 19:28 |
| *** ramashri has quit IRC | 19:29 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for large ops stable/havana bug 1329042 https://review.openstack.org/99457 | 19:31 |
| uvirtbot | Launchpad bug 1329042 in nova "gate-tempest-dsvm-large-ops fails consistently in stable/havana" [Critical,Confirmed] https://launchpad.net/bugs/1329042 | 19:31 |
| *** rfolco has joined #openstack-infra | 19:32 | |
| *** sarob has quit IRC | 19:32 | |
| *** denis_makogon has quit IRC | 19:32 | |
| *** bnemec has joined #openstack-infra | 19:34 | |
| *** ramashri has joined #openstack-infra | 19:34 | |
| *** doug-fish has joined #openstack-infra | 19:34 | |
| lifeless | sdague: we've ad several attempts to talk about https://review.openstack.org/#/c/98154/ and related stuff and not gotten very far. TripleO folk are feeling blocked and confused; I'd like to resolve this. | 19:36 |
| lifeless | sdague: can we make a time to have a solid conversation? | 19:36 |
| sdague | lifeless: sure | 19:38 |
| sdague | I'm game for it now | 19:38 |
| sdague | or set asside some time. But if you keep disappearing when I ack you, that's not useful either :) | 19:39 |
| *** fbo_away is now known as fbo | 19:39 | |
| *** rcarrill` has joined #openstack-infra | 19:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add swift details to experimental-swift-logs.* https://review.openstack.org/99445 | 19:42 |
| *** esker has joined #openstack-infra | 19:43 | |
| *** rcarrillocruz has quit IRC | 19:43 | |
| *** alkari has quit IRC | 19:44 | |
| *** Sukhdev has quit IRC | 19:44 | |
| *** yfried__ has quit IRC | 19:45 | |
| *** yfried__ has joined #openstack-infra | 19:45 | |
| lifeless | sdague: I know :( | 19:46 |
| lifeless | sdague: its breakfast time here in NZ and Cynthia had just derailed her trains | 19:46 |
| lifeless | sdague: not a euphism | 19:46 |
| lifeless | sdague: so in about 15m would be great | 19:46 |
| sdague | ok | 19:47 |
| sdague | 15 minutes from now is good for me | 19:47 |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/publications: Minor updates, new gerrit and zuul screenshots https://review.openstack.org/99173 | 19:49 |
| *** ramashri has quit IRC | 19:49 | |
| *** sweston has quit IRC | 19:51 | |
| openstackgerrit | Dan Stangel proposed a change to openstack-infra/gitdm: Update gitdm to use gerrit 2.8 workflow names https://review.openstack.org/99466 | 19:52 |
| *** nati_ueno has joined #openstack-infra | 19:52 | |
| *** UtahDave has joined #openstack-infra | 19:52 | |
| Alex_Gaynor | jogo: Ok, I've got a fix I believe, sending a patch as soon as tests finish | 19:53 |
| *** sarob has joined #openstack-infra | 19:54 | |
| *** yfried__ has quit IRC | 19:56 | |
| *** yfried__ has joined #openstack-infra | 19:56 | |
| openstackgerrit | Alex Gaynor proposed a change to openstack-dev/hacking: Fixed warning H302 when used with six.moves https://review.openstack.org/99467 | 19:57 |
| *** sweston has joined #openstack-infra | 19:57 | |
| *** yfried has joined #openstack-infra | 19:57 | |
| *** sarob has quit IRC | 19:58 | |
| *** dprince has quit IRC | 19:59 | |
| *** _nadya_ has quit IRC | 19:59 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/git-review: Testing with Bouncy Castle https://review.openstack.org/99468 | 20:00 |
| *** thuc has quit IRC | 20:00 | |
| *** thuc has joined #openstack-infra | 20:01 | |
| *** yfried__ has quit IRC | 20:01 | |
| *** pdmars has quit IRC | 20:02 | |
| *** markmcclain has joined #openstack-infra | 20:02 | |
| *** ArxCruz_ has joined #openstack-infra | 20:04 | |
| *** mwagner_lap has quit IRC | 20:04 | |
| *** ArxCruz has quit IRC | 20:04 | |
| *** adalbas has quit IRC | 20:04 | |
| *** thuc has quit IRC | 20:05 | |
| lifeless | sdague: okiedoky I'm esconced in my office | 20:05 |
| sdague | ok, cool | 20:05 |
| lifeless | no distractions other than well everything. But no family :) | 20:05 |
| *** Ryan_Lane1 has joined #openstack-infra | 20:06 | |
| sdague | heh | 20:06 |
| lifeless | clarkb: btw fixed /COMMIT_MSG in gertty last night | 20:06 |
| lifeless | sdague: ^ like that, butterflybrain me | 20:06 |
| lifeless | anyhow | 20:06 |
| lifeless | sdague: so | 20:06 |
| lifeless | sdague: I'd like to achieve two things; I'd like to increase my understanding of e-r so that I can understand the reservations you're expressing about tripleo's integration there | 20:07 |
| sdague | lifeless: sure | 20:07 |
| lifeless | sdague: and I'd like to increase your understanding of tripleo-ci and its path to gatehood + the compromise situation we're in | 20:07 |
| *** Ryan_Lane has quit IRC | 20:08 | |
| *** zns has quit IRC | 20:08 | |
| sdague | sure | 20:08 |
| sdague | where do you want to start | 20:08 |
| lifeless | sdague: so, I'm not sure if I should ask 20 questions, or if you want to just tell me where I seem to be confused and go from there ;) | 20:08 |
| lifeless | I'm sure there are some basic axioms we don't hold in common driving this | 20:08 |
| sdague | ok, so lets start on e-r, because I think I can frame that | 20:08 |
| sdague | e-r grew up from 'gate on fire' as basically the #1 tool to help us fight that | 20:09 |
| *** e0ne has joined #openstack-infra | 20:09 | |
| sdague | and has grown very organically since then | 20:09 |
| *** markmcclain has quit IRC | 20:09 | |
| sdague | my concern, right now, about adding things that aren't actually the integrated gate into it, is we're at a tipping point in the dashboards in getting really confusing again | 20:10 |
| *** ramashri has joined #openstack-infra | 20:10 | |
| sdague | given some of the ironic bits that landed already, I already have to mentally filter out the dashboards of 'not relevant' to the 15 hr gate backlog I've got | 20:10 |
| sdague | everything in there used to be relevant to the task at hand | 20:11 |
| *** alkari has joined #openstack-infra | 20:11 | |
| lifeless | ok, so there is a tragedy of the commons effect on folk debugging the integrated gate reliability | 20:11 |
| lifeless | is it purely mental overhead or are there technical considerations as well ? | 20:11 |
| sdague | lifeless: yes, there is a massive tragedy of the commons on that | 20:12 |
| lifeless | e.g. performance, scale, latency,.... | 20:12 |
| *** cody-somerville has quit IRC | 20:12 | |
| sdague | so there is definitely mental overhead | 20:12 |
| sdague | I don't know if there is yet a performance impact on ES | 20:12 |
| sdague | I'm not really sure which pieces you are indexing there | 20:13 |
| sdague | though, ES hasn't been able to keep up with the data flow in 3 days now | 20:13 |
| lifeless | ok | 20:13 |
| *** bnemec has quit IRC | 20:13 | |
| sdague | so we're actually pretty blind, last I looked we're 12 hrs behind indexing job output | 20:13 |
| *** nati_ueno has quit IRC | 20:13 | |
| lifeless | so, factoid, tripleo today is a fairly small fraction of dvsm log load | 20:13 |
| sdague | and we really need to be within 5 minutes | 20:13 |
| lifeless | but it won't stay that way | 20:13 |
| sdague | lifeless: yep, I assumed as such | 20:14 |
| lifeless | ok | 20:14 |
| sdague | the value on the bot that reports to gerrit decreases dramatically after 15 minutes | 20:14 |
| sdague | to the point that we give up, because people are probably looking at their fail already | 20:15 |
| lifeless | so for now, I think E-R scale needs to handle dvsm and work done to scale that will easily compensate for the low tripleo load; as tripleo grows this will change but I expect tripleo to be behind the problem-curve for some time yet | 20:15 |
| sdague | yep, so that part we need to anticipate | 20:15 |
| lifeless | '...dvsm, and....' | 20:15 |
| sdague | the mental load bit is a real problem today | 20:15 |
| lifeless | ok | 20:15 |
| lifeless | pcrews: yo | 20:15 |
| pcrews | lifeless: howdy | 20:16 |
| sdague | which is why I last responded with if we are going to expand scope beyond the core gate, we need to work on the dashboards first to support that, because we *can't* make it any harder for people to work on gate bugs | 20:16 |
| lifeless | pcrews: up for some E-R hacking, sdague and I are discussing some things and the outcome is likely going to be a few bits of work | 20:16 |
| lifeless | sdague: so interestingly the dashboards are less interesting for us than the comments on reviews, but not uninteresting, if you see what I mean. | 20:17 |
| pcrews | lifeless: most definitely :) | 20:17 |
| sdague | lifeless: so that confuses me | 20:17 |
| sdague | how are you addressing actually fixing the races? | 20:17 |
| lifeless | sdague: I think I can unconfuse you when we swing back around to the tripleo situation | 20:17 |
| lifeless | sdague: if you can mentally box it up for now, lets stay on your needs | 20:18 |
| sdague | lifeless: honestly, I think that's the bulk of it | 20:18 |
| lifeless | sdague: ok | 20:18 |
| sdague | but, just as an asside | 20:18 |
| *** dkehn has quit IRC | 20:19 | |
| lifeless | so I'm not familiar with ER's codebase in detail, but from what I can tell there are two dashbords - base and uncategorised? | 20:19 |
| sdague | realize that all the folks that do e-r things, including the reviews, are also the ones that are the biggest gate firefighters | 20:19 |
| clarkb | lifeless: neat I will have to find your change and apply it locally | 20:19 |
| sdague | so when you come asking things when the gate is backed up, you will get flamed :) | 20:19 |
| sdague | or you'll find them short of patience at least | 20:20 |
| sdague | which is the longer tradgey of the commons of the gate | 20:20 |
| sdague | asside done | 20:20 |
| lifeless | sdague: yep, and we have the same small-group, quick dropoff pattern visible within tripleo specific jobs | 20:20 |
| *** bnemec has joined #openstack-infra | 20:20 | |
| fungi | unless you come in asking "who wants to go grab a beer with me after i push out this patch fixing the biggest gate blocking bug?" | 20:21 |
| clarkb | sdague: so fwiw logstash appears to be catching up big time | 20:21 |
| sdague | clarkb: cool! | 20:21 |
| sdague | fungi: exactly :) | 20:21 |
| lifeless | sdague: so a) know it, b) part of it in a largly disjoint set of jobs :/ | 20:21 |
| clarkb | sdague: but that may be related to the big purple spot on the node graph | 20:21 |
| *** nati_ueno has joined #openstack-infra | 20:21 | |
| lifeless | sdague: ok so lets quickly capture a design that will fix your mental load | 20:21 |
| *** arnaud__ has joined #openstack-infra | 20:22 | |
| fungi | clarkb: so taking crm114 processing out of the pipeline seems to have done it? | 20:22 |
| clarkb | fungi: no, I only did that on one node | 20:22 |
| lifeless | sdague: I propose the following criteria: a) must not make e-r maintainers hurl, b) must back sdague and other gate firefighters happy when looking at dashbboards | 20:22 |
| fungi | oh, hrm | 20:22 |
| clarkb | fungi: I think the disk situation on the es nodes was more damaging | 20:22 |
| sdague | lifeless: yep | 20:22 |
| fungi | ahh, yup. mostly-full filesystems == i/o performance problems | 20:22 |
| *** arnaud__ has quit IRC | 20:23 | |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/publications: Minor updates, new gerrit and zuul screenshots https://review.openstack.org/99173 | 20:23 |
| clarkb | fungi: so, I need to plan adding a second volume to all those nodes | 20:23 |
| lifeless | sdague: so given that, do you have a suggestion on how to address it, or do we need to design from scratch ? | 20:23 |
| *** arnaud__ has joined #openstack-infra | 20:23 | |
| clarkb | fungi: do you think an online ext4 resize on them is safeish? | 20:23 |
| sdague | lifeless: I can see a couple of approaches | 20:23 |
| fungi | clarkb: lemme take a tally on our cinder utilization in that account... | 20:23 |
| lifeless | sdague: if we can capture something pcrews can go and hack on, I rather suspect he will :) | 20:23 |
| clarkb | fungi: thanks | 20:23 |
| sdague | 1) create a new tag in the queries files that says this is part of: 'foo' | 20:24 |
| *** cody-somerville has joined #openstack-infra | 20:24 | |
| sdague | and if it's part of anything, we exclude it from the main graphs that we have, and make it basically a seperate namespace for the dashboards | 20:24 |
| clarkb | fungi: I just recheck experimental'd a config change to see if the zuul swift stuff will work | 20:24 |
| clarkb | fungi: I am very excited about this too :) | 20:24 |
| sdague | actually, that's probably a better idea than the other ideas I've got | 20:25 |
| lifeless | ok | 20:25 |
| sdague | basically create namespaces within er | 20:25 |
| lifeless | pcrews: https://git.openstack.org/openstack-infra/elastic-recheck is the codebase in question | 20:25 |
| sdague | then tripleo can be a namespace, which includes a full set of dashboards that we have now | 20:25 |
| lifeless | pcrews: http://status.openstack.org/elastic-recheck/ is the deployed thing | 20:25 |
| sdague | the dashboards are a little adhoc, appologies for the organic there, so we might need to poke a bit on the right way to do that | 20:26 |
| fungi | clarkb: yeah, we're using 23.1tb of our 25tb quota, so we could spare a 300gb volume for each of the 6 es nodes, but after that we're basically tapped out without either moving logs to swift or asking for another quota bump | 20:26 |
| pcrews | lifeless: ack - I've been reading up on that system a bit out of personal interest :) | 20:26 |
| clarkb | fungi: ok, we already bumped that quota right? | 20:27 |
| pcrews | our group was working on similar-ish stuff for MySQL way back when...fun stuff | 20:27 |
| sdague | pcrews: cool :) | 20:27 |
| fungi | clarkb: yeah, it was originally at 10tb and we asked them to up it to 25tb | 20:27 |
| clarkb | fungi: I don't expect the logs stuff to be a magical overnight cure as we won't migrate the data | 20:27 |
| lifeless | pcrews: so, if you can get what sdague has described working, it will allow tripleo and ironic to use e-r which will be cool | 20:27 |
| clarkb | fungi: we may need to ask for more quota in the interim | 20:27 |
| lifeless | sdague: do you want a spec, or just pcrews patches ? | 20:28 |
| sdague | It's probably worth an etherpad to sketch out | 20:28 |
| lifeless | sdague: are there any special tricks to hacking on this? like, do you need local elasticsearch infrastructure, or you can use the deployed api endpoints? | 20:28 |
| sdague | but I'm ok with patches as well | 20:28 |
| lifeless | http://etherpad.openstack.org/p/elastic-recheck-namespace | 20:29 |
| sdague | you can use the deployed endpoint | 20:29 |
| clarkb | sdague: of course as soon as I say it is catching up it slows down and backs up a bit :( | 20:29 |
| sdague | clarkb: oom? | 20:29 |
| clarkb | sdague: no just normal activity I think | 20:29 |
| sdague | ok | 20:29 |
| sdague | well the slope was good | 20:29 |
| sdague | hopefully it keeps up on it | 20:29 |
| clarkb | yeah I think we are hitting a local disturbance in the force | 20:29 |
| clarkb | but we will see | 20:29 |
| sdague | clarkb: gate reset? | 20:30 |
| sdague | yeh, we just reset | 20:30 |
| *** _nadya_ has joined #openstack-infra | 20:30 | |
| clarkb | ya | 20:30 |
| sdague | a few minutes ago | 20:30 |
| sdague | that blips us | 20:30 |
| *** enikanorov has quit IRC | 20:30 | |
| *** gary-smith has left #openstack-infra | 20:31 | |
| *** sarob has joined #openstack-infra | 20:31 | |
| clarkb | phschwartz: do you know if asking for even more terabytes of volume would be a bad thing? | 20:32 |
| sdague | lifeless: ok, so part 1, lets saw we have a sketch now, and we can work on the details with pcrews as we go | 20:32 |
| clarkb | phschwartz: we don't want to abuse these resources :) so honest answers are good | 20:32 |
| sdague | so part 2? | 20:32 |
| *** flaper87 is now known as flaper87|afk | 20:32 | |
| lifeless | ok part 2, tripleos needs | 20:32 |
| sdague | asside, pcrews what TZ you in? | 20:33 |
| lifeless | when tripleo tests fail, its sometimes races in openstack, and othertimes flaky infrastructure | 20:33 |
| lifeless | like | 20:33 |
| *** doude has quit IRC | 20:33 | |
| lifeless | pypi.openstack.org deciding to be unreachable for 30 seconds | 20:33 |
| *** marcoemorais has joined #openstack-infra | 20:33 | |
| *** shayneburgess has joined #openstack-infra | 20:33 | |
| sdague | sure | 20:33 |
| lifeless | and kills a dozen jobs all at once | 20:33 |
| *** jamielennox is now known as jamielennox|away | 20:34 | |
| lifeless | we can't today run tripleo jobs in the main infrastructure because you can't pretend to be baremetal with any fidelity in public clouds | 20:34 |
| sdague | so this also relates to the fact that you don't run tempest yet | 20:34 |
| lifeless | it does | 20:34 |
| lifeless | we have a patchset up to do that | 20:34 |
| lifeless | andrea within HP is working on that | 20:34 |
| sdague | because I think your swing of races in openstack vs. infra flakey is going to dramatically change after that | 20:34 |
| lifeless | we need to run tempest in three different configurations | 20:34 |
| lifeless | a) against the seed, b) against the undercloud, c) against a deployed overcloud | 20:34 |
| lifeless | thats one dimension | 20:34 |
| lifeless | as we add more configurations that folk deploy, we'll build a matrix, which I know you're familiar with :) | 20:35 |
| pcrews | sdague: I am in Seattle | 20:35 |
| sdague | sure | 20:35 |
| lifeless | but right now we have a huge spurious failure rate | 20:35 |
| sdague | lifeless: is that a sequence, or are you running a, b, c as parallel | 20:35 |
| *** sarob has quit IRC | 20:35 | |
| lifeless | sdague: those will be three different jobs in my head | 20:36 |
| sdague | lifeless: kk | 20:36 |
| lifeless | sdague: but it could be we deploy, test, deploy, test, deploy test | 20:36 |
| sdague | yep, that's fine | 20:36 |
| * pcrews has to attend to dog + caloric input needs then will be back to start working on e-r stuff. | 20:36 | |
| sdague | pcrews: cool :) | 20:36 |
| lifeless | so we have two immediate uses for ER | 20:36 |
| clarkb | pcrews: be careful going outside today. The weather is too nice for seattle | 20:36 |
| *** vhoward has joined #openstack-infra | 20:36 | |
| lifeless | one is to help derekh and me and pcrews and others who are focused on CI prioritise what we need to fix to fix these infrastructury things | 20:37 |
| lifeless | which sometimes will be making openstack code more robust and sometimes bringing new services into the CI environment - you know the drill | 20:37 |
| sdague | yep | 20:37 |
| lifeless | the second thing is that we want to *avoid* folk recheck no bugging jobs | 20:37 |
| lifeless | its wasteful, they should only recheck if its really spurious | 20:37 |
| sdague | sure, that's fair. | 20:37 |
| sdague | especially as I think today a recheck is retriggering the jenkins nodes as well | 20:38 |
| sdague | for the jobs that live in both check queues | 20:38 |
| lifeless | your questions about whether reviewers can be reasoned with the other day - thats when I realised we weren't clicking :) | 20:38 |
| pcrews | clarkb: I still feel rather leery daring to go too far outside w/o some sort of jacket :) | 20:39 |
| lifeless | but I can answer them now - we have 10 times as many open reviews as reviewers, and reviewers don't have the time or patience to be reasoned with over test failures | 20:39 |
| *** esker has quit IRC | 20:39 | |
| lifeless | sdague: I believe it does, yes. | 20:39 |
| *** portante has quit IRC | 20:39 | |
| *** yjiang5 has quit IRC | 20:39 | |
| *** dosaboy has quit IRC | 20:39 | |
| *** asadoughi has quit IRC | 20:39 | |
| *** vhoward has quit IRC | 20:39 | |
| *** fbo has quit IRC | 20:39 | |
| *** dkliban has quit IRC | 20:39 | |
| *** cdub has quit IRC | 20:39 | |
| *** tteggel has quit IRC | 20:39 | |
| *** katyafervent_awa has quit IRC | 20:39 | |
| *** bnemec_away has quit IRC | 20:39 | |
| *** jd__ has quit IRC | 20:39 | |
| *** Guest15213 has quit IRC | 20:39 | |
| *** mnaser has quit IRC | 20:39 | |
| *** msuriar has quit IRC | 20:39 | |
| lifeless | mordred: review 95234 please | 20:40 |
| sdague | ok, so the namespace bit I think will help here | 20:40 |
| *** msuriar has joined #openstack-infra | 20:40 | |
| *** dosaboy has joined #openstack-infra | 20:40 | |
| *** cdub has joined #openstack-infra | 20:40 | |
| *** Vivek has joined #openstack-infra | 20:40 | |
| *** portante has joined #openstack-infra | 20:40 | |
| *** portante has quit IRC | 20:40 | |
| *** portante has joined #openstack-infra | 20:40 | |
| *** asadoughi has joined #openstack-infra | 20:40 | |
| *** vhoward has joined #openstack-infra | 20:40 | |
| *** fbo has joined #openstack-infra | 20:40 | |
| *** dkliban has joined #openstack-infra | 20:40 | |
| *** yjiang5 has joined #openstack-infra | 20:40 | |
| *** dkliban has quit IRC | 20:40 | |
| *** dkliban has joined #openstack-infra | 20:40 | |
| lifeless | mordred: erm no not that one. 99277 | 20:40 |
| *** yjiang5 has quit IRC | 20:40 | |
| *** yjiang5 has joined #openstack-infra | 20:40 | |
| *** Vivek is now known as Guest56682 | 20:40 | |
| *** jd__ has joined #openstack-infra | 20:40 | |
| sdague | there remains sort of a meta issue though, which I think we remain not clicking on. That I just want to float to let sift. | 20:40 |
| lifeless | sdague: We've got a big picture goal of getting tripleo-ci up the maturity curve to where dvsm is | 20:40 |
| *** tteggel has joined #openstack-infra | 20:41 | |
| *** beekneemech has joined #openstack-infra | 20:41 | |
| *** katyafervent_awa has joined #openstack-infra | 20:41 | |
| lifeless | sdague: which means we're *actively* pushing back on changes that reinvent bits of the related machinery (like e-r) and instead pushing towards tighter integration | 20:41 |
| lifeless | sdague: (so that we get the virtuous circle of working on the same things etc etc) | 20:41 |
| lifeless | anyhow, hopefully there was something new in there for you :) | 20:42 |
| sdague | lifeless: well, except tripleo seems to reinvent a lot of things already :) | 20:42 |
| *** mwagner_lap has joined #openstack-infra | 20:42 | |
| sdague | it's funny to watch devtest conceptually keep reconverging on devstack | 20:42 |
| *** mnaser has joined #openstack-infra | 20:42 | |
| lifeless | sdague: how do you mean? | 20:43 |
| *** hashar has quit IRC | 20:44 | |
| *** zz_gondoi is now known as gondoi | 20:44 | |
| sdague | honestly, my mental energy isn't all here after a week of fire fighting to really have that conversation | 20:44 |
| lifeless | ok | 20:44 |
| sdague | but it definitely feels like there is a lot of more general disconnect of tripleo vs. the rest of openstack | 20:45 |
| lifeless | I want to have it, because I don't see that theres any substantial duplication (and we keeping splitting things out to make them reusable, precisely to avoid having multiple cpoies of things) | 20:45 |
| mordred_phone | lifeless, sdague maybe we should arrange some face time at some point | 20:46 |
| sdague | mordred_phone: sure | 20:46 |
| lifeless | sdague: ok, so we've covered the essentials - thank you | 20:46 |
| sdague | lifeless: agreed, there is a game plan here | 20:47 |
| lifeless | sdague: speaking of areas where the may be overlap, what does devstack do for permitting carrying local bugfixes etc, e.g. 'I need review X Y and Z of nova' ? | 20:47 |
| sdague | lifeless: it doesn't | 20:47 |
| sdague | I was trying to fix it in zuul | 20:47 |
| *** doude has joined #openstack-infra | 20:47 | |
| sdague | well, it could | 20:47 |
| sdague | you can specify branch | 20:47 |
| sdague | but we should fix it in zuul - https://review.openstack.org/#/c/94365/ | 20:48 |
| lifeless | sdague: 96659 might interest you - its a handwave today based on a thing adam_g threw up | 20:48 |
| lifeless | sdague: its aimed at the downstream folk when a patch is being reviewed and discussed but things are broken without it | 20:48 |
| jesusaurus | clarkb: i just noticed that your change https://review.openstack.org/#/c/99263/1 will disappear if https://review.openstack.org/#/c/98561/3 is merged | 20:49 |
| sdague | sure, that's a downstream solution | 20:49 |
| *** zns has joined #openstack-infra | 20:49 | |
| *** shardy_afk is now known as shardy | 20:50 | |
| *** penguinRaider has joined #openstack-infra | 20:50 | |
| *** esker has joined #openstack-infra | 20:50 | |
| sdague | I also think it would be good if the tripleo team thought about tempest-dsvm as their unit tests. Because I think a lot of the issues that are going to trip you up in stability could be shored up in tempest-dsvm. | 20:50 |
| *** esker has quit IRC | 20:50 | |
| jesusaurus | clarkb: nevermind me. im blind | 20:51 |
| sdague | would love to see more cross polination there, especially as it would be good place to learn some tempest things which I'm sure the tripleo team is going to need to grok to stabalize everything out | 20:51 |
| lifeless | sdague: so we need to get tempest running first | 20:52 |
| sdague | lifeless: sure | 20:52 |
| lifeless | sdague: though I could see some value from just pure dvsm things | 20:52 |
| sdague | yeh, this isn't an either or, it's an and | 20:52 |
| lifeless | sdague: I'd actually really like it if I could spend a few days with you deploying openstack somewhere | 20:53 |
| lifeless | sdague: we see very different things to dvsm IME | 20:53 |
| sdague | lifeless: I'm sure you do | 20:53 |
| sdague | I've run some smaller clouds based on puppet, I did as well | 20:54 |
| sdague | but the heat issues you are seeing are things that could be nailed down in dvsm | 20:54 |
| *** vhoward has left #openstack-infra | 20:54 | |
| lifeless | absolutely | 20:54 |
| sdague | because right now, heat's largely untested | 20:54 |
| lifeless | jogo has a patch up to address some of that | 20:54 |
| sdague | I'm not saying it will solve all the problems, I just feel like I get the brush off a lot when I suggestion spending some effort there from tripleo folks :) | 20:55 |
| *** thuc has joined #openstack-infra | 20:55 | |
| lifeless | sdague: ah | 20:55 |
| lifeless | sdague: so i think the thing is, that without the ability to run tempest against a tripleo cloud, its hard to get into the virtuous circle mode | 20:56 |
| lifeless | its not like devstack where you hit a button and then can poke at your custom tempest test | 20:56 |
| sdague | sure, that's possible | 20:56 |
| *** mwagner_lap has quit IRC | 20:56 | |
| lifeless | which lopos back to we need to fix that | 20:56 |
| *** fbo is now known as fbo_away | 20:56 | |
| sdague | though if the tests don't run in dsvm, there will be pushback. Because tempest needs to be self gating. | 20:56 |
| *** weshay has quit IRC | 20:57 | |
| lifeless | sdague: I'd be delighted to have a tripleo-undercloud-tempest gate job | 20:57 |
| lifeless | ecstatic even | 20:57 |
| sdague | lifeless: right, but we're optimistically a year away from that I think | 20:57 |
| sdague | based on past experience | 20:57 |
| lifeless | sdague: or more; we have multivendor regions now, and huawei are on track to be another sizeable region | 20:59 |
| lifeless | once we get the nodepool and jenkins stuff in to avoid public IP usage | 20:59 |
| *** vhoward has joined #openstack-infra | 20:59 | |
| sdague | I think that's part of the challenge of devtest being different than devstack, is there is currently a parallel universe of setup which has thusfar prevented the tripleo team from getting as far as tempest | 20:59 |
| lifeless | the problem there is that devstack is an installer, devtest is a meta-installer - it installs an installer, not a cloud | 21:00 |
| sdague | yeh, well people have extended devstack to do all kinds of things at this point | 21:00 |
| *** cody-somerville has quit IRC | 21:01 | |
| sdague | and maybe these things have to stay parallel universes | 21:01 |
| sdague | but there are definitely things lost in that fact, like expertise with tempest, as that's something which is setup by default ready to run at the end of devstack | 21:02 |
| sdague | anyway, this is the meta conversation | 21:02 |
| *** reed has joined #openstack-infra | 21:02 | |
| sdague | I don't know that I've got real concrete ideas to make it better, but wanted to at least float it | 21:02 |
| lifeless | ack | 21:02 |
| sdague | and I should go and ride my bike, as it's finally not raining | 21:03 |
| *** sarob has joined #openstack-infra | 21:03 | |
| lifeless | enjoy :) - thank you! | 21:03 |
| sdague | yep, take care | 21:03 |
| *** rfolco has quit IRC | 21:03 | |
| *** andreykurilin_ has quit IRC | 21:03 | |
| *** bnemec has quit IRC | 21:03 | |
| *** andreykurilin__ has joined #openstack-infra | 21:03 | |
| clarkb | lifeless: reviewed your gertty bug fix | 21:04 |
| * morganfainberg is now wanting to go for a bike ride instead of writing code / booking travel. | 21:04 | |
| clarkb | lifeless: I think there is a slight bug but it fixes a much bigger bug so +1 | 21:05 |
| morganfainberg | sdague, damn it :P | 21:05 |
| *** doude has quit IRC | 21:06 | |
| *** mfer has quit IRC | 21:06 | |
| *** penguinRaider has quit IRC | 21:06 | |
| jerryz | sdague, morganfainberg: you two have fun ride towards the rainbow after the rain. | 21:07 |
| *** sarob has quit IRC | 21:07 | |
| morganfainberg | jerryz, i'm in SoCal and it's a bit too warm out atm. | 21:07 |
| morganfainberg | jerryz, i'm jealous of sdague since he's getting to go for a ride | 21:07 |
| *** eharney has quit IRC | 21:08 | |
| *** achuprin_ has joined #openstack-infra | 21:08 | |
| *** michchap_ has joined #openstack-infra | 21:09 | |
| reed | hello | 21:09 |
| jogo | sdague: for heat, I am gonna add another test to the large-ops job that does the same thing as the current test but with heat | 21:09 |
| jogo | sdague: for starters | 21:09 |
| *** andreykurilin__ has quit IRC | 21:10 | |
| *** amcrn has quit IRC | 21:10 | |
| *** achuprin has quit IRC | 21:10 | |
| *** michchap has quit IRC | 21:10 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for large ops stable/havana bug 1329042 https://review.openstack.org/99457 | 21:10 |
| uvirtbot | Launchpad bug 1329042 in nova "gate-tempest-dsvm-large-ops fails consistently in stable/havana" [Critical,Confirmed] https://launchpad.net/bugs/1329042 | 21:10 |
| *** andreykurilin_ has joined #openstack-infra | 21:11 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Groups portal infra code refactor https://review.openstack.org/99481 | 21:11 |
| *** mbacchi has quit IRC | 21:11 | |
| *** shayneburgess has quit IRC | 21:11 | |
| jogo | sdague: but that got bumped down the queue as the gate is still shaky as is | 21:11 |
| *** aysyd has quit IRC | 21:12 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Groups portal infra code refactor https://review.openstack.org/99481 | 21:12 |
| *** mrda-away is now known as mrda | 21:13 | |
| *** cody-somerville has joined #openstack-infra | 21:16 | |
| *** sarob has joined #openstack-infra | 21:17 | |
| *** bcrochet has quit IRC | 21:18 | |
| *** jlk has quit IRC | 21:18 | |
| *** annegentle has quit IRC | 21:18 | |
| *** grantbow has quit IRC | 21:18 | |
| *** adarazs has quit IRC | 21:18 | |
| *** loquacities has quit IRC | 21:18 | |
| *** SlickNik has quit IRC | 21:18 | |
| *** mat128 has quit IRC | 21:18 | |
| *** davidlenwell has quit IRC | 21:18 | |
| *** dkranz has quit IRC | 21:18 | |
| *** loquacities has joined #openstack-infra | 21:18 | |
| *** mmitchell_ has joined #openstack-infra | 21:18 | |
| *** adarazs has joined #openstack-infra | 21:18 | |
| *** SlickNik has joined #openstack-infra | 21:18 | |
| *** davidlenwell has joined #openstack-infra | 21:19 | |
| *** shayneburgess has joined #openstack-infra | 21:19 | |
| *** Guest81438 has joined #openstack-infra | 21:24 | |
| *** amcrn has joined #openstack-infra | 21:24 | |
| *** redrobot is now known as Guest57969 | 21:25 | |
| *** jlk has joined #openstack-infra | 21:26 | |
| *** jlk has quit IRC | 21:26 | |
| *** jlk has joined #openstack-infra | 21:26 | |
| *** Guest81438 is now known as redrobot | 21:27 | |
| *** lcostantino has quit IRC | 21:28 | |
| *** annegentle has joined #openstack-infra | 21:29 | |
| *** bcrochet has joined #openstack-infra | 21:30 | |
| *** rlandy has quit IRC | 21:31 | |
| *** grantbow has joined #openstack-infra | 21:33 | |
| dangers | reed: I seem to be hitting the "Cannot store contact information" error when updating my OpenStack profile (https://bugs.launchpad.net/openstack-org/+bug/1317957). Can you help me out? | 21:34 |
| uvirtbot | Launchpad bug 1317957 in openstack-org "Server error when updating contacts" [Critical,Confirmed] | 21:34 |
| *** Sukhdev has joined #openstack-infra | 21:34 | |
| *** ArxCruz_ has quit IRC | 21:35 | |
| clarkb | dangers: have you signed up with the openstack foundation? this is required. If you have make sure the primary email address matches the address in gerrit and foundation profile side | 21:35 |
| dangers | clarkb: definitely signed up, ICLA, etc, but I'll double check on the email addresses | 21:36 |
| reed | dangers, most likely you haven't signed up for a profile on openstack.org | 21:37 |
| dangers | hmm | 21:37 |
| fungi | dangers: you went to https://www.openstack.org/join/ and put in a foundation membership application? | 21:37 |
| *** james_li has quit IRC | 21:37 | |
| dangers | oh dang, yeah it looks like my openstack.org profile email != my gerrit/launchpad email :-( | 21:37 |
| fungi | dangers: if so, make sure that the primary e-mail address displayed at https://www.openstack.org/profile/ matches the preferred e-mail address at https://review.openstack.org/#/settings/contact | 21:38 |
| dangers | fungi: yup that's the problem | 21:38 |
| clarkb | dangers: you can make them different after upating your contact info (its weird yes) | 21:38 |
| clarkb | dangers: but to do a contact info update which is different than setting your gerrit primary email address they have to be the sam | 21:38 |
| fungi | yeah, they only have to be the same initially | 21:38 |
| *** CaptTofu_ has quit IRC | 21:38 | |
| dangers | although the "Second Email Address" is configured as such, but guessing the Primary Email has to match | 21:38 |
| reed | dangers, sorry for that... we hope that such complexity will go away soon | 21:38 |
| dangers | No worries | 21:39 |
| dangers | I understand the complexity of the situation :-) | 21:39 |
| lifeless | clarkb: fixed | 21:39 |
| *** CaptTofu_ has joined #openstack-infra | 21:39 | |
| *** ramashri has quit IRC | 21:39 | |
| *** rwsu_ has quit IRC | 21:39 | |
| * dangers flips some bits | 21:39 | |
| * dangers also takes the opportunity to fix t-shirt size | 21:40 | |
| openstackgerrit | lifeless proposed a change to stackforge/gertty: Support comments in commits https://review.openstack.org/99272 | 21:40 |
| dangers | clarkb: fungi reed thanks for your help, that fixed it | 21:41 |
| reed | cool, welcome dangers | 21:42 |
| *** dkehn has joined #openstack-infra | 21:42 | |
| dangers | well so I've been committing to openstack-infra/gitdm since 2013 with no issue, but trying to make changes elsewhere this just started showing up | 21:42 |
| *** rwsu has joined #openstack-infra | 21:42 | |
| clarkb | lifeless: that did it thanks | 21:43 |
| *** CaptTofu_ has quit IRC | 21:43 | |
| *** dkehnx has quit IRC | 21:44 | |
| clarkb | lifeless: though I think pytz is a noop now | 21:45 |
| *** jergerber has joined #openstack-infra | 21:45 | |
| lifeless | oh indeed | 21:45 |
| openstackgerrit | lifeless proposed a change to stackforge/gertty: Support comments in commits https://review.openstack.org/99272 | 21:46 |
| mordred | lifeless: re: 99277 - it should not be necessary | 21:48 |
| jogo | clarkb: what part of elasticsearch is unable to keep up? | 21:48 |
| jogo | is it too many logs? | 21:48 |
| clarkb | jogo: that is the simple way of putting it | 21:48 |
| clarkb | jogo: we don't have enough disk | 21:48 |
| clarkb | mordred: re 99272? | 21:48 |
| jogo | thats is contributing to why ES cannot keep up? | 21:48 |
| mordred | clarkb: 99288 | 21:49 |
| mordred | gah | 21:49 |
| mordred | clarkb: 99277 - pbr | 21:49 |
| clarkb | jogo: if it can't write the index then it doesn't keep up | 21:49 |
| clarkb | mordred: ah ok I thought maybe tyop | 21:49 |
| *** shayneburgess has quit IRC | 21:49 | |
| *** mriedem has quit IRC | 21:49 | |
| jogo | clarkb: ahh and thats why dropping the number of days down will help | 21:49 |
| clarkb | oh right because of setuptools magic | 21:49 |
| clarkb | jogo: yes | 21:49 |
| mordred | lifeless: oh - well, it gets autoregisterd as test - but not as testr | 21:50 |
| clarkb | jogo: until we can get more quota and bump the volume sizes on those nodes | 21:50 |
| mordred | we could do more magic - but it's also probably fine to just land the patch | 21:50 |
| *** _nadya_ has quit IRC | 21:50 | |
| clarkb | jogo: also I think we may be feeling some upper bound pressure and need to do more tuning and or rethink the approach here | 21:50 |
| jogo | clarkb: rethink which part? | 21:50 |
| clarkb | jogo: the amount of data in elasticsearch | 21:50 |
| jogo | do you think info logs should log all incoming requests http://logs.openstack.org/43/98443/8/gate/gate-tempest-dsvm-neutron/0b944e9/logs/screen-n-api.txt.gz?level=INFO#_2014-06-11_20_39_33_249 | 21:51 |
| lifeless | mordred: if its registered as test, then perhaps we should just change the tox.ini w.r.t. it ? | 21:51 |
| jogo | sdague: ^ | 21:51 |
| clarkb | jogo: which may mean supplementing es with other tools | 21:51 |
| jogo | clarkb: sounds like something good to chat about in germany if we can wait that long | 21:51 |
| clarkb | possibly | 21:52 |
| *** sballe has joined #openstack-infra | 21:52 | |
| *** sballe_ has quit IRC | 21:52 | |
| mordred | lifeless: maybe- but I think we have a bunch of testr still in the wild, so we should fix that too | 21:52 |
| *** mwagner_lap has joined #openstack-infra | 21:52 | |
| jogo | clarkb: if we don't log all incoming API requests at info we will drop the api server lags way down | 21:53 |
| mordred | clarkb: what's the gertty magic for "jump to change" ? | 21:53 |
| jogo | just don't know if that is a good idea or not | 21:53 |
| *** rcarrillocruz has joined #openstack-infra | 21:53 | |
| *** markmcclain has joined #openstack-infra | 21:54 | |
| lifeless | mordred: so, merge the patch? | 21:54 |
| clarkb | mordred: ^O | 21:54 |
| lifeless | mordred: now, I need a timeslive from you :) | 21:54 |
| clarkb | jogo: I am not sure I can answer that | 21:54 |
| mordred | clarkb: it would be neat if that was in the help - also, thanks | 21:55 |
| lifeless | mordred: 97753 | 21:55 |
| clarkb | jogo: the tl;dr preview is I think we should consider logging in json, switching logstash to fluentd, and using es for short term indexes (shorter than today) with hadoop for the big picture stuff sdague wants to do | 21:55 |
| jogo | clarkb: I was hoping sdague was around, I think he could | 21:55 |
| jogo | oh that sounds exciting | 21:56 |
| *** rcarrill` has quit IRC | 21:56 | |
| mordred | clarkb: "we should consider logging in json" sounds crazy | 21:56 |
| lifeless | clarkb: <3 | 21:56 |
| clarkb | lifeless: I am happy that this doens't make you think I am crazy | 21:56 |
| lifeless | structured logging is win | 21:57 |
| clarkb | mordred: you would think that. But we spend tons of CPU on making the data structured | 21:57 |
| clarkb | mordred: we can completely avoid that overhead if we start with structured data | 21:57 |
| clarkb | it also makes hadoop much more reasonable | 21:57 |
| mordred | hrm. werid | 21:57 |
| mordred | also - ruby - but I defer to you on this | 21:57 |
| *** Guest57969 has quit IRC | 21:58 | |
| clarkb | ruby? | 21:58 |
| jogo | fluentd | 21:58 |
| clarkb | oh well logstash is ruby too | 21:59 |
| mordred | lifeless: can you expand on your comment on 95569? | 21:59 |
| mordred | lifeless: I'm not sure what that means in python | 22:00 |
| lifeless | mordred: itertools.tee and itertools.slice | 22:00 |
| lifeless | mordred: what I wrote was python :) | 22:00 |
| mordred | hrm. | 22:01 |
| lifeless | mordred: specifically, right now you open a file, do some data processing within that context and write to the file | 22:01 |
| lifeless | split the concerns | 22:01 |
| lifeless | do some data processing in a generator that yields changelog entries | 22:01 |
| lifeless | then pass that generator to tee to get two iterators | 22:01 |
| lifeless | then use one to write to a file and one to flesh out long_description | 22:01 |
| mordred | nod. k. I'll try that - thanks | 22:02 |
| mordred | lifeless: what was the timeslice you needed from me? | 22:02 |
| reed | well, San Francisco won't host the next Americas Cup... after all that | 22:02 |
| *** lcheng has quit IRC | 22:02 | |
| * reed yells at the cloud | 22:02 | |
| lifeless | mordred: cody's question and my pbr patch, one is interactive with me, the other isn't | 22:02 |
| *** virmitio has quit IRC | 22:02 | |
| clarkb | but fluentd apparently writes critical portions in C and doesn't do a lot of heavy processing like logstash | 22:02 |
| *** redrobot has left #openstack-infra | 22:02 | |
| clarkb | I think it will be a win if we can start with structured data | 22:02 |
| mordred | lifeless: which pbr patch? and what's cody's question? | 22:03 |
| mordred | clarkb: would we need to add a json output thing to oslo.log? | 22:03 |
| clarkb | mordred: I think its already done https://pypi.python.org/pypi/python-json-logger | 22:03 |
| mordred | like, would that be the right way to get json log data? and/or should we just add a fluentd output module for oslo.log and spit things directly? | 22:04 |
| clarkb | but I haven't actually looked to see if that works | 22:04 |
| mordred | neat | 22:04 |
| lifeless | mordred: pbr - 94107 | 22:04 |
| clarkb | but json should also make manipulation for human display easier too | 22:04 |
| lifeless | mordred: the gozer machines thing | 22:05 |
| *** amcrn has quit IRC | 22:05 | |
| *** dkliban is now known as dkliban_afk | 22:05 | |
| clarkb | mordred: there is also python fluent-logger | 22:06 |
| clarkb | but I don't think we change the decoupled logging for tests that we have today | 22:07 |
| clarkb | instead we have the tests dump json lines to files | 22:07 |
| clarkb | then pick them up afterwards and shove them into fluentd so the python-json-logger is a better approach | 22:07 |
| *** signed8bit has quit IRC | 22:07 | |
| *** signed8bit has joined #openstack-infra | 22:08 | |
| *** sarob has quit IRC | 22:09 | |
| *** sarob has joined #openstack-infra | 22:10 | |
| mordred | nod | 22:10 |
| *** chianingwang has quit IRC | 22:12 | |
| *** andreykurilin_ has quit IRC | 22:12 | |
| *** shayneburgess has joined #openstack-infra | 22:14 | |
| *** chianingwang has joined #openstack-infra | 22:14 | |
| clarkb | but I need to find time to throw something like that together and see if it is sane | 22:14 |
| *** nati_ueno has quit IRC | 22:14 | |
| *** sarob has quit IRC | 22:14 | |
| clarkb | jogo: potentially worth digging in on in darmstadt | 22:15 |
| *** gondoi is now known as zz_gondoi | 22:16 | |
| *** lcheng has joined #openstack-infra | 22:17 | |
| *** _nadya_ has joined #openstack-infra | 22:17 | |
| *** markmcclain has quit IRC | 22:19 | |
| *** maxbit has quit IRC | 22:20 | |
| *** reed has quit IRC | 22:21 | |
| *** afazekas has joined #openstack-infra | 22:22 | |
| *** dangers is now known as dangers_away | 22:22 | |
| *** gokrokve has quit IRC | 22:25 | |
| clarkb | it is possible that logstash will be a lot more useable for us if we start with structured data | 22:25 |
| clarkb | but the grok filters kill us | 22:25 |
| *** prad has quit IRC | 22:26 | |
| *** lcheng has quit IRC | 22:26 | |
| *** dims__ has quit IRC | 22:28 | |
| *** signed8bit has quit IRC | 22:28 | |
| openstackgerrit | João Cravo proposed a change to openstack-infra/jenkins-job-builder: Add support for reverse build trigger https://review.openstack.org/95734 | 22:29 |
| clarkb | fungi: crud I think zuul is unhappy with that log server stuff | 22:29 |
| clarkb | fungi: I am looking into it now | 22:29 |
| *** markmcclain has joined #openstack-infra | 22:30 | |
| *** arnaud__ has quit IRC | 22:30 | |
| *** nati_ueno has joined #openstack-infra | 22:31 | |
| *** jergerber has quit IRC | 22:31 | |
| *** dkehn has quit IRC | 22:32 | |
| *** otherwiseguy has quit IRC | 22:32 | |
| *** sarob has joined #openstack-infra | 22:32 | |
| *** reaper has quit IRC | 22:34 | |
| *** virmitio has joined #openstack-infra | 22:35 | |
| *** shayneburgess has quit IRC | 22:36 | |
| *** virmitio has left #openstack-infra | 22:36 | |
| *** smarcet has quit IRC | 22:37 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Fix swift logserver prefixes. https://review.openstack.org/99498 | 22:38 |
| clarkb | jhesketh: fungi mordred ^ | 22:38 |
| *** e0ne has quit IRC | 22:38 | |
| clarkb | now to figure out how to kick the existing job out as the fix will require a zuul restart | 22:38 |
| mordred | clarkb on it | 22:38 |
| *** shayneburgess has joined #openstack-infra | 22:39 | |
| *** e0ne has joined #openstack-infra | 22:39 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Use WatchedFileHandler to avoid copytruncate. https://review.openstack.org/95935 | 22:39 |
| clarkb | woot pushing a new patchset worked | 22:40 |
| clarkb | so don't check experimental any config changes :) | 22:40 |
| *** CaptTofu_ has joined #openstack-infra | 22:41 | |
| *** thuc has quit IRC | 22:42 | |
| clarkb | mordred: I was going to propose a layout.yaml change to use a non default value but the type in voluptuous is int so no can do :( | 22:42 |
| *** afazekas has quit IRC | 22:42 | |
| mordred | clarkb: stupid type checking | 22:42 |
| clarkb | oh wait that change is still wrong | 22:42 |
| *** thuc has joined #openstack-infra | 22:42 | |
| clarkb | new patchset shortly | 22:42 |
| *** e0ne has quit IRC | 22:43 | |
| *** maxbit has joined #openstack-infra | 22:43 | |
| ianw | sdague: around? | 22:43 |
| *** bknudson has quit IRC | 22:45 | |
| *** maxbit has quit IRC | 22:45 | |
| *** CaptTofu_ has quit IRC | 22:46 | |
| *** maxbit has joined #openstack-infra | 22:46 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Fix swift logserver prefixes. https://review.openstack.org/99498 | 22:46 |
| clarkb | mordred: now with actually working | 22:46 |
| *** thuc has quit IRC | 22:46 | |
| *** zzelle_ has quit IRC | 22:49 | |
| *** CaptTofu_ has joined #openstack-infra | 22:50 | |
| mattoliverau | Morning | 22:50 |
| clarkb | mattoliverau: o/ | 22:50 |
| *** jhesketh has quit IRC | 22:50 | |
| *** jhesketh has joined #openstack-infra | 22:51 | |
| jhesketh | Morning | 22:52 |
| clarkb | jhesketh: morning can you see 99498 eally quick? it sorts out some additional zuul swift derpage | 22:52 |
| jhesketh | clarkb: yep, was just looking at it | 22:52 |
| jhesketh | thanks for catching that | 22:52 |
| clarkb | thanks! | 22:52 |
| clarkb | jhesketh: well I ran an experimental job that wasn't going anywhere. After digging in zuul logs it became clear | 22:53 |
| clarkb | jhesketh: and my workaround didn't work because voluptuous hates me | 22:53 |
| clarkb | jhesketh: I feel like we are very close now though | 22:54 |
| jhesketh | clarkb: yeah, thanks for looking into it :-) | 22:55 |
| *** arnaud__ has joined #openstack-infra | 22:55 | |
| jhesketh | (lgtm) | 22:55 |
| *** doug-fish has left #openstack-infra | 22:55 | |
| *** e0ne has joined #openstack-infra | 22:55 | |
| *** arnaud__ has quit IRC | 22:56 | |
| *** maxbit has quit IRC | 22:57 | |
| *** maxbit has joined #openstack-infra | 22:57 | |
| *** thuc has joined #openstack-infra | 22:58 | |
| *** thuc_ has joined #openstack-infra | 22:59 | |
| *** thuc has quit IRC | 22:59 | |
| *** e0ne has quit IRC | 23:00 | |
| *** thuc_ has quit IRC | 23:00 | |
| *** jamielennox|away is now known as jamielennox | 23:00 | |
| *** thuc has joined #openstack-infra | 23:00 | |
| *** arnaud__ has joined #openstack-infra | 23:01 | |
| *** UtahDave has quit IRC | 23:03 | |
| *** cp16net has quit IRC | 23:05 | |
| *** e0ne has joined #openstack-infra | 23:09 | |
| *** maxbit has quit IRC | 23:10 | |
| *** e0ne_ has joined #openstack-infra | 23:11 | |
| *** e0ne has quit IRC | 23:11 | |
| *** maxbit has joined #openstack-infra | 23:14 | |
| *** e0ne_ has quit IRC | 23:15 | |
| *** CaptTofu_ has quit IRC | 23:16 | |
| sdague | ianw: just got back from a ride, what's up? | 23:16 |
| *** CaptTofu_ has joined #openstack-infra | 23:16 | |
| ianw | sdague: thanks, just wanted to ping you on https://review.openstack.org/#/c/99047/ ([Fedora] Devstack fails to install in firewalld less system) | 23:17 |
| ianw | sdague: causing gate issues because hpcloud & rackspace images differ | 23:17 |
| ianw | sdague: i've put in a new change per dtroyer's comments | 23:17 |
| sdague | jogo: yes, INFO should log all API requests | 23:17 |
| sdague | that's kind of the nature of info | 23:17 |
| sdague | jogo: re heat tests, large-ops isn't the problem. Any ops is the problem. | 23:18 |
| sdague | ianw: so there is a redhat bugzilla on it | 23:18 |
| ianw | sdague: https://bugzilla.redhat.com/show_bug.cgi?id=1099031 | 23:19 |
| uvirtbot | ianw: Error: Could not parse XML returned by bugzilla.redhat.com: HTTP Error 404: Not Found | 23:19 |
| ianw | uvirtbot: i guess bugzilla doesn't like you :) | 23:19 |
| uvirtbot | ianw: Error: "i" is not a valid command. | 23:19 |
| sdague | heh | 23:20 |
| sdague | ianw: ok, the patch you put in is ok with me. I would like to see things like that get fixed properly upstream though | 23:20 |
| *** CaptTofu_ has quit IRC | 23:21 | |
| mordred | clarkb: +A | 23:21 |
| *** thedodd has quit IRC | 23:21 | |
| *** yamahata has quit IRC | 23:21 | |
| *** thuc has quit IRC | 23:22 | |
| ianw | sdague: totally agree | 23:23 |
| *** thuc has joined #openstack-infra | 23:23 | |
| *** lcostantino has joined #openstack-infra | 23:23 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Fix swift logserver prefixes. https://review.openstack.org/99498 | 23:25 |
| *** thuc has quit IRC | 23:27 | |
| NobodyCam | question any reason a job would pass all the tests and then reset and start running all over? | 23:28 |
| NobodyCam | 93133 http://paste.openstack.org/show/PgPMbdRnkssVCWZiE8vu/ | 23:29 |
| *** maxbit has quit IRC | 23:29 | |
| NobodyCam | is now running again | 23:29 |
| *** maxbit has joined #openstack-infra | 23:29 | |
| *** dims__ has joined #openstack-infra | 23:30 | |
| *** CaptTofu_ has joined #openstack-infra | 23:32 | |
| jesusaurus | NobodyCam: 93133 appears to be going through the gate-queue tests. the tests that ran about 6 hours ago where check-queue tests | 23:33 |
| NobodyCam | jesusaurus: The paste is from the gate check just a few minutes ago | 23:34 |
| jesusaurus | that paste link is giving me a 500 | 23:34 |
| NobodyCam | it just passed and then started all over | 23:34 |
| *** chianingwang has quit IRC | 23:34 | |
| *** Sukhdev has quit IRC | 23:34 | |
| jesusaurus | it may be that a change ahead of your change failed, causing everything after the failed change to start again | 23:35 |
| NobodyCam | http://paste.openstack.org/show/PgPMbdRnkssVCWZiE8vu | 23:35 |
| *** maxbit has quit IRC | 23:37 | |
| NobodyCam | I'll bet 98928 starts to recheck too | 23:37 |
| NobodyCam | (but I wouldn't be that much) | 23:37 |
| *** shayneburgess has quit IRC | 23:37 | |
| *** asettle has joined #openstack-infra | 23:38 | |
| sdague | clarkb: it's really weird that we get to that 20k floor and get stuck | 23:45 |
| sdague | is it possible the extra workers from 2 days ago are hurting us? | 23:45 |
| sdague | because we were clearing out eventually | 23:45 |
| Alex_Gaynor | sdague: if you've got a sec, can you take a look at https://review.openstack.org/#/c/99467/ | 23:46 |
| *** chianingwang has joined #openstack-infra | 23:46 | |
| sdague | Alex_Gaynor: would that logic let us get rid of the rest of it? | 23:47 |
| Alex_Gaynor | sdague: hahaha, nope. FWIW here's what the whole thing looks like: https://github.com/django/django/blob/master/django/utils/module_loading.py#L110 | 23:48 |
| Alex_Gaynor | I don't htink all of that is actually needed for hacking's use case, but there's deifnitely a bunch of stuff | 23:48 |
| *** jhesketh has quit IRC | 23:48 | |
| *** briancurtin has joined #openstack-infra | 23:48 | |
| *** jhesketh has joined #openstack-infra | 23:49 | |
| clarkb | sdague: ya I think either that or we are queueing up too much stuff in the pipeline | 23:52 |
| clarkb | I will dive into it again tomorrow | 23:53 |
| Alex_Gaynor | sdague: thanks, (and thank god hacking has its own pipeline) | 23:53 |
| clarkb | there is clearly something off | 23:53 |
| mordred | Alex_Gaynor: :) | 23:54 |
| sdague | clarkb: ok, great. Thanks again | 23:54 |
| clarkb | sdague: no problem and I wish I had an answer for you | 23:54 |
| sdague | I wish we had all the answers. :) but such is not to be. | 23:56 |
| *** dkehn has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!