| jeblair | gah | 00:00 |
|---|---|---|
| mgagne | jeblair: you can have multiple "most recent" based on major/minor versions. | 00:00 |
| mgagne | jeblair: snapshots are linked to X.Y.Z version. | 00:00 |
| zaro | jeblair: you can have multiple recently built versions. foo-1.1-SNAPSHOT would be most recent for foo-1.1 | 00:00 |
| *** vipul-away is now known as vipul | 00:00 | |
| zaro | foo-2.1-SNAPSHOT would be most recent for foo-2.1 | 00:01 |
| *** mrodden has quit IRC | 00:01 | |
| fungi | it also seems like this workflow caters to projects which go very long periods accumulating new fixes/features without releasing in public. again less of a free-software-oriented methodology... it seems backward to optimize your release process for people who are going to use unreleased snapshots of your code anyway | 00:01 |
| jeblair | mgagne: right. that's only useful if you're doing simultaneous development on multiple revisions. that's not something we expect post-versioned projects to do generally. | 00:01 |
| *** ^demon has quit IRC | 00:01 | |
| mgagne | fungi: you assume this is the intention being this system. | 00:02 |
| jeblair | clarkb: i still don't understand your objection to the install script. | 00:02 |
| clarkb | jeblair: https://review.openstack.org/#/c/60581/6/modules/openstack_project/files/nodepool/scripts/convert_node_to_xenserver.sh line 194ish is where I think things get crazy | 00:02 |
| clarkb | it is botting into the xen isntaller which is going to overwrite the underlying image aiui | 00:03 |
| clarkb | *booting | 00:03 |
| mgagne | fungi: we use Maven extensively and we often release. Please don't mix intentions into the discussion. | 00:03 |
| jeblair | clarkb: that is one heck of a script. | 00:03 |
| clarkb | jeblair: yup took me a while to sort out where the "dd" happens | 00:03 |
| fungi | mgagne: then why the obsessiveness over versioning schemes for unreleased software? | 00:04 |
| fungi | mgagne: do you rebuild already-released artifacts? | 00:04 |
| mgagne | fungi: I believed the info provided here was enough to understand the workflow and how it is used. | 00:04 |
| zaro | it is what it is.. | 00:05 |
| clarkb | jeblair: I think this is the only real way to overcome this hurdle without a glance, but it seems very fragile | 00:06 |
| zaro | fungi: btw sometimes you do. | 00:06 |
| *** denis_makogon has quit IRC | 00:07 | |
| fungi | it may just be something i'm unable to align my thinking to. linux distributions have similarly struggled for many years with ways to provide reusable java-based applications in a consistent fashion | 00:07 |
| fungi | at least when it comes to integrating them with normal package-management methodologies, modularity/reusability and so on | 00:08 |
| pleia2 | clarkb: if you have time after reviews, time for my weekly https://etherpad.openstack.org/p/infra-pubs-cleanup nudge (also added a note about creating a branch for sysadmin-codereview - just started updating it for LCA) | 00:09 |
| *** krotscheck has joined #openstack-infra | 00:09 | |
| zaro | fungi: normal is probably not same for everyone :) | 00:09 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix format in installation.rst https://review.openstack.org/61548 | 00:10 |
| clarkb | pleia2: gah I completely forgot about that, I will do that in a minute | 00:10 |
| clarkb | fungi: https://review.openstack.org/#/c/61556/ may be of interest to you | 00:10 |
| harlowja | mordred u ok with https://review.openstack.org/#/c/52986/, should be done making changes there :) | 00:10 |
| harlowja | jeblair to, no more changes from me :) | 00:10 |
| clarkb | pleia2: so one thing I realized is that we should probably come up with a list of names for those talks. The existing dir names aren't very nice in many cases | 00:11 |
| pleia2 | clarkb: hah, yeah, "tutorial" is not optimal | 00:11 |
| clarkb | pleia2: for sysadmin-publications you don't want that to be based on tip of master? | 00:12 |
| pleia2 | actually, most aren't | 00:12 |
| *** dcramer_ has joined #openstack-infra | 00:12 | |
| *** ^d has joined #openstack-infra | 00:12 | |
| clarkb | er not master, the template branch | 00:12 |
| pleia2 | clarkb: hmm, will probably be cleaner if I use template, yes, thanks | 00:12 |
| pleia2 | I'm sure my version has strange files lurking about :) | 00:12 |
| clarkb | pleia2: ok, I will create that branch, but maybe we should wave that etherpad in front of the rest of -infra and get a list of names that we want to use for the other branches | 00:13 |
| pleia2 | clarkb: yeah, good idea, I'm thinking mailing list | 00:13 |
| pleia2 | I'll draft something up now | 00:13 |
| clarkb | pleia2: sounds good thanks (your new branch has been created) | 00:13 |
| pleia2 | thanks :) | 00:13 |
| fungi | clarkb: jeblair: on 61556 we had decided git protocol was okay unless it was in scripts or puppet modules which might get reused elsewhere (in which case http(s) was more appropriate)? | 00:13 |
| fungi | just trying to referesh my memory | 00:14 |
| clarkb | fungi: right, I am ok using git:// on slaves as forging sha1s that zuul hands to the slaves is hard | 00:14 |
| fungi | that's what i thought. lgtm | 00:14 |
| fungi | clarkb: i was more thinking we had discussed that proxies make life hard on people wanting to use git ptotocol | 00:15 |
| fungi | but same end result | 00:15 |
| *** dcramer_ has quit IRC | 00:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: Switch devstack checkout to use git.o.o https://review.openstack.org/61556 | 00:17 |
| *** pmathews has quit IRC | 00:22 | |
| *** fifieldt has joined #openstack-infra | 00:25 | |
| zaro | ohh crap! cracked my laptop screen. | 00:25 |
| pleia2 | oh no :( | 00:25 |
| mgagne | oh no! | 00:25 |
| mgagne | (oh yeah!) | 00:26 |
| fungi | that's definitely no good | 00:26 |
| zaro | pleia2: don't grab the camera area to pick up your laptop. | 00:26 |
| zaro | bad, real bad. | 00:26 |
| fungi | ooh, yeah, in general don't try to pick up a laptop by its screen or the surrounding bezel. that can flex in unfortunate ways | 00:26 |
| zaro | thinkpad didn't have that problme. | 00:27 |
| mgagne | how about by the headphone jack or the power plug | 00:27 |
| fungi | i like to pick them up by the escape key | 00:27 |
| fungi | that way they can't get away | 00:27 |
| mgagne | zaro: you could hammer a nail with a thinkpad, it isn't a fair comparison. | 00:28 |
| openstackgerrit | Don Dugger proposed a change to openstack-infra/config: Set repo URLs for gantt and ganttclient https://review.openstack.org/61616 | 00:28 |
| morganfainberg | fungi, can you bring a patchset back from abandoned? (related to the embargod patches from earlier today) | 00:28 |
| zaro | yeah, i was used to picking it up that way fro that thinkpad | 00:28 |
| *** AaronGr is now known as AaronGr_afk | 00:28 | |
| fungi | if zaro has one of those hp bricks, you can wait down your trunk in icy weather with those | 00:28 |
| fungi | morganfainberg: probably. checking now | 00:29 |
| morganfainberg | fungi, https://review.openstack.org/#/c/51973/ | 00:29 |
| morganfainberg | fungi, if you can. | 00:29 |
| fungi | mordred: restored | 00:29 |
| clarkb | zaro: don't get the toxic ooze on you | 00:29 |
| morganfainberg | fungi, thank you! | 00:29 |
| zaro | fungi: can admins do that in gerrit 2.4? | 00:30 |
| fungi | morganfainberg: also, restored. (sorry mordred) | 00:30 |
| *** dstanek has quit IRC | 00:30 | |
| fungi | zaro: only admins and change owners in 2.4. i think we can make an acl for that in later versions? | 00:30 |
| *** herndon has joined #openstack-infra | 00:30 | |
| zaro | yes, definately can. | 00:30 |
| *** dstanek has joined #openstack-infra | 00:30 | |
| fungi | that will be awesome | 00:30 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: add private-repository, java-opts, settings and global-settings to maven-target https://review.openstack.org/60161 | 00:31 |
| zaro | ok. i'm hosed. | 00:34 |
| * zaro looks for external monitor | 00:34 | |
| *** eharney has quit IRC | 00:36 | |
| *** shashank_ has joined #openstack-infra | 00:37 | |
| *** vipul is now known as vipul-away | 00:39 | |
| *** branen_ has joined #openstack-infra | 00:41 | |
| *** Hefeweizen has joined #openstack-infra | 00:42 | |
| *** dims has joined #openstack-infra | 00:43 | |
| *** Adri2000 has quit IRC | 00:44 | |
| *** branen has quit IRC | 00:45 | |
| *** yamahata_ has joined #openstack-infra | 00:45 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: make launchpad integration optional https://review.openstack.org/60952 | 00:47 |
| *** sdake_ has quit IRC | 00:49 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: make launchpad integration optional https://review.openstack.org/60952 | 00:51 |
| *** dizquierdo has quit IRC | 00:51 | |
| *** weshay has quit IRC | 00:53 | |
| clarkb | sdague: have you been following the os-loganalyze changes? | 00:54 |
| clarkb | sdague: they are fairly large :/ | 00:54 |
| sdague | honestly, I need to carve off a chunk of time for them | 00:55 |
| sdague | so the approach is in general good, it is a lot all at once. | 00:56 |
| sdague | also, I don't want to paper over some of the upstream errors, like filtering out lines that aren't useful | 00:56 |
| clarkb | I need to find a chunk of time and review them too. wanted to make sure they were on your radar | 00:56 |
| sdague | I was trying to make sure I wasn't being overly critical because it was someone new and they were changing all my logic :) | 00:57 |
| fungi | it looked like he'd commented on some. i want to carve off a nonexistent chunk of my time to test the console variants | 00:57 |
| jeblair | sdague: when that happens, just give them ownership of the project. :) | 00:57 |
| sdague | heh :) | 00:57 |
| fungi | when it breaks, they get to keep both pieces | 00:58 |
| sdague | yeh, the filter chaining is the right approach, and the only one that will help with letting it do console processing too | 00:58 |
| sdague | I just keep feeling there must be a way to do it a little clearer | 00:58 |
| *** yamahata_ has quit IRC | 00:59 | |
| clarkb | jeblair: so going back to the nodepool installation phases change. Is that something we might want to think about a little more or just accept it as a necessary evil? | 00:59 |
| *** yamahata_ has joined #openstack-infra | 01:00 | |
| *** ^d has quit IRC | 01:00 | |
| *** Adri2000 has joined #openstack-infra | 01:02 | |
| jeblair | clarkb: i don't feel opposed to giving nodepool more flexibility there. it sounds like you might have a beef with what the xenserver script is actually using that facility for though. if that's the issue, maybe concentrate on that? | 01:02 |
| jeblair | clarkb: in general, i think a lot of eyes should probably look at this. | 01:02 |
| clarkb | jeblair: my beef is with the reason for the change. We don't need to support this in nodepool if we don't support this fragile image building mecahnism | 01:03 |
| jeblair | clarkb: the only thing i feel (somewhat) strongly about is that if we have no public glance servers, and we can do this relatively simply with the tools we have, we shouldn't say that's a requirement. | 01:05 |
| jeblair | (shouldn't say public glance servers are a requirement) | 01:05 |
| clarkb | I think part of my problem is I am expecting this to be too much like using KVM | 01:07 |
| clarkb | eg I want this to run on a normal kernel and be installable like any other package, but that isn't possible because Xen | 01:07 |
| fungi | i feel like avoiding workarounds for a lack of glance services is unlikely to put any additional pressure on our providers to implement them, so we're probably not really helping ourselves (other than postponing contributions which would be simpler if we had $nonexistent_thing) | 01:08 |
| fungi | as for kernels, i think recent linux upstream source implements everything xen reasonably needs for kernel-space (so while you may need recompiled kernels, you don't need third-party patches any more at least) | 01:10 |
| *** vipul-away is now known as vipul | 01:10 | |
| clarkb | fungi: that is for guests, you still have to run the hypervisor as the special thing | 01:10 |
| clarkb | which is why this particular image build step is necessary, we are replacing linux with xenserver as the thing booted by rackspace | 01:11 |
| fungi | i thought with xen 3.0 and later linux 3.x kernels, you could do dom0 without special patches (but i could be mistaken) | 01:11 |
| clarkb | fungi: you can | 01:12 |
| fungi | now xenSERVER is another beast entirely | 01:12 |
| clarkb | but that isn't what we are working around here | 01:12 |
| *** jcooley_ has quit IRC | 01:12 | |
| *** jergerber has quit IRC | 01:13 | |
| fifieldt | so much scrollback | 01:17 |
| fifieldt | how was the gerrit upgrade? | 01:17 |
| fifieldt | or was it jenkins | 01:17 |
| * fifieldt cradles head | 01:17 | |
| fungi | fifieldt: how's... taiwan? | 01:18 |
| jeblair | fifieldt: the gerrit upgrade is a potentially long road; we're not sure how long, we're just at the beginning. | 01:18 |
| fifieldt | fungi, taiwan is awesome :) food everywhere | 01:18 |
| fungi | i'll have to check it out sometime | 01:19 |
| fifieldt | oh, so there's really not going to be a test server for a while then jeblair, ok | 01:19 |
| fifieldt | for sure | 01:19 |
| fifieldt | we're heading to taipei on monday to crank up the apartment hunt | 01:19 |
| jeblair | fifieldt: nice, i could get into that. :) | 01:19 |
| fungi | fifieldt: you're not planning to live with the poultry then? | 01:19 |
| fifieldt | only through until the weekend :) | 01:20 |
| fifieldt | but they are delicious | 01:20 |
| fifieldt | and there's this excellent coastal running route/bike track | 01:20 |
| fifieldt | oh, and interenet. need to find out where the AP is here so I can plug into it - maxing out the wireless in a speed test | 01:21 |
| *** salv-orlando_ has joined #openstack-infra | 01:24 | |
| *** salv-orlando has quit IRC | 01:27 | |
| *** salv-orlando has joined #openstack-infra | 01:27 | |
| *** salv-orlando_ has quit IRC | 01:28 | |
| *** jerryz has joined #openstack-infra | 01:28 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Enable patchset-created for #openstack-state-management channel https://review.openstack.org/61605 | 01:28 |
| *** ljjjustin has joined #openstack-infra | 01:30 | |
| *** salv-orlando_ has joined #openstack-infra | 01:33 | |
| *** jcooley_ has joined #openstack-infra | 01:33 | |
| *** mriedem has joined #openstack-infra | 01:34 | |
| *** salv-orlando_ has quit IRC | 01:34 | |
| *** salv-orlando__ has joined #openstack-infra | 01:34 | |
| *** nosnos has joined #openstack-infra | 01:35 | |
| *** pcrews has quit IRC | 01:36 | |
| *** salv-orlando has quit IRC | 01:36 | |
| *** salv-orlando__ is now known as salv-orlando | 01:36 | |
| *** yaguang has joined #openstack-infra | 01:39 | |
| *** dstanek has quit IRC | 01:41 | |
| *** yaguang has quit IRC | 01:44 | |
| *** praneshp has left #openstack-infra | 01:44 | |
| *** moted has quit IRC | 01:44 | |
| *** xchu has joined #openstack-infra | 01:46 | |
| *** dkranz has joined #openstack-infra | 01:46 | |
| *** bingbu has joined #openstack-infra | 01:46 | |
| *** dcramer_ has joined #openstack-infra | 01:46 | |
| *** pcrews has joined #openstack-infra | 01:52 | |
| *** rongze has joined #openstack-infra | 01:52 | |
| *** DennyZhang has joined #openstack-infra | 01:53 | |
| *** krotscheck has quit IRC | 01:58 | |
| *** yaguang has joined #openstack-infra | 02:01 | |
| clarkb | jeblair: I have reviewed https://review.openstack.org/#/c/60581/6/modules/openstack_project/files/nodepool/scripts/convert_node_to_xenserver.sh I feel a bit better about it after that. I think we can make the script much more clear and readable with some refactoring and better commenting. Which helps address some of my concners about fragility | 02:02 |
| *** moted has joined #openstack-infra | 02:04 | |
| *** rcarrillocruz has joined #openstack-infra | 02:06 | |
| *** dstanek has joined #openstack-infra | 02:06 | |
| *** guohliu has joined #openstack-infra | 02:09 | |
| *** rcarrillocruz has quit IRC | 02:11 | |
| clarkb | I need to buy mriedem beer | 02:13 |
| mriedem | yay! | 02:14 |
| clarkb | triaging all the things | 02:14 |
| mriedem | https://bugs.launchpad.net/tempest/+bug/1257561? | 02:14 |
| uvirtbot | Launchpad bug 1257561 in tempest "test_create_backup test not noticing that server has become active" [Undecided,Confirmed] | 02:14 |
| clarkb | mriedem: that and all of the other bug triage | 02:14 |
| mriedem | i'm going to push up a patch to at least make that message a bit more clear | 02:14 |
| mriedem | active == active but task_state:none != task_state:image_backup | 02:14 |
| mriedem | i was looking for a good e-r query on that one but it's failing in a utility method so not a good e-r query to pinpoint that single test fail | 02:15 |
| clarkb | yeah | 02:16 |
| clarkb | ok time for me to walk home | 02:16 |
| *** jcooley_ has quit IRC | 02:24 | |
| *** ryanpetrello has quit IRC | 02:24 | |
| *** rongze has quit IRC | 02:28 | |
| *** rongze has joined #openstack-infra | 02:29 | |
| *** vkozhukalov has joined #openstack-infra | 02:38 | |
| *** klrmn has joined #openstack-infra | 02:43 | |
| *** krotscheck has joined #openstack-infra | 02:48 | |
| *** reed has quit IRC | 02:48 | |
| *** krotscheck has quit IRC | 02:49 | |
| *** eharney has joined #openstack-infra | 02:51 | |
| *** dcramer_ has quit IRC | 02:54 | |
| *** pcrews has quit IRC | 02:57 | |
| *** julim has quit IRC | 02:57 | |
| *** jhesketh has quit IRC | 02:59 | |
| *** SushilKM has joined #openstack-infra | 03:06 | |
| *** esker has joined #openstack-infra | 03:13 | |
| *** shashank_ has quit IRC | 03:14 | |
| *** esker has quit IRC | 03:14 | |
| *** jcooley_ has joined #openstack-infra | 03:20 | |
| *** syerrapragada1 has joined #openstack-infra | 03:21 | |
| *** syerrapragada has quit IRC | 03:22 | |
| *** jcooley_ has quit IRC | 03:26 | |
| *** rongze_ has joined #openstack-infra | 03:33 | |
| *** jhesketh__ has quit IRC | 03:34 | |
| *** jhesketh has joined #openstack-infra | 03:36 | |
| *** rongze has quit IRC | 03:36 | |
| *** DennyZhang has quit IRC | 03:37 | |
| *** jhesketh_ has joined #openstack-infra | 03:40 | |
| *** mriedem has quit IRC | 03:59 | |
| *** sarob has joined #openstack-infra | 04:03 | |
| *** sdake_ has joined #openstack-infra | 04:05 | |
| *** sarob has quit IRC | 04:05 | |
| *** sarob has joined #openstack-infra | 04:06 | |
| *** marun has joined #openstack-infra | 04:06 | |
| *** AaronGr_afk is now known as AaronGr | 04:08 | |
| *** AaronGr is now known as AaronGr_Zzz | 04:09 | |
| *** fifieldt has quit IRC | 04:11 | |
| *** adalbas has quit IRC | 04:11 | |
| *** senk has joined #openstack-infra | 04:13 | |
| *** SushilKM has quit IRC | 04:17 | |
| *** senk has quit IRC | 04:17 | |
| *** senk has joined #openstack-infra | 04:18 | |
| *** senk has quit IRC | 04:22 | |
| *** pmathews has joined #openstack-infra | 04:23 | |
| openstackgerrit | Don Dugger proposed a change to openstack-infra/config: Add Gantt repositories. https://review.openstack.org/60380 | 04:24 |
| *** sarob has joined #openstack-infra | 04:24 | |
| *** pmathews has quit IRC | 04:24 | |
| *** jergerber has joined #openstack-infra | 04:24 | |
| *** sarob has quit IRC | 04:28 | |
| *** senk has joined #openstack-infra | 04:29 | |
| *** chandankumar has joined #openstack-infra | 04:36 | |
| *** jhesketh_ has quit IRC | 04:36 | |
| *** jergerber has quit IRC | 04:39 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: cloverphp publisher https://review.openstack.org/52940 | 04:40 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Enable ignore_cache flag on jenkins_jobs.ini https://review.openstack.org/59515 | 04:41 |
| *** dstanek has quit IRC | 04:43 | |
| *** dkliban has quit IRC | 04:46 | |
| *** saper has quit IRC | 04:46 | |
| *** rongze_ has quit IRC | 04:52 | |
| *** herndon has quit IRC | 04:54 | |
| openstackgerrit | Sean Roberts proposed a change to openstack-infra/config: added jenkins jobs to stackforge project milk https://review.openstack.org/61656 | 04:55 |
| *** senk has quit IRC | 04:55 | |
| *** sarob has joined #openstack-infra | 04:57 | |
| *** gyee has quit IRC | 04:58 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/gerritlib: Allow startWatching to take in watcher connection limits https://review.openstack.org/61657 | 05:00 |
| *** yolanda has joined #openstack-infra | 05:03 | |
| *** nati_ueno has quit IRC | 05:03 | |
| *** talluri has joined #openstack-infra | 05:06 | |
| openstackgerrit | Sean Roberts proposed a change to openstack-infra/config: added jenkins jobs to stackforge project milk modified indent in layout.yaml https://review.openstack.org/61656 | 05:06 |
| *** talluri_ has joined #openstack-infra | 05:08 | |
| *** talluri has quit IRC | 05:08 | |
| *** talluri_ has quit IRC | 05:08 | |
| *** dkliban has joined #openstack-infra | 05:08 | |
| *** talluri has joined #openstack-infra | 05:08 | |
| *** sarob has quit IRC | 05:10 | |
| *** sarob has joined #openstack-infra | 05:10 | |
| *** sarob has quit IRC | 05:11 | |
| *** yongli_away is now known as yongli | 05:14 | |
| *** SergeyLukjanov has joined #openstack-infra | 05:17 | |
| *** fifieldt has joined #openstack-infra | 05:20 | |
| *** rongze has joined #openstack-infra | 05:22 | |
| *** sarob has joined #openstack-infra | 05:23 | |
| *** sarob has quit IRC | 05:25 | |
| *** sarob has joined #openstack-infra | 05:26 | |
| *** sarob has quit IRC | 05:26 | |
| *** rongze has quit IRC | 05:27 | |
| *** sdake_ has quit IRC | 05:32 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 05:33 | |
| *** _SergeyLukjanov has quit IRC | 05:33 | |
| *** elasticio has joined #openstack-infra | 05:36 | |
| *** elasticio has quit IRC | 05:42 | |
| *** SergeyLukjanov has joined #openstack-infra | 05:51 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add Savanna d-g jobs to tempest exp pipeline https://review.openstack.org/61125 | 05:53 |
| *** dkliban has quit IRC | 05:56 | |
| *** zhiyan has joined #openstack-infra | 06:00 | |
| *** jcooley_ has joined #openstack-infra | 06:06 | |
| *** odyssey4me has joined #openstack-infra | 06:14 | |
| *** SergeyLukjanov has quit IRC | 06:17 | |
| *** rongze has joined #openstack-infra | 06:20 | |
| *** vipul has quit IRC | 06:22 | |
| *** vipul has joined #openstack-infra | 06:22 | |
| *** jcooley_ has quit IRC | 06:30 | |
| *** jcooley_ has joined #openstack-infra | 06:30 | |
| *** jcooley_ has quit IRC | 06:31 | |
| *** DinaBelova has joined #openstack-infra | 06:31 | |
| *** jcooley_ has joined #openstack-infra | 06:31 | |
| *** DinaBelova has quit IRC | 06:33 | |
| *** SushilKM has joined #openstack-infra | 06:38 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: allow configdrive vs. metadata server to be configured https://review.openstack.org/59303 | 06:44 |
| *** lifeless has quit IRC | 06:48 | |
| *** xchu has quit IRC | 06:48 | |
| *** vkozhukalov has quit IRC | 06:51 | |
| *** nicedice has quit IRC | 07:00 | |
| *** xchu has joined #openstack-infra | 07:06 | |
| *** yongli has quit IRC | 07:06 | |
| *** zhiyan has quit IRC | 07:06 | |
| *** guohliu has quit IRC | 07:06 | |
| *** odyssey4me has quit IRC | 07:09 | |
| *** zhiyan has joined #openstack-infra | 07:14 | |
| *** lifeless has joined #openstack-infra | 07:15 | |
| *** fifieldt has quit IRC | 07:17 | |
| *** odyssey4me has joined #openstack-infra | 07:17 | |
| *** jcooley_ has quit IRC | 07:18 | |
| *** guohliu has joined #openstack-infra | 07:19 | |
| *** jcooley_ has joined #openstack-infra | 07:22 | |
| *** jcoufal has joined #openstack-infra | 07:27 | |
| *** NikitaKonovalov has joined #openstack-infra | 07:27 | |
| *** paul-- has quit IRC | 07:30 | |
| *** guohliu has quit IRC | 07:34 | |
| *** boris-42 has quit IRC | 07:43 | |
| *** Loquacities has quit IRC | 07:44 | |
| *** jcooley_ has quit IRC | 07:46 | |
| *** wenlock has joined #openstack-infra | 07:47 | |
| *** guohliu has joined #openstack-infra | 07:47 | |
| *** rcarrillocruz has joined #openstack-infra | 07:49 | |
| *** jcooley_ has joined #openstack-infra | 07:50 | |
| *** nati_ueno has joined #openstack-infra | 07:52 | |
| *** praneshp has joined #openstack-infra | 07:54 | |
| praneshp | I need some assistance with https://review.openstack.org/#/c/61615/ | 07:55 |
| *** Loquacities has joined #openstack-infra | 07:55 | |
| *** mkoderer has quit IRC | 07:55 | |
| praneshp | I keep seeing jenkins failure wiht error: 'source_dir' must be a directory name (got `/home/jenkins/workspace/gate-entropy-docs/docs`) | 07:55 |
| *** wenlock_ has joined #openstack-infra | 07:56 | |
| *** wenlock has quit IRC | 07:56 | |
| *** wenlock_ is now known as wenlock | 07:56 | |
| *** belliott has quit IRC | 07:56 | |
| praneshp | python setup.py build_sphinx fails with the same error but not sue what's wrong. docs is a directory. | 07:56 |
| *** belliott has joined #openstack-infra | 07:57 | |
| *** mkoderer_ has joined #openstack-infra | 07:57 | |
| *** jcooley_ has quit IRC | 07:58 | |
| *** ljjjustin has quit IRC | 08:02 | |
| *** rongze has quit IRC | 08:02 | |
| *** rongze has joined #openstack-infra | 08:06 | |
| *** rongze has quit IRC | 08:11 | |
| *** zhiyan has quit IRC | 08:15 | |
| *** denis_makogon has joined #openstack-infra | 08:19 | |
| *** yongli has joined #openstack-infra | 08:22 | |
| *** zhiyan has joined #openstack-infra | 08:26 | |
| *** vkozhukalov has joined #openstack-infra | 08:28 | |
| *** wenlock has quit IRC | 08:38 | |
| *** paul-- has joined #openstack-infra | 08:40 | |
| *** fbo_away is now known as fbo | 08:42 | |
| *** xchu has quit IRC | 08:43 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Pin sphinx to <1.2 https://review.openstack.org/61288 | 08:45 |
| *** nprivalova has joined #openstack-infra | 08:46 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: update documentation with specific of rax setup https://review.openstack.org/61301 | 08:48 |
| *** fifieldt has joined #openstack-infra | 08:49 | |
| *** afazekas has joined #openstack-infra | 08:51 | |
| *** hashar has joined #openstack-infra | 08:54 | |
| *** xchu has joined #openstack-infra | 08:55 | |
| *** praneshp has quit IRC | 08:57 | |
| *** yassine has joined #openstack-infra | 08:59 | |
| *** jhesketh has quit IRC | 09:03 | |
| *** jpich has joined #openstack-infra | 09:05 | |
| *** salv-orlando has quit IRC | 09:05 | |
| *** derekh has joined #openstack-infra | 09:05 | |
| notmyname | jog0: clarkb: jeblair: http://not.mn/gate_status.html | 09:06 |
| *** sharwell has quit IRC | 09:09 | |
| *** paul-- has quit IRC | 09:16 | |
| *** senk has joined #openstack-infra | 09:16 | |
| *** ekarlso has quit IRC | 09:17 | |
| *** ekarlso has joined #openstack-infra | 09:17 | |
| *** guohliu has quit IRC | 09:19 | |
| jog0 | notmyname: neato | 09:20 |
| notmyname | jog0: open the image in a new tab and change the from parameter to -168hours | 09:21 |
| notmyname | ie one week | 09:21 |
| *** yassine has quit IRC | 09:21 | |
| jog0 | notmyname: or just one -7days | 09:21 |
| *** jooools has joined #openstack-infra | 09:21 | |
| jog0 | what is the equation for overall? | 09:21 |
| *** yassine has joined #openstack-infra | 09:22 | |
| notmyname | jog0: I contend my graph shows the status of things more accurately than yours on the elastic-recheck page. I started from yours, but I think there are a couple of important changes | 09:22 |
| *** yassine has quit IRC | 09:22 | |
| jog0 | notmyname: checking now | 09:22 |
| jog0 | for starters the 1 hour window is too small | 09:23 |
| notmyname | e important difference is that is uses the graphite summarize() function to bucket the events into time slots (1 hour buckets in this case)--as opposed to your 5 hour moving average. the overall is the multiplication of the serice | 09:23 |
| jog0 | what is smartsummarize | 09:23 |
| *** yassine has joined #openstack-infra | 09:23 | |
| notmyname | why is 1 hour too small? | 09:23 |
| *** jorisroovers has joined #openstack-infra | 09:23 | |
| *** yassine has quit IRC | 09:23 | |
| jog0 | too small of a data set | 09:23 |
| jog0 | how many gatebs in an hour | 09:23 |
| jog0 | notmyname: what timezone are you in? | 09:24 |
| notmyname | pacific (I shouldn't be awake) | 09:24 |
| jog0 | notmyname: heh, I am in UTC+1 so I should be awake | 09:24 |
| notmyname | that is actually one question I had. how often do the statsd events get generated | 09:24 |
| *** yassine has joined #openstack-infra | 09:24 | |
| jog0 | notmyname: let me find the graph to answer that | 09:25 |
| *** yassine has quit IRC | 09:25 | |
| *** jishaomei has joined #openstack-infra | 09:25 | |
| *** yassine has joined #openstack-infra | 09:26 | |
| notmyname | btw, a five hour bucket has the same shape to it | 09:27 |
| *** jorisroovers has quit IRC | 09:27 | |
| notmyname | also, is there a counter for "jobs that have been reset"? I'd like to graph that too | 09:28 |
| jog0 | http://graphite.openstack.org/graphlot/?from=-24hour&until=-0hour&target=hitcount(%20sum(stats.zuul.pipeline.gate.job.gate-tempest-dsvm-full.{SUCCESS,FAILURE}),%225hours%22) | 09:28 |
| jog0 | notmyname: pastebin of URL http://paste.openstack.org/show/54868/ | 09:29 |
| jog0 | notmyname: I didn't notice that one before | 09:29 |
| jog0 | notmyname: ++ to reset graph | 09:29 |
| jog0 | git.openstack.org/cgit/openstack-infra/elastic-recheck/ | 09:30 |
| jog0 | notmyname: so 1 hour hitcount is < 15 jobs over last 24 hours | 09:31 |
| notmyname | I don't think that's right, but let me go read something first | 09:32 |
| jog0 | notmyname: so when I switched the overall success rate to 5 hours it looks a little better | 09:33 |
| jog0 | notmyname: in short, awesome thanks for doing this. If you think your graph is better (which I think it is) you can push a patch to the repo above | 09:34 |
| notmyname | my thinking around using 1 hour is that the jobs take about 1 hour to run (or 45-50 minutes ish) | 09:34 |
| jog0 | I really like the overalls success rate line | 09:34 |
| jog0 | I need to read up on smar tsummeraize | 09:34 |
| *** marun has quit IRC | 09:36 | |
| jog0 | notmyname: so long term I am imagining status.openstack.org/elastic-recheck/ has 1 or two graphs on it, and if you click a button or link you can get a whole bunch more graphs at different granularities etc | 09:36 |
| *** SushilKM has quit IRC | 09:37 | |
| jog0 | 'Smarter experimental version of summarize. | 09:37 |
| jog0 | Smarter experimental version of summarize. | 09:37 |
| jog0 | thats very descriptive | 09:37 |
| *** marun has joined #openstack-infra | 09:37 | |
| *** flaper87|afk is now known as flaper87 | 09:40 | |
| *** fifieldt has quit IRC | 09:40 | |
| *** dizquierdo has joined #openstack-infra | 09:40 | |
| *** SushilKM has joined #openstack-infra | 09:43 | |
| *** denis_makogon has quit IRC | 09:44 | |
| notmyname | jog0: updated for 5 hour bucket over the last week. (the point stays the same--we've got a _lot_ of gate failures that make integration hard) | 09:45 |
| notmyname | jog0: http://not.mn/gate_status.html | 09:45 |
| notmyname | and now I need to go to bed :-) | 09:46 |
| jog0 | notmyname: can you put a path up with your graph | 09:52 |
| jog0 | notmyname: and yes, this is why I am leaning on people to fix the gate because its hella bad | 09:52 |
| jog0 | notmyname: night, see you in a few hours ! | 09:52 |
| jog0 | online thatis | 09:52 |
| *** johnthetubaguy has joined #openstack-infra | 09:52 | |
| *** johnthetubaguy1 has joined #openstack-infra | 09:54 | |
| *** johnthetubaguy has quit IRC | 09:57 | |
| *** jishaomei has quit IRC | 09:58 | |
| *** che-arne has joined #openstack-infra | 09:59 | |
| *** paul-- has joined #openstack-infra | 09:59 | |
| *** paul-- has quit IRC | 10:03 | |
| *** ativelkov has quit IRC | 10:03 | |
| *** chandankumar_ has joined #openstack-infra | 10:05 | |
| *** apevec has joined #openstack-infra | 10:05 | |
| apevec | mordred, fungi - Swift so special it must be removed from automatic updates from global? https://review.openstack.org/61539 | 10:06 |
| chmouel | apevec: swift is targetting vanilla precise | 10:08 |
| *** chandankumar has quit IRC | 10:08 | |
| apevec | chmouel, ok, I just wonder how that will all work in gate | 10:08 |
| apevec | we'll end up in version conflict like we have now in grizzly | 10:09 |
| apevec | actually, looking now at check results, Swift stable/havana has bitrotted for other reasons | 10:12 |
| apevec | even gate-swift-pep8 fails | 10:12 |
| *** AlexF has joined #openstack-infra | 10:14 | |
| *** xchu has quit IRC | 10:14 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Always run check-tempest-dsvm-full twice https://review.openstack.org/57347 | 10:15 |
| *** thingee has quit IRC | 10:15 | |
| *** nosnos_ has joined #openstack-infra | 10:15 | |
| *** thingee has joined #openstack-infra | 10:15 | |
| *** nosnos_ has quit IRC | 10:16 | |
| *** nosnos_ has joined #openstack-infra | 10:16 | |
| *** nosnos has quit IRC | 10:17 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:17 | |
| *** ativelkov has joined #openstack-infra | 10:19 | |
| *** nosnos_ has quit IRC | 10:20 | |
| *** odyssey4me has quit IRC | 10:23 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add git2lp mapping for stevedore https://review.openstack.org/61696 | 10:28 |
| *** nprivalova has quit IRC | 10:28 | |
| *** AlexF has quit IRC | 10:28 | |
| *** senk has quit IRC | 10:30 | |
| *** paul-- has joined #openstack-infra | 10:31 | |
| *** paul-- has quit IRC | 10:33 | |
| *** salv-orlando has joined #openstack-infra | 10:34 | |
| *** ativelkov has left #openstack-infra | 10:39 | |
| *** rongze has joined #openstack-infra | 10:41 | |
| *** salv-orlando has quit IRC | 10:42 | |
| *** NikitaKonovalov has quit IRC | 10:42 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add graph for gate hit count https://review.openstack.org/61429 | 10:44 |
| *** nprivalova has joined #openstack-infra | 10:45 | |
| *** rongze has quit IRC | 10:46 | |
| *** heyongli has joined #openstack-infra | 10:46 | |
| *** johnthetubaguy1 is now known as johnthetubaguy | 10:47 | |
| *** bingbu has quit IRC | 10:48 | |
| *** yolanda has quit IRC | 10:55 | |
| *** nprivalova has quit IRC | 10:57 | |
| *** lcestari has joined #openstack-infra | 10:58 | |
| *** yamahata_ has quit IRC | 10:58 | |
| *** hashar is now known as hasharAway | 11:00 | |
| openstackgerrit | Serg Melikyan proposed a change to openstack-infra/config: Add eavesdrop to monitor Murano channel https://review.openstack.org/61703 | 11:04 |
| *** yolanda has joined #openstack-infra | 11:09 | |
| jog0 | jeblair: FYI https://review.openstack.org/#/c/59152/ | 11:11 |
| *** hasharAway is now known as hashar | 11:14 | |
| SergeyLukjanov | jog0, oh, nice catch, it could be probably really surprising | 11:17 |
| *** markmc has joined #openstack-infra | 11:17 | |
| *** boris-42 has joined #openstack-infra | 11:17 | |
| jog0 | SergeyLukjanov: the pep8 stuff? | 11:17 |
| SergeyLukjanov | jog0, yep | 11:18 |
| jog0 | SergeyLukjanov: yeah the pep8 tool is strange | 11:18 |
| jog0 | turns out pep8 is self contradicting these days | 11:18 |
| *** hashar_ has joined #openstack-infra | 11:18 | |
| jog0 | http://pep8.readthedocs.org/en/latest/intro.html#error-codes | 11:19 |
| SergeyLukjanov | jog0, yeah... | 11:19 |
| *** hashar has quit IRC | 11:20 | |
| *** hashar_ is now known as hashar | 11:20 | |
| *** branen_ has quit IRC | 11:28 | |
| *** rossella_s has joined #openstack-infra | 11:34 | |
| *** adalbas has joined #openstack-infra | 11:36 | |
| *** hashar has quit IRC | 11:42 | |
| *** salv-orlando has joined #openstack-infra | 11:43 | |
| *** salv-orlando has quit IRC | 11:43 | |
| *** talluri has quit IRC | 11:52 | |
| BobBall | Anyone know how to upload a commit for havana? https://wiki.openstack.org/wiki/StableBranch suggests "git review stable/havana" - but I get an error that the branch isn't found | 11:53 |
| apevec | BobBall, which project? | 11:55 |
| BobBall | nova | 11:55 |
| apevec | if you checked out stable/havan, .gitreview should point to the right branch | 11:55 |
| apevec | so just git-review works | 11:55 |
| BobBall | heh - even better :) | 11:56 |
| BobBall | bop | 11:56 |
| BobBall | nope* | 11:56 |
| BobBall | fatal: ambiguous argument 'stable/havana': unknown revision or path not in the working tree. | 11:56 |
| BobBall | that's after just "git review" | 11:56 |
| BobBall | http://paste.openstack.org/show/54882/ | 11:57 |
| *** weshay has joined #openstack-infra | 11:58 | |
| apevec | BobBall, how did you create local stable/havana branch? | 11:59 |
| *** talluri has joined #openstack-infra | 12:02 | |
| mordred | BobBall: what version of git review are you running? | 12:03 |
| *** senk has joined #openstack-infra | 12:04 | |
| mordred | apevec: it's compllicated. | 12:04 |
| BobBall | very old version of git-review no doubt :) | 12:05 |
| BobBall | just upgrading | 12:06 |
| mordred | BobBall: :) | 12:06 |
| *** talluri has quit IRC | 12:06 | |
| BobBall | Was on 1.20 | 12:06 |
| BobBall | that worked | 12:06 |
| BobBall | I love OpenStack... | 12:06 |
| BobBall | if at first you don't succeed, upgrade! | 12:06 |
| mordred | BobBall: fyi, if you ever can't upgrade git-review for some reason, you can always run "git push gerrit HEAD:refs/publish/stable/havana" | 12:07 |
| mordred | which is what git review is doing under the covers | 12:07 |
| apevec | mordred, while at it, swift stable/havana doesn't have defaultbranch=stable/havana in .gitreview - should I add it or better not mess w/ Swift feud? | 12:07 |
| mordred | apevec: I think it's probably fine to add - it shouldn't cause any issues | 12:07 |
| apevec | ok, I'll propose | 12:08 |
| mordred | apevec: thanks! | 12:09 |
| *** ArxCruz has joined #openstack-infra | 12:09 | |
| jog0 | FYI: gate failure rate is under 25% again | 12:10 |
| mordred | woot | 12:10 |
| jog0 | its 22% | 12:10 |
| *** talluri has joined #openstack-infra | 12:10 | |
| apevec | jog0, what happened?? | 12:11 |
| apevec | :) | 12:11 |
| apevec | mordred, https://review.openstack.org/61714 https://review.openstack.org/61715 | 12:11 |
| apevec | the latter since John doesn't want full "updated from global" | 12:12 |
| mordred | apevec: awesome. to his point, don't we also need the tox.ini change in there/ | 12:12 |
| mordred | ? | 12:12 |
| apevec | right, that wasn't proposed | 12:13 |
| maurosr | guys I'm kind of new on infra stuff, I heard about it but where do I find information about some bug being hit on the gate? I want to know when something started | 12:13 |
| mordred | maurosr: jog0 is your man | 12:13 |
| jog0 | maurosr: http://status.openstack.org/elastic-recheck/ | 12:13 |
| maurosr | jog0, mordred : tks guys! | 12:14 |
| jog0 | apevec: no one thing in particular just lots of things | 12:14 |
| jog0 | apevec: although there was this the other day https://twitter.com/sdague/status/410873451960733696/photo/1 | 12:14 |
| jog0 | apevec: and this too https://twitter.com/sdague/status/408326773210742784/photo/1/large | 12:15 |
| jog0 | apevec: and now the gate queue is 38 and west coast isn't up yet | 12:15 |
| jog0 | maurosr: whats the bug number? | 12:15 |
| jog0 | we use logstash.openstack.org to find out when a bug started (although we only have two weeks of logs) | 12:15 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add git2lp mapping for all Murano projects https://review.openstack.org/61716 | 12:16 |
| maurosr | jog0: 1259907 grenade => failling to attach volume | 12:16 |
| sdague | morning folks | 12:17 |
| maurosr | hit it locally few minutes ago, trying to figure out if it's fixed | 12:17 |
| jog0 | sdague: morning, notmyname was up late hacking on graphite graphs | 12:17 |
| jog0 | http://not.mn/gate_status.html | 12:17 |
| sdague | yeh I was looking at scrollback | 12:17 |
| sdague | I think that's got the same artifacting problem when the sample size is too low | 12:18 |
| sdague | if it goes immediately from 0% pass to 100% pass, it's wrong, the buckets are too small | 12:18 |
| jog0 | sdague: I think moving average is better b/c of that | 12:19 |
| sdague | right | 12:19 |
| jog0 | but the total gate graph thing is awesome | 12:20 |
| jog0 | maurosr: its hard to say when 1259907 started because we don't have a good fingerprint for it | 12:20 |
| jog0 | to search logstash | 12:20 |
| sdague | you can in theory do moving average over events instead of hours, but the results didn't seem right when I did that | 12:21 |
| jog0 | sdague: yeah I saw the same thing | 12:21 |
| sdague | jog0: how do I take a graphite graph back into something where I can actually understand the series and functions better? | 12:21 |
| jog0 | which is why I have the hit count graph | 12:21 |
| maurosr | jog0: np, tks anyway, at least I know there is such tools now, tks | 12:22 |
| jog0 | sdague: instead of render in the url | 12:22 |
| jog0 | type in graphplot | 12:22 |
| *** rfolco has joined #openstack-infra | 12:23 | |
| jog0 | so http://graphite.openstack.org/render/?from => http://graphite.openstack.org/graphplot/?from | 12:23 |
| *** paul-- has joined #openstack-infra | 12:23 | |
| *** kruskakli has joined #openstack-infra | 12:24 | |
| kruskakli | howdy, how can I setup a dedicated user for my gerrit trigger plugin? | 12:25 |
| *** dstanek has joined #openstack-infra | 12:27 | |
| *** paul-- has quit IRC | 12:27 | |
| sdague | jog0: I don't think the math is right on the graph | 12:27 |
| sdague | it's just multiplying success rates, which means that overall graph is going to show a lower rate because it won't account for 2 fails on one item | 12:28 |
| jog0 | 2 fails on one item? | 12:30 |
| *** zhiyan has quit IRC | 12:32 | |
| jog0 | sdague: also can you respond to my comments on https://bugs.launchpad.net/tempest/+bug/1253896 | 12:33 |
| uvirtbot | Launchpad bug 1253896 in tempest "test_minimum_basic_scenario fails with SSHException: Error reading SSH protocol banner" [Critical,Confirmed] | 12:33 |
| jog0 | sdague: that bug got worse recently and it may be metadata server / config drive related for neutron | 12:34 |
| sdague | jog0: if devstack-full and grenade fail | 12:34 |
| jog0 | sdague: if we assume all gate jobs are independant (which they are not) then I *think* this math is right | 12:35 |
| *** saper has joined #openstack-infra | 12:36 | |
| sdague | jog0: hmmm... by calculus? | 12:36 |
| sdague | maybe I'll buy that | 12:36 |
| sdague | it doesn't, however track the other jobs in the gate | 12:37 |
| sdague | which would also contribute | 12:37 |
| jog0 | sdague: yup its a lower bound | 12:37 |
| sdague | so http://logstash.openstack.org/#eyJmaWVsZHMiOltdLCJzZWFyY2giOiJtZXNzYWdlOlwiU1NIVGltZW91dDogQ29ubmVjdGlvbiB0byB0aGVcIiBBTkQgbWVzc2FnZTpcInZpYSBTU0ggdGltZWQgb3V0LlwiIEFORCBmaWxlbmFtZTpcImNvbnNvbGUuaHRtbFwiIiwidGltZWZyYW1lIjoiODY0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsIm9mZnNldCI6MCwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwibW9kZSI6InRlcm1zIiwiYW5hbHl6ZV9maWVsZCI6ImJ1aWxkX25hbWUiLCJzdGFtcCI6MTM4Njg1MTc1Mzk5NH0= | 12:37 |
| jog0 | err upper | 12:37 |
| sdague | is 24hrs | 12:37 |
| *** rongze has joined #openstack-infra | 12:37 | |
| sdague | the neutron-pg job is not on metadata server | 12:38 |
| sdague | that being said, this could just be a contributing factor | 12:38 |
| jog0 | yeah | 12:39 |
| jog0 | anyway gtg, lunch time | 12:39 |
| sdague | I'm still not convinced on the overall math though, because not all gate jobs run the same things | 12:40 |
| *** dims has quit IRC | 12:41 | |
| *** rongze has quit IRC | 12:41 | |
| *** dims has joined #openstack-infra | 12:41 | |
| jog0 | sdague: agreed, I think we can present a upper bound for how well the gate is for most cases | 12:41 |
| sdague | maybe, I think it needs more thought | 12:42 |
| jog0 | sdague: agreed, but I like the idea in general | 12:42 |
| sdague | and, honestly, will it change the fact that no one is looking at bugs? | 12:42 |
| jog0 | anyway I'll be back later. if you can poke at 1253896 that would be great, as its the top bug now | 12:43 |
| jog0 | sdague: I do think having a nice number we can point to for how bad it is will help convey the importance | 12:43 |
| BobBall | Is there a syntax to reverify on two bugs? Should I just add two comments or randomly pick one of them? | 12:44 |
| BobBall | recheck* | 12:45 |
| openstackgerrit | Serg Melikyan proposed a change to openstack-infra/config: Add eavesdrop to monitor Murano channel https://review.openstack.org/61703 | 12:48 |
| sdague | jog0: honestly, I can't start every morning digging into the top gate bug. I actually have other things to do. :) | 12:48 |
| mordred | BobBall: yeah- it only works for one bug number right now | 12:48 |
| BobBall | k | 12:48 |
| BobBall | also - does e-r see when I claim a different bug should be rechecked to the one it said? | 12:48 |
| mordred | fungi: you up this morning yet? | 12:49 |
| mordred | BobBall: I do not believe so | 12:49 |
| BobBall | I think the match for 1253896 is also catching 1254752 by mistake | 12:49 |
| *** dstanek has quit IRC | 12:50 | |
| *** prad has quit IRC | 12:50 | |
| *** bpokorny has joined #openstack-infra | 12:51 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 12:54 | |
| *** _SergeyLukjanov has quit IRC | 12:54 | |
| *** sharwell_ has joined #openstack-infra | 12:55 | |
| *** yamahata_ has joined #openstack-infra | 12:55 | |
| *** senk has quit IRC | 12:57 | |
| *** senk has joined #openstack-infra | 12:57 | |
| *** sdake_ has joined #openstack-infra | 12:58 | |
| *** yamahata_ has quit IRC | 13:02 | |
| *** senk has quit IRC | 13:02 | |
| *** senk has joined #openstack-infra | 13:04 | |
| *** yamahata_ has joined #openstack-infra | 13:04 | |
| *** NikitaKonovalov has joined #openstack-infra | 13:06 | |
| *** jcoufal_ has joined #openstack-infra | 13:08 | |
| *** yamahata_ has quit IRC | 13:09 | |
| *** jcoufal has quit IRC | 13:10 | |
| *** rongze has joined #openstack-infra | 13:12 | |
| *** dims has quit IRC | 13:17 | |
| *** Abhishek_ has joined #openstack-infra | 13:17 | |
| *** dkliban has joined #openstack-infra | 13:19 | |
| *** dims has joined #openstack-infra | 13:19 | |
| *** mriedem has joined #openstack-infra | 13:19 | |
| *** thomasem has joined #openstack-infra | 13:22 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:23 | |
| *** yaguang has quit IRC | 13:24 | |
| *** yamahata_ has joined #openstack-infra | 13:24 | |
| *** che-arne has quit IRC | 13:24 | |
| *** che-arne has joined #openstack-infra | 13:25 | |
| *** bpokorny has quit IRC | 13:27 | |
| *** chandankumar_ has quit IRC | 13:29 | |
| *** hashar has joined #openstack-infra | 13:29 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack/requirements: Add oslo.rootwrap to global requirements https://review.openstack.org/61738 | 13:30 |
| *** chandankumar has joined #openstack-infra | 13:30 | |
| *** smarcet has joined #openstack-infra | 13:31 | |
| *** smarcet has left #openstack-infra | 13:32 | |
| *** smarcet has joined #openstack-infra | 13:32 | |
| *** rcarrillocruz1 has joined #openstack-infra | 13:32 | |
| *** zhiyan has joined #openstack-infra | 13:32 | |
| *** rcarrillocruz has quit IRC | 13:33 | |
| *** nprivalova has joined #openstack-infra | 13:33 | |
| *** yamahata_ has quit IRC | 13:35 | |
| *** dkranz is now known as dkranz-in-transi | 13:38 | |
| *** yamahata_ has joined #openstack-infra | 13:40 | |
| *** dkranz-in-transi has quit IRC | 13:41 | |
| openstackgerrit | Chuck Short proposed a change to openstack/requirements: Bump webob requirements to include 1.3 https://review.openstack.org/61742 | 13:43 |
| *** dstanek has joined #openstack-infra | 13:44 | |
| apevec | fungi, mordred - https://review.openstack.org/#/q/Ida5e440d1bdb9f8e9031277ea53a02d2ef171438,n,z was not proposed for Glance ? | 13:45 |
| *** nprivalova has quit IRC | 13:46 | |
| apevec | looks like it also has competition like https://review.openstack.org/60834 (didn't keep same changeid across project so hard to say where all it was proposed) | 13:47 |
| apevec | such CI fixes should've been just force pushed ... | 13:47 |
| *** chuck_ has joined #openstack-infra | 13:49 | |
| apevec | fungi, so to unblock stable/havana and merge CVE fixes, we need | 13:51 |
| apevec | 1) https://review.openstack.org/#/q/status:open+branch:stable/havana+topic:openstack/requirements,n,z merged | 13:51 |
| apevec | 2) tox.ini update in stable/havana branches | 13:51 |
| *** bpokorny has joined #openstack-infra | 13:52 | |
| *** yamahata_ has quit IRC | 13:54 | |
| *** yamahata_ has joined #openstack-infra | 13:54 | |
| *** yamahata_ has quit IRC | 13:57 | |
| *** yamahata_ has joined #openstack-infra | 13:57 | |
| *** yamahata_ has quit IRC | 13:58 | |
| *** katyafervent has quit IRC | 13:58 | |
| *** yamahata_ has joined #openstack-infra | 13:59 | |
| *** nprivalova has joined #openstack-infra | 13:59 | |
| fungi | mordred: yeah | 14:01 |
| *** dprince has joined #openstack-infra | 14:01 | |
| *** rongze_ has joined #openstack-infra | 14:01 | |
| *** blamar has quit IRC | 14:01 | |
| *** eharney has quit IRC | 14:03 | |
| *** chuck_ has quit IRC | 14:03 | |
| *** adalbas is now known as adalbafk | 14:03 | |
| *** rongze has quit IRC | 14:04 | |
| sdague | ttx: https://bugs.launchpad.net/tempest/+bug/1179008 - can you use your email trick to mark tempest fixed-released on that? | 14:04 |
| uvirtbot | Launchpad bug 1179008 in python-neutronclient "rename requires files to standard names" [Medium,In progress] | 14:04 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add e-r query for bug 1230354 https://review.openstack.org/61749 | 14:05 |
| uvirtbot | Launchpad bug 1230354 in tempest "tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario fails sporadically" [Undecided,Confirmed] https://launchpad.net/bugs/1230354 | 14:05 |
| ttx | sdague: I can /try/ | 14:05 |
| ttx | sdague: last time that one was so complex it even failed the Emailtrick | 14:05 |
| ttx | sdague: sent | 14:06 |
| mordred | hey | 14:07 |
| mordred | sorry I'm late | 14:07 |
| mordred | gah | 14:07 |
| mordred | wrong channel | 14:07 |
| fungi | heh | 14:07 |
| *** zhiyan has quit IRC | 14:08 | |
| *** julim has joined #openstack-infra | 14:09 | |
| *** zhiyan has joined #openstack-infra | 14:09 | |
| *** yamahata_ has quit IRC | 14:12 | |
| *** yamahata_ has joined #openstack-infra | 14:13 | |
| fungi | apevec: it does at least look like the requirements changes are getting some traction in stable/havana (i don't have much influence there other than to fix them if there's something wrong with them | 14:14 |
| apevec | fungi, yep, but tox.ini updates are not even proposed yet for stable/havana | 14:15 |
| apevec | and status on master is various | 14:15 |
| fungi | oh, i see. the ones which jeblair proposed were only to master | 14:17 |
| fungi | i can propose backports to havana for those | 14:17 |
| apevec | right, and there was some competition too so some got -1 | 14:17 |
| apevec | then some pushed their own versions with reduced scope e.g. swift/swiftclient | 14:18 |
| *** tma996 has joined #openstack-infra | 14:18 | |
| fungi | right, well that's their prerogative if they're disinterested in the patches | 14:19 |
| *** changbl has quit IRC | 14:19 | |
| fungi | it looks like the proposer of the competing packages (sushil) has not proposed any backports, so i don't think that's likely to be an issue | 14:20 |
| *** rongze has joined #openstack-infra | 14:20 | |
| fungi | er, competing patches | 14:20 |
| fungi | apevec: also, i did look over your release notes draft just after you disconnected, and it seemed to match the advisories i know we've published for havana since release | 14:21 |
| apevec | fungi, thanks | 14:21 |
| fungi | and as to your question about an advisory/cve database, i think bdpayne was working on something like that | 14:21 |
| apevec | ah cool, that would be nice to have | 14:22 |
| apevec | now it's copy paste from announcement | 14:22 |
| fungi | not sure how far he's gotten though (if it wasn't him, it was someone in the ossg anyway) | 14:22 |
| apevec | although, now that I think of it, announcement email should be possible to parse | 14:22 |
| fungi | yes, we use a template so it's reasonably normalized | 14:23 |
| *** sandy__ has joined #openstack-infra | 14:23 | |
| fungi | slight variations such as situations where there's more than one cve for a single advisory, or multiple patches to different projects | 14:23 |
| *** rongze_ has quit IRC | 14:24 | |
| *** sandywalsh_ has quit IRC | 14:24 | |
| *** dkranz-in-transi has joined #openstack-infra | 14:27 | |
| *** dkranz-in-transi is now known as dkranz | 14:28 | |
| fungi | apevec: though for the tox.ini patches, those aren't actually impacting gating right? from what i understand the issue that's solving is for people running tox locally ending up with prerelease dependencies rather than release dependency versions only | 14:28 |
| fungi | mordred: ^ | 14:28 |
| fungi | if so, the benefit of backporting those seems minimal | 14:28 |
| apevec | that's what I wasn't sure exactly | 14:28 |
| mordred | I thnk we should backport them for correctness | 14:29 |
| apevec | I didn't see confirmation that pre-releases are never pypi mirrored | 14:29 |
| mordred | they were suposted to have gone in last cycle | 14:29 |
| fungi | okay, fair enough | 14:29 |
| fungi | apevec: the risk is mainly for dependencies outside our sphere of control, some of whom do publish prerelease versions of packages to pypi | 14:30 |
| *** sdake_ has quit IRC | 14:30 | |
| fungi | those don't end up sucked into our mirror though, for the most part, so our tests don't run with them (which is why we forgot it wasn't fixed) | 14:31 |
| apevec | fungi, where exactly is the filter for prereleases? | 14:32 |
| apevec | in pypi-mirror code? | 14:32 |
| fungi | apevec: tox 1.4 (maybe 1.4.1) has some idea of what a prerelease version string looks like, and only retrieves those if requested explicitly, which tox does by passing --pre | 14:33 |
| fungi | er, pip 1.4/1.4.1 i meant | 14:33 |
| apevec | ah right | 14:33 |
| fungi | when we build the mirror, we're using pip to install dependencies, and its default behavior these days is to pull release versions only | 14:33 |
| apevec | ok, so priority is to push test-requirements backport | 14:34 |
| apevec | then tox.ini but that's not critical for 2013.2.1 release | 14:34 |
| fungi | but tox, when it added support for skipping prerelease versions, wanted to make that a non-default option so as not to change its behavior | 14:34 |
| apevec | just correctness | 14:34 |
| jog0 | sdague: heh, I see your solution was to send out an email thanks | 14:34 |
| fungi | apevec: yep | 14:34 |
| apevec | ok, I'll chase now other stable-maints to review remaining updated from global on stable/havana | 14:35 |
| fungi | nonetheless, i'm getting started on the backports... shouldn't probably take long (i expect most of them to be identical to what got proposed for master) | 14:35 |
| *** SushilKM has quit IRC | 14:35 | |
| fungi | but yeah, not urgent | 14:35 |
| apevec | adam_g, ttx - please review https://review.openstack.org/#/q/status:open+branch:stable/havana+topic:openstack/requirements,n,z | 14:36 |
| apevec | BTW how can those automatic proposals from Jenkins be abandoned? For example this one https://review.openstack.org/61539 | 14:37 |
| *** prad has joined #openstack-infra | 14:38 | |
| apevec | fungi, btw fixed deps in 61722 | 14:38 |
| fungi | apevec: i can probably abandon it, but then a new one will get proposed. we probably need a filter for projects which adamantly wish not to participate in requirements synchronization | 14:39 |
| fungi | or else a discussion at the tc level about whether this is negotiable | 14:39 |
| apevec | fungi, yes, that's what I'd suggest | 14:39 |
| apevec | to remove Swift from automatic proposals | 14:39 |
| BobBall | Should gate be setting INSTALL_TESTONLY_PACKAGES = true? I think with ceilometer disabled, libxslt-dev doesn't get installed, but it is actually required for tests (glance+trove list it, but also needed for anything that creates a venv as far as I can tell) | 14:40 |
| *** yaguang has joined #openstack-infra | 14:40 | |
| apevec | fungi, but yeah, this is definitely TC discussion material | 14:40 |
| BobBall | it defaults to false, but is listed as an absolute requirement for ceilometer but testonly for glance and trove (looking at Ubuntu) | 14:40 |
| *** prad_ has joined #openstack-infra | 14:40 | |
| *** heyongli has quit IRC | 14:40 | |
| *** prad has quit IRC | 14:40 | |
| dims | fungi, the change for increasing 100->500 results in logstash/kibana (https://github.com/openstack-infra/config/commit/58a1ef2578e588cd13fb0fa596509cba39c95305) does not seem to have been applied. is that something that has to be done by hand? | 14:40 |
| *** nprivalova has quit IRC | 14:41 | |
| *** eharney has joined #openstack-infra | 14:42 | |
| *** herndon has joined #openstack-infra | 14:42 | |
| *** rongze_ has joined #openstack-infra | 14:43 | |
| *** dcramer_ has joined #openstack-infra | 14:43 | |
| *** CaptTofu has quit IRC | 14:43 | |
| dims | another question, i wanted to try logstash-cli, does anyone know if elasticsearch.openstack.org:9200 is correct url to use? (param in CLI shows "[--esurl=ESURL] # URL to connect to elasticsearch") | 14:44 |
| *** CaptTofu has joined #openstack-infra | 14:44 | |
| *** rongze has quit IRC | 14:44 | |
| *** adrian_otto has joined #openstack-infra | 14:44 | |
| *** blamar has joined #openstack-infra | 14:45 | |
| *** alcabrera has joined #openstack-infra | 14:45 | |
| *** nprivalova has joined #openstack-infra | 14:46 | |
| fungi | BobBall: is INSTALL_TESTONLY_PACKAGES a localrc toggle? | 14:46 |
| fungi | dims: not sure whether kibana restarts or loads config updates automagically, but i can at least check whether the config update got onto the server and we can follow up with clarkb once he's awake if we want to restart it | 14:47 |
| *** prad_ has quit IRC | 14:47 | |
| dims | thanks fungi | 14:47 |
| *** oubiwan__ has joined #openstack-infra | 14:47 | |
| fungi | dims: as for the elasticsearch api endpoint, i thought i remembered discussion of a proxy for it on logstash.o.o but sdague or jog0 might know. otherwise we can check with clarkb on that too | 14:48 |
| dims | fungi, will ask them, they may be around now. thanks | 14:48 |
| *** CaptTofu has quit IRC | 14:49 | |
| sdague | dims: the endpoint should be in the er code | 14:49 |
| sdague | I don't remember it off the top of my head | 14:49 |
| *** prad has joined #openstack-infra | 14:50 | |
| *** jergerber has joined #openstack-infra | 14:51 | |
| dims | sdague, ah looks like http://logstash.openstack.org/elasticsearch - will give that a shot. thanks | 14:52 |
| mordred | apevec: please dont' remove swift from auto proposals | 14:54 |
| mordred | apevec: if they ignore them they ignore them - but the list of proposals is also the list of who participates in the mirror | 14:54 |
| apevec | ah ok | 14:54 |
| apevec | that was just my proposal based on what Swift PTL said in a review | 14:55 |
| apevec | mordred, so Swift team will have to -2 them forever? | 14:56 |
| mordred | yes | 14:59 |
| mordred | and they can deal with that however they want to deal with that | 14:59 |
| mordred | sometimes the cost of being different needs to rest on the shoulders of the people who have chosen to be different | 14:59 |
| sdague | is the backend for this actually running - http://status.openstack.org/bugday/ | 15:02 |
| sdague | I would have expected more changes in those graphs | 15:02 |
| mordred | sdague: no - I believe ttx runs that by hand | 15:03 |
| sdague | it's not on a cron? | 15:03 |
| sdague | that's less useful | 15:04 |
| ttx | sdague: bug-update-by-email fail | 15:04 |
| sdague | ttx: awesome | 15:04 |
| ttx | sdague: that bug is now officially non-updatable ! | 15:04 |
| sdague | ttx: how about mordred's above coment | 15:04 |
| sdague | ttx: yay... launchpad | 15:04 |
| *** jcooley_ has joined #openstack-infra | 15:05 | |
| ttx | sdague: last solution is to push a Launchpad question about it and ask them to go in DB to update it | 15:05 |
| ttx | https://answers.launchpad.net/launchpad/+addquestion if you feel like it | 15:05 |
| ttx | sdague: runs on cron | 15:05 |
| ttx | sdague: on old-wiki | 15:06 |
| ttx | sdague: pleia2 was looking into migrating it | 15:06 |
| * ttx looks at data | 15:06 | |
| ttx | hmm suspiciously late | 15:07 |
| ttx | errr | 15:08 |
| *** Mithrandir has quit IRC | 15:10 | |
| ttx | sdague: weird, the cron entry was disabled | 15:11 |
| sdague | can you undisable please? | 15:11 |
| ttx | sdague: done, and running script to update data | 15:12 |
| sdague | cool | 15:12 |
| ttx | No idea since when it's stuck :) | 15:12 |
| ttx | speaks volumes abot how useful it is | 15:12 |
| fungi | ttx: did you check the last modified time on the crontab before updating? | 15:12 |
| ttx | too late, but trying other means | 15:12 |
| ttx | Sep 11 13:32 | 15:13 |
| fungi | i wonder if the wtmp from the is still around | 15:14 |
| fungi | er, from then | 15:14 |
| ttx | fungi: hmm, maybe I disabled it by error when I enabled rc tracking | 15:14 |
| ttx | sep 11 sounds about when I would have enabled that | 15:15 |
| fungi | i don't personally remember logging in and disabling any cronjobs around that time, but that was also many, many beers ago | 15:15 |
| ttx | sdague: you should get new data in 20min | 15:16 |
| ttx | but no past data for the last days I'm afraid | 15:16 |
| sdague | sure, that's fine | 15:17 |
| *** sdake_ has joined #openstack-infra | 15:18 | |
| *** sdake_ has quit IRC | 15:18 | |
| *** sdake_ has joined #openstack-infra | 15:18 | |
| *** mkerrin has quit IRC | 15:19 | |
| *** rcarrillocruz1 is now known as rcarrillocruz | 15:20 | |
| SergeyLukjanov | does anyone now how to add uvirtbot to the channel? :) | 15:22 |
| fungi | SergeyLukjanov: soren manages that, so you'd have to talk him into it | 15:22 |
| SergeyLukjanov | fungi, oh, thanks | 15:22 |
| notmyname | mordred: I'm not trying to be different to be different. I'm trying to target precise. and I've asked for a simple resolver. none of our values in requirements conflict with the ones in global requirements | 15:23 |
| mriedem | uh oh | 15:24 |
| mriedem | Error <urlopen error _ssl.c:489: The handshake operation timed out> while getting https://pypi.python.org/packages/source/h/httplib2/httplib2-0.8.tar.gz#md5=94cb8a3b196dfd19253c46609489d9f1 | 15:24 |
| mriedem | http://logs.openstack.org/25/61625/2/check/check-tempest-dsvm-postgres-full/8030e95/console.html | 15:24 |
| fungi | dims: found out why kibana didn't update... seems elastic-recheck puppetry is breaking it on that server | 15:24 |
| mordred | notmyname: I know. we've talked about this, and I'm not unhappy with what you are doing | 15:25 |
| *** jgrimm has joined #openstack-infra | 15:25 | |
| mordred | notmyname: I'm simply saying that you are making a choice that is different than how openstack has decided to address this problem | 15:25 |
| mordred | notmyname: and because of that, it means you may have to -1 some patches from time to time | 15:25 |
| mordred | once we have a resolver, then we'll revisit how the automation affects you - but there's not much else that can be done right now | 15:26 |
| notmyname | mordred: also I do want to stay part of the auto updates to (1) enusure we stay compatible as we are today and (2) hopefully keep reminding you that you're not solving our use case | 15:26 |
| *** AlexF has joined #openstack-infra | 15:26 | |
| fungi | mordred: or provide patches to improve the requirements proposal script? if so we could get a feature request bug open for that at a minimum | 15:27 |
| mordred | fungi: it's to hard right now | 15:27 |
| * fungi nods | 15:27 | |
| mordred | fungi: dep solving for pep is something dstufft is looking at | 15:27 |
| mordred | it's non-trivial | 15:27 |
| mordred | there is an easier thing notmyname and I talked about | 15:27 |
| mordred | that I have yet to try | 15:28 |
| mordred | but I have not gotten to it yet | 15:28 |
| fungi | oh, cool | 15:28 |
| mordred | however- in the interest of completeness ... supporting the version of the library that happens to be in precise is NOT an openstack goal or design | 15:28 |
| mordred | it is, in fact, another way swift has chosen to be different. I'm not saying it's an invalid technical choice - but it is a different choice, an dhaving and pursuing a different technical choice there from the rest of openstack is - in fact - being different | 15:29 |
| mriedem | new infra bug: https://bugs.launchpad.net/openstack-ci/+bug/1260371 | 15:30 |
| uvirtbot | Launchpad bug 1260371 in openstack-ci "Downloading httplib2 fails due to "The handshake operation timed out"" [Undecided,New] | 15:30 |
| fungi | mriedem: yeah, i'm digging into that since you linked the log. thanks for opening | 15:30 |
| mriedem | fungi: np | 15:30 |
| mriedem | it's not package specific | 15:30 |
| dims | fungi, ah ok | 15:31 |
| fungi | mriedem: yeah, i'm more trying to figure out why a change to tempest ended up trying to satisfy a directly-declared tempest requirement from pypi.python.org (i checked, tempest is in the requirements projects.txt so it's not that we missed adding it there) | 15:32 |
| *** eharney_ has joined #openstack-infra | 15:33 | |
| fungi | mordred: you may also be interested in this one... | 15:33 |
| fungi | Running setup.py egg_info for package from file:///opt/stack/new/tempest | 15:33 |
| fungi | [...] | 15:33 |
| fungi | Could not fetch URL https://pypi.python.org/simple/pbr/: There was a problem confirming the ssl certificate: blah blah | 15:34 |
| *** sharwell_ has left #openstack-infra | 15:34 | |
| *** eharney has quit IRC | 15:34 | |
| fungi | (it then goes on to try to retrieve httplib2 from pypi.python.org and fails miserably, ending the job) | 15:35 |
| mordred | uhm. wtf | 15:36 |
| fungi | and http://logs.openstack.org/25/61625/2/check/check-tempest-dsvm-postgres-full/8030e95/logs/devstack-gate-setup-host.txt.gz suggests that the mirror selection worked as intended | 15:36 |
| fungi | have we maybe missed overriding it for the tempest user? | 15:37 |
| fungi | looks like maybe we only cover root, jenkins and stack... tempest runs under yet another account so maybe that's where this is creeping in | 15:38 |
| *** pcrews has joined #openstack-infra | 15:38 | |
| mordred | wow. yeah. that would be it | 15:39 |
| jeblair | fungi: almost certainly. | 15:39 |
| *** datsun180b has joined #openstack-infra | 15:39 | |
| fungi | i'll work up a patch | 15:39 |
| mordred | dstufft: is there a way to set a machine-wide pip.conf? | 15:39 |
| fungi | /etc/pip.conf maybe? | 15:39 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add job to check projects.yaml alphabetized https://review.openstack.org/61777 | 15:40 |
| fungi | SergeyLukjanov: my hero! | 15:40 |
| SergeyLukjanov | fungi, it's very simple atm and I not really know bash :) | 15:40 |
| fungi | SergeyLukjanov: i expected something very simple, so that's great. i can help you with any bash details if something's not quite right | 15:41 |
| *** rnirmal has joined #openstack-infra | 15:41 | |
| mordred | fungi: WELL... | 15:42 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add job to check projects.yaml alphabetized https://review.openstack.org/61777 | 15:42 |
| SergeyLukjanov | fungi, thank you, so, please, take a look on it when have some time | 15:42 |
| mordred | pip respects an env var - PIP_CONF_FILE | 15:42 |
| mordred | fungi: we could make a pip.conf in /etc and just put PIP_CONF_FILE in /etc/bashrc :) | 15:43 |
| SergeyLukjanov | fungi, and if it'll be ok and merged than we can make it voting and I'll sort projects.yaml | 15:43 |
| fungi | mordred: imminently doable | 15:43 |
| mordred | so that in case a FIFTH user shows up | 15:43 |
| mordred | it's covered | 15:43 |
| jeblair | mordred: or just export PIP_CONF_FILE in devstack-gate? | 15:43 |
| *** hashar_ has joined #openstack-infra | 15:43 | |
| *** dizquierdo has quit IRC | 15:44 | |
| mordred | jeblair: would that make it all the way down into the tempest runs? I guess those are all subshells, yeah? | 15:45 |
| mordred | sounds great | 15:45 |
| jeblair | mordred: yeah, i _think_ that's the better way to go because i don't think the sudo calls are generally invoked as login shells. but hey, i just woke up. | 15:45 |
| mordred | jeblair: I like it - sounds cleaner | 15:46 |
| fungi | wfm. i'll take that tack | 15:46 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add job to check projects.yaml alphabetized https://review.openstack.org/61777 | 15:46 |
| *** hashar has quit IRC | 15:46 | |
| *** hashar_ is now known as hashar | 15:47 | |
| *** dstanek has quit IRC | 15:47 | |
| *** adrian_otto has quit IRC | 15:48 | |
| *** dstanek has joined #openstack-infra | 15:48 | |
| jeblair | SergeyLukjanov: thank you! | 15:49 |
| SergeyLukjanov | jeblair, np, just hope that it's ok ;) I've checked it locally and it works | 15:50 |
| *** vkozhukalov has quit IRC | 15:51 | |
| fungi | mordred: jeblair: are we also going to need to do something about .pydistutils.cfg ? | 15:52 |
| mgagne | SergeyLukjanov: nice! | 15:52 |
| openstackgerrit | Roman Prykhodchenko proposed a change to openstack-infra/config: Adds devstack-gate tests for Ironic https://review.openstack.org/53917 | 15:53 |
| *** yaguang has quit IRC | 15:53 | |
| *** antonym has quit IRC | 15:54 | |
| *** katyafervent has joined #openstack-infra | 15:54 | |
| *** zhiyan has quit IRC | 15:54 | |
| *** oubiwan__ has quit IRC | 15:56 | |
| *** senk has quit IRC | 15:57 | |
| *** talluri has quit IRC | 15:57 | |
| mgagne | I remember people talking about how to manage images generated by DIB. Mainly the disk space issue. Anything about it? (wiki, irc logs, etherpad, etc.) | 15:58 |
| *** talluri has joined #openstack-infra | 15:58 | |
| *** SushilKM has joined #openstack-infra | 15:59 | |
| *** Guest53164 has joined #openstack-infra | 16:00 | |
| *** oubiwan__ has joined #openstack-infra | 16:01 | |
| dkranz | ttx: Do you have a script to say, change all bugs in a project from Fix Committed to Fix Released? | 16:02 |
| *** talluri has quit IRC | 16:02 | |
| *** adalbas has joined #openstack-infra | 16:04 | |
| *** adrian_otto has joined #openstack-infra | 16:04 | |
| *** oubiwa___ has joined #openstack-infra | 16:04 | |
| *** oubiwan__ has quit IRC | 16:06 | |
| *** yamahata_ has quit IRC | 16:07 | |
| *** ywu has joined #openstack-infra | 16:07 | |
| *** Guest53164 has quit IRC | 16:07 | |
| apevec | dkranz, he haz many nice scripts https://github.com/ttx/openstack-releasing | 16:08 |
| *** zhiyan has joined #openstack-infra | 16:08 | |
| dkranz | apevec: Thanks! | 16:09 |
| apevec | process_bugs.py should be the one | 16:09 |
| *** dhellmann is now known as dhellmann_ | 16:10 | |
| BobBall | fungi: sorry for the slow repsonse... yes, INSTALL_TESTONLY_PACKAGES is in localrc | 16:14 |
| *** herndon has quit IRC | 16:14 | |
| *** soren has quit IRC | 16:15 | |
| *** senk has joined #openstack-infra | 16:15 | |
| *** markmcclain has joined #openstack-infra | 16:16 | |
| *** soren has joined #openstack-infra | 16:16 | |
| *** ^d has joined #openstack-infra | 16:18 | |
| *** ^d has joined #openstack-infra | 16:18 | |
| *** rcleere has joined #openstack-infra | 16:19 | |
| *** senk has quit IRC | 16:20 | |
| fungi | BobBall: might be more of a #openstack-qa question, but boils down to whether we want devstack to declare all the things tempest needs to be able to run, or whether we should expect to install test requirements for the individual projects automatically in devstack. i know there has been some debate over this topic in the past... | 16:20 |
| *** eharney_ is now known as eharney | 16:20 | |
| BobBall | okies - I'll wander over there | 16:21 |
| *** NikitaKonovalov has quit IRC | 16:21 | |
| *** SergeyLukjanov has quit IRC | 16:21 | |
| ttx | apevec, dkranz: now at https://github.com/openstack-infra/release-tools | 16:21 |
| *** mrodden has joined #openstack-infra | 16:22 | |
| *** changbl has joined #openstack-infra | 16:22 | |
| apevec | ttx, ah thanks, I missed the move | 16:22 |
| *** Ryan_Lane has quit IRC | 16:22 | |
| ttx | apevec: very recent | 16:22 |
| *** oubiwa___ has quit IRC | 16:22 | |
| *** senk has joined #openstack-infra | 16:23 | |
| *** nprivalova has quit IRC | 16:24 | |
| *** SushilKM has quit IRC | 16:25 | |
| *** SushilKM has joined #openstack-infra | 16:26 | |
| *** senk has quit IRC | 16:28 | |
| *** adalbafk has quit IRC | 16:29 | |
| *** acfleury has joined #openstack-infra | 16:30 | |
| *** oubiwan__ has joined #openstack-infra | 16:31 | |
| *** acfleury has quit IRC | 16:32 | |
| *** adalbas has quit IRC | 16:32 | |
| *** oubiwan__ has quit IRC | 16:32 | |
| *** adalbas has joined #openstack-infra | 16:32 | |
| *** AaronGr_Zzz is now known as AaronGr | 16:33 | |
| *** markmcclain has quit IRC | 16:33 | |
| *** senk has joined #openstack-infra | 16:34 | |
| *** rcleere has quit IRC | 16:34 | |
| *** Abhishek_ has quit IRC | 16:38 | |
| *** herndon has joined #openstack-infra | 16:39 | |
| *** Ryan_Lane has joined #openstack-infra | 16:41 | |
| *** branen has joined #openstack-infra | 16:41 | |
| *** flaper87 is now known as flaper87|afk | 16:42 | |
| *** Guest53164 has joined #openstack-infra | 16:43 | |
| *** Abhishek_ has joined #openstack-infra | 16:45 | |
| *** mfer has joined #openstack-infra | 16:47 | |
| *** boris-42 has quit IRC | 16:48 | |
| *** avishayb has quit IRC | 16:48 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/elastic-recheck: Machine remins in HARD_REBOOT status https://review.openstack.org/61815 | 16:49 |
| *** ekarlso has quit IRC | 16:50 | |
| *** Mithrandir has joined #openstack-infra | 16:50 | |
| *** tsufiev has quit IRC | 16:51 | |
| *** ekarlso has joined #openstack-infra | 16:51 | |
| *** melwitt has joined #openstack-infra | 16:52 | |
| *** gyee has joined #openstack-infra | 16:53 | |
| sandy__ | hey y'all ... who manages stackforge? I have a question. | 16:53 |
| clarkb | fungi: hmm kibana probably doesn't restart on a config change automagically, did that get sorted out? | 16:55 |
| fungi | clarkb: no, broken puppet manifest | 16:56 |
| clarkb | fungi: oh :( | 16:56 |
| fungi | clarkb: haven't had time to dig deeper yet | 16:56 |
| fungi | err: Failed to apply catalog: Could not find dependency Exec[::elastic_recheck::install_elastic-recheck] for Service[elastic-recheck] at /opt/config/production/modules/elastic_recheck/manifests/bot.pp:85 | 16:56 |
| clarkb | sandy__: you do! :) you can ask related questions here, definitely on topic | 16:56 |
| fungi | clarkb: probably just a simple typo | 16:57 |
| *** mkerrin has joined #openstack-infra | 16:57 | |
| dstanek | is there a standard way to generate apidoc from sphinx for projects? | 16:58 |
| fungi | clarkb: actually, scoping issue? | 16:58 |
| *** emagana has joined #openstack-infra | 16:58 | |
| *** senk has quit IRC | 16:59 | |
| clarkb | dstanek: whatever novaclient does is probably pretty standard | 16:59 |
| clarkb | fungi: possibly, that require with the long dame there would've been put in place to work around other scoping issues | 16:59 |
| *** senk has joined #openstack-infra | 16:59 | |
| *** adrian_otto has quit IRC | 17:00 | |
| fungi | if it's a typo, it's not immediately obvious that's the case anyway, which is why i suspect scoping | 17:00 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Don't save mysql logs https://review.openstack.org/61820 | 17:00 |
| dstanek | clarkb: thanks, looking now | 17:00 |
| clarkb | fungi: ya, I don't see it either. bot.pp does include elastic_recheck which should bring that exec into scope then we refer to it with a rooted name | 17:01 |
| *** rongze_ has quit IRC | 17:01 | |
| dstanek | clarkb: the problem i have is that jenkins is using 'setup.py build_sphinx' and i need to find a way to generate the apidocs | 17:01 |
| clarkb | btw the reason I am on IRC so early is the neutron folks are talking third part testing in the alt meeting room now | 17:01 |
| mgagne | fungi: should be Exec['install_elastic-recheck'] | 17:02 |
| *** adrian_otto has joined #openstack-infra | 17:02 | |
| fungi | mgagne: i'll give that a shot, thanks! | 17:02 |
| mgagne | fungi: resources are not like variables. they have no scope | 17:02 |
| fungi | ahh... | 17:02 |
| sandy__ | clarkb, cool ... we have a repo that we'd like to move over, but I don't know if it's a good fit for stackforge. The motivation to move it over is some companies can only pull from "blessed" repos (not all of github) | 17:02 |
| fungi | tricky | 17:02 |
| mgagne | fungi: that's why resource titles have to be unique across your puppetmaster | 17:03 |
| sandy__ | clarkb, it's our ceilometer storage driver tests https://github.com/rackerlabs/ceilometer-load-tests ... it's framework, but doesn't need gerrit or jenkins, etc. It just needs to live somewhere that keeps legal happy. | 17:03 |
| sandy__ | *it's a framework | 17:03 |
| fungi | dstanek: so, maybe the missing detail here is having standardized docstrings in public functions and methods which sphinx can compile into the otherwise empty files in the docs/source subdir of the repo | 17:03 |
| clarkb | sandy__: I personally don't think stackforge would appropriate in that case. The reason to use stackforge is to get gerrit and jenkins. Not as a "blessed" repo location | 17:04 |
| *** senk has quit IRC | 17:04 | |
| clarkb | sandy__: if however you want all of the above I could see it fitting in | 17:04 |
| sandy__ | clarkb, yeah, that was my thoughts as well | 17:04 |
| sandy__ | *thought | 17:04 |
| sandy__ | clarkb, let me chat with them and see how big a deal it is | 17:04 |
| sandy__ | clarkb, thanks | 17:05 |
| dstanek | fungi: the normal way to do this would be to run 'sphinx-apidoc' before building the sphinx docs | 17:05 |
| jeblair | dstanek: pbr has a hook for that | 17:05 |
| dstanek | fungi: as a work around i created an extension to do that, but i'm not sure that it's the correct way | 17:05 |
| *** tsufiev has joined #openstack-infra | 17:06 | |
| openstackgerrit | Roman Prykhodchenko proposed a change to openstack-infra/config: Adds devstack-gate tests for Ironic https://review.openstack.org/53917 | 17:06 |
| dstanek | jeblair: when i add a [pbr] section to my setup.cfg i get a failure | 17:06 |
| fungi | ah, right. jeblair has a good point. if you're packaged with pbr it should be happening with its sphinx integration. otherwise you'd have to duplicate that behavior elsewhere | 17:06 |
| dstanek | jeblair: i wanted to just do this: https://review.openstack.org/#/c/52929/6/setup.cfg | 17:06 |
| fungi | dstanek: i think you also need to add it as a setup_requires in setup.py? | 17:07 |
| dstanek | jeblair: but when i do i get http://logs.openstack.org/29/52929/6/check/gate-keystone-docs/54d716b/ | 17:07 |
| *** klrmn has left #openstack-infra | 17:07 | |
| *** gema has quit IRC | 17:07 | |
| fungi | dstanek: like http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/setup.py#n19 | 17:07 |
| *** senk has joined #openstack-infra | 17:08 | |
| dstanek | fungi: already there http://git.openstack.org/cgit/openstack/keystone/tree/setup.py :-) | 17:08 |
| *** roaet has joined #openstack-infra | 17:08 | |
| fungi | mm, yeah okay if this is keystone then i expect it would be | 17:08 |
| roaet | This is the channel for devstack questions, righ? | 17:08 |
| dstanek | fungi: why is that? | 17:08 |
| fungi | roaet: pure devstack discussions usually happen on #openstack-qa | 17:08 |
| roaet | fungi: got it, thanks | 17:09 |
| *** roaet has left #openstack-infra | 17:09 | |
| fungi | dstanek: because most of the integrated server projects are pbr-ized at this point | 17:09 |
| fungi | keystone being one of them | 17:09 |
| dstanek | fungi: ah i see | 17:09 |
| *** jerdfelt has quit IRC | 17:09 | |
| dstanek | i thought you meant you'd expect that error | 17:09 |
| *** afazekas has quit IRC | 17:10 | |
| fungi | nah, perusing the error log now | 17:10 |
| *** jcoufal_ has quit IRC | 17:10 | |
| *** tma996 has quit IRC | 17:10 | |
| dstanek | it's almost it's almost like setuptools gets messed up somehow | 17:10 |
| fungi | my eye is drawn to "Unknown distribution option: 'setup_requires'" | 17:11 |
| dstanek | fungi: i believe this happens if i just add [pbr] to the setup.cfg without any options | 17:12 |
| *** senk has quit IRC | 17:12 | |
| *** amotoki has joined #openstack-infra | 17:13 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add e-r query for bug 1259907 https://review.openstack.org/61826 | 17:14 |
| uvirtbot | Launchpad bug 1259907 in openstack-ci "check-grenade-dsvm marked as FAILED - n-api/g-api Logs have errors" [Undecided,New] https://launchpad.net/bugs/1259907 | 17:14 |
| mordred | I have seen weird interactions with sphinx plugin things and pbr and I have not yet sorted out why | 17:14 |
| *** SergeyLukjanov has joined #openstack-infra | 17:14 | |
| fungi | dstanek: so looking at nova as a comparison, it has no [pbr] setion in setup.cfg bit does have a [build_sphinx] section like http://git.openstack.org/cgit/openstack/nova/tree/setup.cfg#n154 | 17:14 |
| dstanek | fungi: keystone has the same section | 17:15 |
| *** danger_fo_away is now known as dangers | 17:15 | |
| clarkb | mordred: doesn't pbr just do autodoc though? or does a switch need to be flipped | 17:16 |
| *** yolanda has quit IRC | 17:16 | |
| mordred | clarkb yes. that's what he's doing | 17:16 |
| *** yolanda has joined #openstack-infra | 17:16 | |
| clarkb | mordred: I mean no config change | 17:16 |
| mordred | fungi: build_sphinx is different | 17:16 |
| mordred | build_spohinx is options for sphinx | 17:16 |
| *** melwitt has quit IRC | 17:17 | |
| dstanek | clarkb: sphinx will autodoc automatically | 17:17 |
| mordred | pbr section is options to pbr - and in that section, dstanek is telling pbr to do magic sphinx things | 17:17 |
| *** melwitt has joined #openstack-infra | 17:17 | |
| dstanek | i wanted something more like sphinx-apidoc | 17:17 |
| *** NikitaKonovalov has joined #openstack-infra | 17:18 | |
| mordred | dstanek: [pbr] | 17:18 |
| mordred | autodoc_index_modules = True | 17:18 |
| clarkb | gotcha setting http://git.openstack.org/cgit/openstack-dev/pbr/tree/pbr/packaging.py#n730 | 17:18 |
| mordred | is in ironic's setup.cfg and they aren't breaking - I wonder what's different in keystone | 17:18 |
| * mordred is looking ocally | 17:19 | |
| mordred | O M Gmy internet sucks | 17:19 |
| zaro | good morning | 17:21 |
| dstanek | mordred: the interesting thing is that even when i remove the [pbr] section something in my environment is still messed up | 17:21 |
| clarkb | zaro: morning | 17:21 |
| mordred | dstanek: fantastic | 17:22 |
| dstanek | testing ironic locally now to see if it works | 17:22 |
| fungi | does pbr treat treat "true" the same as "True" in setup.cfg? | 17:23 |
| *** markmc has quit IRC | 17:24 | |
| clarkb | I think so, the ini parser sorts it out | 17:24 |
| fungi | but yeah, in that log setup.py seems to think keystone has no build_sphinx section in setup.cfg | 17:25 |
| *** zhiyan has quit IRC | 17:26 | |
| *** gema has joined #openstack-infra | 17:26 | |
| *** jaypipes has quit IRC | 17:28 | |
| *** AlexF has quit IRC | 17:29 | |
| dstanek | fungi: i read the error as the kwargs passed to setup() are incorrect | 17:30 |
| fungi | i wonder if it's failing to load setup.cfg at all when it's being invoked at that point, given that it doesn't seem to grok setup_requires there either | 17:31 |
| clarkb | fungi: that could be it, I think the setup_requires thing is distutils instead of setuptools is being used | 17:31 |
| dstanek | fungi: does a subprocess run or something? lots of stuff happens before that error | 17:31 |
| *** oshritf has joined #openstack-infra | 17:32 | |
| fungi | it looks like autodoc is trying to index setup.py there | 17:32 |
| fungi | do we normally blacklist that for sphinx? | 17:32 |
| *** blamar has quit IRC | 17:33 | |
| fungi | dstanek: i'm thinking it may be due to the direct invocation of sphinx-build in https://review.openstack.org/#/c/52929/6/tox.ini | 17:33 |
| oshritf | Will appriciate help with the following error message: "fatal: ICLA contributor agreement requires current contact information" while trying to upload a new change with git review. | 17:33 |
| *** hashar has quit IRC | 17:34 | |
| *** nati_ueno has quit IRC | 17:34 | |
| dstanek | fungi: that's something i added for tox; it's not being run here | 17:34 |
| dstanek | fungi: that would run if you 'tox -e docs' | 17:34 |
| fungi | oshritf: https://wiki.openstack.org/wiki/CLA-FAQ | 17:35 |
| *** praneshp has joined #openstack-infra | 17:35 | |
| fungi | dstanek: oh, right. it's not being called | 17:35 |
| fungi | the docs job really is just calling python setup.py build_sphinx | 17:36 |
| fungi | which it seems to find fine at http://logs.openstack.org/29/52929/6/check/gate-keystone-docs/54d716b/console.html.gz#_2013-12-08_21_07_22_024 | 17:36 |
| *** rcleere has joined #openstack-infra | 17:37 | |
| *** oshritf has quit IRC | 17:38 | |
| *** marcol has joined #openstack-infra | 17:38 | |
| *** adrian_otto has left #openstack-infra | 17:39 | |
| *** praneshp has quit IRC | 17:39 | |
| *** NikitaKonovalov has quit IRC | 17:41 | |
| *** oshritf has joined #openstack-infra | 17:41 | |
| *** marcol has left #openstack-infra | 17:41 | |
| *** klrmn has joined #openstack-infra | 17:41 | |
| *** amotoki has quit IRC | 17:42 | |
| *** NikitaKonovalov has joined #openstack-infra | 17:43 | |
| *** gothicmindfood has joined #openstack-infra | 17:43 | |
| *** marcol has joined #openstack-infra | 17:44 | |
| *** gothicmindfood has left #openstack-infra | 17:44 | |
| clarkb | fungi: dstanek what is the difference between autodoc_tree_index_modules and autodoc_index_modules? and does one work while the other fails? | 17:45 |
| fungi | ohhhh | 17:45 |
| devananda | jog0: is it possible to add pep257 to a project's pep/flake check? | 17:45 |
| *** emagana has quit IRC | 17:46 | |
| clarkb | looks like the one with tree in it does apidoc? | 17:46 |
| fungi | clarkb: though ironic's using autodoc_index_modules same as dstanek's https://review.openstack.org/#/c/53486/3/setup.cfg | 17:46 |
| *** dkranz is now known as dkranz_at_lunch | 17:48 | |
| mordred | fungi: THANK YOU. that is what's happening | 17:48 |
| *** jooools has quit IRC | 17:48 | |
| mordred | sphinx api doc is finding and attempting to api index setup.py, and it's doing it in a weird context | 17:49 |
| *** crank has quit IRC | 17:49 | |
| *** oubiwan__ has joined #openstack-infra | 17:49 | |
| *** yolanda has quit IRC | 17:49 | |
| * mordred now remembers tracking down that root cause before but not really knowing how to fix it or work around it | 17:49 | |
| fungi | mordred: yeah, i was trying to find where/how we blacklist it from doing that on other projects but was coming up short | 17:49 |
| dstanek | mordred: is that due to the sys.path.inserts? | 17:49 |
| *** crank has joined #openstack-infra | 17:50 | |
| fungi | was digging into doc/source/conf.py examples in hopes it was buried in there somewhere | 17:50 |
| fungi | sys.path.insert(0, os.path.abspath('../..')) | 17:50 |
| fungi | that? | 17:51 |
| fungi | it does seem that ironic isn't doing that, while keystone is | 17:51 |
| *** reed has joined #openstack-infra | 17:51 | |
| fungi | so could definitely be a possible cause | 17:51 |
| *** rossella_s has quit IRC | 17:52 | |
| mordred | fungi: might be. | 17:53 |
| *** jpich has quit IRC | 17:54 | |
| *** eharney has quit IRC | 17:54 | |
| *** sharwell has joined #openstack-infra | 17:54 | |
| *** crank has quit IRC | 17:56 | |
| *** yassine has quit IRC | 17:58 | |
| dstanek | mordred: fungi: removing the paths make the command succeed, but doesn't generate the api docs; so that's definitely progress | 17:58 |
| dstanek | that may be my fault... | 17:58 |
| *** derekh has quit IRC | 17:58 | |
| *** marcol has quit IRC | 17:59 | |
| *** crank has joined #openstack-infra | 17:59 | |
| clarkb | dstanek: nothing in doc/source/api? | 17:59 |
| mordred | apidoc.main(['apidoc', '.', '-H', 'Modules', '-o', source_dir]) | 18:00 |
| mordred | we should probably fix that and pass it better options that tell it explicitly what the module is | 18:00 |
| mordred | since we know where it is | 18:00 |
| *** nati_ueno has joined #openstack-infra | 18:00 | |
| dstanek | clarkb: it was my fault; i had debugging code in pbr | 18:00 |
| mordred | since it's configured in setup.cfg | 18:00 |
| dstanek | it looks much better now! thanks | 18:00 |
| mordred | w00t | 18:00 |
| notmyname | jog0: clarkb: still around? I wanted to go over the gate job graph I made | 18:01 |
| clarkb | I am, though I need to step away in a bit to walk to the office | 18:02 |
| clarkb | but now is fine | 18:02 |
| notmyname | clarkb: ok. I wanted to validate the checks that I was doing (ie the graphite functions I'm using) | 18:03 |
| *** NikitaKonovalov has quit IRC | 18:03 | |
| *** Ryan_Lane has quit IRC | 18:03 | |
| notmyname | I think the big question was movingAverage vs summarize | 18:03 |
| notmyname | clarkb: but I walso wanted to clarify why I was using the stairstep (rather than slope) graph | 18:04 |
| notmyname | since the data is bucketed into time brackets, the stairstep I think is a more accurate reflection of the state of things. iterpolation doesn't make as much sense, IMO | 18:04 |
| notmyname | (link http://not.mn/gate_status.html ) | 18:05 |
| clarkb | notmyname: right, these are discrete events and taking aggregates over time makes sense to me too | 18:05 |
| notmyname | ok | 18:05 |
| *** NikitaKonovalov has joined #openstack-infra | 18:05 | |
| notmyname | also, the red line for the total isn't the pass rate. it's actually the chance that a new patch will pass the gates. and in that sense, I think the calc is ok because the gate jobs are independent | 18:05 |
| oshritf | will try again :) anyone have any suggestion to how to approach a "fatal: ICLA contributor agreement requires current contact information." error while git review? I tried to fill in a contact information on the setting page, this didnt work at first as I didnt have a profile, but now that I've created one, looks like the contact information is set ok (however, I'm still getting the same message while trying to upload a chang | 18:05 |
| clarkb | notmyname: one thing I thought of is it might be nice to do what the node graph at the bottom of the zuul status page does and stack the values | 18:06 |
| notmyname | clarkb: I think I need to relable the axis | 18:06 |
| notmyname | clarkb: stacked values don't make sense because they aren't added together to mean anything | 18:06 |
| fungi | oshritf: is it possible you have more than one account? what ssh username are you using? | 18:06 |
| notmyname | clarkb: ie 2 jobs with a 90% pass rate don't stack to a 180% pass rate | 18:06 |
| clarkb | notmyname: I was thinking of failure rates | 18:06 |
| *** eharney has joined #openstack-infra | 18:07 | |
| notmyname | clarkb: ya, that would work, but I'd have to change it from percentages to the raw number | 18:07 |
| clarkb | two jobs with a 10% failures rate add up to a 20% chance that a change doesn't merge | 18:07 |
| clarkb | because of statistical independence (which may not be true in all cases) | 18:07 |
| notmyname | clarkb: no, they have a 19% chance of failing (.9 * .9) | 18:08 |
| fungi | oshritf: go to https://review.openstack.org/#/settings/ and confirm that it shows your ssh user in the username field, then go to https://review.openstack.org/#/settings/contact and make sure it says "Contact information last updated on (some date/time, probably today)" | 18:08 |
| *** oshritf has quit IRC | 18:10 | |
| clarkb | notmyname: er right, so stacking with addition doesn't make sense | 18:10 |
| fungi | oshritf can't seem to be bothered to stay in channel to get answers | 18:11 |
| clarkb | problem I am trying to solve there is reading a bunch of overlapping lines to come up with an aggregate is hard, but that is what your Overall rate is providing so I should stop thinkg about it :) | 18:11 |
| notmyname | :-) | 18:12 |
| *** vkozhukalov has joined #openstack-infra | 18:12 | |
| notmyname | clarkb: ok, I've got a movingAverage version. it's basically the same shape, but it's slightly noisier. IMO the summarize makes more sense, but I'm happy to share the other if you're interested | 18:12 |
| clarkb | notmyname: sure why not | 18:13 |
| *** senk has joined #openstack-infra | 18:13 | |
| *** johnthetubaguy has quit IRC | 18:13 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Import the openstackid authentication project https://review.openstack.org/61250 | 18:15 |
| notmyname | clarkb: moving average version at http://not.mn/gate_status2.html | 18:15 |
| notmyname | clarkb: actually I think summarize is better because otherwise we're averaging over the moving 5 hour buckets. which basically creates data where there is none | 18:16 |
| notmyname | which is the noise you see on the moving average graph | 18:16 |
| *** arosen has joined #openstack-infra | 18:17 | |
| clarkb | I think the reason for the longer window is these tests take close to an hour to run so you don't get much data in an hour window | 18:17 |
| arosen | Hi, I've been seeing these gate failures on gate-nova-docs and I was wondering if someone could point or the error or why it's failing? For example, https://review.openstack.org/#/c/61437/ | 18:17 |
| *** senk has quit IRC | 18:17 | |
| notmyname | ya, I'm not arguing the bucket size. just that when trying to have a moving average over the buckets doesn't really do much but add noise (because of the reason to use the stairstep graph mentioned earlier) | 18:18 |
| clarkb | arosen: sphinx released a new version (1.2) that broke build_sphinx because of something in distutils tl;dr we capped sphinx to <1.2 in requirements | 18:18 |
| *** Ryan_Lane has joined #openstack-infra | 18:18 | |
| clarkb | notmyname: ya | 18:18 |
| apevec | arosen, bug 1259511 | 18:18 |
| fungi | arosen: grizzly looks like it needs sphinx pinned to <1.2 | 18:18 |
| uvirtbot | Launchpad bug 1259511 in openstack-ci "sphinx docs gating jobs are broken (source_dir must be a directory)" [Critical,Fix released] https://launchpad.net/bugs/1259511 | 18:18 |
| *** andreaf has quit IRC | 18:19 | |
| arosen | cool, thanks guys! | 18:19 |
| apevec | arosen, for havana it's fixed by update from global https://review.openstack.org/#/q/status:open+branch:stable/havana+topic:openstack/requirements,n,z | 18:19 |
| fungi | for grizzly we're probably going to need the iso8601 problem solved before we can solve the sphinx issue there | 18:20 |
| apevec | well, not fixed yet, still going through gates :( | 18:20 |
| arosen | apevec: cool, hopefully those will all merge soon | 18:20 |
| apevec | arosen, yes, need that before cve fixes can merge and tag 2013.2.1 release | 18:20 |
| apevec | fungi, oh yes, that too | 18:21 |
| fungi | apevec: no luck getting glance devs to look at https://review.openstack.org/55939 i take it? | 18:21 |
| apevec | got two glance core and stable-maints to +2 but PTL was not willing | 18:21 |
| * fungi sighs | 18:22 | |
| reed | mordred works mostly on Sunday afternoon http://stackalytics.com/report/users/mordred ? | 18:22 |
| apevec | https://review.openstack.org/55939 | 18:22 |
| fungi | can we stop supporting glance in grizzly if they're unwilling to have it fixed? though ripping it out of integration testing would not be easy either | 18:22 |
| apevec | technically I could push it, have enough +2s | 18:22 |
| fungi | stackalytics tells me i should get more sleep | 18:24 |
| notmyname | clarkb: that's all I got. thanks for the feedback. my "final" version is at http://not.mn/gate_status.html | 18:24 |
| clarkb | notmyname: looks good thanks | 18:25 |
| clarkb | fungi: I love that it catches my pre football sunday morning activity | 18:25 |
| jeblair | reed: must be an error or artifact -- i definetely don't work on sunday afternoons: http://stackalytics.com/report/users/corvus | 18:25 |
| jeblair | yeah, we seem to all have that bubble | 18:26 |
| fungi | notmyname: are those date labels under utc noon or midnight? trying to figure out whether the dips to near-zero success rates are at peak activity or trough | 18:26 |
| *** crank has quit IRC | 18:26 | |
| reed | jeblair, I don't :) http://stackalytics.com/report/users/smaffulli | 18:26 |
| reed | I suspect the time is in UTC | 18:27 |
| reed | no, makes no sense either | 18:27 |
| notmyname | fungi: I have no idea. whatever the graphite server is serving up, I'd guess (rather than my server's time, which is UTC) | 18:27 |
| fungi | looks like stackalytics doesn't (yet) mine the irc logs on eavesdrop | 18:27 |
| *** praneshp has joined #openstack-infra | 18:29 | |
| clarkb | graphite server will try to be smart based on what the browser tells it | 18:29 |
| fungi | my browser should tell everyone i want utc | 18:30 |
| *** dkranz_at_lunch is now known as dkranz | 18:30 | |
| fungi | at least the workstation i'm running it on uses utc | 18:30 |
| fungi | but if that's the case, that status graph is lagging behind a few hours. maybe that's because of summarize | 18:31 |
| fungi | in which case the labels are underneath utc midnight, so those troughs represent relatively (but not substantially) lower activity times | 18:32 |
| mordred | reed: I found that interesting | 18:32 |
| reed | I don't think in UTC time, I have a hard time rephasing my brain twice a year with daylight saving already | 18:34 |
| reed | i want local time always | 18:34 |
| clarkb | reed: DST is why I am slowly forcing myself to use UTC more | 18:34 |
| jeblair | mordred: it's wrong | 18:34 |
| reed | what time is 1500 UTC? | 18:35 |
| mordred | apevec: what are we waiting on for that glance fix? | 18:35 |
| *** Ryan_Lane has quit IRC | 18:35 | |
| apevec | mordred, Glance PTL to agree | 18:35 |
| apevec | but in theory we have in enough +2 | 18:35 |
| mordred | apevec: why? at what point did we start needing a PTL to agree to a backport patch? | 18:35 |
| apevec | I was just trying to get PTL buyin | 18:36 |
| clarkb | mordred: http://logs.openstack.org/10/61110/4/gate/gate-requirements-integration-dsvm/a81535e/logs/screen-key.txt.gz is a weird one | 18:36 |
| *** sarob has joined #openstack-infra | 18:36 | |
| clarkb | reed: 3 and a half hours ago | 18:36 |
| apevec | mordred, I was just being super nice | 18:36 |
| mordred | apevec: :) | 18:36 |
| * apevec goes to aprove | 18:36 | |
| reed | clarkb, pacific time? | 18:36 |
| reed | yeah | 18:37 |
| reed | ok, thanks clarkb | 18:37 |
| *** krotscheck has joined #openstack-infra | 18:37 | |
| mordred | clarkb: that seems like something not installed, which seems like a weird thing to get through | 18:37 |
| *** rcarrillocruz1 has joined #openstack-infra | 18:37 | |
| *** crank has joined #openstack-infra | 18:37 | |
| *** rcarrillocruz has quit IRC | 18:37 | |
| *** melwitt has quit IRC | 18:37 | |
| clarkb | mordred: ya | 18:38 |
| clarkb | reed: anytime :) | 18:38 |
| *** mdenny has joined #openstack-infra | 18:38 | |
| sarob | mordred: could i borrow your lobes for a few minutes on https://review.openstack.org/#/c/61342/? | 18:39 |
| mordred | sarob: wow. my lobes. | 18:39 |
| mordred | looking | 18:39 |
| clarkb | sarob: mordred: that is the symptom of zuul having a bad locally cached repo | 18:39 |
| sarob | mordred: me wee little brain is stuck | 18:39 |
| clarkb | it will need to be replaced on the zuul host | 18:39 |
| apevec | mordred, approved but now I remember that it will fail due to sphinx | 18:40 |
| clarkb | manage-projects is still not happy | 18:40 |
| mordred | clarkb: sigh | 18:40 |
| clarkb | apevec: you may need to bundle them | 18:40 |
| apevec | should I just include sphinx cap in that patch | 18:40 |
| apevec | clarkb, yeah | 18:40 |
| mordred | clarkb: what is the fix - remove the cached repo on zuul.o.o? or is there a better way? | 18:40 |
| sarob | mordred clarkb so im not as stupid as i thought? omb | 18:40 |
| sarob | omg rather | 18:41 |
| * sarob sigh | 18:41 | |
| clarkb | mordred: I have been moving the repo aside into /root/broke-repos or some such (there is a dir there), then in roots history is a git clone command withteh right GIT_SSH variable that you should use to reclone the repo | 18:41 |
| clarkb | mordred: however, I think zuul may properly reclone for you now | 18:41 |
| sarob | mordred clarkb so it sounds like you guys are working on it | 18:42 |
| clarkb | I can't do it right this second, I have found myself still at home and need to AFK for a bit, but can fix it when I am back to a keyboard | 18:42 |
| mordred | I'm on it | 18:42 |
| *** melwitt has joined #openstack-infra | 18:43 | |
| clarkb | perfect thanks | 18:43 |
| apevec | jog0, how long does it take to pass the gate? This one https://review.openstack.org/61449 has been in gate for 10h | 18:43 |
| morganfainberg | clarkb, mordred, I'm trying to figure out what I can do to help w/ stable/grizzly merges, getting the lovely iso8601 conflict issue on tempest-dsvm-neutron. | 18:43 |
| clarkb | apevec: sdague sent mail to the list saying ~12 hours right now | 18:43 |
| clarkb | morganfainberg: apevec is fixing that as we speak :) | 18:43 |
| morganfainberg | clarkb, mordred, if there is anything I can do to help resolve that (not sure what needs to be done) I'm willing to chip in. | 18:43 |
| morganfainberg | clarkb, oh great! | 18:44 |
| fungi | morganfainberg: https://review.openstack.org/55939 is our latest hope | 18:44 |
| morganfainberg | clarkb, like i said, just coming to offer help :) | 18:44 |
| openstackgerrit | Ben Nemec proposed a change to openstack-dev/hacking: Enforce import group ordering https://review.openstack.org/54403 | 18:44 |
| morganfainberg | fungi, nice. hope it works! | 18:44 |
| mordred | sarob: ok. should be fixed now | 18:44 |
| fungi | morganfainberg: well, it seems to solve the problem, it's more of a social question of convincing teh glance maintainers | 18:44 |
| clarkb | morganfainberg: thank you for the offer | 18:44 |
| *** nati_uen_ has joined #openstack-infra | 18:46 | |
| sarob | mordred: my hero, thx dude | 18:46 |
| annegentle | hey clarkb fungi jeblair mordred any thoughts on hosting docs for people on stackforge that aren't doing sphinx builds? Can you publish to rtd.org without sphinx builds? re: https://review.openstack.org/#/c/61296/1/doc/build_doc.sh | 18:47 |
| mordred | annegentle: heh. well, no - rtd is a sphinx documentation hosting site | 18:47 |
| annegentle | mordred: that's what I thought | 18:48 |
| fungi | clarkb: the other day when this cropped up for stevedore, doing a git reset --hard in the repo didn't help but just deleting the repo was enough for zuul to reclone it on the next job | 18:48 |
| mordred | annegentle: I'd imagine that publishing the output of that build somewhere should be pretty easy though | 18:48 |
| annegentle | mordred: we may need to think about a docs.openstack.org/stackforge area? | 18:48 |
| *** hemanth_ has joined #openstack-infra | 18:48 | |
| mordred | annegentle: perhaps | 18:48 |
| annegentle | mordred: other than I don't want to take responsibility for it at all | 18:48 |
| annegentle | mordred: there's that | 18:48 |
| clarkb | fungi: ya that is a recently added feature/bugfix for zuul wasn't sure if it was running yet | 18:48 |
| annegentle | mordred: I once had docs.openstack.org/incubation | 18:48 |
| mordred | annegentle: yeah - that's the, you know, thing - in theory that might be nice - I have no idea why would work on it or care for it :) | 18:48 |
| mordred | who | 18:49 |
| mordred | not why | 18:49 |
| annegentle | mordred: because it says docs | 18:49 |
| annegentle | mordred: people think it's me | 18:49 |
| *** nati_ueno has quit IRC | 18:49 | |
| annegentle | mordred: there's also the use of openstack "property" -- concerned or not? | 18:49 |
| mordred | yah! that's the part I get and why I think we dont want to just publish directly to docs.o.o | 18:49 |
| mordred | I'm not concerned about that - I just dont' want people to think those are the openstack docs team's docs | 18:49 |
| clarkb | there is nothing preventing them from using sphinx >_> | 18:49 |
| *** yolanda has joined #openstack-infra | 18:50 | |
| hemanth_ | sdague: hey | 18:50 |
| annegentle | mordred: ok, cool, yeah. I just hate to keep turning away projects with interested doc folks, seems like a pipeline of contributors I'd miss out on | 18:50 |
| mordred | clarkb: nice | 18:50 |
| mordred | annegentle: yeah - that's my concern too - it's sort of like the tempest sequencing problem we were talking about the other day | 18:50 |
| annegentle | mordred: yep zactly | 18:50 |
| mordred | we want people to add tempest tests for their things, but we don't want them to do it before they're incubated - but that means that they start writing things elsewhere | 18:50 |
| mordred | annegentle: we _do_ own stackforge.org | 18:51 |
| fungi | the "projects need to show they're ready for official resources without draining those same resources" paradox | 18:51 |
| mordred | but we stopped using it when we stoped running a parallel infrstructure | 18:51 |
| mordred | fungi: ++ | 18:51 |
| annegentle | mordred: fungi yeah agree | 18:52 |
| *** gyee has quit IRC | 18:52 | |
| annegentle | mordred: is running a parallel infra / docs site even passably resource-able? (If that makes sense) | 18:52 |
| mordred | in theory we could published to the same server easily enough - and ftp publication of doc artifacts is one of the easier things we do - and theorectically could do a docs.stackforge.org | 18:52 |
| annegentle | mordred: I can't keep up with primary hardly | 18:52 |
| mordred | but I'd like for jeblair to weigh in on that | 18:52 |
| annegentle | mordred: we might need to think about that... ya | 18:52 |
| mordred | parallel infra is a terrible idea | 18:53 |
| mordred | but if it's a directory on the same server with a vhost | 18:53 |
| mordred | we have everything parameterized already | 18:53 |
| annegentle | mordred: If one of my missions is to make pluggable docs, so pre-publishing during incubation to docs.stackforge.org might help | 18:53 |
| *** che-arne has quit IRC | 18:54 | |
| *** nati_uen_ has quit IRC | 18:54 | |
| fungi | today's count... we now host 102 stackforge projects and 114 official projects in our gerrit (stackforge is 47% of our total now) | 18:55 |
| *** nati_ueno has joined #openstack-infra | 18:55 | |
| annegentle | mordred: at one point I had a branded output that said "built for openstack" | 18:55 |
| annegentle | mordred: so we could brand with stackforge possibly | 18:55 |
| jeblair | mordred, annegentle: if we make integrating with devstack a requirement for new projects, it hardly seems fair to exclude docs from that kind of pre-incubation opportunity. also, they distribute tarballs, etc. | 18:55 |
| annegentle | mordred: but yeah it's all work | 18:55 |
| annegentle | jeblair: yep | 18:55 |
| jeblair | i like the vhost idea. | 18:56 |
| mordred | k. I was thinking that seemed like low overhead for us - but wanted to make sure that didn't raise concerns on your side | 18:56 |
| fungi | agreed. extending the vhost to cover tarballs.stackforge.org might not be a bad idea either since there had been concern expressed about that in the past as well | 18:56 |
| jeblair | mordred: it raises concerns for sure. | 18:56 |
| hemanth_ | iccha: https://github.rackspace.com/O3Eng/glance/pull/27 | 18:57 |
| jeblair | mordred: but i'm not sure i can do anything about those concerns, other than say if stackforge folks want this to happen quickly, pitch in to infra. :) | 18:57 |
| mordred | jeblair: I wonder if it's possible to transfer a domain into a rax dns account | 18:57 |
| mordred | jeblair: right now stackforge.org is on zerigo | 18:57 |
| *** nicedice has joined #openstack-infra | 18:57 | |
| jeblair | mordred: i'm pretty sure it is | 18:58 |
| fungi | mordred: domain registration is generally independent from authoritative dns. the registrar just needs to inject the right glue into the tld | 18:58 |
| apevec | shardy, is that hacking update which broke heat-pep8 ? https://review.openstack.org/61276 | 18:59 |
| jeblair | fungi: stackforge.org might be a good place to experiment with an alternative ADNS system. | 18:59 |
| fungi | jeblair: also a very, very good idea | 18:59 |
| *** blamar has joined #openstack-infra | 18:59 | |
| fungi | guinea pig and then move openstack.org if it works out | 18:59 |
| fungi | record count for stackforge.org seems like it would be tiny. no more than half a dozen probably | 19:00 |
| fungi | and that's counting ns, soa, et cetera | 19:00 |
| fungi | unless we're extending it beyond docs and tarballs anyway | 19:01 |
| jeblair | 1st step: see if hp or rax dns supports zone transfers | 19:01 |
| clarkb | jeblair: I don't think designate does yet, but accepting bind zone files is supposed to come in the next api or some such | 19:02 |
| mordred | kiall: ^^ ? | 19:02 |
| *** apevec has quit IRC | 19:02 | |
| *** jgrimm has quit IRC | 19:03 | |
| annegentle | jeblair: REally good point | 19:04 |
| *** Ryan_Lane has joined #openstack-infra | 19:06 | |
| fungi | jeblair: by "supports zone transfers" you mean acts as a slave nameserver? | 19:06 |
| fungi | jeblair: or you mean will they allow us to perform a zone transfer so we can dump copies of the zones we want to move? | 19:07 |
| fungi | (just tested the latter and rs seems to have zone axfr disabled on their masters) | 19:07 |
| jeblair | fungi: my ideal api is "upload this bind zone file" in some way. something that supports that in some manner. | 19:07 |
| jeblair | fungi: and then the zone file is kept in git. | 19:08 |
| clarkb | jeblair: at openstack on ales I was told by someone at rackspace that that feature is coming beacuse it is one of their needs too | 19:08 |
| fungi | jeblair: got it. so either we do a silent master they can slave from, or we make api calls into their dns to push updates | 19:08 |
| *** Abhishek_ has quit IRC | 19:08 | |
| jeblair | fungi: anything other than that requires work; either writing a something-to-series-of-api-calls translator, or running our own. | 19:08 |
| * fungi nods | 19:09 | |
| jeblair | clarkb: would be nice! | 19:09 |
| fungi | yeah, so direct zonefile uploads/imports on the provider. definitely would be a nice feature | 19:09 |
| * mordred knows he is in the minority, but does not want to move in that direction, so just wants to voice that again. obviously - will of the group, whatnot | 19:10 | |
| jeblair | mordred: huh what? | 19:10 |
| mordred | running our own | 19:11 |
| fungi | though bind zonefile parsing is nontrivial (unless you use libbind directly maybe? does it make the parser available?) | 19:11 |
| jeblair | mordred: oh, yes, obviously; that's why i listed it last. | 19:11 |
| *** Ryan_Lane has quit IRC | 19:11 | |
| clarkb | fungi: twisted names has a parser for it :) | 19:11 |
| *** freyes has joined #openstack-infra | 19:11 | |
| clarkb | I think it probably avoids lots of twistedness too | 19:11 |
| jeblair | mordred: i find it preferable to 'do nothing'. | 19:11 |
| *** krotscheck has quit IRC | 19:12 | |
| dstanek | mordred: it would seem that pbr's autodoc_tree_index_modules is broken for the same reason that i was having trouble | 19:12 |
| fungi | just basically surmising that unless their dns infrastructure actually *is* bind, actual bind zonefile imports may present some gotchas (dynamic things like generate macros for example) | 19:12 |
| *** NikitaKonovalov has quit IRC | 19:12 | |
| clarkb | fungi: yup, I think that is one of the reasons the feature doesn't exist yet | 19:13 |
| jeblair | fungi: the common subset that most dns tools support would be sufficient. | 19:14 |
| *** senk has joined #openstack-infra | 19:14 | |
| *** NikitaKonovalov has joined #openstack-infra | 19:14 | |
| clarkb | ok really AFKing now | 19:14 |
| clarkb | so hard to leave IRC alone | 19:14 |
| fungi | i expect we could make do with that and just not use any of the more esoteric features of bind's parser | 19:14 |
| jeblair | fungi: yes, i'm just trying to store a dns record in a file without inventing a new way of doing that. :) | 19:15 |
| *** NikitaKonovalov has quit IRC | 19:15 | |
| fungi | scoping also has a tendency to get weird, but they probably would at least implement support for those macros since standard bind zone transfers already add them to reduce the payload size | 19:16 |
| mordred | dstanek: because of trying to parse setup.py? | 19:18 |
| *** senk has quit IRC | 19:18 | |
| dstanek | mordred: i believe so - trying to confirm now | 19:19 |
| openstackgerrit | David Kranz proposed a change to openstack-infra/devstack-gate: Always dump errors to console https://review.openstack.org/61850 | 19:19 |
| dstanek | mordred: what doesn't make sense is if i run the sphinx-apidoc from the command line using . it works fine | 19:20 |
| dstanek | doing it from pbr fails, but if i change . to keystone it works | 19:21 |
| mordred | dstanek: it does? and the docs are created properly? | 19:21 |
| *** kraman has joined #openstack-infra | 19:22 | |
| kraman | mordred: hi | 19:22 |
| mordred | jeblair: so, I've been talking with kraman about using zuul as the basis for the git engine in solum | 19:22 |
| mordred | jeblair: and where that got us is the question about whether it could just be the solum workflow engine itself | 19:23 |
| mordred | jeblair: they had been looking at using the taskflow library to be able to express the workflow things they needed | 19:23 |
| mordred | jeblair: so where that led was the question about whether zuul using taskflow made any sense | 19:24 |
| mordred | jeblair: obviously, at that point, it stopped being a me conversation :) | 19:24 |
| *** claytonc has joined #openstack-infra | 19:24 | |
| kraman | mordred: jeblair: claytonc joining is working solum with me and had similar questions | 19:24 |
| claytonc | i should have brought cookies | 19:26 |
| kraman | mordred: jeblair: the current workflow capabilities in zuul are probably not sufficient as they wont allow for complex decisions like rollbacks or retries. If you were to use taskflow in zuul, those capabilities would be possible | 19:26 |
| *** ^d is now known as ^demon|lunch | 19:27 | |
| claytonc | reading back through the log | 19:28 |
| claytonc | so mordred your point about infra evolving into an aaS matches an opinion I had been privately playing through | 19:29 |
| claytonc | over time, the needs of infra become indistinguishable from a generic git / build / test flow infrastructure, although multitenancy and isolation do play a bit into that | 19:29 |
| *** vipul is now known as vipul-away | 19:30 | |
| mordred | claytonc: right- with the one exception(to the generic pbit) that similar to a paas, we don't really want to let people 'skip' things in the flow | 19:31 |
| claytonc | right. the minimum requirement i think for a viable solum system is the ability to, on request (an http endpoint call) fetch the contents of a repo / folder / zip / svn system and inject those into a running container/vm at a predetermined location, such that the container/vm can then "do something with it" | 19:32 |
| claytonc | it doesn't have to be a repo | 19:32 |
| claytonc | it could be nexus, yum, a ppa, etc | 19:32 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Correct elastic-recheck install subscription scope https://review.openstack.org/61851 | 19:33 |
| mordred | that's right. and that's essentially what we do with zuul and friends here - especially once we add turbo-hipster into the mix | 19:33 |
| *** dims has quit IRC | 19:33 | |
| kraman | mordred: claytonc: another thing discussed was that the Infara…aaS could be eternal and just trigger the rest of solum workflow | 19:33 |
| clarkb | fungi: on 61851 can you do a quick grep for ::elastic_recheck to see if there are any other cases of that? | 19:34 |
| mordred | claytonc: (turbo-hipster is the beginnings of the non-jenkins thing that does things based on events from zuul) | 19:34 |
| fungi | clarkb: good idea--checking | 19:34 |
| *** dims has joined #openstack-infra | 19:34 | |
| fungi | clarkb: seems no | 19:35 |
| *** yolanda has quit IRC | 19:35 | |
| mordred | kraman, claytonc: http://git.openstack.org/cgit/stackforge/turbo-hipster | 19:35 |
| claytonc | mordred: ok, so and then that call from solum needs to dispatch the call to zuul, zuul would have a preconfigured flow already that would then manage the details of moving data from point A (repo/zip/archive) to point B (inside of running VM/container that solum established) | 19:35 |
| clarkb | fungi: awesome I couldn't remember if more than one instance of that happened | 19:35 |
| fungi | clarkb: and using that change in a development environment fixed puppetry for logstash.o.o | 19:35 |
| sdague | hemanth_: https://review.openstack.org/#/c/60908/ - reverify no bug has been removed | 19:35 |
| clarkb | fungi: so it has been tested? even better | 19:35 |
| * fungi nods | 19:35 | |
| clarkb | fungi: approved | 19:36 |
| claytonc | and then in that VM for point B (which for the purposes of Solum would prefer to be completely opaque), zuul would simply need to invoke one endpoint/script/something for the VM to do its stuff | 19:36 |
| kraman | claytonc: mordred: concern that default solum install would still have 2 workflow engines, one for source fetch side, and one for rest of solum | 19:36 |
| clarkb | zaro: did you make it to the office today? I may just give up on doing that at this point | 19:36 |
| *** salv-orlando has joined #openstack-infra | 19:36 | |
| claytonc | kraman: so thinking more about this, i think we could say that the two separate engines are different | 19:36 |
| mordred | claytonc: that's exactly right | 19:36 |
| claytonc | because zuul is handling a very proscribed flow | 19:36 |
| claytonc | it doesn't vary much | 19:36 |
| claytonc | at least today | 19:37 |
| hemanth_ | sdague: so, does it mean jenkins doesn't re-run tests anymore? | 19:37 |
| kraman | claytonc: thats assuming that there would be only one flow per install of solum | 19:37 |
| clarkb | claytonc: it actually can vary quite a bit | 19:37 |
| clarkb | wikimedia and others are using it with very different workflows | 19:37 |
| claytonc | clarkb: i mean for solum's needs | 19:37 |
| claytonc | kraman: you're right there can be multiple flows | 19:37 |
| claytonc | but i think this flow is actually pretty concrete | 19:37 |
| kraman | claytonc: agree that this one is, but dont want to make auumptions about others | 19:38 |
| claytonc | the "solum" workflow is a different kind of use case - it's making rest calls to things and fetching data and updating state | 19:38 |
| claytonc | that's always gotta exist, but the key here is that it *doesn't* have to talk to git internally | 19:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Correct elastic-recheck install subscription scope https://review.openstack.org/61851 | 19:38 |
| claytonc | it's the operators choice to configure zuul in such a way that it can pull many different type of things | 19:38 |
| claytonc | now there are some details here | 19:38 |
| kraman | claytonc: specially considering that there were opinions during F2F where (i forget who) wanted the git flow and lang-pack flow to be linked and run tests before accepting the build | 19:38 |
| mordred | yup. and the triggers and pipelines are configurable | 19:38 |
| praneshp | I need some assistance with https://review.openstack.org/#/c/61615/ . I keep seeing jenkins failure wiht error: 'source_dir' must be a directory name (got `/home/jenkins/workspace/gate-entropy-docs/docs`), not sure what the issue is. docs is a directory. | 19:39 |
| claytonc | i think one detail is security - in a proper MT paas you have isolation of git repos and high security | 19:39 |
| clarkb | praneshp: you need to pin Sphinx to <1.2 in your requirements | 19:39 |
| zaro | clarkb: ohh why give up? | 19:39 |
| clarkb | zaro: because I am still somehow at home | 19:39 |
| claytonc | and today, solum is the only one that knows enough to take security A of git repo 1 and make sure zuul knows enough to that zuul can access git repo 1 and then send it to VM 1 that Solum also created | 19:39 |
| praneshp | clarkb justin the test-requirements right? | 19:39 |
| claytonc | however that seems solvable with appropriate input to zuul | 19:40 |
| zaro | clarkb: i'm at the office. had to come in to use external monitor | 19:40 |
| clarkb | praneshp: the recent release of Sphinx 1.2 doesn't play nice with distutils and that error is a result | 19:40 |
| mordred | clarkb: ++ | 19:40 |
| clarkb | praneshp: wherever you have sphinx listed today | 19:40 |
| praneshp | its already pinned that way. | 19:40 |
| mordred | gah | 19:40 |
| mordred | claytonc: ++ | 19:40 |
| kraman | claytonc: mordred I can see solum setting up config for zuul per app basis | 19:40 |
| zaro | clarkb: will probably need to be in office until i get another laptop. | 19:40 |
| *** SergeyLukjanov_ has joined #openstack-infra | 19:40 | |
| kraman | mordred: would that work? | 19:40 |
| claytonc | kraman: so i think the way i would frame this argument is "zuul is responsible for managing a complete flow from a source endpoint into a destination vm, an invocation of a script in that destination vm (parameterized by the solum call), and the end running state on the vm | 19:40 |
| praneshp | clarkb its already pinned that way | 19:41 |
| claytonc | kraman: i'm thinking one zuul per solum install that handles this kind of flow | 19:41 |
| fungi | zaro: do you have a cot you can use there, or do you have to sleep on a sofa in the breakroom? | 19:41 |
| mordred | kraman: yup. zuul already has per-app configs, that are tied to configurable pipeline definitions | 19:41 |
| *** rcarrillocruz1 has quit IRC | 19:41 | |
| mordred | claytonc: ++ | 19:41 |
| kraman | claytonc: so you are ok with zuul driving the vm build etc? | 19:41 |
| zaro | fungi: no cot or sofa here, would be nice though :) | 19:42 |
| claytonc | kraman: yeah i think we should draw it out in a picture and show the isolation interface between solum and it | 19:42 |
| claytonc | and it definitely needs some example input - output - expectations | 19:42 |
| kraman | works for me | 19:42 |
| mordred | I think that's still a different thing - but I think zuul tells a thing that it needs a vm that looks like X | 19:42 |
| clarkb | praneshp: I don't knwo what the problem is then | 19:42 |
| kraman | that part of the workflow is pretty simple as well | 19:42 |
| kraman | no complex logic | 19:42 |
| claytonc | right | 19:42 |
| mordred | but yes to picture with interfaces :) | 19:42 |
| fungi | praneshp: one possibility is that one of your other requirements is pulling in a newer sphinx than you have pinned. the dep solver in pip is currently fairly braindead | 19:43 |
| kraman | so flow is solum post trigger -> setup up zuul config -> tigger zuul -> zuul fet & build vm -> trigger solum with built vm artifact | 19:43 |
| kraman | claytonc: mordred ^ | 19:43 |
| praneshp | yeag fungi, looking into it now. Thanks | 19:43 |
| fungi | praneshp: try putting the sphinx line at the top of requirements.txt instead of in test-requirements.txt and see if it helps | 19:43 |
| *** SushilKM__ has joined #openstack-infra | 19:43 | |
| claytonc | mordred: is zuul view authenticated today? | 19:44 |
| praneshp | its already on the first line :) | 19:44 |
| claytonc | are jobs viewable only with authentication? | 19:44 |
| praneshp | oh requirements.txt | 19:44 |
| mordred | claytonc: nope | 19:44 |
| praneshp | ok, let me try | 19:44 |
| fungi | praneshp: yeah, because requirements is processed before test-requirements | 19:44 |
| lifeless | ?win 75 | 19:44 |
| lifeless | bah | 19:44 |
| kraman | mordred: claytonc hm, would need to add that | 19:44 |
| claytonc | kraman: we could simply make zuul invisible to solum users | 19:44 |
| fungi | praneshp: so my guess is something in requirements is also depending on sphinx and lacks that cap | 19:44 |
| *** senk has joined #openstack-infra | 19:44 | |
| claytonc | and either mediate or proxy the items in the short term | 19:44 |
| *** vipul-away is now known as vipul | 19:44 | |
| mordred | claytonc: the 'view' is actually just a json status blb that we transform into the view | 19:45 |
| kraman | claytonc: why prevent users from seeing why their build failed | 19:45 |
| claytonc | kraman: not prevent users | 19:45 |
| claytonc | kraman: multi tenant situation | 19:45 |
| mordred | claytonc: so if you had a different view that had auth, it could easily filter the status contents | 19:45 |
| claytonc | user a shouldn't see user b's jobs | 19:45 |
| claytonc | mordred: yeah | 19:45 |
| kraman | if we add auth, that shoudl solve multi tenant | 19:45 |
| mordred | in fact, we have job filtering in the status view | 19:45 |
| claytonc | kraman: we could impose auth above zuul | 19:45 |
| mordred | so it's really just about injecting auth-associated filters | 19:45 |
| claytonc | zuul is a true engine | 19:45 |
| mordred | into the html | 19:45 |
| *** SushilKM has quit IRC | 19:46 | |
| claytonc | run and operated by the operator | 19:46 |
| claytonc | tenants would have limited access to it anyway | 19:46 |
| kraman | mordred: adding auth filters already possible? | 19:46 |
| *** SergeyLukjanov has quit IRC | 19:46 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 19:46 | |
| mordred | kraman: no - but I think injecting that into the presentation layer should be very straightforward | 19:46 |
| kraman | k | 19:47 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Promote Elastic-Recheck page as a tab by itself https://review.openstack.org/61856 | 19:47 |
| kraman | mordred: claytonc k, im sold :) | 19:47 |
| mordred | woot | 19:47 |
| *** SergeyLukjanov_ has joined #openstack-infra | 19:47 | |
| claytonc | :) | 19:47 |
| kraman | need time to play with some of this. mordred… hoping you can help me | 19:47 |
| mordred | absolutely | 19:47 |
| *** SergeyLukjanov has quit IRC | 19:47 | |
| *** SergeyLukjanov_ has quit IRC | 19:48 | |
| *** SergeyLukjanov has joined #openstack-infra | 19:48 | |
| mordred | I'll make sure to prep my laptop with a few repos and notes for a little hacking tomorrow too | 19:48 |
| mordred | (I have an 8 hour plane flight) | 19:48 |
| kraman | mordred: claytonc: cool, if we can have a messy prototype by next meeting it will greatly help understand this | 19:48 |
| kraman | maybe 2 meetings | 19:48 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Elastic Recheck should be a separate tab https://review.openstack.org/61857 | 19:48 |
| mordred | ++ | 19:48 |
| kraman | mordred: if you can forward any notes on setting this up, would appreciate it | 19:49 |
| kraman | claytonc: thanks | 19:49 |
| *** senk has quit IRC | 19:49 | |
| clarkb | kraman: have you seen http://ci.openstack.org/zuul? | 19:49 |
| kraman | clarkb: very briefly, havent dug into it yet | 19:50 |
| fungi | it's actually remarkably thorough documentation (there are lots of organizations which have installed and gotten it working straight from the docs) | 19:50 |
| clarkb | and http://status.openstack.org/zuul ? Those are sort of the two user facing docs and dashboard available to folks that we have up today | 19:50 |
| *** zhiyan has joined #openstack-infra | 19:50 | |
| *** yolanda has joined #openstack-infra | 19:50 | |
| kraman | fungi: sounds good. ill start there | 19:50 |
| fungi | obviously your needs are probably more generalized than the suggestions in the documentation since they assume a fairly openstackish developer workflow, but it should provide a good example at least | 19:51 |
| kraman | yep | 19:51 |
| *** melwitt has quit IRC | 19:58 | |
| *** ArxCruz has quit IRC | 20:01 | |
| *** spredzy has quit IRC | 20:02 | |
| dstanek | mordred: good times! http://paste.openstack.org/show/54910/ fixes my issue by allowing me to specifically set the directory | 20:04 |
| clarkb | dstanek: even with the default ['.' | 20:05 |
| clarkb | * ['.'] ? | 20:05 |
| *** kraman has left #openstack-infra | 20:06 | |
| clarkb | nevermind sounds like you are using that to specifically change the root | 20:06 |
| clarkb | dstanek: want to propose that against pbr? I don't see anything wrong with making that configurable | 20:06 |
| clarkb | dstanek: though you may want an env var like the other options have | 20:07 |
| *** markmcclain has joined #openstack-infra | 20:07 | |
| *** denis_makogon_ has joined #openstack-infra | 20:08 | |
| dstanek | clarkb: yes, i can dress it up a bit and submit a patch | 20:08 |
| dstanek | i'm not sure that anyone can actually be using the autodoc_tree option right now because of the setup issue | 20:08 |
| *** spredzy has joined #openstack-infra | 20:09 | |
| clarkb | yeah, maybe even remoev the default ['.'] value and make it a required option if autodoc tree index modules is enabled | 20:09 |
| clarkb | won't be nicely backward compatible but will make the brokeness apparent to folks | 20:09 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Use a global pip.conf for tempest tox calls https://review.openstack.org/61861 | 20:10 |
| *** denis_makogon_ is now known as denis_makogon | 20:10 | |
| dstanek | would it make sense to use autodoc_tree_index_modules for the path instead of a boolean? | 20:11 |
| dstanek | clarkb: ^ | 20:12 |
| *** rfolco has quit IRC | 20:12 | |
| *** oubiwan__ has quit IRC | 20:12 | |
| *** dcramer_ has quit IRC | 20:12 | |
| clarkb | dstanek: I think so particularly if we decide that we need to be not so nice for backward compat to point out brokeness | 20:12 |
| fungi | right... "ERROR: your API docs were never actually working, so you need to supply a path." | 20:13 |
| *** vipul is now known as vipul-away | 20:13 | |
| *** vipul-away is now known as vipul | 20:13 | |
| dstanek | clarkb: fungi: i actually think right now the command would fail using the option; do you know if anyone is successfully using it? | 20:13 |
| fungi | good point--no idea but i doubt it given the circumstances | 20:14 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add e-r query for bug 1259907 https://review.openstack.org/61826 | 20:14 |
| *** jcoufal has joined #openstack-infra | 20:14 | |
| uvirtbot | Launchpad bug 1259907 in openstack-ci "check-grenade-dsvm marked as FAILED - n-api/g-api Logs have errors" [Undecided,New] https://launchpad.net/bugs/1259907 | 20:15 |
| clarkb | fungi: is PIP_CONF_FILE something pip understands on its own? | 20:15 |
| *** reed has quit IRC | 20:15 | |
| fungi | clarkb: clarkb oops, i misspelled it | 20:17 |
| fungi | clarkb: i'll add a link to the commit message for the pip docs on it too | 20:18 |
| clarkb | fungi: thanks | 20:18 |
| *** praneshp has quit IRC | 20:21 | |
| *** crank has quit IRC | 20:22 | |
| *** zweiblumen has joined #openstack-infra | 20:22 | |
| *** dcramer_ has joined #openstack-infra | 20:22 | |
| *** crank has joined #openstack-infra | 20:22 | |
| *** praneshp has joined #openstack-infra | 20:25 | |
| *** ryanpetrello has joined #openstack-infra | 20:26 | |
| clarkb | can someone do me a huge favor and leave this comment http://paste.openstack.org/show/54912/ on https://review.openstack.org/#/c/61110/ ? I cannot leave the comment because then I would have to remove my approval but want to allow the reverify bug to work... | 20:28 |
| *** vipul is now known as vipul-away | 20:28 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Use a global pip.conf for tempest tox calls https://review.openstack.org/61861 | 20:29 |
| *** hashar has joined #openstack-infra | 20:29 | |
| fungi | clarkb: done | 20:30 |
| clarkb | fungi: ty | 20:30 |
| fungi | yw | 20:30 |
| clarkb | 61861 lgtm now | 20:30 |
| *** krtaylor has quit IRC | 20:32 | |
| sdague | hey, you guys got a bug for this yet - http://logs.openstack.org/78/61778/4/check/check-tempest-dsvm-neutron/db02710/console.html ? | 20:32 |
| *** reed has joined #openstack-infra | 20:32 | |
| sdague | 2013-12-12 19:41:33.350 | Downloading/unpacking httplib2 from https://pypi.python.org/packages/source/h/httplib2/httplib2-0.8.tar.gz#md5=94cb8a3b196dfd19253c46609489d9f1 (from tempest==2014.1.dev2418.gc124397) | 20:33 |
| sdague | 2013-12-12 19:41:33.350 | Error <urlopen error _ssl.c:489: The handshake operation timed out> while getting https://pypi.python.org/packages/source/h/httplib2/httplib2-0.8.tar.gz#md5=94cb8a3b196dfd19253c46609489d9f1 | 20:33 |
| sdague | 2013-12-12 19:41:33.350 | Cleaning up... | 20:33 |
| *** UtahDave has joined #openstack-infra | 20:33 | |
| clarkb | sdague: yup 61861 is the fix and has a buglink | 20:33 |
| sdague | clarkb: did you propose an e-r for it? | 20:33 |
| *** SushilKM__ has quit IRC | 20:33 | |
| clarkb | sdague: I did not | 20:34 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Document adding bug signatures to e-r. https://review.openstack.org/61298 | 20:34 |
| *** herndon has quit IRC | 20:35 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add bug for pypi timeout hitting infra https://review.openstack.org/61866 | 20:37 |
| fungi | so we *think* that's the fix. hopefully testing will allow us to review the logs and confirm | 20:38 |
| sdague | clarkb: ok, I just added it | 20:39 |
| sdague | one of my bug fixes in tempest bounced off it a second ago | 20:39 |
| fungi | i need to step out briefly for an errand, but should be back in roughly an hour | 20:39 |
| * fungi out and about | 20:39 | |
| clarkb | fungi: ok see you then | 20:39 |
| *** yolanda has quit IRC | 20:39 | |
| clarkb | jeblair: mordred: reviews on 61861 when check tests come back would be great | 20:40 |
| *** dcramer_ has quit IRC | 20:40 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add bug for pypi timeout hitting infra https://review.openstack.org/61866 | 20:41 |
| jeblair | fungi: what did you determine regarding pydistutils.cfg? | 20:41 |
| jeblair | fungi: afaik, it would still be used by transitive deps. | 20:41 |
| jeblair | fungi: if it's not able to set with a var like pip, then we may want to just copy both of them into ~tempest instead. :/ | 20:42 |
| jeblair | clarkb, mordred: ^ | 20:42 |
| clarkb | jeblair: looks like systemprefix/lib/pythonver/distutils/distutils.cfg can be used | 20:42 |
| clarkb | http://docs.python.org/2/install/#location-and-names-of-config-files | 20:43 |
| *** dcramer_ has joined #openstack-infra | 20:44 | |
| clarkb | which is a bit tricky because tox is involved so prefix will be in .tox/virtualenvname/lib/pythonver/andsoon | 20:44 |
| dkranz | clarkb, fungi : Here is the patch to dump non-whitelisted log errors on failed tests. Should help with better bug reporting. https://review.openstack.org/#/c/61543/ | 20:44 |
| clarkb | I wonder if virtualenv will copy that file when you create a virtualenv? | 20:45 |
| dkranz | clarkb, fungi : The change is in devstack-gare | 20:45 |
| dkranz | Grr. devstack-gate | 20:45 |
| clarkb | jeblair: slightly leaning towards using the homdir | 20:47 |
| clarkb | jeblair: I think dealing with this per virtualenv that also allows site packages could get confusing really quickly | 20:47 |
| jeblair | clarkb: me too. i don't expect more users. at least, not any great number. possibly a code comment where users are added would help us keep from bitrotting. | 20:48 |
| clarkb | ++ | 20:48 |
| clarkb | dkranz: will take a look after lunch | 20:48 |
| clarkb | I am going to dive into code review then | 20:48 |
| clarkb | I am almost completely caught up on code review and need to stay caught up | 20:49 |
| *** vipul-away is now known as vipul | 20:49 | |
| dkranz | clarkb: THANKS | 20:49 |
| *** jcooley_ has quit IRC | 20:49 | |
| openstackgerrit | Felipe Reyes proposed a change to openstack-infra/jenkins-job-builder: Adds Mercurial SCM support https://review.openstack.org/61547 | 20:51 |
| *** reed has quit IRC | 20:53 | |
| *** zweiblumen has quit IRC | 20:58 | |
| *** datsun180b has quit IRC | 20:59 | |
| *** melwitt has joined #openstack-infra | 20:59 | |
| clarkb | I am going to go grab lunch really quick now back in ab it | 21:03 |
| *** alcabrera has quit IRC | 21:03 | |
| *** boris-42 has joined #openstack-infra | 21:03 | |
| *** gyee has joined #openstack-infra | 21:04 | |
| *** zhiyan has quit IRC | 21:05 | |
| *** rwsu has quit IRC | 21:05 | |
| *** reed has joined #openstack-infra | 21:05 | |
| *** pabelanger has quit IRC | 21:05 | |
| *** yassine has joined #openstack-infra | 21:07 | |
| *** lcestari has quit IRC | 21:08 | |
| *** dkranz is now known as dkranz_in_transi | 21:14 | |
| *** AlexF has joined #openstack-infra | 21:14 | |
| *** freyes has quit IRC | 21:16 | |
| *** apevec has joined #openstack-infra | 21:17 | |
| *** dkranz_in_transi has quit IRC | 21:19 | |
| apevec | nice, gate is speeding up, 11.5h https://review.openstack.org/61449 | 21:22 |
| *** oubiwan__ has joined #openstack-infra | 21:22 | |
| zaro | fungi: got time to give this one a look? https://review.openstack.org/#/c/47937 | 21:23 |
| zaro | fungi: particularly the rst. | 21:23 |
| *** krtaylor has joined #openstack-infra | 21:25 | |
| *** smarcet has left #openstack-infra | 21:26 | |
| *** dkliban has quit IRC | 21:26 | |
| openstackgerrit | David Stanek proposed a change to openstack-dev/pbr: Fixes autodoc_tree_index_modules https://review.openstack.org/61875 | 21:29 |
| *** sarob_ has joined #openstack-infra | 21:30 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 21:31 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Document adding bug signatures to e-r. https://review.openstack.org/61298 | 21:32 |
| sdague | so if the root issue on a bug is we need a possibly new kind of test job in infrastructure, is that openstack-ci able? | 21:32 |
| sdague | specifically am triaging this bug - https://bugs.launchpad.net/tempest/+bug/1149928 | 21:33 |
| uvirtbot | Launchpad bug 1149928 in tempest "Add glance v2 image data upload to swift test" [Undecided,Incomplete] | 21:33 |
| sdague | we actually have that test | 21:33 |
| sdague | we just don't have that configuration | 21:33 |
| *** sarob has quit IRC | 21:33 | |
| apevec | hmm, curl to pypi timing out, don't we have pypi mirror? http://logs.openstack.org/14/61714/2/check/check-tempest-dsvm-neutron/d9f3130/console.html | 21:34 |
| openstackgerrit | David Stanek proposed a change to openstack-dev/pbr: Fixes autodoc_tree_index_modules https://review.openstack.org/61875 | 21:34 |
| *** sarob_ has quit IRC | 21:34 | |
| apevec | + curl -O https://pypi.python.org/packages/source/p/pip/pip-1.4.1.tar.gz | 21:34 |
| apevec | curl: (28) SSL connection timeout | 21:35 |
| *** jhesketh_ has joined #openstack-infra | 21:35 | |
| *** oubiwan__ is now known as oubiwann-lambda | 21:35 | |
| *** krotscheck has joined #openstack-infra | 21:36 | |
| *** nati_ueno has quit IRC | 21:36 | |
| jeblair | apevec: we have discovered that when tempest started using a venv it stopped using the mirror; fix in progress | 21:36 |
| clarkb | sdague: glance cant support old and new together? | 21:37 |
| clarkb | eg change devstack to do both and done | 21:38 |
| apevec | jeblair, but wasn't above failure in devstack? | 21:38 |
| openstackgerrit | Dean Troyer proposed a change to openstack-infra/elastic-recheck: Add query for bug 1256970 https://review.openstack.org/61878 | 21:40 |
| uvirtbot | Launchpad bug 1256970 in devstack "Failure uploading bundle bundle.img to testbucket" [Undecided,New] https://launchpad.net/bugs/1256970 | 21:40 |
| sdague | clarkb: it's not the glance version | 21:40 |
| sdague | it's that it has to be backed by swift | 21:40 |
| clarkb | sdague: and the existing job isn't configured that way? (I thought it was) | 21:43 |
| *** sarob has joined #openstack-infra | 21:43 | |
| *** rnirmal has quit IRC | 21:45 | |
| *** SergeyLukjanov has joined #openstack-infra | 21:45 | |
| pleia2 | jeblair: followed up on freenode ticket, openstackinfra account should be good to go re: founder limit removed | 21:47 |
| pleia2 | jeblair: so one day, when you're bored... :) | 21:47 |
| dims | jeblair, is the plan to generate json file with data from recheckwatch that can be used in elastic-recheck? | 21:48 |
| *** sarob has quit IRC | 21:48 | |
| jeblair | pleia2: awesome, thanks! | 21:48 |
| jeblair | dims: sdague is working on collecting recheck comments in elastic-recheck | 21:48 |
| *** marun has quit IRC | 21:48 | |
| *** sarob has joined #openstack-infra | 21:48 | |
| dims | jeblair, k thanks | 21:49 |
| sdague | dims: yeh I was working on foundation code for that yesterday | 21:49 |
| sdague | I probably won't get back to it until monday | 21:49 |
| sdague | clarkb: no, glance goes to the filesystem IIRC | 21:50 |
| clarkb | I thought some of the gate flakyness previously was swift giving glance a hard time | 21:51 |
| *** sarob has quit IRC | 21:51 | |
| *** dkliban has joined #openstack-infra | 21:51 | |
| *** ^demon|lunch has quit IRC | 21:52 | |
| dims | sdague, ok just curious | 21:54 |
| *** hogepodge has joined #openstack-infra | 21:55 | |
| sdague | clarkb: no it was just swift going over limits | 21:55 |
| apevec | where is it written who are PTLs ? | 21:56 |
| fungi | okay, back... reading | 21:56 |
| apevec | ttx, adam_g, we forgot to add Heat and Ceilo PTLs to stable-maint, now that they're part of stable releases, right? | 21:57 |
| *** dkranz_in_transi has joined #openstack-infra | 21:57 | |
| *** dkranz_in_transi is now known as dkranz | 21:58 | |
| apevec | ok, found [openstack-dev] PTL Election Conclusion and Results | 21:58 |
| clarkb | sdague: gotcha, so it was swift being directly tested rather than via glance | 21:59 |
| *** AlexF has quit IRC | 22:00 | |
| *** sandy__ has quit IRC | 22:00 | |
| clarkb | fungi: tl;dr we think we shuold set it in the user homedir for tempest and document where we add users that thsi is necessary | 22:00 |
| david-lyle | apevec: funny thing there, I doubt I'm on that list | 22:00 |
| fungi | apevec: you can also mine ptls out of the list in the openstack/governance repo now, i believe | 22:01 |
| fungi | clarkb: i agree--that was my original intent, but then some global solution we could tie into for future account additions got floated | 22:01 |
| apevec | david-lyle, didn't you get my email the other, you should've been added | 22:01 |
| fungi | apevec: http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml | 22:02 |
| fungi | i've started treating that as the official record | 22:02 |
| jeblair | clarkb: how long does it take logstash to parse a single log file? | 22:02 |
| apevec | david-lyle, http://lists.openstack.org/pipermail/openstack-stable-maint/2013-December/001810.html | 22:03 |
| *** changbl has quit IRC | 22:03 | |
| *** ryanpetrello has quit IRC | 22:03 | |
| apevec | fungi, thanks | 22:03 |
| *** AlexF has joined #openstack-infra | 22:03 | |
| apevec | david-lyle, double-checking, you're in: https://review.openstack.org/#/admin/groups/120,members | 22:04 |
| clarkb | jeblair: not very long now that we prefilter out all DEBUG. I can look at the logs on the pusher nodes to give exact numbers | 22:04 |
| clarkb | jeblair: I think it is in the just under a second range | 22:04 |
| *** saper_ has joined #openstack-infra | 22:05 | |
| *** julim_ has joined #openstack-infra | 22:05 | |
| *** mriedem has quit IRC | 22:05 | |
| david-lyle | apevec: looks like it | 22:05 |
| *** saper has quit IRC | 22:05 | |
| *** mriedem has joined #openstack-infra | 22:05 | |
| jeblair | clarkb: i'll try to dig into logs, thx | 22:05 |
| *** pabelanger has joined #openstack-infra | 22:06 | |
| clarkb | jeblair: in the logs yo ushould see entries that look like pushing ##### log lines into logstash, then lines for requesting the next job | 22:06 |
| clarkb | the difference in timestamps should be pretty accurate there (there is some buffering but that gets pipelined so we can mostly ignore that I think) | 22:06 |
| *** AlexF has quit IRC | 22:07 | |
| jeblair | clarkb: got it, thx | 22:07 |
| *** julim has quit IRC | 22:07 | |
| *** markmcclain has quit IRC | 22:07 | |
| *** mfer has quit IRC | 22:09 | |
| *** boris-42 has quit IRC | 22:11 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Cover the tempest user in select_mirror https://review.openstack.org/61861 | 22:11 |
| *** sandy__ has joined #openstack-infra | 22:12 | |
| clarkb | fungi: see comments | 22:13 |
| *** vkozhukalov has quit IRC | 22:15 | |
| fungi | clarkb: gah, right | 22:15 |
| fungi | i also am going to add a comment where accounts are being added in that script with a reminder to update the function | 22:15 |
| clarkb | fungi: slightly worried the cp's might not work if tempest can't read those files... | 22:15 |
| clarkb | fungi: wasn't sure if you were working around that there with -u root hence my lack of -1ing | 22:16 |
| fungi | clarkb: also a good idea... i'll chown | 22:16 |
| clarkb | wfm | 22:16 |
| *** paul-- has joined #openstack-infra | 22:16 | |
| jeblair | clarkb: it seems to be about 2300 lines/sec. | 22:17 |
| jeblair | clarkb: what are the implications if we made log processing take ~4x as long? | 22:18 |
| jeblair | clarkb: (so yes, you were right, that's something like a range of .8-3 seconds depending on file, including download time) | 22:19 |
| clarkb | jeblair: ~4x as long is probably fine. That is still short enough that we can keep up with any potential backlog | 22:22 |
| clarkb | jeblair: before we eliminated DEBUG logs we were looking at a minute per file | 22:22 |
| clarkb | jeblair: are yo uwanting to kill some of the worker nodes? | 22:22 |
| *** dprince has quit IRC | 22:22 | |
| jeblair | clarkb: no, i'm experimenting with crm114 | 22:22 |
| *** hashar has quit IRC | 22:23 | |
| clarkb | jeblair: oh cool (and by cool, I mean incredibly awesome I am super excited :) ) | 22:23 |
| *** claytonc has left #openstack-infra | 22:23 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/devstack-gate: Cover the tempest user in select_mirror https://review.openstack.org/61861 | 22:23 |
| clarkb | jeblair: so yeah I think we can take that hit | 22:23 |
| jeblair | cool, i'll keep going on this course then. :) | 22:23 |
| fungi | we'll think back on this as the day when skynet was still a twinkle in a log analyzer's eye | 22:23 |
| *** eharney has quit IRC | 22:24 | |
| jeblair | fungi: i still want to use nupic for it, but i haven't had time to learn that yet. :/ | 22:24 |
| fungi | or maybe we won't, since all organic life will be cleansed from the earth | 22:24 |
| clarkb | jeblair: nupic? | 22:24 |
| clarkb | google has shown me | 22:24 |
| fungi | clarkb: they gave a very neat preso at oscon | 22:25 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added config options to not overwrite jobs desc https://review.openstack.org/52080 | 22:25 |
| clarkb | jeblair: super excited to see what you have when you are ready to show it off | 22:26 |
| * clarkb goes back to reviewing. so close to being all caught up | 22:26 | |
| * fungi smells a fosdem paper in the making | 22:27 | |
| SergeyLukjanov | re: git2lp removal from jeepyb - all deps was merged to os-infra/config, so, we can now completely remove mappings from jeepyb - https://review.openstack.org/#/c/60596/ | 22:28 |
| clarkb | fungi: I am going to talk about the whole setup at LCA next month. :) | 22:29 |
| *** ^demon|lunch has joined #openstack-infra | 22:29 | |
| *** ^demon|lunch has quit IRC | 22:29 | |
| mgagne | SergeyLukjanov: soon removal of hard-coded direct_release list? =) | 22:30 |
| SergeyLukjanov | mgagne, yup | 22:30 |
| mgagne | SergeyLukjanov: awesome :D | 22:30 |
| SergeyLukjanov | mgagne, right after it'll be checked | 22:30 |
| *** thomasem has quit IRC | 22:30 | |
| SergeyLukjanov | mgagne, I'm thinking about how to safely check it | 22:30 |
| *** thomasem has joined #openstack-infra | 22:31 | |
| SergeyLukjanov | the full check requires merging some change that fixes bug in some lp project | 22:31 |
| *** thomasem has quit IRC | 22:31 | |
| SergeyLukjanov | I don't think that there could be any bugs but I'd like to check that it works | 22:31 |
| *** pabelanger has quit IRC | 22:32 | |
| *** herndon has joined #openstack-infra | 22:33 | |
| *** denis_makogon has quit IRC | 22:33 | |
| *** dangers is now known as danger_fo_away | 22:38 | |
| *** krtaylor has quit IRC | 22:38 | |
| *** pabelanger has joined #openstack-infra | 22:40 | |
| *** herndon has quit IRC | 22:41 | |
| clarkb | fungi: last question, does ~/.pip need chowning too? | 22:45 |
| *** ladquin has joined #openstack-infra | 22:46 | |
| *** senk has joined #openstack-infra | 22:47 | |
| clarkb | fungi: nevermnid I can read | 22:47 |
| dkranz | clarkb: For you, probably pasting the correct code would be easier then helping me, if that works for you. | 22:48 |
| SergeyLukjanov | could we use python script as slave_script? | 22:48 |
| SergeyLukjanov | oh, find an example | 22:49 |
| *** senk has quit IRC | 22:51 | |
| *** dcramer_ has quit IRC | 22:52 | |
| clarkb | dkranz: commented | 22:52 |
| clarkb | SergeyLukjanov: ? | 22:52 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: add reference to launchpad https://review.openstack.org/61890 | 22:52 |
| dkranz | clarkb: Great. Will do. | 22:54 |
| *** mriedem1 has joined #openstack-infra | 22:54 | |
| SergeyLukjanov | clarkb, looks like it'll be better to rewrite projects yaml sort check on python with yaml lib | 22:54 |
| clarkb | SergeyLukjanov: oh gotcha | 22:54 |
| *** mriedem has quit IRC | 22:54 | |
| SergeyLukjanov | clarkb, looking now on what libs we have on slaves | 22:54 |
| clarkb | AaronGr: did you catch jeblair's comment about possibly digging into the manage-project derpage | 22:58 |
| *** pabelanger has quit IRC | 22:58 | |
| clarkb | AaronGr: is looking into fixing that something you might be interested in? not sure where you are in getting settled | 22:58 |
| fungi | clarkb: i think i am the only infra core staying behind and holding down the electrons while lca happens | 22:59 |
| fungi | sounds like we will be well-represented there | 23:00 |
| clarkb | fungi: :/ I think that is right | 23:00 |
| fungi | i'll just blame all the broken things on te three of you that week. nobody will be the wiser ;) | 23:00 |
| clarkb | you shuold | 23:00 |
| anteaya | fungi: do you want some sand mailed to you? | 23:01 |
| *** jergerber has quit IRC | 23:02 | |
| fungi | anteaya: nah, there's plenty of sand here. i may just get a few cases of aussie beer and hold an anti-lca on the opposite end of the planet | 23:02 |
| anteaya | now there's a thought | 23:02 |
| AaronGr | clarkb: i'm at the 'doesn't know squat' stage -- i'll feel a bit more comfortable next week after a prolonged q&a i'm scheduled for. | 23:03 |
| anteaya | well at least you are showing up, AaronGr | 23:03 |
| fungi | AaronGr: that's an excellent stage to be at. then when you break it you have an excuse (unlike the rest of us) | 23:03 |
| openstackgerrit | David Kranz proposed a change to openstack-infra/devstack-gate: Always dump errors to console https://review.openstack.org/61850 | 23:03 |
| anteaya | points for that | 23:03 |
| openstackgerrit | Dean Troyer proposed a change to openstack-infra/elastic-recheck: Add query for bug 1256970 https://review.openstack.org/61878 | 23:03 |
| *** bpokorny has quit IRC | 23:03 | |
| uvirtbot | Launchpad bug 1256970 in devstack "Failure uploading bundle bundle.img to testbucket" [Medium,Confirmed] https://launchpad.net/bugs/1256970 | 23:03 |
| AaronGr | i'm hoping by just being present I'll at least figure out what questions to ask :) | 23:04 |
| clarkb | AaronGr: ok | 23:04 |
| fungi | AaronGr: that's all i did (actually, i think that still may be all i do) | 23:04 |
| *** ianw has quit IRC | 23:04 | |
| clarkb | AaronGr: if you haven't yet we should get you sorted through with launchpad, gerrit, and the CLA | 23:05 |
| AaronGr | CLA? | 23:05 |
| *** ianw has joined #openstack-infra | 23:05 | |
| dkranz | clarkb: I'm not sure your code is right. | 23:05 |
| clarkb | AaronGr: https://wiki.openstack.org/wiki/CLA#If_you.27re_a_developer contributor license agreement | 23:05 |
| fungi | AaronGr: https://wiki.openstack.org/wiki/How_To_Contribute (better starting point) | 23:05 |
| dkranz | clarkb: If the build passes, but check_logs fails, won't this return success? | 23:05 |
| fungi | oh, though CLA does just redirect there now | 23:06 |
| clarkb | AaronGr: you'll want to read thruogh the wiki article and feel free to ask questions | 23:06 |
| clarkb | dkranz: oh maybe | 23:06 |
| *** dkranz is now known as dkranz_dinner | 23:08 | |
| clarkb | dkranz: after tools check logs runs we need to update $res if tools check logs fails | 23:08 |
| AaronGr | clarkb: ok, i'm hooked up with launchpad, i'm on the review site, i'm even watching the openstack-infra project. I'm digging through ci and the puppet repo, anything else you'd suggest as a study aid? | 23:09 |
| fungi | AaronGr: mailing lists | 23:09 |
| fungi | http://lists.openstack.org/ | 23:09 |
| fungi | sign up for some. at least -infra and -announce and probably -dev as well | 23:09 |
| anteaya | and logs: http://eavesdrop.openstack.org/ | 23:09 |
| clarkb | yeah -dev as well, just be prepared to have it fill up quickly :) | 23:10 |
| clarkb | so filter as appropriate | 23:10 |
| fungi | e-mail filters at the ready, yes | 23:10 |
| AaronGr | ok -- thank god for sieve. signing up for the ones you suggested, and bracing. | 23:10 |
| fungi | an mua with good thread control is also a must, so you can kill uninteresting (to you) threads after a message or two | 23:10 |
| fungi | also, whatever helps you cope with the unending onslaught of information overload | 23:11 |
| fungi | some people meditate. i choose to drink | 23:11 |
| anteaya | AaronGr: o.o is openstack.org | 23:11 |
| anteaya | AaronGr: e-r is elastic-recheck | 23:11 |
| anteaya | AaronGr: d-g is devstack-gate | 23:11 |
| *** jcoufal has quit IRC | 23:12 | |
| AaronGr | that loud crackling you hear is the sound of lightbulbs going on in my head. | 23:12 |
| *** yassine has quit IRC | 23:12 | |
| fungi | AaronGr: you might (and i know this is early but) want to start perusing https://git.openstack.org/ and paying slightly increased attention to things within the openstack-infra and (to a slightly lesser extent) openstack-dev repos | 23:13 |
| fungi | stackforge projects you can more or less ignore there. we help facilitate development for them by providing system resources but basically nothing else | 23:14 |
| *** jgrimm has joined #openstack-infra | 23:14 | |
| clarkb | dkranz_dinner: I have written comments on how to fix that check | 23:14 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1256970 https://review.openstack.org/61878 | 23:15 |
| uvirtbot | Launchpad bug 1256970 in devstack "Failure uploading bundle bundle.img to testbucket" [Medium,Confirmed] https://launchpad.net/bugs/1256970 | 23:15 |
| AaronGr | fungi: thank you. I'm almost afraid to ask, but other than the ui or pulling down copies of each, any easy way to watch repo changes ? | 23:16 |
| anteaya | AaronGr: yeah, acronyms bring me to a screaching halt | 23:16 |
| fungi | mmm... as in watch what's merging to them in realtime? http://status.openstack.org/zuul/ | 23:16 |
| clarkb | also in your gerrit settings you can setup project watches whcih will generate even more email | 23:17 |
| *** dizquierdo has joined #openstack-infra | 23:17 | |
| *** fifieldt has joined #openstack-infra | 23:18 | |
| AaronGr | maybe i'll wait on that, this is quite a bit of info already, it's going to take a little while to digest the contents. | 23:18 |
| anteaya | AaronGr: here are all the merged changes to openstack-infra/config from most recent in decending order: https://review.openstack.org/#/q/status:merged+project:openstack-infra/config,n,z | 23:18 |
| fungi | AaronGr: https://review.openstack.org/#/settings/projects when you're ready to get to that | 23:18 |
| AaronGr | oh, i'm watching openstack-infra/config right now | 23:19 |
| fungi | anteaya: though cgit also gives you that same view, basically | 23:19 |
| clarkb | AaronGr: fwiw I stopped enabling email on my project watches, and poll the web ui instead | 23:19 |
| fungi | AaronGr: so on that watched projects config page, note that if you just want to see what's getting approved for some projects, that's the (poorly-named for us) "submitted changes" column | 23:19 |
| anteaya | fungi: does it now | 23:19 |
| SergeyLukjanov | fungi, clarkb, guys, could you please take a look at CR about adding savanna d-g jobs to tempest exp pipeline? https://review.openstack.org/#/c/61125/ | 23:20 |
| fungi | anteaya: http://git.openstack.org/cgit/openstack-infra/config/log/ | 23:20 |
| fungi | SergeyLukjanov: sure | 23:21 |
| anteaya | AaronGr: cgit is also the name of git.openstack.org | 23:21 |
| clarkb | SergeyLukjanov: yup I just gave it a +2 | 23:21 |
| SergeyLukjanov | fungi, clarkb, thx! | 23:21 |
| AaronGr | ok. i think i'm following so far. thanks fungi, clarkb, anteaya for the information, it's really quite helpful. time to get ahead of the email/filters, and keep digging :) | 23:21 |
| anteaya | fungi: so it does, part of the fun of chiming in for $new_person is I get to learn more stuff too | 23:21 |
| anteaya | AaronGr: happy digging, don't be afraid to ask for clarity | 23:22 |
| fungi | (or sanity, though that's in short supply around here) | 23:22 |
| AaronGr | anteaya: i'll try to make sure you don't regret the offer, expect tougher questions in the weeks ahead. | 23:22 |
| fungi | zaro: i am dedicating the next 30 minutes to digging in on https://review.openstack.org/47937 (sorry for taking so long) | 23:23 |
| anteaya | AaronGr: never | 23:23 |
| anteaya | AaronGr: ask away, be prepared for me not to know the answer as well | 23:23 |
| anteaya | then we can both find out | 23:23 |
| anteaya | :D | 23:23 |
| *** ryanpetrello has joined #openstack-infra | 23:23 | |
| AaronGr | deal! | 23:24 |
| anteaya | great | 23:25 |
| *** herndon has joined #openstack-infra | 23:27 | |
| *** paul-- has quit IRC | 23:31 | |
| lifeless | can anyone help me understand why http://logs.openstack.org/80/60380/2/check/gate-config-layout/c66bf38/console.html is failing ? | 23:31 |
| clarkb | lifeless: sure, looking | 23:31 |
| *** melwitt has quit IRC | 23:31 | |
| fungi | Job gate-grenade-dsvm-stable not defined | 23:31 |
| clarkb | lifeless: Job gate-grenade-dsvm-stable not defined is the error, now I am looking at code | 23:31 |
| *** apevec has quit IRC | 23:32 | |
| clarkb | oh right, because it got renamed to s/stable/forward/ iirc to better reflect what it was actually doing | 23:32 |
| fungi | needs a rebase? | 23:32 |
| clarkb | lifeless: under your experimental pipeline in layout.yaml change the name | 23:33 |
| fungi | ahh, yup | 23:33 |
| *** ^demon|lunch has joined #openstack-infra | 23:33 | |
| fungi | lifeless: in your defense, the name of that job changed between when you started that patch and now | 23:34 |
| *** dstanek has quit IRC | 23:35 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add Savanna d-g jobs to tempest exp pipeline https://review.openstack.org/61125 | 23:35 |
| *** dizquierdo has quit IRC | 23:36 | |
| fungi | zaro: a tip... you can also see that rst rendered as a draft by going to http://docs-draft.openstack.org/37/47937/13/check/gate-ci-docs/0739e5a/doc/build/html/review-security.html | 23:38 |
| lifeless | I still see lots of gate-grenade-dsvm-stable entries after rebasing | 23:38 |
| lifeless | maybe they are all mine ;) | 23:38 |
| lifeless | ah yeah | 23:38 |
| fungi | yep, git grep gate-grenade-dsvm-stable on master returns nada | 23:39 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Add Gantt repositories. https://review.openstack.org/60380 | 23:39 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add job to check projects.yaml alphabetized https://review.openstack.org/61777 | 23:39 |
| *** dstanek has joined #openstack-infra | 23:42 | |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack-infra/devstack-gate: Copy libvirt log files after tempest run https://review.openstack.org/61892 | 23:43 |
| clarkb | jenkins01 died :( | 23:43 |
| fungi | gah! | 23:43 |
| clarkb | looks like it OOMed | 23:43 |
| fungi | context successfully switched. what do you need help with first? | 23:43 |
| clarkb | There is insufficient memory for the Java Runtime Environment to continue. | 23:43 |
| fungi | wow, looking | 23:44 |
| clarkb | fungi: grabbing enough info about the failrue so that we can restart the service and debug later | 23:44 |
| fifieldt | nice :) | 23:44 |
| fungi | yup, doing | 23:44 |
| *** sarob has joined #openstack-infra | 23:44 | |
| clarkb | according to dmesg it wasn't OOMkiller just the JVM killing itself | 23:44 |
| fungi | it oomed inside the jvm? nothing from the klog anywa | 23:45 |
| * jeblair is on deck | 23:45 | |
| fungi | y | 23:45 |
| fungi | that was nice of it. looking at the graph to see where it ended up | 23:45 |
| *** jerryz_ has joined #openstack-infra | 23:45 | |
| fungi | though i can't imagine there's any reason to leave it dead. should we go ahead and start it asap? | 23:45 |
| clarkb | fungi: I want to copy aside the logs really quick | 23:46 |
| clarkb | easier to sift through this way | 23:46 |
| fungi | go for it | 23:46 |
| clarkb | ok done | 23:46 |
| clarkb | you going to start the jenkins service? | 23:47 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=819&rra_id=all and http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=821&rra_id=all make it look like we could assign a lot more ram to the jvm if we wanted | 23:47 |
| fungi | starting | 23:47 |
| SergeyLukjanov | jvm likes to kill itself when OOM in PermGen | 23:47 |
| *** prad has quit IRC | 23:47 | |
| clarkb | fungi: we should probably think about upgrading again, I know the nodepool change needed some work | 23:48 |
| clarkb | *nodepool change for jenkins-dev | 23:48 |
| clarkb | once we have that in and tested we can just do rolling upgrades | 23:48 |
| fungi | it really only pushed about 130mib into swap, and about 30% of ram was in use for cache | 23:49 |
| * clarkb goes to refind those changes | 23:49 | |
| clarkb | fungi: remember newer jenkins cuts down on the number of threadsd needed for slaves significantly | 23:49 |
| clarkb | was it 3 threads per slave to 1? | 23:49 |
| fungi | yep, that was what someone said anyway | 23:49 |
| fungi | zaro said that i think | 23:49 |
| *** krotscheck has left #openstack-infra | 23:49 | |
| fungi | so it shot itself in the head when it reached 20gib used | 23:50 |
| jeblair | fungi: jenkins02 is using >20gib | 23:50 |
| clarkb | I was going to try holding nodes for debugging that ssh exception in tempest, but /me redirects to nodepool on jenkins-dev | 23:51 |
| fungi | so it is | 23:51 |
| fungi | we may want to rolling restart jenkins02 once we get 01 up and running jobs | 23:51 |
| jeblair | fungi: seems like a good idea. | 23:51 |
| fungi | i hate the idea of "planned reboots" to free ram, but palliative care i guess | 23:52 |
| jeblair | also, i was thinking once we have fully dynamic slaves, spinning up jenkins03 and jenkins04. mostly so we don't have to redistribute the long running ones manually. | 23:52 |
| fungi | i think that's a capital idea | 23:52 |
| lifeless | fungi: think of it as killer-monkey time : | 23:53 |
| jeblair | fungi, clarkb: btw performance nodes on rax seem to be cheaper than their standard counterparts for the same amount of ram. | 23:53 |
| fungi | interesting | 23:53 |
| fungi | maybe beefer stuff like jenkins masters and elasticsearch workers need to move there sooner than we'd discussed in that case | 23:53 |
| fungi | er, beefier | 23:53 |
| clarkb | the log fiels are in my homedir on jenkins01 | 23:53 |
| NobodyCam | would this be the correct place to ask a i18n question? | 23:54 |
| fungi | NobodyCam: i'm not sure if the i18n team has an irc channel... fifieldt? | 23:54 |
| fifieldt | #openstack-translation | 23:55 |
| fungi | then i guess they do ;) | 23:55 |
| fifieldt | I can also assist, if needed | 23:55 |
| fifieldt | fungi, I'm not sure it's registered | 23:55 |
| fifieldt | but I put it on the wiki anyway :) | 23:55 |
| hemanth_ | sdague: could you please lift your -1 on https://review.openstack.org/#/c/60908/? | 23:55 |
| NobodyCam | ahh TY | 23:56 |
| fungi | jenkins01 seems to be running many, many jobs | 23:56 |
| fungi | should i go ahead and put jenkins02 into shutdown mode now or wait a bit? | 23:56 |
| *** dkranz_dinner is now known as dkranz | 23:56 | |
| sdague | hemanth_: sure | 23:56 |
| *** jerryz__ has joined #openstack-infra | 23:57 | |
| *** SergeyLukjanov has quit IRC | 23:57 | |
| *** yamahata_ has joined #openstack-infra | 23:58 | |
| *** jerryz_ has quit IRC | 23:58 | |
| hemanth_ | sdague: thank you! | 23:58 |
| *** salv-orlando has quit IRC | 23:58 | |
| clarkb | fungi: should we let it settle a bit? | 23:59 |
| clarkb | though just getting it done with is probably good for sanity | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!