| *** saschpe has quit IRC | 00:00 | |
| jhesketh__ | heh, nothing exciting, just a rebase | 00:00 |
|---|---|---|
| *** shardy has quit IRC | 00:02 | |
| *** shardy has joined #openstack-infra | 00:03 | |
| *** saschpe has joined #openstack-infra | 00:05 | |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Fix tempest installation for neutron api job https://review.openstack.org/83626 | 00:07 |
| openstackgerrit | Benjamin Staffin proposed a change to openstack-infra/jenkins-job-builder: Add support for Slave Utilization plugin https://review.openstack.org/82693 | 00:07 |
| marun | fungi, clarkb: ^^ I keep getting little things wrong :( | 00:07 |
| marun | how do you normally debug these jobs? do you run things locally? | 00:07 |
| marun | iterating through gerrit is pretty painful for someone ignorant like me. | 00:08 |
| *** mrodden has joined #openstack-infra | 00:08 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Added pagination to comments API https://review.openstack.org/83568 | 00:10 |
| jeblair | marun: yeah, easiest way is to do it on your own cloud server. the devstack-gate repo has some instructions on how to do that in the README | 00:10 |
| *** dstanek has joined #openstack-infra | 00:10 | |
| marun | jeblair: danke | 00:10 |
| *** lcheng has quit IRC | 00:11 | |
| *** lcheng has joined #openstack-infra | 00:12 | |
| jogo | do WIP patches get expired? | 00:13 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added pagination directives https://review.openstack.org/83575 | 00:13 |
| clarkb | jogo in the past they did but I think fungi fixed that behavior | 00:14 |
| jogo | ah sweet | 00:14 |
| clarkb | I have a handful of old WIP changes that are still alive | 00:14 |
| sdague | would there be a way to configure that per project? | 00:15 |
| sdague | I honestly find the 6 month old WIP patches to be really distracting | 00:15 |
| jogo | clarkb: even if they have a -1? | 00:15 |
| sdague | jogo: -1 stuff expires regardless | 00:15 |
| jeblair | sdague: the idea of wip is you aren't supposed to see it | 00:15 |
| sdague | I see it all the time | 00:16 |
| sdague | https://review.openstack.org/#/q/status:open+-Verified-1+-CodeReview-2+project:openstack-dev/devstack,n,z | 00:16 |
| *** SumitNaiksatam has quit IRC | 00:16 | |
| jeblair | sdague: it doesn't show up in the important changes view; if you're using another view, you might want to filter it out | 00:16 |
| jogo | sdague: no way to hide WIPs | 00:16 |
| jeblair | yes you can hide WIPs! that's the point! | 00:17 |
| sdague | jeblair: important changes is completely useless to me | 00:17 |
| sdague | because it's 600+ reviews | 00:17 |
| *** _nadya_ has joined #openstack-infra | 00:17 | |
| marun | .... | 00:17 |
| jeblair | sdague: great, fine, don't use it, but if you work based on searches, then you should consider filtering things you don't want to see, like wip, from those. | 00:18 |
| sdague | ok, how do I filter wip ? | 00:18 |
| sdague | I guess I didn't realize I code | 00:18 |
| marun | i think it'd be useful to be able to query by things like bug/blueprint severity… | 00:18 |
| *** reed has quit IRC | 00:18 | |
| marun | hah | 00:18 |
| sdague | heh | 00:18 |
| clarkb | sdague: -status:workinprogress iirc | 00:18 |
| sdague | s/code/could/ | 00:18 |
| sdague | clarkb: nice | 00:19 |
| sdague | ok, that's helpful | 00:19 |
| clarkb | marun: there is a thing, reviewday | 00:19 |
| marun | clarkb: ooh | 00:20 |
| jeblair | marun: or consider writing a patch to gerrit that adds a priority field to changes. then we could set it based on launchpad status. | 00:20 |
| marun | clarkb: that dprince, always doing cool things I don't know about | 00:20 |
| marun | jeblair: how friendly are they do patch submissions? | 00:20 |
| marun | do -> to | 00:20 |
| marun | jeblair: I can think of a bunch of changes to make :) | 00:21 |
| marun | (not that I have any love for java, mind) | 00:21 |
| pleia2 | marun: we're friendly | 00:21 |
| *** bknudson has joined #openstack-infra | 00:21 | |
| jeblair | marun: i'll be honest, we've had very mixed results; starting on the ml or in #gerrit would probably be advisable. | 00:21 |
| marun | jeblair: ok | 00:21 |
| jogo | so silly question | 00:21 |
| marun | pleia2: so, on the topic of gerrit... | 00:22 |
| pleia2 | marun: I meant reviewday :) | 00:22 |
| marun | pleia2: ah! | 00:22 |
| jogo | http://status.openstack.org/zuul/ shows my config patch as in the check queue for 22 minutes and none of the jobs are running yet | 00:22 |
| *** nkinder has joined #openstack-infra | 00:22 | |
| marun | pleia2: sorry. got mixed up. | 00:22 |
| *** _nadya_ has quit IRC | 00:22 | |
| jogo | I thought the build time was generally <20 minutes for a node to become ready | 00:22 |
| pleia2 | I think zaro has had most success working with gerrit folks | 00:22 |
| jeblair | jogo: (also your patch to change the calculation hasn't landed yet) | 00:22 |
| marun | pleia2: is there a hosted version of reviewday? | 00:22 |
| pleia2 | marun: http://status.openstack.org/reviews/ | 00:23 |
| marun | pleia2: ah, so that's where that comes from... | 00:23 |
| pleia2 | yep! | 00:23 |
| jeblair | jogo: we're trying to use hpcloud 1.1 more, which has been throwing servers into error state a lot | 00:23 |
| marun | pleia2: I think someone here mentioned that and it fell out of my head. | 00:23 |
| jeblair | jogo: which prompted this change: https://review.openstack.org/#/c/83587/ | 00:24 |
| marun | pleia2: what do you think of allowing sorting by bug/blueprint severity? | 00:24 |
| jogo | jeblair: yeah I need to test that patch first ... | 00:24 |
| marun | pleia2: and how is the score calculated? | 00:24 |
| pleia2 | marun: pretty sure it takes bug severity into account | 00:24 |
| pleia2 | marun: there's not much to it if you want to have a browse: https://git.openstack.org/cgit/openstack-infra/reviewday/tree/ | 00:25 |
| *** ogelbukh has quit IRC | 00:25 | |
| marun | pleia2: ah, and of course I can just submit a patch through gerrit | 00:25 |
| pleia2 | marun: right :) | 00:25 |
| marun | pleia2: yay infra | 00:25 |
| *** ogelbukh has joined #openstack-infra | 00:25 | |
| *** alexandra has joined #openstack-infra | 00:26 | |
| marun | pleia2: awesome, I definitely have some things I'd like to contribute. | 00:26 |
| marun | pleia2: have to run, ttyl! | 00:26 |
| *** hogepodge has joined #openstack-infra | 00:27 | |
| *** lcheng has quit IRC | 00:28 | |
| pleia2 | marun: great! | 00:28 |
| *** lcheng has joined #openstack-infra | 00:29 | |
| jogo | jeblair: yeah I need to test that patch first ... | 00:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Add gate-nova-specs-docs job https://review.openstack.org/83615 | 00:30 |
| *** marun has quit IRC | 00:30 | |
| *** matsuhashi has joined #openstack-infra | 00:31 | |
| jogo | how long until https://review.openstack.org/#/c/83615/ propagates out | 00:33 |
| *** wenlock has quit IRC | 00:34 | |
| jeblair | jogo: should be out by 01:00 | 00:34 |
| jogo | ahh thanks | 00:34 |
| *** lcheng has quit IRC | 00:44 | |
| *** lcheng has joined #openstack-infra | 00:45 | |
| *** thuc has quit IRC | 00:47 | |
| *** thuc has joined #openstack-infra | 00:48 | |
| *** thuc has quit IRC | 00:52 | |
| *** derekh has joined #openstack-infra | 00:57 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/gerritlib: Add empty-commit flag when creating a project with no commits https://review.openstack.org/83116 | 00:57 |
| *** markwash has quit IRC | 00:58 | |
| *** hogepodge has quit IRC | 00:59 | |
| *** lcheng has quit IRC | 01:00 | |
| *** dguerri has quit IRC | 01:02 | |
| *** gokrokve has joined #openstack-infra | 01:03 | |
| *** lcheng has joined #openstack-infra | 01:04 | |
| *** dguerri has joined #openstack-infra | 01:05 | |
| mordred | anteaya: ^^ nice | 01:05 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Stop waiting for resources in ERROR state https://review.openstack.org/83587 | 01:05 |
| mordred | jeblair: I finally found someone at HP Cloud who is interested in looking at the information we produce about how the cloud is doing | 01:06 |
| mordred | jeblair: and when I say interested, I mean got really excited | 01:06 |
| clarkb | mordred: \o/ | 01:08 |
| openstackgerrit | Jay Faulkner proposed a change to openstack/requirements: Add ironic-python-agent to projects.txt https://review.openstack.org/83638 | 01:09 |
| *** UtahDave has quit IRC | 01:10 | |
| sdague | mordred: nice | 01:11 |
| *** SumitNaiksatam has joined #openstack-infra | 01:14 | |
| openstackgerrit | A change was merged to openstack-infra/gerritlib: Add empty-commit flag when creating a project with no commits https://review.openstack.org/83116 | 01:15 |
| *** david-lyle has joined #openstack-infra | 01:16 | |
| anteaya | mordred: thanks | 01:18 |
| anteaya | and yay for someone at hp being really excited about infra info | 01:18 |
| *** lcheng has quit IRC | 01:19 | |
| *** ProfFalken has quit IRC | 01:20 | |
| *** ProfFalken has joined #openstack-infra | 01:21 | |
| *** HenryG has joined #openstack-infra | 01:21 | |
| *** dcramer_ has joined #openstack-infra | 01:24 | |
| *** amcrn has quit IRC | 01:26 | |
| *** Ryan_Lane1 has quit IRC | 01:27 | |
| *** Sukhdev has quit IRC | 01:31 | |
| *** derekh has quit IRC | 01:32 | |
| *** nosnos has joined #openstack-infra | 01:37 | |
| clarkb | hey does anyone know approximately how many people were at the austin design summit? | 01:38 |
| clarkb | annegentle: jogo mordred jeblair vishy ? | 01:38 |
| *** khyati_ has quit IRC | 01:41 | |
| * fungi pictures it occupying half a conference room, because the other half was being used to store boxes and excess folding tables | 01:41 | |
| clarkb | fungi: you are around \o/ do you have a current ATC count for icehouse? | 01:42 |
| * clarkb is clearly writing a conference proposal | 01:42 | |
| *** alexpilotti has joined #openstack-infra | 01:42 | |
| fungi | clarkb: i haven't generated icehouse-cycle-only atc lists (for summit passes we also include anyone who contributed during the havana cycle too) | 01:43 |
| clarkb | right | 01:43 |
| fungi | i can get you some rough numbers, just a sec | 01:43 |
| clarkb | stackalytics indicates integrated is 732 | 01:43 |
| *** yamahata has joined #openstack-infra | 01:43 | |
| *** gokrokve has quit IRC | 01:43 | |
| clarkb | and openstack is 1,163 | 01:43 |
| clarkb | *1162 | 01:43 |
| clarkb | I can use those numbers | 01:44 |
| fungi | wfm | 01:44 |
| fungi | it's somewhere around that, probably | 01:44 |
| *** gokrokve has joined #openstack-infra | 01:44 | |
| fungi | clarkb: what conference are you proposing to? | 01:44 |
| clarkb | osbridge | 01:45 |
| clarkb | fungi: I can paste my proposal in a bit when it looks written | 01:45 |
| *** gokrokve_ has joined #openstack-infra | 01:45 | |
| fungi | never heard of it--clearly they should want you to come talk ;) | 01:45 |
| clarkb | http://opensourcebridge.org/ | 01:46 |
| fungi | i need to propose to a few conferences for later in the year, it's just hard to plan pact the horizon of my house move in a few months | 01:46 |
| fungi | s/pact/past/ | 01:46 |
| fungi | nice... my dyslexia made it "the open source citizens brigade" | 01:47 |
| fungi | looks like a good excuse to visit portland, but that one is actually smack in the middle of when i'm moving. oh well | 01:48 |
| *** gokrokve has quit IRC | 01:48 | |
| clarkb | :/ | 01:49 |
| fungi | yeah, end of june is when i have to be out of the old place | 01:49 |
| fungi | though if you want to come see north carolina in the heat of october, http://www.allthingsopen.org/call-for-speakers.php | 01:52 |
| fungi | i need to submit an abstract for that one before they close the cfp. crap | 01:52 |
| *** alexandra is now known as alex-luncheon | 01:52 | |
| clarkb | fungi: when does their cfp end? | 01:53 |
| fungi | the e-mail i got didn't say, neither does their website | 01:54 |
| fungi | last year was the first one they did in nc. a few hiccups but worked out pretty well (they've done a few in atlanta prior) | 01:56 |
| *** thuc has joined #openstack-infra | 01:58 | |
| * mordred enjoyed all things open | 01:58 | |
| fungi | i was just sad that frencil didn't have a big enough room so i couldn't squeeze in to see his demo | 01:59 |
| *** wenlock has joined #openstack-infra | 02:01 | |
| *** thuc has quit IRC | 02:03 | |
| fungi | Ryan_Lane: fun! https://bugzilla.wikimedia.org/show_bug.cgi?id=62497 | 02:04 |
| *** lcheng has joined #openstack-infra | 02:07 | |
| *** bhuvan_ has quit IRC | 02:08 | |
| *** yjiang has joined #openstack-infra | 02:21 | |
| *** lcheng has quit IRC | 02:21 | |
| *** lcheng has joined #openstack-infra | 02:22 | |
| *** saschpe has quit IRC | 02:24 | |
| *** bknudson has quit IRC | 02:24 | |
| *** saschpe has joined #openstack-infra | 02:24 | |
| *** jammyjam has quit IRC | 02:25 | |
| *** alex-luncheon is now known as Alexandra | 02:39 | |
| clarkb | lifeless: is https://bugs.launchpad.net/openstack-ci/+bug/1298731 something that infra needs to fix? | 02:40 |
| uvirtbot | Launchpad bug 1298731 in openstack-ci "ssh command limits for toci virsh don't match ironic commands" [Undecided,New] | 02:40 |
| clarkb | lifeless: want to make sure that doesn't get lost somewhere | 02:40 |
| *** tchaypo has joined #openstack-infra | 02:43 | |
| *** krotscheck has quit IRC | 02:43 | |
| lifeless | clarkb: its in an infra codebase (openstack-infra/tripleo-ci) | 02:44 |
| clarkb | gotcha | 02:44 |
| lifeless | clarkb: but it in terms of resourcing | 02:44 |
| lifeless | clarkb: I suspect NobodyCam will pick it up as he's been working on tripleo integration for a bit | 02:45 |
| clarkb | great thanks. (just making sure you weren't blocking on us for something) | 02:45 |
| lifeless | appreciated | 02:45 |
| *** medieval1_ has joined #openstack-infra | 02:46 | |
| *** Sukhdev has joined #openstack-infra | 02:48 | |
| *** mgagne has quit IRC | 02:49 | |
| *** medieval1 has quit IRC | 02:49 | |
| *** zhiyan_ is now known as zhiyan | 02:50 | |
| *** dims has quit IRC | 02:53 | |
| *** jhesketh_ has quit IRC | 02:56 | |
| *** paul-- has joined #openstack-infra | 03:04 | |
| *** matsuhashi has quit IRC | 03:04 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Display only first 6 chars for git commit id https://review.openstack.org/83556 | 03:05 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add demo support for openstack's live zuul feed https://review.openstack.org/82014 | 03:06 |
| *** packet has joined #openstack-infra | 03:07 | |
| *** matsuhashi has joined #openstack-infra | 03:10 | |
| *** lcheng has quit IRC | 03:10 | |
| *** nosnos has quit IRC | 03:17 | |
| jogo | clarkb: I heard it wasn't very many at austin <50? | 03:19 |
| jogo | clarkb: but i wasn't there | 03:19 |
| *** harlowja is now known as harlowja_away | 03:19 | |
| *** changbl has joined #openstack-infra | 03:20 | |
| *** pcrews has quit IRC | 03:24 | |
| *** wenlock_ has joined #openstack-infra | 03:25 | |
| *** mgagne has joined #openstack-infra | 03:29 | |
| *** matsuhashi has quit IRC | 03:31 | |
| *** gokrokve_ has quit IRC | 03:32 | |
| *** dcramer_ has quit IRC | 03:33 | |
| *** jamespage has quit IRC | 03:35 | |
| *** jamespage has joined #openstack-infra | 03:36 | |
| *** jamespage has joined #openstack-infra | 03:36 | |
| jogo | whee http://logs.openstack.org/76/80876/4/gate/gate-tempest-dsvm-neutron-large-ops/094f5d3/logs/devstacklog.txt.gz#_2014-03-28_03_21_43_732 | 03:36 |
| jogo | silly pypi curl | 03:36 |
| *** dcramer_ has joined #openstack-infra | 03:46 | |
| *** alexpilotti has quit IRC | 03:46 | |
| *** ProfFalken has quit IRC | 03:47 | |
| *** ProfFalken has joined #openstack-infra | 03:48 | |
| *** jdurgin has quit IRC | 03:48 | |
| *** rcleere has joined #openstack-infra | 03:53 | |
| *** matsuhashi has joined #openstack-infra | 03:53 | |
| *** thuc_ has joined #openstack-infra | 03:53 | |
| *** mgagne1 has joined #openstack-infra | 03:56 | |
| *** Ryan_Lane1 has joined #openstack-infra | 03:57 | |
| *** Ryan_Lane has quit IRC | 03:57 | |
| *** Ryan_Lane1 is now known as Ryan_Lane | 03:57 | |
| *** Ryan_Lane has joined #openstack-infra | 03:57 | |
| *** mgagne has quit IRC | 03:59 | |
| *** jdurgin has joined #openstack-infra | 04:01 | |
| *** lcheng has joined #openstack-infra | 04:02 | |
| *** nosnos has joined #openstack-infra | 04:06 | |
| *** mgagne1 has quit IRC | 04:07 | |
| *** mgagne has joined #openstack-infra | 04:08 | |
| *** vkozhukalov has joined #openstack-infra | 04:15 | |
| *** chandankumar_ has joined #openstack-infra | 04:23 | |
| *** chandankumar_ has quit IRC | 04:23 | |
| *** wenlock_ has quit IRC | 04:25 | |
| *** lcheng has quit IRC | 04:26 | |
| *** wchrisj has quit IRC | 04:27 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 04:30 |
| *** wenlock has quit IRC | 04:33 | |
| *** rcleere has quit IRC | 04:34 | |
| *** dstanek has quit IRC | 04:40 | |
| *** yjiang has quit IRC | 04:40 | |
| *** packet has quit IRC | 04:43 | |
| *** Sukhdev has quit IRC | 04:48 | |
| *** Shrews has quit IRC | 04:51 | |
| *** Shrews has joined #openstack-infra | 04:52 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: set default configurations for non-test mode https://review.openstack.org/78458 | 04:52 |
| *** nicedice has quit IRC | 05:01 | |
| *** masayukig has quit IRC | 05:02 | |
| *** jhesketh_ has joined #openstack-infra | 05:02 | |
| *** dstanek has joined #openstack-infra | 05:09 | |
| *** nati_ueno has quit IRC | 05:09 | |
| *** wchrisj has joined #openstack-infra | 05:11 | |
| *** tchaypo has quit IRC | 05:12 | |
| *** alff has joined #openstack-infra | 05:13 | |
| *** alff_ has joined #openstack-infra | 05:13 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add in progress bars for jobs https://review.openstack.org/82015 | 05:16 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Format change headers like infra's status page https://review.openstack.org/82016 | 05:16 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add a total progress bar to each change https://review.openstack.org/82017 | 05:17 |
| *** paul-- has quit IRC | 05:26 | |
| *** marun has joined #openstack-infra | 05:27 | |
| *** dstanek has quit IRC | 05:27 | |
| *** wchrisj has quit IRC | 05:28 | |
| *** gyee has quit IRC | 05:33 | |
| *** alff__ has joined #openstack-infra | 05:34 | |
| *** alff has quit IRC | 05:36 | |
| *** markwash has joined #openstack-infra | 05:39 | |
| *** afazekas has joined #openstack-infra | 05:41 | |
| *** tchaypo has joined #openstack-infra | 05:45 | |
| *** afazekas has quit IRC | 05:46 | |
| *** skraynev_afk is now known as skraynev | 05:48 | |
| *** mrmartin has joined #openstack-infra | 05:48 | |
| *** alff_ has quit IRC | 05:49 | |
| *** alff__ has quit IRC | 05:49 | |
| *** maxbit has quit IRC | 05:51 | |
| *** talluri has joined #openstack-infra | 05:59 | |
| *** ArxCruz has quit IRC | 06:00 | |
| *** afazekas has joined #openstack-infra | 06:01 | |
| *** thuc_ has quit IRC | 06:03 | |
| *** thuc has joined #openstack-infra | 06:03 | |
| *** mrda is now known as mrda_weekend | 06:06 | |
| *** _nadya_ has joined #openstack-infra | 06:07 | |
| *** thuc has quit IRC | 06:08 | |
| *** tchaypo has quit IRC | 06:11 | |
| *** tchaypo has joined #openstack-infra | 06:14 | |
| *** vkozhukalov has quit IRC | 06:15 | |
| *** vkozhukalov1 has joined #openstack-infra | 06:15 | |
| *** mrmartin has quit IRC | 06:16 | |
| *** _nadya_ has quit IRC | 06:17 | |
| *** _nadya_ has joined #openstack-infra | 06:19 | |
| *** gtt116_ has joined #openstack-infra | 06:22 | |
| *** gtt116 has quit IRC | 06:22 | |
| *** ildikov_ has quit IRC | 06:27 | |
| *** basha has joined #openstack-infra | 06:30 | |
| *** Ryan_Lane has quit IRC | 06:35 | |
| *** saju_m has joined #openstack-infra | 06:38 | |
| *** dstanek has joined #openstack-infra | 06:39 | |
| *** denis_makogon has joined #openstack-infra | 06:39 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add support to status page to collapse changesets https://review.openstack.org/82018 | 06:45 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Shorten update spinner fadeout on status page https://review.openstack.org/82019 | 06:46 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add a filter box to status.html https://review.openstack.org/82279 | 06:47 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add expand by default option to status page https://review.openstack.org/82290 | 06:48 |
| *** vkozhukalov1 has quit IRC | 06:49 | |
| nibalizer | does anyone know where the puppetmaster-passenger package is specifically installed in the openstack-infra/config repo? | 06:56 |
| nibalizer | i don't see it? | 06:56 |
| nibalizer | is there extra setup scripts called on puppet masters? | 06:56 |
| *** _nadya_ has quit IRC | 07:03 | |
| *** dstanek has quit IRC | 07:04 | |
| *** Manishanker has joined #openstack-infra | 07:06 | |
| *** mihgen has joined #openstack-infra | 07:09 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 07:12 |
| *** thuc has joined #openstack-infra | 07:14 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 07:15 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 07:17 |
| *** tianst20 has quit IRC | 07:18 | |
| *** thuc has quit IRC | 07:19 | |
| *** matsuhashi has quit IRC | 07:20 | |
| *** matsuhas_ has joined #openstack-infra | 07:24 | |
| *** katyafervent_awa is now known as katyafervent | 07:25 | |
| *** markwash has quit IRC | 07:28 | |
| *** dstanek has joined #openstack-infra | 07:32 | |
| *** jlibosva has joined #openstack-infra | 07:33 | |
| *** mgagne has quit IRC | 07:38 | |
| *** dstanek has quit IRC | 07:48 | |
| *** markwash has joined #openstack-infra | 07:49 | |
| *** ildikov_ has joined #openstack-infra | 07:49 | |
| *** yjiang|2 has joined #openstack-infra | 07:50 | |
| *** mihgen has quit IRC | 07:53 | |
| *** markwash has quit IRC | 07:55 | |
| *** Alexandra has quit IRC | 07:57 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Display only first 6 chars for git commit id https://review.openstack.org/83556 | 08:02 |
| *** flaper87|afk is now known as flaper87 | 08:05 | |
| *** ihrachys is now known as ihrachys|wfh | 08:06 | |
| *** vkozhukalov has joined #openstack-infra | 08:16 | |
| *** talluri has quit IRC | 08:21 | |
| *** hashar has joined #openstack-infra | 08:26 | |
| *** jgallard has joined #openstack-infra | 08:28 | |
| *** jcoufal has joined #openstack-infra | 08:32 | |
| *** mihgen has joined #openstack-infra | 08:33 | |
| *** starmer has joined #openstack-infra | 08:34 | |
| openstackgerrit | Benjamin Staffin proposed a change to openstack-infra/jenkins-job-builder: Add support for Slave Utilization plugin https://review.openstack.org/82693 | 08:36 |
| *** dizquierdo has joined #openstack-infra | 08:45 | |
| *** dstanek has joined #openstack-infra | 08:45 | |
| *** vkozhukalov has left #openstack-infra | 08:49 | |
| *** basha has quit IRC | 08:50 | |
| *** talluri has joined #openstack-infra | 08:52 | |
| *** matsuhas_ has quit IRC | 08:53 | |
| *** hashar has quit IRC | 08:55 | |
| *** hashar has joined #openstack-infra | 08:55 | |
| *** matsuhas_ has joined #openstack-infra | 08:57 | |
| *** ildikov_ has quit IRC | 09:03 | |
| *** yolanda has joined #openstack-infra | 09:05 | |
| *** wenlock has joined #openstack-infra | 09:05 | |
| *** saju_m has quit IRC | 09:05 | |
| *** pblaho has joined #openstack-infra | 09:08 | |
| *** derekh has joined #openstack-infra | 09:09 | |
| *** wenlock has quit IRC | 09:09 | |
| *** yjiang|2 has quit IRC | 09:11 | |
| *** e0ne has joined #openstack-infra | 09:12 | |
| *** yassine has joined #openstack-infra | 09:13 | |
| *** johnthetubaguy has joined #openstack-infra | 09:16 | |
| *** johnthetubaguy has quit IRC | 09:17 | |
| *** e0ne has quit IRC | 09:22 | |
| *** starmer has quit IRC | 09:22 | |
| *** saju_m has joined #openstack-infra | 09:25 | |
| *** jcoufal has quit IRC | 09:25 | |
| *** jcoufal has joined #openstack-infra | 09:26 | |
| *** dizquierdo has quit IRC | 09:27 | |
| *** e0ne has joined #openstack-infra | 09:28 | |
| *** e0ne_ has joined #openstack-infra | 09:28 | |
| *** markmc has joined #openstack-infra | 09:31 | |
| *** e0ne has quit IRC | 09:32 | |
| *** ildikov_ has joined #openstack-infra | 09:33 | |
| *** jcoufal has quit IRC | 09:37 | |
| *** jcoufal has joined #openstack-infra | 09:37 | |
| *** johnthetubaguy has joined #openstack-infra | 09:38 | |
| *** denis_makogon has quit IRC | 09:39 | |
| *** jp_at_hp has joined #openstack-infra | 09:40 | |
| *** adarazs has quit IRC | 09:41 | |
| *** rediskin has joined #openstack-infra | 09:43 | |
| *** julienvey has joined #openstack-infra | 09:56 | |
| *** andreaf has quit IRC | 09:58 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Documented module functions. https://review.openstack.org/82849 | 10:00 |
| *** matsuhas_ has quit IRC | 10:00 | |
| *** dizquierdo has joined #openstack-infra | 10:05 | |
| rediskin | hi all. I have a jenkins job, but seems I have no sudo permissions on vm | 10:12 |
| *** yamahata has quit IRC | 10:12 | |
| rediskin | https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/rally.yaml | 10:12 |
| rediskin | is it depends on node? I using bare-precise and bare-centos6 | 10:13 |
| rediskin | I should use other node? Or there is other way to get sudo? | 10:13 |
| rediskin | here is the log http://logs.openstack.org/42/83142/1/check/check-rally-install-bare-centos6/c178f6a/console.html | 10:13 |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/reviewstats: Documented module functions. https://review.openstack.org/82849 | 10:13 |
| *** nosnos has quit IRC | 10:16 | |
| *** e0ne_ has quit IRC | 10:16 | |
| *** flaper87 is now known as flaper87|afk | 10:18 | |
| *** alexpilotti has joined #openstack-infra | 10:20 | |
| *** e0ne has joined #openstack-infra | 10:23 | |
| *** andreaf has joined #openstack-infra | 10:23 | |
| *** mfer has joined #openstack-infra | 10:24 | |
| rediskin | Apparently *-bare nodes does not provides sudo permissions. But why? | 10:25 |
| rediskin | I can't even do any apt-get/yum on such nodes | 10:25 |
| *** david-lyle has quit IRC | 10:27 | |
| *** david-lyle has joined #openstack-infra | 10:27 | |
| *** ominakov has joined #openstack-infra | 10:30 | |
| *** dpyzhov has left #openstack-infra | 10:31 | |
| *** mburned is now known as mburned_out | 10:32 | |
| rediskin | Also I see some jobs using - revoke-sudo on bare-precise node. But there is no sudo permissions on bare-precise node | 10:33 |
| *** dstanek has quit IRC | 10:33 | |
| sdague | rediskin: I believe that's all working as designed | 10:33 |
| *** ProfFalken has quit IRC | 10:34 | |
| sdague | what is your test attempting to do? | 10:34 |
| *** ProfFalken has joined #openstack-infra | 10:34 | |
| sdague | maybe we can work backwards from there | 10:34 |
| rediskin | my test attempting to install rally. it requires sudo for apt-get/yum install dependencies | 10:34 |
| *** jcoufal has quit IRC | 10:34 | |
| sdague | you may be able to do that on the devstack precise nodes | 10:35 |
| *** jcoufal has joined #openstack-infra | 10:35 | |
| rediskin | sdague: understood. what about centos node? | 10:36 |
| sdague | I think all the centos nodes are multi use, so that's not allowed | 10:36 |
| rediskin | sdague: may I propose change to enable sudo on bare-* nodes? or it will got -2? | 10:36 |
| sdague | the unit test nodes, which centos is one, is not designed for that. | 10:36 |
| sdague | it will get a -2 | 10:37 |
| rediskin | so how do I get node for my test? | 10:37 |
| sdague | you'll have to wait for one of the infra cores to get in channel. SergeyLukjanov and fungi are usually first up | 10:38 |
| SergeyLukjanov | I'm here, reading scrollback right now | 10:38 |
| *** ociuhandu has quit IRC | 10:39 | |
| SergeyLukjanov | sdague, morning | 10:41 |
| sdague | morning SergeyLukjanov | 10:41 |
| *** flaper87|afk is now known as flaper87 | 10:42 | |
| * SergeyLukjanov trying to remember details re sudo | 10:45 | |
| *** CaptTofu has quit IRC | 10:52 | |
| *** CaptTofu has joined #openstack-infra | 10:52 | |
| *** CaptTofu has quit IRC | 10:56 | |
| *** amotoki has joined #openstack-infra | 10:57 | |
| *** dstanek has joined #openstack-infra | 10:57 | |
| *** mattymo has joined #openstack-infra | 10:59 | |
| *** che-arne has joined #openstack-infra | 10:59 | |
| *** jamielennox is now known as jamielennox|away | 11:15 | |
| *** dstanek has quit IRC | 11:17 | |
| *** morganfainberg is now known as morganfainberg_Z | 11:20 | |
| *** dstanek has joined #openstack-infra | 11:20 | |
| *** julim has joined #openstack-infra | 11:21 | |
| *** julim has quit IRC | 11:22 | |
| *** rossella_ has joined #openstack-infra | 11:31 | |
| *** rossella_ has quit IRC | 11:31 | |
| *** julim has joined #openstack-infra | 11:31 | |
| *** Shrews has quit IRC | 11:33 | |
| *** Shrews has joined #openstack-infra | 11:33 | |
| *** dstanek has quit IRC | 11:46 | |
| *** mestery has joined #openstack-infra | 11:47 | |
| *** saju_m has quit IRC | 11:50 | |
| *** dkranz has quit IRC | 11:54 | |
| *** mburned_out is now known as mburned | 11:57 | |
| *** yolanda has quit IRC | 11:57 | |
| *** yolanda has joined #openstack-infra | 11:59 | |
| *** dguerri is now known as _dguerri | 12:01 | |
| *** weshay has quit IRC | 12:06 | |
| *** weshay has joined #openstack-infra | 12:06 | |
| *** david-lyle has quit IRC | 12:06 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Display only first 6 chars for git commit id https://review.openstack.org/83556 | 12:08 |
| SergeyLukjanov | jhesketh_ ^^ | 12:09 |
| *** weshay has quit IRC | 12:10 | |
| jhesketh_ | SergeyLukjanov: cool | 12:11 |
| jhesketh_ | SergeyLukjanov: I've replied to patchset 4 | 12:12 |
| *** weshay has joined #openstack-infra | 12:12 | |
| jhesketh_ | SergeyLukjanov: did you want to make the attribute tag only applied when actually shortened? | 12:13 |
| SergeyLukjanov | jhesketh_, >10 chars sounds correct | 12:13 |
| SergeyLukjanov | jhesketh_, but for commit id it's correct to cut and not add '...' | 12:14 |
| *** dstanek has joined #openstack-infra | 12:14 | |
| *** mattymo has quit IRC | 12:15 | |
| jhesketh_ | SergeyLukjanov: yep, makes sense | 12:15 |
| SergeyLukjanov | jhesketh_, so, I prefer to cut only commit id | 12:15 |
| SergeyLukjanov | jhesketh_, and display change id as is | 12:15 |
| *** pdmars has joined #openstack-infra | 12:15 | |
| jhesketh_ | SergeyLukjanov: Why not cut commit-id's at 6 chars and anything else >10 chars with '...'? | 12:15 |
| *** pdmars has quit IRC | 12:16 | |
| *** pdmars_ has joined #openstack-infra | 12:16 | |
| SergeyLukjanov | jhesketh_, I don't think that we could have in change.id something else than change id and commit id | 12:17 |
| SergeyLukjanov | jhesketh_, change id with length 10 is too big for us :) | 12:17 |
| *** pdmars_ has quit IRC | 12:17 | |
| SergeyLukjanov | jhesketh_, so, probably it's a bit useless to add conditions to cover such cases | 12:17 |
| *** pdmars has joined #openstack-infra | 12:17 | |
| SergeyLukjanov | anyway, I can do it if you think that it's really needed | 12:18 |
| jhesketh_ | hmm, interesting point | 12:18 |
| jhesketh_ | I'm happy not to block on that :-) | 12:18 |
| SergeyLukjanov | gtg - lunch | 12:18 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix tempest installation for neutron api job https://review.openstack.org/83626 | 12:19 |
| jhesketh_ | sure, I'm nearly off to bed | 12:19 |
| jhesketh_ | SergeyLukjanov: it'd be good to see the attr only applied to changes that need it though as it reducs clutter | 12:19 |
| jhesketh_ | *reduces | 12:19 |
| *** mattymo has joined #openstack-infra | 12:20 | |
| sdague | SergeyLukjanov: instead of truncating in code, what about defining a bounding box on the css so it will just cut it off if it takes up too much space? | 12:20 |
| jhesketh_ | sdague: using the attr tag allows the user to expand to get the full id | 12:20 |
| sdague | ah, gotcha | 12:20 |
| SergeyLukjanov | sdague, we have a nice on hover fot abbr | 12:20 |
| *** pdmars has quit IRC | 12:22 | |
| *** mestery has quit IRC | 12:22 | |
| *** pdmars has joined #openstack-infra | 12:22 | |
| sdague | first time looking at the app.js extract. Any idea if there is some thinking to do more of these display bits with templates? | 12:22 |
| jhesketh_ | sdague: how so? | 12:23 |
| *** dcramer_ has quit IRC | 12:24 | |
| sdague | jhesketh_: there was a thread a few months ago about http://handlebarsjs.com/ which would let you pull all the html building out of js and be a base template | 12:25 |
| sdague | that just has variables | 12:25 |
| jhesketh_ | right | 12:26 |
| jhesketh_ | well the scope I'm working in is to modularise the existing javascript so I can pull certain things out easily (for example just a change block) | 12:26 |
| sdague | sure | 12:26 |
| jhesketh_ | so it's not something I'm thinking of doing, but others may be | 12:26 |
| sdague | yeh, I just found adding features was getting increasingly cumbersome with the constant need to do .append | 12:27 |
| *** maxbit has joined #openstack-infra | 12:27 | |
| sdague | anyway, just curious | 12:28 |
| jhesketh_ | yeah, it's not ideal | 12:29 |
| *** adalbas has joined #openstack-infra | 12:29 | |
| jhesketh_ | it's getting broken into separate format functions at the very least | 12:29 |
| *** aysyd has joined #openstack-infra | 12:29 | |
| sdague | sure, I suppose that will make it easier to turn into templates later | 12:30 |
| jhesketh_ | indeed... I also wonder if there would be much difference between in defining the elements in a template vs handling the status.json yourself | 12:31 |
| sdague | jhesketh_: meaning? | 12:31 |
| jhesketh_ | well if everything were loops and variables in a template I wonder if you wanted to do something different to the template if you should just parse the json yourselve | 12:32 |
| jhesketh_ | *yourself | 12:32 |
| jhesketh_ | ie, there wouldn't be much logic in the javascript left to reuse | 12:32 |
| jhesketh_ | I'm not sure how true that is, just thinking aloud | 12:32 |
| sdague | well I see the current application as old school php | 12:32 |
| sdague | all the logic and presentation in one blob | 12:33 |
| sdague | maybe other folks are differently, but I think about layout best when I'm actually looking at the declarative markup | 12:33 |
| sdague | and then logic for variable definition in code | 12:33 |
| jhesketh_ | yeah, I agree | 12:34 |
| *** mwagner_bbl has quit IRC | 12:35 | |
| jhesketh_ | I think more of it along the lines of building objects (ie the dom) | 12:35 |
| jhesketh_ | and I guess if you took those out of javascript you'd only be left with the few UI elements to handle in JS (eg the filter box/expand+collapse functions) | 12:35 |
| sdague | right, then the javascript would just be the controller, basically | 12:35 |
| jhesketh_ | yep | 12:35 |
| sdague | and the view would be in the templates | 12:35 |
| jhesketh_ | but the point I was leading to is that if you then wanted to just display a changeset on a separate page for some reason, you may as well parse the json yourself and load it into your html/template (assuming you needed something different to the original template) | 12:36 |
| jhesketh_ | not that I think it's a bad idea, I just wonder on the usefulness | 12:36 |
| *** dstanek has quit IRC | 12:36 | |
| sdague | jhesketh_: yeh, this wasn't really about the changeset issue | 12:36 |
| sdague | this was more general on the whole app | 12:37 |
| jhesketh_ | yep | 12:37 |
| sdague | I agree, if you just want snippets, grab the json | 12:37 |
| jhesketh_ | sure | 12:39 |
| jhesketh_ | and to clarify, I would be keen to see a templated version | 12:39 |
| sdague | http://paste.openstack.org/show/74540/ - this is a good instance, where there is really only 1 variable there | 12:39 |
| sdague | jhesketh_: cool, maybe when I need a break from python I'll see about proposing something :) | 12:40 |
| *** jp_at_hp has quit IRC | 12:40 | |
| jhesketh_ | :-) | 12:41 |
| *** dkranz has joined #openstack-infra | 12:41 | |
| *** basha has joined #openstack-infra | 12:42 | |
| *** dstanek has joined #openstack-infra | 12:45 | |
| *** rfolco has joined #openstack-infra | 12:45 | |
| *** hashar has quit IRC | 12:46 | |
| *** chandan_kumar has quit IRC | 12:51 | |
| Shrews | fungi: have there been any reports of unit tests failing because of something eating all memory, engaging the OOM killer? I've been seeing this randomly in ironic tests, and Jenkins has now seen it: http://logs.openstack.org/05/83105/3/check/gate-ironic-python26/dab25e8/console.html | 12:51 |
| fungi | SergeyLukjanov: jhesketh_: a minor point, but if you abbreviate git commit shas, best to do 7 digits since that's the same length most of the git command-line tools also abbreviate to | 12:51 |
| *** basha has quit IRC | 12:51 | |
| fungi | Shrews: looking | 12:51 |
| *** ociuhandu has joined #openstack-infra | 12:53 | |
| sdague | sweet... pypi is pooping on certs - http://logs.openstack.org/27/78527/8/gate/gate-tempest-dsvm-neutron/e22128f/logs/devstacklog.txt.gz | 12:53 |
| Shrews | fungi: i didn't start seeing this until i rebuilt my tox venv. makes me think the latest version of something is wonky | 12:53 |
| *** rlandy has joined #openstack-infra | 12:54 | |
| sdague | Shrews: I know neutron was hitting those issues, but that was just because of how their tests were written | 12:54 |
| fungi | Shrews: definitely sounds like something is leaky... not sure whether it's just one of ironic's requirements, but seems likely or i think this would be more widespread | 12:54 |
| anteaya | Shrews: what version of tox are you using? | 12:54 |
| Shrews | 1.6.1 | 12:55 |
| fungi | Shrews: could definitely be explained by a new release of one of the reqs though | 12:55 |
| sdague | Shrews: do your unit tests normally take 40 minutes? | 12:56 |
| Shrews | sdague: no | 12:56 |
| sdague | http://logs.openstack.org/05/83105/3/check/gate-ironic-python26/dab25e8/console.html#_2014-03-28_00_52_30_332 | 12:56 |
| sdague | notice the run time from testr start to when it blows up | 12:56 |
| fungi | yeah, maybe something went out of control iterating/looping | 12:56 |
| sdague | 2014-03-28 00:52:30.332 | running testr | 12:56 |
| sdague | 2014-03-28 01:31:14.428 | running=OS_STDOUT_CAPTURE=1 OS_STDERR_CAPTURE=1 OS_TEST_TIMEOUT=60 ${PYTHON:-python} -m subunit.run discover -t ./ ./ --list | 12:56 |
| Shrews | fungi: if it isn't widespread, it certainly points to something ironic specific | 12:56 |
| Shrews | but it's so RANDOM | 12:57 |
| sdague | Shrews: could be some global sharing | 12:57 |
| fungi | if random.random()>0.99: eat_all_ram() | 12:58 |
| sdague | so when tests are running in different orders 2 conflict | 12:58 |
| sdague | Shrews: welcome to the wild world of hunting down race conditions :) | 12:58 |
| Shrews | sdague: i've visited that world before. swore i'd never go back :( | 12:58 |
| * Shrews blames fungi for all the things not working | 12:58 | |
| * fungi feels so special! | 12:59 | |
| *** thuc has joined #openstack-infra | 13:00 | |
| *** thuc_ has joined #openstack-infra | 13:00 | |
| *** ameade has joined #openstack-infra | 13:01 | |
| *** jeckersb_gone is now known as jeckersb | 13:01 | |
| *** alff has joined #openstack-infra | 13:01 | |
| *** alff_ has joined #openstack-infra | 13:01 | |
| *** dkranz has quit IRC | 13:02 | |
| anteaya | Shrews: for tracking down races in unit tests: testr run --parallel --concurrency N --until-failure | 13:03 |
| anteaya | for full explanation: http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2014-02-20.log | 13:03 |
| anteaya | timestamp 2014-02-20T19:14:25 | 13:03 |
| *** thuc has quit IRC | 13:04 | |
| *** dprince has joined #openstack-infra | 13:05 | |
| Shrews | anteaya: thanks | 13:06 |
| *** alff has quit IRC | 13:06 | |
| *** talluri has quit IRC | 13:07 | |
| *** alff_ has quit IRC | 13:07 | |
| *** jhesketh__ has quit IRC | 13:08 | |
| ryanpetrello | I’ve noticed a py33 test failure for pecan | 13:10 |
| ryanpetrello | http://logs.openstack.org/37/80637/7/check/gate-pecan-python33/578f35d/console.html | 13:10 |
| ryanpetrello | this familiar to anybody? | 13:10 |
| ryanpetrello | "/usr/local/jenkins/slave_scripts/run-unittests.sh: line 39: .tox/py33/bin/pip: No such file or directory" | 13:10 |
| anteaya | np | 13:11 |
| fungi | /usr/local/jenkins/slave_scripts/run-unittests.sh: line 34: tox: command not found | 13:11 |
| fungi | somehow tox ended up not being preinstalled on that slave, or else something caused the jenkins user not to find it | 13:12 |
| fungi | i'll check another from the same image | 13:12 |
| fungi | make sure we're not dealing with a botched image build | 13:13 |
| *** zigo has quit IRC | 13:13 | |
| *** lcostantino has joined #openstack-infra | 13:14 | |
| *** zigo has joined #openstack-infra | 13:14 | |
| *** basha has joined #openstack-infra | 13:14 | |
| fungi | that ran in hpcloud region b, where we don't have a ton of slaves building compared to other clouds, so it could go unnoticed there for a while if so | 13:14 |
| *** CaptTofu has joined #openstack-infra | 13:15 | |
| *** eharney has quit IRC | 13:15 | |
| *** saju_m has joined #openstack-infra | 13:15 | |
| *** basha has quit IRC | 13:15 | |
| fungi | yep, found another from the same image in the same state, so i'll see what the image build log had to say about that one | 13:16 |
| *** mriedem has joined #openstack-infra | 13:17 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack-infra/config: Upload oslo.messaging final releases to pypi https://review.openstack.org/83740 | 13:19 |
| *** markmcclain has joined #openstack-infra | 13:19 | |
| fungi | 2014-03-28 02:34:29,629 INFO nodepool.image.build.hpcloud-region-b.py3k-precise: ssl.CertificateError: hostname 'pypi.python.org' doesn't match either of '*.c.ssl.fastly.net', 'c.ssl.fastly.net', '*.target.com' | 13:20 |
| fungi | dstufft: ^ any guess what that's about? | 13:20 |
| *** dkranz has joined #openstack-infra | 13:20 | |
| sdague | yeh, that's hitting a bunch of tests in the gate | 13:20 |
| sdague | also... *.target.com does not inspire confidence | 13:20 |
| dstufft | yea I do | 13:20 |
| dstufft | We switched to a different Fastly certificate last night, apparently one or more nodes didn't get the new cert rollout | 13:21 |
| fungi | that's *so* neat! | 13:21 |
| dstufft | let me check what the TTL is on the DNS entries | 13:21 |
| anteaya | fungi: regarding new projects, I see 3 at https://review.openstack.org/#/q/status:open+project:openstack-infra/config+branch:master+topic:new-project,n,z | 13:22 |
| fungi | random pypi https fails for breakfast | 13:22 |
| anteaya | fungi: all look good to go to my eyes, but the neutron-specs one won't go I'm sure due to the specs discussion | 13:23 |
| *** dizquierdo has quit IRC | 13:23 | |
| fungi | anyway, i'll delete that image, but also this falls into the class of "puppet failed to do something during the nodepool image build, so why didn't it error out and abort the script?" | 13:23 |
| *** ArxCruz has joined #openstack-infra | 13:23 | |
| fungi | pabelanger: ^ i know you were looking into ways we might mitigate that | 13:23 |
| sdague | 35 test jobs failed so far because of it | 13:23 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcIlNTTCBwZWVyIGNlcnRpZmljYXRlIG9yIFNTSCByZW1vdGUga2V5IHdhcyBub3QgT0tcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTcyODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NjAxMjk5MDYxNX0= | 13:23 |
| *** markmcclain has quit IRC | 13:24 | |
| dstufft | sigh | 13:24 |
| dstufft | the TTL is 1 day | 13:24 |
| sdague | fungi: we got an appropriate bug on this yet, or should I file one? | 13:25 |
| fungi | okay, deleted the broken nodepool image and all ready nodes built from it | 13:25 |
| fungi | sdague: i haven't started sniffing the morning's bug pile yet, so not sure | 13:25 |
| fungi | checking now | 13:26 |
| *** thomasem has joined #openstack-infra | 13:27 | |
| fungi | sdague: bug 1298929 | 13:27 |
| uvirtbot | Launchpad bug 1298929 in openstack-ci "failed download pip from pypi" [Undecided,New] https://launchpad.net/bugs/1298929 | 13:27 |
| fungi | also, bug 1298827 is related but i can mark it resolved now | 13:27 |
| uvirtbot | Launchpad bug 1298827 in openstack-ci "tox: command not found on unit-test slave" [Undecided,New] https://launchpad.net/bugs/1298827 | 13:27 |
| sdague | fungi: great | 13:28 |
| fungi | anteaya: thanks for going through those. i'll hopefully have a look at them in a little while | 13:29 |
| fungi | i have the first pass of the junk removal contractor coming momentarily to haul away the stuff we don't want to move but can't effectively sell or give away | 13:30 |
| fungi | so my time is going to be a little divided for the next few hours | 13:30 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: added bug for new pypi cert issue https://review.openstack.org/83743 | 13:30 |
| dstufft | http://status.python.org/incidents/m13xm4994zpn | 13:32 |
| *** johnthetubaguy has quit IRC | 13:32 | |
| anteaya | k | 13:32 |
| dstufft | sdague: fungi sorry y'all :( | 13:32 |
| anteaya | fungi: so you have removed the latest images so we are going back to yesterday's images for nodes? | 13:33 |
| sdague | dstufft: no worries. openstack ci is just nagios for the internet | 13:33 |
| anteaya | fungi: when should those be available? | 13:33 |
| *** basha has joined #openstack-infra | 13:33 | |
| *** basha has quit IRC | 13:34 | |
| fungi | anteaya: so far it was just one image for one node type in one provider which seemed affected, and we keep the previous day's image on hand, so it'salready switched to that one now | 13:34 |
| anteaya | okay thanks | 13:34 |
| *** atiwari has joined #openstack-infra | 13:35 | |
| dstufft | fungi: sdague Is there a particular DC that the errors are happening in by any chance? | 13:35 |
| dstufft | or a provider | 13:35 |
| anteaya | py3k-precise-hp-region-b | 13:35 |
| dstufft | anteaya: is that to me? :) | 13:36 |
| fungi | not sure about the test failures for pypi downloads, but the image which failed to build because of a pypi download issue was in hpcloud region b | 13:36 |
| dstufft | fungi: do you know where that is located geographically? | 13:36 |
| anteaya | dstufft: no, that is the node type and provider | 13:37 |
| anteaya | in case anyone else is reading the logs | 13:37 |
| fungi | dstufft: east coast of the usa i think, but not sure where (hp people in here may have more of an idea) | 13:37 |
| fungi | anteaya: well, the pypi download errors could also be happening elsewhere. i was specifically addressing the location of the broken nodepool image which resulted from the same underlying issue, but was causing a different set of job failures | 13:38 |
| isviridov | fungi anteaya I'm about meetbot in #magentodb channel any ideas when it can be merged? https://review.openstack.org/#/c/81251/ | 13:38 |
| sdague | dstufft: it's pretty spread out - http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU1NMIHBlZXIgY2VydGlmaWNhdGUgb3IgU1NIIHJlbW90ZSBrZXkgd2FzIG5vdCBPS1wiIEFORCB0YWdzOmNvbnNvbGUiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTYwMTMzNjU1NTYsIm1vZGUiOiJzY29yZSIsImFuYWx5emVfZmllbGQiOiJidWlsZF9ub2RlIn0= | 13:39 |
| sdague | looks like more hp than rax | 13:39 |
| sdague | but there is one rax hit | 13:39 |
| sdague | region b more affected than others, but it's not limitted to that | 13:40 |
| anteaya | fungi: yes, I was just putting "one image for one node type in one provider" into my own words | 13:40 |
| *** thuc has joined #openstack-infra | 13:41 | |
| *** basha has joined #openstack-infra | 13:41 | |
| *** krotscheck has joined #openstack-infra | 13:41 | |
| *** alexpilotti has quit IRC | 13:42 | |
| *** jgrimm has joined #openstack-infra | 13:42 | |
| *** prad has joined #openstack-infra | 13:42 | |
| *** krotscheck has left #openstack-infra | 13:43 | |
| anteaya | mordred: where did the patch go, I was sure you had written it, to get long urls to be shortened in channel? | 13:43 |
| anteaya | mordred: looking at your patches, I can't seem to find it | 13:43 |
| *** thuc_ has quit IRC | 13:44 | |
| sdague | anteaya: I thought it was a proposed idea from a bot in someone else's channel | 13:44 |
| anteaya | isviridov: hi I have no ability to merge that patch but I have upvoted | 13:44 |
| anteaya | sdague: hmmm, so not an infra bot? | 13:44 |
| rediskin | SergeyLukjanov: so what do you think about enabling sudo in bare-* images? | 13:44 |
| *** thuc has quit IRC | 13:45 | |
| isviridov | anteaya, thank you. But who has? | 13:45 |
| *** bknudson has joined #openstack-infra | 13:45 | |
| dhellmann | I'd like to make the cross-testing jobs for oslo.test voting. When you have time, can I get reviews for https://review.openstack.org/#/c/82580/1, please? | 13:45 |
| sdague | the race is on... will https://review.openstack.org/#/c/83743/ land before the tempest change that failed on that race get's processed? | 13:45 |
| rediskin | SergeyLukjanov: may I propose this change? or it doomed to -2? | 13:45 |
| anteaya | isviridov: SergeyLukjanov fungi clarkb mordred jeblair | 13:46 |
| *** YorikSar has joined #openstack-infra | 13:46 | |
| *** dcramer_ has joined #openstack-infra | 13:46 | |
| anteaya | and fungi is dealing with a contracter at his house this morning | 13:46 |
| soren | Does jjb offer any way to do conditionals? I have a job-template that does a bunch of stuff. A few of the projects that use it also have another job. I want to trigger this other job, but only for these few jobs that have it. Does that make sense? | 13:47 |
| *** basha has quit IRC | 13:47 | |
| *** zigo has quit IRC | 13:47 | |
| *** zigo has joined #openstack-infra | 13:47 | |
| dstufft | I don't know how to kibani, how can I turn http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwidGFyZ2V0LmNvbVwiIEFORCB0YWdzOmNvbnNvbGUiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTYwMTQzODA5NDUsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0= into a link that shows the node names? | 13:48 |
| *** basha has joined #openstack-infra | 13:48 | |
| isviridov | SergeyLukjanov, hi could you merge https://review.openstack.org/#/c/81251/ ? Looks like there is only one meeting today | 13:48 |
| soren | If I could specify "Run after" like in the Jenkins UI, I could just make the subsequent job have that attribute, but since that's just a different view of the "builds to trigger" that obviously won't do. | 13:48 |
| anteaya | dhellmann: I upvoted but since I don't fully understand all the ramifications for oslo.test voting, it may not mean much | 13:48 |
| *** basha has quit IRC | 13:49 | |
| *** prad has quit IRC | 13:49 | |
| fungi | rediskin: we normally prevent jobs running on those node types from doing things as root to help ensure that developers can run the same tests on their own machines without compromising/destroying them. there are some patches in progress which could allow specific jobs to have root access on those nodes, however i don't think they've all made it through review yet | 13:49 |
| Mithrandir | soren: you can tell it to try to trigger non-existing jobs, it'll just not do anything. | 13:49 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: added bug for new pypi cert issue https://review.openstack.org/83743 | 13:49 |
| *** markmcclain has joined #openstack-infra | 13:50 | |
| soren | Mithrandir: Heh. | 13:50 |
| soren | Mithrandir: That's true. | 13:50 |
| fungi | rediskin: normally we perform installation tests into python virtualenvs, but if you specifically need to test system-level installation it may make more sense to use devstack-precise nodes and set up a custom devstack-gate gate_hook function which runs the commands you need to test instead | 13:50 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Documented module functions. https://review.openstack.org/82849 | 13:50 |
| soren | Mithrandir: I guess I could just do that. | 13:50 |
| soren | Mithrandir: Hm... Yeah, that should work. Thanks! | 13:50 |
| rediskin | fungi: thanks | 13:51 |
| Mithrandir | soren: it'll complain in the UI, but it's not broken per se. | 13:51 |
| *** johnthetubaguy has joined #openstack-infra | 13:51 | |
| *** zigo has quit IRC | 13:51 | |
| Mithrandir | soren: there are some plugins to do conditional builds and such, not sure if those would be more what you're looking for. | 13:51 |
| *** amotoki has quit IRC | 13:51 | |
| *** markmcclain1 has joined #openstack-infra | 13:51 | |
| *** markmcclain has quit IRC | 13:51 | |
| *** prad has joined #openstack-infra | 13:52 | |
| SergeyLukjanov | isviridov, looks like oslo meeting will be in 5 mins | 13:53 |
| isviridov | SergeyLukjanov, yeap. Probably after meeting | 13:53 |
| dhellmann | SergeyLukjanov: there's no meeting today, we didn't call one -- we just have that slot reserved | 13:53 |
| sdague | dstufft: ah, I was hoping the view would be embedded | 13:53 |
| *** mwagner_bbl has joined #openstack-infra | 13:53 | |
| dstufft | sdague: no it was, I was just poking arond myself | 13:54 |
| dstufft | to look at a different error to the same problem | 13:54 |
| sdague | dstufft: so go to the left | 13:54 |
| dstufft | (the error pip gives, I was hoping to see what version of pip it was) | 13:54 |
| sdague | click on the little arrow next to build_node (which creates a js popup) | 13:54 |
| dhellmann | anteaya: the point of those jobs is to ensure that changes in oslo.test do not land if it will break libraries using oslo.test | 13:54 |
| sdague | then click on Score | 13:54 |
| sdague | that will take you to a page which facets the results by that term | 13:54 |
| *** zigo has joined #openstack-infra | 13:55 | |
| dstufft | sdague: cool thanks | 13:55 |
| soren | Mithrandir: I considered that. I could conditionally trigger another build based on "{somevar_that_is_only_set_for_these_few_jobs} != ''" or whatnot, but that seems horrible. | 13:55 |
| sdague | I suppose we probably want to add the target bit to that last bug as well | 13:56 |
| SergeyLukjanov | dhellmann, ok, thx for info | 13:56 |
| SergeyLukjanov | isviridov, approved | 13:56 |
| *** markmcclain1 has quit IRC | 13:56 | |
| *** e0ne has quit IRC | 13:58 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add a second kind of message that can trigger pypi fail https://review.openstack.org/83751 | 13:58 |
| *** jp_at_hp has joined #openstack-infra | 13:58 | |
| anteaya | dhellmann: makes sense to have them voting then | 13:58 |
| openstackgerrit | Mark McLoughlin proposed a change to openstack-infra/config: Upload oslo.messaging final releases to pypi https://review.openstack.org/83740 | 13:58 |
| *** zigo has quit IRC | 13:59 | |
| *** zigo has joined #openstack-infra | 13:59 | |
| *** CaptTofu has quit IRC | 13:59 | |
| *** CaptTofu has joined #openstack-infra | 14:00 | |
| *** CaptTofu has quit IRC | 14:00 | |
| *** wenlock has joined #openstack-infra | 14:00 | |
| *** krotscheck has joined #openstack-infra | 14:01 | |
| *** jlibosva has quit IRC | 14:03 | |
| *** paul-- has joined #openstack-infra | 14:03 | |
| *** johnthetubaguy has quit IRC | 14:03 | |
| *** johnthetubaguy has joined #openstack-infra | 14:04 | |
| *** wenlock has quit IRC | 14:05 | |
| *** eharney has joined #openstack-infra | 14:05 | |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Run rally-install jobs on devstack-precise nodes https://review.openstack.org/83754 | 14:06 |
| *** wchrisj has joined #openstack-infra | 14:06 | |
| *** signed8bit has joined #openstack-infra | 14:07 | |
| dstufft | sdague: fungi is it possible to get one of the systems that the pypi hit failed on and see what ip address resolves for pypi.python.org there? | 14:08 |
| *** russellb is now known as rustlebee | 14:08 | |
| *** dansmith is now known as damnsmith | 14:09 | |
| *** dkranz has quit IRC | 14:09 | |
| *** signed8bit has quit IRC | 14:09 | |
| *** signed8bit has joined #openstack-infra | 14:10 | |
| *** johnthetubaguy has quit IRC | 14:10 | |
| sdague | dstufft: most of these systems are recycled right away | 14:10 |
| *** vhoward- has joined #openstack-infra | 14:10 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add OpenStack Meetbot to #magnetodb channel https://review.openstack.org/81251 | 14:11 |
| *** johnthetubaguy has joined #openstack-infra | 14:11 | |
| *** johnthetubaguy has quit IRC | 14:11 | |
| dstufft | yea I figured :/ Fastly ran a check on all their pops and it's returning the correct cert on all of them, so trying to get an IP that it's happening on, and I (of course) can't repro locally | 14:11 |
| *** dkliban_afk has joined #openstack-infra | 14:12 | |
| *** johnthetubaguy has joined #openstack-infra | 14:12 | |
| sdague | hilarious | 14:13 |
| sdague | the last e-r change to detect this | 14:13 |
| sdague | failed on the same issue - http://logs.openstack.org/51/83751/1/check/gate-elastic-recheck-docs/46c7aa5/console.html | 14:13 |
| dstufft | sdague: how hard would it be to run a patched copy of pip? would that be super hard? | 14:14 |
| sdague | where would it come from? | 14:15 |
| sdague | you could always propose a devstack change with the right tracing and try to rerun it until fail | 14:15 |
| dstufft | ah, can you point me to that repo? :) | 14:16 |
| sdague | https://github.com/openstack-dev/devstack | 14:17 |
| *** openstack has joined #openstack-infra | 14:21 | |
| *** dkranz has joined #openstack-infra | 14:22 | |
| *** markmcclain has joined #openstack-infra | 14:23 | |
| *** bnemec is now known as beekneemech | 14:25 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Parameterize library cross-test job https://review.openstack.org/83759 | 14:25 |
| dstufft | weee https://review.openstack.org/#/c/83760/ | 14:25 |
| *** alexpilotti has joined #openstack-infra | 14:27 | |
| BobBall | dstufft: I'm constantly hitting the pypi issue http://70dcd673047f0ebed6a7-dda4e480e005aaa13ec303551d2d8155.r49.cf5.rackcdn.com/06/83606/3/devstacklog.txt.gz | 14:28 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add a second kind of message that can trigger pypi fail https://review.openstack.org/83751 | 14:29 |
| BobBall | If you want a host, I can easily get one :) | 14:29 |
| dstufft | BobBall: if you can get me a host that would be awesome | 14:29 |
| fungi | dstufft: sdague: we do have the ip addresses of machines which saw those failures (they're printed at the start of the job console logs) if it helps at all | 14:29 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Parameterize library cross-test job https://review.openstack.org/83759 | 14:29 |
| *** wenlock has joined #openstack-infra | 14:30 | |
| BobBall | pypi.python.org. 86053 IN CNAME c.global-ssl.fastly.net. | 14:30 |
| BobBall | it's got "global-ssl" in there now | 14:30 |
| BobBall | not "c.ssl" | 14:30 |
| dstufft | BobBall: yea, what's the IP address that resolves too though | 14:30 |
| BobBall | c.global-ssl.fastly.net. 30 IN A 199.27.72.175 | 14:30 |
| BobBall | http://paste.openstack.org/show/74552/ | 14:31 |
| *** david-lyle has joined #openstack-infra | 14:31 | |
| dstufft | BobBall: and that's from a host where it's failing on? | 14:32 |
| openstackgerrit | Thomas Herve proposed a change to openstack-infra/config: Add a new job for heat-templates https://review.openstack.org/83761 | 14:32 |
| BobBall | hmmm - I thought so but I'll double check. | 14:32 |
| therve | fungi, ^^, tried to follow your advices | 14:32 |
| *** pcrews has joined #openstack-infra | 14:32 | |
| *** Manishanker has quit IRC | 14:33 | |
| fungi | BobBall: the cert i'm seeing served from that includes *.python.org | 14:33 |
| BobBall | dstufft: confirmed, yes | 14:33 |
| fungi | openssl s_client -connect 199.27.72.175:443 </dev/null|openssl x509 -text | 14:34 |
| dstufft | yea | 14:34 |
| dstufft | that may be the wrong reason though | 14:34 |
| *** wenlock has quit IRC | 14:34 | |
| BobBall | Want access to the host? | 14:34 |
| dstufft | that was just a gues son my end :) | 14:34 |
| dstufft | BobBall: that would be great | 14:34 |
| BobBall | Chuck me a public key | 14:34 |
| dstufft | https://github.com/dstufft.keys | 14:34 |
| BobBall | done - jenkins@162.242.246.92 | 14:35 |
| NobodyCam | good morning Infra, I have a question about https://bugs.launchpad.net/openstack-ci/+bug/1298731 | 14:35 |
| uvirtbot | Launchpad bug 1298731 in openstack-ci "ssh command limits for toci virsh don't match ironic commands" [Undecided,New] | 14:35 |
| jeblair | fungi, sdague: good morning | 14:36 |
| fungi | NobodyCam: lifeless warned us you'd be coming ;) | 14:36 |
| fungi | morning jeblair | 14:36 |
| NobodyCam | lol | 14:36 |
| dstufft | BobBall: you were able to reproduce with pip? or a different command? | 14:36 |
| NobodyCam | morning fungi | 14:36 |
| NobodyCam | fungi: so lifeless said "you need to setup a restricted ssh key with the ci_commands file, and then get ironic running against it" | 14:36 |
| sdague | morning jeblair | 14:37 |
| BobBall | dstufft: pip; see logs at workspace/testing/logs/devstacklog.txt.gz | 14:37 |
| NobodyCam | arethere any doc or examples I can read / look over? | 14:37 |
| dstufft | hmm | 14:38 |
| dstufft | rm requests-2.2.1.tar.gz && pip install -d . requests is succeeding every time | 14:38 |
| *** rwsu has quit IRC | 14:38 | |
| fungi | NobodyCam: i'm not 100% sure i parse his suggestion/bug report, but i think he may mean per-key command restrictions in the ssh configuration | 14:38 |
| *** e0ne has joined #openstack-infra | 14:38 | |
| fungi | NobodyCam: let me dig up some documentation and see if it joves with what you're trying to make work | 14:38 |
| fungi | jives | 14:39 |
| fungi | by jove | 14:39 |
| jeblair | fungi, sdague: btw, rax has opened a few more tickets saying that jenkins slaves must be compromised because they are doing lookups on 'ahuyehue.info'. the timing of the blacklisting and our logs don't support that for me. | 14:39 |
| *** wenlock has joined #openstack-infra | 14:39 | |
| NobodyCam | :) awesome fungi TY | 14:39 |
| jeblair | fungi, sdague: my current theory is that their blacklisting agent isn't keeping up with ip addresses moving across accounts. i'm asking for logs with timing info from them to confirm. | 14:39 |
| fungi | jeblair: i suspect that to be the case (and have for a while) | 14:40 |
| fungi | jeblair: particularly since it explains why we'd get fresh instances in rax which can't do any lookups through their resolvers | 14:40 |
| BobBall | dstufft: running as jenkins or stack? | 14:40 |
| sdague | jeblair: yeh, that sounds pretty sound | 14:40 |
| dstufft | BobBall: jenkins user | 14:40 |
| dstufft | BobBall: note: I'm not very familar at all with devstack :) | 14:41 |
| *** rwsu has joined #openstack-infra | 14:41 | |
| sdague | the joys of the cloud, keeping up with whack-a-mole things like blacklists | 14:41 |
| BobBall | works as stack too | 14:42 |
| *** primeministerp has quit IRC | 14:42 | |
| jeblair | fungi, sdague: i think that confirms that in this case our work is insufficient to address Bug 1270382 | 14:42 |
| uvirtbot | Launchpad bug 1270382 in openstack-ci "Name resolution failure on gate job" [Medium,Confirmed] https://launchpad.net/bugs/1270382 | 14:42 |
| sdague | jeblair: yes, definitely if they are doing it at the network level | 14:43 |
| jeblair | so we may want to consider either forwarding to 8.8.8.8 on all slaves, or making all slaves recursive resolvers | 14:43 |
| *** e0ne has quit IRC | 14:43 | |
| *** primeministerp has joined #openstack-infra | 14:43 | |
| jeblair | (either one at this point should be a simple config switch) | 14:43 |
| *** aglarendil has joined #openstack-infra | 14:43 | |
| sdague | so are they actually letting dns out of their network? | 14:43 |
| fungi | jeblair: right, probably the only other potentially effective workaround would be to have a nodepool acceptance test which tried to resolve something | 14:43 |
| sdague | that would seem to defeat the purpose of dns blacklisting | 14:43 |
| dstufft | BobBall: hrmpf | 14:43 |
| fungi | sdague: the point of blacklisting is apparently that dns-based reflection attacks are overloading their resolvers | 14:44 |
| sdague | oh | 14:44 |
| sdague | not to actually isolate bad nodes | 14:44 |
| fungi | sdague: and they want to reduce the impact on their resolvers from compromised servers | 14:44 |
| fungi | sdague: right. it's almost certainly to keep their resolvers from crumbling under load induced by their own compromised customers | 14:45 |
| *** salv-orlando_ has joined #openstack-infra | 14:46 | |
| jeblair | fungi: yeah, the 'ahuyehue.info' thing points in that direction too (it's apparently used in amplification attacks) | 14:46 |
| fungi | it was already pretty bad back when i still worked for an isp, and my friends who continue to do so say that it's gotten markedly worse in the past couple years | 14:46 |
| *** salv-orlando has quit IRC | 14:47 | |
| * NobodyCam is happy to not wark at a ISP any more! | 14:49 | |
| fungi | NobodyCam: so here's a not bad blog post (the gritty details are in the openssh manpages if you want them) http://binblog.info/2008/10/20/openssh-going-flexible-with-forced-commands/ | 14:51 |
| BobBall | dstufft: want me to just try to re-stack to see if it fails in the same way? | 14:51 |
| *** thuc_ has joined #openstack-infra | 14:51 | |
| dstufft | BobBall: sure! | 14:51 |
| dstufft | if you can repro with a restack we can add some debugging to devstack | 14:51 |
| *** salv-orlando_ has quit IRC | 14:51 | |
| *** jooools has quit IRC | 14:53 | |
| BobBall | interestingly it seems to always happen at the same place... i.e. other pip installs have worked? | 14:53 |
| BobBall | it did indeed fail in the same place | 14:53 |
| dstufft | BobBall: hrm | 14:53 |
| dstufft | can you point me to where it failed at? | 14:53 |
| dstufft | what line of code that is | 14:53 |
| BobBall | the failure log is at http://70dcd673047f0ebed6a7-dda4e480e005aaa13ec303551d2d8155.r49.cf5.rackcdn.com/06/83606/3/devstacklog.txt.gz | 14:54 |
| *** jooools has joined #openstack-infra | 14:54 | |
| BobBall | the specific line is used loads though | 14:54 |
| dstufft | that is strange o.O | 14:55 |
| fungi | BobBall: when you say fails at the same place, you mean specifically when downloading pytz? | 14:55 |
| BobBall | yes | 14:55 |
| *** zehicle_at_dell has quit IRC | 14:56 | |
| dstufft | https://drone.io/github.com/ionelmc/python-aspectlib/4 is a failure from earlier (the first report) | 14:56 |
| dstufft | hrm | 14:56 |
| dstufft | both using pip 1.4.1 | 14:56 |
| NobodyCam | fungi: Ty reading over now | 14:56 |
| dstufft | wonder if this is a pip bug | 14:56 |
| dstufft | but that doesn't make sense | 14:56 |
| dstufft | because it works sometimes | 14:56 |
| dstufft | BobBall: if you restack a third time it fails again? | 14:57 |
| BobBall | trying now | 14:57 |
| BobBall | it's stuck on reading package list - but might have been my fault :) | 14:57 |
| dstufft | if I recall devstack downloads pip from a particular url | 14:57 |
| dstufft | I might throw together a pip fork that prints the ip address that it was connecting on | 14:58 |
| *** mgagne has joined #openstack-infra | 14:58 | |
| BobBall | third time it worked... | 14:58 |
| dstufft | of course it did | 14:58 |
| dstufft | because problems can't be easy | 14:59 |
| fungi | this is why we can't have nice bugs | 14:59 |
| BobBall | ah - failed now | 14:59 |
| *** salv-orlando has joined #openstack-infra | 14:59 | |
| BobBall | failed on pycrypto | 15:00 |
| BobBall | same error | 15:00 |
| dstufft | oh | 15:00 |
| dstufft | duh | 15:00 |
| dstufft | of coruse I can't repro locally | 15:00 |
| *** hogepodge has joined #openstack-infra | 15:00 | |
| BobBall | I've got a meeting now - sorry guys | 15:00 |
| dstufft | my pip doesn't point to pypi at all | 15:00 |
| BobBall | feel free to play with it and dstufft if you want to add ppl to the authorisedkeys go ahead | 15:00 |
| dstufft | BobBall: cool thanks | 15:00 |
| *** dizquierdo has joined #openstack-infra | 15:01 | |
| fungi | dstufft: maybe an easy way out is to run a tcpdump session on low-verbosity and make note of the last port 443 destination the machine hit before the script bailed? | 15:01 |
| *** mgagne1 has joined #openstack-infra | 15:02 | |
| *** mgagne has quit IRC | 15:02 | |
| fungi | assuming the nondeterminism here follows a particular remote ip address, if those remote addresses are being layer-3/layer-4 load balanced it could be hard to tell from just cursory checks, and could need a lot of volume to correlate | 15:05 |
| * fungi suspects multi-tier load distribution, with a farm of servers behind a non-ssl-terminating proxy at each cdn pop | 15:06 | |
| *** CaptTofu has joined #openstack-infra | 15:07 | |
| *** reed has joined #openstack-infra | 15:08 | |
| ttx | fungi: is there an anonymous way to get a list of gerrit changes with the current version ? Or you have to use SSH or HTML screenscraping ? | 15:09 |
| ttx | the REST API seems to be >=2.6 | 15:09 |
| *** yamahata has joined #openstack-infra | 15:10 | |
| sdague | ttx: not until gerrit upgrade | 15:11 |
| fungi | ttx: those are pretty much your options afaik... the two interfaces are ssh and https, the first is always authenticated and the second won't return raw results for search queries | 15:11 |
| sdague | ttx: what are you trying to generate? | 15:11 |
| ttx | sdague, fungi: ok thx | 15:11 |
| ttx | sdague: writing a tool to automate TC agenda creation / email notification | 15:11 |
| *** rcleere has joined #openstack-infra | 15:11 | |
| *** e0ne has joined #openstack-infra | 15:12 | |
| sdague | yeh, gerritlib is your friend | 15:12 |
| ttx | so I just retrieve a list of open governance changes (title and link) | 15:12 |
| fungi | ttx: if you wanted to tweak openstack-infra/openstackwatch it could probably start publishing what you want in its rss feed | 15:12 |
| ttx | sdague: yeah, will do | 15:12 |
| *** sheeprine has joined #openstack-infra | 15:12 | |
| fungi | ttx: would something like http://rss.cdn.openstack.org/nova.xml but for the governance repo work for your needs? | 15:13 |
| fungi | ttx: or do you need a more fine-tuned query than just "all open changes" | 15:14 |
| ttx | fungi: bit more fine-tuned. Will just query by ssh | 15:14 |
| fungi | k | 15:14 |
| sheeprine | Hi, just a quick question, is review.openstack.org ipv6 ready ? Because there is an AAAA record in the DNS, the host is reponding but the http service is timing out. | 15:14 |
| fungi | sheeprine: i'm connecting to it via ipv6 on the address listed in its aaaa rr | 15:15 |
| fungi | sheeprine: tried https? | 15:15 |
| sheeprine | will give it a try | 15:15 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add tox env for testing jenkins-job-builder https://review.openstack.org/83776 | 15:15 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Parameterize library cross-test job https://review.openstack.org/83759 | 15:15 |
| NobodyCam | hey hey fungi dumb neubee question.. where are the ssh keys comming from I'm look ing over TripleO's repo (https://github.com/openstack-infra/tripleo-ci/tree/master/elements/testenv-worker) and do not see it, thou I have had very little coffee sofar this morning :) | 15:16 |
| sheeprine | fungi: http is redirecting me to https | 15:17 |
| sheeprine | but https hangs | 15:17 |
| fungi | sheeprine: i can telnet to the http port on its ipv6 address and get / returns a redirect to https as it should. establishing an ssl socket to the https port on it gives me page content when i get / | 15:17 |
| *** rlandy has quit IRC | 15:17 | |
| *** packet has joined #openstack-infra | 15:18 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Make the oslo.test cross-testing jobs voting https://review.openstack.org/82580 | 15:18 |
| fungi | NobodyCam: presumably the machines in question are built by nodepool? if so, the best place to start tracking that down would be from the nodepool prep script for it... http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/nodepool/scripts/prepare_tripleo.sh | 15:20 |
| dhellmann | fungi: thanks for the review on ^^ | 15:20 |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Enable voting for pecan pypy tests. https://review.openstack.org/83779 | 15:20 |
| fungi | dhellmann: lgtm now | 15:20 |
| dhellmann | fungi: great! | 15:21 |
| ihrachys|wfh | is https://bugs.launchpad.net/openstack-ci/+bug/1298929 safe to recheck? | 15:23 |
| uvirtbot | Launchpad bug 1298929 in openstack-ci "failed download pip from pypi" [Undecided,New] | 15:23 |
| sdague | ihrachys|wfh: yes | 15:23 |
| ihrachys|wfh | sdague: tnx! | 15:24 |
| *** paul-- has quit IRC | 15:24 | |
| sheeprine | fungi: I'm testing it from two different locations, and everything is hanging at the SSL handshake | 15:24 |
| anteaya | sheeprine: are you behind a firewall or using a proxy? | 15:25 |
| sheeprine | anteaya: nope, no problem accessing other ipv6 servers | 15:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add a test for subnodes https://review.openstack.org/83602 | 15:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add a very basic functional test https://review.openstack.org/82952 | 15:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add per-test database fixture https://review.openstack.org/82865 | 15:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add tests for the allocator https://review.openstack.org/82628 | 15:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive https://review.openstack.org/82147 | 15:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Finish initial docs https://review.openstack.org/82854 | 15:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add ready-script and multi-node support https://review.openstack.org/82252 | 15:25 |
| fungi | sheeprine: i agree that is very, very odd. i'm currently hitting it over a gre tunnel from my house to hurricane electric where i hop on the 6bone... but the server is in rackspace which also peers with hurricane electric so maybe there's an issue at the border | 15:26 |
| *** hashar has joined #openstack-infra | 15:26 | |
| * fungi pokes at some traceroutes | 15:26 | |
| dstufft | found the rogue server | 15:27 |
| fungi | dstufft: hooray! it's awesome that you're better at debugging fastly's services than they are | 15:27 |
| sheeprine | I'm in france btw, and my two machines are behind a proxad peering | 15:27 |
| ryanpetrello | sdague: is this ready to be reverted yet? https://github.com/openstack-infra/config/commit/53659e5f | 15:27 |
| fungi | ...wait, no, not awesome... what's that other word? | 15:27 |
| fungi | terrible. that's what i meant | 15:27 |
| ryanpetrello | sdague: I created https://review.openstack.org/#/c/83779/1 in response to it | 15:27 |
| dstufft | fungi: heh, they missed a machine when checking their consistency | 15:28 |
| dstufft | should be all fixed now | 15:28 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add de-duplication to periodic triggers https://review.openstack.org/83610 | 15:28 |
| fungi | thanks again dstufft, as always! | 15:28 |
| sdague | ryanpetrello: they are still failing at a good clip from what I see | 15:28 |
| sdague | ryanpetrello: http://status.openstack.org/elastic-recheck/ | 15:29 |
| sdague | #3 on the list | 15:29 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Parameterize library cross-test job https://review.openstack.org/83759 | 15:29 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add tox env for testing jenkins-job-builder https://review.openstack.org/83776 | 15:29 |
| dstufft | fungi: unles sof course there are more machines :) | 15:29 |
| dstufft | gotta pick up my daughter though | 15:29 |
| dstufft | be back in a bit | 15:29 |
| ryanpetrello | sdague: so it looks like it’s an issue across the board w/ the pypy workers, not just certain projects? | 15:30 |
| ryanpetrello | sdague: if that’s the case, I’ll get abandon my pecan voting: true and just wait until this bug is fixed | 15:30 |
| ryanpetrello | s/get/just | 15:30 |
| *** mrodden has quit IRC | 15:30 | |
| sdague | ryanpetrello: yes, I think it's across the board | 15:31 |
| ryanpetrello | kthx | 15:31 |
| fungi | ryanpetrello: i think that bug is still outstanding. Alex_Gaynor was talking to the debian/ubuntu/ppa pacakger for pypy about getting it fixed but i haven't seen a new package yet | 15:31 |
| ryanpetrello | okay, got it | 15:32 |
| fungi | looks like maybe the bug didn't get linked in sdague's commit message as related | 15:32 |
| fungi | ryanpetrello: we're tracking it with bug 1290562 (though it's not really an infra bug because it has to be fixed elsewhere) | 15:34 |
| uvirtbot | Launchpad bug 1290562 in openstack-ci "failing pypy jobs with "option --single-version-externally-managed not recognized"" [High,In progress] https://launchpad.net/bugs/1290562 | 15:34 |
| ryanpetrello | fungi: sounds good | 15:34 |
| ryanpetrello | for now, I’ll just abandon my pecan change for voting: true | 15:34 |
| ryanpetrello | now that I understand what’s actually up | 15:34 |
| *** afazekas has quit IRC | 15:34 | |
| ryanpetrello | I’ll rely on this to be fixed and pypy to default to `voting: true` again at some point | 15:34 |
| *** saju_m has quit IRC | 15:35 | |
| *** thuc has joined #openstack-infra | 15:41 | |
| *** packet has quit IRC | 15:41 | |
| *** mrodden has joined #openstack-infra | 15:43 | |
| *** malini has joined #openstack-infra | 15:43 | |
| *** thuc_ has quit IRC | 15:44 | |
| *** thuc has quit IRC | 15:45 | |
| malini | hello! The Marconi team needs to run tests against a newer version of Mongo (not currently provided at the gate) & we are considering 3rd part testing for that. | 15:48 |
| malini | But I am not sure what the best practices are , except for what is here http://ci.openstack.org/third_party.html | 15:49 |
| malini | Is this a good use case for 3rd party testing? | 15:49 |
| *** mgagne1 is now known as mgagne | 15:50 | |
| *** markwash has joined #openstack-infra | 15:50 | |
| *** markmcclain has quit IRC | 15:51 | |
| fungi | bug 1299057 is why i -1 new project changes which import from remote repos with excess topic branches. several of the 7 projects they want cleaned up have 50+ unwanted branches | 15:52 |
| uvirtbot | Launchpad bug 1299057 in openstack-ci "Remove outdated branches in fuel-* repos" [Wishlist,In progress] https://launchpad.net/bugs/1299057 | 15:52 |
| jeblair | fungi: ++ | 15:53 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Raise min_demand due to slow node boot times https://review.openstack.org/81088 | 15:53 |
| *** nati_ueno has joined #openstack-infra | 15:54 | |
| *** reed has quit IRC | 15:54 | |
| jeblair | malini: i don't think so -- third party tests should be used for things that can't be tested in the cloud | 15:55 |
| *** CaptTofu has quit IRC | 15:55 | |
| jeblair | malini: does marconi _need_ a newer version of mongo, or does it also want to support a newer version? | 15:56 |
| malini | jeblair: what would be agood example? load tests? | 15:56 |
| zaro | morning | 15:56 |
| sheeprine | fungi: my bad, it was an MTU problem caused by a tunnel ipv6 is working fine now | 15:56 |
| jeblair | malini: tests on real hardware, mostly | 15:56 |
| fungi | malini: or tests requiring access to specific hardware | 15:56 |
| fungi | sheeprine: great! glad there's nothing for me to fix anyway ;) | 15:56 |
| malini | jeblair: marconi needs a newer mongo | 15:56 |
| jeblair | malini: then that's not really a testing issue, that's a project requirements issue | 15:57 |
| *** jcoufal has quit IRC | 15:57 | |
| jeblair | malini: openstack as a whole tries to support the latest ubuntu and fedora releases, without breaking compatibility with the latest lts and rhel releases | 15:57 |
| fungi | malini: does marconi not intend to target stable server distributions like the other openstack server projects do? | 15:57 |
| zaro | fungi: my gerrit summit slides not on http://docs.openstack.org/infra/publications yet? | 15:57 |
| zaro | fungi: you still working on that issue? | 15:58 |
| fungi | zaro: i need to see if my fix for that job merged yet so i can re-trigger it | 15:58 |
| malini | fungi: 'stable server distributions' as in 'mongo that comes default with the linux distro' ? | 15:58 |
| dstufft | BobBall: btw i'm done with your host, thanks a ton :) | 15:59 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews https://review.openstack.org/79280 | 15:59 |
| jeblair | malini: so generally, we run tests on the latest lts and precise (because those are the only releases that last long enough to be supported through an openstack release's life cycle | 15:59 |
| fungi | malini: right. the mongo which would be under security support from a stable linux distro (rhel, ubuntu lts, et cetera) | 15:59 |
| jeblair | latest lts and rhel i mean | 15:59 |
| malini | jeblair, fungi: & precise has Mongo 2.0.4 which is pretty old :( | 15:59 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port https://review.openstack.org/79281 | 15:59 |
| fungi | malini: are you actively working with ubuntu to correct that deficiency? | 16:00 |
| pleia2 | good morning | 16:00 |
| zaro | pleia2: morning | 16:00 |
| *** matrohon has quit IRC | 16:00 | |
| malini | fungi: no..will tht be the correct path forward? | 16:00 |
| fungi | malini: it feels like something we shouldn't work around by just punting and telling users to go find their own mongo version | 16:00 |
| *** dangers_away is now known as dangers | 16:00 | |
| jeblair | malini: yeah, if ubuntu adds a newer version to either precise or the cloud archive, we can use it; it would be good to do the same with redhat for their equivalent | 16:01 |
| malini | fungi: agreed..ceilometer also has the same issue, & they dont have any tests against mongo at the gate (But domebody needs to cross check me on tht) | 16:01 |
| jeblair | malini: (also, for juno we will be running tests for the master branch on ubuntu trusty, but not right away) | 16:01 |
| Alex_Gaynor | fungi, ryanpetrello: Sadly no, no new package, I hadn't realized they'd been globally voting: false'd, sorry bout that@Q | 16:01 |
| malini | jeblair: what is cloud archive? | 16:02 |
| fungi | malini: it might be moot if marconi isn't graduating from incubation until juno, since that might allow you to only have to support running on ubuntu 14.04 and rhel 7.x | 16:02 |
| ryanpetrello | Alex_Gaynor: no problem, I’m fine now that I know the whole story | 16:02 |
| jeblair | malini: it's an extra archive you can add to ubuntu that has some newer versions of things required by openstack | 16:02 |
| fungi | Alex_Gaynor: yep, still watching the (openstack, pip, debian) bugs for new developments | 16:03 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Adds sanitised sourceforge source repo requirement to docs https://review.openstack.org/83796 | 16:03 |
| sdague | given that marconi doesn't need to run in jobs with libvirt, it's probably a valid approach | 16:03 |
| Alex_Gaynor | fungi: sorry, I'll go ping everyone. starting with dstufft :-) | 16:03 |
| jeblair | malini: zul can help with that, though he might be pretty busy right now | 16:03 |
| dstufft | what'd I do now | 16:03 |
| *** mihgen has quit IRC | 16:03 | |
| *** jooools has quit IRC | 16:04 | |
| zul | jeblair: yes he is pretty busy | 16:04 |
| jeblair | sdague: i think we have uca installed on devstack nodes | 16:04 |
| jeblair | i'm going to restart nodepool nowe to pick up those two build-related changes | 16:05 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Correct test_remote to support detached HEAD state https://review.openstack.org/83798 | 16:05 |
| fungi | jeblair: sdague: i thought dims was still testing that by patching it into devstack-gate | 16:05 |
| jeblair | fungi: er, wow, we don't add cloud archive? | 16:06 |
| jeblair | i thought we did that since forever | 16:06 |
| malini | jeblair, fungi: Trusty looks promising http://packages.ubuntu.com/trusty/i386/database/mongodb-server &Marconi is not graduating this cycle. So we can just use the sqla driver for now | 16:07 |
| fungi | jeblair: every time we tried, we broke nova and tempest because it installs newer libvirt than we can support | 16:07 |
| jeblair | fungi: oh. i guess we tried enough times i thought it stuck. | 16:07 |
| *** jogo is now known as flashgordon | 16:07 | |
| fungi | it definitely git sticky | 16:07 |
| fungi | got | 16:08 |
| jeblair | malini: sounds like a plan; as soon as we get trusty nodes in we can start testing there | 16:08 |
| fungi | but yeah, at this rate we'll be running on trusty before we're able to use uca on precise i expect | 16:08 |
| malini | thanks jeblair & fungi | 16:09 |
| jeblair | flashgordon: just restarted nodepool with your 'moar servers' change and my "don't wait on error" change, hopefully it should start getting a bit more responsive | 16:09 |
| malini | jeblair, fungi: is there any ETA yet on when we'll Trusty at the gate? | 16:11 |
| jeblair | malini: probably a few weeks after it's released, depending on when either our providers get an image in place or whether we can use glance to upload our own | 16:12 |
| flashgordon | jeblair: neato | 16:12 |
| malini | jeblair: thanks! I will check back around then | 16:13 |
| fungi | and the release is about 3 weeks out from now | 16:13 |
| flashgordon | I forgot I actually called that branch name moar ;) | 16:13 |
| sdague | jeblair: no, it's definitely never really worked, which is why we're just using base precise | 16:13 |
| *** ihrachys|wfh is now known as eehruchyshka|fri | 16:13 | |
| *** changbl has quit IRC | 16:14 | |
| flashgordon | jeblair: at some point we need to loop back to HP and sort out why itsi failing | 16:14 |
| sdague | and, honestly, given that fact that we've failed to make it work for this long, I'm cautious on any policy in using UCA going forward, because the bug fix rate doesn't seem to keep up with our needs | 16:14 |
| jeblair | flashgordon: i added bits to the logs and we're collecting data now | 16:15 |
| *** thuc_ has joined #openstack-infra | 16:15 | |
| *** thuc_ has quit IRC | 16:15 | |
| jeblair | flashgordon: we've got hpcloud 1.1 mixed in at about 200 nodes (so something like 20% of nodes should end up there) | 16:16 |
| *** thuc_ has joined #openstack-infra | 16:16 | |
| jeblair | hopefully enough to get data but not enough to slow things down considerably | 16:16 |
| *** mriedem has quit IRC | 16:17 | |
| jeblair | mordred: do you anything about a host that calls itself "storyboard-dev.openstack.org"? because it's emailing me. | 16:18 |
| krotscheck | I remember us tossing that name around a while ago for a dev instance, but I don’t recall it ever being set up. | 16:19 |
| mgagne | Is there an easy way to add yourself to reviewers without having to type your own name each time? Luckily my name isn't Nabuchodonosor but hey =) | 16:19 |
| jeblair | mgagne: you can use your account number | 16:19 |
| jeblair | mgagne: but i don't think there's an "add me" button | 16:20 |
| sdague | mgagne: leave a 0 review comment | 16:20 |
| mgagne | jeblair: cool - I am not a number, I am a free man! | 16:20 |
| flashgordon | jeblair: awesome, I like how we are now using two mostly CD openstack(ish) clouds to run infra on | 16:20 |
| jeblair | mgagne: mordred is the new number two. | 16:21 |
| jeblair | mordred: i have confirmed there is a machine in our account called "storyboard-dev.openstack.org" with an ipv6 address that matches the one emailing me. | 16:23 |
| jeblair | mordred: yet there is no storyboard-dev in dns or in puppet | 16:24 |
| sdague | jeblair: from a devstack perspective, region-b looks like a straight win | 16:24 |
| jeblair | mordred: i suspect that this is an incomplete attempt to set up that server; can it be deleted? | 16:24 |
| clarkb | morning | 16:25 |
| sdague | tough what's up with the difference between the check nodes in rax-iad - http://dague.net/testing/check-tempest-dsvm-full.png | 16:25 |
| jeblair | sdague: cool. we do spin up 30g instances there and then limit the ram to 8g in order to get enough cpu. but i'm glad that after doing that, it works well. :) | 16:25 |
| fungi | sdague: bearing in mind that we're using instances with 4x the ram we need so we can get them to go faster | 16:25 |
| *** markmcclain has joined #openstack-infra | 16:25 | |
| clarkb | sdague: rax-iad are pvhvm images | 16:26 |
| sdague | clarkb: ok, and the rest of rax is still not on those images? | 16:26 |
| clarkb | sdague: correct, we should fix that | 16:26 |
| clarkb | sdague: the variability in base image made us cautious | 16:26 |
| sdague | fair, though no worse than all the rest of rax | 16:26 |
| *** nati_ueno has quit IRC | 16:27 | |
| sdague | hp azs actually degraded a lot in their variability as well | 16:27 |
| jeblair | i think he meant the fact that the contents of the base image were actually different | 16:27 |
| sdague | oh | 16:27 |
| jeblair | eg, missing packages, etc | 16:27 |
| sdague | right | 16:27 |
| sdague | the cffi issue tripped us up on that | 16:27 |
| *** nicedice has joined #openstack-infra | 16:27 | |
| sdague | fwiw hp az[\d] used to have a std deviation of 3 or 4 mins, now it's up to the ~10 that rax nodes tend to have | 16:28 |
| clarkb | sdague: I think we are seeing the effects of a smaller pool of hypervisors | 16:28 |
| sdague | as they migrate off to 1.1? | 16:29 |
| clarkb | jeblair: fungi: should I propose a change to switch to pvhm for rax? | 16:29 |
| JayF | got a bit of an interesting test failure here. http://logs.openstack.org/38/83638/1/gate/gate-requirements-integration-dsvm/44d9954/logs/devstacklog.txt.gz -- looks like we got a strangely bad cert on a devstack run. I think a recheck no bug will fix the error, but it's weird enough that I'm curious how it happened (see the last line of the traceback at the end of the log) | 16:29 |
| clarkb | sdague: yup | 16:29 |
| jeblair | clarkb: any chance it can wait until the label refactor patch lands? :) | 16:29 |
| *** mriedem has joined #openstack-infra | 16:29 | |
| clarkb | jeblair: oh sure | 16:29 |
| clarkb | jeblair: the label refactor is much more important | 16:30 |
| clarkb | JayF: pypi updated their certs but DNS lagged | 16:30 |
| sdague | JayF: https://bugs.launchpad.net/openstack-ci/+bug/1298929 | 16:30 |
| uvirtbot | Launchpad bug 1298929 in openstack-ci "certificate errors downloading pip or installing pip packages" [Undecided,New] | 16:30 |
| jeblair | also "'*.target.com'" ?? | 16:30 |
| sdague | jeblair: yep, inspiring, right :) | 16:30 |
| fungi | clarkb: they also had at least one "stuck" server with the wrong cert | 16:31 |
| *** YorikSar has quit IRC | 16:31 | |
| clarkb | jeblair: pypi was the group that stole all of those credit card numbers >_> | 16:31 |
| JayF | jeblair: exactly | 16:31 |
| fungi | fastly did, i mean | 16:31 |
| JayF | jeblair: seeing fastly is not that disconcerting, seeing target /is/ | 16:31 |
| clarkb | sdague: I am going to write the e-r change to switch queries over to tag filtering for file names | 16:31 |
| jeblair | i trust target for all of my domestic merchandise and python packaging needs! | 16:32 |
| *** reed has joined #openstack-infra | 16:32 | |
| sdague | clarkb: ++ | 16:32 |
| flashgordon | clarkb: ++ can you add a test to make sure we don't add in filenames too | 16:32 |
| sdague | hopefully dstufft's fix will resolve that one. It's currently our top race bug | 16:32 |
| *** eehruchyshka|fri is now known as ihrachys|afk | 16:33 | |
| sdague | flashgordon: we're not doing any parsing of the query yet, I think that's beyond scope | 16:33 |
| *** comstud is now known as bearhands | 16:35 | |
| flashgordon | sdague: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/elastic_recheck/tests/unit/test_load_queries.py#n33 | 16:35 |
| *** gyee has joined #openstack-infra | 16:35 | |
| sdague | flashgordon: ok, I didn't realize there was syntax checking in there | 16:36 |
| clarkb | flashgordon: filenames are still valid | 16:37 |
| clarkb | flashgordon: maybe you know a bug only happens in grenade old logs | 16:37 |
| clarkb | flashgordon: I don't think telling everyone to stop using filename is the correct answer here | 16:37 |
| fungi | last logstash hit on the cert bug was more than an hour ago | 16:38 |
| sdague | fungi: cool | 16:39 |
| BobBall | dstufft: Great! glad I could help! | 16:39 |
| flashgordon | clarkb: kk | 16:39 |
| fungi | quiet since 15:16 utc | 16:40 |
| dstufft | jeblair: JayF hrm, why is seeing target.com disconcerting? | 16:41 |
| fungi | dstufft: good point... it was in-store purchases which ended up compromised after all | 16:43 |
| BobBall | btw - anyone know why nodepool is such a memory hog? Over time it will eat up 512M of ram... | 16:43 |
| BobBall | I have to keep restarting it :) | 16:43 |
| *** jaypipes is now known as leakypipes | 16:43 | |
| dstufft | fungi: well it's not like target.com has control over that certificate anyways | 16:43 |
| dstufft | only fastly does | 16:43 |
| *** VINOD_ has joined #openstack-infra | 16:43 | |
| fungi | dstufft: i took it as a glib joke ;) | 16:44 |
| dstufft | oh | 16:44 |
| jeblair | BobBall: no, i've tried and haven't found the culprit yet. if you have any ideas, please let me know. | 16:44 |
| dstufft | fungi: I've had people for real tell me that the extra names on the SAN cert were some sort of security problem | 16:44 |
| dstufft | andi'm lik no not really :( | 16:44 |
| *** HenryG has quit IRC | 16:44 | |
| BobBall | okies - np | 16:44 |
| *** hashar has quit IRC | 16:45 | |
| *** HenryG has joined #openstack-infra | 16:45 | |
| *** ameade has quit IRC | 16:45 | |
| jeblair | BobBall: i have learned that using a lot of threads comes with a surprisingly high memory cost, but that should just end up with a higher floor, it shouldn't be a leak. | 16:45 |
| *** ameade has joined #openstack-infra | 16:45 | |
| BobBall | I'll have a poke | 16:46 |
| jeblair | BobBall: an earlier version had a problem with image builds using a lot of ram, but that's been fixed. | 16:46 |
| jeblair | now it leaks slower. | 16:46 |
| BobBall | what was uising the ram in that case? | 16:46 |
| *** VINOD_ has quit IRC | 16:47 | |
| *** vhoward- has left #openstack-infra | 16:48 | |
| *** adalbas has quit IRC | 16:48 | |
| jeblair | BobBall: good question, i can't seem to find the commit right now | 16:49 |
| *** ominakov has quit IRC | 16:51 | |
| *** dcramer_ has quit IRC | 16:52 | |
| clarkb | sdague: flashgordon bug query 1285833 any idea why we can't look for that error in nova logs? | 16:53 |
| clarkb | it should be fine to have that there, its just odd that a message in all api logs would be an error except for nova | 16:54 |
| *** zns has joined #openstack-infra | 16:55 | |
| sdague | clarkb: I have no idea why it's excluding n-api | 16:56 |
| sdague | I'd be happy if it changed | 16:56 |
| *** SumitNaiksatam has quit IRC | 16:56 | |
| *** adalbas has joined #openstack-infra | 17:00 | |
| jeblair | BobBall: ah, the problem was related to keypairs -- we used to leak keypairs on failed image creations, then we ended up retrieving 40k keypair records using novaclient which used lots of ram | 17:00 |
| jeblair | BobBall: so that was fixed in 6ab771728ef83937db63762a848cbad476f44205 | 17:01 |
| jeblair | (and our observed high memory usage was really a side effect of that) | 17:01 |
| BobBall | Hmmmm - good point | 17:01 |
| *** malini has left #openstack-infra | 17:02 | |
| BobBall | so that one was just down to novaclient _using_ ram rather than leaking ram | 17:03 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Switch queries over to tags instead of filename https://review.openstack.org/83825 | 17:03 |
| clarkb | sdague: flashgordon ^ proper review on that would be good though I didn't just search and replace I actually looked at each query | 17:04 |
| *** khyati has joined #openstack-infra | 17:05 | |
| *** thuc_ has quit IRC | 17:06 | |
| *** markwash_ has joined #openstack-infra | 17:07 | |
| *** thuc has joined #openstack-infra | 17:07 | |
| *** markwash has quit IRC | 17:07 | |
| *** markwash_ is now known as markwash | 17:07 | |
| sdague | clarkb: sure, will do in a bit. | 17:07 |
| *** thuc_ has joined #openstack-infra | 17:09 | |
| *** gokrokve has joined #openstack-infra | 17:09 | |
| *** thuc has quit IRC | 17:09 | |
| *** pblaho has quit IRC | 17:09 | |
| *** nati_ueno has joined #openstack-infra | 17:11 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Replace more searches with filters. https://review.openstack.org/83828 | 17:11 |
| clarkb | and that is a follow up | 17:11 |
| *** SumitNaiksatam has joined #openstack-infra | 17:14 | |
| *** harlowja_away is now known as harlowja | 17:15 | |
| *** e0ne has quit IRC | 17:18 | |
| sdague | clarkb: we know this works? | 17:19 |
| sdague | AND tags:screen-*-api.txt | 17:19 |
| *** saju_m has joined #openstack-infra | 17:19 | |
| *** CaptTofu has joined #openstack-infra | 17:22 | |
| clarkb | sdague: it should, the field isn't special compared to filename | 17:23 |
| *** ProfFalken has quit IRC | 17:23 | |
| sdague | ok | 17:23 |
| sdague | a couple other things in there | 17:23 |
| jeblair | BobBall: yes, but i think it may have been holding onto that; maybe caching the info from the response... | 17:23 |
| *** ProfFalken has joined #openstack-infra | 17:24 | |
| jeblair | BobBall: which might suggest a place to investigate for the current problems.. | 17:24 |
| jeblair | BobBall: in my local searching for a 'leak', i didn't actually spin up and destroy machines (but did nearly everything else). perhaps novaclient is caching data from those operations, and simply the number of instances it deals with over time add up. | 17:25 |
| *** changbl has joined #openstack-infra | 17:27 | |
| *** _dguerri has quit IRC | 17:28 | |
| clarkb | sdague: responded. Will push a new patchset shortly | 17:28 |
| *** ProfFalken has quit IRC | 17:28 | |
| *** Kiall has quit IRC | 17:29 | |
| *** Shrews has quit IRC | 17:29 | |
| *** vponomaryov has left #openstack-infra | 17:30 | |
| sdague | clarkb: great | 17:30 |
| clarkb | sdague: https://review.openstack.org/#/c/83825/1/queries/1288427.yaml you think that should be console instead of devstacklog? | 17:31 |
| *** julienvey has quit IRC | 17:31 | |
| clarkb | sdague: and https://review.openstack.org/#/c/83825/1/queries/1288312.yaml console instead of console.html? | 17:31 |
| sdague | so if it's as FAIL: | 17:32 |
| *** dkliban_afk is now known as dkliban | 17:33 | |
| sdague | which is matching a tempest test, then console.html is fine | 17:33 |
| sdague | otherwise, we should probably default to tags:console | 17:33 |
| clarkb | ok | 17:33 |
| sdague | because it might show up in grenade logs | 17:33 |
| clarkb | ah | 17:33 |
| sdague | so 1288427.yaml - console | 17:34 |
| sdague | 1288312.yaml - console.html | 17:34 |
| *** bknudson has left #openstack-infra | 17:34 | |
| clarkb | sdague: and 1294874.yaml should be console? | 17:34 |
| *** saju_m has quit IRC | 17:35 | |
| *** saju_m has joined #openstack-infra | 17:36 | |
| *** lcheng has joined #openstack-infra | 17:36 | |
| sdague | clarkb: yes | 17:36 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Replace more searches with filters. https://review.openstack.org/83828 | 17:37 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Switch queries over to tags instead of filename https://review.openstack.org/83825 | 17:37 |
| *** moted has quit IRC | 17:38 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:39 | |
| pleia2 | clarkb: can you log into old-wiki and get me the info I need to finish this patch? https://review.openstack.org/#/c/82106/ (probably just cron job and config.js) | 17:39 |
| pleia2 | (or anyone) | 17:40 |
| *** Ryan_Lane has joined #openstack-infra | 17:40 | |
| pleia2 | once this is in, bug 1127411 is the last remaining to get rid of this server | 17:41 |
| uvirtbot | Launchpad bug 1127411 in openstack-ci "Move the $project.openstack.org redirects off of old wiki" [Medium,Triaged] https://launchpad.net/bugs/1127411 | 17:41 |
| *** markmc has quit IRC | 17:42 | |
| clarkb | pleia2: looking at the change now | 17:42 |
| fungi | pleia2: oh, sorry, i said i'd get you that and then got sidetracked | 17:42 |
| pleia2 | fungi: no worries, I know you've been busy :) | 17:43 |
| fungi | i'm pretty sure everybody is ;) | 17:43 |
| *** BobBall has quit IRC | 17:44 | |
| pleia2 | yeah, just trying to chip away at these servers we should get rid of | 17:44 |
| clarkb | pleia2: I am digging aruond on old wiki | 17:46 |
| clarkb | I feel so blind on this machine | 17:46 |
| *** vhoward- has joined #openstack-infra | 17:46 | |
| pleia2 | :) | 17:46 |
| *** bknudson has joined #openstack-infra | 17:46 | |
| clarkb | pleia2: */30 * * * * | 17:47 |
| clarkb | for bugdaystats/refresh_data | 17:47 |
| pleia2 | thanks | 17:47 |
| clarkb | and you want config.js contents? | 17:47 |
| fungi | pleia2: http://paste.openstack.org/show/74582/ | 17:47 |
| clarkb | config.js coming up | 17:48 |
| fungi | and http://paste.openstack.org/show/74583/ | 17:48 |
| pleia2 | fungi: hm, commented out? | 17:48 |
| clarkb | ya | 17:49 |
| fungi | pleia2: seems that way | 17:49 |
| fungi | pleia2: -rw------- 1 ttx crontab 370 Mar 7 20:54 /var/spool/cron/crontabs/ttx | 17:50 |
| fungi | i suspect he enables or manually runs it or something? | 17:50 |
| pleia2 | neat, so once this lands we'll actually have *working* bugdaystats too! | 17:50 |
| clarkb | fungi: the files in output/ are old | 17:50 |
| pleia2 | thanks guys | 17:50 |
| clarkb | fungi: not sure if he is running it elsewhere maybe? | 17:50 |
| clarkb | oh wait nevermind those should be old /me digs more for actual outpout | 17:51 |
| pleia2 | old-wiki is definitely where status.o.o points | 17:51 |
| fungi | yeah, i'm thinking bugdaystats got disabled in cron a few weeks ago based on that crontab | 17:52 |
| clarkb | fungi: the files are in /home/ttx/bugstats | 17:52 |
| clarkb | fungi: and they end on march 7th | 17:52 |
| clarkb | so I think you are correct | 17:52 |
| fungi | seems confirmed | 17:52 |
| *** ociuhandu has quit IRC | 17:52 | |
| fungi | pleia2: as for the other remaining task in 1127411, i think it was decided yesterday that we just delete them from dns and see who screams | 17:53 |
| pleia2 | fungi: excellent | 17:54 |
| clarkb | ++ | 17:54 |
| fungi | because they've been permanent redirects for a very long time, and current documents supposedly haven't linked using those names for ages | 17:54 |
| *** david-lyle is now known as david-lyle_afk | 17:54 | |
| fungi | at least not for havana, and grizzly is now eol | 17:54 |
| * fungi gets back to making it so | 17:55 | |
| * pleia2 nods | 17:55 | |
| fungi | but i think i remember documentation getting cleaned up in that regard circa folsom | 17:55 |
| *** krtaylor has quit IRC | 17:55 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Fix missing attribute error in subnodes https://review.openstack.org/83843 | 17:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add a test for subnodes https://review.openstack.org/83602 | 17:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add a very basic functional test https://review.openstack.org/82952 | 17:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add per-test database fixture https://review.openstack.org/82865 | 17:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add tests for the allocator https://review.openstack.org/82628 | 17:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive https://review.openstack.org/82147 | 17:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Finish initial docs https://review.openstack.org/82854 | 17:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add ready-script and multi-node support https://review.openstack.org/82252 | 17:56 |
| jeblair | rebase on master (with flashgordon's change) plus... a bugfix caught by a test! | 17:56 |
| *** derekh has quit IRC | 17:56 | |
| pleia2 | hooray for tests | 17:56 |
| clarkb | mriedem: responded to your e-r comment | 17:57 |
| mriedem | clarkb: ok | 17:57 |
| clarkb | mriedem: basically I want to do a 1:1 mapping in that change then everyone can sweep through the queries and improve them | 17:57 |
| clarkb | mriedem: for example the change that is a child of that change does some of this cleanup | 17:57 |
| *** dcramer_ has joined #openstack-infra | 17:58 | |
| clarkb | jeblair: where is flashgordon's change? | 17:58 |
| jeblair | clarkb: merged | 17:58 |
| clarkb | there is thanks | 17:59 |
| clarkb | *it is | 17:59 |
| rossella_s | jeblair: thanks for the patch! | 18:00 |
| rossella_s | sdague, jeblair are you around to finish the discussion we started yesterday? | 18:00 |
| jeblair | rossella_s: no problem; it's even passing tests now | 18:00 |
| rossella_s | jeblair: nice, when do you think it will be possible to see the job in action? | 18:02 |
| *** mkerrin has quit IRC | 18:02 | |
| jeblair | rossella_s: we'll need to restart zuul after it merges; so probably next week at the earliest | 18:03 |
| clarkb | a zuul change too? /me feels like he has missed a bunch of stuff yesterday | 18:03 |
| rossella_s | jeblair: ok, great! I will wait then. Thanks a lot! | 18:03 |
| clarkb | that reminds me I need to fix my zuul patch... not enough hours in a day | 18:04 |
| clarkb | jeblair: reviewing the nodepool stack now, hopefully we can get that in today | 18:04 |
| *** vhoward- has left #openstack-infra | 18:05 | |
| sdague | here's a fun new fail | 18:05 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcImNvdWxkbid0IGNvbm5lY3QgdG8gaG9zdFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzk2MDI5OTAwMjQxfQ== | 18:05 |
| sdague | rax only | 18:06 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Fixed contenteditable https://review.openstack.org/83847 | 18:06 |
| *** thuc_ has quit IRC | 18:07 | |
| *** thuc has joined #openstack-infra | 18:08 | |
| jeblair | mordred, sdague: it would be pretty cool if the install_pip_tarball thing didn't download from the net | 18:10 |
| clarkb | fungi: anteaya jeblair anyone looking at new-projects for today? | 18:10 |
| jeblair | clarkb: i have not | 18:12 |
| ryanpetrello | is there anybody in here I can nag into taking a peek at https://review.openstack.org/#/c/83564/ ? I’m eager to give it a try :) | 18:12 |
| *** thuc has quit IRC | 18:12 | |
| clarkb | I want to get nodepool changes in, but I can probably look at new-projects afterwards if no one beats me to it | 18:12 |
| *** thuc has joined #openstack-infra | 18:13 | |
| jeblair | clarkb: 2/3 lgtm, i'll push a new patch for the problem i saw | 18:15 |
| *** jp_at_hp has quit IRC | 18:16 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Create neutron-specs git repo https://review.openstack.org/82512 | 18:16 |
| *** ildikov_ has quit IRC | 18:17 | |
| pleia2 | so, tripleo wants to have a local ubuntu/fedora mirror in their ci, I'm going to put the puppet stuff together to make this happen, we're not doing this for -infra, but would it be valuable for me to put it in config/ anyway if we want/need it in the future? | 18:17 |
| pleia2 | instead of shoving it into tripleo-specific repos | 18:18 |
| jeblair | actually, i see another problem | 18:18 |
| jeblair | pleia2: i thought the point of the tripleo ci was to be part of the project infrastructure | 18:19 |
| jeblair | pleia2: i think that means if there are dependencies, they need to be managed that way. | 18:20 |
| pleia2 | jeblair: well, there is still a separate tripleo-ci repo | 18:20 |
| pleia2 | but I agree | 18:20 |
| *** andreaf has quit IRC | 18:20 | |
| pleia2 | putting it in config/ makes the most sense to me | 18:20 |
| fungi | pleia2: it seems to me like something we might legitimately want for future job stability (keeping in mind the bugs we get from time to time because of a packaging mirror going down or getting updated in wrong/non-transparent ways) | 18:20 |
| pleia2 | fungi: yeah | 18:20 |
| clarkb | ++ | 18:20 |
| pleia2 | great | 18:20 |
| jeblair | fungi: agreed | 18:20 |
| clarkb | its definitely something that once in a while we seem to agree we should have a mirror then the problems go away | 18:20 |
| fungi | similar to how we might want to switch to pypi mirroring in each provider for improved latency/throughput/stability and so on | 18:21 |
| clarkb | jeblair: nodepool stack lgtm | 18:22 |
| jeblair | fungi: yep. and dns once we can have a private network with neutron | 18:22 |
| jeblair | (everywhere) | 18:22 |
| fungi | i'm jumping back on the nodepool changes now. i too would like to see them merge and have a weekend to bake in | 18:22 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add the openstack-security-notes project https://review.openstack.org/73157 | 18:23 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add StackTach project https://review.openstack.org/83208 | 18:24 |
| clarkb | jeblair: I have a comment for stacktach | 18:24 |
| clarkb | jeblair: did you want to address that comment? | 18:25 |
| *** dims has joined #openstack-infra | 18:25 | |
| jeblair | clarkb: not really. feel free. | 18:25 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add StackTach project https://review.openstack.org/83208 | 18:27 |
| clarkb | there it is | 18:27 |
| clarkb | I am going to give others a few minutes to review, but then I will approve the bunch so that repo creation can happen | 18:28 |
| *** vkozhukalov has joined #openstack-infra | 18:30 | |
| *** vkozhukalov has left #openstack-infra | 18:30 | |
| *** jgallard has quit IRC | 18:31 | |
| *** signed8bit has quit IRC | 18:32 | |
| devananda | clarkb: hi! I'm wondering if the transifex error is either more wide spread, or a red herring. it seems to be in nova's logs too: http://logs.openstack.org/8d/8df565d39cd3216bde8488e656957a49dd6949fa/post/nova-upstream-translation-update/d8f54e6/console.html | 18:32 |
| clarkb | devananda: I have no idea | 18:33 |
| clarkb | devananda: check with the translation team maybe? | 18:33 |
| *** dstanek has quit IRC | 18:33 | |
| clarkb | devananda: https://review.openstack.org/#/c/83672/ shows a diff | 18:34 |
| clarkb | but thats mostly just keeping the .pot file up to date and updating .po files accordingly | 18:34 |
| mordred | jeblair: oh - the one in devstack? | 18:34 |
| *** dstanek has joined #openstack-infra | 18:34 | |
| jeblair | mordred: yeah | 18:35 |
| jeblair | mordred: http://logs.openstack.org/06/83606/3/gate/gate-tempest-dsvm-neutron/224f7b6/logs/devstacklog.txt.gz | 18:35 |
| mordred | jeblair: well, we could certainly pre-fetch get-pip.py in devstack-gate | 18:35 |
| jeblair | mordred: it should be in the nodepool prep scripts i think | 18:35 |
| mordred | s/devstack-gate/nodepool-prep/ | 18:35 |
| mordred | yeah | 18:35 |
| clarkb | devananda: I am inclined to believe that this is a widespread problem | 18:36 |
| devananda | great :( | 18:36 |
| clarkb | devananda: is there a bug yet? we can send mail to tom fifieldt and daisy to see if they know what is going on | 18:36 |
| devananda | no. I'll open one now | 18:36 |
| devananda | wait. yes | 18:36 |
| devananda | clarkb: https://bugs.launchpad.net/ironic/+bug/1298645 | 18:37 |
| uvirtbot | Launchpad bug 1298645 in ironic "translated .po files do not contain any translation text" [High,Triaged] | 18:37 |
| mordred | jeblair: funny enough - we DO already download get-pip.py in nodepool scripts | 18:37 |
| clarkb | devananda: want to craft and email and send it to daisy and tom? Or do they have a list? we can send it to a list if they have one /me checks | 18:37 |
| mordred | jeblair: we should just put it somewhere so that we can tell destack about it :) | 18:37 |
| devananda | clarkb: there's an irc channel | 18:37 |
| clarkb | devananda: they do http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-i18n | 18:37 |
| clarkb | I think I am even on that list | 18:37 |
| fungi | clarkb: i'm cool with all three the new-project changes, so approve at your leisure | 18:37 |
| devananda | i just jumped in and am pinging them | 18:37 |
| clarkb | fungi: danke | 18:38 |
| clarkb | devananda: awesome let me know if I should find that channel too | 18:38 |
| devananda | clarkb: ack | 18:38 |
| anteaya | clarkb: just got back, looks like all three new-projects patches are in shape to merge | 18:39 |
| clarkb | anteaya: yup, I just approved all of them | 18:39 |
| anteaya | clarkb: are you doing the manage-project honours today? | 18:39 |
| anteaya | answers my question | 18:39 |
| clarkb | anteaya: once git01-git05 have the new repos I can run manage-projects | 18:39 |
| anteaya | k | 18:39 |
| fungi | mordred: agreed--though as a separate aside, it might be helpful for devstack to grow a downloads list similar to its apts/debs lists where urls for things it could conceivably want are abstracted out and can then be more easily pre-cached to a specific location it looks for | 18:39 |
| jeblair | mordred: cool, can you take care of that please? | 18:39 |
| clarkb | that will probably happen a little after noon PST so others should feel free to trigger manage-projects while I eat lunch | 18:39 |
| mordred | jeblair: yes | 18:40 |
| *** dizquierdo has quit IRC | 18:40 | |
| mordred | fungi: funny story - diskimage-builder has such an abstraction | 18:40 |
| fungi | mordred: proof that devstack should just use dib | 18:40 |
| mordred | fungi: it would be interesting to see if it's possible to re-use their code | 18:41 |
| mordred | or that | 18:41 |
| mordred | :) | 18:41 |
| jeblair | fungi: how would devstack use dib? | 18:41 |
| jeblair | what would it do with the images it built? | 18:41 |
| * mordred assumes fungi was joking | 18:41 | |
| * jeblair has been trolled | 18:41 | |
| fungi | yes, mostly joking... though it could potentially use parts of dib without using it to build disk images directly | 18:41 |
| mordred | yah | 18:41 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Fixed #37, project UI shows edit buttons when not superuser. https://review.openstack.org/83856 | 18:42 |
| openstackgerrit | A change was merged to openstack-infra/config: Add StackTach project https://review.openstack.org/83208 | 18:42 |
| *** afazekas has joined #openstack-infra | 18:43 | |
| openstackgerrit | A change was merged to openstack-infra/config: Create neutron-specs git repo https://review.openstack.org/82512 | 18:44 |
| openstackgerrit | A change was merged to openstack-infra/config: Add the openstack-security-notes project https://review.openstack.org/73157 | 18:44 |
| *** sheeprine has left #openstack-infra | 18:47 | |
| *** openstackgerrit has quit IRC | 18:48 | |
| *** openstackgerrit has joined #openstack-infra | 18:48 | |
| *** dstanek has quit IRC | 18:49 | |
| *** thuc has quit IRC | 18:49 | |
| *** dguerri has joined #openstack-infra | 18:50 | |
| *** kiall has joined #openstack-infra | 18:52 | |
| *** ProfFalken has joined #openstack-infra | 18:54 | |
| *** thuc has joined #openstack-infra | 18:55 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Adds sanitised sourceforge source repo requirement to docs https://review.openstack.org/83796 | 18:56 |
| *** _nadya_ has joined #openstack-infra | 18:58 | |
| *** _nadya_ has quit IRC | 18:59 | |
| *** harlowja is now known as harlowja_away | 19:00 | |
| *** david-lyle_afk is now known as david-lyle | 19:00 | |
| *** _nadya_ has joined #openstack-infra | 19:00 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Add data load instructions to dev install doc https://review.openstack.org/83129 | 19:03 |
| *** signed8bit has joined #openstack-infra | 19:04 | |
| *** _nadya_ has quit IRC | 19:05 | |
| *** e0ne has joined #openstack-infra | 19:07 | |
| *** wenlock_ has joined #openstack-infra | 19:07 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Move bugdaystats to to status.openstack.org https://review.openstack.org/82106 | 19:07 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Remove unimplemented features in list view. https://review.openstack.org/83865 | 19:08 |
| fungi | sourceforge! | 19:09 |
| *** ociuhandu has joined #openstack-infra | 19:09 | |
| fungi | i'd almost forgotten about them | 19:10 |
| sdague | mordred: is get-pip in nodepool, or the actual pip tar ball? | 19:10 |
| *** dstanek has joined #openstack-infra | 19:10 | |
| pleia2 | fungi: probably for the best | 19:10 |
| sdague | because if it's only get-pip, that's still got to hit the network | 19:11 |
| mordred | sdague: get-pip.py is the thing you download | 19:11 |
| mordred | it has everything bundled in it | 19:11 |
| sdague | oh, it does? | 19:11 |
| mordred | yes | 19:11 |
| sdague | https://github.com/openstack-dev/devstack/blob/master/tools/install_pip.sh#L65 | 19:12 |
| *** wenlock_ has quit IRC | 19:12 | |
| sdague | so just put it in the right place :) | 19:12 |
| *** zehicle has joined #openstack-infra | 19:12 | |
| mordred | there we go! | 19:12 |
| mordred | we should set use_get_pip to be default | 19:12 |
| *** ociuhandu has quit IRC | 19:13 | |
| sdague | is it not because it's not versioned? | 19:13 |
| mordred | possibly so | 19:13 |
| *** CaptTofu has quit IRC | 19:14 | |
| sdague | and we still have a 1.5 issue, right? | 19:14 |
| mordred | don't think so | 19:14 |
| mordred | we have an issue with newer tox | 19:14 |
| mordred | but ... | 19:14 |
| mordred | let's say two things | 19:14 |
| *** CaptTofu has joined #openstack-infra | 19:14 | |
| mordred | we should put in logic in the tarball fork to look for the file in $FILES first | 19:14 |
| clarkb | mordred: sdague right pip 1.5.latest should work fine | 19:14 |
| mordred | then we should have nodepool prep scripts put both get-pip.py and a pip tarball in $FILES | 19:15 |
| clarkb | it happens that making unittests use pip 1.5.latest is a pita with the state of tox | 19:15 |
| *** denis_makogon_ has joined #openstack-infra | 19:15 | |
| sdague | mordred: yes, I'm already with you | 19:15 |
| dstufft | get-pip.py --no-index --find-links=$FILES | 19:15 |
| sdague | Sean Dague proposed a change to openstack-dev/devstack: cache the pip tarball https://review.openstack.org/83866 | 19:15 |
| dstufft | you might like ^? | 19:15 |
| *** denis_makogon_ is now known as denis_makogon | 19:16 | |
| *** _nadya_ has joined #openstack-infra | 19:16 | |
| mordred | dstufft: well, we have to find get-pip.py | 19:17 |
| mordred | dstufft: oh - I getcha | 19:17 |
| mordred | that way we can always use get-pip as the entrypoint to the process, and control the version by what's in FILES | 19:17 |
| *** harlowja_away is now known as harlowja | 19:17 | |
| dstufft | yea | 19:17 |
| sdague | except where we don't have get-pip | 19:18 |
| dstufft | get-pip.py is really just like calling pip | 19:18 |
| dstufft | except with some arguments already pre-set | 19:18 |
| clarkb | mriedem: sdague flashgordon mtreinish re https://review.openstack.org/#/c/83825/ what do you guys think about my comment there? Should I fix the -1 in that change? | 19:18 |
| sdague | devstack needs to be able to bootstrap from nothing | 19:18 |
| *** melwitt has joined #openstack-infra | 19:18 | |
| dstufft | you can embed get-pip.py in it if you want | 19:18 |
| mordred | sdague: yah - if there is no FILES/get-pip.py - I think you still want to download | 19:19 |
| mordred | sdague: and if there is no FILES/pip*tar.gz, you want to download | 19:19 |
| *** CaptTofu has quit IRC | 19:19 | |
| sdague | yep | 19:19 |
| mriedem | clarkb: on a call, i'll have to check it later | 19:19 |
| clarkb | my ssh's in for loops say all three new projects have repos on all 5 git replicas | 19:19 |
| sdague | so after my last patch we'll do both | 19:19 |
| clarkb | fungi: jeblair mordred SergeyLukjanov anteaya et al I am going to run manage-projects now | 19:19 |
| mordred | sdague: but if there is FILES/get-pip.py, you want to use it - and if there is FILES/pip*tar.gz - you want to do the form dstufft said above | 19:20 |
| *** CaptTofu has joined #openstack-infra | 19:20 | |
| mordred | clarkb: ++ | 19:20 |
| mordred | sdague: cool | 19:20 |
| *** CaptTofu has quit IRC | 19:20 | |
| *** zehicle has quit IRC | 19:21 | |
| *** zehicle has joined #openstack-infra | 19:21 | |
| clarkb | it is running | 19:21 |
| clarkb | log is in roots homedir and dated | 19:21 |
| anteaya | clarkb: yay | 19:24 |
| *** dcramer_ has quit IRC | 19:25 | |
| *** eharney has quit IRC | 19:26 | |
| lifeless | NobodyCam: you'll get more help from derekh :> | 19:29 |
| lifeless | NobodyCam: but | 19:29 |
| lifeless | NobodyCam: see openstack-infra/tripleo-ci repository, in that look at the ci_commands file and you can grep around to see how its installed for a key | 19:29 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add bug for rax network blipping on get-pip https://review.openstack.org/83868 | 19:30 |
| NobodyCam | lifeless: Ty.. I was looking into the isse then got side tracked... | 19:31 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/bugdaystats: Wrap results on index.html in container div https://review.openstack.org/83870 | 19:31 |
| *** packet has joined #openstack-infra | 19:33 | |
| sdague | clarkb: so actually it would be good to fix the think mriedem called out in - https://review.openstack.org/#/c/83825/2 | 19:34 |
| *** ildikov_ has joined #openstack-infra | 19:34 | |
| sdague | because I think when we did the devstacklog we lost most of those hits | 19:34 |
| sdague | if you change that one, then I'll +A it through after | 19:34 |
| clarkb | sdague: ok | 19:34 |
| sdague | then I'll run a delete of all the old bugs before we do more here, just so we aren't fixing things we might delete anyway | 19:35 |
| mriedem | clarkb: sdague: off my call, so i'll agree with sdague since it sounds like he's agreeing with me :) | 19:35 |
| sdague | heh | 19:35 |
| sdague | that's slightly cheating | 19:35 |
| *** jcoufal has joined #openstack-infra | 19:36 | |
| clarkb | manage-projects is done | 19:36 |
| clarkb | anteaya: want to check things? | 19:36 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Replace more searches with filters. https://review.openstack.org/83828 | 19:37 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Switch queries over to tags instead of filename https://review.openstack.org/83825 | 19:37 |
| clarkb | mriedem: sdague ^ | 19:37 |
| *** dcramer_ has joined #openstack-infra | 19:38 | |
| * anteaya will check | 19:38 | |
| *** jcoufal has quit IRC | 19:38 | |
| anteaya | I can clone all three | 19:40 |
| anteaya | thanks clarkb | 19:40 |
| *** david-lyle has quit IRC | 19:40 | |
| *** david-lyle has joined #openstack-infra | 19:40 | |
| clarkb | awesome, I will look at groups and acls and add group members | 19:40 |
| openstackgerrit | Jim Rollenhagen proposed a change to openstack-infra/config: Add PyPy gate to ironic-python-agent https://review.openstack.org/83873 | 19:41 |
| mriedem | clarkb: so i expected all of the queries to change from like tags:"console.html" to tags:console ? | 19:42 |
| clarkb | mriedem: sdague: right that is what I am trying to say. That is a different change and needs to be evaluated on a per query basis | 19:42 |
| clarkb | this change is just taking us from using filename to tags | 19:42 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: periodic cleanup of bugs https://review.openstack.org/83874 | 19:43 |
| clarkb | or was the initial intent at least | 19:43 |
| sdague | mriedem: not always | 19:43 |
| mriedem | so i just got lucky on picking on the first one? | 19:43 |
| sdague | yeh | 19:43 |
| mriedem | oh | 19:43 |
| sdague | it's probably worth a pass through to fix more of them | 19:43 |
| sdague | but lets do that on another round | 19:44 |
| sdague | after the deletes | 19:44 |
| sdague | because it's pointless to fix fingerprints that are getting removed | 19:44 |
| sdague | so review on the deletes would be good as well - https://review.openstack.org/83874 | 19:44 |
| anteaya | annegentle nkinder http://git.openstack.org/cgit/openstack/openstack-security-notes/tree/ | 19:44 |
| nkinder | anteaya: yay! | 19:45 |
| annegentle | anteaya: thank you!! | 19:45 |
| anteaya | welcome | 19:45 |
| anteaya | clarkb: did the heavy lifting | 19:45 |
| sdague | that brings us down to 74 bugs that we've seen in the last 14 days | 19:45 |
| sdague | 74 .... ooof | 19:45 |
| clarkb | nkinder: I was going to add you to the core group then you can add whoever you like | 19:46 |
| clarkb | sdague: ya that list was big :) | 19:46 |
| nkinder | clarkb: I was just going to ask about that, but thought I'd check if you gave me that priv. before asking | 19:47 |
| sdague | I just deleted 26 | 19:47 |
| clarkb | nkinder: done | 19:47 |
| sdague | so we were at an even 100 | 19:47 |
| nkinder | clarkb: thanks! | 19:48 |
| clarkb | markmcclain: you are the seed neutron-specs-core member | 19:48 |
| clarkb | and sandy walsh has been added to the stacktach groups | 19:49 |
| *** dstanek has quit IRC | 19:50 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: periodic cleanup of bugs https://review.openstack.org/83874 | 19:50 |
| sdague | yay, rebase button | 19:50 |
| clarkb | I think that concludes new project friday | 19:51 |
| clarkb | ACLs check out | 19:51 |
| sdague | clarkb: while I'm thinking about er things | 19:51 |
| anteaya | sandywalsh says thanks clarkb | 19:51 |
| anteaya | we are talking in pm he has been having difficulty with freenode rooms today | 19:51 |
| sdague | the puppet policy intentionally was set to trigger a page refresh on a commit of new queries | 19:52 |
| sdague | but I don't think that's actually happening | 19:52 |
| clarkb | anteaya: no problem | 19:52 |
| sdague | is that just because the puppet granularity is too high? | 19:52 |
| clarkb | sdague: it isn't we undid it | 19:52 |
| sdague | oh | 19:52 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/bugdaystats: Add link to JSON file on index.html https://review.openstack.org/83878 | 19:52 |
| sdague | ok, bummer | 19:52 |
| clarkb | sdague: running all those queries every time a change merged was quite noticeable on the cluster side | 19:53 |
| clarkb | granted this was before growing the cluster | 19:53 |
| sdague | ok | 19:53 |
| clarkb | sdague: so now we run them 20 minutes apart once per hour per script | 19:53 |
| clarkb | sdague: we can probably drop that to once every 10 minutes, twice and hour | 19:53 |
| clarkb | but doing all three at the same time was sad times | 19:53 |
| sdague | ok | 19:54 |
| sdague | that's fair | 19:54 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Switch queries over to tags instead of filename https://review.openstack.org/83825 | 19:54 |
| clarkb | sdague: want me to propose a patch for every half hour? | 19:54 |
| sdague | yeh, that would be great | 19:54 |
| clarkb | er should read 10 minutes apart, once every half hour, but I think you grokked | 19:55 |
| *** bookwar has quit IRC | 19:58 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/bugdaystats: Recreate index.html with each run https://review.openstack.org/83880 | 19:59 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Run e-r batch jobs twice an hour ten minutes apart https://review.openstack.org/83881 | 19:59 |
| pleia2 | ok, no more bugdaystats rabbit hole | 19:59 |
| clarkb | sdague: ^ that should do it | 19:59 |
| anteaya | pleia2: yay | 19:59 |
| clarkb | pleia2: I meant to check on tripleo-f20 today | 19:59 |
| pleia2 | anteaya: hehe, all day has been "oh, I should fix that too" | 19:59 |
| pleia2 | clarkb: it built! :D | 20:00 |
| pleia2 | clarkb: tripleo-f20-tripleo-test-cloud-3255167.slave.openstack.org | ACTIVE | - | Running | 20:00 |
| anteaya | pleia2: I see that | 20:00 |
| clarkb | woot | 20:00 |
| anteaya | pleia2: like picking up shells at the beach | 20:00 |
| pleia2 | anteaya: right | 20:00 |
| anteaya | there's a nice one, oh and that one too | 20:00 |
| anteaya | :D | 20:00 |
| pleia2 | hehe | 20:01 |
| clarkb | pleia2: do we have a job to run on that node yet? | 20:01 |
| * pleia2 digs up patch | 20:01 | |
| pleia2 | clarkb: https://review.openstack.org/#/c/81793/ | 20:02 |
| clarkb | thanks looking | 20:02 |
| clarkb | hrm no derekh | 20:02 |
| clarkb | honestly I think that job should be added to the layout.yaml as an experimental job /me leaves a comment | 20:03 |
| pleia2 | he sometimes comes back in late evening ireland time, or monday | 20:03 |
| * anteaya heads out for a walk | 20:04 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/bugdaystats: Recreate index.html with each run https://review.openstack.org/83880 | 20:05 |
| *** dprince has quit IRC | 20:06 | |
| *** krtaylor has joined #openstack-infra | 20:07 | |
| *** yolanda has quit IRC | 20:09 | |
| *** yassine has quit IRC | 20:16 | |
| *** eharney has joined #openstack-infra | 20:23 | |
| *** _nadya_ has quit IRC | 20:25 | |
| *** melwitt has quit IRC | 20:25 | |
| *** melwitt has joined #openstack-infra | 20:26 | |
| *** julim has quit IRC | 20:27 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Enable cross-testing of oslo.test with dib and t-i-e https://review.openstack.org/83885 | 20:29 |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Run unit tests in tripleo-image-elements https://review.openstack.org/83886 | 20:29 |
| *** thuc has quit IRC | 20:32 | |
| *** thuc has joined #openstack-infra | 20:33 | |
| *** julim has joined #openstack-infra | 20:33 | |
| *** thuc_ has joined #openstack-infra | 20:35 | |
| *** zehicle has quit IRC | 20:36 | |
| *** thuc_ has quit IRC | 20:36 | |
| *** zehicle has joined #openstack-infra | 20:36 | |
| *** thuc_ has joined #openstack-infra | 20:36 | |
| *** thuc_ has quit IRC | 20:37 | |
| *** thuc has quit IRC | 20:37 | |
| *** mwagner_bbl has quit IRC | 20:37 | |
| *** thuc has joined #openstack-infra | 20:38 | |
| *** zhiyan is now known as zhiyan_ | 20:38 | |
| *** lcostantino has quit IRC | 20:39 | |
| *** Sukhdev has joined #openstack-infra | 20:45 | |
| *** alexpilotti has quit IRC | 20:45 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Run unit tests in tripleo-image-elements https://review.openstack.org/83886 | 20:46 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: update tests that should be tags:console https://review.openstack.org/83892 | 20:47 |
| *** jeckersb is now known as jeckersb_gone | 20:50 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Run unit tests in tripleo-image-elements https://review.openstack.org/83886 | 20:51 |
| *** CaptTofu has joined #openstack-infra | 20:52 | |
| *** dims has quit IRC | 20:53 | |
| *** andreaf has joined #openstack-infra | 20:54 | |
| *** _nadya_ has joined #openstack-infra | 20:55 | |
| *** ociuhandu has joined #openstack-infra | 20:56 | |
| *** julim has quit IRC | 20:59 | |
| *** mfer has quit IRC | 21:00 | |
| *** mfer has joined #openstack-infra | 21:00 | |
| *** mfer has quit IRC | 21:01 | |
| *** Ryan_Lane has quit IRC | 21:01 | |
| *** mestery has joined #openstack-infra | 21:02 | |
| sdague | ttx: you still awake? | 21:04 |
| *** Ryan_Lane has joined #openstack-infra | 21:04 | |
| *** denis_makogon has quit IRC | 21:07 | |
| *** amcrn has joined #openstack-infra | 21:08 | |
| *** jergerber has joined #openstack-infra | 21:08 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: update tests that should be tags:console https://review.openstack.org/83892 | 21:14 |
| *** wenlock has quit IRC | 21:22 | |
| *** aysyd has quit IRC | 21:23 | |
| openstackgerrit | Martin Konrad proposed a change to openstack-infra/jenkins-job-builder: SCM module: Add support for multiple Git remotes. https://review.openstack.org/80084 | 21:23 |
| fungi | partway through the remaining nodepool change stack. need to take a break and grab some dinner... bbiaw | 21:26 |
| *** ameade has quit IRC | 21:26 | |
| devananda | clarkb: got a response on the i18n ML | 21:28 |
| devananda | clarkb: https://review.openstack.org/#/c/81925/ | 21:28 |
| openstackgerrit | Andrey Kurilin proposed a change to openstack/requirements: Add junitxml to global requirements https://review.openstack.org/83903 | 21:31 |
| *** markmcclain has quit IRC | 21:31 | |
| *** khyati has quit IRC | 21:36 | |
| mordred | I just -1'd that ^^ | 21:36 |
| clarkb | devananda: roger, does that give you a path forward for ironic? | 21:36 |
| clarkb | mordred: its so that people can get junitxml with pypi-mirror | 21:37 |
| clarkb | mordred: we have created a monster | 21:37 |
| devananda | clarkb: we'll try it. does it seem like a viable fix to you? | 21:37 |
| clarkb | devananda: ya | 21:37 |
| devananda | clarkb: great | 21:37 |
| mordred | clarkb: wait - no, why do they need junitxml in _our_ pypi-mirror? | 21:37 |
| mordred | clarkb: pypi-mirror itself takes a list of requirements repos | 21:38 |
| mordred | so if they're building their own mirror, they can make their own list | 21:38 |
| clarkb | mordred: right, but I bet they don't realize that | 21:38 |
| mordred | we do not put things in global-requirements to serve mirrors, we build mirrors to serve global requirements | 21:38 |
| clarkb | mordred: I agree, your -1 is correct (may need to be a -2) | 21:39 |
| clarkb | I was mostly half jokingly commenting on why someone may want that change | 21:39 |
| *** dangers is now known as dangers_away | 21:39 | |
| mordred | clarkb: if someone on requirements-core +2/+A's that with a -1 on it it during a requirements freeze, I believe that's grounds for horsewhipping | 21:39 |
| *** mriedem has quit IRC | 21:40 | |
| clarkb | what is the sitrep on pypy tests? https://review.openstack.org/#/c/83873/1/modules/openstack_project/files/zuul/layout.yaml | 21:40 |
| clarkb | are we still in the hack of a workaround for that? | 21:40 |
| clarkb | fungi: ^ | 21:40 |
| flashgordon | is there any good way to parse rst and extract the titles? | 21:41 |
| sdague | mordred: I just -2ed it | 21:42 |
| clarkb | flashgordon: there are a bunch of rst2stuff utilities that come with doctools | 21:42 |
| clarkb | flashgordon: rst2xml may make grepping easier | 21:42 |
| sdague | the best kind of -2, with no comments :) | 21:42 |
| sdague | it's friday and all | 21:42 |
| clarkb | flashgordon: but, you should be able to to multiline sed it I would think | 21:42 |
| *** khyati has joined #openstack-infra | 21:42 | |
| sdague | flashgordon: you maybe be over thinking things :) | 21:43 |
| *** signed8bit has quit IRC | 21:46 | |
| *** dkranz has quit IRC | 21:46 | |
| clarkb | also the rst tools are python, I am sure you can import doctools and hack on the parser direclt | 21:46 |
| clarkb | y | 21:46 |
| *** rfolco has quit IRC | 21:47 | |
| *** _nadya_ has quit IRC | 21:50 | |
| *** miqui has quit IRC | 21:51 | |
| clarkb | jeblair: fungi: where are we on nodepool? | 21:51 |
| jeblair | clarkb: we have sufficient +2 on the breaking change, so we could go aheand and merge that | 21:52 |
| clarkb | I should be around for at least another hour and a half to two hours | 21:53 |
| *** thomasem has quit IRC | 21:53 | |
| clarkb | jeblair: will defer to your judgement on this one as you are the author, do ping me if you need me to do anything to move that along | 21:53 |
| jeblair | clarkb: i'll merge the breaking change now; stop puppet, merge the config change, then start puppet | 21:54 |
| *** khyati has quit IRC | 21:54 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 21:54 |
| clarkb | jeblair: ok | 21:54 |
| jeblair | i don't think it's worth going further than that because we won't actually be able to use the multi stuff until the whole tree is merged | 21:54 |
| *** rcleere has quit IRC | 21:54 | |
| clarkb | jeblair: right so incorporate labels to get past the DB and config change | 21:54 |
| clarkb | then let the other pieces happen more naturally | 21:55 |
| clarkb | jeblair: you will massage the DB too? | 21:55 |
| jeblair | clarkb: yep, will do | 21:55 |
| *** ominakov has joined #openstack-infra | 21:55 | |
| *** e0ne_ has joined #openstack-infra | 21:55 | |
| clarkb | jeblair: what is the process there? drop the entire schema or are you going to do a manual migration? | 21:55 |
| jeblair | clarkb: i'll probably just alter table | 21:55 |
| clarkb | jeblair: may be good to write that up and update the thread about the breaking change on the ml | 21:55 |
| jeblair | clarkb: ++ | 21:56 |
| *** dcramer_ has quit IRC | 21:57 | |
| *** e0ne has quit IRC | 21:57 | |
| clarkb | ALTER TABLE nodepool.node CHANGE image label TEXT; ? (that is mostly from memory of what the schema looks like so don't copy pasta :) ) | 21:58 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Add 'labels' as a configuration primitive https://review.openstack.org/82147 | 21:58 |
| jeblair | alter table node change image_name label_name varchar(255) not null; | 22:00 |
| jeblair | clarkb: close | 22:00 |
| *** ominakov has quit IRC | 22:01 | |
| *** pdmars has quit IRC | 22:02 | |
| *** prad has quit IRC | 22:02 | |
| *** e0ne_ has quit IRC | 22:03 | |
| *** e0ne has joined #openstack-infra | 22:04 | |
| openstackgerrit | A change was merged to openstack-infra/config: Use labels in nodepool config https://review.openstack.org/83618 | 22:05 |
| *** wchrisj has quit IRC | 22:06 | |
| *** bknudson has quit IRC | 22:06 | |
| jeblair | ok here goes | 22:07 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Fix image/label name typo in stats https://review.openstack.org/83912 | 22:11 |
| jeblair | clarkb: whoops ^; that's in production now | 22:11 |
| *** mwagner_bbl has joined #openstack-infra | 22:11 | |
| *** dcramer_ has joined #openstack-infra | 22:11 | |
| clarkb | jeblair: ok, will review | 22:11 |
| clarkb | jeblair: I also notice in the debug log the AllocationsGrantTarget lines all show 0 despite having nodes granted | 22:12 |
| jeblair | that sounds problematic | 22:13 |
| clarkb | ya not sure if just a logging bug or an actual bug yet | 22:13 |
| jeblair | it certainly is not building many | 22:14 |
| jeblair | clarkb: the nodes it is building _did_ show up in allocationgranttarget lines | 22:14 |
| jeblair | i lean toward the logging being correct | 22:14 |
| *** khyati has joined #openstack-infra | 22:15 | |
| clarkb | jeblair: allocation = max(allocation, 0) must be coming up as 0? | 22:15 |
| clarkb | jeblair: ya I think logging is correct | 22:15 |
| clarkb | the __repr__ line looked good to me | 22:15 |
| *** _nadya_ has joined #openstack-infra | 22:18 | |
| *** mrodden has quit IRC | 22:20 | |
| *** marun has quit IRC | 22:21 | |
| clarkb | jeblair: I think it may be possibly for total_min_ready to be 0 (false) which causes the ration to be 0.0 and delta to be negative making max(allocation, 0) the max calculation between a negative number and 0 | 22:21 |
| *** wenlock has joined #openstack-infra | 22:23 | |
| *** marun has joined #openstack-infra | 22:23 | |
| *** andreaf has quit IRC | 22:24 | |
| jeblair | clarkb: are you looking in the allocator or in nodepool? | 22:25 |
| clarkb | jeblair: allocator | 22:26 |
| clarkb | jeblair: in makeAllocations | 22:26 |
| *** khyati has quit IRC | 22:26 | |
| jeblair | clarkb: but total_min_ready isn't a direct input there | 22:28 |
| *** mgagne has quit IRC | 22:28 | |
| clarkb | jeblair: ya, so over in nodepool.py is getNeededNodes() loop on line 1058 doing the right thing? | 22:28 |
| *** SumitNaiksatam has quit IRC | 22:28 | |
| clarkb | for image in target.images.values(): <- specifically that loop | 22:29 |
| clarkb | target.images was the old way of doing things | 22:29 |
| jeblair | clarkb: i think it's something else; patch in a sec | 22:30 |
| *** _nadya_ has quit IRC | 22:33 | |
| flashgordon | sdague: I am definitly over thinking it | 22:33 |
| flashgordon | but its working nicely | 22:34 |
| sdague | flashgordon: yeh, I just fear putting so much effort into rules up front, vs. letting the thing naturally evolve, as I think there will be a lot of backing stuff out later | 22:34 |
| sdague | or it just constraining what can be done | 22:35 |
| jeblair | clarkb: i can't find that line of code | 22:35 |
| flashgordon | sdague: perhaps | 22:35 |
| clarkb | jeblair: line 1062 in nodepool.py | 22:35 |
| *** _nadya_ has joined #openstack-infra | 22:35 | |
| *** markwash_ has joined #openstack-infra | 22:35 | |
| flashgordon | I am just looking for a list of h1 and h2 (and h3) html tags | 22:35 |
| flashgordon | for starters we can gate on h1 is all there | 22:36 |
| *** mrodden has joined #openstack-infra | 22:36 | |
| sdague | sure, it just seems like if the whole thing makes sense to a human, without one of those, it shouldn't be machine penalized | 22:36 |
| clarkb | jeblair: er, why is my code different than cgits? | 22:36 |
| clarkb | bah I didn't pull I only fetch one sec | 22:37 |
| clarkb | jeblair: sorry for the noise I was one commit behind | 22:37 |
| *** markwash has quit IRC | 22:37 | |
| *** markwash_ is now known as markwash | 22:37 | |
| flashgordon | the concern is we may miss sections | 22:38 |
| flashgordon | its hard to say that doesn't make sense because section x is missing | 22:38 |
| *** afazekas has quit IRC | 22:42 | |
| *** esp has joined #openstack-infra | 22:42 | |
| *** rediskin has left #openstack-infra | 22:43 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev https://review.openstack.org/61321 | 22:43 |
| esp | hey folks, got a gerrit question regarding changing preferred email addresses | 22:44 |
| *** Sukhdev has quit IRC | 22:44 | |
| mordred | esp: sup/ | 22:45 |
| mordred | ? | 22:45 |
| esp | hey mordred, this is rather silly | 22:45 |
| mordred | esp: I love silly | 22:45 |
| esp | so if I change my preferred email address in launchpad, it gets reflected in gerrit correct? | 22:45 |
| esp | but it looks like if I started a patch using the older address | 22:46 |
| esp | and then try to amend the commit this doesn’t work so well. | 22:46 |
| openstackgerrit | Andrey Kurilin proposed a change to openstack/requirements: Add junitxml to global requirements https://review.openstack.org/83903 | 22:46 |
| esp | mordred: err goes something like this committer email address <my old email> does not match your user account. | 22:47 |
| mordred | esp: nope - all email address information is managed in gerrit | 22:47 |
| mordred | esp: https://review.openstack.org/#/settings/contact | 22:47 |
| esp | mordred: kk | 22:47 |
| mordred | or - wait | 22:48 |
| esp | mordred: let me try to register the old one in there and see if works again.. | 22:48 |
| mordred | esp: you might need to try git commit --amend --reset-author | 22:48 |
| * esp takes a sip of coffee | 22:48 | |
| esp | mordred: k, let me give it a go | 22:49 |
| clarkb | jeblair: now that I have the right code in my editor, I wonder if the delta is calculated wrong | 22:50 |
| esp | phew, thanks mordred! | 22:50 |
| clarkb | ratio * total_current is what we currently have | 22:50 |
| esp | mordred: that did the trick.. I should write that down.. | 22:51 |
| *** packet has quit IRC | 22:51 | |
| mordred | esp: w00t | 22:51 |
| clarkb | it doesn't really give us a desired count because nodepool.py:1125 is setting current from the length of number of nodes we currenlty have which makes sense | 22:51 |
| esp | have a good one mordred! | 22:52 |
| clarkb | jeblair: I think desired count should equal round(ratio * amount) | 22:52 |
| mordred | esp: you too! | 22:52 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Fix the allocation distribution https://review.openstack.org/83923 | 22:52 |
| clarkb | jeblair: oh wait total_current is += to amount so that is covered | 22:52 |
| jeblair | clarkb: ^ take a look at that... | 22:52 |
| *** esp has left #openstack-infra | 22:52 | |
| clarkb | jeblair: why would length of targets change while iterating? | 22:53 |
| clarkb | er I don't grok why the old one didn't work but the new one looks good | 22:53 |
| clarkb | we basically divide allotments based on what hasn't been alloted rather than the global scope of things | 22:54 |
| jeblair | clarkb: yeah -- if you look at the earlier change, line 244: https://review.openstack.org/#/c/82147/7/nodepool/allocation.py | 22:54 |
| jeblair | clarkb: that had a similar effect of ensuring that by the end of the list, ratio was 1.0 | 22:55 |
| jeblair | clarkb: so this change tries to do the same thing | 22:55 |
| jeblair | clarkb: (we could get something similar by removing the line "total_current -= agt.request_target.current" which has the advantage of scaling the ratio in less weird ways, but i don't think it would converge on 1.0 in that case, so you could still end up dropping nodes) | 22:56 |
| clarkb | ya | 22:56 |
| clarkb | this way eventually your ratio goes to 1 | 22:56 |
| clarkb | which may mean one jenkins master gets slightly more nodes | 22:57 |
| clarkb | but should be ok in the big picture of things | 22:57 |
| *** thuc has quit IRC | 22:57 | |
| *** jergerber has quit IRC | 22:57 | |
| jeblair | yep. it doesn't behave quite as well in the case of massive differences in node numbers as i would like (it will eventually settle, but it might do some pretty big swings along the way). but i think that's what we had before. | 22:58 |
| jeblair | so i'm aiming for getting that back, and maybe with more unit tests we can tweak it later | 22:58 |
| jeblair | and fwiw, that change still passes the allocator unit tests i wrote so far | 22:59 |
| clarkb | :/ | 22:59 |
| *** thuc_ has joined #openstack-infra | 23:00 | |
| jeblair | okay, restarting with those 2 changes now | 23:01 |
| jeblair | it has, in fact, decided to launch many many nodes. | 23:01 |
| jeblair | which, on balance, is probably better than none | 23:01 |
| clarkb | ++ | 23:02 |
| jeblair | the new allocator output looks good | 23:03 |
| clarkb | yup | 23:04 |
| clarkb | Exception: Server 315d8373-e841-4a3b-b0b7-5a86d62288ef for node id: 3305532 status: ERROR <- is that the hpcloud east failures? | 23:04 |
| jesusaurus | probably not. the east failures are packet loss | 23:05 |
| jeblair | ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-83b693f2-b1c2-4cfe-9813-7658c2f470dc) | 23:05 |
| jeblair | is that? ^ | 23:05 |
| clarkb | jeblair: right if a node doesn't get networkign wouldn't it go itno an error state? | 23:06 |
| *** SlickNik is now known as gerrit | 23:06 | |
| jeblair | clarkb: i dunno? | 23:06 |
| *** paul-- has joined #openstack-infra | 23:06 | |
| clarkb | jeblair: sorry that was for jesusaurus | 23:06 |
| clarkb | tab complete failed me | 23:06 |
| *** gerrit is now known as Guest35494 | 23:06 | |
| *** Guest35494 is now known as SlickNik | 23:07 | |
| *** zns has quit IRC | 23:07 | |
| jesusaurus | clarkb: no, it would be running and the nic would be linked, but not all packets would make it to the other end | 23:07 |
| jesusaurus | so, from a nova perspective, everything would look fine, not ERROR | 23:08 |
| clarkb | jesusaurus: fwiw we have seen other issues like a high error rate in east | 23:08 |
| jesusaurus | hmm | 23:08 |
| clarkb | that apparently are nto related to the networking | 23:08 |
| jesusaurus | might be worth mentioning to support if you're seeing that | 23:08 |
| jeblair | | tripleo-test-cloud | reviewstats | 7847395e-c9d6-40d5-b880-050e70c9151d | None | | 23:08 |
| jeblair | what's that? | 23:08 |
| clarkb | jeblair: I think lifeless was going to host reviewstats on tripleo-test-cloud but russellb resurrected it | 23:10 |
| clarkb | jeblair: I assume you are looking at an alien list | 23:10 |
| clarkb | and ideally our tenant wouldn't be used for things like that | 23:10 |
| lifeless | jeblair: oh yeah, delete that | 23:11 |
| jeblair | lifeless: please don't host random stuff in the openstack-infra tenant of the tripleo test cloud | 23:11 |
| lifeless | jeblair: sure, sorry. te-broker is currently in that category | 23:11 |
| lifeless | jeblair: its not however random | 23:11 |
| *** CaptTofu has quit IRC | 23:12 | |
| jeblair | lifeless: yeah | 23:12 |
| lifeless | jeblair: you have two tenants in the cloud btw - nodepool and infra | 23:12 |
| *** CaptTofu has joined #openstack-infra | 23:12 | |
| *** dhellmann is now known as dhellmann_ | 23:13 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Fix image/label name typo in stats https://review.openstack.org/83912 | 23:14 |
| jeblair | lifeless: ack; i see creds for both. i was looking at the nodepool one. | 23:14 |
| *** david-lyle has quit IRC | 23:16 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Lower tripleo max-servers to 35 https://review.openstack.org/83932 | 23:16 |
| *** CaptTofu has quit IRC | 23:16 | |
| jeblair | restarted puppet on nodepool | 23:21 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Fix the allocation distribution https://review.openstack.org/83923 | 23:22 |
| *** melwitt has quit IRC | 23:26 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 23:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Lower tripleo max-servers to 35 https://review.openstack.org/83932 | 23:28 |
| *** gyee has quit IRC | 23:31 | |
| *** zns has joined #openstack-infra | 23:36 | |
| *** pcrews_ has joined #openstack-infra | 23:39 | |
| *** sandywalsh has joined #openstack-infra | 23:40 | |
| *** pcrews has quit IRC | 23:42 | |
| *** thuc_ has quit IRC | 23:44 | |
| *** CaptTofu has joined #openstack-infra | 23:44 | |
| *** thuc has joined #openstack-infra | 23:44 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Upgraded jasmine https://review.openstack.org/83936 | 23:47 |
| *** thuc has quit IRC | 23:49 | |
| *** gokrokve has quit IRC | 23:49 | |
| *** gokrokve has joined #openstack-infra | 23:50 | |
| *** jhesketh_ has quit IRC | 23:54 | |
| *** julim has joined #openstack-infra | 23:54 | |
| *** gokrokve has quit IRC | 23:54 | |
| *** leakypipes has quit IRC | 23:55 | |
| *** leakypipes has joined #openstack-infra | 23:55 | |
| fungi | sorry, got pulled into more packing/cleaning after dinner, back now | 23:57 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added unit tests for string_util. https://review.openstack.org/83938 | 23:58 |
| fungi | clarkb: pypy situation is Alex_Gaynor is circling back around to find out whether we can get fixed/updated pypy packages which solve the issue, i think. no new movement on openstack/pip/debian bug reports recently | 23:58 |
| fungi | clarkb: working through the rest of the nodepool docs completion change now | 23:58 |
| *** homeless has quit IRC | 23:58 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
