| *** ^demon|away has quit IRC | 00:05 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: updated htmlifier for safety and features https://review.openstack.org/38568 | 00:06 |
|---|---|---|
| sdague | clarkb: you want to take a spin on that | 00:06 |
| sdague | I got rid of the glaring security hole, plus cleaned up other things | 00:07 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: updated htmlifier for safety and features https://review.openstack.org/38568 | 00:15 |
| clarkb | ya reviewing now | 00:16 |
| *** ewindisch has joined #openstack-infra | 00:17 | |
| *** mriedem1 has joined #openstack-infra | 00:24 | |
| *** mriedem has quit IRC | 00:25 | |
| mikal | clarkb: openstack.stillhq.com/reviews/report.html | 00:27 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Conditionally install Python 3 development libs. https://review.openstack.org/38103 | 00:27 |
| clarkb | fungi: jeblair ^ want to review sdague's change really quick? if not I am going to approve it | 00:28 |
| fungi | i'll take a quick peek | 00:28 |
| *** erfanian has joined #openstack-infra | 00:29 | |
| clarkb | ty | 00:30 |
| fungi | sdague: "Pull out a save path from a url." was that a typo of "safe?" i'm going to assume so, otherwise i'm not sure what the word means in that context | 00:32 |
| clarkb | fungi: it is a typo | 00:33 |
| sdague | typo | 00:33 |
| sdague | but I need to go cook dinner now, so won't be able to fix that until later | 00:33 |
| clarkb | sdague: thats fine I am happy to merge as is | 00:33 |
| clarkb | sdague: I left some other improvement comments but nothing not worth merging over | 00:34 |
| fungi | i don't mind the typo, just asking so i understand whati'm revewing | 00:34 |
| *** anteaya has quit IRC | 00:36 | |
| fungi | sdague: in safe_path() you're passing in root='/srv/static/logs/' from application() but then building newpath = os.path.abspath("%s/%s" % (root, raw)) | 00:37 |
| fungi | so you get abspath('/srv/static/logs//something') there | 00:37 |
| fungi | the double-slash should get cleaned up by abspath, but want to make sure that's intentional | 00:38 |
| clarkb | fungi: abspath does clean it up. I am sure it is not intentional. Its probably worht a comment | 00:38 |
| fungi | if it's there to allow root to be optionally non-terminated then that's probably okay, but it's safest to always terminate the root path with a / so the find provides a more explicit result | 00:40 |
| clarkb | fungi: if you refresh I have left a comment that os.path.join would be beneficial there | 00:41 |
| clarkb | fungi: I think using os.path.join will make you happy and prevent double slashes | 00:41 |
| sdague | it's actually a little intentional | 00:42 |
| *** mjfork has quit IRC | 00:42 | |
| fungi | i think passing in /srv/static/logs (without the slash) is probably a bad idea though, so catering to the possibility is questionable | 00:42 |
| fungi | though in most cases it's unlikely to lead to anything exploitable | 00:43 |
| sdague | yeh, that's why I passed it with the /, and ran it through abspath to clean | 00:43 |
| clarkb | fungi: correct I think root should have a trailing slash. | 00:43 |
| sdague | first thing went on the grill... so able to chat for a minute :) | 00:43 |
| fungi | the risk is a bit pathological, but to quote futurama out of context, "technically correct is the best kind of correct" | 00:44 |
| sdague | heh | 00:44 |
| sdague | futurama is one of my favorite shows ever | 00:45 |
| fungi | mine as well | 00:45 |
| clarkb | if I could buy slurm and man chow I probably would | 00:45 |
| fungi | BACHELOR chow | 00:45 |
| clarkb | thats what it was called | 00:45 |
| fungi | i preferred soylent cola (made from the best stuff on earth: people!) | 00:46 |
| fungi | anyway, dropping the modem again for more conferencing | 00:46 |
| Ryan_Lane | I'm upgrading mediawiki on wiki.openstack.org | 00:46 |
| clarkb | fungi: you ok with me approving? | 00:46 |
| fungi | clarkb: yeah, should be safe enough | 00:46 |
| * clarkb hits the go button | 00:47 | |
| fungi | +2'd | 00:47 |
| *** ianw has quit IRC | 00:47 | |
| sdague | I'm pretty sure - http://shop.naturespath.com/Panda-Puffs-Cereal/p/NPA-860044&c=NaturesPath@EnviroKidz is basically bachelor chow | 00:47 |
| clarkb | sdague: approved | 00:47 |
| clarkb | Ryan_Lane: gl hf | 00:47 |
| sdague | at least one guy in the office treated it as such for about 2 months | 00:47 |
| *** ianw has joined #openstack-infra | 00:48 | |
| openstackgerrit | A change was merged to openstack-infra/config: updated htmlifier for safety and features https://review.openstack.org/38568 | 00:49 |
| * clarkb runs home between before puppet can apply that | 00:51 | |
| *** rfolco has quit IRC | 00:54 | |
| *** nati_ueno has quit IRC | 00:54 | |
| *** alexpilotti has quit IRC | 01:00 | |
| *** sgviking has quit IRC | 01:00 | |
| *** sgviking has joined #openstack-infra | 01:01 | |
| *** ewindisch has quit IRC | 01:02 | |
| *** gyee has quit IRC | 01:02 | |
| *** yamahata has quit IRC | 01:04 | |
| *** nati_ueno has joined #openstack-infra | 01:07 | |
| *** markmcclain has quit IRC | 01:08 | |
| *** ewindisch has joined #openstack-infra | 01:15 | |
| *** nati_ueno has quit IRC | 01:18 | |
| *** yamahata has joined #openstack-infra | 01:20 | |
| *** ewindisch has quit IRC | 01:29 | |
| *** UtahDave has quit IRC | 01:31 | |
| *** lbragstad has quit IRC | 01:52 | |
| *** lbragstad has joined #openstack-infra | 01:53 | |
| *** pcrews has quit IRC | 01:56 | |
| *** ewindisch has joined #openstack-infra | 02:06 | |
| *** olaph has quit IRC | 02:12 | |
| *** olaph has joined #openstack-infra | 02:14 | |
| clarkb | just as I sit down for dinner docs jobs go unstable again. /me looks into it | 02:16 |
| *** mriedem1 has quit IRC | 02:21 | |
| clarkb | umounted, fscked clean, mounted again seems to be ok | 02:21 |
| clarkb | it was a different error this time around | 02:21 |
| clarkb | jeblair: fungi ^ you may want to look at dmesg on static.o.o | 02:22 |
| clarkb | and now time for dinner | 02:22 |
| *** ewindisch has quit IRC | 02:29 | |
| *** HenryG has quit IRC | 03:02 | |
| *** yaguang has joined #openstack-infra | 03:07 | |
| *** yaguang has quit IRC | 03:16 | |
| *** cody-somerville has joined #openstack-infra | 03:18 | |
| *** dkliban has quit IRC | 03:26 | |
| *** yaguang has joined #openstack-infra | 03:30 | |
| *** changbl has quit IRC | 03:30 | |
| *** jjmb has joined #openstack-infra | 03:34 | |
| *** jjmb1 has joined #openstack-infra | 03:34 | |
| *** jjmb has quit IRC | 03:38 | |
| *** pcrews has joined #openstack-infra | 03:39 | |
| *** dkliban has joined #openstack-infra | 03:40 | |
| *** dkliban has quit IRC | 03:56 | |
| *** erfanian has quit IRC | 03:56 | |
| *** changbl has joined #openstack-infra | 04:14 | |
| *** yaguang has quit IRC | 04:16 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:22 | |
| *** changbl has quit IRC | 04:22 | |
| *** changbl has joined #openstack-infra | 04:24 | |
| *** yaguang has joined #openstack-infra | 04:29 | |
| *** cody-somerville has quit IRC | 04:32 | |
| *** nati_ueno_2 has quit IRC | 04:34 | |
| *** nati_ueno has joined #openstack-infra | 04:35 | |
| *** afazekas has joined #openstack-infra | 04:35 | |
| *** changbl has quit IRC | 04:37 | |
| *** chuckieb|2 has joined #openstack-infra | 04:40 | |
| *** koolhead17 has joined #openstack-infra | 04:42 | |
| *** chuckieb has quit IRC | 04:43 | |
| *** lbragstad1 has joined #openstack-infra | 05:08 | |
| *** lbragstad has quit IRC | 05:08 | |
| * mordred throws a goat at clarkb | 05:25 | |
| clarkb | mordred: the openstack party has goats? if I had known that | 05:26 |
| mordred | clarkb: sOOO many goats | 05:26 |
| * Alex_Gaynor has linkedin endorsements for goats | 05:27 | |
| jesusaurus | i hear you can teach goats to openstack | 05:28 |
| mordred | 7mmm | 05:28 |
| mordred | loat | 05:28 |
| mordred | gaot | 05:28 |
| mordred | dammit | 05:28 |
| *** nati_ueno_2 has joined #openstack-infra | 05:28 | |
| clarkb | wow | 05:29 |
| * mordred throws goats indiscriminaantly into the air | 05:29 | |
| clarkb | that many goats? | 05:29 |
| mordred | ALL the goats | 05:29 |
| clarkb | mordred: does this mean you IRC on your phone now? | 05:29 |
| mordred | I'm so done with the places with the alcohol | 05:29 |
| clarkb | mordred: also if you see fungi and/or jeblair it would be super duper if you could mention to them about the docs-draft derpage | 05:30 |
| mordred | now you've got to deal with me on IRC on a laptop | 05:30 |
| clarkb | mordred: oh and if you see spearce you should talk to him about our WIP needs | 05:30 |
| clarkb | mordred: and perhaps ask for ideas on making that upstreamable | 05:30 |
| mordred | not tonight | 05:30 |
| mordred | counterproductive | 05:30 |
| mordred | is spearce here? | 05:30 |
| clarkb | mordred: yes, he and Chris Dibona give the google open source update | 05:30 |
| clarkb | which at this point must be "We do GSoC and Gerrit" | 05:31 |
| clarkb | "oh and android is open source every 9 months or so" | 05:31 |
| * clarkb is cranky about not having a 4.3 image for his phone because verizon | 05:31 | |
| *** nati_ueno_2 has quit IRC | 05:32 | |
| *** nati_ueno has quit IRC | 05:32 | |
| *** nati_ueno has joined #openstack-infra | 05:32 | |
| * mordred lobs a goat at verizon | 05:34 | |
| * mordred hefts a scimitar at clarkb | 05:34 | |
| clarkb | mordred: when is the seattle thing with tripleo folks? | 05:37 |
| mordred | sept | 05:37 |
| mordred | clarkb: btw, jeblair and fungi and I had a good talk in person today about dib in d-g | 05:37 |
| mordred | clarkb: which was largely what was in that etherpad | 05:38 |
| clarkb | oh good. is the use case starting to gel? | 05:38 |
| mordred | yah | 05:38 |
| clarkb | I should check if they gave us beta access | 05:39 |
| clarkb | mordred: also tomorrow you should comp that account :) | 05:39 |
| mordred | clarkb: I've also got a half-done dib element for us | 05:39 |
| mordred | clarkb: bah | 05:39 |
| mordred | clarkb: tomorrow I have a talk | 05:39 |
| clarkb | have you written the talk yet? | 05:39 |
| mordred | clarkb: yup | 05:39 |
| clarkb | oh if you see anyone from mirantis you might want to mention that their stackalytics thing seems to ignore us despite being hosted on stackforge | 05:40 |
| mordred | clarkb: I saw everyone who works at mirantis | 05:41 |
| SergeyLukjanov | hey guys, I'm from Mirantis | 05:41 |
| SergeyLukjanov | I'll address your comments to stackalytics team | 05:42 |
| clarkb | SergeyLukjanov: hi there. I was looking at stackalytics earlier this evening and the openstack-infra/* repos seem to be missing | 05:42 |
| clarkb | granted we have a lot of them maybe it was too much work or something. In any case openstack-infra/config should probably be included | 05:42 |
| SergeyLukjanov | clarkb, mm, yep, see it too | 05:42 |
| * mordred hands SergeyLukjanov a cake | 05:42 | |
| clarkb | mordred: run out of goats? | 05:43 |
| * mordred knows that mirantis doesnt' like being handed goats | 05:43 | |
| * mordred thinks mirantis might be his favorite OS company | 05:43 | |
| SergeyLukjanov | the same story with openstack-dev projects | 05:44 |
| SergeyLukjanov | I'll talk today with Ilya Shakhat that runs the stackalytics project about extending number of monitored projects | 05:45 |
| clarkb | SergeyLukjanov: awesome, thanks | 05:46 |
| SergeyLukjanov | btw I have a small question about jobs that we can use in savanna) | 05:47 |
| SergeyLukjanov | i've proposed the change https://review.openstack.org/#/c/37987/1/modules/openstack_project/files/zuul/layout.yaml | 05:47 |
| SergeyLukjanov | it includes -docs jobs | 05:47 |
| SergeyLukjanov | is it ok to use them for savanna project? | 05:48 |
| clarkb | SergeyLukjanov: it is ok to use the gate-*-docs jobs in the check and gate queue. Typically it is not ok to use them in the post, pre-release and release queues as the *-docs jobs want to publish to docs.openstack.org whcih we are trying to keep for openstack projects | 05:48 |
| clarkb | SergeyLukjanov: however, We do have read the docs jobs that you can setup | 05:49 |
| clarkb | SergeyLukjanov: the read the docs jobs tell read the docs to update your docs whenever code merges. You shoudl eb able to see them in the layout file if you search for rtfd | 05:49 |
| SergeyLukjanov | clarkb, we are already using rtfd jobs | 05:49 |
| clarkb | oh I see that now, yeah. Today rtfd is probably the best options for stackforge projects | 05:50 |
| SergeyLukjanov | clarkb, I'll remove gate-*-docs from post/release | 05:50 |
| SergeyLukjanov | so, it'll be able to see on docs w/o merging or manually building them for patches to docs | 05:51 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add more jobs for Savanna projects https://review.openstack.org/37987 | 05:55 |
| SergeyLukjanov | I've removed gate-savanna-docs has been removed from post/pre-release/release, could you please take a look at this patch set? | 05:58 |
| *** boris-42 has joined #openstack-infra | 06:01 | |
| *** SergeyLukjanov has quit IRC | 06:03 | |
| *** melwitt has quit IRC | 06:10 | |
| clarkb | yup plan to tomorrow morning but now sleep | 06:20 |
| *** nati_ueno_2 has joined #openstack-infra | 06:26 | |
| *** afazekas is now known as afazekas__ | 06:27 | |
| *** nati_ueno has quit IRC | 06:29 | |
| *** odyssey4me has joined #openstack-infra | 06:34 | |
| Vivek | mordred: Hi | 06:34 |
| Vivek | mordred: I fell asleep last night. | 06:35 |
| *** dansmith has quit IRC | 06:38 | |
| *** dansmith has joined #openstack-infra | 06:39 | |
| *** koolhead17 has quit IRC | 06:42 | |
| *** boris-42 has quit IRC | 07:01 | |
| *** dkehn_ has joined #openstack-infra | 07:02 | |
| *** dkehn has quit IRC | 07:03 | |
| *** hashar has joined #openstack-infra | 07:08 | |
| *** EmilienM has quit IRC | 07:11 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:12 | |
| *** EmilienM has joined #openstack-infra | 07:16 | |
| jeblair | clarkb, mordred, fungi: it appears dm-1 (docs-draft) has failed yet again (just about 20 minutes ago); i see clarkb fixed it a few hours ago | 07:18 |
| jeblair | i think that's three times that the same lv has failed; lvm says that it is on a single pv, so i'm going to add a new cinder volume and move it, deprecating the current pv that it's on | 07:18 |
| jeblair | lvs --segments -o +pe_ranges | 07:24 |
| jeblair | that shows which pvs the lv is on | 07:24 |
| jeblair | currently running pvmove -n docs-draft /dev/xvdh1 /dev/xvdi1 | 07:24 |
| jeblair | also unmounted and fscking the partition | 07:25 |
| *** hashar_ has joined #openstack-infra | 07:29 | |
| openstackgerrit | A change was merged to openstack-infra/config: Don't remove lost+found from static.o.o. https://review.openstack.org/37805 | 07:31 |
| jeblair | reading the lost+found directory triggers the error | 07:32 |
| *** hashar has quit IRC | 07:33 | |
| *** hashar_ is now known as hashar | 07:33 | |
| *** Ryan_Lane has quit IRC | 07:33 | |
| *** nayward has joined #openstack-infra | 07:35 | |
| jeblair | nm, different error. | 07:39 |
| *** jpich has joined #openstack-infra | 07:39 | |
| *** rpodolyaka has left #openstack-infra | 07:42 | |
| jeblair | repeated invocations of 'fsck -f' without mounting continue to try to correct similar errors; i think we may need a new filesystem as well | 07:44 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add progress and estimated completion times to zuul status https://review.openstack.org/38598 | 07:48 |
| jeblair | okay, i ran it 3 times and the most recent time it corrected no errors | 07:49 |
| jeblair | so i'm going to remount it and not touch lost+found | 07:49 |
| jeblair | after the pvmove is done (in a few hours) and we remove the pv that encountered timeouts, we may want to consider making a new filesystem on a new lv and starting an rsync | 07:50 |
| jeblair | docs-draft is remounted | 07:51 |
| *** SergeyLukjanov has quit IRC | 07:52 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:54 | |
| *** johnthetubaguy has joined #openstack-infra | 08:00 | |
| *** mjura has joined #openstack-infra | 08:01 | |
| *** mjura has left #openstack-infra | 08:03 | |
| *** mjura_ has joined #openstack-infra | 08:03 | |
| *** mjura_ has quit IRC | 08:04 | |
| *** boris-42 has joined #openstack-infra | 08:06 | |
| *** SergeyLukjanov has quit IRC | 08:32 | |
| *** SergeyLukjanov has joined #openstack-infra | 08:36 | |
| *** derekh has joined #openstack-infra | 08:39 | |
| *** nati_ueno_2 has quit IRC | 09:01 | |
| *** amotoki has joined #openstack-infra | 09:28 | |
| *** afazekas__ has quit IRC | 09:51 | |
| *** markmc has joined #openstack-infra | 10:01 | |
| markmc | is the wiki in read-only mode or something? | 10:02 |
| * markmc sees | 10:02 | |
| markmc | 2013-07-25T00:46:46 <Ryan_Lane> I'm upgrading mediawiki on wiki.openstack.org | 10:02 |
| markmc | I don't see a login button today, basically | 10:02 |
| jpich | markmc: Hm... I can't see the button either but accessing https://wiki.openstack.org/w/index.php?title=Special:OpenIDLogin directly works | 10:03 |
| markmc | jpich, super, thanks | 10:03 |
| * markmc filed https://bugs.launchpad.net/openstack-ci/+bug/1204841 | 10:05 | |
| uvirtbot | Launchpad bug 1204841 in openstack-ci "No login button on wiki" [Undecided,New] | 10:05 |
| *** jhesketh__ has quit IRC | 10:06 | |
| *** jhesketh__ has joined #openstack-infra | 10:06 | |
| openstackgerrit | Thomas Goirand proposed a change to openstack/requirements: Adds oslo.sphinx https://review.openstack.org/38617 | 10:21 |
| *** hashar has quit IRC | 10:22 | |
| *** fbo_away is now known as fbo | 10:25 | |
| *** yaguang has quit IRC | 10:29 | |
| *** yaguang has joined #openstack-infra | 10:41 | |
| sdague | bummer that the wiki isn't under puppet control | 10:43 |
| *** yaguang has quit IRC | 10:46 | |
| *** afazekas has joined #openstack-infra | 10:51 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Pathignore plugin support added https://review.openstack.org/37909 | 10:56 |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: HTML Publisher plugin support added https://review.openstack.org/37876 | 10:56 |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: TextFinder plugin support added https://review.openstack.org/37849 | 10:56 |
| *** afazekas has quit IRC | 10:57 | |
| *** yaguang has joined #openstack-infra | 10:58 | |
| *** yaguang has quit IRC | 10:58 | |
| *** weshay has joined #openstack-infra | 11:00 | |
| *** dprince has joined #openstack-infra | 11:01 | |
| *** afazekas has joined #openstack-infra | 11:12 | |
| *** johnthetubaguy has quit IRC | 11:13 | |
| *** hashar has joined #openstack-infra | 11:27 | |
| sdague | jeblair: did you have any further thoughts on the openstack-qa mail alias? | 11:31 |
| *** alexpilotti has joined #openstack-infra | 11:32 | |
| *** sungju_ has joined #openstack-infra | 11:34 | |
| *** alexpilotti has quit IRC | 11:42 | |
| *** jjmb1 has quit IRC | 11:43 | |
| *** pabelanger has quit IRC | 11:44 | |
| *** rfolco has joined #openstack-infra | 11:58 | |
| *** boris-42 has quit IRC | 12:01 | |
| *** boris-42 has joined #openstack-infra | 12:03 | |
| *** bnemec has quit IRC | 12:04 | |
| *** sungju_ has quit IRC | 12:06 | |
| *** SergeyLukjanov has quit IRC | 12:07 | |
| *** dprince has quit IRC | 12:07 | |
| *** bnemec has joined #openstack-infra | 12:07 | |
| *** prad has quit IRC | 12:10 | |
| *** dhellman_ has joined #openstack-infra | 12:14 | |
| *** dkranz has quit IRC | 12:15 | |
| *** prad has joined #openstack-infra | 12:17 | |
| *** novel1 is now known as mirrorbox | 12:20 | |
| *** mirrorbox has joined #openstack-infra | 12:20 | |
| *** dkliban has joined #openstack-infra | 12:23 | |
| *** woodspa has joined #openstack-infra | 12:25 | |
| openstackgerrit | will soula proposed a change to openstack-infra/jenkins-job-builder: Adding the Sauce OnDemand Plugin https://review.openstack.org/37947 | 12:30 |
| *** derekh has quit IRC | 12:37 | |
| *** jergerber has joined #openstack-infra | 12:37 | |
| *** johnthetubaguy has joined #openstack-infra | 12:43 | |
| jd__ | does that error raises a bell http://logs.openstack.org/40/38540/1/check/gate-pbr-devstack-vm-rawinstall/296/console.html ? (mordred maybe?) | 12:44 |
| *** derekh has joined #openstack-infra | 12:47 | |
| *** HenryG has joined #openstack-infra | 12:47 | |
| *** cppcabrera has joined #openstack-infra | 12:48 | |
| *** cppcabrera has left #openstack-infra | 12:48 | |
| *** alexpilotti has joined #openstack-infra | 12:49 | |
| *** mriedem has joined #openstack-infra | 12:49 | |
| *** anteaya has joined #openstack-infra | 12:50 | |
| *** jergerber has quit IRC | 12:51 | |
| *** mgagne has joined #openstack-infra | 12:53 | |
| *** pabelanger has joined #openstack-infra | 12:59 | |
| *** prad has quit IRC | 13:05 | |
| *** dkliban has quit IRC | 13:06 | |
| *** changbl has joined #openstack-infra | 13:08 | |
| *** ewindisch has joined #openstack-infra | 13:10 | |
| *** dprince has joined #openstack-infra | 13:11 | |
| *** boris-42 has quit IRC | 13:13 | |
| *** alexpilotti_ has joined #openstack-infra | 13:17 | |
| *** dims has joined #openstack-infra | 13:19 | |
| *** alexpilotti has quit IRC | 13:20 | |
| *** alexpilotti_ is now known as alexpilotti | 13:20 | |
| *** dkehn_ is now known as dkehn | 13:25 | |
| *** pentameter has joined #openstack-infra | 13:28 | |
| *** amotoki has quit IRC | 13:29 | |
| *** vogxn has joined #openstack-infra | 13:32 | |
| *** markmcclain has joined #openstack-infra | 13:46 | |
| *** vijendar has joined #openstack-infra | 13:48 | |
| *** burt has joined #openstack-infra | 13:49 | |
| *** chuckieb|2 has quit IRC | 13:49 | |
| *** prad has joined #openstack-infra | 13:50 | |
| *** leifmadsen has quit IRC | 13:51 | |
| *** changbl has quit IRC | 13:52 | |
| *** michchap has quit IRC | 13:53 | |
| openstackgerrit | A change was merged to openstack/requirements: Updated minimal version of pymongo to 2.4. https://review.openstack.org/38306 | 13:53 |
| *** dims has quit IRC | 13:58 | |
| *** leifmadsen has joined #openstack-infra | 14:02 | |
| *** _TheDodd_ has joined #openstack-infra | 14:05 | |
| jd__ | dhellmann: do you happen to know why https://review.openstack.org/#/c/38072/ failed? I'm having the same issues on a pbr patchset I sent | 14:06 |
| *** vogxn has quit IRC | 14:08 | |
| *** dkliban has joined #openstack-infra | 14:08 | |
| *** xchu has joined #openstack-infra | 14:08 | |
| *** HenryG has quit IRC | 14:09 | |
| *** datsun180b has joined #openstack-infra | 14:15 | |
| *** datsun180b has joined #openstack-infra | 14:15 | |
| *** avtar has joined #openstack-infra | 14:17 | |
| jpich | jd__: FWIW today I had issues with setuptools locally as well and only managed to "fix" them by using the openstack pypi mirror in order to avoid the 404. I'm surprised this job is calling to upstream pypi | 14:18 |
| jd__ | jpich: equally :) | 14:18 |
| *** datsun180b_ has joined #openstack-infra | 14:20 | |
| *** HenryG has joined #openstack-infra | 14:21 | |
| *** datsun180b has quit IRC | 14:24 | |
| *** datsun180b_ is now known as datsun180b | 14:24 | |
| *** jjmb has joined #openstack-infra | 14:28 | |
| Shrews | As did I. I think a new version is out that hasn't yet propogated to all of the mirrors. Although it may have by now. | 14:28 |
| Shrews | re: setuptools, that is | 14:29 |
| *** dkranz has joined #openstack-infra | 14:29 | |
| *** cp16net is now known as cp16net|away | 14:31 | |
| mordred | jd__: that's failing building the mirror that it uses to test that building a mirror works | 14:43 |
| jd__ | mordred: why is that part of pbr tests? | 14:44 |
| mordred | scuse me, misspoke. it's building the mirror that it uses to test that installing all of the projects works. | 14:45 |
| *** mjfork has joined #openstack-infra | 14:45 | |
| mordred | which is does so that it can inject the currently tested version of pbr into that mirror | 14:45 |
| mordred | so that we can test that different install scenarios haven't gone bad | 14:46 |
| mordred | we don't normally have that many problems in the pbr test - also because pbr doesn't get nearly as many patches as the rest of the project... but if it stays a problem, we might have to get cleverer | 14:46 |
| jd__ | ok, that makes more sense :) | 14:49 |
| *** SergeyLukjanov has joined #openstack-infra | 14:50 | |
| *** changbl has joined #openstack-infra | 14:50 | |
| mordred | actually - I see a bug in the integration script now that pip 1.4 is out... | 14:50 |
| *** jjmb has quit IRC | 14:55 | |
| *** xchu has quit IRC | 14:58 | |
| *** _TheDodd_ has quit IRC | 15:03 | |
| *** _TheDodd_ has joined #openstack-infra | 15:04 | |
| *** changbl has quit IRC | 15:05 | |
| *** mkoderer has joined #openstack-infra | 15:08 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Stop trying to install setuptools<0.7 https://review.openstack.org/38647 | 15:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Install jeepyb and requirements from REPODIR https://review.openstack.org/38184 | 15:08 |
| mkoderer | Hi folks, I am not able to login to the OpenStack Wiki | 15:08 |
| mkoderer | seems the login button is gone ;) could someone have a look | 15:09 |
| Shrews | mordred: so does 38647 mean you must upgrade to pip 1.4 if you use pbr? | 15:10 |
| *** mrodden has quit IRC | 15:10 | |
| sdague | mordred: what's your thinking about unit test on python scripts in the config repo? | 15:11 |
| sdague | or if something is getting that complicated, it should come out into another repo? | 15:11 |
| mkoderer | sdague: I am not able to change the QA meeting wiki .. seems the wiki is somehow broken | 15:12 |
| mordred | Shrews: no. it means that that integration script installs things via pip for testing which is now 1.4 by default from upstream | 15:12 |
| sdague | mkoderer: https://bugs.launchpad.net/openstack-ci/+bug/1204841 | 15:12 |
| uvirtbot | Launchpad bug 1204841 in openstack-ci "No login button on wiki" [Undecided,Confirmed] | 15:12 |
| mkoderer | sdague: great thanks! | 15:13 |
| mordred | Shrews: we DO want to get everyone to upgrade to 1.4 though, because it fixes a bunch of issues, and also allows us to safely upload pre-release software to pypi without breaking peope | 15:13 |
| mordred | people | 15:13 |
| mordred | but we have not gotten to that point yet | 15:13 |
| *** nayward has quit IRC | 15:13 | |
| mordred | sdague: yes. if they've gotten that complex, they should probably have their own home | 15:13 |
| *** CaptTofu has quit IRC | 15:13 | |
| mordred | sdague: although I could certainly be argued off of that stance - it's not a strong one | 15:14 |
| *** CaptTofu has joined #openstack-infra | 15:14 | |
| mordred | jd__: https://review.openstack.org/38647 should fix the pbr integration test gate | 15:14 |
| sdague | mordred: I think it's a fine stance | 15:14 |
| *** gyee has joined #openstack-infra | 15:15 | |
| sdague | I wonder if all the htmlifier filters in config should become a project together | 15:15 |
| sdague | subunit2html is in there as well, and not really unit tested | 15:15 |
| sdague | and I get scared when I make changes to that | 15:15 |
| mordred | sdague: so, subunit2html wants to move to subunit itself I believe | 15:15 |
| mordred | but clarkb wanted to clean it up before submitting it there | 15:15 |
| sdague | is subunit in git? | 15:16 |
| jd__ | mordred: cool I'll recheck once it's in then | 15:16 |
| *** changbl has joined #openstack-infra | 15:16 | |
| sdague | or does that mean having to fight with bzr to make submissions | 15:16 |
| mordred | htmlifier probably could be its own thing | 15:16 |
| mordred | sdague: fight with bzr | 15:16 |
| mordred | sdague: or, fight with lifeless to get him to move it to git | 15:16 |
| sdague | hmph. I actually had some more prettinesses I wanted to add to subunit2html | 15:17 |
| mordred | oh - well please do! | 15:17 |
| mordred | also, we could make it its own thing | 15:17 |
| sdague | so it's a little easier if it lives in a gerrit tree, honestly | 15:17 |
| mordred | it's certainly a generally useful program | 15:17 |
| mordred | sdague: I fully support making it its own gerrit tree with its own pypi releases | 15:18 |
| sdague | ok | 15:18 |
| *** mrodden has joined #openstack-infra | 15:18 | |
| *** cp16net|away is now known as cp16net | 15:18 | |
| sdague | it seems a little weird to push something like the openstack log htmlifier to pypi, as it's so specific though | 15:22 |
| mordred | is it? couldn't it be used to htmlify other logs? | 15:23 |
| sdague | it's pretty specific to the log format in OpenStack | 15:23 |
| mordred | hrm | 15:23 |
| mordred | k | 15:23 |
| sdague | I'm not sure I have a strong opinion realistically, I just don't want to be seen poluting with non generic stuff. | 15:24 |
| sdague | my real goal is just to get unit testing around it though, because I borked a format thing on my last update (not terribly, but it should be different than it is) | 15:25 |
| *** danger_fo_away is now known as danger_fo | 15:26 | |
| dansmith | there was something else about the postgres gate job that was different, but I can't remember what it is | 15:31 |
| dansmith | jeblair said something last week I think | 15:31 |
| jeblair | sdague: i think it might become the first part of a log submission, processing, and retrieval system, if that helps you think about how to structure the repo/project. | 15:31 |
| dansmith | I'm trying to reproduce a fail that only happens there, but just moving to postgres isn't enough, apparently | 15:31 |
| jeblair | dansmith: zeromq | 15:32 |
| dansmith | oh, hmm. | 15:32 |
| sdague | postgresl is running on zmq now? | 15:32 |
| jeblair | maybe | 15:32 |
| sdague | if so, that's news to me | 15:32 |
| jeblair | er, no that's the gate-tempest-devstack-vm-postgres-zeromq-full job | 15:32 |
| jeblair | dansmith: i was wrong, gate-tempest-devstack-vm-postgres is only DEVSTACK_GATE_TEMPEST=1 and DEVSTACK_GATE_POSTGRES=1 | 15:33 |
| dansmith | hmm, okay | 15:33 |
| *** gyee has quit IRC | 15:33 | |
| jeblair | dansmith: DEVSTACK_GATE_TEMPEST_FULL=1 is not set on gate-tempest-devstack-vm-postgres | 15:34 |
| *** gyee has joined #openstack-infra | 15:34 | |
| dansmith | jeblair: wait, what? meaning it's not a full tempest run? it takes long enough that it seems like it's a full run | 15:34 |
| dansmith | oh | 15:34 |
| dansmith | gate-tempest-devstack-vm-postgres-full is the test that fails for me | 15:34 |
| dansmith | which should be full and I don't see anything in the localrc about zmq | 15:35 |
| jeblair | dansmith: yep, DEVSTACK_GATE_TEMPEST=1 DEVSTACK_GATE_TEMPEST_FULL=1 DEVSTACK_GATE_POSTGRES=1 | 15:35 |
| *** krtaylor has quit IRC | 15:35 | |
| dansmith | well, then I can't reproduce it here so jenkins is clearly wrong :P | 15:35 |
| jeblair | dansmith: sorry if i've lead you astray :( | 15:35 |
| dansmith | jeblair: it's all good :) | 15:36 |
| clarkb | good morning | 15:38 |
| clarkb | jeblair: is the docs-draft pv situation good now? (eg you have migrated to a new pv and removed the old one?) | 15:38 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Add a CDR log file https://review.openstack.org/38651 | 15:38 |
| clarkb | reducing the d-g pool size does seem to have made the slave situation much better. I haven't caught any more occurences of the double test run and premature node delete | 15:41 |
| jeblair | clarkb: yes, i have removed the pv from the system and renamed it in the cinder interface to "this-has-timeout-errors" | 15:41 |
| clarkb | jeblair: re https://review.openstack.org/#/c/38565/1 I think that will help us debug the problem as it adds some logging and doesn't make the situation so painful (hopefully meaning we can observe it without making ttx angry :) ) | 15:43 |
| jeblair | clarkb: next steps i think are to handle the (unconfirmed) possibility of a permanently bad FS: create a new lv, rsync docs-draft to it, poke around the old one a bit (including the l+f folder) and then consider switching to the new volume. | 15:43 |
| jeblair | clarkb: we have sufficient logging | 15:44 |
| *** johnthetubaguy has quit IRC | 15:44 | |
| jeblair | clarkb: can you point to nodes with problems that i can look into? | 15:44 |
| *** johnthetubaguy has joined #openstack-infra | 15:44 | |
| clarkb | jeblair: yeah there is a bunch of captured logs in the bug I opened. let me find that | 15:45 |
| jeblair | clarkb: is this just to address the timeout problem? | 15:45 |
| clarkb | jeblair: https://bugs.launchpad.net/openstack-ci/+bug/1204625 has a paste full of relavent logs | 15:46 |
| uvirtbot | Launchpad bug 1204625 in openstack-ci "Devstack gate launch jobs timeout confusing Jenkins and devstack gate" [High,Triaged] | 15:46 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Add a CDR log file https://review.openstack.org/38651 | 15:46 |
| clarkb | jeblair: the timeouts were the inspiration yes | 15:46 |
| annegentle | uh guys, the Wiki Signin link is gone? Mac OSX Chrome 27 | 15:47 |
| clarkb | annegentle: yes, https://bugs.launchpad.net/openstack-ci/+bug/1204841 that bug has a nasty workaround | 15:47 |
| uvirtbot | Launchpad bug 1204841 in openstack-ci "No login button on wiki" [Critical,Confirmed] | 15:47 |
| annegentle | woah | 15:47 |
| clarkb | I intend on asking Ryan_Lane to share a screen session with me when he fixes it so that I don't feel so useless when it comes to the wiki | 15:48 |
| *** adam_g has quit IRC | 15:48 | |
| jeblair | clarkb: ok, so that's a known problem. the best fix for that would be to detect the case where we are trying to add a node that is already in jenkins and set its state to ready if the state is building. | 15:48 |
| *** adam_g has joined #openstack-infra | 15:48 | |
| *** adam_g has joined #openstack-infra | 15:48 | |
| clarkb | and I will poke at it in a bit on my own probably | 15:48 |
| annegentle | heh we thought we were goign crazy | 15:48 |
| jeblair | clarkb: +1 wiki | 15:48 |
| jeblair | clarkb: (i feel like adding another race condition to fix a race condition isn't going to make us any happier) | 15:49 |
| *** johnthetubaguy has quit IRC | 15:49 | |
| anteaya | talking about buttons | 15:49 |
| anteaya | can we get an easier to discover button for signing up/logging into the foundation website? | 15:50 |
| clarkb | jeblair: the way around the race is to set the node offline first, then check if it is idle. I just realized this. I can update the patch if you think making it correct is useful | 15:50 |
| anteaya | who do I talk to about this button? | 15:50 |
| sdague | dansmith: you have a failing run on postgresql that you can point me at logs? | 15:50 |
| dansmith | http://logs.openstack.org/11/38211/3/check/gate-tempest-devstack-vm-postgres-full/34823/console.html.gz | 15:51 |
| dansmith | I'm doing a full tempest run locally to see if it's a symptom of residue or something | 15:51 |
| dansmith | but running that test alone or with the rest of its class doesn't poke it for me | 15:51 |
| jeblair | clarkb: i still feel like we should just fix the bug in the system we have, which is supposed to prevent this. | 15:51 |
| ttx | sdague: I'm a bit disconnected from reality while at OSCON -- what's the situation on the Neutron testsuite re-enablement ? | 15:51 |
| dansmith | I'm sure it's related to my change, I just need to add some stuff to figure out why and need to be able to repro it of course | 15:51 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Swap the order of configparser imports https://review.openstack.org/38653 | 15:51 |
| clarkb | jeblair: ok, I will work on detecting jenkins node state and handling that properly | 15:53 |
| clarkb | ttx: nachi thought he had a fix, but once the fix was subjected to the rigors of the gate it wasn't reliable | 15:53 |
| clarkb | ttx: so we are still not voting on that test | 15:53 |
| ttx | clarkb: ok. didn't know the fix was not enough[tm] | 15:53 |
| jeblair | clarkb: also think about it this way -- it's always an error to attempt to delete a node running a job; that's not something we want to mask (in this case it's benign, but in other cases (running a test twice on a node) it's not) | 15:54 |
| ttx | clarkb: why aren't you with us ? HP being cheap on Seatlle-PDX shuttle tickets or what | 15:54 |
| clarkb | ttx: looking at the most recent several hours of logstash data I have the neutron test failed ~40% of the time in the gate queue | 15:55 |
| jeblair | i need to run; i have just rolled out of bed, worked 1.5 hours and not done any of the things one does in the morning, including eating. | 15:55 |
| clarkb | ttx: I have driven a couple thousand miles this month and didn't want to do more. And I don't think they would've given me admission | 15:55 |
| ttx | clarkb: so close and yet so far :) | 15:56 |
| annegentle | ttx: ping if you've got a sec | 15:56 |
| ttx | annegentle: need to go to plenary room, should be back soon if wifi is not busted | 15:56 |
| ttx | brb | 15:57 |
| annegentle | ttx: ok good luck | 15:57 |
| mordred | sdague: also, re: ssh tests of nodes in tempest not working - do you know what the issue is and is it something easy to describe to someone so that they might work on it? | 15:57 |
| sdague | ttx: on the neutron tests, last time I looked we were still seeing the race | 15:58 |
| * mordred agrees with anteaya about foundation login button/link | 15:58 | |
| sdague | natchi now thinks it's a timeout issue, so there is proposed tempest fix to bump the timeout | 15:58 |
| sdague | we'll see if that makes it better | 15:58 |
| sdague | mordred: on the ssh tests, I think the crux of it is the ssh success detection isn't good | 15:58 |
| pabelanger | any infra-core mind approving https://review.openstack.org/#/c/37689/ ? | 15:58 |
| clarkb | sdague: it is. logstash indicates a ~40% failure rate in the gate (which is much much higher than our typical breakage) | 15:59 |
| clarkb | pabelanger: done. I am guessing jeblair didn't do it because traveling + OSCON | 15:59 |
| mordred | sdague: ah, as in it's tough to know if the ssh command to the node was actually good? | 15:59 |
| sdague | so a more reliable way of testing success on ssh is probably all that's need. I think right now it's looking for cirros log in text, shuold instead run some basic comment | 15:59 |
| sdague | mordred: right | 15:59 |
| sdague | and I think the check bit rotted | 16:00 |
| pabelanger | clarkb: np, thanks | 16:00 |
| sdague | because the tests were off | 16:00 |
| mordred | yeah. they all fail at this point | 16:00 |
| pabelanger | adding some asterisk patchsets now :) | 16:00 |
| mordred | but you expect that the setup should be there in as much as users/keys should be done properly | 16:00 |
| ttx | annegentle: back | 16:00 |
| ttx | so far | 16:00 |
| sdague | mordred: there may be some subtlety in getting paramiko to deal with something too | 16:00 |
| openstackgerrit | A change was merged to openstack-infra/config: Add sshkey for pabelanger https://review.openstack.org/37689 | 16:01 |
| mordred | mmm. my favorite | 16:01 |
| *** changbl has quit IRC | 16:01 | |
| sdague | I'll be honest, last time I looked in detail the ssh actually was working, but the check it was running to know if it worked was bad | 16:01 |
| clarkb | mordred: https://review.openstack.org/#/c/38647/ failed | 16:01 |
| sdague | and I haven't looked deeper | 16:01 |
| * mordred shoots self in face | 16:02 | |
| clarkb | mordred: don't do that, we need you to fix it :) | 16:02 |
| mordred | jesus | 16:02 |
| mordred | https://pypi.python.org/packages/source/s/setuptools/setuptools-0.9.8.tar.gz#md5=243076241781935f7fcad370195a4291 | 16:02 |
| mordred | is a 404 | 16:02 |
| mordred | dstufft: ^^^ | 16:03 |
| mordred | dstufft: danger will robinson | 16:03 |
| mordred | 0.9.7 is there | 16:04 |
| *** changbl has joined #openstack-infra | 16:06 | |
| dstufft | mordred: try now | 16:07 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Swap the order of configparser imports https://review.openstack.org/38653 | 16:07 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Install jeepyb and requirements from REPODIR https://review.openstack.org/38184 | 16:07 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Stop trying to install setuptools<0.7 https://review.openstack.org/38647 | 16:07 |
| mordred | dstufft: yay! | 16:07 |
| openstackgerrit | A change was merged to openstack-infra/config: Added OpenStack docs jobs for Trove. https://review.openstack.org/38191 | 16:07 |
| dstufft | mordred: the Fastly Purge Status API (which tells us if a purge completed or not) is down right now (or was yesterday) so I had to remove the status checks temporarily | 16:08 |
| dstufft | so some purges might not happen successfully :/ | 16:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Swap the order of configparser imports https://review.openstack.org/38653 | 16:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Install jeepyb and requirements from REPODIR https://review.openstack.org/38184 | 16:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Stop trying to install setuptools<0.7 https://review.openstack.org/38647 | 16:08 |
| mordred | dstufft: loverly | 16:08 |
| *** pcrews has quit IRC | 16:09 | |
| ttx | annegentle: I have high lag but I can probably try to talk | 16:10 |
| clarkb | how logstash feels this week http://memedad.com/meme/19065 but elasticsearch hasn't crashed yet so progress! | 16:10 |
| zaro | clarkb: in jjb, is it possible to wrap multiple job groups from one job-group? | 16:10 |
| clarkb | zaro: I don't think so | 16:11 |
| clarkb | I am trying to remember how job-groups expand and I think they are not recursive | 16:11 |
| *** markmc has quit IRC | 16:12 | |
| mordred | dstufft: also, amusingly, there are no things that match 'setuptools<0.7' for pip 1.4 | 16:12 |
| mordred | dstufft: because, you know, they all had letters | 16:12 |
| *** SergeyLukjanov has quit IRC | 16:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: Use the new oslo.sphinx library for the doc theme https://review.openstack.org/35874 | 16:12 |
| dstufft | mordred: yea lol | 16:12 |
| mordred | clarkb: have I mentioned that I think a neat thing to track/report/graph would be number of gate failures for things that passed in check queue | 16:14 |
| mordred | clarkb: and which percentage of those were things kicked out for merge errors | 16:14 |
| *** dkliban is now known as dkliban_lunch | 16:15 | |
| clarkb | keeping track of merge failures would definitely be interesting | 16:15 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add an option for the buildChooser https://review.openstack.org/36893 | 16:15 |
| clarkb | mordred: still around? | 16:19 |
| mordred | clarkb: yup | 16:19 |
| clarkb | mordred: opinions on logging more IRC channels? Do we just want to approve those puppet changes or do we need to get some additional level of approval | 16:19 |
| jeblair | clarkb: i haven't looked at specific changes but i think we should require the approval of the relevant ptl | 16:21 |
| *** pcrews has joined #openstack-infra | 16:21 | |
| mordred | that seems reasonable to me | 16:21 |
| clarkb | wfm | 16:21 |
| clarkb | sdague: https://review.openstack.org/#/c/38152/ | 16:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Document volume maintenance on static https://review.openstack.org/37589 | 16:22 |
| * jeblair goes to get breakfast and then to the convention center | 16:22 | |
| clarkb | have fun | 16:23 |
| *** bclifford has joined #openstack-infra | 16:24 | |
| clarkb | pabelanger: https://review.openstack.org/#/c/38189/ if you think I shouldn't approve based on your +1 just let me know | 16:24 |
| openstackgerrit | A change was merged to openstack-infra/config: Support sip:conference@pbx.openstack.org https://review.openstack.org/38189 | 16:26 |
| *** Ryan_Lane has joined #openstack-infra | 16:26 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: run bitrot jobs after pypi mirror update https://review.openstack.org/38663 | 16:28 |
| pabelanger | clarkb: sounds good | 16:29 |
| dansmith | sdague: so, in that log link I sent, | 16:35 |
| dansmith | sdague: the tempest console logs go many minutes longer than the compute and api logs... | 16:35 |
| clarkb | _TheDodd_: re https://review.openstack.org/#/c/36886/ I have a feeling that one jeblair and fungi have a chance to review it they will be happy with it | 16:35 |
| clarkb | _TheDodd_: hopefully sometime early next week. I also think that this is the sort of change we probably want to announce to the openstack-dev mailing list. Do you want to draft a short announcement in an etherpad? | 16:36 |
| clarkb | _TheDodd_: then when we merge 36886 you can send that out | 16:36 |
| _TheDodd_ | clarkb: I would be happy to. | 16:40 |
| _TheDodd_ | clarkb: Hopefully I will be able to get around to it tomorrow. | 16:41 |
| clarkb | _TheDodd_: no rush. jeblair and fungi are at OSCON so I don't expect their reviews until next week | 16:41 |
| *** UtahDave has joined #openstack-infra | 16:41 | |
| clarkb | Ryan_Lane: for the wiki fixing | 16:42 |
| clarkb | Ryan_Lane: sounds like you have already fixed the problem \o/ any chance you can share a screen session later today and show me what you have done? If you are at OSCON and that is just clunky maybe we can do something like that next week | 16:43 |
| clarkb | Ryan_Lane: I just feel fairly useless when it comes to the wiki and think that should change | 16:43 |
| _TheDodd_ | clarkb: What do you think would be most beneficial for me to mention in the etherpad? I will go ahead and reference this file ( https://etherpad.openstack.org/drive-automation-from-commitmsg ) so that everyone knows that we are pushing for header style bug references, etc. | 16:43 |
| clarkb | _TheDodd_: I am thinking that a description of the new workflow support would be good | 16:44 |
| _TheDodd_ | Great. | 16:44 |
| *** afrittoli has quit IRC | 16:46 | |
| *** andreaf has joined #openstack-infra | 16:46 | |
| clarkb | ok now I really need to AFK for a bit so that I can make my way to the office | 16:47 |
| Ryan_Lane | clarkb: I'm not as OSCON, but I can definitely share a screen session with you | 16:47 |
| Ryan_Lane | the problem was with the skin. I need to upstream a change | 16:47 |
| Ryan_Lane | the skin doesn't handle openid login links properly | 16:47 |
| *** krtaylor has joined #openstack-infra | 16:50 | |
| *** mjfork has quit IRC | 16:50 | |
| *** mjfork_ has joined #openstack-infra | 16:55 | |
| *** hashar has quit IRC | 16:56 | |
| *** jpich has quit IRC | 16:56 | |
| *** koolhead17 has joined #openstack-infra | 16:56 | |
| *** nati_ueno has joined #openstack-infra | 16:57 | |
| *** mjfork has joined #openstack-infra | 16:57 | |
| *** dkliban_lunch is now known as dkliban | 16:58 | |
| *** rnirmal has joined #openstack-infra | 17:00 | |
| *** mjfork_ has quit IRC | 17:00 | |
| *** mjfork_ has joined #openstack-infra | 17:01 | |
| *** Ryan_Lane has quit IRC | 17:02 | |
| *** SergeyLukjanov has joined #openstack-infra | 17:02 | |
| *** mjfork has quit IRC | 17:02 | |
| *** mjfork_ is now known as mjfork | 17:02 | |
| *** mjfork has quit IRC | 17:07 | |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/config: Add reload support to Asterisk https://review.openstack.org/38665 | 17:09 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Swap the order of configparser imports https://review.openstack.org/38653 | 17:09 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Install jeepyb and requirements from REPODIR https://review.openstack.org/38184 | 17:09 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Stop trying to install old pip and setuptools https://review.openstack.org/38647 | 17:09 |
| * mordred cries | 17:09 | |
| clarkb | mordred: more trouble? | 17:09 |
| clarkb | also I am only halfway through my semi AFK. So many distractions | 17:09 |
| *** derekh has quit IRC | 17:10 | |
| *** mjfork has joined #openstack-infra | 17:10 | |
| *** zul has quit IRC | 17:11 | |
| *** avtar has quit IRC | 17:12 | |
| mordred | clarkb: yup. | 17:14 |
| clarkb | ok back to AFK almost done | 17:15 |
| *** Shrews has quit IRC | 17:21 | |
| *** ewindisch has quit IRC | 17:23 | |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/config: Invert modules.conf logic https://review.openstack.org/38667 | 17:24 |
| *** Shrews has joined #openstack-infra | 17:24 | |
| *** SergeyLukjanov has quit IRC | 17:27 | |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/config: Add reload support to Asterisk https://review.openstack.org/38665 | 17:28 |
| *** SergeyLukjanov has joined #openstack-infra | 17:28 | |
| *** ewindisch has joined #openstack-infra | 17:29 | |
| *** SergeyLukjanov has quit IRC | 17:30 | |
| *** SergeyLukjanov has joined #openstack-infra | 17:31 | |
| *** HenryG has quit IRC | 17:31 | |
| *** cody-somerville has joined #openstack-infra | 17:31 | |
| *** SergeyLukjanov has quit IRC | 17:34 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add progress and estimated completion times to zuul status https://review.openstack.org/38598 | 17:34 |
| *** datsun180b has quit IRC | 17:34 | |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/config: Add reload support to Asterisk https://review.openstack.org/38665 | 17:40 |
| *** pcrews has quit IRC | 17:41 | |
| *** sarob has joined #openstack-infra | 17:42 | |
| jeblair | i will create a new docs-draft now and start an rsync | 17:42 |
| *** cody-somerville has quit IRC | 17:42 | |
| *** datsun180b has joined #openstack-infra | 17:46 | |
| fungi | that sounds safest | 17:46 |
| clarkb | jeblair: did it break again? I just got to my desk | 17:48 |
| *** sarob has quit IRC | 17:50 | |
| jeblair | clarkb: no; this is the next steps i outlined earlier | 17:50 |
| *** Ryan_Lane has joined #openstack-infra | 17:50 | |
| jeblair | clarkb: basically i don't trust the filesystem at this point; it was behaving too weirdly. | 17:51 |
| *** sarob has joined #openstack-infra | 17:51 | |
| clarkb | ok | 17:51 |
| jeblair | clarkb: (2 fsck's in a row on an unmounted fs fixed errors. 3rd one was okay, but still) | 17:51 |
| *** bclifford has quit IRC | 17:52 | |
| clarkb | I am writing the d-g change to double check jenkins and d-g state beofre erroring now | 17:52 |
| jeblair | i have an rsync running now. if there _is_ a problem with the current filesystem, that rsync may trigger an error, so we should be alert for that. | 17:52 |
| *** melwitt has joined #openstack-infra | 17:53 | |
| jeblair | clarkb: cool, thanks | 17:53 |
| pabelanger | sip:conference@openstack.org seems to be working now that I reload the dialplan | 17:53 |
| jeblair | clarkb: (also, increasing the job timeout is another quick thing we can do to mitigate that if we want to increase the pool again) | 17:53 |
| clarkb | jeblair: https://review.openstack.org/#/c/38400/ increases the timeout if you want to do that | 17:53 |
| clarkb | I -1'd it because I wanted your opinion on it | 17:54 |
| clarkb | I think we are ok at 45 node right now | 17:54 |
| *** SergeyLukjanov has joined #openstack-infra | 17:55 | |
| pabelanger | russellb: already getting scanned on pbx.openstack.org | 17:55 |
| pabelanger | maybe should do fail2ban or something | 17:55 |
| pabelanger | or changing the default SIP port will help too | 17:55 |
| pabelanger | since we are doing SIP URI dialing | 17:55 |
| *** jpeeler has quit IRC | 17:55 | |
| pabelanger | we can just update the DNS SRV records | 17:56 |
| *** nati_ueno has quit IRC | 17:57 | |
| *** nati_ueno has joined #openstack-infra | 17:58 | |
| *** bnemec has quit IRC | 17:58 | |
| *** jpeeler has joined #openstack-infra | 18:00 | |
| *** bnemec has joined #openstack-infra | 18:01 | |
| *** jpeeler has quit IRC | 18:02 | |
| *** avtar has joined #openstack-infra | 18:02 | |
| *** jpeeler has joined #openstack-infra | 18:02 | |
| *** jpeeler has joined #openstack-infra | 18:02 | |
| *** vijendar has quit IRC | 18:05 | |
| fungi | pabelanger: just give me a heads up if you want that changed, though it will probably require another trouble ticket with rackspace to change the second one | 18:05 |
| *** vijendar has joined #openstack-infra | 18:07 | |
| zaro | clarkb: the gearman-plugin is no longer listed on the jenkins plugin manager. it might be because of how we released ver 0.0.3. | 18:07 |
| *** vijendar has quit IRC | 18:07 | |
| zaro | clarkb: can you make a release of the zmq plugin to see if it will appear? | 18:08 |
| *** pabelanger has quit IRC | 18:08 | |
| *** odyssey4me has quit IRC | 18:09 | |
| *** cp16net is now known as cp16net|away | 18:09 | |
| clarkb | zaro: ya I can do that later today | 18:10 |
| *** Ryan_Lane has quit IRC | 18:14 | |
| *** Ryan_Lane has joined #openstack-infra | 18:14 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/devstack-gate: Handle Jenkins node already being added to Jenkins https://review.openstack.org/38674 | 18:15 |
| *** cp16net|away is now known as cp16net | 18:16 | |
| clarkb | jeblair: ^ I think that should do it | 18:16 |
| *** gyee has quit IRC | 18:19 | |
| *** sarob has quit IRC | 18:22 | |
| *** sarob has joined #openstack-infra | 18:22 | |
| *** chuckieb has joined #openstack-infra | 18:26 | |
| *** jpeeler1 has joined #openstack-infra | 18:30 | |
| jeblair | i see one unstable docs job, but the filesystem, block devices and server seem okay | 18:31 |
| *** jpeeler has quit IRC | 18:31 | |
| jeblair | i am assuming it's a jenkins scp plugin fluke for now | 18:31 |
| jeblair | (possibly it timed out because of the additional load on the fs from the rsync) | 18:32 |
| *** bclifford has joined #openstack-infra | 18:32 | |
| jeblair | clarkb: i am at a talk by james turnbull on logstash | 18:32 |
| jeblair | (but am sadly missing pleia2's talk) | 18:33 |
| *** sarob has quit IRC | 18:33 | |
| *** jpeeler1 is now known as jpeeler | 18:33 | |
| *** afazekas has quit IRC | 18:33 | |
| *** jpeeler has quit IRC | 18:33 | |
| *** jpeeler has joined #openstack-infra | 18:33 | |
| dhellmann | jd__: I left a +2 on mordred's change to the pbr tests; I think that's what's breaking things for both of us? | 18:33 |
| jd__ | dhellmann: I think he fast approved the one needed | 18:34 |
| dhellmann | oh, not https://review.openstack.org/#/c/38647/? | 18:34 |
| jd__ | dhellmann: this one, but he did fast approved before you +2 | 18:35 |
| jd__ | Look at 6:11 PM | 18:35 |
| mordred | dhellmann: yes | 18:35 |
| jd__ | now I keep having http://logs.openstack.org/40/38540/1/check/gate-pbr-devstack-vm-rawinstall/297/console.html | 18:35 |
| mordred | and thank you | 18:35 |
| jd__ | dunnow why | 18:35 |
| mordred | while you're at it: https://review.openstack.org/#/c/38653/ | 18:35 |
| jd__ | too soon recheck? | 18:36 |
| mordred | yeah. it hasn't landed yet. there was another thing | 18:36 |
| *** bclifford has left #openstack-infra | 18:36 | |
| dhellmann | jd__, mordred : ah, it wasn't approved when I got there :-) | 18:36 |
| jeblair | turnbull: "anyone here who works for a distro please choose an an init system; i'm getting sick of it" :) | 18:37 |
| mordred | jeblair: :) | 18:37 |
| dhellmann | mordred: your commit messages are like something out of lewis carroll | 18:37 |
| jeblair | "actual actual lifecycle of a log: record -> delete" | 18:38 |
| clarkb | jeblair: learn all the logstash things | 18:38 |
| * clarkb just had an idea | 18:39 | |
| clarkb | we should do "internal" learn the system things. Ryan_Lane can show us the wiki, I can show logstash, jeblair can do d-g and so on | 18:39 |
| clarkb | and use that to try and keep the team up to date on the broad list of things and be focused on the team and not external conference audiences. Kind of like refining the bootcamp topics | 18:40 |
| annegentle | jeblair: ping | 18:40 |
| jeblair | annegentle: pong but i'm supposed to be paying attention to a conference talk :( | 18:41 |
| clarkb | annegentle: I am happy to help if this isn't something that requires jeblair | 18:41 |
| annegentle | jeblair: ok no worries just wanted to see how it went with Todd re:repos | 18:41 |
| clarkb | (I secretly want him to become a logstash ninja) | 18:41 |
| annegentle | jeblair: holler when you can chat | 18:42 |
| *** jpeeler has quit IRC | 18:42 | |
| jeblair | annegentle: short version; went well, expecting progress by next week | 18:42 |
| clarkb | sdague: got a minute to talk about the number of tempest check tests? specifically can we stop running neutron full if the non full job doesn't even pass? | 18:42 |
| clarkb | sdague: or vice versa, put the effort into making the full job pass and forget the non full job | 18:42 |
| clarkb | sdague: I mean changes to neutron itself don't even trigger the full neutron job... | 18:43 |
| clarkb | sdague: the other one I wanted to talk about was the cells job. I am tempted to make that nova only until it is passing at least smoketests | 18:45 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: change date link to something more understandable https://review.openstack.org/38680 | 18:45 |
| *** dina_belova has joined #openstack-infra | 18:45 | |
| sdague | clarkb: I thought neutron ran both? | 18:46 |
| sdague | clarkb: so neutron full is probably a long way from getting fixed | 18:47 |
| clarkb | sdague: tempest and nova run both neutron jobs, neutron does not | 18:48 |
| sdague | really? | 18:48 |
| clarkb | really | 18:48 |
| sdague | hmph | 18:48 |
| clarkb | https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml#L741-L754 | 18:48 |
| sdague | ok, so we should change that | 18:48 |
| clarkb | maybe we make neutron and tempest run both and remove it from nova? | 18:48 |
| *** GheRivero has quit IRC | 18:48 | |
| sdague | no, nova needs to run both as well | 18:48 |
| sdague | because most of the fixes are actually on the nova side | 18:49 |
| *** GheRivero has joined #openstack-infra | 18:49 | |
| clarkb | ok | 18:49 |
| sdague | the fails are all basically interactions between nova and neutron | 18:49 |
| sdague | I think restricting cells to nova, devstack, and tempest is fine | 18:50 |
| sdague | those are the projects that will need changes to make it work, as far as I know | 18:50 |
| sdague | though I thought alaski said he was working on it | 18:50 |
| clarkb | sdague: he said possibly late this week | 18:50 |
| sdague | well, it's thursday :) | 18:50 |
| sdague | that's pretty late | 18:50 |
| clarkb | I bring this up because some of the pain with testng right now is d-g resource related, we think we have a fix lined up but until then the use of 7 d-g slaves per patchset is pretty painful particularly if we aren't get useful testing out of it | 18:51 |
| clarkb | so trying to clean up where possible (if the tests are needed we should keep them, just checking that we actually need to be running a test 500 times a day that we know will fail) | 18:52 |
| alaski | I'm currently sidetracked at the moment, though I have started looking into cells failures. | 18:53 |
| alaski | If it's causing issues I say turn it off for now | 18:53 |
| sdague | clarkb: if cells gets restricted to nova, devstack, and tempest, is that helpful? | 18:55 |
| *** krtaylor has quit IRC | 18:55 | |
| *** pabelanger has joined #openstack-infra | 18:55 | |
| sdague | because my guess is 95% of the required fixes will be in those 3 projects, so that should let cells make progress, but reduce d-g usage | 18:55 |
| *** afazekas has joined #openstack-infra | 18:56 | |
| clarkb | a little, but probably not worth it is nova is the majority of the test load. I think we will be ok, just wanted to double check that cleanup wasn't necessary (the neutron stuff bothered me more because it is inconsistent and hasn't passed for much much longer) | 18:56 |
| *** dina_belova is now known as DinaBelova | 18:56 | |
| *** HenryG has joined #openstack-infra | 18:57 | |
| *** alexpilotti has quit IRC | 18:57 | |
| sdague | clarkb: so once this merges - https://review.openstack.org/#/c/38513/, it might be the root issue on the neutron smoke side | 18:57 |
| clarkb | cool, I will keep an eye on it | 18:58 |
| jeblair | clarkb: what's our logstash processing rate? | 19:00 |
| clarkb | jeblair: ~5k events per second | 19:00 |
| clarkb | jeblair: and the bottleneck appears to be in logstash filtering | 19:01 |
| jeblair | apparently very large sites process millions of events/sec. but i don't have an idea of what kind of resources are behind that | 19:02 |
| clarkb | I think it probably helps a lot to put the filtering on the same node as the service | 19:02 |
| clarkb | which is something that jenkins slaves make tricky for us | 19:03 |
| dkranz | clarkb: The changes to add #openstack-qa and #savanna to irc logging are both pending. | 19:03 |
| dkranz | clarkb: I didn't understand how to do what you suggested in the review comment so perhaps you could just push one through | 19:03 |
| dkranz | clarkb: The other can then resubmit? | 19:03 |
| clarkb | dkranz: you can rebase one of the changes on the other, eg git review -d 37945 && git review -d 38152 && git rebase branch_for_37945 | 19:04 |
| clarkb | dkranz: they don't have to be merged to do that, just create a normal gerrit dependency. Or we can approve one and let the other sort it out | 19:05 |
| *** vijendar has joined #openstack-infra | 19:05 | |
| clarkb | dkranz: I was also hoping that sdague would give his +1 to the qa logging | 19:05 |
| HenryG | dtroyer: ping | 19:05 |
| *** jpeeler has joined #openstack-infra | 19:05 | |
| ekarlso- | anyone here wanna help me understand why http://logs.openstack.org/79/38679/1/check/gate-billingstack-python26/120/console.html.gz fails ? | 19:05 |
| dkranz | clarkb: He did in IRC but I can ask him to put it in the ticket if you prefer. | 19:06 |
| clarkb | ekarlso-: ya, its a missing dependency that was masked by older versions of testrepository. Add 'discover' to your test-requirements | 19:06 |
| clarkb | dkranz: a +1 in the review would be awesome | 19:06 |
| ekarlso- | oh | 19:06 |
| sdague | clarkb: done | 19:09 |
| sdague | clarkb: also one more round of changes on the htmlifier out there, which I think is about where I'm good before we live it | 19:10 |
| clarkb | sdague: dkranz logging inbound | 19:10 |
| clarkb | sdague: will review | 19:11 |
| sdague | mostly because my fun jquery stuff on the front to make all the debug lines disappear is going to have to wait, as getting testr functioning in the tempest gate really needs attension | 19:11 |
| clarkb | ++ | 19:11 |
| clarkb | testr in the gate is going to be baller | 19:12 |
| *** HenryG has quit IRC | 19:12 | |
| sdague | yeh, so we're probably going to need some help to get to the bottom of things | 19:12 |
| sdague | there are at least 6 races right now | 19:12 |
| *** HenryG has joined #openstack-infra | 19:12 | |
| *** chuckieb has quit IRC | 19:14 | |
| *** rahmu has quit IRC | 19:17 | |
| *** BobBall has quit IRC | 19:17 | |
| *** alaski has quit IRC | 19:17 | |
| *** adarazs has quit IRC | 19:17 | |
| *** comstud has quit IRC | 19:17 | |
| *** rockstar has quit IRC | 19:17 | |
| *** avtar has quit IRC | 19:17 | |
| *** SergeyLukjanov has quit IRC | 19:17 | |
| *** datsun180b has quit IRC | 19:17 | |
| *** rnirmal has quit IRC | 19:17 | |
| *** UtahDave has quit IRC | 19:17 | |
| *** dprince has quit IRC | 19:17 | |
| *** rfolco has quit IRC | 19:17 | |
| *** yamahata has quit IRC | 19:17 | |
| *** nibalizer has quit IRC | 19:17 | |
| *** westmaas has quit IRC | 19:17 | |
| *** ogelbukh has quit IRC | 19:17 | |
| *** mordred has quit IRC | 19:17 | |
| *** sshturm__ has quit IRC | 19:17 | |
| *** nijaba has quit IRC | 19:17 | |
| *** gaelL has quit IRC | 19:17 | |
| *** sgviking has quit IRC | 19:17 | |
| *** koolhead17 has quit IRC | 19:17 | |
| *** andreaf has quit IRC | 19:17 | |
| *** CaptTofu has quit IRC | 19:17 | |
| *** leifmadsen has quit IRC | 19:17 | |
| *** mgagne has quit IRC | 19:17 | |
| *** locke105 has quit IRC | 19:17 | |
| *** akscram has quit IRC | 19:17 | |
| *** sdake_ has quit IRC | 19:17 | |
| *** Ryan_Lane has quit IRC | 19:17 | |
| *** bnemec has quit IRC | 19:17 | |
| *** Shrews has quit IRC | 19:17 | |
| *** mrodden has quit IRC | 19:17 | |
| *** dkehn has quit IRC | 19:17 | |
| *** maurosr has quit IRC | 19:17 | |
| *** plomakin has quit IRC | 19:17 | |
| *** primeministerp has quit IRC | 19:17 | |
| *** bodepd has quit IRC | 19:17 | |
| *** Mithrandir has quit IRC | 19:17 | |
| *** ladquin has quit IRC | 19:17 | |
| *** jd__ has quit IRC | 19:17 | |
| *** vijendar has quit IRC | 19:17 | |
| *** dkliban has quit IRC | 19:17 | |
| *** prad has quit IRC | 19:17 | |
| *** pentameter has quit IRC | 19:17 | |
| *** anteaya has quit IRC | 19:17 | |
| *** mriedem has quit IRC | 19:17 | |
| *** jeblair has quit IRC | 19:17 | |
| *** thingee has quit IRC | 19:17 | |
| *** saper has quit IRC | 19:17 | |
| *** markmcclain has quit IRC | 19:17 | |
| *** mestery has quit IRC | 19:18 | |
| *** tian has quit IRC | 19:18 | |
| *** mikal has quit IRC | 19:18 | |
| *** odyi has quit IRC | 19:18 | |
| *** sdake has quit IRC | 19:18 | |
| *** jeremyb has quit IRC | 19:18 | |
| *** HenryG has quit IRC | 19:18 | |
| *** GheRivero has quit IRC | 19:18 | |
| *** DinaBelova has quit IRC | 19:18 | |
| *** melwitt has quit IRC | 19:18 | |
| *** uvirtbot has quit IRC | 19:18 | |
| *** dripton has quit IRC | 19:18 | |
| *** retr0h has quit IRC | 19:18 | |
| *** blamar has quit IRC | 19:18 | |
| *** shanewang has quit IRC | 19:18 | |
| *** jhesketh__ has quit IRC | 19:18 | |
| *** lbragstad1 has quit IRC | 19:18 | |
| *** chmouel has quit IRC | 19:18 | |
| *** jpeeler has quit IRC | 19:18 | |
| *** _TheDodd_ has quit IRC | 19:18 | |
| *** burt has quit IRC | 19:18 | |
| *** woodspa has quit IRC | 19:18 | |
| *** weshay has quit IRC | 19:18 | |
| *** pleia2 has quit IRC | 19:18 | |
| *** dhellmann has quit IRC | 19:18 | |
| *** jog0 has quit IRC | 19:18 | |
| *** devananda has quit IRC | 19:18 | |
| *** koobs has quit IRC | 19:18 | |
| *** afazekas has quit IRC | 19:18 | |
| *** pabelanger has quit IRC | 19:18 | |
| *** nati_ueno has quit IRC | 19:18 | |
| *** olaph has quit IRC | 19:18 | |
| *** ianw has quit IRC | 19:18 | |
| *** openstackgerrit has quit IRC | 19:18 | |
| *** SlickNik has quit IRC | 19:18 | |
| *** hughsaunders has quit IRC | 19:18 | |
| *** briancline has quit IRC | 19:18 | |
| *** Adri2000 has quit IRC | 19:18 | |
| *** sdague has quit IRC | 19:18 | |
| *** cyeoh has quit IRC | 19:18 | |
| *** ChanServ has quit IRC | 19:18 | |
| *** lifeless has quit IRC | 19:18 | |
| *** zaro has quit IRC | 19:18 | |
| *** davidhadas has quit IRC | 19:18 | |
| *** soren has quit IRC | 19:18 | |
| *** SpamapS has quit IRC | 19:18 | |
| *** fbo has quit IRC | 19:18 | |
| *** EmilienM has quit IRC | 19:18 | |
| *** Guest63475 has quit IRC | 19:18 | |
| *** jcooley has quit IRC | 19:18 | |
| *** sileht has quit IRC | 19:18 | |
| *** jgriffith has quit IRC | 19:18 | |
| *** cp16net has quit IRC | 19:18 | |
| *** NobodyCam has quit IRC | 19:18 | |
| *** vipul has quit IRC | 19:18 | |
| *** Daviey has quit IRC | 19:18 | |
| *** juice has quit IRC | 19:18 | |
| *** mirrorbox has quit IRC | 19:18 | |
| *** antonym has quit IRC | 19:18 | |
| *** spiffxp has quit IRC | 19:18 | |
| *** kiall has quit IRC | 19:18 | |
| *** oubiwann has quit IRC | 19:18 | |
| *** adam_g has quit IRC | 19:18 | |
| *** jaypipes has quit IRC | 19:18 | |
| *** crank has quit IRC | 19:18 | |
| *** jbjohnso_ has quit IRC | 19:18 | |
| *** annegentle has quit IRC | 19:18 | |
| *** EntropyWorks has quit IRC | 19:18 | |
| *** ttx has quit IRC | 19:18 | |
| *** stevebaker has quit IRC | 19:18 | |
| *** dstufft has quit IRC | 19:18 | |
| *** wayneeseguin has quit IRC | 19:18 | |
| *** notmyname has quit IRC | 19:18 | |
| *** russellb has quit IRC | 19:18 | |
| *** jlk has quit IRC | 19:18 | |
| *** vishy has quit IRC | 19:18 | |
| *** fungi has quit IRC | 19:18 | |
| *** clarkb has quit IRC | 19:18 | |
| *** kpepple_ has quit IRC | 19:18 | |
| *** danger_fo has quit IRC | 19:18 | |
| *** Alex_Gaynor has quit IRC | 19:18 | |
| *** changbl has quit IRC | 19:18 | |
| *** dkranz has quit IRC | 19:18 | |
| *** shardy has quit IRC | 19:18 | |
| *** s1rp has quit IRC | 19:18 | |
| *** jesusaurus has quit IRC | 19:18 | |
| *** ekarlso- has quit IRC | 19:18 | |
| *** simonmcc has quit IRC | 19:18 | |
| *** Ng has quit IRC | 19:18 | |
| *** lyz has quit IRC | 19:18 | |
| *** LinuxJedi has quit IRC | 19:18 | |
| *** dtroyer has quit IRC | 19:18 | |
| *** Vivek has quit IRC | 19:18 | |
| *** mancdaz has quit IRC | 19:18 | |
| *** mtreinish has quit IRC | 19:18 | |
| *** lcheng has quit IRC | 19:18 | |
| *** davidlenwell has quit IRC | 19:18 | |
| *** jkt has quit IRC | 19:18 | |
| *** harlowja has quit IRC | 19:18 | |
| *** dhouck has quit IRC | 19:18 | |
| *** mkoderer has quit IRC | 19:18 | |
| *** ewindisch has quit IRC | 19:18 | |
| *** mjfork has quit IRC | 19:18 | |
| *** arosen has quit IRC | 19:18 | |
| *** swifterdarrell has quit IRC | 19:18 | |
| *** larsks has quit IRC | 19:18 | |
| *** klrmn has quit IRC | 19:18 | |
| *** HenryG has joined #openstack-infra | 19:19 | |
| *** jpeeler has joined #openstack-infra | 19:19 | |
| *** vijendar has joined #openstack-infra | 19:19 | |
| *** afazekas has joined #openstack-infra | 19:19 | |
| *** pabelanger has joined #openstack-infra | 19:19 | |
| *** GheRivero has joined #openstack-infra | 19:19 | |
| *** DinaBelova has joined #openstack-infra | 19:19 | |
| *** Ryan_Lane has joined #openstack-infra | 19:19 | |
| *** avtar has joined #openstack-infra | 19:19 | |
| *** bnemec has joined #openstack-infra | 19:19 | |
| *** nati_ueno has joined #openstack-infra | 19:19 | |
| *** SergeyLukjanov has joined #openstack-infra | 19:19 | |
| *** melwitt has joined #openstack-infra | 19:19 | |
| *** datsun180b has joined #openstack-infra | 19:19 | |
| *** ewindisch has joined #openstack-infra | 19:19 | |
| *** Shrews has joined #openstack-infra | 19:19 | |
| *** mjfork has joined #openstack-infra | 19:19 | |
| *** rnirmal has joined #openstack-infra | 19:19 | |
| *** koolhead17 has joined #openstack-infra | 19:19 | |
| *** andreaf has joined #openstack-infra | 19:19 | |
| *** UtahDave has joined #openstack-infra | 19:19 | |
| *** changbl has joined #openstack-infra | 19:19 | |
| *** adam_g has joined #openstack-infra | 19:19 | |
| *** mrodden has joined #openstack-infra | 19:19 | |
| *** CaptTofu has joined #openstack-infra | 19:19 | |
| *** mkoderer has joined #openstack-infra | 19:19 | |
| *** _TheDodd_ has joined #openstack-infra | 19:19 | |
| *** dkranz has joined #openstack-infra | 19:19 | |
| *** dkliban has joined #openstack-infra | 19:19 | |
| *** leifmadsen has joined #openstack-infra | 19:19 | |
| *** prad has joined #openstack-infra | 19:19 | |
| *** burt has joined #openstack-infra | 19:19 | |
| *** markmcclain has joined #openstack-infra | 19:19 | |
| *** pentameter has joined #openstack-infra | 19:19 | |
| *** dprince has joined #openstack-infra | 19:19 | |
| *** mgagne has joined #openstack-infra | 19:19 | |
| *** anteaya has joined #openstack-infra | 19:19 | |
| *** mriedem has joined #openstack-infra | 19:19 | |
| *** woodspa has joined #openstack-infra | 19:19 | |
| *** rfolco has joined #openstack-infra | 19:19 | |
| *** weshay has joined #openstack-infra | 19:19 | |
| *** jhesketh__ has joined #openstack-infra | 19:19 | |
| *** EmilienM has joined #openstack-infra | 19:19 | |
| *** dkehn has joined #openstack-infra | 19:19 | |
| *** lbragstad1 has joined #openstack-infra | 19:19 | |
| *** olaph has joined #openstack-infra | 19:19 | |
| *** yamahata has joined #openstack-infra | 19:19 | |
| *** sgviking has joined #openstack-infra | 19:19 | |
| *** ianw has joined #openstack-infra | 19:19 | |
| *** lifeless has joined #openstack-infra | 19:19 | |
| *** zaro has joined #openstack-infra | 19:19 | |
| *** SlickNik has joined #openstack-infra | 19:19 | |
| *** nibalizer has joined #openstack-infra | 19:19 | |
| *** locke105 has joined #openstack-infra | 19:19 | |
| *** openstackgerrit has joined #openstack-infra | 19:19 | |
| *** akscram has joined #openstack-infra | 19:19 | |
| *** maurosr has joined #openstack-infra | 19:19 | |
| *** pleia2 has joined #openstack-infra | 19:19 | |
| *** jeblair has joined #openstack-infra | 19:19 | |
| *** dhellmann has joined #openstack-infra | 19:19 | |
| *** thingee has joined #openstack-infra | 19:19 | |
| *** plomakin has joined #openstack-infra | 19:19 | |
| *** uvirtbot has joined #openstack-infra | 19:19 | |
| *** westmaas has joined #openstack-infra | 19:19 | |
| *** mestery has joined #openstack-infra | 19:19 | |
| *** tian has joined #openstack-infra | 19:19 | |
| *** mirrorbox has joined #openstack-infra | 19:19 | |
| *** ladquin has joined #openstack-infra | 19:19 | |
| *** davidhadas has joined #openstack-infra | 19:19 | |
| *** soren has joined #openstack-infra | 19:19 | |
| *** ogelbukh has joined #openstack-infra | 19:19 | |
| *** jbjohnso_ has joined #openstack-infra | 19:19 | |
| *** mikal has joined #openstack-infra | 19:19 | |
| *** sdake_ has joined #openstack-infra | 19:19 | |
| *** mordred has joined #openstack-infra | 19:19 | |
| *** jog0 has joined #openstack-infra | 19:19 | |
| *** chmouel has joined #openstack-infra | 19:19 | |
| *** primeministerp has joined #openstack-infra | 19:19 | |
| *** bodepd has joined #openstack-infra | 19:19 | |
| *** Mithrandir has joined #openstack-infra | 19:19 | |
| *** sshturm__ has joined #openstack-infra | 19:19 | |
| *** dripton has joined #openstack-infra | 19:19 | |
| *** nijaba has joined #openstack-infra | 19:19 | |
| *** retr0h has joined #openstack-infra | 19:19 | |
| *** saper has joined #openstack-infra | 19:19 | |
| *** rahmu has joined #openstack-infra | 19:19 | |
| *** gaelL has joined #openstack-infra | 19:19 | |
| *** jd__ has joined #openstack-infra | 19:19 | |
| *** arosen has joined #openstack-infra | 19:19 | |
| *** odyi has joined #openstack-infra | 19:19 | |
| *** devananda has joined #openstack-infra | 19:19 | |
| *** larsks has joined #openstack-infra | 19:19 | |
| *** blamar has joined #openstack-infra | 19:19 | |
| *** sdake has joined #openstack-infra | 19:19 | |
| *** stevebaker has joined #openstack-infra | 19:19 | |
| *** koobs has joined #openstack-infra | 19:19 | |
| *** Guest63475 has joined #openstack-infra | 19:19 | |
| *** shanewang has joined #openstack-infra | 19:19 | |
| *** shardy has joined #openstack-infra | 19:19 | |
| *** s1rp has joined #openstack-infra | 19:19 | |
| *** jesusaurus has joined #openstack-infra | 19:19 | |
| *** jcooley has joined #openstack-infra | 19:19 | |
| *** jaypipes has joined #openstack-infra | 19:19 | |
| *** ekarlso- has joined #openstack-infra | 19:19 | |
| *** crank has joined #openstack-infra | 19:19 | |
| *** sileht has joined #openstack-infra | 19:19 | |
| *** jgriffith has joined #openstack-infra | 19:19 | |
| *** cp16net has joined #openstack-infra | 19:19 | |
| *** NobodyCam has joined #openstack-infra | 19:19 | |
| *** swifterdarrell has joined #openstack-infra | 19:19 | |
| *** Daviey has joined #openstack-infra | 19:19 | |
| *** SpamapS has joined #openstack-infra | 19:19 | |
| *** vipul has joined #openstack-infra | 19:19 | |
| *** klrmn has joined #openstack-infra | 19:19 | |
| *** juice has joined #openstack-infra | 19:19 | |
| *** annegentle has joined #openstack-infra | 19:19 | |
| *** EntropyWorks has joined #openstack-infra | 19:19 | |
| *** hughsaunders has joined #openstack-infra | 19:19 | |
| *** briancline has joined #openstack-infra | 19:19 | |
| *** ttx has joined #openstack-infra | 19:19 | |
| *** LinuxJedi has joined #openstack-infra | 19:19 | |
| *** dtroyer has joined #openstack-infra | 19:19 | |
| *** simonmcc has joined #openstack-infra | 19:19 | |
| *** Vivek has joined #openstack-infra | 19:19 | |
| *** jeremyb has joined #openstack-infra | 19:19 | |
| *** cyeoh has joined #openstack-infra | 19:19 | |
| *** sdague has joined #openstack-infra | 19:19 | |
| *** Adri2000 has joined #openstack-infra | 19:19 | |
| *** Ng has joined #openstack-infra | 19:19 | |
| *** mancdaz has joined #openstack-infra | 19:19 | |
| *** antonym has joined #openstack-infra | 19:19 | |
| *** fbo has joined #openstack-infra | 19:19 | |
| *** dstufft has joined #openstack-infra | 19:19 | |
| *** mtreinish has joined #openstack-infra | 19:19 | |
| *** wayneeseguin has joined #openstack-infra | 19:19 | |
| *** notmyname has joined #openstack-infra | 19:19 | |
| *** lcheng has joined #openstack-infra | 19:19 | |
| *** spiffxp has joined #openstack-infra | 19:19 | |
| *** BobBall has joined #openstack-infra | 19:19 | |
| *** russellb has joined #openstack-infra | 19:19 | |
| *** jlk has joined #openstack-infra | 19:19 | |
| *** vishy has joined #openstack-infra | 19:19 | |
| *** fungi has joined #openstack-infra | 19:19 | |
| *** ChanServ has joined #openstack-infra | 19:19 | |
| *** kiall has joined #openstack-infra | 19:19 | |
| *** clarkb has joined #openstack-infra | 19:19 | |
| *** oubiwann has joined #openstack-infra | 19:19 | |
| *** kpepple_ has joined #openstack-infra | 19:19 | |
| *** danger_fo has joined #openstack-infra | 19:19 | |
| *** Alex_Gaynor has joined #openstack-infra | 19:19 | |
| *** davidlenwell has joined #openstack-infra | 19:19 | |
| *** jkt has joined #openstack-infra | 19:19 | |
| *** lyz has joined #openstack-infra | 19:19 | |
| *** harlowja has joined #openstack-infra | 19:19 | |
| *** dhouck has joined #openstack-infra | 19:19 | |
| *** rockstar has joined #openstack-infra | 19:19 | |
| *** comstud has joined #openstack-infra | 19:19 | |
| *** adarazs has joined #openstack-infra | 19:19 | |
| *** alaski has joined #openstack-infra | 19:19 | |
| *** sendak.freenode.net sets mode: +o ChanServ | 19:19 | |
| *** HenryG_ has joined #openstack-infra | 19:19 | |
| sdague | so, d-g forces ceilo on? | 19:19 |
| russellb | pabelanger: let's start with fail2ban, IMO | 19:19 |
| *** _TheDodd_ has quit IRC | 19:19 | |
| *** _TheDodd_ has joined #openstack-infra | 19:19 | |
| *** avtar has quit IRC | 19:20 | |
| *** SergeyLukjanov has quit IRC | 19:20 | |
| *** datsun180b has quit IRC | 19:20 | |
| *** rnirmal has quit IRC | 19:20 | |
| *** UtahDave has quit IRC | 19:20 | |
| *** dprince has quit IRC | 19:20 | |
| *** rfolco has quit IRC | 19:20 | |
| *** yamahata has quit IRC | 19:20 | |
| *** nibalizer has quit IRC | 19:20 | |
| *** westmaas has quit IRC | 19:20 | |
| *** ogelbukh has quit IRC | 19:20 | |
| *** mordred has quit IRC | 19:20 | |
| *** sshturm__ has quit IRC | 19:20 | |
| *** nijaba has quit IRC | 19:20 | |
| *** gaelL has quit IRC | 19:20 | |
| *** sgviking has quit IRC | 19:20 | |
| russellb | fungi: you mentioned changing the SRV records, i don't see them, should they be there? $ dig -t SRV _sip._udp openstack.org | 19:20 |
| dhellmann | sdague: what's the deal with the keystone client issue? are you still waiting on us to do something with ceilometer? | 19:20 |
| pabelanger | russellb, ya, testing it out now locally | 19:21 |
| *** avtar has joined #openstack-infra | 19:21 | |
| *** SergeyLukjanov has joined #openstack-infra | 19:21 | |
| *** datsun180b has joined #openstack-infra | 19:21 | |
| *** rnirmal has joined #openstack-infra | 19:21 | |
| *** UtahDave has joined #openstack-infra | 19:21 | |
| *** dprince has joined #openstack-infra | 19:21 | |
| *** rfolco has joined #openstack-infra | 19:21 | |
| *** yamahata has joined #openstack-infra | 19:21 | |
| *** sgviking has joined #openstack-infra | 19:21 | |
| *** nibalizer has joined #openstack-infra | 19:21 | |
| *** westmaas has joined #openstack-infra | 19:21 | |
| *** ogelbukh has joined #openstack-infra | 19:21 | |
| *** mordred has joined #openstack-infra | 19:21 | |
| *** sshturm__ has joined #openstack-infra | 19:21 | |
| *** nijaba has joined #openstack-infra | 19:21 | |
| *** gaelL has joined #openstack-infra | 19:21 | |
| *** SergeyLukjanov has quit IRC | 19:21 | |
| *** zehicle_at_dell has joined #openstack-infra | 19:21 | |
| russellb | pabelanger: cool ... we documented it in ATDG, but need to make sure all the patterns are still current | 19:21 |
| *** SergeyLukjanov has joined #openstack-infra | 19:21 | |
| clarkb | sdague: yes iirc | 19:22 |
| sdague | dhellmann: yes, there is a unit test fail to land that | 19:22 |
| *** HenryG has quit IRC | 19:22 | |
| * dhellmann mutters under his breath | 19:23 | |
| pabelanger | russellb, ya, I'm more concerned with the default filters that ship with the fail2ban package right now | 19:23 |
| pabelanger | but will check it out | 19:23 |
| dhellmann | sdague: can you point me at a review or something? | 19:23 |
| sdague | dhellmann: I did bring it up on the ceilo meeting today, jd__ said he'd look at it | 19:23 |
| sdague | but more eyes good | 19:23 |
| russellb | pabelanger: oh didn't know it included asterisk support | 19:23 |
| dhellmann | sdague: ok, if he's on it that's fine | 19:23 |
| sdague | https://review.openstack.org/#/c/36905/ | 19:23 |
| sdague | dhellmann: I don't know if he did | 19:23 |
| pabelanger | still waiting for leifmadsen to drop of the book at my place, like he said the last time he was in town | 19:23 |
| dhellmann | I saw something in the backlog about it earlier, but didn't have a chance to look into it | 19:23 |
| *** ewindisch has quit IRC | 19:23 | |
| *** mjfork has quit IRC | 19:23 | |
| *** arosen has quit IRC | 19:23 | |
| *** swifterdarrell has quit IRC | 19:23 | |
| *** larsks has quit IRC | 19:23 | |
| *** klrmn has quit IRC | 19:23 | |
| pabelanger | russellb, it doesn't just other apps | 19:23 |
| sdague | clarkb: ok, so we should probably try to sync the d-g defaults back to devstack | 19:24 |
| *** sarob has joined #openstack-infra | 19:24 | |
| dhellmann | sdague: after we have this fixed, I would like to prioritize a gate test using that nova plugin we have | 19:24 |
| sdague | dhellmann: sure | 19:24 |
| sdague | 1.0.2 is where you guys uncapped, right? | 19:25 |
| jd__ | dhellmann: sdague: if that can't wait Monday, go ahead, I won't do that before then | 19:26 |
| dhellmann | jd__: OK | 19:26 |
| dhellmann | sdague: someone else did that so I'd have to check | 19:26 |
| *** zehicle_at_dell has quit IRC | 19:27 | |
| *** krtaylor has joined #openstack-infra | 19:28 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: update global requires on clients for keystone fix https://review.openstack.org/38686 | 19:28 |
| sdague | man, one day I'll stop chasing this requirements thread.... | 19:28 |
| sdague | currently preventing me from installing devstack | 19:29 |
| *** larsks has joined #openstack-infra | 19:29 | |
| *** Ryan_Lane has quit IRC | 19:30 | |
| *** openstack has joined #openstack-infra | 19:32 | |
| *** Ryan_Lane has joined #openstack-infra | 19:32 | |
| *** mjfork has joined #openstack-infra | 19:35 | |
| *** arosen has joined #openstack-infra | 19:35 | |
| *** swifterdarrell has joined #openstack-infra | 19:35 | |
| *** klrmn has joined #openstack-infra | 19:35 | |
| *** dprince has quit IRC | 19:37 | |
| dansmith | is grenade borken? | 19:38 |
| sdague | yes | 19:38 |
| sdague | sorry, no | 19:38 |
| sdague | wrong window | 19:39 |
| sdague | dansmith: you got a link to a fail? | 19:39 |
| dansmith | not yet, I just see at least one of mine is failing on grenade and I don't think there's any way it could | 19:40 |
| sdague | any config changes? | 19:40 |
| *** dims has joined #openstack-infra | 19:40 | |
| sdague | we're gating on grenade everywhere | 19:40 |
| dansmith | nope, no config changes | 19:40 |
| *** HenryG_ has quit IRC | 19:41 | |
| dansmith | and one of the ones after it passes grenade | 19:41 |
| dansmith | see 36363 | 19:41 |
| sdague | well, there is always the possibility of a race | 19:41 |
| sdague | that we didn't find | 19:41 |
| Ryan_Lane | clarkb: I'm also interested in learning logstash | 19:41 |
| *** HenryG_ has joined #openstack-infra | 19:41 | |
| Ryan_Lane | I'd like to use it at wikimedia | 19:41 |
| *** dkliban has quit IRC | 19:41 | |
| dansmith | yeah, all my review numbers are palindromes, cause I'm awesome | 19:41 |
| *** afazekas is now known as afazekas_zz | 19:41 | |
| clarkb | dansmith: I need that skill | 19:43 |
| clarkb | Ryan_Lane: we are using it for our test logs, and it seems to be stable now. Current issue is it can't keep up with the load | 19:43 |
| dansmith | clarkb: :P | 19:43 |
| sdague | yeh, but it means you can mess with him a lot by making things one off | 19:43 |
| Ryan_Lane | heh. that's what I worry about | 19:44 |
| Ryan_Lane | because we have about 1000 systems and about 400 instances and we want to send all syslog data to it | 19:45 |
| clarkb | Ryan_Lane: and in our case it seems to be related to the weird way we squeeze everything through a funnel | 19:45 |
| clarkb | Ryan_Lane: if you can distribute the work load to as near the log source as possible you should be fine | 19:45 |
| clarkb | Ryan_Lane: we don't trust our jenkins slaves so we don't have that luxury | 19:45 |
| sdague | dansmith: looks like a more fundamental fail - https://jenkins.openstack.org/job/gate-grenade-devstack-vm/21225/console | 19:45 |
| * Ryan_Lane nods | 19:45 | |
| *** HenryG_ has quit IRC | 19:45 | |
| dansmith | sdague: ah | 19:46 |
| clarkb | Ryan_Lane: also it does a good job of scaling horizontally so throwing hardware at the problem is a possibility | 19:46 |
| *** HenryG has joined #openstack-infra | 19:46 | |
| Ryan_Lane | yep | 19:46 |
| sdague | clarkb: any ideas about that last link | 19:46 |
| clarkb | sdague: looking | 19:46 |
| Ryan_Lane | we're going to have an elastic search cluster anyway, so I may be able to reuse it for this as well | 19:46 |
| clarkb | sdague: ya one sec | 19:47 |
| clarkb | sdague: dansmith https://bugs.launchpad.net/openstack-ci/+bug/1204625 is the bug | 19:47 |
| uvirtbot | Launchpad bug 1204625 in openstack-ci "Devstack gate launch jobs timeout confusing Jenkins and devstack gate" [High,Triaged] | 19:47 |
| clarkb | sdague: dansmith https://review.openstack.org/#/c/38400/ and https://review.openstack.org/#/c/38674/ are two potential things that will mitigate it, but I really wanted jeblair to weight in on those changes before I approve anything | 19:48 |
| *** burt has quit IRC | 19:48 | |
| clarkb | sdague: dansmith: the tl;dr is that jenkins and devstack-gate get out of sync because some of the d-g jobs timeout | 19:48 |
| dansmith | okay | 19:48 |
| clarkb | and that causes weirdness | 19:48 |
| maurosr | easy review: https://review.openstack.org/#/c/38688/ porting this fix https://review.openstack.org/#/c/30057/ to api-v3 fixed-ips | 19:49 |
| clarkb | sdague: if you want to review 38674 that would be helpful | 19:49 |
| maurosr | ops | 19:49 |
| maurosr | sorry, wrong channel | 19:49 |
| Ryan_Lane | ok. redeployed the updated wiki skin with a live hack for the login link | 19:50 |
| sdague | clarkb: looking | 19:50 |
| Ryan_Lane | I'd generally not do so, but the old skin has an XSS vulnerability | 19:50 |
| *** _TheDodd_ has quit IRC | 19:50 | |
| clarkb | Ryan_Lane: want to fire up a screen on wiki.o.o that I can screen -x to? | 19:51 |
| Ryan_Lane | I always forget how to do this | 19:51 |
| Ryan_Lane | I have a screen session running on it | 19:51 |
| Ryan_Lane | if you're also root, you can just screen -x, right? | 19:51 |
| clarkb | Ryan_Lane: if you have a screen running I should be able to just attach to it. yup | 19:51 |
| sdague | clarkb: no additional typing or fields in those jenkins exceptions? | 19:52 |
| clarkb | sdague: no :( | 19:52 |
| sdague | then seems fine to me | 19:52 |
| clarkb | sdague: you can see the exception in myjenkins.py | 19:52 |
| clarkb | Ryan_Lane: I am attached | 19:52 |
| Ryan_Lane | ok | 19:52 |
| Ryan_Lane | I'll give you a walkthrough | 19:52 |
| clarkb | awesome | 19:52 |
| dansmith | man I really need to write a recheck-all script or something, because ten patches needing rechecks sucks :/ | 19:52 |
| Ryan_Lane | mediawiki is installed under /srv/mediawiki | 19:52 |
| Ryan_Lane | I have two versions always there, under slot0 and slot1 | 19:52 |
| Ryan_Lane | slot0 is currently running | 19:53 |
| clarkb | Ryan_Lane: how can you tell which one is running? | 19:53 |
| clarkb | w symlink> | 19:53 |
| Ryan_Lane | the apache alias is for /srv/mediawiki/w | 19:53 |
| clarkb | got it | 19:53 |
| Ryan_Lane | I use the wikimedia foundation deployment branches | 19:54 |
| *** sarob has quit IRC | 19:54 | |
| Ryan_Lane | it's abnormal, but there's a reason for it | 19:54 |
| *** nayward has joined #openstack-infra | 19:54 | |
| *** sarob has joined #openstack-infra | 19:54 | |
| Ryan_Lane | if you notice, that branch has a large number of extension submodules | 19:54 |
| Ryan_Lane | all wikimedia foundation extensions that are deployed to wikimedia sites are there | 19:54 |
| Ryan_Lane | they are known to work with this version of mediawiki | 19:55 |
| Ryan_Lane | we have 5 extensions that don't come from wikimedia | 19:55 |
| clarkb | dansmith: there is a tool like that in zuul/tools/* iirc | 19:55 |
| clarkb | Ryan_Lane: I see them | 19:55 |
| openstackgerrit | Diane Fleming proposed a change to openstack-infra/config: Update yaml file to add new OpenStack End User Guide https://review.openstack.org/38694 | 19:55 |
| Ryan_Lane | extensions/EmbedVideo/ extensions/OpenID/ extensions/ReplaceText/ extensions/googleAnalytics/ skins/strapping/ | 19:55 |
| Ryan_Lane | OpenID will soon also be deployed on wikimedia sites | 19:55 |
| Ryan_Lane | and that'll be one less we have to manage manually | 19:56 |
| dansmith | clarkb: oh, nice | 19:56 |
| Ryan_Lane | so that we can easily switch between versions, I move some things outside of the mediawiki path | 19:56 |
| Ryan_Lane | images and Settings.php | 19:56 |
| *** pcrews has joined #openstack-infra | 19:57 | |
| Ryan_Lane | LocalSettings.php has been modified to require the shared Settings.php | 19:57 |
| clarkb | and that allows you to not need to configure the two slots independently | 19:57 |
| Ryan_Lane | yep | 19:57 |
| Ryan_Lane | and /w/images is an alias in apache pointing at /srv/mediawiki/images | 19:58 |
| sarob | anyone up for reviewing https://review.openstack.org/#/c/38502/ | 19:58 |
| Ryan_Lane | how to upgrade: | 19:58 |
| Ryan_Lane | 1. figure out which slot is being used, prepare to upgrade the other one | 19:58 |
| sarob | its holding up publishing some content I need for a meetup tonight | 19:59 |
| clarkb | sarob: I can review when Ryan_Lane is done giving me the howto | 19:59 |
| Ryan_Lane | don't ever use /srv/mediawiki/w, always use the slot (otherwise git will fuck things up) | 19:59 |
| sarob | okay, thanks! | 19:59 |
| clarkb | Ryan_Lane: good to know | 19:59 |
| Ryan_Lane | it sets submodule config paths to the directory you are in | 19:59 |
| Ryan_Lane | so it'll reference w, which will cause problems later | 20:00 |
| Ryan_Lane | 2: fetch the changes via git fetch | 20:01 |
| Ryan_Lane | 3: find the newest wmf branch | 20:01 |
| Ryan_Lane | 4: create a local branch that tracks the newest remote branch | 20:02 |
| Ryan_Lane | funny enough, there's already a newer branch | 20:02 |
| Ryan_Lane | we could upgrade again right now. let me see if this is already deployed on wikimedia | 20:02 |
| clarkb | you guys go fast | 20:02 |
| *** datsun180b has quit IRC | 20:03 | |
| *** HenryG has quit IRC | 20:03 | |
| Ryan_Lane | we do weekly major upgrades | 20:03 |
| Ryan_Lane | http://www.mediawiki.org/wiki/Special:Version <-- | 20:03 |
| Ryan_Lane | running this new branch | 20:03 |
| Ryan_Lane | mediawiki.org always gets deployed to first | 20:03 |
| Ryan_Lane | it's lesser used and lets devs eat their dogfood | 20:03 |
| Ryan_Lane | so, let's actually do an upgrade | 20:04 |
| clarkb | ++ | 20:04 |
| Ryan_Lane | 5: checkout the new branch | 20:04 |
| Ryan_Lane | 6: update submodules: git submodule update —init | 20:04 |
| *** sdake has quit IRC | 20:05 | |
| clarkb | do you typically want to wait for the new version to be deployed on mediawiki.org before upgrading wiki.o.o? | 20:06 |
| Ryan_Lane | yes. always :) | 20:06 |
| *** dkliban has joined #openstack-infra | 20:06 | |
| Ryan_Lane | otherwise they may not be ready to actually deploy it | 20:06 |
| Ryan_Lane | or may rollback because of something being broken | 20:06 |
| clarkb | I see | 20:07 |
| Ryan_Lane | a major downside to using the wmf branches is that they don't backport security fixes | 20:07 |
| Ryan_Lane | you have to upgrade | 20:07 |
| Ryan_Lane | the good thing is that we're using almost nothing bug wikimedia code and that will just work with upgrades | 20:07 |
| Ryan_Lane | s/bug/but/ | 20:07 |
| * Ryan_Lane yawns | 20:09 | |
| clarkb | is the alternative to the wmf branches using "stable" releases and hoping that the plugins all work? | 20:10 |
| Ryan_Lane | yes | 20:10 |
| Ryan_Lane | which sucks | 20:10 |
| Ryan_Lane | extension management has always been the worst part of using MW | 20:10 |
| Ryan_Lane | there's some movement to fix that, but I'll believe it when I see it | 20:10 |
| clarkb | this seems like a reasonable solution even if it does mean we sit and wait for all of the submodules to update :) | 20:11 |
| Ryan_Lane | if we keep these more up to date more often this takes less time | 20:11 |
| Ryan_Lane | we could actually run a cron with: git fetch && git submodule fetch | 20:12 |
| Ryan_Lane | and this would take basically no time | 20:12 |
| clarkb | Ryan_Lane: ++ | 20:12 |
| clarkb | all done | 20:12 |
| *** hashar has joined #openstack-infra | 20:12 | |
| Ryan_Lane | ok... | 20:13 |
| Ryan_Lane | wikimedia occasionally deprecates extensions | 20:13 |
| *** woodspa has quit IRC | 20:13 | |
| Ryan_Lane | if we are using them, we should stop. if we aren't using them, we just delete them | 20:13 |
| fungi | russellb: did anyone get back to you on the srv rrs? your dig command was missing a . | 20:13 |
| fungi | russellb: dig -t SRV _sip._udp.openstack.org | 20:14 |
| russellb | oh, oops | 20:14 |
| Ryan_Lane | the other slot has a live-hacked strapping skin | 20:14 |
| Ryan_Lane | so I ensured it's copied | 20:14 |
| russellb | thanks. d'oh | 20:14 |
| fungi | russellb: also the _tcp version is there as well | 20:14 |
| Ryan_Lane | otherwise I'd just do a git pull | 20:14 |
| russellb | fungi: yep see it now :-) | 20:14 |
| fungi | no worries | 20:14 |
| * fungi goes back to being all conferency | 20:15 | |
| clarkb | Ryan_Lane: so you track master with strapping? | 20:15 |
| clarkb | Ryan_Lane: and the other extensions looks like. got it | 20:15 |
| *** nayward has quit IRC | 20:15 | |
| Ryan_Lane | well.... | 20:15 |
| Ryan_Lane | I had pulled in some security fixes for opend | 20:15 |
| Ryan_Lane | *openid | 20:15 |
| Ryan_Lane | they are in master now | 20:16 |
| clarkb | Ryan_Lane: and they are causing the merge conflict? | 20:16 |
| Ryan_Lane | I had cherry-picked them when they weren't yet merged | 20:16 |
| clarkb | so a git reset --hard origin/master is needed? | 20:16 |
| * Ryan_Lane has early access | 20:16 | |
| Ryan_Lane | they had been modified before being merged | 20:16 |
| Ryan_Lane | so the changes conflicted | 20:16 |
| Ryan_Lane | I knew that was the case because I had done it | 20:17 |
| Ryan_Lane | it would be good to have a place to document these things | 20:17 |
| clarkb | Ryan_Lane: perhaps at /srv/mediawiki/NOTES? or /root/NOTES? | 20:17 |
| Ryan_Lane | yeah, that'll work | 20:17 |
| Ryan_Lane | extension authors for mediawiki usually try to ensure backwards compatibility | 20:18 |
| Ryan_Lane | and master is meant to be usable | 20:18 |
| Ryan_Lane | so doing a git pull on the other extensions is usually fine | 20:18 |
| *** hashar has quit IRC | 20:19 | |
| Ryan_Lane | ok. now all of them are updated | 20:19 |
| Ryan_Lane | let me write a quick script that we should puppetize | 20:19 |
| Ryan_Lane | the backup script using /srv/mediawiki/w is fine, since it doesn't do anything with git | 20:21 |
| *** DinaBelova has quit IRC | 20:21 | |
| clarkb | Ryan_Lane: my paranoia wonders if the script will be runnable from a web browser | 20:21 |
| Ryan_Lane | and we want to reference the running version of MW | 20:21 |
| Ryan_Lane | nope | 20:21 |
| Ryan_Lane | we alias w and images directly | 20:22 |
| Ryan_Lane | /srv/mediawiki is not accessible | 20:22 |
| clarkb | ok | 20:22 |
| * Ryan_Lane always aliases things in and hates modifying doc roots | 20:22 | |
| *** hashar has joined #openstack-infra | 20:22 | |
| clarkb | cool I see that in the vhost | 20:23 |
| Ryan_Lane | ok… so, now to backup | 20:23 |
| Ryan_Lane | there could be schema upgrades with the upgrade, so we want to ensure we have a current backup | 20:23 |
| Ryan_Lane | schema upgrades are backwards compatible | 20:23 |
| Ryan_Lane | but it's still a good idea :) | 20:24 |
| openstackgerrit | Diane Fleming proposed a change to openstack-infra/config: Update yaml file to add new OpenStack End User Guide https://review.openstack.org/38694 | 20:27 |
| clarkb | Ryan_Lane: are we at a point where we can change the symlink? do we have to stop apache before doing that? | 20:27 |
| Ryan_Lane | one sec. talking to EFF | 20:28 |
| clarkb | np | 20:28 |
| clarkb | brb | 20:29 |
| *** koolhead17 has quit IRC | 20:29 | |
| *** sdake has joined #openstack-infra | 20:29 | |
| *** sdake has quit IRC | 20:29 | |
| *** sdake has joined #openstack-infra | 20:29 | |
| clarkb | back | 20:32 |
| *** sdake has quit IRC | 20:32 | |
| clarkb | sarob: I reviewed that change. There is one tiny nit pick I would like to see cleaned up, I can approve it once that is done | 20:35 |
| *** sarob has quit IRC | 20:35 | |
| *** sarob has joined #openstack-infra | 20:35 | |
| Ryan_Lane | sorry. in person EFF ambush | 20:36 |
| *** sdake has joined #openstack-infra | 20:37 | |
| *** sdake has quit IRC | 20:37 | |
| *** sdake has joined #openstack-infra | 20:37 | |
| clarkb | sarob: let me know if you can't make that change because you are at a conference or something. I can clean it up and push it through | 20:38 |
| clarkb | Ryan_Lane: no problem | 20:38 |
| *** woodspa has joined #openstack-infra | 20:40 | |
| fungi | Ryan_Lane: that's one of the coolest excuses ever | 20:42 |
| Ryan_Lane | :D | 20:42 |
| * fungi wants to get ambushed by the eff sometimwe | 20:42 | |
| *** SergeyLukjanov has quit IRC | 20:42 | |
| clarkb | fungi: can you review sarob's change too? | 20:42 |
| lifeless | sdake: mordred: I accept regular patches, no need for bzr. | 20:42 |
| Ryan_Lane | them: "Hi. When is https by default for all wikipedia users happening?" | 20:42 |
| sdague | clarkb: badness? | 20:42 |
| clarkb | fungi: its fine if not, just a new jenkins job | 20:42 |
| sdague | https://review.openstack.org/#/c/38691/ | 20:42 |
| Ryan_Lane | me: "NIce to meet you too." | 20:42 |
| openstackgerrit | colinmcnamara proposed a change to openstack-infra/config: Update CI systems to validate and publish section openstack-training https://review.openstack.org/38502 | 20:42 |
| clarkb | oh colinmcnamara addressed my comment \o/ | 20:43 |
| fungi | clarkb: listening to frencil from sparkfun talk about open hardware, but can multi-task for a bit | 20:43 |
| clarkb | sdague: that looks like the same bug dansmith ran into :/ | 20:43 |
| sdague | clarkb: ok, wonder what's making it worse today | 20:44 |
| clarkb | sdague: the key bit is the inprogress job failing at the beginning of the console outputs | 20:44 |
| fungi | Ryan_Lane: ha. that's classic | 20:44 |
| clarkb | sdague: its related to test load, more tests = less d-g slaves = higher chance of d-g launch jobs timing out | 20:44 |
| sdague | right, fair | 20:44 |
| clarkb | Ryan_Lane: ready to go when you are even if it looks like I am busy multitasking :) | 20:44 |
| Ryan_Lane | maybe 10 mins | 20:44 |
| *** sdake has quit IRC | 20:46 | |
| openstackgerrit | Thomas Goirand proposed a change to openstack/requirements: Removes the cap for SQLAlchemy https://review.openstack.org/38035 | 20:46 |
| *** jjmb has joined #openstack-infra | 20:47 | |
| *** dkranz has quit IRC | 20:48 | |
| sdague | lifeless: working on and reworking patches outside of an scm it really manual. I'd rather just keep subunit2html over in gerrit if it means loosing git to work on it to move it elsewhere. | 20:49 |
| *** sdake has joined #openstack-infra | 20:50 | |
| *** sdake has quit IRC | 20:50 | |
| *** sdake has joined #openstack-infra | 20:50 | |
| clarkb | sdague: jeblair just approved my change that should hopefully make d-g nodes happier | 20:51 |
| *** sdake has quit IRC | 20:51 | |
| *** Ryan_Lane has quit IRC | 20:51 | |
| *** sdake has joined #openstack-infra | 20:51 | |
| *** sdake has quit IRC | 20:51 | |
| *** sdake has joined #openstack-infra | 20:51 | |
| clarkb | hopefully the gate clears up long enough for it to get through (there have been issues with some of the tests in there not related to the derpyness we are seeing | 20:51 |
| sdague | clarkb: awesome | 20:51 |
| clarkb | if necessary I can shove it in with extra force | 20:51 |
| sdague | d-g kick to the groin! | 20:52 |
| *** Ryan_Lane has joined #openstack-infra | 20:52 | |
| *** jjmb1 has joined #openstack-infra | 20:52 | |
| Ryan_Lane | clarkb: ok. ready to keep going | 20:52 |
| clarkb | Ryan_Lane: I am ready | 20:53 |
| Ryan_Lane | ok | 20:53 |
| Ryan_Lane | so, after doing a backup, it's necessary to run mediawiki's update script | 20:53 |
| Ryan_Lane | you run it before you switch the software over | 20:53 |
| Ryan_Lane | since the old software will work with the new schema | 20:53 |
| Ryan_Lane | (I wish OpenStack worked like this ;) ) | 20:53 |
| clarkb | Ryan_Lane: can you document that php command here in the channel? | 20:54 |
| Ryan_Lane | so, notice: https://wiki.openstack.org/wiki/Special:Version | 20:54 |
| Ryan_Lane | sure. | 20:54 |
| clarkb | (the channel is logged) | 20:54 |
| Ryan_Lane | it's php maintenance/update.php --quick | 20:54 |
| clarkb | run within the slot you are upgrading | 20:54 |
| Ryan_Lane | from inside of the slot you are upgrading | 20:54 |
| Ryan_Lane | indeed | 20:54 |
| *** jjmb has quit IRC | 20:54 | |
| Ryan_Lane | note that the wiki is still up and it running the older version | 20:54 |
| clarkb | yup | 20:55 |
| Ryan_Lane | then you just switch the link | 20:55 |
| Ryan_Lane | and check https://wiki.openstack.org/wiki/Special:Version again | 20:55 |
| Ryan_Lane | new version is running. no downtime | 20:55 |
| clarkb | nice, so restarting apache is not necessary? | 20:56 |
| Ryan_Lane | nope | 20:56 |
| Ryan_Lane | because we just switch the link | 20:56 |
| Ryan_Lane | and the link is the alias | 20:56 |
| *** jjmb has joined #openstack-infra | 20:57 | |
| Ryan_Lane | assuming you don't make any live hacks or pull in pre-merge security fixes, upgrading is very straightforward | 20:57 |
| clarkb | perfect, I feel like I have a much better grasp of this now. Though if something goes wrong I think my abilities to debug and correct are still minimal but this is a start | 20:57 |
| Ryan_Lane | great | 20:58 |
| Ryan_Lane | there's a few things that should get puppetized | 20:58 |
| Ryan_Lane | I should open bugs | 20:58 |
| clarkb | ++ | 20:58 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Increase d-g launch timeout to 20 min https://review.openstack.org/38701 | 20:58 |
| Ryan_Lane | we should have backups running in a cron | 20:58 |
| clarkb | Ryan_Lane: dtroyer just mentioned that he noticed much larger margins after yesterdays upgrade | 20:59 |
| Ryan_Lane | and we should puppetize the wiki config | 20:59 |
| Ryan_Lane | margins? | 20:59 |
| jeblair | clarkb: gah you already have a patch for that don't you? | 20:59 |
| clarkb | Ryan_Lane: page margins I guess so skin related maybe? | 20:59 |
| *** jjmb1 has quit IRC | 20:59 | |
| clarkb | jeblair: I do but I only did 15 minutes. we can useyours | 20:59 |
| Ryan_Lane | ah, meaning the page content area is smaller? | 20:59 |
| openstackgerrit | colinmcnamara proposed a change to openstack-infra/config: Update CI systems to validate and publish section openstack-training https://review.openstack.org/38502 | 20:59 |
| Ryan_Lane | it's possible. I'm using an unpatched upstream version now. it's the same skin used by redhat for their openstack wiki | 21:00 |
| clarkb | Ryan_Lane: that was how I grokked it | 21:00 |
| clarkb | Ryan_Lane: I have detached from screen. feel free to close it anytime | 21:02 |
| jeblair | docs-draft rsync is a 58% done | 21:02 |
| Ryan_Lane | cool | 21:02 |
| clarkb | Ryan_Lane: and thank you for running me through that | 21:02 |
| dtroyer | Ryan_Lane: right, the left and right margins are now together over 1/4 the page width | 21:02 |
| Ryan_Lane | clarkb: totally welcome | 21:02 |
| Ryan_Lane | dtroyer: hm. I'll need to take a look at the CSS to see why | 21:02 |
| dtroyer | Ryan_Lane: ok, thanks. my command tables are wrapping like crazy now… ;) | 21:03 |
| Ryan_Lane | heh | 21:03 |
| Ryan_Lane | oh wow. awesome. | 21:03 |
| Ryan_Lane | I forgot that this upgrade allows editing from mobile skin now | 21:04 |
| dtroyer | that's a plus | 21:04 |
| clarkb | jeblair: fungi mordred I plan to go through the IRC logs from today and write up a ci.openstack.org doc on this process | 21:04 |
| clarkb | I figure hosting it not on the wiki is a good thing :) | 21:04 |
| fungi | clarkb: agreed. don't want a catch-22 outage to push me to read instructions through the wayback machine | 21:05 |
| *** rfolco has quit IRC | 21:05 | |
| clarkb | fungi: https://review.openstack.org/#/c/38701/ if you approve that I can babysit | 21:06 |
| jeblair | clarkb: thanks! | 21:07 |
| *** krtaylor has quit IRC | 21:08 | |
| *** sarob has quit IRC | 21:08 | |
| fungi | clarkb: lgtm. approved | 21:11 |
| * Ryan_Lane grumbles | 21:11 | |
| *** sarob has joined #openstack-infra | 21:12 | |
| Ryan_Lane | mobile editing doesn't work with openid | 21:12 |
| Ryan_Lane | so, yeah…. | 21:12 |
| clarkb | oh well I guess | 21:12 |
| *** jjmb has quit IRC | 21:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: Increase d-g launch timeout to 20 min https://review.openstack.org/38701 | 21:13 |
| *** jjmb has joined #openstack-infra | 21:15 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Enable warnerrors for pbr to catch Sphinx warnings https://review.openstack.org/38006 | 21:17 |
| clarkb | jog0: why not make rootwrap a small !python thing? | 21:18 |
| clarkb | it might do weird things to dependencies I suppose | 21:18 |
| jog0 | clarkb: I think that is an option | 21:18 |
| jog0 | btw http://logs.openstack.org/17/36617/2/gate/gate-nova-python27/29976/console.html | 21:19 |
| jog0 | nova unit tests are borking left and right | 21:19 |
| clarkb | yeah I think someone logged an issue again openstack-ci about it and I haven't had time to swing around and reassign it to nova | 21:19 |
| jog0 | clarkb: I cannot reproduce locally at least not yet | 21:20 |
| jog0 | haven't done a tox -r yet | 21:20 |
| clarkb | jog0: you can try grabbing the subunit log from one of the failed tests then `testr load $fileyoudownloaded && testr run --analyze-isolation` | 21:20 |
| clarkb | jog0: that will use the test partitions that were used when it failed. | 21:21 |
| jog0 | hmm | 21:21 |
| jog0 | just ran tox -r .... so will have to wait | 21:21 |
| clarkb | and it will try to tell you which tests if any conflict with it | 21:21 |
| clarkb | jog0: ok, no rush | 21:21 |
| clarkb | (I would have to build a new nova dev env to do it myself so you are ahead of me) | 21:21 |
| jog0 | clarkb: heh | 21:21 |
| *** DinaBelova has joined #openstack-infra | 21:22 | |
| openstackgerrit | colinmcnamara proposed a change to openstack-infra/config: Update CI systems to validate and publish section openstack-training https://review.openstack.org/38502 | 21:22 |
| clarkb | jog0: the other thing I like to do is `testr run $test_name` a lot of tests fail when run on their own and if they do knowing that makes debugging eaiser | 21:22 |
| *** jjmb has quit IRC | 21:22 | |
| jog0 | yeah I tried that no luck | 21:22 |
| jog0 | I hate it when things don't fail | 21:22 |
| openstackgerrit | colinmcnamara proposed a change to openstack-infra/config: Update CI systems to validate and publish section openstack-training https://review.openstack.org/38502 | 21:26 |
| *** DinaBelova has quit IRC | 21:27 | |
| jog0 | clarkb: I tried testr load and connot reproduce | 21:29 |
| clarkb | :/ | 21:29 |
| fungi | clarkb: jeblair: on https://review.openstack.org/38066 i guess the real problem is that jeepyb needs its setup.py updated to require a newer (non-broken) pbr? | 21:30 |
| jog0 | ' testr run --analyze-isolation ' did nothing | 21:30 |
| jog0 | ohh I bet this is a python version thing | 21:30 |
| jog0 | saw one earlier | 21:30 |
| clarkb | jog0: analyze-isolation should run off and run the test alone, if that passes it bisects the test list until it can find a minimal set of tests that when run together result in the failure. It is possible that it is non deterministic beyond just test order | 21:31 |
| jog0 | clarkb: it ran nothig not sure why | 21:32 |
| *** DinaBelova has joined #openstack-infra | 21:32 | |
| clarkb | jog0: did the testr load load anything? | 21:32 |
| jog0 | clarkb: I think so | 21:33 |
| clarkb | --analyize-isolation runs based on the previous run. load is a way of changing out the previous run | 21:33 |
| *** bclifford has joined #openstack-infra | 21:33 | |
| jog0 | let me paste my terminal output | 21:33 |
| clarkb | fungi: thats possible. do we need to be less restrictive in the setup.py? | 21:34 |
| jog0 | testr load says Ran tests in 37.906s ; then PASSED | 21:34 |
| openstackgerrit | A change was merged to openstack-infra/config: Update CI systems to validate and publish section openstack-training https://review.openstack.org/38502 | 21:34 |
| jog0 | i did this inside the tox py27 venv | 21:34 |
| clarkb | jog0: it should say failed if the job failed. you grabbed http://logs.openstack.org/63/36363/17/check/gate-nova-python26/29748/subunit_log.txt.gz and gunzipped it? | 21:35 |
| jog0 | anyway I gotta jet (boarding one soon) | 21:35 |
| jog0 | I did http://logs.openstack.org/36/36436/5/check/gate-nova-python│ | 21:35 |
| jog0 | and gunzipped yeah | 21:35 |
| jog0 | kk out for real | 21:35 |
| fungi | clarkb: mmm, no. the problem seems to be that relying on setup.py to install dependencies doesn't work when it runs with an older version of that dependency which prevents setup.py from running... chicken and egg problem | 21:36 |
| clarkb | fungi: so it isn't downloading the newer version because the one it already has is sufficient? | 21:36 |
| clarkb | we can probably do a git clean to fix that | 21:36 |
| *** DinaBelova has quit IRC | 21:37 | |
| fungi | clarkb: possibly, though i think it's more porobably that setup.py is throwing an exception due to the broken pbr before it gets far enough to upgrade pbr | 21:39 |
| fungi | yeesh, not enough internets | 21:39 |
| clarkb | fungi: yeah but by removing broken pbr first it should grab a new one | 21:40 |
| clarkb | and hopefulyl bypass the brokeness | 21:40 |
| fungi | so change the jeepyb puppet module exec to pip uninstall pbr||true before running jeepyb's setup.py? | 21:41 |
| *** rnirmal has quit IRC | 21:42 | |
| fungi | i know we can manually do that to break the deadlock, but not sure if i need to think about others who might also use this module (who aren't us), or other servers where we haven't noticed this is broken, or in the future if the same sort of issue reemerges | 21:44 |
| clarkb | fungi: yeah something like that may be necessary | 21:44 |
| fungi | the fact that jeepyb uses pbr before possibly upgrading pbr leaves it open to this sort of issue, in a way which may be particular to pbr | 21:46 |
| clarkb | yeah I think this is definitely a pbr type of problem, because it is required to bootstrap itself | 21:46 |
| fungi | maybe mordred has an opinion, if he's got an internet at the moment | 21:46 |
| clarkb | fungi: you should just hunt him down in the conference center :) | 21:47 |
| *** changbl has quit IRC | 21:47 | |
| fungi | i will in fact see his talk in about an hour | 21:47 |
| jeblair | yes, let's beat on mordred because this is sposed to work imho | 21:47 |
| fungi | so i can merely abuse him with it then | 21:47 |
| jeblair | (if we have to do something yucky to fix this one deadlock case, ok, but we should avoid accepting this as a systemic problem) | 21:48 |
| fungi | i agree | 21:48 |
| clarkb | jeblair: fungi: I intend on approving BobBall's meetbot change tomorrow when meetings are quiet | 21:50 |
| jeblair | clarkb: link? | 21:50 |
| fungi | i should look it over then, if i haven't already | 21:50 |
| clarkb | jeblair: fungi https://review.openstack.org/#/c/36825/ | 21:51 |
| jeblair | clarkb: thx; was not watching meetbot repo | 21:51 |
| *** UtahDave has quit IRC | 21:53 | |
| openstackgerrit | Anne Gentle proposed a change to openstack-infra/config: Update yaml file to add new OpenStack End User Guide https://review.openstack.org/38694 | 21:53 |
| *** UtahDave has joined #openstack-infra | 21:54 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Stop trying to install old pip and setuptools https://review.openstack.org/38647 | 21:56 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Install jeepyb and requirements from REPODIR https://review.openstack.org/38184 | 21:56 |
| *** alexpilotti has joined #openstack-infra | 21:57 | |
| clarkb | sdague: I reviewed your htmlifier change | 21:58 |
| *** dhellmann is now known as dhellmann_ | 21:59 | |
| *** vipul is now known as vipul-away | 22:01 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Swap the order of configparser imports https://review.openstack.org/38653 | 22:03 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Handle Jenkins node already being added to Jenkins https://review.openstack.org/38674 | 22:04 |
| clarkb | \o/ zuul is merging all the things | 22:04 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Report an unknown time if a change ahead has failed https://review.openstack.org/38715 | 22:05 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Add missing projects to pre-cache list https://review.openstack.org/38180 | 22:06 |
| clarkb | jeblair: I think the recent zuul improvements are awesome btw. On the fly reconfigs, queue priority, kicking out potentially unmergable changes and so on | 22:07 |
| clarkb | despite the ridiculous test load zuul itself seems to be cool and collected. | 22:07 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: change date link to something more understandable https://review.openstack.org/38680 | 22:07 |
| sdague | clarkb: thanks | 22:07 |
| sdague | code review is so useful to prevent me from doing stupid things :) | 22:08 |
| sdague | ok, time to ride my bike home, see you guys tomorrow | 22:08 |
| jeblair | clarkb: thx; wait till you see the new status screen. :) | 22:08 |
| clarkb | jeblair: are you going to draw the gate queue? | 22:09 |
| lifeless | sdague: so moving stuff to git is possible, need tuits | 22:10 |
| jeblair | clarkb: hehe, not yet. :) just this: https://review.openstack.org/#/c/38598/ | 22:10 |
| * clarkb is waiting for the next launch jobs to kick off now that the d-g chang merged | 22:10 | |
| * fungi has seen it, is excited | 22:10 | |
| clarkb | jeblair: shiny | 22:10 |
| lifeless | sdague: doesn't git have a bzr translator though? | 22:10 |
| *** vijendar has quit IRC | 22:11 | |
| lifeless | all that said though, subunit2html is an infra thing; I don't particularly care where it lives - there's no 'consolidate all users of libfoo into projectfoo' meme | 22:11 |
| lifeless | I have a bunch of generic filters in subunit but am enthusiastic for others to live whereever they want to | 22:12 |
| *** mrodden has quit IRC | 22:17 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add progress and estimated completion times to zuul status https://review.openstack.org/38598 | 22:17 |
| openstackgerrit | Evgeny Fadeev proposed a change to openstack-infra/askbot-theme: changed css for the language nav https://review.openstack.org/38717 | 22:18 |
| *** vipul-away is now known as vipul | 22:18 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Make it easier to get to the cacti graphs https://review.openstack.org/38718 | 22:19 |
| *** prad has quit IRC | 22:19 | |
| clarkb | jeblair: comments on patchset two of 38598 | 22:21 |
| jeblair | ttx: http://en.wikipedia.org/wiki/Conway%27s_law | 22:21 |
| clarkb | jeblair: whoops I accidentally approved that zuul change, I removed my +A so it won't actually merge | 22:24 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add progress and estimated completion times to zuul status https://review.openstack.org/38598 | 22:26 |
| jeblair | clarkb: thx | 22:26 |
| *** pentameter has quit IRC | 22:29 | |
| *** jhesketh__ has quit IRC | 22:31 | |
| *** DinaBelova has joined #openstack-infra | 22:32 | |
| *** mrodden has joined #openstack-infra | 22:32 | |
| *** rahmu_ has joined #openstack-infra | 22:33 | |
| *** BobBall_ has joined #openstack-infra | 22:33 | |
| *** jhesketh__ has joined #openstack-infra | 22:33 | |
| *** alaski_ has joined #openstack-infra | 22:33 | |
| ttx | jeblair: what about conway's law ? | 22:34 |
| *** comstud_ has joined #openstack-infra | 22:34 | |
| *** rockstar` has joined #openstack-infra | 22:34 | |
| *** adarazs_ has joined #openstack-infra | 22:34 | |
| *** adarazs has quit IRC | 22:34 | |
| *** rockstar has quit IRC | 22:34 | |
| *** rahmu has quit IRC | 22:34 | |
| *** alaski has quit IRC | 22:34 | |
| *** BobBall has quit IRC | 22:34 | |
| *** comstud has quit IRC | 22:34 | |
| *** rahmu_ is now known as rahmu | 22:34 | |
| *** DinaBelova has quit IRC | 22:37 | |
| jeblair | ttx: nothing in particular, just found it interesting. :) | 22:40 |
| *** hashar has quit IRC | 22:40 | |
| *** alexpilotti has quit IRC | 22:41 | |
| openstackgerrit | A change was merged to openstack-infra/askbot-theme: changed css for the language nav https://review.openstack.org/38717 | 22:42 |
| clarkb | jeblair: https://review.openstack.org/#/c/38507/1/tests/test_scheduler.py the reset back to head doesn't break anything because the fake change is referred to via a different ref? | 22:42 |
| clarkb | (in this case r) | 22:42 |
| *** comstud_ is now known as comstud | 22:43 | |
| *** danger_fo is now known as danger_fo_away | 22:44 | |
| *** pcrews has quit IRC | 22:46 | |
| *** vipul is now known as vipul-away | 22:49 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Delay initialization of local repos https://review.openstack.org/38507 | 22:55 |
| clarkb | the launch jobs haven't exploded since my patch went in. I think we are good | 22:55 |
| jeblair | clarkb: yeah, r points to the fake commit | 22:56 |
| jeblair | clarkb: then the 'hexsha' attr of that gets used later | 22:57 |
| *** vipul-away is now known as vipul | 22:57 | |
| *** zul has joined #openstack-infra | 22:58 | |
| *** CaptTofu has quit IRC | 22:59 | |
| *** pcrews has joined #openstack-infra | 23:00 | |
| *** sdake has quit IRC | 23:00 | |
| *** avtar has quit IRC | 23:00 | |
| lifeless | so whats needed to get https://review.openstack.org/#/c/38543/ landed ? | 23:00 |
| clarkb | lifeless: I want jeblair to review it | 23:04 |
| clarkb | lifeless: if he is ok with it I can approve it | 23:04 |
| clarkb | and do the babysitting and so on. | 23:04 |
| clarkb | jeblair: ^ | 23:04 |
| *** sdake has joined #openstack-infra | 23:05 | |
| *** changbl has joined #openstack-infra | 23:09 | |
| *** beagles has joined #openstack-infra | 23:10 | |
| jeblair | clarkb, lifeless: the line above the find command is where we delete a list of files generated by this run of the mirror script | 23:11 |
| jeblair | clarkb, lifeless: (which is an intermediate step to removing the code that generates the list of files from this run of the mirror script) | 23:11 |
| lifeless | jeblair: the line above deletes the html indices that would collide with each other | 23:14 |
| lifeless | jeblair: the line I added generates a per-source list that doesn't conflict | 23:14 |
| lifeless | jeblair: I'd also be happy with alternatives like 'use rsync' if you wanted to enable anonymous rsync on the server | 23:15 |
| lifeless | jeblair: but spidering is super slow, really don't want to do that. | 23:15 |
| *** bclifford has left #openstack-infra | 23:18 | |
| sarob | last manuals build failed | 23:19 |
| sarob | https://jenkins.openstack.org/job/manuals-upstream-translation-update/243/ | 23:19 |
| sarob | sad panda | 23:20 |
| sarob | can i bug someone to take a look? | 23:21 |
| clarkb | sarob: thats the translation update. is that the one you are wanting us to look into? looks like transifex has done something funny with their certs | 23:22 |
| sarob | yeah, i saw that. also failed on training-manuals | 23:22 |
| sarob | but i cant find where in the output | 23:22 |
| clarkb | sarob: https://jenkins.openstack.org/job/openstack-training/1/console looks like it failed to parse something? | 23:23 |
| clarkb | sarob: SEVERE: Unparseable XPointer: controller-os | 23:25 |
| sarob | hmm, found it | 23:25 |
| *** pcrews has quit IRC | 23:26 | |
| jeblair | lifeless: *nod* this feels like it is possible that it is in conflict with the way we decided to handle the canonical mirror file list. it may be okay, but it's giving me pause right now, and i'm listening to monty talk about tripleo atm. | 23:27 |
| clarkb | sarob: does that get you sorted? | 23:28 |
| sarob | no unfortunately | 23:28 |
| clarkb | sarob: does it continue to fail after fixing the controller-os thing? | 23:29 |
| sarob | my local maven doesnt handle remote include statement, but the ci was | 23:29 |
| sarob | the section exists | 23:29 |
| sarob | so the error is wierd | 23:29 |
| clarkb | oh are the docs referring back to github to build themselves? | 23:30 |
| sarob | yeah, using <xi:include href="https://raw.github.com/openstack/openstack-manuals/... | 23:31 |
| sarob | it was working | 23:31 |
| clarkb | in general you probably don't want to depend on github for anything build related. github isn't reliable enough | 23:31 |
| clarkb | I wonder if it tried fetching something from github and failed because github 404'd or 500'd | 23:32 |
| sarob | its the only way i know of bringing the manuals content | 23:32 |
| sarob | i dont think so | 23:32 |
| clarkb | isn't the content in the current repo? | 23:32 |
| sarob | it seems to have a problem with using include xpointer statement | 23:32 |
| clarkb | I am probably misunderstanding something fundamental | 23:33 |
| sarob | for this one yes | 23:33 |
| *** DinaBelova has joined #openstack-infra | 23:33 | |
| sarob | alot of the content in the near future will be from different repos | 23:33 |
| jeblair | sarob: what repos? | 23:33 |
| sarob | openstack-operations | 23:34 |
| sarob | and some others | 23:34 |
| sarob | idea is to not recreate existing content for the training manuals | 23:34 |
| sarob | use existing as much as possible | 23:34 |
| jeblair | clarkb: this actually sounds like it needs d-g style integration to do properly. | 23:35 |
| clarkb | jeblair: well, they don't run in the gate queue so they inherently have race conditions | 23:36 |
| *** sgviking has quit IRC | 23:36 | |
| clarkb | jeblair: so they can probably get away with referring to remote things that are not github. eg git.o.o | 23:37 |
| sarob | explain? | 23:37 |
| clarkb | sarob: by referring to other repos it is possible that you will get outdated content if we don't properly order things | 23:38 |
| *** DinaBelova has quit IRC | 23:38 | |
| jeblair | clarkb: yes, but check/gate tests/drafts should; and somewhat awkwardly the post queue for them would need to observe strict ordering. | 23:38 |
| *** pcrews has joined #openstack-infra | 23:38 | |
| jeblair | clarkb: i agree that a good first step is to ignore that problem and just refer to a reliable source | 23:38 |
| sarob | oh, i get that | 23:38 |
| sarob | no other way to reuse content | 23:38 |
| sarob | we will finalize right after release docs are completed and static | 23:39 |
| jeblair | (making a job that assembles multiple repos non-concurrent would probably be good enough for the post queue) | 23:41 |
| *** mjfork_ has joined #openstack-infra | 23:44 | |
| *** mjfork has quit IRC | 23:44 | |
| *** mjfork_ is now known as mjfork | 23:44 | |
| jeblair | sarob, clarkb: maybe the best solution for now would be to 'git clone --depth 1' other needed repos | 23:45 |
| *** mjfork has quit IRC | 23:45 | |
| *** mjfork has joined #openstack-infra | 23:45 | |
| clarkb | jeblair: maybe. I don't think that addressed the xpointer issue though. I am not convinced it was a reliability of github problme | 23:45 |
| clarkb | unless it got back a file without the controller-os id in it because github can be unreliable | 23:46 |
| sarob | at time of build, how would local repo help? | 23:46 |
| jeblair | clarkb: ok, but that is also an issue. | 23:46 |
| clarkb | sarob: because the files would already be present and maven won't have to retrive them manually which may fail | 23:46 |
| clarkb | sarob: particularly when talking to github | 23:46 |
| sarob | so i would bring in all the repos in the repo i am working on? | 23:47 |
| *** lbragstad1 has quit IRC | 23:47 | |
| sarob | sounds messier that using github | 23:47 |
| clarkb | sarob: yes, prior to running the maven build have a step that clones all of your dependencies | 23:47 |
| clarkb | it is a little messy but far more debuggable | 23:48 |
| sarob | true | 23:48 |
| jeblair | sarob: trust us -- github fails all the time at the rate we run jobs | 23:48 |
| clarkb | and if you are only using openstack repos you can fetch them from far more reliable sources | 23:48 |
| sarob | i want to make it as reliable as possible | 23:48 |
| sarob | so if i have 10 repos cloned | 23:49 |
| sarob | im working on a branch for a few days | 23:49 |
| sarob | very likely ill be behind some commits | 23:49 |
| clarkb | jeblair: do you think there is value in trying to run that post job again to see if that particular failure is consistent? might indicate if github is at fault | 23:50 |
| jeblair | clarkb: up to you; i don't have bandwidth to debug. i'm just saying that regardless, we want to stop using github. | 23:50 |
| clarkb | jeblair: indeed. I will go ahead and trigger it using the gearman client in zuul/tools/ | 23:51 |
| *** zehicle_at_dell has joined #openstack-infra | 23:53 | |
| clarkb | sarob: https://jenkins.openstack.org/job/openstack-training/2/console it is consistent | 23:54 |
| sarob | for this debuging i have removed all the remote github href | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!