| *** CaptTofu has quit IRC | 00:01 | |
| *** CaptTofu has joined #openstack-infra | 00:02 | |
| *** e0ne has joined #openstack-infra | 00:05 | |
| *** CaptTofu has quit IRC | 00:06 | |
| *** e0ne has quit IRC | 00:09 | |
| *** dcramer_ has quit IRC | 00:10 | |
| *** zehicle_at_dell has quit IRC | 00:14 | |
| *** zehicle_at_dell has joined #openstack-infra | 00:14 | |
| *** zehicle has quit IRC | 00:14 | |
| *** zehicle has joined #openstack-infra | 00:14 | |
| *** marktraceur has quit IRC | 00:18 | |
| *** matsuhashi has joined #openstack-infra | 00:18 | |
| *** wchrisj has joined #openstack-infra | 00:20 | |
| *** marktraceur has joined #openstack-infra | 00:20 | |
| *** marktraceur has quit IRC | 00:30 | |
| *** marktraceur has joined #openstack-infra | 00:35 | |
| *** yamahata has joined #openstack-infra | 00:36 | |
| *** marktraceur has quit IRC | 00:40 | |
| *** alexpilotti has quit IRC | 00:45 | |
| *** marktraceur has joined #openstack-infra | 00:47 | |
| *** wchrisj has quit IRC | 00:50 | |
| *** marktraceur has quit IRC | 00:52 | |
| *** oomichi has joined #openstack-infra | 00:53 | |
| *** marktraceur has joined #openstack-infra | 00:58 | |
| *** e0ne has joined #openstack-infra | 01:06 | |
| *** e0ne has quit IRC | 01:07 | |
| *** saschpe has quit IRC | 01:09 | |
| *** saschpe has joined #openstack-infra | 01:10 | |
| *** fifieldt has joined #openstack-infra | 01:19 | |
| *** matjazp has joined #openstack-infra | 01:21 | |
| *** matjazp has quit IRC | 01:23 | |
| *** nosnos has joined #openstack-infra | 01:30 | |
| *** zns has joined #openstack-infra | 01:33 | |
| *** wenlock has joined #openstack-infra | 01:36 | |
| *** pfallenop has quit IRC | 01:38 | |
| *** pfallenop has joined #openstack-infra | 01:40 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for nova unit test race bug 1303533 https://review.openstack.org/85625 | 01:40 |
|---|---|---|
| uvirtbot | Launchpad bug 1303533 in nova "nova.tests.integrated.v3.test_suspend_server.SuspendServerSamplesJsonTest.test_post_resume fails sporadically" [Undecided,New] https://launchpad.net/bugs/1303533 | 01:40 |
| *** wenlock has quit IRC | 01:42 | |
| *** zns has quit IRC | 01:45 | |
| *** yamahata has quit IRC | 01:45 | |
| *** wenlock has joined #openstack-infra | 02:02 | |
| *** e0ne has joined #openstack-infra | 02:05 | |
| *** e0ne has quit IRC | 02:06 | |
| *** wchrisj has joined #openstack-infra | 02:07 | |
| *** timrc-afk is now known as timrc | 02:20 | |
| *** wchrisj has quit IRC | 02:21 | |
| *** matjazp has joined #openstack-infra | 02:21 | |
| *** zz_gondoi is now known as gondoi | 02:22 | |
| *** matjazp has quit IRC | 02:26 | |
| *** chandan_kumar has joined #openstack-infra | 02:29 | |
| *** chandan_kumar has quit IRC | 02:30 | |
| *** Ryan_Lane has quit IRC | 02:34 | |
| *** jamielenz is now known as jamielennox|away | 02:37 | |
| *** jamielennox|away is now known as jamielennox | 02:38 | |
| *** esker has quit IRC | 02:42 | |
| *** imcsk8_lap has quit IRC | 02:43 | |
| *** imcsk8_lap has joined #openstack-infra | 02:43 | |
| *** matsuhashi has quit IRC | 02:47 | |
| *** YorikSar has quit IRC | 02:49 | |
| *** gondoi is now known as zz_gondoi | 02:55 | |
| *** asettle is now known as Alex-lunceon | 03:00 | |
| *** Alex-lunceon is now known as alex-luncheon | 03:00 | |
| *** nosnos has quit IRC | 03:02 | |
| *** matsuhashi has joined #openstack-infra | 03:04 | |
| *** amotoki has joined #openstack-infra | 03:04 | |
| *** Ryan_Lane has joined #openstack-infra | 03:05 | |
| *** e0ne has joined #openstack-infra | 03:05 | |
| *** dcramer_ has joined #openstack-infra | 03:08 | |
| *** Ryan_Lane has quit IRC | 03:09 | |
| *** e0ne has quit IRC | 03:09 | |
| *** nosnos has joined #openstack-infra | 03:20 | |
| *** matjazp has joined #openstack-infra | 03:21 | |
| *** dcramer_ has quit IRC | 03:23 | |
| *** matjazp has quit IRC | 03:24 | |
| *** matsuhashi has quit IRC | 03:29 | |
| *** chandan_kumar has joined #openstack-infra | 03:30 | |
| *** yamahata has joined #openstack-infra | 03:30 | |
| *** chandan_kumar has quit IRC | 03:31 | |
| *** chandan_kumar has joined #openstack-infra | 03:31 | |
| *** alff has joined #openstack-infra | 03:32 | |
| *** alff_ has joined #openstack-infra | 03:32 | |
| *** pcrews has joined #openstack-infra | 03:36 | |
| *** alex-luncheon is now known as asettle | 03:44 | |
| *** chandan_kumar has quit IRC | 03:48 | |
| *** zhiyan_ is now known as zhiyan | 03:50 | |
| *** nosnos has quit IRC | 03:51 | |
| *** e0ne has joined #openstack-infra | 04:05 | |
| *** e0ne has quit IRC | 04:08 | |
| *** chandan_kumar has joined #openstack-infra | 04:09 | |
| lifeless | mordred: sudo vim /etc/systemd/logind.conf | 04:09 |
| lifeless | HandleLidSwitch=ignore | 04:09 |
| *** YorikSar has joined #openstack-infra | 04:12 | |
| *** yfried has quit IRC | 04:14 | |
| *** miqui has quit IRC | 04:16 | |
| *** pcrews has quit IRC | 04:18 | |
| *** matjazp has joined #openstack-infra | 04:21 | |
| *** matjazp has quit IRC | 04:25 | |
| *** mihgen has joined #openstack-infra | 04:27 | |
| *** mihgen has quit IRC | 04:29 | |
| *** zhiyan is now known as zhiyan_ | 04:39 | |
| *** chandan_kumar has quit IRC | 05:02 | |
| *** mrmartin has joined #openstack-infra | 05:03 | |
| *** e0ne has joined #openstack-infra | 05:05 | |
| *** e0ne has quit IRC | 05:09 | |
| *** _nadya_ has joined #openstack-infra | 05:11 | |
| *** saju_m has joined #openstack-infra | 05:13 | |
| *** mihgen has joined #openstack-infra | 05:16 | |
| *** ildikov_ has quit IRC | 05:18 | |
| *** matjazp has joined #openstack-infra | 05:21 | |
| *** mrmartin has quit IRC | 05:23 | |
| *** _nadya_ has quit IRC | 05:24 | |
| *** matjazp has quit IRC | 05:25 | |
| *** e0ne has joined #openstack-infra | 05:26 | |
| *** matjazp has joined #openstack-infra | 05:27 | |
| *** chandan_kumar has joined #openstack-infra | 05:27 | |
| *** Guest96737 has quit IRC | 05:27 | |
| *** lillie has joined #openstack-infra | 05:28 | |
| *** vishy has quit IRC | 05:28 | |
| *** matjazp has quit IRC | 05:28 | |
| *** lillie is now known as Guest21911 | 05:28 | |
| *** jbryce has quit IRC | 05:28 | |
| *** skraynev_afk is now known as skraynev | 05:29 | |
| *** mrda has quit IRC | 05:29 | |
| *** adam_g has quit IRC | 05:29 | |
| *** stevebaker has quit IRC | 05:29 | |
| *** StevenK has quit IRC | 05:29 | |
| *** wendar_ has joined #openstack-infra | 05:29 | |
| *** pabelanger has quit IRC | 05:29 | |
| *** ryanpetrello has quit IRC | 05:29 | |
| *** lifeless has quit IRC | 05:30 | |
| *** thingee has quit IRC | 05:30 | |
| *** wendar has quit IRC | 05:30 | |
| *** StevenK has joined #openstack-infra | 05:30 | |
| *** lifeless has joined #openstack-infra | 05:31 | |
| *** e0ne has quit IRC | 05:31 | |
| *** thingee has joined #openstack-infra | 05:32 | |
| *** ryanpetrello has joined #openstack-infra | 05:32 | |
| *** jbryce has joined #openstack-infra | 05:32 | |
| *** mrda has joined #openstack-infra | 05:32 | |
| *** pabelanger has joined #openstack-infra | 05:33 | |
| *** stevebaker has joined #openstack-infra | 05:33 | |
| *** adam_g has joined #openstack-infra | 05:33 | |
| *** adam_g has quit IRC | 05:33 | |
| *** adam_g has joined #openstack-infra | 05:33 | |
| *** vishy has joined #openstack-infra | 05:34 | |
| *** yfried has joined #openstack-infra | 05:39 | |
| *** jlibosva has joined #openstack-infra | 05:51 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Support Matrix Tie Parent Plugin in wrappers https://review.openstack.org/85642 | 05:53 |
| *** pcrews has joined #openstack-infra | 05:57 | |
| *** Ryan_Lane has joined #openstack-infra | 05:58 | |
| *** VINOD has joined #openstack-infra | 06:01 | |
| *** VINOD has quit IRC | 06:03 | |
| *** jaypipes has joined #openstack-infra | 06:03 | |
| *** VINOD has joined #openstack-infra | 06:04 | |
| *** marun_afk has quit IRC | 06:08 | |
| *** alff has quit IRC | 06:08 | |
| *** alff has joined #openstack-infra | 06:09 | |
| *** ihrachyshka has joined #openstack-infra | 06:10 | |
| *** pcrews has quit IRC | 06:11 | |
| *** VINOD has quit IRC | 06:12 | |
| *** andreykurilin_ has joined #openstack-infra | 06:24 | |
| *** ildikov_ has joined #openstack-infra | 06:24 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to Git https://review.openstack.org/75931 | 06:27 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor if: nesting in build_mirror https://review.openstack.org/59961 | 06:27 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor calls to pip https://review.openstack.org/75945 | 06:27 |
| *** e0ne has joined #openstack-infra | 06:33 | |
| *** mkoderer has joined #openstack-infra | 06:34 | |
| *** rwsu has joined #openstack-infra | 06:35 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Allow to specify multiple branches per project in Gerrit trigger https://review.openstack.org/85648 | 06:37 |
| *** ihrachyshka has quit IRC | 06:40 | |
| *** dkliban has quit IRC | 06:41 | |
| *** saju_m has quit IRC | 06:41 | |
| *** matjazp has joined #openstack-infra | 06:42 | |
| *** e0ne has quit IRC | 06:43 | |
| *** e0ne has joined #openstack-infra | 06:44 | |
| *** afazekas_ has joined #openstack-infra | 06:47 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Allow to specify multiple branches per project in Gerrit trigger https://review.openstack.org/85648 | 06:48 |
| *** e0ne has quit IRC | 06:48 | |
| mikal | anteaya: check this out -- http://www.rcbops.com/gerrit/reports/neutron-cireport.html | 06:49 |
| mikal | So pretty and graphy | 06:50 |
| mattoliverau | mikal: purdy | 06:52 |
| *** oomichi has quit IRC | 06:52 | |
| mikal | mattoliverau: that's all Neill's doing | 06:53 |
| *** andreykurilin_ has quit IRC | 06:54 | |
| mattoliverau | mikal: Neill's the man, he makes us look much more professional :) | 06:54 |
| *** saju_m has joined #openstack-infra | 06:54 | |
| mikal | So true | 06:57 |
| mikal | That's really all of your jobs, to make me look competent | 06:57 |
| mikal | Get on that | 06:57 |
| mattoliverau | mikal: Sir, yes sir! | 06:57 |
| *** jhesketh has joined #openstack-infra | 06:58 | |
| *** e0ne has joined #openstack-infra | 06:58 | |
| mikal | LOL | 06:58 |
| *** afazekas has quit IRC | 06:59 | |
| *** Pradnya has joined #openstack-infra | 06:59 | |
| Pradnya | Hii | 07:01 |
| Pradnya | Anyone knows how to change commit titile? I am getting error in pep8 for commit title >50 char | 07:02 |
| *** chandan_kumar has quit IRC | 07:02 | |
| *** flaper87|afk is now known as flaper87 | 07:04 | |
| StevenK | Pradnya: git commit --amend | 07:04 |
| StevenK | That allows you to edit the commit message | 07:05 |
| mattoliverau | Pradnya: what StevenK said ^^ | 07:06 |
| mattoliverau | Night all, I'm going to call it a day.. dinner wont cook itself. | 07:07 |
| Pradnya | actually i have already submitted the code and its under review. While running Jenkin, pep8 gave error for commit title. So now can i change the commit title or i have to resubmit the code again | 07:07 |
| *** chandan_kumar has joined #openstack-infra | 07:07 | |
| Pradnya | Steaven :?? | 07:08 |
| StevenK | Pradnya: Okay, so on your git branch, you can git commit --amend to change your existing commit, leaving the Change-Id line alone, and then git review to resubmit it | 07:09 |
| Pradnya | okay | 07:10 |
| Pradnya | Steven: I will try that | 07:10 |
| *** dizquierdo has joined #openstack-infra | 07:17 | |
| *** matrohon has joined #openstack-infra | 07:18 | |
| *** jgallard has joined #openstack-infra | 07:22 | |
| *** ihrachyshka has joined #openstack-infra | 07:23 | |
| *** jcoufal has joined #openstack-infra | 07:25 | |
| *** Pradnya has quit IRC | 07:31 | |
| *** dkliban has joined #openstack-infra | 07:34 | |
| *** hashar has joined #openstack-infra | 07:36 | |
| *** derekh has joined #openstack-infra | 07:40 | |
| *** sahumada has quit IRC | 07:44 | |
| *** mrda has quit IRC | 07:45 | |
| *** derekh has quit IRC | 07:46 | |
| *** mrda has joined #openstack-infra | 07:46 | |
| *** mrda has left #openstack-infra | 07:46 | |
| *** alff has quit IRC | 07:53 | |
| *** markmc has joined #openstack-infra | 07:58 | |
| *** julienvey has joined #openstack-infra | 07:59 | |
| *** sahumada has joined #openstack-infra | 07:59 | |
| *** alff has joined #openstack-infra | 08:00 | |
| *** alff_ has quit IRC | 08:01 | |
| *** alff has quit IRC | 08:01 | |
| *** jpich has joined #openstack-infra | 08:07 | |
| *** jlibosva has quit IRC | 08:07 | |
| *** che-arne has joined #openstack-infra | 08:08 | |
| *** jlibosva has joined #openstack-infra | 08:09 | |
| *** eloib_ has joined #openstack-infra | 08:09 | |
| *** mkerrin has joined #openstack-infra | 08:14 | |
| *** che-arne has quit IRC | 08:15 | |
| *** matjazp has quit IRC | 08:16 | |
| *** nkinder has quit IRC | 08:22 | |
| *** nkinder has joined #openstack-infra | 08:27 | |
| *** miarmak has joined #openstack-infra | 08:30 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Tests for checkstyle publisher https://review.openstack.org/85561 | 08:30 |
| *** matsuhashi has joined #openstack-infra | 08:35 | |
| *** mihgen has quit IRC | 08:36 | |
| *** Ryan_Lane has quit IRC | 08:38 | |
| *** e0ne has quit IRC | 08:38 | |
| *** e0ne has joined #openstack-infra | 08:38 | |
| *** nosnos has joined #openstack-infra | 08:40 | |
| *** e0ne_ has joined #openstack-infra | 08:41 | |
| *** e0ne has quit IRC | 08:41 | |
| *** dizquierdo has quit IRC | 08:42 | |
| *** jlibosva has quit IRC | 08:45 | |
| *** jlibosva has joined #openstack-infra | 08:45 | |
| *** Ryan_Lane has joined #openstack-infra | 08:46 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Allow 'Build on the same node' in trigger-builds builder https://review.openstack.org/85668 | 08:51 |
| *** Ryan_Lane has quit IRC | 08:56 | |
| *** ominakov has joined #openstack-infra | 08:59 | |
| *** talluri has joined #openstack-infra | 09:01 | |
| *** jp_at_hp has joined #openstack-infra | 09:03 | |
| *** salv-orlando has joined #openstack-infra | 09:10 | |
| *** ihrachyshka has quit IRC | 09:15 | |
| *** ogelbukh has quit IRC | 09:17 | |
| *** mihgen has joined #openstack-infra | 09:18 | |
| *** che-arne has joined #openstack-infra | 09:21 | |
| *** e0ne_ has quit IRC | 09:25 | |
| *** e0ne has joined #openstack-infra | 09:26 | |
| *** ogelbukh has joined #openstack-infra | 09:26 | |
| *** dizquierdo has joined #openstack-infra | 09:27 | |
| *** saju_m has quit IRC | 09:27 | |
| *** ogelbukh has quit IRC | 09:28 | |
| *** e0ne has quit IRC | 09:28 | |
| *** nkinder has quit IRC | 09:28 | |
| *** e0ne has joined #openstack-infra | 09:28 | |
| *** ogelbukh has joined #openstack-infra | 09:28 | |
| *** alexpilotti has joined #openstack-infra | 09:40 | |
| *** yassine has joined #openstack-infra | 09:41 | |
| *** ihrachyshka has joined #openstack-infra | 09:48 | |
| *** ociuhandu has quit IRC | 09:59 | |
| *** horsson has quit IRC | 10:03 | |
| *** nkinder has joined #openstack-infra | 10:04 | |
| sdague | morning folks | 10:06 |
| openstackgerrit | Peter Jönsson proposed a change to openstack-infra/jenkins-job-builder: Add support for Stash as a repository viewer https://review.openstack.org/81218 | 10:11 |
| *** wenlock has quit IRC | 10:14 | |
| *** matsuhashi has quit IRC | 10:16 | |
| *** matsuhashi has joined #openstack-infra | 10:16 | |
| *** matsuhashi has quit IRC | 10:20 | |
| *** jgallard has quit IRC | 10:24 | |
| *** yolanda has joined #openstack-infra | 10:24 | |
| *** dkranz has quit IRC | 10:26 | |
| *** saju_m has joined #openstack-infra | 10:32 | |
| *** ociuhandu has joined #openstack-infra | 10:39 | |
| *** e0ne has quit IRC | 10:40 | |
| *** e0ne has joined #openstack-infra | 10:41 | |
| *** che-arne|2 has joined #openstack-infra | 10:41 | |
| *** jooools has joined #openstack-infra | 10:44 | |
| *** che-arne has quit IRC | 10:44 | |
| *** e0ne has quit IRC | 10:45 | |
| *** saju_m has quit IRC | 10:49 | |
| SergeyLukjanov | sdague, morning | 10:50 |
| SergeyLukjanov | sdague, looks like everything is ok with gate | 10:51 |
| sdague | yep | 10:51 |
| rcarrillocruz | guys, i'm trying to fix a bug on launchpad-gerrit integration. As such I pulled the openstack-infra/config to get a review.openstack.org alike server in a VM. I tweaked the manifests to configure it on my local development so it doesn't point to upstream servers and now gerrit and apache are up and running. However, I was expecting to see all openstack projects in the projects section of my local gerrit, doesn't it prepopulat | 10:59 |
| *** unicell has quit IRC | 11:13 | |
| *** alexpilotti has quit IRC | 11:16 | |
| *** mwagner_lap has quit IRC | 11:23 | |
| *** saju_m has joined #openstack-infra | 11:24 | |
| *** roeyc has joined #openstack-infra | 11:24 | |
| *** matsuhashi has joined #openstack-infra | 11:26 | |
| *** pblaho has joined #openstack-infra | 11:27 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Use older than newest tox. Newest tox has a bug https://review.openstack.org/85576 | 11:28 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Minor grammar correction https://review.openstack.org/85578 | 11:29 |
| *** e0ne has joined #openstack-infra | 11:31 | |
| *** e0ne_ has joined #openstack-infra | 11:33 | |
| *** e0ne has quit IRC | 11:33 | |
| *** che-arne|2 is now known as che-arne | 11:34 | |
| *** e0ne_ has quit IRC | 11:37 | |
| *** mbacchi has joined #openstack-infra | 11:38 | |
| *** yolanda has quit IRC | 11:39 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Fix for not loading middleware https://review.openstack.org/85693 | 11:42 |
| *** annegentle has quit IRC | 11:43 | |
| *** zz_gondoi is now known as gondoi | 11:46 | |
| *** matsuhashi has quit IRC | 11:49 | |
| *** dkranz has joined #openstack-infra | 11:49 | |
| *** matsuhashi has joined #openstack-infra | 11:50 | |
| *** jgallard has joined #openstack-infra | 11:52 | |
| *** VINOD_ has joined #openstack-infra | 11:53 | |
| *** matsuhas_ has joined #openstack-infra | 11:53 | |
| *** matsuhashi has quit IRC | 11:54 | |
| *** e0ne has joined #openstack-infra | 11:56 | |
| *** gondoi is now known as zz_gondoi | 11:56 | |
| *** jamielennox is now known as jamielennox|away | 11:56 | |
| *** e0ne_ has joined #openstack-infra | 11:59 | |
| *** nosnos has quit IRC | 11:59 | |
| *** e0ne has quit IRC | 12:02 | |
| *** rfolco has joined #openstack-infra | 12:03 | |
| *** yolanda has joined #openstack-infra | 12:03 | |
| *** pblaho has quit IRC | 12:06 | |
| *** mwagner_lap has joined #openstack-infra | 12:10 | |
| *** pdmars has joined #openstack-infra | 12:10 | |
| *** pdmars has quit IRC | 12:10 | |
| *** pdmars has joined #openstack-infra | 12:11 | |
| *** aysyd has joined #openstack-infra | 12:13 | |
| rcarrillocruz | nm, got it now. projects.yaml and projects.ini are not for pre-populating repos at gerrit installation phase, they are used by jeepyb | 12:18 |
| *** yamahata has quit IRC | 12:20 | |
| *** bogdando has quit IRC | 12:22 | |
| *** bogdando has joined #openstack-infra | 12:22 | |
| *** adalbas has joined #openstack-infra | 12:25 | |
| *** zigo has quit IRC | 12:26 | |
| *** mrmartin has joined #openstack-infra | 12:34 | |
| *** rlandy has joined #openstack-infra | 12:38 | |
| *** alexpilotti has joined #openstack-infra | 12:39 | |
| *** reed has joined #openstack-infra | 12:39 | |
| *** ihrachyshka has quit IRC | 12:40 | |
| *** ihrachyshka has joined #openstack-infra | 12:40 | |
| *** alexpilotti has quit IRC | 12:40 | |
| *** sdake_ has quit IRC | 12:45 | |
| openstackgerrit | Victor Stinner proposed a change to openstack/requirements: Trollius: allow version 0.2 and later https://review.openstack.org/85699 | 12:45 |
| *** andreaf has quit IRC | 12:45 | |
| *** sw_away is now known as sandywalsh | 12:47 | |
| *** miqui has joined #openstack-infra | 12:47 | |
| *** weshay has joined #openstack-infra | 12:50 | |
| *** alexpilotti has joined #openstack-infra | 12:53 | |
| *** amotoki has quit IRC | 12:54 | |
| *** thuc has joined #openstack-infra | 12:58 | |
| *** alexpilotti has quit IRC | 12:58 | |
| *** wenlock has joined #openstack-infra | 12:59 | |
| jpich | Hello! I'm hoping to get a pointer or two on adding a new job to the Horizon gate. There's a tox environment already created for it but because it also depends on Selenium, I'm not sure if something else will be required. I noticed there's a specific script related to the selenium unit tests, however the run_tests.sh command in it is hard-coded | 13:00 |
| jpich | https://github.com/openstack-infra/config/blob/master/modules/jenkins/files/slave_scripts/run-selenium.sh | 13:00 |
| jpich | Should I look at updating that script to accept different flags? Is the env in tox.ini necessary at all? | 13:00 |
| *** jeckersb_gone is now known as jeckersb | 13:01 | |
| *** ameade has joined #openstack-infra | 13:02 | |
| *** freyes_ has joined #openstack-infra | 13:02 | |
| *** wenlock has quit IRC | 13:03 | |
| *** weshay has quit IRC | 13:04 | |
| *** weshay has joined #openstack-infra | 13:04 | |
| *** dprince has joined #openstack-infra | 13:04 | |
| *** zehicle has quit IRC | 13:05 | |
| *** zehicle_at_dell has quit IRC | 13:05 | |
| fungi | jpich: i agree it looks like that script doesn't use tox, and is what the current horizon selenium job runs via https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/macros.yaml#n146 | 13:07 |
| *** rustlebee is now known as russellb | 13:08 | |
| jpich | fungi: Thank you for that extra link. Would you recommend updating the run-selenium.sh script to understand different arguments? | 13:10 |
| fungi | jpich: updating that script to call tox directly, or even updating the existing job to use runrun-tox.sh, might help with the flexibility there | 13:10 |
| fungi | er, run-tox.sh | 13:10 |
| *** mfer has joined #openstack-infra | 13:10 | |
| *** xchu_ has joined #openstack-infra | 13:10 | |
| *** xchu_ has quit IRC | 13:11 | |
| fungi | jpich: perhaps if your new job used run-tox.sh instead of run-selenium.sh, that would provide a good migration path away from run-selenium.sh for the existing job too, and then we could drop the additional script and builder macro once it's working correctly | 13:11 |
| *** alexpilotti has joined #openstack-infra | 13:11 | |
| fungi | here i'm assuming you can actually run selenium under tox... if not, then maybe that was why there was a separate script | 13:12 |
| *** homeless has joined #openstack-infra | 13:12 | |
| jpich | fungi: My guess is that the script was added because of the Xvfb requirement | 13:13 |
| *** changbl has quit IRC | 13:13 | |
| fungi | jpich: oh, are there additional commands which get run from run_tests.sh to set up xvfb which can't be done from inside a tox-managed virtualenv? | 13:14 |
| fungi | if that's the case, then perhaps modifying horizon's run_tests.sh, the run-selenium.sh slave script and the selenium builder macro with additional options is the easier solution | 13:15 |
| jpich | fungi: I have no idea. Maybe clarkb would have a suggestion since he submitted the initial patch? | 13:15 |
| *** mriedem has joined #openstack-infra | 13:15 | |
| *** derekh has joined #openstack-infra | 13:15 | |
| *** eharney has joined #openstack-infra | 13:16 | |
| fungi | jpich: perhaps, though he won't likely be around for a few more hours yet... he'll probably see this in his scrollback once he's up for the day | 13:16 |
| jpich | fungi: Ok. Thank you for the tips, I'll have a look and see what I can do! It's not always obvious to me how to test such a change when it involves multiple files | 13:17 |
| *** weshay has quit IRC | 13:18 | |
| *** weshay has joined #openstack-infra | 13:18 | |
| fungi | jpich: basically if it *can* be run directly from tox without needing any additional setup first, then we can almost certainly make the run-tox.sh slave script manage it and lose a lot of the other special tooling around running selenium jobs | 13:19 |
| *** dcramer_ has joined #openstack-infra | 13:19 | |
| fungi | or possibly even just call it from run-unittests.sh at that point | 13:21 |
| jpich | fungi: I suspect it needs some special setup (setting up a display server) but I think I saw a patch to add this to Horizon's run_tests.sh recently, I'll dig it up and see if it could work | 13:21 |
| *** dims has joined #openstack-infra | 13:22 | |
| jpich | The existing run_selenium.sh is for unit tests but the job I'm hoping to add is for integration tests - we'll see how that goes :) | 13:22 |
| *** freyes_ has quit IRC | 13:25 | |
| *** maxbit has joined #openstack-infra | 13:25 | |
| *** signed8bit has joined #openstack-infra | 13:26 | |
| *** matsuhas_ has quit IRC | 13:27 | |
| *** matsuhashi has joined #openstack-infra | 13:28 | |
| *** andreaf has joined #openstack-infra | 13:28 | |
| fungi | jpich: oh, so if integration tests need to set up horizon and other services, then you probably need some of the devstack-gate job framework for that anyway, in which case the tox-based method probably isn't going to make much difference | 13:29 |
| *** jeckersb is now known as jeckersb_gone | 13:29 | |
| *** e0ne_ has quit IRC | 13:30 | |
| *** matsuhas_ has joined #openstack-infra | 13:31 | |
| *** matsuhashi has quit IRC | 13:32 | |
| *** matsuhas_ has quit IRC | 13:34 | |
| jpich | fungi: Doh, I probably should have started with mentioning that then. But if there was a way to get Xvfb set up from the Horizon side, that probably would help remove the need for specific infra scripts for both unit and integration tests, if I understand correctly | 13:34 |
| fungi | jpich: agreed, it's worth investigating. thanks@ | 13:35 |
| *** e0ne has joined #openstack-infra | 13:35 | |
| anteaya | mikal: ohhhhh | 13:36 |
| anteaya | mattoliverau: nice job making mikal look competent and Neill too | 13:37 |
| *** jnoller has joined #openstack-infra | 13:37 | |
| *** jnoller has quit IRC | 13:37 | |
| fungi | apparently if they don't win him the nova ptl election, they're all fired ;) | 13:37 |
| anteaya | ha ha ha | 13:38 |
| *** zz_gondoi is now known as gondoi | 13:38 | |
| *** jnoller has joined #openstack-infra | 13:38 | |
| *** jnoller has quit IRC | 13:38 | |
| *** jnoller has joined #openstack-infra | 13:39 | |
| *** jnoller has quit IRC | 13:39 | |
| *** damnsmith is now known as dansmith | 13:39 | |
| *** jnoller has joined #openstack-infra | 13:39 | |
| *** jnoller has quit IRC | 13:39 | |
| *** thomasem has joined #openstack-infra | 13:39 | |
| *** thuc has quit IRC | 13:39 | |
| rcarrillocruz | guys, if i want to disable gerrit replication, should I remove the replication.config file or will it also work to put 'mirror = false' for each '[remote]' section? | 13:40 |
| *** thuc has joined #openstack-infra | 13:40 | |
| fungi | rcarrillocruz: if you're using our puppet manifests, you can probably just empty out the array of replication targets (though you may want to keep the local mirror in /var/lib/git) | 13:41 |
| *** jnoller has joined #openstack-infra | 13:41 | |
| *** jnoller has quit IRC | 13:41 | |
| rcarrillocruz | yep, using the review.pp, tweaked it to make it work in local environment etc | 13:41 |
| *** jnoller has joined #openstack-infra | 13:41 | |
| *** jnoller has quit IRC | 13:41 | |
| *** jnoller has joined #openstack-infra | 13:42 | |
| *** jnoller has quit IRC | 13:42 | |
| rcarrillocruz | where do you mean emptying the array of targets? in replication.config or other place? | 13:42 |
| *** nkinder has quit IRC | 13:42 | |
| fungi | rcarrillocruz: so you probably want to keep http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/review.pp#n147 but could remove the other replication array elements | 13:42 |
| rcarrillocruz | ok | 13:42 |
| rcarrillocruz | thx fungi | 13:42 |
| *** zehicle_at_dell has joined #openstack-infra | 13:42 | |
| *** zehicle has joined #openstack-infra | 13:42 | |
| *** jnoller has joined #openstack-infra | 13:42 | |
| *** jnoller has quit IRC | 13:42 | |
| fungi | basically you probably don't want the github and git0N entries | 13:43 |
| fungi | the iterator in replication.config.erb builds the relevant sections from the data its passed in that array | 13:44 |
| *** lcostantino has joined #openstack-infra | 13:44 | |
| *** jnoller has joined #openstack-infra | 13:44 | |
| *** jnoller has quit IRC | 13:44 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Fixes to parameters and layout in documentation https://review.openstack.org/83591 | 13:44 |
| *** thuc has quit IRC | 13:44 | |
| *** jnoller has joined #openstack-infra | 13:45 | |
| *** jnoller has quit IRC | 13:45 | |
| *** jnoller has joined #openstack-infra | 13:45 | |
| *** jnoller has quit IRC | 13:45 | |
| *** yamahata has joined #openstack-infra | 13:45 | |
| *** jnoller has joined #openstack-infra | 13:46 | |
| *** jnoller has quit IRC | 13:46 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Missing docstrings added https://review.openstack.org/85718 | 13:46 |
| *** jnoller has joined #openstack-infra | 13:46 | |
| *** jnoller has quit IRC | 13:46 | |
| *** jnoller has joined #openstack-infra | 13:47 | |
| *** jnoller has quit IRC | 13:47 | |
| *** jnoller has joined #openstack-infra | 13:47 | |
| *** jnoller has quit IRC | 13:47 | |
| *** dcramer_ has quit IRC | 13:48 | |
| *** jnoller has joined #openstack-infra | 13:48 | |
| *** jnoller has quit IRC | 13:48 | |
| *** jnoller has joined #openstack-infra | 13:48 | |
| *** jnoller has quit IRC | 13:48 | |
| *** dstanek_zzz is now known as dstanek | 13:48 | |
| *** jnoller has joined #openstack-infra | 13:49 | |
| *** jnoller has quit IRC | 13:49 | |
| *** e0ne has quit IRC | 13:50 | |
| *** e0ne has joined #openstack-infra | 13:51 | |
| *** jnoller has joined #openstack-infra | 13:51 | |
| *** jnoller has quit IRC | 13:51 | |
| *** jnoller has joined #openstack-infra | 13:51 | |
| *** jnoller has quit IRC | 13:52 | |
| *** jnoller has joined #openstack-infra | 13:52 | |
| *** jnoller has quit IRC | 13:52 | |
| *** jnoller has joined #openstack-infra | 13:53 | |
| *** jnoller has quit IRC | 13:53 | |
| *** jnoller has joined #openstack-infra | 13:53 | |
| *** jnoller has quit IRC | 13:53 | |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: jenkins-job-builder: Allow read/write stdin/stdout https://review.openstack.org/84778 | 13:53 |
| *** jnoller has joined #openstack-infra | 13:54 | |
| *** jnoller has quit IRC | 13:54 | |
| *** jnoller has joined #openstack-infra | 13:54 | |
| *** jnoller has quit IRC | 13:54 | |
| *** jnoller has joined #openstack-infra | 13:55 | |
| *** jnoller has quit IRC | 13:55 | |
| *** jnoller has joined #openstack-infra | 13:55 | |
| *** jnoller has quit IRC | 13:55 | |
| sdague | fungi: feeling on this one - https://review.openstack.org/#/c/84829/ ? | 13:55 |
| *** jnoller has joined #openstack-infra | 13:56 | |
| *** jnoller has quit IRC | 13:56 | |
| *** chandan_kumar has quit IRC | 13:56 | |
| *** jnoller has joined #openstack-infra | 13:56 | |
| *** jnoller has quit IRC | 13:56 | |
| fungi | sdague: ahh, i guess i hadn't finished reviewing that one yet. i'll switch back to it once i'm done with the oslo cross-tests review | 13:57 |
| sdague | sounds great | 13:57 |
| sdague | thank yuo | 13:57 |
| *** jnoller has joined #openstack-infra | 13:57 | |
| *** jnoller has quit IRC | 13:57 | |
| *** maxbit has quit IRC | 13:57 | |
| *** jnoller has joined #openstack-infra | 13:57 | |
| *** jnoller has quit IRC | 13:57 | |
| *** lascii has joined #openstack-infra | 13:59 | |
| sdague | anteaya: when do polls close for voting? | 14:00 |
| *** VINOD_ has quit IRC | 14:01 | |
| *** malini_afk is now known as malini | 14:01 | |
| *** e0ne_ has joined #openstack-infra | 14:03 | |
| therve | sdague, Thrusday | 14:04 |
| sdague | cool, thanks | 14:05 |
| fungi | sdague: which i'll have to finish after the vmt meeting | 14:05 |
| *** alexpilotti has quit IRC | 14:05 | |
| *** alexpilotti has joined #openstack-infra | 14:06 | |
| *** e0ne has quit IRC | 14:06 | |
| *** maxbit has joined #openstack-infra | 14:07 | |
| *** alaski_ has quit IRC | 14:08 | |
| *** lascii is now known as alaski | 14:08 | |
| *** pcm_ has joined #openstack-infra | 14:08 | |
| *** skraynev is now known as skraynev_afk | 14:10 | |
| *** changbl has joined #openstack-infra | 14:11 | |
| *** jgriffit2 is now known as jgriffith | 14:13 | |
| anteaya | sdague: Elections are underway and will remain open for you to cast your vote | 14:14 |
| anteaya | until at least 1300 utc April 11, 2014. | 14:14 |
| anteaya | http://lists.openstack.org/pipermail/openstack-dev/2014-April/031919.html | 14:15 |
| *** pdmars has quit IRC | 14:16 | |
| *** pdmars has joined #openstack-infra | 14:17 | |
| *** mburned is now known as mburned_out | 14:17 | |
| sdague | anteaya: ok, thanks. All my votes are in, just curious when we'd have results. Given that we've got so many PTL elections running this time. | 14:17 |
| dhellmann | fungi: when you look at that cross-test change, please pay special attention to the regex for making all the jobs non-voting -- I don't want to turn off other jobs, and I'm not sure those are named uniquely enough | 14:17 |
| *** pdmars has quit IRC | 14:17 | |
| anteaya | sdague: makes sense | 14:17 |
| fungi | dhellmann: yeah, i've got a few things raising my eyebrow in there. once i get freed up to finish reviewing we can discuss | 14:18 |
| anteaya | yes I pegged you as part of the rush that votes within 5 minutes from the time the ballots land | 14:18 |
| *** jgrimm has joined #openstack-infra | 14:18 | |
| *** pdmars has joined #openstack-infra | 14:18 | |
| dhellmann | fungi: ok, I'm going to be in moving around for a while this morning taking the car to the shop, so if you leave comments I'll review and we can talk this afternoon? | 14:19 |
| fungi | dhellmann: sounds good | 14:19 |
| dhellmann | fungi: cool, thanks | 14:20 |
| *** pdmars has quit IRC | 14:20 | |
| *** pdmars has joined #openstack-infra | 14:21 | |
| *** msabramo has quit IRC | 14:22 | |
| *** pdmars has quit IRC | 14:22 | |
| *** pdmars has joined #openstack-infra | 14:22 | |
| *** thuc has joined #openstack-infra | 14:22 | |
| *** matjazp has joined #openstack-infra | 14:23 | |
| *** thuc_ has joined #openstack-infra | 14:23 | |
| *** thuc_ has quit IRC | 14:24 | |
| *** thuc_ has joined #openstack-infra | 14:24 | |
| *** rcleere has joined #openstack-infra | 14:25 | |
| *** thuc has quit IRC | 14:27 | |
| *** zehicle has quit IRC | 14:31 | |
| *** nkinder has joined #openstack-infra | 14:31 | |
| *** zehicle_at_dell has quit IRC | 14:31 | |
| *** msabramo has joined #openstack-infra | 14:33 | |
| *** afazekas_ is now known as afazekas | 14:34 | |
| *** julim has joined #openstack-infra | 14:34 | |
| *** changbl has quit IRC | 14:34 | |
| *** zns has joined #openstack-infra | 14:34 | |
| anteaya | ttx can I request that changes to openstack/governance are frozen until after the conclusion of the tc election? | 14:34 |
| anteaya | https://review.openstack.org/#/q/status:open+project:openstack/governance,n,z | 14:34 |
| anteaya | I had to respond to a handful of confused folks who aren't eligible to vote and I used the programs.yaml file in my responses | 14:35 |
| fungi | anteaya: probably only need freezes to programs.yaml | 14:35 |
| anteaya | right, yes freeze for programs.yaml | 14:35 |
| anteaya | I don't reference the rest of governance | 14:35 |
| ttx | anteaya: you could reference to a past version of programs.yaml | 14:36 |
| ttx | the one from the start of the election | 14:36 |
| *** thuc_ has quit IRC | 14:36 | |
| fungi | i can certainly tweak the script to pull that file from a specific ref rather than the tip of master | 14:36 |
| *** thuc has joined #openstack-infra | 14:36 | |
| anteaya | I could, my concern if the confused folks aren't able to read the elegibility requirements correctly, should they navigate to a current version that may add to their confusion | 14:36 |
| fungi | but yes, for consistency i agree we probably need to use the same list of "official" projects for both the ptl and tc elections | 14:37 |
| anteaya | would there be a down side to a freeze for programs.yaml until after the tc election has concluded? | 14:37 |
| *** thuc has quit IRC | 14:41 | |
| *** mkoderer has quit IRC | 14:41 | |
| openstackgerrit | Randy Coulman proposed a change to openstack-infra/jenkins-job-builder: Support the Store SCM plugin https://review.openstack.org/85729 | 14:41 |
| SergeyLukjanov | ttx, sdague, are we already approving CRs in requirements in master for Juno? | 14:42 |
| *** dcramer_ has joined #openstack-infra | 14:42 | |
| ttx | SergeyLukjanov: sure, requirements was branched out for release | 14:42 |
| *** wenlock has joined #openstack-infra | 14:42 | |
| *** talluri has quit IRC | 14:44 | |
| SergeyLukjanov | ttx, thx, just want to clarify ;) | 14:45 |
| *** krotscheck has quit IRC | 14:46 | |
| *** krotscheck has joined #openstack-infra | 14:46 | |
| fungi | ttx: also, i noticed that the m-p requirements sync proposals worked after your patch and are getting approved. awesome | 14:48 |
| ttx | fungi: yep, it worked! | 14:49 |
| *** saju_m has quit IRC | 14:50 | |
| jgriffith | fungi: I should get cinder onboard with the *-specs movement.... | 14:51 |
| jgriffith | fungi: who do I need to work with to get the repo created? | 14:51 |
| *** che-arne has quit IRC | 14:51 | |
| fungi | jgriffith: you could propose a change like https://review.openstack.org/82512 but just s/neutron/cinder/ | 14:52 |
| jgriffith | fungi: perfect...thank you | 14:52 |
| fungi | and then we can process it this friday when we do our weekly run for new projects | 14:52 |
| mtreinish | jgriffith: you can also use: http://git.openstack.org/cgit/openstack-dev/specs-cookiecutter/ to set things up | 14:53 |
| fungi | oh! right, i forgot that was a thing now | 14:53 |
| jgriffith | mtreinish: fungi cookiecutter it is.. thanks! | 14:53 |
| jgriffith | I'll work on that later today | 14:53 |
| * jgriffith has been dieing to publish something with that for some time now | 14:53 | |
| *** jlibosva has quit IRC | 14:54 | |
| *** mgagne1 has quit IRC | 14:54 | |
| ttx | anteaya: the downside would be to block TC's progress. I have been asking about cancelling TC meetings while elections where udner way but that was rejected | 14:55 |
| ttx | we record a lot of decisions in that specific file | 14:55 |
| mtreinish | jgriffith: cool you get to be the first specs-cookiecutter guinea pig | 14:55 |
| anteaya | how would it block tc progress | 14:55 |
| mtreinish | let me know if something doesn't work | 14:55 |
| ttx | anteaya: preventing decisions to be made ? | 14:55 |
| jgriffith | mtreinish: will do | 14:55 |
| *** pcrews has joined #openstack-infra | 14:55 | |
| *** mburned_out is now known as mburned | 14:55 | |
| ttx | anteaya: blocks 4 of 5 resolutions here: https://review.openstack.org/#/q/status:open+project:openstack/governance,n,z | 14:56 |
| ttx | you could argue that those decisions are "less important", but still, that's basically blocking part of TC's work for n weeks | 14:56 |
| anteaya | just merging them | 14:57 |
| fungi | ttx: what do you think about tagging the governance repo at the start of elections in future cycles? that would make documenting atc eligibility easier | 14:57 |
| anteaya | oh yeah | 14:57 |
| anteaya | tagging | 14:57 |
| ttx | fungi: sure, tagging would work. We could even do that now | 14:57 |
| ttx | but I think that wouldn't solve the confusion | 14:57 |
| *** yfried has quit IRC | 14:57 | |
| ttx | since the tag wasn't mentioned in election doc | 14:57 |
| fungi | well, it makes it easier to provide a url to the version of that file which we're referencing | 14:57 |
| *** sdake has joined #openstack-infra | 14:58 | |
| *** sdake has quit IRC | 14:58 | |
| *** sdake has joined #openstack-infra | 14:58 | |
| fungi | again, easier in future cycles, since we can mention it initially | 14:58 |
| ttx | right, definitely a good idea for future cycles. maybe a good idea for now | 14:58 |
| *** unicell has joined #openstack-infra | 14:58 | |
| jeblair | seems like an appropriate use of tags | 14:59 |
| fungi | also, i can make the electorate roll generation key off a tag that way | 14:59 |
| anteaya | I can mention the tag in the tc election material | 14:59 |
| ttx | anteaya: I can push a tag for the current election if you point me to the right commit | 14:59 |
| anteaya | great let me dig something up | 15:00 |
| ttx | or I bet fungi can do it now | 15:00 |
| anteaya | he will probably beat me | 15:00 |
| fungi | yeah, we're just looking for the state of the repo as it was at 06:00 utc last friday | 15:00 |
| fungi | finding now | 15:00 |
| *** thuc has joined #openstack-infra | 15:00 | |
| anteaya | the current commit is fine | 15:00 |
| anteaya | the last commit was 12 days ago: http://git.openstack.org/cgit/openstack/governance/log/reference/programs.yaml | 15:01 |
| *** mihgen has quit IRC | 15:01 | |
| *** blamar has joined #openstack-infra | 15:01 | |
| fungi | d24ac6f is the tip of master, and 5 days old | 15:01 |
| fungi | which is old enough to cover when the election began | 15:02 |
| fungi | so it's the same commit we would have tagged at the start of the election, in all likelihood | 15:02 |
| fungi | committerGerrit Code Review <review@openstack.org>2014-04-02 08:55:50 (GMT) | 15:03 |
| ttx | does "april-2014-elections" sound like a good tag name ? | 15:03 |
| jeblair | storyboard meeting time | 15:03 |
| anteaya | yes | 15:03 |
| anteaya | thanks | 15:03 |
| fungi | ttx: wfm | 15:03 |
| *** nkinder has quit IRC | 15:03 | |
| ttx | fungi: shall be up | 15:03 |
| *** ogelbukh has quit IRC | 15:03 | |
| *** yamahata has quit IRC | 15:04 | |
| fungi | ttx: or do you think being able to sort the tags in alphanumeric order matching chronology would help? | 15:04 |
| *** yamahata has joined #openstack-infra | 15:04 | |
| fungi | 2014.1-elections or something? | 15:04 |
| ttx | fungi: hmm, too late ? | 15:05 |
| ttx | also little value imho | 15:05 |
| fungi | ttx: eh, probably not important. this bike shed is already painted ;) | 15:05 |
| *** thedodd has joined #openstack-infra | 15:05 | |
| anteaya | look at taht lovely tag: http://git.openstack.org/cgit/openstack/governance/tag/?id=april-2014-elections | 15:06 |
| anteaya | thanks | 15:06 |
| fungi | anteaya: yeah, http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml?id=april-2014-elections | 15:06 |
| fungi | anteaya: that url will forever point to the version we used | 15:07 |
| *** whayutin_ has joined #openstack-infra | 15:07 | |
| anteaya | awesome, I was just wondering how to get that | 15:07 |
| anteaya | the dropdown menu beside the switch button only sees master | 15:08 |
| *** david-lyle has joined #openstack-infra | 15:08 | |
| openstackgerrit | João Vale proposed a change to openstack-infra/jenkins-job-builder: Support variable-specific substitutions in templates. https://review.openstack.org/85383 | 15:08 |
| fungi | anteaya: yeah, that's for selecting branches | 15:08 |
| anteaya | ah | 15:08 |
| pabelanger | morning! | 15:08 |
| pabelanger | Any issues getting https://review.openstack.org/#/c/83980/ merged into nodepool this week? | 15:08 |
| pabelanger | just waiting for approval I believe | 15:09 |
| *** weshay has quit IRC | 15:10 | |
| fungi | pabelanger: and 83975 presumably | 15:11 |
| jeblair | mordred: re https://review.openstack.org/#/c/80976/ (puppet kick) | 15:11 |
| jeblair | mordred: apparently it's deprecated: https://projects.puppetlabs.com/issues/15735 | 15:11 |
| *** alff has joined #openstack-infra | 15:12 | |
| *** alff_ has joined #openstack-infra | 15:12 | |
| jeblair | mordred: they suggest we use mcollective | 15:12 |
| jeblair | mordred: in which case, we might as well use salt, since it's similar (but i believe we felt that salt's remote execution was nicer) | 15:12 |
| jeblair | mordred: or go back to basics and do what they are both designed to simplify, and use ssh | 15:12 |
| pabelanger | fungi, ya, figured I'd like the tail end of the patchset | 15:13 |
| mordred | jeblair: awesome | 15:13 |
| *** mdenny has quit IRC | 15:13 | |
| *** MIDENN_ has joined #openstack-infra | 15:14 | |
| mordred | jeblair: how about I go back to basics, write down the description of the problem we're wanting to solve and the 2.5 solutions we've come up with so far | 15:14 |
| pabelanger | I'm in the process of re-evaluating salt again, but would be interested in trying to keep sync with -infra. | 15:14 |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Refactor if: nesting in build_mirror https://review.openstack.org/59961 | 15:15 |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Refactor calls to Git https://review.openstack.org/75931 | 15:15 |
| *** che-arne has joined #openstack-infra | 15:15 | |
| mordred | jeblair: because I feel like every time we try to loop in someone new on this, they get blocked on grokking the problem | 15:15 |
| *** nkinder has joined #openstack-infra | 15:15 | |
| mordred | pabelanger: what we're talking about is using salt to orchestrate puppet runs | 15:15 |
| fungi | "wait, you have multiple servers and want to coordinate actions on them in a specific order? are you insane?" ;) | 15:15 |
| *** maxbit has quit IRC | 15:16 | |
| fungi | oh, just for the "update puppet when one of this list of repos changes" use case | 15:16 |
| fungi | got it | 15:16 |
| anteaya | mordred: could you write down that description on an etherpad? | 15:16 |
| *** sahumada has quit IRC | 15:17 | |
| *** zehicle_at_dell has joined #openstack-infra | 15:17 | |
| *** pblaho has joined #openstack-infra | 15:17 | |
| *** zehicle has joined #openstack-infra | 15:17 | |
| *** ogelbukh has joined #openstack-infra | 15:17 | |
| *** alexpilotti has quit IRC | 15:17 | |
| fungi | and that was where the salt-based solution got stymied. we had no problem setting up a jenkins slave which could cause all salt minions to run puppet apply on specific merge events... the trick was getting the puppet master to pull an updated stete of the repo before proceeding | 15:17 |
| sdague | fungi: can you look at logs on elastic recheck? | 15:18 |
| *** thuc has quit IRC | 15:18 | |
| anteaya | fungi: that is what I talked to UtahDave about on friday | 15:18 |
| anteaya | ensuring /var/lib/git is in all the repos | 15:19 |
| *** thuc has joined #openstack-infra | 15:19 | |
| zaro | morning | 15:19 |
| *** sahumada has joined #openstack-infra | 15:19 | |
| anteaya | we were going to talk today about finding a way to do that | 15:19 |
| anteaya | morning zaro | 15:19 |
| matjazp | mordred: I have a question regarding Moodle installation for OpenStack Training Manuals | 15:19 |
| *** alexpilotti has joined #openstack-infra | 15:19 | |
| *** gyee has joined #openstack-infra | 15:20 | |
| pabelanger | mordred, Yup, I was considering doing the same, now that I'm also using nodepool. I tried salt ~9 months ago, but had some issues with it. I know they just did a big merge of branches, so plan on testing it again | 15:20 |
| *** sahumada has quit IRC | 15:22 | |
| *** thuc has quit IRC | 15:23 | |
| msabramo | anyone here work on python-jenkins? | 15:23 |
| matjazp | mordred: we're trying to integrate Launchpad IDs into Moodle and I see that you already integrated it with your Storyboard project. Are there any other issues we should consider when integrating Moodle with Launchpad IDs? | 15:23 |
| *** jgrimm has quit IRC | 15:24 | |
| *** ihrachyshka has quit IRC | 15:24 | |
| *** jgrimm has joined #openstack-infra | 15:25 | |
| *** sahumada has joined #openstack-infra | 15:25 | |
| *** mgagne has joined #openstack-infra | 15:26 | |
| fungi | anteaya: i'm not talking about the manage-projects use case. i'm talking about the earlier puppet apply use case | 15:27 |
| matjazp | mordred: sarob said, that openstack-infra is considering hosting this Moodle server with Quizzes for OpenStack Training Manuals when things are finished | 15:27 |
| fungi | anteaya: he was going to put together an example of how to get the puppet master to pull updates into /opt/production/config before the event went out to all salt minions to do a puppet apply | 15:28 |
| *** zul has quit IRC | 15:28 | |
| fungi | anteaya: but i worry that these two separate use cases are confusing people as to what and how we were talking about solving those different problems by using salt | 15:28 |
| *** pdmars has quit IRC | 15:29 | |
| *** mgagne1 has joined #openstack-infra | 15:29 | |
| *** pdmars has joined #openstack-infra | 15:29 | |
| *** alexpilotti has quit IRC | 15:30 | |
| jeblair | fungi: yeah, severing the ideas of 'trigger on commit' from 'run things in order' is a good idea | 15:30 |
| *** mgagne has quit IRC | 15:31 | |
| *** sahumada has quit IRC | 15:31 | |
| *** pdmars has quit IRC | 15:32 | |
| fungi | the common thread between those two specific use cases (manage-projects orchestration and puppet apply orchestration) is that we have more than one action which needs to take place on separate systems in a specific sequence | 15:32 |
| *** alexpilotti has joined #openstack-infra | 15:32 | |
| *** maxbit has joined #openstack-infra | 15:32 | |
| *** pdmars has joined #openstack-infra | 15:32 | |
| *** pdmars has quit IRC | 15:33 | |
| *** zul has joined #openstack-infra | 15:33 | |
| *** sahumada has joined #openstack-infra | 15:33 | |
| *** pdmars has joined #openstack-infra | 15:33 | |
| *** zehicle has quit IRC | 15:33 | |
| *** zehicle_at_dell has quit IRC | 15:33 | |
| *** pdmars has quit IRC | 15:34 | |
| *** pdmars has joined #openstack-infra | 15:35 | |
| *** pdmars has quit IRC | 15:35 | |
| *** pdmars has joined #openstack-infra | 15:36 | |
| *** wchrisj has joined #openstack-infra | 15:36 | |
| *** pdmars has quit IRC | 15:37 | |
| *** pdmars has joined #openstack-infra | 15:37 | |
| anteaya | which if we could do with all salt, would be no problem, the issue is the coordinating with the puppet taht is already there | 15:38 |
| *** pdmars has quit IRC | 15:38 | |
| *** zigo__ has joined #openstack-infra | 15:38 | |
| *** pdmars has joined #openstack-infra | 15:39 | |
| *** msabramo has quit IRC | 15:40 | |
| *** pdmars has quit IRC | 15:40 | |
| fungi | anteaya: i believe it was discussed that was the case for the manage-projects situation (because we have to wait for a puppet-managed exec of create-cgitrepos to run successfully on five machines before starting gerrit replication on another), but the puppet apply orchestration is a different situation | 15:41 |
| *** pdmars has joined #openstack-infra | 15:41 | |
| *** pdmars has quit IRC | 15:41 | |
| *** roeyc has quit IRC | 15:41 | |
| fungi | in the puppet apply orchestration dilemma, we just need the puppet master (which is also the samt master) to pull git refs in a repository before sending an event to all the minions | 15:42 |
| *** alexpilotti has quit IRC | 15:42 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Factor out common code between cli utilities https://review.openstack.org/85565 | 15:42 |
| *** pdmars has joined #openstack-infra | 15:42 | |
| anteaya | yes, the solution I am trying to find right now it for manage-projects | 15:42 |
| *** pdmars has quit IRC | 15:42 | |
| *** alexpilotti has joined #openstack-infra | 15:43 | |
| *** alexpilotti has quit IRC | 15:43 | |
| anteaya | fungi: what is happening now, in the apply orchestration dilemma? | 15:43 |
| fungi | the problem we started trying to address first was puppet apply orchestration, which was never solved completely. then the manage-projects race consition surfaced more recently as something urgent we might want to address using salt | 15:43 |
| anteaya | yes | 15:43 |
| *** esker has joined #openstack-infra | 15:43 | |
| *** pdmars has joined #openstack-infra | 15:43 | |
| *** alexpilotti has joined #openstack-infra | 15:44 | |
| anteaya | y'know what forget I asked | 15:44 |
| *** afazekas has quit IRC | 15:45 | |
| anteaya | let me solve one thing | 15:45 |
| fungi | in the puppet apply orchestration case, i was still looking for an example of how to make sure the salt master ran an action locally on itself (git pull) via its relationship with itself as also a salt minion, before running a separate action on all the minions (puppet apply) | 15:45 |
| anteaya | I don't seem to do well in the juggling of many things | 15:45 |
| fungi | no worries | 15:45 |
| derekh | Hi, look like the tripleo F20 nodes started building on friday once the backlog was cleared, | 15:46 |
| fungi | basically we have all the other bits worked out for the puppet apply problem (how to trigger from a merge event on a repo, how to have salt call puppet apply on every minion at once, et cetera) | 15:46 |
| derekh | the f20 jobs are now passing but | 15:46 |
| derekh | nodepoll seems to be have a preference on building precise nodes, it looks lit its os maintaining f20 at the min-ready level | 15:47 |
| derekh | untill the demand for precise nodes is gone | 15:47 |
| *** msabramo has joined #openstack-infra | 15:47 | |
| derekh | which leaves jobs in the queue waiting on F20 nodes, have you seen this befor in other queues | 15:49 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Openstackid Mysql connection Hiera variables https://review.openstack.org/85745 | 15:50 |
| *** yfried has joined #openstack-infra | 15:50 | |
| fungi | derekh: right, if you need more f20 nodes, we can adjust the min-ready... generally nodepool will calculate the needed ratio based on the number of jobs waiting for more nodes to be assigned | 15:50 |
| *** matjazp_ has joined #openstack-infra | 15:51 | |
| fungi | so min-ready isn't a hard "always have this many available" value, it's more of a guideline for helping calculate the ratio taking demand into account | 15:52 |
| derekh | fungi: I can adjust min-ready and it looks like this would be a good workaround for what I'm observing, | 15:52 |
| derekh | fungi: but creating nodes based on a ratio of waiting jobs doesn't seem to be working | 15:52 |
| *** thuc has joined #openstack-infra | 15:53 | |
| fungi | derekh: i think there may be a separate issue there--doesn't look like there are anywhere near 40 tripleo jobs running | 15:53 |
| fungi | checking under the hood now | 15:53 |
| *** thuc_ has joined #openstack-infra | 15:54 | |
| *** matjazp has quit IRC | 15:54 | |
| *** thuc__ has joined #openstack-infra | 15:55 | |
| *** mburned is now known as mburned_out | 15:55 | |
| *** mburned_out is now known as mburned | 15:55 | |
| *** bearhands is now known as comstud | 15:56 | |
| *** thuc has quit IRC | 15:57 | |
| *** thuc_ has quit IRC | 15:58 | |
| fungi | derekh: yeah, more than a third of the capacity there is currently stuck in a delete state for >3 hours | 15:58 |
| fungi | most look to be in that state since some time on friday | 15:59 |
| fungi | i'll try to clear them | 15:59 |
| derekh | fungi: ok, thanks | 15:59 |
| *** nkinder has quit IRC | 16:02 | |
| *** hashar has quit IRC | 16:02 | |
| fungi | looks like some of these might map to instances in an error state in nova, since the delete attempts are timing out | 16:04 |
| *** changbl has joined #openstack-infra | 16:05 | |
| *** jgallard has quit IRC | 16:05 | |
| *** alff_ has quit IRC | 16:05 | |
| *** alff has quit IRC | 16:05 | |
| *** pblaho has quit IRC | 16:06 | |
| *** matjazp has joined #openstack-infra | 16:06 | |
| fungi | derekh: but worth noting, http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb#n73 is currently saying "at rest, keep 5x as many tripleo-precise nodes as tripleo-f20" | 16:07 |
| openstackgerrit | Cyril Roelandt proposed a change to openstack/requirements: Paramiko: bump to 1.13.0 https://review.openstack.org/81132 | 16:07 |
| *** jcoufal has quit IRC | 16:07 | |
| *** maxbit has quit IRC | 16:07 | |
| *** prad has joined #openstack-infra | 16:07 | |
| *** matjazp_ has quit IRC | 16:09 | |
| *** sahumada has quit IRC | 16:10 | |
| *** matjazp has quit IRC | 16:10 | |
| *** matjazp has joined #openstack-infra | 16:10 | |
| *** saju_m has joined #openstack-infra | 16:12 | |
| *** hogepodge has joined #openstack-infra | 16:12 | |
| *** sahumada has joined #openstack-infra | 16:12 | |
| mrmartin | re | 16:13 |
| mrmartin | fungi: can I get a review for this small patch: https://review.openstack.org/#/c/85745/ ? | 16:14 |
| *** hogepodge has quit IRC | 16:14 | |
| jeblair | mrmartin, fungi: do we need to keep those in hiera, or can we just put the values directly in puppet? | 16:15 |
| mrmartin | we need to keep in hiera, because we want to change this connection string temporarily to live openstack user database | 16:16 |
| *** e0ne_ has quit IRC | 16:16 | |
| *** chandan_kumar has joined #openstack-infra | 16:16 | |
| *** e0ne has joined #openstack-infra | 16:16 | |
| mrmartin | but the db name and user name parameters can be different for prod db, so I moved out defaults to site.pp | 16:16 |
| derekh | fungi: ok, I'm counting 2 f20 nodes and 23 precise nodes, should I try to reset the nodes in a deleting state? | 16:17 |
| fungi | mrmartin: i don't see why they would need to be in hiera to accomplish that | 16:17 |
| *** hogepodge has joined #openstack-infra | 16:17 | |
| fungi | derekh: i didn't set the deletes to go in parallel, so the for loop is blocking on nodes which take 10 minutes to get a response back from nova before giving up | 16:17 |
| mrmartin | fungi: I think this db connection string is a sensitive data. | 16:18 |
| *** alff has joined #openstack-infra | 16:18 | |
| *** nkinder has joined #openstack-infra | 16:18 | |
| jeblair | mrmartin: the user and database names? | 16:18 |
| fungi | mrmartin: sensitive in what way? does it include the password? | 16:18 |
| *** alff_ has joined #openstack-infra | 16:19 | |
| anteaya | fungi: is /var/lib/git a directory? | 16:19 |
| mrmartin | fungi: it is not including the password. | 16:19 |
| derekh | fungi: ok, so should jI ust wait and see if their successful? | 16:19 |
| fungi | mrmartin: if you want to take the typical corporate "security by obscurity" stance to determine what is sensitive in this regard, the fact that we publish the configuration of all our systems could be considered sensitive | 16:19 |
| derekh | *should I just | 16:19 |
| fungi | mrmartin: we should just use good, strong passwords and make sure we don't publish them, in my opinion | 16:20 |
| fungi | anteaya: that is a directory, yes. | 16:20 |
| *** andreykurilin_ has joined #openstack-infra | 16:21 | |
| anteaya | yay | 16:21 |
| anteaya | http://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.file.html#salt.modules.file.directory_exists | 16:21 |
| *** e0ne has quit IRC | 16:21 | |
| mrmartin | fungi, jeblair: so you suggest just keep the password in hiera? | 16:21 |
| fungi | mrmartin: yes | 16:21 |
| *** sahumada has quit IRC | 16:22 | |
| jeblair | mrmartin: we also have the hostnames in hiera | 16:22 |
| mrmartin | so in that case we need to remove ss_mysql_host and id_mysql_host parameters also | 16:22 |
| fungi | mrmartin: at least for now, since all we're using hiera for is secred credential data. in the future we may end up having public and non-public hiera files, so that we can take advantage of hiera as a data store | 16:22 |
| jeblair | fungi, mrmartin: when we decided to do that i'm not sure if we knew whether the trove databases were tenant scoped or not, so it seemed like a reasonable precaution | 16:23 |
| fungi | mrmartin: the ss_mysql_host and id_mysql_host values are special in that they were intended to (eventually) be trove databases, and rackspace generates very long hash-based hostnames for the database server | 16:23 |
| mrmartin | ok, so what are the settings of this two host parameters? | 16:23 |
| *** sahumada has joined #openstack-infra | 16:24 | |
| mrmartin | I can send a modified patch for review. | 16:24 |
| fungi | jeblair: if i confirm that 3306/tcp isn't reachable over the internet at large on those, are you okay with moving them from hiera into the manifests? | 16:25 |
| fungi | maybe test reaching them from a different tenant in the same region? | 16:25 |
| *** jpich has quit IRC | 16:25 | |
| jeblair | fungi: you should test a different tenant; i'm pretty sure they're already rfc1918 addrs | 16:26 |
| jeblair | fungi: but if so, yes that wfm. | 16:26 |
| fungi | mrmartin: jeblair: okay, will have an update for you momentarily | 16:26 |
| * jeblair pays the phone bill for the asterisk server | 16:26 | |
| *** sdake_ has joined #openstack-infra | 16:27 | |
| *** sdake_ has quit IRC | 16:27 | |
| *** sdake_ has joined #openstack-infra | 16:27 | |
| mrmartin | fungi, jeblair: I need to leave now, because the office is closing, may I ask you to drop a comment to this review: https://review.openstack.org/#/c/85745/ ? | 16:27 |
| mrmartin | thnx. | 16:27 |
| fungi | mrmartin: i will be happy to | 16:27 |
| jeblair | mrmartin: will do, thank you! | 16:27 |
| *** mrmartin has quit IRC | 16:28 | |
| *** zigo has joined #openstack-infra | 16:28 | |
| fungi | derekh: i'm re-running the loop with an explicit 15-second timeout | 16:29 |
| derekh | fungi: ok | 16:29 |
| fungi | so whatever's left over in a delete state in nodepool here in the next few minutes will almost certainly be broken instances in nova--then i'll make you a uuid list of those | 16:30 |
| derekh | fungi: ok, thanks | 16:30 |
| *** sahumada has quit IRC | 16:32 | |
| *** zigo has quit IRC | 16:33 | |
| *** markmcclain has joined #openstack-infra | 16:33 | |
| *** beekneemech is now known as bnemec | 16:34 | |
| *** zigo has joined #openstack-infra | 16:34 | |
| *** saju_m has quit IRC | 16:35 | |
| *** sahumada has joined #openstack-infra | 16:35 | |
| anteaya | fungi: when a node is created with the intention it will be part of the git farm, is this the script that is run to create the node? http://git.openstack.org/cgit/openstack-infra/config/tree/launch/launch-node.py | 16:35 |
| fungi | anteaya: yes | 16:35 |
| anteaya | great thanks | 16:35 |
| fungi | anteaya: specifically we follow http://git.openstack.org/cgit/openstack-infra/config/tree/launch/README | 16:36 |
| anteaya | thanks | 16:36 |
| anteaya | so it knows itself to be a production server | 16:37 |
| anteaya | that doesn't differentiate it from any other servers though, does it? | 16:37 |
| anteaya | the reason I ask | 16:37 |
| clarkb | morning | 16:38 |
| anteaya | I need git servers to know they are git servers | 16:38 |
| anteaya | in salt, I can give them a grain file that tells them they are a git server | 16:38 |
| anteaya | just wondering what is there already | 16:38 |
| anteaya | morning clarkb | 16:38 |
| jeblair | anteaya: the puppet cert generated for the server identifies it by name (eg, git0X.openstack.org) | 16:39 |
| jeblair | anteaya: when puppet runs on the new server (as part of that script), it talks to puppetmaster with that cert, and the puppetmaster gives it the info (manifest) needed to become that server | 16:39 |
| anteaya | okay so I can start with the server name | 16:40 |
| anteaya | thanks | 16:40 |
| *** zigo__ has quit IRC | 16:40 | |
| clarkb | fungi: jpich: I am pretty sure you can get Xvfb from the horizon side. This was implemented in a less horizon specific way to potentially allow other projects to do similar testing | 16:41 |
| *** zigo has quit IRC | 16:41 | |
| clarkb | however, if this isn't flexible enough I say hit it with a hammer. Getting Xvfb to run isn't terribly difficult | 16:41 |
| anteaya | the default grain file contains the domain field, so I see if I can just use that | 16:42 |
| anteaya | oh and the fqdn, so that is better | 16:42 |
| zaro | clarkb: coming in today? | 16:43 |
| fungi | jeblair: unfortunately, the trove instance in question is reachable on 3306 from a different tenant in the same region | 16:44 |
| *** flashgordon is now known as jogo | 16:44 | |
| fungi | pvo: is that intentional? ^ | 16:44 |
| *** denis_makogon_ has joined #openstack-infra | 16:45 | |
| *** zigo has joined #openstack-infra | 16:46 | |
| clarkb | zaro: no, today is work from $park day | 16:46 |
| clarkb | currently at home but seattle center or mohai area are likely to happen | 16:46 |
| clarkb | fungi: I didn't think it was supposed to be intentional | 16:47 |
| *** sahumada has quit IRC | 16:47 | |
| *** denis_makogon_ is now known as denis_makogon | 16:47 | |
| clarkb | fungi: the unencrypted connections are 'secure' because they should be segregated | 16:47 |
| clarkb | sdague: what e-r logs were you hoping to look at? | 16:47 |
| fungi | clarkb: is_bug=True then | 16:47 |
| zaro | clarkb: may join you, need your help debugging my puppet test machine. | 16:48 |
| *** yassine has quit IRC | 16:48 | |
| jeblair | fungi: ok, let's consider the host name secret for now... i'm logged into rax webui, should i open a ticket? | 16:49 |
| fungi | jeblair: please do | 16:49 |
| *** sahumada has joined #openstack-infra | 16:49 | |
| clarkb | zaro: I will let you know when I move, likely to happen after lunch. That way one battery charge should be sufficient | 16:50 |
| fungi | jeblair: ftr, i connected via telnet into the ip address to which that trove instance dns rr refers, and was able to reach it from ci-puppetmaster (in the openstackci tenant) but also from salt-trigger.slave (in the openstackjenkins tenant) | 16:50 |
| clarkb | btw I noticed over the weekend that nova and neutron unittest console logs are in the ~140MB range when they fail | 16:51 |
| clarkb | I dropped notes about it in #openstack-qa and #openstack-nova, but will probably write up bugs now | 16:51 |
| jeblair | fungi, clarkb: ticket created in openstackci account | 16:54 |
| pleia2 | good morning, packing for pycon \o/ | 16:54 |
| *** SumitNaiksatam has quit IRC | 16:55 | |
| *** harlowja has joined #openstack-infra | 16:56 | |
| *** marun has joined #openstack-infra | 16:57 | |
| jeblair | fungi: speaking of pycon; i will probably be unreachable on a plane tomorrow during the meeting, will you be available to chair? | 16:57 |
| pleia2 | same here | 16:58 |
| *** marktraceur has quit IRC | 16:58 | |
| fungi | jeblair: sure, no problem | 16:59 |
| fungi | also everyone remember i'll be basically unreachable next week and the week after (though i'll mention it again during the meeting) | 16:59 |
| *** markmcclain has quit IRC | 16:59 | |
| *** markmcclain has joined #openstack-infra | 17:00 | |
| *** markmcclain has quit IRC | 17:00 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: make contactstore option overridable https://review.openstack.org/85753 | 17:00 |
| clarkb | https://bugs.launchpad.net/neutron/+bug/1303913 | 17:01 |
| uvirtbot | Launchpad bug 1303913 in nova "Console logs for unittest failures are > 100MB" [Undecided,New] | 17:01 |
| clarkb | sdague: jogo mtreinish ^ | 17:01 |
| mtreinish | clarkb: that's a big log file | 17:01 |
| clarkb | mtreinish: yes, it is basically useless. Not sure how that started ahppening | 17:01 |
| mtreinish | clarkb: it looks like the logging fixture is picking up at the debug level | 17:02 |
| jogo | clarkb: damn | 17:02 |
| mtreinish | I wonder if the default changed | 17:02 |
| *** zns has quit IRC | 17:02 | |
| clarkb | but it does explain why the e-r bot complains when those tests fail | 17:03 |
| clarkb | mtreinish: oh that could be | 17:03 |
| jogo | clarkb: ohh its a subunit vs console issue | 17:03 |
| clarkb | jogo: ? | 17:03 |
| jogo | " We should continue to preserve complete logging in the subunit log (we do need the verbose information), but we don't need to fill the console log with noisy redundant data." | 17:04 |
| clarkb | I think it is testr dumping tons of data to the console on failures | 17:04 |
| clarkb | jogo: right, so I am just suggesting that if we need 140MB logs that we do that with the subunit log | 17:04 |
| clarkb | not with the console log | 17:04 |
| clarkb | because as a contributor to nova if I have to sift through 140MB of log to find out what happened I will be sad | 17:05 |
| jogo | why isn't console.html compressed? | 17:05 |
| clarkb | jogo: because it happens on a delay | 17:05 |
| jogo | oh right | 17:05 |
| jogo | so the tons of logs are because a lot of neutron tests aren't unit | 17:05 |
| clarkb | well nova too | 17:06 |
| *** Ryan_Lane has joined #openstack-infra | 17:07 | |
| pleia2 | doing the overview talk as a keynote in June \o/ http://2014.dorscluc.org/en/2014/04/06/our-first-confirmed-keynote-speaker/ | 17:08 |
| *** marktraceur has joined #openstack-infra | 17:08 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:08 | |
| jogo | pleia2: croatia? whoa | 17:09 |
| pleia2 | jogo: yeah :) | 17:09 |
| *** derekh has quit IRC | 17:10 | |
| fungi | that sounds like some awesome fun | 17:11 |
| fungi | congrats pleia2 | 17:11 |
| jeblair | pleia2: cool; i've wanted to visit croatia for quite a while! | 17:11 |
| reed | croatia is cool | 17:12 |
| pleia2 | thanks, congrats to all of us for building a keynote-worthy open source infrastructure, people really like learning about it :) | 17:12 |
| *** aconrad has joined #openstack-infra | 17:12 | |
| pleia2 | closest I've been is budapest | 17:12 |
| fungi | hopefully the more who learn about it, the more new contributors we'll get pitching in to improve and run it | 17:12 |
| jogo | clarkb: I don't see where neutron sets the logs to debug | 17:13 |
| pleia2 | fungi: hope so! | 17:13 |
| *** amcrn has joined #openstack-infra | 17:13 | |
| mtreinish | jogo: should I just push out a patch that changes the log level for the fixture in nova to INFO? | 17:13 |
| mtreinish | or did you have another approach? | 17:13 |
| jogo | mtreinish: we want debug logs in nova | 17:16 |
| *** sabari has joined #openstack-infra | 17:16 | |
| jogo | we have af fixture that makes nova fail if there is a misformatted log statement | 17:16 |
| jogo | mtreinish: that doesn't mean we need to keep all the logs around though | 17:16 |
| mtreinish | jogo: you can turn off nuke-handlers which will preserve the log | 17:16 |
| mtreinish | but they won't get picked up by the fixture | 17:17 |
| clarkb | mtreinish: jogo my suggestion in an email thread about a month back was to make it another test configurable flag | 17:17 |
| clarkb | the testr can set it to INFO and everythign else can fall back on debug | 17:17 |
| jogo | clarkb: what do you mean? | 17:18 |
| clarkb | jogo: just like OS_STDOUT_CAPTURE | 17:19 |
| clarkb | make it configurable | 17:19 |
| mtreinish | clarkb: I think the issue is the fixture to check the formatting | 17:19 |
| clarkb | oh | 17:19 |
| clarkb | interesting ok | 17:19 |
| jogo | mtreinish: correct | 17:19 |
| mtreinish | jogo: let me try it with nuke_handlers=False | 17:19 |
| *** blamar has quit IRC | 17:20 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Make ironic-undercloud job non-voting https://review.openstack.org/85468 | 17:20 |
| jogo | clarkb: log formatting using a ',' instad of a '%' results on lazy processing of the logs | 17:20 |
| jogo | mtreinish: you are talking about neutron now right? | 17:21 |
| jogo | the log formatting stuff is in nova | 17:21 |
| jogo | ohh I see now | 17:22 |
| mtreinish | jogo: I haven't looked at neutron yet | 17:22 |
| *** mgagne1 is now known as mgagne | 17:22 | |
| mtreinish | clarkb, jogo: https://review.openstack.org/85759 | 17:23 |
| jogo | so I am -1 on that | 17:23 |
| jogo | what we should do is turn off debug logs for the integrated tests only | 17:24 |
| clarkb | jogo: honestly, I don't see that making a huge difference | 17:24 |
| clarkb | jogo: because the most tests are larger scale than unittests | 17:24 |
| clarkb | jeblair: these zuul timer trigger races in the zuul tests are quite fun to sort out | 17:25 |
| jogo | clarkb: it may not help a lot but it can't hurt | 17:26 |
| jogo | much | 17:27 |
| pabelanger | fungi, I too though of killing builds in progress of nodepool, but couldn't come up with a clean way to do it ATM. | 17:27 |
| jogo | ideally we would use debug level logs, but not preserve debug logs in gate | 17:27 |
| clarkb | jogo: exactly... see my comments about making it configurable :) | 17:27 |
| openstackgerrit | A change was merged to openstack/requirements: Raise oslo.messaging to 1.3.0 https://review.openstack.org/85337 | 17:28 |
| openstackgerrit | A change was merged to openstack/requirements: Taskflow 0.2 released https://review.openstack.org/85423 | 17:28 |
| *** ominakov has quit IRC | 17:28 | |
| jogo | I wasn't sure what you meant by that. we always want log level to be debug -- just not preserve them | 17:28 |
| *** mrmartin has joined #openstack-infra | 17:28 | |
| mtreinish | jogo: what kind of formatting error does that fixture catch? I want to test the nuke_handlers thing | 17:29 |
| mtreinish | because I'm pretty sure it should work | 17:29 |
| clarkb | jogo: no, if you don't want to preserve them then you don't log at that level | 17:29 |
| clarkb | jogo: there is no point in doing all of that IO | 17:29 |
| pleia2 | so, trying to set up a pypi mirror for tripleo, is our mirror configured such that I could mirror from that or do/should I use our identical modules to mirror from upstream? | 17:31 |
| jogo | mtreinish: missing arguements etc things such as: log.DEBUG("%(foo)s", {'bar':'1'}) | 17:31 |
| pleia2 | (either way, hoping to use our existing module, just with an altered upstream) | 17:31 |
| clarkb | pleia2: you should be able to build the mirror from pypi.o.o | 17:31 |
| pleia2 | clarkb: cool, thank you | 17:32 |
| pleia2 | should have a WIP change for that up by the end of the day then | 17:32 |
| jogo | clarkb: ^ is why. original commit I70790aa39c0774726de71e5fa5751c45e7e34ffd | 17:32 |
| jogo | mtreinish: it cought things such as I8a2b64222deaf5320bcc2e5215b6bd46565a505f | 17:32 |
| *** atiwari has joined #openstack-infra | 17:33 | |
| *** mriedem has quit IRC | 17:33 | |
| clarkb | jogo: I agree with everything in that commit message but it needs to be configurable | 17:34 |
| *** saschpe has quit IRC | 17:34 | |
| *** jeckersb_gone is now known as jeckersb | 17:35 | |
| jeblair | clarkb: sorry about the races; would you like me to help with anything? | 17:36 |
| fungi | jeblair: after some additional testing, we have a new potential reason to move forward with mrmartin's https://review.openstack.org/85745 in preparation for switching to production credentials. the vm will need to connect to the production cloudsites db for the foundation site, and currently cloudsites databases are completely reachable on 3306/tcp from the entire internet, so obscuring more | 17:36 |
| fungi | credentials than just the password could be warranted | 17:36 |
| sdague | clarkb: I'm trying to figure out why a bug isn't showing up in the chron output | 17:36 |
| clarkb | jeblair: no, I think I am close to having them fixed. Debugging why swapping in a layout without a timer breaks test_idle but not the other two timer tests | 17:37 |
| mrmartin | fungi: it means, that the patch in this form is acceptable? | 17:37 |
| jeblair | fungi: what do they look like? do they share a single hostname or is it a per-db hostname, etc? | 17:37 |
| fungi | mrmartin: it means we should at least discuss additional temporary precautions to work around that problem | 17:37 |
| fungi | jeblair: the database name and username are both the same somewhat random-looking string | 17:38 |
| mrmartin | fungi: do you think we can deliver some solution within a reasonable timeframe? | 17:38 |
| fungi | jeblair: and no hostname, just a raw ipv4 address, based on the information i have | 17:39 |
| *** matjazp has quit IRC | 17:39 | |
| jogo | clarkb: sounds reasonable but not sure how making it configurable helps | 17:41 |
| *** saschpe has joined #openstack-infra | 17:41 | |
| clarkb | jogo: we can set it to > DEBUG in the gate | 17:42 |
| sdague | clarkb: basically bug 1280532 should be in the ER results, but it isn't | 17:43 |
| uvirtbot | Launchpad bug 1280532 in nova "Detach volume fails with "Unexpected KeyError" in EC2 interface." [High,In progress] https://launchpad.net/bugs/1280532 | 17:43 |
| sdague | and I'm completely confused why | 17:43 |
| clarkb | sdague: gotcha, I can take a look here shortly | 17:43 |
| sdague | when I run the graph script locally it's in the json | 17:43 |
| jeblair | fungi: thanks; yeah i agree obscurity is probably warranted here. | 17:44 |
| sdague | definitively not in - http://status.openstack.org/elastic-recheck/data/all.json | 17:44 |
| *** mriedem has joined #openstack-infra | 17:44 | |
| clarkb | sdague: gotcha, it may be a stuck puppet or similar | 17:44 |
| sdague | clarkb: ok, cool. Thanks for looking at it | 17:45 |
| *** thuc__ has quit IRC | 17:45 | |
| jogo | clarkb: but thats where we want to catch these issues | 17:45 |
| clarkb | jogo: what issues? | 17:45 |
| *** mrodden has quit IRC | 17:45 | |
| jogo | bad log statements | 17:45 |
| *** thuc has joined #openstack-infra | 17:46 | |
| *** ildikov_ has quit IRC | 17:46 | |
| clarkb | jogo: oh, right so I think the current setup for catching them is not right | 17:46 |
| clarkb | jogo: you are relying on side effects of tests to test specific things rather than testing specific things | 17:46 |
| clarkb | jogo: can't oslo.logging enforce this stuff for us and fail in any case without proper logging? | 17:46 |
| jogo | clarkb: true, it is definitely a hack | 17:46 |
| jogo | clarkb: hmm don't think so | 17:46 |
| clarkb | jogo: ya, I am trying to come up with a better way to test it | 17:47 |
| jogo | clarkb: I still think it would be nice if we can run in debug logs but just not save them | 17:47 |
| clarkb | and this may be the most direct. Won't get you perfect coverage but certainly is better than nothing | 17:47 |
| mrmartin | fungi, jeblair: what you think about accepting of those hiera variables and put the prod db connection parameters there? And later we can update this for an optimal solution, when the cloudsites db access is secured. | 17:47 |
| jogo | maybe we can make gate's testr not output logs for failed tests? | 17:47 |
| clarkb | jogo: but it should | 17:48 |
| jogo | so we can shrink console.html | 17:48 |
| clarkb | jogo: thats the point, I think this has made debugging any failre impossible | 17:48 |
| fungi | mrmartin: yes, that's basically a summary of what i just updated my vote on the patch with--i'm working on the hiera end of that now | 17:48 |
| clarkb | jogo: we have traded simplicity and use for catching one very small case | 17:48 |
| mtreinish | jogo: That's what nuke_handlers=False does the fixture captures the info level or above. But it leaves the other logger handlers around so debug messages still get run | 17:48 |
| *** e0ne has joined #openstack-infra | 17:49 | |
| sdague | jogo: we definitely need lots for failed tests | 17:49 |
| mtreinish | but they won't be injected into the subunit stream because the fixture didn't grab them | 17:49 |
| jogo | clarkb: these cases happen a lot | 17:49 |
| sdague | s/lots/logs/ | 17:49 |
| *** freyes_ has joined #openstack-infra | 17:49 | |
| jeblair | mrmartin, fungi: i think putting them in hiera is fine, but we should give them appropriate names -- eg, the production db shouldn't be called dev | 17:49 |
| *** thuc has quit IRC | 17:50 | |
| jogo | clarkb: so if there is no way around this, I am fine with turning off debug logs in the gate. just don't make it the default behavior. so when folks run it locally | 17:50 |
| *** blamar has joined #openstack-infra | 17:50 | |
| mrmartin | jeblair, as I see in the emails it is not the production db, just a clone of it, used for dev.openstack.org. | 17:50 |
| jeblair | mrmartin: ok, cool; should be fine then | 17:50 |
| jeblair | mrmartin: which emails? | 17:51 |
| mrmartin | jeblair: I'll forward it to you. | 17:51 |
| jogo | but if nuke_handlers=False works even better | 17:52 |
| fungi | jeblair: in this case it actually is the dev db--just a dev db which is a copy of the production db | 17:52 |
| mtreinish | jogo: well once numpy finishes compiliing hopefully I'll know :) | 17:52 |
| jeblair | fungi: got it | 17:52 |
| mtreinish | it's been too long since I last ran nova unit tests | 17:52 |
| fungi | jeblair: and being hosted from cloudsites like the production db rather than a dump imported into trove | 17:52 |
| jeblair | fungi: probably a good way to test | 17:53 |
| fungi | jeblair: so anyway, the arguments about why it could be needed for production are basically valid for this phase of dev testing as well | 17:53 |
| clarkb | jeblair: I am running the current iteration of my timer tests fix through testr run --parallel --until-failure to see if I can catch any more corner cases | 17:53 |
| clarkb | jeblair: but if this continues to pass after a few more runs I will push it | 17:54 |
| *** dizquierdo has quit IRC | 17:54 | |
| sdague | jogo: why don't you make the fixture also set the logger to a string buffer that gets cycled at each test? Then you would execute it, but it wouldn't actually output anywhere? | 17:56 |
| jogo | mtreinish: heh | 17:56 |
| jogo | sdague: good idea | 17:56 |
| sdague | jogo: because the current change is terrible, and is just going to make us disk bankrupt way too fast | 17:56 |
| sdague | so the immediate change has to be reverted ASAP | 17:57 |
| *** annegentle_ has joined #openstack-infra | 17:57 | |
| jogo | sdague: I would rather just turn off debug logs in gate (make configurable as clarkb said) | 17:58 |
| *** talluri has joined #openstack-infra | 17:58 | |
| *** mburned is now known as mburned_out | 17:58 | |
| *** mburned_out is now known as mburned | 17:58 | |
| sdague | jogo: well I think you have to revert first | 17:58 |
| sdague | every hour that this lives around we lose a lot of disk space | 17:59 |
| sdague | then come up with the right fix | 17:59 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Handle NullChange "merges" gracefully. https://review.openstack.org/85217 | 17:59 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add base test class to make subclassing possible https://review.openstack.org/85204 | 17:59 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 17:59 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Fix races in timer trigger tests. https://review.openstack.org/85230 | 17:59 |
| sdague | GET http://cacti.openstack.org/ | 18:00 |
| sdague | Can't connect to cacti.openstack.org:80 (Connection refused) | 18:00 |
| clarkb | jeblair: jhesketh ^ bam there we go. 85230 and 85204 are ready imo, the other two need to address some of jheskeths concerns, but I can make them less serialized in gerrit once the first two go in | 18:00 |
| sdague | cacti not up? | 18:00 |
| clarkb | sdague: correct, waiting for patches to arrive | 18:00 |
| sdague | ok | 18:00 |
| *** Sukhdev has joined #openstack-infra | 18:00 | |
| jeblair | /dev/mapper/main-logs 12T 11T 1.2T 91% /srv/static/logs | 18:00 |
| jeblair | /dev/mapper/main-tarballs 197G 31G 167G 16% /srv/static/tarballs | 18:00 |
| jeblair | /dev/mapper/main-docs--draft 1.4T 997G 360G 74% /srv/static/docs-draft | 18:00 |
| sdague | jeblair: yeh, I was hoping for trend line | 18:01 |
| sdague | but that's also handy | 18:01 |
| jogo | sdague clarkb mtreinish: http://stackoverflow.com/questions/9534245/python-logging-to-stringio-handler | 18:01 |
| sdague | jogo: the important thing is you are currently dossing infra | 18:01 |
| sdague | so revert first | 18:01 |
| sdague | then lets sort out the right fix | 18:01 |
| jogo | sdague: I would rather just make it configurable that is just as easy | 18:02 |
| jogo | patch coming now | 18:02 |
| jogo | gonna use the flag OS_DEBUG | 18:02 |
| *** hogepodge_ has joined #openstack-infra | 18:03 | |
| jogo | unless you have a better idea for a flag | 18:03 |
| jogo | clarkb: ^ | 18:03 |
| sdague | or revert, which we know won't hurt anything, then work on configurable after, because you'll need to land patches in multiple trees to change the defaults for nova and infra | 18:03 |
| *** mrodden has joined #openstack-infra | 18:03 | |
| clarkb | jogo: OS_LOG_LEVEL=debug or similar | 18:03 |
| *** nkinder has quit IRC | 18:03 | |
| fungi | sdague: we can get the trend line once ubuntu gets patched cacti in precise (it's already in trusty as of this morning, so just a matter of a little while longer i expect) | 18:03 |
| sdague | fungi: sure | 18:03 |
| fungi | we're still collecting stats, just have the web interface offline to mitigate | 18:04 |
| *** hogepodge has quit IRC | 18:04 | |
| *** hogepodge_ is now known as hogepodge | 18:04 | |
| sdague | fungi: yep, no worries | 18:04 |
| sdague | it's probably worth making a post run test that fails changes that make logs larger than a certain value | 18:05 |
| sdague | so that we don't end up landing DOSing changes like this again | 18:05 |
| clarkb | jeblair: while I have you thinking about zuul tests though. Any idea of how you would like to see me handle jhesketh's comments in 85217 and 76057. For 76057 you wanted to rip out the specail getGitUrl fake which I did. This means we can't really use a thing to test a "gerrit" location. I assume we can just leave that bit unimplemented? For making the polling happen, do you think that is a good way | 18:06 |
| clarkb | to do it? eg have tests update the upstream repo after some delay? | 18:06 |
| mtreinish | jogo: this is taking a bit longer then I though. The unit tests ate all my ram and crashed the guest, sigh... | 18:07 |
| jogo | haha run a subset | 18:07 |
| jeblair | sdague: ++; could add to run-unittests.sh; should also probably rm the logs in question? after all, these jobs actually are failing, right (because the tests fail?) | 18:07 |
| clarkb | jeblair: correct | 18:07 |
| sdague | jogo: how are the unit tests consuming that much memory? | 18:08 |
| sdague | that's crazy pants | 18:08 |
| jogo | sdague: amen | 18:08 |
| sdague | jogo: can we please back up and revert the issue that got us here? | 18:08 |
| sdague | then go sensibly forward | 18:09 |
| *** matjazp has joined #openstack-infra | 18:09 | |
| sdague | instead of running around and hot patching until we think maybe it's not all crazy? | 18:09 |
| jogo | sdague: testing patch now | 18:09 |
| *** nkinder has joined #openstack-infra | 18:09 | |
| jogo | http://paste.openstack.org/show/75235 | 18:09 |
| *** mwagner_lap has quit IRC | 18:10 | |
| sdague | jogo: but then you also need to land infra changes | 18:10 |
| mtreinish | sdague: 8GBs too :) could it be all the logs sitting in memory? | 18:11 |
| sdague | mtreinish: probably | 18:11 |
| sdague | which is why I really want to *revert* | 18:11 |
| sdague | and do this sensibly | 18:11 |
| mtreinish | sdague: I'm testing https://review.openstack.org/#/c/85759/ | 18:11 |
| *** _nadya_ has joined #openstack-infra | 18:11 | |
| mtreinish | I think it's because the other handlers are there with that patch | 18:12 |
| jogo | sdague: if clarkb really wants to then I am fine with it. but don't think revert will be any faster | 18:12 |
| jogo | still will take an hour or two to get through gate etc | 18:12 |
| jeblair | clarkb: i don't understand 'This means we can't really use a thing to test a "gerrit" location.' ? | 18:12 |
| sdague | jogo: a revert we can fast path | 18:12 |
| sdague | and it will be known to not cause the issue | 18:12 |
| sdague | then we can actually work on sensible changes and try a couple different approaches | 18:13 |
| clarkb | jeblair: we are using fetch_url to avoid using the old fake in the test suite. jhesketh is suggesting that we have some tests not use fetch_url to test the other code which was never tested in the first place | 18:13 |
| jeblair | clarkb: out of scope | 18:13 |
| sdague | because I think what you actually want doesn't require us to use the default logger at all | 18:13 |
| mtreinish | jogo: yeah so my approach is a no go: http://logs.openstack.org/59/85759/1/check/gate-nova-python27/a6d1615/console.html | 18:14 |
| mtreinish | unless we want to like double the ram on the unit test guests :) | 18:14 |
| sdague | but we're not going to work on the disposable string buffer approach because that's going to take too long to get right | 18:14 |
| jeblair | clarkb: the delay before updating the upstream repo sounds like a good test | 18:15 |
| *** thuc has joined #openstack-infra | 18:16 | |
| *** zns has joined #openstack-infra | 18:16 | |
| jogo | sdague clarkb: https://review.openstack.org/85779 | 18:16 |
| jogo | just needs an infra patch | 18:16 |
| clarkb | you can set it in tox.ini | 18:16 |
| *** thuc has quit IRC | 18:16 | |
| mtreinish | jogo: why not just set it in the testr.conf | 18:16 |
| clarkb | or .testr.conf | 18:17 |
| mtreinish | or the tox.ini :) | 18:17 |
| jogo | we want to default to debug | 18:17 |
| jogo | so people running this locally will catch issues | 18:17 |
| *** thuc has joined #openstack-infra | 18:17 | |
| clarkb | jogo: no we don't. Default should be whatever we gate against | 18:17 |
| jogo | clarkb: then we want use debug logs | 18:17 |
| sdague | jogo: please just push the REVERT | 18:17 |
| jogo | so we can gate on log statements working | 18:17 |
| jogo | sdague: so this patch landed weeks ago at this point, which is why I am hesitant to take any drastic actions | 18:18 |
| sdague | jogo: then why did it only go crazy this weekend? | 18:18 |
| jogo | clarkb: this patch has cought plenty of bugs | 18:18 |
| jogo | sdague: I have no idea | 18:18 |
| *** _nadya_ has quit IRC | 18:19 | |
| sdague | jogo: what is the patch that is the root of this? | 18:19 |
| jogo | sdague: I didn't realize things got much worse over the wekend | 18:19 |
| clarkb | jogo: sdague I don't think it went crazy over the weekened | 18:20 |
| sdague | jogo: if you revert us back to a sane place, I'm very willing to work on you on an approach that gets the results you want without killing upstream storage | 18:20 |
| * jogo is confused | 18:20 | |
| clarkb | it was just the first cahnce I had to debug why e-r was complaing | 18:20 |
| *** khyati_ has joined #openstack-infra | 18:20 | |
| sdague | clarkb: ok, gotcha | 18:20 |
| clarkb | so this weekend was the first it got noticed | 18:20 |
| jogo | ahh | 18:20 |
| sdague | so basically this makes us completely blind to all nova unit tests issues | 18:20 |
| sdague | ? | 18:21 |
| jogo | sdague: how so? | 18:21 |
| *** moted has joined #openstack-infra | 18:21 | |
| jogo | ohh the logs are too big | 18:21 |
| sdague | if we can't index those logs | 18:21 |
| clarkb | it also makes it impossible for devs to debug their own stuff | 18:22 |
| *** mrodden has quit IRC | 18:22 | |
| clarkb | which is the biggest issue here imo | 18:22 |
| *** mrodden has joined #openstack-infra | 18:22 | |
| clarkb | we made the console logs useless for devs | 18:22 |
| jogo | clarkb: can you find other examples of massive console logs | 18:23 |
| jogo | besides the two in https://bugs.launchpad.net/neutron/+bug/1303913 | 18:23 |
| uvirtbot | Launchpad bug 1303913 in nova "Console logs for unittest failures are > 100MB" [Undecided,New] | 18:23 |
| mtreinish | jogo: would it make sense to do what tempest does and dump the logs to a file? | 18:24 |
| jogo | mtreinish: we have that in subunit stream | 18:24 |
| mtreinish | no so we switch the fixture to be more useful | 18:24 |
| mtreinish | so set it to info | 18:24 |
| sdague | no, the whole point is we don't actually want the logs in the base case | 18:24 |
| sdague | we just want the logging code to execute | 18:24 |
| jeblair | clarkb: left comments on https://review.openstack.org/#/c/76057/ | 18:24 |
| sdague | which is why I'm pushing the revert - https://review.openstack.org/85780 | 18:25 |
| jogo | clarkb: I am not sure how frequent this is happening -- it *may* only be really bad when its an integration test | 18:25 |
| sdague | jogo: well because this is breaking ER, we can't seen how bad it is | 18:25 |
| jogo | sdague: I am -1 on this, as I think this is still premature. how often is this happening | 18:25 |
| clarkb | jogo: gate-neutron-python27 85492,5,fa325bf gate-nova-python26 67500,7,3bd3332 gate-neutron-python27 82729,3,a067a84 gate-nova-python26 83256,14,294f78f gate-nova-python26 83256,13,86daccd I am just going off of what e-r says | 18:25 |
| jogo | sdague: we can look at file sizes etc | 18:25 |
| clarkb | jeblair: thanks | 18:26 |
| jogo | clarkb: any others? | 18:26 |
| jogo | those are the ones you already included | 18:26 |
| clarkb | jogo: sure, I just stopped after a while | 18:26 |
| clarkb | some are some aren't | 18:26 |
| jogo | http://logs.openstack.org/91/82891/1/check/gate-nova-python27/2bd331e/ | 18:26 |
| clarkb | just look at e-r sb | 18:27 |
| clarkb | sdague: e-r json files are from today | 18:28 |
| clarkb | sdague: looking at the age of the e-r git checkout next | 18:28 |
| sdague | clarkb: yep, but they aren't right | 18:28 |
| *** hogepodge has quit IRC | 18:28 | |
| *** mrmartin has quit IRC | 18:29 | |
| clarkb | e-r looks old and puppet looks stuck. kicking it | 18:30 |
| clarkb | ah, I broke it | 18:30 |
| clarkb | 0,30 is not a valid minute | 18:30 |
| sdague | so we're adding another 13 MB of subunit on every python unit tests vs. before - http://logs.openstack.org/68/76968/7/gate/gate-nova-python27/81c06b3/ | 18:31 |
| clarkb | man 5 crontab furiously disagrees with puppet on this one hrm | 18:31 |
| sdague | so probably 1 GB of storage wasted every day based on this | 18:32 |
| clarkb | does puppet want a puppet list? | 18:32 |
| sdague | clarkb: hmmm... I've only used singletons or */5 like values | 18:32 |
| sdague | so I don't know | 18:32 |
| jogo | clarkb sdague: so this issue is very infrequent | 18:32 |
| clarkb | ya puppet wants it as ['0','30'] | 18:32 |
| clarkb | ugh | 18:32 |
| jogo | 6x a day | 18:32 |
| *** yolanda has quit IRC | 18:33 | |
| jogo | so we do have time to sort this out | 18:33 |
| sdague | jogo: the extra 13 MB per run is on every run | 18:33 |
| *** _nadya_ has joined #openstack-infra | 18:33 | |
| sdague | we're wasting 1 GB / day on this | 18:33 |
| sdague | jogo: before this change the subunit log was 1.3MB compressed | 18:34 |
| sdague | now it's 14 MB compressed | 18:35 |
| sdague | on every run | 18:35 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make e-r crons valid puppet https://review.openstack.org/85784 | 18:35 |
| clarkb | I am going to test that really quick | 18:36 |
| sdague | clarkb: cool | 18:36 |
| jogo | sdague: true but an extra 1GB a day isn't a lot | 18:38 |
| *** hogepodge has joined #openstack-infra | 18:38 | |
| mtreinish | jogo: do you think this will work?: https://review.openstack.org/#/c/85759/2 | 18:38 |
| sdague | jogo: it adds up pretty quick | 18:38 |
| sdague | especially as we're basically at our volume limit | 18:38 |
| *** ildikov_ has joined #openstack-infra | 18:39 | |
| *** morganfainberg_Z is now known as morganfainberg | 18:39 | |
| *** markmcclain has joined #openstack-infra | 18:39 | |
| *** andreaf has quit IRC | 18:40 | |
| jogo | sdague: 14MB versus a the 350MB of a tempest run? | 18:40 |
| sdague | a tempest run isn't 350 MB compressed | 18:40 |
| jogo | also neutron has an unrelated issue so | 18:41 |
| jogo | we can hack at that | 18:41 |
| jogo | hack away at | 18:41 |
| rcarrillocruz | mordred , lifeless : thanks for your suggestions on the launchpad-gerrit sync about creating my own token | 18:41 |
| rcarrillocruz | been able to sync my gerrit reviews against a test bug now! https://bugs.launchpad.net/openstack-ci/+bug/1303847 | 18:41 |
| uvirtbot | Launchpad bug 1303847 in openstack-ci "Test bug, do not triage" [Undecided,New] | 18:41 |
| * rcarrillocruz dances | 18:41 | |
| jogo | mtreinish: interesting | 18:42 |
| rcarrillocruz | i can now starting working on the abandon change hook | 18:42 |
| jogo | trying it out now | 18:42 |
| jogo | if that works that would be really nice | 18:42 |
| *** thuc has quit IRC | 18:42 | |
| mtreinish | jogo: heh well at least this way it doesn't eat >8GB of ram | 18:42 |
| sdague | mtreinish: that's a good thing :) | 18:43 |
| *** thuc has joined #openstack-infra | 18:43 | |
| jeblair | sdague: when you have a sec: https://review.openstack.org/#/c/85587/1 | 18:43 |
| jogo | mtreinish: that fixed the ram issues? whoa | 18:44 |
| sdague | jeblair: I think we should be good | 18:44 |
| jeblair | thx | 18:44 |
| mtreinish | jogo: writing to null instead of a stream. Apparently nova logs quite a lot | 18:45 |
| *** thuc_ has joined #openstack-infra | 18:45 | |
| jogo | mtreinish: ahhh | 18:45 |
| openstackgerrit | A change was merged to openstack-infra/config: Make e-r crons valid puppet https://review.openstack.org/85784 | 18:45 |
| *** thuc has quit IRC | 18:45 | |
| sdague | mtreinish: yeh, given that nova unit tests generate roughly the same amount of log data as a whole tempest run, all services :) | 18:45 |
| *** thuc_ has quit IRC | 18:45 | |
| clarkb | sdague: I am approving gating on neutron grenade job gyi | 18:46 |
| clarkb | *fyi | 18:46 |
| sdague | clarkb: great | 18:46 |
| mtreinish | clarkb: awesome | 18:46 |
| sdague | hopefully neutron didn't land any breaks | 18:46 |
| sdague | they were good last time I looked | 18:46 |
| clarkb | we will know very soon :) | 18:46 |
| sdague | yep | 18:46 |
| *** thuc has joined #openstack-infra | 18:46 | |
| *** thuc has quit IRC | 18:47 | |
| *** thuc has joined #openstack-infra | 18:47 | |
| clarkb | oh and ptl voting is open | 18:47 |
| mtreinish | all that's left for neutron now is a full parallel gate | 18:47 |
| clarkb | apparently I have contributed to !infra | 18:47 |
| jogo | mtreinish: its working for me! | 18:47 |
| sdague | clarkb: :) | 18:47 |
| mtreinish | jogo: sweet | 18:48 |
| jogo | mtreinish: a little more testing needed to double check | 18:48 |
| sdague | mtreinish: I assume that will take the subunit stream back to sanity as well | 18:48 |
| mtreinish | clarkb: I was dissapointed too, I couldn't vote for cinder, my last cinder commit was a couple weeks too old | 18:49 |
| mtreinish | sdague: yeah it should | 18:49 |
| mtreinish | only info will get put in the subunit stream | 18:49 |
| openstackgerrit | David Kranz proposed a change to openstack-infra/devstack-gate: DO NOT MERGE: Try to use nose to get a stacktrace https://review.openstack.org/85786 | 18:49 |
| *** jp_at_hp has quit IRC | 18:49 | |
| sdague | mtreinish: yeh, lgtm | 18:50 |
| clarkb | mtreinish: I am pleasantly surprised, I couldn't even tell you which changes got me into the nova and neutron pools | 18:50 |
| *** gondoi is now known as zz_gondoi | 18:50 | |
| openstackgerrit | Jay Faulkner proposed a change to openstack-infra/config: Disable pypy jobs in ironic-python-agent https://review.openstack.org/85787 | 18:50 |
| sdague | jogo: this whole approach seems kind of error prone though to making sure debug messages are valid syntax, because I'm sure a bunch of them don't get touched. Have you considered a static analysis tool on them instead? | 18:51 |
| *** saju_m has joined #openstack-infra | 18:51 | |
| mtreinish | clarkb: looks like https://review.openstack.org/#q,Id0f196d7b5680e5950e4a27d66042bf00ccd49e6,n,z and https://review.openstack.org/#q,I6be917181fa8d18764db89e741190964546d0632,n,z :) | 18:51 |
| *** talluri has quit IRC | 18:51 | |
| jogo | mtreinish: false alarm, it didn't work :/ | 18:52 |
| mtreinish | jogo: isn't that a real alarm then? | 18:52 |
| jogo | sdague: I don' think static analysis would work very well | 18:52 |
| jogo | mtreinish: heh perhaps, I added a log statement that was bad and it didn't get picked up | 18:53 |
| mtreinish | jogo: let me take a look at how the formatting fixture is working | 18:53 |
| mtreinish | because that should run every log message | 18:53 |
| sdague | jogo: ok, how are you ensuring that all debug lines are hit? | 18:53 |
| jogo | yeah the fakelogger does dirty things | 18:53 |
| jogo | sdague: I am not. just going for if it has unit test coverage then log statements will work | 18:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Add rally-scenarios job https://review.openstack.org/84719 | 18:54 |
| *** zz_gondoi is now known as gondoi | 18:54 | |
| openstackgerrit | A change was merged to openstack-infra/config: Enable voting on grenade-dsvm-neutron job https://review.openstack.org/85587 | 18:56 |
| boris-42 | guys does anybody knows how to fix this ? http://logs.openstack.org/38/85738/1/check/gate-rally-python33/7cf1bc1/console.html | 18:56 |
| boris-42 | seems like during installation of python-glance client something weird happens.. | 18:56 |
| boris-42 | in py33 .. | 18:56 |
| jogo | mtreinish: I like the direction you are going in, hopefully you can get it to work | 18:57 |
| clarkb | boris-42: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 173: ordinal not in range(128) is the error. Happens when installing jsonpatch | 18:58 |
| clarkb | boris-42: I think that means jsonpatch is not python33 compatible during install? | 18:58 |
| clarkb | looks like they do something funny like open and read a file in setup.py which fails | 18:59 |
| boris-42 | clarkb yep | 18:59 |
| mtreinish | jogo: this doesn | 18:59 |
| boris-42 | clarkb hm so we should try to fix jsonpatch | 18:59 |
| boris-42 | clarkb or remove jsonpatch from glance client?) | 19:00 |
| mtreinish | jogo: this doesn't work: https://docs.python.org/2/library/logging.html#logging.Handler.handleError | 19:00 |
| *** hogepodge has quit IRC | 19:00 | |
| mtreinish | because you shouldn't need to monkey patch it | 19:00 |
| clarkb | boris-42: ya, you may get away with an older version for now if the older version works | 19:00 |
| JayF | I don't think it'll have a significant impact on capacity, but https://review.openstack.org/85787 is to disable pypy tests which are completely timing out due to a deadlock, so it's keeping a worker busy for 40m doing nothing. If you guys review and approve this, it'll not be eating up that machine anymore :) | 19:00 |
| *** mwagner_lap has joined #openstack-infra | 19:00 | |
| jogo | mtreinish: yeah its ugly, but I blame python | 19:00 |
| clarkb | boris-42: 1.5 was uploaded on march 22nd | 19:00 |
| boris-42 | clarkb hm I should speak with glance guys | 19:01 |
| *** ociuhandu has quit IRC | 19:01 | |
| boris-42 | clarkb cause I am not able to fix this from rally =( | 19:01 |
| *** thedodd has quit IRC | 19:01 | |
| jogo | mtreinish: I did 'You could, however, replace this with a custom handler if you wish' | 19:02 |
| *** thedodd has joined #openstack-infra | 19:03 | |
| mtreinish | jogo: yeah it's that whole wish part. I'm just wondering why it was necessary. But, I'll take your word for it. | 19:05 |
| *** jnoller has joined #openstack-infra | 19:05 | |
| *** jnoller has quit IRC | 19:05 | |
| jogo | mtreinish: if you know a better way I am all ears. | 19:05 |
| * jogo finds lunch | 19:05 | |
| JayF | clarkb: would you like the todo out of there too re: https://review.openstack.org/85787 ? | 19:06 |
| *** jnoller has joined #openstack-infra | 19:06 | |
| *** jnoller has quit IRC | 19:06 | |
| mtreinish | jogo: also it looks like NullHandler wasn't in py26 | 19:06 |
| mtreinish | so I'm not sure this approach will work | 19:06 |
| clarkb | JayF: yes, I don't think the layout.yaml is an appropriate place to trak that | 19:06 |
| sdague | anyone know what's up with turbo hipster -1ing everything? | 19:06 |
| clarkb | JayF: it will get lost. ironic should track it somewere ironicy | 19:06 |
| *** blamar has quit IRC | 19:06 | |
| JayF | clarkb: that's okay with me, I'll update it to have no comments | 19:06 |
| davidlenwell | did you guys fix the thing where it failed if the commit message was moe than 50 chars ? | 19:07 |
| *** blamar has joined #openstack-infra | 19:07 | |
| openstackgerrit | Jay Faulkner proposed a change to openstack-infra/config: Disable pypy jobs in ironic-python-agent https://review.openstack.org/85787 | 19:07 |
| clarkb | davidlenwell: hahahahaha infra fixes it by ignoring hacking :P | 19:07 |
| davidlenwell | oh ha ha | 19:08 |
| clarkb | davidlenwell: you can ignore all of hacking r just that specific rule in your flake8 config in tox.ini | 19:08 |
| *** dprince has quit IRC | 19:08 | |
| *** jnoller has joined #openstack-infra | 19:08 | |
| *** matjazp has quit IRC | 19:08 | |
| clarkb | fungi: before you take that well deserved vacation to do real life stuff, we should sit down and figure out puppet master backups. Tomorrow after meeting work? | 19:09 |
| jogo | davidlenwell: its 72 | 19:09 |
| *** hogepodge has joined #openstack-infra | 19:09 | |
| fungi | clarkb: works for me | 19:09 |
| *** matjazp_ has joined #openstack-infra | 19:10 | |
| sdague | do we have anyone that has access to turbo hipster in this hemispher? | 19:11 |
| *** maxbit has joined #openstack-infra | 19:11 | |
| clarkb | sdague: not that I know of | 19:11 |
| sdague | ok, it appears to be drunk | 19:11 |
| clarkb | they should be waking soon though | 19:11 |
| mtreinish | jogo: I figured out why it doesn't work. NullHandler just runs pass and doesn't actually do anything | 19:13 |
| mtreinish | so I don't think handleError ever gets called | 19:13 |
| sdague | jeblair: so where is the right entry point for the unit tests here to prevent this from happening again? | 19:14 |
| *** nati_ueno has joined #openstack-infra | 19:14 | |
| *** pcm_ has quit IRC | 19:15 | |
| *** ominakov has joined #openstack-infra | 19:15 | |
| *** ominakov has quit IRC | 19:15 | |
| *** pcrews_ has joined #openstack-infra | 19:16 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Run bup backups on the eavesdrop server. https://review.openstack.org/85789 | 19:17 |
| clarkb | that ^ is a WIP, until I create the user on the backup server | 19:18 |
| clarkb | I also need to update the commit message with the bug | 19:18 |
| sdague | mtreinish: does the /dev/null approach actually work? | 19:19 |
| *** pcrews has quit IRC | 19:19 | |
| sdague | actually, that will break on windows | 19:19 |
| sdague | so that's no dice | 19:19 |
| mtreinish | sdague: dunno I thought I push it out as a test | 19:19 |
| mtreinish | people run the unit tests on windows!? | 19:20 |
| sdague | mtreinish: I don't know if hyperv runs the unit tests | 19:20 |
| *** unicell has quit IRC | 19:22 | |
| clarkb | fungi: any chance you can quickly update https://review.openstack.org/#/c/85435/1 to address jhesketh's comment? | 19:22 |
| clarkb | fungi: I think fixing that would be a good thing to push through :) | 19:23 |
| *** mrodden1 has joined #openstack-infra | 19:23 | |
| *** matjazp_ has quit IRC | 19:23 | |
| *** matjazp has joined #openstack-infra | 19:23 | |
| *** _nadya_ has quit IRC | 19:23 | |
| *** zns has quit IRC | 19:25 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: triggered-builds: Support property file injection https://review.openstack.org/84734 | 19:26 |
| *** markmcclain has quit IRC | 19:26 | |
| *** mrodden has quit IRC | 19:27 | |
| fungi | clarkb: yeah, was starting to do that, but i need to research the circumstances under which it came to be. i only noticed it was wrong, but don't have any actual context as to why it's that way | 19:27 |
| mtreinish | sdague: it failed locally for me with: IOError: [Errno 24] Too many open files: '/dev/null' | 19:28 |
| mtreinish | :) | 19:28 |
| clarkb | fungi: gotcha, from what I can tell that doesn't actually cause any functional problems other than possibly building images twice a day? | 19:28 |
| mtreinish | way too many logs | 19:28 |
| *** thedodd has quit IRC | 19:29 | |
| fungi | clarkb: well, it caused a problem in that we cargo-culted the wrong value into openstack-infra/config and then nobody noticed when we tried to update the config that it didn't switch the time image builds ran | 19:29 |
| clarkb | fungi: ya, it is definitely not most correct :) | 19:30 |
| fungi | because the previous value in the config was also the baked-in default in the code | 19:30 |
| *** thedodd has joined #openstack-infra | 19:31 | |
| *** thedodd has quit IRC | 19:31 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add subunit log test sanity https://review.openstack.org/85797 | 19:32 |
| jeblair | sdague: probably...oh look | 19:32 |
| sdague | jeblair: yeh, I think I found it | 19:32 |
| jeblair | sdague: yes :) | 19:32 |
| *** malini is now known as malini_afk | 19:32 | |
| *** thedodd has joined #openstack-infra | 19:33 | |
| jeblair | sdague, jhesketh: maybe in the future we could add an argument to the swift uploader to specify a max size, and it could error out appropriately (which probably means upload an error report instead of the logs) | 19:34 |
| *** e0ne has quit IRC | 19:34 | |
| *** e0ne has joined #openstack-infra | 19:34 | |
| *** hashar has joined #openstack-infra | 19:34 | |
| sdague | jeblair: sure | 19:35 |
| *** matjazp has quit IRC | 19:36 | |
| *** mrmartin has joined #openstack-infra | 19:36 | |
| anteaya | fungi: I am trying to find the absolute path for /var/lib/git, so in this example: https://review.openstack.org/#/c/82512/3/modules/openstack_project/files/review.projects.yaml would the patch be openstack/neutron-specs/var/lib/git | 19:36 |
| therve | jeblair, Hi, if you have some time to look at https://review.openstack.org/#/c/83761/ later that'd be great | 19:37 |
| clarkb | therve: do you really need devstack to do that? | 19:37 |
| clarkb | therve: heat can't do it without nova, cinder, neutron, swift, glance et al installed? | 19:38 |
| *** doddstack has joined #openstack-infra | 19:38 | |
| *** UtahDave has joined #openstack-infra | 19:38 | |
| anteaya | *would the path be | 19:38 |
| *** UtahDave has left #openstack-infra | 19:38 | |
| fungi | anteaya: there would be a /var/lib/git/openstack/neutron-specs.git directory on your gerrit server | 19:39 |
| *** e0ne has quit IRC | 19:39 | |
| anteaya | ah, so we need the project.git file in the /var/lib/git/openstack dir | 19:39 |
| sdague | mtreinish: yeh I think this needs a more thought out solution. I think a string buffer that gets reset regularly would be solid | 19:39 |
| anteaya | k thanks | 19:39 |
| *** thedodd has quit IRC | 19:40 | |
| *** freyes_ has quit IRC | 19:42 | |
| mtreinish | sdague: and how would we periodically reset it? Every setup()? | 19:42 |
| sdague | mtreinish: yeh | 19:42 |
| anteaya | fungi: and a /var/lib/git/openstack/neutron-specs.git file on all 5 git servers | 19:42 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Added send-to options support to email-ext plugin https://review.openstack.org/73601 | 19:45 |
| mgagne | zaro: ping | 19:46 |
| clarkb | therve: I ask because if the heat template checking can be done without devstack we can run the test more quickly using fewer resources | 19:46 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add support for Stash as a repository viewer https://review.openstack.org/81218 | 19:46 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add subunit log test sanity https://review.openstack.org/85797 | 19:48 |
| *** _nadya_ has joined #openstack-infra | 19:48 | |
| clarkb | therve: reading https://git.openstack.org/cgit/openstack/heat-templates/tree/tools/validate-templates it does not appear to need root or openstack to be running. But that assumes heat template-validate doesn't talk to things like nova | 19:49 |
| clarkb | therve: can you confirm and if so we should update that job | 19:49 |
| *** matjazp has joined #openstack-infra | 19:50 | |
| fungi | anteaya: yes | 19:50 |
| *** yfried has quit IRC | 19:51 | |
| anteaya | fungi: great thanks | 19:51 |
| *** yfried has joined #openstack-infra | 19:51 | |
| sdague | fungi: so I provided comments back on https://review.openstack.org/#/c/84829/ | 19:51 |
| *** gondoi is now known as zz_gondoi | 19:51 | |
| sdague | what I discovered was that it was very important not only to check zuul positions, but to test that branches for non zuul changes were checked out correctly | 19:52 |
| sdague | because that's easy to screw up | 19:52 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Small documentation tweaks https://review.openstack.org/84950 | 19:52 |
| zaro | mgagne: yo! | 19:52 |
| mgagne | zaro: hi! | 19:52 |
| *** mrodden1 is now known as mrodden | 19:52 | |
| mgagne | zaro: did you started having issues with JJB where macros aren't substituted properly anymore? | 19:52 |
| sdague | for instance, the initial algorithm created variables like $OVERRIDE_OSLO.CONFIG_BRANCH | 19:52 |
| *** zz_gondoi is now known as gondoi | 19:53 | |
| sdague | which, you know, don't work... | 19:53 |
| mgagne | zaro: My jobs are now scheduled on {node} | 19:53 |
| *** ociuhandu has joined #openstack-infra | 19:53 | |
| zaro | mgagne: no, have not. | 19:53 |
| zaro | mgagne: are you on master or some version? | 19:54 |
| mgagne | zaro: master | 19:54 |
| mtreinish | sdague: oh fun nonatomic io to the stream object makes this fun. Basically a bunch of: "ValueError: I/O operation on closed file"... | 19:54 |
| zaro | mgagne: when did you notice? | 19:54 |
| mgagne | zaro: now =) | 19:54 |
| sdague | mtreinish: you should leave it to jogo :) | 19:55 |
| mtreinish | sdague: heh, yeah that's what I'm thinking | 19:55 |
| zaro | mgagne: i think openstack infra is using master as well and i haven't heard any wackoness in channel so must still be working for us. | 19:56 |
| zaro | mgagne: i'll do a test now and see if i can repo. | 19:56 |
| mgagne | zaro: hmmm checking again on my side, could be so weird use cases | 19:56 |
| mgagne | zaro: some jobs are ok | 19:56 |
| *** _nadya_ has quit IRC | 19:57 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Output name of ini file when it's not valid https://review.openstack.org/85553 | 19:58 |
| *** jooools has quit IRC | 19:58 | |
| mgagne | zaro: ok, don't know how it worked before by my jobs weren't job-template | 19:58 |
| *** SumitNaiksatam has quit IRC | 19:59 | |
| mgagne | zaro: I'm wondering if macro substitutions once worked with job definitions (not job-template) | 19:59 |
| mgagne | zaro: or my jobs were broken and I didn't know (I don't run this particular job often) | 19:59 |
| zaro | mgagne: i don't think so. it needs to be a template | 19:59 |
| mgagne | zaro: ok, might have broken that jobs months ago and didn't know, sorry for the noise :-/ | 20:00 |
| zaro | np | 20:00 |
| *** Sukhdev has quit IRC | 20:00 | |
| *** sabari has quit IRC | 20:02 | |
| *** harlowja has quit IRC | 20:03 | |
| *** Vivek has joined #openstack-infra | 20:05 | |
| *** Vivek is now known as Guest456 | 20:06 | |
| clarkb | pleia2: https://review.openstack.org/#/c/82106/ reviewed | 20:07 |
| clarkb | pleia2: couple of things that need to be addressed, but I also have a bigger question in the cover message that may or may not be addressed. Let me know what you think | 20:08 |
| *** dims has quit IRC | 20:08 | |
| *** fboz has joined #openstack-infra | 20:09 | |
| *** wayneseguin has joined #openstack-infra | 20:09 | |
| *** russellb_ has joined #openstack-infra | 20:09 | |
| *** fboz is now known as fbo | 20:09 | |
| *** cbkyeoh has joined #openstack-infra | 20:09 | |
| *** vishy has quit IRC | 20:10 | |
| *** Clabbe has quit IRC | 20:10 | |
| *** cyeoh has quit IRC | 20:10 | |
| *** wayneeseguin has quit IRC | 20:10 | |
| *** boris-42 has quit IRC | 20:10 | |
| *** russellb has quit IRC | 20:10 | |
| *** Guest96472 has quit IRC | 20:10 | |
| *** fbo_away has quit IRC | 20:10 | |
| *** daenney has quit IRC | 20:10 | |
| *** cbkyeoh is now known as cyeoh | 20:10 | |
| sdague | anyone know why subunit_log.txt doesn't respect locale? | 20:10 |
| *** russellb_ is now known as russellb | 20:10 | |
| *** wayneseguin is now known as wayneeseguin | 20:10 | |
| *** vishy has joined #openstack-infra | 20:12 | |
| *** SumitNaiksatam has joined #openstack-infra | 20:12 | |
| *** SumitNaiksatam has quit IRC | 20:13 | |
| *** SumitNaiksatam has joined #openstack-infra | 20:13 | |
| pleia2 | clarkb: ok, thanks | 20:15 |
| *** markmcclain has joined #openstack-infra | 20:18 | |
| *** harlowja has joined #openstack-infra | 20:19 | |
| sdague | mtreinish: though, follow on question, you know how to set the loglevel of just one module through FakeLogger? | 20:23 |
| *** sarob has joined #openstack-infra | 20:23 | |
| *** maxbit has quit IRC | 20:24 | |
| *** mrmartin has quit IRC | 20:25 | |
| *** aysyd has quit IRC | 20:25 | |
| *** aysyd has joined #openstack-infra | 20:26 | |
| mordred | anteaya: hey - sorry, was afk | 20:26 |
| mordred | anteaya: I will definitely write down thoughts - although I think that fungi has expressed them decently above | 20:26 |
| mordred | but to summarize - I think the most important problem to solve is the sequencing problem - that is "how do I tell puppet to run on one machine before I tell puppet to run on another machine" | 20:27 |
| anteaya | mordred: okay | 20:27 |
| anteaya | right, salt needs to know what to look for | 20:27 |
| mordred | well, I think taht's a driver script issue - such as... | 20:27 |
| anteaya | there is a file presence function or a directory presence funtion | 20:27 |
| *** amcrn has quit IRC | 20:28 | |
| mordred | salt git0*.openstack.org run-puppet ; salt review.openstack.org run-puppet | 20:28 |
| * mordred waves hands in the air | 20:28 | |
| *** freyes_ has joined #openstack-infra | 20:28 | |
| *** sabari has joined #openstack-infra | 20:28 | |
| *** boris-42 has joined #openstack-infra | 20:28 | |
| jeblair | dhellmann, fungi: i'm going to propose an alternate implementation of https://review.openstack.org/#/c/85487 ; it may not completely solve the issue fungi was getting at, but i think it will make things simpler | 20:28 |
| mtreinish | sdague: yeah the fakeLogger fixture has a name param | 20:29 |
| mtreinish | you just give it the module name there and it will only work on that one | 20:29 |
| anteaya | sudo salt -G 'fqdn:git*' file.directory_exists /var/lib/git | 20:29 |
| sdague | hmmm.. but that won't help me with the override | 20:29 |
| anteaya | different command for the file but yeah | 20:30 |
| sdague | neutron unit tests need to dump nova.api.extension log messages | 20:30 |
| clarkb | jeblair: woot I was just looking at that and trying to come up with a proper solution but I think the tooling simply doesn't exist yet. We need something that expresses the matrix in an easy to read/verify way that can be cross checked against a layout.yaml | 20:30 |
| clarkb | I think the hard bit is the representation | 20:30 |
| anteaya | now I have to work on where to put it | 20:30 |
| mordred | anteaya: why would you do a file.directory_Exists? | 20:31 |
| *** melwitt has joined #openstack-infra | 20:31 | |
| *** pcrews__ has joined #openstack-infra | 20:32 | |
| *** markmcclain has quit IRC | 20:32 | |
| *** markmcclain has joined #openstack-infra | 20:32 | |
| anteaya | well for a while I thought that was the requirement, but I am wrong | 20:32 |
| mordred | anteaya: sorry - I believe I've been remiss in speaking clearly :( | 20:32 |
| anteaya | now I realize I am looking for a file within /var/lib/git | 20:32 |
| mordred | I don't think you are | 20:32 |
| anteaya | no you are fine | 20:32 |
| anteaya | am I not ensureing that /var/lib/git/dir/project.git exists on all 5 git servers? | 20:33 |
| fungi | clarkb: jeblair: and i think the fact that in a cursory review i found at least one asymmetry and another defined jib not being run the the layout, the "this is too complex to do this way" is not a theoretical concern | 20:33 |
| mordred | I think all we need is "salt -G 'fdqn:git*' cmd.run 'puppet agent --test'" | 20:33 |
| mordred | nope | 20:33 |
| mordred | because that's already done by puppet | 20:33 |
| anteaya | where | 20:33 |
| sdague | mtreinish: there doesn't seem to be anything like setlevel in the FakeLogger code | 20:33 |
| anteaya | I thought it wasn't | 20:33 |
| clarkb | fungi: agreed | 20:33 |
| sdague | which is what I need | 20:33 |
| anteaya | that is the race | 20:33 |
| *** amcrn has joined #openstack-infra | 20:33 | |
| mordred | right - it's a race because we don't control _When_ puppet runs | 20:34 |
| sdague | because i need to setlevel neutron.api.extension log level to ERROR, but leave the rest | 20:34 |
| anteaya | mordred: okay explain | 20:34 |
| dhellmann | jeblair, fungi : yeah, that was sort of painful to put together, so something better would be good | 20:34 |
| *** pcrews_ has quit IRC | 20:34 | |
| *** andreaf has joined #openstack-infra | 20:34 | |
| * anteaya wishes she could at least look at a git server and poke around, not changing anything | 20:34 | |
| mordred | anteaya: if, instead of puppet agent running | 20:34 |
| dhellmann | jeblair, fungi : I didn't exactly want to write a parser for updating the needed files :-/ | 20:34 |
| mordred | anteaya: we ran this script on the puppet master - at first ina cron job: | 20:35 |
| mordred | https://review.openstack.org/#/c/80976/2/run_all.sh | 20:35 |
| mordred | anteaya: and then, we took this script: https://review.openstack.org/#/c/80976/2/run_remote_puppet.sh - which is called from the other - and changed the "run_remote_puppet" function | 20:35 |
| fungi | dhellmann: agreed--and i think it would just get worse as things grow further. that way lies madness | 20:35 |
| mordred | to do "salt -G 'fqdn:$1' cmd.run 'puppet agent --test'" (hand wavey on the syntax there) instead of puppet kick | 20:36 |
| *** mgagne has quit IRC | 20:36 | |
| mordred | then, I'm sure there will be 100 bugs - but in general that's the sketch of step one of the need | 20:36 |
| *** mgagne has joined #openstack-infra | 20:36 | |
| mordred | it may want to look VERY different | 20:36 |
| dhellmann | fungi: I thought about having one job that would run all of the tests for the other libs, but that will make it more difficult to set the voting/non-voting status for a given use | 20:36 |
| anteaya | this seems to be a different direction than get manage-project working | 20:36 |
| anteaya | and to be honest, I feel _extremely_ unproductive in infra | 20:37 |
| dhellmann | fungi: and to be fair, it should be easier to update the configs when it is one library at a time | 20:37 |
| anteaya | mostly because every time I try to do anything, i am redirected many times | 20:37 |
| clarkb | dhellmann: fungi: it certainly feels more like an integration test and that is currently how we do integration testing | 20:37 |
| anteaya | so I go away and answer questions and then try again | 20:37 |
| jeblair | dhellmann: oh, hrm, that was part of my plan | 20:37 |
| mordred | anteaya: I am very sorry for that | 20:38 |
| dhellmann | clarkb: yeah, it is, but I don't want to start gating an oslo lib on a project just because someone adopts it | 20:38 |
| anteaya | it isn't your fault | 20:38 |
| anteaya | it is the nature of working in the environment when there is such a large gap in knowledge | 20:38 |
| anteaya | I don't know much | 20:38 |
| anteaya | and I am slow to pick up | 20:38 |
| clarkb | fyi I have approved https://review.openstack.org/#/c/85485/ but it won't get applied until npf | 20:38 |
| dhellmann | jeblair: yeah, I would like to have *some* control over who gets to gate us | 20:38 |
| anteaya | so it is really hard for me to move at the speed of others | 20:38 |
| anteaya | and I get frustrated because I don't want to hold up other people | 20:39 |
| anteaya | but the result is taht I produce nothing | 20:39 |
| anteaya | and I am tired of producing nothing | 20:39 |
| anteaya | so even if it is just really small | 20:39 |
| anteaya | I would like to try to produce something | 20:39 |
| dhellmann | jeblair: I guess if we limit it to projects under openstack/* -- which is basically what I did here | 20:40 |
| mordred | anteaya: well, I think it's great that you're working on this - is a task we've been putting off for quite a while - I know it can be frustrating, and I'll try to be clearer in the future | 20:40 |
| jeblair | mordred, anteaya: i think the really simple thing of just running the puppet commands in sequence is the way to go; we _urgently_ need this working and it doesn't have to be fancy | 20:40 |
| anteaya | can I keep working on what I am doing | 20:41 |
| jeblair | mordred, anteaya: in other words -- the less work we can put into this, the better | 20:41 |
| anteaya | at least until I can get a patch up? | 20:41 |
| *** saju_m has quit IRC | 20:41 | |
| mordred | anteaya: at this point in time, we basically want to use salt to orchestrate runs of puppet | 20:41 |
| dhellmann | jeblair: another concern I had with One Big Job was timeouts, but that may be unfounded | 20:41 |
| clarkb | fungi: re https://review.openstack.org/#/c/85745/1 has hiera been updated? | 20:42 |
| mordred | anteaya: we don't want to re-encode the work that puppet is doing in salt - and I'm sorry if I've indicated that that's what we want | 20:42 |
| mordred | anteaya: that said- I'd love to see what you're working on | 20:42 |
| *** aysyd has quit IRC | 20:42 | |
| *** aysyd has joined #openstack-infra | 20:42 | |
| *** sweston has joined #openstack-infra | 20:42 | |
| mtreinish | sdague: it just grabs the logger and runs setLevel during setup. | 20:43 |
| *** dprince has joined #openstack-infra | 20:43 | |
| *** mgagne_ has joined #openstack-infra | 20:43 | |
| fungi | clarkb: oh, yep, i just realized that having "I'll go ahead and get the corresponding keys for these values into hiera so that this won't break." in one comment and then "I've updated this." in a subsequent comment was perhaps slightly vague. it is in hiera now | 20:44 |
| *** mgagne has quit IRC | 20:44 | |
| mtreinish | sdague: so you could just write a subclass fixture that has a setLevel method | 20:44 |
| clarkb | fungi: ya, I wasn't quite sure what was updated :) | 20:44 |
| clarkb | fungi: thanks approving | 20:44 |
| *** mgagne_ is now known as mgagne | 20:44 | |
| *** mgagne has quit IRC | 20:45 | |
| *** mgagne has joined #openstack-infra | 20:45 | |
| jeblair | dhellmann: so i was going to make one job per oslo lib; so you would be able to set non-voting for all of the runs of the oslo.foo cross-test jobs... | 20:45 |
| dhellmann | jeblair: how would we make that symmetric? | 20:45 |
| anteaya | mordred: so far my understanding is that changes to review.projects.yaml triggers the running of create-cgit-repos | 20:46 |
| dhellmann | jeblair: with the apps, that is | 20:46 |
| *** matjazp has quit IRC | 20:46 | |
| anteaya | mordred: create-cgit-repos creates a new .git file in /var/lib/git/directory/project.git for all 5 servers for the git farm | 20:46 |
| jeblair | dhellmann: indeed, i think my attempt won't work | 20:46 |
| sdague | mtreinish: so I could do a 2nd log fixture? | 20:46 |
| *** dcramer_ has quit IRC | 20:47 | |
| anteaya | if manage-projects runs before all 5 git servers have /var/lib/git/directory/project.git on them, manage-projects fails | 20:47 |
| jeblair | dhellmann: i'm starting to think that either your approach, or the "single job" are the best approaches... | 20:47 |
| mordred | anteaya: that's right | 20:48 |
| jeblair | dhellmann, clarkb, fungi: what if we had a couple of variants of the 'single job'? the main one gates X*Y for all of the openstack and oslo libs | 20:48 |
| anteaya | I am working on the code (and trying to find the place to put it) so that salt tracks when /var/lib/git/directory/new-project.git exists on all 5 git servers and then fires manage-projects | 20:48 |
| mtreinish | sdague: you can, but what exactly are you trying to do? Do you want to set the level per module? Or override the default set in the base setup on a per case basis? | 20:48 |
| mordred | right. so, that's why I was saying I think there is a simpler way | 20:48 |
| jeblair | dhellmann, clarkb, fungi: then there's a non-voting variant that stackforge repos can subscribe to | 20:48 |
| sdague | mtreinish: I want to always set neutron.api.extensions to ERROR level | 20:48 |
| sdague | in unit tests | 20:48 |
| *** matjazp has joined #openstack-infra | 20:48 | |
| mordred | anteaya: because fi you do taht, you're starting to re-encode the logic of "change to projects.yaml triggers create-cgit-repos" - which is already written | 20:49 |
| jeblair | dhellmann, clarkb, fungi: however, if we did that, i don't think that the oslo jobs should run the non-voting variant (because the change queues will merge) | 20:49 |
| openstackgerrit | A change was merged to openstack-infra/config: Openstackid Mysql connection Hiera variables https://review.openstack.org/85745 | 20:49 |
| fungi | jeblair: a "run all the unit tests" integration test job? sounds reasonable | 20:49 |
| anteaya | mordred: the problem is taht it might be simpler for you, because you understand the current puppet config | 20:49 |
| jeblair | dhellmann, clarkb, fungi: correction: however, if we did that, i don't think that the oslo projects should run the non-voting variant (because the change queues will merge) | 20:49 |
| mordred | all we really need to do is make sure that puppet runs on all 5 git servers before puppet runs on review.o.o | 20:49 |
| anteaya | I might need to write the code in order to comprehend the simpler way | 20:49 |
| mtreinish | sdague: yeah then I'd try adding a second fixture after the one there with the name='neutron.api.extensions' and level=logging.ERROR | 20:49 |
| sdague | yep, trying that now | 20:50 |
| dhellmann | jeblair: yeah, I just set this up non-voting to start because there were so many new jobs on so many projects and I didn't want to be the guy that brought all merges to a halt | 20:50 |
| *** jnoller has quit IRC | 20:50 | |
| mordred | so we just need to ensure a successful run of one command on git* before running the same command on review.o.o | 20:50 |
| anteaya | how | 20:50 |
| mordred | the specifics of what those commands accomplish isn't actually as important to the thing that it running them | 20:50 |
| anteaya | we can know the command was fired | 20:50 |
| *** pcrews_ has joined #openstack-infra | 20:50 | |
| anteaya | puppet has no way to know if it finished | 20:50 |
| anteaya | nibalizer went over that with me | 20:50 |
| openstackgerrit | A change was merged to openstack-infra/config: Add acl overrides on stable branches for tripleo https://review.openstack.org/85485 | 20:50 |
| mordred | I disagree with nibalizer | 20:51 |
| anteaya | he calls it "dropping the kids off at the mall" | 20:51 |
| dhellmann | jeblair: for a big job, would we just run all unit tests for openstack/* ? | 20:51 |
| mordred | I aplologize, I believe that nibalizer is incorrect in this case | 20:51 |
| jeblair | mordred: we _can_ know if it finished and succeeded? | 20:51 |
| * anteaya continues to listen | 20:51 | |
| jeblair | dhellmann: yeah, however, that might take a really long time to run | 20:52 |
| clarkb | fungi: https://review.openstack.org/#/c/85385/3 I have no idea where the email sorting thing is actually fixed. Would it be terrible to make that two changes? | 20:52 |
| dhellmann | jeblair: right | 20:52 |
| jeblair | clarkb, fungi: ^ i've just about talked myself back into the current oslo cross-test state. :( | 20:52 |
| clarkb | jeblair: ok | 20:52 |
| fungi | clarkb: sure, -1 it and i'll do a new patchset | 20:52 |
| dhellmann | jeblair: I can also see project foo not wanting to wait for project bar's unit tests to pass against an oslo lib before the patch in foo can merge | 20:52 |
| dhellmann | jeblair: I do like the idea of a tool to make it easier to manage these jobs | 20:53 |
| *** pcrews__ has quit IRC | 20:53 | |
| *** freyes_ has quit IRC | 20:53 | |
| mordred | jeblair: yeah. if it fails, then the invocation of puppet agent --test returns a status of "1" | 20:53 |
| jeblair | mordred, anteaya: yeah, and then salt will tell us that, right? | 20:54 |
| anteaya | salt can check if /var/lib/git/directory/project.git exists on all -G 'fdqn:git*' servers | 20:54 |
| anteaya | I haven't tested salt to see what return codes it deals with | 20:55 |
| *** pcrews has joined #openstack-infra | 20:55 | |
| *** pcrews_ has quit IRC | 20:55 | |
| anteaya | but there is a returncode function so it can check on the retunr code | 20:55 |
| mordred | so, if salt asks puppet to run on git01.openstack.org, and that asks create-cgit-repo to run on git01.openstack.org - then either the program runs successfully and we can move on to the next server, or the run of puppet fails and we stop and panic and don't run puppet on the next server | 20:55 |
| anteaya | what would trigger a return status of "1" for puppet agent --test? | 20:56 |
| anteaya | I don't know what that does | 20:57 |
| anteaya | from what I have seen, puppet knows about itself | 20:57 |
| *** amotoki has joined #openstack-infra | 20:57 | |
| mordred | anteaya: anything in the puppet run, such as create-cgit-repos failing | 20:57 |
| anteaya | but very little about outcomes | 20:57 |
| anteaya | so if create-cgit-repos throws an exception? | 20:57 |
| mordred | anteaya: all puppet agent --test does is run puppet - if any of the things it expects to do as a result of that are unsuccessful, it will return 1 | 20:57 |
| mordred | yes | 20:57 |
| mordred | or just if create-cgit-repos returns 1 - puppet will too | 20:57 |
| *** hashar has quit IRC | 20:58 | |
| anteaya | well I can try that if you want | 20:58 |
| anteaya | so you want salt to tell puppet to run on the git servers rather than the subscription to review.projects.yaml to tell it to run? | 20:59 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: localize variables https://review.openstack.org/85309 | 20:59 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: allow for generic branch overrides https://review.openstack.org/84829 | 20:59 |
| *** Sukhdev has joined #openstack-infra | 20:59 | |
| mordred | yes. I think it'll be a much more straightforward first step - then after that works, we can talk about different ways to trigger salt action that might be more exciting | 20:59 |
| clarkb | fungi: does the email get sorted because you open accounts.csv first in the new change? | 21:00 |
| openstackgerrit | lifeless proposed a change to openstack-infra/reviewstats: March updates for tripleo-core https://review.openstack.org/85815 | 21:00 |
| anteaya | jeblair: your thoughts? | 21:00 |
| lifeless | russellb: https://review.openstack.org/85815 | 21:00 |
| sdague | fungi: updated https://review.openstack.org/84829 | 21:00 |
| russellb | lifeless: done | 21:01 |
| lifeless | russellb: \o/ | 21:01 |
| fungi | sdague: lgtm | 21:01 |
| *** MIDENN_ has quit IRC | 21:01 | |
| *** matjazp has quit IRC | 21:02 | |
| *** MIDENN_ has joined #openstack-infra | 21:02 | |
| fungi | clarkb: well, it gets sorted because i prepopulate the list of e-mail addresses from the preferred e-mail (the one listed in the accounts table) before adding those from the account_external_ids table | 21:02 |
| fungi | "sorted" is a poor term there | 21:02 |
| *** pblaho has joined #openstack-infra | 21:03 | |
| fungi | we make sure that the first address listed is the one from the accounts table rather than some arbitrary address from the external ids | 21:03 |
| jeblair | anteaya: i agree -- i think there may be some interesting things we can do later as you are suggesting, but keeping this very simple (even if it's not always optimal) is the best first step | 21:03 |
| *** sweston has quit IRC | 21:03 | |
| anteaya | okay I will try this | 21:03 |
| *** salv-orlando has quit IRC | 21:04 | |
| anteaya | so I would have to remove the subscription trigger in git_backend I think it is | 21:04 |
| anteaya | so do you want the salt master to fire this? | 21:04 |
| anteaya | or the salt trigger | 21:04 |
| anteaya | the salt trigger is set up but has no jobs | 21:04 |
| *** Ryan_Lane1 has joined #openstack-infra | 21:04 | |
| *** Ryan_Lane has quit IRC | 21:05 | |
| *** denis_makogon has quit IRC | 21:05 | |
| anteaya | I think it would be the salt trigger, firing off changes to review.projects.yaml | 21:05 |
| *** salv-orlando has joined #openstack-infra | 21:05 | |
| openstackgerrit | A change was merged to openstack/requirements: Update the keyring dependency https://review.openstack.org/79796 | 21:06 |
| openstackgerrit | A change was merged to openstack/requirements: Upgrades Pbr to exclude 0.7 due to Windows issue https://review.openstack.org/84030 | 21:06 |
| *** dcramer_ has joined #openstack-infra | 21:06 | |
| openstackgerrit | Jenkins proposed a change to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/85816 | 21:06 |
| *** pblaho has quit IRC | 21:06 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: March updates for tripleo-core https://review.openstack.org/85815 | 21:08 |
| mordred | anteaya: I don't think we need salt-trigger to trigger this for right now | 21:08 |
| anteaya | okay I will get the salt master to deal with this | 21:08 |
| *** mgagne has quit IRC | 21:08 | |
| *** daenney has joined #openstack-infra | 21:09 | |
| *** mgagne has joined #openstack-infra | 21:09 | |
| *** mgagne is now known as Guest11621 | 21:09 | |
| *** dprince has quit IRC | 21:09 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Add access control https://review.openstack.org/84873 | 21:09 |
| *** GheRivero_ has joined #openstack-infra | 21:10 | |
| mordred | anteaya: I thnk even that can be easy. for now, as a step one, if you can just look at patchset 1 in https://review.openstack.org/#/c/80976 | 21:10 |
| fungi | right, this is what i meant by confusing the different use cases. salt-trigger.slave was an answer to making puppet agent run everywhere when changes merge to openstack-infra/config | 21:10 |
| mordred | anteaya: and investigate replacing the ssh call with a call to salt cmd.run | 21:10 |
| *** matjazp has joined #openstack-infra | 21:10 | |
| *** sweston has joined #openstack-infra | 21:11 | |
| fungi | salt-trigger.slave was *only* intended as a bridge from zuul events to salt actions | 21:11 |
| *** Guest11621 is now known as mgagne_ | 21:11 | |
| *** mgagne_ has quit IRC | 21:11 | |
| *** mgagne_ has joined #openstack-infra | 21:11 | |
| mordred | yah. what I'm suggesting now is that we geta thing in place that does the thing we want - then next we can work on ways to trigger it that are not cron | 21:13 |
| mordred | sigh | 21:15 |
| mordred | omg | 21:15 |
| mordred | stab stab stabbity stab stab | 21:15 |
| mordred | anteaya: ok - there may be a fun thing for you to poke in to here | 21:15 |
| mordred | salt -s 'pypi.slave.openstack.org' cmd.run '/bin/false' | 21:16 |
| mordred | returns ... | 21:16 |
| mordred | wait for it | 21:16 |
| mordred | 0 | 21:16 |
| mordred | return code from cmd.run does not seem to be the return code of the command run | 21:16 |
| *** jcoufal has joined #openstack-infra | 21:17 | |
| clarkb | does it at least return the return code somehow? | 21:17 |
| *** eharney has quit IRC | 21:20 | |
| *** mfer has quit IRC | 21:20 | |
| *** maxbit has joined #openstack-infra | 21:22 | |
| mordred | no | 21:23 |
| lifeless | clarkb: return codes are so not web scale | 21:23 |
| mordred | lifeless: I forgot | 21:23 |
| clarkb | lifeless: clearly all webscale software should just assume everything worked | 21:24 |
| clarkb | that way we don't need to handle failures | 21:24 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Move bugdaystats to to status.openstack.org https://review.openstack.org/82106 | 21:24 |
| mordred | clarkb: if the returncode is 0 - that means it succeeded! | 21:24 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for nova unit test race bug 1303533 https://review.openstack.org/85625 | 21:25 |
| uvirtbot | Launchpad bug 1303533 in nova "nova.tests.integrated.v3.test_suspend_server.SuspendServerSamplesJsonTest.test_post_resume fails sporadically" [Undecided,New] https://launchpad.net/bugs/1303533 | 21:25 |
| clarkb | mordred: exactly | 21:25 |
| *** dcramer_ has quit IRC | 21:25 | |
| andreaf | mordred: hi - do you have a moment for a review https://review.openstack.org/#/c/79314/ it has already a +2 - I'm setting up experimental keystone v3 tempest checks | 21:25 |
| clarkb | pleia2: lgtm thanks | 21:26 |
| *** atiwari has quit IRC | 21:26 | |
| clarkb | pleia2: and ya I was mistaken about the served by apache stuff | 21:26 |
| clarkb | ok time for me to relocate to outside | 21:26 |
| jogo | mtreinish: looping back to the debug log stuff | 21:27 |
| jogo | sdague mtreinish: what do you want to leave to me? I got distracted by the amazing weather here | 21:27 |
| pleia2 | clarkb: thanks :) | 21:27 |
| sdague | jogo: see -nova | 21:27 |
| sdague | I want the revert, then I think you need a custom ephemeral logger | 21:28 |
| sdague | which is going to take a couple days to write | 21:28 |
| nibalizer | anteaya: i may have misled you sorry about that | 21:29 |
| nibalizer | puppet absolutely knows the return code of an exec | 21:29 |
| jogo | sdague: agreed, but was hoping the answer wasn't leave it up to me :) | 21:30 |
| mtreinish | jogo: none of my approaches for a clever solution work. I think it's all somewhere in the backlog. | 21:30 |
| nibalizer | and will throw a puppet error if the exec returns false | 21:30 |
| *** alff has quit IRC | 21:30 | |
| *** alff_ has quit IRC | 21:30 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add cross-project unit tests for oslo libraries https://review.openstack.org/85487 | 21:30 |
| jogo | mtreinish: you try string buffer logger? | 21:30 |
| *** ameade has quit IRC | 21:30 | |
| nibalizer | it even has a timeout (defaults to 5 minutes) where puppet will abort execution if it takes too long for the exec to run | 21:30 |
| *** harlowja is now known as harlowja_away | 21:31 | |
| mtreinish | jogo: this?: http://paste.openstack.org/show/75256/ | 21:31 |
| nibalizer | the 'drop kids off at the mall' is about you have no idea what the exec has done really, sine it is managing the system outside of puppet | 21:31 |
| anteaya | yeah salt -s <selector> cmd.run argument returns 0 because it succeeded, you need to get the returncode of the job | 21:32 |
| jogo | mtreinish: ttps://docs.python.org/2/library/logging.handlers.html#memoryhandler looks promising | 21:32 |
| anteaya | I had to get something to eat | 21:32 |
| jeblair | nibalizer: you know whether it has succeeded or not -- and don't run B if A fails is satisfactory | 21:33 |
| *** miqui has quit IRC | 21:33 | |
| sdague | jogo: you broke it the first time :) | 21:33 |
| mtreinish | jogo: I'll give that one a try | 21:33 |
| jogo | mtreinish: I think you need to drop the fakelogger all together | 21:33 |
| *** hogepodge has quit IRC | 21:33 | |
| *** mburned is now known as mburned_out | 21:33 | |
| jogo | sdague: I didn't break it I just made things exponentially worse | 21:33 |
| jogo | the issue was already there | 21:33 |
| sdague | jogo: also, I'm proposing failing projects if their uncompressed subunit exceeds 50 MB | 21:33 |
| jogo | but same difference | 21:33 |
| sdague | "I didn't break it I just made things exponentially worse" | 21:34 |
| sdague | that is the new T-shirt moto for Juno summit | 21:34 |
| jogo | sdague: heh | 21:34 |
| jogo | sdague: I would wear that | 21:34 |
| *** harlowja_away is now known as harlowja | 21:35 | |
| nibalizer | jeblair: ya | 21:35 |
| anteaya | mordred: cmd.retcode: http://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.cmdmod.html | 21:36 |
| mordred | ooh | 21:37 |
| mordred | ok. that at least returns the return code | 21:38 |
| anteaya | yes | 21:38 |
| anteaya | which is what you wanted | 21:38 |
| anteaya | was it not? | 21:38 |
| mordred | yes. it returns it strangely - but I think that' workable :) | 21:39 |
| anteaya | mordred: can you paste the output you are seeing? | 21:40 |
| *** dcramer_ has joined #openstack-infra | 21:40 | |
| anteaya | what I get is consistent with how all salt commands output | 21:40 |
| *** alff has joined #openstack-infra | 21:42 | |
| *** alff_ has joined #openstack-infra | 21:42 | |
| *** atiwari has joined #openstack-infra | 21:43 | |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: README.rst: Remove space before bullets https://review.openstack.org/85851 | 21:43 |
| mordred | anteaya: yeah - it's consistent I think - it's printing it on the screen rather than returning the return code | 21:45 |
| fungi | sdague: hard limits like that seem to end up being malleable and perpetually inching upward. we've seen it with timeouts... "but i'm just adding one more test, it's not my fault that the first thousand are inefficient" | 21:46 |
| *** nkinder has quit IRC | 21:46 | |
| mtreinish | fungi: it's totally all your fault... | 21:46 |
| fungi | and so we agree to bump the limit up a little on the promise that there some cleanup or refactoring just around the corner | 21:46 |
| *** tykeal has left #openstack-infra | 21:46 | |
| *** nkinder has joined #openstack-infra | 21:46 | |
| sdague | fungi: sure, however it would catch the immediate overruns | 21:47 |
| sdague | because what we actually mostly care about is order of magnitude jumps | 21:47 |
| anteaya | mordred: here is what I get: http://paste.openstack.org/show/75258/ | 21:47 |
| sdague | which is what we saw with nova | 21:47 |
| fungi | given that run time and test output are variable, it just gets harder and harder to add another test until someone finally ups the limit | 21:47 |
| clarkb | outdoor relocation successful | 21:47 |
| anteaya | mordred: that is how salt outputs | 21:47 |
| *** dkranz has quit IRC | 21:47 | |
| anteaya | mordred: did you want the output differently? | 21:47 |
| mordred | anteaya: yeah. grok it now | 21:48 |
| *** esker has quit IRC | 21:48 | |
| anteaya | k | 21:48 |
| fungi | sdague: so while i agree that it will root out huge jumps (hopefully), it will be another limit we're perpetually bumping | 21:49 |
| *** zehicle_at_dell has joined #openstack-infra | 21:49 | |
| jeblair | mordred, anteaya: maybe this should be a python program that invokes salt's cmd.retcode so that the return values can be processed appropriately | 21:49 |
| *** zehicle has joined #openstack-infra | 21:49 | |
| anteaya | mordred: http://paste.openstack.org/show/75259/ | 21:49 |
| *** lcostantino has quit IRC | 21:50 | |
| anteaya | how do you want to process the returncode? | 21:50 |
| mordred | jeblair: ++ | 21:51 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add subway map to status page https://review.openstack.org/84314 | 21:53 |
| *** jeckersb is now known as jeckersb_gone | 21:53 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Run check-tripleo jobs on stale changes https://review.openstack.org/85854 | 21:54 |
| *** changbl has quit IRC | 21:54 | |
| clarkb | lifeless: ^ | 21:54 |
| lifeless | clarkb: \o/ thank you! | 21:54 |
| *** nkinder has quit IRC | 21:55 | |
| *** GheRivero has quit IRC | 21:55 | |
| lifeless | clarkb: one thought inline on the review | 21:55 |
| *** GheRivero_ is now known as GheRivero | 21:55 | |
| *** mriedem has quit IRC | 21:56 | |
| *** matjazp has quit IRC | 21:57 | |
| *** wenlock_ has joined #openstack-infra | 22:00 | |
| *** pdmars has quit IRC | 22:01 | |
| mordred | jeblair, anteaya: re: return codes - I think "run_all" is what we want - and I think we do want to call from python | 22:01 |
| mordred | jeblair, anteaya: for example: http://paste.openstack.org/show/75261/ | 22:01 |
| *** markmcclain has quit IRC | 22:02 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Run check-tripleo jobs on stale changes https://review.openstack.org/85854 | 22:03 |
| clarkb | lifeless: ^ | 22:03 |
| jeblair | mordred: cool, but why would host not be in output? and if it is not, why is that success? | 22:03 |
| *** wenlock_ has quit IRC | 22:05 | |
| *** mriedem has joined #openstack-infra | 22:06 | |
| *** markmcclain has joined #openstack-infra | 22:06 | |
| *** gondoi is now known as zz_gondoi | 22:07 | |
| *** dcramer_ has quit IRC | 22:08 | |
| *** amcrn has quit IRC | 22:08 | |
| *** hogepodge has joined #openstack-infra | 22:09 | |
| *** maxbit has quit IRC | 22:10 | |
| anteaya | mordred: can you show me the documentation for cmd.run_all? | 22:12 |
| anteaya | I can't seem to find it | 22:12 |
| *** amcrn has joined #openstack-infra | 22:12 | |
| jeblair | anteaya: it was on the page you sent earlier: http://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.cmdmod.html#salt.modules.cmdmod.run_all | 22:13 |
| anteaya | thanks | 22:13 |
| mattoliverau | Morning all | 22:14 |
| anteaya | some of my results: http://paste.openstack.org/show/75264/ | 22:16 |
| *** aysyd has quit IRC | 22:16 | |
| anteaya | hi mattoliverau | 22:16 |
| *** UtahDave has joined #openstack-infra | 22:16 | |
| mattoliverau | anteaya: reading scroll back.. yup we must all vote mikal, otherwise how else am I to live the life style I have become accustomed :P | 22:17 |
| anteaya | :D | 22:17 |
| anteaya | I notice I have no host data | 22:19 |
| anteaya | what output are you getting mordred? | 22:19 |
| *** adalbas has quit IRC | 22:19 | |
| anteaya | I do have stderr | 22:19 |
| mordred | anteaya: http://paste.openstack.org/show/75267/ | 22:20 |
| anteaya | which to be honest looks more important than returncode | 22:20 |
| mordred | anteaya: taht's what I get when I run that script | 22:20 |
| mordred | both are important - a program may print somethign on stderr and still succeed - so that's ok | 22:20 |
| mordred | but we want to abort if something fails - and then be able to print out the contents of its stderr | 22:20 |
| anteaya | so where are you seeing host data? | 22:21 |
| mordred | anteaya: what do you mean by host data? | 22:21 |
| anteaya | oh it is the first argument you give when you call client.cmd | 22:22 |
| mordred | yah | 22:22 |
| *** jeckersb_gone is now known as jeckersb | 22:22 | |
| anteaya | jeblair: had asked about this line: if host in output: | 22:23 |
| mordred | ah - that's because in an earlier test there were times when the return dict did not have an entry for the host | 22:24 |
| mordred | I think run_all will always return a dict for the host | 22:24 |
| openstackgerrit | A change was merged to openstack-infra/config: Add experimental keystonev3 jobs https://review.openstack.org/79314 | 22:26 |
| *** jaypipes has quit IRC | 22:27 | |
| *** hogepodge has quit IRC | 22:30 | |
| jeblair | mordred: k | 22:31 |
| *** jgrimm has quit IRC | 22:31 | |
| *** prad has quit IRC | 22:33 | |
| *** signed8bit has quit IRC | 22:33 | |
| anteaya | mordred: okay well I look forward to your patch with that it in | 22:34 |
| *** andreykurilin_ has quit IRC | 22:34 | |
| mordred | but the format is that each of the hosts that ran something are ina dict of dicts keyed on hostname | 22:34 |
| *** zhiyan_ has quit IRC | 22:34 | |
| *** adalbas has joined #openstack-infra | 22:35 | |
| mordred | so if you did cmd.run 'git*.openstack.org' - you'd get back {'git01.openstack.org': {retcode: 0, stdout: ""}, 'git02.openstack.org': '{ ... | 22:35 |
| *** mgagne_ is now known as mgagne | 22:35 | |
| *** zhiyan_ has joined #openstack-infra | 22:35 | |
| *** signed8bit has joined #openstack-infra | 22:39 | |
| *** andreaf has quit IRC | 22:40 | |
| *** thuc has quit IRC | 22:41 | |
| *** amotoki has quit IRC | 22:41 | |
| *** mburned_out is now known as mburned | 22:41 | |
| *** thuc has joined #openstack-infra | 22:41 | |
| anteaya | why not select all the git servers | 22:41 |
| anteaya | why do them one at a time? | 22:41 |
| *** niska has quit IRC | 22:42 | |
| anteaya | salt -G 'fqdn:git*' | 22:42 |
| anteaya | that selects all teh git servers | 22:42 |
| anteaya | then you can do cmd.run command | 22:42 |
| mordred | anteaya: I thnk that's exactly right | 22:42 |
| jhesketh | Morning | 22:43 |
| mattoliverau | anteaya: I never recieved an email with a link to the election. | 22:43 |
| anteaya | morning jhesketh | 22:43 |
| mattoliverau | morning jhesketh | 22:43 |
| anteaya | mattoliverau: can you email me and tristanC | 22:43 |
| anteaya | so we can track it | 22:43 |
| mattoliverau | Will do :) | 22:43 |
| anteaya | mattoliverau: http://lists.openstack.org/pipermail/openstack-dev/2014-April/031919.html | 22:44 |
| anteaya | under what to do if you don't see the email | 22:44 |
| anteaya | and thanks | 22:44 |
| mattoliverau | OK | 22:44 |
| *** rcleere has quit IRC | 22:44 | |
| *** zns has joined #openstack-infra | 22:44 | |
| *** alexpilotti has quit IRC | 22:45 | |
| *** thuc has quit IRC | 22:46 | |
| *** arosen has joined #openstack-infra | 22:47 | |
| *** nkinder has joined #openstack-infra | 22:47 | |
| arosen | Hi guys, I was hoping someone could point out to me why the tests i added here https://review.openstack.org/#/c/85801/ novadocker/virt/docker/vifs.py aren't running: https://jenkins02.openstack.org/job/gate-nova-docker-python26/24/console | 22:47 |
| * arosen novadocker/tests/virt/docker/test_vifs.py | 22:48 | |
| *** jhesketh has quit IRC | 22:48 | |
| *** niska has joined #openstack-infra | 22:48 | |
| *** greghaynes has joined #openstack-infra | 22:49 | |
| anteaya | mordred: so what happens now? | 22:49 |
| *** msabramo has quit IRC | 22:49 | |
| *** jhesketh has joined #openstack-infra | 22:49 | |
| clarkb | arosen: are they in the discover path? check .testr.conf | 22:49 |
| mordred | anteaya: afk - let me write slightly more code, then hand it to you with some thoughts on next step | 22:51 |
| arosen | clarkb: Looks like it: the other tests in the same dir are running: https://github.com/stackforge/nova-docker/blob/master/.testr.conf | 22:51 |
| clarkb | ya it is discovering in the root of the repo | 22:51 |
| clarkb | so that isn't it | 22:52 |
| anteaya | mordred: k, I am afk too | 22:52 |
| anteaya | will read scrollback | 22:52 |
| *** mriedem has quit IRC | 22:52 | |
| *** thuc has joined #openstack-infra | 22:54 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-infra/config: Run unit tests in tripleo-image-elements https://review.openstack.org/83886 | 22:54 |
| *** mrodden has quit IRC | 22:54 | |
| *** thuc has quit IRC | 22:55 | |
| *** thuc_ has joined #openstack-infra | 22:55 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Added config options to not overwrite jobs desc https://review.openstack.org/52080 | 22:55 |
| *** thuc_ has quit IRC | 22:56 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Support recursive processing of paths https://review.openstack.org/68895 | 22:56 |
| *** thuc has joined #openstack-infra | 22:56 | |
| *** thuc_ has joined #openstack-infra | 22:57 | |
| arosen | clarkb: actually it looks like those tests ran looking here: http://logs.openstack.org/01/85801/3/check/gate-nova-docker-python27/5422c8e/testr_results.html.gz test_plug_vifs_bridge_two_interfaces - pt1.119: novadocker.tests.virt.docker.test_vifs.DockerGenericVIFDriverTestCase.test_plug_vifs_bridge_two_interfaces | 22:58 |
| arosen | clarkb: but looking at console.html they don't show up: http://logs.openstack.org/01/85801/3/check/gate-nova-docker-python27/5422c8e/console.html | 22:58 |
| *** _nadya_ has joined #openstack-infra | 22:58 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Pin facter everywhere. https://review.openstack.org/85864 | 22:58 |
| clarkb | jeblair: fungi mordred ^ | 22:58 |
| clarkb | arosen: console log isn't listing all tests | 22:59 |
| clarkb | arosen: I think you are fine | 22:59 |
| arosen | clarkb: why is the console log not listing all the tests though? | 22:59 |
| *** Sukhdev has quit IRC | 22:59 | |
| clarkb | arosen: because it wasn't asked to. testr doesn't list all tests by default | 22:59 |
| *** saschpe has quit IRC | 23:00 | |
| *** thuc has quit IRC | 23:00 | |
| arosen | clarkb: is this all controlled by .testr.conf? Seems like we would want to enabled this. | 23:00 |
| clarkb | no it is controlled by tox.ini | 23:01 |
| clarkb | arosen: fwiw I really like the default behavipr | 23:01 |
| clarkb | listing every test that passed is just noise | 23:01 |
| arosen | clarkb: sorry gotta run to a meeting X.x brb | 23:01 |
| *** _nadya_ has quit IRC | 23:02 | |
| *** jamespage has quit IRC | 23:04 | |
| *** gnuoy` has quit IRC | 23:04 | |
| *** jamespage has joined #openstack-infra | 23:05 | |
| *** jamespage has joined #openstack-infra | 23:05 | |
| *** gnuoy has joined #openstack-infra | 23:05 | |
| *** UtahDave1 has joined #openstack-infra | 23:05 | |
| *** UtahDave has quit IRC | 23:05 | |
| *** UtahDave1 is now known as UtahDave | 23:05 | |
| *** thuc_ has quit IRC | 23:06 | |
| *** thuc has joined #openstack-infra | 23:06 | |
| *** msabramo has joined #openstack-infra | 23:07 | |
| *** jamielennox|away is now known as jamielennox | 23:07 | |
| *** wenlock has quit IRC | 23:08 | |
| *** thuc has quit IRC | 23:11 | |
| openstackgerrit | A change was merged to openstack-infra/config: Pin facter everywhere. https://review.openstack.org/85864 | 23:11 |
| *** Sukhdev has joined #openstack-infra | 23:13 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Pin facter on centos boxes https://review.openstack.org/85865 | 23:13 |
| clarkb | fungi: jeblair mordred ^ | 23:13 |
| jeblair | clarkb: i think you forgot to add the file | 23:14 |
| clarkb | jeblair: gah one sec | 23:14 |
| lifeless | erm | 23:14 |
| lifeless | HTTP ERROR: 500 | 23:14 |
| lifeless | Problem accessing /. Reason: | 23:14 |
| lifeless | Guice provision errors: | 23:14 |
| lifeless | 1) Cannot open ReviewDb | 23:14 |
| lifeless | while locating com.google.gerrit.server.config.RequestScopedReviewDbProvider | 23:14 |
| lifeless | at com.google.gerrit.server.config.GerritRequestModule.configure(GerritRequestModule.java:73) | 23:14 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Pin facter on centos boxes https://review.openstack.org/85865 | 23:15 |
| lifeless | while locating com.google.gerrit.reviewdb.server.ReviewDb | 23:15 |
| *** mrda has joined #openstack-infra | 23:15 | |
| clarkb | fungi: mordred did we upgrade mysql on that node? | 23:15 |
| lifeless | clarkb: ^ | 23:15 |
| clarkb | gerrit may need kicking | 23:15 |
| lifeless | ah its back | 23:15 |
| lifeless | clearly I'm reviewing too much | 23:15 |
| fungi | lifeless: clarkb yep, restarting now | 23:16 |
| *** doddstack has quit IRC | 23:16 | |
| *** msabramo has quit IRC | 23:16 | |
| fungi | should we just reboot gerrit? | 23:16 |
| clarkb | fungi: ya, why don't we wait for reboot time | 23:16 |
| fungi | just to note, mysql upgrades restart mysqld | 23:17 |
| *** david-lyle has quit IRC | 23:20 | |
| *** gyee has quit IRC | 23:21 | |
| *** jcoufal has quit IRC | 23:22 | |
| *** yamahata has quit IRC | 23:24 | |
| dstufft | so are you guys upgrading your openssl :V | 23:24 |
| pleia2 | typically do security updates automatically, but this one might be worth pre-empting, fungi? | 23:25 |
| fungi | our silence in here should speak volumes | 23:25 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Changed task states https://review.openstack.org/85868 | 23:25 |
| dstufft | also revoking all your private keys :( | 23:25 |
| pleia2 | \o/ | 23:25 |
| fungi | remaking all the things | 23:25 |
| dstufft | GJ OpenSSL on 0-daying the entire internet | 23:26 |
| pleia2 | heh, right | 23:26 |
| dstufft | fungi: didn't mean to imply y'all wernt on it btw :) Just going through the motions to make sure stuff I care about is being handled :] | 23:27 |
| fungi | dstufft: i'm glad you care ;) | 23:27 |
| *** jhesketh has quit IRC | 23:27 | |
| *** whayutin_ has quit IRC | 23:28 | |
| *** reed has quit IRC | 23:28 | |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Bump pep8, pyflakes and flake8 versions https://review.openstack.org/85496 | 23:29 |
| *** alchen99 has joined #openstack-infra | 23:30 | |
| *** homeless has quit IRC | 23:30 | |
| *** flaper87 is now known as flaper87|afk | 23:33 | |
| *** amcrn has quit IRC | 23:33 | |
| *** salv-orlando has quit IRC | 23:34 | |
| *** markmcclain has quit IRC | 23:35 | |
| *** markmcclain has joined #openstack-infra | 23:35 | |
| fungi | crappity. apt-get dist-upgrade is pulling in jenkins 1.558 | 23:37 |
| fungi | i was able to stop the madness after it upgraded jenkins.o.o and before it upgraded jenkins01 | 23:38 |
| clarkb | yeah dist-upgrade is scary times :) | 23:39 |
| clarkb | we need dist-upgrade with all packages but witout all upgrades :) | 23:39 |
| clarkb | or someting | 23:39 |
| *** jhesketh has joined #openstack-infra | 23:39 | |
| *** bhuvan has joined #openstack-infra | 23:39 | |
| *** koolhead17 has quit IRC | 23:40 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Updated UI for tasks status https://review.openstack.org/85870 | 23:40 |
| *** sabari has quit IRC | 23:45 | |
| *** rlandy has quit IRC | 23:47 | |
| *** zns has quit IRC | 23:49 | |
| *** mburned is now known as mburned_out | 23:50 | |
| *** mburned_out is now known as mburned | 23:50 | |
| *** markmcclain has quit IRC | 23:52 | |
| *** alexpilotti has joined #openstack-infra | 23:53 | |
| *** koolhead17 has joined #openstack-infra | 23:53 | |
| *** unicell has joined #openstack-infra | 23:56 | |
| *** Ryan_Lane1 has quit IRC | 23:57 | |
| *** Ryan_Lane1 has joined #openstack-infra | 23:57 | |
| *** gyee has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!