| *** oubiwan__ has quit IRC | 00:00 | |
| *** CaptTofu has quit IRC | 00:03 | |
| *** CaptTofu has joined #openstack-infra | 00:04 | |
| *** CaptTofu has quit IRC | 00:08 | |
| jesusaurus | oh for the origin thing? | 00:10 |
|---|---|---|
| jesusaurus | yeah... assuming 'origin' was silly of them | 00:11 |
| *** rms_13 has quit IRC | 00:17 | |
| openstackgerrit | Björn Michaelsen proposed a change to openstack-infra/git-review: add query option https://review.openstack.org/76197 | 00:20 |
| *** yamahata has quit IRC | 00:28 | |
| nibalizer | mordred: i think if we use --ignore-dependencies on the puppet module install for puppetboard we'll be good | 00:36 |
| nibalizer | i could add a pass in install_modules.sh that does that, there may be other modules that we want to bring in without bringing their dependencies | 00:37 |
| *** sarob_ has quit IRC | 00:56 | |
| *** sarob_ has joined #openstack-infra | 00:57 | |
| *** dstufft|laptop has quit IRC | 01:00 | |
| *** sarob_ has quit IRC | 01:01 | |
| *** CaptTofu has joined #openstack-infra | 01:07 | |
| *** Daisy_ has joined #openstack-infra | 01:10 | |
| *** CaptTofu has quit IRC | 01:11 | |
| *** dstufft|laptop has joined #openstack-infra | 01:16 | |
| *** mwagner_lap has quit IRC | 01:20 | |
| *** zhiyan_ is now known as zhiyan | 01:24 | |
| *** sarob_ has joined #openstack-infra | 01:29 | |
| *** yamahata has joined #openstack-infra | 01:30 | |
| *** sarob_ has quit IRC | 01:33 | |
| *** nosnos has joined #openstack-infra | 01:37 | |
| *** nosnos has quit IRC | 01:38 | |
| *** nosnos has joined #openstack-infra | 01:40 | |
| *** sarob_ has joined #openstack-infra | 01:59 | |
| *** yaguang has joined #openstack-infra | 02:00 | |
| *** wenlock has joined #openstack-infra | 02:00 | |
| *** mriedem has quit IRC | 02:01 | |
| *** sarob_ has quit IRC | 02:08 | |
| *** yamahata has quit IRC | 02:09 | |
| *** jhesketh has joined #openstack-infra | 02:18 | |
| *** sarob_ has joined #openstack-infra | 02:29 | |
| *** zns has joined #openstack-infra | 02:32 | |
| *** sarob_ has quit IRC | 02:34 | |
| lifeless | mordred: https://review.openstack.org/#/c/71872/1 | 02:41 |
| lifeless | mordred: I thought you said we didn't need to do this ? | 02:42 |
| lifeless | mordred: I am confuse. | 02:42 |
| *** wenlock_ has joined #openstack-infra | 02:45 | |
| *** oubiwan__ has joined #openstack-infra | 02:51 | |
| *** rockstar has quit IRC | 02:55 | |
| *** CaptTofu has joined #openstack-infra | 02:55 | |
| lifeless | rustlebee: https://review.openstack.org/#/c/75764/ | 02:57 |
| *** CaptTofu has quit IRC | 03:00 | |
| *** jhesketh has quit IRC | 03:03 | |
| *** david-lyle has joined #openstack-infra | 03:10 | |
| *** bknudson has quit IRC | 03:14 | |
| *** oubiwan__ has quit IRC | 03:19 | |
| *** Daisy_ has quit IRC | 03:22 | |
| *** Daisy_ has joined #openstack-infra | 03:22 | |
| *** oubiwan__ has joined #openstack-infra | 03:28 | |
| *** sarob_ has joined #openstack-infra | 03:30 | |
| *** zns has quit IRC | 03:37 | |
| *** oubiwan__ has quit IRC | 03:37 | |
| *** basha has joined #openstack-infra | 03:38 | |
| *** rcarrillocruz has joined #openstack-infra | 03:42 | |
| *** rcarrillocruz1 has quit IRC | 03:44 | |
| *** sarob_ has quit IRC | 03:44 | |
| *** yamahata has joined #openstack-infra | 03:44 | |
| *** Ryan_Lane has quit IRC | 03:46 | |
| *** Ryan_Lane has joined #openstack-infra | 03:49 | |
| mordred | lifeless: I was wrong - it can be helpful | 04:00 |
| *** mtreinish has quit IRC | 04:01 | |
| *** mtreinish has joined #openstack-infra | 04:03 | |
| *** sdake__ has quit IRC | 04:04 | |
| *** Daisy_ has quit IRC | 04:06 | |
| *** mtreinish has quit IRC | 04:08 | |
| *** mwagner_lap has joined #openstack-infra | 04:08 | |
| *** mtreinish has joined #openstack-infra | 04:15 | |
| *** basha has quit IRC | 04:30 | |
| *** basha has joined #openstack-infra | 04:35 | |
| *** oubiwan__ has joined #openstack-infra | 04:35 | |
| *** oubiwan__ has quit IRC | 04:36 | |
| *** oubiwan__ has joined #openstack-infra | 04:36 | |
| *** rpodolyaka has joined #openstack-infra | 04:37 | |
| *** sarob_ has joined #openstack-infra | 04:41 | |
| *** chandan_kumar has joined #openstack-infra | 04:43 | |
| *** CaptTofu has joined #openstack-infra | 04:43 | |
| *** CaptTofu has quit IRC | 04:48 | |
| *** chandan_kumar has quit IRC | 04:52 | |
| *** oubiwan__ has quit IRC | 04:54 | |
| *** Daisy_ has joined #openstack-infra | 04:54 | |
| *** saju_m has joined #openstack-infra | 04:59 | |
| *** chandan_kumar has joined #openstack-infra | 05:01 | |
| *** wenlock has quit IRC | 05:04 | |
| *** sarob_ has quit IRC | 05:13 | |
| *** sarob_ has joined #openstack-infra | 05:21 | |
| *** gokrokve has quit IRC | 05:24 | |
| *** gokrokve_ has joined #openstack-infra | 05:26 | |
| *** gokrokve_ has quit IRC | 05:31 | |
| *** gokrokve has joined #openstack-infra | 05:38 | |
| *** nosnos has quit IRC | 05:39 | |
| *** nosnos has joined #openstack-infra | 05:39 | |
| *** talluri has joined #openstack-infra | 05:41 | |
| *** hdd_ has quit IRC | 05:46 | |
| *** sarob_ has quit IRC | 05:54 | |
| *** saju_m has quit IRC | 05:54 | |
| *** saju_m has joined #openstack-infra | 05:55 | |
| *** saju_m has quit IRC | 06:00 | |
| *** saju_m has joined #openstack-infra | 06:02 | |
| *** rpodolyaka has quit IRC | 06:07 | |
| *** rpodolyaka has joined #openstack-infra | 06:08 | |
| *** wdtnbr has quit IRC | 06:09 | |
| openstackgerrit | daisy-ycguo proposed a change to openstack-infra/config: Fix an error of upstream_translation_horizon.sh https://review.openstack.org/79243 | 06:11 |
| openstackgerrit | Kei YAMAZAKI proposed a change to openstack-infra/jenkins-job-builder: Added support for CI Game plugin https://review.openstack.org/79244 | 06:16 |
| *** ildikov_ has quit IRC | 06:17 | |
| *** basha has quit IRC | 06:28 | |
| *** sarob_ has joined #openstack-infra | 06:30 | |
| *** irisaf has joined #openstack-infra | 06:31 | |
| *** CaptTofu has joined #openstack-infra | 06:32 | |
| openstackgerrit | Kei YAMAZAKI proposed a change to openstack-infra/jenkins-job-builder: Added support for CI Game plugin https://review.openstack.org/79244 | 06:36 |
| *** CaptTofu has quit IRC | 06:37 | |
| *** nosnos_ has joined #openstack-infra | 06:51 | |
| *** nosnos has quit IRC | 06:51 | |
| *** wenlock_ has quit IRC | 06:53 | |
| *** vponomaryov has joined #openstack-infra | 06:54 | |
| *** nosnos has joined #openstack-infra | 06:56 | |
| *** ildikov_ has joined #openstack-infra | 06:58 | |
| *** nosnos_ has quit IRC | 06:58 | |
| *** pblaho has joined #openstack-infra | 06:59 | |
| *** irisaf has quit IRC | 07:04 | |
| *** sarob_ has quit IRC | 07:04 | |
| *** nosnos has quit IRC | 07:05 | |
| *** gokrokve has quit IRC | 07:05 | |
| *** gokrokve has joined #openstack-infra | 07:06 | |
| *** nosnos has joined #openstack-infra | 07:08 | |
| *** gokrokve has quit IRC | 07:09 | |
| *** tianst20 has quit IRC | 07:11 | |
| *** tian has quit IRC | 07:11 | |
| *** nosnos has quit IRC | 07:12 | |
| *** nosnos has joined #openstack-infra | 07:12 | |
| *** davidhadas has quit IRC | 07:16 | |
| *** rpodolyaka has quit IRC | 07:24 | |
| *** yolanda_ has joined #openstack-infra | 07:28 | |
| *** sarob_ has joined #openstack-infra | 07:30 | |
| *** jhesketh__ has joined #openstack-infra | 07:31 | |
| *** jhesketh__ has quit IRC | 07:31 | |
| *** gokrokve has joined #openstack-infra | 07:39 | |
| *** matsuhashi has joined #openstack-infra | 07:39 | |
| *** mattoliverau has quit IRC | 07:41 | |
| *** rcarrillocruz1 has joined #openstack-infra | 07:41 | |
| *** dizquierdo has joined #openstack-infra | 07:42 | |
| *** sarob_ has quit IRC | 07:42 | |
| *** rcarrillocruz has quit IRC | 07:42 | |
| *** basha has joined #openstack-infra | 07:43 | |
| *** gokrokve has quit IRC | 07:44 | |
| *** jlibosva has joined #openstack-infra | 07:48 | |
| *** mkoderer has joined #openstack-infra | 07:50 | |
| *** eharney has joined #openstack-infra | 07:55 | |
| *** basha has quit IRC | 07:56 | |
| *** basha has joined #openstack-infra | 07:59 | |
| *** rcarrillocruz1 has quit IRC | 08:00 | |
| *** rlandy has joined #openstack-infra | 08:01 | |
| *** rcarrillocruz has joined #openstack-infra | 08:01 | |
| *** jcoufal has joined #openstack-infra | 08:08 | |
| *** davidhadas has joined #openstack-infra | 08:09 | |
| *** dstufft|laptop has quit IRC | 08:12 | |
| *** flaper87|afk is now known as flaper87 | 08:14 | |
| *** rlandy has quit IRC | 08:15 | |
| *** rlandy has joined #openstack-infra | 08:15 | |
| *** jooools has joined #openstack-infra | 08:17 | |
| *** rpodolyaka has joined #openstack-infra | 08:20 | |
| *** CaptTofu has joined #openstack-infra | 08:20 | |
| *** saju_m has quit IRC | 08:22 | |
| *** CaptTofu has quit IRC | 08:25 | |
| *** jgallard has joined #openstack-infra | 08:25 | |
| *** saju_m has joined #openstack-infra | 08:28 | |
| *** ihrachys|afk is now known as ihrachys | 08:28 | |
| *** sarob_ has joined #openstack-infra | 08:30 | |
| *** davidhadas has quit IRC | 08:31 | |
| *** rpodolyaka has quit IRC | 08:31 | |
| *** davidhadas has joined #openstack-infra | 08:31 | |
| *** hashar has joined #openstack-infra | 08:32 | |
| *** sarob_ has quit IRC | 08:35 | |
| *** gokrokve has joined #openstack-infra | 08:39 | |
| *** gokrokve has quit IRC | 08:44 | |
| *** bada_ has joined #openstack-infra | 08:47 | |
| *** dizquierdo has quit IRC | 08:48 | |
| *** matsuhashi has quit IRC | 08:50 | |
| *** saju_m has quit IRC | 08:50 | |
| *** bada has quit IRC | 08:51 | |
| *** yolanda_ has quit IRC | 08:52 | |
| *** matsuhashi has joined #openstack-infra | 08:54 | |
| *** yolanda has joined #openstack-infra | 08:56 | |
| *** rossella_s has joined #openstack-infra | 08:57 | |
| *** jlibosva has quit IRC | 08:58 | |
| *** jlibosva has joined #openstack-infra | 09:00 | |
| *** wenlock has joined #openstack-infra | 09:02 | |
| *** saju_m has joined #openstack-infra | 09:04 | |
| *** wenlock has quit IRC | 09:07 | |
| *** saju_m has quit IRC | 09:10 | |
| *** yassine has joined #openstack-infra | 09:11 | |
| *** saju_m has joined #openstack-infra | 09:11 | |
| *** talluri has quit IRC | 09:12 | |
| *** derekh has joined #openstack-infra | 09:12 | |
| *** talluri has joined #openstack-infra | 09:13 | |
| *** jpich has joined #openstack-infra | 09:16 | |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Add http(s) protocol support to fetch_review and list_reviews https://review.openstack.org/79280 | 09:25 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Git review assumes the wrong ssh default port https://review.openstack.org/79281 | 09:25 |
| *** jlibosva has quit IRC | 09:27 | |
| *** bada__ has joined #openstack-infra | 09:27 | |
| *** rpodolyaka has joined #openstack-infra | 09:27 | |
| *** sarob_ has joined #openstack-infra | 09:30 | |
| *** bada_ has quit IRC | 09:30 | |
| *** fbo_away is now known as fbo | 09:31 | |
| *** rossella_ has joined #openstack-infra | 09:34 | |
| *** rossella__ has joined #openstack-infra | 09:34 | |
| *** rossella__ has quit IRC | 09:34 | |
| *** rossella_s has quit IRC | 09:34 | |
| *** rossella_ has quit IRC | 09:34 | |
| *** rossella_s has joined #openstack-infra | 09:35 | |
| *** Daisy_ has quit IRC | 09:35 | |
| *** rpodolyaka has quit IRC | 09:39 | |
| *** gokrokve has joined #openstack-infra | 09:39 | |
| *** jlibosva has joined #openstack-infra | 09:41 | |
| *** mattoliverau has joined #openstack-infra | 09:42 | |
| *** gokrokve has quit IRC | 09:44 | |
| *** pbelanyi has joined #openstack-infra | 09:56 | |
| *** andreaf has joined #openstack-infra | 09:56 | |
| *** pblaho has quit IRC | 10:00 | |
| *** jp_at_hp has joined #openstack-infra | 10:02 | |
| *** sarob_ has quit IRC | 10:04 | |
| *** pblaho has joined #openstack-infra | 10:06 | |
| *** CaptTofu has joined #openstack-infra | 10:08 | |
| *** CaptTofu has quit IRC | 10:13 | |
| *** johnthetubaguy has joined #openstack-infra | 10:18 | |
| *** johnthetubaguy1 has joined #openstack-infra | 10:20 | |
| *** johnthetubaguy has quit IRC | 10:22 | |
| openstackgerrit | Nikola Dipanov proposed a change to openstack-dev/hacking: Make H202 check honor pep8 #noqa comment https://review.openstack.org/79287 | 10:25 |
| *** sarob_ has joined #openstack-infra | 10:30 | |
| *** basha has quit IRC | 10:30 | |
| *** basha has joined #openstack-infra | 10:34 | |
| *** davidhadas has quit IRC | 10:34 | |
| *** davidhadas has joined #openstack-infra | 10:36 | |
| *** rpodolyaka has joined #openstack-infra | 10:36 | |
| *** gokrokve has joined #openstack-infra | 10:39 | |
| *** markmc has joined #openstack-infra | 10:41 | |
| *** dcramer__ has quit IRC | 10:42 | |
| *** gokrokve has quit IRC | 10:44 | |
| *** rpodolyaka has quit IRC | 10:45 | |
| *** rcarrillocruz1 has joined #openstack-infra | 10:47 | |
| *** zehicle has quit IRC | 10:47 | |
| *** rcarrillocruz has quit IRC | 10:49 | |
| *** CaptTofu has joined #openstack-infra | 10:56 | |
| kashyap | clarkb, pleia2, Heya, when you're around - re: Fedora on Gate: just checking if you here have found any other blocking issues from a Fedora distro perspective. I see that prep scripts in config git repo already seem to have some Fedora support | 10:58 |
| kashyap | (s/if you here have/here, if you have) | 10:58 |
| openstackgerrit | Nikola Dipanov proposed a change to openstack-dev/hacking: Make H202 check honor pep8 #noqa comment https://review.openstack.org/79287 | 11:03 |
| *** sarob_ has quit IRC | 11:04 | |
| *** saju_m has quit IRC | 11:05 | |
| *** jpich has quit IRC | 11:12 | |
| openstackgerrit | Nikola Dipanov proposed a change to openstack-dev/hacking: Make H202 check honor pep8 #noqa comment https://review.openstack.org/79287 | 11:17 |
| *** dkranz has quit IRC | 11:19 | |
| *** jpich has joined #openstack-infra | 11:19 | |
| *** eharney has quit IRC | 11:20 | |
| *** andreaf has quit IRC | 11:21 | |
| *** yaguang has quit IRC | 11:23 | |
| *** hashar has quit IRC | 11:28 | |
| *** basha has quit IRC | 11:29 | |
| *** basha has joined #openstack-infra | 11:30 | |
| *** sarob_ has joined #openstack-infra | 11:30 | |
| *** CaptTofu has quit IRC | 11:36 | |
| *** lcestari has joined #openstack-infra | 11:39 | |
| *** gokrokve has joined #openstack-infra | 11:39 | |
| *** alexpilotti has joined #openstack-infra | 11:40 | |
| *** gokrokve_ has joined #openstack-infra | 11:41 | |
| *** rpodolyaka has joined #openstack-infra | 11:42 | |
| *** basha has quit IRC | 11:42 | |
| *** gokrokve has quit IRC | 11:44 | |
| *** ArxCruz has joined #openstack-infra | 11:44 | |
| *** gokrokve_ has quit IRC | 11:45 | |
| *** andreaf has joined #openstack-infra | 11:48 | |
| *** CaptTofu has joined #openstack-infra | 11:51 | |
| *** rpodolyaka has quit IRC | 11:52 | |
| *** yamahata has quit IRC | 11:54 | |
| *** jgallard has quit IRC | 11:56 | |
| *** mburned_out is now known as mburned | 11:59 | |
| *** jcoufal has quit IRC | 12:02 | |
| *** sarob_ has quit IRC | 12:02 | |
| *** jcoufal has joined #openstack-infra | 12:02 | |
| *** aysyd has joined #openstack-infra | 12:08 | |
| *** matsuhashi has quit IRC | 12:11 | |
| *** hashar has joined #openstack-infra | 12:11 | |
| *** matsuhashi has joined #openstack-infra | 12:13 | |
| *** ArxCruz has quit IRC | 12:13 | |
| *** dkranz has joined #openstack-infra | 12:13 | |
| *** lcostantino has joined #openstack-infra | 12:19 | |
| *** talluri has quit IRC | 12:24 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: cloner to easily clone dependent repositories https://review.openstack.org/70373 | 12:27 |
| *** david-lyle has quit IRC | 12:28 | |
| *** mwagner_lap has quit IRC | 12:28 | |
| *** sarob_ has joined #openstack-infra | 12:30 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: update query to handle additional dns issues https://review.openstack.org/79301 | 12:30 |
| *** ArxCruz has joined #openstack-infra | 12:33 | |
| *** mriedem has joined #openstack-infra | 12:34 | |
| *** eharney has joined #openstack-infra | 12:38 | |
| *** gokrokve has joined #openstack-infra | 12:39 | |
| *** gokrokve has quit IRC | 12:44 | |
| *** dims_ has quit IRC | 12:44 | |
| *** dcramer__ has joined #openstack-infra | 12:44 | |
| *** bada_ has joined #openstack-infra | 12:45 | |
| *** bada__ has quit IRC | 12:46 | |
| sdague | mriedem: so I know you're pretty big on it, however given that ER changes include the bug# as the filename, you always want it in the commit message as well? | 12:46 |
| sdague | I always figured there was enough context there | 12:46 |
| mriedem | sdague: it was more for when i'm looking through email or git history later | 12:47 |
| sdague | ok, that's fair | 12:47 |
| mriedem | plus i can't gerrit link to the bug report from the filename and i'm lazy | 12:47 |
| fungi | i've come to realize most contributors don't follow the old gnu changelog format of including the names of files and functions being modified in their commit messages for added context | 12:48 |
| *** bada has joined #openstack-infra | 12:48 | |
| sdague | mriedem: :) | 12:48 |
| *** dkliban has quit IRC | 12:48 | |
| sdague | fungi: well, the old gnu changelog format also predates any reasonable version control tools | 12:48 |
| sdague | SCCS for the win... or lose | 12:49 |
| mriedem | sdague: i was more -1 for leaving git.openstack.org in the query than i was the commit title | 12:49 |
| fungi | very true. it's still nice when looking through git log without -p | 12:49 |
| sdague | mriedem: oh, so removing git.openstack.org was more important to me | 12:49 |
| sdague | because I actually think that we're going to catch a few more things in this net over time | 12:49 |
| mriedem | sdague: so this is just one of those infra bugs that we'll keep open for tracking forever? | 12:50 |
| sdague | mriedem: maybe. Honestly, I don't understand why DNS is failing in rax | 12:50 |
| sdague | DNS shouldn't fail | 12:51 |
| *** pdmars has joined #openstack-infra | 12:51 | |
| *** bada_ has quit IRC | 12:51 | |
| fungi | especially since the records being looked up are hosted *in* rax dns services | 12:51 |
| *** smarcet has joined #openstack-infra | 12:51 | |
| *** rpodolyaka has joined #openstack-infra | 12:51 | |
| mriedem | sdague: ah, related-bug: #1286963 would have linked it to the bug report too | 12:52 |
| mriedem | since it's marked as infra bug | 12:52 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: update query to match additional instances of dns issues https://review.openstack.org/79301 | 12:52 |
| sdague | mriedem: yep | 12:52 |
| *** CaptTofu has quit IRC | 12:52 | |
| fungi | the e-mail thread we had going with rax support over blacklisting some of our servers on their resolvers for making too many queries kinda petered out, but i still anticipate we'll end up rolling out caching local resolvers onto all our systems as a performance improvement anyway. test jobs could then start out with a retry loop to prime that cache with things we know will get looked up over the | 12:53 |
| fungi | course of a job | 12:53 |
| *** CaptTofu has joined #openstack-infra | 12:53 | |
| fungi | which should mostly mitigate that failure mode, except in the face of prolonged resolver outages in the provider | 12:53 |
| sdague | so this isn't single use nodes only | 12:54 |
| sdague | sorry, this also affects single use nodes | 12:54 |
| sdague | which confuses me on the blacklisting | 12:54 |
| sdague | unless we are basically hitting a blacklist on an IP they assigned to someone else | 12:54 |
| sdague | which... would be really bad | 12:54 |
| fungi | well, i don't know if we have lookup failures from time-to-time on our long-lived infra servers, but i *do* know that review, lists and ask were all blacklisted a few weeks ago for doing too many lookups | 12:54 |
| mriedem | sdague: so this only shows up on rax nodes, and the bug report is targeted at rax | 12:54 |
| *** bada has quit IRC | 12:54 | |
| mriedem | sdague: what do you think about limiting the query to rax nodes only? | 12:55 |
| sdague | mriedem: yes | 12:55 |
| *** lukego has joined #openstack-infra | 12:55 | |
| *** jeckersb_gone is now known as jeckersb | 12:55 | |
| sdague | mriedem: honestly, because we don't have cloud as a field, I expect limitting on that via regex is probably more expensive in ER | 12:55 |
| *** dims_ has joined #openstack-infra | 12:56 | |
| mriedem | cloud? i meant build_node | 12:56 |
| sdague | mriedem: right | 12:56 |
| sdague | but you are going to have to do that with a * | 12:56 |
| fungi | the great part was that they only actively informed us about having blacklisted one of those three servers (after the fact, but still better than the other two which they just silently started blocking) | 12:56 |
| mriedem | yeah | 12:56 |
| mriedem | sdague: yeah to * | 12:56 |
| sdague | so, based on everythign I know, doing that means you gave away any performance you thought you were gaining | 12:56 |
| *** Daisy_ has joined #openstack-infra | 12:56 | |
| *** Daisy_ has quit IRC | 12:57 | |
| sdague | and just putting more load on ES | 12:57 |
| mriedem | sdague: we're doing * on filename in several queries already right? | 12:57 |
| mriedem | how is that different? | 12:57 |
| mriedem | logs*screen-n-cpu.txt | 12:57 |
| sdague | we are, which I actually think we need to revisit | 12:57 |
| mriedem | for example | 12:57 |
| *** Daisy_ has joined #openstack-infra | 12:57 | |
| sdague | because I expect that's costing us | 12:57 |
| *** CaptTofu has quit IRC | 12:58 | |
| mriedem | :( | 12:58 |
| mriedem | well it's not even 8am yet and i'm on 2 the fence about 2 patches already | 12:59 |
| mriedem | gdi | 12:59 |
| sdague | :) | 12:59 |
| mriedem | and i can't type | 12:59 |
| *** rpodolyaka has quit IRC | 13:00 | |
| *** bada has joined #openstack-infra | 13:00 | |
| *** mfer has joined #openstack-infra | 13:04 | |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/config: Add experimental keystonev3 pipeline https://review.openstack.org/79314 | 13:04 |
| *** sarob_ has quit IRC | 13:04 | |
| andreaf | sdague: ping | 13:05 |
| sdague | andreaf: what's up? | 13:05 |
| *** mbacchi has joined #openstack-infra | 13:05 | |
| *** zns has joined #openstack-infra | 13:06 | |
| andreaf | sdague: I'm trying to add an experimental keystone v3 check - could you have a look at https://review.openstack.org/#/c/79212 and https://review.openstack.org/#/c/79314/ to see if it makes sense? | 13:06 |
| sdague | andreaf: sure, though probably later on today, getting my bearings for the morning still | 13:07 |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/config: Add experimental keystonev3 pipeline https://review.openstack.org/79314 | 13:07 |
| andreaf | sdague: sure, thx | 13:07 |
| *** bada has quit IRC | 13:08 | |
| sdague | fungi: https://review.openstack.org/#/c/78698/ - how you feel about putting that through? | 13:10 |
| sdague | which would finally lock down heat | 13:10 |
| *** jgallard has joined #openstack-infra | 13:10 | |
| * fungi brings the heat | 13:11 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: remove python33 from marconi gate https://review.openstack.org/79316 | 13:12 |
| *** rfolco has joined #openstack-infra | 13:13 | |
| *** AJaeger has joined #openstack-infra | 13:15 | |
| *** zns has quit IRC | 13:15 | |
| AJaeger | Zuul shows me two long running openstack-manuals post jobs at: http://status.openstack.org/zuul/ | 13:16 |
| AJaeger | running for 22 hours looks broken ;( | 13:16 |
| fungi | AJaeger: i noticed those as well. i went digging in zuul's logs and the only theory i have so far is that the gearman messages about them never made it out over the network. at least i didn't see any acknowledgement in the debug log that a worker started those, though i could be overlooking it | 13:19 |
| *** dcramer__ has quit IRC | 13:19 | |
| AJaeger | fungi, thanks | 13:19 |
| fungi | corvus: when you're awake, ^ may be of interest | 13:19 |
| AJaeger | Feel free to kill the jobs if you like. | 13:19 |
| AJaeger | They only publish to the website and any later run will publish again... | 13:19 |
| *** CaptTofu has joined #openstack-infra | 13:20 | |
| openstackgerrit | A change was merged to openstack-infra/config: enable heat-slow in the integrated-gate https://review.openstack.org/78698 | 13:20 |
| AJaeger | fungi, one more question: The translation job suddenly fails: https://jenkins02.openstack.org/job/openstack-install-deploy-guide-ja/47/console - it misses gettext on the server | 13:20 |
| AJaeger | Did something change recently? | 13:20 |
| AJaeger | msgfmt is the tool missing | 13:21 |
| fungi | AJaeger: rackspace did supposedly do their base image updates last week, so maybe we don't install that package and it used to be preinstalled on images but isn't any longer? i'll take a look | 13:22 |
| AJaeger | Compare it with this run (the other hanging job): https://jenkins01.openstack.org/job/openstack-install-deploy-guide-ja/34/consoleText | 13:23 |
| AJaeger | There it worked... | 13:23 |
| AJaeger | fungi, thanks for looking into it! | 13:23 |
| *** bada has joined #openstack-infra | 13:26 | |
| *** rustlebee is now known as russellb | 13:26 | |
| *** julim has joined #openstack-infra | 13:26 | |
| *** hashar has quit IRC | 13:26 | |
| *** hashar has joined #openstack-infra | 13:27 | |
| sdague | fungi / mriedem - we used to have an ER query for this, right? - http://logs.openstack.org/79/71479/18/gate/gate-tempest-dsvm-full/5f3fcb4/console.html.gz | 13:28 |
| mriedem | sdague: i want to say yes but there have been a lot of jenkins traces like that | 13:29 |
| mriedem | with subtle differences | 13:29 |
| sdague | yeh, so this is new - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkNhdXNlZCBieTogamF2YS5pby5JT0V4Y2VwdGlvbjogU1NIIGNoYW5uZWwgaXMgY2xvc2VkXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTQ0NTgxMzIwMzZ9 | 13:29 |
| sdague | just last 24 hrs | 13:29 |
| fungi | fun. any correlation to a specific jenkins master? | 13:30 |
| sdague | no | 13:30 |
| *** sarob_ has joined #openstack-infra | 13:30 | |
| sdague | but they are all rax-iad | 13:30 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkNhdXNlZCBieTogamF2YS5pby5JT0V4Y2VwdGlvbjogU1NIIGNoYW5uZWwgaXMgY2xvc2VkXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTQ0NTgxMzIwMzYsIm1vZGUiOiJzY29yZSIsImFuYWx5emVfZmllbGQiOiJidWlsZF9ub2RlIn0= | 13:31 |
| fungi | that first one was jenkins05 but it seems responsive enough | 13:31 |
| sdague | it's over at least 5 jenkins masters | 13:31 |
| fungi | were they tightly-grouped timewise? maybe maintenance in iad... i'll look at teh rax status page | 13:31 |
| sdague | so it's spread out over a 18 hrs window | 13:31 |
| fungi | too fun | 13:32 |
| *** matsuhashi has quit IRC | 13:32 | |
| fungi | nothing immediately concerning on their status page | 13:32 |
| AJaeger | fungi, regarding msgfmt - the failing one is indeed a bare-precise-rax-iad-2480603, the pass was on bare-precise-hpcloud-az2-2477439. And another fail on bare-precise-rax-iad-2495899 | 13:33 |
| fungi | it does worry me that they very recently announced a "priority networking maintenance on the 12th" for not-iad (dfw/ord). like, "oops, this broke and we ninja-fixed it in iad, now let's schedule an emergency window to do it everywhere else" | 13:34 |
| fungi | AJaeger: i confirmed that we msgfmt is provided by the gettext package and we don't explicitly install it in our puppet manifest... holding a couple of nodes in different providers now to compare | 13:36 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: jenkins ssh explosion query https://review.openstack.org/79322 | 13:36 |
| *** nosnos has quit IRC | 13:38 | |
| *** bknudson has joined #openstack-infra | 13:38 | |
| *** gokrokve has joined #openstack-infra | 13:39 | |
| *** zns has joined #openstack-infra | 13:40 | |
| fungi | AJaeger: confirmed. gettext is not installed on bare-precise nodes in rax-dfw or rax-iad, but is installed on nodes in hpcloud-az2 | 13:40 |
| AJaeger | fungi, which file do I need to touch to install it - or will you fix it? | 13:41 |
| fungi | AJaeger: i'll whip up a quick patch to our puppet manifest to add it to all slaves | 13:41 |
| AJaeger | Thanks a lot, fungi! | 13:41 |
| fungi | AJaeger: though it won't take effect for about another 12 hours when our daily image rebuilds happen | 13:41 |
| AJaeger | no problem for now. | 13:42 |
| *** dprince has joined #openstack-infra | 13:43 | |
| *** yamahata has joined #openstack-infra | 13:44 | |
| *** gokrokve has quit IRC | 13:44 | |
| *** smarcet has quit IRC | 13:46 | |
| *** jeckersb is now known as jeckersb_gone | 13:47 | |
| *** jcoufal has quit IRC | 13:48 | |
| *** dkliban has joined #openstack-infra | 13:50 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Explicitly install gettext on job workers https://review.openstack.org/79323 | 13:51 |
| *** jgrimm has quit IRC | 13:51 | |
| *** jeckersb_gone is now known as jeckersb | 13:52 | |
| *** prad has joined #openstack-infra | 13:53 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: mock out lp in tests that don't really need it https://review.openstack.org/79325 | 13:53 |
| *** prad has quit IRC | 13:53 | |
| *** dcramer__ has joined #openstack-infra | 13:54 | |
| fungi | pleia2: i added you as a requested reviewer on https://review.openstack.org/79323 in case you want to make sure the gettext package name (or the package providing msgfmt) isn't different in some funky way on latest fedora | 13:54 |
| *** matrohon has joined #openstack-infra | 13:56 | |
| *** ociuhandu has quit IRC | 13:56 | |
| *** mgagne has joined #openstack-infra | 13:57 | |
| andreaf | I'm trying to use tox with zuul, to create a venv - but I consistently get an error on voluptuous>=0.7 - is there a known fix to this? | 13:57 |
| fungi | andreaf: what error exactly? | 13:58 |
| AJaeger | fungi, thanks for the patch. | 13:58 |
| *** yamahata has quit IRC | 13:58 | |
| andreaf | fungi: cannot find the package in the pypi repo | 13:59 |
| *** mgagne has quit IRC | 13:59 | |
| andreaf | Downloading/unpacking voluptuous>=0.7 (from -r /git/repos/openstack-infra/zuul/requirements.txt (line 15)) | 13:59 |
| andreaf | http://pypi.openstack.org/openstack/voluptuous/ uses an insecure transport scheme (http). Consider using https if pypi.openstack.org has it available | 13:59 |
| andreaf | Could not find a version that satisfies the requirement voluptuous>=0.7 (from -r /git/repos/openstack-infra/zuul/requirements.txt (line 15)) (from versions: 0.6.1, 0.6) | 13:59 |
| andreaf | Cleaning up... | 13:59 |
| andreaf | No distributions matching the version for voluptuous>=0.7 (from -r /git/repos/openstack-infra/zuul/requirements.txt (line 15)) | 13:59 |
| andreaf | Storing debug log for failure in /home/af/.pip/pip.log | 13:59 |
| andreaf | fungi: sorry about the multiline copy / paste | 13:59 |
| fungi | andreaf: why is it pulling from pypi.openstack.org? | 14:00 |
| *** mgagne has joined #openstack-infra | 14:00 | |
| *** jcoufal has joined #openstack-infra | 14:00 | |
| *** Daisy_ has quit IRC | 14:00 | |
| AJaeger | andreaf, http://pypi.openstack.org/openstack/voluptuous/ has 0.6.1 as latest version | 14:00 |
| AJaeger | pypi.python has 0.8.4 | 14:00 |
| fungi | right. zuul isn't an openstack project, it's an openstack-infra project, so its requirements are not necessarily satisfied by the openstack pypi mirror | 14:01 |
| andreaf | fungi: well usually it's more stable than pypi.python | 14:01 |
| andreaf | fungi: go it | 14:01 |
| andreaf | fungi: thanks | 14:01 |
| fungi | andreaf: pypi.openstack.org is not a full mirror of pypi. also it may seem more stable than pypi.python.org but it's a small virtual machine shared by a bunch of other stuff and is a single point of failure | 14:01 |
| *** beekneemech is now known as bnemec | 14:02 | |
| *** jooools has quit IRC | 14:02 | |
| fungi | while pypi.python.org is behind a cdn and also offers https with a server certificate pip can validate | 14:02 |
| andreaf | fungi: I've been trying to setup my own mirror but it keeps failing | 14:02 |
| fungi | andreaf: have you tried bandersnatch? | 14:02 |
| AJaeger | fungi, one more openstack-manuals issue: Our bugs are not marked as "Fix released" anymore - could you review https://review.openstack.org/#/c/79181/ , please? | 14:02 |
| * AJaeger does not want to manually update 30+ bugs again ;( | 14:03 | |
| fungi | andreaf: https://pypi.python.org/pypi/bandersnatch | 14:03 |
| *** mwagner_lap has joined #openstack-infra | 14:03 | |
| andreaf | fungi: thanks I will try that | 14:03 |
| *** jooools has joined #openstack-infra | 14:03 | |
| fungi | andreaf: i haven't used it, but seems to be the most popular tool for creating and updating a full pypi mirror these days | 14:03 |
| *** sarob_ has quit IRC | 14:03 | |
| *** smarcet has joined #openstack-infra | 14:03 | |
| *** yamahata has joined #openstack-infra | 14:04 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: mock out lp in tests that don't really need it https://review.openstack.org/79325 | 14:04 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Make H202 check honor pep8 #noqa comment https://review.openstack.org/79287 | 14:05 |
| *** malini_afk is now known as malini | 14:06 | |
| fungi | SergeyLukjanov: mordred: regarding https://review.openstack.org/79181 did we miss migrating the rest of the project options to review.projects.yaml in https://review.openstack.org/62190 before https://review.openstack.org/74309 removed them? | 14:07 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: update query to match additional instances of dns issues https://review.openstack.org/79301 | 14:08 |
| openstackgerrit | Randy Coulman proposed a change to openstack-infra/jenkins-job-builder: Fix typo in ircbot's default matrix-notifier. https://review.openstack.org/79328 | 14:08 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: jenkins ssh explosion query https://review.openstack.org/79322 | 14:08 |
| SergeyLukjanov | fungi, morning | 14:08 |
| *** yamahata has quit IRC | 14:08 | |
| SergeyLukjanov | fungi, looking now | 14:08 |
| fungi | AJaeger: i don't see where were those set previously in openstack-infra/jeepyb:jeepyb/projects.py and am trying to figure out where we accidentally removed them. when were they previously working? | 14:10 |
| AJaeger | Fungi hey worked for sure last year. | 14:11 |
| AJaeger | https://review.openstack.org/#/c/74309/3/jeepyb/projects.py contains "openstack-manuals" and "api-site" | 14:11 |
| AJaeger | The others in 79181 are new. | 14:11 |
| SergeyLukjanov | AJaeger, fungi, yup | 14:12 |
| fungi | AJaeger: ahh, yes looks like it | 14:13 |
| fungi | AJaeger: sorry about that! | 14:13 |
| AJaeger | it seems you migrated everything with the exception of these two ;( | 14:13 |
| AJaeger | fungi, no big deal. | 14:13 |
| fungi | AJaeger: i agree, that shouldn't have happened. sorry i missed it when reviewing | 14:14 |
| SergeyLukjanov | fungi, yup, we'll missed this :( | 14:14 |
| AJaeger | fungi, let's persuade SergeyLukjanov to review and approve ;) | 14:14 |
| AJaeger | SergeyLukjanov, please :) | 14:14 |
| *** dkliban has quit IRC | 14:15 | |
| *** yamahata has joined #openstack-infra | 14:15 | |
| AJaeger | there are more missing: Meetbot, nose-html-output,... | 14:15 |
| SergeyLukjanov | AJaeger, done, sorry about that | 14:15 |
| *** freyes has quit IRC | 14:16 | |
| SergeyLukjanov | AJaeger, oh, thanks for catching, I'll check the lists right now | 14:16 |
| AJaeger | SergeyLukjanov, I'll check myself and send a patch - you can review ;) | 14:16 |
| *** smarcet has quit IRC | 14:16 | |
| SergeyLukjanov | AJaeger, ok, thank you! | 14:16 |
| openstackgerrit | Ziad Sawalha proposed a change to openstack/requirements: Add PEP-257 Testing Tools https://review.openstack.org/79054 | 14:17 |
| *** ociuhandu has joined #openstack-infra | 14:18 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Add direct-release for meetbot and nose-html-output https://review.openstack.org/79330 | 14:19 |
| *** yamahata has quit IRC | 14:20 | |
| AJaeger | SergeyLukjanov, fungi: You might want to doublecheck the list, there are some further projects that IMO could use direct-release. | 14:20 |
| openstackgerrit | Randy Coulman proposed a change to openstack-infra/jenkins-job-builder: Change default Reply-To header to $DEFAULT_REPLYTO. https://review.openstack.org/79331 | 14:21 |
| *** bada_ has joined #openstack-infra | 14:22 | |
| AJaeger | Might be that those do not get bugreports ;) I stumbled over jenkins-job-builder and nodepool. | 14:22 |
| *** bada has quit IRC | 14:22 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add direct-release back for manual related repositories https://review.openstack.org/79181 | 14:24 |
| fungi | AJaeger: those both have releases | 14:24 |
| *** bada__ has joined #openstack-infra | 14:25 | |
| AJaeger | fungi, didn't know - thanks. | 14:25 |
| fungi | AJaeger: direct release is for projects which don't tag releases and want their bugs to be automatically set to fix-released when the fixes merge to the repository | 14:25 |
| AJaeger | fungi, I know now what it does ;) | 14:25 |
| *** sweston has quit IRC | 14:25 | |
| fungi | ahh, okay | 14:25 |
| *** yamahata has joined #openstack-infra | 14:26 | |
| fungi | this is one of the things which should get much better in storyboard | 14:26 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: mock out lp in tests that don't really need it https://review.openstack.org/79325 | 14:26 |
| *** odyssey4me has joined #openstack-infra | 14:26 | |
| *** bada_ has quit IRC | 14:27 | |
| *** esker has quit IRC | 14:27 | |
| *** homeless has joined #openstack-infra | 14:28 | |
| AJaeger | fungi, my comment was that I did not know those had releases. | 14:29 |
| fungi | AJaeger: right. we actually tag and release a lot of openstack-infra tools to pypi, more than most people realize i think | 14:29 |
| AJaeger | at least more than I realize ;) | 14:30 |
| AJaeger | All my infra problems are solved for today ;) - back to other work. Thanks, fungi and SergeyLukjanov ! | 14:30 |
| *** wchrisj has joined #openstack-infra | 14:30 | |
| *** sarob_ has joined #openstack-infra | 14:30 | |
| SergeyLukjanov | AJaeger, thanks for catching missed direct-release project, /me comparing the lists now | 14:30 |
| fungi | nodepool technically hasn't had a release yet, but i suspect it's getting close to the point where we should have one. there are plenty of people starting to use it now | 14:31 |
| *** smarcet has joined #openstack-infra | 14:31 | |
| fungi | jjb on the other hand has released 6 times so far | 14:31 |
| fungi | and is due for another | 14:32 |
| *** CaptTofu has quit IRC | 14:32 | |
| *** david-lyle has joined #openstack-infra | 14:33 | |
| *** CaptTofu has joined #openstack-infra | 14:33 | |
| *** yamahata has quit IRC | 14:33 | |
| SergeyLukjanov | fungi, can I ask you to take a look on savanna renaming changes? | 14:34 |
| fungi | SergeyLukjanov: sure, i can take a break and go over them | 14:34 |
| SergeyLukjanov | fungi, awesome, thank you! | 14:34 |
| fungi | SergeyLukjanov: it's just the three of these which aren't the patch to savanna itself, yes? https://review.openstack.org/#/q/status:open+branch:master+topic:savanna-sahara,n,z | 14:36 |
| *** yamahata has joined #openstack-infra | 14:36 | |
| openstackgerrit | Randy Coulman proposed a change to openstack-infra/jenkins-job-builder: Support the new Always trigger type. https://review.openstack.org/79332 | 14:36 |
| SergeyLukjanov | fungi, https://review.openstack.org/#/c/78960/ https://review.openstack.org/#/c/79097/ and after repo rename https://review.openstack.org/#/c/79099/ | 14:36 |
| *** rwsu has joined #openstack-infra | 14:36 | |
| SergeyLukjanov | fungi, exactly! | 14:36 |
| SergeyLukjanov | fungi 79099 to be landed after repo rename | 14:36 |
| fungi | got it | 14:37 |
| *** mkerrin has quit IRC | 14:38 | |
| *** CaptTofu has quit IRC | 14:38 | |
| fungi | we should pick a time when there aren't meetings going on so we can merge 78960 without causing disruption | 14:39 |
| *** gokrokve has joined #openstack-infra | 14:39 | |
| *** wenlock has joined #openstack-infra | 14:40 | |
| *** unicell has quit IRC | 14:40 | |
| zns | sdague: you commented in gerrit that flake8_docstring seems MIT licensed. Did you see that somewhere specifically or did you read the license and conclude it is an MIT license? | 14:41 |
| sdague | I read the license | 14:41 |
| sdague | https://bitbucket.org/icordasc/flake8-docstrings/src/4f657f86b5dae935e26df19248d33da196abbefc/LICENSE?at=default | 14:41 |
| sdague | http://opensource.org/licenses/MIT | 14:41 |
| *** yamahata__ has joined #openstack-infra | 14:41 | |
| *** gokrokve_ has joined #openstack-infra | 14:41 | |
| *** yamahata has quit IRC | 14:42 | |
| *** yamahata__ has quit IRC | 14:43 | |
| *** gokrokve has quit IRC | 14:43 | |
| zns | Cool. Thanks. I'll update the comments. | 14:44 |
| *** AaronGr has quit IRC | 14:44 | |
| fungi | SergeyLukjanov: you probably want to make sure openstack-sahara finds its way into https://review.openstack.org/#/c/78483/6/modules/openstack_project/files/accessbot/channels.yaml as well | 14:44 |
| *** yamahata has joined #openstack-infra | 14:45 | |
| *** yamahata has quit IRC | 14:48 | |
| *** dolphm has quit IRC | 14:48 | |
| sdague | yay, sqla 0.9 now used in the gate - https://review.openstack.org/#/c/79129/ | 14:48 |
| *** davidlenwell has quit IRC | 14:48 | |
| *** juice has quit IRC | 14:49 | |
| *** zns has quit IRC | 14:49 | |
| SergeyLukjanov | fungi, oh, thanks for the tip, this CR was in my backlog, I'll add commit after that | 14:49 |
| openstackgerrit | Ziad Sawalha proposed a change to openstack/requirements: Add PEP-257 Testing Tools https://review.openstack.org/79054 | 14:49 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: cloner to easily clone dependent repositories https://review.openstack.org/70373 | 14:49 |
| openstackgerrit | A change was merged to openstack/requirements: Switch to SQLAlchemy 0.9 https://review.openstack.org/79129 | 14:49 |
| fungi | sdague: expat license, i think, technically speaking | 14:49 |
| *** packet has joined #openstack-infra | 14:49 | |
| *** AaronGreen has joined #openstack-infra | 14:49 | |
| *** BlartVersenwald is now known as NobodyCam | 14:49 | |
| *** dolphm_ has joined #openstack-infra | 14:49 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: mock out lp in tests that don't really need it https://review.openstack.org/79325 | 14:49 |
| *** davidlenwell_ has joined #openstack-infra | 14:49 | |
| openstackgerrit | Randy Coulman proposed a change to openstack-infra/jenkins-job-builder: Make email-ext's AlwaysTrigger the default. https://review.openstack.org/79337 | 14:50 |
| *** dolphm_ is now known as dolphm | 14:50 | |
| *** yamahata has joined #openstack-infra | 14:50 | |
| clarkb | morning, I am not actually here yet | 14:51 |
| sdague | fungi: really? I did a word compare against MIT and it seemed to match | 14:51 |
| *** zns has joined #openstack-infra | 14:51 | |
| clarkb | kashyap: pleia2 has a change up to enable fedora in the gate for tripleo. I don't think there are any more blockers to that than code review | 14:51 |
| *** sarob_ has quit IRC | 14:51 | |
| *** juice has joined #openstack-infra | 14:51 | |
| fungi | sdague: "mit license" is vague because there are multiple free software licenses which originated at mit. the expat license is one of them | 14:51 |
| SergeyLukjanov | clarkb, howdy | 14:51 |
| zns | Does anyone have any objection to adding https://pypi.python.org/pypi/freezegun to the test requirements? Or are there alternatives in use today? | 14:51 |
| fungi | sdague: the x11 license is an example of another | 14:51 |
| *** gokrokve_ has quit IRC | 14:51 | |
| *** jooools has quit IRC | 14:51 | |
| *** hdd_ has joined #openstack-infra | 14:52 | |
| sdague | fungi: sure, I was just referencing this - http://opensource.org/licenses/MIT as source of truth :) | 14:52 |
| fungi | sdague: but i am mostly being a pedant. these days people are less familiar with other mit licenses | 14:52 |
| *** jooools has joined #openstack-infra | 14:52 | |
| fungi | sdague: it's only slightly less vague than saying something is released under "the bsd license" | 14:52 |
| *** freyes has joined #openstack-infra | 14:53 | |
| fungi | osi seems to be a little less concerned about specificity there than the fsf is | 14:54 |
| kashyap | clarkb, I see, does that imply it'll be in nodepool soon? | 14:57 |
| clarkb | kashyap: once the change merges yes | 14:57 |
| *** rpodolyaka has joined #openstack-infra | 14:57 | |
| fungi | ha, so it turns out one of the stray suggestions to send mail to openstack-ci-admins@lists.launchpad.net is on a stale japanese translatiojn of the gerritjenkinsgithub article | 14:58 |
| mordred | zns: I believe there is a mock or fixture that does that somewhere in the nova source tree | 14:58 |
| *** julim_ has joined #openstack-infra | 14:58 | |
| *** thedodd has joined #openstack-infra | 14:58 | |
| *** zehicle_at_dell has joined #openstack-infra | 14:59 | |
| mordred | zns: but I believe it's handwritten ... I'd be fine with adding it - but only if we can come up with a plan to migrate the existing code which needs that functionality to the new library | 14:59 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add logging to the new #openstack-sahara channel https://review.openstack.org/78960 | 14:59 |
| zns | mordred: OK. Thanks. I'll look into that. Probably needs a blueprint to nova too. | 15:00 |
| *** blamar has joined #openstack-infra | 15:00 | |
| *** julim has quit IRC | 15:01 | |
| *** blamar has quit IRC | 15:02 | |
| *** blamar has joined #openstack-infra | 15:02 | |
| kashyap | clarkb, Okay, thank you for the details. | 15:02 |
| SergeyLukjanov | sdague, could you please take a look on savanna renaming change from the pov of keeping d-g running? | 15:02 |
| *** jgrimm has joined #openstack-infra | 15:03 | |
| kashyap | clarkb, I personally am not testing TripleO, currently mostly focusing on Libvirt/KVM/QEMU (some areas of QEMU) related issues in OpenStack | 15:03 |
| *** CaptTofu has joined #openstack-infra | 15:05 | |
| *** corvus is now known as jeblair | 15:07 | |
| *** gokrokve has joined #openstack-infra | 15:08 | |
| *** dizquierdo has joined #openstack-infra | 15:08 | |
| sdague | SergeyLukjanov: url? | 15:08 |
| *** rpodolyaka has quit IRC | 15:09 | |
| SergeyLukjanov | sdague, thx! | 15:09 |
| SergeyLukjanov | sdague, https://review.openstack.org/#/c/79097/ for repo renaming | 15:09 |
| SergeyLukjanov | sdague, and https://review.openstack.org/#/c/79099/ to allow new repos be cloned https://review.openstack.org/#/c/79099/ | 15:09 |
| *** rpodolyaka has joined #openstack-infra | 15:10 | |
| sdague | SergeyLukjanov: hmmm... does that actually do the things you expect? | 15:10 |
| jeblair | sdague: about http://logs.openstack.org/79/71479/18/gate/gate-tempest-dsvm-full/5f3fcb4/console.html.gz | 15:10 |
| sdague | I thought with the zuul overwrite on references | 15:10 |
| jeblair | sdague: that looks like a zuul-aborted job | 15:10 |
| sdague | jeblair: ok, well if so, it's a completely new signature for one | 15:10 |
| *** esker has joined #openstack-infra | 15:11 | |
| SergeyLukjanov | sdague, hm | 15:11 |
| jeblair | sdague: yeah, i believe it varies based on when the signal happens | 15:11 |
| sdague | jeblair: then why would it only happen on rax-iad nodes? | 15:11 |
| sdague | just luck? | 15:12 |
| *** mgagne has quit IRC | 15:12 | |
| jeblair | sdague: i'm guessing? INFO: ---- devstack-precise-rax-iad-2344307_exec-0 Request to stop AWT: hudson.plugins.gearman.ExecutorWorkerThread@7ae6c454 | 15:12 |
| jeblair | sdague: i believe that's the gearman-worker log for that; i'll double check the zuul log | 15:12 |
| *** hdd_ has quit IRC | 15:13 | |
| *** rpodolyaka has quit IRC | 15:14 | |
| *** BobBall has quit IRC | 15:15 | |
| *** jraim has quit IRC | 15:15 | |
| sdague | jeblair: well the pattern is too suspicious to me to say it's working as normal - http://logstash.openstack.org/#eyJmaWVsZHMiOiBbXSwgInNlYXJjaCI6ICJtZXNzYWdlOlwiQ2F1c2VkIGJ5OiBqYXZhLmlvLklPRXhjZXB0aW9uOiBTU0ggY2hhbm5lbCBpcyBjbG9zZWRcIiBBTkQgZmlsZW5hbWU6Y29uc29sZS5odG1sIiwgInRpbWVmcmFtZSI6ICI2MDQ4MDAiLCAiZ3JhcGhtb2RlIjogImNvdW50IiwgIm9mZnNldCI6IDB9 | 15:16 |
| sdague | because it's almost 100 fails, only in one az | 15:16 |
| sdague | over an 18hr window | 15:16 |
| *** pcrews has joined #openstack-infra | 15:16 | |
| jeblair | sdague: it's not a zuul abort | 15:17 |
| *** jraim has joined #openstack-infra | 15:17 | |
| *** mgagne has joined #openstack-infra | 15:17 | |
| sdague | jeblair: ok, cool | 15:17 |
| jeblair | sdague: yeah, jenkins took it offline because it wasn't responding, then back online, then when it tried to run the job, decided it was actually not responsive afterall | 15:18 |
| jeblair | sdague: (and so took it offline again, which is what that gearman-plugin log msg actually was) | 15:18 |
| jeblair | it seems like fungi was leaning toward dc network fubar; i'd tend to agree | 15:19 |
| fungi | sdague: did you nail down a more specific window of time on that one? curious whether we had evidence it's still ongoing there | 15:20 |
| *** mgagne has quit IRC | 15:21 | |
| sdague | fungi: it's in the logstash query | 15:21 |
| *** rpodolyaka has joined #openstack-infra | 15:21 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: mock out lp in tests that don't really need it https://review.openstack.org/79325 | 15:22 |
| *** mgagne has joined #openstack-infra | 15:23 | |
| *** mgagne has quit IRC | 15:23 | |
| *** mgagne has joined #openstack-infra | 15:23 | |
| *** matrohon has quit IRC | 15:23 | |
| *** unicell has joined #openstack-infra | 15:24 | |
| *** zehicle_at_dell has quit IRC | 15:24 | |
| jeblair | fungi, AJaeger: yeah, i can't track down those missing jobs; at least not without the work that needs to be done to revise gear logging to make it less verbose so we can actually turn it on | 15:24 |
| *** luisg has joined #openstack-infra | 15:25 | |
| jeblair | fungi: also, maybe we should symlink /var/log/zuul to opt | 15:25 |
| *** eharney has quit IRC | 15:25 | |
| fungi | jeblair: not a bad idea at all | 15:25 |
| *** luis_ has joined #openstack-infra | 15:26 | |
| *** markmcclain has joined #openstack-infra | 15:26 | |
| jeblair | AJaeger: so those jobs will probably sit there until we restart zuul, but they won't interfere with anything | 15:26 |
| openstackgerrit | Sylvain Baubeau proposed a change to openstack-infra/jenkins-job-builder: Add workspace scm https://review.openstack.org/79345 | 15:26 |
| *** luis_ has quit IRC | 15:26 | |
| *** sweston has joined #openstack-infra | 15:27 | |
| *** eharney has joined #openstack-infra | 15:29 | |
| *** hashar has quit IRC | 15:29 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gerritbot: Support SSL https://review.openstack.org/79049 | 15:30 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Support SSL https://review.openstack.org/79047 | 15:30 |
| *** sarob_ has joined #openstack-infra | 15:30 | |
| *** zehicle_at_dell has joined #openstack-infra | 15:31 | |
| *** sweston has quit IRC | 15:31 | |
| *** bada_ has joined #openstack-infra | 15:32 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a script to manage IRC perms https://review.openstack.org/78483 | 15:32 |
| SergeyLukjanov | fungi, clarkb, jeblair, mordred, sdague, re project renaming, I've take a look on the quantum renaming wiki page and I see that folks force push devstack update commits, probably the same should be done for savanna | 15:34 |
| SergeyLukjanov | do you have any thoughts on it, please? | 15:34 |
| *** sarob_ has quit IRC | 15:35 | |
| fungi | SergeyLukjanov: that was done because devstack was gating on quantum/neutron, so we couldn't merge a change which fixed them | 15:35 |
| SergeyLukjanov | fungi, savanna job is non-voting on devstack, so, we'll be able to check it and merge | 15:36 |
| fungi | SergeyLukjanov: savanna/sahara gates on devstack, but devstask does not gate on savanna/sahara right? | 15:36 |
| fungi | right, good | 15:36 |
| SergeyLukjanov | fungi, yup | 15:36 |
| jeblair | SergeyLukjanov: is https://review.openstack.org/#/q/status:open++topic:savanna-sahara,n,z the full list of reviews? | 15:36 |
| *** bada__ has quit IRC | 15:36 | |
| fungi | that's part of why we want names vetted and settled on prior to graduation from incubation | 15:36 |
| SergeyLukjanov | jeblair, yup, the two config CRs and d-g | 15:36 |
| fungi | becomes much harder afterward | 15:36 |
| SergeyLukjanov | jeblair, + one pending for devstack | 15:37 |
| SergeyLukjanov | fungi, yup, that's why we're trying to do it right now ;) | 15:37 |
| *** matrohon has joined #openstack-infra | 15:37 | |
| SergeyLukjanov | so, probably, the better approach is to just rename all repos, force push sahara to d-g and land devstack update | 15:38 |
| SergeyLukjanov | than sahara will be unblocked | 15:38 |
| *** cody-somerville has joined #openstack-infra | 15:38 | |
| *** e0ne has joined #openstack-infra | 15:38 | |
| *** sweston has joined #openstack-infra | 15:38 | |
| fungi | SergeyLukjanov: the force-push is because otherwise you'd have to land savanna and python-savannaclient fixes together? | 15:39 |
| SergeyLukjanov | fungi, I mean allow d-g to clone new sahara repos | 15:39 |
| SergeyLukjanov | fungi, I think that we'll land sahara aliases to savanna before repos renaming | 15:39 |
| jeblair | SergeyLukjanov: does python-savannaclient need similar changes? | 15:40 |
| *** sandywalsh has joined #openstack-infra | 15:40 | |
| *** gokrokve has quit IRC | 15:40 | |
| SergeyLukjanov | jeblair, similar to https://review.openstack.org/#/c/79063/? | 15:40 |
| SergeyLukjanov | jeblair, if yes - it's on the go, /me fighting home internet slowness | 15:41 |
| jeblair | SergeyLukjanov: yes | 15:41 |
| jeblair | ok | 15:41 |
| jeblair | SergeyLukjanov, fungi: i think the suggestion to force-merge the d-g change is because it and devstack need to change together | 15:42 |
| russellb | is getting a new empty git repo created an infra/config patch? or something manual I should put in a request for? | 15:42 |
| jeblair | russellb: infra/config patch | 15:42 |
| fungi | jeblair: oh, i see | 15:42 |
| russellb | jeblair: ok thanks | 15:42 |
| jeblair | russellb: i forget, have you done one yet? | 15:42 |
| russellb | don't think so | 15:42 |
| russellb | i can look through git history though | 15:42 |
| jeblair | russellb: http://ci.openstack.org/stackforge.html | 15:42 |
| jeblair | russellb: basically that, but s/stackforge/openstack/ or whatever as appropriate | 15:43 |
| fungi | russellb: you can see plenty of examples merged on the last couple of fridays | 15:43 |
| russellb | perfect thanks | 15:43 |
| SergeyLukjanov | jeblair, in fact we will not need any force-merge because savanna jobs are non voting :) | 15:43 |
| russellb | we want a repo for doing design reviews, instead of blueprint whiteboard madness | 15:43 |
| russellb | it's going to be awesome. | 15:43 |
| SergeyLukjanov | jeblair, so, I'm going to remove my attempts to override savanna repo urls and to upload corrsponding devstack change | 15:44 |
| jeblair | russellb: ah, yeah, i've been following that; it's exciting! | 15:44 |
| jeblair | SergeyLukjanov: cool, then we should be able to merge the d-g change normally ignoring the failures, then after that the devstack change should work and we can look at its tests before merging | 15:45 |
| SergeyLukjanov | jeblair, exactly | 15:45 |
| *** dcramer__ has quit IRC | 15:46 | |
| *** rpodolyaka has quit IRC | 15:46 | |
| SergeyLukjanov | just to clarify: [land sahara aliases to savanna and client] -> [rename all repos, jobs, etc] -> [push change to d-g that'll replace savanna repos clone/usage with sahara] -> [check that devstack change is working and merge it] -> [sahara dev is unblocked, could perform internal renamings] | 15:46 |
| jeblair | SergeyLukjanov: yes, except what do you mean by "push change to d-g"? isn't that change this one: https://review.openstack.org/#/c/79099/ | 15:48 |
| jeblair | SergeyLukjanov: you have an etherpad right, what's the link? | 15:48 |
| *** sparkycollier has joined #openstack-infra | 15:48 | |
| SergeyLukjanov | jeblair, it should be updated due to the last changes | 15:49 |
| SergeyLukjanov | jeblair, it's 79099 | 15:49 |
| jeblair | SergeyLukjanov: i think it should be updated to also remove savanna; is that what you mean? | 15:49 |
| *** fcarpenter has joined #openstack-infra | 15:50 | |
| SergeyLukjanov | jeblair, yup | 15:51 |
| *** SumitNaiksatam has quit IRC | 15:51 | |
| jeblair | SergeyLukjanov: then yep, that all sounds right. | 15:52 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: mock out lp in tests that don't really need it https://review.openstack.org/79325 | 15:52 |
| SergeyLukjanov | jeblair, thank you! | 15:52 |
| * SergeyLukjanov needs to update everything, should return back in an hour if hit will not fight me :) | 15:52 | |
| SergeyLukjanov | jeblair, here is a pad - https://etherpad.openstack.org/p/savanna-renaming-process | 15:53 |
| SergeyLukjanov | jeblair, re devstack, should we keep users able to enable savanna by specifying savanna service? | 15:53 |
| jeblair | SergeyLukjanov: for a little bit, but i don't think you need to do that for too long | 15:54 |
| SergeyLukjanov | jeblair, ok, for I release for ex. and than remove in early J | 15:54 |
| jeblair | SergeyLukjanov: probably just long enough for CI systems to update | 15:54 |
| jeblair | SergeyLukjanov: i'd do it much shorter than that, but i think it won't hurt if you want to leave it longer. | 15:55 |
| SergeyLukjanov | jeblair, oh, I don't think that we need it for CI, due to the intention to update d-g before devstack | 15:55 |
| SergeyLukjanov | jeblair, ok | 15:55 |
| openstackgerrit | A change was merged to openstack-infra/config: Use a sudoers file for jenkins sudo rights https://review.openstack.org/72777 | 15:56 |
| jeblair | fungi, clarkb, mordred, SergeyLukjanov: ^ that change could possibly break many or all jobs on the next image build | 15:57 |
| SergeyLukjanov | jeblair, :) | 15:57 |
| jeblair | fungi, clarkb, mordred, SergeyLukjanov: i'm planning on triggering one manually in a little while and keeping an eye on it | 15:57 |
| *** afazekas has joined #openstack-infra | 15:58 | |
| fungi | jeblair: thanks for the heads up | 15:58 |
| *** sarob_ has joined #openstack-infra | 15:59 | |
| *** hashar has joined #openstack-infra | 15:59 | |
| *** dcramer__ has joined #openstack-infra | 16:00 | |
| *** Hefeweizen has quit IRC | 16:00 | |
| mordred | jeblair: kk | 16:01 |
| *** sparkycollier has quit IRC | 16:01 | |
| *** sparkycollier has joined #openstack-infra | 16:01 | |
| *** rpodolyaka has joined #openstack-infra | 16:05 | |
| *** zehicle_at_dell has quit IRC | 16:06 | |
| *** davidhadas has quit IRC | 16:08 | |
| *** vkozhukalov has joined #openstack-infra | 16:08 | |
| *** sarob_ has quit IRC | 16:08 | |
| *** spredzy has quit IRC | 16:08 | |
| *** spredzy has joined #openstack-infra | 16:09 | |
| *** zehicle_at_dell has joined #openstack-infra | 16:11 | |
| *** dangers_` is now known as dangers | 16:12 | |
| *** yamahata has quit IRC | 16:13 | |
| *** Hefeweizen has joined #openstack-infra | 16:18 | |
| *** gyee has joined #openstack-infra | 16:19 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add logging to the new #openstack-sahara channel https://review.openstack.org/78960 | 16:21 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Revert "Make H202 check honor pep8 #noqa comment" https://review.openstack.org/79354 | 16:21 |
| jogo | sdague bnemec ^ | 16:22 |
| bnemec | jogo: I was +2 on that because I just saw this: https://github.com/openstack/oslo.messaging/blob/master/tests/test_qpid.py#L125 | 16:24 |
| bnemec | And we can't fix it because it's in third-party code. | 16:24 |
| bnemec | (I suppose we could raise a bug with the qpid people) | 16:25 |
| jogo | bnemec: yeah thats the better solution | 16:25 |
| jogo | I have seen H202 catch real bugs ... I still have yet to see a compelling reason to enable skipping it | 16:25 |
| jogo | it just promotes lazy devs | 16:25 |
| sdague | jogo: honestly, I'm actually -2 on your insistence that we don't do this globally | 16:25 |
| jogo | sdague: go on ... | 16:25 |
| sdague | I still believe #noqa should be globally applied to all hacking rules | 16:26 |
| *** gokrokve has joined #openstack-infra | 16:26 | |
| sdague | and then let projects decide they want exceptions in places | 16:26 |
| jeblair | sdague: ++ | 16:26 |
| jogo | so I can get behind that, but we need a config option to support that | 16:26 |
| sdague | jogo: no we don't | 16:26 |
| ArxCruz | sdague: can this patch be reverted: https://review.openstack.org/#/c/77067 | 16:26 |
| ArxCruz | we don't have mongodb in power | 16:26 |
| sdague | that should be hacking default behavior | 16:27 |
| ArxCruz | this patch is broken our CI | 16:27 |
| *** jgrimm has quit IRC | 16:27 | |
| jeblair | sdague, jogo: even the first rule of pep8 is about knowing when to be INconsistent | 16:27 |
| jogo | sdague: if a project doesn't want to support noqa for something ... they should be able to turn it off in config file | 16:27 |
| *** sarob_ has joined #openstack-infra | 16:27 | |
| sdague | jogo: I still disagree | 16:28 |
| jogo | jeblair: sure, I still have yet to see when H202 should be skipped | 16:28 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: mock out lp in tests that don't really need it https://review.openstack.org/79325 | 16:28 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: First step of renaming Savanna to Sahara https://review.openstack.org/79097 | 16:28 |
| sdague | if they don't want #noqa then they can write a local check to kill commits that have it | 16:28 |
| jogo | sdague: that is a really ugly answer | 16:28 |
| sdague | jogo: so is the current approach | 16:28 |
| jeblair | jogo: maybe never, but it doesn't matter -- hacking should be a useful tool for developers to help reviewers and give them the flexibility to make good choices | 16:29 |
| sdague | where lots of people don't want to use hacking because they have to fight you over every single #noqa exception | 16:29 |
| *** gmoro has quit IRC | 16:29 | |
| ArxCruz | dtroyer: can this patch be reverted: https://review.openstack.org/#/c/77067 I post my comments there | 16:29 |
| jeblair | jogo: anytime someone uses #noqa in a project, reviewes should pay extra attention -- but that's where the discussion should happen | 16:29 |
| sdague | ArxCruz: can you provide the actual fix which is to make exceptions for an architecture | 16:30 |
| jogo | I don't think they pay enough | 16:30 |
| sdague | jeblair: ++ | 16:30 |
| sdague | jogo: that's not your call to make | 16:30 |
| *** sarob_ has quit IRC | 16:30 | |
| *** jungleboyj has quit IRC | 16:30 | |
| jogo | sdague: I have seen several cases that didn't even need a noqa (from oslo... import _) | 16:30 |
| *** sarob_ has joined #openstack-infra | 16:30 | |
| *** sdake has joined #openstack-infra | 16:31 | |
| *** sdake has quit IRC | 16:31 | |
| *** sdake has joined #openstack-infra | 16:31 | |
| sdague | the whole point of this project was to make life *easier* on projects, and I think right now it creates an imposition as much as a helping hand | 16:31 |
| *** mkoderer has quit IRC | 16:31 | |
| *** zehicle_at_dell has quit IRC | 16:31 | |
| sdague | anyway, I'm -2 on the revert | 16:31 |
| *** sparkycollier has quit IRC | 16:31 | |
| fungi | jogo: however, #noqa is easy to grep for, and a source of low-hanging-fruit corrections someone can go in and make/test easily enough if they're so inclined | 16:31 |
| jogo | sdague: so I am all for noqa on everything that we think makes sense | 16:31 |
| fungi | it does a good job of marking things which got excluded from analysis, so it's easier to find them later | 16:31 |
| jogo | but I don't think every rule should support noqa | 16:32 |
| *** rpodolyaka has quit IRC | 16:32 | |
| jogo | for example the docstring rules | 16:32 |
| sdague | jogo: I disagree, it should be global | 16:32 |
| jogo | why would you need a noqa for that? | 16:32 |
| sdague | and the projects decide when it makes sense | 16:32 |
| sdague | jogo: I don't care, but if a project believes they need it, that's their call | 16:32 |
| *** atiwari has joined #openstack-infra | 16:32 | |
| fungi | i think the easier question is "why would you omit #noqa support only for specific checks?" probably less code just to support it everywhere | 16:33 |
| sdague | and it's a *ton* better to let them disable one instance, than to disable the entire rule | 16:33 |
| jogo | so two things on this specific patch: | 16:33 |
| jogo | 1) in general for noqa patches like this we can cut a new stable release since its backwards compat | 16:33 |
| *** Ryan_Lane1 has joined #openstack-infra | 16:33 | |
| jogo | 2) the aspect of this that cought my eye the most is how quickly this went in, especially something that we don't all agree on | 16:34 |
| jogo | If there was more discussion then sure, but there really wasn't | 16:34 |
| sdague | jogo: yeh, don't care. I think we have a fundamental diagreement on noqa which needs resolution otherwise this is just going to continue. | 16:34 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Rename Savanna to Sahara https://review.openstack.org/79099 | 16:35 |
| *** rpodolyaka has joined #openstack-infra | 16:35 | |
| *** sarob_ has quit IRC | 16:36 | |
| sdague | because I actually believe you are the only person who believes noqa shouldn't be global | 16:36 |
| jogo | sdague: that sounds about right, well noqa isn't global in pep8 either thats what I am basing this on | 16:36 |
| sdague | yeh, pep8 doesn't tons of things wrong :) | 16:36 |
| jogo | although we know how much everyone loves pep8 ;) | 16:36 |
| jogo | anyway so ++ on a addressing the noqa debaut at its root ... but until then I still don't know of any good cases for noqa on H202 | 16:37 |
| *** jgrimm has joined #openstack-infra | 16:37 | |
| sdague | I don't care | 16:37 |
| sdague | I'm +A on any new noqa rule | 16:37 |
| jogo | -2 means you clearly do | 16:37 |
| clarkb | ok now I am here | 16:37 |
| sdague | no, I don't care that you don't know of any good cases | 16:37 |
| sdague | my "I don't care", is I don't care that your opinion requires justification on this particular action | 16:38 |
| sdague | because I don't think adding noqa processing in hacking should require any justification | 16:38 |
| sdague | because it should do it | 16:38 |
| *** jlibosva has quit IRC | 16:39 | |
| ArxCruz | sdague: https://review.openstack.org/79356 | 16:39 |
| sdague | ArxCruz: so I think that remains the wrong fix | 16:40 |
| sdague | what code path is requiring mongo-db? | 16:40 |
| *** gmoro has joined #openstack-infra | 16:40 | |
| ArxCruz | sdague: I don't know | 16:41 |
| ArxCruz | maurosr: ^ | 16:41 |
| ArxCruz | do you know ? | 16:41 |
| sdague | so you need to know, because the right thing is not to install or configure things in way that would pull in mongodb | 16:41 |
| sdague | and if that means changing the logic a bit, so be it | 16:41 |
| sdague | but going back to silent fails is really the wrong thing | 16:41 |
| sdague | also, why didn't power CI vote comment on this? | 16:42 |
| SergeyLukjanov | jeblair, due to the agreement on renaming process, probably, we can schedule gerrit maintenance for it, please? :) | 16:42 |
| ArxCruz | sdague: As I mentioned in the change, we are beginning our infra, and we are not listening all the projects | 16:43 |
| ArxCruz | sadly devstack is one of the projects we are not checking yey | 16:43 |
| ArxCruz | yet* | 16:43 |
| *** Ryan_Lane has quit IRC | 16:43 | |
| *** Ryan_Lane1 is now known as Ryan_Lane | 16:43 | |
| *** Ryan_Lane has joined #openstack-infra | 16:43 | |
| sdague | ArxCruz: sure, but devstack changes should probably be high on the list, as otherwise these kinds of breaks are going to hit you a lot | 16:44 |
| sdague | given that fedora isn't tested in devstack anywhere upstream | 16:44 |
| *** Ryan_Lane1 has joined #openstack-infra | 16:44 | |
| ArxCruz | sdague: indeed, we are increasing our infra to accommodate more projects :) | 16:44 |
| *** pblaho has quit IRC | 16:48 | |
| *** harlowja has joined #openstack-infra | 16:49 | |
| *** e0ne has quit IRC | 16:49 | |
| *** torandu has joined #openstack-infra | 16:50 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Split IRC message if they are long. https://review.openstack.org/79360 | 16:50 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Use correct time.sleep argument. https://review.openstack.org/79361 | 16:50 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Fix exception logging. https://review.openstack.org/79121 | 16:50 |
| clarkb | jogo: sdague mtreinish ^ not tested yet but split into multiple changes and actually should fix the problem now | 16:51 |
| *** yamahata has joined #openstack-infra | 16:52 | |
| jogo | clarkb: nice | 16:52 |
| sdague | sweet | 16:52 |
| *** yjiang5 has joined #openstack-infra | 16:52 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Create nova-specs git repo https://review.openstack.org/79363 | 16:53 |
| ArxCruz | sdague: it seems that mongodb it's being used by ceilometer and marconi | 16:53 |
| SergeyLukjanov | fungi, clarkb, jeblair, mordred, looks like, all CRs needed to rename savanna - https://review.openstack.org/#/q/status:open+topic:savanna-sahara+-savanna,n,z + I'm working on updating Tempest too, but it's not required due to the temp TEMPEST+=,savanna in devstack | 16:54 |
| sdague | ArxCruz: so don't enable those services | 16:54 |
| SergeyLukjanov | sdague, ^^ | 16:54 |
| fungi | SergeyLukjanov: great! taking another look now | 16:54 |
| SergeyLukjanov | fungi, thank you! | 16:55 |
| sdague | clarkb: your series looks good to me | 16:55 |
| sdague | thanks for diving through that | 16:55 |
| *** dizquierdo has quit IRC | 16:57 | |
| *** davidhadas has joined #openstack-infra | 16:58 | |
| ArxCruz | sdague: but the silent fail it's only for ppc64, until we create a mongodb version to ppc64 | 16:58 |
| harlowja | lifeless u make it back ok? :-P | 16:58 |
| harlowja | no plane issues like the other plane that is MIA | 16:58 |
| sdague | ArxCruz: or turn off those services that require it | 16:59 |
| sdague | because you aren't working with them anyway | 16:59 |
| dtroyer | ArxCruz: another option is to add an archand/or NOPRIME restriction to the packages that are missing in the prereq files in files/{apts,rpms}/*, than make sure they are specifically installed when required | 17:01 |
| *** khyati has joined #openstack-infra | 17:01 | |
| *** sarob_ has joined #openstack-infra | 17:02 | |
| dtroyer | you'll need to track down the services requiring them (looks like you have that) and the complete list of missing packages | 17:02 |
| ArxCruz | dtroyer: the missing package is the mongodb-server | 17:02 |
| dtroyer | just the one? | 17:02 |
| ArxCruz | yes | 17:02 |
| dtroyer | mongodb is already handled specifically for fedora | 17:03 |
| ArxCruz | dtroyer: mongodb yes, but mongodb-server no | 17:03 |
| yjiang5 | hi, all, I hit a Trubo-hipster failure in https://review.openstack.org/#/c/69731/ but I don't think my patch has anything related to migration. Should I report a but for it for recheck? I'm a bit surprised to find no bug for turbo hipster in http://status.openstack.org/rechecks/ , so not sure if we need do this for turbo hipster. | 17:04 |
| dtroyer | point is you may need something similar | 17:04 |
| *** amcrn has quit IRC | 17:04 | |
| *** jpich has quit IRC | 17:04 | |
| *** bhuvan has joined #openstack-infra | 17:04 | |
| *** bhuvan_ has joined #openstack-infra | 17:05 | |
| *** sabari2 has joined #openstack-infra | 17:05 | |
| yjiang5 | oops, should I ask this in #openstack-qa? | 17:05 |
| fungi | yjiang5: turbo-hipster is an independent third-party test system. you want to give mikal or jhesketh a heads up on it, probably | 17:05 |
| ArxCruz | dtroyer: the is_arch works in the files/rpms/service ? | 17:05 |
| yjiang5 | fungi: thanks. | 17:05 |
| dtroyer | ArxCruz: no, just distro, testonly and noprime | 17:06 |
| *** yamahata__ has joined #openstack-infra | 17:07 | |
| *** hashar has quit IRC | 17:08 | |
| *** nati_ueno has joined #openstack-infra | 17:09 | |
| *** jcoufal has quit IRC | 17:09 | |
| *** jcoufal has joined #openstack-infra | 17:09 | |
| *** jcoufal has quit IRC | 17:10 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:11 | |
| *** krotscheck has joined #openstack-infra | 17:12 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: fix setting of default values for missing parameters in jenkins.ini file. https://review.openstack.org/78458 | 17:14 |
| rcarrillocruz1 | hey zaro, thanks for the review, I'll do a proper regex match for releases, re: https://review.openstack.org/#/c/79205/ | 17:14 |
| *** Ryan_Lane has quit IRC | 17:15 | |
| *** yamahata has quit IRC | 17:15 | |
| *** yamahata has joined #openstack-infra | 17:16 | |
| *** odyssey4me has quit IRC | 17:16 | |
| *** eharney has quit IRC | 17:17 | |
| fungi | SergeyLukjanov: on 79097, were you planning to add a separate (wip or whatever) change to switch the bug groups after you rename projects on launchpad? | 17:17 |
| *** yamahata has quit IRC | 17:18 | |
| *** yamahata__ has quit IRC | 17:18 | |
| *** yamahata has joined #openstack-infra | 17:18 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Split IRC message if they are long. https://review.openstack.org/79360 | 17:19 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/elastic-recheck: Use correct time.sleep argument. https://review.openstack.org/79361 | 17:19 |
| *** rpodolyaka has quit IRC | 17:19 | |
| *** rpodolyaka has joined #openstack-infra | 17:20 | |
| *** yamahata__ has joined #openstack-infra | 17:20 | |
| *** CaptTofu has quit IRC | 17:22 | |
| *** markmc has quit IRC | 17:22 | |
| *** CaptTofu has joined #openstack-infra | 17:22 | |
| *** CaptTofu has quit IRC | 17:23 | |
| *** CaptTofu_ has joined #openstack-infra | 17:23 | |
| *** hogepodge has joined #openstack-infra | 17:23 | |
| *** dhellmann is now known as dhellmann_ | 17:23 | |
| jogo | sdague jeblair: so I hope to cut a new hacking by the end of the week, and seeing that I appear to be the only one who doesn't want noqa everywhere ... lets just make hacking 0.9 support noqa everywhere | 17:24 |
| sdague | jogo: +1 :) | 17:24 |
| jeblair | sounds lovely! | 17:24 |
| *** rpodolyaka has quit IRC | 17:26 | |
| jogo | although not sure what noqa would mean for multi line rules like license | 17:26 |
| jogo | stuff | 17:26 |
| *** amcrn has joined #openstack-infra | 17:27 | |
| *** johnthetubaguy1 has quit IRC | 17:27 | |
| *** rpodolyaka has joined #openstack-infra | 17:28 | |
| *** johnthetubaguy has joined #openstack-infra | 17:29 | |
| sdague | we can consider that a fail open case | 17:29 |
| sdague | where there is no sensible way to specify that anyway | 17:30 |
| sdague | jeblair: big open question, what's your current thinking on 14.04 in infrastructure | 17:30 |
| *** Ryan_Lane has joined #openstack-infra | 17:30 | |
| *** jungleboyj has joined #openstack-infra | 17:31 | |
| *** basha has joined #openstack-infra | 17:32 | |
| jeblair | sdague: as soon as rax and hp provide images (assuming we don't start using glance before then) we can start putting them in the mix; i would normally expect juno to start gating on them but not icehouse. | 17:32 |
| *** sdake_ has quit IRC | 17:32 | |
| jeblair | sdague: traditionally, rax and hp have been about a few weeks to a month behind anyway | 17:32 |
| *** sdake_ has joined #openstack-infra | 17:33 | |
| *** sdake_ has quit IRC | 17:33 | |
| *** sdake_ has joined #openstack-infra | 17:33 | |
| sdague | jeblair: ok, cool. If we see images early, it would be good to have non voting devstack jobs up as soon as possible. | 17:34 |
| *** ArxCruz has quit IRC | 17:34 | |
| sdague | because I expect we'll want need to work up the stack and fix a couple things along the way | 17:34 |
| jeblair | sdague: the last time we actually did a full slave migration, it also ended up taking a month or two just to get all the projects moved over anyway. | 17:34 |
| *** andreaf has quit IRC | 17:34 | |
| sdague | that's fair | 17:34 |
| *** ArxCruz has joined #openstack-infra | 17:34 | |
| jeblair | sdague: yeah; certainly non-voting early can help us get ahead on that | 17:34 |
| sdague | I just wasn't sure if we had seen any beta images out there | 17:34 |
| sdague | to get that devstack job started | 17:34 |
| jeblair | sdague: i haven't gone looking | 17:35 |
| sdague | as I expect whatever head start we can get, will help | 17:35 |
| jeblair | sdague: so, big picture plan is probably something like: "non-voting asap, gating early in juno" | 17:35 |
| sdague | +1 | 17:35 |
| jeblair | sdague: (and of course precise will stick around to continue to serve <= icehouse) | 17:36 |
| *** johnthetubaguy1 has joined #openstack-infra | 17:36 | |
| sdague | I just know there are some edge conditions around 13.10 that people seem to hit, that I expect we'll probably have a few things to fix | 17:36 |
| sdague | yeh, so I guess we'll have a fun question on grenade jobs | 17:36 |
| sdague | but I actually think that can wait for summit | 17:37 |
| sdague | and actually figure out how many real issues we hit | 17:37 |
| *** johnthetubaguy has quit IRC | 17:37 | |
| *** lascii is now known as alaski | 17:39 | |
| SergeyLukjanov | fungi, yup, I'll add it | 17:39 |
| *** sweston has quit IRC | 17:39 | |
| *** pbelanyi has quit IRC | 17:39 | |
| *** dripton has quit IRC | 17:39 | |
| harlowja | anyone know of a good monotonic time library that we can use in openstack (seems like the ones on pypi are sorta stale) | 17:40 |
| harlowja | *monotonic clock exists in 3.3 (but thats not so useful, lol) | 17:40 |
| *** dripton has joined #openstack-infra | 17:41 | |
| jogo | clarkb: https://review.openstack.org/#/c/79360/2/elastic_recheck/bot.py why 400 | 17:45 |
| clarkb | jogo: its an arbitrary guess | 17:46 |
| clarkb | jogo: to do it correctly you would have to add all the overhead for each IRC message, subtract that from 512 and use thatnumber | 17:47 |
| clarkb | 400 should be conservative enough that we can ignore that for now | 17:47 |
| jogo | ahhh | 17:47 |
| jeblair | (overhead includes things like channel names, nicks, etc) | 17:47 |
| jogo | thats the context I was missing, thanks | 17:47 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Fix exception logging. https://review.openstack.org/79121 | 17:48 |
| *** gyee has quit IRC | 17:49 | |
| *** mbacchi has quit IRC | 17:49 | |
| pleia2 | bug day tomorrow, just sent a reminder to the list too | 17:50 |
| *** derekh has quit IRC | 17:51 | |
| clarkb | jeblair: I am trying to add more logging to the zuul merger, specific to log which changes are triggering actions so that you can trace from zuul server to zuul merger and back again. But adding number, patchset, oldrev and newrev to _makeMergerItem fails a bunch of tests. Is that overloaded in a funny way in the test suite? | 17:53 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Split IRC message if they are long. https://review.openstack.org/79360 | 17:54 |
| *** markmcclain has quit IRC | 17:54 | |
| *** dkehn__ is now known as dkehn_ | 17:55 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Use correct time.sleep argument. https://review.openstack.org/79361 | 17:55 |
| jeblair | clarkb: i don't think the merger is faked at all in the test suite | 17:56 |
| *** mwagner_lap has quit IRC | 17:56 | |
| clarkb | jeblair: gah nevermind, I simply can't write code :) | 17:56 |
| jeblair | clarkb: yay test suite works! :) | 17:57 |
| sdague | clarkb: all your er patches merged | 17:58 |
| SpamapS | fungi: hey, important urgent thing to tell you | 17:58 |
| SpamapS | fungi: _the cloud stayed up all weekend_ .. ;-) | 17:58 |
| fungi | SpamapS: sorry to hear that, i'll get right on breaking it for you then | 17:59 |
| SpamapS | fungi: kthxbai! :) | 17:59 |
| *** nicedice has quit IRC | 17:59 | |
| *** nicedice has joined #openstack-infra | 18:00 | |
| *** jungleboyj has quit IRC | 18:00 | |
| fungi | another third-party testing inservice is starting in -meeting | 18:01 |
| openstackgerrit | A change was merged to openstack-infra/config: remove python33 from marconi gate https://review.openstack.org/79316 | 18:01 |
| *** jungleboyj has joined #openstack-infra | 18:01 | |
| fungi | (or so said the e-mail from jaypipes) | 18:01 |
| jaypipes | fungi: yes, ty! :) | 18:01 |
| jaypipes | #startmeeting | 18:01 |
| openstack | jaypipes: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 18:01 |
| fungi | jaypipes: you may want to do that in -meeting | 18:01 |
| fungi | ;) | 18:02 |
| *** mriedem1 has joined #openstack-infra | 18:02 | |
| jaypipes | oh, doh... | 18:02 |
| *** dcramer__ has quit IRC | 18:02 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add more logging to zuul merger process https://review.openstack.org/79385 | 18:02 |
| lifeless | harlowja: sure did | 18:03 |
| clarkb | jeblair: ^ that is what I am trying to do. Basically making it easier to figure out what the merger is doing | 18:03 |
| harlowja | lifeless sweet, no MIA planes allowed, ha | 18:03 |
| *** ildikov_ has quit IRC | 18:03 | |
| *** mriedem has quit IRC | 18:04 | |
| clarkb | jeblair: that is partially in response to seeing the zuul merger's main thread die (process was still going but pid file was removed and nothing was logged) and needing more log info | 18:04 |
| *** lukego has quit IRC | 18:05 | |
| clarkb | jeblair: in a merger for a different zuul. and since I think it is happening in the cmd file I don't expect that to be easily unittestable | 18:05 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Enable story and project 'deletion' https://review.openstack.org/77764 | 18:05 |
| jeblair | clarkb: i think that happens in infra zm as well | 18:05 |
| clarkb | jeblair: oh good useful for everyone then | 18:05 |
| jeblair | clarkb: and for our irc bots | 18:05 |
| jeblair | clarkb: do we have a USR2 handler for the merger? if not, maybe we should | 18:06 |
| clarkb | jeblair: yeah I think it is related to being hit with a signal like USR2 from somewhere | 18:06 |
| clarkb | jeblair: at the very least I think we should log unhandled exceptions like that | 18:06 |
| *** yassine has quit IRC | 18:07 | |
| jeblair | clarkb: i'm a big fan of logging unhandled exceptions | 18:08 |
| *** zhiyan is now known as zhiyan_ | 18:08 | |
| *** sandywalsh_ has joined #openstack-infra | 18:08 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Keep py3.X compatibility for urllib/urllib2 https://review.openstack.org/76124 | 18:08 |
| *** sandywalsh_ has quit IRC | 18:08 | |
| *** mkerrin has joined #openstack-infra | 18:09 | |
| *** mkerrin has quit IRC | 18:10 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Creating a new story now requires authentication. https://review.openstack.org/79390 | 18:11 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Create story modal allows multi-task creation. https://review.openstack.org/78408 | 18:11 |
| clarkb | jeblair: it does catch USR2 with the stack dump handler. I will have to remember to get a stack trace with that next time I see this | 18:11 |
| jeblair | clarkb: check if zm01 or zm02 are missing pid files now? | 18:12 |
| clarkb | jeblair: they both have pid files right now | 18:12 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Cancel a build even if not found https://review.openstack.org/79391 | 18:12 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: dequeue abandoned changes https://review.openstack.org/65467 | 18:12 |
| jogo | has the pypy issue been resolved? | 18:13 |
| jeblair | there was a pypy issue? | 18:13 |
| jogo | allegedly: [openstack-dev] [Nova client] gate-python-novaclient-pypy | 18:14 |
| * fungi fast-forwards his dev ml backlog by a week | 18:15 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:17 | |
| *** jgallard has quit IRC | 18:18 | |
| *** zhiyan_ is now known as zhiyan | 18:19 | |
| fungi | jogo: gah. that's exactly the issue we had been having on py3k-precise nodepool nodes which got me to originally roll the pypy tests back to the old static nodes because setuptools 2.1 was breaking, then later when setuptools 2.2 came out i was unable to reproduce the issue so i moved the jobs. now it's cropping up again... why? | 18:20 |
| *** talluri has joined #openstack-infra | 18:20 | |
| fungi | looking into it | 18:21 |
| fungi | of course... setuptools 3.1! | 18:21 |
| fungi | released saturday | 18:21 |
| *** basha has quit IRC | 18:21 | |
| fungi | i'll manually reproduce this and then see if mordred's proposed change which would have solved the issue for 2.1 works with 3.1 | 18:22 |
| *** dkliban has joined #openstack-infra | 18:22 | |
| clarkb | fungi: :/ | 18:22 |
| jogo | fungi: haha ouch | 18:23 |
| *** dhellmann_ is now known as dhellmann | 18:23 | |
| *** moted has quit IRC | 18:23 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Fixed padding on modals. https://review.openstack.org/79393 | 18:23 |
| pleia2 | fungi and/or jeblair - can I have you look at this change for fedora? https://review.openstack.org/#/c/69510/ I've had to rebase a couple times, would be nice to have it finally land | 18:23 |
| *** bhuvan_ has quit IRC | 18:24 | |
| *** bhuvan has quit IRC | 18:24 | |
| *** bhuvan_ has joined #openstack-infra | 18:25 | |
| *** bhuvan has joined #openstack-infra | 18:25 | |
| *** davidlenwell_ is now known as davidlenwell | 18:26 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 18:28 | |
| *** denis_makogon_ has joined #openstack-infra | 18:29 | |
| *** lyxus has quit IRC | 18:29 | |
| *** SumitNaiksatam has quit IRC | 18:30 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 18:30 | |
| *** esker has quit IRC | 18:30 | |
| *** dprince has quit IRC | 18:30 | |
| *** mkerrin has joined #openstack-infra | 18:31 | |
| clarkb | jeblair: hrm my except Exception won't actually catch anything there best I can tell. Since python is probably failing within the default signal handlers rather than throwing an exception | 18:31 |
| clarkb | and python-daemon does stuff with those iirc | 18:32 |
| *** denis_makogon_ is now known as denis_makogon | 18:33 | |
| *** zhiyan is now known as zhiyan_ | 18:34 | |
| clarkb | I wonder if it is getting SIGCHLD | 18:34 |
| clarkb | from git | 18:34 |
| *** fbo is now known as fbo_away | 18:34 | |
| fungi | clarkb: even better detail on the pypy failure... happens on rax but not hpcloud! (presumably because of differing initial versions of things installed or not installed) | 18:37 |
| *** nati_uen_ has joined #openstack-infra | 18:37 | |
| *** gyee has joined #openstack-infra | 18:37 | |
| *** jp_at_hp has quit IRC | 18:38 | |
| *** sweston has joined #openstack-infra | 18:38 | |
| *** nati_ueno has quit IRC | 18:40 | |
| fungi | that might also explain why mordred was having a hard time reproducing it last time it cropped up | 18:43 |
| clarkb | dstufft: https://pypi.python.org/pypi/python-daemon/1.6 can you explain why latest version is 1.5.5? | 18:46 |
| clarkb | dstufft: pip will install 1.6 | 18:46 |
| *** freyes has quit IRC | 18:47 | |
| dstufft | clarkb: maybe | 18:47 |
| dstufft | let me look | 18:47 |
| clarkb | the pid file handling appears to be different which is fine but odd that you don't get 1.6 as latest | 18:48 |
| dstufft | clarkb: 1.6 is set to "hidden" on PyPI | 18:49 |
| dstufft | which means it won't show up in the UX unless you browse to it manually | 18:49 |
| fungi | clearly not the intended use for that pypi feature | 18:50 |
| clarkb | dstufft: but it isn't hidden from pip | 18:50 |
| *** krotscheck has quit IRC | 18:50 | |
| fungi | clarkb: i imagine pypi "hidden" being broken in similar ways to gerrit "draft" | 18:50 |
| *** fcarpenter has left #openstack-infra | 18:50 | |
| dstufft | pip doesn't use the UX | 18:51 |
| dstufft | it's a UX only thing | 18:51 |
| dstufft | sec | 18:51 |
| fungi | sometimes referred to as an "attractive nuisance" | 18:51 |
| dstufft | let me show you what it looks like if you don't have all versions but one hidden | 18:51 |
| dstufft | https://pypi.python.org/pypi/Django/ | 18:51 |
| dstufft | vs https://pypi.python.org/pypi/python-daemon/ | 18:51 |
| dstufft | it's kind of dumb, Warehouse gets rid of it | 18:52 |
| *** markmcclain has joined #openstack-infra | 18:52 | |
| dstufft | https://pypi-preview.a.ssl.fastly.net/project/python-daemon/ | 18:52 |
| fungi | i strongly suspect that the python-daemon devs thought they were making 1.6 a prerelease by hiding it | 18:53 |
| dstufft | probably | 18:53 |
| clarkb | fungi: ya | 18:53 |
| dstufft | a lot of people do think that | 18:53 |
| dstufft | there's a reason I'm trying to kill that feature | 18:53 |
| clarkb | fungi: and I wonder if it is related to us losing lock files where we use 1.6 | 18:53 |
| dstufft | "feature" | 18:53 |
| jeblair | dstufft: ++ | 18:53 |
| dstufft | there's some somewhat sane historical reasons for why it is that way | 18:54 |
| dstufft | but those historical reasons no longer matter | 18:54 |
| jeblair | clarkb, fungi: but if 1.6 was uploaded in 2010, surely we've been using it the whole time | 18:54 |
| clarkb | jeblair: no it depends on how we install it I think | 18:54 |
| *** afazekas_ has joined #openstack-infra | 18:54 | |
| *** afazekas_ is now known as afazekas_wfp | 18:54 | |
| clarkb | jeblair: zm01 is 1.5.5 | 18:54 |
| clarkb | mordreds zm is 1.6 | 18:55 |
| jeblair | oh | 18:55 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Minor fixes to styling and coherency https://review.openstack.org/77442 | 18:55 |
| dstufft | FWIW there is no way to hide a file from pip | 18:55 |
| clarkb | I am trying to figure that out now | 18:55 |
| dstufft | it's either uploaded and available, or deleted | 18:55 |
| clarkb | but I think mordred is using the pip installs everything change | 18:55 |
| clarkb | we are not so he gets everything via pip we get some stuff via apt | 18:55 |
| clarkb | this probably isn't related to the problems we are seeing | 18:56 |
| jeblair | clarkb: we have 1.6 on eavesdrop | 18:56 |
| clarkb | it was just really odd when I started looking that direction | 18:56 |
| *** basha has joined #openstack-infra | 18:56 | |
| clarkb | jeblair: oh maybe it is related then | 18:56 |
| clarkb | jeblair: I noticed this beacuse 1.6 comes with a file to do pid files. 1.5.5 seems to have a lock file too and a thread file | 18:57 |
| jeblair | clarkb: you've lost me. i'm confused on 2 points. you say "pip will install 1.6" and "mordred is using pip to install everything" and "mordred's installed 1.5.5" | 18:57 |
| jeblair | clarkb: that's the first point | 18:57 |
| jeblair | clarkb: and why does 1.6 on eavesdrop make you think it's related if mordred's zm is running 1.5.5? | 18:57 |
| clarkb | mordreds is 1.6 | 18:57 |
| *** reed has joined #openstack-infra | 18:57 | |
| jeblair | clarkb: oh, ok, i got that bit backward | 18:57 |
| *** e0ne has joined #openstack-infra | 18:58 | |
| *** moted has joined #openstack-infra | 18:58 | |
| clarkb | jeblair: we can probably check really quickly if this is the problem by setting python-daemon==1.5.5 in our requirements files | 18:58 |
| jeblair | clarkb: how is that quick? | 18:59 |
| clarkb | jeblair: it doesn't require us to redo a bunch of how puppet installs stuff | 18:59 |
| clarkb | granted dstufft says we won't be able to do that with pip anyways | 18:59 |
| dstufft | what can't you do? | 18:59 |
| clarkb | dstufft: install non hidden packages | 19:00 |
| dstufft | you can't say "pip install something==only-the-non-hidden" correct | 19:00 |
| jeblair | clarkb: i think dstufft said you can install any version with pip | 19:00 |
| jeblair | oh | 19:00 |
| fungi | specifically tell pip to determine and install the latest non-hidden version on pypi | 19:00 |
| jeblair | clarkb: yeah, but we can tell it to install !=1.6 | 19:00 |
| clarkb | jeblair: ya | 19:00 |
| dstufft | the hidden flag is strictly 100% UX only | 19:00 |
| dstufft | pip knows nothing about it | 19:00 |
| dstufft | (this is a lie but it's close enough) | 19:01 |
| *** luis_ has joined #openstack-infra | 19:01 | |
| jeblair | clarkb: it's not the hiddeness that makes us want to not install 1.6, it's that 1.6 is broken, so that's entirely appropriate. :) | 19:01 |
| jeblair | (allegedly broken) | 19:01 |
| clarkb | jeblair: that | 19:01 |
| *** basha has quit IRC | 19:02 | |
| clarkb | jeblair: statusbot was having similar trouble right? | 19:02 |
| jeblair | clarkb: yeah | 19:02 |
| clarkb | on eavesdrop. This is sufficiently weird enough that I am willing to bet it is a problem | 19:03 |
| dstufft | what is "eavesdrop" | 19:03 |
| jeblair | i'm still really annoyed that daemon isn't in stdlib. python seems to be increasingly not 'batteries included'. | 19:04 |
| jeblair | dstufft: where our irc bots run | 19:04 |
| *** rpodolyaka has quit IRC | 19:04 | |
| dstufft | oh | 19:04 |
| dstufft | jeblair: heh, you'd hate my thoughts on the stdlib | 19:04 |
| *** luisg has quit IRC | 19:04 | |
| *** luis_ has quit IRC | 19:05 | |
| * dstufft wants to gut the stdlib | 19:05 | |
| *** luisg has joined #openstack-infra | 19:05 | |
| jeblair | dstufft: maybe :). i think a huge part of why python is successful is because it had a useful stdlib. | 19:05 |
| clarkb | I am going to diff 1.6 and 1.5.5 now | 19:05 |
| *** basha has joined #openstack-infra | 19:06 | |
| *** rpodolyaka has joined #openstack-infra | 19:06 | |
| dstufft | jeblair: I think a large stdlib is a sign your packaging tools suck, and that there's not really much of a reason to tie the release cycle of say a http library to the release cycle of the Python language :D | 19:07 |
| clarkb | most of the changes seem to be around unicode vs non unicode strings | 19:08 |
| jeblair | dstufft: yeah, we disagree, which is probably enough for now until we have beers :) | 19:08 |
| clarkb | but the pidfile stuff is very different | 19:08 |
| dstufft | jeblair: :D | 19:09 |
| fungi | oh boy... so the pypy-based unit test issue gets worse... apparently i can't upgrade setuptools within the virtualenv: http://paste.openstack.org/show/73052/ | 19:09 |
| *** mriedem has joined #openstack-infra | 19:10 | |
| jeblair | fungi: have you determined the diff btwn hpcloud and rax? | 19:12 |
| fungi | jeblair: that's where i'm headed next in hopes it will enlighten me | 19:12 |
| *** johnthetubaguy1 has quit IRC | 19:13 | |
| *** mriedem1 has quit IRC | 19:15 | |
| *** sarob_ has quit IRC | 19:16 | |
| *** dcramer__ has joined #openstack-infra | 19:17 | |
| clarkb | so I don't think python-daemon is at fault just something weird | 19:25 |
| *** talluri has quit IRC | 19:25 | |
| clarkb | jeblair: what do you think about having atexit functions that we can call to do stack dumps? | 19:25 |
| clarkb | would make logs super verbose when we want to stop zuul | 19:25 |
| *** JuanManuelOlle has joined #openstack-infra | 19:26 | |
| *** talluri has joined #openstack-infra | 19:26 | |
| *** e0ne has quit IRC | 19:26 | |
| *** basha has quit IRC | 19:26 | |
| *** basha has joined #openstack-infra | 19:28 | |
| *** talluri has quit IRC | 19:30 | |
| *** zns has quit IRC | 19:31 | |
| *** e0ne has joined #openstack-infra | 19:32 | |
| *** bhuvan_ has quit IRC | 19:33 | |
| *** bhuvan has quit IRC | 19:33 | |
| jogo | sdague: so sqlalchemy 0.9.3 | 19:37 |
| *** malini is now known as malini_afk | 19:37 | |
| jogo | https://review.openstack.org/#/c/79338/ | 19:37 |
| jogo | but pip freeze shows 0.8.5 | 19:37 |
| jogo | is the use global reqs logic in gate working? | 19:38 |
| *** nati_uen_ has quit IRC | 19:38 | |
| jogo | fungi: ^ | 19:38 |
| *** nati_ueno has joined #openstack-infra | 19:38 | |
| *** basha has quit IRC | 19:39 | |
| fungi | jogo: is sqlalchemy 0.9.3 in our mirror? | 19:39 |
| jogo | fungi: yup | 19:39 |
| jogo | http://pypi.openstack.org/openstack/SQLAlchemy/ | 19:39 |
| mriedem | https://github.com/openstack/keystone/blob/master/requirements.txt#L12 | 19:40 |
| *** gokrokve_ has joined #openstack-infra | 19:40 | |
| mriedem | so you won't be able to run sqla 0.9 in deployments until all projects move up, we had that problem with sqla 0.8 | 19:40 |
| *** rpodolyaka has quit IRC | 19:41 | |
| *** jungleboyj has quit IRC | 19:41 | |
| fungi | jogo: so... look at the last-modified timestamp on the url you just pasted | 19:41 |
| fungi | SQLAlchemy-0.9.3.tar.gz10-Mar-2014 15:09 4.2M | 19:41 |
| fungi | and then compare that to when the jobs installed SQLAlchemy from the mirror | 19:41 |
| mriedem | yeah, we need to recheck | 19:41 |
| jogo | fungi: I checked newer patches too | 19:42 |
| jogo | fungi: http://logs.openstack.org/85/78985/8/check/check-tempest-dsvm-neutron/7afdf54/logs/pip-freeze.txt.gz | 19:42 |
| fungi | jogo: okay, so a non-devstack job which started after 15:09 has older sqlalchemy? | 19:42 |
| mriedem | jogo: well why that other job pull in sqla 0.9 when the requirements.txt isn't updated yet? | 19:43 |
| mriedem | b/c nova is capped on sqla < 0.8.99 right now | 19:43 |
| fungi | oh, right if it's updated in openstack/requirements then devstack jobs should get it too | 19:43 |
| *** jungleboyj has joined #openstack-infra | 19:43 | |
| jogo | fungi: yeah | 19:43 |
| mriedem | so devstack jobs run against versions that the individual projects are capping against? | 19:43 |
| jogo | but they don't appear to begetting it | 19:44 |
| *** krotscheck has joined #openstack-infra | 19:44 | |
| jogo | mriedem: we try to run against global reqs only | 19:44 |
| *** gokrokve has quit IRC | 19:44 | |
| *** vkozhukalov has quit IRC | 19:44 | |
| mriedem | that seems crazy to me | 19:45 |
| mriedem | but if you're running devstack for all different jobs with different reqs, i guess you'd have to | 19:45 |
| jogo | mriedem: its a comprimise on making sure we have a standard set of deps | 19:45 |
| *** bada_ has quit IRC | 19:45 | |
| fungi | http://logs.openstack.org/85/78985/8/check/check-tempest-dsvm-neutron/7afdf54/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-03-10_17_24_25_993 | 19:45 |
| mriedem | yeah, s/different jobs/different projects/ | 19:45 |
| jogo | but it appears to not be working as I thought it did | 19:45 |
| fungi | that should be getting it what it needs, i think | 19:45 |
| *** mbacchi has joined #openstack-infra | 19:45 | |
| jogo | fungi: yeah but I don't see those versions overwritting the per repo | 19:46 |
| *** bada has joined #openstack-infra | 19:46 | |
| sdague | jogo so that means someone broke global requirements, and we never sorted it out | 19:47 |
| fungi | http://logs.openstack.org/85/78985/8/check/check-tempest-dsvm-neutron/7afdf54/logs/devstacklog.txt.gz#_2014-03-10_17_28_42_805 | 19:47 |
| openstackgerrit | Ricardo Carrillo Cruz proposed a change to openstack-infra/gerritbot: Add support for notification of releases (and ignore pre-releases) on IRC https://review.openstack.org/79205 | 19:48 |
| jogo | sdague: :/ where did the logic live to do global reqs? | 19:48 |
| sdague | in the global requirements repo | 19:48 |
| *** nati_ueno has quit IRC | 19:48 | |
| *** ociuhandu has quit IRC | 19:48 | |
| *** rossella_s has quit IRC | 19:48 | |
| jogo | ahh update.py | 19:49 |
| sdague | grr | 19:49 |
| sdague | someone in one of their "being helpful" changes in devstack - http://logs.openstack.org/29/79129/2/gate/gate-tempest-dsvm-full/4e9387a/logs/devstacklog.txt.gz#_2014-03-10_13_39_05_980 | 19:49 |
| *** nati_ueno has joined #openstack-infra | 19:51 | |
| jogo | sdague: I58357757ac67a919bf70178b76f65fa0a9e16242 or Ia670198332af5945a56d708cd83d9239df0c2287 | 19:52 |
| jogo | both look fishy | 19:52 |
| sdague | yeh, I wonder how broken we are now | 19:53 |
| *** hashar has joined #openstack-infra | 19:53 | |
| jogo | yeah ... this could be very fun | 19:54 |
| *** mbacchi has quit IRC | 19:54 | |
| jogo | mriedem: why do you need to ask good questions, things were easier with our heads in the sand | 19:54 |
| mriedem | jogo: making up for last week :) | 19:55 |
| *** sarob_ has joined #openstack-infra | 19:55 | |
| mriedem | when i was basically MIA all week | 19:55 |
| *** afazekas_wfp has quit IRC | 19:55 | |
| sdague | nice catch mrda_away | 19:55 |
| sdague | nice catch mriedem | 19:56 |
| *** mbacchi has joined #openstack-infra | 19:56 | |
| sdague | so I'm going to do the sledge hammer fix | 19:56 |
| sdague | https://review.openstack.org/79431 | 19:56 |
| mrodden | ruh roh | 19:56 |
| mriedem | mc style or peter gabriel style? | 19:56 |
| sdague | then someone else can figure out how to do the don't write into people's devstack dir fixes | 19:56 |
| SergeyLukjanov | fungi, jeblair, clarkb, I hope that I've finished checking where should we rename something for savanna -> sahara; here are savanna+client changes that adds aliases https://review.openstack.org/#/q/status:open+topic:savanna-sahara+savanna,n,z and infra/docs/tempest/devstack/d-g changes https://review.openstack.org/#/q/status:open+topic:savanna-sahara+-savanna,n,z | 19:57 |
| jogo | this should be interesting | 19:58 |
| *** nati_ueno has quit IRC | 19:59 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Create nova-specs git repo https://review.openstack.org/79363 | 19:59 |
| *** nati_ueno has joined #openstack-infra | 19:59 | |
| jogo | sdague: any ideas of how this broke? | 20:00 |
| sdague | refactoring in devstack, and it just missed it | 20:01 |
| sdague | we don't actually have a synthetic test here | 20:01 |
| *** bhuvan has joined #openstack-infra | 20:02 | |
| *** bhuvan_ has joined #openstack-infra | 20:02 | |
| sdague | https://review.openstack.org/#/c/77815/1 might fix it | 20:03 |
| *** zns has joined #openstack-infra | 20:03 | |
| jogo | ahh that make sense | 20:04 |
| jogo | and if that is the bug, then we are only 1 week off | 20:04 |
| *** alexpilotti has quit IRC | 20:04 | |
| jogo | so not bad | 20:04 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Create story modal allows multi-task creation. https://review.openstack.org/78408 | 20:04 |
| *** esker has joined #openstack-infra | 20:04 | |
| sdague | we'll see what the runs look | 20:04 |
| sdague | like | 20:04 |
| sdague | jogo you starting to classify more of the gate bugs? | 20:05 |
| sdague | we are definitely creeping up again, I plucked out a couple this morning | 20:05 |
| jogo | sdague: still a little behind, haven't had a chance | 20:05 |
| jogo | but noticed things are on the rise | 20:05 |
| jogo | only a few new postgres-full failires in last 24 hours | 20:06 |
| jogo | will dig into more after lunch | 20:07 |
| *** bhuvan__ has joined #openstack-infra | 20:08 | |
| *** bhuvan___ has joined #openstack-infra | 20:08 | |
| *** zzelle_ has joined #openstack-infra | 20:08 | |
| *** bhuvan_ has quit IRC | 20:08 | |
| *** bhuvan has quit IRC | 20:08 | |
| *** Sukhdev has joined #openstack-infra | 20:09 | |
| *** thuc has joined #openstack-infra | 20:09 | |
| *** emagana has joined #openstack-infra | 20:10 | |
| *** thomasem has joined #openstack-infra | 20:13 | |
| *** jamielennox is now known as jamielennox|away | 20:15 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed a change to openstack-infra/gerritbot: Add support for notification of releases on IRC https://review.openstack.org/79205 | 20:16 |
| *** afazekas_wfp has joined #openstack-infra | 20:17 | |
| *** yolanda has quit IRC | 20:19 | |
| fungi | Alex_Gaynor: i don't suppose you happen to be around for a perplexing pypy question? | 20:22 |
| Alex_Gaynor | fungi: I am indeed around | 20:22 |
| *** mbacchi has quit IRC | 20:23 | |
| fungi | Alex_Gaynor: submitted for your consideration, to seemingly-enough similar ubuntu virtual machines... | 20:24 |
| fungi | Alex_Gaynor: this one behaves as expected... http://paste.openstack.org/show/73058/ | 20:24 |
| fungi | Alex_Gaynor: while this one has apparently been sent out into the cornfield... http://paste.openstack.org/show/73060/ | 20:25 |
| *** malini_afk is now known as malini | 20:25 | |
| fungi | Alex_Gaynor: what particular differences on those systems should i be looking for to help pinpoint the cause? | 20:25 |
| Alex_Gaynor | fungi: I... do not... know... dstufft ? | 20:26 |
| fungi | Alex_Gaynor: agreed. this could be virtualenv weeds | 20:26 |
| fungi | Alex_Gaynor: on that second system, the same steps with -p python instead of -p pypy exhibit no problem | 20:27 |
| fungi | and also -p python3.3 | 20:28 |
| *** bhuvan___ has quit IRC | 20:29 | |
| *** bhuvan__ has quit IRC | 20:29 | |
| fungi | it looks like some sort of order-of-execition bug which crops up when upgrading setuptools and something else in the same pip command. installing these individually in either order works fine | 20:31 |
| *** ildikov_ has joined #openstack-infra | 20:31 | |
| *** mrda_away is now known as mrda | 20:31 | |
| Alex_Gaynor | fungi: Yeah, that super sounds like a pip bug that's incidentally triggered by pypy on some archs because our hashes are different... or something. | 20:32 |
| Alex_Gaynor | fungi: definitely sounds like it's dstufft's problem | 20:32 |
| * clarkb is back from lunch | 20:32 | |
| sdague | clarkb: you got time to talk about logstash fields? | 20:32 |
| clarkb | sdague: sure | 20:32 |
| clarkb | I need a break from zuul merger stuff anyways | 20:32 |
| sdague | cool | 20:32 |
| clarkb | sdague: by the way you will love aggregators in es 1.0 | 20:33 |
| *** jlibosva has joined #openstack-infra | 20:33 | |
| clarkb | it basically allows you to do what you are doing by hand with facets aiui | 20:33 |
| sdague | so I'd like to be able to get us away from wildcards, because I think they are part of the performance issues | 20:33 |
| clarkb | sdague: agree | 20:33 |
| *** sarob__ has joined #openstack-infra | 20:34 | |
| clarkb | wildcards are expensive | 20:34 |
| sdague | which means we need to tag or field better | 20:34 |
| clarkb | which means fixing the grenade logs | 20:34 |
| sdague | sure, or coming up with another field there | 20:34 |
| clarkb | sdague: oh you know I could have the tag be basename(filepath) | 20:34 |
| clarkb | then filter on tags | 20:34 |
| sdague | yeh, that would be good | 20:34 |
| sdague | honestly, it might be handy to have a tag for "console" in general | 20:35 |
| sdague | now that the console is in a bunch of different logs | 20:35 |
| clarkb | sdague: distinct from console.html? ya | 20:35 |
| sdague | yeh | 20:35 |
| sdague | that includes all the various console logs | 20:35 |
| clarkb | yup we could do that too | 20:35 |
| sdague | because sometimes we want all of them | 20:35 |
| *** ociuhandu has joined #openstack-infra | 20:35 | |
| sdague | and some times we want to be specific | 20:35 |
| clarkb | basically console would be like oslofmt | 20:35 |
| clarkb | and we could still have the filename tag | 20:36 |
| hashar | good evening ::] | 20:36 |
| sdague | yep | 20:36 |
| clarkb | whcih should be basename(filename) then we can filter on tag isntead of filename | 20:36 |
| clarkb | sdague: the other field we wildcard on is build uuid | 20:36 |
| sdague | where do we wildcard that? | 20:36 |
| clarkb | do we need a short_uuid field? | 20:36 |
| *** sarob_ has quit IRC | 20:36 | |
| clarkb | sdague: in e-r where it finds files for a job | 20:36 |
| clarkb | build uuid is the only absolute way to match jobs but gerrit comments don't have the full uuid | 20:37 |
| hashar | jeblair: thank you to have looked at Zuul race condition when abandoning patches :-] | 20:37 |
| sdague | clarkb: right so short_uuid field would be good | 20:37 |
| hashar | jeblair: drove me mad yesterday and I eventually gave up :-( | 20:37 |
| *** julim_ has quit IRC | 20:38 | |
| sdague | clarkb: so this is today - http://paste.openstack.org/show/73064/ | 20:38 |
| *** mburned is now known as mburned_out | 20:38 | |
| sdague | providing a basename(filename) tag would drop a bunch of that | 20:38 |
| *** sarob__ has quit IRC | 20:38 | |
| clarkb | yup | 20:39 |
| jeblair | hashar: you're welcome; i love it when we actually get logs from a failed test with a race condition, when that happens we can always find the cause | 20:39 |
| sdague | I'm trying to figure out if we have a better plan in build_name | 20:39 |
| sdague | maybe we can think about that as well | 20:39 |
| sdague | though the filename thing I think will help a ton | 20:39 |
| hashar | jeblair: joshua hinted me about the race, but I could not diagnose it out of the debug log :/ | 20:39 |
| jeblair | hashar: there are some other race conditions that we can't track down because we can't figure out how to get logs from tests that time out | 20:40 |
| sdague | clarkb: one other thing I'd like, is if we could compute out cloud as a field instead of just build_host | 20:40 |
| jeblair | clarkb: i'm not excited about stack dumps at exit | 20:40 |
| sdague | just to make the baked in term statistics more straight forward | 20:41 |
| clarkb | jeblair: ok | 20:41 |
| *** rpodolyaka has joined #openstack-infra | 20:41 | |
| hashar | jeblair: doesn't Zuul react to some signal to generate a thread dump? Maybe when Jenkins terminate the process you could intercept that and attempt to generate a thread dump. Not sure it would help though. | 20:41 |
| jeblair | hashar: oh, the thread dumps are different -- not related to the timed out tests; clarkb is trying to diagnose what's happening to the zuul merger process (it's pidfile is deleted after a while on one of his hosts) | 20:42 |
| jeblair | hashar: the problem with test timeouts is related to testtools where if the test timeout fixture fires, it kills the process and testtools never gets the logs | 20:43 |
| *** alexpilotti has joined #openstack-infra | 20:44 | |
| jeblair | i've been running one of the jobs tha fails intermittently in a loop all morning, and it still hasn't failed | 20:44 |
| hashar | :-((( | 20:44 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: neutron fails unit tests in the gate https://review.openstack.org/79444 | 20:44 |
| *** bhuvan_ has joined #openstack-infra | 20:44 | |
| *** bhuvan has joined #openstack-infra | 20:44 | |
| hashar | jeblair: regarding the race condition I am facing while testing abandoned change would it help to add a sleep() somewhere ? | 20:45 |
| jeblair | hashar: i think the change i pushed below your change should fix it | 20:45 |
| hashar | jeblair: to let sometime for gearman to process the packet and start the build so it can be properly abandoned ? | 20:45 |
| *** jeckersb is now known as jeckersb_gone | 20:45 | |
| *** luisg has quit IRC | 20:45 | |
| hashar | jeblair: I am asking because the py26 test failed, might be a different reason though ( http://logs.openstack.org/67/65467/8/check/gate-zuul-python26/3f775cd/testr_results.html.gz ) | 20:46 |
| clarkb | sdague: for the node stuff, do you want image label too or just AZ? | 20:46 |
| *** rpodolyaka has quit IRC | 20:46 | |
| clarkb | sdague: eg bare-precise-hpcloud-az1 or hpcloud-az1? | 20:46 |
| jeblair | clarkb: 2 fields? | 20:46 |
| sdague | clarkb: 2 fields might be best | 20:46 |
| *** rpodolyaka has joined #openstack-infra | 20:47 | |
| jeblair | that way we can say "this only happens on precise/14.04" or "this only happens in rax" or "this only happens on precise in rax" | 20:47 |
| clarkb | that works, in that case build_label:bare-precise and build_az:hpcloud-az1 is what I will go with | 20:47 |
| sdague | sure | 20:47 |
| sdague | build_image ? | 20:47 |
| jeblair | clarkb: s/az/provider/ to match nodepool terms? | 20:47 |
| clarkb | jeblair: sure | 20:47 |
| sdague | build_label I think doesn't have a lot of meaning | 20:48 |
| clarkb | sdague: wfm | 20:48 |
| jeblair | sdague, clarkb: and yeah build_image matches nodepool :) | 20:48 |
| sdague | +1 | 20:48 |
| sdague | that will be great | 20:48 |
| jeblair | hashar: ok, i'll look at that failure in a bit. apparently, that test i had running in a loop just hung! | 20:49 |
| clarkb | jeblair: sdague: can we go with III-III-PPP-PPP-DIGITS as a generic name format? | 20:49 |
| clarkb | for our disposable slaves we seem to do that today, trying to figure out how I can infer with a high degree of accuracy | 20:49 |
| jeblair | clarkb: hrm. we should have made the separator different between those fields | 20:50 |
| clarkb | ++ not sure if we can do that now though | 20:50 |
| jeblair | clarkb: i was kinda planning on making bare-precise just be 'precise' at some point. | 20:50 |
| *** lukego has joined #openstack-infra | 20:50 | |
| fungi | clarkb: jeblair how about provider-zone-flavor-distro-nodeid | 20:50 |
| clarkb | hrm, I could split it, then work backwards | 20:50 |
| fungi | s/flavor/variant/ (don't want to confuse openstack terms) | 20:51 |
| clarkb | oh wait no because tripleo | 20:51 |
| clarkb | tripleo az is triplep-test-cloud ... | 20:51 |
| jeblair | we could rename it 'tripleo-actual' | 20:51 |
| sdague | if we are redoing taxonomy - provider-zone-distro-flavor-nodeid | 20:51 |
| sdague | would seem to go from big to small | 20:52 |
| jeblair | we don't have a lot of separator characters available in dns | 20:52 |
| fungi | so rename tripleo-test-cloud tripleo-whateverzone | 20:52 |
| sdague | yeh, just drop the extra - | 20:52 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add #openstack-sahara channel to accessbot https://review.openstack.org/79446 | 20:52 |
| fungi | and then settle on that as an emergent descriptor | 20:53 |
| sdague | tripleo-testcloud would be fine | 20:53 |
| * fungi is strongly considering making py3k-precise nodes only in hpcloud until i can work out the pypy/pip/setuptools weirdness on the rax nodes | 20:54 | |
| *** JuanManuelOlle has quit IRC | 20:54 | |
| *** pete5 has joined #openstack-infra | 20:56 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Better logstash field data. https://review.openstack.org/79447 | 20:56 |
| clarkb | sdague: ^ is the first bit | 20:56 |
| clarkb | fungi: I think that is a not terrible idea | 20:56 |
| jeblair | clarkb: what's the short uuid for? | 20:57 |
| sdague | matching exact builds to reviews | 20:57 |
| clarkb | jeblair: for filtering on the uuid handed back to gerrit from zuul | 20:57 |
| clarkb | jeblair: right now we do shortuuid* which is a search and more expensive than an exact filter | 20:58 |
| sdague | we actually had issues of overmatching before with er because we'd match by review id | 20:58 |
| sdague | but if someone did a recheck | 20:58 |
| sdague | then we were actually matching against multiple runs | 20:58 |
| clarkb | right change id + patchset was not sufficient | 20:58 |
| sdague | or even check + gate runs | 20:58 |
| *** jlibosva has quit IRC | 20:59 | |
| jeblair | clarkb: ah, i see, this is for the e-r bot (and occasional human use when looking for entries from a specific run when you find that run in the review comments) | 20:59 |
| sdague | clarkb: actually, until january were weren't even matching by patchset | 20:59 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Fixed padding on modals. https://review.openstack.org/79393 | 20:59 |
| clarkb | jeblair: yup | 20:59 |
| sdague | so it was even more awesome overmatching | 20:59 |
| *** ArxCruz has quit IRC | 21:00 | |
| sdague | jeblair: right, and this is actually a *very* frequent query for the bot | 21:00 |
| jeblair | cool, understand and lgtm | 21:00 |
| clarkb | I am going to let the provider/image stuff stew a little longer | 21:00 |
| *** weiwei has joined #openstack-infra | 21:00 | |
| weiwei | hi, I've been pointed here looking for someone to help with ssh key problem | 21:01 |
| *** jooools has quit IRC | 21:01 | |
| clarkb | weiwei: go for it | 21:01 |
| * clarkb can help | 21:01 | |
| weiwei | git review -s gives me permission denied | 21:01 |
| *** smarcet has left #openstack-infra | 21:01 | |
| weiwei | but we can't work out what's wrong | 21:01 |
| clarkb | weiwei: can you paste the output from git review -s to paste.openstack.org or similar? | 21:02 |
| weiwei | http://paste.openstack.org/show/73066/ | 21:02 |
| weiwei | clarkb, getting there :) | 21:02 |
| *** vkozhukalov has joined #openstack-infra | 21:02 | |
| weiwei | Launchpad Id: dob12460 | 21:02 |
| clarkb | weiwei: if you go to https://review.openstack.org/#/settings/ does dob12460 match your username there? | 21:03 |
| weiwei | keyname id_rsa.pub | 21:03 |
| krotscheck | clarkb: You keep mentioning how I can just go directly into the storyboard database to look around. How would I go about doing that? | 21:03 |
| clarkb | krotscheck: well I thought you had access to the node but I may be wrong about that | 21:03 |
| krotscheck | clarkb: I don't. and jeblair -1'd my patch. | 21:03 |
| weiwei | clarkb, this is what I'm wondering... | 21:04 |
| *** luisg has joined #openstack-infra | 21:04 | |
| weiwei | username = davidobrien but launchpad id = dob12460 | 21:04 |
| clarkb | weiwei: can you check? | 21:04 |
| *** mfer has quit IRC | 21:04 | |
| *** dcramer__ has quit IRC | 21:04 | |
| clarkb | ah oh | 21:04 |
| clarkb | weiwei: then you just need to ssh as davidobrien | 21:04 |
| weiwei | clarkb, how do I tell git to do that? | 21:05 |
| *** bhuvan_ has quit IRC | 21:05 | |
| *** bhuvan has quit IRC | 21:05 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add query for glance failing it's unit tests https://review.openstack.org/79450 | 21:05 |
| Alex_Gaynor | Can someone explain how I'm supposed to debug this: https://jenkins02.openstack.org/job/gate-python-openstacksdk-python27/9/console ? it seems to be showing me raw subunit stuff instead of nice formatted tracebacks | 21:05 |
| clarkb | weiwei: `git config --global gitreview.username davidobrien` | 21:05 |
| clarkb | weiwei: that will set the gitreview option globally on your machine | 21:05 |
| weiwei | ah | 21:05 |
| weiwei | didn't know that one | 21:06 |
| weiwei | looked at others :( | 21:06 |
| weiwei | let me try... | 21:06 |
| clarkb | where global is for your user | 21:06 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: neutron fails unit tests in the gate https://review.openstack.org/79444 | 21:06 |
| clarkb | Alex_Gaynor: import errors8openstack.tests.test_auth is the important bit, there was an import error | 21:06 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Temporarily stop using rax for py3k-precise nodes https://review.openstack.org/79451 | 21:06 |
| *** amcrn has quit IRC | 21:06 | |
| *** packet has quit IRC | 21:06 | |
| clarkb | Alex_Gaynor: lifeless has code to make that not suck in the testr UI but he hasn't cut a release yet | 21:07 |
| *** pete5 has quit IRC | 21:07 | |
| Alex_Gaynor | clarkb: but I can't see what it actually failed at importing :/ | 21:07 |
| clarkb | Alex_Gaynor: so you can install from source or beg lifeless for a release :) | 21:07 |
| *** pete5 has joined #openstack-infra | 21:07 | |
| Alex_Gaynor | lifeless: please release! :-) | 21:07 |
| Alex_Gaynor | clarkb: I guess it's probably some dep I didn't put in the right place | 21:07 |
| *** pete5 has quit IRC | 21:07 | |
| *** pete5 has joined #openstack-infra | 21:08 | |
| clarkb | Alex_Gaynor: I would have to fetch the code and try importing it by hand | 21:08 |
| Alex_Gaynor | clarkb: It works locally :/ I guess it's somethign I don't have in requirements.tt | 21:08 |
| clarkb | Alex_Gaynor: use the tox luke | 21:09 |
| mattoliverau | Morning everyone, I'm back from the rackspace conference in the US, so back to regular duties! I hope everyone is well :) | 21:09 |
| *** afazekas_wfp has quit IRC | 21:09 | |
| Alex_Gaynor | clarkb: I get the same useless message out of tox locally | 21:09 |
| krotscheck | clarkb, jeblair: For the sake of sanity, can one of you shell into the storyboard.o.o DB and set is_active to 1 on tasks, stories, and projects? | 21:09 |
| weiwei | clarkb, not working. same error. Still trying to connect as dob12460 | 21:09 |
| * krotscheck is still grokking alembic data migrations and forgot to set a default. | 21:09 | |
| clarkb | Alex_Gaynor: yeah so now, .tox/py27/bin/python for file in files: import file | 21:09 |
| weiwei | scp -P 29418 dob12460@review.openstack.org:hooks/commit-msg .git/hooks/commit-msg | 21:10 |
| jeblair | mattoliverau: welcome back! | 21:10 |
| Alex_Gaynor | clarkb: ah, clever | 21:10 |
| jeblair | mattoliverau: if you have a moment, i don't think the cacti redirect worked as expected | 21:10 |
| weiwei | clarkb, $ git config --global gitreview.username davidobrien | 21:10 |
| weiwei | yeah? | 21:10 |
| clarkb | weiwei: yeah | 21:10 |
| mattoliverau | jeblair: thanks! | 21:10 |
| Alex_Gaynor | clarkb: thanks | 21:10 |
| mattoliverau | jeblair: I'll take a look | 21:10 |
| clarkb | weiwei: you might also need to `git remote rm gerrit` in that repo and rerun git review -s | 21:11 |
| clarkb | weiwei: depending on how far along git review -s got | 21:11 |
| jeblair | krotscheck: can you make another migration to fix it? | 21:11 |
| weiwei | clarkb, yeah I'm looking at the first part of the errors... | 21:11 |
| weiwei | Problem running 'git remote update gerrit' | 21:11 |
| *** packet has joined #openstack-infra | 21:11 | |
| *** amcrn has joined #openstack-infra | 21:12 | |
| *** packet has quit IRC | 21:12 | |
| *** Ajaeger1 has joined #openstack-infra | 21:12 | |
| weiwei | clarkb, ok, that's making more sense | 21:12 |
| krotscheck | jeblair: Are you going to make me do that? | 21:12 |
| Ajaeger1 | Hi, I'd apprecate if fungi's patch for https://review.openstack.org/#/c/79323/ gets reviewed and approved to fix openstack-manual translation jobs. Anybody willing to review, please? | 21:13 |
| clarkb | Ajaeger1: looking | 21:14 |
| weiwei | clarkb, so now git review -s doesn't actually return anything, but it doesn't appear to be failing | 21:14 |
| Ajaeger1 | clarkb: thanks! | 21:14 |
| jeblair | krotscheck: i think the idea was to run a continuously deployable system; this error seems to be the result of incorrect code that's in the tree, so shouldn't the fix be there? | 21:15 |
| clarkb | weiwei: right it should be silent if all goes well. you can `git remote -v` and see if the gerrit remote looks correct | 21:15 |
| Ajaeger1 | clarkb: that was fast - thanks a lot. | 21:15 |
| *** mburned_out is now known as mburned | 21:15 | |
| clarkb | Ajaeger1: it is nice and simple | 21:15 |
| Ajaeger1 | clarkb: kudos to fungi for finding and fixing ;) | 21:15 |
| jeblair | krotscheck: (if, for some reason, we accidentally set all the tasks to deleted by running a command, i would have no issues with logging into the server to fix it) | 21:15 |
| weiwei | clarkb, I have fetch and push for gerrit and origin | 21:16 |
| weiwei | no errors | 21:16 |
| clarkb | weiwei: I think you are set | 21:16 |
| jeblair | krotscheck: (but this seems like a problem that _should_ be fixed in the code. even if there are no other known production storyboard deployments, this is the sort of thing we would need to do to support them, so it's worth practicing, right?) | 21:16 |
| weiwei | clarkb, so hopefully I can move on to step 19: git checkout master; git remote update; git pull origin master | 21:16 |
| * weiwei follows steps for new contributor... | 21:17 | |
| openstackgerrit | A change was merged to openstack-infra/config: Explicitly install gettext on job workers https://review.openstack.org/79323 | 21:17 |
| weiwei | clarkb, thanks a lot | 21:17 |
| weiwei | clarkb++ | 21:17 |
| clarkb | weiwei: no problem | 21:17 |
| *** jamielennox|away is now known as jamielennox | 21:17 | |
| weiwei | eh? no karma bot? | 21:17 |
| weiwei | :( | 21:17 |
| krotscheck | jeblair: it's really more of a question of "How soon do you want to dogfood". I can write a patch easily enough, but the average review turnaorund time on storyboard these days is a week. | 21:19 |
| krotscheck | jeblair: I'll go ahead and write a patch then. | 21:19 |
| krotscheck | jeblair: turns out I have to remove a unique name constraint anyway. | 21:19 |
| *** lcostantino has quit IRC | 21:20 | |
| weiwei | Ajaeger1, ssh issue fixed | 21:21 |
| weiwei | code review username didn't match | 21:21 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: Revert "Switch to SQLAlchemy 0.9" https://review.openstack.org/79454 | 21:22 |
| *** dstufft has quit IRC | 21:22 | |
| *** dstufft has joined #openstack-infra | 21:22 | |
| Ajaeger1 | weiwei: Ah! Great you found some experts to help solving it! | 21:23 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add query for glance failing it's unit tests https://review.openstack.org/79450 | 21:23 |
| weiwei | yes, glad to get bumps fixed so quickly | 21:23 |
| weiwei | thanks group :) | 21:23 |
| clarkb | dims_: how is cloud archive going? | 21:24 |
| *** vponomaryov has left #openstack-infra | 21:24 | |
| clarkb | jeblair: in https://review.openstack.org/#/c/79385/ are you ok with the code that puts change,patchset in the merger logs? I think that is independently useful, and I will hunt down the other stuff separately | 21:25 |
| sdague | so it looks like sqla 0.9 probably doesn't actually work | 21:26 |
| clarkb | sdague: :( | 21:26 |
| sdague | I put a fast revert in the requirements, though it will take a while to land | 21:26 |
| openstackgerrit | A change was merged to openstack/requirements: Adding support for Django 1.6 https://review.openstack.org/77015 | 21:27 |
| *** rfolco has quit IRC | 21:28 | |
| jeblair | clarkb: this doesn't make sense to me: http://paste.openstack.org/show/73074/ | 21:29 |
| *** bhuvan has joined #openstack-infra | 21:31 | |
| jeblair | clarkb: whis is "UNKNOWN" showing up there? why is "gerrit" showing up in the egg name? why are there two entries? where are all the other dependencies that should have been installed? | 21:31 |
| *** bhuvan_ has joined #openstack-infra | 21:31 | |
| *** jhesketh has joined #openstack-infra | 21:31 | |
| clarkb | jeblair: pbr is failing to give you a name + version | 21:31 |
| clarkb | I don't know why gerrit shows up in the egg name | 21:31 |
| jhesketh | Morning | 21:31 |
| clarkb | there are two entries because of that UNKNOWN thing | 21:32 |
| clarkb | this is weird /me tries to reprocude | 21:32 |
| jeblair | jhesketh: good morning; i looked at your zuul patch, i'm going to try to get to the os-loganalyze patch | 21:32 |
| jeblair | jhesketh: but i've been pleasantly distracted by finally making some headway on the nondeterministic zuul test failures | 21:32 |
| *** dangers is now known as dangers_away | 21:33 | |
| jhesketh | jeblair: great, thanks | 21:33 |
| *** spredzy has quit IRC | 21:33 | |
| *** spredzy has joined #openstack-infra | 21:34 | |
| jeblair | clarkb: i did "rm -fr zuul.egg-info UNKNOWN.egg-info" and things seem to be moving | 21:35 |
| clarkb | jeblair: interesting, I couldn't reproduce with pip 1.5.4 in virtualenv 1.11.4 | 21:35 |
| clarkb | whcih are newer than we run in the test slaves | 21:35 |
| jogo | jhesketh: so turbo hipster | 21:35 |
| jhesketh | jogo: yeah, just noticed it's been failing all night | 21:36 |
| jeblair | clarkb: yeah, one or both of those egg-info dirs must have been bogus from some prior action and caused the problem. | 21:36 |
| jhesketh | I think something might have change in nova that's causing one of our datasets to fail | 21:36 |
| *** CaptTofu_ has quit IRC | 21:36 | |
| jogo | you have one grumpy turbo hipster ... his favorite coffeee shoop most have closed for renovations | 21:37 |
| jhesketh | seems that way! | 21:37 |
| jogo | jhesketh: assuming it is nova broke things, lets find out how and revert | 21:37 |
| jogo | slash fix | 21:37 |
| jhesketh | well not necessarily nova's fault | 21:38 |
| jhesketh | but it looks like it's just one dataset that is failing so it isn't moving up through the upgrade process correctly | 21:38 |
| *** bhuvan_ has quit IRC | 21:40 | |
| *** bhuvan has quit IRC | 21:40 | |
| jogo | jhesketh: anyway if it is nova let me know and I can help shepard a fix | 21:41 |
| jhesketh | thanks jogo, will do | 21:41 |
| jhesketh | for now I'll put that dataset in non-voting | 21:41 |
| jeblair | clarkb: okay, here's another weird pbr thing: http://paste.openstack.org/show/73076/ | 21:42 |
| clarkb | jeblair: looking | 21:43 |
| jeblair | clarkb: pbr is a zuul requirement, and is apparently installed in such a way that it can be used to install zuul, but is not actually available for import | 21:43 |
| *** Ajaeger1 has quit IRC | 21:43 | |
| clarkb | jeblair: huh, I wonder if that is because it is a setup requires | 21:44 |
| mordred | jeblair: yup. that's why we reworked the tox.ini files in teh rest of openstack | 21:44 |
| clarkb | jeblair: is pbr also in requirements.txt? | 21:44 |
| clarkb | mordred: ? | 21:44 |
| jeblair | clarkb: yeah, it's at the top of that | 21:44 |
| mordred | there is a weird interaction between setup_Requires and pip-requires which can leave you missing anything that was in setup_requires from your installed requirements | 21:44 |
| mordred | it's nota pbpr specific issue | 21:44 |
| mordred | Requirement already satisfied (use --upgrade to upgrade): pbr>=0.5.21,<1.0 in ./pbr-0.6-py2.7.egg (from zuul==2.0.0.81.gac6c77d) | 21:45 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add more logging to zuul merger process https://review.openstack.org/79385 | 21:45 |
| jeblair | mordred: so, what's the solution? | 21:45 |
| mordred | means it's finding the egg that setup_Requires pulls locally, so it's not finding it | 21:45 |
| clarkb | jeblair: ^ I am happier with that patchset, will work on the dying stuff differently | 21:45 |
| mordred | s/finding/instaslling/ | 21:45 |
| mordred | but that egg is not installed | 21:45 |
| clarkb | mordred: so what do we do to fix it? | 21:46 |
| mordred | clarkb: thinking of how to fix in this context in the real way | 21:47 |
| mordred | we fix in tox by pip installing requirements first before we pip install . | 21:47 |
| mordred | but - wierdly - I don't thin kthis is an issue outside of git repos | 21:47 |
| * mordred goes to test real quick | 21:47 | |
| clarkb | mordred: gotcha thanks | 21:47 |
| mordred | clarkb: do you need short-term workaroudn? | 21:48 |
| clarkb | mordred: jeblair might | 21:48 |
| jeblair | mordred: i think "pip install this thing from the git repo i have just checked out" is something we are trying to support | 21:48 |
| fungi | clarkb: mordred: jeblair: ano objections to merging https://review.openstack.org/79451 while i work through https://launchpad.net/bugs/1290562 ? | 21:48 |
| jeblair | mordred: it's kindof the key to CDing our projects | 21:48 |
| mordred | yup | 21:48 |
| mordred | I agre | 21:48 |
| fungi | er, any | 21:48 |
| *** sarob_ has joined #openstack-infra | 21:48 | |
| clarkb | fungi: looking | 21:48 |
| clarkb | fungi: wfm | 21:49 |
| jeblair | mordred: ok, cool, just wanted to make sure there wasn't like "don't do this in a git repo" subtext to the "not an issue outside of git repos" comment. | 21:49 |
| mordred | nope | 21:49 |
| jeblair | mordred: for the moment, i will just rm the egg and pip install pbr in the venv to continue my testing | 21:49 |
| mordred | ++ | 21:49 |
| mordred | jeblair: I've had discussions with dstufft before about trying to track down why this happens | 21:50 |
| dstufft | what's up | 21:50 |
| fungi | dstufft: https://github.com/pypa/pip/issues/1632 | 21:50 |
| dstufft | oh is this related to the ticket from Jeremy Stanley? I forget whose username that is heh | 21:50 |
| *** torandu has left #openstack-infra | 21:50 | |
| jeblair | dstufft: fungi == Jeremy | 21:51 |
| fungi | dstufft: weird. github tells me "fungi" authored that ;) | 21:51 |
| dstufft | ah yea, looking at it on the web tells me that | 21:51 |
| dstufft | fungi: email doesn't :( | 21:51 |
| fungi | github is clearly broken | 21:51 |
| dstufft | http://d.stufft.io/image/3a2E001e362K | 21:51 |
| fungi | that's wacku | 21:51 |
| fungi | wacky | 21:51 |
| dstufft | is the failing service provider Rackspace by any chance | 21:52 |
| fungi | dstufft: mmm... how about we go with "yes" on that one | 21:52 |
| dstufft | ok, that makes it easier for me to spin up a server and see if I can reproduce | 21:52 |
| fungi | dstufft: the failure demonstration is http://paste.openstack.org/show/73060/ (though we jump through a couple hoops to install python3.3 and pypy backports from ppas on those) | 21:53 |
| dstufft | I have a standup in a few minutes for python-infra, I'll see if I can take a poke after that | 21:53 |
| fungi | dstufft: i can also get you an ssh key on an affected virtual machine after i get back from dinner | 21:54 |
| dstufft | fungi: ok | 21:54 |
| mordred | jeblair: I just tested locally and wound up with a fully working install ... I think you may have just had some cruft from something failed before | 21:54 |
| fungi | dstufft: these are throwaway servers anyway, so we don't reuse them... i've just held one to prevent it from being immediately deleted | 21:54 |
| jeblair | mordred: nope, i removed cruft | 21:55 |
| mordred | yeah/ and it still broke for you again? | 21:55 |
| jeblair | mordred: you could 'python -c "import pbr"'? | 21:55 |
| mordred | yes | 21:55 |
| dstufft | fungi: ok | 21:55 |
| dstufft | fungi: well whenever, my key is on https://github.com/dstufft.keys | 21:55 |
| fungi | dstufft: thanks! i'll ping you when i get back in a little while with the ip address | 21:56 |
| jeblair | mordred: i think i have a different repro sequence for that, hang on a sec. | 21:57 |
| mordred | jeblair: awesome | 21:57 |
| * fungi disappears for a while. fooooooood | 21:57 | |
| *** vkozhukalov has quit IRC | 21:58 | |
| *** dims_ has quit IRC | 21:58 | |
| openstackgerrit | A change was merged to openstack-infra/config: Temporarily stop using rax for py3k-precise nodes https://review.openstack.org/79451 | 21:58 |
| jeblair | mordred: nope, you're right, it's not reproducible. i think there may have been a pbr0.5.x egg in my gear directory | 21:59 |
| jeblair | which was later removed | 21:59 |
| jeblair | when i was trying to figure out why it wasn't working | 21:59 |
| *** jeckersb_gone is now known as jeckersb | 22:00 | |
| jeblair | mordred: fwiw, gear and zuul both have a >=6.0 requirement, so i wouldn't expect that to be an issue | 22:00 |
| jeblair | but i have no idea what's going on so... | 22:00 |
| jeblair | mordred: thanks for looking into it anyway | 22:00 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: WIP : Try Icehouse precise proposed UCA - Not for review https://review.openstack.org/74889 | 22:01 |
| *** nati_uen_ has joined #openstack-infra | 22:02 | |
| *** nati_uen_ has quit IRC | 22:02 | |
| *** nati_uen_ has joined #openstack-infra | 22:02 | |
| *** lukego has quit IRC | 22:02 | |
| malini | hello..one of my jobs is running into an 'out of disk space' error . http://logs.openstack.org/49/63449/21/experimental/check-tempest-dsvm-marconi/046a0a6/logs/screen-marconi-server.txt.gz Any pointers to what could be going on ? | 22:03 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Be more careful with locks https://review.openstack.org/79469 | 22:03 |
| jeblair | hashar, jhesketh, clarkb: ^ that is one part of a fix to one of the nondeterministic zuul failures | 22:04 |
| *** nati_ueno has quit IRC | 22:05 | |
| *** dkranz has quit IRC | 22:05 | |
| * hashar takes a 101 course on threading and locking | 22:06 | |
| *** thomasem has quit IRC | 22:06 | |
| *** vkozhukalov has joined #openstack-infra | 22:06 | |
| hashar | jeblair: sorry that is way over my skills :-( | 22:06 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Set the correct default for is_active. https://review.openstack.org/79471 | 22:07 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Shut down the gearman server last in tests https://review.openstack.org/79472 | 22:07 |
| jogo | sdague: https://review.openstack.org/79431 haha | 22:07 |
| jogo | ^ bad | 22:07 |
| jeblair | hashar, jhesketh, clarkb: https://review.openstack.org/79472 is what was triggering the error in zuul | 22:07 |
| clarkb | jeblair: looking | 22:07 |
| hashar | ahhh | 22:08 |
| clarkb | I am finally writing tests for the git replica thing, more complicated than I expected | 22:08 |
| hashar | jeblair: yeah I got some stacktrace locally because the test suite can no more connect to the gear server. I thought it was my setup :] | 22:08 |
| jogo | sdague: so I think new migrate is broken http://logs.openstack.org/31/79431/1/check/gate-tempest-dsvm-large-ops/36c4203/logs/devstacklog.txt.gz | 22:08 |
| *** Sukhdev has quit IRC | 22:08 | |
| jeblair | hashar: it normally gets a little hairy at the end and prints tracebacks -- it's only an issue if it hangs | 22:09 |
| jogo | being things just hang | 22:09 |
| sdague | jogo: no, it's a stdout buffering problem | 22:09 |
| jeblair | hashar: if you ever see it hang, save the logs! :) | 22:09 |
| sdague | it's ceilometer | 22:09 |
| *** thuc has quit IRC | 22:09 | |
| sdague | ceilometer is doing something slightly bizarre to self migrate from migrate -> alembic | 22:10 |
| *** andreaf has joined #openstack-infra | 22:10 | |
| *** thuc has joined #openstack-infra | 22:10 | |
| sdague | and it depends on sqla behavior in 0.8 | 22:10 |
| jogo | ahhh | 22:10 |
| jogo | how did you figure that out? any other logs? | 22:10 |
| sdague | local reproduce | 22:10 |
| clarkb | jogo: there is a gear change related to 79472, it puts a sleep between reconnect attempts | 22:10 |
| clarkb | jeblair: ^ whoops | 22:10 |
| clarkb | jeblair: the change has my +2 | 22:11 |
| jogo | sdague: ahh | 22:11 |
| jeblair | clarkb: can you link to it in your review of 79472? | 22:11 |
| clarkb | jeblair: sure | 22:11 |
| hashar | 2014-03-10 23:11:24,604 gear.Worker.Zuul RPC Listener DEBUG Unable to connect to <gear.Connection 0x10b9ec710 host: 127.0.0.1 port: 50659> | 22:11 |
| hashar | that is what I am getting | 22:11 |
| clarkb | jeblair: done | 22:12 |
| jeblair | hashar: does it hang? | 22:12 |
| *** pdmars has quit IRC | 22:12 | |
| hashar | nop | 22:12 |
| jeblair | hashar: then it's probably fine | 22:12 |
| hashar | huge paste http://paste.openstack.org/show/73081/ | 22:12 |
| hashar | using my abandoning patch | 22:13 |
| hashar | ohh and the assertion failed though I tried to reproduce it for a few hours on sunday :/ | 22:13 |
| jeblair | hashar: yeah, i'm about to look at that again | 22:14 |
| jeblair | hashar: since i thought my patch should have prevented it | 22:14 |
| hashar | jeblair: and I thought abandoning a patch automatically would be an easy change... =) | 22:14 |
| *** thuc has quit IRC | 22:14 | |
| jhesketh | jeblair: will take a look shortly | 22:14 |
| *** ianw has quit IRC | 22:14 | |
| *** ianw has joined #openstack-infra | 22:15 | |
| jeblair | jhesketh: no rush on that | 22:15 |
| *** dims_ has joined #openstack-infra | 22:15 | |
| jeblair | clarkb: i wonder what exception was causing https://review.openstack.org/#/c/76589 ? | 22:15 |
| *** aysyd has quit IRC | 22:16 | |
| clarkb | jeblair: I think it may be networking going hard down | 22:16 |
| clarkb | whcih may not actually have helped you above | 22:17 |
| jeblair | clarkb: i doubt it; just so we're clear, derekh's patch is to the server, so it's not a reconnect attempt, it's a "start trying to accept incoming sockets again" attempt | 22:17 |
| *** SumitNaiksatam has quit IRC | 22:17 | |
| clarkb | oh | 22:18 |
| *** julim has joined #openstack-infra | 22:18 | |
| *** e0ne has quit IRC | 22:18 | |
| *** rfolco has joined #openstack-infra | 22:24 | |
| *** spm has joined #openstack-infra | 22:24 | |
| *** spm has left #openstack-infra | 22:24 | |
| *** jgrimm has quit IRC | 22:25 | |
| *** rlandy has quit IRC | 22:25 | |
| *** mburned is now known as mburned_out | 22:26 | |
| *** bhuvan_ has joined #openstack-infra | 22:26 | |
| *** bhuvan has joined #openstack-infra | 22:26 | |
| *** jungleboyj has quit IRC | 22:27 | |
| jogo | just wrote my first rpython program | 22:27 |
| mikal | Is anyone else getting "connection reset by peer" from pypi.python.org? | 22:30 |
| SergeyLukjanov | jeblair, fungi, clarkb, apologize for being boring... but could we schedule/announce a mid week gerrit maintenance for savanna->sahara rename to make it in advance? | 22:31 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added UI switches for authentication state. https://review.openstack.org/79478 | 22:35 |
| *** thedodd has quit IRC | 22:35 | |
| *** dkliban has quit IRC | 22:36 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 22:36 |
| clarkb | jeblair: jhesketh ^ that adds testing for the new feature, but it hard codes those config options | 22:37 |
| *** mriedem has quit IRC | 22:37 | |
| clarkb | jeblair: jhesketh is there a good way of toggling zuul.conf stuff in tests? I couldn't find a way to do it nicely | 22:37 |
| jhesketh | not that I know of | 22:37 |
| jeblair | SergeyLukjanov: when would you propose? | 22:38 |
| mikal | jogo: the problems with turbo-hipster seem to be related to our pypi mirror going bad, and not being able to rebuild the mirror because we're having connection problems to pypi.python.org | 22:39 |
| jogo | ouch | 22:41 |
| hashar | jeblair: I am heading bed, thanks for your attention on those patches :-] | 22:41 |
| *** rfolco has quit IRC | 22:42 | |
| *** bhuvan_ has quit IRC | 22:42 | |
| *** bhuvan has quit IRC | 22:42 | |
| *** UtahDave has joined #openstack-infra | 22:42 | |
| jeblair | hashar: good night | 22:42 |
| mikal | fungi: is there a way to get pypi-mirror to run with --use_mirrors for its pip commands? | 22:42 |
| *** mwagner_lap has joined #openstack-infra | 22:43 | |
| clarkb | mikal: you can set your ~/.pip/pip.conf | 22:43 |
| clarkb | mikal: and do something silly like use our mirror >_> | 22:43 |
| mikal | pip.conf seems to want an explicit list of mirrors though | 22:43 |
| mikal | clarkb: what's the URL of your mirror? | 22:43 |
| sdague | jogo: https://review.openstack.org/#/c/79482/ - that seems to be the fix we need | 22:44 |
| clarkb | mikal: http://pypi.openstack.org/openstack | 22:45 |
| *** UtahDave has left #openstack-infra | 22:45 | |
| SergeyLukjanov | jeblair, for me the sooner is better, so, it depends on when you're ok to schedule it | 22:45 |
| SergeyLukjanov | jeblair, Wed will be great for me | 22:46 |
| clarkb | jhesketh: jeblair: actually what if I set remote location to upstream_root + project and set poll to true? | 22:49 |
| dstufft | --use-mirrors is gone | 22:49 |
| jeblair | SergeyLukjanov: i have to run for a bit; but i'd like to know what time you think best. i'm worried about finding a time this week when you and an infra root are both available and that also won't be disruptive to the project (things look like they are still a bit busy) | 22:49 |
| dstufft | pip 1.5 noops it | 22:49 |
| dstufft | maybe 1.4 nooped it too I don't remember | 22:49 |
| clarkb | jhesketh: jeblair: I have just realized that the synchronization around that stuff to make it mimic live systems will be hard in the tests | 22:49 |
| *** oubiwan__ has joined #openstack-infra | 22:49 | |
| clarkb | jhesketh: jeblair: but by having it talk to upstream root we don't need to build in replication but test the other code paths | 22:49 |
| clarkb | mordred: AaronGreen it would be great for you guys to test https://review.openstack.org/76057 latest patchset at some point too | 22:50 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Set the correct default for is_active. https://review.openstack.org/79471 | 22:50 |
| *** bknudson has left #openstack-infra | 22:51 | |
| *** bearhands is now known as comstud | 22:52 | |
| SergeyLukjanov | jeblair, I'll be available in any time that'll work for infra root, from the savanna pov - dev is fully blocked by renaming repos to rename the rest stuff, validate and push RC | 22:52 |
| jhesketh | clarkb: as in the unit tests would rely on being able to talk to upstream openstack? | 22:53 |
| clarkb | jhesketh: no, in the tests everything is cloned from upstream_root (a variable pointing at local test repos). I could just set location_override to that value to test code paths without worrying about replication | 22:53 |
| *** denis_makogon has quit IRC | 22:54 | |
| jhesketh | right, okay | 22:54 |
| jhesketh | I need to re-read your patch I think | 22:54 |
| *** yamahata has quit IRC | 22:54 | |
| jhesketh | but that sounds sensible | 22:55 |
| *** yamahata__ has quit IRC | 22:55 | |
| *** david-lyle has quit IRC | 22:56 | |
| *** yamahata__ has joined #openstack-infra | 22:56 | |
| *** yamahata has joined #openstack-infra | 22:56 | |
| *** hashar has quit IRC | 22:56 | |
| *** derekh has joined #openstack-infra | 22:57 | |
| *** gokrokve_ has quit IRC | 22:58 | |
| jogo | anyone know what 'python -S' does? | 22:58 |
| SergeyLukjanov | jogo, AFAIR it means that the site-packages directory will not be added to the path | 23:00 |
| SergeyLukjanov | jogo, so, only stdlib will be available | 23:01 |
| jogo | SergeyLukjanov: I ask with regard to rootwrap | 23:02 |
| jogo | ohh that may work | 23:02 |
| jogo | only dep for rootwrap is six | 23:03 |
| jogo | SergeyLukjanov: '-S' drops startup time by half | 23:03 |
| mordred | jogo: this is still "rootwrap is too slow" ? | 23:03 |
| SergeyLukjanov | jogo, because small number of default imports | 23:04 |
| *** rpodolyaka has quit IRC | 23:04 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 23:04 |
| clarkb | jhesketh: ^ is what I am thinking | 23:04 |
| jogo | mordred: yeah | 23:05 |
| jogo | turns out neutron is just a giant rootwrap app | 23:05 |
| jogo | mordred: I tried Cython and RPython but both are slow startup | 23:05 |
| jogo | 'python -S' helped a lot | 23:05 |
| sdague | jogo: so, honestly, are the rootwrap rules tight enough to believe it is has any security | 23:05 |
| jogo | sdague: hehe no comment | 23:06 |
| sdague | because nova includes tee in rootwrap | 23:06 |
| *** andreaf has quit IRC | 23:06 | |
| sdague | at which point, all bets are off | 23:06 |
| SergeyLukjanov | jeblair, so, renaming on Wed (best) / Thu will make savanna team to fix any potential issues and stabilize code till the next week to have a week before the scheduled RC to test all stuff... | 23:06 |
| sdague | I think rootwrap in most of it's uses is completely security theatre at this point | 23:06 |
| *** amcrn has quit IRC | 23:07 | |
| mordred | sdague: what is it that it theorectically does that sudo doesn't? and why not just bite the bullet and say "this is a daemon whose job in life is to orchestrate root-only resources" and run nova-compute as root? | 23:08 |
| sdague | it theoretically filters arguments | 23:08 |
| sdague | however, there is a lot of theory there | 23:08 |
| dstufft | security theatre is the best theatre | 23:08 |
| clarkb | sdague: with tee we can overwrite the rootwrap rules right? then have a party? | 23:09 |
| *** rpodolyaka has joined #openstack-infra | 23:09 | |
| sdague | clarkb: exactly | 23:09 |
| dstufft | my favorite was the sudo rule that restricted everything but ``python`` | 23:10 |
| *** Sukhdev has joined #openstack-infra | 23:10 | |
| *** bhuvan has joined #openstack-infra | 23:10 | |
| *** bhuvan_ has joined #openstack-infra | 23:10 | |
| sdague | so the neutron rules actually look vaguely sane | 23:10 |
| *** amcrn has joined #openstack-infra | 23:10 | |
| sdague | much more so than nova | 23:10 |
| clarkb | jeblair: I did leave a comment on your be more careful with locks change | 23:12 |
| sdague | https://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters however... is a sad sad joke :) | 23:12 |
| mordred | I mean, seriously - nova-compute manipulates vms and network bridges on compute hosts - those hosts have no other purpose in life than managing those vms - so if someone compromised the python process and could do additinoal things ... what evil could they do that matters which is not already covered by the things we're giving to nova-compute in the first palce? | 23:12 |
| *** CaptTofu has joined #openstack-infra | 23:13 | |
| mordred | (given how many things we give it access to do) | 23:13 |
| sdague | mordred: 100% agree | 23:13 |
| jogo | mordred: mine bitcoins! | 23:13 |
| jogo | oh wait that is userland | 23:14 |
| sdague | I mean you give compute tee, dd, schred, chown, chmod, cp | 23:14 |
| sdague | with no restrictions | 23:14 |
| jogo | does nova support running sudo as rootwrap in trunk? | 23:14 |
| mordred | sdague: if you have dd, then you have the whole box | 23:14 |
| sdague | mordred: there are so many ways you have the whole box in those rules | 23:14 |
| mordred | because you can dd over /etc/sudoers | 23:14 |
| sdague | or tee | 23:14 |
| clarkb | mordred: or rootwrap | 23:15 |
| mordred | or chwon | 23:15 |
| jogo | http://git.openstack.org/cgit/openstack/nova/tree/etc/nova/rootwrap.d/compute.filters | 23:15 |
| mordred | I vote that it's theatre | 23:15 |
| *** Sukhdev has quit IRC | 23:15 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: WIP : Try Icehouse precise proposed UCA - Not for review https://review.openstack.org/74889 | 23:15 |
| sdague | that is, however, the nova-compute process | 23:15 |
| jogo | I don't see dd in compute filters | 23:15 |
| locke105 | chmod 777 /etc/* ftw... | 23:15 |
| sdague | which doesn't help with the fact that neutron actually seems to use rootwrap appropriately, including really strict params | 23:16 |
| sdague | jogo: https://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters#L91 | 23:16 |
| jogo | sdague: ahh | 23:16 |
| jogo | wow | 23:17 |
| jogo | so sdague I don'tthink we support running just sudo in nova | 23:17 |
| *** emagana has quit IRC | 23:17 | |
| sdague | jogo: well, honestly, if it wasn't freeze, I'd be really tempted to push a commit to delete compute rootwrap, because it's a farse :) | 23:18 |
| *** CaptTofu has quit IRC | 23:18 | |
| sdague | so I'll wait until juno | 23:18 |
| sdague | we just need to realize the computes are untrusted | 23:18 |
| jogo | heh yeah | 23:18 |
| sdague | I only realized the tee thing the other day, then read the rest of it | 23:18 |
| *** julim has quit IRC | 23:19 | |
| mikal | clarkb: is the openstack pypi mirror rsyncable? | 23:19 |
| jogo | sdague mordred: so n-cpu aside neutron still has issues | 23:20 |
| clarkb | mikal: no | 23:20 |
| jogo | which is why i think the 'python -S' option may help | 23:21 |
| jogo | mordred: Ithink that would need to be a pbr option no? | 23:21 |
| sdague | jogo: sure though how much of this is just because it's implementing gheto-sdn? | 23:21 |
| sdague | which would go away if it was using opendaylight instead | 23:21 |
| sdague | in which case, punt, and wait for that | 23:22 |
| *** flaper87 is now known as flaper87|afk | 23:22 | |
| mordred | clarkb, mikal when I have more than 5 minutes and when I'm done with manage-projects I'd like to rethink how we're doign mirror - we spend WAY too much time on it | 23:22 |
| mordred | but not now | 23:22 |
| mordred | maybe in atlanta | 23:22 |
| jogo | sdague: couldn't say, so if adding 'python -S' support isn't easy I agree. | 23:22 |
| *** rfolco has joined #openstack-infra | 23:22 | |
| sdague | because I really think there isn't a lot of value in patching up ghetto-sdn | 23:22 |
| jogo | but it may be an easy bump | 23:22 |
| sdague | it's days are numbered | 23:22 |
| *** sabari2 has quit IRC | 23:23 | |
| jogo | sdague: well this would help all the rootwrap things | 23:23 |
| sdague | sure, but if you call out to a real sdn, how much of that just goes away | 23:23 |
| sdague | like all the ovs calls, the ip calls, etc | 23:23 |
| sdague | because the one place in nova where we see a real impact is on the network side as well | 23:24 |
| jogo | sdague: yes, but as long as rootwrap is around at all. its worth making faster | 23:24 |
| sdague | but with something like opendaylight all that goes away, because it's just ovs | 23:24 |
| sdague | jogo: maybe | 23:24 |
| jogo | as I said it depends how hard this is | 23:25 |
| sdague | I think you're going to shed 95% of the calls | 23:25 |
| mordred | jogo: what needs to happen to support it? | 23:25 |
| sdague | and it immediately becomes a noop | 23:25 |
| mordred | jogo: you want it in the shebang line? | 23:25 |
| jogo | mordred: that would be the idea, trying it locally first | 23:25 |
| jogo | to see how insane itis | 23:25 |
| mordred | jogo: k. well, the template for that is in pbr | 23:25 |
| jogo | mordred: thanks thats what I thought | 23:25 |
| jogo | sdague: jumping topics http://status.openstack.org/elastic-recheck/ | 23:26 |
| jogo | massive spike in 1254872 | 23:26 |
| jogo | is that a spike in hits or in overall load | 23:26 |
| sdague | yeh, I'm not very convinced that's well structured | 23:26 |
| sdague | honestly, I looked at a couple of those bugs today | 23:26 |
| jogo | that what is well structured? | 23:26 |
| sdague | the bug | 23:27 |
| sdague | so look at some of the runs | 23:27 |
| mordred | jogo: oh - wait - it's not - the shebang line is basic setuptools | 23:27 |
| mordred | one sec | 23:27 |
| *** yamahata__ has quit IRC | 23:27 | |
| jogo | mordred: that explains why a git grep didn't find anything | 23:27 |
| jogo | mordred: anyway I may be getting ahead of myself | 23:27 |
| sdague | I think that is actually not just one bug, it's a bunch of smaller ones | 23:27 |
| sdague | that need diving on | 23:27 |
| jogo | sdague: yeah | 23:28 |
| jogo | the libvirt stuff is hairy | 23:28 |
| sdague | well, we borked logging enough | 23:28 |
| sdague | danpb and I sorted that one this morning | 23:28 |
| sdague | so hopefully runs tomorrow will be helpful | 23:28 |
| mordred | jogo: | 23:28 |
| mordred | header = easy_install.get_script_header("", executable, is_wininst) | 23:28 |
| sdague | the success runs in that bug are super suspicious | 23:28 |
| mordred | jogo: http://git.openstack.org/cgit/openstack-dev/pbr/tree/pbr/packaging.py#n504 | 23:28 |
| mordred | jogo: you'd need to edit header | 23:29 |
| mordred | jogo: or override teh call and do something else | 23:29 |
| jogo | mordred: cool, well once I try this locally ... | 23:29 |
| mordred | jogo: I'll find some info for you further | 23:29 |
| jogo | sdague: well the spike in libvirt isn't due to a new libvirt version at least | 23:31 |
| *** yamahata has quit IRC | 23:31 | |
| jogo | sdague: also Bug 1273386 scares me | 23:33 |
| *** esker has quit IRC | 23:33 | |
| jogo | the successes mean we don't always fail if we break the kernel | 23:33 |
| sdague | well we are just matching on a kernel trace | 23:34 |
| *** rcarrillocruz has joined #openstack-infra | 23:34 | |
| sdague | the kernel recovers much of the time | 23:34 |
| sdague | anyway, it's time for me to stop working. | 23:34 |
| sdague | but there is a ton of stuff in the race bugs to actually try to resolve during release | 23:34 |
| *** rfolco has quit IRC | 23:34 | |
| jogo | sdague: I thought we have a bug for http://logs.openstack.org/46/73646/5/gate/gate-tempest-dsvm-postgres-full/1cb4ffb/logs/screen-key.txt.gz#_2014-03-10_17_52_53_789 | 23:35 |
| sdague | so the more hands the better | 23:35 |
| jogo | sdague: agreed | 23:35 |
| sdague | jogo: did you delete it | 23:35 |
| *** rcarrillocruz1 has quit IRC | 23:35 | |
| sdague | that one comes and goes at a rate, where it's possible to not see it for 2 weeks | 23:36 |
| jogo | its still there | 23:36 |
| mordred | jogo: oh - easy peasy - | 23:36 |
| sdague | jogo: ok, where didn't it match? | 23:37 |
| jogo | sdague: not sure, my guess is missing file? | 23:37 |
| jogo | queries/1253482.yaml: AND message:"Address already in use" | 23:37 |
| sdague | did this just fail? | 23:37 |
| sdague | anyway, stopping work | 23:38 |
| sdague | see you tomorrow | 23:38 |
| jogo | o/ | 23:39 |
| jogo | sdague: ahh keystone log parsing derp | 23:39 |
| jogo | clarkb: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQWRkcmVzcyBhbHJlYWR5IGluIHVzZVwiICAgQU5EIGZpbGVuYW1lOmxvZ3Mqc2NyZWVuLWtleS50eHQgQU5EIGJ1aWxkX2NoYW5nZTpcIjczNjQ2XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NDQ5NDc2NDQ1MiwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ== | 23:39 |
| jogo | that doesn't look right | 23:39 |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: WIP: Support -S flag in scripts https://review.openstack.org/79492 | 23:39 |
| mordred | jogo: that ^^ WILL NOT WORK -but shows what you need to do to hook in to the magic - the other bits needed would be in getting the config option from setup.cfg | 23:40 |
| *** gokrokve has joined #openstack-infra | 23:40 | |
| jogo | mordred: thanks | 23:41 |
| clarkb | jogo: can you give me the query instead? | 23:41 |
| clarkb | jogo: or shorten the url | 23:41 |
| * clarkb hates that they base64 encode the urls | 23:41 | |
| jeblair | SergeyLukjanov: i think earlier in the morning would probably be best if we we wanted to do wednesday, but i think we'd need to see if fungi is available for that. | 23:43 |
| jeblair | clarkb: logstash queue looks :( | 23:43 |
| clarkb | jeblair: ya it is :( | 23:44 |
| jogo | clarkb: message:"Address already in use" AND filename:logs*screen-key.txt AND build_change:"73646" | 23:44 |
| jogo | jeblair that is a :( graph | 23:44 |
| clarkb | jogo: we are indexing 7k events per second in 4 nodes and 3.5k on 2 nodes | 23:44 |
| clarkb | jeblair: ^ | 23:44 |
| clarkb | jeblair: its just not keeping up, I wonder if we changed logging dramtically somewhere | 23:45 |
| fungi | jeblair: SergeyLukjanov: i'm okay with whatever time we think works. early is just a matter of having enough coffee on hand | 23:45 |
| *** ArxCruz has joined #openstack-infra | 23:46 | |
| fungi | dstufft: ssh jenkins@23.253.82.112 for a system where this is reproducible and ssh jenkins@15.185.123.189 for one where it is not (but which we consider to be configured similarly) | 23:46 |
| jeblair | fungi: sadly, the dst change means i'm not up until what most people around here call "8am" or people near where you live call "11am" | 23:47 |
| jeblair | fungi: so i may not be much help | 23:47 |
| fungi | dstufft: reproducing is a matter of 'virtualenv -p pypy foo ; foo/bin/pip install -U setuptools mccabe' | 23:47 |
| clarkb | I was up at 6am local time today, but that was for special stuff | 23:47 |
| clarkb | don't expect that any time soon | 23:47 |
| dstufft | fungi: ok, it may be a little while before I switch to that, dealing with a thing | 23:47 |
| fungi | jeblair: i get up at 10:00 utc, regardless of what the locals think the time is | 23:48 |
| StevenK | fungi: That must make sprints fun. | 23:48 |
| fungi | dstufft: i think we're stable for now with a workaround of not runnig pypy jobs in rackspace, so no hurry | 23:48 |
| fungi | StevenK: i do long-distance events, not sprints | 23:48 |
| fungi | StevenK: easier on the knees | 23:48 |
| *** oubiwan__ has left #openstack-infra | 23:48 | |
| StevenK | Haha | 23:49 |
| jeblair | fungi: so if you're okay with this, shall we assume it meets with SergeyLukjanov's approval (i think he was pretty clear anytime is good as long as the date is early) | 23:50 |
| jeblair | fungi: you want to send an announcement with whatever time you feel like? | 23:50 |
| fungi | jeblair: sure--i'll announce the outage and we'll see whether zhiwei is around later and interested in tacking the stackforge/cookbook-openstack-metering->telemetry rename onto it too | 23:51 |
| SergeyLukjanov | I'm here from my phone and I confirm any date time works for me :) Wednesday is great / jeblair fungi | 23:51 |
| *** bknudson has joined #openstack-infra | 23:52 | |
| *** zzelle_ has quit IRC | 23:52 | |
| fungi | SergeyLukjanov: jeblair's around starting at ~1500utc if that's not too late in the day, otherwise i can go earlier (1200utc?) | 23:52 |
| SergeyLukjanov | fungi: Do you know from which tz zhiwei is? | 23:53 |
| *** bada_ has joined #openstack-infra | 23:53 | |
| fungi | SergeyLukjanov: cn i think (it's all one tz apparently?). i tend to hear from him around now or maybe a few hours later | 23:53 |
| SergeyLukjanov | fungi: Both times are very comfort for me (+4) | 23:54 |
| SergeyLukjanov | fungi: I'll try to contact him too if don't see his approval when wake up | 23:55 |
| fungi | SergeyLukjanov: that means 1500utc is 7pm local time for you? if that's not too bad for your schedule, it means jeblair can deal with other emergencies which might crop up while i'm in the middle of the rename steps | 23:56 |
| *** ArxCruz has quit IRC | 23:56 | |
| *** bada has quit IRC | 23:56 | |
| fungi | if so, that seems like a safe option | 23:56 |
| jogo | mordred: sigh: | 23:57 |
| jogo | time rootwrap ip a: 0.2 seconds | 23:57 |
| SergeyLukjanov | fungi: ++ | 23:57 |
| jogo | time ip a: 0.002s | 23:57 |
| jeblair | fungi: 1300 is about when the project woke up this morning | 23:57 |
| SergeyLukjanov | fungi: I'm online from 9am to 2am local in avg. :) | 23:58 |
| *** CaptTofu has joined #openstack-infra | 23:58 | |
| SergeyLukjanov | jeblair, fungi, thank you very much! | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!