| *** thorst_ has quit IRC | 00:00 | |
| *** nwkarsten has joined #openstack-infra | 00:01 | |
| *** sihan has joined #openstack-infra | 00:01 | |
| mriedem | etherpad.o.o down? | 00:03 |
|---|---|---|
| sihan | Hi guys, what is gerrit format comment for UI? Our ci result does not show color, and it did not show on the top, only show in the commit message section. | 00:03 |
| *** kzaitsev_mb has joined #openstack-infra | 00:04 | |
| *** nwkarsten has quit IRC | 00:05 | |
| fungi | mriedem: midnight utc database backups. still down for you? | 00:07 |
| *** bhavik has joined #openstack-infra | 00:08 | |
| *** sdake has joined #openstack-infra | 00:09 | |
| openstackgerrit | Merged openstack/diskimage-builder: Fix the bug that "mktemp: failed to create directory" https://review.openstack.org/335775 | 00:10 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 00:10 |
| clarkb | zl01, 03 and 06 done manually. the oyhers are running against playbook without async now | 00:10 |
| clarkb | sihan: you can find it in zuul or system-config/modules/openstack-project/manifests/review.pp iirc | 00:11 |
| *** ffrank has quit IRC | 00:13 | |
| mriedem | fungi: now it's back | 00:14 |
| *** darvon has quit IRC | 00:15 | |
| *** darvon has joined #openstack-infra | 00:15 | |
| *** PollyZ has joined #openstack-infra | 00:15 | |
| *** sdake_ has joined #openstack-infra | 00:16 | |
| *** thumpba has joined #openstack-infra | 00:17 | |
| thumpba | i need to update my ICLA with my mailing address so i can commit, how can i do this? | 00:17 |
| *** IlyaG has quit IRC | 00:19 | |
| *** kaisers1 has joined #openstack-infra | 00:19 | |
| *** sdake has quit IRC | 00:19 | |
| *** PollyZ has quit IRC | 00:21 | |
| thumpba | nevermind https://review.openstack.org/#/settings/contact seems to do the trick | 00:21 |
| *** piet has joined #openstack-infra | 00:21 | |
| *** zeih has joined #openstack-infra | 00:21 | |
| *** kaisers has quit IRC | 00:21 | |
| *** bhunter71 has joined #openstack-infra | 00:22 | |
| *** markvoelker has joined #openstack-infra | 00:22 | |
| *** SumitNaiksatam has quit IRC | 00:23 | |
| *** ddieterly has joined #openstack-infra | 00:24 | |
| *** zeih has quit IRC | 00:26 | |
| *** pt_15 has quit IRC | 00:26 | |
| *** xarses has joined #openstack-infra | 00:26 | |
| *** markvoelker has quit IRC | 00:27 | |
| *** PollyZ has joined #openstack-infra | 00:29 | |
| *** Apoorva has joined #openstack-infra | 00:30 | |
| *** itisha has quit IRC | 00:30 | |
| *** thumpba has quit IRC | 00:31 | |
| *** nwkarsten has joined #openstack-infra | 00:31 | |
| *** lvdongbing has joined #openstack-infra | 00:33 | |
| *** PollyZ has quit IRC | 00:34 | |
| *** piet has quit IRC | 00:36 | |
| *** karthik__ has quit IRC | 00:38 | |
| *** adrian_otto has quit IRC | 00:39 | |
| *** shashank_hegde has quit IRC | 00:39 | |
| *** ddieterly has quit IRC | 00:41 | |
| *** Jeffrey4l has joined #openstack-infra | 00:42 | |
| *** PollyZ has joined #openstack-infra | 00:43 | |
| *** Sukhdev has quit IRC | 00:47 | |
| *** PollyZ has quit IRC | 00:47 | |
| *** yamahata has joined #openstack-infra | 00:48 | |
| *** woodster_ has quit IRC | 00:49 | |
| *** amotoki has joined #openstack-infra | 00:54 | |
| *** rcernin has quit IRC | 00:55 | |
| *** PollyZ has joined #openstack-infra | 00:56 | |
| *** thorst_ has joined #openstack-infra | 00:59 | |
| *** amitgandhinz has quit IRC | 01:01 | |
| *** PollyZ has quit IRC | 01:01 | |
| *** amitgandhinz has joined #openstack-infra | 01:01 | |
| *** chlong has quit IRC | 01:06 | |
| *** sihan has quit IRC | 01:06 | |
| *** thorst_ has quit IRC | 01:07 | |
| *** Apoorva has quit IRC | 01:07 | |
| *** tqtran_ has quit IRC | 01:07 | |
| *** pvaneck has quit IRC | 01:08 | |
| *** xyang1 has quit IRC | 01:10 | |
| *** PollyZ has joined #openstack-infra | 01:10 | |
| *** Apoorva has joined #openstack-infra | 01:11 | |
| *** zhurong has joined #openstack-infra | 01:12 | |
| *** sdake_ has quit IRC | 01:13 | |
| *** PollyZ has quit IRC | 01:15 | |
| *** Apoorva has quit IRC | 01:15 | |
| *** zshuo has joined #openstack-infra | 01:16 | |
| *** karthik__ has joined #openstack-infra | 01:18 | |
| *** zeih has joined #openstack-infra | 01:22 | |
| *** thorst_ has joined #openstack-infra | 01:23 | |
| *** PollyZ has joined #openstack-infra | 01:24 | |
| *** thorst_ has quit IRC | 01:24 | |
| *** zeih has quit IRC | 01:26 | |
| *** julim has quit IRC | 01:26 | |
| *** PollyZ has quit IRC | 01:28 | |
| *** kzaitsev_mb has quit IRC | 01:30 | |
| *** yanyanhu has joined #openstack-infra | 01:30 | |
| *** gouthamr has quit IRC | 01:34 | |
| *** claudiub has quit IRC | 01:34 | |
| *** rwsu has joined #openstack-infra | 01:36 | |
| *** iurygregory has joined #openstack-infra | 01:36 | |
| *** bradm_ has joined #openstack-infra | 01:36 | |
| *** dulek has joined #openstack-infra | 01:36 | |
| *** mkoderer__ has joined #openstack-infra | 01:36 | |
| *** ianw has joined #openstack-infra | 01:36 | |
| *** strigazi has joined #openstack-infra | 01:36 | |
| *** trown|outtypewww has joined #openstack-infra | 01:36 | |
| *** hamzy has joined #openstack-infra | 01:36 | |
| *** mkoderer has joined #openstack-infra | 01:36 | |
| *** gothicmindfood has joined #openstack-infra | 01:36 | |
| *** PollyZ has joined #openstack-infra | 01:37 | |
| *** karthik__ has quit IRC | 01:38 | |
| *** ddieterly has joined #openstack-infra | 01:38 | |
| openstackgerrit | Bin Hu proposed openstack-infra/irc-meetings: Add Meeting Information for New Project Gluon https://review.openstack.org/336330 | 01:39 |
| *** zhurong has quit IRC | 01:40 | |
| *** sdake has joined #openstack-infra | 01:40 | |
| *** zhurong has joined #openstack-infra | 01:41 | |
| *** PollyZ has quit IRC | 01:43 | |
| *** ddieterly has quit IRC | 01:43 | |
| *** ams__ has quit IRC | 01:44 | |
| *** ianychoi has joined #openstack-infra | 01:49 | |
| *** PollyZ has joined #openstack-infra | 01:51 | |
| *** PollyZ has quit IRC | 01:56 | |
| *** ddieterly has joined #openstack-infra | 01:57 | |
| *** PollyZ has joined #openstack-infra | 01:59 | |
| *** savihou has quit IRC | 02:04 | |
| *** savihou has joined #openstack-infra | 02:05 | |
| *** Apoorva has joined #openstack-infra | 02:10 | |
| *** signed8bit_Zzz is now known as signed8bit | 02:10 | |
| *** sdake has quit IRC | 02:12 | |
| *** nwkarsten has quit IRC | 02:13 | |
| *** Apoorva has quit IRC | 02:16 | |
| *** thorst_ has joined #openstack-infra | 02:19 | |
| *** thorst_ has quit IRC | 02:20 | |
| *** Daisy has joined #openstack-infra | 02:20 | |
| *** thorst_ has joined #openstack-infra | 02:20 | |
| *** zeih has joined #openstack-infra | 02:23 | |
| *** markvoelker has joined #openstack-infra | 02:24 | |
| *** zeih has quit IRC | 02:27 | |
| *** markvoelker has quit IRC | 02:28 | |
| *** thorst_ has quit IRC | 02:29 | |
| *** ddieterly has quit IRC | 02:29 | |
| *** chlong has joined #openstack-infra | 02:34 | |
| *** shashank_hegde has joined #openstack-infra | 02:34 | |
| *** PollyZ has quit IRC | 02:35 | |
| *** mriedem has quit IRC | 02:38 | |
| *** rossella_s has quit IRC | 02:39 | |
| *** rossella_s has joined #openstack-infra | 02:39 | |
| *** karthik__ has joined #openstack-infra | 02:41 | |
| *** shashank_hegde has quit IRC | 02:42 | |
| *** kushal has joined #openstack-infra | 02:47 | |
| *** piet has joined #openstack-infra | 02:51 | |
| *** PollyZ has joined #openstack-infra | 02:52 | |
| *** ramishra has joined #openstack-infra | 02:52 | |
| *** yuanying_ has quit IRC | 02:52 | |
| *** karthik__ has quit IRC | 02:54 | |
| *** Daisy has quit IRC | 02:56 | |
| *** Daisy has joined #openstack-infra | 02:56 | |
| *** PollyZ has quit IRC | 02:56 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: set check-osc-job to voting https://review.openstack.org/336344 | 02:57 |
| *** rook- has quit IRC | 02:59 | |
| *** coolsvap has joined #openstack-infra | 02:59 | |
| *** gyee has quit IRC | 03:00 | |
| *** rook- has joined #openstack-infra | 03:00 | |
| *** karthik__ has joined #openstack-infra | 03:00 | |
| *** kushal has quit IRC | 03:01 | |
| *** jamielennox is now known as jamielennox|away | 03:03 | |
| *** asettle has joined #openstack-infra | 03:08 | |
| *** kushal has joined #openstack-infra | 03:09 | |
| *** Daisy has quit IRC | 03:11 | |
| *** Daisy has joined #openstack-infra | 03:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Clear up "already provided" message https://review.openstack.org/290968 | 03:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make element-info output env variables https://review.openstack.org/335265 | 03:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add IMAGE_ELEMENT_YAML https://review.openstack.org/334786 | 03:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Making element overriding explicit https://review.openstack.org/334785 | 03:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Convert pkg-map and svc-map copies to explicit variables https://review.openstack.org/335308 | 03:13 |
| *** nwkarsten has joined #openstack-infra | 03:14 | |
| *** asettle has quit IRC | 03:15 | |
| *** Daisy has quit IRC | 03:16 | |
| *** nwkarsten has quit IRC | 03:18 | |
| masayukig | jhesketh or someone in infra, could you get logs of health.openstack.org ? Now, http://health.openstack.org/build_name/gate-monasca-api-python27/test_runs?callback=angular.callbacks._0&datetime_resolution=hour&start_date=2016-05-31T03:00:00.000Z&stop_date=2016-07-01T03:00:00.000Z is returning a 500. | 03:18 |
| masayukig | I'd like to see what's happening. | 03:19 |
| *** PollyZ has joined #openstack-infra | 03:19 | |
| *** harlowja_at_home has joined #openstack-infra | 03:20 | |
| *** zeih has joined #openstack-infra | 03:23 | |
| *** gsilvis has quit IRC | 03:24 | |
| *** markvoelker has joined #openstack-infra | 03:25 | |
| *** gsilvis has joined #openstack-infra | 03:25 | |
| *** links has joined #openstack-infra | 03:26 | |
| *** zeih has quit IRC | 03:28 | |
| *** Qiming has quit IRC | 03:28 | |
| *** yamamoto_ has joined #openstack-infra | 03:28 | |
| *** zshuo has quit IRC | 03:28 | |
| *** markvoelker has quit IRC | 03:29 | |
| *** PollyZ has quit IRC | 03:31 | |
| *** zshuo has joined #openstack-infra | 03:31 | |
| *** jamielennox|away is now known as jamielennox | 03:31 | |
| *** Qiming has joined #openstack-infra | 03:32 | |
| *** karthik__ has quit IRC | 03:33 | |
| *** binbincong has quit IRC | 03:33 | |
| *** karthik__ has joined #openstack-infra | 03:33 | |
| *** chlong has quit IRC | 03:34 | |
| *** harlowja_at_home has quit IRC | 03:35 | |
| *** thorst_ has joined #openstack-infra | 03:35 | |
| *** Daisy has joined #openstack-infra | 03:38 | |
| *** Liuqing has joined #openstack-infra | 03:39 | |
| *** Daisy has quit IRC | 03:40 | |
| *** Daisy has joined #openstack-infra | 03:41 | |
| *** thorst_ has quit IRC | 03:44 | |
| *** Daisy has quit IRC | 03:45 | |
| *** piet has quit IRC | 03:46 | |
| *** binbincong has joined #openstack-infra | 03:49 | |
| *** karthi___ has joined #openstack-infra | 03:50 | |
| *** yuanying has joined #openstack-infra | 03:50 | |
| *** karthik__ has quit IRC | 03:52 | |
| openstackgerrit | Adam Spiers proposed openstack-infra/project-config: Fix gerrit ACLs for openstack-resource-agents-specs https://review.openstack.org/335764 | 03:52 |
| *** sputnik13 has joined #openstack-infra | 03:52 | |
| *** PollyZ has joined #openstack-infra | 03:54 | |
| *** PollyZ has quit IRC | 03:59 | |
| *** Daisy has joined #openstack-infra | 03:59 | |
| *** Daisy has quit IRC | 03:59 | |
| *** Daisy has joined #openstack-infra | 04:00 | |
| *** sputnik13_ has joined #openstack-infra | 04:02 | |
| *** zhurong has quit IRC | 04:03 | |
| *** Daisy has quit IRC | 04:04 | |
| *** Daisy has joined #openstack-infra | 04:04 | |
| jhesketh | masayukig: Here's what happens when I hit that page: http://paste.openstack.org/show/524379/ | 04:04 |
| *** tqtran has joined #openstack-infra | 04:05 | |
| *** sputnik13_ has quit IRC | 04:06 | |
| masayukig | jhesketh : thanks! | 04:06 |
| *** pgadiya has joined #openstack-infra | 04:07 | |
| masayukig | I have no idea to fix this, though... :( | 04:07 |
| *** PollyZ has joined #openstack-infra | 04:08 | |
| *** Daisy has quit IRC | 04:09 | |
| *** Daisy has joined #openstack-infra | 04:10 | |
| *** tqtran has quit IRC | 04:10 | |
| *** Daisy has quit IRC | 04:12 | |
| jhesketh | masayukig: yeah sorry I'm not around to help today either :-( | 04:12 |
| *** Daisy has joined #openstack-infra | 04:12 | |
| masayukig | no worries, thanks :) | 04:13 |
| *** sputnik13_ has joined #openstack-infra | 04:13 | |
| *** PollyZ has quit IRC | 04:13 | |
| *** _ody has joined #openstack-infra | 04:13 | |
| *** Apoorva has joined #openstack-infra | 04:13 | |
| *** zhurong has joined #openstack-infra | 04:13 | |
| *** Daisy has quit IRC | 04:14 | |
| *** Daisy has joined #openstack-infra | 04:15 | |
| *** sputnik13_ has quit IRC | 04:15 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Make element-info output env variables https://review.openstack.org/335265 | 04:16 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Convert pkg-map and svc-map copies to explicit variables https://review.openstack.org/335308 | 04:16 |
| *** nwkarsten has joined #openstack-infra | 04:16 | |
| *** signed8bit is now known as signed8bit_Zzz | 04:18 | |
| *** Daisy has quit IRC | 04:19 | |
| *** yamamoto_ has quit IRC | 04:19 | |
| *** Apoorva has quit IRC | 04:20 | |
| *** nwkarsten has quit IRC | 04:21 | |
| *** Daisy has joined #openstack-infra | 04:21 | |
| *** yamamoto_ has joined #openstack-infra | 04:21 | |
| *** IlyaG has joined #openstack-infra | 04:22 | |
| masayukig | mtreinish : hrm, it seems like memcached doesn't accept a key contains spaces.. : MemcacheIllegalInputError: Key contains spaces: '_lockopenstack_health.api:_query_test_runs_by_build_name|gate-monasca-api-python27 2016-05-31 03:00:00+00:00 2016-07-01 03:00:00+00:00' | 04:23 |
| *** zeih has joined #openstack-infra | 04:24 | |
| *** Daisy has quit IRC | 04:25 | |
| *** Daisy has joined #openstack-infra | 04:25 | |
| *** markvoelker has joined #openstack-infra | 04:25 | |
| *** Daisy has quit IRC | 04:27 | |
| *** Daisy has joined #openstack-infra | 04:28 | |
| *** coolsvap has quit IRC | 04:28 | |
| *** zeih has quit IRC | 04:29 | |
| *** pgadiya has quit IRC | 04:29 | |
| *** markvoelker has quit IRC | 04:30 | |
| *** sputnik13 has quit IRC | 04:30 | |
| *** rlandy has quit IRC | 04:31 | |
| *** shashank_hegde has joined #openstack-infra | 04:32 | |
| *** Daisy has quit IRC | 04:33 | |
| *** asselin__ has joined #openstack-infra | 04:33 | |
| *** karthi___ has quit IRC | 04:34 | |
| *** karthik__ has joined #openstack-infra | 04:34 | |
| *** PollyZ has joined #openstack-infra | 04:35 | |
| *** coolsvap has joined #openstack-infra | 04:35 | |
| *** IlyaG has quit IRC | 04:36 | |
| *** sree has joined #openstack-infra | 04:36 | |
| *** asselin__ has quit IRC | 04:38 | |
| *** yamamoto_ has quit IRC | 04:38 | |
| *** IlyaG has joined #openstack-infra | 04:39 | |
| *** psachin has joined #openstack-infra | 04:39 | |
| *** PollyZ has quit IRC | 04:40 | |
| *** bhavik has quit IRC | 04:40 | |
| *** sputnik13 has joined #openstack-infra | 04:40 | |
| *** nwkarsten has joined #openstack-infra | 04:40 | |
| *** xarses has quit IRC | 04:41 | |
| *** xarses has joined #openstack-infra | 04:41 | |
| *** sree_ has joined #openstack-infra | 04:42 | |
| *** sree_ is now known as Guest46399 | 04:43 | |
| *** thorst_ has joined #openstack-infra | 04:43 | |
| *** nwkarsten has quit IRC | 04:45 | |
| *** sree has quit IRC | 04:45 | |
| *** PollyZ has joined #openstack-infra | 04:48 | |
| *** thorst_ has quit IRC | 04:49 | |
| *** karthi___ has joined #openstack-infra | 04:52 | |
| *** IlyaG has quit IRC | 04:52 | |
| *** karthik__ has quit IRC | 04:53 | |
| *** PollyZ has quit IRC | 04:53 | |
| *** signed8bit_Zzz is now known as signed8bit | 04:53 | |
| *** signed8bit has quit IRC | 04:54 | |
| *** zhurong has quit IRC | 04:58 | |
| *** piet has joined #openstack-infra | 04:58 | |
| *** e0ne has joined #openstack-infra | 04:59 | |
| *** zhurong has joined #openstack-infra | 05:00 | |
| openstackgerrit | vinay kumar muddu proposed openstack/diskimage-builder: Build wholedisk images capable of UEFI secure boot https://review.openstack.org/333888 | 05:00 |
| *** Sukhdev has joined #openstack-infra | 05:03 | |
| *** SumitNaiksatam has joined #openstack-infra | 05:07 | |
| *** e0ne has quit IRC | 05:07 | |
| *** ayoung has quit IRC | 05:09 | |
| *** chlong has joined #openstack-infra | 05:15 | |
| *** PollyZ has joined #openstack-infra | 05:16 | |
| *** pgadiya has joined #openstack-infra | 05:16 | |
| *** Guest46399 has quit IRC | 05:17 | |
| *** ianw is now known as ianw_pto | 05:19 | |
| *** sree has joined #openstack-infra | 05:20 | |
| *** PollyZ has quit IRC | 05:20 | |
| *** yamamoto_ has joined #openstack-infra | 05:24 | |
| *** zeih has joined #openstack-infra | 05:25 | |
| *** markvoelker has joined #openstack-infra | 05:26 | |
| *** yamahata has quit IRC | 05:27 | |
| *** PollyZ has joined #openstack-infra | 05:29 | |
| *** liusheng has quit IRC | 05:29 | |
| *** zeih has quit IRC | 05:30 | |
| *** liusheng has joined #openstack-infra | 05:30 | |
| *** markvoelker has quit IRC | 05:31 | |
| openstackgerrit | Babu Shanmugam proposed openstack-infra/project-config: Adding a new JOB for networking-ovn project https://review.openstack.org/330926 | 05:31 |
| *** dingyichen has joined #openstack-infra | 05:32 | |
| *** PollyZ has quit IRC | 05:35 | |
| *** yamahata has joined #openstack-infra | 05:37 | |
| *** Liuqing has quit IRC | 05:38 | |
| *** ramishra has quit IRC | 05:39 | |
| *** Liuqing has joined #openstack-infra | 05:39 | |
| *** ramishra has joined #openstack-infra | 05:40 | |
| *** Daisy has joined #openstack-infra | 05:41 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Move element-info a standard entry-point https://review.openstack.org/330893 | 05:42 |
| *** PollyZ has joined #openstack-infra | 05:43 | |
| *** zhurong has quit IRC | 05:43 | |
| *** thorst_ has joined #openstack-infra | 05:46 | |
| *** PollyZ has quit IRC | 05:47 | |
| *** zhurong has joined #openstack-infra | 05:48 | |
| *** YorikSar has quit IRC | 05:51 | |
| *** javeriak has joined #openstack-infra | 05:52 | |
| *** YorikSar has joined #openstack-infra | 05:53 | |
| *** thorst_ has quit IRC | 05:53 | |
| *** javeriak has quit IRC | 05:53 | |
| *** amitgandhinz has quit IRC | 05:56 | |
| *** amitgandhinz has joined #openstack-infra | 05:56 | |
| *** PollyZ has joined #openstack-infra | 05:56 | |
| *** tqtran has joined #openstack-infra | 05:58 | |
| *** Ravikiran_K has joined #openstack-infra | 05:59 | |
| *** yuanying has quit IRC | 06:01 | |
| *** yuanying has joined #openstack-infra | 06:02 | |
| *** PollyZ has quit IRC | 06:02 | |
| *** sputnik13 has quit IRC | 06:02 | |
| *** Daisy has quit IRC | 06:02 | |
| *** sputnik13 has joined #openstack-infra | 06:02 | |
| *** Daisy has joined #openstack-infra | 06:02 | |
| *** tqtran has quit IRC | 06:03 | |
| *** fawadkhaliq has joined #openstack-infra | 06:03 | |
| *** claudiub has joined #openstack-infra | 06:03 | |
| *** aeng has quit IRC | 06:04 | |
| *** isq has joined #openstack-infra | 06:05 | |
| *** yuanying has quit IRC | 06:05 | |
| *** yuanying has joined #openstack-infra | 06:08 | |
| *** fawadkhaliq has quit IRC | 06:09 | |
| *** fawadkhaliq has joined #openstack-infra | 06:10 | |
| *** PollyZ has joined #openstack-infra | 06:10 | |
| *** ccamacho has quit IRC | 06:13 | |
| openstackgerrit | Mathieu Mitchell proposed openstack-infra/project-config: Bump VM memory for latest stable CoreOS IPA https://review.openstack.org/336278 | 06:14 |
| *** PollyZ has quit IRC | 06:15 | |
| *** sputnik13 has quit IRC | 06:17 | |
| *** gouthamr has joined #openstack-infra | 06:17 | |
| *** tlian has quit IRC | 06:18 | |
| *** pgadiya has quit IRC | 06:20 | |
| *** chem|off has quit IRC | 06:25 | |
| *** chem|off has joined #openstack-infra | 06:25 | |
| *** zeih has joined #openstack-infra | 06:25 | |
| *** chem|off is now known as chem | 06:26 | |
| *** pgadiya has joined #openstack-infra | 06:26 | |
| *** armax has quit IRC | 06:26 | |
| *** markvoelker has joined #openstack-infra | 06:27 | |
| *** andreas_s has joined #openstack-infra | 06:28 | |
| *** Sukhdev has quit IRC | 06:29 | |
| *** rcernin has joined #openstack-infra | 06:29 | |
| *** zeih has quit IRC | 06:30 | |
| openstackgerrit | Mathieu Mitchell proposed openstack-infra/project-config: Introduce experimental job to test TinyIPA with less RAM https://review.openstack.org/327994 | 06:30 |
| yamamoto_ | "Unable to load this page! An API error occurred." on openstack health is known? | 06:31 |
| yamamoto_ | eg. http://status.openstack.org/openstack-health/#/job/periodic-tempest-dsvm-neutron-full-ssh-master | 06:32 |
| *** markvoelker has quit IRC | 06:32 | |
| openstackgerrit | Mathieu Mitchell proposed openstack-infra/project-config: Introduce experimental job to test TinyIPA with less RAM https://review.openstack.org/327994 | 06:33 |
| masayukig | yamamoto_ : yeah, I already pushed a patch for it : https://review.openstack.org/336367 | 06:34 |
| yamamoto_ | masayukig: thank you | 06:34 |
| openstackgerrit | Merged openstack-infra/project-config: Add a heat job to use devstack plugin https://review.openstack.org/317811 | 06:36 |
| *** zeih has joined #openstack-infra | 06:36 | |
| *** PollyZ has joined #openstack-infra | 06:37 | |
| *** mat128|afk is now known as mat128|holiday | 06:38 | |
| *** ccamacho has joined #openstack-infra | 06:41 | |
| *** zhurong has quit IRC | 06:41 | |
| *** nwkarsten has joined #openstack-infra | 06:42 | |
| *** PollyZ has quit IRC | 06:42 | |
| *** zhurong has joined #openstack-infra | 06:44 | |
| *** isq has quit IRC | 06:44 | |
| *** Nikolay_St has joined #openstack-infra | 06:46 | |
| *** nwkarsten has quit IRC | 06:46 | |
| *** piet has quit IRC | 06:46 | |
| *** ociuhandu has joined #openstack-infra | 06:47 | |
| *** isq has joined #openstack-infra | 06:47 | |
| *** camunoz has quit IRC | 06:49 | |
| *** PollyZ has joined #openstack-infra | 06:51 | |
| *** thorst_ has joined #openstack-infra | 06:52 | |
| *** jed56 has joined #openstack-infra | 06:54 | |
| *** PollyZ has quit IRC | 06:56 | |
| *** pcaruana has joined #openstack-infra | 06:57 | |
| yolanda | good morning | 06:58 |
| *** thorst_ has quit IRC | 06:59 | |
| *** ociuhandu_ has joined #openstack-infra | 07:00 | |
| *** ociuhandu has quit IRC | 07:00 | |
| *** ociuhandu_ is now known as ociuhandu | 07:00 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: As initial step to deploy with HA, start using pacemaker module https://review.openstack.org/335511 | 07:01 |
| *** javeriak has joined #openstack-infra | 07:02 | |
| *** tesseract- has joined #openstack-infra | 07:02 | |
| *** karthi___ has quit IRC | 07:03 | |
| *** PollyZ has joined #openstack-infra | 07:04 | |
| *** yuanying has quit IRC | 07:05 | |
| *** yuanying has joined #openstack-infra | 07:08 | |
| *** PollyZ has quit IRC | 07:09 | |
| *** Daisy_ has joined #openstack-infra | 07:10 | |
| *** asettle has joined #openstack-infra | 07:13 | |
| *** Daisy has quit IRC | 07:14 | |
| openstackgerrit | Merged openstack-infra/project-config: Add new project Zunclient to openstack repo https://review.openstack.org/317699 | 07:14 |
| *** javeriak has quit IRC | 07:15 | |
| *** esikachev has joined #openstack-infra | 07:15 | |
| *** gouthamr has quit IRC | 07:16 | |
| *** asettle has quit IRC | 07:17 | |
| AJaeger | morning, yolanda ! | 07:17 |
| *** yamahata has quit IRC | 07:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Add experimental wheel-build-ubuntu-xenial-amd64 job" https://review.openstack.org/336289 | 07:19 |
| openstackgerrit | Merged openstack-infra/project-config: Set wheel-build-ubuntu-xenial-amd64 voting https://review.openstack.org/336291 | 07:19 |
| openstackgerrit | Merged openstack-infra/project-config: Fix gerrit ACLs for openstack-resource-agents-specs https://review.openstack.org/335764 | 07:19 |
| openstackgerrit | Merged openstack-infra/project-config: Read only mode for fuel-upgrade project https://review.openstack.org/335841 | 07:22 |
| *** chlong has quit IRC | 07:23 | |
| *** dtardivel has joined #openstack-infra | 07:24 | |
| openstackgerrit | Merged openstack-infra/project-config: update osc-check-plugin script https://review.openstack.org/336061 | 07:24 |
| *** jlanoux has joined #openstack-infra | 07:24 | |
| *** gnuoy` is now known as gnuoy | 07:25 | |
| *** javeriak has joined #openstack-infra | 07:25 | |
| *** ihrachys has joined #openstack-infra | 07:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Puppet-refstack doesn't have any beaker jobs https://review.openstack.org/332654 | 07:27 |
| *** arxcruz has joined #openstack-infra | 07:27 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Networking OVN Data to Grafana https://review.openstack.org/335791 | 07:28 |
| *** markvoelker has joined #openstack-infra | 07:28 | |
| *** kushal has quit IRC | 07:28 | |
| *** kushal has joined #openstack-infra | 07:29 | |
| *** ihrachys has quit IRC | 07:31 | |
| *** markvoelker has quit IRC | 07:32 | |
| *** openstackgerrit has quit IRC | 07:33 | |
| *** ihrachys has joined #openstack-infra | 07:33 | |
| *** openstackgerrit has joined #openstack-infra | 07:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rework publishing jobs https://review.openstack.org/336405 | 07:34 |
| AJaeger | clarkb, this reorg changes the release jobs - it removes 800 jobs ^ | 07:34 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rework publishing jobs https://review.openstack.org/336405 | 07:35 |
| *** Daisy_ has quit IRC | 07:35 | |
| *** Daisy has joined #openstack-infra | 07:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Add rally job for Watcher service and support for rally-gate https://review.openstack.org/333860 | 07:37 |
| AJaeger | clarkb: my change will merge-conflict heavily in projects.yaml. If my change merges first, I'll rebase yours. | 07:37 |
| openstackgerrit | sandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 07:39 |
| AJaeger | yolanda, do you have time to review 336405, please? It's largish... | 07:39 |
| yolanda | AJaeger, sure, i can take a look in a while | 07:40 |
| *** Daisy has quit IRC | 07:41 | |
| *** rvba has joined #openstack-infra | 07:41 | |
| *** rvba has quit IRC | 07:41 | |
| *** rvba has joined #openstack-infra | 07:41 | |
| AJaeger | thanks, yolanda - and thanks for your review batch this morning | 07:41 |
| yolanda | no problem, just tell me if you need something more, i'll review as time permits | 07:42 |
| *** ajmiller has joined #openstack-infra | 07:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Run glanceclient functional tests in Identity v3-only https://review.openstack.org/313572 | 07:43 |
| *** fawadkhaliq has quit IRC | 07:44 | |
| AJaeger | yolanda: I'm fine now, thanks | 07:44 |
| *** ajmiller_ has quit IRC | 07:44 | |
| *** jpich has joined #openstack-infra | 07:46 | |
| openstackgerrit | Merged openstack-infra/project-config: kuryr: add libnetwork and k8s to gerritbot https://review.openstack.org/335922 | 07:46 |
| *** amoralej|off is now known as amoralej | 07:48 | |
| *** Kiall_ has quit IRC | 07:50 | |
| *** ihrachys_ has joined #openstack-infra | 07:51 | |
| *** ccamacho is now known as ccamacho|afk | 07:51 | |
| *** ihrachys_ has quit IRC | 07:52 | |
| *** ihrachys has quit IRC | 07:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Further kilo cleanup https://review.openstack.org/336413 | 07:54 |
| *** rcernin has quit IRC | 07:57 | |
| *** thorst_ has joined #openstack-infra | 07:57 | |
| *** ifarkas has joined #openstack-infra | 07:58 | |
| *** PollyZ has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** shardy has joined #openstack-infra | 08:01 | |
| *** PollyZ has quit IRC | 08:04 | |
| *** thorst_ has quit IRC | 08:04 | |
| *** Na3iL has joined #openstack-infra | 08:05 | |
| AJaeger | tonyb, do you want to review the change above? ^ | 08:06 |
| *** yamahata has joined #openstack-infra | 08:06 | |
| *** dtantsur|afk is now known as dtantsur | 08:07 | |
| tonyb | AJaeger: Thanks I'll have a look after dinner | 08:07 |
| *** asettle has joined #openstack-infra | 08:09 | |
| *** hashar has joined #openstack-infra | 08:10 | |
| *** Daisy has joined #openstack-infra | 08:10 | |
| *** rcernin has joined #openstack-infra | 08:11 | |
| AJaeger | tonyb: enjoy dinner! | 08:12 |
| *** eezhova has joined #openstack-infra | 08:13 | |
| *** PollyZ has joined #openstack-infra | 08:13 | |
| *** Daisy has quit IRC | 08:15 | |
| *** Daisy has joined #openstack-infra | 08:16 | |
| *** acoles_ is now known as acoles | 08:16 | |
| *** mrmartin has joined #openstack-infra | 08:17 | |
| frickler | clarkb: merged the pypy changes and fixed the remaining issues hopefully, please crosscheck (https://review.openstack.org/336272, not sure why gerritbot didn't log my update) | 08:17 |
| *** PollyZ has quit IRC | 08:17 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** openstackgerrit has joined #openstack-infra | 08:18 | |
| *** derekh has joined #openstack-infra | 08:19 | |
| *** Daisy_ has joined #openstack-infra | 08:20 | |
| AJaeger | thanks, frickler ! | 08:20 |
| *** sorantis has joined #openstack-infra | 08:20 | |
| *** jianghuaw has joined #openstack-infra | 08:22 | |
| *** Daisy has quit IRC | 08:23 | |
| *** kushal has quit IRC | 08:24 | |
| jianghuaw | Is there anyway to make nodepool keep the node if the test failed. | 08:24 |
| *** Daisy_ has quit IRC | 08:24 | |
| *** Daisy has joined #openstack-infra | 08:31 | |
| *** ams__ has joined #openstack-infra | 08:31 | |
| *** ihrachys has joined #openstack-infra | 08:31 | |
| *** Jeffrey4l has quit IRC | 08:33 | |
| *** yamamoto_ has quit IRC | 08:33 | |
| *** yamamoto has joined #openstack-infra | 08:35 | |
| *** zshuo has quit IRC | 08:36 | |
| *** ifarkas_ has joined #openstack-infra | 08:36 | |
| *** yanyanhu has quit IRC | 08:37 | |
| *** _degorenko|afk is now known as degorenko | 08:37 | |
| *** ifarkas has quit IRC | 08:37 | |
| hashar | jianghuaw: in short now unless you manually create the job from the command line and pass --hold-on-failure | 08:38 |
| hashar | jianghuaw: relevant code would be in nodepool/nodepool.py in NodeCompleteThread.handleEvent() | 08:39 |
| *** nwkarsten has joined #openstack-infra | 08:44 | |
| frickler | clarkb: nova will need https://review.openstack.org/336432 before the python35 jobs can run properly. I also found a python3.5 incompatibility issue in nova, will push a patch for that later | 08:46 |
| *** abregman has joined #openstack-infra | 08:47 | |
| *** nwkarsten has quit IRC | 08:48 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 08:51 |
| *** YorikSar has quit IRC | 08:51 | |
| *** YorikSar has joined #openstack-infra | 08:52 | |
| *** shashank_hegde has quit IRC | 08:52 | |
| *** Jeffrey4l has joined #openstack-infra | 08:55 | |
| *** ccamacho|afk has quit IRC | 08:55 | |
| *** javeriak has quit IRC | 08:57 | |
| *** javeriak has joined #openstack-infra | 08:58 | |
| *** ccamacho has joined #openstack-infra | 08:59 | |
| *** zshuo has joined #openstack-infra | 09:00 | |
| *** electrofelix has joined #openstack-infra | 09:00 | |
| *** thorst_ has joined #openstack-infra | 09:02 | |
| *** Daisy has quit IRC | 09:02 | |
| jianghuaw | @hashar: thanks. | 09:03 |
| hashar | jianghuaw: a hack I am doing from time to time is to have the job triggered as usual, then rush to the nodepool CLI command and manually hold the node | 09:03 |
| hashar | jianghuaw: with something like nodepool hold 123456 | 09:03 |
| hashar | but maybe nodepool job-create --hold-on-failure is the way to go. I am not running that version of nodepool yet though :( | 09:04 |
| jianghuaw | hashar: Yes, nodepool hold # can hold a target node. | 09:06 |
| jianghuaw | It will be helpful if there is any parameter in nodepool to keep node for failures. Specially useful for random failure. | 09:06 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here https://review.openstack.org/111011 | 09:07 |
| *** PollyZ has joined #openstack-infra | 09:07 | |
| *** thorst_ has quit IRC | 09:09 | |
| *** salv-orl_ has quit IRC | 09:09 | |
| *** salv-orlando has joined #openstack-infra | 09:10 | |
| *** PollyZ has quit IRC | 09:12 | |
| *** andymaier has joined #openstack-infra | 09:15 | |
| *** sree has quit IRC | 09:16 | |
| *** piet has joined #openstack-infra | 09:17 | |
| *** eezhova has quit IRC | 09:19 | |
| *** ajmiller has quit IRC | 09:20 | |
| *** ajmiller has joined #openstack-infra | 09:20 | |
| AJaeger | frickler: python35 jobs are non-voting - so we can figure that one out one by one - which is good. | 09:22 |
| AJaeger | frickler: thanks for helping to move this forward! | 09:23 |
| frickler | AJaeger: I'll push a new patch addressing your comments once my local tox run finishes successfully | 09:24 |
| AJaeger | hashar, jianghuaw have a look at https://review.openstack.org/#/c/332976/ - that's an auto-hold feature for failing jobs | 09:24 |
| hashar | AJaeger: you and jeblair are magic! | 09:25 |
| *** eezhova has joined #openstack-infra | 09:27 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/project-config: Add python35 jobs https://review.openstack.org/336272 | 09:28 |
| AJaeger | hashar: jeblair is for sure ;) | 09:29 |
| *** john-davidge has joined #openstack-infra | 09:29 | |
| *** markvoelker has joined #openstack-infra | 09:29 | |
| *** fawadkhaliq has joined #openstack-infra | 09:30 | |
| *** fawadkhaliq has quit IRC | 09:30 | |
| *** mrmartin has quit IRC | 09:30 | |
| *** john-dav_ has joined #openstack-infra | 09:32 | |
| *** dtantsur is now known as dtantsur|bbl | 09:33 | |
| *** markvoelker has quit IRC | 09:34 | |
| *** john-davidge has quit IRC | 09:35 | |
| *** ajmiller has quit IRC | 09:36 | |
| *** ajmiller has joined #openstack-infra | 09:36 | |
| *** Daisy has joined #openstack-infra | 09:37 | |
| *** ajmiller has quit IRC | 09:38 | |
| *** yolanda has quit IRC | 09:38 | |
| *** ajmiller has joined #openstack-infra | 09:38 | |
| *** yolanda has joined #openstack-infra | 09:38 | |
| *** john-dav_ is now known as john-davidge | 09:41 | |
| *** Daisy has quit IRC | 09:41 | |
| jianghuaw | Ajaeger: thanks for the sharing. That's useful. | 09:45 |
| *** david-lyle has quit IRC | 09:45 | |
| *** fawadkhaliq has joined #openstack-infra | 09:46 | |
| *** yolanda has quit IRC | 09:47 | |
| *** zshuo has quit IRC | 09:48 | |
| *** yolanda has joined #openstack-infra | 09:49 | |
| *** abregman is now known as abregman|afk | 09:50 | |
| *** Daisy has joined #openstack-infra | 09:53 | |
| *** tosky has joined #openstack-infra | 09:55 | |
| *** ociuhandu has quit IRC | 09:55 | |
| *** maishsk has quit IRC | 09:56 | |
| *** ifarkas_ has quit IRC | 09:58 | |
| *** maishsk has joined #openstack-infra | 09:58 | |
| *** david-lyle has joined #openstack-infra | 09:59 | |
| *** tqtran has joined #openstack-infra | 10:01 | |
| *** jcoufal has quit IRC | 10:04 | |
| *** yolanda has quit IRC | 10:05 | |
| *** lvdongbing has quit IRC | 10:05 | |
| *** tqtran has quit IRC | 10:05 | |
| *** piet has quit IRC | 10:06 | |
| *** yolanda has joined #openstack-infra | 10:06 | |
| *** gmann has quit IRC | 10:06 | |
| *** piet has joined #openstack-infra | 10:08 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add Meeting Information for New Project Gluon https://review.openstack.org/336330 | 10:09 |
| *** thorst_ has joined #openstack-infra | 10:09 | |
| *** rcernin has quit IRC | 10:10 | |
| *** _nadya_ has joined #openstack-infra | 10:11 | |
| *** ggnel_t has quit IRC | 10:13 | |
| *** zqfan has quit IRC | 10:13 | |
| *** thorst_ has quit IRC | 10:14 | |
| *** kushal has joined #openstack-infra | 10:16 | |
| *** zhurong has quit IRC | 10:17 | |
| openstackgerrit | Bernard Cafarelli proposed openstack/diskimage-builder: Export YUM variable in centos bases https://review.openstack.org/336471 | 10:18 |
| *** linuxmaniac_ has joined #openstack-infra | 10:19 | |
| *** linuxmaniac has quit IRC | 10:19 | |
| *** fawadkhaliq has quit IRC | 10:22 | |
| *** fawadkhaliq has joined #openstack-infra | 10:23 | |
| *** linuxmaniac_ has quit IRC | 10:23 | |
| *** rcernin has joined #openstack-infra | 10:23 | |
| *** linuxmaniac has joined #openstack-infra | 10:23 | |
| AJaeger | infra-root, I checked bindep 2.0 changes and it looks fine with current image - but rax-dfw still has old images. Will those be updated automatically? Or do we need manual upload again? | 10:23 |
| *** dtantsur|bbl is now known as dtantsur | 10:24 | |
| *** sorantis has quit IRC | 10:27 | |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add repositories for Charm layers https://review.openstack.org/334880 | 10:27 |
| *** fawadkhaliq has quit IRC | 10:28 | |
| *** amotoki has quit IRC | 10:28 | |
| *** PollyZ has joined #openstack-infra | 10:29 | |
| *** fawadkhaliq has joined #openstack-infra | 10:29 | |
| *** markvoelker has joined #openstack-infra | 10:30 | |
| openstackgerrit | German proposed openstack-infra/git-review: Story 2000662 https://review.openstack.org/336475 | 10:33 |
| *** Ravikiran_K has quit IRC | 10:34 | |
| *** PollyZ has quit IRC | 10:34 | |
| *** markvoelker has quit IRC | 10:35 | |
| *** piet has quit IRC | 10:35 | |
| *** jed56 has quit IRC | 10:35 | |
| *** fawadkhaliq has quit IRC | 10:35 | |
| *** ifarkas has joined #openstack-infra | 10:36 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Move ironic-inspector grenade job to the check pipeline https://review.openstack.org/336477 | 10:36 |
| *** maishsk_ has joined #openstack-infra | 10:37 | |
| *** fawadkhaliq has joined #openstack-infra | 10:37 | |
| *** maishsk has quit IRC | 10:37 | |
| *** fawadkhaliq has quit IRC | 10:40 | |
| *** fawadkhaliq has joined #openstack-infra | 10:40 | |
| *** maishsk_ has quit IRC | 10:41 | |
| *** PollyZ has joined #openstack-infra | 10:42 | |
| *** nwkarsten has joined #openstack-infra | 10:46 | |
| *** fawadk has joined #openstack-infra | 10:46 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Add jobs based on ipmitool drivers to ironic-python-agent https://review.openstack.org/329428 | 10:46 |
| *** fawadkhaliq has quit IRC | 10:47 | |
| *** fawadk has quit IRC | 10:47 | |
| *** PollyZ has quit IRC | 10:47 | |
| *** fawadkhaliq has joined #openstack-infra | 10:47 | |
| dtantsur | lucasagomes, ^^^ | 10:47 |
| lucasagomes | dtantsur, cool, will take alook | 10:48 |
| *** vgridnev has quit IRC | 10:50 | |
| openstackgerrit | sandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 10:50 |
| *** nwkarsten has quit IRC | 10:51 | |
| *** sorantis has joined #openstack-infra | 10:51 | |
| *** eezhova has quit IRC | 10:51 | |
| *** sdake has joined #openstack-infra | 10:51 | |
| *** maishsk has joined #openstack-infra | 10:52 | |
| *** fawadk has joined #openstack-infra | 10:52 | |
| *** fawadkhaliq has quit IRC | 10:52 | |
| openstackgerrit | Lau Mercadal proposed openstack-infra/git-review: Story 2000662 https://review.openstack.org/336475 | 10:53 |
| *** ociuhandu has joined #openstack-infra | 10:53 | |
| *** sdake_ has joined #openstack-infra | 10:54 | |
| *** vgridnev has joined #openstack-infra | 10:55 | |
| *** sdake has quit IRC | 10:56 | |
| *** PollyZ has joined #openstack-infra | 10:56 | |
| *** yolanda has quit IRC | 10:57 | |
| *** sdake_ has quit IRC | 10:57 | |
| *** fawadkhaliq has joined #openstack-infra | 10:58 | |
| *** amotoki has joined #openstack-infra | 10:59 | |
| *** maishsk has quit IRC | 10:59 | |
| *** fawadk has quit IRC | 10:59 | |
| *** nwkarsten has joined #openstack-infra | 11:00 | |
| *** jcoufal has joined #openstack-infra | 11:00 | |
| *** asettle has quit IRC | 11:01 | |
| *** PollyZ has quit IRC | 11:02 | |
| *** maishsk has joined #openstack-infra | 11:03 | |
| *** mgould has joined #openstack-infra | 11:03 | |
| mgould | morning everyone! | 11:04 |
| *** amotoki has quit IRC | 11:04 | |
| mgould | can people please review a simple change to tripleo gate? https://review.openstack.org/#/c/336049/ | 11:04 |
| *** sdake has joined #openstack-infra | 11:04 | |
| *** sdake has quit IRC | 11:04 | |
| mgould | removes non-docs checks for docs-only patches, because I recently spent a fortnight trying to get a docs-only patch through tripleo CI and don't want anyone to have to go through that pain again | 11:05 |
| *** pgadiya has quit IRC | 11:05 | |
| *** fawadkhaliq has quit IRC | 11:05 | |
| *** fawadkhaliq has joined #openstack-infra | 11:05 | |
| AJaeger | mgould: those skip-ifs are really evil, they can have nasty side effects. Please have some tripleo cores review them as well, I'd like to see consensus by tripleo team that they want this... | 11:06 |
| *** sdake has joined #openstack-infra | 11:07 | |
| mgould | AJaeger: I think the +1s are from tripleo cores, but I'll check | 11:07 |
| *** fawadk has joined #openstack-infra | 11:07 | |
| *** fawadkhaliq has quit IRC | 11:08 | |
| AJaeger | mgould: there's a single +1 | 11:08 |
| mgould | AJaeger: oh, yeah, all the others were for previous patchsets. Sorry. | 11:10 |
| *** kzaitsev_mb has joined #openstack-infra | 11:10 | |
| *** fawadkhaliq has joined #openstack-infra | 11:11 | |
| AJaeger | mgould: your change does not work | 11:11 |
| AJaeger | did you check the log file? | 11:11 |
| *** fawadk has quit IRC | 11:12 | |
| *** fawadkhaliq has quit IRC | 11:12 | |
| AJaeger | mgould: http://logs.openstack.org/49/336049/3/check/gate-project-config-layout/bdfedc2/console.html#_2016-06-30_15_33_43_224346 | 11:12 |
| *** fawadkhaliq has joined #openstack-infra | 11:12 | |
| mgould | AJaeger: thanks, looking now | 11:13 |
| mgould | is there a way I can test this locally? | 11:13 |
| AJaeger | mgould: run "tox -e zuul" and check the output, best pipe it to a file like "tox -e zuul|tee out" | 11:13 |
| AJaeger | mgould: problem is that there's a skip-if in line 1351 for triploe-ci-centos-7 - and that one matches your jobs as well and thus wins. | 11:14 |
| *** fawadkhaliq has quit IRC | 11:14 | |
| mgould | AJaeger: thanks! | 11:14 |
| *** yolanda has joined #openstack-infra | 11:14 | |
| *** fawadkhaliq has joined #openstack-infra | 11:15 | |
| AJaeger | you need to split up your change - add to that skip if to exclcude the triploe-ci, and handle the rest elsewhere | 11:15 |
| AJaeger | And then double check your chnage ,please - these skip-ifs are indeed nasty | 11:15 |
| mgould | right, will do | 11:16 |
| * AJaeger has seen more than once questions here: Why does that job not run - and skip-if was often the answer ;( | 11:16 | |
| mgould | is there a less nasty way of achieving what I want? | 11:16 |
| AJaeger | That's why I'd like broad consensus, so that you know what you do... | 11:16 |
| apuimedo | Hi guys! Time for the daily kuryr related questions :$ | 11:16 |
| apuimedo | as part of the recommended split of our packages into three repos | 11:16 |
| AJaeger | mgould: There's the files conditions which means: Run this job only if certain files change. But that's difficult for some kind of jobs to do, have a look how it's used in layout.yaml | 11:17 |
| apuimedo | infra created the new kuryr-libnetwork and kuryr-kubernetes | 11:17 |
| apuimedo | and asked us if they should be cloned from somewhere else | 11:17 |
| apuimedo | or if they should be created anew | 11:17 |
| *** yolanda has quit IRC | 11:17 | |
| *** electrofelix has quit IRC | 11:17 | |
| *** fawadkhaliq has quit IRC | 11:18 | |
| *** fawadkhaliq has joined #openstack-infra | 11:18 | |
| *** claudiub|2 has joined #openstack-infra | 11:18 | |
| AJaeger | apuimedo: yeah - that was your one time chnage for importing it. Now everything needs to go through gerrit. | 11:19 |
| apuimedo | we said "anew". This was the right decision for kuryr-kubernetes. But it turns out it is not the best course of action for kuryr-libnetwork, which should have been cloned from 'kuryr' (and then start dropping parts in both 'kuryr' and kuryr-libnetwork) | 11:19 |
| apuimedo | AJaeger: good. I wanted to know if it was possibe to have a do-over for that one to preserve history | 11:19 |
| apuimedo | but if it is not, well, what can we do, we fscked up :P | 11:20 |
| *** rodrigods has quit IRC | 11:20 | |
| apuimedo | we'll go through gerrit | 11:20 |
| *** rodrigods has joined #openstack-infra | 11:20 | |
| AJaeger | apuimedo: at project creation, it's an automatic import. | 11:20 |
| *** lucasagomes is now known as lucas-hungry | 11:20 | |
| AJaeger | Anything to do now, would be a manual setup - and for that you need to discuss with one of the admins - I'm not one - what is possible and whether somebody has time to do this. | 11:21 |
| apuimedo | AJaeger: understood | 11:21 |
| AJaeger | apuimedo: That's why we ask explicitely at project creation time - to save all of us time on it. | 11:21 |
| *** claudiub has quit IRC | 11:21 | |
| apuimedo | I also considered deleting the project and creating it again :P | 11:21 |
| apuimedo | (the subproject) | 11:21 |
| apuimedo | but that feels a bit weird too | 11:22 |
| AJaeger | apuimedo: manual task as well ;) | 11:22 |
| *** electrofelix has joined #openstack-infra | 11:22 | |
| apuimedo | ok | 11:22 |
| apuimedo | thanks a lot for the info AJaeger | 11:22 |
| AJaeger | apuimedo: come back later (US time) and see whether an admin is able to help. | 11:22 |
| apuimedo | AJaeger: how do I find out who the admins are? | 11:22 |
| *** fawadk has joined #openstack-infra | 11:23 | |
| *** fawadkhaliq has quit IRC | 11:23 | |
| AJaeger | apuimedo: just ask here and one will answer if they're awake ;) | 11:23 |
| *** abregman|afk is now known as abregman | 11:24 | |
| apuimedo | very well! | 11:24 |
| apuimedo | thanks again | 11:24 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for GitLab notifier and connection property https://review.openstack.org/330178 | 11:24 |
| *** sdake has quit IRC | 11:25 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Use 'false' and 'true' in default docstrings https://review.openstack.org/331220 | 11:25 |
| *** amotoki has joined #openstack-infra | 11:25 | |
| *** abregman is now known as abregman|afk | 11:25 | |
| *** Liuqing has quit IRC | 11:26 | |
| *** fawadk has quit IRC | 11:26 | |
| *** fawadkhaliq has joined #openstack-infra | 11:26 | |
| *** alaski is now known as lascii | 11:27 | |
| *** salv-orl_ has joined #openstack-infra | 11:30 | |
| *** bradm_ is now known as bradm | 11:30 | |
| *** fawadkhaliq has quit IRC | 11:31 | |
| *** ldnunes has joined #openstack-infra | 11:31 | |
| *** fawadkhaliq has joined #openstack-infra | 11:31 | |
| *** markvoelker has joined #openstack-infra | 11:31 | |
| *** amotoki has quit IRC | 11:33 | |
| *** salv-orlando has quit IRC | 11:33 | |
| *** amoralej is now known as amoralej|lunch | 11:33 | |
| *** eezhova has joined #openstack-infra | 11:34 | |
| *** yolanda has joined #openstack-infra | 11:35 | |
| *** sdague has joined #openstack-infra | 11:35 | |
| *** markvoelker has quit IRC | 11:36 | |
| openstackgerrit | Merged openstack/diskimage-builder: Export YUM variable in centos bases https://review.openstack.org/336471 | 11:36 |
| *** chem has quit IRC | 11:36 | |
| *** fawadkhaliq has quit IRC | 11:37 | |
| *** PollyZ has joined #openstack-infra | 11:37 | |
| *** fawadkhaliq has joined #openstack-infra | 11:37 | |
| *** fawadkhaliq has quit IRC | 11:39 | |
| *** fawadkhaliq has joined #openstack-infra | 11:39 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci https://review.openstack.org/324777 | 11:41 |
| *** PollyZ has quit IRC | 11:42 | |
| *** fawadkhaliq has quit IRC | 11:42 | |
| *** fawadkhaliq has joined #openstack-infra | 11:42 | |
| *** eezhova has quit IRC | 11:42 | |
| *** Na3iL has quit IRC | 11:42 | |
| *** fawadk has joined #openstack-infra | 11:46 | |
| *** fawadkhaliq has quit IRC | 11:46 | |
| *** javeriak has quit IRC | 11:47 | |
| *** fawadk has quit IRC | 11:48 | |
| stevemar | AJaeger: silly question about 336344 | 11:48 |
| *** fawadkhaliq has joined #openstack-infra | 11:48 | |
| stevemar | isn't it already in check and gate? | 11:48 |
| *** ddieterly has joined #openstack-infra | 11:48 | |
| stevemar | oh no, it's not :( | 11:48 |
| openstackgerrit | Merged openstack-infra/storyboard: Add a way to retrieve the history of edited comments https://review.openstack.org/333409 | 11:50 |
| *** fawadkhaliq has quit IRC | 11:50 | |
| *** ccamacho is now known as ccamacho|lunch | 11:50 | |
| *** ldnunes has quit IRC | 11:50 | |
| *** eezhova has joined #openstack-infra | 11:50 | |
| *** fawadkhaliq has joined #openstack-infra | 11:50 | |
| *** ldnunes has joined #openstack-infra | 11:50 | |
| *** PollyZ has joined #openstack-infra | 11:50 | |
| *** eezhova has quit IRC | 11:51 | |
| *** yamamoto has quit IRC | 11:52 | |
| *** mrmartin has joined #openstack-infra | 11:54 | |
| AJaeger | stevemar: I didn't check - but since non-voting jobs are only in check, I would be surprised if it's already in both ;) | 11:54 |
| *** edmondsw has joined #openstack-infra | 11:54 | |
| stevemar | AJaeger: no you are right | 11:54 |
| stevemar | AJaeger: i think http://paste.openstack.org/show/524551/ should work? | 11:54 |
| stevemar | AJaeger: i will send out a note to the ML and ask project PTLs to look at it | 11:55 |
| *** itisha has joined #openstack-infra | 11:55 | |
| *** pt_15 has joined #openstack-infra | 11:55 | |
| *** PollyZ has quit IRC | 11:55 | |
| *** Daisy has quit IRC | 11:55 | |
| Zara | we've noticed a test reporting success that actually errors out (and then fails in the gate)-- the linting test as seen on this patch: https://review.openstack.org/#/c/331748/ | 11:55 |
| Zara | wondering if anyone knows what's up? | 11:56 |
| *** Daisy has joined #openstack-infra | 11:56 | |
| Zara | it's possible I've missed a memo, been ill this week | 11:56 |
| *** javeriak has joined #openstack-infra | 11:56 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: set check-osc-job to voting https://review.openstack.org/336344 | 11:57 |
| *** fawadkhaliq has quit IRC | 11:58 | |
| *** fawadk has joined #openstack-infra | 11:58 | |
| AJaeger | Zara: looking at http://logs.openstack.org/48/331748/2/gate/gate-storyboard-webclient-npm-run-lint/36f22df/console.html#_2016-07-01_11_07_01_426056, it looks like broken npm packages ;( | 12:00 |
| AJaeger | stevemar: you cannot combine this together - let me check your patch in detail... | 12:00 |
| AJaeger | stevemar: we have separate template sections - and conditions. | 12:01 |
| AJaeger | Your first need to create a template to use it - and the conditions apply to jobs. | 12:01 |
| *** raildo-afk is now known as raildo | 12:01 | |
| *** asettle has joined #openstack-infra | 12:02 | |
| AJaeger | Zara, try a recheck | 12:03 |
| Zara | AJaeger: hm, I think it should fail; I'm wondering why it reports as success when the error is present... the error itself isn't weird; there's an extraneous variable. http://logs.openstack.org/48/331748/2/check/gate-storyboard-webclient-npm-run-lint/441c8c5/console.html.gz#_2016-06-21_11_28_28_920865 | 12:03 |
| *** fawadkhaliq has joined #openstack-infra | 12:03 | |
| AJaeger | Zara, ahh, that one - let me show you a fix... | 12:04 |
| *** fawadk has quit IRC | 12:04 | |
| *** PollyZ has joined #openstack-infra | 12:04 | |
| AJaeger | Zara, fixed with https://review.openstack.org/334582 | 12:04 |
| *** markusry has joined #openstack-infra | 12:04 | |
| AJaeger | And your job was run before that - so, the recheck I just triggered will make it fail | 12:04 |
| *** maishsk has quit IRC | 12:05 | |
| *** mrmartin has quit IRC | 12:05 | |
| Zara | aha | 12:05 |
| Zara | AJaeger: that explains it, then; the first check was run before that. thanks for the link! :) | 12:06 |
| *** aysyd has joined #openstack-infra | 12:06 | |
| *** ddieterly has quit IRC | 12:07 | |
| *** asettle has quit IRC | 12:07 | |
| *** fawadkhaliq has quit IRC | 12:07 | |
| *** fawadkhaliq has joined #openstack-infra | 12:08 | |
| *** jed56 has joined #openstack-infra | 12:09 | |
| mordred | apuimedo: I can do an import for you | 12:09 |
| *** PollyZ has quit IRC | 12:09 | |
| openstackgerrit | Miles Gould proposed openstack-infra/project-config: Skip non-doc checks for docs-only TripleO patches https://review.openstack.org/336049 | 12:09 |
| *** derekjhyang has quit IRC | 12:11 | |
| *** Daisy has quit IRC | 12:12 | |
| *** Daisy has joined #openstack-infra | 12:13 | |
| *** fawadk has joined #openstack-infra | 12:13 | |
| *** asselin__ has joined #openstack-infra | 12:14 | |
| *** fawadkhaliq has quit IRC | 12:14 | |
| openstackgerrit | Miles Gould proposed openstack-infra/project-config: Skip non-doc checks for docs-only TripleO patches https://review.openstack.org/336049 | 12:14 |
| *** Na3iL has joined #openstack-infra | 12:14 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: set check-osc-job to voting https://review.openstack.org/336344 | 12:16 |
| *** kdas_ has joined #openstack-infra | 12:16 | |
| *** erlon has joined #openstack-infra | 12:16 | |
| *** markvoelker has joined #openstack-infra | 12:16 | |
| openstackgerrit | Merged openstack-infra/system-config: Do not use async with zl restarter playbook https://review.openstack.org/336317 | 12:17 |
| *** Daisy has quit IRC | 12:17 | |
| *** bauzas is now known as bauwser | 12:17 | |
| *** fawadkhaliq has joined #openstack-infra | 12:18 | |
| *** PollyZ has joined #openstack-infra | 12:18 | |
| *** amotoki has joined #openstack-infra | 12:18 | |
| *** fawadkhaliq has quit IRC | 12:18 | |
| *** kushal is now known as Guest29039 | 12:18 | |
| *** psilvad has joined #openstack-infra | 12:19 | |
| *** pradk has joined #openstack-infra | 12:19 | |
| *** fawadk has quit IRC | 12:20 | |
| *** Guest29039 has quit IRC | 12:20 | |
| *** afazekas_ is now known as afazekas|real | 12:20 | |
| *** andymaier has quit IRC | 12:20 | |
| *** zumbi has quit IRC | 12:21 | |
| *** rlandy has joined #openstack-infra | 12:21 | |
| *** zumbi has joined #openstack-infra | 12:21 | |
| *** kdas_ is now known as kushal | 12:22 | |
| *** kushal has joined #openstack-infra | 12:22 | |
| *** PollyZ has quit IRC | 12:22 | |
| *** lucas-hungry is now known as lucasagomes | 12:23 | |
| *** ddieterly has joined #openstack-infra | 12:24 | |
| *** inc0 has joined #openstack-infra | 12:24 | |
| *** nwkarsten has quit IRC | 12:25 | |
| *** fawadkhaliq has joined #openstack-infra | 12:25 | |
| *** nwkarsten has joined #openstack-infra | 12:25 | |
| *** baoli has joined #openstack-infra | 12:27 | |
| *** junoyoon has joined #openstack-infra | 12:27 | |
| *** amotoki has quit IRC | 12:27 | |
| *** eezhova has joined #openstack-infra | 12:27 | |
| *** baoli_ has joined #openstack-infra | 12:28 | |
| *** ddieterly has quit IRC | 12:29 | |
| junoyoon | Hi. while saving the contact info in https://review.openstack.org/#/settings/contact, it shows the "Code Review - Error Server Error Cannot store contact information". Is there someone to look into this? | 12:29 |
| *** _nadya_ has quit IRC | 12:30 | |
| *** fawadkhaliq has quit IRC | 12:30 | |
| *** nwkarsten has quit IRC | 12:30 | |
| *** _nadya_ has joined #openstack-infra | 12:30 | |
| *** zumbi has quit IRC | 12:31 | |
| *** PollyZ has joined #openstack-infra | 12:31 | |
| *** baoli has quit IRC | 12:32 | |
| frickler | junoyoon: please look at https://ask.openstack.org/en/question/56720/cannot-store-contact-information-when-updating-info-in-openstack-gerrit/ and see if that helps | 12:32 |
| *** asselin__ has quit IRC | 12:34 | |
| *** _nadya_ has quit IRC | 12:35 | |
| junoyoon | frickler: Thanks!! It now works/ | 12:35 |
| *** yolanda has quit IRC | 12:35 | |
| *** trown|outtypewww is now known as trown | 12:36 | |
| *** PollyZ has quit IRC | 12:36 | |
| *** yolanda has joined #openstack-infra | 12:36 | |
| *** javeriak has quit IRC | 12:37 | |
| *** javeriak has joined #openstack-infra | 12:37 | |
| *** zumbi has joined #openstack-infra | 12:38 | |
| *** claudiub|2 has quit IRC | 12:38 | |
| *** asettle has joined #openstack-infra | 12:41 | |
| *** PollyZ has joined #openstack-infra | 12:45 | |
| *** yamamoto has joined #openstack-infra | 12:45 | |
| *** ddieterly has joined #openstack-infra | 12:46 | |
| *** asettle has quit IRC | 12:47 | |
| *** rfolco has joined #openstack-infra | 12:47 | |
| *** esikachev has quit IRC | 12:48 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for Clang Scan-Build plugin version 1.7 https://review.openstack.org/330150 | 12:49 |
| *** matt-borland has joined #openstack-infra | 12:50 | |
| *** PollyZ has quit IRC | 12:50 | |
| *** jsavak has joined #openstack-infra | 12:50 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Update OSA job skip for all jobs except docs/releasenotes https://review.openstack.org/331574 | 12:51 |
| odyssey4me | AJaeger ^ I think that one is right. | 12:51 |
| *** pfallenop has quit IRC | 12:52 | |
| *** pfallenop has joined #openstack-infra | 12:52 | |
| *** pfallenop has joined #openstack-infra | 12:52 | |
| *** salv-orl_ has quit IRC | 12:53 | |
| *** d0ugal has quit IRC | 12:54 | |
| *** d0ugal has joined #openstack-infra | 12:54 | |
| *** d0ugal has quit IRC | 12:54 | |
| *** d0ugal has joined #openstack-infra | 12:54 | |
| *** jsavak has quit IRC | 12:54 | |
| *** d0ugal has quit IRC | 12:55 | |
| *** d0ugal has joined #openstack-infra | 12:55 | |
| *** d0ugal has quit IRC | 12:55 | |
| *** d0ugal has joined #openstack-infra | 12:55 | |
| *** jsavak has joined #openstack-infra | 12:55 | |
| AJaeger | odyssey4me: skip-ifs are dangerous, I need to be fully awake and concentrated to review them ;). Might not come to that today - and need the log files to double check as well. | 12:55 |
| *** ddieterly has quit IRC | 12:56 | |
| *** javeriak has quit IRC | 12:57 | |
| *** PollyZ has joined #openstack-infra | 12:57 | |
| *** esikachev has joined #openstack-infra | 12:57 | |
| *** tlian has joined #openstack-infra | 12:58 | |
| odyssey4me | AJaeger yep, no problem :) | 12:58 |
| apuimedo | mordred: that would be awesome | 12:59 |
| mordred | apuimedo: you want the current contents of the kuryr repo pushed over top of the current contents of kuryr-libnetwork? | 13:00 |
| *** d0ugal has quit IRC | 13:00 | |
| apuimedo | mordred: yes, exactly | 13:00 |
| *** d0ugal has joined #openstack-infra | 13:00 | |
| *** PollyZ has quit IRC | 13:00 | |
| apuimedo | since it is a split | 13:00 |
| apuimedo | it makes sense that the new repo takes the content of the old and trims away | 13:00 |
| apuimedo | than to put everything anew | 13:01 |
| mordred | ++ | 13:01 |
| *** dizquierdo has joined #openstack-infra | 13:02 | |
| *** binbincong has quit IRC | 13:02 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI https://review.openstack.org/336312 | 13:02 |
| *** bin_ has joined #openstack-infra | 13:02 | |
| *** d0ugal has quit IRC | 13:02 | |
| Zara | during the sprint, people asked about making task notes read only by default. there's a patch in review here (I can merge, just want to check people want it so we don't end up undoing it again later): https://review.openstack.org/#/c/325294/ now's the time to indicate if you feel strongly. :) | 13:02 |
| apuimedo | thanks ;-) | 13:02 |
| *** d0ugal has joined #openstack-infra | 13:03 | |
| *** d0ugal has quit IRC | 13:03 | |
| *** d0ugal has joined #openstack-infra | 13:03 | |
| *** d0ugal has quit IRC | 13:03 | |
| *** markusry has quit IRC | 13:03 | |
| mordred | apuimedo: done | 13:03 |
| mordred | apuimedo: I also put up a gitreview patch to change it | 13:03 |
| openstackgerrit | James Page proposed openstack-infra/project-config: Add repositories for Charm interfaces https://review.openstack.org/334502 | 13:03 |
| *** Guest12790 has joined #openstack-infra | 13:03 | |
| apuimedo | great! | 13:03 |
| apuimedo | mordred: which patch? | 13:04 |
| *** Guest12790 has quit IRC | 13:04 | |
| apuimedo | oh, of course | 13:04 |
| apuimedo | .gitreview | 13:04 |
| *** junoyoon has quit IRC | 13:04 | |
| *** d0ugal has joined #openstack-infra | 13:04 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI https://review.openstack.org/336312 | 13:04 |
| apuimedo | thanks for that | 13:04 |
| apuimedo | very thoughtful | 13:04 |
| *** Ravikiran_K has joined #openstack-infra | 13:04 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Update eavesdrop site name https://review.openstack.org/336552 | 13:05 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Rename the eavesdrop site to be clearer https://review.openstack.org/331503 | 13:05 |
| *** kushal has quit IRC | 13:06 | |
| *** markusry has joined #openstack-infra | 13:08 | |
| *** cshastri has joined #openstack-infra | 13:10 | |
| *** amoralej|lunch is now known as amoralej | 13:11 | |
| fungi | AJaeger: rax-dfw images should eventually update. probably ran into an upload issue yesterday or something | 13:14 |
| *** psilvad has quit IRC | 13:15 | |
| *** psilvad has joined #openstack-infra | 13:15 | |
| *** psachin has quit IRC | 13:16 | |
| *** fawadkhaliq has joined #openstack-infra | 13:17 | |
| *** fawadkhaliq has quit IRC | 13:21 | |
| *** azvyagintsev has joined #openstack-infra | 13:21 | |
| *** amotoki has joined #openstack-infra | 13:21 | |
| *** psilvad has quit IRC | 13:21 | |
| *** psilvad has joined #openstack-infra | 13:22 | |
| *** abregman|afk is now known as abregman | 13:24 | |
| AJaeger | ok, so I'll wait another day and check again. | 13:24 |
| azvyagintsev | Hi folks and gurus ! i'm thinking about implementation CI for openstack/murano-apps . current idea is - combine hardware lab , and devstack-murano slave(via nodepool) (i guess, currently it works in that way in openstack/murano-deployment ci) | 13:25 |
| yolanda | hi mordred, around? can you review some of my changes with topic puppet-infracloud? | 13:25 |
| azvyagintsev | i can flame with someone from infra about this ? exactly - about best practice for devstack + *some_app* + hardware lab | 13:25 |
| AJaeger | fungi, do you want to merge clarkb's python35 patch today - or wait until Tuesday? Depending on that I'd like to push - or not - my rework of the publishing jobs https://review.openstack.org/336405. Those two conflict heavily and if mine merges first, I'll rebase clarkb's. | 13:25 |
| fungi | AJaeger: clarkb indicated he expected the py35 change to need some rebase treadmill, so merging your publishing job rework seems fine | 13:27 |
| *** psilvad has quit IRC | 13:27 | |
| *** mriedem has joined #openstack-infra | 13:27 | |
| *** signed8bit has joined #openstack-infra | 13:28 | |
| *** psilvad has joined #openstack-infra | 13:28 | |
| fungi | we were just hesitant to push the py35 change heading into a holiday weekend for a significant chunk of our community (canada is off today, usa monday) | 13:28 |
| AJaeger | fungi, frickler was so nice to rebase already, so clarkb's one is good to merge right now... | 13:28 |
| *** sree has joined #openstack-infra | 13:28 | |
| *** akshai has joined #openstack-infra | 13:28 | |
| *** ayoung has joined #openstack-infra | 13:29 | |
| AJaeger | fungi, understood - even if I see that myself not as problematic since the jobs are non-voting... | 13:29 |
| fungi | AJaeger: if clarkb pops in here in a bit, we can revisit | 13:30 |
| *** ddieterly has joined #openstack-infra | 13:30 | |
| *** ddieterly is now known as ddieterly[away] | 13:30 | |
| AJaeger | yeah, if we do the python35 today, we should do it early today - in case of problems... | 13:31 |
| *** kushal has joined #openstack-infra | 13:33 | |
| *** eezhova has quit IRC | 13:34 | |
| *** dansmith is now known as superdan | 13:34 | |
| *** tpsilva has joined #openstack-infra | 13:34 | |
| *** Liuqing has joined #openstack-infra | 13:36 | |
| *** bryan_att has joined #openstack-infra | 13:37 | |
| *** Na3iL has quit IRC | 13:38 | |
| fungi | azvyagintsev: chances are you'll find people in the third-party ci meetings with more experience in running a ci on bare metal. everything we do upstream is currently in virtual server instances | 13:38 |
| *** ddieterly[away] is now known as ddieterly | 13:39 | |
| *** Jeffrey4l has quit IRC | 13:40 | |
| azvyagintsev | fungi: thx! hm, did you ran vm inside vm ?(not cirros, ubuntu\debian at least) (i mean, at least for something, is nested performance enough for tests ?) | 13:40 |
| *** ajmiller has quit IRC | 13:41 | |
| *** ajmiller has joined #openstack-infra | 13:41 | |
| *** abregman is now known as abregman|afk | 13:42 | |
| fungi | azvyagintsev: there is a little of that, though performance is pretty terrible for that sort of thing so mocked interactions and avoiding nested virtualization where possible usually provide sufficient alternatives | 13:42 |
| openstackgerrit | Merged openstack-infra/puppet-jenkins: Remove unused GnuPG keyring https://review.openstack.org/333529 | 13:42 |
| openstackgerrit | Merged openstack-infra/system-config: Project renames may need governance updates https://review.openstack.org/330190 | 13:42 |
| azvyagintsev | ok, thanks | 13:42 |
| fungi | azvyagintsev: for example, beaker normally expects to run things in a virtual machine/container, but we configure it to treat the local system as if it were a remote vm | 13:43 |
| openstackgerrit | Merged openstack-infra/system-config: Filter nonexistent groups in who-approves.py https://review.openstack.org/292594 | 13:43 |
| *** daemontool has joined #openstack-infra | 13:43 | |
| fungi | azvyagintsev: and when complete contextual separation is required, we run jobs that span multiple virtual machines rather than trying to nest multiple virtual machines inside a virtual machine | 13:44 |
| openstackgerrit | Merged openstack-infra/system-config: Publish eavesdrop to yaml2ical dir https://review.openstack.org/331502 | 13:44 |
| fungi | azvyagintsev: for example, jobs that utilize a multi-node devstack deployment | 13:45 |
| *** eezhova has joined #openstack-infra | 13:45 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Adding property-file into the pipeline publisher https://review.openstack.org/311150 | 13:45 |
| *** ajmiller has quit IRC | 13:46 | |
| *** ajmiller has joined #openstack-infra | 13:47 | |
| azvyagintsev | hmm, lost point - 'we run jobs that span multiple virtual machines' - it's not for case, when node should spawn VM inside openstack , right ? | 13:47 |
| *** gouthamr has joined #openstack-infra | 13:47 | |
| *** thiagop has joined #openstack-infra | 13:48 | |
| *** r-mibu has quit IRC | 13:48 | |
| *** r-mibu has joined #openstack-infra | 13:48 | |
| *** asettle has joined #openstack-infra | 13:49 | |
| openstackgerrit | Merged openstack-infra/git-review: Use consistent formatting for deprecations https://review.openstack.org/326538 | 13:50 |
| *** rbrndt has joined #openstack-infra | 13:50 | |
| *** gouthamr_ has joined #openstack-infra | 13:51 | |
| *** cshastri has quit IRC | 13:52 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Adds matrix restriction to trigger-builds plugin https://review.openstack.org/331112 | 13:53 |
| fungi | azvyagintsev: well, you can perhaps test those two things separately? e.g., test that your software is able to boot a vm (something small like cirros), and then test your software that should run in that vm by installing it locally on the job node | 13:53 |
| AJaeger | fungi, do we have enough resources to take roughly 40 more repos on? We have around 20 new fuel repos and 15+ charm repos in the project-config queue.. | 13:54 |
| fungi | oh wow | 13:54 |
| fungi | well, additional repos don't themselves necessarily consume any additional "resources" | 13:54 |
| *** gouthamr has quit IRC | 13:55 | |
| openstackgerrit | Merged openstack-infra/system-config: Add a node for artifact signing jobs https://review.openstack.org/333065 | 13:55 |
| AJaeger | disk space + jobs ;) | 13:55 |
| fungi | the majority of new repos add very few jobs | 13:55 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add "every online node" factory to Parameterized Trigger builder https://review.openstack.org/321313 | 13:55 |
| AJaeger | nothing out of the normal from what I can see for each, just more repos than usual ;) | 13:55 |
| *** gouthamr_ is now known as gouthamr | 13:55 | |
| *** sree has quit IRC | 13:56 | |
| *** sorantis has quit IRC | 13:56 | |
| azvyagintsev | fungi: that an main problem- no i can't. i'm going to test murano-apps. for this i need openstack + murano , then i import murano-app => run it => app spawn 1-5 VMs, and so on | 13:56 |
| * AJaeger hasn't reviewed in full detail yet, just did some glance over | 13:56 | |
| azvyagintsev | so its require pretty huge(in tests meaning) resources | 13:56 |
| fungi | azvyagintsev: yep, sounds like a good case for a third-party ci system then. that's pretty far out of scope for our existing test infrastructure | 13:58 |
| fungi | azvyagintsev: our ci manages virtual machines directly, you want jobs that manage virtual machines instead. you're likely going to have some additional struggles with protecting credentials, so you likely need something like what nodepool does, but for creating and destroying tenants in your test cloud rather than creating and destroying individual nova instances | 14:00 |
| *** mhickey has joined #openstack-infra | 14:00 | |
| *** eharney has joined #openstack-infra | 14:00 | |
| azvyagintsev | ah, so anyway thanks :) for me- it's pretty clean how-to achieve it, but not clean - how i should integrate my idea with infra. so, if you sure that it's fully for third-party ci...i don't have q for you :) thx one more time! | 14:01 |
| fungi | azvyagintsev: that way if a job leaks the keystone credentials for the tenant it's using, that doesn't matter because you're going to blow that tenant away as soon as your job completes and use a different tenant for the next job | 14:01 |
| *** amitgandhinz has quit IRC | 14:01 | |
| azvyagintsev | yeah, i think about tenant-per-job, so, i don't want to recreate it each run, at least for now | 14:01 |
| *** amitgandhinz has joined #openstack-infra | 14:02 | |
| *** Liuqing has quit IRC | 14:02 | |
| fungi | azvyagintsev: if you reuse tenants, i wonder how you prevent a test for a random gerrit change from publishing the credentials for the tenant it uses | 14:02 |
| *** tonytan4ever has joined #openstack-infra | 14:03 | |
| azvyagintsev | yup, but as i saw - third-party CI usually use internal openstack, w\o external access | 14:03 |
| fungi | azvyagintsev: unless you consider that running untrusted changes from gerrit actually _is_ a form of external access | 14:04 |
| *** yolanda has quit IRC | 14:04 | |
| *** esikachev has quit IRC | 14:05 | |
| *** yolanda has joined #openstack-infra | 14:05 | |
| fungi | not a ton of difference between "i can log into your environment and run something" or "i can post something to gerrit and get it to run in your environment" | 14:05 |
| azvyagintsev | some kind. bit test tenant don't have admin access and so on...but yes, i guess you right. not its work on you'r side ? some pre-job hook ? | 14:05 |
| azvyagintsev | *but test | 14:06 |
| *** Liuqing has joined #openstack-infra | 14:06 | |
| *** yamamoto has quit IRC | 14:06 | |
| azvyagintsev | i mean, how exactly you create test-tenant, w\o sharing admin tenant creds in logs ? | 14:06 |
| *** tonytan4ever has quit IRC | 14:07 | |
| *** tonytan4ever has joined #openstack-infra | 14:07 | |
| fungi | like i said, one way to do it is with a separate service. like how we entrust keystone credentials to nodepool and have it create virtual machines, then zuul assigns jobs to those virtual machines. you could write a service which has your cloud admin credentials and creates tenants, which your job scheduler then assigns jobs to | 14:08 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: pingtest: don't show error if no images needs to be cleaned-up https://review.openstack.org/335655 | 14:10 |
| *** david-lyle has quit IRC | 14:10 | |
| azvyagintsev | catch it. ok, thx for describing | 14:11 |
| *** kgiusti has joined #openstack-infra | 14:11 | |
| *** daemontool has quit IRC | 14:12 | |
| azvyagintsev | btw, sounds like - we need another one tool, tenant-pool :D | 14:14 |
| *** daemontool has joined #openstack-infra | 14:14 | |
| *** TaylorH has joined #openstack-infra | 14:19 | |
| *** Srinu has joined #openstack-infra | 14:19 | |
| Srinu | Hi | 14:19 |
| Srinu | Asselin_: hi | 14:20 |
| Srinu | Asselin_: can we use two jenkins master in the same network r not | 14:21 |
| Srinu | #openstack-cinder | 14:22 |
| *** TaylorHuang has quit IRC | 14:22 | |
| *** yamamoto has joined #openstack-infra | 14:23 | |
| fungi | azvyagintsev: it's also possible that would be a worthwhile extension to nodepool (the resource tracking framework itself could probably be entirely reused for that at least), but i don't know. it would be a pretty large scope change so i'm not sure how i feel about that. would want to hear feedback from other nodepool devs | 14:23 |
| Srinu | Can we use two jenkins master in the same network | 14:24 |
| Srinu | Fungi: hi | 14:24 |
| mordred | fungi, azvyagintsev: I think it would not be a terrible thing to add to nodepool - and would work nicely even in upstream infra when our clouds grow domain support so that we could have nodepool have domain-admin | 14:25 |
| *** links has quit IRC | 14:25 | |
| fungi | Srinu: we used to have 8 jenkins masters in one network, tied to a single instance of zuul. so yes, that is possible. it's the reason we originally created the jenkins-gearman plugin and gearman-based zuul v2 | 14:25 |
| mordred | fungi, azvyagintsev: it would, however, absolutely require domain support in our clouds for infra to run such a thing | 14:25 |
| azvyagintsev | yeah, its perfectly for nodepool - just injecting some pre-generated operc file, and then connect it like env-file for slave. sounds good | 14:25 |
| mordred | fungi, azvyagintsev: although I recommend s/tenant/project/ in your brains :) | 14:26 |
| fungi | s/nodepool/cloudpool/ | 14:26 |
| mordred | \o/ | 14:26 |
| azvyagintsev | )) | 14:26 |
| fungi | my brain will still always have tenant stuck in it, since that's what we public cloud providers called them way back in the beforetime, pre-openstack | 14:27 |
| Srinu | Fungi: i have one jenkins master with gearman enable and how can i add anther jenkins to it | 14:27 |
| fungi | Srinu: you just connect the additional jenkins to gearman the same way you did the first one | 14:28 |
| *** links has joined #openstack-infra | 14:28 | |
| fungi | Srinu: and if you're using nodepool, configure it with the new jenkins master as a second "target" | 14:28 |
| fungi | Srinu: and also as a second zmq-publisher | 14:29 |
| fungi | nodepool and zuul are designed to work with an arbitrary number of jenkins masters | 14:30 |
| Srinu | Fungi:i installed 2 jenkins master with 2 different slaves. I enable gearman in both the jenkins with different zuul | 14:30 |
| *** zhurong has joined #openstack-infra | 14:30 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/diskimage-builder: Remove python-setuptools from yum excludes https://review.openstack.org/336586 | 14:31 |
| fungi | Srinu: why a different zuul? you should be able to have one zuul with multiple jenkins masters | 14:31 |
| *** esikachev has joined #openstack-infra | 14:31 | |
| *** xyang1 has joined #openstack-infra | 14:32 | |
| clarkb | AJaeger: fungi I guess we could merge it today if we want. we should send email explaining what new job is, details on how to add py35 tox env if necessary and how to make job voting if/when it passes | 14:32 |
| clarkb | I have not drafted that yet and wont be able to for a biy due to morning chores | 14:33 |
| *** baoli_ has quit IRC | 14:33 | |
| *** baoli has joined #openstack-infra | 14:34 | |
| AJaeger | morning, clarkb. I guess, in that case, we should draft - or even send - the email today and merge on Tuesday. | 14:34 |
| *** salv-orlando has joined #openstack-infra | 14:34 | |
| *** Srinu has quit IRC | 14:34 | |
| AJaeger | clarkb: then my offer stands: If https://review.openstack.org/#/c/336405/ merges earlier, I'll rebase the your python35 change. | 14:36 |
| * AJaeger now needs to do some chores as well... | 14:36 | |
| clarkb | AJaeger: ya dont worry about conflicting with it, I or you or frickler (thhanks!) can rebase as necessary | 14:36 |
| AJaeger | clarkb: my change removes 800 jobs... | 14:38 |
| AJaeger | bbl | 14:38 |
| clarkb | AJaeger: ooh I should review it then :) | 14:38 |
| *** armax has joined #openstack-infra | 14:39 | |
| AJaeger | clarkb: and some low-hanging fruits: https://review.openstack.org/#/c/336413/ and https://review.openstack.org/#/c/336263/ - that's all cleanup I came up with... | 14:39 |
| AJaeger | now really until later ;) | 14:39 |
| *** Srinu has joined #openstack-infra | 14:40 | |
| *** salv-orlando has quit IRC | 14:40 | |
| Srinu | Fungi: can you suggest me links to do that | 14:41 |
| *** slaweq has quit IRC | 14:41 | |
| mordred | clarkb: the change looks good to me and already has another +2 - do you want me to hold off on approving so you can look at it? | 14:41 |
| *** zz_dimtruck is now known as dimtruck | 14:41 | |
| apuimedo | dhellmann: mordred mentioned in the ml that the use of python packages can bring some issues and that you could tell me more about it | 14:42 |
| mordred | apuimedo: fungi might also remember the issue that oslo ran in to with namespace packages | 14:42 |
| *** yamamoto has quit IRC | 14:43 | |
| apuimedo | yes, they moved from oslo.config to oslo_config, right? | 14:44 |
| *** snarwade has quit IRC | 14:44 | |
| Srinu | Fungi:hi | 14:45 |
| *** baoli_ has joined #openstack-infra | 14:45 | |
| Srinu | Jeblair:hi | 14:45 |
| clarkb | mordred: nope go for it | 14:46 |
| Srinu | Clarkb: hi | 14:46 |
| clarkb | Srinu: hello | 14:46 |
| Srinu | Clarkb: how can i make 2 jenkins to use single zuul to get the streams by enabling gearman | 14:47 |
| *** baoli has quit IRC | 14:48 | |
| clarkb | Srinu: configure them to talk to the same gearman server as zuul | 14:48 |
| Srinu | Clarkb: i have two separate jenkins with different slaves.. | 14:49 |
| mordred | Srinu: one gearman server. zuul talks to gearman. each jenkins talks to gearman. | 14:49 |
| Srinu | Clarkb: i want to make to listen same zuul | 14:49 |
| Srinu | Clarkb: i enable gearman in both jenkins | 14:50 |
| *** chem has joined #openstack-infra | 14:50 | |
| *** xarses has quit IRC | 14:50 | |
| *** dingyichen has quit IRC | 14:53 | |
| Srinu | Mordred: i enable gearman in both jenkins masterp | 14:53 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Properties: fix copyartifact permissions with several projects https://review.openstack.org/334656 | 14:54 |
| *** pilgrimstack has quit IRC | 14:54 | |
| *** pilgrimstack has joined #openstack-infra | 14:55 | |
| *** arxcruz has quit IRC | 14:55 | |
| clarkb | Srinu: just make sure that zuul and the jenkins masters use the same gearman server and it will do what you want | 14:56 |
| *** infra-red has joined #openstack-infra | 14:56 | |
| *** yolanda has quit IRC | 14:56 | |
| *** yolanda has joined #openstack-infra | 14:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Rework publishing jobs https://review.openstack.org/336405 | 14:57 |
| Srinu | Clarkb: in another jenkins master i have to use gearman ip to connect to first jenkins gearman right | 14:57 |
| *** infra-red has quit IRC | 14:59 | |
| *** dizquierdo has quit IRC | 15:00 | |
| *** sputnik13 has joined #openstack-infra | 15:00 | |
| bunting | Hi, what could cause permissions of a folder to be diffrent between two diffrent functional tests? Since I am running into "Permission denied: '/var/cache/glance'" | 15:00 |
| fungi | mordred: apuimedo: if memory serves, it's that they fight over which one controls the __init__.py for the namespace | 15:01 |
| clarkb | Srinu: use the same server for both so should be the same ip | 15:01 |
| *** links has quit IRC | 15:02 | |
| fungi | apuimedo: https://specs.openstack.org/openstack/oslo-specs/specs/kilo/drop-namespace-packages.html has some details, it seems | 15:02 |
| *** makowals has quit IRC | 15:02 | |
| clarkb | bunting: possibly different writers setting different permissions | 15:03 |
| *** aysyd has quit IRC | 15:03 | |
| clarkb | bunting: I would start with figuring out what writrs the files and work from there | 15:03 |
| *** piet has joined #openstack-infra | 15:04 | |
| *** aysyd has joined #openstack-infra | 15:06 | |
| bunting | clarkb: This is the glance-api trying to create the folder. I'm looking at a glance_store functional test (which runs into the problem) vs a nova functioal test (which works). | 15:06 |
| AJaeger | mordred: thanks for review | 15:06 |
| clarkb | bunting: well nova functional tests may not write to /var/cache at all | 15:07 |
| *** jistr is now known as jistr|mtg | 15:07 | |
| clarkb | bunting: you will need root perms tto create that dir | 15:07 |
| *** pcaruana has quit IRC | 15:08 | |
| AJaeger | frickler, clarkb :I'm rebasing 336272 - the py35 patch - now | 15:08 |
| Srinu | Clarkb: i installed one jenkins with gearman enable with one slave and another jenkins master with different slave but i am facing this issue when on jenkins master report sucess another jenkins master node is going offline | 15:10 |
| *** _nadya_ has joined #openstack-infra | 15:10 | |
| *** salv-orlando has joined #openstack-infra | 15:10 | |
| *** xarses has joined #openstack-infra | 15:13 | |
| *** rcernin has quit IRC | 15:14 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add python35 jobs https://review.openstack.org/336272 | 15:14 |
| *** dimtruck is now known as zz_dimtruck | 15:15 | |
| clarkb | Srinu: maybe they both ran jobs? Ehat you describe isnt necessarily an issue if both masters are activelyy running jobs | 15:16 |
| *** ajmiller_ has joined #openstack-infra | 15:16 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI https://review.openstack.org/336312 | 15:17 |
| *** salv-orlando has quit IRC | 15:17 | |
| fungi | jeblair: i guess you can un-wip (and approve?) 330698 after the zuul.o.o distro upgrade today | 15:18 |
| *** hongbin_ has joined #openstack-infra | 15:18 | |
| Srinu | Clarkb: they are running jobs but when one reaches success another jenkins master slave node went offline | 15:19 |
| *** ajmiller has quit IRC | 15:19 | |
| *** locust has joined #openstack-infra | 15:20 | |
| *** xarses has quit IRC | 15:20 | |
| *** xarses has joined #openstack-infra | 15:20 | |
| *** pradk_ has joined #openstack-infra | 15:20 | |
| AJaeger | clarkb, what about splitting the layout.yaml change up - only add the new jobs to infra repos. That way we can guinia pig a bit... | 15:21 |
| *** amitgandhinz has quit IRC | 15:21 | |
| *** hongbin_ has quit IRC | 15:21 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update ScriptTrigger plugin https://review.openstack.org/331866 | 15:21 |
| *** jamesmcarthur has joined #openstack-infra | 15:22 | |
| AJaeger | but probably not worth it since it's non-voting and in addition - not replacing | 15:22 |
| *** amitgandhinz has joined #openstack-infra | 15:22 | |
| *** hongbin_ has joined #openstack-infra | 15:22 | |
| clarkb | AJaeger: we could do that, we can also split by db vs not db additions | 15:22 |
| fungi | Srinu: is there any chance you have your two jenkins masters sharing a slave node or something? | 15:22 |
| clarkb | but ya concern is less about it breaking and more helping people understand and fix any issues | 15:22 |
| fungi | jeblair: is it safe to go ahead and launch the new zuul.o.o instance now so that we can prep the firewall changes, or is there a risk that it will connect to gerrit and try to do things? granted its gearman will have no job registrations/workers/mergers available so it won't be able to do much more than make annoying comments if anything | 15:23 |
| *** kushal has quit IRC | 15:23 | |
| *** amotoki has quit IRC | 15:23 | |
| fungi | oh, and leave bad votes of course | 15:24 |
| fungi | ahh, but our puppet-zuul module doesn't auto-start the service! | 15:24 |
| *** sputnik13 has quit IRC | 15:24 | |
| fungi | so should be 100% safe right? | 15:24 |
| jeblair | fungi: i think there is a good chance of that, but yes, hopefully it will just wait for mergers to show up before actually saying anything. so i think it should be okay if you keep a close eye on it | 15:24 |
| *** Jeffrey4l has joined #openstack-infra | 15:24 | |
| jeblair | fungi: (hopefully it configures it to start on boot though, and launch-node tells it to reboot) | 15:24 |
| *** Jeffrey4l has quit IRC | 15:24 | |
| fungi | d'oh! right | 15:25 |
| *** Srinu has quit IRC | 15:25 | |
| clarkb | you could disable puppet on old host, update key info to be temporarily invalid then make new host | 15:25 |
| AJaeger | clarkb: let's leave as is and I'm fine with merging completely - splitting it up is a boring split ;) | 15:25 |
| fungi | i guess i can launch it, then ssh in immediately after the reboot and stop the zuul service | 15:25 |
| clarkb | AJaeger: ok | 15:26 |
| *** Kiall has joined #openstack-infra | 15:26 | |
| *** esikachev has quit IRC | 15:26 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add plugin="junit" attribute for JUnit plugin https://review.openstack.org/332238 | 15:26 |
| fungi | collateral damage should be minimal as long as i stop it right away | 15:26 |
| jeblair | clarkb: yeah, but i think it's not going to actually do anything as fungi says, so i think the risk is low enough to just manually stop it asap | 15:26 |
| *** vhosakot has joined #openstack-infra | 15:27 | |
| *** esikachev has joined #openstack-infra | 15:27 | |
| fungi | jeblair: oh, weird... we have /opt double-mounted on the current zuul.o.o | 15:28 |
| *** daemontool has quit IRC | 15:28 | |
| fungi | df lists it twice (with the same blockdev) | 15:28 |
| fungi | that's... odd | 15:28 |
| *** nwkarsten has joined #openstack-infra | 15:28 | |
| *** tesseract- has quit IRC | 15:28 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update OpenShift Pipeline plugin https://review.openstack.org/334632 | 15:32 |
| *** PollyZ has joined #openstack-infra | 15:32 | |
| *** zhurong has quit IRC | 15:32 | |
| *** amitgandhinz has quit IRC | 15:32 | |
| *** esikachev has quit IRC | 15:33 | |
| *** nwkarsten has quit IRC | 15:33 | |
| *** Kiall has quit IRC | 15:33 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update GitHub pull request builder plugin https://review.openstack.org/335632 | 15:33 |
| fungi | jeblair: another related question... it looks like we have a 200gb cinder volume (10447e03-2bf6-4ea1-b7c0-d16ee90a673b) at xvdb but instead we're using the ephemeral disk at xvde for /opt | 15:34 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update bazaar plugin https://review.openstack.org/335646 | 15:34 |
| fungi | i guess that's less of a question and more of a statement, to be followed by the question "do we need it?" | 15:34 |
| fungi | also i'm pretty sure we no longer need the tmpfs mounted at /var/lib/zuul/git since that predates the separate mergers | 15:36 |
| fungi | and /opt seems to be mostly filled with cruft | 15:37 |
| locust | hello, is there existing config for nodepool | 15:37 |
| clarkb | I think zuul foesnt need that now that mergers are separate | 15:37 |
| locust | I know there exists project-config-example | 15:38 |
| openstackgerrit | Merged openstack-infra/system-config: Rename the eavesdrop site to be clearer https://review.openstack.org/331503 | 15:38 |
| clarkb | locust: yes though practice says they arent very reusable because clouds differ as do image needs and so on | 15:38 |
| fungi | locust: your question puzzles me. we run nodepool, so we have _a_ configuration (it's in openstacl-infra/project-config) but it's probably not usable as _your_ configuration | 15:38 |
| locust | fungi: hm | 15:39 |
| jeblair | clarkb, fungi: i think the cinder volume was used for the ginormous debug logs we needed a while back? so i think the answer is no we don't need it anymore | 15:39 |
| locust | I am deploing third party CI, so thinking what to start with? | 15:40 |
| fungi | locust: does the puppet-openstackci module provide a templated one? | 15:40 |
| fungi | no, i guess it just expects you to provide your own project-config | 15:41 |
| locust | as I see --- not | 15:41 |
| *** ifarkas has quit IRC | 15:41 | |
| fungi | oh! it does | 15:42 |
| fungi | there's the openstack-infra/project-config-example repo which has a simple one in it | 15:42 |
| fungi | http://git.openstack.org/cgit/openstack-infra/project-config-example/tree/nodepool/nodepool.yaml | 15:42 |
| *** thiagop is now known as thiagop-afk | 15:42 | |
| locust | fungi: | 15:43 |
| *** SumitNaiksatam has quit IRC | 15:43 | |
| *** yamamoto has joined #openstack-infra | 15:43 | |
| fungi | jeblair: also now that the memory leak is under control, it looks like we could easily get by with a 15gb flavor instead of 30gb? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 15:43 |
| *** david-lyle has joined #openstack-infra | 15:44 | |
| locust | fungi: yes, but it is not working from the box | 15:44 |
| fungi | locust: nothing is going to work "out of the box" | 15:44 |
| jeblair | fungi: wow, i'm surprised that's so high | 15:44 |
| *** salv-orlando has joined #openstack-infra | 15:45 | |
| jeblair | fungi: we *should* need half that. but i agree we appear to need more. | 15:45 |
| jeblair | (i think something is still wonky) | 15:45 |
| fungi | locust: it's a daemon for managing virtual machine resources, so without knowing your cloud provider, your image needs, et cetera there's no way to make it work "out of the box" | 15:45 |
| *** markusry has quit IRC | 15:47 | |
| locust | fungi: I thought this is just for creating images | 15:47 |
| locust | like build some thign like bootable iso | 15:47 |
| *** amotoki has joined #openstack-infra | 15:47 | |
| fungi | locust: probably the best place to start is asking, what do you want it to do? (make images, but with what in them? boot virtual machines, but where and how many?) | 15:47 |
| AJaeger | are we still having a downtime at 20:00 UTC? Should we send a reminder? | 15:47 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Update OSA job skip for all jobs except docs/releasenotes https://review.openstack.org/331574 | 15:48 |
| fungi | locust: perhaps it's best to read http://docs.openstack.org/infra/nodepool/ and then come back with clearer questions | 15:48 |
| *** roxanagh_ has joined #openstack-infra | 15:48 | |
| locust | I want image of "All in one" deployment of openstack to test cinder driver | 15:48 |
| *** karthik__ has joined #openstack-infra | 15:48 | |
| locust | I do not expect to have lots of VM's may be one | 15:49 |
| locust | is it enoungh to support driver testing? | 15:49 |
| fungi | locust: you want an image of a devstack deployment to use for local testing of something? | 15:49 |
| *** PollyZ has quit IRC | 15:49 | |
| fungi | i don't know what you mean by "driver testing" | 15:49 |
| *** roxanagh_ has quit IRC | 15:49 | |
| *** amotoki has quit IRC | 15:49 | |
| *** amotoki has joined #openstack-infra | 15:50 | |
| *** karthik__ has quit IRC | 15:50 | |
| *** jistr|mtg is now known as jistr | 15:50 | |
| *** andreas_s has quit IRC | 15:50 | |
| locust | fungi: I am making Cinder to be "Friendly" to some new Device | 15:50 |
| fungi | AJaeger: yeah, i was going to wait until closer, like maybe 18:00 utc for a 2-hour warning | 15:50 |
| fungi | locust: yeah, i know what a driver is, but please elaborate on _what_ about it you expect to be testing? are you just running some local tests you've devised to help you write that driver? | 15:51 |
| locust | fungi: and to provide commit of this updates I need to setup CI on my side, to test my code | 15:51 |
| *** mandre has quit IRC | 15:52 | |
| *** salv-orlando has quit IRC | 15:52 | |
| locust | fungi: no, I am using tempest to proof that my code is valid and to commit it to Cinder repo | 15:52 |
| *** hashar has quit IRC | 15:53 | |
| AJaeger | fungi, ok. Was expecting an email - but that can go out around 18:00 as well... | 15:53 |
| fungi | locust: are you saying you expect to test changes other people propose to cinder to make sure they don't break your driver? if so, you're not going to get by with just one vm, i don't think. you'll be booting a lot of them and not reusing them because you're giving someone else (by way of testing their proposed patches) root access on those virtual machines | 15:53 |
| *** Nikolay_St has quit IRC | 15:53 | |
| *** pilgrimstack has quit IRC | 15:53 | |
| *** PollyZ has joined #openstack-infra | 15:54 | |
| * AJaeger needs a break now - if I don't come back online today, I wish everybody here based in the US a great 4th of July weekend! | 15:55 | |
| locust | fungi: m... my first prioryty target is to test MY code | 15:55 |
| locust | fungi: in the whay it is acceptable by Cinder comunyty | 15:56 |
| fungi | locust: okay, in that case you're probably better off starting from installing devstack in a virtual machine. it's designed to be very easy to set up | 15:56 |
| *** sree has joined #openstack-infra | 15:56 | |
| locust | fungi: yes, I've already done this | 15:56 |
| locust | and my code is passing tests | 15:57 |
| *** david-lyle_ has joined #openstack-infra | 15:57 | |
| fungi | locust: then it sounds like you've met your stated goal? | 15:57 |
| *** david-lyle_ has quit IRC | 15:57 | |
| *** eezhova has quit IRC | 15:57 | |
| *** mandre has joined #openstack-infra | 15:58 | |
| locust | so now I am working on contributing to Cinder, and folks at Cinder will not accept my code just because it is avesome, and I have done local tests | 15:58 |
| *** yamamoto has quit IRC | 15:58 | |
| locust | fungi: they want to see logs, and see interractive code testing | 15:58 |
| *** karthik__ has joined #openstack-infra | 15:58 | |
| fungi | locust: right, as far as i know the cinder team wants _you_ to test _other_people's_ proposed patches to cinder to make sure they don't break your driver | 15:59 |
| *** dtantsur is now known as dtantsur|afk | 15:59 | |
| *** Swami has joined #openstack-infra | 15:59 | |
| locust | and this too | 15:59 |
| *** jlanoux has quit IRC | 15:59 | |
| locust | fungi: ok, yes | 16:00 |
| *** sree has quit IRC | 16:00 | |
| *** david-lyle_ has joined #openstack-infra | 16:00 | |
| fungi | locust: i think there must be some critical context missing either on my end or your end of this discussion. you say you _just_ want something to make a virtual machine image for you, but that's not the extent of what the cinder team is going to require you to have in place as far as i know | 16:01 |
| fungi | locust: the best place to start is probably talking to other people who are doing similar things with third-party ci for cinder drivers. here is a good place to discuss the tools we've written which you can assemble into a ci system, but how to assemble those components to best test changes to cinder against your driver/hardware isn't really something we can help with | 16:01 |
| fungi | that's more of a policy question | 16:02 |
| *** tqtran has joined #openstack-infra | 16:03 | |
| locust | fungi: hm, you are write, bu not all Cinder developers are familiare with Puppet, Nodepool | 16:03 |
| fungi | locust: there's a bunch of third-party ci operators who meet weekly on mondays (i don't know if they're planning to meet this coming monday because it's a holiday for a lot of the people who normally participate) http://eavesdrop.openstack.org/#Third_Party_Meeting | 16:03 |
| fungi | that's a good opportunity to get to know other people who are doing the things you're talking about needing to do | 16:03 |
| pleia2 | good morning | 16:04 |
| *** david-lyle has quit IRC | 16:04 | |
| *** david-lyle_ is now known as david-lyle | 16:04 | |
| fungi | locust: the cinder devs can tell you what they expect from you, and the other third-party ci operators testing cinder drivers can tell you what specific software and configuration they've put in place to meet those expectations (much of which might be software we've written) | 16:05 |
| *** baoli_ has quit IRC | 16:05 | |
| *** Apoorva has joined #openstack-infra | 16:06 | |
| locust | fungi: yes, but I thought that here I can find good clues of how to start with One simple VM running ubuntu with devstack | 16:06 |
| *** Apoorva has quit IRC | 16:06 | |
| *** _nadya_ has quit IRC | 16:06 | |
| locust | fungi: not solve all my problems but find a ground to start with | 16:07 |
| fungi | locust: you've already started with one simple vm running ubuntu with devstack. what we have is software to automate doing that on a mass scale | 16:07 |
| *** PollyZ has quit IRC | 16:07 | |
| *** tqtran has quit IRC | 16:07 | |
| fungi | locust: and that automation at scale expects you've met a variety of other prerequisites such as having a tenant/project in an openstack cloud available with some quota set | 16:08 |
| fungi | the sorts of configuration you'll need to plug into nodepool (we can't know what that will be, which is why there is no nodepool config that will work "out of the box") | 16:09 |
| *** _nadya_ has joined #openstack-infra | 16:09 | |
| *** tphummel has joined #openstack-infra | 16:10 | |
| locust | fungi: so for instance if I look at nodepool.yaml at the fild of elements of diskimages: evry folder in elements must be writen by my hands? | 16:11 |
| locust | fugi: like write my own automation of image building? | 16:11 |
| *** mriedem has quit IRC | 16:11 | |
| fungi | locust: to summarize, nodepool's designed to use diskimage-builder to create images, upload them to a cloud, boot a defined number of single-use virtual machines from those images and register them to a job worker (such as a jenkins), and to track demand expressed by zuul (a job scheduler) to determine how many instances to boot, and to delete those instances once their jobs complete | 16:12 |
| *** mriedem has joined #openstack-infra | 16:12 | |
| *** mriedem has quit IRC | 16:12 | |
| jeblair | clarkb: python error in 336311, otherwise solution lgtm | 16:13 |
| *** edmondsw has quit IRC | 16:13 | |
| clarkb | jeblair: woo, can I blame juggling babies when writing that change? | 16:14 |
| clarkb | we are in the "we can almost walk now let me show you" stage | 16:14 |
| locust | fungi: and I need to write my own image killing and booting according to zuul output? | 16:15 |
| fungi | locust: the example config is reusable. it looks like it's designed to expect a local jenkins master running on the same machine as nodepool, and a local zuul server also running on that machine, and will communicate with an openstack cloud keystone endpoint specified in your clouds.yaml as "local_01" to build and upload a predefined ubuntu image and boot up to 2 virtual machines at a time from that | 16:15 |
| fungi | image | 16:15 |
| jeblair | clarkb: you should just blame the babies! i bet they get boolean coercions wrong all the time! :) | 16:15 |
| *** ihrachys has quit IRC | 16:15 | |
| mordred | jeblair: I find it hard to think that clarkb's babies would ever make a boolean mistake | 16:15 |
| fungi | locust: it looks like there is a readme in the project-config-example repo that expresses some of this http://git.openstack.org/cgit/openstack-infra/project-config-example/tree/README.md | 16:16 |
| locust | fungi: thank you | 16:17 |
| pleia2 | I'm going to reply to my initial email about the Gerrit downtime today as a reminder, have there been any changes to that? (2000 UTC, 2 hour window) | 16:17 |
| locust | fungi: gona read this one | 16:17 |
| fungi | pleia2: we're still on schedule afaik | 16:17 |
| pleia2 | fungi: thanks, sending reminder then :) | 16:18 |
| fungi | i was just trying to finish establishing parameters for the replacement zuul so i can know the new ip addresses and start writing up the necessary config changes | 16:18 |
| *** john-davidge has quit IRC | 16:18 | |
| *** coolsvap has quit IRC | 16:18 | |
| jeblair | oh good, i was just about to go try to find out what i had committed to :) | 16:18 |
| pleia2 | hehe | 16:18 |
| fungi | jeblair: you committed to "being around" as far as i'm concerned, so however much or little you want to do is appreciated | 16:19 |
| jeblair | fungi: and now i know when to be around! :) | 16:19 |
| *** jpich has quit IRC | 16:19 | |
| jeblair | clarkb: i dropped off the planet right after we wrote the zmq fixes -- did they get merged, and appropriate things restarted? | 16:20 |
| clarkb | jeblair: they did though jhesketh had a much more extensive change you should consider | 16:21 |
| *** Apoorva has joined #openstack-infra | 16:22 | |
| clarkb | I need to check if all my restarts completed last night | 16:22 |
| jeblair | clarkb: any hints as to which change that might be? | 16:23 |
| clarkb | it was linked to in comments of your change | 16:23 |
| fungi | jeblair: so you're cool with using a 15gb flavor this time around, or should i stick to 30gb? | 16:24 |
| jeblair | oh, it's on the "zuulv25-tests" branch | 16:24 |
| *** PollyZ has joined #openstack-infra | 16:25 | |
| *** amotoki has quit IRC | 16:27 | |
| jeblair | fungi: 15g sgtm | 16:28 |
| *** shashank_hegde has joined #openstack-infra | 16:28 | |
| jeblair | clarkb: 317481; reviewed | 16:28 |
| *** pfallenop has quit IRC | 16:29 | |
| *** watersoul has quit IRC | 16:29 | |
| fungi | jeblair: as far as i can tell, the existing zuul is only referenced anywhere by hostname, so no explicit ip addresses to patch in any of our repos | 16:30 |
| fungi | we'll just need to reapply iptables rules in a few places after we change dns | 16:31 |
| jeblair | btw, i consider zuul v2.5 finished and have switched back to working on v3. | 16:31 |
| jeblair | fungi: agreed | 16:31 |
| fungi | jeblair: awesome | 16:31 |
| *** daemontool has joined #openstack-infra | 16:31 | |
| fungi | and also restart everything that talks to gearman (nodepool, zuul mergers, zuul launchers) | 16:32 |
| *** roxanagh_ has joined #openstack-infra | 16:32 | |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: Unlink monasca gate jobs from neutron testes https://review.openstack.org/336649 | 16:33 |
| fungi | i'm making sure the a/aaaa records for zuul and static are 5 minutes | 16:33 |
| *** gyee has joined #openstack-infra | 16:33 | |
| fungi | pretty sure i previously made certain all the virtual hosts served from static.o.o are just cnames | 16:33 |
| fungi | but i'll double-check that shortly | 16:34 |
| fungi | okay, i managed to jump on the replacement zuul as soon as ssh would let me in after the reboot and stopped the daemon | 16:35 |
| fungi | so less than a minute of it running | 16:35 |
| pleia2 | I can check on cnames real quick if you'd like | 16:36 |
| fungi | pleia2: that would be a huge help--thanks! | 16:36 |
| pleia2 | on it | 16:36 |
| *** ociuhandu has quit IRC | 16:37 | |
| *** hongbin_ has quit IRC | 16:38 | |
| *** asettle has quit IRC | 16:38 | |
| *** lucasagomes is now known as lucas-afk | 16:38 | |
| pleia2 | tarballs.o.o and devstack.org are both A records | 16:38 |
| fungi | zuul records were already 5 minutes, static records were 60 but i've lowered them to 5. the new ttl should be propagated everywhere with plenty of time to spare | 16:38 |
| fungi | pleia2: thanks, i'll switch those to cnames now | 16:39 |
| *** piet has quit IRC | 16:39 | |
| pleia2 | ci, docs-draft, security, governance, specs, summit, trystack and releases are all cnams | 16:39 |
| pleia2 | cnames | 16:39 |
| *** shashank_hegde has quit IRC | 16:39 | |
| *** derekh has quit IRC | 16:39 | |
| pleia2 | do we have anything outside of static.pp? | 16:39 |
| *** nwkarsten has joined #openstack-infra | 16:40 | |
| fungi | not to my knowledge, no | 16:40 |
| *** hongbin_ has joined #openstack-infra | 16:40 | |
| pleia2 | alright, checked everything there | 16:40 |
| openstackgerrit | Kaiyan Sheng proposed openstack-infra/project-config: Unlink monasca gate jobs from neutron tests https://review.openstack.org/336649 | 16:40 |
| fungi | hrm, tarballs will be slightly racy... i need to delete the a/aaaa records since cnames can't coexist with them | 16:40 |
| patrickeast | hey, anyone have any tips of troubleshooting stack.sh never finishing on subnodes with devstack-gate? it looks like https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L632-L634 is running since that txt file exists, but its empty | 16:41 |
| *** jlanoux has joined #openstack-infra | 16:41 | |
| *** Kiall has joined #openstack-infra | 16:41 | |
| *** amoralej is now known as amoralej|off | 16:41 | |
| patrickeast | i can see stack.sh running if i look at ps aux | 16:41 |
| patrickeast | but dunno how to get its output to see whats going on | 16:41 |
| *** pfallenop has joined #openstack-infra | 16:43 | |
| fungi | pleia2: devstack.org will need to be changed at the same time we adjust dns for static.o.o since (for similar reasons) you can't cname the soa of a zone | 16:44 |
| *** Apoorva has quit IRC | 16:44 | |
| pleia2 | fungi: ok, so just dropping the ttl for now? | 16:44 |
| *** geekinutah is now known as nerdonmountain | 16:44 | |
| fungi | pleia2: yep. also trystack.org is in that same boat | 16:44 |
| *** Apoorva has joined #openstack-infra | 16:44 | |
| pleia2 | ah yes, I missed the serveraliases => | 16:45 |
| pleia2 | we also have serveraliases => ['*.devstack.org'], | 16:45 |
| pleia2 | but I suspect that's just www. | 16:45 |
| *** sputnik13 has joined #openstack-infra | 16:45 | |
| fungi | pleia2: yep, and www.devstack.org is already a cname to static.openstack.org | 16:46 |
| pleia2 | great | 16:46 |
| *** javeriak has joined #openstack-infra | 16:46 | |
| *** ccamacho|lunch has quit IRC | 16:46 | |
| fungi | okay ttls for trystack.org and devstack.org records are 5 minutes now as well | 16:46 |
| *** jamesmcarthur has quit IRC | 16:46 | |
| *** pfallenop has quit IRC | 16:48 | |
| *** jsavak has quit IRC | 16:49 | |
| *** hongbin_ has quit IRC | 16:49 | |
| *** salv-orlando has joined #openstack-infra | 16:49 | |
| *** jsavak has joined #openstack-infra | 16:49 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI https://review.openstack.org/336312 | 16:50 |
| *** hongbin_ has joined #openstack-infra | 16:50 | |
| fungi | static.openstack.org is an 8gb flavor right now. looks like it could get by reasonably on a lot less but its bandwidth utilization spikes up from time to time so i'd worry about possibly hitting a lower bw cap if we did | 16:50 |
| *** kzaitsev_mb has quit IRC | 16:50 | |
| fungi | looks like we don't need to size _up_ though so i say we stick with 8gb | 16:51 |
| clarkb | I have confirmed that all the restarts completed using https://review.openstack.org/#/c/336317/ | 16:51 |
| clarkb | fungi: ++ especially for the bw | 16:51 |
| clarkb | bw gets halved as you go to each smaller flavor iir | 16:51 |
| *** acoles is now known as acoles_ | 16:52 | |
| *** ddieterly is now known as ddieterly[away] | 16:52 | |
| fungi | the cpu/io/buffers are high from time to time, but that's because we've still not successfully disabled the mlocate cronjob | 16:53 |
| *** bryan_att has quit IRC | 16:53 | |
| *** _nadya_ has quit IRC | 16:53 | |
| *** bin_ has quit IRC | 16:54 | |
| * fungi shakes fist at mlocate wanting to reindex terabytes of log data | 16:54 | |
| *** asettle has joined #openstack-infra | 16:54 | |
| *** pfallenop has joined #openstack-infra | 16:55 | |
| *** pfallenop has joined #openstack-infra | 16:55 | |
| *** salv-orlando has quit IRC | 16:56 | |
| jeblair | clarkb: i wonder if mordred's async change to ansible would make that work | 16:56 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Spec for adding diskimage_builder module in-chroot https://review.openstack.org/336664 | 16:56 |
| fungi | do we want to use https://etherpad.openstack.org/p/upgrade-zuul-trusty or https://etherpad.openstack.org/p/newton-infra-distro-upgrade-plans or https://etherpad.openstack.org/p/infra-operating-system-upgrades or a new etherpad for this window? | 16:56 |
| jeblair | mordred: btw, what's the status of that? did it get made into a pr? did it merge? will it be in next release? | 16:56 |
| *** tonytan4ever has quit IRC | 16:57 | |
| *** tosky has quit IRC | 16:57 | |
| *** ddieterly[away] is now known as ddieterly | 16:58 | |
| *** _nadya_ has joined #openstack-infra | 16:58 | |
| *** sambetts is now known as sambetts|afk | 17:00 | |
| clarkb | jeblair: it might, but its also unnecessary here | 17:00 |
| clarkb | since we do one host at a time, keeping an ssh connection open is cheap | 17:01 |
| jeblair | clarkb: i thought we liked async for these things because we ran into ssh timeouts, or increased the risk of the connection dropping for some reason? | 17:02 |
| *** PollyZ has quit IRC | 17:02 | |
| *** piet has joined #openstack-infra | 17:02 | |
| *** hongbin_ has quit IRC | 17:02 | |
| mordred | jeblair: https://github.com/ansible/ansible/pull/16458 | 17:02 |
| mordred | jeblair: I do not think my patch would help with this - but we could potentially also do a patch to retry on spurious connection failures | 17:03 |
| clarkb | jeblair: my experience is the opposite, I wasn't really around when you all decided to switch to async wholesale | 17:03 |
| clarkb | jeblair: asnyc leads to consistent and frequent ssh errors between puppetmaster and the zuul launchers | 17:04 |
| clarkb | no asnyc worked 100% of the time in my small sample | 17:04 |
| mordred | I believe that async _should_ work between puppetmaster and the zuul launchers, and we should fix it if it doesn't. also, wait_for, since it has a timeout, should not need async. | 17:05 |
| mordred | it was my understanding that our use of async in z-l was to make timeouts work | 17:05 |
| *** jsavak has quit IRC | 17:05 | |
| clarkb | ah ok that makes sense | 17:05 |
| jeblair | i think we are talking about 5 different things here | 17:05 |
| mordred | yup | 17:05 |
| *** hongbin_ has joined #openstack-infra | 17:06 | |
| *** jsavak has joined #openstack-infra | 17:06 | |
| jeblair | zuul-launcher's use of async was my choice, and yes, it is the only ansible-native way of doing timeouts that i know of and that's why i chose it | 17:06 |
| *** javeriak has quit IRC | 17:06 | |
| jeblair | i did not choose any other use of async and can't speak as to why, but i head heard other people talk about ssh timeouts in the context of puppetmaster run-all | 17:07 |
| clarkb | jeblair: I think other people have begun to cargo cult your use of async even when it is not necessary | 17:07 |
| clarkb | unforutnately, this has led to reliability issues, possibly related to the same ones that zuul ran into and is trying to address with patches to ansible | 17:07 |
| mordred | right. it's actually a different but potentially related reliability issue | 17:08 |
| mordred | I'm looking right now to see if we can make a follow up that would handle connection issues | 17:08 |
| mordred | so that if we do need/want to use async for various things, we do not avoid it because of bugs | 17:08 |
| mordred | clarkb: any chance you have any captured errors/tracebacks from when it was giving you ssh problems? | 17:09 |
| jeblair | clarkb, mordred: i also used async in playbooks/restart_jjb for the same reason -- to get a timeout | 17:09 |
| Shrews | mordred: i get the feeling we're going to end up rewriting most of the async code at some point | 17:09 |
| mordred | Shrews: yah | 17:09 |
| jeblair | clarkb, mordred: https://review.openstack.org/277539 and http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-02-08.log.html#t2016-02-08T19:01:16 | 17:09 |
| clarkb | mordred: I have already closed that screen session but I pasted the relevant bit of the error in here, let me find it in scrollback | 17:10 |
| *** jsavak has quit IRC | 17:10 | |
| *** hongbin_ has quit IRC | 17:10 | |
| clarkb | mordred: fatal: [zl03.openstack.org]: FAILED! => {"failed": true, "msg": "Failed to connect to the host via ssh."} | 17:10 |
| mordred | Shrews: we may also want to chat with ansible core about playbook timeouts in general | 17:11 |
| jeblair | clarkb: have you spoken with pabelanger about this? | 17:11 |
| Shrews | mordred: agreed. fest is a good time for that | 17:11 |
| mordred | jeblair: restart_jjb was for re-running jjb right? | 17:11 |
| mordred | Shrews: agreed | 17:11 |
| *** hongbin_ has joined #openstack-infra | 17:12 | |
| clarkb | jeblair: yes he indicated I should run the playbook from my laptop, instead I disabled async and it just worked | 17:12 |
| *** electrofelix has quit IRC | 17:12 | |
| jeblair | mordred: typo: playbooks/run_jjb and yes. | 17:12 |
| *** hashar has joined #openstack-infra | 17:12 | |
| clarkb | since wait_for is used and has built in timeouts the async is not necessary | 17:12 |
| clarkb | and will either succeed to ssh once then wait or fail and die eraly instead of failing an hour into an async poll | 17:13 |
| jeblair | clarkb: makes sense. just thought it would be useful to find out what prompted him to use async. | 17:13 |
| *** hashar is now known as hasharAway | 17:13 | |
| jeblair | clarkb: but today is canada day. so maybe we can follow up later. | 17:13 |
| *** mhickey has quit IRC | 17:13 | |
| mordred | jeblair: isn't every day canada day? | 17:14 |
| Shrews | o/` Oh, Canada o/` | 17:14 |
| *** adrian_otto has joined #openstack-infra | 17:14 | |
| jeblair | clarkb, mordred, Shrews: if we don't like async for zuul-launcher, we could have z-l pgkill the group (that is our fallback anyway -- i just wanted to try to do the right ansible-ish thing) | 17:14 |
| mordred | jeblair: I think async for zuul-launcher is correct usage | 17:15 |
| clarkb | I think it makes sense for zuul launcher | 17:15 |
| *** tqtran has joined #openstack-infra | 17:15 | |
| clarkb | particularly since one of jenkins big scale issues was maintaining all of those ssh connections at once | 17:15 |
| mordred | yah | 17:15 |
| Shrews | async just needs some polish, but is the right way | 17:15 |
| jeblair | (but also, i worry about timeouts on 'idle' ssh connections) | 17:15 |
| mordred | for the restart playbook, wrapping the wait_for timeout in a second timeout just seems like overkill | 17:16 |
| *** markusry has joined #openstack-infra | 17:16 | |
| jeblair | (that's handwavey -- it's just something to potentially understand) | 17:16 |
| mordred | jeblair: totally... although I believe if wait_for can't deal with 'idle' connections that would also be a bug we should fix for sure :) | 17:16 |
| jeblair | yeah, i think clarkb's thing is good, assuming no ssh issues. but maybe those were imagined or otherwise don't exist. | 17:16 |
| openstackgerrit | Diana Whitten proposed openstack-infra/vinz: Vinz, the web app https://review.openstack.org/336143 | 17:16 |
| mordred | woah. vinz changes | 17:17 |
| Shrews | mordred: it would be kinda cool if timeouts were supported at the 'block' level | 17:17 |
| mordred | Shrews: yup | 17:17 |
| Shrews | block-rescue-always-timeout | 17:17 |
| jeblair | Shrews: ++ | 17:17 |
| AJaeger | team, kite and python-kiteclient is dead - but there's a second +2A missing on https://review.openstack.org/#/c/312175/ to fully retire python-kiteclient. Is that something an infra-root should do - or should we ping the ex-kite folks again? | 17:18 |
| openstackgerrit | Diana Whitten proposed openstack-infra/vinz: Vinz, the web app https://review.openstack.org/336143 | 17:19 |
| *** ddieterly is now known as ddieterly[away] | 17:19 | |
| pabelanger | ohai! I opted for using async on zuul-launcher graceful restarts because I didn't think a single SSH connection idling for 3h10mins would be reliable. But agree the problem clarkb had was something I didn't initially think about | 17:19 |
| pabelanger | it would be great if async could simply ignore SSH failures and keep try connections until the timeout value was reached | 17:20 |
| pabelanger | rather then bailing on SSH connection failures | 17:20 |
| clarkb | pabelanger: or some total number of consecutive fails | 17:20 |
| pabelanger | clarkb: ya, that would be great too | 17:22 |
| openstackgerrit | Merged openstack-infra/project-config: Update eavesdrop site name https://review.openstack.org/336552 | 17:23 |
| *** hongbin_ has quit IRC | 17:23 | |
| jeblair | clarkb, pabelanger: cool -- well, we have data for one side of the question. i guess we'll see how reliable the really long ssh connection is now. :) | 17:23 |
| jeblair | pabelanger: and shouldn't you be canadaing? | 17:24 |
| *** javeriak has joined #openstack-infra | 17:24 | |
| pabelanger | Ya, I'm happy to try it without async | 17:24 |
| pabelanger | see what results we get | 17:24 |
| *** salv-orl_ has joined #openstack-infra | 17:25 | |
| pabelanger | jeblair: I have been! kids are down for naps and noticed IRC blinking at mt | 17:25 |
| pabelanger | me* | 17:25 |
| *** hongbin_ has joined #openstack-infra | 17:25 | |
| pabelanger | was about to drop off again to fetch a beer | 17:26 |
| pabelanger | good luck on the upgrades today | 17:26 |
| jeblair | pabelanger: thanks, and good luck on the beer! | 17:26 |
| *** rkukura has quit IRC | 17:26 | |
| *** karthik__ has quit IRC | 17:26 | |
| clarkb | pabelanger: does canada have a beer called "Canada"? | 17:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/vinz: Update to use pbr and other infra standards https://review.openstack.org/336681 | 17:27 |
| clarkb | I recommend drinking beers named after your country on country day | 17:27 |
| *** salv-orl_ has quit IRC | 17:27 | |
| mordred | nibalizer, hurgleburgler: ^^ I was going to leave a -1 comment on some of the setuptools stuff in the previous patch, but then decided that would be rude and would be easier for me to just make a patch | 17:27 |
| *** akshai has quit IRC | 17:28 | |
| mordred | clarkb: eww. does that I mean I need to drink America on Monday? | 17:28 |
| mordred | that can't possibly have good results | 17:28 |
| clarkb | mordred: I intend on drinking at least one can | 17:28 |
| clarkb | for the lols | 17:28 |
| clarkb | mordred: also you are in texas which means you can only get America and Shiner neither of which I would drink in large quantities | 17:29 |
| cinerama | lone star is the national beer of texas | 17:29 |
| cinerama | (it says so on the tin) | 17:29 |
| clarkb | clearly I am biased but still not anything I would drink in large quantities :) | 17:29 |
| *** akshai has joined #openstack-infra | 17:29 | |
| clarkb | in other news Gerrit GCing has picked up quite a bit recently, we should consider a restart | 17:30 |
| clarkb | maybe when zuul is off? | 17:30 |
| mordred | clarkb: agree re: restart | 17:30 |
| clarkb | mordred: if you can find it Fremont Summer Ale out of seattle is great drink in quantity beer | 17:30 |
| *** sdake has joined #openstack-infra | 17:30 | |
| mordred | clarkb: however, I disagree re: shiner | 17:30 |
| *** shashank_hegde has joined #openstack-infra | 17:30 | |
| persia | cinerama: Yes, but one only drinks that on 2nd March | 17:31 |
| *** PollyZ has joined #openstack-infra | 17:31 | |
| jeblair | i wonder if i can get shiner around here | 17:31 |
| mordred | jeblair: I believe we export it now | 17:31 |
| jeblair | (not hipster enough? too hipster? i don't know!) | 17:31 |
| mordred | and I know I've had shiner in Palo Alto before | 17:31 |
| AJaeger | jeblair: there're two changes by ianw in project-config related to zuul-worker: https://review.openstack.org/333653 (Don't use devuser for zuul-worker) and https://review.openstack.org/#/c/334264/1 (Specify the ZUUL_USER_SSH_PUBLIC_KEY) - could you and others review them, please? | 17:31 |
| clarkb | I can get it at fred meyet | 17:31 |
| mordred | that does not answer your questoin though | 17:31 |
| nibalizer | mordred: thanks :) | 17:32 |
| jeblair | AJaeger: we're talking about *beer*! ;) | 17:32 |
| AJaeger | jeblair: yeah, I know ;) | 17:32 |
| *** sdake_ has joined #openstack-infra | 17:32 | |
| AJaeger | jeblair: sorry, for having interrupted such an essential topic | 17:33 |
| jeblair | priorities! | 17:33 |
| *** hongbin_ has quit IRC | 17:34 | |
| *** jamesmcarthur has joined #openstack-infra | 17:34 | |
| * AJaeger hides in the corner - and considers what kind of beer is in his cellar. Definitely neither America nor Germany ;) | 17:34 | |
| *** markus__ has joined #openstack-infra | 17:34 | |
| *** jsavak has joined #openstack-infra | 17:35 | |
| *** sdake has quit IRC | 17:35 | |
| *** fawadkhaliq has joined #openstack-infra | 17:35 | |
| jeblair | oh neat, that's how you get data into dib | 17:35 |
| TheJulia | Is gerrit throwing proxy errors for every other click for anyone else? | 17:35 |
| *** markusry has quit IRC | 17:36 | |
| *** erlon has quit IRC | 17:36 | |
| greghaynes | Yep, that is the whole reason the extra-data.d phase exists - to copy data in to the image | 17:37 |
| jeblair | AJaeger: +1 from me -- i'd like to have + votes from greghaynes and pabelanger. | 17:37 |
| * greghaynes adds to review queue | 17:37 | |
| hurgleburgler | mordred that patch is still very much in flux, so any suggestions that you want to make would be great | 17:37 |
| AJaeger | thanks, jeblair. | 17:38 |
| jeblair | TheJulia: clarkb mentioned increased GC activity which may be slowing it enough to cause what you report; he suggested maybe we should restart it in a few hours... | 17:38 |
| mordred | hurgleburgler: cool - I'm thrilled you're hacking on t | 17:38 |
| AJaeger | TheJulia: happens for me right now quite often as well - just not every other click ;) | 17:39 |
| *** fawadkhaliq has quit IRC | 17:39 | |
| AJaeger | jeblair, clarkb : Yeah, seems to be time again ;( | 17:39 |
| TheJulia | AJaeger: okay, maybe once every three clicks :) | 17:40 |
| *** markus__ has quit IRC | 17:40 | |
| *** markusry has joined #openstack-infra | 17:40 | |
| fungi | yeah, we've got a maintenance window coming up at 20:00 utc anyway, so we'll kive it a kick while zuul et al are down | 17:42 |
| *** mriedem has joined #openstack-infra | 17:43 | |
| *** signed8bit is now known as signed8bit_Zzz | 17:43 | |
| *** kzaitsev_mb has joined #openstack-infra | 17:44 | |
| *** jamesmcarthur has quit IRC | 17:44 | |
| *** jed56 has quit IRC | 17:45 | |
| *** dtardivel has quit IRC | 17:47 | |
| *** sdague has quit IRC | 17:47 | |
| *** kzaitsev_mb has quit IRC | 17:49 | |
| *** watersoul has joined #openstack-infra | 17:51 | |
| *** baoli has joined #openstack-infra | 17:51 | |
| *** baoli has quit IRC | 17:53 | |
| *** baoli has joined #openstack-infra | 17:53 | |
| *** rbrndt has quit IRC | 17:53 | |
| *** pradk has quit IRC | 17:54 | |
| *** pradk_ is now known as pradk | 17:54 | |
| *** sputnik13 has quit IRC | 17:54 | |
| *** sarob has joined #openstack-infra | 17:55 | |
| *** sarob has quit IRC | 17:55 | |
| *** signed8bit_Zzz is now known as signed8bit | 17:55 | |
| *** sarob has joined #openstack-infra | 17:55 | |
| *** salv-orlando has joined #openstack-infra | 17:58 | |
| *** trown is now known as trown|lunch | 17:59 | |
| *** asettle has quit IRC | 18:00 | |
| *** daemontool has quit IRC | 18:00 | |
| *** sdague has joined #openstack-infra | 18:02 | |
| *** bmjen has joined #openstack-infra | 18:03 | |
| notmorgan | oh look it's a hurgleburgler | 18:05 |
| *** chem` has joined #openstack-infra | 18:05 | |
| notmorgan | o/ | 18:06 |
| *** salv-orlando has quit IRC | 18:06 | |
| notmorgan | morning.. or afternoon...or whatever everyone! :) | 18:06 |
| bkero | howdy | 18:06 |
| hurgleburgler | Allo! (◠‿◠✿)ノ | 18:06 |
| *** chem has quit IRC | 18:07 | |
| *** jsavak has quit IRC | 18:08 | |
| *** baoli has quit IRC | 18:08 | |
| *** baoli has joined #openstack-infra | 18:09 | |
| *** sputnik13 has joined #openstack-infra | 18:09 | |
| *** jsavak has joined #openstack-infra | 18:09 | |
| *** akshai has quit IRC | 18:11 | |
| *** _nadya_ has quit IRC | 18:15 | |
| *** mgould is now known as mgould|afk | 18:15 | |
| *** _nadya_ has joined #openstack-infra | 18:17 | |
| *** Sukhdev has joined #openstack-infra | 18:17 | |
| EmilienM | is it me or gerrit is super slow? | 18:18 |
| *** rbrndt has joined #openstack-infra | 18:18 | |
| *** baoli has quit IRC | 18:18 | |
| *** baoli has joined #openstack-infra | 18:18 | |
| *** pvaneck has joined #openstack-infra | 18:19 | |
| *** ddieterly[away] has quit IRC | 18:19 | |
| *** baoli has quit IRC | 18:20 | |
| *** akshai has joined #openstack-infra | 18:20 | |
| *** baoli has joined #openstack-infra | 18:21 | |
| *** inc0 has quit IRC | 18:21 | |
| pleia2 | looks like the post job to update specs.o.o didn't run/work for the wiki spec yesterday, was that a known issue? I could use some help fixing the situation | 18:22 |
| EmilienM | got 502 errors on gerrit | 18:22 |
| EmilienM | should we restart it? | 18:23 |
| pleia2 | EmilienM: it's been mentioned, the maintenance window is coming up in 90 minutes anyway, so it doesn't make sense to restart early | 18:24 |
| EmilienM | ok | 18:24 |
| pleia2 | EmilienM: go enjoy Canada Day :D | 18:24 |
| EmilienM | pleia2: how do you know? :) | 18:24 |
| EmilienM | pleia2: but yeah, I should leave | 18:25 |
| pleia2 | my calendar told me (and I have a sister lives in Alberta) | 18:25 |
| pleia2 | s/lives/living | 18:25 |
| *** Liuqing has quit IRC | 18:26 | |
| sdague | I would second that gerrit is the sad right now | 18:27 |
| *** pblaho has quit IRC | 18:27 | |
| notmorgan | pleia2: i like using simple presnet tense instead of present progressive when present progressive is appropriate | 18:28 |
| *** jcoufal has quit IRC | 18:28 | |
| *** inc0 has joined #openstack-infra | 18:28 | |
| notmorgan | pleia2: :P | 18:28 |
| pleia2 | notmorgan: writing is hard | 18:28 |
| fungi | pleia2: it probably raced the post job for the other spec i approved at the same time | 18:29 |
| fungi | that's the usual cause | 18:29 |
| pleia2 | fungi: ah, makes sense | 18:29 |
| notmorgan | pleia2: right?! | 18:29 |
| fungi | pleia2: if so, the next spec change which merges should solve it | 18:29 |
| notmorgan | pleia2: i also realize... wow, i spent too much time in english classes / writing in college :P | 18:29 |
| pleia2 | notmorgan: haah | 18:29 |
| sdague | pleia2: except, the current 1 in 4 502 error rate on gerrit is making getting reviews done before the outage... difficult | 18:30 |
| *** jlanoux has quit IRC | 18:30 | |
| *** bmjen has quit IRC | 18:30 | |
| notmorgan | sdague: yay 502s :( | 18:30 |
| *** amrith has quit IRC | 18:30 | |
| mtreinish | oh, that explains why my gertty sync is still going. I thought it was just something on my end | 18:31 |
| notmorgan | i mean.. the opposite of that. | 18:31 |
| *** sputnik13 has quit IRC | 18:31 | |
| *** amrith has joined #openstack-infra | 18:31 | |
| *** hurgleburgler has quit IRC | 18:31 | |
| *** salv-orlando has joined #openstack-infra | 18:32 | |
| *** bmjen has joined #openstack-infra | 18:34 | |
| *** xyang1 has quit IRC | 18:35 | |
| *** hurgleburgler has joined #openstack-infra | 18:36 | |
| *** _nadya_ has quit IRC | 18:38 | |
| *** ayoung has quit IRC | 18:39 | |
| *** rossella_s has quit IRC | 18:39 | |
| *** rossella_s has joined #openstack-infra | 18:39 | |
| *** salv-orlando has quit IRC | 18:40 | |
| *** xyang1 has joined #openstack-infra | 18:41 | |
| *** PollyZ has quit IRC | 18:41 | |
| fungi | nibalizer: it looks like https://review.openstack.org/310948 hasn't garnered much interest yet, though has suggestions from clarkb and ianw so i didn't approve it yesterday | 18:42 |
| *** PollyZ has joined #openstack-infra | 18:42 | |
| nibalizer | fungi: ok | 18:42 |
| *** Sukhdev has quit IRC | 18:45 | |
| *** PollyZ has quit IRC | 18:45 | |
| *** karthik__ has joined #openstack-infra | 18:45 | |
| *** thiagop-afk is now known as TheJulia_ | 18:46 | |
| *** TheJulia_ is now known as thiagop | 18:46 | |
| *** PollyZ has joined #openstack-infra | 18:47 | |
| *** HeOS has quit IRC | 18:53 | |
| mriedem | when is the gerrit outage? | 18:53 |
| mriedem | oh nvm | 18:53 |
| *** tpsilva has quit IRC | 18:58 | |
| *** nwkarsten has quit IRC | 19:04 | |
| *** PollyZ_ has joined #openstack-infra | 19:04 | |
| *** chem` has quit IRC | 19:05 | |
| *** PollyZ has quit IRC | 19:07 | |
| *** roxanagh_ has quit IRC | 19:09 | |
| *** PollyZ_ has quit IRC | 19:09 | |
| *** karthik__ has quit IRC | 19:09 | |
| *** PollyZ has joined #openstack-infra | 19:09 | |
| *** ddieterly has joined #openstack-infra | 19:09 | |
| *** karthik__ has joined #openstack-infra | 19:12 | |
| *** Sukhdev has joined #openstack-infra | 19:17 | |
| *** cody-somerville has joined #openstack-infra | 19:18 | |
| fungi | i'm accumulating server upgrade notes for zuul and static at https://etherpad.openstack.org/p/upgrade-zuul-trusty | 19:18 |
| fungi | i've already got the replacement servers launched and identified there | 19:18 |
| *** Sukhdev has quit IRC | 19:23 | |
| *** sputnik13 has joined #openstack-infra | 19:23 | |
| *** trown|lunch is now known as trown | 19:23 | |
| *** adrian_otto has quit IRC | 19:23 | |
| *** Apoorva has quit IRC | 19:24 | |
| *** sarob has quit IRC | 19:29 | |
| *** Ravikiran_K has quit IRC | 19:31 | |
| *** shardy is now known as shardy_afk | 19:32 | |
| *** harlowja_at_home has joined #openstack-infra | 19:33 | |
| *** yolanda has quit IRC | 19:34 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: package-installs: add list to arch and "not-arch" list https://review.openstack.org/335748 | 19:34 |
| zigo | pabelanger: Hi there! Any progress on the reprepro job? | 19:35 |
| zigo | Or any hints to us, on how you want it made? | 19:36 |
| zigo | Any of the 2 would work ... | 19:36 |
| *** pt_15 has quit IRC | 19:36 | |
| *** zz_zz_ja has quit IRC | 19:36 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul: Make job registration with labels optional https://review.openstack.org/336311 | 19:38 |
| clarkb | jeblair: ^ that took entirely too long to get pushed today | 19:38 |
| clarkb | but there we have it | 19:38 |
| clarkb | I am going to try sedning out the python35 email before the downtime stuff to | 19:39 |
| clarkb | we are adding ~240 jobs for python35 which is far less than we hvae removed so far \o/ | 19:39 |
| *** jswarren_ has joined #openstack-infra | 19:39 | |
| *** zz_ja has joined #openstack-infra | 19:40 | |
| *** zz_dimtruck is now known as dimtruck | 19:40 | |
| *** bswartz has quit IRC | 19:40 | |
| *** xyang1 has quit IRC | 19:41 | |
| *** hrubi has quit IRC | 19:41 | |
| *** coreycb has quit IRC | 19:41 | |
| *** spiffxp has quit IRC | 19:41 | |
| *** boris-42 has quit IRC | 19:41 | |
| *** laurelm has quit IRC | 19:41 | |
| *** kambiz has quit IRC | 19:41 | |
| *** xyang1 has joined #openstack-infra | 19:41 | |
| *** spiffxp has joined #openstack-infra | 19:41 | |
| *** stevebaker has quit IRC | 19:41 | |
| *** tristanC has quit IRC | 19:41 | |
| *** andymccr has quit IRC | 19:41 | |
| *** SpamapS has quit IRC | 19:41 | |
| *** mgould|afk has quit IRC | 19:41 | |
| *** elarson has quit IRC | 19:41 | |
| *** itsuugo has quit IRC | 19:41 | |
| *** furlongm_ has joined #openstack-infra | 19:42 | |
| *** eharney has quit IRC | 19:42 | |
| *** yamahata has quit IRC | 19:42 | |
| *** liusheng has quit IRC | 19:42 | |
| *** gsilvis has quit IRC | 19:42 | |
| *** notmyname has quit IRC | 19:42 | |
| *** j_king has quit IRC | 19:42 | |
| *** jswarren has quit IRC | 19:42 | |
| *** fungi_lounge has quit IRC | 19:42 | |
| *** jtomasek has quit IRC | 19:42 | |
| *** furlongm has quit IRC | 19:42 | |
| *** thingee has quit IRC | 19:42 | |
| *** _alastor_ has quit IRC | 19:42 | |
| *** eglute has quit IRC | 19:42 | |
| *** jgriffith has quit IRC | 19:42 | |
| *** kevinbenton has quit IRC | 19:42 | |
| *** zeus has quit IRC | 19:42 | |
| *** uberjay has quit IRC | 19:42 | |
| *** lyz has quit IRC | 19:42 | |
| *** salv-orlando has joined #openstack-infra | 19:42 | |
| *** sflanigan has quit IRC | 19:42 | |
| *** gabriel-bezerra has quit IRC | 19:42 | |
| *** liusheng has joined #openstack-infra | 19:42 | |
| *** tristanC has joined #openstack-infra | 19:43 | |
| *** mgould|afk has joined #openstack-infra | 19:43 | |
| *** _alastor_ has joined #openstack-infra | 19:44 | |
| *** j_king has joined #openstack-infra | 19:44 | |
| *** boris-42 has joined #openstack-infra | 19:44 | |
| *** gsilvis has joined #openstack-infra | 19:44 | |
| AJaeger | clarkb: https://review.openstack.org/336263 and https://review.openstack.org/336413 are further job removals | 19:44 |
| clarkb | AJaeger: noted | 19:44 |
| *** kevinbenton has joined #openstack-infra | 19:44 | |
| fungi | zigo: it's canada day, so pabelanger is hopefully enjoying some time with his family. as to your question, i don't know the status on those at the moment though we're about to start a maintenance window for some server upgrades so i can't research it right now | 19:44 |
| zigo | fungi: Thanks for the update. | 19:45 |
| zigo | fungi: FYI, mordred wrote that pabelanger would work on it, which is why I asked. | 19:45 |
| fungi | pleia2: should we irc a 10-minute warning? | 19:46 |
| *** ayoung has joined #openstack-infra | 19:46 | |
| *** stevebaker has joined #openstack-infra | 19:46 | |
| *** itsuugo has joined #openstack-infra | 19:47 | |
| *** zeus has joined #openstack-infra | 19:47 | |
| *** elarson has joined #openstack-infra | 19:47 | |
| *** jgriffith has joined #openstack-infra | 19:47 | |
| *** notmyname has joined #openstack-infra | 19:47 | |
| *** andymccr has joined #openstack-infra | 19:47 | |
| *** coreycb has joined #openstack-infra | 19:47 | |
| *** kambiz has joined #openstack-infra | 19:47 | |
| *** eglute has joined #openstack-infra | 19:47 | |
| *** laurelm has joined #openstack-infra | 19:47 | |
| *** zeus is now known as Guest31961 | 19:47 | |
| *** hrubi has joined #openstack-infra | 19:48 | |
| *** uberjay has joined #openstack-infra | 19:48 | |
| *** lyz has joined #openstack-infra | 19:48 | |
| pleia2 | fungi: yeah, I can do | 19:48 |
| fungi | thanks! | 19:49 |
| clarkb | would people here like to proof my python35 email? otherwise I am going to send it nowish | 19:49 |
| *** sdake_ has quit IRC | 19:50 | |
| fungi | i'm in favor of you just sending it | 19:50 |
| karthik__ | ianw_pto, AJaeger: I have made some changes to the devstack-gate for Openstack/cinder... Please will you be able to review it in your free time? | 19:50 |
| *** salv-orlando has quit IRC | 19:50 | |
| mordred | clarkb: I support the words you have in your brian | 19:51 |
| mordred | brain | 19:51 |
| mordred | not in your brian - that's weird | 19:51 |
| clarkb | I didn't know I had a Brian | 19:51 |
| *** Nikolay_St has joined #openstack-infra | 19:51 | |
| *** Guest31961 is now known as zeus | 19:51 | |
| *** zeus has joined #openstack-infra | 19:51 | |
| pleia2 | #status notice The infrastructure team is taking Gerrit offline for maintenance beginning shortly after 20:00 UTC to upgrade the Zuul and static.openstack.org servers. We aim to have it back online around 22:00 UTC. | 19:52 |
| openstackstatus | pleia2: sending notice | 19:52 |
| clarkb | ok mail sent | 19:52 |
| fungi | anyone helping on the maintenance, feel free to brainstorm things i've forgotten at https://etherpad.openstack.org/p/upgrade-zuul-trusty | 19:52 |
| -openstackstatus- NOTICE: The infrastructure team is taking Gerrit offline for maintenance beginning shortly after 20:00 UTC to upgrade the Zuul and static.openstack.org servers. We aim to have it back online around 22:00 UTC. | 19:53 | |
| fungi | i figure at the start of the window we'll stop zuul and gerrit, halt static.openstack.org and then immediately update dns records | 19:53 |
| * clarkb +2's AJaeger's change but doesn't approve due to ^ | 19:53 | |
| *** SpamapS has joined #openstack-infra | 19:54 | |
| *** pt_15 has joined #openstack-infra | 19:54 | |
| *** Daisy has joined #openstack-infra | 19:54 | |
| *** sflanigan has joined #openstack-infra | 19:55 | |
| *** sflanigan has joined #openstack-infra | 19:55 | |
| openstackstatus | pleia2: finished sending notice | 19:55 |
| *** eharney has joined #openstack-infra | 19:55 | |
| clarkb | I am impressed I think we removed about 1300 jobs in this job cleanup AJaeger has been doing | 19:56 |
| *** jtomasek has joined #openstack-infra | 19:56 | |
| *** yamahata has joined #openstack-infra | 19:56 | |
| *** gabriel-bezerra has joined #openstack-infra | 19:56 | |
| jeblair | clarkb, cinerama, mordred: look what i found! http://amo-probos.org/static/shiner.jpg | 19:56 |
| * jeblair is prepared for maintenance! | 19:57 | |
| fungi | best maintenance prep EVAR | 19:57 |
| * fungi notes they have that at the grocery here too. probably just as terrible | 19:58 | |
| *** asettle has joined #openstack-infra | 19:58 | |
| pleia2 | hehe | 19:58 |
| *** asettle has quit IRC | 19:58 | |
| fungi | anybody want to claim steps on the plan at https://etherpad.openstack.org/p/upgrade-zuul-trusty ? | 19:58 |
| fungi | also please add things i've forgotten if you know them ;) | 19:59 |
| *** Nikolay_St has quit IRC | 19:59 | |
| *** Daisy has quit IRC | 19:59 | |
| *** harlowja_at_home has quit IRC | 19:59 | |
| *** piet has quit IRC | 19:59 | |
| *** piet has joined #openstack-infra | 20:00 | |
| jeblair | fungi: are we halting static to try to avoid the interminable detaching state? | 20:00 |
| fungi | jeblair: YEP | 20:00 |
| *** inc0 has quit IRC | 20:00 | |
| jeblair | ok | 20:00 |
| jeblair | neat | 20:00 |
| fungi | my clock says we're 60 seconds into the window now, btw. so free to start any time we like | 20:00 |
| *** HeOS has joined #openstack-infra | 20:01 | |
| jeblair | i claimed #2, so i'll start on that :) | 20:01 |
| fungi | pleia2: want to do the status alert? | 20:01 |
| pleia2 | right, on it | 20:02 |
| jeblair | that horizon test looks like it's not particularly about to wrap up, so i will hard stop things now. | 20:02 |
| pleia2 | #status alert Gerrit is offline for maintenance until approximately 22:00 UTC | 20:02 |
| openstackstatus | pleia2: sending alert | 20:02 |
| fungi | i notice i forgot to add the reverse dns records on the replacement servers, so i'll add a step for me to fix that too | 20:03 |
| *** ddieterly is now known as ddieterly[away] | 20:03 | |
| jeblair | gerrit is stopping, i have saved zuul queues | 20:03 |
| fungi | i'll halt static.o.o now | 20:03 |
| * clarkb is around and can help as necessary | 20:03 | |
| jeblair | gerrit and zuul have stopped | 20:03 |
| fungi | and working on dns updates now | 20:04 |
| jeblair | i'm going to stop launchers and mergers now | 20:04 |
| -openstackstatus- NOTICE: Gerrit is offline for maintenance until approximately 22:00 UTC | 20:05 | |
| *** nwkarsten has joined #openstack-infra | 20:05 | |
| *** ChanServ changes topic to "Gerrit is offline for maintenance until approximately 22:00 UTC" | 20:05 | |
| *** weshay has quit IRC | 20:06 | |
| pleia2 | I can do 8, but I'll need some direction | 20:06 |
| clarkb | pleia2: I can walk you through it its basically ssh/ansible for loop to service iptables-persistent restart | 20:07 |
| *** kgiusti has left #openstack-infra | 20:07 | |
| clarkb | pleia2: you can run iptables -L -n before and after to see the ip addrs update | 20:07 |
| * clarkb will work on a list of those hosts on the etherpad | 20:07 | |
| *** karthi___ has joined #openstack-infra | 20:07 | |
| *** Apoorva has joined #openstack-infra | 20:08 | |
| openstackstatus | pleia2: finished sending alert | 20:08 |
| pleia2 | clarkb: ah, easy enough | 20:08 |
| *** karthik__ has quit IRC | 20:08 | |
| fungi | forward dns done now (including devstack.org and trystack.org). working on reverse dns | 20:09 |
| jeblair | clarkb, pleia2: i think we don't have many iptables to restart | 20:10 |
| *** nwkarsten has quit IRC | 20:10 | |
| *** roxanagh_ has joined #openstack-infra | 20:10 | |
| clarkb | jeblair: oh right because things connect to zuul not the other way around | 20:10 |
| fungi | yeah, the only one i could think of was graphite | 20:10 |
| jeblair | clarkb, pleia2: yeah -- i think it's only graphite? | 20:10 |
| clarkb | the things I have listed all connect to zuul so should be fine | 20:10 |
| pleia2 | even better | 20:10 |
| *** achanda has joined #openstack-infra | 20:10 | |
| jeblair | you can totally still use ansible tho :) | 20:11 |
| *** weshay has joined #openstack-infra | 20:11 | |
| *** _nadya_ has joined #openstack-infra | 20:11 | |
| *** Swami has quit IRC | 20:12 | |
| fungi | reverse dns done | 20:12 |
| *** Sukhdev has joined #openstack-infra | 20:12 | |
| pleia2 | if it's just the one I might just play service restart | 20:12 |
| fungi | i'm working on the static.o.o volumes now | 20:12 |
| *** javeriak_ has joined #openstack-infra | 20:12 | |
| jeblair | hrm. we have no logs on the zuul mergers | 20:12 |
| clarkb | they don't even run systmed/journald yet | 20:12 |
| jeblair | sorry, i meant service logs from z-m itself | 20:13 |
| *** markusry has quit IRC | 20:13 | |
| *** javeriak has quit IRC | 20:13 | |
| *** EricGonczer_ has joined #openstack-infra | 20:14 | |
| *** PollyZ has quit IRC | 20:14 | |
| rm_work | ah rofl came to ask if anyone else was having gerrit issues. saw topic T_T | 20:15 |
| achanda | getting 503 from gerrit | 20:15 |
| fungi | achanda: see /topic | 20:15 |
| jeblair | rm_work: thanks! it's been 5 years we've been doing this -- *someone* was bound to actually read the topic someday ;) | 20:15 |
| achanda | fungi, thanks, my bad! | 20:15 |
| *** PollyZ has joined #openstack-infra | 20:15 | |
| rm_work | happened EXACTLY when i a was futzing with vpn / bastion / routes | 20:16 |
| rm_work | such timing | 20:16 |
| jeblair | rm_work: we try | 20:16 |
| rm_work | lol | 20:16 |
| clarkb | jeblair: for step 9 gear should reconnect on its own? maybe? | 20:16 |
| fungi | okay, all volumes detached successfully from old static.o.o. definitely should just halt servers before doing volume detach if we have the opportunity | 20:16 |
| fungi | this goes so much faster | 20:17 |
| *** bswartz has joined #openstack-infra | 20:17 | |
| jeblair | okay, i dunno what's up with the zm logs -- i will carefully restart them when we are done to try to further diagnose that | 20:17 |
| *** salv-orlando has joined #openstack-infra | 20:17 | |
| fungi | volume add is going a bit slower than volume remove. i suspect nova waits for the instance to settle or something between successive attachments | 20:18 |
| jeblair | clarkb: i think/hope it should do it on its own | 20:18 |
| jeblair | clarkb: it at least performs a new getaddrinfo every time it tries to connect | 20:18 |
| jeblair | clarkb: so it would have to be cached lower than that layer to fail | 20:18 |
| fungi | i guess just wait on #9 and see if any of those don't dtrt | 20:19 |
| jeblair | the only thing left trying to connect to zuul's geard is nodepool, right? (i stopped the launchers and mergers -- mostly just because the mergers rarely get restarted) | 20:20 |
| clarkb | jeblair: yup should be | 20:21 |
| fungi | yay, all volumes (supposedly) attached successfully to the new static.o.o. i'm checking its work now ;) | 20:21 |
| *** EricGonczer_ has quit IRC | 20:21 | |
| jeblair | clarkb: does nodepool need a restart for any other reasons? (pick up any recent changes?) | 20:21 |
| fungi | grr... static.openstack.org/main14 is still showing "available" | 20:22 |
| clarkb | jeblair: not that I am aware of | 20:22 |
| jeblair | ok, then i guess we can leave it be and see what happens | 20:22 |
| jeblair | i'm also going to take the opportunity to delete zl git dirs | 20:23 |
| *** EricGonczer_ has joined #openstack-infra | 20:23 | |
| clarkb | jeblair: zm too maybe? | 20:23 |
| clarkb | or is that what yo umeant? | 20:23 |
| jeblair | clarkb: that is indeed what i meant :) | 20:23 |
| fungi | oh [expletive deleted] | 20:23 |
| fungi | we have a configdrive | 20:24 |
| jeblair | so we lost a slot? | 20:24 |
| fungi | ... yeah | 20:24 |
| clarkb | we shouldn't need a config drive there | 20:24 |
| jeblair | can we return the configdrive unused? | 20:24 |
| jeblair | i would like a refund | 20:24 |
| clarkb | jeblair: not without making a new instance I think | 20:24 |
| jeblair | clarkb: okay, but we can exchange our instance for one without a configdrive? | 20:25 |
| clarkb | yes we should be able to do that, I do not know why we are config driving now on those hosts | 20:25 |
| fungi | i can solve this by detaching all these, attaching them back to the old server, starting it back up, using vgreduce to shuffle any used extents off main14, then detach them all again and attach the used 13 back to the new server | 20:25 |
| fungi | but that will eat up a lot of time | 20:25 |
| jeblair | fungi: i agree that's a solution -- clarkb's may be faster | 20:25 |
| jeblair | Shrews, mordred: ^ possibly of interest | 20:25 |
| fungi | yes, i'd much prefer clarkb's solution first | 20:25 |
| *** Swami has joined #openstack-infra | 20:26 | |
| * clarkb pulls up the launcer to see if it is configurable there | 20:26 | |
| *** salv-orlando has quit IRC | 20:26 | |
| clarkb | you might have to checkout an old launch node script | 20:26 |
| *** ddieterly[away] is now known as ddieterly | 20:26 | |
| fungi | is configdrive irreversible after the instance is created? | 20:26 |
| jeblair | fungi: i think that's what clarkb was telling me | 20:26 |
| clarkb | fungi: yes I don't think there is a way of telling nova to remove it | 20:26 |
| fungi | as in do i need to detach all these and replace this instance and update dns again? | 20:26 |
| clarkb | sdague: or mriedem might be able to confirm | 20:27 |
| clarkb | it is configurable in launch-node.py | 20:27 |
| jeblair | clarkb: agreed, and defaults to true | 20:27 |
| *** woodster_ has joined #openstack-infra | 20:27 | |
| fungi | in that case, not a whole lot faster than the vgreduce option, but maybe still superior since it won't lose us a terabyte | 20:27 |
| clarkb | --config-drive=False or similar to turn it off | 20:28 |
| *** shardy_afk has quit IRC | 20:28 | |
| fungi | okay, so any alternative proposals? shall i press forward replacing this instance? | 20:28 |
| jeblair | fungi: (doesn't the vgreduce require at least one full fsck? -- i was imagining it to take hours) | 20:28 |
| fungi | jeblair: no fsck required. it moves extents online | 20:28 |
| fungi | or at least _can_ move extents while online | 20:28 |
| jeblair | and our volume is smaller than 14tb? | 20:28 |
| jeblair | er | 20:28 |
| jeblair | our volume and our fs? | 20:29 |
| clarkb | I am investigating if there is a way to remove the config drive | 20:29 |
| fungi | sorry, lvreduce first, then vgreduce | 20:29 |
| mriedem | config drive is static on the instance once you've created it | 20:29 |
| fungi | but both work online | 20:29 |
| jeblair | fungi: i'm still +1 on the new server | 20:29 |
| mordred | I'm also +1 on the new server | 20:29 |
| fungi | okay, i'll detach these and so that | 20:29 |
| mordred | and also wow what a 'fun' fail | 20:29 |
| jeblair | fungi: (right -- but there's an ext4 filesystem that's 14tb, isn't there?) | 20:29 |
| jeblair | mordred: do you have any idea why we defaulted that to 'true' in launch-node? | 20:29 |
| fungi | jeblair: oh, grr. you're right, need to resize2fs. that does require a fsck | 20:30 |
| mordred | jeblair: habit from elsewhere. no reason we need to in launch-node | 20:30 |
| fungi | anyway, the new static.o.o is halted and i'm booting a replacement while i detach these volumes | 20:31 |
| mordred | jeblair: in the general case, having configdrive=True means you can use glean instead of cloud-init which means you can use the keypair extension without needing to ever hit the scales-poorly metadata service | 20:31 |
| jeblair | - default=False) | 20:32 |
| jeblair | + default=True) | 20:32 |
| mordred | jeblair: but we have not gotten to booting from our own base images yet | 20:32 |
| jeblair | that was in e14471dcc6daa4f33bba84ac1d3e7c091ada10ea with no explanation in the commit msg | 20:32 |
| jeblair | mordred: ok, good to know | 20:32 |
| clarkb | mriedem: ty for confirming | 20:32 |
| jeblair | mordred: so config_drive may be the future for us -- we may need to adapt to having one less cinder slot | 20:33 |
| *** kzaitsev_mb has joined #openstack-infra | 20:33 | |
| fungi | er, --config-drive doesn't take a value, and --no-config-drive doesn't seem to work? | 20:33 |
| mordred | jeblair: yah - it was forward-looking, but we haven't caught up with it yet. obviously running out of volume mounts early due to config-drive will change the design of the future a bit | 20:33 |
| mordred | jeblair: yah | 20:33 |
| jeblair | fungi: haha! | 20:33 |
| jeblair | fungi: store_true with default=true is a bad idea | 20:33 |
| jeblair | mordred: ^ :) | 20:33 |
| mordred | jeblair: it's certainly something we can't avoid dealing with | 20:33 |
| mordred | jeblair: yay! | 20:33 |
| fungi | yep, it's hilarious. i'll clone this and patch it | 20:33 |
| mordred | fungi: sorry | 20:34 |
| *** xyang1 has quit IRC | 20:34 | |
| fungi | nah, it's fine. really is sorta funny ;) | 20:34 |
| jeblair | like, maybe argparse should give you a heads-up on that one, huh? :) | 20:35 |
| *** javeriak_ has quit IRC | 20:35 | |
| *** PollyZ has quit IRC | 20:35 | |
| jeblair | "I noticed you really seem to want this value to be true..." | 20:35 |
| fungi | "--no-config-drive" is now working for me ;) | 20:35 |
| *** karthikp_ has joined #openstack-infra | 20:37 | |
| *** karthi___ has quit IRC | 20:37 | |
| *** iurygregory has quit IRC | 20:38 | |
| fungi | here's a new helpful nova error | 20:40 |
| mordred | yay | 20:40 |
| *** piet has quit IRC | 20:40 | |
| *** cody-somerville has quit IRC | 20:40 | |
| *** jsavak has quit IRC | 20:40 | |
| fungi | Cannot 'detach_volume' instance 78711ec1-78ce-41ce-b8dc-7d0c1fd3b7f1 while it is in task_state powering-off (HTTP 409) (Request-ID: req-96de3c63-093c-4943-951f-00387cb5184b) | 20:40 |
| *** jsavak has joined #openstack-infra | 20:41 | |
| fungi | apparently sometimes nova _doesn't_ like that? | 20:41 |
| fungi | i got through the first 3 before nova started complaining on the rest | 20:41 |
| fungi | guess i need to boot it back up again | 20:41 |
| mordred | fungi: so - to recap - detaching volumes when the server is running sometimes does not work. detaching volumes when the server is halted sometimes does not work | 20:41 |
| *** _nadya_ has quit IRC | 20:42 | |
| fungi | i think you can reduce that problem statement considerably ;) | 20:42 |
| pleia2 | hehe | 20:42 |
| fungi | yay nova! | 20:42 |
| fungi | Cannot 'reboot' instance 78711ec1-78ce-41ce-b8dc-7d0c1fd3b7f1 while it is in task_state powering-off | 20:42 |
| fungi | sdague: that ^ look in any way familiar? | 20:43 |
| *** eharney has quit IRC | 20:45 | |
| *** Sukhdev has quit IRC | 20:46 | |
| mordred | fungi: he's on vacation probaly. mriedem ^^ ? | 20:46 |
| fungi | oh, he had popped in a few minutes ago | 20:47 |
| mordred | fungi: he's theoretically half-here | 20:47 |
| fungi | anyway, i seem to be successfully detaching again, so whatever was going on was temporary | 20:47 |
| mordred | fungi: oh good | 20:47 |
| mriedem | i'm here | 20:47 |
| mtreinish | mordred: it's after 4pm on a friday, he's gone :) | 20:47 |
| mriedem | powering-off is a task state | 20:47 |
| mriedem | sdague is going camping for 2 weeks or something | 20:47 |
| mriedem | so yeah he's gone | 20:47 |
| fungi | oh, awesome. he should definitely be camping and not in here | 20:48 |
| mordred | yup | 20:48 |
| * mriedem is writing the soul crushing blueprint freeze email to the dev list | 20:48 | |
| mordred | mriedem: just rename whatever emotion you're feeling "happy" | 20:49 |
| fungi | "your ideas are sub-par and will not be included in the next release. drown your sorrows in booze" | 20:49 |
| *** rfolco has quit IRC | 20:49 | |
| fungi | awesome. all 14 volumes are back to available finally | 20:49 |
| mordred | fungi: I feel like you were just suggesting to me personally that it's time to start drinking | 20:49 |
| mriedem | mordred: i don't feel emotion | 20:49 |
| mordred | mriedem: cool. so if you redefine that, you can constantly be "happy" | 20:50 |
| mriedem | sure | 20:50 |
| mriedem | "your face is telling me that my face should be smiling" | 20:50 |
| mordred | ++ | 20:50 |
| fungi | updating dns now | 20:51 |
| fungi | )remembered to set reverse dns on the new server) | 20:51 |
| *** salv-orlando has joined #openstack-infra | 20:51 | |
| fungi | also, no configdrive this time, i confirmed | 20:52 |
| fungi | just the rootfs and the ephemeral disk | 20:52 |
| *** EricGonc_ has joined #openstack-infra | 20:52 | |
| *** EricGonczer_ has quit IRC | 20:55 | |
| fungi | a/aaaa records for all 3 domains updated to the new-new addresses for static.o.o now | 20:57 |
| fungi | starting to attach volumes | 20:57 |
| *** jsavak has quit IRC | 20:58 | |
| *** asettle has joined #openstack-infra | 20:59 | |
| *** salv-orlando has quit IRC | 21:00 | |
| *** raildo is now known as raildo-afk | 21:01 | |
| *** Sukhdev has joined #openstack-infra | 21:01 | |
| fungi | finally--all 14 volumes attached successfully and showing in use this time | 21:02 |
| fungi | gonna see if it can assemble the vg now | 21:02 |
| mordred | fungi: woot! | 21:03 |
| fungi | yep, we have volumes docs-draft, logs, mirror, static and tarballs present | 21:04 |
| fungi | adding to fstab now | 21:04 |
| *** aysyd has quit IRC | 21:07 | |
| jeblair | i cleaned up all the git repos on the mergers | 21:07 |
| jeblair | so that should be faster when we restart | 21:07 |
| *** Sukhdev has quit IRC | 21:08 | |
| *** Sukhdev_ has joined #openstack-infra | 21:08 | |
| fungi | oh, er, ew... need to chown all this too | 21:08 |
| fungi | uids/gids are not the same as on the old server | 21:09 |
| clarkb | ugh | 21:09 |
| clarkb | that could take days? | 21:09 |
| *** akshai has quit IRC | 21:09 | |
| clarkb | might be easier to change the uids/gids | 21:09 |
| *** mriedem is now known as mriedem_away | 21:10 | |
| fungi | yeah, i'm thinking so | 21:10 |
| jeblair | ++ | 21:10 |
| *** sdake has joined #openstack-infra | 21:10 | |
| *** psilvad has quit IRC | 21:11 | |
| fungi | we have a bunch of different uids/gids showing up though, so i'm going to try and start the old server now and see what they were before | 21:11 |
| jeblair | interesting -- jenkins should be the only interesting one, right? | 21:13 |
| clarkb | jeblair: and root but root should be the same :) | 21:13 |
| fungi | i would think | 21:13 |
| jeblair | clarkb: yeah, but what about *systemd-root*? | 21:13 |
| * jeblair ducks | 21:13 | |
| nibalizer | jeblair: booooo | 21:13 |
| *** EricGonc_ has quit IRC | 21:14 | |
| nibalizer | maybe we should just be putting uid in the puppet resources for our users | 21:14 |
| nibalizer | that would be one way to get consistency, might be scary to roll out though | 21:14 |
| fungi | i think it's just called "lennart" (uid ∞) | 21:14 |
| jeblair | nibalizer: we do that for our user-users... it would be hard to do that for our services though | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| jeblair | nibalizer: (in that we can't guarantee as the authors of the zuul module that your zuul user should be $uid and won't conflict with some site-local thing) | 21:15 |
| nibalizer | ohhhhh | 21:15 |
| nibalizer | jeblair: yea and you have things like the postgres package adding a postgres user of uid <next uid> | 21:16 |
| jeblair | i suppose we could make that a configurable input, via hiera... | 21:16 |
| jeblair | nibalizer: right | 21:16 |
| nibalizer | and hopefully we don't end up having an ldap for <20 users | 21:17 |
| jeblair | yeah, we thought about it for afs, but then figured easier to just set the uids | 21:17 |
| jeblair | (that's the main reason we have it for our human users) | 21:18 |
| fungi | reviewday:x:1000:1000 jenkins:x:1005:1006 releasestatus:x:1006:1007 recheck:x:1007:1008 | 21:18 |
| fungi | 1000:1000 is currently ubuntu:ubuntu so i'm cleaning it out of any group memberships and deleting its homedir | 21:19 |
| mordred | fungi: such a useful user | 21:20 |
| *** asettle has quit IRC | 21:24 | |
| fungi | okay, old uids/gids restored through the use of vipw/vigr and the same with -s | 21:25 |
| pleia2 | nice work, fungi | 21:25 |
| *** jsavak has joined #openstack-infra | 21:25 | |
| fungi | i'm going to attempt to start apache now | 21:25 |
| pleia2 | we routinely forget about u/gids :\ | 21:25 |
| mordred | pleia2: u/gids are silly | 21:26 |
| mordred | pleia2: we should get rid of user accounts and just have code | 21:27 |
| pleia2 | they're something | 21:27 |
| pleia2 | mordred: run everything as root! | 21:27 |
| pleia2 | wait, no | 21:27 |
| fungi | i've pulled up an old log successfully, browsed tarballs via https... | 21:27 |
| fungi | anything anybody else wants to test out on the new static.o.o? | 21:28 |
| clarkb | docs draft | 21:28 |
| pleia2 | releases.o.o is forbidden | 21:28 |
| pleia2 | http://releases.openstack.org/newton/schedule.html | 21:28 |
| pleia2 | hm, might be my problem | 21:29 |
| fungi | i'm getting forbidden on a recent docs-draft from my browser history too | 21:29 |
| fungi | You don't have permission to access /82/335582/1/check/gate-bindep-docs/e355ca4//doc/build/html/news.html on this server. | 21:29 |
| jeblair | AH01630: client denied by server configuration: /srv/static/releases/newton/schedule.html | 21:30 |
| pleia2 | yeah, it's not just me | 21:30 |
| jeblair | apache config? | 21:30 |
| fungi | that's what i'm thinking | 21:30 |
| jeblair | the 'allow from' thing maybe? | 21:30 |
| fungi | probably apache 2.4isms | 21:30 |
| fungi | if so, we should be able to work through those with everything else back online again | 21:31 |
| pleia2 | yeah | 21:31 |
| jeblair | Satisfy Any | 21:31 |
| jeblair | hrm, that's the new form, isn't it? | 21:31 |
| jeblair | oh no | 21:32 |
| jeblair | "Require all granted" is the thing we want | 21:32 |
| jeblair | and yeah, it's missing | 21:32 |
| jeblair | so ++ bring up and push fixing changes through | 21:32 |
| fungi | yep | 21:32 |
| *** jsavak has quit IRC | 21:32 | |
| *** jsavak has joined #openstack-infra | 21:33 | |
| jeblair | are we ready for me to start starting zuul? | 21:33 |
| fungi | so we're up to the starting zuul step i guess"? | 21:33 |
| fungi | i think so--go for it | 21:33 |
| fungi | should have had more than enough time for the most recent round of dns updates to propagate | 21:34 |
| *** zzelle_ has joined #openstack-infra | 21:34 | |
| *** tonytan4ever has joined #openstack-infra | 21:35 | |
| fungi | obviously we shouldn't reenqueue anything until gerrit is back online though | 21:37 |
| jeblair | fungi: yeah, also, i forgot to copy the queue files over, so it will take a little extra | 21:38 |
| pleia2 | yeah, lmk when I should bring gerrit back | 21:38 |
| *** gouthamr_ has joined #openstack-infra | 21:38 | |
| jeblair | okay, zuul and all its components appear to be running | 21:39 |
| jeblair | (i just spent some extra time checking that out) | 21:39 |
| jeblair | it even seems to be okay with starting without being able to connect to gerrit, which is nice | 21:39 |
| *** jsavak has quit IRC | 21:39 | |
| *** jsavak has joined #openstack-infra | 21:39 | |
| jeblair | the launchers have grabbed nodes from nodepool | 21:39 |
| pleia2 | ok, graphite iptables now? | 21:39 |
| fungi | yeah could have really done the graphite iptables update any time | 21:40 |
| pleia2 | done, gerrit? | 21:40 |
| fungi | good to know we can safely restart zuul with gerrit offline | 21:40 |
| fungi | i wondered about that bit | 21:40 |
| jeblair | pleia2: go for it | 21:40 |
| fungi | yep, gerrit time! | 21:40 |
| pleia2 | gerrit starting up | 21:40 |
| clarkb | exciting | 21:41 |
| pleia2 | Starting Gerrit Code Review: OK | 21:41 |
| *** gouthamr has quit IRC | 21:41 | |
| * fungi is so glad we conservatively budgeted 2 hours for this maintenance, leaving us an hour to deal with unforeseen challenges | 21:41 | |
| pleia2 | hopefully it's just java-ing now | 21:41 |
| jeblair | 2016-07-01 21:41:45,753 INFO zuul.source.Gerrit: Updating <Change 0x7fde8c6ac7d0 336719,1> | 21:41 |
| fungi | it reconnects QUICKLY! | 21:42 |
| pleia2 | yep, there it goes, web UI is up | 21:42 |
| jeblair | re-enqueing | 21:42 |
| fungi | who wants to hack on apache config fixes? | 21:42 |
| jeblair | and running jobs | 21:42 |
| jeblair | oh, heh | 21:42 |
| jeblair | we forgot to copy the zuul time database over | 21:42 |
| jeblair | we could probably do that even now and it would be okay | 21:43 |
| jeblair | or we could just leave it to rebuild | 21:43 |
| jeblair | (i vote leave it -- it will rebuild quickly) | 21:43 |
| fungi | i have no opinion. i'm fine with it just regenerating that on the fly from fresh samples | 21:43 |
| pleia2 | I can do the apache changes real quick | 21:43 |
| fungi | pleia2: awesome. i'm happy to review/approve those | 21:44 |
| jeblair | ooh, grafana is in xenial | 21:44 |
| fungi | hrm, python-ironicclient change in the gate with a ton of POST_FAILUREs | 21:45 |
| pleia2 | good grief, this apache template file | 21:45 |
| clarkb | I think we can just rebuild it | 21:46 |
| fungi | sahara-tests change exhibiting similar issues | 21:46 |
| *** watersoul_ has joined #openstack-infra | 21:47 | |
| *** lascii is now known as alaski | 21:47 | |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Add Apache 2.4 syntax for static.o.o https://review.openstack.org/336722 | 21:48 |
| fungi | yeah, i think all completing jobs are generating POST_FAILURE results | 21:48 |
| jeblair | i'll track down their launcher logs for the cause | 21:48 |
| *** baoli has quit IRC | 21:48 | |
| *** puiterwi1 has joined #openstack-infra | 21:48 | |
| *** furlongm has joined #openstack-infra | 21:48 | |
| *** javeriak has joined #openstack-infra | 21:48 | |
| fungi | i have a feeling there's something not quite right with log uploads | 21:48 |
| *** gouthamr_ has quit IRC | 21:48 | |
| *** puiterwijk has quit IRC | 21:49 | |
| jeblair | "rsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]" | 21:49 |
| *** puiterwi1 is now known as puiterwijk | 21:49 | |
| *** ddieterly is now known as ddieterly[away] | 21:49 | |
| mordred | host key? | 21:50 |
| fungi | rsync is definitely installed on static.o.o | 21:50 |
| fungi | oh, do they have the old host key cached? | 21:50 |
| fungi | do we puppet that? | 21:50 |
| *** nibalize1 has joined #openstack-infra | 21:50 | |
| fungi | it's not in hiera/group/zuul-merger.yaml | 21:51 |
| *** Adri2000_ has joined #openstack-infra | 21:51 | |
| jeblair | yes, they cache the keys | 21:51 |
| fungi | maybe it just caches them on first use? | 21:51 |
| *** david-lyle has quit IRC | 21:51 | |
| fungi | i can copy the old host keys over to the new server | 21:51 |
| jeblair | i will remove /home/zuul/.ssh/known_hosts from all launchers | 21:51 |
| *** tristanC_ has joined #openstack-infra | 21:51 | |
| fungi | or do you want to have them re-cache the new one? | 21:51 |
| fungi | okay, that's probably better | 21:51 |
| jeblair | let's use the new one | 21:51 |
| *** gsilvis_ has joined #openstack-infra | 21:51 | |
| fungi | yeah, generated with stronger defaults and whatnot | 21:51 |
| jeblair | done | 21:52 |
| jeblair | i could restart zuul and re-enqueue again | 21:52 |
| fungi | yeah, that's probably less painful than telling everyone to recheck all of these | 21:52 |
| *** afazekas|real has quit IRC | 21:53 | |
| *** harlowja has quit IRC | 21:53 | |
| *** markmcclain has quit IRC | 21:53 | |
| *** sigmavirus24 has quit IRC | 21:53 | |
| *** coreyob_ has quit IRC | 21:53 | |
| *** afazekas has joined #openstack-infra | 21:53 | |
| *** hasharAway has quit IRC | 21:53 | |
| *** furlongm_ has quit IRC | 21:53 | |
| *** Adri2000 has quit IRC | 21:53 | |
| *** tristanC has quit IRC | 21:53 | |
| *** nibalizer has quit IRC | 21:53 | |
| *** sc` has quit IRC | 21:53 | |
| *** coreyob has joined #openstack-infra | 21:53 | |
| *** markmcclain has joined #openstack-infra | 21:53 | |
| *** javeriak has quit IRC | 21:53 | |
| *** harlowja has joined #openstack-infra | 21:53 | |
| *** watersoul has quit IRC | 21:53 | |
| openstackgerrit | lynxis lazus proposed openstack-infra/jenkins-job-builder: docker-build-publish: add registry + server support https://review.openstack.org/336276 | 21:53 |
| *** hashar has joined #openstack-infra | 21:54 | |
| *** javeriak has joined #openstack-infra | 21:54 | |
| *** tlian has quit IRC | 21:55 | |
| *** tlian has joined #openstack-infra | 21:56 | |
| *** sigmavirus24 has joined #openstack-infra | 21:57 | |
| *** sc` has joined #openstack-infra | 21:57 | |
| jeblair | zuul restarted and re-enqueing | 21:57 |
| fungi | jeblair: thanks! | 21:57 |
| *** gsilvis has quit IRC | 21:57 | |
| jeblair | np. i'll push up my playbooks in a sec :) | 21:57 |
| fungi | i'm watching closely to see if stuff uploads this time | 21:57 |
| *** nibalize1 is now known as nibalizer | 21:58 | |
| *** jsavak has quit IRC | 21:58 | |
| *** salv-orlando has joined #openstack-infra | 21:58 | |
| openstackgerrit | lynxis lazus proposed openstack-infra/jenkins-job-builder: docker-build-publish: add registry + server support https://review.openstack.org/336276 | 21:59 |
| *** jsavak has joined #openstack-infra | 21:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add playbooks to start/stop all of zuul https://review.openstack.org/336724 | 21:59 |
| *** vhosakot has quit IRC | 21:59 | |
| jeblair | those act on all 17(!) zuul hosts :) | 21:59 |
| fungi | 17 _for_now_ ;) | 22:00 |
| fungi | i guess that has the added benefit of killing running jobs sooner so nodepool can reclaim the available capacity? | 22:00 |
| *** salv-orlando has quit IRC | 22:00 | |
| jeblair | yeah, that's why i did it | 22:01 |
| *** tonytan4ever has quit IRC | 22:01 | |
| fungi | still getting POST_FAILURE on jobs | 22:01 |
| *** tonytan4ever has joined #openstack-infra | 22:01 | |
| fungi | it did recreate /home/zuul/.ssh/known_hosts | 22:02 |
| *** rfolco has joined #openstack-infra | 22:02 | |
| fungi | and that looks like the correct ip address in its entry | 22:02 |
| fungi | so presumably now failing on something else | 22:03 |
| fungi | lots of exceptions while sending job completion packet | 22:04 |
| *** zeih has quit IRC | 22:05 | |
| fungi | oh, those are form 10 minutes ago | 22:05 |
| fungi | so probably associated with the restart | 22:05 |
| jeblair | "rsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: unexplained error (code 255) at io.c(226) | 22:05 |
| jeblair | again a helpful message | 22:05 |
| jeblair | zuul@zl01:/var/log/zuul$ /usr/bin/ssh -i /var/lib/zuul/ssh/id_rsa -S none -o StrictHostKeyChecking=no jenkins@static.openstack.org | 22:06 |
| jeblair | Permission denied (publickey). | 22:06 |
| *** javeriak_ has joined #openstack-infra | 22:06 | |
| *** nwkarsten has joined #openstack-infra | 22:06 | |
| *** javeriak has quit IRC | 22:07 | |
| *** zzelle_ has quit IRC | 22:07 | |
| jeblair | the jenkins authorized_keys files on old/new static servers seem to match | 22:07 |
| fungi | oh! | 22:08 |
| fungi | this one's my fault. i changed jenkins's uid/gid | 22:08 |
| fungi | fixing | 22:08 |
| jeblair | fungi: what did you miss? (i'm still not seeing it) | 22:08 |
| fungi | hrm, no | 22:09 |
| *** karthikp_ has quit IRC | 22:10 | |
| jeblair | oh there it is | 22:11 |
| *** nwkarsten has quit IRC | 22:11 | |
| fungi | okay, fixed | 22:11 |
| jeblair | fungi: /home/jenkins uid is 3000 | 22:11 |
| fungi | it was _both_ :/ | 22:11 |
| jeblair | fungi: yep, lgtm | 22:11 |
| jeblair | ssh access from zl01 works | 22:11 |
| jeblair | i will do the restart dance again | 22:11 |
| fungi | i had to rerun vipw and vigr (and -s) to get rid of the "extra" jenkins at 3000:3001 | 22:12 |
| fungi | and then re-chown ~jenkins | 22:12 |
| jeblair | aaaah | 22:12 |
| *** ddieterly[away] is now known as ddieterly | 22:12 | |
| *** rfolco has quit IRC | 22:12 | |
| fungi | puppet had created that entry but not the others for reviewday, recheck and releasestatus | 22:12 |
| *** hashar has quit IRC | 22:13 | |
| fungi | so i got overzealous in adding them and forgot to remove/replace teh jenkins account | 22:13 |
| *** matt-borland has quit IRC | 22:14 | |
| *** javeriak has joined #openstack-infra | 22:15 | |
| *** javeriak_ has quit IRC | 22:15 | |
| *** rlandy has quit IRC | 22:16 | |
| jeblair | zuul is restarted and reenqueing | 22:16 |
| fungi | thanks jeblair | 22:16 |
| fungi | pleia2: minor suggestion for 336722, i think we can reasonably simplify that addition | 22:17 |
| pleia2 | fungi: thanks | 22:17 |
| fungi | if anybody is using our static.openstack.org puppet class, an unexpected apache 2.4ism is the least of their worries | 22:18 |
| fungi | so i'd rather we didn't make those templates any more complex than they already are if we can help it | 22:18 |
| fungi | if it were in one of our other puppet modules i would be more inclined to agree with the additional level of caution | 22:19 |
| *** berendt has joined #openstack-infra | 22:19 | |
| berendt | tarballs.openstack.org is still offline/not working, is this related to the maintenance of gerrit/static? | 22:20 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Add Apache 2.4 syntax for static.o.o https://review.openstack.org/336722 | 22:20 |
| fungi | berendt: tarballs.openstack.org should have been up and available for the past ~40 minutes | 22:20 |
| berendt | Ah, it is :) | 22:20 |
| *** sheeprine has quit IRC | 22:21 | |
| berendt | I think the review proposed by Elizabeth is related to this issue. | 22:21 |
| fungi | berendt: oh! | 22:21 |
| fungi | berendt: yep, it's working via https but not http | 22:21 |
| fungi | and i hadn't checked http | 22:21 |
| fungi | and yes, 336722 will get the http version working again too | 22:21 |
| pleia2 | interesting that it's working via https | 22:21 |
| fungi | indeed | 22:21 |
| berendt | yes, little bit strange | 22:21 |
| *** Douhet has quit IRC | 22:23 | |
| *** yamamoto has joined #openstack-infra | 22:24 | |
| *** Douhet has joined #openstack-infra | 22:24 | |
| fungi | pleia2: i went ahead and punted 336722,2 straight to the gate | 22:26 |
| pleia2 | fungi: thanks | 22:26 |
| fungi | so we won't have to wait for resources in the check pipeline | 22:26 |
| *** piet_ has joined #openstack-infra | 22:27 | |
| fungi | since we've sort of blasted our quota with these several zuul restarts in a row now | 22:27 |
| fungi | yay job logs uploading again though! http://logs.openstack.org/22/331522/2/gate/gate-horizon-pep8/9a58e4d/ | 22:27 |
| *** salv-orlando has joined #openstack-infra | 22:31 | |
| jeblair | w00t | 22:32 |
| *** jsavak has quit IRC | 22:32 | |
| *** achanda has quit IRC | 22:32 | |
| fungi | ahh, yep http://tarballs.openstack.org/ being broken has caused some jobs which rely on downloading things from it at runtime to break | 22:34 |
| fungi | but looks like the fix is merging in seconfs | 22:34 |
| fungi | seconds too | 22:34 |
| fungi | here it comes | 22:34 |
| *** tlian has quit IRC | 22:35 | |
| *** jianghuaw has quit IRC | 22:35 | |
| fungi | i guess uploads for our apply jobs are slow | 22:35 |
| openstackgerrit | Merged openstack-infra/system-config: Add Apache 2.4 syntax for static.o.o https://review.openstack.org/336722 | 22:35 |
| fungi | there it is | 22:35 |
| pleia2 | yay | 22:36 |
| *** javeriak has quit IRC | 22:36 | |
| *** jianghuaw has joined #openstack-infra | 22:36 | |
| fungi | machinery exercised | 22:36 |
| *** achanda has joined #openstack-infra | 22:36 | |
| *** asettle has joined #openstack-infra | 22:37 | |
| jeblair | i will fix root's known hosts on puppetmaster | 22:38 |
| jeblair | needs static/zuul entries removed | 22:38 |
| fungi | oh, we still need to do that even with ansible launching? | 22:38 |
| *** rossella_s has quit IRC | 22:39 | |
| fungi | yeah, that explains what i was just starting to look into, which is why there's no puppet mentioned in teh syslog of our new static.o.o | 22:39 |
| jeblair | fungi: i don't think it removes old entries? | 22:39 |
| fungi | agreed. that seems to be the case | 22:39 |
| *** rossella_s has joined #openstack-infra | 22:39 | |
| jeblair | okay, done | 22:39 |
| fungi | i thought it was connecting by ip address alone due to the fact that we can have multiples with the same name and some not in dns | 22:40 |
| fungi | looks like it just connected now, so seems to be the case that there's some complication there with conflicting host keys | 22:42 |
| jeblair | i also am running our single-host adhoc script | 22:42 |
| fungi | oh, is that why i saw puppet fire up? | 22:42 |
| fungi | thought it seemed quick on the draw | 22:42 |
| *** asettle has quit IRC | 22:42 | |
| jeblair | fungi: yeah, i think the run-all cron missed it this time | 22:43 |
| jeblair | so it would have just been the ad-hoc | 22:43 |
| jeblair | though that appears not to have helped | 22:43 |
| fungi | i'm guessing teh ad-hoc play doesn't pull system-config updates? | 22:44 |
| fungi | so it's still working off a git pull from before that merged | 22:44 |
| jeblair | probably not, so it's probably not very useful for this sort of thing | 22:44 |
| *** piet_ has quit IRC | 22:45 | |
| fungi | and the existing run went a little long so we're missing the 22:45 pulse | 22:45 |
| jeblair | okay, i'll run the update puppet play and the adhoc again | 22:45 |
| fungi | looks like it was probably over by about 30 seconds | 22:45 |
| fungi | _now_ /optsystem-config/production is updated | 22:46 |
| fungi | er, /opt/system-config/production | 22:46 |
| jeblair | i ran ansible-playbook -f 10 ${ANSIBLE_PLAYBOOKS}/update_puppet.yaml | 22:46 |
| jeblair | it's still doing something.... | 22:47 |
| jeblair | TASK [command] ***************************************************************** | 22:47 |
| fungi | got it, and that does the needed git pull i guess? | 22:47 |
| jeblair | yep | 22:47 |
| jeblair | TASK [git] ********************************************************************* | 22:47 |
| jeblair | changed: [localhost] | 22:47 |
| jeblair | that already happened | 22:47 |
| jeblair | ah, install_modules is the slow bit | 22:47 |
| jeblair | okay, running the adhoc on static now | 22:48 |
| *** tonytan4ever has quit IRC | 22:48 | |
| bkero | oof, those task names could use some love | 22:48 |
| *** karthikp_ has joined #openstack-infra | 22:49 | |
| fungi | updated apache vhost configs that time | 22:49 |
| fungi | restarted apache | 22:49 |
| fungi | i'm still getting 403 forbidden | 22:50 |
| pleia2 | oh bother | 22:51 |
| *** tonytan4ever has joined #openstack-infra | 22:51 | |
| fungi | and i see why | 22:51 |
| *** piet_ has joined #openstack-infra | 22:52 | |
| *** rbrndt has quit IRC | 22:52 | |
| fungi | looks like /etc/apache2/sites-enabled/50-tarballs.openstack.org.conf only has directory blocks for /srv/static/election and /srv/static/governance under the http vhost | 22:52 |
| fungi | and /srv/static/tarballs under the https vhost | 22:52 |
| fungi | that's exceptionally odd | 22:53 |
| jeblair | well, what about releases? is that simpler? | 22:53 |
| jeblair | oh | 22:54 |
| fungi | same pattern. /srv/static/election and /srv/static/governance under the http vhost | 22:54 |
| jeblair | yeah. | 22:54 |
| jeblair | i get what you're saying. | 22:54 |
| *** karthik__ has joined #openstack-infra | 22:54 | |
| jeblair | this looks either very auto-generated or very copy-pastad. | 22:54 |
| fungi | i'm checking over the template it's using now | 22:55 |
| *** ddieterly is now known as ddieterly[away] | 22:55 | |
| *** harlowja_ has joined #openstack-infra | 22:55 | |
| *** ddieterly[away] has quit IRC | 22:55 | |
| fungi | *sigh* | 22:55 |
| fungi | scope.lookupvar('openstack_project::static::directories') | 22:56 |
| fungi | i wonder if someone got scope-happy and updated these somehow breakagey | 22:56 |
| fungi | directories is not in the openstack_project::static scope it's supposed to be from the scope of teh vhost instantiating the template | 22:57 |
| *** tristanC_ is now known as tristanC | 22:58 | |
| *** karthikp_ has quit IRC | 22:58 | |
| fungi | i think this was somehow a no-op on 2.2 | 22:58 |
| fungi | the same template expansion silliness can be observed on the old server as well | 22:58 |
| fungi | but i think it must have defaulted to permissiveness | 22:58 |
| *** harlowja has quit IRC | 22:59 | |
| jeblair | fungi: just for sanity, i manually edited the releases conf file and made the http section look like the https section | 23:00 |
| fungi | jeblair: thanks. i'm hacking on a patch for the templates now | 23:00 |
| jeblair | (in an attempt to find any other lurking errors) | 23:00 |
| jeblair | and it seems to work | 23:00 |
| *** zzzeek has quit IRC | 23:01 | |
| *** zzzeek has joined #openstack-infra | 23:02 | |
| *** ams__ has quit IRC | 23:04 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Fix variable scopes in static vhost template https://review.openstack.org/336736 | 23:05 |
| fungi | jeblair: pleia2: ^ | 23:05 |
| *** zeih has joined #openstack-infra | 23:05 | |
| fungi | interestingly, the redirect vhost template got it right | 23:05 |
| *** roxanagh_ has quit IRC | 23:06 | |
| pleia2 | ah, scopes | 23:07 |
| fungi | oh, hold up, there's some duplication in there i also missed | 23:07 |
| pleia2 | k | 23:07 |
| openstackgerrit | Jef Mallal proposed openstack-infra/jenkins-job-builder: Add support for retry in multijob plugin https://review.openstack.org/329169 | 23:08 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Fix variable scopes in static vhost template https://review.openstack.org/336736 | 23:09 |
| *** roxanagh_ has joined #openstack-infra | 23:09 | |
| fungi | actually, no duplication, i just re-confused myself when writing the commit message, but now i've spotted a discrepancy in how the puppet class is defining these | 23:09 |
| *** zeih has quit IRC | 23:10 | |
| pleia2 | doh | 23:10 |
| *** zzzeek has quit IRC | 23:11 | |
| fungi | it looks like https://review.openstack.org/292675 actually didn't quite grok that these are shared templates being used across multiple vhosts | 23:11 |
| fungi | so 336736 is not the fix for this | 23:12 |
| fungi | aliases and directories _are_ defined at the class level | 23:12 |
| fungi | so those are getting added to every chost | 23:13 |
| fungi | er, vhost | 23:13 |
| fungi | worse, the if/else in there is causing it to apply to all vhosts because it's defined at the class level | 23:14 |
| *** zzzeek has joined #openstack-infra | 23:14 | |
| fungi | i'm going to push a revert of the elections alias change for the moment, once i resolve the merge conflicts on it | 23:16 |
| *** Apoorva_ has joined #openstack-infra | 23:17 | |
| *** Swami has quit IRC | 23:20 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Revert "Add election alias to governance.openstack.org" https://review.openstack.org/336744 | 23:21 |
| *** Apoorva has quit IRC | 23:21 | |
| fungi | pleia2: jeblair: ^ that's the fix (for now) | 23:21 |
| fungi | hopefully the explanation in the commit message makes sense in context | 23:21 |
| pleia2 | it does, sorry for missing it when I reviewed the initial patch, that static server /o\ | 23:22 |
| *** Apoorva_ has quit IRC | 23:22 | |
| fungi | meh, it's a confoundingly complex situation in there | 23:23 |
| pleia2 | yeah | 23:24 |
| fungi | the governance site should really just get its own vhost template | 23:24 |
| *** javeriak has joined #openstack-infra | 23:24 | |
| fungi | the shared template is intended for sites that do nothing unusual, so if you see a change adding something to that shared template for the benefit of a single site it's probably in error | 23:24 |
| *** yamamoto has quit IRC | 23:24 | |
| fungi | the only "sane" way to extend that with custom values that aren't already implemented in the apache vhost resource is probably to somehow subclass it or something | 23:26 |
| *** tonytan4ever has quit IRC | 23:27 | |
| fungi | or maybe there's a way to add some lookup hash where you can add keys based on some value known from the vhost scope | 23:27 |
| fungi | so the hash is in the class scope but has keys with the site names and then the template checks the hash keyed by the site name | 23:28 |
| fungi | anyway, complicated | 23:28 |
| fungi | custom template for that one site would be a simpler solution | 23:28 |
| *** piet_ has quit IRC | 23:29 | |
| fungi | writing that replacement now | 23:32 |
| *** gyee has quit IRC | 23:33 | |
| *** Sukhdev_ has quit IRC | 23:33 | |
| *** sdague has quit IRC | 23:33 | |
| pleia2 | we're pretty close, should status ok soon | 23:35 |
| *** karthik__ has quit IRC | 23:35 | |
| *** ianychoi has quit IRC | 23:39 | |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Add election alias to governance.openstack.org" https://review.openstack.org/336744 | 23:39 |
| *** davideagnello has joined #openstack-infra | 23:40 | |
| *** davideagnello has quit IRC | 23:40 | |
| *** rfolco has joined #openstack-infra | 23:41 | |
| *** xarses has quit IRC | 23:42 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add election alias to governance.openstack.org https://review.openstack.org/336750 | 23:43 |
| fungi | pleia2: jeblair: ^ i've resubmitted jhesketh's change more safely | 23:43 |
| *** shashank_hegde has quit IRC | 23:44 | |
| pleia2 | fungi: want to add a ref: to the previous one in the commit message? | 23:45 |
| fungi | sure, can do | 23:45 |
| *** piet has joined #openstack-infra | 23:46 | |
| *** ajmiller_ has quit IRC | 23:47 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add election alias to governance.openstack.org https://review.openstack.org/336750 | 23:49 |
| rm_work | why am I seeing telnet:// links on zuul's status page for my changes that are in queue? | 23:50 |
| rm_work | is that normal? O_o | 23:50 |
| rm_work | i'm used to it linking me to jenkins | 23:50 |
| fungi | rm_work: yes, we no longer run jenkins at all anywhere | 23:50 |
| rm_work | ah | 23:50 |
| rm_work | ooooooookay then :P that is unexpected but would explain that | 23:51 |
| fungi | rm_work: the telnet:// links are usable for now (zuul provides a tcp socket stream of the console logs for jobs while they're still running) and eventually someone will provide a websocket implementation to allow you to stream them in your browser as a webpage | 23:51 |
| rm_work | cool, so telnet follows the log | 23:51 |
| rm_work | neat | 23:51 |
| fungi | rm_work: yeah, there were... announcements | 23:51 |
| rm_work | yeah i see | 23:51 |
| rm_work | ah, mailing list? | 23:52 |
| fungi | so hopefully not a surprise to people who follow the mailing lists. yep | 23:52 |
| rm_work | yeah, i got kicked from the mailing list :/ | 23:52 |
| *** karthikp_ has joined #openstack-infra | 23:52 | |
| fungi | oh? what happened? | 23:52 |
| rm_work | it kept saying my email was sending too many bounces!? and i kept verifying my email with the link it would send... and eventually it gave me links with tokens that didn't work, so i gave up | 23:52 |
| rm_work | the email for reference is my @rackspace.com address, lol | 23:53 |
| fungi | rackspace e-mail address by any change? | 23:53 |
| fungi | yeah | 23:53 |
| clarkb | When that happened to me I switched email providers | 23:53 |
| rm_work | i honestly barely read it anyway, so i gave up | 23:53 |
| fungi | rackspace has (had?) some sort of misbehaving "fireeye" mail filter | 23:53 |
| fungi | kept randomly bouncing e-mails for weeks if not months | 23:53 |
| rm_work | there's so many hundreds of emails that i just filter out the ones with my project's name in the subject anyway | 23:54 |
| *** rfolco has quit IRC | 23:54 | |
| rm_work | i think that's literally everyone I know's config... | 23:54 |
| rm_work | lol | 23:54 |
| rm_work | not that it's your fault/problem :P | 23:54 |
| rm_work | I just wouldn't be super surprised that no one sees announcements like that, heh | 23:54 |
| fungi | huh. i actually skim the general, -dev, -infra, -docs, -i18n, -stable-maint, -security and probably others i'm forgetting | 23:55 |
| *** yamamoto has joined #openstack-infra | 23:55 | |
| fungi | then again, i also don't have trouble keeping up with debian-devel and several other debian mailing lists too | 23:55 |
| rm_work | yeah, you would be the first i've talked to that actually does that ^_^ | 23:55 |
| fungi | and the oss-sec ml, and some high-volume cryptography mailing lists | 23:56 |
| rm_work | i think i have 29,000 unread emails in my openstack-dev folder | 23:56 |
| rm_work | that's over... | 23:56 |
| *** thiagop has quit IRC | 23:56 | |
| fungi | oh, and python mailing lists | 23:56 |
| rm_work | i forget how long, but not THAT long T_T | 23:56 |
| rm_work | if i tried to deal with the MLs i'm on I'd go insane | 23:56 |
| fungi | i learned long ago that keeping up with very high volumes of e-mail is only doable if you use mail clients that are engineered to make it convenient to deal with very high volumes of e-mail | 23:57 |
| pleia2 | mutt <3 | 23:57 |
| fungi | 'tis what i use too | 23:57 |
| clarkb | does mutt speak imap yet? seems like last I tried to use it was local mail only | 23:58 |
| fungi | uh, have you ttied it in the past 15 years? | 23:58 |
| fungi | er, tried | 23:58 |
| clarkb | its been a long time | 23:58 |
| clarkb | 9 years maybe | 23:58 |
| pleia2 | yeah, I only use it with imap these days | 23:59 |
| *** javeriak has quit IRC | 23:59 | |
| clarkb | ya 07ish iirc | 23:59 |
| *** salv-orl_ has joined #openstack-infra | 23:59 | |
| pleia2 | fungi: I am losing track already, what needs to land before we can #ok ? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
