| mordred | jeblair: did we success the afs docs? | 00:00 |
|---|---|---|
| jeblair | :( | 00:01 |
| mordred | uhoh | 00:01 |
| jeblair | 2016-09-28 23:57:37,198 DEBUG zuul.NodeWorker.ubuntu-trusty-osic-cloud1-s3500-4614916: Ansible post output: changed: [node -> 127.0.0.1] => {"changed": true, "cmd": "/usr/bin/find /afs/openstack.org/docs/infra/system-config -name .root-marker -printf '%P\n' | /usr/bin/xargs -I{} dirname {} | /usr/bin/sort > /tmp/tmpTRiOUh/staging/tmpYtcDn8/dst-markers", "delta": "0:00:00.006632", "end": "2016-09-28 23:57:37.177470", "rc": 0, "start": ... | 00:02 |
| jeblair | ... "2016-09-28 23:57:37.170838", "stderr": "/usr/bin/find: `/afs/openstack.org/docs/infra/system-config': No such file or directory", "stdout": "", "stdout_lines": [], "warnings": []} | 00:02 |
| *** gongysh has joined #openstack-infra | 00:02 | |
| pabelanger | 2.152 kilo-jobs/hr ! | 00:02 |
| pabelanger | think that is a new record | 00:02 |
| kfox1111 | mordred: yeah, it was a docker user permission thing. | 00:04 |
| *** ijw has joined #openstack-infra | 00:04 | |
| pabelanger | jeblair: did we enable openafs-client on our nodes? | 00:05 |
| jeblair | pabelanger: it's because the subdir doesn't exist yet | 00:06 |
| pabelanger | Ah | 00:06 |
| jeblair | i need that find command to produce an empty file if the directory doesn't exist | 00:06 |
| pabelanger | right, this is on the zuul-launcher | 00:06 |
| pabelanger | brain starting to shutdown for the day | 00:06 |
| kfox1111 | thats fun.... | 00:07 |
| kfox1111 | getting this out of zuul: telnet://[2001:4800:1ae1:18:f816:3eff:fe36:af4a]:19885 | 00:07 |
| kfox1111 | ipv6. | 00:07 |
| jeblair | kfox1111: welcome to the future! | 00:07 |
| *** dtardivel has quit IRC | 00:07 | |
| jeblair | circa 1994 | 00:08 |
| *** adrian_otto has joined #openstack-infra | 00:09 | |
| *** zhurong has quit IRC | 00:10 | |
| clarkb | jeblair: and now jobs use tls too | 00:11 |
| clarkb | its even more future | 00:11 |
| *** mriedem has joined #openstack-infra | 00:12 | |
| *** kaisers3 has joined #openstack-infra | 00:14 | |
| *** kaisers2 has quit IRC | 00:15 | |
| greghaynes | \O/ | 00:15 |
| *** kaisers2 has joined #openstack-infra | 00:15 | |
| *** kzaitsev_mb has joined #openstack-infra | 00:15 | |
| *** kaisers1 has quit IRC | 00:16 | |
| jpmaxman | fungi yeah it may be as simplea s a misconfig by someone trying to use openstack ID | 00:19 |
| *** Julien-zte has joined #openstack-infra | 00:19 | |
| jpmaxman | I think removing the apache cache of the error | 00:19 |
| jpmaxman | will help | 00:19 |
| jpmaxman | and sorry was awk | 00:20 |
| jpmaxman | *afk | 00:20 |
| jpmaxman | We are keeping a close eye on it | 00:20 |
| jpmaxman | thanks for the approval of access that will help :) | 00:20 |
| pabelanger | greghaynes: mind adding https://review.openstack.org/#/c/378985 to your review list for tomorrow | 00:21 |
| *** kzaitsev_mb has quit IRC | 00:22 | |
| greghaynes | pabelanger: ah, sure | 00:22 |
| pabelanger | danke | 00:22 |
| pabelanger | I've confirmed this work better with this approach | 00:22 |
| *** kzaitsev_mb has joined #openstack-infra | 00:23 | |
| *** tonytan4ever has joined #openstack-infra | 00:24 | |
| *** smarcet has joined #openstack-infra | 00:25 | |
| *** sdake_ has quit IRC | 00:25 | |
| *** adrian_otto has quit IRC | 00:27 | |
| *** jamielennox is now known as jamielennox|away | 00:29 | |
| *** tonytan4ever has quit IRC | 00:30 | |
| *** gongysh has quit IRC | 00:32 | |
| *** thcipriani is now known as thcipriani|afk | 00:33 | |
| *** jamielennox|away is now known as jamielennox | 00:33 | |
| *** xarses has joined #openstack-infra | 00:33 | |
| *** Goneri has joined #openstack-infra | 00:34 | |
| *** ijw_ has joined #openstack-infra | 00:36 | |
| *** ijw has quit IRC | 00:39 | |
| *** shu-mutou has joined #openstack-infra | 00:42 | |
| *** tiswanso has joined #openstack-infra | 00:42 | |
| *** lezbar has quit IRC | 00:43 | |
| *** lezbar has joined #openstack-infra | 00:44 | |
| *** thorst has joined #openstack-infra | 00:44 | |
| *** roxanaghe has joined #openstack-infra | 00:51 | |
| *** gouthamr has joined #openstack-infra | 00:52 | |
| *** Goneri has quit IRC | 00:52 | |
| *** sdake has joined #openstack-infra | 00:52 | |
| *** EricGonczer_ has joined #openstack-infra | 00:53 | |
| *** roxanaghe has quit IRC | 00:53 | |
| *** nadya has joined #openstack-infra | 00:53 | |
| *** amitgandhinz has joined #openstack-infra | 00:54 | |
| *** gouthamr has quit IRC | 00:58 | |
| *** nadya has quit IRC | 00:59 | |
| *** vinaypotluri has quit IRC | 01:02 | |
| *** pahuang__ has quit IRC | 01:04 | |
| *** kzaitsev_mb has quit IRC | 01:06 | |
| *** kaisers has quit IRC | 01:11 | |
| *** yanyanhu_ has joined #openstack-infra | 01:13 | |
| *** markvoelker has joined #openstack-infra | 01:13 | |
| *** Guest92 has joined #openstack-infra | 01:14 | |
| *** hongbin has joined #openstack-infra | 01:15 | |
| *** Guest92_ has joined #openstack-infra | 01:15 | |
| *** mtanino has quit IRC | 01:16 | |
| *** gouthamr has joined #openstack-infra | 01:18 | |
| *** zhurong has joined #openstack-infra | 01:19 | |
| *** Guest92 has quit IRC | 01:19 | |
| *** ijw_ has quit IRC | 01:20 | |
| *** rkukura_ has joined #openstack-infra | 01:21 | |
| *** pahuang__ has joined #openstack-infra | 01:21 | |
| *** jamesdenton has joined #openstack-infra | 01:21 | |
| *** rkukura has quit IRC | 01:23 | |
| *** rkukura_ is now known as rkukura | 01:23 | |
| *** Guest92_ has quit IRC | 01:25 | |
| *** Jeffrey4l has joined #openstack-infra | 01:27 | |
| *** amitgandhinz has quit IRC | 01:29 | |
| *** mriedem has quit IRC | 01:29 | |
| *** gildub has joined #openstack-infra | 01:30 | |
| *** Apoorva_ has joined #openstack-infra | 01:39 | |
| *** ijw has joined #openstack-infra | 01:40 | |
| *** ddieterly has joined #openstack-infra | 01:41 | |
| *** jamielennox is now known as jamielennox|away | 01:42 | |
| *** Apoorva has quit IRC | 01:43 | |
| *** caowei has joined #openstack-infra | 01:44 | |
| *** Apoorva_ has quit IRC | 01:44 | |
| *** pahuang__ has quit IRC | 01:45 | |
| *** esikachev has joined #openstack-infra | 01:46 | |
| *** armax has joined #openstack-infra | 01:46 | |
| *** esikachev has quit IRC | 01:50 | |
| *** woodster_ has joined #openstack-infra | 01:51 | |
| *** tuanla has joined #openstack-infra | 01:52 | |
| *** thorst has quit IRC | 01:53 | |
| *** tuanla has quit IRC | 01:53 | |
| *** tuanluong has joined #openstack-infra | 01:54 | |
| *** pahuang__ has joined #openstack-infra | 01:57 | |
| *** ijw has quit IRC | 02:01 | |
| *** salv-orlando has joined #openstack-infra | 02:01 | |
| *** gouthamr has quit IRC | 02:05 | |
| *** salv-orlando has quit IRC | 02:05 | |
| *** tphummel has quit IRC | 02:13 | |
| *** tuannguyen has joined #openstack-infra | 02:13 | |
| *** pahuang__ has quit IRC | 02:15 | |
| *** amitgandhinz has joined #openstack-infra | 02:25 | |
| *** pahuang__ has joined #openstack-infra | 02:28 | |
| *** armax has quit IRC | 02:31 | |
| *** lezbar has quit IRC | 02:37 | |
| *** lezbar has joined #openstack-infra | 02:38 | |
| *** EricGonczer_ has quit IRC | 02:38 | |
| *** winggundamth has quit IRC | 02:40 | |
| *** tuannguyen has quit IRC | 02:43 | |
| *** yuanying_ has quit IRC | 02:50 | |
| *** sdake has quit IRC | 02:52 | |
| *** nadya has joined #openstack-infra | 02:54 | |
| *** tuannguyen has joined #openstack-infra | 02:57 | |
| *** EricGonczer_ has joined #openstack-infra | 02:59 | |
| *** nadya has quit IRC | 02:59 | |
| *** yamamoto_ has joined #openstack-infra | 02:59 | |
| *** amitgandhinz has quit IRC | 02:59 | |
| *** apetrich has quit IRC | 03:02 | |
| *** david-lyle has quit IRC | 03:04 | |
| *** yuanying has joined #openstack-infra | 03:05 | |
| *** apetrich has joined #openstack-infra | 03:06 | |
| openstackgerrit | zhangyanxian proposed openstack-infra/project-config: Fix typo in layout.yaml https://review.openstack.org/379095 | 03:07 |
| *** smarcet1981 has joined #openstack-infra | 03:08 | |
| openstackgerrit | zhangyanxian proposed openstack-infra/project-config: Fix typo in layout.yaml https://review.openstack.org/379095 | 03:08 |
| *** jamielennox|away is now known as jamielennox | 03:11 | |
| *** EricGonczer_ has quit IRC | 03:12 | |
| *** ijw has joined #openstack-infra | 03:15 | |
| *** sdake has joined #openstack-infra | 03:16 | |
| smarcet1981 | fungi: yes i think that the apache2 instance was configured wrongly | 03:18 |
| smarcet1981 | mainly the 503 error that i am seeing are from resource server | 03:18 |
| smarcet1981 | that uses the idp to validate the access tokens | 03:18 |
| smarcet1981 | and yes the apache errors are the "AH01075: Error dispatching request" and "AH01067: Failed to read FastCGI header" | 03:19 |
| *** sdake has quit IRC | 03:19 | |
| smarcet1981 | fungi: at any rate i took a measure of the avg memory used by php fpm process on openstackid and its 25/30 mb | 03:20 |
| *** ijw has quit IRC | 03:20 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: OFAgent CI should be run for only liberty and mitaka https://review.openstack.org/379103 | 03:21 |
| smarcet1981 | so i guess that is fair to said that we could support around 300 process at once, also i noticed that the pm.max_requests was set to high ( 5000) | 03:22 |
| *** ijw has joined #openstack-infra | 03:22 | |
| *** armax has joined #openstack-infra | 03:22 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update localrc for ironic-multinode case https://review.openstack.org/352790 | 03:27 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Setup ssh-key on subnodes for Ironic https://review.openstack.org/364830 | 03:27 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update ENABLED_SERVICE on subnode with ironic https://review.openstack.org/368611 | 03:27 |
| *** flepied has joined #openstack-infra | 03:31 | |
| *** sdake has joined #openstack-infra | 03:31 | |
| armax | AJaeger: it looks like your change (https://review.openstack.org/#/c/377738) didn’t quite work as expected | 03:33 |
| *** links has joined #openstack-infra | 03:33 | |
| *** yuanying has quit IRC | 03:33 | |
| *** ddieterly has quit IRC | 03:33 | |
| armax | AJaeger: http://logs.openstack.org/periodic/periodic-neutron-py35-with-neutron-lib-master/a4bf407/console.html#_2016-09-28_23_47_39_064749 | 03:33 |
| armax | I can’t spot the obvious though | 03:33 |
| *** yuanying has joined #openstack-infra | 03:34 | |
| *** armax has quit IRC | 03:34 | |
| *** tuannguyen has quit IRC | 03:38 | |
| *** vikrant has joined #openstack-infra | 03:41 | |
| *** PalTale has quit IRC | 03:44 | |
| *** kaisers3 has quit IRC | 03:45 | |
| *** watersoul_ has joined #openstack-infra | 03:45 | |
| *** mrunge has quit IRC | 03:45 | |
| *** Julien-z_ has joined #openstack-infra | 03:46 | |
| *** claudiub|2 has joined #openstack-infra | 03:46 | |
| *** fernnest__ has joined #openstack-infra | 03:46 | |
| *** rcarrillocruz has quit IRC | 03:46 | |
| *** Douhet has quit IRC | 03:46 | |
| *** Julien-zte has quit IRC | 03:47 | |
| *** fernnest_ has quit IRC | 03:47 | |
| *** pfallenop has quit IRC | 03:47 | |
| *** pcaruana has quit IRC | 03:47 | |
| *** ramishra has quit IRC | 03:47 | |
| *** gildub has quit IRC | 03:47 | |
| *** claudiub has quit IRC | 03:47 | |
| *** watersoul has quit IRC | 03:47 | |
| *** dtantsur has quit IRC | 03:47 | |
| *** tomhambleton has quit IRC | 03:48 | |
| *** d0ugal has quit IRC | 03:48 | |
| *** gildub has joined #openstack-infra | 03:48 | |
| *** eeiden has quit IRC | 03:48 | |
| *** fkautz_ has quit IRC | 03:48 | |
| *** kaisers2 has quit IRC | 03:49 | |
| *** yuanying has quit IRC | 03:49 | |
| *** Kiall has quit IRC | 03:49 | |
| *** M-docaedo_vector has quit IRC | 03:50 | |
| *** kaisers has joined #openstack-infra | 03:50 | |
| *** tomhambleton has joined #openstack-infra | 03:50 | |
| *** xgerman has quit IRC | 03:50 | |
| *** fkautz_ has joined #openstack-infra | 03:50 | |
| *** Kiall has joined #openstack-infra | 03:50 | |
| *** eeiden has joined #openstack-infra | 03:52 | |
| *** yuanying has joined #openstack-infra | 03:52 | |
| *** xgerman has joined #openstack-infra | 03:52 | |
| *** oomichi has quit IRC | 03:52 | |
| *** petems has quit IRC | 03:52 | |
| *** smarcet1981 has quit IRC | 03:52 | |
| *** fernnest has quit IRC | 03:52 | |
| *** hongbin has quit IRC | 03:53 | |
| *** petems has joined #openstack-infra | 03:53 | |
| *** dtantsur has joined #openstack-infra | 03:54 | |
| *** doffm has quit IRC | 03:54 | |
| *** darvon_ has joined #openstack-infra | 03:54 | |
| *** oomichi has joined #openstack-infra | 03:54 | |
| *** lezbar has quit IRC | 03:54 | |
| *** mrunge has joined #openstack-infra | 03:54 | |
| *** lezbar has joined #openstack-infra | 03:55 | |
| *** doffm has joined #openstack-infra | 03:55 | |
| *** fernnest has joined #openstack-infra | 03:55 | |
| *** amitgandhinz has joined #openstack-infra | 03:55 | |
| *** rcarrillocruz has joined #openstack-infra | 03:55 | |
| *** ramishra has joined #openstack-infra | 03:55 | |
| *** tiswanso has quit IRC | 03:56 | |
| *** darvon has quit IRC | 03:56 | |
| *** d0ugal has joined #openstack-infra | 03:56 | |
| *** pcaruana has joined #openstack-infra | 03:59 | |
| *** pfallenop has joined #openstack-infra | 03:59 | |
| *** darvon_ is now known as darvon | 04:00 | |
| *** lifeless has quit IRC | 04:02 | |
| *** salv-orlando has joined #openstack-infra | 04:03 | |
| *** kaisers1 has joined #openstack-infra | 04:03 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update localrc for ironic-multinode case https://review.openstack.org/352790 | 04:04 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Setup ssh-key on subnodes for Ironic https://review.openstack.org/364830 | 04:04 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update ENABLED_SERVICE on subnode with ironic https://review.openstack.org/368611 | 04:04 |
| *** lifeless has joined #openstack-infra | 04:05 | |
| *** ramishra has quit IRC | 04:06 | |
| *** salv-orlando has quit IRC | 04:07 | |
| *** megm_ has joined #openstack-infra | 04:08 | |
| *** megm has quit IRC | 04:08 | |
| *** dkehn_ has quit IRC | 04:12 | |
| *** PalTale has joined #openstack-infra | 04:12 | |
| *** Rockyg has quit IRC | 04:19 | |
| *** ramishra has joined #openstack-infra | 04:24 | |
| *** ijw has quit IRC | 04:25 | |
| *** amitgandhinz has quit IRC | 04:29 | |
| *** marst has quit IRC | 04:37 | |
| *** tongli has quit IRC | 04:39 | |
| *** coolsvap has joined #openstack-infra | 04:41 | |
| *** salv-orlando has joined #openstack-infra | 04:42 | |
| *** salv-orlando has quit IRC | 04:42 | |
| *** salv-orlando has joined #openstack-infra | 04:43 | |
| *** flepied has quit IRC | 04:46 | |
| *** gongysh has joined #openstack-infra | 04:46 | |
| *** psachin has joined #openstack-infra | 04:52 | |
| *** nadya has joined #openstack-infra | 04:55 | |
| *** amotoki has joined #openstack-infra | 04:57 | |
| frickler | armax: AJaeger: I'm guessing that the job needs to run on a new image built that includes your patched run-tox-with-neutron-lib-master.sh, so that error should disappear on the next run | 04:58 |
| openstackgerrit | Merged openstack-infra/project-config: Rename py27-with-upper-constraints to py27-check-uc https://review.openstack.org/378732 | 04:59 |
| *** sdake has quit IRC | 05:00 | |
| *** nadya has quit IRC | 05:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix typo in layout.yaml https://review.openstack.org/379095 | 05:01 |
| *** amotoki has quit IRC | 05:02 | |
| *** yamamoto_ has quit IRC | 05:04 | |
| *** yamahata has joined #openstack-infra | 05:05 | |
| *** Guest92 has joined #openstack-infra | 05:15 | |
| *** amotoki has joined #openstack-infra | 05:17 | |
| *** vsaienko has joined #openstack-infra | 05:24 | |
| *** amitgandhinz has joined #openstack-infra | 05:25 | |
| *** narayrak has joined #openstack-infra | 05:28 | |
| openstackgerrit | Ondřej Nový proposed openstack-infra/project-config: Use git.openstack.org not Github for Openstack projects https://review.openstack.org/375853 | 05:32 |
| *** vsaienko has quit IRC | 05:32 | |
| *** Guest92 has quit IRC | 05:38 | |
| openstackgerrit | Ondřej Nový proposed openstack-infra/project-config: Use https not git protocol for upstream repos https://review.openstack.org/379153 | 05:39 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Revert "Make Horizon integration tests non-voting" https://review.openstack.org/379154 | 05:41 |
| *** e0ne has joined #openstack-infra | 05:41 | |
| *** adriant has quit IRC | 05:43 | |
| *** pgadiya has joined #openstack-infra | 05:44 | |
| *** yamamoto_ has joined #openstack-infra | 05:46 | |
| *** tosky has joined #openstack-infra | 05:46 | |
| *** esikachev has joined #openstack-infra | 05:47 | |
| *** gongysh has quit IRC | 05:48 | |
| *** esikachev has quit IRC | 05:51 | |
| *** nadya has joined #openstack-infra | 05:54 | |
| *** nadya has quit IRC | 05:57 | |
| *** amitgandhinz has quit IRC | 05:59 | |
| *** hashar has joined #openstack-infra | 05:59 | |
| *** woodster_ has quit IRC | 06:00 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Add neutron-lib-master periodic job for networking-midonet https://review.openstack.org/377292 | 06:02 |
| *** pahuang__ has quit IRC | 06:05 | |
| *** yuval has joined #openstack-infra | 06:09 | |
| *** nherciu has joined #openstack-infra | 06:11 | |
| *** nadya has joined #openstack-infra | 06:12 | |
| *** M-docaedo_vector has joined #openstack-infra | 06:16 | |
| *** andreas_s has joined #openstack-infra | 06:21 | |
| *** rcernin has joined #openstack-infra | 06:22 | |
| *** rlu has joined #openstack-infra | 06:24 | |
| *** Guest92 has joined #openstack-infra | 06:28 | |
| openstackgerrit | linzi proposed openstack-infra/project-config: Add networking-peregrine project. https://review.openstack.org/376167 | 06:31 |
| *** nadya has quit IRC | 06:33 | |
| *** aeng__ has quit IRC | 06:37 | |
| *** Guest92 has quit IRC | 06:41 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 06:46 |
| *** lezbar has quit IRC | 06:47 | |
| *** lezbar has joined #openstack-infra | 06:48 | |
| *** mhickey has joined #openstack-infra | 06:48 | |
| openstackgerrit | Dong Ma proposed openstack-infra/jenkins-job-builder: Update logparser plugin https://review.openstack.org/379194 | 06:51 |
| *** rwsu has joined #openstack-infra | 06:52 | |
| *** dtardivel has joined #openstack-infra | 06:53 | |
| *** drifterza has joined #openstack-infra | 06:55 | |
| *** kaisers_ has joined #openstack-infra | 06:55 | |
| *** amitgandhinz has joined #openstack-infra | 06:56 | |
| *** jaosorior has joined #openstack-infra | 06:58 | |
| *** winggundamth has joined #openstack-infra | 06:59 | |
| *** martinkopec has joined #openstack-infra | 07:04 | |
| *** amoralej|off is now known as amoralej | 07:06 | |
| *** ssancheztrujillo has quit IRC | 07:10 | |
| *** nzoueidi has joined #openstack-infra | 07:12 | |
| *** camunoz has quit IRC | 07:25 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 07:25 |
| *** matrohon has joined #openstack-infra | 07:26 | |
| *** hichihara has quit IRC | 07:27 | |
| *** sandanar has joined #openstack-infra | 07:28 | |
| *** jlanoux has joined #openstack-infra | 07:29 | |
| *** amitgandhinz has quit IRC | 07:30 | |
| *** bexelbie has quit IRC | 07:35 | |
| *** bexelbie has joined #openstack-infra | 07:35 | |
| *** salv-orlando has quit IRC | 07:37 | |
| *** OferBarber has joined #openstack-infra | 07:38 | |
| *** OferBarber_ has quit IRC | 07:41 | |
| pradiprwt | rcarrillocruz, can you please review https://review.openstack.org/#/c/377639 | 07:42 |
| *** oanson has joined #openstack-infra | 07:43 | |
| *** vsaienko has joined #openstack-infra | 07:45 | |
| *** Naeil has joined #openstack-infra | 07:45 | |
| *** eliqiao has quit IRC | 07:46 | |
| *** nzoueidi has quit IRC | 07:46 | |
| *** jpich has joined #openstack-infra | 07:48 | |
| *** salv-orlando has joined #openstack-infra | 07:50 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:53 | |
| *** Ahharu has quit IRC | 07:56 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| openstackgerrit | Merged openstack-infra/groups: Add Flag module v3 commons refactor patch https://review.openstack.org/378596 | 08:05 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Preparing multinode networking for Ironic https://review.openstack.org/335981 | 08:06 |
| *** watersoul_ has quit IRC | 08:11 | |
| *** esikachev has joined #openstack-infra | 08:12 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE TESTING https://review.openstack.org/316436 | 08:12 |
| *** pilgrimstack has quit IRC | 08:13 | |
| *** yaume has joined #openstack-infra | 08:14 | |
| *** sflanigan_ has quit IRC | 08:15 | |
| *** pilgrimstack has joined #openstack-infra | 08:17 | |
| *** vsaienko has quit IRC | 08:18 | |
| *** vsaienko has joined #openstack-infra | 08:18 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Flag module v3 patch applied to commons modules https://review.openstack.org/379263 | 08:19 |
| *** gildub has quit IRC | 08:21 | |
| *** amitgandhinz has joined #openstack-infra | 08:26 | |
| *** strigazi_AFK is now known as strigazi | 08:29 | |
| *** vsaienko has quit IRC | 08:29 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:29 | |
| strigazi | Hi AJaeger, we get a lot of these timeouts http://logs.openstack.org/07/379207/1/check/gate-functional-dsvm-magnum-mesos/59bfa03/logs/devstacklog.txt.gz#_2016-09-29_07_22_39_132 | 08:29 |
| *** dragonmaster has joined #openstack-infra | 08:32 | |
| pradiprwt | Hi yolanda, can you please review https://review.openstack.org/#/c/377639 | 08:36 |
| *** vsaienko has joined #openstack-infra | 08:37 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| wznoinsk | morning | 08:39 |
| dragonmaster | morning | 08:40 |
| yolanda | good morning | 08:44 |
| wznoinsk | anyone advanced in nodepool enough could give me a hint maybe... http://paste.openstack.org/show/583437/ <- nodepool tries to ssh connect to the newly created node, while sshd is up on the node it gives 'still booting' kind of message and nodepool simply fails and deletes the node (before the nodes is completely up) | 08:44 |
| *** Naeil has quit IRC | 08:44 | |
| AJaeger | strigazi: let's tell cloudnull and infra-root about those. http://logs.openstack.org/07/379207/1/check/gate-functional-dsvm-magnum-mesos/59bfa03/console.html shows it's in OSIC and they are investigating | 08:46 |
| *** gildub has joined #openstack-infra | 08:49 | |
| *** yamahata has quit IRC | 08:49 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 08:50 | |
| AJaeger | strigazi: regarding https://review.openstack.org/#/c/352905, see my comment on https://review.openstack.org/#/c/377415/1 - you should not touch the translation files at all. The bots take care of it eventually... | 08:51 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Add lost job to master and newton branches pipelines https://review.openstack.org/379282 | 08:53 |
| *** matrohon has quit IRC | 08:54 | |
| *** vsaienko has quit IRC | 08:54 | |
| *** kzaitsev_mb has quit IRC | 08:55 | |
| *** berendt has joined #openstack-infra | 08:56 | |
| strigazi | AJaeger, most of our jobs break with this timeout, ok I'll wait | 08:57 |
| *** jpena|off is now known as jpena | 08:57 | |
| *** Naeil has joined #openstack-infra | 09:00 | |
| *** ihrachys has joined #openstack-infra | 09:01 | |
| *** amitgandhinz has quit IRC | 09:01 | |
| *** narayrak has quit IRC | 09:01 | |
| *** narayrak has joined #openstack-infra | 09:02 | |
| *** jaosorior is now known as jaosorior_lunch | 09:05 | |
| *** jed56 has quit IRC | 09:05 | |
| *** eranrom_ is now known as eranrom | 09:06 | |
| zigo | AJaeger: This patch should go through the gating queue, but it isn't, any idea why? https://review.openstack.org/372527 | 09:06 |
| *** eranrom has quit IRC | 09:07 | |
| *** eranrom has joined #openstack-infra | 09:07 | |
| AJaeger | strigazi: best to come back US morning and ping again - but I hope an infra-root will read and react. | 09:09 |
| AJaeger | zigo: are you using gerrty or the browser? | 09:09 |
| zigo | AJaeger: Browser, and checked zuul.o.o | 09:09 |
| *** zz_dimtruck has quit IRC | 09:10 | |
| *** wendar has quit IRC | 09:10 | |
| AJaeger | zigo: open the review again, go to the upper right corner and check "Related changes" | 09:10 |
| eranrom | Hi. I used to be a channel operator on #openstack-storlets I assume this violate some policy and I am no longer an op there. I do need to be able to change the topic though. Any help will be greatly appreciated. | 09:10 |
| AJaeger | Your change is stacked on top of another one. That one needs to merge first | 09:10 |
| zigo | Oh ... | 09:10 |
| zigo | Got it. | 09:10 |
| zigo | AJaeger: I'll ask the contributor to re-do his CR since other similar stuff got merged already, thanks ! | 09:11 |
| AJaeger | wait - that one merged, didn't it? | 09:11 |
| AJaeger | then rebase | 09:11 |
| zigo | It was abandonned because I used a mergeCommit to push the .gitreview fix. | 09:11 |
| zigo | (needed because of broken unit tests in the old version) | 09:12 |
| AJaeger | Ah, so rebase the change properly... | 09:12 |
| *** watersoul has joined #openstack-infra | 09:12 | |
| AJaeger | eranrom: no, that should not violate policies. | 09:13 |
| AJaeger | eranrom: run "/msg chanserv access #openstack-storlets list" and ask on of the operators to add you. | 09:14 |
| AJaeger | eranrom: see http://docs.openstack.org/infra/system-config/irc.html | 09:14 |
| eranrom | AJaeger: ok, thanks! will do. | 09:14 |
| eranrom | pabelanger: yolanda: nibalizer: I randomly picked you from the ops list of #openstack-storlets. Can you please help adding me to the channel as an op? Thanks very much! | 09:18 |
| *** esikachev has quit IRC | 09:18 | |
| *** sambetts|afk is now known as sambetts | 09:18 | |
| *** electrofelix has joined #openstack-infra | 09:19 | |
| AJaeger | eranrom: why not ask hrou ? | 09:19 |
| eranrom | AJaeger: He is no longer active, doing other things for few month now... | 09:20 |
| *** mhickey has quit IRC | 09:26 | |
| AJaeger | eranrom: That's a bad handover ;( Ok, understood | 09:26 |
| eranrom | AJaeger: Agree. Perhaps I should ask to remove him while at it :-) | 09:27 |
| *** esikachev has joined #openstack-infra | 09:27 | |
| *** esikachev has quit IRC | 09:32 | |
| *** eliqiao has joined #openstack-infra | 09:35 | |
| *** sshnaidm is now known as sshnaidm|afk | 09:37 | |
| pradiprwt | Thanks AJaeger for giving your vote on patch https://review.openstack.org/#/c/377639, hoping others core reviwers vote , do you know any core reviewer who is active now? | 09:40 |
| *** tnovacik has joined #openstack-infra | 09:46 | |
| *** sdake has joined #openstack-infra | 09:47 | |
| *** Julien-z_ has quit IRC | 09:47 | |
| *** esikachev has joined #openstack-infra | 09:47 | |
| *** nadya has joined #openstack-infra | 09:49 | |
| *** john-davidge has joined #openstack-infra | 09:53 | |
| *** amitgandhinz has joined #openstack-infra | 09:57 | |
| *** nadya has quit IRC | 10:02 | |
| *** zhurong has quit IRC | 10:03 | |
| *** amotoki has quit IRC | 10:06 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/tripleo-ci: Add Mistral to the CI status https://review.openstack.org/379335 | 10:08 |
| *** yanyanhu_ has quit IRC | 10:14 | |
| *** e0ne has quit IRC | 10:16 | |
| openstackgerrit | Merged openstack-infra/project-config: Add networking-peregrine project. https://review.openstack.org/376167 | 10:16 |
| openstackgerrit | Merged openstack-infra/project-config: Added Swift related Debian packaging projects https://review.openstack.org/375851 | 10:19 |
| openstackgerrit | Merged openstack-infra/project-config: Added forgoten openstack/deb-python-castellan https://review.openstack.org/375854 | 10:19 |
| openstackgerrit | Merged openstack-infra/project-config: Use git.openstack.org not Github for Openstack projects https://review.openstack.org/375853 | 10:21 |
| openstackgerrit | Merged openstack-infra/project-config: Use https not git protocol for upstream repos https://review.openstack.org/379153 | 10:22 |
| *** kzaitsev_mb has joined #openstack-infra | 10:22 | |
| openstackgerrit | Merged openstack-infra/system-config: Mirror centos virt mirror to AFS https://review.openstack.org/378978 | 10:23 |
| openstackgerrit | Michal Dulko proposed openstack-infra/project-config: Make Cinder multinode grenade job voting https://review.openstack.org/379346 | 10:24 |
| openstackgerrit | Merged openstack-infra/project-config: zuul/layout/puppet: run scenario003 for puppet-designate https://review.openstack.org/378997 | 10:25 |
| wznoinsk | hi all | 10:30 |
| *** amitgandhinz has quit IRC | 10:31 | |
| openstackgerrit | Alexander Evseev proposed openstack-infra/project-config: Use arrays in job parameter definitions https://review.openstack.org/379356 | 10:33 |
| wznoinsk | I have an issue with nodepool's ssh connecting to newly provisioned baremetal nodes too early and failing straight away | 10:34 |
| Zara | hm, there's a story in storyboard-dev that looks like it might be a real story rather than test data; might be worth checking with the author if anyone knows how? https://storyboard-dev.openstack.org/#!/story/46 | 10:34 |
| wznoinsk | would anyone have any ideas/suggestions on what could be a fix/workaround for such situation? | 10:34 |
| *** gildub has quit IRC | 10:35 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/project-config: selective caching of git repos https://review.openstack.org/322487 | 10:36 |
| *** ihrachys has quit IRC | 10:37 | |
| *** nadya has joined #openstack-infra | 10:37 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/project-config: selective caching of git repos https://review.openstack.org/322487 | 10:38 |
| *** yamamoto_ has quit IRC | 10:40 | |
| *** salv-orlando has quit IRC | 10:40 | |
| *** salv-orlando has joined #openstack-infra | 10:41 | |
| frickler | clarkb: maybe the issues with the multinode jobs are related to neutron setting up a route for 10/8, this is in the worlddump "10.0.0.0/8 via 172.24.5.5 dev br-ex | 10:43 |
| frickler | clarkb: at the same time ens3 has a local address from 10/8. while the connected route wins for outgoing packets, I've seen issues with arp replies getting blocked in similar situations | 10:44 |
| *** kzaitsev_mb has quit IRC | 10:47 | |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Add access to set tags for k8s-docker-suite-app-murano project https://review.openstack.org/379362 | 10:47 |
| *** kzaitsev_mb has joined #openstack-infra | 10:51 | |
| *** salv-orlando has quit IRC | 10:54 | |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Add access to set tags for k8s-docker-suite-app-murano project https://review.openstack.org/379362 | 10:56 |
| AJaeger | yolanda: could you review https://review.openstack.org/376211 and https://review.openstack.org/377968, please? | 10:57 |
| yolanda | hi sure | 10:58 |
| AJaeger | thanks | 10:58 |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Add access to set tags for k8s-docker-suite-app-murano project https://review.openstack.org/379362 | 10:59 |
| *** jkilpatr has quit IRC | 11:00 | |
| *** caowei has quit IRC | 11:02 | |
| *** _degorenko is now known as degorenko | 11:02 | |
| *** nherciu has quit IRC | 11:02 | |
| *** jaosorior_lunch is now known as jaosorior | 11:02 | |
| *** nherciu has joined #openstack-infra | 11:03 | |
| Zara | story question #2-- there's a story here about gertty behaviour on macOS; afaict from the gerrty installation instructions, macOS isn't supported, so I was going to comment to that effect, but really I should make sure. https://storyboard.openstack.org/#!/story/2000732 | 11:05 |
| *** dprince has joined #openstack-infra | 11:06 | |
| *** rhallisey has joined #openstack-infra | 11:07 | |
| AJaeger | project-config cores, FYI: I -2 a large change that I don't agree with as is. I'd like to have a discussion for such kind of cleanup. There are some nice ideas but this goes too far IMHO. I'm willing to lift my -2 after a discussion happened. For details, see https://review.openstack.org/#/c/379356/ | 11:10 |
| *** OferBarber_ has joined #openstack-infra | 11:13 | |
| *** mat128|afk is now known as mat128 | 11:14 | |
| onovy | AJaeger: nice catch, sry, fixing | 11:16 |
| *** OferBarber has quit IRC | 11:16 | |
| Zara | wow | 11:17 |
| *** vsaienko has joined #openstack-infra | 11:18 | |
| *** shardy has joined #openstack-infra | 11:18 | |
| openstackgerrit | Ondřej Nový proposed openstack-infra/project-config: Added .git suffix for github.com repos https://review.openstack.org/379272 | 11:18 |
| *** yamamoto has joined #openstack-infra | 11:18 | |
| *** ccamacho is now known as ccamacho|lunch | 11:18 | |
| *** OferBarber has joined #openstack-infra | 11:18 | |
| *** yamamoto_ has joined #openstack-infra | 11:18 | |
| *** yamamoto has quit IRC | 11:18 | |
| *** openstackgerrit has quit IRC | 11:19 | |
| *** openstackgerrit has joined #openstack-infra | 11:19 | |
| *** rtheis has joined #openstack-infra | 11:21 | |
| *** OferBarber_ has quit IRC | 11:21 | |
| openstackgerrit | Ondřej Nový proposed openstack-infra/project-config: Added .git suffix for github.com repos https://review.openstack.org/379272 | 11:25 |
| *** amitgandhinz has joined #openstack-infra | 11:28 | |
| *** dtantsur has quit IRC | 11:29 | |
| *** lucas-afk is now known as lucasagomes | 11:33 | |
| *** jkilpatr has joined #openstack-infra | 11:35 | |
| *** apetrich has quit IRC | 11:39 | |
| *** ldnunes has joined #openstack-infra | 11:39 | |
| *** apetrich has joined #openstack-infra | 11:41 | |
| *** thorst has joined #openstack-infra | 11:41 | |
| *** jpena is now known as jpena|lunch | 11:43 | |
| *** EricGonczer_ has joined #openstack-infra | 11:44 | |
| *** furlongm has quit IRC | 11:46 | |
| *** furlongm has joined #openstack-infra | 11:46 | |
| AJaeger | onovy: thanks for these cleanups! | 11:47 |
| onovy | you are welcome :) | 11:47 |
| *** furlongm has quit IRC | 11:50 | |
| *** furlongm has joined #openstack-infra | 11:50 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:50 | |
| *** rodrigods has quit IRC | 11:51 | |
| *** rodrigods has joined #openstack-infra | 11:51 | |
| *** rohanion has joined #openstack-infra | 11:52 | |
| *** EricGonczer_ has quit IRC | 11:52 | |
| rohanion | hi everyone | 11:52 |
| rohanion | I have a strange problem with testing on a tempest devstack vm | 11:53 |
| rohanion | http://logs.openstack.org/35/294535/10/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/ab4259f/logs/devstacklog.txt.gz 2016-09-27 17:55:40.371 | 11:53 |
| AJaeger | rohanion: better ask on #openstack-qa or even #openstack-keystone | 11:55 |
| rohanion | it's not a keystone issue I suppose | 11:56 |
| *** tosky has quit IRC | 11:56 | |
| rohanion | ok thank you | 11:56 |
| AJaeger | rohanion: I does not look like an infra problem to me, so those other two should be your first contact. | 11:57 |
| *** stevebak` is now known as stevebaker | 11:57 | |
| rohanion | Fine | 11:57 |
| rohanion | thank you! | 11:58 |
| *** amoralej is now known as amoralej|lunch | 11:59 | |
| *** psilvad has joined #openstack-infra | 12:00 | |
| *** Douhet has joined #openstack-infra | 12:00 | |
| *** amitgandhinz has quit IRC | 12:03 | |
| *** tiswanso has joined #openstack-infra | 12:03 | |
| *** artmr has joined #openstack-infra | 12:07 | |
| *** kgiusti has joined #openstack-infra | 12:11 | |
| *** tiswanso has quit IRC | 12:12 | |
| *** tuanluong has quit IRC | 12:13 | |
| *** raildo has joined #openstack-infra | 12:14 | |
| *** tosky has joined #openstack-infra | 12:14 | |
| *** kgiusti has quit IRC | 12:15 | |
| *** edmondsw has joined #openstack-infra | 12:15 | |
| *** ccamacho|lunch is now known as ccamacho | 12:15 | |
| openstackgerrit | Merged openstack-infra/system-config: Start using br_infracloud as bridge name https://review.openstack.org/368727 | 12:17 |
| *** trown|outtypewww is now known as trown | 12:18 | |
| *** number80 has joined #openstack-infra | 12:20 | |
| number80 | hello | 12:20 |
| number80 | I'd like to give third-party CI perms to give +V on a project, do I need to add them to the <PROJECT>-ci group or do I need to do something else? | 12:21 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Hide arrows to expand task details if there are no details https://review.openstack.org/379417 | 12:22 |
| number80 | And if it's option 1., it seems that I can't do it as rpm-packaging PTL | 12:23 |
| *** EricGonczer_ has joined #openstack-infra | 12:23 | |
| AJaeger | number80: yeah, you should need them to rpm-packaging-ci. Let's see who owns that... | 12:24 |
| number80 | ack | 12:25 |
| AJaeger | https://review.openstack.org/#/admin/groups/1008,members and https://review.openstack.org/#/admin/groups/uuid-ca033ee2dd2d63109cf7827ebb5658419c1005c9,info | 12:25 |
| AJaeger | it was never set up. | 12:25 |
| *** ihrachys has joined #openstack-infra | 12:25 | |
| *** salv-orlando has joined #openstack-infra | 12:25 | |
| AJaeger | number80: you need an infra-root that adds rpm-packaging-core as owner to the group - then you can self-administrate it. | 12:25 |
| AJaeger | infra-root, could somebody help the rpm packaging team, please? ^ | 12:26 |
| number80 | AJaeger: thanks :) | 12:26 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Add lost job to master and newton branches pipelines https://review.openstack.org/379282 | 12:28 |
| openstackgerrit | Isaac Beckman proposed openstack-infra/project-config: selective caching of git repos https://review.openstack.org/322487 | 12:29 |
| *** yaume_ has joined #openstack-infra | 12:29 | |
| *** rkukura has quit IRC | 12:30 | |
| *** salv-orlando has quit IRC | 12:30 | |
| mordred | AJaeger: done | 12:30 |
| *** zhurong has joined #openstack-infra | 12:31 | |
| *** kgiusti has joined #openstack-infra | 12:32 | |
| *** yaume has quit IRC | 12:32 | |
| *** mriedem has joined #openstack-infra | 12:33 | |
| *** dtantsur has joined #openstack-infra | 12:38 | |
| *** rfolco_ has joined #openstack-infra | 12:38 | |
| *** rlandy has joined #openstack-infra | 12:38 | |
| pabelanger | wznoinsk: what operating system? | 12:40 |
| *** smarcet1981 has joined #openstack-infra | 12:41 | |
| pabelanger | wznoinsk: I've been working on fixing the issues upstream, since we have the same problems: https://review.openstack.org/#/c/378985/ and https://review.openstack.org/#/c/374371/ | 12:41 |
| *** jheroux has joined #openstack-infra | 12:41 | |
| rcarrillocruz | heya | 12:45 |
| rcarrillocruz | is the external connectivity issue on OSIC being investigated ? | 12:45 |
| *** burgerk has joined #openstack-infra | 12:46 | |
| wznoinsk | pabelanger: ironic node is xenial, nodepool is trusty | 12:47 |
| mordred | rcarrillocruz: yah - cloudnull was looking in to it yesterday | 12:47 |
| rcarrillocruz | k thx | 12:48 |
| *** shardy has quit IRC | 12:48 | |
| wznoinsk | pabelanger: I'm thinking of fixing it here only, add a (configurable) retry mechanism for these ssh checks | 12:48 |
| pabelanger | wznoinsk: did you used diskimage-builder for your images? | 12:49 |
| wznoinsk | pabelanger: yes, the problem is the node is still in the bootup process (it's waiting for one of the NICs to timeout waiting for DHCP) while the nodepool tries to connect over ssh | 12:49 |
| wznoinsk | pabelanger: re https://review.openstack.org/#/c/374371/, wouldn't generating host keys at the dib stage (in simple-init) be wrong? | 12:50 |
| *** e0ne has joined #openstack-infra | 12:50 | |
| pabelanger | wznoinsk: Right, so you could increase your SSH timeout in nodepool.yaml too. | 12:51 |
| wznoinsk | sorry, I see the other change | 12:51 |
| *** amitgandhinz has joined #openstack-infra | 12:51 | |
| pabelanger | ya, we created a new element, runtime-ssh-host-keys, to manage that. So we can setup a proper dependency on ssh.service | 12:51 |
| wznoinsk | pabelanger: the problem is nodepool is able to connect to sshd on the node but not establish a session, I saw sshd is returning a message 'still booting' or alike | 12:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Don't specify an executable if it's not needed https://review.openstack.org/379439 | 12:52 |
| pabelanger | wznoinsk: check to see if ssh host keys are generated on the node | 12:52 |
| pabelanger | likely SSHd is getting restarted some place | 12:52 |
| wznoinsk | I'm pretty sure they are, this is 1:1 copy of a working image used on VMs now problematic on baremetal | 12:53 |
| wznoinsk | pabelanger: ok, will check the exact timings | 12:53 |
| pabelanger | mordred: do you mind commenting on: https://review.openstack.org/#/c/374371/ ? | 12:53 |
| pabelanger | wznoinsk: sure, I hope to have a new release of glean / diskimage-builder today / tomorrow. I'll post results for our VMs | 12:54 |
| *** tdasilva has quit IRC | 12:54 | |
| *** yamamoto_ has quit IRC | 12:54 | |
| *** tuannguyen has joined #openstack-infra | 12:54 | |
| *** psilvad has quit IRC | 12:56 | |
| *** EricGonc_ has joined #openstack-infra | 12:56 | |
| *** psilvad has joined #openstack-infra | 12:56 | |
| *** ddieterly has joined #openstack-infra | 12:56 | |
| *** david-lyle has joined #openstack-infra | 12:57 | |
| *** vikrant has quit IRC | 12:57 | |
| *** rlu has quit IRC | 12:58 | |
| *** EricGonczer_ has quit IRC | 12:58 | |
| *** ddieterly has quit IRC | 12:59 | |
| *** ddieterly has joined #openstack-infra | 12:59 | |
| *** rlu has joined #openstack-infra | 12:59 | |
| *** jpena|lunch is now known as jpena | 12:59 | |
| *** ddieterly has quit IRC | 12:59 | |
| Zara | https://review.openstack.org/#/c/371515/ awaits reviews (infra-manual). It's a patch to explain how the gerrit plugin behaves when given 'story' and 'task' in commit messages | 12:59 |
| Zara | (and there are also some heavily commented tutorial scripts for the python storyboardclient, based on things people needed during the bugsquash https://review.openstack.org/#/c/371620/) | 13:01 |
| *** psilvad has quit IRC | 13:01 | |
| *** trash_ is now known as trash | 13:01 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Add a note for br_ironic_vxlan tunnel https://review.openstack.org/379447 | 13:01 |
| *** psilvad has joined #openstack-infra | 13:05 | |
| *** oanson has quit IRC | 13:05 | |
| *** Goneri has joined #openstack-infra | 13:05 | |
| *** esberglu has joined #openstack-infra | 13:06 | |
| *** OferBarber_ has joined #openstack-infra | 13:07 | |
| *** zhurong has quit IRC | 13:09 | |
| *** psilvad has quit IRC | 13:09 | |
| *** zhurong has joined #openstack-infra | 13:10 | |
| *** OferBarber has quit IRC | 13:10 | |
| *** jamesden_ has joined #openstack-infra | 13:11 | |
| *** dragonmaster has quit IRC | 13:12 | |
| *** baoli has joined #openstack-infra | 13:13 | |
| *** baoli_ has joined #openstack-infra | 13:14 | |
| *** matt-borland has joined #openstack-infra | 13:15 | |
| *** yamamoto has joined #openstack-infra | 13:16 | |
| *** links has quit IRC | 13:17 | |
| *** baoli has quit IRC | 13:17 | |
| *** mdrabe has joined #openstack-infra | 13:19 | |
| *** xyang1 has joined #openstack-infra | 13:19 | |
| *** jaosorior has quit IRC | 13:21 | |
| *** zhurong has quit IRC | 13:21 | |
| *** jaosorior has joined #openstack-infra | 13:22 | |
| *** amoralej|lunch is now known as amoralej | 13:22 | |
| *** tnovacik has quit IRC | 13:25 | |
| *** tuannguyen has quit IRC | 13:25 | |
| *** tuannguyen has joined #openstack-infra | 13:26 | |
| *** eharney has joined #openstack-infra | 13:26 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate computes from 31 to 40 to Puppetmaster Ansible inventory https://review.openstack.org/379459 | 13:27 |
| *** salv-orlando has joined #openstack-infra | 13:27 | |
| *** tnovacik has joined #openstack-infra | 13:27 | |
| mordred | pabelanger: done | 13:27 |
| mordred | wznoinsk: I agree with your comment on that change, but the depends-on should take care of it | 13:27 |
| *** zhurong has joined #openstack-infra | 13:27 | |
| *** mriedem has quit IRC | 13:28 | |
| wznoinsk | mordred: ah, yes, missed that | 13:28 |
| *** cardeois has joined #openstack-infra | 13:28 | |
| pabelanger | that is the original patchset for runtime-ssh-host-key, which does work. | 13:29 |
| pabelanger | 378985 is an update to some edge cases I found | 13:29 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add chocolate computes from 41 to 50 to Puppetmaster Ansible inventory https://review.openstack.org/379464 | 13:29 |
| *** tiswanso has joined #openstack-infra | 13:30 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Bug on Summit time convertion routines https://review.openstack.org/379465 | 13:31 |
| *** salv-orlando has quit IRC | 13:31 | |
| *** rkukura has joined #openstack-infra | 13:31 | |
| *** mtanino has joined #openstack-infra | 13:32 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Bug on Summit time convertion routines https://review.openstack.org/379465 | 13:32 |
| *** pgadiya has quit IRC | 13:34 | |
| rcarrillocruz | fungi , pabelanger : can you pls have a look at https://review.openstack.org/#/c/378996/1/manifests/site.pp when you get a chance? | 13:39 |
| *** OferBarber has joined #openstack-infra | 13:40 | |
| *** claudiub has joined #openstack-infra | 13:40 | |
| pabelanger | sure | 13:40 |
| rcarrillocruz | thx | 13:41 |
| *** rkukura has quit IRC | 13:42 | |
| pabelanger | rcarrillocruz: comments left | 13:42 |
| openstackgerrit | Merged openstack-infra/system-config: Add chocolate computes from 31 to 40 to Puppetmaster Ansible inventory https://review.openstack.org/379459 | 13:42 |
| *** OferBarber_ has quit IRC | 13:42 | |
| strigazi | Hi all, we get a lot of these timeouts http://logs.openstack.org/07/379207/1/check/gate-functional-dsvm-magnum-mesos/59bfa03/logs/devstacklog.txt.gz#_2016-09-29_07_22_39_132 | 13:42 |
| pabelanger | Hmm, I thought mordred had a patch up on how to fix that | 13:43 |
| dhellmann | can someone help me with gerrit acls? I'm trying to figure out which permissions allowed someone to make a bad stable/newton branch in the networking-hyperv project. Is it "create" under refs/heads/* that needs to be removed? | 13:43 |
| *** tiswanso has quit IRC | 13:43 | |
| *** claudiub|2 has quit IRC | 13:43 | |
| strigazi | Andreas told me that there is an issue in the OSIC DC, at least that's what I understood | 13:44 |
| *** martinkopec has quit IRC | 13:44 | |
| pabelanger | strigazi: you could update your job to do: https://review.openstack.org/#/c/378999/ | 13:44 |
| pabelanger | setting DIB_REPOLOCATION_pip_and_virtualenv | 13:44 |
| *** martinkopec has joined #openstack-infra | 13:44 | |
| strigazi | pabelanger: I'm having a look | 13:44 |
| *** smarcet1981 has quit IRC | 13:44 | |
| *** tiswanso has joined #openstack-infra | 13:45 | |
| openstackgerrit | Merged openstack-infra/system-config: Add chocolate computes from 41 to 50 to Puppetmaster Ansible inventory https://review.openstack.org/379464 | 13:45 |
| pabelanger | strigazi: export DIB_REPOLOCATION_pip_and_virtualenv=/home/jenkins/cache/files/get-pip.py before creating your DIB | 13:45 |
| mordred | actually ... | 13:46 |
| mordred | don't use that path | 13:46 |
| mordred | it turns out to be incorrect | 13:46 |
| pabelanger | Ah | 13:46 |
| dansmith | can anyone help me debug why our post_test_hook seems to not be running? | 13:46 |
| mordred | /opt/stack/new/devstack/files/get-pip.py | 13:46 |
| mordred | I think | 13:46 |
| dhellmann | anteaya : maybe you know the answer to my gerrit acl question? ^^ | 13:46 |
| mordred | one sec | 13:46 |
| *** tiswanso has quit IRC | 13:46 | |
| anteaya | dhellmann: I will look | 13:47 |
| strigazi | mordred pabelanger, Can I do anything else? | 13:47 |
| mordred | strigazi: yah - one sec ... trying to sort out where the cached file is | 13:48 |
| *** burgerk has quit IRC | 13:48 | |
| strigazi | mordred, sure, thanks | 13:48 |
| mordred | pabelanger: I thought we were pre-caching files on these nodes | 13:48 |
| *** tiswanso has joined #openstack-infra | 13:48 | |
| anteaya | dhellmann: that repo can also submit merge commits, with the pushMerge rule on refs/for/refs/* | 13:49 |
| anteaya | dhellmann: since I'm looking at it | 13:49 |
| pabelanger | mordred: we should be, I'm checking the DIB logs | 13:49 |
| mordred | pabelanger: I just logged in to a centos and a precise node | 13:49 |
| mordred | and all I see in /opt is: | 13:49 |
| mordred | [mordred@centos-7-osic-cloud1-disk-4632385 ~]$ ls /opt | 13:49 |
| mordred | build_git git nodepool-scripts requirements zuul | 13:49 |
| mordred | oh - ok. the original path is right | 13:50 |
| mordred | ./home/jenkins/cache/files/get-pip.py | 13:50 |
| *** tdasilva has joined #openstack-infra | 13:50 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: fix permissions on winstacker repositories https://review.openstack.org/379478 | 13:50 |
| anteaya | dhellmann: they can create any named branch with create on refs/heads/* | 13:50 |
| dhellmann | anteaya :good catch, ^^ should fix | 13:50 |
| anteaya | dhellmann: they also can create stable/anything themeselves since the refs/heads/stable/* permissions point to their team | 13:51 |
| anteaya | dhellmann: what permissions would you like them to have? what makes the stable/newton branch they created bad? | 13:51 |
| asselin__ | can anyone tell me how big the openstack paste database is? ours is > 1GB which is a ridiculous about of text. | 13:52 |
| dhellmann | anteaya: it wasn't cut from a release | 13:52 |
| anteaya | dhellmann: ah | 13:52 |
| dhellmann | anteaya : they should not have any branch or tagging permissions at all | 13:52 |
| *** vsaienko has quit IRC | 13:52 | |
| anteaya | dhellmann: who should? | 13:52 |
| pabelanger | mordred: Oh, interesting /home/jenkins/cache/files is missing on xenial | 13:52 |
| pabelanger | but exists for trusty | 13:52 |
| mordred | GAH | 13:52 |
| dhellmann | anteaya : the default is for release managers to do that | 13:52 |
| mordred | we MOVE it in devstack-gate | 13:52 |
| *** tiswanso has quit IRC | 13:53 | |
| dhellmann | anteaya : so if we remove any settings in their files, the defaults should be used | 13:53 |
| pabelanger | Ah, that explains it | 13:53 |
| anteaya | dhellmann: what repo should I look at that is set up correctly? | 13:53 |
| mordred | so if you use devstack-gate as a wrapper, it'll be in /opt/stack/new/devstack/files/get-pip.py ... if you don't, it'll be in /home/jenkins/cache/files | 13:53 |
| anteaya | as my example? | 13:53 |
| *** dkehn has joined #openstack-infra | 13:53 | |
| pabelanger | mordred: ouch | 13:53 |
| mordred | maybe we should just cache the files in one location, have that be a known location, not move them anyway, and set up things to consume them from the cache location | 13:53 |
| *** vsaienko has joined #openstack-infra | 13:55 | |
| pabelanger | seems reasonable | 13:55 |
| mordred | I can only imagine how many patches it'll take to safely do that though | 13:55 |
| mordred | I'm not convinced the /opt/stack/new/devstack/files/get-pip.py path is correct, btw | 13:55 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: fix permissions on winstacker repositories https://review.openstack.org/379478 | 13:55 |
| strigazi | for now we must check both locations? | 13:55 |
| dhellmann | anteaya : os-win is like oslo.config and networking-hyperv is like nova | 13:56 |
| *** Guest92 has joined #openstack-infra | 13:56 | |
| pabelanger | we could stop devstack-gate from rm -rf ~/cache to start | 13:56 |
| mordred | strigazi: no - it'll be in one location - just still trying to find the right one to tell you :) | 13:56 |
| *** spzala has joined #openstack-infra | 13:56 | |
| mordred | pabelanger: reading setup_workspace is seems it'll be in /opt/stack/cache/files/get-pip.py | 13:56 |
| anteaya | dhellmann: what that patch has just done is create a second file named networking-hyperv.config | 13:57 |
| *** yamamoto has quit IRC | 13:57 | |
| anteaya | dhellmann: I'm not sure what gerrit will do if it has two competing configs for the same repo | 13:57 |
| dhellmann | anteaya : ? | 13:57 |
| *** rossella_ has joined #openstack-infra | 13:57 | |
| dhellmann | i only see one file, and it's being modified | 13:57 |
| *** matrohon has joined #openstack-infra | 13:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Pull get-pip.py from jenkins cache https://review.openstack.org/378999 | 13:58 |
| mordred | let's see if that works | 13:58 |
| mordred | strigazi: ^^ if that patch turns the dvsm jobs for that patch green, we'll be in good shape | 13:58 |
| dansmith | so we landed this project-config change yesterday: https://review.openstack.org/#/c/376537/6 | 13:59 |
| dansmith | but it doesn't seem to be taking effect yet | 13:59 |
| anteaya | dhellmann: my apologies when I opened it in the gui the first time I saw it as renaming os-win.config to networking-hyperv.config | 13:59 |
| pabelanger | mordred: So, we are already updating the path with zuulv3 changes: https://review.openstack.org/#/c/337821/3/nodepool/elements/cache-devstack/source-repository-pip So, we can have a discussion about where the right place to cache things are moving forward | 13:59 |
| dhellmann | anteaya : ah, np | 13:59 |
| dansmith | I don't see that new hook defined in the devstack-gate run.. does something else have to happen? | 13:59 |
| anteaya | dhellmann: I refreshed and now I see two separate files, sorry | 13:59 |
| *** rkukura has joined #openstack-infra | 13:59 | |
| pabelanger | /opt/dib_cache | 14:00 |
| pabelanger | or something along that lines | 14:00 |
| mordred | pabelanger: do we have a d-g patch yet that looks in both places? | 14:00 |
| *** lezbar has quit IRC | 14:00 | |
| mordred | before it does its copy? | 14:00 |
| *** rohanion has quit IRC | 14:00 | |
| *** Guest92 has quit IRC | 14:00 | |
| strigazi | mordred, thanks | 14:01 |
| pabelanger | mordred: not yet, 337821 keeps the original pathes in place so we can then start migrating projects to the new location | 14:01 |
| anteaya | dhellmann: in this file, do you want to leave networking-hyperv with the ability to create stable branches? https://review.openstack.org/#/c/379478/2/gerrit/acls/openstack/networking-hyperv.config | 14:01 |
| *** lezbar has joined #openstack-infra | 14:01 | |
| dhellmann | anteaya : no. am I doing that? | 14:01 |
| anteaya | dhellmann: yes, you added a new section | 14:02 |
| anteaya | dhellmann: I'll inline comment | 14:02 |
| dhellmann | anteaya : please do. I'm cargo-culting these files, so I don't know what line is wrong | 14:02 |
| *** mriedem has joined #openstack-infra | 14:02 | |
| anteaya | dhellmann: ah okay thanks for the heads up on that | 14:03 |
| *** szaher_ has quit IRC | 14:04 | |
| *** tiswanso has joined #openstack-infra | 14:05 | |
| *** yamamoto has joined #openstack-infra | 14:05 | |
| *** szaher has joined #openstack-infra | 14:05 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: fix permissions on winstacker repositories https://review.openstack.org/379478 | 14:05 |
| *** Guest92 has joined #openstack-infra | 14:06 | |
| *** thorst is now known as thorst_afk | 14:06 | |
| AJaeger | anteaya: the new section is fine, check the neutron or nova files for examples | 14:07 |
| anteaya | AJaeger: okay if you want to take over I'm fine with that | 14:07 |
| anteaya | I just don't want to double team dhellmann | 14:07 |
| anteaya | happy to review whatever you work out with him AJaeger | 14:07 |
| *** rbrndt has joined #openstack-infra | 14:07 | |
| AJaeger | anteaya: no need to take over, you give great advise. Just wanted to point out that single point, it's new with newton | 14:08 |
| cloudnull | mornings | 14:08 |
| * mordred hands cloudnull a coffee | 14:08 | |
| dhellmann | anteaya, AJaeger : yeah, that's a new special case situation. it will revert to "normal" after the final release next week. | 14:08 |
| * cloudnull drinks | 14:08 | |
| AJaeger | dhellmann: LGTM now. | 14:09 |
| dhellmann | AJaeger : thanks | 14:09 |
| cloudnull | so reading scroll back, are we seeing egress ingress issues within the OSIC or are the error nodes and failures mentioned by AJaeger related to multi-node setup? | 14:11 |
| strigazi | mordred: I guess it didn't work http://logs.openstack.org/99/378999/4/check/gate-nodepool-python27-db-ubuntu-xenial/09ab15d/console.html#_2016-09-29_14_08_23_815783 | 14:12 |
| cloudnull | clarkb: I've been working on seeing whats up with tunneling on our GATEWAY_NET_V6 but i've not nailed that down quite yet. | 14:12 |
| cloudnull | it's been itermitent and idk | 14:12 |
| AJaeger | cloudnull: strigazi reported those, and reading backscroll I see strigazi, mordred, and pabelanger are discussing it. | 14:12 |
| cloudnull | *idk why | 14:12 |
| cloudnull | ah. | 14:13 |
| clarkb | frickler: ugh | 14:13 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Update bridge name to match the real one https://review.openstack.org/379490 | 14:13 |
| clarkb | cloudnull: frickler had an idea in scrollback appare tly neutron is setting a route for 10/8 on our overlay neyworks | 14:14 |
| clarkb | cloudnull: I do not know why buy that could certainly explain connectivity issues given the addresses in use | 14:14 |
| *** ddieterly has joined #openstack-infra | 14:14 | |
| clarkb | why is neutron setting a route for a network range it does not control? | 14:15 |
| frickler | clarkb: might be the result of https://review.openstack.org/356026 | 14:15 |
| anteaya | dhellmann: do you need fungi's help in untangling the bad networking-hyperv stable/newton branch? | 14:15 |
| AJaeger | clarkb, pabelanger, mordred, fungi: Could you look at this project-config change for our images, please? https://review.openstack.org/#/c/377159/ To me it seems that we're duplicating information between devstack and project-config in a fragile way. Need some experts to look closer (not urgent) | 14:15 |
| cloudnull | clarkb: yea, idk why that would be . | 14:15 |
| frickler | clarkb: defaul subnetpool is 10/8 indeed | 14:15 |
| anteaya | dhellmann: if yes, I'd like to discuss the timing of that prior to merging that acl | 14:15 |
| anteaya | dhellmann: it case it has an impact | 14:15 |
| pabelanger | clarkb: cloudnull: working on DIB image issue in osic-cloud1-s3500 | 14:16 |
| dhellmann | anteaya : I need to review it and see if we have to recreate it, or if they are just going to need to do the patch updates for .gitreview and reno by hand | 14:16 |
| clarkb | frickler: thanks that definitely seems to be a likely cause | 14:16 |
| mordred | clarkb, frickler: I feel like we had a chat with sc68cal a few weeks ago tracking down a similar bug | 14:16 |
| dhellmann | anteaya : either way it should be safe to merge that patch, because if we have to recreate the branch I'll do that for them | 14:16 |
| clarkb | its annoying because we configure neutron in the test to use specific ramges | 14:17 |
| anteaya | dhellmann: okay so you don't need fungi to help then? | 14:17 |
| mordred | clarkb: ++ | 14:17 |
| *** rkukura has quit IRC | 14:17 | |
| clarkb | then it decides that no it needs all of 0/8 and anyone using 10/8 is acrewed | 14:17 |
| mordred | I would contend that is a bug :) | 14:17 |
| dhellmann | anteaya : I don't know yet. I'll ask him if I determine that I do. | 14:17 |
| dhellmann | anteaya : that'll be a little while; I'm in a meeting. | 14:18 |
| anteaya | dhellmann: you want nova-release-branch to be the group in charge of stable/newton for hyperv? | 14:18 |
| anteaya | nova? | 14:18 |
| *** flepied has joined #openstack-infra | 14:18 | |
| dhellmann | anteaya : no, does it still say nova somewhere? | 14:18 |
| anteaya | dhellmann: I'll wait to your finished then, thanks | 14:18 |
| anteaya | dhellmann: yes | 14:18 |
| *** annegentle has joined #openstack-infra | 14:18 | |
| anteaya | dhellmann: sorry that was in patchset2 | 14:19 |
| dhellmann | I don't see nova anywhere in my copy of the file | 14:19 |
| dhellmann | ok | 14:19 |
| anteaya | dhellmann: I too am trying to do too many things at once | 14:19 |
| dhellmann | anteaya :-) | 14:19 |
| anteaya | dhellmann: will stop now until I can focus | 14:19 |
| anteaya | sorry | 14:19 |
| *** jpeeler1 is now known as jpeeler | 14:19 | |
| *** jpeeler has quit IRC | 14:19 | |
| *** jpeeler has joined #openstack-infra | 14:19 | |
| *** martinkopec has quit IRC | 14:20 | |
| pabelanger | clarkb: ya, we are running into some issues in osic-cloud1 with trying to reuse the same image across all three providers | 14:22 |
| *** yamamoto has quit IRC | 14:22 | |
| irtermite | pabelanger: clarkb: due to type or due to something you need cloudnull and me to look at? | 14:22 |
| frickler | clarkb: where do you set special ranges? maybe that can be amended to subnetpools | 14:23 |
| pabelanger | irtermite: no, this is how nodepool does our uploads. | 14:23 |
| *** yamamoto has joined #openstack-infra | 14:23 | |
| irtermite | right on, thanks pabelanger | 14:23 |
| *** rlu has quit IRC | 14:24 | |
| *** flepied has quit IRC | 14:24 | |
| *** sdake has quit IRC | 14:24 | |
| *** EricGonc_ has quit IRC | 14:25 | |
| *** hongbin has joined #openstack-infra | 14:25 | |
| *** smarcet1981 has joined #openstack-infra | 14:26 | |
| *** sdake has joined #openstack-infra | 14:26 | |
| clarkb | frickler: we have always set the floating ip range and the fixed range http://logs.openstack.org/41/379141/2/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/1ecb283/logs/localrc.txt.gz | 14:26 |
| *** yamamoto has quit IRC | 14:27 | |
| clarkb | frickler: the fixed range is specifically set to avoid issues with private networls in various clouds and the fixed range is configured to be properly routable on multinode hosts | 14:27 |
| *** yamamoto has joined #openstack-infra | 14:27 | |
| *** yamamoto has quit IRC | 14:27 | |
| openstackgerrit | matthew wagoner proposed openstack-infra/shade: WIP: Add test for os_zone Ansible module https://review.openstack.org/379498 | 14:27 |
| clarkb | frickler: maybe subnetpool ramge should default to fixed range? | 14:28 |
| clarkb | but also why is neutron setting that route there at all? that implies you want VMs to be able to talk to hypervisors? | 14:28 |
| clarkb | that seems wrong | 14:29 |
| *** salv-orlando has joined #openstack-infra | 14:29 | |
| pabelanger | clarkb: are you okay with reverting https://review.openstack.org/#/c/378043/ (dib image id for glance uploads) until we figure out how serialize the image uploads for osic-cloud1? | 14:29 |
| pabelanger | Or maybe just revert it for osic-cloud1 | 14:29 |
| pabelanger | so we don't have to upload images every where | 14:30 |
| frickler | clarkb: it's the other way round I guess, users that just deploy devstack want to be able to talk to their VM from the hypervisor, so devstack does all kinds of crazy things to ensure first-user experience is fine | 14:30 |
| frickler | clarkb: how you work around that in gate is another issue, then. I think for now you would need to set SUBNETPOOL_PREFIX_ | 14:31 |
| frickler | clarkb: _V4 properly matching your fixed range (damn c&p of eol ;) | 14:31 |
| *** inc0 has joined #openstack-infra | 14:31 | |
| clarkb | I dunno I think we should fix this in devstack so we stop brealing everyone that dares to use 10/8 properly | 14:32 |
| frickler | clarkb: so that would be supporting https://review.openstack.org/378063 then | 14:32 |
| clarkb | pabelanger: yes thats fine also you'll need to edit the osic additions that went on top of that change | 14:33 |
| cloudnull | if we want to revert back to the single region for a while that'd be fine too. | 14:33 |
| *** yamahata has joined #openstack-infra | 14:33 | |
| clarkb | frickler: or just make it default to the fixed range or something | 14:33 |
| clarkb | cloudnull: I dont think we need to we will.just end up using more images than strictly necessary | 14:34 |
| *** sdake_ has joined #openstack-infra | 14:34 | |
| *** kzaitsev_mb has quit IRC | 14:35 | |
| *** rkukura has joined #openstack-infra | 14:35 | |
| *** lezbar has quit IRC | 14:35 | |
| *** tongli has joined #openstack-infra | 14:36 | |
| *** lezbar has joined #openstack-infra | 14:36 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Stop using template-name for osic-cloud1 https://review.openstack.org/379506 | 14:36 |
| pabelanger | clarkb: opted for ^, to avoid image uploads across all of our clouds | 14:36 |
| *** sdake has quit IRC | 14:36 | |
| clarkb | frickler: we actually have had to jump through similar hoops for testing purposes (communication to VMs) so I dont ha e a huge issue with that in the test setup. More annoyed that devstack is ignoring previously carefully selected ranges that are known to work. So I think its fine to just tweak defaults there | 14:36 |
| pabelanger | cloudnull: so, you are ready for us to remove osic-cloud1 from nodepool? | 14:37 |
| cloudnull | yup | 14:37 |
| clarkb | pabelanger: it shouldnt cause image uploads outside the normal upload process | 14:37 |
| clarkb | pabelanger: I would just make it consistent across all providers | 14:37 |
| pabelanger | clarkb: okay | 14:37 |
| cloudnull | we can keep it in play if that makes life easier but it can come out at any time. | 14:37 |
| *** nherciu_ has joined #openstack-infra | 14:38 | |
| *** kaisers_ has quit IRC | 14:38 | |
| *** amotoki has joined #openstack-infra | 14:38 | |
| dmsimard | Anyone else seeing a lot of POST_FAILUREs ? | 14:39 |
| *** burgerk has joined #openstack-infra | 14:39 | |
| dmsimard | Looks related to log upload because the logs aren't available | 14:39 |
| dmsimard | Two recent examples: https://review.openstack.org/#/c/363142/ && https://review.openstack.org/#/c/378995/ | 14:40 |
| dmsimard | I'd love to investigate the failure but I can't really do that if I don't have logs :( | 14:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Use the DIB image id in glance uploads" https://review.openstack.org/379506 | 14:41 |
| *** nherciu has quit IRC | 14:41 | |
| clarkb | dmsimard: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all we may have run out of disk again | 14:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Remove osic-cloud1 provider from nodepool https://review.openstack.org/379513 | 14:42 |
| dmsimard | clarkb: oh no :( | 14:42 |
| *** john-davidge has quit IRC | 14:42 | |
| clarkb | fungi was the last set of deletes manual? we didmt update the cron to reduce retention? | 14:43 |
| *** kaisers_ has joined #openstack-infra | 14:43 | |
| clarkb | if so we will likely need to reapply the same intervention and probably update the cleaning cron | 14:44 |
| *** yuval has quit IRC | 14:45 | |
| *** gordc has joined #openstack-infra | 14:46 | |
| Zara | hi, I mentioned this earlier but it was too early in the day-- there's a story in storyboard-dev; it looks like it might not be a test story; if not I'd like to give the author a heads-up (and find out how he landed on dev), but I don't know how to contact him. I can leave a comment but storyboard-dev doesn't send emails, so it may not be seen. https://storyboard-dev.openstack.org/#!/story/46 Maybe i | 14:46 |
| Zara | t is just a test story and I don't know enough about it. | 14:46 |
| Zara | mentioning it in case someone gets questions later; it might ring a bell | 14:46 |
| *** rkukura has quit IRC | 14:47 | |
| pabelanger | clarkb: fungi: I can look at logs.o.o, if we have it documented how to clean up | 14:47 |
| *** amotoki has quit IRC | 14:48 | |
| clarkb | pabelanger: we hve the cron thats currnetly set to 3 months but I want to say fungi had to run it manually at 2 months retention to give us breathing room | 14:48 |
| dmsimard | pabelanger, clarkb: worthy of a status update ? | 14:49 |
| pabelanger | clarkb: ya, looks like fungi has something in his home directory | 14:50 |
| *** aviau has quit IRC | 14:50 | |
| *** aviau has joined #openstack-infra | 14:51 | |
| mtreinish | fungi: it's starting to look like if we build the latest libwebsockets release from source and then rebuild mosquitto to use that things don't crash | 14:52 |
| mtreinish | I'll keep it running in the background (mostly because everytime I say it works in channel it crashes) for a while, but it's already been up for abount an hour without a crash | 14:53 |
| *** jamielennox is now known as jamielennox|away | 14:53 | |
| *** OferBarber has quit IRC | 14:53 | |
| *** sdake_ is now known as sdake_dnd | 14:53 | |
| zhenguo | can some help to +A this https://review.openstack.org/#/c/377268/ , thanks you! | 14:54 |
| *** zhurong has quit IRC | 14:54 | |
| mtreinish | fungi: but if that is the fix, any suggestions on how best to automate the build and install? | 14:54 |
| AJaeger | zhenguo: it helps a lot if you tell us a bit more about the change so that the right people in the channel can help | 14:54 |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/devstack-gate: Set SUBNETPOOL_PREFIX_V4 to match FIXED_RANGE https://review.openstack.org/379521 | 14:55 |
| frickler | clarkb: ^^ I think that should help for now | 14:55 |
| zhenguo | AJaeger: thanks, it's about adding a new project to statusbot and meetbot | 14:56 |
| pabelanger | clarkb: fungi: running log_archive_maintenance.sh now | 14:56 |
| zigo | Oh, first time I see this: https://review.openstack.org/#/c/379433/ <--- It says POST failure. | 14:57 |
| zigo | How can I tell what happened? | 14:57 |
| zigo | There's no log ... | 14:57 |
| frickler | zigo: looks like the disk of the logserver is full, hopefully fixed soon | 14:58 |
| zigo | frickler: Oh, ok, so I guess everything is broken! :P | 14:58 |
| AJaeger | zigo: wait and recheck once there's space again | 14:58 |
| zigo | Thanks. | 14:58 |
| clarkb | pabelanger: thank you, I am dealing with sick kids as they wake so not really "here" yet | 14:59 |
| *** annegentle has quit IRC | 14:59 | |
| anteaya | clarkb: :( hope they start feeling better soon | 14:59 |
| *** sdake has joined #openstack-infra | 14:59 | |
| fungi | pabelanger: oh, did we fill up again? | 14:59 |
| *** electrical is now known as fibre | 14:59 | |
| pabelanger | fungi: ya | 15:00 |
| *** fibre is now known as electrical | 15:00 | |
| *** kgiusti has quit IRC | 15:00 | |
| zigo | anteaya: Kids always recover quick, but it's also always depressing to see them sick. :/ | 15:00 |
| *** pabelanger has quit IRC | 15:00 | |
| *** pabelanger has joined #openstack-infra | 15:00 | |
| *** jamielennox|away is now known as jamielennox | 15:00 | |
| *** sdake_dnd has quit IRC | 15:00 | |
| zigo | All my hart with clark's familly as well. | 15:00 |
| dmsimard | fungi: lots of POST_FAILURES with no logs: https://review.openstack.org/#/c/363142/ && https://review.openstack.org/#/c/378995/ && http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 15:00 |
| zigo | dmsimard: Disk full, please wait until fixed. | 15:00 |
| fungi | clarkb: i think the first attempt to update the cronjob altered the wrong copy. we have two and i meant to submit a patch to remove the lingering one in system-config | 15:00 |
| dmsimard | zigo: I know, I'm the one who reported it intially :) | 15:00 |
| anteaya | zigo: I'm glad you have experience with children with strong immune systems | 15:01 |
| pabelanger | #status log manually running log_archive_maintenance.sh to make room for logs on static.o.o | 15:01 |
| openstackstatus | pabelanger: finished logging | 15:01 |
| *** thcipriani|afk is now known as thcipriani | 15:02 | |
| fungi | pabelanger: make sure to edit up a version that doesn't bother doing compression and skips the docs-draft volume | 15:02 |
| fungi | the one you spotted in my homedir is probably that | 15:02 |
| *** isaacb has joined #openstack-infra | 15:02 | |
| *** vinaypotluri has joined #openstack-infra | 15:03 | |
| pabelanger | fungi: Ya, I'm using the version in your home directory, but we should consider also adding this into system-config, since we run it a few times | 15:03 |
| *** r-mibu has quit IRC | 15:03 | |
| *** r-mibu has joined #openstack-infra | 15:03 | |
| fungi | pabelanger: heh, ironically we have one in system-config that's not doing anything since it moved to puppet-openstackci and whoever did that never finished teh cleanup part of the transition | 15:04 |
| *** mdrabe has quit IRC | 15:04 | |
| *** jlanoux has quit IRC | 15:04 | |
| *** neiljerram_hg has joined #openstack-infra | 15:04 | |
| pabelanger | Doh | 15:04 |
| neiljerram_hg | Hi. I just got a POST_FAILURE on https://review.openstack.org/#/c/378727/. I'm afraid I haven't had one of those before - does someone here know what it means? | 15:05 |
| fungi | yeah, when i tried to lower the retention officially to 90 days i think i only edited that copy. but then log volume was still so high that 90 days would have been too long anyway and it probably needs to be dropped further to 60 | 15:05 |
| pabelanger | neiljerram_hg: we ran out of HDD space on the log server, you can recheck now | 15:05 |
| neiljerram_hg | pabelanger, Ah OK, thanks, I'll do that. | 15:06 |
| *** bhavik1 has joined #openstack-infra | 15:07 | |
| *** claudiub has quit IRC | 15:09 | |
| *** smarcet1981 has quit IRC | 15:09 | |
| *** mdrabe has joined #openstack-infra | 15:10 | |
| *** rkukura has joined #openstack-infra | 15:11 | |
| *** lucasagomes is now known as lucas-hungry | 15:12 | |
| *** kzaitsev_mb has joined #openstack-infra | 15:13 | |
| *** bhavik1 has quit IRC | 15:13 | |
| frickler | clarkb: do you have 10.1.0.0/20 as fixed range because you need more than a /24 for your private network, or would it be o.k. if the default private subnet gets 10.1.0.0/24 only? if not, one might also set SUBNETPOOL_SIZE_V4 to something smaller than /24 | 15:14 |
| clarkb | frickler: I believe we made it bigger because some tests do want more than a /24 | 15:14 |
| clarkb | s/tests/jobs/ | 15:14 |
| *** amotoki has joined #openstack-infra | 15:14 | |
| clarkb | frickler: but my point is that devstack should continue to honor those values regardless of what they are set to | 15:14 |
| clarkb | since people using a portion of 10/8 in the past may have explicitly set those settings to avoid problems with networking on their end | 15:15 |
| pabelanger | fungi: clarkb: if you have time, would like to see about landing https://review.openstack.org/#/c/374371/ today and get a new glean release if possible | 15:15 |
| *** tiswanso has quit IRC | 15:15 | |
| *** rkukura has quit IRC | 15:15 | |
| frickler | clarkb: I tend to agree with you, but you need to discuss this with other ppl. I can only help you with a workaround | 15:15 |
| pradiprwt | Can any core reviewer please review this patch https://review.openstack.org/#/c/377639/ | 15:16 |
| *** isaacb has quit IRC | 15:17 | |
| *** kaisers_ has quit IRC | 15:18 | |
| *** amotoki has quit IRC | 15:20 | |
| *** coolsvap has quit IRC | 15:22 | |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/devstack-gate: Set SUBNETPOOL_PREFIX_V4 to match FIXED_RANGE https://review.openstack.org/379521 | 15:23 |
| *** yamahata has quit IRC | 15:24 | |
| openstackgerrit | Merged openstack-infra/project-config: Add one new Fuel's plugin https://review.openstack.org/377639 | 15:24 |
| *** rossella_ has quit IRC | 15:25 | |
| *** kaisers_ has joined #openstack-infra | 15:25 | |
| *** xarses has quit IRC | 15:25 | |
| *** smarcet1981 has joined #openstack-infra | 15:26 | |
| *** sdake is now known as sdake_dnd | 15:26 | |
| *** tiswanso has joined #openstack-infra | 15:26 | |
| smarcet1981 | fungi: morning re: https://review.openstack.org/#/c/379022/ are u ok with the changes ? | 15:26 |
| *** amotoki has joined #openstack-infra | 15:27 | |
| *** yamamoto has joined #openstack-infra | 15:28 | |
| *** amotoki has quit IRC | 15:28 | |
| *** thorst_afk is now known as thorst | 15:28 | |
| *** adrian_otto has joined #openstack-infra | 15:29 | |
| openstackgerrit | Merged openstack-infra/system-config: Update bridge name to match the real one https://review.openstack.org/379490 | 15:29 |
| clarkb | frickler: https://review.openstack.org/379543 is sort of what I had in mind | 15:31 |
| *** tiswanso has quit IRC | 15:31 | |
| fungi | smarcet1981: oh, sure. i didn't know if anyone else was going to review it, but it's been almost a day at this point so i've just gone ahead and approved | 15:32 |
| *** yamamoto has quit IRC | 15:33 | |
| *** jlanoux has joined #openstack-infra | 15:34 | |
| *** amotoki has joined #openstack-infra | 15:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: Add missing import https://review.openstack.org/379544 | 15:34 |
| *** baoli_ has quit IRC | 15:34 | |
| dmsimard | pradiprwt: still seeing a lot of failures, can you confirm there is space available now ? | 15:35 |
| dmsimard | pradiprwt: er, sorry | 15:35 |
| dmsimard | pabelanger: ^ | 15:35 |
| *** kaisers_ has quit IRC | 15:35 | |
| *** amotoki has quit IRC | 15:36 | |
| pabelanger | dmsimard: which jobs? We are still cleaning up HDD, and have about 20GB free now | 15:36 |
| dmsimard | pabelanger: ah, I realize that while I have just got the gerrit email the jobs may have run a while ago | 15:37 |
| dmsimard | pabelanger: I'll wait a bit and fire a recheck | 15:37 |
| *** amoralej is now known as amoralej|off | 15:37 | |
| *** tuannguyen has quit IRC | 15:37 | |
| smarcet1981 | fungi: thx a lot ! :) | 15:37 |
| *** Guest92_ has joined #openstack-infra | 15:39 | |
| *** mdrabe has quit IRC | 15:40 | |
| *** mdrabe has joined #openstack-infra | 15:40 | |
| *** Guest92 has quit IRC | 15:42 | |
| *** tnovacik has quit IRC | 15:43 | |
| *** xarses has joined #openstack-infra | 15:43 | |
| *** krtaylor has quit IRC | 15:44 | |
| *** hashar has quit IRC | 15:45 | |
| *** tonytan4ever has joined #openstack-infra | 15:45 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackid: php5-fpm 503 errors https://review.openstack.org/379022 | 15:45 |
| *** annegentle has joined #openstack-infra | 15:45 | |
| *** andreas_s has quit IRC | 15:46 | |
| *** tuannguyen has joined #openstack-infra | 15:46 | |
| *** jlanoux has quit IRC | 15:46 | |
| *** jlanoux has joined #openstack-infra | 15:47 | |
| *** Guest92_ has quit IRC | 15:48 | |
| *** Guest92 has joined #openstack-infra | 15:49 | |
| *** openstackgerrit has quit IRC | 15:49 | |
| *** sandanar has quit IRC | 15:50 | |
| fungi | pabelanger: looking at http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all after i manually expired all logs older than 60 days, it took us a month to fill up again. i'm guessing that means that we're still collecting too many logs to safely do 90 days retention and should update the script to make it 60 until someone can figure out how to get the log quantity | 15:50 |
| fungi | reduced? | 15:50 |
| *** openstackgerrit has joined #openstack-infra | 15:50 | |
| pabelanger | fungi: Ya, I think we need to figure out who is eating up all the space. I can poke around now | 15:51 |
| *** ddieterly is now known as ddieterly[away] | 15:52 | |
| jeblair | (we are running more jobs than ever before, so the answer could be 'no one') | 15:52 |
| *** jlanoux_ has joined #openstack-infra | 15:52 | |
| pabelanger | True | 15:52 |
| fungi | yep | 15:52 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openstackci: Reduce job log retention to 90 days https://review.openstack.org/379553 | 15:52 |
| fungi | anyway, there's the change ^ i _should_ have proposed originally | 15:52 |
| *** jlanoux has quit IRC | 15:52 | |
| anteaya | 12 T is a lot of logs | 15:53 |
| pabelanger | +2 | 15:53 |
| *** rkukura has joined #openstack-infra | 15:53 | |
| fungi | oh, whoops. i have an incorrect subject | 15:53 |
| anteaya | do we even have the conversation about trying to have more space? | 15:53 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-openstackci: Reduce job log retention to 60 days https://review.openstack.org/379553 | 15:54 |
| fungi | anteaya: for that, we need to shard the content across servers (probably afs servers?) | 15:54 |
| anteaya | would it be worth exploring that? | 15:54 |
| fungi | it's been discussed some already | 15:54 |
| anteaya | sorry I've missed that discussion, were there any strong opinions? | 15:55 |
| *** bhavik1 has joined #openstack-infra | 15:55 | |
| jeblair | anteaya: https://review.openstack.org/269928 | 15:56 |
| anteaya | thank you | 15:56 |
| fungi | jeblair found it faster than i could | 15:56 |
| *** uxdanielle has joined #openstack-infra | 15:56 | |
| *** nadya has quit IRC | 15:56 | |
| *** claudiub has joined #openstack-infra | 15:57 | |
| *** yamahata has joined #openstack-infra | 15:57 | |
| jeblair | i would have found it faster if i hadn't made gertty's interactive search case-sensitive (/me notes to change that) | 15:57 |
| fungi | make it accept \c at the end of a search string ;) | 15:58 |
| uxdanielle | Hi folks! The OpenStack UX team is looking for a few more people to help with some Horizon research. We're targeting operators with basic familiarity with using Horizon in a production environment. Send me a message on IRC (uxdanielle) if you have an hour today or tomorrow to help out! | 15:58 |
| * fungi observes that's probably a vi-ism | 15:58 | |
| jeblair | fungi: also, not the best ux for interactive search | 15:59 |
| *** tosky has quit IRC | 15:59 | |
| *** tiswanso has joined #openstack-infra | 15:59 | |
| *** berendt has quit IRC | 15:59 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Remove unused log_archive_maintenance.sh script https://review.openstack.org/379559 | 16:00 |
| * clarkb knows not this vi ism and just tells vim to case insensitive search always | 16:00 | |
| fungi | jeblair: oh, true | 16:00 |
| clarkb | actually its case sensitive if I add in upper case characters | 16:02 |
| clarkb | and insensitive if I just use lower case chars | 16:02 |
| *** ddieterly[away] is now known as ddieterly | 16:02 | |
| fungi | that's an interesting heuristic | 16:03 |
| *** pcaruana has quit IRC | 16:03 | |
| *** woodster_ has joined #openstack-infra | 16:03 | |
| *** rcernin has quit IRC | 16:05 | |
| *** yaume has joined #openstack-infra | 16:06 | |
| pabelanger | fungi: clarkb: so, I am noticing that tripleo jobs are using .xz format for compression, do you know why that would be? | 16:07 |
| *** EricGonczer_ has joined #openstack-infra | 16:07 | |
| clarkb | pabelanger: should give them slightly better compression | 16:07 |
| timrc_ | Random question, taking jobs running on non-static nodes as the example, is there only one ansible launcher server/gearman worker running all of the ansible playbooks or are multuple ansible launcher servers/gearman workers launched to handle high loads? | 16:07 |
| fungi | pabelanger: probably because they're excited by shiny things that may or may not work well in some browsers? | 16:07 |
| jroll | fungi: the "usually people are lazy" heuristic - if one takes the time for a capital letter it must be for a reason :) | 16:08 |
| clarkb | timrc_: we have 7 of them just like we had with jenkins iirc | 16:08 |
| fungi | jroll: ahh, psychology! | 16:08 |
| jroll | :D | 16:08 |
| timrc_ | clarkb: I looked into puppet-zuul and system-config but not see where that is defined? Maybe I'm just terrible at grep'ing though. Would you know where to look? | 16:08 |
| *** tiswanso has quit IRC | 16:09 | |
| fungi | timrc_: check the public hiera | 16:09 |
| clarkb | timrc_: starts at https://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n838 | 16:09 |
| *** narayrak has quit IRC | 16:10 | |
| clarkb | timrc_: you can see the list of gearman workers that zuul opens the firewall to. And then below that the site.pp defs for the launchers themselves | 16:10 |
| fungi | hrm, it's not where i was expecting | 16:10 |
| *** yaume_ has quit IRC | 16:10 | |
| clarkb | fungi: its not in hiera its hardcoded in a list in site.pp | 16:10 |
| clarkb | at least for the firewall rules | 16:10 |
| *** esikachev has quit IRC | 16:10 | |
| fungi | yep, for some reason i was thinking we had them in a hiera key too for something else | 16:10 |
| fungi | but apparently not | 16:11 |
| fungi | that's just te static nodes | 16:11 |
| anteaya | dhellmann: thanks for your patience Doug | 16:11 |
| clarkb | pabelanger: I also want to say that beacuse there is the tarball and we also copy the logs individusally that all the tripleo jobs copy their logs twice | 16:11 |
| clarkb | pabelanger: might be nice to only copy them once | 16:12 |
| *** Swami has joined #openstack-infra | 16:12 | |
| clarkb | though I don't know how big of a dent that would make on the storage situation | 16:12 |
| adrian_otto | Good morning infra team. The magnum team is having considerable problems with all reviews in CI. Basically it looks like nothing has passed gate tests due to apparent timeout issues. Example: http://logs.openstack.org/06/374906/7/check/gate-functional-dsvm-magnum-api/018e58c/console.html#_2016-09-29_13_19_21_200204 | 16:12 |
| pabelanger | clarkb: okay, so I see some issues with tripleo, they are including git repos in logs files. EG: puppet modules | 16:12 |
| *** jordanP has joined #openstack-infra | 16:12 | |
| dhellmann | anteaya : np, the process keeps changing every cycle... | 16:12 |
| adrian_otto | has anything changed with respect to gate job timeout settings recently. | 16:12 |
| pabelanger | clarkb: that explains some large files | 16:12 |
| anteaya | dhellmann: why stay the same? | 16:12 |
| anteaya | :) | 16:12 |
| anteaya | you trendsetter you | 16:12 |
| clarkb | pabelanger: oh interesting | 16:12 |
| clarkb | adrian_otto: no, and tahts not a timeout you have linked to | 16:13 |
| adrian_otto | clarkb: oh? What's going on there? | 16:13 |
| clarkb | adrian_otto: tahts just the helpful message saying "this should only take this long" | 16:13 |
| dhellmann | anteaya : I know, right? | 16:13 |
| adrian_otto | our gates normally take ~50 min | 16:14 |
| clarkb | adrian_otto: http://logs.openstack.org/06/374906/7/check/gate-functional-dsvm-magnum-api/018e58c/console.html#_2016-09-29_13_42_19_490845 says look at the main log for devstack | 16:14 |
| anteaya | dhellmann: :) | 16:14 |
| *** tiswanso has joined #openstack-infra | 16:14 | |
| timrc_ | clarkb: Ah okay, it's clear to me now. Or at least I think it's clear to me now ;) One ansible launcher is started on every host that matches the regex. | 16:14 |
| anteaya | adrian_otto: clarkb Failed to connect to bootstrap.pypa.io port 443: Connection timed out | 16:15 |
| clarkb | adrian_otto: http://logs.openstack.org/06/374906/7/check/gate-functional-dsvm-magnum-api/018e58c/logs/devstacklog.txt.gz#_2016-09-29_13_42_17_444 | 16:15 |
| *** piet_ has joined #openstack-infra | 16:15 | |
| clarkb | anteaya: yup | 16:15 |
| pabelanger | clarkb: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/common_vars.bash#n22 is the issue | 16:15 |
| *** lucas-hungry is now known as lucasagomes | 16:15 | |
| pabelanger | it is grabbing the /etc folder on the node, which contains a lot of things we install when we build DIBS | 16:16 |
| clarkb | pabelanger: need to add more excludes? | 16:16 |
| clarkb | or maybe chagne the /etc to /etc/thisthing /etc/thisotherthing etc | 16:16 |
| pabelanger | clarkb: yes or invert the logic and grab only what they care about | 16:16 |
| clarkb | ya | 16:16 |
| *** Apoorva has joined #openstack-infra | 16:16 | |
| anteaya | I'm going with bootstrap.pypa.io isn't under our maintenance | 16:17 |
| *** Apoorva_ has joined #openstack-infra | 16:17 | |
| adrian_otto | thanks anteaya | 16:17 |
| anteaya | adrian_otto: welcome | 16:17 |
| *** Apoorva_ has quit IRC | 16:17 | |
| *** Apoorva has quit IRC | 16:17 | |
| clarkb | it isn't but dstufft may know what is going on and how to fix it | 16:17 |
| *** Apoorva has joined #openstack-infra | 16:17 | |
| dstufft | huh? | 16:18 |
| clarkb | dstufft: issues downloading get-pip | 16:18 |
| *** tiswanso has quit IRC | 16:19 | |
| clarkb | dstufft: example starts at http://logs.openstack.org/06/374906/7/check/gate-functional-dsvm-magnum-api/018e58c/logs/devstacklog.txt.gz#_2016-09-29_13_37_38_907 | 16:19 |
| *** tphummel has joined #openstack-infra | 16:19 | |
| dstufft | looks like a Fastly issue, I wouldn't be able tod o anything except tell them | 16:20 |
| fungi | dstufft: clarkb: i wouldn't rule out local network issues in that environment. it tried to download it from four different (ipv4) addresses according to the log | 16:22 |
| fungi | and timed out reaching 443/tcp on each one | 16:22 |
| clarkb | or maybe fastly has blacklisted our source(s) | 16:22 |
| fungi | also possible i guess | 16:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/tripleo-ci: Update exludes list for TARCMD https://review.openstack.org/379576 | 16:22 |
| fungi | as part of our "mirror arbitrary files" effort to cover small guest images, we might consider using it to mirror that script too | 16:23 |
| *** Guest92 is now known as rushil | 16:23 | |
| fungi | and then jobs can install it from the local mirror rather than hammering the upstream cdn | 16:23 |
| *** rushil has quit IRC | 16:23 | |
| *** Guest92 has joined #openstack-infra | 16:24 | |
| *** esikachev has joined #openstack-infra | 16:25 | |
| clarkb | we already cache it | 16:25 |
| *** lifeless has quit IRC | 16:25 | |
| *** tiswanso has joined #openstack-infra | 16:26 | |
| clarkb | mordred: was fiddling with using the cahed file in dib builds in job runs yeaterday | 16:26 |
| *** ijw has joined #openstack-infra | 16:26 | |
| *** tiswanso has quit IRC | 16:26 | |
| *** lezbar has quit IRC | 16:26 | |
| *** tiswanso has joined #openstack-infra | 16:27 | |
| *** lezbar has joined #openstack-infra | 16:27 | |
| *** tuannguyen has quit IRC | 16:27 | |
| *** piet_ has quit IRC | 16:27 | |
| *** sdake_dnd is now known as sdake | 16:27 | |
| *** rbrndt has quit IRC | 16:28 | |
| *** piet_ has joined #openstack-infra | 16:28 | |
| *** tuannguyen has joined #openstack-infra | 16:28 | |
| *** baoli has joined #openstack-infra | 16:28 | |
| pabelanger | clarkb: Ya, we found an issue where devstack-gate is actually moving ~jenkins/cache to another file location, then rm -rf the original. So, the location of the cache changes depending on which job is running | 16:29 |
| clarkb | yes thats due to disk space issues | 16:29 |
| clarkb | we dont want multiple copies of files | 16:29 |
| clarkb | we could maybe use symlinks | 16:29 |
| *** matrohon has quit IRC | 16:30 | |
| *** e0ne has quit IRC | 16:30 | |
| pabelanger | ya | 16:30 |
| *** ijw has quit IRC | 16:30 | |
| *** drifterza has quit IRC | 16:31 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Create puppet-openstack-integration Gerrit group https://review.openstack.org/379583 | 16:32 |
| EmilienM | mwhahaha: ^ | 16:32 |
| *** lifeless has joined #openstack-infra | 16:32 | |
| *** rkukura has quit IRC | 16:33 | |
| fungi | clarkb: oh, great point, for very small files like that, there's no reason not to bake them into our images | 16:33 |
| *** nadya has joined #openstack-infra | 16:34 | |
| fungi | blob mirroring would be for files larger than we want to bake into images but small enough that jobs can still make use of them | 16:34 |
| *** derekh has quit IRC | 16:34 | |
| *** nadya has quit IRC | 16:34 | |
| *** Guest92 is now known as rushil | 16:34 | |
| fungi | so why are we trying to download it in that job? because it's looking for the cache in the wrong place? | 16:35 |
| clarkb | its not using the cache at all | 16:35 |
| clarkb | I think its a dib build | 16:35 |
| anteaya | mwhahaha: you are alex shultz? | 16:36 |
| *** gyee has joined #openstack-infra | 16:36 | |
| clarkb | mordred has a change to test using it with dib against nodepool | 16:36 |
| mwhahaha | anteaya: yea | 16:38 |
| anteaya | thank you | 16:38 |
| openstackgerrit | Kirill Zaitsev proposed openstack-infra/project-config: Mark murano-pkg-check ready for translations https://review.openstack.org/366420 | 16:38 |
| *** jordanP has quit IRC | 16:39 | |
| *** liusheng has quit IRC | 16:39 | |
| openstackgerrit | Kirill Zaitsev proposed openstack-infra/project-config: Mark murano-pkg-check ready for translations https://review.openstack.org/366420 | 16:39 |
| *** psachin has quit IRC | 16:40 | |
| *** liusheng has joined #openstack-infra | 16:40 | |
| *** tiswanso has quit IRC | 16:41 | |
| jlvillal | Is one preferred over the other? http://zuul.openstack.org/ vs http://status.openstack.org/zuul/ | 16:41 |
| *** rajinir has joined #openstack-infra | 16:42 | |
| anteaya | I use status.o.o/zuul myself | 16:42 |
| piet_ | Does anyone have an hour to contribute to the UX team? Danielle is running a study to investigate the usability of the searchlight plugin in horizon. | 16:43 |
| anteaya | we have a lot of zuul workers according to the graph 4K compared to the number of nodes we appear to be using at the moment 500 | 16:44 |
| *** rbrndt_ has joined #openstack-infra | 16:44 | |
| *** adrian_otto has quit IRC | 16:44 | |
| *** priteau has joined #openstack-infra | 16:44 | |
| anteaya | piet_: you might get a better response post release week | 16:46 |
| jeblair | anteaya: zuul workers is wrong, it tends to be inflated for some reason we don't know of. the other lines on that graph are correct though (jobs running / waiting) | 16:47 |
| piet_ | anteaya Unfortunately, they're going to need to take down the instance running horizon/searchlight so a bit of a fire drill | 16:47 |
| anteaya | ah the graph is wrong, thank you | 16:47 |
| anteaya | yes the other lines appear consistent with current node usage, thank you | 16:47 |
| anteaya | piet_: ah okay | 16:48 |
| openstackgerrit | Beth Elwell proposed openstack-infra/storyboard-webclient: WIP - Converting less to scss https://review.openstack.org/379595 | 16:51 |
| *** amitgandhinz has quit IRC | 16:52 | |
| *** trown is now known as trown|lunch | 16:53 | |
| *** tkelsey has joined #openstack-infra | 16:54 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config: Create puppet-openstack-integration Gerrit group https://review.openstack.org/379583 | 16:55 |
| *** esikachev has quit IRC | 16:56 | |
| *** amitgandhinz has joined #openstack-infra | 16:57 | |
| jlvillal | anteaya: thanks. That is the one I usually use. But I noticed the other by accident. | 16:57 |
| *** amotoki has joined #openstack-infra | 16:57 | |
| anteaya | jlvillal: yup, they both exist, I have never really used zuul.o.o though I am aware of it | 16:58 |
| anteaya | I believe the information is the same though the code for displaying is different | 16:58 |
| electrofelix | zaro: I've just hit a very odd error with gerrit: https://review.openstack.org/372673 actually has a newer patch uploaded that is not showing up since if I go to https://review.openstack.org/378520, which depends on the updated patch and follow a gitweb link to the patch https://review.openstack.org/gitweb?p=openstack/git-upstream.git;a=commitdiff;h=505a98b6fe4a8e75b65df001afffac41416b8a71 the review link goes to https://rev | 16:59 |
| electrofelix | seems like gerrit has accepted the commit, but failed to create/update the review reference | 16:59 |
| *** claudiub has quit IRC | 17:00 | |
| electrofelix | I'm sure I can work around it by updating the commit date to force an update, but it's a bit odd | 17:01 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add comment about proxy env pass through https://review.openstack.org/271244 | 17:01 |
| *** amotoki has quit IRC | 17:03 | |
| *** krtaylor has joined #openstack-infra | 17:03 | |
| jeblair | electrofelix: if the missing commit was uploaded around 00:00 utc, the error may have been due to our daily db backups. (a known problem that i don't believe is being worked on) | 17:04 |
| *** psilvad has joined #openstack-infra | 17:05 | |
| *** ihrachys has quit IRC | 17:05 | |
| jeblair | electrofelix: (if so, the commit will be in the git repo, but the gerrit db will be missing a reference to it) | 17:05 |
| electrofelix | jeblair: was uploaded about 15/20 minutes ago | 17:06 |
| jeblair | that may not be it then :) | 17:06 |
| asselin__ | does infra have any runbook entry to remove entries in paste.openstack.org? | 17:06 |
| *** ddieterly is now known as ddieterly[away] | 17:07 | |
| electrofelix | https://review.openstack.org/372673 certainly shows the problem, there is revision 8 and a revision 10, but no revision 9 which is the one that didn't get created... | 17:07 |
| zaro | electrofelix: wow! i don't think i've ever seen that before. | 17:08 |
| *** tuannguyen has quit IRC | 17:08 | |
| fungi | electrofelix: patchset 9 got uplaoded 15/20 minutes ago, or 10 did? | 17:09 |
| electrofelix | fungi: patchset 9 got uploaded about 15/20 minutes ago, patchset 10 about 5 minutes ago which was just a commit --amend to work around the issue by forcing the commit sha1 change | 17:10 |
| fungi | i guess ps10 was only about 8 minutes ago, so presumably you do indeed mean the missing patchset was 10/15 minutes ago | 17:10 |
| fungi | yep | 17:10 |
| fungi | definitely weird to see outside of the daily backup window | 17:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: fix afs publisher https://review.openstack.org/379609 | 17:11 |
| jeblair | pabelanger, AJaeger, fungi: ^ | 17:11 |
| fungi | but the issue arising commonly when db backups are running is presumably triggered by some issue querying/updating the db, so in theory the same issue could arise under similar circumstances outside backup time too | 17:11 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: make the announce script verbose https://review.openstack.org/379611 | 17:12 |
| jeblair | i have tested that patch locally with a slightly more realistic directory layout, and it seems to work. so i think if we land that, restart the publishers, and make a few changes to the config i'm about to write up, we should be good to try the afs docs thing again. | 17:12 |
| *** kgiusti has joined #openstack-infra | 17:12 | |
| zaro | electrofelix: not available when attempting to get PS 9 with REST eitehr | 17:13 |
| electrofelix | zaro: indeed, though the commit is available via gitweb, so it got accepted, so it's just that no change ref was created for it | 17:14 |
| zaro | electrofelix: that seems like a force push? | 17:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Add remove-prefix to afs publishers https://review.openstack.org/379615 | 17:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add service name to afs zuul-launcher sites https://review.openstack.org/379616 | 17:17 |
| electrofelix | zaro: it's not any branch in the repo, just that the commit is accessible when looked up directly so I expect it'll get gc'ed at some stage | 17:17 |
| electrofelix | zaro: I was using git-review, but I hit Ctrl-C during the first attempt to upload, so it appears that prevented Gerrit from completing something on it's side | 17:17 |
| electrofelix | in this case it's been triggered by me aborting something, but it seems like the fail behaviour is incorrect, it should either a) discard the change complete or b) add a new ref for it even if the client connection goes away | 17:18 |
| anteaya | asselin__: the workflow for removing entries in paste.o.o is pm fungi with the paste reference and reason | 17:20 |
| *** jaosorior has quit IRC | 17:20 | |
| anteaya | it is done by request only and by hand | 17:20 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Increase disk_allocation_ratio to 4 https://review.openstack.org/379618 | 17:21 |
| asselin__ | anteaya, thanks. fungi I'm asking b/c I'd like to do the same in our env. In fact, see if we can expire some old pastes. Might just be delete row in database, but thought I'd ask if it's documented. (I didn't find any docs if it is) | 17:22 |
| anteaya | ah | 17:23 |
| openstackgerrit | matthew wagoner proposed openstack-infra/shade: WIP: Add test for os_zone Ansible module https://review.openstack.org/379498 | 17:23 |
| fungi | anteaya: asselin__: well, not just me. any infra-root admin can do it but we only entertain it under extreme circumstances such as illegal or security-sensitive content | 17:23 |
| zaro | electrofelix: probably the same issue? https://bugs.chromium.org/p/gerrit/issues/detail?id=3896 | 17:23 |
| *** jpena is now known as jpena|off | 17:23 | |
| anteaya | asselin__: yes, we don't have a workflow for deleting pastes | 17:23 |
| anteaya | asselin__: since we keep them | 17:23 |
| *** tnovacik has joined #openstack-infra | 17:23 | |
| anteaya | unless as fungi says there are concerns about the content of a specific paste | 17:23 |
| *** cjloader is now known as jcloader | 17:24 | |
| anteaya | then those pastes that are deemed inappropriate are removed from the db | 17:24 |
| anteaya | on an individual basis | 17:24 |
| fungi | asselin__: i think it's just delete the row(s) from the db, but i honestly don't recall. we do it so infrequently that it's not documented and end up figuring it out from scratch eact time | 17:24 |
| fungi | each time | 17:24 |
| AJaeger | team: Could you look at this project-config change for our images, please? https://review.openstack.org/#/c/377159/ To me it seems that we're duplicating information between devstack (see depends-on) and project-config in a fragile way. Need some experts to look closer (not urgent) | 17:25 |
| asselin__ | fungi, ok thanks | 17:25 |
| *** degorenko is now known as _degorenko | 17:25 | |
| *** jcloader is now known as cjloader | 17:26 | |
| electrofelix | zaro: possibly, though that doesn't indicate whether it's concerned with the entire chain as an atomic set, or that individual changes were left in an inconsistent state that still allowed them to be picked up by a commit submitted subsequently using the broken change as a parent | 17:27 |
| fungi | electrofelix: you're dbailey-k in gerrit? i'm trying to correlate events in logs for this but its logging is very disjoint (and it looks like you were uploading a stack of changes each of which ends up being a separate connection with limited context) | 17:27 |
| electrofelix | fungi: yes, that's correct | 17:27 |
| *** jkilpatr has quit IRC | 17:27 | |
| fungi | yeah, looks like you were very active before and after this too, so i can't narrow it down to a precise timestamp unfortunately | 17:29 |
| fungi | oh, here it is maybe: | 17:29 |
| fungi | [2016-09-29 16:49:52,470 +0000] abd968e2 dbailey-k a/1054 git-receive-pack./openstack/git-upstream.git 0ms 501ms killed | 17:29 |
| electrofelix | fungi: that's probably the one | 17:30 |
| *** pvaneck has joined #openstack-infra | 17:30 | |
| electrofelix | uploading 3 commits, aborted the git-review upload, then uploaded the same set again and only the the subsequent patches updated changes as the first one was in an odd state where the commit was in the git repo backend but not in gerrit's refs | 17:30 |
| electrofelix | so it never updated the ref for the first one since it could find the SHA1 | 17:31 |
| *** ddieterly[away] is now known as ddieterly | 17:31 | |
| AJaeger | clarkb: https://review.openstack.org/379145 changes bashate jobs to use xenial everywhere - do you agree or should we do the usual split? | 17:32 |
| clarkb | AJaeger: we run bashate against d-g which is master only. So its fine for us | 17:32 |
| *** jkilpatr has joined #openstack-infra | 17:33 | |
| clarkb | devstack is probably the other big user of bashate. If they have no issues with it I think its fine | 17:33 |
| EmilienM | AJaeger: quick question on https://review.openstack.org/#/c/379583/ | 17:33 |
| * clarkb is finally feeling raedy to do real work this morning | 17:33 | |
| clarkb | pabelanger: where is that revert/template-hostname change I can go ahead and make sure it gets in | 17:34 |
| AJaeger | EmilienM: answered with link. Groups defines the launchpad bug group. | 17:36 |
| EmilienM | AJaeger: oops :) | 17:36 |
| pabelanger | clarkb: https://review.openstack.org/#/c/379506/ ready to land | 17:36 |
| EmilienM | AJaeger: can I add it back to other modules then? | 17:37 |
| AJaeger | EmilienM: sure. | 17:37 |
| *** jlanoux_ has quit IRC | 17:37 | |
| *** nadya has joined #openstack-infra | 17:37 | |
| clarkb | pabelanger: fwiw the issue you describe in that commit message shouldn't cause any problems. Openstack allows you to reuse names all you want and things get assigned uuids | 17:37 |
| clarkb | pabelanger: so say you create two images with name foo, one gets uuid 1 and the other uuid 2. Nodepool should boot using the uuid of either 1 or 2 and everything works | 17:38 |
| clarkb | I think whatever is causing us problems is likely a more subtle issue in how shade handles deduplication | 17:38 |
| pabelanger | ah | 17:38 |
| clarkb | liek maybe it doesn't check the state of the image when it decides it a duplicate and just uses the uuid given | 17:38 |
| openstackgerrit | Merged openstack-infra/project-config: Run neutron-linuxbridge job on nova libvirt/vif.py changes https://review.openstack.org/371065 | 17:39 |
| clarkb | so you have a SAVING image that nodepool is trying to use then as soon as it finishes saving it works again (which I think we saw behavior like that yesterday where boots didn't work for a short period of time then started working) | 17:39 |
| EmilienM | AJaeger: in fact it's good, we have https://launchpad.net/puppet-openstack-integration | 17:40 |
| AJaeger | EmilienM: and are you using that for reporting bugs? | 17:40 |
| clarkb | pabelanger: I will leave a comment with a tl'dr and approve the change | 17:41 |
| EmilienM | AJaeger: yes | 17:41 |
| AJaeger | ok. | 17:41 |
| AJaeger | then the change is fine - just the commit message is wrong. Please update it to explain what's going on. | 17:42 |
| EmilienM | AJaeger: sorry for confusion /me facepalm | 17:42 |
| EmilienM | ok | 17:42 |
| *** tiswanso has joined #openstack-infra | 17:42 | |
| AJaeger | no worries, EmilienM | 17:42 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Create puppet-openstack-integration Gerrit group https://review.openstack.org/379583 | 17:42 |
| AJaeger | EmilienM: you reverted the change for the ACL file, it will not pass the gate now... | 17:43 |
| *** tnovacik has quit IRC | 17:43 | |
| AJaeger | Best to "git review -d 379583,2" and then amend it | 17:44 |
| anteaya | EmilienM: the normalization test cares what order the gerrit groups are in in the acl file | 17:44 |
| zaro | electrofelix: maybe create a bug but for that? repo case using `push origin HEAD:refs/for/master` instead of git-review? | 17:44 |
| EmilienM | ah indeed | 17:45 |
| anteaya | EmilienM: this is what mwhahaha's second patch fixed | 17:45 |
| clarkb | pabelanger: nevermind found an issue with the change itself so I -1'd | 17:45 |
| EmilienM | I missed it :) | 17:45 |
| *** tnovacik has joined #openstack-infra | 17:45 | |
| EmilienM | thanks Alex :P | 17:45 |
| *** inc0_ has joined #openstack-infra | 17:45 | |
| pabelanger | clarkb: okay, looking | 17:45 |
| jeblair | pabelanger: can you review https://review.openstack.org/379616 please? | 17:45 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Create puppet-openstack-integration Gerrit group https://review.openstack.org/379583 | 17:46 |
| pabelanger | jeblair: I think we might need a puppet change too | 17:46 |
| jeblair | i'd like to get 379609, 379615, and 379616 all in asap so we can eval the afs docs stuff today | 17:46 |
| jeblair | pabelanger: for what? | 17:46 |
| pabelanger | jeblair: for zuul.conf: http://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/templates/zuul.conf.erb#n78 | 17:47 |
| jeblair | pabelanger: for 'user'? it's already in there | 17:47 |
| pabelanger | actually | 17:47 |
| pabelanger | yes, please ignore | 17:47 |
| *** e0ne has joined #openstack-infra | 17:48 | |
| *** inc0 has quit IRC | 17:48 | |
| jeblair | mordred: would you review/approve 379609, 379615, and 379616 please? | 17:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Remove osic-cloud1 provider from nodepool https://review.openstack.org/379513 | 17:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Revert "Use the DIB image id in glance uploads" https://review.openstack.org/379506 | 17:52 |
| pabelanger | clarkb: thanks, I thought I removed them before uploading | 17:52 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add requirements check to os-api-ref https://review.openstack.org/379631 | 17:53 |
| *** nadya has quit IRC | 17:53 | |
| mordred | jeblair: on it | 17:54 |
| mordred | jeblair: done | 17:54 |
| jeblair | mordred: thx | 17:55 |
| electrofelix | zaro: will do | 17:55 |
| *** electrofelix has quit IRC | 17:55 | |
| *** trown|lunch is now known as trown | 17:56 | |
| *** lin_yang has joined #openstack-infra | 17:56 | |
| fungi | electrofelix: zaro: sorry for the delay, juggling a lot of stuff, but here's the backtrace: http://paste.openstack.org/show/583537 | 17:56 |
| *** rkukura has joined #openstack-infra | 17:56 | |
| fungi | oh, electrofelix just left | 17:56 |
| *** SumitNaiksatam has joined #openstack-infra | 17:57 | |
| *** nadya has joined #openstack-infra | 17:58 | |
| clarkb | vsaienko: jlvillal btw multinode is currently broken if using neutron + devstack on osic. This may impact the work you are doing if you see ssh failures | 17:59 |
| openstackgerrit | Merged openstack-infra/zuul: Ansible launcher: fix afs publisher https://review.openstack.org/379609 | 17:59 |
| cloudnull | clarkb: I still can't nail that totally down :( | 17:59 |
| jlvillal | clarkb: Thanks for letting us know! I have been wondering why our job is failing. | 17:59 |
| clarkb | cloudnull: its not your fault | 17:59 |
| clarkb | cloudnull: pretty sure its neutron/devstack's now | 17:59 |
| clarkb | cloudnull: https://review.openstack.org/379543 seems to get around the problem | 18:00 |
| openstackgerrit | Merged openstack-infra/project-config: Add remove-prefix to afs publishers https://review.openstack.org/379615 | 18:00 |
| *** sshnaidm is now known as sshnaidm|afk | 18:00 | |
| clarkb | long story short neutron + devstack set a route for 10.0.0.0/8 on br-ex (which is itself a bridge with vxlan connectivity to the other host on top of network IPs in 10.0.0.0/8) | 18:00 |
| clarkb | so frickler was on the right track earlier today | 18:00 |
| clarkb | problem is my workaround does not work because it causes the auto network allocation tests to fail | 18:01 |
| *** tkelsey has quit IRC | 18:01 | |
| *** neiljerram_hg has quit IRC | 18:01 | |
| clarkb | sc68cal is looking into sorting things out | 18:01 |
| mnaser | forgive me if this is a bit unrelated to -infra, bu ti feel like folks here might be useful in finding an answer... is there no current solution for jenkins + cloud builders + keystone v3 besides nodepool? | 18:01 |
| *** Naeil has quit IRC | 18:02 | |
| cloudnull | clarkb: oh. thought that was resolved a while back where we set exclude ranges in the osic? | 18:03 |
| clarkb | cloudnull: this is new :) | 18:03 |
| clarkb | cloudnull: and its totally our ends fault not osics | 18:03 |
| cloudnull | ah. fair enough | 18:03 |
| fungi | mnaser: jclouds plugin for jenkins might support keystone v3 if jclouds itself does? though we tried jclouds and it wasn't reliable at our scale at least | 18:04 |
| *** ddieterly is now known as ddieterly[away] | 18:04 | |
| cloudnull | If we need to be more accomodating (or change things up to better support something like this) let me know. | 18:04 |
| clarkb | cloudnull: so unless you want to work on making devstack + neutron work again I think you are good to ignore this for now | 18:04 |
| fungi | mnaser: granted that was several years ago, so presumably it's either improved or disappeared in that time | 18:04 |
| * cloudnull begins ignoring devstack+neutron for now :) | 18:05 | |
| clarkb | cloudnull: a "cheap" fix would be to use 192.168/16 addresses because those don't overlap with 10.0.0.0/8 but I really think we should continue to work with networks in 10.0.0.0/8 because they are super common | 18:05 |
| *** john-davidge has joined #openstack-infra | 18:06 | |
| mnaser | fungi, neither jclouds or openstack plugin have v3 support :( | 18:06 |
| mnaser | we have some customers that do a lot of CI and this kinda blocks us from going to keystone v3 .. looks like some work is being done towards v3 in the openstack plugin but its stuck - https://github.com/jenkinsci/openstack-cloud-plugin/pull/83 | 18:07 |
| fungi | clarkb: picking an unlikely /16 out of 172.16/12 seems like it would be less prone to encounter collisions in providers to me | 18:07 |
| cloudnull | clarkb: im a big fan of 172.16/12 | 18:07 |
| cloudnull | what fungi said | 18:07 |
| *** jpich has quit IRC | 18:07 | |
| fungi | i use 172.23.x.0/24 networks for my various vlans at home even | 18:08 |
| clarkb | we already use 172.16/12 for floating IPs for this reason | 18:08 |
| clarkb | currently 172.24.4.0/23 iirc | 18:08 |
| fungi | we really need a /12 worth of fips in our jobs?!? | 18:08 |
| clarkb | no | 18:08 |
| clarkb | but we don't use any of 192.168 | 18:08 |
| clarkb | so its least likely to conflict I think | 18:08 |
| clarkb | that said really this should just work | 18:09 |
| *** rbrndt_ has quit IRC | 18:09 | |
| AJaeger | clarkb: mtreinish gave a +1 on the bashate xenial change https://review.openstack.org/#/c/379145/ | 18:10 |
| clarkb | AJaeger: cool I can review and approve | 18:10 |
| openstackgerrit | Sarafraj Singh proposed openstack-infra/devstack-gate: Do not merge: Patch to test LM in nova https://review.openstack.org/378015 | 18:10 |
| *** john-davidge has quit IRC | 18:10 | |
| *** rkukura has quit IRC | 18:10 | |
| AJaeger | great, clarkb | 18:10 |
| mtreinish | AJaeger, clarkb: oh I actually just thought of one thing. Is it gonna run with py2 or py3? I don't know what the bashate py3 story is | 18:11 |
| clarkb | mtreinish: still py2 | 18:12 |
| *** Sukhdev has joined #openstack-infra | 18:12 | |
| mtreinish | ok, then it's fine | 18:12 |
| clarkb | mtreinish: since `python` is python2 on sane distros :P | 18:12 |
| mtreinish | heh, yeah it just sucks for me | 18:12 |
| openstackgerrit | Travis Truman (automagically) proposed openstack-infra/project-config: Add OSA ceilometer gnocchi functional test https://review.openstack.org/379642 | 18:13 |
| mtreinish | clarkb: I've been hit by that too many times in local testing | 18:13 |
| clarkb | mtreinish: ya i bet its really annoying :) | 18:14 |
| clarkb | mtreinish: if you use virtualenv does it sanitize that for you? | 18:14 |
| clarkb | or does it just get confused | 18:14 |
| mtreinish | yeah, thats what I do now | 18:14 |
| anteaya | is travis truman automagically a human? | 18:14 |
| clarkb | anteaya: I think thats the username? | 18:14 |
| mtreinish | I just active the venv in my default shell env | 18:14 |
| anteaya | it strikes me as an odd choice | 18:14 |
| anteaya | https://review.openstack.org/#/q/owner:%22Travis+Truman+(automagically)%22+status:open | 18:15 |
| openstackgerrit | Merged openstack-infra/project-config: Create puppet-openstack-integration Gerrit group https://review.openstack.org/379583 | 18:15 |
| cloudnull | anteaya: yup he's a person I've met him :) | 18:16 |
| *** baoli has quit IRC | 18:16 | |
| *** baoli has joined #openstack-infra | 18:17 | |
| cloudnull | he's one of our OpenStack-Ansible contributors. | 18:17 |
| clarkb | I kind of liek that username now | 18:17 |
| anteaya | okay thought he was using it to tell us he automated patch submisison | 18:18 |
| anteaya | submission* | 18:18 |
| cloudnull | which would be awesome ! :) | 18:18 |
| anteaya | or not | 18:18 |
| *** xarses has quit IRC | 18:18 | |
| anteaya | since we already have identified that as a security concern | 18:18 |
| mtreinish | clarkb: where the by default in a venv thing falls apart is when you active a second venv. (that itself works fine) The deactivate function breaks out of all the venvs | 18:19 |
| anteaya | so would be discussing disabling the account if that were the case | 18:19 |
| clarkb | mtreinish: ah | 18:19 |
| *** automagically has joined #openstack-infra | 18:19 | |
| clarkb | mtreinish: anymore I just try to use paths into venvs and avoid activation entirely | 18:19 |
| cloudnull | anteaya: rest assured automagically is a person. :) | 18:19 |
| automagically | Indeed, I’m real | 18:20 |
| automagically | And even sometimes not so magical | 18:20 |
| anteaya | automagically: hello | 18:20 |
| mtreinish | clarkb: yeah, that's what I try to do too. But it's sometimes hard to break old habits | 18:20 |
| *** woodster_ has quit IRC | 18:20 | |
| automagically | o/ anteaya | 18:20 |
| anteaya | automagically: glad you are a person, thank you for confirming | 18:20 |
| *** xarses has joined #openstack-infra | 18:20 | |
| * mtreinish thinks it's a turing test | 18:20 | |
| *** eranrom has quit IRC | 18:20 | |
| *** eranrom has joined #openstack-infra | 18:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Use xenial for bashate https://review.openstack.org/379145 | 18:21 |
| dims | AJaeger : os-api-ref is a rush order? (can't find you on #openstack-release) | 18:21 |
| clarkb | ugh I think my brain stopped working today. I just had to man symlink no thats not it, man link no thats not it, man ln aha! | 18:21 |
| *** tpsilva has joined #openstack-infra | 18:21 | |
| openstackgerrit | Mike Fedosin proposed openstack-infra/project-config: Create glare-specs project https://review.openstack.org/370749 | 18:21 |
| *** tnovacik has quit IRC | 18:21 | |
| clarkb | I can never remember if its target name or name target | 18:21 |
| anteaya | dims: are you waiting for something? | 18:22 |
| anteaya | dims: if yes, please link | 18:22 |
| dims | anteaya : nope, just asking about a release requested by AJaeger (https://review.openstack.org/#/c/379644/) | 18:22 |
| anteaya | ah | 18:22 |
| mtreinish | clarkb: heh, I think everyone has that problem | 18:22 |
| fungi | mtreinish: perhaps automagically is a mechanical turk? | 18:23 |
| *** gyee has quit IRC | 18:24 | |
| anteaya | can he protect the wiki from the things other mechanical turks send to us? | 18:24 |
| automagically | anteaya: That I cannot do | 18:24 |
| anteaya | shame | 18:24 |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/devstack-gate: Set SUBNETPOOL_PREFIX_V4 to match FIXED_RANGE https://review.openstack.org/379521 | 18:24 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Use the DIB image id in glance uploads" https://review.openstack.org/379506 | 18:25 |
| AJaeger | dims: let me come to #openstack-release | 18:25 |
| pabelanger | clarkb: fungi: Have cycles to review https://review.openstack.org/#/c/374371/ ? stops reloading openssh-server from glean | 18:25 |
| *** lseki has joined #openstack-infra | 18:26 | |
| *** claudiub has joined #openstack-infra | 18:29 | |
| *** nadya has quit IRC | 18:29 | |
| *** claudiub|2 has joined #openstack-infra | 18:31 | |
| jeblair | i used to always fail my turing tests in school | 18:32 |
| *** claudiub has quit IRC | 18:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Import to_* from module_utils.basic https://review.openstack.org/379652 | 18:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Work with ansible 2.1 https://review.openstack.org/379653 | 18:34 |
| mordred | jeblair: ^^ | 18:35 |
| mordred | jeblair: the second one is the important one | 18:35 |
| mordred | (the work with ansible 2.1) | 18:35 |
| jeblair | mordred: 'work' is kind of a giveaway keyword | 18:35 |
| mordred | jeblair: :) | 18:35 |
| *** xarses has quit IRC | 18:36 | |
| *** dtardivel has quit IRC | 18:37 | |
| *** xarses has joined #openstack-infra | 18:38 | |
| *** amitgandhinz has quit IRC | 18:39 | |
| *** amitgandhinz has joined #openstack-infra | 18:40 | |
| *** spzala has quit IRC | 18:40 | |
| *** xarses has quit IRC | 18:40 | |
| *** spzala has joined #openstack-infra | 18:41 | |
| *** xarses has joined #openstack-infra | 18:42 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add imageUploadLock() API https://review.openstack.org/378983 | 18:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Work with ansible 2.1 https://review.openstack.org/379653 | 18:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Import to_* from module_utils.basic https://review.openstack.org/379652 | 18:43 |
| *** kzaitsev_mb has quit IRC | 18:43 | |
| *** mhickey has joined #openstack-infra | 18:45 | |
| *** ddieterly[away] is now known as ddieterly | 18:46 | |
| *** spzala has quit IRC | 18:46 | |
| *** rbrndt_ has joined #openstack-infra | 18:48 | |
| *** ddieterly is now known as ddieterly[away] | 18:49 | |
| *** sdake has quit IRC | 18:50 | |
| *** sdake has joined #openstack-infra | 18:50 | |
| *** esikachev has joined #openstack-infra | 18:51 | |
| *** ihrachys has joined #openstack-infra | 18:51 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/irc-meetings: arch_wg: alt time for APAC friendly meetings https://review.openstack.org/379768 | 18:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Revert "Work with ansible 2.1" https://review.openstack.org/379769 | 18:53 |
| *** Rockyg has joined #openstack-infra | 18:53 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Run gate-tempest-dsvm-full-devstack-plugin-nfs-nv on xenial https://review.openstack.org/379770 | 18:53 |
| mriedem | eharney: ^ | 18:53 |
| *** krtaylor has quit IRC | 18:54 | |
| *** ddieterly[away] is now known as ddieterly | 18:55 | |
| eharney | mriedem: excellent | 18:55 |
| mriedem | i'm not sure how the nfs job passed before that | 18:55 |
| mriedem | with snapshots at least | 18:55 |
| eharney | mriedem: also see https://review.openstack.org/#/c/369519/ | 18:55 |
| *** ihrachys has quit IRC | 18:55 | |
| eharney | which includes the same change | 18:55 |
| *** salv-orlando has quit IRC | 18:56 | |
| mriedem | eharney: -1 on that one | 18:56 |
| mriedem | the depends-on are backward | 18:56 |
| lseki | hello guys, could someone help me please? | 18:58 |
| lseki | I have a Jenkins build that is triggered by Zuul, but the Zuul parameters are not passed to the build | 18:58 |
| clarkb | lseki: thats likely due to the newish jenkins security thing to prevent plugins from passing parameters | 18:58 |
| clarkb | lseki: you can either add the params to your build or whitelist the gearman plugin iirc | 18:59 |
| *** rbrndt_ has quit IRC | 18:59 | |
| lseki | clarkb: oh thanks, I'll try it! | 19:00 |
| *** yamahata has quit IRC | 19:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Don't specify an executable if it's not needed https://review.openstack.org/379439 | 19:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 19:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Use command module instead of zuul_runner https://review.openstack.org/369604 | 19:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files https://review.openstack.org/369605 | 19:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async https://review.openstack.org/371733 | 19:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Ansible launcher: Add missing import https://review.openstack.org/379544 | 19:01 |
| *** cardeois has quit IRC | 19:02 | |
| *** Goneri has quit IRC | 19:03 | |
| *** cardeois has joined #openstack-infra | 19:03 | |
| mat128 | lseki, clarkb: https://wiki.jenkins-ci.org/display/JENKINS/Plugins+affected+by+fix+for+SECURITY-170 | 19:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async https://review.openstack.org/371733 | 19:04 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add imageUploadLock() API https://review.openstack.org/378983 | 19:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Don't specify an executable if it's not needed https://review.openstack.org/379439 | 19:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 19:04 |
| mat128 | https://jenkins.io/blog/2016/05/11/security-update/ | 19:04 |
| mordred | jeblair: ^^ took the fix patches and squashed them into the appropriate earlier patches | 19:04 |
| *** rkukura has joined #openstack-infra | 19:05 | |
| *** cardeois has quit IRC | 19:06 | |
| *** nadya has joined #openstack-infra | 19:06 | |
| *** apetrich has quit IRC | 19:08 | |
| *** baoli has quit IRC | 19:08 | |
| *** baoli has joined #openstack-infra | 19:08 | |
| *** esberglu_ has joined #openstack-infra | 19:09 | |
| *** rbrndt has joined #openstack-infra | 19:09 | |
| fungi | pabelanger: greghaynes: i approved 374371 but presumably we need a dib release containing the dependency change before we tag a glean release with that right? | 19:09 |
| *** apetrich has joined #openstack-infra | 19:10 | |
| *** esberglu has quit IRC | 19:10 | |
| *** esberglu_ is now known as esberglu | 19:10 | |
| pabelanger | fungi: greghaynes was kinda enough to tag 1.20.0 the night before | 19:11 |
| pabelanger | so we are ready to also tag glean | 19:11 |
| *** psilvad has quit IRC | 19:11 | |
| pabelanger | I'd actually like to get diskimage-builder 1.20.1 including: https://review.openstack.org/#/c/378985/ | 19:11 |
| pabelanger | so fix some keyboard input issues with ssh-keygen -A | 19:12 |
| * AJaeger waves good night | 19:12 | |
| cloudnull | o/ AJaeger | 19:12 |
| pabelanger | AJaeger: enjoy the apple strudels | 19:12 |
| anteaya | good night AJaeger | 19:13 |
| *** salv-orlando has joined #openstack-infra | 19:18 | |
| cinerama | pabelanger, you might want to bring that up in the meeting and/or channel | 19:19 |
| lseki | clarkb, mat128 : It worked! thank you!!! | 19:19 |
| *** SumitNaiksatam has left #openstack-infra | 19:20 | |
| openstackgerrit | Merged openstack-infra/glean: Stop managing openssh-server from glean https://review.openstack.org/374371 | 19:20 |
| *** bhavik1 has quit IRC | 19:23 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack-infra/project-config: Add OSA ceilometer gnocchi functional test https://review.openstack.org/379642 | 19:26 |
| *** adrian_otto has joined #openstack-infra | 19:29 | |
| *** esikachev has quit IRC | 19:30 | |
| *** Apoorva has quit IRC | 19:30 | |
| *** ihrachys has joined #openstack-infra | 19:31 | |
| *** vsaienko has quit IRC | 19:34 | |
| *** vsaienko has joined #openstack-infra | 19:34 | |
| *** matrohon has joined #openstack-infra | 19:36 | |
| *** asselin_ has joined #openstack-infra | 19:38 | |
| *** pradiprwt_ has joined #openstack-infra | 19:39 | |
| *** pradiprwt has quit IRC | 19:39 | |
| *** yarkot has quit IRC | 19:39 | |
| *** thiagolib has quit IRC | 19:39 | |
| *** tchaypo has quit IRC | 19:39 | |
| *** purp_too has quit IRC | 19:39 | |
| *** jamesden_ has quit IRC | 19:39 | |
| *** pradiprwt_ is now known as pradiprwt | 19:39 | |
| *** purp has joined #openstack-infra | 19:40 | |
| *** jamesden_ has joined #openstack-infra | 19:40 | |
| *** baoli has quit IRC | 19:40 | |
| *** tchaypo has joined #openstack-infra | 19:41 | |
| *** baoli has joined #openstack-infra | 19:41 | |
| *** thiagolib has joined #openstack-infra | 19:41 | |
| *** Goneri has joined #openstack-infra | 19:42 | |
| *** yarkot has joined #openstack-infra | 19:42 | |
| *** zxiiro_ has joined #openstack-infra | 19:42 | |
| *** cloudnull has quit IRC | 19:43 | |
| *** verdurin has quit IRC | 19:43 | |
| *** asselin__ has quit IRC | 19:43 | |
| *** zxiiro has quit IRC | 19:43 | |
| *** scottda has quit IRC | 19:43 | |
| *** flaper87 has quit IRC | 19:43 | |
| *** khappone_ has quit IRC | 19:43 | |
| *** zhenguo has quit IRC | 19:43 | |
| *** briancli1e has quit IRC | 19:43 | |
| *** tdurakov has quit IRC | 19:43 | |
| *** ashtokolov has quit IRC | 19:43 | |
| *** cloudnull has joined #openstack-infra | 19:43 | |
| *** lindsayk_ has quit IRC | 19:43 | |
| *** khappone has joined #openstack-infra | 19:43 | |
| *** salv-orl_ has joined #openstack-infra | 19:44 | |
| *** zxiiro_ is now known as zxiiro | 19:44 | |
| *** sjmc7 has joined #openstack-infra | 19:45 | |
| *** tdurakov has joined #openstack-infra | 19:45 | |
| *** toabctl_ has joined #openstack-infra | 19:46 | |
| *** scottda has joined #openstack-infra | 19:46 | |
| *** ashtokolov has joined #openstack-infra | 19:46 | |
| *** paul_ has joined #openstack-infra | 19:46 | |
| *** salv-orlando has quit IRC | 19:47 | |
| *** vsaienko has quit IRC | 19:47 | |
| *** cardeois has joined #openstack-infra | 19:47 | |
| *** sulo_ has joined #openstack-infra | 19:47 | |
| *** amitgandhinz has quit IRC | 19:48 | |
| rcarrillocruz | in case you didnd't see, ansible 2.1.2.0 has been released | 19:48 |
| *** NobodyCam_ has joined #openstack-infra | 19:48 | |
| *** amitgandhinz has joined #openstack-infra | 19:48 | |
| sjmc7 | hi folks. i had a group created in gerrit last week, searchlight-ui-core - could someone add me as the initial member so i can populate it please? https://review.openstack.org/#/admin/groups/1584,members | 19:49 |
| *** paul___ has quit IRC | 19:50 | |
| *** katyafervent_awa has quit IRC | 19:50 | |
| *** toabctl has quit IRC | 19:50 | |
| *** Adri2000 has quit IRC | 19:50 | |
| *** samueldmq has quit IRC | 19:50 | |
| *** briancline has joined #openstack-infra | 19:50 | |
| *** NobodyCam has quit IRC | 19:50 | |
| *** AJaeger has quit IRC | 19:50 | |
| *** sulo has quit IRC | 19:50 | |
| *** nt has quit IRC | 19:50 | |
| *** toabctl_ is now known as toabctl | 19:50 | |
| *** Mmike has quit IRC | 19:50 | |
| rcarrillocruz | i'm upgrading the requirements on the launcher role | 19:50 |
| *** Mmike has joined #openstack-infra | 19:50 | |
| *** NobodyCam_ is now known as NobodyCam | 19:50 | |
| *** samueldmq has joined #openstack-infra | 19:50 | |
| mtreinish | rcarrillocruz, pleia2, clarkb, fungi, jeblair, mordred, nibalizer: if you get a sec: https://review.openstack.org/#/q/status:open+topic:mosquitto-redux should all be fairly quick | 19:50 |
| clarkb | its like a matrix movie. mosquitto-reloaded | 19:51 |
| *** katyafervent_awa has joined #openstack-infra | 19:51 | |
| pabelanger | We really should bump ansible on puppetmaster again, I think we are still pinned at 2.0.0.2 | 19:51 |
| fungi | pabelanger: and that version doesn't work with the launch-node.py script | 19:51 |
| fungi | so yes, if newer ansible works for the other things puppetmaster does, then i'm all for it | 19:52 |
| openstackgerrit | Jens Rosenboom proposed openstack-infra/devstack-gate: Set SUBNETPOOL_PREFIX_V4 to match FIXED_RANGE https://review.openstack.org/379521 | 19:52 |
| mtreinish | clarkb: heh, yeah a bit out of order though | 19:52 |
| *** timrc_ is now known as timrc | 19:52 | |
| *** verdurin has joined #openstack-infra | 19:52 | |
| mordred | pabelanger: so - 2.2 is in RC ... | 19:52 |
| timrc | You never know what breaks when ansible bumps a version. | 19:52 |
| pabelanger | fungi: ya, it would be good to get a list of the failures together and see what we need to do to fix them | 19:52 |
| mordred | pabelanger: maybe we wait for 2.2 and then do our vetting of "are we sure that it works" for that | 19:52 |
| *** nt has joined #openstack-infra | 19:52 | |
| rcarrillocruz | i'm all for upgrading ansible in puppetmater | 19:52 |
| mordred | rather than doing it now to get on 2.1 and then in another week to get on 2.2 | 19:52 |
| rcarrillocruz | cos 2.0.0.2 does not work for the launcher | 19:52 |
| fungi | pabelanger: several of us have been running launch-node.py out of virtualenvs with newer ansible in them to get around the old ansible on puppetmaster.o.o until we've heard it's safe for us to upgrade | 19:52 |
| rcarrillocruz | as it relies on looping features introduced on 2.1.x | 19:52 |
| *** zhenguo has joined #openstack-infra | 19:53 | |
| fungi | timrc: who knows what bugs lurk in the heart of an ansible release? the shadow knows.... bwahahahahah | 19:53 |
| *** artmr has quit IRC | 19:53 | |
| *** lucasagomes is now known as lucas-afk | 19:54 | |
| *** woodster_ has joined #openstack-infra | 19:54 | |
| pabelanger | mordred: Ya, if we don't think 2.0.0.2 to 2.2.0.0 is too bad | 19:55 |
| mordred | pabelanger: it shouldn't be ... we're not actually very advanced ansible users ... but also 2.2 should have the fix for the bug clarkb submitted | 19:55 |
| clarkb | mordred: the inventory issue? | 19:56 |
| clarkb | I saw that there was a thing that merged but no communication of what it does. Didn't look like a revert and instaed we might have to toggle configuration? | 19:56 |
| *** cardeois_ has joined #openstack-infra | 19:56 | |
| *** ihrachys has quit IRC | 19:56 | |
| *** spzala_ has joined #openstack-infra | 19:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Don't specify an executable if it's not needed https://review.openstack.org/379439 | 19:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 19:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Use command module instead of zuul_runner https://review.openstack.org/369604 | 19:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Put script string in directly instead of in files https://review.openstack.org/369605 | 19:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add async action plugin to override upstream async https://review.openstack.org/371733 | 19:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Rename callback_plugins dir to callback https://review.openstack.org/371887 | 19:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Rename zuul_runner to command https://review.openstack.org/369603 | 19:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: fix 'when' https://review.openstack.org/379792 | 19:57 |
| *** vsaienko has joined #openstack-infra | 19:58 | |
| *** cardeois has quit IRC | 19:58 | |
| *** lin_yang has quit IRC | 19:59 | |
| *** tchaypo has quit IRC | 19:59 | |
| *** lin_yang has joined #openstack-infra | 20:00 | |
| *** tchaypo has joined #openstack-infra | 20:00 | |
| *** nherciu_ has quit IRC | 20:00 | |
| *** spzala_ has quit IRC | 20:01 | |
| *** spzala has joined #openstack-infra | 20:03 | |
| timrc | fungi: I had a crazy one just recently... the ansible apt_repository module was computing the octal value of 0644 and then (accidentally) computing it again, so files it was writing to /etc/apt/sources.list.d/ were basically getting chmod 420'd... which apt-cache does not like (apt-cache is way to straight-edge) | 20:04 |
| fungi | wow | 20:05 |
| mordred | timrc: hahaha | 20:05 |
| fungi | otc(oct(mode))? | 20:05 |
| fungi | that's fun | 20:05 |
| timrc | Yep... only if you allowed apt_repository to use the "default". If you explicitly told it to set permissions, it behaved correctly. | 20:06 |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Disable mqtt output plugin on logstash workers" https://review.openstack.org/377130 | 20:06 |
| fungi | and why carrying explicit base prefixes all the way through to the last step is useful | 20:06 |
| fungi | that way you can make your conversion functions idempotent | 20:07 |
| *** Adri2000 has joined #openstack-infra | 20:08 | |
| *** sdake has quit IRC | 20:08 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: [WIP] Add provider upload framework https://review.openstack.org/379795 | 20:09 |
| mordred | fungi: we could just ditch all these encodings and stick to pure binary everywhere | 20:10 |
| *** yaume has quit IRC | 20:10 | |
| fungi | binary is twice as complicated as we need really, we could just switch to unary numbers | 20:10 |
| mordred | ++ | 20:10 |
| timrc | lolol | 20:10 |
| Shrews | mordred: 00101011 00101011 | 20:11 |
| *** ldnunes has quit IRC | 20:11 | |
| fungi | wow, and here i thought i was making a joke https://en.wikipedia.org/wiki/Unary_numeral_system | 20:11 |
| *** ddieterly is now known as ddieterly[away] | 20:11 | |
| openstackgerrit | Chris Dent proposed openstack-infra/devstack-gate: DNM Test using the optional placement db in nova https://review.openstack.org/379797 | 20:12 |
| mtreinish | fungi: do you have any suggestions for how to build libwebsockets and mosquitto from source on firehose (well besides doing it by hand, which I assume we didn't want to do) | 20:12 |
| clarkb | fungi: its like counting to 10 on your fingers | 20:12 |
| fungi | mtreinish: well, you could rebuild the debs for them and upload them into an ubuntu ppa | 20:13 |
| fungi | mtreinish: what releases of libwebsockets and mosquitto do you suggest? | 20:13 |
| *** yamahata has joined #openstack-infra | 20:13 | |
| *** AJaeger has joined #openstack-infra | 20:13 | |
| mordred | fungi: I just learned WAY too much about unary | 20:14 |
| timrc | fungi: So actually they aren't using oct but int(str, base) base=8 and then feeding that to chmod somewhere down the road. | 20:14 |
| fungi | mordred: you and me both. and here i thought i paid attention in number theory | 20:14 |
| mtreinish | fungi: well it just needs to be a newer libwebsockets. I tested the latest locally 2.0.3 and it didn't blow up | 20:14 |
| *** ldnunes has joined #openstack-infra | 20:14 | |
| mtreinish | fungi: I didn't bother doing a bisect to figure out the min version that fixes it thoguh | 20:14 |
| timrc | os.chmod^ | 20:14 |
| *** piet_ has quit IRC | 20:14 | |
| mordred | fungi: I know I didn't pay attention - I was too busy in my voice and diction class doing alexander exercises | 20:15 |
| mtreinish | the problem is you need to recompile mosquitto to use the newer version of libwesockets | 20:15 |
| fungi | mtreinish: yeah, which is probably generally easy from a package build perspective | 20:15 |
| mtreinish | so we could reuse 1.4.8 or use the latest version 1.4.10 | 20:15 |
| *** kgiusti has left #openstack-infra | 20:15 | |
| mtreinish | sure, I guess that's what I'll do then | 20:16 |
| mtreinish | heh, I wonder how common it is to build deb packages on an arch box? :) | 20:16 |
| *** ldnunes has quit IRC | 20:17 | |
| timrc | better than building them on centos boxen.. maybe? | 20:17 |
| mtreinish | heh, probably | 20:17 |
| fungi | mtreinish: ubuntu has libwebsockets7 1.7.1 in xenial but 1.7.3 available in yakkety we could probably drag into a xenial system to rebuild mosquitto against. debian has 1.7.3 in testing and unstable, and 2.0.2 in experimental. so if we need to forward port the packaging it's probably already done | 20:17 |
| fungi | mtreinish: i'd use something like pdebuild or just debootstrap a chroot and then use debuild inside that | 20:18 |
| fungi | mtreinish: what version of libwebsockets were you testing with? | 20:19 |
| mtreinish | fungi: yeah that probably would work. BUt, I'd probably just spin up a xenial vm and do it in there | 20:19 |
| fungi | likely we could rebuild the debian unstable package of 2.0.2 on xenial trivially, and then rebuild the mosquitto package from xenial but just bump the package rev | 20:19 |
| mtreinish | fungi: I was using 2.0.3 | 20:20 |
| mtreinish | which was the latest tag in the git repo | 20:20 |
| fungi | chances are the debian 2.0.2 package can be updated to 2.0.3 unless a lot changed about teh structure of the repo or its built toolchain | 20:20 |
| mtreinish | I imagine 2.0.2 is probably fine too | 20:20 |
| mtreinish | fungi: I'm a bit confused: https://github.com/warmcat/libwebsockets/releases (no 2.0.2) | 20:21 |
| fungi | if a watch file is set up in the package, might be able to just uupdate and rebuild even | 20:21 |
| *** baoli has quit IRC | 20:21 | |
| fungi | mtreinish: dunno, http://metadata.ftp-master.debian.org/changelogs//main/libw/libwebsockets/libwebsockets_2.0.2-1_changelog doesn't say anything about making that version number up | 20:22 |
| fungi | but maybe the maintainer did | 20:22 |
| *** baoli has joined #openstack-infra | 20:22 | |
| mtreinish | hmm: https://github.com/warmcat/libwebsockets/commit/b064fb5bc9f21416a98c424b347cee141ef9df6e | 20:22 |
| mtreinish | I guess they just didn't tag it | 20:22 |
| mtreinish | its also mentioned in the changelog on the git repo | 20:23 |
| openstackgerrit | Tim Burke proposed openstack-infra/meetbot: Make #links links https://review.openstack.org/379805 | 20:23 |
| fungi | it's _possible_ the latest mosquitto packages might need some tweaking to handle the libsebsockets8 transition, but that's probably just a one-liner edit | 20:23 |
| mtreinish | fungi: yeah probably, I'd have to take a more detailed look at how they setup the package to be sure | 20:25 |
| mtreinish | but yeah, it'd be a one liner if there's an issue | 20:25 |
| fungi | i mean, presumably if there _were_ an actual problem building mosquitto against libwebsockets8 you would have run into it | 20:25 |
| fungi | so at worst it'll just be tweaking package metadata for dependencies | 20:26 |
| mtreinish | fungi: oh, I assumed you were talking about the metadata :) | 20:26 |
| fungi | i was | 20:26 |
| fungi | just clarifying why i was only talking about the metadat | 20:26 |
| fungi | a | 20:26 |
| cloudnull | clarkb: pabelanger: -qq are these error node lauch attempts related to any known issue? -- http://grafana.openstack.org/dashboard/db/nodepool-osic?panelId=11&fullscreen -- I know about the multi-node devstack gate issue but is there something else that is causing those? | 20:30 |
| fungi | anyway, i've got to go deal with the yard. bbiaw | 20:30 |
| pabelanger | fungi: clarkb: mordred: anybody mind tagging glean 1.8.0 (2e6ba9)? Or me doing it? | 20:31 |
| clarkb | cloudnull: I thunk its the image issue pabelanger ran inyo so we are just going to stop deduping images for now | 20:31 |
| pabelanger | cloudnull: known, yes. Hoping our soon to be tagged glean release will finally fix things | 20:31 |
| cloudnull | ah. ok. | 20:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Ansible launcher: fix environment quoting https://review.openstack.org/379812 | 20:33 |
| *** lezbar has quit IRC | 20:33 | |
| mordred | pabelanger: I'm fine with you doing it | 20:34 |
| *** lezbar has joined #openstack-infra | 20:34 | |
| *** jkilpatr has quit IRC | 20:34 | |
| *** Jeffrey4l has quit IRC | 20:35 | |
| *** Jeffrey4l has joined #openstack-infra | 20:35 | |
| mtreinish | wow, mqtt on lostash is working finally | 20:35 |
| mtreinish | it's also a lot of messages :) | 20:36 |
| *** manfred has joined #openstack-infra | 20:36 | |
| *** gtmanfred has quit IRC | 20:36 | |
| *** cburgess_ has quit IRC | 20:36 | |
| *** manfred is now known as gtmanfred | 20:36 | |
| *** nadya has quit IRC | 20:37 | |
| *** lindsayk_ has joined #openstack-infra | 20:37 | |
| *** edmondsw has quit IRC | 20:37 | |
| *** tonytan4ever has quit IRC | 20:38 | |
| *** smarcet1981 has quit IRC | 20:38 | |
| *** cburgess has joined #openstack-infra | 20:38 | |
| *** ddieterly[away] is now known as ddieterly | 20:40 | |
| johnsom | mordred On the get-pip.py patch. I think we need to pass it in as a file:// URL | 20:40 |
| johnsom | I'm testing that now | 20:40 |
| *** lin_yang has quit IRC | 20:40 | |
| jesusaur | mtreinish: awesome | 20:42 |
| *** baoli has quit IRC | 20:42 | |
| *** sdake has joined #openstack-infra | 20:43 | |
| mtreinish | jesusaur: I need to write a little scipt now to see how many messages per second we're pushing at the firehose right now :) | 20:43 |
| *** spzala has quit IRC | 20:43 | |
| *** ldnunes has joined #openstack-infra | 20:43 | |
| *** spzala has joined #openstack-infra | 20:43 | |
| *** baoli has joined #openstack-infra | 20:44 | |
| *** melwitt has quit IRC | 20:44 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 20:45 |
| clarkb | mtreinish: we burst to 10k indexing per second iirc | 20:45 |
| clarkb | mtreinish: since mqtt doesnt index we likely qctually have higher throughput in total | 20:45 |
| *** briancline has quit IRC | 20:46 | |
| *** jamesdenton has quit IRC | 20:46 | |
| mtreinish | wow, thats a higher than I was expecting | 20:46 |
| *** trown is now known as trown|outtypewww | 20:47 | |
| mtreinish | but I guess it makes sense, we have a lot of logs | 20:47 |
| *** sdake has quit IRC | 20:48 | |
| *** annegentle has quit IRC | 20:48 | |
| *** spzala has quit IRC | 20:48 | |
| *** spzala has joined #openstack-infra | 20:49 | |
| jeblair | mordred, fungi, clarkb, pabelanger: i'm ready to restart zuul launchers for the afs fix. i can do a graceful or hard restart. graceful means 10 hours, hard means instant. hard will terminate all running jobs, however, they will all be auto-rescheduled. i lean toward hard since there aren't any release critical activities at the moment, the gate is not that deep, and we are well under capacity. thoughts? | 20:49 |
| clarkb | jeblair: I actually have a change in a restart job loop (top of check) that I need to figure out, but I think that means graceful won't work unless I force it out some other way | 20:50 |
| clarkb | jeblair: tl;dr hard is probably ok double so because ^ | 20:50 |
| jeblair | clarkb: indeed :) | 20:51 |
| *** melwitt has joined #openstack-infra | 20:51 | |
| pabelanger | sure, curious to see how nodepool handles it :) | 20:52 |
| *** lezbar has quit IRC | 20:52 | |
| *** melwitt is now known as Guest99228 | 20:52 | |
| *** lezbar has joined #openstack-infra | 20:52 | |
| *** amoralej|off has quit IRC | 20:53 | |
| *** briancline has joined #openstack-infra | 20:53 | |
| *** jesusaur has quit IRC | 20:53 | |
| *** baoli has quit IRC | 20:53 | |
| openstackgerrit | Chris Dent proposed openstack-infra/devstack-gate: DNM Test using the optional placement db in nova https://review.openstack.org/379797 | 20:53 |
| mordred | jeblair: hard ++ | 20:54 |
| mordred | oh yeah - a hard restart is a nice nodepool slam :) | 20:54 |
| *** baoli has joined #openstack-infra | 20:54 | |
| *** cindy has joined #openstack-infra | 20:54 | |
| *** spzala has quit IRC | 20:54 | |
| *** jesusaur has joined #openstack-infra | 20:54 | |
| *** amoralej has joined #openstack-infra | 20:55 | |
| *** panda is now known as panda|Za | 20:55 | |
| *** vsaienko has quit IRC | 20:56 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add hard-restart zuul-launcher playbook https://review.openstack.org/379823 | 20:56 |
| *** panda|Za is now known as panda|zZ | 20:57 | |
| *** baoli has quit IRC | 20:57 | |
| *** spzala has joined #openstack-infra | 20:57 | |
| *** ijw has joined #openstack-infra | 20:58 | |
| *** baoli has joined #openstack-infra | 20:58 | |
| *** dangers_away has quit IRC | 20:59 | |
| *** dangers has joined #openstack-infra | 20:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Replace the SUCCESS and FAILURE https://review.openstack.org/379824 | 21:00 |
| *** jpena|off has quit IRC | 21:00 | |
| *** matt-borland has quit IRC | 21:00 | |
| mordred | jeblair: yah - you were right ^^ I was overzealous in my removal of those when I was doing the if success patch | 21:00 |
| jeblair | i'm restarting zuul now | 21:00 |
| *** amotoki has joined #openstack-infra | 21:00 | |
| jeblair | er launchers | 21:00 |
| *** jkilpatr has joined #openstack-infra | 21:01 | |
| *** jpena|off has joined #openstack-infra | 21:02 | |
| *** raildo has quit IRC | 21:02 | |
| *** spzala has quit IRC | 21:02 | |
| *** mhickey has quit IRC | 21:02 | |
| fungi | jeblair: sounds good | 21:02 |
| *** e0ne has quit IRC | 21:05 | |
| *** anteaya has quit IRC | 21:05 | |
| *** Apoorva has joined #openstack-infra | 21:05 | |
| *** amotoki has quit IRC | 21:05 | |
| *** inc0_ has quit IRC | 21:07 | |
| *** spzala has joined #openstack-infra | 21:07 | |
| *** caphrim007 has quit IRC | 21:07 | |
| *** ijw has quit IRC | 21:08 | |
| jeblair | for some reason that playbook did not succeed in starting the launchers; i have run another playbook to do that | 21:08 |
| jeblair | they are running now | 21:08 |
| *** e0ne has joined #openstack-infra | 21:08 | |
| mgagne | why are some projects using 'gate-{name}-python27-db-{node}' and other 'gate-{name}-python27-{node}' ? Have some projects found that unit testing without database is less "accurate"? Why choose one over the other? | 21:09 |
| clarkb | mgagne: some projects want access to DBs that are preconfigured. Others don't need this | 21:09 |
| clarkb | thats really the only difference, whether or not the job will set up a pre canned set of DBs and users and schemas | 21:09 |
| mgagne | so it's up to the project to choose what it needs. -db happens to have mysql preinstalled/configured for them? | 21:09 |
| fungi | mgagne: the -db variants run a few additional builder macros that precondifure databases for mysql and postgres and grant full access to add/drop for non-root users | 21:09 |
| clarkb | yup and postgresql and maybe mongodb | 21:10 |
| mgagne | thanks for the explanation =) | 21:10 |
| *** lezbar has quit IRC | 21:10 | |
| mordred | in zuul v3, I imagine the "setup a database for them" can be a nice ansible role they can add to the job | 21:10 |
| fungi | mgagne: the details of those builders are here http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n919 | 21:10 |
| jeblair | mordred: yes, though we have that in v2 with a jjb builder | 21:10 |
| *** lezbar has joined #openstack-infra | 21:10 | |
| *** thorst has quit IRC | 21:11 | |
| mgagne | yea I'm in those files, just looking for the rational between those 2 =) | 21:11 |
| fungi | mgagne: basically the use case is jobs that don't use sudo but need to perform administrative database functions | 21:11 |
| fungi | so this gives them control of those databases | 21:11 |
| *** thorst has joined #openstack-infra | 21:11 | |
| jeblair | mordred: (but yes, the use case is handled in v3!) | 21:11 |
| mgagne | cool =) | 21:12 |
| *** vsaienko has joined #openstack-infra | 21:12 | |
| *** Goneri has quit IRC | 21:13 | |
| *** sdake has joined #openstack-infra | 21:13 | |
| *** briancline has quit IRC | 21:14 | |
| *** annegentle has joined #openstack-infra | 21:15 | |
| *** matrohon has quit IRC | 21:15 | |
| *** thorst has quit IRC | 21:16 | |
| *** ldnunes has quit IRC | 21:16 | |
| *** burgerk has quit IRC | 21:16 | |
| *** annegent_ has joined #openstack-infra | 21:16 | |
| *** cardeois has joined #openstack-infra | 21:18 | |
| *** cardeois has quit IRC | 21:18 | |
| *** cardeois has joined #openstack-infra | 21:19 | |
| *** briancline has joined #openstack-infra | 21:19 | |
| *** rfolco_ has quit IRC | 21:20 | |
| mtreinish | boo, cacti is breaking apart right as we opened the flood gates on firehose: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=301&page=2 | 21:20 |
| *** fguillot has quit IRC | 21:20 | |
| *** sabari has left #openstack-infra | 21:20 | |
| *** annegentle has quit IRC | 21:20 | |
| mtreinish | now we don't get to clearly see if there is a spike in anything with all of logstash going through it | 21:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Don't specify an executable if it's not needed https://review.openstack.org/379439 | 21:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 21:20 |
| *** cardeois_ has quit IRC | 21:21 | |
| *** gyee has joined #openstack-infra | 21:22 | |
| *** rtheis has quit IRC | 21:22 | |
| *** cardeois has quit IRC | 21:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Enable pipelining for ansible-playbook https://review.openstack.org/331499 | 21:24 |
| *** jheroux has quit IRC | 21:24 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update localrc for ironic-multinode case https://review.openstack.org/352790 | 21:25 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Setup ssh-key on subnodes for Ironic https://review.openstack.org/364830 | 21:25 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update ENABLED_SERVICE on subnode with ironic https://review.openstack.org/368611 | 21:25 |
| *** aeng has joined #openstack-infra | 21:25 | |
| *** baoli has quit IRC | 21:26 | |
| *** baoli has joined #openstack-infra | 21:27 | |
| *** anteaya has joined #openstack-infra | 21:28 | |
| *** amitgandhinz has quit IRC | 21:28 | |
| *** amitgandhinz has joined #openstack-infra | 21:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Sleep at the end of the loop https://review.openstack.org/379832 | 21:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Don't specify an executable if it's not needed https://review.openstack.org/379439 | 21:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 21:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Add async action plugin to override upstream async https://review.openstack.org/371733 | 21:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Enable pipelining for ansible-playbook https://review.openstack.org/331499 | 21:33 |
| *** adrian_otto has quit IRC | 21:35 | |
| *** eharney has quit IRC | 21:35 | |
| *** nadya has joined #openstack-infra | 21:37 | |
| *** thorst has joined #openstack-infra | 21:38 | |
| *** tonytan4ever has joined #openstack-infra | 21:38 | |
| *** mdrabe has quit IRC | 21:40 | |
| *** mdrabe has joined #openstack-infra | 21:40 | |
| *** nadya has quit IRC | 21:42 | |
| *** thorst has quit IRC | 21:42 | |
| *** tonytan4ever has quit IRC | 21:44 | |
| *** e0ne has quit IRC | 21:45 | |
| *** e0ne has joined #openstack-infra | 21:45 | |
| *** e0ne has quit IRC | 21:45 | |
| *** e0ne has joined #openstack-infra | 21:46 | |
| *** Guest92 has joined #openstack-infra | 21:46 | |
| *** yamahata has quit IRC | 21:46 | |
| *** e0ne has quit IRC | 21:46 | |
| *** EricGonczer_ has quit IRC | 21:47 | |
| *** e0ne has joined #openstack-infra | 21:47 | |
| *** e0ne has quit IRC | 21:47 | |
| *** flepied has joined #openstack-infra | 21:47 | |
| *** e0ne has joined #openstack-infra | 21:47 | |
| *** e0ne has quit IRC | 21:48 | |
| *** dtantsur has quit IRC | 21:48 | |
| *** e0ne has joined #openstack-infra | 21:48 | |
| *** e0ne has quit IRC | 21:49 | |
| *** e0ne has joined #openstack-infra | 21:49 | |
| *** rushil has quit IRC | 21:49 | |
| *** e0ne has quit IRC | 21:49 | |
| *** tuannguyen has joined #openstack-infra | 21:51 | |
| *** adriant has joined #openstack-infra | 21:51 | |
| *** Goneri has joined #openstack-infra | 21:53 | |
| *** baoli has quit IRC | 21:53 | |
| *** baoli has joined #openstack-infra | 21:54 | |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: Move opensuse utils to zypper so they can be shared by SUSE-based distros https://review.openstack.org/379835 | 21:54 |
| *** xyang1 has quit IRC | 21:55 | |
| *** caphrim007 has joined #openstack-infra | 21:55 | |
| *** tuannguyen has quit IRC | 21:55 | |
| openstackgerrit | Markos Chandras proposed openstack/diskimage-builder: Move opensuse utils to zypper so they can be shared by SUSE-based distros https://review.openstack.org/379835 | 21:56 |
| *** esberglu has quit IRC | 21:57 | |
| *** woodster_ has quit IRC | 22:00 | |
| *** Apoorva_ has joined #openstack-infra | 22:00 | |
| *** amitgandhinz has quit IRC | 22:01 | |
| *** mdrabe has quit IRC | 22:03 | |
| *** Apoorva has quit IRC | 22:04 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:05 | |
| *** Goneri has quit IRC | 22:05 | |
| *** baoli has quit IRC | 22:06 | |
| *** baoli has joined #openstack-infra | 22:07 | |
| jhesketh | Morning | 22:10 |
| *** ijw has joined #openstack-infra | 22:11 | |
| *** spzala has quit IRC | 22:11 | |
| clarkb | good morning | 22:11 |
| *** jamesden_ has quit IRC | 22:13 | |
| *** cindy has quit IRC | 22:20 | |
| *** drawsmcgraw has left #openstack-infra | 22:20 | |
| *** cindy has joined #openstack-infra | 22:21 | |
| anteaya | morning jhesketh | 22:22 |
| *** john-davidge has joined #openstack-infra | 22:23 | |
| *** Guest99228 is now known as melwitt | 22:23 | |
| *** tonytan4ever has joined #openstack-infra | 22:27 | |
| *** lezbar has quit IRC | 22:27 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 22:27 |
| *** lezbar has joined #openstack-infra | 22:27 | |
| *** john-davidge has quit IRC | 22:28 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/germqtt: Add config option to set QOS of published messages https://review.openstack.org/350169 | 22:29 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/germqtt: Cleanup sample config file https://review.openstack.org/379842 | 22:31 |
| *** mriedem has quit IRC | 22:32 | |
| *** kzaitsev_mb has quit IRC | 22:33 | |
| *** moravec has joined #openstack-infra | 22:35 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Enable container job on multinode https://review.openstack.org/288915 | 22:38 |
| openstackgerrit | Eric Brown proposed openstack-infra/project-config: Add doc publish and py3x jobs to syntribos https://review.openstack.org/379844 | 22:38 |
| *** baoli has quit IRC | 22:41 | |
| *** baoli has joined #openstack-infra | 22:41 | |
| *** Kiall has quit IRC | 22:41 | |
| *** baoli has quit IRC | 22:42 | |
| *** baoli has joined #openstack-infra | 22:42 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:46 | |
| *** pahuang has joined #openstack-infra | 22:46 | |
| *** pahuang has quit IRC | 22:50 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/lpmqtt: Add config option to set the imap idle timeout https://review.openstack.org/379849 | 22:51 |
| *** sjmc7 has quit IRC | 22:52 | |
| *** tuannguyen has joined #openstack-infra | 22:52 | |
| *** kzaitsev_mb has quit IRC | 22:56 | |
| *** tuannguyen has quit IRC | 22:57 | |
| *** rbrndt has quit IRC | 22:57 | |
| *** pahuang has joined #openstack-infra | 22:58 | |
| *** ddieterly is now known as ddieterly[away] | 22:58 | |
| *** pfallenop has quit IRC | 22:59 | |
| mtreinish | clarkb: hmm, looking at the topics on firehose, I think we're only getting mqtt messages from logstash-worker01 | 22:59 |
| fungi | ooh! fox in docs | 22:59 |
| mtreinish | unless I made a typo in the puppet and all the workers are using logstash/logstash-worker01 as the topic for the mqtt plugin | 22:59 |
| fungi | that'll be fun to say repeatedly | 22:59 |
| *** xarses has quit IRC | 22:59 | |
| fungi | mtreinish: quick suggestion as to file i can look at on, say, worker02 to see if it incorrectly says worker01? | 23:00 |
| *** hongbin has quit IRC | 23:00 | |
| clarkb | fungi: /etc/logstash/conf.d/something or toher like that | 23:01 |
| mtreinish | it should be /etc/logstash/conf.d/99-output.conf iirc | 23:01 |
| fungi | topic => "logstash/logstash-worker02" | 23:01 |
| *** amitgandhinz has joined #openstack-infra | 23:02 | |
| mtreinish | hmm, then I guess it's not publishing from any other workers | 23:02 |
| *** gordc has quit IRC | 23:02 | |
| clarkb | that worker should have a connection to the mosquitto server ya? | 23:02 |
| clarkb | I guess thats the next thing to check? | 23:02 |
| fungi | i see the java process on worker02 but no gearman workers? | 23:03 |
| *** kzaitsev_mb has joined #openstack-infra | 23:03 | |
| clarkb | fungi: I agree so that likely explains it | 23:03 |
| fungi | i wonder if we somehow bombed out all the workers again | 23:03 |
| mtreinish | clarkb: well it depends on the ruby mqtt lib really. Like the python one only opens a connection when it goes to publish a message | 23:03 |
| clarkb | those workers are supposed to retry with a long delay to handle logstash restarts but maybe its not working properly | 23:03 |
| clarkb | fungi: ya I am going to guess they all died when we restarted logasth | 23:04 |
| * clarkb checks a log | 23:04 | |
| *** gouthamr has joined #openstack-infra | 23:04 | |
| mtreinish | (it keeps an open connection when subscribing, but not for publishing) | 23:04 |
| *** tpsilva has quit IRC | 23:04 | |
| fungi | ends at: error: [Errno 111] Connection refused | 23:04 |
| clarkb | error: [Errno 111] Connection refused | 23:04 |
| mtreinish | heh, connection refused from what? | 23:04 |
| clarkb | ya so I think we aren't getting the tcp connection back up again when logstash restarts as quickly as we'd like | 23:04 |
| clarkb | mtreinish: the gearman worker to logstash jvm | 23:04 |
| mtreinish | ah, ok | 23:05 |
| mtreinish | the gearman worker does connect to a few different things, so I wasn't sure | 23:05 |
| fungi | should i restart the gearman worker processes in a loop across all the servers again like last time? | 23:05 |
| clarkb | fungi: ya I think so | 23:05 |
| *** dprince has quit IRC | 23:05 | |
| jlvillal | clarkb: Is there a bug or patch set being worked on for the multi-node failures? Or has it already been fixed? | 23:06 |
| jlvillal | I wanted to subscribe myself so I would find out about updates. | 23:06 |
| clarkb | fungi: reading the logs carefully we did wait the full 90 seconds before attemtping to reconnect | 23:07 |
| clarkb | fungi: so I think logstash may take longer than that to reestablish its tcp scoket | 23:07 |
| clarkb | jlvillal: I am not sure, I asked if anyone in neutron had filed one yet and didn't get an answer so probably not | 23:07 |
| fungi | sounds likely | 23:07 |
| jlvillal | clarkb: Ah thanks. | 23:07 |
| clarkb | jlvillal: give me a few and I will file one | 23:07 |
| mtreinish | clarkb, jlvillal: is that failure e-r fingerprintable? (doesn't know what is failing) | 23:08 |
| clarkb | mtreinish: its already fingerprinted by the ssh to subnode fails check | 23:08 |
| jeblair | mtreinish: i'm going to reparent 375780 on HEAD and merge | 23:08 |
| mtreinish | clarkb: ah, ok | 23:08 |
| clarkb | mtreinish: the underlying cause for that in osic with neutron is neutron breaking routing for 10.0.0.0/8 | 23:08 |
| jlvillal | clarkb: thanks | 23:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Fix header level for IMAP and MX section https://review.openstack.org/375780 | 23:09 |
| mtreinish | jeblair: ok | 23:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add a note to docs for websockets being disabled https://review.openstack.org/379536 | 23:09 |
| mtreinish | jeblair: fwiw, it was based on top of another doc patch | 23:09 |
| mtreinish | was there an issue with the ssl docs patch? | 23:09 |
| jeblair | mtreinish: yep, i had a question about the one it was based on | 23:09 |
| mtreinish | ah, see that now | 23:10 |
| jeblair | but this didn't need that one, and i need a change to land to test afs docs :) | 23:10 |
| fungi | mtreinish: clarkb: 4 worker processes running on each of all 20 servers now | 23:10 |
| *** jamielennox is now known as jamielennox|away | 23:11 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove osic-cloud1 provider from nodepool https://review.openstack.org/379513 | 23:11 |
| *** sulo_ is now known as sulo | 23:11 | |
| mtreinish | fungi: cool, now I'm seeing a mix of logstash topics on the firehose | 23:11 |
| *** sdake has quit IRC | 23:12 | |
| *** smarcet1981 has joined #openstack-infra | 23:13 | |
| *** baoli has quit IRC | 23:13 | |
| jeblair | pabelanger: oops, we're missing k5start on launchers | 23:13 |
| fungi | as in missing the actual package, or missing the invocation? | 23:14 |
| pabelanger | :) | 23:14 |
| jeblair | package | 23:14 |
| fungi | that makes more sense. i coulda sworn i reviewed a change to use k5start for this | 23:14 |
| pabelanger | let me see how we install it today with puppet | 23:15 |
| fungi | might be a good idea for the afs class to just add it | 23:15 |
| pabelanger | ya | 23:15 |
| jeblair | fungi: yeah i was thinking that too... | 23:15 |
| jeblair | (package name is kstart btw) | 23:15 |
| *** priteau has quit IRC | 23:16 | |
| mtreinish | clarkb, fungi: heh we really need a tool to measure the message throughput on firehose now, there's a lot going through it | 23:16 |
| jeblair | fungi, pabelanger: oh, it does... afs seems to be missing on the launchers | 23:16 |
| clarkb | mtreinish: jlvillal frickler sc68cal https://bugs.launchpad.net/neutron/+bug/1629133 has been created | 23:16 |
| openstack | Launchpad bug 1629133 in neutron "New neutron subnet pool support breaks multinode testing." [Undecided,New] | 23:16 |
| jeblair | i guess *that's* what we forgot to do? :) | 23:16 |
| pabelanger | jeblair: fungi: Ya, puppet-openafs has it | 23:17 |
| fungi | er, the afs=>true class parameter to the openstack_project::server class? | 23:17 |
| jeblair | yep | 23:17 |
| fungi | aha, i concur | 23:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Install AFS on zuul launchers https://review.openstack.org/379851 | 23:18 |
| jeblair | fungi, pabelanger: ^ | 23:18 |
| jeblair | minor detail | 23:18 |
| pabelanger | +2 | 23:18 |
| fungi | so openafs::client does indeed install the kstart package | 23:18 |
| jeblair | i believe so | 23:19 |
| *** smarcet1981 has quit IRC | 23:19 | |
| jeblair | fungi: past us is at least as clever as present us | 23:19 |
| fungi | http://git.openstack.org/cgit/openstack-infra/puppet-openafs/tree/manifests/client.pp#n21 | 23:19 |
| fungi | yep | 23:19 |
| openstackgerrit | Merged openstack-infra/system-config: Fix header level for IMAP and MX section https://review.openstack.org/375780 | 23:19 |
| fungi | lgtm | 23:19 |
| *** ddieterly[away] is now known as ddieterly | 23:20 | |
| mordred | I wanted to get a +2 in on that one too :) | 23:21 |
| fungi | SpamapS: pure gold! | 23:21 |
| mordred | SpamapS: wow. you win all the things | 23:22 |
| mordred | like, seriously | 23:22 |
| jlvillal | clarkb: thanks | 23:22 |
| jlvillal | That darn subnetpool has broken a lot of things... | 23:22 |
| clarkb | jlvillal: well subnetpool itself is fine | 23:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 23:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Use command module instead of zuul_runner https://review.openstack.org/369604 | 23:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Put script string in directly instead of in files https://review.openstack.org/369605 | 23:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Use command module instead of async https://review.openstack.org/371733 | 23:23 |
| clarkb | it just currently uses a very not great for testing range by default | 23:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Rename zuul_runner to command https://review.openstack.org/369603 | 23:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Enable pipelining for ansible-playbook https://review.openstack.org/331499 | 23:23 |
| mtreinish | SpamapS: I am looking forward to your wall | 23:23 |
| jlvillal | clarkb: I know that a devstack patch regarding subnetpool seems to have broken a few of the 3rd Party CIs. | 23:24 |
| mordred | jeblair: ^^ I think that should do it - and we could skip patch 5 if we wanted to | 23:24 |
| clarkb | jlvillal: ya it broke them for other very similar reasons | 23:24 |
| clarkb | jlvillal: at least in the one case I saw it was because the CI setup was using 172.something which wasn't included by 10/8 so failed | 23:24 |
| *** rajinir has quit IRC | 23:25 | |
| jlvillal | clarkb: FYI: https://review.openstack.org/#/c/378063/ | 23:25 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add docs on using SSL/TLS MQTT with firehose https://review.openstack.org/375779 | 23:25 |
| mtreinish | jeblair: ^^^ | 23:26 |
| clarkb | jlvillal: I don't think we can do that since tempest appears to rely on the feature now | 23:26 |
| clarkb | jlvillal: we should just find a way to use smaller ranges that don't interfere with test setups | 23:26 |
| jlvillal | Ah okay. Makes sense. | 23:26 |
| mtreinish | clarkb: which feature? | 23:26 |
| clarkb | mtreinish: subnetpools for auto network allocation | 23:27 |
| openstackgerrit | Merged openstack-infra/system-config: Install AFS on zuul launchers https://review.openstack.org/379851 | 23:27 |
| clarkb | mtreinish: there is one test that fails on my fix and its because my alternative ipv6 default range isn't big enough | 23:27 |
| jeblair | mtreinish: woo! | 23:27 |
| jlvillal | clarkb: On a different note. I changed patch to use setup_ssh, with some mods to setup_ssh: https://review.openstack.org/#/c/364830/ | 23:27 |
| clarkb | jlvillal: cool | 23:28 |
| mtreinish | clarkb: hmm, which test. I don't think we've changed anything that directly touches the subnetpools api in a while | 23:28 |
| mtreinish | oh, unless it's the get me a network test doing something under the covers in nova | 23:28 |
| mtreinish | but all of that should be configurable and something you can turn off via the tempest.conf if your env doesn't support it | 23:29 |
| mtreinish | granted that's not a good solution for the gate if it's something we want to be testing | 23:29 |
| clarkb | mtreinish: http://logs.openstack.org/43/379543/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/8d108f9/console.html#_2016-09-29_16_16_29_930439 that one | 23:29 |
| mtreinish | clarkb: ah ok, yeah that's mriedem's get me a network stuff | 23:30 |
| clarkb | SpamapS: not enough 'cyber' | 23:33 |
| *** vsaienko has quit IRC | 23:33 | |
| *** ijw has quit IRC | 23:33 | |
| *** ijw has joined #openstack-infra | 23:34 | |
| *** tonytan4ever has quit IRC | 23:35 | |
| *** kzaitsev_mb has quit IRC | 23:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Split playbook into vars, pre-playbook and playbook https://review.openstack.org/371734 | 23:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Stop running commands with async https://review.openstack.org/371733 | 23:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Enable pipelining for ansible-playbook https://review.openstack.org/331499 | 23:35 |
| *** amitgandhinz has quit IRC | 23:35 | |
| *** markvoelker has quit IRC | 23:36 | |
| *** pfallenop has joined #openstack-infra | 23:36 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Update exludes list for TARCMD https://review.openstack.org/379576 | 23:37 |
| *** ijw has quit IRC | 23:38 | |
| *** ijw has joined #openstack-infra | 23:38 | |
| SpamapS | clarkb: says who? | 23:39 |
| clarkb | debate watchers everywhere | 23:39 |
| pabelanger | I approve of more 'the cyber' | 23:40 |
| *** lezbar has quit IRC | 23:40 | |
| *** lseki has quit IRC | 23:40 | |
| *** lezbar has joined #openstack-infra | 23:40 | |
| *** pfallenop has quit IRC | 23:41 | |
| openstackgerrit | matthew wagoner proposed openstack-infra/shade: Add test for os_group Ansible module https://review.openstack.org/379854 | 23:42 |
| fungi | the security aspect of cyber is very, very tough. and maybe it’s hardly doable | 23:43 |
| *** Swami has quit IRC | 23:43 | |
| *** zhurong has joined #openstack-infra | 23:45 | |
| *** thorst has joined #openstack-infra | 23:47 | |
| *** cindy has quit IRC | 23:47 | |
| *** ijw has quit IRC | 23:48 | |
| *** ijw has joined #openstack-infra | 23:49 | |
| *** ijw has quit IRC | 23:50 | |
| *** ddieterly is now known as ddieterly[away] | 23:51 | |
| *** jamielennox|away is now known as jamielennox | 23:52 | |
| *** EricGonczer_ has joined #openstack-infra | 23:53 | |
| *** jamesdenton has joined #openstack-infra | 23:54 | |
| *** ddieterly[away] has quit IRC | 23:54 | |
| *** pfallenop has joined #openstack-infra | 23:54 | |
| *** sdake has joined #openstack-infra | 23:55 | |
| *** EricGonczer_ has quit IRC | 23:57 | |
| *** sflanigan_ has joined #openstack-infra | 23:57 | |
| *** sflanigan_ has quit IRC | 23:57 | |
| *** sflanigan_ has joined #openstack-infra | 23:57 | |
| *** EricGonczer_ has joined #openstack-infra | 23:57 | |
| *** thorst has quit IRC | 23:58 | |
| *** vsaienko has joined #openstack-infra | 23:58 | |
| *** dangers is now known as dangers_away | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!