| *** david-lyle has joined #openstack-infra | 00:00 | |
| *** baoli has quit IRC | 00:00 | |
| *** david-lyle_ has joined #openstack-infra | 00:03 | |
| docaedo | I was looking at a really unimportant bug (https://bugs.launchpad.net/app-catalog/+bug/1553572) and happened to have just spun up a new "test" server for app-catalog stuff, and found the issue | 00:05 |
|---|---|---|
| openstack | Launchpad bug 1553572 in Community App Catalog "Last modified date missing from api/v1/assets" [Medium,Triaged] - Assigned to Christopher Aedo (docaedo) | 00:05 |
| pabelanger | clarkb: ack | 00:05 |
| *** david-lyle has quit IRC | 00:05 | |
| *** eggshell has quit IRC | 00:06 | |
| *** tphummel has quit IRC | 00:07 | |
| *** david-lyle_ has quit IRC | 00:09 | |
| *** edtubill has joined #openstack-infra | 00:11 | |
| *** sdake has joined #openstack-infra | 00:16 | |
| *** Swami has quit IRC | 00:21 | |
| openstackgerrit | Christopher Aedo proposed openstack-infra/puppet-apps_site: Include package python-dateutils https://review.openstack.org/358909 | 00:23 |
| *** Jeffrey4l_ has joined #openstack-infra | 00:23 | |
| *** pvaneck has quit IRC | 00:24 | |
| *** sputnik13 has quit IRC | 00:24 | |
| *** tonytan4ever has joined #openstack-infra | 00:25 | |
| *** nwkarsten has quit IRC | 00:26 | |
| *** nwkarsten has joined #openstack-infra | 00:27 | |
| *** zeroDivisible has quit IRC | 00:27 | |
| *** mdrabe has quit IRC | 00:29 | |
| *** zeroDivisible has joined #openstack-infra | 00:29 | |
| *** tonytan4ever has quit IRC | 00:31 | |
| *** nwkarsten has quit IRC | 00:31 | |
| *** Julien-zte has joined #openstack-infra | 00:34 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Create a new meeting for WOS-mentoring https://review.openstack.org/356467 | 00:36 |
| *** caowei has quit IRC | 00:43 | |
| openstackgerrit | Merged openstack-infra/project-config: Increase packaging-deb timeouts https://review.openstack.org/358857 | 00:43 |
| docaedo | If any infra cores would like to review a really exciting patch, it would be much appreciated. https://review.openstack.org/358909 will make the Community App Catalog *gloriously* dynamic (by updating the recently added apps section based on actual dates, vs. current randomness) | 00:43 |
| ianw | fyi, as discussed with rcarrillocruz i'm going to see what i can do about this new review-dev as discussed -> http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-08-16-19.02.log.html#l-194 | 00:48 |
| *** gouthamr has joined #openstack-infra | 00:52 | |
| *** piet has quit IRC | 00:53 | |
| *** nwkarsten has joined #openstack-infra | 00:56 | |
| *** tqtran has quit IRC | 00:56 | |
| *** fguillot is now known as fguillot_afk | 00:57 | |
| *** fguillot_afk has quit IRC | 00:57 | |
| *** Apoorva has quit IRC | 00:59 | |
| *** fguillot has joined #openstack-infra | 01:00 | |
| *** nwkarsten has quit IRC | 01:00 | |
| *** chem has quit IRC | 01:02 | |
| *** Hal1 has joined #openstack-infra | 01:04 | |
| *** nwkarsten has joined #openstack-infra | 01:04 | |
| *** zxiiro-away has quit IRC | 01:04 | |
| *** rockstar has quit IRC | 01:05 | |
| *** rockstar has joined #openstack-infra | 01:05 | |
| *** esberglu has joined #openstack-infra | 01:06 | |
| *** sdake_ has joined #openstack-infra | 01:06 | |
| *** gyee has quit IRC | 01:07 | |
| *** zxiiro-away has joined #openstack-infra | 01:07 | |
| *** Hal has quit IRC | 01:07 | |
| *** thorst_ has quit IRC | 01:07 | |
| *** thorst has joined #openstack-infra | 01:08 | |
| *** sdake has quit IRC | 01:10 | |
| *** jamielennox is now known as jamielennox|away | 01:11 | |
| *** chem has joined #openstack-infra | 01:12 | |
| *** zhurong has joined #openstack-infra | 01:13 | |
| *** yanyanhu has joined #openstack-infra | 01:13 | |
| *** eranrom has joined #openstack-infra | 01:15 | |
| *** salv-orl_ has joined #openstack-infra | 01:15 | |
| *** jamielennox|away is now known as jamielennox | 01:16 | |
| *** thorst has quit IRC | 01:16 | |
| *** salv-orlando has quit IRC | 01:18 | |
| *** sdake_ has quit IRC | 01:19 | |
| *** eranrom has quit IRC | 01:19 | |
| *** sdake has joined #openstack-infra | 01:22 | |
| *** nwkarsten has quit IRC | 01:22 | |
| *** yanyanhu has quit IRC | 01:23 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add memory to overcloud vms up to 6144 https://review.openstack.org/357532 | 01:25 |
| *** zhurong has quit IRC | 01:25 | |
| *** salv-orl_ has quit IRC | 01:26 | |
| *** zhurong has joined #openstack-infra | 01:26 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 01:26 |
| clarkb | ubuntu xenial image after reseting the cache is 7.3GB qcow2 | 01:28 |
| clarkb | so got almost a gigabyte back | 01:28 |
| *** baoli has joined #openstack-infra | 01:28 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/devstack-gate: Remove q-lbaas from tempest pre-installed stuff. https://review.openstack.org/358258 | 01:28 |
| clarkb | and the vhd is 18GB so about 3 GB saved there | 01:28 |
| clarkb | I have started image uploads for the new xenial image in all clouds. Goal is to get ntpdate'd xenial iamges everywhere so we can update d-g with that revert | 01:31 |
| *** hongbin has joined #openstack-infra | 01:35 | |
| *** hockeynut has quit IRC | 01:36 | |
| *** raunak has quit IRC | 01:37 | |
| clarkb | it would be really neat if we could read only mount a single volume on many hosts | 01:38 |
| clarkb | then instead of 18GB images we could have 600MB images with a 17GB cinder volume | 01:39 |
| *** coolsvap has quit IRC | 01:39 | |
| *** andymaier has joined #openstack-infra | 01:41 | |
| *** markvoelker has joined #openstack-infra | 01:44 | |
| *** markvoelker_ has joined #openstack-infra | 01:46 | |
| *** xarses has joined #openstack-infra | 01:48 | |
| *** markvoelker has quit IRC | 01:50 | |
| *** caowei has joined #openstack-infra | 01:52 | |
| pabelanger | clarkb: wow, so, why the drop in size? Stale packages getting pulled into the images? | 01:54 |
| *** nwkarsten has joined #openstack-infra | 01:54 | |
| clarkb | pabelanger: I think our git cache bloats over time | 01:56 |
| pabelanger | Ah | 01:56 |
| clarkb | I didnt delete the old cache just moved it aside so we can compare | 01:56 |
| *** mtanin___ has quit IRC | 02:07 | |
| *** annegentle has joined #openstack-infra | 02:07 | |
| *** yuanying has quit IRC | 02:10 | |
| *** zhurong has quit IRC | 02:13 | |
| *** annegentle has quit IRC | 02:14 | |
| *** thorst has joined #openstack-infra | 02:14 | |
| *** baoli has quit IRC | 02:15 | |
| *** yamahata has quit IRC | 02:15 | |
| *** zhurong has joined #openstack-infra | 02:15 | |
| *** shashank_hegde has quit IRC | 02:16 | |
| *** rfolco has quit IRC | 02:18 | |
| *** thorst has quit IRC | 02:22 | |
| *** raunak has joined #openstack-infra | 02:28 | |
| *** ramishra has quit IRC | 02:40 | |
| *** esberglu has quit IRC | 02:43 | |
| *** roxanagh_ has joined #openstack-infra | 02:47 | |
| *** jamielennox is now known as jamielennox|away | 02:49 | |
| adriant | Hey, any reason why a new project I'm trying to get added to the openstack gerrit is taking forever? | 02:51 |
| *** roxanagh_ has quit IRC | 02:52 | |
| adriant | I resolved the only comment posted on it, but haven't gotten any updates since. | 02:52 |
| adriant | patch in question: https://review.openstack.org/#/c/353818/ | 02:53 |
| *** tqtran has joined #openstack-infra | 02:54 | |
| *** admcleod has joined #openstack-infra | 02:57 | |
| *** gouthamr_ has joined #openstack-infra | 02:57 | |
| *** admcleod_ has quit IRC | 02:57 | |
| *** gouthamr has quit IRC | 02:58 | |
| *** tqtran has quit IRC | 03:00 | |
| *** gouthamr_ is now known as gouthamr | 03:01 | |
| *** andymaier has quit IRC | 03:03 | |
| *** nwkarsten has quit IRC | 03:03 | |
| *** jamielennox|away is now known as jamielennox | 03:06 | |
| *** fguillot has quit IRC | 03:06 | |
| *** ianychoi has joined #openstack-infra | 03:07 | |
| *** thorst has joined #openstack-infra | 03:20 | |
| *** vinaypotluri has quit IRC | 03:21 | |
| *** nwkarsten has joined #openstack-infra | 03:23 | |
| *** thorst has quit IRC | 03:26 | |
| *** salv-orlando has joined #openstack-infra | 03:30 | |
| *** Goneri has quit IRC | 03:31 | |
| *** vikrant has joined #openstack-infra | 03:34 | |
| *** vikrant is now known as vikrant|brb | 03:34 | |
| openstackgerrit | Donovan Jones proposed openstack-infra/shade: Allow object storage endpoint to return 404 for missing /info endpoint https://review.openstack.org/358937 | 03:34 |
| armax | hi infra wizards, there’s a change in the gate queue (358753) that’s meant to alleviate some pressure on the gate | 03:36 |
| armax | if that could be bumped up, that would help get rid of the recent failures like this one | 03:37 |
| armax | empest.exceptions.BuildErrorException: Server 04fe97b1-e46e-4bea-a378-afc6ec04fb7d failed to build and is in ERROR status | 03:37 |
| armax | 2016-08-23 02:30:43.168929 | Details: {u'created': u'2016-08-23T01:49:28Z', u'message': u'Build of instance 04fe97b1-e46e-4bea-a378-afc6ec04fb7d aborted: Failed to allocate the network(s), not rescheduling.', u'code': 500} | 03:37 |
| *** salv-orlando has quit IRC | 03:38 | |
| *** asselin_ has joined #openstack-infra | 03:38 | |
| *** yamahata has joined #openstack-infra | 03:40 | |
| *** zul has quit IRC | 03:41 | |
| *** asselin has quit IRC | 03:42 | |
| *** gouthamr has quit IRC | 03:44 | |
| *** shashank_hegde has joined #openstack-infra | 03:45 | |
| *** zul has joined #openstack-infra | 03:46 | |
| *** vikrant|brb is now known as vikrant | 03:47 | |
| *** roxanagh_ has joined #openstack-infra | 03:48 | |
| *** aeng has quit IRC | 03:51 | |
| *** roxanagh_ has quit IRC | 03:52 | |
| *** vinaypotluri has joined #openstack-infra | 03:54 | |
| *** nwkarsten has quit IRC | 03:57 | |
| *** yuanying has joined #openstack-infra | 03:59 | |
| *** M-docaedo_vector has quit IRC | 04:00 | |
| *** sflanigan has quit IRC | 04:02 | |
| *** hongbin has quit IRC | 04:03 | |
| *** aeng has joined #openstack-infra | 04:07 | |
| *** Jaison has joined #openstack-infra | 04:15 | |
| *** timello has quit IRC | 04:15 | |
| *** markvoelker has joined #openstack-infra | 04:21 | |
| *** ilyashakhat has joined #openstack-infra | 04:22 | |
| *** markvoelker_ has quit IRC | 04:22 | |
| *** sarob has joined #openstack-infra | 04:23 | |
| *** thorst has joined #openstack-infra | 04:24 | |
| *** Jaison has quit IRC | 04:24 | |
| *** nwkarsten has joined #openstack-infra | 04:25 | |
| *** jraju has joined #openstack-infra | 04:26 | |
| *** baoli has joined #openstack-infra | 04:27 | |
| *** sarob has quit IRC | 04:27 | |
| *** markvoelker has quit IRC | 04:28 | |
| *** timello has joined #openstack-infra | 04:29 | |
| *** thorst has quit IRC | 04:31 | |
| *** baoli has quit IRC | 04:31 | |
| *** M-docaedo_vector has joined #openstack-infra | 04:32 | |
| *** _nadya_ has joined #openstack-infra | 04:36 | |
| *** jerryz has joined #openstack-infra | 04:37 | |
| *** salv-orlando has joined #openstack-infra | 04:37 | |
| *** AJaeger has joined #openstack-infra | 04:39 | |
| *** jtomasek has quit IRC | 04:42 | |
| *** salv-orlando has quit IRC | 04:49 | |
| *** jamielennox is now known as jamielennox|away | 04:49 | |
| *** edtubill has quit IRC | 04:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:51 | |
| *** tqtran has joined #openstack-infra | 04:57 | |
| *** armax has quit IRC | 04:57 | |
| *** salv-orlando has joined #openstack-infra | 04:58 | |
| *** tqtran has quit IRC | 05:01 | |
| *** Hal1 has quit IRC | 05:01 | |
| *** Hal has joined #openstack-infra | 05:02 | |
| *** claudiub has joined #openstack-infra | 05:02 | |
| *** roxanagh_ has joined #openstack-infra | 05:02 | |
| *** roxanagh_ has quit IRC | 05:03 | |
| *** Sukhdev has joined #openstack-infra | 05:05 | |
| *** Sukhdev has quit IRC | 05:07 | |
| *** Sukhdev has joined #openstack-infra | 05:07 | |
| *** jaosorior has joined #openstack-infra | 05:10 | |
| *** yanyanhu has joined #openstack-infra | 05:11 | |
| *** tphummel has joined #openstack-infra | 05:12 | |
| *** kzaitsev_mb has quit IRC | 05:12 | |
| *** senk has joined #openstack-infra | 05:14 | |
| *** eranrom has joined #openstack-infra | 05:16 | |
| *** _nadya_ has quit IRC | 05:21 | |
| *** eranrom has quit IRC | 05:21 | |
| *** sdake_ has joined #openstack-infra | 05:21 | |
| *** jamielennox|away is now known as jamielennox | 05:23 | |
| *** sdake has quit IRC | 05:24 | |
| *** markvoelker has joined #openstack-infra | 05:29 | |
| openstackgerrit | Guido Günther proposed openstack-infra/jenkins-job-builder: Fix logparser for 2.0 module https://review.openstack.org/358956 | 05:29 |
| *** thorst has joined #openstack-infra | 05:30 | |
| *** ilyashakhat has quit IRC | 05:31 | |
| *** raunak has quit IRC | 05:35 | |
| *** raunak has joined #openstack-infra | 05:36 | |
| *** thorst has quit IRC | 05:37 | |
| *** sandanar has joined #openstack-infra | 05:37 | |
| *** senk has quit IRC | 05:39 | |
| *** markvoelker has quit IRC | 05:39 | |
| *** ilyashakhat has joined #openstack-infra | 05:40 | |
| *** sdake_ has quit IRC | 05:41 | |
| *** tphummel has quit IRC | 05:41 | |
| *** raunak has quit IRC | 05:42 | |
| *** AJaeger has quit IRC | 05:42 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/shade: test commit for osc3.0.1 https://review.openstack.org/358967 | 05:47 |
| *** Sukhdev has quit IRC | 05:48 | |
| *** AnarchyAo has joined #openstack-infra | 05:48 | |
| *** ilyashakhat has quit IRC | 05:50 | |
| *** AnarchyAo has quit IRC | 05:50 | |
| *** r-mibu has quit IRC | 05:52 | |
| *** dstufft has quit IRC | 05:53 | |
| *** dstufft has joined #openstack-infra | 05:54 | |
| *** nwkarsten has quit IRC | 05:58 | |
| openstackgerrit | Merged openstack-infra/system-config: Set iLO/public/provisioning addresses and metadata for compute043.vanilla https://review.openstack.org/358598 | 05:59 |
| *** roxanagh_ has joined #openstack-infra | 06:04 | |
| *** asselin__ has joined #openstack-infra | 06:04 | |
| openstackgerrit | Guido Günther proposed openstack-infra/jenkins-job-builder: Fix logparser for 2.0 module https://review.openstack.org/358956 | 06:04 |
| *** senk has joined #openstack-infra | 06:04 | |
| *** wcriswell has quit IRC | 06:05 | |
| *** _oanson has joined #openstack-infra | 06:07 | |
| *** asselin_ has quit IRC | 06:07 | |
| openstackgerrit | Merged openstack-infra/system-config: Enable compute005.vanilla and set all IPs and metadata https://review.openstack.org/358631 | 06:08 |
| *** roxanagh_ has quit IRC | 06:08 | |
| *** jamielennox is now known as jamielennox|away | 06:11 | |
| *** r-mibu has joined #openstack-infra | 06:12 | |
| *** pcaruana has joined #openstack-infra | 06:14 | |
| *** woodster_ has quit IRC | 06:19 | |
| *** esikachev has joined #openstack-infra | 06:22 | |
| *** shashank_hegde has quit IRC | 06:24 | |
| *** pt_15 has joined #openstack-infra | 06:25 | |
| *** AJaeger has joined #openstack-infra | 06:33 | |
| *** thorst has joined #openstack-infra | 06:34 | |
| *** markvoelker has joined #openstack-infra | 06:36 | |
| *** abregman has joined #openstack-infra | 06:37 | |
| *** aeng has quit IRC | 06:37 | |
| *** AJaeger has quit IRC | 06:38 | |
| *** wcriswell has joined #openstack-infra | 06:41 | |
| *** thorst has quit IRC | 06:42 | |
| *** markvoelker has quit IRC | 06:42 | |
| *** andreas_s has joined #openstack-infra | 06:43 | |
| openstackgerrit | Merged openstack-infra/system-config: Set compute038.vanilla IPs and metadata https://review.openstack.org/358638 | 06:47 |
| *** aeng has joined #openstack-infra | 06:50 | |
| *** eranrom has joined #openstack-infra | 06:51 | |
| *** AJaeger has joined #openstack-infra | 06:51 | |
| *** florianf has joined #openstack-infra | 06:55 | |
| *** yolanda has quit IRC | 06:56 | |
| *** sflanigan has joined #openstack-infra | 06:56 | |
| openstackgerrit | Madhuri Kumari proposed openstack-infra/project-config: Rename Zun gate tests. https://review.openstack.org/358988 | 06:57 |
| *** nwkarsten has joined #openstack-infra | 06:59 | |
| *** tqtran has joined #openstack-infra | 06:59 | |
| *** tqtran has quit IRC | 07:03 | |
| *** nwkarsten has quit IRC | 07:03 | |
| *** yolanda has joined #openstack-infra | 07:04 | |
| yolanda | good morning | 07:04 |
| AJaeger | good morning, yolanda ! | 07:05 |
| yolanda | hi, AJaeger , back from holiday? | 07:06 |
| yolanda | did you have a good time? | 07:06 |
| *** fmccrthy has quit IRC | 07:06 | |
| *** rackertom has quit IRC | 07:06 | |
| AJaeger | yolanda: was great, thanks! REally relaxing - and my kids were happy ;) | 07:06 |
| *** watersoul has joined #openstack-infra | 07:06 | |
| *** zubchick has quit IRC | 07:07 | |
| *** rackertom has joined #openstack-infra | 07:07 | |
| *** zubchick has joined #openstack-infra | 07:08 | |
| *** watersoul_ has quit IRC | 07:08 | |
| *** fmccrthy has joined #openstack-infra | 07:08 | |
| yolanda | AJaeger, i'm going holiday next week | 07:09 |
| *** esikachev has quit IRC | 07:12 | |
| AJaeger | yolanda: where are you going? | 07:12 |
| yolanda | i'll stay in Spain, but a bit more in the south, to the beach | 07:13 |
| yolanda | near a town called Torrevieja | 07:13 |
| *** penguinolog has joined #openstack-infra | 07:13 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Change hpuswest for vanilla on controller and compute node definitions https://review.openstack.org/358992 | 07:15 |
| AJaeger | yolanda: I wish you a great vacation! | 07:16 |
| *** salv-orl_ has joined #openstack-infra | 07:16 | |
| yolanda | thanks! where have you gone? | 07:16 |
| AJaeger | To the north - an island in the Baltic Sea. A little bit colder than I expect it will be for you ;) | 07:17 |
| AJaeger | Still in Germany - so nice temparature, sea and lots to do ... | 07:17 |
| jaosorior | AJaeger: so what jobs is bindep actually used for? | 07:18 |
| *** tesseract- has joined #openstack-infra | 07:18 | |
| *** andymaier has joined #openstack-infra | 07:18 | |
| AJaeger | jaosorior: did you read http://lists.openstack.org/pipermail/openstack-dev/2016-August/101590.html ? | 07:18 |
| *** salv-orlando has quit IRC | 07:19 | |
| jaosorior | AJaeger: I didn't. I went for the documentation. | 07:19 |
| AJaeger | project-config cores, could you review https://review.openstack.org/358446 https://review.openstack.org/354861 (already +2 by yolanda) and https://review.openstack.org/358769 | 07:19 |
| *** Na3iL has joined #openstack-infra | 07:20 | |
| AJaeger | jaosorior: So, let's update documentation to not confuse you - do you want to give it a go? Or do you have still questions after reading that email? | 07:20 |
| *** salv-orl_ has quit IRC | 07:21 | |
| openstackgerrit | Merged openstack-infra/system-config: Correct iLO IP and rack number for compute19.chocolate https://review.openstack.org/358675 | 07:21 |
| *** vinaypotluri has quit IRC | 07:21 | |
| jaosorior | AJaeger: it seems clearer now. Thanks. So; is there another way of managing dependencies for devstack based tests? | 07:22 |
| AJaeger | jaosorior: let me dig out a link for you... | 07:22 |
| jaosorior | AJaeger: sorry for the extra work; I've actually had a hard time digging out where to do these kind of things. And even got the wrong impression of bindep. | 07:23 |
| AJaeger | jaosorior: http://docs.openstack.org/infra/manual/drivers.html#package-requirements - contains a link to http://git.openstack.org/cgit/openstack-dev/devstack/tree/files | 07:24 |
| AJaeger | jaosorior: sorry to hear that - I would really appreciate if you could help the next person on this journey. | 07:24 |
| AJaeger | jaosorior: So, do you want to patch - or explain to me what confused you and I'll try changing it? | 07:25 |
| *** salv-orlando has joined #openstack-infra | 07:25 | |
| jaosorior | AJaeger: Would be nice to have a more explicit explanation of how bindep is used in openstack (also to specify that it's not used in devstack based jobs). And also some examples on how to use it would be nice. There is some explanation of the syntax, but one has to dig into the openstack projects to concretely see how it's used. | 07:26 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Change hpuswest for vanilla on controller and compute node definitions https://review.openstack.org/358992 | 07:27 |
| *** jpich has joined #openstack-infra | 07:27 | |
| AJaeger | jaosorior: for examples, here's one: https://review.openstack.org/#/c/358811/ | 07:28 |
| AJaeger | jaosorior: I'll write a section... | 07:28 |
| *** matrohon has joined #openstack-infra | 07:30 | |
| jaosorior | AJaeger: yeah, pabelanger passed me some examples. Which was useful. Though it would be useful for the reader to get some examples in the documentation. | 07:30 |
| AJaeger | jaosorior: could you review 358811, please? | 07:30 |
| AJaeger | Suggestions on what else to add are welcome ;) | 07:31 |
| *** Na3iL has quit IRC | 07:31 | |
| jaosorior | AJaeger: now I'm not sure if adding the devstack comment is necessary, since you did pass a link where that's mentioned. So I guess it's fine | 07:33 |
| *** asettle has joined #openstack-infra | 07:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/bindep: Document OpenStack usage https://review.openstack.org/358998 | 07:33 |
| *** yaume has joined #openstack-infra | 07:34 | |
| AJaeger | jaosorior: Brief documentation here ^ | 07:34 |
| *** vincentll has joined #openstack-infra | 07:34 | |
| jaosorior | nice! | 07:35 |
| AJaeger | thanks for review. | 07:37 |
| jaosorior | AJaeger: thanks for the commits and the explanation | 07:37 |
| AJaeger | jaosorior: so, add your bindep.txt file in barbican and leave the devstack change out - and talk to the QA team on how to add the dependencies in the best way for your plugin. That should be a separate change IMHO | 07:38 |
| *** yamamoto has quit IRC | 07:38 | |
| *** markvoelker has joined #openstack-infra | 07:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Add check-requirements to openstack-ansible-specs https://review.openstack.org/358411 | 07:38 |
| jaosorior | AJaeger: will do. Thanks | 07:39 |
| openstackgerrit | Merged openstack-infra/project-config: Add os_watcher to OpenStack-Ansible https://review.openstack.org/358883 | 07:39 |
| *** thorst has joined #openstack-infra | 07:39 | |
| *** DrifterZA has joined #openstack-infra | 07:40 | |
| *** ifarkas_afk is now known as ifarkas | 07:42 | |
| *** markvoelker has quit IRC | 07:43 | |
| *** matthewbodkin has joined #openstack-infra | 07:43 | |
| *** e0ne has joined #openstack-infra | 07:45 | |
| *** thorst has quit IRC | 07:46 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:48 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 07:48 |
| *** roxanagh_ has joined #openstack-infra | 07:52 | |
| *** yaume has quit IRC | 07:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove DocBook XML publishing for trove https://review.openstack.org/358446 | 07:53 |
| *** yaume has joined #openstack-infra | 07:53 | |
| *** DrifterZA has quit IRC | 07:53 | |
| *** adriant_ has joined #openstack-infra | 07:54 | |
| *** sleviim has joined #openstack-infra | 07:55 | |
| *** rcernin has quit IRC | 07:56 | |
| *** roxanagh_ has quit IRC | 07:56 | |
| *** esikachev has joined #openstack-infra | 07:58 | |
| openstackgerrit | Merged openstack-infra/system-config: Change hpuswest for vanilla on controller and compute node definitions https://review.openstack.org/358992 | 07:58 |
| *** zzzeek has quit IRC | 08:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Install Guide Jobs to Barbican https://review.openstack.org/358769 | 08:00 |
| sshnaidm | the zuul status page shows half of IPs as IPv6, how can I connect to telnet://2001:4800:1ae1:18:f816:3eff:fe45:326f:19885 ?? I don't have IPv6 with my ISP | 08:00 |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| openstackgerrit | Yuval Brik proposed openstack-infra/project-config: Karbor (Smaug) Fullstack Path Fix https://review.openstack.org/359019 | 08:01 |
| AJaeger | sshnaidm: don't you have a place you can login that has IPv6? If not, you really have to wait - we do not have public IPv4 addresses for each of our test nodes. | 08:01 |
| sshnaidm | AJaeger, can't think about such place.. | 08:02 |
| *** _oanson is now known as oanson | 08:02 | |
| *** esikachev has quit IRC | 08:03 | |
| *** ggnel_t has joined #openstack-infra | 08:03 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| sshnaidm | AJaeger, in my country no ISP has IPv6 and even don't plan to have it, like in many others btw | 08:04 |
| *** openstackgerrit has joined #openstack-infra | 08:04 | |
| *** yaume has quit IRC | 08:05 | |
| *** yuanying has quit IRC | 08:05 | |
| *** vsaienko2 has left #openstack-infra | 08:06 | |
| *** hashar has joined #openstack-infra | 08:06 | |
| sshnaidm | NAT is our everything | 08:07 |
| *** adriant_ has quit IRC | 08:07 | |
| *** jtomasek has joined #openstack-infra | 08:07 | |
| AJaeger | sshnaidm: in that case you have to wait until the job has completed, the log files will be available as usual from logs.openstack.org... | 08:07 |
| sleviim | hi anteaya, how are you? | 08:08 |
| *** adriant__ has joined #openstack-infra | 08:11 | |
| Jokke_ | sshnaidm: https://tunnelbroker.net/ | 08:11 |
| Jokke_ | that might help for cases like these | 08:12 |
| *** Goneri has joined #openstack-infra | 08:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Node Launches to nodepool dashboard https://review.openstack.org/358699 | 08:13 |
| openstackgerrit | Merged openstack-infra/project-config: Remove q-lbaas from this tempest list, as it is being removed https://review.openstack.org/358259 | 08:13 |
| adriant__ | AJaeger: any update on this review: https://review.openstack.org/#/c/353818/ | 08:14 |
| sshnaidm | Jokke_, thanks, will try | 08:15 |
| *** lucas-dinner is now known as lucasagomes | 08:15 | |
| *** yamamoto has joined #openstack-infra | 08:16 | |
| sleviim | anteaya: it seems like it works :) | 08:17 |
| *** derekh has joined #openstack-infra | 08:18 | |
| *** esikachev has joined #openstack-infra | 08:18 | |
| *** apetrich has quit IRC | 08:19 | |
| *** apetrich has joined #openstack-infra | 08:21 | |
| *** pgadiya has joined #openstack-infra | 08:22 | |
| AJaeger | adriant__: it needs two core reviewers to look at, it's on my long list after vacation and I'll review eventually - if no other project-config core beats me to it ;) | 08:23 |
| *** esikachev has quit IRC | 08:23 | |
| *** Na3iL has joined #openstack-infra | 08:24 | |
| *** sarob has joined #openstack-infra | 08:24 | |
| openstackgerrit | sandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 08:24 |
| *** coolsvap has joined #openstack-infra | 08:25 | |
| *** sarob has quit IRC | 08:28 | |
| openstackgerrit | Bartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029 | 08:31 |
| *** dingyichen has quit IRC | 08:33 | |
| mrmartin | clarkb, anteaya: I'll check the event duplication on groups.o.o sometimes it happens with the events imported through meetup.com api. | 08:34 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Adding a 1GB swap file to the undercloud. https://review.openstack.org/359035 | 08:35 |
| *** yaume has joined #openstack-infra | 08:35 | |
| *** pt_15 has quit IRC | 08:36 | |
| *** dizquierdo has joined #openstack-infra | 08:39 | |
| *** esikachev has joined #openstack-infra | 08:39 | |
| *** markvoelker has joined #openstack-infra | 08:39 | |
| *** yamamoto has quit IRC | 08:40 | |
| *** ifarkas has quit IRC | 08:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/359041 | 08:41 |
| *** ifarkas has joined #openstack-infra | 08:42 | |
| *** esikachev has quit IRC | 08:43 | |
| *** salv-orl_ has joined #openstack-infra | 08:44 | |
| *** markvoelker has quit IRC | 08:44 | |
| *** salv-orlando has quit IRC | 08:45 | |
| *** thorst has joined #openstack-infra | 08:45 | |
| *** vincentll has quit IRC | 08:46 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Fix vlan on vanilla controller and compute machines https://review.openstack.org/359046 | 08:46 |
| *** vincentll has joined #openstack-infra | 08:48 | |
| *** yamamoto has joined #openstack-infra | 08:48 | |
| *** yaume has quit IRC | 08:48 | |
| *** salv-orl_ has quit IRC | 08:49 | |
| *** salv-orlando has joined #openstack-infra | 08:50 | |
| *** nwkarsten has joined #openstack-infra | 08:51 | |
| *** thorst has quit IRC | 08:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:52 | |
| *** adriant__ has quit IRC | 08:53 | |
| *** nwkarsten has quit IRC | 08:55 | |
| *** yaume has joined #openstack-infra | 08:57 | |
| *** gongysh has joined #openstack-infra | 08:57 | |
| *** electrofelix has joined #openstack-infra | 09:00 | |
| *** eranrom has quit IRC | 09:00 | |
| *** eranrom has joined #openstack-infra | 09:01 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: [NO MERGE] Test write performances https://review.openstack.org/359054 | 09:01 |
| *** ifarkas_ has joined #openstack-infra | 09:02 | |
| *** eranrom has quit IRC | 09:03 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use separated SSL endpoint environment file https://review.openstack.org/356488 | 09:09 |
| *** kaisers_ has joined #openstack-infra | 09:10 | |
| *** _nadya_ has joined #openstack-infra | 09:12 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Correct vanilla Neutron ranges https://review.openstack.org/359058 | 09:12 |
| *** sambetts|afk is now known as sambetts | 09:14 | |
| *** d0ugal has quit IRC | 09:15 | |
| *** d0ugal has joined #openstack-infra | 09:16 | |
| *** gongysh has quit IRC | 09:18 | |
| *** aarefiev_ is now known as aarefiev | 09:19 | |
| *** caowei has quit IRC | 09:20 | |
| *** salv-orlando has quit IRC | 09:21 | |
| *** salv-orlando has joined #openstack-infra | 09:21 | |
| openstackgerrit | Merged openstack-infra/system-config: Fix vlan on vanilla controller and compute machines https://review.openstack.org/359046 | 09:22 |
| *** caowei has joined #openstack-infra | 09:22 | |
| *** berendt has joined #openstack-infra | 09:23 | |
| *** AnarchyAo has joined #openstack-infra | 09:23 | |
| *** AnarchyAo has quit IRC | 09:23 | |
| *** AnarchyAo has joined #openstack-infra | 09:23 | |
| *** AnarchyAo has quit IRC | 09:23 | |
| *** AnarchyAo has joined #openstack-infra | 09:23 | |
| *** AnarchyAo has quit IRC | 09:24 | |
| *** AnarchyAo has joined #openstack-infra | 09:24 | |
| *** AnarchyAo has quit IRC | 09:24 | |
| *** AnarchyAo has joined #openstack-infra | 09:24 | |
| *** AnarchyAo has quit IRC | 09:24 | |
| *** nwkarsten has joined #openstack-infra | 09:27 | |
| AJaeger | adriant: I commented on your review. Once I have an answer to that I can +2. | 09:28 |
| *** Goneri has quit IRC | 09:30 | |
| *** nwkarsten has quit IRC | 09:31 | |
| *** javeriak has joined #openstack-infra | 09:33 | |
| *** pgadiya_ has joined #openstack-infra | 09:33 | |
| *** Goneri has joined #openstack-infra | 09:33 | |
| *** kzaitsev_mb has quit IRC | 09:33 | |
| *** pgadiya has quit IRC | 09:34 | |
| *** lucasagomes is now known as lucas-afk | 09:40 | |
| *** nwkarsten has joined #openstack-infra | 09:40 | |
| *** roxanagh_ has joined #openstack-infra | 09:40 | |
| *** amotoki has joined #openstack-infra | 09:40 | |
| *** markvoelker has joined #openstack-infra | 09:40 | |
| *** tosky has joined #openstack-infra | 09:41 | |
| *** dtantsur|afk is now known as dtantsur | 09:42 | |
| *** jerryz has quit IRC | 09:43 | |
| *** markvoelker has quit IRC | 09:44 | |
| *** nwkarsten has quit IRC | 09:44 | |
| *** roxanagh_ has quit IRC | 09:45 | |
| zigo | AJaeger: Hi there! | 09:47 |
| zigo | AJaeger: Regarding your comment, in which file should I put deb-python-fixtures so that it's officially in packaging-deb? | 09:47 |
| zigo | I forgot which file. | 09:47 |
| openstackgerrit | Volodymyr Stoiko proposed openstack-infra/project-config: Add fuel-plugin-rally project https://review.openstack.org/359076 | 09:51 |
| *** amotoki has quit IRC | 09:51 | |
| *** yanyanhu has quit IRC | 09:52 | |
| *** ifarkas has quit IRC | 09:54 | |
| *** ifarkas_ is now known as ifarkas | 09:54 | |
| AJaeger | zigo: I gave a link in my review to our fine manual, please read and follow it. | 09:55 |
| zigo | Thanks. | 09:55 |
| * zigo hides behind his desk... | 09:55 | |
| AJaeger | No need to hide, I don't plan throwing anything through IRC ;) | 09:56 |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config: Do not run all tempest tests on designate grenade job https://review.openstack.org/359080 | 09:59 |
| *** zhurong has quit IRC | 09:59 | |
| *** tqtran has joined #openstack-infra | 10:01 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add deb-python-fixtures to packaging-deb https://review.openstack.org/358819 | 10:01 |
| *** yaume_ has joined #openstack-infra | 10:02 | |
| *** caowei has quit IRC | 10:04 | |
| *** tqtran has quit IRC | 10:05 | |
| *** yaume has quit IRC | 10:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/359041 | 10:08 |
| *** mikelk has joined #openstack-infra | 10:12 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Use ovs-interface-nondefault instead of ovs-native job https://review.openstack.org/338944 | 10:13 |
| *** mikelk has quit IRC | 10:14 | |
| *** mikelk has joined #openstack-infra | 10:15 | |
| *** ccamacho is now known as ccamacho|afk | 10:15 | |
| *** Julien-zte has quit IRC | 10:18 | |
| *** _degorenko|afk is now known as degorenko | 10:18 | |
| *** mikelk has quit IRC | 10:20 | |
| *** mikelk has joined #openstack-infra | 10:20 | |
| *** sarob has joined #openstack-infra | 10:25 | |
| *** sarob has quit IRC | 10:29 | |
| odyssey4me | I everyone. Now that https://review.openstack.org/358883 has merged, can you please add me to https://review.openstack.org/#/admin/groups/1538,members | 10:30 |
| rcarrillocruz | odyssey4me: done | 10:31 |
| odyssey4me | thanks rcarrillocruz | 10:32 |
| *** kzaitsev_mb has joined #openstack-infra | 10:33 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Pass TRIPLEO_ROOT directory to heat_deploy_times.sh https://review.openstack.org/356946 | 10:35 |
| *** yaume has joined #openstack-infra | 10:36 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Replace hpuswest naming for vanilla on hiera keys https://review.openstack.org/359103 | 10:37 |
| *** javeriak has quit IRC | 10:38 | |
| *** ramishra has joined #openstack-infra | 10:39 | |
| *** kzaitsev_mb has quit IRC | 10:39 | |
| *** yaume_ has quit IRC | 10:39 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Replace hpuswest for vanilla on certfile hiera key https://review.openstack.org/359107 | 10:41 |
| *** amotoki has joined #openstack-infra | 10:42 | |
| AJaeger | rcarrillocruz: could you review https://review.openstack.org/#/c/345441/ , please? The dependency has merged... | 10:42 |
| rcarrillocruz | +A | 10:43 |
| AJaeger | thanks | 10:44 |
| *** javeriak has joined #openstack-infra | 10:45 | |
| *** thorst has joined #openstack-infra | 10:47 | |
| *** thorst has quit IRC | 10:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:52 | |
| *** coolsvap is now known as coolsvap_ | 10:53 | |
| *** amotoki has quit IRC | 10:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Test api-ref theming with openstackdocstheme https://review.openstack.org/345441 | 10:54 |
| *** rodrigods has quit IRC | 10:59 | |
| *** rodrigods has joined #openstack-infra | 10:59 | |
| *** oanson has quit IRC | 10:59 | |
| *** icey has joined #openstack-infra | 11:00 | |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Move 'Save' button up in 'Preferences' page https://review.openstack.org/359119 | 11:00 |
| *** Na3iL has quit IRC | 11:01 | |
| *** dizquierdo is now known as dizquierdo_afk | 11:01 | |
| *** jkilpatr has quit IRC | 11:02 | |
| *** thorst has joined #openstack-infra | 11:04 | |
| openstackgerrit | shizhihui proposed openstack-infra/project-config: Make py35 voting for Horizon https://review.openstack.org/359123 | 11:05 |
| openstackgerrit | Volodymyr Stoiko proposed openstack-infra/project-config: Add fuel-plugin-rally project https://review.openstack.org/359076 | 11:08 |
| *** ccamacho|afk is now known as ccamacho | 11:09 | |
| *** markvoelker has joined #openstack-infra | 11:09 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add management of /etc/nova/ssl/private directory https://review.openstack.org/358668 | 11:15 |
| openstackgerrit | Merged openstack-infra/system-config: Temporarily add rabbit keys to hiera https://review.openstack.org/357021 | 11:15 |
| *** ramishra has quit IRC | 11:20 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Enable SSL for undercloud-only job https://review.openstack.org/359131 | 11:20 |
| *** markvoelker has quit IRC | 11:20 | |
| * AJaeger just got "Could not connect to mirror.regionone.osic-cloud1.openstack.org:80" in http://logs.openstack.org/22/359122/1/check/gate-openstackdocstheme-releasenotes/136d9b8/console.html | 11:21 | |
| *** ramishra has joined #openstack-infra | 11:22 | |
| *** thiagolib has quit IRC | 11:23 | |
| *** sarob has joined #openstack-infra | 11:24 | |
| *** thiagolib has joined #openstack-infra | 11:24 | |
| *** roxanagh_ has joined #openstack-infra | 11:28 | |
| *** sarob has quit IRC | 11:28 | |
| *** dprince has joined #openstack-infra | 11:29 | |
| *** dizquierdo_afk is now known as dizquierdo | 11:30 | |
| *** roxanagh_ has quit IRC | 11:32 | |
| *** jtomasek has quit IRC | 11:37 | |
| *** jkilpatr has joined #openstack-infra | 11:37 | |
| *** ldnunes has joined #openstack-infra | 11:37 | |
| *** ccamacho is now known as ccamacho|lunch | 11:41 | |
| *** nwkarsten has joined #openstack-infra | 11:42 | |
| *** amotoki has joined #openstack-infra | 11:42 | |
| *** rcernin has joined #openstack-infra | 11:43 | |
| *** rfolco has joined #openstack-infra | 11:47 | |
| *** nwkarsten has quit IRC | 11:47 | |
| openstackgerrit | Merged openstack-infra/system-config: Replace hpuswest naming for vanilla on hiera keys https://review.openstack.org/359103 | 11:48 |
| *** jtomasek has joined #openstack-infra | 11:49 | |
| openstackgerrit | Merged openstack-infra/system-config: Replace hpuswest for vanilla on certfile hiera key https://review.openstack.org/359107 | 11:49 |
| openstackgerrit | Volodymyr Stoiko proposed openstack-infra/project-config: Add fuel-plugin-rally project https://review.openstack.org/359076 | 11:49 |
| *** jaosorior has quit IRC | 11:50 | |
| *** asettle has quit IRC | 11:51 | |
| *** jaosorior has joined #openstack-infra | 11:51 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Security update for Panelizer module https://review.openstack.org/359155 | 11:52 |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Add a margin to the bottom of all pages https://review.openstack.org/359119 | 11:56 |
| *** Goneri has quit IRC | 11:57 | |
| *** Na3iL has joined #openstack-infra | 11:57 | |
| *** yamahata has quit IRC | 12:00 | |
| *** pgadiya_ is now known as pgadiya | 12:03 | |
| *** Goneri has joined #openstack-infra | 12:04 | |
| *** xyang1 has joined #openstack-infra | 12:05 | |
| *** ansmith has joined #openstack-infra | 12:07 | |
| *** tpsilva has joined #openstack-infra | 12:08 | |
| openstackgerrit | Ilya Shakhat proposed openstack-infra/project-config: Add new project "os-failures" https://review.openstack.org/355819 | 12:08 |
| *** mordred has quit IRC | 12:10 | |
| *** kaisers_ has quit IRC | 12:11 | |
| *** Shrews has quit IRC | 12:11 | |
| *** zhurong has joined #openstack-infra | 12:11 | |
| openstackgerrit | Volodymyr Stoiko proposed openstack-infra/project-config: Add fuel-plugin-rally project https://review.openstack.org/359076 | 12:12 |
| *** andymaier has quit IRC | 12:13 | |
| *** phschwartz has quit IRC | 12:14 | |
| *** mordred has joined #openstack-infra | 12:15 | |
| *** Shrews has joined #openstack-infra | 12:16 | |
| *** annegentle has joined #openstack-infra | 12:16 | |
| *** esikachev has joined #openstack-infra | 12:18 | |
| *** javeriak has quit IRC | 12:20 | |
| *** rcernin has quit IRC | 12:23 | |
| *** kgiusti has joined #openstack-infra | 12:24 | |
| *** andymaier has joined #openstack-infra | 12:25 | |
| *** gouthamr has joined #openstack-infra | 12:26 | |
| openstackgerrit | Ramana Raja proposed openstack-infra/project-config: remove manila's glusterfs xenial jobs https://review.openstack.org/359167 | 12:27 |
| *** vikasc has left #openstack-infra | 12:27 | |
| *** rhallisey_ has joined #openstack-infra | 12:28 | |
| *** rcernin has joined #openstack-infra | 12:28 | |
| *** abregman has quit IRC | 12:28 | |
| *** kushal has joined #openstack-infra | 12:30 | |
| *** dtardivel has joined #openstack-infra | 12:31 | |
| *** jed56 has joined #openstack-infra | 12:31 | |
| *** jcoufal has joined #openstack-infra | 12:32 | |
| *** coolsvap_ is now known as coolsvap | 12:36 | |
| openstackgerrit | Merged openstack-infra/system-config: Correct vanilla Neutron ranges https://review.openstack.org/359058 | 12:36 |
| *** phschwartz has joined #openstack-infra | 12:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Allow image and flavor by name for create_server https://review.openstack.org/355251 | 12:39 |
| *** mdrabe has joined #openstack-infra | 12:40 | |
| *** edmondsw has joined #openstack-infra | 12:40 | |
| *** asettle has joined #openstack-infra | 12:40 | |
| *** vikrant has quit IRC | 12:41 | |
| *** kushal has quit IRC | 12:42 | |
| *** abregman has joined #openstack-infra | 12:44 | |
| *** rlandy has joined #openstack-infra | 12:45 | |
| mugsie | anyone around to +W https://review.openstack.org/#/c/359080/ ? It is cause gate failures on most patches\ | 12:46 |
| mugsie | it is causing* | 12:46 |
| AJaeger | rcarrillocruz, mordred, yolanda? ^ Any of you around to help mugsie? I've given my +2 already | 12:47 |
| AJaeger | mugsie: that comment would have been nice in the commit message ;) | 12:48 |
| mugsie | AJaeger: yeah, it was kinda rushed :( - I should have | 12:48 |
| mugsie | We are in the mid cycle, trying to get some of our outstanding features merged | 12:48 |
| yolanda | approved | 12:49 |
| *** dtantsur is now known as dtantsur|mtg | 12:50 | |
| zigo | AJaeger: I get a "No space left on device" when building a package, probably because using a ramdisk to build. Do you think it's fine to increase the flavor? | 12:50 |
| zigo | It really was at the end of the build :( | 12:51 |
| AJaeger | zigo, better ask the rest of the team... | 12:51 |
| mugsie | yolanda: thanks! | 12:51 |
| zigo | AJaeger: The other way would be to *not* use a ramdisk, but then it would build slower. | 12:51 |
| rcarrillocruz | sorry, was at lunch | 12:52 |
| *** baoli has joined #openstack-infra | 12:54 | |
| zigo | AJaeger: I'll just disable the tmpfs for now, and then discuss... | 12:54 |
| *** chlong has quit IRC | 12:56 | |
| *** woodster_ has joined #openstack-infra | 12:56 | |
| mordred | zigo: sorry, it's not possible to use a different flavor | 12:58 |
| *** pvinci has joined #openstack-infra | 12:58 | |
| zigo | mordred: Is there only a single flavor type available? | 12:59 |
| mordred | zigo: yah | 12:59 |
| zigo | Ok. | 12:59 |
| mordred | zigo: sorry bout that | 12:59 |
| zigo | mordred: It should be fine without using the ramdisk then. | 12:59 |
| asselin__ | rcarrillocruz, hey, I figured out most of the issues yesterday with launch_node playbook. Next is to figure out the input file change. You seem to be using a different format than what cloud-launch wants. Why no profiles? | 12:59 |
| zigo | mordred: Maybe I could hack something to stop using a ramdisk for only a subset of packages... | 13:00 |
| rcarrillocruz | asselin__: as i explained earlier, that change is to have feature parity to the current launch_node.py | 13:00 |
| rcarrillocruz | you should have your own resources.yaml to feed it the role | 13:00 |
| openstackgerrit | Merged openstack-infra/project-config: Do not run all tempest tests on designate grenade job https://review.openstack.org/359080 | 13:00 |
| *** coolsvap is now known as coolsvap_ | 13:00 | |
| rcarrillocruz | rather than use the playbook that creates it on the fly | 13:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add support for fetching console logs from servers https://review.openstack.org/358232 | 13:00 |
| rcarrillocruz | there's no advantage to use that over launch-node.py | 13:00 |
| *** devananda is now known as devananda|OSE | 13:01 | |
| *** bin_ has joined #openstack-infra | 13:01 | |
| rcarrillocruz | a profile is a way to reuse common resources | 13:02 |
| rcarrillocruz | there's no point in using a profile in the launch-node playbook , as you'll just create one server on the fly | 13:02 |
| *** rhallisey_ is now known as rhallisey | 13:03 | |
| *** tqtran has joined #openstack-infra | 13:03 | |
| *** rcernin has quit IRC | 13:03 | |
| rcarrillocruz | http://git.openstack.org/cgit/openstack-infra/system-config/tree/cloud_launcher/clouds_layouts.yml | 13:03 |
| rcarrillocruz | that's ^ the main purpose for profiles | 13:03 |
| asselin__ | rcarrillocruz, I guess my question then is: how do you use cloud-launcher without a profile | 13:03 |
| rcarrillocruz | you can totally use the launcher role with a profiel | 13:04 |
| rcarrillocruz | just have a cloud with the per-cloud specific resources defined | 13:04 |
| asselin__ | rcarrillocruz, I didn't see it in the docs or the example....and not good enough w/ ansible to rev engineer what the resource files is supposed to look like. | 13:05 |
| rcarrillocruz | http://rcarrillocruz.com/deploying-multiple-openstack-clouds-with-ansible-in-a-data-driven-fashion/ | 13:05 |
| *** DrifterZA has joined #openstack-infra | 13:06 | |
| rcarrillocruz | "'On these items you can either re-use the profiles previously defined by name or define per-cloud specific resources." | 13:06 |
| *** tqtran has quit IRC | 13:07 | |
| mordred | rcarrillocruz: I think we might need to copy that blog post into the docs ... I can never remember where it is | 13:08 |
| *** yuval has joined #openstack-infra | 13:08 | |
| rcarrillocruz | indeed | 13:08 |
| rcarrillocruz | improving docs , as putting something else than the current dummy README, is on my todo list | 13:08 |
| mordred | :) | 13:09 |
| rcarrillocruz | :D | 13:09 |
| mordred | it's always on my todo list | 13:09 |
| *** matt-borland has joined #openstack-infra | 13:09 | |
| AJaeger | mordred: just add a link to the README ;) | 13:09 |
| *** caowei has joined #openstack-infra | 13:10 | |
| asselin__ | rcarrillocruz, ok, thanks I see it now: - name: nonprofilescloud | 13:10 |
| openstackgerrit | Bartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029 | 13:10 |
| AJaeger | project-config cores, I would appreciate review of https://review.openstack.org/#/c/358734/ to get rid of some extra jobs for docs project, please. | 13:10 |
| *** andymaier has quit IRC | 13:12 | |
| *** lucas-afk is now known as lucas-hungry | 13:12 | |
| *** chlong has joined #openstack-infra | 13:13 | |
| *** mikelk has quit IRC | 13:13 | |
| openstackgerrit | Bartosz Kupidura proposed openstack-infra/puppet-apps_site: w[wip] Glare support for app-catalog https://review.openstack.org/359029 | 13:14 |
| *** _ari_ has joined #openstack-infra | 13:16 | |
| *** roxanagh_ has joined #openstack-infra | 13:16 | |
| *** _ari_ has quit IRC | 13:17 | |
| *** javeriak has joined #openstack-infra | 13:17 | |
| *** raunak has joined #openstack-infra | 13:18 | |
| *** hrubi_ has joined #openstack-infra | 13:18 | |
| *** senk has quit IRC | 13:18 | |
| *** hrubi has quit IRC | 13:18 | |
| yuval | Hey, would appreciate you review for Smaug (Karbor) fullstack fix: https://review.openstack.org/#/c/359019/ | 13:20 |
| *** roxanagh_ has quit IRC | 13:21 | |
| *** julim has joined #openstack-infra | 13:21 | |
| yuval | *your :) | 13:21 |
| *** ccamacho|lunch is now known as ccamacho | 13:21 | |
| rcarrillocruz | mordred: was it you who created /root/certs/gencert.sh script or maybe fungi? | 13:21 |
| rcarrillocruz | i wonder why it just creates the csr and key, but not the cert | 13:21 |
| *** _ari_ has joined #openstack-infra | 13:21 | |
| rcarrillocruz | talking about puppetmaster.openstack.org machine btw | 13:21 |
| *** andymaier has joined #openstack-infra | 13:22 | |
| *** Jeffrey4l_ has quit IRC | 13:22 | |
| asselin__ | rcarrillocruz, does this look right? `cuz it doesn't work: http://paste.openstack.org/show/562463/ 'item_cloud' is undefined. | 13:23 |
| *** pgadiya has quit IRC | 13:23 | |
| *** andymaier has quit IRC | 13:23 | |
| rcarrillocruz | asking cos i'm not sure if there's a pattern i should use to create the certs, or can I just create the cert for the infracloud controller with 365 days ? | 13:23 |
| *** andymaier_ has joined #openstack-infra | 13:24 | |
| *** david-lyle has joined #openstack-infra | 13:24 | |
| *** tonytan4ever has joined #openstack-infra | 13:24 | |
| rcarrillocruz | how are you running it asselin__ | 13:25 |
| *** piet has joined #openstack-infra | 13:27 | |
| openstackgerrit | Bartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog https://review.openstack.org/359029 | 13:27 |
| asselin__ | rcarrillocruz, http://paste.openstack.org/show/562465/ | 13:27 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Add management of /etc/nova/ssl/private directory https://review.openstack.org/358668 | 13:29 |
| mordred | rcarrillocruz: twasn't me | 13:30 |
| rcarrillocruz | validate_certs is not an option of the launcher, but an oscc option | 13:30 |
| rcarrillocruz | try to remove it and give it a run | 13:30 |
| anteaya | rcarrillocruz: fungi tends to create certs for infra services | 13:30 |
| *** Julien-zte has joined #openstack-infra | 13:30 | |
| anteaya | that I have seen | 13:30 |
| rcarrillocruz | good, i'll wait for him then, thanks | 13:30 |
| anteaya | welcome | 13:31 |
| anteaya | congratulations for being at the cert stage | 13:31 |
| anteaya | well done | 13:31 |
| rcarrillocruz | well yeah, let's see what other yak i have to shave after i put a sane cert | 13:31 |
| asselin__ | rcarrillocruz, same error http://paste.openstack.org/show/562468/ | 13:32 |
| rcarrillocruz | ansible --version | 13:32 |
| rcarrillocruz | ? | 13:32 |
| *** yaume has quit IRC | 13:32 | |
| *** yaume has joined #openstack-infra | 13:32 | |
| *** nwkarste_ has joined #openstack-infra | 13:33 | |
| asselin__ | http://paste.openstack.org/show/562469/ | 13:33 |
| *** jheroux has joined #openstack-infra | 13:35 | |
| *** sdake has joined #openstack-infra | 13:36 | |
| rcarrillocruz | i don't see it | 13:37 |
| *** sdake_ has joined #openstack-infra | 13:37 | |
| rcarrillocruz | i'll spin a dsvm to check | 13:37 |
| *** jcoufal_ has joined #openstack-infra | 13:38 | |
| *** DrifterZA has quit IRC | 13:38 | |
| *** DrifterZA has joined #openstack-infra | 13:39 | |
| *** raunak has quit IRC | 13:39 | |
| *** dprince has quit IRC | 13:40 | |
| *** jraju has quit IRC | 13:40 | |
| *** raunak has joined #openstack-infra | 13:40 | |
| *** jcoufal has quit IRC | 13:40 | |
| *** sdake has quit IRC | 13:41 | |
| *** mikelk has joined #openstack-infra | 13:41 | |
| *** dprince has joined #openstack-infra | 13:41 | |
| *** raunak has quit IRC | 13:42 | |
| *** sdague has joined #openstack-infra | 13:43 | |
| *** abregman has quit IRC | 13:44 | |
| *** thiagop has joined #openstack-infra | 13:44 | |
| *** sandanar has quit IRC | 13:45 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstackid: Move other-requirements.txt to bindep.txt https://review.openstack.org/354860 | 13:47 |
| *** eandersson_ has quit IRC | 13:47 | |
| *** kushal has joined #openstack-infra | 13:47 | |
| *** eranrom has joined #openstack-infra | 13:49 | |
| *** eranrom has quit IRC | 13:49 | |
| *** piet has quit IRC | 13:50 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add ssl_key_file_contents to compute nodes https://review.openstack.org/359208 | 13:50 |
| *** eranrom has joined #openstack-infra | 13:50 | |
| *** eranrom has quit IRC | 13:51 | |
| *** raildo has joined #openstack-infra | 13:51 | |
| *** eranrom has joined #openstack-infra | 13:52 | |
| *** eranrom has quit IRC | 13:52 | |
| *** javeriak_ has joined #openstack-infra | 13:52 | |
| *** eharney has joined #openstack-infra | 13:53 | |
| *** javeriak has quit IRC | 13:53 | |
| *** paulobanon has joined #openstack-infra | 13:55 | |
| *** kzaitsev_mb has quit IRC | 13:56 | |
| *** tonytan4ever has quit IRC | 13:56 | |
| sdague | fyi, osic looks extra fubar | 13:56 |
| sdague | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22%2Ftmp%2Fansible%2Fbin%2Fansible:%20No%20such%20file%20or%20directory%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1&from=864000s | 13:56 |
| *** eranrom has joined #openstack-infra | 13:56 | |
| sdague | basically it can't produce multinode envs correctly | 13:56 |
| sdague | this is why there is a 16 hr gate | 13:57 |
| mordred | yes. this is known. it's not supposed to be in the multi-node providers | 13:57 |
| * mordred looks | 13:57 | |
| sdague | mordred: actually, it's even fubar on single node it seems | 13:57 |
| mordred | ok. that's a different thing | 13:57 |
| *** dprince has quit IRC | 13:58 | |
| mordred | sdague: is it running multi-node test though? | 13:58 |
| sdague | gate-tempest-dsvm-neutron-full-ubuntu-xenial - 42 failures in 24 hours osic | 13:58 |
| timrc | I don't want to be annoying, but I really want to understand the problem that occured yesterday with osc. It seems like osc was tested with older packages than what it actually installed with. This happened because a new version of occ was released after osc passed tests which caused a breaking change (that would have otherwise been caught). This seems like quite the race condition. Do I have | 13:58 |
| timrc | this right? If so, my question is why don't automatically propose a new requirements.txt for packages like osc which include the global requirements and upper constraints that were actually used to pass tests? This would eliminate such a race condition... I'm sure I'm missing something though. | 13:58 |
| *** dprince has joined #openstack-infra | 13:58 | |
| *** hongbin has joined #openstack-infra | 13:58 | |
| sdague | mordred: in this failure cluster, I don't see any | 13:58 |
| mordred | timrc: we just simply didn't have a gate job. we have added the gate job that was missing, so we should be good now | 13:59 |
| *** kzaitsev_mb has joined #openstack-infra | 13:59 | |
| sdague | http://status.openstack.org/elastic-recheck/ | 13:59 |
| paulobanon | #openstack-release | 13:59 |
| *** kaisers_ has joined #openstack-infra | 13:59 | |
| *** piet has joined #openstack-infra | 13:59 | |
| sdague | I'm at openstack days east, so real debug is hard, but given this failure rate, osic should probably be fully disabled | 14:00 |
| sdague | otherwise no code is going to merge this week | 14:00 |
| *** xarses has quit IRC | 14:01 | |
| rcarrillocruz | asselin__: looks like pypi ansible 2.1.1.0 has include/with_items nested broken | 14:01 |
| rcarrillocruz | try this | 14:01 |
| rcarrillocruz | pip install ansible==2.1.0.0 | 14:01 |
| rcarrillocruz | and let me know if you no longer get that item_cloud failure | 14:01 |
| *** eranrom has quit IRC | 14:02 | |
| *** irtermite has joined #openstack-infra | 14:02 | |
| *** eranrom has joined #openstack-infra | 14:02 | |
| irtermite | @all at some point today, we will be updating the ssl cert for cloud1.osic.org. If you notice anything odd, hit me here. | 14:02 |
| *** tqtran has joined #openstack-infra | 14:02 | |
| *** pcaruana has quit IRC | 14:02 | |
| *** oanson has joined #openstack-infra | 14:02 | |
| rcarrillocruz | and as a matter of fact, let me throw something at the gate | 14:02 |
| irtermite | ping cloudnull | 14:02 |
| mordred | sdague: thanks - poking at it fo sho | 14:03 |
| *** eranrom has quit IRC | 14:03 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs https://review.openstack.org/359215 | 14:03 |
| *** eranrom has joined #openstack-infra | 14:03 | |
| *** david-lyle has quit IRC | 14:04 | |
| *** esberglu has joined #openstack-infra | 14:04 | |
| *** kaisers_ has quit IRC | 14:04 | |
| odyssey4me | mordred sdague It may be useful to provide some sort of filter to allow jobs from specific repositories to go to specific providers. Or perhaps long running jobs (ie !docs, !releasenotes, !linters, etc) to go to specific providers. | 14:04 |
| timrc | mordred: So provided there's complete test coverage, it's guaranteed that any dep a package like osc implicitly installs should be non-breaking? | 14:04 |
| odyssey4me | Blocking a whole provider just because one set of jobs are failing seems counter-productive. | 14:04 |
| *** yuval has quit IRC | 14:04 | |
| *** tbarron|gone is now known as tbarron | 14:05 | |
| *** coolsvap_ is now known as coolsvap | 14:05 | |
| odyssey4me | I can, for instance, confirm that OSIC jobs are working perfectly for OpenStack-Ansible jobs - even if they aren't for devstack jobs. | 14:05 |
| irtermite | odyssey4me: *thumbsup* | 14:05 |
| *** bhunter71 has joined #openstack-infra | 14:05 | |
| asselin__ | rcarrillocruz, yup, that error goes away | 14:06 |
| sdague | odyssey4me: well, you have to change all the job definitions to chunk them up that way. If you want to build a special class for some of these, go ahead | 14:07 |
| *** berendt has quit IRC | 14:07 | |
| sdague | but it becomes a logical mess to manage that given the 5000+ job definitions | 14:07 |
| zigo | mordred: Could you please remove the current python-cryptography-vectors from the debian-openstack repo? The version from sid of python-cryptography fails to build, I'd like to use the official backport instead. Also, we need to figure out a way so that *we* can do such operation. Your thougths would be welcome. | 14:07 |
| mordred | http://logs.openstack.org/71/343571/11/check/gate-tempest-dsvm-neutron-full-ubuntu-trusty-mitaka/93ac9e9/console.html#_2016-08-23_12_04_43_696209 | 14:07 |
| mordred | sdague: ^^ | 14:07 |
| *** tonytan4ever has joined #openstack-infra | 14:08 | |
| mordred | odyssey4me: Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.requests.packages.urllib3.connection.HTTPConnection object at 0x7f004550f950>, 'Connection to mirror.regionone.osic-cloud1.openstack.org timed out. (connect timeout=60.0)')': /pypi/simple/paramiko/ | 14:08 |
| *** rods has quit IRC | 14:08 | |
| odyssey4me | mordred oh dear, bad mirror | 14:08 |
| rcarrillocruz | as such, i expect https://review.openstack.org/#/c/359216/ to fail | 14:08 |
| rcarrillocruz | i'll see | 14:08 |
| *** hieulq_ has joined #openstack-infra | 14:08 | |
| rcarrillocruz | tehre are 2-3 bugs opened for include/with_items and loop_var, must be one of them | 14:09 |
| asselin__ | rcarrillocruz, how is this related to shade? | 14:09 |
| mordred | the mirror seems to be returning for me at the moment from my laptop | 14:09 |
| odyssey4me | mordred that's weird though, because we've had several successful jobs on OSIC today | 14:09 |
| rcarrillocruz | nothing | 14:09 |
| rcarrillocruz | it's just some dummy change | 14:09 |
| * mordred poking further | 14:09 | |
| *** eantyshev has joined #openstack-infra | 14:09 | |
| *** rbrndt has joined #openstack-infra | 14:09 | |
| * odyssey4me heads to logstash.o.o | 14:09 | |
| rcarrillocruz | i had to revert it regardless | 14:09 |
| rcarrillocruz | but that gave me perfect excuse to see the behaviour in the gate | 14:09 |
| sdague | odyssey4me: the number of nodes in osic just doubled yesterday | 14:09 |
| asselin__ | rcarrillocruz, got it. | 14:09 |
| *** rods has joined #openstack-infra | 14:10 | |
| AJaeger | I had this mirror problem earlier today as well: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/latest.log.html#t2016-08-23T11:21:35 | 14:10 |
| sdague | once those all got consumed I can imagine it overwhelming the mirrors | 14:10 |
| *** lucas-hungry is now known as lucasagomes | 14:10 | |
| odyssey4me | sdague hmm, yeah - so a scale issue which has never been seen before because no one provider has ever given so many nodes in a single region | 14:10 |
| odyssey4me | interesting | 14:10 |
| *** eranrom has quit IRC | 14:10 | |
| AJaeger | 3 jobs for the same change got the timeouts. And then another time later one - several jobs for same change | 14:10 |
| asselin__ | rcarrillocruz, ok I really see now: it should install latest version of ansible to show the bug: ansible>=2.1.0.0 | 14:11 |
| asselin__ | rcarrillocruz, what's strange is it doesn't happen when using a profile | 14:11 |
| sdague | odyssey4me: anyway, right now the fail rate basically has destroyed code merge | 14:11 |
| rcarrillocruz | asselin__: because the looping mechanism differs | 14:11 |
| *** eranrom has joined #openstack-infra | 14:11 | |
| rcarrillocruz | the logic is quite different | 14:12 |
| odyssey4me | asselin__ rcarrillocruz does that happen to nivolve with_flattenned ? | 14:12 |
| rcarrillocruz | even though i refactored code and reuse as much as possible | 14:12 |
| rcarrillocruz | odyssey4me: i don't use with_flattened on the role so i can't really tell | 14:12 |
| *** ddieterly has joined #openstack-infra | 14:12 | |
| asselin__ | rcarrillocruz, btw, I really like the refactored code compared to when I last used it. (multiple files vs all in one file) | 14:12 |
| rcarrillocruz | yup | 14:13 |
| rcarrillocruz | there was a ton of duplicated code | 14:13 |
| rcarrillocruz | much cleaner now | 14:13 |
| odyssey4me | rcarrillocruz yeah, with_flattened is notorious for bad behaviour - just don't use it | 14:13 |
| mordred | I just jumped on an OSIC node and it is able to contact th emirror | 14:13 |
| mordred | so there don't seem to be systemic routing issues between osic nodes and the mirror | 14:14 |
| sdague | mordred: just because it worked once, doesn't mean it's not a real issue | 14:15 |
| mordred | sdague: sigh. really? wow, that's super helpful | 14:16 |
| mordred | come on man | 14:16 |
| *** zz_dimtruck is now known as dimtruck | 14:17 | |
| *** pcaruana has joined #openstack-infra | 14:17 | |
| *** reed has quit IRC | 14:17 | |
| *** edtubill has joined #openstack-infra | 14:17 | |
| pabelanger | morning | 14:18 |
| odyssey4me | mordred sdague the trend of successful jobs for OSIC in logstash is pretty good | 14:18 |
| *** reed has joined #openstack-infra | 14:19 | |
| * odyssey4me tried to figure out how to share a search | 14:19 | |
| *** mikelk has quit IRC | 14:19 | |
| mordred | sdague: this stopped happening two hours ago as best I can tell from that logstash query | 14:19 |
| *** rajinir has joined #openstack-infra | 14:19 | |
| mordred | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22%2Ftmp%2Fansible%2Fbin%2Fansible:%20No%20such%20file%20or%20directory%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1&from=864000s | 14:19 |
| mordred | and it seems to have been an issue for about 30 minutes | 14:19 |
| sdague | mordred: ok | 14:20 |
| odyssey4me | oh, that's a fun one | 14:20 |
| *** calebb has quit IRC | 14:20 | |
| mordred | pabelanger: in the apache error log on that mirror server, it's listing [Tue Aug 23 12:37:06.614393 2016] [core:notice] [pid 2849:tid 140327933454208] AH00051: child pid 5643 exit signal Segmentation fault (11), possible coredump in /etc/apache2 | 14:21 |
| sdague | it actually looks like there are 2 spikes | 14:21 |
| sdague | one at 5am, and one at 8am | 14:21 |
| mordred | there are 7 seg faults | 14:21 |
| mordred | apache segfaulting seems like a Bad Thing | 14:22 |
| mordred | but there's no additional info | 14:22 |
| pabelanger | did that just start happening? | 14:22 |
| *** oanson has quit IRC | 14:23 | |
| mordred | pabelanger: nope. | 14:23 |
| mordred | pabelanger: there are 8 in error.log.1 | 14:23 |
| sdague | ok, running out of battery and need to give up my seat. mordred thanks for looking. | 14:23 |
| mordred | [Mon Aug 22 06:25:56.562139 2016] [mpm_event:notice] [pid 2849:tid 140327933454208] AH00493: SIGUSR1 received. Doing graceful restart | 14:23 |
| mordred | sdague: we'll get it sorted - sorry for the toruble | 14:24 |
| mordred | pabelanger: that log line above ^^ | 14:24 |
| pabelanger | Hmm | 14:24 |
| mordred | is the graceful restart after which this started happening | 14:24 |
| sdague | yeh, no worries, glad it looks like it may have self resolved | 14:24 |
| pabelanger | I wonder if other mirrors are doing that | 14:24 |
| mordred | sdague: well, I'd love to find root cause - serving static files from mirrors shoudl be a fairly rocksolid thing | 14:24 |
| pabelanger | we updated logrotate the other day | 14:24 |
| mordred | pabelanger: worth checking | 14:24 |
| pabelanger | maybe it is misconifugred | 14:25 |
| *** amitgandhinz has quit IRC | 14:25 | |
| mordred | here is error.log.1 : http://paste.openstack.org/show/562480/ | 14:25 |
| mordred | here is error.log: http://paste.openstack.org/show/562481/ | 14:26 |
| mordred | there are no segfaults in error.log.2.gz | 14:26 |
| *** dtantsur|mtg is now known as dtantsur | 14:26 | |
| *** amitgandhinz has joined #openstack-infra | 14:26 | |
| *** amitgandhinz has quit IRC | 14:26 | |
| mordred | it would be neat if there WAS a core dump | 14:26 |
| pabelanger | ya, we need to enable that in apache | 14:27 |
| *** amitgandhinz has joined #openstack-infra | 14:27 | |
| pabelanger | also, we haven't setup ipv6 DNS records on osic-cloud1 mirror | 14:27 |
| pabelanger | doing that now | 14:27 |
| mordred | cool | 14:27 |
| mordred | that'll be nice | 14:27 |
| *** sdague has quit IRC | 14:28 | |
| *** david-lyle has joined #openstack-infra | 14:28 | |
| AJaeger | pabelanger, mordred: Once you fixed the mirror, could either of you review https://review.openstack.org/#/c/358734/ , please? That removes some jobs for docs team. | 14:28 |
| *** adam_g has quit IRC | 14:29 | |
| pabelanger | Ah, we'll check to schedule the work for ipv6 on osic-cloud1 mirror | 14:29 |
| pabelanger | it lacks ipv6 right now | 14:29 |
| AJaeger | thanks, mordred ! | 14:30 |
| mordred | pabelanger: aroo? | 14:30 |
| mordred | pabelanger: OH | 14:31 |
| rcarrillocruz | asselin__: https://github.com/ansible/ansible/issues/17148 looks like a good candidate | 14:31 |
| dulek | Hi, can we get https://review.openstack.org/#/c/355678/ in? This will make running Cinder multinode grenade tests easier on patches in review. | 14:31 |
| pabelanger | I don't see anything in syslog that would restart apache | 14:31 |
| mordred | pabelanger: we made that mirror before there was ipv6 in osic | 14:31 |
| pabelanger | mordred: yes | 14:31 |
| *** gbraad has quit IRC | 14:31 | |
| *** hieulq__ has joined #openstack-infra | 14:31 | |
| mordred | nod. this makes sense to me | 14:32 |
| *** jaosorior is now known as jaosorior_away | 14:32 | |
| *** gbraad has joined #openstack-infra | 14:32 | |
| openstackgerrit | Peter Stachowski proposed openstack-infra/project-config: [trove] Add more nv scenario tests https://review.openstack.org/354881 | 14:32 |
| *** hieulq_ has quit IRC | 14:33 | |
| *** adam_g has joined #openstack-infra | 14:34 | |
| *** adam_g has quit IRC | 14:34 | |
| *** adam_g has joined #openstack-infra | 14:34 | |
| *** mikelk has joined #openstack-infra | 14:34 | |
| *** sdake_ has quit IRC | 14:35 | |
| pabelanger | Aug 23 06:25:01 mirror CRON[27393]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )) | 14:35 |
| pabelanger | apache2 restarts line up with the cron.daily logrotate job | 14:35 |
| mordred | ok. well that's good | 14:35 |
| pabelanger | ya | 14:35 |
| pabelanger | now for coredump | 14:35 |
| *** piet has quit IRC | 14:35 | |
| mordred | now I guess the question is - did we upgrade apache or something in between those restarts? | 14:36 |
| *** davidlenwell has quit IRC | 14:36 | |
| *** vern has quit IRC | 14:36 | |
| pabelanger | doesn't look like it | 14:36 |
| pabelanger | nothing in /var/log/apt | 14:37 |
| pabelanger | well, nothing related to apache2 | 14:37 |
| *** davidlenwell has joined #openstack-infra | 14:37 | |
| pabelanger | so, we have no swap | 14:38 |
| pabelanger | I wonder if we are OOMing | 14:38 |
| openstackgerrit | Merged openstack-infra/groups: Security update for Panelizer module https://review.openstack.org/359155 | 14:38 |
| mordred | wouldn't that show up as oomkiller though? | 14:38 |
| mordred | it certainly doesn't seem like we have extra memory though | 14:39 |
| pabelanger | ya, don't see oomkiller in logs | 14:39 |
| *** esikachev has quit IRC | 14:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Cleanup DocBook XML publishing https://review.openstack.org/358734 | 14:40 |
| *** xarses has joined #openstack-infra | 14:40 | |
| *** sdake has joined #openstack-infra | 14:41 | |
| mordred | pabelanger: I don't see any spikes, leaks or anything else in cacti graphs :( | 14:41 |
| mordred | pabelanger: we haven't updated any packages there since aug 19 | 14:42 |
| *** sleviim has quit IRC | 14:44 | |
| mordred | pabelanger: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22%2Ftmp%2Fansible%2Fbin%2Fansible:%20No%20such%20file%20or%20directory%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1&from=864000s | 14:46 |
| *** calebb has joined #openstack-infra | 14:48 | |
| pabelanger | ya, lines up when mirror was down | 14:49 |
| *** tqtran has quit IRC | 14:49 | |
| pabelanger | we need to add: CoreDumpDirectory /var/cache/apache2/ | 14:50 |
| pabelanger | into apache2.conf | 14:50 |
| *** Goneri has quit IRC | 14:52 | |
| pabelanger | just had another 1 too | 14:53 |
| pabelanger | [Tue Aug 23 14:39:47.398544 2016] [core:notice] [pid 2849:tid 140327933454208] AH00051: child pid 6620 exit signal Segmentation fault (11), possible coredump in /etc/apache2 | 14:53 |
| *** javeriak_ has quit IRC | 14:53 | |
| mordred | WEIRD | 14:54 |
| *** david-lyle has quit IRC | 14:54 | |
| *** raunak has joined #openstack-infra | 14:55 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Replace ssl cert for infracloud vanilla controller https://review.openstack.org/359254 | 14:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Ensure per-resource caches work without global cache https://review.openstack.org/358776 | 14:57 |
| openstackgerrit | Merged openstack-infra/shade: Allow object storage endpoint to return 404 for missing /info endpoint https://review.openstack.org/358937 | 14:57 |
| *** yamamoto has quit IRC | 14:57 | |
| pabelanger | mordred: I'm going to put osic-cloud1 mirror into emergency and manually enable coredumps. I'll get a patch up for puppet too, but we'll need to land it off hours since it requires apache2 restart | 14:57 |
| *** pvinci has quit IRC | 14:58 | |
| pabelanger | Actually, we need to decided if we want to restart mirror osic-cloud1 to pick up the config change now | 14:58 |
| pabelanger | or just wait until we land the puppet patch | 14:58 |
| *** weshay is now known as weshay_afk | 14:58 | |
| *** hockeynut has joined #openstack-infra | 14:58 | |
| *** hieulq__ has quit IRC | 15:01 | |
| *** yamamoto has joined #openstack-infra | 15:01 | |
| *** vinaypotluri has joined #openstack-infra | 15:01 | |
| *** hieulq_ has joined #openstack-infra | 15:01 | |
| *** vern has joined #openstack-infra | 15:03 | |
| nwkarste_ | it looks like the openstackci puppet module hasn't been updated with the new split logstash::indexer parameters https://github.com/openstack-infra/puppet-openstackci/blob/master/manifests/logstash_worker.pp#L49 https://github.com/openstack-infra/puppet-logstash/blob/master/manifests/indexer.pp#L38 | 15:06 |
| *** armax has joined #openstack-infra | 15:06 | |
| *** yamamoto has quit IRC | 15:06 | |
| anteaya | nwkarste_: have you considered offering a patch? | 15:07 |
| *** edtubill has quit IRC | 15:07 | |
| *** Goneri has joined #openstack-infra | 15:07 | |
| nwkarste_ | anteaya: sure i'll do it | 15:07 |
| dhellmann | has someone already reported the tarball job failure for smaugclient? http://logs.openstack.org/74/74a8a033aafbc0cdc6f984b2ffb4cd327498fbd6/release/python-smaugclient-tarball/9e47bfe/console.html | 15:07 |
| *** weshay_afk is now known as weshay | 15:07 | |
| anteaya | nwkarste_: wonderful | 15:08 |
| fungi | rcarrillocruz: skimming scrollback, on a conference call right now, but didn't we work out a way to use self-signed certs for the last infra-cloud deployment? | 15:08 |
| rm_work | Hey, so ... *some* of the Zuul telnet links for running jobs are using IPv6 links ... which is great! Except, my corp network doesn't support IPv6 internally... T_T | 15:08 |
| rm_work | Is that expected? | 15:08 |
| clarkb | rm_work: yes | 15:08 |
| fungi | rm_work: yes, some of our job nodes only have ipv6 addresses | 15:08 |
| clarkb | mordred: pabelanger what puppet change for the mirror? | 15:08 |
| *** salv-orlando has quit IRC | 15:08 | |
| timrc | Latest shade does not install successfully in a clean venv. It breaks installing positional which breaks if pytz is not already installed. If you install pytz first and then install shade, things seem good. | 15:08 |
| *** jtomasek has quit IRC | 15:08 | |
| *** salv-orlando has joined #openstack-infra | 15:08 | |
| fungi | rm_work: if the node has a global ipv4 address we list that for the console, and any that only have global v6 we fall back on that for the url | 15:09 |
| pabelanger | clarkb: I'm writing a patch to enable coredumps for apache2 | 15:09 |
| mordred | timrc: oh fantastic | 15:09 |
| pabelanger | but will require apache2 to restart | 15:09 |
| rm_work | ok, interesting | 15:09 |
| anteaya | dhellmann: I have not seen a tarball job failure reported yet personally, no | 15:09 |
| rm_work | so I'm basically out of luck until it's done and posted | 15:09 |
| clarkb | pabelanger: gotcha | 15:09 |
| fungi | pabelanger: trying to track down those intermittent segfaults we get with the event worker on trusty? | 15:09 |
| mordred | timrc: I did not experience the problem you describe | 15:10 |
| rm_work | or unless I can get an ipv6 link working :P | 15:10 |
| clarkb | rm_work: or set up a tunnel of some sort | 15:10 |
| rm_work | yeah | 15:10 |
| rm_work | working on that | 15:10 |
| rcarrillocruz | fungi: yeah, i generated a self-signed cert on the puppetmaster machine by using the gencert.sh , then openssl command to craete cert from csr+key | 15:10 |
| clarkb | timrc: I am not able to reproduce that behavior | 15:10 |
| timrc | Hrm | 15:10 |
| clarkb | timrc: make sure your virtualenv is up to date | 15:10 |
| rcarrillocruz | is it ok if i leave the csr and key there? /root/certs | 15:10 |
| fungi | pabelanger: turning on coredumps was going to be my next step for looking into that but i never got to it | 15:10 |
| dhellmann | anteaya : ok, thanks. I've pointed it out to saggi in #openstack-dev and since smaug is an independent project I'll let the team work on debugging it | 15:10 |
| timrc | Let me do a pastebin. | 15:10 |
| openstackgerrit | Michal Dulko proposed openstack-infra/project-config: Move cinder multinode grenade job to check https://review.openstack.org/359275 | 15:10 |
| fungi | rcarrillocruz: yeah, that's fine by me | 15:10 |
| rcarrillocruz | was more looking on the usual way of storing that stuff really | 15:11 |
| anteaya | dhellmann: very good | 15:11 |
| fungi | rcarrillocruz: it's mainly been our staging area for generating csrs and then dumping the resulting keys/certs and chain certs into hiera | 15:11 |
| rcarrillocruz | i genreated a new one with 365 days | 15:11 |
| rcarrillocruz | ++ | 15:11 |
| rcarrillocruz | just pushed to gate the new cert | 15:11 |
| openstackgerrit | Merged openstack-infra/nodepool: Don't delete building DIB images https://review.openstack.org/358843 | 15:12 |
| clarkb | timrc: since virtualenv bundles pip and setuptools | 15:13 |
| mordred | clarkb: I am doubtful we're going to be able to release shade with the needed patches today - purely due to gate depth. I could be wrong, of course, but I'm currently pessimistic that it'll happen before tomorrow | 15:13 |
| mordred | clarkb: in positive news though, the shade-nodepool-dsvm job totally caught a bug | 15:14 |
| timrc | clarkb: mordred: Here's the paste: http://paste.openstack.org/show/77jv1TCSFnMIHJyBXgtW/ | 15:14 |
| mordred | timrc: thanks | 15:14 |
| clarkb | timrc: ya try updating virtualenv or at least upgrade setuptools and pip in the virtualenv before installing shade | 15:15 |
| *** Julien-zte has quit IRC | 15:15 | |
| fungi | okay, mediawiki finally released their security fixes after i disappeared last night, so i'll be working on that between now and the meeting https://lists.wikimedia.org/pipermail/mediawiki-announce/2016-August/000195.html | 15:15 |
| mordred | timrc: that's a really old virtualenv. yah - what clarkb said | 15:15 |
| anteaya | fungi: yay | 15:16 |
| timrc | clarkb: Let me try... this VM should be getting booted with a daily built Trusty image though so hrm :/ | 15:16 |
| rm_work | clarkb / fungi: is telnet://2001:4800:1ae1:18:f816:3eff:feb9:537:19885 connectable for you? I'm trying from a machine that DOES have verified working ipv6 connectivity and not getting a connection | 15:16 |
| mordred | timrc: fwiw, I never use distro packaged virtualenv - but if you do need to, definitely update pip/setuptools in the venv before you do anything in it to make it useful | 15:16 |
| mordred | for that matter, I never use distro packaged pip either - but that's not your issue here | 15:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Enabled coredumps for apache2 on AFS mirrors https://review.openstack.org/359278 | 15:16 |
| *** annegentle has quit IRC | 15:16 | |
| pabelanger | fungi: clarkb: first stab^ | 15:16 |
| *** klindgren has quit IRC | 15:17 | |
| fungi | timrc: one way around it if you're using the distro package of virtualenv is to virtualenv an intermediary venv, pip install latest virtualenv inside that, and then run that virtualenv to create your desired venvs | 15:17 |
| fungi | that's my usual bootstrapping trick, since i detest pip installing anything system-wide | 15:17 |
| *** ePrVRSBBhG has joined #openstack-infra | 15:17 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Remove unnecessary NodePoolBuilder thread https://review.openstack.org/356676 | 15:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add new ZK method for sending cluster heartbeat https://review.openstack.org/358868 | 15:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add new ZK method for registering a watch. https://review.openstack.org/358837 | 15:17 |
| fungi | i just keep a tree of venvs in my homedir with the tools i use, and deep-link from ~/bin/whatever to ~/pyenvs/whatever/bin/whatever | 15:18 |
| anteaya | fungi: I've advised smaug folks to add an infra meeting agenda item about renames, since they want one. I have told them it won't be prior to feature freeze and them attending is the best way to have one scheduled | 15:19 |
| anteaya | in case they show up but don't have an agenda item | 15:19 |
| timrc | clarkb, fungi, mordred: Cool. Thanks for the help. | 15:19 |
| cloudnull | mornings | 15:19 |
| fungi | anteaya: sounds good, we already have another project with a rename requested too, so will likely address them both at the same time | 15:19 |
| clarkb | rm_work: egat us your telnet/nc command? | 15:19 |
| anteaya | fungi: thought as much, thank you | 15:19 |
| clarkb | rm_work: note tge trailing 19885 is the port not part of the addr | 15:19 |
| rm_work | telnet 2001:4800:1ae1:18:f816:3eff:feb9:537 19885 | 15:19 |
| *** andymaier_ has quit IRC | 15:19 | |
| pabelanger | clarkb: I haven't seen an ubuntu-xenial launch failure since you uploaded the images last night | 15:19 |
| anteaya | cloudnull: morning, so some backscroll for you | 15:19 |
| pleia2 | good morning | 15:19 |
| *** ePrVRSBBhG has quit IRC | 15:20 | |
| anteaya | cloudnull: something about being able to find the mirror on osic? | 15:20 |
| anteaya | moring pleia2 | 15:20 |
| clarkb | pabelanger: huh also only ovh gra1 and osic got new images all the othera failed :/ | 15:20 |
| fungi | rm_work: not all telnet clients have ipv6 support by default. also it's not really a telnet server just a streaming tcp socket, you might find it saner to install netcat-openbsd and then use the nc command instead of telnet | 15:20 |
| rm_work | hmm k | 15:20 |
| mordred | cloudnull: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22%2Ftmp%2Fansible%2Fbin%2Fansible:%20No%20such%20file%20or%20directory%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1&from=864000s | 15:20 |
| pabelanger | clarkb: boo | 15:20 |
| fungi | (note that netcat-traditional also does not support rav v6 addresses, but netcat-openbsd does) | 15:20 |
| clarkb | pabelanger: I will requeue the others shortly in hopes of getting d-g updated sometime soon | 15:20 |
| mordred | cloudnull: we are also looking at things on the node, since there were some segfaults | 15:21 |
| SamYaple | mornings cloudnull | 15:21 |
| fungi | s/rav/raw/ | 15:21 |
| *** ifarkas is now known as ifarkas_afk | 15:21 | |
| rm_work | fungi / clarkb: I tested with "telnet -6 google.com 80" and it connects successfully via ipv6, that's why i was asking if that server worked for you guys or not | 15:21 |
| SamYaple | netcat-openbsd is the only syntax i know | 15:21 |
| mordred | but the segfaults were just a few and there was 30 minutes of lack of connectivity - so we're not really sure what the heck was going on | 15:21 |
| *** dizquierdo has quit IRC | 15:22 | |
| *** DmZDsfZoQv has joined #openstack-infra | 15:22 | |
| fungi | rm_work: i'm getting no response out of 2001:4800:1ae1:18:f816:3eff:feb9:537 (not even with ping6) so the node has probably already been deleted | 15:22 |
| *** baoli has quit IRC | 15:22 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Set whole disk image options directly in devstack https://review.openstack.org/359285 | 15:22 |
| fungi | rm_work: is it for a currently running job? | 15:22 |
| rm_work | fungi: yes the job is still running according to zuul | 15:22 |
| *** baoli has joined #openstack-infra | 15:23 | |
| rm_work | 286381,14 | 15:23 |
| *** david-lyle has joined #openstack-infra | 15:23 | |
| fungi | yeah, nodepool hasn't deleted it yet | 15:23 |
| *** piet has joined #openstack-infra | 15:23 | |
| zigo | pabelanger: Hey there! Could you please remove python-cryptography-vectors from the repo? I would like to use the version from official jessie-backports instead. | 15:23 |
| jeblair | fungi, rm_work: no answer on ssh either | 15:23 |
| zigo | pabelanger: The version from Sid fails to build ... | 15:23 |
| zigo | pabelanger: That's a major blocker for other stuff to build. | 15:24 |
| rm_work | jeblair: so, this node just doesn't like me T_T | 15:24 |
| mordred | rm_work: it has emotional issues | 15:24 |
| fungi | | b08d8dc2-373c-4883-aa6b-0d69db680e25 | ubuntu-trusty-osic-cloud1-3788505 | ACTIVE | GATEWAY_NET_V6=2001:4800:1ae1:18:f816:3eff:feb9:537, 10.0.77.47 | template-ubuntu-trusty-1471909100 | | 15:24 |
| rm_work | and by me, i mean, the universe hates me today | 15:24 |
| Shrews | pabelanger: can you +A these nodepool reviews that add example files? https://review.openstack.org/357329 and https://review.openstack.org/357330 | 15:24 |
| mordred | heh. I already +2'd them | 15:24 |
| Shrews | mordred: i just want them in to get the default branch change in that you already +A'd :) | 15:25 |
| mordred | Shrews: yah | 15:25 |
| *** abregman has joined #openstack-infra | 15:25 | |
| *** andreas_s has quit IRC | 15:25 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/devstack-gate: Do not set ephemeral size based on driver https://review.openstack.org/326061 | 15:25 |
| Shrews | mordred: you could push them through too if you feel so inclined | 15:26 |
| cloudnull | mordred: so the mirrors are busted or do we suspect there's a routing issue at the edge? | 15:26 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Run host lookup first for configure_mirror.sh https://review.openstack.org/359289 | 15:26 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Include dib-builddate.txt for configure_mirror.sh https://review.openstack.org/359290 | 15:26 |
| fungi | rm_work: any chance https://review.openstack.org/286381 could nuke the network on the job node? | 15:26 |
| jeblair | rm_work, fungi: so i wonder if either the node has hung, or if some of the network configuration changes that job does (gate-neutron-lbaasv2-dsvm-api-namespace-nv) have affected our ability to connect from the outside. | 15:26 |
| jeblair | fungi: right that :) | 15:26 |
| mordred | cloudnull: right now they're working - so I have no legit clue why they were not working for that period | 15:26 |
| pabelanger | Shrews: looks like mordred has you covered | 15:27 |
| Shrews | aye. danke | 15:27 |
| mordred | cloudnull: pabelanger is adding coredump config to apache so we can look into the segfault | 15:27 |
| *** davidlenwell has quit IRC | 15:27 | |
| rm_work | fungi: I don't think so ... | 15:27 |
| cloudnull | maybe an issue at the DFW edge? | 15:27 |
| pabelanger | zigo: remove from which repo? | 15:27 |
| mordred | cloudnull: but yeah - it's possible there was a 30 minute issue | 15:27 |
| cloudnull | oh, was the mirror server segfaulting? | 15:27 |
| mordred | also - we dont have ipv6 on the mirror server | 15:27 |
| *** njohnston has left #openstack-infra | 15:27 | |
| mordred | so all of the nodes are bouncing through the neutron router too | 15:27 |
| mordred | so one could imagine an issue with neutron for a bit | 15:27 |
| cloudnull | ++ | 15:28 |
| cloudnull | I can look into that | 15:28 |
| mordred | cloudnull: yah - yesterday we started having occasional apache segfaults with no other info | 15:28 |
| cloudnull | :( | 15:28 |
| mordred | cloudnull: http://paste.openstack.org/show/562481/ and http://paste.openstack.org/show/562480/ | 15:28 |
| jeblair | we've seen apache segfault on mirrors in other clouds, but not all of them. (like, i think it happens more often in ord) | 15:28 |
| dougwig | jeblair, fungi, rm_work - nothing in that job should've affected inbound access, i think. | 15:29 |
| cloudnull | are jobs in the OSIC working now? -cc sdague ? | 15:29 |
| mordred | cloudnull: yes | 15:29 |
| zigo | pabelanger: The jessie-newton-backports one. | 15:29 |
| mordred | cloudnull: the failures stopped 2 hours ago | 15:29 |
| mordred | jeblair: it was not in error.2 ... only in 1 and current - but we did not see any changes in the server that would correlate with the introduction of the segfaults | 15:29 |
| *** yamahata has joined #openstack-infra | 15:30 | |
| mordred | although it's possible that the lack of them in error.2 is circumstantial | 15:30 |
| rm_work | jeblair / clarkb / fungi: Looks like the job just re-queued >_> | 15:30 |
| rm_work | did you guys do that? | 15:30 |
| *** dprince has quit IRC | 15:30 | |
| pabelanger | We have a nice wave going on in osic-cloud1 too: http://grafana.openstack.org/dashboard/db/nodepool-osic?from=1471962597278&to=1471966197278 | 15:30 |
| fungi | rm_work: if zuul thinks the node has fallen off the network, it blames the provider and restarts the job on a new node | 15:30 |
| rm_work | hmm lol k | 15:31 |
| pabelanger | need to see what is going on there, if job failures or just running short lived jobs | 15:31 |
| fungi | rm_work: if it continues to loop like this, then... probably the change itself | 15:31 |
| openstackgerrit | Merged openstack-infra/nodepool: Add an example logging.conf for development https://review.openstack.org/357329 | 15:31 |
| cloudnull | symetrical :) | 15:31 |
| cloudnull | ^ pabelanger | 15:31 |
| openstackgerrit | Merged openstack-infra/nodepool: Add a fake-secure.conf https://review.openstack.org/357330 | 15:31 |
| openstackgerrit | Merged openstack-infra/nodepool: Set default branch to feature/zuulv3 https://review.openstack.org/357326 | 15:31 |
| rm_work | another one just requeued... | 15:31 |
| *** yamamoto has joined #openstack-infra | 15:31 | |
| *** abregman is now known as abregman|mtg | 15:31 | |
| rm_work | i don't think it's the change. but yeah, we'll wait and see what happens | 15:31 |
| openstackgerrit | Merged openstack-infra/nodepool: Add zookeeper-servers to fake config https://review.openstack.org/357327 | 15:32 |
| pabelanger | rm_work: which review are you looking at? | 15:32 |
| *** dprince has joined #openstack-infra | 15:32 | |
| openstackgerrit | Merged openstack-infra/system-config: Add ssl_key_file_contents to compute nodes https://review.openstack.org/359208 | 15:32 |
| cloudnull | mordred: maybe something w/ the event mpm settings allowing memory consumption to get too high? | 15:32 |
| rm_work | https://review.openstack.org/286381 | 15:32 |
| *** yamamoto has quit IRC | 15:32 | |
| *** yamamoto has joined #openstack-infra | 15:33 | |
| rm_work | pabelanger: ^^ it passed recently and the only changes since then shouldn't be able to break it, but ... <_< | 15:33 |
| *** yamamoto has quit IRC | 15:33 | |
| rm_work | we'll just wait and see what happens, could just be something intermittent | 15:33 |
| mordred | cloudnull: we were thinking something related to oom ... but we didn't see any mentions of oomkiller running | 15:33 |
| dougwig | rm_work: let's see what happens. we did just enable some templates that mess with namespaces. | 15:33 |
| *** david-lyle has quit IRC | 15:33 | |
| fungi | rm_work: if you find another one that's frozen, i can try to grab the nova console log before it gets deleted | 15:33 |
| rm_work | fungi: k... going to look and figure out if there's any way it COULD be this change | 15:34 |
| fungi | wish i'd thought to do that on the last one while it was still showing active in nova | 15:34 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Set the ssl_key_file_contents to mandatory https://review.openstack.org/359294 | 15:34 |
| clarkb | this mornings ubuntu-precise iamge build took less than an hour | 15:35 |
| anteaya | yay! | 15:35 |
| rm_work | fungi: telnet://63.251.114.233:19885 | 15:35 |
| rm_work | fungi: that one looks to be nonresponsive | 15:35 |
| *** piet has quit IRC | 15:35 | |
| cloudnull | is log rotate doing a graceful restart? maybe logs are filling and its rotating more often than it should or calling multiple restarts? we just added the log rotate bits right? | 15:35 |
| *** senk has joined #openstack-infra | 15:37 | |
| fungi | rm_work: weird, that one's in internap not osic | 15:37 |
| *** esikachev has joined #openstack-infra | 15:37 | |
| mordred | cloudnull: it is doing a graceful - but we're not seeing failures everytime it does | 15:37 |
| *** sdague has joined #openstack-infra | 15:38 | |
| pabelanger | zigo: okay, deleted | 15:38 |
| cloudnull | mordred: hum, interesting... I'd be curious to see what the output is from the logs. | 15:39 |
| cloudnull | mordred: do you have the mirror instance(s) UUID(s) on hand? | 15:40 |
| rm_work | hmm | 15:40 |
| mordred | cloudnull: one sec | 15:40 |
| cloudnull | I can go look up the compute nodes and see if there's something else a-foot here. | 15:40 |
| rm_work | fungi: maybe i'm not able to connect for a different reason :P | 15:40 |
| mordred | cloudnull: 54bce385-4b3e-4a14-aa2a-f87f5ddd6bc0 | 15:40 |
| *** davidlenwell has joined #openstack-infra | 15:40 | |
| cloudnull | o/ SamYaple -- missed your earlier ping :) | 15:40 |
| mordred | cloudnull: "hostId": "2f26ddc2dafefc5d995c6daeda125b7159bd210399d1763c4b3a2a82", | 15:40 |
| mordred | cloudnull: in case that's useful | 15:41 |
| fungi | rm_work: well, i'm taking an inordinate amount of time to track down credentials for checking that one because the all-clouds.yaml on our puppetmaster is incomplete for some reason | 15:41 |
| cloudnull | mordred: tyvm | 15:41 |
| rm_work | fungi: yeah nm looks like THAT one is a firewall issue, because it's not a RAX node, we can only get to RAX nodes :P | 15:41 |
| *** vincentll has quit IRC | 15:41 | |
| rm_work | fungi: it's ok, we'll figure it out, hopefully the next runs of those other jobs will be fine | 15:41 |
| clarkb | fungi: missing the internap jenkins account? | 15:41 |
| mordred | fungi: oh - that seems unpleasing | 15:41 |
| rcarrillocruz | :S | 15:41 |
| cloudnull | mordred: if we turn-on LBaaS would more than 1 mirror server help things ? | 15:41 |
| fungi | clarkb: missing username and password for it, yes. probably incorrect hiera keys | 15:41 |
| *** esikachev has quit IRC | 15:42 | |
| mordred | cloudnull: I don't know enough yet - I'd like to grok the problem first if we can ... at the moment this is still total mystery | 15:42 |
| cloudnull | or if we did HAP w/ multiple mirror server backends ? | 15:42 |
| cloudnull | ok | 15:42 |
| clarkb | cloudnull: and hope they don't all die at the same time | 15:42 |
| fungi | anyway, i need to get back to this wiki security update for now | 15:43 |
| cloudnull | clarkb: ++ | 15:43 |
| fungi | rm_work: yeah, i'm able to get to the console on 63.251.114.233 anyway | 15:43 |
| rm_work | yeah | 15:43 |
| fungi | so sounds like egress filtering in your office | 15:43 |
| rm_work | i am able to from another location | 15:44 |
| rm_work | yep | 15:44 |
| rm_work | interestingly the RAX nodes are whitelisted | 15:44 |
| rm_work | or, not surprisingly | 15:44 |
| rm_work | i'll verify better next time, if we see another similar issue on our jobs | 15:45 |
| *** piet has joined #openstack-infra | 15:45 | |
| rm_work | but, i can say given which jobs passed already and which re-queued, it shouldn't be anything to do with the change | 15:45 |
| fungi | rm_work: we had to pick a fairly oddball port number in an attempt to avoid colliding with any services that might try to listen on the job nodes (including teh various service ports devstack made up) | 15:45 |
| *** piet has quit IRC | 15:45 | |
| *** zhurong has quit IRC | 15:46 | |
| *** tesseract- has quit IRC | 15:46 | |
| *** piet has joined #openstack-infra | 15:47 | |
| *** kaisers_ has joined #openstack-infra | 15:48 | |
| *** esikachev has joined #openstack-infra | 15:49 | |
| *** yaume has quit IRC | 15:50 | |
| *** ddieterly is now known as ddieterly[away] | 15:50 | |
| openstackgerrit | Changcheng Intel proposed openstack-infra/jenkins-job-builder: update base_email_ext to adapt Email-ext plugin https://review.openstack.org/355139 | 15:50 |
| mordred | fungi: have I mentoined how silly I think egress filtering is? | 15:50 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Use aliasByNode for Node Launches panel https://review.openstack.org/359299 | 15:50 |
| pleia2 | heh | 15:50 |
| fungi | mordred: i have a choir you're free to preach to | 15:51 |
| jeblair | pabelanger, mordred, cloudnull: i'm still catching up -- but do i understand correctly that pabelanger is adding the ipv6 address for the osic mirror to dns now? | 15:51 |
| openstackgerrit | Cyril Roelandt proposed openstack-dev/hacking: Add a check to make sure the right assert* method is used https://review.openstack.org/354185 | 15:52 |
| jeblair | pabelanger, mordred, cloudnull: and because it wasn't in dns, that's why all the logs indicate the requests come from a single ipv4 addr -- the nat server doing v6->v4 translation? | 15:52 |
| pabelanger | jeblair: Yes, but not at the moment. I think we need to schedule it to avoid some downtime | 15:52 |
| fungi | why would that cause downtime? | 15:52 |
| openstackgerrit | Merged openstack-infra/system-config: Replace ssl cert for infracloud vanilla controller https://review.openstack.org/359254 | 15:52 |
| mordred | there is no ipv6 address | 15:52 |
| *** mmedvede has quit IRC | 15:53 | |
| fungi | oh, the mirror has no ipv6 address configured? | 15:53 |
| jeblair | pabelanger, mordred, cloudnull: and so if we're looking for a component problem, not only should we consider neutron, but also the v4->v6 nat system? | 15:53 |
| *** kaisers_ has quit IRC | 15:53 | |
| pabelanger | fungi: right, we need to first do that and update the network | 15:53 |
| mordred | jeblair: yah | 15:53 |
| irtermite | pabelanger: mordred: fungi: cloudnull: is this mirror an instance hosted on osic? | 15:53 |
| mordred | and I believe cloudnull is looking at host logs | 15:54 |
| mordred | irtermite: yes | 15:54 |
| cloudnull | I am | 15:54 |
| fungi | indeed, eth0 has only a linklocal v6 addy | 15:54 |
| irtermite | and it has no ipv6 because that was how it was originally deployed mordred? | 15:54 |
| *** sdague has quit IRC | 15:54 | |
| mordred | we need to either add a new nic to the existing system, or just boot a new mirror | 15:54 |
| mordred | irtermite: yes. that's right | 15:54 |
| *** ddieterly[away] is now known as ddieterly | 15:54 | |
| pabelanger | I mean, we could launch a replacement server then just switch the DNS | 15:54 |
| mordred | when we booted this mirror, the GATEWAY_NET_v6 network did not exist | 15:54 |
| *** timello has quit IRC | 15:54 | |
| jeblair | [i went to look at apache logs to see what they looked like around the time of the errors, and ... well, it's difficult to discern patterns with the nat address :( ] | 15:55 |
| irtermite | understood, that's what I figured | 15:55 |
| pabelanger | likely faster then waiting for a window to add ipv6 | 15:55 |
| mordred | ++ | 15:55 |
| cloudnull | ++ | 15:55 |
| irtermite | would it not help to add another interface to it and put it on gateway_net_v6? | 15:55 |
| jeblair | pabelanger: do we have a free ipv4? | 15:55 |
| fungi | well, actually it has a linklocal address and a ula | 15:55 |
| fungi | but regardless, no global v6 address | 15:55 |
| pabelanger | jeblair: I haven't not checked. I can do that now | 15:55 |
| *** ggnel_t has quit IRC | 15:55 | |
| clarkb | jeblair: we should have a quota of 5 fips in that project so as long as the cloud has one then we should be fine | 15:55 |
| *** _nadya_ has quit IRC | 15:55 | |
| clarkb | we can also move the fip to the new server | 15:56 |
| mordred | irtermite: we could theorecticaly do that - but it's likely more work than just spinning up a new one - this is a fairly stateless server | 15:56 |
| clarkb | (which may be disruptive) | 15:56 |
| *** mmedvede_ is now known as mmedvede | 15:56 | |
| irtermite | yea mordred, i would agree. can't hurt to have another mirror | 15:56 |
| jeblair | clarkb: yeah, i imagine it might be; i like the replace with new server+ip option if we can | 15:56 |
| *** pcaruana has quit IRC | 15:56 | |
| mordred | the floating ip we currently have is on GATEWAY_NET ... does the neutron there have a router from GATEWAY_NET to GATEWAY_NET_V6 ? | 15:56 |
| *** matrohon has quit IRC | 15:56 | |
| jeblair | (even if it's just for 30 seconds or so, it would tank a bunch of jobs) | 15:57 |
| irtermite | mordred: mirror_v4, mirror_v6 + cloudnull ;) | 15:57 |
| *** Sukhdev has joined #openstack-infra | 15:57 | |
| irtermite | mordred: no I do not believe we have a router between | 15:57 |
| pabelanger | jeblair: our current quota for FIPs in 1 of 3 in openstackci | 15:57 |
| mordred | k. then the new server may have to be a little extra special | 15:57 |
| mordred | or we may need to figure out a router | 15:58 |
| clarkb | mordred: why would we have to route between them? | 15:58 |
| irtermite | ??? | 15:58 |
| clarkb | mordred: can't we just have one interface on GATEWAY_NET and one on GATEWAY_NET_V6? | 15:58 |
| mordred | because we'll need an IPv4 FIP attached ot the private ipv4 address we get from GATEWAY_NET_V6 | 15:58 |
| mordred | clarkb: we can also do that | 15:58 |
| irtermite | clarkb: that's what i suggested | 15:58 |
| irtermite | orrrrr, just spin up a new mirror on both stacks and THEN destroy the old one | 15:59 |
| pabelanger | yes, we've done that a few times now | 15:59 |
| clarkb | ya I think spin up a new one with two interfaces, check both v4 and v6 work happily, update dns, delete the old one | 15:59 |
| *** matthewbodkin has quit IRC | 15:59 | |
| irtermite | well, you know my vote clarkb and mordred | 16:00 |
| irtermite | but, I'm just your account manager ;) | 16:00 |
| irtermite | what do I know? ;) | 16:00 |
| *** jaosorior_away is now known as jaosorior | 16:00 | |
| jeblair | irtermite: i think you and clarkb and pabelanger are all agreeing, right? or am i missing something? | 16:00 |
| mordred | yah. I think we're all on the same page | 16:01 |
| irtermite | jeblair: yup... it would appear so | 16:01 |
| cloudnull | mordred: so there are several instances on the node where the mirror is, however the node was not our of mem or slamming the CPU. Dual socket 48cores, 256MEM, constant tx/rx but nothing that the nic cant handle. so I can't imagine it was a noisey neighbor issue for that host or the host causing KVM OOM issues. | 16:01 |
| jeblair | ok. cool. carry on. | 16:01 |
| *** timello has joined #openstack-infra | 16:01 | |
| fungi | the remaining question is whether we want to scale our osic max-servers back down temporarily while we do that | 16:01 |
| cloudnull | 's/not our/not out/' | 16:01 |
| mordred | fungi: it's not currently causing problems | 16:01 |
| *** david-lyle has joined #openstack-infra | 16:01 | |
| irtermite | NO SCALE DOWN FOR YOU! | 16:01 |
| mordred | the problems existed for 30 minutes about 3 hours ago | 16:01 |
| mordred | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22after%20connection%20broken%20by%20'ConnectTimeoutError%5C%22%20AND%20tags%3A%5C%22console%5C%22 | 16:02 |
| cloudnull | this very well could've been an issue within the RAX DC --cc irtermite | 16:02 |
| irtermite | ACK | 16:02 |
| *** cody-somerville has joined #openstack-infra | 16:02 | |
| irtermite | haven't heard anything though | 16:03 |
| *** gyee has joined #openstack-infra | 16:03 | |
| irtermite | nothing globally impacting anyway, cloudnull https://status.rackspace.com/ | 16:03 |
| cloudnull | maybe we can reachout to dcops and see if there was an incident or issue ~3 hours ago? | 16:03 |
| zigo | pabelanger: http://mirror.dfw.rax.openstack.org/debian-openstack/pool/main/p/python-cryptography-vectors/ <--- There's still a binary package remaining there, probably because there was some stuff in the POST. | 16:04 |
| irtermite | boom | 16:04 |
| irtermite | https://status.rackspace.com/index/viewincidents?start=1471924800 | 16:04 |
| irtermite | cloudnull: ^^ | 16:04 |
| zigo | pabelanger: I waited until they are all done, it should be fine if you delete the package now. | 16:04 |
| zigo | pabelanger: How do you delete the package btw, is there a kind of API for reprepro? | 16:04 |
| cloudnull | mordred: ^^ | 16:04 |
| zigo | pabelanger: I'm asking so we can design some kind of API together ... | 16:04 |
| cloudnull | maybe we were in the subset of customers that were impacted? | 16:05 |
| *** ganesan has joined #openstack-infra | 16:05 | |
| pabelanger | zigo: yes, there is some commands in reprepro to remove a package | 16:05 |
| irtermite | at your desk cloudnull? have time to sit with Johnny and me for SSL cert, or want to resolve this issue first? | 16:05 |
| cloudnull | :) i am | 16:05 |
| irtermite | BRT | 16:05 |
| fungi | cloudnull: irtermite: wow. i had skimmed the status page and never realized that "cloud monitoring" was a valid category for network outages | 16:05 |
| pabelanger | zigo: Ya, not right now, but we can also loop mordred in also. We should be able the process via a job | 16:05 |
| zigo | pabelanger: Let's say I would add a file "reprepro-delete" to the uploads folder, and your script would process it? | 16:06 |
| fungi | cloudnull: irtermite: i didn't even bother clicking into that incident because i figured it was just a problem impacting the monitoring service | 16:06 |
| cloudnull | sorry ? :\ | 16:06 |
| clarkb | doing more rough maths we are averaging just over an hour per image build now. Which is about half as long as it took before. So that is an improvement. I am also working on getting these xenial images uploaded so that we can have ntpdate for d-g | 16:06 |
| pabelanger | zigo: removed now | 16:07 |
| zigo | Thanks so much. | 16:07 |
| pabelanger | basically: reprepro --confdir /etc/reprepro/debian-openstack remove jessie-newton-backports python3-cryptography-vectors | 16:07 |
| fungi | cloudnull: just trying to correct my previous assumptions... are "cloud monitoring" outages a catch-all for outages impacting other services too? | 16:07 |
| pabelanger | reprepro --confdir /etc/reprepro/debian-openstack --nokeepunreferencedfiles deleteunreferenced | 16:07 |
| jeblair | fungi, clarkb, irtermite: if i'm tz-mathing correctly, that incident was about 10 hours ago, but our errors were about 4 hours ago...? | 16:07 |
| pabelanger | vos release mirror.deb-openstack | 16:07 |
| *** Sukhdev has quit IRC | 16:08 | |
| *** piet has quit IRC | 16:08 | |
| pabelanger | zigo: Ya, we'd need to pass a list of files to delete to reprepro-delete job | 16:08 |
| clarkb | pabelanger: zigo what would trigger the need for a delete? | 16:08 |
| pabelanger | zigo: then add some logic into zuul to only run said job when the file changes | 16:08 |
| clarkb | pabelanger: zigo udnerstanding that might help determine how we want to do it | 16:08 |
| clarkb | jeblair: yes 10-11 hours ago or so for the rax incident | 16:09 |
| *** pcaruana has joined #openstack-infra | 16:09 | |
| pabelanger | clarkb: once scenerio might be incorrectly publishing a package to the wrong repo | 16:09 |
| clarkb | pabelanger: that should almost never happen if we automate things right? | 16:10 |
| pabelanger | we have 2 today, jessie-newton and jessie-newton-backports | 16:10 |
| *** piet has joined #openstack-infra | 16:10 | |
| *** tphummel has joined #openstack-infra | 16:10 | |
| clarkb | pabelanger: eg if we get the builds working properly we shouldn't have to worry about that much | 16:10 |
| pabelanger | clarkb: right, shouldn't happen but we also don't impose validation | 16:10 |
| pabelanger | clarkb: it could only happen if somebody patched the wrong branch for example | 16:10 |
| clarkb | pabelanger: could tehy just revert to fix? | 16:10 |
| pabelanger | that wouldn't remove the package from reprepro today | 16:11 |
| clarkb | (assuming revert makes new package and reuploads I Think taht should work) | 16:11 |
| pabelanger | since we already build and published it | 16:11 |
| pabelanger | Oh | 16:11 |
| pabelanger | Hmm | 16:11 |
| pabelanger | that would work | 16:11 |
| pabelanger | but zigo would need to increment his version number to be greater then the broken package | 16:11 |
| pabelanger | which is possible | 16:11 |
| clarkb | so revert + version bump | 16:12 |
| pabelanger | ya, that would fix this example | 16:12 |
| pabelanger | the issue is, you need to delete a released package for some reason | 16:12 |
| pabelanger | and remove it from reprepro with now replacement | 16:12 |
| pabelanger | no* | 16:13 |
| openstackgerrit | Nate Johnston proposed openstack-infra/project-config: Make neutron-fwaas functional job not experimental https://review.openstack.org/359320 | 16:13 |
| irtermite | jeblair: hrm, good point (timestamp) | 16:13 |
| *** hashar has quit IRC | 16:14 | |
| mordred | Shrews: we have per-resource expiration in clouds.yaml for shade-nodepool test: http://logs.openstack.org/97/315697/7/check/gate-dsvm-nodepool-src-shade/0df6717/logs/etc/openstack/clouds.yaml.txt.gz | 16:15 |
| mordred | Shrews: any chance you know what puts those settings there? | 16:15 |
| clarkb | pabelanger: ya I think that situation is the one we actually need to solve for | 16:15 |
| *** DrifterZA has quit IRC | 16:15 | |
| mordred | Shrews: I cannot find _anywhere_ that writes out expiration times | 16:15 |
| mordred | Shrews: (I want to add a setting is why I'm asking) | 16:16 |
| *** jpich has quit IRC | 16:16 | |
| Shrews | mordred: i don't understand the question. those are manually added to clouds.yaml | 16:18 |
| mordred | Shrews: yah - where? | 16:18 |
| pabelanger | clarkb: going to start work on the replacement mirror in osic-cloud1 | 16:18 |
| Shrews | mordred: umm, by the user that owns it? | 16:18 |
| mordred | Shrews: like, they end up in clouds.yaml for the test job | 16:18 |
| *** jaosorior has quit IRC | 16:18 | |
| clarkb | pabelanger: ok fo t forget to pass both nics to the boot command which launch node may not do? | 16:18 |
| Shrews | mordred: oh. isn't it a fixture? | 16:19 |
| clarkb | actually the new ansible stuff probably just hasa list you can set? | 16:19 |
| mordred | Shrews: _something_ is adding it to /etc/openstack/clouds.yaml | 16:19 |
| Shrews | mordred: shade/tests/unit/fixtures/clouds/clouds_cache.yaml | 16:20 |
| Shrews | maybe? | 16:20 |
| pabelanger | clarkb: ya, so keep the current network (openstackci-subnet1) and add GATEWAY_NET_v6 right? | 16:20 |
| Shrews | mordred: oh, in /etc... maybe in devstack itself | 16:21 |
| clarkb | pabelanger: or switch openstackci-subnet1 to GATEWAY_NET and dont fip | 16:21 |
| pabelanger | clarkb: ya, lets do that | 16:21 |
| mordred | Shrews: nope. the cache settings in the clouds.yaml file are different ... yah - I looked in devstack but couldn't find it - I'll look again though | 16:21 |
| mordred | Shrews: this is extra weird :) | 16:21 |
| pabelanger | clarkb: any preference of NIC order? | 16:21 |
| clarkb | pabelanger: I dont think it matters. If I hadto choose v6 first since thats the rest of the cloud for ys | 16:22 |
| jeblair | fungi: updated https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting to add an item on the doc-in-afs spec -- basically an RFC and RFVolunteers before we vote on it next week. | 16:22 |
| jeblair | AJaeger: are you back? | 16:22 |
| Shrews | mordred: yeah, i dunno man | 16:23 |
| AJaeger | jeblair: yes, I am | 16:23 |
| fungi | jeblair: thanks! | 16:23 |
| jeblair | AJaeger: welcome! i hope you were able to stay away sufficiently when you were away. ;) | 16:23 |
| jeblair | AJaeger: will you have time to join the infra meeting today? | 16:24 |
| mordred | Shrews: nodepool's devstack plugin | 16:25 |
| mordred | Shrews: wow. that was fun | 16:25 |
| Shrews | mordred: neat | 16:25 |
| jeblair | AJaeger: i want to discuss https://review.openstack.org/276482 and make sure you have an opportunity to participate | 16:25 |
| AJaeger | jeblair: I'll join the meeting - and I was sufficiently away ;) | 16:26 |
| AJaeger | thanks, jeblair | 16:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Add floating-ip batching settings to clouds.yaml https://review.openstack.org/359327 | 16:27 |
| zaro | morning | 16:27 |
| mordred | Shrews: ^^ that's why I was looking for it | 16:27 |
| mordred | :) | 16:27 |
| *** shashank_hegde has joined #openstack-infra | 16:31 | |
| rcarrillocruz | in other news: | 16:32 |
| rcarrillocruz | controller00.vanilla.ic.openstack.org : ok=15 changed=5 unreachable=0 failed=0 | 16:32 |
| rcarrillocruz | wootz! | 16:32 |
| rcarrillocruz | puppet on controller infracloud converges | 16:32 |
| *** florianf has quit IRC | 16:32 | |
| mordred | rcarrillocruz: woot! | 16:33 |
| *** yamamoto has joined #openstack-infra | 16:33 | |
| *** eharney has quit IRC | 16:35 | |
| *** sputnik13 has joined #openstack-infra | 16:35 | |
| rcarrillocruz | afk for a bit | 16:36 |
| *** zul has quit IRC | 16:36 | |
| rcarrillocruz | see ya in the meeting | 16:36 |
| *** jraju has joined #openstack-infra | 16:38 | |
| *** fernnest has joined #openstack-infra | 16:38 | |
| *** jraju has quit IRC | 16:39 | |
| AJaeger | team, do we have zuul-cloner installed as /usr/zuul-env/bin/zuul-cloner on the proposal node? I see very strange failures in post jobs that run on proposal node but work fine elsewhere. Could it be that the version on the proposal node is very old? | 16:41 |
| *** yamamoto has quit IRC | 16:41 | |
| *** AnarchyAo has joined #openstack-infra | 16:42 | |
| *** AnarchyAo has quit IRC | 16:42 | |
| *** cody-somerville has quit IRC | 16:42 | |
| *** AnarchyAo has joined #openstack-infra | 16:42 | |
| *** _nadya_ has joined #openstack-infra | 16:43 | |
| *** _nadya_ has quit IRC | 16:43 | |
| fungi | AJaeger: http://paste.openstack.org/show/562508/ | 16:44 |
| *** bethwhite_ has quit IRC | 16:44 | |
| fungi | AJaeger: so, yes, looks old-ish | 16:45 |
| AJaeger | thanks, fungi. How can we update this to current zuul-cloner? The one that is able to be used in post and periodic jobs... | 16:45 |
| fungi | AJaeger: latest from git should be zuul==2.5.1.dev4 # git sha 569b7a3 | 16:46 |
| AJaeger | that's new enough ;9 | 16:46 |
| fungi | AJaeger: probably we need to double-check the puppet we have for creating that env on persistent job nodes and make sure it's correctly set to upgrade | 16:47 |
| *** asselin has joined #openstack-infra | 16:47 | |
| *** zul has joined #openstack-infra | 16:48 | |
| *** yamahata has quit IRC | 16:48 | |
| *** asettle has quit IRC | 16:48 | |
| clarkb | ok osic, internap, and bluebox should all have up to date xenial images with ntpdate installed | 16:48 |
| clarkb | working on ovh and rax | 16:49 |
| *** asettle has joined #openstack-infra | 16:49 | |
| *** asettle has quit IRC | 16:49 | |
| *** asettle has joined #openstack-infra | 16:49 | |
| clarkb | also image builds continue to be quicker so we may weant to consider semi periodic cache cleanups (but also dig into why having a cache makes builds slower and not faster) | 16:49 |
| *** awayne has quit IRC | 16:49 | |
| clarkb | greghaynes: FYI ^ dib caching behavior makes things slower | 16:49 |
| AJaeger | fungi, https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/slave_common.pp#n155 - ensures only installation. | 16:49 |
| greghaynes | clarkb: wah | 16:50 |
| clarkb | greghaynes: I moved the old cache dir aside and builds are now half as long (from 2 hours to about 50 minutes) | 16:50 |
| *** asettle has quit IRC | 16:50 | |
| pabelanger | mordred: any ideas why openstack network list in osic-cloud1 returns Connection failure that may be retried. ? | 16:51 |
| clarkb | pabelanger: see if neutron client does the same thing? | 16:51 |
| AJaeger | fungi, I don't see how we can ensure that those are uptodate - my puppet knowledge is nearly zero. | 16:51 |
| pabelanger | clarkb: sure, testing | 16:52 |
| fungi | AJaeger: we includes more than just you, luckily | 16:52 |
| *** e0ne has quit IRC | 16:52 | |
| pabelanger | clarkb: that works as expected | 16:52 |
| clarkb | neat. Does openstackclient have a trace flag? if so you can compare the API calls between the two clients and see if they differ | 16:53 |
| ganesan | I am getting the Auth Expection when nodepool try to ssh the nodes(hitting this prob very long time and couldnot fix). I verified the ssh keys manually and it works | 16:53 |
| AJaeger | fungi, this problem hits us now with projects using constraints everywhere with failing translation jobs (running on the proposal node) | 16:53 |
| pabelanger | clarkb: sure, let me get some food before I shave this yak | 16:53 |
| AJaeger | So, any help here is welcome | 16:53 |
| ganesan | Is it possible to check the ssh keys injected into an image | 16:53 |
| fungi | #status log The https://wiki.openstack.org/ site (temporarily hosted from wiki-upgrade-test.o.o) has been updated from Mediawiki 1.27.0 to 1.27.1 per https://lists.wikimedia.org/pipermail/mediawiki-announce/2016-August/000195.html | 16:53 |
| openstackstatus | fungi: finished logging | 16:53 |
| ganesan | like I could mount the image and check the given ssh keys are injected into the image created by nodepool builder | 16:54 |
| fungi | ganesan: you should be able to mount the image on a loop block device, yes | 16:55 |
| fungi | mount -o loop /path/to/image/file | 16:55 |
| *** jed56 has quit IRC | 16:55 | |
| fungi | i think it's that easy anyway, though been a while since i've needed to | 16:55 |
| clarkb | for a qcow2 you have to nbd it I think but for raw that should work. THere are all sorts of directions on doing it on the internets if you google for mounting $image type | 16:56 |
| ganesan | fungi: thanks | 16:56 |
| fungi | er, that's `mount -o loop /path/to/image/file my_mountpoint` i guess | 16:56 |
| fungi | where my_mountpoint is some local directory you're going to use as the mountpoint | 16:57 |
| fungi | and yeah, for raw. qcow2 needs some extra decoding | 16:57 |
| *** mtanino has joined #openstack-infra | 16:57 | |
| greghaynes | clarkb: huh... are those logs being put on nodepool.o.o? | 16:58 |
| greghaynes | clarkb: something I can look at a before/after | 16:58 |
| clarkb | greghaynes: yup, http://nodepool.openstack.org, compare today's image logs to those from a few days ago | 16:58 |
| clarkb | rax image uploads just succeeded | 16:59 |
| clarkb | so just waiting on ovh now before we can merge the ntpdate in d-g change | 16:59 |
| *** derekh has quit IRC | 16:59 | |
| *** dtantsur is now known as dtantsur|afk | 17:01 | |
| clarkb | greghaynes: we can actually poke at those logs together later today | 17:02 |
| mordred | pabelanger: oh, lovely | 17:02 |
| clarkb | I will try and prefetch them onto laptop so that we don't have to derp with tether for them | 17:02 |
| greghaynes | clarkb: ah, yea | 17:02 |
| fungi | AJaeger: so... i think we should probably just set https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/zuul-env-reqs.txt to whatever version we want to install (2.5.0?) and let it get the release from pypi. however, before we do, i notice that there is no subscribe/notify between File['/etc/zuul-env-reqs.txt'] and Python::Virtualenv['/usr/zuul-env'], | 17:03 |
| fungi | which will be needed to get it to update | 17:03 |
| *** jerryz has joined #openstack-infra | 17:04 | |
| *** javeriak has joined #openstack-infra | 17:04 | |
| *** Apoorva has joined #openstack-infra | 17:04 | |
| clarkb | mordred: is current plan to get shade/occ things merged up today then restart nodepool tomorrow? | 17:04 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install https://review.openstack.org/358919 | 17:04 |
| mordred | clarkb: yah. I mean, gate willing | 17:04 |
| *** Apoorva has quit IRC | 17:04 | |
| clarkb | mordred: ok, looks like my nodepool change merged so we don't have to wait for that one | 17:05 |
| AJaeger | 2.5.0 as version should be ok | 17:05 |
| *** Apoorva has joined #openstack-infra | 17:05 | |
| *** Guest25180 is now known as med_ | 17:06 | |
| *** med_ has joined #openstack-infra | 17:06 | |
| *** med_ is now known as medberry | 17:06 | |
| *** medberry is now known as med_ | 17:06 | |
| fungi | wo here has familiarity with the python::virtualenv puppet class? latest release looks like it only has a way to create a virtualenv, but no mechanism to upgrade/replace one? https://github.com/stankevich/puppet-python/blob/1.14.2/manifests/virtualenv.pp | 17:06 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job https://review.openstack.org/359146 | 17:07 |
| fungi | is there some special puppet dance you're expected to follow to remove and replace a resource? | 17:07 |
| *** AnarchyAo has joined #openstack-infra | 17:07 | |
| *** AnarchyAo has quit IRC | 17:08 | |
| *** AnarchyAo has joined #openstack-infra | 17:08 | |
| *** AnarchyAo has quit IRC | 17:08 | |
| mordred | pabelanger: neutron support in python-openstackclient is very new | 17:08 |
| *** lucasagomes is now known as lucas-afk | 17:08 | |
| *** AnarchyAo has joined #openstack-infra | 17:08 | |
| mordred | pabelanger: and shaky in places | 17:08 |
| *** AnarchyAo has quit IRC | 17:08 | |
| *** AnarchyAo has joined #openstack-infra | 17:08 | |
| *** AnarchyAo has quit IRC | 17:08 | |
| mordred | pabelanger: using neutron client at this point is still probably more betterer | 17:08 |
| fungi | or maybe it magically knows to upgrade packages in a virtualenv when the requirements file changes | 17:08 |
| *** AnarchyAo has joined #openstack-infra | 17:08 | |
| *** asselin has quit IRC | 17:09 | |
| ganesan | fungi: there is a script to mount qcow2 file - http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/README.rst#n61 | 17:09 |
| *** sambetts is now known as sambetts|afk | 17:09 | |
| *** ddieterly is now known as ddieterly[away] | 17:10 | |
| *** abregman|mtg is now known as abregman | 17:10 | |
| *** mtanin___ has joined #openstack-infra | 17:10 | |
| ganesan | and I see the id_rsa.pub keys are injected into the image under /home/jenkins/.ssh/authorized_keys | 17:10 |
| *** hashar has joined #openstack-infra | 17:10 | |
| ganesan | but still I am getting Auth exception | 17:11 |
| clarkb | fungi: I think the dance may be to run pip within the virtualenv | 17:11 |
| *** AnarchyAo has joined #openstack-infra | 17:11 | |
| *** hashar is now known as hasharAway | 17:11 | |
| fungi | ahh, yep, looks like if a requirements parameter is specified, there's an exec on it with a refreshonly which is installing packages, but i don't see a subscribe or notify to it from the requirements file | 17:11 |
| openstackgerrit | Marc Aubry proposed openstack-infra/project-config: Add python34-jobs on Almanach https://review.openstack.org/359345 | 17:12 |
| fungi | i wonder if we could notify Exec[" exec { " | 17:12 |
| fungi | python_requirements_initial_install_${requirements}_${venv_dir}"] | 17:12 |
| *** mtanino has quit IRC | 17:12 | |
| fungi | er, notify Exec["python_requirements_initial_install_${requirements}_${venv_dir}"] | 17:12 |
| *** abregman has quit IRC | 17:12 | |
| fungi | but yeah, i suppose we could also just have our own exec subscribed to the file and make it require the venv resource | 17:13 |
| fungi | i'll give that a shot | 17:13 |
| clarkb | ganesan: usually the best way to debug that is to manually boot an instance off that image then attempt sshing to it as the nodepool user | 17:13 |
| *** ilyashakhat has joined #openstack-infra | 17:14 | |
| *** asselin has joined #openstack-infra | 17:15 | |
| *** brad_behle has joined #openstack-infra | 17:17 | |
| *** mikelk has quit IRC | 17:19 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: fix announce.sh for projects with setup_requires https://review.openstack.org/359351 | 17:19 |
| *** hieulq_ has quit IRC | 17:20 | |
| clarkb | greghaynes: waiting for todays ubuntu trusty build to finish then I will have logs from the 21st and today to compare | 17:20 |
| clarkb | they are all at http://nodepool.openstack.org too | 17:21 |
| *** ilyashakhat has quit IRC | 17:21 | |
| brad_behle | Hello, I'm trying to find what exact ubuntu-trusty image is used for the gate jobs? openstack-infra/devstack-gate/blob/master/README.rst has a link to system-config modules/openstack_project/templates/nodepool/nodepool.yaml.erb, but that file doesn't exist. | 17:22 |
| *** piet has quit IRC | 17:22 | |
| *** piet has joined #openstack-infra | 17:22 | |
| *** piet has quit IRC | 17:23 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Update zuul-env on job nodes https://review.openstack.org/359352 | 17:25 |
| fungi | AJaeger: ^ | 17:25 |
| *** _nadya_ has joined #openstack-infra | 17:25 | |
| clarkb | brad_behle: the file moved to openstack-infra/project-config/nodepool/nodepool.yaml | 17:25 |
| *** oanson has joined #openstack-infra | 17:25 | |
| AJaeger | thanks a lot, fungi! | 17:25 |
| clarkb | brad_behle: the dib images are all defined at the end of that file. That repo also includes some of the dib elements we use to build the images | 17:26 |
| AJaeger | brad_behle: could you update the README.rst, please? | 17:26 |
| *** zul has quit IRC | 17:26 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/devstack-gate: Update link to nodepool.yaml in README.rst https://review.openstack.org/359355 | 17:28 |
| fungi | AJaeger: brad_behle: ^ there | 17:28 |
| fungi | i was already updating it | 17:28 |
| *** yamahata has joined #openstack-infra | 17:28 | |
| AJaeger | thanks, fungi | 17:28 |
| *** sarob has joined #openstack-infra | 17:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks https://review.openstack.org/357517 | 17:29 |
| *** senk has quit IRC | 17:30 | |
| *** Swami has joined #openstack-infra | 17:30 | |
| *** senk has joined #openstack-infra | 17:30 | |
| *** samueldmq has joined #openstack-infra | 17:31 | |
| *** nwkarste_ has quit IRC | 17:31 | |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Add optional H204 to check that assert(Not)In is used https://review.openstack.org/359358 | 17:31 |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Add optional H205 to check that assertTrue/False is used https://review.openstack.org/359359 | 17:31 |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Add optional H206 to check that assertIs(Not)Instance is used https://review.openstack.org/359360 | 17:31 |
| *** nwkarsten has joined #openstack-infra | 17:32 | |
| *** zz_ja is now known as zz_zz_ja | 17:32 | |
| brad_behle | clarkb, AJaeger, Thanks! | 17:33 |
| *** hockeynut has quit IRC | 17:33 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:34 | |
| *** tqtran has joined #openstack-infra | 17:35 | |
| *** oanson has quit IRC | 17:36 | |
| *** nwkarsten has quit IRC | 17:36 | |
| pabelanger | mordred: Ya, it appears that way. http://paste.openstack.org/show/562521/ | 17:36 |
| *** kaisers_ has joined #openstack-infra | 17:37 | |
| clarkb | pabelanger: opuch | 17:37 |
| pabelanger | ya, I cannot use launch-node right now for osic-cloud1 | 17:38 |
| pabelanger | tying to see what changed | 17:38 |
| *** e0ne has joined #openstack-infra | 17:39 | |
| *** ilyashakhat has joined #openstack-infra | 17:39 | |
| *** tqtran has quit IRC | 17:39 | |
| *** shashank_hegde has quit IRC | 17:40 | |
| brad_behle | clarkb: I don't have accounts with any of those cloud providers, I was hoping I could find a link to download the exact version of ubuntu-trusty and deploy a few VMs of it myself. the networking-ovn project needs one of its gate jobs to have specific kernel features that I don't think are in the ubuntu images that are out there | 17:40 |
| *** ilyashakhat has quit IRC | 17:40 | |
| *** eharney has joined #openstack-infra | 17:40 | |
| *** ilyashakhat has joined #openstack-infra | 17:40 | |
| brad_behle | I wanted to set up a few vms and see if those kernel patches are really needed, so we can figure out how to get them into the gate job we are going to create. | 17:40 |
| clarkb | brad_behle: we don't currently publish them (we should but they are huge and unwieldy), but you can build them yourself using dib. The tools/build-image.sh script in project-config should make it easy | 17:40 |
| clarkb | brad_behle: we use the normal ubuntu cloud vm kernels | 17:41 |
| *** cody-somerville has joined #openstack-infra | 17:41 | |
| clarkb | so we aren't doing anything specail to remove modules or add them | 17:41 |
| *** _nadya_ has quit IRC | 17:41 | |
| *** nwkarsten has joined #openstack-infra | 17:41 | |
| clarkb | brad_behle: it shouldn't be hard to determine if the ubuntu trusty/xenial kernels have what you need | 17:42 |
| *** kaisers_ has quit IRC | 17:42 | |
| pabelanger | clarkb: mordred: failure log: http://paste.openstack.org/show/562522/ | 17:42 |
| *** csomerville has joined #openstack-infra | 17:42 | |
| *** nwkarsten has quit IRC | 17:42 | |
| *** tosky has quit IRC | 17:42 | |
| *** shashank_hegde has joined #openstack-infra | 17:42 | |
| *** nwkarsten has joined #openstack-infra | 17:43 | |
| brad_behle | clarkb: Okay, I'll take a look at build-image.sh and dib and try to build them myself. Thanks. | 17:45 |
| *** jcoufal_ has quit IRC | 17:45 | |
| clarkb | brad_behle: do you know what version of the kernel/modules you need? | 17:45 |
| *** cody-somerville has quit IRC | 17:45 | |
| clarkb | ubuntu xenial is a 4.4 kernel iirc | 17:45 |
| clarkb | ianw is also working on fedora 24 images which I don't think are functional yet but should have a 4.5 kernel looks like | 17:46 |
| clarkb | oh maybe 4.6 | 17:46 |
| clarkb | so newer | 17:46 |
| *** senk has quit IRC | 17:46 | |
| brad_behle | clarkb: I just started on this an hour ago, so I don't know exactly, but I think they are patches that aren't in a released kernel yet, right now the developers are applying kernel patches to the test systems. I did see a reference to 4.6 somewhere. | 17:46 |
| *** ganesan has quit IRC | 17:47 | |
| brad_behle | clarkb: on the test vagrant environment, uname -a shows: Linux compute2.ursula 4.6.0+ #1 SMP Wed Jun 8 15:23:19 CDT 2016 x86_64 ... | 17:47 |
| brad_behle | ursula is the name of the ansible project used to deploy. | 17:47 |
| clarkb | pabelanger: that almost looks like ti can't http to that url? | 17:48 |
| *** nwkarsten has quit IRC | 17:48 | |
| pabelanger | clarkb: ya, was just about to ask cloudnull to take a look | 17:48 |
| clarkb | brad_behle: you probably need to srot out what features/versions you need first? | 17:48 |
| pabelanger | cloudnull: mind helping with an issue we are seeing in osic-cloud1? http://paste.openstack.org/show/562522/ | 17:48 |
| brad_behle | My first step was just to find out what the gate jobs are using, install it and run the existing tests with the new code, watch some tests fail, then figure out what kernel patches/features are needed | 17:48 |
| *** vhosakot has joined #openstack-infra | 17:48 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Remove stale octavia job that never gets run, and is broken https://review.openstack.org/359367 | 17:48 |
| clarkb | pabelanger: that port is listening and does accept connections for me | 17:49 |
| clarkb | brad_behle: that seems backward to me but ok... | 17:49 |
| brad_behle | clarkb: I'll actually be doing both in parallel, trying to set up an environment and also tracking down the people building the test kernels we have that are working and see what patches they are pulling in | 17:50 |
| *** nwkarsten has joined #openstack-infra | 17:51 | |
| *** rbrndt has quit IRC | 17:51 | |
| fungi | brad_behle: yeah, publishing the images we're using is something we considered, but they're in the neighborhood of 5gb compressed each and have our access credentials baked into them, so would take some local surgery to make it possible for someone else to log in | 17:52 |
| clarkb | fungi: its actually 8GB compressed now and 20ish uncompressed | 17:52 |
| fungi | oh, yeeowch | 17:52 |
| fungi | guess i haven't looked at the image sizes in a while | 17:52 |
| brad_behle | clarkb, fungi: I don't suppose you have seen this requirement before, where a gate job has needed kernel patches or something else not in a standard cloud distro at the moment? | 17:52 |
| clarkb | those numbers fell slightly when I claered out the cache we are in the middle of rebuilding everything with the new cache | 17:52 |
| clarkb | brad_behle: not really | 17:53 |
| fungi | brad_behle: for upstream jobs we usually recommend running on a distro that has a new enough kernel (e.g. fedora 24) | 17:53 |
| clarkb | russellb at one time cared for a special image with a custom kernel compile iirc but that was it | 17:53 |
| clarkb | I think that lasted about a cycle? | 17:53 |
| fungi | but if you need a completely nonstandard kernel, not just a newer kernel, you'd currently require a separate node type with its own image | 17:54 |
| pabelanger | clarkb: only 1 launch failure in osic-cloud1 in the last 6 hours! | 17:54 |
| clarkb | pabelanger: nice | 17:54 |
| fungi | which we're hesitant to add without extremely good use cases due to the impact on our already strained image updating solution | 17:54 |
| *** e0ne has quit IRC | 17:55 | |
| *** zul has joined #openstack-infra | 17:55 | |
| clarkb | brad_behle: a better understanding of the requirements would definitely help | 17:55 |
| brad_behle | fungi: Okay, good to know. | 17:55 |
| clarkb | which is why I would sort those out first | 17:55 |
| fungi | though it might be interesting to some up with a way of saving job state on nodes and supporting controlled reboots during job runtime. it wouldn't be a trivial change to our framework though | 17:56 |
| fungi | s/some/come/ | 17:56 |
| brad_behle | clarkb: Yeah, agreed. Once I understand the requirements better, I'll definitely let you know :-) | 17:56 |
| *** eharney has quit IRC | 17:56 | |
| clarkb | fungi: I think zuulv3 will support that since ansible can have a reboot and wait portion of the playbook | 17:56 |
| timrc | We should add this image and node type for April 1 https://www.gnu.org/software/hurd/hurd/running/cloud.html | 17:57 |
| clarkb | timrc: and run pep8 on it right? | 17:58 |
| *** _nadya_ has joined #openstack-infra | 17:58 | |
| clarkb | (I assume hurd has working python but that might be a bad assumption) | 17:58 |
| fungi | clarkb: oh, that'll open up an entire new class of job options if we support rebooting at runtime. awesome | 17:58 |
| timrc | no virtio drivers though, that's rough | 17:58 |
| *** dimtruck is now known as zz_dimtruck | 17:59 | |
| *** baoli has quit IRC | 17:59 | |
| fungi | clarkb: for example rebooting a subnode in a multi-node job onto a separate decompressed image to work around lack of nested vir acceleration | 18:00 |
| fungi | er, virt | 18:00 |
| *** e0ne has joined #openstack-infra | 18:00 | |
| *** _sarob has joined #openstack-infra | 18:01 | |
| AJaeger | infra cores, could you review my changes to move other-requirements to bindep, please? https://review.openstack.org/#/q/topic:bindep-mv+projects:openstack-infra+status:open is the list of open changes | 18:01 |
| *** zz_dimtruck is now known as dimtruck | 18:01 | |
| *** ansmith has quit IRC | 18:01 | |
| *** jamielennox|away is now known as jamielennox | 18:01 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Merge zuul-git-prep-upper-constraints/zuul-release-git-prep-upper-constraints https://review.openstack.org/352575 | 18:02 |
| clarkb | and now just ovh bhs1 lacks the new xenial iamge | 18:03 |
| *** baoli has joined #openstack-infra | 18:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Merge zuul-git-prep-upper-constraints/zuul-release-git-prep-upper-constraints https://review.openstack.org/352575 | 18:04 |
| *** sarob has quit IRC | 18:04 | |
| pabelanger | clarkb: jeblair: I'd like to see if we can clean-up the following error message: http://paste.openstack.org/show/562528/ | 18:04 |
| *** DrifterZA has joined #openstack-infra | 18:04 | |
| pabelanger | clarkb: jeblair: it doesn't do much today in nodepool except increase the size of our log file | 18:04 |
| AJaeger | fungi, for infra-specs I have a sphinx warning fix - could you review https://review.openstack.org/352218 , please? | 18:04 |
| clarkb | pabelanger: the ssh connect time out should be user configurable | 18:05 |
| clarkb | pabelanger: maybe we just need to increase the timeout? | 18:05 |
| pabelanger | clarkb: Ya, we could do that. I just need to figure out which cloud it is | 18:05 |
| AJaeger | mtreinish, mordred : could you review my constraints change for cookiecutter, please? https://review.openstack.org/#/c/352758/ | 18:06 |
| *** jcoufal has joined #openstack-infra | 18:06 | |
| cloudnull | pabelanger clarkb looking no w | 18:07 |
| *** ilyashakhat has quit IRC | 18:07 | |
| cloudnull | sorry split brained today | 18:07 |
| *** eharney has joined #openstack-infra | 18:08 | |
| anteaya | just today? | 18:08 |
| anteaya | I consider that normal operating mode | 18:08 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test. https://review.openstack.org/346949 | 18:08 |
| cloudnull | ah neutron cant deal with X-Forward-For in liberty. | 18:09 |
| cloudnull | when we upgrade to mitaka thats supposedly fixed | 18:09 |
| cloudnull | ^ clarkb pabelanger | 18:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Include ip address for ssh_connect exception https://review.openstack.org/359369 | 18:10 |
| cloudnull | ^^ RE: network list failing | 18:10 |
| pabelanger | clarkb: jeblair: that should give us some additional information on failure ^ | 18:10 |
| *** _nadya_ has quit IRC | 18:10 | |
| pabelanger | cloudnull: okay, cool. That explains the failure | 18:10 |
| *** csomerville has quit IRC | 18:10 | |
| cloudnull | yea we treid to add rewrite rules on the F5 to deal with that however it failed misserably and caused other issues | 18:11 |
| cloudnull | so we left it as-is and know its a limitation | 18:11 |
| cloudnull | nova net-list still works to list networks though | 18:11 |
| cloudnull | which may be of use? | 18:12 |
| *** hockeynut has joined #openstack-infra | 18:13 | |
| anteaya | fungi: the meetbot in -meeting didn't show the results of a vote that just occured | 18:13 |
| anteaya | is there a solution? | 18:13 |
| anteaya | fungi: keystone has moved on, but it is odd behaviour for the meetbot | 18:14 |
| *** rbrndt has joined #openstack-infra | 18:14 | |
| *** bin_ has quit IRC | 18:14 | |
| fungi | anteaya: not sure. are the vote options there case sensitive? i know it's also picky about having more than one space separating commands and data | 18:15 |
| pabelanger | cloudnull: okay, let me try that | 18:15 |
| *** amitgandhinz has quit IRC | 18:15 | |
| pleia2 | anteaya: I wonder if adding so many invalid choices at the end confused things? | 18:15 |
| anteaya | it is possible | 18:15 |
| pleia2 | (it really is just yes/no, not all the rest) | 18:15 |
| anteaya | I just witnessed it and thought it was odd | 18:16 |
| *** caowei has quit IRC | 18:16 | |
| *** amitgandhinz has joined #openstack-infra | 18:16 | |
| anteaya | they don't seem concerned | 18:16 |
| pleia2 | yeah | 18:16 |
| anteaya | so we don't have to do anything I guess, just datapoint | 18:16 |
| *** caowei has joined #openstack-infra | 18:16 | |
| fungi | one person did #vote Yes (two spaces) and the rest did #vote yes or #vote no (lower-case) | 18:16 |
| *** csomerville has joined #openstack-infra | 18:17 | |
| *** jcoufal has quit IRC | 18:17 | |
| anteaya | would the two spaces make meetbot forget the results? | 18:17 |
| fungi | while the bot said the valid choices were Yes and No (upper-cased first letter) | 18:17 |
| anteaya | oh | 18:18 |
| anteaya | which noone typed | 18:18 |
| fungi | so it's possible it thought none of those were valid votes | 18:18 |
| anteaya | so yeah if case senstivive it got no votes | 18:18 |
| anteaya | wow | 18:18 |
| *** degorenko is now known as _degorenko|afk | 18:18 | |
| fungi | i don't know if #vote followed by more than one space is a problem, but we recently saw someone trying to #startmeeting something (two spaces) and it consistently ignoring them | 18:19 |
| openstackgerrit | Henry Gessau proposed openstack-infra/project-config: Fix neutron failure rates dashboard integrated jobs list https://review.openstack.org/358462 | 18:19 |
| anteaya | interesting | 18:19 |
| *** javeriak_ has joined #openstack-infra | 18:19 | |
| fungi | so at least some meetbot commands are sensitive to having an extra separating space | 18:19 |
| anteaya | it would seem so | 18:19 |
| anteaya | now I know | 18:19 |
| *** csomerville has quit IRC | 18:20 | |
| anteaya | I'll try to share our theory during open discussion | 18:20 |
| anteaya | fungi: pleia2 thank you | 18:20 |
| *** javeriak has quit IRC | 18:21 | |
| *** electrofelix has quit IRC | 18:21 | |
| *** Sukhdev has joined #openstack-infra | 18:22 | |
| clarkb | that part of the parser (the command part) is all in meetbot proper | 18:23 |
| *** javeriak has joined #openstack-infra | 18:23 | |
| clarkb | the case sensitivity is in the bits I added, we could make it case insensitive if we wanted | 18:23 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Switch from old recaptcha to recaptcha-nocaptcha https://review.openstack.org/358202 | 18:23 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Clean up old recaptcha parameters https://review.openstack.org/358210 | 18:23 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Parameterize database connection settings https://review.openstack.org/358195 | 18:23 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Update scope.lookupvar() calls to shorter @ lookup https://review.openstack.org/358194 | 18:23 |
| anteaya | clarkb: what was the reason for adding case sensitivity? | 18:23 |
| *** raunak has quit IRC | 18:23 | |
| fungi | yolanda: ^ updated from scope[] to @ per your suggestion | 18:23 |
| openstackgerrit | Merged openstack-infra/infra-specs: Fix warnings https://review.openstack.org/352218 | 18:24 |
| *** pvaneck has joined #openstack-infra | 18:25 | |
| *** javeriak_ has quit IRC | 18:25 | |
| *** pfallenop has quit IRC | 18:25 | |
| *** jcoufal has joined #openstack-infra | 18:26 | |
| clarkb | anteaya: things are case sensitive by default typically we didnt add it souch as we didnt make an effort to make it insensitive | 18:27 |
| *** AnarchyAo has left #openstack-infra | 18:28 | |
| *** csomerville has joined #openstack-infra | 18:28 | |
| anteaya | oh sorry, I interpreted "the case sensitivity is in the bits I added" as you added case sensitivity | 18:28 |
| anteaya | but you mean it was included in other things you added | 18:28 |
| anteaya | +1 make meetbot case insensitive | 18:29 |
| openstackgerrit | Merged openstack-dev/cookiecutter: Adjust tox.ini for constraints https://review.openstack.org/352758 | 18:29 |
| *** pfallenop has joined #openstack-infra | 18:30 | |
| mordred | pabelanger: I see your note above about launch_node and osic | 18:30 |
| mordred | pabelanger: did you get anywhere with that or should I look? | 18:31 |
| pabelanger | mordred: I'm trying to hack around it | 18:32 |
| pabelanger | by using nics | 18:32 |
| pabelanger | and not network | 18:32 |
| pabelanger | the reason this works in nodepool, is because we do that, nics | 18:32 |
| pabelanger | otherwise, we'd have the same problem | 18:32 |
| clarkb | pabelanger: mordred it might work if you tell shade to not neutron in clouds.yanl | 18:32 |
| clarkb | similar to what we did in tripleo cloud | 18:33 |
| pabelanger | we can try that | 18:33 |
| pabelanger | but I have a server launching now with my hack | 18:33 |
| clarkb | cool | 18:33 |
| pabelanger | but, it would be good to add --nics support to launch-node | 18:34 |
| clarkb | happy to use nics too if that works | 18:34 |
| pabelanger | I think we need to do that either way, since I cannot see how network support more then 1 | 18:34 |
| mordred | clarkb: hang on | 18:34 |
| AJaeger | fungi wrote a change to update zuul-cloner envs for system-config, please review https://review.openstack.org/359352 - this fixes translation sync with constraints. | 18:34 |
| mordred | this is working fine for me when I'm doing shade in the repl | 18:35 |
| mordred | so gimme a sec to see where it's going south in launch_node | 18:35 |
| *** amitgandhinz has quit IRC | 18:35 | |
| *** ilyashakhat has joined #openstack-infra | 18:36 | |
| *** amitgandhinz has joined #openstack-infra | 18:36 | |
| mordred | Network ['GATEWAY_NET_V6'] is not a valid network in openstackci-osic-cloud1:RegionOne | 18:36 |
| pabelanger | ya, that is the error | 18:36 |
| mordred | that says to me that something is passing GATEWAY_NET_V6 into shade as a list | 18:36 |
| mordred | 'Network {network} is not a valid network in' | 18:36 |
| mordred | ' {cloud}:{region}'.format( | 18:36 |
| mordred | network=network, | 18:37 |
| pabelanger | Oh, hmm | 18:37 |
| mordred | I don't see anything that would cause such a thing to be true though | 18:38 |
| mordred | pabelanger: are you passing in more command line than I see in that paste? | 18:38 |
| pabelanger | mordred: that is likely my fault, I was using a list to see if I could do more then 1 network | 18:39 |
| mordred | ah - you cannot | 18:39 |
| pabelanger | ya, that's why I switched to nics | 18:40 |
| mordred | network is a convenience setting for if you have a single network you're specifying | 18:40 |
| mordred | cool. that will work much betterer | 18:40 |
| pabelanger | when I revert --network works correctly | 18:40 |
| mordred | woot | 18:40 |
| *** kzaitsev_mb has quit IRC | 18:41 | |
| clarkb | we probably want multi nic support anywyas ya? | 18:41 |
| pabelanger | I think so | 18:41 |
| *** dtantsur|afk has quit IRC | 18:41 | |
| mordred | yah - that's just what nics is there for. now, that said - we could also add support to shade to handle network as a list with not much effort | 18:42 |
| mordred | since the input format of nics is annoying | 18:42 |
| pabelanger | I would not complain about that | 18:43 |
| *** ddieterly[away] is now known as ddieterly | 18:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support more than one network in create_server https://review.openstack.org/359378 | 18:45 |
| mordred | there | 18:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support more than one network in create_server https://review.openstack.org/359378 | 18:46 |
| mordred | sorry, updated docstring | 18:46 |
| *** zul has quit IRC | 18:46 | |
| mordred | I should really make that check to see if any of the entries are already dictlike ... | 18:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support more than one network in create_server https://review.openstack.org/359378 | 18:48 |
| *** _nadya_ has joined #openstack-infra | 18:48 | |
| mordred | there. that's more complete and also now not a syntax error :) | 18:48 |
| *** nwkarsten has quit IRC | 18:48 | |
| mordred | (you should still hack around this in launch_node for now, because gate length today) | 18:49 |
| *** nwkarsten has joined #openstack-infra | 18:49 | |
| pabelanger | Sure, that is no issue | 18:49 |
| pabelanger | but I'll update launch-node.py to use that now | 18:49 |
| *** _nadya_ has quit IRC | 18:52 | |
| openstackgerrit | Marc Aubry proposed openstack-infra/project-config: Add python34-jobs and python35-jobs to Almanach https://review.openstack.org/359345 | 18:53 |
| *** nwkarste_ has joined #openstack-infra | 18:54 | |
| *** nwkarsten has quit IRC | 18:54 | |
| rcarrillocruz | back | 18:55 |
| *** raunak has joined #openstack-infra | 18:55 | |
| *** raunak has quit IRC | 18:56 | |
| *** javeriak has quit IRC | 18:56 | |
| *** dtardivel has quit IRC | 18:57 | |
| *** ilyashakhat has quit IRC | 18:57 | |
| *** markvoelker has joined #openstack-infra | 18:58 | |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 19:00 |
| *** hasharAway is now known as hashar | 19:00 | |
| *** zul has joined #openstack-infra | 19:01 | |
| *** markvoelker has quit IRC | 19:01 | |
| *** Hal has quit IRC | 19:02 | |
| jeblair | fungi: if you're on the waiting list, does that mean you have to stay in the beer garden? | 19:02 |
| *** tqtran has joined #openstack-infra | 19:04 | |
| *** ddieterly is now known as ddieterly[away] | 19:04 | |
| *** piet has joined #openstack-infra | 19:05 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 19:05 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 19:06 |
| fungi | jeblair: that sounds like a compelling argument in favor of procrastination | 19:06 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 19:06 |
| *** kushal has quit IRC | 19:09 | |
| *** dizquierdo has joined #openstack-infra | 19:11 | |
| *** tonytan_brb has joined #openstack-infra | 19:11 | |
| *** DrifterZA has quit IRC | 19:12 | |
| *** spzala has joined #openstack-infra | 19:13 | |
| *** ansmith has joined #openstack-infra | 19:13 | |
| *** tonytan4ever has quit IRC | 19:14 | |
| *** Hal has joined #openstack-infra | 19:15 | |
| *** jtomasek has joined #openstack-infra | 19:18 | |
| pabelanger | new address for osic-cloud1 mirror | 19:20 |
| pabelanger | IPV4=172.99.106.178 | 19:20 |
| pabelanger | IPV6=2001:4800:1ae1:18:f816:3eff:fe6f:dd6f | 19:20 |
| *** jtomasek has quit IRC | 19:20 | |
| *** jtomasek_ has joined #openstack-infra | 19:20 | |
| pabelanger | need to confirm IPv6 security group access | 19:20 |
| *** pcaruana has quit IRC | 19:20 | |
| *** andymaier_ has joined #openstack-infra | 19:21 | |
| pabelanger | actually, cloud_launcher can do that | 19:22 |
| clarkb | pabelanger: I want to say I always open both even if cloud doesnt have v6 for precisely this reason | 19:22 |
| clarkb | and if it isnt open already then we should definitely do it that eay | 19:22 |
| irtermite | Heads up... doing SSL renew now | 19:22 |
| dmsimard | Is review.o.o working for you guys ? | 19:23 |
| pabelanger | clarkb: ya, add the rule into cloud_launcher last week, just need to confirm openstackci-osic-cloud1 is there | 19:24 |
| *** ddieterly[away] is now known as ddieterly | 19:24 | |
| pabelanger | irtermite: yup, just had a big spike in errors in osic-cloud1 | 19:24 |
| dmsimard | Unable to reach review.o.o on my end.. http://paste.openstack.org/show/562542/ | 19:25 |
| irtermite | pabelanger: thank you. all good now | 19:25 |
| irtermite | check again pabelanger | 19:26 |
| sdake | gerrit seems pokey | 19:26 |
| sdake | is it just me? | 19:26 |
| pabelanger | irtermite: http://grafana.openstack.org/dashboard/db/nodepool-osic | 19:26 |
| dmsimard | sdake: I'm seeing it too but openstack-infra is in a meeting | 19:26 |
| pabelanger | irtermite: what I was watching | 19:26 |
| sdake | dmsimard roger | 19:26 |
| *** kaisers_ has joined #openstack-infra | 19:26 | |
| irtermite | small error... selected the old intermediate with the new cert on mistake. fixed it right away, pabelanger | 19:26 |
| *** gyee has quit IRC | 19:27 | |
| irtermite | annnnd spike gone, pabelanger | 19:27 |
| irtermite | sorry about that | 19:27 |
| fungi | i seem to be getting to gerrit okay, though it is a mite slow | 19:28 |
| *** salv-orlando has quit IRC | 19:28 | |
| fungi | might check http://cacti.openstack.org/ for anything anomalous? | 19:28 |
| dmsimard | irtermite: that spike you mention, was that anything to do with review.o.o ? | 19:28 |
| rcarrillocruz | dmsimard: it was fair, it took a long time for me to load up | 19:28 |
| *** salv-orlando has joined #openstack-infra | 19:28 | |
| *** hockeynut has quit IRC | 19:28 | |
| pabelanger | irtermite: yup, will keep an eye on it | 19:28 |
| irtermite | dmsimard: nope... that was us updating the ssl cert | 19:29 |
| irtermite | it's better now >.< | 19:29 |
| dmsimard | irtermite: ok | 19:29 |
| irtermite | dmsimard: accidentally added the wrong intermediate but quickly fixed | 19:29 |
| *** kaisers_ has quit IRC | 19:30 | |
| irtermite | always safe to blame me when things break here... | 19:32 |
| *** salv-orlando has quit IRC | 19:32 | |
| *** ewindisch_ has joined #openstack-infra | 19:35 | |
| *** ewindisch has quit IRC | 19:36 | |
| *** ewindisch_ is now known as ewindisch | 19:36 | |
| *** piet has quit IRC | 19:36 | |
| *** esikachev has quit IRC | 19:38 | |
| mordred | clarkb, pabelanger: https://review.openstack.org/#/c/358232/ made it through the gate gauntlet and is ready for re-review | 19:41 |
| mordred | Shrews: ^^ | 19:41 |
| openstackgerrit | Marc Aubry proposed openstack-infra/project-config: Add python34-jobs and python35-jobs to Almanach https://review.openstack.org/359345 | 19:42 |
| *** markusry has quit IRC | 19:44 | |
| *** raunak has joined #openstack-infra | 19:45 | |
| *** markusry has joined #openstack-infra | 19:45 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling (local loop) https://review.openstack.org/319591 | 19:45 |
| *** markusry has quit IRC | 19:45 | |
| *** andymaier_ has quit IRC | 19:46 | |
| dougwig | fungi: another node that went offline: telnet -6 2001:4800:1ae1:18:f816:3eff:fe0b:a9e0 19885 | 19:47 |
| dougwig | fungi: just re-queued, so it might still be around. | 19:47 |
| fungi | if only i weren't chaiting a meeting | 19:47 |
| fungi | chairing | 19:47 |
| rm_work | :( | 19:47 |
| dougwig | fungi: ok, we'll catch another later. | 19:48 |
| *** Apoorva has quit IRC | 19:48 | |
| *** andymaier_ has joined #openstack-infra | 19:49 | |
| fungi | it's still live (288b718b-f453-4f4c-8460-9475f3af02fd in osic) but i'm spacing on the openstack server console syntax (that isn't it apparently) | 19:49 |
| *** xarses has quit IRC | 19:50 | |
| *** Na3iL has quit IRC | 19:50 | |
| *** andymaier_ has quit IRC | 19:50 | |
| *** xarses has joined #openstack-infra | 19:51 | |
| pabelanger | clarkb: next issue with osic-cloud1 mirror replacement, only 1 interface was configured by cloud-init, meaning we are still missing ipv6 | 19:51 |
| pabelanger | clarkb: will dive into it shortly | 19:51 |
| clarkb | pabelanger: wow fun | 19:51 |
| *** xarses has quit IRC | 19:51 | |
| *** andymaier_ has joined #openstack-infra | 19:51 | |
| *** xarses has joined #openstack-infra | 19:52 | |
| *** Apoorva has joined #openstack-infra | 19:52 | |
| pabelanger | config-drive is enabled | 19:52 |
| clarkb | pabelanger: there should be a cloud init log somewhere iirc | 19:52 |
| clarkb | pabelanger: maybe that will say what it did and why | 19:53 |
| mordred | clarkb, pabelanger: I think the stock ubuntu images are only configured with one nic listening to dhcp | 19:53 |
| *** asettle has joined #openstack-infra | 19:53 | |
| mordred | and I don't think cloud-init is involved | 19:54 |
| mordred | I'm pretty sure if you just add another auto dhcp line to /etc/network/interfaces | 19:54 |
| mordred | and do an ifup eth1 | 19:54 |
| mordred | it _should_ work | 19:54 |
| clarkb | well it shouldnt dhcp there | 19:54 |
| clarkb | I dont think | 19:54 |
| russellb | brad_behle: we just compile the ovs kernel module from the ovs tree for networking-ovn jobs. are you saying that's not good enough for some cases? | 19:54 |
| pabelanger | http://paste.openstack.org/show/562543/ | 19:54 |
| clarkb | ptovably just need to ifup it | 19:54 |
| clarkb | because ipv6 magic | 19:55 |
| krotscheck | Any cores around to assist with a few npm-based build updates? Adding docs builds, devstack, etc. https://review.openstack.org/#/q/topic:npm+status:open | 19:55 |
| mordred | clarkb: ++ | 19:55 |
| mordred | clarkb: yah - whatever the magic is - I mostly think we just need to enable the nic | 19:55 |
| brad_behle | russellb: From what I understand, some of the SNAT and Floating IP work for OVN requires some very recent linux kernel enhancements (I think in conntrack) | 19:56 |
| russellb | brad_behle: yes, but all of that is backported into the ovs kernel module from the ovs git tree | 19:56 |
| pabelanger | mordred: clarkb: Ya, adding configuration for it works. So, we'll need to add this to system-config | 19:56 |
| mordred | pabelanger: cool | 19:56 |
| mordred | pabelanger: I support adding such config to system-config | 19:57 |
| *** markvoelker has joined #openstack-infra | 19:57 | |
| pabelanger | okay, let me get the volume going again | 19:57 |
| krotscheck | AJaeger: Thanks :D | 19:57 |
| russellb | brad_behle: https://github.com/openvswitch/ovs/blob/master/FAQ.md ... see "Q: Are all features available with all datapaths?" and the "Linux OVS tree" column | 19:57 |
| *** ddieterly is now known as ddieterly[away] | 19:58 | |
| *** asettle has quit IRC | 19:58 | |
| russellb | clarkb: i think i caught a question about vxlan and ipv6 in scrollback a couple days ago? the link above covers that same thing -- vxlan ipv6 is available for ovs in 4.3+ or if you compile the ovs kernel module from the ovs tree (compatible back to 3.11 or something) | 19:58 |
| clarkb | russellb: so if the job builds the module and loads it is good? | 19:58 |
| russellb | clarkb: yes | 19:59 |
| russellb | that's what all OVN jobs do today | 19:59 |
| clarkb | russellb: ya we dont have ovs nearly that new | 19:59 |
| * russellb nods | 19:59 | |
| *** amotoki has quit IRC | 20:00 | |
| fungi | okay, so it's `openstack console log show` apparently, not namespaced under the server subcommand tree? | 20:01 |
| clarkb | russellb: mosyl just mind boggled ipv6 is so second class citizen with newish networking tools | 20:01 |
| *** xarses_ has joined #openstack-infra | 20:01 | |
| dhellmann | fungi : regarding your scheduling question; a maintenance window the day after milestone 3 feels a bit tight. I guess we're running out of good times at this point in the cycle, though. | 20:02 |
| fungi | dhellmann: so on the scheduling... do you feel like there'll be spillover into friday for the milestone releases? | 20:02 |
| mordred | clarkb: perhaps we should get newer ovs? | 20:02 |
| *** dizquierdo has quit IRC | 20:02 | |
| dhellmann | fungi : possibly in the morning us eastern if we have late tag submissions the day before | 20:02 |
| clarkb | mordred: that seems like a lot of work for all the distros | 20:02 |
| mordred | clarkb: oh. right. nevermind | 20:02 |
| clarkb | mordred: since we need it for the underlying multinode network stuff | 20:02 |
| mordred | clarkb: let's not do that | 20:02 |
| mordred | yah | 20:02 |
| *** ansmith has quit IRC | 20:02 | |
| mordred | I rescind my thought | 20:03 |
| russellb | compile it from master! | 20:03 |
| *** xarses has quit IRC | 20:03 | |
| russellb | what could go wrong | 20:03 |
| fungi | dhellmann: if it helps, we're scheduling the window for a much longer period than we expect to actually need (this is our first time trying online reindexing in production), but we're also talking about starting the work at 18:00 utc | 20:03 |
| mordred | clarkb: https://review.openstack.org/#/c/358232/ nudge | 20:03 |
| pabelanger | mirror.regionone.osic-cloud1.openstack.org replacement online, both with ipv4 / ipv6 | 20:04 |
| pabelanger | IPV4=172.99.106.183 | 20:04 |
| pabelanger | IPV6=2001:4800:1ae1:18:f816:3eff:fe0c:2c24 | 20:04 |
| pabelanger | will be updating DNS in a minute | 20:04 |
| mordred | pabelanger: woot! | 20:04 |
| irtermite | woot pabelanger ! | 20:04 |
| ianw | fungi | anyone : here's launch-node.py atm -> http://paste.openstack.org/show/562545/ ... that node comes up, has an ipv6 address in interfaces but it's not on eth0 | 20:04 |
| irtermite | uh, jinx, mordred ? | 20:04 |
| dhellmann | fungi : hmm, that's around noon US eastern | 20:04 |
| mordred | ianw: looking | 20:04 |
| fungi | dhellmann: 2pm us eastern according to my clock? | 20:04 |
| * dhellmann looks again | 20:05 | |
| dhellmann | oh, 18 not 16 | 20:05 |
| rcarrillocruz | well done pabelanger ;-) | 20:05 |
| brad_behle | russellb: Okay, I'll take a look at that FAQ. I'm in the process of building the ubuntu-trusty image that the gate uses, I plan to use that image to run the latest tests, to see if any fail due to needing an updated kernel | 20:05 |
| dhellmann | fungi : ok, that's a little better. let me confer with my veteran expert, but I think if we end up with a late tag we can wait and apply it monday | 20:05 |
| brad_behle | russellb, that would be great if we didn't require anything extra from the base image | 20:05 |
| dhellmann | fungi : I'll track down ttx here at the conference and see what he thinks and let you know | 20:06 |
| fungi | dhellmann: we can also delay/abort if it looks like release work is still underway. it's not terribly critical that we do it next week, it just seemed like one of the sooner possibilities | 20:06 |
| dhellmann | fungi : ok, let's say a tentative yes for now | 20:06 |
| fungi | thanks dhellmann! | 20:06 |
| russellb | brad_behle: OK, let me know what you find, i'm not aware of any reason we need a custom image | 20:06 |
| brad_behle | russellb: Will do, thanks! | 20:06 |
| pabelanger | and DNS update | 20:06 |
| pabelanger | wow | 20:06 |
| pabelanger | ipv6 traffic already | 20:07 |
| pabelanger | \o/ | 20:07 |
| mordred | ianw: that seems like a potential bug in the rackspace nova-agent or something, and should be solvable by rebooting? | 20:07 |
| *** ilyashakhat has joined #openstack-infra | 20:07 | |
| ianw | mordred: yes, or just ifdown && ifup ... but that doesn't really help finish launch-node.py :) | 20:07 |
| ianw | i mean, i could easily add that, if we just think it's something silly upstream & not my fault | 20:07 |
| pabelanger | #status log mirror.regionone.osic-cloud1.openstack.org upgraded to support both ipv4 / ipv6. DNS has also been updated. | 20:07 |
| openstackstatus | pabelanger: finished logging | 20:07 |
| openstackgerrit | Merged openstack-infra/project-config: Created DSVM Job for NPM Projects https://review.openstack.org/348056 | 20:08 |
| mordred | ianw: I'm not sure it's a thing launch-node is doing though :( | 20:08 |
| mordred | ianw: I mean, we could put in a launch-node "ifdown && ifup" ... | 20:08 |
| ianw | mordred: yeah, that's what i mean i could do, if it's not user-error on my behalf | 20:09 |
| mordred | ianw: I do not believe it is | 20:09 |
| mordred | ianw: I expect the things you did to result in a working in-guest network | 20:09 |
| ianw | yeah, i'm surprised nobody has noticed their rax vm's not booting with a working ipv6 address | 20:10 |
| *** esikachev has joined #openstack-infra | 20:10 | |
| mordred | this does seem to be new behavior | 20:10 |
| mordred | the last rax servers we spun up did not exhibit this to my knowledge | 20:11 |
| *** bhunter71 has quit IRC | 20:12 | |
| *** e0ne has quit IRC | 20:12 | |
| *** edmondsw has quit IRC | 20:13 | |
| rcarrillocruz | ianw: yeah, that's new | 20:13 |
| rcarrillocruz | cos i created firehose not that long ago | 20:13 |
| rcarrillocruz | and i do remember running ipv6 dns commands | 20:13 |
| phschwartz | jeblair: you around? I am getting a zuul error and wanted to see if you have seen it http://paste.openstack.org/show/562547/ | 20:15 |
| phschwartz | jeblair: been happening since early this morning. It has our whole set of queues blocked | 20:15 |
| ianw | mordred | rcarrillocruz : ok, trying with an ifdown && ifup in there | 20:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Re-enable the shared/independent queue test https://review.openstack.org/358010 | 20:17 |
| jeblair | phschwartz: looking | 20:17 |
| phschwartz | jeblair: I have never seen it out of no where from working projects start to throw project.name doesn't exist. We are tracking master in this env so not sure if anything changed, figured you would have the quickest insite | 20:18 |
| phschwartz | jeblair: and to note, what ever is causing this, is causing zuul to push the cpu top 100% and eat almost all ram | 20:20 |
| *** esikachev has quit IRC | 20:21 | |
| *** ddieterly[away] is now known as ddieterly | 20:21 | |
| *** ansmith has joined #openstack-infra | 20:21 | |
| dhellmann | fungi : I've talked with ttx, and we think we'll be ok with that gerrit maintenance window. We'll make the final call on any late tags earlier that day at the release team meeting and should have time to process them all before you start. | 20:22 |
| *** kgiusti has quit IRC | 20:22 | |
| jeblair | phschwartz: that doesn't ring a bell. has anything changed in your gerrit or zuul configuration recently? | 20:24 |
| *** e0ne has joined #openstack-infra | 20:25 | |
| AJaeger | infra-root, fungi wrote a change to update zuul-cloner envs for system-config, please review https://review.openstack.org/359352 - this fixes translation sync with constraints on the proposal slave. | 20:26 |
| fungi | dhellmann: ttx: perfect. as i said, let us know that day if release work is still underway and we can delay starting or reschedule as needed | 20:26 |
| dhellmann | fungi : yep, I'll leave myself a note to do that | 20:26 |
| fungi | dhellmann: i'll try to remember to check in with #openstack-release before we start as well | 20:27 |
| sdake | pabelanger - say I had to rebase https://review.openstack.org/#/c/349278/ | 20:27 |
| sdake | and your +2 fell off | 20:27 |
| sdake | any chance you could reaapply it | 20:27 |
| *** hockeynut has joined #openstack-infra | 20:27 | |
| mordred | clarkb, Shrews: while I've got you: https://review.openstack.org/#/c/359378/ and https://review.openstack.org/#/c/315697 are ready to fly (trying to burn down shade patches in the outstanding queue as quickly as the gate gives a green) | 20:28 |
| sdake | mordred - 12 pages of patches to review!! | 20:28 |
| sdake | 7 days to do it in | 20:28 |
| sdake | mission impossibe? | 20:28 |
| sdake | (yes) | 20:28 |
| ianw | mordred: ok, new error, from ansible ... -> http://paste.openstack.org/show/562549/ | 20:28 |
| phschwartz | jeblair: one project has been added, but it was working after it was. | 20:28 |
| *** sigmavirus is now known as sigmavirus|away | 20:29 | |
| jeblair | phschwartz: did you reconfigure zuul after adding the project? | 20:29 |
| *** NobodyCam has quit IRC | 20:29 | |
| phschwartz | jeblair: yeah, it auto reconfigures every 15 min | 20:29 |
| pabelanger | ianw: make sure you run launch-node.py as root, there is a bug where we don't copy hieradata properly | 20:29 |
| *** NobodyCam has joined #openstack-infra | 20:30 | |
| phschwartz | jeblair: only change since then could have been a new pull of master of zuul | 20:30 |
| ianw | pabelanger: ok ... should we add that to readme? | 20:30 |
| jeblair | phschwartz: you restarted zuul? | 20:30 |
| fungi | our beaker jobs for centos-7 seem to have started failing with a bundler error... anybody already looking into it? http://logs.openstack.org/94/358194/3/check/gate-openstackci-beaker-centos-7/5f54990/console.html#_2016-08-23_18_56_02_297505 | 20:30 |
| pabelanger | ianw: we do have a patch to fix it, it hasn't landed yet | 20:30 |
| jeblair | pabelanger: how about we land it? | 20:31 |
| phschwartz | jeblair: looks like it was restarted last night. issues started after that | 20:31 |
| pabelanger | 326649 | 20:31 |
| phschwartz | jeblair: also seeing http://paste.openstack.org/show/562550/ in the logs. Just noticed it | 20:31 |
| ianw | yeah, i'd prefer that than running as root ... to much chance i destroy something else :) | 20:31 |
| phschwartz | I might move us to a pin og the last stable release of zuul | 20:31 |
| *** coolsvap has quit IRC | 20:31 | |
| pabelanger | ianw: I think the issue you are having with ansible-playbook has been fixed in shade, but we don't have a new release yet | 20:32 |
| anteaya | fungi: it looks to me like the gem rubocop-rspec failed to download: http://logs.openstack.org/94/358194/3/check/gate-openstackci-beaker-centos-7/5f54990/console.html#_2016-08-23_18_56_01_976067 | 20:33 |
| mordred | pabelanger: getting closer | 20:33 |
| mordred | we're on the final burn-down list :) | 20:33 |
| *** itisha has joined #openstack-infra | 20:33 | |
| *** pt_15 has joined #openstack-infra | 20:33 | |
| fungi | anteaya: indeed. i wonder if something's wrong with a very recent release of that gem or something | 20:34 |
| pabelanger | ianw: http://paste.openstack.org/show/562551/ should work around the failure | 20:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: launch-node.py: restart interface https://review.openstack.org/359416 | 20:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: launch-node.py : save key when failing early https://review.openstack.org/359417 | 20:34 |
| jeblair | phschwartz: it might be helpful to know the sequence of relevant changes to the configuration of gerrit and zuul that preceded the first occurances of those errors, as well as more log context around them. | 20:34 |
| anteaya | fungi: it releaseed Aug 3 | 20:35 |
| anteaya | surely we have run beaker tests since that | 20:35 |
| jeblair | phschwartz: basically, they are both errors that should never happen, so in order to track them down, we'll need a lot more data | 20:35 |
| fungi | anteaya: hrm... almost three weeks ago | 20:35 |
| fungi | anteaya: yeah, the job was passing earlier today | 20:35 |
| anteaya | or maybe not: rubocop-rspec requires Ruby version >= 2.2.0 | 20:35 |
| anteaya | we are ruby 1.9 are we not? | 20:35 |
| anteaya | or did we move to ruby 2+ | 20:35 |
| fungi | anteaya: on centos 7? not sure | 20:35 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: launch-node.py: save key when failing early https://review.openstack.org/359417 | 20:36 |
| fungi | anteaya: there's a warning further up about "Rubygems 2.0.14 is not threadsafe" so i'm guessing it's that | 20:36 |
| phschwartz | jeblair: what could I provide to get some help. I am stuck and our zuul is too. lol | 20:36 |
| anteaya | fungi: yeah I'm just seeing that | 20:36 |
| anteaya | not sure how we could have been passing tests for the last 3 weeks if that is the error yet failing today | 20:37 |
| mordred | phschwartz: did you see: 20:34:36 jeblair | phschwartz: it might be helpful to know the sequence of relevant changes to the configuration of gerrit and zuul that preceded the first occurances of those errors, as well as more log | 20:37 |
| mordred | | context around them. | 20:37 |
| *** gyee has joined #openstack-infra | 20:37 | |
| *** csomerville has quit IRC | 20:37 | |
| *** e0ne has quit IRC | 20:37 | |
| phschwartz | mordred: no, didn't get that log line. yay irc that hates me | 20:38 |
| mordred | yay! | 20:38 |
| anteaya | in any case, my read is that upgrading to ruby 2.2 or downgrading rubocop-rspec gem to maybe 1.5.1 is the route | 20:38 |
| mordred | phschwartz: there was also: 20:35:15 jeblair | phschwartz: basically, they are both errors that should never happen, so in order to track them down, we'll need a lot more data | 20:38 |
| ianw | pabelanger: ok, trying again ... | 20:38 |
| phschwartz | jeblair: so, no changes to gerrit have been made. the only changes to zuul were moving a job from being a single job called twice to having a passed suffix on the jjb template, and then moving the job from experimental to being in the check queue | 20:39 |
| pabelanger | anteaya: fungi: I believe EmilienM is also fighting that fire now too. | 20:39 |
| EmilienM | yes | 20:39 |
| anteaya | EmilienM: awesome | 20:39 |
| jeblair | phschwartz: you said you added a project to gerrit? | 20:39 |
| phschwartz | jeblair: not gerrit, to zuul | 20:39 |
| anteaya | I can't find the version notes on the gems so I have no idea why 3 version were released the first week of august | 20:40 |
| EmilienM | https://tickets.puppetlabs.com/browse/MODULES-3776 | 20:40 |
| * AJaeger waves good night | 20:40 | |
| anteaya | AJaeger: good night | 20:40 |
| EmilienM | we're doing https://review.openstack.org/#/c/359385/ | 20:40 |
| anteaya | AJaeger: thank you for a great day | 20:40 |
| phschwartz | jeblair: pm'ed you 2 links | 20:40 |
| jeblair | phschwartz: can you paste a much longer log segment that preceeded the first error? and let me know what the names of the jobs and projects you changed are? | 20:40 |
| AJaeger | thanks, anteaya ! Enjoy the rest of the day! | 20:40 |
| EmilienM | anteaya: you need to do the same for https://github.com/openstack-infra/puppet-openstack_infra_spec_helper | 20:41 |
| phschwartz | jeblair: let me get that paste for you | 20:41 |
| EmilienM | nibalizer: fyi ^ | 20:41 |
| anteaya | AJaeger: thank you | 20:41 |
| anteaya | EmilienM: thank you | 20:41 |
| ianw | pabelanger: same issue | 20:41 |
| EmilienM | jeblair: an FYI about https://review.openstack.org/#/c/356675/ -- I would like you to look when you have time, in the commit message you'll find out a nice feature we could have in zuul later | 20:42 |
| nibalizer | EmilienM: whats up? | 20:42 |
| EmilienM | nibalizer: all puppet syntax jobs fail, https://tickets.puppetlabs.com/browse/MODULES-3776 | 20:42 |
| anteaya | nibalizer: we need https://review.openstack.org/#/c/359385/1 in infra spec helper | 20:43 |
| EmilienM | nibalizer: we're solving the problem with https://review.openstack.org/#/c/359385/ for now, you'll need to do the same for infra modules | 20:43 |
| EmilienM | nibalizer: not sure all infra modules use infra spec helper though | 20:43 |
| ianw | also, why so much ... [WARNING]: log file at /var/log/ansible.log is not writeable and we cannot create it, aborting | 20:43 |
| jeblair | EmilienM: that will be easier in zuulv3 :) | 20:43 |
| pabelanger | ianw: log? | 20:43 |
| pabelanger | ianw: file permissions issues | 20:43 |
| ianw | pabelanger: yeah ... but seems like we should not be logging to that? | 20:44 |
| phschwartz | jeblair: http://paste.openstack.org/show/562552/ http://paste.openstack.org/show/562553/ | 20:44 |
| phschwartz | jeblair: large chunk of both debug.log and zuul.log | 20:44 |
| EmilienM | jeblair: great, please comment then, I'm curious how :) | 20:44 |
| *** ansmith has quit IRC | 20:45 | |
| ianw | pabelanger: same as in http://paste.openstack.org/show/562549/ ...error while accessing the file /etc/puppet/hieradata/production/common.yaml | 20:45 |
| nibalizer | EmilienM: oh hahhahah rip | 20:45 |
| jeblair | phschwartz: thanks -- can you grab a chunk of lines before the first instance of those errors? | 20:45 |
| *** raildo has quit IRC | 20:46 | |
| nibalizer | jesusaur: do we still need https://review.openstack.org/#/c/350835/ ? | 20:46 |
| pabelanger | ianw: right, we need to land https://review.openstack.org/#/c/326649 | 20:47 |
| *** priteau has joined #openstack-infra | 20:47 | |
| pabelanger | ianw: that fixes the first problem | 20:48 |
| phschwartz | jeblair: let me dig, there are thousands of those errors. 10-15 going in a min | 20:48 |
| pabelanger | ianw: the patch I linked to you, you might not need | 20:48 |
| jesusaur | nibalizer: I'm not sure why that affected gozer and not infra, but it doesn't seem like there have been any issues with that upstream | 20:48 |
| jeblair | EmilienM: overrides of job attributes via project-local job-specification -- http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html#jobs | 20:48 |
| EmilienM | jeblair: that's awesome :) | 20:48 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin puppetlabs-spec-helper https://review.openstack.org/359421 | 20:49 |
| *** kzaitsev_mb has joined #openstack-infra | 20:49 | |
| nibalizer | jesusaur: EmilienM https://review.openstack.org/359421 | 20:49 |
| EmilienM | nibalizer: +1 | 20:49 |
| ianw | pabelanger: ok, cool, let's start with that then ... | 20:49 |
| jesusaur | nibalizer: also today there was a rubocop upgrade that now requires ruby ~> 2.0 | 20:50 |
| *** markvoelker has quit IRC | 20:52 | |
| nibalizer | fun | 20:52 |
| phschwartz | jeblair: It will be a couple more min. There are 400k lines of that error in the log | 20:52 |
| jeblair | phschwartz: you will probably need to restart zuul to fix this | 20:53 |
| *** ilyashakhat has quit IRC | 20:53 | |
| openstackgerrit | Merged openstack-infra/shade: Add support for fetching console logs from servers https://review.openstack.org/358232 | 20:54 |
| *** eggshell has joined #openstack-infra | 20:54 | |
| *** eggshell has quit IRC | 20:54 | |
| *** eggshell has joined #openstack-infra | 20:55 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: TripleO scenario001 experimental job https://review.openstack.org/356675 | 20:55 |
| fungi | thanks EmilienM! | 20:55 |
| EmilienM | fungi: well, all credits go to mwhahaha (Alex) | 20:55 |
| fungi | thanks, mwhahaha! | 20:56 |
| anteaya | yeah should we credit mwhahaha on that patch commit message, nibalizer | 20:56 |
| fungi | and nibalizer! | 20:56 |
| ianw | pabelanger: so we set log_path in /etc/ansible/ansible.cfg -- perhaps we shouldn't and just let syslog handle it -> http://docs.ansible.com/ansible/intro_configuration.html#log-path | 20:56 |
| mwhahaha | :o | 20:56 |
| ianw | or open permissions on /var/log/ansible.log, that seems bad | 20:57 |
| nibalizer | anteaya: ok | 20:57 |
| anteaya | nibalizer: thank you | 20:57 |
| *** gouthamr has quit IRC | 20:58 | |
| pabelanger | ianw: or have launch-node.py handle it. | 20:58 |
| *** jkilpatr has quit IRC | 20:58 | |
| mordred | clarkb: how's your upper-constraints/devstack zen? | 20:58 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin puppetlabs-spec-helper https://review.openstack.org/359421 | 20:58 |
| anteaya | can constraints and zen be used in the same sentence? | 20:58 |
| mordred | :) | 20:58 |
| pabelanger | ianw: we can add it to JobDir(), thats how we do it with zuul-launcher | 20:58 |
| *** xarses_ is now known as xarses | 20:59 | |
| *** ccarmack has joined #openstack-infra | 21:00 | |
| *** jswarren has joined #openstack-infra | 21:00 | |
| dougwig | fungi: just reset again, but missed grabbing the link. and given the overall runtime of most of the check queue, i think there's something weird going on. | 21:01 |
| *** amotoki has joined #openstack-infra | 21:01 | |
| jeblair | dougwig: i'm not following | 21:01 |
| *** rfolco has quit IRC | 21:01 | |
| *** tonytan_brb has quit IRC | 21:01 | |
| pabelanger | dougwig: it is likely ansible losing connection with the node the job runs | 21:02 |
| *** asettle has joined #openstack-infra | 21:02 | |
| rm_work | pabelanger: well it makes sense that ansible would lose connection, because the nodes are becoming totally unconnectable | 21:02 |
| pabelanger | there is some logic in zuul-launcher to requeue the job if network is unavailable | 21:02 |
| rm_work | that's the issue we're seeing | 21:02 |
| dougwig | jeblair: dsvm nodes seem to be resetting intermittently, only seen so far on nodes with ipv6 addresses. instead of a runtime of about an hour, we're at nearly 3 and counting. and this one review isn't alone. | 21:02 |
| dougwig | jeblair: i've watched one job almost "finish" and reset three times now. | 21:02 |
| rm_work | basically the nodes become completely unresponsive, thus causing a reset | 21:02 |
| rm_work | it's happening on more than just our change, I think | 21:03 |
| ianw | pabelanger: that makes sense, i'll do that | 21:03 |
| jeblair | dougwig: what jobs are involved? | 21:03 |
| *** jcoufal has quit IRC | 21:03 | |
| dougwig | jeblair: i have to step away, but rm_work can give details. back in 30. | 21:04 |
| rm_work | I'm looking for other examples | 21:04 |
| rm_work | so far i've seen it happen specifically on the octavia gate with: | 21:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Directly use pip instead of setup_develop in plugin https://review.openstack.org/359425 | 21:05 |
| rm_work | gate-neutron-lbaasv2-dsvm-api-namespace-nv | 21:05 |
| mordred | clarkb, jeblair, pabelanger: ^^ sigh - getting caught by devstack's defaulting to using upper-constraints in our shade-nodepool job | 21:05 |
| rm_work | gate-neutron-lbaasv2-dsvm-* | 21:05 |
| anteaya | rm_work: is it just affecting octavia patches? | 21:05 |
| rm_work | but I think it's probably doing it in other projects/CRs as well | 21:05 |
| rm_work | because I have seen others have jobs reset | 21:06 |
| phschwartz | jeblair: Here is one around the first error http://paste.openstack.org/show/562556/ | 21:06 |
| rm_work | and look at the queue length at the moment... | 21:06 |
| *** amotoki has quit IRC | 21:06 | |
| phschwartz | jeblair: working on second | 21:06 |
| mordred | Shrews: you may also want to look at that, although you may not | 21:06 |
| anteaya | rm_work: can you recall which projects? | 21:06 |
| rm_work | i'm looking for other examples at the moment | 21:06 |
| anteaya | rm_work: thanks | 21:06 |
| jeblair | rm_work: the queue length is being reduced at the rate i would expect | 21:06 |
| pabelanger | jeblair: I am seeing a lot of exit code 3 on zl01: http://paste.openstack.org/show/562557/ | 21:06 |
| anteaya | rm_work: having the check queue long the week prior to feature freeze is expected | 21:06 |
| rm_work | there's a good number of CRs with 6+h runtimes, where some jobs are JUST starting | 21:07 |
| rm_work | which to me indicates they were reset | 21:07 |
| jeblair | rm_work: that is likely yes | 21:07 |
| rm_work | similar to what we were seeing happen to us | 21:07 |
| rm_work | that's what i mean by "look at the length of the queue" | 21:07 |
| rm_work | but give me a moment, i'm manually searching for cases | 21:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Support dual-stack neutron networks https://review.openstack.org/357517 | 21:07 |
| * mordred cries | 21:08 | |
| openstackgerrit | John L. Villalovos proposed openstack-dev/hacking: Add documentation about off-by-default options https://review.openstack.org/359427 | 21:08 |
| *** psilvad has quit IRC | 21:08 | |
| rm_work | I think this is one: telnet 2001:4800:1ae1:18:f816:3eff:fe64:b2bf 19885 | 21:08 |
| rm_work | that's from a tempest CR | 21:08 |
| rm_work | https://review.openstack.org/355103 | 21:08 |
| jeblair | rm_work: okay, i understand what you mean now and agree -- it's just that 'length of the queue' did not convey that to me, as the length is nominally what i would expect and decreasing at a reasonable rate. but we can move on. :) | 21:09 |
| rm_work | gate-tempest-dsvm-layer4 | 21:09 |
| rm_work | right yeah, realized it was ambiguous | 21:09 |
| clarkb | mordred: and use pip -e just to maintain compat with the old develop stuff? | 21:10 |
| jeblair | rm_work, pabelanger: so there's likely something in those jobs that borks the (ipv6?) network. it might help to get a full list of the affected jobs and triangulate that way. | 21:10 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Removed directory changes in npm-dsvm-macro https://review.openstack.org/359428 | 21:10 |
| jeblair | pabelanger: your grep for exit codes and x-referencing by node id to find what job ran might help. | 21:10 |
| pabelanger | jeblair: yes, I am going to try and find if there is a pattern | 21:10 |
| mordred | clarkb: yah | 21:10 |
| rm_work | jeblair: those jobs have passed in previous and later runs | 21:10 |
| rm_work | jeblair: and some of them wouldn't even be touching the network | 21:10 |
| *** _ari_ has quit IRC | 21:10 | |
| jeblair | rm_work: well, every dsvm job touches the netwerk, right? | 21:10 |
| rm_work | hmm | 21:10 |
| clarkb | jeblair: pabelanger are they running on trusty or xenial? we may not have the ipv6 private stuff in trusty yet | 21:11 |
| jeblair | clarkb: http://paste.openstack.org/show/562557/ says both | 21:11 |
| rm_work | i guess technically, but minimally and in a way that's been pretty well tested? | 21:11 |
| rm_work | so let me see | 21:11 |
| phschwartz | jeblair: I think nibalizer tracked it down. Looks like we have reviews with a depends-on another review that is not in zuul so zuul is bombing on it | 21:11 |
| rm_work | if there are any non-dsvm | 21:11 |
| jeblair | phschwartz: that case should be covered | 21:11 |
| rm_work | yeah ok, it is quite possible it's DSVM jobs only | 21:12 |
| phschwartz | jeblair: hmm, it seems to get that error in the one I just sent every .002 seconds which causes cpu and mem usage to spike | 21:12 |
| mordred | clarkb: I mean - we'll see if it works | 21:12 |
| *** julim has quit IRC | 21:12 | |
| rm_work | so maybe you are correct that it's something with the dsvm process hosing the network, but I don't think it has anything to do with the specific code in the CRs | 21:12 |
| rm_work | jeblair: ^^ | 21:12 |
| *** michauds has joined #openstack-infra | 21:13 | |
| mordred | clarkb: this is what failure looks like: http://logs.openstack.org/17/357517/5/check/gate-dsvm-nodepool-src-shade/dbbbdfc/logs/screen-nodepool.txt.gz | 21:13 |
| mordred | clarkb: error of not having new enough os-client-config, even though the requirements.txt file has it | 21:13 |
| mordred | SO - if we don't get that error, then the nodepool change works :) | 21:13 |
| clarkb | mordred: ya I think that your change should fix that IF the plugin is evaluated after everything else installing occ | 21:14 |
| clarkb | mordred: I do not know if that is the case but +2 for now and the self testing should find out :) | 21:14 |
| mordred | ++ | 21:14 |
| *** bhunter71 has joined #openstack-infra | 21:15 | |
| jeblair | phschwartz: can you provide, say, a few hundred more lines of logs before that point? | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| *** kaisers_ has joined #openstack-infra | 21:15 | |
| *** david-lyle has quit IRC | 21:15 | |
| *** spzala has quit IRC | 21:16 | |
| *** dizquierdo has joined #openstack-infra | 21:16 | |
| *** esikachev has joined #openstack-infra | 21:17 | |
| *** spzala has joined #openstack-infra | 21:17 | |
| *** ddieterly is now known as ddieterly[away] | 21:17 | |
| *** ddieterly[away] is now known as ddieterly | 21:17 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/glean: Add logging around interface carrier detection https://review.openstack.org/359430 | 21:18 |
| *** david-lyle has joined #openstack-infra | 21:18 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Switch ironic-multinode job to wholedisk agent_ssh https://review.openstack.org/359431 | 21:19 |
| clarkb | its also worth noting that I saw similar behavoir with ansible from puppetmaster to other dfw hosts | 21:20 |
| rm_work | jeblair / fungi: it SEEMS like it's always the same cloud(s) that fails on the dsvm jobs and requeues | 21:20 |
| *** kaisers_ has quit IRC | 21:20 | |
| rm_work | it's one of the one(s?) that is ipv6 only | 21:20 |
| clarkb | pabelanger: ^ that is what prompted me to switch how the ansible launchers restart playbook worked | 21:20 |
| clarkb | and there is ipv6 in rax too | 21:20 |
| *** dizquierdo has quit IRC | 21:21 | |
| clarkb | pabelanger: and you reported it was rock solid from home which was ipv4 only at the time (I am guessing based on your talk of new HE tunnel) | 21:21 |
| *** dimtruck is now known as zz_dimtruck | 21:21 | |
| clarkb | rm_work: they are all clouds that ipv6 | 21:21 |
| clarkb | oh wait no there is a bluebox in there which is ipv4 only | 21:21 |
| rm_work | clarkb: I mean *only* ipv6 | 21:21 |
| *** spzala has quit IRC | 21:21 | |
| *** esikachev has quit IRC | 21:21 | |
| rm_work | because if it supports both, you display the ipv4 link for telnet, right? | 21:21 |
| pabelanger | clarkb: Ah, right | 21:22 |
| clarkb | rm_work: yes I understood you but there is more than osic with the error and osic is our only ipv6 only cloud | 21:22 |
| rm_work | ah i've only seen it happen with the ipv6 telnet'd jobs | 21:22 |
| clarkb | rm_work: see jeblair's paste above | 21:22 |
| rm_work | you have a better way of identifying the affected nodes? | 21:22 |
| rm_work | ah | 21:22 |
| jeblair | (it's pabelanger's paste ftr :) | 21:22 |
| pabelanger | I should have a sample of jobs from zl01 in a few minutes | 21:22 |
| rm_work | wasn't sure if the exit-code-3 thing was related | 21:23 |
| pabelanger | A lot of dsvm job | 21:23 |
| *** hashar has quit IRC | 21:23 | |
| *** zz_dimtruck is now known as dimtruck | 21:23 | |
| rm_work | cool, so you're probably close (at least you can identify the nodes with issues) | 21:23 |
| jeblair | it's identifying all network errors though -- there could be multiple underlying causes | 21:23 |
| *** matt-borland has quit IRC | 21:23 | |
| clarkb | jeblair: good point | 21:23 |
| rm_work | hmm | 21:23 |
| rm_work | i'll leave it to you guys i guess then, you seem to have it covered | 21:23 |
| rm_work | but i'll be around for a while | 21:24 |
| jeblair | rm_work: i won't be fixing this :) | 21:24 |
| rm_work | lol | 21:24 |
| *** pradk has quit IRC | 21:24 | |
| rm_work | where do you think the problem is? | 21:24 |
| jeblair | rm_work: the assumption i'm working from is that a class of devstack jobs borks the network on nodes that only have ipv6 | 21:24 |
| openstackgerrit | Merged openstack-infra/puppet-openstack_infra_spec_helper: Pin puppetlabs-spec-helper https://review.openstack.org/359421 | 21:25 |
| *** ccarmack has left #openstack-infra | 21:25 | |
| *** cody-somerville has joined #openstack-infra | 21:25 | |
| rm_work | yeah, seems right to me | 21:25 |
| clarkb | might be worth filtering out the jobs affected | 21:25 |
| clarkb | eg is it always with neutron and never with nova network etc | 21:25 |
| rm_work | that paste does indicate it's PRIMARILY osic nodes, the non-osic nodes showing up may be anomalies unrelated | 21:25 |
| rm_work | I saw one with *tempest* | 21:25 |
| jeblair | i think the next step is to collect the list of those jobs (pabelanger is doing this) so we can look for a pattern and maybe take a guess as to what's going wrong | 21:25 |
| clarkb | yup ++ | 21:26 |
| *** csomerville has joined #openstack-infra | 21:26 | |
| *** esikachev has joined #openstack-infra | 21:26 | |
| rm_work | ah, yeah | 21:26 |
| pabelanger | http://paste.openstack.org/show/562753/ | 21:26 |
| pabelanger | list of failures for today from zl01 | 21:26 |
| rm_work | all i can easily do is manually look around the zuul status page, not so helpful :P | 21:26 |
| rm_work | ah cool | 21:26 |
| *** dprince has quit IRC | 21:26 | |
| ianw | do we know about this erorr -> Bundler::GitError: The git source https://git.openstack.org/openstack-infra/puppet-openstack_infra_spec_helper is not yet checked out. Please run `bundle install` before trying to start your application | 21:26 |
| ianw | http://logs.openstack.org/17/359417/2/check/gate-openstackci-beaker-centos-7/082de10/console.html | 21:27 |
| anteaya | ianw: yes | 21:27 |
| anteaya | the fix is in the gate | 21:27 |
| ianw | anteaya: cool, thanks | 21:27 |
| anteaya | sorry merged: https://review.openstack.org/#/c/359421/ | 21:27 |
| *** eharney has quit IRC | 21:27 | |
| jeblair | phschwartz: can you ack or nak my request? | 21:27 |
| clarkb | just a quick scan of that list looks like its predominantly tests that use neutron | 21:27 |
| pabelanger | clarkb: jeblair: need to step away for family time, but will poke more at it later tonight | 21:28 |
| ianw | ahh, ok just missed it. will recheck | 21:28 |
| clarkb | neutron itself, ironic, nodepool, etc all use neutron | 21:28 |
| phschwartz | jeblair: sorry was restarting zuul. Will grab from the log | 21:28 |
| rm_work | clarkb: i was seeing that too but i think it's just an issue of percentage of all tests that use neutron skews things :P | 21:28 |
| jeblair | phschwartz: thx | 21:28 |
| rm_work | ah, yeah i mean almost anything in openstack uses neutron in some capacity <_< | 21:28 |
| clarkb | rm_work: I think we still have a higher percentage that use nova net fwiw | 21:29 |
| rm_work | really? | 21:29 |
| clarkb | rm_work: so I don't know that that is the case | 21:29 |
| rm_work | that's surprising | 21:29 |
| clarkb | rm_work: yes because it is/was the default | 21:29 |
| clarkb | and is/was more reliable | 21:29 |
| rm_work | aren't they actually finally *deleting that code* in the next cycle or something? lol | 21:29 |
| *** cody-somerville has quit IRC | 21:29 | |
| rm_work | guess maybe not >_> | 21:29 |
| *** Swami has quit IRC | 21:29 | |
| clarkb | rm_work: they only just got things switched in the last couple weeks | 21:30 |
| anteaya | ianw: sounds good | 21:30 |
| clarkb | this very problem could be related :P I am trying to see if any of those in pabelanger's list are nova net jobs to rule it out | 21:30 |
| zigo | pabelanger: https://review.openstack.org/#/c/358819/ <--- Could you +2 adding deb-python-fixtures again please? | 21:30 |
| *** hockeynut has quit IRC | 21:31 | |
| rm_work | again, i should just let you guys handle this probably :P i'm just distracting | 21:31 |
| clarkb | oh we have kolla jobs in ther ewhich don't devstack | 21:31 |
| jeblair | mordred: http://logs.openstack.org/25/359425/1/check/gate-dsvm-nodepool/4896a4b/logs/devstacklog.txt.gz#_2016-08-23_21_29_04_118 needs a sudo? | 21:31 |
| mordred | jeblair: yah. thanks | 21:32 |
| clarkb | so probably not related to that unless they do similar things | 21:32 |
| *** salv-orlando has joined #openstack-infra | 21:32 | |
| michauds | Is this the proper channel to report an issue with gerrit? | 21:32 |
| clarkb | michauds: yes | 21:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Directly use pip instead of setup_develop in plugin https://review.openstack.org/359425 | 21:32 |
| phschwartz | jeblair: http://paste.openstack.org/show/562758/ | 21:33 |
| clarkb | jeblair: pabelanger and this error means the ssh poll for async ansible failed to connect right? | 21:33 |
| jeblair | clarkb: yes it's "AnsibleHostUnreachable" | 21:34 |
| michauds | clarkb: I can't seem to POST data to https://review.openstack.org/gerrit_ui/rpc/AccountSecurity to update my offline contact information. | 21:34 |
| *** gouthamr has joined #openstack-infra | 21:34 | |
| *** piet has joined #openstack-infra | 21:35 | |
| clarkb | michauds: make sure that you have set up your accounts properly. Did you follow http://docs.openstack.org/infra/manual/developers.html#account-setup to set that up? | 21:35 |
| *** jheroux has quit IRC | 21:35 | |
| rm_work | clarkb: the kolla jobs I see are "dsvm" jobs though? | 21:36 |
| jeblair | phschwartz: it looks like there may be lines missing between those 2 pastes? | 21:36 |
| clarkb | rm_work: I don't think they run devstack, they just use dsvm for supporting setup | 21:36 |
| jeblair | phschwartz: maybe there's a limit for how many lines in a paste? | 21:36 |
| rm_work | ah | 21:36 |
| jeblair | phschwartz: or overall size i guess | 21:37 |
| phschwartz | jeblair: not sure I did 1k lines starting from the last line of the first AttributeError | 21:37 |
| clarkb | spot checking a rax xenial host we have the correct ipv6 tempaddr value of 0 | 21:37 |
| michauds | clarkb: yes, I have a Launchpad account and am also part of Openstack Foundation :) | 21:38 |
| michauds | clarkb: Oh I need to upgrade to Foundation Member | 21:40 |
| michauds | heh | 21:40 |
| *** vhosakot has quit IRC | 21:40 | |
| *** vhosakot has joined #openstack-infra | 21:41 | |
| dougwig | rm_work, jeblair, fungi: back. | 21:41 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Switch ironic-multinode job to wholedisk agent_ssh https://review.openstack.org/359431 | 21:42 |
| *** Goneri has quit IRC | 21:43 | |
| *** thorst has quit IRC | 21:44 | |
| *** sdake has quit IRC | 21:45 | |
| *** asettle has quit IRC | 21:45 | |
| *** vhosakot has quit IRC | 21:45 | |
| *** sdake has joined #openstack-infra | 21:45 | |
| *** sdake has quit IRC | 21:45 | |
| *** sdake has joined #openstack-infra | 21:46 | |
| *** thorst has joined #openstack-infra | 21:47 | |
| *** thiagop has quit IRC | 21:47 | |
| *** eranrom has quit IRC | 21:47 | |
| *** eranrom has joined #openstack-infra | 21:48 | |
| *** larainema has quit IRC | 21:49 | |
| clarkb | ok sitting with greghaynes and we seem to lose the most time doing the mv of the image from chroot to the bind mounted image dest | 21:49 |
| *** piet has quit IRC | 21:49 | |
| clarkb | he thinks what is happening is the apt cache never gets cleaned up so we are copying more and more and more files | 21:50 |
| clarkb | also possibly we need to pack/gc our git repo cache | 21:50 |
| clarkb | so the dib caching need curating | 21:50 |
| *** dfflanders has joined #openstack-infra | 21:50 | |
| *** vhosakot has joined #openstack-infra | 21:51 | |
| clarkb | with that understood back to ansible network fails | 21:51 |
| zigo | pabelanger: I'm having a very hard time to build webkit2gtk, which is needed to build sphinx. It seems it takes a huge amount of time to build. I wonder if we could just "cheat" here, and just download it from official jessie-backports. There shouldn't be many packages like that, just this one, hopefully. Your thoughts? | 21:51 |
| *** thorst has quit IRC | 21:51 | |
| *** esikachev has quit IRC | 21:53 | |
| *** spzala has joined #openstack-infra | 21:53 | |
| *** csomerville has quit IRC | 21:54 | |
| *** ddieterly is now known as ddieterly[away] | 21:57 | |
| *** mariojv has joined #openstack-infra | 21:59 | |
| *** eggshell has quit IRC | 22:00 | |
| *** piet has joined #openstack-infra | 22:01 | |
| *** tonytan4ever has joined #openstack-infra | 22:02 | |
| *** amotoki has joined #openstack-infra | 22:02 | |
| *** xarses has quit IRC | 22:03 | |
| *** mariojv has left #openstack-infra | 22:04 | |
| *** priteau has quit IRC | 22:05 | |
| *** Gorian|work has joined #openstack-infra | 22:06 | |
| *** tonytan4ever has quit IRC | 22:07 | |
| *** amotoki has quit IRC | 22:07 | |
| *** vhosakot has quit IRC | 22:09 | |
| *** ddieterly[away] is now known as ddieterly | 22:09 | |
| *** notmorgan is now known as morganfainberg | 22:09 | |
| *** morganfainberg is now known as morgan | 22:09 | |
| *** morgan is now known as notmorgan | 22:11 | |
| *** nwkarsten has joined #openstack-infra | 22:12 | |
| *** nwkarsten has quit IRC | 22:12 | |
| *** nwkarsten has joined #openstack-infra | 22:12 | |
| dougwig | project-config and devstack-gate cores, lbaas v1 delete is just about ready to merge, but will break these two things (nodepool default service list, and devstack-gate default features): https://review.openstack.org/#/c/358257/ https://review.openstack.org/#/c/358258/ | 22:13 |
| dougwig | both still reference q-lbaas, even though all CI jobs now use the neutron-lbaas devstack plugin | 22:13 |
| jeblair | phschwartz: your paste has 479 lines, which is why i think you hit the paste limit. can you re-paste as multiple chunks | 22:14 |
| *** piet has quit IRC | 22:14 | |
| *** baoli has quit IRC | 22:14 | |
| clarkb | dougwig: you will need to update devstack-gate first | 22:14 |
| clarkb | dougwig: otherwise you will break everyone | 22:14 |
| dougwig | clarkb: aye, that's what i'm trying to do. https://review.openstack.org/#/c/358258/ | 22:15 |
| *** andymaier_ has quit IRC | 22:15 | |
| *** nwkarste_ has quit IRC | 22:15 | |
| clarkb | pabelanger: rm_work: in this setup ansible sshs over and over again to poll the build status. It does seem odd that it would fail to ssh after it has succeeded many times unless something has changed on the test node | 22:16 |
| *** nwkarsten has quit IRC | 22:17 | |
| rm_work | yeah, the whole thing seems odd to me | 22:18 |
| clarkb | jeblair: pabelanger and we are not using paramiko correct? zuul launchers are creating teh ssh subprocess? | 22:20 |
| rm_work | I have to go for a bit... | 22:20 |
| rm_work | i'll be back around later though :/ | 22:21 |
| jeblair | clarkb: zuul -> ansible -> openssh | 22:21 |
| jeblair | clarkb: also, worth noting that ansible uses controlmaster, so there is a persistent connection | 22:21 |
| jeblair | (which apparently dies) | 22:21 |
| clarkb | oh its not making successive connections to poll? /me looks up controlmaster | 22:21 |
| mordred | maybe we need to add a thing to .ssh/config to send keepalives? | 22:22 |
| *** AnarchyAo has joined #openstack-infra | 22:22 | |
| *** fguillot has joined #openstack-infra | 22:23 | |
| clarkb | what keeps the master running? looks like it will only reuse if it already exists? | 22:23 |
| jeblair | mordred: no -- there is activity every few seconds thanks to the ansible polling | 22:23 |
| jeblair | clarkb: ansible starts it for each host it connects to | 22:23 |
| clarkb | ah and the 60s ControlPersist says wait 60s before you die if you were the master | 22:24 |
| jeblair | clarkb: if you want to see them, the processes on the launcher look like: zuul 31088 0.2 0.0 44588 1608 ? Ss 22:23 0:00 ssh: /home/zuul/.ansible/cp/ansible-ssh-2001:4800:1ae1:18:f816:3eff:feab:c8ca-22-jenkins [mux] | 22:24 |
| openstackgerrit | Merged openstack-infra/project-config: Remove q-lbaas from the nodepool pre-configured list https://review.openstack.org/358257 | 22:24 |
| mgagne_ | clarkb: how long does it take for a new CI mirror to be built from scratch? | 22:26 |
| clarkb | that sort of error implies that the master was not there for some reason though because a new connection failed to connect. /me begins to understand the setup | 22:26 |
| *** sdague has joined #openstack-infra | 22:26 | |
| clarkb | mgagne_: its fast since its mostly just an http server in front of an afs cache | 22:26 |
| jeblair | mgagne_: maybe 30 mins? | 22:26 |
| mgagne_ | clarkb: right so afs just needs to heat up its cache to be effective? | 22:27 |
| jeblair | mgagne_: my union has a 4 hour minimum though ;) | 22:27 |
| jhesketh | Morning | 22:27 |
| clarkb | mgagne_: yup, | 22:27 |
| jeblair | mgagne_: yeah, after a couple of slow-ish jobs, it'll be warm. someone suggested pip installing the upper-constraints file to make that happen out of band; dunno if that's been tried | 22:28 |
| clarkb | mgagne_: which can be done if you pip download only the global reqs and equivelent type things for ubuntu/debian/centos mirrors | 22:28 |
| *** dfflanders has quit IRC | 22:28 | |
| clarkb | jeblair: ya not sure if anyone has gone through the trouble since it hasn't seemed to be necessary | 22:28 |
| jeblair | ++ | 22:28 |
| *** xarses has joined #openstack-infra | 22:29 | |
| mgagne_ | :D | 22:29 |
| mgagne_ | good to know | 22:29 |
| *** esberglu has quit IRC | 22:29 | |
| mgagne_ | clarkb, jeblair: we would be ready to offer new resources for CI infra which would be hosted in a different region than the current one | 22:30 |
| *** dimtruck is now known as zz_dimtruck | 22:30 | |
| *** sdague has quit IRC | 22:31 | |
| jeblair | mgagne_: cool -- do you want us to decrease/stop nyj01 or use both? | 22:31 |
| mgagne_ | jeblair: I suggest we use both for now until someone asks us to stop using nyj01 for "reasons" | 22:31 |
| *** ddieterly is now known as ddieterly[away] | 22:31 | |
| jeblair | mgagne_: sure thing -- is the new one ready now? | 22:32 |
| mgagne_ | jeblair: yes, ready like 5m ago | 22:32 |
| mgagne_ | jeblair: mtl01 is the region name | 22:32 |
| mgagne_ | jeblair: for "management" account, there is a public /28 available | 22:33 |
| clarkb | jeblair: is it possible we are hitting the 108 byte socket path limit? my quick check for a max length ipv6 addr representation is 85 bytes though | 22:33 |
| clarkb | based on the control path in use | 22:33 |
| jeblair | clarkb: it's dying in the middle of the job i believe | 22:33 |
| mgagne_ | jeblair: for nodepool, quota will be over 120 instances at least, maybe more | 22:34 |
| *** michauds has quit IRC | 22:36 | |
| *** thorst_ has joined #openstack-infra | 22:36 | |
| *** xyang1 has quit IRC | 22:36 | |
| jeblair | mgagne_: cool, thanks! i'm about to eod -- maybe another infra-root can start the work to add it | 22:36 |
| *** ddieterly[away] has quit IRC | 22:36 | |
| mgagne_ | jeblair: I will be off to the ops meetup too since might not be responsive this week. This is just a heads up about what's coming for our contribution. | 22:37 |
| *** thorst_ has quit IRC | 22:39 | |
| *** ccamacho has quit IRC | 22:42 | |
| anteaya | morning jhesketh | 22:43 |
| *** sdake has quit IRC | 22:44 | |
| *** sdake has joined #openstack-infra | 22:44 | |
| clarkb | jeblair: looking at logs on zl01 it is dying after copying the main sh file it looks like I assume the next zuul_runner there is attempting to run the script | 22:44 |
| clarkb | so yes I agree its dying in the middle of the job | 22:44 |
| *** yamahata has quit IRC | 22:47 | |
| *** spzala has quit IRC | 22:48 | |
| *** esikachev has joined #openstack-infra | 22:49 | |
| *** yamamoto has joined #openstack-infra | 22:51 | |
| *** Hal has quit IRC | 22:52 | |
| openstackgerrit | Chris Krelle proposed openstack-infra/glean: Adjust wait time for interfaces to become available https://review.openstack.org/359471 | 22:52 |
| *** esikachev has quit IRC | 22:54 | |
| *** rbrndt has quit IRC | 22:55 | |
| clarkb | pabelanger: jeblair I have held 3803271 which is an osic job running a neutron tempest test. Not sure how frequent these "fails" are but hopeflly will catch one if we open the nets | 22:55 |
| jeblair | clarkb: you going to go ahead and open an ssh connection? | 22:56 |
| *** fguillot is now known as fguillot_afk | 22:57 | |
| clarkb | jeblair: ya | 22:58 |
| *** fguillot_afk is now known as fguillot | 22:58 | |
| clarkb | also tailing zl01's log to see if I can hold one fast enough if I see it | 22:58 |
| jeblair | clarkb: probably not :( but it might be worth setting an auto-hold for gate-tempest-dsvm-neutron-full-ubuntu-xenial ? | 22:59 |
| jeblair | clarkb: (okay, i guess if you're faster than the publisher playbook, you might be able to make it :) | 23:00 |
| openstackgerrit | Merged openstack-infra/nodepool: Include ip address for ssh_connect exception https://review.openstack.org/359369 | 23:00 |
| clarkb | ya we'll just hae to see might not be possible | 23:00 |
| *** spzala has joined #openstack-infra | 23:00 | |
| clarkb | just held 3802386 | 23:01 |
| clarkb | it failed to ssh | 23:01 |
| clarkb | and 3802471 | 23:01 |
| clarkb | and 3802440 lets see if nodepool doesnt' delete any of those | 23:02 |
| clarkb | now one thing I am noticing is they all come in clumps | 23:02 |
| clarkb | which maybe implies its not a specific job side thing breaking us | 23:02 |
| *** thorst_ has joined #openstack-infra | 23:03 | |
| *** chlong has quit IRC | 23:03 | |
| *** tonytan4ever has joined #openstack-infra | 23:03 | |
| *** larainema has joined #openstack-infra | 23:04 | |
| clarkb | oh cool one of the fails was on the host I had held earlier. Unfortunately my ssh connection to it is gone gone gone | 23:04 |
| clarkb | cloudnull: you don't happen to be around do you? | 23:04 |
| * clarkb checks console logs | 23:04 | |
| *** spzala has quit IRC | 23:05 | |
| clarkb | console log shows nothing | 23:05 |
| *** thorst_ has quit IRC | 23:05 | |
| *** thorst_ has joined #openstack-infra | 23:06 | |
| clarkb | and we aren't copying job logs because ssh doesn't work right? | 23:07 |
| clarkb | this is mysterious and fun | 23:07 |
| clarkb | "fun" | 23:07 |
| * clarkb attempts to get in via the mirror on the 10 net | 23:08 | |
| *** tonytan4ever has quit IRC | 23:08 | |
| *** vhosakot has joined #openstack-infra | 23:08 | |
| *** nwkarsten has joined #openstack-infra | 23:08 | |
| ianw | jeblair: in http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/ansiblelaunchserver.py#n113 ; the ansible.cfg in JobDir is found because because that's the pwd for the ansible-playbook call? ie. no ANSIBLE_CONFIG setting | 23:09 |
| *** yamahata has joined #openstack-infra | 23:09 | |
| clarkb | I'm in \o/ | 23:09 |
| clarkb | so that does work, the ipv4 stack is stilltehre | 23:10 |
| ianw | jeblair: ahh, doh, yeah, cwd= in the call, now i see it | 23:10 |
| *** ddieterly has joined #openstack-infra | 23:10 | |
| *** thorst_ has quit IRC | 23:10 | |
| *** ddieterly is now known as ddieterly[away] | 23:11 | |
| *** rhallisey has quit IRC | 23:11 | |
| clarkb | hrm doesn't look like 3803271 has been detected as a fail by ansible yet | 23:12 |
| clarkb | let me just double check that the one I caught on zl03 exhibits the same behavior | 23:12 |
| *** Hal has joined #openstack-infra | 23:12 | |
| *** ddieterly[away] is now known as ddieterly | 23:13 | |
| *** nwkarsten has quit IRC | 23:13 | |
| clarkb | I has more datas I can hit the ipv6 addr from the mirror host | 23:14 |
| clarkb | but not from rackspace | 23:14 |
| clarkb | (I am tunneling through my irc box for ipv6) | 23:15 |
| mordred | clarkb: oh - that's weird | 23:18 |
| clarkb | pabelanger: jeblair cloudnull ubuntu-trusty-osic-cloud1-3802440 with uuid d44117b0-4835-4910-990a-fa1157d54dc8 and IPs 2001:4800:1ae1:18:f816:3eff:fe45:ab1b, 10.0.23.242 is the one I have "captured" | 23:18 |
| clarkb | mordred: ^ you too | 23:18 |
| mordred | \o/ | 23:18 |
| clarkb | ssh from the mirror in osic cloud1 can hit both those IPs | 23:19 |
| clarkb | I cannot hit the IPv6 from rackspace and the ipv4 is local only | 23:19 |
| clarkb | this makes me think its maybe not a problem with the instance itself | 23:19 |
| clarkb | and instead may be something in the cloud? | 23:19 |
| mordred | clarkb: the 'public' ipv4? | 23:19 |
| mordred | clarkb: or the 'private' ipv4 | 23:20 |
| clarkb | mordred: the test instances only have the private ipv4 | 23:20 |
| mordred | ah. nod | 23:20 |
| clarkb | so I did ssh from home to mirror host via public v4, then ssh to test instance ipv6 and private ipv4 and both work | 23:21 |
| pabelanger | just catching up on backscroll | 23:21 |
| clarkb | I double checked that sshing from my rackspace irc screen box to the test instance ipv6 fails | 23:21 |
| mordred | clarkb: I can verify that I can get in from the mirror host | 23:21 |
| mordred | not that I was doubting you | 23:21 |
| clarkb | mordred: can you double check you can't hit it from your "local" ipv6? | 23:22 |
| mordred | clarkb: I do not have "local" ipv6 | 23:22 |
| clarkb | its possible this is a rax to osic ipv6 issue | 23:22 |
| pabelanger | mgagne_: jeblair: are we using the same credentials for mtl01? | 23:22 |
| clarkb | since my irc box and the zuul launcher are all in rax | 23:22 |
| clarkb | pabelanger: maybe you can test ^ | 23:22 |
| mordred | oh - my irc box is in vexxhost ... | 23:22 |
| mordred | one sec | 23:22 |
| clarkb | mordred: kk thanks | 23:22 |
| *** Julien-zte has joined #openstack-infra | 23:22 | |
| clarkb | if it is broken for vexxhost too then I think its likely in osic proper | 23:23 |
| pabelanger | clarkb: not yet, still don't have broker setup properly | 23:23 |
| mordred | clarkb: I cannot get there from vexxhost eithre | 23:23 |
| mordred | cloudnull: ^^ | 23:23 |
| pleia2 | I have native v6 here, can't get to it | 23:23 |
| clarkb | I think the next thing would be for cloudnull to examine networking in osic for us | 23:24 |
| nibalizer | jesusaur: so looks like this failed around rubocop ? https://review.openstack.org/#/c/350835/ | 23:24 |
| mordred | clarkb: so - I was afk for a sec ... other than "thjis ipv6 doesn't work" ... what's the problem we're trying to sort ... is this the "ssh to nodes breaks" problem? | 23:25 |
| *** Gorian|work has quit IRC | 23:25 | |
| mordred | clarkb: so.... there are two ipv6 addresses on eth0 | 23:25 |
| mordred | inet6 2001:4800:1ae1:18:f816:3eff:fe45:ab1b/64 scope global dynamic | 23:25 |
| mordred | valid_lft 2589662sec preferred_lft 602462sec | 23:25 |
| mordred | inet6 fe80::f816:3eff:fe45:ab1b/64 scope link | 23:25 |
| mordred | valid_lft forever preferred_lft forever | 23:25 |
| mordred | is that 'normal' ? | 23:25 |
| mordred | yah. vexxhost does that. nm | 23:26 |
| clarkb | ya the scope link is your I forget the name for it address | 23:26 |
| pleia2 | fe80 is local | 23:26 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remve old cache files on the mirror server https://review.openstack.org/348806 | 23:26 |
| *** hongbin has quit IRC | 23:26 | |
| clarkb | and the other one is the global addresseable one | 23:26 |
| * pleia2 nods | 23:26 | |
| clarkb | and your routes are smart enough to use the proper src addr when doing things | 23:26 |
| mordred | yah. soo ... route -6n looks a little odd to me - but I'm still not fully used to looking at v6 route tables | 23:27 |
| mordred | I guess my question is "is something about the v6 address on br-ex messing up v6 routing" | 23:27 |
| clarkb | ::/0 :: !n -1 1 9265 lo | 23:27 |
| clarkb | that actually might be what is braeking us? | 23:27 |
| mordred | yah | 23:27 |
| jesusaur | nibalizer: oh rubocop-rspec... that's not the same but likely very related | 23:28 |
| mordred | that was my next question | 23:28 |
| clarkb | I think that says default route is out lo | 23:28 |
| clarkb | 2001:4800:1ae1:18::/64 :: UAe 256 1 0 eth0 is what makes it work in the cloud from the mirror host | 23:28 |
| mordred | clarkb: because we can ssh to these when nodepool creates them | 23:28 |
| pabelanger | right | 23:28 |
| mordred | it's just after stuff runs on them that something goes south | 23:28 |
| mordred | nod | 23:28 |
| anteaya | jesusaur: rubocop-rspec uses rubocop | 23:28 |
| pabelanger | clarkb: what if you remove it? | 23:28 |
| clarkb | pabelanger: I think we have to update it to say eth0 | 23:28 |
| clarkb | I want to look at a host that works really quick | 23:29 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/yaml2ical: Update hacking test-requirement https://review.openstack.org/359478 | 23:29 |
| clarkb | ::/0 fe80::def UGDAe 1024 4 42 ens3 | 23:29 |
| *** yuanying has joined #openstack-infra | 23:30 | |
| clarkb | we have ^ on a xenial host where things are working | 23:30 |
| clarkb | let me compare trusty to trusty | 23:30 |
| *** fguillot has quit IRC | 23:30 | |
| clarkb | ::/0 fe80::def UGDAe 1024 2 0 eth0 | 23:30 |
| clarkb | thats trusty | 23:30 |
| mordred | GAH STAB STAB | 23:31 |
| mordred | the devstack plugin stuff totally didn't work | 23:31 |
| mordred | because it installs python-openstackclient after running the plugin | 23:31 |
| clarkb | mordred: I was worried about that | 23:31 |
| jesusaur | nibalizer: so I guess that change needs to pin rubocop-rspec too? | 23:32 |
| clarkb | we are using RAs here? | 23:33 |
| pabelanger | clarkb: ya, looks like you are on the right track | 23:33 |
| *** spzala has joined #openstack-infra | 23:34 | |
| mordred | clarkb: https://review.openstack.org/359479 | 23:34 |
| clarkb | Aug 23 23:28:56 ubuntu dhclient: Error printing text. is a fun one in syslog | 23:34 |
| anteaya | jesusaur: https://review.openstack.org/#/c/359421/ | 23:35 |
| clarkb | pabelanger: mordred pleia2 so I think either the cloud is sending us a new bad RA or maybe neutron is? | 23:35 |
| jesusaur | nibalizer: looks like rubocop 1.5.0 is the last version to support ruby 1.9 | 23:36 |
| *** salv-orlando has quit IRC | 23:36 | |
| clarkb | like we aren't isolating neutron's RAs for client networks from our actual interfaces maybe | 23:36 |
| jesusaur | s/rubocop/rubocop-rspec/ | 23:36 |
| clarkb | but I am still sort of swimming through logs trying to figure out when that route is updated | 23:36 |
| jesusaur | anteaya: nibalizer: oh i was confused about what change i was looking at | 23:37 |
| anteaya | jesusaur: so the fix merged an hour after your change failed | 23:37 |
| anteaya | so maybe recheck? | 23:37 |
| *** fguillot has joined #openstack-infra | 23:37 | |
| mordred | clarkb: if that doesn't work, I'm going to see about making the nodepool plugin install into a virtualenv | 23:37 |
| mordred | in fact, I may just do that | 23:38 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/yaml2ical: Manual sync to global-requirements https://review.openstack.org/359480 | 23:38 |
| jesusaur | anteaya: yeah | 23:38 |
| clarkb | people that know ipv6 better than I do, does linux log RA updates anywhere? | 23:38 |
| *** spzala has quit IRC | 23:39 | |
| *** hexlibris has joined #openstack-infra | 23:40 | |
| clarkb | like dhcp keeps everything in /var/lib/dhcp | 23:40 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Use cached images https://review.openstack.org/359481 | 23:40 |
| pleia2 | hm, I've only ever seen updates in syslog, but I'm not sure if that was some kind of debug level | 23:41 |
| clarkb | neutron does have radvd processes running but reading their configs they are scoped to a neutron addr which should be "detached" from eth0 | 23:43 |
| *** paulobanon has quit IRC | 23:43 | |
| clarkb | supposedly net.ipv6.conf.eth0.accept_ra = 1 means accept RAs if forwarding is disabled and we also have net.ipv6.conf.eth0.forwarding = 1 | 23:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Install nodepool and shade into a virtualenv https://review.openstack.org/359425 | 23:45 |
| mordred | clarkb: are we only seeing the detach failures on neutron devstacks? | 23:46 |
| clarkb | on a host that works we have net.ipv6.conf.eth0.forwarding = 0 | 23:46 |
| openstackgerrit | Dmitry Ilyin proposed openstack-infra/project-config: Enable voting checks for the Fuel unit tests Puppet 4.5 https://review.openstack.org/357335 | 23:46 |
| clarkb | mordred: I wasn't able to confirm that but its heavily neutron from the list | 23:46 |
| jeblair | mordred: normally i'm all ick on venvs, but this seems like a good pattern -- we're not interested in being part of an openstack deployment, we just happen to need one. | 23:47 |
| clarkb | mordred: anyways I think maybe something is flipping that sysctl and that may be breaking us | 23:47 |
| mordred | jeblair: yah. that was my thinking | 23:47 |
| mordred | clarkb: ++ | 23:47 |
| *** Julien-zte has quit IRC | 23:47 | |
| mordred | lib/neutron_plugins/services/l3: sudo sysctl -w net.ipv6.conf.all.forwarding=1 | 23:47 |
| clarkb | mordred: yay? | 23:47 |
| pabelanger | there we go | 23:47 |
| * clarkb goes to grab a beer now | 23:47 | |
| mordred | I think that would be it | 23:47 |
| clarkb | :) | 23:47 |
| mordred | I mean, it should stop doing that :) | 23:47 |
| jeblair | http://codesearch.openstack.org/?q=net.ipv6.conf&i=nope&files=&repos= | 23:47 |
| jeblair | mordred: you beat me | 23:47 |
| mordred | jeblair: git grep ftw | 23:48 |
| clarkb | mordred: I agree | 23:48 |
| mordred | jeblair: (I may have alreayd been hacking in devstack in a window) | 23:48 |
| mordred | clarkb: now ... how do we communicate "hey devstack, this is an interface that you should not enable forwarding on" | 23:48 |
| mordred | sc68cal: ^^ any chance you're around? | 23:49 |
| clarkb | mordred: another option is we can set net.ipv6.conf.eth0.accept_ra = 1 to 2 which means always accept ras | 23:49 |
| clarkb | regardless of forwarding state | 23:49 |
| clarkb | (which is apparently not in line with rfcs but they are only meant to be read not followed right?) | 23:49 |
| * sc68cal connects and reads scrollback | 23:49 | |
| mordred | sc68cal: tl;dr - devstack may be hosing the ipv6 connections on our ipv6 only cloud | 23:50 |
| anteaya | sc68cal: it started this morning | 23:50 |
| clarkb | I think that is the wrong solution since this means that devstack would hose other peoples machiens in a similar situation | 23:50 |
| mordred | yah | 23:50 |
| pabelanger | clarkb: agree | 23:50 |
| clarkb | so should fix in devstack with its forwading flippage | 23:50 |
| *** esikachev has joined #openstack-infra | 23:50 | |
| anteaya | sc68cal: or better, we started looking at it this morning | 23:50 |
| jeblair | clarkb: i have a slight concern about the number of hits in that codesearch query | 23:51 |
| * clarkb pulls it uop | 23:51 | |
| clarkb | oh wow | 23:51 |
| clarkb | we might have to do the other thing too :/ | 23:51 |
| mordred | jeblair: most of the hits seem to be in charms | 23:51 |
| jeblair | mordred: let me rephrase -- i have a concern about the breadth of projects that have similar looking settings :) | 23:52 |
| mordred | ++ | 23:52 |
| jeblair | like -- it might be some voodoo that has been copied around prolifically | 23:52 |
| *** _sarob has quit IRC | 23:53 | |
| sc68cal | ok, I think I follow what's going on. | 23:54 |
| mordred | \o/ | 23:54 |
| *** sarob has joined #openstack-infra | 23:54 | |
| pabelanger | jeblair: are you able to catch me up on the (new?) mtl01 region in internap? Is that a new region we are launching or just moving an AFS mirror to it? | 23:55 |
| *** rlandy has quit IRC | 23:55 | |
| *** esikachev has quit IRC | 23:55 | |
| sc68cal | is this for rfc6204w3 ? | 23:56 |
| clarkb | sc68cal: this is for run neutron in a cloud that is ipv6 only | 23:56 |
| openstackgerrit | Merged openstack-infra/puppet-openstack_infra_spec_helper: Pin json_pure gem for ruby1.9 support https://review.openstack.org/350835 | 23:56 |
| jeblair | pabelanger: new region, we'll use both. i don't know the answer about creds -- probably worth just trying :) | 23:56 |
| *** spzala has joined #openstack-infra | 23:56 | |
| clarkb | sc68cal: so the instances we get from taht cloud get RAs, then neutron shows up and says no don't do that and now we can't route | 23:56 |
| sc68cal | clarkb: right, but you have a interface that is recieving RAs from an upstream source, and also advertises to links connected to the node | 23:57 |
| sc68cal | basically https://tools.ietf.org/html/rfc6204 | 23:57 |
| pabelanger | jeblair: sure, I'll poke around a bit tonight. | 23:57 |
| mordred | sc68cal: http://paste.openstack.org/show/562777/ is the interfaces on a borked host and also the routes | 23:57 |
| clarkb | sc68cal: we have an interface that receives RAs because neutron is the underlying cloud. Then neutron in that instance does whatever the heck neutron does | 23:57 |
| sc68cal | right | 23:58 |
| sc68cal | it's the same issue that customer premise equipment has in ISPs | 23:58 |
| clarkb | well the issue is neutron should not touch eth0 in the gate | 23:58 |
| clarkb | ever please don't do it | 23:58 |
| *** zhurong has joined #openstack-infra | 23:58 | |
| sc68cal | customer router recieves RAs from ISP equipment but also needs to distribute RA's to boxes connected to it | 23:58 |
| clarkb | (replace eth0 with whatever the actual interface is) | 23:58 |
| *** dingyichen has joined #openstack-infra | 23:58 | |
| *** AnarchyAo has quit IRC | 23:59 | |
| clarkb | so I think the issue here is the assumption that it can edit the all settings in sysctl rather than the subset it needs | 23:59 |
| mordred | current theory is that we think that the neutron running on the host is also managing to distribute RAs to the host itself, yeah? | 23:59 |
| openstackgerrit | John L. Villalovos proposed openstack-dev/hacking: Fix issues detected by pycodestyle https://review.openstack.org/359487 | 23:59 |
| clarkb | mordred: maybe? I checked the radvd config and its scoped to an interface | 23:59 |
| clarkb | and that interface shouldn't have a path to eth0 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!