| *** gmann has quit IRC | 00:05 | |
| *** jamesmcarthur has joined #openstack-infra | 00:05 | |
| *** gmann has joined #openstack-infra | 00:06 | |
| *** hongbin has quit IRC | 00:07 | |
| *** markvoelker has quit IRC | 00:10 | |
| *** markvoelker has joined #openstack-infra | 00:10 | |
| *** lbragstad has quit IRC | 00:12 | |
| *** sflanigan has joined #openstack-infra | 00:13 | |
| *** markvoelker has quit IRC | 00:15 | |
| *** bobh has joined #openstack-infra | 00:22 | |
| *** bobh has quit IRC | 00:31 | |
| *** ijw has joined #openstack-infra | 00:33 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Sync after writing partition table https://review.openstack.org/475203 | 00:37 |
|---|---|---|
| *** MatthewLi has joined #openstack-infra | 00:37 | |
| *** thorst has joined #openstack-infra | 00:39 | |
| *** ijw_ has joined #openstack-infra | 00:49 | |
| *** ijw has quit IRC | 00:49 | |
| ianw | i'll try and look this afternoon, but dib seems broken in two ways now ... ^ and dealing with the debian stable release | 00:49 |
| *** bobh has joined #openstack-infra | 00:50 | |
| fungi | ianw: for debian stretch (released yesterday)? | 00:51 |
| fungi | or maybe that was today | 00:51 |
| ianw | yeah, well we just use "stable" which has now become ... unstable :) | 00:51 |
| *** thorst has quit IRC | 00:51 | |
| fungi | heh. momentarily unstable, until we adjust ;) | 00:52 |
| *** ijw_ has quit IRC | 00:52 | |
| *** thorst has joined #openstack-infra | 00:52 | |
| fungi | but yes, we should be installing by release codename since that's how we differentiate our image labels | 00:53 |
| fungi | building debian-jessie images that are actually stretch would be extra weird | 00:53 |
| fungi | i assume that's the case already for our ci images at least | 00:54 |
| *** amotoki_away is now known as amotoki | 00:54 | |
| ianw | i think so, too scared to look atm :) | 00:55 |
| ianw | one thing at a time :) | 00:55 |
| *** thorst has quit IRC | 00:56 | |
| *** stakeda has joined #openstack-infra | 00:57 | |
| fungi | right, don't go looking for trouble! | 01:00 |
| *** LindaWang has joined #openstack-infra | 01:01 | |
| *** namnh has joined #openstack-infra | 01:01 | |
| *** alex_xu has quit IRC | 01:01 | |
| *** alex_xu has joined #openstack-infra | 01:01 | |
| *** bobh has quit IRC | 01:04 | |
| *** cuongnv has joined #openstack-infra | 01:07 | |
| ianw | you'll never guess what package starting with "s" is causing problems | 01:07 |
| *** jamesmcarthur has quit IRC | 01:10 | |
| *** cshastri has joined #openstack-infra | 01:10 | |
| *** liujiong has joined #openstack-infra | 01:10 | |
| fungi | systemg? | 01:22 |
| *** stakeda has quit IRC | 01:30 | |
| *** ianychoi has joined #openstack-infra | 01:33 | |
| *** zz_dimtruck is now known as dimtruck | 01:36 | |
| *** bobh has joined #openstack-infra | 01:37 | |
| *** stakeda has joined #openstack-infra | 01:41 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Install systemd earlier for strech https://review.openstack.org/475206 | 01:41 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Install systemd earlier for Stretch https://review.openstack.org/475206 | 01:42 |
| ianw | hrm, do suse nodes have an issue to? | 01:46 |
| ianw | gate-dsvm-nodepool-ubuntu-src: and gate-dsvm-nodepool-opensuse-src: have been queued for over an hour | 01:46 |
| *** hichihara has joined #openstack-infra | 01:47 | |
| ianw | % (action, command, out, err)) | 01:50 |
| ianw | Exception: Unable to run ready script | 01:50 |
| ianw | command: cd /opt/nodepool-scripts && ./configure_mirror.sh ubuntu-xenial-osic-cloud1-s3500-9376402 | 01:50 |
| ianw | + host -W 30 git.openstack.org | 01:50 |
| ianw | ;; connection timed out; no servers could be reached | 01:50 |
| clarkb | my unbound change probavly derped | 01:56 |
| clarkb | :( | 01:56 |
| ianw | clarkb: yeah, it gives a synatx error | 01:57 |
| ianw | http://paste.openstack.org/show/613056/ | 01:58 |
| *** hongbin has joined #openstack-infra | 01:59 | |
| ianw | it does seem like it's right | 02:00 |
| *** tuanluong has joined #openstack-infra | 02:00 | |
| ianw | i think it might be the ""'s | 02:02 |
| *** bobh has quit IRC | 02:04 | |
| *** namnh has quit IRC | 02:05 | |
| *** Julien-zte has joined #openstack-infra | 02:14 | |
| fungi | that's the symptom i was seeing with the new ubuntu-trusty images as well (which are now rolled back to the previous ones and paused there) | 02:16 |
| ianw | nahh, it's just got to be under "server:" | 02:16 |
| *** dave-mccowan has joined #openstack-infra | 02:16 | |
| ianw | ok ... i think we want to rebuild at this point, let me get a change out | 02:17 |
| fungi | i guess we can unpause the trusty images too if that's the issue | 02:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Fix unbound logging config syntax https://review.openstack.org/475209 | 02:19 |
| ianw | if you want to check "ssh root@46.254.11.23" i got that going with ^ | 02:19 |
| ianw | http://paste.openstack.org/show/613057/ | 02:21 |
| ianw | i'll just grab some lunch, but i can merge and rebuild images ... they should be ok for when .eu wakes up | 02:24 |
| *** mdrabe has joined #openstack-infra | 02:26 | |
| *** mdrabe has quit IRC | 02:31 | |
| *** Julien-zte has quit IRC | 02:33 | |
| ianw | ohhh ... there might be issues merging it if the nodes are stuck, hmm, i guess reverting xenial will do it | 02:36 |
| ianw | that would be 340 | 02:36 |
| ianw | ok, i'm going to delete xenial 340, since we know it's a config syntax issue | 02:37 |
| ianw | ok, that's cleared out | 02:40 |
| *** nicolasbock has joined #openstack-infra | 02:42 | |
| ianw | alright, top of the gate starting to move along a bit ... i think that might get us there | 02:43 |
| clarkb | I've approved we can probably force merge if necessary too | 02:47 |
| clarkb | and sorry for that my derp | 02:47 |
| *** gomarivera_ has joined #openstack-infra | 02:48 | |
| *** gomarivera_ has quit IRC | 02:48 | |
| *** gomarivera_ has joined #openstack-infra | 02:49 | |
| clarkb | looks like things are moving. I'll keep an eye on irc so let me know if I can help | 02:49 |
| ianw | yep, i think that's sufficient, but i can redo the builds so we get the unbound log file anyway | 02:49 |
| *** gomarivera has quit IRC | 02:50 | |
| *** thorst has joined #openstack-infra | 02:53 | |
| *** yamamoto has joined #openstack-infra | 02:53 | |
| *** thorst has quit IRC | 02:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix unbound logging config syntax https://review.openstack.org/475209 | 03:00 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Revert "Move ubuntu-trusty image pause to correct section" https://review.openstack.org/475213 | 03:01 |
| *** ramishra has joined #openstack-infra | 03:05 | |
| *** jamesmcarthur has joined #openstack-infra | 03:10 | |
| *** hichihara has quit IRC | 03:13 | |
| *** jamesmcarthur has quit IRC | 03:14 | |
| *** gouthamr has quit IRC | 03:17 | |
| *** ykarel_ has joined #openstack-infra | 03:19 | |
| *** gongysh has joined #openstack-infra | 03:27 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Install systemd earlier for Stretch https://review.openstack.org/475206 | 03:27 |
| *** jamesmcarthur has joined #openstack-infra | 03:28 | |
| *** dave-mccowan has quit IRC | 03:33 | |
| *** jamesmcarthur has quit IRC | 03:39 | |
| *** bakbad has joined #openstack-infra | 03:40 | |
| *** hongbin has quit IRC | 03:46 | |
| *** thorst has joined #openstack-infra | 03:54 | |
| *** gomarivera_ has quit IRC | 03:54 | |
| *** gomarivera has joined #openstack-infra | 03:54 | |
| *** gouthamr has joined #openstack-infra | 03:58 | |
| *** thorst has quit IRC | 03:58 | |
| *** gomarivera has quit IRC | 03:59 | |
| *** gouthamr has quit IRC | 04:02 | |
| openstackgerrit | Merged openstack-infra/release-tools master: README: Rename --no_dry_run to --no-dry-run https://review.openstack.org/474995 | 04:09 |
| *** nicolasbock has quit IRC | 04:14 | |
| *** laurelm has joined #openstack-infra | 04:15 | |
| *** namnh has joined #openstack-infra | 04:24 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: requirements: update ansible version https://review.openstack.org/475228 | 04:29 |
| *** winggundamth has joined #openstack-infra | 04:31 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: requirements: remove ordereddict https://review.openstack.org/475229 | 04:32 |
| *** ramishra has quit IRC | 04:32 | |
| *** SotK_ has quit IRC | 04:35 | |
| *** nicolasbock has joined #openstack-infra | 04:36 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 04:38 | |
| *** SotK has joined #openstack-infra | 04:42 | |
| *** salv-orlando has joined #openstack-infra | 04:51 | |
| *** salv-orlando has quit IRC | 04:55 | |
| *** tuanluong has quit IRC | 04:59 | |
| *** sree has joined #openstack-infra | 05:01 | |
| *** salv-orlando has joined #openstack-infra | 05:01 | |
| ianw | bah, i put a bunch of status about nb03 in #openstack-dib | 05:03 |
| ianw | pabelanger: ^ rebooted, cleared out ... is building again, see how it goes. need to keep an eye on if it's leaking | 05:03 |
| ianw | or if it just got full disk and then spiraled out of control | 05:03 |
| *** zaro_ is now known as zaro | 05:04 | |
| *** prateek has joined #openstack-infra | 05:04 | |
| *** apetrich has joined #openstack-infra | 05:06 | |
| openstackgerrit | Adrian Turjak proposed openstack-infra/project-config master: new project - Adjutant https://review.openstack.org/474867 | 05:14 |
| *** pcaruana has joined #openstack-infra | 05:14 | |
| *** salv-orl_ has joined #openstack-infra | 05:16 | |
| *** salv-orlando has quit IRC | 05:18 | |
| ianw | heh, centos might have been spared failures due to it's failing to build loop | 05:18 |
| *** jascott1 has quit IRC | 05:20 | |
| *** ltomasbo|away is now known as ltomasbo | 05:21 | |
| *** nmathew has joined #openstack-infra | 05:22 | |
| *** dhajare has joined #openstack-infra | 05:22 | |
| *** e0ne has joined #openstack-infra | 05:33 | |
| *** jamesmcarthur has joined #openstack-infra | 05:39 | |
| *** ykarel_ is now known as ykarel | 05:41 | |
| *** dhajare has quit IRC | 05:42 | |
| *** jamesmcarthur has quit IRC | 05:44 | |
| *** jbadiapa_ has joined #openstack-infra | 05:46 | |
| *** jtomasek has joined #openstack-infra | 05:48 | |
| *** e0ne has quit IRC | 05:48 | |
| *** tushar has joined #openstack-infra | 05:54 | |
| *** tumbarka has quit IRC | 05:54 | |
| *** thorst has joined #openstack-infra | 05:54 | |
| *** thorst has quit IRC | 05:59 | |
| AJaeger | fungi, thanks for fixing and sorry for leading to wrong kind of nodes | 05:59 |
| *** dhajare has joined #openstack-infra | 06:00 | |
| *** aeng has quit IRC | 06:00 | |
| *** tuanluong has joined #openstack-infra | 06:02 | |
| *** e0ne has joined #openstack-infra | 06:02 | |
| *** tuanluong has quit IRC | 06:02 | |
| *** e0ne has quit IRC | 06:06 | |
| *** AJaeger has quit IRC | 06:09 | |
| *** jtomasek has quit IRC | 06:12 | |
| *** aeng has joined #openstack-infra | 06:15 | |
| ianw | hmm ... centos-7 hit the "no partition error" | 06:17 |
| *** rcernin has joined #openstack-infra | 06:19 | |
| *** makowals has quit IRC | 06:19 | |
| *** makowals has joined #openstack-infra | 06:19 | |
| *** Hal has quit IRC | 06:19 | |
| *** winggundamth has quit IRC | 06:20 | |
| *** markus_z has joined #openstack-infra | 06:22 | |
| *** andreas_s has joined #openstack-infra | 06:26 | |
| *** aeng has quit IRC | 06:26 | |
| *** bhavik1 has joined #openstack-infra | 06:28 | |
| *** bhavik1 has quit IRC | 06:39 | |
| *** aeng has joined #openstack-infra | 06:40 | |
| kevinbenton | fungi: sorry, i meant 'external' from the perspective of the virtual cloud being built | 06:47 |
| kevinbenton | fungi: so when tempest creates an SSH connection into an booted VM via it's floating IP, it's sourcing the connection from an address on the same subnet as the floating IP | 06:48 |
| *** jtomasek has joined #openstack-infra | 06:49 | |
| kevinbenton | fungi: the reason I'm asking is that I think we had a serious regression in a DVR upgrade path that basically stopped floating IPs from being able to communicate with things outside of the external network | 06:50 |
| kevinbenton | fungi: (loss of default route in a namespace) | 06:50 |
| kevinbenton | fungi: so i'm figuring out if grenade missed it because of missing coverage, or just because it wouldn't have caught it by virtue of being directly attached to the VMs' floating IP network | 06:51 |
| *** dhinesh has quit IRC | 06:52 | |
| *** salv-orl_ has quit IRC | 06:55 | |
| *** salv-orlando has joined #openstack-infra | 06:57 | |
| *** ramishra has joined #openstack-infra | 07:00 | |
| *** Julien-zte has joined #openstack-infra | 07:02 | |
| *** martinkopec has joined #openstack-infra | 07:03 | |
| *** tesseract has joined #openstack-infra | 07:05 | |
| *** amotoki is now known as amotoki_away | 07:05 | |
| *** shardy has joined #openstack-infra | 07:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Sync after writing partition table https://review.openstack.org/475203 | 07:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Sync after writing partition table https://review.openstack.org/475203 | 07:14 |
| *** jpich has joined #openstack-infra | 07:15 | |
| *** yamamoto has quit IRC | 07:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: [fuel] Grant permissions to fuel-stable-core https://review.openstack.org/474541 | 07:18 |
| openstackgerrit | Merged openstack-infra/project-config master: Promote manila lvm centos 7 job to voting https://review.openstack.org/472302 | 07:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Make the linter job voting https://review.openstack.org/473859 | 07:20 |
| *** jascott1 has joined #openstack-infra | 07:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch Fedora job to voting https://review.openstack.org/474301 | 07:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Add experimental py35 job to barbican-tempest-plugin https://review.openstack.org/474572 | 07:21 |
| *** winggundamth has joined #openstack-infra | 07:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add py35 job to instack-undercloud https://review.openstack.org/475026 | 07:21 |
| openstackgerrit | Merged openstack-infra/project-config master: neutron: Pass python version to gate hook https://review.openstack.org/474585 | 07:22 |
| *** jascott1 has quit IRC | 07:25 | |
| *** jaosorior has joined #openstack-infra | 07:26 | |
| *** e0ne has joined #openstack-infra | 07:26 | |
| *** Hal has joined #openstack-infra | 07:27 | |
| *** Dinesh_Bhor has joined #openstack-infra | 07:31 | |
| openstackgerrit | Merged openstack-infra/project-config master: Cleanup lvm-multibackend job usage for nova/cinder https://review.openstack.org/474786 | 07:33 |
| *** rcernin has quit IRC | 07:34 | |
| openstackgerrit | Unana Okpoyo proposed openstack-infra/project-config master: A Devstack-plugin that installs a VMAX backend https://review.openstack.org/474924 | 07:34 |
| openstackgerrit | Jan Provaznik proposed openstack-infra/project-config master: Add new puppet-ganesha project https://review.openstack.org/474474 | 07:36 |
| *** jpena|off is now known as jpena | 07:36 | |
| *** xarses has joined #openstack-infra | 07:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Place subunit result file in the correct place https://review.openstack.org/474312 | 07:36 |
| openstackgerrit | Merged openstack-infra/project-config master: [Zun] Turn dsvm job to voting https://review.openstack.org/471842 | 07:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Add OSH-AddOns to IRC channel notification https://review.openstack.org/471207 | 07:36 |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/subunit2sql master: Add vcs status to version option https://review.openstack.org/475275 | 07:36 |
| openstackgerrit | Merged openstack-infra/project-config master: New project - OpenStack-Spaceport https://review.openstack.org/472791 | 07:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Add coverage job to tempest check https://review.openstack.org/470856 | 07:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Gerritbot: track current branches for sahara repositories https://review.openstack.org/471860 | 07:37 |
| *** rcernin has joined #openstack-infra | 07:37 | |
| *** rcernin has quit IRC | 07:37 | |
| *** markmcd has quit IRC | 07:37 | |
| *** rcernin has joined #openstack-infra | 07:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create the requirements stable maintence team https://review.openstack.org/470419 | 07:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Create tap-as-a-service-dashboard project https://review.openstack.org/468747 | 07:39 |
| *** markmcd has joined #openstack-infra | 07:39 | |
| *** dizquierdo has joined #openstack-infra | 07:41 | |
| *** gibi has joined #openstack-infra | 07:41 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add neutron-dynamic-routing job with ryu master https://review.openstack.org/461584 | 07:44 |
| openstackgerrit | Merged openstack-infra/project-config master: Add log copier to dib jobs https://review.openstack.org/474828 | 07:45 |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** ccamacho has joined #openstack-infra | 07:50 | |
| *** amoralej|off is now known as amoralej | 07:51 | |
| *** tuanluong has joined #openstack-infra | 07:53 | |
| *** alexchadin has joined #openstack-infra | 07:54 | |
| *** aarefiev_afk is now known as aarefiev | 07:54 | |
| *** thorst has joined #openstack-infra | 07:55 | |
| *** openstackgerrit has joined #openstack-infra | 07:56 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/project-config master: Create tempest non-voting gate in kuryr-kubernetes https://review.openstack.org/474993 | 07:56 |
| *** mhickey has joined #openstack-infra | 07:59 | |
| *** thorst has quit IRC | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** rossella_s has joined #openstack-infra | 08:02 | |
| *** SumitNaiksatam has quit IRC | 08:03 | |
| *** dizquierdo has quit IRC | 08:08 | |
| *** flepied has joined #openstack-infra | 08:10 | |
| *** rossella_s has quit IRC | 08:14 | |
| *** dtantsur|afk is now known as dtantsur | 08:16 | |
| *** lucas-afk is now known as lucasagomes | 08:16 | |
| *** florianf has joined #openstack-infra | 08:17 | |
| *** SumitNaiksatam has joined #openstack-infra | 08:17 | |
| *** rossella_s has joined #openstack-infra | 08:21 | |
| *** hashar has joined #openstack-infra | 08:22 | |
| *** namnh has quit IRC | 08:25 | |
| *** efoley has joined #openstack-infra | 08:27 | |
| *** ralonsoh has joined #openstack-infra | 08:28 | |
| kornicameister | hi guys, I am looking for some help around my problem with python27 gate | 08:30 |
| kornicameister | is there anyone that could help out | 08:30 |
| kornicameister | ? | 08:30 |
| *** amotoki_away is now known as amotoki | 08:31 | |
| *** jascott1 has joined #openstack-infra | 08:32 | |
| *** pgadiya has joined #openstack-infra | 08:35 | |
| *** electrofelix has joined #openstack-infra | 08:40 | |
| *** psachin has joined #openstack-infra | 08:41 | |
| *** amotoki is now known as amotoki_away | 08:44 | |
| openstackgerrit | Andy McCrae proposed openstack-infra/project-config master: Propose a job for keystone rolling upgrades https://review.openstack.org/471419 | 08:47 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: bifrost: Fix node for bifrost jobs https://review.openstack.org/475290 | 08:47 |
| *** rossella_s has quit IRC | 08:47 | |
| *** e0ne has quit IRC | 08:47 | |
| openstackgerrit | Christian Berendt proposed openstack-infra/jeepyb master: Remove leading whitespaces from the Gerrit welcome message https://review.openstack.org/310926 | 08:53 |
| *** gomarivera has joined #openstack-infra | 08:55 | |
| *** gomarivera has quit IRC | 08:59 | |
| *** derekh has joined #openstack-infra | 09:00 | |
| openstackgerrit | Pratik Shah proposed openstack-infra/project-config master: [WIP] Added rough changes to check experimental unit tests for Omni project using Zuul https://review.openstack.org/475300 | 09:01 |
| *** yamamoto has joined #openstack-infra | 09:02 | |
| *** salv-orlando has quit IRC | 09:03 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: bifrost: Fix node for bifrost jobs https://review.openstack.org/475290 | 09:03 |
| openstackgerrit | Merged openstack-infra/release-tools master: lp-tag.py: be prepared for empty lines in input https://review.openstack.org/473992 | 09:05 |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Removing duplicate set in ovb-settings https://review.openstack.org/475304 | 09:11 |
| *** dizquierdo has joined #openstack-infra | 09:11 | |
| *** salv-orlando has joined #openstack-infra | 09:17 | |
| *** e0ne has joined #openstack-infra | 09:18 | |
| *** leifmadsen has quit IRC | 09:25 | |
| *** leifmadsen has joined #openstack-infra | 09:26 | |
| *** rossella_s has joined #openstack-infra | 09:31 | |
| *** Guest11 has joined #openstack-infra | 09:33 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/project-config master: Create tempest non-voting gate in kuryr-kubernetes https://review.openstack.org/474993 | 09:37 |
| *** hrubi has quit IRC | 09:39 | |
| *** hrubi has joined #openstack-infra | 09:39 | |
| dalvarez | hi guys, do you know if this is going to be fixed or i did something wrong: https://storyboard.openstack.org/#!/story/2000955 | 09:40 |
| dalvarez | thanks a lot | 09:40 |
| *** sambetts|afk is now known as sambetts | 09:42 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:52 | |
| *** tmorin has joined #openstack-infra | 09:54 | |
| *** akscram has quit IRC | 09:55 | |
| *** thorst has joined #openstack-infra | 09:56 | |
| *** stakeda has quit IRC | 09:57 | |
| *** bakbad has quit IRC | 09:58 | |
| *** bakbad has joined #openstack-infra | 09:58 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Fix stackviz collect data https://review.openstack.org/474559 | 10:00 |
| *** thorst has quit IRC | 10:00 | |
| *** cschwede has quit IRC | 10:02 | |
| *** otherwiseguy has quit IRC | 10:02 | |
| *** dhinesh has joined #openstack-infra | 10:03 | |
| *** katkapilatova has joined #openstack-infra | 10:03 | |
| *** liujiong has quit IRC | 10:05 | |
| *** otherwiseguy has joined #openstack-infra | 10:07 | |
| *** dhinesh has quit IRC | 10:07 | |
| *** cschwede has joined #openstack-infra | 10:07 | |
| *** sree_ has joined #openstack-infra | 10:10 | |
| *** sree_ is now known as Guest7537 | 10:11 | |
| *** dimtruck is now known as zz_dimtruck | 10:12 | |
| *** cshastri has quit IRC | 10:12 | |
| dalvarez | frickler, thanks for looking into it, should it be fixed now then? | 10:12 |
| *** sree has quit IRC | 10:13 | |
| *** rossella_s has quit IRC | 10:13 | |
| *** Guest11 has quit IRC | 10:16 | |
| frickler | dalvarez: I don't know, but the old logs are expired and deleted, so it's difficult to tell even what the error was | 10:18 |
| dalvarez | frickler, ah true... in my first comment I paste the error i got locally which IIRC it was the same i saw in gate | 10:20 |
| *** rhallisey has joined #openstack-infra | 10:21 | |
| *** persia has joined #openstack-infra | 10:27 | |
| *** toabctl has quit IRC | 10:28 | |
| *** hwoarang has quit IRC | 10:28 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/475321 | 10:29 |
| *** hwoarang has joined #openstack-infra | 10:29 | |
| *** rossella_s has joined #openstack-infra | 10:29 | |
| *** toabctl has joined #openstack-infra | 10:30 | |
| *** frickler has quit IRC | 10:30 | |
| *** frickler has joined #openstack-infra | 10:30 | |
| *** yamamoto has quit IRC | 10:30 | |
| *** yamamoto has joined #openstack-infra | 10:31 | |
| *** Kevin_Zheng has quit IRC | 10:31 | |
| *** claudiub has joined #openstack-infra | 10:32 | |
| *** wfoster has quit IRC | 10:37 | |
| *** yamamoto has quit IRC | 10:37 | |
| *** Julien-zte has quit IRC | 10:38 | |
| *** rossella_s has quit IRC | 10:38 | |
| *** Guest48624 has quit IRC | 10:39 | |
| *** lucasagomes has quit IRC | 10:39 | |
| *** LindaWang has quit IRC | 10:39 | |
| *** jamesmcarthur has joined #openstack-infra | 10:40 | |
| *** wfoster has joined #openstack-infra | 10:40 | |
| *** sdague has joined #openstack-infra | 10:41 | |
| *** rook has joined #openstack-infra | 10:41 | |
| *** rook is now known as Guest52034 | 10:42 | |
| *** Guest11 has joined #openstack-infra | 10:42 | |
| *** katkapilatova has left #openstack-infra | 10:43 | |
| openstackgerrit | Dmitry Nikishov proposed openstack-infra/project-config master: Add fuel-plugin-detach-netnode https://review.openstack.org/475323 | 10:44 |
| *** jamesmcarthur has quit IRC | 10:44 | |
| *** hrubi has quit IRC | 10:45 | |
| *** aviau has quit IRC | 10:47 | |
| *** aviau has joined #openstack-infra | 10:47 | |
| *** katkapilatova has joined #openstack-infra | 10:47 | |
| *** Guest7537 has quit IRC | 10:48 | |
| openstackgerrit | Pratik Shah proposed openstack-infra/project-config master: Added changes to run unit tests for Omni project using Zuul https://review.openstack.org/475300 | 10:50 |
| *** katkapilatova has left #openstack-infra | 10:51 | |
| *** tojuvone has joined #openstack-infra | 10:52 | |
| *** tojuvone has left #openstack-infra | 10:52 | |
| *** katkapilatova has joined #openstack-infra | 10:53 | |
| *** nmathew has quit IRC | 10:54 | |
| *** rossella_s has joined #openstack-infra | 10:56 | |
| *** jkilpatr has joined #openstack-infra | 10:57 | |
| *** lucasagomes has joined #openstack-infra | 10:58 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: jenkins: scripts: install-distro-packages: Use --no-recommends on zypper https://review.openstack.org/475331 | 11:01 |
| *** tushar has quit IRC | 11:01 | |
| *** tushar has joined #openstack-infra | 11:01 | |
| *** hrubi has joined #openstack-infra | 11:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/475321 | 11:04 |
| *** lucasagomes is now known as lucas-hungry | 11:04 | |
| openstackgerrit | Lajos Katona proposed openstack-infra/project-config master: Enable q-trunk in heat dsvm functional jobs https://review.openstack.org/473700 | 11:06 |
| oanson | Hello. On my environment (Fedora 25, updated) devstack is broken, failing on setting root privileges on mysqk (mariadb-10.1.24). Gates seem to work, but they use Ubuntu Xenial with mysql (not mariadb). Anyone else ran into this issue? | 11:06 |
| *** winggundamth has quit IRC | 11:07 | |
| *** rossella_ has joined #openstack-infra | 11:07 | |
| *** HeOS has quit IRC | 11:08 | |
| oanson | Error is for command: +lib/databases/mysql:configure_database_mysql:87 sudo mysqladmin -u root password password : You cannot use 'password' command as mysqld runs with grant tables disabled (was started with --skip-grant-tables). Use: "mysqladmin flush-privileges password '*'" instead | 11:09 |
| oanson | And for sudo mysql -uroot -psecrete -h127.0.0.1 -e 'GRANT ALL PRIVILEGES ON *.* TO '\''root'\''@'\''%'\'' identified by '\''password\'';' : ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) | 11:09 |
| *** yland has joined #openstack-infra | 11:10 | |
| *** flepied has quit IRC | 11:10 | |
| *** rossella_s has quit IRC | 11:10 | |
| *** Guest11 has quit IRC | 11:12 | |
| *** lucasagomes has joined #openstack-infra | 11:14 | |
| openstackgerrit | Hieu LE proposed openstack-infra/project-config master: Enable horizon for designate grenade gate https://review.openstack.org/475337 | 11:14 |
| *** salv-orl_ has joined #openstack-infra | 11:15 | |
| *** katkapilatova has left #openstack-infra | 11:15 | |
| *** salv-orlando has quit IRC | 11:18 | |
| *** dhinesh has joined #openstack-infra | 11:20 | |
| *** hrubi has quit IRC | 11:22 | |
| *** ccamacho is now known as ccamacho|lunch | 11:22 | |
| *** dhinesh has quit IRC | 11:25 | |
| *** flepied has joined #openstack-infra | 11:27 | |
| *** hrubi has joined #openstack-infra | 11:28 | |
| *** yamamoto has joined #openstack-infra | 11:32 | |
| *** ldnunes has joined #openstack-infra | 11:36 | |
| *** jpena is now known as jpena|lunch | 11:37 | |
| *** prateek has quit IRC | 11:39 | |
| *** gordc has joined #openstack-infra | 11:39 | |
| *** yamamoto has quit IRC | 11:40 | |
| *** links has joined #openstack-infra | 11:40 | |
| *** yamamoto has joined #openstack-infra | 11:41 | |
| *** Julien-zte has joined #openstack-infra | 11:41 | |
| *** pgadiya has quit IRC | 11:41 | |
| *** links has quit IRC | 11:41 | |
| *** xarses has quit IRC | 11:42 | |
| *** bakbad has quit IRC | 11:42 | |
| dhellmann | mordred : is there any reason to hold off on a pbr 3.1.0 release? | 11:43 |
| openstackgerrit | Egor Borisov proposed openstack-infra/jenkins-job-builder master: Add integration with Allure Report plugin. https://review.openstack.org/475185 | 11:44 |
| *** armax has joined #openstack-infra | 11:44 | |
| *** LindaWang has joined #openstack-infra | 11:45 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Add script parsing status multinode periodic jobs https://review.openstack.org/475197 | 11:45 |
| *** yamamoto has quit IRC | 11:46 | |
| *** thorst has joined #openstack-infra | 11:48 | |
| *** hrubi has quit IRC | 11:49 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add log streaming test https://review.openstack.org/471079 | 11:51 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Add multinode jobs to promotion https://review.openstack.org/475200 | 11:54 |
| *** akscram has joined #openstack-infra | 11:56 | |
| *** gongysh has quit IRC | 11:56 | |
| *** rfolco has joined #openstack-infra | 11:58 | |
| openstackgerrit | Saad Zaher proposed openstack-infra/project-config master: Added Freezer tempest plugin repo https://review.openstack.org/472732 | 11:59 |
| *** wolverineav has joined #openstack-infra | 12:00 | |
| *** rossella_ has quit IRC | 12:02 | |
| *** trown|brb is now known as trown | 12:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: github: gracefully handle unknown event https://review.openstack.org/473249 | 12:03 |
| *** dprince has joined #openstack-infra | 12:03 | |
| *** apetrich has quit IRC | 12:05 | |
| *** apetrich has joined #openstack-infra | 12:08 | |
| *** cshen has joined #openstack-infra | 12:08 | |
| *** amoralej is now known as amoralej|lunch | 12:09 | |
| *** jtomasek_ has joined #openstack-infra | 12:09 | |
| *** yamamoto has joined #openstack-infra | 12:16 | |
| *** lucas-hungry has quit IRC | 12:16 | |
| *** Hal has quit IRC | 12:17 | |
| *** rossella_ has joined #openstack-infra | 12:18 | |
| *** askb has quit IRC | 12:19 | |
| *** LindaWang has quit IRC | 12:20 | |
| *** LindaWang has joined #openstack-infra | 12:20 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add support for zuul.d configuration split https://review.openstack.org/473764 | 12:20 |
| *** jtomasek_ has quit IRC | 12:26 | |
| *** dtantsur is now known as dtantsur|brb | 12:26 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Fix stackviz collect data https://review.openstack.org/474559 | 12:27 |
| *** rossella_ has quit IRC | 12:28 | |
| *** jaypipes has joined #openstack-infra | 12:28 | |
| *** mhickey has quit IRC | 12:31 | |
| *** rossella_ has joined #openstack-infra | 12:32 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** mdrabe has joined #openstack-infra | 12:35 | |
| *** Hal has joined #openstack-infra | 12:35 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: github: prevent getRepoPermission to raise AttributeError https://review.openstack.org/475368 | 12:36 |
| *** rlandy has joined #openstack-infra | 12:37 | |
| *** kgiusti has joined #openstack-infra | 12:38 | |
| *** tuanluong has quit IRC | 12:42 | |
| lennyb | oanson, I am facing same issue on Fedora24 | 12:47 |
| oanson | lennyb, I don't have a solution yet :( | 12:47 |
| *** sshnaidm is now known as sshnaidm|afk | 12:48 | |
| lennyb | oanson, me neither. We need Fedora :(. | 12:48 |
| sbezverk | fungi: ping | 12:51 |
| *** gouthamr has joined #openstack-infra | 12:53 | |
| *** rossella_ has quit IRC | 12:54 | |
| *** LindaWang has quit IRC | 12:54 | |
| *** cuongnv has quit IRC | 12:55 | |
| *** flepied has quit IRC | 12:58 | |
| *** tosky has joined #openstack-infra | 13:00 | |
| robcresswell | Anyone seen an issue with pip installing the wrong version when using the project tarballs? I'm assuming this is a stale build dir or cache or something, but its in a venv so I assumed that would work around that issue. | 13:00 |
| *** mriedem has joined #openstack-infra | 13:00 | |
| *** jcoufal has joined #openstack-infra | 13:02 | |
| odyssey4me | we're still seeing a lot of DNS resolution failures clarkb fungi :( for example: http://logs.openstack.org/05/471805/7/check/gate-openstack-ansible-tests-ansible-func-centos-7/4aeb436/console.html#_2017-06-19_13_00_36_300577 | 13:03 |
| lennyb | oanson, is there a irc channel for global-requirements? Somehow I have a feeling that it's related | 13:06 |
| *** rossella_ has joined #openstack-infra | 13:06 | |
| *** ccamacho|lunch is now known as ccamacho | 13:07 | |
| robcresswell | lennyb: #openstack-requirements ? | 13:08 |
| *** ykarel has quit IRC | 13:08 | |
| *** amoralej|lunch is now known as amoralej | 13:08 | |
| *** esberglu has joined #openstack-infra | 13:08 | |
| *** sree has joined #openstack-infra | 13:09 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:10 | |
| *** jcoufal has quit IRC | 13:11 | |
| *** flepied has joined #openstack-infra | 13:12 | |
| *** jamesmcarthur has joined #openstack-infra | 13:13 | |
| lennyb | robcresswell, thanks | 13:13 |
| *** jamesmcarthur has quit IRC | 13:13 | |
| *** jamesmcarthur has joined #openstack-infra | 13:13 | |
| fungi | odyssey4me: yeah, the only additional change we've made is to enable error logging in unbound so we can try to better pinpoint the issue, though we suspect the problem is global for the .org tld so other than trying to convince afilias that they have a problem there may not be much else we can do | 13:15 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: jenkins: scripts: propose_update.sh: Use sync-test-repos.sh from OSA repo https://review.openstack.org/475381 | 13:16 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: gitignore: Ignore .venv https://review.openstack.org/475382 | 13:16 |
| odyssey4me | fungi but this is untenable - our failure rate is debilitating, and this was never a problem before friday | 13:16 |
| odyssey4me | so we do need to figure out something | 13:17 |
| *** HeOS has joined #openstack-infra | 13:18 | |
| *** jcoufal has joined #openstack-infra | 13:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: base64 encode user_data sent to create server https://review.openstack.org/475385 | 13:18 |
| fungi | odyssey4me: we were seeing it well before friday from what i understood. any dns records in particular impacting you (git.o.o? tarballs.o.o? something else?). wondering if there are ways we could inject extended ttls for pre-cached values into our images or something | 13:19 |
| fungi | odyssey4me: you're going through the local caching resolver configured on our images, right? | 13:19 |
| *** ericyoung has joined #openstack-infra | 13:19 | |
| odyssey4me | fungi we don't reconfigure the resolvers, so I assume so | 13:20 |
| odyssey4me | any containers resolve through the hosts too | 13:20 |
| odyssey4me | the failure I'm seeing most often is images.linuxcontainers.org: http://logs.openstack.org/05/471805/7/check/gate-openstack-ansible-tests-ansible-func-centos-7/4aeb436/console.html#_2017-06-19_13_00_36_300577 | 13:21 |
| fungi | odyssey4me: that would match our observations that there's something amiss with one or more of the .org tld nameservers | 13:21 |
| openstackgerrit | Merged openstack/diskimage-builder master: Install systemd earlier for Stretch https://review.openstack.org/475206 | 13:22 |
| fungi | i'll be at a better computer in a few minutes and should be able to work out whether we're already collecting the new logs we're generating or need to adjust jobs to copy those out | 13:23 |
| mordred | fungi: https://serverfault.com/questions/18748/overriding-some-dns-entries-in-bind-for-internal-networks | 13:23 |
| mordred | fungi: we could run bind and override individual records then point our unbound at our bind ... although that seems very sadmaking if we wind up there | 13:24 |
| sshnaidm | do we have a mirror (afs?) for downloading a cirros image? | 13:24 |
| *** jcoufal has quit IRC | 13:24 | |
| sshnaidm | http://download.cirros-cloud.net drops connections | 13:24 |
| mordred | fungi: that same article shows syntax for doing the same with unbound directly | 13:25 |
| *** makowals has quit IRC | 13:26 | |
| fungi | sshnaidm: we preinstall cirros and maybe other images onto the filesystems of our images, based on which ones devstack claims to need | 13:26 |
| sshnaidm | fungi, where is it located? | 13:27 |
| *** jcoufal has joined #openstack-infra | 13:28 | |
| mordred | sshnaidm: /opt/cache/files | 13:29 |
| mordred | sshnaidm: http://paste.openstack.org/show/613107/ | 13:29 |
| sshnaidm | mordred, thanks | 13:30 |
| *** Guest52034 is now known as rook | 13:30 | |
| *** Goneri has joined #openstack-infra | 13:30 | |
| *** makowals has joined #openstack-infra | 13:32 | |
| *** namnh has joined #openstack-infra | 13:34 | |
| *** alexchadin has quit IRC | 13:34 | |
| *** xarses has joined #openstack-infra | 13:34 | |
| *** baoli has joined #openstack-infra | 13:36 | |
| namnh | Hi everyone, could you please help me to review this patch: https://review.openstack.org/#/c/474824/. It got +2 from one core-reviewer. | 13:36 |
| *** xarses_ has joined #openstack-infra | 13:36 | |
| *** xyang1 has joined #openstack-infra | 13:37 | |
| *** gcb has joined #openstack-infra | 13:38 | |
| *** xarses has quit IRC | 13:39 | |
| *** LindaWang has joined #openstack-infra | 13:39 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config master: Add a non-voting neutron and networking-ovn job to ovsdbapp https://review.openstack.org/475390 | 13:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: base64 encode user_data sent to create server https://review.openstack.org/475385 | 13:44 |
| dmellado | clarkb: fungi would you mind having a look at this patch when you get the chance? https://review.openstack.org/#/c/474993/ | 13:44 |
| dmellado | thanks in advance! | 13:44 |
| dmellado | it's in order to create a non-voting kuryr-k8s gate | 13:44 |
| *** jamesmcarthur has quit IRC | 13:44 | |
| openstackgerrit | Tin Lam proposed openstack-infra/project-config master: Add new repo openstack-helm-infra under OSH https://review.openstack.org/469079 | 13:50 |
| *** sfinucan_ is now known as sfinucan | 13:51 | |
| *** yamamoto has quit IRC | 13:52 | |
| *** yamamoto has joined #openstack-infra | 13:52 | |
| *** salv-orl_ has quit IRC | 13:55 | |
| *** rbrndt has joined #openstack-infra | 13:55 | |
| *** gomarivera has joined #openstack-infra | 13:57 | |
| *** dtantsur|brb is now known as dtantsur | 13:57 | |
| *** yamamoto has quit IRC | 13:57 | |
| *** salv-orlando has joined #openstack-infra | 13:57 | |
| *** Julien-zte has quit IRC | 13:59 | |
| *** Julien-zte has joined #openstack-infra | 14:00 | |
| *** hrubi has joined #openstack-infra | 14:02 | |
| *** ykarel has joined #openstack-infra | 14:02 | |
| *** baoli has quit IRC | 14:03 | |
| *** baoli has joined #openstack-infra | 14:06 | |
| *** camunoz has joined #openstack-infra | 14:08 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove use of six and ordereddict libraries https://review.openstack.org/475135 | 14:09 |
| yuval | oanson: lennyb: I think I found a solution to that | 14:12 |
| *** yamamoto has joined #openstack-infra | 14:12 | |
| *** prateek has joined #openstack-infra | 14:13 | |
| *** prateek has quit IRC | 14:14 | |
| * lennyb listening to yuval | 14:14 | |
| *** zz_dimtruck is now known as dimtruck | 14:14 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Remove py2 SocketServer import https://review.openstack.org/475399 | 14:16 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add note to README about Python3 https://review.openstack.org/475070 | 14:16 |
| *** dave-mccowan has joined #openstack-infra | 14:17 | |
| *** prateek has joined #openstack-infra | 14:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Temporarily copy unbound.log in test-results https://review.openstack.org/475402 | 14:19 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/devstack-gate master: Temporarily copy unbound.log in cleanup_host https://review.openstack.org/475403 | 14:19 |
| fungi | not sure whether we need both, but there they are for completeness. i'm going to try to trip a failure by rechecking 475403 and seeing what we catch there | 14:20 |
| lennyb | yuval, what solution are you thinking about? | 14:24 |
| *** hrubi has quit IRC | 14:24 | |
| *** gongysh has joined #openstack-infra | 14:24 | |
| *** gongysh has quit IRC | 14:26 | |
| *** gouthamr has quit IRC | 14:26 | |
| *** gouthamr has joined #openstack-infra | 14:27 | |
| *** rossella_ has quit IRC | 14:28 | |
| *** sree has quit IRC | 14:29 | |
| fungi | the d-g change seems to properly collect the log at least: http://logs.openstack.org/03/475403/1/check/gate-dg-hooks-dsvm/b2280ae/logs/unbound_log.txt.gz | 14:29 |
| *** sree has joined #openstack-infra | 14:29 | |
| *** salv-orlando has quit IRC | 14:30 | |
| *** njohnston_ is now known as njohnston | 14:30 | |
| *** LindaWang has quit IRC | 14:31 | |
| yuval | lennyb: not sure, but I think https://review.openstack.org/#/c/475410/ | 14:32 |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config master: Move experimental cephfs jobs to check queue https://review.openstack.org/475411 | 14:33 |
| *** sree has quit IRC | 14:33 | |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config master: Move manila cephfs jobs to check queue https://review.openstack.org/475411 | 14:35 |
| *** jamesmcarthur has joined #openstack-infra | 14:35 | |
| *** lbragstad has joined #openstack-infra | 14:35 | |
| *** psachin has quit IRC | 14:36 | |
| *** yland has quit IRC | 14:37 | |
| *** hongbin has joined #openstack-infra | 14:38 | |
| *** pradk has joined #openstack-infra | 14:43 | |
| *** andrewbogott has quit IRC | 14:43 | |
| *** andrewbogott has joined #openstack-infra | 14:43 | |
| clarkb | fungi: +2'd the d-g change, we can probably go ahead and approve that one? left comment on the project-config change | 14:46 |
| *** bobh has joined #openstack-infra | 14:46 | |
| lennyb | yuval, thanks, I will check it and comment | 14:46 |
| *** rhallisey has quit IRC | 14:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: base64 encode user_data sent to create server https://review.openstack.org/475385 | 14:48 |
| *** AJaeger has joined #openstack-infra | 14:50 | |
| *** dhinesh has joined #openstack-infra | 14:51 | |
| mordred | clarkb, fungi: +A the d-g one | 14:51 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Test with bonded public interface https://review.openstack.org/385562 | 14:51 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Move PublicVirtualFixedIPs to enable-tls.yaml https://review.openstack.org/475419 | 14:51 |
| mordred | clarkb: once we land that shade patch ^^ we're good for a shade release - which will no-longer use novaclient | 14:51 |
| mordred | clarkb: (so we should obviously watch nodepool when we cut that one) | 14:52 |
| clarkb | mordred: ya was just reading that, I think there is a bug in the code from oslo :) it should use locale.getpreferredencoding not sys.getdefaultencoding as one represents the users pref (which better represents what stdin will be) and the other is what python is built to default to which is entirely up to who knows (I think) | 14:53 |
| *** yland has joined #openstack-infra | 14:54 | |
| *** prometheanfire has joined #openstack-infra | 14:54 | |
| *** yland is now known as Guest85443 | 14:55 | |
| *** dhinesh has quit IRC | 14:55 | |
| *** yamahata_ has joined #openstack-infra | 14:55 | |
| mordred | clarkb: kk | 14:56 |
| clarkb | mordred: I don't know that it is that big of a deal though, seems like it tries extra hard to find any encoding that works | 14:57 |
| *** makowals has quit IRC | 14:57 | |
| mordred | clarkb: yah - honestly, it should probably crab locale.getpreferredencoding and sys.getdefaultencoding and try both | 14:59 |
| *** makowals has joined #openstack-infra | 15:00 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Ping test horizon https://review.openstack.org/472758 | 15:01 |
| *** chlong has joined #openstack-infra | 15:03 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Increase OVN testing coverage https://review.openstack.org/475423 | 15:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: base64 encode user_data sent to create server https://review.openstack.org/475385 | 15:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Try all the detected encodings, not just one https://review.openstack.org/475424 | 15:13 |
| mordred | clarkb: k. that updates based on your thought - and follows up with one that does a list | 15:14 |
| *** hieulq_ has joined #openstack-infra | 15:14 | |
| *** ykarel has quit IRC | 15:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Add a Puppet module for the PTG IRC Bot https://review.openstack.org/475429 | 15:23 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Track ptgbot using gerritbot and storyboard https://review.openstack.org/475430 | 15:23 |
| *** e0ne has quit IRC | 15:26 | |
| *** sree has joined #openstack-infra | 15:28 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Use 'build_reno' setuptools extension if available https://review.openstack.org/475034 | 15:29 |
| *** armax has quit IRC | 15:30 | |
| *** rcernin has quit IRC | 15:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove use of six and ordereddict libraries https://review.openstack.org/475135 | 15:31 |
| *** amotoki_away is now known as amotoki | 15:32 | |
| *** xarses_ has quit IRC | 15:32 | |
| clarkb | fungi: mordred http://logs.openstack.org/03/475403/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/9e56ed1/console.html#_2017-06-19_15_16_13_869549 d-g failed in tempest :( | 15:33 |
| *** pcaruana has quit IRC | 15:33 | |
| clarkb | http://logs.openstack.org/03/475403/1/check/gate-tempest-dsvm-neutron-full-opensuse-422-nv/0180520/console.html#_2017-06-19_15_23_00_119260 in different jobs :( | 15:33 |
| *** baoli has quit IRC | 15:35 | |
| *** baoli has joined #openstack-infra | 15:35 | |
| fungi | clarkb: yep, i saw that as well. will recheck once the remainder finish | 15:35 |
| *** flepied has quit IRC | 15:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove use of six library https://review.openstack.org/475135 | 15:36 |
| *** mtanino has joined #openstack-infra | 15:36 | |
| *** baoli has quit IRC | 15:40 | |
| openstackgerrit | Sylvain Baubeau proposed openstack-infra/jenkins-job-builder master: Add support for black/white lists of labels in ghprb module https://review.openstack.org/475443 | 15:41 |
| *** jascott1 has quit IRC | 15:45 | |
| lennyb | yuval, your patch causes keystone to fail | 15:45 |
| *** Hal has quit IRC | 15:45 | |
| prometheanfire | what's the proper way of going about adding a new team, we had one approved but I'm not sure as to the next step https://review.openstack.org/470419 | 15:47 |
| prometheanfire | team/group | 15:47 |
| prometheanfire | gerrit groups | 15:49 |
| *** inc0 has quit IRC | 15:49 | |
| *** inc0 has joined #openstack-infra | 15:50 | |
| *** rhallisey has joined #openstack-infra | 15:50 | |
| *** mhickey has joined #openstack-infra | 15:50 | |
| prometheanfire | looks like we need someone added to https://review.openstack.org/#/admin/groups/1789,members | 15:51 |
| prometheanfire | https://docs.openstack.org/infra/manual/creators.html#update-the-gerrit-group-members | 15:51 |
| *** inc0 has quit IRC | 15:52 | |
| prometheanfire | so at least tonyb | 15:52 |
| *** inc0 has joined #openstack-infra | 15:52 | |
| pabelanger | right, we manually do it today. We just look for the original patch to add the project, the add that use to the group | 15:53 |
| clarkb | or ptl of project if part of existing project | 15:53 |
| pabelanger | ya | 15:53 |
| clarkb | in this case it is stable so I think tonyb wins? | 15:53 |
| *** baoli has joined #openstack-infra | 15:55 | |
| *** electrofelix has quit IRC | 15:56 | |
| *** hashar is now known as hasharAway | 15:57 | |
| *** andreas_s has quit IRC | 15:57 | |
| *** shardy has quit IRC | 15:58 | |
| *** bobh has quit IRC | 15:58 | |
| *** baoli has quit IRC | 16:00 | |
| dmellado | AJaeger: ping welcome back, I'm not really sure on how to modify this to also track non-master branches but future stables ones | 16:00 |
| *** armax has joined #openstack-infra | 16:01 | |
| *** prateek has quit IRC | 16:01 | |
| dmellado | re: https://review.openstack.org/#/c/474993/7/zuul/layout.yaml | 16:01 |
| dmellado | could you point me to doc or explain this to me? | 16:01 |
| dmellado | pabelanger: clarkb maybe you can do this too, thanks in advance! | 16:01 |
| *** bhavik1 has joined #openstack-infra | 16:01 | |
| clarkb | prometheanfire: I have added tonyb | 16:02 |
| *** baoli has joined #openstack-infra | 16:02 | |
| *** martinkopec has quit IRC | 16:02 | |
| *** namnh has quit IRC | 16:03 | |
| *** prateek has joined #openstack-infra | 16:03 | |
| clarkb | dmellado: you do a negative lookahead regex like on line 3067 to not match current stable branches, but thne it will match any other branch including future stable and master | 16:03 |
| dmellado | clarkb: oh, I see, thanks! | 16:04 |
| *** dprince has quit IRC | 16:06 | |
| clarkb | fungi: I got the recheck in | 16:06 |
| dmellado | so then branch: ^(?!stable/(?:pike)).*$ | 16:06 |
| dmellado | would match the future stable/pike when available | 16:06 |
| *** gyee has joined #openstack-infra | 16:06 | |
| dmellado | and for now would only do master | 16:07 |
| *** dizquierdo has quit IRC | 16:07 | |
| clarkb | dmellado: that would say don't match pike, and will match any other branch including master | 16:07 |
| dmellado | got it | 16:07 |
| fungi | thanks clarkb! | 16:08 |
| prometheanfire | clarkb: thanks | 16:08 |
| fungi | dmellado: so what you probably want is one that explicitly says not to match stable/newton or stable/ocata | 16:09 |
| fungi | (you can ignore stable/mitaka as eol is underway there) | 16:09 |
| dmellado | fungi: well, for kuryr-kubernetes we only have master so far | 16:09 |
| dmellado | so maybe I can put just branch:master | 16:09 |
| dmellado | I was thinking on the future stable releases | 16:10 |
| *** sree has quit IRC | 16:10 | |
| *** Apoorva has joined #openstack-infra | 16:10 | |
| dmellado | https://github.com/openstack/kuryr-kubernetes | 16:10 |
| *** hieulq_ has quit IRC | 16:10 | |
| *** sree has joined #openstack-infra | 16:10 | |
| dmellado | but maybe it would make sense to put | 16:11 |
| dmellado | branch: ^(?!stable/(newton|ocata)).*$ | 16:11 |
| dmellado | this would pick up master and then the future pike unless explicitly added | 16:11 |
| * dmellado learns a bit more each time he has to fight with infra xD | 16:11 | |
| dmellado | thanks! | 16:11 |
| clarkb | yup I think that is what AJaeger had in mind | 16:11 |
| *** pcaruana has joined #openstack-infra | 16:11 | |
| fungi | dmellado: oh, if you want it to never match any stable branch (not sure why you would make that choice, but whatever) you could just tell it to never match anything starting with stable/ or you could just tell it to only ever match master | 16:11 |
| *** yamamoto has quit IRC | 16:12 | |
| *** hrubi has joined #openstack-infra | 16:12 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/project-config master: Create tempest non-voting gate in kuryr-kubernetes https://review.openstack.org/474993 | 16:12 |
| dmellado | I guess I got it this time, thanks clarkb, fungi ;) | 16:13 |
| dmellado | and of course AJaeger ;) | 16:13 |
| clarkb | fungi: if it is going to fail in check I'd like it to fail on dns lookups :) | 16:13 |
| fungi | clarkb: me too | 16:13 |
| *** sree has quit IRC | 16:14 | |
| *** Apoorva has quit IRC | 16:15 | |
| *** Apoorva has joined #openstack-infra | 16:16 | |
| *** flepied has joined #openstack-infra | 16:17 | |
| *** markus_z has quit IRC | 16:19 | |
| *** Julien-zte has quit IRC | 16:22 | |
| *** ijw has joined #openstack-infra | 16:22 | |
| *** Julien-zte has joined #openstack-infra | 16:23 | |
| *** Julien-zte has quit IRC | 16:23 | |
| *** armax has quit IRC | 16:24 | |
| *** Julien-zte has joined #openstack-infra | 16:24 | |
| *** Julien-zte has quit IRC | 16:24 | |
| *** lucasagomes is now known as lucas-afk | 16:25 | |
| *** tmorin has quit IRC | 16:25 | |
| *** Julien-zte has joined #openstack-infra | 16:25 | |
| *** Julien-zte has quit IRC | 16:25 | |
| *** Julien-zte has joined #openstack-infra | 16:26 | |
| *** Julien-zte has quit IRC | 16:26 | |
| *** Julien-zte has joined #openstack-infra | 16:27 | |
| *** ijw has quit IRC | 16:27 | |
| *** Julien-zte has quit IRC | 16:27 | |
| *** jpich has quit IRC | 16:27 | |
| *** gcb has quit IRC | 16:28 | |
| *** bobh has joined #openstack-infra | 16:29 | |
| *** ykarel has joined #openstack-infra | 16:30 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/project-config master: Create tempest non-voting gate in kuryr-kubernetes https://review.openstack.org/474993 | 16:31 |
| *** liusheng has quit IRC | 16:31 | |
| *** liusheng has joined #openstack-infra | 16:32 | |
| *** efoley has quit IRC | 16:32 | |
| *** dprince has joined #openstack-infra | 16:33 | |
| *** jascott1 has joined #openstack-infra | 16:37 | |
| *** yamamoto has joined #openstack-infra | 16:42 | |
| *** yamamoto has quit IRC | 16:42 | |
| *** Guest85443 has quit IRC | 16:42 | |
| *** trown is now known as trown|lunch | 16:45 | |
| *** Hunner_ is now known as Hunner | 16:47 | |
| *** jascott1 has quit IRC | 16:48 | |
| *** jascott1 has joined #openstack-infra | 16:49 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Sync after writing partition table https://review.openstack.org/475203 | 16:49 |
| *** aimeeu has left #openstack-infra | 16:51 | |
| *** ykarel has quit IRC | 16:52 | |
| *** ltomasbo is now known as ltomasbo|away | 16:52 | |
| *** ralonsoh has quit IRC | 16:53 | |
| *** jascott1 has quit IRC | 16:53 | |
| *** dizquierdo has joined #openstack-infra | 16:53 | |
| *** Swami has joined #openstack-infra | 16:54 | |
| *** chlong has quit IRC | 16:56 | |
| *** tesseract has quit IRC | 16:59 | |
| fungi | infra-root: http://www.openwall.com/lists/oss-security/2017/06/19/1 is exciting... we may want to think about orchestrating reboots, at least for some more sensitive systems, once we confirm kernel updates are applied | 17:01 |
| *** derekh has quit IRC | 17:01 | |
| *** jascott1 has joined #openstack-infra | 17:01 | |
| fungi | if you look at the bottom of https://lists.debian.org/debian-security-announce/2017/maillist.html sorted by date, you can see the (currently 4) advisories debian has issued today. i expect ubuntu is announcing similar ones | 17:02 |
| *** Guest85443 has joined #openstack-infra | 17:02 | |
| *** bexelbie has quit IRC | 17:03 | |
| *** bexelbie has joined #openstack-infra | 17:05 | |
| fungi | ahh, yep, two so far at the bottom of https://lists.ubuntu.com/archives/ubuntu-security-announce/2017-June/date.html are from today (exim and glibc) | 17:05 |
| *** jascott1 has quit IRC | 17:06 | |
| clarkb | wee | 17:06 |
| clarkb | fungi: I'm trying to finish up this gerrit upgrade doc then can help with that | 17:06 |
| fungi | i'm referencing announcement mailing list archives mainly because neither debian nor ubuntu update their vulnerability listing web sites quickly, so aren't showing these yet | 17:07 |
| *** jascott1 has joined #openstack-infra | 17:07 | |
| * fungi is proud of the openstack vmt's website publication automation which happens faster than we can send e-mail copies of the announcements | 17:07 | |
| jeblair | fungi: oh, do we need to send email faster? i can help with that. ;) | 17:08 |
| openstackgerrit | Merged openstack-infra/shade master: Convert remaining nova tests to requests_mock https://review.openstack.org/475190 | 17:08 |
| fungi | jeblair: the thuoght has crossed my mind... if we smashed the release announcements and artifact signing jobs together into an ossa repo job... | 17:08 |
| openstackgerrit | Merged openstack-infra/shade master: Translate final nova calls to REST https://review.openstack.org/475191 | 17:09 |
| openstackgerrit | Merged openstack-infra/shade master: Remove novaclient from shade's dependencies https://review.openstack.org/475192 | 17:09 |
| pabelanger | fungi: I have to run out for an errand now, I am not sure how much help I'll be today | 17:09 |
| jeblair | fungi: unfortunately, i'm getting in a metal tube today, so can't help much with the updates :( | 17:09 |
| fungi | jeblair: but it's nice to be able to have actual vmt member signatures in the announcements we send to mailing lists, so probably not perfect for automating | 17:09 |
| *** jascott1 has quit IRC | 17:10 | |
| fungi | jeblair: pabelanger: you'll be with us in spirit! | 17:10 |
| *** jascott1 has joined #openstack-infra | 17:10 | |
| *** amoralej is now known as amoralej|off | 17:11 | |
| pabelanger | fungi: clarkb: I'm also traveling today too. I expect I'll be around limited this week. As of friday, I am also on PTO for 2 weeks. Back to work on July 10th. | 17:11 |
| fungi | anyway, probably not super-urgent, but i'll keep tabs on ubuntu package situation (my hopes for centos getting kernel updates quickly are not so high) | 17:11 |
| *** tushar1 has joined #openstack-infra | 17:11 | |
| * mordred also travelling today | 17:11 | |
| fungi | i am disappearing from thursday through monday, lots of family coming to visit for a long weekend | 17:11 |
| *** chlong has joined #openstack-infra | 17:11 | |
| *** dhajare has quit IRC | 17:11 | |
| *** ramishra has quit IRC | 17:12 | |
| *** salv-orlando has joined #openstack-infra | 17:12 | |
| *** jpena is now known as jpena|off | 17:13 | |
| clarkb | infra-root https://etherpad.openstack.org/p/gerrit-2.13.-upgrade-steps is ready for review now | 17:14 |
| *** salv-orl_ has joined #openstack-infra | 17:15 | |
| fungi | yay! thanks | 17:18 |
| fungi | also, my d-g change to collect the unbound log is still passing :/ | 17:18 |
| *** rcernin has joined #openstack-infra | 17:18 | |
| *** salv-orlando has quit IRC | 17:18 | |
| clarkb | well if it merges then it will get applied to a lot mroe job runs | 17:19 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Temporarily copy unbound.log in test-results https://review.openstack.org/475402 | 17:19 |
| fungi | clarkb: ^ fixed the copy/paste error you spotted. thanks again | 17:19 |
| *** rbrndt has quit IRC | 17:20 | |
| clarkb | fungi: thinking about ^ more, does the zl ansible executor stuff care if the target is outside of the workspace? I want to say jenkins had a fit if it was | 17:20 |
| clarkb | jeblair: ^ do you know off the top of your head? | 17:20 |
| *** yamahata_ has quit IRC | 17:21 | |
| fungi | ooh, good question | 17:21 |
| mordred | clarkb: I know in v3 we do not care about remote file paths outside of the workspace path | 17:21 |
| mordred | in 2.5 I _doubt_ we'd care, becuase caring would take a lot of effort | 17:22 |
| *** vhosakot has joined #openstack-infra | 17:22 | |
| *** rodrigods has quit IRC | 17:23 | |
| *** rodrigods has joined #openstack-infra | 17:23 | |
| jeblair | mordred: i agree. clarkb: i *think* that's fine. | 17:24 |
| *** smarcet_ has joined #openstack-infra | 17:24 | |
| clarkb | cool I have +2'd | 17:24 |
| clarkb | if others can review I am happy to babysit and keep an eye out for that issue | 17:25 |
| *** mhickey has quit IRC | 17:25 | |
| fungi | i went skimming macros.yaml and all sources in other scp publishers are relative (probably owing to jenkins expectations more than anything) | 17:26 |
| mordred | clarkb: I have also +2'd it - today is a travel day so I'm not sure I can help watch it | 17:26 |
| fungi | so i agree we don't have any precedent | 17:26 |
| fungi | maybe we see what we catch with the devstack-gate change first rather than risk breaking most other jobs in our ci | 17:27 |
| *** dhajare has joined #openstack-infra | 17:27 | |
| *** flepied has quit IRC | 17:27 | |
| fungi | i've wip'd it for now | 17:28 |
| *** xarses has joined #openstack-infra | 17:28 | |
| mordred | clarkb, Shrews: either of you want to look at the user-data base64 encoding patches: https://review.openstack.org/#/c/475385 https://review.openstack.org/#/c/475424 | 17:29 |
| *** jcoufal_ has joined #openstack-infra | 17:30 | |
| tushar1 | clarkb : Hi | 17:30 |
| *** jcoufal_ has quit IRC | 17:30 | |
| tushar1 | clarkb : In last week I discussed the gearman plugin issue which I am facing in third party CI | 17:30 |
| smarcet_ | fungi : afternoon i am trying to set up locally this https://github.com/openstack/openstack-manuals, bc need to migrate google search to swiftype search | 17:30 |
| mordred | clarkb, Shrews: the user who reported the bug has verified that ps2 of the first patch fixed his issue | 17:30 |
| *** jcoufal_ has joined #openstack-infra | 17:31 | |
| *** rossella_ has joined #openstack-infra | 17:31 | |
| smarcet_ | fungi: basically i was crawling at docs but not found any clear instructions on how to set up the site locally, should i check on puppet script ? | 17:31 |
| fungi | smarcet_: are you running into challenges with the build instructions in the readme? | 17:31 |
| tushar1 | clarkb : I checked jenkins logs, I didnt found any error message there | 17:31 |
| *** sambetts is now known as sambetts|afk | 17:32 | |
| fungi | smarcet_: oh, i can get you a link to the apache vhost we use for docs.openstack.org if that helps | 17:32 |
| smarcet_ | fungi: well from waht i understand the build instructions on readme it just to build up the doc, just wana to host the site to change something on header | 17:32 |
| smarcet_ | fungi: yes that would help thx | 17:32 |
| tushar1 | clarkb : the error message in zuul.log is - job is not registered with gearman , | 17:32 |
| *** jcoufal has quit IRC | 17:32 | |
| fungi | smarcet_: the template is at http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/docs.vhost.erb | 17:33 |
| smarcet_ | fungi: thx! | 17:33 |
| tushar1 | clarkb : my question here- is there any dependency of version of zuul,gearman plugin and jenkins? | 17:33 |
| fungi | smarcet_: any time! let us know if you have other questions. the search feature would be useful to have working again | 17:34 |
| tushar1 | currently our setup is having zuul 2.1.0 jenkins -1.642 and gearman plugin with 0.1.1 | 17:34 |
| smarcet_ | fungi: sure really appreciate it :) i will thx! | 17:34 |
| *** JudeC has joined #openstack-infra | 17:34 | |
| *** jamesmcarthur has quit IRC | 17:35 | |
| clarkb | tushar1: there shouldn't be, but we don't run jenkins anymore so I'm not sure if latest jenkins still works | 17:35 |
| tushar1 | clarkb : okay | 17:36 |
| fungi | tushar1: normally the jenkins gearman plugin will register jobs into gearman when a new slave is attached to the master, but you should be able to trigger it for any jobs registered to labels/slaves in your jenkins configuration by disabling and reenabling the plugin | 17:36 |
| *** florianf has quit IRC | 17:36 | |
| fungi | tushar1: also something like `echo "status" | nc localhost 4730 | grep somejob` on your zuul server (assuming the gearman service is local to zuul) should show you what labels are registered for somejob | 17:37 |
| smarcet_ | fungi: btw i dont see it the node docs.openstack.org here https://github.com/openstack-infra/system-config/blob/master/manifests/site.pp | 17:38 |
| fungi | tushar1: an example from ours... http://paste.openstack.org/show/613125/ | 17:38 |
| fungi | smarcet_: it's a vhost running on files01.openstack.org, so the entrypoint is http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n657 | 17:39 |
| smarcet_ | ok got it thx | 17:39 |
| tushar1 | fungi : clarkb: yes `echo "status" | nc localhost 4730 | grep somejob` this command should show the output, but in our case it is not showing any job registered with is command | 17:39 |
| fungi | tushar1: oh, there's also a semi-recent change in jenkins which may break the gearman plugin if you don't apply a workaround... i'll see if i can find the mailing list thread about that | 17:41 |
| fungi | smarcet_: also if you do a dns lookup of docs.openstack.org you'll see that it's a cname for files.openstack.org which is in turn a cname for files01.openstack.org (in case you want to know how to figure this out for similar situations in the future) | 17:42 |
| *** dhinesh has joined #openstack-infra | 17:42 | |
| tushar1 | fungi : okk | 17:42 |
| *** yamamoto has joined #openstack-infra | 17:42 | |
| smarcet_ | fungi: ok thx for the info :) | 17:43 |
| fungi | tushar1: nevermind, i was thinking of the change which makes jobs running in jenkins not receive zuul envvars from the gearman plugin, but that wouldn't impact job registration into gearman | 17:43 |
| fungi | tushar1: are your jenkins and zuul/gearman services running on the same server or is there a chance you have a firewall blocking some traffic between the jenkins server and the zuul/gearman server (particularly 4730/tcp)? | 17:44 |
| clarkb | mordred: is it bad that I think the novaclient logic is buggy too? Shouldn't we do the safe_encode if python2 or python3 and not just if python2? | 17:45 |
| tushar1 | fungi : jenkins and zuul/gearman running on same server | 17:45 |
| tushar1 | thats why there is no chance of firewall blocking | 17:45 |
| clarkb | mordred: mostly worried that python3 case only works with utf8 and it could be something different, not a major problem for those of us on linux but ya | 17:45 |
| fungi | tushar1: can you paste the output of `echo status | nc localhost 4730` to http://paste.openstack.org/ and let us know the url it gives you? | 17:46 |
| tushar1 | fungi : http://paste.openstack.org/show/613126/ | 17:47 |
| *** vhosakot has quit IRC | 17:47 | |
| *** dhajare has quit IRC | 17:48 | |
| *** yamamoto has quit IRC | 17:48 | |
| clarkb | mordred: I left comments not really sure they are worth -1's as they are improvements in the commonest case | 17:50 |
| mordred | clarkb: thanks | 17:50 |
| smarcet_ | fungi: so i have another question , here https://github.com/openstack/openstack-manuals/blob/master/www/index.html i am seeing code block that seems to me jinga2 python engine template syntax, is that correct ? | 17:50 |
| fungi | tushar1: i see a build:noop-check-communication in the list there, which i believe indicates the plugin is communicating | 17:51 |
| smarcet_ | fungi: i mean its seems that its using a template engine, but no sure which one | 17:51 |
| fungi | tushar1: also the set_description and stop imply that as well, i think | 17:51 |
| tushar1 | fungi : yes it is showing 1 job build:noop-check-communication | 17:52 |
| smarcet_ | smarcet: also i am a bit lost , bc the www folder does not seems to be hosted under a python uWSGI or something similar | 17:52 |
| fungi | smarcet_: these are probably questions for the docs team, but it does look like at least something in there is using jinja2 according to https://github.com/openstack/openstack-manuals/blob/master/test-requirements.txt#L10 | 17:53 |
| smarcet_ | fungi: got it thx | 17:53 |
| tushar1 | but I cant see all the jobs which are there in projects.yaml | 17:54 |
| *** tosky__ has joined #openstack-infra | 17:54 | |
| tushar1 | fungi : let me share my projects.yaml | 17:54 |
| fungi | smarcet_: as for uwsgi or anything, we just build static content from that source tree and serve the result | 17:54 |
| smarcet_ | fungi: oh got it thx for the info :) | 17:54 |
| openstackgerrit | Darragh Bailey proposed openstack/python-jenkins master: Migration to using requests https://review.openstack.org/205637 | 17:55 |
| *** tosky has quit IRC | 17:55 | |
| *** tosky__ is now known as tosky | 17:56 | |
| fungi | smarcet_: you can also browse the file tree via (public, global) afs if you're able to get the client installed: https://docs.openstack.org/infra/system-config/afs.html#client-configuration | 17:56 |
| smarcet_ | fungi: oh cool thx :) | 17:56 |
| fungi | tushar1: if it's just a misconfiguration, then odds are you're missing a node assignment in your job definitions or your node assignment doesn't match the name or label of any slave registered on your jenkins master | 17:58 |
| fungi | tushar1: but yes seeing a sample of your jjb configuration can't hurt | 17:58 |
| tushar1 | fungi : this is part of the projects.yaml - http://paste.openstack.org/show/613128/ | 17:59 |
| fungi | tushar1: and do any of your slaves in jenkins have a name or label of d-p-c or d-p-c-fc | 18:00 |
| fungi | ? | 18:00 |
| *** camunoz has quit IRC | 18:00 | |
| *** yamahata_ has joined #openstack-infra | 18:00 | |
| tushar1 | fungi : yes all the slave starts with d-p-c or d-p-c-fc, appending the ID after it | 18:01 |
| tushar1 | for example d-p-c-1234 | 18:01 |
| *** baoli has quit IRC | 18:01 | |
| *** cschwede has quit IRC | 18:01 | |
| *** SumitNaiksatam has quit IRC | 18:02 | |
| *** baoli has joined #openstack-infra | 18:02 | |
| fungi | tushar1: well, the node id has to be an exact match for the node name or a node label attached to the node. pull up the configuration for one of those nodes in the jenkins webui and see what the "label" field says on it | 18:02 |
| *** cschwede has joined #openstack-infra | 18:02 | |
| *** baoli has quit IRC | 18:02 | |
| fungi | i would expect to see there be a label on d-p-c-1234 which is just "d-p-c" if it's set up correctly for this | 18:02 |
| *** otherwiseguy has quit IRC | 18:03 | |
| *** flepied has joined #openstack-infra | 18:04 | |
| *** otherwiseguy has joined #openstack-infra | 18:05 | |
| *** zigo has quit IRC | 18:05 | |
| *** baoli has joined #openstack-infra | 18:06 | |
| tushar1 | fungi: https://bugs.launchpad.net/gearman-plugin/+bug/1353891 | 18:08 |
| openstack | Launchpad bug 1353891 in Jenkins Gearman Plugin "Jobs generated by jenkins job builder are not registered" [Undecided,Fix released] - Assigned to Khai Do (zaro0508) | 18:08 |
| dhellmann | pabelanger, clarkb, jeblair, fungi: the docs team is ready to move ahead with the migration of content into project repositories. This patch to add the new unified doc build job is one of the first steps we need. Can you add it to your review lists, please? https://review.openstack.org/#/c/471881/ | 18:09 |
| tushar1 | fungi : ^ In this bug it is discussed about node name , node label ,target and job name | 18:09 |
| *** zigo has joined #openstack-infra | 18:09 | |
| *** trown|lunch is now known as trown | 18:10 | |
| *** netsin has joined #openstack-infra | 18:10 | |
| tushar1 | Fungi : I think node label and target name should be same | 18:10 |
| dhellmann | having all of these cross-repo depends-on in the migration patches is causing zuul to not want to run tests, so I'd like to get some of the early dependencies out of the way asap | 18:10 |
| *** salv-orlando has joined #openstack-infra | 18:10 | |
| tushar1 | which is d-p-c in our case | 18:10 |
| tushar1 | fungi : correct if I am wrong | 18:11 |
| *** jascott1 has quit IRC | 18:11 | |
| *** mriedem1 has joined #openstack-infra | 18:11 | |
| *** thorst_ has joined #openstack-infra | 18:12 | |
| *** rguillebert_ has joined #openstack-infra | 18:12 | |
| *** andrewbogott_ has joined #openstack-infra | 18:12 | |
| *** scottda_ has joined #openstack-infra | 18:12 | |
| *** evgenyl_ has joined #openstack-infra | 18:12 | |
| *** rajinir_ has joined #openstack-infra | 18:12 | |
| *** cargonza_ has joined #openstack-infra | 18:12 | |
| *** pvaneck has joined #openstack-infra | 18:12 | |
| *** prateek_ has joined #openstack-infra | 18:13 | |
| *** electrical_ has joined #openstack-infra | 18:13 | |
| *** jascott1 has joined #openstack-infra | 18:14 | |
| *** camunoz has joined #openstack-infra | 18:14 | |
| *** mugsie_ has joined #openstack-infra | 18:14 | |
| *** dosaboy_ has joined #openstack-infra | 18:15 | |
| *** Adri2000_ has joined #openstack-infra | 18:15 | |
| *** prateek_ has quit IRC | 18:15 | |
| *** pradk has quit IRC | 18:17 | |
| *** hrubi_ has joined #openstack-infra | 18:18 | |
| *** kencjohnston_ has joined #openstack-infra | 18:18 | |
| *** mnencia_ has joined #openstack-infra | 18:18 | |
| *** patriciadomin_ has joined #openstack-infra | 18:18 | |
| *** nhandler_ has joined #openstack-infra | 18:18 | |
| *** gaelL_ has joined #openstack-infra | 18:18 | |
| *** mrunge_ has joined #openstack-infra | 18:19 | |
| *** salv-orl_ has quit IRC | 18:19 | |
| *** hrubi has quit IRC | 18:19 | |
| *** prateek has quit IRC | 18:19 | |
| *** mtanino has quit IRC | 18:19 | |
| *** mriedem has quit IRC | 18:19 | |
| *** thorst has quit IRC | 18:19 | |
| *** nicolasbock has quit IRC | 18:19 | |
| *** electrical has quit IRC | 18:19 | |
| *** scottda has quit IRC | 18:19 | |
| *** rguillebert has quit IRC | 18:19 | |
| *** rockstar has quit IRC | 18:19 | |
| *** andrewbogott has quit IRC | 18:19 | |
| *** kencjohnston has quit IRC | 18:19 | |
| *** evgenyl has quit IRC | 18:19 | |
| *** cargonza has quit IRC | 18:19 | |
| *** patriciadomin has quit IRC | 18:19 | |
| *** nhandler has quit IRC | 18:19 | |
| *** jamielennox has quit IRC | 18:19 | |
| *** mugsie has quit IRC | 18:19 | |
| *** mrmartin has quit IRC | 18:19 | |
| *** d0ugal has quit IRC | 18:19 | |
| *** dosaboy has quit IRC | 18:19 | |
| *** nt has quit IRC | 18:19 | |
| *** michaelxin has quit IRC | 18:19 | |
| *** jiggangalator has quit IRC | 18:19 | |
| *** krtaylor has quit IRC | 18:19 | |
| *** clayton has quit IRC | 18:19 | |
| *** mnencia has quit IRC | 18:19 | |
| *** rajinir has quit IRC | 18:19 | |
| *** lennyb has quit IRC | 18:19 | |
| *** mrunge has quit IRC | 18:19 | |
| *** Adri2000 has quit IRC | 18:19 | |
| *** fungi has quit IRC | 18:19 | |
| *** gaelL has quit IRC | 18:19 | |
| *** mnencia_ is now known as mnencia | 18:19 | |
| *** andrewbogott_ is now known as andrewbogott | 18:19 | |
| *** rguillebert_ is now known as rguillebert | 18:19 | |
| *** electrical_ is now known as electrical | 18:19 | |
| *** scottda_ is now known as scottda | 18:19 | |
| *** evgenyl_ is now known as evgenyl | 18:19 | |
| *** cargonza_ is now known as cargonza | 18:19 | |
| *** mriedem1 is now known as mriedem | 18:19 | |
| *** dtantsur is now known as dtantsur|afk | 18:19 | |
| *** rajinir_ is now known as rajinir | 18:20 | |
| *** clayton has joined #openstack-infra | 18:20 | |
| *** nt has joined #openstack-infra | 18:21 | |
| *** jamielennox has joined #openstack-infra | 18:21 | |
| *** hrubi_ has quit IRC | 18:23 | |
| *** onovy has quit IRC | 18:23 | |
| *** lbragstad has quit IRC | 18:24 | |
| *** onovy has joined #openstack-infra | 18:24 | |
| *** nicolasbock has joined #openstack-infra | 18:24 | |
| *** hrubi has joined #openstack-infra | 18:24 | |
| *** fungi has joined #openstack-infra | 18:25 | |
| *** krtaylor has joined #openstack-infra | 18:25 | |
| *** d0ugal has joined #openstack-infra | 18:25 | |
| *** lennyb has joined #openstack-infra | 18:27 | |
| fungi | fun, i think i got hit by a netsplit or something else went terrible between my shell server in rackspace dfw and whatever freenode server i was using | 18:27 |
| *** toabctl has quit IRC | 18:27 | |
| clarkb | looks like a fairly large netsplit | 18:27 |
| fungi | dhellmann: thanks, starred. i'm still taking a last look through the spec too\ | 18:27 |
| *** SumitNaiksatam has joined #openstack-infra | 18:27 | |
| *** bobh has quit IRC | 18:28 | |
| fungi | i would expect to see there be a label on d-p-c-1234 which is just "d-p-c" if it's set up correctly for this | 18:28 |
| fungi | tushar1: ^ | 18:28 |
| fungi | tushar1: i read your subsequent comments from the channel log, and it doesn't look like you confirmed what the actual label is on the slaves in your jenkins master | 18:29 |
| *** flepied has quit IRC | 18:30 | |
| fungi | if neither the slave name nor label matches the node pattern set in the job definition, the gearman plugin won't register it | 18:30 |
| fungi | it doesn't do substring/prefix matching or anything fancy like that | 18:30 |
| *** toabctl has joined #openstack-infra | 18:31 | |
| *** michaelxin has joined #openstack-infra | 18:31 | |
| tushar1 | fungi : I verified from Jenkins UI , the label is node label is d-p-c | 18:32 |
| *** prometheanfire has left #openstack-infra | 18:33 | |
| fungi | tushar1: okay, rules that out | 18:33 |
| clarkb | fungi: if you are out thursday maybe we want to aim for either tomorrow after meeting or wednesday upgrade of review-dev.openstack.org? I'd like another root around just to sanity check things with me since this is dry run for production and sounds like most other roots will be out and about this week | 18:34 |
| *** smarcet_ has quit IRC | 18:35 | |
| fungi | tushar1: next thing to keep in mind is that the plugin only registers jobs when a new slave is added to your jenkins master, or when the plugin itself is reenabled. the gearman queue is ephemeral and will be emptied when zuul/gearman is restarted so you may need to disable and reenable the gearman plugin or delete and replace a slave to get registrations to show up afterward | 18:35 |
| *** camunoz has quit IRC | 18:35 | |
| fungi | clarkb: i'm good with either or both of those times | 18:36 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Add scenario jobs to tripleo repositories https://review.openstack.org/475482 | 18:36 |
| clarkb | fungi: does 2030UTC work? assuming no TC meeting tomorrow after 1900UTC infra meeting | 18:36 |
| clarkb | that should give anyone else that wants to read over the etherpad a chance to do so after meeting before we start | 18:37 |
| fungi | there is a tc meeting tomorrow | 18:37 |
| fungi | we're going to go over the cross-project goals options and distill them to a subset to propose to the ml | 18:37 |
| clarkb | ah ok, then lets go for after tc meeting, 2115? | 18:37 |
| fungi | wfm | 18:37 |
| tushar1 | fungi : new slaves are added to jenkins and also I tried to re-enabling the gearman from Jenkins UI | 18:37 |
| clarkb | tushar1: is it possible that zuul and jenkins are atlking to different gearman servers? | 18:38 |
| tushar1 | I there any other way to restart the gearman? | 18:38 |
| dhellmann | fungi : thanks | 18:38 |
| fungi | clarkb: i don't have anything on my schedule between the end of the tc meeting (21:00) and the start of tc office hours (01:00) | 18:38 |
| fungi | so should be plenty of time | 18:38 |
| clarkb | fungi: sounds good, thanks | 18:39 |
| tushar1 | fungi : both are talking with localhost and both are on same server | 18:39 |
| fungi | clarkb: connectivity test no-op job is showing up in the local gearman service, just no jobs tied to actual slaves/labels apparently | 18:40 |
| *** gomarivera has quit IRC | 18:40 | |
| *** rbrndt has joined #openstack-infra | 18:41 | |
| fungi | tushar1: i'm pretty much out of ideas. as clarkb mentioned we haven't used jenkins or the jenkins gearman plugin for nearly a year at this point, so you might need suggestions from someone who is still actively doing that... you might try asking in #openstack-third-party-ci or, failing that, on the openstack-infra@lists.openstack.org mailing list | 18:41 |
| clarkb | tushar1: fungi the only other thing I can think of is disable the job and enable the job manually to force reregistration | 18:42 |
| clarkb | also offline the slave then online the slave | 18:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Try all the detected encodings, not just one https://review.openstack.org/475424 | 18:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: base64 encode user_data sent to create server https://review.openstack.org/475385 | 18:43 |
| *** gomarivera has joined #openstack-infra | 18:43 | |
| *** gomarivera has quit IRC | 18:43 | |
| mordred | clarkb: ^^ how does that look? | 18:43 |
| *** gomarivera has joined #openstack-infra | 18:43 | |
| clarkb | mordred: close looks like you'll get typerror if the input is bytes but you should just skip the encode | 18:44 |
| clarkb | mordred: maybe _safe_encode should just return instead of raising? | 18:44 |
| *** bhavik1 has quit IRC | 18:44 | |
| *** smarcet has joined #openstack-infra | 18:45 | |
| tushar1 | fungi : Thanks , will connect to #openstack-third-party-ci | 18:45 |
| mordred | clarkb: well - if we can't encode, then the API call will not do what the user is expecting | 18:45 |
| *** pcaruana has quit IRC | 18:46 | |
| mordred | clarkb: oh - you're thining return userdata as-is? | 18:46 |
| tushar1 | lets see , I may get some clue there | 18:46 |
| clarkb | mordred: ya, because it could be bytes already | 18:46 |
| mordred | nod | 18:46 |
| clarkb | if it is bytes arleady then we just convert to base64 and are done | 18:46 |
| clarkb | rather than doing utf8 -> bytes -> base64 | 18:46 |
| *** salv-orlando has quit IRC | 18:46 | |
| tushar1 | clarkb : you mean to disable and enable the job manually from jenkins UI? | 18:47 |
| *** salv-orlando has joined #openstack-infra | 18:48 | |
| clarkb | tushar1: yes, that should force job registration | 18:48 |
| clarkb | tushar1: you can also tcpdump -i any -A port 4730 when you do that to see if any packets are sent to register jobs | 18:48 |
| clarkb | tushar1: those packets should contain CAN_DO in them | 18:48 |
| *** camunoz has joined #openstack-infra | 18:48 | |
| *** dkehn_ has joined #openstack-infra | 18:48 | |
| tushar1 | clarkb : okay , I am trying that as well | 18:48 |
| *** dhajare has joined #openstack-infra | 18:49 | |
| *** jamesmcarthur has joined #openstack-infra | 18:50 | |
| *** dkehn has quit IRC | 18:51 | |
| *** kgiusti has quit IRC | 18:52 | |
| *** kgiusti1 has joined #openstack-infra | 18:52 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Try all the detected encodings, not just one https://review.openstack.org/475424 | 18:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: base64 encode user_data sent to create server https://review.openstack.org/475385 | 18:53 |
| mordred | clarkb: ok. I think that should do it | 18:53 |
| mordred | clarkb: I believe things might go wrong if thesystem encoding is EBCIDIC - but honestly I don't care about that | 18:53 |
| *** Guest85443 has quit IRC | 18:56 | |
| mwhahaha | i have a question, is there a cirros image available locally as part of the infra mirrors? | 18:56 |
| clarkb | mordred: first change lgtm, second has a logic thing | 18:57 |
| clarkb | mwhahaha: no, we bake the cirros image onto our test node rootfs instead | 18:57 |
| *** rossella_ has quit IRC | 18:58 | |
| mwhahaha | clarkb: interesting, where is it on the image? | 18:58 |
| clarkb | mordred: I think you might want to have if binary: then return else if not text then raise, then have the fallthrough outer block do the magic of converting text to bytes regardless of encoding | 18:58 |
| *** rlandy is now known as rlandy|brb | 18:58 | |
| clarkb | mwhahaha: ~/cache/files, but if you use devstack-gate it moves them to the $BASE/cache/files because we try to stick $BASE on an ephemeral drive with more room if possible | 18:59 |
| *** tushar1 has quit IRC | 19:00 | |
| mordred | clarkb: yes - you're absolutely right | 19:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Try all the detected encodings, not just one https://review.openstack.org/475424 | 19:01 |
| mordred | mwhahaha, clarkb: http://paste.openstack.org/show/613107/ | 19:02 |
| mordred | that question got asked this morning too | 19:02 |
| clarkb | mordred: I think latest patchset still has the problem | 19:02 |
| mwhahaha | i was asking cause in tripleo we're downloading it as part of our validation, just wondering if we could stop | 19:02 |
| mordred | mwhahaha: yah - definitely :) | 19:02 |
| clarkb | mordred: basically remove the isinstance for six.text_type entirely | 19:03 |
| clarkb | mordred: wow thats a lot more in the files/ dir than I expected | 19:03 |
| mordred | clarkb: oh - and just always do decode/encode even if it's already a string type that can directly encode? | 19:03 |
| fungi | i'm going to go get something to eat... back in a while | 19:03 |
| *** shardy has joined #openstack-infra | 19:04 | |
| clarkb | mordred: not sure I understand that. THe problem with the code now is the fallthrough block with all the encodings will never run | 19:04 |
| clarkb | mordred: because either you are binary, text, or neither | 19:04 |
| *** dizquierdo has quit IRC | 19:05 | |
| clarkb | mordred: and all three of those cases have their own one line blocks to raise/return | 19:05 |
| mordred | unfortunately not | 19:05 |
| mordred | six.string_types | 19:05 |
| mordred | Possible types for text data. This is basestring() in Python 2 and str in Python 3. | 19:05 |
| mordred | six.text_type | 19:05 |
| mordred | Type for representing (Unicode) textual data. This is unicode() in Python 2 and str in Python 3. | 19:05 |
| clarkb | mordred: ya but you are checking if not that | 19:05 |
| clarkb | mordred: which means you will match every other possible object | 19:05 |
| clarkb | basicaly everything below line 5471 will neverrun | 19:06 |
| clarkb | I think you just want to remove the if type == text check and return and have it fall through | 19:06 |
| mordred | yah - and will raise if you're not string_types - but then if you are a basestring in python2, the rest will run | 19:06 |
| mordred | but if you're a unicode in python2, you'll just encode to utf-8 and be done with it | 19:06 |
| mordred | hrm. nope - I see it now | 19:07 |
| clarkb | if you want to optimize out the common utf8 case then you wrap the type == text in a try except and only return on success | 19:07 |
| *** dhajare has quit IRC | 19:08 | |
| *** alex_xu has quit IRC | 19:11 | |
| *** alex_xu has joined #openstack-infra | 19:11 | |
| Shrews | oh, didn't see the encoding talk for shade here. mordred, should I remove +A? | 19:13 |
| *** baoli has quit IRC | 19:14 | |
| *** baoli has joined #openstack-infra | 19:14 | |
| * Shrews lets it ride | 19:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: base64 encode user_data sent to create server https://review.openstack.org/475385 | 19:21 |
| mordred | clarkb, Shrews: ^^ - that shoudl take care of the +A ... also, I squashed it | 19:22 |
| *** jamesmcarthur has quit IRC | 19:22 | |
| *** jascott1 has quit IRC | 19:23 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Temporarily copy unbound.log in cleanup_host https://review.openstack.org/475403 | 19:24 |
| clarkb | fungi: ^ woot now we can troll logstash for incidences of that error against devstack jobs then check against the unbound log | 19:25 |
| clarkb | probably wait for end of day just to make sure the jobs ran with d-g update | 19:26 |
| *** bobh has joined #openstack-infra | 19:28 | |
| *** nicolasbock has quit IRC | 19:28 | |
| *** armax has joined #openstack-infra | 19:29 | |
| *** baoli has quit IRC | 19:29 | |
| *** rpioso has joined #openstack-infra | 19:30 | |
| *** dave-mccowan has quit IRC | 19:30 | |
| *** baoli has joined #openstack-infra | 19:30 | |
| *** mhickey has joined #openstack-infra | 19:32 | |
| clarkb | fungi: (message:"failed: Temporary failure in name resolution." OR message:"Temporary failure resolving" OR message:"wget: unable to resolve") AND tags:"console" is the rough query I am using over the last 6 hours to keep an eye on that | 19:34 |
| clarkb | fungi: seems like name resolution errors are not all consistent :/ | 19:34 |
| *** baoli has quit IRC | 19:34 | |
| clarkb | it definitely seems to be .orgs though not seeing any thing for eg github | 19:35 |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Don't remove top-container element for server REST API calls https://review.openstack.org/475501 | 19:39 |
| *** rlandy|brb is now known as rlandy | 19:39 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Test with bonded public interface https://review.openstack.org/385562 | 19:42 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Move PublicVirtualFixedIPs to ssl-public-vip.yaml https://review.openstack.org/475419 | 19:42 |
| *** makowals has quit IRC | 19:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Use Event in streaming test https://review.openstack.org/475505 | 19:43 |
| *** camunoz has quit IRC | 19:45 | |
| *** dave-mccowan has joined #openstack-infra | 19:46 | |
| *** dkehn_ is now known as dkehn | 19:50 | |
| *** yland has joined #openstack-infra | 19:51 | |
| *** rockyg has joined #openstack-infra | 19:54 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Ping test horizon https://review.openstack.org/472758 | 19:55 |
| *** makowals has joined #openstack-infra | 19:58 | |
| *** jtomasek_ has joined #openstack-infra | 19:59 | |
| *** jascott1 has joined #openstack-infra | 20:00 | |
| *** smarcet has quit IRC | 20:00 | |
| *** camunoz has joined #openstack-infra | 20:01 | |
| *** baoli has joined #openstack-infra | 20:01 | |
| *** sshnaidm is now known as sshnaidm|afk | 20:05 | |
| *** mrmartin has joined #openstack-infra | 20:09 | |
| *** mtanino has joined #openstack-infra | 20:09 | |
| *** rockstar has joined #openstack-infra | 20:09 | |
| *** jtomasek_ has quit IRC | 20:10 | |
| clarkb | fungi: http://logs.openstack.org/69/469669/5/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/b63e07b/logs/devstacklog.txt.gz#_2017-06-19_20_01_43_395 there is a .com. Also http://logs.openstack.org/69/469669/5/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/b63e07b/logs/unbound_log.txt.gz is basically empty :/ | 20:13 |
| hwoarang | dirk: ping? could I pm you about a dib issue or perhaps you could join #openstack-dib as well? :) | 20:13 |
| clarkb | we can increase the unbound verbosity. Is it possible that we are not talking to unbound though? | 20:14 |
| clarkb | I think we may also want a d-g change to grab the /etc/resolv.conf | 20:14 |
| clarkb | and maybe netstat -lnp | grep 53 | 20:15 |
| * clarkb works on that | 20:15 | |
| *** gouthamr has quit IRC | 20:15 | |
| *** JudeC has quit IRC | 20:16 | |
| *** flepied has joined #openstack-infra | 20:16 | |
| mordred | clarkb: fwiw, netstat -lnp | grep 53 does show a listenin unbound on a node I happen to have held | 20:18 |
| *** xyang1 has quit IRC | 20:18 | |
| clarkb | mordred: ya for me too, but maybe it is rogue dhcp or something breaking in some cases? | 20:18 |
| clarkb | at this point I think we need all the infos | 20:19 |
| mordred | clarkb: agree | 20:19 |
| *** jcoufal_ has quit IRC | 20:19 | |
| mordred | clarkb: I don't have a /var/log/unbound.log - but this is an old node | 20:20 |
| *** dimtruck is now known as zz_dimtruck | 20:20 | |
| *** shardy has quit IRC | 20:21 | |
| *** hasharAway has quit IRC | 20:21 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Save additional DNS info https://review.openstack.org/475520 | 20:21 |
| clarkb | that captures resolv.conf and ss -lntup | grep ':53' output | 20:22 |
| *** xyang1 has joined #openstack-infra | 20:22 | |
| mordred | clark: Jun 19 20:21:13 ubuntu unbound[24632]: /etc/unbound/unbound.conf.d/unbound-logging.conf:1: error: syntax error | 20:22 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: Make the Ironic multinode grenade job voting https://review.openstack.org/475521 | 20:22 |
| clarkb | mordred: ya I think that got fixed by ianw yesterday | 20:23 |
| clarkb | mordred: there was issue with missing server: directive in the conf file I added | 20:23 |
| mordred | oh - gotcha | 20:23 |
| clarkb | the man page made it easy to miss that (its my excuse) | 20:23 |
| *** bobh has quit IRC | 20:24 | |
| clarkb | also I wonder if since these unbound servers are configured to be forwarding is NXDOMAIN (or similar) would be treated as an error? | 20:24 |
| *** bobh has joined #openstack-infra | 20:24 | |
| mordred | clarkb: I just tried doing a lookup on a thing that doesn't exist and it did not log | 20:25 |
| clarkb | basically maybe google/opends will mask the actual error from unbound and we'll get a situation normal that name doesn't exist | 20:25 |
| clarkb | whereas if we were recursing we'd get the actual error of either timeout or server otherwise derped | 20:25 |
| *** armax has quit IRC | 20:28 | |
| mordred | clarkb: ok - from some poking - I can't get it to make that config file at all | 20:29 |
| mordred | clarkb: but if I set use-syslog: yes | 20:29 |
| fungi | clarkb: i noticed earlier today that at least one of the "broken package mirror" er queries was inadvertently matching on jobs suffering from dns failures too | 20:29 |
| mordred | clarkb: and verbosity 3 - I can definitely get info about what's going on | 20:29 |
| mordred | clarkb: but it still doens't log errors about NXDOMAIN at verbosity 0 | 20:29 |
| clarkb | fungi: ya I've got my own query I am slowly modifying as I find extra things that match | 20:29 |
| *** lbragstad has joined #openstack-infra | 20:30 | |
| clarkb | mordred: can you change your forwarding resolver to be some non existant IP and see if that results in errors logged? | 20:30 |
| *** gardlt has joined #openstack-infra | 20:32 | |
| mordred | clarkb: ok. so - thing number one - unbound can't create the log file - it doesn't have permissions | 20:32 |
| mordred | clarkb: if we touch /var/log/unbound.log first then chown it to unbound, unbound can log to that file | 20:32 |
| clarkb | mordred: that should be handled by the dib build, I touch then chown | 20:32 |
| mordred | clarkb: ok - so that's probablyfine on newer nodes then | 20:32 |
| clarkb | and we get contents in the log file above, its just the line saying "I've started" | 20:32 |
| * mordred tries resolver | 20:32 | |
| clarkb | ya pretty sure that bit works | 20:32 |
| *** markvoelker has joined #openstack-infra | 20:33 | |
| fungi | i'm happy to write/review a change raising the verbosity level, but will need another image update to go into effect | 20:33 |
| openstackgerrit | Mikhail S Medvedev proposed openstack/diskimage-builder master: DNM, Revert "Sync after writing partition table" https://review.openstack.org/475524 | 20:34 |
| *** smarcet has joined #openstack-infra | 20:34 | |
| mordred | clarkb: hah | 20:34 |
| mordred | clarkb: root@ubuntu-xenial-rax-ord-9191322:~# host asf.blerg.org;; connection timed out; no servers could be reached | 20:34 |
| mordred | clarkb: produced no logging | 20:34 |
| mordred | clarkb: that's after settingthe forwarders to bong addresses | 20:35 |
| clarkb | mordred: yay I mean :( | 20:35 |
| clarkb | mordred: the man page says that verosity: 0 is errors only but maybe that doesn't count as an error | 20:35 |
| mordred | yah - I'm trying different verbosities now | 20:35 |
| *** zz_dimtruck is now known as dimtruck | 20:36 | |
| mordred | clarkb: no useful logging at all | 20:38 |
| mordred | clarkb: even on debug level | 20:38 |
| *** thorst_ has quit IRC | 20:39 | |
| mordred | clarkb: it logs various things- just none to provide useful information - either for NXDOMAI or for bad-server | 20:39 |
| *** JudeC has joined #openstack-infra | 20:40 | |
| *** thorst has joined #openstack-infra | 20:41 | |
| *** chlong has quit IRC | 20:41 | |
| mordred | clarkb: http://paste.openstack.org/show/613136/ | 20:42 |
| *** gouthamr has joined #openstack-infra | 20:43 | |
| clarkb | mordred: so looks like info might be what we want | 20:43 |
| clarkb | I worry that that log file will be huge in a devstack gate run | 20:43 |
| clarkb | even though we minimize access to the external world | 20:43 |
| mordred | clarkb: info doesn't have anything | 20:43 |
| clarkb | well it has the no data answer | 20:44 |
| clarkb | which is potentially what is happenign to us if google/openddns can't get an answer upstream of us | 20:44 |
| clarkb | fungi: in good news our logs file system is increasing its free space | 20:45 |
| *** thorst has quit IRC | 20:45 | |
| mordred | clarkb: so - you'd THINK that setting verbosity to 2 would get you all of those info lins | 20:46 |
| mordred | clarkb: but you'd be wrong | 20:46 |
| clarkb | mordred: 3 is info then? | 20:46 |
| mordred | http://paste.openstack.org/show/613137/ | 20:46 |
| mordred | clarkb: that's whatyou get with 2 and a bad forwarder | 20:46 |
| clarkb | mordred: maybe we want to hack a d-g change that sets it to three and restarts unbound just to see how big that file gets for a devstack tempest run | 20:46 |
| clarkb | I can work on that hack | 20:46 |
| clarkb | and if it isn't too bad then we push into the images | 20:46 |
| clarkb | mordred: wait so there are multiple levels of info? | 20:47 |
| mordred | clarkb: http://paste.openstack.org/show/613138/ | 20:48 |
| mordred | clarkb: that's verbosity 3 - same thing - bad forwarder | 20:48 |
| clarkb | looks like it does exponential backoff trying to resolve it | 20:49 |
| mordred | yah | 20:49 |
| *** srwilkers_ has joined #openstack-infra | 20:49 | |
| srwilkers_ | hey, had a quick question. is it possible to get a tarball publisher job without being in governance? or is being in governance a prerequisite for publishing tarballs from master for CI purposes? | 20:52 |
| *** rocky_g has joined #openstack-infra | 20:52 | |
| *** StaceyF has joined #openstack-infra | 20:52 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: DNM this is testing the size of unbound logs https://review.openstack.org/475532 | 20:54 |
| clarkb | that is a really hacky way to check how big this will get | 20:54 |
| *** chlong has joined #openstack-infra | 20:55 | |
| openstackgerrit | Merged openstack-infra/shade master: base64 encode user_data sent to create server https://review.openstack.org/475385 | 20:55 |
| clarkb | srwilkers_: did you have something specific in mind. We publish things there like etcd at this point so I don't think it is a requirement. But knowing what you want would help us udnerstand if tarballs.o.o makes sense | 20:55 |
| *** rockyg has quit IRC | 20:55 | |
| *** Goneri has quit IRC | 20:55 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Default OVB variable https://review.openstack.org/475533 | 20:56 |
| srwilkers_ | clarkb, we've got a few repos associated with openstack-helm. we've got infra (for things like databases, rabbit, etcd), addons (logging tools, etc), and openstack-helm. | 20:57 |
| *** StaceyF has quit IRC | 20:57 | |
| *** nhandler_ is now known as nhandler | 20:57 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: DNM this is testing the size of unbound logs https://review.openstack.org/475532 | 20:57 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Save additional DNS info https://review.openstack.org/475520 | 20:57 |
| fungi | srwilkers_: we haven't (up to now at least) used tc governance as criteria for whether projects can publish artifacts to tarballs.openstack.org | 20:58 |
| srwilkers_ | to test all three repos, we considered: running tests on infra and publishing tarballs on each successful merge. then pull that in to use as a known success to test openstack-helm, then publish that on merge to test addons | 20:58 |
| clarkb | fungi: mordred ^ sorry found a minor issue with running ss | 20:58 |
| clarkb | srwilkers_: if you gate don't you get that for free (you only merge code on success then can just check out that code?) | 20:58 |
| mordred | infra-root: shade is ready for a release - this one will include removal of novaclient | 20:59 |
| fungi | mordred: thanks for the heads up | 21:00 |
| portdirect | clarkb: thats true :) though it would be nice to be able to use the built artifacts rather than having to build again each time | 21:01 |
| fungi | srwilkers_: to restate clarkb's point, if you can have your jobs consume the branch tip instead of consuming a tarball of your branch tip, then not only do you not need that additional step but you also can take advantage of change dependencies prior to merging | 21:01 |
| fungi | (by substituting a tool like zuul-cloner for regular git clone that is) | 21:02 |
| srwilkers_ | oh nice | 21:02 |
| srwilkers_ | do you have an example of that on hand (ie: do you know if any projects do that currently?) | 21:02 |
| clarkb | srwilkers_: all openstack projects should do it | 21:02 |
| clarkb | and yes it does mean you have to rebuild your "artifact" however that happens. For us its making sdists/wheels for python packages for the most part | 21:03 |
| srwilkers_ | okay, awesome. thanks clarkb and fungi :) | 21:04 |
| fungi | all of the devstack-gate based jobs do that automatically by checking out remote commits prepared for them by zuul for all projects in their $PROJECTS list, but there are also lots of projects using, say, zuul-cloner to checkout another repo like openstack-requirements (for obtaining the state of an upper-constraints.txt file) or related projects for bespoke integration testing | 21:04 |
| fungi | we have a commit message footer zuul looks for where any lines starting with Depends-On: indicate other gerrit changes which should be merged to related repositories when zuul-cloner or devstack-gate checks them out in the course of a ci job | 21:05 |
| fungi | srwilkers_: http://codesearch.openstack.org/?q=zuul-cloner probably has far more examples than you could ever want | 21:06 |
| srwilkers_ | fungi, awesome. cheers dude | 21:07 |
| * fungi is surprised srwilkers_ knows he's a dude | 21:08 | |
| *** gardlt has quit IRC | 21:08 | |
| * fungi abides | 21:08 | |
| *** gardlt has joined #openstack-infra | 21:08 | |
| * fungi has now used up his allowed film references for the week | 21:08 | |
| clarkb | srwilkers_: if the cost is high beacuse you are building images or long compiles or whatever then you may want to reconsider the intermediate artifact stashing. This is largely an unsolved problem though and is potentially racey. | 21:09 |
| clarkb | multiple writes and readers woo | 21:10 |
| fungi | agreed, it's a relatively common pattern used now or in the past by other projects (trove, ironic, kolla...) and is something we have ideas for improving after zuul v3 is in full production | 21:10 |
| *** dprince has quit IRC | 21:10 | |
| *** gardlt_ has joined #openstack-infra | 21:10 | |
| fungi | the solution use today is far from perfect, ut we | 21:11 |
| *** sree has joined #openstack-infra | 21:11 | |
| fungi | we're at least reasonably aware of the related limitations and gotchas should counselling be required | 21:11 |
| *** gardlt has quit IRC | 21:13 | |
| *** gardlt_ has quit IRC | 21:15 | |
| *** sree has quit IRC | 21:16 | |
| *** ldnunes has quit IRC | 21:16 | |
| clarkb | I'm beginning to think we may want to hold a node in osic (where dns works the least) then put queries to opends and google dns in a loop with some sane sleep between queries to test | 21:16 |
| *** trown is now known as trown|outtypewww | 21:16 | |
| clarkb | then hope they don't blacklist us | 21:16 |
| clarkb | see if things work without unbound or if they don't | 21:16 |
| *** jascott1 has quit IRC | 21:21 | |
| *** rfolco has quit IRC | 21:21 | |
| mordred | clarkb: ++ | 21:21 |
| *** jascott1 has joined #openstack-infra | 21:21 | |
| *** ttx has quit IRC | 21:21 | |
| *** dalvarez has quit IRC | 21:21 | |
| *** uberjay has quit IRC | 21:21 | |
| *** dulek has quit IRC | 21:21 | |
| *** sdake has quit IRC | 21:21 | |
| *** jmccrory has quit IRC | 21:21 | |
| *** eglute has quit IRC | 21:21 | |
| *** sambetts|afk has quit IRC | 21:21 | |
| *** comstud has quit IRC | 21:21 | |
| *** mgagne has quit IRC | 21:21 | |
| *** asettle has quit IRC | 21:21 | |
| *** dimtruck has quit IRC | 21:21 | |
| *** Anticimex has quit IRC | 21:21 | |
| *** jcook has quit IRC | 21:21 | |
| *** kevinbenton has quit IRC | 21:21 | |
| *** abelur has quit IRC | 21:21 | |
| *** cody-somerville has joined #openstack-infra | 21:21 | |
| *** cody-somerville has quit IRC | 21:21 | |
| *** cody-somerville has joined #openstack-infra | 21:21 | |
| *** jmccrory_ has joined #openstack-infra | 21:21 | |
| *** Anticimex has joined #openstack-infra | 21:21 | |
| *** sdake has joined #openstack-infra | 21:21 | |
| *** eglute_s has joined #openstack-infra | 21:21 | |
| *** comstud has joined #openstack-infra | 21:21 | |
| *** dalvarez has joined #openstack-infra | 21:21 | |
| *** sambetts_ has joined #openstack-infra | 21:21 | |
| *** jcook has joined #openstack-infra | 21:21 | |
| *** uberjay has joined #openstack-infra | 21:21 | |
| *** dulek has joined #openstack-infra | 21:21 | |
| *** dimtruck_ has joined #openstack-infra | 21:21 | |
| *** abelur has joined #openstack-infra | 21:22 | |
| *** mgagne has joined #openstack-infra | 21:22 | |
| *** Guest39557 has joined #openstack-infra | 21:22 | |
| ianw | if you block 53/udp does it fall back to tcp in some graceful fashion | 21:22 |
| *** sdake is now known as Guest8722 | 21:22 | |
| *** dimtruck_ is now known as dimtruck | 21:22 | |
| *** jascott1_ has joined #openstack-infra | 21:22 | |
| *** kevinbenton has joined #openstack-infra | 21:22 | |
| *** ttx has joined #openstack-infra | 21:22 | |
| *** jmccrory_ is now known as jmccrory | 21:23 | |
| *** jascott1 has quit IRC | 21:24 | |
| *** smarcet has left #openstack-infra | 21:24 | |
| fungi | ianw: not really, no | 21:24 |
| clarkb | http://logs.openstack.org/32/475532/2/check/gate-swift-dsvm-functional-ubuntu-xenial-nv/b881b93/logs/unbound_log.txt.gz is interesting, we attempt ipv6 opendns in bluebox cloud but it fails because no ipv6 there, then we fall back on google ipv4 dns | 21:25 |
| clarkb | I guess forward-addr's are not ordered by priority? | 21:26 |
| fungi | the primary "tcp fallback" mechanism with dns is nameserver-to-nameserver when using axfr in a master/slave relationship, or for some queries exceeding 512 bytes where you might otherwise end up with fragmented datagrams (tcp deals with fragmentation far more reliably than udp) | 21:26 |
| fungi | since the vast majority of dns queries and responses fit in a single packet, tcp retransmits are more overhead than just retrying the request | 21:27 |
| clarkb | also I'm not sure that unbound is actually caching based on that log file | 21:28 |
| clarkb | it queries for pypi.python.org a lot which has a very large ttl | 21:28 |
| *** jascott1_ has quit IRC | 21:29 | |
| fungi | if unbound is in any way a sane nameserver, it will have some mechanism for dumping and inspecting the contents of its cache | 21:30 |
| *** ccamacho has left #openstack-infra | 21:30 | |
| ianw | fungi: yeah ... i'm thinking of something like "use-vc" in glibc ; which ISTR using on some very dodgy links at one point. but thinking about it, it's getting back "not exist" ... rather than missing packets anyway? | 21:31 |
| *** jascott1 has joined #openstack-infra | 21:33 | |
| fungi | ianw: right, there's every chance that we're getting a negative response and then caching that for the negative cache ttl indicated by the domain. still need more detailed logging to confirm i think | 21:36 |
| *** amotoki is now known as amotoki_away | 21:36 | |
| *** jascott1 has quit IRC | 21:38 | |
| clarkb | ok on one of my old held nodes unbound-control dump_cache shows it caching things | 21:38 |
| clarkb | going to check a current node now | 21:38 |
| clarkb | it is definitely caching on new nodes too | 21:40 |
| clarkb | ok pypi.python.org is CNAMED to a fastly.net name which has a really short ttl on its A record | 21:40 |
| *** haleyb_ has quit IRC | 21:41 | |
| fungi | hah, i just realized the board meeting and tomorrow's tc meeting overlap | 21:41 |
| fungi | start at the same time, even | 21:42 |
| clarkb | using tcpdump I 'm fairly certain we are only going external for the short ttl'd A record | 21:45 |
| *** jkilpatr has quit IRC | 21:45 | |
| *** chlong has quit IRC | 21:46 | |
| clarkb | unbound just logs that in a funny way? | 21:47 |
| *** mhickey has quit IRC | 21:48 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Bump Ansible 2.1 from 2.1.5 to 2.1.6 for ARA gate jobs https://review.openstack.org/475575 | 21:48 |
| clarkb | looks like neutron's dns server resolves a bunch of queries for download.cirros-cloud.net too but it is also caching since I don't see those queries go off host | 21:51 |
| clarkb | (cirros checking if it is up to date on boot or something?) | 21:51 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Use cirros images on host https://review.openstack.org/475576 | 21:51 |
| clarkb | ianw: fungi: mordred I think we should consider getting https://review.openstack.org/#/c/475520/ in just to rule out dns server being dead. | 21:55 |
| mordred | clarkb: wfm | 21:55 |
| fungi | today i learned that ss is less typing than netstat | 21:57 |
| clarkb | fungi: it is the ip replacement of netstat | 21:57 |
| clarkb | ss:netstat ip:ifconfig | 21:57 |
| *** rpioso has left #openstack-infra | 21:58 | |
| *** rpioso has quit IRC | 21:58 | |
| fungi | indeed, i'd either never encountered it or sacrificed those braincells to the beer gods | 21:58 |
| *** srwilkers_ has quit IRC | 21:58 | |
| *** baoli has quit IRC | 21:59 | |
| clarkb | `unbound-control stats_noreset` is another interesting command we can add to logging data if we want | 22:00 |
| clarkb | it shows total number of queries and total number of misses | 22:00 |
| clarkb | *cache misses | 22:00 |
| mordred | oh - nice | 22:00 |
| *** Goneri has joined #openstack-infra | 22:00 | |
| clarkb | it actually looks pretty decent on this held node | 22:02 |
| clarkb | 423 queries, 39 misses | 22:02 |
| *** neiljerram has quit IRC | 22:02 | |
| *** makowals has quit IRC | 22:03 | |
| *** markvoelker has quit IRC | 22:03 | |
| *** wolverineav has quit IRC | 22:03 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: Make the Ironic multinode grenade job voting https://review.openstack.org/475521 | 22:04 |
| *** markvoelker has joined #openstack-infra | 22:04 | |
| *** wolverineav has joined #openstack-infra | 22:04 | |
| *** neiljerram has joined #openstack-infra | 22:04 | |
| *** makowals has joined #openstack-infra | 22:05 | |
| *** esberglu has quit IRC | 22:06 | |
| *** Apoorva_ has joined #openstack-infra | 22:06 | |
| *** esberglu has joined #openstack-infra | 22:06 | |
| *** Apoorva has quit IRC | 22:07 | |
| clarkb | https://www.unbound.net/documentation/howto_optimise.html has some things we might consider like allowing more threads and using os-reuseport | 22:07 |
| *** markvoelker_ has joined #openstack-infra | 22:08 | |
| clarkb | so_reuseport needs kernel >=3.9 though | 22:08 |
| *** markvoelker has quit IRC | 22:08 | |
| *** wolverineav has quit IRC | 22:09 | |
| *** esberglu has quit IRC | 22:10 | |
| *** markvoelker_ has quit IRC | 22:11 | |
| *** rocky_g has quit IRC | 22:12 | |
| *** bobh has quit IRC | 22:13 | |
| clarkb | the ubuntu unbound is linked against libevent so threads may not actually matter there | 22:13 |
| *** markvoelker has joined #openstack-infra | 22:14 | |
| *** ijw__ has joined #openstack-infra | 22:15 | |
| *** gardlt has joined #openstack-infra | 22:17 | |
| clarkb | specifically out of the stats data I think we want threadX.requestlist.exceeded if that number is >0 that means we dropped queries | 22:18 |
| *** jkilpatr has joined #openstack-infra | 22:18 | |
| clarkb | so maybe we collect the stats and check for that vlaue then add threads if necessary? | 22:18 |
| clarkb | we can also just add threads | 22:18 |
| *** esberglu has joined #openstack-infra | 22:19 | |
| *** rcernin has quit IRC | 22:21 | |
| *** gardlt has quit IRC | 22:26 | |
| *** gardlt has joined #openstack-infra | 22:26 | |
| *** gardlt has quit IRC | 22:27 | |
| *** gardlt has joined #openstack-infra | 22:28 | |
| *** jkilpatr has quit IRC | 22:29 | |
| *** dimtruck is now known as zz_dimtruck | 22:30 | |
| *** baoli has joined #openstack-infra | 22:32 | |
| *** baoli has quit IRC | 22:32 | |
| *** thorst has joined #openstack-infra | 22:32 | |
| *** Goneri has quit IRC | 22:34 | |
| *** jascott1 has joined #openstack-infra | 22:35 | |
| *** jascott1 has quit IRC | 22:36 | |
| *** andrewbogott has quit IRC | 22:36 | |
| *** andrewbogott has joined #openstack-infra | 22:36 | |
| *** thorst has quit IRC | 22:37 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Tune unbound for our test systems https://review.openstack.org/475592 | 22:37 |
| clarkb | something like ^ would be what we want based on the docs, I tested that on xenial | 22:37 |
| *** gordc has quit IRC | 22:37 | |
| clarkb | I'm not sure we need to do that yet though. Would be good to pin this problem down before making changes like that | 22:37 |
| *** MasterOfBugs has joined #openstack-infra | 22:41 | |
| openstackgerrit | Merged openstack-infra/shade master: Don't remove top-container element for server REST API calls https://review.openstack.org/475501 | 22:43 |
| clarkb | mtreinish: are you back online? | 22:45 |
| * fungi expects the sun is just now rising in the land of the rising sun | 22:46 | |
| clarkb | mtreinish: I've got a change that sdague would like you to review https://review.openstack.org/#/c/474036/ should slightly reduce job runtimes | 22:46 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Move ubuntu-trusty image pause to correct section" https://review.openstack.org/475213 | 22:47 |
| *** wolverineav has joined #openstack-infra | 22:48 | |
| *** MasterOfBugs has quit IRC | 22:50 | |
| *** pramodrj07 has joined #openstack-infra | 22:50 | |
| *** camunoz has quit IRC | 22:50 | |
| *** rbrndt has quit IRC | 22:50 | |
| *** rbrndt has joined #openstack-infra | 22:50 | |
| *** rbrndt has quit IRC | 22:50 | |
| *** salv-orlando has quit IRC | 22:50 | |
| *** thorst has joined #openstack-infra | 22:51 | |
| *** thorst has quit IRC | 22:51 | |
| clarkb | part of me wants to get those config changes for unbound in just to see if it fixes the problems | 22:52 |
| fungi | if it did, though, we would be left wondering what caused it to become an issue so suddenly and for such an extended period globally | 22:56 |
| clarkb | ya | 22:56 |
| *** gardlt has quit IRC | 22:57 | |
| *** gardlt has joined #openstack-infra | 22:57 | |
| *** gomarivera has quit IRC | 22:58 | |
| *** yland has quit IRC | 22:59 | |
| *** gardlt_ has joined #openstack-infra | 23:00 | |
| *** gomarivera has joined #openstack-infra | 23:00 | |
| *** yland has joined #openstack-infra | 23:00 | |
| *** yland is now known as Guest66617 | 23:00 | |
| *** gardlt has quit IRC | 23:02 | |
| *** jascott1 has joined #openstack-infra | 23:03 | |
| *** gomarivera has quit IRC | 23:04 | |
| clarkb | fatal: unable to access 'https://github.com/openstack/nova/';: Could not resolve host: github.com <- this is an interesting way to find people using github when they shouldn't be | 23:04 |
| *** gardlt_ has quit IRC | 23:04 | |
| *** wolverineav has quit IRC | 23:04 | |
| clarkb | that is from gate-compute-hyperv-pep8-ubuntu-xenial | 23:04 |
| *** jaypipes has quit IRC | 23:04 | |
| *** wolverineav has joined #openstack-infra | 23:05 | |
| clarkb | according to logstash 3 hyperv jobs all failed to resole github.com at the same time in osic | 23:05 |
| clarkb | that implies it isn't a problem with a single instance in osic (eg not a performance issue with unbound) | 23:05 |
| clarkb | cloudnull: there wouldn't be a firewall throttling those connections in osic would there? | 23:06 |
| *** xarses has quit IRC | 23:06 | |
| fungi | pat on udp connections in rapid succession can start exhibiting all sorts of strange behaviors under port exhaustion pressure too | 23:07 |
| clarkb | oh ya that could lead to fun | 23:07 |
| fungi | "connections" | 23:07 |
| fungi | not entirely a kosher term where udp is concerned | 23:07 |
| fungi | state tracking i suppose | 23:07 |
| clarkb | well some firewalls will state track udp so effectively connections | 23:08 |
| clarkb | or is that what youmean by pat? | 23:08 |
| fungi | port address translation (layer 4 "overload" nat) implies some minimal state maintenance on the translating device | 23:09 |
| fungi | so as to remember what global ports map to which internal port numbers at any given point in time | 23:09 |
| clarkb | what I've seen is firewalls want to allow return udp traffic as if it were an established tcp connection so they will infer what that raffic looks like and let it through using hand wavy state tracking | 23:10 |
| fungi | as opposed to a 1:1 (layer 3) nat, which only needs to retain address mappings | 23:10 |
| clarkb | without doing address translation | 23:11 |
| fungi | oh, sure, in non-nat (filtering router) situations | 23:11 |
| fungi | yep, and since there is minimal signaling of the end of a udp session, they generally impose some fairly short timeout before they recycle the tracked state | 23:11 |
| clarkb | yup and if there is any delay in the return it gets dropped as a result | 23:12 |
| fungi | and that tends to be the case equally for filtering routers and overload port address translators | 23:12 |
| clarkb | or if it comes back asymettrically so it doesn't match existing state | 23:12 |
| fungi | but where behavior gets dicey is when you have so many of them that you start getting as many tracked udp sessions as you can map to individual port numbers for a single address | 23:13 |
| fungi | that's really a nat-specific issue | 23:13 |
| fungi | well, pat-specific especially | 23:13 |
| fungi | are we forwarding to external resolvers over ipv6, ipv4, or a mix of the two? | 23:14 |
| clarkb | we configure ipv4 and ipv6, based on my reading of unbound logs I'm not sure it prioritizes them by order | 23:15 |
| clarkb | so it is possible that we are sending over ipv4 and the pat tables are sad | 23:15 |
| clarkb | fungi: http://logs.openstack.org/32/475532/2/check/gate-tempest-dsvm-py35-ubuntu-xenial/9b0b48d/logs/unbound_log.txt.gz that ran in osic and it definitely appears to use 8.8.8.8 | 23:16 |
| *** wolverineav has quit IRC | 23:16 | |
| clarkb | fungi: so maybe the change we want to make is have only ipv6 resolvers in ipv6 clouds, and only ipv4 in ipv4 clouds | 23:17 |
| clarkb | and just avoid the nat/pat problem as much as possible | 23:17 |
| *** wolverineav has joined #openstack-infra | 23:17 | |
| clarkb | this will have to be an init script/systemd unit though | 23:17 |
| fungi | yeah, that'll require some logic | 23:17 |
| *** ianychoi has quit IRC | 23:17 | |
| clarkb | mordred: ianw ^ thoughts? | 23:18 |
| fungi | i suppose a nodepool ready script could do it too, but having working name resolution earlier in boot might be nice (and could also speed up the initial ssh connection so it doesn't take forever trying to do a reverse lookup fr the incoming client) | 23:19 |
| *** PramodJ has joined #openstack-infra | 23:19 | |
| *** tosky has quit IRC | 23:22 | |
| *** pramodrj07 has quit IRC | 23:22 | |
| *** jamesmcarthur has joined #openstack-infra | 23:26 | |
| *** jamesmcarthur has quit IRC | 23:28 | |
| *** jamesmca_ has joined #openstack-infra | 23:28 | |
| *** gildub has joined #openstack-infra | 23:29 | |
| *** wolverineav has quit IRC | 23:29 | |
| *** gardlt has joined #openstack-infra | 23:39 | |
| *** hongbin has quit IRC | 23:40 | |
| *** zz_dimtruck is now known as dimtruck | 23:49 | |
| *** JudeC has quit IRC | 23:50 | |
| *** pvaneck has quit IRC | 23:52 | |
| *** xarses has joined #openstack-infra | 23:53 | |
| *** andreww has quit IRC | 23:53 | |
| clarkb | this is extra complicated because unbound is sys v init script on ubuntu xenial | 23:58 |
| clarkb | it is sys v unit on centos/fedora | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!