| *** gyee has quit IRC | 00:03 | |
| pabelanger | ugh | 00:04 |
|---|---|---|
| pabelanger | http://logs.openstack.org/46/584546/1/gate/infra-puppet-apply-3-centos-7/c821706/job-output.txt.gz#_2018-07-23_23_59_46_980276 | 00:04 |
| pabelanger | epel mirror failed | 00:04 |
| pabelanger | guess we should be using AFS mirror there | 00:05 |
| clarkb | pabelanger: thats odd, it failed on the limestone upstream mirror and job ran in limestone | 00:10 |
| clarkb | I wonder if those mirrors are all ipv4 only | 00:10 |
| clarkb | (which forces our job thorugh NAT) | 00:10 |
| pabelanger | not sure | 00:11 |
| pabelanger | we actually don't need to install epel there, as centos already has it setup | 00:11 |
| pabelanger | will refactor script in a bit | 00:11 |
| *** jamesmcarthur has quit IRC | 00:11 | |
| *** shu-mutow has joined #openstack-infra | 00:12 | |
| *** jamesmcarthur has joined #openstack-infra | 00:13 | |
| *** markvoelker has quit IRC | 00:19 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Skip epel mirror if already setup https://review.openstack.org/585056 | 00:19 |
| pabelanger | ianw: clarkb: lets see if ^ works | 00:20 |
| *** ijw has joined #openstack-infra | 00:21 | |
| *** jamesmcarthur has quit IRC | 00:24 | |
| dave-mccowan | what's the right way to restart a merge that failed a gate during +W (after the underlying gate has been fixed)? recheck? reverify? -W then +W? | 00:25 |
| clarkb | dave-mccowan: recheck | 00:25 |
| clarkb | I think +W will work as well (current gerrit doesnt require -W first) but only approvers can do +W anyone can recheck | 00:26 |
| dave-mccowan | clarkb with this job: https://review.openstack.org/#/c/499419/55 none of those seem to be retriggering the merge workflow. | 00:28 |
| corvus | dave-mccowan: (also, assuming it wasn't just due entirely to outside influences, generally speaking, it's worth considering whether additional testing (co-gating with other projects, adding a new job, etc) might prevent getting into that situation) | 00:28 |
| corvus | dave-mccowan: it's currently in the check queue; when it's done and passes, it'll show up in gate again. | 00:29 |
| *** shrircha has quit IRC | 00:29 | |
| *** shrircha has joined #openstack-infra | 00:30 | |
| corvus | dave-mccowan: (openstack has a policy that a change has to pass check before moving to gate) | 00:30 |
| *** graphene has quit IRC | 00:30 | |
| dave-mccowan | corvus ah, i should have checked the queue. i was expecting to see the -2 from jenkins to clear first. thanks! | 00:31 |
| *** graphene has joined #openstack-infra | 00:32 | |
| corvus | dave-mccowan: yeah, that only happens when it goes into gate, but check doesn't clear. so if it passes, you'll see a +1, then almost immediately it'll get cleared. :) | 00:32 |
| corvus | maybe when we have the new results api available, we can do something like that | 00:33 |
| *** threestrands has quit IRC | 00:38 | |
| *** gfidente has quit IRC | 00:38 | |
| *** hongbin has joined #openstack-infra | 00:39 | |
| *** stakeda has joined #openstack-infra | 00:39 | |
| *** s-shiono has joined #openstack-infra | 00:43 | |
| *** owalsh_ is now known as owalsh | 00:44 | |
| *** markvoelker has joined #openstack-infra | 00:47 | |
| tonyb | Any chance a brave soul would like to approve: https://review.openstack.org/583065 ? | 00:47 |
| logan- | clarkb pabelanger: yea thats odd.. http://fedora-epel.mirror.lstn.net/7/x86_64/repodata/ shows the files have not been updated since 7/14 and it looks like epel has not synced since 7/16 per http://mirror.lstn.net/.... I wonder if our upstream disappeared. will look into it. thanks | 00:48 |
| pabelanger | logan-: we use rsync://pubmirror1.math.uh.edu/fedora-buffet/epel for openstack-infra | 00:50 |
| *** felipemonteiro has joined #openstack-infra | 00:51 | |
| felipemonteiro | is it possible to apply irrelevant-files stanza to a project-template, e.g. integrated-gate-py35 -- from what I can tell it doesn't look like it | 00:52 |
| logan- | thanks pabelanger. well mystery solved http://paste.openstack.org/raw/726489/ :) good catch | 00:55 |
| *** rcernin_ has joined #openstack-infra | 00:56 | |
| pabelanger | logan-: yah, we stopped mirroring from kernel.org for same reason | 00:56 |
| *** bdodd has quit IRC | 00:56 | |
| *** rcernin has quit IRC | 00:58 | |
| openstackgerrit | Merged openstack-infra/system-config master: Bring ze11.o.o online https://review.openstack.org/584546 | 01:08 |
| ianw | pabelanger: speaking of mirroring issues; notice anything :) http://grafana.openstack.org/d/ACtl1JSmz/afs?panelId=25&fullscreen&orgId=1&from=now-30d&to=now | 01:09 |
| ianw | i'll bump the fedora quota | 01:09 |
| pabelanger | ianw: oh, did we actually delete old fedora content? | 01:09 |
| ianw | hrm | 01:09 |
| pabelanger | ianw: eg: fedora-27 can be delete | 01:09 |
| *** gongysh has quit IRC | 01:10 | |
| pabelanger | but not sure if anybody has removed the images yet | 01:10 |
| pabelanger | http://mirror.dfw.rax.openstack.org/fedora/atomic/stable/ | 01:10 |
| pabelanger | might not help either | 01:10 |
| ianw | umm, i think dib is still straggling on that | 01:10 |
| ianw | but i should be able to finish of the zuulv3 transition there this week | 01:10 |
| pabelanger | yah, if we nuke atomic images, we can safe a bunch of space I htink | 01:11 |
| pabelanger | I'm not sure if anything is using that any more | 01:11 |
| ianw | i think i looked at the rsync for that at some point | 01:11 |
| *** slaweq has joined #openstack-infra | 01:11 | |
| pabelanger | was it for magnum? | 01:11 |
| ianw | oh, that's right, i was looking at dropping the alternative archs | 01:12 |
| ianw | 99G there | 01:12 |
| clarkb | yes magnum used the atomic images | 01:12 |
| pabelanger | watching puppet run now for ze11.o.o firewall changes | 01:13 |
| ianw | pabelanger: ahh, there's even a change for it https://review.openstack.org/#/c/526259/ | 01:13 |
| pabelanger | ianw: needs a rebase, will +2 :) | 01:14 |
| *** slaweq has quit IRC | 01:15 | |
| *** links has joined #openstack-infra | 01:17 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: fedora-mirror: skip alternative archs in atomic image mirrors https://review.openstack.org/526259 | 01:19 |
| *** yamamoto has joined #openstack-infra | 01:20 | |
| ianw | pabelanger: ^ i think i'll bump the quota slightly, and manually try that out anyway | 01:20 |
| *** jiapei has joined #openstack-infra | 01:20 | |
| pabelanger | ianw: yah, too bad there isn't specific sub directories for releases, would make it much easier to rsync | 01:21 |
| pabelanger | not sure how best to drop fedora-23, 24, 25, etc | 01:21 |
| ianw | ok, have fedora lock and running a manual update after bumping quota to 500g | 01:23 |
| *** yamamoto has quit IRC | 01:24 | |
| *** Bhujay has joined #openstack-infra | 01:24 | |
| *** edmondsw has joined #openstack-infra | 01:27 | |
| *** felipemonteiro has quit IRC | 01:29 | |
| *** yamahata has quit IRC | 01:31 | |
| *** edmondsw has quit IRC | 01:31 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Skip epel mirror if already setup https://review.openstack.org/585056 | 01:46 |
| pabelanger | ianw: clarkb: ^ 2.0 for epel check | 01:46 |
| pabelanger | ze11.o.o is running jobs | 01:48 |
| pabelanger | #status log ze11.openstack.org is online and running jobs. | 01:49 |
| openstackstatus | pabelanger: finished logging | 01:49 |
| ianw | pabelanger: left a comment ... it looks a little odd ... "rpm -qa | grep ..." is probably simpler? | 01:56 |
| pabelanger | ianw: ah, k. If you want to push up a change that is okay with me, about to call it shortly | 01:58 |
| pabelanger | otherwise, can look at it in the morning | 01:58 |
| ianw | ok | 01:59 |
| ianw | ahh, we already do it in devstack, using the yum repolist. which is basically your first method, but just abstracted to the tool. may as well do it like that since it's known to work | 02:00 |
| jiapei | Hi all, I'm setting up nodepool according to https://docs.openstack.org/infra/openstackci/third_party_ci.html#start-nodepool, but I find that nodepool-builder can't be started, then I use "ps -elf | grep nodepool", and find that nodepool start with the --no-builder option as shown http://paste.openstack.org/show/726492/. Besides, log at /var/log/nodepool/builder-debug.log is empty. So how can I debug it? Anyone can | 02:02 |
| jiapei | help me with the problem? Thank you😃 | 02:02 |
| *** ijw has quit IRC | 02:04 | |
| *** gongysh has joined #openstack-infra | 02:07 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Skip epel mirror if already setup https://review.openstack.org/585056 | 02:08 |
| *** atarakt has joined #openstack-infra | 02:10 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** gcb has joined #openstack-infra | 02:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Use mode 0o755 for executor server mkdirs https://review.openstack.org/585068 | 02:16 |
| pabelanger | infra-root: ze11.o.o is holding up great | 02:19 |
| pabelanger | and with that, I am EOD | 02:19 |
| *** armaan has quit IRC | 02:19 | |
| *** armaan has joined #openstack-infra | 02:20 | |
| *** jamesmcarthur has joined #openstack-infra | 02:24 | |
| *** ktsuyuzaki is now known as kota_ | 02:28 | |
| *** jamesmcarthur has quit IRC | 02:29 | |
| *** jamesmcarthur has joined #openstack-infra | 02:30 | |
| *** dave-mccowan has quit IRC | 02:31 | |
| *** graphene has quit IRC | 02:37 | |
| *** graphene has joined #openstack-infra | 02:39 | |
| *** felipemonteiro has joined #openstack-infra | 02:44 | |
| clarkb | pabelanger: thank you! | 02:46 |
| *** atarakt has quit IRC | 02:47 | |
| *** psachin` has joined #openstack-infra | 02:49 | |
| *** eernst has joined #openstack-infra | 02:53 | |
| *** yamahata has joined #openstack-infra | 02:57 | |
| *** ramishra has joined #openstack-infra | 03:01 | |
| *** eernst has quit IRC | 03:07 | |
| *** eernst has joined #openstack-infra | 03:08 | |
| *** eernst has quit IRC | 03:09 | |
| *** dklyle has quit IRC | 03:14 | |
| *** edmondsw has joined #openstack-infra | 03:15 | |
| *** Bhujay has quit IRC | 03:17 | |
| *** roman_g has quit IRC | 03:18 | |
| *** edmondsw has quit IRC | 03:19 | |
| *** r-daneel has joined #openstack-infra | 03:21 | |
| *** graphene has quit IRC | 03:25 | |
| *** graphene has joined #openstack-infra | 03:26 | |
| *** ijw has joined #openstack-infra | 03:27 | |
| *** jamesmcarthur has quit IRC | 03:29 | |
| *** dklyle has joined #openstack-infra | 03:30 | |
| *** ijw has quit IRC | 03:32 | |
| *** adrianreza has quit IRC | 03:34 | |
| *** felipemonteiro has quit IRC | 03:38 | |
| *** r-daneel has quit IRC | 03:39 | |
| *** gongysh has quit IRC | 03:42 | |
| *** bdodd has joined #openstack-infra | 03:44 | |
| *** jamesmcarthur has joined #openstack-infra | 03:45 | |
| *** bdodd has quit IRC | 03:48 | |
| *** jamesmcarthur has quit IRC | 03:50 | |
| *** felipemonteiro has joined #openstack-infra | 03:50 | |
| openstackgerrit | Vu Cong Tuan proposed openstack-dev/hacking master: Switch to stestr https://review.openstack.org/581541 | 03:52 |
| *** bdodd has joined #openstack-infra | 03:52 | |
| *** lpetrut has joined #openstack-infra | 03:53 | |
| *** eernst has joined #openstack-infra | 03:58 | |
| *** felipemonteiro has quit IRC | 04:00 | |
| *** dhajare has joined #openstack-infra | 04:01 | |
| *** ykarel has joined #openstack-infra | 04:03 | |
| *** mschuppert has joined #openstack-infra | 04:05 | |
| *** zhangfei has joined #openstack-infra | 04:06 | |
| *** felipemonteiro has joined #openstack-infra | 04:08 | |
| *** jamesmcarthur has joined #openstack-infra | 04:10 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Murano convert weekly meeting to office hours https://review.openstack.org/584734 | 04:11 |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** felipemonteiro has quit IRC | 04:14 | |
| *** armaan has quit IRC | 04:15 | |
| *** psachin` has quit IRC | 04:15 | |
| *** slaweq has quit IRC | 04:16 | |
| *** hongbin has quit IRC | 04:16 | |
| *** jamesmcarthur has quit IRC | 04:16 | |
| *** r-daneel has joined #openstack-infra | 04:17 | |
| *** armaan has joined #openstack-infra | 04:18 | |
| *** psachin` has joined #openstack-infra | 04:19 | |
| *** luzC has quit IRC | 04:25 | |
| *** jamesmcarthur has joined #openstack-infra | 04:26 | |
| *** eernst has quit IRC | 04:27 | |
| *** luzC has joined #openstack-infra | 04:28 | |
| *** jamesmcarthur has quit IRC | 04:31 | |
| *** jamesmcarthur has joined #openstack-infra | 04:32 | |
| *** ijw has joined #openstack-infra | 04:33 | |
| *** ijw has quit IRC | 04:38 | |
| *** e0ne has joined #openstack-infra | 04:40 | |
| *** e0ne has quit IRC | 04:40 | |
| *** ihar has quit IRC | 04:40 | |
| *** jamesmcarthur has quit IRC | 04:40 | |
| *** david-lyle has joined #openstack-infra | 04:42 | |
| *** lpetrut has quit IRC | 04:42 | |
| *** dklyle has quit IRC | 04:43 | |
| *** jamesmcarthur has joined #openstack-infra | 04:45 | |
| *** honza has quit IRC | 04:47 | |
| *** honza has joined #openstack-infra | 04:48 | |
| *** honza is now known as Guest56850 | 04:49 | |
| *** e0ne has joined #openstack-infra | 04:53 | |
| *** jamesmcarthur has quit IRC | 04:54 | |
| *** e0ne has quit IRC | 04:55 | |
| *** iyamahat has joined #openstack-infra | 04:57 | |
| *** edmondsw has joined #openstack-infra | 05:03 | |
| *** ijw has joined #openstack-infra | 05:04 | |
| *** Douhet has quit IRC | 05:05 | |
| *** armaan has quit IRC | 05:05 | |
| *** Douhet has joined #openstack-infra | 05:06 | |
| *** armaan has joined #openstack-infra | 05:06 | |
| *** jamesmcarthur has joined #openstack-infra | 05:07 | |
| *** edmondsw has quit IRC | 05:08 | |
| *** gongysh has joined #openstack-infra | 05:08 | |
| *** armaan has quit IRC | 05:09 | |
| *** armaan has joined #openstack-infra | 05:10 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/os-loganalyze master: Add report dialog to enable user feedback https://review.openstack.org/579782 | 05:11 |
| *** ijw has quit IRC | 05:11 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** jamesmcarthur has quit IRC | 05:12 | |
| *** tojuvone has left #openstack-infra | 05:14 | |
| *** slaweq has quit IRC | 05:15 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Rename didAnyJobFail() to hasAnyJobFailed() https://review.openstack.org/585134 | 05:16 |
| *** fnordahl has quit IRC | 05:16 | |
| *** kjackal has joined #openstack-infra | 05:17 | |
| *** kjackal_ has joined #openstack-infra | 05:17 | |
| *** gongysh has quit IRC | 05:21 | |
| *** fnordahl has joined #openstack-infra | 05:23 | |
| *** jamesmcarthur has joined #openstack-infra | 05:28 | |
| *** viks_ has joined #openstack-infra | 05:30 | |
| *** cshastri has joined #openstack-infra | 05:31 | |
| *** jamesmcarthur has quit IRC | 05:32 | |
| *** quiquell|off is now known as quiquell | 05:38 | |
| *** jamesmcarthur has joined #openstack-infra | 05:41 | |
| *** armaan has quit IRC | 05:43 | |
| *** zhangfei has quit IRC | 05:43 | |
| *** armaan has joined #openstack-infra | 05:44 | |
| *** ccamacho has joined #openstack-infra | 05:44 | |
| *** yamamoto has joined #openstack-infra | 05:46 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Add instructions on building static web components https://review.openstack.org/581256 | 05:46 |
| *** yamamoto has quit IRC | 05:48 | |
| *** vivsoni_ has joined #openstack-infra | 05:52 | |
| *** vivsoni has quit IRC | 05:54 | |
| *** zhangfei has joined #openstack-infra | 05:56 | |
| *** yamamoto has joined #openstack-infra | 05:56 | |
| *** yamamoto has quit IRC | 06:00 | |
| *** kjackal_ has quit IRC | 06:05 | |
| *** kjackal has quit IRC | 06:05 | |
| *** janki has joined #openstack-infra | 06:06 | |
| AJaeger | config-core, https://review.openstack.org/#/c/583024/ has its dependencies merged - and https://review.openstack.org/584571 is ready as well. Could you review these at your pace, please? | 06:06 |
| *** jamesmcarthur has quit IRC | 06:09 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Run propose-update-constraints under Python3.6 https://review.openstack.org/583065 | 06:15 |
| *** slaweq has quit IRC | 06:16 | |
| *** graphene has quit IRC | 06:17 | |
| *** graphene has joined #openstack-infra | 06:18 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove glance_store queens translations https://review.openstack.org/584571 | 06:22 |
| *** armaan_ has joined #openstack-infra | 06:26 | |
| *** zhangfei has quit IRC | 06:28 | |
| *** armaan has quit IRC | 06:29 | |
| *** gongysh has joined #openstack-infra | 06:32 | |
| *** lpetrut has joined #openstack-infra | 06:33 | |
| *** pcaruana has joined #openstack-infra | 06:34 | |
| *** alexchadin has joined #openstack-infra | 06:35 | |
| *** felipemonteiro has joined #openstack-infra | 06:39 | |
| *** yamahata has quit IRC | 06:43 | |
| *** yamahata has joined #openstack-infra | 06:44 | |
| AJaeger | thanks, ianw | 06:52 |
| *** jamesmcarthur has joined #openstack-infra | 06:52 | |
| *** quiquell is now known as quiquell|bbl | 06:54 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Scope all_known_hosts in mult-node-known-hosts https://review.openstack.org/584993 | 06:56 |
| *** jamesmcarthur has quit IRC | 06:57 | |
| *** hashar has joined #openstack-infra | 06:58 | |
| *** felipemonteiro has quit IRC | 07:00 | |
| *** kjackal has joined #openstack-infra | 07:03 | |
| *** kjackal_ has joined #openstack-infra | 07:03 | |
| *** david-lyle has quit IRC | 07:05 | |
| *** david-lyle has joined #openstack-infra | 07:06 | |
| *** udesale has joined #openstack-infra | 07:06 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| *** pguimaraes has joined #openstack-infra | 07:12 | |
| *** pguimaraes has quit IRC | 07:12 | |
| *** slaweq has quit IRC | 07:12 | |
| *** pguimaraes has joined #openstack-infra | 07:12 | |
| *** alexchadin has quit IRC | 07:13 | |
| *** alexchadin has joined #openstack-infra | 07:15 | |
| *** slaweq has joined #openstack-infra | 07:16 | |
| *** tesseract has joined #openstack-infra | 07:18 | |
| *** prometheanfire has quit IRC | 07:26 | |
| *** jamesmcarthur has joined #openstack-infra | 07:28 | |
| *** markvoelker has quit IRC | 07:29 | |
| *** gfidente has joined #openstack-infra | 07:29 | |
| *** gfidente has quit IRC | 07:29 | |
| *** gfidente has joined #openstack-infra | 07:29 | |
| *** zhangfei has joined #openstack-infra | 07:31 | |
| *** shardy has joined #openstack-infra | 07:32 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix for proper LVM support https://review.openstack.org/576168 | 07:32 |
| *** Dobroslaw has left #openstack-infra | 07:32 | |
| *** jamesmcarthur has quit IRC | 07:32 | |
| *** ykarel is now known as ykarel|lunch | 07:33 | |
| *** tosky has joined #openstack-infra | 07:36 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for groups.o.o https://review.openstack.org/584638 | 07:38 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for graphite.o.o https://review.openstack.org/584745 | 07:38 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn the future parser on for codesearch.o.o https://review.openstack.org/585161 | 07:38 |
| *** lpetrut has quit IRC | 07:48 | |
| *** prometheanfire has joined #openstack-infra | 07:49 | |
| *** quiquell|bbl is now known as quiquell | 07:52 | |
| *** lpetrut has joined #openstack-infra | 07:55 | |
| *** Bhujay has joined #openstack-infra | 07:55 | |
| *** rcernin_ has quit IRC | 07:56 | |
| *** janki has quit IRC | 07:58 | |
| *** janki has joined #openstack-infra | 07:58 | |
| *** alexchadin has quit IRC | 08:01 | |
| *** alexchadin has joined #openstack-infra | 08:03 | |
| *** graphene has quit IRC | 08:07 | |
| *** Bhujay has quit IRC | 08:07 | |
| *** derekh has joined #openstack-infra | 08:07 | |
| *** jpich has joined #openstack-infra | 08:07 | |
| *** roman_g has joined #openstack-infra | 08:07 | |
| *** graphene has joined #openstack-infra | 08:08 | |
| *** rcernin_ has joined #openstack-infra | 08:09 | |
| *** yamamoto has joined #openstack-infra | 08:09 | |
| *** jamesmcarthur has joined #openstack-infra | 08:09 | |
| *** jamesmcarthur has quit IRC | 08:14 | |
| *** eumel8 has joined #openstack-infra | 08:14 | |
| *** eumel8 has quit IRC | 08:15 | |
| *** e0ne has joined #openstack-infra | 08:15 | |
| *** shu-mutow has quit IRC | 08:18 | |
| *** graphene has quit IRC | 08:21 | |
| *** dklyle_ has joined #openstack-infra | 08:21 | |
| *** ykarel|lunch is now known as ykarel | 08:21 | |
| *** graphene has joined #openstack-infra | 08:22 | |
| *** david-lyle has quit IRC | 08:23 | |
| *** yamamoto has quit IRC | 08:26 | |
| *** auristor has quit IRC | 08:29 | |
| *** ralonsoh has joined #openstack-infra | 08:31 | |
| *** dtantsur|afk is now known as dtantsur | 08:33 | |
| *** florianf has joined #openstack-infra | 08:33 | |
| *** armaan_ has quit IRC | 08:34 | |
| *** armaan has joined #openstack-infra | 08:34 | |
| *** pbourke has quit IRC | 08:36 | |
| *** auristor has joined #openstack-infra | 08:37 | |
| *** pbourke has joined #openstack-infra | 08:37 | |
| *** vivsoni_ has quit IRC | 08:39 | |
| *** florianf has quit IRC | 08:39 | |
| *** vivsoni has joined #openstack-infra | 08:39 | |
| *** gfidente has quit IRC | 08:46 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 08:46 | |
| *** jamesmcarthur has joined #openstack-infra | 08:56 | |
| *** priteau has joined #openstack-infra | 08:56 | |
| *** jamesmcarthur has quit IRC | 09:01 | |
| *** janki has quit IRC | 09:01 | |
| *** gouthamr has quit IRC | 09:02 | |
| *** dmellado has quit IRC | 09:03 | |
| *** stevebaker has quit IRC | 09:03 | |
| *** udesale has quit IRC | 09:06 | |
| *** kiennt26 has joined #openstack-infra | 09:06 | |
| *** armaan has quit IRC | 09:07 | |
| *** armaan has joined #openstack-infra | 09:08 | |
| *** jento has quit IRC | 09:16 | |
| *** jento has joined #openstack-infra | 09:17 | |
| openstackgerrit | Carlos Goncalves proposed openstack-infra/project-config master: DNM: Switch to publish-openstack-docs-pti in Octavia https://review.openstack.org/585184 | 09:23 |
| *** alexchadin has quit IRC | 09:24 | |
| *** panda|off is now known as panda | 09:25 | |
| *** udesale has joined #openstack-infra | 09:28 | |
| *** markvoelker has joined #openstack-infra | 09:33 | |
| *** jamesmcarthur has joined #openstack-infra | 09:37 | |
| *** vivsoni has quit IRC | 09:39 | |
| *** vivsoni has joined #openstack-infra | 09:41 | |
| *** jamesmcarthur has quit IRC | 09:41 | |
| *** dhajare has quit IRC | 09:42 | |
| jiapei | Hi, I'm encountered an nodepool-builder issue http://paste.openstack.org/show/726507/, it says "DibFailedError: DIB failed creating /opt/nodepool_dib/217", could anyone help me? | 09:42 |
| *** oanson has quit IRC | 09:45 | |
| *** oanson has joined #openstack-infra | 09:45 | |
| *** stephenfin has quit IRC | 09:47 | |
| *** jaosorior has quit IRC | 09:47 | |
| *** stephenfin has joined #openstack-infra | 09:49 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-askbot master: Fix template variable https://review.openstack.org/585195 | 09:51 |
| *** alexchadin has joined #openstack-infra | 09:55 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-askbot master: Add test to ensure service is active https://review.openstack.org/585196 | 09:58 |
| *** alexchadin has quit IRC | 10:00 | |
| *** zoli is now known as zoli|lunch | 10:01 | |
| *** markvoelker has quit IRC | 10:04 | |
| *** betherly-afk is now known as betherly_ | 10:06 | |
| *** jamesmcarthur has joined #openstack-infra | 10:10 | |
| *** kjackal has quit IRC | 10:10 | |
| *** kjackal_ has quit IRC | 10:10 | |
| *** gongysh has quit IRC | 10:15 | |
| *** kjackal_ has joined #openstack-infra | 10:16 | |
| *** kjackal has joined #openstack-infra | 10:16 | |
| *** kiennt26 has quit IRC | 10:17 | |
| *** jamesmcarthur has quit IRC | 10:18 | |
| *** gfidente has joined #openstack-infra | 10:19 | |
| *** gfidente has quit IRC | 10:19 | |
| *** gfidente has joined #openstack-infra | 10:19 | |
| *** gouthamr has joined #openstack-infra | 10:20 | |
| *** jaosorior has joined #openstack-infra | 10:22 | |
| *** dmellado has joined #openstack-infra | 10:24 | |
| *** alexchadin has joined #openstack-infra | 10:25 | |
| *** e0ne has quit IRC | 10:27 | |
| *** alexchadin has quit IRC | 10:30 | |
| *** alexchadin has joined #openstack-infra | 10:30 | |
| *** alexchadin has quit IRC | 10:31 | |
| *** alexchadin has joined #openstack-infra | 10:31 | |
| *** alexchadin has quit IRC | 10:31 | |
| *** alexchadin has joined #openstack-infra | 10:32 | |
| *** alexchadin has quit IRC | 10:32 | |
| *** e0ne has joined #openstack-infra | 10:32 | |
| *** edmondsw has joined #openstack-infra | 10:35 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-askbot master: Fix apache compatibility for Xenial https://review.openstack.org/585196 | 10:38 |
| *** stevebaker has joined #openstack-infra | 10:38 | |
| *** edmondsw has quit IRC | 10:40 | |
| *** e0ne has quit IRC | 10:40 | |
| *** alexchadin has joined #openstack-infra | 10:41 | |
| *** dhajare has joined #openstack-infra | 10:44 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on future parser for ask-staging.o.o https://review.openstack.org/585205 | 10:44 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for ask.o.o https://review.openstack.org/585206 | 10:44 |
| *** hashar has quit IRC | 10:47 | |
| *** quiquell has quit IRC | 10:48 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Remove stackalytics.o.o from site.pp https://review.openstack.org/585208 | 10:49 |
| *** rcernin_ has quit IRC | 10:52 | |
| *** alexchadin has quit IRC | 10:54 | |
| *** quiquell has joined #openstack-infra | 10:55 | |
| *** alexchad_ has joined #openstack-infra | 10:57 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-askbot master: Fix apache compatibility for Xenial https://review.openstack.org/585196 | 10:58 |
| *** markvoelker has joined #openstack-infra | 11:01 | |
| *** alexchad_ has quit IRC | 11:02 | |
| *** rosmaita has joined #openstack-infra | 11:05 | |
| *** alexchadin has joined #openstack-infra | 11:06 | |
| *** panda is now known as panda|lunch | 11:10 | |
| *** yamamoto_ has joined #openstack-infra | 11:14 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Turn off binary symlink in nodejs module https://review.openstack.org/585291 | 11:15 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add workaround for ethercalc bug https://review.openstack.org/585292 | 11:15 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add idempotency test https://review.openstack.org/585293 | 11:15 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add test for ethercalc::apache https://review.openstack.org/585294 | 11:15 |
| *** dhajare has quit IRC | 11:17 | |
| *** gongysh has joined #openstack-infra | 11:21 | |
| *** gcb has quit IRC | 11:23 | |
| *** yamamoto has joined #openstack-infra | 11:25 | |
| openstackgerrit | Merged openstack/diskimage-builder master: block-device lvm: fix umount phase https://review.openstack.org/503958 | 11:26 |
| *** yamamoto_ has quit IRC | 11:27 | |
| *** udesale has quit IRC | 11:28 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 11:29 | |
| *** yamamoto has quit IRC | 11:29 | |
| *** yamamoto has joined #openstack-infra | 11:31 | |
| *** zoli|lunch is now known as zoli | 11:31 | |
| *** yamamoto has quit IRC | 11:32 | |
| *** yamamoto has joined #openstack-infra | 11:32 | |
| *** markvoelker has quit IRC | 11:34 | |
| *** s-shiono has quit IRC | 11:34 | |
| *** agopi is now known as agopi|brb | 11:36 | |
| *** zhangfei has quit IRC | 11:38 | |
| *** wolverineav has joined #openstack-infra | 11:39 | |
| *** agopi|brb has quit IRC | 11:41 | |
| *** r-daneel has quit IRC | 11:46 | |
| *** rh-jelabarre has joined #openstack-infra | 11:47 | |
| *** yamamoto has quit IRC | 11:47 | |
| *** e0ne has joined #openstack-infra | 11:53 | |
| *** dhill_ has joined #openstack-infra | 11:53 | |
| *** linkmark has joined #openstack-infra | 11:54 | |
| *** edmondsw has joined #openstack-infra | 11:57 | |
| *** yamamoto has joined #openstack-infra | 11:57 | |
| *** e0ne has quit IRC | 11:58 | |
| *** yamamoto has quit IRC | 11:59 | |
| *** yamamoto has joined #openstack-infra | 12:01 | |
| *** boden has joined #openstack-infra | 12:01 | |
| *** eharney has quit IRC | 12:04 | |
| *** e0ne has joined #openstack-infra | 12:07 | |
| *** jiapei has quit IRC | 12:10 | |
| *** katkapilatova has joined #openstack-infra | 12:10 | |
| *** r-daneel has joined #openstack-infra | 12:11 | |
| *** trown|outtypewww is now known as trown | 12:11 | |
| *** kjackal_ has quit IRC | 12:11 | |
| *** kjackal has quit IRC | 12:12 | |
| *** markvoelker has joined #openstack-infra | 12:12 | |
| *** strigazi has quit IRC | 12:13 | |
| *** strigazi has joined #openstack-infra | 12:14 | |
| *** r-daneel has quit IRC | 12:14 | |
| *** eharney has joined #openstack-infra | 12:17 | |
| *** armaan has quit IRC | 12:26 | |
| *** armaan has joined #openstack-infra | 12:26 | |
| *** yamamoto has quit IRC | 12:27 | |
| *** mriedem has joined #openstack-infra | 12:28 | |
| *** psachin` has quit IRC | 12:30 | |
| *** yamamoto has joined #openstack-infra | 12:30 | |
| sshnaidm|ruck | is mirror.us-west-1.packethost.openstack.org:8080 unavailable? | 12:31 |
| sshnaidm|ruck | pabelanger, clarkb ^^ | 12:31 |
| *** kjackal has joined #openstack-infra | 12:31 | |
| *** kjackal_ has joined #openstack-infra | 12:31 | |
| *** panda|lunch is now known as panda | 12:36 | |
| *** camunoz has joined #openstack-infra | 12:36 | |
| *** eharney has quit IRC | 12:37 | |
| *** rlandy has joined #openstack-infra | 12:39 | |
| *** r-mibu has joined #openstack-infra | 13:04 | |
| *** kgiusti has joined #openstack-infra | 13:05 | |
| openstackgerrit | Carlos Goncalves proposed openstack-infra/project-config master: Switch to publish-openstack-docs-pti in Octavia https://review.openstack.org/585184 | 13:08 |
| *** ianychoi has quit IRC | 13:08 | |
| *** ianychoi has joined #openstack-infra | 13:08 | |
| *** tpsilva has joined #openstack-infra | 13:10 | |
| *** dave-mccowan has joined #openstack-infra | 13:11 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add workaround for ethercalc bug https://review.openstack.org/585292 | 13:13 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add idempotency test https://review.openstack.org/585293 | 13:13 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add test for ethercalc::apache https://review.openstack.org/585294 | 13:13 |
| *** jamesden_ has joined #openstack-infra | 13:14 | |
| *** graphene has quit IRC | 13:14 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add idempotency test https://review.openstack.org/585293 | 13:15 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ethercalc master: Add test for ethercalc::apache https://review.openstack.org/585294 | 13:15 |
| *** agopi|brb has joined #openstack-infra | 13:15 | |
| *** agopi|brb is now known as agopi | 13:15 | |
| *** graphene has joined #openstack-infra | 13:15 | |
| *** dave-mcc_ has joined #openstack-infra | 13:20 | |
| *** dave-mccowan has quit IRC | 13:22 | |
| *** eharney has joined #openstack-infra | 13:24 | |
| *** mriedem is now known as mriedem_away | 13:24 | |
| *** agopi is now known as agopi|brb | 13:25 | |
| *** ihar has joined #openstack-infra | 13:27 | |
| *** agopi|brb has quit IRC | 13:29 | |
| *** agopi|brb has joined #openstack-infra | 13:31 | |
| *** gongysh has quit IRC | 13:36 | |
| *** bdodd has quit IRC | 13:40 | |
| *** ccamacho1 has joined #openstack-infra | 13:43 | |
| *** ccamacho has quit IRC | 13:43 | |
| *** bdodd has joined #openstack-infra | 13:45 | |
| *** mihalis68 has joined #openstack-infra | 13:47 | |
| *** jamesmca_ has joined #openstack-infra | 13:49 | |
| *** shardy has quit IRC | 13:50 | |
| pabelanger | sshnaidm|ruck: unknown do you see job failures? | 13:50 |
| sshnaidm|ruck | pabelanger, yes | 13:51 |
| sshnaidm|ruck | pabelanger, http://logs.openstack.org/87/582387/11/check/tripleo-ci-centos-7-containers-multinode/861e47b/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2018-07-24_11_34_21 | 13:52 |
| *** jamesden_ has quit IRC | 13:52 | |
| *** jamesde__ has joined #openstack-infra | 13:52 | |
| sshnaidm|ruck | http://mirror.us-west-1.packethost.openstack.org:8080/rdo/centos7/95/9e/959e1d7f755ee681b6f23b498d262a9e4dd6326f_4cbb1814/repodata/repomd.xml: [Errno 14] curl#7 - \\"Failed connect to mirror.us-west-1.packethost.openstack.org:8080; No route to host | 13:52 |
| *** bdodd has quit IRC | 13:54 | |
| pabelanger | sshnaidm|ruck: you can also check cacti.o.o to see status | 13:54 |
| pabelanger | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=699&nodeid=node1_699&host_group_data= | 13:54 |
| scas | project renames require gerrit downtime, yes? | 13:54 |
| pabelanger | does look dead | 13:54 |
| *** quiquell has quit IRC | 13:54 | |
| pabelanger | sshnaidm|ruck: can you propose a change to project-config setting packethost max-servers to 0 | 13:55 |
| scas | ENOCAFFEINE s/project/repo/ | 13:55 |
| *** shardy has joined #openstack-infra | 13:56 | |
| sshnaidm|ruck | pabelanger, which repo? | 13:56 |
| pabelanger | sshnaidm|ruck: openstack-infra/project-config | 13:57 |
| scas | for the sake of my question, i suppose both are one and the same | 13:58 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config master: Disable packethost cloud https://review.openstack.org/585369 | 13:59 |
| sshnaidm|ruck | pabelanger, like that? ^^ | 13:59 |
| sshnaidm|ruck | pabelanger, what does tell you it's dead in cacti? | 13:59 |
| pabelanger | sshnaidm|ruck: you can see lack of traffic on snmp graph | 14:00 |
| *** neiloy has joined #openstack-infra | 14:01 | |
| *** sthussey has joined #openstack-infra | 14:01 | |
| pabelanger | actually, let me check openstack api first | 14:01 |
| sshnaidm|ruck | much easier would be have up/down graph.. | 14:01 |
| *** r-daneel has joined #openstack-infra | 14:02 | |
| *** ykarel is now known as ykarel|away | 14:04 | |
| pabelanger | trying to reboot mirror01.us-west-1.packethost.openstack.org via openstack API | 14:06 |
| *** hamzy_ has quit IRC | 14:07 | |
| pabelanger | is john_studarus on IRC? | 14:08 |
| pabelanger | okay, mirror.us-west-1.packethost.openstack.org came back online after a reboot | 14:10 |
| pabelanger | checking logs | 14:10 |
| *** mriedem_away is now known as mriedem | 14:11 | |
| *** ykarel|away has quit IRC | 14:12 | |
| pabelanger | sshnaidm|ruck: seems we don't need 585369 right now | 14:12 |
| sshnaidm|ruck | pabelanger, ok | 14:13 |
| pabelanger | #status log mirror.us-west-1.packethost.openstack.org reboot via openstack API due to not responding to SSH / HTTP requests. Server now back online. | 14:13 |
| openstackstatus | pabelanger: finished logging | 14:13 |
| pabelanger | clarkb: ^looks like something to debug with john, not sure his IRC handle | 14:14 |
| *** felipemonteiro has joined #openstack-infra | 14:17 | |
| *** mriedem1 has joined #openstack-infra | 14:18 | |
| *** jamesmca_ has quit IRC | 14:20 | |
| *** mriedem has quit IRC | 14:21 | |
| *** mdrabe has joined #openstack-infra | 14:21 | |
| *** john_studarus has joined #openstack-infra | 14:21 | |
| *** wolverineav has quit IRC | 14:23 | |
| *** wolverineav has joined #openstack-infra | 14:23 | |
| john_studarus | pabelanger You said there's an issue with the mirror? | 14:24 |
| *** mriedem1 is now known as mriedem | 14:24 | |
| *** bdodd has joined #openstack-infra | 14:25 | |
| john_studarus | Also, all the Grafana API stats for Packet have disappeared. What's the API call behind those graphs? | 14:25 |
| *** yamamoto has quit IRC | 14:25 | |
| *** yamamoto has joined #openstack-infra | 14:27 | |
| *** felipemonteiro has quit IRC | 14:27 | |
| pabelanger | john_studarus: thanks, we seem to lost network access to the mirror this morning: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64951&rra_id=all curious if you see anything on your side | 14:27 |
| pabelanger | john_studarus: we had to reboot it via openstack APi and seem to bring it back online | 14:27 |
| *** yamamoto has quit IRC | 14:27 | |
| *** yamamoto has joined #openstack-infra | 14:27 | |
| pabelanger | john_studarus: which URL are you using for grafana? | 14:28 |
| pabelanger | http://grafana.openstack.org/d/U462abNik/nodepool-packethost | 14:28 |
| pabelanger | we also seem to be having build issues for nodes? | 14:28 |
| *** eharney_ has joined #openstack-infra | 14:28 | |
| john_studarus | Create Server, List Server, etc - all those graphs are empty | 14:29 |
| *** jamesmcarthur has joined #openstack-infra | 14:29 | |
| *** katkapilatova has left #openstack-infra | 14:29 | |
| john_studarus | so 11:30am UTC mirror stopped working? | 14:30 |
| *** eharney has quit IRC | 14:30 | |
| *** links has quit IRC | 14:31 | |
| pabelanger | john_studarus: I'm guessing something is wrong with cloud? | 14:31 |
| pabelanger | we don't seem to be getting responses anymore | 14:31 |
| *** ralonsoh_ has joined #openstack-infra | 14:31 | |
| pabelanger | john_studarus: yah, about 11:30 utc | 14:31 |
| corvus | john_studarus, pabelanger: re grafana, all nodepool api graphs stop around the same time, which is around when we restarted the launchers yesterday | 14:32 |
| corvus | john_studarus, pabelanger: re grafana, all nodepool api graphs stop around the same time, which is around when we restarted the launchers yesterday | 14:32 |
| corvus | sorry | 14:32 |
| pabelanger | ah, I was just looking into that | 14:32 |
| corvus | john_studarus, pabelanger, Shrews, clarkb: i think we have a nodepool regression | 14:32 |
| *** ralonsoh has quit IRC | 14:33 | |
| john_studarus | so yesterday I took a hypervisor out of service that had disk space issues about 11am PDT | 14:33 |
| john_studarus | it didn't have any VMs on it | 14:33 |
| pabelanger | corvus: agree | 14:35 |
| pabelanger | john_studarus: yah, don't think the mirror was on that server | 14:36 |
| john_studarus | so both the Limestone and Packet Grafana graphs show nothing under the Create Server, Get Server, Delete Server | 14:36 |
| john_studarus | OVH grafana charts are the same way so that's a different issue then | 14:37 |
| pabelanger | john_studarus: yah, there is a regression in nodepool it looks like, we restarted it yesterday | 14:37 |
| *** armaan has quit IRC | 14:37 | |
| *** armaan has joined #openstack-infra | 14:38 | |
| john_studarus | so let me focus on why the mirror went offline - just wanted to make sure there wasn't a larger API issue | 14:38 |
| Shrews | corvus: pabelanger: clarkb: having to deal with something here atm... any idea what's causing the regression? | 14:39 |
| pabelanger | https://git.zuul-ci.org/cgit/nodepool/tree/nodepool/task_manager.py#n78 might not be triggering? | 14:40 |
| pabelanger | that seems to be a recent change for openstacksdk | 14:40 |
| corvus | Shrews, pabelanger: moving to #zuul | 14:40 |
| *** yamamoto has quit IRC | 14:43 | |
| *** janki has joined #openstack-infra | 14:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Support parent job pause https://review.openstack.org/585389 | 14:46 |
| *** jamesmcarthur has quit IRC | 14:46 | |
| john_studarus | so the mirror, along with 6 other VM were "Instance is paused unexpectedly. Ignore." at 11:57 | 14:48 |
| *** rpioso|afk is now known as rpioso | 14:49 | |
| *** hamzy_ has joined #openstack-infra | 14:50 | |
| pabelanger | john_studarus: possible something paused them? | 14:50 |
| *** jamesmcarthur has joined #openstack-infra | 14:51 | |
| john_studarus | checking if these 7 have something in common... | 14:51 |
| corvus | "Ignore", i like that. if only we could. :) | 14:51 |
| john_studarus | "While synchronizing instance power states, found 11 instances in the database and 18 instances on the hypervisor." | 14:52 |
| *** hamzy_ has quit IRC | 14:52 | |
| john_studarus | so that's the 7 - there were on the hypervisor but not in the database | 14:52 |
| *** hamzy_ has joined #openstack-infra | 14:52 | |
| *** jamesmcarthur has quit IRC | 14:53 | |
| corvus | john_studarus, pabelanger: it looks like the missing api graphs are a regression on nodepool's part and unrelated to any other issue. i believe everything is working as expected in nodepool, we just inadvertently changed the metric names in a recent patch. we'll work on a fix. | 14:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Make statsd key look like the keys from shade https://review.openstack.org/585397 | 14:53 |
| *** yamamoto has joined #openstack-infra | 14:54 | |
| *** udesale has joined #openstack-infra | 14:56 | |
| *** ralonsoh has joined #openstack-infra | 14:57 | |
| clarkb | john_studarus: pabelanger: mirror was built before the networking was redone. Then when networking stuff was finished I simply rebooted the mirror node and it came back and was happy (I did this through the openstack api) | 14:59 |
| clarkb | john_studarus: pabelanger I guess nova had lost track of it despite being managed through the api? | 15:00 |
| *** eernst has joined #openstack-infra | 15:00 | |
| *** hongbin has joined #openstack-infra | 15:00 | |
| clarkb | I guess pabelanger rebooted it through the api as well | 15:00 |
| *** ralonsoh_ has quit IRC | 15:01 | |
| *** cshastri has quit IRC | 15:03 | |
| clarkb | john_studarus: is this the sort of thing where having us rebuild the instance would be helpful to sync of db state? | 15:03 |
| *** rcernin_ has joined #openstack-infra | 15:04 | |
| *** jcoufal has joined #openstack-infra | 15:04 | |
| john_studarus | let me open a ticket with the Platform9 folks and see if they've seen this before | 15:06 |
| john_studarus | I'm still tracking down the other six VMs that had the same issue | 15:06 |
| john_studarus | or rather, were listed in that same warning message | 15:06 |
| pabelanger | clarkb: I did, reboot api | 15:07 |
| pabelanger | john_studarus: other question I had, can you confirm the compute nodes are not converting the glance images into raw format? I don't think this cloud is using ceph right? | 15:07 |
| clarkb | pabelanger: nova default is to use raw even without ceph fwiw | 15:08 |
| *** hongbin has quit IRC | 15:08 | |
| john_studarus | there's no ceph | 15:08 |
| *** hongbin has joined #openstack-infra | 15:08 | |
| pabelanger | clarkb: yah, I suspect the config was changes to qcow2 and we are force converting them | 15:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Emit 'Manager ran task' log lines https://review.openstack.org/585401 | 15:09 |
| *** alexchadin has quit IRC | 15:10 | |
| AJaeger | config-core, https://review.openstack.org/#/c/583024/ has its dependencies merged, review would be welcome to remove an extra experimental job. | 15:13 |
| *** jamesmcarthur has joined #openstack-infra | 15:17 | |
| *** ccamacho1 has quit IRC | 15:18 | |
| *** udesale has quit IRC | 15:18 | |
| *** mdrabe has quit IRC | 15:19 | |
| *** lpetrut has quit IRC | 15:19 | |
| *** eernst has quit IRC | 15:22 | |
| *** eernst has joined #openstack-infra | 15:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Make statsd key look like the keys from shade https://review.openstack.org/585397 | 15:23 |
| *** eernst has quit IRC | 15:24 | |
| *** eernst has joined #openstack-infra | 15:24 | |
| *** mdrabe has joined #openstack-infra | 15:26 | |
| scas | if the project name i want was once in-use, but moved to the attic, should the attic repo go away? | 15:28 |
| clarkb | scas: we don't delete repos which was one reason for having the attic. Now we just "shutdown" in place | 15:29 |
| *** rcernin_ has quit IRC | 15:30 | |
| scas | makes sense. i'm looking to rename the entry point for the chef project, but i noticed that there was one in the attic of the same name | 15:31 |
| pabelanger | clarkb: john_studarus: https://git.openstack.org/cgit/openstack-infra/puppet-infracloud/commit/?id=9dc122c3ddd852f4572ffff0c6d5ed6f84fa2a96 was the settings we made on infracloud to not to force raw images on compute nodes, if we could confirm on packethost | 15:31 |
| tommylikehu | hi, could anyone take a look at this patch, https://review.openstack.org/#/c/583970/ thanks:) | 15:32 |
| *** kjackal_ has quit IRC | 15:32 | |
| *** pcaruana has quit IRC | 15:32 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed summit event serializer https://review.openstack.org/585405 | 15:33 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed summit event serializer https://review.openstack.org/585405 | 15:33 |
| clarkb | pabelanger: probably need to sort out the networking and mirror going away first. Removing that hypervisor seems to have made networking much more reliable, but had ~6 jobs over last 18 hours with ssh trouble. I'll need to dig up those instance uuids | 15:35 |
| pabelanger | clarkb: agree, just wanted to share info before I forgot | 15:37 |
| *** gyee has joined #openstack-infra | 15:37 | |
| scas | for sanity's sake, there are no practical implications from reusing a name in the attic where that repo was last touched in 2013? | 15:39 |
| *** lpetrut has joined #openstack-infra | 15:39 | |
| clarkb | scas: repo names have historically been unique | 15:39 |
| *** felipemonteiro has joined #openstack-infra | 15:39 | |
| clarkb | I think ideally we would continue to maintain that rule | 15:40 |
| clarkb | scas: it would probably help if you could concretely state what you are trying to do | 15:40 |
| scas | yeah. this one is in an odd place. the entry point literally has the word 'repo' in the name because that's how it was done years ago. it's giving me grief trying to standardize | 15:41 |
| scas | openstack/openstack-chef-repo looks odd and nonsensical, even though 'chef-repo' meant something in yesteryear | 15:41 |
| scas | i've had questions about the naming itself. modern chef documentation says to use freeform names, not emphasizing the old terms | 15:42 |
| *** rlandy is now known as rlandy|brb | 15:42 | |
| scas | so when someone comes along, their first question is usually 'why do you call it repo?' | 15:42 |
| pabelanger | clarkb: mordred: corvus: I was looking at zuul-executors last night, and trying to see how we can increase performance. I've noticed for reasons I don't know yet, we are doing a lot of swapping. EG: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64005&rra_id=all Given that /var/lib/zuul and swap share the same drive, I wonder if moving them to separate drives will help IO a | 15:44 |
| pabelanger | little. Also wonder if we should make try testing a SSD volume, to compare speeds. | 15:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Make statsd key look like the keys from shade https://review.openstack.org/585397 | 15:44 |
| clarkb | pabelanger: or we need more memory | 15:45 |
| pabelanger | clarkb: ya, was also considering maybe adjusting our memory governor on a zuul-executor (tuning) to see if that helps | 15:46 |
| pabelanger | right now it is 5%, IIRC | 15:46 |
| *** dhajare has joined #openstack-infra | 15:47 | |
| pabelanger | clarkb: was also looking into vm.swappiness, but still don't fully understand it | 15:47 |
| scas | from what i'm gathering, renaming the repo is more friction than refamiliarizing myself with 301 redirects for the docs i'm going to start publishing | 15:49 |
| clarkb | scas: you'd like to rename openstack/openstack-chef-repo to a name that already exists in the attic? | 15:49 |
| clarkb | was the name in the attic also the original name used that go replaced by openstack-chef-repo? | 15:50 |
| scas | openstack-attic/openstack-chef exists. it used to be the entry for the chef cookbooks, but, yes, it was replaced entirely by openstack-chef-repo | 15:50 |
| scas | i was going to go with the totally unused chef-openstack. when i started digging in the yamls, i noticed everything else said openstack-chef | 15:51 |
| *** eharney_ is now known as eharney | 15:51 | |
| scas | short of a whole project rename, that seemed like a thread to follow | 15:51 |
| *** jamesmcarthur has quit IRC | 15:52 | |
| *** lpetrut has quit IRC | 15:52 | |
| clarkb | My initial thought on how to do that would be to move openstack-chef out of the attic, push history update to it explaining the back and forth, then retire openstack-chef-repo | 15:52 |
| clarkb | then it is only one project rename instead of multiple | 15:53 |
| *** jamesmcarthur has joined #openstack-infra | 15:53 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:54 | |
| clarkb | pabelanger: I'm guessing most of the memory use is ansible processes too? so directly tied to number of jobs we can run | 15:54 |
| scas | indeed. that makes more sense. | 15:54 |
| corvus | pabelanger: does the ram governor ever trigger? because yeah, i would think that kind of swap activity is exactly the sort of thing we'd be trying to prevent. | 15:54 |
| pabelanger | corvus: I don't believe so, looking at logs now. Is that something we could add to statsd to better see (if governor is active and which) | 15:56 |
| corvus | pabelanger: i thought it was logged | 15:57 |
| *** felipemonteiro has quit IRC | 15:57 | |
| corvus | pabelanger, clarkb: still -- the swap activity graph isn't huge. it's got an average io rate of like 300kbps. it might be okay to ignore it. | 15:58 |
| pabelanger | corvus: yah, see them in debug log, but nothing for memory | 15:58 |
| pabelanger | just load | 15:59 |
| corvus | and iowait isn't huge. | 15:59 |
| clarkb | I want to say linux will use swap for caching files too as its (slightly) quicker than going to the fs proper for those files | 16:00 |
| clarkb | since we don't have a lot of free mem on the servers it may be doing that for git and ansible files | 16:00 |
| corvus | pabelanger: ze11 seems to be running many more builds than the others | 16:00 |
| pabelanger | corvus: yah, seems to be handing them better for some reason | 16:01 |
| pabelanger | handling* | 16:01 |
| *** jpich has quit IRC | 16:01 | |
| pabelanger | check server to make sure I used right flavor | 16:01 |
| pabelanger | checking* | 16:01 |
| pabelanger | performance1-8, so that is correct | 16:02 |
| *** agopi|brb has quit IRC | 16:03 | |
| *** lpetrut has joined #openstack-infra | 16:03 | |
| *** agopi|brb has joined #openstack-infra | 16:03 | |
| pabelanger | corvus: we are running a newer kernel on ze11.o.o | 16:04 |
| pabelanger | Linux ze01 4.13.0-32-generic #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | 16:04 |
| pabelanger | Linux ze11 4.15.0-29-generic #31~16.04.1-Ubuntu SMP Wed Jul 18 08:54:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | 16:05 |
| corvus | maybe it'll just take a while to equalize. or maybe the kernel is better. or maybe the hardware is better. (or maybe it has fewer vm neighbors) | 16:05 |
| pabelanger | either way, I like the boost we are getting :) | 16:06 |
| corvus | ya | 16:06 |
| corvus | we still had a period of queueing between 1100 and 1330 | 16:08 |
| pabelanger | if we are up for experimenting, could consider stopping one of the older zuul-executors, updating kernel and see if that helps. maybe new hwe kernel with more improvements | 16:08 |
| corvus | if we have to restart for some other reason, let's. but let's not do it just for that right now, due to the backlog | 16:09 |
| *** e0ne has quit IRC | 16:10 | |
| pabelanger | wfm | 16:10 |
| clarkb | pabelanger: I was thinking about the rackdns ssl verification problem more and realized that the other openstack clients could talk to rax identity just fine. I think this is because python requests vendors certs. If rackdns uses requests we might be able to just update that library to get rackdns working. The problem is that I'm not sure how w einstall rackdns (from source maybe) and would be | 16:10 |
| clarkb | surprised if it works with modern requests (if it uses requests at all) | 16:10 |
| *** graphene has quit IRC | 16:10 | |
| *** graphene has joined #openstack-infra | 16:11 | |
| clarkb | requests is installed in that venv | 16:12 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy-tempest-dsvm-nova-os-vif https://review.openstack.org/583024 | 16:14 |
| pabelanger | 104 jobs in ze11.o.o, haven't see that number before! | 16:15 |
| clarkb | mordred: do you know how we install rackdns was it from source and not from pypi? I can't find the package on pypi | 16:15 |
| clarkb | but reading requests docs before requests 2.16 they bundled certs and would only update certs if you updated requests. You can also install certifi | 16:16 |
| clarkb | pabelanger: ^ all that to say I think updating requests and/or certifi may fix the dns problems we had | 16:16 |
| pabelanger | clarkb: ah, good idea, didn't think of requests | 16:16 |
| *** trown is now known as trown|lunch | 16:16 | |
| clarkb | But I want to create a new venv for that so we don't break (the already somewhat broken) existing venv | 16:16 |
| pabelanger | ++ | 16:17 |
| clarkb | pabelanger: pretty sure this is why you could boot instances against rax because requests was more up to date in that env | 16:17 |
| *** panda is now known as panda|off | 16:18 | |
| *** zoli is now known as zoli|gone | 16:19 | |
| *** zoli|gone is now known as zoli | 16:19 | |
| clarkb | pabelanger: /home/clarkb/rackdns-venv exists now but I don't have any records to update currently. I expect this will work though | 16:23 |
| *** lucasagomes is now known as lucas-afk | 16:24 | |
| *** ccamacho has joined #openstack-infra | 16:24 | |
| *** ccamacho has quit IRC | 16:25 | |
| *** ccamacho has joined #openstack-infra | 16:25 | |
| *** eernst has quit IRC | 16:26 | |
| *** eernst has joined #openstack-infra | 16:26 | |
| pabelanger | ack | 16:26 |
| *** rlandy|brb is now known as rlandy | 16:28 | |
| *** yamamoto has quit IRC | 16:33 | |
| openstackgerrit | Samuel Cassiba proposed openstack-infra/project-config master: Unretire the openstack-chef project https://review.openstack.org/585471 | 16:35 |
| openstackgerrit | Samuel Cassiba proposed openstack-infra/project-config master: Unretire the openstack-chef project https://review.openstack.org/585471 | 16:36 |
| *** vivsoni has quit IRC | 16:36 | |
| *** vivsoni has joined #openstack-infra | 16:36 | |
| *** sshnaidm|ruck is now known as sshnaidm|bbl | 16:37 | |
| *** dtantsur is now known as dtantsur|afk | 16:39 | |
| *** jlk has joined #openstack-infra | 16:48 | |
| *** jlk has joined #openstack-infra | 16:48 | |
| *** rlandy is now known as rlandy|afk | 16:49 | |
| *** eernst has quit IRC | 16:49 | |
| *** eernst has joined #openstack-infra | 16:51 | |
| *** derekh has quit IRC | 16:55 | |
| *** myoung is now known as myoung|lunch | 16:58 | |
| *** ralonsoh has quit IRC | 17:02 | |
| *** agopi|brb has quit IRC | 17:03 | |
| *** efried is now known as efried_pto | 17:05 | |
| *** yamahata has quit IRC | 17:11 | |
| *** iyamahat has quit IRC | 17:13 | |
| *** gfidente has quit IRC | 17:13 | |
| *** slaweq has quit IRC | 17:16 | |
| *** lpetrut has quit IRC | 17:16 | |
| *** ramishra has quit IRC | 17:17 | |
| *** tesseract has quit IRC | 17:17 | |
| *** iyamahat has joined #openstack-infra | 17:18 | |
| *** ccamacho has quit IRC | 17:20 | |
| *** armaan has quit IRC | 17:25 | |
| *** iyamahat_ has joined #openstack-infra | 17:25 | |
| *** iyamahat has quit IRC | 17:29 | |
| *** e0ne has joined #openstack-infra | 17:30 | |
| *** felipemonteiro__ has joined #openstack-infra | 17:33 | |
| *** felipemonteiro_ has quit IRC | 17:33 | |
| *** trown|lunch is now known as trown | 17:37 | |
| *** shardy has quit IRC | 17:38 | |
| *** jamesmcarthur has quit IRC | 17:39 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:42 | |
| *** yamahata has joined #openstack-infra | 17:43 | |
| *** gary_perkins has quit IRC | 17:43 | |
| *** felipemonteiro__ has quit IRC | 17:46 | |
| *** felipemonteiro__ has joined #openstack-infra | 17:47 | |
| *** felipemonteiro_ has quit IRC | 17:47 | |
| *** dhajare has quit IRC | 17:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-tempest-dsvm-nova-os-vif https://review.openstack.org/585493 | 17:49 |
| *** felipemonteiro__ has quit IRC | 17:51 | |
| *** r-mibu has quit IRC | 17:51 | |
| *** gary_perkins has joined #openstack-infra | 17:51 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-tempest-dsvm-nova-os-vif https://review.openstack.org/585493 | 17:51 |
| *** jamesmcarthur has joined #openstack-infra | 17:54 | |
| *** hamzy_ has quit IRC | 17:55 | |
| openstackgerrit | Samuel Cassiba proposed openstack-infra/project-config master: Unretire the openstack-chef project https://review.openstack.org/585471 | 17:56 |
| *** ekhugen has quit IRC | 17:58 | |
| *** jamesmcarthur has quit IRC | 17:58 | |
| *** ijw has joined #openstack-infra | 18:00 | |
| *** ekhugen has joined #openstack-infra | 18:00 | |
| *** ijw has quit IRC | 18:08 | |
| *** jamesden_ has joined #openstack-infra | 18:12 | |
| *** cfriesen_ has joined #openstack-infra | 18:12 | |
| *** jamesde__ has quit IRC | 18:13 | |
| cfriesen_ | maybe a weird question, but is there a way for normal users to determine who the core reviewers are for a given project on review.openstack.org? | 18:13 |
| *** timburke_ is now known as timburke | 18:14 | |
| *** e0ne has quit IRC | 18:14 | |
| *** jlk has left #openstack-infra | 18:14 | |
| rcarrillocruz | yeah, that's kept revision control | 18:14 |
| *** slaweq has joined #openstack-infra | 18:15 | |
| *** zerick has quit IRC | 18:16 | |
| *** mriedem1 has joined #openstack-infra | 18:16 | |
| *** zerick_ has joined #openstack-infra | 18:16 | |
| *** mriedem has quit IRC | 18:16 | |
| rcarrillocruz | or not, maybe just the groups | 18:16 |
| rcarrillocruz | https://github.com/openstack-infra/project-config/commit/9f306e087710eff918dc925e8dcd32a60cbb49c0 | 18:16 |
| rcarrillocruz | i think members are just thru UI | 18:17 |
| clarkb | cfriesen_: yes if you are logged in you can see groul membsrship in Gerrit | 18:18 |
| *** bobh has joined #openstack-infra | 18:19 | |
| cfriesen_ | specifically, I can see the members of "starlingx-core" at https://review.openstack.org/#/admin/groups/1920,members, but I'm trying to find a way to see who the actual code owners are for each sub-project. Like https://review.openstack.org/#/q/project:openstack/stx-integ for instance. | 18:20 |
| clarkb | cfriesen_: if you go projects -> list -> stx-integ -> access you will see what groups have which permissions. https://review.openstack.org/#/admin/projects/openstack/stx-integ,access | 18:23 |
| *** agopi|brb has joined #openstack-infra | 18:23 | |
| clarkb | cfriesen_: in this case starlingx-core and starlingx-release | 18:24 |
| clarkb | cfriesen_: is that the info you are looking for? | 18:26 |
| cfriesen_ | clarkb: yes, thanks | 18:26 |
| *** mriedem1 is now known as mriedem | 18:27 | |
| *** agopi|brb is now known as agopi | 18:27 | |
| *** eernst has quit IRC | 18:31 | |
| *** eernst has joined #openstack-infra | 18:33 | |
| *** eernst has quit IRC | 18:33 | |
| *** eernst has joined #openstack-infra | 18:33 | |
| *** eernst has quit IRC | 18:35 | |
| *** eernst has joined #openstack-infra | 18:36 | |
| *** myoung|lunch is now known as myoung | 18:37 | |
| *** hamzy_ has joined #openstack-infra | 18:39 | |
| *** zerick_ has quit IRC | 18:40 | |
| *** zerick has joined #openstack-infra | 18:41 | |
| *** eernst has quit IRC | 18:44 | |
| *** eernst has joined #openstack-infra | 18:46 | |
| *** mihalis68 has quit IRC | 18:46 | |
| *** eernst has quit IRC | 18:47 | |
| *** eernst has joined #openstack-infra | 18:47 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/infra-specs master: Make update-config-management priority effort https://review.openstack.org/585514 | 18:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Make statsd key look like the keys from shade https://review.openstack.org/585397 | 18:57 |
| clarkb | cfriesen_: you can get all of that from the api too, but I'm not sure what all the steps for that are | 18:57 |
| *** armaan has joined #openstack-infra | 18:57 | |
| *** bobh has quit IRC | 18:58 | |
| clarkb | Meeting time over in #openstack-meeting | 18:59 |
| *** rlandy|afk is now known as rlandy | 19:03 | |
| *** alex_xu has quit IRC | 19:03 | |
| *** alex_xu has joined #openstack-infra | 19:07 | |
| *** jamesmcarthur has joined #openstack-infra | 19:11 | |
| *** jamesmca_ has joined #openstack-infra | 19:12 | |
| *** yamamoto has joined #openstack-infra | 19:14 | |
| *** jamesmcarthur has quit IRC | 19:16 | |
| *** eharney has quit IRC | 19:17 | |
| *** yamamoto has quit IRC | 19:19 | |
| *** armaan has quit IRC | 19:24 | |
| *** armaan has joined #openstack-infra | 19:25 | |
| *** mdrabe has quit IRC | 19:27 | |
| *** armaan has quit IRC | 19:29 | |
| *** cfriesen_ has left #openstack-infra | 19:29 | |
| *** rwsu has quit IRC | 19:34 | |
| *** jamesmca_ has quit IRC | 19:34 | |
| *** jamesmcarthur has joined #openstack-infra | 19:37 | |
| *** pguimaraes has quit IRC | 19:39 | |
| *** agopi is now known as agopi|brb | 19:42 | |
| smcginnis | Looks like a release went into a blackhole back in June. | 19:56 |
| smcginnis | Would someone be able to reenqueue https://review.openstack.org/#/c/573456/ for me? | 19:56 |
| mriedem | clarkb: seems we're not tracking this type of timeout failure in e-r http://logs.openstack.org/19/584319/1/gate/tempest-full/f22ef43/job-output.txt.gz#_2018-07-23_00_57_14_222550 - does that look familiar | 19:57 |
| *** rtjure has quit IRC | 19:57 | |
| *** jamesmcarthur has quit IRC | 19:58 | |
| *** jcoufal has quit IRC | 19:58 | |
| mriedem | looks like lots of tempest network scenario tests are slow, but not marked as slow, | 19:58 |
| mriedem | tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic [176.416681s] ... ok | 19:58 |
| mriedem | gmann: ^ | 19:59 |
| clarkb | mriedem: http://status.openstack.org/elastic-recheck/#1686542 is the existing check that tries to check for that, probably need ot update for that signature though | 19:59 |
| mriedem | yeah i was thinking that should catch it but way too generic | 20:00 |
| mriedem | i could open a bug against tempest to look at why these scenario tests are taking so long, | 20:00 |
| mriedem | and if they should be marked as slow | 20:00 |
| clarkb | ya the issue is timeouts happen all over, useful to know when they happen but hard to drill in using that query | 20:00 |
| smcginnis | I believe this would be the reenqueue commands for the osc-placement release: http://paste.openstack.org/show/726552/ | 20:01 |
| *** jamesmcarthur has joined #openstack-infra | 20:01 | |
| clarkb | smcginnis: thanks I'll work to queue that up | 20:02 |
| *** agopi|brb is now known as agopi | 20:02 | |
| clarkb | smcginnis: I think we actually need to run the the jobs that create the tag then that will auto enqueue the jobs for the tag | 20:02 |
| clarkb | since the tag doesn't exist yet | 20:03 |
| smcginnis | clarkb: Ah, you're right. | 20:03 |
| clarkb | smcginnis: now to figure out where that happens | 20:03 |
| *** eharney has joined #openstack-infra | 20:04 | |
| clarkb | smcginnis: looks like release-post is what we want to enqueue | 20:04 |
| smcginnis | clarkb: Yeah, then that should trigger the other ones I believe. | 20:05 |
| clarkb | yup | 20:05 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for tempest scenario run timeout bug 1783405 https://review.openstack.org/585534 | 20:06 |
| openstack | bug 1783405 in tempest "Networking scenario tests are randomly timing out" [Undecided,New] https://launchpad.net/bugs/1783405 | 20:06 |
| smcginnis | Something wrong with http://zuul.openstack.org/ ? | 20:07 |
| clarkb | my zuul enqueue is just hanging | 20:08 |
| smcginnis | Yeah, web interface isn't looking happy. | 20:09 |
| pabelanger | we seem to have stopped launching jobs looking at grafana.o.o | 20:09 |
| pabelanger | err: starting builgs | 20:10 |
| clarkb | zuul scheduler just finished building a new layout | 20:10 |
| *** sshnaidm|bbl is now known as sshnaidm|ruck | 20:11 | |
| clarkb | it seems very busy producing another one immediately after | 20:11 |
| clarkb | and now its back again | 20:13 |
| corvus | that's most of what it does these days | 20:13 |
| corvus | but it was doing it rather slowly | 20:13 |
| clarkb | anyays I decided to enqueue mordreds nodepool change since I was enqueing things anyway | 20:13 |
| clarkb | thats in, now to try enqueing smcginnis' release-post change | 20:13 |
| corvus | now it seems to be doing things more quickly | 20:13 |
| clarkb | corvus: enqueue-ref --ref flag is for the git ref like master and --newrev is for the commit sha? | 20:15 |
| corvus | clarkb: yeah; should be examples in bash history | 20:15 |
| smcginnis | Thanks clarkb | 20:16 |
| clarkb | smcginnis: --ref master --newrev 6a346c82f6821f100fc985d83c31b92a359ca076 look right to you? | 20:16 |
| clarkb | I can cross check against jobs that ran in the past | 20:17 |
| *** jamesmcarthur has quit IRC | 20:17 | |
| clarkb | corvus: only for tags unfortunately | 20:17 |
| clarkb | --ref refs/heads/master looks like according to jobs that ran | 20:18 |
| *** rcernin_ has joined #openstack-infra | 20:18 | |
| corvus | clarkb: yeah, if that's what's in the zuul inventory, that's what you want. | 20:18 |
| smcginnis | clarkb: Yeah, I think that looks right. | 20:18 |
| clarkb | and oldrev is the commit prior? I'm double checking that against a job that ran | 20:19 |
| clarkb | smcginnis: ok should be enqueued now | 20:20 |
| *** jtomasek has quit IRC | 20:20 | |
| smcginnis | Awesome, watching its progress. Thanks for the help. | 20:20 |
| smcginnis | melwitt: osc-placement release should be going through soon. ^ | 20:21 |
| *** jamesmcarthur has joined #openstack-infra | 20:21 | |
| clarkb | mordred: care to review https://review.openstack.org/#/c/584638/3 really uickly to update the ansible group for groups.o.o ? | 20:22 |
| clarkb | mordred: I can babysit if you want ot approve to | 20:23 |
| *** jamesmcarthur has quit IRC | 20:25 | |
| mordred | clarkb: done | 20:28 |
| *** camunoz has quit IRC | 20:28 | |
| *** janki has quit IRC | 20:31 | |
| clarkb | tyty | 20:31 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Fix typo in ansible-role-openstack-operations project https://review.openstack.org/585543 | 20:38 |
| *** d0ugal has quit IRC | 20:44 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Make statsd key look like the keys from shade https://review.openstack.org/585397 | 20:44 |
| *** gary_perkins has quit IRC | 20:45 | |
| *** gary_perkins has joined #openstack-infra | 20:46 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Add variables to project https://review.openstack.org/584230 | 20:48 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Use definition list for job status https://review.openstack.org/584780 | 20:48 |
| *** agopi is now known as agopi|brb | 20:51 | |
| clarkb | smcginnis: looks like the release-post jobs ran and now things are queued in tag | 20:53 |
| clarkb | Shrews: mordred corvus I'm going to restart nodepool launchers and builders just as soon as I confirm that the installed version is for the change that merged above | 20:55 |
| *** hamzy_ has quit IRC | 20:55 | |
| mordred | clarkb: ++ | 20:55 |
| corvus | clarkb: thx | 20:55 |
| fungi | clarkb: pabelanger: on the rackdns api issue, i discovered that we need a new api token but our new account in that tenant lacks permission to create one | 20:55 |
| fungi | and i did indeed work around it via the rackspace web dashboard | 20:56 |
| fungi | pretty sure the old api key died with the old tenant account when it was deactivated | 20:56 |
| clarkb | fungi: ok what I found was that ssl verification was failing | 20:56 |
| clarkb | fungi: and that rackdns seemed to just use the username and pssword (based on output from debug logging) | 20:57 |
| clarkb | could be a combo of the two is what we need | 20:57 |
| clarkb | corvus: mordred nodepool==3.1.1.dev45 # git sha 67824d8 look correct to you? | 20:57 |
| clarkb | seems to match the sha1 I expect | 20:57 |
| mordred | clarkb: yes | 20:58 |
| *** kgiusti has left #openstack-infra | 20:58 | |
| *** rtjure has joined #openstack-infra | 20:58 | |
| clarkb | nl04 restarted to start | 20:58 |
| clarkb | http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1 should start to see data again if statsd fix works | 20:59 |
| fungi | ssbarnea: thanks for the attention to git-review in the meeting. triage of open changes with some summary and recommendations would be great. i'll prioritize bug fixes and support for new gerrit releases for sure, along with testing improvements. a lot of the remainder will likely need some culling | 20:59 |
| clarkb | yup I see graph data again | 20:59 |
| ssbarnea | fungi: great! :) | 21:00 |
| clarkb | I'll give nl04 a few minutes just to see if we notice anything weird then restart everything else | 21:01 |
| mordred | yay! | 21:01 |
| fungi | ssbarnea: as clarkb mentioned i'm on vacation (clearly not doing a good job of that) but should be around after the middle of next week | 21:01 |
| *** trown is now known as trown|outtypewww | 21:02 | |
| fungi | clarkb: oh, if rackdns uses the username and password then we should be able to get it working again | 21:02 |
| clarkb | fungi: ya it seemed to be based on the output from --debug | 21:02 |
| fungi | awesome | 21:02 |
| clarkb | and according to that was failing on ssl verification | 21:02 |
| * fungi fades back into vacation | 21:02 | |
| clarkb | so I've got a new rackdns venv in my homedir to test with next time a dns record needs an update (new venv has newer python requests which vendors trust chains) | 21:03 |
| *** gfidente has joined #openstack-infra | 21:05 | |
| *** gfidente has joined #openstack-infra | 21:05 | |
| clarkb | ok still seems happy on nl04, going to restart the other daemons | 21:08 |
| *** graphene has quit IRC | 21:17 | |
| *** graphene has joined #openstack-infra | 21:18 | |
| clarkb | everything but nb03 is restarted as nb03 is running and image build currently | 21:18 |
| clarkb | john_studarus: just a heads up that grafana should have nodepool graph data again | 21:22 |
| john_studarus | ok! | 21:23 |
| *** rcernin_ has quit IRC | 21:30 | |
| *** slaweq has quit IRC | 21:30 | |
| pabelanger | clarkb: john_studarus: yan, however looks like we might of had an outage all day for packethost? a lot of nodes building all day | 21:31 |
| clarkb | looks like it started around 1330UTC | 21:32 |
| john_studarus | was it deleting them? or just never got out of building state? | 21:33 |
| clarkb | john_studarus: it looks like it is building them until our timeout then deleting them | 21:33 |
| clarkb | openstack.exceptions.ResourceTimeout: Timeout waiting for the server to come up. | 21:34 |
| *** edmondsw has quit IRC | 21:34 | |
| *** agopi|brb is now known as agopi | 21:35 | |
| pabelanger | FIP issue again? | 21:35 |
| clarkb | fip issue was happening post boot | 21:35 |
| clarkb | (which is why we noticed in jobs) | 21:36 |
| john_studarus | I'm seeing a combination of "Failed to allocate network(s)" and "Failure prepping block device" | 21:38 |
| clarkb | niehter of those are familiar to me | 21:39 |
| *** agopi is now known as agopi|brv | 21:48 | |
| john_studarus | clarkb: "...Unable to convert image to raw..." | 21:49 |
| clarkb | john_studarus: ah, ya so it was as pabelanger suspected, we are converting from qcow2 to raw | 21:49 |
| clarkb | john_studarus: is it failing due to lack of disk space? or does qemu-img not like our qcow2? | 21:50 |
| john_studarus | but it might be due to disk space issue | 21:50 |
| john_studarus | let me grow that volume | 21:50 |
| john_studarus | it's at 93% | 21:50 |
| pabelanger | Yah, strikes again | 21:50 |
| john_studarus | Stderr: u'qemu-img: error while writing sector 8712960: No space left on device\n' | 21:51 |
| clarkb | john_studarus: I think nova keeps the old converted images around as a cache, but we rotate images daily | 21:51 |
| clarkb | john_studarus: what we did in infracloud was configured nova to just boot qcow2 and not do a conversion and it worked fine for that cloud | 21:51 |
| *** priteau has quit IRC | 21:51 | |
| pabelanger | Yup! | 21:52 |
| clarkb | pabelanger: have those details handy? | 21:52 |
| pabelanger | yah, 1 sec. inthe puppet change I linked this morning | 21:52 |
| pabelanger | https://git.openstack.org/cgit/openstack-infra/puppet-infracloud/commit/?id=9dc122c3ddd852f4572ffff0c6d5ed6f84fa2a96 | 21:52 |
| *** agopi|brv has quit IRC | 21:53 | |
| pabelanger | libvirt images_type=qcow2 | 21:53 |
| pabelanger | and force_raw_images false | 21:53 |
| clarkb | john_studarus: another option here may be to have nova not cache those imges so aggressively? | 21:53 |
| john_studarus | I have 3 * 450GB of SSD space free I can allocate on that system | 21:54 |
| clarkb | ah if you have more disk space that may be a good quick solution | 21:54 |
| john_studarus | k | 21:54 |
| pabelanger | +1 | 21:55 |
| *** boden has quit IRC | 21:55 | |
| clarkb | I think nova converts to raw beacuse raw image format performs better, but we found for our use case at least it didn't make much difference and vastly simplified the hypervisor management to not convert | 21:55 |
| pabelanger | yah, also lets vms boot faster once we upload images, as there is no covert to raw | 21:55 |
| pabelanger | if raw needed, we could also just upload a raw | 21:56 |
| clarkb | ya that is another option, we could just upload raw and avoid a conversion entirely | 21:56 |
| pabelanger | maybe we should just default nodepool-builder to raw for openstack :) since raw is usually default | 21:58 |
| pabelanger | and only do qcow2 when we know a cloud has disable the setting | 21:59 |
| *** yamamoto has joined #openstack-infra | 21:59 | |
| clarkb | possibly, though it is a lot less efficient at rest. I wish openstack did the simple thing by default and allowed people to tune for raw if they think it will make a difference | 21:59 |
| pabelanger | ++ | 22:00 |
| *** kjackal has quit IRC | 22:01 | |
| logan- | i think its good as is. the raw conversion hit is just a one time hit, if you disable it then it does the conversion on the fly for each nodepool instance you boot when you do disk i/o. i prefer to leave it on for that reason.. ill take a few minutes longer image rollout over slower test nodes :) | 22:02 |
| clarkb | logan-: if you boot qcow2 then no conversion is necessary though | 22:03 |
| clarkb | oh you mean block level read and writes for the VM? | 22:03 |
| logan- | right | 22:03 |
| logan- | and all of that work at runtime is not cached, so when youre booting hundreds of instances over the lifetime of that image i think its prudent to just do the one-time conversion | 22:04 |
| *** yamamoto has quit IRC | 22:04 | |
| clarkb | logan-: ya there is a trade off (typically the problem has been disk space and boots being incredibly slow due to the conversion happening per hypervisor) | 22:05 |
| logan- | yeah :( | 22:05 |
| pabelanger | IMO, if we are already building a raw / qcow2 image in nodepool-builder, and upload qcow2 to only convert to raw, we might as well change to upload raw | 22:05 |
| pabelanger | and avoid the convert | 22:05 |
| clarkb | from a user pespecitve I sort of see it as if I upload raw then I know I am ok with the disk size tradeoff, if I upload qcow2 I don't mind simplicity for some disk read/write overhead | 22:05 |
| clarkb | pabelanger: network bandwidth is another consideration, it may be faster (and possibly more relaible) to upload qcow2 to the cloud and have it convert than it is to upload raw | 22:06 |
| logan- | whats the size diff for raw? id imagine that will have a huge impact on upload times and bandwidth consumption | 22:06 |
| clarkb | pabelanger: I think our qcow2 images are about 1/4 the size of raw | 22:06 |
| *** yamamoto has joined #openstack-infra | 22:06 | |
| logan- | ah | 22:06 |
| pabelanger | clarkb: maybe, I think nodepoolv3 uploads are been more reliable in general | 22:06 |
| *** tosky has quit IRC | 22:07 | |
| john_studarus | I'm doubling the image storage | 22:07 |
| john_studarus | I can triple it if needed | 22:07 |
| clarkb | logan-: pabelanger looks like 8GB ish vs 15GB ish, however that 15GB ish may be fstrimmed | 22:07 |
| pabelanger | plus, once we have 1 image online and nodes using it, there really isn't much pressure to rotate to next image... unless something is broken | 22:07 |
| pabelanger | clarkb: yah, that sounds right | 22:08 |
| clarkb | nope du reports 8 ish GB fstrimmed and 15GB is actual size | 22:08 |
| clarkb | pabelanger: uploads are more reliable over time in that we are better at retrying them if and when they fail. I don't think we changed much about reliability of a single upload | 22:09 |
| pabelanger | Completely agree | 22:10 |
| clarkb | logan-: and when you mix in tools like ceph the choice of what is better becomes easier | 22:10 |
| clarkb | mostly I'm just happy to not use vhd | 22:10 |
| clarkb | since vhd tooling is pretty lacking | 22:10 |
| pabelanger | there we go, 5 nodes online in packethost | 22:11 |
| *** slaweq has joined #openstack-infra | 22:11 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for tempest scenario run timeout bug 1783405 https://review.openstack.org/585534 | 22:11 |
| openstack | bug 1783405 in tempest "Slow tests randomly timing out jobs (which aren't marked slow)" [High,Confirmed] https://launchpad.net/bugs/1783405 | 22:11 |
| john_studarus | clarkb: increase done... up to 2.6TB from 450GB | 22:15 |
| clarkb | john_studarus: thanks@ | 22:15 |
| *** slaweq has quit IRC | 22:16 | |
| pabelanger | clarkb: john_studarus: we likely can also tune nodepool boot-timeout to be something less then 3600 seconds. Maybe 600? | 22:17 |
| clarkb | pabelanger: it is set to 300 (5 minutes now right?) if so 600 seems reasonable | 22:17 |
| pabelanger | oh, hmm. It is | 22:17 |
| clarkb | part of my concern is that the way jobs get assigned nodes now mean the longer that timeout is the longer it takes to get a node from a cloud | 22:17 |
| pabelanger | clarkb: is launch-retries coming into play here? | 22:18 |
| clarkb | pabelanger: yes I think its 3x the timeout if each one times out | 22:18 |
| clarkb | there is a balance in here somewhere :) | 22:18 |
| clarkb | I don't know what it is | 22:18 |
| pabelanger | yah, and launch-retries is the same provider too | 22:19 |
| pabelanger | IIRC | 22:19 |
| clarkb | yup | 22:19 |
| pabelanger | okay, I see 15 nodes online now | 22:19 |
| clarkb | in use too | 22:21 |
| clarkb | pabelanger: part of the problem in determining a good timeout is we don't know how long the image conversion will take in each cloud | 22:22 |
| clarkb | pabelanger: we could probably ballpark it checking how long it takes locally | 22:22 |
| pabelanger | yah, good argument not to convert :) | 22:23 |
| clarkb | ya that does remove a lot of that uncertainty | 22:23 |
| pabelanger | for most parts, time to ready metric in nodepool is a good indicator, we could maybe buffer X% extra | 22:24 |
| pabelanger | (assuming no convert) | 22:24 |
| clarkb | pabelanger: ya | 22:24 |
| tonyb | So yesterday my chnage to switch propose-updates from xenial -> bionic got merged but the logs indictae that it got run on a xenial node. which resulted in py3.6 not being there. | 22:29 |
| tonyb | so my question is how do I determine if it was just a race in zuul with the new script runnin on the old job defn. *or* if my chnage was wrong and wont do what I want? | 22:30 |
| tonyb | http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/propose-updates/3191fe4/job-output.txt.gz#_2018-07-24_06_22_30_279216 | 22:30 |
| *** rcernin has joined #openstack-infra | 22:30 | |
| *** sthussey has quit IRC | 22:30 | |
| *** hongbin has quit IRC | 22:30 | |
| clarkb | tonyb: the zuul-info/inventory.yaml file is where I would start. That includes the config inheritance path | 22:31 |
| clarkb | tonyb: http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/propose-updates/3191fe4/zuul-info/inventory.yaml and grep inheritance_path | 22:31 |
| clarkb | hrm it just says master though so we need to look at timestamps next | 22:32 |
| mordred | http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/propose-updates/3191fe4/job-output.txt.gz#_2018-07-24_06_19_24_373343 | 22:32 |
| tonyb | clarkb: Okay I was goign to ask how I translate: openstack-infra/project-config/zuul.d/jobs.yaml@master#935 into a SHA | 22:32 |
| mordred | that shows the sha of the requirement repo | 22:33 |
| tonyb | mordred: that's requirements I *think* I'm looking for the SHAs of project-config and *zuul* aren't I? | 22:33 |
| mordred | oh - you are aren't you | 22:33 |
| *** neiloy has quit IRC | 22:33 | |
| tonyb | mordred: .... this job is defined in project-config IIRC | 22:33 |
| clarkb | tonyb: where is the change to switch to bionic? | 22:34 |
| mordred | yeah | 22:34 |
| tonyb | clarkb: let me find it | 22:35 |
| clarkb | I'm not seeing bionic on that job in update to date project-config (but I'm probably just missing it) | 22:35 |
| tonyb | https://review.openstack.org/#/c/583065/ | 22:35 |
| tonyb | Hmmm propose-updates vs propose-updates-constraints | 22:36 |
| clarkb | ya its a different job | 22:36 |
| tonyb | Oh phooey | 22:36 |
| * tonyb goes to work out why/how they're different and if one isn't needed | 22:36 | |
| *** rlandy has quit IRC | 22:37 | |
| *** jamesmcarthur has joined #openstack-infra | 22:39 | |
| tonyb | Oh rats. I'm messed that thing up badly. | 22:41 |
| *** Nil_ has quit IRC | 22:42 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Revert "Run propose-update-constraints under Python3.6" https://review.openstack.org/585566 | 22:43 |
| tonyb | clarkb, mordred: ^^^ I'll push a correct version real-soon now | 22:43 |
| *** jamesmcarthur has quit IRC | 22:44 | |
| clarkb | tonyb: I've gone ahead and single core approved that one | 22:44 |
| tonyb | clarkb: Thanks | 22:45 |
| *** graphene has quit IRC | 22:45 | |
| clarkb | I'm going ahead and restarting nodepool builder on nb03 because the previous build failed | 22:46 |
| *** graphene has joined #openstack-infra | 22:46 | |
| *** slaweq has joined #openstack-infra | 22:50 | |
| *** felipemonteiro has joined #openstack-infra | 22:52 | |
| *** jamesmcarthur has joined #openstack-infra | 22:54 | |
| *** slaweq has quit IRC | 22:54 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Run propose-updates for requirements-constraints to Python3.6 https://review.openstack.org/585569 | 23:03 |
| *** Nil_ has joined #openstack-infra | 23:03 | |
| *** mschuppert has quit IRC | 23:03 | |
| * tonyb has no idea if I can really do that ^^ but it's worth a try | 23:04 | |
| *** rfolco has quit IRC | 23:04 | |
| *** rfolco has joined #openstack-infra | 23:05 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 23:05 |
| openstackgerrit | Merged openstack/diskimage-builder master: Call kpartx remove in umount, not cleanup https://review.openstack.org/578657 | 23:05 |
| *** graphene has quit IRC | 23:07 | |
| *** rh-jelabarre has quit IRC | 23:07 | |
| *** graphene has joined #openstack-infra | 23:08 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** rh-jelabarre has joined #openstack-infra | 23:13 | |
| *** graphene has quit IRC | 23:13 | |
| *** tpsilva has quit IRC | 23:15 | |
| *** graphene has joined #openstack-infra | 23:15 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Run propose-update-constraints under Python3.6" https://review.openstack.org/585566 | 23:15 |
| *** slaweq has quit IRC | 23:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Fix zuul reporting build failure with only non-voting jobs https://review.openstack.org/584990 | 23:16 |
| *** gongysh has joined #openstack-infra | 23:18 | |
| *** jamesden_ has quit IRC | 23:24 | |
| *** jamesmcarthur has quit IRC | 23:24 | |
| *** agopi has joined #openstack-infra | 23:25 | |
| *** jamesde__ has joined #openstack-infra | 23:25 | |
| *** jamesde__ has quit IRC | 23:26 | |
| *** wolverineav has quit IRC | 23:26 | |
| *** jamesden_ has joined #openstack-infra | 23:27 | |
| *** jamesden_ has quit IRC | 23:27 | |
| *** wolverineav has joined #openstack-infra | 23:27 | |
| *** jamesden_ has joined #openstack-infra | 23:28 | |
| *** jamesden_ has quit IRC | 23:29 | |
| *** jamesden_ has joined #openstack-infra | 23:30 | |
| *** jamesden_ has quit IRC | 23:30 | |
| *** jamesden_ has joined #openstack-infra | 23:31 | |
| *** jamesden_ has quit IRC | 23:32 | |
| *** jamesden_ has joined #openstack-infra | 23:32 | |
| *** jamesden_ has quit IRC | 23:33 | |
| *** wolverineav has quit IRC | 23:33 | |
| *** jamesden_ has joined #openstack-infra | 23:34 | |
| *** jamesden_ has quit IRC | 23:35 | |
| *** jamesden_ has joined #openstack-infra | 23:35 | |
| *** jamesden_ has quit IRC | 23:36 | |
| *** jamesden_ has joined #openstack-infra | 23:37 | |
| *** eernst has quit IRC | 23:37 | |
| *** jamesden_ has quit IRC | 23:37 | |
| *** gfidente has quit IRC | 23:37 | |
| *** r-daneel has quit IRC | 23:38 | |
| *** jamesden_ has joined #openstack-infra | 23:38 | |
| *** jamesden_ has quit IRC | 23:39 | |
| *** jamesde__ has joined #openstack-infra | 23:39 | |
| *** jamesde__ has quit IRC | 23:40 | |
| *** jamesden_ has joined #openstack-infra | 23:41 | |
| *** eernst_ has joined #openstack-infra | 23:41 | |
| *** jamesden_ has quit IRC | 23:41 | |
| *** jamesden_ has joined #openstack-infra | 23:42 | |
| *** eernst_ has quit IRC | 23:43 | |
| *** jamesden_ has quit IRC | 23:43 | |
| *** eernst has joined #openstack-infra | 23:43 | |
| *** jamesden_ has joined #openstack-infra | 23:44 | |
| *** jamesden_ has quit IRC | 23:44 | |
| *** eernst has quit IRC | 23:45 | |
| *** jamesden_ has joined #openstack-infra | 23:45 | |
| *** jamesden_ has quit IRC | 23:46 | |
| *** eernst has joined #openstack-infra | 23:46 | |
| *** jamesden_ has joined #openstack-infra | 23:47 | |
| *** jamesden_ has quit IRC | 23:48 | |
| *** jamesden_ has joined #openstack-infra | 23:49 | |
| *** felipemonteiro has quit IRC | 23:49 | |
| *** jamesden_ has quit IRC | 23:49 | |
| *** jamesden_ has joined #openstack-infra | 23:50 | |
| *** jamesden_ has quit IRC | 23:50 | |
| *** rpioso is now known as rpioso|afk | 23:51 | |
| *** eernst has quit IRC | 23:51 | |
| *** jamesden_ has joined #openstack-infra | 23:52 | |
| *** jamesden_ has quit IRC | 23:53 | |
| *** yolanda_ has joined #openstack-infra | 23:56 | |
| *** linkmark has quit IRC | 23:57 | |
| *** yolanda has quit IRC | 23:58 | |
| *** wolverineav has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!