| jeblair | mordred: is there a "reason" for that? it might help me remember which to use :) | 00:00 |
|---|---|---|
| *** dingyichen has joined #openstack-infra | 00:00 | |
| *** thingee_ has quit IRC | 00:00 | |
| mordred | jeblair: shell spawns a full subshell, command just forks the given process | 00:01 |
| pabelanger | jeblair: mordred: we may what to think about how to properly bootstrap a node with required ansible dependency for the common tasks too. I can see that being a problem for other zuulv3 users too | 00:01 |
| jeblair | mordred: aha, that does help, thanks. | 00:01 |
| mordred | pabelanger: yah - I agree - I'm not crazy about the options I've come up with for that so far - but I do think it's worth pondering until we come up with something we do like | 00:02 |
| clarkb | pabelanger: pre run task? | 00:02 |
| clarkb | that has benefit of reruning job should installing random libs fail | 00:02 |
| pabelanger | clarkb: ya | 00:02 |
| pabelanger | configure-mirror is trusted playbook :( | 00:02 |
| mordred | clarkb: installing them in a playbook isn't too bad - it's that random users who may want to use, say, the apt module, need to know to install python-apt to do so | 00:02 |
| clarkb | mordred: right but if you do that early enough in all jobs then few should notice right? | 00:03 |
| clarkb | mordred: bsaically put it in bsae job that you just always install a few deps? | 00:03 |
| jeblair | pabelanger: can you expand "configure-mirror is trusted playbook :(" ? | 00:03 |
| mordred | sure - but then we're installing software in a base job, and that migh taint things for things like devstack | 00:03 |
| pabelanger | jeblair: 492716 will need to be force merged, because configure-mirror role comes from base job (project-config) | 00:04 |
| clarkb | ya, other method would be to trap common tasks where this is a problem then just shell/command their equivalents | 00:04 |
| clarkb | but thats gross from a compatibility and documentation perspective | 00:04 |
| mordred | clarkb: yup | 00:04 |
| clarkb | "why doesn't this work like in the docs?" | 00:04 |
| mordred | clarkb: exactly - which is why I like none of the solutions so far - but luckily if someone writes a job that fails on python-apt not being installed for their apt task, it's super easy to add it to that job | 00:05 |
| jeblair | pabelanger: only if it runs on internap. we may be able to recheck-bash it in | 00:05 |
| mordred | so if we don't invent any magic in the short term, it's not like people will be blocked | 00:05 |
| jeblair | pabelanger: why don't you +3 it and we'll see? :) | 00:06 |
| pabelanger | jeblair: Ha, yup. Just passed on infracloud | 00:06 |
| clarkb | mordred: or just get ansible to work with built in tasks out of the box :) | 00:06 |
| mordred | clarkb: :) | 00:06 |
| jeblair | the *actual* problem is that apt batteries are not included in the python stdlib :) | 00:07 |
| pabelanger | mordred: clarkb: the fun really starts when python is missing on remote node, then you get to write playbooks with raw connection :) | 00:07 |
| clarkb | (thats actually one nice thing about how puppet installs itself, it does a good job of getting all the things into its installation) | 00:07 |
| clarkb | jeblair: ya so in eg puppet it copies the executables and libs it needs to make it happen | 00:07 |
| clarkb | because same problem with ruby | 00:07 |
| pabelanger | I wonder why apt index are so stale on internap | 00:08 |
| fungi | looks like python-apt doesn't have any other python deps (well, it depends on python-apt-common which in turn has none) so that seems a safe enough addition | 00:09 |
| mordred | the streaming console logs never get old | 00:09 |
| jeblair | i have some questions: | 00:09 |
| jeblair | 1) why is jenkins commenting on 492716 | 00:09 |
| jeblair | 2) why is 492716,2 in zuulv3 check | 00:09 |
| jeblair | 3) why is 492716,2 not in zuulv3 gate | 00:09 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Use shell for apt-get update https://review.openstack.org/492716 | 00:09 |
| *** thorst has quit IRC | 00:09 | |
| mordred | jeblair: 1) maybe updated project-config hasn't gone through puppet cycle yet? | 00:10 |
| *** thorst has joined #openstack-infra | 00:10 | |
| pabelanger | 2) tox-py35-on-zuul still running too | 00:10 |
| jeblair | mordred: 1) plausible! | 00:10 |
| mordred | jeblair: and 2.5 just merged 492716 | 00:10 |
| pabelanger | eep | 00:11 |
| *** thorst has quit IRC | 00:11 | |
| mordred | jeblair: but 2) is a great question | 00:11 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for cinder group update bug 1709739 https://review.openstack.org/492720 | 00:11 |
| openstack | bug 1709739 in Cinder ""Invalid volume: Volume is not local to this node" in neutron multinode job (pike)" [Undecided,Confirmed] https://launchpad.net/bugs/1709739 | 00:11 |
| jeblair | oh i think i have an answer: | 00:12 |
| pabelanger | confused on 2 now, was it not expected to be in zuulv3 check? | 00:12 |
| jeblair | it was not in v3 gate because v3 check had not reported yet | 00:12 |
| jeblair | (only v2 check had reported) | 00:12 |
| TheJulia | fungi: oh awesome! | 00:13 |
| jeblair | so possibly the answer to everything is the v2 puppet cycle | 00:13 |
| pabelanger | cool | 00:13 |
| fungi | "eventually consistent!" | 00:14 |
| mordred | jeblair: ah - so it likely would have gone into the v3 gate, since v3 updates its own config and doens't wait on puppet - but v2 beat it to the punch | 00:14 |
| jeblair | mordred: yeah; if v2 had taken a bit longer to merge it may have. | 00:15 |
| * TheJulia thought the new thing was consistent inconsistency | 00:15 | |
| jeblair | TheJulia: then we are all set! :) | 00:15 |
| *** priteau has joined #openstack-infra | 00:16 | |
| *** k2so is now known as sdake | 00:16 | |
| mordred | TheJulia: we'll eventually be consistent with the world's new view on preferring consistent inconsistency over inconsistent consistency | 00:16 |
| *** Sukhdev has quit IRC | 00:16 | |
| *** sflanigan has joined #openstack-infra | 00:17 | |
| *** sflanigan has joined #openstack-infra | 00:17 | |
| TheJulia | I feel like I need to learn "belter" now... | 00:17 |
| *** funzo has joined #openstack-infra | 00:17 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for cinder group update bug 1709739 https://review.openstack.org/492720 | 00:18 |
| openstack | bug 1709739 in Cinder ""Invalid volume: Volume is not local to this node" in neutron multinode job (pike)" [Undecided,Confirmed] https://launchpad.net/bugs/1709739 | 00:18 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Stop trying to build networking-bagpipe with DLRN https://review.openstack.org/492339 | 00:20 |
| mordred | pabelanger: happy clicky? https://review.openstack.org/#/c/492713/ | 00:20 |
| *** priteau has quit IRC | 00:20 | |
| * TheJulia goes back to her rock | 00:21 | |
| pabelanger | mordred: done | 00:21 |
| *** sbezverk has quit IRC | 00:21 | |
| mordred | \o/ | 00:21 |
| pabelanger | mordred: jeblair: do we need to restart zuulv3 or is 492671 safe to land: ssh key secret | 00:21 |
| *** funzo has quit IRC | 00:22 | |
| jeblair | pabelanger: should be safe, we'll just need to restart to get the new behavior | 00:23 |
| pabelanger | great | 00:23 |
| *** slaweq has joined #openstack-infra | 00:25 | |
| mriedem | is there a hard timeout on a job that appears hung? i tried to telnet but not getting anything back | 00:27 |
| mriedem | oh hold the phone | 00:27 |
| *** xarses has quit IRC | 00:27 | |
| pabelanger | which node? | 00:28 |
| *** slaweq has quit IRC | 00:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-openstack-artifacts base job https://review.openstack.org/492713 | 00:35 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Bindmount /etc/lsb-release into bubblewrap https://review.openstack.org/490200 | 00:36 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Hack in linux bridge vxlan support https://review.openstack.org/492654 | 00:48 |
| *** LindaWang has joined #openstack-infra | 00:50 | |
| *** zhurong has joined #openstack-infra | 00:51 | |
| *** skelso has joined #openstack-infra | 00:52 | |
| *** sekelso has joined #openstack-infra | 00:55 | |
| mriedem | don't know how i'd tell which node | 00:56 |
| mriedem | seems to be super slow | 00:56 |
| jeblair | mriedem: the ip address will identify it | 00:56 |
| mriedem | ah yeah jenkins just killed it | 00:57 |
| mriedem | sorry, zuul | 00:57 |
| mriedem | telnet://37.153.172.122:19885 | 00:57 |
| jeblair | | 10382972 | citycloud-lon1 | None | ubuntu-xenial | zuul | zl03 | ubuntu-xenial-citycloud-lon1-10382972 | ubuntu-xenial-citycloud-lon1-10382972 | 1c9d5887-c7e2-4ff3-948e-63cac88671bc | 37.153.172.122 | used | 00:02:51:06 | None | | 00:57 |
| *** skelso has quit IRC | 00:58 | |
| *** esberglu has joined #openstack-infra | 00:58 | |
| jeblair | i heard tell of some serious io problems from lon1 on sporadic nodes earlier | 00:58 |
| mriedem | yes, citycloud-lon1 has been a problem for a few days | 00:58 |
| mriedem | http://status.openstack.org/elastic-recheck/#1709506 is related | 00:59 |
| fungi | we suspect i/o starvation for one or more hypervisor hosts in that region | 01:01 |
| fungi | it's been reported to themm | 01:01 |
| *** esberglu has quit IRC | 01:02 | |
| *** markvoelker has joined #openstack-infra | 01:03 | |
| *** xarses has joined #openstack-infra | 01:06 | |
| pabelanger | ya, another bad node in citycloud-lon1 it looks like: http://logs.openstack.org/71/491171/5/gate/gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial/2baec8f/logs/devstack-gate-setup-host.txt | 01:07 |
| *** Apoorva_ has quit IRC | 01:09 | |
| *** thorst has joined #openstack-infra | 01:11 | |
| *** cuongnv has joined #openstack-infra | 01:13 | |
| *** thorst has quit IRC | 01:17 | |
| *** funzo has joined #openstack-infra | 01:18 | |
| *** funzo has quit IRC | 01:23 | |
| *** caphrim007 has joined #openstack-infra | 01:23 | |
| *** slaweq has joined #openstack-infra | 01:27 | |
| *** skelso has joined #openstack-infra | 01:29 | |
| openstackgerrit | Rui Chen proposed openstack-infra/shade master: Support to get resource by id https://review.openstack.org/492080 | 01:29 |
| *** jpena|off is now known as jpena | 01:30 | |
| *** yamahata has quit IRC | 01:31 | |
| *** slaweq has quit IRC | 01:31 | |
| *** sekelso has quit IRC | 01:31 | |
| *** liujiong has joined #openstack-infra | 01:32 | |
| *** PsionTheory has quit IRC | 01:35 | |
| *** lathiat_ has quit IRC | 01:35 | |
| *** lathiat has joined #openstack-infra | 01:36 | |
| *** sekelso has joined #openstack-infra | 01:36 | |
| *** hongbin has joined #openstack-infra | 01:38 | |
| *** jpena is now known as jpena|off | 01:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add https://api.rubygems.org/ to reverse proxy cache https://review.openstack.org/492760 | 01:39 |
| *** skelso has quit IRC | 01:39 | |
| pabelanger | ianw: clarkb: ^ adds https://api.rubygems.org/ to our reverse proxy cache. Looks like all we might need | 01:39 |
| ianw | ok, i know little about the backend proxying requirements of ruby gems though! :) | 01:41 |
| pabelanger | ianw: same, just looked at wireshark when gem install command run. I didn't see any 302 returned | 01:42 |
| *** xarses_ has joined #openstack-infra | 01:48 | |
| *** ramishra has joined #openstack-infra | 01:53 | |
| *** jamesmcarthur has joined #openstack-infra | 01:53 | |
| *** jamesmcarthur has quit IRC | 01:58 | |
| *** sree has joined #openstack-infra | 02:10 | |
| *** ianychoi_ has joined #openstack-infra | 02:11 | |
| *** hongbin_ has joined #openstack-infra | 02:11 | |
| *** thorst has joined #openstack-infra | 02:12 | |
| *** cinerama` has joined #openstack-infra | 02:13 | |
| *** sree has quit IRC | 02:14 | |
| *** lathiat_ has joined #openstack-infra | 02:16 | |
| *** priteau has joined #openstack-infra | 02:17 | |
| *** gongysh has joined #openstack-infra | 02:17 | |
| *** thorst has quit IRC | 02:17 | |
| *** cinerama has quit IRC | 02:18 | |
| *** hongbin has quit IRC | 02:18 | |
| *** ianychoi has quit IRC | 02:18 | |
| *** jistr has quit IRC | 02:18 | |
| *** lathiat has quit IRC | 02:18 | |
| *** ramishra has quit IRC | 02:18 | |
| *** quite has quit IRC | 02:18 | |
| *** markmcd has quit IRC | 02:18 | |
| *** spligak has quit IRC | 02:18 | |
| *** clayton has quit IRC | 02:18 | |
| *** rkukura has quit IRC | 02:18 | |
| *** mudpuppy has quit IRC | 02:18 | |
| *** lindsayk has quit IRC | 02:18 | |
| *** kzaitsev1pi has quit IRC | 02:18 | |
| *** kzaitsev_pi has joined #openstack-infra | 02:19 | |
| *** funzo has joined #openstack-infra | 02:19 | |
| *** mudpuppy has joined #openstack-infra | 02:19 | |
| *** clayton has joined #openstack-infra | 02:19 | |
| *** markmcd has joined #openstack-infra | 02:20 | |
| *** openstackgerrit has quit IRC | 02:21 | |
| *** armax has quit IRC | 02:21 | |
| *** lifeless_ has joined #openstack-infra | 02:21 | |
| *** rkukura_ has joined #openstack-infra | 02:21 | |
| *** purp has quit IRC | 02:21 | |
| *** armax has joined #openstack-infra | 02:21 | |
| *** priteau has quit IRC | 02:22 | |
| *** jistr has joined #openstack-infra | 02:23 | |
| *** xarses_ has quit IRC | 02:23 | |
| *** purp has joined #openstack-infra | 02:24 | |
| *** funzo has quit IRC | 02:24 | |
| *** kzaitsev1pi has joined #openstack-infra | 02:25 | |
| *** hrubi_ has joined #openstack-infra | 02:27 | |
| *** ramishra has joined #openstack-infra | 02:27 | |
| *** rkukura has joined #openstack-infra | 02:28 | |
| *** lifeless has quit IRC | 02:28 | |
| *** rkukura has quit IRC | 02:28 | |
| *** hrubi has quit IRC | 02:28 | |
| *** kzaitsev_pi has quit IRC | 02:28 | |
| *** slaweq has joined #openstack-infra | 02:28 | |
| *** yamahata has joined #openstack-infra | 02:28 | |
| *** rkukura_ is now known as rkukura | 02:28 | |
| *** quite has joined #openstack-infra | 02:30 | |
| *** openstackgerrit has joined #openstack-infra | 02:30 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: External Calendar Sync https://review.openstack.org/487683 | 02:30 |
| *** ekhugen has quit IRC | 02:30 | |
| *** lindsayk has joined #openstack-infra | 02:31 | |
| *** slaweq has quit IRC | 02:32 | |
| *** ekhugen has joined #openstack-infra | 02:33 | |
| *** ykarel_ has joined #openstack-infra | 02:42 | |
| *** hongbin_ has quit IRC | 02:43 | |
| *** jtomasek has joined #openstack-infra | 02:44 | |
| *** jtomasek has quit IRC | 02:48 | |
| *** ykarel__ has joined #openstack-infra | 02:51 | |
| *** StaceyF has joined #openstack-infra | 02:52 | |
| *** yamamoto has quit IRC | 02:52 | |
| *** yamamoto has joined #openstack-infra | 02:53 | |
| openstackgerrit | Rui Chen proposed openstack-infra/shade master: Support to get resource by id https://review.openstack.org/492080 | 02:54 |
| *** ykarel_ has quit IRC | 02:55 | |
| *** hongbin has joined #openstack-infra | 02:55 | |
| *** EricGonczer_ has joined #openstack-infra | 02:56 | |
| *** spzala has joined #openstack-infra | 02:59 | |
| *** caphrim0_ has joined #openstack-infra | 03:01 | |
| *** caphrim007 has quit IRC | 03:05 | |
| *** dave-mccowan has quit IRC | 03:06 | |
| *** EricGonczer_ has quit IRC | 03:08 | |
| *** thorst has joined #openstack-infra | 03:13 | |
| *** ykarel__ has quit IRC | 03:17 | |
| *** sekelso has quit IRC | 03:17 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Switch to use local buildlogs mirror https://review.openstack.org/492786 | 03:18 |
| *** ykarel__ has joined #openstack-infra | 03:20 | |
| *** nicolasbock has joined #openstack-infra | 03:21 | |
| *** funzo has joined #openstack-infra | 03:21 | |
| *** thorst has quit IRC | 03:21 | |
| *** mriedem has quit IRC | 03:22 | |
| *** sbezverk has joined #openstack-infra | 03:24 | |
| *** funzo has quit IRC | 03:26 | |
| *** mriedem has joined #openstack-infra | 03:27 | |
| *** slaweq has joined #openstack-infra | 03:29 | |
| *** namnh has joined #openstack-infra | 03:33 | |
| *** slaweq has quit IRC | 03:34 | |
| *** gongysh has quit IRC | 03:34 | |
| *** ykarel__ has quit IRC | 03:35 | |
| *** StaceyF has quit IRC | 03:38 | |
| *** psachin has joined #openstack-infra | 03:41 | |
| *** udesale has joined #openstack-infra | 03:41 | |
| *** psachin has quit IRC | 03:43 | |
| *** gmann has joined #openstack-infra | 03:44 | |
| *** psachin has joined #openstack-infra | 03:51 | |
| johnsom | Thank you infra folks for making magic happen and getting the gates going again today! Our RC is back on track! | 03:53 |
| *** hongbin has quit IRC | 03:56 | |
| *** SamYaple has quit IRC | 03:57 | |
| *** ykarel__ has joined #openstack-infra | 04:03 | |
| *** sbezverk has quit IRC | 04:07 | |
| *** SamYaple has joined #openstack-infra | 04:11 | |
| *** spligak has joined #openstack-infra | 04:14 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Move neutron multinode job to check pipeline https://review.openstack.org/492797 | 04:16 |
| *** thorst has joined #openstack-infra | 04:17 | |
| *** priteau has joined #openstack-infra | 04:18 | |
| *** hamzy has joined #openstack-infra | 04:18 | |
| *** hareesh has joined #openstack-infra | 04:21 | |
| *** thorst has quit IRC | 04:22 | |
| *** funzo has joined #openstack-infra | 04:22 | |
| *** priteau has quit IRC | 04:22 | |
| *** funzo has quit IRC | 04:26 | |
| *** adisky__ has joined #openstack-infra | 04:31 | |
| *** jogo has quit IRC | 04:31 | |
| *** liujiong_lj has joined #openstack-infra | 04:33 | |
| *** liujiong has quit IRC | 04:35 | |
| *** ykarel__ has quit IRC | 04:38 | |
| *** dingyichen has quit IRC | 04:40 | |
| *** ykarel__ has joined #openstack-infra | 04:43 | |
| *** vhosakot has joined #openstack-infra | 04:47 | |
| *** tnovacik has joined #openstack-infra | 04:47 | |
| *** vhosakot has quit IRC | 04:52 | |
| *** vhosakot has joined #openstack-infra | 04:52 | |
| *** rkukura has quit IRC | 04:52 | |
| *** ihrachys has joined #openstack-infra | 04:54 | |
| *** jogo has joined #openstack-infra | 04:56 | |
| *** jogo has joined #openstack-infra | 04:56 | |
| *** rkukura has joined #openstack-infra | 04:57 | |
| ianw | #status log restarted mirror.ord.rax.openstack.org per investigation in https://bugs.launchpad.net/openstack-gate/+bug/1708707 which suggested apache segfaults causing pypi download failures. Will monitor | 05:00 |
| openstack | Launchpad bug 1708707 in OpenStack-Gate "Pip finds hash mismatch for package during installation" [Undecided,New] | 05:00 |
| openstackstatus | ianw: finished logging | 05:00 |
| ianw | pabelanger: /dev/mapper/main-proxycache 99G 97G 2.1G 98% /var/cache/apache2 <-- this seems ... tight | 05:02 |
| pabelanger | ianw: which mirror is that? | 05:03 |
| ianw | mirror.ord.rax.openstack.org | 05:03 |
| *** yamamoto has quit IRC | 05:03 | |
| *** yamamoto has joined #openstack-infra | 05:04 | |
| ianw | htcacheclean is running, maybe it will prune it out | 05:04 |
| pabelanger | ya, think we cap at 80GB | 05:05 |
| ianw | only if it can clean faster than it pulls in new stuff though? | 05:06 |
| pabelanger | Ya | 05:07 |
| pabelanger | can't help tonight sadly | 05:07 |
| pabelanger | good luck | 05:07 |
| ianw | pabelanger: no probs, it's not an issue, just something to keep an eye on | 05:08 |
| *** tnovacik has quit IRC | 05:09 | |
| *** claudiub has joined #openstack-infra | 05:09 | |
| *** sree has joined #openstack-infra | 05:10 | |
| *** yamamoto has quit IRC | 05:11 | |
| *** sree has quit IRC | 05:15 | |
| *** ihrachys has quit IRC | 05:16 | |
| *** vhosakot has quit IRC | 05:16 | |
| *** thorst has joined #openstack-infra | 05:17 | |
| *** thorst has quit IRC | 05:22 | |
| *** funzo has joined #openstack-infra | 05:23 | |
| *** funzo has quit IRC | 05:23 | |
| *** funzo has joined #openstack-infra | 05:23 | |
| *** samP_ has joined #openstack-infra | 05:26 | |
| *** funzo has quit IRC | 05:27 | |
| *** slaweq has joined #openstack-infra | 05:31 | |
| *** slaweq has quit IRC | 05:35 | |
| *** claudiub has quit IRC | 05:38 | |
| *** ykarel__ has quit IRC | 05:43 | |
| *** gongysh has joined #openstack-infra | 05:43 | |
| openstackgerrit | Artur Basiak proposed openstack-infra/project-config master: Publish monasca-events-api documentation https://review.openstack.org/492805 | 05:46 |
| *** ykarel has joined #openstack-infra | 05:48 | |
| kevinbenton | ianw: hey, one of the jobs for 485333,20 is stuck. is there anyway to just retrigger that job? | 05:48 |
| *** armax has quit IRC | 05:51 | |
| *** armax has joined #openstack-infra | 05:52 | |
| ianw | kevinbenton: is it really stuck? we've seen issues with things taking a long time to cleanup, but they do | 05:52 |
| *** armax has quit IRC | 05:52 | |
| kevinbenton | ianw: well when i telnet to it it's not showing anything in the console | 05:52 |
| kevinbenton | ianw: like the usual cleaning up message | 05:52 |
| *** armax has joined #openstack-infra | 05:52 | |
| kevinbenton | ianw: oh, sorry! | 05:53 |
| kevinbenton | it's doing something now | 05:53 |
| *** armax has quit IRC | 05:53 | |
| kevinbenton | ianw: now it legitimately timed out :) | 05:53 |
| *** armax has joined #openstack-infra | 05:53 | |
| ianw | ubuntu-xenial-citycloud-lon1-10386405 ... yep it's one of our screwy london nodes | 05:54 |
| *** armax has quit IRC | 05:54 | |
| *** jamesmcarthur has joined #openstack-infra | 05:54 | |
| *** armax has joined #openstack-infra | 05:54 | |
| ianw | kevinbenton: from what we can tell, there is an unhappy compute node in that region, and sometimes jobs get sent to it. i believe it's been reported via several channels | 05:54 |
| *** armax has quit IRC | 05:54 | |
| kevinbenton | ianw: ack | 05:55 |
| *** armax has joined #openstack-infra | 05:55 | |
| *** armax has quit IRC | 05:55 | |
| *** armax has joined #openstack-infra | 05:56 | |
| *** armax has quit IRC | 05:56 | |
| *** armax has joined #openstack-infra | 05:57 | |
| *** armax has quit IRC | 05:57 | |
| *** armax has joined #openstack-infra | 05:57 | |
| *** armax has quit IRC | 05:58 | |
| *** jamesmcarthur has quit IRC | 05:59 | |
| *** gmann has quit IRC | 06:01 | |
| openstackgerrit | Artur Basiak proposed openstack-infra/project-config master: Publish monasca-events-api documentation https://review.openstack.org/492805 | 06:02 |
| *** slaweq has joined #openstack-infra | 06:04 | |
| *** vtapia has left #openstack-infra | 06:08 | |
| *** aeng has quit IRC | 06:14 | |
| *** tesseract has joined #openstack-infra | 06:16 | |
| *** martinkopec has joined #openstack-infra | 06:17 | |
| *** thorst has joined #openstack-infra | 06:18 | |
| *** priteau has joined #openstack-infra | 06:18 | |
| *** yamamoto has joined #openstack-infra | 06:19 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] create intermediate LVMNode object, refactor slightly https://review.openstack.org/492812 | 06:20 |
| *** rcernin has joined #openstack-infra | 06:22 | |
| *** Sukhdev has joined #openstack-infra | 06:22 | |
| *** thorst has quit IRC | 06:23 | |
| *** priteau has quit IRC | 06:23 | |
| *** funzo has joined #openstack-infra | 06:23 | |
| *** yamamoto has quit IRC | 06:24 | |
| *** ramishra has quit IRC | 06:26 | |
| *** ramishra has joined #openstack-infra | 06:27 | |
| *** funzo has quit IRC | 06:28 | |
| *** rtjure has joined #openstack-infra | 06:32 | |
| *** samP_ has quit IRC | 06:32 | |
| *** dhajare has joined #openstack-infra | 06:32 | |
| *** thorre_se has joined #openstack-infra | 06:33 | |
| *** Sukhdev has quit IRC | 06:34 | |
| *** thorre has quit IRC | 06:36 | |
| *** thorre_se is now known as thorre | 06:36 | |
| *** ykarel has quit IRC | 06:41 | |
| *** ykarel has joined #openstack-infra | 06:42 | |
| *** dizquierdo has joined #openstack-infra | 06:43 | |
| *** eranrom has joined #openstack-infra | 06:43 | |
| *** liujiong_lj is now known as liujiong | 06:47 | |
| *** ramishra has quit IRC | 06:51 | |
| *** yamamoto has joined #openstack-infra | 06:52 | |
| *** ramishra has joined #openstack-infra | 06:53 | |
| *** ccamacho has joined #openstack-infra | 06:53 | |
| *** cshastri has joined #openstack-infra | 06:56 | |
| *** amotoki has joined #openstack-infra | 06:57 | |
| *** Swami has quit IRC | 06:59 | |
| *** dhajare has quit IRC | 07:02 | |
| *** jaosorior has quit IRC | 07:03 | |
| *** pgadiya has joined #openstack-infra | 07:03 | |
| kklimonda | how is logs.openstack.org managed? I don't see it in puppet (only references in some yaml files) | 07:06 |
| *** yamamoto has quit IRC | 07:07 | |
| *** aarefiev_afk is now known as aarefiev | 07:08 | |
| *** eroux has quit IRC | 07:09 | |
| *** dizquierdo has quit IRC | 07:14 | |
| *** jaosorior has joined #openstack-infra | 07:15 | |
| *** markus_z has joined #openstack-infra | 07:16 | |
| *** eroux has joined #openstack-infra | 07:19 | |
| *** thorst has joined #openstack-infra | 07:19 | |
| *** gmann has joined #openstack-infra | 07:21 | |
| *** thorst has quit IRC | 07:23 | |
| *** funzo has joined #openstack-infra | 07:24 | |
| *** bogdando has joined #openstack-infra | 07:25 | |
| bogdando | hi.folks, let's merge this please https://review.openstack.org/#/c/492463/ | 07:25 |
| bogdando | do we have elastic-recheck cores around? | 07:25 |
| *** gmann has quit IRC | 07:28 | |
| *** slaweq has quit IRC | 07:28 | |
| *** funzo has quit IRC | 07:29 | |
| *** slaweq has joined #openstack-infra | 07:29 | |
| *** tmorin has joined #openstack-infra | 07:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Zuul v3: add description to check pipeline https://review.openstack.org/492710 | 07:35 |
| *** gmann has joined #openstack-infra | 07:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add documentation jobs for monasca-api https://review.openstack.org/490569 | 07:40 |
| *** sree has joined #openstack-infra | 07:40 | |
| *** rwsu has joined #openstack-infra | 07:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: networking-midonet: Remove v2 jobs from grafana dashboard https://review.openstack.org/492424 | 07:40 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Fix Generic job timeout bug match https://review.openstack.org/492463 | 07:41 |
| *** ralonsoh has joined #openstack-infra | 07:41 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add gerritbot notications for osprofiler into #openstack-performance https://review.openstack.org/492437 | 07:42 |
| *** e0ne has joined #openstack-infra | 07:42 | |
| openstackgerrit | Artur Basiak proposed openstack-infra/project-config master: Publish monasca-events-api documentation https://review.openstack.org/492805 | 07:43 |
| *** slaweq has quit IRC | 07:47 | |
| *** sflanigan has quit IRC | 07:49 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: [DNM] create intermediate LVMNode object, refactor slightly https://review.openstack.org/492812 | 07:51 |
| *** dizquierdo has joined #openstack-infra | 07:58 | |
| *** rwsu has quit IRC | 07:58 | |
| *** alexchadin has joined #openstack-infra | 07:58 | |
| *** rwsu has joined #openstack-infra | 08:00 | |
| *** openstackgerrit has quit IRC | 08:02 | |
| *** yamamoto has joined #openstack-infra | 08:04 | |
| *** gmann has quit IRC | 08:07 | |
| *** derekh has joined #openstack-infra | 08:14 | |
| *** priteau has joined #openstack-infra | 08:16 | |
| *** yamamoto has quit IRC | 08:17 | |
| *** shardy has joined #openstack-infra | 08:21 | |
| *** cuongnv has quit IRC | 08:23 | |
| *** funzo has joined #openstack-infra | 08:25 | |
| *** funzo has quit IRC | 08:29 | |
| *** benj_ has quit IRC | 08:30 | |
| *** rwsu has quit IRC | 08:37 | |
| *** bhavik1 has joined #openstack-infra | 08:43 | |
| *** yamamoto has joined #openstack-infra | 08:44 | |
| *** benj_ has joined #openstack-infra | 08:46 | |
| *** ykarel has quit IRC | 08:47 | |
| *** ykarel has joined #openstack-infra | 08:48 | |
| *** lucas-afk is now known as lucasagomes | 08:49 | |
| *** rwsu has joined #openstack-infra | 08:49 | |
| *** bhavik1 has quit IRC | 08:51 | |
| *** electrofelix has joined #openstack-infra | 08:53 | |
| *** dtantsur|afk is now known as dtantsur | 08:54 | |
| *** yamamoto has quit IRC | 08:56 | |
| *** bhavik1 has joined #openstack-infra | 08:58 | |
| *** priteau has quit IRC | 08:58 | |
| *** sambetts|afk is now known as sambetts | 09:02 | |
| Dinesh_Bhor | AJaeger_: Hi you there? | 09:05 |
| *** priteau has joined #openstack-infra | 09:08 | |
| *** alexchadin has quit IRC | 09:09 | |
| *** ykarel is now known as ykarel|lunch | 09:12 | |
| *** markvoelker has quit IRC | 09:17 | |
| *** sshnaidm|afk is now known as sshnaidm|off | 09:19 | |
| *** thorst has joined #openstack-infra | 09:21 | |
| strigazi | hi, when jenkins votes -2 we always need to run two times the pipeline with recheck? | 09:22 |
| odyssey4me | strigazi yes | 09:24 |
| *** yamamoto has joined #openstack-infra | 09:24 | |
| strigazi | odyssey4me thanks | 09:24 |
| *** thorst has quit IRC | 09:25 | |
| *** funzo has joined #openstack-infra | 09:26 | |
| *** openstackgerrit has joined #openstack-infra | 09:28 | |
| *** mriedem has quit IRC | 09:30 | |
| *** funzo has quit IRC | 09:31 | |
| *** priteau has quit IRC | 09:33 | |
| *** yamamoto has quit IRC | 09:35 | |
| *** ianychoi_ is now known as ianychoi | 09:36 | |
| *** bhavik1 has quit IRC | 09:39 | |
| *** rwsu has quit IRC | 09:44 | |
| *** priteau has joined #openstack-infra | 09:54 | |
| *** rwsu has joined #openstack-infra | 09:56 | |
| *** priteau has quit IRC | 09:59 | |
| *** sdague has joined #openstack-infra | 10:00 | |
| *** pgadiya has quit IRC | 10:01 | |
| *** sshnaidm|off has quit IRC | 10:02 | |
| *** priteau has joined #openstack-infra | 10:04 | |
| *** gmann has joined #openstack-infra | 10:04 | |
| *** ociuhandu has quit IRC | 10:08 | |
| *** pgadiya has joined #openstack-infra | 10:18 | |
| *** markvoelker has joined #openstack-infra | 10:18 | |
| *** thorst has joined #openstack-infra | 10:22 | |
| *** psachin has quit IRC | 10:22 | |
| *** markvoelker has quit IRC | 10:23 | |
| *** sree has quit IRC | 10:24 | |
| *** thorst has quit IRC | 10:26 | |
| *** funzo has joined #openstack-infra | 10:26 | |
| *** liujiong has quit IRC | 10:27 | |
| *** funzo has quit IRC | 10:31 | |
| *** yamamoto has joined #openstack-infra | 10:32 | |
| *** sshnaidm has joined #openstack-infra | 10:34 | |
| *** sshnaidm is now known as sshnaidm|off | 10:34 | |
| *** psachin has joined #openstack-infra | 10:35 | |
| *** namnh has quit IRC | 10:36 | |
| *** ykarel|lunch is now known as ykarel | 10:38 | |
| *** markvoelker has joined #openstack-infra | 10:39 | |
| *** ykarel_ has joined #openstack-infra | 10:42 | |
| *** markvoelker_ has joined #openstack-infra | 10:44 | |
| *** markvoelker has quit IRC | 10:44 | |
| *** markvoelker_ has quit IRC | 10:44 | |
| *** markvoelker has joined #openstack-infra | 10:45 | |
| *** ykarel has quit IRC | 10:46 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Rework the getthelogs helper script for wget recursive https://review.openstack.org/492178 | 10:46 |
| *** yamamoto has quit IRC | 10:48 | |
| *** jkilpatr has quit IRC | 10:52 | |
| *** rwsu has quit IRC | 10:56 | |
| *** LindaWang has quit IRC | 10:56 | |
| *** shardy is now known as shardy_lunch | 10:59 | |
| *** zhurong has quit IRC | 11:00 | |
| *** dhajare has joined #openstack-infra | 11:02 | |
| *** priteau has quit IRC | 11:04 | |
| Dinesh_Bhor | yolanda: Hi, you there? | 11:06 |
| *** mriedem has joined #openstack-infra | 11:07 | |
| *** rwsu has joined #openstack-infra | 11:08 | |
| *** slaweq has joined #openstack-infra | 11:10 | |
| *** jkilpatr has joined #openstack-infra | 11:12 | |
| *** gongysh has quit IRC | 11:17 | |
| *** udesale has quit IRC | 11:18 | |
| *** thorst has joined #openstack-infra | 11:22 | |
| *** priteau has joined #openstack-infra | 11:23 | |
| *** priteau has quit IRC | 11:24 | |
| *** makowals_ has quit IRC | 11:24 | |
| *** jkilpatr has quit IRC | 11:24 | |
| *** dhajare has quit IRC | 11:24 | |
| *** shardy_lunch has quit IRC | 11:24 | |
| *** e0ne has quit IRC | 11:24 | |
| *** cshastri has quit IRC | 11:24 | |
| *** bogdando has quit IRC | 11:24 | |
| *** martinkopec has quit IRC | 11:24 | |
| *** caphrim0_ has quit IRC | 11:24 | |
| *** lindsayk has quit IRC | 11:24 | |
| *** yamahata has quit IRC | 11:24 | |
| *** hareesh has quit IRC | 11:26 | |
| *** igormarnat has quit IRC | 11:27 | |
| *** igormarnat has joined #openstack-infra | 11:27 | |
| *** funzo has joined #openstack-infra | 11:27 | |
| *** bogdando has joined #openstack-infra | 11:27 | |
| *** thorst has quit IRC | 11:27 | |
| *** makowals has joined #openstack-infra | 11:28 | |
| *** shardy_lunch has joined #openstack-infra | 11:30 | |
| *** yamahata has joined #openstack-infra | 11:31 | |
| *** funzo has quit IRC | 11:32 | |
| *** martinkopec has joined #openstack-infra | 11:33 | |
| *** shardy_lunch is now known as shardy | 11:33 | |
| *** yamamoto has joined #openstack-infra | 11:35 | |
| *** atarakt has quit IRC | 11:39 | |
| *** mdbooth has quit IRC | 11:39 | |
| *** atarakt has joined #openstack-infra | 11:41 | |
| *** mdbooth has joined #openstack-infra | 11:42 | |
| *** sbezverk has joined #openstack-infra | 11:42 | |
| *** slaweq has quit IRC | 11:42 | |
| *** rwsu has quit IRC | 11:42 | |
| *** dizquierdo has quit IRC | 11:45 | |
| *** yamamoto has quit IRC | 11:48 | |
| *** yamamoto has joined #openstack-infra | 11:48 | |
| *** sbezverk has quit IRC | 11:50 | |
| *** e0ne has joined #openstack-infra | 11:51 | |
| *** dhajare has joined #openstack-infra | 11:51 | |
| *** cshastri has joined #openstack-infra | 11:51 | |
| *** caphrim007 has joined #openstack-infra | 11:51 | |
| *** jkilpatr has joined #openstack-infra | 11:51 | |
| *** ldnunes has joined #openstack-infra | 11:52 | |
| *** priteau has joined #openstack-infra | 11:53 | |
| *** priteau has quit IRC | 11:53 | |
| *** lucasagomes is now known as lucas-hungry | 11:54 | |
| *** yamamoto has quit IRC | 11:55 | |
| *** dhajare has quit IRC | 11:55 | |
| *** yamamoto has joined #openstack-infra | 11:57 | |
| *** dhajare has joined #openstack-infra | 11:59 | |
| *** EricGonczer_ has joined #openstack-infra | 12:01 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Switch to use local buildlogs mirror https://review.openstack.org/492786 | 12:03 |
| *** arturb has quit IRC | 12:07 | |
| *** thorst has joined #openstack-infra | 12:08 | |
| *** dhajare has quit IRC | 12:09 | |
| *** markvoelker_ has joined #openstack-infra | 12:10 | |
| *** rhallisey has joined #openstack-infra | 12:10 | |
| *** markvoelker has quit IRC | 12:13 | |
| *** trown|outtypewww is now known as trown | 12:14 | |
| *** LindaWang has joined #openstack-infra | 12:14 | |
| *** pgadiya has quit IRC | 12:16 | |
| *** sshnaidm|off has quit IRC | 12:17 | |
| *** EricGonczer_ has quit IRC | 12:19 | |
| *** Douhet has quit IRC | 12:19 | |
| *** mwhahaha has quit IRC | 12:19 | |
| *** dasanind has quit IRC | 12:19 | |
| *** rockstar has quit IRC | 12:19 | |
| *** fdegir has quit IRC | 12:19 | |
| *** simonmcc has quit IRC | 12:19 | |
| *** TheJulia has quit IRC | 12:19 | |
| *** puiterwijk|cld has quit IRC | 12:19 | |
| *** betherly has quit IRC | 12:19 | |
| *** berendt has quit IRC | 12:19 | |
| *** puiterwijk|cld has joined #openstack-infra | 12:19 | |
| *** Douhet has joined #openstack-infra | 12:19 | |
| *** rockstar has joined #openstack-infra | 12:19 | |
| *** berendt has joined #openstack-infra | 12:19 | |
| *** TheJulia has joined #openstack-infra | 12:19 | |
| *** fdegir has joined #openstack-infra | 12:19 | |
| *** gouthamr has joined #openstack-infra | 12:20 | |
| *** dasanind has joined #openstack-infra | 12:20 | |
| *** simonmcc has joined #openstack-infra | 12:20 | |
| *** betherly has joined #openstack-infra | 12:20 | |
| *** mwhahaha has joined #openstack-infra | 12:20 | |
| *** EricGonczer_ has joined #openstack-infra | 12:20 | |
| *** gongysh has joined #openstack-infra | 12:20 | |
| *** gongysh has quit IRC | 12:20 | |
| mhayden | pabelanger: thanks for the comments in https://review.openstack.org/492601 -- i've got a patch up for openstack-ansible-tests that actually gets the nodepool mirror url and passes it to that role | 12:21 |
| *** clayton has quit IRC | 12:21 | |
| *** purp has quit IRC | 12:21 | |
| *** mandre_mtg has quit IRC | 12:22 | |
| *** makowals has quit IRC | 12:22 | |
| AJaeger_ | strigazi: read http://lists.openstack.org/pipermail/openstack-dev/2017-August/120608.html | 12:22 |
| *** rtjure has quit IRC | 12:23 | |
| AJaeger_ | Dinesh_Bhor: just ask here, others might be able to help as well. I'm not really around... | 12:23 |
| *** clayton has joined #openstack-infra | 12:24 | |
| *** mandre has joined #openstack-infra | 12:24 | |
| *** dave-mccowan has joined #openstack-infra | 12:24 | |
| *** makowals has joined #openstack-infra | 12:24 | |
| *** rtjure has joined #openstack-infra | 12:26 | |
| *** purp has joined #openstack-infra | 12:26 | |
| Dinesh_Bhor | AJaeger_: yeah, the problem is solved now. thanks for reply | 12:27 |
| *** baoli has quit IRC | 12:27 | |
| *** jamesmcarthur has joined #openstack-infra | 12:28 | |
| *** xyang1 has joined #openstack-infra | 12:29 | |
| *** dizquierdo has joined #openstack-infra | 12:31 | |
| *** Hunner has quit IRC | 12:31 | |
| *** gongysh has joined #openstack-infra | 12:35 | |
| *** Hunner has joined #openstack-infra | 12:36 | |
| *** Hunner has quit IRC | 12:36 | |
| *** Hunner has joined #openstack-infra | 12:36 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Remove query for bug 1696834 https://review.openstack.org/492985 | 12:38 |
| openstack | bug 1696834 in OpenStack Compute (nova) "Intermittent "KeyError: 'allocations'" in functional tests" [Medium,Fix released] https://launchpad.net/bugs/1696834 - Assigned to Chris Dent (cdent) | 12:38 |
| *** jtomasek has joined #openstack-infra | 12:39 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Remove query for bug 1696834 https://review.openstack.org/492985 | 12:40 |
| openstack | bug 1696834 in OpenStack Compute (nova) "Intermittent "KeyError: 'allocations'" in functional tests" [Medium,Fix released] https://launchpad.net/bugs/1696834 - Assigned to Chris Dent (cdent) | 12:40 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Remove query for bug 1709739 https://review.openstack.org/492985 | 12:41 |
| openstack | bug 1709739 in tempest ""Invalid volume: Volume is not local to this node" in neutron multinode job (pike)" [Undecided,In progress] https://launchpad.net/bugs/1709739 - Assigned to Xing Yang (xing-yang) | 12:41 |
| mnaser | infra-root would it be ok if i propose dropping citycloud-lon1 to max-servers: 0? | 12:43 |
| *** pradk has joined #openstack-infra | 12:44 | |
| mnaser | 492390,5 is about to fail gate after ~3hours of CI jobs because we landed on it | 12:44 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/492923 | 12:45 |
| *** makowals has quit IRC | 12:45 | |
| *** makowals has joined #openstack-infra | 12:47 | |
| *** pblaho has quit IRC | 12:47 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add mirror host for images.linuxcontainers.org reverse proxy https://review.openstack.org/492192 | 12:51 |
| *** jpena|off is now known as jpena | 12:54 | |
| *** jaypipes has joined #openstack-infra | 12:57 | |
| *** LindaWang has quit IRC | 12:57 | |
| *** jcoufal has joined #openstack-infra | 12:58 | |
| *** LindaWang has joined #openstack-infra | 12:59 | |
| *** jaypipes has quit IRC | 13:00 | |
| *** jaypipes has joined #openstack-infra | 13:00 | |
| *** jamesmcarthur has quit IRC | 13:02 | |
| *** mpaolino has joined #openstack-infra | 13:03 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Remove query for bug 1709739 https://review.openstack.org/492985 | 13:04 |
| openstack | bug 1709739 in tempest ""Invalid volume: Volume is not local to this node" in neutron multinode job (pike)" [Undecided,In progress] https://launchpad.net/bugs/1709739 - Assigned to Xing Yang (xing-yang) | 13:04 |
| *** lucas-hungry is now known as lucasagomes | 13:05 | |
| *** fried_rice has joined #openstack-infra | 13:07 | |
| fried_rice | AJaeger_ greetings | 13:07 |
| fried_rice | mordred See https://review.openstack.org/#/c/490750/ -- how/when does that publish job get run? | 13:08 |
| pabelanger | mnaser: ya, clarkb wants to do an audit of our lon1 mirror to confirm it is not related. But starting to look we might have to disable that region | 13:08 |
| *** sbezverk has joined #openstack-infra | 13:09 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Removed noise from log (OAUTH2.0) https://review.openstack.org/492997 | 13:11 |
| *** sekelso has joined #openstack-infra | 13:12 | |
| *** rlandy has joined #openstack-infra | 13:12 | |
| openstackgerrit | Sean Handley proposed openstack/os-client-config master: DataCentred supports Keystone V3 and Glance V2. https://review.openstack.org/493002 | 13:12 |
| *** skelso has joined #openstack-infra | 13:15 | |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Remove OpenStack/Ceph/Virt repo from CentOS https://review.openstack.org/493003 | 13:15 |
| mhayden | pabelanger: a review for discussion ^^ | 13:15 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add kpartx as a requirement to build images https://review.openstack.org/493004 | 13:16 |
| *** Hal has joined #openstack-infra | 13:16 | |
| *** Hal is now known as Guest66310 | 13:17 | |
| *** sekelso has quit IRC | 13:17 | |
| *** baoli has joined #openstack-infra | 13:18 | |
| *** lihi has quit IRC | 13:18 | |
| *** oanson has quit IRC | 13:19 | |
| *** dimak has quit IRC | 13:19 | |
| *** dimak has joined #openstack-infra | 13:19 | |
| *** oanson has joined #openstack-infra | 13:19 | |
| openstackgerrit | Merged openstack-infra/openstackid master: Removed noise from log (OAUTH2.0) https://review.openstack.org/492997 | 13:19 |
| *** lihi has joined #openstack-infra | 13:20 | |
| pabelanger | mhayden: support it, but left a -1 and comments. We have somethings we need to fix first | 13:22 |
| *** dprince has joined #openstack-infra | 13:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add zuul.project.src_dir variable https://review.openstack.org/491915 | 13:26 |
| *** sshnaidm has joined #openstack-infra | 13:27 | |
| *** funzo has joined #openstack-infra | 13:29 | |
| mnaser | clarkb pabelanger http://logs.openstack.org/90/492390/5/gate/gate-functional-dsvm-magnum-api-ubuntu-xenial/c203e1f/logs/devstacklog.txt.gz#_2017-08-11_12_02_11_081 | 13:29 |
| mnaser | Resolving mirror.lon1.citycloud.openstack.org (mirror.lon1.citycloud.openstack.org)... 37.153.172.44 | 13:30 |
| mnaser | do the mirror and workers sit on the same network? | 13:30 |
| *** udesale has joined #openstack-infra | 13:30 | |
| mnaser | if so, could we somehow reach it over the internet network rather than over NAT.. im willing to bet it would be much faster. | 13:30 |
| *** jaypipes is now known as leakypipes | 13:30 | |
| pabelanger | mnaser: mirror and nodes are in different projects, need to confirm if shared network | 13:31 |
| pabelanger | but ya, right now they hit public web | 13:32 |
| openstackgerrit | Merged openstack-infra/system-config master: Add https://api.rubygems.org/ to reverse proxy cache https://review.openstack.org/492760 | 13:32 |
| mnaser | yeah cause that's what is timing us out now :( | 13:32 |
| pabelanger | mnaser: Hmm, could be possible AFS cache was cold. Was this first time job ran in citycloud-lon1? | 13:32 |
| *** esberglu has joined #openstack-infra | 13:33 | |
| mnaser | pabelanger let me do a small check and see | 13:33 |
| *** pradk has quit IRC | 13:33 | |
| *** funzo has quit IRC | 13:33 | |
| pabelanger | mnaser: that would be my guess, since fedora-alt is new | 13:33 |
| mnaser | pabelanger it does indeed seem like it | 13:33 |
| pabelanger | I believe there is a way to check with AFS client, but not sure the command. Will have to google | 13:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 13:35 |
| *** bmjen has quit IRC | 13:36 | |
| *** makowals has quit IRC | 13:36 | |
| mnaser | pabelanger ill keep an eye out on the region of failure and let you know if this happens again | 13:36 |
| mnaser | if it happens more than once in the same region at least | 13:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Rename tox_command_line in docs to tox_extra_args https://review.openstack.org/489758 | 13:37 |
| pabelanger | mnaser: ya, lon1 would be slow first time too. Data lives in DFW I believe | 13:37 |
| fungi | kklimonda: not sure if you found it yet, but the logs.o.o site is a vhost on static.openstack.org managed via the openstack_project::static class in the openstack-infra/system-config repo | 13:37 |
| *** bobh has joined #openstack-infra | 13:39 | |
| *** makowals has joined #openstack-infra | 13:39 | |
| kklimonda | fungi: thanks, I didn't have time to look into it. | 13:39 |
| fungi | mnaser: yeah, we were discussing also reverting the change to return our quota in citycloud-sto2 and see if the network issues from a few weeks ago have been resolved, since the engineer we had been talking to about that one seemed to be on to some leads there but then fell silent | 13:39 |
| mnaser | fungi it could be just afs cache misses in our case but i heard a lot of other folks have issues so i figured it was related | 13:39 |
| *** bmjen has joined #openstack-infra | 13:41 | |
| pabelanger | ++ to try citycloud-sto2 again | 13:42 |
| fungi | mnaser: well, we were definitely seeing performance issues there throughout the day yesterday too. there is an outside chance that becuase the way they migrated the mirror instance caused cloud-init to replace it's ssh host key and so the flock wrapper around htcacheclean never made it on the server, we got it back into a sad state that way, but i have doubts that's the whole story there | 13:42 |
| mnaser | ouch | 13:43 |
| *** weshay is now known as weshay_doc | 13:43 | |
| *** dizquierdo has quit IRC | 13:43 | |
| *** hongbin has joined #openstack-infra | 13:44 | |
| pabelanger | fungi: clarkb: we'll need to land https://review.openstack.org/#/c/492712/ for infracloud, puppet provisioned our DNS back to 8.8.8.8 again | 13:44 |
| fungi | i kinda figured that might happen, thanks for confirming and finding where to set it properly | 13:44 |
| *** ldnunes has quit IRC | 13:47 | |
| *** jpena is now known as jpena|off | 13:48 | |
| *** markvoelker_ has quit IRC | 13:52 | |
| *** felipemonteiro has joined #openstack-infra | 13:55 | |
| *** trown is now known as trown|brb | 13:57 | |
| *** ldnunes has joined #openstack-infra | 14:00 | |
| *** mat128 has joined #openstack-infra | 14:04 | |
| *** tnovacik has joined #openstack-infra | 14:05 | |
| *** skelso has quit IRC | 14:05 | |
| samueldmq | dhellmann: thanks for http://logs.openstack.org/67/492767/2/check/gate-releases-tox-list-changes-ubuntu-xenial/0d2ac66/console.html#_2017-08-11_02_13_59_846806 | 14:07 |
| samueldmq | cc lbragstad | 14:07 |
| *** skelso has joined #openstack-infra | 14:07 | |
| *** mat128 has quit IRC | 14:14 | |
| *** tnovacik has quit IRC | 14:15 | |
| *** Guest66310 has quit IRC | 14:16 | |
| *** trown|brb is now known as trown | 14:17 | |
| *** sshnaidm is now known as sshnaidm|off | 14:17 | |
| openstackgerrit | Julia Kreger proposed openstack/diskimage-builder master: Add netbase to ensure /etc/protocols is placed for debian https://review.openstack.org/490656 | 14:18 |
| *** jtomasek has quit IRC | 14:19 | |
| *** sambetts is now known as sambetts|afk | 14:21 | |
| *** rbrndt has joined #openstack-infra | 14:21 | |
| *** mat128 has joined #openstack-infra | 14:22 | |
| *** bogdando has quit IRC | 14:22 | |
| *** psachin has quit IRC | 14:24 | |
| *** mat128 has quit IRC | 14:24 | |
| *** caphrim007 has quit IRC | 14:29 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Increase timeout for removal https://review.openstack.org/493026 | 14:29 |
| *** jamesmcarthur has joined #openstack-infra | 14:29 | |
| *** funzo has joined #openstack-infra | 14:29 | |
| *** jamesmcarthur has quit IRC | 14:30 | |
| *** Hal has joined #openstack-infra | 14:30 | |
| *** jamesmcarthur has joined #openstack-infra | 14:31 | |
| *** Hal is now known as Guest12736 | 14:31 | |
| *** armax has joined #openstack-infra | 14:31 | |
| *** jtomasek has joined #openstack-infra | 14:33 | |
| *** funzo has quit IRC | 14:34 | |
| *** SamYaple has quit IRC | 14:35 | |
| *** SamYaple has joined #openstack-infra | 14:35 | |
| *** rwsu has joined #openstack-infra | 14:36 | |
| *** mat128 has joined #openstack-infra | 14:36 | |
| *** Julien-zte has quit IRC | 14:36 | |
| *** udesale has quit IRC | 14:37 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add support for NODEPOOL_RUGYGEMS_PROXY https://review.openstack.org/493028 | 14:40 |
| pabelanger | fungi: clarkb: EmilienM: mwhahaha: ^finally support for rubygems reverse proxy cache. Tested in infracloud-vanilla and works as expected | 14:40 |
| pabelanger | 15.184.66.2 - - [11/Aug/2017:14:33:56 +0000] "GET /api.rubygems/quick/Marshal.4.8/semantic_puppet-1.0.1.gemspec.rz HTTP/1.1" 200 1461 "-" "Ruby, RubyGems/2.0.14.1 x86_64-linux Ruby/2.0.0 (2015-12-16 patchlevel 648)" | 14:41 |
| *** mat128 has quit IRC | 14:49 | |
| *** bobh has quit IRC | 14:49 | |
| *** mat128 has joined #openstack-infra | 14:52 | |
| *** EricGonczer_ has quit IRC | 14:54 | |
| *** sbezverk has quit IRC | 14:54 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:55 | |
| *** ramishra has quit IRC | 14:55 | |
| *** felipemonteiro has quit IRC | 14:55 | |
| *** sbezverk has joined #openstack-infra | 14:55 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack-infra/project-config master: Skip non-doc bifrost jobs on reno-only changes https://review.openstack.org/493034 | 14:56 |
| *** rcernin has quit IRC | 14:57 | |
| *** EricGonczer_ has joined #openstack-infra | 14:58 | |
| *** thingee_ has joined #openstack-infra | 14:58 | |
| pabelanger | fungi: clarkb: mnaser: did we want to try launch another instances in vexxhost today? | 14:59 |
| mnaser | pabelanger we could, let me make some adjustments on my side, i want to push up a change on the flavor name | 15:00 |
| mnaser | i think 2 core thing wasnt successful because jobs were running with NPROC concurrency | 15:00 |
| *** xarses_ has joined #openstack-infra | 15:00 | |
| *** EricGonc_ has joined #openstack-infra | 15:01 | |
| *** EricGonczer_ has quit IRC | 15:02 | |
| *** bh526r has joined #openstack-infra | 15:04 | |
| *** cshastri has quit IRC | 15:05 | |
| *** ykarel_ has quit IRC | 15:05 | |
| *** skelso has quit IRC | 15:07 | |
| *** claudiub has joined #openstack-infra | 15:09 | |
| *** mpaolino has quit IRC | 15:09 | |
| *** ramishra has joined #openstack-infra | 15:09 | |
| *** bswartz has joined #openstack-infra | 15:09 | |
| *** skelso has joined #openstack-infra | 15:10 | |
| *** annegentle has joined #openstack-infra | 15:14 | |
| *** Dinesh_Bhor has quit IRC | 15:15 | |
| *** dklyle has joined #openstack-infra | 15:15 | |
| *** david-lyle has quit IRC | 15:15 | |
| *** ihrachys has joined #openstack-infra | 15:16 | |
| *** dprince has quit IRC | 15:17 | |
| *** LindaWang has quit IRC | 15:21 | |
| *** shardy has quit IRC | 15:24 | |
| *** jtomasek has quit IRC | 15:25 | |
| *** Guest12736 has quit IRC | 15:25 | |
| *** jtomasek has joined #openstack-infra | 15:26 | |
| *** jaosorior has quit IRC | 15:27 | |
| AJaeger_ | odyssey4me: you don not need to rebase in gerrit - see https://review.openstack.org/491195 a recheck would have been enough. We always rebase before running any of the check or gate jobs | 15:27 |
| *** e0ne has quit IRC | 15:27 | |
| odyssey4me | AJaeger_ I thought it did the check using a checkout, then a rebase in gate | 15:28 |
| fungi | (that is, zuul transparently rebases it to test it) | 15:28 |
| *** jtomasek has quit IRC | 15:28 | |
| *** baoli has quit IRC | 15:28 | |
| fungi | odyssey4me: all jobs zuul runs | 15:28 |
| fungi | whether in the check or gate pipeline | 15:28 |
| odyssey4me | from my own experience it appears to me that it works that way, but I don't have evidence off-hand to prove it | 15:28 |
| AJaeger_ | odyssey4me: we want the same test runs in both check and gate if nothing else merges... | 15:29 |
| AJaeger_ | odyssey4me: believe us ;) It's designed this way :) | 15:29 |
| odyssey4me | except that in gate it is definitely rebased is there is another patch in the queue for the same repo | 15:29 |
| fungi | if you recheck a change after some other change merges to the repo, you get tested with the new state of the repo even if you don't rebase your change | 15:29 |
| AJaeger_ | odyssey4me: thanks for caring to get my change in ;) | 15:30 |
| fungi | this is also why we automatically leave verify -1 votes on changes once they merge-conflict with the tip of the branch to which they're targeted | 15:30 |
| fungi | because we can no longer test them | 15:30 |
| *** funzo has joined #openstack-infra | 15:30 | |
| *** baoli has joined #openstack-infra | 15:30 | |
| odyssey4me | AJaeger_ I'm keeping an eye on it, and I'll port it back too once it's done. That script helps people testing outside of infra to get speedier results. | 15:30 |
| fungi | odyssey4me: where you _do_ need to rebase a change is if its parent is another unmerged change in review and you upload a new patchset for the parent | 15:31 |
| odyssey4me | fungi it's possible that my experience was based on before we used zuul_prep for the repo, but instead used git_prep | 15:31 |
| fungi | gerrit-git-prep does the same | 15:32 |
| *** skelso has quit IRC | 15:32 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Add the parameter parsing functionality to all of the search views https://review.openstack.org/455025 | 15:32 |
| AJaeger_ | odyssey4me: this reminds me, I always wanted to ask the team here for your script: | 15:32 |
| AJaeger_ | team, http://git.openstack.org/cgit/openstack/openstack-ansible/tree/scripts/fastest-infra-wheel-mirror.py is a script for users of ansible to use the Infra mirrors - is this something we're fine with? | 15:33 |
| fungi | or rather, the gerrit-git-prep.sh script _did_ the same up to the point where it was retired and removed from project-config ;) | 15:33 |
| *** martinkopec has quit IRC | 15:33 | |
| odyssey4me | AJaeger_ ah yes, I did actually ask before building it - although it was in an informal way | 15:33 |
| *** caphrim007 has joined #openstack-infra | 15:33 | |
| odyssey4me | I got the answer that the mirrors were freely and openly available on purpose to be used for testing | 15:33 |
| odyssey4me | but, things do change over time | 15:33 |
| *** ociuhandu has joined #openstack-infra | 15:34 | |
| AJaeger_ | odyssey4me: the difference between check and gate is: If you have a change in gate in front of your change, it's based on top of that change. If you have multiple changes in check then each is rebased on master but not on each other since we don't know how they merge | 15:34 |
| AJaeger_ | odyssey4me: thanks for confirming | 15:34 |
| *** funzo has quit IRC | 15:34 | |
| odyssey4me | Note that the script is *only* used as a wheel source for pip and is used alongside pypi - it's *only* used for someone building an AIO (dev/test - kinda like devstack) although someone could use it outside of that I guess. | 15:36 |
| *** kjackal_ has quit IRC | 15:36 | |
| fungi | odyssey4me: basically, when zuul gets ready to run a job, it asks a zuul merger process to stack the required changes onto their target branch tip(s) in the determined order, and then serves that from the server in $ZUUL_URL as a unique named ref listed in the $ZUUL_REF envvar. you can see the last state of gerrit-git-prep.sh when it was removed, on line 49 it attempts to checkout the supplied | 15:37 |
| fungi | $ZUUL_REF from $ZUUL_URL: https://review.openstack.org/454462 | 15:37 |
| *** bauzas is now known as bauzas_afk | 15:37 | |
| odyssey4me | fungi odd then how I saw some different results - it was some time ago though, I'll endeavour to find and report another one if I ever see it again | 15:38 |
| fungi | it has worked more or less the same way basically for as long as zuul has existed | 15:38 |
| fungi | odyssey4me: it's possible your jobs weren't using any of our standard checkout methods and instead had some bespoke tooling to supply the git repos they were using, i suppose | 15:38 |
| fungi | that's the only way i can imagine it would happen | 15:39 |
| *** skelso has joined #openstack-infra | 15:39 | |
| odyssey4me | fungi unlikely, this was specifically two patches in one repo of ours - not something we pull in from somewhere else | 15:39 |
| *** Hal has joined #openstack-infra | 15:39 | |
| odyssey4me | but as I say, this is part of my old, fading memory so I could be wrong | 15:39 |
| fungi | odyssey4me: if you ever see that behavior, please report it because we definitely would consider it a severe bug in the ci system | 15:40 |
| fungi | going back many years now | 15:40 |
| *** Hal is now known as Guest8912 | 15:40 | |
| odyssey4me | will definitely do! | 15:40 |
| odyssey4me | hopefully in the next cycle we'll be doing less of the bespoke tooling, and merging efforts more with infra and other ansible users | 15:41 |
| fungi | neat! | 15:41 |
| *** ramishra has quit IRC | 15:41 | |
| dims | AJaeger_ : we are branching devstack/grenade/requirements over on -release - there's a note on translations in https://github.com/openstack/releases/blob/master/PROCESS.rst#rc1, want to join us? | 15:41 |
| clarkb | pabelanger: ya lon1 mirror was operating as expected with htcacheclean and the io situation wasnt significanlty different (though load was much lower due to fewer processes running) | 15:44 |
| clarkb | pabelanger: I think we should probably disable lon1 | 15:44 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Reduced OpenID log noise * added a base class for all openid exceptions * moved invalid assoc exception from 500 to 400 * fixed bug on openid mememto request https://review.openstack.org/493053 | 15:45 |
| *** jzerebec1i is now known as jzerebecki | 15:46 | |
| *** aarefiev is now known as aarefiev_afk | 15:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename allow-secrets to pre-review https://review.openstack.org/493054 | 15:48 |
| pabelanger | clarkb: k, give me a few minutes and I'll propose a patch | 15:49 |
| *** kjackal_ has joined #openstack-infra | 15:49 | |
| *** ihrachys has quit IRC | 15:50 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/devstack-gate master: Update grenade settings for stable/pike https://review.openstack.org/493057 | 15:53 |
| openstackgerrit | Merged openstack-infra/openstackid master: Reduced OpenID log noise * added a base class for all openid exceptions * moved invalid assoc exception from 500 to 400 * fixed bug on openid mememto request https://review.openstack.org/493053 | 15:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Disable citycloud-lon1 region in nodepool https://review.openstack.org/493058 | 15:55 |
| pabelanger | clarkb: fungi: ^nodepool change for lon1 | 15:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename allow-secrets to post-review https://review.openstack.org/493059 | 15:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename allow-secrets to pre-review https://review.openstack.org/493054 | 15:57 |
| clarkb | did anyone else want to weigh in on lon1 removal before it gets approved? | 15:57 |
| *** Apoorva has joined #openstack-infra | 15:58 | |
| *** skelso has quit IRC | 15:59 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/devstack-gate master: Fix newton forward upgrade test https://review.openstack.org/493060 | 15:59 |
| *** tmorin has left #openstack-infra | 16:01 | |
| *** skelso has joined #openstack-infra | 16:03 | |
| *** armax has quit IRC | 16:04 | |
| *** sbezverk has quit IRC | 16:05 | |
| jeblair | clarkb: i could add ":(" but i don't think that's helpful so carry on | 16:05 |
| fungi | i agree in principle, and trust that you know which key makes the 0 | 16:05 |
| jeblair | fungi: did you know on *some* keyboards it's right next to the key that makes the O? can you believe it? | 16:06 |
| fungi | jeblair: i even have a keyboard or two like that | 16:07 |
| jeblair | amazing! | 16:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Skip non-doc bifrost jobs on reno-only changes https://review.openstack.org/493034 | 16:07 |
| fungi | it's why i always use fonts that make my 0 look like ø | 16:07 |
| clarkb | I;ve enjoyed source code pro for this reason | 16:08 |
| fungi | is that the name of a font? | 16:08 |
| jeblair | sometimes li uses fonts that make j's look like i and u's look like v | 16:08 |
| fungi | that's called classical latin ;) | 16:09 |
| jeblair | (the history of J is fascinating, btw) | 16:09 |
| clarkb | fungi: yup its an open source font from adobe | 16:09 |
| clarkb | are Ws two Vs? | 16:09 |
| *** ociuhandu has quit IRC | 16:10 | |
| fungi | if you speak german, a w is just one v (and the v is an f!) | 16:10 |
| fungi | in welsh, the u is an i sound too. fun stuff | 16:11 |
| *** apetrich has quit IRC | 16:11 | |
| jeblair | i don't know as much about w, i should learn | 16:11 |
| fungi | also, welsh has that fun ll that is nearly impossible to pronounce unless you're born in a small fishing village | 16:12 |
| *** apetrich has joined #openstack-infra | 16:13 | |
| fungi | 493058 | 16:14 |
| fungi | er, mispaste | 16:14 |
| jeblair | i pulled it up anyway | 16:14 |
| jeblair | apparently i respond to very abbreviated commands | 16:15 |
| fungi | heh | 16:15 |
| jeblair | pabelanger: more zuulv3 internap fun: http://logs.openstack.org/59/493059/1/check/tox-pep8/a891af6/job-output.txt.gz#_2017-08-11_16_01_23_882463 | 16:16 |
| fungi | clarkb: so i had to look it up... i'm actually using a circa 1982 reproduction commodore 64 screen font at the moment | 16:16 |
| fungi | (courtesy of cool-retro-term) | 16:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Hack in linux bridge vxlan support https://review.openstack.org/492654 | 16:16 |
| mgagne | jeblair: is it supposed to resolve outside the infra? | 16:17 |
| clarkb | fungi: https://en.wikipedia.org/wiki/Source_Code_Pro is source code pro | 16:17 |
| jeblair | mgagne: yes, i think that hostname is constructed wrong (we build it up from variables) | 16:17 |
| mgagne | how about mirror.mtl01.internap.openstack.org instead ? | 16:17 |
| *** vhosakot has joined #openstack-infra | 16:17 | |
| fungi | mgagne: right, that's what it was supposed to be | 16:18 |
| jeblair | mgagne: yes that's what it should be | 16:18 |
| * mgagne feels better now | 16:18 | |
| mgagne | I thought it was related to our infra =) | 16:18 |
| fungi | not an internap problem, no, just a zuul v3 (or perhaps job-specific?) bug | 16:18 |
| fungi | well, role-specific i mean | 16:18 |
| mgagne | oh btw, how hard to rename internap for inap everywhere the occurence is found? =) | 16:19 |
| jeblair | mgagne: oh no, sorry -- we're just adding more clouds to our small zuul v3 installation, and it's uncovering bugs in (zuul/nodepool/our zuulv3 jobs) | 16:19 |
| *** gongysh has quit IRC | 16:19 | |
| mgagne | jeblair: yea, I saw the review related to zuulv3 | 16:19 |
| *** trown is now known as trown|lunch | 16:19 | |
| *** dtantsur is now known as dtantsur|afk | 16:20 | |
| jeblair | mgagne: it would probably take several days; should we do that? | 16:20 |
| mgagne | jeblair: we got instructions to rename everything to inap asap | 16:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Disable citycloud-lon1 region in nodepool https://review.openstack.org/493058 | 16:21 |
| mgagne | I know that resources are very much busy due to release so I guess it won't be easy to rename everything without disturbing CI | 16:22 |
| jeblair | i think the steps would be: 1) new entries in dns 2) add a second provider in nodepool to let it upload images 3) switch the quota to the new provider 4) remove the old provider | 16:23 |
| jeblair | just so we don't go insane, we'd probably want to 5) rebuild the mirror server with the new name | 16:23 |
| jlvillal | AJaeger_: Do you know if openstack-dev/devstack-vagrant is still in a working condition? Haven't seen a commit to it in over two months. I saw your name as one of the committers. | 16:23 |
| jlvillal | I've tried using it, but fail so far. | 16:24 |
| *** markvoelker has joined #openstack-infra | 16:25 | |
| mgagne | jeblair: ok, I'm not familiar with the procedure and I know you people probably have plenty in your plate already. If we can find time, it would be greatly appreciated. | 16:25 |
| pabelanger | wow, new DNS, neat | 16:25 |
| *** rlandy is now known as rlandy|brb | 16:27 | |
| jeblair | mgagne: is the request related to just our use in openstack-ci? or is it a rebranding of the service in general. ie, should we also look at updating the cloud profile name in os-client-config, etc? | 16:28 |
| pabelanger | jeblair: would it be better to have nodepool.cloud inventory variable for zuulv3, instead of provider.name? | 16:29 |
| mgagne | jeblair: we can say that every occurrences of "internap" should be removed and be replaced by "inap". | 16:29 |
| jeblair | pabelanger: it looks like that's what we use to make that hostname, so i guess we should add it.... how is it that infracloud works? | 16:30 |
| mgagne | jeblair: I can understand if "internap" should be kept in some places for backward compat reasons. | 16:30 |
| mgagne | jeblair: so yes, I think this includes os-client-config | 16:30 |
| jeblair | mgagne: yeah, i expect for occ they'll want to keep both | 16:30 |
| *** amotoki is now known as amotoki__away | 16:30 | |
| pabelanger | jeblair: infracloud-vanilla provider name and cloud are the same, but the regonname is regionOne. Which is slightly different then internal, where we have region name in provider name | 16:31 |
| *** jamesmcarthur has quit IRC | 16:31 | |
| jeblair | pabelanger: gotcha. so switching to cloud will work in all cases. | 16:32 |
| jeblair | pabelanger: you want to add that to zuul? | 16:32 |
| pabelanger | Sure, give me a second | 16:32 |
| *** jamesmcarthur has joined #openstack-infra | 16:33 | |
| *** jogo has quit IRC | 16:34 | |
| *** derekh has quit IRC | 16:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add inap cloud https://review.openstack.org/493072 | 16:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Stop using internap in favor if inap https://review.openstack.org/493073 | 16:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Remove internap in favor of inap https://review.openstack.org/493074 | 16:35 |
| jeblair | mgagne: ^ that's step 2-4 | 16:35 |
| mgagne | :D | 16:36 |
| *** psachin has joined #openstack-infra | 16:36 | |
| *** jamesmcarthur has quit IRC | 16:37 | |
| mgagne | jeblair: will grafana dashboards and metrics need to be renamed too? | 16:37 |
| mgagne | didn't read yet if it was done already | 16:37 |
| pabelanger | jeblair: Hmm, I think we might also need to update nodepool and set the cloud in zk? | 16:37 |
| jeblair | mgagne: it might be best if you proposed the os-client-config change? it should be pretty straightforward... | 16:37 |
| mgagne | jeblair: will check next week to make it happen and see how we can support both names if possible | 16:38 |
| jeblair | mgagne: i forgot grafana :) the easiest thing to do is to just make new dashboards and retire the old; actually moving the data is annoying and probably not necessary. | 16:38 |
| *** EricGonc_ has quit IRC | 16:38 | |
| mgagne | yea, I'm not emotionally attached to the data history | 16:38 |
| jeblair | mgagne: When used as a proper noun, should Inap be capitalized or not? | 16:39 |
| mgagne | I will need to check. I suspect it's INAP but I'm not sure | 16:40 |
| mgagne | INAP looks to be used in all our internal communications so far | 16:40 |
| *** tesseract has quit IRC | 16:41 | |
| *** lucasagomes is now known as lucas-afk | 16:41 | |
| *** sshnaidm|off has quit IRC | 16:42 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Remove internap in favor of inap https://review.openstack.org/493074 | 16:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add inap cloud https://review.openstack.org/493072 | 16:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Stop using internap in favor if inap https://review.openstack.org/493073 | 16:42 |
| mgagne | thanks! | 16:43 |
| jeblair | mgagne: no problem -- i'm going to revise those one more time for nodepool v3 | 16:43 |
| *** ihrachys has joined #openstack-infra | 16:44 | |
| *** dklyle has quit IRC | 16:44 | |
| *** dklyle has joined #openstack-infra | 16:45 | |
| *** dprince has joined #openstack-infra | 16:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Remove internap in favor of inap https://review.openstack.org/493074 | 16:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add inap cloud https://review.openstack.org/493072 | 16:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Stop using internap in favor if inap https://review.openstack.org/493073 | 16:45 |
| jeblair | there we go | 16:46 |
| clarkb | pabelanger: odyssey4me so it turns out that there are places that may assume an ovs bridge as the bridge type. Found it in devstack for neutron at least and maybe ironic/tripleo as well? Working on a patch to address the devstack+neutron side and we'll see if that fixes but this just got a lot more complicated I think | 16:48 |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/project-config master: Setup translations for Pike https://review.openstack.org/493080 | 16:48 |
| odyssey4me | clarkb :( | 16:48 |
| odyssey4me | in the interests of making all neutron use-cases more equal, I support your efforts! :) | 16:49 |
| fungi | mgagne: judging from http://codesearch.openstack.org/?q=internap there are probably also some occurrences in the shade library, on the www.openstack.org site, the planet.openstack.org blog aggregator... | 16:49 |
| mgagne | we have a blog? omg | 16:49 |
| odyssey4me | anyway, I'm out for the w/end - have a fantastic day and even better w/end all! | 16:49 |
| clarkb | odyssey4me: ya I think use of eg ovs shouldn't assume an ovs bridge but some device it must add to the ovs bridge it iwll manage | 16:49 |
| clarkb | but getting that changed everywhere... thats another story | 16:49 |
| clarkb | odyssey4me: enjoy | 16:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename allow-secrets to post-review https://review.openstack.org/493059 | 16:50 |
| fungi | mgagne: plus, lots of projects have references to contributors with internap.com e-mail addresses | 16:51 |
| * fungi wonders why they didn't prefer .net but whatever | 16:51 | |
| *** kjackal_ has quit IRC | 16:53 | |
| *** sshnaidm|off has joined #openstack-infra | 16:55 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Hack in linux bridge vxlan support https://review.openstack.org/492654 | 16:56 |
| mgagne | fungi: yes, that's part of the "rename" we have to do | 16:56 |
| mgagne | ¯\_(ツ)_/¯ | 16:56 |
| fungi | i applaud your artistic use of the "tu" katakana | 16:58 |
| *** jogo has joined #openstack-infra | 16:59 | |
| *** markvoelker has quit IRC | 16:59 | |
| mgagne | :P | 16:59 |
| *** markus_z has quit IRC | 17:00 | |
| *** rlandy|brb is now known as rlandy | 17:00 | |
| *** Apoorva_ has joined #openstack-infra | 17:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Add node.cloud to zookeeper https://review.openstack.org/493086 | 17:01 |
| fungi | (or "tsu" depending on which romanization you prefer) | 17:01 |
| *** e0ne has joined #openstack-infra | 17:02 | |
| *** mdrabe is now known as mdrabe_afk | 17:02 | |
| pabelanger | clarkb: odyssey4me: I too approve :) | 17:03 |
| pabelanger | clarkb: mind looking at https://review.openstack.org/493028 sets up .gemrc with reverse proxy | 17:03 |
| clarkb | pabelanger: ya that said likely the simplest fix right now is to just get rdo added in d-g if we need it (are on centos) | 17:04 |
| *** Apoorva has quit IRC | 17:04 | |
| clarkb | since the tendrils of this ovs bridge seem to reach far | 17:04 |
| *** e0ne has quit IRC | 17:05 | |
| pabelanger | clarkb: Ya, agree. I can look into adding logic to test that | 17:06 |
| *** e0ne has joined #openstack-infra | 17:07 | |
| *** psachin has quit IRC | 17:08 | |
| *** e0ne_ has joined #openstack-infra | 17:09 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create nodepool.cloud inventory variable https://review.openstack.org/493088 | 17:12 |
| clarkb | pabelanger: for ^ you probably want to update ovs_vxlan_bridge in functions.sh to add the yum commands to add a repo if running on centos. This all gets shoved into a variable because we run it on >1 host | 17:12 |
| *** e0ne has quit IRC | 17:12 | |
| openstackgerrit | Ryan proposed openstack-infra/bindep master: Add ability to list all deps https://review.openstack.org/492693 | 17:13 |
| pabelanger | clarkb: ack | 17:13 |
| *** felipemonteiro_ has quit IRC | 17:14 | |
| rybridges | New changes for my bindep review https://review.openstack.org/#/c/492693/ | 17:15 |
| rybridges | You can now specify whether to print one dep per line or csv format | 17:15 |
| *** persia has quit IRC | 17:18 | |
| *** tnovacik has joined #openstack-infra | 17:19 | |
| *** rbrndt has quit IRC | 17:22 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: remove v2 grenade job from master https://review.openstack.org/493091 | 17:23 |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config master: Increase timeouts for Manila DSVM jobs https://review.openstack.org/493092 | 17:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Add support for NODEPOOL_RUGYGEMS_PROXY https://review.openstack.org/493028 | 17:26 |
| *** electrofelix has quit IRC | 17:30 | |
| *** funzo has joined #openstack-infra | 17:32 | |
| *** annegentle has quit IRC | 17:33 | |
| *** simondodsley has joined #openstack-infra | 17:33 | |
| *** e0ne_ has quit IRC | 17:35 | |
| *** funzo has quit IRC | 17:36 | |
| EmilienM | pabelanger: thanks a lot for the rubygems proxy thing | 17:38 |
| *** Swami has joined #openstack-infra | 17:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename allow-secrets to post-review https://review.openstack.org/493059 | 17:40 |
| *** e0ne has joined #openstack-infra | 17:42 | |
| *** yamamoto has quit IRC | 17:43 | |
| *** e0ne has quit IRC | 17:43 | |
| dhellmann | mtreinish, AJaeger_ : I'm working on pike job setup patches and looking for how to reproduce adding periodic stable jobs for tempest. The reference I have is https://review.openstack.org/#/c/375271/1 but the periodic-stable pipeline referenced there is no longer present in layout.yaml. What's the new approach for that? | 17:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Zuulv3: remove infra-publish-jobs from zuul repos https://review.openstack.org/493097 | 17:44 |
| *** ralonsoh has quit IRC | 17:45 | |
| dhellmann | oh, nevermind, it's just moved in a weird way, I see it | 17:45 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add stable-compat jobs for tempest and devstack-gate for pike https://review.openstack.org/493098 | 17:48 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add pike bitrot jobs https://review.openstack.org/493099 | 17:48 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add periodic pike jobs for tempest https://review.openstack.org/493100 | 17:48 |
| *** trown|lunch is now known as trown | 17:51 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add stable-compat jobs for tempest and devstack-gate for pike https://review.openstack.org/493098 | 17:52 |
| pabelanger | EmilienM: np, hopefully we can land the change today | 17:53 |
| pabelanger | EmilienM: puppet-openstack-integration would need a minor patch, since you use root | 17:53 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add stable-compat jobs for tempest and devstack-gate for pike https://review.openstack.org/493098 | 17:53 |
| *** yamahata has quit IRC | 17:53 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add pike bitrot jobs https://review.openstack.org/493099 | 17:53 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add stable-compat jobs for tempest and devstack-gate for pike https://review.openstack.org/493098 | 17:53 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add periodic pike jobs for tempest https://review.openstack.org/493100 | 17:53 |
| pabelanger | clarkb: fungi /dev/mapper/main-proxycache 103077152 103060732 36 100% /var/cache/apache2 | 17:55 |
| pabelanger | that is mirror.ord.rax.openstack.org | 17:55 |
| *** markvoelker has joined #openstack-infra | 17:55 | |
| pabelanger | http://logs.openstack.org/63/491463/6/gate/gate-tripleo-ci-centos-7-scenario003-multinode-oooq-puppet/f7f033f/logs/undercloud/home/jenkins/install_packages.sh.log.txt.gz | 17:56 |
| clarkb | ya ianw pointed that out yesterday | 17:57 |
| fungi | pabelanger: so should we drop the htcacheclean down lower than 80gb or run it more often, or something else? | 17:58 |
| clarkb | pabelanger: other rax mirrors are fine, I think we want to see how old some of the cache contents are | 17:58 |
| clarkb | fungi: possibly both if we haven't leaked cache contents | 17:58 |
| johnsom | Hi folks, were just talking about some release notes jobs failing in the releases channel: | 17:58 |
| johnsom | http://logs.openstack.org/60/491560/3/check/gate-releases-docs-ubuntu-xenial/0a1411d/console.html#_2017-08-11_08_24_08_395666 | 17:58 |
| johnsom | http://logs.openstack.org/69/690ccfd43fb141c26652e119f1d702b65414a194/post/octavia-releasenotes/238e6ab/console.html#_2017-08-11_17_50_10_439343 | 17:59 |
| johnsom | THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. | 17:59 |
| fungi | i worry that filesystem performance could be causing us to not be able to delete the cache faster than we accumulate (which was a concern of mine from the beginning when i realized we needed an outside command to expire cache contents) | 17:59 |
| pabelanger | clarkb: ya, looking into that now | 17:59 |
| johnsom | Both using RAX hosts | 17:59 |
| pabelanger | fungi: I think that might be the case | 17:59 |
| *** bobh has joined #openstack-infra | 17:59 | |
| pabelanger | johnsom: see http://status.openstack.org/elastic-recheck/#1708707 know issue we are working to fix | 18:00 |
| johnsom | Cool, thanks | 18:00 |
| *** camunoz has quit IRC | 18:01 | |
| pabelanger | fungi: I think you might be onto something. htcacheclean was running and flock was from | 18:01 |
| pabelanger | root 21389 0.0 0.0 4452 560 ? Ss 13:00 0:00 /bin/sh -c flock -n /var/run/htcacheclean.lock htcacheclean -n -p /var/cache/apache2/proxy -t -l 81920M > /dev/null | 18:01 |
| pabelanger | so, pushing 5 hours to do 80GB | 18:01 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 18:01 |
| clarkb | pabelanger: johnsom its also potentially related to the cache fs filling as apache may be serving subsets of data | 18:02 |
| pabelanger | fungi: I restarted the flock, and seem to be getting 600K/s reads on the mount | 18:02 |
| *** Apoorva_ has quit IRC | 18:03 | |
| pabelanger | clarkb: are we doing pypi via reverse proxy cache? | 18:03 |
| clarkb | pabelanger: oh not any longer good point | 18:03 |
| clarkb | pabelanger: fungi is this mounted on a cinder volume or the epheneral disk? | 18:03 |
| *** Apoorva has joined #openstack-infra | 18:03 | |
| clarkb | also we have a lot of data from july | 18:04 |
| clarkb | so we are leaking | 18:04 |
| pabelanger | both afs and proxy are AFS | 18:04 |
| fungi | lvm on cinder | 18:04 |
| pabelanger | clarkb: how did know that? | 18:05 |
| fungi | ephemeral disk (/dev/xvde2) is mounted at /opt and not very large | 18:05 |
| *** annegentle has joined #openstack-infra | 18:06 | |
| pabelanger | clarkb: ls on /var/cache/apache2/proxy I can see data from july | 18:06 |
| clarkb | pabelanger: http://paste.openstack.org/show/618219/ ya I just started digging into dirs with old timestamps | 18:06 |
| clarkb | so I think that means our cinder volume is eslow? | 18:06 |
| fungi | looks like we get a 42.9 GB ephemeral disk there, and it's carved into 2gb swap and ~36gb /opt | 18:06 |
| pabelanger | k | 18:06 |
| fungi | (i know that math doesn't add up, comparing fdisk to df) | 18:06 |
| pabelanger | maybe ord is slower then dfw for cinder for some reason | 18:07 |
| *** thingee_ has quit IRC | 18:07 | |
| fungi | or could just be this particular hypervisor host or the device our cinder volume landed on in ord is slow | 18:07 |
| clarkb | we might try just replacing the volume to start | 18:08 |
| pabelanger | we can do that without shutdown right? | 18:09 |
| fungi | not completely | 18:09 |
| fungi | oh, actually yes we can | 18:10 |
| jeblair | i feel like i'm missing something -- what do we suspect are contributing factors to pip hash mismatch? | 18:10 |
| fungi | but it would probably take a while | 18:10 |
| clarkb | jeblair: we don't pabelanger corrected me that we stoped proxying pypi | 18:10 |
| fungi | jeblair: apache not being able to cache files because its cache filesystem is full | 18:10 |
| clarkb | jeblair: so its all afs (and was only proxied with d-g) | 18:10 |
| pabelanger | htcacheclean is has been running for 4 hours on dfw, but only at 67% capacity | 18:10 |
| clarkb | pabelanger: there is old stuff on dfw too though | 18:10 |
| fungi | but yes, that was before we realized it's not the apache cache that's going through any longer | 18:10 |
| clarkb | pabelanger: oh right because its under the size limit | 18:11 |
| clarkb | I think htcacheclean may just mostly leave things alone if under the size limit | 18:11 |
| jeblair | okay, so basically we have 2 problems here -- a slow/full cache which is affecting some stuff, and the pip hash mismatch. currently, no suspected overlap between the problems. | 18:11 |
| clarkb | jeblair: right | 18:11 |
| fungi | yeah | 18:11 |
| pabelanger | agree | 18:11 |
| jeblair | did someone say they suspected pip hash mismatches may be due to apache segfaults? | 18:12 |
| *** e0ne has joined #openstack-infra | 18:12 | |
| *** e0ne has quit IRC | 18:12 | |
| clarkb | jeblair: yes ianw has been looking into it and seemed to think there was correlation there | 18:12 |
| clarkb | there are details on the bug, one sec while I get that pulled up | 18:13 |
| fungi | dmesg recorded a "[Fri Aug 11 17:46:50 2017] afs: disk cache read error in CacheItems slot 1540722 off 123257780/125000020 code -4/80" | 18:13 |
| clarkb | http://bugs.launchpad.net/bugs/1708707 | 18:13 |
| openstack | Launchpad bug 1708707 in OpenStack-Gate "Pip finds hash mismatch for package during installation" [Undecided,New] | 18:13 |
| pabelanger | so, rax-ord is our largest region right now (195), so it is likely that 80GB htcacheclean on 100GB disk, is just cutting it too close. Maybe we reduce htcacheclean by 10GB to start | 18:14 |
| jeblair | fungi: that seems a few minutes too early | 18:14 |
| fungi | yeah | 18:14 |
| fungi | there's also this in the lastlog: reboot system boot Fri Aug 11 04:56 - 18:14 (13:17) 0.0.0.0 | 18:14 |
| clarkb | pabelanger: ya I think we want to increase the delta between max disk and max cache size | 18:15 |
| clarkb | pabelanger: that way htcacheclean has more time to delete things once we go over that limit | 18:15 |
| jeblair | fungi: neat | 18:15 |
| clarkb | its basically a race between adding contents to the cache and deleting them once we go over the cache limit | 18:15 |
| clarkb | fungi: ya ianw did that | 18:15 |
| *** esberglu has quit IRC | 18:16 | |
| clarkb | (its in sb, not sure if in bug) | 18:16 |
| fungi | oh, okay, i missed that. thanks | 18:16 |
| *** esberglu has joined #openstack-infra | 18:16 | |
| *** sdague has quit IRC | 18:17 | |
| *** sdague has joined #openstack-infra | 18:17 | |
| jeblair | wow i miss zuulv3 debug info already | 18:17 |
| *** Guest8912 has quit IRC | 18:17 | |
| fungi | based on the early numbers i ran when figuring out these sizes, limiting it to 50gb would have been safe i just figured 80gb gave us more headroom for future growth. didn't realize at the time that htcacheclean could have trouble keeping up between 80gb and teh 100gb hard limit there | 18:18 |
| fungi | so i'm in favor of dropping it to 50gb for now and seeing whether that helps matters | 18:19 |
| clarkb | ++ | 18:19 |
| jeblair | mirror.ord.rax.openstack.org_access.log:2001:4801:7828:101:be76:4eff:fe10:6e6a - - [11/Aug/2017:17:50:10 +0000] "GET /wheel/ubuntu-16.04-x86_64/pyperclip/pyperclip-1.5.27-py2-none-any.whl HTTP/1.1" 200 7463 "-" "pip/9.0.1 | 18:19 |
| fungi | i'll put that change together real quick | 18:19 |
| clarkb | though we may need to stop in rax-ord temporarily to get it cleaned up | 18:19 |
| jeblair | there's the apache log line for that error | 18:19 |
| clarkb | but step 0 is change the limit | 18:19 |
| clarkb | jeblair: 7463 seems small for a whl | 18:20 |
| *** rbrndt has joined #openstack-infra | 18:20 | |
| *** bobh has quit IRC | 18:20 | |
| clarkb | truncated GET maybe? | 18:20 |
| fungi | clarkb: yeah, may make the most sense to drop nodepool max-servers to 0 there in the near term too while it catches up | 18:20 |
| clarkb | fungi: ya we can also use htcacheclean -r if we stop apache | 18:20 |
| clarkb | which is "clean thoroughly" but requiers apache be stopped | 18:20 |
| jeblair | -rw-rw-r-- 1 10003 3000 7203 Jul 4 06:52 /afs/openstack.org/mirror/wheel/ubuntu-16.04-x86_64/p/pyperclip/pyperclip-1.5.27-py2-none-any.whl | 18:20 |
| fungi | still worried though that if htcacheclean can't remove files faster than we add them then no amount of difference between the clean size and teh filesystem capacity is going to help | 18:21 |
| *** esberglu has quit IRC | 18:21 | |
| clarkb | fungi: ya | 18:21 |
| jeblair | what are we caching? | 18:22 |
| jeblair | (so far, our caches have been *tiny* and they barely move) | 18:22 |
| clarkb | jeblair: docker images is hte big thing | 18:22 |
| fungi | rpms, debs, npm, images, soon gems | 18:22 |
| clarkb | images being the only thing I expect to really cause this trouble | 18:22 |
| fungi | i'm not so sure | 18:23 |
| *** niska has quit IRC | 18:23 | |
| fungi | images are large, but that means they're fast to clean from the cache, right? | 18:23 |
| jeblair | i think i'm most concerned by the churn | 18:23 |
| fungi | it's massive numbers of tiny files which are going to cause htcacheclean headaches | 18:23 |
| *** ilpianista_ has quit IRC | 18:23 | |
| *** masayukig[m] has quit IRC | 18:23 | |
| clarkb | fungi: maybe? I'm not sure if it shards them or not | 18:23 |
| fungi | i would expect | 18:23 |
| fungi | oh, perhaps | 18:23 |
| jeblair | basically, if we have continuous turnover, we have a (much?) less effective cache | 18:24 |
| fungi | jeblair: well, i don't think it's continuous turnover. clarkb noted files with dates from july | 18:25 |
| *** yamahata has joined #openstack-infra | 18:25 | |
| *** priteau has joined #openstack-infra | 18:25 | |
| *** gmann has quit IRC | 18:25 | |
| clarkb | hrm http://paste.openstack.org/show/618221/ likely not to help | 18:25 |
| dims | johnsom : found any clues yet? | 18:26 |
| jeblair | oh i think i was looking at the wrong pip log line. it errored on blockdiag not pyperclip | 18:27 |
| fungi | yeah, /var/cache/apache2 is "only" at 34% inodes used, but if a vast majority of them are in a single directory... | 18:27 |
| jeblair | mirror.ord.rax.openstack.org_access.log:2001:4801:7828:101:be76:4eff:fe10:6e6a - - [11/Aug/2017:12:29:35 +0000] "GET /pypi/packages/1c/17/653639940e2a4f20e42af1738ec3350e972a11d0209ecd65c6034876cf60/blockdiag-1.5.3-py2.py3-none-any.whl HTTP/1.1" 200 2722535 "-" "pip/9.0.1 | 18:27 |
| johnsom | dims Above they acknowledged the issue and are working on it. | 18:27 |
| dims | thanks johnsom | 18:27 |
| jeblair | -rw-r--r-- 1 10001 root 2722270 Jul 27 2016 /afs/openstack.org/mirror/pypi/web/packages/1c/17/653639940e2a4f20e42af1738ec3350e972a11d0209ecd65c6034876cf60/blockdiag-1.5.3-py2.py3-none-any.whl | 18:27 |
| dims | johnsom : y all hits i saw were on rax | 18:27 |
| johnsom | Yep | 18:28 |
| fungi | i wonder if there's a way to tell mod_proxy to shard those files into subtrees, and whether doing that would improve htcacheclean's performance | 18:28 |
| clarkb | fungi: ya I think you can control how it shards those dirs, trying to find docs on it now | 18:28 |
| *** masayukig[m] has joined #openstack-infra | 18:29 | |
| *** markvoelker has quit IRC | 18:29 | |
| jeblair | fungi: why might it improve performance? (or is this a shot in the dark?) | 18:29 |
| clarkb | fungi: https://httpd.apache.org/docs/2.4/mod/mod_cache_disk.html#cachedirlength | 18:29 |
| clarkb | jeblair: stating large dirs is hard on fs iirc | 18:29 |
| fungi | i haven't looked into htcacheclean's implementation, but if it's operating on an index of that directory it could really get bogged down | 18:30 |
| *** baoli has quit IRC | 18:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Reduce htcleancache to 70GB https://review.openstack.org/493110 | 18:30 |
| jeblair | pabelanger: i thought 50, not 70? | 18:30 |
| clarkb | https://httpd.apache.org/docs/2.4/mod/mod_cache.html we are using roughly the sample there I think | 18:30 |
| fungi | jeblair: so mostly a stab in the dark | 18:30 |
| pabelanger | jeblair: Oh, I missed that | 18:31 |
| pabelanger | let me update | 18:31 |
| *** baoli has joined #openstack-infra | 18:31 | |
| openstackgerrit | Merged openstack/os-client-config master: Update the documentation link for doc migration https://review.openstack.org/491374 | 18:31 |
| fungi | pabelanger: yeah, let's do 50. i was in th emiddle of writing the change but keep getting sidetracked gabbing | 18:31 |
| fungi | thanks! | 18:31 |
| clarkb | so maybe we want to go to the default dir length of 2 and do 8 dir levels for roughly equivalent cross product | 18:31 |
| clarkb | I worry that we'll have to completely wipe out the old cache if we do that though (which maybe is fine just rm anything with longer name) | 18:32 |
| *** niska has joined #openstack-infra | 18:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Reduce htcacheclean to 50GB https://review.openstack.org/493110 | 18:32 |
| pabelanger | fungi: np, I missed backscroll | 18:32 |
| jeblair | so the blockdiag transfer was short -- it's shorter than the size on disk. but it still returned 200 and i don't see any segfaults right around that time. | 18:32 |
| *** GregHouse has quit IRC | 18:32 | |
| *** funzo has joined #openstack-infra | 18:32 | |
| *** Hal has joined #openstack-infra | 18:33 | |
| *** Hal is now known as Guest74276 | 18:33 | |
| jeblair | oh wait i mathed wrong | 18:34 |
| jeblair | apache sent more data than what was on disk, so it looks entirely normal from the server side | 18:34 |
| fungi | so possible for the client to have stopped reading, maybe? | 18:35 |
| jeblair | i suspect that was an aborted transfer on the client side, or network issue (rst?) | 18:35 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Make apache cache structure deeper https://review.openstack.org/493112 | 18:36 |
| pabelanger | okay, I have screen running on mirror.rax.ord using htcacheclean with 50GB | 18:36 |
| clarkb | there is the idea for reorganizing the cache structure. I expect that we'll have to completely delete the existing cache when we put that in place | 18:37 |
| *** sshnaidm|off has quit IRC | 18:37 | |
| fungi | hits over the past 10 days for the pip hash mismatch logstash query are basically all from rax-ord | 18:37 |
| *** funzo has quit IRC | 18:37 | |
| *** GregHouse has joined #openstack-infra | 18:37 | |
| fungi | network traffic on that mirror is topping out at 200mbps outbound | 18:38 |
| fungi | i wonder if we need to rebuild it with a bigger flavor? | 18:39 |
| pabelanger | oh, maybe | 18:39 |
| jeblair | fungi: that could do it | 18:40 |
| fungi | yeah, would explain short reads if their packet shaping layer is dropping stuff | 18:40 |
| fungi | i'll start launching a replacement | 18:41 |
| jeblair | at 17:50 we were "only" at 100mbit outbound | 18:41 |
| *** ilpianista_ has joined #openstack-infra | 18:42 | |
| jeblair | (of course that's a 5m average, there could have been a spike) | 18:42 |
| fungi | yeah, whether or not this solves the pip hash mismatches, we're gonna need a bigger boat anyway | 18:42 |
| *** Sukhdev has joined #openstack-infra | 18:42 | |
| *** nicolasbock has quit IRC | 18:43 | |
| jeblair | noticable increase starting in july | 18:43 |
| *** yamamoto has joined #openstack-infra | 18:44 | |
| fungi | yeah, same for the iowait on the cpu graph | 18:44 |
| fungi | probably coincides with when we added the tarballs site cache and kolla started using it to get cached copies of their nightly images from there | 18:45 |
| jeblair | it's currently a 4g server; i don't recall the limits, but 200mbit seems plausible | 18:45 |
| pabelanger | we did start caching tarballs.o.o in july with reverse proxy that could account for increase traffic | 18:46 |
| jeblair | johnsom, dims, fungi: updated https://bugs.launchpad.net/openstack-gate/+bug/1708707 | 18:46 |
| openstack | Launchpad bug 1708707 in OpenStack-Gate "Pip finds hash mismatch for package during installation" [Undecided,New] | 18:46 |
| fungi | yeah, that pretty closely coincides with when i added the cinder volume for the apache proxy: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64027&rra_id=all | 18:46 |
| fungi | which was so that we could start caching the tarballs site | 18:47 |
| pabelanger | we also had multiple htcacheclean processes running, causing load (and assuming IO) to climb | 18:47 |
| clarkb | ya they all wouldve been stating things | 18:48 |
| fungi | so, 8gb performance seem reasonable? | 18:48 |
| clarkb | fungi: I think so | 18:48 |
| *** yamamoto has quit IRC | 18:49 | |
| pabelanger | wfm | 18:49 |
| pabelanger | fungi: clarkb: the 50% reduction for infracloud was approved the other day, I didn't WIP it. With that is mind, did we want to start image uploads again? | 18:51 |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Remove OpenStack/Ceph/Virt repo from CentOS https://review.openstack.org/493003 | 18:51 |
| fungi | probably so | 18:51 |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Remove UUID token from Patrole jobs https://review.openstack.org/493118 | 18:51 |
| openstackgerrit | Felipe Monteiro proposed openstack-infra/project-config master: Remove UUID token from Patrole jobs https://review.openstack.org/493118 | 18:52 |
| jeblair | pabelanger: yes, i think infracloud has been doing much better. enabling uploads and keeping an eye on things would be good i think. | 18:54 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Temporarily stop using rax-ord https://review.openstack.org/493119 | 18:56 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Revert "Temporarily stop using rax-ord" https://review.openstack.org/493120 | 18:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Pause image uploads in infracloud (both regions)" https://review.openstack.org/493121 | 18:56 |
| pabelanger | fungi: jeblair: ^ | 18:56 |
| fungi | let's get 493119 merged while i've got the replacement mirror building, and once the swap is complete we can approve the revert | 18:57 |
| pabelanger | +2 | 18:57 |
| *** mdrabe_afk is now known as mdrabe | 18:58 | |
| *** jamesmcarthur has joined #openstack-infra | 18:58 | |
| clarkb | and approved | 19:00 |
| *** annegentle has quit IRC | 19:00 | |
| *** sekelso has joined #openstack-infra | 19:05 | |
| *** skelso has quit IRC | 19:07 | |
| *** jamesmcarthur has quit IRC | 19:07 | |
| *** jamesmcarthur has joined #openstack-infra | 19:08 | |
| fungi | okay, replacement mirror server is built and in reverse dns. once nodes in use there reach 0 i'll swap a/aaaa dns entries, stop apache and openafs, move the cinder volumes, delete the old server, reboot the new server, clear the ansible inventory cache, and update the host key cached on the puppetmaster, then un-wip 493120 | 19:08 |
| fungi | anybody think of any steps i'm missing there? | 19:09 |
| clarkb | fungi: would it be easier to just make a new cinder volume and reboot now? | 19:10 |
| fungi | not especially | 19:10 |
| clarkb | ok | 19:10 |
| fungi | just more stuff i have to delete later | 19:10 |
| clarkb | actually if you use new volume now and reboot you could in theory juts update dns and things will shift over on their own? | 19:10 |
| clarkb | either way works | 19:11 |
| *** mat128 has quit IRC | 19:11 | |
| fungi | and a few extra minutes of downtime aren't a big deal when we already have the region down in nodepool | 19:11 |
| clarkb | ya | 19:11 |
| fungi | also we lose the old caches that way | 19:11 |
| pabelanger | clarkb: want to +3 493121? image uploads again for infracloud | 19:11 |
| pabelanger | fungi: nice | 19:12 |
| fungi | (if we create new volumes i mean) | 19:12 |
| *** EricGonczer_ has joined #openstack-infra | 19:12 | |
| clarkb | pabelanger: done | 19:12 |
| pabelanger | ya, maybe just new cinder volumes | 19:13 |
| pabelanger | either way works for me | 19:13 |
| fungi | i'd rather not unnecessarily wipe the afs cache there if we can help it, and so if i have to move one volume i might as well move both. losing the apache cache is less of a concern if we're going forward with 493112 but still that means we end up re-priming the apache cache twice here | 19:13 |
| clarkb | fungi: yup I had only considered the apache cache when first proposing the idea, definitely makes sense to keep the afs cache | 19:14 |
| clarkb | I am going to work on lunch. Be back in a bit | 19:14 |
| pabelanger | ya, wiping apache cache would be fine. It is full ATM and going to take a fair bit of time for htcacheclean to purge | 19:14 |
| fungi | pabelanger: luckily we have a fair bit of time while waiting for nodes to dwindle there | 19:15 |
| openstackgerrit | Major Hayden proposed openstack-infra/system-config master: Add yum.mariadb.org reverse proxy https://review.openstack.org/493204 | 19:15 |
| fungi | i've deleted the remaining ready nodes in rax-ord | 19:17 |
| *** annegentle has joined #openstack-infra | 19:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Temporarily stop using rax-ord https://review.openstack.org/493119 | 19:17 |
| fungi | and manually applied that ^ | 19:18 |
| openstackgerrit | Major Hayden proposed openstack-infra/project-config master: Add proxy host for MariaDB yum mirror https://review.openstack.org/493205 | 19:18 |
| fungi | oh, before i forget... | 19:19 |
| fungi | #status log The apps.openstack.org server has been stopped, snapshotted one last time, and deleted. | 19:20 |
| openstackstatus | fungi: finished logging | 19:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Pause image uploads in infracloud (both regions)" https://review.openstack.org/493121 | 19:21 |
| fungi | we've still got right at 100 nodes in use in ord, so it'll take a few hours for those to finish winding down | 19:21 |
| fungi | but no more in building/ready | 19:22 |
| *** markvoelker has joined #openstack-infra | 19:27 | |
| *** jamesmcarthur has quit IRC | 19:28 | |
| *** EricGonc_ has joined #openstack-infra | 19:28 | |
| *** EricGonczer_ has quit IRC | 19:29 | |
| *** jamesmcarthur has joined #openstack-infra | 19:30 | |
| *** vhosakot has quit IRC | 19:31 | |
| *** vhosakot has joined #openstack-infra | 19:32 | |
| *** Apoorva has quit IRC | 19:32 | |
| *** funzo has joined #openstack-infra | 19:33 | |
| *** jcoufal has quit IRC | 19:33 | |
| *** funzo has quit IRC | 19:38 | |
| *** adisky__ has quit IRC | 19:39 | |
| *** baoli has quit IRC | 19:41 | |
| *** fried_rice has quit IRC | 19:41 | |
| *** baoli has joined #openstack-infra | 19:41 | |
| *** baoli has quit IRC | 19:42 | |
| *** vhosakot has quit IRC | 19:43 | |
| *** baoli has joined #openstack-infra | 19:43 | |
| *** vhosakot has joined #openstack-infra | 19:44 | |
| *** jamesmcarthur has quit IRC | 19:45 | |
| *** yamamoto has joined #openstack-infra | 19:46 | |
| *** jkilpatr has quit IRC | 19:49 | |
| sdague | http://logs.openstack.org/42/473842/2/gate/gate-grenade-dsvm-neutron-ubuntu-xenial/fe28786/console.html#_2017-08-11_19_05_48_073877 | 19:53 |
| sdague | no hosts matched, nothing to do? | 19:53 |
| sdague | that seemed to fail really really early | 19:53 |
| *** fried_rice has joined #openstack-infra | 19:53 | |
| pabelanger | thats from ansible | 19:54 |
| sdague | yeh, but we never ran anything | 19:54 |
| pabelanger | setup_host playbook looks to have failed | 19:54 |
| sdague | ok | 19:54 |
| pabelanger | http://logs.openstack.org/42/473842/2/gate/gate-grenade-dsvm-neutron-ubuntu-xenial/fe28786/logs/devstack-gate-setup-host.txt | 19:54 |
| *** sekelso has quit IRC | 19:54 | |
| sdague | rax-ord it seems | 19:55 |
| *** sekelso has joined #openstack-infra | 19:55 | |
| sdague | I was just looking at various gate fails in progress to see what's going one | 19:55 |
| sdague | going on | 19:55 |
| *** yamamoto has quit IRC | 19:55 | |
| sdague | there is at least one requirements job the blew up in the gate with mirror checksum issues | 19:56 |
| pabelanger | ya, fungi is working on rolling out a new mirror in rax-ord. Increasing capacity | 19:57 |
| *** rwsu has quit IRC | 19:57 | |
| fungi | it's possible that one was my fault. i deleted ready nodes while we were winding down utilization there and it's possible my delete loop raced with a job picking up a ready node | 19:57 |
| sdague | ok, cool | 19:57 |
| fungi | like, if it transitioned from ready to used in the two or three seconds between when my one-liner built the list of nodes still in a ready state and marked them for deletion | 19:58 |
| *** markvoelker has quit IRC | 19:59 | |
| fungi | speaking of which, we're under 40 still in use there, so hopefully shouldn't be much longer before i can swap mirrors | 19:59 |
| *** masayukig[m] has quit IRC | 20:01 | |
| *** ilpianista_ has quit IRC | 20:01 | |
| *** sshnaidm|off has joined #openstack-infra | 20:02 | |
| *** ihrachys has quit IRC | 20:02 | |
| *** masayukig[m] has joined #openstack-infra | 20:03 | |
| *** mudpuppy has quit IRC | 20:04 | |
| jeblair | maybe i should go ahead and spin up the inap mirror | 20:05 |
| *** mudpuppy has joined #openstack-infra | 20:06 | |
| *** priteau has quit IRC | 20:06 | |
| jeblair | fungi: should i build it with xenial? | 20:11 |
| *** jamesmcarthur has joined #openstack-infra | 20:12 | |
| fungi | jeblair: not a bad idea... i haven't tried doing one on xenial yet | 20:13 |
| fungi | our default with the launch script seems to still be trusty | 20:13 |
| *** gouthamr has quit IRC | 20:14 | |
| jeblair | giving it a shot | 20:14 |
| fungi | assuming it works (i can't imagine why it wouldn't) maybe it's time we updated that default | 20:15 |
| *** slagle has quit IRC | 20:15 | |
| clarkb | pabelanger: where do we confiure the rdo repo in our image builds? | 20:15 |
| pabelanger | clarkb: https://review.openstack.org/#/c/493003/ | 20:15 |
| pabelanger | it is infra-package-needs now | 20:15 |
| pabelanger | was install_puppet.sh | 20:15 |
| clarkb | pabelanger: oh so we yum install that package and it adds the repo? | 20:16 |
| pabelanger | ya | 20:16 |
| pabelanger | create .repo file on disk, enabled by default | 20:16 |
| *** Apoorva has joined #openstack-infra | 20:16 | |
| *** ilpianista_ has joined #openstack-infra | 20:17 | |
| jeblair | | fault | {u'message': u'No valid host was found. There are not enough hosts available.', u'code': 500, u'created': u'2017-08-11T20:14:08Z'} | | 20:17 |
| jeblair | :( | 20:17 |
| pabelanger | so devstack-gate would install centos-release-openstack-ocata, disable repo, then yum install openswitch --enable-repo=openstack-rdo-ocata (untested) | 20:18 |
| jeblair | that was with an A flavor, it's going better with a B flavor now | 20:20 |
| clarkb | pabelanger: I don't like that it is pinned to the openstack release... especially if we have to special case it in places | 20:23 |
| pabelanger | clarkb: ya, been like that for a while with RDO | 20:23 |
| clarkb | means we'll have to update $repo every release just to get ovs | 20:23 |
| pabelanger | ya | 20:24 |
| clarkb | pabelanger: I thought there was a rdo/latest or whatever that you could point at? | 20:24 |
| pabelanger | there is, but that would be like master releases | 20:24 |
| clarkb | but for ovs I'm guessing its fairly stable | 20:24 |
| pabelanger | https://trunk.rdoproject.org/ | 20:24 |
| clarkb | since we arn't actually installing openstack just ovs | 20:24 |
| fungi | jeblair: which was the flavor we had for the previous mirror/. | 20:25 |
| fungi | ? | 20:25 |
| pabelanger | clarkb: a while back I asked RDO to move ovs into its own repo, they didn't object | 20:25 |
| pabelanger | clarkb: I can push on that again | 20:25 |
| pabelanger | maybe at PTG | 20:25 |
| *** jkilpatr has joined #openstack-infra | 20:25 | |
| openstackgerrit | Merged openstack-infra/system-config master: Reduce htcacheclean to 50GB https://review.openstack.org/493110 | 20:28 |
| jeblair | fungi: B is previous flavor | 20:29 |
| clarkb | pabelanger: fungi it almost looks like htcacheclean tries to build a complete list of things to delete then do them all at once rather than going incrementally | 20:29 |
| jeblair | | Aug 11 20:26:13 mirror puppet-agent[4067]: Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known) | 20:30 |
| clarkb | it might be advantageous to run it in daemon mode for that reason | 20:30 |
| pabelanger | clarkb: yes, I am thinking the same thing | 20:30 |
| jeblair | fungi: that's the last log line i have from launch-node, and it's sitting there -- did you see anything like that? | 20:30 |
| pabelanger | clarkb: Ya, I can test daemon mode now and see if there is a difference | 20:31 |
| clarkb | pabelanger: we probably won't see it immediately but once it loads up all the state (however long that takes) it won't have to rebuild it again just work on deltas (of course it may rebuild it all anyways) | 20:31 |
| jeblair | fungi: oh hey it started moving again | 20:31 |
| jeblair | wow 5 mins later | 20:31 |
| pabelanger | clarkb: Ya, delta would be faster, for sure | 20:32 |
| fungi | jeblair: looks familiar, but since stdout gets spammed with the entire ansible inventory at one point, it's scrolled off the top of my 2k line terminal history | 20:32 |
| *** tnovacik has quit IRC | 20:32 | |
| *** gouthamr has joined #openstack-infra | 20:32 | |
| *** funzo has joined #openstack-infra | 20:34 | |
| *** ccamacho has quit IRC | 20:35 | |
| clarkb | there are also claims on the internets that you can just run a find for items +mtime $value and delete old ones | 20:38 |
| *** funzo has quit IRC | 20:38 | |
| *** jamesmcarthur has quit IRC | 20:38 | |
| *** weshay_doc is now known as weshay | 20:39 | |
| jeblair | #status log created mirror.mtl01.inap.openstack.org to replace mirror.mtl01.internap.openstack.org (internap -> inap rename) | 20:39 |
| openstackstatus | jeblair: finished logging | 20:39 |
| jeblair | now i probably need to add some volumes | 20:40 |
| jeblair | pabelanger: oh i just realized -- we *don't* need a new internap mirror unless/until we change the cloud name | 20:42 |
| *** spzala has quit IRC | 20:42 | |
| clarkb | also we should apparently mount the fs noatime | 20:42 |
| clarkb | fungi: ^ | 20:42 |
| jeblair | because we build the hostname from the cloud name, not the provider | 20:42 |
| jeblair | clarkb: don't we do that for all our filesystems? | 20:42 |
| jeblair | wow, apparently not | 20:42 |
| jeblair | "Be sure to add it to /etc/fstab." is all we say on the subject | 20:43 |
| pabelanger | clarkb: so, we could edit /etc/apache2/default and update setting for HTCACHECLEAN_PATH. apache2 service already starts htcacheclean process itsself | 20:43 |
| pabelanger | then tweak the interval settings | 20:43 |
| pabelanger | and we can remove crontab | 20:43 |
| clarkb | pabelanger: and hte sizes | 20:43 |
| clarkb | but ya thta should work | 20:43 |
| pabelanger | yup | 20:43 |
| pabelanger | default is 300M | 20:44 |
| pabelanger | since rax-ord is getting replaced, I can quickly test it there | 20:44 |
| clarkb | but also suppoesdly we can just run find /path/to/cache/ -mtime 7 -delete; | 20:45 |
| clarkb | this won't give us a max size to stay under but may delete more aggressively | 20:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add inap cloud definition https://review.openstack.org/493226 | 20:47 |
| *** annegentle has quit IRC | 20:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add inap cloud https://review.openstack.org/493072 | 20:48 |
| *** bobh has joined #openstack-infra | 20:48 | |
| clarkb | also maybe its time to figure out squid | 20:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Remove internap in favor of inap https://review.openstack.org/493074 | 20:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add inap cloud https://review.openstack.org/493072 | 20:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Stop using internap in favor of inap https://review.openstack.org/493073 | 20:49 |
| clarkb | (though I like explicitly pointin at urls rather than using http_proxy it seems so much simpler | 20:49 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add zuul.project.src_dir variable https://review.openstack.org/491915 | 20:50 |
| *** thorst has quit IRC | 20:52 | |
| *** baoli has quit IRC | 20:52 | |
| pabelanger | clarkb: okay, /etc/default/apache2 changes work | 20:53 |
| pabelanger | will propose a patch shortly | 20:53 |
| *** rhallisey has quit IRC | 20:54 | |
| openstackgerrit | Ryan proposed openstack-infra/bindep master: Add ability to list all deps https://review.openstack.org/492693 | 20:55 |
| *** markvoelker has joined #openstack-infra | 20:56 | |
| *** sekelso has quit IRC | 20:59 | |
| fungi | clarkb: interesting... did we not use noatime? | 20:59 |
| clarkb | fungi: doesn't look like it | 20:59 |
| fungi | looks like we did not | 20:59 |
| fungi | s/we/i/ | 20:59 |
| clarkb | / has it | 21:00 |
| clarkb | but not the new volume | 21:00 |
| clarkb | which could explain a difference between clouds too where we have the cache on / | 21:00 |
| jeblair | fungi, clarkb: relatime may be the default now | 21:00 |
| *** Apoorva_ has joined #openstack-infra | 21:00 | |
| fungi | on modern kernels it is, but i don't remember when that changed | 21:00 |
| fungi | i think you can -onoatime,remount that on the fly can't you? we don't need a downtime to set it if we want | 21:01 |
| fungi | worth trying and seeing if it objects i guess | 21:01 |
| jeblair | fungi: it's the case on the internap (trusty) mirror | 21:01 |
| jeblair | i mean, noatime is still better | 21:01 |
| jeblair | just saying it's only 'bad' not 'terribad' | 21:01 |
| clarkb | jeblair: ha | 21:01 |
| fungi | agreed, but likely not the cause of our troubles here | 21:01 |
| fungi | might not be helping though, agreed | 21:01 |
| *** jistr is now known as jistr|off | 21:02 | |
| fungi | just happened to glimpse the fstab on nodepool.o.o and see /opt is mounted without noatime there too | 21:02 |
| clarkb | fungi: `mount -onoatime,remount /var/cache/apache2` ? you think that is safe to give a go on ord's mirror? | 21:03 |
| fungi | clarkb: worth a try. for starters if it's not going to work it will just tell you anyway, but also there are only 3 jobs still running there and i doubt any of them are still using the mirror at this stage | 21:03 |
| clarkb | ah ok will give it a go | 21:03 |
| *** Apoorva has quit IRC | 21:04 | |
| clarkb | it worked | 21:04 |
| fungi | is that additive with the other options already specified in /etc/fstab? | 21:04 |
| clarkb | ya errors=remote-ro seemed to stick at least | 21:05 |
| fungi | anyway, wondering what a good solution might be to get our fstabs consistent. probably just need to do an audit with the help of ansible and then manually handle the resulting list | 21:05 |
| clarkb | as did the barriers option | 21:05 |
| jeblair | fungi: ++ | 21:06 |
| jeblair | clarkb: what creates the subdirs in /var/cache/apache2? | 21:06 |
| clarkb | jeblair: puppet creates the one for proxy, apache package install creats the other iirc | 21:06 |
| jeblair | (i saw some on the root filesystem in my new mirror, but then mounted an empty filesystem over them; wondering if i need to intervene to recreate them) | 21:06 |
| jeblair | ok, based on apache package install, i'll unmount, move, remount | 21:07 |
| *** trown is now known as trown|outtypewww | 21:07 | |
| fungi | when i added volumes i manually copied them out and back in (with apache2 stopped) | 21:07 |
| fungi | just to be on the safe side | 21:07 |
| clarkb | fungi: you'll want to apply the noatime and update fstab on the new mirror node in ord | 21:07 |
| pabelanger | clarkb: https://serverfault.com/questions/320767/apaches-htcacheclean-doesnt-scale-how-to-tame-a-huge-apache-disk-cache interesting approach in 2nd comment | 21:08 |
| jeblair | okay, i think mirror.inap should be good to go (i included noatime in fstab) | 21:08 |
| *** dprince has quit IRC | 21:09 | |
| clarkb | pabelanger: neat | 21:09 |
| clarkb | pabelanger: that may actually be worth trying before we go to the daemon | 21:10 |
| pabelanger | ya, going to try that now | 21:10 |
| clarkb | pabelanger: keep in mind our number of top level dirs is different | 21:10 |
| clarkb | (its currently huge) | 21:11 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Skip long-running jobs for only placement-api-ref changes https://review.openstack.org/493242 | 21:12 |
| pabelanger | clarkb: 185811 currently | 21:12 |
| *** dave-mccowan has quit IRC | 21:12 | |
| clarkb | pabelanger: ya 64 ** 3 is limit I think | 21:13 |
| clarkb | upper case + lower case alphabet, 10 digits, _ and @ all added together is 64 then there are three characters per name so 64 **3 | 21:13 |
| *** Guest74276 has quit IRC | 21:14 | |
| clarkb | pabelanger: if we put my change in place to shorten to 2 characters it will be 4096 as in the example you linked | 21:14 |
| pabelanger | clarkb: ya, might be best | 21:14 |
| clarkb | I'm just not sure what the effect of changing those values under apache is. Guessing it will just start building a new cache with new structure and we can rm the old one completely but thats a guess | 21:15 |
| *** sekelso has joined #openstack-infra | 21:15 | |
| clarkb | we could test it in rax.ord before enabling that region again | 21:16 |
| clarkb | fungi: ^ thoughts on doing that? | 21:16 |
| *** ldnunes has quit IRC | 21:16 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Rename allow-secrets to post-review https://review.openstack.org/493059 | 21:17 |
| *** vhosakot has quit IRC | 21:20 | |
| fungi | clarkb: seems safe enough to try | 21:20 |
| *** vhosakot has joined #openstack-infra | 21:20 | |
| fungi | we _just_ reached 0 nodes there | 21:20 |
| jeblair | clarkb, fungi: you could also experiment on inap | 21:21 |
| fungi | wanna test on the old mirror, or after i swap the new one in? | 21:21 |
| fungi | or jeblair's suggestion | 21:21 |
| jeblair | it has no data now; you could prime it with some data, then make the switch. it will be a nice small test case | 21:21 |
| jeblair | easy to repeat if you want to refine the approach | 21:21 |
| jeblair | then try out on ord | 21:21 |
| *** priteau has joined #openstack-infra | 21:21 | |
| pabelanger | ya, inap sounds like a good idea | 21:21 |
| jeblair | i'm all done there, so go right ahead | 21:22 |
| clarkb | ok lets make some requests through the proxies | 21:22 |
| clarkb | is it mirror.mtl01.inap.openstack.org? | 21:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Zuulv3: remove infra-publish-jobs from zuul repos https://review.openstack.org/493097 | 21:24 |
| *** bh526r has quit IRC | 21:24 | |
| *** priteau has quit IRC | 21:24 | |
| clarkb | looks lik eit is, adding it to the ansible emergency file now | 21:25 |
| clarkb | ok hand editing the vhots file to use 2 * 8 instead of 3 * 5 | 21:26 |
| clarkb | hrm while we are at it do we want to shorten up the dir depth? | 21:26 |
| clarkb | I did 8 because 16 is roughly the same as 15 | 21:27 |
| clarkb | maybe lets keep the depth the same at 5 | 21:27 |
| pabelanger | sure | 21:27 |
| *** kjackal_ has joined #openstack-infra | 21:28 | |
| *** Hal has joined #openstack-infra | 21:28 | |
| fungi | i'm offlining the old mirror.ord.rax now and moving the cinder volumes | 21:28 |
| *** Hal is now known as Guest60127 | 21:28 | |
| pabelanger | WFM, disconnected | 21:28 |
| jeblair | i'm going to release zuul-sphinx 0.1.4 so we can start using the new directives | 21:28 |
| clarkb | ok based on quick testing it will continue to use the old cache too. I requested a pbr file through proxy and it made three char entries in cache. I then requested it again afte rthe change and no new caching. When I requested a different file I got the two character cache | 21:29 |
| clarkb | so this actually like is fairly safe to apply as is then we can let the old stuff die on the vine. Let me update my change to switch depth back to 5 and keep that constant | 21:29 |
| *** markvoelker has quit IRC | 21:30 | |
| jeblair | clarkb: some of it will never die though, right? | 21:30 |
| fungi | depends on if htcacheclean will eventually clean it out, i guess | 21:30 |
| jeblair | if we're planning on optimizing how htcacheclean runs based on dir structure, it still might be worth doing a purge after making the change and just let the cache rebuild | 21:31 |
| clarkb | jeblair: ya its possible we never clear it all out depending on htcacheclean behavior | 21:31 |
| jeblair | pushed zuul-sphinx 0.1.4 | 21:31 |
| clarkb | jeblair: ++ to clearing it out but we should be able to do that at our leisure after the change goes into place and not worry about synchronizing it all | 21:32 |
| jeblair | ya | 21:32 |
| clarkb | jeblair: basically put 2 char in place for a day or two, then just rm everything with three char prefix | 21:32 |
| jeblair | ++ | 21:32 |
| clarkb | I'm going to test I can delete the three char prefixes on inap safely now | 21:33 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Make apache cache structure saner https://review.openstack.org/493112 | 21:35 |
| *** funzo has joined #openstack-infra | 21:35 | |
| *** vhosakot_ has joined #openstack-infra | 21:36 | |
| *** Apoorva_ has quit IRC | 21:38 | |
| *** EricGonc_ has quit IRC | 21:38 | |
| *** vhosakot has quit IRC | 21:38 | |
| *** EricGonczer_ has joined #openstack-infra | 21:38 | |
| *** Apoorva has joined #openstack-infra | 21:38 | |
| clarkb | ya that seems to work fine. I ran `sudo find ./ -maxdepth 1 -name '???' -exec rm -rf {} \;` from within /var/cache/apache2/proxy to delete the dirs I didn't want | 21:39 |
| *** funzo has quit IRC | 21:40 | |
| clarkb | so we can run ^ in a day or two after 493112 is in place | 21:40 |
| *** thorst has joined #openstack-infra | 21:44 | |
| *** ihrachys has joined #openstack-infra | 21:46 | |
| *** sekelso has quit IRC | 21:46 | |
| pabelanger | cool | 21:48 |
| fungi | dns updated, volumes reattached to new server and mounted (noatime!), old server deleted, ansible inventory cache cleared, puppetmaster cached copy of mirror's ssh host key updated... | 21:48 |
| *** thorst has quit IRC | 21:48 | |
| fungi | server is rebooting now | 21:48 |
| fungi | to make sure everything boots correctly unattended | 21:49 |
| clarkb | fungi: rsponded to your question on 493112 with data learned from testing against mtl01.inap | 21:49 |
| fungi | we likely still want to do a pass with htcacheclean on the new mirror.ord.rax before turning it back up in nodepool | 21:50 |
| fungi | it doesn't really seem to have made any headway while mounted to the previous server | 21:50 |
| clarkb | ++ | 21:50 |
| clarkb | fungi: don't forget to mount noatime too and update fstab | 21:50 |
| fungi | already did! | 21:51 |
| clarkb | woot | 21:51 |
| fungi | i've manually started a flock'd htcacheclean at -l 51200M under screen | 21:53 |
| fungi | will leave that going while i make dinner | 21:53 |
| clarkb | doing maths I think 2 * 5 gives us a total of 4096 ** 5 total cacheable entities? which is a ton so should be plenty | 21:54 |
| *** xyang1 has quit IRC | 21:54 | |
| *** annegentle has joined #openstack-infra | 21:56 | |
| *** spzala has joined #openstack-infra | 22:00 | |
| *** vhosakot_ has quit IRC | 22:02 | |
| *** vhosakot has joined #openstack-infra | 22:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Add jobvar and rolevar directives https://review.openstack.org/493248 | 22:06 |
| *** bobh has quit IRC | 22:06 | |
| pabelanger | clarkb: cool, and we can even implement the htcacheclean from serverfault to make things potentially faster too | 22:07 |
| *** markvoelker has joined #openstack-infra | 22:10 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add node.cloud to zookeeper https://review.openstack.org/493086 | 22:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Use new sphinx roles in docs https://review.openstack.org/493250 | 22:10 |
| pabelanger | clarkb: care to +3 https://review.openstack.org/492712 dnsserver changes for infracloud. they were restored this morning | 22:16 |
| *** lbragstad has quit IRC | 22:17 | |
| *** ihrachys has quit IRC | 22:26 | |
| *** vhosakot has quit IRC | 22:32 | |
| *** xarses_ has quit IRC | 22:33 | |
| *** markvoelker has quit IRC | 22:33 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud master: Remove dns_servers from provider-subnet-infracloud https://review.openstack.org/492712 | 22:33 |
| *** rbrndt has quit IRC | 22:35 | |
| *** funzo has joined #openstack-infra | 22:36 | |
| fungi | wow, finished with dinner, htcacheclean has removed 4k of utilization in /var/cache/apache2 on mirror.ord.rax | 22:36 |
| fungi | basically freed one block on the filesystem | 22:37 |
| fungi | over the course of some 45 minutes | 22:37 |
| fungi | clarkb: have a link to the recommendations you saw for using find to do this? | 22:38 |
| *** rlandy has quit IRC | 22:38 | |
| pabelanger | fungi: ya, I had run htcacheclean for over 90mins, and didn't make a dent | 22:39 |
| pabelanger | it was still hashing directories I think | 22:39 |
| fungi | le sighs | 22:39 |
| *** funzo has quit IRC | 22:40 | |
| clarkb | fungi: https://serverfault.com/questions/585793/htcacheclean-doesnt-seem-to-be-cleaning-the-cache | 22:40 |
| pabelanger | clarkb: ya, was going to ask if we just deleted directories manually under apache, if it cared | 22:41 |
| *** spzala has quit IRC | 22:41 | |
| fungi | clarkb: what about the recommendation there to set dir_index in the fs? | 22:42 |
| clarkb | fungi: I think thats default for ext4 isnt it? | 22:43 |
| clarkb | pabelanger: testing on inap mirror it didnt seem to care | 22:43 |
| fungi | maybe tune2fs -l doesn't show it? | 22:43 |
| fungi | yeah, even after setting tune2fs -O dir_index on it, still doesn't show up in the listing so i guess it's just assumed on ext4 | 22:46 |
| fungi | no, wait, it does | 22:47 |
| fungi | somehow i managed to copy some non-ascii zero-width whitespace on the end of it into my buffer the first time | 22:48 |
| fungi | and yeah, seems to already be on in other regions too, so this is not the problem | 22:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Increase puppet integration job timeout to 90m https://review.openstack.org/492544 | 22:50 |
| *** annegentle has quit IRC | 22:50 | |
| pabelanger | at this point, maybe we should just attach a new volume for proxy cache and land clarkb apache change? | 22:51 |
| pabelanger | shouldn't take long to reprime those reverse proxy caches | 22:52 |
| *** jamesmcarthur has joined #openstack-infra | 22:53 | |
| *** jamesmcarthur has quit IRC | 22:57 | |
| fungi | well, i'm giving find a try first since there's apparently nothing fancy we need to do there | 22:57 |
| *** kjackal_ has quit IRC | 23:01 | |
| fungi | yeah, this is going waaaay faster | 23:01 |
| fungi | at the moment i'm just removing entries older than a month | 23:02 |
| fungi | but it's already cleared a good 300mib | 23:02 |
| *** claudiub has quit IRC | 23:04 | |
| *** spzala has joined #openstack-infra | 23:04 | |
| *** Sukhdev has quit IRC | 23:04 | |
| jeblair | fungi: shouldn't cache clearing be more sophisticated than that? like, don't we want to delete things that haven't been hit in a month? not things that we've been sucessfully caching for > 1 month? | 23:07 |
| fungi | yep, i expect that's what htcacheclean is trying to do | 23:08 |
| fungi | on the other hand, without atime, there is no filesystem-level indicator of what's been accessed when | 23:08 |
| *** sdague has quit IRC | 23:09 | |
| clarkb | apaches cache time limit is 24 hpurs by default too iirc | 23:09 |
| clarkb | so can delete verything older than that in theory | 23:09 |
| jeblair | fungi: right, so htcacheclean is checking metadata? | 23:09 |
| clarkb | ya it wrotes a headers file and a data file and I thi k it uses headers to know how valid stuff is | 23:09 |
| fungi | clarkb: you mean it updates the last modified time on stuff in the cache even if there is no newer version it needs to re-retrieve? | 23:09 |
| fungi | not entirely clear to me why apache wouldn't itself just delete old data if it's unconditionally refreshing it every 24 hours | 23:11 |
| jeblair | (also, unconditional expiration seems counter to what we'd desire; most of our actual data is permanent, just indexes change) | 23:12 |
| jeblair | i said "most" | 23:12 |
| *** hongbin has quit IRC | 23:12 | |
| fungi | seems more likely the files in the cache with a filesystem mtime >30 days are still able to be served up by apache when requested and it wouldn't bother replacing them if the header check for the remote site said it hadn't been modified more recently | 23:13 |
| *** pbourke has quit IRC | 23:13 | |
| fungi | so i agree this is removing files from the cache which are likely to be viable | 23:14 |
| fungi | just not aware of a more sophisticated way to go about getting us under 100% used on the fs without either blowing away (at least part of) the cache or doing something likely as underperforming as htcacheclean itself | 23:14 |
| *** pbourke has joined #openstack-infra | 23:15 | |
| *** esberglu has joined #openstack-infra | 23:17 | |
| *** caphrim007 has quit IRC | 23:17 | |
| fungi | so with the other options being get rid of the cache entirely and start fresh or wait (possibly literally) forever for htcacheclean to run on the current dataset, removing some files from the cache based on their age seems like a reasonable compromise | 23:17 |
| *** caphrim007 has joined #openstack-infra | 23:17 | |
| *** clayton has quit IRC | 23:18 | |
| *** clayton has joined #openstack-infra | 23:18 | |
| fungi | worth noting, we're also using htcacheclean's -n option | 23:19 |
| *** krtaylor has quit IRC | 23:21 | |
| *** esberglu has quit IRC | 23:21 | |
| *** caphrim007 has quit IRC | 23:21 | |
| pabelanger | ya, was looking at that too. Wonder if we are being too nice :) | 23:22 |
| fungi | ooh! another thing we might consider... in apache 2.4 htcacheclean lets you delete specific urls | 23:22 |
| clarkb | fungi: it will treat the old data as invalid at 24 hours and attempt to refresh it | 23:22 |
| clarkb | aiui | 23:23 |
| fungi | like, for example, the pypi mirror we're not not pumping through it | 23:23 |
| clarkb | but it doesnt rm it | 23:23 |
| fungi | clarkb: so you're saying any files in /var/cache/apache2/proxy with an mtime >1 (day) are not something apache will ever serve from the cache again? | 23:24 |
| fungi | i guess that's to avoid trusting the timestamps/headers from the remote site? | 23:27 |
| pabelanger | ya, think so: https://httpd.apache.org/docs/2.4/mod/mod_cache_socache.html#cachesocachemaxtime | 23:28 |
| pabelanger | actually, that is mod_cach_socache | 23:28 |
| pabelanger | we are using cache_disk | 23:28 |
| fungi | right | 23:28 |
| pabelanger | I don't see any expire / age options for mod_cache_disk | 23:29 |
| pabelanger | just file size min / max | 23:29 |
| fungi | looks like mod_cache_socache is intended more for smaller http responses | 23:29 |
| fungi | documentation talks about backing up socache usage with mod_cache_disk handling the larger stuff | 23:30 |
| pabelanger | The mod_cache_disk module makes no attempt to regulate the amount of disk space used by the cache, although it will gracefully stand down on any disk error and behave as if the cache was never present. | 23:32 |
| pabelanger | https://httpd.apache.org/docs/2.4/caching.html | 23:32 |
| pabelanger | seems htcleancache is independant of that, and decides how to expire the data | 23:33 |
| pabelanger | htcacheclean* | 23:33 |
| *** vhosakot has joined #openstack-infra | 23:33 | |
| fungi | i think the important bit on that page is in the "Three-state RFC2616 HTTP caching" section near the top where it talks specifically about the "Stale" state | 23:34 |
| fungi | "... An HTTP cache should contact the origin server and check whether the content is still fresh before serving stale content to a client. The origin server will either respond with replacement content if not still valid, or ideally, the origin server will respond with a code to tell the cache the content is still fresh, without the need to generate or send the content again. ..." | 23:34 |
| tonyb | Is anyone seeign issues with etherpad? I get to make 1 edit before being 'force reconnected', and then the edit I made has been lost :( | 23:34 |
| fungi | tonyb: how huge is the pad in question? | 23:35 |
| pabelanger | fungi: Ya, in fact we ignore some of those headers with docker reverse proxy, otherwise we didn't cache anything | 23:35 |
| jeblair | fungi: yeah *checking* again is fine, but blindly deleting and refetching is :( | 23:35 |
| tonyb | fungi: ~200 characters | 23:36 |
| tonyb | fungi: +formatting | 23:36 |
| *** funzo has joined #openstack-infra | 23:36 | |
| fungi | jeblair: right, i'm saying i think the docs there agree with my (and your) assumption that old files in the cache are still potentially valid for fulfilling future requests if the remote server agrees and apache refreshes the staleness data | 23:37 |
| fungi | tonyb: okay, because i've seen that occasionally with very large pads, but this doesn't sound like the case | 23:37 |
| fungi | tonyb: i'm not having any trouble staying connected to a pad and updating content, but the one i'm testing also doesn't have anyone else editing it at the moment | 23:38 |
| clarkb | fungi: sorry juggling kids let me try to find the 24 hour docs | 23:38 |
| fungi | tonyb: are there other editors connected to yours? | 23:38 |
| clarkb | https://httpd.apache.org/docs/2.4/mod/mod_cache.html#cachemaxexpire | 23:38 |
| tonyb | fungi: it has one other editor | 23:39 |
| clarkb | so it will recheck origin afterwards and presumably recache the content if the content is deleted under it | 23:39 |
| clarkb | whereas if you delete somethi g less than a day old it can try to serve it directly | 23:39 |
| tonyb | fungi: It's probably at my end I'll try it from my phone .... | 23:39 |
| fungi | tonyb: yeah, sorry, best guess is you're having local network issues or there's a bad peering point somewhere on the internet you're traversing | 23:40 |
| *** nick__ has joined #openstack-infra | 23:40 | |
| *** funzo has quit IRC | 23:41 | |
| fungi | clarkb: right, so deleting things more than a day old is still cooling the cache to some extent by forcing apache to retrieve those files again if a client asks for them when it wouldn't otherwise, right? | 23:41 |
| clarkb | yes I think so | 23:41 |
| fungi | it would have instead only checked with the origin to confirm the file was still fresh | 23:41 |
| fungi | and not necessarily retrieved that file all over again | 23:41 |
| fungi | so anyway, all that aside, i still don't have a great answer for freeing up space on mirror.ord.rax in a way which won't cool the current cache somewhat by removing older files and forcing apache to potentially need to refetch them again | 23:42 |
| tonyb | fungi: okay. Same behaviour from my phone (on 4G not wifi) so Meh I'll try again tomorrow. Thanks again | 23:42 |
| *** Hypercube32 has quit IRC | 23:42 | |
| fungi | tonyb: if you give me the url to the pad i can try to see if it's something specific to that pad i guess? | 23:43 |
| *** zzzeek has quit IRC | 23:43 | |
| tonyb | fungi: https://etherpad.openstack.org/p/requirements-cycle-process | 23:43 |
| *** zzzeek has joined #openstack-infra | 23:44 | |
| fungi | tonyb: not seeming to have trouble so far | 23:45 |
| fungi | maybe it's a ttl/round-trip related problem? | 23:45 |
| fungi | my average rtt is 95.848ms and that's going over an ipv6 gre tunnel to another provider... ~11 hops not counting the ones beneath the tunnel i can't see | 23:47 |
| tonyb | fungi: Perhaps, it's 17 hops according to tcptracepath | 23:48 |
| fungi | also no appreciable spikes in latency nor packet loss noticed | 23:48 |
| tonyb | fungi: rtt min/avg/max/mdev = 226.543/226.736/226.894/0.121 ms according to ping -c10 | 23:48 |
| fungi | gotta love being on the opposite side of the world | 23:48 |
| tonyb | :D | 23:48 |
| fungi | i don't have a great way to test 200+ms rtt situations to that server unfortunately | 23:49 |
| * tonyb considers fireing up a broswer in the US. | 23:49 | |
| fungi | do you normally have any issues using etherpad.o.o? | 23:49 |
| fungi | or do you not use it often? | 23:49 |
| tonyb | fungi: No it's normally fine this is quite strange | 23:50 |
| tonyb | fungi: I use it semi-regularly more at this time of the cycle | 23:50 |
| *** Apoorva_ has joined #openstack-infra | 23:50 | |
| fungi | okay, so it's probably _not_ the latency in particular. i assume 227ms average is typical (perhaps even good?) for the distance being covered | 23:50 |
| tonyb | Yeah pretty typical | 23:51 |
| fungi | if only we could beam your packets straight through the earth's mantle | 23:51 |
| tonyb | ;P | 23:51 |
| tonyb | etherpad needs javascript right, so I can't use links or similar text web browser via ssh to the US | 23:51 |
| fungi | yeah, you'd need to do something like remote x11 | 23:52 |
| fungi | etherpad is extremely js-heavy | 23:52 |
| *** xarses_ has joined #openstack-infra | 23:53 | |
| tonyb | hmm elinks claims to support js .... | 23:53 |
| fungi | very minimally | 23:53 |
| fungi | i use (and like!) elinks, but its js support is like maybe enough to handle page refresh scripts | 23:54 |
| *** Apoorva has quit IRC | 23:54 | |
| tonyb | Yeah I get 'you must have javascript' | 23:54 |
| fungi | been a while since i x-hosted an application from another machine, but i would assume modern distros don't make that extremely hard (probably just need to make sure you pass -X to your ssh client) | 23:54 |
| *** Apoorva_ has quit IRC | 23:55 | |
| fungi | granted, x-hosting at 200+ms is likely extremely painful in its own right | 23:55 |
| tonyb | Yeah | 23:55 |
| tonyb | If it still is an issue on Monday I'll get extreme | 23:55 |
| tonyb | but for right now etherpad edits aren't on the critcal path. I really only asked incase it was a symptom of a probelm y'll needed to know about which it seems not. | 23:56 |
| *** baoli has joined #openstack-infra | 23:57 | |
| fungi | well, it's a problem i can't personally seem to recreate, but that doesn't mean it isn't a problem | 23:57 |
| fungi | i also haven't hears anyone else mention it. then again, it's friday night here and saturday morning there... likely not a lot of people around trying to use it anyway | 23:57 |
| fungi | s/hears/heard/ | 23:57 |
| tonyb | Yeah. It's probably a sign I should go ride a bike .... | 23:58 |
| * tonyb isn't good at reading signs | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!