| *** igordc has quit IRC | 00:05 | |
| *** weifan has quit IRC | 00:08 | |
| *** sgw has quit IRC | 00:12 | |
| *** dchen has joined #openstack-infra | 00:12 | |
| *** igordc has joined #openstack-infra | 00:24 | |
| mordred | corvus: awesome. I look forward to debugging it! | 00:27 |
|---|---|---|
| *** igordc has quit IRC | 00:30 | |
| harlowja | stephenfin, sure (my damn irc logging faileed) | 00:34 |
| harlowja | stephenfin, do u know anyone in storaged group ;) | 00:35 |
| harlowja | i'll trade doc8 for 1 contact, lol | 00:36 |
| *** yamamoto has quit IRC | 00:45 | |
| *** ruffian_sheep has joined #openstack-infra | 00:55 | |
| *** zhangfei has joined #openstack-infra | 00:59 | |
| *** gyee has quit IRC | 01:00 | |
| *** ricolin_ is now known as ricolin | 01:04 | |
| *** yamamoto has joined #openstack-infra | 01:07 | |
| *** yamamoto has quit IRC | 01:08 | |
| *** imacdonn has quit IRC | 01:16 | |
| *** ykarel|away has quit IRC | 01:17 | |
| *** imacdonn has joined #openstack-infra | 01:17 | |
| *** whoami-rajat has joined #openstack-infra | 01:34 | |
| *** yamamoto has joined #openstack-infra | 01:47 | |
| *** rh-jelabarre has quit IRC | 01:48 | |
| *** rh-jelabarre has joined #openstack-infra | 01:49 | |
| *** yamamoto has quit IRC | 01:54 | |
| *** armax has quit IRC | 02:03 | |
| *** zhangfei has quit IRC | 02:15 | |
| *** tkajinam has quit IRC | 02:21 | |
| *** tkajinam has joined #openstack-infra | 02:21 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:26 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:01 | |
| *** armax has joined #openstack-infra | 03:01 | |
| *** bhavikdbavishi has quit IRC | 03:02 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:02 | |
| *** armax has quit IRC | 03:06 | |
| *** ruffian_sheep has quit IRC | 03:26 | |
| *** psachin has joined #openstack-infra | 03:36 | |
| *** ricolin_ has joined #openstack-infra | 03:40 | |
| *** sgw has joined #openstack-infra | 03:41 | |
| *** ricolin has quit IRC | 03:42 | |
| *** rpittau|afk is now known as rpittau | 03:44 | |
| *** ruffian_sheep has joined #openstack-infra | 03:51 | |
| *** ruffian_sheep has quit IRC | 03:56 | |
| *** ykarel has joined #openstack-infra | 03:59 | |
| *** rh-jelabarre has quit IRC | 04:03 | |
| *** udesale has joined #openstack-infra | 04:04 | |
| *** _erlon_ has quit IRC | 04:12 | |
| *** ramishra has joined #openstack-infra | 04:14 | |
| *** armax has joined #openstack-infra | 04:15 | |
| *** apetrich has quit IRC | 04:20 | |
| *** pcaruana has joined #openstack-infra | 04:37 | |
| *** rpittau is now known as rpittau|Imonapla | 04:38 | |
| *** rpittau|Imonapla is now known as rpittau|flying | 04:38 | |
| *** dpawlik has joined #openstack-infra | 04:51 | |
| *** ykarel has quit IRC | 04:54 | |
| *** tkajinam has quit IRC | 05:01 | |
| *** armax has quit IRC | 05:04 | |
| *** ykarel has joined #openstack-infra | 05:08 | |
| *** ruffian_sheep has joined #openstack-infra | 05:35 | |
| *** raukadah is now known as chandankumar | 05:36 | |
| *** ykarel_ has joined #openstack-infra | 05:40 | |
| *** udesale has quit IRC | 05:42 | |
| *** ykarel has quit IRC | 05:42 | |
| *** udesale has joined #openstack-infra | 05:42 | |
| *** yamamoto has joined #openstack-infra | 05:44 | |
| *** ruffian_sheep has quit IRC | 05:44 | |
| *** armax has joined #openstack-infra | 05:45 | |
| *** dulek has quit IRC | 05:47 | |
| *** jistr has quit IRC | 05:47 | |
| *** kaisers has quit IRC | 05:47 | |
| *** mgoddard has quit IRC | 05:48 | |
| *** kaisers has joined #openstack-infra | 05:48 | |
| *** jaicaa has quit IRC | 05:48 | |
| *** jistr has joined #openstack-infra | 05:48 | |
| *** jaicaa has joined #openstack-infra | 05:49 | |
| *** roman_g has quit IRC | 05:49 | |
| *** roman_g has joined #openstack-infra | 05:49 | |
| *** dulek has joined #openstack-infra | 05:50 | |
| *** mgoddard has joined #openstack-infra | 05:51 | |
| *** tkajinam has joined #openstack-infra | 05:55 | |
| *** armax has quit IRC | 05:57 | |
| *** rcernin has quit IRC | 06:02 | |
| *** jbadiapa has quit IRC | 06:03 | |
| *** udesale has quit IRC | 06:12 | |
| *** udesale has joined #openstack-infra | 06:17 | |
| *** ykarel__ has joined #openstack-infra | 06:17 | |
| *** ykarel_ has quit IRC | 06:19 | |
| *** joeguo has quit IRC | 06:20 | |
| *** iurygregory has joined #openstack-infra | 06:39 | |
| *** odicha has joined #openstack-infra | 06:46 | |
| *** pgaxatte has joined #openstack-infra | 07:02 | |
| *** ykarel__ is now known as ykarel | 07:08 | |
| *** slaweq has joined #openstack-infra | 07:09 | |
| *** zbr has quit IRC | 07:15 | |
| *** zbr has joined #openstack-infra | 07:15 | |
| *** ginopc has joined #openstack-infra | 07:21 | |
| *** ykarel is now known as ykarel|lunch | 07:23 | |
| *** tosky has joined #openstack-infra | 07:24 | |
| openstackgerrit | Merged opendev/irc-meetings master: After transforming publiccloud wg to sig, meeting info changes as well. https://review.opendev.org/671509 | 07:45 |
| *** jbadiapa has joined #openstack-infra | 07:49 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Install system dependencies for tox-molecule https://review.opendev.org/671029 | 07:56 |
| *** lucasagomes has joined #openstack-infra | 07:59 | |
| *** ralonsoh has joined #openstack-infra | 08:00 | |
| *** ruffian_sheep has joined #openstack-infra | 08:03 | |
| *** ccamacho has joined #openstack-infra | 08:04 | |
| *** derekh has joined #openstack-infra | 08:08 | |
| openstackgerrit | Jan Kubovy proposed opendev/gear master: Add BSD/Darwin support. https://review.opendev.org/671674 | 08:11 |
| *** rpittau|flying is now known as rpittau | 08:13 | |
| *** pkopec has joined #openstack-infra | 08:13 | |
| openstackgerrit | Jan Kubovy proposed opendev/gear master: Add BSD/Darwin support. https://review.opendev.org/671674 | 08:16 |
| *** ykarel|lunch is now known as ykarel | 08:24 | |
| *** dchen has quit IRC | 08:25 | |
| *** gfidente has joined #openstack-infra | 08:28 | |
| *** gtarnaras has joined #openstack-infra | 08:33 | |
| *** siqbal has joined #openstack-infra | 08:46 | |
| *** panda is now known as panda|drappt | 08:50 | |
| *** ricolin_ is now known as ricolin | 08:59 | |
| *** tkajinam has quit IRC | 09:05 | |
| *** derekh has quit IRC | 09:09 | |
| *** derekh has joined #openstack-infra | 09:19 | |
| *** xek has joined #openstack-infra | 09:20 | |
| *** yolanda has quit IRC | 09:30 | |
| *** joeguo has joined #openstack-infra | 09:33 | |
| openstackgerrit | Jan Kubovy proposed opendev/gear master: Add BSD/Darwin support. https://review.opendev.org/671674 | 09:33 |
| zbr | i wonder why we keep out of sync repos on github like https://github.com/openstack-infra/zuul-jobs ? i got confused today and cloned it for a minor change. why not archiving it? | 09:40 |
| *** apetrich has joined #openstack-infra | 09:40 | |
| *** gtarnaras has quit IRC | 09:41 | |
| *** gtarnaras has joined #openstack-infra | 09:46 | |
| *** e0ne has joined #openstack-infra | 09:48 | |
| *** yamamoto has quit IRC | 09:52 | |
| *** joeguo has quit IRC | 09:53 | |
| *** udesale has quit IRC | 10:05 | |
| *** bhavikdbavishi has quit IRC | 10:06 | |
| *** udesale has joined #openstack-infra | 10:06 | |
| *** e0ne has quit IRC | 10:06 | |
| *** yamamoto has joined #openstack-infra | 10:10 | |
| *** apetrich has quit IRC | 10:13 | |
| *** panda|drappt is now known as panda | 10:14 | |
| *** yamamoto has quit IRC | 10:15 | |
| openstackgerrit | Jan Kubovy proposed opendev/gear master: Add BSD/Darwin support. https://review.opendev.org/671674 | 10:16 |
| *** uberjay has quit IRC | 10:18 | |
| *** uberjay has joined #openstack-infra | 10:19 | |
| AJaeger | harlowja: you would need to add stephenfin to https://review.opendev.org/#/admin/groups/357,members | 10:28 |
| *** priteau has joined #openstack-infra | 10:32 | |
| *** priteau has quit IRC | 10:35 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Update heuristing of parallel starting builds. https://review.opendev.org/671702 | 10:36 |
| *** priteau has joined #openstack-infra | 10:37 | |
| jonher | https://opendev.org/openstack/nova/ is very slow to load for some reason "Page: 14730ms" | 10:37 |
| openstackgerrit | Jan Kubovy proposed opendev/gear master: Add BSD/Darwin support. https://review.opendev.org/671674 | 10:42 |
| *** udesale has quit IRC | 10:44 | |
| *** udesale has joined #openstack-infra | 10:45 | |
| *** priteau has quit IRC | 10:47 | |
| *** pgaxatte has quit IRC | 10:53 | |
| *** yamamoto has joined #openstack-infra | 10:53 | |
| *** udesale has quit IRC | 10:54 | |
| *** udesale has joined #openstack-infra | 10:55 | |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Fix node failures when at volume quota https://review.opendev.org/671704 | 10:58 |
| openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Fix node failures when at volume quota https://review.opendev.org/671704 | 10:59 |
| *** yamamoto has quit IRC | 11:00 | |
| *** udesale has quit IRC | 11:02 | |
| donnyd | So I have been trying to isolate why just a few jobs are timing out, and there was a thought that maybe is was contention in the mirror. So I moved the mirror to its own dedicated node, with its own dedicated drive... | 11:09 |
| donnyd | However it still seems a little sluggish. While performance is better than it was, it could still be faster. I am hoping someone can help me run a few tests on it today to see to the root of why | 11:10 |
| stephenfin | harlowja: storaged, you say? Can't say I do, unfortunately /o\ | 11:12 |
| zbr | stephenfin: hi | 11:20 |
| zbr | stephenfin: re doc8, i seen that you created pyqa/doc8 as empty instead of transfering or forking openstack/doc8 or pycontribs/doc8 | 11:21 |
| stephenfin | Yeah, I figured it was easy to force push everything. It's not a fork of either of those projects so much as a relocation | 11:22 |
| zbr | that is not ideal as it prevents users from being redirected from old location to new one. | 11:22 |
| stephenfin | I doubt there are many (if any) people using pycontribs/doc8 at the moment, so I think that shouldn't be an issue | 11:23 |
| zbr | in case of relocation, lets ask one admin from infra to transfer it, it will redirect all users. | 11:23 |
| zbr | sure, forget about pycontribs, i can easily recreate 3-4 pull requests. | 11:23 |
| stephenfin | As for openstack/doc8 though, I didn't even know if it was possible to move something there without a lot of work | 11:23 |
| stephenfin | I have a patch to prepare the README telling people where it's gone to so we _should_ be okay | 11:24 |
| zbr | in fact is not much work, but we need one admin to do it. clarkb it was you or fungi ? | 11:24 |
| zbr | i think that we can fix the readme after, directly on github. the only trick is that the person doing the transfer needs to be able to create repos on new organization | 11:25 |
| zbr | but i guess you can temporary give them this permissions, right? | 11:25 |
| zbr | the other option is: fork and archive. | 11:26 |
| stephenfin | I can, but for a smaller project like this I don't know if it's worth the effort, tbh | 11:26 |
| zbr | user experience is important and you can do it only from start. | 11:26 |
| stephenfin | Or email GitHub and ask them to set up a redirect for us | 11:26 |
| zbr | nope, they will not do it. | 11:26 |
| stephenfin | You sure? I know they can remove the 'Forked from' metadata if necessary | 11:27 |
| zbr | there is a process, well documented, called transfer. they do not do stuff for you. | 11:27 |
| stephenfin | Hmm, let me email and ask. They're very helpful usually | 11:28 |
| stephenfin | If they can't, we can do the move of the openstack/doc8 project on GitHub if you'd like | 11:28 |
| stephenfin | But again, I think for a smaller project like this, a simple note in the README is more than good enough | 11:29 |
| zbr | stephenfin: what is the problem of forking from openstack/doc8? | 11:29 |
| zbr | later we can ask them to archive it. | 11:29 |
| stephenfin | That doesn't solve the redirect problem | 11:29 |
| stephenfin | And it adds "forked from" metadata which isn't correct | 11:29 |
| stephenfin | As above, "It's not a fork [...] so much as a relocation" | 11:30 |
| stephenfin | So we either just push everything to this new pycqa repo and replace everything in the old GitHub/OpenDev repos with a simple note saying "we've moved to pycqa" | 11:31 |
| *** tdasilva has joined #openstack-infra | 11:32 | |
| stephenfin | Or we work through the paperwork of getting someone to move the openstack/doc8 project to the pyqca organization | 11:32 |
| stephenfin | and I'm firmly in favour of the former because it's much easier :) | 11:32 |
| *** tesseract has joined #openstack-infra | 11:33 | |
| stephenfin | (also, we won't be able to do anything about redirects for opendev.org/x/doc8, fwict. Again though, I don't think this matters. The README update is more than okay) | 11:33 |
| stephenfin | anywho | 11:33 |
| * stephenfin -> lunch | 11:33 | |
| zbr | nobody was against if i remember well, so lets see if we can do it using "transfer". much better user experience, no need to reclone or change other stuff. | 11:33 |
| mordred | zbr: I'm working on a script to handle the openstack-infra repos on github - just keep getting other stuff in front of it on the todo list | 11:37 |
| zbr | mordred: ok. but for the moment can you manually perform the transfer for doc8? | 11:38 |
| *** pgaxatte has joined #openstack-infra | 11:38 | |
| *** apetrich has joined #openstack-infra | 11:38 | |
| mordred | zbr: it is possible and not terribly hard - but I'm not fully awake or connected yet so I don't want to do it and make some mistake. one of us can certainly handle it as we awaken | 11:39 |
| mordred | (for me I need at least one more coffee before I go clicking on things) | 11:39 |
| zbr | mordred: thanks. | 11:39 |
| zbr | sure. pressing the transfer button on github needs coffee, i should go and grab another one too. | 11:40 |
| zbr | i think everyone will appreciate the auto-redirect on github, no more need to put post-it notes on older locations. | 11:41 |
| mordred | agree. when things are actually moving, it's best to get the redirect in place when possible | 11:41 |
| *** panda is now known as panda|lunch | 11:42 | |
| *** xek has quit IRC | 11:43 | |
| *** xek has joined #openstack-infra | 11:43 | |
| *** _erlon_ has joined #openstack-infra | 12:05 | |
| *** rh-jelabarre has joined #openstack-infra | 12:08 | |
| *** yamamoto has joined #openstack-infra | 12:13 | |
| *** udesale has joined #openstack-infra | 12:21 | |
| *** tdasilva_ has joined #openstack-infra | 12:29 | |
| *** tdasilva has quit IRC | 12:32 | |
| *** panda|lunch is now known as panda | 12:36 | |
| *** goldyfruit has quit IRC | 12:37 | |
| *** yamamoto has quit IRC | 12:38 | |
| fungi | zbr: mordred has a to do item to clean up the remnants of openstack-infra on github, i believe | 12:46 |
| jamespage | any project-config core's have time for a bit of a tidy friday review? https://review.opendev.org/#/c/670058 | 12:47 |
| fungi | zbr: stephenfin: a workaround is that we can transfer a repository to an individual account and then that account can transfer it into a new org | 12:47 |
| mordred | infra-root: I'm going AFK for a bit this morning - back in a few hours | 12:47 |
| fungi | that still maintains the chain of redirects | 12:47 |
| *** udesale has quit IRC | 12:49 | |
| fungi | jonher: yes, corvus has some suspicions as to why gitea might be slow rendering very large repositories (nova, openstack-manuals, neutron) but is seeking volunteers who have time to dig into it (easy to deploy gitea locally from a docker image) and if necessary improving gitea upstream to perform better (the maintainers are very friendly and responsive) | 12:50 |
| jonher | i see | 12:51 |
| *** udesale has joined #openstack-infra | 12:53 | |
| *** aaronsheffield has joined #openstack-infra | 12:54 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:54 | |
| fungi | donnyd: which sorts of requests from the mirror are slow? we have two different types of caches... one is an afs cache serving prebuilt mirrors of packages, and the other is an apache caching proxy passing through requests to the usual repositories on the internet. the breakdown is at https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror/templates/mirror.vhost.j2 | 12:54 |
| fungi | both tend to be initially sluggish with a cold cache (should no longer be the case there hopefully) and also on cache misses | 12:56 |
| donnyd | fungi: Well I moved the machine to a signficantly more performant machine with shared nothing on the fastest storage I can get my hands on and looking at the logs the download speeds I am seeing aren't very fast at all. | 12:57 |
| donnyd | http://logs.openstack.org/96/670796/5/check/openstack-tox-lower-constraints/5597a80/job-output.txt#_2019-07-19_12_47_28_227849 | 12:57 |
| fungi | afs suffers a bit more than the proxy on cache misses, mainly due to the protocol used on the backend | 12:57 |
| fungi | (designed many years ago for a much older, slower internet) | 12:58 |
| *** apetrich has quit IRC | 12:58 | |
| fungi | hrm, okay so 17.6mb spread over a dozen files in the span of 3 seconds | 12:59 |
| *** dtantsur is now known as dtantsur|bbl | 12:59 | |
| fungi | from the afs cache, and they're files that are almost certainly already present i the cache locally | 12:59 |
| donnyd | well if I look elsewhere on a real cloud provider its much much much faster | 12:59 |
| donnyd | http://logs.openstack.org/53/670453/1/gate/openstack-tox-docs/ec624c9/job-output.txt#_2019-07-19_12_55_12_499438 | 12:59 |
| donnyd | for instance this one from vexxhost yields significantly better performance | 13:00 |
| donnyd | and I am thinking this is the root of why the jobs are timing out, the mirror isn't fast enough | 13:00 |
| fungi | nearly 2x, just shy of | 13:00 |
| fungi | 10.3 vs 5.6 | 13:01 |
| donnyd | yea, so something isn't quite right just don't know where to look | 13:01 |
| donnyd | storage performance has been increased about 10x | 13:02 |
| donnyd | https://www.irccloud.com/pastebin/GHkxvN9R/ | 13:02 |
| *** apetrich has joined #openstack-infra | 13:02 | |
| donnyd | that was 1m blocks | 13:02 |
| donnyd | here are 4k blocks | 13:02 |
| donnyd | https://www.irccloud.com/pastebin/wIZTIy7F/ | 13:02 |
| donnyd | so I am confident there is not a storage issue for the mirror. | 13:03 |
| fungi | mirror rebooted 11 hours ago | 13:03 |
| donnyd | yea, when i moved it | 13:03 |
| fungi | okay, cool | 13:03 |
| fungi | just making sure it was a known cause | 13:03 |
| donnyd | migration isn't working in tripleo, so i had to do it the hard way | 13:04 |
| donnyd | I should say the specific version of 3o i am on.. | 13:04 |
| donnyd | So I am hoping to maybe run some network tests | 13:05 |
| donnyd | there are only two components to test | 13:05 |
| donnyd | I am thinking the way I did routing for this network is causeing some issues | 13:05 |
| fungi | one non-provider-related difference is that the fortnebula mirror is running openafs 1.8.3 on ubuntu bionic, while the vexxhost mirror is running openafs 1.6.15 on ubuntu xenial | 13:06 |
| donnyd | but being able to confirm it would be great | 13:06 |
| fungi | it's possible the performance issues are related to software in the guest | 13:06 |
| donnyd | 100% slower??? One would think not, but its surely possible | 13:06 |
| fungi | comparing to a job which ran in rax-ord might be a better comparison (it's also newer openafs on newer ubuntu) | 13:08 |
| donnyd | ok, i will look for one | 13:09 |
| *** rlandy has joined #openstack-infra | 13:09 | |
| donnyd | And i stand corrected. | 13:09 |
| donnyd | http://logs.openstack.org/66/669866/3/check/openstack-tox-pylint/215dcc7/job-output.txt#_2019-07-19_13_02_29_584092 | 13:09 |
| fungi | we're in the process of redoing our mirrors on newer software, but i don't know if we had any real-world benchmarks comparing performance across them | 13:09 |
| donnyd | nearly the same as me | 13:09 |
| fungi | to be fair, it's also possible it's not the guest software, and the hardware in rax-ord is slow or something | 13:10 |
| jrosser | donnyd: some of the jobs run an additional inner ARA report so you can get timing info for individual ansible tasks | 13:10 |
| dmsimard | nested ansibles all the way down | 13:11 |
| jrosser | that might help decide "all apt installs from the mirror are slow" vs. "everything we pull from the internet is slow" | 13:11 |
| fungi | dmsimard: i heard you like ansible so i put ansible in your ansible and now you can ara while you ara | 13:11 |
| donnyd | If someone has time i would like to get a quick throughput test to see if the issue is networking (surely not storage anymore) or something else | 13:12 |
| dmsimard | fungi: let me say that ara integration tests are fun :P | 13:12 |
| fungi | no doubt | 13:12 |
| fungi | donnyd: sure, i can wget something to /dev/null | 13:12 |
| dmsimard | iperf ? | 13:12 |
| fungi | unless you need something more precise | 13:12 |
| donnyd | sure | 13:12 |
| donnyd | iperf 3 preferably | 13:13 |
| *** rlandy_ has joined #openstack-infra | 13:13 | |
| fungi | i'll splat some urandom of a known size onto one server we manage and then wget it from the fn mirror | 13:13 |
| *** rlandy_ has quit IRC | 13:13 | |
| fungi | oh, i can work out iperf3 cli options if that's preferable | 13:13 |
| donnyd | both ? :) | 13:13 |
| *** rlandy_ has joined #openstack-infra | 13:13 | |
| fungi | is that a client/server thing where you run something on both ends and connect them? | 13:14 |
| dmsimard | yeah | 13:14 |
| fungi | i need to do some morning things real quick and then i'll learn me some iperf3 | 13:14 |
| *** yamamoto has joined #openstack-infra | 13:14 | |
| donnyd | you just need to run iperf3 -s | 13:14 |
| *** rlandy_ has quit IRC | 13:15 | |
| *** yamamoto has quit IRC | 13:15 | |
| donnyd | it should open up a tcp server on 5201 | 13:15 |
| *** yamamoto has joined #openstack-infra | 13:15 | |
| *** rlandy_ has joined #openstack-infra | 13:15 | |
| *** rlandy has quit IRC | 13:16 | |
| donnyd | for sure, lmk when its running. thanks fungi | 13:16 |
| fungi | it'll probably be 30-45 minutes before i'm freed up, but will do | 13:16 |
| *** rlandy_ is now known as rlandy | 13:17 | |
| donnyd | ok | 13:17 |
| *** xek has quit IRC | 13:17 | |
| donnyd | np | 13:17 |
| *** eharney has joined #openstack-infra | 13:21 | |
| *** goldyfruit has joined #openstack-infra | 13:34 | |
| *** yamamoto has quit IRC | 13:34 | |
| *** yamamoto has joined #openstack-infra | 13:37 | |
| *** siqbal has quit IRC | 13:37 | |
| *** EmilienM is now known as EvilienM | 13:38 | |
| *** siqbal has joined #openstack-infra | 13:40 | |
| *** yamamoto has quit IRC | 13:41 | |
| *** tdasilva_ is now known as tdasilva | 13:43 | |
| *** TheJulia is now known as needssleep | 13:48 | |
| *** rh-jelabarre has quit IRC | 13:50 | |
| *** ekultails has joined #openstack-infra | 13:51 | |
| *** igordc has joined #openstack-infra | 13:52 | |
| *** rh-jelabarre has joined #openstack-infra | 13:56 | |
| *** eernst has joined #openstack-infra | 13:59 | |
| *** goldyfruit has quit IRC | 13:59 | |
| *** goldyfruit has joined #openstack-infra | 14:00 | |
| *** armax has joined #openstack-infra | 14:12 | |
| *** e0ne has joined #openstack-infra | 14:12 | |
| *** sgw has quit IRC | 14:13 | |
| *** eernst has quit IRC | 14:13 | |
| mnaser | clarkb: have the weirdi ssues about "reference is not a tree" been resolved? | 14:14 |
| mnaser | i have some sort of interesting information | 14:14 |
| mnaser | centos doesnt seem to be impacted, i dont see that issue happen on centos jobs | 14:15 |
| mnaser | i do see t hem on ubuntu bionic | 14:15 |
| *** ykarel is now known as ykarel|away | 14:15 | |
| mnaser | and i can see github clones working fine, but opendev ones failing | 14:15 |
| mnaser | http://logs.openstack.org/05/671105/2/check/openstack-ansible-upgrade-aio_metal-ubuntu-bionic/35cea57/job-output.txt.gz | 14:15 |
| *** yamamoto has joined #openstack-infra | 14:18 | |
| *** ykarel|away has quit IRC | 14:20 | |
| *** igordc has quit IRC | 14:20 | |
| *** dtantsur|bbl is now known as dtantsur | 14:23 | |
| *** yamamoto has quit IRC | 14:26 | |
| stephenfin | fungi: I guess the person to move 'openstack/doc8' to would be 'sigmavirus24' (Ian Stapleton Cordasco) since he's the one that's handling this for us | 14:26 |
| stephenfin | (I have it in my head he worked on OpenStack in the past but idk, tbh) | 14:26 |
| fungi | stephenfin: yep, he did | 14:27 |
| *** sgw has joined #openstack-infra | 14:27 | |
| stephenfin | So, once we've discussed it with him, perhaps we could just add him as an admin to that single repo so he can change the organization? | 14:27 |
| stephenfin | Or do we need to transfer to his personal account first? | 14:28 |
| fungi | the latter | 14:28 |
| stephenfin | fungi: Ack. I'll ping him now to make sure he's okay with us doing that | 14:29 |
| *** odicha has quit IRC | 14:30 | |
| fungi | mnaser: is there a git submodule involved? | 14:31 |
| *** pgaxatte has quit IRC | 14:32 | |
| fungi | looks like no, i think it's doing the equivalent of `git clone https://opendev.org/openstack/ansible-hardening` and trying to checkout stable/stein? | 14:33 |
| *** dpawlik has quit IRC | 14:34 | |
| fungi | yeah, show-ref on a local clone i created confirms that's the commit id for the current state of the stable/stein branch | 14:34 |
| corvus | mnaser: why does that clone instead of using zuul? | 14:36 |
| mnaser | corvus: its an upgrade job, so we need to clone stable/stein, run that, then run master, but if there's a more zuul-y way of doing it, happy to hear | 14:36 |
| corvus | mnaser: i'll work up a patch :) | 14:37 |
| mnaser | in our branch jobs we use the zuul checked out stuff repos but that would be neat | 14:37 |
| fungi | yeah, zuul will have both branches. you just need to git checkout the ones you want at the right time(s) or perform local clones of them if you need two different worktrees present at those different states simultaneously | 14:37 |
| fungi | mnaser: on the topic of the failure git is reporting, however, is it consistent on ubuntu-bionic, or only occasional? | 14:38 |
| *** gtarnaras has quit IRC | 14:39 | |
| fungi | like, if it impacts the same repos on only 1/8 runs, then maybe it's just one of our backends reporting this issue | 14:39 |
| mnaser | fungi: seems consistent on ubuntu-bionic (consistent as in like.. i had it fail 2/2 times) | 14:40 |
| mnaser | and centos worked 2/2 times but that's maybe not enough details | 14:40 |
| fungi | got it | 14:40 |
| mnaser | also, TIL that zuul has both branches | 14:40 |
| fungi | so maybe something different about the version of git on those distros | 14:41 |
| mnaser | yeah that's what i figured :\ | 14:41 |
| fungi | still, i wonder how to reproduce that error. i'll try on a bionic machine (i tested on debian/sid and couldn't reproduce) | 14:41 |
| *** gtarnaras has joined #openstack-infra | 14:42 | |
| *** mattw4 has joined #openstack-infra | 14:42 | |
| *** rlandy has quit IRC | 14:43 | |
| *** dpawlik has joined #openstack-infra | 14:46 | |
| *** rlandy has joined #openstack-infra | 14:46 | |
| fungi | yeah, not able to reproduce manually with the git cli | 14:46 |
| fungi | even on a bionic server | 14:46 |
| fungi | i wonder if the ansible scm module does anything weird | 14:47 |
| corvus | mnaser, fungi: the problem is those are shallow clones and you're trying to checkout a branch which wasn't cloned | 14:48 |
| corvus | this is the reproducer: git clone https://opendev.org/openstack/ansible-hardening --depth=10 && git checkout --force ae01cbc824c7cf8aa7eac054fde481dacc441b68 | 14:49 |
| *** siqbal has quit IRC | 14:49 | |
| fungi | aha, i didn't see the shallow in there | 14:49 |
| mnaser | ahhh, and i bet the centos version is too old to do shallow clones | 14:49 |
| mnaser | so it does full clones | 14:49 |
| corvus | nope, it's a shallow clone | 14:49 |
| mnaser | and it doesn't complain | 14:49 |
| corvus | a full clone would work | 14:49 |
| mnaser | (yeah but centos git is too old for shallow clones, i remember seeing this warning in ansible) | 14:49 |
| mnaser | which explains why it worked on centos and not on bionic | 14:49 |
| corvus | oh i see, yes | 14:49 |
| corvus | mnaser, fungi: so, really, the best solution is to let zuul handle it. the clones will be fast, full, and correct | 14:50 |
| corvus | it looks like that playbook is set up to use zuul repos if they exist | 14:50 |
| AJaeger | config-core, a couple of repo retirements are ready for review: https://review.opendev.org/670923 https://review.opendev.org/671550 https://review.opendev.org/671263 https://review.opendev.org/670058 | 14:50 |
| fungi | absolutely, i was mainly wanting to make sure we don't have some unknown problem with the gitea cluster | 14:50 |
| corvus | but there's some conditional around that which i don't understand | 14:50 |
| mnaser | yeah that code is a bit rough, but a checkout of stable/stein should be possible in there? | 14:50 |
| mnaser | and avoids us cloning anything at all | 14:50 |
| corvus | mnaser: absolutely | 14:50 |
| mnaser | ok great, ill hack away :) | 14:50 |
| mnaser | thanks corvus & fungi | 14:50 |
| *** dpawlik has quit IRC | 14:51 | |
| fungi | mnaser: and as an added bonus, if you depends-on or get queued behind a stable branch change to one of those repos, you get the possible future state from zuul rather than just the branch tip | 14:51 |
| corvus | (maybe the job needs more "required-projects" ?) | 14:51 |
| mnaser | i think that might be it, yeah | 14:51 |
| mnaser | thats probably fallen out of sync | 14:51 |
| mnaser | and maybe ill add a conditional for it to hard fail if inside gate + no local zuul clone | 14:52 |
| mnaser | s/gate/zuul/ | 14:52 |
| corvus | mnaser: that has helped us quite a few times with devstack | 14:52 |
| clarkb | as a sanity check there have been no OOMs since the swapfiles were added | 14:53 |
| fungi | mnaser: technically the bionic version of the job is doing that for you already if you don't fix the shallow bit ;) | 14:53 |
| mnaser | aha, yep | 14:53 |
| clarkb | does ansible default to a shallow clone? | 14:54 |
| clarkb | that seems like a bug if so | 14:54 |
| corvus | clarkb: no | 14:54 |
| mnaser | corvus: as a last question, if we do 'git checkout master' after doing our work, does that automatically get us the original state? | 14:54 |
| mnaser | or we need to store the exact comit and check that out | 14:54 |
| mnaser | this is probably zuul 101 but yeah :( | 14:55 |
| corvus | mnaser: if the change was for the master branch, yes; {{ zuul.branch }} will tell you which. | 14:55 |
| mnaser | ok awesome, thats perfect | 14:55 |
| mnaser | that helps clean u pthings | 14:55 |
| clarkb | mnaser: the idea is that zuul has configured the repo for you in the way it would look if the change(s) merged | 14:56 |
| corvus | mnaser: to handle both sides of the upgrade path, we may need to make things a little more complicated, but getting this to work on just changes to master is a good step :) | 14:56 |
| clarkb | so master isn't real master it is the potential future master (and same with stable/stein and so on) | 14:56 |
| corvus | mnaser: actually, it's not that complicated; you're writing branch-specific jobs, so you probably do just want to hard-code "git checkout $oldbranch" and "git checkout master". that will dtrt no matter which branch triggers the job. | 14:58 |
| *** tosky_ has joined #openstack-infra | 14:59 | |
| corvus | any objections to me restarting zuul now? | 14:59 |
| *** ruffian_sheep has quit IRC | 15:00 | |
| corvus | doing it | 15:00 |
| mnaser | corvus, clarkb: from another angle ive been thinking if i should have a job per branch (i.e. a -master and -stein etc jobs) or stick to something more dynamic | 15:00 |
| fungi | sorry, was just checking the release and release-post pipelines | 15:00 |
| fungi | looks like there are a couple of release-post jobs queued but probably not critical | 15:00 |
| corvus | fungi: i think those are false alarms | 15:01 |
| fungi | cool | 15:01 |
| fungi | i didn't get time to expand the refs on them before the status api went offline | 15:01 |
| fungi | so wasn't certain | 15:01 |
| corvus | they probably just hadn't been dequeued yet after finding no matching jobs | 15:01 |
| clarkb | mnaser: grenade is dynamic. It checks if it is doing a "up to this release" or "forward to next release" upgrade then looks up what the two branches are in a table based on the current branch and the type of upgrade | 15:01 |
| *** tosky has quit IRC | 15:02 | |
| clarkb | (I think only the up to this release type of upgrade is currently tested/used) | 15:02 |
| mnaser | Yeah I think for now we just want to test from n-1 in n .. that would be good for now | 15:02 |
| *** ricolin_ has joined #openstack-infra | 15:03 | |
| *** ricolin has quit IRC | 15:04 | |
| fungi | donnyd: reverse dns on traceroutes seems to indicate the border where you're peering is nearish to mae east. should i try to find a machine in that vicinity to be the other end of the iperf tests? or do you want something with a higher rtt/latency? | 15:08 |
| donnyd | well I was going to test it locally first to watch where the load is being placed. I have 10G to the desktop, so if I can get close to that then it isn't an issue with the way traffic is routed | 15:09 |
| corvus | #status log restarted all of zuul on 3d9498f78db956cee366e3f5b633cf37df4e5bfa | 15:09 |
| openstackstatus | corvus: finished logging | 15:09 |
| fungi | donnyd: ahh, okay, so just need iperf listening and iptables set to allow connections to that socket? | 15:10 |
| clarkb | a single red hat IP does make the vast majority of connections to opendev.org 358938 over yesterday's syslog file. Next up is codesearch at 273206, then nb02 at 80873, then another red hat ip at 62793, some random IP from bulgaria?, then nb01 at 35986 | 15:10 |
| donnyd | yea | 15:10 |
| clarkb | pointing that out beacuse the ip source hashing does likely hurt us given that distribution | 15:10 |
| donnyd | i do believe the defaults are 5201 | 15:10 |
| clarkb | (maybe we can convince red hat to use ipv6) | 15:10 |
| *** tosky_ is now known as tosky | 15:10 | |
| clarkb | that said the current cluster seems to be keeping up so maybe it is a non issue | 15:12 |
| *** zhurong has quit IRC | 15:12 | |
| corvus | re-enqueue finished | 15:13 |
| fungi | donnyd: listening on 5201/tcp in server mode now and ipv4/6 iptables rules amended to allow connections to that from anywhere | 15:15 |
| fungi | looks like it's achieving around 3Gbps? | 15:17 |
| donnyd | yea, not ideal but not too bad either | 15:17 |
| donnyd | https://www.irccloud.com/pastebin/xqqRJyWn/ | 15:17 |
| donnyd | ipv4 first and ipv6 second | 15:17 |
| fungi | significantly higher than what we're seeing in job logs anyway | 15:17 |
| donnyd | yea, way higher | 15:18 |
| fungi | several orders of magnitude | 15:18 |
| clarkb | we probably want to test the network path between the tenants too | 15:18 |
| *** jamesmcarthur has joined #openstack-infra | 15:18 | |
| donnyd | In terms of real data, we should be getting somewhere around 200-300M | 15:18 |
| donnyd | yea, thats not a half bad idea either | 15:18 |
| donnyd | that could be the bottleneck | 15:19 |
| clarkb | and maybe double check we aren't fragmenting | 15:19 |
| fungi | i guess we don't have this mirror server in cacti yet? is there a change waiting in review? | 15:19 |
| donnyd | https://www.irccloud.com/pastebin/7TVTfjQE/ | 15:19 |
| donnyd | No fragmenting from outside | 15:19 |
| fungi | shouldn't -s 1500 force fragmentation? that's the size of the data payload inside the echo | 15:21 |
| donnyd | and throughput on multiple threads is about what I expect my edge fw to be able to do | 15:21 |
| *** mattw4 has quit IRC | 15:22 | |
| *** mattw4 has joined #openstack-infra | 15:22 | |
| donnyd | ipv6 | 15:22 |
| donnyd | [SUM] 0.00-10.00 sec 7.23 GBytes 6.21 Gbits/sec 708 sender | 15:22 |
| donnyd | [SUM] 0.00-10.00 sec 7.22 GBytes 6.20 Gbits/sec receiver | 15:22 |
| donnyd | ipv4 | 15:22 |
| donnyd | [SUM] 0.00-10.00 sec 8.21 GBytes 7.06 Gbits/sec 1168 sender | 15:22 |
| donnyd | [SUM] 0.00-10.00 sec 8.20 GBytes 7.04 Gbits/sec receiver | 15:22 |
| fungi | (unless you've got jumbo frames all the way through between the office and the guest or are doing transparent fragment reassembly somewhere) | 15:22 |
| donnyd | ping -s is don't fragment | 15:22 |
| fungi | um, -s is "size" | 15:23 |
| donnyd | :checks flags for ping | 15:23 |
| donnyd | ping6 -M do -s 1450 mirror01.regionone.fortnebula.opendev.org | 15:24 |
| donnyd | PING mirror01.regionone.fortnebula.opendev.org(2001:470:e045:2:f816:3eff:fe41:3c14 (2001:470:e045:2:f816:3eff:fe41:3c14)) 1450 data bytes | 15:24 |
| donnyd | 1458 bytes from 2001:470:e045:2:f816:3eff:fe41:3c14 (2001:470:e045:2:f816:3eff:fe41:3c14): icmp_seq=1 ttl=62 time=1.98 ms | 15:24 |
| donnyd | what is the mtu set in the VM? | 15:24 |
| donnyd | 1500 should work | 15:24 |
| fungi | ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 | 15:24 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: manager: specify report failure in logs https://review.opendev.org/671760 | 15:26 |
| fungi | donnyd: the ip header will take up 20 bytes and the icmp header another 8, so max payload size to keep the packet below 1500 is -s 1472 i think? | 15:27 |
| fungi | if the mtu is 1500 octets and you are able to send echos through with larger than -s 1472, fragmentation and reassembly are successfully occurring | 15:28 |
| *** e0ne has quit IRC | 15:29 | |
| fungi | or path mtu discovery is working and the request/reply are being split into multiple packets at the endpoints | 15:29 |
| donnyd | ping6 -M do -s 1452 mirror01.regionone.fortnebula.opendev.org | 15:30 |
| donnyd | PING mirror01.regionone.fortnebula.opendev.org(2001:470:e045:2:f816:3eff:fe41:3c14 (2001:470:e045:2:f816:3eff:fe41:3c14)) 1452 data bytes | 15:30 |
| donnyd | 1460 bytes from 2001:470:e045:2:f816:3eff:fe41:3c14 (2001:470:e045:2:f816:3eff:fe41:3c14): icmp_seq=1 ttl=62 time=1.23 ms | 15:30 |
| donnyd | 1460 bytes from 2001:470:e045:2:f816:3eff:fe41:3c14 (2001:470:e045:2:f816:3eff:fe41:3c14): icmp_seq=2 ttl=62 time=1.13 ms | 15:30 |
| *** zbr is now known as zbr|out | 15:30 | |
| donnyd | So probably not an mtu issue then | 15:30 |
| * clarkb pulls up PTG survey | 15:31 | |
| fungi | oh, or both ends have interfaces with an mtu of 1500 so they're both compatibly splitting request/reply into multiple packets based on their own mtus and there are no hops between them with a lower mtu than 1500 | 15:32 |
| *** ykarel|away has joined #openstack-infra | 15:33 | |
| donnyd | there should be no hops below 1500 | 15:34 |
| clarkb | ya I only mentioned it because its a semi common network bw problem with overlay networking | 15:34 |
| clarkb | particularly with neutron where there are hops with low mtus that don't have ip addrs so they can't tell the sender to fragment | 15:35 |
| clarkb | and so you just lose pakcets if they go over a certain size | 15:35 |
| fungi | but yeah, let's move on. doesn't seem related at all to mtus along that path | 15:35 |
| clarkb | My debug openstack networking list order is something like: ensure IP addrs are configured on host via glean/cloud-init, double check security groups, check MTUs, check routing | 15:35 |
| fungi | hrm. the fn mirror just went dark for me | 15:38 |
| fungi | ahh, over ipv6 but v4 is working fine | 15:38 |
| fungi | maybe my home ipv6 tunnel just broke | 15:38 |
| *** mattw4 has quit IRC | 15:39 | |
| fungi | hrm, no, i can reach other ipv6 addresses just fine | 15:39 |
| fungi | donnyd: did the fn ipv6 networking just die on us? | 15:40 |
| donnyd | Yea | 15:46 |
| donnyd | It wasn't mtu | 15:46 |
| fungi | okay, just making sure it was known | 15:46 |
| fungi | seems it came back just a moment ago | 15:46 |
| donnyd | v6 goes through my edge router, which i increased the mtu for... and made things quite unhappy | 15:47 |
| clarkb | the ethers escaped | 15:47 |
| donnyd | yea, my bad | 15:47 |
| fungi | i can imagine. probably not a good idea if the peer isn't also accepting higher mtus | 15:47 |
| fungi | insta-blackhole for larger frames | 15:48 |
| *** pcaruana has quit IRC | 15:49 | |
| donnyd | so things seem to be pretty standard, and likely not an mtu issue. However even if it was iperf3 is still showing quite a bit faster than what the clients are able to achieve | 15:50 |
| *** rpittau is now known as rpittau|afk | 15:50 | |
| donnyd | so maybe an openstack side client test? | 15:50 |
| clarkb | ya I think we should test iperf between tenant networks. via both public ipv6 and private ipv4 | 15:51 |
| clarkb | which takes us down to the routes entry on my debugging list :) | 15:52 |
| *** bhavikdbavishi1 has joined #openstack-infra | 15:53 | |
| *** bhavikdbavishi has quit IRC | 15:54 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 15:54 | |
| donnyd | ok I can run that from the zuul tenant | 15:56 |
| *** iurygregory has quit IRC | 15:57 | |
| *** gyee has joined #openstack-infra | 15:58 | |
| *** ginopc has quit IRC | 16:02 | |
| donnyd | is iperf still running? | 16:04 |
| fungi | i'll fire it back up | 16:06 |
| fungi | i lost my earlier ssh session in the mtu blackout | 16:06 |
| fungi | donnyd: running again | 16:07 |
| *** tosky has quit IRC | 16:08 | |
| *** bnemec is now known as beekneemech | 16:09 | |
| *** kmalloc is now known as needscoffee | 16:11 | |
| *** lucasagomes has quit IRC | 16:15 | |
| donnyd | ipv6 | 16:15 |
| donnyd | [ 4] 0.00-60.00 sec 20.4 GBytes 2.92 Gbits/sec 138 sender | 16:15 |
| donnyd | [ 4] 0.00-60.00 sec 20.4 GBytes 2.92 Gbits/sec receiver | 16:15 |
| donnyd | ipv4 | 16:15 |
| donnyd | [ 4] 0.00-10.00 sec 3.39 GBytes 2.91 Gbits/sec 50 sender | 16:15 |
| donnyd | [ 4] 0.00-10.00 sec 3.39 GBytes 2.91 Gbits/sec receiver | 16:15 |
| donnyd | single thread | 16:15 |
| mnaser | weird question but does anyone know how to checkout a branch from a local repo with ansible | 16:15 |
| mnaser | i.e. i want to checkout the zuul stable/stein of a repo locally (without defining a remote) | 16:16 |
| *** goldyfruit has quit IRC | 16:16 | |
| mnaser | shell: git checkout stable/stein args: chdir: src/opendev.org/openstack/openstack-ansible type of thing | 16:16 |
| donnyd | six threads | 16:16 |
| donnyd | [SUM] 0.00-10.00 sec 6.57 GBytes 5.64 Gbits/sec 735 sender | 16:16 |
| donnyd | [SUM] 0.00-10.00 sec 6.55 GBytes 5.63 Gbits/sec receiver | 16:16 |
| mnaser | "repo" seems to be required which is: git, SSH, or HTTP(S) protocol address of the git repository. | 16:16 |
| *** gtarnaras has quit IRC | 16:17 | |
| donnyd | could you do something like git:///var/code | 16:17 |
| mnaser | thats interesting | 16:17 |
| *** diablo_rojo has joined #openstack-infra | 16:18 | |
| jrosser | git clone file:///blah.git is legit too | 16:18 |
| *** ricolin_ has quit IRC | 16:19 | |
| *** gfidente has quit IRC | 16:19 | |
| clarkb | It may still try to clone from taht remote to wherever you are asking it to put it | 16:19 |
| clarkb | the git module in ansible is the one I run into with ansible lint the most | 16:20 |
| clarkb | beacuse it simply can't express 99% of what git is capable of | 16:20 |
| mnaser | file:/// did try to clone indeed | 16:20 |
| *** goldyfruit has joined #openstack-infra | 16:20 | |
| mnaser | cause you have to give a 'repo' and 'dest' | 16:20 |
| donnyd | nice | 16:21 |
| donnyd | and finally ipv4 with multiple threads | 16:22 |
| donnyd | [SUM] 0.00-10.00 sec 6.52 GBytes 5.60 Gbits/sec 370 sender | 16:22 |
| donnyd | [SUM] 0.00-10.00 sec 6.50 GBytes 5.59 Gbits/sec receiver | 16:22 |
| donnyd | So i am thinking it probably isn't network BW to the mirror | 16:22 |
| *** adriancz has quit IRC | 16:22 | |
| mordred | corvus: the gerrit 2.13 build issue is because it's using openjdk:6 which is wheezy which no longer exists. easy fix. the 2.15 one is more vexing, but I'll figure it out | 16:23 |
| mordred | corvus: there is a message which makes it look like it's targetting java7 instead of 8 for some reason. shrug. easy enough to reproduce locally though | 16:24 |
| clarkb | I have submitted PTG survey forms for opendev and gitea | 16:24 |
| corvus | mordred: sweet -- the 'run gerrit in a container' job passed, so i'm cleaning that up now so we can merge it (i need to make a new fake gerrit host so we don't actually run it in prod) | 16:24 |
| *** ramishra has quit IRC | 16:25 | |
| *** goldyfruit has quit IRC | 16:25 | |
| *** dtantsur is now known as dtantsur|afk | 16:25 | |
| corvus | mordred: that's using the upstream gerrit, so i think we merge my change for the job and your change for the build and then update the job to use our build | 16:26 |
| mordred | corvus: yes, I agree with that | 16:27 |
| mordred | the main difference between upstream and ours will be plugins, so I think it's worthwhile using the upstream images for testing until we have ours | 16:27 |
| *** roman_g has quit IRC | 16:29 | |
| *** beekneemech has quit IRC | 16:30 | |
| *** goldyfruit has joined #openstack-infra | 16:31 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 16:31 |
| clarkb | corvus: the xfce4-terminal paste warnings work on python too | 16:31 |
| *** gtarnaras has joined #openstack-infra | 16:32 | |
| *** gtarnaras has quit IRC | 16:33 | |
| *** gtarnaras has joined #openstack-infra | 16:34 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add gerrit to gitea job https://review.opendev.org/671162 | 16:35 |
| corvus | clarkb: i'm so looking forward to this. i assume you can turn them off with some xresources | 16:35 |
| donnyd | Is there a way to check performance on openafs? | 16:35 |
| *** goldyfruit has quit IRC | 16:35 | |
| *** bnemec has joined #openstack-infra | 16:35 | |
| clarkb | corvus: ya I haven't decided if I want to disable it yet. Its a good sanity check and I can probably live with one a day. But if it gets more persistent then I'll probably find out how to disable it | 16:36 |
| clarkb | donnyd: the easiest way would likely be fetching resources backed by afs on the mirror via http | 16:36 |
| clarkb | (that is also how jobs will access them so is more real world) | 16:36 |
| clarkb | oh you know what I wonder | 16:36 |
| donnyd | http://mirror.fortnebula.com/centos/7/os/x86_64/images/boot.iso | 16:37 |
| donnyd | this is big enough to get an idea | 16:37 |
| clarkb | the new afs mirrors don't have very large afs caches or didn't to start. I thought ianw was workign to fix that but then I disappeared for vacation or something | 16:37 |
| donnyd | however, idk how openafs works | 16:37 |
| clarkb | /afs:/var/cache/openafs:500000 is our cacheinfo which I think is still the samll value | 16:38 |
| clarkb | that is kb iirc | 16:38 |
| *** gtarnaras has quit IRC | 16:38 | |
| clarkb | corvus: ^ if I modify that file by hand (or maybe there is a command to run instead) how do I get afs to see the update? | 16:38 |
| corvus | clarkb: reboot i think | 16:39 |
| donnyd | here is what I get locally and this also reflects the speeds I am seeing in the logs | 16:39 |
| corvus | clarkb: https://review.opendev.org/669006 | 16:39 |
| clarkb | `fs setcachesize` may be a thing too | 16:39 |
| corvus | clarkb: that should work | 16:39 |
| *** chandankumar is now known as raukadah | 16:39 | |
| donnyd | % Total % Received % Xferd Average Speed Time Time Time Current | 16:39 |
| donnyd | Dload Upload Total Spent Left Speed | 16:39 |
| donnyd | 10 507M 10 55.5M 0 0 723k 0 0:11:58 0:01:18 0:10:40 697k | 16:39 |
| clarkb | corvus: thanks I'll udpate that change based on your suggestion | 16:40 |
| clarkb | donnyd: I think we should fix the cache problem first | 16:40 |
| donnyd | ok sounds good | 16:40 |
| clarkb | donnyd: then retest as this may be the issue (we aren't caching effectively so it is laways over the wire) | 16:40 |
| donnyd | thanks for helping to work out this seemingly final issue for FN | 16:40 |
| *** pkopec has quit IRC | 16:41 | |
| fungi | this would also explain why throughput from jobs sampled in two providers with the newer mirrors exhibited equivalent (slower) performance compared to older mirrors | 16:45 |
| fungi | mirror.rax.ord.opendev.org was ~ the same as mirror.regionone.fortnebula.opendev.org | 16:46 |
| fungi | er, mirror.ord.rax.opendev.org was i mean | 16:47 |
| *** diablo_rojo has quit IRC | 16:47 | |
| *** goldyfruit has joined #openstack-infra | 16:47 | |
| donnyd | Seems like rax-ord is having similar timeout issues | 16:48 |
| donnyd | node_provider:"rax-ord" AND filename:job-output.txt AND message:"RUN END RESULT_TIMED_OUT" | 16:48 |
| fungi | in that case the afs cache size may solve a bunch of these | 16:48 |
| fungi | er, afs cache size increase | 16:48 |
| fungi | presumably we're expiring entries much sooner, and so have many more cache misses | 16:49 |
| corvus | yeah, the system is more or less predicated on fitting the entire working set into cache | 16:49 |
| donnyd | makes sense | 16:49 |
| donnyd | the working cache part | 16:50 |
| fungi | especially since afs not super fast at fetching data from the backend (small udp datagrams crossing half the internet) | 16:50 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Set openafs cache sizes for mirror/mirror-update https://review.opendev.org/669006 | 16:52 |
| fungi | thanks clarkb! | 16:52 |
| clarkb | corvus: ^ I think Igot that right as long as the zuul/templates stuff is automatically populated? | 16:52 |
| clarkb | corvus: grep wasn't showing me anything explicitly applying those template files but maybe I wasn't grepping well enough | 16:53 |
| clarkb | oh just found it. new ps in a sec | 16:53 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Set openafs cache sizes for mirror/mirror-update https://review.opendev.org/669006 | 16:54 |
| clarkb | I think that should work now | 16:54 |
| *** tdasilva has quit IRC | 16:56 | |
| corvus | clarkb: one thing, but otherwise good | 16:56 |
| *** tdasilva has joined #openstack-infra | 16:56 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Set openafs cache sizes for mirror/mirror-update https://review.opendev.org/669006 | 16:56 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add tool to analyze check and gate success rates https://review.opendev.org/671777 | 16:59 |
| clarkb | and ^ is the tool I realized I never pushed anywhere. Figured it should at least get up to gerrit | 16:59 |
| *** dustinc_OSCON is now known as dustinc | 17:06 | |
| *** derekh has quit IRC | 17:10 | |
| *** igordc has joined #openstack-infra | 17:10 | |
| *** raissa has joined #openstack-infra | 17:12 | |
| *** raissa has quit IRC | 17:13 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Run a gerrit container (test only) https://review.opendev.org/630406 | 17:13 |
| *** tdasilva_ has joined #openstack-infra | 17:13 | |
| *** raissa has joined #openstack-infra | 17:14 | |
| corvus | clarkb, fungi: do we still need to do the whole ip address change notice if we change gerrit? | 17:14 |
| *** raissa has quit IRC | 17:14 | |
| corvus | i want to say that it shouldn't be a problem for contributors, but maybe for 3pci? so maybe we still try to give a heads up (2 weeks?), but not sweat it as much as we used to? | 17:14 |
| *** raissa has joined #openstack-infra | 17:14 | |
| *** raissa has quit IRC | 17:15 | |
| clarkb | fwiw I think some end users needed it too for corporate firewall to open 29418 (like IBM) | 17:15 |
| corvus | clarkb: but they can use http | 17:15 |
| clarkb | but in the intervening years our contributing orgs seem to have figured out how to network a bit better | 17:15 |
| clarkb | corvus: ya thats true | 17:15 |
| clarkb | I think it is less of a concern | 17:15 |
| *** tdasilva has quit IRC | 17:15 | |
| fungi | yeah, two weeks sounds like plenty these days | 17:16 |
| *** ykarel|away has quit IRC | 17:17 | |
| openstackgerrit | Jeremy Stanley proposed opendev/python-storyboardclient master: fix tox python3 overrides https://review.opendev.org/574347 | 17:17 |
| openstackgerrit | Merged openstack/project-config master: nova-lxd: noop jobs as part of project retirement https://review.opendev.org/670058 | 17:18 |
| openstackgerrit | Merged openstack/project-config master: End Airship Utils gating https://review.opendev.org/670923 | 17:18 |
| openstackgerrit | Merged openstack/project-config master: Add puppet-crane noop jobs https://review.opendev.org/671263 | 17:18 |
| *** tesseract has quit IRC | 17:24 | |
| *** jamesmcarthur has quit IRC | 17:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove airship/berth from infra https://review.opendev.org/671550 | 17:31 |
| *** mattw4 has joined #openstack-infra | 17:32 | |
| AJaeger | clarkb: I rebased to resolve merge conflict, want to +2A again, please? ^ | 17:32 |
| *** gtarnaras has joined #openstack-infra | 17:32 | |
| *** ralonsoh has quit IRC | 17:34 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Run a gerrit container (test only) https://review.opendev.org/630406 | 17:35 |
| *** rlandy is now known as rlandy|brb | 17:36 | |
| *** tdasilva_ has quit IRC | 17:37 | |
| *** tdasilva_ has joined #openstack-infra | 17:37 | |
| AJaeger | thanks, fungi for taking that | 17:37 |
| *** gtarnaras has quit IRC | 17:37 | |
| fungi | no problem | 17:37 |
| mnaser | question | 17:40 |
| mnaser | https://opendev.org/openstack/project-config/src/branch/master/playbooks/proposal/sync_openstack_ansible_common_files.sh -- this can't exist within our OSA repos right? because of the fact that it uses secrets (the proposal bot?) | 17:40 |
| AJaeger | mnaser: you could merge it in-tree with *new* secrets. | 17:41 |
| mnaser | ok so id have to create a new user called "the openstack-ansible proposal bot" | 17:41 |
| mnaser | or something | 17:41 |
| AJaeger | But we have all that infrastructure here, so why would you want to move it? | 17:41 |
| mnaser | AJaeger: just made a change to it and figured we can manage our changes rather than have infra deal with it :) | 17:41 |
| AJaeger | mnaser: it should work. But it needs other scripts as well, let me check the script... | 17:42 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: proposal: add sync for doc/* in openstack-ansible https://review.opendev.org/671782 | 17:42 |
| mnaser | AJaeger: ^ in the meantime :p | 17:42 |
| *** whoami-rajat has quit IRC | 17:43 | |
| AJaeger | mnaser: you can easily sync that over, no need for secrets. let me explain... | 17:44 |
| AJaeger | mnaser: https://opendev.org/openstack/project-config/src/branch/master/playbooks/proposal/propose_update.sh#L29 is what we do for openstack-manuals, here the script lives in the repo: https://opendev.org/openstack/openstack-manuals/src/branch/master/tools/sync-projects.sh | 17:45 |
| AJaeger | mnaser: so, change https://opendev.org/openstack/project-config/src/branch/master/playbooks/proposal/propose_update.sh#L64 to use your in-tree copy and maintain it there. | 17:46 |
| AJaeger | mnaser: but I think for security reasons we wanted to keep those simple and in project-config. so, might need some discussion. In the meantime, let me review your change ;) | 17:47 |
| *** gtarnaras has joined #openstack-infra | 17:48 | |
| *** gtarnaras has quit IRC | 17:48 | |
| *** udesale has quit IRC | 17:48 | |
| *** tosky has joined #openstack-infra | 17:49 | |
| openstackgerrit | Merged zuul/zuul-jobs master: upload-logs-swift: option for disabling the indexer https://review.opendev.org/663355 | 17:52 |
| clarkb | AJaeger: mnaser is there background on what that is doing? | 17:52 |
| *** eernst has joined #openstack-infra | 17:52 | |
| mnaser | clarkb, AJaeger: because we have a lot of roles in OSA, we have this little system that we do to maintain a single copy that is replicated to all of our other repos | 17:52 |
| mnaser | we use it for things like mq_setup and db_setup which is literally the exact tasks being used across all of our repos | 17:53 |
| mnaser | and AJaeger i guess we can eventually move onto using this https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/sync-test-repos.sh | 17:53 |
| clarkb | but why do zuul jobs care about vagrantfiles? | 17:53 |
| mnaser | i thought that was actually gonna take care of it but maybe there was a reason why it wasnt being used | 17:53 |
| clarkb | or .gitignore files? | 17:53 |
| mnaser | clarkb: we have a Vagrantfile at the root of every single role, so we want to keep it sync'd because when we add a new supported OS, we don't want to manually change that file in every single repo | 17:54 |
| clarkb | oh this is to update repos and propose changes back to gerrit? | 17:54 |
| mnaser | yes :) | 17:54 |
| AJaeger | clarkb: yes, it is. | 17:54 |
| clarkb | remember when we would copy python code around liek that? | 17:55 |
| clarkb | what is old is new again I guess | 17:55 |
| openstackgerrit | Merged openstack/project-config master: Remove airship/berth from infra https://review.opendev.org/671550 | 17:56 |
| fungi | ahh, oslo incubator, how we all miss you (or not) | 17:56 |
| AJaeger | and requirements sync... | 17:56 |
| AJaeger | https://review.opendev.org/#/c/671547/ is now ready to finish airship/utils retirement if I could get a second +2, please | 17:57 |
| fungi | that one's still a little more fresh in most people's memoryes | 17:57 |
| fungi | memories | 17:57 |
| mnaser | yeah it's pretty useful for us because we have a lot of common things across our roles that drift apart often | 17:57 |
| mnaser | and having some 30+ roles make it hard to keep them all up to date | 17:57 |
| AJaeger | mnaser: and no way to use a "common library" ;( | 17:57 |
| mnaser | yeah exactly :( | 17:57 |
| mnaser | heres to ansible collections eventually helping | 17:57 |
| fungi | right, we solved that in puppet modules by being able to centralize most of the need there | 17:58 |
| bnemec | I dunno, Go built a whole subsystem around copying code. Maybe we were onto something with incubator... | 17:58 |
| * bnemec barfs | 17:58 | |
| clarkb | bnemec: remember when go relied on git submodules? | 17:58 |
| mnaser | once this merges, is it possible to request rerunning a post job again to test and see if it works? | 17:58 |
| fungi | mnaser: yep, just point to a log for one you want rerun and i'm happy to do it | 17:59 |
| bnemec | clarkb: I do not. I've only had the dubious pleasure of working in Go recently. | 17:59 |
| *** rlandy|brb is now known as rlandy | 17:59 | |
| AJaeger | config-core, a few more reviews - most simple housekeeping - if you have time: https://review.opendev.org/671568 https://review.opendev.org/671412 https://review.opendev.org/671311 https://review.opendev.org/670195 https://review.opendev.org/670778 https://review.opendev.org/658439 , please | 17:59 |
| fungi | mnaser: to be clear, what we actually do is reenqueue the ref in the post pipeline, but basically the same result | 17:59 |
| mnaser | fungi: http://zuul.openstack.org/build/544dca35ca8249b08cc840b8bd82c488 it's not ready yet but in case you need to build your command if you have spare time :) | 18:00 |
| * mnaser looks at AJaeger reviews in the meantime | 18:00 | |
| AJaeger | thanks, mnaser | 18:00 |
| AJaeger | bbl | 18:00 |
| clarkb | hrm was tempest-full replaced globally? | 18:00 |
| *** thiago__ has joined #openstack-infra | 18:00 | |
| clarkb | re https://review.opendev.org/#/c/671568/1 because if that was only changed in neutron I think openstack has potentially opened itself to wedging the gate | 18:01 |
| *** thiago__ is now known as tdasilva | 18:01 | |
| fungi | mnaser: for reference, the command i will run is `sudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=post --project=openstack/openstack-ansible-tests --ref=refs/heads/master --newrev=cf7ee23b237c7235060800e02d55488095b84a3d` | 18:01 |
| clarkb | tempest-full-py3 seems to be what nova is running ... | 18:01 |
| clarkb | slaweq: ^ do you know what the plan around that is? | 18:01 |
| clarkb | does http://zuul.opendev.org/t/openstack/stream/ff5f02530b464b058da02b4a577cd1ff?logfile=console.log kill your browser like it kills mine? | 18:03 |
| *** tdasilva_ has quit IRC | 18:03 | |
| mnaser | mine is ok | 18:03 |
| clarkb | (I get the little wait for script or stop it yellow bar from FF and no text) | 18:03 |
| mnaser | using latest chromium on fedora | 18:03 |
| mnaser | jobs done now fwiw | 18:04 |
| mnaser | or in post-logs | 18:04 |
| clarkb | ya seems to work on chrome | 18:04 |
| *** bnemec is now known as beekneemech | 18:04 | |
| clarkb | so maybe we regressed FF | 18:04 |
| clarkb | mnaser: thanks for checking | 18:04 |
| openstackgerrit | Merged openstack/project-config master: proposal: add sync for doc/* in openstack-ansible https://review.opendev.org/671782 | 18:05 |
| mnaser | fungi: ^ ready to rerun that enqueue when/if you are :) | 18:06 |
| mnaser | (the openstack-ansible-tests change that is) | 18:06 |
| fungi | mnaser: enqueued now | 18:07 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove airship/utils from infrastructure https://review.opendev.org/671547 | 18:07 |
| AJaeger | rebased to resolve merge conflict, review welcome ^ | 18:07 |
| *** jamesmcarthur has joined #openstack-infra | 18:07 | |
| AJaeger | anybody reviewing subunit2sql? https://review.opendev.org/#/c/671412/ moves some jobs around... | 18:09 |
| *** raissa has joined #openstack-infra | 18:10 | |
| clarkb | AJaeger: we should maybe sync up with the qa team on tooling responsibilities. In the past mtreinish in particul was good about working betwene groups but I think we've largely lost that at this point | 18:10 |
| clarkb | devstack often languishes too | 18:10 |
| AJaeger | I see | 18:10 |
| *** ekultails has quit IRC | 18:12 | |
| AJaeger | clarkb: indeed, https://review.opendev.org/#/admin/groups/395,members looks like it could use some new members... | 18:12 |
| openstackgerrit | Jeremy Stanley proposed opendev/python-storyboardclient master: Correct the distname in PBR version discovery https://review.opendev.org/667455 | 18:13 |
| * AJaeger signs off and wishes everybody a great weekend | 18:14 | |
| fungi | thanks AJaeger, you too! | 18:14 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 18:14 |
| corvus | that should be ready now ^ | 18:14 |
| mnaser | fungi, AJaeger, clarkb: thanks, our proposal jobs just worked fine | 18:16 |
| fungi | awesome | 18:16 |
| openstackgerrit | Merged openstack/project-config master: Update Neutron grafana dashboard https://review.opendev.org/671568 | 18:18 |
| openstackgerrit | Merged openstack/project-config master: Add release jobs for ansible-role-collect-logs https://review.opendev.org/671311 | 18:18 |
| openstackgerrit | Merged openstack/project-config master: Add review priority for kayobe repos https://review.opendev.org/670778 | 18:18 |
| *** raissa has quit IRC | 18:20 | |
| *** raissa has joined #openstack-infra | 18:21 | |
| *** raissa has joined #openstack-infra | 18:22 | |
| *** raissa has quit IRC | 18:22 | |
| *** eernst has quit IRC | 18:22 | |
| *** factor has joined #openstack-infra | 18:27 | |
| openstackgerrit | Merged openstack/project-config master: Remove airship/utils from infrastructure https://review.opendev.org/671547 | 18:32 |
| clarkb | corvus: hrm http://logs.openstack.org/06/669006/4/gate/system-config-run-mirror/635e97b/job-output.txt.gz#_2019-07-19_18_28_23_783432 I'm not sure the templates version is overriding the system-config version of the group vars | 18:33 |
| clarkb | I wonder if playbooks/group_vars wins because it is adjacent to the playbook | 18:35 |
| clarkb | maybe I can address this by having a default value if unset then only set it in the tests? | 18:35 |
| *** goldyfruit has quit IRC | 18:35 | |
| * clarkb works on a new ps | 18:35 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add install-devstack test job https://review.opendev.org/670195 | 18:37 |
| openstackgerrit | Jeremy Stanley proposed opendev/python-storyboardclient master: fix tox python3 overrides https://review.opendev.org/574347 | 18:41 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Set openafs cache sizes for mirror/mirror-update https://review.opendev.org/669006 | 18:42 |
| clarkb | corvus: fungi ^ lets see if that does better | 18:42 |
| corvus | clarkb: that passed in check? | 18:43 |
| corvus | also, weird, i would have thought the inventory would have overriden the adjacent value | 18:44 |
| clarkb | corvus: I think it depends on the test node that gets assined because rax has the two devices that total >80GB disk but most other clouds have a single 80GB disk | 18:44 |
| clarkb | so /var/cache/afs or whatever the path is ends up on / in rax which is the samller 20GB or so disk iirc | 18:45 |
| fungi | unless we stick it in /opt and link it from /var/cache | 18:45 |
| fungi | in which case it's closer to 50 | 18:45 |
| clarkb | ya and even then it might be too small for 50GB cache | 18:45 |
| corvus | mordred: any chance you have either one of those image builds working? | 18:47 |
| corvus | mordred: maybe i can build 2.13 if i change 6 to 7? | 18:47 |
| corvus | (i'm to the point in my "add a job running gerrit in a container" where i need to change something in our built image, so i need to be able to build something (doesn't have to be any version in particular)) | 18:48 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 18:53 |
| *** Lucas_Gray has joined #openstack-infra | 18:54 | |
| *** eharney has quit IRC | 18:57 | |
| *** jamesmcarthur has quit IRC | 18:58 | |
| *** Lucas_Gray has quit IRC | 18:59 | |
| *** Lucas_Gray has joined #openstack-infra | 19:00 | |
| *** jamesmcarthur has joined #openstack-infra | 19:01 | |
| clarkb | http://logs.openstack.org/06/669006/5/check/system-config-run-mirror-update/51fd523/job-output.txt.gz#_2019-07-19_18_56_21_014926 do the variables need to be differently named? | 19:02 |
| *** goldyfruit has joined #openstack-infra | 19:02 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Set openafs cache sizes for mirror/mirror-update https://review.opendev.org/669006 | 19:04 |
| clarkb | that actually makes a lot of sense if we end up changing afs implementations later too | 19:04 |
| clarkb | we can set general afs settings at top level then modify as necessary for implementation specific details | 19:05 |
| *** Lucas_Gray has quit IRC | 19:05 | |
| *** Lucas_Gray has joined #openstack-infra | 19:06 | |
| *** jamesmcarthur has quit IRC | 19:07 | |
| fungi | okay, need to go grab a late lunch and run some errands. bbiaw | 19:08 |
| fungi | and yeah, the method for setting options in 669006 seems fine to me | 19:09 |
| *** Lucas_Gray has quit IRC | 19:10 | |
| *** Lucas_Gray has joined #openstack-infra | 19:11 | |
| clarkb | wow I had lunch before fungi | 19:11 |
| fungi | it happens | 19:11 |
| clarkb | though you might consider that breakfast since I sort of skipped that meal | 19:11 |
| fungi | maybe i'll call it "early dinner" instead | 19:11 |
| fungi | i did also skip breakfast, so... i'll call it "food" | 19:12 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: configure-pypi-mirrors: add initial role https://review.opendev.org/581165 | 19:14 |
| *** roman_g has joined #openstack-infra | 19:18 | |
| *** psachin has quit IRC | 19:20 | |
| *** eernst has joined #openstack-infra | 19:22 | |
| *** harlowja has quit IRC | 19:24 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: add host-key-checking toggle https://review.opendev.org/653679 | 19:26 |
| clarkb | corvus: 669006 is ready for rereview now. It has passed the two mirror jobs and got a +1 | 19:33 |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: add host-key-checking toggle https://review.opendev.org/653679 | 19:34 |
| *** harlowja has joined #openstack-infra | 19:40 | |
| *** jamesmcarthur has joined #openstack-infra | 19:42 | |
| *** eharney has joined #openstack-infra | 19:42 | |
| *** jtomasek has quit IRC | 19:45 | |
| *** eernst has quit IRC | 19:46 | |
| *** jamesmcarthur has quit IRC | 19:48 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: configure-pypi-mirrors: add initial role https://review.opendev.org/581165 | 19:50 |
| *** michael-beaver has joined #openstack-infra | 19:54 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Remove opendev-promote-javascript-content https://review.opendev.org/671851 | 19:54 |
| openstackgerrit | James E. Blair proposed opendev/puppet-zuul master: Update to new zuul JS content url https://review.opendev.org/671852 | 19:56 |
| corvus | clarkb: ^ if we need to make any changes, we're gonna want that in place :) | 19:56 |
| clarkb | looking | 19:57 |
| *** betherly has joined #openstack-infra | 20:00 | |
| *** betherly has quit IRC | 20:05 | |
| *** tdasilva has quit IRC | 20:06 | |
| *** tdasilva has joined #openstack-infra | 20:06 | |
| donnyd | fyi something is going on with my edge fw, looking now for root cause now | 20:07 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: configure-pypi-mirrors: add initial role https://review.opendev.org/581165 | 20:08 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Update xterm to >= 3.14.5 https://review.opendev.org/671858 | 20:09 |
| *** Lucas_Gray has quit IRC | 20:10 | |
| clarkb | corvus: can you take a look at https://review.opendev.org/#/c/669006/6 one more time? | 20:11 |
| *** Lucas_Gray has joined #openstack-infra | 20:12 | |
| corvus | +3 | 20:12 |
| clarkb | tyty | 20:12 |
| *** Lucas_Gray has quit IRC | 20:15 | |
| *** yamamoto has joined #openstack-infra | 20:17 | |
| *** jamesmcarthur has joined #openstack-infra | 20:20 | |
| *** jamesmcarthur has quit IRC | 20:26 | |
| *** ociuhandu has joined #openstack-infra | 20:28 | |
| *** Lucas_Gray has joined #openstack-infra | 20:29 | |
| mordred | corvus: sorry - not yet - almost though - patch coming (there are more errors than just the 8 thing | 20:32 |
| corvus | mordred: cool, thx | 20:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build a docker image of the existing gerrit 2.13 https://review.opendev.org/671457 | 20:34 |
| mordred | corvus: ^^ the 2.13 should build now | 20:34 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Update xterm to >= 3.14.5 https://review.opendev.org/671858 | 20:36 |
| *** Lucas_Gray has quit IRC | 20:40 | |
| openstackgerrit | Merged zuul/nodepool master: Remove opendev-promote-javascript-content https://review.opendev.org/671851 | 20:47 |
| *** ociuhandu has quit IRC | 20:50 | |
| *** e0ne has joined #openstack-infra | 20:51 | |
| *** ociuhandu has joined #openstack-infra | 20:51 | |
| *** ociuhandu has quit IRC | 20:53 | |
| *** ociuhandu_ has joined #openstack-infra | 20:53 | |
| *** ociuhandu_ has quit IRC | 20:54 | |
| *** ociuhandu has joined #openstack-infra | 20:54 | |
| *** ociuhandu has quit IRC | 20:55 | |
| *** jamesmcarthur has joined #openstack-infra | 20:55 | |
| *** ociuhandu has joined #openstack-infra | 20:55 | |
| *** ociuhandu has quit IRC | 20:56 | |
| *** ociuhandu_ has joined #openstack-infra | 20:56 | |
| *** e0ne has quit IRC | 20:57 | |
| *** jamesmcarthur has quit IRC | 21:00 | |
| *** betherly has joined #openstack-infra | 21:00 | |
| openstackgerrit | Merged opendev/system-config master: Set openafs cache sizes for mirror/mirror-update https://review.opendev.org/669006 | 21:01 |
| *** ociuhandu_ has quit IRC | 21:01 | |
| donnyd | ok its fixed | 21:02 |
| *** betherly has quit IRC | 21:05 | |
| clarkb | donnyd: and change to update afs cache sizes has merged. I'll reboot the mirror when that gets applied | 21:10 |
| donnyd | Awesome | 21:12 |
| donnyd | hopefully it fixes the issue | 21:12 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Update xterm to >= 3.14.5 https://review.opendev.org/671858 | 21:16 |
| *** yamamoto has quit IRC | 21:18 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Display artifacts in build page https://review.opendev.org/671865 | 21:21 |
| *** EvilienM is now known as EmilienM | 21:30 | |
| *** _erlon_ has quit IRC | 21:30 | |
| *** betherly has joined #openstack-infra | 21:31 | |
| *** jamesmcarthur has joined #openstack-infra | 21:32 | |
| *** betherly has quit IRC | 21:35 | |
| *** jamesmcarthur has quit IRC | 21:36 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add mirror_info documentation https://review.opendev.org/669948 | 21:43 |
| *** tosky has quit IRC | 21:43 | |
| clarkb | I'm going to reboot the fortnebula mirror now. It has updated afs cache settings | 21:54 |
| corvus | clarkb: do you understand the failure of https://review.opendev.org/671852 in http://logs.openstack.org/52/671852/1/check/puppet-beaker-rspec-puppet-4-infra/f62192e/job-output.txt.gz ? | 21:57 |
| clarkb | looking | 21:57 |
| corvus | http://logs.openstack.org/52/671852/1/check/puppet-beaker-rspec-puppet-4-infra/f62192e/job-output.txt.gz#_2019-07-19_20_21_37_707034 | 21:57 |
| corvus | but, like, what's not idempotent? | 21:58 |
| corvus | is it due to install-ara? | 21:58 |
| donnyd | clarkb: how long does it take for the cache to build? | 22:00 |
| fungi | corvus: still looking but there are some nonzero exit codes earlier in that log as well | 22:00 |
| clarkb | corvus: ya i think it installed ara again therefore failed the idempotency check | 22:00 |
| openstackgerrit | James E. Blair proposed opendev/puppet-zuul master: Update to new zuul JS content url https://review.opendev.org/671852 | 22:01 |
| clarkb | fungi: remember that puppet exits non zero to indicate different types of success | 22:01 |
| clarkb | in this case it returns 2 both times to indicate it ran successfully but made changes | 22:01 |
| fungi | ahh, i see 2 and 127... | 22:01 |
| corvus | donnyd: first build of any job should warm the cache. so by the second build you should have good numbers. | 22:01 |
| fungi | also some exiting 1 | 22:01 |
| fungi | this was the 127: http://logs.openstack.org/52/671852/1/check/puppet-beaker-rspec-puppet-4-infra/f62192e/job-output.txt.gz#_2019-07-19_20_17_44_326517 | 22:02 |
| clarkb | corvus: donnyd ya it should warm up quickly if under use | 22:02 |
| donnyd | i just ran a quick check | 22:02 |
| corvus | donnyd: hard to say overall though, since different jobs use different packages; the cache populates one file at a time as it's used. | 22:02 |
| donnyd | donny@office:~> curl -o /dev/null http://mirror.fortnebula.com/centos/7/os/x86_64/images/efiboot.img | 22:02 |
| donnyd | % Total % Received % Xferd Average Speed Time Time Time Current | 22:02 |
| donnyd | Dload Upload Total Spent Left Speed | 22:02 |
| donnyd | 100 8704k 100 8704k 0 0 917k 0 0:00:09 0:00:09 --:--:-- 1220k | 22:02 |
| donnyd | donny@office:~> curl -o /dev/null http://mirror.fortnebula.com/centos/7/os/x86_64/images/efiboot.img | 22:02 |
| donnyd | % Total % Received % Xferd Average Speed Time Time Time Current | 22:02 |
| donnyd | Dload Upload Total Spent Left Speed | 22:02 |
| donnyd | 100 8704k 100 8704k 0 0 229M 0 --:--:-- --:--:-- --:--:-- 229M | 22:02 |
| corvus | yeah, that's what i'd expect | 22:02 |
| clarkb | that seems good | 22:03 |
| donnyd | :) | 22:03 |
| donnyd | much mo better | 22:03 |
| clarkb | ifnra-root should I quickly reboot the other opendev openafs mirrors? | 22:03 |
| corvus | (there will periodically be cache invalidations when the mirrors are updated, so if you see an anomoly, it may be that) | 22:03 |
| corvus | clarkb: can we just use fs setcache on those? | 22:03 |
| clarkb | corvus: maybe? you would know better than me | 22:04 |
| corvus | ie, have it on the filesystem for future reboots, but fs setcache to update currently running mirrors without downtime | 22:04 |
| clarkb | k I'll read some manpages | 22:04 |
| corvus | clarkb: i think running that (as root) should be fine. it's just not going to persist. and we have to make sure we have the right number of 0's in the command. :) | 22:05 |
| corvus | so as long as the values are correctly copy-pasted from the ones on disk, we're great :) | 22:05 |
| clarkb | I'll copy the zeros off the number from disk | 22:06 |
| *** needssleep is now known as TheJulia | 22:11 | |
| clarkb | the manpage says if you apss it -blocks 0 it will use the value set in the cacheinfo file. This is not true. It uses a cached value for what it thinks is in the cacheinfo file | 22:12 |
| clarkb | I've set it with fs setcachesize -blocks 50000000 | 22:12 |
| clarkb | and confirmed via fs getcacheparms | 22:12 |
| clarkb | all three opendev mirrors report the correct cache size now | 22:14 |
| jonher | \o/ | 22:15 |
| clarkb | and fn mirror is now using more cache blocks than it had previously | 22:19 |
| donnyd | http://logs.openstack.org/03/670203/9/check/neutron-fullstack-with-uwsgi/b8fe673/job-output.txt#_2019-07-19_21_09_37_174552 | 22:19 |
| donnyd | http://logs.openstack.org/03/670203/9/check/neutron-fullstack-with-uwsgi/b8fe673/job-output.txt#_2019-07-19_21_07_18_217839 | 22:19 |
| clarkb | 633072 blocks in use | 22:19 |
| donnyd | Yea, there is a pretty sizable difference | 22:19 |
| donnyd | thats more along the lines of what something local should be doing | 22:20 |
| *** eharney has quit IRC | 22:20 | |
| clarkb | and now up to 711743 | 22:21 |
| clarkb | so ya seems to be a good improvement so far | 22:21 |
| *** betherly has joined #openstack-infra | 22:22 | |
| *** betherly has quit IRC | 22:27 | |
| clarkb | donnyd: do you still have the timed out logstash query? I can't seem to find my tab | 22:27 |
| donnyd | node_provider:"fortnebula-regionone" AND filename:job-output.txt AND message:"RUN END RESULT_TIMED_OUT" | 22:28 |
| clarkb | thanks | 22:28 |
| donnyd | edge fw still seems to be acting funny | 22:32 |
| *** gyee has quit IRC | 22:34 | |
| fungi | funny "ha ha" or funny "uh oh"? | 22:34 |
| donnyd | Headed out to dinner | 22:36 |
| donnyd | #2 fungi | 22:36 |
| fungi | :/ | 22:36 |
| fungi | well, seems stable enough | 22:36 |
| donnyd | Like I'm going to need to reboot because there is a parameter that can only be changed at boot | 22:37 |
| fungi | have a good evening/weekend! | 22:37 |
| donnyd | seems to be working(ish) enough | 22:37 |
| donnyd | I will get it tuned up this weekend | 22:37 |
| *** rascasoft has quit IRC | 22:40 | |
| *** rascasoft has joined #openstack-infra | 22:42 | |
| *** betherly has joined #openstack-infra | 22:53 | |
| clarkb | no OOMs since swapfiles went in. Just double checking that again before I call it a week | 22:55 |
| *** gyee has joined #openstack-infra | 22:57 | |
| *** betherly has quit IRC | 22:58 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Display artifacts in build page https://review.opendev.org/671865 | 23:00 |
| *** rlandy has quit IRC | 23:07 | |
| *** mattw4 has quit IRC | 23:08 | |
| *** jamesmcarthur has joined #openstack-infra | 23:08 | |
| *** sgw has quit IRC | 23:14 | |
| *** betherly has joined #openstack-infra | 23:14 | |
| *** betherly has quit IRC | 23:19 | |
| *** jamesmcarthur has quit IRC | 23:22 | |
| *** Lucas_Gray has joined #openstack-infra | 23:22 | |
| *** jamesmcarthur has joined #openstack-infra | 23:23 | |
| *** michael-beaver has quit IRC | 23:23 | |
| *** jamesmcarthur has quit IRC | 23:28 | |
| *** takamatsu has quit IRC | 23:31 | |
| *** betherly has joined #openstack-infra | 23:35 | |
| *** betherly has quit IRC | 23:39 | |
| *** aaronsheffield has quit IRC | 23:44 | |
| *** factor has quit IRC | 23:46 | |
| *** factor has joined #openstack-infra | 23:47 | |
| *** igordc has quit IRC | 23:48 | |
| *** jamesmcarthur has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!