| *** fuentess has quit IRC | 00:00 | |
| clarkb | mordred: last thing to check is that sdk is new enough on bridge and nodepool hosts | 00:00 |
|---|---|---|
| mordred | clarkb: what fo? | 00:00 |
| mordred | for? | 00:00 |
| clarkb | mordred: the addition of the string interpolation? | 00:00 |
| clarkb | so that oscc loads it up properly? that is part of sdk now right? | 00:01 |
| mordred | oh golly - string interp is *super* old | 00:01 |
| clarkb | oh huh | 00:01 |
| tonyb | ianw: Oh rats :( but also \o/ | 00:01 |
| mordred | conoha has been doing that with their regions since the tokyo summit | 00:01 |
| mordred | clarkb: the only issue with it was that I accepted a patch using % style when .format() is what we use | 00:01 |
| *** sthussey has quit IRC | 00:04 | |
| *** rcernin has joined #openstack-infra | 00:11 | |
| *** jamesmcarthur has joined #openstack-infra | 00:11 | |
| *** rcernin_ has quit IRC | 00:12 | |
| *** rcernin has quit IRC | 00:13 | |
| *** rcernin has joined #openstack-infra | 00:14 | |
| *** jamesmcarthur has quit IRC | 00:16 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix zk cluster members listing https://review.openstack.org/612535 | 00:19 |
| *** tobiash has quit IRC | 00:21 | |
| *** ssbarnea has joined #openstack-infra | 00:22 | |
| *** tobiash has joined #openstack-infra | 00:23 | |
| *** jamesmcarthur has joined #openstack-infra | 00:30 | |
| *** longkb has joined #openstack-infra | 00:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Consume rate limiting task manager from openstacksdk https://review.openstack.org/612169 | 00:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove task manager https://review.openstack.org/612170 | 00:46 |
| clarkb | ok I've removed zk* from the emergency file. Will check that we still have a proper cluster after puppet runs then call it a day | 00:47 |
| *** gouthamr has joined #openstack-infra | 00:47 | |
| clarkb | Thnking about doing the switchover maybe put builders on new zk late wednesday my time, then move zuul and the launchers on friday my time sometime? | 00:48 |
| clarkb | I've got dentist appointment wednesday but otherwise I can be quite flexible this week Ithink | 00:48 |
| dmsimard | I love that any job I'm troubleshooting has ara enabled.. no matter the project | 00:50 |
| dmsimard | :D | 00:50 |
| *** dmellado has joined #openstack-infra | 00:51 | |
| dmsimard | even when developing ara, ara helps me troubleshooting the ara integration jobs | 00:53 |
| dmsimard | but then I need to generate a nested ara report, it gets confusing | 00:53 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add support for enabling the ARA callback plugin in install-ansible https://review.openstack.org/611228 | 00:56 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add playbook for deploying the ARA web application https://review.openstack.org/611232 | 00:57 |
| corvus | clarkb: i'm around all this week (on east-coast time, believe it or not since i'm talking to you now) except friday i'm afk | 00:58 |
| mordred | corvus: wow. I *knew* you were east coast, but have also been calculating west coast timezone when deciding whether or not you're likely to respond | 00:58 |
| clarkb | corvus: in that case maybe we switch builders tomorrow then do the launchers and zuul on thursday? | 01:00 |
| clarkb | mostly I want ~36 hours for image builds to happen | 01:00 |
| *** stevebaker has joined #openstack-infra | 01:06 | |
| *** xinliang has joined #openstack-infra | 01:10 | |
| Shrews | corvus: what's happening on the east coast? | 01:17 |
| dmsimard | it's cold | 01:17 |
| clarkb | ATO? | 01:19 |
| *** smarcet has joined #openstack-infra | 01:20 | |
| Shrews | clarkb: if you want to cutover builders tomorrow, i can help monitor then | 01:21 |
| clarkb | Shrews: does after the infra meeting work for you? 1pm pacific or 4pm eastern? | 01:21 |
| *** mrsoul has quit IRC | 01:21 | |
| Shrews | clarkb: wfm | 01:21 |
| Shrews | we should know pretty quickly if there are issues | 01:22 |
| clarkb | cool. Puppet ran on new cluster and it is still a cluster | 01:22 |
| clarkb | so I think we are ready to switch the builders over whenever. I'm going to call it a day now and see you all tomorrow | 01:22 |
| *** imacdonn has quit IRC | 01:23 | |
| *** imacdonn has joined #openstack-infra | 01:23 | |
| *** markvoelker has joined #openstack-infra | 01:25 | |
| *** rlandy has quit IRC | 01:27 | |
| *** hongbin has joined #openstack-infra | 01:27 | |
| *** carl_cai has quit IRC | 01:35 | |
| *** hongbin has quit IRC | 01:36 | |
| *** hongbin has joined #openstack-infra | 01:37 | |
| *** jamesmcarthur has quit IRC | 01:41 | |
| *** hongbin_ has joined #openstack-infra | 01:41 | |
| *** hongbin has quit IRC | 01:43 | |
| *** bhavikdbavishi has joined #openstack-infra | 01:43 | |
| *** bhavikdbavishi has quit IRC | 01:50 | |
| *** jamesmcarthur has joined #openstack-infra | 02:00 | |
| *** hongbin has joined #openstack-infra | 02:05 | |
| *** jamesmcarthur has quit IRC | 02:05 | |
| *** hongbin_ has quit IRC | 02:07 | |
| *** jamesmcarthur has joined #openstack-infra | 02:18 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:23 | |
| *** jamesmcarthur has quit IRC | 02:34 | |
| *** graphene has joined #openstack-infra | 02:38 | |
| *** jamesmcarthur has joined #openstack-infra | 02:39 | |
| *** bhavikdbavishi has quit IRC | 02:40 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Migration to PHP 7.x https://review.openstack.org/611936 | 02:51 |
| *** psachin has joined #openstack-infra | 02:56 | |
| *** dave-mccowan has quit IRC | 02:57 | |
| *** jamesmcarthur has quit IRC | 02:59 | |
| *** munimeha1 has quit IRC | 03:01 | |
| *** smarcet has quit IRC | 03:09 | |
| *** ramishra has joined #openstack-infra | 03:12 | |
| *** graphene has quit IRC | 03:23 | |
| *** graphene has joined #openstack-infra | 03:25 | |
| *** graphene has quit IRC | 03:25 | |
| *** graphene has joined #openstack-infra | 03:27 | |
| *** graphene has quit IRC | 03:33 | |
| *** graphene has joined #openstack-infra | 03:35 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:35 | |
| *** hongbin has quit IRC | 03:52 | |
| *** graphene has quit IRC | 03:53 | |
| *** graphene has joined #openstack-infra | 03:54 | |
| *** udesale has joined #openstack-infra | 03:58 | |
| *** mrhillsman is now known as openlab | 04:05 | |
| *** openlab is now known as mrhillsman | 04:06 | |
| *** smarcet has joined #openstack-infra | 04:09 | |
| *** roman_g has quit IRC | 04:10 | |
| *** yamamoto has quit IRC | 04:17 | |
| *** yamamoto has joined #openstack-infra | 04:17 | |
| *** jamesmcarthur has joined #openstack-infra | 04:18 | |
| *** jamesmcarthur has quit IRC | 04:22 | |
| openstackgerrit | zhulingjie proposed openstack/ansible-role-cloud-launcher master: use include_tasks instead of include https://review.openstack.org/612570 | 04:40 |
| *** smarcet has quit IRC | 04:45 | |
| *** janki has joined #openstack-infra | 04:59 | |
| *** quiquell|off is now known as quiquell | 05:40 | |
| *** kjackal has joined #openstack-infra | 05:40 | |
| *** spsurya has joined #openstack-infra | 05:48 | |
| *** maciejjozefczyk has quit IRC | 06:09 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: New Repo: OpenStack-Helm Docs https://review.openstack.org/611893 | 06:15 |
| AJaeger | config-core, a couple of new repos reviews are up: https://review.openstack.org/612419 https://review.openstack.org/602783 https://review.openstack.org/609531 https://review.openstack.org/611892 https://review.openstack.org/#/c/611893/ | 06:18 |
| *** ccamacho has joined #openstack-infra | 06:31 | |
| *** kjackal has quit IRC | 06:35 | |
| *** ramishra_ has joined #openstack-infra | 06:40 | |
| *** ramishra has quit IRC | 06:43 | |
| *** slaweq has joined #openstack-infra | 06:43 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 06:53 | |
| *** bhavikdbavishi has quit IRC | 06:55 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:55 | |
| *** pcaruana has joined #openstack-infra | 06:56 | |
| *** jaosorior has quit IRC | 07:02 | |
| *** graphene has quit IRC | 07:04 | |
| *** armax has quit IRC | 07:04 | |
| *** felipemonteiro has joined #openstack-infra | 07:04 | |
| *** jaosorior has joined #openstack-infra | 07:05 | |
| *** graphene has joined #openstack-infra | 07:05 | |
| *** maciejjozefczyk has joined #openstack-infra | 07:05 | |
| amorin | hey guys | 07:06 |
| amorin | do you know if those mountains on graphs | 07:06 |
| *** rcernin has quit IRC | 07:06 | |
| amorin | http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1&var-region=ovh-bhs1&var-region=ovh-gra1&from=1540230724666&to=1540262142848 | 07:06 |
| amorin | for gra1 | 07:06 |
| amorin | is it normal behavior? | 07:07 |
| *** graphene has quit IRC | 07:07 | |
| *** rpittau has quit IRC | 07:07 | |
| *** rpittau has joined #openstack-infra | 07:07 | |
| *** shardy has joined #openstack-infra | 07:19 | |
| *** felipemonteiro has quit IRC | 07:22 | |
| *** gfidente has joined #openstack-infra | 07:24 | |
| cgoncalves | ianw, hey. we're seeing another issue with centos DIB: http://logs.openstack.org/79/604479/13/check/octavia-v2-dsvm-scenario-centos-7/615561f/controller/logs/devstacklog.txt.gz#_2018-10-23_01_16_32_211 | 07:27 |
| cgoncalves | ianw, shouldn't it be using the epel mirror from openstack ci? | 07:27 |
| cgoncalves | http://logs.openstack.org/79/604479/13/check/octavia-v2-dsvm-scenario-centos-7/615561f/controller/logs/devstacklog.txt.gz#_2018-10-23_01_13_20_241 | 07:28 |
| ianw | amorin: hey ... it is not normal :) | 07:33 |
| *** ramishra_ is now known as ramishra | 07:33 | |
| ianw | amorin: it seems launches on gra1 are basically always failing, i discussed with clarkb this morning turning it off | 07:34 |
| *** jamesmcarthur has joined #openstack-infra | 07:34 | |
| ianw | amorin: i can jump in to get some id's if it will help | 07:34 |
| ianw | cgoncalves: hrmm, let me see | 07:35 |
| *** aojea has joined #openstack-infra | 07:37 | |
| ianw | cgoncalves: can you make dib run by default with "-x" in this test? also, we have a --logfile argument now, which might be useful to save the output into a separate log | 07:37 |
| *** jamesmcarthur has quit IRC | 07:38 | |
| *** sshnaidm|afk is now known as sshnaidm|pto | 07:41 | |
| cgoncalves | ianw, oh, that would be super useful for debugging purposes indeed! | 07:52 |
| ianw | cgoncalves: ok, as step 1 i'll try out -> https://review.openstack.org/612622 :) | 07:52 |
| cgoncalves | ianw, do we have the centos cloud image mirrored somewhere? | 07:52 |
| ianw | cgoncalves: no, we don't mirror those | 07:53 |
| ianw | it might be something for our reverse proxy | 07:53 |
| *** lastmikoi has joined #openstack-infra | 07:54 | |
| *** xek has joined #openstack-infra | 07:55 | |
| quiquell | Good morning, we are using fedora28 nodeset and looks like dnf.conf exclude the python virtualenv package | 07:56 |
| quiquell | http://logs.openstack.org/90/612290/8/check/tripleo-ci-fedora-28-standalone/cd876e8/logs/undercloud/etc/dnf/dnf.conf.txt.gz | 07:56 |
| ianw | quiquell: yes, this is by design | 07:56 |
| quiquell | ianw: Do you know if this is the default or maybe we are changing something | 07:56 |
| quiquell | ianw: I have test it laso with Fedora-Cloud-Base-28-1.1.x86_64 and is different | 07:57 |
| quiquell | ianw: This is fedora's design or openstack infra stuff ? | 07:57 |
| *** tosky has joined #openstack-infra | 07:57 | |
| cgoncalves | ianw, that patch was freaky quick! thanks! | 07:58 |
| quiquell | ianw: centos was working fine | 07:58 |
| ianw | quiquell: no, this is an infra thing; you can read all about it @ https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#n47 | 07:58 |
| quiquell | ianw: the exclusion is not needed there ? | 07:58 |
| ianw | quiquell: umm, i would have said it was excluded there too, of the top of my head | 07:58 |
| ianw | quiquell: the problem, iirc, was that if a new setuptools/pip package appears, but it still less than the upstream versions we've installed, it could get itself into a big mess | 08:00 |
| quiquell | ianw: At centos we install the python-virtualenv package http://logs.openstack.org/90/612290/6/check/tripleo-ci-centos-7-standalone/4e58975/job-output.txt.gz#_2018-10-22_13_59_07_727009 | 08:00 |
| *** xek_ has joined #openstack-infra | 08:01 | |
| ianw | quiquell: is it possible that yum v dnf just ignores the held package? | 08:01 |
| quiquell | ianw: Don't know I am just approaching this | 08:03 |
| *** xek has quit IRC | 08:03 | |
| ianw | quiquell: given that more recent fedora's i think do a much better job at separating user-installed v packaged tools, the whole thing might be able to be reworked | 08:04 |
| ianw | i'd volunteer to review that, but i don't know about work on it :) i'm not making any excuses that that pip-and-virtualenv element is pretty messy | 08:04 |
| ianw | it's just grown around practicalities, but things do tend to change | 08:05 |
| quiquell | ianw: Nah don't worry is totally ok, we are just hacking around to have our new fedora28 job working | 08:05 |
| quiquell | ianw: So what's the correct way to use virtualenv at fedora28 from nodesets ? | 08:06 |
| ianw | quiquell: i would say "virtualenv -p python3" | 08:08 |
| quiquell | ianw: so pip and virtualenv for python3 is already installed ? | 08:09 |
| *** jpich has joined #openstack-infra | 08:10 | |
| * quiquell have being too lazy not checking it | 08:10 | |
| ianw | quiquell: yes, that's what all the fussing is in about in pip-and-virtualenv package :) | 08:11 |
| quiquell | ianw: Yep, ok thanks | 08:12 |
| quiquell | ianw: So we are suppose to have it at centos too ? | 08:12 |
| ianw | cgoncalves: hrm so it really seems like it runs h05-rpm-epel-release http://logs.openstack.org/79/604479/13/check/octavia-v2-dsvm-scenario-centos-7/615561f/controller/logs/devstacklog.txt.gz#_2018-10-23_01_13_19_913 | 08:12 |
| ianw | quiquell: yep, all platforms should have a working, latest version of "pip" and "virtualenv" | 08:13 |
| quiquell | ianw: ack thanks so much | 08:13 |
| ianw | np, i hope it helps more than hinders :) | 08:15 |
| quiquell | ianw: sure it will do the job | 08:15 |
| cgoncalves | ianw, shouldn't it? it is a dep of pip-and-virtualenv | 08:16 |
| *** shardy has quit IRC | 08:17 | |
| ianw | cgoncalves: yeah it should ... without "-x" it doesn't show exactly what it did ... but in theory it should have re-written the epel repo to the mirror ... | 08:18 |
| *** ginopc has quit IRC | 08:20 | |
| *** ginopc has joined #openstack-infra | 08:21 | |
| amorin | ianw: the weird thing is that it seems to be working at the beginning of hours | 08:21 |
| *** eernst has joined #openstack-infra | 08:21 | |
| *** dtantsur|afk is now known as dtantsur | 08:22 | |
| amorin | http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1&var-region=ovh-bhs1&var-region=ovh-gra1&from=1540245245361&to=1540248719132 | 08:22 |
| amorin | why does it stop after minute 35 | 08:22 |
| ianw | amorin: i'm running the port cleaning script every 20 minutes, and it clears out some 600+ ports | 08:22 |
| *** e0ne has joined #openstack-infra | 08:23 | |
| ianw | i wonder if there's a background of very high frequency of failure, but a few get through when the ports are cleared back to zero? | 08:24 |
| ianw | hrm, not sure that makes sense, because nodepool is asynchronous to the cleaning script. it's not like it pauses and waits for all free ports | 08:24 |
| amorin | nodepool is supposed to spawn all instances, right? | 08:25 |
| ianw | amorin: umm, yes, nothing else is creating vm's if that's what you mean | 08:26 |
| amorin | ok | 08:26 |
| amorin | I mean the available line, should be near the max line | 08:26 |
| amorin | all the time, right? | 08:27 |
| *** eernst has quit IRC | 08:29 | |
| *** eernst has joined #openstack-infra | 08:29 | |
| *** derekh has joined #openstack-infra | 08:29 | |
| *** electrofelix has joined #openstack-infra | 08:29 | |
| ianw | amorin: well, it shouldn't look like that :) | 08:29 |
| ianw | amorin: let me see what's in logs ... | 08:29 |
| amorin | ok | 08:30 |
| ianw | HttpException: 403: Client Error for url: https://compute.gra1.cloud.ovh.net/v2/dcaab5e32b234d56b626f72581e3644c/servers, {"forbidden": {"message": "The number of d | 08:30 |
| ianw | efined ports: 636 is over the limit: 600", "code": 403}} | 08:30 |
| ianw | also occasionally | 08:31 |
| ianw | HttpException: 403: Client Error for url: https://compute.gra1.cloud.ovh.net/v2/dcaab5e32b234d56b626f72581e3644c/servers, {"forbidden": {"message": "Maximum number of ports exceeded", "code": 403}} | 08:31 |
| amorin | ok | 08:32 |
| amorin | so the courb look like that mostly because of port leaking | 08:32 |
| *** gouthamr has quit IRC | 08:32 | |
| amorin | that prevent nodepool to spawn new instances | 08:32 |
| ianw | hrm, but every 20 minutes we run and clear out the ports | 08:32 |
| ianw | so the only way we get 600+ ports in 20 minutes is if the vm boot is failing *and* leaving ports behind | 08:33 |
| ianw | then nodepool is just looping making non-working vm's, if that makes sense | 08:33 |
| amorin | yup | 08:33 |
| ianw | if the nodes come up and are running something, no way we go through that many in that short time | 08:34 |
| amorin | we found something on GRA1 that we are currently fixing | 08:34 |
| amorin | I will let you know | 08:34 |
| ianw | we really need to look at nodepool's logs to better correlate the openstacksdk errors with the vm's it's trying to boot in the logs ... it's all so jumbled up | 08:34 |
| *** dmellado has quit IRC | 08:34 | |
| *** stevebaker has quit IRC | 08:35 | |
| ianw | amorin: hrm, "port list" shows 600+ ports, i wonder if the clearing isn't working? | 08:37 |
| ianw | http://paste.openstack.org/show/732692/ is a sample of the ports it cleared on the last run | 08:38 |
| ianw | amorin: let me stop the region and clear all the ports and see where we are | 08:39 |
| *** ifat_afek has joined #openstack-infra | 08:40 | |
| ianw | alright, the ports are being removed now | 08:43 |
| ianw | port list | grep DOWN | wc -l | 08:43 |
| ianw | 614 | 08:43 |
| quiquell | ianw: So just to be sure, the opnestack-infra zuul's centos nodesets has already setuptools/pip/virtualenv too ? | 08:44 |
| *** priteau has joined #openstack-infra | 08:45 | |
| ianw | quiquell: yes, it should do | 08:45 |
| ianw | amorin: so i think all we're seeing is "openstack.exceptions.ResourceTimeout: Timeout waiting for the server to come up" ... i.e. our side in the sdk is timing out on the boot with no response. i can get you some id's if it helps | 08:46 |
| ianw | 2c0a4629-b588-47ce-89a6-e4e094d7e846 might be a recent one | 08:48 |
| amorin | checking | 08:49 |
| *** shardy has joined #openstack-infra | 08:49 | |
| amorin | on my side: I see some neutron timeout errors | 08:50 |
| ianw | c479bd47-e31c-4c36-88c9-1655bd8e3b9f maybe another one with a weird error | 08:52 |
| ianw | openstack.cloud.exc.OpenStackCloudCreateException: Error creating server: c479bd47-e31c-4c36-88c9-1655bd8e3b9f | 08:52 |
| ianw | that's it :/ | 08:52 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add epel element to centos7 testing https://review.openstack.org/612636 | 08:56 |
| ianw | cgoncalves: from dib's side, i thought i put this in for testing but must have forgot ... ^ let's see if dib gate shows any issues | 08:56 |
| cgoncalves | ianw, ack :) | 08:57 |
| *** carl_cai has joined #openstack-infra | 08:59 | |
| *** eernst has quit IRC | 09:00 | |
| *** kjackal has joined #openstack-infra | 09:08 | |
| *** adriant has quit IRC | 09:09 | |
| *** adriant has joined #openstack-infra | 09:10 | |
| *** gouthamr has joined #openstack-infra | 09:11 | |
| *** adriant has quit IRC | 09:13 | |
| *** adriant has joined #openstack-infra | 09:14 | |
| *** adriant has quit IRC | 09:16 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove redundant sources change/update https://review.openstack.org/563739 | 09:16 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add a post-root.d phase https://review.openstack.org/611806 | 09:16 |
| ianw | amorin: ok, something really weird is going on, i just removed 600+ ports and they're back... | 09:18 |
| *** dmellado has joined #openstack-infra | 09:18 | |
| ianw | 200 or so, anyway | 09:19 |
| ianw | oh, maybe puppet got to it, the max-servers was set again | 09:20 |
| *** jpena|off is now known as jpena | 09:22 | |
| *** tosky has quit IRC | 09:24 | |
| *** tosky has joined #openstack-infra | 09:24 | |
| *** ifat_afek has quit IRC | 09:33 | |
| *** stevebaker has joined #openstack-infra | 09:36 | |
| ianw | amorin: ok, back to no leaked ports. i can try starting a server or two, if you like | 09:42 |
| *** jamesmcarthur has joined #openstack-infra | 09:43 | |
| ianw | #status log nl04 in emergency with ovh-gra1 set to 0 for now | 09:43 |
| openstackstatus | ianw: finished logging | 09:43 |
| *** jamesmcarthur has quit IRC | 09:47 | |
| *** quiquell is now known as quiquell|brb | 09:50 | |
| *** udesale has quit IRC | 09:51 | |
| *** udesale has joined #openstack-infra | 09:52 | |
| *** jpich has quit IRC | 09:53 | |
| *** jpich has joined #openstack-infra | 09:54 | |
| *** dhill_ has quit IRC | 09:55 | |
| *** rossella_s has quit IRC | 09:57 | |
| *** shardy has quit IRC | 09:57 | |
| *** shardy has joined #openstack-infra | 09:58 | |
| *** ifat_afek has joined #openstack-infra | 10:02 | |
| *** xek_ has quit IRC | 10:03 | |
| *** xek has joined #openstack-infra | 10:07 | |
| *** eernst has joined #openstack-infra | 10:07 | |
| *** quiquell|brb is now known as quiquell | 10:08 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove the "emit-ara-html" role https://review.openstack.org/610381 | 10:14 |
| *** ssbarnea_ has joined #openstack-infra | 10:16 | |
| e0ne | hi. could anybody please help we why depends-on flag doesn't work for https://review.openstack.org/#/c/612652/? | 10:21 |
| *** ianychoi has quit IRC | 10:22 | |
| *** ianychoi has joined #openstack-infra | 10:25 | |
| *** ifat_afek has quit IRC | 10:25 | |
| *** eernst has quit IRC | 10:30 | |
| *** apetrich has quit IRC | 10:38 | |
| *** pbourke has quit IRC | 10:47 | |
| *** pbourke has joined #openstack-infra | 10:48 | |
| *** apetrich has joined #openstack-infra | 10:54 | |
| priteau | Topic on #openstack-meeting-alt is stuck to "Documentation (Meeting topic: trove)" when no other meeting is active, could it be reset? | 10:55 |
| *** yamamoto has quit IRC | 11:03 | |
| *** yamamoto has joined #openstack-infra | 11:04 | |
| *** yamamoto has quit IRC | 11:08 | |
| *** udesale has quit IRC | 11:14 | |
| *** tosky has quit IRC | 11:18 | |
| *** tosky has joined #openstack-infra | 11:18 | |
| *** longkb has quit IRC | 11:19 | |
| *** yamamoto has joined #openstack-infra | 11:21 | |
| cmurphy | e0ne: it looks to me like it's working, i can see them linked in http://zuul.openstack.org/status | 11:27 |
| *** panda is now known as panda|lunch | 11:27 | |
| *** jpena is now known as jpena|lunch | 11:33 | |
| e0ne | cmurphy: thanks. there was an extra space in the commit message :( | 11:35 |
| *** janki has quit IRC | 11:41 | |
| *** eharney has joined #openstack-infra | 11:45 | |
| *** markvoelker has quit IRC | 11:45 | |
| *** ldnunes has joined #openstack-infra | 11:51 | |
| *** ansmith has quit IRC | 11:52 | |
| *** dhill_ has joined #openstack-infra | 11:59 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Fix meeting ID for Cyborg https://review.openstack.org/612676 | 12:01 |
| *** hwoarang has quit IRC | 12:04 | |
| *** hwoarang has joined #openstack-infra | 12:06 | |
| *** rh-jelabarre has joined #openstack-infra | 12:07 | |
| *** dave-mccowan has joined #openstack-infra | 12:07 | |
| *** janki has joined #openstack-infra | 12:14 | |
| *** bhavikdbavishi has quit IRC | 12:22 | |
| *** auristor has quit IRC | 12:24 | |
| *** janki has quit IRC | 12:25 | |
| *** auristor has joined #openstack-infra | 12:25 | |
| *** janki has joined #openstack-infra | 12:25 | |
| *** janki has quit IRC | 12:27 | |
| *** rlandy has joined #openstack-infra | 12:29 | |
| *** tobberydberg has quit IRC | 12:30 | |
| *** udesale has joined #openstack-infra | 12:31 | |
| *** adriancz has joined #openstack-infra | 12:32 | |
| *** janki has joined #openstack-infra | 12:34 | |
| *** markvoelker has joined #openstack-infra | 12:36 | |
| quiquell | ianw: Have a question regarding openstackclient RPM, this is the place ? | 12:36 |
| *** jcoufal has joined #openstack-infra | 12:37 | |
| *** markvoelker has quit IRC | 12:37 | |
| *** jchhatbar has joined #openstack-infra | 12:38 | |
| *** jamesdenton has joined #openstack-infra | 12:39 | |
| *** janki has quit IRC | 12:40 | |
| *** jpena|lunch is now known as jpena | 12:40 | |
| dtroyer | quiquell: there may be someone here who would know about that, however we do not produce distro packaging at the project level directly, that is done in separate projects or downstream by distros. | 12:41 |
| quiquell | dtroyer: ack, thanks | 12:41 |
| *** kgiusti has joined #openstack-infra | 12:47 | |
| *** trown|outtypewww is now known as trown | 12:47 | |
| fungi | ianw: quiquell: does `python3 -m venv` not work on fedora? then it doesn't matter what virtualenv package you've got installed since you wouldn't be using it anyway | 12:49 |
| quiquell | fungi: You have to use the ones already installed at fedora28 in zuul nodsets | 12:50 |
| *** jchhatbar is now known as janki | 12:50 | |
| fungi | quiquell: no, i mean, why call virtualenv at all? python3 has a venv module built-in | 12:51 |
| fungi | as long as the distro hasn't stripped it from the python3 stdlib | 12:52 |
| quiquell | fungi: We are going step by step :-) | 12:52 |
| quiquell | fungi: Sure we will use the module at the end, but we have a long road in front of us | 12:53 |
| fungi | priteau: i've reset the default topic in #openstack-meeting-alt and #openstack-meeting-4 now (both were stale) | 12:53 |
| priteau | Thank you fungi | 13:00 |
| *** yamamoto has quit IRC | 13:02 | |
| fungi | that tends to happen if the meetbot gets caught on the other side of a netsplit from chanserv in the middle of a meeting, or if it gets restarted during one (it's not stateful through restarts, which are needed to pick up configuration changes, though we try to be careful about merging those when meetings are underway) | 13:02 |
| *** ansmith has joined #openstack-infra | 13:04 | |
| *** bnemec has joined #openstack-infra | 13:08 | |
| *** psachin has quit IRC | 13:10 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update clouds.yaml for citycloud with new auth info https://review.openstack.org/612538 | 13:10 |
| ssbarnea | fungi: venv is not virtualenv, and as long we support py27 we have reasons to stick with virtualenv. later we can swap but we don't really want to use two different v-env tools at the same time. | 13:11 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove OpenStack-Chef meeting https://review.openstack.org/612691 | 13:11 |
| ssbarnea | fungi: I have question related to pypi mirrors which can be broken, as seen in http://logs.openstack.org/91/610491/9/gate/tripleo-ci-centos-7-containers-multinode/f54eb7e/job-output.txt.gz#_2018-10-23_03_33_30_244926 where it fails to find "pbr". | 13:11 |
| ssbarnea | how do we configure mirrors on our jobs? do we use the --extra-index-url for a fallback mirror or not really? | 13:12 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove Daisycloud meeting https://review.openstack.org/612692 | 13:13 |
| *** quiquell is now known as quiquell|lunch | 13:13 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove Glare meeting https://review.openstack.org/612693 | 13:14 |
| *** e0ne has quit IRC | 13:19 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove ironic-bfv and ironic-ui meetings https://review.openstack.org/612695 | 13:19 |
| *** sthussey has joined #openstack-infra | 13:19 | |
| fungi | ssbarnea: i was talking about what they're running on fedora, specifically, where it's python3-only. also the venv module has worked fine as a virtualenv stand-in for me so far. what does tripleo do with virtualenv that venv doesn't support? | 13:20 |
| *** markmcd has quit IRC | 13:21 | |
| fungi | (on python3 i mean) | 13:21 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove diskimage-builder meeting https://review.openstack.org/612696 | 13:21 |
| ssbarnea | fungi: enough to make me afraid of using it;) mainly today i was working to fix the hack that injects the libselinux bindings into virtualenv. | 13:22 |
| ssbarnea | fungi: don't get me wrong, i am not against venv. is just that I already have too much diversity to deal with. | 13:23 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove JJB meeting https://review.openstack.org/612697 | 13:24 |
| fungi | ssbarnea: the error you linked looks like probably a network issue within ovh bhs1. if connectivity between two machines there is failing, i have little hope for connectivity across the internet to pypi | 13:25 |
| ssbarnea | fungi: it can be a glitch, i would prefer to see it that it attempted to get it from two sources before failing. don't you agree? | 13:26 |
| fungi | ssbarnea: it's possible we could try that now that we're simply proxying instead of building a mirror (for that matter, the timeout might have been the proxy failing to reach pypi). back when we built full pypi mirrors instead it made more sense to not have pip try to reach additional indices | 13:28 |
| ssbarnea | fungi: can you indicate me where this could be implemented? | 13:29 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove various unused Neutron meetings https://review.openstack.org/612698 | 13:29 |
| fungi | what i'm worried about is that pip won't use it for a fallback, but will actually try to hit all the indices every time (because they could have differing package versions) so we'd effectively be telling our ci jobs to start hitting pypi every time pip is invoked | 13:30 |
| *** bobh has joined #openstack-infra | 13:31 | |
| fungi | adding tons of additional calls across the internet we aren't making now, and putting a lot of additional load on pypi's cdn | 13:32 |
| fungi | granted they probably don't care, but pushing these through a local cache is polite to our donor clouds as well | 13:32 |
| *** smarcet has joined #openstack-infra | 13:33 | |
| fungi | ssbarnea: how often are you encountering these failures, that increasing inefficiency of all jobs in our ci system is a reasonable workaround? | 13:33 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove training guide/labs team meetings https://review.openstack.org/612699 | 13:34 |
| *** mriedem has joined #openstack-infra | 13:34 | |
| fungi | and what are the chances that if the job node has trouble reaching the mirror host in that same cloud, or the mirror is in turn having trouble reaching the pypi cdn from that region, that the job node will still have no problem reaching pypi from there? | 13:34 |
| *** roman_g has joined #openstack-infra | 13:34 | |
| *** yamamoto has joined #openstack-infra | 13:35 | |
| ssbarnea | fungi: i see two occurences in the last 7 days, i think both were on gate jobs, which caused serious delay. What is the inefficiency introduced by the fallback? | 13:35 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove Solum team meeting https://review.openstack.org/612700 | 13:36 |
| fungi | ssbarnea: so two out of tens of thousands? | 13:37 |
| fungi | ssbarnea: inefficiency introduced by the fallback is that it's not a fallback. pip will hit pypi.org every time it's invoked if pypi.org is one of the listed indices. pip doesn't use them as "fallbacks" but additional indices it thinks it should check, so it checks all of them every time | 13:38 |
| *** markmcd has joined #openstack-infra | 13:38 | |
| fungi | ssbarnea: as to where we perform the configuration, it's in this task: https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/configure-mirrors/tasks/mirror.yaml | 13:38 |
| fungi | took me a moment to find | 13:38 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove Puppet-OpenStack team meeting https://review.openstack.org/612701 | 13:38 |
| *** agopi has quit IRC | 13:39 | |
| fungi | ssbarnea: we set index-url to the "mirror" proxy in the local region so that pip won't try to hit the pypi.org index, and then we use extra-index-url to add our mirror of prebuilt wheels | 13:40 |
| fungi | https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/configure-mirrors/templates/etc/pip.conf.j2 | 13:40 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove OpenStackClient team meeting https://review.openstack.org/612702 | 13:41 |
| *** boden has joined #openstack-infra | 13:42 | |
| ssbarnea | fungi: ohh, this means that we cannot really do it because pip supports only two URLs, and we are already using both. | 13:42 |
| *** roman_g has quit IRC | 13:43 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/irc-meetings master: Remove OSops team meeting https://review.openstack.org/612703 | 13:43 |
| *** roman_g has joined #openstack-infra | 13:45 | |
| ssbarnea | fungi i am wondering why it does not attempt any retry for that timeout, I see that pip does have some ability to retry based on https://github.com/pypa/pip/issues/5844 | 13:45 |
| *** kiennt26 has joined #openstack-infra | 13:50 | |
| openstackgerrit | Merged openstack-infra/zuul master: web: Increase height and padding of zuul-job-result https://review.openstack.org/610980 | 13:51 |
| ssbarnea | fungi: "Couldn't find index page for" makes me believe it did receive a 404 from the mirror, which would indicate a serious issue with the mirror. a no response is much better than a 404 for a package. strange I do remember seeing the same kind of error from pypi CDN few months back, randomly breaking some of my travis builds. | 13:51 |
| openstackgerrit | Merged openstack-infra/zuul master: encrypt_secret: support OpenSSL 1.1.1 https://review.openstack.org/611414 | 13:54 |
| openstackgerrit | Nicola Peditto proposed openstack-infra/project-config master: Added template 'publish-to-pypi-python3' to Iotronic projects. https://review.openstack.org/612705 | 13:55 |
| *** kiennt26 has quit IRC | 13:56 | |
| *** edmondsw has joined #openstack-infra | 14:00 | |
| fungi | ssbarnea: pip supports multiple urls, but it queries them all as separate indices in case they include different things | 14:01 |
| fungi | ssbarnea: if the apache proxy on the mirror received an error when it tried to get that index from the pypi cdn, then it could have resulted in the behavior observed | 14:02 |
| *** panda|lunch is now known as panda | 14:04 | |
| ssbarnea | fungi: ok, thanks. for the moment case closed, if we see it re-ocurring often enough we can reopen it and think about alternatives. | 14:04 |
| fungi | yeah, i think most of the failures we see of that nature are actually the pypi cdn failing to return what we want, so going straight to the cdn from the test nodes and bypassing the proxy isn't likely to yield much of an improvement, i don't expect | 14:06 |
| *** ramishra has quit IRC | 14:06 | |
| fungi | pypi (like most resources queried over the internet) is not reliable at the scale we tend to operate at in our ci jobs | 14:07 |
| fungi | putting a caching proxy between the test nodes and pypi helps absorb some of that, but we're still going to see more failures of that nature than when we built our own copies of pypi | 14:08 |
| fungi | that simply became unworkable once tensorflow and some other machine learning projects started inserting gigabytes a day of ml data snapshots into pypi | 14:08 |
| *** quiquell|lunch is now known as quiquell | 14:10 | |
| fungi | clarkb: something seems to have started up around 0800z today that's chewing up a noticeable amount of system cpu and increasing load average. the delay makes me think it's not related to the xenial upgrade but i'm not immediately seeing what's causing it | 14:15 |
| *** fresta_ is now known as fresta | 14:16 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Ignore removed provider in _cleanupLeakedInstances https://review.openstack.org/608670 | 14:18 |
| fungi | we seem to have picked up a little bit of swap activity since the upgrade which we weren't seeing before (according to cacti). wonder if we didn't have a swap device or simply ran on an image preconfigured with swappiness overridden to 0 | 14:18 |
| fungi | not enough to account for the system cpu utilization though | 14:18 |
| fungi | looks like it could be either apache or nodejs (or maybe both?) waiting on something | 14:20 |
| fungi | but regardless, pads have gotten really slow to load | 14:20 |
| *** janki has quit IRC | 14:21 | |
| fungi | disk of the etherpad-mysql-5.6 trove instance is getting close to full (17.7/20gb). i'll see about increasing that | 14:23 |
| *** rpioso|afk is now known as rpioso | 14:26 | |
| *** smarcet has quit IRC | 14:29 | |
| *** e0ne has joined #openstack-infra | 14:29 | |
| *** quiquell is now known as quiquell|off | 14:29 | |
| *** aojeagarcia has joined #openstack-infra | 14:35 | |
| *** aojea has quit IRC | 14:35 | |
| *** felipemonteiro has joined #openstack-infra | 14:38 | |
| clarkb | fungi: top seems to show apache being the bigger consumer of cpu time than anything else? | 14:47 |
| fungi | it varies | 14:47 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cleanup node requests that are declined by all current providers https://review.openstack.org/610915 | 14:48 |
| clarkb | possible its a meltdown mitigation cost? that for some reason just wasn't present in trusty? (though we checked all were PTIing at the time) | 14:48 |
| clarkb | that would explain a rise in system cpu cost at least as it has to shuffle things around anytime you syscall and back | 14:49 |
| clarkb | fungi: related to the zuul executors we might try using the hwe kernel on that node to see if it helps? | 14:49 |
| fungi | doesn't explain though why it just started 7 hours ago | 14:52 |
| clarkb | true | 14:52 |
| fungi | unattended-upgrades fired at 0700z on the etherpad server | 14:53 |
| fungi | openssh, man-db, systemd, ureadahead, ufw | 14:54 |
| fungi | i wonder if something is reindexing? | 14:54 |
| fungi | #status log doubled size of disk for etherpad-mysql-5.6 trove instance from 20gb to 40gb (contains 17.7gb data) | 14:55 |
| openstackstatus | fungi: finished logging | 14:56 |
| fungi | hrm, it's also using all its ram | 14:56 |
| clarkb | it being the db? | 14:57 |
| fungi | yeah | 14:57 |
| *** ccamacho has quit IRC | 14:57 | |
| fungi | the trove instance is going to restart here in a bit | 14:58 |
| fungi | i'm doubling its memory allocation | 14:58 |
| clarkb | related to the db, it occurred to me if we switched from gz to xz for the compression on backup artifacts we would probably save quite a bit of disk space | 14:58 |
| clarkb | fungi: ++ | 14:58 |
| lbragstad | that would explain the 503s from etherpad.o.o :) | 14:58 |
| clarkb | but I need to test that hypothesis before pushing it out | 14:58 |
| clarkb | fungi: re something indexing I don't see any processes on the app server itself that look like that | 14:59 |
| clarkb | I do see some unnecessary things like a battery manager :/ | 14:59 |
| clarkb | fungi: if the db size up doesn't fix things, maybe we install dstat? | 15:00 |
| clarkb | that should give us much better data over time? | 15:01 |
| fungi | sgtm | 15:01 |
| fungi | #status log doubled memory allocation for etherpad-mysql-5.6 trove instance from 2gb to 4gb (contains indicated ~2gb active use) | 15:02 |
| openstackstatus | fungi: finished logging | 15:02 |
| evrardjp | thanks fungi for handling that event :) | 15:02 |
| openstackgerrit | Merged openstack-infra/zuul master: Exclude .keep files from .gitignore https://review.openstack.org/611990 | 15:03 |
| openstackgerrit | Merged openstack-infra/zuul master: Add a sanity check for all refs returned by Gerrit https://review.openstack.org/599011 | 15:03 |
| openstackgerrit | Merged openstack-infra/zuul master: Reload tenant in case of new project branches https://review.openstack.org/600088 | 15:03 |
| fungi | looks like i may need to restart etherpad-lite service too | 15:03 |
| clarkb | ya not sure if it will reconnect on its own | 15:03 |
| fungi | it crashed, looks like | 15:04 |
| fungi | running again now | 15:04 |
| evrardjp | thanks! | 15:04 |
| clarkb | https://etherpad.openstack.org/p/clarkb-test looks happy | 15:04 |
| *** hamzy has quit IRC | 15:04 | |
| clarkb | yup thank you fungi | 15:05 |
| fungi | seems pretty snappy, but at this point hard to know whether it was the nodejs restart or the trove ram/disk increases | 15:05 |
| *** cfriesen has joined #openstack-infra | 15:11 | |
| cfriesen | can anyone tell me why https://review.openstack.org/#/c/611498 didn't merge yesterday? | 15:11 |
| cfriesen | it's got +W and +1 from zuul, but no +2 | 15:12 |
| *** jamesmcarthur has joined #openstack-infra | 15:12 | |
| fungi | Depends-On: I2861839532049bf0c8a2bf89311c4c56186fc0fb | 15:12 |
| cfriesen | that's merged | 15:13 |
| fungi | merged at 20:26z | 15:13 |
| fungi | your change was approved before the depends-on merged | 15:13 |
| mordred | yeah. what fungi said | 15:13 |
| clarkb | but also 6e34371af089cc71c5e54c6921a644cf4391d77a is the parent commit which is not the current patchset of the parent change | 15:13 |
| fungi | oh, yep, i think that's the actual problem | 15:13 |
| fungi | because the depends-on is to another change in the same repo (which is kind of odd but whatever) so should have shared a change queue | 15:14 |
| cfriesen | that's not actually my commit, some of our guys didn't know about the implicit depends in the same repo | 15:14 |
| fungi | so it's that the parent in gerrit got another patchset and 611498 wasn't rebased to reparent it | 15:14 |
| cfriesen | so sounds like I need to rebase this? | 15:15 |
| cfriesen | on what actually merged | 15:15 |
| clarkb | fwiw the little orange dot there tries to tell you this, its just really bad ui from gerrit on actually making that clear | 15:17 |
| *** ccamacho has joined #openstack-infra | 15:17 | |
| *** ccamacho has quit IRC | 15:17 | |
| cfriesen | ah, I was wondering about the orange dot. just hadn't looked it up yet. | 15:17 |
| cfriesen | thanks for the help | 15:17 |
| fungi | yeah, https://review.openstack.org/611498 has a parent of 611494,1 but 611494,2 is what ended up merging. so gerrit considers it not mergeable because its parent will never exist in the branch | 15:17 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Cleanup node requests that are declined by all current providers https://review.openstack.org/610915 | 15:17 |
| fungi | cfriesen: a rebase should solve it | 15:18 |
| cfriesen | great, thanks | 15:18 |
| dtroyer | I'm seeing a new failure in publish-stx-specs at cd7827ba0e3fb3dd2ff5fc77e0bc4c7ba81f4969, looks like we may have exceeded our quota? | 15:22 |
| *** fuentess has joined #openstack-infra | 15:22 | |
| dtroyer | shoot, that's not a link… http://logs.openstack.org/cd/cd7827ba0e3fb3dd2ff5fc77e0bc4c7ba81f4969/post/publish-stx-specs/9232619/ara-report/ | 15:24 |
| dtroyer | that's a link... | 15:24 |
| openstackgerrit | Merged openstack-infra/zuul master: Use merger to get list of files for pull-request https://review.openstack.org/603287 | 15:25 |
| openstackgerrit | Merged openstack-infra/zuul master: Add support for authentication/STARTTLS to SMTP https://review.openstack.org/603833 | 15:25 |
| openstackgerrit | Merged openstack-infra/zuul master: encrypt_secret: Allow file scheme for public key https://review.openstack.org/581429 | 15:25 |
| *** armax has joined #openstack-infra | 15:25 | |
| clarkb | dtroyer: yes, we'll need to bump the quota. Would probably be good to understand what is using the disk space so quickly | 15:26 |
| dtroyer | clarkb: right… we did just add a handful of documents in the last maybe week… I haven't gone through looking at usage yet | 15:26 |
| fungi | large binary objects maybe? | 15:27 |
| dtroyer | so q, if we find something large that gets removed, its removed from afs too? | 15:27 |
| dtroyer | fungi: that's what I'm thinking | 15:27 |
| dtroyer | or images or ppt | 15:27 |
| fungi | yes, afs publication is basically rsync --delete | 15:27 |
| dtroyer | its stuff from the wiki and I suspect soeone who wanted to upload ppts now finally did | 15:28 |
| dtroyer | fungi: good, I was hoping for that :) | 15:28 |
| dtroyer | why did I ever doubt? :) | 15:28 |
| *** smarcet has joined #openstack-infra | 15:28 | |
| fungi | we put a "root marker" file at the root of each tree handled by a particular job so that the publisher knows not to descend into any child dir with one of those in it | 15:31 |
| fungi | and so only cleans up its own files and not those for which another job is responsible | 15:31 |
| clarkb | dtroyer: if the content is something we want/need then we don't need to delete it. Mostly want to double check there isn't unwanted or unexpected disk consumption | 15:31 |
| fungi | (at least in theory) | 15:31 |
| *** dtantsur is now known as dtantsur|brb | 15:32 | |
| *** kopecmartin is now known as kopecmartin|off | 15:33 | |
| fungi | i don't think it's specifically specs at fault, just browsing around /afs/openstack.org/project/starlingx.io/www/specs/ a bit | 15:34 |
| fungi | checking other parts of the tree now | 15:34 |
| *** ianychoi_ has joined #openstack-infra | 15:36 | |
| fungi | du -sh /afs/openstack.org/project/starlingx.io/www says 98M | 15:36 |
| fungi | is the quota only 100mb? | 15:36 |
| clarkb | fungi: ya beacuse zuul is using like 5/100mb on its volume | 15:37 |
| fungi | k | 15:37 |
| clarkb | so we used zuul as a starting point | 15:37 |
| *** cfriesen has quit IRC | 15:37 | |
| *** hamzy has joined #openstack-infra | 15:38 | |
| *** e0ne has quit IRC | 15:39 | |
| *** ianychoi has quit IRC | 15:40 | |
| *** gfidente is now known as gfidente|afk | 15:46 | |
| fungi | but yeah, just skimming around i think a bunch of it is merely boilerplate for the dozens of stx projects publishing their docs under there now | 15:47 |
| fungi | copies of jquery, icons, et cetera | 15:48 |
| fungi | zuul only has a handful of repos compared to stx | 15:48 |
| *** agopi has joined #openstack-infra | 15:48 | |
| clarkb | fungi: are you in a spot to bump the quota there or should I spin up my aklog shell to do it? | 15:53 |
| clarkb | maybe go to 1GB? | 15:53 |
| *** smarcet has quit IRC | 15:56 | |
| fungi | i can take care of it in just a moment | 15:59 |
| clarkb | thanks! | 15:59 |
| * dtroyer catches up | 16:02 | |
| dtroyer | thanks guys… yeah, we have 30+ separate sphinx builds writing into that tree… we are thinking about doing some sort of meta-build to cut some of that duplication out and to allow things like internal Sphinx references between more of those… longer term | 16:03 |
| *** smarcet has joined #openstack-infra | 16:03 | |
| *** ginopc has quit IRC | 16:05 | |
| *** udesale has quit IRC | 16:06 | |
| openstackgerrit | Merged openstack-infra/zuul master: web: add config-errors notifications drawer https://review.openstack.org/597147 | 16:07 |
| *** felipemonteiro has quit IRC | 16:09 | |
| *** gyee has joined #openstack-infra | 16:16 | |
| fungi | clarkb: dtroyer: i've done `fs setquota -path /afs/openstack.org/project/starlingx.io -max 1000000` | 16:17 |
| fungi | and now `fs listquota -path /afs/openstack.org/project/starlingx.io` says 100054 of 1000000 used (10%) | 16:17 |
| *** smarcet has quit IRC | 16:17 | |
| fungi | #status log increased quota for project.starlingx volume from 100mb to 1gb | 16:18 |
| openstackstatus | fungi: finished logging | 16:18 |
| dtroyer | thanks fungi… should I push up another review to re-run that job? | 16:18 |
| fungi | if that's easy | 16:21 |
| fungi | otherwise i can reenqueue the last ref into the post pipeline | 16:22 |
| dtroyer | if you could do that I'd appreciate it, I'm about to get busy for a few hours… | 16:22 |
| *** agopi has quit IRC | 16:24 | |
| *** smarcet has joined #openstack-infra | 16:27 | |
| fungi | sure (i mean, i'm busy too, but it's a fairly simple command) | 16:27 |
| fungi | dtroyer: i've just now done `sudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=post --project=openstack/stx-specs --ref=refs/heads/master --newrev=cd7827ba0e3fb3dd2ff5fc77e0bc4c7ba81f4969` | 16:28 |
| fungi | (based on the ref for the failed job run you linked, assuming that it was the most recent) | 16:29 |
| *** gfidente|afk has quit IRC | 16:39 | |
| *** armstrong has joined #openstack-infra | 16:46 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: DNM: testing zookeeper oddities https://review.openstack.org/612750 | 16:46 |
| *** jamesmcarthur has quit IRC | 16:46 | |
| *** smarcet has quit IRC | 16:46 | |
| *** carl_cai has quit IRC | 16:47 | |
| *** dtruong has quit IRC | 16:56 | |
| *** derekh has quit IRC | 17:00 | |
| *** smarcet has joined #openstack-infra | 17:01 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add support for enabling the ARA callback plugin in install-ansible https://review.openstack.org/611228 | 17:03 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add playbook for deploying the ARA web application https://review.openstack.org/611232 | 17:03 |
| *** gothicmindfood has quit IRC | 17:05 | |
| *** bobh has quit IRC | 17:08 | |
| *** e0ne has joined #openstack-infra | 17:08 | |
| *** bobh has joined #openstack-infra | 17:09 | |
| *** gothicmindfood has joined #openstack-infra | 17:10 | |
| *** hamzy has quit IRC | 17:12 | |
| *** hamzy has joined #openstack-infra | 17:12 | |
| *** dtantsur|brb is now known as dtantsur | 17:13 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: DNM: testing zookeeper oddities https://review.openstack.org/612750 | 17:13 |
| *** bobh has quit IRC | 17:14 | |
| clarkb | fungi: dtroyer http://logs.openstack.org/cd/cd7827ba0e3fb3dd2ff5fc77e0bc4c7ba81f4969/post/publish-stx-specs/d864000/ara-report/ looks happy fwiw | 17:16 |
| *** lbragstad is now known as lbragstad_f00d | 17:17 | |
| *** aojeagarcia has quit IRC | 17:18 | |
| fungi | great! | 17:18 |
| *** dtantsur is now known as dtantsur|afk | 17:29 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: WIP: Run dstat and generate graphs in unit tests https://review.openstack.org/612765 | 17:30 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: DNM: testing zookeeper oddities https://review.openstack.org/612750 | 17:31 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Run zookeeper datadir on tmpfs during testing https://review.openstack.org/612766 | 17:33 |
| *** jpena is now known as jpena|off | 17:34 | |
| *** agopi has joined #openstack-infra | 17:34 | |
| *** agopi has quit IRC | 17:39 | |
| *** tpsilva has joined #openstack-infra | 17:40 | |
| corvus | i've created adns1; moving on to ns1 now | 17:41 |
| *** trown is now known as trown|lunch | 17:42 | |
| *** e0ne has quit IRC | 17:45 | |
| *** lbragstad_f00d is now known as lbragstad | 17:45 | |
| *** felipemonteiro has joined #openstack-infra | 17:46 | |
| *** smarcet has quit IRC | 17:46 | |
| Shrews | clarkb: still on for builder zk rehoming in 2 hrs? | 17:47 |
| clarkb | Shrews: ya let me remove my -1 wip | 17:47 |
| clarkb | https://review.openstack.org/#/c/612441/1 is the change | 17:48 |
| Shrews | got it | 17:48 |
| Shrews | clarkb: i'm almost wondering if we should do a total shutdown of the builder processes before merging that | 17:53 |
| Shrews | we've never switched to a *different* cluster on the fly | 17:53 |
| clarkb | Shrews: or merge it with nb01-03 in the emergency file so that we can coordinate when it applies (and shutdown the builders as part of that) | 17:53 |
| Shrews | clarkb: i think that would be the best plan | 17:54 |
| clarkb | I'll go add them to that list now | 17:54 |
| Shrews | ++ | 17:54 |
| clarkb | all three builders listed now in the emergency file | 17:55 |
| * Shrews needs to step away for a bit before the meeting. biab | 17:57 | |
| *** hamzy has quit IRC | 17:59 | |
| clarkb | I too need to get a few things done away from the computer before the meeting happens | 17:59 |
| *** hamzy has joined #openstack-infra | 18:00 | |
| *** gary_perkins has quit IRC | 18:00 | |
| *** kjackal has quit IRC | 18:02 | |
| *** agopi has joined #openstack-infra | 18:06 | |
| *** bobh has joined #openstack-infra | 18:07 | |
| *** jamesmcarthur has joined #openstack-infra | 18:07 | |
| *** jamesmcarthur has quit IRC | 18:08 | |
| *** jamesmcarthur has joined #openstack-infra | 18:09 | |
| *** bobh has quit IRC | 18:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add opendev nameservers (2/2) https://review.openstack.org/610066 | 18:14 |
| *** smarcet has joined #openstack-infra | 18:14 | |
| *** apetrich has quit IRC | 18:16 | |
| *** apetrich has joined #openstack-infra | 18:17 | |
| *** jpich has quit IRC | 18:20 | |
| corvus | remote: https://review.openstack.org/612770 Add initial zone info | 18:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Gerritbot: add zone-opendev.org to -infra https://review.openstack.org/612771 | 18:22 |
| corvus | clarkb, fungi: would you plase review those 3 changes asap? | 18:23 |
| fungi | corvus: has anyone talked to mnaser yet to get reverse dns updated on ns2? | 18:23 |
| fungi | (and yes, already looking at them) | 18:23 |
| corvus | fungi: not yet, that's next on my list | 18:23 |
| *** hamzy has quit IRC | 18:23 | |
| fungi | okay, cool just noticed it was still at a default generated ptr | 18:23 |
| *** hamzy has joined #openstack-infra | 18:24 | |
| corvus | fungi: and i believe we decided that we wanted manual dig queries to work before we asked jimmy to set up the glue records, so i'll do that after we work through the initial bootstrap | 18:24 |
| *** smarcet has quit IRC | 18:24 | |
| clarkb | corvus: I think we need to edit acls or groups for 612770 | 18:24 |
| clarkb | I only have +-1 | 18:24 |
| *** kjackal has joined #openstack-infra | 18:25 | |
| clarkb | shoudl I add infra-root as a group to the opendev zone file core group? | 18:26 |
| fungi | looks like we simply need to update the group membership | 18:26 |
| clarkb | ya | 18:27 |
| fungi | yeah, that's what i was about to ask as well, though pretty sure the answer is yes | 18:27 |
| clarkb | I went ahead and added infra core | 18:27 |
| clarkb | can be changed later if necessary | 18:27 |
| fungi | i see that, thanks! | 18:27 |
| *** bobh has joined #openstack-infra | 18:28 | |
| fungi | we're missing reverse dns on the v6 address of adns1? | 18:29 |
| corvus | well that's weird | 18:29 |
| fungi | ahh, nope, wrong address | 18:30 |
| fungi | i was looking at reverse dns for the v6 address of ns2 | 18:30 |
| corvus | whew | 18:30 |
| fungi | so nothing to see here, move along ;) | 18:30 |
| *** e0ne has joined #openstack-infra | 18:31 | |
| *** e0ne has quit IRC | 18:31 | |
| *** felipemonteiro has quit IRC | 18:34 | |
| *** irclogbot_2 has joined #openstack-infra | 18:35 | |
| *** chandankumar is now known as chkumar|off | 18:37 | |
| fungi | the anomalous system cpu load on the etherpad server doesn't seem to have returned after the nodejs restart and trove resizing | 18:44 |
| fungi | still quite snappy | 18:44 |
| *** panda has quit IRC | 18:45 | |
| *** panda has joined #openstack-infra | 18:45 | |
| clarkb | fungi: yup it seems to be happy | 18:46 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Handle project names in the new Story modal controller https://review.openstack.org/612778 | 18:47 |
| *** gary_perkins has joined #openstack-infra | 18:51 | |
| *** cfriesen has joined #openstack-infra | 18:52 | |
| melwitt | can anyone remind me which irc bot is needed in channel to make elastic-recheck comments on gerrit work? | 18:53 |
| dtroyer | fungi: thank you, it does look happy on the web side too | 18:54 |
| cfriesen | melwitt: recheckwatchbot ? | 18:55 |
| cfriesen | or maybe that's just how it identifies itself in logs | 18:55 |
| clarkb | melwitt: cfriesen you don't need the irc bot to be in channel to comment on the gerrit changes | 18:55 |
| clarkb | melwitt: cfriesen it should comment on all gerrit changes it can identify as failing for a particular reason within the timeout window we give it. Then on top of that it can also comment on irc if you configure it for that | 18:56 |
| melwitt | ok. I can't recall what the issue was last time it stopped commenting. I had thought it had to do with an irc bot | 18:56 |
| melwitt | I'm not 100% sure it stopped commenting but was trying to see how I can check if any of the necessary conditions have not been met | 18:57 |
| clarkb | Shrews: maybe you want to review/approve https://review.openstack.org/#/c/612441/ now so that it is ready for us after the meeting? the emergency file is set now | 18:57 |
| *** bobh has quit IRC | 18:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Merger: automatically add new hosts to the known_hosts file https://review.openstack.org/608453 | 18:57 |
| melwitt | I was trying to determine whether or not it had stopped commenting | 18:57 |
| clarkb | melwitt: https://review.openstack.org/#/q/owner:%22Elastic+Recheck+(8871)%22 is the gerrit account that should comment. I forget how to search by commented by | 18:57 |
| clarkb | you should be able to query gerrit for changes that that account has commented on though | 18:58 |
| melwitt | right, I can search "comment:elastic" and find things | 18:58 |
| clarkb | https://review.openstack.org/#/q/commentby:%22Elastic+Recheck+(8871)%22 is the query | 18:59 |
| melwitt | ah, thanks | 18:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Merger: automatically add new hosts to the known_hosts file https://review.openstack.org/608453 | 18:59 |
| clarkb | ifnra meeting starting in a minute over in #openstack-meeting | 18:59 |
| ssbarnea | yep, didn't had time to even check the agenda | 19:00 |
| Shrews | clarkb: done | 19:00 |
| *** agopi has quit IRC | 19:03 | |
| ianw | fungi: (re your comment ages ago), yeah, iirc the problem was stuff does "virtualenv -p python3", so that was what we wanted to work. but i don't know how built-in venv is either, try it on bionic :) there you need python3-venv | 19:04 |
| fungi | right | 19:04 |
| ianw | fungi: i get very lost trying to understand what the future of something that does what "virtualenv" is | 19:04 |
| * fungi shakes fist at debian python manitainers | 19:04 | |
| ssbarnea | afaik venv is part of python3 and is a replacement of virtualenv, the only catch is that there is no venv for py27, but there is a working virtualenv for py3. | 19:07 |
| ssbarnea | which made venv for me a no go from start, enough problems with the old virtualenv to combine them with the new ones. | 19:07 |
| ianw | ... and it's not *quite* a part of base python3 on all distros ... heh, see, it's that simple :) | 19:08 |
| *** smarcet has joined #openstack-infra | 19:08 | |
| fungi | there's been some frequent talk of backporting the venv module to upstream 2.7.x but sounds like it's nontrivial due to some differences in module relocation support for python3 i think | 19:08 |
| fungi | but yeah, distros stripping out bits of the pythonstdlib and sticking them in packages which don't get installed with the interpreter is annoying | 19:09 |
| clarkb | ++ | 19:10 |
| ianw | then you've got pipenv going on too. and that has a gif screencast of a terminal session on it's github page, which makes it better too | 19:10 |
| ssbarnea | ianw: probably with a youtube channel, instagram account too. | 19:13 |
| ianw | :) | 19:14 |
| fungi | technical support via snapchat | 19:15 |
| *** e0ne has joined #openstack-infra | 19:15 | |
| dmsimard | no twitter ? | 19:16 |
| *** smarcet has quit IRC | 19:18 | |
| fungi | twitter is so 2018 | 19:21 |
| *** jamesmcarthur has quit IRC | 19:22 | |
| *** pcaruana has quit IRC | 19:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Switch nodepool builders to zk cluster https://review.openstack.org/612441 | 19:23 |
| *** jamesmcarthur has joined #openstack-infra | 19:23 | |
| *** jamesmcarthur has quit IRC | 19:27 | |
| *** david-lyle has joined #openstack-infra | 19:27 | |
| *** dklyle has quit IRC | 19:28 | |
| *** jamesmcarthur has joined #openstack-infra | 19:28 | |
| *** jamesmcarthur has quit IRC | 19:30 | |
| *** jamesmcarthur has joined #openstack-infra | 19:32 | |
| *** bobh has joined #openstack-infra | 19:32 | |
| *** trown|lunch is now known as trown | 19:36 | |
| ssbarnea | anything urgent on the agenda? ;) | 19:37 |
| *** bobh has quit IRC | 19:38 | |
| fungi | on the infra meeting agenda? not that i'm aware | 19:40 |
| *** armstrong has quit IRC | 19:46 | |
| *** agopi has joined #openstack-infra | 19:50 | |
| *** lbragstad has quit IRC | 19:50 | |
| *** lbragstad has joined #openstack-infra | 19:53 | |
| *** david-lyle is now known as dklyle | 19:53 | |
| amorin | hey guys | 19:55 |
| dmsimard | infra-root: I'm back if we want to chat bridge.o.o | 19:55 |
| fungi | after infra meeting maybe | 19:55 |
| fungi | hi amorin! any news? | 19:56 |
| amorin | ianw: check the nodepool status on gra1, sounds in better shape | 19:56 |
| amorin | fungi: yes | 19:56 |
| amorin | http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1&from=now-3h&to=now | 19:56 |
| amorin | we fixed an issue on gra1 | 19:56 |
| amorin | I think it will help a lot | 19:56 |
| fungi | ooh, that's... looking better as of ~1815utc? | 19:57 |
| fungi | thanks!!! | 19:57 |
| amorin | we fid not apply on bhs1 yet | 19:57 |
| amorin | we will tomorrow | 19:57 |
| ianw | amorin: excellent! did clarkb turn it back on? | 19:57 |
| *** jamesmcarthur has quit IRC | 19:58 | |
| clarkb | I haven't touched it | 19:58 |
| clarkb | unless I messed up an emergency file edit | 19:58 |
| clarkb | no nl04 is still in that file | 19:59 |
| ianw | hrm, no it's still in there ... but nobody turned it back to 79? | 19:59 |
| ianw | oh for heavens sake | 19:59 |
| ianw | opensatck.org | 19:59 |
| clarkb | oh ha | 19:59 |
| fungi | those people are splitters | 19:59 |
| fungi | buncha poseurs | 20:00 |
| clarkb | Shrews: I can run the kick.sh for nb0*.openstack.org if you want to shut down the running builders and let me know when you are happy they are off? | 20:00 |
| ianw | #status log nb04.opensatck.org removed from emergency | 20:00 |
| openstackstatus | ianw: finished logging | 20:00 |
| Shrews | clarkb: on it | 20:00 |
| clarkb | Shrews: the kick.sh should edit the config and restart the service | 20:00 |
| clarkb | I'll go talk to the release team in the interim | 20:00 |
| dmsimard | ianw: replied to your comment on https://review.openstack.org/#/c/611228/ sorry to have missed it | 20:01 |
| ianw | amorin: well all's well that ends well :) it still seems that we're cleaning up a lot of leaked ports, but it does seem we're not running out of ports before the next cleanup run. is that a known problem? | 20:01 |
| *** jamesmcarthur has joined #openstack-infra | 20:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Gerritbot: add zone-opendev.org to -infra https://review.openstack.org/612771 | 20:03 |
| Shrews | hrm, i thought builder shutdown would kill any running dib processes | 20:03 |
| clarkb | Shrews: I think if dib is blocking on a long running process like the image conversion step it may not process signals? | 20:04 |
| clarkb | I want to say it does go away after a while? | 20:04 |
| Shrews | clarkb: should be safe to just kill right? | 20:04 |
| clarkb | Shrews: ya, it might leak some stuff but we are already counting on cleaning all that up anyway | 20:04 |
| ianw | it might leave behind mounts, probably best to do a reboot | 20:05 |
| *** bobh has joined #openstack-infra | 20:05 | |
| Shrews | ianw: a server reboot? | 20:05 |
| aspiers | how good are the results from inline replies to mail notifications from Gerrit reviews? | 20:05 |
| clarkb | we can also wait for dib to finish if it won't be too long | 20:05 |
| Shrews | clarkb: well i already killed it | 20:05 |
| clarkb | Shrews: heh ok :) | 20:05 |
| aspiers | i.e. if I inline reply to a Gerrit mail notification, will it create a mess in the review? | 20:06 |
| clarkb | aspiers: I don't think the version of gerrit we run supports it at all | 20:06 |
| aspiers | I found https://gerrit-review.googlesource.com/Documentation/intro-user.html#reply-by-email but it doesn't go into detail | 20:06 |
| clarkb | aspiers: so no, it should just noop | 20:06 |
| *** openstackgerrit has quit IRC | 20:06 | |
| aspiers | oh OK, thanks | 20:06 |
| clarkb | aspiers: the gerrit mailing list makes it seem like that particular feature is still a work in progress | 20:06 |
| clarkb | its mostly works except for the weird corner cases people have run into | 20:06 |
| Shrews | clarkb: ianw: should we reboot (nb02)? | 20:06 |
| *** jamesmcarthur has quit IRC | 20:06 | |
| aspiers | gotcha | 20:06 |
| clarkb | Shrews: let me look | 20:07 |
| clarkb | ya there are a bunch of weird mounts. Shrews do we want to disable the nodepool builder service first so that it doesn't start up again and start building before we start it? | 20:07 |
| clarkb | sudo systemctl disable nodepool-builder (I think) | 20:08 |
| clarkb | then puppet will enable it, update the config, and start it | 20:08 |
| Shrews | clarkb: yeah | 20:08 |
| clarkb | we should probably do the same on the other two builders too so that they are all running a consistent kernel and anything else pending a reboot | 20:09 |
| *** jamesmcarthur has joined #openstack-infra | 20:09 | |
| Shrews | clarkb: nb01 appears to have those weird mounts too, and i didn't kill anything there | 20:09 |
| *** bobh has quit IRC | 20:09 | |
| clarkb | Shrews: I wonder if dib crashes leak those too | 20:09 |
| Shrews | *shrug* | 20:10 |
| ianw | yeah, it's possible that failed builds can sometimes leak mounts, depending on how they failed | 20:10 |
| clarkb | in any case I think the reboots should be done on nb01 nb02 and nb03 for update consistency with kernels and the like | 20:10 |
| clarkb | and disable nodepool-builder first | 20:10 |
| Shrews | *nod* | 20:10 |
| ianw | considering it's been up for 277 days, maybe a few leaks isn't that bad :) (i welcome any reviews that help the cleanup path) | 20:11 |
| Shrews | clarkb: nodepool-builder disabled on all 3 | 20:11 |
| Shrews | clarkb: issueing reboots now | 20:11 |
| clarkb | nb01 looks good, no nodepool-builder running and much cleaner mounts | 20:13 |
| dmsimard | so, re: bridge.o.o... my understanding is that it needs to be rebuilt from scratch anyway because it's not sized properly ? is there a plan on how we might scale bridge.o.o beyond a single machine or do we want to stay on a single server ? | 20:13 |
| Shrews | clarkb: yep. all 3 back, no builder running | 20:14 |
| clarkb | nb02 too | 20:14 |
| clarkb | Shrews: when I kick.sh should I start with nb01.o.o and make sure it all looks good before doing the other two? | 20:14 |
| clarkb | Shrews: also I'm ready to do ^ if you are | 20:14 |
| Shrews | clarkb: yeah | 20:14 |
| clarkb | ok running kick against nb01 now | 20:15 |
| clarkb | its takes a couple minutes to figure out the inventory and groups | 20:16 |
| clarkb | Shrews: puppet is done on nb01 | 20:18 |
| clarkb | looks like it didn't actuall enable and start the builder | 20:18 |
| Shrews | nope | 20:18 |
| clarkb | I wonder if we have a bug with the sysv init script compat laye | 20:19 |
| *** smarcet has joined #openstack-infra | 20:19 | |
| corvus | dmsimard: for the forseeable future, a single machine | 20:19 |
| clarkb | Shrews: considering we manually disabled the service I'm good to manually enable and start the service if you are | 20:19 |
| clarkb | Shrews: then we can rerun puppet to make sure everything is happy | 20:19 |
| Shrews | clarkb: go ahead. i'm monitoring the log now... | 20:19 |
| clarkb | ok | 20:19 |
| clarkb | it is running | 20:20 |
| clarkb | seems to be running dib implying its happy with the zk? | 20:20 |
| clarkb | I see the connection from nb01 on zk01 | 20:21 |
| Shrews | yeah | 20:21 |
| clarkb | Shrews: I'll rerun kick.sh against nb01 to make sure the puppet is happy with my manual service enablement | 20:21 |
| Shrews | 2018-10-23 20:20:16,645 INFO kazoo.client: Connecting to 2001:4800:7817:103:be76:4eff:fe04:e359:2181 | 20:21 |
| Shrews | 2018-10-23 20:20:16,677 INFO kazoo.client: Zookeeper connection established, state: CONNECTED | 20:22 |
| clarkb | (want it to steady state as a noop) | 20:22 |
| clarkb | it nooped. Ready for me to run it against nb02 and nb03? | 20:23 |
| *** kgiusti has left #openstack-infra | 20:23 | |
| clarkb | Shrews: ^ I'll wait for your go ahead | 20:24 |
| Shrews | go | 20:24 |
| clarkb | sudo ./kick.sh nb02.openstack.org:nb03.openstack.org is running | 20:25 |
| clarkb | there is probably a better way to glob that | 20:25 |
| clarkb | nb03 is done. I will manually start the service there now | 20:27 |
| clarkb | er nb02, 03 is still running | 20:27 |
| clarkb | I see nb02 on zk01 now | 20:28 |
| Shrews | yep, it connected, creating a new image | 20:28 |
| clarkb | starting nb03 builder now | 20:29 |
| Shrews | oops | 20:29 |
| clarkb | and I see nb03 on zk01 now | 20:29 |
| clarkb | oh? | 20:29 |
| Shrews | 2018-10-23 20:29:37,954 INFO kazoo.client: Connecting to 2001:4800:7815:102:be76:4eff:fe02:f134:2181 | 20:30 |
| Shrews | 2018-10-23 20:29:37,959 WARNING kazoo.client: Connection dropped: socket connection error: Network is unreachable | 20:30 |
| Shrews | 2018-10-23 20:29:37,962 INFO kazoo.client: Connecting to 2001:4800:7817:103:be76:4eff:fe04:e359:2181 | 20:30 |
| Shrews | 2018-10-23 20:29:37,963 WARNING kazoo.client: Connection dropped: socket connection error: Network is unreachable | 20:30 |
| Shrews | 2018-10-23 20:29:37,964 INFO kazoo.client: Connecting to 23.253.236.126:2181 | 20:30 |
| Shrews | nb03 could only connect to that last one | 20:30 |
| clarkb | Shrews: is that on 03? I wonder if ipv6 doesn't work in that cloud | 20:30 |
| clarkb | ianw: ^ any idea if that is expected to work? | 20:30 |
| Shrews | yeah, only the ipv4 worked | 20:30 |
| clarkb | Shrews: I think its ok for now if that builder falls back to ipv4 | 20:30 |
| clarkb | unfortunately we are finding that happens in a few spots because ipv6 isn't quite working as expected :( | 20:31 |
| Shrews | nb03 is connected though and building a new image | 20:31 |
| clarkb | nb03 only has a scope:Link address for ipv6 | 20:31 |
| clarkb | I think ipv6 not working there is expected. Possibly we need to have kazoo check if it can ipv6 before ipv6ing? | 20:32 |
| Shrews | clarkb: i'm not sure why it chose ipv4 for the last address though | 20:32 |
| corvus | thoe hosts are zk03/zk01/zk01 -- so it tried both the v6 and v4 for zk01. | 20:32 |
| Shrews | oh, heh | 20:32 |
| clarkb | https://review.openstack.org/#/c/611920/ is the chagne I wrote for gear to do similar on the binding side | 20:32 |
| clarkb | but you can use the AI_ADDRCONFIG flag when connecting too aiui | 20:32 |
| Shrews | well that's controlled in the kazoo layer | 20:33 |
| clarkb | AI_ADDRCONFIG says give me addresses that are valid for configured AF_INET types on this host | 20:33 |
| clarkb | ya probably needs to be fixed in kazoo | 20:33 |
| clarkb | in any case things are working well from what I see so far. I guess we watch it and make sure that images get uploaded into the clouds then prepare for thursday morning zuul+launcher switch? | 20:34 |
| *** gfidente has joined #openstack-infra | 20:34 | |
| ianw | clarkb: i don't think 03 will have ipv6 | 20:34 |
| clarkb | corvus: Shrews does a noonish eastern time work for you on thursday to do cutover? | 20:34 |
| clarkb | ianw: ya I don't see it there via ifconfig so likely a kazoo bug | 20:34 |
| Shrews | clarkb: yes, i should be available then | 20:35 |
| *** ssbarnea_ has quit IRC | 20:36 | |
| clarkb | corvus: I figure we should do a full restart of zuul too and mark the sha1 so that we can cut a zuul release in the near future too | 20:36 |
| *** ansmith has quit IRC | 20:37 | |
| *** hamzy has quit IRC | 20:40 | |
| *** hamzy has joined #openstack-infra | 20:40 | |
| *** xek has quit IRC | 20:41 | |
| corvus | clarkb, Shrews: before 11 and after 2 eastern work better for me; but you don't *need* me, so if you want to do that i can catch up later. | 20:42 |
| fungi | also i had to bail on the conference so am at home and can help at any of those times | 20:43 |
| *** agopi has quit IRC | 20:44 | |
| clarkb | corvus: ok, mostly want to do it early enough that the release team can do things later in the day (sounds like they may be behind on release stuff so us getting done earlier is better than later)> I could possibly do 7am Pacific but I may not be very useful :) | 20:44 |
| clarkb | lets pencil in 7am pacific if that works for shrews and fungi | 20:45 |
| clarkb | then I will pretend it is a fishign trip and wake up early | 20:45 |
| fungi | fishing for server upgrades | 20:45 |
| *** openstackgerrit has joined #openstack-infra | 20:46 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Add epel element to centos7 testing https://review.openstack.org/612636 | 20:46 |
| corvus | oh, wow, i guess i could have gone to ato. oh well. | 20:46 |
| clarkb | the changes to do the switchover should all be pushed. The only work outstanding is coordinating the shutdown, update, start and restore | 20:46 |
| fungi | corvus: oh, were you in nc and didn't realize it was ato week? | 20:46 |
| corvus | fungi: yep! i'm good at calendars and stuff. | 20:47 |
| * fungi just assumed you were in this tz for the conference | 20:47 | |
| *** carl_cai has joined #openstack-infra | 20:48 | |
| clarkb | I kinda want to do the terrible thing of stopping zk01 now to see what happens :) | 20:49 |
| clarkb | maybe we do that after all the images are built tomorrow | 20:49 |
| corvus | clarkb: be a chaos monkey. we'll be fine. :) | 20:50 |
| clarkb | in this particular moment builders continue to look happy so I am going to step away from the computer for "lunch" | 20:51 |
| corvus | (it should only set the image build process back a little) | 20:51 |
| clarkb | I'll keep and eye on the build process and maybe be a chaos monkey too. Also review the storyboard attachments spec when I get back | 20:51 |
| clarkb | fwiw the zk nodes are in cacti too so we'll have good data as we turn stuff up I hope | 20:51 |
| clarkb | `echo stat | nc localhost 2181` is the incantation to see zk server stats and leader/follower info | 20:52 |
| clarkb | if anyone is wondering | 20:52 |
| Shrews | oh, if we restart the launchers, we'll probably need to consider ianw's change (https://review.openstack.org/605898) that modifies stats labels (assuming it merges) | 20:53 |
| Shrews | i'm suspecting that will break our graphs | 20:53 |
| clarkb | Shrews: I think we can clean that up as a followon | 20:53 |
| clarkb | won't impact functionality, just reporting | 20:53 |
| openstackgerrit | Merged openstack-infra/zuul master: Run zookeeper datadir on tmpfs during testing https://review.openstack.org/612766 | 20:54 |
| ianw | Shrews: yeah, i planned to update graphs with those stats :) | 20:55 |
| *** bobh has joined #openstack-infra | 20:56 | |
| *** bobh has quit IRC | 20:59 | |
| *** bobh_ has joined #openstack-infra | 20:59 | |
| amorin | ianw: if api is stable enough, i'll dig into port leak tomorrow | 21:00 |
| *** bobh_ has quit IRC | 21:00 | |
| ianw | amorin: ++ thanks | 21:01 |
| *** bobh has joined #openstack-infra | 21:01 | |
| *** e0ne has quit IRC | 21:04 | |
| *** bobh has quit IRC | 21:06 | |
| *** trown is now known as trown|outtypewww | 21:08 | |
| *** priteau has quit IRC | 21:09 | |
| clarkb | two images have been built and are being uploaded. Continues to look happy | 21:09 |
| *** felipemonteiro has joined #openstack-infra | 21:13 | |
| *** xek has joined #openstack-infra | 21:14 | |
| *** gfidente has quit IRC | 21:14 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** xek has quit IRC | 21:17 | |
| *** dklyle has quit IRC | 21:19 | |
| *** dklyle has joined #openstack-infra | 21:20 | |
| *** jamesmcarthur has quit IRC | 21:29 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard-webclient master: Show Email Addresses when Searching https://review.openstack.org/589713 | 21:29 |
| clarkb | diablo_rojo: I've reviewed the spec | 21:34 |
| diablo_rojo | clarkb, awesome! Thank you. | 21:34 |
| diablo_rojo | I'll give it another day or two and have updates up after that. | 21:34 |
| *** boden has quit IRC | 21:37 | |
| *** spsurya has quit IRC | 21:38 | |
| *** slaweq has quit IRC | 21:39 | |
| *** rtjure has quit IRC | 21:39 | |
| *** agopi has joined #openstack-infra | 21:48 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Run test zookeeper on top of tmpfs https://review.openstack.org/612816 | 21:49 |
| *** felipemonteiro has quit IRC | 21:52 | |
| *** felipemonteiro has joined #openstack-infra | 21:54 | |
| *** slaweq has joined #openstack-infra | 22:05 | |
| *** felipemonteiro has quit IRC | 22:12 | |
| *** eharney has quit IRC | 22:13 | |
| *** smarcet has quit IRC | 22:18 | |
| openstackgerrit | Andrey Volkov proposed openstack-infra/project-config master: New Airship project - Utils https://review.openstack.org/612820 | 22:20 |
| *** kjackal has quit IRC | 22:23 | |
| *** kjackal_v2 has joined #openstack-infra | 22:23 | |
| *** rcernin has joined #openstack-infra | 22:24 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Add epel element to centos7 testing https://review.openstack.org/612636 | 22:25 |
| *** bnemec has quit IRC | 22:25 | |
| *** rh-jelabarre has quit IRC | 22:26 | |
| *** mriedem has quit IRC | 22:26 | |
| ianw | cgoncalves: ^ hrm "mirrorlist" v "metalink" in the repo ... i'm not sure why i saw it working in the octavia gate | 22:27 |
| *** ansmith has joined #openstack-infra | 22:29 | |
| *** slaweq has quit IRC | 22:38 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: Add the process environment to zuul.conf parser https://review.openstack.org/612824 | 22:47 |
| *** kjackal_v2 has quit IRC | 22:48 | |
| *** carl_cai has quit IRC | 22:54 | |
| *** diablo_rojo has quit IRC | 22:57 | |
| *** hamzy has quit IRC | 22:59 | |
| *** threestrands has joined #openstack-infra | 23:02 | |
| *** tosky has quit IRC | 23:02 | |
| *** adriant has joined #openstack-infra | 23:08 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Do not merge https://review.openstack.org/612828 | 23:08 |
| *** diablo_rojo has joined #openstack-infra | 23:08 | |
| *** tpsilva has quit IRC | 23:11 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** rlandy is now known as rlandy|bbl | 23:17 | |
| *** xarses_ has quit IRC | 23:32 | |
| *** xarses_ has joined #openstack-infra | 23:33 | |
| *** sthussey has quit IRC | 23:38 | |
| *** roman_g has quit IRC | 23:43 | |
| *** slaweq has quit IRC | 23:45 | |
| *** roman_g has joined #openstack-infra | 23:45 | |
| *** gyee has quit IRC | 23:46 | |
| *** smarcet has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!