| *** weifan has quit IRC | 00:01 | |
| *** mattw4 has quit IRC | 00:19 | |
| fungi | ianw: still no working ipv6 on it though, right? | 00:22 |
|---|---|---|
| *** mattw4 has joined #openstack-infra | 00:22 | |
| fungi | and sorry, freak storm has knocked out all broadband to the island so after dinner i gave up waiting for them to fix it and broke out the wireless modem to be able to check in | 00:22 |
| ianw | fungi: yeah, ipv6 is enabled in the security groups | 00:38 |
| *** markvoelker has joined #openstack-infra | 00:40 | |
| ianw | i'm just going to quickly reboot to get to the afs-next @9f4a9105 as requested by upstream | 00:43 |
| ianw | well, that didn't fix ipv6 either | 00:45 |
| *** owalsh has quit IRC | 00:47 | |
| *** mattw4 has quit IRC | 00:47 | |
| *** yamamoto has quit IRC | 00:49 | |
| ianw | no matter what i tried yesterday, i couldn't get mirror.iad talking to the log host on port 6666 without putting them both into an open security group | 00:49 |
| *** owalsh has joined #openstack-infra | 00:50 | |
| *** ricolin has joined #openstack-infra | 00:55 | |
| *** markvoelker has quit IRC | 00:59 | |
| ianw | ok, mirror iad can't ping it's gateway "ping -6 fe80::def%eth0" | 01:02 |
| *** owalsh has quit IRC | 01:17 | |
| *** sreejithp has quit IRC | 01:21 | |
| *** owalsh has joined #openstack-infra | 01:22 | |
| *** smarcet has joined #openstack-infra | 01:31 | |
| *** michael-beaver has quit IRC | 01:39 | |
| *** eernst has joined #openstack-infra | 01:42 | |
| *** eernst has quit IRC | 01:46 | |
| *** markvoelker has joined #openstack-infra | 01:56 | |
| *** apetrich has quit IRC | 01:57 | |
| *** mattw4 has joined #openstack-infra | 01:57 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add reboot-host role https://review.opendev.org/666748 | 01:58 |
| *** goldyfruit has quit IRC | 01:59 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] testing reboot host role https://review.opendev.org/666749 | 02:00 |
| *** markvoelker has quit IRC | 02:01 | |
| *** smarcet has quit IRC | 02:04 | |
| *** Goneri has quit IRC | 02:09 | |
| *** goldyfruit has joined #openstack-infra | 02:17 | |
| *** smarcet has joined #openstack-infra | 02:24 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] testing reboot host role https://review.opendev.org/666749 | 02:31 |
| *** smarcet has quit IRC | 02:41 | |
| *** smarcet has joined #openstack-infra | 02:42 | |
| *** markvoelker has joined #openstack-infra | 02:57 | |
| *** markvoelker has quit IRC | 03:02 | |
| ianw | oh hrm, zuul-jobs isn't in system-config, oh well | 03:04 |
| *** psachin has joined #openstack-infra | 03:06 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] testing reboot host stream reconnnect https://review.opendev.org/666749 | 03:07 |
| *** smarcet has left #openstack-infra | 03:08 | |
| *** yamamoto has joined #openstack-infra | 03:11 | |
| *** ykarel has joined #openstack-infra | 03:20 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:24 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:34 | |
| *** bhavikdbavishi has quit IRC | 03:36 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:36 | |
| *** smarcet has joined #openstack-infra | 03:39 | |
| *** yamamoto has quit IRC | 03:53 | |
| *** smarcet has quit IRC | 03:58 | |
| *** markvoelker has joined #openstack-infra | 03:58 | |
| *** yamamoto has joined #openstack-infra | 04:01 | |
| *** markvoelker has quit IRC | 04:02 | |
| *** udesale has joined #openstack-infra | 04:05 | |
| *** dave-mccowan has quit IRC | 04:09 | |
| *** slaweq has joined #openstack-infra | 04:14 | |
| *** ramishra has joined #openstack-infra | 04:17 | |
| amorin | fungi, corvus, about that instance without any IP, I also noticed that from time to time,you were deleting some ports on neutron side before deleting the instance | 04:20 |
| amorin | it's triggering a race condition in nova side on our hypervisors | 04:20 |
| amorin | I am wondering if this is a normal behavior | 04:20 |
| amorin | maybe that's what happened on your instance | 04:20 |
| *** zzehring has quit IRC | 04:24 | |
| *** slaweq has quit IRC | 04:25 | |
| *** ykarel has quit IRC | 04:34 | |
| *** whoami-rajat has joined #openstack-infra | 04:35 | |
| *** bobh has joined #openstack-infra | 04:39 | |
| *** mattw4 has quit IRC | 04:39 | |
| *** mhu has quit IRC | 04:44 | |
| *** mhu has joined #openstack-infra | 04:44 | |
| *** ianw is now known as ianw_pto | 04:44 | |
| *** pcaruana has joined #openstack-infra | 04:45 | |
| *** bobh has quit IRC | 04:47 | |
| *** ykarel has joined #openstack-infra | 04:49 | |
| *** markvoelker has joined #openstack-infra | 04:59 | |
| *** kjackal has joined #openstack-infra | 05:02 | |
| *** janki has joined #openstack-infra | 05:02 | |
| *** sreejithp has joined #openstack-infra | 05:03 | |
| *** sreejithp has quit IRC | 05:03 | |
| *** markvoelker has quit IRC | 05:04 | |
| *** kopecmartin|off is now known as kopecmartin | 05:12 | |
| *** jbadiapa has quit IRC | 05:13 | |
| *** virendra has joined #openstack-infra | 05:32 | |
| *** ykarel_ has joined #openstack-infra | 05:32 | |
| *** ykarel has quit IRC | 05:34 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Filter out unprotected branches from builds if excluded https://review.opendev.org/666664 | 05:48 |
| *** rpioso is now known as rpioso|afk | 05:56 | |
| *** rcernin has quit IRC | 05:59 | |
| *** markvoelker has joined #openstack-infra | 06:00 | |
| *** ykarel_ is now known as ykarel|meeting | 06:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/665869 | 06:04 |
| *** markvoelker has quit IRC | 06:04 | |
| *** pgaxatte has joined #openstack-infra | 06:15 | |
| *** jtomasek has joined #openstack-infra | 06:16 | |
| *** jtomasek has quit IRC | 06:17 | |
| *** jtomasek has joined #openstack-infra | 06:17 | |
| *** e0ne has joined #openstack-infra | 06:22 | |
| *** dpawlik has joined #openstack-infra | 06:33 | |
| *** udesale has quit IRC | 06:36 | |
| *** udesale has joined #openstack-infra | 06:37 | |
| *** janki has quit IRC | 06:53 | |
| *** ykarel_ has joined #openstack-infra | 06:59 | |
| *** jbadiapa has joined #openstack-infra | 07:00 | |
| *** markvoelker has joined #openstack-infra | 07:01 | |
| *** ykarel|meeting has quit IRC | 07:01 | |
| *** slaweq has joined #openstack-infra | 07:04 | |
| *** iurygregory has joined #openstack-infra | 07:10 | |
| *** ykarel_ is now known as ykarel | 07:13 | |
| *** ginopc has joined #openstack-infra | 07:17 | |
| *** tesseract has joined #openstack-infra | 07:19 | |
| *** tosky has joined #openstack-infra | 07:21 | |
| *** mhu has quit IRC | 07:27 | |
| *** mhu has joined #openstack-infra | 07:27 | |
| *** markvoelker has quit IRC | 07:28 | |
| *** tkajinam has quit IRC | 07:31 | |
| *** janki has joined #openstack-infra | 07:31 | |
| *** ccamacho has joined #openstack-infra | 07:36 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 07:36 |
| *** jpena|off is now known as jpena | 07:37 | |
| *** dchen has quit IRC | 07:39 | |
| *** dchen has joined #openstack-infra | 07:40 | |
| *** ricolin_ has joined #openstack-infra | 07:40 | |
| *** ricolin has quit IRC | 07:43 | |
| *** tkajinam has joined #openstack-infra | 07:44 | |
| *** rpittau|afk is now known as rpittau | 07:47 | |
| *** dchen has quit IRC | 07:52 | |
| *** ralonsoh has joined #openstack-infra | 07:52 | |
| *** ykarel has quit IRC | 07:53 | |
| *** ykarel has joined #openstack-infra | 07:55 | |
| *** ykarel is now known as ykarel|lunch | 07:56 | |
| *** jpich has joined #openstack-infra | 07:56 | |
| *** rascasoft has joined #openstack-infra | 08:02 | |
| ralonsoh | hello, I have an small issue here | 08:05 |
| ralonsoh | http://logs.openstack.org/83/666583/1/check/grenade-py3/34faefe/logs/devstack-gate-setup-host.txt | 08:05 |
| ralonsoh | 2019-06-20 18:47:57.419 | localhost | FAILED | network_sanity_check : Perform ping check | rc=1 >> | 08:05 |
| ralonsoh | PING mirror.iad.rax.opendev.org(2001:4802:7802:104:be76:4eff:fe20:4b35 (2001:4802:7802:104:be76:4eff:fe20:4b35)) 56 data bytes | 08:05 |
| ralonsoh | the grenade job is trying to ping to mirror.iad.rax.opendev.org and it's resolving the ipv6 address | 08:06 |
| ralonsoh | but is failing | 08:06 |
| ralonsoh | did you see this error before? | 08:06 |
| *** apetrich has joined #openstack-infra | 08:07 | |
| slaweq | frickler: infra-root: can You check what ralonsoh said ^^? Thx in advance | 08:07 |
| *** udesale has quit IRC | 08:11 | |
| *** dirk has joined #openstack-infra | 08:11 | |
| *** udesale has joined #openstack-infra | 08:12 | |
| *** rascasoft has quit IRC | 08:15 | |
| *** rascasoft has joined #openstack-infra | 08:15 | |
| *** lucasagomes has joined #openstack-infra | 08:17 | |
| *** virendra has quit IRC | 08:25 | |
| *** virendra has joined #openstack-infra | 08:26 | |
| virendra | can anyone suggest, is there an issue with CI-WATCH ? unable to see newly added patchset log on it. | 08:27 |
| *** janki has quit IRC | 08:28 | |
| *** tkajinam_ has joined #openstack-infra | 08:30 | |
| *** tkajinam has quit IRC | 08:33 | |
| *** tkajinam_ has quit IRC | 08:35 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| *** imacdonn has quit IRC | 08:41 | |
| *** imacdonn has joined #openstack-infra | 08:41 | |
| *** ociuhandu has joined #openstack-infra | 08:46 | |
| *** ociuhandu has quit IRC | 08:46 | |
| *** ociuhandu has joined #openstack-infra | 08:46 | |
| *** ociuhandu has quit IRC | 08:50 | |
| *** ociuhandu has joined #openstack-infra | 08:50 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 08:56 |
| *** ykarel|lunch is now known as ykarel | 08:56 | |
| *** rascasoft has quit IRC | 08:56 | |
| *** janki has joined #openstack-infra | 08:57 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 09:03 |
| *** owalsh has left #openstack-infra | 09:06 | |
| *** tosky__ has joined #openstack-infra | 09:06 | |
| *** ykarel has quit IRC | 09:07 | |
| *** ginopc has quit IRC | 09:07 | |
| *** ginopc has joined #openstack-infra | 09:07 | |
| *** tosky has quit IRC | 09:08 | |
| *** ricolin_ has quit IRC | 09:10 | |
| *** ricolin has joined #openstack-infra | 09:11 | |
| derekh | Seems to be a problem with http://codesearch.openstack.org/ - "ERROR:open /home/hound/data/idx-85adc3f221c62aae/raw/LICENSE: too many open files" | 09:20 |
| *** owalsh has joined #openstack-infra | 09:27 | |
| *** tosky__ is now known as tosky | 09:43 | |
| *** lpetrut has joined #openstack-infra | 09:45 | |
| *** gfidente has quit IRC | 09:55 | |
| *** ykarel_ has joined #openstack-infra | 09:57 | |
| *** ricolin has quit IRC | 09:57 | |
| *** ykarel_ is now known as ykarel | 09:58 | |
| *** spsurya has joined #openstack-infra | 10:07 | |
| *** lpetrut has quit IRC | 10:10 | |
| *** lpetrut has joined #openstack-infra | 10:10 | |
| *** gfidente has joined #openstack-infra | 10:18 | |
| *** Wasaac has joined #openstack-infra | 10:24 | |
| *** yamamoto has quit IRC | 10:25 | |
| *** udesale has quit IRC | 10:31 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 10:38 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 10:39 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 10:40 |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** yamamoto has quit IRC | 10:59 | |
| *** yamamoto has joined #openstack-infra | 11:00 | |
| *** Wasaac has quit IRC | 11:04 | |
| *** jbadiapa has quit IRC | 11:09 | |
| zbr|ruck | does anyone knows zuul returns empty status on changes? like https://zuul.opendev.org/api/tenant/openstack/status/change/645273 | 11:14 |
| *** janki has quit IRC | 11:14 | |
| zbr|ruck | i see this behavior more and more often, and while going to it own status page I can see the status, but api returns empty response "[]" | 11:14 |
| *** smarcet has joined #openstack-infra | 11:16 | |
| *** ginux has joined #openstack-infra | 11:19 | |
| *** janki has joined #openstack-infra | 11:19 | |
| *** ginux is now known as Guest88123 | 11:19 | |
| *** ginopc has quit IRC | 11:21 | |
| *** Guest88123 is now known as ginopc | 11:21 | |
| *** Lucas_Gray has joined #openstack-infra | 11:21 | |
| *** Goneri has joined #openstack-infra | 11:21 | |
| *** jpena is now known as jpena|lunch | 11:26 | |
| *** jpich has quit IRC | 11:35 | |
| *** jpich has joined #openstack-infra | 11:36 | |
| *** ykarel is now known as ykarel|away | 11:51 | |
| *** ykarel_ has joined #openstack-infra | 11:54 | |
| *** ykarel|away has quit IRC | 11:56 | |
| *** EmilienM is now known as EvilienM | 11:59 | |
| *** udesale has joined #openstack-infra | 12:10 | |
| *** ykarel_ has quit IRC | 12:11 | |
| *** priteau has joined #openstack-infra | 12:16 | |
| *** ab-a has quit IRC | 12:18 | |
| *** virendra has quit IRC | 12:18 | |
| *** goldyfruit has quit IRC | 12:22 | |
| *** ab-a has joined #openstack-infra | 12:24 | |
| *** ekultails has joined #openstack-infra | 12:29 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 12:35 |
| *** rlandy has joined #openstack-infra | 12:36 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 12:36 |
| *** priteau has quit IRC | 12:36 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 12:36 |
| *** priteau has joined #openstack-infra | 12:40 | |
| *** vkmc has quit IRC | 12:40 | |
| *** jamielennox has quit IRC | 12:41 | |
| fungi | amorin: that's an interesting observation... i don't think we manage neutron ports independently with nodepool, except to clean up ports which are already in a "down" state for at least three minutes (normally we expect instance deletion to cause nova to tell neutron to delete the associated ports associated, but in some clouds we've seen nova's request to neutron time out and then the instance gets | 12:47 |
| fungi | deleted but the ports hang around indefinitely consuming port quota) https://zuul-ci.org/docs/nodepool/releasenotes.html#relnotes-3-3-1-new-features | 12:48 |
| amorin | hum | 12:48 |
| fungi | though the mystery instance was in use and still locked by zuul, so unless the port for it went down for an unrelated reason and stayed down for at least 3 minutes, i wouldn't expect nodepool to remove it with the instance still up | 12:48 |
| *** jbadiapa has joined #openstack-infra | 12:48 | |
| *** ginux has joined #openstack-infra | 12:50 | |
| amorin | the result of this is that, if the instance scheduling took more that 3 minutes, then the port is deleted while nova was trying to boot the instance | 12:50 |
| *** ginux is now known as Guest67129 | 12:50 | |
| fungi | ralonsoh: slaweq: the current mirror.iad.rax.opendev.org has no working ipv6 connectivity, we have a trouble ticket open with the provider about it. i'll propose a change to remove its aaaa record from dns for now | 12:50 |
| *** ginopc has quit IRC | 12:50 | |
| *** Guest67129 has quit IRC | 12:53 | |
| slaweq | fungi: thx a lot | 12:53 |
| amorin | we have a lot of edge cases on OSF aggregate with nova booting the instance, plugging the interface, but the port was deleted in the meantime, so neutron do nothing and the instance boot timeout waiting for a reply from neutron | 12:54 |
| *** ginopc has joined #openstack-infra | 12:55 | |
| *** michael-beaver has joined #openstack-infra | 12:55 | |
| openstackgerrit | Jeremy Stanley proposed opendev/zone-opendev.org master: Temporarily remove AAAA for mirror01.iad.rax https://review.opendev.org/666830 | 12:56 |
| fungi | infra-root: it would get good to get ^ merged quickly | 12:57 |
| fungi | derekh: hound seems to run into open files issues from time to time. i'll take a quick look | 12:57 |
| fungi | derekh: codesearch seems to be working for me at the moment. were you getting that error on a particular search? is it still happening? | 12:59 |
| *** jpena|lunch is now known as jpena | 12:59 | |
| fungi | zbr|ruck: i get that behavior if i don't include the patchset index (i.e. of i request 662265 instead of 662265,1) | 13:02 |
| *** rfolco has joined #openstack-infra | 13:02 | |
| *** sthussey has joined #openstack-infra | 13:02 | |
| fungi | amorin: thanks, i'll noodle on that with some of the nodepool maintainers. they probably didn't consider that a port might be created in a down state in advance of the instance being scheduled | 13:03 |
| amorin | fungi: thanks! | 13:04 |
| *** ykarel_ has joined #openstack-infra | 13:05 | |
| *** rascasoft has joined #openstack-infra | 13:08 | |
| ralonsoh | fungi, thanks! | 13:08 |
| *** mriedem has joined #openstack-infra | 13:08 | |
| *** ykarel_ is now known as ykarel | 13:09 | |
| derekh | fungi: this was the search, same error happened just now http://codesearch.openstack.org/?q=provisioning%20was%20aborted&i=nope&files=&repos= | 13:13 |
| derekh | fungi: but then worked a few seconds later | 13:14 |
| *** goldyfruit has joined #openstack-infra | 13:15 | |
| *** jchhatbar has joined #openstack-infra | 13:18 | |
| fungi | derekh: yeah, its log was also complaining about vcs pull/sync errors due to pipe calls getting refused for the same reason. i'm restarting the service now to hopefully free up leaked file descriptors | 13:19 |
| fungi | it'll take a few minutes to get back up and running | 13:20 |
| derekh | fungi: no problem, thanks | 13:20 |
| fungi | #status log restarted hound on codesearch.o.o due to persistent "too many open files" errors | 13:20 |
| openstackstatus | fungi: finished logging | 13:20 |
| *** janki has quit IRC | 13:21 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 13:21 |
| *** zzehring has joined #openstack-infra | 13:23 | |
| *** aedc has joined #openstack-infra | 13:26 | |
| *** rpioso|afk is now known as rpioso | 13:26 | |
| *** markvoelker has joined #openstack-infra | 13:29 | |
| *** dave-mccowan has joined #openstack-infra | 13:30 | |
| *** FlorianFa has joined #openstack-infra | 13:30 | |
| *** yboaron_ has joined #openstack-infra | 13:31 | |
| *** yboaron_ has quit IRC | 13:35 | |
| *** eharney has joined #openstack-infra | 13:38 | |
| *** priteau has quit IRC | 13:41 | |
| *** sreejithp has joined #openstack-infra | 13:41 | |
| zbr|ruck | fungi: yep, you are right, my mistake. what i still do not understand is why zuul_statics script fails to gather status on some changes (apparently not clear pattern found yet). i double checked and the script does query with change,patch always. | 13:41 |
| *** ginopc has quit IRC | 13:42 | |
| *** ginopc has joined #openstack-infra | 13:42 | |
| *** bhavikdbavishi has quit IRC | 13:43 | |
| *** dims has quit IRC | 13:45 | |
| *** jchhatbar has quit IRC | 13:47 | |
| *** priteau has joined #openstack-infra | 13:51 | |
| *** ginopc has quit IRC | 13:53 | |
| *** dims has joined #openstack-infra | 13:53 | |
| fungi | race condition between queuing and status being provided by the api maybe? | 13:57 |
| *** markvoelker has quit IRC | 13:57 | |
| *** jamesmcarthur has joined #openstack-infra | 13:58 | |
| *** ginopc has joined #openstack-infra | 13:59 | |
| *** dims has quit IRC | 14:01 | |
| *** dims has joined #openstack-infra | 14:03 | |
| corvus | zbr|ruck: don't forget the ",patchset" part of the change id | 14:17 |
| *** jamesmcarthur has quit IRC | 14:18 | |
| *** dpawlik has quit IRC | 14:20 | |
| *** smarcet has quit IRC | 14:24 | |
| *** apetrich has quit IRC | 14:26 | |
| *** ykarel has quit IRC | 14:26 | |
| *** ykarel has joined #openstack-infra | 14:26 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Temporarily remove AAAA for mirror01.iad.rax https://review.opendev.org/666830 | 14:27 |
| *** rajinir has joined #openstack-infra | 14:30 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 14:31 |
| *** lpetrut has quit IRC | 14:32 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 14:33 |
| *** yboaron_ has joined #openstack-infra | 14:34 | |
| *** whoami-rajat has quit IRC | 14:34 | |
| *** dklyle has joined #openstack-infra | 14:37 | |
| *** yboaron_ has quit IRC | 14:39 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Increase port cleanup interval https://review.opendev.org/666852 | 14:40 |
| *** lseki has joined #openstack-infra | 14:42 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 14:42 |
| *** rh-jelabarre has quit IRC | 14:43 | |
| *** bnemec is now known as beekneemech | 14:44 | |
| *** TheJulia is now known as needssleep | 14:46 | |
| fungi | amorin: do you think 10 minutes like in https://review.opendev.org/666852 would be enough time to take care of most of those races you're seeing? | 14:46 |
| clarkb | note corvus says itmarks on the first pass and deletes on the second | 14:47 |
| *** kopecmartin is now known as kopecmartin|off | 14:47 | |
| clarkb | so should result in 20 minutes before deleting | 14:47 |
| corvus | clarkb: 10-20m | 14:48 |
| corvus | 10-19:99.99999 min :) | 14:48 |
| *** rh-jelabarre has joined #openstack-infra | 14:48 | |
| clarkb | oh right since it is a cron | 14:49 |
| fungi | yeah, could be as short as 10 minutes | 14:49 |
| fungi | on average 15, which is probably what really matters for statistics over racing slow instance scheduling | 14:49 |
| fungi | since we assume there's always going to be an over/under anyway and the goal is to simply lose that race an acceptable percentage of the time | 14:53 |
| *** markvoelker has joined #openstack-infra | 14:54 | |
| *** markvoelker has quit IRC | 14:58 | |
| slaweq | fungi: hi, I have one additional question about redirect from https://github.com/openstack/tobiko to some other mirror | 14:58 |
| slaweq | fungi: can it be just redirected to https://opendev.org/x/tobiko ? | 14:59 |
| slaweq | fungi: or do we need some other mirror for it? | 14:59 |
| fungi | slaweq: github only allows you to redirect repositories to other places on github, not to external sites | 14:59 |
| slaweq | fungi: ok, thx | 15:00 |
| fungi | https://opendev.org/openstack/tobiko already redirects to https://opendev.org/x/tobiko of course | 15:00 |
| fungi | slaweq: we could simply delete https://github.com/openstack/tobiko if you like, to avoid confusion | 15:00 |
| fungi | (assuming you're one of its maintainers, i haven't checked) | 15:01 |
| slaweq | fungi: deletion of this repo from github would be great for us | 15:01 |
| slaweq | fungi: I'm not | 15:01 |
| slaweq | fungi: I will tell fressi who is maintainer of this repo to confirm it to You :) | 15:01 |
| slaweq | fungi: thx for info, I think that we will ping You on Monday | 15:02 |
| fungi | slaweq: thanks, looks like if fressi or abregman request it we'll be fine according to https://review.opendev.org/#/admin/groups/tobiko-release | 15:03 |
| slaweq | yes, I just wrote this to them | 15:04 |
| slaweq | but they are not working today so they will contact You next week | 15:04 |
| slaweq | thx a lot for help | 15:04 |
| fungi | no worries, happy to help whenever they're available | 15:04 |
| *** ykarel is now known as ykarel|away | 15:05 | |
| clarkb | trying to catch up on things in general, do we know why iad mirror is not happy with ipv6? | 15:08 |
| fungi | infra-root: fanatical support got back to us on the mirror.iad.rax.opendev.org ipv6 connectivity problem suggesting a host migration. i replied that they can do a reboot migration at any time. the outage would be brief enough and things have been strange enough with responses from that mirror over the course of the week that i expect it will go unnoticed | 15:08 |
| fungi | but i'll status log time of the reboot for correlation purposes | 15:08 |
| clarkb | oh ok so hypervisor problem possibly? | 15:09 |
| fungi | possibly | 15:09 |
| clarkb | where did we end up with afs on bionic generally? is the rc kernel much more stable? | 15:09 |
| fungi | we're now on a non-mainline build with some proposed commits from the kafs maintainer(s) | 15:10 |
| fungi | but yes, haven't seen the server spontaneously enter shutoff state for a couple days now | 15:10 |
| fungi | and i haven't seen new complaints about stale content, but it's also possible we just haven't had the right conditions to notice | 15:11 |
| clarkb | we can check logstash if we think we are stable otherwise | 15:12 |
| *** bobh has joined #openstack-infra | 15:13 | |
| fungi | yeah, that's why i'm going to mark the reboot time so if we see a cluster of errors around it then we know it was probably that | 15:13 |
| clarkb | ++ | 15:13 |
| fungi | i'll follow ianw_pto's notes in https://etherpad.openstack.org/p/opendev-mirror-afs to get the netconsole going again once it's back up | 15:13 |
| fungi | and assuming that fixes the connectivity, i'll push up a revert for 666830 | 15:14 |
| fungi | for the record, uname -a says we're presently booted on 5.2.0-rc1-afs-next-9f4a9105 | 15:15 |
| fungi | presumably that's what we'll come back up with, but i'll double-check of course | 15:16 |
| fungi | and also modprobe kafs and remount /afs if the requisite changes haven't merged yet | 15:16 |
| *** tosky has quit IRC | 15:18 | |
| clarkb | fungi: have pointers to those changes? | 15:18 |
| fungi | clarkb: https://review.opendev.org/#/q/topic:kafs | 15:19 |
| fungi | i think 623974 is the one that sets up the kafs module autoloading and /afs mounting at boot (among other things) | 15:20 |
| *** Lucas_Gray has quit IRC | 15:21 | |
| clarkb | but the bsae of the stack is replacing ntpd | 15:21 |
| clarkb | I'll start there | 15:21 |
| fungi | clarkb: or you could continue your vacation? | 15:21 |
| clarkb | fungi: well technically I planned to be semi back today. I'm no longer in the desert | 15:22 |
| clarkb | But it likely will be a slow recovery day :) | 15:22 |
| fungi | ahh, too bad for you! | 15:22 |
| fungi | vacations should last through the weekend in my opinion ;) | 15:22 |
| corvus | clarkb: i also saved the +3 on 662657 for you since you had previous comments | 15:22 |
| corvus | fungi is correct | 15:22 |
| clarkb | ya'll are making me rethink my day :) | 15:23 |
| clarkb | clearly no one would notice if I just cruised through to monday | 15:23 |
| fungi | you really should. taking vacation for a week should mean taking vacation for a _week_ | 15:24 |
| corvus | not taking a weak vacation | 15:24 |
| fungi | yes! | 15:24 |
| fungi | there's nothing so critical here it can't wait | 15:24 |
| clarkb | fwiw the rock digging was a huge success, though I did not expect it to be quite so easy | 15:24 |
| clarkb | you bsaically dig thundereggs out of a layer of ancient volcano ash and that layer is full of them | 15:25 |
| fungi | rocks conveniently placed near the surface of the earth's crust | 15:25 |
| fungi | nature is more convenient than folks give it credit for | 15:25 |
| fungi | our planet is waaaay more convenient than most of the other ones i've seen out there. at least for the moment | 15:25 |
| corvus | this one looks very excited to have been found: http://www.geologypage.com/wp-content/uploads/2016/10/Thunder-egg-696x702.jpg | 15:27 |
| *** mudpuppy has joined #openstack-infra | 15:28 | |
| *** cmurphy is now known as cmorpheus | 15:28 | |
| clarkb | none of ours are quite so interesting (though we only cut a handful) | 15:28 |
| clarkb | supposedly the rock museum near us will cut rocks for a fraction of the price so have to go and do that later | 15:29 |
| corvus | also, you never know if they'll hatch later | 15:29 |
| clarkb | Ok the ntp change and the bup change both +2'd but not approved as I won't be around to watch those | 15:32 |
| clarkb | you've convinced me to make it a lazier friday | 15:32 |
| fungi | then i have succeeded! | 15:33 |
| fungi | thanks for the reviews, i'll approve those once the server migration is behind us so we don't get confused by two many behavior changes at once | 15:34 |
| fungi | also too many | 15:34 |
| fungi | possibly even more than two many | 15:34 |
| fungi | fanatical support confirms the migration request has been initiated and estimates 1.5 hours for the data transfer to complete at which point the reboot will happen | 15:36 |
| fungi | update was ~15 minutes ago, so probably by 17:00z | 15:36 |
| zbr|ruck | sorry to ask again but where was the code that determine what is sent to logstash? i have one or two key files which I want to index as they contains often fatal errors. | 15:37 |
| *** goldyfruit has quit IRC | 15:41 | |
| auristor | how is the iad mirror running kafs? | 15:41 |
| fungi | auristor: doing well. no spontaneous shutdowns now and haven't gotten any new reports of stale content | 15:42 |
| auristor | great | 15:42 |
| clarkb | zbr|ruck: https://opendev.org/openstack/project-config/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml | 15:42 |
| fungi | auristor: currently booted from 5.2.0-rc1-afs-next-9f4a9105 | 15:43 |
| *** bobh has quit IRC | 15:46 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: DNM: fail devstack jobs https://review.opendev.org/666880 | 15:46 |
| *** gyee has joined #openstack-infra | 15:46 | |
| *** yamamoto has quit IRC | 15:48 | |
| *** yamamoto has joined #openstack-infra | 15:49 | |
| zbr|ruck | clarkb: thanks clark. | 15:52 |
| clarkb | https://i.imgur.com/EmutD2i.jpg bad photo of thundereggs | 15:52 |
| fungi | some neat formations | 15:53 |
| *** clarkb has quit IRC | 15:53 | |
| *** yamamoto has quit IRC | 15:54 | |
| fungi | #status log mirror.iad.rax.opendev.org restarted at 15:46 utc for host migration | 15:55 |
| openstackstatus | fungi: finished logging | 15:55 |
| *** markvoelker has joined #openstack-infra | 15:55 | |
| fungi | i've done the modprobe and mount on it | 15:55 |
| fungi | testing ipv6 connectivity now | 15:55 |
| *** jamesmcarthur has joined #openstack-infra | 15:56 | |
| fungi | still no v6 connectivity as far as i can tell | 15:56 |
| *** rpittau is now known as rpittau|afk | 15:57 | |
| *** markvoelker has quit IRC | 15:59 | |
| *** ginopc has quit IRC | 16:00 | |
| *** jpena is now known as jpena|off | 16:01 | |
| openstackgerrit | Merged zuul/nodepool master: Increase port cleanup interval https://review.opendev.org/666852 | 16:01 |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Include overcloud deploy in logstash config https://review.opendev.org/666883 | 16:02 |
| *** clarkb has joined #openstack-infra | 16:02 | |
| clarkb | apparently reloaded weechat plugins disconnects you from servers | 16:02 |
| *** lucasagomes has quit IRC | 16:04 | |
| *** jpich has quit IRC | 16:05 | |
| *** pgaxatte has quit IRC | 16:07 | |
| *** panda is now known as panda-pto | 16:08 | |
| clarkb | ok I think I've recovered enough of my irc client situation to call it a day :) | 16:11 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add command processor to zuul-web https://review.opendev.org/666307 | 16:11 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add repl server for debug purposes https://review.opendev.org/579962 | 16:12 |
| *** priteau has quit IRC | 16:17 | |
| zbr|ruck | clarkb: does my proposed patch ^ look ok for collecting overcloud_deploy.log file? | 16:18 |
| zbr|ruck | the file endsup as .log.txt.gz on logs server but I suspect that the pattern does not include the .gz part. | 16:19 |
| *** mattw4 has joined #openstack-infra | 16:21 | |
| *** icarusfactor has joined #openstack-infra | 16:22 | |
| *** factor has quit IRC | 16:24 | |
| *** mgoddard has quit IRC | 16:24 | |
| *** mgoddard has joined #openstack-infra | 16:24 | |
| *** spsurya has quit IRC | 16:24 | |
| *** factor__ has joined #openstack-infra | 16:26 | |
| *** icarusfactor has quit IRC | 16:26 | |
| *** mattw4 has quit IRC | 16:27 | |
| *** mattw4 has joined #openstack-infra | 16:28 | |
| *** factor__ has quit IRC | 16:29 | |
| *** factor__ has joined #openstack-infra | 16:29 | |
| *** gfidente has quit IRC | 16:35 | |
| *** factor__ has quit IRC | 16:38 | |
| *** factor__ has joined #openstack-infra | 16:38 | |
| *** udesale has quit IRC | 16:39 | |
| *** udesale has joined #openstack-infra | 16:39 | |
| *** jamesmcarthur has quit IRC | 16:44 | |
| *** udesale has quit IRC | 16:50 | |
| *** derekh has quit IRC | 16:51 | |
| *** woojay has quit IRC | 16:51 | |
| *** factor__ has quit IRC | 16:52 | |
| *** factor__ has joined #openstack-infra | 16:52 | |
| *** weifan has joined #openstack-infra | 16:55 | |
| *** markvoelker has joined #openstack-infra | 16:56 | |
| *** factor__ has quit IRC | 17:00 | |
| *** factor has joined #openstack-infra | 17:00 | |
| *** markvoelker has quit IRC | 17:01 | |
| *** diablo_rojo has joined #openstack-infra | 17:04 | |
| *** ralonsoh has quit IRC | 17:04 | |
| *** aedc has quit IRC | 17:07 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 17:07 |
| *** jamesmcarthur has joined #openstack-infra | 17:07 | |
| *** whoami-rajat has joined #openstack-infra | 17:09 | |
| *** smarcet has joined #openstack-infra | 17:19 | |
| *** jamesmcarthur has quit IRC | 17:23 | |
| *** igordc has joined #openstack-infra | 17:25 | |
| *** ykarel|away has quit IRC | 17:28 | |
| *** hwoarang has quit IRC | 17:31 | |
| *** hwoarang has joined #openstack-infra | 17:32 | |
| guilhermesp | hi everyone! could someone point me where you guys implements zuul logging? like ara reports and stuff? | 17:35 |
| corvus | guilhermesp: the zuul base jobs in the opendev/base-jobs repo deal with that | 17:36 |
| *** ociuhandu has quit IRC | 17:38 | |
| guilhermesp | thanks corvus ! | 17:39 |
| *** tesseract has quit IRC | 17:50 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 17:53 |
| *** markvoelker has joined #openstack-infra | 17:56 | |
| *** eharney has quit IRC | 18:01 | |
| *** markvoelker has quit IRC | 18:01 | |
| *** ramishra has quit IRC | 18:08 | |
| *** hwoarang has quit IRC | 18:13 | |
| *** hwoarang has joined #openstack-infra | 18:14 | |
| *** tosky has joined #openstack-infra | 18:19 | |
| *** factor has quit IRC | 18:28 | |
| *** icarusfactor has joined #openstack-infra | 18:28 | |
| *** iurygregory has quit IRC | 18:38 | |
| *** eharney has joined #openstack-infra | 18:40 | |
| *** smarcet has quit IRC | 18:44 | |
| *** goldyfruit has joined #openstack-infra | 18:51 | |
| *** icarusfactor has quit IRC | 18:57 | |
| *** icarusfactor has joined #openstack-infra | 18:57 | |
| *** markvoelker has joined #openstack-infra | 18:57 | |
| *** psachin has quit IRC | 18:59 | |
| *** goldyfruit has quit IRC | 19:16 | |
| openstackgerrit | Arun S A G proposed zuul/nodepool master: Clouds running nova-network won't have public network endpoints https://review.opendev.org/666905 | 19:18 |
| *** whoami-rajat has quit IRC | 19:19 | |
| yoctozepto | could you look at https://review.opendev.org/666881 - the failed job is missing from the summary table | 19:24 |
| yoctozepto | kolla-ansible-ubuntu-source-upgrade-ceph finger://ze06.openstack.org/dc855182298d4b4490b568b9122dbcb7 : RETRY_LIMIT in 56m 10s (non-voting) | 19:25 |
| yoctozepto | (and what about this finger url) | 19:25 |
| *** markvoelker has quit IRC | 19:26 | |
| fungi | yoctozepto: there's been talk of switching those to link to e.g. http://zuul.opendev.org/t/openstack/build/dc855182298d4b4490b568b9122dbcb7 now that we have that | 19:27 |
| fungi | but yeah, the finger url appearing there as a placeholder generally means we had no logs | 19:28 |
| yoctozepto | ok, so like an internal failure of Zuul? | 19:29 |
| fungi | also, there's work done to move console log collection so that it always happens, can't remember if that's merged yet but maybe it would have at least gotten a console log if not | 19:29 |
| fungi | yoctozepto: i think it usually means a very early (and consistent) failure from the first "pre" playbook run by the job, though i'm not positive | 19:30 |
| fungi | it could be anything which crashes the node before remote log collection is able to run | 19:30 |
| yoctozepto | I see, but it should be included in the table anyway, no? | 19:31 |
| fungi | the javacript which generates that table by parsing review comments expects very particular patterns which that line doesn't match | 19:33 |
| fungi | we've also been discussing doing https://review.opendev.org/655783 or something similar which only exposes the most recent comment | 19:34 |
| *** smarcet has joined #openstack-infra | 19:35 | |
| *** EvilienM is now known as EmilienM | 19:39 | |
| yoctozepto | I see, thanks, fungi! | 19:40 |
| fungi | yoctozepto: if you recheck and catch the change as soon as that job gets enqueued and then stream the console from it, you may be able to at least see what was going on immediately before the job died | 19:42 |
| fungi | er, rather, as soon as that job gets a node assigned and the job starts | 19:42 |
| fungi | easier to do when there's not much backlog | 19:42 |
| fungi | current node request backlog is ~800 and falling according to http://grafana.openstack.org/d/T6vSHcSik/zuul-status so once that hits 0 it should be fairly quick to do | 19:45 |
| fungi | maybe around 20:30z | 19:47 |
| *** yamamoto has joined #openstack-infra | 19:49 | |
| yoctozepto | fungi: the last time a different job failed so it is hard to choose one to track :-) | 19:51 |
| fungi | i see. so this seems to be random? not just a case of that change breaking that job? | 19:52 |
| yoctozepto | the error was unrelated to change being tested, so must be cosmic rays ;D | 19:52 |
| yoctozepto | the previous one no | 19:53 |
| yoctozepto | this one was without logs so hard to tell | 19:53 |
| *** yamamoto has quit IRC | 19:53 | |
| yoctozepto | ;P | 19:53 |
| fungi | 666881 only adds/removes comments, so the odds that it caused that job to fail are small | 19:54 |
| fungi | but it's possible something has consistently broken that job nonetheless, outside that change | 19:55 |
| fungi | oh, i misread. it adds a pull | 19:55 |
| fungi | so could be | 19:55 |
| yoctozepto | fungi: was about to tell you | 19:56 |
| yoctozepto | though the pull happens as part of upgrade | 19:56 |
| yoctozepto | this only makes it more optimal | 19:56 |
| *** goldyfruit has joined #openstack-infra | 19:59 | |
| fungi | ahh | 20:00 |
| yoctozepto | unless we make those Ethernet ports burn due to faster flow of docker images | 20:00 |
| yoctozepto | no reason it made jobs fail | 20:00 |
| yoctozepto | could be something else we have | 20:01 |
| yoctozepto | or cosmic rays ;D | 20:01 |
| fungi | or clouds | 20:01 |
| fungi | we blame most problems on clouds | 20:01 |
| yoctozepto | yeah, they cloud our vision, we can't see the real problems anymore | 20:02 |
| *** tosky__ has joined #openstack-infra | 20:03 | |
| *** tosky has quit IRC | 20:03 | |
| *** e0ne has quit IRC | 20:17 | |
| *** markvoelker has joined #openstack-infra | 20:23 | |
| *** rfolco has quit IRC | 20:25 | |
| *** roman_g has quit IRC | 20:28 | |
| *** markvoelker has quit IRC | 20:28 | |
| jrist | can I get a workflow +1 when/if someone gets a chance? https://review.opendev.org/#/c/665259/1 | 20:36 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Do not require a CLA for the sandbox https://review.opendev.org/666918 | 20:41 |
| corvus | fungi, clarkb, mordred: ^ i'm helping to onboard a new contributor and they ran into that | 20:42 |
| fungi | absolutely | 20:44 |
| fungi | thanks for spotting that | 20:44 |
| fungi | opendev does not require a cla. that openstack does is an implementation detail of openstack's contributor onboarding, not opendev's | 20:44 |
| *** Goneri has quit IRC | 20:45 | |
| corvus | evrardjp: in ansible 2.8, there is a jinja2 function now() to get the current datetime | 20:51 |
| corvus | evrardjp: https://docs.ansible.com/ansible/latest/user_guide/playbooks_templating.html#get-the-current-time | 20:51 |
| fungi | ooh, that seems like a great solution | 20:52 |
| corvus | evrardjp: you can request 2.8 from zuul with https://zuul-ci.org/docs/zuul/user/config.html#attr-job.ansible-version | 20:52 |
| corvus | evrardjp: if you want to proceed with the current ansible_date_time work, i'm still okay with that, but if you'd prefer to back it out and try now() instead, i'd support that too. | 20:53 |
| *** kjackal has quit IRC | 20:55 | |
| openstackgerrit | Merged openstack/project-config master: Retiring TripleO-UI and associated projects. https://review.opendev.org/665259 | 20:56 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 21:02 |
| *** pcaruana has quit IRC | 21:16 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 21:16 |
| *** markvoelker has joined #openstack-infra | 21:24 | |
| *** markvoelker has quit IRC | 21:29 | |
| *** openstackgerrit has quit IRC | 21:33 | |
| *** jtomasek has quit IRC | 21:38 | |
| *** smarcet has quit IRC | 21:43 | |
| *** persia_ is now known as persia | 21:45 | |
| *** slaweq has quit IRC | 21:55 | |
| *** openstackgerrit has joined #openstack-infra | 22:06 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:06 |
| *** tosky__ is now known as tosky | 22:14 | |
| *** slaweq has joined #openstack-infra | 22:16 | |
| *** sreejithp has quit IRC | 22:21 | |
| *** slaweq has quit IRC | 22:24 | |
| *** markvoelker has joined #openstack-infra | 22:25 | |
| *** markvoelker has quit IRC | 22:30 | |
| *** gyee has quit IRC | 22:40 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:41 |
| *** dave-mccowan has quit IRC | 22:49 | |
| *** rlandy has quit IRC | 23:00 | |
| *** igordc has quit IRC | 23:02 | |
| *** rajinir has quit IRC | 23:09 | |
| *** slaweq has joined #openstack-infra | 23:13 | |
| *** weifan has quit IRC | 23:16 | |
| *** weifan has joined #openstack-infra | 23:18 | |
| *** weifan has quit IRC | 23:19 | |
| *** weifan has joined #openstack-infra | 23:20 | |
| *** tosky has quit IRC | 23:22 | |
| openstackgerrit | Kendall Nelson proposed opendev/storyboard master: word highlight https://review.opendev.org/629189 | 23:23 |
| *** slaweq has quit IRC | 23:25 | |
| *** weifan has quit IRC | 23:25 | |
| *** markvoelker has joined #openstack-infra | 23:26 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 23:29 |
| *** markvoelker has quit IRC | 23:30 | |
| *** goldyfruit has quit IRC | 23:40 | |
| *** weifan has joined #openstack-infra | 23:42 | |
| *** weifan has quit IRC | 23:44 | |
| *** goldyfruit has joined #openstack-infra | 23:44 | |
| *** weifan has joined #openstack-infra | 23:45 | |
| jrist | thanks fungi | 23:45 |
| *** weifan has quit IRC | 23:45 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/project-config master: Retire independent ansible role https://review.opendev.org/665237 | 23:59 |
| *** jamesmcarthur has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
