| mordred | ianw: ok. merged pin | 00:00 |
|---|---|---|
| mordred | clarkb: yah | 00:00 |
| ianw | clarkb: but popen is non-blocking? it wouldn't sit for 2 hours?\ | 00:00 |
| clarkb | ianw: this is a remote popen though | 00:00 |
| mordred | ianw: sorry about that - I somehow thought I'd already landed that patch :( | 00:00 |
| clarkb | ianw: so it may be blocking on the call to that non blocking popen to even happen? | 00:00 |
| ianw | mordred: heh, so did I, didn't read my notification emails close enough | 00:01 |
| mordred | clarkb: this does seem like it's a good chance of being a fun bug | 00:01 |
| clarkb | hrm though maybe the cmd includes the ssh command? | 00:02 |
| clarkb | it isn't clear to me how that popen is running on a remote host | 00:03 |
| ianw | clarkb: yeah, my reading of that is that this is that popen is creating the connection to the remote end that is used to run the commands | 00:04 |
| *** Emine has quit IRC | 00:04 | |
| ianw | it's not like an RPC where that code is being executed on the remote node | 00:04 |
| kmalloc | ianw: yeah. | 00:04 |
| kmalloc | ianw: my patch has now started failing all over | 00:04 |
| kmalloc | ianw: not sure if it's related to my patch or because something else is causing issues | 00:05 |
| clarkb | ianw: yup build_command is wrapping the actual command in the ssh command | 00:05 |
| kmalloc | ianw: i'd rather see a 0.7.0 pin merge so a quick release can be cut (point release) then we can correct/fix and see the stuff through gate a little less OMG RIGHT NOW | 00:05 |
| kmalloc | mordred: ^ | 00:05 |
| kmalloc | my fix needs some love. | 00:06 |
| dhellmann | I'm having some trouble getting https://review.openstack.org/#/c/625624/1 to merge. It has passed the check queue and I've approved it, and nothing is happening. The parent patch did merge fine. Does anyone have any ideas about what might be going on? | 00:06 |
| mordred | kmalloc: ++ | 00:06 |
| *** rlandy has quit IRC | 00:06 | |
| kmalloc | mordred: mostly because, like I said something is horked. | 00:06 |
| ianw | dhellmann: hrm, no jobs setup? | 00:06 |
| * kmalloc stares at the new failures. | 00:07 | |
| dhellmann | ianw : the parent patch establishes the jobs | 00:07 |
| kmalloc | it's like our fake cache is doing something ... bad? | 00:07 |
| kmalloc | because we're getting nones back... | 00:07 |
| kmalloc | mordred: all cirrus image not existing. | 00:07 |
| mordred | kmalloc: that does, in fact, seem bad | 00:08 |
| mordred | kmalloc: is your new wrapper returning the wrong thing? | 00:08 |
| dhellmann | a recheck at least puts it back into the check queue. maybe going through again will do it | 00:08 |
| kmalloc | oh crap i suck. | 00:08 |
| kmalloc | mordred: no, i did it right | 00:08 |
| kmalloc | mordred: it just does a straight return f(*args, **kwargs) | 00:09 |
| kmalloc | mordred: i was thinking i missed the return | 00:09 |
| kmalloc | the new wrapper is about as basic as you can possibly get. | 00:09 |
| *** dave-mccowan has joined #openstack-infra | 00:10 | |
| kmalloc | mordred: something broke between when i proposed the change and when i added the Needed-by | 00:10 |
| clarkb | the key thing with the ansible double tap is that the randbits don't change | 00:10 |
| clarkb | so its rerunning it at a really low level in ansible | 00:10 |
| ianw | kmalloc: hrm, maybe devsatck merged a cirros update? | 00:11 |
| clarkb | ianw: to 0.3.6 | 00:12 |
| ianw | kmalloc: https://review.openstack.org/#/c/624716/ ... does this line up? | 00:12 |
| kmalloc | ianw: looking | 00:12 |
| ianw | penstack.exceptions.ResourceNotFound: No Image found for cirros-0.3.5-x86_64-disk | 00:12 |
| kmalloc | ianw: yup | 00:13 |
| kmalloc | ianw: that looks like the issue. | 00:13 |
| clarkb | this fixes config drive support in cirros | 00:13 |
| kmalloc | ianw: openstack.exceptions.ResourceNotFound: No Image found for cirros-0.3.5-x86_64-disk that is what i'm seeing in all the failing tests. | 00:13 |
| ianw | hrm, that change merged well before this run | 00:13 |
| ianw | http://logs.openstack.org/70/625370/3/check/openstacksdk-functional-devstack/ff562ff/ | 00:14 |
| kmalloc | before my first patch? | 00:14 |
| kmalloc | 12-14 16:40ish | 00:14 |
| clarkb | the sdk tests probably shouldn't assume the cirros version is static | 00:14 |
| clarkb | instead just check for any cirros? | 00:14 |
| ianw | indeed the devstack side of ^^ is showing 0.3.6 ... so where's the 0.3.5 in the testr results coming from | 00:15 |
| kmalloc | ianw: yep, so it merged between when i proposed the 1st patch and the update. :) | 00:15 |
| kmalloc | ianw: thanks! :) this makes me feel a little better. | 00:15 |
| clarkb | ianw: its hardcoded in sdk's test suite | 00:15 |
| kmalloc | it's not something i did. | 00:15 |
| kmalloc | yeah SDK is going to fail until we fix that. | 00:15 |
| ianw | oh yeah, i see now ... ok, we'll that explains that | 00:15 |
| kmalloc | ianw: otherwise my patch should be good to go. | 00:16 |
| ianw | urgh, so the 0.7.0 pin will fail to go in because of that anyway | 00:16 |
| kmalloc | yep. | 00:16 |
| ianw | SIGH! this has been the longest series of yak shaving ever ... | 00:16 |
| kmalloc | =/ | 00:16 |
| kmalloc | sorry man | 00:16 |
| * mordred hands ianw a pile of beers | 00:16 | |
| kmalloc | mordred: did you order extra yaks? | 00:16 |
| kmalloc | cause i feel like there are a few more waiting to show up. | 00:17 |
| mordred | ianw: so maybe we should just put the cirros fix into the 0.7.0 pin | 00:17 |
| kmalloc | i'd support that | 00:17 |
| ianw | nobody's fault ... just amazing how far down everything goes ... | 00:17 |
| kmalloc | since it's a test-suite change | 00:17 |
| ianw | yeah, i think they'll have to be merged otherwise it's circular dependency | 00:17 |
| kmalloc | not a functional "this is busted code wise" change | 00:17 |
| kmalloc | and then we can land dogpile fix, and unpin req, unpin SDK | 00:17 |
| mordred | yeah | 00:18 |
| kmalloc | man that seems like a seriously circuitous way to get to this fix | 00:18 |
| *** dave-mccowan has quit IRC | 00:18 | |
| mordred | that sounds like a good sequence | 00:18 |
| clarkb | dhellmann: looks like it is queued in the gate now | 00:18 |
| kmalloc | it is the most correct way though. | 00:18 |
| mordred | ianw: you want to update the sdk patch? or want me to? | 00:18 |
| ianw | mordred: i can take a look | 00:19 |
| dhellmann | \o/ | 00:19 |
| dhellmann | I wonder if the problem is I approved it before there were jobs because I approved that patch and then the parent so they would merge together | 00:19 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql master: Fix handling of xfail and uxsuccess results in subunit2sql cli https://review.openstack.org/625739 | 00:19 |
| mordred | ianw: I'm thinking for expediency, just changing it to 0.3.6 for now - and we can followup with a better solution | 00:20 |
| ianw | yeah | 00:20 |
| mtreinish | clarkb, fungi: I just came across a bug in subunit2sql (fix here: https://review.openstack.org/#/c/625739/ ) and I'm wondering if there are any exceptions from it in the gearman worker logs? | 00:21 |
| mtreinish | anytime there was a xfail or uxsuccess result in the stream (for an existing test) it would raise an exception | 00:22 |
| mtreinish | it could be that we just don't have any suites using that status, but I feel like there would be at least one somewhere | 00:22 |
| clarkb | mtreinish: there are for xfail | 00:23 |
| ianw | oh, we can probably drop the -src job as non-voting, because moving the jobs to xenial merged (the func tests are on-voting & thus the gate doesn't actually check them in nodepool) | 00:25 |
| mtreinish | clarkb: hmm, do you know which test? I can check the db | 00:25 |
| clarkb | mtreinish: I don't know the test case but it appears http://logs.openstack.org/88/611088/12/gate/openstack-tox-py35/f4dcfd6/testrepository.subunit.gz is one of the files that had an issue | 00:26 |
| *** sthussey has quit IRC | 00:32 | |
| *** Emine has joined #openstack-infra | 00:34 | |
| clarkb | dhellmann: ya I wonder if zuul processed the nequeue to the gate before it updated the gate job config post merge | 00:34 |
| mtreinish | clarkb: yeah that's definitely failing to update: http://paste.openstack.org/show/737516/ | 00:36 |
| mtreinish | it also probably means we have a bunch of incomplete entries in the db for nova unit tests | 00:41 |
| clarkb | frickler: amorin: I have to pop out soon, but I've got https://etherpad.openstack.org/p/bhs1-test-node-slowness up todate with my benchmarking info. It looked good to me but I didn't want to potentially break any of fricklers test nodes. frickler can you run my script on those nodes or something similar to double check all the instances have reasonable tempest runtimes? | 00:44 |
| *** gouthamr_ has joined #openstack-infra | 00:49 | |
| clarkb | now that ansible is running the full base playbook we are back to 45 minutes between ansible runs | 00:51 |
| *** dmellado has joined #openstack-infra | 00:53 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for subunit workers https://review.openstack.org/615661 | 01:00 |
| *** stevebaker has joined #openstack-infra | 01:03 | |
| *** bobh has joined #openstack-infra | 01:08 | |
| *** jamesmcarthur has joined #openstack-infra | 01:14 | |
| *** gyee has quit IRC | 01:18 | |
| *** jamesmcarthur has quit IRC | 01:19 | |
| *** bobh has quit IRC | 01:20 | |
| *** eernst_ has joined #openstack-infra | 01:21 | |
| *** hongbin has joined #openstack-infra | 01:42 | |
| *** Emine has quit IRC | 02:00 | |
| *** eernst_ has quit IRC | 02:09 | |
| *** hongbin_ has joined #openstack-infra | 02:28 | |
| *** hongbin has quit IRC | 02:29 | |
| *** wolverineav has quit IRC | 02:32 | |
| *** hwoarang has quit IRC | 02:32 | |
| *** hwoarang has joined #openstack-infra | 02:34 | |
| *** bobh has joined #openstack-infra | 02:38 | |
| *** psachin has joined #openstack-infra | 02:41 | |
| *** ykarel|away has joined #openstack-infra | 02:46 | |
| *** mrsoul has joined #openstack-infra | 02:58 | |
| *** anteaya has quit IRC | 02:59 | |
| *** ykarel|away is now known as ykarel | 03:05 | |
| *** dpawlik has joined #openstack-infra | 03:06 | |
| *** dpawlik has quit IRC | 03:10 | |
| *** apetrich has quit IRC | 03:15 | |
| *** hwoarang has quit IRC | 03:16 | |
| *** hwoarang has joined #openstack-infra | 03:17 | |
| *** wolverineav has joined #openstack-infra | 03:18 | |
| *** hongbin has joined #openstack-infra | 03:20 | |
| *** hongbin_ has quit IRC | 03:21 | |
| *** rcernin has quit IRC | 03:25 | |
| *** zhangfei has joined #openstack-infra | 03:26 | |
| *** rcernin has joined #openstack-infra | 03:27 | |
| *** rcernin has quit IRC | 03:28 | |
| *** rcernin has joined #openstack-infra | 03:28 | |
| *** hwoarang has quit IRC | 03:30 | |
| *** hwoarang has joined #openstack-infra | 03:32 | |
| *** ramishra has joined #openstack-infra | 03:32 | |
| *** bobh has quit IRC | 03:34 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:34 | |
| *** wolverineav has quit IRC | 03:34 | |
| *** bhavikdbavishi has quit IRC | 03:35 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:35 | |
| *** hongbin has quit IRC | 03:43 | |
| *** mriedem has joined #openstack-infra | 03:49 | |
| *** bobh has joined #openstack-infra | 03:51 | |
| *** zhangfei has quit IRC | 04:10 | |
| *** udesale has joined #openstack-infra | 04:11 | |
| *** bobh has quit IRC | 04:12 | |
| *** ykarel is now known as ykarel|afk | 04:15 | |
| *** ykarel|afk has quit IRC | 04:19 | |
| *** Bhujay has joined #openstack-infra | 04:20 | |
| *** Bhujay has quit IRC | 04:21 | |
| *** Bhujay has joined #openstack-infra | 04:21 | |
| *** Bhujay has quit IRC | 04:22 | |
| *** Bhujay has joined #openstack-infra | 04:23 | |
| *** rf0lc0 has joined #openstack-infra | 04:28 | |
| *** rfolco has quit IRC | 04:29 | |
| *** eernst has joined #openstack-infra | 04:29 | |
| *** zhangfei has joined #openstack-infra | 04:30 | |
| *** mriedem has quit IRC | 04:31 | |
| *** bhavikdbavishi has quit IRC | 04:33 | |
| *** wolverineav has joined #openstack-infra | 04:39 | |
| *** ykarel|afk has joined #openstack-infra | 04:39 | |
| *** ykarel|afk is now known as ykarel | 04:39 | |
| *** wolverineav has quit IRC | 04:43 | |
| *** hwoarang has quit IRC | 04:46 | |
| *** hwoarang has joined #openstack-infra | 04:49 | |
| *** janki has joined #openstack-infra | 04:50 | |
| *** chkumar|out is now known as chandankumar | 04:53 | |
| *** eernst has quit IRC | 04:59 | |
| *** armax has quit IRC | 05:07 | |
| *** kmalloc has quit IRC | 05:14 | |
| *** diablo_rojo has quit IRC | 05:25 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:44 | |
| *** bhavikdbavishi has quit IRC | 06:06 | |
| *** yolanda has joined #openstack-infra | 06:07 | |
| *** yboaron has joined #openstack-infra | 06:17 | |
| *** gagehugo has quit IRC | 06:23 | |
| *** wolverineav has joined #openstack-infra | 06:30 | |
| *** ccamacho has quit IRC | 06:31 | |
| *** mgagne_ has quit IRC | 06:35 | |
| *** mgagne has joined #openstack-infra | 06:39 | |
| *** gouthamr has quit IRC | 06:55 | |
| *** hwoarang has quit IRC | 06:57 | |
| *** hwoarang has joined #openstack-infra | 07:00 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:01 | |
| *** wolverineav has quit IRC | 07:04 | |
| *** e0ne has joined #openstack-infra | 07:11 | |
| *** ccamacho has joined #openstack-infra | 07:14 | |
| *** quiquell|off is now known as quiquell | 07:15 | |
| *** csatari has joined #openstack-infra | 07:17 | |
| *** bhavikdbavishi has quit IRC | 07:18 | |
| *** apetrich has joined #openstack-infra | 07:18 | |
| csatari | I do not receive the confirmation mail form openstack-discuss and my IT says, that an SPF entry should be set for lists.openstack.org. Can anyone help in this here? | 07:21 |
| *** ccamacho has quit IRC | 07:21 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:22 | |
| *** jtomasek has joined #openstack-infra | 07:25 | |
| *** pgaxatte has joined #openstack-infra | 07:25 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Update test-mirror-workspace-git-repos, add test https://review.openstack.org/624575 | 07:26 |
| AJaeger | ianw: happy testing ^ | 07:28 |
| *** imacdonn has quit IRC | 07:29 | |
| *** imacdonn has joined #openstack-infra | 07:29 | |
| ianw | AJaeger: ... :) i also added a note on that because it's hard to get your head around | 07:34 |
| ianw | https://review.openstack.org/#/c/624578/ | 07:34 |
| AJaeger | ianw: yeah, noticed - and pinged yesterday on #zuul for it ;) Thanks | 07:34 |
| *** dpawlik has joined #openstack-infra | 07:35 | |
| AJaeger | I wanted to get some more review on that one... | 07:35 |
| *** AJaeger has quit IRC | 07:42 | |
| *** dpawlik has quit IRC | 07:42 | |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| *** AJaeger has joined #openstack-infra | 07:45 | |
| Linkid | Hi | 07:46 |
| Linkid | I was thinking about using docker to deploy a new service on the OpenStack Infra | 07:48 |
| Linkid | Maybe it could be useful to use ansible (or puppet, for now) to deploy the configuration file, and to launch docker | 07:49 |
| Linkid | What do you think about that ? | 07:52 |
| ianw | Linkid: there's work in progress to use docker in our control plane | 07:52 |
| ianw | https://review.openstack.org/#/c/605585/ | 07:53 |
| ianw | Linkid: using that, and something along the lines of https://review.openstack.org/#/c/606028/1, it should be largely possible to test all of this via CI jobs | 07:53 |
| *** slaweq has joined #openstack-infra | 07:53 | |
| *** quiquell is now known as quiquell|brb | 07:54 | |
| *** dpawlik has joined #openstack-infra | 07:56 | |
| *** hwoarang has quit IRC | 07:56 | |
| *** shardy has joined #openstack-infra | 07:56 | |
| *** hwoarang has joined #openstack-infra | 07:57 | |
| *** rpittau has joined #openstack-infra | 07:59 | |
| *** longkb has joined #openstack-infra | 08:01 | |
| *** zhangfei has quit IRC | 08:02 | |
| *** apetrich has quit IRC | 08:06 | |
| *** wolverineav has joined #openstack-infra | 08:19 | |
| *** quiquell|brb is now known as quiquell | 08:19 | |
| *** gagehugo has joined #openstack-infra | 08:20 | |
| *** jamesdenton has quit IRC | 08:22 | |
| *** ccamacho has joined #openstack-infra | 08:22 | |
| *** wolverineav has quit IRC | 08:23 | |
| *** yboaron has quit IRC | 08:23 | |
| *** jamesdenton has joined #openstack-infra | 08:23 | |
| Linkid | and what about using docker compose ? | 08:23 |
| Linkid | will it be possible to use it too ? | 08:24 |
| *** zhangfei has joined #openstack-infra | 08:24 | |
| Linkid | but I think I will reread the spec about that and update mine | 08:26 |
| Linkid | thanks for the links :) | 08:29 |
| *** bhavikdbavishi has quit IRC | 08:35 | |
| *** apetrich has joined #openstack-infra | 08:38 | |
| *** priteau has joined #openstack-infra | 08:47 | |
| *** ykarel|lunch is now known as ykarel | 08:51 | |
| *** jpena|off is now known as jpena | 08:56 | |
| *** jpich has joined #openstack-infra | 09:05 | |
| *** yboaron has joined #openstack-infra | 09:06 | |
| *** bhavikdbavishi has joined #openstack-infra | 09:08 | |
| *** jesusaur has quit IRC | 09:15 | |
| *** rcernin has quit IRC | 09:18 | |
| *** quiquell has quit IRC | 09:21 | |
| *** jesusaur has joined #openstack-infra | 09:22 | |
| *** Bhujay has quit IRC | 09:24 | |
| *** quiquell has joined #openstack-infra | 09:25 | |
| amorin | hey frickler | 09:30 |
| amorin | hey clarkb , I red the etherpad updates | 09:30 |
| amorin | sounds good | 09:30 |
| amorin | before moving forward I'd like to be sure that this is correct with frickler tests | 09:31 |
| quiquell | Hello it's review.o.o down ? | 09:34 |
| egonzalez | quiquell not for me | 09:34 |
| *** xek has joined #openstack-infra | 09:42 | |
| *** bhavikdbavishi has quit IRC | 09:45 | |
| frickler | ianw: for the nodepool job with dogpile.cache, you probably need a pre task to clone the repo, similar to this one https://review.openstack.org/#/c/624494/9/playbooks/skydive-pre-run.yaml . I can try to set this up later for you if you like | 09:47 |
| *** derekh has joined #openstack-infra | 09:48 | |
| frickler | amorin: I'll look into running clarkb's tests later, will have to check the network/mirror things | 09:50 |
| *** dkehn has quit IRC | 09:57 | |
| amorin | k | 10:01 |
| *** Bhujay has joined #openstack-infra | 10:02 | |
| *** Bhujay has quit IRC | 10:03 | |
| *** Bhujay has joined #openstack-infra | 10:04 | |
| *** Bhujay has quit IRC | 10:05 | |
| *** Bhujay has joined #openstack-infra | 10:05 | |
| *** Bhujay has quit IRC | 10:06 | |
| *** kaisers has joined #openstack-infra | 10:07 | |
| *** Bhujay has joined #openstack-infra | 10:07 | |
| *** electrofelix has joined #openstack-infra | 10:12 | |
| *** quiquell has quit IRC | 10:14 | |
| *** quiquell has joined #openstack-infra | 10:15 | |
| *** dpawlik has quit IRC | 10:30 | |
| *** dpawlik_ has joined #openstack-infra | 10:30 | |
| *** pbourke has quit IRC | 10:36 | |
| *** pbourke has joined #openstack-infra | 10:37 | |
| *** udesale has quit IRC | 10:40 | |
| *** yamamoto has quit IRC | 11:01 | |
| openstackgerrit | Merged openstack-infra/opendev-website master: Add publishing of content to opendev-website https://review.openstack.org/625665 | 11:08 |
| *** longkb has quit IRC | 11:13 | |
| *** zhangfei has quit IRC | 11:19 | |
| *** sean-k-mooney has quit IRC | 11:21 | |
| *** longkb has joined #openstack-infra | 11:23 | |
| *** longkb has quit IRC | 11:23 | |
| *** yamamoto has joined #openstack-infra | 11:38 | |
| *** EmilienM|off is now known as EmilienM | 11:41 | |
| *** Bhujay has quit IRC | 11:46 | |
| *** Bhujay has joined #openstack-infra | 11:52 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:52 | |
| *** Bhujay has quit IRC | 11:53 | |
| *** Bhujay has joined #openstack-infra | 11:53 | |
| *** yamamoto has quit IRC | 11:54 | |
| *** yamamoto has joined #openstack-infra | 11:54 | |
| *** Bhujay has quit IRC | 11:54 | |
| *** Bhujay has joined #openstack-infra | 11:55 | |
| *** bhavikdbavishi has quit IRC | 11:56 | |
| *** Bhujay has quit IRC | 11:56 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:56 | |
| *** Bhujay has joined #openstack-infra | 11:56 | |
| *** Bhujay has quit IRC | 11:57 | |
| *** Bhujay has joined #openstack-infra | 11:58 | |
| *** Bhujay has quit IRC | 11:59 | |
| *** Bhujay has joined #openstack-infra | 11:59 | |
| *** wolverineav has joined #openstack-infra | 11:59 | |
| *** Bhujay has quit IRC | 12:00 | |
| *** Bhujay has joined #openstack-infra | 12:01 | |
| *** rpittau has quit IRC | 12:02 | |
| *** rpittau has joined #openstack-infra | 12:03 | |
| *** wolverineav has quit IRC | 12:04 | |
| *** tpsilva has joined #openstack-infra | 12:06 | |
| *** rpittau is now known as rpittau|lunch | 12:08 | |
| frickler | amorin: clarkb: these are the results of running fio on my test nodes, looks pretty sane to me. I'll look into running devstack now to confirm. https://ethercalc.openstack.org/kk3ot86hyvdk | 12:11 |
| *** shardy has quit IRC | 12:11 | |
| *** janki has quit IRC | 12:17 | |
| evrardjp | hello folks! I've added periodic testing into a project (which merged yesterday: https://review.openstack.org/#/c/624654/ ), but I don't see if it has run (nothing in http://zuul.openstack.org/builds?project=openstack%2Fopenstack-helm-images&pipeline=periodic ) -- any clue on how to find if it was scheduled? | 12:18 |
| AJaeger | evrardjp: AFAIK this needs an infra-root to look at log-files. | 12:24 |
| AJaeger | evrardjp: I didn't see anything suspicious | 12:25 |
| *** udesale has joined #openstack-infra | 12:27 | |
| *** ramishra has quit IRC | 12:27 | |
| * frickler goes digging into zuul logs | 12:28 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Delay Github fileschanges workaround to pipeline processing https://review.openstack.org/625584 | 12:30 |
| *** eharney has joined #openstack-infra | 12:33 | |
| frickler | evrardjp: 2018-12-18 06:00:28,756 DEBUG zuul.layout: Job <Job openstack-helm-images-ceph-config-helper-ubuntu_xenial branches: None source: openstack-infra/project-config/zuul.d/jobs.yaml@master#26> did not match files in | 12:35 |
| frickler | <Branch 0x7fdb23d2ebe0 openstack/openstack-helm-images refs/heads/master updated None..None> | 12:35 |
| *** jpena is now known as jpena|lunch | 12:35 | |
| frickler | evrardjp: you may not have "files:" filters on periodic jobs, as they are not being run against a specific change | 12:36 |
| evrardjp | Well that's interesting | 12:36 |
| evrardjp | oh that's it :) | 12:36 |
| evrardjp | any creative idea on how to re-use the existing jobs for periodics? | 12:37 |
| *** ramishra has joined #openstack-infra | 12:37 | |
| frickler | evrardjp: make a variant without the files selector, I'd say | 12:37 |
| evrardjp | yeah that's the simplest | 12:38 |
| evrardjp | I was looking for a fancy way to ignore that directive into periodics :) | 12:38 |
| frickler | evrardjp: maybe there is one that I'm not aware of. otherwise might be a useful idea for a feature request | 12:39 |
| evrardjp | yeah I guess it's probably worth asking in zuul channel. Thanks for the help frickler ! | 12:40 |
| *** quiquell is now known as quiquell|brb | 12:45 | |
| AJaeger | evrardjp: have a parent without files and inherit a job with files added to it? | 12:50 |
| evrardjp | AJaeger: that's what I am writing right now | 12:50 |
| evrardjp | but it means I will have two jobs: jobname-checks and jobname-periodics (or something like that) | 12:50 |
| AJaeger | evrardjp: ok. saw your question in #zuul , might want to wait for the US west coast to wake up and comment | 12:51 |
| AJaeger | evrardjp: yes, you would have those two | 12:51 |
| evrardjp | I was wondering if it was possible to bypass by passing some argument to the job in the pipeline for example | 12:51 |
| evrardjp | (so in the project definition) | 12:51 |
| AJaeger | evrardjp: understood - and curious as well... | 12:51 |
| evrardjp | we'll see :) | 12:52 |
| *** dkehn has joined #openstack-infra | 12:52 | |
| *** e0ne has quit IRC | 12:53 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Pagure driver https://review.openstack.org/604404 | 12:54 |
| *** rh-jelabarre has joined #openstack-infra | 12:58 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 12:59 | |
| *** bhavikdbavishi has quit IRC | 13:00 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 13:00 | |
| *** trown|outtypewww is now known as trown | 13:06 | |
| EmilienM | I wonder why we don't provide HTTPS to http://lists.openstack.org - and we have to give our password on unsecure channels | 13:08 |
| fungi | EmilienM: mailman can already be asked to send anyone's password over an unsecured, plaintext channel (smtp) so https wouldn't be much help. also you only need to set a password if you want to manage list preferences; you can just omit the password field on the web form or not supply one when subscribing via e-mail | 13:11 |
| *** rpittau|lunch is now known as rpittau | 13:12 | |
| *** jrist has quit IRC | 13:12 | |
| Tengu | not a good reason..... nowadays, firefox and other modern browser complain about non-secure channels when you get forms. | 13:12 |
| EmilienM | fair enough but still why aren't we providing https for everything? | 13:12 |
| EmilienM | (I'm just wondering if we have some limitations on the SSL certificates or something) | 13:13 |
| fungi | csatari: why are they saying an spf record is needed for lists.openstack.org? i recommend not using an e-mail address at any domain which insists on other domains setting spf records in dns for any sending mta. spf is most decidedly not a requirement to send e-mail according to the ietf | 13:14 |
| *** quiquell|brb is now known as quiquell | 13:14 | |
| fungi | EmilienM: mainly that buying and renewing ssl certs is annoying/cumbersome, as is the process of filing expense reports for them. and before you say "let's encrypt" trying to deal with bootstrapping it through configuration management is still a nontrivial exercise we're working through a spec for | 13:16 |
| Tengu | let's encrypt... ? | 13:16 |
| EmilienM | yeah https://review.openstack.org/#/c/587283/ | 13:16 |
| fungi | Tengu: i agree that aggressive browser behavior is not a good reason | 13:16 |
| EmilienM | I now remember this specs | 13:16 |
| fungi | Tengu: "let's encrypt" is a joint effort between some browser vendors and other interest groups aimed at providing domain-validated x.509 certs from a blessed ca remotely via an api | 13:18 |
| *** boden has joined #openstack-infra | 13:18 | |
| Tengu | fungi: well, 2018 is a big turn toward full traffic encrpytion anyway. so it might be good to move in that direction. on my own I'm checking how to distribute LE certificates across a multi-node infra, using freeipa Vault and the like. | 13:19 |
| fungi | seems like putting all your certs in one place, even if it's a fairly safe one, mostly invalidates the main benefits of le | 13:21 |
| fungi | that being, only each server needs to know the private key, and it never needs to leave that server | 13:21 |
| fungi | so with distinct per-server private keys there's no single point of compromise which could reveal them all | 13:22 |
| *** ramishra has quit IRC | 13:22 | |
| *** ramishra has joined #openstack-infra | 13:22 | |
| fungi | er, i meant to say putting all your keys (not your public certs) in one place invalidates that benefit | 13:23 |
| Tengu | there are two uses for certificate : encryption and domain validations. The way x509 works is crappy from the very beginning, but I don't think it's a reason to expose user's private information such as email, name, firstname and [dedicated] password on clear traffic. anyway. guess we'll just have to cope with popups from modern browsers saying "you're transmitting data over insecure channel" and | 13:23 |
| Tengu | the like. | 13:23 |
| Tengu | on my own, I don't care, I have per-service dedicate password, not even related. others don't work like that. | 13:23 |
| fungi | i expect switching to mailman 3 will push us to move the service to https because it does a lot more with authentication and also goes to more of an effort to secure it | 13:24 |
| Tengu | and serving different certificates for the same domain from different server might cause an issue as well, especially for users using extensions like certPatrol :). as said: x509 is too old and doesn't scale. | 13:25 |
| fungi | we might just get a cert for it anyway even while still running mailman 2 if it becomes more of a problem than just satisfying a checkbox | 13:26 |
| fungi | but honestly, if you care about securing the password used for setting your list preferences, don't set one. mailman 2 doesn't consider it a particularly secure piece of data | 13:27 |
| *** rlandy has joined #openstack-infra | 13:27 | |
| Tengu | it's more a matter of good practices imho - your customer is able to send data over Internet, you just encrypt the road. Guess we aren't on the same ground here ;). | 13:28 |
| fungi | https in this case would only serve to instill a false sense of security, which in my (long-time information security professional) experience is worse than no security | 13:28 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Fix skipped job counted as failed https://review.openstack.org/625910 | 13:28 |
| *** jrist has joined #openstack-infra | 13:30 | |
| *** sean-k-mooney has joined #openstack-infra | 13:31 | |
| Tengu | fungi: two sides of the same coin ;). I get your point and respect it. So I'll wait for the mailman3 thingy, hoping it will push forward for some tls things :). | 13:34 |
| Tengu | and will continue to use dedicated passwords, as it's the only real thing to do, with or without TLS :] | 13:34 |
| fungi | yes, reusing the same password between two systems is detrimental to your health anyway ;) | 13:35 |
| Tengu | true ;). | 13:36 |
| *** yamamoto has quit IRC | 13:37 | |
| *** yamamoto has joined #openstack-infra | 13:38 | |
| Tengu | just a pity I'm limited in the aliases at work though :(. I'm used to get per-service dedicated email as well in order to cut leaks. | 13:38 |
| *** kgiusti has joined #openstack-infra | 13:39 | |
| *** zhangfei has joined #openstack-infra | 13:42 | |
| *** jpena|lunch is now known as jpena | 13:42 | |
| *** yamamoto has quit IRC | 13:43 | |
| *** rf0lc0 is now known as rfolco | 13:48 | |
| *** spa-87 has joined #openstack-infra | 13:51 | |
| *** zhangfei has quit IRC | 13:52 | |
| fungi | csatari: is yours one of the nokia.com addresses? if so, i've looked into that situation for your colleagues. for some reason outlook.com is telling our mta that it has accepted the subscription confirmation e-mail for delivery. we don't have any insight into it beyond that. we did confirm that manually force-subscribing one of those addresses resulted in them starting to receive messages from the ml | 13:52 |
| fungi | normally, so it's quite frustrating that the mail host there would silently eat just the subscription confirmations and not at least report an error/rejection | 13:52 |
| fungi | all i can conclude is that there is some overly-aggressive spam filter silently discarding the subscription confirmation messages | 13:54 |
| *** e0ne has joined #openstack-infra | 13:57 | |
| *** bhavikdbavishi has quit IRC | 13:57 | |
| Tengu | erf.. outlook.com... microsoft mail service is a pain, they blacklist without any reason, they don't even use public lists, and you have no way to know why you were blacklisted -.-. I've stopped trying to send them emails. | 14:03 |
| *** e0ne has quit IRC | 14:07 | |
| *** gfidente has joined #openstack-infra | 14:07 | |
| fungi | agreed. unfortunately some of our contributors work for companies who 1. outsource their e-mail hosting there and 2. demand that all employees use their work e-mail when engaging with open source communities during work hours or as part of their job | 14:08 |
| Tengu | sure. I'm "happy" seeing I'm not the only one having issue with this specific provider. | 14:09 |
| Tengu | probably the worse to deal with. 0 transparency. even google is better on that field, with a lot of doc/help :/. microsoft just points to some 3rd party service costing an liver for some obscure "validation" we can have, and doesn't even ensure domains getting that validation are delivered. yay. | 14:10 |
| *** psachin has quit IRC | 14:12 | |
| fungi | well, the services we offer to the community are "best effort" anyway. we can't promise to satisfy the expectations of every company or mail provider on the internet | 14:15 |
| Tengu | :) | 14:15 |
| fungi | so we focus on satisfying the ietf's expectations instead, and tell folks who wish to interact with us that conformance to established rfcs and ietf standards is the best way to accomplish that | 14:16 |
| Tengu | regarding spf record, my new mail infra checks it, if it has a valid one it bypasses the greylisting. that's the only thing it's used for. | 14:17 |
| *** ykarel is now known as ykarel|away | 14:18 | |
| Tengu | after greylisting, there's the good old spamassassin. | 14:18 |
| *** yamamoto has joined #openstack-infra | 14:22 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Pagure driver https://review.openstack.org/604404 | 14:22 |
| evrardjp | AJaeger: FYI we are discussing two different ways in #zuul channel: Adding a files: [] or files: null into the project's periodic pipeline job definitions or the other way around, define files: in check/gates of the project's jobs definitions. | 14:22 |
| evrardjp | so basically creating nameless variants | 14:23 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add a note on testing trusted roles https://review.openstack.org/624578 | 14:26 |
| *** mriedem has joined #openstack-infra | 14:27 | |
| frickler | mordred: https://review.openstack.org/#/c/605585/20 is on top of two other patches. not sure whether any of these needs closer attention when merging | 14:29 |
| csatari | fungi: Yes, I'm with a nokia.com email address. Let me get back to our IT with your answer. It would be good to have an automatic solution ... | 14:30 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: kata-containers: add /usr/sbin to the PATH https://review.openstack.org/625679 | 14:31 |
| mordred | frickler: oh - so it is | 14:32 |
| fungi | csatari: it's particularly interesting to me that your mail hosting accepts messages sent through the mailing list just fine, but seems to be consistently discarding the subscription confirmation e-mails from the exact same server | 14:32 |
| fungi | accepts *and delivers* messages sent through the mailing list just fine | 14:33 |
| mordred | frickler: maybe I'll wait until clarkb is awake to land the iptables one - just in case | 14:34 |
| mordred | ( | 14:34 |
| mordred | (I think it should be safe - but better safe than sorry) | 14:34 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/nodepool master: [wip] Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 14:34 |
| *** shardy has joined #openstack-infra | 14:35 | |
| csatari | fungi: Yes, it is very interesting. I guess there are also others here using outlook365. | 14:35 |
| *** ykarel|away has quit IRC | 14:36 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Pagure driver https://review.openstack.org/604404 | 14:37 |
| *** e0ne has joined #openstack-infra | 14:43 | |
| frickler | amorin: clarkb: I did only manage to run two single devstack tests today https://ethercalc.openstack.org/3s24nymcu7lw | 14:45 |
| frickler | those look rather normal to me, though, so maybe we can just return the region into normal usage? otherwise we'd have to wait until tomorrow for some further testing | 14:46 |
| *** dpawlik_ has quit IRC | 14:47 | |
| smcginnis | Hey infra folks - I had assumed after this change merged it would be reflected in the UI - https://review.openstack.org/#/c/620664/ | 14:50 |
| smcginnis | Any additional steps needed to get that so Cinder is set up like Designate? | 14:50 |
| frickler | smcginnis: can you check whether you can set the priority on a patch? it might not be shown in the overview before that actually happens | 14:51 |
| frickler | mugsie: ^^ do you remember if anything else needed to happen when we did that for designate? | 14:52 |
| *** boden has quit IRC | 14:53 | |
| *** Bhujay has quit IRC | 14:53 | |
| *** ramishra_ has joined #openstack-infra | 14:54 | |
| *** ykarel has joined #openstack-infra | 14:55 | |
| *** ramishra has quit IRC | 14:56 | |
| fungi | csatari: yeah, if it helps, skimming our mta logs we're delivering to the following other domains hosted with outlook.com whose users have successfully subscribed to the openstack-discuss ml: blizzard.com bristol.ac.uk cern.ch godaddy.com mellanox.com mit.edu nttdata.com | 14:57 |
| fungi | pdvwireless.com vmware.com zetup.se | 14:57 |
| *** bhavikdbavishi has joined #openstack-infra | 14:57 | |
| fungi | so something in particular seems to be blocking subscription confirmations for nokia.com but not those other domains at outlook.com | 14:58 |
| fungi | s/blocking/accepting and then discarding/ | 14:58 |
| csatari | fungi: Thanks for the info. | 14:59 |
| smcginnis | frickler: Sorry, not sure what you mean by the priority. | 14:59 |
| frickler | smcginnis: sorry for not being more explicit. iiuc your patch should add a "Review-Priority" label that cinder-core members can set on a patch. until someone actually does that, I think it won't show up elsewhere | 15:02 |
| smcginnis | frickler: With the permissions set, I should be one of the folks that are able to do that, but it is not given as an option. | 15:03 |
| frickler | smcginnis: ah, indeed, I found a bug on your patch comparing with the designate one. will upload a fix in a moment | 15:05 |
| fungi | smcginnis: are you looking at a patchset which you've reviewed previously? gettit embeds all that in javascript which ends up cached browser-side, so when your permissions change you may need to pull up the vote modal and then force-refresh your browser once for it to take effect | 15:07 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Fix cinder gerrit acl https://review.openstack.org/625938 | 15:07 |
| fungi | but i'll wait for frickler's patch before guessing | 15:07 |
| frickler | smcginnis: config-core: ^^ | 15:07 |
| smcginnis | Ah! Thanks frickler | 15:07 |
| fungi | yeah, that typo would definitely explain it ;) | 15:09 |
| fungi | good eye, frickler! | 15:09 |
| fungi | seems gerrit was never able to load that changed acl at all, looking at https://review.openstack.org/#/admin/projects/openstack/cinder,access | 15:11 |
| fungi | or sometimes easier to compare what you think should have been loaded to the git view replicated to https://git.openstack.org/cgit/openstack/cinder/plain/project.config?h=refs/meta/config | 15:14 |
| fungi | since gerrit has a nasty habit of hiding permissions your account's not granted in the acl view | 15:15 |
| *** quiquell is now known as quiquell|off | 15:15 | |
| fungi | but you can clearly see there that the acl currently loaded in gerrit doesn't match what was previously approved | 15:15 |
| *** dpawlik has joined #openstack-infra | 15:16 | |
| *** dpawlik has quit IRC | 15:21 | |
| *** aojea has joined #openstack-infra | 15:21 | |
| *** bhavikdbavishi has quit IRC | 15:21 | |
| *** eharney has quit IRC | 15:22 | |
| *** ginopc has quit IRC | 15:24 | |
| *** ginopc has joined #openstack-infra | 15:25 | |
| *** gfidente has quit IRC | 15:26 | |
| *** ramishra_ has quit IRC | 15:27 | |
| *** jtomasek has quit IRC | 15:29 | |
| clarkb | frickler: is it ok if I run a few more test on your test nodes? | 15:29 |
| clarkb | but I agree everything I saw yesterday looked normal so leaning towards turning on the region again | 15:30 |
| *** rh-jelabarre has quit IRC | 15:30 | |
| *** gfidente has joined #openstack-infra | 15:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fix cinder gerrit acl https://review.openstack.org/625938 | 15:35 |
| smcginnis | Thanks frickler and fungi ^ | 15:35 |
| *** wolverineav has joined #openstack-infra | 15:36 | |
| clarkb | oh fio was run on all the nodes and they all looked sane. In that case ya I think we can clean up the test nodes and reenable in nodepool | 15:37 |
| *** rh-jelabarre has joined #openstack-infra | 15:39 | |
| *** wolverineav has quit IRC | 15:40 | |
| *** pgaxatte has quit IRC | 15:41 | |
| fungi | smcginnis: it'll take a bit of time for our automation to push that updated acl into production, but if it's still not working by, say, 16:30z please do let us know | 15:41 |
| clarkb | mordred: yes I believe the FORWARD DENY change to be safe. If we want to be really careful we can manually apply that on a host or two before merging it and make sure packets still flow, but I don't know of any cases where our instances act as routers | 15:43 |
| smcginnis | fungi: Will do, thanks | 15:44 |
| *** armax has joined #openstack-infra | 15:44 | |
| *** agopi has quit IRC | 15:46 | |
| *** agopi has joined #openstack-infra | 15:46 | |
| *** agopi has quit IRC | 15:47 | |
| *** agopi has joined #openstack-infra | 15:49 | |
| fungi | smcginnis: looks like it just took effect in the last few minutes if you want to try again | 15:56 |
| *** jamesmcarthur has joined #openstack-infra | 15:59 | |
| *** jamesmcarthur has quit IRC | 15:59 | |
| *** jamesmcarthur has joined #openstack-infra | 15:59 | |
| *** ccamacho has quit IRC | 15:59 | |
| smcginnis | Verified it shows up and works as expected. Perfect! | 16:00 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Use OVH BHS1 again https://review.openstack.org/625943 | 16:00 |
| *** diablo_rojo has joined #openstack-infra | 16:01 | |
| clarkb | infra-root ^ should be low impact over the holidays if things are still unhappy, but our testing shows what appears to be happy test nodes | 16:01 |
| *** dave-mccowan has joined #openstack-infra | 16:12 | |
| mordred | clarkb: I think I kinda think we can just land it - right? | 16:13 |
| clarkb | mordred: ya probably so | 16:13 |
| clarkb | mordred: and docker will enforce that anyway on the nodes running docker daemon | 16:13 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Add second time for Monasca team meeting https://review.openstack.org/625609 | 16:13 |
| mordred | clarkb: cool | 16:15 |
| *** dave-mccowan has quit IRC | 16:18 | |
| *** dave-mccowan has joined #openstack-infra | 16:20 | |
| *** sthussey has joined #openstack-infra | 16:20 | |
| *** spa-87 has left #openstack-infra | 16:22 | |
| *** dave-mccowan has quit IRC | 16:22 | |
| *** irdr has quit IRC | 16:24 | |
| *** dave-mccowan has joined #openstack-infra | 16:27 | |
| *** bobh has joined #openstack-infra | 16:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use OVH BHS1 again https://review.openstack.org/625943 | 16:29 |
| *** jonher has quit IRC | 16:34 | |
| *** fresta has quit IRC | 16:35 | |
| *** fresta_ has joined #openstack-infra | 16:35 | |
| *** jonher has joined #openstack-infra | 16:35 | |
| clarkb | looking at the yum failures we are seeing now it seems that egonzalez's kolla changes have fixed the opendaylight issue. But now noticing there are a bunch of metadata checksum failures with the oraclelinux repo | 16:35 |
| clarkb | I'm guessing most all of this is fallout of the centos 7.6 (and then oraclelinux downstreaming that) release | 16:35 |
| *** jonher has quit IRC | 16:36 | |
| *** fresta has joined #openstack-infra | 16:36 | |
| *** jonher has joined #openstack-infra | 16:37 | |
| *** fresta_ has quit IRC | 16:40 | |
| *** jonher_ has joined #openstack-infra | 16:40 | |
| *** fresta- has joined #openstack-infra | 16:40 | |
| mordred | infra-root: openstacksdk 0.21.0 has been released - this contains the dogpile.cache version pin, as well as fixes for the various things we've encountered over the last couple of weeks | 16:40 |
| clarkb | yay | 16:40 |
| Shrews | \o/ | 16:40 |
| clarkb | sorry about the cirros thing, I didn't eralize that would have functional test fallout like that (and encouraged we get that in so that the config drive based tests actually test config drive) | 16:41 |
| *** fresta has quit IRC | 16:42 | |
| *** jonher has quit IRC | 16:43 | |
| *** jonher_ is now known as jonher | 16:43 | |
| mordred | clarkb: well, I think we've been meaning to make those tests less hard-coded for a while, but just hadn't gotten around to it | 16:44 |
| pabelanger | heads up, ansible-lint 4.0.0 is out if you start to see linters fail could be related | 16:46 |
| pabelanger | I'm getting hit with new [204] Lines should be no longer than 120 chars errors locally | 16:47 |
| mordred | pabelanger: oh goodie | 16:47 |
| pabelanger | https://docs.ansible.com/ansible-lint/rules/default_rules.html | 16:47 |
| pabelanger | is the new defaults | 16:47 |
| clarkb | at the rate we are finding ansible bugs I expect to find a new one today :P | 16:49 |
| clarkb | fwiw I'm not sure if its worth filing a bug report with ansible with the minimal info we have from yesterdays fun issue | 16:49 |
| *** udesale has quit IRC | 16:49 | |
| clarkb | but the more I look at it the more I'm convinced that something in the low level ssh exec ran the same command twice | 16:50 |
| clarkb | because the randbits are supposed to be unique per invocation and are not unique in this case | 16:50 |
| *** e0ne has quit IRC | 16:50 | |
| clarkb | I think even if you used task retries each retry will (or is supposed to) get different rand bits | 16:50 |
| clarkb | bhs1 is building test nodes now | 16:51 |
| mordred | clarkb: yes - I'm also more convinced about that | 16:52 |
| clarkb | the include_tasks issue I filed has a PR to fix it proposed now which isnice | 16:54 |
| clarkb | hopefully we see that in 2.8 (and maybe backported to 2.7? unsure of backport policy there) | 16:55 |
| clarkb | bhs1 has available and in use nodes now (keep an eye out for any unexpected behavior there, but testing did look much happier) | 16:55 |
| *** shardy has quit IRC | 16:57 | |
| frickler | clarkb: did you remove my test nodes? /me needs to get some food now first | 16:59 |
| *** ginopc has quit IRC | 17:00 | |
| clarkb | frickler: I didn't, nodepool will handle that with its quota calculations (or it should) | 17:00 |
| clarkb | frickler: I think we can remove the test nodes whenever you are ready ( assuming nodepool quota handling is working in that region) | 17:00 |
| *** dayou has quit IRC | 17:01 | |
| pabelanger | clarkb: we can ask for 2.7 backport | 17:02 |
| frickler | clarkb: I'm fine with removing them later, if no quota issues arise. would save amorin from having to do another round of distributing stuff in case it turns out we do need to be further debugging | 17:02 |
| clarkb | frickler: ++ | 17:02 |
| *** boden has joined #openstack-infra | 17:03 | |
| clarkb | pabelanger: my biggest concern is that I get paranoid when I discover the configuration management system can fail silently and report success :) | 17:03 |
| clarkb | pabelanger: so a backport would be nice for piece of mind, though I believe we've corrected the only instance of the bug in our playbooks | 17:03 |
| pabelanger | clarkb: yah, I admit I haven't followed along with the issue you found. But agree that is bad | 17:03 |
| clarkb | pabelanger: basically if you include_tasks in a tasks file that was itself included via include_tasks then ansible stops running the play and returns success or starts the next play in the playbook | 17:04 |
| clarkb | the fix makes both the include_tasks sequencing like that work and ansible will raise a warning if it hits a similar crash situation | 17:05 |
| *** tosky has joined #openstack-infra | 17:06 | |
| *** rpittau has quit IRC | 17:06 | |
| openstackgerrit | Merged openstack-infra/system-config master: Set iptables forward drop by default https://review.openstack.org/624501 | 17:08 |
| *** yboaron has quit IRC | 17:08 | |
| openstackgerrit | Merged openstack-infra/system-config master: Collect syslogs from nodes in ansible tests https://review.openstack.org/624827 | 17:08 |
| openstackgerrit | Merged openstack-infra/system-config master: Import install-docker role https://review.openstack.org/605585 | 17:08 |
| clarkb | Here we go | 17:08 |
| *** e0ne has joined #openstack-infra | 17:13 | |
| *** chandankumar is now known as chkumar|out | 17:13 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/infra-specs master: StoryBoard Story Attachments https://review.openstack.org/607377 | 17:16 |
| diablo_rojo | clarkb, got the spec updated with the latest round of jhesketh and fungi comments in time for the meeting | 17:16 |
| *** jamesmcarthur has quit IRC | 17:16 | |
| clarkb | diablo_rojo: thanks. Do youthink it is in a spot where we should put it up for approval? | 17:17 |
| *** dayou has joined #openstack-infra | 17:17 | |
| clarkb | I expect we'll put the opendev spec up for approval as well. Then have a probably long approval period since I'm afk starting thursday, but maybe plan to merge both just before the new year? | 17:17 |
| clarkb | or around the new year anyway | 17:17 |
| clarkb | corvus: ^ thoughts? | 17:17 |
| diablo_rojo | clarkb, yeah I think so | 17:17 |
| diablo_rojo | I definitely think we can get this merged whenever. If we need to update as the implementation develops, that's fine, but I think we are agreed on the basic approach. | 17:18 |
| clarkb | ++ | 17:18 |
| *** e0ne has quit IRC | 17:19 | |
| *** armax_ has joined #openstack-infra | 17:19 | |
| *** armax has quit IRC | 17:19 | |
| *** armax_ is now known as armax | 17:19 | |
| fungi | sgtm | 17:20 |
| *** eharney has joined #openstack-infra | 17:22 | |
| *** gfidente has quit IRC | 17:26 | |
| *** jpich has quit IRC | 17:26 | |
| frickler | ianw: the other nodepool failures on your glean stack look like plain job timeouts to me, but I'm not sure about this one, please take another look http://logs.openstack.org/20/620420/5/check/nodepool-functional-py35-redhat-src/6c77fa2/ | 17:31 |
| frickler | corvus: I did ask earlier but probably got lost in backlog: do you still need your three held nodes? 46d old by now | 17:33 |
| *** gfidente has joined #openstack-infra | 17:38 | |
| *** bobh has quit IRC | 17:39 | |
| *** bobh has joined #openstack-infra | 17:40 | |
| corvus | clarkb: i think opendev is ready for voting | 17:46 |
| corvus | frickler: no, sorry about that. i'll clean them up | 17:46 |
| clarkb | the ansible base server run is happening now. Iptables updates should happen soon | 17:48 |
| *** jamesmcarthur has joined #openstack-infra | 17:52 | |
| *** _alastor_ has joined #openstack-infra | 17:53 | |
| *** bdodd has quit IRC | 17:54 | |
| *** _alastor_ has quit IRC | 17:55 | |
| *** _alastor_ has joined #openstack-infra | 17:56 | |
| *** bdodd has joined #openstack-infra | 17:56 | |
| *** derekh has quit IRC | 18:01 | |
| clarkb | mordred: doesn't look like our iptables role restarts iptables to apply the new rules? | 18:03 |
| clarkb | mordred: I've confirmed the FORWARD DROP default is in place in the /etc/iptables/rules.v4 file but it doesn't appear to be applied | 18:03 |
| clarkb | I need to pop out for a few and then prep for the meeting, but ^ appears to be the stat we are in now | 18:04 |
| mordred | clarkb: hrm. I guess we should add a handler to do a restart | 18:04 |
| mordred | clarkb: although we'll also likely want to do a blanket ansible ' | 18:04 |
| mordred | clarkb: although we'll also likely want to do a blanket ansible '*' -m shell -e 'restart iptables' | 18:05 |
| *** _alastor_ has quit IRC | 18:05 | |
| mordred | since we won't trigger a handler on subsequent patches | 18:05 |
| clarkb | oh wait maybe we have a handler and they only run at end of play not end of role? | 18:08 |
| mordred | oh - yeah. | 18:09 |
| clarkb | that must be it because its set now | 18:09 |
| mordred | clarkb: yes - we have handlers | 18:09 |
| mordred | so - yay! | 18:10 |
| *** wolverineav has joined #openstack-infra | 18:10 | |
| clarkb | I rmember this behavior tripping up our unbound setup on jobs | 18:10 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/nodepool master: [wip] Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 18:11 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/elastic-recheck master: Identify *POST* timeout failures individually https://review.openstack.org/625573 | 18:15 |
| *** armax has quit IRC | 18:17 | |
| *** wolverineav has quit IRC | 18:18 | |
| mnaser | how can I disable swift in the new devstack zuulv3 jobs? | 18:20 |
| *** trown is now known as trown|lunch | 18:20 | |
| mnaser | swift,s-{account,container-sync,container,proxy}: false in devstack_services | 18:20 |
| clarkb | mnaser: I think you add a localrc option that disabled swift | 18:21 |
| mnaser | i thought i can use devstack_services and it generated it magically | 18:21 |
| clarkb | I'm not sure if it works for deletions, or if that only works for addition | 18:21 |
| mnaser | aaah | 18:21 |
| mnaser | i missed s-object | 18:21 |
| mnaser | http://logs.openstack.org/48/625448/12/check/magnum-kubernetes-conformance/4098b25/controller/logs/_.localrc_auto.txt | 18:21 |
| mnaser | of course when i ask i find out | 18:21 |
| mnaser | clarkb: according to the role you can use it to disable, false will add a disable_service line | 18:21 |
| clarkb | there was a base job with no services enable to make the minimal use case easier (but that might be because listing everything to disable is buggy over time as services are added) | 18:22 |
| clarkb | ah | 18:22 |
| * mnaser is hacking around 2 node jobs that run conformance tests | 18:22 | |
| mnaser | is there any reason why the controller is unable to ssh into vms running in that cloud on a 2 node job? | 18:24 |
| clarkb | it should be able to for nova livemigration | 18:24 |
| clarkb | might be ssh as root | 18:24 |
| *** ykarel has quit IRC | 18:24 | |
| clarkb | oh wait the VMs | 18:25 |
| clarkb | you have to hit them on the floating IP network | 18:25 |
| clarkb | the private network is isolated from host networking | 18:25 |
| mnaser | it hits a floating ip | 18:25 |
| mnaser | OH | 18:25 |
| mnaser | you know what | 18:25 |
| mnaser | i remember | 18:25 |
| clarkb | that should work | 18:25 |
| *** _alastor_ has joined #openstack-infra | 18:25 | |
| mnaser | okay, i shouldn't be working on weekends | 18:26 |
| mnaser | devstack gate defaults to qemu only | 18:26 |
| clarkb | and is even tested on multinode tempest and grenade jobs | 18:26 |
| mnaser | and this is running against nested virt cloud only, so i had to override it to kvm | 18:26 |
| mnaser | and then since the override i hit other issues | 18:26 |
| mnaser | http://logs.openstack.org/48/625448/9/check/magnum-kubernetes-conformance/0899c50/ara-report/result/886aa306-7fbd-4409-bba7-f1c473a4bb91/ -- at the time it would get connection refused but i assume the lack of nested virt just made it not boot on time | 18:26 |
| clarkb | there are bugs with metadata service being slow though its not 100% failure | 18:27 |
| clarkb | I would dump the instance console log to debug | 18:27 |
| mnaser | clarkb: while i have you here.. in this review .. https://review.openstack.org/#/c/625448/ .. my conformance.yaml feels pretty wrong | 18:27 |
| mnaser | it seems like overriding job.run will not run more than one but replace it | 18:27 |
| mnaser | so i had to copy the contents of devstack.yml manually which feels pretty wrong.. | 18:28 |
| *** irdr has joined #openstack-infra | 18:28 | |
| *** wolverineav has joined #openstack-infra | 18:28 | |
| corvus | mnaser: make a job which runs the devstack playbook as a pre-playbook | 18:29 |
| clarkb | ya I think it only nests the pre and post playbooks | 18:29 |
| clarkb | run is unique to the current job workload | 18:29 |
| corvus | but that would have to be in the devstack repo | 18:29 |
| mnaser | so i should add a job to openstack-dev/devstack with that as a pre that is consumed downstream | 18:29 |
| mnaser | similar to the -consumer jobs that mordred has written | 18:29 |
| clarkb | mnaser: I want to say that already exists for eg tempest or sdk/shade jobs | 18:29 |
| mnaser | yeah it does, but it calls tox functional | 18:30 |
| clarkb | mnaser: but you override run | 18:30 |
| clarkb | so you should be able to run anything in run: ? | 18:30 |
| mnaser | ah -consumer has built in call for `playbooks/tox/pre.yaml` in pre-run | 18:31 |
| mnaser | so that might break in my case | 18:31 |
| mnaser | or do unnecessary things | 18:31 |
| clarkb | mnaser: pre and post nest like nesting dolls | 18:31 |
| mnaser | https://github.com/openstack-dev/devstack/blob/master/.zuul.yaml#L548-L570 -- right but the job already calls playbooks/tox/pre.yml there | 18:32 |
| clarkb | I believe that your jobs variables will apply to that pre run | 18:32 |
| clarkb | so you just use their pre, and define your own run.yaml | 18:33 |
| mnaser | right but im not running my tests using tox, it's just an ansible playbook job | 18:33 |
| *** wolverineav has quit IRC | 18:33 | |
| clarkb | but isn't tox part of the run.yaml? | 18:33 |
| clarkb | which allows you to supply your own run playbook that will not run tox | 18:33 |
| mnaser | hmm i was worried about `playbooks/tox/pre.yaml` possibly doing unnecessary things | 18:34 |
| *** e0ne has joined #openstack-infra | 18:34 | |
| *** wolverineav has joined #openstack-infra | 18:34 | |
| mnaser | but i guess i could just ignore it and let it do its thing | 18:34 |
| *** wolverineav has quit IRC | 18:34 | |
| *** wolverineav has joined #openstack-infra | 18:34 | |
| clarkb | the idea there aiui is to try and bottle up a common invocation, but still eb flexible enough to do something different. pre should just do cloud setup based on the vars you set. Then run is your actual test workload. If you reuse their run then you get tox. Or supply your own and consume the prebuilt cloud | 18:35 |
| mnaser | yeah it looks like there's not much harm done inside the tox/pre.yaml | 18:35 |
| mnaser | just bindep, test-setup (which might be a noop) and ensure-tox .. so i can live with those extra things happening even if i wont use them | 18:36 |
| clarkb | also you can totally have tox run ansible | 18:37 |
| clarkb | whether or not that makes sense is another story :) | 18:37 |
| mnaser | hmm yeah that could have benefits aka use my own version of ansible | 18:37 |
| mnaser | but extra complexity i dont wanna deal with | 18:37 |
| mnaser | hmm i might just create devstack-consumer job | 18:38 |
| mnaser | which doesnt do any tox stuff but has devstack in pre-run | 18:38 |
| clarkb | mwhahaha: EmilienM ssbarnea|rover ansible lint release appears to have exploded the tripleo gate | 18:39 |
| clarkb | not sure if you all have noticed yet | 18:39 |
| EmilienM | oh noes | 18:39 |
| EmilienM | ssbarnea|rover ^ | 18:39 |
| EmilienM | clarkb: any log file handy? | 18:39 |
| clarkb | http://logs.openstack.org/85/625585/1/gate/openstack-tox-linters/9e6a509/job-output.txt.gz | 18:40 |
| clarkb | is one example | 18:40 |
| EmilienM | thanks | 18:40 |
| *** e0ne has quit IRC | 18:40 | |
| EmilienM | 206 Variables should have spaces before and after: {{ var_name }} | 18:40 |
| EmilienM | damn | 18:40 |
| clarkb | and 120 characters is new line length limit | 18:40 |
| clarkb | etc | 18:40 |
| clarkb | don't compare to empty string | 18:40 |
| EmilienM | damn ansible-lint | 18:40 |
| EmilienM | we should pin | 18:41 |
| clarkb | 4.0.0 is the new release according to pabelanger in scrollback | 18:41 |
| clarkb | so I think you want < 4.0.0 | 18:41 |
| fungi | at least they went with a nice big shiny new major version bump there | 18:42 |
| mnaser | EmilienM: clarkb fwiw we pin ansible-lint in openstack-ansible | 18:42 |
| pabelanger | yes, 4.0.0 released today, with new default rules | 18:42 |
| mnaser | that kinda makes our lives a bit easier | 18:42 |
| pabelanger | https://docs.ansible.com/ansible-lint/rules/default_rules.html | 18:42 |
| fungi | most openstack projects pin their linters and other static code analysis tools and explicitly bump them at the start of a new release cycle to contain the madness | 18:44 |
| *** jpena is now known as jpena|off | 18:44 | |
| boden | hi. hopefully an easy question... will patches that depend on others run their zuul gate once the depends on patch lands? | 18:46 |
| *** rlandy is now known as rlandy|biab | 18:47 | |
| clarkb | boden: if they share the same gate queue value then yes | 18:47 |
| clarkb | boden: for example nova cinder swift glance keystone neutron are all in the "integrated" gate queue and they will have the behavior with each other | 18:48 |
| *** gyee has joined #openstack-infra | 18:48 | |
| boden | clarkb thanks.. they must not share the same gate queue then | 18:48 |
| EmilienM | woot we aleady pin ansible-lint in most of places | 18:53 |
| EmilienM | not tripleo-upgrade repo though but that's seems it so far | 18:53 |
| EmilienM | clarkb: thanks for the notification! | 18:53 |
| EmilienM | I'll keep an eye in case I missed some other repos | 18:53 |
| clarkb | ~5 minutes to the last infra meeting of the year | 18:54 |
| fungi | just enough time for everyone to pour some eggnog | 18:56 |
| fungi | where "eggnog" is code for whatever you've got | 18:56 |
| clarkb | I just had some orange juice | 18:57 |
| *** priteau has quit IRC | 18:58 | |
| mnaser | clarkb: https://review.openstack.org/#/c/625989/ came up with this | 19:00 |
| mordred | mnaser: kinda makes me want to invert that | 19:01 |
| mordred | mnaser: make devstack-consumer parent: devstack liek you have it - but then make devstack-tox-functional-consumer parent on devstack-consumer | 19:02 |
| mnaser | mordred: ok let me see how i can structure that cleanly | 19:02 |
| mordred | mnaser: I mean - I think what you have there is perfectly fine | 19:02 |
| mnaser | yeah but it'd be cleaner | 19:02 |
| mnaser | im just tryna see how it works in terms of devstack-tox-base => devstack-tox-functional => devstack-tox-functional-consumer existing tree | 19:03 |
| mnaser | mordred: if you have anything that comes to your mind, let me know but i'm not tooooo sure | 19:03 |
| *** armax has joined #openstack-infra | 19:04 | |
| *** e0ne has joined #openstack-infra | 19:05 | |
| *** eharney has quit IRC | 19:08 | |
| openstackgerrit | Merged openstack-infra/opendev-website master: Publish opendev website to afs on merge. https://review.openstack.org/625671 | 19:09 |
| *** eharney has joined #openstack-infra | 19:11 | |
| *** mriedem is now known as mriedem_afk | 19:22 | |
| corvus | ianw, mordred, frickler: can someone explain test_firewall to me in https://review.openstack.org/605585 ? why is that needed when we have the changes to test_base? and why is there no job that includes it in its files stanza? | 19:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Remove test_firewall.py https://review.openstack.org/625995 | 19:24 |
| corvus | ianw, mordred, frickler: okay, that's that. i'm a little confused because that was my comment on PS13, and mordred agreed. it's not clear to me how we ended up merging that despite agreeing it isn't necessary. | 19:26 |
| openstackgerrit | Merged openstack-infra/system-config master: Run a local MySQL service on StoryBoard servers https://review.openstack.org/623290 | 19:35 |
| *** aojea has quit IRC | 19:36 | |
| *** rlandy|biab is now known as rlandy | 19:40 | |
| *** agopi_ has joined #openstack-infra | 19:48 | |
| *** wolverineav has quit IRC | 19:48 | |
| *** wolverineav has joined #openstack-infra | 19:49 | |
| *** agopi has quit IRC | 19:50 | |
| *** trown|lunch is now known as trown | 19:51 | |
| *** agopi_ has quit IRC | 19:51 | |
| *** agopi_ has joined #openstack-infra | 19:52 | |
| ianw | frickler: yeah, as you noticed on that nodepool dogpile from source job, our install jobs are pretty openstack specific | 19:54 |
| *** wolverineav has quit IRC | 19:54 | |
| ianw | i started looking at use-cached-repos, which led me down a rabbit hole of ansible bugs to https://github.com/ansible/ansible/pull/50070 | 19:54 |
| ianw | anyway, i think we can probably automate handling of github repos, might be useful for other things too | 19:55 |
| mordred | ianw: don't know if you saw - but we cut an openstacksdk release - so hopefully we're a step closer to unravelling that issue? | 19:55 |
| *** manjeets has quit IRC | 19:55 | |
| *** rkukura has quit IRC | 19:56 | |
| ianw | mordred: excellent, i think with the workaround we're all good on that front | 19:58 |
| ianw | i'd hoped to get the networkmanager stuff, and f29 and potentially a rhel8 beta image up with it, but i think out of time for this year | 19:58 |
| corvus | clarkb, fungi, ianw, mordred: i restarted gitea with code indexing -- example: http://38.108.68.96/explore/code?q=zk | 19:59 |
| ianw | as mentioned in the meeting, don't want to push out something that breaks everyone's networking at this point | 19:59 |
| clarkb | corvus: seems quick. I wonder what index backend it is using | 20:00 |
| corvus | maybe something called 'bleve'? | 20:00 |
| dmsimard | gitea is pretty sweet. I love the built-in mirror feature. | 20:01 |
| corvus | https://github.com/blevesearch/bleve | 20:01 |
| ianw | corvus: lgtm ... i do like codesearch's file matching feature, this doesn't seem to have an extra option like that | 20:01 |
| clarkb | google says bleve is "boiling liquid expanding vapor explosion" | 20:01 |
| corvus | dmsimard: yes it's nice | 20:01 |
| corvus | though i should mention, i don't actually expect we'll set up the repos as mirrors. i did in the poc, but in reality, i think we want to have gitea think it's authoritative and push to it. | 20:01 |
| *** agopi__ has joined #openstack-infra | 20:02 | |
| mordred | yah | 20:02 |
| corvus | (you'll note in the poc, the repos say they are mirrors of the thing we are going to retire :) | 20:02 |
| corvus | clarkb: jives with their logo :) | 20:02 |
| corvus | ianw: good point | 20:02 |
| clarkb | ha indeed | 20:02 |
| dmsimard | Mirror is a pull, we can push with gerrit right now so /shrug | 20:02 |
| mordred | dmsimard: exactly | 20:02 |
| *** agopi_ has quit IRC | 20:03 | |
| ianw | corvus: hrm, it's a keyword not a regex search? or am i doing it wrong? | 20:04 |
| corvus | ianw: i've only tried keyword | 20:04 |
| fungi | yeah, if it's something like a tree of trigrams and not a full text index that could explain why it's fast | 20:05 |
| clarkb | bleve docs make it seem simpler than regex | 20:05 |
| clarkb | which isn't suprising lucene is simpler queryl ang too | 20:05 |
| clarkb | you can require terms and exlude terms | 20:05 |
| *** agopi_ has joined #openstack-infra | 20:10 | |
| *** agopi__ has quit IRC | 20:10 | |
| corvus | bleve has the idea of fields; i think gitea just uses the "Content" field. so i bet it could be improved without too much difficulty. | 20:11 |
| *** wolverineav has joined #openstack-infra | 20:14 | |
| corvus | https://github.com/go-gitea/gitea/blob/master/modules/indexer/repo.go#L96 | 20:15 |
| *** agopi_ has quit IRC | 20:16 | |
| *** agopi_ has joined #openstack-infra | 20:16 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for elasticsearch.openstack.org https://review.openstack.org/615665 | 20:17 |
| clarkb | system-config is still installing ansible-list 3.5.1. I don't see any reason for it to not install 4.0.0 though | 20:19 |
| clarkb | http://logs.openstack.org/95/625995/1/gate/tox-linters/73b1520/tox/linters-1.log anyone have ideas | 20:19 |
| clarkb | could be that we are caching the indexes for longer than the epxected 10 minutes on hte mirror nodes? | 20:19 |
| clarkb | but if I pull up the index locally on that mirror I see ansible-lint 4.0.0 | 20:20 |
| *** agopi_ has quit IRC | 20:20 | |
| *** agopi_ has joined #openstack-infra | 20:20 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Cap ansible-lint at < 4.0.0 https://review.openstack.org/625999 | 20:21 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Add cgroup support to ram sensor https://review.openstack.org/549506 | 20:21 |
| clarkb | That cap will ensure we are ready to merge a fix if it is a delayed cache eviction problem | 20:22 |
| ianw | clarkb: hrm, this i just after your change to ignore the cache busting merged? | 20:22 |
| clarkb | ianw: ya, though we are definitely seeing some jobs install 4.0.0 and fail (tripleo for example) | 20:23 |
| clarkb | running tox locally I get 4.0.0 though | 20:23 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove test_firewall.py https://review.openstack.org/625995 | 20:26 |
| clarkb | I wonder if we are ignoring the server side cache control max age of 10 minutes | 20:27 |
| clarkb | and falling back to our one day cache default | 20:27 |
| *** _alastor_ has quit IRC | 20:29 | |
| *** jtomasek has joined #openstack-infra | 20:29 | |
| clarkb | in my web-browser I see cache control max-age is still 600s | 20:30 |
| clarkb | I'll tcpdump pip again | 20:30 |
| *** _alastor_ has joined #openstack-infra | 20:30 | |
| *** rlandy is now known as rlandy|brb | 20:30 | |
| ianw | clarkb: how would your browser see 4.0.0 in the index but pip not? | 20:31 |
| clarkb | ianw: ya exactly. I don't know | 20:31 |
| ianw | or unless something like whatever pip sends doesn't trigger a remote lookup, until a browser hits the mirror, it's serving up 3.x -- i.e. the browser busts the cache | 20:32 |
| clarkb | and using the same mirror locally installing ansible-lint installs 4.0.0 | 20:32 |
| clarkb | maybe | 20:32 |
| *** bobh has quit IRC | 20:32 | |
| *** bobh has joined #openstack-infra | 20:33 | |
| clarkb | that said if we see max-age 600 I would expect the apache cache to honor that so we should only cache the index for 10 minutes | 20:33 |
| clarkb | unless the ignore cache control docs aren't clear that it affects the backend reponse headers too | 20:33 |
| *** jtomasek has quit IRC | 20:34 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: [dnm] testing against base-test https://review.openstack.org/626002 | 20:35 |
| clarkb | ianw: http://logs.openstack.org/85/625585/1/gate/openstack-tox-linters/9e6a509/zuul-info/host-info.ubuntu-xenial.yaml that failed on ansible lint 4.0.0 in inap and ran before our passing job there that installed 3.5.1 | 20:36 |
| clarkb | I don't know that it is the cache | 20:36 |
| mnaser | do we have a zuul role that installs golang right now? i couldn't find one but i know we have things that use golang | 20:37 |
| clarkb | mnaser: on the kata jobs I just instaleld it from the distro because it was quite new iirc | 20:37 |
| clarkb | yup that is what i did | 20:38 |
| mnaser | clarkb: ok, i might write up a small role to be able to deploy specific versions then | 20:38 |
| clarkb | mnaser: I would be wary of relying on github's hosting for that as in your comments | 20:38 |
| clarkb | looks like ubuntu bionic has 1.7-1.0 and xenial has 1.6, 1.7, 1.9 and 1.10 | 20:39 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add cgroup support to ram sensor https://review.openstack.org/549506 | 20:40 |
| mnaser | clarkb: i dont think it relies on github | 20:41 |
| mnaser | looks like its dl.google.com for golang | 20:41 |
| clarkb | mnaser: the url you had was for a githubcontent.com url | 20:41 |
| clarkb | in your comments for the one playbook | 20:41 |
| mnaser | oh yes | 20:41 |
| mnaser | im dropping that gimme stuff | 20:41 |
| mnaser | because its just a wrapper and we should go stuff directly using ansible | 20:41 |
| mnaser | instead of using a wrapper which calls shell commands via ansible | 20:42 |
| mnaser | just gonna pull down bins via https://dl.google.com/go/go1.11.4.darwin-amd64.pkg directly | 20:42 |
| clarkb | I would just install it from the distro since you are ubuntuing | 20:42 |
| *** agopi_ has quit IRC | 20:42 | |
| clarkb | we mirror that, should be plenty reliable and its a quite new version of go | 20:42 |
| mnaser | oh xenial has 1.10 | 20:42 |
| mnaser | thats what i actually need | 20:42 |
| mnaser | good enough | 20:42 |
| clarkb | yes and bionic too | 20:42 |
| mnaser | ill drop it into bindep too with a custom profile | 20:42 |
| *** agopi_ has joined #openstack-infra | 20:44 | |
| *** agopi_ has quit IRC | 20:44 | |
| *** agopi_ has joined #openstack-infra | 20:44 | |
| *** agopi_ has quit IRC | 20:46 | |
| *** rlandy|brb is now known as rlandy | 20:48 | |
| *** mriedem_afk is now known as mriedem | 20:50 | |
| clarkb | http://logs.openstack.org/f2/f24073cf1e3210901da0eb1816a07a20b4e726a6/post/opendev-website-publish/a279205/job-output.txt.gz#_2018-12-18_19_11_36_139589 is why we have no opendev website content. If I try to cd into /afs/openstack.org/project/opendev.org I get permission denied | 20:50 |
| clarkb | I want to say we ran into similar with starlingx | 20:50 |
| clarkb | and it had to do with not tying the .openstack.org RW volume to the openstack.org RO volume | 20:50 |
| clarkb | fungi: corvus ^ do you recall? | 20:50 |
| clarkb | oh though if we can't write to it with that afs tab then that might be a write permission issue too | 20:51 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing tree checkout tagging https://review.openstack.org/626006 | 20:52 |
| pabelanger | mkdir: cannot create directory ‘/afs/.openstack.org/project/opendev.org’: Permission denied | 20:52 |
| pabelanger | clarkb: looks like it | 20:52 |
| *** eernst has joined #openstack-infra | 20:54 | |
| fungi | clarkb: i think i updated the docs... looking | 20:56 |
| fungi | in particular there's no need for a read-only replica | 20:56 |
| clarkb | ah | 20:57 |
| clarkb | it looks like I need to reboot sothat I have an up to date kernel with afs module | 20:57 |
| clarkb | until then aklog not working | 20:57 |
| clarkb | and also it is lunch time so I might take that as an excuse to go through the pain fo a reboot | 20:58 |
| fungi | oh | 20:58 |
| *** jamesmcarthur has quit IRC | 20:58 | |
| fungi | "If the volume’s mountpoint lies within another volume, you may also need to vos release that parent volume before it will show up in the read-only path." | 20:59 |
| *** gfidente has quit IRC | 20:59 | |
| fungi | did you vos release the parent volume? | 20:59 |
| fungi | that's what i missed the first time, i think, which is why i updated the docs | 20:59 |
| fungi | the child mountpoint won't exist until the parent volume is updated | 21:00 |
| clarkb | I didnt set afs up, I think corvus did? | 21:00 |
| fungi | oh, got it. i'll see if i can spot it | 21:00 |
| clarkb | system config already had the parent job stuff in place and ls on afs showed the dir was there | 21:01 |
| *** mriedem has quit IRC | 21:02 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/system-config master: Adds ServerAliveInterval to ansible ssh_args to avoid frozen connections https://review.openstack.org/626009 | 21:04 |
| *** mriedem has joined #openstack-infra | 21:04 | |
| fungi | yeah, so far not finding anything missing | 21:04 |
| fungi | there's definitely a /afs/.openstack.org/project/opendev.org | 21:04 |
| fungi | File /afs/openstack.org/project/opendev.org (536871059.1.1) contained in volume 536871059 | 21:05 |
| fungi | also matches the .openstack.org path output from fs listvol | 21:06 |
| fungi | Volume status for vid = 536871059 named project.opendev | 21:06 |
| fungi | Current disk quota is 100000 | 21:06 |
| openstackgerrit | Merged openstack-infra/glean master: Fix argv patching in unit tests https://review.openstack.org/618962 | 21:08 |
| openstackgerrit | Merged openstack-infra/glean master: Pass arguments rather than distro to utility functions https://review.openstack.org/618963 | 21:08 |
| openstackgerrit | Merged openstack-infra/glean master: Add NetworkManager distro plugin support https://review.openstack.org/618964 | 21:08 |
| fungi | `fs listacl /afs/.openstack.org/project/opendev.org` lacks "docs rlidwk" | 21:09 |
| fungi | it only has "system:administrators rlidwka" | 21:10 |
| fungi | so i think that's it | 21:10 |
| mordred | fungi: ah - so maybe it just needs an fs sa | 21:10 |
| fungi | we need to grant the docs principal access | 21:10 |
| mordred | yeah | 21:10 |
| fungi | er, account/group i mean, not principal | 21:11 |
| fungi | does `fs setacl /afs/.openstack.org/project/opendev.org docs write` look correct? | 21:12 |
| *** dpawlik has joined #openstack-infra | 21:12 | |
| fungi | also should we grant read to system:anyuser as well? | 21:12 |
| fungi | the others seem to have "system:anyuser rl" so i'm thinking yes | 21:12 |
| fungi | trying those now, though it's taking its time to respond to me | 21:13 |
| fungi | okay, acls look like they match the others now | 21:14 |
| fungi | i can retrigger that commit in post | 21:14 |
| fungi | i wonder why we didn't record zuul info for http://logs.openstack.org/f2/f24073cf1e3210901da0eb1816a07a20b4e726a6/post/opendev-website-publish/a279205/ | 21:15 |
| clarkb | it is becauseit waslocalhost only job | 21:18 |
| clarkb | fwiw my aklog still not working here so maybe tumbleweed broke its afs packages | 21:18 |
| fungi | i've reenqueued now | 21:18 |
| clarkb | thnk you | 21:18 |
| fungi | it's waiting for a node assignment | 21:19 |
| fungi | console stream is http://zuul.openstack.org/stream/ba1fd7f611c74ea881754e74d26c5346?logfile=console.log | 21:20 |
| clarkb | I should set up a debuntu vm just for afs | 21:21 |
| fungi | the synchronize task worked this time, or so it says | 21:21 |
| fungi | https://opendev.org/ has content! | 21:21 |
| *** dpawlik has quit IRC | 21:22 | |
| *** rcernin has joined #openstack-infra | 21:22 | |
| fungi | also storyboard and storyboard-dev have mysql servers running on them now. hopefully tomorrow i can do a brief outage to switch the services over | 21:23 |
| *** rkukura has joined #openstack-infra | 21:24 | |
| clarkb | yay | 21:25 |
| clarkb | so it was dir level acls | 21:26 |
| openstackgerrit | Merged openstack-infra/glean master: A systemd skip for Debuntu systems https://review.openstack.org/620420 | 21:26 |
| openstackgerrit | Merged openstack-infra/glean master: Change openstack-dev to openstack-discuss https://review.openstack.org/622329 | 21:26 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add cgroup support to ram sensor https://review.openstack.org/549506 | 21:26 |
| fungi | clarkb: if you're going to do a vm for openafs, debian/unstable and debian/testing have openafs 1.8.2, same as ubuntu cosmic | 21:26 |
| *** jamesmcarthur has joined #openstack-infra | 21:26 | |
| *** jamesmcarthur has quit IRC | 21:27 | |
| *** jamesmcarthur has joined #openstack-infra | 21:27 | |
| corvus | clarkb, fungi: everything ok with afs now? | 21:27 |
| fungi | so newer than the 1.8.0 pre-release bionic has at any rate | 21:27 |
| fungi | corvus: yep, we just needed to grant acls for the website volume | 21:28 |
| clarkb | that is what I've got too but have pioctl errors | 21:28 |
| fungi | everything else looks correct | 21:28 |
| clarkb | I think because kernel module and kernel are mismtached somehow | 21:28 |
| fungi | corvus: and after adding the usual acls for docs publication and world-readability, retriggering the build published us a https://opendev.org/ | 21:29 |
| corvus | yay! i'll talk to the freenode folks next | 21:30 |
| fungi | thanks! | 21:30 |
| *** jamesmcarthur has quit IRC | 21:30 | |
| *** tpsilva has quit IRC | 21:31 | |
| *** rcernin has quit IRC | 21:37 | |
| *** dpawlik has joined #openstack-infra | 21:37 | |
| *** eharney has quit IRC | 21:40 | |
| *** dpawlik has quit IRC | 21:42 | |
| mordred | TIL: the nodejs package depends on python2.7-minimal | 21:42 |
| fungi | wow | 21:43 |
| *** manjeets has joined #openstack-infra | 21:48 | |
| clarkb | actually I wonder how much I can use kafs for | 21:49 |
| clarkb | ianw ^ I think you were looking into that. Im assuming a 4.19 kernel can read afs things with kafs | 21:49 |
| *** e0ne has quit IRC | 21:51 | |
| *** trown is now known as trown|outtypewww | 21:56 | |
| *** wolverineav has quit IRC | 21:59 | |
| *** wolverineav has joined #openstack-infra | 22:00 | |
| *** rcernin has joined #openstack-infra | 22:04 | |
| *** boden has quit IRC | 22:04 | |
| ianw | clarkb: it seems the client side of things is still coming along for kafs, although the main bits are there | 22:05 |
| *** wolverineav has quit IRC | 22:06 | |
| *** wolverineav has joined #openstack-infra | 22:08 | |
| *** jamesmcarthur has joined #openstack-infra | 22:11 | |
| *** kgiusti has left #openstack-infra | 22:15 | |
| *** jamesmcarthur has quit IRC | 22:16 | |
| *** yboaron has joined #openstack-infra | 22:16 | |
| corvus | ianw: does it work enough to use it on the mirror nodes? -- they are unauthenticated read-only | 22:19 |
| ianw | corvus: you read my mind : https://review.openstack.org/#/c/623974/ :) | 22:21 |
| ianw | i think it could, but the client packaging is holding things up but apparently that's being worked on. also i think we want fedora 29 kernels for all bug fixes ... that's been a saga between glean/openstacksdk/etc getting going | 22:22 |
| ianw | from the kafs perspective, it would be great if we could show it actually working like this to strengthen an argument for inclusion in rhel8 (and ergo centos8). currently there's discussions about if it should be or not but showing real world usage is better than just words | 22:25 |
| corvus | ianw: it might help to grab syslog from that job | 22:27 |
| clarkb | ianw: cern might also be able to show real world usage? iirc they are big rhel/centos users and big afs users | 22:28 |
| *** manjeets has quit IRC | 22:30 | |
| *** _ody has joined #openstack-infra | 22:30 | |
| ianw | corvus: you read my mind on collecting syslog from jobs :) https://review.openstack.org/#/c/624827/ | 22:30 |
| clarkb | ianw: btw dmsimard caught a fun system-config ansible job behavior. We install a cron to run ansible and that cron ansible will update the nested ara with failed runs :) | 22:31 |
| corvus | ianw: but the kafs change is failing on role-integration -- that doesn't inherit from the ansible jobs, right? | 22:31 |
| ianw | clarkb: oh, haha, i didn't think of that :) | 22:31 |
| clarkb | we may want to disable the cron or configure it to use a different ansible.cfg so it doesn't use ara | 22:31 |
| clarkb | it is a non issue except it pollutes the debugging data | 22:31 |
| ianw | corvus: ah, true, yeah might need to port. anyway, i got a bit stuck because the client packages were mounting /kafs (not /afs) and the instructions about how to mount were unclear | 22:32 |
| ianw | but i think in the last few days, the client packages have been updated and now mount at /afs again | 22:32 |
| corvus | ianw: huh. um. i kinda think "/afs" should be considered an API | 22:33 |
| corvus | ok. whew. | 22:33 |
| ianw | yeah, i've had a little discussion with kafs developers over the job in private mails, they're aware but it's a busy time | 22:34 |
| ianw | my real initial hope was to get them interested with enough framework that it could integrate with them, and expound the greatness of zuul :) | 22:34 |
| ianw | certainly i think we could be useful 3rd party ci for the client tools, etc, if the jobs are written | 22:35 |
| *** rcernin has quit IRC | 22:37 | |
| _ody | Hey folks, I am looking for an email address to redirect the openstack space on the Puppet forge to. Tobias Urdin informed to use infra-root@openstack.org but looks like that's already being used and forge emails are used as logins so needs to be unique. | 22:39 |
| _ody | Does + address work for you folks? | 22:40 |
| *** dkehn has quit IRC | 22:40 | |
| corvus | interesting. i don't see a record of a puppetforge login in our credentials file | 22:40 |
| *** dkehn has joined #openstack-infra | 22:41 | |
| clarkb | I don't actually know if our mail host for that account works with +strings | 22:41 |
| clarkb | corvus: I seem to recall mordred maybe setting something up a long long time ago for puppet modules? | 22:41 |
| *** rcernin has joined #openstack-infra | 22:41 | |
| corvus | _ody: do you know what account is using that address? can we do a recovery to find out? | 22:41 |
| corvus | clarkb: credentials file or it didn't happen :) | 22:41 |
| _ody | corvus: I don't. I'd need to ping devs to look in db | 22:41 |
| clarkb | also if we do that recovery can we use the existing account for the openstack puppet modules? | 22:42 |
| corvus | _ody: i have done a "forgot password" thing on that account | 22:42 |
| corvus | i will go look for mail now | 22:42 |
| clarkb | corvus: thank you | 22:42 |
| ianw | corvus: yeah, i just did the same, we got one | 22:42 |
| corvus | ianw: all you then :) | 22:42 |
| _ody | clarkb: Up to you and Tobias, really. You folks can have the openstack space if you want it. | 22:43 |
| *** rcernin has quit IRC | 22:43 | |
| corvus | ianw: i'm out of the passwords file if you want to update it | 22:43 |
| ianw | ok, let me generate one and see what this account is about | 22:43 |
| *** rcernin has joined #openstack-infra | 22:45 | |
| _ody | Puppet forge, sadly doesn't understand orgs so forge.puppet.com/openstack maps directly to the openstack user. | 22:45 |
| ianw | https://forge.puppet.com/users/openstackinfra | 22:45 |
| ianw | is it | 22:45 |
| *** manjeets has joined #openstack-infra | 22:45 | |
| ianw | ok, credentials updated for future reference | 22:47 |
| ianw | _ody: hrm, so we'll never be able to set the email to infra-root@openstack.org if it's taken by the openstackinfra user? | 22:48 |
| *** bobh has quit IRC | 22:48 | |
| _ody | ianw: Looks that way. | 22:48 |
| clarkb | in the last 12 hours bhs1 is not dominating our query for timed out jobs so I think it is doing well. There is still a non zero number of timeouts but it is middle of the pack | 22:48 |
| clarkb | _ody: ianw and oepnstackinfra can't publish to openstack/ because those are individual users? | 22:48 |
| _ody | clarkb: Correct. | 22:49 |
| ianw | i can't see any way to close your account via the web interface -- clearly we don't use openstackinfra account | 22:49 |
| corvus | wait i don't understand that last thing | 22:49 |
| clarkb | ianw: we should be able to try the +string email thing really quicklly and if it delivers to that mail host we should be able to use it | 22:49 |
| corvus | why can't openstackinfra publish to openstack/ ? | 22:49 |
| clarkb | corvus: because that path root is a user's namespace. And openstackinfra and openstack are distinct users. They aren't logical paths | 22:50 |
| corvus | oooh i see | 22:50 |
| corvus | we can make aliases. we have a bunch. | 22:51 |
| corvus | so if + addressing doesn't work, we can try that. | 22:51 |
| ianw | my test + mail has neither been rejected nor seems to have made it to the inbox | 22:52 |
| corvus | did it land in a folder named for what's after the +? did what's after the + exist as a folder already? | 22:52 |
| ianw | i sent it to infra-root+test | 22:53 |
| *** yboaron has quit IRC | 22:56 | |
| corvus | ianw: i see it filed in the test folder | 22:56 |
| corvus | _ody, ianw, clarkb: so i think we could use "infra-root+puppet-openstack@openstack.org" or something | 22:58 |
| ianw | yeah, i just realised my imap isn't subscribed to a bunch of folders | 22:58 |
| *** rh-jelabarre has quit IRC | 23:00 | |
| ianw | corvus: it looks like i can change the github username without too much fuss at this point (without having to set it all up again). how strongly do you feel? to avoid similar things to ^ what name do you suggest? | 23:00 |
| clarkb | http://logs.openstack.org/58/619258/1/check/nodepool-functional-py35-ubuntu-src/0fa1b77/controller/logs/screen-nodepool-launcher.txt.gz#_Dec_18_19_47_53_329401 is a zookeeper connection loss in nodepool functional tests on bhs1 since we reenabled bhs1 | 23:02 |
| clarkb | looking at the dstat data disk IO is in the undreds of MB/s around that time | 23:03 |
| clarkb | far better than the single digit MB/s in the past | 23:03 |
| _ody | email confirmation on the way | 23:04 |
| corvus | ianw: not very strongly, i doubt we will wedge ourselves there. i was thinking something involving opendev -- however, maybe we keep the openstack-ish username for managing the openstack orgs, and make a new one for the opendev zuul app.... | 23:05 |
| clarkb | I think that is a good indication we've fixed the disk io troubles. But may indicate there is something else going on as well? | 23:05 |
| ianw | _ody: ok, got it. so what do you want, a secret for writing a job to publish things? | 23:06 |
| clarkb | it predominantly hits the nodepool ubuntu job, but also affects the other nodepool jobs | 23:06 |
| _ody | ianw: I am no longer working on openstack related things, I am not even a dev or operator anymore so trying to turn over to current PTL of the community module project. | 23:07 |
| _ody | That person seemed to ping you folks has he was directed at infra-root. | 23:07 |
| *** bobh has joined #openstack-infra | 23:07 | |
| ianw | _ody: oh, ok, Tobias will want that going forward then | 23:08 |
| *** eernst has quit IRC | 23:08 | |
| ianw | i'll reset the password, and store it in our repo. we can then generate a secret for use in automated publishing | 23:08 |
| clarkb | ianw: sounds good | 23:08 |
| clarkb | looks like we have connectionlosses in gra1 and inap mtl01 too | 23:08 |
| clarkb | but bhs1 does it more | 23:08 |
| clarkb | looks like rax dfw did it once on the far end of the hits | 23:10 |
| clarkb | its also a largely new behavior with most hits being the last couple days (we on'y turned on bhs1 earlier today) | 23:11 |
| clarkb | that could be related to slow week for nodepool last week | 23:11 |
| clarkb | maybe we want ot put the functional test zk disk on a tmpfs too (we put unittests there already) | 23:13 |
| *** wolverineav has quit IRC | 23:14 | |
| _ody | ianw: Thanks! | 23:15 |
| ianw | _ody tobias-urdin : ok, thanks, the openstack account is now updated and we have the credentials in our usual place | 23:15 |
| *** bobh has quit IRC | 23:17 | |
| *** wolverineav has joined #openstack-infra | 23:17 | |
| *** wolverineav has quit IRC | 23:17 | |
| *** wolverineav has joined #openstack-infra | 23:17 | |
| *** bobh has joined #openstack-infra | 23:18 | |
| *** _ody has left #openstack-infra | 23:20 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Run devstack zookeeper on tmpfs https://review.openstack.org/626038 | 23:21 |
| clarkb | I don't actually know if that will make things better or worse considering the memory pressure involved | 23:21 |
| clarkb | dstat seems to show we should have enough memroy in the jobs for that | 23:22 |
| *** bobh has quit IRC | 23:22 | |
| openstackgerrit | MarcH proposed openstack-infra/git-review master: Make it possible to configure draft as default push mode https://review.openstack.org/220426 | 23:23 |
| clarkb | as a sanity check all 4 of the current integrated gate failures did not run on bhs1 | 23:25 |
| clarkb | I think things are mostly looking good and maybe zookeeper has a specific sad there | 23:26 |
| clarkb | ianw: ^ thinking out loud here part of the trouble may be running dib + qemu + zookeeper + mysql and all of openstack | 23:26 |
| clarkb | dib is going to build the first image then nodepool will immediately start booting that while dib moves to build the second image | 23:26 |
| *** armax has quit IRC | 23:29 | |
| *** efried has joined #openstack-infra | 23:29 | |
| *** efried has quit IRC | 23:34 | |
| *** _alastor_ has quit IRC | 23:37 | |
| openstackgerrit | MarcH proposed openstack-infra/git-review master: Add hint to just "git remote rename origin gerrit" when no .gitreview https://review.openstack.org/626043 | 23:39 |
| *** tosky has quit IRC | 23:40 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Trim devstack services used in testing https://review.openstack.org/626044 | 23:41 |
| *** smarcet has joined #openstack-infra | 23:51 | |
| *** wolverineav has quit IRC | 23:53 | |
| *** yolanda has quit IRC | 23:55 | |
| *** dave-mccowan has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!