| *** yamamoto has joined #openstack-infra | 00:01 | |
| *** mattw4 has quit IRC | 00:10 | |
| *** weifan has joined #openstack-infra | 00:13 | |
| donnyd | clarkb: can you run fio in another instance to give me back some data. I need to know if I am actually making progress or not | 00:17 |
|---|---|---|
| *** lseki has quit IRC | 00:17 | |
| donnyd | https://www.irccloud.com/pastebin/zgLxUacE/ | 00:18 |
| donnyd | This is what I am getting in an instance now... was more like 4k iops in the vm | 00:18 |
| clarkb | donnyd: ya let me see | 00:19 |
| donnyd | If this is not good, then I have a bit more tuning I can do to get a couple more k if need be... but that will be about all it can do | 00:20 |
| clarkb | donnyd: hrm doesn't seem to like my credentials | 00:21 |
| clarkb | did those get wiped/changed during the updates? | 00:21 |
| donnyd | yea | 00:21 |
| donnyd | The ones I gave you for openstackci and openstackzuul should be working | 00:21 |
| donnyd | https://www.irccloud.com/pastebin/3tUjxjKl/ | 00:21 |
| donnyd | Checking | 00:22 |
| donnyd | They are there | 00:22 |
| clarkb | donnyd: did the passwords change? | 00:23 |
| donnyd | no | 00:23 |
| clarkb | I updated the passwords on our side, but using the ones you set and the ones I set doesn't seem to work | 00:23 |
| clarkb | ok let me recheck my clouds.yaml | 00:23 |
| *** weifan has quit IRC | 00:28 | |
| clarkb | yup got it was os_project_domain_name chagned from Default to default I think | 00:29 |
| *** weifan has joined #openstack-infra | 00:29 | |
| clarkb | donnyd: so I should go ahead and reupload our image as qcow2 then boot an instance and run some tests? | 00:32 |
| donnyd | yea you can go ahead | 00:32 |
| *** rfarr_ has quit IRC | 00:32 | |
| donnyd | The openstackci project should be good to go | 00:33 |
| *** rfarr_ has joined #openstack-infra | 00:33 | |
| *** tdasilva has quit IRC | 00:33 | |
| clarkb | what about openstackzuul? | 00:34 |
| clarkb | (thats where I'll upload to since it will run the test instances) | 00:34 |
| *** tdasilva has joined #openstack-infra | 00:34 | |
| *** hamzy has quit IRC | 00:35 | |
| donnyd | still working on the ipv6 part | 00:35 |
| *** smarcet has left #openstack-infra | 00:36 | |
| clarkb | ok image is uploading | 00:37 |
| *** gyee has quit IRC | 00:37 | |
| clarkb | except I am doing the raw image woops | 00:37 |
| clarkb | I'll stop it, delete and start again | 00:37 |
| *** jamesmcarthur has joined #openstack-infra | 00:40 | |
| *** weifan has quit IRC | 00:41 | |
| donnyd | :) | 00:42 |
| donnyd | They would still have worked | 00:43 |
| clarkb | ya but the upload takes forever because the raw image size isn't small :/ | 00:43 |
| ianw | auristor: is there anything sensitive in the audit logs? | 00:43 |
| *** ijw has quit IRC | 00:43 | |
| clarkb | donnyd: oh re fio are you asking us to run in one of your cloud instances or on another cloud? | 00:46 |
| clarkb | (I read it previously as on your cloud but not sure if that is what you emant now looking at it again) | 00:46 |
| donnyd | yea, I was curious to see | 00:46 |
| donnyd | If I am 50K IOPS away, then I may need to procure some better gear for this workload | 00:47 |
| clarkb | I think devstack jobs actually record that /me looks for some links | 00:48 |
| *** rfarr_ has quit IRC | 00:50 | |
| *** weifan has joined #openstack-infra | 00:50 | |
| *** rfarr_ has joined #openstack-infra | 00:50 | |
| clarkb | donnyd: if you download http://logs.openstack.org/12/665512/3/check/tempest-full/c5e5fe3/controller/logs/dstat-csv_log.txt.gz then put it in https://lamada.eu/dstat-graph/ it will give you some idea | 00:51 |
| clarkb | there are spikes in the 10k ranges but mostly its much loewr than that | 00:51 |
| *** jamesmcarthur has quit IRC | 00:51 | |
| clarkb | I think the interesting thing will be to run tempest on these nodes though as there are other factors too | 00:52 |
| donnyd | ok, so I am surely in the ballpark | 00:52 |
| clarkb | and I'll work on that. Though dinner is served here now so maybe that will have to be a morning activity | 00:52 |
| donnyd | Ok, the openstackzuul project should be good to hook | 00:53 |
| donnyd | testing now | 00:53 |
| *** jamesmcarthur has joined #openstack-infra | 00:53 | |
| donnyd | your image is done | 00:54 |
| clarkb | ya I just created my ssh key too | 00:54 |
| clarkb | if you are happy with the network I'll boot a node, start nova unittests then eat dinner | 00:54 |
| donnyd | it works | 00:56 |
| donnyd | test complete | 00:56 |
| donnyd | :) | 00:56 |
| clarkb | k booting instance then thanks | 00:56 |
| donnyd | try again | 00:57 |
| donnyd | selinux policies with containers and NFS have been killling me | 00:57 |
| clarkb | image has no associated data? trying again | 00:58 |
| donnyd | yea, its an selinux thing on the glance side.. not sure why it does that yet... | 00:58 |
| donnyd | It still does seem to take quite a while for nova to pull down the image | 00:59 |
| *** ccamacho has quit IRC | 01:00 | |
| clarkb | I think nova may convert it to raw by default too | 01:00 |
| donnyd | yea that could be it... but I don't think its at that stage yet. | 01:01 |
| donnyd | https://www.irccloud.com/pastebin/TrMfvC4I/ | 01:01 |
| clarkb | ya not sure how it gets logged | 01:02 |
| donnyd | i stand corrected | 01:02 |
| donnyd | 42436 277229 46.3 0.0 476904 36284 ? Sl 01:01 0:04 qemu-img convert -t none -O raw -f qcow2 /var/lib/nova/instances/_base/e956c6c8779473194d0964072073a2efe98d85d9.part /var/lib/nova/instances/_base/e956c6c8779473194d09640 | 01:02 |
| *** weifan has quit IRC | 01:02 | |
| clarkb | that can be changed (and force it to boot off qcow2, not sure how that will affect io though) | 01:02 |
| *** ccamacho has joined #openstack-infra | 01:02 | |
| donnyd | If performance is close enough, I won't really worry too much. Will only have to wait once a day | 01:03 |
| clarkb | and I'm in | 01:04 |
| *** yamamoto has quit IRC | 01:04 | |
| donnyd | yey... that only took 3 days of refactoring to get back to where we started | 01:04 |
| donnyd | LOL | 01:04 |
| donnyd | After this test is done, I will bring up the rest of the hypervisors and we will be in business | 01:09 |
| *** tdasilva_ has joined #openstack-infra | 01:10 | |
| *** jamesmcarthur has quit IRC | 01:10 | |
| *** jamesmcarthur has joined #openstack-infra | 01:10 | |
| *** imacdonn has quit IRC | 01:10 | |
| clarkb | the nova unittests are running | 01:11 |
| *** imacdonn has joined #openstack-infra | 01:11 | |
| *** tdasilva has quit IRC | 01:12 | |
| *** jamesmcarthur has quit IRC | 01:15 | |
| donnyd | enjoy your dinners | 01:15 |
| *** hongbin has joined #openstack-infra | 01:16 | |
| clarkb | http://paste.openstack.org/show/753733/ looks much bette | 01:17 |
| clarkb | particularly the db tests that are listed with times there | 01:17 |
| *** redrobot has quit IRC | 01:17 | |
| clarkb | previously worst case was like 160s now 49s | 01:17 |
| clarkb | we'll need to run tempest too which I can do in the morning | 01:18 |
| clarkb | thanks! | 01:18 |
| donnyd | I will take a 100% improvement | 01:18 |
| donnyd | Ok cool | 01:18 |
| donnyd | I will work on bringing up the rest of the gear then. Api's might be up and donwn | 01:18 |
| clarkb | total runtime was 501s before now 397s | 01:19 |
| donnyd | So that is closer to what I saw the other clouds doing | 01:19 |
| donnyd | they were somewhere in that 400 range | 01:19 |
| clarkb | yup | 01:20 |
| donnyd | Well enjoy your evening. I am hopeful tomorrow will be the day | 01:21 |
| donnyd | Is it ok if I take down your instance? | 01:22 |
| corvus | clarkb: i'd like to add dstat_graph to jobs (see https://review.opendev.org/610100 ) but that just clones from github... we need a way to avoid doing that in every post playbook... got any ideas? add a new caching proxy for random github projects? DIB element to cache it on our images? beg a DD to package it? install docker and run it in a docker container? | 01:22 |
| donnyd | I have a couple sysctls on the nfs server that require a rebbot | 01:22 |
| donnyd | If its running tests, it's not a big deal... can be done any time | 01:23 |
| clarkb | corvus: we could clone it into our images | 01:23 |
| clarkb | donnyd: yes you can take down the instance | 01:23 |
| donnyd | ok thanks | 01:23 |
| corvus | clarkb: yeah, that feels like the most actionable thing i can think of. we can write the zuul-jobs role to optionall take a path to a cached version, otherwise clone from github, so it's still univesally usable | 01:24 |
| clarkb | ++ | 01:24 |
| clarkb | I'll also reset passwords to their values in hiera tomorrow morning | 01:24 |
| clarkb | hrm maybe I hsould just go ahead and do that now? | 01:25 |
| corvus | k, i'll plan on doing that. but right now, i'm halfway through "leviathan wakes" and that's not nearly far enough. | 01:25 |
| clarkb | ha | 01:25 |
| clarkb | ya I'll reset passwords to hiera values now so that others can interact with the fortnebula cloud if necessary | 01:25 |
| clarkb | thats done. Values in hiera are correct now however I also need to update clouds.yaml for the project_domain_name change from Default to default. That will happen tomorrow | 01:30 |
| donnyd | awesome. see you tomorrow clarkb | 01:30 |
| donnyd | I will have the rest of things up in the morning | 01:31 |
| *** rfarr_ has quit IRC | 01:33 | |
| *** rfarr_ has joined #openstack-infra | 01:34 | |
| *** jamesmcarthur has joined #openstack-infra | 01:37 | |
| *** jamesmcarthur has quit IRC | 01:41 | |
| *** ccamacho has quit IRC | 01:57 | |
| *** ccamacho has joined #openstack-infra | 01:58 | |
| *** ccamacho has quit IRC | 01:58 | |
| *** ccamacho has joined #openstack-infra | 01:59 | |
| *** rfarr_ has quit IRC | 02:05 | |
| *** rfarr_ has joined #openstack-infra | 02:06 | |
| *** hwoarang has quit IRC | 02:09 | |
| *** rfarr_ has quit IRC | 02:09 | |
| *** rfarr_ has joined #openstack-infra | 02:10 | |
| *** ccamacho has quit IRC | 02:21 | |
| *** ccamacho has joined #openstack-infra | 02:22 | |
| *** ccamacho has quit IRC | 02:31 | |
| *** ccamacho has joined #openstack-infra | 02:31 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:32 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:35 | |
| *** ccamacho has quit IRC | 02:36 | |
| *** bhavikdbavishi has quit IRC | 02:36 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:36 | |
| *** jamesmcarthur has joined #openstack-infra | 02:37 | |
| *** ccamacho has joined #openstack-infra | 02:38 | |
| *** yamamoto has joined #openstack-infra | 02:41 | |
| *** rfarr_ has quit IRC | 02:42 | |
| *** tdasilva_ has quit IRC | 02:42 | |
| *** hwoarang has joined #openstack-infra | 02:42 | |
| *** rfarr_ has joined #openstack-infra | 02:43 | |
| *** tdasilva_ has joined #openstack-infra | 02:43 | |
| *** tdasilva_ has quit IRC | 02:44 | |
| *** tdasilva_ has joined #openstack-infra | 02:45 | |
| *** rfarr_ has quit IRC | 02:45 | |
| *** rfarr_ has joined #openstack-infra | 02:46 | |
| *** tdasilva_ has quit IRC | 02:48 | |
| *** ccamacho has quit IRC | 02:48 | |
| *** tdasilva_ has joined #openstack-infra | 02:48 | |
| *** rfarr_ has quit IRC | 02:50 | |
| *** rfarr_ has joined #openstack-infra | 02:50 | |
| *** ccamacho has joined #openstack-infra | 02:51 | |
| auristor | ianw: I'm back | 02:54 |
| *** ccamacho has quit IRC | 02:55 | |
| *** ccamacho has joined #openstack-infra | 02:56 | |
| ianw | auristor: hey; i hope we captured enough in logs to be helpful | 02:56 |
| *** yamamoto has quit IRC | 02:57 | |
| auristor | I'm looking at the July 1 14:42 case. openafs "vos" needs a mode to timestamp the verbose output :( | 02:57 |
| *** rfarr_ has quit IRC | 02:59 | |
| *** rfarr_ has joined #openstack-infra | 03:00 | |
| auristor | what host is 104.130.4.160 ? | 03:02 |
| ianw | that's mirror.iad.rax.opendev.org ... there's a dmseg log in the tar file that should have the utc dmesg for that and our gra1 kafs mirror | 03:03 |
| auristor | grep 536870950 afs*.dfw.dafileserver.audit.log | grep -v "CODE 0" | 03:04 |
| auristor | will give you what I'm looking at | 03:04 |
| auristor | The 102 errors are VNOVNODE which means the cache manager was requesting a file that no longer exists. | 03:05 |
| auristor | The 106 errors are the VOFFLINE messages. | 03:05 |
| *** ccamacho has quit IRC | 03:06 | |
| auristor | what the audit log tells me is that the kafs client is only reading from afs02.dfw and isn't failing over to afs01.dfw | 03:07 |
| *** yamamoto has joined #openstack-infra | 03:07 | |
| auristor | which is host 38.108.68.37 ? | 03:07 |
| ianw | hrmm... not sure what's up the reverse dns | 03:08 |
| ianw | that looks vexxhosty ... looking around | 03:09 |
| auristor | 38.108.68.37 received a VOFFLINE (106) from afs02.dfw and retried the request to afs01.dfw which succeeded | 03:10 |
| ianw | https://opendev.org/opendev/system-config/src/branch/master/inventory/openstack.yaml#L509 | 03:10 |
| ianw | mirror01.sjc1.vexxhost.openstack.org | 03:10 |
| auristor | openafs 1.6 ? | 03:11 |
| ianw | yep | 03:11 |
| auristor | :) | 03:11 |
| *** ykarel|away has joined #openstack-infra | 03:11 | |
| auristor | kafs is retrying sometimes. It did at the 10:49:02 when afs02.dfw returned VOFFLINE | 03:13 |
| auristor | interesting. it retried the first vnode which received a VOFFLINE but it didn't retry the requests for the next three vnodes | 03:14 |
| auristor | I won't be able to discuss with dhowells until Wednesday. I know he is racing to finish some other work for the next merge window. | 03:16 |
| ianw | no probs ... do you think we have enough in there for now that we should disable the audit logs? | 03:17 |
| auristor | yes | 03:17 |
| *** hamzy has joined #openstack-infra | 03:18 | |
| auristor | The only other thing you could share would be the FileLog which will provide the timestamps at which the volume callback messages were scheduled for delivery. | 03:18 |
| *** rfarr_ has quit IRC | 03:21 | |
| *** rfarr_ has joined #openstack-infra | 03:21 | |
| ianw | auristor: ok i can grab them too, just a min | 03:22 |
| *** ykarel|away is now known as ykarel | 03:24 | |
| *** diablo_rojo has quit IRC | 03:26 | |
| *** diablo_rojo has joined #openstack-infra | 03:27 | |
| *** ccamacho has joined #openstack-infra | 03:29 | |
| ianw | auristor: at top level in http://people.redhat.com/~iwienand/ ... for completeness I'll regenerate the tarball with them | 03:30 |
| *** diablo_rojo has quit IRC | 03:32 | |
| *** rfarr_ has quit IRC | 03:34 | |
| auristor | to give you some explanation of some of the FileLog messages | 03:34 |
| *** rfarr_ has joined #openstack-infra | 03:34 | |
| *** ccamacho has quit IRC | 03:35 | |
| auristor | An entry such as "CheckHost_r: Probing all interfaces of host 147.135.133.8:7001 failed, code -1" indicates that a rpc was received from the indicated cache manager but rpc the fileserver issued to the client's callback service timed out before it was responded to. | 03:35 |
| auristor | that is indicative of an openafs cache manager experiencing a soft-deadlock | 03:36 |
| *** tdasilva_ has quit IRC | 03:37 | |
| *** psachin has joined #openstack-infra | 03:37 | |
| auristor | "BVCB: volume callback for Host 00007F6671273CC8 (23.253.125.17:7001) failed" might be a client that has shutdown, a NAT port mapping that has timed out, or other communication difficulties. | 03:37 |
| *** tdasilva_ has joined #openstack-infra | 03:38 | |
| auristor | "fssync: breaking all call backs for volume 536870983" is logged when the volserver takes a volume offline for a potential data change such as a re-clone operation. | 03:38 |
| *** rfarr_ has quit IRC | 03:39 | |
| ianw | ok, so that's mostly the points of interest? | 03:40 |
| *** rfarr_ has joined #openstack-infra | 03:40 | |
| auristor | These clients appear to be experiencing soft-deadlocks regularly | 03:41 |
| auristor | 10 23.253.125.17:7001 | 03:41 |
| auristor | 10 91.123.202.253:7001 | 03:41 |
| auristor | 15 37.153.172.158:7001 | 03:41 |
| auristor | 20 158.69.80.87:7001 | 03:41 |
| auristor | 21 104.239.192.5:7001 | 03:41 |
| auristor | 21 149.202.161.140:7001 | 03:41 |
| auristor | 23 38.108.68.37:7001 | 03:41 |
| auristor | 29 104.130.4.160:7001 | 03:41 |
| auristor | 30 147.135.133.8:7001 | 03:41 |
| auristor | 32 38.145.32.16:7001 | 03:41 |
| auristor | 33 199.204.45.64:7001 | 03:41 |
| ianw | heh, i'd say that's just about all of them? :) | 03:41 |
| auristor | thats only the top dozen :) | 03:42 |
| *** whoami-rajat has joined #openstack-infra | 03:42 | |
| *** rfarr_ has quit IRC | 03:43 | |
| *** rfarr_ has joined #openstack-infra | 03:44 | |
| auristor | I would say that the cache is too small on those clients | 03:44 |
| *** jamesmcarthur has quit IRC | 03:49 | |
| *** jamesmcarthur has joined #openstack-infra | 03:50 | |
| *** igordc has quit IRC | 03:51 | |
| *** yamamoto has quit IRC | 03:54 | |
| *** tdasilva_ has quit IRC | 03:55 | |
| *** jamesmcarthur has quit IRC | 03:57 | |
| *** ykarel is now known as ykarel|afk | 03:59 | |
| *** kjackal has joined #openstack-infra | 04:01 | |
| ianw | hrm | 04:01 |
| *** yamamoto has joined #openstack-infra | 04:02 | |
| ianw | 23.253.125.17 is files.openstack.org ... i wonder if we've even upped the cache on that? | 04:02 |
| ianw | not really ... /afs:/var/cache/openafs:10000000 | 04:04 |
| *** ykarel|afk has quit IRC | 04:06 | |
| auristor | its the -stat cache size that most often leads to soft-deadlocks | 04:09 |
| *** rfarr_ has quit IRC | 04:12 | |
| *** rfarr_ has joined #openstack-infra | 04:13 | |
| *** udesale has joined #openstack-infra | 04:14 | |
| *** yamamoto has quit IRC | 04:20 | |
| *** hongbin has quit IRC | 04:21 | |
| *** rfarr_ has quit IRC | 04:22 | |
| *** rfarr_ has joined #openstack-infra | 04:23 | |
| *** ricolin has joined #openstack-infra | 04:25 | |
| *** jamesmcarthur has joined #openstack-infra | 04:29 | |
| *** ykarel|afk has joined #openstack-infra | 04:34 | |
| *** jamesmcarthur has quit IRC | 04:36 | |
| *** ramishra has joined #openstack-infra | 04:36 | |
| *** rfarr_ has quit IRC | 04:36 | |
| *** rfarr_ has joined #openstack-infra | 04:37 | |
| *** yamamoto has joined #openstack-infra | 04:57 | |
| *** ykarel|afk is now known as ykarel | 05:02 | |
| *** dansmith has quit IRC | 05:04 | |
| *** jamesmcarthur has joined #openstack-infra | 05:05 | |
| *** yamamoto has quit IRC | 05:07 | |
| *** rfarr_ has quit IRC | 05:09 | |
| *** jamesmcarthur has quit IRC | 05:10 | |
| *** rfarr_ has joined #openstack-infra | 05:10 | |
| *** rfarr_ has quit IRC | 05:20 | |
| *** rfarr_ has joined #openstack-infra | 05:21 | |
| *** dansmith has joined #openstack-infra | 05:22 | |
| *** ykarel_ has joined #openstack-infra | 05:26 | |
| *** ykarel has quit IRC | 05:28 | |
| *** rfarr_ has quit IRC | 05:30 | |
| *** rfarr_ has joined #openstack-infra | 05:30 | |
| *** rfarr_ has quit IRC | 05:34 | |
| *** rfarr_ has joined #openstack-infra | 05:35 | |
| *** udesale has quit IRC | 05:35 | |
| *** ykarel_ is now known as ykarel|meeting | 05:36 | |
| *** udesale has joined #openstack-infra | 05:36 | |
| *** kjackal has quit IRC | 05:36 | |
| *** jamesmcarthur has joined #openstack-infra | 05:42 | |
| *** rfarr_ has quit IRC | 05:46 | |
| *** rfarr_ has joined #openstack-infra | 05:46 | |
| *** jamesmcarthur has quit IRC | 05:47 | |
| *** tdasilva has joined #openstack-infra | 06:00 | |
| *** rfarr_ has quit IRC | 06:01 | |
| *** rfarr_ has joined #openstack-infra | 06:01 | |
| *** rakhmerov has quit IRC | 06:02 | |
| ianw | #status log restarted afs file servers without auditlog. see notes in https://etherpad.openstack.org/p/opendev-mirror-afs for log recent log bundle | 06:05 |
| openstackstatus | ianw: finished logging | 06:05 |
| *** yamamoto has joined #openstack-infra | 06:09 | |
| *** jtomasek has joined #openstack-infra | 06:15 | |
| *** kjackal has joined #openstack-infra | 06:19 | |
| *** dchen has quit IRC | 06:21 | |
| *** dchen has joined #openstack-infra | 06:21 | |
| *** rfarr_ has quit IRC | 06:25 | |
| *** rfarr_ has joined #openstack-infra | 06:25 | |
| *** rfarr_ has quit IRC | 06:28 | |
| *** rfarr_ has joined #openstack-infra | 06:29 | |
| *** slaweq has joined #openstack-infra | 06:30 | |
| *** pgaxatte has joined #openstack-infra | 06:31 | |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add mirror-update01.opendev.org https://review.opendev.org/668610 | 06:31 |
| *** yamamoto has quit IRC | 06:33 | |
| *** Emine has joined #openstack-infra | 06:36 | |
| *** Emine has quit IRC | 06:37 | |
| *** yamamoto has joined #openstack-infra | 06:37 | |
| *** yamamoto has quit IRC | 06:37 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add mirror-update01.opendev.org server https://review.opendev.org/668612 | 06:38 |
| *** rfarr_ has quit IRC | 06:39 | |
| *** rfarr_ has joined #openstack-infra | 06:40 | |
| *** yamamoto has joined #openstack-infra | 06:41 | |
| *** rfarr_ has quit IRC | 06:45 | |
| *** rfarr_ has joined #openstack-infra | 06:46 | |
| *** raukadah is now known as chandankumar | 06:46 | |
| *** dpawlik has joined #openstack-infra | 06:49 | |
| *** pgaxatte has quit IRC | 06:53 | |
| *** pgaxatte has joined #openstack-infra | 06:54 | |
| *** pgaxatte has quit IRC | 06:54 | |
| *** bhavikdbavishi has quit IRC | 07:01 | |
| *** pgaxatte has joined #openstack-infra | 07:02 | |
| *** pgaxatte has quit IRC | 07:03 | |
| *** pgaxatte has joined #openstack-infra | 07:04 | |
| *** pgaxatte has quit IRC | 07:04 | |
| *** ginopc has joined #openstack-infra | 07:08 | |
| *** xek has joined #openstack-infra | 07:12 | |
| *** pgaxatte has joined #openstack-infra | 07:15 | |
| *** tesseract has joined #openstack-infra | 07:15 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move rsync mirror updates to new opendev.org mirror-update host https://review.opendev.org/667782 | 07:17 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add mirror-update01.opendev.org server https://review.opendev.org/668612 | 07:17 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: update keytab testing https://review.opendev.org/668620 | 07:17 |
| *** rfarr_ has quit IRC | 07:20 | |
| *** rfarr_ has joined #openstack-infra | 07:20 | |
| *** witek has joined #openstack-infra | 07:21 | |
| *** tosky has joined #openstack-infra | 07:22 | |
| *** rfarr_ has quit IRC | 07:22 | |
| *** rfarr_ has joined #openstack-infra | 07:23 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Mailing lists change openstack-dev to openstack-discuss https://review.opendev.org/668435 | 07:23 |
| openstackgerrit | Merged openstack/diskimage-builder master: install gnupg2 by default in debian-minimal https://review.opendev.org/668554 | 07:32 |
| *** yamamoto has quit IRC | 07:33 | |
| *** ricolin_ has joined #openstack-infra | 07:35 | |
| *** dchen has quit IRC | 07:35 | |
| *** dchen has joined #openstack-infra | 07:37 | |
| *** ricolin has quit IRC | 07:37 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add mirror-update01.opendev.org server https://review.opendev.org/668612 | 07:38 |
| *** yamamoto has joined #openstack-infra | 07:38 | |
| *** evrardjp is now known as evrardjp_on_holi | 07:40 | |
| *** evrardjp_on_holi is now known as evrardjp_away | 07:40 | |
| *** ociuhandu has joined #openstack-infra | 07:42 | |
| *** tosky has quit IRC | 07:43 | |
| *** tosky has joined #openstack-infra | 07:43 | |
| *** jamesmcarthur has joined #openstack-infra | 07:44 | |
| *** jamesmcarthur has quit IRC | 07:48 | |
| *** e0ne has joined #openstack-infra | 07:49 | |
| *** ricolin__ has joined #openstack-infra | 07:49 | |
| *** rfarr_ has quit IRC | 07:50 | |
| *** jamesmcarthur has joined #openstack-infra | 07:51 | |
| *** rfarr_ has joined #openstack-infra | 07:51 | |
| *** ricolin_ has quit IRC | 07:51 | |
| *** jpich has joined #openstack-infra | 07:53 | |
| *** rpittau|afk is now known as rpittau | 07:54 | |
| *** jamesmcarthur has quit IRC | 07:55 | |
| *** yamamoto has quit IRC | 07:55 | |
| *** rfarr_ has quit IRC | 07:55 | |
| *** rfarr_ has joined #openstack-infra | 07:56 | |
| *** pkopec has joined #openstack-infra | 07:56 | |
| *** yamamoto has joined #openstack-infra | 07:57 | |
| *** lucasagomes has joined #openstack-infra | 07:59 | |
| *** dchen has quit IRC | 08:00 | |
| *** pcaruana has joined #openstack-infra | 08:01 | |
| *** dchen has joined #openstack-infra | 08:01 | |
| *** ralonsoh has joined #openstack-infra | 08:02 | |
| *** dchen has quit IRC | 08:04 | |
| *** udesale has quit IRC | 08:05 | |
| *** dchen has joined #openstack-infra | 08:05 | |
| *** ykarel_ has joined #openstack-infra | 08:05 | |
| *** udesale has joined #openstack-infra | 08:06 | |
| *** ykarel|meeting has quit IRC | 08:07 | |
| *** dchen has quit IRC | 08:09 | |
| *** yamamoto has quit IRC | 08:09 | |
| *** dchen has joined #openstack-infra | 08:10 | |
| *** trident has quit IRC | 08:10 | |
| *** trident has joined #openstack-infra | 08:13 | |
| *** dchen has quit IRC | 08:15 | |
| *** yamamoto has joined #openstack-infra | 08:16 | |
| *** dchen has joined #openstack-infra | 08:16 | |
| *** dchen has quit IRC | 08:17 | |
| *** jamesmcarthur has joined #openstack-infra | 08:22 | |
| *** tkajinam has quit IRC | 08:26 | |
| *** jamesmcarthur has quit IRC | 08:28 | |
| *** rfarr_ has quit IRC | 08:28 | |
| *** rfarr_ has joined #openstack-infra | 08:29 | |
| *** ykarel_ has quit IRC | 08:32 | |
| *** rfarr_ has quit IRC | 08:35 | |
| *** rfarr_ has joined #openstack-infra | 08:35 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 08:36 | |
| *** ykarel_ has joined #openstack-infra | 08:37 | |
| *** ykarel_ is now known as ykarel|lunc | 08:37 | |
| *** ykarel|lunc is now known as ykarel|lunch | 08:38 | |
| *** rfarr_ has quit IRC | 08:41 | |
| *** rfarr_ has joined #openstack-infra | 08:42 | |
| *** rfarr_ has quit IRC | 08:45 | |
| *** rfarr_ has joined #openstack-infra | 08:45 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:46 | |
| *** rfarr_ has quit IRC | 08:48 | |
| *** rfarr_ has joined #openstack-infra | 08:48 | |
| *** ykarel|lunch is now known as ykarel | 08:54 | |
| *** jamesmcarthur has joined #openstack-infra | 09:00 | |
| *** yamamoto has quit IRC | 09:01 | |
| *** ricolin_ has joined #openstack-infra | 09:04 | |
| *** rfarr_ has quit IRC | 09:06 | |
| *** rfarr_ has joined #openstack-infra | 09:06 | |
| *** ricolin__ has quit IRC | 09:07 | |
| *** Lucas_Gray has joined #openstack-infra | 09:11 | |
| *** rfarr_ has quit IRC | 09:12 | |
| *** rfarr_ has joined #openstack-infra | 09:13 | |
| *** yamamoto has joined #openstack-infra | 09:13 | |
| *** jaosorior has joined #openstack-infra | 09:13 | |
| *** priteau has joined #openstack-infra | 09:17 | |
| *** tdasilva has quit IRC | 09:21 | |
| *** yamamoto has quit IRC | 09:22 | |
| *** rfarr_ has quit IRC | 09:23 | |
| *** rfarr_ has joined #openstack-infra | 09:23 | |
| *** rfarr_ has quit IRC | 09:33 | |
| *** rfarr_ has joined #openstack-infra | 09:33 | |
| *** ricolin_ is now known as ricolin | 09:33 | |
| *** bhavikdbavishi has quit IRC | 09:34 | |
| *** udesale has quit IRC | 09:35 | |
| *** udesale has joined #openstack-infra | 09:36 | |
| *** rfarr_ has quit IRC | 09:41 | |
| *** rfarr_ has joined #openstack-infra | 09:41 | |
| *** whoami-rajat has quit IRC | 09:42 | |
| *** rfarr_ has quit IRC | 09:46 | |
| *** rfarr_ has joined #openstack-infra | 09:47 | |
| *** ricolin_ has joined #openstack-infra | 09:47 | |
| *** yamamoto has joined #openstack-infra | 09:47 | |
| jrosser | E:Failed to fetch http://mirror.ord.rax.opendev.org/debian-security/dists/stretch/main/binary-amd64/Packages.gz Hash Sum mismatch | 09:49 |
| *** ricolin has quit IRC | 09:49 | |
| *** rfarr_ has quit IRC | 09:52 | |
| *** yamamoto has quit IRC | 09:52 | |
| *** rfarr_ has joined #openstack-infra | 09:52 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Mailing lists change openstack-dev to openstack-discuss https://review.opendev.org/668435 | 09:53 |
| *** yamamoto has joined #openstack-infra | 09:55 | |
| *** priteau has quit IRC | 09:57 | |
| *** ykarel is now known as ykarel|meeting | 09:58 | |
| *** yamamoto has quit IRC | 09:59 | |
| *** roman_g has quit IRC | 10:02 | |
| *** roman_g has joined #openstack-infra | 10:03 | |
| *** jamesmcarthur has quit IRC | 10:04 | |
| *** yamamoto has joined #openstack-infra | 10:04 | |
| *** yamamoto has quit IRC | 10:04 | |
| *** roman_g_ has joined #openstack-infra | 10:05 | |
| *** priteau has joined #openstack-infra | 10:06 | |
| *** psachin has quit IRC | 10:08 | |
| *** roman_g has quit IRC | 10:08 | |
| sshnaidm|ruck | what does mean "Triggered by: https://review.opendev.org/gitweb?p=openstack/tripleo-ci.git;a=commitdiff;h=None" ? Isn't it from patch? | 10:09 |
| *** roman_g_ is now known as roman_g | 10:10 | |
| *** rfarr_ has quit IRC | 10:16 | |
| *** rfarr_ has joined #openstack-infra | 10:16 | |
| ianw | jrosser: can you give a timestamp? we put that into rotation yesterday, should be running openafs 1.8.3. we haven't been seeing issues afaik with the other server running that (dfw.rax) ... | 10:26 |
| *** tdasilva has joined #openstack-infra | 10:27 | |
| jrosser | We just had a bunch of openstack-ansible jobs fail I think all the same | 10:27 |
| * jrosser just in the server room, 20mins or so and I can find a log | 10:28 | |
| *** rfarr_ has quit IRC | 10:28 | |
| *** rfarr_ has joined #openstack-infra | 10:29 | |
| *** yamamoto has joined #openstack-infra | 10:30 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:31 | |
| ianw | jrosser: ok, we'll want to correlate it i think to any possible release of a volume or logs on the host. we can also check via logstash. i gotta disappear now, but will be back for meeting and check on things tomorrow too | 10:32 |
| *** ykarel|meeting is now known as ykarel | 10:35 | |
| jrosser | ianw: http://logs.openstack.org/87/668287/1/gate/openstack-ansible-deploy-aio_metal-debian-stable/d3a3b3d/job-output.txt.gz#_2019-07-02_08_35_08_621625 | 10:36 |
| jrosser | http://logs.openstack.org/18/665218/1/gate/openstack-ansible-deploy-aio_metal-debian-stable/bf8ca5c/job-output.txt.gz#_2019-07-02_08_35_47_265743 | 10:37 |
| *** yamamoto has quit IRC | 10:37 | |
| jrosser | http://logs.openstack.org/64/666564/2/gate/openstack-ansible-functional-debian-stable/37baf18/job-output.txt.gz#_2019-07-02_08_39_18_677330 | 10:38 |
| jrosser | theres more but hopefully thats enough to start with | 10:38 |
| *** bhavikdbavishi has quit IRC | 10:40 | |
| *** jpich has quit IRC | 10:43 | |
| *** rfarr_ has quit IRC | 10:43 | |
| *** rfarr_ has joined #openstack-infra | 10:43 | |
| *** kjackal has quit IRC | 10:44 | |
| openstackgerrit | Merged opendev/system-config master: do backports for Debian 'buster' https://review.opendev.org/668157 | 10:51 |
| *** ginopc has quit IRC | 10:52 | |
| *** udesale has quit IRC | 10:54 | |
| *** ginopc has joined #openstack-infra | 10:59 | |
| *** rfarr_ has quit IRC | 10:59 | |
| *** rfarr_ has joined #openstack-infra | 11:00 | |
| *** udesale has joined #openstack-infra | 11:00 | |
| AJaeger | jrosser: I just downloaed the file from your first change and it has the correct hash sum, so might have been a race that is fixed now. Still, I think ianw will want to check log files. | 11:02 |
| *** kjackal has joined #openstack-infra | 11:07 | |
| jrosser | AJaeger: ok, i'll recheck those jobs | 11:07 |
| *** ykarel is now known as ykarel|afk | 11:09 | |
| *** rfarr_ has quit IRC | 11:13 | |
| *** udesale has quit IRC | 11:13 | |
| *** rfarr_ has joined #openstack-infra | 11:14 | |
| *** ykarel|afk has quit IRC | 11:16 | |
| *** yamamoto has joined #openstack-infra | 11:30 | |
| *** rfarr_ has quit IRC | 11:31 | |
| *** kjackal has quit IRC | 11:31 | |
| *** rfarr_ has joined #openstack-infra | 11:32 | |
| *** rfarr_ has quit IRC | 11:36 | |
| *** rfarr_ has joined #openstack-infra | 11:36 | |
| *** ykarel has joined #openstack-infra | 11:37 | |
| *** tdasilva has quit IRC | 11:38 | |
| *** rh-jelabarre has joined #openstack-infra | 11:39 | |
| *** psachin has joined #openstack-infra | 11:39 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Spec: Add a Kubernetes Operator for Zuul https://review.opendev.org/659180 | 11:40 |
| *** priteau has quit IRC | 11:41 | |
| *** jaosorior has quit IRC | 11:42 | |
| *** kjackal has joined #openstack-infra | 11:43 | |
| openstackgerrit | Graham Hayes proposed zuul/nodepool master: Implement an Azure driver https://review.opendev.org/554432 | 11:44 |
| *** ykarel is now known as ykarel|meeting | 11:52 | |
| *** jamesmcarthur has joined #openstack-infra | 12:01 | |
| *** pgaxatte has quit IRC | 12:04 | |
| *** jamesmcarthur has quit IRC | 12:05 | |
| *** rfarr_ has quit IRC | 12:06 | |
| *** pgaxatte has joined #openstack-infra | 12:06 | |
| *** rfarr_ has joined #openstack-infra | 12:06 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Spec: Add a Kubernetes Operator for Zuul https://review.opendev.org/659180 | 12:16 |
| *** bhavikdbavishi has joined #openstack-infra | 12:18 | |
| *** electrofelix has joined #openstack-infra | 12:22 | |
| *** jcoufal has joined #openstack-infra | 12:28 | |
| *** pkopec has quit IRC | 12:29 | |
| *** yamamoto has quit IRC | 12:31 | |
| *** jamesmcarthur has joined #openstack-infra | 12:34 | |
| *** tdasilva has joined #openstack-infra | 12:35 | |
| *** rlandy has joined #openstack-infra | 12:35 | |
| *** pkopec has joined #openstack-infra | 12:37 | |
| *** jamesmcarthur has quit IRC | 12:39 | |
| *** jaosorior has joined #openstack-infra | 12:39 | |
| *** jamesmcarthur has joined #openstack-infra | 12:42 | |
| *** eharney has quit IRC | 12:43 | |
| *** udesale has joined #openstack-infra | 12:43 | |
| *** jpich has joined #openstack-infra | 12:45 | |
| *** rfarr__ has joined #openstack-infra | 12:50 | |
| *** sthussey has joined #openstack-infra | 12:53 | |
| *** rkukura has quit IRC | 12:53 | |
| *** rfarr_ has quit IRC | 12:53 | |
| *** aaronsheffield has joined #openstack-infra | 12:54 | |
| *** priteau has joined #openstack-infra | 12:55 | |
| *** ricolin_ is now known as ricolin | 12:56 | |
| *** ekultails has joined #openstack-infra | 12:59 | |
| *** ykarel|meeting is now known as ykarel | 13:01 | |
| *** pkopec has quit IRC | 13:04 | |
| *** jamesmcarthur has quit IRC | 13:05 | |
| *** pkopec has joined #openstack-infra | 13:06 | |
| *** priteau has quit IRC | 13:13 | |
| *** ccamacho has joined #openstack-infra | 13:13 | |
| *** priteau has joined #openstack-infra | 13:13 | |
| *** lseki has joined #openstack-infra | 13:14 | |
| *** mriedem has joined #openstack-infra | 13:14 | |
| *** adriancz has joined #openstack-infra | 13:19 | |
| *** ykarel is now known as ykarel|afk | 13:20 | |
| *** eharney has joined #openstack-infra | 13:26 | |
| *** smarcet has joined #openstack-infra | 13:27 | |
| *** rfarr__ has quit IRC | 13:28 | |
| *** yamamoto has joined #openstack-infra | 13:28 | |
| *** smarcet has left #openstack-infra | 13:28 | |
| *** jamesmcarthur has joined #openstack-infra | 13:32 | |
| *** rfarr has joined #openstack-infra | 13:33 | |
| *** sreejithp has joined #openstack-infra | 13:40 | |
| *** bhavikdbavishi has quit IRC | 13:44 | |
| *** liuyulong has joined #openstack-infra | 13:45 | |
| *** yamamoto has quit IRC | 13:47 | |
| *** iurygregory_ has joined #openstack-infra | 13:49 | |
| *** Jeffrey4l_ has joined #openstack-infra | 13:50 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Update documentation for k8s in-cluster config https://review.opendev.org/668679 | 13:51 |
| *** iurygregory has quit IRC | 13:51 | |
| *** Jeffrey4l has quit IRC | 13:54 | |
| mudpuppy | sup | 14:14 |
| *** Fidde has joined #openstack-infra | 14:15 | |
| openstackgerrit | Brian Haley proposed opendev/irc-meetings master: Create a meeting for Networking OVN project https://review.opendev.org/668013 | 14:17 |
| openstackgerrit | Merged openstack/project-config master: Load extra Zuul config from zuul-jobs https://review.opendev.org/668563 | 14:18 |
| openstackgerrit | Pete Vander Giessen proposed openstack/project-config master: Add microstack project. https://review.opendev.org/668681 | 14:18 |
| sshnaidm|ruck | Maybe problem with iax docker mirror: TTPError: 502 Server Error: Bad Gateway for url: http://mirror.iad.rax.opendev.org:8082/cloudflare/registry-v2/docker/registry/v2/blobs/sha256/6a/6 | 14:21 |
| sshnaidm|ruck | http://logs.openstack.org/71/668671/1/check/tripleo-ci-centos-7-scenario004-standalone/30b6866/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-07-02_14_07_10 | 14:21 |
| frickler | cloudflare reported network issues earlier | 14:23 |
| openstackgerrit | Brian Haley proposed opendev/irc-meetings master: Create a meeting for Networking OVN project https://review.opendev.org/668013 | 14:27 |
| openstackgerrit | Merged zuul/nodepool master: Allow nodepool for using in-cluster configs https://review.opendev.org/668172 | 14:32 |
| *** Lucas_Gray has quit IRC | 14:32 | |
| *** Lucas_Gray has joined #openstack-infra | 14:33 | |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Cache dstast_graph on nodepool images https://review.opendev.org/668683 | 14:33 |
| corvus | clarkb, mordred: ^ does that look right? | 14:33 |
| *** redrobot has joined #openstack-infra | 14:33 | |
| mordred | corvus: probably? | 14:34 |
| mordred | corvus: also - you know dag archived dstat | 14:35 |
| mordred | (it'll obviously keep working - but such is life) | 14:35 |
| corvus | mordred: yeah, when there's an alternative, i'm sure we can switch :) | 14:36 |
| *** yamamoto has joined #openstack-infra | 14:39 | |
| *** yamamoto has quit IRC | 14:39 | |
| *** sreejithp has quit IRC | 14:43 | |
| *** pkopec is now known as pkopec|lunch | 14:46 | |
| *** zbr has quit IRC | 14:53 | |
| *** zbr has joined #openstack-infra | 14:54 | |
| *** pgaxatte has quit IRC | 14:57 | |
| *** evrardjp_away has quit IRC | 14:58 | |
| *** evrardjp has joined #openstack-infra | 14:59 | |
| *** mattw4 has joined #openstack-infra | 15:02 | |
| *** Fidde has quit IRC | 15:03 | |
| *** yamamoto has joined #openstack-infra | 15:04 | |
| *** edmondsw_ has joined #openstack-infra | 15:04 | |
| *** gyee has joined #openstack-infra | 15:04 | |
| openstackgerrit | Lon Hohberger proposed openstack/diskimage-builder master: ironic-agent: Use targetcli & system-python-devel on rhel8 https://review.opendev.org/668687 | 15:07 |
| openstackgerrit | Merged openstack/project-config master: Cache dstast_graph on nodepool images https://review.opendev.org/668683 | 15:10 |
| *** Lucas_Gray has quit IRC | 15:10 | |
| openstackgerrit | Lon Hohberger proposed openstack/diskimage-builder master: ironic-agent: Use targetcli & python3-devel on rhel8 https://review.opendev.org/668687 | 15:11 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Add playbooks folder and zuul job that installs k8s https://review.opendev.org/668029 | 15:17 |
| *** bhavikdbavishi has joined #openstack-infra | 15:22 | |
| *** smarcet has joined #openstack-infra | 15:22 | |
| *** liuyulong has quit IRC | 15:22 | |
| *** smarcet has left #openstack-infra | 15:22 | |
| *** e0ne has quit IRC | 15:27 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 15:29 | |
| *** yamamoto has quit IRC | 15:30 | |
| *** bhavikdbavishi has quit IRC | 15:30 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 15:30 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Move project config in-repo https://review.opendev.org/668698 | 15:31 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Remove zuul-jobs config https://review.opendev.org/668699 | 15:32 |
| corvus | hrm, zuul did not pick up that change the way i expected; let me see if we've reloaded the tenant config | 15:35 |
| openstackgerrit | Merged zuul/nodepool master: Update documentation for k8s in-cluster config https://review.opendev.org/668679 | 15:37 |
| *** rkukura has joined #openstack-infra | 15:39 | |
| *** tdasilva has quit IRC | 15:40 | |
| *** tdasilva has joined #openstack-infra | 15:41 | |
| corvus | ah, it did work, it just had an invalid config, and didn't report it because it's not attached to the pipeline yet (we don't have a .* project in the zuul tenant) | 15:45 |
| clarkb | we set a lot of IP addresses in local.conf during devstack jobs | 15:52 |
| clarkb | makes pulling a local.conf from a job to run locally a major pain | 15:52 |
| clarkb | (if anyone is wonderig why this doesn't work for them, remove all the host ip stuff in local.conf) | 15:52 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Zuul-jobs: load extra config in openstack tenant https://review.opendev.org/668703 | 15:53 |
| corvus | clarkb, mordred, AJaeger: ^ i got ahead of myself, can you review that soon? | 15:53 |
| *** ykarel|afk is now known as ykarel | 15:54 | |
| clarkb | I can | 15:54 |
| donnyd | clarkb: How hard would it be to build up a freebsd box for zuul? | 15:55 |
| donnyd | oh and everything should be good to go | 15:55 |
| donnyd | I may swap out the glance backend, but that *should* be transparent | 15:56 |
| clarkb | donnyd: freebsd as the node running zuul or freebsd images for running tests on with zuul? | 15:56 |
| *** yamamoto has joined #openstack-infra | 15:56 | |
| clarkb | I think freebsd as node running zuul is likely pretty straightforward and may just work | 15:56 |
| donnyd | The nfs mount point is seemingly causing some major issues with podman or docker based containers for glance only | 15:56 |
| donnyd | testing code on a freebsd image | 15:57 |
| donnyd | https://review.opendev.org/#/c/668694/1 | 15:57 |
| clarkb | I don't think diskimage-builder can build images for freebsd today. It could be updated to do that likely, possibly requiring a freebsd builder node to host those builds | 15:57 |
| donnyd | Just curious because Tim asked | 15:57 |
| donnyd | So we just need resources and time is what I am hearing | 15:58 |
| clarkb | We can also point zuul/nodepool at preexisting images in a cloud. There are a bunch of reasons why we like to rebuild images daily to update caches and not worry about security updates though so if we do that we'll need to think about thei mplications of it | 15:58 |
| clarkb | donnyd: ya, basically nothing inherent in zuul's design should prevent it. Its more a matter of it hasn't been done by us before so likely lots to learn :) | 15:58 |
| *** ginopc has quit IRC | 16:00 | |
| *** jamesmcarthur has quit IRC | 16:03 | |
| *** kjackal has quit IRC | 16:03 | |
| AJaeger | corvus: +2A on 668703 | 16:05 |
| *** mattw4 has quit IRC | 16:06 | |
| *** yamamoto has quit IRC | 16:06 | |
| corvus | clarkb, donnyd: my order of preference would be: 1) freebsd has something like 'debootstrap' which just happens to magically work on ubuntu and we can use the existing dib methodology [seems unlikely]. 2) we have a freebsd builder which can run something like 'debootstrap' on freebsd and add dib support for that [this ought to work -- they make those release isos *somehow* right?] 3) we dig out | 16:08 |
| corvus | the old nodepool 0.1 code that does snapshot based builds and make a new nodepool builder that uses snapshots instead of dib. [a lot of work (this is mostly a nodepool coding project), but gives us an extremely flexible builder] | 16:08 |
| openstackgerrit | Marios Andreou proposed openstack/diskimage-builder master: WIP: Adds release->rhel8 for ironic-agent pkg-map https://review.opendev.org/668436 | 16:09 |
| corvus | or, i gues 4) we just use upstream images for freebsd and forget about our local caching, etc. | 16:09 |
| *** gyee has quit IRC | 16:12 | |
| *** iurygregory_ has quit IRC | 16:16 | |
| openstackgerrit | Merged openstack/project-config master: Zuul-jobs: load extra config in openstack tenant https://review.opendev.org/668703 | 16:17 |
| *** jamesmcarthur has joined #openstack-infra | 16:17 | |
| *** rkukura_ has joined #openstack-infra | 16:17 | |
| *** rkukura has quit IRC | 16:20 | |
| *** rkukura_ is now known as rkukura | 16:20 | |
| *** rpittau is now known as rpittau|afk | 16:21 | |
| *** gyee has joined #openstack-infra | 16:25 | |
| *** lucasagomes has quit IRC | 16:32 | |
| *** jpich has quit IRC | 16:35 | |
| openstackgerrit | Nate Johnston proposed openstack/project-config master: Mark networking-ovn-tempest-dsvm-ovs-release as voting https://review.opendev.org/668708 | 16:36 |
| clarkb | tempest is running in the test node now | 16:38 |
| donnyd | on mine? | 16:38 |
| clarkb | donnyd: yup | 16:38 |
| clarkb | devstack took 27 minutes without using cached data which is inline with other clouds iirc | 16:38 |
| clarkb | so looking good so far | 16:38 |
| donnyd | Well that is good news | 16:39 |
| clarkb | http://paste.openstack.org/show/753783/ compared to http://logs.openstack.org/38/662538/1/check/tempest-full/fafc02f/controller/logs/devstacklog.txt.gz#_2019-05-31_22_06_26_022 | 16:39 |
| clarkb | it was actually quite a bit slower in the job I grabbed my local.conf from ^ | 16:39 |
| clarkb | now to see how tempest does | 16:40 |
| *** rlandy has quit IRC | 16:41 | |
| clarkb | the trick for reusing a zuul job's local.conf is to comment out error on clone and all the entries with HOST in them | 16:41 |
| clarkb | devstack will autodetect ip addresses if you comment out the HOST entries | 16:41 |
| *** rlandy has joined #openstack-infra | 16:42 | |
| clarkb | this is going to take a bit of time so I am going to get a bike ride in now while waiting. Back in a bit | 16:42 |
| donnyd | Ok cool | 16:42 |
| donnyd | LMK how it works out | 16:42 |
| *** rlandy has quit IRC | 16:43 | |
| *** rlandy has joined #openstack-infra | 16:44 | |
| *** yamamoto has joined #openstack-infra | 16:46 | |
| *** priteau has quit IRC | 16:48 | |
| *** udesale has quit IRC | 16:49 | |
| openstackgerrit | Merged zuul/nodepool master: Remove devstack plugin functional test jobs https://review.opendev.org/667156 | 16:52 |
| openstackgerrit | Brian Haley proposed openstack/project-config master: Mark networking-ovn-tempest-dsvm-ovs-release voting https://review.opendev.org/668714 | 16:56 |
| *** yamamoto has quit IRC | 16:56 | |
| *** ykarel is now known as ykarel|away | 16:58 | |
| openstackgerrit | Merged zuul/nodepool master: Reparent nodepool-functional-openstack-src https://review.opendev.org/667995 | 17:01 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Move in-repo test jobs into zuul-tests.d https://review.opendev.org/668716 | 17:01 |
| *** tesseract has quit IRC | 17:01 | |
| *** jamesmcarthur has quit IRC | 17:02 | |
| *** jamesmcarthur has joined #openstack-infra | 17:02 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Move in-repo test jobs into zuul-tests.d https://review.opendev.org/668716 | 17:04 |
| *** goldyfruit has quit IRC | 17:04 | |
| *** ricolin_ has joined #openstack-infra | 17:05 | |
| *** ykarel|away has quit IRC | 17:05 | |
| *** goldyfruit has joined #openstack-infra | 17:05 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add zuul-jobs-tox-linters https://review.opendev.org/668052 | 17:07 |
| *** ricolin has quit IRC | 17:07 | |
| *** weifan has joined #openstack-infra | 17:14 | |
| *** ricolin_ has quit IRC | 17:14 | |
| *** ociuhandu_ has joined #openstack-infra | 17:14 | |
| *** ociuhandu_ has quit IRC | 17:15 | |
| *** mattw4 has joined #openstack-infra | 17:15 | |
| *** ardaguclu has joined #openstack-infra | 17:15 | |
| *** witek has quit IRC | 17:15 | |
| *** weifan has quit IRC | 17:15 | |
| *** ardaguclu has left #openstack-infra | 17:16 | |
| *** ociuhandu has quit IRC | 17:18 | |
| *** electrofelix has quit IRC | 17:26 | |
| *** hemna has joined #openstack-infra | 17:30 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 17:31 | |
| hemna | hey guys, working on a ci job definition for ceph-iscsi. how to I get zuul to allow me to clone a repo not owned/run by openstack as part of the job? | 17:31 |
| hemna | https://review.opendev.org/#/c/667108/4/.zuul.yaml@167 | 17:32 |
| hemna | I just want devstack to allow cloning of a couple github.com/open-iscsi projects | 17:32 |
| *** ociuhandu has joined #openstack-infra | 17:33 | |
| *** ociuhandu has quit IRC | 17:38 | |
| *** ralonsoh has quit IRC | 17:42 | |
| *** goldyfruit has quit IRC | 17:43 | |
| corvus | hemna: if it's not something that we're gating, then it should be installed from a release rather than from source | 17:44 |
| hemna | corvus: there is no dist package for them | 17:44 |
| hemna | they are installed from their clone directly | 17:44 |
| hemna | that's the only option for tcmu-runner, targetcli-fb and ceph-iscsi projects | 17:45 |
| corvus | hemna: openstack software should only depend on released software | 17:45 |
| hemna | that doesn't help me test at all | 17:46 |
| corvus | hemna: are you planning on releasing something that depends on unreleased software? | 17:46 |
| hemna | the official ceph-iscsi docs say to install from github | 17:46 |
| *** mattmceuen has left #openstack-infra | 17:46 | |
| hemna | http://docs.ceph.com/docs/mimic/rbd/iscsi-target-cli-manual-install/ | 17:47 |
| hemna | that's just how it's done | 17:47 |
| corvus | hemna: we're sort of in the realm of policy here, and previously openstack has established this policy that to be friendly to users and delpoyers, it would depend only on released software, so that our install instructions didn't say "download this source code from github and install it yourself". so i think if you want to expand the scope to cover depending on software without formal releases, | 17:48 |
| corvus | that's probably a good topic for the TC | 17:48 |
| clarkb | donnyd: so tempest ran to completion but with 24 failed jobs. Some of the problems are related to ambiguous network specifications booting instances. There seems to have been some OOMing too :/ | 17:48 |
| hemna | well, if I can't get a gate job for this, then the ceph-iscsi driver is dead for cinder | 17:48 |
| clarkb | donnyd: I'll specify the IP addrs that devstack should auto figure out in case taht is necessary for the networking pieces and the OOMing might be fallout of tempest going sideways | 17:48 |
| donnyd | OOMing? | 17:49 |
| donnyd | Out of Memory? | 17:49 |
| AJaeger | config-core, could you review corvus' change https://review.opendev.org/668699 to move jobs to zuul-jobs, please? | 17:49 |
| clarkb | donnyd: ya | 17:49 |
| corvus | hemna: well, don't let it die here -- i'm just trying to point you to the right venue for the conversation | 17:49 |
| clarkb | donnyd: I don't think the cloud is to blame for that as the amount of memory on the host matches our other instances | 17:50 |
| hemna | so there is no way to even test this then | 17:50 |
| corvus | hemna: maybe an outcome is one of the distros sees this as a problem they can fix and packages it up. | 17:50 |
| AJaeger | config-core, and also review https://review.opendev.org/#/c/668052 , please | 17:50 |
| hemna | :( | 17:51 |
| donnyd | Hrm.. well the nodes they run on have plenty of the memory | 17:51 |
| clarkb | donnyd: ya I'll rerun and make sure this wasn't fallout of the network changes I made in local.conf | 17:51 |
| corvus | hemna: there are definitely ways to test this. i can tell you several ways to make that job work, but if you pushed up those patches, we'd have to -1 them because they are out of policy. so, please, please, please, just talk to the TC about this, okay? :) | 17:51 |
| hemna | I can't change distros | 17:51 |
| donnyd | when you say "ambiguous network specifications", what do you mean | 17:52 |
| donnyd | also does nested virt need to be enabled? | 17:52 |
| AJaeger | hemna: openSUSE has these three packages, so test using openSUSE-15 | 17:52 |
| *** gyee has quit IRC | 17:52 | |
| clarkb | donnyd: nova was failing to boot instances because a specific network wasn't specified and it was ambiguous according to the nova logs | 17:52 |
| clarkb | donnyd: its the error you get if there are several possible networks that can be used and nova doesn't know which to use (note this is in the nested cloud not your cloud) | 17:53 |
| hemna | how do I change the job definition to force using openSUSE then ? | 17:53 |
| clarkb | donnyd: no we disable nested virt in these jobs as it is unreliable so it is fine to leave that off | 17:53 |
| clarkb | hemna: you can specify a new nodeset. If you look in devstack's zuul config there are examples of this | 17:54 |
| AJaeger | hemna: devstack-single-node-opensuse-150 is in devstack repo defined, keystone has keystone-dsvm-functional-federation-opensuse15, have a look how they do it | 17:55 |
| AJaeger | hemna: there are other jobs defined as well... if you need a review, I'm sure dirk and cmurphy can help | 17:55 |
| donnyd | Is there anything you need me to do? | 17:55 |
| clarkb | donnyd: at this point I don't think so. I'll work on narrowing down the failures if they are reproduceable and let you know what I find | 17:56 |
| hemna | ok, so just specify nodeset: devstack-single-node-opensuse-150 as part of the job? | 17:56 |
| *** jamesmcarthur has quit IRC | 17:56 | |
| donnyd | ok sounds great | 17:56 |
| openstackgerrit | Merged zuul/zuul-jobs master: Move project config in-repo https://review.opendev.org/668698 | 17:57 |
| AJaeger | hemna: that is a job name, check how the job does it | 17:57 |
| AJaeger | bbl | 17:58 |
| *** ijw has joined #openstack-infra | 17:58 | |
| *** weifan has joined #openstack-infra | 18:02 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Move in-repo test jobs into zuul-tests.d https://review.opendev.org/668716 | 18:03 |
| *** tdasilva has quit IRC | 18:03 | |
| *** armax has joined #openstack-infra | 18:04 | |
| openstackgerrit | Merged zuul/zuul master: Filter out unprotected branches from builds if excluded https://review.opendev.org/666664 | 18:05 |
| *** gyee has joined #openstack-infra | 18:06 | |
| *** goldyfruit has joined #openstack-infra | 18:10 | |
| *** ykarel|away has joined #openstack-infra | 18:13 | |
| *** igordc has joined #openstack-infra | 18:13 | |
| hemna | ok well, it looks like the devstack-plugin-ceph isn't supported on opensuse | 18:20 |
| hemna | so that kills it. | 18:20 |
| hemna | :( | 18:20 |
| *** weifan has quit IRC | 18:21 | |
| *** weifan has joined #openstack-infra | 18:22 | |
| *** ykarel|away has quit IRC | 18:23 | |
| donnyd | clarkb: when you are done with your tempest testing please LMK | 18:23 |
| donnyd | I am going to swap from nfs to swift for the backend | 18:23 |
| *** weifan has quit IRC | 18:23 | |
| donnyd | I can upload your image with a quickness, so downtime will be minimal | 18:23 |
| *** weifan has joined #openstack-infra | 18:24 | |
| clarkb | donnyd: will do | 18:24 |
| clarkb | its probably another hour away? Not 100% sure | 18:24 |
| donnyd | No rush. I am still doing a little tuning | 18:25 |
| clarkb | I've discovered that tempest should run as a tempest user (at least it does in the gate) and I don't have a tempest user at the end of my devstack run so sorting that out | 18:25 |
| clarkb | tempest was a lot easier to run a couple years ago :/ | 18:26 |
| *** ardaguclu has joined #openstack-infra | 18:28 | |
| *** ardaguclu has quit IRC | 18:28 | |
| *** ardaguclu has joined #openstack-infra | 18:29 | |
| clarkb | anyone know where the tempest user comes from? I see d-g adds it in the legacy jobs but not fiding it in tempest or devstack v3 roles | 18:31 |
| clarkb | aha there it is | 18:31 |
| clarkb | and we do that because the user running tempest needs to be able to sudo? | 18:32 |
| clarkb | Also another piece of data, I've realized I didn't create a swap file which our normal jobs will do. This may explain the OOMing | 18:34 |
| clarkb | but also does that mean tempest can't run without digging into swap? | 18:35 |
| clarkb | that seems like a pretty bad regresion | 18:35 |
| mordred | clarkb: one would think that devstack would create the tempest user - or that there would be a tempest plugin for devstack that would create the tempest user ... but one would think many things | 18:36 |
| clarkb | mordred: nope devstack does not do it. The devstack jobs do though | 18:37 |
| clarkb | and I agree that seems backwards | 18:37 |
| clarkb | if tempest wants its own user then give it one during devstacking | 18:37 |
| clarkb | fwiw I've decided I don't ened to make a tempest user because the zuul user can do passwordless sudo so just running it as zuul | 18:37 |
| clarkb | I'm also going to be quite sad if I have to create a swapfile just to run devstack successfully | 18:38 |
| clarkb | I think base devstack + tempest should not be swapping | 18:39 |
| clarkb | I guess I can go ahead and preempitvely create that file while tempest is running | 18:39 |
| clarkb | because more interested in performance results than cloud memory use | 18:39 |
| * clarkb does this | 18:39 | |
| *** weifan has quit IRC | 18:40 | |
| *** weifan has joined #openstack-infra | 18:40 | |
| *** weifan has quit IRC | 18:40 | |
| *** weifan has joined #openstack-infra | 18:41 | |
| *** weifan has quit IRC | 18:41 | |
| *** bhavikdbavishi has quit IRC | 18:41 | |
| *** weifan has joined #openstack-infra | 18:41 | |
| *** weifan has quit IRC | 18:42 | |
| *** weifan has joined #openstack-infra | 18:42 | |
| *** weifan has quit IRC | 18:43 | |
| *** weifan has joined #openstack-infra | 18:43 | |
| *** bhavikdbavishi has joined #openstack-infra | 18:43 | |
| *** weifan has quit IRC | 18:44 | |
| clarkb | SKIPPED: nova-network is gone. I wonder if we can delete those tests then? | 18:44 |
| *** weifan has joined #openstack-infra | 18:44 | |
| *** armax has quit IRC | 18:44 | |
| *** weifan has quit IRC | 18:44 | |
| *** weifan has joined #openstack-infra | 18:44 | |
| clarkb | ~15 minutes to our weekly meeting. Join us in #openstack-meeting | 18:45 |
| *** weifan has quit IRC | 18:45 | |
| *** ardaguclu has quit IRC | 18:45 | |
| *** weifan has joined #openstack-infra | 18:45 | |
| *** ardaguclu has joined #openstack-infra | 18:46 | |
| *** weifan has quit IRC | 18:46 | |
| *** weifan has joined #openstack-infra | 18:46 | |
| *** weifan has quit IRC | 18:47 | |
| *** ardaguclu has left #openstack-infra | 18:48 | |
| clarkb | hemna: there are other ways to address it. I think corvus' point was to ensure that going down the road of other methods was not going to surprise the TC | 18:49 |
| clarkb | (or our users) | 18:49 |
| *** weifan has joined #openstack-infra | 18:50 | |
| *** weifan has quit IRC | 18:50 | |
| *** weifan has joined #openstack-infra | 18:51 | |
| openstackgerrit | Nicholas Shobe proposed opendev/bindep master: Add dependency version examples to README https://review.opendev.org/668740 | 18:54 |
| *** weifan has quit IRC | 18:55 | |
| *** smarcet has joined #openstack-infra | 18:56 | |
| clarkb | ya test node is almost 200MB into swap now | 18:56 |
| clarkb | just cross 200MB swap now | 18:57 |
| clarkb | so if anyone wants ot improve zuul throughput: fixing the swapping in tempest/devstack jobs will likely significantly speed up all of those jobs thus freeing up resources for other tests | 18:57 |
| * clarkb tucks that away for next time peopel complain | 18:57 | |
| clarkb | I should write an email about it after our meeting /me makes a note to do this | 18:58 |
| *** weifan has joined #openstack-infra | 19:00 | |
| *** pkopec|lunch has quit IRC | 19:05 | |
| *** xek has quit IRC | 19:08 | |
| clarkb | now up to 360MB swappage | 19:09 |
| donnyd | Should we include swap in the flavor? | 19:16 |
| *** kmarc has quit IRC | 19:16 | |
| clarkb | donnyd: it isn't necessary our jobs will create a swap file if there is no device. (we have to do that in a bunch of clouds) | 19:16 |
| *** kmarc has joined #openstack-infra | 19:16 | |
| clarkb | it should do the right thing if you add swap to the falvor though | 19:16 |
| donnyd | Not sure if that will actually make it perform better though | 19:18 |
| clarkb | ya I don't think we are too worried about swap performance. It is there as a safety valve but should be avoied | 19:18 |
| *** tosky has quit IRC | 19:19 | |
| ianw | clarkb: yep, i think you're on the right track, ord seems not to have 1.8.3 packages. investigating ... | 19:20 |
| yoctozepto | 2019-07-02 19:17 Err:15 http://mirror.ord.rax.opendev.org/ubuntu bionic-updates/universe amd64 Packages | 19:21 |
| yoctozepto | File has unexpected size (1320734 != 1321741). Mirror sync in progress? [IP: 2001:4801:7824:103:be76:4eff:fe10:9c2 80] | 19:21 |
| yoctozepto | Hashes of expected file: | 19:21 |
| yoctozepto | - Filesize:1321741 [weak] | 19:21 |
| yoctozepto | - SHA256:8b11598039ba0cb9f3a9717e3119feaa865f113ed9e51d173935b38763cf1aea | 19:21 |
| yoctozepto | - SHA1:4f2e9579a49005777803b51eb4fb905881ae0db1 [weak] | 19:21 |
| yoctozepto | - MD5Sum:820f33b9f056e1c9e2579db15d997b7d [weak] | 19:21 |
| yoctozepto | Release file created at: Tue, 02 Jul 2019 16:32:17 +0000 | 19:21 |
| yoctozepto | sorry for multiline | 19:21 |
| *** jamesmcarthur has joined #openstack-infra | 19:21 | |
| yoctozepto | seems to be failing still | 19:22 |
| clarkb | yoctozepto: ya that is related to ianw's message just prior to yours | 19:23 |
| clarkb | yoctozepto: it is using the wrong version of openafs unexpectedly | 19:23 |
| yoctozepto | clarkb: thanks, I skimmed the message, lone 'ord' did not catch my eye ;/ | 19:24 |
| *** panda has quit IRC | 19:27 | |
| *** jamesmcarthur has quit IRC | 19:28 | |
| *** panda has joined #openstack-infra | 19:30 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: ensure latest package and reorder install https://review.opendev.org/668745 | 19:33 |
| *** bhavikdbavishi has quit IRC | 19:33 | |
| ianw | yoctozepto / clarkb: ^ i think ord.rax was in an odd pocket where i'd created the server a while ago, but we didn't put it into rotation until just a few days ago. i think a totally new server would have got 1.8.3 packages installed | 19:34 |
| ianw | however, i'll still need to manual reboot after that anyway | 19:34 |
| donnyd | This is just a random bit of info in my testing | 19:36 |
| donnyd | cpu utilization for my controllers when using NFS for glance were at 60(ish) percent | 19:37 |
| donnyd | no real reason.. just was | 19:37 |
| donnyd | just swapped out the backend and it dropped by down to where it should at... at 8% | 19:38 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Spec: Add a Kubernetes Operator for Zuul https://review.opendev.org/659180 | 19:46 |
| *** jamesmcarthur has joined #openstack-infra | 19:47 | |
| ijw | So it seems that networking-vpp's mirror on github is present but no longer synced, and the date looks supiciously like it was the date of conversion from openstack/networking-vpp to x/networking-vpp. Is there something I'm supposed to have done, there? | 19:48 |
| clarkb | ijw: we stopped syncing all x/.* repos to github beacuse they are no logner part of openstack. The idea there being we reduce confusion over what is and isn't openstack by being specific. What projects can do instead of relying on the default openstack/ replication is setup zuul jobs that replicate to arbitrary git remotes including github | 19:50 |
| clarkb | there was an email thread about it on the openstack-discuss list which I can dig up if you give me a couple minutes | 19:50 |
| *** jamesmcarthur has quit IRC | 19:50 | |
| yoctozepto | ianw: should not you leave linux-headers-... with present instead of latest? (understanding-wise because it is pinned anyway) | 19:50 |
| fungi | soon the openstack github org admins will go through and mark all those stale repos as "archived" | 19:50 |
| clarkb | ijw: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005007.html | 19:51 |
| ijw | Yeah, ok - the change doesn't surprise me, but leaving the repos present but orphaned caught someone out | 19:51 |
| clarkb | ianw: re devstack + tempest swapping I'll try to put an email together to openstack-discuss about it after lunch | 19:52 |
| ianw | yoctozepto: my theory is that should pull in the latest kernel updates, which will be applied with the latest openafs package when a host reboots ... theory anyway :) | 19:52 |
| fungi | ijw: the alternative was to delete them all immediately, but then that makes it harder t transfer them to other orgs later | 19:52 |
| clarkb | donnyd: infra-root http://logs.openstack.org/38/662538/1/check/tempest-full/fafc02f/job-output.txt.gz#_2019-05-31_22_56_53_731181 compared to http://paste.openstack.org/show/753791/ on fortnebula. I think we are in the same ballpark especially given that devstack runs quicker on fortnebula than that job ran it | 19:53 |
| *** Fidde has joined #openstack-infra | 19:53 | |
| fungi | seems so | 19:53 |
| corvus | clarkb, mordred: https://review.opendev.org/668699 can merge now (and then the rest of the changes can be self-testing) | 19:53 |
| *** factor__ has joined #openstack-infra | 19:54 | |
| ianw | yoctozepto: actually, no i guess you're right because it's using {{ ansible_kernel }} | 19:54 |
| *** bnemec has quit IRC | 19:54 | |
| ianw | so it will only ever install the version currently running | 19:54 |
| yoctozepto | ianw: exactly | 19:54 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: ensure latest package and reorder install https://review.opendev.org/668745 | 19:55 |
| *** icarusfactor has quit IRC | 19:56 | |
| clarkb | yoctozepto: ianw I think they are equivalent in this case beacuse the version is specified | 19:56 |
| clarkb | +2 either way | 19:56 |
| ianw | yeah, but it might try re-installing every run, i dunno | 19:56 |
| donnyd | Well after some tweaking, it's likely to get even faster | 19:57 |
| yoctozepto | ianw, clarkb: yeah, they are functionally equivalent, though present will not trigger check for update (which is unnecessary) and better reflects the task action | 19:58 |
| *** tdasilva has joined #openstack-infra | 19:58 | |
| donnyd | clarkb: So the test was good then? | 20:00 |
| clarkb | donnyd: ya | 20:03 |
| donnyd | So what is next? | 20:04 |
| clarkb | I'll boot a mirror after lunch and we'll add it to nodepool | 20:05 |
| clarkb | that will build images then wwe can tell zuul to use testnodes there | 20:05 |
| clarkb | donnyd: I dont think you need to do anything at this point but I'll let you know if that changes | 20:06 |
| *** bnemec has joined #openstack-infra | 20:07 | |
| donnyd | ok cool | 20:07 |
| donnyd | Well I will just keep humming away over here then | 20:07 |
| openstackgerrit | Merged openstack/project-config master: Remove zuul-jobs config https://review.opendev.org/668699 | 20:11 |
| donnyd | clarkb: can I reboot your instance while i finish off my last bit of testing? | 20:12 |
| clarkb | yes | 20:13 |
| donnyd | ok thanks | 20:13 |
| ianw | clarkb/corvus : in https://review.opendev.org/#/q/topic:mirror-update-ansible+(status:open+OR+status:merged) i think the dns change can merge, then there's the follow-on to add to inventory. if i could get some eyes, i can merge that today with babysitting | 20:13 |
| ianw | oh and a follow-on for testing slightly longer keytabs, to make sure i'm handling line-breaks ok | 20:14 |
| *** factor__ has quit IRC | 20:14 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add zuul-jobs-tox-linters https://review.opendev.org/668052 | 20:14 |
| *** eharney has quit IRC | 20:15 | |
| *** factor has joined #openstack-infra | 20:16 | |
| *** factor has quit IRC | 20:18 | |
| *** factor has joined #openstack-infra | 20:24 | |
| *** ijw has quit IRC | 20:32 | |
| *** Lucas_Gray has joined #openstack-infra | 20:34 | |
| *** Lucas_Gray has quit IRC | 20:37 | |
| *** ijw has joined #openstack-infra | 20:37 | |
| openstackgerrit | Matt McEuen proposed opendev/irc-meetings master: Adjust Airship meeting time https://review.opendev.org/668755 | 20:40 |
| *** psachin has quit IRC | 20:42 | |
| *** evrardjp has quit IRC | 20:51 | |
| *** Lucas_Gray has joined #openstack-infra | 20:52 | |
| *** evrardjp has joined #openstack-infra | 20:52 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-client: ensure latest package and reorder install https://review.opendev.org/668745 | 20:52 |
| ianw | yoctozepto / clarkb: ^ i really do like our testing finding issues that hadn't thought of :) although i should have put a comment first time around because i remember now why it was done like that | 20:53 |
| *** ekultails has quit IRC | 20:58 | |
| *** pcaruana has quit IRC | 20:58 | |
| *** eharney has joined #openstack-infra | 20:59 | |
| *** Lucas_Gray has quit IRC | 21:03 | |
| *** Lucas_Gray has joined #openstack-infra | 21:04 | |
| *** sshnaidm|afk has quit IRC | 21:06 | |
| *** priteau has joined #openstack-infra | 21:07 | |
| *** priteau has quit IRC | 21:07 | |
| *** sshnaidm has joined #openstack-infra | 21:08 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add fortnebula cloud to cloud launcher https://review.opendev.org/668251 | 21:09 |
| *** sshnaidm is now known as sshnaidm|afk | 21:09 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use qcow2 images on fortnebula https://review.opendev.org/668757 | 21:09 |
| clarkb | infra-root ^ those are two small housekeeping changes for fortnebula cloud | 21:09 |
| clarkb | I'm going to work on booting the mirror now | 21:09 |
| *** jcoufal_ has joined #openstack-infra | 21:12 | |
| clarkb | hrm all-clouds.yaml hasn't updated on bridge.o.o? | 21:13 |
| *** ijw has quit IRC | 21:13 | |
| *** ijw has joined #openstack-infra | 21:14 | |
| *** mattw4 has quit IRC | 21:15 | |
| *** jcoufal has quit IRC | 21:16 | |
| donnyd | clarkb: Do I have 5 minutes? | 21:17 |
| donnyd | just wanted to run one more test... i promise | 21:17 |
| *** mattw4 has joined #openstack-infra | 21:17 | |
| donnyd | If you are ready to pull the trigger it can wait | 21:17 |
| clarkb | donnyd: yup | 21:18 |
| clarkb | I've discovered a flaw in our config of the bastion | 21:18 |
| clarkb | so figuring that out first | 21:18 |
| *** ijw has quit IRC | 21:19 | |
| donnyd | Ok cool | 21:19 |
| clarkb | infra-root the issue with bridge clouds.yaml files is service-bridge.yaml is only ever run during testing | 21:20 |
| clarkb | do we need to add that to run_all.sh or does someone else understand this better than me? | 21:21 |
| clarkb | I wasn't around when we split that stuff up so may have missed some details | 21:21 |
| mordred | clarkb: hrm | 21:22 |
| *** jcoufal_ has quit IRC | 21:22 | |
| *** eharney has quit IRC | 21:22 | |
| *** jamesmcarthur has joined #openstack-infra | 21:22 | |
| mordred | clarkb: I'm pretty sure corvus groks it all - but is also afk atm | 21:22 |
| mordred | clarkb: looking | 21:23 |
| mordred | clarkb: can you say more words about what issue you're observing? | 21:23 |
| clarkb | mordred: I updated the bridge all-clouds.yaml and clouds.yaml files to include the new fortnebula cloud. But those chagnes in opendev/system-config are not applied to bridge.o.o | 21:23 |
| clarkb | mordred: I think they are not applied because service-bridge.yaml which makes those changes is not run in production but is run in testing | 21:24 |
| clarkb | I'm not sure if that is or was an intentional change as part of the base.yaml split up | 21:25 |
| mordred | clarkb: well - ok, yeah - I'm also not sure what the difference between bridge and service-bridge is | 21:25 |
| *** bnemec has quit IRC | 21:25 | |
| *** jamesmcarthur has quit IRC | 21:27 | |
| *** bnemec has joined #openstack-infra | 21:27 | |
| clarkb | if anyone else recalls let me know. We'll need that sorted so that cloud launcher works and configures security groups and keys and stuff | 21:31 |
| clarkb | I guess I can modify the clouds.yaml files manually while we search memory | 21:31 |
| donnyd | well... my power just went out... so it may be a few minutes before things come back up | 21:32 |
| donnyd | my generator is sitting outside waiting to be installed | 21:32 |
| clarkb | oh no | 21:33 |
| clarkb | I'll work on something else for a bit then | 21:34 |
| donnyd | I just can't win | 21:34 |
| donnyd | LOL | 21:34 |
| clarkb | when it rains it pours | 21:34 |
| clarkb | we had some really fun weather here yesterday. A tornado (really weak one) even knocked over some trees in portland | 21:35 |
| donnyd | well... I guess I will find out how things are handled when stuff goes bad | 21:35 |
| clarkb | fungi: http://logs.openstack.org/62/666162/1/check/openstackci-beaker-puppet-4/3e13a54/job-output.txt.gz#_2019-07-02_19_16_53_145830 the error appears to be a lack of zlib dev? I'm gonna update bindep to see if that makes it happy | 21:36 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add zlib dev packages to bindep https://review.opendev.org/668759 | 21:39 |
| donnyd | my ups was 2 minutes 2 short | 21:39 |
| donnyd | LOL | 21:39 |
| openstackgerrit | Clark Boylan proposed opendev/puppet-mediawiki master: Canonicalize clone URLs https://review.opendev.org/666162 | 21:40 |
| clarkb | fungi: ^ lets see if that is happier | 21:40 |
| fungi | clarkb: thanks! i wonder why it stopped working | 21:48 |
| clarkb | fungi: well and its odd that the job doesn't seem to fail on all system-config changes? I don't quite undersatnd that aspect of it yet | 21:48 |
| clarkb | oh wait | 21:50 |
| clarkb | its because they are different repos | 21:50 |
| clarkb | I bet the change was dropping global bindep fallback | 21:51 |
| fungi | oh, indeed... if it had no bindep.txt then the timing is about right | 21:52 |
| clarkb | ya so system-config change likely completely unnecessary | 21:52 |
| fungi | suppose we can just add it in the same puppet-mediawiki change | 21:55 |
| fungi | but i'll have to do that latter... being dragged away again now | 21:55 |
| fungi | (or feel free to push it yourself if you want) | 21:55 |
| ianw | clarkb: i feel like service-bridge should probably run as part of run-all.sh ; it was running as part of base.yaml in https://review.opendev.org/#/c/656871/ ... my feeling it was it was just overlooked | 21:56 |
| openstackgerrit | Clark Boylan proposed openstack/openstack-zuul-jobs master: Make infra puppet beaker jobs parent to legacy-base https://review.opendev.org/668761 | 21:56 |
| clarkb | infra-root ^ that should fix puppet jobs | 21:57 |
| clarkb | fungi: ^ I'm fixing it via zuul job config then allpuppet modules will be happy | 21:57 |
| clarkb | ianw: ok I'll work on that change now then | 21:57 |
| openstackgerrit | Clark Boylan proposed opendev/puppet-mediawiki master: Canonicalize clone URLs https://review.opendev.org/666162 | 22:00 |
| clarkb | ianw: and other infra-roots do you think all of playbooks/bridge should go into playbooks/service-bridge? | 22:03 |
| clarkb | I'm not understanding why they are different | 22:03 |
| clarkb | most of the content of playbooks/bridge was moved into playbooks/service-bridge in the change ianw linked | 22:03 |
| clarkb | I'm happy to propose running them as separate playbooks for now too | 22:03 |
| clarkb | oh I think maybe it was because we want all of the service stuff to happen after base.yaml | 22:04 |
| clarkb | but some bridge stuff has to happen first | 22:04 |
| clarkb | in that case I'll push it up as two separate things | 22:04 |
| donnyd | ok, its back up | 22:05 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Apply service-bridge.yaml in run_all.sh https://review.opendev.org/668762 | 22:06 |
| clarkb | ianw: ^ there | 22:06 |
| ianw | +2, thanks | 22:09 |
| ianw | just need to do morning runs, then i'll come back and make sure ord is fixed first priority | 22:09 |
| clarkb | ok now to write that email about devstack+tempest swapping | 22:14 |
| donnyd | going to dinner... be back later | 22:15 |
| corvus | ianw, clarkb: +3 668762 -- we probably just forgot it, or it's possible that we thought we were going to get to the point where we would actually have a service-bridge job by the time that merged | 22:20 |
| corvus | anyway, easy enough to revert that along with the addition of the service-bridge job when we're ready :) | 22:20 |
| clarkb | corvus: ++ | 22:23 |
| clarkb | ianw: I've identified at least one cause of memory pressure on tempest jobs: cinder-backup consumes 785MB of memory according to dstat | 22:24 |
| clarkb | adding that into my email | 22:24 |
| clarkb | hrm dstat may not look at resident memroy but virtual memory? | 22:32 |
| clarkb | the peakmem tracker shows different numbers | 22:32 |
| clarkb | corvus: can you review https://review.opendev.org/#/c/668761/ too? the -1 from zuul there is due to the mirror problem that ianw is fixing | 22:38 |
| *** jcoufal has joined #openstack-infra | 22:39 | |
| corvus | clarkb: +3 should i recheck? | 22:39 |
| corvus | (i think the answer is yes so i did) | 22:40 |
| corvus | (we can at least roll the dice again while ianw is out) | 22:40 |
| clarkb | ya recheck is fine, thanks | 22:40 |
| *** rfarr_ has joined #openstack-infra | 22:43 | |
| *** rfarr has quit IRC | 22:46 | |
| *** panda has quit IRC | 22:52 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add base integration roles https://review.opendev.org/668061 | 22:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Shard project-pipeline config https://review.opendev.org/668763 | 22:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add base integration roles https://review.opendev.org/668061 | 22:57 |
| *** panda has joined #openstack-infra | 22:59 | |
| *** tkajinam has joined #openstack-infra | 23:02 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Make infra puppet beaker jobs parent to legacy-base https://review.opendev.org/668761 | 23:02 |
| clarkb | fungi's wiki change passes tests now https://review.opendev.org/#/c/666162/3 if anyone else wants to review taht real quick | 23:06 |
| *** slaweq has quit IRC | 23:07 | |
| *** Fidde has quit IRC | 23:11 | |
| *** dchen has joined #openstack-infra | 23:12 | |
| *** aaronsheffield has quit IRC | 23:13 | |
| *** mriedem has quit IRC | 23:20 | |
| *** rfarr_ has quit IRC | 23:22 | |
| *** rfarr_ has joined #openstack-infra | 23:23 | |
| *** mattw4 has quit IRC | 23:23 | |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Remove immutable attributes before cleanup https://review.opendev.org/668766 | 23:26 |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Remove immutable attributes before cleanup https://review.opendev.org/668766 | 23:26 |
| *** rfarr_ has quit IRC | 23:27 | |
| *** rfarr_ has joined #openstack-infra | 23:28 | |
| ianw | building 1.8.3 on ord so this gets fixed up now | 23:30 |
| *** sthussey has quit IRC | 23:32 | |
| *** lseki has quit IRC | 23:33 | |
| *** dchen has quit IRC | 23:37 | |
| *** dchen has joined #openstack-infra | 23:39 | |
| *** dchen has joined #openstack-infra | 23:41 | |
| *** dchen has quit IRC | 23:42 | |
| *** dchen has joined #openstack-infra | 23:44 | |
| ianw | #status log mirror.ord.rax.opendev.org rebooted and running openafs 1.8.3; should fix up periodic "hash mismatch" errors seen in jobs | 23:46 |
| openstackstatus | ianw: finished logging | 23:46 |
| *** dchen has quit IRC | 23:46 | |
| *** dchen has joined #openstack-infra | 23:47 | |
| *** dchen has quit IRC | 23:49 | |
| *** dchen has joined #openstack-infra | 23:51 | |
| *** weifan has quit IRC | 23:53 | |
| *** weifan has joined #openstack-infra | 23:53 | |
| *** weifan has quit IRC | 23:57 | |
| ianw | infra-root: if i could get some eyes on https://review.opendev.org/#/c/668612/ that will allow me to do mirror-update migration; the host is up and ready | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!