| *** markvoelker has quit IRC | 00:00 | |
| clarkb | ok well maybe I should go find dinner while we wait on round two here | 00:01 |
|---|---|---|
| *** tosky has quit IRC | 00:01 | |
| clarkb | I suppose it is somewhat possible that py35 issues are related to the change? Though py36 and py37 (locally) both work | 00:01 |
| clarkb | yup dinner is happening I'll check in after for merged code and if things ahve merged look to restart the scheduler | 00:06 |
| pabelanger | ack | 00:06 |
| *** yamamoto has joined #openstack-infra | 00:13 | |
| *** bobh has quit IRC | 00:14 | |
| *** ijw has joined #openstack-infra | 00:16 | |
| *** yamamoto has quit IRC | 00:22 | |
| *** bobh has joined #openstack-infra | 00:23 | |
| *** bobh has quit IRC | 00:24 | |
| *** rlandy|ruck has quit IRC | 00:24 | |
| *** bobh has joined #openstack-infra | 00:25 | |
| *** bobh_ has joined #openstack-infra | 00:28 | |
| *** bobh has quit IRC | 00:33 | |
| *** mattw4 has quit IRC | 00:37 | |
| *** lseki has quit IRC | 00:37 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] letsencrypt update idea https://review.openstack.org/652801 | 00:37 |
| *** roman_g has quit IRC | 00:39 | |
| *** kgiusti has left #openstack-infra | 00:41 | |
| mnaser | infra-root: I think I've discovered the issue of our strange hands. I think there may be a an ansible repo that has a problem that is resulting in our roles failing to clone | 00:45 |
| mnaser | try this: for i in `seq 20`; do rm -rf ansible-role-python_venv_build; git clone https://git.openstack.org/openstack/ansible-role-python_venv_build; done | 00:45 |
| mnaser | at some point, I get "fatal: The remote end hung up unexpectedly" | 00:45 |
| mnaser | in one of the attempts | 00:45 |
| mnaser | oh this is even weirder | 00:46 |
| mnaser | https://www.irccloud.com/pastebin/VvuzI3oD/ | 00:47 |
| mnaser | it looks like the commits for stable/stein are somehow flipping changing around? | 00:47 |
| pabelanger | mnaser: maybe issue with one of git servers? And since going via load balancer, you are bouncing around | 00:47 |
| ianw | bad node in the load balancer? | 00:47 |
| mnaser | that's my guess | 00:48 |
| pabelanger | mnaser: you could try cloning directory to each, to find which repo is the issue | 00:48 |
| mnaser | oh that's a good idea | 00:48 |
| mnaser | how many git servers are there? | 00:48 |
| clarkb | 8 | 00:48 |
| mnaser | git0[1-8] right? | 00:48 |
| clarkb | and they listen on 8080 and 4443 | 00:48 |
| clarkb | yup | 00:48 |
| pabelanger | http://git01.openstack.org:8080/cgit | 00:48 |
| pabelanger | yah | 00:48 |
| *** yamamoto has joined #openstack-infra | 00:49 | |
| mnaser | hmm interesting it cloned fine from all 8 | 00:49 |
| mnaser | but over http not https cause I got cert warnings | 00:49 |
| mnaser | im going to add them all as origins | 00:50 |
| mnaser | and see what I get | 00:50 |
| pabelanger | clarkb: what is with warnings on: https://review.openstack.org/#/c/652788/ | 00:50 |
| clarkb | thereis a git env var to igbore cert errors | 00:50 |
| pabelanger | Requirements ['docker-image'] not met by build 603f16194acc4cf5b5dec0055348be7f | 00:50 |
| ianw | mnaser: any chance you have timestamps? | 00:50 |
| mnaser | I have failing gate jobs, and it just failed locally for me too on my machine | 00:50 |
| mnaser | ianw: http://logs.openstack.org/14/652314/1/check/openstack-ansible-deploy-aio_metal-centos-7/de04e8b/job-output.txt.gz#_2019-04-15_18_01_19_100792 is one | 00:51 |
| clarkb | pabelanger it means the image build ended uo in limestone and skipped iirc | 00:51 |
| clarkb | pabelanger: when we restart zuul.those will be failures | 00:51 |
| ianw | mnaser: hrm, so i think this lines up -> http://paste.openstack.org/show/749331/ | 00:52 |
| mnaser | yeah we do a lot of clones so it may be hard to tell what/who | 00:53 |
| mnaser | but looping over all 8 gits | 00:53 |
| mnaser | hasn't resulted in any problems | 00:53 |
| pabelanger | clarkb: kk | 00:53 |
| mnaser | for i in `seq 20`; do rm -rf ansible-role-python_venv_build; git clone https://git.openstack.org/openstack/ansible-role-python_venv_build; done <-- this replicates it, let me try that on each server | 00:53 |
| pabelanger | clarkb: should we recheck them to be safe? | 00:54 |
| *** bobh_ has quit IRC | 00:54 | |
| ianw | mnaser: what ip is that coming from? i can filter the logs for that | 00:54 |
| mnaser | ianw: right now im doing them from our office so 173.176.58.69 | 00:55 |
| mnaser | there should be a few failures not long ago hitting the main endpoint | 00:55 |
| mnaser | im just trying 20 attempts on each git0X right now | 00:55 |
| mnaser | wonder if it could be a maxconn thing, are haproxy stats exposed anywhere | 00:55 |
| ianw | mnaser: ok, i see them from a while ago | 00:56 |
| ianw | ahh, i think we do send that into grafana | 00:56 |
| clarkb | pabelanger: I'm not too worried about it it mostly tests that the image builds which we will gate on too | 00:56 |
| pabelanger | k | 00:56 |
| pabelanger | 652787 failed again | 00:56 |
| mnaser | http://grafana.openstack.org/d/MYvSHcSiz/git-load-balancer?orgId=1 this one | 00:56 |
| pabelanger | gearman connection loss | 00:57 |
| pabelanger | rechecked | 00:57 |
| ianw | mnaser: yep, jinx :) | 00:57 |
| mnaser | ok. I just ran 20 'clones' against every individual git server, with no issues | 00:58 |
| ianw | mnaser: ok, seeing requests for git.o.o | 00:58 |
| mnaser | but against the LB, 5/20 resulted in fatal: The remote end hung up unexpectedly | 00:58 |
| *** bobh has joined #openstack-infra | 00:58 | |
| mnaser | lemme try to do this for another repo | 00:58 |
| *** bobh has quit IRC | 00:59 | |
| mnaser | (it always seem to hang on this one) | 00:59 |
| mnaser | ok | 01:00 |
| ianw | mnaser: maybe tcpdump on your end so we could see the outgoing port? is there another way to narrow that down? | 01:00 |
| mnaser | stable/stein is two different things | 01:00 |
| mnaser | on different git servers | 01:00 |
| mnaser | git0{1,2,3,4,7,8} have -- Merge "Ensure venv_wheel_build_enable is evaluated as boolean" into stable/stein -- as HEAD of stable/stein | 01:01 |
| mnaser | git{5,6} have -- Merge "Ensure the build host is the first repo server" into stable/stein -- as HEAD of stable/stein | 01:02 |
| mnaser | http://git01.openstack.org:8080/cgit/openstack/ansible-role-python_venv_build/ vs http://git05.openstack.org:8080/cgit/openstack/ansible-role-python_venv_build/ | 01:02 |
| ianw | i'm not seeing any puppeting errors for 5/6 on bridge.o.o at least | 01:02 |
| ianw | so they are minimally alive | 01:03 |
| ianw | root 10085 10060 0 2018 ? 00:00:00 git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack-infra/project-config/ | 01:06 |
| ianw | i'm guessing that's not working ... | 01:06 |
| mnaser | yeah that can do it | 01:10 |
| mnaser | I guess | 01:10 |
| mnaser | I dunno how we're getting hit by it | 01:10 |
| *** bobh has joined #openstack-infra | 01:11 | |
| ianw | the last replication attempt for this was -> http://paste.openstack.org/show/749332/ | 01:12 |
| ianw | what's the missing change? | 01:13 |
| *** eharney has quit IRC | 01:14 | |
| ianw | 652252,2 | 01:15 |
| mnaser | ianw: yep | 01:17 |
| ianw | [root@git05 ansible-role-python_venv_build.git]# ls refs/changes/10/652610/ | 01:17 |
| ianw | 1 | 01:17 |
| ianw | http://paste.openstack.org/show/749333/ | 01:18 |
| ianw | i dunno what to make of that log | 01:18 |
| ianw | does NOT_ATTEMPTED actually mean it wasn't attempted? | 01:19 |
| mnaser | thats beyond my knowledge :) | 01:20 |
| paladox | Yup. | 01:20 |
| ianw | i think it's a red herring | 01:21 |
| ianw | http://paste.openstack.org/show/749334/ ... it completed ok | 01:25 |
| mnaser | im still seeing some inconsistency but idk | 01:27 |
| *** hwoarang has quit IRC | 01:28 | |
| mnaser | cgit still shows things are different | 01:28 |
| ianw | f88af92dd34e2fe3e10f7af0e106d4e001285345 is in the git05 repo | 01:28 |
| *** hwoarang has joined #openstack-infra | 01:28 | |
| ianw | but refs/heads/stable/stein is different! | 01:28 |
| mnaser | yeeeep | 01:29 |
| ianw | http://paste.openstack.org/show/749335/ | 01:31 |
| ianw | Cancelling replication event [d231dcdd] push cgit@git05.openstack.org:/var/lib/git/openstack/ansible-role-python_venv_build.git ? | 01:31 |
| ianw | oh dear ... | 01:33 |
| clarkb | fungi and dmsimard were debugging replication issues earlier today | 01:34 |
| dmsimard | o/ | 01:34 |
| ianw | http://paste.openstack.org/show/749336/ | 01:34 |
| clarkb | for that we may want to force replication on that repo? | 01:34 |
| ianw | is it possible creating a new repo has wiped out all the pending replications? | 01:35 |
| dmsimard | the issue was that the pattern for github was not matching due to lack of ^ at the beginning of the pattern | 01:35 |
| * dmsimard reads back | 01:35 | |
| ianw | $ cat replication_log.2019-04-15 | grep ff6a8e3f | grep 'Cancelling ' | wc -l | 01:36 |
| ianw | 32389 | 01:36 |
| ianw | if i haven't messed that up ... does that mean we dropped 32,389 replication tasks? | 01:36 |
| mnaser | uhoh | 01:37 |
| mnaser | why is there 32389 replication tasks for ff6a8e3f ? :\ | 01:37 |
| mnaser | also this was replication to the openstack git farm and not github | 01:37 |
| *** jamesmcarthur has joined #openstack-infra | 01:37 | |
| ianw | i think that's just the internal task id | 01:37 |
| dmsimard | what time are those ? do they match roughly 13 hours ago ? | 01:37 |
| ianw | that's utc | 01:38 |
| openstackgerrit | Trinh Nguyen proposed openstack-infra/irc-meetings master: Add Telemetry team meeting https://review.openstack.org/652809 | 01:38 |
| dmsimard | the replication spam was likely while I was fixing the broken github replication | 01:38 |
| ianw | since it's 01:37 UTC, and that happened at about 13:00 ... that's ~ 12-13 hours | 01:38 |
| dmsimard | ok I'm caught up :D | 01:39 |
| dmsimard | now what's wrong ? | 01:39 |
| ianw | dmsimard: well, TLDR afaict atm is that at 2019-04-15 13:07:07,797 it looks like we dropped all pending replication tasks in gerrit | 01:40 |
| ianw | and it looks like the git mirrors are out of sync | 01:40 |
| dmsimard | but they were rescheduled afterwards, no ? | 01:40 |
| dmsimard | ok I'm at an actual keyboard now | 01:42 |
| ianw | hrm, checking queu | 01:42 |
| ianw | there is no replication queue in "show-queue" | 01:44 |
| clarkb | soo maybe we just needed to force a rereplication globally after fixing the config? | 01:44 |
| clarkb | there is a replication ssh command to do that | 01:44 |
| ianw | http://paste.openstack.org/show/749335/ appears to be the lifecycle of how the stein ref got missed on git05 | 01:44 |
| *** bhavikdbavishi has joined #openstack-infra | 01:45 | |
| ianw | clarkb: replication start --all ? | 01:45 |
| dmsimard | there is a pattern | 01:46 |
| clarkb | that sounds right | 01:46 |
| dmsimard | but yeah you can run everything :p | 01:46 |
| ianw | ok; any idea why we had all those replications in the queue that got cancelled? | 01:47 |
| clarkb | it wont be quick but should resolve the discrepancies | 01:47 |
| dmsimard | ianw: It might coincide with a reload of the config file | 01:47 |
| ianw | dmsimard: is it likely you had maybe triggered that? and then a new project creation came along and cancelled everything in flight? | 01:47 |
| dmsimard | the project creation had worked | 01:47 |
| dmsimard | but it had not been replicated to github | 01:48 |
| dmsimard | due to the missing ^ | 01:48 |
| ianw | i don't know why the new project seemed to cancel everything ... http://paste.openstack.org/show/749336/ | 01:48 |
| ianw | dmsimard: is openstack-infra/publications invovled ? | 01:48 |
| ianw | or was, rather | 01:48 |
| dmsimard | not afaik, the new project was openstack/networking-omnipath I believe | 01:49 |
| dmsimard | frickler pointed out that there are errors in the jeepyb logs because it tries to create the github repos for ara but they've been moved | 01:49 |
| dmsimard | I did not have the time to look into this one but it didn't seem fatal | 01:50 |
| dmsimard | I see those as separate problems, though | 01:51 |
| ianw | this looks like omnipath -> http://paste.openstack.org/show/749337/ | 01:52 |
| *** whoami-rajat has joined #openstack-infra | 01:53 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix dynamic loading of trusted layouts https://review.openstack.org/652787 | 01:53 |
| ianw | i feel like what might have happened her is that the reload has triggered a global replication event; and i don't know why but it looks like a new project creation has cancelled all inflight events | 01:53 |
| ianw | so, any objection if i start a replication start --all to try and get us back in sync? | 01:53 |
| clarkb | I think dmsimard updating the replication config may have cancelled the events | 01:54 |
| dmsimard | ianw: +1 on replication start | 01:55 |
| dmsimard | it can't hurt :) | 01:55 |
| clarkb | and ya ++ to rerunning | 01:55 |
| ianw | clarkb: oh, that would make sense. | 01:55 |
| ianw | so takers on why a new project cancels all replication? | 01:56 |
| dmsimard | it wasn't the new project | 01:56 |
| dmsimard | or maybe it was | 01:56 |
| dmsimard | I'm confused | 01:56 |
| ianw | "cancelling replication event" (the error we get) appears to have no google hits | 01:56 |
| clarkb | I think it wasthe replication config that cancelled it not the new project? | 01:57 |
| clarkb | before we only ever changedthe config by restarting gerrit | 01:57 |
| clarkb | which then did a full replication | 01:57 |
| ianw | clarkb: i dunno, i'm going on the event id in http://paste.openstack.org/show/749336/ | 01:57 |
| clarkb | but now we can updatethe config with auto reload | 01:58 |
| clarkb | ianw what is the timestamp on review_site/etc/replication.config? | 01:58 |
| clarkb | we autoreload replication config on changes to that file now. if the timestamp on the file matches the events that may explain it | 01:59 |
| ianw | $ ls -l replication.config | 01:59 |
| ianw | -r--r--r-- 1 root root 3736 Apr 15 17:28 replication.config | 01:59 |
| *** apetrich has quit IRC | 01:59 | |
| dmsimard | ianw: probably puppet once I removed the emergency file | 02:00 |
| ianw | 37435 tasks after the "replication start --all" | 02:00 |
| dmsimard | maybe we can see matching logs around 17:28 | 02:00 |
| ianw | which is the region of the ~32,000 that got cancelled | 02:00 |
| *** bobh has quit IRC | 02:01 | |
| ianw | yah, so the question for me now is what may jump in and kill this replication, leaving us again inconsistent | 02:01 |
| ianw | something touching replication.config? | 02:01 |
| dmsimard | can we reproduce it ? | 02:02 |
| clarkb | that is my hunch since we just updated gerrit to autoreload the replication config | 02:02 |
| dmsimard | if we can I would file a bug -- things should be rescheduled appropriately | 02:02 |
| ianw | hrm, yeah, so maybe that points at the replication *not* being rescheduled ... i can not see that it was | 02:03 |
| ianw | (i think that's the same thing dmsimard just said :) | 02:03 |
| *** ijw has quit IRC | 02:04 | |
| dmsimard | out of curiosity, was it a conscious decision to have gerrit push to gitea ? gitea has a built-in mirror feature but I don't know what kind of latency we'd be looking at | 02:04 |
| ianw | would touching replication.config now replicate it? | 02:04 |
| clarkb | dmsimard: Im not sure what we would get turning ut the other way around | 02:07 |
| clarkb | this way we get gerrit events and very quick updates | 02:07 |
| dmsimard | yes, this is probably me being curious about the mirror feature more than anything else :p | 02:08 |
| *** jamesmcarthur has quit IRC | 02:08 | |
| openstackgerrit | Merged openstack-infra/zuul master: Config errors should not affect config-projects https://review.openstack.org/652788 | 02:10 |
| *** hongbin has joined #openstack-infra | 02:12 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:13 | |
| *** bhavikdbavishi has quit IRC | 02:14 | |
| ianw | i guess we can just put this down to weirdness around playing with replication at this point, i'll stamp a log in case we have more issues | 02:16 |
| ianw | #status log started re-replication of all projects on review.openstack.org to bring everything in sync due to some missing references on git servers; likely related to one-off replication reconfiguration previously | 02:17 |
| openstackstatus | ianw: finished logging | 02:17 |
| *** bhavikdbavishi1 has quit IRC | 02:18 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:20 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 02:24 | |
| clarkb | puppet should update zuul01 to the change that just merged above in a few minutes. I'll be restarting the zuul scheduler at that point | 02:24 |
| *** bhavikdbavishi has quit IRC | 02:24 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:24 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add release note for broken trusted config loading fix https://review.openstack.org/652793 | 02:29 |
| *** nicolasbock has quit IRC | 02:29 | |
| clarkb | infra-root zuul==3.7.2.dev85 # git sha 0bb220c is installed on zuul01 now which is the change prior to ^ | 02:29 |
| clarkb | since I don't need the release note for functionality I am going to save queues, stop zuul scheduler then start zuul scheduler and restore queues now | 02:30 |
| clarkb | hrm there are a couple tag and release things queued | 02:31 |
| pabelanger | yah, maybe double check with release team | 02:31 |
| clarkb | I guess I should wait for those to complete | 02:31 |
| clarkb | ya I pinged them and let them know. I'll wait for the current queued things to finish | 02:32 |
| *** bobh has joined #openstack-infra | 02:32 | |
| clarkb | there are two changes in the gate and ~2 in release/tag/release-post | 02:33 |
| clarkb | probably about 15 minutes away from being able to restart the scheduler | 02:33 |
| pabelanger | wfm | 02:34 |
| *** bobh has quit IRC | 02:37 | |
| *** hwoarang has quit IRC | 02:37 | |
| *** jamesmcarthur has joined #openstack-infra | 02:39 | |
| *** hwoarang has joined #openstack-infra | 02:39 | |
| *** ricolin has joined #openstack-infra | 02:45 | |
| *** jamesmcarthur has quit IRC | 02:46 | |
| *** weshay_pto is now known as weshay | 02:49 | |
| clarkb | ok release jobs are done | 02:55 |
| clarkb | proceeding with zuul scheduler restart now | 02:55 |
| clarkb | #status log Restarted zuul-scheduler and zuul-web on commit 0bb220c | 03:02 |
| openstackstatus | clarkb: finished logging | 03:02 |
| clarkb | pabelanger: ^ its done | 03:02 |
| *** jamesmcarthur has joined #openstack-infra | 03:02 | |
| clarkb | I think first thing tomorrow we cut a release | 03:02 |
| pabelanger | clarkb: great | 03:04 |
| clarkb | and with that I'm going to call it a day | 03:09 |
| pabelanger | clarkb: thanks! | 03:10 |
| pabelanger | I am too | 03:10 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] letsencrypt update idea https://review.openstack.org/652801 | 03:19 |
| *** ramishra has joined #openstack-infra | 03:20 | |
| *** gregoryo has joined #openstack-infra | 03:20 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] letsencrypt update idea https://review.openstack.org/652801 | 03:30 |
| *** jamesmcarthur has quit IRC | 03:39 | |
| *** jamesmcarthur has joined #openstack-infra | 03:40 | |
| *** diablo_rojo has quit IRC | 03:41 | |
| *** jamesmcarthur has quit IRC | 03:45 | |
| *** _erlon_ has quit IRC | 04:05 | |
| *** imacdonn has quit IRC | 04:06 | |
| *** imacdonn has joined #openstack-infra | 04:07 | |
| *** jamesmcarthur has joined #openstack-infra | 04:11 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] letsencrypt update idea https://review.openstack.org/652801 | 04:12 |
| *** ykarel|away has joined #openstack-infra | 04:16 | |
| *** jamesmcarthur has quit IRC | 04:18 | |
| *** hwoarang has quit IRC | 04:21 | |
| *** ykarel_ has joined #openstack-infra | 04:22 | |
| *** zhurong has quit IRC | 04:23 | |
| *** ykarel|away has quit IRC | 04:24 | |
| *** hwoarang has joined #openstack-infra | 04:26 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix for orphaned DELETED nodes https://review.openstack.org/652729 | 04:35 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] letsencrypt update idea https://review.openstack.org/652801 | 04:38 |
| *** hongbin has quit IRC | 04:40 | |
| *** bobh has joined #openstack-infra | 04:47 | |
| *** bobh has quit IRC | 04:51 | |
| *** e0ne has joined #openstack-infra | 04:53 | |
| *** auristor has quit IRC | 04:56 | |
| *** e0ne has quit IRC | 04:56 | |
| *** zhurong has joined #openstack-infra | 04:57 | |
| *** auristor has joined #openstack-infra | 05:03 | |
| *** gyee has quit IRC | 05:08 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] letsencrypt update idea https://review.openstack.org/652801 | 05:11 |
| *** auristor has quit IRC | 05:14 | |
| *** benj_ has quit IRC | 05:15 | |
| *** auristor has joined #openstack-infra | 05:18 | |
| *** e0ne has joined #openstack-infra | 05:21 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support fail-fast in project pipelines https://review.openstack.org/652764 | 05:22 |
| *** jbadiapa has quit IRC | 05:25 | |
| *** e0ne has quit IRC | 05:25 | |
| *** benj_ has joined #openstack-infra | 05:26 | |
| *** kjackal has joined #openstack-infra | 05:29 | |
| *** ykarel_ is now known as ykarel | 05:31 | |
| *** quiquell|off is now known as quiquell|rover | 05:49 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support fail-fast in project pipelines https://review.openstack.org/652764 | 05:59 |
| *** lpetrut has joined #openstack-infra | 06:02 | |
| *** kopecmartin|off is now known as kopecmartin | 06:07 | |
| *** lpetrut has quit IRC | 06:16 | |
| *** lpetrut has joined #openstack-infra | 06:16 | |
| *** kjackal has quit IRC | 06:23 | |
| *** pcaruana has joined #openstack-infra | 06:26 | |
| *** nickv1985 has quit IRC | 06:27 | |
| *** masayukig has quit IRC | 06:27 | |
| *** rpittau|afk has quit IRC | 06:28 | |
| *** kmalloc has quit IRC | 06:28 | |
| *** vdrok has quit IRC | 06:28 | |
| *** serverascode has quit IRC | 06:29 | |
| *** TheJulia has quit IRC | 06:29 | |
| *** kmalloc has joined #openstack-infra | 06:30 | |
| *** hogepodge has quit IRC | 06:32 | |
| *** serverascode has joined #openstack-infra | 06:32 | |
| *** johnsom has quit IRC | 06:32 | |
| *** serverascode has quit IRC | 06:38 | |
| *** kmalloc has quit IRC | 06:40 | |
| *** TheJulia has joined #openstack-infra | 06:43 | |
| *** markvoelker has joined #openstack-infra | 06:43 | |
| *** dangtrinhnt has quit IRC | 06:44 | |
| *** benj_ has quit IRC | 06:44 | |
| *** benj_ has joined #openstack-infra | 06:45 | |
| *** TheJulia has quit IRC | 06:47 | |
| *** slaweq has joined #openstack-infra | 06:49 | |
| *** TheJulia has joined #openstack-infra | 06:52 | |
| *** pgaxatte has joined #openstack-infra | 06:52 | |
| *** nickv1985 has joined #openstack-infra | 06:52 | |
| *** kmalloc has joined #openstack-infra | 06:53 | |
| *** rpittau|afk has joined #openstack-infra | 06:53 | |
| *** johnsom has joined #openstack-infra | 06:55 | |
| *** apetrich has joined #openstack-infra | 06:55 | |
| *** masayukig has joined #openstack-infra | 06:55 | |
| *** quiquell|rover is now known as quique|rover|brb | 06:55 | |
| *** hogepodge has joined #openstack-infra | 06:56 | |
| *** gfidente has joined #openstack-infra | 06:56 | |
| *** vdrok has joined #openstack-infra | 06:58 | |
| *** ginopc has joined #openstack-infra | 07:01 | |
| *** serverascode has joined #openstack-infra | 07:02 | |
| *** bobh has joined #openstack-infra | 07:04 | |
| *** e0ne has joined #openstack-infra | 07:06 | |
| *** rpittau|afk is now known as rpittau | 07:06 | |
| *** florianf has joined #openstack-infra | 07:07 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:09 | |
| *** bobh has quit IRC | 07:09 | |
| *** e0ne has quit IRC | 07:09 | |
| *** e0ne has joined #openstack-infra | 07:16 | |
| *** rcernin has quit IRC | 07:20 | |
| *** tosky has joined #openstack-infra | 07:23 | |
| *** bhavikdbavishi has quit IRC | 07:28 | |
| *** dkushwaha has quit IRC | 07:28 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:29 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Ensure correct state in MQTT connection https://review.openstack.org/652932 | 07:29 |
| *** jbadiapa has joined #openstack-infra | 07:31 | |
| *** ykarel is now known as ykarel|lunch | 07:33 | |
| *** bobh has joined #openstack-infra | 07:37 | |
| *** bobh has quit IRC | 07:41 | |
| *** yamamoto has quit IRC | 07:41 | |
| *** helenafm has joined #openstack-infra | 07:42 | |
| *** ccamacho has joined #openstack-infra | 07:48 | |
| *** yamamoto has joined #openstack-infra | 07:48 | |
| *** iurygregory has joined #openstack-infra | 07:52 | |
| *** e0ne has quit IRC | 07:52 | |
| *** roman_g has joined #openstack-infra | 07:58 | |
| *** e0ne has joined #openstack-infra | 07:58 | |
| *** e0ne has quit IRC | 07:59 | |
| *** harlowja has quit IRC | 08:01 | |
| *** jpich has joined #openstack-infra | 08:03 | |
| *** johnsom has quit IRC | 08:04 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] letsencrypt update idea https://review.openstack.org/652801 | 08:04 |
| *** johnsom has joined #openstack-infra | 08:05 | |
| *** masayukig_ has joined #openstack-infra | 08:05 | |
| *** rpittau_ has joined #openstack-infra | 08:05 | |
| *** masayukig has quit IRC | 08:05 | |
| *** masayukig_ is now known as masayukig | 08:05 | |
| *** rpittau_ is now known as rpittua | 08:05 | |
| *** rpittua is now known as rpittau_ | 08:05 | |
| *** rpittau has quit IRC | 08:06 | |
| *** rpittau_ is now known as rpittau | 08:06 | |
| *** kjackal has joined #openstack-infra | 08:06 | |
| *** florianf has quit IRC | 08:08 | |
| *** priteau has joined #openstack-infra | 08:09 | |
| *** e0ne has joined #openstack-infra | 08:10 | |
| openstackgerrit | M V P Nitesh proposed openstack/diskimage-builder master: Adding 'oel' as a new dib element https://review.openstack.org/625501 | 08:13 |
| *** harlowja has joined #openstack-infra | 08:15 | |
| *** tkajinam has quit IRC | 08:24 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] letsencrypt update idea https://review.openstack.org/652801 | 08:25 |
| *** Lucas_Gray has joined #openstack-infra | 08:26 | |
| *** janki has joined #openstack-infra | 08:28 | |
| *** ykarel|lunch is now known as ykarel | 08:30 | |
| *** lucasagomes has joined #openstack-infra | 08:34 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/reviewstats master: Add Kuryr to projects https://review.openstack.org/652943 | 08:35 |
| *** adriancz has joined #openstack-infra | 08:36 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 08:37 | |
| *** bhavikdbavishi has quit IRC | 08:38 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 08:38 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Add Telemetry team meeting https://review.openstack.org/652809 | 08:39 |
| openstackgerrit | Daniel Mellado proposed openstack-infra/project-config master: Announce stable branches for Kuryr https://review.openstack.org/652947 | 08:49 |
| *** gregoryo has quit IRC | 08:54 | |
| openstackgerrit | Daniel Mellado proposed openstack-infra/reviewstats master: Add Kuryr to projects https://review.openstack.org/652943 | 08:56 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] letsencrypt update idea https://review.openstack.org/652801 | 09:04 |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add support for 'oldValue' comment-added field https://review.openstack.org/649900 | 09:31 |
| *** dtantsur|afk is now known as dtantsur | 09:35 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] letsencrypt update idea https://review.openstack.org/652801 | 09:40 |
| *** yamamoto has quit IRC | 09:45 | |
| *** yamamoto has joined #openstack-infra | 09:46 | |
| *** yamamoto has quit IRC | 09:46 | |
| *** yamamoto has joined #openstack-infra | 09:47 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add support for 'oldValue' comment-added field https://review.openstack.org/649900 | 09:49 |
| *** yamamoto has quit IRC | 09:52 | |
| *** rcernin has joined #openstack-infra | 10:01 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Use handlers for letsencrypt cert updates https://review.openstack.org/652801 | 10:12 |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add support for 'oldValue' comment-added field https://review.openstack.org/649900 | 10:27 |
| *** yamamoto has joined #openstack-infra | 10:30 | |
| *** yamamoto has quit IRC | 10:39 | |
| *** yamamoto has joined #openstack-infra | 10:40 | |
| *** yamamoto has quit IRC | 10:47 | |
| *** nicolasbock has joined #openstack-infra | 10:55 | |
| *** yamamoto has joined #openstack-infra | 10:58 | |
| *** jpich has quit IRC | 10:58 | |
| *** jpich has joined #openstack-infra | 10:59 | |
| *** panda is now known as panda|lunch | 11:05 | |
| *** Lucas_Gray has quit IRC | 11:09 | |
| *** jpich has quit IRC | 11:12 | |
| *** Lucas_Gray has joined #openstack-infra | 11:12 | |
| *** jpich has joined #openstack-infra | 11:13 | |
| *** jpich has quit IRC | 11:14 | |
| *** pcaruana has quit IRC | 11:17 | |
| *** sshnaidm has quit IRC | 11:17 | |
| openstackgerrit | Brendan proposed openstack-infra/zuul master: gerrit: Add support for 'oldValue' comment-added field https://review.openstack.org/649900 | 11:18 |
| *** quiquell|rover is now known as quique|rover|eat | 11:25 | |
| *** sshnaidm has joined #openstack-infra | 11:26 | |
| *** jpich has joined #openstack-infra | 11:30 | |
| *** jpich has quit IRC | 11:30 | |
| *** ccamacho has quit IRC | 11:34 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support fail-fast in project pipelines https://review.openstack.org/652764 | 11:37 |
| *** dtantsur is now known as dtantsur|brb | 11:45 | |
| *** bobh has joined #openstack-infra | 11:48 | |
| *** Lucas_Gray has quit IRC | 11:52 | |
| *** lpetrut has quit IRC | 11:52 | |
| *** bobh has quit IRC | 11:53 | |
| *** lpetrut has joined #openstack-infra | 12:00 | |
| *** pcaruana has joined #openstack-infra | 12:04 | |
| *** jcoufal has joined #openstack-infra | 12:07 | |
| *** quique|rover|eat is now known as quiquell|rover | 12:07 | |
| openstackgerrit | boden proposed openstack-infra/project-config master: update vmware-nsx jobs https://review.openstack.org/652680 | 12:08 |
| *** rh-jelabarre has joined #openstack-infra | 12:10 | |
| *** bobh has joined #openstack-infra | 12:10 | |
| *** priteau has quit IRC | 12:17 | |
| *** bobh has quit IRC | 12:18 | |
| *** priteau has joined #openstack-infra | 12:18 | |
| *** rlandy has joined #openstack-infra | 12:19 | |
| *** rlandy is now known as rlandy|ruck | 12:20 | |
| *** jamesmcarthur has joined #openstack-infra | 12:21 | |
| *** panda|lunch is now known as panda | 12:22 | |
| *** jpich has joined #openstack-infra | 12:25 | |
| *** jamesmcarthur has quit IRC | 12:30 | |
| *** bhavikdbavishi has quit IRC | 12:31 | |
| *** jbadiapa has quit IRC | 12:32 | |
| *** kgiusti has joined #openstack-infra | 12:35 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:35 | |
| *** jamesmcarthur has joined #openstack-infra | 12:48 | |
| *** aaronsheffield has joined #openstack-infra | 12:57 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Gather host keys for connection-type network_cli https://review.openstack.org/652778 | 12:58 |
| *** jamesmcarthur has quit IRC | 12:58 | |
| Shrews | infra-root: i saw that nl0[123] had nodepool-launcher restarted last night. don't see that running at all on nl04. investigating... | 13:00 |
| pabelanger | Shrews: ah, I forgot to check nl04 yesterday, wonder if corvus did too | 13:01 |
| mordred | Shrews: oh goodie! | 13:01 |
| Shrews | Apr 14 18:19:47 nl04 kernel: [38625591.436648] nodepool-launch invoked oom-killer | 13:04 |
| Shrews | *sigh* | 13:04 |
| Shrews | !status log nodepool-launcher was not running on nl04 due to OOM. Restarted | 13:05 |
| openstack | Shrews: Error: "status" is not a valid command. | 13:05 |
| Shrews | grr | 13:05 |
| Shrews | #status log nodepool-launcher was not running on nl04 due to OOM. Restarted | 13:06 |
| openstackstatus | Shrews: finished logging | 13:06 |
| fungi | consistent at least | 13:06 |
| fungi | eerie they all went at around the same timeframe | 13:06 |
| *** yamamoto has quit IRC | 13:07 | |
| Shrews | uh... it didn't restart | 13:07 |
| *** bobh has joined #openstack-infra | 13:07 | |
| *** pcaruana has quit IRC | 13:07 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Add openstacksdk-functional-devstack-networking job to Neutron dashboard https://review.openstack.org/652993 | 13:08 |
| Shrews | ok, now it is | 13:09 |
| Shrews | fungi: yeah | 13:10 |
| fungi | i suppose that points to some triggering event they share in common (central network disruption, configuration change deploy, package update) | 13:12 |
| openstackgerrit | boden proposed openstack-infra/project-config master: update vmware-nsx jobs https://review.openstack.org/652680 | 13:12 |
| mnaser | has replication caught up since yesterday btw? | 13:16 |
| fungi | oh, yeah, that only took maybe 90 minutes once fixed and triggered | 13:16 |
| *** ccamacho has joined #openstack-infra | 13:20 | |
| *** Goneri has joined #openstack-infra | 13:20 | |
| *** mriedem has joined #openstack-infra | 13:21 | |
| *** eharney has joined #openstack-infra | 13:22 | |
| *** bhavikdbavishi has quit IRC | 13:23 | |
| *** yamamoto has joined #openstack-infra | 13:24 | |
| *** yamamoto has quit IRC | 13:24 | |
| *** dtantsur|brb is now known as dtantsur | 13:24 | |
| *** yamamoto has joined #openstack-infra | 13:24 | |
| *** lseki has joined #openstack-infra | 13:25 | |
| *** yamamoto has quit IRC | 13:25 | |
| *** yamamoto has joined #openstack-infra | 13:25 | |
| *** jpich has quit IRC | 13:25 | |
| *** jpich has joined #openstack-infra | 13:26 | |
| *** egonzalez has quit IRC | 13:30 | |
| *** egonzalez has joined #openstack-infra | 13:30 | |
| *** efried_pto is now known as efried | 13:31 | |
| *** tobiash has quit IRC | 13:31 | |
| *** tobiash has joined #openstack-infra | 13:32 | |
| *** lennyb has quit IRC | 13:32 | |
| *** pcaruana has joined #openstack-infra | 13:33 | |
| openstackgerrit | Malek Karray proposed openstack-infra/storyboard-webclient master: Allowing the user to choose what Columns are seen https://review.openstack.org/651270 | 13:41 |
| *** sthussey has joined #openstack-infra | 13:44 | |
| *** lennyb has joined #openstack-infra | 13:45 | |
| *** bobh has quit IRC | 13:50 | |
| smcginnis | Looks like all release tag-release jobs failed last night with the "Unable to freeze job graph" error here: http://paste.openstack.org/show/749355/ | 13:53 |
| smcginnis | Wondering if that could be caused from https://review.openstack.org/#/c/652659/ | 13:54 |
| smcginnis | And if so, how/why and wondering how we would go about increasing that job timeout. | 13:54 |
| *** yamamoto has quit IRC | 13:57 | |
| *** yamamoto has joined #openstack-infra | 13:58 | |
| fungi | smcginnis: i think so. looks like we need that timeout set in opendev/base-jobs/zuul.yaml because tag-releases is defined there as a "final" job (meaning variants are forbidden). not sure why that didn't get raised as an error by zuul in 652659 (maybe something to do with speculative execution and opendev/base-jobs being a trusted config repo?) | 13:58 |
| *** yamamoto has quit IRC | 13:58 | |
| smcginnis | fungi: OK, we were thinking we could probably work around that timeout anyway, so I guess I can just revert that. | 13:59 |
| smcginnis | fungi: We had a few patches approved last night and unfortunately the error emails don't give any reference to which ones were failed. | 13:59 |
| smcginnis | I can figure out what was impacted, but looks like we'll have to do some kind of clean up work on those. | 14:00 |
| fungi | oh, part of why we didn't find out until tag-releases ran is that it's not run in check/gate (for obvious reasons) | 14:01 |
| smcginnis | Yeah, was thinking that. I don't suppose there's an easy zuul check we can run only on patches that modify zuul.yaml files? | 14:01 |
| frickler | zuul still might or maybe even should be able to verify this at definition time and not only at runtime | 14:02 |
| *** kopecmartin is now known as kopecmartin|off | 14:02 | |
| fungi | hrm, tag-releases isn't actually defined at opendev/base-jobs/zuul.yaml@master#112 | 14:02 |
| fungi | that's the base job | 14:02 |
| fungi | confusing error for sure | 14:02 |
| frickler | openstack-infra/project-config/zuul.d/jobs.yaml@master#1089 from http://logs.openstack.org/80/804d609460e6b165e0549e6759b1d0ee277a970a/release-post/tag-releases/a1dd193/zuul-info/inventory.yaml | 14:03 |
| fungi | http://zuul.opendev.org/t/openstack/job/tag-releases | 14:04 |
| fungi | https://opendev.org/openstack-infra/project-config/src/branch/master/zuul.d/jobs.yaml#L1089 | 14:06 |
| fungi | does indeed have final:true there so that's where we'd need to set the timeout if we wanted one (or remove the finality, if we determine it's an unwarranted protection in this particular case) | 14:06 |
| *** ramishra has quit IRC | 14:06 | |
| *** markvoelker has quit IRC | 14:07 | |
| frickler | fungi: IIUC we want to avoid doing variants of the job that might expose the secrets, so setting the timeout in the job definition directly seems to be the proper solution | 14:07 |
| fungi | yeah, i guess that's more risk for jobs defining this one as a parent, not for variants? | 14:09 |
| fungi | though maybe there's some way to expose them with a variant too and i'm just not thinking of it | 14:10 |
| *** Lucas_Gray has joined #openstack-infra | 14:13 | |
| frickler | yeah, maybe just setting "protected" instead of "final" might be good enough in that case | 14:13 |
| *** eernst has joined #openstack-infra | 14:13 | |
| *** jbadiapa has joined #openstack-infra | 14:15 | |
| *** priteau has quit IRC | 14:19 | |
| smcginnis | Just curous, what is the functional difference between "protected" and "final"? | 14:24 |
| *** iurygregory_ has joined #openstack-infra | 14:25 | |
| AJaeger | smcginnis: you can inherit a protected only in the same repo, you cannot inherit a final job at all | 14:26 |
| smcginnis | Makes sense. Thanks AJaeger. | 14:26 |
| *** rcernin has quit IRC | 14:27 | |
| *** janki has quit IRC | 14:27 | |
| *** iurygregory has quit IRC | 14:27 | |
| *** dtantsur has quit IRC | 14:27 | |
| *** iurygregory_ is now known as iurygregory | 14:27 | |
| *** dpawlik has quit IRC | 14:28 | |
| openstackgerrit | Malek Karray proposed openstack-infra/storyboard-webclient master: Allowing the user to choose what Columns are seen https://review.openstack.org/651270 | 14:29 |
| *** dtantsur has joined #openstack-infra | 14:31 | |
| *** iurygregory_ has joined #openstack-infra | 14:32 | |
| *** iurygregory has quit IRC | 14:33 | |
| *** iurygregory_ is now known as iurygregory | 14:33 | |
| corvus | clarkb: the registry is at 32% usage today, so i guess we have ~3 days worth of storage? seems like we should move it to a large cinder volume. also the docs at https://docs.docker.com/registry/garbage-collection/ (search for "Note: You should ensure that the registry is in read-only mode") explain why we always need to run gc with the registry offline -- so i guess we should plan on a nightly | 14:35 |
| corvus | stop/gc/start cron? | 14:35 |
| *** lpetrut has quit IRC | 14:36 | |
| mordred | corvus: needing to stop it to gc is so cloud native | 14:37 |
| *** priteau has joined #openstack-infra | 14:37 | |
| mgagne | infra-root: fyi, we need to perform an emergency maintenance on inap-mtl01 | 14:37 |
| corvus | mordred: right? | 14:37 |
| corvus | mordred: i think after this we should find a better registry | 14:37 |
| mordred | corvus: ++ | 14:38 |
| *** bobh has joined #openstack-infra | 14:39 | |
| *** quiquell|rover is now known as quiquell|off | 14:41 | |
| *** bobh has quit IRC | 14:41 | |
| mordred | corvus: maybe we shoudl consider the swift driver for the docker registry? | 14:44 |
| mordred | corvus: https://docs.docker.com/registry/storage-drivers/swift/ | 14:44 |
| mordred | I hear object storage is all the rage with the cloud-native set | 14:44 |
| corvus | mordred: ooh, we could avoid using cinder then (and therefore have an arbitrary cap on our size) | 14:44 |
| mordred | corvus: ++ | 14:45 |
| corvus | it looks like it can or does use tempurl? | 14:45 |
| corvus | (i wonder why) | 14:46 |
| corvus | mordred: also, check out the docs for "accesskey" | 14:46 |
| mordred | corvus: that's amazing | 14:48 |
| openstackgerrit | boden proposed openstack-infra/project-config master: update vmware-nsx related jobs https://review.openstack.org/652680 | 14:48 |
| *** dklyle has quit IRC | 14:49 | |
| *** dklyle has joined #openstack-infra | 14:50 | |
| *** pgaxatte has quit IRC | 14:56 | |
| *** jamesmcarthur has joined #openstack-infra | 14:56 | |
| *** helenafm has quit IRC | 14:57 | |
| *** ykarel is now known as ykarel|away | 14:57 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Temporarily disable inap-mtl01 for maintenance https://review.openstack.org/653017 | 14:57 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Revert "Temporarily disable inap-mtl01 for maintenance" https://review.openstack.org/653018 | 14:57 |
| fungi | mgagne: ^ | 14:57 |
| fungi | config-core: can we get 653017 expedited for mgagne? | 14:57 |
| mgagne | fungi: ty | 14:57 |
| clarkb | reviewing now | 14:57 |
| mordred | rosmaita: are virtdriver image properties documented anywhere other than https://wiki.openstack.org/wiki/VirtDriverImageProperties ? | 14:58 |
| mnaser | fungi: +2'd | 14:58 |
| clarkb | and I +3'd | 14:58 |
| fungi | i've marked the revert wip for now so it won't be accidentally merged | 14:58 |
| mnaser | mordred, corvus: the cool thing about using it with swift is afaik it doesn't proxy things through | 14:59 |
| mnaser | creates a one time url with tempurl and hands that over to docker client | 14:59 |
| clarkb | mnaser: that must be what the tmpurls are for | 14:59 |
| mnaser | and docker clients pull directly from object storage | 14:59 |
| corvus | mnaser, mordred: that explains the tempurl reqs | 14:59 |
| mnaser | afaik | 14:59 |
| mordred | ah - that is pretty cool | 15:00 |
| mnaser | fwiw: https://opendev.org/vexxhost/ansible-role-docker-distribution :P | 15:00 |
| mnaser | haven't gotten around adding CI to that though | 15:00 |
| clarkb | swift stores the storage size problem, but we still want to clean up after the registry. Any idea what that looks like if using swift? | 15:02 |
| clarkb | maybe we can configure it to set expire dates on the swift objects? | 15:02 |
| mnaser | https://github.com/docker/distribution/blob/0d3efadf0154c2b8a4e7b6621fff9809655cc580/registry/storage/driver/swift/swift.go#L605-L658 | 15:03 |
| mnaser | https://github.com/lazyfrosch/docker-registry-cleanup is an interesting approach | 15:05 |
| mnaser | it starts it up in read only mode | 15:05 |
| *** iurygregory has quit IRC | 15:05 | |
| *** kaiokmo has quit IRC | 15:06 | |
| rosmaita | mordred: https://docs.openstack.org/glance/latest/admin/useful-image-properties.html | 15:06 |
| *** yboaron has quit IRC | 15:07 | |
| mordred | rosmaita: thanks! I also bugged the nova folks about it and made some grumpy noises about trait:<trait-name> in their channel | 15:07 |
| rosmaita | mordred: a problem with cross-project stuff is that nothing is ever completely up to date | 15:08 |
| *** sreejithp has joined #openstack-infra | 15:09 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Remove some non-oslo projects from oslo.json https://review.openstack.org/653023 | 15:18 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Load subproject data from governance https://review.openstack.org/653024 | 15:18 |
| openstackgerrit | Merged openstack-infra/project-config master: Temporarily disable inap-mtl01 for maintenance https://review.openstack.org/653017 | 15:22 |
| *** ykarel|away is now known as ykarel | 15:22 | |
| *** hamzy has quit IRC | 15:22 | |
| *** mujahidali has joined #openstack-infra | 15:24 | |
| *** pcaruana has quit IRC | 15:27 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated validation rules https://review.openstack.org/653027 | 15:27 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Updated validation rules https://review.openstack.org/653027 | 15:28 |
| *** pcaruana has joined #openstack-infra | 15:30 | |
| *** e0ne has quit IRC | 15:31 | |
| *** iurygregory has joined #openstack-infra | 15:31 | |
| Shrews | infra-root: i'd like to add some logging lines to the launcher on nl04 and restart it to track down an annoying issue. Any issues with me doing that now-ish? | 15:38 |
| fungi | no objection from me | 15:38 |
| clarkb | I have no objection. Beware the ansible/puppet setting things back unexpectedly if you don't add it to hte emergency file | 15:39 |
| *** pcaruana has quit IRC | 15:40 | |
| *** pcaruana has joined #openstack-infra | 15:46 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/ https://review.openstack.org/604404 | 15:46 |
| Shrews | ok, got what i needed. running as normal now | 15:46 |
| *** gyee has joined #openstack-infra | 15:50 | |
| *** nickv1985 has quit IRC | 15:54 | |
| *** nickv1985 has joined #openstack-infra | 15:54 | |
| *** pcaruana has quit IRC | 15:55 | |
| *** bhavikdbavishi has joined #openstack-infra | 16:00 | |
| *** ykarel is now known as ykarel|away | 16:00 | |
| *** pcaruana has joined #openstack-infra | 16:02 | |
| Shrews | amorin: I just sent you an email regarding a problematic instance we have in OVH. Hoping you can help when you get the chance. | 16:02 |
| *** jpich has quit IRC | 16:04 | |
| *** jamesmcarthur has quit IRC | 16:04 | |
| *** dtruong has left #openstack-infra | 16:05 | |
| clarkb | prometheanfire: tonyb[m] github is complaning about our sqlalchemy requirement. Says we should bump to >= 1.3.0 | 16:06 |
| *** ykarel|away has quit IRC | 16:07 | |
| *** lucasagomes has quit IRC | 16:07 | |
| *** pcaruana has quit IRC | 16:08 | |
| fungi | clarkb: you looked at the root alias inbox, huh? | 16:13 |
| *** jamesmcarthur has joined #openstack-infra | 16:13 | |
| fungi | i've been systematically moving those to the service.github mailbox. i recommend you don't peek in there | 16:14 |
| clarkb | fungi: those actually still go to my gmail account too because I haven't removed my personal user. We keep finding a new reason the github sahred account needs updating so I haven't removed mine yet | 16:14 |
| fungi | we get dozens of notifications a day from them on average | 16:14 |
| clarkb | but ya I also occasioanlly peak into that imap connection | 16:14 |
| fungi | including yesterday(?) when they notified us of vulnerabilities for the wrong version of ansible | 16:15 |
| clarkb | my all time favorite was demanding we upgrade to a prerelease pyyaml over the default to safe loading | 16:15 |
| fungi | we had something like ansible>=2.5,<2.6 declared and the warning was about a vulnerability in 2.6 | 16:15 |
| pabelanger | fungi: so, you are getting that from github vulnerability scanner thing they run? | 16:18 |
| clarkb | pabelanger: ya | 16:18 |
| pabelanger | there is a API to disable it | 16:18 |
| pabelanger | maybe we should update manage-projects | 16:18 |
| fungi | i get the impression you have to set it per repository, though i haven't looked closely | 16:20 |
| pabelanger | yah, I think that is right | 16:20 |
| fungi | and am looking forward to the day we (opendev infra) just don't have any group ownership of github mirrors | 16:20 |
| *** Lucas_Gray has quit IRC | 16:20 | |
| mujahidali | Hi infra team, I am trying to update nodepool from(0.3.0 to 0.5.0 can’t upgrade to latest due to some dependencies). I am not able to ping the instances spawned using the image uploaded by nodepool. For 0.3.0 version I was running prepare-node script which was doing some networking stuff for me, now I am not sure from which section I need to invoke the same script. new nodepool.yml http://paste.openstack.org/show/749374/ and ol | 16:22 |
| mujahidali | d nodepool.yaml http://paste.openstack.org/show/749369/ error http://paste.openstack.org/show/749376/ | 16:22 |
| mujahidali | Can anyone help me to figure out the correct nodepool config file ? or any other missing config ? | 16:23 |
| clarkb | mujahidali: you probably need to set config drive enabled on that provider | 16:23 |
| clarkb | and possibly add the simple-init element | 16:24 |
| clarkb | mujahidali: the problem there is you need something like glean (simple-init) or cloud-init to bootstrap networking and credentials on the instance after booting | 16:27 |
| clarkb | mujahidali: the simple-init element will install glean which only uses config drive as a data source. You can theoretically also install cloud-init then other data sources will work, but I don't have any exampels of that | 16:28 |
| mujahidali | clarkb: thanks, I will add simple-init element and set config-drive true for provider and retry. | 16:29 |
| *** ykarel|away has joined #openstack-infra | 16:31 | |
| Shrews | clarkb: looks like we have about 13 instances total in ovh-gra1 in a similar state. i sent a follow up email with those | 16:34 |
| clarkb | Shrews: thanks! | 16:35 |
| Shrews | region BHS1 seems clean | 16:36 |
| *** pcaruana has joined #openstack-infra | 16:36 | |
| fungi | the gh spam is getting worse. i see some announcements about the same dependency in different files of the same repo, notified via separate e-mails :/ | 16:38 |
| *** mattw4 has joined #openstack-infra | 16:39 | |
| *** kjackal has quit IRC | 16:42 | |
| *** rpittau is now known as rpittau|afk | 16:43 | |
| *** kjackal has joined #openstack-infra | 16:43 | |
| *** armax has quit IRC | 16:48 | |
| *** jamesmcarthur_ has joined #openstack-infra | 16:48 | |
| *** jamesmcarthur has quit IRC | 16:52 | |
| *** markvoelker has joined #openstack-infra | 16:52 | |
| *** ccamacho has quit IRC | 16:53 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 16:55 | |
| *** bhavikdbavishi has quit IRC | 16:56 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 16:56 | |
| *** markvoelker has quit IRC | 16:56 | |
| fungi | 167 notifications from github today about suspected vulnerable dependencies in our mirrors | 16:56 |
| fungi | (and counting) | 16:56 |
| *** yolanda_ has quit IRC | 16:57 | |
| *** ginopc has quit IRC | 17:00 | |
| *** ijw has joined #openstack-infra | 17:00 | |
| pabelanger | ouch | 17:00 |
| fungi | up to 177 now | 17:01 |
| fungi | basically one e-mail for every file, branch and repo which mentions a version of sqlalchemy older than the new release | 17:01 |
| fungi | like they couldn't batch these up or something? | 17:02 |
| *** priteau has quit IRC | 17:11 | |
| *** diablo_rojo has joined #openstack-infra | 17:21 | |
| *** bhavikdbavishi has quit IRC | 17:23 | |
| *** bhavikdbavishi has joined #openstack-infra | 17:25 | |
| *** ricolin has quit IRC | 17:25 | |
| *** hamzy has joined #openstack-infra | 17:27 | |
| openstackgerrit | Merged openstack-infra/openstackid master: Migrated Mail from native to Sendgrid API https://review.openstack.org/651794 | 17:31 |
| *** bhavikdbavishi has quit IRC | 17:32 | |
| *** e0ne has joined #openstack-infra | 17:32 | |
| *** armax has joined #openstack-infra | 17:34 | |
| *** dtantsur is now known as dtantsur|afk | 17:36 | |
| *** kaiokmo has joined #openstack-infra | 17:38 | |
| *** ijw has quit IRC | 17:40 | |
| *** eernst has quit IRC | 17:41 | |
| *** ijw has joined #openstack-infra | 17:42 | |
| *** dklyle has quit IRC | 17:45 | |
| clarkb | Shrews: I'd like to merge https://review.openstack.org/#/c/650379/3 today if possible, but I'll defer to you on whether or not that impacts your debugging efforts | 17:45 |
| Shrews | clarkb: i've completed my debugging efforts for the ovh stuff | 17:47 |
| Shrews | so go for it | 17:47 |
| clarkb | ok I'll queue that up for after today's meeting | 17:48 |
| *** priteau has joined #openstack-infra | 17:49 | |
| *** e0ne has quit IRC | 17:49 | |
| *** igordc has joined #openstack-infra | 17:50 | |
| *** mattw4 has quit IRC | 18:00 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackid master: Add sendgrid support https://review.openstack.org/651798 | 18:05 |
| prometheanfire | clarkb: github is? | 18:06 |
| *** iurygregory has quit IRC | 18:07 | |
| prometheanfire | clarkb: well, for train we probably can, we are close, 1-2 projects still have issues with sqlalchemy>=1.3.0 | 18:07 |
| openstackgerrit | Merged openstack-infra/system-config master: Added openstackid dev sendgrid support https://review.openstack.org/651806 | 18:08 |
| fungi | in the past we haven't updated stable branches for such things, considering updating dependencies something that distros handle | 18:08 |
| fungi | as they're going to backport the security fixes to something which claims to still be the version our stable branch depends on, essentially | 18:08 |
| clarkb | prometheanfire: ya they email you when they detect dependencies that have known security bugs | 18:09 |
| fungi | or in many cases, misdetect | 18:11 |
| prometheanfire | clarkb: ah, that github feature, ya | 18:11 |
| prometheanfire | well, it's in progress | 18:12 |
| prometheanfire | see https://review.openstack.org/651591 | 18:12 |
| fungi | our root alias is now up to 183 notifications from them today alone | 18:12 |
| fungi | it's absurd | 18:12 |
| *** electrofelix has quit IRC | 18:13 | |
| clarkb | I took it as motivation to clean up some subscriptions I didn't need on my personal email | 18:15 |
| *** ijw has quit IRC | 18:15 | |
| *** ijw has joined #openstack-infra | 18:16 | |
| mordred | prometheanfire: "feature" | 18:21 |
| *** ijw has quit IRC | 18:22 | |
| *** jamesmcarthur_ has quit IRC | 18:25 | |
| *** jamesmcarthur has joined #openstack-infra | 18:26 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Materials API bug fixing https://review.openstack.org/653076 | 18:31 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Materials API bug fixing https://review.openstack.org/653076 | 18:32 |
| *** priteau has quit IRC | 18:33 | |
| *** ijw has joined #openstack-infra | 18:35 | |
| *** jamesmcarthur has quit IRC | 18:35 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Presentation Links api bug fixing https://review.openstack.org/653078 | 18:37 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Presentation Links api bug fixing https://review.openstack.org/653078 | 18:37 |
| *** irclogbot_0 has quit IRC | 18:39 | |
| *** jamesmcarthur has joined #openstack-infra | 18:40 | |
| *** irclogbot_1 has joined #openstack-infra | 18:40 | |
| *** ykarel|away has quit IRC | 18:41 | |
| clarkb | Infra meeting in ~10 minutes | 18:50 |
| clarkb | we'll be in #openstack-meeting for that | 18:50 |
| *** mujahidali has quit IRC | 19:14 | |
| *** eharney has quit IRC | 19:17 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Load subproject data from governance https://review.openstack.org/653024 | 19:21 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Handle all exceptions loading pickled data https://review.openstack.org/653106 | 19:21 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: Add review.opendev.org https://review.openstack.org/653108 | 19:22 |
| *** bobh has joined #openstack-infra | 19:25 | |
| Shrews | infra-root: just fyi, we finally hit the thing that caused nodepool builder to leak images over the weekend, and verified that the new code handled it properly. so... win | 19:27 |
| mordred | Shrews: \o/ | 19:27 |
| *** bobh has quit IRC | 19:28 | |
| fungi | right on! | 19:29 |
| *** bobh has joined #openstack-infra | 19:37 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add test_setup_reset_connection setting https://review.openstack.org/653130 | 19:38 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Remove some non-oslo projects from oslo.json https://review.openstack.org/653023 | 19:49 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Load subproject data from governance https://review.openstack.org/653024 | 19:49 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Handle all exceptions loading pickled data https://review.openstack.org/653106 | 19:49 |
| openstackgerrit | Ben Nemec proposed openstack-infra/reviewstats master: Import Zuul job settings https://review.openstack.org/653133 | 19:49 |
| *** dklyle has joined #openstack-infra | 19:58 | |
| corvus | dtroyer: what is it you're looking to do? | 20:00 |
| *** jamesmcarthur has quit IRC | 20:00 | |
| dtroyer | corvus: starlingx will be carrying a stable/stein branch for additional backports during Train, I'd like to be able to run at least a handful of tests for PRs to that branch | 20:01 |
| dtroyer | for Nova only | 20:01 |
| corvus | dtroyer: where does github come into play? | 20:01 |
| dtroyer | that is where the branch lives https://github.com/starlingx-staging/stx-nova/tree/stx/stein.1 | 20:02 |
| corvus | dtroyer: is it not going to move into gerrit (along with these? https://git.starlingx.io/cgit ) | 20:02 |
| clarkb | corvus: aiui the old plan was not to do that because people didn't want to have a fork of nova hosted on the opnstack infrastructure. We might have a bit more leeway when things are opendev'd | 20:03 |
| dtroyer | no. nothing more from github moves into gerrit, that was settled last summer. This is what used to be the 'fork' of nova that WRS carried | 20:03 |
| dtroyer | also, this is only for the train cycle, I'll have words if it extends beyond that | 20:04 |
| corvus | dtroyer: is that repo going to be eliminated eventually? | 20:04 |
| dtroyer | yes | 20:04 |
| corvus | gotcha | 20:04 |
| clarkb | I take it the backports people want to make to stein won't fit into openstack's stable maintenance policy? | 20:04 |
| clarkb | or we arent unforked enough yet? | 20:04 |
| dtroyer | correct, things like Artom's NUMA live migration patches which almost made stein but didn't | 20:05 |
| dtroyer | no, this is the only thing left of the forks. We have 11 'things' on the list to backport when they merge upstream | 20:05 |
| mordred | dtroyer: so - the fork that's there isn't something that needs to be tested/gated itself - as much as it needs to be available to be pulled in to jobs for other things - yeah? | 20:05 |
| *** igordc has quit IRC | 20:05 | |
| dtroyer | I am just looking to sanity check the PRs proposed to github before merging them. otherwise we have to create the docker image and test it the long way | 20:06 |
| dtroyer | or di a devstack/tempest run manually. | 20:06 |
| dtroyer | this is not going to be a frequent thing, I can live with that | 20:06 |
| dtroyer | if Zuul + github isn't a happy thing, that's fine | 20:07 |
| mordred | nod - so you _would_ like to run tests on PRs to that repo - but it's primarily a place where some stuff you're working to get rid of is living | 20:07 |
| dtroyer | more specifically 'stuff from upstream train we need before the train release' | 20:08 |
| clarkb | thinking out loud here | 20:12 |
| clarkb | what are the chances nova would host a less stable stein branch? | 20:12 |
| clarkb | cinder did similar in the past for "driverfixes" | 20:12 |
| dtroyer | clarkb: it may be a realistic possibility now but given the history I chose not to persue that. | 20:14 |
| *** eharney has joined #openstack-infra | 20:14 | |
| dtroyer | I don't know of the 'for train only' would make it more or less likely for that to be a good idea | 20:14 |
| *** weshay has quit IRC | 20:14 | |
| *** weshay has joined #openstack-infra | 20:15 | |
| clarkb | fwiw the trouble we've had with zuul + github are largely on the engagement side of things. I think if a project is off doing most of its own stuff and only consuming zuul its been hard for us to build effective communication back and forth | 20:15 |
| clarkb | I don't expect dtroyer will have problems with that. Which makes my biggest concern here the reaction to the forking that happened | 20:15 |
| dtroyer | that is good to know, I was wondering if it was technical or cultural | 20:15 |
| corvus | this is not the only problem i have with supporting github in opendev's zuul | 20:16 |
| clarkb | corvus: ya there are other concerns from a software freenes perspective. But from a "things don't physically work" that has been the issue | 20:16 |
| dtroyer | the reaction so far has been minimal… more on twitter than anywhere else I've seen | 20:16 |
| corvus | reaction to what? (i don't use twitter so i don't know what people are saying on it) | 20:17 |
| dtroyer | reaction to stx carrying a fork of stable/stein. I'm trying to do it as transparently as possible, which helped | 20:17 |
| dtroyer | (for nova) | 20:18 |
| corvus | ah | 20:18 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add a opendev migration repo name mapping script https://review.openstack.org/653138 | 20:18 |
| fungi | woohoo! i got the rename mapping generator working, just need to fiddle with output formatting now | 20:18 |
| fungi | and with that, going to go get dinner with some friends, then get back to it | 20:19 |
| *** pcaruana has quit IRC | 20:19 | |
| clarkb | I think from our perspective a less stable stein branch in gerrit would be easiest to accomodate | 20:21 |
| *** priteau has joined #openstack-infra | 20:23 | |
| dtroyer | clarkb: ok, I'll talk over beer to some folks about it in Denver and see where we may stand. thanks | 20:24 |
| *** priteau has quit IRC | 20:27 | |
| *** kgiusti has left #openstack-infra | 20:28 | |
| anteaya | clarkb: I missed the meeting, I see 1500UTC was decided as the start time, and you are going to update the emails? | 20:29 |
| clarkb | anteaya: I am, but I need to take a break as this headache isn't going away | 20:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Prune the intermediate registry before GCing https://review.openstack.org/653148 | 20:30 |
| anteaya | clarkb: I'm sorry to hear that, I hope some time away from keyboard bring relief | 20:31 |
| corvus | clarkb: then do not look at that change unless you find jq and awk an effective remedy for headaches ^ | 20:31 |
| anteaya | brings* | 20:31 |
| mordred | corvus: jq and awk are an effective remedy for something? | 20:31 |
| clarkb | I just had some toast so I can take the painkillers. Hopefully that gets me sorted. Back in a bit | 20:31 |
| anteaya | I favour a big glass of water and darkness myself | 20:31 |
| corvus | mordred: i've decided awk is under-appreciated and i should use it more | 20:33 |
| corvus | mordred: it's like... i dunno, as effective as perl, but more arcane and compact! :) | 20:33 |
| corvus | every character of those awk programs represents about 5 minutes of careful work | 20:34 |
| openstackgerrit | Ben Nemec proposed openstack-infra/project-config master: Remove reviewstats Zuul job config https://review.openstack.org/653150 | 20:34 |
| *** mnencia_ has joined #openstack-infra | 20:34 | |
| *** mnencia has quit IRC | 20:35 | |
| *** mnencia_ is now known as mnencia | 20:35 | |
| corvus | clarkb, mordred: i'm going to poke at the swift driver next | 20:36 |
| *** ijw has quit IRC | 20:42 | |
| pabelanger | ianw: any ideas why DIB is rasing an exception? http://paste.openstack.org/show/749392/ | 20:44 |
| pabelanger | also seems to be leaking mounts | 20:44 |
| pabelanger | http://paste.openstack.org/show/749393/ | 20:47 |
| pabelanger | let me move to better channel | 20:47 |
| *** ijw has joined #openstack-infra | 20:47 | |
| *** ijw has quit IRC | 20:54 | |
| *** hamzy has quit IRC | 20:54 | |
| *** bobh has quit IRC | 20:56 | |
| mordred | corvus: I have amazed and astounded people while doing consulting engagements simply with command-line awk | 20:56 |
| corvus | mordred: cool! you can let me know if i did it right :) | 20:57 |
| *** Goneri has quit IRC | 21:00 | |
| mordred | corvus: you'd think that ... but I've already learned things from your patch :) | 21:00 |
| mordred | corvus: also - I love that you have a command-line python in there too | 21:00 |
| corvus | heh, i copied that from a similar ansible thing from a while back -- i could not find a way to get the exact format i needed with 'date' | 21:01 |
| *** jamesmcarthur has joined #openstack-infra | 21:01 | |
| mordred | corvus: I feel like 'date' is sort of like 'find' - if I knew the right combo of flags it should be fully turing complete | 21:02 |
| mordred | but like find - there is no way as a human that I could possibly know the right combo of flags | 21:02 |
| corvus | fortunately, stackoverflow knows... though it's also been trying to tell me spoilers about GoT... so i've been using it less today. | 21:03 |
| corvus | (i'm guessing their product managers did not see that one coming) | 21:04 |
| mordred | corvus: GoT is the one show where we've chosen to actively watch it immediately when available - because avoiding spoilers is so darned hard | 21:05 |
| *** jcoufal has quit IRC | 21:07 | |
| *** kaiokmo has quit IRC | 21:10 | |
| *** gfidente has quit IRC | 21:22 | |
| *** kjackal has quit IRC | 21:23 | |
| *** ijw has joined #openstack-infra | 21:25 | |
| corvus | registry_1 | panic: Swift authentication failed: Bad Request | 21:35 |
| corvus | sigh | 21:35 |
| corvus | mordred: ^ i'm not having much luck with the swift driver | 21:35 |
| corvus | i assume there's something wonky with the parameters | 21:35 |
| corvus | registry_1 | panic: Failed to retrieve info about container insecure_registry (Operation forbidden) | 21:36 |
| corvus | i've managed to get 2 kinds of errors | 21:36 |
| corvus | i guess the second one is better? i'm not sure. | 21:36 |
| mordred | corvus: can you show me how you're mapping in values to their input parameters? | 21:36 |
| corvus | yep | 21:36 |
| mordred | sweet | 21:36 |
| corvus | mordred: http://paste.openstack.org/show/749396/ | 21:37 |
| corvus | mordred: i elided 2 things with <brackets> | 21:38 |
| corvus | otherwise that's the config file i have | 21:38 |
| mordred | corvus: can you try changing tenant to tenantid | 21:38 |
| corvus | did, same error | 21:39 |
| mordred | hrm. I don't suppose you've found a way to get the docker to print a trace of its http interactions have you? | 21:40 |
| corvus | no, i was hoping logging: level: debug would do that but :( | 21:41 |
| mordred | corvus: does the insecure_registry container exist? | 21:42 |
| ianw | pabelanger: 2019-04-16 20:35:57.885 | chown: cannot access '/etc/unbound/conf.d/forwarding.conf': Not a directory | 21:42 |
| corvus | mordred: gah! no. but intermediate_registry does exist, and still produces the same orror. | 21:43 |
| mordred | corvus: poop | 21:44 |
| corvus | mordred: apparently we're here: https://github.com/docker/distribution/blob/master/registry/storage/driver/swift/swift.go#L222 | 21:46 |
| corvus | mordred: oh, wow, i also put in the wrong region | 21:47 |
| corvus | mordred: apparently i need a break. sorry for the goose chase and thanks for the help. | 21:47 |
| mordred | corvus: oh! well - sure thing! | 21:47 |
| corvus | it seems to be doing something without errors now :) | 21:47 |
| mordred | corvus: \o/ | 21:47 |
| * mordred is super helpful | 21:48 | |
| *** Goneri has joined #openstack-infra | 21:48 | |
| corvus | i seem to be able to push images into it | 21:50 |
| corvus | though it looks like any layers > 40MB may have failed... | 21:54 |
| *** ijw has quit IRC | 21:55 | |
| *** ijw has joined #openstack-infra | 21:55 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed materials api bugs https://review.openstack.org/653160 | 21:58 |
| mordred | corvus: chunksize should default to 5M | 22:00 |
| mordred | so I do't know why layers over 40M would be a problem | 22:00 |
| mordred | (if it was layers over 5M I'd say chunk size was borked) | 22:01 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed materials api bugs https://review.openstack.org/653160 | 22:02 |
| corvus | it's really hard to tell what's going on; tcpdump traffic between my workstation and cloud files continues long after the 'docker push' command has failed | 22:03 |
| corvus | i don't know if the docker daemon is still pushing stuff to the registry, or the registry is still pushing stuff to swift, or if the docker daemon is pushing stuff to swift... | 22:03 |
| corvus | but this is the output at the end of a push command: http://paste.openstack.org/show/749401/ | 22:04 |
| mordred | uh | 22:04 |
| corvus | meanwhile, the registry occasionally emits a log line like this: swift.PutContent("/docker/registry/v2/repositories/sshd/_uploads/020f4ceb-637b-4a3e-9540-926fba453663/hashstates/sha256/29704593")" | 22:05 |
| mordred | corvus: that unauthorized line doesn't look like an error from swift | 22:05 |
| corvus | mordred: oh... hrm... | 22:06 |
| corvus | mordred: i wonder if i messed up the auth config when i switched over to using swift | 22:06 |
| *** sreejithp has quit IRC | 22:10 | |
| fungi | and back | 22:11 |
| corvus | that doesn't seem to be the issue | 22:12 |
| fungi | and yes, not to knock perl, but i do prefer awk myself | 22:13 |
| corvus | apparently you can't delete a folder in a rax cloudfiles container unless it's empty | 22:16 |
| corvus | and you can't delete a rax cloudfiles container unless *it* is empty | 22:16 |
| corvus | "For bulk deletes, we recommend using a free third-party cloud storage browser such as Cyberduck or Cloudberry Explorer." | 22:16 |
| fungi | "For bulk deletes, we recommend switching to a better service" | 22:19 |
| fungi | okay, back to the repo namespace mapping generator. it's scary when one script needs to grok yaml, json *and* csv | 22:19 |
| ianw | fungi: easy, convert it all to xml :) | 22:20 |
| fungi | oh, right! the "enterprise" solution | 22:20 |
| fungi | on its five-year mission to chart new serialization | 22:20 |
| mordred | infra-root: I must now AFK - it is time to go pick up many pounds of crawfish | 22:26 |
| fungi | i fully endorse this priority of yours | 22:26 |
| fungi | gumbo? étouffée? | 22:27 |
| fungi | even just straight up steamed is good | 22:28 |
| fungi | i chose the wrong beach for crawdads | 22:28 |
| fungi | we make do with shrimp | 22:29 |
| *** rcernin has joined #openstack-infra | 22:38 | |
| pabelanger | ianw: yah, fixing that, but was surprised to see dib leak the mounts | 22:43 |
| corvus | mordred: i have a theory -- i think the auth error (and subsequent failures) are due to this failure: | 22:46 |
| corvus | Apr 16 15:38:05 fuligin dockerd[23164]: time="2019-04-16T15:38:05.624032957-07:00" level=debug msg="Failed to check for presence of layer sha256:fd3cdb0bc0a09eddd99259cda1de9f0303bbea05004a9a87c3871a22199bc614 (sha256:41381f6a2bc0fc86e5b258de9a54bd86691f6bca07b589cb68c241b390de8e66) in localhost:5000/sshd" error="unauthorized: authentication required" | 22:46 |
| corvus | as that's a 'get' opreation, perhaps that's using tempurl stuff, whereas the uploads are not | 22:47 |
| corvus | or.... | 22:50 |
| corvus | this is somehow running afoul of the rax swift/cdn stuff | 22:50 |
| *** Goneri has quit IRC | 22:50 | |
| corvus | gah... the registry swift driver config lets you specify the tempurl methods, but it does *not* let you override what the server returns; the value is only used if the server doesn't return anything | 22:53 |
| corvus | that seems really backwards | 22:53 |
| *** tkajinam has joined #openstack-infra | 22:54 | |
| *** whoami-rajat has quit IRC | 23:02 | |
| tonyb[m] | clarkb: Can you forward me/us the details? why would github care? | 23:06 |
| *** jamesmcarthur has quit IRC | 23:06 | |
| *** Goneri has joined #openstack-infra | 23:06 | |
| ianw | pabelanger: are you on a current release? we did merge a fix hopefully for leaking | 23:06 |
| clarkb | tonyb[m]: github scans your dependency lists for versions with known security issues then spams you when they find them | 23:06 |
| tonyb[m] | Ahh okay | 23:07 |
| *** jamesmcarthur has joined #openstack-infra | 23:07 | |
| tonyb[m] | Does it tell you which branch? | 23:07 |
| pabelanger | ianw: no, https://github.com/ansible-network/windmill-config/blob/master/ansible/group_vars/nodepool-builder.yaml#L14 | 23:07 |
| pabelanger | I had to pin to get fedora-29 images working | 23:07 |
| fungi | tonyb[m]: it does, yes | 23:07 |
| pabelanger | I should try to uncap and see if it's working again | 23:07 |
| ianw | pabelanger: yeah, https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=528456407131c8771588fd3d53f91785b728752e is the change | 23:08 |
| clarkb | in this case it was like 180 emails so I deleted them | 23:09 |
| pabelanger | ianw: ack, thanks | 23:09 |
| fungi | i stashed all the ones which came to the infra root alias | 23:09 |
| fungi | in case they're needed | 23:09 |
| fungi | but we have some 2+k notices like that from github over past months in the same mailbox | 23:10 |
| *** jamesmcarthur has quit IRC | 23:11 | |
| fungi | infra-root: config-core: infra-puppet-core: i meant to bring this up during the meeting, but the namespace decision reached for openstack is that *all* official repos which aren't currently in the openstack namespace are moving to the openstack namespace (this includes things in openstack-infra and openstack-dev), so if there are infra repos we *don't* want moved, we should work with clarkb to get | 23:13 |
| fungi | them out of openstack governance | 23:13 |
| fungi | (before friday) | 23:14 |
| *** Goneri has quit IRC | 23:14 | |
| fungi | or at least come to some agreement to whitelist specific things in the short term | 23:15 |
| corvus | i think that's swell; i expect the next big task for us is to set up governance for opendev to normalize "opendev/" repos | 23:15 |
| *** jcoufal has joined #openstack-infra | 23:15 | |
| fungi | yes, i agree | 23:15 |
| corvus | in the interim, i do think openstack/system-config might be a teensy bit awkward | 23:15 |
| fungi | sure, as i said, i'm not against whitelisting things in the generator script | 23:16 |
| corvus | so we may want to look for things like that where we might want to do that ^ :) | 23:16 |
| fungi | i'll have the list up shortly (containing the 925 moves the algorithms have identified) | 23:17 |
| fungi | er, 825 | 23:17 |
| corvus | (it doesn't bother me wearing my infra hat, it's more that wearing my openstack hat, i think openstack/system-config will do a disservice to the branding effort that the openstack project is trying to do) | 23:17 |
| fungi | sure, i also think there are more things which are likely to move out of the openstack namespace after migration (for example, talking the osf staff and board into moving repos under the jurisdiction of board working groups and staff efforts to an "osf" namespace) | 23:19 |
| *** tosky has quit IRC | 23:19 | |
| *** guimaluf has quit IRC | 23:21 | |
| *** rlandy|ruck is now known as rlandy|ruck|biab | 23:22 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add a opendev migration repo name mapping script https://review.openstack.org/653138 | 23:24 |
| fungi | this is a preliminary list based on ^ http://paste.openstack.org/show/749402/ | 23:24 |
| pabelanger | fungi: nice, ethercalc stuff good | 23:27 |
| tonyb[m] | [tony@thor ~]$ ssh review gerrit ls-projects | grep inaugust | 23:30 |
| tonyb[m] | inaugust/inaugust.com | 23:30 |
| tonyb[m] | inaugust/src.sh | 23:30 |
| tonyb[m] | inaugust/ttrun | 23:30 |
| tonyb[m] | mordred: shouls ^^^ that be a thing? | 23:31 |
| fungi | yep, welcome to opendev! | 23:31 |
| tonyb[m] | okay | 23:31 |
| fungi | those aren't moving, as they're already in their intended namespace | 23:31 |
| tonyb[m] | I see some scripting updates in my future | 23:31 |
| fungi | yeah, for many of us | 23:31 |
| tonyb[m] | okay | 23:32 |
| fungi | you can no longer expect any particular namespaces, so may want to do some filtering | 23:32 |
| fungi | on a positive note, after friday all official openstack projects will be in the "openstack" namespace | 23:32 |
| fungi | so if that's what you care about, it may make your filtering rule remarkably trivial ;) | 23:32 |
| tonyb[m] | I think I care about zuul, airship and starlingX too | 23:33 |
| tonyb[m] | but none of it is too hard | 23:33 |
| fungi | each of those will have their own namespaces too (and you just guessed the names for them) | 23:33 |
| tonyb[m] | hehe | 23:34 |
| fungi | note however around line 181 in my example paste, some of them are dropping basename prefixes when they move | 23:35 |
| tonyb[m] | so ... gerrit ls-projects | grep -Ei '^(openstack|staringx|zuul|airship)' will basically do what I want? | 23:35 |
| fungi | you may want a / there too, i don't know how open-ended you care to be | 23:35 |
| tonyb[m] | okay | 23:35 |
| tonyb[m] | So another question I just remembered I wanted to ask .... | 23:36 |
| *** rh-jelabarre has quit IRC | 23:36 | |
| tonyb[m] | There is a project on gihub (https://github.com/redhat-cip/hardware) for which travis isn't working | 23:36 |
| tonyb[m] | can I a) setup zuul to run jobs on any PRs for that repo ; and b) assuming the answer to a is 'sure' are there docs? | 23:37 |
| fungi | tonyb[m]: see if http://lists.openstack.org/pipermail/openstack-infra/2019-January/006269.html answers it for you | 23:39 |
| fungi | if not, happy to dig deeper | 23:39 |
| *** jcoufal has quit IRC | 23:41 | |
| fungi | (tl;dr is that zuul.opendev.org exists to test software hosted on review.opendev.org and possibly some of those projects dependencies on github or other external services, but is not intended as a stand-alone tool for projects hosted outside review.opendev.org) | 23:41 |
| tonyb[m] | okay so this library is used by ironic | 23:42 |
| fungi | the argument could be made that's third-party testing of an ironic dependency | 23:42 |
| fungi | however, if the project in question is under an open license and the maintainers are interested in moving it to review.opendev.org i think we'd be thrilled to help | 23:42 |
| tonyb[m] | but some members of the core team don't want to move it to opendev :( | 23:43 |
| * tonyb[m] would love to move it | 23:43 | |
| clarkb | thats actually a big part of my concern doing testing for github projects. experience says in those cases communication is poor and we get blamed rather that collaborated eith in fixing things | 23:44 |
| fungi | if that project is a pain point for ironic and the goal is to run some jobs to integration-test pull requests for it with ironic to make sure they don't break ironic, then that's probably in keeping with the other examples we have | 23:44 |
| tonyb[m] | Okay I don't think I can quite make that argument | 23:47 |
| tonyb[m] | I think I just need to work out how to fix Travis | 23:47 |
| tonyb[m] | :( | 23:47 |
| * tonyb[m] wouldnm | 23:47 | |
| fungi | but as a rule, we don't want opendev to be seen as endorsing some proprietary hosting service, and directly integrating with it for the benefit of projects who don't want to host in opendev's infrastructure is a bridge to far | 23:47 |
| tonyb[m] | 't enable anythign without the permission of the otehr core reviewers | 23:47 |
| tonyb[m] | okay | 23:47 |
| tonyb[m] | That's a clear answer and I have a path forward | 23:48 |
| fungi | excellent! it wasn't entirely clear for many of us for a long time, which is why we needed to come up with a good position statement | 23:48 |
| *** sshnaidm is now known as sshnaidm|afk | 23:50 | |
| fungi | it's hard for us to toe the free/libre open source tools line and at the same time say we're okay with serving as an a la carte menu of services for projects who don't believe that's important | 23:51 |
| pabelanger | I actually had a question, is there policy in place that would prevent me from creating a repo in opendev.org/inaugust ? | 23:52 |
| pabelanger | or opendev.org/vexxhost | 23:52 |
| fungi | this is something i was wanting to bring up, though it's not necessary to solve before friday | 23:53 |
| openstackgerrit | Merged openstack-infra/system-config master: Add Puppet-Version: !X skip to apply tests https://review.openstack.org/652472 | 23:53 |
| fungi | we likely need some way to track points of contact for namespaces | 23:53 |
| pabelanger | yah, I could see in some case, project doesn't want to use opendev but also doesn't wait their namespace to be used | 23:53 |
| fungi | at the moment we know who they are, but that sort of relationship doesn't scale forever | 23:54 |
| fungi | or maybe even past june | 23:54 |
| clarkb | but also one thing at a time so that we can remain sane :P | 23:54 |
| fungi | yes | 23:54 |
| pabelanger | yah | 23:54 |
| fungi | which is why i hadn't planned to bring it up yet | 23:54 |
| pabelanger | okay cool | 23:54 |
| pabelanger | hopefully people don't try to do a land grab on things | 23:55 |
| fungi | on the "project doesn't want to use opendev" point, i'm not sure i care that we have some way for them to squat namespaces in our service | 23:55 |
| *** rlandy|ruck|biab is now known as rlandy|ruck | 23:55 | |
| fungi | currently the only way (afaik) to create a namespace in our gitea is to create a git repository in it through our new project creation workflow, and that has people reviewing config changes as gatekeepers | 23:56 |
| pabelanger | yah | 23:56 |
| fungi | so hopefully anything wild would come to our attention fairly quickly | 23:56 |
| *** hwoarang has quit IRC | 23:57 | |
| fungi | i think if and when request volume rises to the point where we need to take more people out of the workflow, we'll likely add more safeguards at the same time | 23:57 |
| *** hwoarang has joined #openstack-infra | 23:58 | |
| clarkb | config-core I just sent email out about the new project ceration freeze to the infra mailing list | 23:59 |
| fungi | thanks! | 23:59 |
| clarkb | now to send email about the gerrit fun starting at 1500UTC | 23:59 |
| fungi | i'll be putting together some (individualized) e-mails to various mailing lists with the straw man renames list | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!