| *** igordc has quit IRC | 00:09 | |
| *** mlavalle has quit IRC | 00:15 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Add output for mis-configured element scripts https://review.opendev.org/680785 | 00:18 |
|---|---|---|
| *** nicolasbock has quit IRC | 00:27 | |
| *** rfolco has quit IRC | 00:45 | |
| *** Goneri has joined #openstack-infra | 00:46 | |
| *** yamamoto has joined #openstack-infra | 00:55 | |
| *** yamamoto has quit IRC | 00:59 | |
| *** claudiub has quit IRC | 01:13 | |
| *** yamamoto has joined #openstack-infra | 01:15 | |
| *** sgw has quit IRC | 01:16 | |
| *** Goneri has quit IRC | 01:31 | |
| *** sgw has joined #openstack-infra | 01:34 | |
| *** jcoufal has joined #openstack-infra | 01:49 | |
| sean-k-mooney | donnyd: clarkb: so the old lable works and the new label fails | 02:00 |
| sean-k-mooney | i dont know why but thats a thing | 02:01 |
| clarkb | sean-k-mooney: ok I can help look into it more monday | 02:02 |
| *** jcoufal_ has joined #openstack-infra | 02:02 | |
| sean-k-mooney | no worries its 3 am so im gong to go to sleep but im going to swap back to useing the old lable on monday and we can swap to the new one agian once its working | 02:03 |
| *** jcoufal has quit IRC | 02:04 | |
| *** jcoufal_ has quit IRC | 02:08 | |
| *** apetrich has quit IRC | 02:10 | |
| *** yamamoto has quit IRC | 02:18 | |
| *** markvoelker has joined #openstack-infra | 02:21 | |
| *** markvoelker has quit IRC | 02:25 | |
| *** markvoelker has joined #openstack-infra | 02:27 | |
| *** jcoufal has joined #openstack-infra | 02:34 | |
| *** markvoelker has quit IRC | 02:37 | |
| *** vesper11 has quit IRC | 02:41 | |
| *** dklyle has quit IRC | 02:45 | |
| *** vesper11 has joined #openstack-infra | 02:47 | |
| *** jcoufal has quit IRC | 02:51 | |
| *** rfolco has joined #openstack-infra | 02:52 | |
| *** eernst has joined #openstack-infra | 03:08 | |
| *** eernst has quit IRC | 03:10 | |
| *** dklyle has joined #openstack-infra | 03:13 | |
| *** yamamoto has joined #openstack-infra | 03:40 | |
| *** yamamoto has quit IRC | 04:11 | |
| *** yamamoto has joined #openstack-infra | 04:20 | |
| *** rh-jelabarre has quit IRC | 04:28 | |
| *** markvoelker has joined #openstack-infra | 04:31 | |
| *** dklyle has quit IRC | 04:32 | |
| *** markvoelker has quit IRC | 04:35 | |
| *** dklyle has joined #openstack-infra | 04:43 | |
| cmurphy | corvus: pretty sure i've misunderstood what you meant https://review.opendev.org/680798 what do i need to do to override that timeout? | 04:46 |
| *** gfidente has joined #openstack-infra | 05:17 | |
| *** bdodd has quit IRC | 05:23 | |
| *** yamamoto has quit IRC | 05:23 | |
| *** yamamoto has joined #openstack-infra | 05:26 | |
| *** slaweq has quit IRC | 05:33 | |
| *** kjackal has joined #openstack-infra | 05:44 | |
| *** yamamoto has quit IRC | 05:49 | |
| *** kjackal has quit IRC | 05:50 | |
| *** signed8b_ has quit IRC | 05:54 | |
| *** mtreinish has joined #openstack-infra | 05:57 | |
| *** yamamoto has joined #openstack-infra | 06:02 | |
| *** yamamoto has quit IRC | 06:26 | |
| *** yamamoto has joined #openstack-infra | 06:32 | |
| *** yamamoto has quit IRC | 06:39 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| *** pcaruana has quit IRC | 07:14 | |
| *** slaweq has quit IRC | 07:15 | |
| *** yamamoto has joined #openstack-infra | 07:18 | |
| *** dklyle has quit IRC | 07:20 | |
| *** david-lyle has joined #openstack-infra | 07:20 | |
| *** yamamoto has quit IRC | 07:23 | |
| *** pcaruana has joined #openstack-infra | 07:24 | |
| *** david-lyle has quit IRC | 07:27 | |
| *** yamamoto has joined #openstack-infra | 07:30 | |
| *** yamamoto has quit IRC | 07:31 | |
| *** markvoelker has joined #openstack-infra | 07:35 | |
| *** markvoelker has quit IRC | 07:40 | |
| *** slaweq has joined #openstack-infra | 08:01 | |
| *** yamamoto has joined #openstack-infra | 08:02 | |
| *** yamamoto has quit IRC | 08:07 | |
| *** diga has joined #openstack-infra | 08:14 | |
| zbr | please wf https://review.opendev.org/#/c/673893/ | 08:35 |
| *** e0ne has joined #openstack-infra | 08:36 | |
| *** e0ne has quit IRC | 08:43 | |
| *** markvoelker has joined #openstack-infra | 08:51 | |
| *** markvoelker has quit IRC | 08:55 | |
| *** jtomasek has joined #openstack-infra | 08:57 | |
| *** ociuhandu has joined #openstack-infra | 09:16 | |
| *** ociuhandu has quit IRC | 09:21 | |
| *** ralonsoh has joined #openstack-infra | 09:33 | |
| *** kjackal has joined #openstack-infra | 09:39 | |
| *** yamamoto has joined #openstack-infra | 09:42 | |
| *** yamamoto has quit IRC | 09:47 | |
| *** yamamoto has joined #openstack-infra | 09:47 | |
| *** yamamoto has quit IRC | 09:50 | |
| *** kjackal has quit IRC | 09:52 | |
| *** kjackal has joined #openstack-infra | 09:53 | |
| *** yamamoto has joined #openstack-infra | 09:59 | |
| *** yamamoto has quit IRC | 10:04 | |
| *** elod has quit IRC | 10:19 | |
| *** kjackal has quit IRC | 10:32 | |
| *** kjackal has joined #openstack-infra | 10:40 | |
| *** dciabrin_ has joined #openstack-infra | 10:53 | |
| *** dciabrin has quit IRC | 10:55 | |
| *** elod has joined #openstack-infra | 11:08 | |
| *** markvoelker has joined #openstack-infra | 11:30 | |
| donnyd | sean-k-mooney: that makes absolutely zero sense. | 11:34 |
| *** markvoelker has quit IRC | 11:35 | |
| fungi | cmurphy: see my inline comments | 11:58 |
| *** nicolasbock has joined #openstack-infra | 12:13 | |
| *** kjackal has quit IRC | 12:36 | |
| *** kjackal has joined #openstack-infra | 12:40 | |
| *** gfidente has quit IRC | 13:04 | |
| *** bobh has joined #openstack-infra | 13:05 | |
| *** bobh has quit IRC | 13:10 | |
| *** kjackal has quit IRC | 13:20 | |
| donnyd | Soooo I am getting the electrical for my generator wired in today. I will need to scale FN down to zero. I would have gladly given more notice had I gotten it myself | 13:31 |
| fungi | good luck with the rewiring! | 13:31 |
| * fungi desperately needs to get a home generator and ups wall installed | 13:32 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Scaling FN down to 0 for backup gen install https://review.opendev.org/680805 | 13:32 |
| donnyd | thanks fungi | 13:33 |
| donnyd | I will only be an assistant today. | 13:33 |
| fungi | slightly reduces the risk of electrocution | 13:33 |
| donnyd | LOL, yea... the majority of risk reduction comes from turning off all the power | 13:34 |
| donnyd | Usually with wiring projects i leave it on... but I think its best for nobody to get kilt today | 13:34 |
| fungi | good call | 13:35 |
| donnyd | I will submit a review when FN is back online.. One good part is I will be able to upgrade my edge rtr so hopefully performance is even better. | 13:36 |
| donnyd | Now to go cleanup my garage so the panel can actually be installed | 13:38 |
| donnyd | thanks fungi | 13:38 |
| donnyd | fungi: on a side note those walls are not UPS | 13:38 |
| donnyd | they are more like backup power... | 13:38 |
| donnyd | I talked with Tesla at length on the topic because that is originally what i was gonna do... throw a couple power walls on and never look back | 13:39 |
| donnyd | but the outage length when they come on is too long for electronics | 13:39 |
| donnyd | so yea, more like backup power than UPS | 13:39 |
| donnyd | have a great weekend !! | 13:40 |
| *** dklyle has joined #openstack-infra | 13:40 | |
| clarkb | do we need to disable the swift logs in fn too? | 13:41 |
| fungi | donnyd: ahh, i was thinking more like the wall-size upses in data centers, i wasn't aware tesla was selling residential solutions along those lines | 13:44 |
| donnyd | clarkb: yea | 13:46 |
| donnyd | my power is projected to be out for 8 hours | 13:47 |
| donnyd | clarkb: is that in system-config or porject config | 13:47 |
| donnyd | project-config | 13:47 |
| *** rosmaita has left #openstack-infra | 13:48 | |
| donnyd | fungi: those are too rich for my blood | 13:49 |
| fungi | probably mine too | 13:49 |
| *** jtomasek has quit IRC | 13:49 | |
| fungi | won't know until tomorrow night when i get home what sort of storm damage my disposable income may be going to | 13:50 |
| * donnyd crosses finger fungi home is ok | 13:53 | |
| fungi | but yeah, you basically need to be powering the gear through the battery backup for it to work out for electronics... i guess the tesla power walls rely on something like an ats to switch the mains, and those can take in the 1-3 seconds range | 13:54 |
| fungi | on the other hand, with something like that you can get by coupling it with a high-throughput but short-duration ups | 13:56 |
| fungi | (as long as the ats doesn't fail to cut over) | 13:56 |
| fungi | basically just big line conditioners | 13:57 |
| donnyd | I have powered UPS for everything right now and they can hold it for about 9 minutes | 13:58 |
| donnyd | and in my case the generator would have already been running for 4 of those 9 | 13:58 |
| donnyd | and the price was about 15K less | 13:58 |
| fungi | indeed, sounds like a winner | 13:58 |
| donnyd | clarkb: do i need to do anything for the swift logs | 13:59 |
| clarkb | sorry its early for me, ya we should remove fn from the swift entries in opemdev/base-jobs | 14:00 |
| clarkb | https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml#L19 and the playbooks for base-test and base-minimal | 14:04 |
| clarkb | (I cant push that as I am not at proper computer) | 14:04 |
| openstackgerrit | Donny Davis proposed opendev/base-jobs master: Temp removal of FN for power outage https://review.opendev.org/680807 | 14:08 |
| *** signed8bit has joined #openstack-infra | 14:09 | |
| *** signed8bit has quit IRC | 14:10 | |
| donnyd | done thanks clarkb | 14:10 |
| clarkb | fungi: your recheck for post failure may be fixed by ^ | 14:11 |
| clarkb | if that doesnt merge in a pass we might force merge it | 14:11 |
| fungi | ahh, yep | 14:11 |
| donnyd | hrm, any ideas what the POST_FAILURE is from? | 14:12 |
| fungi | i wonder if zuul could grow some log storage resilience by trying a different backend from rotation on failure | 14:12 |
| donnyd | is that from it being unable to upload the log? | 14:12 |
| clarkb | donnyd: likely yes | 14:13 |
| clarkb | fungi pabelanger hasbeen poking at that, ansible makes it abit weird | 14:13 |
| clarkb | wecould push the cloud selection down into the python and do it there otherwise its difficult aiui | 14:13 |
| pabelanger | yes, I want to make that a little more resilient (upload-logs-swift), for example https://github.com/ansible/project-config/blob/master/playbooks/base-minimal-test/post-logs.yaml#L17 is how we handle it for zuul.a.c, but it still isn't the best setup. Part of the issue is, you cannot use loop until success or block loops in ansible. That needs to be done at the task level, so for upload-logs-swift, | 14:17 |
| pabelanger | we need to do a little refactor to allow it to retry on failure, if we want it to go to another cloud | 14:17 |
| pabelanger | then there is going to be some magic to pop off the list, the bad cloud, to random try the remaining | 14:17 |
| openstackgerrit | Merged opendev/base-jobs master: Temp removal of FN for power outage https://review.opendev.org/680807 | 14:19 |
| *** bobh has joined #openstack-infra | 14:21 | |
| *** virendra-sharma has joined #openstack-infra | 14:22 | |
| *** ralonsoh has quit IRC | 14:24 | |
| *** bobh has quit IRC | 14:25 | |
| *** nicolasbock has quit IRC | 14:33 | |
| donnyd | fungi: lol. now the linters POST_FAIL | 14:33 |
| donnyd | FN is still up, I won't take anything down until all the jobs scale down and are out | 14:34 |
| fungi | roll of the dice | 14:34 |
| corvus | are we expecting post failures? | 14:34 |
| fungi | ahh, so these POST_FAILURE results may be failure to store logs in another provider | 14:34 |
| corvus | 2019-09-07 12:58:44,008 DEBUG zuul.AnsibleJob.output: [e: aaa5abc26053462db5db34bf25931607] [build: 5269795a794346fdb2f2fdddfea652ad] Ansible output: b'fatal: [localhost -> localhost]: FAILED! => {"changed": false, "cloud": "defaults", "msg": "The account is disabled for user: 4aec54448a4d48c79bda8582529e1a27. (HTTP 401) (Request-ID: req-7363b95e-3995-447d-a11d-d3bec4a0d83d)", "region_name": | 14:35 |
| corvus | "BHS1"}' | 14:35 |
| fungi | okay, so this may be a recurrence of the ovh account disablement from earlier in the week? | 14:36 |
| corvus | i'm starting to doubt the premise behind the migration away from the static log server | 14:36 |
| corvus | i also see the same error for gra1 | 14:37 |
| fungi | i think this was the case last time too | 14:37 |
| donnyd | the swift for FN is also still operational | 14:38 |
| fungi | some billing confusion | 14:38 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Stop storing logs on OVH https://review.opendev.org/680808 | 14:39 |
| corvus | fungi, clarkb: ^ instapprove? | 14:39 |
| corvus | maybe force merge | 14:39 |
| corvus | (because, ironically, merging the fn change has increased the chances that will fail) | 14:40 |
| corvus | (and yeah, that's alanis-style irony) | 14:40 |
| virendra-sharma | I have added repo add-apt-repository ppa:deadsnakes/ppa for python3.7, which install python3.7 but I found pip installing package with pip2.7any steps for pip to point pip3.7 ? Because after devstack installation, it automatically point to python3.7 & pip3.7 | 14:42 |
| fungi | corvus: oh! thanks, i was in the middle of trying to fix the merge conflicts to revert teh revert | 14:43 |
| donnyd | it is kinda comical | 14:44 |
| fungi | virendra-sharma: don't invoke `pip` instead you want `pip3` or `python3.7 -m pip` | 14:44 |
| fungi | virendra-sharma: also, devstack questions are probably more on-topic in #openstack-qa | 14:45 |
| *** jcoufal has joined #openstack-infra | 14:46 | |
| openstackgerrit | Merged opendev/base-jobs master: Stop storing logs on OVH https://review.opendev.org/680808 | 14:49 |
| clarkb | want tp followup woth amorin on the thread I started the other day? I'm distracted by the weekend currently | 14:50 |
| fungi | i'm replying now, yes | 14:51 |
| fungi | and reply sent | 14:54 |
| *** jcoufal_ has joined #openstack-infra | 14:54 | |
| *** jcoufal has quit IRC | 14:57 | |
| *** jcoufal has joined #openstack-infra | 14:59 | |
| *** markvoelker has joined #openstack-infra | 15:00 | |
| *** jcoufal_ has quit IRC | 15:02 | |
| *** rfolco has quit IRC | 15:04 | |
| *** markvoelker has quit IRC | 15:05 | |
| donnyd | fungi should we recheck the FN job now that OVH is out | 15:06 |
| fungi | donnyd: ayup, i see you have now | 15:13 |
| *** factor has quit IRC | 15:15 | |
| *** icarusfactor has joined #openstack-infra | 15:15 | |
| amorin | hey guys | 15:20 |
| fungi | howdy amorin! | 15:21 |
| fungi | got my e-mail, i guess | 15:21 |
| *** jcoufal has quit IRC | 15:21 | |
| amorin | yup | 15:21 |
| amorin | true, it started again | 15:21 |
| fungi | i'm about to disappear to get lunch but corvus or clarkb may be around to check stuff if needed | 15:22 |
| amorin | no problem, i will try to escalate the issue | 15:22 |
| amorin | internaly | 15:22 |
| fungi | much appreciated! | 15:22 |
| fungi | i'll likely be back in a little while too | 15:22 |
| *** jcoufal has joined #openstack-infra | 15:23 | |
| *** jcoufal has quit IRC | 15:24 | |
| *** factor has joined #openstack-infra | 15:32 | |
| amorin | the guy in charge of billing in our team will be there within an hour | 15:32 |
| amorin | I'll keep you in touch | 15:32 |
| *** icarusfactor has quit IRC | 15:35 | |
| *** dave-mccowan has joined #openstack-infra | 15:38 | |
| *** factor has quit IRC | 15:38 | |
| *** factor has joined #openstack-infra | 15:38 | |
| virendra-sharma | fungi I just added ppa repository and if i use pip3 then i takes pip3.5 (default in ubuntu16) | 15:40 |
| *** dave-mccowan has quit IRC | 15:40 | |
| virendra-sharma | instead i need pip3.7 | 15:41 |
| *** slaweq has quit IRC | 15:45 | |
| openstackgerrit | Merged openstack/project-config master: Scaling FN down to 0 for backup gen install https://review.opendev.org/680805 | 15:48 |
| *** dklyle has quit IRC | 15:49 | |
| *** hongbin has joined #openstack-infra | 16:24 | |
| *** bobh has joined #openstack-infra | 16:27 | |
| amorin | fungi: clarkb I need to go now but the guys in charge of this should do the job before the end of the day (french time). I'll check tomorrow | 16:34 |
| amorin | sorry again for this | 16:34 |
| clarkb | thank you for your help | 16:36 |
| *** virendra-sharma has quit IRC | 16:47 | |
| *** hongbin has quit IRC | 16:56 | |
| *** bobh has quit IRC | 17:06 | |
| *** bobh has joined #openstack-infra | 17:08 | |
| *** zxiiro has quit IRC | 17:10 | |
| *** zxiiro has joined #openstack-infra | 17:11 | |
| *** bobh has quit IRC | 17:12 | |
| *** prometheanfire has quit IRC | 18:01 | |
| *** prometheanfire has joined #openstack-infra | 18:02 | |
| mnaser | i expect us to be able to bring back our logging by the end of this week because we're still trying to rebalance things and make sure the load can be handled i guess | 18:07 |
| clarkb | cool. I think we've made a lot of progress on our side to be friendlier to ceph | 18:10 |
| clarkb | the container sharding is by build uuid and into 4096 containers now so should be spread out really well and we dropped the root ara reports | 18:10 |
| *** diga has quit IRC | 18:22 | |
| openstackgerrit | Paul Belanger proposed openstack/project-config master: Add windmill-ops / ansible-role-elasticsearch https://review.opendev.org/680830 | 18:39 |
| clarkb | pabelanger: any reason to not use one of the popular elasticsearch roles on galaxy? | 18:41 |
| pabelanger | clarkb: mostly because they usually end up doing too much, like repo mgmt, db setup, etc. Also most like an all in one deployment, Plan here is basically role as package manager. But, TBH, I haven't really looked into galaxy, but should | 18:45 |
| *** bobh has joined #openstack-infra | 18:58 | |
| johnsom | FYI, it looks like the RETRY_LIMIT issue continues. Looking at the zuul builds page, there are a bunch of them across most of the projects. | 19:02 |
| johnsom | here is the output from an Octavia job: | 19:02 |
| johnsom | https://www.irccloud.com/pastebin/K3ErvFTC/ | 19:02 |
| johnsom | So far the ones I have seen are running at Rackspace. | 19:03 |
| *** bobh has quit IRC | 19:04 | |
| johnsom | Does it always retry at the same provider? | 19:05 |
| *** bobh has joined #openstack-infra | 19:05 | |
| clarkb | no retries should go to whochever provider can proovidethe node in the queue | 19:07 |
| clarkb | is it failing at tempest again? | 19:07 |
| johnsom | Yeah, in our job. But the builds page shows it happening for unit tests too. | 19:07 |
| *** ociuhandu has joined #openstack-infra | 19:34 | |
| *** markvoelker has joined #openstack-infra | 19:35 | |
| *** markvoelker has quit IRC | 19:40 | |
| *** ociuhandu has quit IRC | 19:45 | |
| fungi | any particular project's unit tests, or lots at random? | 20:04 |
| *** jroll has quit IRC | 20:05 | |
| fungi | looks like keystone has a few today too | 20:06 |
| johnsom | FYI, another failure on a RAX host I had the console open for: | 20:06 |
| johnsom | https://www.irccloud.com/pastebin/Yi6ka9vG/ | 20:06 |
| *** mattw4 has joined #openstack-infra | 20:12 | |
| *** jcoufal has joined #openstack-infra | 20:25 | |
| *** roman_g has joined #openstack-infra | 20:27 | |
| *** jcoufal has joined #openstack-infra | 20:36 | |
| *** dklyle has joined #openstack-infra | 20:37 | |
| *** hwoarang has quit IRC | 20:44 | |
| *** dklyle has quit IRC | 20:44 | |
| *** jcoufal_ has joined #openstack-infra | 20:49 | |
| *** jcoufal__ has joined #openstack-infra | 20:50 | |
| *** jcoufal has quit IRC | 20:51 | |
| *** hwoarang has joined #openstack-infra | 20:52 | |
| *** jcoufal has joined #openstack-infra | 20:53 | |
| *** jcoufal_ has quit IRC | 20:54 | |
| *** jcoufal__ has quit IRC | 20:54 | |
| *** jcoufal_ has joined #openstack-infra | 21:04 | |
| *** ociuhandu has joined #openstack-infra | 21:04 | |
| *** bobh has quit IRC | 21:04 | |
| *** jcoufal has quit IRC | 21:07 | |
| *** jcoufal has joined #openstack-infra | 21:10 | |
| *** bobh has joined #openstack-infra | 21:12 | |
| *** jcoufal_ has quit IRC | 21:12 | |
| *** jcoufal has quit IRC | 21:15 | |
| *** bobh has quit IRC | 21:17 | |
| *** jcoufal has joined #openstack-infra | 21:17 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Add output for mis-configured element scripts https://review.opendev.org/680785 | 21:30 |
| *** ociuhandu has quit IRC | 21:34 | |
| *** ociuhandu has joined #openstack-infra | 21:35 | |
| johnsom | FYI, a retry run of the same test at vexxhost passed fine. | 21:38 |
| *** ociuhandu has quit IRC | 21:40 | |
| *** mattw4 has quit IRC | 21:44 | |
| *** roman_g has quit IRC | 21:54 | |
| *** jcoufal has quit IRC | 22:15 | |
| *** bobh has joined #openstack-infra | 22:29 | |
| *** bobh has quit IRC | 22:33 | |
| *** xenos76 has joined #openstack-infra | 22:34 | |
| *** xenos76 has quit IRC | 23:05 | |
| fungi | the biggest difference with rackspace is that it's running on xen. could that test be sometimes tickling a hypervisor bug and crashing the guest hard? | 23:09 |
| *** bobh has joined #openstack-infra | 23:15 | |
| *** goldyfruit has joined #openstack-infra | 23:29 | |
| *** goldyfruit_ has joined #openstack-infra | 23:35 | |
| *** goldyfruit has quit IRC | 23:37 | |
| *** dklyle has joined #openstack-infra | 23:48 | |
| *** logan- has quit IRC | 23:59 | |
| *** goldyfruit_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!