Tuesday, 2018-10-16

*** eernst has quit IRC00:00
mordredclarkb: is etherpad.o.o runniner older code?00:01
clarkbmordred: no, reading the docs this is still supported00:01
mordred*awesome*00:01
clarkbhttp://etherpad.org/doc/v1.7.0/#index_custom_static_files on current etherpad00:01
clarkbits just you have to create the dir I guess00:01
clarkbI'll update the puppet00:01
*** sthussey has quit IRC00:02
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] ubuntu element - ignore missing openstack repos  https://review.openstack.org/61043000:02
*** rlandy is now known as rlandy|bbl00:03
ianw#status log nl04 in emergency while I fiddle with ovh-gra quotas to see what works00:05
openstackgerritClark Boylan proposed openstack-infra/puppet-etherpad_lite master: Ensure the custom js dir is created  https://review.openstack.org/61075500:05
openstackstatusianw: finished logging00:05
clarkbianw: mordred ^ I am hopeful that is the last fix we need for new etherpad lite00:05
clarkbI actually need to go look at making dinner nowish. But I think you can safely approve that and the server will puppet overnight sometime and I'll get back to it in the morning00:06
ianwclarkb: cool, will have a look00:06
clarkbI wonder how this ends up working with the rspec tests. Maybe we use older version there that does work?00:09
clarkbprobably worth looking into00:09
dmsimardclarkb: gerritbot fixed for #ara, thanks !00:12
*** agopi has quit IRC00:29
*** agopi has joined #openstack-infra00:30
*** jamesmcarthur has joined #openstack-infra00:39
*** longkb has joined #openstack-infra00:40
*** smarcet has quit IRC00:45
smcginnisFixed for #openstack-cinder too!00:46
*** jamesmcarthur has quit IRC00:47
*** ssbarnea|bkp2 has quit IRC00:48
*** ssbarnea has joined #openstack-infra00:48
*** agopi_ has joined #openstack-infra00:49
*** agopi has quit IRC00:50
*** jamesmcarthur has joined #openstack-infra00:51
*** jamesmcarthur has quit IRC00:52
*** jamesmcarthur has joined #openstack-infra00:52
*** smarcet has joined #openstack-infra00:53
*** smarcet has quit IRC00:56
openstackgerritMerged openstack-infra/gerritbot master: Don't autojoin channels  https://review.openstack.org/61074501:05
*** rkukura has joined #openstack-infra01:06
openstackgerritJeremy Stanley proposed openstack-infra/zuul-website master: Add three more case studies  https://review.openstack.org/61075701:06
openstackgerritJeremy Stanley proposed openstack-infra/zuul-website master: Add three more case studies  https://review.openstack.org/61075701:10
*** slaweq has joined #openstack-infra01:11
*** slaweq has quit IRC01:16
*** mrsoul has joined #openstack-infra01:26
tonybWhere in system-config? can I see the size (RAM/Disk) of the VMs used for jobs?01:28
*** rlandy|bbl is now known as rlandy01:28
clarkbtonyb https://docs.openstack.org/infra/manual/testing.html are the docs on test envs01:30
tonybclarkb: Thanks01:31
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Revert "Add ensure-python role"  https://review.openstack.org/61075801:32
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Revert "Add ensure-python role"  https://review.openstack.org/61075801:33
gmannclarkb: AJaeger yes, this will cover the both scenario. 1. testing openstack with lib master version by lib-forward-testing and lib-forward-testing-python3 2. testing openstack with lib pypi version in integrated tempest-full job01:35
pabelangerAJaeger: frickler: coreycb (I think), I'd like to consider a revert of ^. It seems to duplicate functionality we already support with bindep01:35
*** openstackgerrit has quit IRC01:35
pabelangerclarkb: also, sine you on original review01:36
ianwclarkb: are we tracking the "openstack.cloud.exc.OpenStackCloudCreateException: Error creating server: 083df9db-4db9-447f-a9df-73bf26a0dc9d" in ovh?01:37
ianwbhs1 is hitting this all the time01:37
*** yamamoto has quit IRC01:41
*** yamamoto has joined #openstack-infra01:41
clarkbianw not that I know of.01:41
fungipabelanger: how is it a duplicate?01:42
fungiare you suggesting that every project which wants to run python 3.7 tests with a tox job needs an entry in their bindep.txt so that a job which says it's going to run tests under python 3.7 will actually have python 3.7 available?01:43
pabelangerfungi: it provides the same functionality that bindep could, and our tox jobs parent to unittests, which include bindep by default01:44
pabelangerfungi: yes01:44
pabelangerI think that is a sane approach01:44
fungii think it's not sane at all01:47
pabelangerwhy? we do it today for python3-devel01:47
pabelangerfor example: https://git.zuul-ci.org/cgit/zuul/tree/bindep.txt#n3201:48
pabelangeri don't think it is a good idea to have 2 places to install python from01:48
fungiwhich is why i had encouraged coreycb to abandon mass patching to add python3.7 entries to the bindep.txt file of every single python-based project we host01:48
fungibindep.txt is not a tool to control our ci system01:48
fungiit's instructions to a developer to let them know what unusual packages they should install in their development environment to be able to test changes they're writing01:49
pabelangerthat is now how I understood it, otherwise, we can just have a global bindep_fallback.txt again for tox specific things over pre ansible role package installs01:49
fungiif they choose to run `tox -e py37` i think 1. they ought to know they need a python 3.7 interpreter installed, and 2. the error message tox provides if they don't is quite clear01:49
fungithis isn't a global bindep fallback01:50
fungiit's like deciding that our pypi upload jobs should install twine01:50
fungiwe don't make every project declare that there's a ci job which will need twine installed01:51
fungisimilarly we should make every single project indicate that to run unit tests under python 3.7 you need python3.7 installed01:51
fungier, shouldn't01:51
pabelangerif we don't want to mass update bindep.txt file, i can understand that, we could then just installed python3.7 via DIB elements, like we do for python2.7 and python3.6 today.01:52
fungidistros provide packages for things like tox. why do we have an ensure-tox role instead of just making all projects put tox in their bindep.txt?01:52
pabelangerThe issue I have with the patch, is we now have 2 places where somebody can define a python package to be used.  For reasons I thought, bindep.txt was for the CI system to manage OS package dependencies, not expanding out roles in also include that01:53
*** nhicher has quit IRC01:54
fungiour ci jobs take advantage of the fact that projects often declare additional distro packages they need in a bindep.txt file, but any time we're adding things to bindep.txt just to get a ci job working chances are we're doing it wrong01:54
fungiif someone puts tox in their bindep.txt, then there are two places they can define a tox package to use01:55
pabelangerI actually disagree with that statement, bindep.txt at least for how we are using it today in ansible-network, is a good place to get CI depends and application dependencies installed. Otherwise, I am not sure why we have python3-dev in bindep.txt today.01:56
fungii saw ensure-python as a potential future reusable role where we can provide ways to get interesting versions of python onto multiple distros (not just via distro packages, that one happens to be the initial case only)01:56
fungihaving python3-dev, which is not installed by default when you install a python3 interpreter, is needed if the project is using any python dependencies built from sdist and linked against libpython3. it's not necessary for every project01:58
fungiprojects with only pure python dependencies shouldn't need it, for example01:59
fungialso, this is proposed to zuul's stdlib, so #zuul is likely a better place to be having this debate. it's not an openstack-only use case02:00
clarkbthe reason to have python3-dev in bindep is to say this needs python3 to run. To test with a specific version of python to ensure coverage of that version of pythin the job should handle that02:01
clarkbthey are different concerns02:01
*** rh-jelabarre has joined #openstack-infra02:02
*** armax has quit IRC02:03
ianwi'd always considered it binary dependencies of the projects in requirements.txt02:04
pabelangerreplied to orginal patchset02:06
pabelangerianw: yes, that is how I understood it too.02:07
fungiit also seems strange to me, taken from the main bindep use case which is communicating distro package dependencies to the developer, that we would list that they need python3.7 but not any other python versions just because there happens to be some platform somewhere which preinstalls one of those other versions02:07
pabelangerDid we discuss updating our images to provide python3.7?02:08
fungiso a python3.7 bindep entry for ubuntu bionic does tell them that they might want python3.7 installed to run py37 tox tests, but doesn't tell them that to run py35 tox tests they need to be running on ubuntu xenial02:08
clarkbthey are complimentary things. If I install zuul I want python3 installed via bindep for me. If I test zuul and I want to test python3.7 the job should verify and ensure that02:08
clarkbpabelanger: we dont need ti its available on bionic02:09
*** hongbin has joined #openstack-infra02:09
pabelangerI'm okay to discuss it more in #zuul or ML, I'm not really a fan of growing out ensure-python to support all the possible distros to be honest.  But agree, if updating all the projects bindep.txt doesn't scale well, maybe going back to centralized bindep is an option02:14
clarkbbut centralized bindeo doesnt really solve this either?02:14
clarkbthe problem is there are 3 pythons on bionic02:14
clarkb2.7, 3.6 and 3.702:14
pabelangerjust noticed we have a new task in ansible-network jobs, and at first glance looked to duplicate bindep functionality02:14
fungiand as i said earlier, ensure-python can certainly be extended to do other things like install unusual python versions via pyenv or make&&make install, the first case for it just happens to be providing a way to declare a nonstandard python interpreter distro package for jobs which need thaht version02:14
clarkbto run unittests in a specific version you want to ensure that version is the one being used02:15
clarkbthat seems out of the scope of bindep which is instead: give me a python302:15
*** rh-jelabarre has quit IRC02:15
fungii had hoped that something along the lines of ensure-python was already existing in ansible galaxy, but that turns out to not be the case02:16
pabelangerconfused why you could be more specific in bindep profile, for python36 environment: eg: bindep_profile: test python36'02:17
*** diablo_rojo has quit IRC02:17
clarkbpabelanger: because then you have to standardize arbitrary stirngs across hundreds of repos02:17
clarkband then mordred makes a change that conflicts with non standard strings and things break02:17
clarkbsee the "build" profile02:18
pabelangerclarkb: sure, but wasn't that the idea of pushing bindep or jobs into projects, we'd make those changes across all the repos vs central location02:18
fungiand that gets back to abusing bindep to drive the ci system rather than taking advantage of the fact that projects are using bindep and installing the things they tell their developers they should install02:18
*** jamesmcarthur has quit IRC02:18
clarkbpabelanger: its still entirely that way with ensure-python02:19
clarkbits config per project that asserts a value02:19
*** rkukura has quit IRC02:19
pabelangerI guess, I just don't see putting CI requirement into bindep as abusing bindep. We do think it abuse, we need to document that correctly.02:21
clarkbIm not sure its abause as much as impossible to police as we've already proven with "build"02:22
promehteanfiresomeone mind taking a second look at this for loop, getting a perm issue02:22
promehteanfirehttp://logs.openstack.org/68/604268/15/check/dib-dsvm-functests-python2-ubuntu-trusty-extras/6fe000c/job-output.txt.gz#_2018-10-16_01_19_53_06970702:23
clarkband so if a jobs reason to exist is to test with a specific version of python it is reasonable to assert that in the job02:23
promehteanfireah, redirects don't work in sudo (not that way)02:23
ianwso setting max-servers to 1 in gra, it already seems to have decide to launch about 5 nodes02:24
fungiwhen i say abusing, i mean relying on bindep to communicate things to our ci system without considering what it's communicating to the primary audience of the tool and the bindep.txt file in a repo02:26
*** openstackgerrit has joined #openstack-infra02:26
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds  https://review.openstack.org/60426802:26
promehteanfireone of these days02:26
*** annp has joined #openstack-infra02:26
*** jamesmcarthur has joined #openstack-infra02:27
fungibut also, when there's an entry you expect to be present in the bindep.txt file of, say, every single python-based project, chances are it's not one you need to communicate to developers and so only noise we could be solving by making jobs do what is obvious02:28
fungi(the ensure-twine role as an example)02:29
promehteanfirequestion, if I can support all versions of python, should I? (in the image)?02:29
promehteanfireatm, I think it's hard coded to py35(only), but I could change it to anything else02:30
clarkbpromehteanfire: I think we'd like to avoid baking that into the image but have some python2 stuff for historical reasons02:30
clarkbso making it possible to install arbitrary pythons without installing them all might be a good thing?02:30
clarkbpabelanger: is there a specific concern with allowing jobs ti specify their needs directly?02:31
clarkbone I could see is not testing that bindep works generally (but functional or integrarion tests might be better for that?)02:32
promehteanfirethis is how I install the pythons people want02:32
promehteanfirehttps://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/gentoo/pre-install.d/02-gentoo-04-install-desired-python02:32
promehteanfireit's not quite so easy as just use bindep for me at least02:32
pabelangerclarkb: just bindep concern, so far on ansible-net side were looking to leverage bindep.txt for all dependencies. But doesn't break anything, was just surprised by the new task02:33
pabelangerI think I was just more surprised to see the change, I missed the discussion02:34
promehteanfireit's actually a little bit better than that now, though py37 will need similiar workarounds02:36
promehteanfirehttps://review.openstack.org/#/c/608576/4/diskimage_builder/elements/gentoo/pre-install.d/02-gentoo-04-install-desired-python02:36
clarkbpabelanger: and using python specific bindep envs for a complete distros x release x python ver matrix?02:36
clarkbI mostly want to avoid making the decsion to add a new python version to the test matrix complicated for individual projects and require them to have more than a generic python3 dep02:37
clarkbif you want to add more you can but not sure it should be required02:37
pabelangerclarkb: not yet, we are still on python2.7 and python3.6, we skipped python3.5. However, agree once next fedora comes out, we too have to do the python3.7 dance02:38
pabelangerfor that, i was planning more specific bindep_profiles02:38
*** felipemonteiro has joined #openstack-infra02:42
fungipabelanger: the ensure-twine role is in the zuul-jobs repo, not the project-config repo02:43
fungiwe also have ensure-babel, ensure-sphinx...02:43
pabelangerfungi: yah, was trying to say we don't have a shared job today that is using them in zuul-jobs, only in ozj02:44
pabelangererr, project-config02:44
fungi(these are things which _could_ be installed via listing them in bindep.txt or test-requirements.txt or whatever, but makes more sense for jobs which need those to make sure they get installed rather than relying on all the projects to dtrt)02:45
fungipabelanger: i expect the only reason jobs in zuul-jobs don't use ensure-twine is because the credentials have to be provided in the same repo as the playbook using them02:47
fungidue to zuul's safety measures around secrets handling02:47
pabelangerYup! Think that too. This is an interesting topic, because the tox jobs really are our first jobs that we have shared between 3 zuul systems, that I know of, openstack-infra, rdoproject and ansible-network. And really a first for sharing job content in this CI context :)02:48
pabelangerjust to throw another angle into this topic, this could also have been handled in tools/test-setup.sh also02:50
fungiyep02:50
clarkbI actually like that the job that says I use python 3.7 actually ensures that02:51
clarkbit isnt left to some external system to make that true02:51
clarkband the change as written shouldve worked for everyone else?02:51
clarkbthus didnt break anyone right? it only did the ensuring on platforms it knew how to do that for02:52
fungibut again, i think any logic or information which is applicable to a broad class of projects is not something we should be duplicating in every one of them. tools/test-setup.sh as a pattern was meant for performing unusual steps unique to a particular project or small set of projects02:52
clarkb++02:52
pabelangerclarkb: we are projected because python_version isn't set however, once we do set it, it will break for us, because that package syntax isn't valid for fedora or centos02:53
fungitools/test-setup.sh was, at least initially, for prepopulating test database permissions and adding nonstandard filesystems02:54
clarkbpabelanger: but doesnt it guard against unsupported distros? I thought it did02:54
pabelangerclarkb: no02:54
clarkbin any case it isnt going to break anyone running tox already02:54
clarkbit is opt in02:54
pabelangerit will actually fail if we set python_version with fedora02:54
fungiit guards by expecting you to test the change defining a python_version value02:55
pabelangerit should be updated to have when statement for specific distro it supports, ubuntu according to docs02:55
fungithat seems like a reasonable addition, i agree02:55
clarkbor do thr first found include and have a default that noops02:56
*** carl_cai has quit IRC02:56
fungikeep in mind it will also break similarly if you declare a nonexistent python_version for the distro on which you're testing02:56
clarkbI find that easier to understand when its distro specific behavior02:56
fungilike, set python_version=3.5 in a job for an ubuntu-xenial node and it will break02:57
clarkblooks like fedora just has "python3"02:57
clarkbso you avoid this class of problem entirely02:57
fungier, i mean set python_version=3.5 in a job for an ubuntu-bionic node02:57
fungiclarkb: zaneb was saying that on fedora he has packages available for 3.4, 3.5, 3.6 and 3.702:58
clarkbfungi: those dont appear to be in the main repo according to a quick package search02:58
clarkbbut Im on a phone and dont have dnf available to give a concrete answer02:58
fungialso possible i misinterpreted him02:59
fungion debian i can get a few via packages, but find it easier to just compile them all from source instead and have a symlink farm in ~/bin i use03:00
clarkbaha found them03:01
clarkbpython36 and python37 with no punctuation03:01
pabelangerpython37, python3 on fedora-2803:01
clarkbya 36 is only a package if python3 isnt 3.603:02
*** ramishra has joined #openstack-infra03:02
fungiso we could use some string mangling to strip out the . in the fedora logic branch03:02
*** jamesmcarthur has quit IRC03:02
clarkbso its actually more complicated on fedora :/03:02
fungibut is the python3 package installed by default on fedora?03:03
promehteanfireiirc it's been the default for a few releases in fedora03:03
*** zhangfei has joined #openstack-infra03:03
pabelangerfungi: it only is because of DIB python dependency03:03
pabelangerfedora-cloud images, is it missing IIRC03:03
pabelangerpromehteanfire: oh, I need to double check then03:04
fungiagain, considering this from an opt-in perspective, you'd only need to set python_version if calling the job on a platform for which that interpreter is provided from a nonstandard package03:04
fungior rather one not already installed by default03:04
promehteanfireI'm not sure if they went py3-only like I'm doing, but I think they can03:05
pabelangerlast I looked, we need to update pip-and-virtualenv element to support single python version. It is 2 by default03:05
fungione addition which would probably be sane is to check first for the existence of a pythonX.Y executable in the environment's patyh03:06
promehteanfireit'd be nice if we could support the distro's version of pip at least for init (pip can upgrade itself to a constraint if needed)03:07
promehteanfirethat'd also allow us to constrain pip03:07
*** bhavikdbavishi has joined #openstack-infra03:07
promehteanfirefungi++03:07
pabelangerpromehteanfire: we're actually planing on remove pip from our images shortly for a use case. To make sure everything is packaged properly for testing in ansible. Not looking forward to that fun03:08
fungiwell, pip upgrading itself from a distro package means pip and the distro package manager fighting over some of the same files03:08
*** jamesmcarthur has joined #openstack-infra03:08
promehteanfirefungi: true, but that generally happens anyway if using pip systemwide03:09
* promehteanfire wishes that wasn't the case and venvs were used for everything03:09
fungiyeah, i shudder at every case of that we're propagating03:09
promehteanfireI was told ymmv, expect breakage, when I showed the gentoo python team what openstack does with pip (system wide)03:10
promehteanfirethere's a reason we disabled system-wide pip by default for our pip package03:10
clarkbthe debian split with /usr/local addresses the vast majority if problems though03:11
clarkbunfortunately this sanity was not embraced more globally03:11
*** slaweq has joined #openstack-infra03:11
promehteanfireiirc we have site and dist packages in $PYTHON_LIBDIR03:11
fungiwell, and then as a result pip grew the desire to uninstall packages outside its install path03:11
pabelangerI thought we fixed most of our system-wide pip usage when pip10 came out03:11
pabelangeror was that just for devstack03:12
clarkbpabelanger: no interest for that died out03:12
clarkbwe whack a moled the problems03:12
pabelangerclarkb: Oh, what did we end up doing?03:12
clarkband addressed individual issues03:12
clarkbwe just stopped installing system and pip packages that conflicted aiuu03:12
pabelangerah03:12
clarkbnew conflicts could show up tomorrow03:13
pabelangerack03:13
promehteanfires/could/will03:13
pabelangerpromehteanfire: ++03:13
clarkbon debian it doesnt even fight over the bin entries03:14
fungibasically the split between distro packages in usr and pip using usr/local on debian mostly solved the problems, but because red hat stuck to having pip and distro packages use the same tree and files were constantly overwritten when mixing the package managers, pip maintainers decided they needed to assume all python packages on the system were fair game for pip to try to uninstall03:14
*** bobh has joined #openstack-infra03:14
*** rlandy has quit IRC03:15
fungiregardless of distro03:15
*** slaweq has quit IRC03:16
* fungi really needs to get some sleep. more cleaning awaits tomorrow. night all!03:16
promehteanfireheh, ya, pip freeze shows all the distro stuff03:16
promehteanfirefungi: nn03:16
promehteanfiredon't remove this one     portage==2.3.4903:16
*** jamesmcarthur has quit IRC03:19
clarkbhttps://review.openstack.org/#/c/610755/ is the fix for etherpad-dev puppeting if anyone hastime to review03:23
clarkbIm off to enjoy my evening now03:23
*** bobh has quit IRC03:31
*** zhangfei has quit IRC03:32
*** jamesmcarthur has joined #openstack-infra03:45
*** hongbin has quit IRC03:48
*** Bhujay has joined #openstack-infra03:51
*** Bhujay has quit IRC03:52
*** lpetrut has joined #openstack-infra03:52
*** Bhujay has joined #openstack-infra03:52
*** bhavikdbavishi has quit IRC03:54
*** zhangfei has joined #openstack-infra03:56
*** zhangfei has quit IRC03:58
*** zhangfei has joined #openstack-infra03:58
*** felipemonteiro has quit IRC04:04
*** Bhujay has quit IRC04:06
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: enable caching for gentoo builds  https://review.openstack.org/60426804:17
*** agopi_ is now known as agopi04:18
*** ykarel has joined #openstack-infra04:22
*** zhangfei has quit IRC04:28
*** lpetrut has quit IRC04:34
*** bhavikdbavishi has joined #openstack-infra04:35
*** jamesmcarthur has quit IRC04:44
*** bhavikdbavishi has quit IRC04:52
*** bhavikdbavishi has joined #openstack-infra04:52
promehteanfirewhoa, zuul got updated05:09
*** Douhet has quit IRC05:09
*** rkukura has joined #openstack-infra05:21
*** apetrich has quit IRC05:40
promehteanfirefinally PASS: Element gentoo, test-element: build-succeeds05:41
promehteanfiredebian minimal fails, but who uses that anyway :P05:42
*** Douhet has joined #openstack-infra05:44
*** janki has joined #openstack-infra05:47
*** dpawlik has joined #openstack-infra05:55
*** dpawlik has quit IRC06:00
*** apetrich has joined #openstack-infra06:00
*** chkumar|off is now known as chandankumar06:05
*** dpawlik has joined #openstack-infra06:11
*** slaweq has joined #openstack-infra06:11
*** dpawlik has quit IRC06:12
*** dpawlik has joined #openstack-infra06:12
AJaegerconfig-core, could you reveiw https://review.openstack.org/610385, please? dependency is approved but not merged yet - I'll +A once ready.06:16
*** slaweq has quit IRC06:16
openstackgerritMerged openstack-infra/project-config master: Reduce gate window floor to 10  https://review.openstack.org/61072406:18
*** apetrich has quit IRC06:23
*** noama has joined #openstack-infra06:24
openstackgerritIan Wienand proposed openstack-infra/project-config master: Disable ovh-gra1  https://review.openstack.org/61088506:35
dpawlikAJaeger: do you still have problems with gra1?06:37
dpawlikianw: you still have problems with the ports?06:42
*** apetrich has joined #openstack-infra06:44
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: run-test-command: support list in test_command variable  https://review.openstack.org/61088806:46
*** e0ne has joined #openstack-infra06:48
AJaegerdpawlik: we had last night06:52
AJaegerdpawlik: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-10-15.log.html#t2018-10-15T05:08:3906:52
AJaegerdpawlik: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-10-15.log.html#t2018-10-15T20:11:5906:52
dpawlikAJaeger: ack, I will ask run team06:55
*** slaweq has joined #openstack-infra06:56
AJaegerdpawlik: thanks!06:57
dpawlikAJaeger: just ping me when are some problems with infra ;)06:57
*** aojea has joined #openstack-infra06:57
AJaegerdpawlik: will try - let's inform ianw, since he's an infra-root ^, I expect ianw to be  a better contact06:58
ianwdpawlik: yeah, gra1 is not happy06:59
ianwdpawlik: i don't really know the best way to use storyboard for this, but there's tasks for gra1 and bhs107:00
dpawlikianw:07:00
ianwhttps://storyboard.openstack.org/#!/story/200409007:00
dpawlikianw: that regions have been updated to newton. They are little big so they are also problematic07:01
*** rcernin has quit IRC07:01
dpawlikianw: but we still working on it. It should be better, but if its still not working, I will talk with other teams07:01
ianwdpawlik: yeah, it's been ongoing.  gra1 -- i set us to just launch one node in there and it constantly failed, and then leaked ports07:02
ianwthere are id's in that storyboard which might help if someone wants to look at logs07:02
ianwoh the bhs1 side, we have a lot of hosts in ERROR with "No valid host was found. There are not enough hosts available."07:04
*** hashar has joined #openstack-infra07:05
*** jtomasek has joined #openstack-infra07:05
*** ginopc has joined #openstack-infra07:08
*** bhavikdbavishi1 has joined #openstack-infra07:14
*** bhavikdbavishi has quit IRC07:16
*** bhavikdbavishi1 is now known as bhavikdbavishi07:16
*** jpena|off is now known as jpena07:20
*** lpetrut has joined #openstack-infra07:20
*** quiquell|off is now known as quiquell07:24
*** lpetrut has quit IRC07:25
openstackgerritIan Wienand proposed openstack/diskimage-builder master: ubuntu: Add option to ignore missing mirror components  https://review.openstack.org/61043007:28
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Set EPEL mirror during openstack-ci-mirrors  https://review.openstack.org/60916907:28
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Provide for an EPEL mirror during build  https://review.openstack.org/60883407:28
*** markvoelker has quit IRC07:29
*** markvoelker has joined #openstack-infra07:29
*** bhavikdbavishi has quit IRC07:32
*** markvoelker has quit IRC07:34
*** aojea has quit IRC07:42
*** gfidente has joined #openstack-infra07:45
*** tosky has joined #openstack-infra07:52
*** icey has quit IRC07:52
*** icey has joined #openstack-infra07:52
*** bhavikdbavishi has joined #openstack-infra07:53
*** kjackal has joined #openstack-infra07:53
*** roman_g has joined #openstack-infra07:55
openstackgerritMerged openstack-infra/project-config master: Disable ovh-gra1  https://review.openstack.org/61088507:59
*** jpich has joined #openstack-infra08:00
*** dtantsur|afk is now known as dtantsur08:03
*** gnuoy has quit IRC08:10
*** bhavikdbavishi has quit IRC08:14
*** bhavikdbavishi has joined #openstack-infra08:15
quiquellpabelanger: Have fedora28 nodes at zuul run have python3 ?08:17
*** bhavikdbavishi has quit IRC08:19
*** bhavikdbavishi has joined #openstack-infra08:27
*** markvoelker has joined #openstack-infra08:30
*** priteau has joined #openstack-infra08:31
*** zhangfei has joined #openstack-infra08:33
*** e0ne has quit IRC08:37
*** gnuoy has joined #openstack-infra08:41
*** e0ne has joined #openstack-infra08:44
*** ykarel is now known as ykarel|mtg08:51
*** studarus has quit IRC08:52
*** derekh has joined #openstack-infra08:52
openstackgerritStephen Finucane proposed openstack-infra/git-review master: Remove auto-branch name  https://review.openstack.org/61057308:54
stephenfinmordred, fungi: What would one need to do to get +2 rights on openstack-infra/git-review? There's a rather large review queue built up there09:02
*** ykarel|mtg is now known as ykarel09:03
*** markvoelker has quit IRC09:03
*** panda|off has quit IRC09:15
*** Dobroslaw has joined #openstack-infra09:16
*** panda has joined #openstack-infra09:18
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Cleanup node requests that are declined by all current providers  https://review.openstack.org/61091509:25
*** persia has quit IRC09:27
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Use quota handling code for min-ready declines  https://review.openstack.org/61049509:28
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Remove unneeded todo comment  https://review.openstack.org/61049609:28
*** persia has joined #openstack-infra09:28
ssbarneaa bindep question https://ask.openstack.org/en/question/116630/how-to-use-aliases-with-bindep-requirements/09:32
*** sshnaidm|afk has joined #openstack-infra09:32
*** shardy has joined #openstack-infra09:37
stephenfinssbarnea: Pretty sure aliases is a corner case that isn't covered by bindep yet09:44
ssbarneastephenfin: soon we will need something to cover for this, at least for cli (yum/dnf) i was able to get around with wildcards.09:45
stephenfinssbarnea: I imagine it would be an easy feature to add, if you were so inclined?09:45
*** imacdonn has quit IRC09:54
*** imacdonn has joined #openstack-infra09:55
ssbarneayeah, i am not afraid of adding a CR if i find no way to fix a build. at the moment i have impression that I am ok, but this may change soon.09:57
*** longkb has quit IRC09:59
*** ykarel has quit IRC09:59
*** markvoelker has joined #openstack-infra10:00
*** sshnaidm|afk is now known as sshnaidm10:03
*** chrisyang_0660 has quit IRC10:04
*** ykarel has joined #openstack-infra10:08
*** ykarel is now known as ykarel|lunch10:09
*** jamielennox has quit IRC10:11
*** jamielennox has joined #openstack-infra10:13
*** felipemonteiro has joined #openstack-infra10:27
*** markvoelker has quit IRC10:34
*** ykarel_ has joined #openstack-infra10:48
*** snapiri has quit IRC10:50
*** ykarel|lunch has quit IRC10:50
*** jamesmcarthur has joined #openstack-infra10:56
*** ykarel_ is now known as ykarel10:56
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Cleanup node requests that are declined by all current providers  https://review.openstack.org/61091510:58
*** sshnaidm has quit IRC11:00
*** sshnaidm has joined #openstack-infra11:01
*** quiquell is now known as quiquell|lunch11:01
*** shardy has quit IRC11:02
*** electrofelix has joined #openstack-infra11:04
*** dave-mccowan has joined #openstack-infra11:09
*** ansmith has quit IRC11:16
*** e0ne has quit IRC11:18
*** felipemonteiro has quit IRC11:20
*** roman_g has quit IRC11:21
*** roman_g has joined #openstack-infra11:22
*** bhavikdbavishi has quit IRC11:23
*** jpena is now known as jpena|lunch11:29
*** markvoelker has joined #openstack-infra11:31
*** e0ne has joined #openstack-infra11:32
*** quiquell|lunch is now known as quiquell11:37
*** sshnaidm has quit IRC11:38
*** sshnaidm has joined #openstack-infra11:38
*** jistr is now known as jistr|afk11:47
*** nhicher has joined #openstack-infra11:58
hwoarangdirk: have you noticed all the RETRY_LIMIT failures on opensuse-150 images? Error is "No viable v4 or v6 route found to git.openstack.org. The build node is assumed to be invalid"12:00
*** trown|outtypewww is now known as trown12:00
*** rh-jelabarre has joined #openstack-infra12:03
*** markvoelker has quit IRC12:04
*** yamamoto has quit IRC12:04
*** smarcet has joined #openstack-infra12:04
*** yamamoto has joined #openstack-infra12:05
*** yamamoto has quit IRC12:05
*** yamamoto has joined #openstack-infra12:05
fricklerhwoarang: do you have a pointer to a log with that handy?12:06
fricklerhwoarang: also, there is still a held node for opensuse-150 from 11d ago, do you still need that?12:07
hwoarangfrickler: http://logs.openstack.org/56/606056/2/check/openstack-ansible-functional-opensuse-150/48f015a/job-output.txt.gz12:07
hwoarangfrickler: sorry i was on paternity leave and did not have the chance to look more into that. Please release the node and I will let you know if I need one again. Thank you12:07
*** quiquell is now known as quiquell|off12:11
fricklerhwoarang: so your failure might be related to the ipv6 address not being set up on the node, http://logs.openstack.org/56/606056/2/check/openstack-ansible-functional-opensuse-150/48f015a/zuul-info/inventory.yaml has a public v6 address while it is missing here http://logs.openstack.org/56/606056/2/check/openstack-ansible-functional-opensuse-150/48f015a/zuul-info/host-info.opensuse-150.yaml12:11
fricklerhwoarang: I just did a test on the held node and see unbound failing there, too. do you remember whether it worked when you looked at it earlier12:15
fricklerhwoarang: I'm guessing that this might be related to the ksk rollover, unbound.log says: Oct 16 12:12:23 unbound[2689:0] info: validation failure openstack.org. A IN12:16
*** bhavikdbavishi has joined #openstack-infra12:17
fricklerhwoarang: indeed, /var/lib/unbound/root.key shows the new key as "ADDPEND" instead of "VALID". see https://www.unbound.net/root-11sep-11oct.html12:18
*** electrofelix has quit IRC12:18
*** smarcet has quit IRC12:19
*** boden has joined #openstack-infra12:20
*** e0ne has quit IRC12:24
*** electrofelix has joined #openstack-infra12:24
hwoaranghmm12:25
hwoarangnot sure when that regressed12:26
hwoarangor why12:26
hwoarangfrickler: the node i had reserved was for a different issue but it was not failing at this point12:27
*** e0ne has joined #openstack-infra12:27
hwoarangjobs from at least a week ago seem to work fine. Need to investigate i guess but I was hoping dirk may have the magic answer :)12:28
fricklerthe rollover was last thursday, so it would have regressed at that date probably. I tried regenerating the root.key file as mentioned in the unbound advisory above and that seems to resolve the issue. not sure why it would occur in the first place, though, when you have 1.6.8 installed12:28
fricklerhwoarang: could you recheck your job above? I'd hold another node for that, then, in order to crosscheck12:29
dirkhwoarang: 42 is the magic answer12:29
*** Emine has joined #openstack-infra12:30
hwoarangtell me more12:31
dirkfrickler: do you have any pointer ? I'm happy to help debugging just don't know what / where to look12:31
*** felipemonteiro has joined #openstack-infra12:32
fricklerdirk: you could try to set up an instance locally with unbound as forwarding resolver and check whether dnssec works in that case. then we'd have something to crosscheck against our soon-to-be-held instance12:34
*** rlandy has joined #openstack-infra12:35
fricklerdirk: https://www.icann.org/dns-resolvers-updating-latest-trust-anchor has a bit more information about what happened12:36
*** sshnaidm has quit IRC12:36
dirkfrickler: ah, yeah, I heard about the dns key switch. didn't quite connect this12:37
dirkfrickler: so it means the unbound package in leap 15.0 is missing the new root key?12:37
fricklerdirk: not sure yet how this comes about. the new key is present, but in state "ADDPEND", which means that it was added only recently added and will become valid in 30 days.12:39
*** priteau has quit IRC12:39
*** sshnaidm has joined #openstack-infra12:39
fricklerdirk: that issue should only happen with unbound < 1.6.4, though, while you have 1.6.8 installed. hoping to see more on a new held node or maybe if you try locally12:40
*** priteau has joined #openstack-infra12:40
dirkfrickler: I see a root.key in the package, that looks suspicous. can we maybe fix that by releasing an update that includes only the new key?12:41
fricklerdirk: that would be an option probably, yes.12:42
dirkfrickler: I'll try to find somebody for this12:43
toskydid anyone else hit errors when getting SQLAlchemy_Utils-0.33.6-py2.py3-none-any.whl from the internal mirrors?12:45
tosky("Bad magic number for central directory", which I guess means a corrupted file)12:45
*** bhavikdbavishi has quit IRC12:49
openstackgerritCorey Bryant proposed openstack-infra/zuul-jobs master: Limit ensure-python to Debian/Ubuntu use  https://review.openstack.org/61094812:53
dirkfrickler: hwoarang : I submitted a maintenance update: https://build.opensuse.org/request/show/64227412:53
coreycbpabelanger: fungi: clarkb: I saw some comments limiting ensure-python to debian/ubuntu so I added some logic ^^. thanks for the discussion last night, sorry i missed it.12:54
fungicoreycb: great, thanks!12:57
hwoarangthank you dirk12:58
fungistephenfin: clarkb is the ptl, so the one to ask. ssbarnea has helped a bunch recently with getting some of the more obvious git-review fixes triaged and reviewed for the 1.27.0 release just under a month ago but i agree there are still a number that need to be considered and either merged or rejected as out of scope (git-review gets a lot of proposed "features" which aren't really a good fit)13:00
*** kgiusti has joined #openstack-infra13:01
*** bobh has joined #openstack-infra13:04
*** sshnaidm is now known as sshnaidm|afk13:05
mordredstephenfin: I left a comment on that patch13:06
stephenfinta13:06
*** ansmith has joined #openstack-infra13:08
*** sshnaidm|afk has quit IRC13:09
*** Emine has quit IRC13:09
*** efried has joined #openstack-infra13:10
coreycbfungi: clarkb: infra: if you have a sec, can you comment on this please? https://review.openstack.org/#/c/610708/  personally i really would prefer to drop 3.5 in favor of 3.7 unit tests as we originally discussed. but this governance change seems like the current path of least resistance (keep 3.5 and add 3.6 *or* 3.7).13:10
*** yamamoto has quit IRC13:10
*** yamamoto has joined #openstack-infra13:11
*** e0ne has quit IRC13:12
*** dpawlik has quit IRC13:13
*** agopi has quit IRC13:15
*** yamamoto has quit IRC13:15
*** e0ne has joined #openstack-infra13:16
*** dpawlik has joined #openstack-infra13:18
fricklerinfra-root: I looked at the logs regarding my doubly held node yesterday. seem zuul saw an error for the first hold request although nodepool still did hold the node successfully. not sure how to continue with this, though, probably will ignore it unless it happens regularly http://paste.openstack.org/show/732244/13:18
fricklertosky: I just saw a gate failure due to this. maybe some other infra-root can dig into that? http://logs.openstack.org/34/610734/1/gate/requirements-integration/63dc003/job-output.txt.gz#_2018-10-16_13_13_47_68737813:20
*** jistr|afk is now known as jistr13:20
*** mriedem has joined #openstack-infra13:23
fungicoreycb: i think you're caught in the middle of some disagreements on what we should be testing. my position is that we need to complete our transition from xenial to bionic (hence from 3.5 jobs to 3.6 jobs) which people seem to have forgotten we were in the middle of13:24
fricklerfungi: mnaser: there is an old held node for you, too, also for fuentess who doesn't seem to be here, does anybody know the status of that?13:24
fungicoreycb: the intent was never to test 3.5 and 3.6 for a long period of time, only long enough for projects which had been using py35 jobs to make sure py36 jobs were working and set them voting so we didn't have a gap nor a major disruption13:25
coreycbfungi: yes it seems that way. there are a lot of competing thoughts.13:25
fungiadding py37 testing is nice, but perhaps confusing people by happening in the same patches as the py35 removal, as they're separate efforts13:26
coreycbfungi: i agree. if only we could just add py37 and not drop py35, but i don't think infra has resources to support that.13:27
fungii think we've also given the projects who had py35 working long enough to get py36 working, so should be looking to drop py35 asap13:27
Shrewsfrickler: that appears to be a race between when get the child nodes in a tree and when we iterate through those children, getting each individually. We've seen that before in many other scenarios13:27
*** Serhii_Rusin has joined #openstack-infra13:27
*** felipemonteiro has quit IRC13:27
coreycbfungi: do you think temporarily running 3.5, 3.6, and 3.7 unit tests is a no-go?13:29
fricklerShrews: sounds plausible, yes, but not like it could be easily fixed13:29
*** Serhii_Rusin has left #openstack-infra13:30
AJaegercoreycb: we're missing some clouds and are running at production limit - any extra job means slowdown for all of us.13:30
*** dpawlik has quit IRC13:30
fungicoreycb: i don't think it's a no-go, but i think the priorities are the bionic transition (we say in our pti we test on the latest lts), then getting python3 jobs working for teams which haven't switched to python3 yet (it's a cycle goal), then adding py37 jobs (a nice-to-have)13:31
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul master: Handle missing node during hold check  https://review.openstack.org/61096913:32
Shrewsfrickler: i think that ^ will handle it a bit better13:32
coreycbfungi: that's reasonable that the original python3-first goals are priority over py37 jobs13:33
*** Serhii_Rusin has joined #openstack-infra13:33
Shrewsfrickler: *very* easily fixed  :)13:34
coreycbfungi: but if it's not a no-go to support all 3 py3 unit test jobs, then that makes me think i can start adding py37 jobs. and is contrary to AJaeger's comment.13:34
*** agopi has joined #openstack-infra13:36
fungicoreycb: i'm not sure what percentage increase of overall ci resource consumption we're talking about, but i'm guessing not huge since unit tests tend to be fairly quick (compared to functional/integration type jobs) and would only likely make sense for projects which already have voting py36 jobs13:36
*** ociuhandu has joined #openstack-infra13:37
*** jpena|lunch is now known as jpena13:37
*** ociuhandu has quit IRC13:38
*** agopi_ has joined #openstack-infra13:38
coreycbfungi: yes in my experience most unit tests are fairly quick particularly for dependencies. some larger core projects like neutron/nova are not as quick.13:38
*** agopi has quit IRC13:40
*** dpawlik has joined #openstack-infra13:40
coreycbfungi: so perhaps a strategy could be to add py37 jobs (on top of py35 and py36) if and only if py36 unit test jobs are already voting for the project13:40
coreycbof course that would need +1 from infra13:41
fricklerShrews: looks easy indeed, thanks13:41
slaweqfrickler: hi13:41
slaweqfrickler: I have a question, for jobs which runs on Xenial (e.g. neutron-functional) we are using nodeset: devstack-single-node13:42
slaweqfrickler: if I would like to run this job on Bionic, should I use openstack-single-node-bionic instead?13:43
fricklerslaweq: for short term testing, yes. I was just thinking how to move devstack towards bionic in general, though, and one solution for that might be to make devstack-single-node be bionic instead, possibly having devstack-single-node-xenial for jobs that do need xenial explicitly13:45
*** sthussey has joined #openstack-infra13:45
*** eharney has joined #openstack-infra13:45
slaweqfrickler: ok, thx13:46
*** ykarel is now known as ykarel|afk13:50
slaweqfrickler: and what about multinode jobs? e.g  job which now uses ubuntu-xenial-2-node - I don't see ubuntu-nionic-2-node nodeset13:51
*** e0ne has quit IRC13:53
*** janki has quit IRC13:58
openstackgerritStephen Finucane proposed openstack-infra/git-review master: DNM: Test patch  https://review.openstack.org/61097613:58
openstackgerritStephen Finucane proposed openstack-infra/git-review master: DNM: Test patch  https://review.openstack.org/61097613:59
fricklerslaweq: see #openstack-qa13:59
mordredcoreycb, fungi: we could also investigate a combined job - so one that ran "tox -epy35,py36,py37"14:00
*** yamamoto has joined #openstack-infra14:00
mordredwhich would consume fewer overal nodes - and because of shared setup and wheel caching would also take less overall cloud CPU time than three different distinct jobs14:01
toskya unified unit-test job? That would not be bad14:01
slaweqfrickler: thx14:01
*** e0ne has joined #openstack-infra14:01
toskymordred: do you think it would make sense to fold the pep8 job there too?14:01
mordredI'm not sure the subunit2html post-processor would handle that today - so it would be a little bit of work14:01
coreycbmordred: true. for ubuntu it would need to still be 2 jobs with 3.5 on xenial and 3.6/3.7 on bionic but still would save on spin up/down time.14:02
mordredtosky: pep8 reports its errors a bit differently, so it might be a weirder user experience ... but I believe that would be possible if someone chose if we can sort out the error reporting for more than one subunit stream14:02
mordredcoreycb: yah.14:03
*** agopi_ is now known as agopi14:04
openstackgerritDavid Moreau Simard proposed openstack-infra/zuul master: web: Increase height and padding of zuul-job-result  https://review.openstack.org/61098014:04
mordredtosky: WAY back in the day there was a suggestion to write a dynamic unittest class which ran the pep8 stuff as if it were unittests14:04
mordredtosky: nobody ever got around to writing the code to do so though14:04
mordredtosky: otoh - we've got a patch coming to zuul-jobs to process the pep8 output and leave in-line comments about the failures which we'd lose if we folded pep8 in to the subunit stream (most of the subunit failures don't make a ton of sense as in-line comments)14:05
toskyoh14:06
toskyI see14:06
mordredtosky: I should never talk to people when I'm still on my first coffee14:07
toskyO.O14:08
* tosky didn't notice anything weird14:08
*** dpawlik has quit IRC14:11
openstackgerritStephen Finucane proposed openstack-infra/git-review master: Don't set topic when submitting no-topic patches  https://review.openstack.org/61098814:13
*** rkukura has quit IRC14:14
*** munimeha1 has joined #openstack-infra14:19
*** eernst has joined #openstack-infra14:20
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Initialize label statistics to zero  https://review.openstack.org/61099314:23
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Initialize label statistics to zero  https://review.openstack.org/61099314:28
*** ykarel|afk has quit IRC14:29
*** ykarel|afk has joined #openstack-infra14:30
*** eharney has quit IRC14:33
*** eernst has quit IRC14:34
*** Serhii_Rusin has quit IRC14:39
clarkbstephenfin: fungi: after conversations with ssbarnea I'm happy to add a few new cores (maybe stephenfin and ssbarnea to start) if we can agree on some rough starter rules like bug fixes and testing improvements first. Then if we get reliable testing again that we are happy with consider feature changes at that point.14:41
clarkbstephenfin: ssbarnea fungi even then I'm not sure we want to add a ton of features to git review14:41
clarkbits a fairly small predictable and understandable tool today, if needing an update or three to keep up with new gerrit changes14:42
stephenfinclarkb: Yup, Unix philosophy ftw14:43
clarkbssbarnea: "With wilcards both, yum and dnf behave the same: returning an error only if at the end none of the packages is installed. If one of the packages is installed, they will return success." I wish dnf hadn't preserved that behavior from yum :(14:43
*** armax has joined #openstack-infra14:43
clarkbmakes the lives of config management tools difficult because they must install one package at a time to check if all packages install correctly14:43
clarkb(among other confusing results)14:44
*** dpawlik has joined #openstack-infra14:46
openstackgerritClark Boylan proposed openstack-infra/puppet-etherpad_lite master: Bump eplite version in rspec tests  https://review.openstack.org/61100014:47
clarkbanyone willing to review ^ and its parent? THe parent change fixes a bug getting etherpad-dev redeployed on xenial14:47
fungior they have to check back at the end to make sure the packages which were meant to be installed actually are14:47
*** eharney has joined #openstack-infra14:48
*** ykarel|afk has quit IRC14:48
*** ykarel|afk has joined #openstack-infra14:48
*** dpawlik has quit IRC14:50
*** rkukura has joined #openstack-infra14:51
*** fuentess_ has joined #openstack-infra14:51
*** fuentess_ is now known as fuentess14:52
openstackgerritDaniel Lublin proposed openstack-infra/git-review master: Allow choosing which field to use as author when naming branch  https://review.openstack.org/44457414:53
*** felipemonteiro has joined #openstack-infra14:55
*** Emine has joined #openstack-infra14:56
ssbarneaclarkb: (dnf) hehe, well one's bug is someone else's feature14:57
toskyinfra core, as mentioned earlier, I'm hitting an error while getting one file from the mirrors14:57
clarkbssbarnea: indeed14:57
toskySQLAlchemy_Utils-0.33.6-py2.py3-none-any.whl14:57
toskysee http://logs.openstack.org/27/610527/1/gate/sahara-tests-scenario/73bcec5/job-output.txt.gz#_2018-10-16_13_38_57_19691514:57
*** ccamacho has quit IRC14:57
ssbarneaclarkb: but i am happy to hear about better ways of coping with the same issue.14:58
toskyand also http://logs.openstack.org/27/610527/1/gate/sahara-tests-tempest/8adcdff/job-output.txt.gz#_2018-10-16_13_32_16_326018 (different provider)14:58
clarkbtosky: pulling the file locally and running `unzip` on it reports the same error. The files are served from the same AFS filesystem regardless of provider so should be identical across the board15:00
clarkblikely is a corrupt file in this case15:00
*** ccamacho has joined #openstack-infra15:00
clarkblet me see about deleting it15:00
toskythanks15:01
tosky!15:01
*** ramishra has quit IRC15:01
ssbarneaclarkb: regarding git review, i got your point which I would translate as "preserving compatibility is no1 prio". git-review is one of the most used tools around here, so I can imagine what would happen if a new release starts to break our workflows. once approach would be to publish all merges to master as prereleases on pypa. this would make it much easier for (more) people to use "bleading edge" version before we would do a new release.15:01
fungidefinitely easy. we can just make sure to push frequent prerelease tags and our jobs should do the rest15:03
fungiif there's agreement we want to publish pre-releases of git-review i can push a 1.27.1.0a1 tag now15:05
*** rkukura has quit IRC15:05
*** kaiokmo has quit IRC15:08
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added new my affiliations endpoints  https://review.openstack.org/61100615:09
*** kaiokmo has joined #openstack-infra15:10
clarkbI'm getting permission denied rm'ing the sqlalchemy file using my clarkb/admin kerberos account15:10
clarkbwhat am I missing here15:10
clarkbaha aklog15:10
clarkbyay for docs I wrote :)15:10
fungiheh15:10
*** kjackal has quit IRC15:11
clarkbhttps://review.openstack.org/#/c/601855/1/doc/source/afs.rst not merged yet. Debating if I should change the account there to the admin one and maybe I should15:11
*** kjackal has joined #openstack-infra15:12
clarkbI am going to /usr/sbin/vos release -cell openstack.org mirror.wheel.xenialx6415:14
clarkbfungi: ^ that look right to you? and any idea if I need to lock that volume somehow so that our publish jobs don't conflict with me?15:15
openstackgerritMerged openstack-infra/openstackid-resources master: Added new my affiliations endpoints  https://review.openstack.org/61100615:15
* dtantsur sees that the BadZipFile problem is already known15:15
clarkbthat job seems to run in periodic which happens at 0600 so I should be safe doing a release now15:16
clarkbgoing to try it15:16
ssbarneafungi: small nit https://review.openstack.org/#/c/456929/15:16
clarkb"Released volume mirror.wheel.xenialx64 successfully" tosky dtantsur for timestamps after 1517UTC you should see this work now I expect15:17
clarkbwe'll have to see if sqlalchemy utils wheel building is generally corrupted15:17
toskyI remember that something like this happened few days (weeks?) ago15:18
clarkbI'm going to double check quotas and disk usage too15:19
clarkbthough if the write failed due to lack fo disk then we shouldn't release the volume so I don't expect htat is a problem15:19
clarkbwhen we run out of disk the behavior is to stop udpating at all15:19
*** zhangfei has quit IRC15:19
*** chandankumar is now known as chkumar|off15:19
toskyclarkb: one month ago: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-09-17.log.html#t2018-09-17T09:14:5715:19
clarkbya we use 5% of the current quota on that fs15:20
clarkband grafana says we still have disk space on vicepa15:20
toskyinteresting enough, it was the exact same wheel (two minor versions ago)15:21
clarkbtosky: hrm we did recently merge the change to not consume our own wheels to build wheels15:22
toskythis one? https://review.openstack.org/#/c/603064/15:22
clarkbbut yes, suspicious that it is the same package again. My hunch is something about wheel builds for that package are flaky15:22
clarkbtosky: yes that one15:23
toskyianw wrote that immediately after the issue with that wheel15:23
fungiclarkb: lgtm15:25
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Remove airship-armada jobs, as they are all in project  https://review.openstack.org/61101315:25
clarkbtosky: http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/publish-wheel-mirror-ubuntu-xenial/25125c0/python2/master-job.log shows where we built this wheel I think15:27
clarkbtosky: and the retval column says 015:27
clarkbimo this is broken packaging15:27
openstackgerritGraham Hayes proposed openstack-infra/project-config master: Install Pygments before checking README syntax for publish jobs  https://review.openstack.org/61101415:27
clarkbsorry exitval15:27
clarkbtosky: https://github.com/kvesteri/sqlalchemy-utils/blob/master/setup.py#L74 is suspicious to me15:34
clarkbhowever they've set it to false which is what I'd expect it to be if not safe so propbably unrelated15:36
*** sshnaidm has joined #openstack-infra15:37
dtroyerDoes anyone know if we (community) already do any checking of website URLs?  I'm looking for something to do really basic "yup the right page is there" or "the redirect is still working" kinds of things periodically against published docs sites.15:39
clarkbdtroyer: external monitoring? we don't currently run that against any of the sites that I know of. The one check we do with reminders is ssl cert verification and expiry notifications15:41
*** hamzy has quit IRC15:42
toskydtroyer: there is a report of 404 errors for doc pages, AJaeger knows more about it15:42
fungithere is also the whereto checker for docs changes to make sure they don't break relevant redirects15:43
dtroyertosky: ok, thanks.    clarkb: external from the standpoint that it's not on the web server is enough.  I've got 30+ job/repo combinations and don't want to check it by hand forever…15:45
dtroyerfungi: I think that is next on our list of things to understand, we're not doing the versioned repos yet and will need to get deeper there before March.15:45
dtroyerer, versioned docs trees15:46
dtroyerthanks guys15:46
openstackgerritweizj proposed openstack/os-testr master: Update the hacking to latst  https://review.openstack.org/61104815:47
*** felipemonteiro has quit IRC15:49
*** e0ne has quit IRC15:50
*** promehteanfire is now known as prometheanfire16:01
*** ginopc has quit IRC16:02
openstackgerritMerged openstack-dev/pbr master: Add an option to skip generating RELEASENOTES.rst  https://review.openstack.org/58720416:08
*** noama has quit IRC16:10
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Fixed my speaker profile  https://review.openstack.org/61106916:11
prometheanfirednssec stuff for opensuse-150, is that handled?16:13
*** sshnaidm has quit IRC16:13
prometheanfiredirk: dnssec stuff for opensuse-150, is that handled?    https://data.iana.org/root-anchors/16:14
*** hashar is now known as hasharAway16:14
*** quiquell|off has quit IRC16:15
openstackgerritweizj proposed openstack/os-testr master: Update the hacking to latst  https://review.openstack.org/61104816:16
openstackgerritMerged openstack-infra/zuul master: Provide some accounting of node usage in logs  https://review.openstack.org/60585616:17
*** ykarel|afk is now known as ykarel16:18
*** apetrich has quit IRC16:18
*** bhavikdbavishi has joined #openstack-infra16:20
*** d0ugal has quit IRC16:21
openstackgerritClark Boylan proposed openstack-infra/project-config master: Check wheels built are valid zip files  https://review.openstack.org/61107516:22
clarkbtosky: dtantsur ^ fyi I think that may catch it in the future?16:22
*** d0ugal has joined #openstack-infra16:23
toskyI don't know really much about wheels and the process, but I concur that this change should prevent bad zip files16:23
clarkbinfra-root review on https://review.openstack.org/#/c/610755/ much appreciated so I can continue to make progress with the etherpad upgrades16:24
* clarkb finds breakfast16:25
*** gfidente is now known as gfidente|afk16:25
*** jpich has quit IRC16:29
*** apetrich has joined #openstack-infra16:30
*** rlandy is now known as rlandy|biab16:32
*** diablo_rojo has joined #openstack-infra16:33
*** rpioso|afk is now known as rpioso16:36
*** ykarel is now known as ykarel|away16:37
*** eernst has joined #openstack-infra16:42
clarkbinteresting data, the check queue backlog has already at 2 hours and this is after the gate window floor change (which has limited the impact of some integrated gate resets this morning from what I have seen)16:42
*** jpena is now known as jpena|off16:45
*** ykarel|away has quit IRC16:46
corvuswe changed the floor?16:47
clarkbcorvus: yes, pabelanger had suggested it and watching the gate reset many times yesterday I went ahead and pushed the change to half it to 10 yesterday. Looks like it merged overnight16:47
*** panda is now known as panda|off16:48
clarkbconsidering the backlog isn't appreciably better I'm not sure it has had the impact pabelanger thought it would have16:48
openstackgerritMerged openstack-infra/puppet-etherpad_lite master: Ensure the custom js dir is created  https://review.openstack.org/61075516:50
pabelangerclarkb: corvus: looks like AJaeger approved it. But just now also noticing it merged16:54
*** chkumar|off is now known as chandankumar16:57
*** eernst has quit IRC16:58
*** jamesmcarthur has quit IRC17:00
*** jamesmcarthur has joined #openstack-infra17:00
*** jesusaur has quit IRC17:02
clarkbanyway I think we knew it would be a short term experiment. So far data indicates it doesn't change much and we can probably revert17:02
*** dtantsur is now known as dtantsur|afk17:03
*** kopecmartin|ruck is now known as kopecmartin|off17:05
*** hamzy has joined #openstack-infra17:05
*** e0ne has joined #openstack-infra17:08
*** munimeha1 has quit IRC17:10
*** kaiokmo has quit IRC17:17
*** kaiokmo has joined #openstack-infra17:18
*** panda|off has quit IRC17:24
*** panda has joined #openstack-infra17:27
dirkprometheanfire: in general yes, except that we forgot unbound as noone uses that in opensuse17:31
dirkprometheanfire: that update is in the works now17:31
clarkbyay we helped find a bug?17:31
dirkUnfortunately its one of the 'this would have been really helpful, last week'17:32
dirkclarkb: yep17:32
AJaegerpabelanger, clarkb I considered +8 a good time to try the experiment ;)17:32
dirkclarkb: and for your pleasure, it was fixed in tumbleweed weeks ago ;-)17:33
AJaegerconfig-core, could you review https://review.openstack.org/610385 and https://review.openstack.org/610386 , please? The job moved to tempest repo in periodic pipeline, so we can remove it.17:33
openstackgerritMerged openstack-infra/openstackid-resources master: Fixed my speaker profile  https://review.openstack.org/61106917:33
AJaegerclarkb: but I saw quite a few reset in integrated-gate, so that lower number might have helped.17:36
clarkbAJaeger: ya there have been some resets there17:36
AJaegerclarkb: since we prioritize gate pipeline over chck now, I think limiting it might be fine - the gate is too flaky to merge 10 or even 20 changes at once...17:36
AJaegerand we give priority with nodes to it - so, I would not revert directly17:37
*** trown is now known as trown|lunch17:37
clarkbAJaeger: ya I think we can continue to experiment with it. Its just that current data doesn't show it helping much so maybe we should look at other things17:38
fungialso if the gate queues stabilize then the floor becomes irrelevant17:39
AJaegerfungi: yeah...17:40
*** electrofelix has quit IRC17:41
AJaegertosky, dtroyer, http://files.openstack.org/docs-404s/ is the list of 404s that people hit when going to docs.o.o17:41
*** rlandy|biab is now known as rlandy17:46
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Update governance-uc docs publishing  https://review.openstack.org/61111417:51
*** gfidente|afk has quit IRC17:52
AJaegerfungi, could you review https://review.openstack.org/611014 to unblock designate, please?17:55
openstackgerritMerged openstack-infra/project-config master: Remove periodic-tempest-dsvm-oslo-latest-full-master job  https://review.openstack.org/61038517:57
clarkbnew etherpad-dev seems to be working. I am updating dns now as a result17:59
*** e0ne has quit IRC17:59
openstackgerritMerged openstack-infra/zuul-website master: Add three more case studies  https://review.openstack.org/61075718:00
*** olivierb has quit IRC18:02
clarkbhttps://etherpad-dev.openstack.org/p/clarkb-test2 seems to be working from here18:03
*** diablo_rojo has quit IRC18:03
clarkbI'm going to prep for meeting and then look at upgrading etherpad.o.o after the meeting18:05
clarkbI'll delete the old hosts in a few days (to allow local db backups to roll over onto new hosts)18:06
*** diablo_rojo has joined #openstack-infra18:10
*** bhavikdbavishi has quit IRC18:11
AJaegerclarkb, fungi, the security SIG leads proposed 5 weeks ago to abandon anchor, see https://review.openstack.org/#/c/597022/ - but neither of those two leads are in the core group. what about adding them to https://review.openstack.org/#/admin/groups/498,members to allow them to finish the retirement, please?18:11
prometheanfireis it possible to have the certificates api thing to not show up in https://storyboard.openstack.org/#!/project/openstack/requirements (the requirements task is done)?18:13
prometheanfiresame for the kafka-python one18:14
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add mysql to quick-start  https://review.openstack.org/61069718:14
fungiAJaeger: no objection from me18:14
*** Emine has quit IRC18:15
*** e0ne has joined #openstack-infra18:15
AJaegergagehugo, see above my comment about anchor, please18:15
AJaegerfungi: some infra-root would still need to do it - but can wait until after the meeting ;)18:16
openstackgerritMerged openstack-infra/project-config master: Install Pygments before checking README syntax for publish jobs  https://review.openstack.org/61101418:18
*** gyee has joined #openstack-infra18:19
gagehugoAJaeger: that works18:19
AJaegergagehugo: ok, let's hope an admin adds either you or Luke so that you can approve that change and abandon all open reviews and we can finish retirement18:21
clarkbgagehugo: you have two accounts do you want the gmail or att one added to that group?18:21
gagehugoclarkb: gmail18:22
*** e0ne has quit IRC18:22
clarkbdone18:22
*** hamzy has quit IRC18:23
clarkbprometheanfire: I think its waiting for the magnum todo to be set to soem closed state18:23
clarkbI'm nto aware of any other way to indicate a story is no longer active than to update the task state18:23
prometheanfireboo, imo the default view for a project should only have stories with active tasks for that project18:24
AJaegerthanks, clarkb18:24
AJaegergagehugo: if you need help, feel free to ask me...18:25
gagehugoclarkb: thanks18:25
gagehugoAJaeger: so the next step is to abandon the open reviews then?18:26
AJaegergagehugo: yes, exactly get this list empty: https://review.openstack.org/#/q/project:openstack/anchor+is:open18:26
AJaegergagehugo: shall I push the remaining changes for you?18:26
AJaegergagehugo: or want to follow https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project yourself?18:26
gagehugoAJaeger: I can follow that guide, it will be good practice as I've not done this before18:27
gagehugoI'll def ping you if I have questions18:28
AJaegergagehugo: so, steps 1-2 are done, you're at step 3 now - and thus need step 4 plus a governance change to finish retirement.18:28
AJaegergagehugo: ok18:29
gagehugook18:29
gagehugoAJaeger: thanks!18:29
*** rkukura has joined #openstack-infra18:35
*** jamesmcarthur has quit IRC18:38
fungiprometheanfire: it's been proposed to rework the project and project group views so that only stories with active tasks for those projects are considered active18:41
fungior, stated another way, constrain the task query so that only tasks for those projects are taken into consideration when deciding what story state to indicate18:41
prometheanfirefungi: well, have another +1 from me on it :P18:42
prometheanfireor maybe make the query selectable from a dropdown18:42
*** jamesmcarthur has joined #openstack-infra18:52
*** bobh has quit IRC18:54
*** bobh has joined #openstack-infra18:56
*** jamesmcarthur has quit IRC18:56
*** ssbarnea_ has joined #openstack-infra18:57
clarkbinfra meeting over in #openstack-meeting in just a few minutes18:57
*** bobh has quit IRC19:01
*** rkukura has quit IRC19:04
*** dave-mccowan has quit IRC19:07
*** trown|lunch is now known as trown19:07
*** rkukura has joined #openstack-infra19:07
*** dave-mccowan has joined #openstack-infra19:08
*** auristor has quit IRC19:19
*** auristor has joined #openstack-infra19:20
*** dkehn has quit IRC19:25
*** jamesmcarthur has joined #openstack-infra19:25
*** bobh has joined #openstack-infra19:32
*** dkehn has joined #openstack-infra19:33
*** auristor has quit IRC19:34
*** jamesmcarthur has quit IRC19:35
*** bobh has quit IRC19:37
openstackgerritMerged openstack-infra/puppet-cgit master: Filter out link-local ipv6 address in loadbalancer  https://review.openstack.org/57626219:44
*** rkukura has quit IRC19:45
*** rkukura has joined #openstack-infra19:48
openstackgerritMerged openstack-infra/puppet-cgit master: Update Gemfile for Zuulv3  https://review.openstack.org/57198719:49
*** auristor has joined #openstack-infra19:52
*** icey has quit IRC19:52
*** jamesmcarthur has joined #openstack-infra19:54
*** hamzy has joined #openstack-infra19:54
*** jamesmcarthur has quit IRC19:56
*** jamesmcarthur has joined #openstack-infra19:56
*** auristor has quit IRC19:57
*** eernst has joined #openstack-infra19:59
*** olivierb has joined #openstack-infra19:59
*** eernst has quit IRC20:01
*** auristor has joined #openstack-infra20:02
corvusfungi, clarkb: i was just looking into the super-weird bounce message we got for openstack-infra@lists and i think we broke verp with https://review.openstack.org/48498920:02
*** eernst has joined #openstack-infra20:02
corvusi think i can fix it20:02
*** eernst has joined #openstack-infra20:03
fungicorvus: the gmail deferrals for catchrobot at mirantis?20:04
*** eernst_ has joined #openstack-infra20:04
corvusfungi: yep20:04
corvuscatchrobot is not subscribed to the list; i don't know what address we sent to that goes there, but it was one of a batch of something like 30 addrs at mirantis.  verp generates unique bounce addresses to prevent this.20:05
*** bobh has joined #openstack-infra20:06
clarkbetherpad01.openstack.org is booting now20:07
openstackgerritJames E. Blair proposed openstack-infra/system-config master: lists: run mailman verp router before dnslookup  https://review.openstack.org/61115520:07
corvusfungi, clarkb: ^20:07
*** eernst has quit IRC20:08
*** bobh has quit IRC20:12
*** jamesmcarthur has quit IRC20:12
fungihow did moving the mailman_router entry have that impact?20:13
mordredcorvus: 2x+2 - but no +A bcause you said "merge with caution"20:13
fungiahh, yep, it was the verp router20:14
corvusfungi: yeah, we only needed to move the mailman router, but we moved both20:15
clarkbI'm running kick.sh on new etherpad01 server now20:15
corvusmordred: thx, i approved, i should be around for a while... depending on check queue.20:15
clarkbif this finishes successfully my next step will be to stop apache and nodejs on the old server then update dns20:15
clarkbyou may notice a short etherpad service interruption while that happens20:15
corvusif i leave before it merges, i'll either make sure someone is around or WIP it till tomorrow20:16
fungicorvus: thanks, my ability to monitor it is limited at the moment, but lgtm!20:16
corvusthis won't cause those bounces to stop, since they are for an already delivered message; we'll have to wait for gmail to time out on that.20:18
mriedemclarkb: when you get some leisure time, i think you might want to get in on this hot hot placement d-g action https://review.openstack.org/#/c/606853/20:18
clarkbmriedem: ya reviewing cdents list of changes was on my todo list. do we also need to add placement to the enabled services list or is it already there?20:19
*** jamesmcarthur has joined #openstack-infra20:19
mriedemit's been in d-g for awhile now20:19
mriedemsince ocata20:19
rpiosoIn gertty, is there a way to follow up on comments left on a patch set other than the most recent one? I appear to have successfully added the comments; however, when I submitted the review via 'r', they don't appear in Gerrit's web interface or the change's front screen in gertty.20:23
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Use zuul-sphinx for configuration layout  https://review.openstack.org/60427420:25
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Add overview of config options  https://review.openstack.org/60498420:25
clarkbjust waiting for the dns timeout to run out on etherpad.openstack.org so that you end up talking to the new server20:26
clarkbonce I've got that running I'll make sure bup is happy20:26
clarkbhttps://etherpad.openstack.org/p/clarkb-test is my long running test etherpad20:29
clarkbif you see the new lines in pad and in chat about post xenail upgrade I think we are set20:29
clarkbrpioso: is it posible you haven't synced yet?20:30
clarkbrpioso: there should be a little sync todo counter in the top right20:30
clarkbif that is greater than 0 then that could epxlain it20:30
*** rkukura has quit IRC20:30
*** rkukura_ has joined #openstack-infra20:30
*** rkukura_ has quit IRC20:31
rpiosoclarkb: I don't believe that's the issue. The first time I tried, comments for the most recent patch set were successfully submitted. And those were added after the ones on the older patch set.20:31
rpiosoI was hoping the comments on both patch sets would be submitted.20:32
*** rkukura has joined #openstack-infra20:32
*** kgiusti has left #openstack-infra20:33
*** slaweq has quit IRC20:33
clarkbbackups have been disabled on old etherpad.o.o server and I am running a start backup on etherpad01 now to make sure it works20:37
*** bobh has joined #openstack-infra20:38
clarkbok I think etherpad server upgrades are all done except for deleting the old servers which I wanted to keep around for a day or so just in case we have to revert20:41
*** ssbarnea has quit IRC20:42
*** ssbarne__ has joined #openstack-infra20:42
*** rkukura has quit IRC20:42
*** ssbarnea has joined #openstack-infra20:42
*** hamzy has quit IRC20:44
*** ssbarnea_ has quit IRC20:44
*** ansmith has quit IRC20:45
clarkbinfra-root https://review.openstack.org/#/c/529875/ and its parent will simplify things a bit to make the logstash.o.o upgrade to xenial easier. If you are able to review those two changes I can approve them when able to monitor them20:47
fungirpioso: expand the patch set details for the earlier patch and there should be a review button for it20:48
*** jesusaur has joined #openstack-infra20:48
clarkbianw: https://review.openstack.org/#/c/611075/ is a change that may interest you. We had another corrupt sqlalchemy utils whel file. This change adds a zip validity checker to our build/publish jobs for wheels20:49
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Simplify Builds page  https://review.openstack.org/61013720:49
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add more information to build page  https://review.openstack.org/61013820:49
rpiosofungi: Sure enough. Beware! Student driver!20:50
rpiosoclarkb, fungi: Thank you.20:51
funginp20:51
ianwclarkb: hrm ... did you find in the logs what went wrong?20:51
*** bobh has quit IRC20:52
clarkbianw: I did not, the wheel build fine according to the log I could see20:52
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add opendev nameservers (1/2)  https://review.openstack.org/60509220:52
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add opendev nameservers (2/2)  https://review.openstack.org/61006620:52
rpiosofungi: No CodeReview{-1,0,+1} option is provided on that screen.20:54
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Indicate whether a build is voting in the logs  https://review.openstack.org/60585720:54
*** eernst_ has quit IRC20:54
fungirpioso: correct, gerrit won't allow you to vote on labels for non-current patchsets. this mimics the behavior of the webui20:56
corvusbut you can still leave comments, and clicking save there should do it20:56
rpiosocorvus, fungi: Got it! Thanks, again :)20:56
rpiosoI'll take an additional action to vote.20:57
*** priteau has quit IRC20:57
*** ssbarne__ has quit IRC20:57
*** eernst has joined #openstack-infra20:58
fungirpioso: yes, you can (with separate actions either in gertty or the gerrit webui) comment on an earlier patch set and vote on the current patch set20:59
*** priteau has joined #openstack-infra20:59
*** sshnaidm has joined #openstack-infra21:01
*** sshnaidm is now known as sshnaidm|off21:01
*** mriedem is now known as mriedem_away21:02
rpiosoI expanded that old patch set with the new comments, selected Review, entered a message, and saved. The comments have not been submitted, yet. Does that sound right?21:03
clarkbianw http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/publish-wheel-mirror-ubuntu-xenial/25125c0/ is the log fwiw, you may know how to navigate it better21:03
*** eharney has quit IRC21:03
clarkbbut the failed.txt for python2 and 3 are both empty21:03
*** rkukura has joined #openstack-infra21:04
*** trown is now known as trown|outtypewww21:04
*** munimeha1 has joined #openstack-infra21:06
rpiosoHrm It looks like the draft messages have vanished.21:06
fungirpioso: they should be submitted on the next sync21:06
fungiat which point they'll no longer be draft comments21:07
fungiwell, as soon as they're saved they should no longer be draft comments from gertty's perspective21:07
openstackgerritMerged openstack-infra/zuul master: web: expand status panels only when the header is clicked  https://review.openstack.org/60934821:09
openstackgerritMerged openstack-infra/zuul master: web: fix stream url from status json  https://review.openstack.org/60936521:13
*** kaiokmo has quit IRC21:17
*** rkukura has quit IRC21:17
*** rkukura has joined #openstack-infra21:19
*** kaiokmo has joined #openstack-infra21:19
*** jamesmcarthur has quit IRC21:20
*** jamesmcarthur has joined #openstack-infra21:22
*** bobh has joined #openstack-infra21:28
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Extract pep8 messages for inline comments  https://review.openstack.org/58963421:29
corvusmordred, AJaeger: ^ that should pass now21:29
*** rkukura has quit IRC21:30
*** rh-jelabarre has quit IRC21:32
mordredcorvus: ooh!21:33
*** bobh has quit IRC21:33
ianwgary_perkins: thanks for credentials; that's all setup.  I've put some comments on ssl in https://review.openstack.org/#/c/60243621:35
smcginnisHah, Duncan on the Cinder team had suggested commenting on reviews with the pep8 errors and was told it was a horrible idea.21:35
smcginnisIt all takes time.21:35
smcginnisOh, I see that's slightly different than I assumed with the summary line. :)21:36
mordredcorvus: I pushed up https://review.openstack.org/610744 DNM Testing inline pep8 with a depends-on21:37
mordredcorvus: and added several juicy pep8 errors21:37
mordredcorvus: including but not limited to trailing whitespace, incorrect indentation - and SHOCK, no space after a comma!21:37
mordredsmcginnis: what did you think it was going to be?21:38
smcginnismordred: Adding comments to the review to point out the pep8 errors. Or is it?21:39
mordredyup. leaving in-line source code comments with the pep8 error message21:40
openstackgerritClark Boylan proposed openstack-infra/system-config master: Add opendev nameservers (1/2)  https://review.openstack.org/60509221:41
openstackgerritClark Boylan proposed openstack-infra/system-config master: Add opendev nameservers (2/2)  https://review.openstack.org/61006621:41
clarkbI think that may make puppet happy in testing21:41
smcginnismordred: OK, then it is the same as suggested as before. Personally, I think that will be very useful.21:41
mordredme too21:41
mordredWAY less clicky-clicky to figure out what the error was21:42
*** lbragstad is now known as lbragstad-50321:42
smcginnisEspecially for those where clicking on the job link and then clicking on the log link is one too many clicks for the day.21:42
mordredsmcginnis: out of curiosity, what was the concern?21:42
corvusi don't remember a conversation about that, but perhaps it was before gerrit had in-line comment api support?21:42
smcginnismordred: This is like 2-3 years ago now, but from what I remember there was an unconvincing but strong opinion from someone that there was no need to go to such lengths when it's all in the log file.21:43
smcginniscorvus: That could very well be too.21:43
mordredsmcginnis: ah. so it was more of "that seems like a waste of effort" rather than "this will physically impede progress towards joy"21:43
clarkbfwiw I think learning to read logs is a valuable skill/tool, but if computers can reduce overhead they should21:44
mordredto be fair - it did take us 2-3 years to implement the feature :)21:44
smcginnismordred: The former with the attitude of the latter.21:44
openstackgerritGage Hugo proposed openstack-infra/project-config master: Retire project Anchor  https://review.openstack.org/61118121:44
mordredclarkb: oh - also, per our conversaiton the other day about stackviz - it works _perfectly_ fine with other tox runs21:44
clarkbneat21:44
*** hasharAway has quit IRC21:44
*** munimeha1 has quit IRC21:45
mordredhttp://logs.openstack.org/67/610167/3/check/openstacksdk-functional-devstack/4c5c356/controller/logs/stackviz/#/stdin <-- there is an example of using it with openstacksdk functional tests21:45
mordredclarkb: SO - given an extremely small sample size - I think we _could_ add a dstat role to the base tox job pre and add the stackviz role to the base tox job post ... and get stackviz reports for every tox-based thing with subunit files21:47
rpiosofungi: I'm good, and learned more about gertty :)21:47
mordreddeciding we should is a whole other thing - but in terms of could - it should totally work21:47
openstackgerritClark Boylan proposed openstack-infra/project-config master: Revert "Reduce gate window floor to 10"  https://review.openstack.org/61118321:47
openstackgerritGage Hugo proposed openstack-infra/project-config master: Retire project Anchor  https://review.openstack.org/61118121:48
*** sshnaidm|off has quit IRC21:48
clarkbmordred: your stackviz report doesn't seem to have the dstat info, is that because you don't have dstat info in that job?21:48
mordredclarkb: I think we noticed the other day that the dstat info seems to be missing even from normal ones?21:48
clarkboh maybe21:49
clarkbmtreinish: may know why21:49
mordredhttp://logs.openstack.org/91/608091/6/check/tempest-full/9f77a01/controller/logs/stackviz/#/stdin/timeline?test=tempest.api.compute.admin.test_servers.ServersAdminTestJSON.test_rebuild_server_in_error_state21:49
mordredclarkb: ^^ that's from tempest-full in the nova repo21:50
* mordred assumes mtreinish knows all things21:50
mordredespecially related to subunit streams21:51
*** slaweq has joined #openstack-infra21:53
openstackgerritGage Hugo proposed openstack-infra/project-config master: Retire project Anchor - step 4  https://review.openstack.org/61118121:53
openstackgerritBen Nemec proposed openstack-infra/project-config master: Fix Pygments install for release check  https://review.openstack.org/61118421:54
clarkbianw: gary_perkins left this in gerrit as a comment too but I think we should try ot setup https even if it is a self signed cert or snakeoil cert. We can give that cert to oscc clouds.yaml directly to have it verify the connection that way21:54
clarkbits a bit hacky but should still prevent mitm and encrypt the data path21:55
*** mriedem_away has quit IRC21:55
*** smcginnis is now known as smcginnis_vaca21:55
clarkbother clouds have used letsencrypt21:55
*** priteau has quit IRC22:03
*** slaweq has quit IRC22:09
gagehugoAJaeger: Is there any other change in governance needed for anchor? https://review.openstack.org/#/c/611187/22:09
clarkbmordred: browser debugger says invalid dstat csv22:11
mordredoh!22:11
clarkbmordred: if I had to guess its the header causing it22:11
mordredwell, that's at least something22:11
*** slaweq has joined #openstack-infra22:11
clarkbI'm going to move the dstat csv file aside for that tempest one and edit the csv file to remove the header and see if that does it22:12
mordredclarkb: ++22:12
clarkbno that isn't helping it22:15
clarkbI also tried unzipping it22:15
mordredhrm22:15
mordredclarkb: my browser does not show that you edited the file22:15
clarkboh maybe there is a cache acting on it22:16
clarkblibreoffice is fine with the csv parsing22:18
*** boden has quit IRC22:18
clarkbreading the js it almost seems like its looking for Dstat to lead every row?22:19
clarkbit is actually looking for that header that is good to know22:20
mordredclarkb: oh - it pulls the file from stackviz/dstat.csv22:24
clarkbthe gzip part is fine I think22:25
mordredsorry - when I was refreshing I was looking at the one in the logs dir22:25
clarkbthere are quota in the header that are not expected by stackviz22:25
*** rkukura has joined #openstack-infra22:25
clarkbya I moved it around I'm about to gzip it again to test it works if zipped22:26
clarkbit does not22:26
*** jamesmcarthur has quit IRC22:26
clarkbok the quotes are fine its the gzipped file it doesn't like22:26
clarkbI have unzipped it and it works, gzipped it and it doesn't22:27
clarkbI think because the config.json says the dstat file is not gizipped22:27
clarkbI think we solve this if we have stackviz do the gzipping which was buggy when I was fixing things on python322:27
mordredand I guess the javascript isn't sending deflate headers?22:28
clarkbI think the config.json file in the data dir is what it uses to detminer stuff22:28
clarkband that says text/csv22:28
mordrednod22:28
clarkblet me gzip again and see the server response headers22:28
clarkbthey are application/g-zip on return22:29
clarkbI'm guessing it doesnt' check what the server says and jsut goes off of its config22:29
mordredwhere is the javascript?22:30
*** timothyb89 has joined #openstack-infra22:30
clarkbin the openstack/stackviz repo22:30
mordredyah - have you found where it does the fetch yet?22:31
clarkbor do you mean the stuff we serve? the served stuff is all minimized and impossible to read22:31
mordredoh - I meant the source22:31
mordredalso - we can serve map files with this to make it debuggable in the browser22:31
mordredbut that'll take me slightly longer to sort out than right now22:31
*** eernst has joined #openstack-infra22:31
clarkbmordred: app/js/directives/timeline.js: look for dstatLoaded22:32
mordredthanks22:32
clarkbthat broadcasts the dstatloaded even then we trigger an action on that event elesewhere but I think its the stuff in that file that is failing22:32
*** pbourke has quit IRC22:35
*** pbourke has joined #openstack-infra22:35
*** jamesmcarthur has joined #openstack-infra22:38
clarkbI'm quickly lost in what I'm assuming are angular magic things for fetching and decoding resources22:38
clarkbdataset.js seems to do the actual GET22:38
mordredclarkb: yes - and from what I can read the browser should do the gzip handling before the data gets to javascript22:39
mordredbut maybe we need to send an accept-encoding: gzip header?22:40
mordredclarkb: or - jeez, this hurts my brain22:41
openstackgerritClark Boylan proposed openstack-infra/system-config master: Add afs client docs for non Debuntu  https://review.openstack.org/60185522:41
clarkbfungi: ianw ^ how about that for specifying how to kinit as admin?22:42
clarkbmordred: my browser does send accept-encoding gzip22:42
clarkbor maybe it was just deflate22:42
*** jamesmcarthur has quit IRC22:43
*** slaweq has quit IRC22:44
openstackgerritMonty Taylor proposed openstack-infra/storyboard master: Build containers using pbrx  https://review.openstack.org/61119122:46
openstackgerritClark Boylan proposed openstack-infra/system-config master: Add afs client docs for non Debuntu  https://review.openstack.org/60185522:46
*** rcernin has joined #openstack-infra22:49
*** agopi is now known as agopi|brb22:52
*** jamesmcarthur has joined #openstack-infra22:54
*** agopi|brb has quit IRC22:56
openstackgerritMonty Taylor proposed openstack-infra/storyboard master: Build containers using pbrx  https://review.openstack.org/61119122:57
openstackgerritMerged openstack-infra/system-config master: lists: run mailman verp router before dnslookup  https://review.openstack.org/61115523:01
openstackgerritMonty Taylor proposed openstack-infra/storyboard master: Build containers using pbrx  https://review.openstack.org/61119123:01
openstackgerritMonty Taylor proposed openstack-infra/storyboard master: Fix up a few requirements  https://review.openstack.org/61119423:01
*** jamesmcarthur has quit IRC23:07
*** jamesmcarthur has joined #openstack-infra23:07
*** eernst has quit IRC23:09
*** eernst has joined #openstack-infra23:09
*** tosky has quit IRC23:10
*** eernst has quit IRC23:10
*** slaweq has joined #openstack-infra23:11
*** jamesmcarthur has quit IRC23:18
*** jamesmcarthur has joined #openstack-infra23:21
*** dave-mccowan has quit IRC23:24
*** hamzy has joined #openstack-infra23:24
*** jamesmcarthur has quit IRC23:32
*** rkukura has quit IRC23:41
*** slaweq has quit IRC23:44
*** jamesmcarthur has joined #openstack-infra23:44
clarkbcorvus: https://review.openstack.org/#/c/605092/5 passes now for opendev dns23:45
*** ssbarnea|bkp2 has joined #openstack-infra23:45
*** ssbarnea has quit IRC23:46

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!