| fungi | jeblair: in the past i think the suggestion has been to base64 encode and then decode it with an exec? though you can also define binary encoded data in yaml, i have no idea if that freaks puppet/hiera the heck out | 00:00 |
|---|---|---|
| reed | docaedo, i'm responding to Marcela now | 00:00 |
| clarkb | and if you don't want updates well we should stop updating your project | 00:00 |
| adam_g | greghaynes, http://logs.openstack.org/35/264935/2/check/gate-functional-dsvm-astara-appliance-src/536b9f3/logs/devstacklog.txt.gz#_2016-01-20_23_34_42_795 | 00:00 |
| docaedo | reed: oh excellent, thanks! | 00:00 |
| reed | I'll argue that hiding the shade document only because it's missing a piece is bad | 00:00 |
| dougwig | clarkb: another dev made a hand update, and that one was in merge conflict. i figured it was easier to abandon than merge and push it back up empty. | 00:00 |
| dougwig | clarkb: not a normal event. | 00:00 |
| *** bpokorny_ has joined #openstack-infra | 00:00 | |
| clarkb | dougwig: you just ignore it | 00:00 |
| reed | the same goes for fog and java | 00:00 |
| clarkb | dougwig: and the next day it will be updated | 00:00 |
| dougwig | clarkb: oh, great. ok, will do. | 00:01 |
| reed | some (correct) documentation is better than no documentation at all | 00:01 |
| clarkb | dhellmann: similar question for https://review.openstack.org/#/c/257693/ if we don't want updates we shouldn't make updated | 00:01 |
| clarkb | dhellmann: if we do want updates we can leave the changes in place and let them get updated | 00:01 |
| greghaynes | adam_g: Caching get-pip-py file from https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py in /opt/stack/new/.cache/image-create/source-repositories/get_pip_py_5217e630e57404aa87af8b6db2387b35af53ada | 00:01 |
| greghaynes | adam_g: so it definitely is reading the github url from the source-repositories file | 00:01 |
| docaedo | reed: agreed completely, also shade is as far as I know the only lib that is actively tested against multiple clouds, and gets fixed as openstack and the openstack clouds it's tested against change | 00:02 |
| jeblair | fungi, nibalizer: base64 encoding it works for me, that seems like it would fit in well with what we're generally doing... | 00:02 |
| fungi | greghaynes: adam_g: not finding it in master branch tip http://codesearch.openstack.org/?q=pypa%2Fpip%2Fmaster%2Fcontrib%2Fget-pip.py&i=nope&files=&repos= | 00:02 |
| jeblair | fungi, nibalizer: it would be cool if i can just use yaml's base64 and hiera would dtrt... | 00:02 |
| jeblair | i will try that now if no one else has already | 00:03 |
| fungi | jeblair: yeah, no idea if that works, but if you can confirm, big win | 00:03 |
| adam_g | fungi, thank you for pointing out what should be obvious to me. | 00:03 |
| adam_g | greghaynes, i will go away now and fix that | 00:03 |
| reed | docaedo, true, that and the openstack clients | 00:04 |
| *** bpokorny has quit IRC | 00:04 | |
| greghaynes | adam_g: oh? | 00:04 |
| fungi | adam_g: oh, was your project one of the ones in those search results? ;) | 00:04 |
| adam_g | greghaynes, we've been keeping our own pip-and-virtualenv element that looks to be overriding the DIB one | 00:04 |
| reed | docaedo, on the other hand, libcloud seems to be popular among other cloud users, like AWS, GCE, digital ocean etc | 00:05 |
| adam_g | fungi, yes. the first one :) | 00:05 |
| greghaynes | adam_g: hahaa | 00:05 |
| fungi | hah | 00:05 |
| adam_g | that pre-dates me, but today it will get thrown. | 00:05 |
| * reed slaps the hands that put Digital Ocean in the 'clouds' category | 00:05 | |
| *** elo has joined #openstack-infra | 00:06 | |
| docaedo | reed: that is true too - so for someone wanting the ability to address multiple clouds from the same codebase, makes sense | 00:06 |
| clarkb | adam_g: don't worry you don't actually want new pip yet so this is broken but better | 00:06 |
| greghaynes | by default that element should install from pacakge | 00:07 |
| greghaynes | so you should be fine for now | 00:07 |
| greghaynes | actually, youre using source installtype, nvm | 00:07 |
| adam_g | id prefer to just drop ours from tree and use DIBs element | 00:07 |
| mordred | jeblair: let me know if you have to do something other than base64 - we might have to make the heiracopy smarter | 00:07 |
| *** ZZelle_ has quit IRC | 00:08 | |
| mordred | reed: what is the shade piece missing? | 00:08 |
| reed | mordred, a chapter about Swift | 00:09 |
| mordred | heh | 00:09 |
| reed | IIRC Caleb contributed to shade to add that missing piece | 00:09 |
| mordred | turns out a guide to using openstack compute does not need a swift chapter | 00:09 |
| mordred | reed: yah, I believe that is correct | 00:09 |
| reed | but he ran out of time | 00:09 |
| reed | I needed him to focus on another project first... | 00:10 |
| clarkb | he should have more time now that the seahawks have been eliminated | 00:10 |
| reed | aahahaha | 00:10 |
| clarkb | >_> | 00:10 |
| reed | mordred, the piece about swift makes sense in the first app tutorial... it's just that IMHO some documentation is better pushed out incomplete but valid, than sit on it until it's shiny | 00:11 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove zero length files https://review.openstack.org/270550 | 00:11 |
| mordred | reed: I agree | 00:11 |
| reed | 'release early release often' ... | 00:12 |
| nibalizer | jeblair: what is the file? | 00:12 |
| nibalizer | actually... since we're on puppet apply its really easy to do this probably | 00:13 |
| nibalizer | because we can just use the file { } resource and point it at the location that ansible droped the file | 00:13 |
| *** dizquierdo_ has quit IRC | 00:14 | |
| nibalizer | so we could standardize on having groups/<groupname>.d/{file1,file2,file3} | 00:15 |
| nibalizer | and ansible moves over groupname.yaml and groupname.d/* | 00:15 |
| jeblair | nibalizer: oh, so you reckon we can just stick the file in the secrets repo alongside the other yaml files, ignore it in hiera, and just use source=> /opt/system-config/hieradata/production/fqdn/filename ? | 00:15 |
| nibalizer | and the file resource ends up looking line file { '/path/to/secret_file': source => '/etc/puppet/hieradata/groups/<groupname.d>/filename', } | 00:15 |
| jeblair | nibalizer: won't that break the apply test (since the file won't exist in that case?) | 00:16 |
| nibalizer | ya it would break the apply test | 00:17 |
| nibalizer | but we can fix that | 00:17 |
| *** rbrndt has quit IRC | 00:17 | |
| *** PsionTheory has quit IRC | 00:17 | |
| *** weshay has quit IRC | 00:19 | |
| jeblair | nibalizer, fungi: the yaml binary encoding works | 00:19 |
| nibalizer | what is the file though? | 00:20 |
| nibalizer | some secretstore or something? | 00:20 |
| jeblair | nibalizer: the thing i'm working on? a kerberos keytab | 00:20 |
| *** hdd has quit IRC | 00:21 | |
| fungi | jeblair: excellent--i'll likely need to do the same for puppeting in our release signing keyring | 00:21 |
| ianw | pabelanger: doesn't dib-lint complain about the RST files not being there? | 00:21 |
| nibalizer | ah | 00:21 |
| jeblair | nibalizer: which is a binary blob that's used in place of a password or api key to get a credential from a kerberos server | 00:21 |
| jeblair | nibalizer: i tried a test with the data at http://yaml.org/type/binary.html | 00:21 |
| jeblair | nibalizer: and both forms worked | 00:21 |
| nibalizer | i think having it as a plain file is better than base64encoded | 00:21 |
| nibalizer | there are two ways we can just have a file in the secrets repo | 00:22 |
| *** peristeri has quit IRC | 00:23 | |
| *** ccrouch has quit IRC | 00:23 | |
| *** elo has quit IRC | 00:23 | |
| *** ccrouch has joined #openstack-infra | 00:24 | |
| nibalizer | https://github.com/adrienthebo/hiera-file is the hiera-file backend | 00:25 |
| jeblair | fwiw, the keytab is like 126 characters, so it won't be the longest base64 encoded string in our yaml by far :) | 00:25 |
| nibalizer | the others being pem files? | 00:25 |
| jeblair | yeh | 00:25 |
| jeblair | which are non-binary base64 obviously | 00:25 |
| fungi | way shorter than the ascii-armored x.509 key material we have | 00:25 |
| fungi | yeah, the pem encoded certs | 00:26 |
| jeblair | nibalizer: of course if we start using files, we can move those out | 00:26 |
| *** elo has joined #openstack-infra | 00:26 | |
| *** mriedem is now known as mriedem_dinner | 00:28 | |
| *** dims has quit IRC | 00:29 | |
| *** pvaneck has quit IRC | 00:30 | |
| krotscheck | jeblair: flagged for review. wil look at it in themorning after diaprs | 00:31 |
| *** sdake_ has joined #openstack-infra | 00:32 | |
| *** sdake_ has quit IRC | 00:32 | |
| *** bexelbie has quit IRC | 00:33 | |
| *** sdake_ has joined #openstack-infra | 00:33 | |
| *** sdake has quit IRC | 00:34 | |
| jeblair | nibalizer, fungi: um. it turns out i don't even need to change hieraedit. | 00:34 |
| jeblair | ~/hieraedit.py --yaml ./mirror_update.openstack.org -f /home/corvus/bandersnatch.keytab bandersnatch_keytab | 00:35 |
| jeblair | that did the right thing with the data | 00:35 |
| jeblair | (i do need to change it to handle empty yaml files, but that's a separate issue resulting from the puppet apply split) | 00:35 |
| fungi | wow, neat | 00:36 |
| *** otter768 has joined #openstack-infra | 00:36 | |
| jeblair | yeah, python-yaml coerces the output automatically based on its type | 00:37 |
| fungi | oh, yep it does | 00:37 |
| jeblair | it, er, prints the output which isn't optimal, but... | 00:38 |
| jeblair | it works :) | 00:38 |
| greghaynes | clarkb: does nodepool ask for another image-upload in the logs after the last one which finished? | 00:39 |
| clarkb | greghaynes: yes | 00:39 |
| greghaynes | clarkb: as in, do you see the job submitted for a new image upload | 00:39 |
| greghaynes | oh? | 00:39 |
| greghaynes | so is it queued? | 00:39 |
| clarkb | no | 00:39 |
| *** santoshk has quit IRC | 00:39 | |
| greghaynes | huh. so it gets consumed by the builder but the upload doesn't happen | 00:39 |
| clarkb | logs say it is going to do it but gearman says no qeue | 00:39 |
| greghaynes | the builder logs say it is going to upload? | 00:40 |
| clarkb | and no uloads are attempted after | 00:40 |
| greghaynes | or, which logs | 00:40 |
| clarkb | nodepool.nodepool logs that upload job is being submitted | 00:40 |
| greghaynes | ok | 00:40 |
| clarkb | nodepool.builder knows nothing | 00:40 |
| greghaynes | wah | 00:40 |
| clarkb | the job is registered in gearman but nothing is queued and it has one worker | 00:40 |
| greghaynes | hrm | 00:41 |
| *** markvoelker_ has quit IRC | 00:41 | |
| *** otter768 has quit IRC | 00:41 | |
| *** kushal has quit IRC | 00:41 | |
| *** Swami has quit IRC | 00:42 | |
| greghaynes | clarkb: do we run the builder with debug logs on? | 00:42 |
| greghaynes | clarkb: I think yes | 00:42 |
| clarkb | greghaynes: the builder yes but not gearman | 00:42 |
| greghaynes | ya, so no "got job" message is seen? | 00:42 |
| clarkb | let me grep more | 00:43 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: [WIP] testing multiple dib uploads https://review.openstack.org/270557 | 00:43 |
| ianw | clarkb: ^ see how this goes... | 00:43 |
| *** ccrouch has quit IRC | 00:44 | |
| zaro | fungi: was wondering wher projects.ini is located? | 00:44 |
| zaro | the file for jeepyb. can't seem to find it projects-config or system-config repos | 00:45 |
| clarkb | http://paste.openstack.org/show/484482/ | 00:45 |
| clarkb | so a few minutes after my original grepping it completed for that image | 00:45 |
| *** ccrouch has joined #openstack-infra | 00:45 | |
| clarkb | but I don't see any more recent imgae uploading logs | 00:46 |
| greghaynes | clarkb: huh, so the next one ended up uploading? | 00:46 |
| zaro | fungi:NV, found it. | 00:46 |
| clarkb | greghaynes: the one that was "hanging" fnished 4 hours later | 00:47 |
| clarkb | but I don't see one after that | 00:47 |
| greghaynes | clarkb: I wonder if we just arent flushing logs | 00:47 |
| greghaynes | clarkb: so you dont see it start until the upload finishes | 00:47 |
| ianw | greghaynes: all goes through logging module though? | 00:48 |
| greghaynes | ianw: it does. I am kind of grasping at straws... | 00:48 |
| anteaya | zaro: where is it? | 00:48 |
| greghaynes | I assume the logging module would dtrt and flush out in another thread but I dont actually know | 00:48 |
| clarkb | 2016-01-20 18:39:02,084 DEBUG nodepool.builder: Got job image-upload:176178 with data {"image-name": "devstack-precise", "provider": "hpcloud-b1"} is the lsat got job | 00:49 |
| greghaynes | clarkb: and on the nodepool side theres been anohter job submitted since then? | 00:49 |
| anteaya | zaro: this it? http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/review.projects.ini.erb | 00:50 |
| clarkb | greghaynes: ya http://paste.openstack.org/show/484483/ | 00:50 |
| greghaynes | ah, right, it just spams all the uploads and lets gearman do its thing | 00:50 |
| greghaynes | ok | 00:50 |
| zaro | anteaya: yes, that is it. | 00:51 |
| *** ajmiller has joined #openstack-infra | 00:51 | |
| greghaynes | clarkb: hrm, so I kind of suspect it is off uploading the next thing | 00:51 |
| anteaya | zaro: wonderful | 00:51 |
| greghaynes | clarkb: not sure how to test that theory other than network dumping | 00:51 |
| clarkb | I can certainly look for glance connections | 00:52 |
| clarkb | anyone know what port it uses? | 00:52 |
| zaro | anteaya: figured out the issue with access for sandbox. https://code.google.com/p/gerrit/issues/detail?id=3831 | 00:52 |
| clarkb | I suppose Ican ask the catalog | 00:52 |
| *** sdake has joined #openstack-infra | 00:53 | |
| greghaynes | clarkb: 9292 | 00:53 |
| greghaynes | clarkb: heh, thats exactly what I did ;) | 00:53 |
| clarkb | sudo netstat -np | grep 9292 so no | 00:53 |
| clarkb | er says no | 00:53 |
| greghaynes | if its rax itll be swift | 00:53 |
| greghaynes | which I have no idea of the port | 00:53 |
| clarkb | if it is going in order like it was for the first two it will be hpcloud | 00:53 |
| greghaynes | I dunno, that depends on how gearman farms them out | 00:54 |
| *** sdake_ has quit IRC | 00:54 | |
| fungi | zaro: looks exactly like what i expected was happening. thanks for testing and confirming | 00:54 |
| clarkb | gearman should round robin jobs in order of client submission iirc | 00:54 |
| *** kushal has joined #openstack-infra | 00:54 | |
| clarkb | so I think it would be hpcloud | 00:54 |
| clarkb | for dib image id 176176 | 00:55 |
| clarkb | maybe we should add the provider to those log messages | 00:55 |
| *** bpokorny_ has quit IRC | 00:55 | |
| greghaynes | yep | 00:55 |
| greghaynes | the fact that it eventually uploaded the second image is really odd | 00:55 |
| *** bpokorny has joined #openstack-infra | 00:55 | |
| clarkb | greghaynes: we may be getting hella throttled | 00:55 |
| greghaynes | either we are missing it uploading the next or it just goes out to lunch for a while inbetween uploads | 00:56 |
| clarkb | (something ot look into) | 00:56 |
| clarkb | looks like hpcloud uses 443 | 00:57 |
| sc68cal | RIP gate - 31 hours 51 minutes | 00:57 |
| sc68cal | going to go pour one out for you zuul | 00:57 |
| clarkb | ok there is a connection there now to tcpdump it to see if it is active | 00:58 |
| greghaynes | clarkb: awesome | 00:58 |
| *** gildub has joined #openstack-infra | 00:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add mirror_update.openstack.org https://review.openstack.org/270520 | 00:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Document adding a kerberos service principal https://review.openstack.org/270559 | 00:59 |
| *** changbl has quit IRC | 00:59 | |
| clarkb | it isnt chatty at all | 00:59 |
| greghaynes | clarkb: no packets at all? | 01:00 |
| clarkb | 4 so far | 01:00 |
| *** ccrouch has quit IRC | 01:00 | |
| clarkb | all of length 0 | 01:00 |
| greghaynes | clarkb: are the window sizes 0 anywhere? | 01:00 |
| clarkb | so I don't think it is doing an upload | 01:00 |
| *** bexelbie has joined #openstack-infra | 01:00 | |
| *** bexelbie has joined #openstack-infra | 01:00 | |
| *** ccrouch has joined #openstack-infra | 01:00 | |
| clarkb | no 0 windows sizes | 01:00 |
| greghaynes | hrm | 01:00 |
| ianw | clarkb: can you strace the thread, is it waiting on something? | 01:01 |
| *** sputnik13 has quit IRC | 01:01 | |
| *** ddieterly has joined #openstack-infra | 01:01 | |
| *** bpokorny has quit IRC | 01:01 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 01:02 |
| clarkb | are python threads straceable in that way? | 01:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add mirror_update.openstack.org https://review.openstack.org/270520 | 01:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Document adding a kerberos service principal https://review.openstack.org/270559 | 01:02 |
| greghaynes | clarkb: totally | 01:02 |
| clarkb | I expect to be able to strace the entire process and be confuzzled | 01:02 |
| greghaynes | oh | 01:02 |
| clarkb | I can do a thread dump though so lets start there | 01:02 |
| greghaynes | clarkb: if you can find the thread id then it should work | 01:02 |
| clarkb | should give us a python traceback which is hopefully mor ereadable | 01:02 |
| *** baoli_ has quit IRC | 01:03 | |
| ianw | clarkb: if you pick the right one (/proc/pid/task) | 01:03 |
| ianw | yeah, thread bt is probably better anyway | 01:03 |
| *** mriedem_dinner is now known as mriedem | 01:05 | |
| clarkb | http://paste.openstack.org/show/484486/ | 01:05 |
| clarkb | looks like it is waiting for gearman to give it a job | 01:05 |
| clarkb | so let me double check the gearman job queues again | 01:05 |
| greghaynes | hrm | 01:05 |
| clarkb | the jobs are still there with one worker and no queues | 01:06 |
| greghaynes | clarkb: so, actually, there should be a job queued | 01:07 |
| greghaynes | clarkb: since nodepool spams gear with all the uploads | 01:07 |
| clarkb | greghaynes: maybe that isn't happening? | 01:07 |
| clarkb | or the client is going away and gear removes the jobs? | 01:07 |
| greghaynes | yea, I dunno | 01:08 |
| *** Qiming has joined #openstack-infra | 01:08 | |
| greghaynes | seems like nodepool submits the jobs based on http://paste.openstack.org/show/484483/ | 01:08 |
| greghaynes | although gear logs would be more convincing of that | 01:08 |
| clarkb | ya I am going to go read the code that logs that now | 01:08 |
| pabelanger | ianw: not sure. Will need to check, but I know rpmlint is not happy | 01:09 |
| *** salv-orlando has quit IRC | 01:09 | |
| pabelanger | ianw: where is dib-lint? | 01:10 |
| greghaynes | pabelanger: in dib | 01:10 |
| clarkb | submitJob should be async so I doubt it is blocking there and getting confused | 01:10 |
| pabelanger | greghaynes: right, but not in the gate? | 01:10 |
| clarkb | so I agree it should actually be submitting all of those jobs | 01:10 |
| greghaynes | pabelanger: it is in the dib gate | 01:10 |
| greghaynes | clarkb: yep. Do we have logs on the gearman server itself? | 01:10 |
| pabelanger | greghaynes: cool | 01:10 |
| pabelanger | ianw: dib-lint seems happy | 01:10 |
| clarkb | greghaynes: not at debug just warn and up or something iirc | 01:11 |
| clarkb | but let me check | 01:11 |
| greghaynes | clarkb: ah. It would be nice to have a job log on the server | 01:11 |
| clarkb | error: [Errno 32] Broken pipe well there is that | 01:11 |
| greghaynes | ahoy | 01:11 |
| clarkb | at 2016-01-20 19:18:26,753 ERROR gear.Server: Exception in poll loop: | 01:11 |
| clarkb | so that would've dumped all our jobs | 01:11 |
| greghaynes | \O/ we broke geard | 01:11 |
| clarkb | :( | 01:11 |
| clarkb | but the client never find sout to record those jobs as errors | 01:12 |
| clarkb | maybe we should consider running a separate geard for this | 01:12 |
| clarkb | though its really low traffic compared to the jenkins jobs | 01:12 |
| greghaynes | Yea, I'd like to get a good theory for what killed that socket first | 01:13 |
| clarkb | r = self.conn.send(data) is what threw the exception | 01:13 |
| clarkb | so not much info there | 01:13 |
| greghaynes | clarkb: hrm, so nodepool could have just closed the connection | 01:13 |
| greghaynes | clarkb: you know, it might be throttling | 01:13 |
| greghaynes | clarkb: is geard colocated with nodepool? | 01:13 |
| clarkb | greghaynes: no it is on zuul | 01:14 |
| greghaynes | ya | 01:14 |
| greghaynes | :/ | 01:14 |
| greghaynes | clarkb: ok, so we need to make nodepool resilient against that socket dying I think | 01:15 |
| clarkb | greghaynes: yup | 01:16 |
| clarkb | though I thought that gear would at least on the client side say something like woops connection did these jobs are all invalid now | 01:16 |
| clarkb | but Iguess now | 01:16 |
| clarkb | *not | 01:16 |
| greghaynes | We might just not be listening for that | 01:16 |
| greghaynes | actually | 01:16 |
| greghaynes | a few things - do we have tcp keepalives on either side? | 01:16 |
| clarkb | we should on nodepool since we did that for the client connections | 01:17 |
| clarkb | but that may be implemented in requests so may not affect gear | 01:17 |
| greghaynes | this is kind of a good argument for sharing a geard too - since were communicating over that channel a lot for test jobs then the socket should die pretty fast | 01:18 |
| * greghaynes reads some gear client code | 01:18 | |
| *** kaisers1 has joined #openstack-infra | 01:18 | |
| clarkb | right we poll to determine demand | 01:18 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Fix rpmlint script-without-shebang errors https://review.openstack.org/270562 | 01:18 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files https://review.openstack.org/270563 | 01:18 |
| *** elo has quit IRC | 01:19 | |
| pabelanger | ianw: greghaynes: 2 more for rpmlint | 01:19 |
| pabelanger | ^ | 01:19 |
| greghaynes | clarkb: and other gearman stuff is still working with nodepool right? | 01:19 |
| clarkb | greghaynes: ya demand seems to be working fine but it doesn't need persistent client server worker job stuff | 01:19 |
| clarkb | it just polls state at $time | 01:19 |
| *** kaisers has quit IRC | 01:20 | |
| greghaynes | clarkb: over the same socket though? | 01:20 |
| clarkb | yes should be | 01:20 |
| clarkb | yup same self.gearman_client in nodepool.NodePool | 01:20 |
| greghaynes | hrm, so ya I would really like some kind of traceback / error for what happens on the nodepool side when it finds out that the socket exploded | 01:20 |
| clarkb | I can see if we have one | 01:20 |
| greghaynes | based on the gear code I think it should raise an exceptio | 01:21 |
| clarkb | grep isn't showing anything | 01:22 |
| greghaynes | :/ | 01:22 |
| clarkb | grepping for gear and times around when server shows error | 01:23 |
| greghaynes | clarkb: I think the error might not be gear - itll just raise the raw exception I think | 01:23 |
| clarkb | right but it should show up as gear in the traceback | 01:23 |
| greghaynes | s/raw/originl | 01:23 |
| greghaynes | clarkb: not if we catch it in nodepool then log.except it? | 01:24 |
| greghaynes | or log.error | 01:24 |
| *** _amrith_ is now known as amrith | 01:24 | |
| clarkb | log.exception should but log.error wont | 01:24 |
| greghaynes | right | 01:24 |
| clarkb | I also grepped the timestamp less milliseconds and didn't see anythin | 01:24 |
| clarkb | but let me grep for errors | 01:24 |
| greghaynes | hrm | 01:24 |
| *** keedya has joined #openstack-infra | 01:24 | |
| anteaya | zaro: thank you | 01:25 |
| anteaya | zaro: it is only a problem with ssh, hmmmm | 01:25 |
| anteaya | sc68cal: yeah, the poor gate :( | 01:25 |
| clarkb | ok found some things | 01:26 |
| greghaynes | ooo things | 01:26 |
| clarkb | let me work on putting togethre in a paste | 01:26 |
| clarkb | I am typing with one elbow in the air so baby can take a nap, do not recommend | 01:27 |
| greghaynes | haha | 01:27 |
| *** elo has joined #openstack-infra | 01:27 | |
| greghaynes | when are we going to make a baby seat thing which has a laptop extensikon | 01:27 |
| *** rguillebert has quit IRC | 01:27 | |
| nibalizer | oh man | 01:28 |
| nibalizer | put a battery array in the stroller | 01:28 |
| anteaya | clarkb: I'm constantly impressed by your gymnastics | 01:28 |
| nibalizer | so it charges your thibgs | 01:28 |
| anteaya | that wouldn't do anything to the babies health I'm sure | 01:28 |
| greghaynes | Yes! I should absolutely add a laptop tray and batteries to the back of my stroller and basically have a portable standing desk | 01:28 |
| nibalizer | oh man hackrollers have to be a thing | 01:29 |
| *** dingyichen has quit IRC | 01:29 | |
| *** aeng has quit IRC | 01:29 | |
| *** pahuang has quit IRC | 01:29 | |
| dougwig | clarkb: i've had to do that with cats, but not babies. good luck. | 01:29 |
| anteaya | dougwig: elbow in the air or battery array? | 01:30 |
| greghaynes | haha | 01:30 |
| dougwig | anteaya: elbow in the air. i think every parent has used the stroller as a standing desk. :) | 01:31 |
| greghaynes | \O/ it is a thing | 01:31 |
| dougwig | but a deep cycle in the bottom compartment, now that would be a good idea. | 01:31 |
| anteaya | wow | 01:31 |
| anteaya | every day I give thanks I'm not a parent | 01:31 |
| anteaya | today I give thanks twice | 01:32 |
| clarkb | http://paste.openstack.org/show/484487/ that is severely trimmed down | 01:32 |
| clarkb | but it definitely shows we disconnected | 01:32 |
| clarkb | I don't see errors from the job submissions though | 01:33 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files https://review.openstack.org/270563 | 01:33 |
| clarkb | anteaya: hahaha | 01:33 |
| clarkb | having two definitely makes certain things very hard. Occasionally we get out of the house with just one baby leaving the other at home with one baby and it is so much easier | 01:34 |
| clarkb | you can get things done when you have one and it is napping | 01:35 |
| anteaya | yes | 01:35 |
| anteaya | agreed | 01:35 |
| greghaynes | clarkb: looks like there is a gear client.connections_condition which you can listen to and get notifications when connections change | 01:35 |
| clarkb | greghaynes: I am guessing we aren't currently checking that though | 01:35 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files https://review.openstack.org/270563 | 01:36 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Fix rpmlint script-without-shebang errors https://review.openstack.org/270562 | 01:36 |
| anteaya | I'm just amazed that the fact that strong magnetic fields affect growing cells seems to be a side issue in the tangental conversation | 01:36 |
| clarkb | also they are just old enough where jealousy is a real thing. 7 months old and how dare you pick up my sister | 01:36 |
| anteaya | clarkb: nice | 01:36 |
| greghaynes | clarkb: it also looks like gear should have raised the original exception, and if that is actually the case then that is probably a much easier way to handle this | 01:36 |
| anteaya | it will only get worse | 01:36 |
| dougwig | anteaya: nerd babies are doomed on that front. | 01:36 |
| anteaya | dougwig: yup | 01:37 |
| anteaya | so glad I don't have children | 01:37 |
| greghaynes | clarkb: The annoying bit is that we can definitely get out of sync | 01:37 |
| greghaynes | clarkb: as in, just because we think the connection is dead doesn't mean the gear server knows yet | 01:37 |
| anteaya | my cat has figured out that if she touches my neck gently with a paw when I make a move to put her down I'll hold her a little longer | 01:37 |
| clarkb | greghaynes: riht, but you can always close the connection which should send a fin right? then reconnect? | 01:37 |
| anteaya | I can be stuck for an additional 30 minutes if she plays it right | 01:37 |
| pabelanger | greghaynes: ianw the only rpmlint error left is: E: wrong-script-interpreter /usr/share/diskimage-builder/elements/package-installs/bin/package-installs-v2 /usr/local/bin/dib-python | 01:38 |
| clarkb | greghaynes: if networking is really hosed that may not be sufficient though | 01:38 |
| greghaynes | clarkb: not necessarially, it may just be closed because we got an rst out of band | 01:38 |
| pabelanger | Since we added that for inside a chroot, I have to get funky with the RPM package to remove it | 01:38 |
| pabelanger | sadly, I cannot find a debian lintian override option for RPMs | 01:38 |
| *** thorst has joined #openstack-infra | 01:39 | |
| *** Sukhdev has quit IRC | 01:40 | |
| *** katyafervent has quit IRC | 01:41 | |
| *** Daisy has joined #openstack-infra | 01:42 | |
| greghaynes | clarkb: ok, I think next thing is to whip up some test cases | 01:44 |
| greghaynes | clarkb: seems like we have a pretty good handle on what is going on | 01:44 |
| clarkb | ya basically queue some jobs, kill server connection, what next | 01:44 |
| greghaynes | clarkb: I also wonder if the root cause is that we are now uploading at a higher speed and therefore hit some throttling | 01:45 |
| clarkb | well we are still in process so shouldn't be uploading faster | 01:45 |
| clarkb | but it is possible I suppose | 01:45 |
| greghaynes | hrm | 01:45 |
| *** changbl has joined #openstack-infra | 01:45 | |
| greghaynes | or maybe the image-upload spam hit throttling? althogh i'd hope that wouldhnt | 01:46 |
| greghaynes | er, the submit job image-upload spam | 01:46 |
| greghaynes | oh, or, I wonder if gear does not properly handle the case when send buffers get filled | 01:46 |
| clarkb | that would surprise me since its tiny | 01:46 |
| *** Daisy_ has joined #openstack-infra | 01:47 | |
| greghaynes | and explodes rather than trying again later | 01:47 |
| clarkb | greghaynes: it should block on writes iirc | 01:47 |
| clarkb | but I may be mistaken | 01:47 |
| *** Daisy_ has quit IRC | 01:48 | |
| *** Daisy has quit IRC | 01:48 | |
| *** katyafervent has joined #openstack-infra | 01:49 | |
| *** elo has quit IRC | 01:52 | |
| greghaynes | clarkb: ya, it is all blocking so that should 'just work' | 01:52 |
| *** dims has joined #openstack-infra | 01:54 | |
| *** elo has joined #openstack-infra | 01:54 | |
| *** erlon has quit IRC | 01:56 | |
| ianw | isn't this stuff the gear client has figured out? | 01:58 |
| clarkb | it should it is relatievly reliable for zuul | 01:59 |
| greghaynes | Yep, trying to get a theory on why the socket died though, the best I have so far is just throttling | 01:59 |
| *** Zhongjun_ has joined #openstack-infra | 02:00 | |
| *** dingyichen has joined #openstack-infra | 02:00 | |
| *** vivekd has joined #openstack-infra | 02:00 | |
| *** tiswanso has joined #openstack-infra | 02:01 | |
| *** tiswanso has quit IRC | 02:03 | |
| *** tiswanso has joined #openstack-infra | 02:04 | |
| ianw | greghaynes: from the remote end? i.e. you queued up too many jobs in quick succession? | 02:05 |
| greghaynes | Rax does bw throttling and tends to rst connections ove t the limit | 02:06 |
| *** tlian2 has joined #openstack-infra | 02:08 | |
| *** sridhar_ram1 has quit IRC | 02:08 | |
| ianw | but then how do the jobs drop out of the gear queue? | 02:09 |
| *** eric_lopez has joined #openstack-infra | 02:10 | |
| greghaynes | ianw: whenever the client who submitted jobs disconnects all the jobs it submitted are dropped | 02:10 |
| *** tlian has quit IRC | 02:10 | |
| *** elo has quit IRC | 02:12 | |
| ianw | greghaynes: (sorry for being dense) but isn't the build job added by nodepool thread, then consumed by builder thread? aren't they different clients? | 02:12 |
| *** wei-wang has joined #openstack-infra | 02:14 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Support old setuptools for LTS distro venvs https://review.openstack.org/269999 | 02:15 |
| greghaynes | ianw: yes, but the builder only consumes the jobs one at a time, nodepool submits all the upload jobs at once | 02:15 |
| greghaynes | ianw: before the upload jobs were all consumed nodepool lost the gear connection which deleted all the upload jobs it submitted which were queued | 02:15 |
| ianw | ok ... but not getting how the timeone in the builder thread caused the nodepool side to drop the connection? | 02:16 |
| greghaynes | ianw: oh, that isn't the cause of the connection drop, we don't have any clear idea what caused that | 02:17 |
| ianw | ahh, ok | 02:17 |
| *** zhurong has joined #openstack-infra | 02:18 | |
| *** changzhi has joined #openstack-infra | 02:20 | |
| *** thorst has quit IRC | 02:20 | |
| *** thorst has joined #openstack-infra | 02:21 | |
| *** Qiming has quit IRC | 02:22 | |
| ianw | greghaynes: i see, so that's why you're thinking keepalive. cause this connection would be mostly idle | 02:22 |
| *** Qiming has joined #openstack-infra | 02:22 | |
| greghaynes | Ya | 02:22 |
| greghaynes | Although I realized it actually isn't very idle | 02:23 |
| *** yamamoto has joined #openstack-infra | 02:23 | |
| *** thorst has quit IRC | 02:25 | |
| *** woodster_ has quit IRC | 02:26 | |
| *** baoli has joined #openstack-infra | 02:28 | |
| *** rhallisey has quit IRC | 02:28 | |
| *** Jeffrey4l has joined #openstack-infra | 02:31 | |
| *** tlian2 has quit IRC | 02:32 | |
| *** julim has quit IRC | 02:32 | |
| *** dingyichen has quit IRC | 02:33 | |
| *** baoli has quit IRC | 02:33 | |
| *** akshai has joined #openstack-infra | 02:35 | |
| *** yamamoto has quit IRC | 02:37 | |
| *** yamamoto has joined #openstack-infra | 02:37 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Support entry point patching on setuptools < 12 https://review.openstack.org/269999 | 02:37 |
| *** otter768 has joined #openstack-infra | 02:37 | |
| *** hdd has joined #openstack-infra | 02:40 | |
| *** akshai has quit IRC | 02:40 | |
| *** otter768 has quit IRC | 02:42 | |
| *** dingyichen has joined #openstack-infra | 02:46 | |
| *** yamamoto has quit IRC | 02:46 | |
| Thelo | Hi all, quick question: in the zuul gate, what could be the reason for a test to restart? | 02:46 |
| ianw | Thelo: someone uploaded a new change? | 02:47 |
| Thelo | iamw: no | 02:48 |
| *** bpokorny has joined #openstack-infra | 02:48 | |
| clarkb | change ahed of it failed | 02:48 |
| *** keedya has quit IRC | 02:52 | |
| *** vilobhmm111 has left #openstack-infra | 02:54 | |
| *** apoorvad has quit IRC | 02:55 | |
| Thelo | clarkb: go it , thanks | 02:56 |
| Thelo | got it* | 02:56 |
| zxiiro | is there anyone around who can help me figure out why my zuul instance won't parse it's queue? jobs get queued up but no workers ever take care of it. I confirmed that if I directly queue a job in gearman it works. for some reason my zuul side of things doesn't seem to do anything | 02:57 |
| *** dgonzalez has quit IRC | 03:03 | |
| *** eric_lopez has quit IRC | 03:07 | |
| *** tricksters has joined #openstack-infra | 03:08 | |
| *** bardia_ has quit IRC | 03:08 | |
| *** baoli has joined #openstack-infra | 03:10 | |
| *** otter768 has joined #openstack-infra | 03:10 | |
| *** binbincong has joined #openstack-infra | 03:11 | |
| *** dgonzalez has joined #openstack-infra | 03:13 | |
| *** Qiming has quit IRC | 03:15 | |
| *** doug-fish has joined #openstack-infra | 03:16 | |
| *** bpokorny_ has joined #openstack-infra | 03:17 | |
| *** hdd has quit IRC | 03:21 | |
| *** bpokorny has quit IRC | 03:21 | |
| *** doug-fish has quit IRC | 03:21 | |
| *** _et_ has quit IRC | 03:21 | |
| *** bpokorny_ has quit IRC | 03:22 | |
| *** cody-somerville_ has joined #openstack-infra | 03:22 | |
| *** oomichi is now known as oomichi_away | 03:23 | |
| *** thorst has joined #openstack-infra | 03:25 | |
| *** tricksters has quit IRC | 03:26 | |
| *** elo has joined #openstack-infra | 03:26 | |
| *** thorst has quit IRC | 03:30 | |
| Thelo | Hi, I get some error in my CI when doing git check out (http://128.199.204.149/08/268608/3/check/dsvm-tempest-ITRI-DISCO-cinder-driver/81e2aa5/logs/devstack-gate-setup-workspace-new.txt) has anyone faced this before? | 03:32 |
| zxiiro | ok i figured out the zuul problem | 03:33 |
| zxiiro | needed to configure and run a merger process which wasn't immediately obvious to me | 03:33 |
| zxiiro | I shall document this and contribute a getting started guide | 03:36 |
| *** yamamoto_ has joined #openstack-infra | 03:39 | |
| *** dgonzalez has quit IRC | 03:39 | |
| *** dgonzalez has joined #openstack-infra | 03:40 | |
| *** sdake has quit IRC | 03:41 | |
| *** sdake has joined #openstack-infra | 03:41 | |
| *** dims has quit IRC | 03:42 | |
| *** armax has quit IRC | 03:42 | |
| *** eric_lopez has joined #openstack-infra | 03:46 | |
| *** elo has quit IRC | 03:46 | |
| *** arxcruz has quit IRC | 03:49 | |
| *** baoli has quit IRC | 03:54 | |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Bump oslo.versionedobjects minimum to 1.4.0 https://review.openstack.org/270591 | 03:58 |
| *** keedya has joined #openstack-infra | 04:01 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 04:02 |
| *** flwang1 has quit IRC | 04:05 | |
| *** vgridnev has joined #openstack-infra | 04:06 | |
| *** Hal has joined #openstack-infra | 04:07 | |
| *** Hal is now known as Guest28825 | 04:08 | |
| *** aeng has joined #openstack-infra | 04:09 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Run ovs-native jobs on the periodic-qa queue https://review.openstack.org/270595 | 04:10 |
| *** mriedem has quit IRC | 04:12 | |
| *** camunoz has joined #openstack-infra | 04:13 | |
| *** sdake_ has joined #openstack-infra | 04:13 | |
| *** eric_lopez has quit IRC | 04:15 | |
| *** sdake has quit IRC | 04:15 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: WIP: Add new django extraction https://review.openstack.org/266565 | 04:16 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: WIP: magnum-ui use generic django translation processing https://review.openstack.org/270596 | 04:16 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: WIP: magnum-ui use generic django translation processing https://review.openstack.org/270596 | 04:17 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: WIP: Add new django extraction https://review.openstack.org/266565 | 04:17 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: add support for package-installs to gentoo https://review.openstack.org/270597 | 04:19 |
| *** armax has joined #openstack-infra | 04:20 | |
| *** claudiub has quit IRC | 04:21 | |
| *** FallenPe_ has joined #openstack-infra | 04:22 | |
| *** sdake has joined #openstack-infra | 04:22 | |
| *** FallenPe_ has quit IRC | 04:23 | |
| *** sdake_ has quit IRC | 04:24 | |
| *** FallenPegasus has quit IRC | 04:25 | |
| *** hdd has joined #openstack-infra | 04:25 | |
| *** thorst has joined #openstack-infra | 04:27 | |
| *** ddieterly has quit IRC | 04:28 | |
| zxiiro | does zuul's cron timer support things like "H H * * *" or "@daily"? or does it only support numbers in those fields? | 04:28 |
| asselin__ | Thelo, you have a conflict between what you are setting up in your cinder repo and what zuul is setting up. | 04:29 |
| clarkb | it doesnt support @daily | 04:29 |
| asselin__ | Thelo, you should make sure you make your changes after zuul is done | 04:29 |
| zxiiro | ah ok | 04:30 |
| asselin__ | Thelo, either that...or somehow you're re-using the same slave. | 04:30 |
| *** otter768 has quit IRC | 04:31 | |
| *** thorst has quit IRC | 04:34 | |
| *** vgridnev has quit IRC | 04:35 | |
| *** bpokorny has joined #openstack-infra | 04:38 | |
| *** tjones has joined #openstack-infra | 04:40 | |
| *** Thelo_ has joined #openstack-infra | 04:41 | |
| *** Ramanjaneya_ has joined #openstack-infra | 04:42 | |
| *** Thelo_ has quit IRC | 04:46 | |
| *** coolsvap|away is now known as coolsvap | 04:49 | |
| *** nelsnels_ has quit IRC | 04:53 | |
| *** Thelo_ has joined #openstack-infra | 04:55 | |
| *** ashleighfarnham has quit IRC | 04:56 | |
| *** nelsnelson has joined #openstack-infra | 04:56 | |
| *** treaki_ has joined #openstack-infra | 04:57 | |
| *** tjones has quit IRC | 04:58 | |
| *** vgridnev has joined #openstack-infra | 04:58 | |
| *** Ramanjaneya_ has quit IRC | 04:59 | |
| *** Ramanjaneya has joined #openstack-infra | 05:00 | |
| *** treaki__ has quit IRC | 05:01 | |
| *** harlowja_at_home has joined #openstack-infra | 05:02 | |
| *** david-lyle has quit IRC | 05:02 | |
| *** piet has joined #openstack-infra | 05:03 | |
| *** david-lyle has joined #openstack-infra | 05:03 | |
| *** FallenPegasus has joined #openstack-infra | 05:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add mirror_update.openstack.org https://review.openstack.org/270520 | 05:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Document adding a kerberos service principal https://review.openstack.org/270559 | 05:04 |
| *** markvoelker has joined #openstack-infra | 05:07 | |
| *** Thelo_ has quit IRC | 05:10 | |
| *** markvoelker has quit IRC | 05:12 | |
| *** ccrouch has quit IRC | 05:12 | |
| *** markvoelker has joined #openstack-infra | 05:12 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 05:16 |
| *** aeng has quit IRC | 05:23 | |
| *** zhurong has quit IRC | 05:26 | |
| *** rlandy has quit IRC | 05:28 | |
| *** ddieterly has joined #openstack-infra | 05:28 | |
| *** doug-fish has joined #openstack-infra | 05:30 | |
| *** tiswanso has quit IRC | 05:31 | |
| *** yuanying has joined #openstack-infra | 05:32 | |
| *** thorst has joined #openstack-infra | 05:33 | |
| *** ddieterly has quit IRC | 05:33 | |
| *** doug-fish has quit IRC | 05:35 | |
| *** yuanying_ has quit IRC | 05:36 | |
| *** Zhongjun_ has quit IRC | 05:38 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 05:39 |
| *** thorst has quit IRC | 05:40 | |
| *** vivekd has quit IRC | 05:44 | |
| *** doug-fish has joined #openstack-infra | 05:44 | |
| *** vgridnev has quit IRC | 05:45 | |
| gmann | lifeless: pip8 distutils issue for py27/docs etc seems to be fixed with pbr > 1.0 | 05:45 |
| gmann | lifeless: with that pbr version it does not install pip version 8 and ending with 1.7.* which fix the issue - https://review.openstack.org/#/c/270606/1 | 05:46 |
| *** jaosorior has joined #openstack-infra | 05:47 | |
| gmann | lifeless: i tried to find the reason of taking pip < 8 in case of pbr >1.0 but did not get complete stack there | 05:47 |
| gmann | lifeless: is this because of virtualenv thing ? | 05:47 |
| *** doug-fish has quit IRC | 05:49 | |
| lifeless | gmann: uh, I have no idea , pbr doesn't influence pip versions | 05:51 |
| lifeless | gmann: if it did, devstack would have not broken, since pbr is used there too | 05:52 |
| *** harlowja_at_home has quit IRC | 05:52 | |
| gmann | lifeless: yea, but for docs and py27 it seems to fix (my guess virtualenv cap as with pbr it is <14 ?) | 05:53 |
| lifeless | no, pbr doesn't trigger virtualenv installation either, nor constrain it | 05:53 |
| *** zhurong has joined #openstack-infra | 05:53 | |
| lifeless | I don't know what you're seeing, but its not pbr fixing it | 05:53 |
| gmann | lifeless: humm. diff from log seems that failure one picked pip from pbr venv and passed one not | 05:54 |
| gmann | lifeless: passed - http://logs.openstack.org/06/270606/1/check/gate-nova-specs-python27/f689acc/tox/py27-2.log | 05:55 |
| gmann | lifeless: failure http://logs.openstack.org/64/269664/3/check/gate-nova-specs-python27/95f7118/tox/py27-2.log | 05:55 |
| lifeless | gmann: the first one has pip 7.1 | 05:57 |
| gmann | lifeless: yea this is with pbr >1.0 | 05:57 |
| gmann | lifeless: may be pbr <0.11.0 has pip in requirement.txt and endup with pip update to 8 | 05:58 |
| *** aopincar has joined #openstack-infra | 06:00 | |
| *** markvoelker_ has joined #openstack-infra | 06:01 | |
| *** abregman has joined #openstack-infra | 06:02 | |
| *** flwang1 has joined #openstack-infra | 06:02 | |
| *** hichihara has joined #openstack-infra | 06:02 | |
| gmann | lifeless: in failure case (pbr<0.11.0) t trigger to upgrade pip to 8 because pbr has that in requirement.txt - http://logs.openstack.org/64/269664/3/check/gate-nova-specs-python27/95f7118/tox/py27-1.log | 06:04 |
| lifeless | gmann: old pbr does indeed have a dep on pip, which could be messing things up - but we don't support old pbr anywhere except - grudglingly - on kilo :) | 06:04 |
| gmann | Collecting pip (from pbr<1.0,>=0.6->-r /home/jenkins/workspace/gate-nova-specs-python27/requirements.txt (line 2)) | 06:04 |
| gmann | Downloading http://pypi.IAD.openstack.org/packages/py2.py3/p/pip/pip-8.0.0-py2.py3-none-any.whl (1.2MB) | 06:04 |
| lifeless | gmann: regardless, the fix for this will be pip 8.1 | 06:04 |
| * notmorgan sees a lifeless. /me waves. /me decides that is it likely time to check out for the evening... or try to | 06:04 | |
| *** markvoelker has quit IRC | 06:05 | |
| lifeless | notmorgan: you should be long gone. /waves | 06:05 |
| *** markvoelker has joined #openstack-infra | 06:05 | |
| gmann | lifeless: yea pip 8.1 is fix | 06:06 |
| gmann | lifeless: all spec repo seems to use old pbr <1.0 | 06:06 |
| notmorgan | lifeless: yeah. well... you know how it goes :P | 06:06 |
| *** markvoelker_ has quit IRC | 06:06 | |
| *** flwang1 has quit IRC | 06:06 | |
| lifeless | gmann: they probably aren't syncing with requirements. Doing a manual sync would be a good thign :) | 06:07 |
| gmann | lifeless: yea, but not sure why not syncing there. | 06:07 |
| lifeless | because noone has set it up | 06:07 |
| lifeless | I don't think there is any deeper reason | 06:08 |
| gmann | lifeless: humm | 06:08 |
| *** vgridnev has joined #openstack-infra | 06:08 | |
| *** ashishb has joined #openstack-infra | 06:11 | |
| *** oomichi_away is now known as oomichi | 06:11 | |
| *** bpokorny has quit IRC | 06:12 | |
| *** piet has quit IRC | 06:14 | |
| *** ayoung has quit IRC | 06:16 | |
| wei-wang | Hi, does this confirmed: https://bugs.launchpad.net/neutron/+bug/1533194 ?, it seems block the https://review.openstack.org/#/c/269017/7 | 06:17 |
| openstack | Launchpad bug 1533194 in tempest "Gate failures for neutron in TestGettingAddress" [Undecided,New] | 06:17 |
| *** yfried has quit IRC | 06:17 | |
| *** Jeffrey4l has quit IRC | 06:18 | |
| *** Jeffrey4l has joined #openstack-infra | 06:19 | |
| Thelo | wei-wang: looks like this one had the same issue : https://review.openstack.org/#/c/267725/ | 06:22 |
| *** Guest28825 has quit IRC | 06:23 | |
| *** _nadya_ has joined #openstack-infra | 06:24 | |
| wei-wang | Thelo: thanks :-) | 06:24 |
| *** vgridnev has quit IRC | 06:26 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/267962 | 06:27 |
| *** ddieterly has joined #openstack-infra | 06:30 | |
| *** derekh has joined #openstack-infra | 06:31 | |
| *** [1]Thelo has joined #openstack-infra | 06:32 | |
| *** otter768 has joined #openstack-infra | 06:32 | |
| *** unicell has joined #openstack-infra | 06:33 | |
| *** Thelo has quit IRC | 06:34 | |
| *** [1]Thelo is now known as Thelo | 06:34 | |
| *** ddieterly has quit IRC | 06:34 | |
| *** otter768 has quit IRC | 06:37 | |
| *** sdake has quit IRC | 06:38 | |
| *** thorst has joined #openstack-infra | 06:38 | |
| *** Hal has joined #openstack-infra | 06:38 | |
| *** Hal is now known as Guest11845 | 06:38 | |
| *** gyee has quit IRC | 06:39 | |
| *** thorst has quit IRC | 06:45 | |
| *** vgridnev has joined #openstack-infra | 06:47 | |
| *** HeOS has joined #openstack-infra | 06:49 | |
| openstackgerrit | Kuo-tung Kao proposed openstack/requirements: update docker-py to 1.6.0 https://review.openstack.org/270625 | 06:55 |
| *** ashleighfarnham has joined #openstack-infra | 06:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/elastic-recheck: Remove argparse from requirements https://review.openstack.org/270414 | 06:58 |
| openstackgerrit | Khai Do proposed openstack-infra/jeepyb: Fix handling of Gerrit system groups https://review.openstack.org/270628 | 06:59 |
| *** hdd has quit IRC | 06:59 | |
| zaro | fungi, anteaya ^ this and https://review.openstack.org/#/c/270537 should fix the sandbox repo situation. | 07:00 |
| *** ashleighfarnham has quit IRC | 07:01 | |
| *** achanda has quit IRC | 07:06 | |
| *** FallenPegasus has quit IRC | 07:08 | |
| *** gildub has quit IRC | 07:09 | |
| *** yfried has joined #openstack-infra | 07:13 | |
| AJaeger | lifeless: morning, sorry, had to leave last night. | 07:20 |
| *** rcernin has joined #openstack-infra | 07:20 | |
| AJaeger | lifeless: yes, I consider argparse a cleanup - and it might fix a few ones as well. | 07:21 |
| AJaeger | lifeless: But python 2.6 support is not best effort for the clients anymore, they even removed python 2.6 from setup.cfg. The only project where we discussed best effort for on the mailing list was pbr. | 07:21 |
| AJaeger | thanks, zaro for tackling the sandbox! | 07:22 |
| zaro | AJaeger: np, hey 270537 seems to have linter issues again. is that line too long error? | 07:24 |
| zaro | or something else? ugh, puppet logs drive me nuts! | 07:25 |
| AJaeger | zaro: we disabled lines to long for bashate - but let me check | 07:25 |
| lifeless | AJaeger: shrug - I don't hugely care any which way, but I did want folk to think about this rather than knee-jerk reacting, because there is nothing wrong with depending on argparse [because so many thihngs did/do - we have to make pip work any which way) | 07:25 |
| *** wei-wang has quit IRC | 07:25 | |
| AJaeger | zaro, error is: http://logs.openstack.org/37/270537/2/check/gate-system-config-linters/2e5070b/console.html#_2016-01-20_23_49_06_107 | 07:25 |
| AJaeger | zaro: that'S tabs, whitespace, wrong indent | 07:26 |
| *** vgridnev has quit IRC | 07:26 | |
| *** wei-wang has joined #openstack-infra | 07:26 | |
| AJaeger | lifeless: yes, pip has to work with depending on argparse. I understand the situation now much better | 07:27 |
| *** sshnaidm has quit IRC | 07:27 | |
| * AJaeger is not asking for futher releases because of argparse and hopes pip 8.1/8.0.1 comes out soon | 07:28 | |
| zaro | AJaeger: ohh, the [E]. thanks! | 07:28 |
| AJaeger | zaro: yes, the [E]. | 07:29 |
| AJaeger | [W] = Warning | 07:29 |
| *** esikachev has joined #openstack-infra | 07:30 | |
| *** ddieterly has joined #openstack-infra | 07:31 | |
| *** nmagnezi has joined #openstack-infra | 07:34 | |
| *** jcoufal has joined #openstack-infra | 07:35 | |
| *** ddieterly has quit IRC | 07:36 | |
| *** davidhadas_ has joined #openstack-infra | 07:38 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Add helper script to delete Gerrit groups https://review.openstack.org/270537 | 07:38 |
| openstackgerrit | Khai Do proposed openstack-infra/jeepyb: Fix handling of Gerrit system groups https://review.openstack.org/270628 | 07:40 |
| *** thorst has joined #openstack-infra | 07:43 | |
| *** migi_ is now known as migi | 07:43 | |
| *** unicell has quit IRC | 07:49 | |
| *** unicell has joined #openstack-infra | 07:49 | |
| *** andymaier has joined #openstack-infra | 07:50 | |
| *** thorst has quit IRC | 07:50 | |
| *** jyuso1 has joined #openstack-infra | 07:51 | |
| *** jaosorior has quit IRC | 07:53 | |
| *** jtomasek has joined #openstack-infra | 07:55 | |
| *** nihilife1 is now known as nihilifer | 07:55 | |
| *** otter768 has joined #openstack-infra | 08:00 | |
| *** ihrachys has joined #openstack-infra | 08:01 | |
| *** ayoung has joined #openstack-infra | 08:01 | |
| *** otter768_ has joined #openstack-infra | 08:02 | |
| *** arxcruz has joined #openstack-infra | 08:02 | |
| *** mkoderer_ is now known as mkoderer | 08:03 | |
| *** davidhadas_ has quit IRC | 08:03 | |
| *** otter768_ has quit IRC | 08:04 | |
| *** otter76__ has joined #openstack-infra | 08:04 | |
| *** otter768 has quit IRC | 08:04 | |
| *** otter76__ has quit IRC | 08:05 | |
| *** jaosorior has joined #openstack-infra | 08:06 | |
| *** otter768 has joined #openstack-infra | 08:06 | |
| *** achanda has joined #openstack-infra | 08:06 | |
| Thelo | not sure it is the right place to ask but where to take a look at then the gate-grenade-dsvm-multinode job fails (https://jenkins02.openstack.org/job/gate-grenade-dsvm-multinode/1988/console) from the console I dont see the root cause | 08:06 |
| *** mpavone has joined #openstack-infra | 08:07 | |
| *** otter768 has quit IRC | 08:07 | |
| *** otter768 has joined #openstack-infra | 08:07 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add some swap to the undercloud https://review.openstack.org/270274 | 08:08 |
| openstackgerrit | Kuo-tung Kao proposed openstack/requirements: update docker-py to 1.6.0 https://review.openstack.org/270625 | 08:09 |
| *** otter768_ has joined #openstack-infra | 08:09 | |
| *** otter768 has quit IRC | 08:09 | |
| *** otter768 has joined #openstack-infra | 08:11 | |
| *** otter768_ has quit IRC | 08:11 | |
| *** otter768 has quit IRC | 08:11 | |
| *** HeOS has quit IRC | 08:12 | |
| *** achanda has quit IRC | 08:13 | |
| *** Jeffrey4l has quit IRC | 08:13 | |
| *** placko has joined #openstack-infra | 08:13 | |
| *** esikachev has quit IRC | 08:13 | |
| *** markus_z has joined #openstack-infra | 08:14 | |
| *** ildikov has quit IRC | 08:15 | |
| *** Guest11845 has quit IRC | 08:18 | |
| *** Hal has joined #openstack-infra | 08:18 | |
| *** scheuran has joined #openstack-infra | 08:18 | |
| *** Hal is now known as Guest8525 | 08:18 | |
| *** hashar has joined #openstack-infra | 08:20 | |
| *** ajo_ is now known as ajo | 08:24 | |
| *** zeih has joined #openstack-infra | 08:25 | |
| *** Jeffrey4l has joined #openstack-infra | 08:26 | |
| *** isaacb has joined #openstack-infra | 08:26 | |
| *** vgridnev has joined #openstack-infra | 08:28 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config: Enable Zaqar in Heat integration jobs https://review.openstack.org/270659 | 08:28 |
| *** samuelBartel has joined #openstack-infra | 08:29 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc https://review.openstack.org/262028 | 08:30 |
| *** ddieterly has joined #openstack-infra | 08:33 | |
| *** fhubik has joined #openstack-infra | 08:34 | |
| *** fhubik is now known as fhubik_brb | 08:34 | |
| *** yfried has quit IRC | 08:35 | |
| *** yfried has joined #openstack-infra | 08:35 | |
| *** hichihara has quit IRC | 08:35 | |
| *** esikachev has joined #openstack-infra | 08:35 | |
| *** armax has quit IRC | 08:36 | |
| *** ddieterly has quit IRC | 08:37 | |
| *** Guest8525 has quit IRC | 08:38 | |
| *** shardy has joined #openstack-infra | 08:39 | |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: [WIP] trying to add one block node https://review.openstack.org/270423 | 08:39 |
| *** dingyichen has quit IRC | 08:44 | |
| *** andrearo1 is now known as andrearosa | 08:44 | |
| *** jlanoux has joined #openstack-infra | 08:47 | |
| *** thorst has joined #openstack-infra | 08:48 | |
| *** fhubik_brb is now known as fhubik | 08:50 | |
| *** yfried_ has joined #openstack-infra | 08:50 | |
| *** dizquierdo has joined #openstack-infra | 08:52 | |
| *** derekh has quit IRC | 08:52 | |
| *** yfried has quit IRC | 08:55 | |
| *** thorst has quit IRC | 08:55 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/requirements: Adding all specs repo in projects.txt https://review.openstack.org/270671 | 08:57 |
| *** dtantsur|afk is now known as dtantsur | 08:58 | |
| *** yolanda has joined #openstack-infra | 08:58 | |
| gmann | lifeless: ^^ | 08:59 |
| lifeless | gmann: have you pushed up changes to make them verify reqs first? See the README in requirements | 08:59 |
| ihrachys | sdague: ping re mtu patches in devstack[-gate] to fix multinode neutron setups | 08:59 |
| gmann | lifeless: oh, checking.. | 09:00 |
| openstackgerrit | Dina Belova proposed openstack-infra/project-config: Add py27 job for performance-docs https://review.openstack.org/270088 | 09:02 |
| gmann | lifeless: got it, i will check and push accordingly. Thanks | 09:02 |
| AJaeger | gmann: most (all?) of the specs repos you added do not have a check-requirements job | 09:03 |
| AJaeger | gmann: Also, the purpose of requirements to allow cloud admins to install projects together. So, specs are out of scope for those. We can add them but I'm not sure it's worth the effort. | 09:03 |
| gmann | AJaeger: yea, just got that from lifeless. I did not see the readme. I will add and proposed accordingly | 09:03 |
| *** ildikov has joined #openstack-infra | 09:04 | |
| gmann | AJaeger: humm/ but in many case gate can be broken due to lib upgrade | 09:06 |
| gmann | AJaeger: but your point also seems valid | 09:06 |
| lifeless | AJaeger: requirements works in two differnet directions | 09:07 |
| lifeless | AJaeger: if we were adding specs to g-r, your concern would be relevant - but this is just keeping the specs up to date, so we're not running insecure deps on them. like pbr < 1! | 09:07 |
| *** dizquierdo has quit IRC | 09:08 | |
| lifeless | AJaeger: sorry if that came across harsh; 10pm, I shouldn't be here :) | 09:08 |
| gmann | lifeless: too late for you :) | 09:08 |
| gmann | AJaeger: projects.txt used for auto sync req and manual etc right | 09:09 |
| *** sputnik13 has joined #openstack-infra | 09:10 | |
| *** markvoelker has quit IRC | 09:12 | |
| *** HeOS has joined #openstack-infra | 09:12 | |
| *** Liuqing has joined #openstack-infra | 09:12 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/requirements: Add sahara-scenario to projects list https://review.openstack.org/270676 | 09:13 |
| AJaeger | lifeless: no worries | 09:13 |
| AJaeger | lifeless: we can do that syncing indeed | 09:14 |
| AJaeger | lifeless: I had some hard fights in the past when I wanted the syncing but needed extra dependencies for documentation tools as well and finally gave up ;( | 09:15 |
| *** jistr has joined #openstack-infra | 09:16 | |
| lifeless | AJaeger: :/ | 09:16 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Add check-requirements to sahara-scenario project https://review.openstack.org/270680 | 09:18 |
| AJaeger | lifeless: I'll happily +2 the project-config changes if the relevant spec project "owners" are fine with it. | 09:18 |
| *** rossella_s has joined #openstack-infra | 09:18 | |
| lifeless | cool | 09:19 |
| AJaeger | lifeless: I807b061cf697dc79b26ffdfbe023c32600ddd346 was the change to remove some repos | 09:20 |
| *** daemontool has joined #openstack-infra | 09:20 | |
| AJaeger | lifeless: I have to leave now - and wish you a good night! | 09:21 |
| lifeless | AJaeger: ight | 09:21 |
| *** rossella_s has quit IRC | 09:21 | |
| lifeless | night | 09:21 |
| *** rossella_s has joined #openstack-infra | 09:21 | |
| gmann | lifeless: AJaeger same issue here some spec repo has requirement which are not in g-r | 09:21 |
| gmann | AJaeger: and not sure those are valid one to go in g-r | 09:21 |
| AJaeger | gmann: start with those that are fine ;) | 09:22 |
| *** sfinucan has joined #openstack-infra | 09:22 | |
| *** Guest45232 is now known as BobBall | 09:22 | |
| *** derekh has joined #openstack-infra | 09:23 | |
| gmann | AJaeger: yea, in first i got hit by qa one itself :) | 09:23 |
| AJaeger | ;) | 09:23 |
| gmann | yasfb one :) | 09:24 |
| gmann | anyways lets see what all can be done in check requirements and based on that we can add in auto sync | 09:24 |
| daemontool | AJaeger, if you get the chance, could you please help me understand how can i fix this http://logs.openstack.org/89/269189/1/check/gate-freezer-devstack-dsvm/69c6c96/console.html | 09:26 |
| daemontool | ERROR: branch not allowed by features matrix: 1.2.0 | 09:26 |
| daemontool | the review is this one https://review.openstack.org/#/c/269189/ | 09:26 |
| daemontool | we branched stable/1.2.0 and now the dsvm gate job fail with that error | 09:26 |
| daemontool | I think is more or less the same issue as http://lists.openstack.org/pipermail/openstack-dev/2015-November/080021.html | 09:27 |
| daemontool | or anyone around can help me with that | 09:27 |
| *** jordanP has joined #openstack-infra | 09:27 | |
| openstackgerrit | Evgeny Antyshev proposed openstack-infra/ciwatch: Fix "verified" attribute assignment https://review.openstack.org/270688 | 09:30 |
| *** mhickey has joined #openstack-infra | 09:31 | |
| *** exploreshaifali has joined #openstack-infra | 09:31 | |
| *** vgridnev has quit IRC | 09:32 | |
| *** vgridnev has joined #openstack-infra | 09:33 | |
| *** ddieterly has joined #openstack-infra | 09:34 | |
| *** Hal has joined #openstack-infra | 09:35 | |
| *** Hal is now known as Guest85416 | 09:36 | |
| *** ddieterly has quit IRC | 09:38 | |
| *** dtardivel has joined #openstack-infra | 09:38 | |
| *** jcoufal has quit IRC | 09:40 | |
| *** jyuso1 has quit IRC | 09:40 | |
| *** markvoelker has joined #openstack-infra | 09:42 | |
| *** sshnaidm has joined #openstack-infra | 09:44 | |
| *** acabot has quit IRC | 09:45 | |
| *** jordanP has quit IRC | 09:47 | |
| *** markvoelker has quit IRC | 09:48 | |
| *** jpr1 has joined #openstack-infra | 09:50 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Add translation jobs for python-neutronclient https://review.openstack.org/270693 | 09:51 |
| *** jordanP has joined #openstack-infra | 09:51 | |
| *** ociuhandu has quit IRC | 09:52 | |
| *** Liuqing has quit IRC | 09:52 | |
| *** thorst has joined #openstack-infra | 09:53 | |
| *** sdake has joined #openstack-infra | 09:55 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: WIP: Prepare new translation functions for python projects https://review.openstack.org/270697 | 09:57 |
| openstackgerrit | Merged openstack-infra/project-config: Keystone run py34 unit tests only on master https://review.openstack.org/269896 | 09:58 |
| *** acabot has joined #openstack-infra | 09:58 | |
| *** fhubik is now known as fhubik_brb | 09:59 | |
| openstackgerrit | Merged openstack-infra/project-config: Add osprofiler events to #openstack-oslo channel https://review.openstack.org/270238 | 09:59 |
| *** thorst has quit IRC | 10:00 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/project-config: Add non voting docs check jobs for Solar https://review.openstack.org/270698 | 10:01 |
| *** sputnik13 has quit IRC | 10:02 | |
| *** ashishb has quit IRC | 10:02 | |
| *** openstackgerrit has quit IRC | 10:02 | |
| *** openstackgerrit has joined #openstack-infra | 10:02 | |
| *** fhubik_brb is now known as fhubik | 10:03 | |
| *** pblaho has quit IRC | 10:03 | |
| *** electrofelix has joined #openstack-infra | 10:03 | |
| *** rguillebert has joined #openstack-infra | 10:04 | |
| *** sdake has quit IRC | 10:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove fuel-library puppet 3.6 and 3.7 unit jobs https://review.openstack.org/270340 | 10:06 |
| *** daemontool_ has joined #openstack-infra | 10:07 | |
| *** ashishb has joined #openstack-infra | 10:07 | |
| *** daemontool has quit IRC | 10:07 | |
| *** jaosorior has quit IRC | 10:09 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Remove py27-scenario-unit check from sahara https://review.openstack.org/270704 | 10:09 |
| *** jaosorior has joined #openstack-infra | 10:10 | |
| *** jaosorior has quit IRC | 10:10 | |
| *** achanda has joined #openstack-infra | 10:11 | |
| *** jaosorior has joined #openstack-infra | 10:11 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/tripleo-ci: Remove unused urllib https://review.openstack.org/261182 | 10:12 |
| daemontool_ | yolanda, ping | 10:13 |
| yolanda | hi daemontool | 10:13 |
| *** diltram has left #openstack-infra | 10:14 | |
| daemontool_ | hi yolanda I have an issue with if you get the chance, could you please help me understand how can i fix this http://logs.openstack.org/89/269189/1/check/gate-freezer-devstack-dsvm/69c6c96/console.html | 10:14 |
| *** daemontool_ is now known as daemontool | 10:14 | |
| daemontool | ERROR: branch not allowed by features matrix: 1.2.0 | 10:14 |
| *** ashishb has quit IRC | 10:14 | |
| daemontool | we branched stable/1.2.0 and now the dsvm gate job fail with that error | 10:14 |
| yolanda | i'm not very experienced with ACL configs on infra. AJaeger, are you around to take a look? | 10:15 |
| *** dtantsur is now known as dtantsur|brb | 10:15 | |
| daemontool | I think is more or less the same issue as http://lists.openstack.org/pipermail/openstack-dev/2015-November/080021.html | 10:15 |
| daemontool | the review is this one https://review.openstack.org/#/c/269189/ | 10:15 |
| daemontool | yolanda, ok ty | 10:15 |
| *** wei-wang has quit IRC | 10:16 | |
| yolanda | sorry but you seem to know more than me... | 10:16 |
| *** achanda has quit IRC | 10:17 | |
| yolanda | i see code relies on an env var called ALLOWED_BRANCHES | 10:17 |
| *** gildub has joined #openstack-infra | 10:19 | |
| yolanda | daemontool, this thread looks useful as well https://openstack.nimeyo.com/65815/openstack-infra-devstack-gnocchi-unable-devstack-with-stable | 10:19 |
| *** changzhi has quit IRC | 10:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Adds openstack-doc-tools as a group for doc tools https://review.openstack.org/268313 | 10:20 |
| *** doug-fish has joined #openstack-infra | 10:22 | |
| openstackgerrit | Merged openstack-infra/project-config: Make gate-fuel-library-puppet-syntax-4-dsvm-centos7 job voting in the check queue https://review.openstack.org/270256 | 10:23 |
| *** Liuqing has joined #openstack-infra | 10:24 | |
| *** Odd_Blok1 is now known as Odd_Bloke | 10:25 | |
| daemontool | yolanda, yes thanks | 10:26 |
| *** doug-fish has quit IRC | 10:27 | |
| daemontool | yolanda, in the link the guy says "In the past, we set OVERRIDEZUULBRANCH=stable/kilo in the Gnocchi jobs | 10:27 |
| daemontool | for some stable branches (we did for stable/1.0), but honestly patching | 10:27 |
| daemontool | the infra each time we do a stable release is getting painful." | 10:27 |
| yolanda | yes, saw that comment | 10:28 |
| openstackgerrit | Merged openstack-infra/project-config: fix ceilometer integration job to handle slow gate https://review.openstack.org/269846 | 10:28 |
| openstackgerrit | Ghanshyam Mann proposed openstack/requirements: Add yasfb (Sphinx extension) in global requirements https://review.openstack.org/270720 | 10:28 |
| daemontool | where I can set that in the service job (Gnocchi job) | 10:28 |
| daemontool | ? | 10:28 |
| *** vkramskikh has quit IRC | 10:29 | |
| *** StevenK has quit IRC | 10:29 | |
| *** rm_work has quit IRC | 10:29 | |
| *** zeus has quit IRC | 10:29 | |
| *** zeus has joined #openstack-infra | 10:30 | |
| *** zeus is now known as Guest65425 | 10:30 | |
| *** keedya has quit IRC | 10:31 | |
| yolanda | daemontool, i was looking for samples ast http://codesearch.openstack.org/?q=OVERRIDE_ZUUL_BRANCH&i=nope&files=&repos= | 10:31 |
| *** rm_work has joined #openstack-infra | 10:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Make the ironic-parallel job experimental https://review.openstack.org/269757 | 10:32 |
| yolanda | for example in ceilometer they use it http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/ceilometer.yaml | 10:32 |
| openstackgerrit | Merged openstack-infra/project-config: use ceilometerclient from git https://review.openstack.org/269302 | 10:32 |
| *** ddieterly has joined #openstack-infra | 10:34 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:35 | |
| daemontool | ah ok ty | 10:38 |
| daemontool | thanks a lot that's really helpful | 10:38 |
| *** ddieterly has quit IRC | 10:39 | |
| *** ldnunes has joined #openstack-infra | 10:39 | |
| *** markvoelker has joined #openstack-infra | 10:39 | |
| gibi | hello infra folks! Is it normal that if I have a commit chain with 6 commits then only the first two of them are progressing in the gate queue and the others are rejected with 'This change depends on a change that failed to merge.' ? | 10:39 |
| daemontool | yolanda, that same job will be used for all the branches or each branch will use it's own setting? | 10:39 |
| yolanda | daemontool, depending on how you configure, but normally the same job | 10:39 |
| gibi | the chain starts here https://review.openstack.org/#/c/247024 | 10:39 |
| *** amotoki has quit IRC | 10:40 | |
| daemontool | yolanda, ok | 10:40 |
| *** vkramskikh has joined #openstack-infra | 10:41 | |
| *** zhurong has quit IRC | 10:41 | |
| *** bogdando has joined #openstack-infra | 10:43 | |
| bogdando | can someone help with the gate failing https://review.openstack.org/#/c/270698 ? | 10:43 |
| *** markvoelker has quit IRC | 10:44 | |
| *** Guest85416 has quit IRC | 10:46 | |
| *** Jeffrey4l has quit IRC | 10:46 | |
| *** Hal has joined #openstack-infra | 10:47 | |
| *** Hal is now known as Guest96416 | 10:47 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 10:50 |
| *** StevenK has joined #openstack-infra | 10:53 | |
| *** dizquierdo has joined #openstack-infra | 10:55 | |
| *** e0ne has joined #openstack-infra | 10:57 | |
| *** Qiming has joined #openstack-infra | 10:58 | |
| *** ashleighfarnham has joined #openstack-infra | 10:58 | |
| *** thorst has joined #openstack-infra | 10:58 | |
| *** abregman is now known as abregman|afk | 10:58 | |
| *** ashleighfarnham has quit IRC | 11:02 | |
| *** dizquierdo has quit IRC | 11:03 | |
| *** dims has joined #openstack-infra | 11:04 | |
| *** thorst has quit IRC | 11:04 | |
| *** Qiming has quit IRC | 11:06 | |
| *** aysyd has joined #openstack-infra | 11:07 | |
| *** Nikolay_St has joined #openstack-infra | 11:08 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 11:09 |
| clarkb | daemontool: as fungi explained previously ylu have chosen to use a different branching strategy and one that devstack-gate does nkt understand | 11:10 |
| clarkb | daemontool: you can use the openstack branch model or you will bave to teach the tools what to do with your branches | 11:11 |
| *** _nadya_ has quit IRC | 11:12 | |
| *** kzaitsev_mb has quit IRC | 11:14 | |
| clarkb | gibi: it doesnt look like zuul has gotten far enough in its processing to consoder those changes | 11:15 |
| clarkb | gibi I do not see where the merge errors are listed | 11:15 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 11:17 |
| daemontool | clarkb, sorry I'm not sure I fully understand that | 11:18 |
| daemontool | what's the openstack branch model? | 11:18 |
| *** dizquierdo has joined #openstack-infra | 11:18 | |
| daemontool | like stable/kilo, stable/liberty etc? | 11:18 |
| clarkb | daemontool: our tooling understands stable/kilo stable/liberty stable/mitaka stable/newton and so forth | 11:18 |
| daemontool | ok | 11:18 |
| daemontool | so the reason why we are doing that | 11:18 |
| clarkb | stable/1.2.0 is not somethibg it knows whag to do with | 11:19 |
| daemontool | is because we can't branch stable/liberty | 11:19 |
| daemontool | if we didn't meet the openstack release timeline | 11:19 |
| clarkb | you cant't? | 11:19 |
| daemontool | so for this case we where instructed to do that | 11:19 |
| daemontool | I think is make sense | 11:19 |
| *** kzaitsev_mb has joined #openstack-infra | 11:19 | |
| daemontool | s/is/it/ | 11:19 |
| daemontool | but it looks like we can-t | 11:19 |
| clarkb | ok well it means that none of the tools understand what to do with your branch in integration testing | 11:20 |
| clarkb | you can teach the tools or change the branch | 11:20 |
| daemontool | clarkb, anyone that have a tool | 11:20 |
| daemontool | a service | 11:20 |
| daemontool | and wants the tool to work with stable/kilo and so on | 11:20 |
| daemontool | have the same issue | 11:20 |
| clarkb | yes they typically use a branch called stable/kilo | 11:21 |
| daemontool | but is your tool has been added to openstack | 11:21 |
| daemontool | after the stable/kilo timeline | 11:21 |
| daemontool | how do you do that... | 11:21 |
| daemontool | I'm asking because | 11:21 |
| daemontool | I'm not sure how to solve this | 11:21 |
| clarkb | you just make a branch in gerrit the same way you make any other branch | 11:22 |
| clarkb | need a name and branch root | 11:22 |
| daemontool | ok that's what I did | 11:22 |
| *** _nadya_ has joined #openstack-infra | 11:22 | |
| daemontool | *I think* :) | 11:23 |
| clarkb | yes but used stable/1.2.0 | 11:23 |
| daemontool | ok | 11:23 |
| clarkb | so the alternative is to teach devstack-gate what to do with stable/1.2.0 | 11:23 |
| daemontool | ok, I think I need help understanding how to do that | 11:24 |
| clarkb | I dont know that anyone does it so I dont have examplea | 11:24 |
| daemontool | =( | 11:24 |
| daemontool | ok | 11:24 |
| clarkb | I really think if the goal is to have a branch that works with kilo you should just rename your branch | 11:24 |
| *** ajo has quit IRC | 11:24 | |
| *** abregman|afk is now known as abregman | 11:25 | |
| daemontool | that's what I'm going to do, thanks | 11:25 |
| daemontool | a lot | 11:25 |
| *** ociuhandu has joined #openstack-infra | 11:25 | |
| *** ajo has joined #openstack-infra | 11:25 | |
| *** pblaho has joined #openstack-infra | 11:25 | |
| clarkb | for teaching devstack gate you can write your own features.yaml that explains what services to configure on that branch | 11:26 |
| clarkb | then you also need to set the branch override for git checkouts so prpjects with say kilo get kilo checked out instead of master | 11:26 |
| daemontool | I think fuel is on our same situation, monasca too | 11:26 |
| isaacb | yolanda: Hi, about https://review.openstack.org/#/c/262787/8 | 11:27 |
| clarkb | fuel doesnt used devstack gate as far as I know | 11:27 |
| clarkb | not sure about monasca | 11:27 |
| daemontool | that's why then they don't have this issue | 11:27 |
| isaacb | yolanda: I took your advise to fully customize the template | 11:27 |
| daemontool | but any service that try retroactively to have the service work with a specific milestone release | 11:27 |
| daemontool | incur on this I think | 11:27 |
| isaacb | yolanda: You gave me as example the gerrit template | 11:28 |
| daemontool | clarkb, do you think by using the same config of ceilometer for branch override would work? http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/ceilometer.yaml | 11:29 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config: Adds check-requirements for qa-specs repo https://review.openstack.org/270745 | 11:29 |
| clarkb | daemontool: they just have to use the correct branch name then it is automatic | 11:29 |
| daemontool | clarkb, ok | 11:29 |
| daemontool | the ones listed in ALLOWED_BRANCHES | 11:29 |
| daemontool | or something like that if I remember well | 11:29 |
| daemontool | ok | 11:29 |
| yolanda | hi isaacb, the problem i see with that, is with the java-arg | 11:31 |
| yolanda | JAVA_ARGS | 11:31 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 11:31 |
| *** vgridnev has quit IRC | 11:31 | |
| clarkb | daemontool: yes since those are the branches that it knows how to deploy. the underlying issur is do you deploy stable/1.2.0 against master or stable/liberty or etc | 11:31 |
| *** vgridnev has joined #openstack-infra | 11:31 | |
| yolanda | if you limit just to configuring heap_size, you are limiting the possibilities of that param. Downstream users may need to add more arguments, or remove existing ones, so that's why i think the whole line needs to be parameterized | 11:31 |
| *** yamamoto_ has quit IRC | 11:31 | |
| *** dizquierdo has quit IRC | 11:32 | |
| daemontool | clarkb, ok got it | 11:32 |
| daemontool | thanks a lot | 11:32 |
| *** rfolco has joined #openstack-infra | 11:32 | |
| clarkb | daemontool: I think the ceilometer solution is to essentially say stable/1.1.0 is stable/foo | 11:32 |
| clarkb | but they do that mappi g late in the jobs | 11:32 |
| *** kushal has quit IRC | 11:33 | |
| clarkb | yolanda: isaacb the current code doesnt have that limitation fwiw :) | 11:33 |
| isaacb | yolanda: o.k. this make sense, but what about the other parameters | 11:33 |
| yolanda | clarkb, there was a thread some weeks ago, about external jenkins default vs jenkins template | 11:33 |
| daemontool | clarkb, that happen from http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/ceilometer.yaml#n21 to line 24? | 11:33 |
| clarkb | yes I responded | 11:33 |
| daemontool | clarkb, I'm gonig to create stable/liberty | 11:33 |
| *** dizquierdo has joined #openstack-infra | 11:34 | |
| yolanda | i miss your answer, but i've been out due to family problems in the latest times, sorry. What was your opinion? | 11:34 |
| daemontool | so we are unblocked, then if I need to do anything different I'll do it | 11:34 |
| clarkb | daemontool: line 214 | 11:34 |
| clarkb | yolanda: that the current code accepts a file path and is pretty flexible | 11:34 |
| *** kzaitsev_mb has quit IRC | 11:34 | |
| *** jcoufal has joined #openstack-infra | 11:34 | |
| clarkb | yolanda: I didnt understand why we needed to change it | 11:34 |
| *** markvoelker has joined #openstack-infra | 11:34 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 11:35 |
| *** ddieterly has joined #openstack-infra | 11:35 | |
| yolanda | clarkb, isaacb, my concern was to go only with template because it would break downstream consumers, but i was not opposite to templating. clarkb, do you prefer that this is abandoned and we go with external files only? | 11:36 |
| isaacb | clarckb: What happens if the file is changed upstream? | 11:36 |
| clarkb | isaacb: nothing your file is still used | 11:36 |
| isaacb | clarkb: right. But its not updated | 11:37 |
| clarkb | yolanda: correct the current code doesnt have these issues and allows arbitrary defaults files | 11:37 |
| clarkb | isaacb: why not? | 11:37 |
| yolanda | clarkb, although the concern i have, is that we don't have a clear way to go in all puppet-openstack modules. Some of them are using templates, some using external files, without a clear guideline about the prefered way to go | 11:37 |
| clarkb | yolanda: it depends on the file | 11:37 |
| clarkb | a defaukts file has no specific form | 11:38 |
| clarkb | you can put arbitrary shell in it | 11:38 |
| clarkb | that doesnt work well with a template file | 11:38 |
| clarkb | other files have a finite set of values are great for templates | 11:38 |
| *** markvoelker has quit IRC | 11:39 | |
| yolanda | clarkb , well that happens in some cases. In some others we have change a bit the behaviour of our templates, to allow to pass an array of settings or more open content | 11:39 |
| clarkb | isaacb: anyways you tell it use this file over here then any changes to the source end up in the output | 11:39 |
| clarkb | yolanda: but even then it is finite the gerrit config for example | 11:40 |
| clarkb | or elasticsearch config | 11:40 |
| *** ddieterly has quit IRC | 11:40 | |
| yolanda | clarkb i'm ok with following that criteria, i think we can document it , so we have a clear guide to review and what is the prefered way to go with puppet configs | 11:40 |
| clarkb | really I think this defaults file is an exception | 11:41 |
| clarkb | it is arbitrary sourcable shell code | 11:41 |
| clarkb | it isnt a true config file | 11:42 |
| yolanda | clarkb, do you know if we follow the same criteria in all modules we have? | 11:42 |
| robcresswell | Goood morning. We've had Horizon patches in the gate for around 33hrs. Should we be rechecking these, or just wait until things settle down? | 11:42 |
| clarkb | yolanda: I dont think we typically have this probl anywhere else | 11:42 |
| *** fhubik is now known as fhubik_brb | 11:43 | |
| clarkb | robcresswell: before rechecking understanding why something failed is aleays helpful | 11:43 |
| isaacb | yolanda, clarkb: almost every downstream should override the default value for heap size. why not enable to do it simple? | 11:43 |
| clarkb | robcresswell: if new pip is to blame then rechecking wont help until new pip is addressed for example | 11:43 |
| robcresswell | clarkb: These aren't failures. They're just stuck in the gate for over a day. | 11:43 |
| yolanda | heap size is not the only value that a downstream consumer will need to change | 11:43 |
| clarkb | robcresswell: then rechecking wont do anything. the queue is being processed | 11:44 |
| robcresswell | clarkb: Ah, understood. | 11:44 |
| yolanda | we typically had to change timeouts, or add/remove some extra parameter on java defaults | 11:44 |
| isaacb | yolanda: I agree | 11:44 |
| clarkb | isaacb: because we shouldnt sacrifice functionality just to set the heap size | 11:45 |
| isaacb | clarckb: see Ramy comments in: https://review.openstack.org/#/c/263021/ | 11:45 |
| clarkb | isaacb: all I see is a comment to add to another module and to fix a typo | 11:47 |
| *** kushal has joined #openstack-infra | 11:47 | |
| clarkb | anyways I stated my opinion on yhe ml | 11:47 |
| clarkb | I think we need to understand why the current method doesnt work before we change it | 11:48 |
| isaacb | clarkb: according to Ramy this is useful for those using single-node-ci as well. | 11:48 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 11:48 |
| clarkb | and any chnages should hopefully not remove functionality the currwnt method supports | 11:48 |
| clarkb | isaacb: yes but we already support the feature is my ooint | 11:48 |
| clarkb | single node ci can use an arbitrary defaults file like anyone else | 11:49 |
| *** kushal has quit IRC | 11:49 | |
| *** kushal has joined #openstack-infra | 11:49 | |
| *** fhubik_brb is now known as fhubik | 11:49 | |
| clarkb | so one way potentially to do this if you want to make specific vars easy to set is do something like append the values to the file | 11:50 |
| isaacb | clark: what about the original thought to have both the external file and the template. If an external file is passed we use it. otherwise we use the template | 11:51 |
| clarkb | then you can have complete control of complicated shell if you want or you can just set the heap size | 11:51 |
| isaacb | right. yolanda? | 11:52 |
| clarkb | isaacb: I still think the issur with that is going to be attemlting to make a template everyone is happy with | 11:52 |
| clarkb | but maybe all that template does is set key valye pairs and anythibg more needs to set the file | 11:53 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Always go to Story Detail Page after adding Story https://review.openstack.org/270337 | 11:53 |
| yolanda | isaacb, what i don't see is the need to just templatize a concrete value, if we do , i'd add the same flexibility as the external file | 11:53 |
| *** Ramanjaneya has quit IRC | 11:54 | |
| yolanda | but this is limiting the functionalty, as the sample we had with java_args , or as clark said, if an user wants to add extra lines to the default file | 11:54 |
| isaacb | yolanda: using a template is easier to keep track with upstream changes. | 11:55 |
| yolanda | i can see the advantges of going with a template, in terms of having a set of sane defaults values, and only change the value you need when you call them. But i'm also afraid about code duplication and template complexity | 11:55 |
| isaacb | yolanda: what do you mean? | 11:56 |
| *** dizquierdo has quit IRC | 11:56 | |
| isaacb | I can't see any code duplications? | 11:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: openstack-doc-tools,openstackdocstheme: Use linters https://review.openstack.org/270469 | 11:57 |
| yolanda | isaacb, if we go with file + template, it means duplicating jenkins.default file | 11:57 |
| yolanda | one in the default file that people can override, one in the template with sane defaults | 11:57 |
| yolanda | and if we go only with template, we may loose functionality, or we need to do the template complex enough to accept extra vars, or remove the settings not needed in the initial template | 11:58 |
| isaacb | yolanda: I don't mean to have a default file in puppet | 11:58 |
| isaacb | there will be only a template in puppet | 11:58 |
| isaacb | but there is also an option to pass a file | 11:59 |
| yolanda | isaacb yes, but the puppet-jenkins module relies on a jenkins.default file , that is present on the module | 12:00 |
| yolanda | that's initial file that puppet-jenkins take, unless you specify something different on the option | 12:00 |
| *** esikachev has quit IRC | 12:01 | |
| isaacb | the file is no longer present on the module. I turned it to a template. | 12:01 |
| isaacb | But anyone who wish to pass another file as default is welcome | 12:02 |
| *** thorst has joined #openstack-infra | 12:02 | |
| *** dizquierdo has joined #openstack-infra | 12:03 | |
| isaacb | In case someone pass a file we can assume that he does not want the template.... | 12:03 |
| *** Zhongjun_ has joined #openstack-infra | 12:03 | |
| clarkb | right, I think as long as the default template has a sane scope then that is fine | 12:03 |
| *** esikachev has joined #openstack-infra | 12:03 | |
| *** boris-42 has quit IRC | 12:03 | |
| clarkb | isaacb: or we assume they are passing a their own template | 12:04 |
| isaacb | clarkb: this is another option... | 12:04 |
| clarkb | since the templafe engien can take inpufs with no tempalting and output them 1:1 | 12:05 |
| *** sdake has joined #openstack-infra | 12:05 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 12:07 |
| *** claudiub has joined #openstack-infra | 12:07 | |
| *** coolsvap is now known as coolsvap|away | 12:07 | |
| isaacb | yolanda: are you still concerned about having the file parameter option? | 12:08 |
| *** thorst has quit IRC | 12:10 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 12:11 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Add translation jobs for python-neutronclient https://review.openstack.org/270693 | 12:12 |
| *** gildub has quit IRC | 12:12 | |
| yolanda | isaacb i am fine with that, i'd rework the java_args bit as i told you in the review, and maybe not change every setting in the template, i don't think user or path needs to be update in the normal deployments | 12:13 |
| *** dtantsur|brb is now known as dtantsur | 12:13 | |
| *** fhubik is now known as fhubik_brb | 12:14 | |
| *** fhubik_brb is now known as fhubik | 12:14 | |
| yolanda | an external user can pass a file, and that file can be generated also using a template in the module that consumes puppet-jenkins, or the way they need | 12:14 |
| *** EricGonczer_ has joined #openstack-infra | 12:14 | |
| dstufft | okay, so let's see if we can get 8.0.1 finished up | 12:14 |
| *** achanda has joined #openstack-infra | 12:14 | |
| isaacb | yolanda: right. ok. I'll do the changes as we talked and add a new patch set. | 12:16 |
| yolanda | i am ok with that | 12:16 |
| *** vgridnev has quit IRC | 12:16 | |
| AJaeger | dstufft: we're standing by and cheer you up ;) | 12:18 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 12:18 |
| AJaeger | yolanda: I've udpated https://review.openstack.org/#/c/270469/ - thanks | 12:18 |
| *** vgridnev has joined #openstack-infra | 12:18 | |
| *** EricGonc_ has joined #openstack-infra | 12:19 | |
| *** achanda has quit IRC | 12:20 | |
| *** mgould has joined #openstack-infra | 12:20 | |
| mgould | morning! | 12:21 |
| mgould | can anyone take a look at https://review.openstack.org/#/c/266320/, please? One-line change to global-requirements that other patches depend on. | 12:22 |
| *** EricGonczer_ has quit IRC | 12:23 | |
| *** erlon has joined #openstack-infra | 12:24 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:25 | |
| *** dizquierdo_ has joined #openstack-infra | 12:25 | |
| *** doug-fish has joined #openstack-infra | 12:25 | |
| *** oomichi is now known as oomichi_away | 12:26 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Always go to Story Detail Page after adding Story https://review.openstack.org/270337 | 12:26 |
| *** dizquierdo has quit IRC | 12:29 | |
| gibi | clarkb: thanks for the answer. There was a time window when all the 6 patch wass progressed in the queue (for hours) but suddnely the gate queue dropped the last 4 commit | 12:29 |
| *** markvoelker has joined #openstack-infra | 12:29 | |
| gibi | clarkb: based on my recent experience now I have to wait untilt the first 2 commits are merged and then recheck the last 4 | 12:30 |
| *** otter768 has joined #openstack-infra | 12:30 | |
| *** oomichi_away has quit IRC | 12:31 | |
| *** EricGonc_ has quit IRC | 12:31 | |
| *** dizquierdo_ has quit IRC | 12:32 | |
| openstackgerrit | Victor Ryzhenkin proposed openstack-infra/project-config: Add non-voting murano-api gate job with tempest-plugin https://review.openstack.org/270770 | 12:33 |
| *** nelsnelson has quit IRC | 12:34 | |
| *** markvoelker has quit IRC | 12:35 | |
| *** abregman is now known as abregman|mtg | 12:35 | |
| *** otter768 has quit IRC | 12:35 | |
| *** jordanP has quit IRC | 12:36 | |
| *** ddieterly has joined #openstack-infra | 12:36 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 12:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-bandersnatch: Make bandersnatch::httpd class https://review.openstack.org/270486 | 12:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-bandersnatch: Make bandersnatch::cron class https://review.openstack.org/270498 | 12:38 |
| *** ujuc has joined #openstack-infra | 12:40 | |
| scheuran | hi, is there a known problem in the check queue, as jobs take 3 hours and longer to pass? | 12:40 |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch: Add hash_index support https://review.openstack.org/270480 | 12:41 |
| *** ddieterly has quit IRC | 12:41 | |
| AJaeger | scheuran: very high load - couple of problems came together, including the pip 8 release (8.01 should be out soon) | 12:41 |
| AJaeger | please have patience, scheuran | 12:41 |
| scheuran | AJaeger, thanks for the update! | 12:42 |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch: Add support to install from alternate locations https://review.openstack.org/270481 | 12:43 |
| *** thorst has joined #openstack-infra | 12:43 | |
| *** abregman|mtg is now known as abregman | 12:45 | |
| mordred | yolanda: I fixed your comment on 270486 - 498 is still a little busted though | 12:46 |
| *** zeih has quit IRC | 12:47 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Change Hover Cursor for Interactable Titles https://review.openstack.org/270776 | 12:47 |
| *** dizquierdo_ has joined #openstack-infra | 12:47 | |
| dims | sdague : AJaeger : where can i see a list of projects that will end up in the "integrated" gate queue? i know somewhere in project-config. but i tend to get lost in there :) | 12:48 |
| *** erikwilson has joined #openstack-infra | 12:48 | |
| *** trown|outttypeww is now known as trown | 12:48 | |
| AJaeger | dims: perhaps in the debug output of project-config layout job? Not sure ;( | 12:48 |
| openstackgerrit | Merged openstack-infra/puppet-openafs: Add kstart to client https://review.openstack.org/270516 | 12:48 |
| dims | AJaeger : ack will look there. thanks | 12:49 |
| * AJaeger checks as well | 12:49 | |
| AJaeger | dims, http://logs.openstack.org/93/270693/2/check/gate-project-config-layout/ffbe201/console.html#_2016-01-21_12_20_22_103 | 12:51 |
| sdague | dims: it's dynamic | 12:52 |
| sdague | we should probably actually have a job that does the job matrix calculation and ensures the list is a known value | 12:53 |
| *** erikwilson has quit IRC | 12:53 | |
| sdague | someone seems to have messed up the swiftclient job definition, so it's in there now | 12:53 |
| *** zeih has joined #openstack-infra | 12:53 | |
| mordred | Zara: morning! two quick storyboard things you may or may not want to know about ... | 12:53 |
| AJaeger | sdague: see my link above, that'S the zuul debug output stating the content of it | 12:53 |
| AJaeger | sdague: ah, you checked ;) | 12:54 |
| sdague | AJaeger: no, I just noticed it in the gate | 12:54 |
| AJaeger | sdague: gate-swift-dsvm-functional ? | 12:54 |
| sdague | yeh | 12:54 |
| mordred | Zara: a) I just went to projects and searched for "openstack-infra/shade" (I type faster than the type-ahead-find) and hit enter and I got a much longer list than shade. however, searching for shade worked as expected | 12:54 |
| mordred | Zara: b) is there any way to only see a list of stories from projects I have starred? | 12:55 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 12:55 |
| dims | sdague : any chance we can strip more things out from there? other than swift client? | 12:55 |
| yolanda | hi mordred, great | 12:55 |
| sdague | dims: not really | 12:55 |
| sdague | the number of things in there isn't actually a problem | 12:55 |
| mordred | Zara: oh, finally, I just did another search, and the search succeeded, but a popup showed up in the upper right hand corner saying: "400: GET /api/v1/stories/search: Unknown argument: "status"" | 12:56 |
| AJaeger | sdague: seems to be an older change... | 12:56 |
| sdague | dims: horizon might be able to come out | 12:56 |
| yolanda | mordred, i agree, 270498 doesn't honor the commit message | 12:56 |
| mordred | Zara: (I mention the two things that seem like errors not because they are problems for me, but just in case they are useful information) | 12:56 |
| sdague | dims: http://tinyurl.com/zrq35e8 is the problem | 12:56 |
| dims | sdague: how about the cluster of devstack/devstack-gate/grenade/requirements? | 12:56 |
| mordred | yolanda: yah - I'm pretty sure the commit message is the intent and th epatch is wrong :) | 12:56 |
| dims | sdague : looking | 12:57 |
| AJaeger | python-ganttclient? | 12:57 |
| sdague | as far as I can tell the grenade multinode jobs is always failing on the liberty side | 12:57 |
| sdague | honestly, I'd just push to make that non voting for now | 12:57 |
| sdague | because I have no idea what's wrong with it | 12:57 |
| AJaeger | sdague: why is that running integrated-gate jobs? | 12:57 |
| sdague | AJaeger: ? | 12:57 |
| AJaeger | sdague: python-ganttclient uses integrated-gate template - is that correct? | 12:58 |
| sdague | AJaeger: oh, that, it's a dead project | 12:58 |
| dims | sdague : +1 to make it non-voting | 12:58 |
| sdague | dims: push it, as well as the email about it | 12:58 |
| dims | sdague : ack will do shortly | 12:58 |
| AJaeger | sdague: ah, I remember. So, a nop ;) | 12:59 |
| AJaeger | dims, sdague and me can +2 ;) | 12:59 |
| dims | AJaeger : sneaky! :) | 12:59 |
| *** baoli has joined #openstack-infra | 13:02 | |
| *** fhubik is now known as fhubik_brb | 13:04 | |
| *** ihrachys has quit IRC | 13:04 | |
| *** ihrachys has joined #openstack-infra | 13:05 | |
| Zara | mordred: thanks! search has a lot of problems atm; both of those sound familiar. we've been looking at other things, but I want to focus on search after we get email notifications up. there's a patch in review that would help with b) https://review.openstack.org/#/c/257509/2/src/app/profile/template/subscriptions.html | 13:06 |
| *** bcourt has joined #openstack-infra | 13:06 | |
| Zara | mordred: it doesn't filter search results by subscription, but it's a page to view all subscriptions, separated by resource-type (ie: projects, stories) | 13:07 |
| *** jamielennox is now known as jamielennox|away | 13:07 | |
| *** ddieterly has joined #openstack-infra | 13:07 | |
| Zara | mordred: so not quite the samething but I think it would help meet the same goal (ie: quickly get to stuff you're interested in). | 13:07 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Set gate-grenade-dsvm-multinode to non-voting https://review.openstack.org/270788 | 13:09 |
| *** jtomasek has quit IRC | 13:09 | |
| *** gordc has joined #openstack-infra | 13:10 | |
| *** rhallisey has joined #openstack-infra | 13:10 | |
| AJaeger | dims: if you make a job non-voting, then we should remove it from the gate queues everywhere as well. If this is a short temporary solution, then state it and add some TODO, please | 13:11 |
| *** Guest7791 is now known as bradjones | 13:12 | |
| *** bradjones has joined #openstack-infra | 13:12 | |
| *** amotoki has joined #openstack-infra | 13:12 | |
| *** yolanda has quit IRC | 13:13 | |
| mordred | Zara: yah - that's definitely the goal ... | 13:14 |
| * mordred goes to look at the patch | 13:14 | |
| krotscheck | mordred: Are you interested in building wheel mirrors this morning? | 13:16 |
| *** dprince has joined #openstack-infra | 13:17 | |
| *** weshay_xchat has joined #openstack-infra | 13:18 | |
| *** fhubik_brb is now known as fhubik | 13:18 | |
| *** weshay_xchat is now known as weshay | 13:18 | |
| *** zeih has quit IRC | 13:19 | |
| dims | AJaeger : sdague : email sent and review updated with TODO | 13:19 |
| *** daemontool has quit IRC | 13:19 | |
| krotscheck | Else I'll wait until jeblair and/or fungi wake up | 13:20 |
| mordred | krotscheck: I am interested in the building of wheel mirrors, but I have to make some slides for a meeting :( | 13:20 |
| mordred | krotscheck: building wheel mirrors would be more funner | 13:20 |
| krotscheck | mordred: Build slides capable of launching a wheel mirror | 13:21 |
| *** baoli has quit IRC | 13:21 | |
| *** jtomasek_ has joined #openstack-infra | 13:22 | |
| *** baoli has joined #openstack-infra | 13:22 | |
| *** jed56 has quit IRC | 13:23 | |
| mordred | krotscheck: YES | 13:24 |
| *** markvoelker has joined #openstack-infra | 13:24 | |
| *** markvoelker has quit IRC | 13:29 | |
| sdague | AJaeger: I think we know this is a temp situation | 13:31 |
| sdague | this is a linchpin job | 13:31 |
| *** kgiusti has joined #openstack-infra | 13:32 | |
| *** Guest96416 has quit IRC | 13:32 | |
| sdague | mordred: pbr question | 13:34 |
| *** vgridnev has quit IRC | 13:34 | |
| sdague | you in #nova? | 13:34 |
| AJaeger | dims: did you push the update? sdague I'm fine with a temp situation... | 13:35 |
| sdague | AJaeger: I didn't see an update, I'd say lets just approve what's there | 13:36 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Set gate-grenade-dsvm-multinode to non-voting https://review.openstack.org/270788 | 13:36 |
| dims | oops | 13:36 |
| dims | here it is | 13:36 |
| *** vgridnev has joined #openstack-infra | 13:37 | |
| AJaeger | sdague: +2 ;) | 13:37 |
| *** kzaitsev_mb has quit IRC | 13:38 | |
| *** EricGonczer_ has joined #openstack-infra | 13:41 | |
| *** kzaitsev1ws is now known as kzaitsev_ws | 13:41 | |
| *** jpr1 has quit IRC | 13:41 | |
| *** tlian has joined #openstack-infra | 13:41 | |
| Ng | AJaeger: the gerritbot failure in https://review.openstack.org/#/c/266549/ is fixed on freenode - would you be able to add it back to your review queue for another look? :) | 13:42 |
| *** julim has joined #openstack-infra | 13:42 | |
| *** dguitarbite has quit IRC | 13:45 | |
| *** amrith is now known as _amrith_ | 13:46 | |
| *** bogdando has left #openstack-infra | 13:46 | |
| *** dguitarbite has joined #openstack-infra | 13:47 | |
| *** yamamoto has joined #openstack-infra | 13:48 | |
| ihrachys | sdague: any chance you have cycles to review some multinode + neutron patches that should allow neutron upgrades team to proceed on making partial grenade job working for neutron? https://review.openstack.org/#/q/status:open+topic:multinode-neutron-mtu (the first four patches in the list) | 13:49 |
| *** yamamoto has quit IRC | 13:49 | |
| *** Guest65425 is now known as zeus | 13:50 | |
| *** zeus has joined #openstack-infra | 13:50 | |
| sdague | is there an order? it's really hard to figure out with all the depends-on | 13:50 |
| *** markvoelker has joined #openstack-infra | 13:50 | |
| *** bcourt has quit IRC | 13:51 | |
| *** bhunter71 has quit IRC | 13:51 | |
| sdague | ihrachys: https://review.openstack.org/#/c/267604/4 that was a typo, right? | 13:52 |
| *** yamamoto has joined #openstack-infra | 13:52 | |
| *** markvoelker has quit IRC | 13:53 | |
| *** hashar has quit IRC | 13:53 | |
| *** markvoelker has joined #openstack-infra | 13:53 | |
| *** EricGonczer_ has quit IRC | 13:53 | |
| *** exploreshaifali has quit IRC | 13:54 | |
| *** notnownikki has joined #openstack-infra | 13:54 | |
| *** dslevin has quit IRC | 13:54 | |
| *** bcourt has joined #openstack-infra | 13:55 | |
| openstackgerrit | Merged openstack-infra/system-config: Stop listing hosts as groups in the disabled file https://review.openstack.org/270262 | 13:55 |
| *** zhurong has joined #openstack-infra | 13:55 | |
| *** zhurong has quit IRC | 13:56 | |
| *** Liuqing has quit IRC | 13:56 | |
| *** Jeffrey4l has joined #openstack-infra | 13:57 | |
| *** markvoelker has quit IRC | 13:57 | |
| esikachev | hi all! i have a problem with my local ci, need youre help with zuul. who can help me? | 13:58 |
| *** markvoelker has joined #openstack-infra | 13:59 | |
| *** ansiwen is now known as ansiwen|afk | 13:59 | |
| *** EricGonczer_ has joined #openstack-infra | 13:59 | |
| *** fhubik is now known as fhubik_brb | 13:59 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Fix some minor project group issues https://review.openstack.org/269627 | 14:00 |
| *** sdake has quit IRC | 14:01 | |
| *** weshay has quit IRC | 14:01 | |
| *** Jeffrey4l has quit IRC | 14:02 | |
| *** erikwilson has joined #openstack-infra | 14:02 | |
| *** shardy has quit IRC | 14:03 | |
| *** burgerk has joined #openstack-infra | 14:03 | |
| *** fhubik_brb is now known as fhubik | 14:03 | |
| *** ddieterly has quit IRC | 14:03 | |
| openstackgerrit | Merged openstack-infra/system-config: Use groups.txt for disabling hosts for puppet https://review.openstack.org/270296 | 14:03 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add hue-light plugin support https://review.openstack.org/268814 | 14:04 |
| *** annegentle has joined #openstack-infra | 14:04 | |
| *** shardy has joined #openstack-infra | 14:05 | |
| *** daemontool has joined #openstack-infra | 14:05 | |
| *** EricGonczer_ has quit IRC | 14:07 | |
| *** hdd has joined #openstack-infra | 14:07 | |
| *** pc_m has quit IRC | 14:07 | |
| *** lezbar has quit IRC | 14:07 | |
| *** lezbar__ has joined #openstack-infra | 14:07 | |
| *** pc_m has joined #openstack-infra | 14:07 | |
| *** edmondsw has joined #openstack-infra | 14:08 | |
| ihrachys | sdague: no, that is a special syntax to allow unsetting the value with an empty value. then you don't get path_mtu set. | 14:09 |
| ihrachys | sdague: they are pretty independent pieces (those four patches). other patches are still not ready for review or we doubt their need (need more testing). | 14:10 |
| ihrachys | some pieces already merged in neutron, so some Depends-On should be already fulfilled | 14:10 |
| sdague | ihrachys: ok, that needs a comment | 14:10 |
| sdague | because that's going to be refactored away later | 14:11 |
| openstackgerrit | Merged openstack-infra/project-config: Set gate-grenade-dsvm-multinode to non-voting https://review.openstack.org/270788 | 14:11 |
| ihrachys | sdague: that == syntax? | 14:12 |
| sdague | yeh | 14:12 |
| ihrachys | ok on my way :) | 14:12 |
| sdague | anything that's not :- needs a comment | 14:12 |
| *** hdd has quit IRC | 14:13 | |
| sdague | infra-root: can we get a promote on https://review.openstack.org/#/c/270474/ which is part of this keystonemiddleware cache unwind? | 14:14 |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds fairy-slipper build jobs for api-site repo https://review.openstack.org/269809 | 14:14 |
| *** flip214 has quit IRC | 14:14 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 14:15 |
| ihrachys | sdague: uploaded with a comment | 14:15 |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds fairy-slipper build jobs for api-site repo https://review.openstack.org/269809 | 14:16 |
| *** Thelo_ has joined #openstack-infra | 14:16 | |
| *** dizquierdo_ has quit IRC | 14:17 | |
| *** sabeen1 has joined #openstack-infra | 14:17 | |
| *** flip214 has joined #openstack-infra | 14:17 | |
| *** flip214 has joined #openstack-infra | 14:17 | |
| *** achanda has joined #openstack-infra | 14:17 | |
| *** weshay_xchat has joined #openstack-infra | 14:18 | |
| *** sdake has joined #openstack-infra | 14:19 | |
| *** keedya has joined #openstack-infra | 14:19 | |
| *** jlanoux has quit IRC | 14:19 | |
| *** dkranz has joined #openstack-infra | 14:20 | |
| *** fhubik is now known as fhubik_brb | 14:20 | |
| *** sabeen3 has joined #openstack-infra | 14:21 | |
| *** tiswanso has joined #openstack-infra | 14:22 | |
| *** achanda has quit IRC | 14:23 | |
| *** sabeen1 has quit IRC | 14:24 | |
| *** peristeri has joined #openstack-infra | 14:24 | |
| *** ccrouch has joined #openstack-infra | 14:24 | |
| *** tiswanso has quit IRC | 14:26 | |
| *** rlandy has joined #openstack-infra | 14:27 | |
| *** mriedem has joined #openstack-infra | 14:28 | |
| *** yamamoto has quit IRC | 14:28 | |
| *** ihrachys has quit IRC | 14:28 | |
| *** ddieterly has joined #openstack-infra | 14:30 | |
| *** zeih has joined #openstack-infra | 14:31 | |
| *** jtomasek has joined #openstack-infra | 14:31 | |
| AJaeger | sdague: could you review https://review.openstack.org/270469 , please? | 14:31 |
| *** otter768 has joined #openstack-infra | 14:31 | |
| *** yolanda has joined #openstack-infra | 14:32 | |
| *** tiswanso has joined #openstack-infra | 14:32 | |
| *** dizquierdo has joined #openstack-infra | 14:32 | |
| openstackgerrit | Matthew Kassawara proposed openstack-infra/project-config: Add logic for networking-ovn jobs https://review.openstack.org/270444 | 14:33 |
| *** ujuc has quit IRC | 14:33 | |
| *** ddieterly has quit IRC | 14:34 | |
| *** sdake has quit IRC | 14:35 | |
| *** otter768 has quit IRC | 14:36 | |
| *** Ramanjaneya has joined #openstack-infra | 14:37 | |
| *** ihrachys has joined #openstack-infra | 14:39 | |
| *** dslev has joined #openstack-infra | 14:39 | |
| *** fhubik_brb is now known as fhubik | 14:40 | |
| *** aopincar has quit IRC | 14:41 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Change Hover Cursor for Interactable Titles https://review.openstack.org/270776 | 14:41 |
| *** regXboi has joined #openstack-infra | 14:41 | |
| openstackgerrit | Dina Belova proposed openstack-infra/project-config: Add py27 job for performance-docs https://review.openstack.org/270088 | 14:41 |
| *** fawadkhaliq has joined #openstack-infra | 14:42 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Add matrixOptions to throttle properties module https://review.openstack.org/241699 | 14:45 |
| *** ari has joined #openstack-infra | 14:46 | |
| fungi | sdague: 270474 breaking multi-node jobs is acceptable collateral damage for now? | 14:47 |
| *** jordanP has joined #openstack-infra | 14:48 | |
| *** jtomasek_ has quit IRC | 14:49 | |
| *** _amrith_ is now known as amrith | 14:50 | |
| fungi | i've gone ahead and promoted it | 14:51 |
| *** binbincong has quit IRC | 14:51 | |
| *** binbincong has joined #openstack-infra | 14:52 | |
| *** treaki_ has quit IRC | 14:52 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Remove *-validate-content job from Kolla https://review.openstack.org/270839 | 14:52 |
| *** ildikov has quit IRC | 14:52 | |
| sdague | fungi: yeh, multinode is just broken | 14:52 |
| fungi | i missed that development. there was already a new bug breaking the multinode tempest jobs? | 14:53 |
| fungi | do you happen to know who was looking into it, if anyone? | 14:53 |
| sdague | no, multinode has been broken for a while again | 14:53 |
| *** nelsnelson has joined #openstack-infra | 14:53 | |
| *** kashyap has left #openstack-infra | 14:53 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: gitlab trigger: Support new "addCiMessage" flag https://review.openstack.org/266087 | 14:54 |
| *** ccrouch has quit IRC | 14:54 | |
| *** mtanino has joined #openstack-infra | 14:54 | |
| *** nelsnels_ has joined #openstack-infra | 14:55 | |
| *** nelsnelson has quit IRC | 14:55 | |
| *** julim has quit IRC | 14:56 | |
| dims | fungi : sdague : integrated gate queue seems to be starving... not much running in that queue - http://status.openstack.org/zuul/ | 14:57 |
| sdague | dims: there was a reset | 14:57 |
| sdague | it's rebuilding | 14:57 |
| dims | sdague : aha. thanks! | 14:58 |
| sdague | well, there was a promote | 14:58 |
| *** ildikov has joined #openstack-infra | 14:58 | |
| sdague | so it's restacking everything | 14:58 |
| fungi | yeah, i promoted the memcached patch to the top, so everything had its jobs restarted | 14:58 |
| dims | whew. cool thanks | 14:58 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add some swap to the undercloud https://review.openstack.org/270274 | 14:59 |
| *** jlanoux has joined #openstack-infra | 14:59 | |
| *** ashleighfarnham has joined #openstack-infra | 14:59 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on Refresh token Response: https://review.openstack.org/270846 | 15:00 |
| *** Thelo_ has quit IRC | 15:00 | |
| *** vgridnev has quit IRC | 15:01 | |
| *** julim has joined #openstack-infra | 15:02 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:03 | |
| *** fawadkhaliq has quit IRC | 15:03 | |
| *** yamamoto has joined #openstack-infra | 15:03 | |
| *** piet has joined #openstack-infra | 15:04 | |
| *** burgerk has quit IRC | 15:04 | |
| *** ashleighfarnham has quit IRC | 15:04 | |
| *** boris-42 has joined #openstack-infra | 15:07 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config: End Gating for Tuskar Projects https://review.openstack.org/270850 | 15:07 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Fix some minor project group issues https://review.openstack.org/269627 | 15:07 |
| *** piet has quit IRC | 15:09 | |
| *** sdake has joined #openstack-infra | 15:12 | |
| fungi | zaro: thanks for your suggestions on 263971 (and your upstream bug fixes in gerrit)! | 15:15 |
| *** sdake_ has joined #openstack-infra | 15:17 | |
| *** ccrouch has joined #openstack-infra | 15:19 | |
| *** bhunter71 has joined #openstack-infra | 15:20 | |
| *** achanda has joined #openstack-infra | 15:20 | |
| *** rbrndt has joined #openstack-infra | 15:20 | |
| *** achanda has quit IRC | 15:20 | |
| *** achanda has joined #openstack-infra | 15:20 | |
| *** jaypipes has joined #openstack-infra | 15:21 | |
| mriedem | dims: https://review.openstack.org/#/c/270857/ | 15:21 |
| mriedem | clarkb: was looking at it yesterday | 15:21 |
| *** sdake has quit IRC | 15:22 | |
| mriedem | it's a problem with the multinode job where we're using fixed ips and trying to ssh from one private net to another | 15:22 |
| mriedem | *i think* | 15:22 |
| dims | mriedem : +1 worth a try | 15:22 |
| *** burgerk has joined #openstack-infra | 15:22 | |
| *** tonytan4ever has joined #openstack-infra | 15:22 | |
| *** cschwede has quit IRC | 15:23 | |
| *** FallenPegasus has joined #openstack-infra | 15:24 | |
| pabelanger | fungi: Asked yesterday, but got missed in the backscroll. Is there anything special you'd like to see for the upstream summit talk around submitting presentations? EG: do something as a group, or just do individual contributions? | 15:25 |
| *** fawadkhaliq has joined #openstack-infra | 15:25 | |
| *** zz_dimtruck is now known as dimtruck | 15:26 | |
| *** ddieterly has joined #openstack-infra | 15:26 | |
| fungi | pabelanger: there was a list of ideas in an etherpad... did i link that in the meeting? | 15:27 |
| *** Ramanjaneya has quit IRC | 15:27 | |
| fungi | it was in ttx's e-mail to the -dev ml | 15:27 |
| *** Daisy has joined #openstack-infra | 15:27 | |
| *** migi has quit IRC | 15:29 | |
| *** jsavak has joined #openstack-infra | 15:30 | |
| *** ansiwen|afk has quit IRC | 15:30 | |
| pabelanger | fungi: Right, I have it. Mostly curious if the infra team wanted to submit something formal (eg: help prepare the talk) or just do it individually. Is there any preference here on what we want to talk about? | 15:30 |
| *** fhubik is now known as fhubik_brb | 15:30 | |
| *** fhubik_brb is now known as fhubik | 15:31 | |
| *** fhubik has quit IRC | 15:31 | |
| openstackgerrit | Merged openstack-infra/openstackid: Fix on Refresh token Response: https://review.openstack.org/270846 | 15:31 |
| *** EricGonczer_ has joined #openstack-infra | 15:31 | |
| *** burgerk has quit IRC | 15:32 | |
| EmilienM | hello infra! I have a patch standing for some time in os-loganalyze, could someone have a look? https://review.openstack.org/#/c/265447/ Thanks! | 15:32 |
| fungi | pabelanger: if anyone else volunteers, it would be good to coordinate submissions so we can avoid proposing overlapping topics. i'm fine with individual presenters, groups, panels, whatever | 15:33 |
| fungi | it's more a question of picking a format that best suits the topic | 15:33 |
| *** achanda has quit IRC | 15:33 | |
| fungi | and what the track chairs will ultimately approve | 15:34 |
| openstackgerrit | Nicko Glayre proposed openstack-infra/jenkins-job-builder: Adding support for maven-metadata-plugin https://review.openstack.org/257430 | 15:34 |
| pabelanger | fungi: Sure, I was also considering sending something to the -infra ML to see if anybody else is interested in collaborating. If not, I have some ideas based on the etherpad link | 15:34 |
| *** dprince has quit IRC | 15:35 | |
| *** fhubik has joined #openstack-infra | 15:35 | |
| fungi | pabelanger: sounds great! | 15:35 |
| openstackgerrit | Julien Danjou proposed openstack/requirements: Make upper constraint actually be "upper" constraints https://review.openstack.org/270865 | 15:35 |
| fungi | more discussion in a different venue than irc will hopefully increase visibility and garner more volunteers | 15:36 |
| *** asselin__ has quit IRC | 15:37 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add matrixOptions to throttle properties module https://review.openstack.org/241699 | 15:37 |
| *** sdake_ is now known as sdake | 15:37 | |
| *** jyuso1 has joined #openstack-infra | 15:37 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Migrate scenario gate on sahara-scenario repo https://review.openstack.org/270868 | 15:37 |
| *** jsavak has quit IRC | 15:38 | |
| *** markvoelker has quit IRC | 15:38 | |
| *** tiswanso has quit IRC | 15:38 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Migrate scenario gate on sahara-scenario repo https://review.openstack.org/270868 | 15:39 |
| *** tiswanso has joined #openstack-infra | 15:39 | |
| *** asselin_ has joined #openstack-infra | 15:39 | |
| *** annegent_ has joined #openstack-infra | 15:39 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config: Retire the Tuskar project https://review.openstack.org/270869 | 15:39 |
| *** andymaier has quit IRC | 15:40 | |
| *** Ramanjaneya has joined #openstack-infra | 15:41 | |
| *** hdd has joined #openstack-infra | 15:41 | |
| *** annegentle has quit IRC | 15:42 | |
| *** fawadkhaliq has quit IRC | 15:42 | |
| *** jyuso1 has quit IRC | 15:42 | |
| *** esikachev has quit IRC | 15:43 | |
| *** amotoki has quit IRC | 15:44 | |
| *** Daisy has quit IRC | 15:45 | |
| *** fhubik has quit IRC | 15:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-bandersnatch: Make bandersnatch::cron class https://review.openstack.org/270498 | 15:47 |
| EmilienM | krotscheck: hey, do we have a follow-up on https://review.openstack.org/#/c/253616/ ? | 15:47 |
| krotscheck | EmilienM: Yes. | 15:48 |
| EmilienM | krotscheck: looks like https://review.openstack.org/#/c/238754 is stalled | 15:48 |
| krotscheck | EmilienM: Yeah, the reason for that is that the effort was split, once again, into different efforts. | 15:48 |
| pabelanger | fungi: great, already sent | 15:49 |
| krotscheck | EmilienM: Long story short- First land wheel mirror to make things faster, then land AFS mirrors, then add additional mirrors to AFS mirror. | 15:49 |
| *** hashar has joined #openstack-infra | 15:49 | |
| krotscheck | EmilienM: I'm still on it. | 15:49 |
| krotscheck | EmilienM: Current wheel effort is here https://review.openstack.org/#/q/branch:master+topic:wheel_mirror+status:open | 15:50 |
| EmilienM | krotscheck: nice summary and good to know you're still on it. Thanks! Please let me know if you need help on something | 15:50 |
| krotscheck | EmilienM: AFS mirror effort is here https://review.openstack.org/#/q/branch:master+topic:afs-mirror+status:open | 15:50 |
| krotscheck | EmilienM: Willdo :) | 15:50 |
| EmilienM | krotscheck: indeed, lot of work, wow | 15:50 |
| *** tjones has joined #openstack-infra | 15:51 | |
| sdague | mordred: you had a +2 on earlier version of this patch - https://review.openstack.org/#/c/267133/ - can you look again? | 15:52 |
| jeblair | fungi, sdague: so did we just break multinode more? there's the ip issue that clarkb et al were working on yesterday and now also the memcache thing? | 15:53 |
| *** nmagnezi has quit IRC | 15:54 | |
| *** achanda has joined #openstack-infra | 15:55 | |
| openstackgerrit | Nicko Glayre proposed openstack-infra/jenkins-job-builder: Adding support for maven-metadata-plugin https://review.openstack.org/257430 | 15:57 |
| *** zeih has quit IRC | 15:57 | |
| *** angdraug has joined #openstack-infra | 15:57 | |
| sdague | honestly, I don't know. There is currently so much broken it's hard to see what's going on | 15:57 |
| *** shardy has quit IRC | 15:57 | |
| *** jpr has joined #openstack-infra | 15:58 | |
| *** zeih has joined #openstack-infra | 15:58 | |
| sdague | ok, I'm going to pull that devstack change out of the gate | 15:58 |
| sdague | you might be right, about it failing on the multinode case in a new way | 15:59 |
| *** shardy has joined #openstack-infra | 15:59 | |
| openstackgerrit | Dina Belova proposed openstack-infra/project-config: Add py27 job for performance-docs https://review.openstack.org/270088 | 16:00 |
| *** diazjf has joined #openstack-infra | 16:00 | |
| *** markvoelker has joined #openstack-infra | 16:01 | |
| *** isaacb has quit IRC | 16:01 | |
| *** achanda has quit IRC | 16:02 | |
| *** FallenPegasus has quit IRC | 16:02 | |
| mordred | sdague: done | 16:03 |
| *** zeih has quit IRC | 16:03 | |
| jeblair | if you have "class {foo}" and "class {foo::bar}" is there an implied depedency in puppet? | 16:03 |
| cbader_ | I have a question about the argparse issue yesterday was there a solution I am running into this on creating a common-ci server. Issue with pip install on zuul and diskimage-builder? | 16:04 |
| mordred | jeblair: I do not believe so | 16:04 |
| *** sshnaidm has quit IRC | 16:04 | |
| mordred | jeblair: like, if you include foo::bar does that imply including fo? | 16:04 |
| mordred | foo? | 16:04 |
| *** scheuran has quit IRC | 16:04 | |
| *** placko has quit IRC | 16:05 | |
| jeblair | oh sorry | 16:05 |
| jeblair | mordred: i meant does foo::bar auto-require foo | 16:05 |
| jeblair | for ordering | 16:05 |
| jeblair | (puppet does perform _some_ auto-requiring) | 16:05 |
| mordred | jeblair: ah. that I do not know | 16:05 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Logging in no longer causes page refresh. https://review.openstack.org/270882 | 16:06 |
| mordred | crinkle: ^^ ? | 16:06 |
| mordred | actually - I bet EmilienM knows ... | 16:06 |
| EmilienM | o/ | 16:06 |
| EmilienM | no | 16:06 |
| mordred | EmilienM: see jeblair's questoin? | 16:06 |
| EmilienM | you need to explicitly include ::foo | 16:06 |
| EmilienM | and then ::foo::bar | 16:06 |
| mordred | EmilienM: but if you include ::foo and include ::foo::bar will it include ::foo _before_ it includes ::foo::bar? | 16:07 |
| EmilienM | including ::foo::bar does not include ::foo for you | 16:07 |
| EmilienM | I know the DSL is now aware about resource ordering in the manifest | 16:07 |
| jeblair | (will it automatically add a "require => Class['foo']" relationship) | 16:07 |
| jeblair | (to foo::bar) | 16:07 |
| EmilienM | so I highly suggest to first declare ::foo and then ::foobar | 16:07 |
| EmilienM | so I highly suggest to first declare ::foo and then ::foo::bar | 16:07 |
| *** pcrews has joined #openstack-infra | 16:07 | |
| EmilienM | jeblair: not afik | 16:08 |
| jeblair | ok, thanks | 16:08 |
| EmilienM | let me confirm | 16:08 |
| EmilienM | jeblair: include != require | 16:08 |
| EmilienM | https://docs.puppetlabs.com/puppet/latest/reference/lang_relationships.html#syntax-the-require-function | 16:09 |
| EmilienM | so I think there is no implicit require | 16:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add mirror_update.openstack.org https://review.openstack.org/270520 | 16:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Document adding a kerberos service principal https://review.openstack.org/270559 | 16:09 |
| cbader_ | clarkb, was there a solution to the remove argparse issue? | 16:09 |
| *** rcernin has quit IRC | 16:09 | |
| jeblair | mordred, krotscheck: updated ^ | 16:09 |
| AJaeger | cbader_: we wait for pip 8.0.1. argparse removal is a nice cleanup | 16:10 |
| krotscheck | jeblair: Thanks, will look at it as soon as I've fixed something for storyboard | 16:10 |
| *** Ramanjaneya has quit IRC | 16:10 | |
| *** yamahata has joined #openstack-infra | 16:10 | |
| cbader_ | Ajaeger, we are waiting for pip 8.0.1 is that correct? | 16:10 |
| *** ddieterly has quit IRC | 16:11 | |
| *** ddieterly has joined #openstack-infra | 16:11 | |
| *** yamamoto has quit IRC | 16:11 | |
| AJaeger | cbader_: yes. | 16:11 |
| *** yamamoto has joined #openstack-infra | 16:11 | |
| *** yamamoto has quit IRC | 16:11 | |
| AJaeger | cbader_: at least my understanding | 16:12 |
| crinkle | mordred: jeblair no foo::bar does not autorequire foo | 16:12 |
| cbader_ | Ajaeger, is there any idea when that might happen, is there any work around. | 16:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add storyboard exim router/transports https://review.openstack.org/270331 | 16:12 |
| cbader_ | Ajeager, my openstackci is down so not reporting anything at this time so lots of preasure. | 16:12 |
| *** yfried_ has quit IRC | 16:12 | |
| *** amotoki has joined #openstack-infra | 16:13 | |
| AJaeger | cbader_: dstufft was working on this earlier today, not sure what his status is | 16:14 |
| jeblair | krotscheck: i'll be back in a few minutes then i'll be ready to mirror | 16:14 |
| krotscheck | jeblair: Thanks | 16:14 |
| *** slagle has quit IRC | 16:14 | |
| *** cschwede has joined #openstack-infra | 16:14 | |
| cbader_ | Ajaeger, thank you for the update, I will watch for update and hope it comes. | 16:14 |
| *** erikwilson has quit IRC | 16:14 | |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add lib-forward-testing for OSprofiler https://review.openstack.org/269905 | 16:15 |
| cbader_ | Has anyone seen this issue on creation of common-ci. Execution of '/usr/local/bin/pip install -q --upgrade python-jenkins' returned 1: Detected a distutils installed project ('six') which we cannot uninstall. | 16:16 |
| fungi | sdague: jeblair: the main reason i asked is because of the numerous comments from other reviewers saying that memcache needed to become a dependency of most services, not just keystone, that it was a quirk of aio that it worked there at all, and so would break on devstack nodes where keystone didn't get set up (e.g. the second node in our multinode jobs) | 16:16 |
| *** fawadkhaliq has joined #openstack-infra | 16:17 | |
| fungi | and i didn't see any of those comments addressed before you approved | 16:17 |
| clarkb | cbader_: that is the pip problem | 16:17 |
| fungi | yeah, no pip 8.0.1 yet | 16:17 |
| openstackgerrit | Merged openstack-infra/project-config: always make upper-constraints available to python jobs https://review.openstack.org/267133 | 16:18 |
| Sam-I-Am | good morning... whats the status of https://review.openstack.org/#/c/270417/ | 16:18 |
| cbader_ | clarkb, ok thanks that is what I thought but wanted to just check so I don't spend anymore time trying to figure out. | 16:18 |
| *** bnemec has quit IRC | 16:18 | |
| cbader_ | So all will be made right by pip 8.0.1 :-b | 16:19 |
| fungi | not necessarily all | 16:19 |
| *** placko has joined #openstack-infra | 16:19 | |
| clarkb | that particular issue should be corrected, but not necessarily everything | 16:19 |
| clarkb | oh wait no that one may not be | 16:19 |
| clarkb | I read six as argparse for some reason | 16:19 |
| cbader_ | I just want the current ones fixed so I can get my openstackci backup and running. | 16:20 |
| *** bnemec has joined #openstack-infra | 16:20 | |
| clarkb | cbader_: you can use an older pip | 16:20 |
| fungi | last i heard, broken stdlib shadowing (argparse) and issues with system-site-packages in virtualenvs are targeted for fixing in 8.0.1 | 16:20 |
| *** keedya has quit IRC | 16:20 | |
| openstackgerrit | Xingchao Yu proposed openstack-infra/project-config: Create puppet-oslo module https://review.openstack.org/270895 | 16:20 |
| *** rajinir has joined #openstack-infra | 16:20 | |
| sdague | fungi: yeh, sorry, my bad | 16:21 |
| cbader_ | so my six issue on upgrade on python-jenkins might be something else any clue were? | 16:21 |
| Sam-I-Am | sdague: got a min? | 16:21 |
| *** vgridnev has joined #openstack-infra | 16:22 | |
| cbader_ | so the python-jenkins issue says, The metadata provided by distutils does not contain a list of files which have been installed, so pip does not know which files to uninstall. | 16:22 |
| *** Ramanjaneya has joined #openstack-infra | 16:22 | |
| clarkb | cbader_: yes that is the new pip | 16:22 |
| fungi | problems exposed by pip 8 with distro packages of things like six, netaddr, et cetera aren't entirely settled on how (or whether) they'd be fixed in pip | 16:22 |
| sdague | Sam-I-Am: sure | 16:22 |
| *** HeOS has quit IRC | 16:22 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config: End Gating for Tuskar Projects https://review.openstack.org/270850 | 16:22 |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config: Retire the Tuskar project https://review.openstack.org/270869 | 16:22 |
| cbader_ | clarkb, ok will wait with everyone else and tell bosses on hook for fix. hope they accept it. | 16:22 |
| *** asalkeld has joined #openstack-infra | 16:23 | |
| clarkb | cbader_: or you can use older pip | 16:23 |
| *** krtaylor has quit IRC | 16:23 | |
| sdague | fungi: can you re promote this - https://review.openstack.org/#/c/270417/ | 16:23 |
| cbader_ | clarkb, I tried that yesterday but seemed to cause other issues. I guess I can try again. | 16:23 |
| sdague | that bounced off the multinode fail | 16:23 |
| clarkb | cbader_: it needs to be the most recent previous version probably | 16:24 |
| krotscheck | jeblair: Ready when you are. | 16:24 |
| * krotscheck goes to reviews jeblair's patches. | 16:24 | |
| sdague | which is now non voting, so maybe we can fix it | 16:24 |
| rajinir | asselin: Appreciate if you can review this - https://review.openstack.org/#/c/267733/ | 16:24 |
| rajinir | clarkb: Need help with review to move this forward - https://review.openstack.org/#/c/267733/ | 16:25 |
| *** fawadkhaliq has quit IRC | 16:25 | |
| *** bardia has joined #openstack-infra | 16:26 | |
| *** hashar has quit IRC | 16:27 | |
| cbader_ | clarkb, I think it is 7.1.2 from yesterday. need to lookup how to do forgot. | 16:28 |
| *** jvrbanac_ has joined #openstack-infra | 16:29 | |
| *** dizquierdo has quit IRC | 16:29 | |
| sdague | fungi: thanks | 16:30 |
| *** piet has joined #openstack-infra | 16:30 | |
| fungi | sdague: yep, done ;) | 16:30 |
| *** jvrbanac_ has quit IRC | 16:30 | |
| *** matrohon has joined #openstack-infra | 16:31 | |
| pabelanger | greghaynes: ianw: any ideas on the diskimage-builder gate failures? | 16:32 |
| *** jvrbanac has joined #openstack-infra | 16:32 | |
| *** erikwilson has joined #openstack-infra | 16:32 | |
| anteaya | cbader_: remember it is you who are responsible for your third party ci operation not the infra team nor anyone else in openstack or the python community | 16:32 |
| *** otter768 has joined #openstack-infra | 16:32 | |
| dstufft | fungi: they'll be fixed | 16:32 |
| dstufft | for now | 16:32 |
| anteaya | cbader_: we do our best to help you but the responsibility is yours | 16:32 |
| *** [1]Thelo has joined #openstack-infra | 16:33 | |
| fungi | dstufft: thanks for the update! | 16:34 |
| cbader_ | anteaya, I understand it is all my responsibility, but is it nice to see the community help resolve issue or recommend how to fix issues. | 16:35 |
| *** Thelo has quit IRC | 16:35 | |
| *** [1]Thelo is now known as Thelo | 16:35 | |
| *** isaacb has joined #openstack-infra | 16:35 | |
| anteaya | cbader_: you are part of the community | 16:35 |
| *** krtaylor has joined #openstack-infra | 16:35 | |
| *** mpavone has quit IRC | 16:36 | |
| cbader_ | anteaya, I am trying to be, new to this type of work, thrown in a very large pool. | 16:36 |
| *** otter768 has quit IRC | 16:37 | |
| cbader_ | anteaya, I am commited to this endevour and hopefully contributing in some small some day but there yet. | 16:37 |
| anteaya | expressing appreciation goes a long way | 16:37 |
| jeblair | some people think this is the deep end of the pool, but really, we've just gone off the deep end | 16:38 |
| dstufft | fungi: no problem :) | 16:38 |
| dstufft | 8.0.1 will probably be today | 16:38 |
| *** erikwilson has quit IRC | 16:38 | |
| dstufft | assuming travis stops taking forever and/or I don't fall asleep again like last night | 16:39 |
| jeblair | krotscheck: okay, what needs doing? | 16:39 |
| cbader_ | the person I am trying to replace has made hugh commits to this group and I am trying to just survive. | 16:39 |
| Swanson | dumb question. When is the 3rd party ci meeting? | 16:39 |
| anteaya | well he is sitting right next to you | 16:39 |
| krotscheck | jeblair: At this point, I believe we need to create the wheel mirror nodes. | 16:39 |
| Swanson | google is my friend, I suppose | 16:39 |
| anteaya | Swanson: not a dumb question | 16:39 |
| jeblair | krotscheck: all relevant patches landed? | 16:39 |
| krotscheck | jeblair: If I'm not mistaken, fungi has already added all the hiera keys for ssh things. | 16:39 |
| krotscheck | jeblair: Yeah, sec, lemme find it for you. | 16:40 |
| fungi | krotscheck: jeblair: well, yes and no | 16:40 |
| anteaya | Swanson: here is the meeting page of all of the openstack meetings: http://eavesdrop.openstack.org/ | 16:40 |
| cbader_ | but he has his own job and is not available to help but only very little at all. | 16:40 |
| krotscheck | jeblair: https://review.openstack.org/#/c/165240/26/manifests/site.pp | 16:40 |
| anteaya | Swanson: here is the third party meetings I chair: http://eavesdrop.openstack.org/#Third_Party_Meeting | 16:40 |
| fungi | krotscheck: jeblair: we got stuck with getting mordred's groups.txt refactor merged yesterday which my addition to the groups list for the mirror workers needs to get added to | 16:40 |
| fungi | checking now on where that ended up | 16:40 |
| cbader_ | his comment is use the community first | 16:40 |
| *** erikwilson has joined #openstack-infra | 16:41 | |
| anteaya | Swanson: here is the third party meeting asselin_ chairs: http://eavesdrop.openstack.org/#Third_Party_Working_Group_Meeting | 16:41 |
| Swanson | anteaya, Thanks! | 16:41 |
| anteaya | cbader_: I'm not going to take up any more channel space speaking with you on this item | 16:41 |
| anteaya | Swanson: welcome thanks for asking | 16:41 |
| jeblair | fungi, krotscheck: oh, host keys in hiera -- are we doing that in public hiera? | 16:42 |
| mordred | fungi: it landed | 16:42 |
| Swanson | anteaya, I had it on my calendar once. Must have been once cause I only remember making one meeting. | 16:42 |
| anteaya | Swanson: good thing you asked, there are all the meetings | 16:42 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: ignore deleted files when listing deliverable changes https://review.openstack.org/270910 | 16:42 |
| fungi | jeblair: we could do it in public hiera instead, sure. for now i just put it in a wheel-mirror.yaml group hiera file on the puppetmaster | 16:42 |
| Swanson | anteaya, thanks again | 16:43 |
| anteaya | zaro: thanks for those patches | 16:43 |
| anteaya | Swanson: welcome | 16:43 |
| anteaya | zaro: I commented | 16:43 |
| fungi | mordred: awesome, i'll finish rebasing and push this up real fast | 16:43 |
| jeblair | nibalizer: where are we on public hiera? | 16:43 |
| krotscheck | jeblair: expected node names are here: https://review.openstack.org/#/c/164927/19/jenkins/jobs/projects.yaml | 16:44 |
| fungi | jeblair: this is one of many, many things we could move to public hiera in my opinion | 16:44 |
| *** ldnunes_ has joined #openstack-infra | 16:44 | |
| *** fawadkhaliq has joined #openstack-infra | 16:44 | |
| *** bpokorny has joined #openstack-infra | 16:44 | |
| fungi | i'd love to see private hiera stripped down to just secret key material | 16:45 |
| *** placko has quit IRC | 16:45 | |
| jeblair | krotscheck, fungi: so the fact that we need to update puppet when adding a new mirror is :( | 16:45 |
| fungi | any anything else security-sensitive (like trove hostnames, because $reasons) | 16:45 |
| *** ashleighfarnham has joined #openstack-infra | 16:45 | |
| jeblair | krotscheck, fungi: we do not need to do that today, so this is a bit of a regression | 16:45 |
| *** tjones has quit IRC | 16:45 | |
| jeblair | krotscheck, fungi: perhaps the way to fix this though is to just fall forward into afs | 16:45 |
| fungi | yeah, i was looking at it as a temporary situation while you were working on centralized updates | 16:46 |
| *** burgerk has joined #openstack-infra | 16:46 | |
| jeblair | fungi: well, to be fair, i wasn't doing that until the day before yesterday | 16:46 |
| *** hdd has quit IRC | 16:46 | |
| jeblair | but now that i am, i guess that works :) | 16:46 |
| fungi | i didn't approve it until yesterday | 16:46 |
| jeblair | ha! | 16:46 |
| jeblair | always thinking ahead ;) | 16:46 |
| *** hdd has joined #openstack-infra | 16:46 | |
| *** ldnunes has quit IRC | 16:47 | |
| mordred | nibalizer: we have public hiera | 16:47 |
| openstackgerrit | Anne Gentle proposed openstack/requirements: Only needed for API reference doc builds, fairy-slipper 0.1.0 https://review.openstack.org/270915 | 16:47 |
| mordred | gah | 16:47 |
| mordred | jeblair: we have public hiera | 16:47 |
| jeblair | mordred: do we have anything in it? | 16:48 |
| mordred | we do | 16:48 |
| jeblair | mordred: where is it? | 16:48 |
| mordred | jeblair: system-config/hiera | 16:48 |
| *** esikachev has joined #openstack-infra | 16:48 | |
| mordred | jeblair: we, in fact, have many things in it | 16:49 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Put wheel-mirror workers in a common group https://review.openstack.org/270495 | 16:49 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 16:49 |
| fungi | mordred: jeblair: krotscheck: ^ | 16:49 |
| *** rcernin has joined #openstack-infra | 16:49 | |
| fungi | i can propose a change to move these host keys into public hiera in that case. presumably the global site manifest doesn't need adjusting? | 16:50 |
| mordred | fungi: nope. it should not | 16:50 |
| jeblair | fungi: right, so we don't even need to wait on that | 16:50 |
| *** ddieterly has quit IRC | 16:50 | |
| krotscheck | Well, I was _trying_ to do the unified mirrors thing, but then SOMEONE decided that AFS was a better idea ;) | 16:51 |
| mordred | krotscheck: AFS is always a better idea | 16:51 |
| *** yfried_ has joined #openstack-infra | 16:51 | |
| *** ddieterly has joined #openstack-infra | 16:51 | |
| *** bpokorny has quit IRC | 16:51 | |
| mordred | krotscheck: well, except bacon. AFS is not a better idea than bacon | 16:51 |
| *** vgridnev has quit IRC | 16:51 | |
| fungi | afs has been a better idea since before there were ideas | 16:51 |
| *** bpokorny has joined #openstack-infra | 16:52 | |
| *** bnemec has quit IRC | 16:52 | |
| jeblair | krotscheck: i've been thinking about your question about how to vos release the wheel mirrors... | 16:52 |
| fungi | or 1989 anyway | 16:52 |
| krotscheck | jeblair: Any ideas? | 16:52 |
| anteaya | can you have chocolate coated afs? | 16:52 |
| jeblair | krotscheck: the path for those are something like .../mirror/web/wheel/<arch>/stuff right? | 16:52 |
| krotscheck | jeblair: Cron job on the mirror master maybe? | 16:52 |
| krotscheck | yep | 16:52 |
| jeblair | krotscheck: i'm actually thinking that we make a volume per-arch | 16:53 |
| jeblair | krotscheck: and then have the job run vos release at the end | 16:53 |
| *** bpokorny has quit IRC | 16:53 | |
| krotscheck | jeblair: Well, for now, anyway. Once we do the unified mirror thing I feel that we'll need to think about the folder structure a little more. | 16:53 |
| *** Daisy has joined #openstack-infra | 16:53 | |
| jeblair | krotscheck: what's the unified mirror thing? | 16:53 |
| krotscheck | jeblair: One mirror to serve them all (npm, wheel, pypi, maven, rubygems, etc) | 16:54 |
| jeblair | that's not what we're doing? | 16:54 |
| *** fawadkhaliq has quit IRC | 16:54 | |
| *** haleyb has joined #openstack-infra | 16:54 | |
| *** fawadkhaliq has joined #openstack-infra | 16:54 | |
| jeblair | the last time i reviewed patches, that's exactly what this was about... | 16:55 |
| jeblair | when did that change? | 16:55 |
| anteaya | zxiiro: thank you: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-21.log.html#t2016-01-21T03:36:54 | 16:55 |
| krotscheck | jeblair: Yes. I the case of a wheel mirror, I wonder if it makes more sense to do /pypi/simple and /wheel/ARCH, rather than /pypi/simple /pypi/wheel/ARCH | 16:55 |
| krotscheck | My original patch chain was the former. | 16:56 |
| *** gyee has joined #openstack-infra | 16:56 | |
| krotscheck | (It created /pypi, /wheel, /npm, /rubygems, etc.) | 16:56 |
| jeblair | yeah, that's what i remember working with you on and reviewing | 16:57 |
| *** akuznetsova has quit IRC | 16:57 | |
| *** Nikolay_St has quit IRC | 16:57 | |
| *** vgridnev has joined #openstack-infra | 16:58 | |
| *** Daisy has quit IRC | 16:58 | |
| *** placko has joined #openstack-infra | 16:58 | |
| *** davidlenwell has quit IRC | 16:58 | |
| zxiiro | anteaya: no problem. I'll try to capture as much as I can soon. | 16:58 |
| *** baoli has quit IRC | 16:59 | |
| jeblair | krotscheck: this spec: https://review.openstack.org/252678 | 16:59 |
| *** davidlenwell has joined #openstack-infra | 16:59 | |
| jeblair | though, now that we're having this conversation, i'm starting to get a little worried that we haven't approved it | 16:59 |
| *** jklare has quit IRC | 17:00 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Logging in no longer causes page refresh. https://review.openstack.org/270882 | 17:00 |
| krotscheck | jeblair: Hrm. I didn't get comments. | 17:00 |
| krotscheck | Lemme look | 17:00 |
| anteaya | zxiiro: thanks for wanting to contribute back, we appreciate it | 17:00 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: Optimize announce subject line https://review.openstack.org/270923 | 17:01 |
| *** jklare has joined #openstack-infra | 17:01 | |
| *** vgridnev has quit IRC | 17:01 | |
| *** bnemec has joined #openstack-infra | 17:02 | |
| krotscheck | jeblair: Ok, so the wheel things are not directly attached to this spec. | 17:02 |
| *** akuznetsova has joined #openstack-infra | 17:02 | |
| openstackgerrit | Dina Belova proposed openstack-infra/project-config: Add py27 job for performance-docs https://review.openstack.org/270088 | 17:02 |
| krotscheck | jeblair: The AFS things, however, kindof are. | 17:02 |
| *** kzaitsev_mb has joined #openstack-infra | 17:02 | |
| *** rafa_ has joined #openstack-infra | 17:03 | |
| *** vgridnev has joined #openstack-infra | 17:03 | |
| jeblair | krotscheck: well, i mean any of this work can pivot to using afs pretty easily -- we just change the apache docroot | 17:03 |
| fungi | jeblair: krotscheck: i'm eager to get that spec up for a council vote, probably should have pushed to have it proposed for vote before now | 17:03 |
| jeblair | fungi: ++ | 17:04 |
| jeblair | so, i thought i was signing up to do the work in that spec | 17:04 |
| jeblair | i was clearly mistaken | 17:04 |
| openstackgerrit | Isaac Beckman proposed openstack-infra/puppet-jenkins: Customize jenkins.default by template https://review.openstack.org/262787 | 17:04 |
| jeblair | and have no idea what's actually going on | 17:04 |
| *** baoli has joined #openstack-infra | 17:05 | |
| *** knikolla has joined #openstack-infra | 17:05 | |
| mordred | jeblair, krotscheck: also, yes to /pypi/simple and /wheel/ARCH rather than /pypi/wheel/ARCH - I thnk that aligns more better with that spec | 17:05 |
| mordred | krotscheck: /pypi/wheel/ARCH came from being an in-place stepping-stone from the current situation, no? | 17:05 |
| jeblair | mordred: yeah, that makes sense | 17:05 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/infra-specs: Unified Mirrors https://review.openstack.org/252678 | 17:05 |
| krotscheck | Ok, updated. | 17:06 |
| *** blogan is now known as patient-zer0-bl0 | 17:06 | |
| jeblair | if we were going to put the wheel mirror at /pypi/wheel/arch then that blocks the use of afs on the pypi mirrors | 17:06 |
| *** patient-zer0-bl0 is now known as patient-0-bl0gan | 17:06 | |
| *** dtantsur is now known as dtantsur|afk | 17:06 | |
| jeblair | unless of course we do mod-rewrite gymnastics | 17:07 |
| *** coolsvap|away has quit IRC | 17:07 | |
| *** apoorvad has joined #openstack-infra | 17:07 | |
| *** salv-orlando has joined #openstack-infra | 17:10 | |
| *** isaacb has quit IRC | 17:10 | |
| *** Hal has joined #openstack-infra | 17:10 | |
| jeblair | krotscheck, fungi, mordred: are there obstacles with us proceeding with the unified mirror spec? | 17:10 |
| *** Hal is now known as Guest84688 | 17:11 | |
| mordred | jeblair: I think the only ones I'm aware of from the past are infra-root being available to do the node spin ups | 17:11 |
| jeblair | mordred: which is what i thought i was pitching in for :) | 17:11 |
| mordred | yah | 17:11 |
| mordred | jeblair: so I think we've just overcome that obstacle :) | 17:12 |
| *** eharney has quit IRC | 17:12 | |
| jeblair | mordred: i think there are a lot of puppet changes in the work items there though, and i don't know if they have been written, much less merged | 17:13 |
| *** nmagnezi has joined #openstack-infra | 17:13 | |
| krotscheck | jeblair: YOu mean the unified mirror changes? I had a patch chain ready to go that did it all on the assumption that not-afs. | 17:14 |
| krotscheck | https://review.openstack.org/#/q/status:open+branch:master+topic:unified_mirror | 17:14 |
| krotscheck | Then fungi asked to land wheel first because stability | 17:14 |
| krotscheck | So I let those bitrot for 2 weeks while wheel things happened | 17:14 |
| jeblair | krotscheck: yeah, let's not worry about afs for a bit -- it really should have very little impact on this, which is why my plan was to execute your spec and then switch to afs | 17:15 |
| jeblair | krotscheck, fungi: i had no idea we had dropped the unified mirror effort in favor of just wheels | 17:15 |
| krotscheck | Not dropped. Delayed | 17:15 |
| krotscheck | And, well, if we're going to be provisioining new mirror servers _anyway_ might as well start them off with AFS. | 17:16 |
| notmorgan | sdague: fwiw, i don't think python-memcache supports ipv6, it may have changed though? https://review.openstack.org/#/c/270474/6 | 17:16 |
| mordred | I agre both that we don't necessarily need to block on AFS for this - and also that jeblair's afs patches seem far enough along that just doing that might be less work | 17:17 |
| notmorgan | sdague: let me dig into that... we had an issue with that in the past. | 17:17 |
| sdague | notmorgan: that's fine, but it's definitely not going to work to try to talk to memcache server on the API node, from the workers, on 127.0.0.1 | 17:17 |
| notmorgan | ah ok | 17:18 |
| notmorgan | ++ | 17:18 |
| mordred | sdague: we could make 127.0.0.1 routable ... | 17:18 |
| notmorgan | i have a longer term fix for ipv6 but it needs oslo.cache | 17:18 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add baremetal hpuswest node definition https://review.openstack.org/208751 | 17:18 |
| notmorgan | so, i'm ok with living with it for a short bit using $SERVICE_HOST :) | 17:18 |
| sdague | mordred: sadly, that's not the craziest code concept I've been in today | 17:19 |
| haleyb | notmorgan: i think it does, at least i found that info after posting that comment | 17:19 |
| mordred | sdague: sadly, I believe you | 17:19 |
| notmorgan | haleyb: ok cool. it didn't for a while | 17:19 |
| sdague | mordred: also, that's why we can't have nice things | 17:19 |
| notmorgan | haleyb: i know. | 17:19 |
| krotscheck | jeblair: I would like to do the AFS thing sooner rather than later, so I can remove the cognitive load of "Getting an NPM mirror up" from my brainspace. | 17:19 |
| krotscheck | Doing Version 1 of mirrors, and then verison 2 of mirrors, is just adding unsettled todo items. | 17:20 |
| notmorgan | mordred: oh lets just use 7.0.0.0/8 as a space to camp as "internal" ips.. no one uses it, right? /s [past job insane leaking, sorry] | 17:20 |
| anteaya | did anyone do anything with jenkins07 lately? | 17:20 |
| jeblair | krotscheck: i believe the fastest way to get all of the mirrors we want up and running (afs or no) is to have the flexibility afforded by the unified mirror system | 17:20 |
| *** Swami has joined #openstack-infra | 17:21 | |
| jeblair | krotscheck: it sounds like we might be in agreement on that | 17:21 |
| * mordred is in agreement on that | 17:21 | |
| *** unicell has quit IRC | 17:21 | |
| krotscheck | Ok. So let's crush this like a brogrammer. | 17:21 |
| fungi | jeblair: qa was asking for wheel mirroring as an urgent need since we stopped installing some of the expensive-to-build dependencies from distro packages (which added 10+ minutes to every devstack run for example), and since greghaynes had the wheel mirror work already lined up i asked to proceed with it as a priority over node.js package mirroring | 17:21 |
| * krotscheck can't believe he actually said that. | 17:21 | |
| notmorgan | krotscheck: ... i ... | 17:21 |
| notmorgan | krotscheck: you have broken my head already today. | 17:21 |
| notmorgan | just with that statement | 17:21 |
| *** jogo has quit IRC | 17:22 | |
| anteaya | jenkins07 eth0 traffic is flatlined eth1 traffic continues | 17:22 |
| fungi | i basically just asked that we make sure the wheel mirror design was not at odds with being extended to also support javascript dependency mirroring, and that's what spurred the unified mirroring design krotscheck put together | 17:22 |
| krotscheck | Right, and the wheel things I built on top of greghaynes's work is AFS supportive. | 17:23 |
| fungi | so i don't feel like we dropped unified mirror work in favor of wheel mirror work | 17:23 |
| *** placko has quit IRC | 17:23 | |
| fungi | if anything, it increased priority on the unified mirror design | 17:24 |
| *** jistr has quit IRC | 17:24 | |
| jeblair | fungi: are you okay with us 'crushing' the unified mirror work then? | 17:24 |
| *** jogo has joined #openstack-infra | 17:24 | |
| *** jogo has joined #openstack-infra | 17:24 | |
| jeblair | krotscheck: looks like https://review.openstack.org/238754 needs an update? | 17:25 |
| * fungi is not up on brogrammer slang ;) | 17:25 | |
| krotscheck | jeblair: On it. | 17:25 |
| krotscheck | jeblair: Will rebase on top of your mirror_update | 17:25 |
| jeblair | fungi: i think it means something like 'just get it done' :) | 17:25 |
| fungi | by all means, please! | 17:26 |
| *** esikachev has quit IRC | 17:26 | |
| jeblair | drat | 17:26 |
| notmorgan | sdague: do we need to do the same thing for stable/liberty? $service_host and install where middleware is installed not where keystone is installed? | 17:26 |
| jeblair | we should have merged https://review.openstack.org/253504 ages ago | 17:26 |
| *** zeih has joined #openstack-infra | 17:27 | |
| notmorgan | sdague: i can roll that up quickly but the liberty fix has +A from you. | 17:27 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:27 | |
| notmorgan | sdague: before i bounce it asking. | 17:27 |
| jeblair | krotscheck: that ^ needs an update, then we need to merge it and kick off image builds | 17:27 |
| jeblair | krotscheck: i'll fix up that one | 17:27 |
| *** rafa_ has left #openstack-infra | 17:27 | |
| *** sriram has joined #openstack-infra | 17:28 | |
| sriram | hey all | 17:28 |
| fungi | yeah, i missed the later comment suggesting that it was safe to merge ahead (though i had sort of thought that might be the case when i saw the change) | 17:28 |
| *** armax has joined #openstack-infra | 17:28 | |
| fungi | krotscheck: i guess just also drop the comment about waiting to merge it from the commit message if you're updating anyway | 17:28 |
| krotscheck | jeblair: erp... ok | 17:28 |
| krotscheck | jeblair: I'm halfway through fixing that | 17:29 |
| jeblair | krotscheck: fixing 504? | 17:29 |
| krotscheck | yeay | 17:29 |
| sriram | so we at openstack poppy are affected by the pip issue, the problem we are seeing is tox starts up and virtualenv also got updated to use pip v8, is there a way to make tox use a specific version of virtualenv to get around this? | 17:29 |
| jeblair | krotscheck, fungi: though actually in order for it to be truly no-op, we should include all of the hosts, old and new, in the trusted hosts list | 17:29 |
| jeblair | krotscheck: so carry on fixing, but make that change ^ | 17:29 |
| sriram | any suggestions would be awesome :) | 17:29 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Update pypi_trusted_hosts https://review.openstack.org/253504 | 17:30 |
| krotscheck | Hoookay | 17:30 |
| krotscheck | sec | 17:30 |
| krotscheck | willdo | 17:30 |
| *** malini has joined #openstack-infra | 17:30 | |
| anteaya | sriram: this has been an issue since pip 8 was released Tuesday 22:00 utc ish | 17:30 |
| sriram | anteaya: yeah | 17:30 |
| anteaya | sriram: there have been various conversations on the right way forward in the logs: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/ | 17:31 |
| anteaya | I don't know as we as a group have come to a firm consenous on how to address the issue | 17:31 |
| jeblair | krotscheck: you found the cacti host list moved to the hiera file, right? | 17:31 |
| anteaya | some people are working around it | 17:31 |
| fungi | sriram: you would need to downgrade virtualenv 14.0.0 to <14 before invoking tox. from what i understand there's a 14.0.1 (with pip 8.0.1) on the way real soon now which may relieve the issues you're encountering | 17:31 |
| krotscheck | jeblair: Just noticed, yeah | 17:31 |
| anteaya | fungi: ah thank you | 17:31 |
| jeblair | i just learned that today too :) | 17:31 |
| krotscheck | jeblair: Seems like a sane place to put a trusted hosts list. | 17:32 |
| fungi | jeblair: now i've just learn't it as well | 17:32 |
| sriram | fungi: ah I see, but where would that go? | 17:32 |
| sriram | we cant put inside tox.ini | 17:32 |
| krotscheck | I'll update the commit message to note that those hosts will have oto be added | 17:32 |
| sriram | or in requirements | 17:32 |
| fungi | sriram: it would need to be a custom job | 17:32 |
| *** yolanda has quit IRC | 17:32 | |
| fungi | sriram: our images are built with virtualenv preinstalled | 17:32 |
| sriram | fungi: ah, where would I need to submit that job? | 17:33 |
| sriram | has someone already worked on getting such a job running? | 17:33 |
| sriram | fungi: ^ | 17:33 |
| *** dprince has joined #openstack-infra | 17:33 | |
| fungi | sriram: the chances of it getting reviewed and approved before virtualenv 14.0.1 releases are pretty slim, so i'm not recommending that as the path forward, just explaining what's involved | 17:33 |
| *** sputnik13 has joined #openstack-infra | 17:34 | |
| sriram | fungi: I see ok, thanks for the explanation :) | 17:34 |
| sriram | much appreciated. | 17:34 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Update pypi_trusted_hosts https://review.openstack.org/253504 | 17:36 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Remove legacy pypi_trusted_hosts https://review.openstack.org/270930 | 17:36 |
| krotscheck | fungi, jeblair: ^^ | 17:36 |
| krotscheck | I'm going to WIP the second to make sure it doesn't merge. | 17:36 |
| jeblair | krotscheck: oh, you can update the hiera file | 17:36 |
| *** malini has left #openstack-infra | 17:37 | |
| krotscheck | I can? | 17:37 |
| *** markus_z has quit IRC | 17:37 | |
| *** e0ne has quit IRC | 17:37 | |
| jeblair | krotscheck: it's a public hiera file in hiera/common.yaml in that repo | 17:37 |
| mordred | krotscheck: (we added new fun) | 17:37 |
| krotscheck | Oh | 17:37 |
| krotscheck | Well then | 17:37 |
| jeblair | one last update! (the rest of ps7 lgtm) | 17:37 |
| krotscheck | What is pypi.openstack.org? | 17:37 |
| krotscheck | It's not in server.pp or template.pp | 17:38 |
| jeblair | dead, i think at this point | 17:38 |
| *** samuelBartel has quit IRC | 17:38 | |
| jeblair | so maybe not fully removed? | 17:38 |
| *** esker has joined #openstack-infra | 17:39 | |
| openstackgerrit | Merged openstack/requirements: Cap keystonemiddleware version https://review.openstack.org/270417 | 17:39 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Remove legacy pypi_trusted_hosts https://review.openstack.org/270930 | 17:40 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Update pypi_trusted_hosts https://review.openstack.org/253504 | 17:40 |
| jeblair | krotscheck: lgtm; fungi, mordred: ^ | 17:41 |
| * krotscheck hopes that whitespace won't trigger a litn error | 17:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/268438 | 17:41 |
| jeblair | krotscheck: https://review.openstack.org/238754 is next? | 17:42 |
| krotscheck | ok, next patch | 17:42 |
| krotscheck | yep | 17:42 |
| *** sfinucan has quit IRC | 17:42 | |
| *** stevebaker has quit IRC | 17:43 | |
| *** _nadya_ has quit IRC | 17:44 | |
| krotscheck | jeblair: I'm going to pull the mirror things off of the instance creation, so we can step through the different mirror types one at a time. | 17:44 |
| jeblair | krotscheck: hrm, i don't follow | 17:46 |
| *** matrohon has quit IRC | 17:46 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node https://review.openstack.org/209698 | 17:47 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition https://review.openstack.org/213980 | 17:48 |
| krotscheck | Empty out https://review.openstack.org/#/c/238754/13/modules/openstack_project/manifests/mirror.pp so we can start with AFS. Then add hosting for pypi, then wheel, then... | 17:48 |
| krotscheck | Trying to create small changes. | 17:49 |
| *** davidlenwell has quit IRC | 17:50 | |
| *** esikachev has joined #openstack-infra | 17:50 | |
| *** kzaitsev_mb has quit IRC | 17:51 | |
| krotscheck | jeblair: the root dir is /afs/.openstack.org/mirror ? | 17:52 |
| *** unicell has joined #openstack-infra | 17:53 | |
| jeblair | krotscheck: drop the leading . in openstack | 17:53 |
| jeblair | krotscheck: the . is the read-write path, without is the read-only | 17:53 |
| krotscheck | Ahh | 17:53 |
| krotscheck | ok | 17:53 |
| *** SumitNaiksatam has joined #openstack-infra | 17:55 | |
| jeblair | krotscheck: i have a meeting coming up; i'll be back in an hour to continue | 17:55 |
| krotscheck | ok | 17:55 |
| jeblair | krotscheck: but if other cores can merge those changes, we'll be well prepared | 17:55 |
| *** davidlenwell has joined #openstack-infra | 17:55 | |
| *** fesp has joined #openstack-infra | 17:55 | |
| mgould | hi everyone! Could someone please take a look at https://review.openstack.org/#/c/266320/ ? | 17:56 |
| mgould | one-line change to g-r, blocking several other patches | 17:56 |
| mgould | [this is the right place to ask, right?] | 17:57 |
| *** _nadya_ has joined #openstack-infra | 17:58 | |
| *** yfried_ has quit IRC | 17:58 | |
| *** rkukura_ has joined #openstack-infra | 18:00 | |
| *** ihrachys has quit IRC | 18:00 | |
| *** jlanoux has quit IRC | 18:00 | |
| *** hdd has quit IRC | 18:01 | |
| *** rkukura has quit IRC | 18:02 | |
| *** rkukura_ is now known as rkukura | 18:02 | |
| *** derekh has quit IRC | 18:03 | |
| *** fawadk has joined #openstack-infra | 18:04 | |
| anteaya | mgould: these are teh folks who can help you: https://review.openstack.org/#/admin/groups/131,members | 18:04 |
| *** flepied1 is now known as flepied | 18:04 | |
| anteaya | they often are in here but not always | 18:04 |
| *** sabeen3 has quit IRC | 18:04 | |
| *** fawadkhaliq has quit IRC | 18:04 | |
| mgould | anteaya, thanks very much! | 18:04 |
| anteaya | mgould: welcome | 18:05 |
| *** skolekonov is now known as _skolekonov | 18:05 | |
| *** Zhongjun_ has quit IRC | 18:08 | |
| *** hdd has joined #openstack-infra | 18:09 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Created mirror.<region>.openstack.org https://review.openstack.org/238754 | 18:09 |
| *** jaosorior has quit IRC | 18:09 | |
| *** jaosorior has joined #openstack-infra | 18:10 | |
| krotscheck | OH bloody... | 18:11 |
| krotscheck | sec | 18:11 |
| *** jordanP has quit IRC | 18:12 | |
| openstackgerrit | Miles Gould proposed openstack/requirements: Bump minimum python-ironicclient version to 0.9.0 https://review.openstack.org/266320 | 18:12 |
| anteaya | any infra-root free to check jenkins07? | 18:13 |
| *** slagle has joined #openstack-infra | 18:13 | |
| anteaya | dougwig helped me look at cacti for it and suspects jenkins might have been OOM'd on that server | 18:13 |
| dougwig | or something else eating memory. | 18:14 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Add mirror.<region>.openstack.org https://review.openstack.org/238754 | 18:14 |
| krotscheck | There we go | 18:14 |
| anteaya | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=116&page=2 | 18:14 |
| jeblair | anteaya: the good news is it looks like nodepool is ignoring it: http://grafana.openstack.org/dashboard/db/nodepool | 18:15 |
| krotscheck | Ok, so I munged the commits and the author is now wacky, but that's fine. | 18:15 |
| anteaya | well done nodepool | 18:15 |
| jeblair | anteaya: and as long as it isn't dragging nodepool down with it, it's probably not terribly urgent | 18:15 |
| anteaya | oh okay | 18:15 |
| mordred | krotscheck: I do not think you need to pass in fqdn from the site.pp - BUT - it also does not hurt anything, so meh | 18:16 |
| anteaya | dougwig: thanks for your help | 18:16 |
| krotscheck | mordred, fungi Patch chain for you at https://review.openstack.org/#/c/238754/15 and https://review.openstack.org/#/c/270520/5 | 18:16 |
| jeblair | anteaya: let me know if it gets worse -- but i'm going to let another infra-root take care of it and focus on the mirror thing with krotscheck | 18:16 |
| jeblair | krotscheck: surprisingly short meeting; i'm back. | 18:16 |
| anteaya | jeblair: understood, didn't mean to interrupt, go mirrors | 18:17 |
| jeblair | gooooo mirrors! | 18:17 |
| anteaya | I'll keep an eye on the graphs | 18:17 |
| *** dizquierdo has joined #openstack-infra | 18:17 | |
| * krotscheck pops his collar. | 18:17 | |
| *** bardia has quit IRC | 18:17 | |
| openstackgerrit | Xingchao Yu proposed openstack-infra/project-config: Create puppet-oslo module https://review.openstack.org/270895 | 18:18 |
| * krotscheck was pleasantly surprised that adding AFS was a single falg. | 18:18 | |
| krotscheck | *flag | 18:18 |
| * krotscheck is rebuilding his mirror vm, just a moment. | 18:19 | |
| *** sridhar_ram has joined #openstack-infra | 18:19 | |
| *** abregman has quit IRC | 18:20 | |
| jeblair | fungi: do you have time to review https://review.openstack.org/270520 ? | 18:20 |
| nibalizer | jeblair: yep public hiera went in | 18:20 |
| nibalizer | seems pretty useful | 18:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-bandersnatch: Make bandersnatch::cron class https://review.openstack.org/270498 | 18:21 |
| *** mhickey has quit IRC | 18:22 | |
| *** trown is now known as trown|lunch | 18:22 | |
| jeblair | mordred, nibalizer: can you double check https://review.openstack.org/253504 ? | 18:23 |
| *** esker has quit IRC | 18:24 | |
| jeblair | greghaynes, clarkb: i'm about to need to kick off image rebuilds; what do i need to know? | 18:24 |
| *** esker has joined #openstack-infra | 18:25 | |
| clarkb | jeblair: I thin kwe will need to delete the dib based images in the db that are building because it isn't resilient against gear going away | 18:25 |
| greghaynes | jeblair: I dont know of anything specific - yesterday the gearman socket died after submitting all the image-upload jobs which killed all the queued jobs... | 18:25 |
| openstackgerrit | Anne Gentle proposed openstack/requirements: For API reference doc builds, fairy-slipper 0.1.0 https://review.openstack.org/270915 | 18:25 |
| mordred | jeblair: lgtm | 18:25 |
| clarkb | greghaynes: right it killed them running but the db is stale, so we have to clean that up | 18:25 |
| mordred | btw - now that we have in-tree hiera - we can stop needing to plumb so many things through | 18:25 |
| clarkb | jeblair: other than that everything should work as before | 18:26 |
| greghaynes | clarkb: ah, yep | 18:26 |
| jeblair | clarkb, greghaynes: thanks, i'll report what i find | 18:26 |
| greghaynes | clarkb: Wait, so what state did the images get stuck in? | 18:26 |
| greghaynes | (in the nodepool db) | 18:26 |
| clarkb | greghaynes: "building" | 18:26 |
| greghaynes | huh. I guess because nodepool sets to building, sends to gear, and then just waits | 18:26 |
| greghaynes | :/ | 18:26 |
| clarkb | yup | 18:26 |
| clarkb | then connection died and no update was made to the db | 18:26 |
| greghaynes | Does the client know when a job is accepted to be worked on? | 18:27 |
| greghaynes | as opposed to just submitted | 18:27 |
| clarkb | we should either resubmit jobs or change the state to delete | 18:27 |
| clarkb | greghaynes: yes it should be told that iirc | 18:27 |
| jeblair | nodepool should probably change the state when it gets the work_fail packet | 18:27 |
| clarkb | with information about the worker | 18:27 |
| jeblair | clarkb: yeah that | 18:27 |
| AJaeger | anteaya: could you approve another linters change for me, please? https://review.openstack.org/270469 | 18:27 |
| greghaynes | jeblair: I think the socket just up and died so it never got a work failed | 18:27 |
| clarkb | jeblair: it does, I don't think it got a work fail because the connection died | 18:27 |
| jeblair | on disconnection the client that submitted the job will get a work_fail | 18:27 |
| jeblair | oh | 18:27 |
| jeblair | it was the client that lost the connection not the worker | 18:28 |
| clarkb | correct | 18:28 |
| greghaynes | I wonder if we should not set images to building until the job is actually accepted | 18:28 |
| *** stevebaker has joined #openstack-infra | 18:28 | |
| greghaynes | as in, once the builder starts building | 18:28 |
| clarkb | greghaynes: ++ | 18:28 |
| *** rkukura has quit IRC | 18:28 | |
| *** knikolla has quit IRC | 18:29 | |
| anteaya | AJaeger: done, trying to keep the channel clear for mirrors work to proceed | 18:30 |
| jeblair | greghaynes, clarkb: we can implement this method on our client to get notified of that: http://gear.readthedocs.org/en/latest/#gear.Client.handleDisconnect | 18:30 |
| AJaeger | krotscheck: http://pypi.openstack.org redirects to doc.openstack.org/infra/... | 18:30 |
| *** rossella_s has quit IRC | 18:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Move pypi config into public hiera https://review.openstack.org/270966 | 18:31 |
| *** jflynn has quit IRC | 18:31 | |
| krotscheck | sec | 18:31 |
| mordred | jeblair, nibalizer, fungi: ^^ not urgent - just wrote that as a POC/example | 18:31 |
| *** rossella_s has joined #openstack-infra | 18:31 | |
| greghaynes | jeblair: ah, and that is for every job. nice | 18:32 |
| nibalizer | jeblair: oh I left come comments on 270520 | 18:32 |
| *** woodster_ has joined #openstack-infra | 18:32 | |
| nibalizer | can likely be fixe in a follow up | 18:32 |
| AJaeger | anteaya: ok | 18:32 |
| *** fesp has quit IRC | 18:32 | |
| *** mgould has quit IRC | 18:32 | |
| *** zeih has quit IRC | 18:33 | |
| *** otter768 has joined #openstack-infra | 18:33 | |
| jeblair | nibalizer: responding, but it'll take me a minute | 18:33 |
| esikachev | AJaeger: take a look on my comment https://review.openstack.org/#/c/270868 i want to use sahara-scenario master branch for testing sahara master and stable/liberty. how to do it? | 18:33 |
| *** kdas_ has joined #openstack-infra | 18:34 | |
| *** gyee has quit IRC | 18:34 | |
| nibalizer | for 253504 it makes me think we should move pypi trusted hosts into hiera | 18:34 |
| anteaya | AJaeger: thanks | 18:34 |
| jeblair | nibalizer: yes we should, that's what prompted my question earlier :) | 18:34 |
| jeblair | oh that was a different change | 18:35 |
| *** kushal has quit IRC | 18:35 | |
| jeblair | sorry, but still yes. maybe we can do it as a followup though | 18:35 |
| *** daemontool has quit IRC | 18:35 | |
| jeblair | nibalizer: responded on 270520 | 18:36 |
| *** jaosorior_ has joined #openstack-infra | 18:36 | |
| *** kdas_ is now known as kushal | 18:36 | |
| *** kushal has joined #openstack-infra | 18:36 | |
| AJaeger | esikachev: I'll check later... | 18:37 |
| esikachev | ok, thanks | 18:37 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on Client Update worflow https://review.openstack.org/270969 | 18:37 |
| jeblair | nibalizer: (afs factoid: when traversing through an afs mountpoint from one volume to another, if you are leaving a read-only volume, afs will prefer the read-only version of the volume that you are moving to. so if you set up /afs/openstack.org as a read-only volume (it is) then for any volumes you mount under it, generally afs will prefer the read-only versions of those) | 18:37 |
| *** otter768 has quit IRC | 18:38 | |
| *** dtroyer has quit IRC | 18:38 | |
| *** electrofelix has quit IRC | 18:39 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Add mirror.<region>.openstack.org https://review.openstack.org/238754 | 18:39 |
| jeblair | nibalizer: (so there's a convention to, at the root volume for a cell, _also_ create a /afs/.openstack.org mountpoint which is explicitly a read-write mount, and then afs will prefer the read-write version for any mounts under it) | 18:39 |
| krotscheck | Forgot a require ^^ | 18:40 |
| jeblair | the upshot is that the easy default thing in almost all circumstances is you end up on read-only volumes, which is exactly what we want for mirrors | 18:40 |
| *** edmondsw has quit IRC | 18:41 | |
| *** dtroyer has joined #openstack-infra | 18:41 | |
| mordred | jeblair: I'm guessing homedirs in eos worked without /afs/.eos.ncsu.ed/ because there were no read-only volumes for user lockers? | 18:41 |
| fungi | okay, back now--sorry. ended up wrestling a local computer issue during lunch. catching up on scrollback now | 18:42 |
| jeblair | mordred: yeah, as soon as there is a volume without a read-only version, you end up on the read-write path | 18:42 |
| mordred | fungi: we're crushing mirrors | 18:42 |
| jeblair | mordred: i think there is an .eos.ncsu.edu | 18:42 |
| mordred | jeblair: yes, I believe there is | 18:42 |
| mordred | mordred@camelot:~/src/openstack-infra/system-config$ ls /afs/.eos.ncsu.edu | 18:43 |
| mordred | adm contrib dist engrwww lockers public service source users | 18:43 |
| mordred | chass courses engrservers info project sadm software system www | 18:43 |
| mordred | turns out its easy to check | 18:43 |
| *** fawadk has quit IRC | 18:43 | |
| openstackgerrit | Merged openstack-infra/project-config: openstack-doc-tools,openstackdocstheme: Use linters https://review.openstack.org/270469 | 18:43 |
| nibalizer | jeblair: ok | 18:43 |
| nibalizer | TIL things | 18:43 |
| *** kzaitsev_mb has joined #openstack-infra | 18:43 | |
| openstackgerrit | Craig Vyvial proposed openstack/requirements: bump troveclient to 2.0.0 https://review.openstack.org/270973 | 18:43 |
| mordred | jeblair: too bad: ls: cannot open directory /afs/eos.ncsu.edu/source/ultrix42: Permission denied | 18:44 |
| jeblair | bwaha | 18:44 |
| *** bpokorny has joined #openstack-infra | 18:44 | |
| fungi | krotscheck: jeblair: pypi.openstack.org was a vhost on static.openstack.org, at least in its late stages i think? | 18:45 |
| mordred | I thnk so? | 18:45 |
| sriram | fungi: ping | 18:45 |
| sriram | so where is the repository that we submit additional custom jenkins job to? | 18:46 |
| sriram | I'm thinking of submitting it, and if pip and virtualenv are fixed before that | 18:46 |
| sriram | then I can abandon that patch | 18:46 |
| *** sabeen has joined #openstack-infra | 18:46 | |
| fungi | anteaya: yeah, looks like jenkins07 is dead, probably ran away with available memory and got killed for its trouble. i'll get it cleaned up and restarted if nobody else indicates they're already on it | 18:47 |
| *** malini has joined #openstack-infra | 18:48 | |
| *** mriedem has quit IRC | 18:48 | |
| anteaya | fungi: thank you | 18:48 |
| anteaya | fungi: I haven't seen anyone else indicate they are fixing it | 18:48 |
| nibalizer | jeblair: i have discovered the issue | 18:49 |
| *** dizquierdo has quit IRC | 18:49 | |
| *** tonytan4ever has quit IRC | 18:49 | |
| nibalizer | you created /opt/system-config/hieradata/production/fqdn/mirror_update.openstack.org and needed to create that + .yaml at the end | 18:49 |
| fungi | nibalizer: your concerns on 270520 seem to be addressed in jeblair's comments. are you still opposed? | 18:49 |
| *** sabeen3 has joined #openstack-infra | 18:49 | |
| jeblair | nibalizer: doh | 18:49 |
| nibalizer | fungi: im still poking | 18:49 |
| anteaya | sriram: you can create a file for project specific jobs in this directory: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs | 18:49 |
| nibalizer | I think when jeblair says he tested it he tested it with python | 18:50 |
| nibalizer | so Im making sure that puppet dtrt | 18:50 |
| jeblair | no i tested it with puppet | 18:50 |
| SpamapS | Could somebody point me to the way to have docs.openstack.org/developer/ populated for any repo? (Specifically os-performance-tools) ? | 18:50 |
| jeblair | but i did not test _that_ file | 18:50 |
| nibalizer | oh then yes I'm good to go | 18:50 |
| jeblair | i tested a file with .yaml at the end :) | 18:50 |
| * nibalizer changes vote | 18:50 | |
| mordred | SpamapS: you need the doc-publishing macros | 18:50 |
| mordred | SpamapS: one sec | 18:50 |
| jeblair | nibalizer: i'll go fix puppetmaster | 18:50 |
| jeblair | thank | 18:50 |
| jeblair | s | 18:50 |
| *** jpr has quit IRC | 18:51 | |
| *** mriedem has joined #openstack-infra | 18:51 | |
| anteaya | SpamapS: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/openstack-publish-jobs.yaml | 18:51 |
| nibalizer | jeblair: np, I've lost many hours to that mistake | 18:51 |
| *** mrmartin has joined #openstack-infra | 18:51 | |
| *** sabeen has quit IRC | 18:51 | |
| jeblair | puppetmaster fixed | 18:52 |
| mordred | SpamapS: you want - openstack-publish-jobs and doc-publisher-site: docs.openstack.org defined for the project in jenkins/jobs/projects.yaml | 18:52 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 18:52 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files https://review.openstack.org/270563 | 18:53 |
| jeblair | SpamapS: do you mind if infra-core is core on os-performance-tools? | 18:53 |
| SpamapS | jeblair: I'd love for infra-core to be core on that. :) | 18:53 |
| jeblair | SpamapS: yay! happiness abounds! mtreinish ^ | 18:54 |
| SpamapS | jeblair: since it straddles QA and infra I think that makes perfect sense. | 18:54 |
| mordred | SpamapS: then openstack-server-publish-jobs in zuul/layout.yaml if you want docs pushed on every commit - or openstack-client-publish-jobs if you want them only on releases | 18:54 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added pypi hosting for AFS mirror slave https://review.openstack.org/270975 | 18:54 |
| SpamapS | also.. I've been playing with influxdb lately | 18:54 |
| SpamapS | And, I kind of want to say that subunit2sql and some of graphite should be moved to influxdb | 18:55 |
| SpamapS | mtreinish: ^ | 18:55 |
| nibalizer | jeblair: while you're in that mind state: https://review.openstack.org/#/c/244917/ has a jeblair -1 and a spamaps response | 18:55 |
| AJaeger | SpamapS: once that project-config change is in, send a change for openstack-manuals/www/developer to update the index page on docs.openstack.org | 18:55 |
| openstackgerrit | Merged openstack-infra/project-config: Migrate scenario gate on sahara-scenario repo https://review.openstack.org/270868 | 18:55 |
| SpamapS | nibalizer: hah, I was going to get the docs publishing before I asked jeblair to re-consider. ;) | 18:55 |
| mordred | SpamapS: I thought influxdb was still alpha-ish | 18:55 |
| SpamapS | mordred: it's working really well for what we're doing with it on the SoftLayer/IBM cloud. :) | 18:56 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Add helper script to delete Gerrit groups https://review.openstack.org/270537 | 18:56 |
| nibalizer | mordred: but we're 'elive devops' you see so... | 18:56 |
| mordred | nibalizer: what's an elive? | 18:56 |
| *** e0ne has joined #openstack-infra | 18:56 | |
| greghaynes | graphite isn't exactly what I would call 'stable' | 18:56 |
| *** pvaneck has joined #openstack-infra | 18:57 | |
| mordred | nibalizer: am I not elive enough to understand? | 18:57 |
| SpamapS | The reason I say this is that it is _heavily_ optimized for time-series storage and queries. | 18:57 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added wheel hosting for AFS mirror slave https://review.openstack.org/270979 | 18:57 |
| SpamapS | which is exactly what subunit2sql is | 18:57 |
| *** angdraug has quit IRC | 18:57 | |
| nibalizer | mordred: | 18:57 |
| nibalizer | mordred: i meant to say 'elite' | 18:57 |
| SpamapS | as much fun as it has been making the sql part of subunit2sql as fast as possible... | 18:57 |
| greghaynes | it seems like influx is better at doign large numbers of metrics too | 18:57 |
| SpamapS | I'd rather it just lived in a data store that is built for that. | 18:57 |
| mordred | nibalizer: I know - I was making fun of you | 18:58 |
| jeblair | greghaynes: better than carbon and whisper? | 18:58 |
| jeblair | greghaynes: because those can _seriously_ scale. | 18:58 |
| nibalizer | mordred: elite devops dosen't need to spell, we have automation for that | 18:58 |
| mordred | nibalizer: ++ | 18:58 |
| greghaynes | jeblair: well they are rrdtool style, so for each metric you waste the full metric size on disk | 18:58 |
| jeblair | greghaynes: yes. that's part of how they are fast. | 18:59 |
| openstackgerrit | Merged openstack-infra/openstackid: Fix on Client Update worflow https://review.openstack.org/270969 | 18:59 |
| fungi | jenkins07 verdict is indeed that it ran out of memory, but in the jvm not kernel virtual memory. left this in jenkins.log as it terminated: There is insufficient memory for the Java Runtime Environment to continue. Native memory allocation (malloc) failed to allocate 612368384 bytes for committing reserved memory. | 18:59 |
| greghaynes | jeblair: yep, so depends whether you want fater per metric or not to use all that space up | 18:59 |
| clarkb | fungi: that is the thread leak I think | 18:59 |
| greghaynes | at least thats what I noticed in my few mins of influxdbing since I have had that hangup with carbon/whisper before | 18:59 |
| *** nmagnezi has quit IRC | 18:59 | |
| clarkb | fungi: we don't run out of heap we run out of memory whereever thread stacks go | 18:59 |
| fungi | anyway, i've deleted all jenkins07 nodes in nodepool and cleaned out all its slaves from the config. it's starting back up now | 19:00 |
| krotscheck | jeblair: Had to recheck the bandersnatch::httpd thing because devstack | 19:00 |
| jeblair | krotscheck: thanks | 19:00 |
| fungi | clarkb: makes sense | 19:00 |
| jeblair | how did that pass earlier? | 19:01 |
| greghaynes | also IME whisper-proxy or w/e it is called tends to go out to lunch when you have it writing to a large number of metrics | 19:01 |
| nibalizer | there are a couple influxdb modules on the foreg https://forge.puppetlabs.com/tags/influxdb | 19:01 |
| jeblair | greghaynes: it's called carbon-cache, and you can make a fleet of them to scale it. | 19:01 |
| *** zeih has joined #openstack-infra | 19:01 | |
| *** esikachev has quit IRC | 19:01 | |
| jeblair | anyway, can we talk about this later? | 19:02 |
| greghaynes | Yep | 19:02 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:02 | |
| sriram | fungi, anteaya: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs, looks like that would create separate jobs, what we need is to wrap all our existing jobs, such that a bash script gets run first. | 19:03 |
| sriram | or maybe I am misunderstanding what is intended. | 19:03 |
| sriram | could you guys shed some light there please :) | 19:03 |
| *** zeih has quit IRC | 19:04 | |
| krotscheck | jeblair: Going to update the wheel slave patches now. | 19:06 |
| *** Sukhdev has joined #openstack-infra | 19:07 | |
| openstackgerrit | Khai Do proposed openstack-infra/jeepyb: Fix handling of Gerrit system groups https://review.openstack.org/270628 | 19:07 |
| anteaya | sriram: well we are trying to focus on getting mirrors working right now, so trying to keep other conversations to a minumum, does looking at other project specific jobs in their files shed some light? | 19:08 |
| anteaya | sriram: keep in mind, no other project has code for this | 19:08 |
| anteaya | fungi: thank you | 19:10 |
| *** FallenPegasus has joined #openstack-infra | 19:10 | |
| krotscheck | Oh wait, forgot the AFS rewrite things... | 19:10 |
| *** eharney has joined #openstack-infra | 19:11 | |
| *** ybathia has joined #openstack-infra | 19:11 | |
| openstackgerrit | Khai Do proposed openstack-infra/jeepyb: Fix handling of Gerrit system groups https://review.openstack.org/270628 | 19:11 |
| *** yamahata has quit IRC | 19:12 | |
| *** fesp has joined #openstack-infra | 19:13 | |
| *** sshnaidm has joined #openstack-infra | 19:14 | |
| jeblair | clarkb, fungi, mordred: http://logs.openstack.org/86/270486/2/gate/gate-openstackci-beaker-trusty-dsvm/9a35480/console.html failed due to pip8, but it worked in the previous run... why? | 19:14 |
| clarkb | jeblair: image updates maybe? | 19:15 |
| jeblair | what's our solution for that? | 19:15 |
| openstackgerrit | Merged openstack-infra/system-config: Update pypi_trusted_hosts https://review.openstack.org/253504 | 19:15 |
| jeblair | clarkb: we haven't pinned pip on images have we? | 19:16 |
| fungi | yeah, i suggested yesterday that we should consider deleting the new snapshot-based images (dib-based were failing to build at the time) but nobody seemed to chime in agreeing | 19:16 |
| jeblair | clarkb: so you're thinking one of the images is pre-pip8 still? | 19:16 |
| clarkb | jeblair: ya that would be the first thing I would check | 19:16 |
| jeblair | fungi: well, i'm about to need new ones across the board so it's not a good med-term strategy | 19:16 |
| clarkb | fungi: well at the time we thought we would have a fix in a couple hours :) | 19:16 |
| *** ociuhandu has quit IRC | 19:16 | |
| jeblair | and pip8.0.1 won't fix us anyway | 19:17 |
| jeblair | unless they fully revert | 19:17 |
| mordred | I agree with pinning to <8 on the base images | 19:17 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 19:17 |
| fungi | dstufft seemed to imply a couple hours ago that there was actually a fix coming to address the distutils-based system package problem as well | 19:17 |
| clarkb | the problem with pinning is it will just update itself | 19:17 |
| clarkb | apparently pip in virutalenv upgrades itself | 19:17 |
| clarkb | (not an issue for beaker tests though) | 19:18 |
| fungi | well, also we need to pin virtualenv<14 and not so much pip<8 | 19:18 |
| jeblair | no it's pip | 19:18 |
| fungi | oh, this is not a tox-based job. got it | 19:18 |
| jeblair | our problems are outside the virtualenv | 19:18 |
| clarkb | fungi: right but I think it was lifeless that said pinning virutalenv won't help beacuse it will upgrade its pip | 19:18 |
| anteaya | fungi: 7 hours ago | 19:18 |
| clarkb | jeblair: for this subset of jobs yes | 19:18 |
| *** elo has joined #openstack-infra | 19:18 | |
| jeblair | did anyone write a pin-pip8 macro? | 19:18 |
| anteaya | I did not | 19:19 |
| jeblair | mordred: you wrote an update to install_puppet.sh, right? did that merge? | 19:19 |
| mordred | I did | 19:19 |
| fungi | anteaya: almost 3 hours ago (16:32:48 utc) | 19:19 |
| mordred | jeblair: https://review.openstack.org/#/c/270503/ | 19:19 |
| mordred | clarkb, fungi: ^^ | 19:19 |
| mordred | needs another +2 | 19:19 |
| anteaya | fungi: sorry | 19:20 |
| jeblair | mordred, fungi: i will write the pin-pip8 macro that will help that, and all our other changes, pass the beaker tests | 19:20 |
| anteaya | fungi: 7 hours ago he announced he was working on 8.0.1 again, not the distutils-based system package problem | 19:20 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: skip most tests on a nova locale import https://review.openstack.org/270987 | 19:21 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 19:21 |
| fungi | mordred: approved, though note that get-pip.py now supports being passed a version specifier directly | 19:21 |
| fungi | as of yesterday i think | 19:21 |
| mordred | neat | 19:22 |
| zaro | zxiiro: morning | 19:22 |
| fungi | it'll be fun tracking down all the places we've hacked around pip 8 so that we can undo them again some day. i'm not hopeful it'll be this year given how long unrolling similar pip/virtualenv/tox pins have taken us in the past | 19:22 |
| greghaynes | if only https://review.openstack.org/#/c/245297/ was ready to merge then it would be trivial to make dib images with whatever pip version you want | 19:22 |
| zaro | zxiiro: what version of zuul are you using? | 19:23 |
| *** malini has quit IRC | 19:23 | |
| fungi | kinda worried openstack will be a pip7.x and older only project for many moons | 19:23 |
| mordred | fungi: that does, in fact, concern me | 19:23 |
| zxiiro | zaro: morning, i'm using the one from pip install | 19:23 |
| mordred | fungi: especially since I've spent the last 2 years yelling at everyone that they should ALWAYS use the latest pip | 19:24 |
| jeblair | i just want to use pip from packages. | 19:24 |
| jeblair | but i can defer that discussion for later. | 19:24 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config: Convert kolla-mesos to linters https://review.openstack.org/270990 | 19:24 |
| zxiiro | zaro: 2.1.0 | 19:24 |
| lifeless | clarkb: pinning virtualenv will work for <14 | 19:24 |
| jeblair | mordred, fungi: do we not run install_puppet on our images? | 19:24 |
| *** rkukura has joined #openstack-infra | 19:25 | |
| mordred | jeblair: I thought we did | 19:25 |
| clarkb | jeblair we do | 19:25 |
| lifeless | clarkb: in 14, virtualenv starts doing a -U and will pull in latest pip etc newer than the virtualenv release | 19:25 |
| mordred | I thought that was how we installed pip on our images | 19:25 |
| jeblair | where does the get-pip.py script land? | 19:25 |
| jeblair | oh | 19:25 |
| greghaynes | mordred: pip will be install from system packages due to simple-init | 19:25 |
| clarkb | lifeless gotcha | 19:25 |
| jeblair | "rm get-pip.py" | 19:25 |
| lifeless | clarkb: (which for now, is pip 8; once 8.1 comes out (today hopefully!) virtualenv 14 will install that automatically | 19:25 |
| mordred | greghaynes: what? | 19:25 |
| fungi | jeblair: nodepool/elements/puppet/install.d/05-puppet and nodepool/scripts/prepare_node.sh in system-config | 19:25 |
| greghaynes | I am not sure if the puppet does a removal of that and then installs new pip | 19:26 |
| *** tonytan4ever has joined #openstack-infra | 19:26 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: only run api-guide on tox.ini or api-guide changes https://review.openstack.org/270992 | 19:26 |
| *** trown|lunch is now known as trown | 19:26 | |
| jeblair | so if we want to pin pip, we have to download get-pip again? | 19:26 |
| jeblair | only the devstack images cache that? | 19:26 |
| greghaynes | mordred: http://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/simple-init/package-installs.yaml | 19:26 |
| greghaynes | mordred: I think we uninstall that later on though so it isn't actually an issue | 19:26 |
| mordred | jeblair: if pip is already on the box, we should just be able to do pip install 'pip <8' | 19:27 |
| clarkb | jeblair pip is installed by get pip at that point get pip is no longer needed | 19:27 |
| mtreinish | jeblair: ok, I added infra core to os-performance-tools-core | 19:27 |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: include keystonev2 role assignments https://review.openstack.org/270993 | 19:27 |
| jeblair | okay | 19:27 |
| zaro | zxiiro: are you using gearman that is packaged with zuul? | 19:27 |
| zaro | gearman server | 19:27 |
| fungi | jeblair: sounds likely. it's cached in devstack images because devstack calls it (and downloads it if not already present) | 19:27 |
| zxiiro | zaro: yes I'm using zuul's gearman | 19:27 |
| mtreinish | SpamapS: I'm not opposed to exploring other data stores, but the issue with that is basically we have to pivot everything over to be different which is a lot of work | 19:27 |
| jeblair | oh, we only have to use get-pip when we want to install it after uninstalling the package | 19:27 |
| mordred | jeblair: yes | 19:27 |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder: Force dib-python symlink creation https://review.openstack.org/270994 | 19:27 |
| jeblair | gotcha | 19:27 |
| mtreinish | considering it took > 1 year to get to this point | 19:28 |
| jeblair | so just "pip install -U pip<8" ? | 19:28 |
| fungi | jeblair: and the caching was actually explicitly added for that case because we were seeing semi-frequent outages of where it was hosted tanking all devstack-based jobs | 19:28 |
| mordred | jeblair: yes - with quotes, so it's not a redirect | 19:28 |
| jeblair | heh | 19:28 |
| lifeless | jeblair: you don't need the -U | 19:28 |
| lifeless | jeblair: and you'll need the quotes as mordred says | 19:28 |
| *** SumitNaiksatam has quit IRC | 19:28 | |
| SpamapS | mtreinish: Right, not saying it's free. But I think you're stuck at this point. The materialized view approach I mentioned earlier this week will have to be done query-by-query.. and it's only going to get worse. | 19:29 |
| *** pradk has quit IRC | 19:29 | |
| *** edmondsw has joined #openstack-infra | 19:29 | |
| openstackgerrit | Merged openstack-infra/system-config: Pin pip to less than version 8 in install_puppet.sh https://review.openstack.org/270503 | 19:31 |
| mordred | yay! | 19:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Pin pip to <8 for openstackci-beaker jobs https://review.openstack.org/270995 | 19:31 |
| jeblair | mordred, fungi, clarkb: ^ | 19:31 |
| jeblair | mordred: yay, now it's safe to spin up new servers! | 19:31 |
| *** Guest84688 has quit IRC | 19:32 | |
| *** jpr has joined #openstack-infra | 19:33 | |
| anteaya | sriram: see https://review.openstack.org/#/c/270995/1, use the pin-pip macro in your job like the patch shows | 19:34 |
| mordred | jeblair: btw - we should land your shade launch_node patch | 19:34 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added wheel hosting for AFS mirror slave https://review.openstack.org/270979 | 19:34 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added pypi hosting for AFS mirror slave https://review.openstack.org/270975 | 19:34 |
| krotscheck | THere we go | 19:34 |
| krotscheck | jeblair: How are you generating that index.html file in the pypi directory? | 19:34 |
| jeblair | krotscheck: i think bandersnatch does it | 19:35 |
| krotscheck | jeblair: Gotcah | 19:35 |
| sriram | anteaya: yeah, but virtualenv 14 comes with pip8 installed, we probably need to pin that as well. | 19:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add a shade launch-node https://review.openstack.org/234975 | 19:36 |
| mordred | jeblair: ^^ updated | 19:36 |
| mordred | jeblair: I rebased it and included the "remove the inventory cache" lines | 19:36 |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 19:37 |
| jeblair | krotscheck: yeah i think https://bitbucket.org/pypa/bandersnatch/pull-requests/16/add-option-to-dir-hash-index-files/diff#Lsrc/bandersnatch/mirror.pyF177T207 is where i changed it for the dir hashing | 19:37 |
| krotscheck | jeblair: Gotcha | 19:37 |
| anteaya | sriram: you can create a macro, like that patch demonstrates, and use the macro | 19:37 |
| sriram | anteaya: yes | 19:37 |
| krotscheck | jeblair: Well, I've got the vhost for pypi up, and best I can tell my local vm is now mirroring the pypi directory from AFS> | 19:37 |
| krotscheck | Wheel things still need to be figured out re: jobs etc | 19:38 |
| krotscheck | But that's encouraging! | 19:38 |
| jeblair | krotscheck: cool :) | 19:38 |
| jeblair | mordred: i can't diff your rebase+changes :( | 19:38 |
| mordred | jeblair: I added 3 lines to the bottom | 19:39 |
| jeblair | mordred: i know, i want to see them but it's going to take me a while | 19:39 |
| mordred | jeblair: scuse me - 5 lines - the last 5 lines above if __name__ == '__main__' | 19:39 |
| mordred | jeblair: I did not have to fix anything in the rebase - it was just based on an old version of a parent patch | 19:40 |
| *** Sukhdev has quit IRC | 19:40 | |
| jeblair | mordred: yeah, i totally understand that. but because you rebased and changed at the same time, it's hard for me to diff. | 19:40 |
| openstackgerrit | Merged openstack-infra/infra-specs: Move docs.openstack.org/releases to releases.openstack.org https://review.openstack.org/266506 | 19:40 |
| AJaeger | I see pip 8.0.1 on https://pypi.python.org/pypi/pip | 19:41 |
| mordred | jeblair: totally | 19:41 |
| mordred | jeblair: and sorry | 19:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add a shade launch-node https://review.openstack.org/234975 | 19:41 |
| jeblair | mordred: it's okay. i fixed it. | 19:41 |
| mordred | woot | 19:41 |
| fungi | AJaeger: there aren't any votes against 262545. there are a few recent suggestions which don't seem to significantly alter the plan and so could be addressed in a separate change updating that spec. do you want to acknowledge them before i approve? | 19:41 |
| *** yamahata has joined #openstack-infra | 19:41 | |
| AJaeger | fungi, acknowledge = saying yes, I'll update spec as needed? | 19:42 |
| jpr | testing out image build for dpc via nodepool according to http://docs.openstack.org/infra/openstackci/third_party_ci.html#start-nodepool but nothing appears to be happening. | 19:42 |
| jeblair | mordred: what workflow caused you to end up rebasing that? | 19:42 |
| jpr | rerunning nodepool with debug reports it's connecting to zuul's gearman and the build is in process | 19:42 |
| mordred | jeblair: the patch was in merge conflict on gerrit | 19:42 |
| *** knikolla has joined #openstack-infra | 19:42 | |
| *** daemontool has joined #openstack-infra | 19:42 | |
| *** salv-orlando has quit IRC | 19:42 | |
| mordred | jeblair: so I pulled it down and did a rebase - which showed that it was depending on an old version of a patch that had landed, so I removed the old version of that patch in the rebase screen | 19:43 |
| jpr | zuul is configured to run gearman but there doesn't seem to be anything happening that is build orientied, no evidence in the logs or in the dib cache | 19:43 |
| mordred | jeblair: then I said to myself "oh, I also added inventory cache invalidation to the non-shade one- I should do that" - and did it before pushing up just the rebased version | 19:43 |
| anteaya | pip 8.0.1 appears to require virtualenv >=1.10 | 19:43 |
| fungi | AJaeger: sure, just whatever replies you wanted to leave to those comments (yes i'll update the spec later with this, no this is a bad idea for whatever reason, et cetera) | 19:44 |
| jpr | any hints on where to look? | 19:44 |
| AJaeger | fungi: on it ;) | 19:44 |
| *** mrmartin has quit IRC | 19:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add a shade launch-node https://review.openstack.org/234975 | 19:45 |
| jeblair | mordred: okay, there's your change and then a rebase as separates | 19:45 |
| *** bardia has joined #openstack-infra | 19:45 | |
| mordred | cool | 19:45 |
| jeblair | i understand now :) | 19:45 |
| *** Eva-i has quit IRC | 19:46 | |
| *** Sukhdev has joined #openstack-infra | 19:46 | |
| jeblair | mordred: you want to +A for form's sake? | 19:47 |
| *** bardia has quit IRC | 19:47 | |
| *** bardia has joined #openstack-infra | 19:47 | |
| *** Ramanjaneya has quit IRC | 19:49 | |
| lifeless | pip 8.0.1 is out | 19:49 |
| lifeless | virtualenv will be out soon | 19:49 |
| lifeless | this should fix everything for us | 19:49 |
| jeblair | it will not fix everything for us | 19:49 |
| *** ashleighfarnham has quit IRC | 19:49 | |
| jeblair | unless they completely reverted the uninstall thing | 19:49 |
| *** e0ne has quit IRC | 19:49 | |
| krotscheck | jeblair: Shifting to daycare mode. Will be available when naps happen. | 19:49 |
| lifeless | jeblair: its been reverted back to a warning | 19:49 |
| anteaya | krotscheck: happy naps | 19:49 |
| jeblair | krotscheck: cool, i'll keep pushing things | 19:49 |
| jeblair | lifeless: okay, then it will fix everything for us :) | 19:49 |
| dstufft | If any of y'all are distro people y'all should read https://lists.debian.org/debian-python/2016/01/msg00073.html and help me make it happen (both on Debian and on others) | 19:50 |
| SpamapS | lifeless: \o/ | 19:50 |
| *** ashleighfarnham has joined #openstack-infra | 19:50 | |
| lifeless | afk for a few minutes, C-> kindy | 19:50 |
| *** Nikolay_St has joined #openstack-infra | 19:50 | |
| AJaeger | fungi, commented | 19:51 |
| AJaeger | dstufft: thanks, for the release | 19:52 |
| dstufft | you probably won't be completely fixed until virtualenv 14.0.1 | 19:52 |
| dstufft | I forgot to actually default the new auto update stuff to True | 19:53 |
| dstufft | so 14.0.1 doesn't auto update :/ | 19:53 |
| dstufft | er 14.0.0 doesn't | 19:53 |
| anteaya | dstufft: thanks for keeping us informed :) | 19:54 |
| openstackgerrit | Ian Cordasco proposed openstack-dev/hacking: Update flake8 requirement for 0.11.0 https://review.openstack.org/270544 | 19:54 |
| mordred | dstufft: good email | 19:55 |
| *** ayoung has quit IRC | 19:56 | |
| jeblair | dstufft, mordred: just for my own curiousity -- why is it that when you pip -U over a system-installed package, it needs to actually remove files from the system one at all? doesn't sys.path precedence come into play? | 19:56 |
| fungi | i doubt we can be sure what's fixed for us for some time now that we're not using latest pip in many (most?) places | 19:57 |
| *** SumitNaiksatam has joined #openstack-infra | 19:57 | |
| dstufft | the answer is a bit complicated | 19:57 |
| AJaeger | is puppet currently disabled on running on jenkins? Last run was more than 2 hours ago | 19:57 |
| dstufft | Python itself doesn't have any concept of a package that is an "OS" package | 19:57 |
| dstufft | Debuntu hack this into it, other distros don't | 19:57 |
| jeblair | this is probably why i'm thinking that way | 19:58 |
| *** vgridnev has quit IRC | 19:58 | |
| dstufft | on like, Fedora, pip and the OS will be installing into exactly the same folder | 19:58 |
| fungi | yeah, the dist-packages thing is a debian hacl | 19:58 |
| fungi | hack | 19:58 |
| *** amotoki has quit IRC | 19:58 | |
| mordred | well... dist-packages is even crazier | 19:58 |
| *** amotoki has joined #openstack-infra | 19:58 | |
| jeblair | okay, so that's why this kinda sorta works on debian but not elsewhere | 19:58 |
| fungi | coupled with installing into /usr/local instead of /usr | 19:58 |
| mordred | becuase it's not that some things go into site and some into dist | 19:59 |
| zaro | anteaya: RE: https://review.openstack.org/#/c/270628/3//COMMIT_MSG@10 | 19:59 |
| openstackgerrit | Merged openstack-infra/project-config: Pin pip to <8 for openstackci-beaker jobs https://review.openstack.org/270995 | 19:59 |
| dstufft | dist-packages is just so Debian can have OS packages in /usr but pip packages in /usr/local | 19:59 |
| mordred | nope - | 19:59 |
| mordred | eveyrhitng goes into dist | 19:59 |
| dstufft | without making it impossible for someone to install their own Python in /usr/local | 19:59 |
| mordred | they could have done ... | 19:59 |
| *** jcoufal has quit IRC | 19:59 | |
| mordred | oh. jeez | 19:59 |
| dstufft | (whch will use /usr/local/../site-packages/ | 19:59 |
| zaro | anteaya: i don't think that's an accurate statement | 19:59 |
| mordred | dstufft: that's even worse than I thought | 19:59 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 19:59 |
| anteaya | zaro: well that was what fungi said was the cause | 19:59 |
| zaro | anteaya: yeah, i don't think it's accurate | 19:59 |
| anteaya | zaro: the contents of the patch isn't different from any other acl patch | 20:00 |
| fungi | anteaya: zaro: what did i say? | 20:00 |
| dstufft | I plan to write a PEP that is something like the Debian thing | 20:00 |
| anteaya | zaro: it was the timing | 20:00 |
| dstufft | except with some edges shorn off | 20:00 |
| zaro | it has 'Registered Users' in it. | 20:00 |
| anteaya | fungi: 270628 | 20:00 |
| dstufft | I just haven't gotten around to it yet :/ | 20:00 |
| dstufft | ETIME | 20:00 |
| *** Nikolay_St has quit IRC | 20:00 | |
| zaro | fungi: https://review.openstack.org/#/c/270628/3//COMMIT_MSG@10 | 20:00 |
| jeblair | dstufft, mordred: thanks | 20:00 |
| *** silos has joined #openstack-infra | 20:01 | |
| *** daemontool has quit IRC | 20:01 | |
| anteaya | zaro fungi http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-19.log.html#t2016-01-19T21:47:51 | 20:01 |
| anteaya | zaro: my patch didn't trigger the creation of that group | 20:01 |
| anteaya | zaro: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-19.log.html#t2016-01-19T21:47:08 | 20:02 |
| *** EricGonczer_ has quit IRC | 20:02 | |
| zaro | anteaya: i could be wrong about it being your patch but maybe another one that had 'Registered Users' in it would also cause it | 20:02 |
| dstufft | jeblair: long term, it's exactly my intention that we rely on sys.path shadowing instead of trying to uninstall stuff for OS provided packages like that (because we can't really reliably remove them anyways, apt-get upgrade can bring them back and such) | 20:02 |
| *** daemontool has joined #openstack-infra | 20:02 | |
| anteaya | zaro: did you read the scrollback I linked? | 20:02 |
| anteaya | zaro: fungi found the trigger and group creation timestamp | 20:02 |
| zaro | reading again | 20:02 |
| anteaya | zaro: thanks | 20:02 |
| fungi | anteaya: zaro: what i said was that the first change to acls after we upgraded gerrit caused manage-projects to run. the sequence number on the new registered users group indicates that (based on which changes introduced the group numbers immediately before and after it) | 20:03 |
| dstufft | but we need a way to say "this package is provided by the OS"... whcih is .egg-info/INSTALLER (in my plan in my head) which we can't rely on yet nobody uses it D: | 20:03 |
| *** austin81 has joined #openstack-infra | 20:03 | |
| jeblair | dstufft: gotcha | 20:03 |
| fungi | anteaya: zaro: when it ran, it tried to create a registered users group, and for the first time ever gerrit allowed it to due to the bug described in gerrit issue 3831 | 20:04 |
| *** fesp has quit IRC | 20:04 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: make announce.sh more robust for first releases https://review.openstack.org/271011 | 20:05 |
| *** bcourt has quit IRC | 20:05 | |
| jpr | asselin: any thoughts on what would cause `nodepool image-build dpc` to fail in the stock third-party-ci setup? | 20:05 |
| zaro | fungi, anteaya : ahh i see. | 20:06 |
| *** EricGonczer_ has joined #openstack-infra | 20:06 | |
| anteaya | zaro: thanks for asking, sorry I wasn't clear before | 20:06 |
| anteaya | zaro: so the gerrit bug is right | 20:07 |
| zaro | anteaya: so your right your change wasn't the culprit. There are already lots of acls files that contain 'Registered Users' in them and that's the cause. | 20:07 |
| anteaya | zaro: no it didn't even need to contain Registered Users | 20:07 |
| zaro | existing acls files | 20:07 |
| anteaya | it just needed to touch acl files | 20:07 |
| anteaya | fungi believes this acl was the trigger: https://review.openstack.org/#/c/259373/4/gerrit/acls/openstack/sahara-scenario.config | 20:08 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: skip most tests on a locale import https://review.openstack.org/271014 | 20:08 |
| AJaeger | dougwig: could you review the above, please ^ | 20:08 |
| fungi | anteaya: zaro: in fact the broken openstack-dev/sandbox approval permissions which prompted us to tro and add that change in the first place turns out to be because the registered users group it was already using (since the time we upgraded to 2.11) was not the system registered users group | 20:08 |
| anteaya | but the bug is correct in that we should never be able to create a user group named the same as a system group | 20:08 |
| *** aysyd has quit IRC | 20:09 | |
| dougwig | AJaeger looking | 20:09 |
| yamahata | hello. Can you please take care of https://review.openstack.org/#/c/265437/ ? | 20:09 |
| *** elo has quit IRC | 20:09 | |
| fungi | zaro: anteaya: though alternatively, jeepyb should probably also not try to create groups which already exist, and should check for system groups too not just regular groups | 20:09 |
| anteaya | fungi: true | 20:09 |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: include keystonev2 role assignments https://review.openstack.org/270993 | 20:09 |
| AJaeger | yamahata: I'll take care of it | 20:10 |
| zaro | yes, that's what the jeepyb change is about | 20:10 |
| *** sdake_ has joined #openstack-infra | 20:10 | |
| fungi | ahh, cool, i was just looking at the comment not the rest of the commit | 20:10 |
| fungi | spread thin at the moment | 20:10 |
| anteaya | right | 20:10 |
| yamahata | AJaeger: thank you so much. I've been waiting it for a while. | 20:10 |
| zaro | i added additional comment to the last PS. | 20:11 |
| anteaya | are we taking a break or still focusing on mirrors? | 20:11 |
| *** esker has quit IRC | 20:11 | |
| zaro | was the commit msg not clear? | 20:11 |
| *** silos has quit IRC | 20:11 | |
| *** markvoelker has quit IRC | 20:11 | |
| *** rfolco has quit IRC | 20:11 | |
| *** gildub has joined #openstack-infra | 20:11 | |
| fungi | anteaya: mirrors and pip-related issues are interleaved at the moment since to some extent the one was preventing the other | 20:11 |
| anteaya | okay | 20:12 |
| anteaya | we were just making such wonderful progress for a while | 20:12 |
| fungi | but also i'm just trying to catch up on administrivia i'm behind on before i disappear all next week | 20:12 |
| anteaya | fungi: good call | 20:12 |
| *** sdake has quit IRC | 20:12 | |
| anteaya | zaro: I'll look again after you fix that commit message | 20:13 |
| *** cschwede has quit IRC | 20:14 | |
| zaro | fungi, anteaya: not to beat a dead horse. the reason the 'Registered Users' group got created was because there's existing acls files in project-config that contains 'Registered Users'. Jeepyb created that group on first run of managed_projects after gerrit 2.11 upgrade. | 20:14 |
| zaro | i will try add that info to commit msg | 20:14 |
| fungi | zaro: that's exactly what i said | 20:14 |
| anteaya | zaro: ah there it is | 20:14 |
| anteaya | zaro: thanks | 20:15 |
| *** daemontool_ has joined #openstack-infra | 20:15 | |
| *** ldnunes_ has quit IRC | 20:15 | |
| fungi | note that puppet only runs manage-projects when certain files change (via subscribe/notify) so it was technically triggered by a change merging soon after the upgrade | 20:15 |
| zaro | fungi: yes, not sure if anteaya was clear on that. | 20:15 |
| anteaya | I wasn't, i had missed the part where a preexisting acl file created Registered Users | 20:16 |
| anteaya | that was my failing | 20:16 |
| fungi | anteaya: recall that it was already in use in the openstack-dev/sandbox acl. that was what led you to start troubleshooting the mismatch in our documentation about that repo in the first place | 20:17 |
| zaro | i also actually didn't realized until a few mins ago :) | 20:17 |
| *** daemontool has quit IRC | 20:17 | |
| anteaya | fungi: yes I remember, you subsequently pointed out that the created group had a link to in the sandbox access gitweb permissions | 20:18 |
| fungi | but it also appears in dozens of other acl files for other repos too | 20:18 |
| *** cschwede has joined #openstack-infra | 20:18 | |
| *** ybathia has quit IRC | 20:18 | |
| anteaya | but in any case I believe I am clear now | 20:18 |
| anteaya | thank you for helping me to understand this better | 20:18 |
| AJaeger | zaro, fungi: So, what exactly is wrong in the ACL file? | 20:18 |
| fungi | AJaeger: nothing at all | 20:18 |
| anteaya | AJaeger: nothing | 20:18 |
| openstackgerrit | Merged openstack-infra/infra-specs: Make translation set up consistent https://review.openstack.org/262545 | 20:19 |
| anteaya | AJaeger: gerrit allows you to create user groups in the same name as system groups via ssh in 2.11 | 20:19 |
| *** bcourt has joined #openstack-infra | 20:19 | |
| AJaeger | Ah.... | 20:19 |
| anteaya | but not via the web ui or REST | 20:19 |
| anteaya | zaro has filed a bug | 20:19 |
| anteaya | and patches to work around the issue in jeepyb | 20:20 |
| fungi | AJaeger: the issue is that if an acl refers to a system group right now jeepyb tries to create it and, in 2.11, gerrit allows it to succeed (after which point those acls refer to the new empty regular group of the same name rather than the system group) | 20:20 |
| AJaeger | fungi, got it now, thanks | 20:20 |
| zaro | now on to fixing the root cause.. | 20:20 |
| anteaya | zaro: thank you :) | 20:20 |
| *** amotoki has quit IRC | 20:21 | |
| AJaeger | I now get an "internal server error" on http://puppetboard.openstack.org/ | 20:22 |
| zaro | fungi: another quick fix for sandbox is to just change group file entry for Registered Users to 'global:Registered-Users ' | 20:23 |
| openstackgerrit | Merged openstack-infra/project-config: networking-odl: enable test with ODL release of beryllium-snapshot https://review.openstack.org/265437 | 20:23 |
| *** dimtruck is now known as zz_dimtruck | 20:24 | |
| AJaeger | sdague: I followed your nova patch, what do you think of https://review.openstack.org/#/c/271014 ? | 20:24 |
| fungi | zaro: yeah, but as mentioned this is theoretically impacting lots of other repos too so we'll need to clean it up in a lot of places | 20:24 |
| fungi | not just the sandbox repo | 20:24 |
| zaro | ohh right. | 20:25 |
| AJaeger | fungi: only 92 places ;) | 20:25 |
| *** zhurong has joined #openstack-infra | 20:25 | |
| zaro | fungi: i can test those patches on review-dev but will need to take if off puppet control. | 20:26 |
| zaro | were you still wanting to keep it puppted? | 20:26 |
| *** Nikolay_St has joined #openstack-infra | 20:26 | |
| AJaeger | ah, puppetboard is back | 20:26 |
| *** vgridnev has joined #openstack-infra | 20:26 | |
| *** dtardivel has quit IRC | 20:27 | |
| fungi | zaro: that was mostly just important while we were trying to get kernel updates rolled out for the security issue earlier this week. we can disable puppet on it again but note that the mechanism for doing that has now changed: http://docs.openstack.org/infra/system-config/sysadmin.html#disable-enable-puppet | 20:27 |
| *** pavel_bondar_ has joined #openstack-infra | 20:27 | |
| *** flwang1 has joined #openstack-infra | 20:28 | |
| zaro | cool. thanks | 20:29 |
| tonytan4ever | Hey folks, is pip related issue fixed ? | 20:29 |
| tonytan4ever | Or do we still need to change jenkins jobs to fix that ? | 20:30 |
| *** krtaylor has quit IRC | 20:31 | |
| *** piet has quit IRC | 20:31 | |
| *** markvoelker has joined #openstack-infra | 20:31 | |
| clarkb | tonytan4ever: fix is in progress | 20:31 |
| tonytan4ever | rgr. Thanks | 20:31 |
| anteaya | I believe we need both pip 8.0.1 and virtualenv 14.0.1 to be fixed, we are waiting on virtualenv 14.0.1 | 20:32 |
| clarkb | justvwaiting on mew venv | 20:32 |
| clarkb | yup | 20:32 |
| clarkb | then we can rebuild images | 20:32 |
| tonytan4ever | K | 20:32 |
| *** sabeen3 has quit IRC | 20:32 | |
| tonytan4ever | Thanks folks. | 20:32 |
| pavel_bondar_ | tonytan4ever: what is the pip issue? | 20:32 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: make announce.sh more robust for first releases https://review.openstack.org/271011 | 20:33 |
| *** silos has joined #openstack-infra | 20:33 | |
| *** aeng has joined #openstack-infra | 20:33 | |
| dstufft | waiting on travis :\ | 20:33 |
| tonytan4ever | pavel_bondar_: So all the new change cannot be built because of a pip issue. | 20:33 |
| asselin | jpr, I don't know, but seems there are a lot of recent changes to nodepool going on. | 20:33 |
| anteaya | dstufft: :\ | 20:33 |
| dstufft | Also, if y'all care about being able to isntall things on Linux without a compiler via pip, there's a distutils-sig thread about it | 20:33 |
| *** sabeen has joined #openstack-infra | 20:33 | |
| *** kzaitsev_mb has quit IRC | 20:33 | |
| asselin | jpr, I suggest you use an older version of nodepool: https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/single_node_ci_data.yaml#n38 | 20:34 |
| dstufft | it talks about ABI and other crap that I'm woefully unprepaired to fully grasp | 20:34 |
| dstufft | but might be interesting to some of you | 20:34 |
| jpr | asselin: thanks for the heads up. I've debugged a bit | 20:34 |
| *** ashleighfarnham has quit IRC | 20:34 | |
| *** otter768 has joined #openstack-infra | 20:34 | |
| *** hashar has joined #openstack-infra | 20:34 | |
| asselin | jpr, oh, ok, even better. | 20:34 |
| *** zz_dimtruck is now known as dimtruck | 20:35 | |
| jpr | asselin: no promises to figure it out, but good to know things might have changed a bit | 20:35 |
| asselin | jpr, I suggest the 0.2.0 tag https://git.openstack.org/cgit/openstack-infra/nodepool | 20:35 |
| asselin | in case you need to go back | 20:35 |
| jpr | asselin: so just replace "master" with the preferred release tag? | 20:35 |
| *** silos has left #openstack-infra | 20:35 | |
| pavel_bondar_ | tonytan4ever: I have an issue that new version of package I maintain can not be pulled down by openstack CI, so I am curious if it is the same issue | 20:36 |
| pavel_bondar_ | tonytan4ever: is there launchpad ticket for that so I could read more about it? | 20:36 |
| asselin | jpr, yes | 20:36 |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Bump upper-constraints on oslo.db to 4.3.1 https://review.openstack.org/271030 | 20:37 |
| mriedem | dims: double yay ^ | 20:37 |
| fungi | dstufft: yeah, i've been following that one. we're actually working on solving it ourselves at the moment by centrally building wheels of everything for each platform on which we test and then propping up mirrors of that in ways pip can find automatically in each of our service providers | 20:37 |
| fungi | well, wheels of everything _we_use_ that is | 20:37 |
| fungi | not wheels of everything on pypi of course. that would be both insane and impossible | 20:37 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Make the rally job experimental https://review.openstack.org/183836 | 20:38 |
| tonytan4ever | pavel_bondar_: Possibly it is related. You can look at the logs | 20:38 |
| tonytan4ever | I am not aware there is a launchpad ticket related to it. | 20:38 |
| *** otter768 has quit IRC | 20:39 | |
| pavel_bondar_ | tonytan4ever: thanks! | 20:39 |
| dims | mriedem :) | 20:39 |
| jpr | asselin: also, will puppet take me to a different revision if I change that field and puppet apply cleanly or should I rebuild from a pristine state | 20:39 |
| jpr | asselin: i'll give the revision change a shot and see if it works to isolate the issue | 20:39 |
| *** Nikolay_St has quit IRC | 20:40 | |
| asselin | jpr, puppet should take you there | 20:40 |
| *** ashleighfarnham has joined #openstack-infra | 20:40 | |
| *** _nadya_ has quit IRC | 20:40 | |
| *** sridhar_ram has quit IRC | 20:41 | |
| *** tiswanso has quit IRC | 20:42 | |
| openstackgerrit | Khai Do proposed openstack-infra/jeepyb: Fix handling of Gerrit system groups in acls files https://review.openstack.org/270628 | 20:42 |
| zaro | anteaya: ^ hope that's more clear | 20:42 |
| asselin | jpr, but puppet doesn't restart nodepool (service), and since this is a downgrade, any database changes might need to be cleaned up.... | 20:43 |
| *** kzaitsev_mb has joined #openstack-infra | 20:43 | |
| jpr | asselin: so the problem appears to be in zuul | 20:43 |
| fungi | tonytan4ever: pavel_bondar_: https://github.com/pypa/pip/issues/3384 | 20:43 |
| anteaya | zaro: a piece of art, my friend | 20:43 |
| *** mhickey has joined #openstack-infra | 20:43 | |
| tonytan4ever | fungi: Cool thanks. | 20:43 |
| anteaya | zaro: that commit message needs a frame around it | 20:43 |
| asselin | jpr, not sure what zuul has to do with nodepool image builds... | 20:44 |
| pavel_bondar_ | fungi: thanks! | 20:44 |
| *** ZZelle_ has joined #openstack-infra | 20:44 | |
| jpr | asselin: it looks like nodepool is contacting zuul->gearman | 20:45 |
| fungi | asselin: in the past week, with the addition of image build workers, nodepool now uses gearman (in our case zuul's gearman) | 20:45 |
| zaro | anteaya: haha, english is not my forte so TY :) | 20:45 |
| *** matrohon has joined #openstack-infra | 20:45 | |
| jpr | fungi: or tries to ;) | 20:45 |
| *** Sukhdev has quit IRC | 20:46 | |
| fungi | jpr: yes, there are apparently still some rough edges we're working through encountered in production which did not surface in our tests | 20:46 |
| fungi | clarkb and greghaynes have been running those down | 20:47 |
| *** jamielennox|away is now known as jamielennox | 20:47 | |
| *** erikwilson has quit IRC | 20:47 | |
| *** baoli has quit IRC | 20:47 | |
| *** Daisy has joined #openstack-infra | 20:47 | |
| clarkb | it should work now as long as your gear connection stsys up | 20:47 |
| clarkb | we had two successful builds and uploafs | 20:47 |
| lifeless | jeblair: dstufft: also the debuntu hack doesn't provide a place for pip to put its new files; it just changes the name on disk of the one concept in the python sysconfig | 20:47 |
| lifeless | so its pretty much useless from that angle | 20:48 |
| anteaya | zaro: you did a wonderful job there | 20:48 |
| *** mhickey has quit IRC | 20:48 | |
| *** baoli has joined #openstack-infra | 20:49 | |
| lbragstad | o/ does anyone know what the replacement is for hound.openstack.org? | 20:50 |
| *** Sukhdev has joined #openstack-infra | 20:50 | |
| anteaya | codesearch.o.o | 20:51 |
| lbragstad | anteaya awesome, thanks! | 20:51 |
| anteaya | welcome | 20:51 |
| anteaya | that answer is only for use if you are wearing a hat | 20:51 |
| *** Daisy has quit IRC | 20:51 | |
| anteaya | lbragstad: ^^ | 20:51 |
| anteaya | lbragstad: you are of course, wearing a hat, are you not? | 20:51 |
| lbragstad | anteaya I forgot what it's like to not wear one... | 20:52 |
| anteaya | lbragstad: :) | 20:52 |
| anteaya | lbragstad: your barber must love you | 20:52 |
| *** yamahata has quit IRC | 20:52 | |
| *** daemontool_ has quit IRC | 20:52 | |
| *** daemontool_ has joined #openstack-infra | 20:53 | |
| asselin | fungi, clarkb but if they use nodepool 0.2.0 they should be fine right? | 20:55 |
| mordred | lifeless: ++ | 20:55 |
| *** rajinir has quit IRC | 20:55 | |
| clarkb | asselin yes I mean you should be finr with master too | 20:55 |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 20:55 |
| *** jpr has quit IRC | 20:55 | |
| *** piet has joined #openstack-infra | 20:55 | |
| asselin | jpr, ^^ ok so master should work. what issue did you run into? | 20:55 |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: include keystonev2 role assignments https://review.openstack.org/270993 | 20:55 |
| greghaynes | clarkb: asselin Is the issue that you no longer can nodepool image-build without a nodepoold running? | 20:56 |
| lifeless | (the intent was to do more... but intent and execution often differ) | 20:56 |
| asselin | greghaynes, oh, is that true? | 20:56 |
| greghaynes | asselin: yes | 20:56 |
| mordred | fungi: we have a lot of build nodes- I bet if we used them during the evening when the queue has otherwise dropped off, we could eventually build wheels of all of pypi | 20:56 |
| *** zhurong has quit IRC | 20:56 | |
| lifeless | mordred: please dog no | 20:56 |
| greghaynes | asselin: you now need to either have a nodepool-builder running or a nodepoold | 20:56 |
| mordred | lifeless: I mean, I do not think it's a good idea | 20:57 |
| mordred | lifeless: but fungi said "impossible" | 20:57 |
| fungi | mordred: many, many, many packages on pypi are simply not wheelable (or even pip-installable) | 20:57 |
| lifeless | mordred: enough things will build bad metadata as wheels, we'd be fiit-gunning ourselves | 20:57 |
| *** gyee has joined #openstack-infra | 20:57 | |
| lifeless | mordred: its likely in fact that the wheel mirror we're bringing up now is going to generate bad metadata and need fixing | 20:57 |
| asselin | greghaynes, I see...yeah docs recommend running a build first before starting nodepoold: https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/doc/source/third_party_ci.rst#n336 | 20:57 |
| fungi | mordred: we avoid depending on those for obvious reasons, but they are numerous | 20:57 |
| lifeless | mordred: though we won't know that till it bits | 20:58 |
| lifeless | bites | 20:58 |
| mordred | lifeless: oh? I thought our current installs went through wheel first, no? | 20:58 |
| greghaynes | They do | 20:58 |
| *** ashleighfarnham has quit IRC | 20:58 | |
| greghaynes | or at least they did | 20:58 |
| lifeless | mordred: yes, a wheel cache that is local | 20:58 |
| *** pavel_bondar_ has quit IRC | 20:58 | |
| lifeless | mordred: and not shared | 20:58 |
| mordred | lifeless: what bad thing are you talking about so I can prepare myself for the carnage? | 20:58 |
| *** ashleighfarnham has joined #openstack-infra | 20:58 | |
| lifeless | mordred: this mitigates the impact of e.g. conditional dependencies expressed through turing complete logic rather than markers | 20:58 |
| greghaynes | lifeless: there was some hard coded stuff in devstack to build wheels for all the packages we would install before pip had the auto-caching bits | 20:59 |
| greghaynes | but same idea | 20:59 |
| fungi | mordred: lifeless: so not something which is going to bite us if we share they across a homogenous set of systems being booted from the same images | 20:59 |
| lifeless | greghaynes: yes, I don't know if we've even pulled that out, but its still local not shared | 20:59 |
| fungi | er, share them | 20:59 |
| mordred | I'm not sure what the shared part breaks? | 20:59 |
| mordred | lifeless: could you splain? | 20:59 |
| lifeless | fungi: if the machines that can see the wheel files are the same python version and same OS, we should be fine | 20:59 |
| mordred | they are | 21:00 |
| mordred | we are not building _a_ wheel cache | 21:00 |
| lifeless | fungi: (though its still possible to have breakage) | 21:00 |
| mordred | we are building multiple wheel cache mirrors | 21:00 |
| greghaynes | Yep, the entire point is that we control what nodes are using our wheel mirror | 21:00 |
| *** dslev has quit IRC | 21:00 | |
| greghaynes | so we know os/packages | 21:00 |
| mordred | essentially one for each OS/python combination we have | 21:00 |
| lifeless | so, there are two broad failures | 21:00 |
| *** gordc has quit IRC | 21:00 | |
| mordred | and we will point the pip.conf at the matching mirror | 21:00 |
| mordred | we will not expect pip to sort it out from amongst a mixed bad | 21:00 |
| mordred | bag | 21:01 |
| lifeless | a) some setup.py code that inspects the environment to determine dependencies/features built into the wheel, but doesn't export that. | 21:01 |
| mordred | nod | 21:01 |
| lifeless | so e.g. if some library (might be C, might be python) is available, build an accelerator, if its not, don't. | 21:01 |
| dstufft | Ugh | 21:01 |
| dstufft | Ther eis probably going to be a 8.0.2 | 21:01 |
| dstufft | because CentOS. | 21:01 |
| fungi | the biggest risk i see is we cache numpy and then get updated diskimages with a newer library than it linked which breaks abi in some unfortunate way | 21:01 |
| mordred | dstufft: \o/ | 21:01 |
| dstufft | (alternatively because requests, not sure exactly who is to blame yet) | 21:01 |
| greghaynes | fungi: eeek | 21:01 |
| dstufft | someone is though | 21:01 |
| lifeless | dstufft: its very unlikely that libblas etc are going to do that; I'm not concerned | 21:01 |
| mordred | fungi: if we get updated numpy ABI breakages in trusty I'm going to kill someone | 21:02 |
| lifeless | bah | 21:02 |
| lifeless | fungi: ^ | 21:02 |
| dstufft | https://github.com/pypa/pip/issues/3415#issuecomment-173704252 | 21:02 |
| greghaynes | Ya, but same idea for pyzmq or somesuch | 21:02 |
| mordred | yah | 21:02 |
| mordred | I mean, we do have to have ALL of the -dev packages installed on the builders | 21:02 |
| lifeless | b) the same kind of thing, but the resulting wheel is actually specifically relevant to something in that environment which the wheel tags on the built wheel do not disclose | 21:03 |
| clarkb | also we did this for a long time with no issues | 21:03 |
| mordred | dstufft: that's amazing | 21:03 |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/project-config: Reduce internap max servers from 72 to 64 https://review.openstack.org/271037 | 21:03 |
| fungi | also, just to confirm, when we're building wheels we rebuild the entire set every time there's a change, right? because otherwise things can get funky | 21:03 |
| *** Sukhdev has quit IRC | 21:03 | |
| lifeless | for instance, - and I know this one isn't going to hit us, but its an example - if you build some packages on 2.6, the wheel depends on argparse, if you build the same pacakge on 2.7, it doesn't. | 21:03 |
| lifeless | but the wheel file claims compat with 2.6 and 2.7 in both cases | 21:03 |
| lifeless | or even claims universal support | 21:03 |
| greghaynes | fungi: yes | 21:04 |
| *** amrith is now known as _amrith_ | 21:04 | |
| lifeless | clarkb: actually, we had issues, we worked around them by blacklisting a couple of packages from memory. And we certainly built bad wheels ourselves of our packages because of requirements-pyN.txt | 21:04 |
| lifeless | clarkb: if the mirror url is per-OS+per-python-version though, we're pretty safe | 21:04 |
| clarkb | uh noy with the mirror | 21:05 |
| fungi | greghaynes: perfect | 21:05 |
| clarkb | we had trouble in devstack | 21:05 |
| dstufft | pip's internal wheel cache forces all wheels to be interpreter specific | 21:05 |
| dstufft | FWIW | 21:05 |
| clarkb | completely different | 21:05 |
| dstufft | because of what lifeless said | 21:05 |
| lifeless | dstufft: did we land that? I thought the issue was still open ? | 21:05 |
| dstufft | mordred: I'm at least 95% sure distros exist just to screw with me | 21:05 |
| dstufft | lifeless: I landed it in pip 8 yea | 21:05 |
| clarkb | the mirror build just ignored failrd builds | 21:05 |
| lifeless | dstufft: cool | 21:05 |
| clarkb | those files didnt end up getting hosted | 21:05 |
| clarkb | devstacks wheel builder however didnt do that | 21:06 |
| lifeless | clarkb: failed builds are fine; I'm talking corrupt builds | 21:06 |
| clarkb | and failed on things | 21:06 |
| *** yamahata has joined #openstack-infra | 21:06 | |
| hashar | isn't pip building the wheels for you ? I myself just snapshot ~/.cache/pip/wheels and reinject it in the instances on startup | 21:06 |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 21:06 |
| AJaeger | is puppet working? There's not a run for over an hour on jenkins,e.g. http://puppetboard.openstack.org/node/jenkins01.openstack.org ? | 21:06 |
| *** bcourt has quit IRC | 21:06 | |
| *** erlon has quit IRC | 21:06 | |
| lifeless | clarkb: this may be all hypothetical, I'm only detailing the failure modes I've seen / know can happen because mordred asked me too | 21:07 |
| *** sabeen2 has joined #openstack-infra | 21:07 | |
| lifeless | clarkb: I'm not saying we *will* encounter them, I'm describing what they are so that if they are encountered its comprehensible | 21:07 |
| fungi | hashar: since we run lots of jobs for different projects with different sets of requirements, we need some process which can generate them all. also our dependencies (as defined by our upper constraints list) change more often than our images | 21:08 |
| mordred | AJaeger: cecking | 21:08 |
| mtreinish | pleia2, jeblair, fungi, clarkb, nibalizer: if you get a sec: https://review.openstack.org/#/c/234447/ should be pretty straightforward | 21:08 |
| anteaya | mtreinish: pleia2 is conferencing atm | 21:08 |
| anteaya | mtreinish: not sure where | 21:08 |
| AJaeger | anteaya: scale? | 21:08 |
| anteaya | AJaeger: ah yeah, that is probably it | 21:09 |
| fungi | hashar: and conveniently pip makes it possible for you to keep your wheels somewhere other than in a local directory | 21:09 |
| mtreinish | anteaya: I'll never remember and just continue to ping her :) | 21:09 |
| mtreinish | she doesn't have to respond | 21:09 |
| hashar | fungi: what I do is that on gate-and-submit I rsync the wheel cache to a central place and namespace the caches with ZUUL_PROJECT/ZUUL_BRANCH/JOB_NAME which seems good enough for me :D | 21:09 |
| mordred | openstackid update-site is not happy, btw | 21:10 |
| *** sabeen has quit IRC | 21:10 | |
| hashar | fungi: been too lazy to figure out how to collect all requirements and generate a cache out of it | 21:10 |
| anteaya | mtreinish: as you wish, just don't fell insulted if it takes her a while to reply | 21:10 |
| AJaeger | thanks for checking, mordred | 21:10 |
| mtreinish | AJaeger: that's a good guess: https://www.socallinuxexpo.org/scale/14x/presentations/building-career-ubuntu-and-foss | 21:10 |
| *** dimtruck is now known as zz_dimtruck | 21:11 | |
| *** salv-orlando has joined #openstack-infra | 21:11 | |
| fungi | hashar: we _may_ have some code you could reuse if you decide to try to | 21:12 |
| hashar | fungi: I went with my own rsync command and some magic jjb tricks :-D | 21:12 |
| *** matrohon has quit IRC | 21:13 | |
| openstackgerrit | Cedric Brandily proposed openstack/requirements: Bump debtcollector version https://review.openstack.org/271039 | 21:13 |
| *** nelsnels_ has quit IRC | 21:14 | |
| * AJaeger waves good-night | 21:15 | |
| anteaya | AJaeger: good night | 21:16 |
| *** matrohon has joined #openstack-infra | 21:16 | |
| anteaya | AJaeger: thank you | 21:17 |
| *** dslev has joined #openstack-infra | 21:19 | |
| *** nelsnelson has joined #openstack-infra | 21:22 | |
| *** matrohon has quit IRC | 21:23 | |
| *** esker has joined #openstack-infra | 21:24 | |
| *** baoli has quit IRC | 21:27 | |
| *** baoli has joined #openstack-infra | 21:29 | |
| *** zz_dimtruck is now known as dimtruck | 21:29 | |
| *** sriram has quit IRC | 21:31 | |
| craige | o/ | 21:34 |
| *** gordc has joined #openstack-infra | 21:34 | |
| *** kgiusti has left #openstack-infra | 21:35 | |
| *** baoli has quit IRC | 21:35 | |
| *** baoli has joined #openstack-infra | 21:36 | |
| *** julim has quit IRC | 21:36 | |
| *** ddieterly has quit IRC | 21:36 | |
| *** baoli has quit IRC | 21:38 | |
| *** baoli has joined #openstack-infra | 21:38 | |
| *** markvoelker has quit IRC | 21:39 | |
| *** markvoelker has joined #openstack-infra | 21:41 | |
| *** shardy has quit IRC | 21:43 | |
| *** nelsnelson has quit IRC | 21:46 | |
| bknudson | I swear I used to have instructions for how to see the tox -e cover output for a review... | 21:46 |
| dmsimard | dstufft: man, you must not have had much sleep since pip8 is out. Thanks for your work :) | 21:48 |
| * dmsimard waits for pip 8.0.2 | 21:48 | |
| jhesketh | Morning | 21:48 |
| *** ajmiller_ has quit IRC | 21:49 | |
| *** thorst has quit IRC | 21:49 | |
| *** thorst has joined #openstack-infra | 21:50 | |
| anteaya | morning jhesketh | 21:50 |
| *** knikolla has quit IRC | 21:51 | |
| bknudson | this must be the wrong system? https://jenkins.openstack.org/view/All/job/keystone-coverage/changes | 21:51 |
| *** ybathia has joined #openstack-infra | 21:51 | |
| *** ddieterly has joined #openstack-infra | 21:52 | |
| *** hashar has quit IRC | 21:52 | |
| jeblair | anteaya, mordred: where do i go to see puppet reports? | 21:53 |
| *** krtaylor has joined #openstack-infra | 21:53 | |
| jeblair | oh weird | 21:53 |
| jeblair | puppetboard appeared broken but now appears fixed | 21:54 |
| anteaya | jeblair: AJaeger had the same experience not long ago | 21:54 |
| anteaya | a few hours at most | 21:54 |
| jeblair | anteaya: you were saying something about jenkins not updating? | 21:54 |
| nibalizer | anteaya: AJaeger yes pleia2 is at scale | 21:54 |
| nibalizer | i have spotted her a couple times already | 21:54 |
| anteaya | nibalizer: nice | 21:54 |
| anteaya | jeblair: ummm, was I? | 21:55 |
| *** Swami has quit IRC | 21:55 | |
| jeblair | anteaya: maybe it was someone else... | 21:55 |
| *** Swami has joined #openstack-infra | 21:55 | |
| *** esker has quit IRC | 21:55 | |
| fungi | i think AJaeger mentioned it maybe, which was why he was looking at puppetboard... trying to figure out why a job update hadn't propagated | 21:55 |
| *** Nikolay_St has joined #openstack-infra | 21:56 | |
| anteaya | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-21.log.html#t2016-01-21T21:06:44 | 21:56 |
| jeblair | it was an 'a' person at least! :) | 21:56 |
| fungi | mordred: said he was taking a look to confirm what was making puppet updates stale | 21:56 |
| *** sdake has joined #openstack-infra | 21:56 | |
| *** dims_ has joined #openstack-infra | 21:56 | |
| anteaya | mordred concluded "openstackid update-site is not happy, btw" | 21:56 |
| fungi | and indeed, it looks like the jenkins masters have not reported for 2 hours now | 21:57 |
| anteaya | jeblair: those 'a' nicks | 21:57 |
| *** burgerk has quit IRC | 21:57 | |
| *** dims has quit IRC | 21:57 | |
| jeblair | puppet_run_all.log is 74M | 21:57 |
| jeblair | ah i see why it's slow in less -- it's not because it's huge, it's because it has huge lines. | 21:58 |
| jeblair | like, the complete output from puppet is on one line | 21:58 |
| fungi | looking at jenkins01, its most recent two puppet reports were 17:51 and 19:51 | 21:58 |
| jeblair | this makes it a bit hard to read | 21:58 |
| fungi | so it's going a while between updates there | 21:58 |
| *** sdake_ has quit IRC | 21:59 | |
| jeblair | this is the log i usually look at to find out why | 21:59 |
| jeblair | but i am unable to do so | 21:59 |
| *** dslev has quit IRC | 21:59 | |
| jeblair | honestly | 22:00 |
| jeblair | i would just like ansible to stop printing the puppet output | 22:00 |
| jeblair | if it's not going to do it in a usable way | 22:00 |
| jeblair | we can always go onto the host and get it from syslog if needed | 22:00 |
| jeblair | so does anyone know how to make that happen? | 22:01 |
| anteaya | I do not | 22:01 |
| jeblair | mordred, Shrews: ^ | 22:01 |
| jeblair | anyone else around here know ansible? | 22:01 |
| mgagne | not to be that guy but: puppet agent -t > /dev/null ? | 22:01 |
| mordred | hey | 22:01 |
| jeblair | mordred: hi! | 22:01 |
| mordred | jeblair: I just jumped on a call | 22:01 |
| mordred | jeblair: but I can help you fix in 30 minutes | 22:01 |
| fungi | mgagne: well, we're calling puppet apply these days, but point taken | 22:02 |
| Shrews | jeblair: i do not know | 22:02 |
| *** markvoelker has quit IRC | 22:04 | |
| fungi | looking in syslog on jenkins01, ansible does seem to be firing puppet apply at a regular interval, but it's not printing puppet apply stdout that i can tell | 22:04 |
| *** ari has quit IRC | 22:05 | |
| nibalizer | jeblair: whats the issue? | 22:05 |
| jeblair | oh, do we lose syslog output with puppet-apply? | 22:05 |
| *** darrenc is now known as darrenc_afk | 22:05 | |
| *** Sukhdev has joined #openstack-infra | 22:06 | |
| fungi | that's what i'm attempting to confirm now | 22:06 |
| jeblair | nibalizer: it looks like puppet-apply does not syslog, so the only output we have is via ansible; ansible is taking the *complete* puppet output and concatenating it on one line in our log file on puppetmaster. | 22:07 |
| nibalizer | jeblair: oh yea thats annoying | 22:07 |
| nibalizer | its a json blob too | 22:07 |
| fungi | yeah, the most we seem to get in local logs now is stuff like: | 22:07 |
| fungi | Jan 21 19:51:24 jenkins01 ansible-puppet: Invoked with puppetmaster=None show_diff=False facter_basename=ansible manifest=/opt/system-config/production/manifests/site.pp environment=None timeout=30m facts=None | 22:07 |
| jeblair | nibalizer: well, more than annoying, i literally have no view in to what's happening anymore | 22:07 |
| dmsimard | I've been working with an ansible callback to get helpful verbose output (i.e, not everything in one line) | 22:08 |
| fungi | would be nice if it could at least tee to a file on each system as well as back over the calling socket. i wonder if worst case we could just do that with a wrapper | 22:08 |
| jeblair | fungi: well, i'm reading ansible-puppet now to see if we can update it to behave that way | 22:09 |
| *** jaosorior_ has quit IRC | 22:09 | |
| clarkb | might be possible to instruct puppet apply to use a log file too | 22:09 |
| nibalizer | jeblair: puppetboard isn't a view? | 22:09 |
| fungi | or |logger or whatever | 22:09 |
| jeblair | nibalizer: nope | 22:09 |
| jeblair | nibalizer: puppetboard says it hasn't updated in 2 hours. why? | 22:09 |
| jeblair | no idea | 22:09 |
| nibalizer | which host? | 22:10 |
| fungi | nibalizer: puppetboard especially doesn't help track down why something isn't reporting to puppetdb | 22:10 |
| jeblair | nibalizer: any jenkins host | 22:10 |
| fungi | nibalizer: all jenkins masters | 22:10 |
| jeblair | nibalizer: also, puppetboard has always been a very limited view | 22:10 |
| jeblair | it does not include all actions or outputs | 22:10 |
| clarkb | --logdest | 22:10 |
| jeblair | nibalizer: *especially* interesting ones :) | 22:10 |
| clarkb | choose between syslog the console and a log file | 22:10 |
| clarkb | says the man page | 22:10 |
| jeblair | it looks like ansible has exit_json and fail_json methods that we pass stdout too | 22:10 |
| jeblair | i'm trying to see if we can just not do that to cause ansible to not log that | 22:11 |
| nibalizer | when you look here | 22:11 |
| nibalizer | http://puppetboard.openstack.org/node/jenkins03.openstack.org | 22:11 |
| jeblair | the stdout argument is not documented | 22:11 |
| nibalizer | both groups and files make no sense | 22:11 |
| nibalizer | so at the very least there is a big in our puppetboard submit pipeline | 22:12 |
| jeblair | fungi: something about groups happened earlier, is that related ^? | 22:12 |
| *** peristeri has quit IRC | 22:13 | |
| fungi | jeblair: it's possible something in https://review.openstack.org/270296 or https://review.openstack.org/270262 is related? | 22:13 |
| nibalizer | ansible jenkins --list-hosts seems correct in output | 22:13 |
| *** pradk has joined #openstack-infra | 22:13 | |
| jeblair | oh | 22:14 |
| *** dslev has joined #openstack-infra | 22:14 | |
| jeblair | i think the args to exit_json are just included in json | 22:14 |
| nibalizer | we only have four hosts disabled according to ansible disabled --list-hosts | 22:14 |
| jeblair | clarkb: what should --logdest be? | 22:14 |
| fungi | nibalizer: also `sudo ansible jenkins --list-hosts` lgtm | 22:15 |
| nibalizer | fungi: I think ansible grouping is working fine because of the testing we just did then | 22:15 |
| clarkb | jeblair: my reading of the man page is you can pass it 'syslog' 'console' or a file path | 22:16 |
| clarkb | jeblair: so depends on how you want it to log | 22:16 |
| jeblair | clarkb: "puppet apply --logdest syslog" ? | 22:16 |
| clarkb | a | 22:16 |
| clarkb | ya | 22:16 |
| fungi | i have a suspicion we're running into a jjb timeout or something along those lines... because jenkins.o.o is still reporting with greater frequency http://puppetboard.openstack.org/node/jenkins.openstack.org | 22:16 |
| clarkb | https://docs.puppetlabs.com/references/3.3.1/man/apply.html | 22:16 |
| nibalizer | has anyone run puppet apply manually on a jenkins master to see what the output is? | 22:16 |
| nibalizer | I generally start there when debugging puppet | 22:16 |
| fungi | and the puppetry/ansiblery for jenkins0?.o.o and jenkins.o.o should be basically identical | 22:17 |
| nibalizer | instead of reading logs | 22:17 |
| jeblair | nibalizer: no, and while i agree that may tell us something in this instance, i believe it is *critically* important that we have logs... | 22:17 |
| fungi | nibalizer: i usually check logs first to avoid tainting evidence of a possible bug/error condition | 22:17 |
| jeblair | nibalizer: over beer sometime, we can debate the wisdom of retrying a failed command withot knowing why it failed :) | 22:17 |
| jeblair | fungi: +1 | 22:17 |
| fungi | i should go grab a beer now, obviously ;) | 22:17 |
| nibalizer | i acutally am gonna go grab a monster | 22:18 |
| nibalizer | back in a flash | 22:18 |
| jeblair | monster flash | 22:18 |
| fungi | hope it doesn't grab back | 22:18 |
| * fungi has monster mash stuck in his head now | 22:18 | |
| *** sridhar_ram has joined #openstack-infra | 22:18 | |
| jeblair | excellent | 22:19 |
| *** yaume has joined #openstack-infra | 22:19 | |
| anteaya | fungi: now I do too | 22:19 |
| *** regXboi has quit IRC | 22:19 | |
| openstackgerrit | James E. Blair proposed openstack-infra/ansible-puppet: Log puppet to syslog instead of json https://review.openstack.org/271057 | 22:20 |
| jeblair | clarkb, mordred, fungi, nibalizer: ^ that's my shot in the dark | 22:20 |
| *** sdake has quit IRC | 22:20 | |
| clarkb | we should probably test it | 22:20 |
| jeblair | any idea how? :) | 22:20 |
| clarkb | just do a manual run somewhere with that flag set | 22:21 |
| jeblair | oh you mean logdest | 22:21 |
| *** thorst has quit IRC | 22:21 | |
| jeblair | sure | 22:21 |
| clarkb | ya | 22:21 |
| anteaya | jeblair: you said it was very long lines, I didn't realize that all the ouput was in one long line | 22:21 |
| clarkb | if you want to test the ansible you can install it all in a virtualenv and use a different module path then run it against a single host | 22:21 |
| *** thorst has joined #openstack-infra | 22:21 | |
| fungi | yeah, i'm +2 on the idea, as long as someone confirms that "syslog" is the value which does what we want | 22:21 |
| *** darrenc_afk is now known as darrenc | 22:22 | |
| jeblair | anteaya: to be fair, i don't know how many long lines it is because the only one i can see is long enough to prevent me from figuring out how many others there are | 22:22 |
| jeblair | anteaya: like, is it one line per server? or one for all the servers? | 22:22 |
| jeblair | i think it's probably the former | 22:22 |
| jeblair | but it's just a guess | 22:22 |
| jeblair | i'm running "puppet apply --logdest syslog" on etherpad | 22:22 |
| anteaya | jeblair: fair point | 22:23 |
| *** diazjf has quit IRC | 22:23 | |
| *** markvoelker has joined #openstack-infra | 22:24 | |
| *** zhurong has joined #openstack-infra | 22:24 | |
| jeblair | i think i need to tell it more than that | 22:24 |
| jeblair | i don't know what our actual apply command is... | 22:24 |
| *** vgridnev has quit IRC | 22:25 | |
| jeblair | 'puppet apply /opt/system-config/production/manifests/site.pp' ? | 22:25 |
| jeblair | yep | 22:25 |
| fungi | jeblair: documented at http://docs.openstack.org/infra/system-config/puppet.html#running-puppet-on-nodes | 22:25 |
| jeblair | Jan 21 22:25:37 etherpad puppet-user[29825]: Finished catalog run in 4.39 seconds | 22:25 |
| clarkb | we should fix the quoting there | 22:25 |
| clarkb | so that it jumps out a bit more | 22:26 |
| jeblair | that did the right thing in syslog with no stdout | 22:26 |
| *** thorst has quit IRC | 22:26 | |
| fungi | excellent. let's merge this | 22:26 |
| fungi | oh, failing jobs | 22:26 |
| *** dkranz has quit IRC | 22:26 | |
| fungi | also SamYaple has a good comment | 22:26 |
| jeblair | 2016-01-21 22:23:04.041 | Detected a distutils installed project ('PyYAML') which we cannot uninstall. The metadata provided by distutils does not contain a list of files which have been installed, so pip does not know which files to uninstall. | 22:26 |
| jeblair | i think our snake just ate its own tail | 22:27 |
| fungi | jeblair: so we've ended up with latest pip | 22:27 |
| clarkb | SamYaple has a good comment on the ansible-puppet change | 22:27 |
| clarkb | so I wonder if we can pass two logdests | 22:27 |
| jeblair | wait | 22:27 |
| jeblair | i don't think it's correct | 22:27 |
| ianw | jeblair: that problem you got there is fixed in 8.0.1 pip | 22:28 |
| jeblair | ianw: i understand. | 22:28 |
| jeblair | believe me. | 22:28 |
| ianw | oh, sorry :) | 22:28 |
| SamYaple | oh i seem to have stirred comments. fyi i dont know what this change does it just popped up on gertty for me, i dont know the big picture | 22:28 |
| fungi | SamYaple: it's matching against the stdout from line 103 though, right? | 22:28 |
| clarkb | SamYaple regardless your comment is correct :) | 22:28 |
| fungi | er, 193 | 22:28 |
| fungi | oh, line 197 is appending to it | 22:29 |
| *** thorst has joined #openstack-infra | 22:29 | |
| clarkb | fungi: no stdout is written though it would go to syslog | 22:29 |
| jeblair | okay, i have replied to SamYaple in gerrit | 22:29 |
| jeblair | but i will also test that case | 22:29 |
| *** baoli has quit IRC | 22:30 | |
| clarkb | I guess it hinges on whether or not puppet apply is affected by the puppet lock file | 22:30 |
| fungi | but yeah, i'm not entirely concerned if we can't tell from the log on the puppetmaster whether puppet is disabled or couldn't run for some other reason | 22:31 |
| *** rossella_s has quit IRC | 22:31 | |
| jeblair | it is not. | 22:31 |
| clarkb | if it isn't then I agree, that is dead code. If it does then we should support it | 22:31 |
| fungi | also that | 22:31 |
| jeblair | i ran 'puppet agent --disable' | 22:31 |
| jeblair | puppet apply still works | 22:31 |
| fungi | no reason to even keep that check around | 22:31 |
| *** rossella_s has joined #openstack-infra | 22:31 | |
| fungi | agreed | 22:31 |
| jeblair | so that message will never show up in the apply case | 22:31 |
| fungi | it's just cruft from when we still called agent --test there | 22:32 |
| *** sdake has joined #openstack-infra | 22:32 | |
| jeblair | there's another conditional block above that can call agent | 22:32 |
| jeblair | so it's still possible to get that output if you run in agent mode | 22:32 |
| jeblair | i'm only changing the log destination for apply though | 22:32 |
| jeblair | so we should keep the test since it's still possible to hit | 22:32 |
| jeblair | but we should not be concerned about it's applicability to apply (because it isn't) | 22:33 |
| clarkb | but will not match in the apply case | 22:33 |
| clarkb | right | 22:33 |
| jeblair | iow, i think the change is in its best form as written | 22:33 |
| *** thorst has quit IRC | 22:34 | |
| *** dslev has quit IRC | 22:34 | |
| jeblair | so now we just need to figure out how to get it merged | 22:34 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Make the v3 only devstack job voting https://review.openstack.org/271064 | 22:34 |
| clarkb | jeblair: is it still too early to build new images? | 22:35 |
| clarkb | that can be time consuming but seems like the least effort | 22:35 |
| *** otter768 has joined #openstack-infra | 22:35 | |
| *** petems has joined #openstack-infra | 22:36 | |
| *** vgridnev has joined #openstack-infra | 22:36 | |
| jeblair | clarkb: the patch i needed to update trusted hosts has merged | 22:36 |
| jeblair | clarkb: i do not know the status of pip/virtualenv if we want to try to get those in too | 22:37 |
| jeblair | clarkb: however, we can also pin pip for that test | 22:37 |
| jeblair | unfortunately, the job change will not apply | 22:37 |
| jeblair | so we could go ahead and try running jenkins-jobs on the masters and see | 22:37 |
| jeblair | fungi: did you fix jenkins07? | 22:38 |
| *** jpr has joined #openstack-infra | 22:38 | |
| *** otter768 has quit IRC | 22:39 | |
| *** weshay_xchat has quit IRC | 22:40 | |
| jeblair | i'm going to run jenkins-jobs on 01 | 22:40 |
| ianw | dstufft said in pypa-dev he will do 8.0.2 pip (fixes a certificate thing) and virtualenv soonish, about a hour ago | 22:40 |
| anteaya | jeblair: yes jenkins 07 came back up | 22:41 |
| anteaya | http://grafana.openstack.org/dashboard/db/nodepool | 22:41 |
| anteaya | currently has 66 nodes in used status on it | 22:41 |
| *** claudiub has quit IRC | 22:42 | |
| fungi | jeblair: yeah, 07 should be back up and fine now | 22:42 |
| anteaya | ianw: thanks for the update | 22:43 |
| *** trown is now known as trown|outttypeww | 22:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Fix normalize_role_assignments() return value https://review.openstack.org/271068 | 22:44 |
| nibalizer | jeblair: clarkb so looks like we spun up a patch? | 22:46 |
| clarkb | nibalizer: yes there is a patch to ansible-puppet that is hitting pip things | 22:46 |
| jeblair | INFO:jenkins_jobs.builder:Number of jobs generated: 8200 | 22:47 |
| jeblair | and it looks like it's updating every one | 22:48 |
| jeblair | and i don't know why -- there's no change | 22:48 |
| clarkb | hrm the cache may not be functioning properly then | 22:49 |
| clarkb | I did have to move the old cache aside at one point because it was invalid yaml | 22:49 |
| jeblair | the current cache appears to be valid yaml | 22:50 |
| *** ajmiller has quit IRC | 22:50 | |
| *** ajmiller has joined #openstack-infra | 22:50 | |
| *** vgridnev has quit IRC | 22:50 | |
| fungi | there were a few changes landed in jjb today | 22:51 |
| *** HeOS has joined #openstack-infra | 22:51 | |
| openstackgerrit | Fausto Marzi proposed openstack/requirements: Add modules for freezer mitaka m2 release https://review.openstack.org/271072 | 22:51 |
| fungi | none of those are jumping out at me as something that would alter our current xml though | 22:52 |
| jeblair | fungi: i don't think the xml is altered | 22:53 |
| jeblair | https://jenkins01.openstack.org/job/aeromancer-branch-tarball/jobConfigHistory/showDiffFiles?histDir1=%2Fvar%2Flib%2Fjenkins%2Fjobs%2Faeromancer-branch-tarball%2Fconfig-history%2F2016-01-21_20-58-52&histDir2=%2Fvar%2Flib%2Fjenkins%2Fjobs%2Faeromancer-branch-tarball%2Fconfig-history%2F2016-01-21_22-47-44 | 22:53 |
| jeblair | fungi: at least based on that | 22:53 |
| *** mriedem has quit IRC | 22:53 | |
| jeblair | trying again with debug logging | 22:53 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Basic folder support https://review.openstack.org/134307 | 22:53 |
| *** tonytan4ever has quit IRC | 22:53 | |
| openstackgerrit | Fausto Marzi proposed openstack/requirements: Add modules for freezer mitaka m2 release https://review.openstack.org/271072 | 22:54 |
| *** zhurong has quit IRC | 22:55 | |
| fungi | the jenkins masters did eventually report again | 22:55 |
| *** harlowja has quit IRC | 22:56 | |
| fungi | they're just reporting at a 1.5-2 hours cadence | 22:56 |
| *** harlowja has joined #openstack-infra | 22:56 | |
| jeblair | weird | 22:56 |
| fungi | though the reports aren't entirely helpful... http://puppetboard.openstack.org/report/jenkins01.openstack.org/9d982249bd054b7f4dd25634b3cfd05d15db07db | 22:56 |
| fungi | also, the last report before the span between reports got considerably longer indicates one thing was changed by puppet... /etc/pip.conf | 22:58 |
| mordred | jeblair: off the phone- how can I help with the ansibles? | 22:58 |
| jeblair | mordred: check out https://review.openstack.org/271057 | 22:58 |
| mordred | jeblair: kk | 22:58 |
| jpr | asselin: the behavior with nodepool 0.2.0 vs master appears to be about the same | 22:58 |
| asselin | jpr, in what regard? | 22:58 |
| *** ayoung has joined #openstack-infra | 22:58 | |
| *** annegent_ has quit IRC | 23:00 | |
| jpr | asselin: the only difference is in the output I'm getting (using --debug). the 0.2.0 only showed messages regarding the "stevedore.exstention" and then reported dpc was already building | 23:00 |
| jpr | 2016-01-21 14:41:42,007 DEBUG stevedore.extension: found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generi | 23:00 |
| jpr | c:Password') | 23:00 |
| jpr | 2016-01-21 14:41:42,050 ERROR nodepool.NodePool: Image dpc is already being built | 23:00 |
| mordred | jeblair: patch looks great - I'd like to do a follow up when things are not broken to make that behavior choice optional and one we opt in to | 23:00 |
| jeblair | mordred: yeah, i was thinking something similar | 23:00 |
| jeblair | mordred: i just wanted to land it quickly | 23:00 |
| jeblair | mordred: which clearly isn't going to happen | 23:00 |
| mordred | jeblair: (largely because the upstreamed version of this module which we upstreamed has actually picked up users - apparently we're not the only ones wanting to do this :) ) | 23:00 |
| jeblair | okay, jjb finally started updating jobs again | 23:01 |
| mordred | jeblair: ++ | 23:01 |
| jeblair | it thinks that some jobs have updated | 23:01 |
| jeblair | and some have not | 23:01 |
| jeblair | the debug output was not useful. | 23:01 |
| asselin | jpr, right I was saying by using 0.2.0 you can build the image with nodepool not running. So sudo service nodepool stop. Delete the image being built, and start a new one. | 23:01 |
| jpr | using master it reported the same but had messages related to gearman after the "EntryPoint" message. that reference polling gearman and then concluded with the same in progress | 23:01 |
| jeblair | oh | 23:02 |
| jeblair | did sdague's change merge? | 23:02 |
| jeblair | the one that checks out requirements all the time? | 23:02 |
| fungi | aha, the timeout increase? | 23:02 |
| fungi | yeah, that would explain it | 23:02 |
| fungi | i did approve that one and didn't think to go manually run jjb everywhere | 23:03 |
| jeblair | no not the timeout -- the constraints thing | 23:03 |
| jpr | asselin: never got to the nodepool start part. I was testing the image build prior to turning it on | 23:03 |
| jeblair | fungi: because when i look a bit further back in the history i see this https://jenkins01.openstack.org/job/aeromancer-branch-tarball/jobConfigHistory/showDiffFiles?histDir1=%2Fvar%2Flib%2Fjenkins%2Fjobs%2Faeromancer-branch-tarball%2Fconfig-history%2F2016-01-13_23-40-55&histDir2=%2Fvar%2Flib%2Fjenkins%2Fjobs%2Faeromancer-branch-tarball%2Fconfig-history%2F2016-01-21_23-00-24 | 23:03 |
| asselin | jpr, ok...nodepool master won't build images when it's not started..... | 23:03 |
| notmorgan | clarkb, fungi, sdague, mtreinish: https://review.openstack.org/#/c/270474/ should be good to go now, (master memcache thing for token validates) | 23:03 |
| jeblair | fungi: so i think that may have been approved today and ever since then, jjb has been timing out trying to update those jobs everywhere | 23:03 |
| asselin | jpr, that was the discussion earlier ^^^ | 23:04 |
| fungi | jeblair: https://review.openstack.org/267133 was approved today | 23:04 |
| fungi | mordred: approved that, and it merged at 16:18 | 23:04 |
| jeblair | mordred: that change updates 7318 jobs | 23:04 |
| jeblair | mordred: would you mind running manual jenkins jobs updates in screen across all of the masters? | 23:05 |
| openstackgerrit | Merged openstack/diskimage-builder: Use pip 7 for ironic https://review.openstack.org/270010 | 23:05 |
| anteaya | this always use pip constraints did not merge: https://review.openstack.org/#/c/267096/ | 23:05 |
| jeblair | anteaya: yeah, that's a nova change, not the jjb change. the jjb change did merge | 23:06 |
| anteaya | this always make upper constraints availble to python jobs did merge: https://review.openstack.org/#/c/267133/ | 23:06 |
| *** dimtruck is now known as zz_dimtruck | 23:06 | |
| *** Sukhdev has quit IRC | 23:06 | |
| jeblair | you can see the output here: http://logs.openstack.org/33/267133/3/check/gate-project-config-compare-xml/364068e/console.html | 23:06 |
| jeblair | it is not short | 23:06 |
| jpr | asselin: so is the order in the docs off then? http://docs.openstack.org/infra/openstackci/third_party_ci.html#start-nodepool | 23:07 |
| jeblair | anyway, we have 8200 jobs, it updated 7318 of them, leaving 882 untouched. | 23:07 |
| jpr | it says to test nodepool image-build dpc first before turning on nodepoold | 23:07 |
| *** unicell1 has joined #openstack-infra | 23:07 | |
| asselin | jpr, yes, it hasn't been updated yet for the most recent version of nodepool. I mentioned that before ^^ | 23:07 |
| anteaya | I can't even load that log file | 23:07 |
| anteaya | as you say it is not short | 23:08 |
| jpr | asselin: OK. the correct process is "turn start nodepool and /then/ test the image-build? | 23:08 |
| *** unicell has quit IRC | 23:08 | |
| asselin | jpr, I don't know how to test it using the new nodepool master. | 23:08 |
| jeblair | mordred: are you still around? | 23:09 |
| asselin | jpr, but if you figure it out, please submit a doc update | 23:09 |
| clarkb | asselin: jpr I am not sure why we would instruct people to uild an image first | 23:09 |
| clarkb | asselin: jpr one reason to use nodepool is that this is all handled for you | 23:10 |
| *** gordc has quit IRC | 23:10 | |
| asselin | clarkb, because there a plenty of errors that happen during image builds | 23:10 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet/bundle: retry failed network or git requests https://review.openstack.org/271080 | 23:10 |
| *** zhurong has joined #openstack-infra | 23:10 | |
| asselin | clarkb, for new setups, it much easier to debug them wi/o noepool running | 23:10 |
| clarkb | asselin: I am not sure I agree, everything is logged for you | 23:11 |
| clarkb | and if you don't have any issues the overhead is lower | 23:11 |
| jpr | asselin: ok. :) so should I just assume the image-build step "works", service nodepool start and go from there to continue with the setup/test | 23:11 |
| zaro | fungi: i'm no longer admin on review-dev. would you be able to fix that? | 23:12 |
| *** Sukhdev has joined #openstack-infra | 23:12 | |
| asselin | clarkb, I've found it easier. You can set DIB_OFFLINE, you can DIB break, etc | 23:12 |
| *** MarkAtwood has joined #openstack-infra | 23:12 | |
| clarkb | but you are actively circumventing the software | 23:12 |
| asselin | clarkb, and all the cache is setup in nodepool so it's not lost | 23:13 |
| zaro | fungi: actually i don't mean the server, i mean gerrit. | 23:13 |
| fungi | jeblair: mordred: i'll go ahead and do the jjb updates dance in scren sessions. i'm around this evening | 23:13 |
| clarkb | asselin: and it leads to confusion when upstream doesn't expect you to not run the softare | 23:13 |
| fungi | zaro: i see you at https://review-dev.openstack.org/#/admin/groups/1,members | 23:13 |
| asselin | clarkb, debugging issues is an important part | 23:13 |
| jeblair | fungi: thanks. i kind of think it should be the policy that people who approve changes that update all the jobs should do that. | 23:13 |
| fungi | i agree | 23:14 |
| jeblair | but that's just my opinion | 23:14 |
| *** Daisy has joined #openstack-infra | 23:14 | |
| mordred | ++ | 23:14 |
| asselin | clarkb, this was the way it's been for a while...only recent nodepool doesn't support it.... | 23:14 |
| *** pahuang has joined #openstack-infra | 23:14 | |
| jeblair | mordred: are you volunteering or just thanking fungi? | 23:14 |
| fungi | mordred: so are you around to do that? | 23:14 |
| anteaya | jeblair: I see this in the file: INFO:jenkins_jobs.builder:Number of jobs generated: 8199 I can't see where it says it updated 7318 of them though | 23:14 |
| clarkb | asselin: right because no one considered that you wouldn't be using nodepool to run nodepool :) | 23:14 |
| clarkb | asselin: we were careful to make sure that it was backward compatible. Nodepool will run an internal build worker by default | 23:15 |
| clarkb | but that does require running the service | 23:15 |
| zaro | fungi: woops that was wako. i think i was trying to do things to review.o.o instead. thanks. | 23:15 |
| jeblair | anteaya: i got that from running jjb. you could probably get it by grepping for something in that log file that it output each time it updated a job | 23:15 |
| *** yaume has quit IRC | 23:15 | |
| jpr | asselin: the problem i'm seeing looks like nodepool is happily submitting the image-build, is told by zuul/gearman its underway but then gearman isn't running under zuul so nothing actually happens. | 23:15 |
| asselin | clarkb, so what's the best way to debug issues with nodepool image build? | 23:15 |
| anteaya | jeblair: ah okay I am trying that | 23:15 |
| *** piet has quit IRC | 23:15 | |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: include keystonev2 role assignments https://review.openstack.org/270993 | 23:16 |
| jpr | asselin: nodepool 0.2.0 didn't seem to change that. you mentioned "deleting the exisitng build", how do I do that? | 23:16 |
| mordred | fungi: I can do it - it's been a while - can you give me the tl;dr so that I don't mess it up? | 23:16 |
| jeblair | anteaya: i would say that it's probably sufficient to look at the log and when you realize your computer doesn't have enough memory to display it, infer that it will affect many of the jobs. | 23:16 |
| asselin | jpr, you'll need to ask clarkb or greghaynes for lastest nodepool questions...I haven't used it yet so can't answer those... | 23:16 |
| clarkb | asselin: use the image logs | 23:16 |
| anteaya | jeblair: fair enough | 23:16 |
| jpr | asselin: ok | 23:16 |
| asselin | clarkb, so no more using any dib debugging helpiers.... | 23:16 |
| clarkb | asselin: I don't know what that means | 23:17 |
| * mordred apologizes to everyone for merging the break-all-the-jobs change. totally my fault | 23:17 | |
| jpr | asselin: clarkb I'll test more in a couple hours. thanks | 23:17 |
| anteaya | mordred: I accept your apology, thank you | 23:17 |
| anteaya | mordred: I'm learning from your experience | 23:18 |
| clarkb | asselin: you can always run dib yourself too | 23:18 |
| clarkb | nodepool isn't doing anything special around that, but the whole idea is that nodepool will take care of it for you | 23:18 |
| *** Daisy has quit IRC | 23:18 | |
| jeblair | mordred: "jenkins-jobs --conf /etc/jenkins_jobs/jenkins_jobs.ini update --delete-old /etc/jenkins_jobs/config" is a reasonable looking command i found in history which seemed to be dtrt | 23:18 |
| asselin | clarkb, oh, then that's good. how do you do that? | 23:18 |
| mordred | jeblair: great. and it's ok to do all of them simultaneously, yeah? | 23:19 |
| jeblair | mordred: yes, preferable even | 23:19 |
| mordred | great | 23:19 |
| clarkb | asselin: http://docs.openstack.org/developer/diskimage-builder/user_guide/building_an_image.html | 23:19 |
| jeblair | mordred: some may die and need to be restarted :( | 23:19 |
| jeblair | mordred: depending on how healthy the given jenkins master is at any given time | 23:19 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Adding support for maven-metadata-plugin https://review.openstack.org/257430 | 23:19 |
| mordred | yay | 23:19 |
| anteaya | that file is 12MB | 23:19 |
| ianw | asselin: depending on what you're doing things can vary, but also check out https://www.technovelty.org/openstack/platform-bootstrap-for-openstack-infrastructure.html that i wrote that might help | 23:20 |
| mordred | jeblair: while I'm getting that set up - for your puppet apply thing- when --logdest=syslog ... there should be almost no outout on stdout, yeah? | 23:20 |
| asselin | clarkb, yeah....but I want to reuse the nodepool cache.... | 23:20 |
| clarkb | asselin: you can | 23:20 |
| jeblair | mordred: yes | 23:20 |
| clarkb | asselin: nodepool isn't doing anything special, it sets are few vars and thats it | 23:20 |
| mordred | jeblair: thanks | 23:20 |
| fungi | mordred: thanks! | 23:20 |
| jeblair | mordred: want to go halfway on that and keep the stdout passthrough? | 23:21 |
| jeblair | i kind of killed it with fire | 23:21 |
| fungi | stdout passthrough is workable if it can just be squashed at logging time | 23:21 |
| mordred | I do - it'll make the module always return the same things in its return struct | 23:21 |
| mordred | jeblair: but I've got a follow up patch half done- so I can just add it back in there | 23:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Pin pip to <8 for ansible lint jobs https://review.openstack.org/271083 | 23:22 |
| mordred | jeblair: jenkins0[1-8 and jenkins.o.o yah? | 23:22 |
| jeblair | mordred: yeah; though i don't think we have an 8 | 23:22 |
| asselin | clarkb, sure...but before it was very easy to manually create one. Anyway I found it very very helpful in the past. | 23:22 |
| mordred | oh - I meant 7 | 23:22 |
| clarkb | asselin: it is still very easy to do that | 23:22 |
| clarkb | asselin: it just requires a separate daemon | 23:22 |
| mordred | k. jjb running everywhere | 23:23 |
| fungi | mordred: also jenkins.o.o usually completes within a few minutes (or even seconds if it managed to keep up) | 23:23 |
| anteaya | my grepping got 4998: http://paste.openstack.org/show/484627/ | 23:24 |
| jeblair | mordred, fungi, clarkb: route A) out of the hole: https://review.openstack.org/271083 then mordred finishes his jjb run then https://review.openstack.org/271057 | 23:24 |
| jeblair | mordred, fungi, clarkb: route B) out of the hole: we build new images that have pip 801 or 802 | 23:24 |
| fungi | mordred: jenkins01 and 02 tend to take a good 50% longer than 03-07 and are also a little more prone to dying from api failures | 23:24 |
| fungi | mordred: but expect 60-90 minutes across the board best case, with failures on some... longer | 23:24 |
| clarkb | jeblair: based on past experience both will be slow, therefore I think we should bite the bullet and do the images | 23:25 |
| clarkb | since that will fix more than just us | 23:25 |
| *** jpr has quit IRC | 23:25 | |
| asselin | clarkb, ok I haven't tried it myself yet, so I can't really say it's easier or not or how to do it and document it | 23:25 |
| asselin | clarkb, ok here's the one from ianw's doc: break=after-error | 23:25 |
| clarkb | asselin: for that you are probably best off running it directly | 23:26 |
| fungi | it does sound like b is the safer bet of the two | 23:26 |
| clarkb | last time I had to update the jenkins jobs world it took > 8 hours | 23:26 |
| clarkb | I think we can get image rebuilds done quicker than that | 23:26 |
| * anteaya votes B | 23:26 | |
| fungi | last time i did it, was only 2-3 hours. though i did end up having to switch a couple of jenkins masters into prepare for shutdown to get them to finish | 23:27 |
| fungi | the jjb updates duration that is | 23:27 |
| jeblair | do we feel comfortable kicking off image builds at the end of our day? | 23:27 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Add Docker Custom Build Environment Plugin support https://review.openstack.org/258739 | 23:27 |
| jeblair | i mean, it's sort of... how can it be worse, right? | 23:27 |
| clarkb | jeblair: exactly :) | 23:28 |
| asselin | clarkb, sure...but it worked beautifully and simply before. now I have figure out what nodepool does so I can reproduce it and be out of sync with nodepool.yaml configuration. http://paste.openstack.org/show/484629/ | 23:28 |
| fungi | jhesketh: are you around to weigh in? it's partway into your day now | 23:28 |
| clarkb | asselin: those comments were welcome during the spec and code review period? | 23:28 |
| clarkb | asselin: I don't think anyone knew that this was a use case | 23:28 |
| *** arxcruz has quit IRC | 23:28 | |
| clarkb | asselin: I personally run dib locally without nodepool, its simpler | 23:28 |
| jhesketh | fungi: here, but haven't been paying attention sorry.. | 23:28 |
| * jhesketh reads | 23:28 | |
| clarkb | asselin: and being able to trim it down to a specific subset of elements aids in debug turnaround | 23:29 |
| clarkb | asselin: so that is both a positive and negative I guess | 23:29 |
| fungi | jhesketh: if we kick off new jjb image updates and you're the only infra-root around when someone notices they've somehow made things even _worse_ than they already are, are you comfortable deleting them or troubleshooting? | 23:29 |
| asselin | clarkb, agree...didn't review it. Didn't realize it wasn't a valid use case. Did document it though: http://docs.openstack.org/infra/openstackci/third_party_ci.html#start-nodepool | 23:29 |
| fungi | s/jjb/nodepool/ | 23:29 |
| fungi | too many tools on teh brainz | 23:29 |
| asselin | clarkb, but sure can figure out the next best way to debug issues and update the docs | 23:30 |
| jhesketh | fungi: yep, so long as we put the older images on hold that's fine | 23:30 |
| *** sdake has quit IRC | 23:30 | |
| jeblair | clarkb, greghaynes: how do i confirm the nodepool builder is idle? | 23:30 |
| asselin | clarkb, reading that again: 2nd issue is manually uploading to provider...is that still valid? | 23:30 |
| greghaynes | jeblair: hrm | 23:31 |
| clarkb | jeblair: I had been checking gear | 23:31 |
| jeblair | clarkb, greghaynes: the last log message is an exception building, so i'm assuming it is | 23:31 |
| *** bradjones_ has joined #openstack-infra | 23:31 | |
| *** bradjones_ has quit IRC | 23:31 | |
| *** bradjones_ has joined #openstack-infra | 23:31 | |
| fungi | jhesketh: nodepool retains the previous successful image builds (in theory) so it should only be clearing out the images from two builds ago anyay | 23:31 |
| fungi | anyway | 23:31 |
| ianw | asselin: i feel like it would be a quick hack to make a "dump command line" that replicates what command you should run to trigger a build manually for debugging. would that help? | 23:31 |
| jeblair | (otherwise i would expect a message saying it was starting something) | 23:31 |
| greghaynes | jeblair: yes, it should spew a "got job ..." right when it starts handling a job | 23:31 |
| clarkb | asselin: yes all of the functionality continues to exist, you just have to have the service connected to gearman | 23:31 |
| asselin | ianw yes it would | 23:31 |
| jeblair | clarkb, greghaynes: so i can go ahead and delete the stale building entries | 23:31 |
| clarkb | jeblair: yup I Think so | 23:31 |
| jhesketh | fungi: more worried that if we kick a build now and then the cron does another build that we lose the good images | 23:32 |
| jeblair | clarkb: can i dib-image-delete them? | 23:32 |
| greghaynes | asselin: clarkb One thing we could do - the way builders are coded it wouldn't be too hard to have a command that runs a build | 23:32 |
| *** bhunter71 has quit IRC | 23:32 | |
| clarkb | jeblair: I don't tnink it was the dib image builds that were stale, it was the uploads which are recorded in the other table | 23:32 |
| greghaynes | jeblair: yep | 23:32 |
| boris-42 | clarkb: hi | 23:32 |
| jeblair | well, there are two that are listed as building | 23:32 |
| boris-42 | clarkb: is keystone already fixed? | 23:32 |
| fungi | jhesketh: the cron doesn't start until 14:14 utc, so we'll presumably know before then? does nodepool have a hold image option? | 23:33 |
| clarkb | jeblair: ah then yes they were probably still building when the disconnect happened | 23:33 |
| boris-42 | clarkb: because work on rally is blocked | 23:33 |
| boris-42 | clarkb: this may help us for now https://review.openstack.org/#/c/270134/ | 23:33 |
| clarkb | boris-42: notmorgan has a patch to put memcached in place and in the interim I believe they reverted the removal of built in caching | 23:33 |
| clarkb | boris-42: so I think so | 23:33 |
| jhesketh | fungi: I don't think it has a hold.. maybe we should stop it from rebuilding? | 23:33 |
| jhesketh | granted, you're right about probably knowing before then | 23:34 |
| boris-42 | clarkb: so we are still facing issues | 23:34 |
| jeblair | clarkb, greghaynes: that appeared to just submit a gearman function and sit there | 23:34 |
| boris-42 | clarkb: can we increase the timeout for now? | 23:34 |
| clarkb | boris-42: can you ask notmorgan about it? he is more familiar with the situation | 23:34 |
| fungi | jhesketh: also, the key bit here is that the usability of our images is pretty unlikely to get even worse than now | 23:34 |
| greghaynes | jeblair: dib-image-delete? | 23:34 |
| jeblair | yep | 23:34 |
| boris-42 | clarkb: in any case I would be happy to unblock development of rally which shouldn't be blocked by this | 23:34 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:34 | |
| clarkb | jeblair: so I did have to edit the db by hand last time | 23:34 |
| clarkb | oh I know why | 23:35 |
| greghaynes | jeblair: hrm, is the corresponding job registerd in dib? | 23:35 |
| clarkb | because there is no file on disk | 23:35 |
| jeblair | right | 23:35 |
| clarkb | so gear worker doesn'yt register a delete function | 23:35 |
| greghaynes | er, registered in gearman, but ya | 23:35 |
| greghaynes | if the file isnt on disk it wont have the delte registered | 23:35 |
| jeblair | this is a case that should be handled though | 23:35 |
| greghaynes | Yep, although I don't have any great ideas for how... | 23:35 |
| jeblair | generally "nodepool delete-thing" should cause it to be deleted even if it's in an inconsistent state | 23:35 |
| greghaynes | since listing jobs is really expensive | 23:36 |
| clarkb | right the worker can't have access to the db | 23:36 |
| clarkb | greghaynes: might be fine to list the jobs in this case since its exceptional | 23:36 |
| clarkb | and if no job present just clean up db | 23:36 |
| greghaynes | and we don't have a way to fan out and ask al lthe workers what files they can delete | 23:36 |
| clarkb | we already list all the things anyways | 23:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Log puppet to syslog instead of json https://review.openstack.org/271057 | 23:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Sync with upstream ansible https://review.openstack.org/271085 | 23:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Make puppet logout destination configurable https://review.openstack.org/271086 | 23:37 |
| clarkb | but thats in the main run loop which the commands won't have access to | 23:37 |
| greghaynes | if we could ask gearman if it has a worker able to perform a specific job that would be nice... | 23:37 |
| clarkb | greghaynes: you can (via listing) | 23:37 |
| greghaynes | ya, I mean efficiently | 23:37 |
| *** EricGonczer_ has quit IRC | 23:38 | |
| fungi | why is listing expensive? just because of the 8k registered jenkins jobs x multiple labels? | 23:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Configure out puppet runs to log to syslog https://review.openstack.org/271087 | 23:38 |
| clarkb | fungi: yes basically | 23:38 |
| greghaynes | fungi: I think so | 23:38 |
| jeblair | it takes 8.5 seconds | 23:38 |
| jhesketh | fungi: so these new nodes are going to pin below pip8? | 23:38 |
| fungi | jhesketh: opposite | 23:39 |
| anteaya | jhesketh: no, pip 8.0.1 is out, it will use it | 23:39 |
| fungi | jhesketh: they'll build with pip 8+ | 23:39 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: WIP: Add initial optional influxdb support for test_runs https://review.openstack.org/271088 | 23:39 |
| jhesketh | ah, right | 23:39 |
| * jhesketh is still catching up sorry | 23:39 | |
| anteaya | jhesketh: no no you are asking good questions | 23:39 |
| fungi | released since we last built images | 23:39 |
| anteaya | jhesketh: thank you | 23:39 |
| *** gildub has quit IRC | 23:39 | |
| yamahata | Hello I need help to debug https://review.openstack.org/#/c/265437/ | 23:39 |
| anteaya | jhesketh: much scrollback to wake up to | 23:39 |
| fungi | this is the sort of brokenness hand-off we should do between timezones more often, honestly | 23:40 |
| yamahata | It caused NOT_REGISTERED job e.g. https://review.openstack.org/#/c/266583/ | 23:40 |
| jhesketh | fungi: so no changes to project-config or anything, just rebuilding to pick up the new version? | 23:40 |
| jeblair | clarkb, greghaynes: okay, i deleted the building dib images... | 23:40 |
| *** dprince has quit IRC | 23:40 | |
| jeblair | nodepool image-list shows no building images | 23:40 |
| greghaynes | awesome | 23:40 |
| fungi | jhesketh: yeah, there are some other changes scattered around that are either pinning pip<8 (devstack) or downgrading to <8 (beaker) which we can in theory unroll once this is working | 23:41 |
| *** bhunter71 has joined #openstack-infra | 23:41 | |
| jeblair | oh nope, i messed it up | 23:41 |
| jeblair | i deleted all of the dib images from the table | 23:41 |
| jeblair | so good news | 23:41 |
| jeblair | the dib image builds are now happening automatically | 23:41 |
| jhesketh | hah | 23:42 |
| jhesketh | :-) | 23:42 |
| fungi | we've effectively held the old images too? | 23:42 |
| clarkb | you'll want to cleanup the on disk situation once we hvae new images | 23:42 |
| jeblair | clarkb: yeah, that's the bad news | 23:42 |
| *** FallenPegasus has quit IRC | 23:42 | |
| jhesketh | fungi: okay thanks :-) | 23:42 |
| jeblair | now for the snapshot images... | 23:42 |
| anteaya | yamahata: so since 265437 our jenkins jobs have failed to update, we are working on fixing that now | 23:42 |
| fungi | er, i guess we haven't conveniently held the old images since nodepool won't have database entries for them so won't know it can use them | 23:42 |
| mordred | fungi: ++ to " this is the sort of brokenness hand-off we should do between timezones more often, honestly" | 23:42 |
| jeblair | fungi: the snapshot images are still there | 23:43 |
| boris-42 | clarkb: btw in logs I see this "WARNING: devstack run took > 20 minutes, this is a very slow node." | 23:43 |
| anteaya | yamahata: that is probably the cause of the NOT_REGISTERED status you are seeing | 23:43 |
| jeblair | fungi: and for dib images, snapshot images mean uploaded versions | 23:43 |
| clarkb | boris-42: I think that check may not have a current estimation for how long devstack actually takes | 23:43 |
| jeblair | fungi: so the uploaded dib images are still usable by nodepool | 23:43 |
| fungi | jeblair: oh, okay. great | 23:43 |
| fungi | we just can't reupload them anywhere | 23:43 |
| jeblair | fungi: though i don't know what you mean by held | 23:43 |
| yamahata | anteaya: oh okay, thanks. then I'll wait for it. I though I messed up with my patches. | 23:43 |
| boris-42 | clarkb: but what I saw is like timeout for job is 50 minutes | 23:43 |
| boris-42 | clarkb: and tests are killed after 10 minutes | 23:43 |
| anteaya | yamahata: no, I don't believe you did, I believe this is a result of a larger issue we are working on fixing | 23:44 |
| boris-42 | clarkb: so it looks like it take about 40 minutes to get OpenStack installed | 23:44 |
| fungi | jeblair: jhesketh was asking if we could hold the current images so nodepool won't delete them tomorrow in case we need to roll back two images, though we'd already dismissed that as being particularly likely to be useful | 23:44 |
| clarkb | boris-42: can you link to the job? | 23:44 |
| boris-42 | clarkb: gate-rally-dsvm-cli ? | 23:44 |
| jhesketh | fungi: also irrelevant for dib now | 23:44 |
| boris-42 | clarkb: http://logs.openstack.org/76/270176/4/check/gate-rally-dsvm-cli/ec85c60/ | 23:45 |
| jeblair | fungi, jhesketh: nodepool always keeps the current and previous image | 23:45 |
| fungi | jeblair: yep, the question was whether we could make it keep one in addition to those | 23:45 |
| jhesketh | jeblair: my concern was when the rebuild happens in a few hours it'll blow away the previous one | 23:45 |
| jhesketh | fungi, jeblair: having caught up I don't think we need to worry too much about it though | 23:45 |
| fungi | anyway, it was a rathole. no need to go further | 23:45 |
| jeblair | jhesketh: but the current one would still be there | 23:45 |
| clarkb | boris-42: that job is misconfigured and not recording timestamps properly | 23:45 |
| jeblair | jhesketh: so you could revert to the one we are using now | 23:46 |
| dstufft | Okay, I'm cutting 8.0.2 now | 23:46 |
| jhesketh | jeblair: I meant the rebuild that'll happen at 14:14 | 23:46 |
| clarkb | boris-42: so it is really hard to help debug | 23:46 |
| fungi | jeblair: he was initially worried about what happens tomorrow after 14:14 when yet another set of updates kick off, if we haven't discovered how broken the coming set is by then | 23:46 |
| jeblair | aaah | 23:46 |
| anteaya | yamahata: if the job is still not registered tomorrow then come back and we will look again | 23:46 |
| jeblair | anyway, for the record, in case we need it, here is the contents of the table before i demolished it: http://paste.openstack.org/show/484630/ | 23:46 |
| boris-42 | clarkb: can we just get that patch with increasing timeout for now in | 23:47 |
| *** rcernin has quit IRC | 23:47 | |
| *** sdake has joined #openstack-infra | 23:47 | |
| boris-42 | clarkb: after that I will add timestapms and the rest of stuff | 23:47 |
| clarkb | boris-42: I won't +2 it but feel free to solicit other peopl's reviews | 23:47 |
| yamahata | anteaya: Got it. thank you so much for hard work. | 23:47 |
| boris-42 | clarkb: ok | 23:47 |
| anteaya | yamahata: thank you for your appreciation, today might be a great day for local testing for you | 23:47 |
| clarkb | I feel pretty strongly that part of the problem here is we continuously relax our testing rather than tighten it and it leads to these problems | 23:47 |
| *** bradjones_ has quit IRC | 23:47 | |
| *** bradjones_ has joined #openstack-infra | 23:47 | |
| *** bradjones_ has quit IRC | 23:47 | |
| *** bradjones_ has joined #openstack-infra | 23:47 | |
| anteaya | clarkb: ++ | 23:48 |
| anteaya | we are far too big to "let it slide" be an appropriate response | 23:48 |
| *** isq_ has quit IRC | 23:48 | |
| boris-42 | clarkb: anteaya so it's not the case of Rally sorry | 23:48 |
| clarkb | boris-42: rally is doing performance testing right? if we need to increase runtime by 50% there has been a regression | 23:49 |
| clarkb | boris-42: we should fix the regression not accet it | 23:49 |
| clarkb | and I beleive the fix is in progress but notmorgan has far more details than I do | 23:49 |
| boris-42 | clarkb: so we first of all should not block for 1 year Rally community until these fixes will be accepted by projects | 23:49 |
| jeblair | clarkb: for snapshots, i need to run this: "nodepool image-list|grep -v None|awk '{print $6}'|sort|uniq" to get a list of snapshot images, and then run nodepool-image-build on each of those in screen in parallel? | 23:49 |
| clarkb | boris-42: they have been accepted is what I am saying, people are actively fixing it | 23:49 |
| boris-42 | clarkb: and can do this in parallel without making harm to rally community | 23:50 |
| clarkb | boris-42: you should coordinate with them first | 23:50 |
| clarkb | jeblair: nodepool image-update | 23:50 |
| boris-42 | clarkb: if I make this job non voting? | 23:50 |
| boris-42 | clarkb: will it work for you? | 23:50 |
| *** dingyichen has joined #openstack-infra | 23:50 | |
| boris-42 | clarkb: I need to unblock development of Rally that is not the is not the root of problem | 23:51 |
| clarkb | boris-42: no what I would like to see is a statement from keystone on whether or not the issue should be correct | 23:51 |
| clarkb | boris-42: if it is in the pipeline then just waiting for it will be faster | 23:51 |
| clarkb | boris-42: if it isn't in the pipeline then we can accept increasing the timeout | 23:51 |
| boris-42 | clarkb: so we are blocking whole community of Rally because of issues in other OpenStack projects? | 23:51 |
| clarkb | boris-42: any job config changes aren't going to got through quickly right now due to the larger set of issues with pip and so on | 23:51 |
| *** darrenc is now known as darrenc_afk | 23:51 | |
| clarkb | boris-42: no I am saying what you want to do won't help anyways | 23:52 |
| clarkb | boris-42: so lets try to do it right in the meantime | 23:52 |
| fungi | boris-42: i thought rally was part of openstack. openstack is blocking development on openstack because openstack is broken | 23:52 |
| clarkb | boris-42: if we find that that isn't viable we can go the timeout bump route without spending any extra time getting it in | 23:52 |
| boris-42 | fungi: if it was bug in Rally then it's okay to block Rally development | 23:52 |
| boris-42 | unitl bug is fixed | 23:52 |
| fungi | we're all in this together | 23:53 |
| boris-42 | that is what I am saying | 23:53 |
| fungi | let's get openstack fixed | 23:53 |
| mordred | boris-42: so - the main thing is - literally everytihng is blocked at the moment | 23:53 |
| mordred | boris-42: so it's entirely possible that by the time the system is unbroken, the actual fix to the other thing that's blocking you will be ready to go | 23:53 |
| boris-42 | mordred: ok | 23:53 |
| fungi | between potential keystone performance regressions, pip 8 behavior changes, grenade multi-node network assumptions, et cetera... everything is on fire | 23:54 |
| *** kzaitsev_mb has quit IRC | 23:54 | |
| notmorgan | clarkb: saw my name | 23:54 |
| mordred | boris-42: yeah, it's a mess right now - believe me, we feel your pain ... | 23:54 |
| notmorgan | clarkb: reading backscroll | 23:54 |
| mordred | boris-42: we can't land our own patches to fix it this instant | 23:54 |
| clarkb | notmorgan: what is expected keystone middleware regression status right now? | 23:54 |
| boris-42 | mordred: ok | 23:54 |
| clarkb | notmorgan: it seemed that rally jobs were affected and I aksed that we just get the keystone fixes in rather than bump the rally job timeout | 23:54 |
| notmorgan | clarkb: we have a requirements fix in, banning the current broken version | 23:54 |
| clarkb | in part because updating jenkins job is slow an laborous right now (just like everything else so may as well go with the correct thing) | 23:55 |
| notmorgan | clarkb: the ksm fix in devstack is in flight, needs reviews | 23:55 |
| clarkb | notmorgan: with contraints updated too? | 23:55 |
| notmorgan | to use a shared memcache | 23:55 |
| boris-42 | clarkb: only one rally job is seriously affected (that is actually blocking stuff) | 23:55 |
| notmorgan | clarkb: yeah, that landed last night | 23:55 |
| boris-42 | clarkb: I don't like when I have jobs that run over 1 hour and I will work on fix | 23:55 |
| *** edmondsw has quit IRC | 23:55 | |
| clarkb | boris-42: ok so this should be better today, have they continued to fail in the last few hours? | 23:55 |
| notmorgan | shared memcache patch is waiting for review/gate, greande one is in flight in gate | 23:55 |
| fungi | notmorgan: did a revert on master happen too, or are jobs integrating ksm master likely still impacted? | 23:55 |
| boris-42 | clarkb: yep they are still failing | 23:55 |
| notmorgan | middleware is not receiving a change. | 23:55 |
| notmorgan | fungi: no revert is happening | 23:56 |
| *** isq has joined #openstack-infra | 23:56 | |
| boris-42 | clarkb: actually only one job is failing | 23:56 |
| boris-42 | clarkb: this one | 23:56 |
| fungi | boris-42: does rally install ksm from source or packages? | 23:56 |
| mordred | fungi: the roll-forward path for ksm is the devstack patch | 23:56 |
| clarkb | boris-42: ok, can you fix the timestamps in your change then? | 23:56 |
| notmorgan | mordred: ++ | 23:56 |
| fungi | boris-42: for the affected job i mean | 23:56 |
| boris-42 | clarkb: i will fix | 23:56 |
| clarkb | because I don't want to bump timestamps without being able to see how long the job is taking | 23:56 |
| boris-42 | fungi: I will | 23:56 |
| notmorgan | https://review.openstack.org/#/q/Ifc17c27744dac5ad55e84752ca6f68169c2f5a86,n,z | 23:56 |
| notmorgan | those two patches are the roll forward plan | 23:56 |
| mordred | woot | 23:56 |
| * mordred does not have devstack +2 | 23:56 | |
| fungi | notmorgan: thanks for the update | 23:56 |
| notmorgan | it should also make the banned middlewareversion happier running too | 23:57 |
| mordred | notmorgan: wasn't there some multi-node issue? | 23:57 |
| notmorgan | since we're caching token validations centrally | 23:57 |
| mordred | notmorgan: or did that also get sorted? | 23:57 |
| notmorgan | mordred: sdage addressed it | 23:57 |
| mordred | awesome | 23:57 |
| notmorgan | mordred: or at least addressed the base of it | 23:57 |
| notmorgan | we still need a bit of work to make devstack better, but it should mostly just work | 23:57 |
| dstufft | 8.0.2 is out now. | 23:57 |
| *** piet has joined #openstack-infra | 23:58 | |
| anteaya | notmorgan: do you want to perhaps remove the "remove from gate" bit in the commit message: https://review.openstack.org/#/c/270474/ | 23:58 |
| mordred | notmorgan: also - while I'm poking you about things ... | 23:58 |
| fungi | dstufft: thanks! | 23:58 |
| *** bpokorny_ has joined #openstack-infra | 23:58 | |
| mordred | notmorgan: did I hear "python-memcached does not support ipv6" earlier? | 23:58 |
| *** sabeen2 has quit IRC | 23:58 | |
| fungi | dstufft: that was mainly just fixing things for centos right? | 23:58 |
| fungi | dstufft: over 8.0.1 i mean | 23:58 |
| notmorgan | anteaya: hmm? i thought i fixed that. | 23:58 |
| *** bpokorny_ has quit IRC | 23:58 | |
| notmorgan | mordred: it didn't for a bit | 23:58 |
| notmorgan | mordred: it might now | 23:58 |
| notmorgan | mordred: that is part of what i aim to fix next with a better use of something like bmemcache | 23:59 |
| notmorgan | but i need middlewaree to move to oslo.cache first | 23:59 |
| jeblair | all of the snapshot builds are proceeding | 23:59 |
| dstufft | fungi: yes | 23:59 |
| notmorgan | so on the short list for review/landing/etc | 23:59 |
| jeblair | as are all of the dib builds | 23:59 |
| dstufft | fungi: no idea if you all were hitting that at all | 23:59 |
| *** bpokorny_ has joined #openstack-infra | 23:59 | |
| fungi | dstufft: ignore me. i should just rtfchangelog ;) | 23:59 |
| notmorgan | mordred: but i hear python-memcached is better about ipv6 now | 23:59 |
| dstufft | centos/fedora/amazon linux/who knows what else | 23:59 |
| notmorgan | just a lot of versions sucked | 23:59 |
| fungi | dstufft: but thanks again | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!