| clarkb | jeblair: tl;dr is exim said no such user for third-party-request@lists.openstack.org | 00:00 |
|---|---|---|
| *** praneshp_ has joined #openstack-infra | 00:00 | |
| clarkb | jeblair: that shows up in the exim fail log. There is no correspodning error in the mailman errorlog | 00:00 |
| *** unicell has joined #openstack-infra | 00:00 | |
| *** emagana has quit IRC | 00:00 | |
| fungi | the suffix addresses working is sort of odd since the mailman router configuration has them as optional | 00:01 |
| *** craigbr has joined #openstack-infra | 00:01 | |
| *** emagana has joined #openstack-infra | 00:01 | |
| anteaya | jeblair: thanks | 00:01 |
| *** praneshp has quit IRC | 00:03 | |
| *** praneshp_ is now known as praneshp | 00:03 | |
| fungi | a dump of the pickle file for it looks sane | 00:03 |
| *** emagana has quit IRC | 00:05 | |
| jeblair | 2014-09-02 23:33:37 1XOxZx-0000d0-0T ** | /var/lib/mailman/mail/mailman post third-party-request <third-party-request@lists.openstack.org> R=system_aliases T=address_pipe: Child process of address_pipe transport returned 2 from command: /var/lib/mailman/mail/mailman | 00:07 |
| *** otherwiseguy has quit IRC | 00:08 | |
| fungi | Sep 02 16:56:53 2014 (25772) No such list "third-party-reques": | 00:09 |
| fungi | (from /var/log/mailman/error) | 00:09 |
| anteaya | something truncated the name? | 00:10 |
| fungi | eh, no that may not be configuration related | 00:10 |
| clarkb | fungi: the timestamps don't quite line up on that either | 00:10 |
| fungi | the timestamp makes it look like it was probably someone's request | 00:10 |
| *** otherwiseguy has joined #openstack-infra | 00:11 | |
| fungi | so... the times i see in the exim rejectlog are *before* puppet created the list at 23:49:08 (per syslog) | 00:12 |
| fungi | nope, false alarm | 00:12 |
| fungi | we rerun that ensure->created on every puppet run | 00:12 |
| openstackgerrit | lifeless proposed a change to openstack-infra/reviewstats: Tripleo project updates. https://review.openstack.org/118503 | 00:12 |
| *** gokrokve has joined #openstack-infra | 00:12 | |
| fungi | list has been created there for a good long while | 00:13 |
| clarkb | fungi: huh that is probably a puppet resource bug then | 00:13 |
| clarkb | but shouldn't affect an existing list. it hasn't in the past that we are aware of | 00:13 |
| fungi | right | 00:13 |
| jeblair | ah | 00:14 |
| fungi | i think it's just noise from the mail alias configuration in the mailman module | 00:14 |
| jeblair | exim -d -bt third-party-request@lists.openstack.org | 00:14 |
| jeblair | run that and you'll have the answer | 00:14 |
| jeblair | --------> mailman_router router <-------- | 00:14 |
| jeblair | file check: /var/lib/mailman/lists/${lc:$local_part}/config.pck | 00:14 |
| jeblair | expanded file: /var/lib/mailman/lists/third-party/config.pck | 00:14 |
| clarkb | expansion isn't working right | 00:15 |
| jeblair | it is | 00:15 |
| clarkb | oh | 00:15 |
| fungi | ha! | 00:15 |
| fungi | oic | 00:15 |
| fungi | bwahahahaha | 00:15 |
| clarkb | because -request is special? | 00:15 |
| clarkb | woo | 00:15 |
| fungi | poor name choice there :/ | 00:15 |
| jeblair | yep. it's one of the special mailman addresses | 00:15 |
| jeblair | like bounces, confirm, jon, leave owner, subscribe, unsubscribe, admin... | 00:15 |
| anteaya | ah | 00:16 |
| anteaya | so the way forward? | 00:16 |
| anteaya | -requests | 00:16 |
| anteaya | ? | 00:16 |
| jeblair | probably rename the list, yeah | 00:16 |
| anteaya | would -requests work? | 00:16 |
| anteaya | different enough? | 00:16 |
| lifeless | also file a bug on mailman, it shouldn't have let that list be created | 00:17 |
| jeblair | i mean, it's exim, so we _could_ work around it, but i don't think this is worth invalidating mailman documentation, etc. | 00:17 |
| anteaya | or should we be more obvious | 00:17 |
| anteaya | we only have 14 subscribers | 00:17 |
| *** tomoe_ has joined #openstack-infra | 00:17 | |
| anteaya | we can get them moved over | 00:17 |
| *** otherwiseguy has quit IRC | 00:17 | |
| *** terryw has joined #openstack-infra | 00:17 | |
| jeblair | i expect we'll probably stop puppet, merge rename change, manually rename on disk, start puppet | 00:19 |
| clarkb | yes, otherwise you wil lend up with two lists | 00:19 |
| anteaya | do I need to get the subscribers to resubscribe? | 00:19 |
| jeblair | anteaya: nope | 00:20 |
| clarkb | puppet can delete though if you want to ensure => absent on the old one | 00:20 |
| *** erlon has quit IRC | 00:20 | |
| anteaya | kk | 00:20 |
| pleia2 | hah, oops | 00:20 |
| jeblair | clarkb: oh, maybe we should do that instead? | 00:20 |
| openstackgerrit | A change was merged to openstack-infra/config: Use zm04 https://review.openstack.org/117909 | 00:20 |
| anteaya | just change the link on third-party.rst | 00:20 |
| fungi | okay, once puppet applies that ^ to zuul, zm04 should start picking up merge-check work | 00:20 |
| *** chuckC has quit IRC | 00:20 | |
| anteaya | what have we agreed to rename the list? | 00:20 |
| jeblair | i'm okay with either way, but maybe the delete/recreate method is less error-prone? | 00:21 |
| jeblair | (either way, i'm thinking we do it tomorrow) | 00:21 |
| clarkb | I know how to review and babysit delete/recreate but renamign a list and any associated cleanup mailman side would be foreign to me | 00:21 |
| clarkb | jeblair: ++ I need to walk home shortly | 00:21 |
| fungi | delete it from puppet configuration (not ensure->absent), rename on disk, readd to puppet with correct name? | 00:21 |
| clarkb | fungi: that will work too | 00:22 |
| jeblair | fungi: oh that's a cleaner rename way, yeah | 00:22 |
| fungi | prevents stopping puppet too | 00:22 |
| *** sballe_ has quit IRC | 00:22 | |
| jeblair | i'm okay doing the rename on disk and troubleshooting if it goes awry | 00:22 |
| anteaya | I'm about to offer a patch to the docs | 00:23 |
| anteaya | just need to know what we are renaming it | 00:23 |
| fungi | anyway, i'm knocking off for the evening, other than keeping an eye on zm04 for the next few minutes | 00:23 |
| jeblair | anteaya: -requests should be fine | 00:23 |
| anteaya | kk | 00:23 |
| jeblair | anteaya: would you also like to write the list changes? the first one should just remove the list definition entirely (ie, just delete it from the file). then the second should add it back, but with the new name. | 00:24 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Edit the name of the third-party-requests ml https://review.openstack.org/118506 | 00:27 |
| anteaya | jeblair: I can try | 00:27 |
| openstackgerrit | Jay Faulkner proposed a change to openstack-infra/config: Publish individual files in IPA post job https://review.openstack.org/118507 | 00:27 |
| anteaya | I'll work from pleia2's patch: https://review.openstack.org/#/c/115392/ | 00:28 |
| JayF | fungi: sdague: clarkb: https://review.openstack.org/#/c/118507/ this is my first attempt at something to make the IPA devstack stuff more palatable, I've laid out a general plan in the commit message, please let me know what you guys think in gerrit :) CC: jroll | 00:29 |
| *** sballe has joined #openstack-infra | 00:31 | |
| *** gokrokve_ has joined #openstack-infra | 00:33 | |
| *** gokrokve has quit IRC | 00:34 | |
| *** yamahata has joined #openstack-infra | 00:34 | |
| *** baoli has joined #openstack-infra | 00:35 | |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Don't modify status widgets outside of main thread https://review.openstack.org/118498 | 00:36 |
| *** yamahata has quit IRC | 00:36 | |
| *** bhuvan_ has quit IRC | 00:36 | |
| *** mrodden has quit IRC | 00:36 | |
| *** yamahata has joined #openstack-infra | 00:37 | |
| *** mrodden has joined #openstack-infra | 00:37 | |
| *** craigbr has quit IRC | 00:37 | |
| *** nijaba has quit IRC | 00:38 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Removes mailing list third-party-request https://review.openstack.org/118508 | 00:41 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Creates third-party-requests mailing list https://review.openstack.org/118509 | 00:41 |
| openstackgerrit | lifeless proposed a change to openstack-infra/reviewstats: Make pdb debugging of openreviews possible. https://review.openstack.org/118510 | 00:44 |
| anteaya | how do I make https://review.openstack.org/#/c/118506/1 a dependency of 118509? | 00:44 |
| anteaya | since it should go in after | 00:44 |
| lifeless | git review -d 118506; git review -x 118509; git review. IIRC. | 00:44 |
| *** ominakov has joined #openstack-infra | 00:44 | |
| anteaya | thanks | 00:47 |
| jeblair | anteaya: but you still want 08 in there, so you should add it into the line lifeless suggests: -d 06, -x 08, -x 09. | 00:47 |
| anteaya | kk | 00:47 |
| anteaya | should I be in the branch with 09 in it or master? | 00:48 |
| anteaya | or will git review -d take me to a new branch with 506 anyway | 00:48 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Removes mailing list third-party-request https://review.openstack.org/118508 | 00:50 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Creates third-party-requests mailing list https://review.openstack.org/118509 | 00:50 |
| jeblair | anteaya: new branch with 506 | 00:51 |
| anteaya | kk, well they are all together then | 00:51 |
| anteaya | I had wanted 06 at the end but it is at the front | 00:51 |
| anteaya | oh well | 00:51 |
| *** gokrokve_ has quit IRC | 00:51 | |
| anteaya | does't matter | 00:51 |
| anteaya | I'll know for next tiem | 00:51 |
| jeblair | oh, heh. you can do -d 08, -x 09, -x 06. | 00:52 |
| anteaya | okay | 00:52 |
| anteaya | this is new: http://paste.openstack.org/show/104962/ | 00:54 |
| anteaya | git review blew up on me | 00:54 |
| anteaya | my choices are git commit --allow-empty or git reset | 00:55 |
| *** mjturek has joined #openstack-infra | 00:56 | |
| jeblair | anteaya: oh, it's probably because starting with 08 meant it also pulled in 06 since it already depended on it.. | 00:59 |
| anteaya | I think so | 00:59 |
| anteaya | reading about git reset options | 00:59 |
| *** terryw has quit IRC | 00:59 | |
| *** r-daneel has quit IRC | 01:00 | |
| *** pcm_ has quit IRC | 01:00 | |
| jeblair | anteaya: git reset; git checkout master; git checkout -b somenewbranch | 01:00 |
| anteaya | kk | 01:00 |
| jeblair | anteaya: then do the git review series again, except use -x for all three | 01:00 |
| anteaya | ah okay | 01:00 |
| jeblair | anteaya: (that way you are cherry-picking each change in sequence, starting with master (so it won't pull in any of the other changes in the current sequence until you tell it to)) | 01:01 |
| *** reed has quit IRC | 01:01 | |
| *** marcusvrn has quit IRC | 01:01 | |
| anteaya | ah ha | 01:01 |
| *** mjturek has quit IRC | 01:01 | |
| *** reed has joined #openstack-infra | 01:03 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Removes mailing list third-party-request https://review.openstack.org/118508 | 01:03 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Creates third-party-requests mailing list https://review.openstack.org/118509 | 01:03 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Edit the name of the third-party-requests ml https://review.openstack.org/118506 | 01:03 |
| *** mmaglana has quit IRC | 01:03 | |
| *** erlon has joined #openstack-infra | 01:04 | |
| anteaya | yay thats the order I wanted, thanks jeblair | 01:04 |
| jeblair | np | 01:04 |
| *** gyee has quit IRC | 01:04 | |
| *** lnxnut has joined #openstack-infra | 01:06 | |
| *** yaguang has joined #openstack-infra | 01:07 | |
| *** lcheng has quit IRC | 01:07 | |
| *** dims has joined #openstack-infra | 01:11 | |
| *** laca has joined #openstack-infra | 01:11 | |
| *** otter768 has joined #openstack-infra | 01:11 | |
| *** mwagner_lap has joined #openstack-infra | 01:14 | |
| *** lnxnut has quit IRC | 01:16 | |
| *** gokrokve has joined #openstack-infra | 01:17 | |
| phschwartz | evening infra from Disney. | 01:17 |
| *** bknudson has joined #openstack-infra | 01:18 | |
| dims | phschwartz: nice :) | 01:21 |
| phschwartz | :) trying to get some work in while on a family vacation. lol | 01:21 |
| clarkb | does the family know? | 01:21 |
| *** dims has quit IRC | 01:23 | |
| *** dims has joined #openstack-infra | 01:23 | |
| *** marcoemorais has quit IRC | 01:23 | |
| phschwartz | Yes, Wife is watching TV in bed next to me | 01:23 |
| *** marcoemorais has joined #openstack-infra | 01:24 | |
| * nibalizer casually wonders if disney has an openstack | 01:24 | |
| clarkb | nibalizer: they run swift | 01:25 |
| phschwartz | nibalizer: In ATL I talked to a few of their engineers. The run a Nova setup and a Swift setup. | 01:26 |
| *** dims_ has joined #openstack-infra | 01:26 | |
| *** zz_dimtruck is now known as dimtruck | 01:26 | |
| dims_ | whoa! loooooooong gate | 01:27 |
| *** dims has quit IRC | 01:28 | |
| *** mestery has quit IRC | 01:31 | |
| *** mestery has joined #openstack-infra | 01:31 | |
| *** sballe has quit IRC | 01:32 | |
| *** mestery_ has joined #openstack-infra | 01:35 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/config: Move os-{apply,collect}-config to python3-jobs https://review.openstack.org/118073 | 01:35 |
| *** trinaths has quit IRC | 01:35 | |
| openstackgerrit | Jia Dong proposed a change to openstack-infra/config: Add Tricircle project https://review.openstack.org/117939 | 01:36 |
| *** camunoz has joined #openstack-infra | 01:36 | |
| lifeless | clarkb: re https://review.openstack.org/#/c/118069/ | 01:36 |
| *** marcoemorais has quit IRC | 01:36 | |
| lifeless | clarkb: what other subunit logs? | 01:36 |
| clarkb | lifeless: the ones we grab for the unittests, I think they have the same issue | 01:36 |
| clarkb | lifeless: I can write up the fix for them though | 01:36 |
| *** camunoz has quit IRC | 01:37 | |
| *** mestery has quit IRC | 01:37 | |
| nibalizer | phschwartz: clarkb sweet | 01:37 |
| lifeless | clarkb: I thought that was the one I patches | 01:38 |
| lifeless | patched | 01:38 |
| lifeless | clarkb: oh, no I see - yes please do | 01:38 |
| lifeless | clarkb: I'm || close to turning on subunit-v2 for the storage backend | 01:38 |
| lifeless | so we'll really want that changed at that point :) | 01:39 |
| *** Ryan_Lane has quit IRC | 01:42 | |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Fix crash on dependency update https://review.openstack.org/118516 | 01:42 |
| homeless | /var/lib/git | 01:43 |
| jeblair | tricircle sounds like tripleo | 01:43 |
| jeblair | not so much the project, but the name :) | 01:44 |
| lifeless | three pleo's ? | 01:44 |
| jeblair | beep boop bleep | 01:44 |
| *** stevemar has joined #openstack-infra | 01:46 | |
| anteaya | jeblair: that's what I thought | 01:46 |
| anteaya | sounds to me like the tried to match tripleo's name as closely as possible | 01:47 |
| *** melwitt has quit IRC | 01:48 | |
| openstackgerrit | Jia Dong proposed a change to openstack-infra/config: Add Tricircle project https://review.openstack.org/117939 | 01:49 |
| jeblair | anteaya: could be complete coincidence. trinities and round shapes show up a lot :) | 01:49 |
| *** Sukhdev has joined #openstack-infra | 01:50 | |
| *** gargola has quit IRC | 01:50 | |
| *** pcrews has quit IRC | 01:50 | |
| anteaya | ah, I had to get them to reduce the use of openstack in their commit message from 3 mentions to 1 | 01:50 |
| anteaya | I'm not feeling coincidence here | 01:50 |
| anteaya | 3 mentions in one sentence | 01:50 |
| anteaya | but lifeless looked and gave me the go-ahead | 01:51 |
| anteaya | so *shrug* I will | 01:51 |
| *** Sukhdev has quit IRC | 01:51 | |
| *** pcrews has joined #openstack-infra | 01:52 | |
| *** nosnos has joined #openstack-infra | 01:52 | |
| *** changbl has joined #openstack-infra | 01:52 | |
| *** imeerovi has joined #openstack-infra | 01:53 | |
| nibalizer | i wish we'd called tripleo ozone | 01:55 |
| nibalizer | that name is so cool | 01:55 |
| anteaya | that would go over well in the trademark filter | 01:55 |
| nibalizer | oh right | 01:56 |
| anteaya | I'm sure noone else has that | 01:56 |
| nibalizer | nope | 01:56 |
| nibalizer | :D | 01:56 |
| anteaya | :D | 01:56 |
| nibalizer | its all mine and your anita | 01:56 |
| nibalizer | we can escape with it | 01:56 |
| anteaya | though I like it as well | 01:56 |
| anteaya | deal | 01:56 |
| nibalizer | and hold the foundation hostage for millions! | 01:56 |
| anteaya | no no | 01:56 |
| anteaya | if i am escape there are horses and beaches | 01:56 |
| nibalizer | tru dat | 01:56 |
| anteaya | and mountains will teepees owls and coyotes | 01:56 |
| anteaya | s/will/with | 01:57 |
| anteaya | and boats | 01:57 |
| anteaya | no hostages | 01:57 |
| anteaya | they just get in the way | 01:57 |
| nibalizer | haha, deal | 01:57 |
| *** JadnG has joined #openstack-infra | 01:57 | |
| * anteaya takes nibalizer's hand and skips away | 01:57 | |
| * nibalizer turns on happy-pop music | 01:57 | |
| anteaya | ha ha ha | 01:57 |
| *** imeerovi has quit IRC | 01:58 | |
| *** baohua has joined #openstack-infra | 02:00 | |
| jeblair | i switch desktops for 5 minutes and this happens :) | 02:00 |
| jeblair | i should switch desktops more often | 02:00 |
| pleia2 | hehe | 02:00 |
| anteaya | ha ha ha | 02:00 |
| *** yongli has quit IRC | 02:02 | |
| JadnG | Hello. Help please. Just create a stackforge project Tricircle. Can anyone review and approve that request. We'd like to contribute source code ASAP. Thanks | 02:02 |
| *** dims_ has quit IRC | 02:03 | |
| JadnG | The link https://review.openstack.org/#/c/117939/ | 02:03 |
| *** yongli has joined #openstack-infra | 02:03 | |
| *** dims has joined #openstack-infra | 02:03 | |
| anteaya | hi JadnG we are reviewing your patch as fast as we are able | 02:04 |
| anteaya | JadnG: this is a very busy week for us in our release cycle | 02:04 |
| JadnG | Thank you very much. It's great helpful | 02:04 |
| anteaya | JadnG: https://wiki.openstack.org/wiki/Juno_Release_Schedule | 02:04 |
| *** baohua has quit IRC | 02:04 | |
| anteaya | JadnG: great, you are quick at offering new patches, that helps | 02:05 |
| *** zhiwei has joined #openstack-infra | 02:05 | |
| JadnG | We also want the tricircle source code being a candidate part of Kilo. Need discussion in the summit. | 02:05 |
| JadnG | Thank you all a lot. | 02:06 |
| *** baohua has joined #openstack-infra | 02:06 | |
| *** adalbas has quit IRC | 02:07 | |
| anteaya | JadnG: ah well you are a stackforge project, if you want to be considered for incubation in kilo you need to start following the tc meetings | 02:07 |
| *** dims has quit IRC | 02:08 | |
| zhiwei | Hi all, I encountered an jenkins issue in my commit, any one can help? | 02:08 |
| anteaya | pleia2: kevin mitchell might be a good person to get some input on style guidelines: https://review.openstack.org/#/c/116708/3/README.rst | 02:08 |
| anteaya | zhiwei: we don't know yet | 02:08 |
| anteaya | zhiwei: can you post the url of the commit? | 02:08 |
| zhiwei | anteaya: http://logs.openstack.org/07/114407/6/check/gate-cookbook-openstack-common-chef-lint/dad68c4/console.html | 02:08 |
| zhiwei | ok, thanks. | 02:08 |
| anteaya | JadnG: start here: https://wiki.openstack.org/wiki/Governance/TechnicalCommittee | 02:09 |
| *** mestery_ is now known as mestery | 02:09 | |
| zhiwei | the buff-ruby_engine gem need ruby version >=1.9.2 | 02:09 |
| zhiwei | but the jenkins server has 1.8 | 02:09 |
| pleia2 | anteaya: ooh, perhaps! | 02:09 |
| anteaya | zhiwei: your test ran on precise | 02:10 |
| *** salv-orlando has quit IRC | 02:10 | |
| anteaya | zhiwei: update your test node to run on trusty | 02:10 |
| zhiwei | 14.04 | 02:10 |
| anteaya | Building remotely on bare-precise-hpcloud-b1-1856260 in workspace /home/jenkins/workspace/gate-cookbook-openstack-common-chef-lint | 02:11 |
| anteaya | unless we are running trusty on nodes called bare-precise? | 02:11 |
| anteaya | zhiwei: it needs to run on 14.04 but it didn't | 02:11 |
| zhiwei | anteaya: please look this url: http://logs.openstack.org/58/118458/1/check/gate-cookbook-openstack-block-storage-chef-lint/409d51f/console.html | 02:12 |
| anteaya | zhiwei: you need to change the test node information | 02:12 |
| zhiwei | I think the jenkins server has multi ruby version, but I don't why in my commit, it uses the wrong version. | 02:12 |
| anteaya | uilding remotely on bare-trusty-hpcloud-b2-1858453 in workspace /home/jenkins/workspace/gate-cookbook-openstack-block-storage-chef-lint | 02:12 |
| anteaya | different test | 02:13 |
| anteaya | that one built on trusty | 02:13 |
| anteaya | the jenkins server has multiple test nodes | 02:13 |
| anteaya | and runs tests on the node type your project specifies | 02:13 |
| *** dims has joined #openstack-infra | 02:13 | |
| *** gokrokve_ has joined #openstack-infra | 02:14 | |
| *** homeless has quit IRC | 02:14 | |
| zhiwei | sorry, I can understand, can you detail it? | 02:14 |
| pleia2 | if your tests need to run on trusty, you need to specify that in the test, right now it just picks any available ubuntu node (precise or trusty) | 02:15 |
| anteaya | what is the name of this project? | 02:15 |
| *** tgohad has quit IRC | 02:15 | |
| anteaya | cookbook-openstack-common | 02:15 |
| anteaya | what a name | 02:15 |
| *** nosnos has quit IRC | 02:16 | |
| anteaya | zhiwei: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml#n2129 | 02:16 |
| *** gokrokve_ has quit IRC | 02:16 | |
| anteaya | remove bare-precise || from that file | 02:16 |
| zhiwei | ok, thanks. | 02:16 |
| pleia2 | and I'd confirm with the team that is what is desired | 02:16 |
| anteaya | godspeed | 02:16 |
| pleia2 | if it should run on precise, the code should be altered, not the test | 02:17 |
| anteaya | zhiwei: oh yes, offer the patch and get it reviewed | 02:17 |
| *** gokrokve has quit IRC | 02:18 | |
| *** stevemar has quit IRC | 02:18 | |
| zhiwei | pleia2: yes, I think I need to explore it why the code can't work on precise. | 02:19 |
| *** stevemar has joined #openstack-infra | 02:19 | |
| anteaya | because precise doesn't have the version of ruby one of your gems needs | 02:19 |
| *** chuckC has joined #openstack-infra | 02:19 | |
| anteaya | so re-evaluate if you really need that gem, or that version of the gem | 02:20 |
| *** yamamoto has quit IRC | 02:20 | |
| zhiwei | this gem is inside in another gem, and I don't provider in the project Gemfile. | 02:20 |
| *** gokrokve has joined #openstack-infra | 02:21 | |
| anteaya | zhiwei: that is a choice your team has to make | 02:22 |
| *** bdpayne has joined #openstack-infra | 02:22 | |
| anteaya | zhiwei: we can't make that choice | 02:22 |
| zhiwei | ok, I will discuss this with our team later. | 02:23 |
| *** tomoe_ has quit IRC | 02:24 | |
| *** gokrokve has quit IRC | 02:25 | |
| anteaya | hands up anyone who still thinks tricircle is a coincidence | 02:27 |
| *** imeerovi has joined #openstack-infra | 02:27 | |
| *** pcrews has quit IRC | 02:28 | |
| zhiwei | anteaya: please check this url: http://logs.openstack.org/07/114407/2/check/gate-cookbook-openstack-common-chef-lint/8efe481/console.html.gz | 02:28 |
| zhiwei | they have almost the same code, but the result is not same. | 02:28 |
| zhiwei | https://review.openstack.org/#/c/114407/ the patch set1 and patch set 6 has minor changes. But the jenkins job failed in patch set 6. | 02:30 |
| *** baohua has quit IRC | 02:31 | |
| *** baohua has joined #openstack-infra | 02:32 | |
| anteaya | An error occurred while installing buff-ruby_engine (0.1.0), and Bundler cannot | 02:32 |
| anteaya | if the mirror you were downloading your gem from had a problem during the download, then the gem couldn't be downloaded | 02:33 |
| anteaya | openstack doesn't have gem mirrors | 02:33 |
| anteaya | so you must be installing from rubygems or somewhere | 02:33 |
| anteaya | so recheck the patch and run the job again | 02:34 |
| anteaya | okay I'm tired | 02:35 |
| anteaya | done for the night | 02:35 |
| zhiwei | ok, thanks. | 02:36 |
| *** amcrn has quit IRC | 02:38 | |
| *** mmaglana has joined #openstack-infra | 02:38 | |
| *** imeerovi has quit IRC | 02:38 | |
| *** david-lyle has joined #openstack-infra | 02:39 | |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Query projects in batches https://review.openstack.org/118519 | 02:41 |
| jeblair | lifeless: if you have a moment, i'd appreciate your thoughts on 105032 and 118519 | 02:41 |
| *** mriedem has quit IRC | 02:42 | |
| lifeless | looking | 02:48 |
| anteaya | oh forgot to say, I am dentisting off the top of the day | 02:49 |
| anteaya | should be back in the afternoon some time | 02:49 |
| *** imeerovi has joined #openstack-infra | 02:49 | |
| *** harlowja is now known as harlowja_away | 02:52 | |
| *** jyuso has joined #openstack-infra | 02:53 | |
| lifeless | jeblair: I had trouble making multiple q= work | 02:53 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/subunit2sql: Add subunit2sql-db-manage utility https://review.openstack.org/118418 | 02:54 |
| lifeless | jeblair: I'm fine with groups of 10, though its likely many gertty users will have more than 10 subscriptions | 02:54 |
| *** yamamoto has joined #openstack-infra | 02:55 | |
| lifeless | jeblair: reviewed | 02:56 |
| *** imeerovi has quit IRC | 02:56 | |
| lifeless | jeblair: I think its hard to choose between them - there will be cases where each is better | 02:56 |
| *** dims has quit IRC | 02:56 | |
| *** daya_k has joined #openstack-infra | 02:56 | |
| lifeless | jeblair: I think I found a logic bug in the batch-of-10 patch | 02:56 |
| *** dims has joined #openstack-infra | 02:57 | |
| jeblair | lifeless: oh yep, you're right about that, thanks | 02:58 |
| lifeless | jeblair: I suspect you're right about only getting the first 500 changes today | 02:59 |
| lifeless | jeblair: with my patch, but thats clearly fixable | 02:59 |
| lifeless | jeblair: I'm pondering whether we should have both optimisations and a heuristic to choose | 02:59 |
| jeblair | lifeless: the 500 limit was true even before your patch; i fixed it for the base case earlier in the series (but you can see it there) | 03:00 |
| lifeless | yah | 03:00 |
| jeblair | lifeless: i expect that you mostly didn't notice it even with your patch since the deltas between queries are are somewhat likely to be < 500 changes | 03:00 |
| jeblair | (depending on how often gertty is not running) | 03:00 |
| *** KanagarajM has joined #openstack-infra | 03:01 | |
| jeblair | or at least, the impact may have been minimal to moderate | 03:01 |
| lifeless | jeblair: I suspect it was in the bucket of 'thats odd, I should look into it' that I had in fact not looked into | 03:01 |
| jeblair | though robbie's test would have definitely been affected | 03:01 |
| *** dims has quit IRC | 03:01 | |
| jeblair | because syncing many projects at once from 0 should be thousands of changes | 03:01 |
| jeblair | (nova currently has 650 open, neutron 527) | 03:02 |
| jeblair | (nova is down from >700 this weekend when i was looking into this) | 03:02 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Query projects in batches https://review.openstack.org/118519 | 03:05 |
| *** hdd has quit IRC | 03:07 | |
| *** erlon has quit IRC | 03:10 | |
| clarkb | fyi if we extrapolate from https://github.com/openstack/openstack/graphs/commit-activity we have already enqueued the last chagne that will get into oepnstack prior to feature freeze | 03:18 |
| *** imeerovi has joined #openstack-infra | 03:18 | |
| clarkb | however zuul only reports a 14 hour time delta | 03:19 |
| clarkb | I assume failed commits account for that? | 03:19 |
| *** praneshp has quit IRC | 03:20 | |
| *** otter768 has quit IRC | 03:21 | |
| *** zz_naotok is now known as naotok | 03:22 | |
| *** imeerovi has quit IRC | 03:23 | |
| *** praneshp has joined #openstack-infra | 03:23 | |
| *** zhiwei has quit IRC | 03:26 | |
| *** dimtruck is now known as zz_dimtruck | 03:26 | |
| *** koolhead17 has joined #openstack-infra | 03:29 | |
| *** koolhead17 has joined #openstack-infra | 03:29 | |
| *** imeerovi has joined #openstack-infra | 03:31 | |
| *** bhuvan has joined #openstack-infra | 03:37 | |
| *** lttrl has quit IRC | 03:37 | |
| *** praneshp has quit IRC | 03:38 | |
| *** imeerovi has quit IRC | 03:39 | |
| *** lttrl has joined #openstack-infra | 03:39 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:39 | |
| *** praneshp has joined #openstack-infra | 03:40 | |
| jyuso | Hi folks,I have a question:I use Gerrit trigger(not Zuul) as Jenkins trigger.If I want to generate a "testr_result.html" file after CI testing,what should I do? | 03:42 |
| *** praneshp_ has joined #openstack-infra | 03:45 | |
| *** praneshp has quit IRC | 03:45 | |
| *** praneshp_ is now known as praneshp | 03:45 | |
| *** radez is now known as radez_g0n3 | 03:46 | |
| *** lcheng has joined #openstack-infra | 03:46 | |
| *** Sukhdev has joined #openstack-infra | 03:52 | |
| *** bdpayne has quit IRC | 03:53 | |
| *** bdpayne has joined #openstack-infra | 03:54 | |
| *** melwitt has joined #openstack-infra | 04:06 | |
| *** mbacchi has quit IRC | 04:07 | |
| *** garyh has quit IRC | 04:09 | |
| *** melwitt1 has joined #openstack-infra | 04:13 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Fix pep8 issues https://review.openstack.org/115187 | 04:13 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/jenkins-job-builder: Start libification of jjb https://review.openstack.org/107724 | 04:14 |
| *** melwitt has quit IRC | 04:16 | |
| *** changbl has quit IRC | 04:24 | |
| *** zz_dimtruck is now known as dimtruck | 04:32 | |
| *** KanagarajM has quit IRC | 04:34 | |
| *** ianw has quit IRC | 04:37 | |
| *** ianw has joined #openstack-infra | 04:37 | |
| *** garyh has joined #openstack-infra | 04:40 | |
| *** bdpayne has quit IRC | 04:45 | |
| *** nosnos has joined #openstack-infra | 04:50 | |
| Alex_Gaynor | Is something broken with zuul? status. doesn't seem to be updating, the requests are just hanging | 04:53 |
| jogo | Alex_Gaynor: status.o.o/zuul is updating for me | 04:55 |
| *** amotoki has joined #openstack-infra | 04:56 | |
| * jogo snipes 105231 | 04:56 | |
| Alex_Gaynor | seems to be workign now, I suppose it's all in my head | 04:56 |
| jogo | Alex_Gaynor: once you mentioned it, it was in my head too | 04:56 |
| jogo | hmm I wonder if there is an automatic way to sort out when something is actually broken | 04:57 |
| jogo | by something I mean unit tests/pep8 failing | 04:57 |
| *** harlowja_at_home has joined #openstack-infra | 04:59 | |
| *** ppai has joined #openstack-infra | 04:59 | |
| *** yfried_ has quit IRC | 05:00 | |
| *** dimtruck is now known as zz_dimtruck | 05:00 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add zaqar to specs.openstack.org https://review.openstack.org/116310 | 05:01 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Fix pep8 issues https://review.openstack.org/115187 | 05:03 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Fix up status page dependencies https://review.openstack.org/98028 | 05:04 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove resolved fingerprints https://review.openstack.org/117916 | 05:05 |
| stevemar | zuul is getting wrecked right now :( | 05:10 |
| stevemar | 140 in the gate | 05:11 |
| *** chandankumar has joined #openstack-infra | 05:13 | |
| *** enikanorov__ has quit IRC | 05:23 | |
| *** bdpayne has joined #openstack-infra | 05:24 | |
| *** ildikov has quit IRC | 05:28 | |
| *** tsg has joined #openstack-infra | 05:32 | |
| *** cipcosma has joined #openstack-infra | 05:34 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add gerrit reviews into patchset approvals https://review.openstack.org/97390 | 05:37 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Allow a pipeline to specify alternative gerrit acc https://review.openstack.org/97391 | 05:37 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support for negative requirements https://review.openstack.org/102726 | 05:38 |
| *** rlandy has joined #openstack-infra | 05:44 | |
| *** Sukhdev has quit IRC | 05:52 | |
| *** harlowja_at_home has quit IRC | 05:55 | |
| *** Ryan_Lane has joined #openstack-infra | 06:01 | |
| *** bdpayne has quit IRC | 06:02 | |
| *** Ryan_Lane has quit IRC | 06:03 | |
| *** baoli has quit IRC | 06:08 | |
| *** imeerovi has joined #openstack-infra | 06:12 | |
| *** denis_makogon has quit IRC | 06:12 | |
| *** denis_makogon has joined #openstack-infra | 06:12 | |
| *** stevemar has quit IRC | 06:16 | |
| *** skolekonov has joined #openstack-infra | 06:16 | |
| *** loki184 has joined #openstack-infra | 06:17 | |
| *** nelsnelson has quit IRC | 06:17 | |
| *** k4n0 has joined #openstack-infra | 06:18 | |
| *** imeerovi has quit IRC | 06:18 | |
| *** afazekas has joined #openstack-infra | 06:18 | |
| *** david-lyle has quit IRC | 06:19 | |
| *** harlowja_at_home has joined #openstack-infra | 06:20 | |
| *** flaper87|afk is now known as flaper87 | 06:30 | |
| *** liam_ has joined #openstack-infra | 06:30 | |
| *** liam_ is now known as Guest81233 | 06:31 | |
| *** andreykurilin_ has joined #openstack-infra | 06:31 | |
| *** HeOS has quit IRC | 06:32 | |
| *** Guest81233 has quit IRC | 06:32 | |
| *** yfried_ has joined #openstack-infra | 06:34 | |
| *** luqas has joined #openstack-infra | 06:41 | |
| *** jcoufal has joined #openstack-infra | 06:43 | |
| * gus just had jenkins fail at "git remote update" - "Could not read from remote repository" | 06:46 | |
| gus | ^ http://logs.openstack.org/86/115486/8/check/gate-neutron-pep8/b480208/console.html | 06:47 |
| AJaeger | gus, just recheck... | 06:55 |
| AJaeger | fungi, could you remove glance_store from jenkins so that the propose translation update works again, please? See https://jenkins.openstack.org/job/glance_store-propose-translation-update/4/console | 06:56 |
| gus | yep - I only did a brief bug search but couldn't see an existing e-r bug. Since it seemed to hit many of my test runs at about the same time, I imagine others have seen it too. | 06:56 |
| *** ildikov has joined #openstack-infra | 06:57 | |
| *** MaxV has joined #openstack-infra | 07:00 | |
| *** doude has joined #openstack-infra | 07:04 | |
| *** chandankumar has quit IRC | 07:07 | |
| *** MaxV has quit IRC | 07:09 | |
| *** ominakov has quit IRC | 07:09 | |
| *** chandan_kumar has joined #openstack-infra | 07:13 | |
| *** lcheng has quit IRC | 07:19 | |
| *** e0ne has joined #openstack-infra | 07:19 | |
| *** rushiagr has joined #openstack-infra | 07:20 | |
| rushiagr | I was trying to create stories into storyboard, but it is not allowing me to | 07:21 |
| rushiagr | the 'save changes' button is not clickable. Is this deliberately done? | 07:21 |
| *** ashaeron has joined #openstack-infra | 07:21 | |
| rushiagr | I found few issues with the storyboard webui, so was trying to report them | 07:22 |
| ppai | anteaya, could you relook at https://review.openstack.org/115250 | 07:25 |
| *** flaper87 is now known as flaper87|afk | 07:26 | |
| ppai | SergeyLukjanov, could you relook at https://review.openstack.org/115250 thanks :) | 07:26 |
| *** Hal has joined #openstack-infra | 07:26 | |
| *** Hal is now known as Guest10457 | 07:27 | |
| openstackgerrit | Carl Loa Odin proposed a change to openstack-infra/jenkins-job-builder: Extend trigger_builds functionality https://review.openstack.org/91051 | 07:28 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Add oslo.utils and oslo.concurrency https://review.openstack.org/117307 | 07:28 |
| *** shardy_afk is now known as shardy | 07:29 | |
| *** lyxus has quit IRC | 07:29 | |
| *** lyxus has joined #openstack-infra | 07:31 | |
| *** yfried_ has quit IRC | 07:32 | |
| *** harlowja_at_home has quit IRC | 07:33 | |
| *** chandan_kumar has quit IRC | 07:33 | |
| openstackgerrit | A change was merged to openstack/requirements: Remove docutils-pin https://review.openstack.org/117602 | 07:34 |
| *** dizquierdo has joined #openstack-infra | 07:35 | |
| *** yfried has joined #openstack-infra | 07:35 | |
| vponomaryov | fungi: just read thread, I am available for open question with moving Manila to openstack namespace. | 07:36 |
| *** wendar_ is now known as wendar | 07:36 | |
| *** e0ne has quit IRC | 07:38 | |
| *** e0ne has joined #openstack-infra | 07:38 | |
| *** acruz has joined #openstack-infra | 07:38 | |
| *** flaper87|afk is now known as flaper87 | 07:39 | |
| *** marun_ has quit IRC | 07:41 | |
| *** andreykurilin_ has quit IRC | 07:41 | |
| *** tsg has quit IRC | 07:42 | |
| *** jlibosva has joined #openstack-infra | 07:43 | |
| *** alexpilotti has joined #openstack-infra | 07:43 | |
| *** lcheng has joined #openstack-infra | 07:45 | |
| *** HeOS has joined #openstack-infra | 07:46 | |
| *** alexpilotti has quit IRC | 07:46 | |
| ttx | lifeless: I replied to your thread. I'm not sure we actually have a problem here | 07:47 |
| *** chandan_kumar has joined #openstack-infra | 07:47 | |
| *** mancdaz has joined #openstack-infra | 07:47 | |
| *** nuritv has joined #openstack-infra | 07:50 | |
| nuritv | Hi There, i have a Question - we have a CI system working on Neutron, and we want to create one for Cinder as well. do we need to register with a different account or can we use our existing account? | 07:52 |
| openstackgerrit | A change was merged to openstack/requirements: Bump boto to 2.32.1 https://review.openstack.org/118160 | 07:52 |
| *** jistr has joined #openstack-infra | 07:53 | |
| *** jgallard has joined #openstack-infra | 07:53 | |
| *** alexpilotti has joined #openstack-infra | 07:54 | |
| *** MaxV has joined #openstack-infra | 07:55 | |
| *** imeerovi has joined #openstack-infra | 07:57 | |
| *** alexpilotti has quit IRC | 08:03 | |
| *** lcheng has quit IRC | 08:06 | |
| *** nuritv has quit IRC | 08:09 | |
| *** mattymo is now known as mmosesohn | 08:11 | |
| *** mmosesohn is now known as mattymo | 08:12 | |
| lifeless | ttx: ack | 08:14 |
| *** nuritv has joined #openstack-infra | 08:15 | |
| *** sdake has quit IRC | 08:16 | |
| *** alexpilotti has joined #openstack-infra | 08:18 | |
| *** ZZelle has quit IRC | 08:18 | |
| *** ZZelle has joined #openstack-infra | 08:19 | |
| *** sdake has joined #openstack-infra | 08:20 | |
| *** melwitt1 has quit IRC | 08:24 | |
| *** derekh has joined #openstack-infra | 08:25 | |
| *** oomichi has quit IRC | 08:25 | |
| *** lttrl has quit IRC | 08:26 | |
| *** luqas has quit IRC | 08:28 | |
| *** chandan_kumar has quit IRC | 08:29 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:30 | |
| *** salv-orlando has joined #openstack-infra | 08:36 | |
| *** alexpilotti has quit IRC | 08:37 | |
| *** pblaho_ has joined #openstack-infra | 08:38 | |
| *** Longgeek has joined #openstack-infra | 08:39 | |
| *** andreykurilin_ has joined #openstack-infra | 08:42 | |
| *** koolhead17 has quit IRC | 08:42 | |
| *** lttrl has joined #openstack-infra | 08:44 | |
| *** ildikov_ has joined #openstack-infra | 08:44 | |
| *** tnovacik has joined #openstack-infra | 08:45 | |
| *** ildikov has quit IRC | 08:47 | |
| *** amuller has joined #openstack-infra | 08:50 | |
| *** mpaolino has joined #openstack-infra | 08:51 | |
| *** amotoki has quit IRC | 08:52 | |
| *** Longgeek has quit IRC | 08:53 | |
| *** nijaba has joined #openstack-infra | 08:53 | |
| *** Longgeek has joined #openstack-infra | 08:53 | |
| *** andreaf has joined #openstack-infra | 08:53 | |
| *** praneshp has quit IRC | 08:57 | |
| *** masayukig has quit IRC | 09:00 | |
| *** masayukig has joined #openstack-infra | 09:00 | |
| *** BobBall_Away is now known as BobBall | 09:01 | |
| *** jp_at_hp has joined #openstack-infra | 09:03 | |
| *** jp_at_hp has quit IRC | 09:03 | |
| *** jp_at_hp has joined #openstack-infra | 09:03 | |
| *** luqas has joined #openstack-infra | 09:04 | |
| *** nuritv has quit IRC | 09:05 | |
| *** andreykurilin_ has quit IRC | 09:06 | |
| *** jistr has quit IRC | 09:06 | |
| *** nuritv has joined #openstack-infra | 09:07 | |
| *** dtantsur|afk is now known as dtantsur | 09:07 | |
| *** yamamoto has quit IRC | 09:08 | |
| *** jistr has joined #openstack-infra | 09:10 | |
| *** andreykurilin_ has joined #openstack-infra | 09:10 | |
| *** andreykurilin_ has quit IRC | 09:15 | |
| *** andreykurilin_ has joined #openstack-infra | 09:18 | |
| *** alexpilotti has joined #openstack-infra | 09:21 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/config: Typo: replace puppet.rsh by puppet.rst https://review.openstack.org/118113 | 09:23 |
| *** Longgeek has quit IRC | 09:24 | |
| *** alexpilotti has quit IRC | 09:26 | |
| *** Longgeek has joined #openstack-infra | 09:27 | |
| *** pblaho_ is now known as pblaho|afk | 09:32 | |
| *** e0ne has quit IRC | 09:35 | |
| *** marun_ has joined #openstack-infra | 09:39 | |
| *** andreykurilin_ has quit IRC | 09:39 | |
| *** ociuhandu has quit IRC | 09:40 | |
| *** tnovacik has quit IRC | 09:41 | |
| *** e0ne has joined #openstack-infra | 09:41 | |
| *** yamamoto has joined #openstack-infra | 09:42 | |
| *** nijaba has quit IRC | 09:46 | |
| *** nijaba has joined #openstack-infra | 09:46 | |
| *** amuller is now known as amuller_lunch | 09:49 | |
| *** zhiwei has joined #openstack-infra | 09:56 | |
| *** unicell has quit IRC | 10:00 | |
| *** _shaps__ is now known as _shaps_ | 10:04 | |
| *** pcm_ has joined #openstack-infra | 10:05 | |
| *** pcm_ has quit IRC | 10:05 | |
| *** pcm_ has joined #openstack-infra | 10:06 | |
| *** jgallard has quit IRC | 10:08 | |
| *** yamahata has quit IRC | 10:11 | |
| *** nijaba has quit IRC | 10:12 | |
| *** dizquierdo has quit IRC | 10:16 | |
| *** pblaho|afk is now known as pblaho_ | 10:20 | |
| *** koolhead17 has joined #openstack-infra | 10:20 | |
| *** saper has joined #openstack-infra | 10:22 | |
| *** e0ne has quit IRC | 10:24 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/config: Add pre-requisites needed for diskimage-builder https://review.openstack.org/102583 | 10:25 |
| *** bhuvan has quit IRC | 10:25 | |
| *** imeerovi has quit IRC | 10:30 | |
| *** amuller_lunch is now known as amuller | 10:31 | |
| *** Longgeek has quit IRC | 10:31 | |
| *** Longgeek has joined #openstack-infra | 10:32 | |
| SergeyLukjanov | anteaya, what's the correct specific recheck syntax for 3rd party? | 10:35 |
| *** Longgeek_ has joined #openstack-infra | 10:36 | |
| *** saper has joined #openstack-infra | 10:36 | |
| *** masayukig has quit IRC | 10:39 | |
| *** Longgeek has quit IRC | 10:39 | |
| *** e0ne has joined #openstack-infra | 10:39 | |
| AJaeger | FYI, last requirements post job failed due to tuskar-ui docutils pin, a patch for tuskar-ui is merged, so next requirements post job should be fine... | 10:40 |
| *** unicell has joined #openstack-infra | 10:40 | |
| *** naotok is now known as zz_naotok | 10:41 | |
| *** imeerovi has joined #openstack-infra | 10:42 | |
| *** ociuhandu has joined #openstack-infra | 10:42 | |
| *** yaguang has quit IRC | 10:44 | |
| *** daya_k has quit IRC | 10:46 | |
| *** Guest10457 has quit IRC | 10:46 | |
| *** pblaho_ is now known as pblaho | 10:49 | |
| *** _nadya_ has joined #openstack-infra | 10:49 | |
| *** bhuvan has joined #openstack-infra | 10:52 | |
| *** bhuvan has quit IRC | 10:56 | |
| *** jraim_ has quit IRC | 11:03 | |
| *** fungi has quit IRC | 11:03 | |
| *** comstud has quit IRC | 11:03 | |
| *** ttx has quit IRC | 11:03 | |
| *** MaxV has quit IRC | 11:04 | |
| *** miqui has quit IRC | 11:05 | |
| *** MaxV has joined #openstack-infra | 11:05 | |
| *** jraim__ has joined #openstack-infra | 11:06 | |
| *** dims has joined #openstack-infra | 11:06 | |
| *** dolphm has quit IRC | 11:06 | |
| *** ppai has quit IRC | 11:06 | |
| *** jyuso has quit IRC | 11:08 | |
| *** comstud has joined #openstack-infra | 11:08 | |
| *** ttx has joined #openstack-infra | 11:09 | |
| *** ttx has quit IRC | 11:09 | |
| *** ttx has joined #openstack-infra | 11:09 | |
| *** MaxV has quit IRC | 11:10 | |
| *** loki184 has quit IRC | 11:10 | |
| *** marun_ has quit IRC | 11:10 | |
| *** marun has joined #openstack-infra | 11:10 | |
| *** dolphm has joined #openstack-infra | 11:11 | |
| *** dims has quit IRC | 11:12 | |
| *** dims has joined #openstack-infra | 11:13 | |
| *** dims_ has joined #openstack-infra | 11:14 | |
| *** baoli has joined #openstack-infra | 11:16 | |
| *** fungi has joined #openstack-infra | 11:16 | |
| *** ppai has joined #openstack-infra | 11:17 | |
| *** dims has quit IRC | 11:17 | |
| krtaylor | SergeyLukjanov, right now, "recheck" with or without anything after it, will restart jobs | 11:17 |
| krtaylor | SergeyLukjanov, sdague proposed a standard way of starting recheck for specific third-party systems, but it was abandoned | 11:18 |
| krtaylor | SergeyLukjanov, I plan on reopening that discussion soon | 11:19 |
| *** pblaho is now known as pblaho|afk | 11:21 | |
| *** mpaolino has quit IRC | 11:21 | |
| krtaylor | SergeyLukjanov, see https://review.openstack.org/#/c/109565/ | 11:25 |
| *** asettle has quit IRC | 11:28 | |
| *** asettle has joined #openstack-infra | 11:28 | |
| *** dims_ has quit IRC | 11:29 | |
| *** dims has joined #openstack-infra | 11:29 | |
| *** masayukig has joined #openstack-infra | 11:31 | |
| *** MaxV has joined #openstack-infra | 11:33 | |
| *** dims has quit IRC | 11:34 | |
| *** pelix has joined #openstack-infra | 11:36 | |
| *** dims has joined #openstack-infra | 11:37 | |
| *** mmedvede has joined #openstack-infra | 11:40 | |
| *** mmaglana has quit IRC | 11:41 | |
| *** mmaglana has joined #openstack-infra | 11:42 | |
| *** yamahata has joined #openstack-infra | 11:45 | |
| *** penguinRaider has joined #openstack-infra | 11:46 | |
| *** mmaglana has quit IRC | 11:47 | |
| *** masayukig has quit IRC | 11:48 | |
| *** masayukig has joined #openstack-infra | 11:48 | |
| *** flaviof_zzz is now known as flaviof | 11:49 | |
| *** dims has quit IRC | 11:50 | |
| *** dims has joined #openstack-infra | 11:50 | |
| *** Hal has joined #openstack-infra | 11:50 | |
| *** Hal is now known as Guest64359 | 11:51 | |
| *** ppai has quit IRC | 11:51 | |
| *** dims_ has joined #openstack-infra | 11:51 | |
| *** Longgeek_ has quit IRC | 11:53 | |
| *** Longgeek has joined #openstack-infra | 11:54 | |
| *** dims has quit IRC | 11:54 | |
| *** Longgeek_ has joined #openstack-infra | 11:54 | |
| *** marun_ has joined #openstack-infra | 11:56 | |
| *** baohua_ has joined #openstack-infra | 11:58 | |
| *** yamahata has quit IRC | 11:58 | |
| *** Longgeek has quit IRC | 11:58 | |
| *** marun has quit IRC | 11:59 | |
| *** kgiusti has joined #openstack-infra | 12:00 | |
| *** yamahata has joined #openstack-infra | 12:00 | |
| *** KanagarajM has joined #openstack-infra | 12:00 | |
| *** baohua has quit IRC | 12:00 | |
| *** daya_k has joined #openstack-infra | 12:02 | |
| *** markmcclain has joined #openstack-infra | 12:02 | |
| *** markmcclain has quit IRC | 12:02 | |
| *** dprince has joined #openstack-infra | 12:03 | |
| *** erlon has joined #openstack-infra | 12:03 | |
| *** HenryG is now known as HenryG_afk | 12:04 | |
| *** markmcclain has joined #openstack-infra | 12:04 | |
| *** ppai has joined #openstack-infra | 12:04 | |
| *** mbacchi has joined #openstack-infra | 12:11 | |
| *** pblaho|afk is now known as pblaho | 12:12 | |
| *** mmaglana has joined #openstack-infra | 12:12 | |
| openstackgerrit | Erno Kuvaja proposed a change to openstack/requirements: Limiting requests <2.4.0 https://review.openstack.org/118617 | 12:13 |
| *** mmaglana_ has joined #openstack-infra | 12:14 | |
| *** aysyd has joined #openstack-infra | 12:16 | |
| *** nosnos has quit IRC | 12:16 | |
| *** mmaglana has quit IRC | 12:17 | |
| *** nosnos has joined #openstack-infra | 12:17 | |
| *** Longgeek_ has quit IRC | 12:19 | |
| *** mmaglana_ has quit IRC | 12:19 | |
| *** nosnos has quit IRC | 12:21 | |
| *** adalbas has joined #openstack-infra | 12:21 | |
| *** dkranz has joined #openstack-infra | 12:21 | |
| *** miqui has joined #openstack-infra | 12:25 | |
| *** dims_ has quit IRC | 12:29 | |
| *** baohua_ has quit IRC | 12:30 | |
| *** dims has joined #openstack-infra | 12:30 | |
| *** Longgeek has joined #openstack-infra | 12:30 | |
| *** cnesa10 has quit IRC | 12:30 | |
| *** e0ne has quit IRC | 12:31 | |
| *** dims_ has joined #openstack-infra | 12:32 | |
| *** dims has quit IRC | 12:34 | |
| *** e0ne has joined #openstack-infra | 12:34 | |
| *** KanagarajM has quit IRC | 12:38 | |
| *** koolhead17 has quit IRC | 12:41 | |
| *** freyes has quit IRC | 12:45 | |
| *** dims_ has quit IRC | 12:45 | |
| *** freyes has joined #openstack-infra | 12:46 | |
| *** dims has joined #openstack-infra | 12:46 | |
| *** che-arne has quit IRC | 12:48 | |
| *** nijaba has joined #openstack-infra | 12:52 | |
| *** nijaba has joined #openstack-infra | 12:52 | |
| *** igor has joined #openstack-infra | 12:52 | |
| *** igor has quit IRC | 12:52 | |
| *** igor has joined #openstack-infra | 12:53 | |
| *** jgallard has joined #openstack-infra | 12:53 | |
| *** doug-fish has joined #openstack-infra | 12:53 | |
| openstackgerrit | Kurt Taylor proposed a change to openstack-infra/config: An alternate recheck syntax for third-party CI systems https://review.openstack.org/118623 | 12:53 |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack/requirements: Update oslosphinx to 2.2.0.0a3 https://review.openstack.org/118624 | 12:54 |
| *** marcusvrn has joined #openstack-infra | 12:55 | |
| *** igor has quit IRC | 12:56 | |
| openstackgerrit | Kurt Taylor proposed a change to openstack-infra/config: An alternate recheck syntax for third-party CI systems https://review.openstack.org/118623 | 12:57 |
| *** vponomaryov has quit IRC | 12:57 | |
| *** vponomaryov has joined #openstack-infra | 12:59 | |
| *** dkranz has quit IRC | 12:59 | |
| *** changbl has joined #openstack-infra | 13:00 | |
| *** sandywalsh has joined #openstack-infra | 13:01 | |
| *** enikanorov has joined #openstack-infra | 13:01 | |
| *** jpeeler has joined #openstack-infra | 13:05 | |
| *** zz_dimtruck is now known as dimtruck | 13:07 | |
| *** daya_k has quit IRC | 13:09 | |
| *** _nadya_ has quit IRC | 13:10 | |
| *** doude has quit IRC | 13:11 | |
| *** dkliban_afk is now known as dkliban | 13:12 | |
| *** doude has joined #openstack-infra | 13:12 | |
| *** mmaglana has joined #openstack-infra | 13:13 | |
| *** dkranz has joined #openstack-infra | 13:15 | |
| openstackgerrit | A change was merged to openstack-infra/git-review: Remove useless constants https://review.openstack.org/114756 | 13:17 |
| *** enikanorov__ has joined #openstack-infra | 13:17 | |
| *** eharney has joined #openstack-infra | 13:18 | |
| *** mmaglana has quit IRC | 13:18 | |
| *** enikanorov has quit IRC | 13:19 | |
| *** mriedem has joined #openstack-infra | 13:21 | |
| *** radez_g0n3 is now known as radez | 13:22 | |
| *** craigbr has joined #openstack-infra | 13:24 | |
| *** julim has joined #openstack-infra | 13:24 | |
| *** dangers_away is now known as dangers | 13:25 | |
| *** bknudson has quit IRC | 13:26 | |
| *** mfer has joined #openstack-infra | 13:26 | |
| *** bookwar has joined #openstack-infra | 13:32 | |
| *** doug-fish1 has joined #openstack-infra | 13:36 | |
| *** mjturek has joined #openstack-infra | 13:36 | |
| *** jp_at_hp has quit IRC | 13:37 | |
| *** doug-fish has quit IRC | 13:39 | |
| *** ominakov has joined #openstack-infra | 13:41 | |
| *** e0ne has quit IRC | 13:43 | |
| *** bknudson has joined #openstack-infra | 13:44 | |
| *** ppai has quit IRC | 13:46 | |
| *** r-daneel has joined #openstack-infra | 13:47 | |
| *** e0ne has joined #openstack-infra | 13:50 | |
| *** ashaeron has quit IRC | 13:50 | |
| *** dimtruck is now known as zz_dimtruck | 13:53 | |
| *** homeless has joined #openstack-infra | 13:54 | |
| *** chuckC has quit IRC | 13:55 | |
| *** e0ne has quit IRC | 13:56 | |
| *** cdent has joined #openstack-infra | 13:56 | |
| *** dhellmann has quit IRC | 13:57 | |
| *** dhellmann has joined #openstack-infra | 13:58 | |
| *** e0ne has joined #openstack-infra | 13:58 | |
| fungi | AJaeger: i cleared the workspaces on the proposal slave for glance_store. presumably a change landed to it after we renamed the project but before they merged my .gitreview update | 14:00 |
| *** signed8bit has joined #openstack-infra | 14:02 | |
| *** pcrews has joined #openstack-infra | 14:04 | |
| fungi | vponomaryov: thanks, i was going to go venturing into #openstack-manila to figure out whether bswartz hangs out in irc. mainly we just want to figure out when a convenient time is to rename your program's git repositories from stackforge/.* to openstack/.* | 14:07 |
| fungi | vponomaryov: if someone from the manila core team can be around next tuesday at 19:00 utc in #openstack-meeting (during the infra team meeting) to discuss scheduling that would be a huge help | 14:08 |
| vponomaryov | fungi: I can be there | 14:08 |
| vponomaryov | fungi: only time is open item? | 14:08 |
| vponomaryov | fungi: how long it takes? Whole process of moving? | 14:08 |
| *** yfried has quit IRC | 14:09 | |
| *** zz_jgrimm is now known as jgrimm | 14:09 | |
| *** penguinRaider has quit IRC | 14:09 | |
| *** zz_dimtruck is now known as dimtruck | 14:10 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Look for and process sem-ver pseudo headers in git https://review.openstack.org/108270 | 14:12 |
| *** wenlock has joined #openstack-infra | 14:13 | |
| *** nelsnelson has joined #openstack-infra | 14:14 | |
| zaro | morning | 14:16 |
| *** annegent_ has joined #openstack-infra | 14:17 | |
| *** nuritv has quit IRC | 14:18 | |
| *** paulrad has joined #openstack-infra | 14:21 | |
| *** dmsimard_away is now known as dmsimard | 14:25 | |
| openstackgerrit | Andrey Nikitin proposed a change to openstack-infra/jeepyb: Update mechanism of creating groups in database and managing access rights with jeepyb https://review.openstack.org/117199 | 14:25 |
| marcusvrn | Good morning, just a question....in the CI system page, under requirements section, there's a topic (All comments from your CI system must contain a link to the wiki page for your CI system.). This topic means that the CI system have to comment on patches....but the CI system doesn't have that behavior (comment on patches) anymore. What should I do? | 14:26 |
| *** otherwiseguy has joined #openstack-infra | 14:26 | |
| zaro | anteaya: i don't think you can restrict a gerrit user from adding comments to a change. you can only restrict them from voting | 14:26 |
| fungi | vponomaryov: yeah, basically we just need to know when will be convenient for your project, and then we coordinate available time among infra root sysadmins to determine when a couple of us will be available to make the change (since it involves a brief outage for zuul and gerrit, and some manual steps we have to perform on the servers) | 14:27 |
| *** tonytan4ever has joined #openstack-infra | 14:27 | |
| vponomaryov | fungi: I guess we can just set it for more convenient time and inform all folks in manila room. | 14:28 |
| *** rushiagr is now known as rushiagr_away | 14:28 | |
| zaro | marcusvrn: link to wiki? | 14:28 |
| fungi | vponomaryov: if all we're doing is renaming those two projects, then it will take us probably roughly half an hour. we usually like to do them on fridays or weekends when the developer activity is at a minimum | 14:28 |
| vponomaryov | fungi: convenient time for you | 14:28 |
| *** andreaf has quit IRC | 14:28 | |
| zaro | marcusvrn: where's the ci system page? | 14:29 |
| marcusvrn | zaro: http://ci.openstack.org/third_party.html | 14:29 |
| zaro | anteaya: ^ | 14:30 |
| openstackgerrit | James E. Blair proposed a change to openstack-dev/pbr: Accept capitalized Sem-Ver headers https://review.openstack.org/118648 | 14:30 |
| vponomaryov | fungi: or send ML with "manila" tag, it is even better, but time choosing is up to you | 14:31 |
| *** wenlock has quit IRC | 14:31 | |
| openstackgerrit | Andrey Nikitin proposed a change to openstack-infra/gerritlib: Update listGroups function in gerrit.py https://review.openstack.org/117225 | 14:31 |
| fungi | vponomaryov: okay, thanks! we'll pick a time and then post to the -dev ml when we're ready to schedule it | 14:31 |
| zaro | marcusvrn: are you not the owner of that CI system? | 14:32 |
| marcusvrn | zaro: where can I link to wiki? the comments are no more shown.. :( | 14:32 |
| vponomaryov | fungi: ok | 14:32 |
| *** jcoufal_ has joined #openstack-infra | 14:32 | |
| marcusvrn | zaro: nope, I only have the credentials | 14:33 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/subunit2sql: Update pep8 ignore list https://review.openstack.org/118650 | 14:33 |
| *** jcoufal has quit IRC | 14:34 | |
| *** mmaglana has joined #openstack-infra | 14:35 | |
| zaro | marcusvrn: can you maybe ask the owner to comply with that requirement? | 14:35 |
| *** HenryG_afk is now known as HenryG | 14:36 | |
| *** ildikov has joined #openstack-infra | 14:36 | |
| *** david-lyle has joined #openstack-infra | 14:36 | |
| *** eharney_ has joined #openstack-infra | 14:36 | |
| *** stevemar has joined #openstack-infra | 14:36 | |
| openstackgerrit | James E. Blair proposed a change to openstack-dev/pbr: Accept capitalized Sem-Ver headers https://review.openstack.org/118648 | 14:37 |
| *** eharney has quit IRC | 14:37 | |
| *** eharney_ is now known as eharney | 14:37 | |
| *** chuckC has joined #openstack-infra | 14:38 | |
| *** gargola has joined #openstack-infra | 14:38 | |
| *** cody-somerville has joined #openstack-infra | 14:39 | |
| *** bswartz has joined #openstack-infra | 14:39 | |
| jeblair | fungi: shall we restart nodepool? | 14:40 |
| *** ildikov_ has quit IRC | 14:40 | |
| marcusvrn | zaro: sry, I misunderstood your question....I'm deploying the CI, so I have to do it | 14:40 |
| fungi | jeblair: let's do this thing. no time like the present | 14:41 |
| fungi | shall i try to gracefullt stop it? | 14:41 |
| fungi | er, gracefully | 14:41 |
| *** ildikov has quit IRC | 14:41 | |
| jeblair | fungi: does that work? i thought it didn't. or at least, not well. | 14:42 |
| fungi | it used to... | 14:42 |
| jeblair | fungi: i guess we can try it for a few minutes :) | 14:42 |
| fungi | just did, stopped instantly | 14:42 |
| fungi | from what i've seen it either works immediately or not at all | 14:42 |
| fungi | shall i start it back up now? | 14:43 |
| jeblair | yup | 14:43 |
| fungi | there it goes, deleting building nodes | 14:43 |
| fungi | launching new servers | 14:43 |
| fungi | OverLimit: Quota exceeded for ram: Requested 8192, but already used 761856 of 76 | 14:44 |
| fungi | 8000 ram | 14:44 |
| *** yfried has joined #openstack-infra | 14:44 | |
| fungi | rax-iad | 14:44 |
| jeblair | fungi: we might want to run an alien check and also look for used > a few hours in a few hours. | 14:44 |
| fungi | i will add that to my morning plans | 14:44 |
| *** zz_gondoi is now known as gondoi | 14:45 | |
| dolphm | can someone clarify if we should be using reverify at all anymore, or if it's *only* "recheck <reasoning>" now? | 14:46 |
| AJaeger | fungi, the gitreview only landed last night and the proposal job run before. Thanks! | 14:46 |
| openstackgerrit | Sylvain Bauza proposed a change to openstack-infra/config: Enable IRC channel logging for #openstack-fr https://review.openstack.org/118651 | 14:46 |
| *** datsun180b has joined #openstack-infra | 14:46 | |
| fungi | dolphm: they're synonymous now | 14:47 |
| marun_ | jogo: re: https://review.openstack.org/#/c/118231/ | 14:47 |
| dolphm | fungi: thank you! cc- ttx | 14:47 |
| ttx | ack | 14:47 |
| *** atiwari has joined #openstack-infra | 14:47 | |
| fungi | dolphm: more or less have been ever since we added the clean check requirement, since you couldn't reenqueue into the gate after failing out without rerunning check jobs. but about a month ago we just went ahead and made them completely synonymous (about the same time we stopped requiring a bug number or a "no bug") | 14:48 |
| marun_ | jogo: given this following traceback, can you help me determine a more appropriate elasticsearch query? | 14:48 |
| marun_ | jogo: http://logs.openstack.org/44/117444/6/gate/gate-neutron-dsvm-functional/ae6c985/console.html#_2014-09-02_05_28_19_205 | 14:48 |
| marun_ | jogo: The problem can apparently reproduce on other tests that use eventlet, btw, though the l3 agent test is the only cause in the gate. | 14:49 |
| AJaeger | fungi, I suggest that the .gitreview gets approved as part of the migration - it will cause problems otherwise... | 14:49 |
| *** amuller_ has joined #openstack-infra | 14:49 | |
| *** koolhead17 has joined #openstack-infra | 14:49 | |
| *** koolhead17 has joined #openstack-infra | 14:49 | |
| *** unicell has quit IRC | 14:50 | |
| fungi | AJaeger: we may want to rethink how that's being used to add gerrit remotes in jobs, so that it can stay decoupled | 14:50 |
| dolphm | fungi: good to know, thanks | 14:50 |
| AJaeger | fungi, that's another option... | 14:51 |
| *** tsg has joined #openstack-infra | 14:51 | |
| *** amuller has quit IRC | 14:51 | |
| fungi | jeblair: so far so good... not jumping significantly high on node count yet, at least | 14:52 |
| jeblair | woo | 14:52 |
| *** wenlock has joined #openstack-infra | 14:53 | |
| zhiyan | fungi: may i get your 2 mins? met a probably in job, http://logs.openstack.org/31/105231/40/check/check-tempest-dsvm-neutron-heat-slow/be2aa4d/console.html#_2014-09-03_13_31_05_933 | 14:54 |
| fungi | building count is falling and used count increasing, though it's still trying to worth through deletions from earlier i think | 14:54 |
| zhiyan | fungi: i have no idea if this is an known issue? | 14:54 |
| *** jcoufal_ has quit IRC | 14:55 | |
| fungi | zhiyan: did you look in the devstack logs? | 14:55 |
| fungi | zhiyan: "Running devstack ... this takes 5 - 8 minutes (logs in logs/devstacklog.txt.gz) ... ERROR: the main setup script run by this job failed - exit code: 1 ... please look at the relevant log files to determine the root cause" | 14:56 |
| zhiyan | i could met that issue in https://review.openstack.org/#/c/118627/ as well..it's a rush fix to unblock some changes.. | 14:56 |
| fungi | zhiyan: http://logs.openstack.org/31/105231/40/check/check-tempest-dsvm-neutron-heat-slow/be2aa4d/logs/devstacklog.txt.gz#_2014-09-03_13_35_53_792 | 14:57 |
| jeblair | fungi: a lot of deletion timeouts and a lot of 'ERROR' states from rax | 14:57 |
| fungi | zhiyan: looks like someone might have broken glanceclient | 14:57 |
| fungi | zhiyan: oh, it's you | 14:57 |
| fungi | zhiyan: your change breaks glanceclient | 14:57 |
| zhiyan | fungi: oh? which one? | 14:58 |
| fungi | zhiyan: 118627 | 14:58 |
| *** bhuvan has joined #openstack-infra | 14:58 | |
| fungi | zhiyan: please look at the test logs, or test your change yourself. integration tests appear to be doing precisely what they're designed to do--keeping your change from merging and breaking glanceclient | 14:59 |
| zhiyan | fungi: actually we just want to use 118627 to fix the issue which latest requests provided | 14:59 |
| *** pblaho is now known as pblaho|afk | 14:59 | |
| *** rushiagr_away is now known as rushiagr | 14:59 | |
| zhiyan | fungi: without that change the glanceclient will be failed, actually... | 14:59 |
| *** mriedem has quit IRC | 15:00 | |
| *** mriedem has joined #openstack-infra | 15:01 | |
| fungi | zhiyan: with that change it fails in a different way ("from requests.packages import urllib3 == ImportError: No module named packages") | 15:01 |
| zhiyan | fungi: because the version which jenkins installed is not the latest version.. | 15:01 |
| fungi | zhiyan: the version of glanceclient being used in that job is missing the change being tested? | 15:03 |
| fungi | zhiyan: that doesn't seem to be the case at all | 15:03 |
| fungi | zhiyan: the line which is throwing an exception is a line added by your proposed change | 15:04 |
| zhiyan | fungi: now i'm not sure why, but from a older test result, i can see a new exception (ProtocolError) which only latest requests provided (2.4.0) could be triggered, so that's why we prepared #118627.. | 15:04 |
| zhiyan | fungi: yes, https://launchpad.net/bugs/1364893 , the reason. | 15:04 |
| fungi | zhiyan: sure, but you can't "from requests.packages import urllib3" because requests.packages is not a module | 15:04 |
| uvirtbot | Launchpad bug 1364893 in python-glanceclient "New version of requests library breaks unit tests" [Undecided,In progress] | 15:04 |
| fungi | zhiyan: that's what the python excpetion being raised means | 15:05 |
| zhiyan | fungi: ok, will try. thanks. | 15:06 |
| *** zhiwei has quit IRC | 15:07 | |
| *** pblaho|afk is now known as pblaho | 15:07 | |
| *** _nadya_ has joined #openstack-infra | 15:08 | |
| fungi | zhiyan: that job ran with requests==2.2.1 | 15:09 |
| fungi | zhiyan: according to http://logs.openstack.org/31/105231/40/check/check-tempest-dsvm-neutron-heat-slow/be2aa4d/logs/pip-freeze.txt.gz | 15:09 |
| zhiyan | fungi: interesting..but recently i see it 2.4.0 | 15:09 |
| *** dizquierdo has joined #openstack-infra | 15:10 | |
| fungi | zhiyan: something else being installed in that integration test has that pinned, so it's running with an earlier version than 2.4.0 | 15:12 |
| *** imeerovi has quit IRC | 15:12 | |
| *** mmaglana has quit IRC | 15:12 | |
| zhiyan | fungi: hum..for this issue, i don't think pining is a good idea, i think we'dbetter just fix it | 15:13 |
| *** bdpayne has joined #openstack-infra | 15:13 | |
| *** mmaglana has joined #openstack-infra | 15:13 | |
| zhiyan | fungi: but back to this topic, i see global requment didn't pin it. https://github.com/openstack/requirements/blob/master/global-requirements.txt#L120 | 15:13 |
| zhiyan | fungi: so do you think we could just skip 118627? (due to we will not use 2.4.0 , and gate will be unblocked), but i think it's not a stable solution.... | 15:15 |
| fungi | zhiyan: right, i don't think one of our projects is likely pinning it... some transitive dependency probably is | 15:15 |
| jroll | might be better to pin global-reqs at < 2.4.0 | 15:15 |
| *** AJaeger has quit IRC | 15:15 | |
| fungi | zhiyan: no, i think it needs to be fixed, but integration testing is complicated when you're trying to interoperate with other dependencies which may declare dependencies on different versions of things you use | 15:16 |
| jroll | (I think we should be pinning at == $version for everything, but that's a different conversation) | 15:16 |
| *** gokrokve has joined #openstack-infra | 15:16 | |
| zhiyan | fungi: do you think pining <2.4.0 is a idea? | 15:16 |
| fungi | jroll: that works until you start depending on things which depend on things | 15:16 |
| zhiyan | temporarily | 15:17 |
| *** Guest64359 has quit IRC | 15:17 | |
| fungi | jroll: and also if you don't care whether the things you depend on have security holes fixed in newer releases | 15:17 |
| *** mmaglana has quit IRC | 15:17 | |
| *** rlandy has quit IRC | 15:17 | |
| *** rhe00 has left #openstack-infra | 15:18 | |
| *** Hal has joined #openstack-infra | 15:18 | |
| jroll | fungi: I realize the problem is much harder than me waving my hands around :) | 15:18 |
| *** Hal is now known as Guest42286 | 15:18 | |
| fungi | jroll: i frequently run around with flailing kermit arms | 15:19 |
| jroll | :D | 15:19 |
| *** vhoward has left #openstack-infra | 15:19 | |
| *** doug-fish1 has left #openstack-infra | 15:20 | |
| fungi | zhiyan: aha! so the reason it's using requests==2.2.1 is that we apt-get install python-requests on ubuntu trusty for devstack | 15:20 |
| fungi | zhiyan: http://logs.openstack.org/31/105231/40/check/check-tempest-dsvm-neutron-heat-slow/be2aa4d/logs/dpkg-l.txt.gz | 15:20 |
| *** acruz has quit IRC | 15:21 | |
| zhiyan | fungi: can you help fix it? | 15:21 |
| jroll | I'd be willing to bet it's a dependency for some other package installed by apt | 15:22 |
| fungi | zhiyan: with the python-requests deb installed, i can recreate this error. not sure if it's a modification of the requests library by the debian package maintainer or something else odd | 15:22 |
| sdague | zhiyan: glance client should probably be able to bridge across this | 15:22 |
| *** jcoufal has joined #openstack-infra | 15:22 | |
| jroll | that's a *lot* of python packages installed by apt :( | 15:22 |
| *** pblaho is now known as pblaho|afk | 15:22 | |
| zhiyan | sdague: it's a temporary way can go, but good enough | 15:23 |
| *** paulrad has quit IRC | 15:23 | |
| *** liam__ has joined #openstack-infra | 15:23 | |
| *** hdd has joined #openstack-infra | 15:23 | |
| *** acruz has joined #openstack-infra | 15:23 | |
| *** paulrad has joined #openstack-infra | 15:24 | |
| zhiyan | fungi: jroll yep, i see the point | 15:24 |
| *** torandu has left #openstack-infra | 15:24 | |
| *** ominakov has quit IRC | 15:24 | |
| sdague | so python-boto is what's dragging it in on devstack | 15:26 |
| sdague | which, honestly, removing from the package list wouldn't be a terrible idea | 15:26 |
| *** Sukhdev has joined #openstack-infra | 15:26 | |
| fungi | zhiyan: yep, so it's a local change introduced into debian's python-requests package... http://anonscm.debian.org/viewvc/python-modules/packages/requests/trunk/debian/patches/02_use-system-chardet-and-urllib3.patch?revision=27448&view=markup | 15:26 |
| zhiyan | i saw novaclient has this same issue as well, probably needs bridge as well in its code | 15:27 |
| *** pballand has joined #openstack-infra | 15:27 | |
| *** ominakov_ has joined #openstack-infra | 15:27 | |
| *** paulrad has quit IRC | 15:28 | |
| fungi | it appears the workaround is to try: from requests.packages import urllib3 and then except ImportError: import urllib3 | 15:28 |
| zhiyan | ok, thanks fungi sdague inputs. | 15:28 |
| *** acruz is now known as arxcruz | 15:28 | |
| fungi | since python-urllib3 is packaged separately | 15:29 |
| *** ominakov_ has quit IRC | 15:29 | |
| openstackgerrit | Dmitry Teselkin proposed a change to openstack-infra/config: Pin pupppetdb-terminus package https://review.openstack.org/118660 | 15:30 |
| fungi | jeblair: hrm... we have a glut of ready nodes now. checking for any obvious imbalance/staleness | 15:30 |
| jeblair | yeah i was just noticing that | 15:30 |
| jeblair | fungi: they are all very new | 15:30 |
| jeblair | oh nm, not that new | 15:30 |
| *** liam__ has quit IRC | 15:30 | |
| jeblair | some of them are several 10s of minutes old | 15:31 |
| fungi | that may be some sort of hysteresis and then lag in going from nodepool-ready to jenkins-ready | 15:31 |
| *** Sincler has joined #openstack-infra | 15:32 | |
| fungi | the bulk are in rax | 15:32 |
| fungi | but fairly rapidly falling | 15:32 |
| jeblair | almost no one is waiting on a centos job :/ | 15:33 |
| *** amotoki has joined #openstack-infra | 15:34 | |
| *** alexpilotti has joined #openstack-infra | 15:34 | |
| *** pblaho|afk is now known as pblaho | 15:34 | |
| *** skolekonov has quit IRC | 15:34 | |
| fungi | switching to an overlap graph instead of a stack graph makes it look like we had an influx of ready as the building sharply fell, and then a lot of delete as the ready fell, so presumably just nodepool managing to get well in front of the demand? | 15:35 |
| jeblair | fungi: well in front of the demand isn't really desirable when we're at capacity | 15:36 |
| fungi | building is climbing again as well | 15:36 |
| jeblair | i mean, nodes sitting for 30 minuts doing nothing isn't great | 15:36 |
| YorikSar | mordred: ping | 15:36 |
| fungi | trying to figure out whether this is similar to what we saw the last time, minus the jumping waaay above quota behavior | 15:36 |
| *** gokrokve has quit IRC | 15:37 | |
| YorikSar | mordred: As I understand, it was you who switched git-review to bare man page instead of Sphinx-generated one. | 15:37 |
| YorikSar | mordred: Maybe we can switch back and use wheels instead? | 15:37 |
| fungi | it looks like building jumps in 10-minute intervals, then falls sharply, then starts to climb again | 15:37 |
| jeblair | fungi: oh, most of them are precise | 15:37 |
| jeblair | fungi: i get it | 15:37 |
| *** jgallard has quit IRC | 15:37 | |
| clarkb | I take it nodepool was restarted /me reads sb | 15:37 |
| fungi | clarkb: yup | 15:37 |
| jeblair | fungi: the round robin allocator is satisfying all of the "demand" for devstack-precise nodes when it gets to be their turn | 15:38 |
| jeblair | fungi: the "demand" for them is from min-ready | 15:38 |
| fungi | aha | 15:38 |
| fungi | so we jump by as much as min-ready for that type | 15:38 |
| jeblair | fungi: normally, min-ready has little impact when at capacity | 15:38 |
| jeblair | but now we're giving it outsized importance | 15:38 |
| *** gokrokve has joined #openstack-infra | 15:39 | |
| clarkb | oh right, we used it to compensate before but now we are getting it all | 15:39 |
| *** annegent_ has quit IRC | 15:39 | |
| *** yamahata has quit IRC | 15:39 | |
| *** liam__ has joined #openstack-infra | 15:39 | |
| *** andreaf has joined #openstack-infra | 15:39 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 15:40 | |
| mordred | YorikSar: why do man pages have anything to do with wheels? | 15:40 |
| YorikSar | mordred: We can package them in wheel and user who installs this wheel won't need Sphinx to get manpage. | 15:41 |
| *** zz_johnthetubagu is now known as johnthetubaguy | 15:41 | |
| mordred | well, kinda. except that we don't know where to install the man page until install time | 15:42 |
| mordred | because, well, insanity | 15:42 |
| *** andreaf_ has joined #openstack-infra | 15:42 | |
| fungi | mordred: YorikSar: also we don't currently have a way to specify different requirements for tarballs vs wheels, right? | 15:43 |
| mordred | so, we can make the manpage generated by Sphinx, but it will not solve or change anything | 15:43 |
| *** mflobo has quit IRC | 15:43 | |
| mordred | fungi there is no way to do that, that's right | 15:43 |
| *** liam__ has quit IRC | 15:43 | |
| YorikSar | mordred: Hm... How does people package manpages then?.. | 15:44 |
| *** zeus has quit IRC | 15:44 | |
| YorikSar | fungi: I think setup_requires should do that. | 15:45 |
| *** chuckC has quit IRC | 15:45 | |
| *** zeus has joined #openstack-infra | 15:45 | |
| YorikSar | fungi: With wheels setup.py don't get executed at all. | 15:45 |
| *** paulrad has joined #openstack-infra | 15:45 | |
| *** andreaf has quit IRC | 15:45 | |
| fungi | YorikSar: usually developers package manpages as static files (or often not at all). also manpages via pip install are sort of broken due to it being hard to maintain cross-platform support for all the various places they could go (and in a virtualenv they make basically no sense whatsoever) | 15:46 |
| openstackgerrit | A change was merged to openstack-infra/subunit2sql: Update pep8 ignore list https://review.openstack.org/118650 | 15:47 |
| fungi | on a related note, based on a recent thread on debian-devel i'm getting the feeling that a lot of distro packagers now view upstream tarballs as at best a relic, at worst a nuisance, and their use in distribution source packages a mere implementation detail | 15:48 |
| mordred | yup | 15:48 |
| fungi | seems many would rather upstreams just tagged a commit in vcs and left instructions on how to deploy that so it can be translated into packaging | 15:48 |
| *** emagana has joined #openstack-infra | 15:49 | |
| mordred | I'm fine with that as long as they don't argue with me on build tooling :) | 15:49 |
| mordred | that is, our releases are all tagged and content is 100% reproducible | 15:49 |
| YorikSar | fungi: That's interesting. I wonder if we ever give up tarballs :) | 15:49 |
| mordred | YorikSar: setuptools is especially dumb wrt manpages | 15:50 |
| fungi | in which case, if we accept this as a commonplace attitude, we could just start considering our tarballs to purely be the pypi packaging mechanism rather than trying to make it double as "upstream source" | 15:50 |
| jeblair | mordred: i can't imagine anyone having a discussion on build tooling that isn't an argument :) | 15:50 |
| mordred | jeblair: :) | 15:50 |
| mordred | fungi I'm in large favor of admitting that our source is in git | 15:50 |
| mordred | since it is | 15:50 |
| *** mmaglana has joined #openstack-infra | 15:51 | |
| YorikSar | mordred: All setuptools installed manpages to to ~/.local/share/man on my machine and that's what I've added to MANPATH :) | 15:51 |
| mordred | yah. systemwide it gets brokener | 15:52 |
| pleia2 | title could have been slightly better, but hey hey http://www.linux.com/news/featured-blogs/200-libby-clark/786065-how-elizabeth-joseph-became-a-sysadmin-on-hps-openstack-infrastructure-team | 15:52 |
| fungi | YorikSar: does that work if you install into a virtualenv? i've not had a lot of luck getting that to pan out | 15:52 |
| *** trinaths1 has joined #openstack-infra | 15:52 | |
| mordred | the problem isn't that | 15:53 |
| pleia2 | mordred: welcome back, I made a stick puppet you for puppetconf https://www.flickr.com/photos/pleia2/14809145127/ | 15:53 |
| mordred | it's that manpaths are different on different systems | 15:53 |
| *** trinaths1 has left #openstack-infra | 15:53 | |
| *** chuckC has joined #openstack-infra | 15:53 | |
| YorikSar | fungi: You can probably use relative paths in MANPATH (like ./.tox/py27/share/man), but I have never wanted that. | 15:53 |
| mordred | pleia2: I saw! thank you!! | 15:53 |
| *** emagana has quit IRC | 15:53 | |
| pleia2 | mordred: yay :) | 15:53 |
| openstackgerrit | Dolph Mathews proposed a change to openstack-infra/elastic-recheck: Add query for critical keystone regression https://review.openstack.org/118668 | 15:54 |
| mordred | YorikSar: you CAN do anything, but we needs to be able to install properly into default systems | 15:54 |
| *** emagana has joined #openstack-infra | 15:55 | |
| YorikSar | mordred: Sure. I'm just saying that using pip/setuptools to install things into otherwise-package-controlled system won't generally work anyway. | 15:55 |
| dolphm | jogo: mtreinish: sdague: elastic-recheck query for a critical keystone regression introduced about 2 hours ago: https://review.openstack.org/118668 | 15:55 |
| mordred | but it does work noe | 15:55 |
| mordred | now | 15:56 |
| mordred | and I'm not going to break that without good reason | 15:56 |
| *** arxcruz has quit IRC | 15:56 | |
| mordred | the "don't mix pip and apt" mantra is unrealistic ... it regularly happens | 15:56 |
| *** bdpayne has quit IRC | 15:57 | |
| dolphm | sdague: thank you | 15:57 |
| sdague | mordred: I've started some selective pruning of apt installed python that's pure python, as I don't think we gain anything from it in devstack. | 15:57 |
| mordred | ++ | 15:57 |
| sdague | but for python/c stuff, I think we need to stay with apt | 15:57 |
| mordred | sdague: I think we should install nothing from apt that we can install from pip | 15:58 |
| mordred | for consistency | 15:58 |
| YorikSar | mordred: Ok, now I'm confused. How does it work now and why won't it work the same way with wheel?.. | 15:58 |
| sdague | mordred: so .... gentoo? | 15:58 |
| *** dtantsur is now known as dtantsur|afk | 15:58 | |
| YorikSar | sdague: Yay! Gentoo! | 15:58 |
| mordred | YorikSar: right now we run code in setup.py to figure out the right path at install time | 15:58 |
| sdague | I really don't want to be compiling source for 30 minutes on a new devstack | 15:59 |
| mordred | since wheel doesn't execute that, wheel git-review will put manpages in the wrong place on some platforms | 15:59 |
| clarkb | sdague: we can probably mitigate a lot of that with ccache and pip installs to a tmp virtualenv in our image build process | 15:59 |
| *** bdpayne has joined #openstack-infra | 15:59 | |
| mordred | sdague: we have other solutions for tjat | 15:59 |
| YorikSar | mordred: Who decides where to put them? pbr? | 15:59 |
| mordred | YorikSar: yes | 15:59 |
| mordred | sdague: Ubuntu is fine ... apt is fine ... | 16:00 |
| YorikSar | sdague: We can build binary package repo for everythin but our stuff :) | 16:00 |
| mordred | sdague: but if it's in global reqs, we should install from pip | 16:00 |
| YorikSar | mordred: Ah... Magic... Ok... | 16:00 |
| sdague | mordred: there are other gate solutions | 16:00 |
| *** enikanorov has joined #openstack-infra | 16:00 | |
| sdague | but not other solutions for developers getting started | 16:01 |
| mordred | sdague: there are other dev solutions too | 16:01 |
| mordred | yes there are | 16:01 |
| sdague | ok | 16:01 |
| mordred | but I'm on my phone ... :) | 16:01 |
| sdague | well, propose away. I mostly just don't want to turn the 10 minute devstack install time into an hour long process, especially when much of that is available as prebuilt binaries | 16:01 |
| mordred | nod | 16:02 |
| *** Longgeek has quit IRC | 16:02 | |
| YorikSar | sdague: We can prebuild distro-specific wheels for gate. | 16:02 |
| sdague | YorikSar: I'm not really talking about the gate here | 16:02 |
| sdague | that's a different optimization | 16:03 |
| YorikSar | sdague: Ah... | 16:03 |
| *** sweston_ has joined #openstack-infra | 16:03 | |
| sdague | devstack is actually used by people outside of the gate :) | 16:03 |
| clarkb | fungi: Bug 1365046 is weird and potentially an incidence of that rax thing? but this time it is in ord | 16:03 |
| uvirtbot | Launchpad bug 1365046 in openstack-ci "Job failed due to no devstack directory" [Undecided,New] https://launchpad.net/bugs/1365046 | 16:03 |
| *** TravT has joined #openstack-infra | 16:03 | |
| mordred | I actually don't really care. but we have global requirements, and we subvert that in the gate for a random collection of things. in most other cases, we try for correctness and then fix the problem | 16:03 |
| *** marcoemorais has joined #openstack-infra | 16:03 | |
| YorikSar | Outside CI devstack tends to take way more than 10 minutes for me. | 16:03 |
| *** cbkyeoh has joined #openstack-infra | 16:03 | |
| jeblair | YorikSar: we cache most of what it needs on the node already, so there's very little network traffic | 16:04 |
| *** eglynn__ has joined #openstack-infra | 16:04 | |
| *** vish1 has joined #openstack-infra | 16:04 | |
| mordred | so, we can keep arbitrarily installing some python things from Ubuntu and I'm fine with it ... but it is surprising given g-r | 16:04 |
| *** pelix1 has joined #openstack-infra | 16:04 | |
| *** hogepodge has joined #openstack-infra | 16:04 | |
| TravT | ok,probably a dumb question. will zuul attempt an auto rebase if a dependent patch is not based on the most recent parent patch? | 16:05 |
| clarkb | hrm http://logs.openstack.org/54/109754/13/check/check-tempest-dsvm-full/10d7fe8/logs/devstack-gate-setup-workspace-new.txt.gz does show openstack-dev/devstack getting updated which is probably the cause? | 16:05 |
| *** bmwiedemann1 has joined #openstack-infra | 16:05 | |
| *** primemin1sterp has joined #openstack-infra | 16:05 | |
| *** markvan_ has joined #openstack-infra | 16:06 | |
| *** bswartz1 has joined #openstack-infra | 16:06 | |
| *** willroberts_ has joined #openstack-infra | 16:06 | |
| *** simonmcc____ has joined #openstack-infra | 16:06 | |
| clarkb | http://logs.openstack.org/54/109754/13/check/check-tempest-dsvm-full/10d7fe8/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-09-03_08_50_25_414 that will do it | 16:06 |
| *** tristanC_ has joined #openstack-infra | 16:06 | |
| clarkb | fungi: false alarm not related to that imge thing | 16:06 |
| fungi | clarkb: http://logs.openstack.org/54/109754/13/check/check-tempest-dsvm-full/10d7fe8/logs/devstack-gate-setup-workspace-new.txt.gz | 16:07 |
| sdague | mordred: well lets deal with the easy parts first, where I don't think there is any drawback, then deal with the other parts later | 16:07 |
| *** afazekas has quit IRC | 16:07 | |
| mordred | ++ | 16:07 |
| *** thingee_ has joined #openstack-infra | 16:07 | |
| *** Adri2000_ has joined #openstack-infra | 16:07 | |
| sdague | instead of start at the hard parts, get in a fight, and never do the easy parts :) | 16:07 |
| *** NobodyCa1 has joined #openstack-infra | 16:07 | |
| *** masayukig_ has joined #openstack-infra | 16:07 | |
| *** dosaboy_ has joined #openstack-infra | 16:07 | |
| fungi | clarkb: it hit some sort of weird https issue cloning nova, and the setup-workspace script exited nonzero as a result | 16:07 |
| *** Adri2000_ is now known as Guest60909 | 16:07 | |
| *** doude_ has joined #openstack-infra | 16:08 | |
| clarkb | fungi: ya | 16:08 |
| *** ctracey_ has joined #openstack-infra | 16:08 | |
| sdague | speaking of which - https://review.openstack.org/#/c/118661/ - which would have exposed the requests thing more places | 16:08 |
| *** signed8b_ has joined #openstack-infra | 16:08 | |
| *** ctracey has quit IRC | 16:08 | |
| *** sweston has quit IRC | 16:08 | |
| *** vishy has quit IRC | 16:08 | |
| *** vish1 is now known as vishy | 16:08 | |
| *** cyeoh has quit IRC | 16:08 | |
| *** pelix has quit IRC | 16:08 | |
| *** SumitNaiksatam has quit IRC | 16:08 | |
| *** primeministerp has quit IRC | 16:08 | |
| *** skath_ has quit IRC | 16:08 | |
| *** Mithrandir has quit IRC | 16:08 | |
| *** erw has quit IRC | 16:08 | |
| *** msuriar has quit IRC | 16:08 | |
| *** k4n0 has quit IRC | 16:08 | |
| *** bswartz has quit IRC | 16:08 | |
| *** sileht has quit IRC | 16:08 | |
| *** rossella_s has quit IRC | 16:08 | |
| *** eglynn has quit IRC | 16:08 | |
| *** AaronGreen has quit IRC | 16:08 | |
| *** tristanC has quit IRC | 16:08 | |
| *** dougwig has quit IRC | 16:08 | |
| *** enikanorov_ has quit IRC | 16:08 | |
| *** freyes has quit IRC | 16:08 | |
| *** tchaypo has quit IRC | 16:08 | |
| *** StevenK has quit IRC | 16:08 | |
| *** rushiagr has quit IRC | 16:08 | |
| *** luqas has quit IRC | 16:08 | |
| *** mancdaz has quit IRC | 16:08 | |
| *** simonmcc___ has quit IRC | 16:08 | |
| *** maurosr has quit IRC | 16:08 | |
| *** signed8bit has quit IRC | 16:08 | |
| *** Svedrin has quit IRC | 16:08 | |
| *** NobodyCam has quit IRC | 16:08 | |
| *** markvan has quit IRC | 16:08 | |
| *** thingee has quit IRC | 16:08 | |
| *** dosaboy has quit IRC | 16:08 | |
| *** masayukig has quit IRC | 16:08 | |
| *** Adri2000 has quit IRC | 16:08 | |
| *** krtaylor has quit IRC | 16:08 | |
| *** GheRivero has quit IRC | 16:08 | |
| *** doude has quit IRC | 16:08 | |
| *** johnthetubaguy has quit IRC | 16:08 | |
| *** asadoughi has quit IRC | 16:08 | |
| *** willroberts has quit IRC | 16:08 | |
| *** mancdaz has joined #openstack-infra | 16:08 | |
| *** cbkyeoh is now known as cyeoh | 16:08 | |
| *** Mithrandir_ has joined #openstack-infra | 16:08 | |
| *** ctracey_ is now known as ctracey | 16:08 | |
| *** sweston_ is now known as sweston | 16:08 | |
| *** msuriar has joined #openstack-infra | 16:08 | |
| *** tristanC_ is now known as tristanC | 16:08 | |
| *** bswartz1 has quit IRC | 16:08 | |
| *** tchaypo has joined #openstack-infra | 16:09 | |
| *** luqas has joined #openstack-infra | 16:09 | |
| *** rossella_s has joined #openstack-infra | 16:09 | |
| *** k4n0_ has joined #openstack-infra | 16:09 | |
| *** bswartz has joined #openstack-infra | 16:09 | |
| fungi | clarkb: which begs the question why it tried to go ahead with the gate hook | 16:09 |
| sdague | mordred: also... aren't you supposed to be on a beach ? | 16:09 |
| *** johnthetubaguy has joined #openstack-infra | 16:09 | |
| clarkb | fungi: right, it should short circuit hard there imo | 16:09 |
| *** maurosr has joined #openstack-infra | 16:09 | |
| *** skath_ has joined #openstack-infra | 16:09 | |
| *** asadoughi has joined #openstack-infra | 16:09 | |
| *** erw_ has joined #openstack-infra | 16:09 | |
| *** rushiagr has joined #openstack-infra | 16:10 | |
| *** dougwig_ has joined #openstack-infra | 16:10 | |
| *** krtaylor has joined #openstack-infra | 16:10 | |
| *** Svedrin has joined #openstack-infra | 16:10 | |
| *** freyes has joined #openstack-infra | 16:10 | |
| *** AaronGr has joined #openstack-infra | 16:10 | |
| *** amcrn has joined #openstack-infra | 16:10 | |
| *** sileht has joined #openstack-infra | 16:10 | |
| *** sileht has quit IRC | 16:10 | |
| *** che-arne has joined #openstack-infra | 16:10 | |
| *** bswartz has quit IRC | 16:10 | |
| *** sileht has joined #openstack-infra | 16:10 | |
| jeblair | fungi: the ready spread doesn't seem so egregious now | 16:11 |
| *** bswartz has joined #openstack-infra | 16:11 | |
| jeblair | it looks like it's a bit more than before, but not radically so | 16:11 |
| *** trinaths has joined #openstack-infra | 16:11 | |
| *** thingee_ is now known as thingee | 16:11 | |
| clarkb | jeblair: do we need to udpate our min ready values to be a lot more conservative? since we should expand to meet demand better now? | 16:11 |
| *** krtaylor has quit IRC | 16:11 | |
| fungi | jeblair: right--at this point i;m mostly evaluating it by how/whether it keeps up with the check pipeline backlog while still fulfilling all gate pipeline needs in a timely fashion | 16:11 |
| *** trinaths has left #openstack-infra | 16:12 | |
| fungi | clarkb: i think that robs us of faster job results during quiet periods if we do | 16:12 |
| jeblair | clarkb: i think min-ready is so that .. yeah what fungi said | 16:12 |
| *** simonmcc____ is now known as simonmcc___ | 16:12 | |
| *** pblaho has quit IRC | 16:12 | |
| jeblair | fungi: i noticed lower gate was lagging on devstack-trusty a while ago, but looks ok now | 16:13 |
| *** StevenK has joined #openstack-infra | 16:13 | |
| *** GheRivero has joined #openstack-infra | 16:13 | |
| fungi | yeah, briefly | 16:13 |
| *** bhuvan has quit IRC | 16:13 | |
| clarkb | hrm good point, it is there to combat lag when we are quieter | 16:14 |
| fungi | though at the expense of making the logic a bit more complicated, we could scale min-ready downward the more we exceed it in use | 16:14 |
| mordred | sdague: I am. I'm not here | 16:14 |
| fungi | but that seems like it would get unnecessarily fiddly | 16:15 |
| jeblair | i notice a lot of red in the gate due to socket timeouts to pypi.o.o | 16:15 |
| jeblair | i think hpcloud<->rax may have had a network thing | 16:15 |
| *** habib has joined #openstack-infra | 16:16 | |
| *** willroberts_ is now known as willroberts | 16:16 | |
| clarkb | jeblair: that may be related to the TLS thing I linked to as well? though that was ord to dfw | 16:16 |
| *** andreaf_ is now known as andreaf | 16:17 | |
| jeblair | clarkb: oh i missed your followup on that. hrm. | 16:17 |
| *** paulrad has quit IRC | 16:17 | |
| *** paulrad has joined #openstack-infra | 16:17 | |
| *** bdpayne_ has joined #openstack-infra | 16:17 | |
| jeblair | clarkb: well, there's an 8 hour time difference | 16:18 |
| clarkb | oh ya | 16:18 |
| *** MaxV has quit IRC | 16:18 | |
| *** MaxV has joined #openstack-infra | 16:19 | |
| *** paulrad has quit IRC | 16:19 | |
| *** paulrad_ has joined #openstack-infra | 16:19 | |
| *** gyee has joined #openstack-infra | 16:20 | |
| *** bdpayne has quit IRC | 16:21 | |
| *** amuller__ has joined #openstack-infra | 16:22 | |
| *** mwagner_lap has quit IRC | 16:22 | |
| *** MaxV has quit IRC | 16:23 | |
| *** NobodyCa1 is now known as NobodyCam | 16:24 | |
| *** paulrad_ has quit IRC | 16:24 | |
| *** annegent_ has joined #openstack-infra | 16:25 | |
| *** amuller_ has quit IRC | 16:26 | |
| clarkb | the zuul events queue is fairly large | 16:28 |
| *** cody-somerville has quit IRC | 16:29 | |
| *** annegent_ has quit IRC | 16:30 | |
| jeblair | now it's small | 16:31 |
| *** cody-somerville has joined #openstack-infra | 16:31 | |
| *** Guest42286 has quit IRC | 16:32 | |
| clarkb | that was quick. I guess the event queue processing backs it up occasionally | 16:32 |
| jeblair | result queue processing is generally slower; it causes queue walks | 16:32 |
| jeblair | but it has priority over event queue processing | 16:32 |
| clarkb | right because results can affect how we process events | 16:33 |
| jeblair | clarkb: most of the backloge in the trigger event queue was pseudo events for a merged change | 16:34 |
| jeblair | so in that case, we have hundreds of trigger events appear all at once, and they get processed pretty quickly too | 16:34 |
| *** bhuvan has joined #openstack-infra | 16:35 | |
| *** koolhead_ has joined #openstack-infra | 16:35 | |
| clarkb | I see | 16:35 |
| *** jistr has quit IRC | 16:35 | |
| *** _nadya_ has quit IRC | 16:36 | |
| *** terryw has joined #openstack-infra | 16:37 | |
| *** krtaylor has joined #openstack-infra | 16:37 | |
| jeblair | wow, neutron has some very old open changes | 16:37 |
| *** otherwiseguy has quit IRC | 16:37 | |
| jeblair | 17248 | 16:37 |
| *** koolhead17 has quit IRC | 16:37 | |
| *** praneshp has joined #openstack-infra | 16:37 | |
| *** luqas has quit IRC | 16:37 | |
| *** annegent_ has joined #openstack-infra | 16:38 | |
| *** jlibosva has quit IRC | 16:38 | |
| clarkb | jeblair: some additional changes just merged and the event queue did grow quickly | 16:39 |
| clarkb | jeblair: yay meta events :) ? | 16:39 |
| *** dizquierdo has quit IRC | 16:40 | |
| *** mriedem1 has joined #openstack-infra | 16:40 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Fix method name call when files cannot be found https://review.openstack.org/106600 | 16:41 |
| *** ildikov has joined #openstack-infra | 16:41 | |
| jeblair | clarkb: yeah, pushes our numbers up without doing much more work! :) | 16:42 |
| *** annegent_ has quit IRC | 16:43 | |
| fungi | looks like we're currently running ~2hr wait for devstack-trusty nodes in the check pipeline | 16:43 |
| *** dkranz has quit IRC | 16:43 | |
| *** mriedem has quit IRC | 16:44 | |
| *** bdpayne_ has quit IRC | 16:44 | |
| *** hogepodge has quit IRC | 16:44 | |
| clarkb | fungi: do we know what the wait was before the nodepool restart when under load? | 16:44 |
| *** arnaud has joined #openstack-infra | 16:45 | |
| fungi | clarkb: it was probably about the same at peak yesterday. don't really have a good statistical sample to go on though | 16:45 |
| fungi | mainly waiting to see whether it gets substantially worse once we've been running with the new algorithm for a few hours | 16:46 |
| clarkb | you guys won't throw produce at me if I suggest another node.js service as a potential alternative to znc as a service will you? | 16:46 |
| clarkb | I guess I can respond to the thread and see how many tomatos you have | 16:47 |
| fungi | i want to turn it into a service-as-a-service meta-discussion | 16:48 |
| fungi | discussion-as-a-service | 16:48 |
| *** hdd has quit IRC | 16:49 | |
| mordred | clarkb: node.js is a thing. we might as well admit it | 16:50 |
| clarkb | mordred: there is a difference between storyboard using it to build and test and running production applications on it | 16:51 |
| clarkb | mordred: but I agree, I don't expect it to go anywhere and it is pretty damn popular | 16:51 |
| *** hogepodge has joined #openstack-infra | 16:51 | |
| mordred | avoiding it isn't helpful generally | 16:52 |
| jeblair | clarkb: i'm pretty sure it's already dead to the hacker news crowd | 16:52 |
| fungi | i want a sandboxed interpreter for basic, and a package manager for my basic applications | 16:52 |
| *** doude_ has quit IRC | 16:52 | |
| morganfainberg | mordred, welcome back btw. | 16:52 |
| *** andreykurilin_ has joined #openstack-infra | 16:52 | |
| mordred | morganfainberg: not really back ... I'm at the beach phoning over breakfast :) | 16:53 |
| mordred | morganfainberg: but thanks! | 16:53 |
| jeblair | morganfainberg: this may be the new normal for mordred | 16:53 |
| morganfainberg | jeblair, hehe | 16:53 |
| morganfainberg | mordred, beach, breakfast, gee, what an awful way to spend a wednesday :) | 16:54 |
| clarkb | is it wednesday already | 16:54 |
| morganfainberg | mordred, well enjoy the rest of the "not really here" time :) | 16:55 |
| *** paulrad has joined #openstack-infra | 16:56 | |
| fungi | clarkb: lies | 16:56 |
| *** alkari has joined #openstack-infra | 16:58 | |
| clarkb | pretty sure it is only tuesday | 16:58 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 16:58 | |
| clarkb | monday was a time warp | 16:58 |
| fungi | oxymoronic labor holiday is oxymoronic | 16:59 |
| *** emagana has quit IRC | 16:59 | |
| *** emagana has joined #openstack-infra | 16:59 | |
| *** tsg has quit IRC | 17:00 | |
| *** marun_ has quit IRC | 17:00 | |
| *** dougwig_ is now known as dougwig | 17:00 | |
| *** tonytan4ever has quit IRC | 17:00 | |
| *** marcoemorais has quit IRC | 17:00 | |
| *** derekh has quit IRC | 17:01 | |
| *** paulrad has quit IRC | 17:01 | |
| *** tsg has joined #openstack-infra | 17:02 | |
| *** marcoemorais1 has joined #openstack-infra | 17:02 | |
| *** amcrn_ has joined #openstack-infra | 17:02 | |
| *** gema has quit IRC | 17:02 | |
| *** jcoufal has quit IRC | 17:03 | |
| *** gokrokve_ has joined #openstack-infra | 17:03 | |
| *** emagana has quit IRC | 17:04 | |
| *** tgohad has joined #openstack-infra | 17:04 | |
| *** emagana has joined #openstack-infra | 17:05 | |
| *** gokrokve has quit IRC | 17:05 | |
| *** marcoemorais1 has quit IRC | 17:06 | |
| *** amcrn has quit IRC | 17:06 | |
| *** marcoemorais2 has joined #openstack-infra | 17:06 | |
| *** marcoemorais2 has quit IRC | 17:06 | |
| *** amcrn_ is now known as amcrn | 17:06 | |
| *** tsg has quit IRC | 17:07 | |
| *** marcoemorais1 has joined #openstack-infra | 17:07 | |
| nibalizer | mordred: wb! | 17:07 |
| *** gondoi is now known as zz_gondoi | 17:07 | |
| *** paulrad has joined #openstack-infra | 17:08 | |
| *** bdpayne has joined #openstack-infra | 17:08 | |
| *** dkranz has joined #openstack-infra | 17:09 | |
| *** bdpayne has quit IRC | 17:10 | |
| *** bdpayne has joined #openstack-infra | 17:10 | |
| *** amotoki has quit IRC | 17:10 | |
| *** palar has joined #openstack-infra | 17:11 | |
| *** yjiang5 has joined #openstack-infra | 17:12 | |
| *** paulrad has quit IRC | 17:12 | |
| *** marun_ has joined #openstack-infra | 17:14 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for critical keystone regression https://review.openstack.org/118668 | 17:14 |
| *** Ryan_Lane has joined #openstack-infra | 17:14 | |
| *** flaviof is now known as flaviof_zzz | 17:14 | |
| *** chuckC has quit IRC | 17:16 | |
| *** mwagner_lap has joined #openstack-infra | 17:16 | |
| jeblair | russellb: for 113349, do you have plans to add the f20 preview node type to hpcloud? | 17:16 |
| openstackgerrit | A change was merged to openstack-infra/config: Start running bashate on config (nonvoting) https://review.openstack.org/112446 | 17:17 |
| jeblair | sdague, dtroyer: ^ that changes the way bashate runs on devstack | 17:18 |
| *** yjiang5 has quit IRC | 17:18 | |
| *** habib has quit IRC | 17:19 | |
| pelix1 | so I identified the root cause of what in https://review.openstack.org/#/c/87810/ broke the compare-xml, but since that was reverted, I have to use a new changeId. Should I include the old changeId into the commit message in the body so that there is a reference back to the original? | 17:19 |
| *** pelix1 is now known as pelix | 17:19 | |
| openstackgerrit | A change was merged to openstack-infra/config: f20 should run full tempest runs https://review.openstack.org/112670 | 17:20 |
| clarkb | pelix: sure, it may help reviewers and historians down the line | 17:20 |
| *** Ajaeger has joined #openstack-infra | 17:20 | |
| *** harlowja_away is now known as harlowja | 17:20 | |
| pelix | clarkb: sure just wondered if there was a standard convention | 17:20 |
| Ajaeger | I see with glance_store a problem uploading to pypi: https://jenkins.openstack.org/job/glance_store-pypi-both-upload/2/console | 17:20 |
| Ajaeger | fungi, clarkb: Any ideas about this error "403 Client Error: You are not allowed to edit 'glance-store' package information" | 17:21 |
| clarkb | pelix: I don't think so, generally if you are undoing a revert with edits I would refer back to the origianl change(s) | 17:21 |
| clarkb | Ajaeger: pypi user permissions are probably not correct I will check shortly | 17:21 |
| Ajaeger | clarkb: thanks! | 17:21 |
| clarkb | jeblair: nibalizer: we should try to get changes like https://review.openstack.org/#/c/107243/ in soonish, as that will allow us to keep moving on the puppet3 front right? | 17:22 |
| nibalizer | clarkb: thats not a blocker | 17:22 |
| nibalizer | its a good thing to ahve | 17:22 |
| nibalizer | but its just noise in the logs, not hard failure | 17:22 |
| clarkb | nibalizer: oh ok | 17:22 |
| clarkb | nibalizer: the stack for the puppet3 master has a bunch of those type of changes at the bottom though | 17:23 |
| *** yjiang5 has joined #openstack-infra | 17:23 | |
| nibalizer | im totally with you on not touching gerrit with a ten foot pole unless we have to | 17:23 |
| clarkb | oh but this is gerrit specifically ok | 17:23 |
| nibalizer | yea, i did that because there was a lot of red and I couldn't easily differentiate between warn and error | 17:23 |
| nibalizer | and we should merge that, if we can | 17:23 |
| clarkb | Ajaeger: so the sdist worked https://pypi.python.org/pypi/glance_store/0.1.3 | 17:23 |
| clarkb | Ajaeger: this could be a twine bug perhaps? | 17:24 |
| Ajaeger | clarkb: I don't follow - no clue about twine... | 17:24 |
| *** gema has joined #openstack-infra | 17:25 | |
| clarkb | Ajaeger: twine is the utility we use to upload to pypi. dstufft wrote it. Basically the utility is that you don't have to run arbitrary code via python setup.py and get more consistency this way | 17:25 |
| Alex_Gaynor | clarkb, Ajaeger: I seem to recall twine has a bug with packages that have -s in their names /cc dstanek | 17:25 |
| Alex_Gaynor | err, dstufft | 17:25 |
| clarkb | Alex_Gaynor: huh that sounds like a funny bug | 17:25 |
| Ajaeger | clarkb, Alex_Gaynor Thanks for the explanation. | 17:25 |
| Alex_Gaynor | Actually, looks like it was underscores: https://github.com/pypa/twine/issues/47 | 17:25 |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/devstack-gate: Set required devstack variables for Ironic+grenade https://review.openstack.org/116761 | 17:26 |
| Ajaeger | Alex_Gaynor: "_s" | 17:26 |
| *** Sukhdev has quit IRC | 17:26 | |
| Alex_Gaynor | Can someone tell dstufft it's his job to fix this bug, it's been annoying us for cryptography as well :-) | 17:27 |
| Ajaeger | dstufft: ping | 17:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Add f20-devstack-virt-preview node type https://review.openstack.org/113349 | 17:27 |
| *** yjiang5 has quit IRC | 17:28 | |
| *** yolanda has quit IRC | 17:28 | |
| clarkb | Ajaeger: thankfully the sdist should work just fine for now | 17:28 |
| jeblair | well, it has impacted openstack now :/ | 17:28 |
| clarkb | Ajaeger: so I think we can get away with the current state, but should fix that twine bug | 17:29 |
| Ajaeger | clarkb: good to know | 17:29 |
| Ajaeger | clarkb: should I file a bug or is the one on twine enough? | 17:29 |
| clarkb | I am inclined to say the one on twine is enough, unless we want to make changes to workaround that bug on our side | 17:30 |
| russellb | jeblair: re f20 node type, i certainly can, was just keeping it small since it's an experimental thing to start with | 17:30 |
| *** cipcosma has quit IRC | 17:30 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/config: Adds Ironic parallel Tempest job https://review.openstack.org/118700 | 17:30 |
| nibalizer | hey Alex_Gaynor, anita asked me to take over your livegrep work since it seems to have stalled out, is that okay with you? | 17:30 |
| russellb | jeblair: i'm also working hard in the background to see if i can get some things going in the centos community to allow this to eventually be centos 7 based instead | 17:30 |
| clarkb | adam_g: that reminds me, how is the sideways upgrade going? | 17:30 |
| russellb | fyi i guess | 17:30 |
| Alex_Gaynor | nibalizer: thats fine; sorry I haven't had more time for it, let me know if there's anything I can do | 17:30 |
| Ajaeger | clarkb: ok | 17:30 |
| nibalizer | Alex_Gaynor: sure, i just didn't want to step on toes | 17:31 |
| krotscheck | Can someone give me a pointer why gate-config-layout died on this review? It seems like a really innocuous change…. https://review.openstack.org/#/c/118435/ | 17:31 |
| jeblair | russellb: no problem. we'll just want to make sure it exists on hpcloud before we make it not-experimental. also ++ on centos7, that means we can continue to test those features consistently throughout the openstack release lifecycle | 17:31 |
| Alex_Gaynor | nibalizer: not at all; thanks for running with it! | 17:31 |
| *** imeerovi has joined #openstack-infra | 17:31 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:31 | |
| nibalizer | krotscheck: gate-config-layout was just derping at that time | 17:31 |
| jeblair | russellb: also, i -1d the change to add the job because it went into silent, not experimental. otherwise all looks good | 17:31 |
| nibalizer | give it a recheck and you should be ok | 17:31 |
| clarkb | krotscheck: there was a bad commit to JJB that broke that job. It has been reverted so you should be fine to recheck | 17:31 |
| jeblair | clarkb: where did we end up on the firewalld question? 113884 | 17:32 |
| jeblair | clarkb: now that devstack has it fixed, do we also want to yank it from our base images, or do we care? | 17:32 |
| krotscheck | clarkb: Derp… Did recheck stop working? | 17:33 |
| *** yjiang5 has joined #openstack-infra | 17:33 | |
| clarkb | krotscheck: no it may just take some time with the feature freeze backlogs | 17:33 |
| clarkb | jeblair: I decided it was a harmless change that would give us consistency in our images | 17:33 |
| clarkb | jeblair: but didn't realize that devstack had corrected on that side when I +2'd | 17:33 |
| clarkb | jeblair: so I think we can probably just abandon that change | 17:34 |
| *** mfink has joined #openstack-infra | 17:34 | |
| jeblair | krotscheck: we also don't leave 'starting check jobs' comments anymore | 17:34 |
| nibalizer | krotscheck: yea what jeblair said | 17:35 |
| *** e0ne has quit IRC | 17:35 | |
| nibalizer | krotscheck: also clarkb and i are looking sideways at this node.js app called 'subway' that does web irc client w/ persistent session | 17:35 |
| *** skath_ has left #openstack-infra | 17:35 | |
| nibalizer | have you heard of it? | 17:35 |
| *** ashaeron has joined #openstack-infra | 17:36 | |
| adam_g | clarkb, its going okay. im hacking in things required to be set by DSG as a patch ontop of the main grenade patch (which still needs review) https://review.openstack.org/#/c/116761/ | 17:36 |
| adam_g | i mean https://review.openstack.org/#/c/118515/ | 17:37 |
| *** paulrad has joined #openstack-infra | 17:37 | |
| *** terryw has quit IRC | 17:37 | |
| adam_g | there seems to be an issue with grenade and ENABLE_HTTPD_MOD_WSGI_SERVICES=True | 17:37 |
| krotscheck | nibalizer: You lost me at “node.js app" | 17:38 |
| *** annegent_ has joined #openstack-infra | 17:38 | |
| adam_g | which AFAICS was not something that stable/icehouse ever deployed, so this is the first time grenade is seeing it. | 17:38 |
| *** alexpilotti has quit IRC | 17:39 | |
| clarkb | adam_g: now that I am looking at that change again (116761) when you modify the localrc to prevent exercising the old side, should you be modifying $BASE/old/grenade/localrc instead of $BASE/new/grenade/localrc? | 17:39 |
| nibalizer | krotscheck: lulz :D | 17:40 |
| clarkb | adam_g: I am not sure which side exercising stuff falls on | 17:40 |
| clarkb | sdague: ^ | 17:41 |
| krotscheck | nibalizer: And then I looked it up and the readme lost me at “mongo” | 17:41 |
| adam_g | clarkb, thats not clear to me either :| i dont see any reference to $BASE/old/grenade/localrc | 17:41 |
| *** alexpilotti has joined #openstack-infra | 17:41 | |
| *** alkari has quit IRC | 17:41 | |
| clarkb | krotscheck: mongo isn't used anymore | 17:41 |
| *** alkari has joined #openstack-infra | 17:41 | |
| *** paulrad has quit IRC | 17:41 | |
| clarkb | krotscheck: and I don't see it in the readme | 17:42 |
| adam_g | clarkb, but things that would be required by the 'old' side get written to $BASE/new/grenade/localrc (ie, BASE_*), which made me think $BASE/new/grenade/localrc is the single localrc used by grenade (there isn't an old/grenade and new/grenade) | 17:42 |
| clarkb | oh I see | 17:42 |
| clarkb | because grenade has a complete view | 17:42 |
| clarkb | and old vs new is just for the things being upgraded | 17:42 |
| adam_g | yeah. actually | 17:43 |
| *** annegent_ has quit IRC | 17:43 | |
| adam_g | the slave runs grenade from $BASE/new/grande | 17:43 |
| clarkb | yup | 17:44 |
| clarkb | I am +2 on the d-g change | 17:44 |
| clarkb | I don't have any ideas on the wsgi stuff | 17:44 |
| clarkb | MOD_WSGI_SERVICES is keystone and horizon? | 17:44 |
| adam_g | clarkb, im probably going to keep that change updated iwth other things i may find that need to get set. | 17:44 |
| adam_g | yeah, but keystone is what is failing to start on the new side | 17:44 |
| clarkb | morganfainberg: ^ maybe you have some ideas | 17:44 |
| clarkb | adam_g: that works for me | 17:45 |
| clarkb | adam_g: maybe WIP it until you are happy with it? | 17:45 |
| adam_g | i saw this locally and punted on it and disabled it, since i saw no other grenade run is using MOD_WSGI yet | 17:45 |
| *** hdd has joined #openstack-infra | 17:46 | |
| *** dane_leblanc has joined #openstack-infra | 17:46 | |
| clarkb | adam_g: with wsgi disabled are you able to do an upgrade and now it is a matter of getting the tests to pass? | 17:46 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Make JJB python 3 compatible https://review.openstack.org/118703 | 17:46 |
| clarkb | pelix: why did the popitem fail? | 17:47 |
| clarkb | pelix: do we know? | 17:47 |
| *** pcrews has quit IRC | 17:47 | |
| pelix | Because we shouldn't have used it :) | 17:47 |
| *** ashaeron has quit IRC | 17:47 | |
| adam_g | clarkb, with wsgi enabled, devstack fails to start keystone on the new side (or at least, its curl check of the service fails, the apache log actually looks okay). it works locally when disabled, checking in experimental now | 17:47 |
| pelix | it removed data from the original dict that gets reused on subsequent iterations since the dicts are never copied in between | 17:48 |
| clarkb | pelix: oh so it was "recursively" digging to the bottom of a dict | 17:48 |
| clarkb | pelix: then it failed | 17:48 |
| clarkb | whereas the iterator will raise iteration exception and we continue properly | 17:48 |
| adam_g | clarkb, beyond that, there are two tempest patches are +A'd and be merging today, that should get tests passing entirely | 17:49 |
| clarkb | adam_g: nice! | 17:49 |
| pelix | Not quite, it was operating on the source dict defining the job group. If the dict was copied, both approaches work, but since it's using references upon a second usage, it wouldn't find the required job-group | 17:49 |
| adam_g | clarkb, the biggest blocker now it seems is getting the grenade patch reviewed @ https://review.openstack.org/#/c/111859/ | 17:50 |
| fungi | jeblair: we're getting a ton of "Exception: Manager rax-foo is no longer running" from nodepoold (for all three rax regions) | 17:50 |
| jeblair | fungi: probably related to the nodepool config change i just approved | 17:50 |
| fungi | oh | 17:50 |
| fungi | "ERROR nodepool.ProviderManager: Unable to list servers while waiting for xxx will retry" | 17:50 |
| clarkb | pelix: gotcha, thanks | 17:51 |
| krotscheck | clarkb: So, using “recheck” in a comment is supposed to clear the jenkins -1, no? | 17:51 |
| clarkb | krotscheck: not anymore | 17:51 |
| krotscheck | ….alright | 17:51 |
| clarkb | krotscheck: we removed the noise of reseting the vote | 17:51 |
| clarkb | krotscheck: that was an implementation detail for the freshness checks which we aren't doing | 17:51 |
| krotscheck | Odd | 17:51 |
| jeblair | krotscheck: care to elaborate? | 17:52 |
| krotscheck | Well, usually, when a user does something, they want some indication that the thing they did was useful. But now adding “recheck” to a review provides no user indicator that Things-Are-Happening(TM) | 17:52 |
| *** morganfainberg is now known as morganfainberg_Z | 17:53 | |
| *** arnaud has quit IRC | 17:53 | |
| jeblair | krotscheck: leaving comments is a terrible user interface for something like this. we should neither encourage nor leave more comments ourselves than are absolutely necessary | 17:53 |
| fungi | jeblair: oh, yep, started complaining at 17:40:37 so that was probably right about when it got applied | 17:53 |
| *** tonytan4ever has joined #openstack-infra | 17:54 | |
| fungi | timestamp on nodepool.yaml confirms | 17:54 |
| krotscheck | jeblair: Ok, so if I want to recheck, and I shouldn’t leave a comment, how do I do that/ | 17:54 |
| krotscheck | ? | 17:54 |
| jeblair | krotscheck: it's not possible yet, because no one has written an alternate interface. in the mean time, we try to minimize it. because comments are there fore people to talk about the change | 17:54 |
| *** koolhead_ has quit IRC | 17:55 | |
| jeblair | krotscheck: if you would like to enhance zuul to support out-of-band enqueing, it would be a welcome change | 17:55 |
| krotscheck | jeblair: Ok, then how does the UI indicate to me that my recheck request has been received and is being processed? | 17:55 |
| jeblair | krotscheck: we ran just fine without spamming 'starting' comments for a few years. if you're really curious, you can check http://status.openstack.org/zuul/ and if you would prefer more immediate feedback, then assistance with jheskeths changes to get the zuul status page in order so that it can be used on the review page would be helpful | 17:56 |
| jeblair | krotscheck: i am aware that feedback to users is good. but the costs in this case are high and other concerns outweigh them. | 17:58 |
| krotscheck | Sure! I’ll put that right after getting infra on storyboard :) | 17:58 |
| jeblair | krotscheck: that's the spirit :) | 17:58 |
| * krotscheck wouldn’t mind working on something other than storyboard for a while. | 17:58 | |
| jeblair | cross pollination++ | 18:00 |
| clarkb | fungi: need any help with debugging that? | 18:00 |
| jeblair | fungi: i think the behavior may be that nodes that were being processed will throw those ecxeptions until their processing times out (about 1h) | 18:01 |
| fungi | jeblair: oh! the manager with which they were associated has been replaced | 18:02 |
| *** aj_ has joined #openstack-infra | 18:02 | |
| fungi | yeah, i think that explains it | 18:02 |
| *** tsg has joined #openstack-infra | 18:03 | |
| *** weshay has joined #openstack-infra | 18:03 | |
| *** tgohad has quit IRC | 18:04 | |
| jeblair | fungi, clarkb: we should probably raise a more specific exception there, and use it to escape some of the loops that we're currently spinning in | 18:05 |
| jeblair | since those loops are designed to mask transient errors, and this is not one | 18:05 |
| clarkb | sounds good | 18:05 |
| *** radez is now known as radez_g0n3 | 18:06 | |
| fungi | makes sense | 18:06 |
| *** mriedem1 is now known as mriedem | 18:07 | |
| *** marcoemorais1 has quit IRC | 18:08 | |
| *** aj_ has quit IRC | 18:08 | |
| *** Mithrandir_ is now known as Mithrandir | 18:08 | |
| *** marcoemorais has joined #openstack-infra | 18:09 | |
| *** marun_ has quit IRC | 18:09 | |
| *** e0ne has joined #openstack-infra | 18:10 | |
| bswartz | I have a question | 18:10 |
| bswartz | how can I create tags for a stackforge project? | 18:10 |
| bswartz | I want to create the juno-3 tag for manila | 18:11 |
| *** shayneburgess has joined #openstack-infra | 18:11 | |
| clarkb | bswartz: there is a gerrit ACL that needs to be set up if it isn't already, then members of the group given these permissiosn can push tags to gerrit with `git push gerrit tagnamehere` | 18:11 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Don't retry forever when a provider is stopped https://review.openstack.org/118710 | 18:11 |
| bswartz | clarkb: where can I see the existing acls? | 18:12 |
| clarkb | bswartz: https://review.openstack.org/#/admin/projects/stackforge/manila,access there | 18:12 |
| jeblair | fungi, clarkb: ^ i think that should do it. more generally, i think nodepool is violating some early assumptions about how providers would work and probably needs a bit of an overhaul there. | 18:12 |
| *** palar_ has joined #openstack-infra | 18:12 | |
| clarkb | bswartz: the manila release group has the ability to push tags | 18:12 |
| clarkb | bswartz: things to keep in mind. Make sure that gerrit knows about the commit you have tagged and that it has already merged to a branch | 18:12 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Define -T/--no-topic to disable review submit with topic https://review.openstack.org/114038 | 18:13 |
| clarkb | bswartz: sign your tags, and treat them as immutable (you don't want to move a tag later) | 18:13 |
| fungi | bswartz: see also https://wiki.openstack.org/wiki/GerritJenkinsGit#Tagging_a_Release | 18:13 |
| bswartz | clarkb: when I tried to tag I got a gpg error | 18:13 |
| bswartz | Is GPG a requirement? | 18:13 |
| fungi | bswartz: very much so, yes | 18:13 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: -F/--force-rebase has no effect if rebase is disabled by config https://review.openstack.org/117489 | 18:13 |
| bswartz | ok | 18:13 |
| bswartz | never needed gpg for anything else | 18:13 |
| bswartz | I'll figure it out | 18:14 |
| fungi | bswartz: also we want to rename your git repositories from stackforge to openstack sometime soon. vponomaryov suggested infra should just pick a time convenient for us and let you know when that's happening, perhaps via a thread on the -dev ml tagged [manila] | 18:15 |
| bswartz | yes I'm aware of that | 18:15 |
| *** palar has quit IRC | 18:15 | |
| fungi | okay, awesome | 18:15 |
| bswartz | I figure it's better to do the rename after J-3 | 18:15 |
| fungi | yep, we have to take zuul and gerrit offline briefly for the rename, so we too will want to wait until after the milestone rush has subsided | 18:16 |
| bswartz | regarding gpg, is there a place I have to register my public key? | 18:16 |
| fungi | bswartz: i wrote this which may help you... https://wiki.openstack.org/wiki/OpenPGP_Web_of_Trust | 18:16 |
| *** mspreitz has joined #openstack-infra | 18:17 | |
| *** annegent_ has joined #openstack-infra | 18:18 | |
| *** jlk_clone has joined #openstack-infra | 18:18 | |
| *** masayukig_ has quit IRC | 18:18 | |
| fungi | i suppose now that we have that, i can link it in the tagging instreuctions instead of the fedoraproject link currently there | 18:18 |
| *** jlk_clone is now known as jlk | 18:19 | |
| *** jlk has joined #openstack-infra | 18:19 | |
| jeblair | jesusaurus: on 115116 my math indicates a fator of 16 reduction in sample storage, not 30. | 18:20 |
| jeblair | factor | 18:20 |
| *** radez_g0n3 is now known as radez | 18:21 | |
| *** HeOS has quit IRC | 18:21 | |
| *** ZZelle_ has joined #openstack-infra | 18:22 | |
| *** adalbas has quit IRC | 18:22 | |
| *** habib has joined #openstack-infra | 18:22 | |
| *** ashaeron has joined #openstack-infra | 18:23 | |
| *** masayukig has joined #openstack-infra | 18:25 | |
| jesusaurus | did i derp that math? i might've | 18:25 |
| ekarlso | phschwartz: around ? | 18:26 |
| jeblair | jesusaurus: i come up with 19206 samples for your proposal, compared to 275220 under current config | 18:27 |
| fungi | ekarlso: he's apparently on a family vacation all this week | 18:27 |
| ekarlso | boooo | 18:27 |
| ekarlso | darn it | 18:27 |
| *** arnaud has joined #openstack-infra | 18:27 | |
| *** marcoemorais has quit IRC | 18:28 | |
| jeblair | jesusaurus: 360+10080+8766=19206 | 18:28 |
| jeblair | jesusaurus: let me know if you think i did something wrong; i have a spreadsheet up calculating alternatives :) | 18:28 |
| *** pcrews has joined #openstack-infra | 18:28 | |
| *** ashaeron has quit IRC | 18:28 | |
| jesusaurus | jeblair: ah, i was just doing some back-of-the-napkin comparisons of 10-minute for 5 years vs 1-hour for 1-year and assumed the others were negligible | 18:28 |
| *** marcoemorais has joined #openstack-infra | 18:28 | |
| jesusaurus | that was probably a bad assumption on my part | 18:29 |
| *** gokrokve has joined #openstack-infra | 18:29 | |
| *** marcoemorais has quit IRC | 18:29 | |
| jeblair | jesusaurus: they are actually -- only changing the year is 0.07x vs 0.06x | 18:29 |
| jeblair | that was probably incomprehensible let me try again | 18:29 |
| *** marcoemorais has joined #openstack-infra | 18:29 | |
| jeblair | jesusaurus: they are actually -- only changing the 5y to 1y retention is 0.07x compared to 0.06x if you change that and also the 6h-1h retention | 18:30 |
| jesusaurus | i dont follow where 0.07x and 0.06x are coming from | 18:31 |
| *** gokrokve_ has quit IRC | 18:31 | |
| jeblair | jesusaurus: sorry, i inverted the math because i prefer to think the other way :) 19206/275214=0.069... | 18:33 |
| *** palar_ has quit IRC | 18:33 | |
| jeblair | 6.9% of current usage | 18:33 |
| *** gokrokve has quit IRC | 18:33 | |
| *** rushiagr is now known as rushiagr_away | 18:33 | |
| jesusaurus | the way i got 30 is: 5yr to 1yr is a factor of 5; 10minute to 1hour is a factor of 6. 5*6 gives me 30. very rough math on my part, and i wouldnt be surprised if i didnt factor in something significant | 18:34 |
| *** adalbas has joined #openstack-infra | 18:34 | |
| *** emagana has quit IRC | 18:34 | |
| *** emagana has joined #openstack-infra | 18:35 | |
| *** daxatlas has quit IRC | 18:35 | |
| *** habib has quit IRC | 18:35 | |
| *** emagana has quit IRC | 18:36 | |
| *** emagana has joined #openstack-infra | 18:36 | |
| jeblair | jesusaurus: ah, yes. that's correct (it gives you the right number for the yearly metric), but at that point, that metric then accounts for ~50% of the total storage, so that explains why you're off by about 2x. | 18:37 |
| jesusaurus | ahh | 18:39 |
| clarkb | fyi logstash doesn't seem to be indexing all of the data I expect it to be indexing (details over in -qa) so I am going to poke at that after lunch | 18:39 |
| jesusaurus | so yeah, assuming the others were negligible was an error on my part | 18:39 |
| jeblair | okay, i'll have some alternate proposals on that review in a minute | 18:40 |
| *** flaviof_zzz is now known as flaviof | 18:40 | |
| *** _nadya_ has joined #openstack-infra | 18:41 | |
| jesusaurus | clarkb: anything i can help with? | 18:41 |
| clarkb | jesusaurus: possibly. I need to dig through logs first to see why it is failing but basically screen-n-cond.txt isn't ending up in logstash but the old screen n cond for grenade does | 18:41 |
| clarkb | jesusaurus: so I don't think it is a parser failure, probably something earlier in the pipeline | 18:42 |
| jesusaurus | huh | 18:42 |
| clarkb | jesusaurus: I will let you if/when I find anything | 18:42 |
| *** zul has quit IRC | 18:43 | |
| *** lttrl has quit IRC | 18:43 | |
| *** annegent_ has quit IRC | 18:44 | |
| *** _nadya_ has quit IRC | 18:46 | |
| *** e0ne has quit IRC | 18:47 | |
| *** zul has joined #openstack-infra | 18:48 | |
| *** amcrn_ has joined #openstack-infra | 18:51 | |
| *** amcrn has quit IRC | 18:54 | |
| *** bswartz has quit IRC | 18:54 | |
| *** bswartz has joined #openstack-infra | 18:55 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/git-review: ../../message https://review.openstack.org/118743 | 18:55 |
| *** bswartz has quit IRC | 18:56 | |
| *** bswartz has joined #openstack-infra | 18:56 | |
| jeblair | jesusaurus: okay, left comments is 115116 | 18:56 |
| jesusaurus | jeblair: thanks | 18:56 |
| jeblair | notmyname: interested in in 115250 ? | 18:58 |
| notmyname | ? | 18:58 |
| *** amuller__ has quit IRC | 18:58 | |
| notmyname | jeblair: oh. umm...just what I said in the comments. is there a discussions somewhere else? | 18:59 |
| jeblair | notmyname: oh did i miss your comments? | 18:59 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/git-review: Work toward Python 3.4 support and testing https://review.openstack.org/118743 | 19:00 |
| notmyname | jeblair: I see it at the bottom (aug 27) | 19:00 |
| sdague | adam_g: is there a reason https://review.openstack.org/#/c/116761 is WIP? | 19:01 |
| jeblair | notmyname: whoops, sorry about that | 19:01 |
| notmyname | jeblair: no worries | 19:01 |
| notmyname | jeblair: can you give any insight or context for https://review.openstack.org/#/c/97092 | 19:02 |
| adam_g | sdague, still fleshing out exactly whats needed in devstack-gate, based on attempts at getting the sideways job running in experimental (https://review.openstack.org/#/c/118515/) | 19:02 |
| *** derekh has joined #openstack-infra | 19:03 | |
| sdague | adam_g: ok | 19:03 |
| sdague | adam_g: are there other patches you feel are ready for review related to the ironic sideways effort? | 19:03 |
| adam_g | sdague, the main grenade patch @ https://review.openstack.org/#/c/111859/ | 19:04 |
| adam_g | sdague, i just put up two fixes for https://bugs.launchpad.net/devstack/+bug/1365105 to devstack and grenade that are relevant but can be worked around in the meantime by disabling mod wsgi | 19:04 |
| uvirtbot | Launchpad bug 1365105 in grenade "grenade fails to upgrade keystone when mod wsgi is enabled" [Undecided,In progress] | 19:04 |
| *** e0ne has joined #openstack-infra | 19:05 | |
| *** Ajaeger has quit IRC | 19:06 | |
| adam_g | sdague, all required tempest changes are now +A'd. https://review.openstack.org/#/c/107814/ would help with related tempest work outside of grenade | 19:06 |
| sdague | nova_bm_create_node_and_port is something I envisioned would actually be mysql < ironic-dump.sql | 19:08 |
| jeblair | notmyname: i'd recommend mordred or clarkb; they are more thoroughly versed in the babel machinery. | 19:08 |
| pelix | in https://review.openstack.org/#/c/106617/10/git_review/tests/test_unit.py between the comments on various patches I'm seeing conflicting requirements. Previous comments indicate I should follow the hacking guide at https://github.com/openstack-dev/hacking/blob/ea638e72f2a5d4bb17710dab35aafddc51309a17/hacking/checks/imports.py#L208-L224 but at the same time I'm being asked to move an existing dynamic import of a std lib module after the p | 19:08 |
| *** dmellado has quit IRC | 19:08 | |
| sdague | adam_g: is there a reason for doing mysql inserts inline? | 19:08 |
| notmyname | jeblair: thanks. just curious about hwere it came from | 19:08 |
| *** otherwiseguy has joined #openstack-infra | 19:08 | |
| *** derekh has quit IRC | 19:08 | |
| adam_g | sdague, we're not actually deploying nova-bm and have no API server to accept the data otherwise. we cant really ship a database dump, as we need the nova-bm node inventory to reflect the 'baremetal' VMs we'll end up using post-migration | 19:09 |
| pelix | Is there a convention, is it alright in this case to move imports not part of your patch, and should I move a std lib import to the end of all the imports because it's a dynamic import as opposed to a static one, or should I keep it grouped with the other std lib imports? | 19:09 |
| *** paulrad has joined #openstack-infra | 19:09 | |
| * clarkb sees he just got volunteered for a thing | 19:10 | |
| mordred | notmyname: we copied most of the babel stuff from swift, fwiw :) | 19:11 |
| *** dmellado has joined #openstack-infra | 19:11 | |
| clarkb | notmyname: re 97092, I believe we need/want those files to be inlcuded in our sdists. We get that cheap in that location iirc | 19:11 |
| clarkb | notmyname: the exact location probably doesn't matter, but if we are consistent it makes the packaging and installation of .po(t) files much easier | 19:11 |
| adam_g | sdague, also, even if devstack could set us up a nova-bm API server, it would have been stopped by the time that migration stuff ran | 19:11 |
| mordred | also, what clarkb says | 19:12 |
| sdague | adam_g: ok, that's sufficient explanation for me | 19:12 |
| *** radez is now known as radez_g0n3 | 19:15 | |
| *** e0ne has quit IRC | 19:16 | |
| clarkb | pelix: I would keep it grouped, I blieve that is what the rest of openstack tries to do | 19:17 |
| clarkb | pelix: unless you need it to be dynamic at post "parse" time | 19:17 |
| pelix | nope, just at parse time | 19:17 |
| pelix | clarkb: thanks | 19:19 |
| *** e0ne has joined #openstack-infra | 19:19 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/git-review: Enable color support based on tty and config https://review.openstack.org/106617 | 19:19 |
| *** homeless has quit IRC | 19:19 | |
| *** gokrokve has joined #openstack-infra | 19:19 | |
| *** homeless has joined #openstack-infra | 19:19 | |
| pelix | clarkb: apparently in neutron dynamic imports are done afterwards, but I'm guessing that's not standardized? | 19:20 |
| *** emagana has quit IRC | 19:20 | |
| clarkb | pelix: huh I thought hacking checked that. maybe hacking doesn't deal with them | 19:20 |
| *** zeus has quit IRC | 19:21 | |
| *** zeus has joined #openstack-infra | 19:22 | |
| *** zeus has quit IRC | 19:22 | |
| *** zeus has joined #openstack-infra | 19:22 | |
| *** marun_ has joined #openstack-infra | 19:24 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/devstack-gate: Deprecate API_RATE_LIMIT setting for Tempest runs https://review.openstack.org/116001 | 19:27 |
| *** sabeen has joined #openstack-infra | 19:28 | |
| *** _nadya_ has joined #openstack-infra | 19:29 | |
| pelix | clarkb: no worries, I've just asked that someone make a clear statement in the review as to whether the suggestion from the reviews should override the documented style. just one of those cases that opinion doesn't work well | 19:31 |
| clarkb | pelix: ya I am fine with it either way really | 19:31 |
| *** _nadya_ has quit IRC | 19:33 | |
| *** _nadya_ has joined #openstack-infra | 19:34 | |
| *** ociuhandu has quit IRC | 19:34 | |
| *** jergerber has joined #openstack-infra | 19:35 | |
| *** homeless has quit IRC | 19:38 | |
| *** Sincler has quit IRC | 19:39 | |
| *** IAX has quit IRC | 19:39 | |
| *** gargola has quit IRC | 19:39 | |
| *** _nadya_ has quit IRC | 19:39 | |
| *** hashar has joined #openstack-infra | 19:39 | |
| *** gargola has joined #openstack-infra | 19:39 | |
| *** homeless has joined #openstack-infra | 19:40 | |
| *** IAX has joined #openstack-infra | 19:40 | |
| *** Sincler has joined #openstack-infra | 19:40 | |
| *** paulrad has quit IRC | 19:41 | |
| *** che-arne has quit IRC | 19:41 | |
| *** paulrad has joined #openstack-infra | 19:41 | |
| *** marun_ has quit IRC | 19:42 | |
| openstackgerrit | Andrey Kurilin proposed a change to openstack/requirements: restrict requests version with !=2.4.0 https://review.openstack.org/118332 | 19:45 |
| *** paulrad has quit IRC | 19:45 | |
| *** weshay has quit IRC | 19:46 | |
| *** paulrad has joined #openstack-infra | 19:46 | |
| *** sballe has joined #openstack-infra | 19:47 | |
| clarkb | before I grab lunch, when do elections happen? is that real soon now? | 19:48 |
| *** hashar has quit IRC | 19:49 | |
| *** paulrad has quit IRC | 19:50 | |
| *** radez_g0n3 is now known as radez | 19:51 | |
| *** marun_ has joined #openstack-infra | 19:55 | |
| pleia2 | mm lunch, might need to find some place outside, it's september so condo is starting to get toasty in the afternoon | 19:55 |
| anteaya | SergeyLukjanov: there is no correct specific recheck syntax for third party: http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-08-19-19.02.log.html timestamp: 19:48:38 <jeblair> #topic comment syntax for third-party CI recheck (dougwig, from third-party CI meeting) | 19:58 |
| *** tgohad has joined #openstack-infra | 19:59 | |
| *** tsg has quit IRC | 19:59 | |
| *** chuckC has joined #openstack-infra | 19:59 | |
| anteaya | ppai: what do you want me to look at with https://review.openstack.org/#/c/115250/ I am already +1, I can't get anymore +1 than I already am | 19:59 |
| anteaya | I'm back and doing poorly | 20:00 |
| anteaya | let me know if there is anything i need to address | 20:00 |
| pleia2 | anteaya: rest :) | 20:00 |
| jeblair | fungi, clarkb: i'm going to do the mailing list rename now | 20:01 |
| fungi | let me know if you need help | 20:01 |
| pleia2 | thanks jeblair | 20:01 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add lxsli and jonpaul-sullivan to tripleo core https://review.openstack.org/118483 | 20:02 |
| dhellmann | The oslo team is considering a FFE for the AMQP 1.0 driver, esp. since it is optional. However, we'd like to ensure there isn't going to be a major issue with getting real testing in place. kgiusti has put together https://review.openstack.org/#/c/115752/ but it's not clear if that approach is acceptable to the infra/qa teams. Can I get some feedback on the ML thread? | 20:02 |
| jeblair | i'm single-approving the changes because i think we all agreed on the process | 20:02 |
| fungi | yep | 20:02 |
| pleia2 | I'll follow up on the mailing list thread once we know all is working | 20:02 |
| *** paulrad has joined #openstack-infra | 20:03 | |
| *** tsg has joined #openstack-infra | 20:03 | |
| *** annegent_ has joined #openstack-infra | 20:05 | |
| clarkb | dhellmann: iirc ianw has a change up to make it possible. I don't expect any major issues | 20:05 |
| dhellmann | clarkb: ok, great, that's what I wanted to make sure of | 20:06 |
| clarkb | 115752 should probably depend on the addition of the bare-f20 nodetype though | 20:06 |
| *** radez is now known as radez_g0n3 | 20:06 | |
| clarkb | ianw: ^ do you expect problems? I know the postgres thing was something you ran into. crinkle says we should see a release of that module this week | 20:06 |
| *** arosen has joined #openstack-infra | 20:06 | |
| *** e0ne has quit IRC | 20:06 | |
| jeblair | dhellmann: main thing to know about non-centos/lits node types is that we can't test on them for the full openstack lifecycle, so when they go out of support, we will drop them and their associated jobs immediately | 20:07 |
| jeblair | non-centos/lts | 20:07 |
| clarkb | ianw: and if they don't release you tested the work around and I am willing to roll with that | 20:07 |
| dhellmann | when the OS goes out of support, you mean? | 20:07 |
| jeblair | dhellmann: yep | 20:07 |
| *** tgohad has quit IRC | 20:07 | |
| *** stevemar has quit IRC | 20:07 | |
| dhellmann | jeblair: ok, that's good to know. I would expect the use of f20 to be temporary until the packages needed are available for the centos/lts nodes to consume. | 20:08 |
| jeblair | dhellmann: sounds like a plan | 20:08 |
| anteaya | I just took a copy of the subscriber list of -request, just in case | 20:08 |
| arosen | hi, I'm kinda stuck on this tox error. It's telling me: --- import errors --- congressclient.tests.v1.test_policyNon-zero exit code (2) from test listing. If I debug it like i normally do i.e: source .tox/py27/bin/activate; i can import all the imports on this test so I'm not quite sure what the problem is. | 20:08 |
| arosen | Any ideas on how to debug this further? nosetests and python setup.py testr both pass. | 20:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Removes mailing list third-party-request https://review.openstack.org/118508 | 20:09 |
| clarkb | arosen: so `import congressclient.tests.v1.test_policy` succeeds in an python interpreter run out of that venv? | 20:10 |
| *** marcoemorais has quit IRC | 20:10 | |
| *** marcoemorais has joined #openstack-infra | 20:11 | |
| *** marcoemorais has quit IRC | 20:12 | |
| arosen | clarkb: yup :/ http://paste.openstack.org/show/105396/ | 20:12 |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: Define -T/--no-topic to disable review submit with topic https://review.openstack.org/114038 | 20:13 |
| *** marcoemorais1 has joined #openstack-infra | 20:13 | |
| clarkb | arosen: interesting try `.tox/py27/bin/python -m testtools.run discover -t ./ . --list` | 20:15 |
| clarkb | arosen: that is the command that is failing but testtools instead of subunit so that the output comes out human readable. And let us see if that is more verbose | 20:15 |
| arosen | clarkb: it just says failed to import :/ http://paste.openstack.org/show/105397/ | 20:16 |
| arosen | let me try playing around with my testr.conf and see if that helps anything. | 20:17 |
| *** fifieldt_ has joined #openstack-infra | 20:17 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1363103 https://review.openstack.org/117923 | 20:19 |
| uvirtbot | Launchpad bug 1363103 in neutron "OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'INSERT INTO routerl3agentbindings" [High,Confirmed] https://launchpad.net/bugs/1363103 | 20:19 |
| *** paulrad has quit IRC | 20:19 | |
| *** paulrad has joined #openstack-infra | 20:19 | |
| jeblair | anteaya, fungi: puppet change is in place. i'm stopping exim and performing the steps listed here (in part 2): http://wiki.list.org/pages/viewpage.action?pageId=4030617 | 20:20 |
| *** HeOS has joined #openstack-infra | 20:20 | |
| *** fifieldt has quit IRC | 20:21 | |
| * anteaya clicks on the linked wikipage | 20:22 | |
| *** mfink has quit IRC | 20:23 | |
| *** julim has quit IRC | 20:24 | |
| *** paulrad has quit IRC | 20:24 | |
| *** ashaeron has joined #openstack-infra | 20:25 | |
| arosen | clarkb: hrm same thing. Is there anyway to get a trace where the import failed :/ | 20:25 |
| jeblair | complete, and restarted exim | 20:26 |
| jeblair | approving change #2 | 20:26 |
| clarkb | arosen: usign the interpreter is what I do which is why I am surprised it doesn't work | 20:27 |
| clarkb | arosen: is this code available for others to reproduce? | 20:27 |
| pleia2 | jeblair: looks good \o/ | 20:27 |
| jeblair | in 10 mins, we'll see if puppet is happy with the list | 20:27 |
| *** sballe has quit IRC | 20:28 | |
| arosen | clarkb: one sec i think i might have found out why with strace | 20:28 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-dev/hacking: Work toward Python 3.4 support and testing https://review.openstack.org/118765 | 20:28 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-dev/pbr: Work toward Python 3.4 support and testing https://review.openstack.org/118766 | 20:28 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/elastic-recheck: Work toward Python 3.4 support and testing https://review.openstack.org/118767 | 20:28 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/reviewstats: Work toward Python 3.4 support and testing https://review.openstack.org/118768 | 20:28 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/storyboard: Work toward Python 3.4 support and testing https://review.openstack.org/118769 | 20:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Creates third-party-requests mailing list https://review.openstack.org/118509 | 20:29 |
| arosen | clarkb: sHere's the patch: https://review.openstack.org/#/c/104375/ | 20:31 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-dev/cookiecutter: warn against sorting requirements https://review.openstack.org/118838 | 20:32 |
| *** cdent has quit IRC | 20:32 | |
| *** kurtmartin has joined #openstack-infra | 20:33 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-dev/cookiecutter: Work toward Python 3.4 support and testing https://review.openstack.org/118855 | 20:33 |
| *** mwagner_lap has quit IRC | 20:34 | |
| anteaya | my heart is still tap-dancing from the anesthetic but I can't eat anymore apples or oatmeal | 20:36 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add Python 3.4 and PyPy to Ubuntu Trusty slaves https://review.openstack.org/118859 | 20:37 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add non-voting gate-git-review-python34 job https://review.openstack.org/118860 | 20:37 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Update StackForge documentation to recommend py34 https://review.openstack.org/118861 | 20:37 |
| *** cnesa10 has joined #openstack-infra | 20:37 | |
| pleia2 | http://puppetboard.openstack.org/report/lists.openstack.org/c4772efb0983936c5319f98d48d266cc79dc8dbc \o/ | 20:37 |
| crinkle | clarkb: we're trying to release puppetlabs-postgresql but it will be a major version bump, so I don't know if that helps you | 20:37 |
| dolphm | this fixes a transient and a critical regression in keystone - can someone prioritize it in the gate for us? https://review.openstack.org/#/c/118667/ | 20:37 |
| *** shayneburgess has quit IRC | 20:38 | |
| jeblair | dolphm: can do | 20:39 |
| dolphm | jeblair: thank you | 20:40 |
| *** shayneburgess has joined #openstack-infra | 20:40 | |
| anteaya | pleia2: your welcome email just got sent out from third-party-requests | 20:41 |
| *** shayneburgess has quit IRC | 20:41 | |
| pleia2 | anteaya: I just resent it to the proper address | 20:41 |
| jeblair | dolphm: done | 20:41 |
| pleia2 | http://lists.openstack.org/pipermail/third-party-requests/2014-September/000000.html | 20:41 |
| pleia2 | so yay, works | 20:42 |
| anteaya | pleia2: ah, well it worked | 20:42 |
| clarkb | crinkle: I think ianw said that the module worked as is if we checked it out directly so major version bump may be ok | 20:42 |
| dolphm | jeblair: /salute | 20:42 |
| clarkb | crinkle: we may need to tweak a few things, postgres is less scary to us because its only used on test slaves | 20:42 |
| anteaya | pleia2: I'll try repling | 20:42 |
| pleia2 | anteaya: followed up on dev and infra about the change too (same thread as initial announcement) | 20:42 |
| jeblair | hrm. alien-list doesn't work on hpcloud because of the pseudo regions | 20:43 |
| anteaya | pleia2: yay | 20:43 |
| *** emagana has joined #openstack-infra | 20:43 | |
| *** tonytan4ever has quit IRC | 20:43 | |
| jeblair | oh, but i can grep | 20:43 |
| *** adalbas has quit IRC | 20:43 | |
| anteaya | pleia2: the reply works, yay and thanks for following up on -dev and -infra | 20:43 |
| *** ildikov has quit IRC | 20:43 | |
| * pleia2 updates email filter | 20:44 | |
| pleia2 | all looks good | 20:44 |
| anteaya | yay | 20:45 |
| fungi | jeblair: i use a delay between two and then filter for those which didn't un-alien in that span: http://paste.openstack.org/show/105419/ | 20:46 |
| fungi | jeblair: and then i iterate over the resulting *.aliens lists for a nova delete | 20:47 |
| *** sballe has joined #openstack-infra | 20:47 | |
| jeblair | fungi: ah cool. oh yeah, you said you were going to do that. i won't step on your toes, unless you'd rather me do it. | 20:47 |
| fungi | i have it running now, since you just reminded me | 20:48 |
| fungi | i got sidetracked submitting 5 pages of py34 transition changes to gerrit | 20:48 |
| fungi | https://review.openstack.org/#/q/status:open+topic:py34,n,z | 20:48 |
| jeblair | fungi: wow, you did the stackforge projects? | 20:49 |
| *** tonytan4ever has joined #openstack-infra | 20:49 | |
| fungi | i iterated over all of what was in gerrit. was easier that way | 20:49 |
| fungi | updated the cookiecutters too | 20:49 |
| jeblair | nice | 20:50 |
| fungi | i'be got a config change proposed to start installing python3.4 and pypy on trusty from the normal ubuntu packages, and a child change to add a non-voting job for it to git-review | 20:50 |
| *** dprince has quit IRC | 20:50 | |
| *** shayneburgess has joined #openstack-infra | 20:50 | |
| *** yfried has quit IRC | 20:50 | |
| *** marcoemorais1 has quit IRC | 20:50 | |
| fungi | and assuming that works, we can discuss whether we want to add non-voting jobs for everyone, or just switch them all from 33 to 34 and let them scream/fix whatever it breaks ;) | 20:51 |
| *** doug-fish has joined #openstack-infra | 20:51 | |
| anteaya | fungi: and you have a -1 already | 20:52 |
| anteaya | woo | 20:52 |
| fungi | at least half the python33 jobs are non-voting now anyway, so we can probably safely just replace those. if the voting 33 jobs aren't too numerous i can pre-test them manually | 20:52 |
| fungi | aha, taskflow uses a tox.ini template/generator | 20:53 |
| *** markmcclain1 has joined #openstack-infra | 20:53 | |
| anteaya | do you need py34 for havana? | 20:54 |
| anteaya | how much longer does havana have? | 20:54 |
| *** markmcclain has quit IRC | 20:54 | |
| *** marcoemorais1 has joined #openstack-infra | 20:54 | |
| clarkb | fungi: oh thank you for the py34 stuff | 20:54 |
| *** ociuhandu has joined #openstack-infra | 20:54 | |
| fungi | anteaya: probably not, but i generated all the backport changes automagically | 20:54 |
| anteaya | ah okay | 20:54 |
| fungi | anteaya: if we ditch havana sooner than we ditch the precise-py3k nodes, then those changes can just get abandoned | 20:55 |
| anteaya | kk | 20:55 |
| fungi | and ultimately they're not (most of them anyway) needed for running python 3.4 jobs since the majority of those changes just edit the default list of environments. the ones which _do_ matter were the projects with "extra" stuff going on in their tox.ini to support 3.3 | 20:56 |
| fungi | but since i needed to identify those, it was easier to just do them all in a batch run | 20:56 |
| * anteaya keeps an eye out for tox.ini files with extra stuff in them | 20:56 | |
| anteaya | batches can be good | 20:57 |
| clarkb | fungi: were projects doing much more than PYTHONHASHSEED=0 or the occasional dependency difference? | 20:57 |
| arosen | clarkb: ah found it! https://review.openstack.org/#/c/104375/6/congressclient/v1/policy.py I do: from openstackclient.common import utils as osc_utils | 20:57 |
| fungi | clarkb: it was mostly py3k-specific dependencies, yeah | 20:58 |
| arosen | that import isn't in the python-congressclient repo | 20:58 |
| arosen | i guess tox creates a source tarball and runs it's tests from that so that's what was missing. | 20:58 |
| arosen | thanks for your help though! | 20:58 |
| clarkb | yes by default that is how tox tests | 20:59 |
| clarkb | however your non sdist installed code would have the same issue in the virtualenv right? | 20:59 |
| *** homeless has quit IRC | 21:00 | |
| *** emagana has quit IRC | 21:00 | |
| anteaya | fungi: this would be an example of a tox.ini file with extra stuff in it: https://review.openstack.org/#/c/118866/1/tox.ini yes? | 21:00 |
| *** emagana has joined #openstack-infra | 21:00 | |
| *** homeless has joined #openstack-infra | 21:01 | |
| *** Sukhdev has joined #openstack-infra | 21:01 | |
| *** markmcclain1 has quit IRC | 21:01 | |
| openstackgerrit | A change was merged to openstack-infra/config: Edit the name of the third-party-requests ml https://review.openstack.org/118506 | 21:01 |
| *** markmcclain has joined #openstack-infra | 21:01 | |
| fungi | anteaya: yeah, keystone does some interesting things there because of nose | 21:02 |
| fungi | anteaya: oh, though that's actually a bad cherry-pick | 21:02 |
| fungi | the master version of that makes a little more sense | 21:02 |
| * fungi fixes | 21:02 | |
| anteaya | kk | 21:03 |
| bdpayne | jeblair Could you help me understand what I need to do with chanserv for the openstack-security op privs? | 21:03 |
| anteaya | fungi: this would be the master version: https://review.openstack.org/#/c/118778/1/tox.ini yes? | 21:04 |
| *** annegent_ has quit IRC | 21:04 | |
| fungi | anteaya: yes | 21:05 |
| anteaya | fungi: the master and the icehouse version look the same to my eyes, what makes this a bad cherry-pick? | 21:05 |
| anteaya | the icehouse version shouldn't have the nose stuff? | 21:06 |
| anteaya | I don't understand | 21:06 |
| fungi | anteaya: it duplicated logic from the py33 venv setup for the master tox.ini into the stable/icehouse (and havana too) tox.ini | 21:06 |
| *** tsg has quit IRC | 21:06 | |
| fungi | anteaya: note that the py33 setup i was copying in the master tox.ini is not present in the stable ones | 21:07 |
| *** elixor has joined #openstack-infra | 21:07 | |
| jeblair | bdpayne: send '/msg chanserv help' | 21:07 |
| jeblair | bdpayne: /msg chanserv op #openstack-security bdpayne | 21:07 |
| jeblair | bdpayne: should do it | 21:07 |
| * anteaya looks again at the py33 setup in the master and stable tox.ini files | 21:08 | |
| jeblair | bdpayne: specifically related to banning folks, you may be interested in 'access +b'. see 'help access' and 'help flags' | 21:09 |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally https://review.openstack.org/117892 | 21:09 |
| bdpayne | hey thanks, that worked | 21:09 |
| *** aysyd has quit IRC | 21:09 | |
| jeblair | bdpayne: np | 21:09 |
| *** emagana_ has joined #openstack-infra | 21:10 | |
| *** zz_johnthetubagu has quit IRC | 21:10 | |
| *** emagana has quit IRC | 21:10 | |
| fungi | anteaya: http://git.openstack.org/cgit/openstack/keystone/diff/tox.ini?id=master&id2=stable/icehouse | 21:11 |
| *** mfer has quit IRC | 21:11 | |
| *** zz_johnthetubagu has joined #openstack-infra | 21:11 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 21:11 | |
| *** miqui has quit IRC | 21:11 | |
| fungi | anteaya: the addition of the [testenv:py33] section wasn't added until after stable/icehouse so doesn't need a similar [testenv:py34] section in the backport either | 21:12 |
| openstackgerrit | A change was merged to openstack-infra/config: Change wording and typos in third_party.rst https://review.openstack.org/116609 | 21:12 |
| *** david-lyle has quit IRC | 21:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: Make py33 job experimental for cloudkitty-dashboard https://review.openstack.org/116637 | 21:12 |
| *** emagana has joined #openstack-infra | 21:13 | |
| *** elixor has quit IRC | 21:13 | |
| *** imcsk8 has quit IRC | 21:13 | |
| *** imcsk8 has joined #openstack-infra | 21:14 | |
| anteaya | fungi: I see the diff | 21:15 |
| *** emagana_ has quit IRC | 21:16 | |
| anteaya | fungi: I'm having difficulty linking backward from that page to see the datestamp for that addition | 21:16 |
| *** emagana_ has joined #openstack-infra | 21:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix deprecation warnings in exim module https://review.openstack.org/116749 | 21:17 |
| openstackgerrit | A change was merged to openstack-infra/config: Conditionally manage puppetdb connection https://review.openstack.org/117377 | 21:17 |
| *** melwitt has joined #openstack-infra | 21:17 | |
| *** tsg has joined #openstack-infra | 21:19 | |
| *** emagana has quit IRC | 21:19 | |
| greghaynes | jeblair: Hey, https://review.openstack.org/#/c/118754/ got -1'd for reasons I dont understand and I was told that it might be a UI bug | 21:20 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix depercation warnings in templates for unatttended upgrades https://review.openstack.org/116747 | 21:20 |
| clarkb | greghaynes: jeblair: the issue is you the cahnge failed the requirements check | 21:20 |
| clarkb | so -1. but the CI table doesn't show that failure because it reports back with a emssage | 21:21 |
| greghaynes | oh! I totally glanced over that becuase it wasnt green or red | 21:21 |
| greghaynes | haha | 21:21 |
| *** melwitt has quit IRC | 21:22 | |
| *** melwitt has joined #openstack-infra | 21:23 | |
| clarkb | the js regex may need to be relaxed to capture non binary results | 21:25 |
| jeblair | clarkb, fungi: http://logs.openstack.org/52/116752/2/gate/check-projects-yaml-upstream/d677179/console.html | 21:27 |
| fungi | jeblair: uhh... weird | 21:27 |
| clarkb | I was about to say mirror problems again but no thats local fil:// | 21:28 |
| *** dhellmann is now known as dhellmann_ | 21:28 | |
| clarkb | perhaps the image was built with a bad repo in the cache? | 21:28 |
| fungi | local disk i/o problem? are we seeing this widespread? | 21:28 |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: I missed an 's' https://review.openstack.org/118893 | 21:28 |
| fungi | digging in logstash now | 21:28 |
| anteaya | jeblair: I realized I missed an 's' in the docs ^^ | 21:29 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a missing 's' to third-party-requests https://review.openstack.org/118893 | 21:30 |
| fungi | clarkb: jeblair: bingo. all in rax-iad | 21:30 |
| jeblair | anteaya: i made a better commit subject for you :) | 21:30 |
| anteaya | you did, thank you | 21:30 |
| fungi | clarkb: jeblair: better still, it's been happening for a couple days, so we didn't get an image there today | 21:31 |
| clarkb | fungi: the image build failed in a similar way? | 21:31 |
| clarkb | fungi: sounds like fun :) | 21:31 |
| *** krtaylor has quit IRC | 21:32 | |
| *** david-lyle has joined #openstack-infra | 21:33 | |
| *** doug-fish has quit IRC | 21:33 | |
| fungi | clarkb: no, i mean it's been happening since before today's image would have built, so presumably we didn't get a new image today (which would likely have caused it to cease happening) | 21:33 |
| clarkb | oh I see. I thought maybe you were talking about vm disk failures in that region, but you mean that particular things failed | 21:34 |
| fungi | i suspect the image builds didn't happen either because of the alien infestation (which are in the process of being deleted now) or because they were interrupted by our nodepool restart around the same time | 21:34 |
| clarkb | rgr | 21:34 |
| fungi | clarkb: just looking at message:"fatal: serious inflate inconsistency" AND tags:console for the past 7 days | 21:35 |
| fungi | clarkb: though fun tidbit, it only hit check-projects-yaml-upstream once | 21:35 |
| *** annegent_ has joined #openstack-infra | 21:35 | |
| fungi | the others are for gate-nova-python27, gate-neutron-python27, gate-ceilometer-pep8 add gate-tempest-docs | 21:36 |
| fungi | so maybe it's not a bad image, but something else going sour in iad | 21:36 |
| fungi | for example http://logs.openstack.org/63/117463/1/gate/gate-nova-python27/03c6c8d/console.html | 21:37 |
| *** annegent_ has quit IRC | 21:37 | |
| clarkb | right because those other jobs will be using different repos | 21:37 |
| *** annegent_ has joined #openstack-infra | 21:38 | |
| clarkb | bad disk? bit flips in memory? the next intel fp bug? | 21:38 |
| fungi | but whatever the underlying cause, it's only in rax-iad and started some time yesterday | 21:38 |
| jroll | fungi: I'm just reading up, but what time approximately? 7 or 8 pm pacific, perhaps? | 21:41 |
| *** tonytan4ever has quit IRC | 21:42 | |
| *** mfink has joined #openstack-infra | 21:42 | |
| clarkb | jroll: http://logs.openstack.org/63/117463/1/gate/gate-nova-python27/03c6c8d/console.html was 1840 UTC today which is ~1140 pacific if I did my maths correctly | 21:42 |
| jroll | fungi: disk io issues? | 21:42 |
| jroll | gah | 21:43 |
| jroll | I'm curious when we started seeing it | 21:43 |
| *** emagana_ has quit IRC | 21:44 | |
| jroll | there was a largish deploy to iad last night | 21:44 |
| * jroll does math | 21:44 | |
| clarkb | oh when it started | 21:44 |
| * clarkb runs the query | 21:44 | |
| jroll | began at 0200 utc this morning? | 21:44 |
| fungi | jroll: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiZmF0YWw6IHNlcmlvdXMgaW5mbGF0ZSBpbmNvbnNpc3RlbmN5XCIgQU5EIHRhZ3M6Y29uc29sZSIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwOTc3OTc3MTk3MH0= | 21:44 |
| jroll | if my math is right | 21:44 |
| clarkb | oh fungi has it | 21:44 |
| jroll | 7pm pacific last night | 21:44 |
| fungi | jroll: 09:36 utc was the earliest sighting | 21:44 |
| jroll | oh, huh, ok | 21:45 |
| fungi | that would be 02:00 pdt | 21:45 |
| jroll | still bad, but at least not something we broke with a deploy :P | 21:45 |
| jroll | (maybe) | 21:45 |
| openstackgerrit | A change was merged to openstack-infra/config: Run requirements jobs for openstack-doc-tools https://review.openstack.org/116805 | 21:45 |
| openstackgerrit | A change was merged to openstack-infra/config: Consolidate tripleo/os-apply-config ACL files https://review.openstack.org/116950 | 21:46 |
| *** mriedem has quit IRC | 21:46 | |
| fungi | local time in iad would have been 05:00 (edt) | 21:47 |
| jroll | right | 21:47 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove /etc/puppet/modules on apply test https://review.openstack.org/117365 | 21:47 |
| jroll | I can't imagine a code change would cause corrupted memory, though, either | 21:47 |
| clarkb | jroll: did you perhaps deploy onto new machines? | 21:48 |
| clarkb | and maybe one of them is having hardware trouble? | 21:48 |
| jroll | clarkb: 99% sure that's a negative | 21:48 |
| jroll | but, old machines could be having issues | 21:48 |
| jroll | clarkb: any way we have instance ids for these VMs? | 21:49 |
| clarkb | its such an odd error, and it doesnt' affect a specific repo which makes me think it must be more of a general issue | 21:49 |
| clarkb | jroll: we can dig them up | 21:49 |
| jroll | indeed | 21:49 |
| *** kgiusti has quit IRC | 21:49 | |
| jroll | clarkb: I'd be interested to see if they were all on the same host or something | 21:49 |
| *** emagana has joined #openstack-infra | 21:49 | |
| openstackgerrit | A change was merged to openstack-infra/config: Disable notifications in storyboard https://review.openstack.org/116973 | 21:49 |
| jroll | clarkb: if you think it's worth the effor, anyway | 21:50 |
| jroll | effort* | 21:50 |
| jroll | I'm not 100% sure I can dig up old instances, but I can try | 21:50 |
| *** flaviof is now known as flaviof_zzz | 21:50 | |
| *** andreaf has quit IRC | 21:51 | |
| fungi | alien nodes total before deletion was 52, though 18 of those were in iad | 21:51 |
| fungi | which explains the over-quota errors nodepool was getting | 21:51 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix typo - run gate/check against oslo.serialization-py33 https://review.openstack.org/116997 | 21:52 |
| openstackgerrit | A change was merged to openstack-infra/config: Be able to generate math formulae with Sphinx mathpng base extension. https://review.openstack.org/117263 | 21:53 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix Berkshelf 3.x GeCode 3.x dependency https://review.openstack.org/117274 | 21:53 |
| clarkb | jroll: ya I can find a few now | 21:53 |
| jroll | clarkb: cool, let me know | 21:53 |
| *** paulrad has joined #openstack-infra | 21:54 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add #openstack-operators to the eavesdrop bot https://review.openstack.org/117312 | 21:57 |
| clarkb | jroll: d6b3da89-df68-4284-9ec5-bd0fa35a321e 622e322a-95c6-4f88-873c-e11382e621dd 80ae393d-35c4-43b3-9d82-2745c7c73090 0ed34e38-f8ce-421b-a1eb-ac7dc20862ee f9626b38-31d7-4735-9d7c-78db13904ca3 | 21:58 |
| *** datsun180b has quit IRC | 21:58 | |
| jroll | whee | 21:58 |
| * jroll looks | 21:58 | |
| *** paulrad has quit IRC | 21:59 | |
| *** baoli has quit IRC | 22:00 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add integration apply test https://review.openstack.org/117323 | 22:00 |
| jroll | clarkb: heh, nice one, all the same host | 22:03 |
| jroll | clarkb: I'll poke people internally, thanks for tracking that down :) | 22:03 |
| clarkb | jroll: thank you for checking on your side | 22:03 |
| jroll | np | 22:03 |
| openstackgerrit | A change was merged to openstack-infra/config: Turn off manpage building https://review.openstack.org/117337 | 22:03 |
| openstackgerrit | A change was merged to openstack-infra/config: Doc cleanup https://review.openstack.org/117319 | 22:04 |
| *** Clabbe has quit IRC | 22:04 | |
| *** pballand has quit IRC | 22:04 | |
| *** marcoemorais1 has quit IRC | 22:05 | |
| *** marcoemorais has joined #openstack-infra | 22:06 | |
| *** shardy is now known as shardy_z | 22:07 | |
| *** openstack has joined #openstack-infra | 22:11 | |
| clarkb | but first trying to figure out if this is a legit derp or if we just need more fds | 22:11 |
| *** eharney has quit IRC | 22:12 | |
| clarkb | it actually looks like it may be leaking PIPEs | 22:13 |
| clarkb | which are used for the crm114 communication | 22:14 |
| clarkb | so bumping the limit is not so good | 22:14 |
| *** emagana has quit IRC | 22:15 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/nodepool: Don't retry forever when a provider is stopped https://review.openstack.org/118710 | 22:16 |
| *** paulrad has joined #openstack-infra | 22:17 | |
| clarkb | fungi: we both +2'd :) | 22:17 |
| clarkb | I can approve | 22:17 |
| fungi | clarkb: feel free | 22:17 |
| *** sdake_ has joined #openstack-infra | 22:18 | |
| *** dkranz has quit IRC | 22:19 | |
| *** dkranz has joined #openstack-infra | 22:20 | |
| *** erlon has quit IRC | 22:20 | |
| *** markmcclain has quit IRC | 22:20 | |
| *** mfink has quit IRC | 22:20 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add readthedocs for heat-translator https://review.openstack.org/117533 | 22:20 |
| openstackgerrit | A change was merged to openstack-dev/pbr: use get_boolean_option for warnerrors https://review.openstack.org/117476 | 22:20 |
| fungi | we're a little over 4 hours to get bare-trusty nodes in the check pipeline now | 22:22 |
| jeblair | 2014-09-03 18:50:01.099 | 441 bashate error(s) found | 22:22 |
| jeblair | in config | 22:22 |
| *** packet has joined #openstack-infra | 22:22 | |
| clarkb | jeblair: I am rereading the filter code in https://git.openstack.org/cgit/openstack-infra/config/tree/modules/log_processor/files/log-gearman-worker.py and no obvious subprocess pipe leaks stand out. We may want to put lines 169-170 in something more like a finally block though | 22:24 |
| clarkb | jeblair: does anything stand out to you? | 22:24 |
| clarkb | I am going to dig into logs further back in time to see if there is an error thrown before we start leaking | 22:25 |
| arosen | I've been running into this probably lately i'm not sure if i'm doing something wrong... When I do git review or git push manually i get back: http://paste.openstack.org/show/105459/ | 22:26 |
| arosen | it only happens when i have multiple patches rebased on each other | 22:26 |
| *** dimtruck is now known as zz_dimtruck | 22:26 | |
| jeblair | clarkb: yeah, a try/finally sounds good. an error on f.close() could prevent other filters folm closing (i'm not sure if there is more than one filter). logq.put could fail if the queue is full; i don't know if that happens either. | 22:27 |
| fungi | arosen: usually that means that you updated the commit without changing its parentage or modifying its content or commit message | 22:27 |
| *** kurtmartin has quit IRC | 22:27 | |
| *** andreykurilin_ has quit IRC | 22:28 | |
| arosen | fungi: right but if i do git commit --amend and change the top commit message I should be able to repush even if the patches below haven't changed right? | 22:28 |
| *** annegent_ has quit IRC | 22:28 | |
| arosen | Iv'e never really had this problem in the past :/ | 22:28 |
| fungi | arosen: right | 22:28 |
| fungi | arosen: how are you building the patch series? git review -d? | 22:29 |
| clarkb | jeblair: I think logq.put() should block if the queue is full | 22:29 |
| *** jgrimm is now known as zz_jgrimm | 22:29 | |
| fungi | (to retrieve the existing series from gerrit into your locak working clone i mean) | 22:29 |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/publications: Update slides for Fossetcon https://review.openstack.org/118915 | 22:30 |
| *** kurtmartin has joined #openstack-infra | 22:30 | |
| pleia2 | bah, how did I manage to push that to template | 22:31 |
| pleia2 | I clearly don't do this often enough :) | 22:31 |
| fungi | pleia2: if you branched from template, you need to update the defaultbranch in .gitreview on the new branch to help you not do that | 22:31 |
| arosen | fungi: I'm not sure the work flow that breaks it. git add file; git commit; git review; git add file; git commit; git review; | 22:31 |
| pleia2 | ah, I never updated .gitreview in this branch | 22:32 |
| pleia2 | fungi: yeah :) | 22:32 |
| clarkb | jeblair: we should probably also close stdout and stderr in https://git.openstack.org/cgit/openstack-infra/config/tree/modules/log_processor/files/log-gearman-worker.py#n95 I will write a patch for these things | 22:32 |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/publications: Update slides for Fossetcon https://review.openstack.org/118918 | 22:33 |
| fungi | arosen: if you have an example you hit, being able to see the contents of git log --online would help | 22:33 |
| fungi | er, --oneline | 22:33 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove Octavia python 3.3 job, as eventlet means that will always fail. https://review.openstack.org/117868 | 22:33 |
| openstackgerrit | A change was merged to openstack-infra/config: Update documented local gerrit git replica path https://review.openstack.org/117977 | 22:33 |
| *** marcoemorais has quit IRC | 22:33 | |
| arosen | fungi: http://paste.openstack.org/show/105466/ | 22:33 |
| *** marcoemorais has joined #openstack-infra | 22:34 | |
| arosen | fungi: see i changed the comment messsage to !22 | 22:34 |
| arosen | e2490d5 --> isn't showing up in git log --online which is weird. | 22:34 |
| fungi | arosen: yeah, the issue is back at the "Fixup gitignore" change | 22:35 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add the tempest-lib project to openstack https://review.openstack.org/117640 | 22:35 |
| fungi | arosen: apparently it is an update of but not significantly different than the version in gerrit of https://review.openstack.org/118657 | 22:36 |
| arosen | fungi: but the patches on top of that changed though | 22:36 |
| arosen | so i should be able to push the new stuff. | 22:36 |
| fungi | arosen: yeah, but that commit changed too, and gerrit doesn't like that it's not got the same gitsha as https://review.openstack.org/118657 | 22:37 |
| *** sdake_ has quit IRC | 22:37 | |
| arosen | fungi: Ah I see: so I did a git rebase -i HEAD~5; then changed the order of commits so the git sha changes | 22:38 |
| fungi | arosen: i would do 'git review -d 118657;git cherry-pick 4602d2d;git cherry-pick 271f8db;git cherry-pick bc5383e;git review' | 22:38 |
| *** imeerovi has quit IRC | 22:38 | |
| *** krtaylor has joined #openstack-infra | 22:38 | |
| fungi | arosen: except when you were reordering, i think you went one commit too far back and re-picked e2490d5 which when it was amended got a new sha of f6b0b73 but was otherwise unchanged (same parent gitsha, same commit message, same diff) | 22:40 |
| *** bknudson has quit IRC | 22:41 | |
| fungi | so gerrit sees you trying to push f6b0b73 as a new patchset to https://review.openstack.org/118657 with zero delta | 22:41 |
| *** emagana has joined #openstack-infra | 22:42 | |
| arosen | fungi: ah i see. Thanks for explaining that to me. That's exactly what happened. | 22:43 |
| *** masayukig has quit IRC | 22:44 | |
| *** otherwiseguy has quit IRC | 22:45 | |
| *** craigbr has quit IRC | 22:50 | |
| *** imeerovi has joined #openstack-infra | 22:50 | |
| *** masayukig has joined #openstack-infra | 22:50 | |
| *** arnaud has quit IRC | 22:51 | |
| *** pcm_ has quit IRC | 22:51 | |
| *** dims has quit IRC | 22:52 | |
| fungi | no meetings at the moment. approving 116527 | 22:52 |
| *** dims has joined #openstack-infra | 22:52 | |
| fungi | well, i'll wait until 23:05 in case someone starts one at 23:00 (given the current gating sluggishness) | 22:52 |
| *** craigbr has joined #openstack-infra | 22:53 | |
| *** marun_ has quit IRC | 22:54 | |
| *** dims has quit IRC | 22:57 | |
| *** gyee has quit IRC | 22:59 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/config: Adds Ironic parallel Tempest job https://review.openstack.org/118700 | 22:59 |
| *** paulrad has quit IRC | 22:59 | |
| *** paulrad has joined #openstack-infra | 22:59 | |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally https://review.openstack.org/117892 | 23:00 |
| jhesketh | Morning | 23:00 |
| *** Sukhdev has quit IRC | 23:00 | |
| fungi | huh... what was the deal with bashate running against openstack-infra/config? | 23:02 |
| openstackgerrit | Aaron Rosen proposed a change to openstack-dev/cookiecutter: Add additional gitignores .swo/.swn https://review.openstack.org/118923 | 23:03 |
| jeblair | fungi: that doesn't imapct eavesdrop | 23:03 |
| fungi | it looks like it's failing to exclude the .tox directory, and is descending into its own sample failure tests | 23:03 |
| fungi | Running bashate on ./.tox/bashate/lib/python2.7/site-packages/bashate/tests/samples/E001_bad.sh | 23:03 |
| fungi | et cetera | 23:03 |
| *** paulrad has quit IRC | 23:04 | |
| anteaya | morning jhesketh | 23:05 |
| sdague | fungi: yeh, there is a thread on trying to figure out the ignore parts of this correctly | 23:06 |
| sdague | on the -dev ML | 23:06 |
| fungi | jeblair: oh, ha yeah... i was thrown off by clarkb's comment that he wasn't approving the change because there were meetings going on. forgetting that the change wasn't touching meetbot configuration | 23:06 |
| *** jergerber has quit IRC | 23:06 | |
| *** masayukig has quit IRC | 23:07 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Handle log processing subprocess cleanup better https://review.openstack.org/118924 | 23:07 |
| clarkb | fungi: oh it didnt? | 23:07 |
| clarkb | I thought Iexpanded it and it did | 23:07 |
| clarkb | jeblair: ^ how does that look to you? I haven't tested it but I figured early feedback would be good | 23:08 |
| jeblair | fungi: i was too, but then i went to try to figure out why i missed it in my search of eavesdrop changes earlier :) | 23:08 |
| fungi | jeblair: i had no excuse... it was my patch after all | 23:09 |
| clarkb | jeblair: secifically do we need to read() the pipes before closing them? | 23:09 |
| clarkb | jeblair: close(2) shouldn't error if there is data in the pipe right? | 23:09 |
| *** wenlock has quit IRC | 23:09 | |
| *** masayukig has joined #openstack-infra | 23:09 | |
| *** SumitNaiksatam has quit IRC | 23:10 | |
| *** signed8b_ has quit IRC | 23:11 | |
| clarkb | oh that was a statusbot list to eavesdrop | 23:11 |
| clarkb | I saw eavesdrop and immediately thought: meetbot | 23:11 |
| jeblair | clarkb: erm, i don't really like the "always abruptly terminate" approach | 23:12 |
| clarkb | jeblair: without it we can hang on wait() | 23:13 |
| jeblair | clarkb: is that happening? | 23:13 |
| clarkb | jeblair: I haven't observed it | 23:13 |
| openstackgerrit | A change was merged to openstack-infra/config: Add #openstack-stable IRC channel https://review.openstack.org/116527 | 23:13 |
| clarkb | jeblair: but with my change to not read the pipe until it is empty it may hang? | 23:14 |
| jeblair | clarkb: i dunno. that's just a lot of change without actually having an error that we're trying to fix | 23:14 |
| *** david-lyle has quit IRC | 23:14 | |
| clarkb | jeblair: it was to make the close() calls simpler | 23:14 |
| clarkb | as crm114 may not die if I just close my side of the pipe | 23:15 |
| clarkb | I may have to read all the data for it to EOF and shutdown nicely? | 23:15 |
| jeblair | clarkb: i'm just saying i do not understand the problem that's trying to be fixed, and can't determine whether your change addresses it | 23:15 |
| clarkb | jeblair: the problem is the Popen is leaky | 23:15 |
| jeblair | clarkb: the symptom is that pipes have leaked | 23:16 |
| clarkb | yes | 23:16 |
| jeblair | clarkb: i helped you speculate as to possible causes, but unless i missed something, i don't think you identified one | 23:16 |
| *** signed8bit has joined #openstack-infra | 23:17 | |
| clarkb | jeblair: I did not pdb/strace it to find where the leak is happening | 23:18 |
| clarkb | but there are several code paths that may throw short circuiting clean up | 23:18 |
| jeblair | clarkb: i agree; but you're doing more than that here, you're changing the way it's closed | 23:18 |
| clarkb | jeblair: yes, to simplify the closure | 23:19 |
| *** openstackstatus has quit IRC | 23:19 | |
| jeblair | clarkb: i don't know why, nor do i know what the implications are for that. for both this program, and the program you are terminating | 23:19 |
| jeblair | clarkb: particularly, i can't answer the question "why should we never let it terminate normally?" | 23:19 |
| clarkb | jeblair: if I remove the kill() then I think I have to do crazy non blocking reads off of the pipes until they are empty | 23:20 |
| *** otherwiseguy has joined #openstack-infra | 23:20 | |
| *** openstackstatus has joined #openstack-infra | 23:20 | |
| *** ChanServ sets mode: +v openstackstatus | 23:20 | |
| *** dane_leblanc has quit IRC | 23:21 | |
| jeblair | clarkb: does that increase the risk of database corruption? | 23:21 |
| clarkb | jeblair: if the reads are not non blocking the parent process may deadlock the subprocess and if Idon't read all of the data to completion the subprocess may not die blocking the parent on wait(). That said the subprocess may not take kindly to being killed | 23:21 |
| *** Sincler has quit IRC | 23:22 | |
| *** ianw has quit IRC | 23:25 | |
| *** ianw has joined #openstack-infra | 23:25 | |
| NobodyCam | Hi -infra : off the wall question: is there a way to recheck a reuuning job with out pushing up a new patch? | 23:25 |
| *** dmsimard is now known as dmsimard_away | 23:26 | |
| NobodyCam | s/reunning/running/ | 23:26 |
| *** asettle has quit IRC | 23:26 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fixing deprecation warnings in logrotate module https://review.openstack.org/116752 | 23:26 |
| anteaya | NobodyCam: not that I know of | 23:28 |
| anteaya | NobodyCam: using recheck will just wait until the current tests finish, as far as I know | 23:28 |
| anteaya | NobodyCam: why do you ask? | 23:28 |
| clarkb | jeblair: I can stab at using select and removing the kill | 23:28 |
| NobodyCam | I have a job that failed py27 due to bug and it like 4 hours away form finishing | 23:29 |
| *** baoli has joined #openstack-infra | 23:29 | |
| *** coolsvap has quit IRC | 23:29 | |
| NobodyCam | anteaya: https://review.openstack.org/#/c/113865 | 23:30 |
| *** chuckC has quit IRC | 23:30 | |
| anteaya | NobodyCam: you aren't approved, change the commit message | 23:30 |
| *** emagana has quit IRC | 23:31 | |
| NobodyCam | oh good idea anteaya TY | 23:31 |
| *** emagana has joined #openstack-infra | 23:32 | |
| clarkb | jeblair: in the mean time what do you think about splitting the cahnge into and merging the try finally in the outer block more quickly? | 23:32 |
| anteaya | NobodyCam: it used to work, let's see if it still does | 23:32 |
| clarkb | jeblair: then we can sort out how the inner close() works in a second patch | 23:32 |
| fungi | NobodyCam: changing the commit message will indeed restart testing, but it will also dump any existing reviewer votes since it effectively creates a new patchset for you | 23:33 |
| *** coolsvap has joined #openstack-infra | 23:33 | |
| NobodyCam | Awesome TY :) | 23:33 |
| *** hogepodge has quit IRC | 23:34 | |
| *** atiwari has quit IRC | 23:34 | |
| *** _buzz_ has quit IRC | 23:35 | |
| *** emagana_ has joined #openstack-infra | 23:36 | |
| *** hemna is now known as hemna__ | 23:36 | |
| *** emagana has quit IRC | 23:36 | |
| *** alexpilotti has quit IRC | 23:37 | |
| *** dangers is now known as dangers_away | 23:37 | |
| *** _buzz has joined #openstack-infra | 23:38 | |
| jroll | clarkb: I learned about new things today: "the hypervisor and xapi have lost connection in such a way where the HV cannot allocate new instance chunks or make certain changes" | 23:40 |
| *** koolhead17 has joined #openstack-infra | 23:40 | |
| clarkb | jroll: all I hear is "Something xen xen something something xen unhappy xen" :) | 23:41 |
| jroll | clarkb: right, exactly :P | 23:41 |
| jroll | but yeah, memory issue | 23:41 |
| jroll | s | 23:41 |
| jroll | thanks again for noticing that :) | 23:41 |
| *** _buzz has quit IRC | 23:42 | |
| *** otherwiseguy has quit IRC | 23:43 | |
| *** otherwiseguy has joined #openstack-infra | 23:43 | |
| *** _buzz has joined #openstack-infra | 23:44 | |
| clarkb | jeblair: as an alternative it may be time to look into using crm114 in daemon/service mode | 23:44 |
| clarkb | jeblair: then we can launch it independently | 23:44 |
| clarkb | and not try to do process management in these small scripts | 23:45 |
| *** mwagner_lap has joined #openstack-infra | 23:45 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Reduce graphite's stats retention policy https://review.openstack.org/115116 | 23:48 |
| *** _buzz has quit IRC | 23:49 | |
| *** _buzz has joined #openstack-infra | 23:49 | |
| *** _buzz has quit IRC | 23:50 | |
| *** gokrokve has quit IRC | 23:50 | |
| fungi | bug 1365139 looks like fun, but i don't have the remaining energy tonight to wontfix it and explain why it's "not going to happen[tm]" | 23:50 |
| uvirtbot | Launchpad bug 1365139 in openstack-ci "partial recheck" [Undecided,New] https://launchpad.net/bugs/1365139 | 23:50 |
| *** reed has quit IRC | 23:53 | |
| clarkb | jeblair: a quick test indicates closing stdin is enough to kill crm | 23:54 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Removed sort field from search. https://review.openstack.org/118927 | 23:54 |
| *** kurtmartin has quit IRC | 23:55 | |
| *** baoli has quit IRC | 23:56 | |
| *** baoli has joined #openstack-infra | 23:56 | |
| *** _buzz has joined #openstack-infra | 23:57 | |
| *** mbacchi has quit IRC | 23:58 | |
| *** chuckC has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!