| anteaya | wow | 00:00 |
|---|---|---|
| clarkb | 433G nodepool_dib | 00:00 |
| anteaya | my goodness | 00:00 |
| tonyb | that is a large number but I don't know what it's expected to be .... | 00:00 |
| anteaya | did something change with logging? | 00:01 |
| clarkb | it looks like we aren't deleting old images | 00:02 |
| clarkb | I am still trying to map files that are present to he database | 00:02 |
| tonyb | clarkb: thanks. | 00:04 |
| *** salv-orl_ has quit IRC | 00:05 | |
| *** amitgandhinz has quit IRC | 00:07 | |
| *** zeih has joined #openstack-infra | 00:07 | |
| clarkb | /dev/mapper/main-opt 504G 390G 115G 78% /opt after cleaning out 4 leaked images | 00:08 |
| lifeless | Nakato: sounds like StevenK is jumping on it after lunch - do you need some more curated bugs found though? I can do that if you like.. | 00:08 |
| clarkb | well 4 images x the number of formats we carry for them | 00:08 |
| *** sflanigan has joined #openstack-infra | 00:09 | |
| openstackgerrit | Sachi King proposed openstack-infra/zuul: Fix test for new WebOb https://review.openstack.org/296180 | 00:09 |
| clarkb | and there is one more image to clean out that fialed to build due to the lack of disk space | 00:09 |
| clarkb | there it goes happy now | 00:10 |
| anteaya | yay! | 00:10 |
| *** gildub has joined #openstack-infra | 00:11 | |
| clarkb | mordred: so this actually brings up the proper benefit to doing the import image and convert, we can keep a single format for all those images (except rax woo) | 00:11 |
| *** chlong has quit IRC | 00:11 | |
| clarkb | also our raw images are coming in closer to 10GB now, thats pretty awesome and probably a result of fstrim? | 00:12 |
| *** zeih has quit IRC | 00:12 | |
| clarkb | all in all 5 images cleared out for a total of ~140GB freed | 00:13 |
| clarkb | images are expensive | 00:13 |
| lifeless | it is terrfiying that 10G is an improvement | 00:13 |
| clarkb | lifeless: yes was 17GB | 00:13 |
| *** Sukhdev has quit IRC | 00:13 | |
| clarkb | qcow comes in just under 6GB typically | 00:13 |
| clarkb | there is 6.8GB of git cache on each of those | 00:14 |
| *** fawadkhaliq has quit IRC | 00:15 | |
| clarkb | if we can keep just the qcow2 format then we drop our overhead by about 34GB per image | 00:16 |
| clarkb | oh wait thats with old raw size. 28GB pre image | 00:16 |
| jhesketh | clarkb: maybe we should grow that volume anyway | 00:17 |
| clarkb | and since rax can't convert we still need the vhd which means we can only drop about 10GB per image | 00:17 |
| clarkb | jhesketh: ya we probably should, I am just trying to work out the options to us | 00:17 |
| *** fawadkhaliq has joined #openstack-infra | 00:17 | |
| jhesketh | yep | 00:17 |
| clarkb | we currently keep 12 images at once if we don't leak | 00:18 |
| clarkb | so thats roughly 120GB saved | 00:18 |
| anteaya | that is 6 types x 2 apiece? | 00:18 |
| clarkb | yes | 00:18 |
| clarkb | ubuntu-trusty, devstack-trusty, debian-jessie, fedora-23, devstack-centos7, centos-7 | 00:19 |
| *** achanda has quit IRC | 00:19 | |
| clarkb | eventually devstack-trusty and devstack-centos7 go away | 00:19 |
| clarkb | but then we add in ubuntu-xenial | 00:19 |
| *** tqtran has quit IRC | 00:19 | |
| clarkb | and presumably fedora-24 | 00:19 |
| *** Swami has quit IRC | 00:19 | |
| clarkb | bigger volume would be good | 00:20 |
| *** gyee has quit IRC | 00:20 | |
| *** jamesmcarthur has joined #openstack-infra | 00:21 | |
| *** mtanino has quit IRC | 00:24 | |
| *** ybathia has quit IRC | 00:25 | |
| *** sambetts|afk has quit IRC | 00:26 | |
| *** jcoufal has joined #openstack-infra | 00:27 | |
| *** jamesmcarthur has quit IRC | 00:27 | |
| *** fawadkhaliq has quit IRC | 00:28 | |
| *** thorst has joined #openstack-infra | 00:29 | |
| *** ddieterly has joined #openstack-infra | 00:30 | |
| *** thorst has quit IRC | 00:30 | |
| *** fawadkhaliq has joined #openstack-infra | 00:30 | |
| *** thorst has joined #openstack-infra | 00:31 | |
| *** zz_dimtruck is now known as dimtruck | 00:32 | |
| *** sambetts has joined #openstack-infra | 00:33 | |
| *** fawadkhaliq has quit IRC | 00:34 | |
| *** fawadkhaliq has joined #openstack-infra | 00:35 | |
| clarkb | I went ahead and started 3 more builder processes to add 12 upload threads | 00:35 |
| clarkb | hopefully this chews through the 31 upload backload | 00:35 |
| clarkb | we will need to make sure these processes are dead before the next round of builds because we can't have multiple builds running at the same time | 00:35 |
| *** thorst has quit IRC | 00:36 | |
| *** ddieterly is now known as ddieterly[away] | 00:38 | |
| Nakato | lifeless: Ah okay. Well, if you have them I'll take them, but not hurting for work currently. | 00:38 |
| *** briancline has quit IRC | 00:38 | |
| *** samueldm1 has quit IRC | 00:39 | |
| *** sdake has quit IRC | 00:41 | |
| *** fawadkhaliq has quit IRC | 00:41 | |
| *** ddieterly[away] is now known as ddieterly | 00:41 | |
| *** McCorkis has joined #openstack-infra | 00:44 | |
| jhesketh | clarkb: cool, thanks for that :-) | 00:44 |
| clarkb | jhesketh: they are running in a screen owned by my user, I will do my best to clean it up later this evening | 00:45 |
| *** briancline has joined #openstack-infra | 00:45 | |
| clarkb | its looking like vexxhost and ovh uploads may or may not happen so we probably can't count on an up to date ubuntu-trusty everywhere even after we get them ueued for upload :( | 00:47 |
| *** sputnik13 has quit IRC | 00:48 | |
| *** roxanaghe has joined #openstack-infra | 00:48 | |
| jhesketh | clarkb: understood | 00:49 |
| *** Sukhdev has joined #openstack-infra | 00:49 | |
| *** Sukhdev has quit IRC | 00:49 | |
| clarkb | down to 24 | 00:50 |
| *** roxanaghe has quit IRC | 00:53 | |
| *** annegentle has quit IRC | 00:55 | |
| fungi | clarkb: _if_ the load is lowish, we could delete remaining images from the handful of locations where uploads aren't succeeding | 00:55 |
| *** annegentle has joined #openstack-infra | 00:55 | |
| fungi | and let nodepool retry to add them there | 00:56 |
| clarkb | fungi: we were flat out today for a while | 00:56 |
| fungi | dunno if it still does that with non-snapshot builds though | 00:56 |
| clarkb | its pretty quiet now though | 00:56 |
| fungi | yep | 00:56 |
| *** akshai_ has quit IRC | 00:56 | |
| fungi | we were definitely pegging our quotas most of the american day though | 00:56 |
| *** piet has quit IRC | 00:56 | |
| *** pvaneck has quit IRC | 00:57 | |
| *** woodster_ has quit IRC | 00:57 | |
| clarkb | I'll see what things look like once the current batch is done | 00:57 |
| *** mylu has quit IRC | 00:58 | |
| fungi | bunch of changes stuck in the release pipeline waiting for wheel-related jobs | 00:58 |
| tonyb | clarkb: is there anything I can watch that will let me know when the uploads are done? | 00:58 |
| clarkb | tonyb: this isn't published anywhere yet | 00:58 |
| tonyb | clarkb: okay that knoda what I thought | 00:58 |
| clarkb | in theory it should just work everyday and no one would care | 00:58 |
| clarkb | but reality is when we upload 6 images in 9 regions a good chunk of them fail because all the things | 00:59 |
| fungi | huh, the duration of our wheel build jobs seems to be a bottleneck since they're serialized (one worker) and we queue one for every release | 00:59 |
| tonyb | clarkb: Sure and normally that's my attitude, it's just today I have an issues that *shoudl* be fixed by the new image. | 00:59 |
| clarkb | fungi: nice | 00:59 |
| fungi | we might as well not trigger those and just run them in a continuous loop | 00:59 |
| clarkb | tonyb: right if things were reliable I would say wait for tomorrow :) | 01:00 |
| clarkb | fungi: or just deal with the lack of wheels | 01:00 |
| clarkb | fungi: its not like any of our wheels are problematic intsalls | 01:00 |
| clarkb | fungi: and with constraints we don't use them right anyway anywys | 01:00 |
| tonyb | clarkb: with a 4 day w/e coming up I was hoping to recheck today ;P | 01:00 |
| fungi | well, that and we already build and upload wheels of them to pypi | 01:00 |
| mriedem | can't say i've seen this failure before http://logs.openstack.org/96/296596/4/check/gate-grenade-dsvm-multinode/0416843/console.html#_2016-03-23_23_05_41_624 | 01:00 |
| clarkb | tonyb: yup I am watching them and will do my best | 01:00 |
| mriedem | hmm http://logs.openstack.org/96/296596/4/check/gate-grenade-dsvm-multinode/0416843/logs/error.txt.gz | 01:01 |
| *** zeih has joined #openstack-infra | 01:01 | |
| clarkb | mriedem: the error is at http://logs.openstack.org/96/296596/4/check/gate-grenade-dsvm-multinode/0416843/console.html#_2016-03-23_23_05_16_524 | 01:01 |
| tonyb | clarkb: Thanks. not nagging | 01:01 |
| clarkb | mriedem: the exit 1 is just a holdover fomr us exiting with that earlier return status | 01:01 |
| clarkb | fungi: the other tricky thing is we don't multinode in rax right now so if we remove the image from ovh and vexxhost our multinode capacity drops a bit, but thats probably ok | 01:02 |
| mriedem | ya http://logs.openstack.org/96/296596/4/check/gate-grenade-dsvm-multinode/0416843/logs/grenade.sh.txt.gz#_2016-03-23_23_04_59_073 | 01:02 |
| clarkb | because once new image is in rax we can add rax multinode back in | 01:02 |
| *** bpokorny_ has joined #openstack-infra | 01:04 | |
| *** amitgandhinz has joined #openstack-infra | 01:05 | |
| fungi | if another infra-puppet-core is okay with 296708 i'll keep an eye on it after it merges to make sure it doesn't disrupt openstackid.org | 01:05 |
| *** zeih has quit IRC | 01:06 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Another ssh grenade failure fingerprint https://review.openstack.org/296854 | 01:07 |
| *** bpokorny has quit IRC | 01:07 | |
| *** fawadkhaliq has joined #openstack-infra | 01:08 | |
| *** zhurong has joined #openstack-infra | 01:08 | |
| *** bpokorny_ has quit IRC | 01:09 | |
| *** fawadkhaliq has quit IRC | 01:11 | |
| *** yamahata has quit IRC | 01:11 | |
| *** angdraug has quit IRC | 01:13 | |
| StevenK | lifeless: After *shopping*, not lunch. :-) | 01:13 |
| StevenK | Anyway, I have a testcase written, diving into the meat now | 01:13 |
| *** gongysh_ has joined #openstack-infra | 01:14 | |
| lifeless | StevenK: \o/ | 01:14 |
| *** Jeffrey4l has joined #openstack-infra | 01:15 | |
| *** amitgandhinz has quit IRC | 01:16 | |
| StevenK | With some fiddling because tox. I want nice things. | 01:16 |
| *** tphummel has quit IRC | 01:18 | |
| *** simondodsley has quit IRC | 01:19 | |
| *** achanda has joined #openstack-infra | 01:19 | |
| *** piet has joined #openstack-infra | 01:22 | |
| *** achanda has quit IRC | 01:23 | |
| *** Daisy has joined #openstack-infra | 01:30 | |
| *** Daisy has quit IRC | 01:31 | |
| *** EricGonczer_ has joined #openstack-infra | 01:31 | |
| *** Daisy has joined #openstack-infra | 01:32 | |
| *** achanda has joined #openstack-infra | 01:34 | |
| *** mtanino has joined #openstack-infra | 01:34 | |
| *** Daisy has quit IRC | 01:36 | |
| *** chlong has joined #openstack-infra | 01:36 | |
| *** mylu has joined #openstack-infra | 01:36 | |
| *** akshai has joined #openstack-infra | 01:41 | |
| *** baoli has joined #openstack-infra | 01:42 | |
| *** mylu has quit IRC | 01:42 | |
| clarkb | tonyb: we are down to 20ish image buts a whole chunk of rax uploads look close to done so hopefully that number drops fast | 01:43 |
| clarkb | tonyb: this is still the backlog queue though once it hits 0 I will start hte current ubuntu-trusty uploads | 01:43 |
| clarkb | something something interop | 01:43 |
| *** apoorvad has quit IRC | 01:44 | |
| *** thorst has joined #openstack-infra | 01:46 | |
| *** thorst has quit IRC | 01:47 | |
| *** Qiming has joined #openstack-infra | 01:48 | |
| *** EricGonczer_ has quit IRC | 01:49 | |
| tonyb | clarkb: Thanks for the update | 01:50 |
| *** aeng has quit IRC | 01:50 | |
| *** Daisy has joined #openstack-infra | 01:51 | |
| *** baoli has quit IRC | 01:52 | |
| *** yamamoto_ has joined #openstack-infra | 01:52 | |
| *** mylu has joined #openstack-infra | 01:55 | |
| *** fawadkhaliq has joined #openstack-infra | 01:56 | |
| openstackgerrit | Steve Kowalik proposed openstack-dev/pbr: Do not convert git tags when searching history https://review.openstack.org/296863 | 01:56 |
| StevenK | lifeless: ^ | 01:56 |
| *** akshai_ has joined #openstack-infra | 01:57 | |
| *** sdake has joined #openstack-infra | 01:57 | |
| *** Jeffrey4l has quit IRC | 01:58 | |
| lifeless | StevenK: nice | 01:58 |
| *** yamahata has joined #openstack-infra | 01:58 | |
| *** baoli has joined #openstack-infra | 01:58 | |
| lifeless | Nakato: ^ | 01:58 |
| *** achanda has quit IRC | 01:59 | |
| *** mtreinish has quit IRC | 01:59 | |
| *** Daisy has quit IRC | 02:00 | |
| *** akshai has quit IRC | 02:00 | |
| *** Daisy has joined #openstack-infra | 02:00 | |
| clarkb | apparently we poll every 2 seconds to check hte status of the imae import task | 02:01 |
| *** mtreinish has joined #openstack-infra | 02:01 | |
| *** Daisy has quit IRC | 02:01 | |
| StevenK | lifeless: Once I worked it out, I was amused at the steps of get the tag, coerce it to a version, get the maximum, convert it to a string so we can immediately coerce it back into a version. | 02:01 |
| *** Daisy_ has joined #openstack-infra | 02:01 | |
| *** Daisy_ has quit IRC | 02:02 | |
| *** Daisy has joined #openstack-infra | 02:03 | |
| *** akshai_ has quit IRC | 02:04 | |
| *** achanda has joined #openstack-infra | 02:05 | |
| *** verdurin has quit IRC | 02:05 | |
| anteaya | fungi: I opened 296708 and stared at it but I don't have any opinion | 02:06 |
| *** akshai has joined #openstack-infra | 02:07 | |
| *** kevinsho_ has joined #openstack-infra | 02:08 | |
| *** watersoul has quit IRC | 02:10 | |
| *** watersoul has joined #openstack-infra | 02:10 | |
| *** ddieterly is now known as ddieterly[away] | 02:10 | |
| *** mriedem has quit IRC | 02:11 | |
| *** mriedem has joined #openstack-infra | 02:11 | |
| *** baoli has quit IRC | 02:13 | |
| *** hichihara has joined #openstack-infra | 02:13 | |
| *** amitgandhinz has joined #openstack-infra | 02:13 | |
| *** Daisy_ has joined #openstack-infra | 02:15 | |
| *** baoli has joined #openstack-infra | 02:18 | |
| *** Daisy has quit IRC | 02:18 | |
| cj | mordred: moo | 02:18 |
| *** tqtran has joined #openstack-infra | 02:18 | |
| cj | mordred: I don't think that gerritlib is being as smrt as it could regarding paramiko exceptions | 02:19 |
| cj | in gerritlib.gerrit._connect(), the only exception it is currently catching is paramiko.SSHException, and the only exception handling it does, if there is a retry_delay specified is to retry. over and over and over | 02:20 |
| *** baoli has quit IRC | 02:20 | |
| cj | so even if the server has always refused to accept the proffered ssh key, it will continue to try it for ever | 02:20 |
| clarkb | cj: thats when you add the public key to the server and next retry will work :) | 02:21 |
| cj | I'm going to patch it up to throw all authentication exceptions up the stack. There doesn't seem to be a way to cycle through multiple credentials at this time | 02:21 |
| cj | clarkb: that's a nice theory. but what if the gerritbot operator has just specified the incorrect key? | 02:22 |
| cj | don't you think they'd like to know about that? | 02:22 |
| cj | rather than failing silently forever? | 02:22 |
| cj | (hint: yes, I would thanks) | 02:22 |
| clarkb | sure I am not entirely serious | 02:22 |
| clarkb | proper error logging and handling is always a good ting | 02:22 |
| cj | I know. neither am I :-) | 02:22 |
| *** tqtran has quit IRC | 02:23 | |
| *** aspiers has quit IRC | 02:24 | |
| *** amitgandhinz has quit IRC | 02:26 | |
| *** fawadkhaliq has quit IRC | 02:27 | |
| *** baoli has joined #openstack-infra | 02:29 | |
| *** jerryz has quit IRC | 02:30 | |
| clarkb | tonyb: bluebox and osic have an up to date ubuntu-trusty now | 02:30 |
| tonyb | clarkb: thanks. | 02:30 |
| *** MarkAtwood has quit IRC | 02:30 | |
| clarkb | rax apperas to have failed somewhere | 02:30 |
| clarkb | like across the board | 02:30 |
| tonyb | clarkb: there isn't a way for me to say "re-run on osic" is there? | 02:30 |
| clarkb | tonyb: there isn't | 02:30 |
| tonyb | clarkb: Oh rats | 02:30 |
| *** baoli has quit IRC | 02:31 | |
| tonyb | clarkb: cool. that's what I thought | 02:31 |
| clarkb | its all demand + quota ratio based | 02:31 |
| *** McCorkis has quit IRC | 02:31 | |
| clarkb | ya I need rax updated to add it back to multinode | 02:31 |
| *** yamamoto_ has quit IRC | 02:34 | |
| clarkb | I have killed the extra builder processes so they won't interfere with tomorrows image build and queued up the remaining uploads for ubuntu-trusty | 02:35 |
| clarkb | 7/9 regions are queued 4 of which should be actively uploading | 02:35 |
| clarkb | its like a star trek episode | 02:35 |
| *** roxanaghe has joined #openstack-infra | 02:36 | |
| clarkb | tonyb: if I have done my maths correctly you have about a 15-20% chance of nabbing a machine on bluebox or osic | 02:37 |
| tonyb | clarkb: hehe | 02:37 |
| *** rkukura has quit IRC | 02:38 | |
| tonyb | clarkb: I'll wait for the odds to be better. I don't want to waste (up to) 4 rechecks worth of CPU time | 02:38 |
| *** baoli has joined #openstack-infra | 02:39 | |
| *** roxanaghe has quit IRC | 02:41 | |
| *** baoli has quit IRC | 02:41 | |
| *** Daisy has joined #openstack-infra | 02:41 | |
| *** baoli has joined #openstack-infra | 02:42 | |
| *** Daisy_ has quit IRC | 02:43 | |
| *** aeng has joined #openstack-infra | 02:44 | |
| *** rlandy has quit IRC | 02:44 | |
| *** jcoufal has quit IRC | 02:47 | |
| *** baoli has quit IRC | 02:48 | |
| *** zeih has joined #openstack-infra | 02:49 | |
| *** baoli has joined #openstack-infra | 02:51 | |
| *** EricGonczer_ has joined #openstack-infra | 02:51 | |
| *** Daisy has quit IRC | 02:51 | |
| *** Daisy has joined #openstack-infra | 02:52 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Create puppet-openstack-guide https://review.openstack.org/296821 | 02:54 |
| *** zeih has quit IRC | 02:54 | |
| *** baoli has quit IRC | 02:54 | |
| clarkb | tonyb: and now internap is ready | 02:57 |
| tonyb | clarkb: thanks /me does a recheck | 02:58 |
| *** IlyaG has joined #openstack-infra | 03:01 | |
| *** Daisy_ has joined #openstack-infra | 03:02 | |
| *** tphummel has joined #openstack-infra | 03:02 | |
| tonyb | clarkb: Gah I got OVH ;P | 03:04 |
| *** tphummel has quit IRC | 03:04 | |
| StevenK | Nakato: https://review.openstack.org/#/c/296863/ if you think it's good, and then I can EOW | 03:04 |
| *** tphummel has joined #openstack-infra | 03:05 | |
| *** Daisy has quit IRC | 03:05 | |
| *** IlyaG has quit IRC | 03:05 | |
| *** EricGonczer_ has quit IRC | 03:06 | |
| *** sflanigan is now known as sflanigan_meetin | 03:07 | |
| Nakato | StevenK: I'll get to it, I'm having lunch. | 03:09 |
| *** piet has quit IRC | 03:09 | |
| StevenK | Nakato: Excellent, no real rush | 03:09 |
| *** ddieterly[away] has quit IRC | 03:10 | |
| clarkb | oh hey vexxhost succeeded woot | 03:11 |
| clarkb | tonyb: so slowly but surely things are updating | 03:11 |
| tonyb | clarkb: :) Did you work out what happend to/at RAX? | 03:11 |
| clarkb | tonyb: no, it often times out | 03:12 |
| clarkb | on the image import task | 03:12 |
| clarkb | which is a giant black box of not open source stuff | 03:12 |
| tonyb | clarkb: :( Yeah makes me sad | 03:12 |
| greghaynes | There is a good commit msg about that | 03:12 |
| greghaynes | about error code 53 IIRC | 03:12 |
| clarkb | also learned something new today | 03:13 |
| clarkb | its possible for raw images to be smaller than vhds by a large amount | 03:13 |
| greghaynes | heh | 03:13 |
| clarkb | not actually smaller but on disk smaller | 03:13 |
| greghaynes | "smaller" | 03:13 |
| clarkb | hey those are bytes saved | 03:13 |
| greghaynes | yea, on disk | 03:13 |
| *** naggappan has quit IRC | 03:13 | |
| clarkb | yup still counts | 03:13 |
| greghaynes | It is a good indication that compression will work out well for that image, too | 03:14 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Create puppet-openstack-guide https://review.openstack.org/296821 | 03:14 |
| *** Nikolay_St has joined #openstack-infra | 03:16 | |
| *** tphummel has quit IRC | 03:17 | |
| *** mriedem has quit IRC | 03:17 | |
| prometheanfire | greghaynes: ping? https://review.openstack.org/294973 | 03:19 |
| *** Daisy_ has quit IRC | 03:19 | |
| *** Daisy has joined #openstack-infra | 03:20 | |
| *** sdake_ has joined #openstack-infra | 03:21 | |
| *** Daisy has quit IRC | 03:21 | |
| greghaynes | prometheanfire: +A'd | 03:21 |
| *** Daisy has joined #openstack-infra | 03:21 | |
| *** yamamoto_ has joined #openstack-infra | 03:21 | |
| *** piet has joined #openstack-infra | 03:22 | |
| *** sdake has quit IRC | 03:22 | |
| prometheanfire | thanks :D | 03:23 |
| *** amitgandhinz has joined #openstack-infra | 03:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Create puppet-openstack-guide https://review.openstack.org/296821 | 03:24 |
| *** piet has quit IRC | 03:26 | |
| *** links has joined #openstack-infra | 03:27 | |
| *** fedexo has joined #openstack-infra | 03:29 | |
| *** rfolco_ has quit IRC | 03:30 | |
| clarkb | tonyb: rax-iad upload worked second time around | 03:34 |
| tonyb | clarkb: I wonder if it caches the partial objects somewhere .... not that it really matters. | 03:35 |
| clarkb | tonyb: actually thats a good point I think it does, the swift upload uses the hash of the image so we should only ever upload that once | 03:35 |
| *** amitgandhinz has quit IRC | 03:36 | |
| *** roxanaghe has joined #openstack-infra | 03:37 | |
| tonyb | clarkb: I'll ask internally | 03:39 |
| clarkb | tonyb: I think thats something we do | 03:39 |
| clarkb | tonyb: shade uploads to swift then runs the import task stuff and the first part should only have to happen once per image if shade is working properly | 03:39 |
| tonyb | clarkb: Ahh yeah okay that makes sense. | 03:40 |
| *** akshai_ has joined #openstack-infra | 03:40 | |
| *** roxanaghe has quit IRC | 03:42 | |
| *** akshai has quit IRC | 03:42 | |
| *** salv-orlando has joined #openstack-infra | 03:43 | |
| *** zeih has joined #openstack-infra | 03:43 | |
| clarkb | greghaynes: have you tried hardcoding a fedora mirror like osuosl? | 03:43 |
| *** sflanigan_meetin is now known as sflanigan | 03:44 | |
| greghaynes | clarkb: yes, it helps a ton | 03:44 |
| greghaynes | clarkb: that's what TripleO CI dies | 03:44 |
| greghaynes | I think some of the dib mirror pinning is broken in Fedora though | 03:44 |
| *** zeih has quit IRC | 03:48 | |
| *** x00350071 has quit IRC | 03:49 | |
| *** achanda has quit IRC | 03:49 | |
| *** aeng has quit IRC | 03:49 | |
| *** salv-orlando has quit IRC | 03:49 | |
| *** x00350071 has joined #openstack-infra | 03:50 | |
| *** aeng has joined #openstack-infra | 03:50 | |
| *** michaelxin has quit IRC | 03:53 | |
| *** lindsayk has quit IRC | 03:53 | |
| *** bswartz has quit IRC | 03:55 | |
| *** bswartz has joined #openstack-infra | 03:56 | |
| *** Nikolay_St has quit IRC | 03:58 | |
| *** Daisy has quit IRC | 03:58 | |
| *** Daisy has joined #openstack-infra | 03:59 | |
| prometheanfire | greghaynes: you talking about this? | 04:01 |
| prometheanfire | 2016-03-24 03:36:17.630 | Curl error (56): Failure when receiving data from the peer for ftp://fedora.mirrors.ovh.net/download.fedora.redhat.com/linux/updates/22/x86_64/drpms/python-dnf-1.0.0-1.fc22_1.1.6-2.fc22.noarch.drpm [response reading failed] | 04:01 |
| greghaynes | prometheanfire: yep, we should pin that to a known good mirror | 04:01 |
| greghaynes | prometheanfire: also, the irony there is that job ran in the ovh DC | 04:02 |
| prometheanfire | so don't recheck yet? | 04:02 |
| greghaynes | so I dont think they can blame networking | 04:02 |
| greghaynes | no, recheck it, no one has taken the time to fix pining | 04:02 |
| prometheanfire | k | 04:02 |
| greghaynes | but someone should | 04:02 |
| *** Rockyg has quit IRC | 04:03 | |
| *** Daisy has quit IRC | 04:03 | |
| *** fawadkhaliq has joined #openstack-infra | 04:03 | |
| *** verdurin has joined #openstack-infra | 04:03 | |
| *** jerryz has joined #openstack-infra | 04:04 | |
| *** MarkAtwood has joined #openstack-infra | 04:08 | |
| *** akshai has joined #openstack-infra | 04:11 | |
| openstackgerrit | Remo Mattei proposed openstack-infra/project-config: Add a mailing list for the Italian i18n local group. https://review.openstack.org/296883 | 04:12 |
| *** MarkAtwood has quit IRC | 04:12 | |
| *** gildub has quit IRC | 04:14 | |
| *** akshai_ has quit IRC | 04:14 | |
| *** gildub has joined #openstack-infra | 04:14 | |
| *** dimtruck is now known as zz_dimtruck | 04:14 | |
| *** michaelxin has joined #openstack-infra | 04:16 | |
| *** IlyaG has joined #openstack-infra | 04:17 | |
| *** IlyaG has quit IRC | 04:21 | |
| *** lindsayk has joined #openstack-infra | 04:22 | |
| *** treaki__ has joined #openstack-infra | 04:23 | |
| *** watanabe_isao has joined #openstack-infra | 04:24 | |
| *** treaki_ has quit IRC | 04:25 | |
| *** Daisy has joined #openstack-infra | 04:25 | |
| *** fawadkhaliq has quit IRC | 04:27 | |
| *** fawadkhaliq has joined #openstack-infra | 04:27 | |
| *** sdake_ is now known as sdake | 04:29 | |
| ianw | it's on the list -- after i fix the periodic build failures when packages update underneath the cache | 04:29 |
| *** aspiers has joined #openstack-infra | 04:30 | |
| *** aeng has quit IRC | 04:30 | |
| openstackgerrit | Remo Mattei proposed openstack-infra/system-config: Adding the Italian Mailing List https://review.openstack.org/296887 | 04:31 |
| *** abregman|afk has quit IRC | 04:33 | |
| *** amitgandhinz has joined #openstack-infra | 04:33 | |
| greghaynes | ah, awesome. I have been meaning to but ETIME | 04:37 |
| *** MarkAtwood has joined #openstack-infra | 04:38 | |
| *** tonytan4ever has quit IRC | 04:40 | |
| clarkb | tonyb: these last few uploads are all still going | 04:41 |
| clarkb | I am going to have to call it a day soon | 04:42 |
| clarkb | jhesketh: ^ maybe you want to keep an eye on those last few? ovh and rax-dfw/ord | 04:42 |
| *** bryan_att has quit IRC | 04:42 | |
| jhesketh | clarkb: sure, they are just waiting to upload? | 04:42 |
| clarkb | jhesketh: they are uploading now, its just a mtter of restarting them if they fail | 04:43 |
| *** fedexo has quit IRC | 04:44 | |
| jhesketh | clarkb: are you doing it in a screen ? | 04:45 |
| clarkb | jhesketh: yup its owned by me | 04:45 |
| clarkb | you can apparently script /dev/null && screen -r if you su to me first | 04:45 |
| clarkb | or you can just check against nodepool image-list | grep building and start your own if they fail | 04:46 |
| *** amitgandhinz has quit IRC | 04:47 | |
| clarkb | it looks like one of them timed out but I think shade retries at least once in that case | 04:47 |
| jhesketh | clarkb: I'll watch them via image-list and if they fail I'll join your session and take a look | 04:47 |
| *** mylu has quit IRC | 04:48 | |
| *** chlong has quit IRC | 04:56 | |
| bkero | So I know that infra has been considering getting a rubygems mirror going. Does this look like it would fit the bill? (Is functional, but could use improvements). https://github.com/bkero/bkero-rubygems_mirror | 04:56 |
| openstackgerrit | Merged openstack-dev/pbr: Do not convert git tags when searching history https://review.openstack.org/296863 | 04:58 |
| *** achanda has joined #openstack-infra | 04:59 | |
| *** salv-orlando has joined #openstack-infra | 05:02 | |
| *** zeih has joined #openstack-infra | 05:03 | |
| *** rossella_s has quit IRC | 05:03 | |
| *** jerryz has quit IRC | 05:03 | |
| lifeless | StevenK: Nakato: |o/ | 05:03 |
| *** rossella_s has joined #openstack-infra | 05:03 | |
| *** baoli has joined #openstack-infra | 05:06 | |
| *** mylu has joined #openstack-infra | 05:07 | |
| *** zeih has quit IRC | 05:08 | |
| *** aspiers_ has joined #openstack-infra | 05:08 | |
| *** aspiers has quit IRC | 05:10 | |
| *** binbincong has quit IRC | 05:10 | |
| *** baoli has quit IRC | 05:11 | |
| *** McCorkis has joined #openstack-infra | 05:11 | |
| openstackgerrit | Merged openstack/diskimage-builder: add pkg-map to pip-and-virtualenv element https://review.openstack.org/294973 | 05:12 |
| *** chlong has joined #openstack-infra | 05:12 | |
| *** sdake has quit IRC | 05:16 | |
| *** salv-orlando has quit IRC | 05:17 | |
| *** tphummel has joined #openstack-infra | 05:18 | |
| *** sdake has joined #openstack-infra | 05:18 | |
| *** abregman has joined #openstack-infra | 05:19 | |
| abregman | morning | 05:19 |
| *** sdake_ has joined #openstack-infra | 05:20 | |
| *** akshai has quit IRC | 05:22 | |
| *** sdake has quit IRC | 05:23 | |
| *** roxanaghe has joined #openstack-infra | 05:25 | |
| *** binbincong has joined #openstack-infra | 05:27 | |
| *** binbincong has quit IRC | 05:27 | |
| *** markvoelker has quit IRC | 05:28 | |
| *** roxanaghe has quit IRC | 05:30 | |
| *** fawadkhaliq has quit IRC | 05:32 | |
| *** gongysh_ has quit IRC | 05:32 | |
| *** fawadkhaliq has joined #openstack-infra | 05:32 | |
| *** mtanino has quit IRC | 05:35 | |
| *** harlowja_at_home has quit IRC | 05:43 | |
| *** amitgandhinz has joined #openstack-infra | 05:43 | |
| *** HeOS has joined #openstack-infra | 05:43 | |
| *** asettle has quit IRC | 05:44 | |
| *** asettle has joined #openstack-infra | 05:44 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Update the Nova Scheduler meeting https://review.openstack.org/296786 | 05:44 |
| *** asettle has quit IRC | 05:45 | |
| *** binbincong has joined #openstack-infra | 05:45 | |
| *** sdake_ is now known as sdake | 05:45 | |
| *** pcaruana has quit IRC | 05:48 | |
| tonyb | jhesketh: any chance you can check on clarkb's upload? | 05:51 |
| *** mylu has quit IRC | 05:51 | |
| jhesketh | tonyb: yep, I'm watching it | 05:51 |
| *** mylu has joined #openstack-infra | 05:52 | |
| jhesketh | tonyb: it's still running as far as I can tell | 05:52 |
| tonyb | jhesketh: cool. do OVH or RAX have the new image? | 05:52 |
| jhesketh | no | 05:52 |
| tonyb | jhesketh: thanks. | 05:53 |
| jhesketh | tonyb: rax-iad should do, but not the other regions | 05:53 |
| *** amitgandhinz has quit IRC | 05:56 | |
| *** Guest65562 has quit IRC | 05:58 | |
| *** Daisy has quit IRC | 05:58 | |
| *** Daisy has joined #openstack-infra | 05:58 | |
| *** tqtran has joined #openstack-infra | 06:01 | |
| *** Daisy has quit IRC | 06:03 | |
| *** Daisy has joined #openstack-infra | 06:03 | |
| *** Douhet has quit IRC | 06:03 | |
| *** bdemers has quit IRC | 06:03 | |
| *** Daisy_ has joined #openstack-infra | 06:04 | |
| *** mylu has quit IRC | 06:04 | |
| *** Douhet has joined #openstack-infra | 06:05 | |
| jhesketh | tonyb: restarted the failed ones... nodepool didn't clean them up so I assumed they were still trying but looks like they had failed | 06:05 |
| *** Daisy_ has quit IRC | 06:05 | |
| *** Daisy_ has joined #openstack-infra | 06:05 | |
| tonyb | jhesketh: Thanks. | 06:06 |
| *** tqtran has quit IRC | 06:06 | |
| *** bdemers has joined #openstack-infra | 06:07 | |
| *** Daisy has quit IRC | 06:07 | |
| * jhesketh continues to watch | 06:07 | |
| *** chlong has quit IRC | 06:08 | |
| *** oanson has joined #openstack-infra | 06:09 | |
| *** Daisy_ has quit IRC | 06:10 | |
| *** MarkAtwood has quit IRC | 06:11 | |
| *** oanson has quit IRC | 06:14 | |
| *** jaosorior has joined #openstack-infra | 06:15 | |
| *** jaosorior has quit IRC | 06:17 | |
| *** jaosorior has joined #openstack-infra | 06:18 | |
| *** x00350071 has quit IRC | 06:21 | |
| *** salv-orlando has joined #openstack-infra | 06:21 | |
| *** chlong has joined #openstack-infra | 06:22 | |
| *** rcernin has joined #openstack-infra | 06:23 | |
| *** gongysh_ has joined #openstack-infra | 06:24 | |
| *** MarkAtwood has joined #openstack-infra | 06:24 | |
| *** salv-orlando has quit IRC | 06:24 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/296903 | 06:25 |
| *** oanson has joined #openstack-infra | 06:26 | |
| *** markvoelker has joined #openstack-infra | 06:29 | |
| *** ybathia has joined #openstack-infra | 06:30 | |
| *** gongysh_ has quit IRC | 06:32 | |
| *** markvoelker has quit IRC | 06:34 | |
| *** zhurong has quit IRC | 06:34 | |
| *** tphummel has quit IRC | 06:36 | |
| *** zhurong has joined #openstack-infra | 06:38 | |
| *** Nikolay_St has joined #openstack-infra | 06:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/288849 | 06:41 |
| *** Daisy has joined #openstack-infra | 06:41 | |
| *** annegentle has quit IRC | 06:42 | |
| *** Daisy has quit IRC | 06:44 | |
| *** Daisy has joined #openstack-infra | 06:45 | |
| *** esker has quit IRC | 06:46 | |
| *** Daisy has quit IRC | 06:49 | |
| *** fawadkhaliq has quit IRC | 06:53 | |
| *** amitgandhinz has joined #openstack-infra | 06:54 | |
| *** mylu has joined #openstack-infra | 07:01 | |
| *** gongysh_ has joined #openstack-infra | 07:02 | |
| *** mylu has quit IRC | 07:06 | |
| *** lifeless has quit IRC | 07:06 | |
| *** amitgandhinz has quit IRC | 07:07 | |
| *** Daisy has joined #openstack-infra | 07:08 | |
| *** mikelk has joined #openstack-infra | 07:11 | |
| *** esikachev has joined #openstack-infra | 07:12 | |
| *** roxanaghe has joined #openstack-infra | 07:13 | |
| *** flepied has quit IRC | 07:15 | |
| *** ybathia has quit IRC | 07:16 | |
| *** flepied has joined #openstack-infra | 07:16 | |
| *** akaszuba has joined #openstack-infra | 07:17 | |
| *** roxanaghe has quit IRC | 07:18 | |
| *** yamahata has quit IRC | 07:21 | |
| *** HeOS has quit IRC | 07:21 | |
| *** flepied has quit IRC | 07:25 | |
| *** lmiccini_ is now known as lmiccini | 07:26 | |
| cj | I was maybe wrong about the exceptions in gerritlib... it looks like paramiko is having some major malfunctions with regard to over-generating exceptions. | 07:26 |
| cj | https://github.com/paramiko/paramiko/issues/387 | 07:26 |
| cj | I'm going to see if I can resolve that problem by figuring out how to extract the key type from the private key data. | 07:26 |
| *** ihrachys has joined #openstack-infra | 07:29 | |
| cj | the return value of this function has the information needed: | 07:29 |
| cj | https://github.com/openssl/openssl/blob/master/crypto/pem/pem_pkey.c#L74 | 07:29 |
| cj | so all I've got to do is translate it so python. easy enough, righ? Ha. | 07:29 |
| *** pcaruana has joined #openstack-infra | 07:32 | |
| *** daemontool has joined #openstack-infra | 07:33 | |
| *** achanda has quit IRC | 07:33 | |
| yolanda | morning | 07:36 |
| *** achanda has joined #openstack-infra | 07:37 | |
| *** tesseract has joined #openstack-infra | 07:41 | |
| *** tesseract is now known as Guest27982 | 07:42 | |
| *** annegentle has joined #openstack-infra | 07:42 | |
| *** ihrachys has quit IRC | 07:45 | |
| *** ihrachys has joined #openstack-infra | 07:45 | |
| *** scheuran has joined #openstack-infra | 07:48 | |
| *** ihrachys has quit IRC | 07:49 | |
| *** ihrachys has joined #openstack-infra | 07:49 | |
| *** annegentle has quit IRC | 07:50 | |
| *** nmagnezi has joined #openstack-infra | 07:54 | |
| *** flepied has joined #openstack-infra | 07:55 | |
| *** achanda has quit IRC | 07:56 | |
| *** markvoelker has joined #openstack-infra | 07:56 | |
| *** salv-orlando has joined #openstack-infra | 07:57 | |
| masayukig | yolanda: so, Now, openstack-health site is something wrong... http://status.openstack.org/openstack-health/#/ Do you know how to get the logs (or something) to debug? | 07:58 |
| *** flepied has quit IRC | 07:58 | |
| *** flepied1 has joined #openstack-infra | 07:58 | |
| yolanda | hi masayukig, i'll enter into the instance and look | 07:58 |
| *** flepied1 has quit IRC | 07:59 | |
| masayukig | yolanda: oh! thanks:) | 07:59 |
| *** ifarkas has joined #openstack-infra | 07:59 | |
| masayukig | API server seems working... http://health.openstack.org/ | 07:59 |
| *** MarkAtwood has quit IRC | 08:00 | |
| *** MarkAtwood has joined #openstack-infra | 08:00 | |
| yolanda | masayukig, i see timeout errors | 08:01 |
| yolanda | TimeoutError: QueuePool limit of size 20 overflow 10 reached, connection timed out, timeout 30 | 08:01 |
| *** flepied has joined #openstack-infra | 08:01 | |
| *** salv-orlando has quit IRC | 08:02 | |
| masayukig | yolanda: thanks :), but hmm... | 08:02 |
| *** markvoelker has quit IRC | 08:02 | |
| *** sdake has quit IRC | 08:02 | |
| *** amitgandhinz has joined #openstack-infra | 08:03 | |
| yolanda | masayukig, how do i better restart it? restarting apache is enough? | 08:03 |
| *** matrohon has joined #openstack-infra | 08:03 | |
| masayukig | yolanda: yeah, it might recover this error. | 08:04 |
| yolanda | it worked | 08:04 |
| *** Hal has joined #openstack-infra | 08:04 | |
| yolanda | masayukig, although there may be some errors in the app, going to paste you output | 08:04 |
| *** Hal is now known as Guest39452 | 08:05 | |
| yolanda | http://paste.openstack.org/show/491669/ | 08:05 |
| masayukig | yolanda: Thanks! I suppose these error happened several times... We need to find the root cause.. :( | 08:06 |
| masayukig | yolanda: thanks again, I'll check the log. | 08:06 |
| jhesketh | hmm so ovh-bhs1 is still failing to upload | 08:06 |
| yolanda | masayukig, glad to help | 08:06 |
| *** sdake has joined #openstack-infra | 08:06 | |
| jhesketh | rax-ord is going but the rest are done | 08:06 |
| *** achanda has joined #openstack-infra | 08:10 | |
| yolanda | jhesketh, ovh is normally very slow regarding i/o right? | 08:11 |
| AJaeger | yolanda, pabelanger, fungi, jhesketh: https://review.openstack.org/#/c/296039 - the removal of db access from python unit tests - is ready to merge. It changes lots of jobs, so will need manual JJB runs. I removed my WIP, feel free to approve if you have time. I'll be out most of today, so can't help further. | 08:14 |
| yolanda | i work today, i have holiday on friday and monday | 08:15 |
| openstackgerrit | Merged openstack-infra/puppet-openstackid: Update prefork MPM apache settings https://review.openstack.org/296708 | 08:15 |
| yolanda | i can have fun with jjb | 08:15 |
| AJaeger | yolanda: I took holiday a day early ;) | 08:16 |
| *** amitgandhinz has quit IRC | 08:16 | |
| yolanda | well done :) | 08:16 |
| yolanda | have a nice easter holiday | 08:17 |
| AJaeger | yolanda: for today it means doing all the things I don't manage to do otherwise like dumping some rubbish or some shopping ;) | 08:18 |
| *** sdake has quit IRC | 08:18 | |
| AJaeger | yolanda: enjoy your easter holiday as well and thanks for having fun with jjb.;) | 08:18 |
| *** mrmartin has joined #openstack-infra | 08:18 | |
| AJaeger | yolanda: so, feel free to +A once you're ready on 296039 | 08:18 |
| AJaeger | I'll check back in later - thanks | 08:18 |
| yolanda | sure, i'll do a first round of reviews to propagate small changes on projects, then go with that | 08:19 |
| *** mylu has joined #openstack-infra | 08:19 | |
| AJaeger | ok | 08:19 |
| openstackgerrit | Evgeny Antyshev proposed openstack-infra/zuul: Warning if .ssh_wrapper not found https://review.openstack.org/255956 | 08:21 |
| *** tmorin has joined #openstack-infra | 08:21 | |
| *** salv-orl_ has joined #openstack-infra | 08:21 | |
| *** oanson has quit IRC | 08:22 | |
| tmorin | hi AJaeger, ianw: we have a short/trivial fix for the networking-bgpvpn jobs ( https://review.openstack.org/#/c/296812/2 ), if you have two minutes... | 08:22 |
| tmorin | matrohon: ^ | 08:22 |
| openstackgerrit | Merged openstack-infra/project-config: Use standard python3-jobs definition for oslo.messaging https://review.openstack.org/296374 | 08:22 |
| *** mylu has quit IRC | 08:23 | |
| openstackgerrit | Merged openstack-infra/project-config: Add docs publishing job to osprofiler https://review.openstack.org/296584 | 08:24 |
| openstackgerrit | Merged openstack-infra/project-config: Add barbican client to project list for lbaas testing https://review.openstack.org/296777 | 08:25 |
| *** sflanigan has quit IRC | 08:26 | |
| *** shardy has joined #openstack-infra | 08:27 | |
| *** dingyichen has quit IRC | 08:29 | |
| *** e0ne has joined #openstack-infra | 08:29 | |
| *** jlanoux has joined #openstack-infra | 08:30 | |
| *** jlanoux has quit IRC | 08:31 | |
| *** HeOS has joined #openstack-infra | 08:31 | |
| *** hichihara has quit IRC | 08:32 | |
| *** jlanoux has joined #openstack-infra | 08:32 | |
| *** jpr has joined #openstack-infra | 08:34 | |
| *** vincentll has joined #openstack-infra | 08:34 | |
| *** MarkAtwood has quit IRC | 08:34 | |
| matrohon | tmorin, AJaeger, ianw : I discussed about 296812 with anteaya and clarkb yesterday | 08:35 |
| *** oanson has joined #openstack-infra | 08:35 | |
| matrohon | tmorin, AJaeger, ianw : they agreed on the change | 08:35 |
| tmorin | ok, great | 08:35 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 08:36 |
| openstackgerrit | Merged openstack-infra/project-config: Remove db set up from python unittests https://review.openstack.org/296039 | 08:37 |
| *** chlong has quit IRC | 08:38 | |
| openstackgerrit | Merged openstack-infra/shade: Use OpenStackCloudException when _delete_server() raises https://review.openstack.org/296731 | 08:38 |
| *** esikachev has quit IRC | 08:41 | |
| *** matrohon has quit IRC | 08:42 | |
| *** yaume has joined #openstack-infra | 08:43 | |
| *** yaume is now known as yaume[away] | 08:43 | |
| jhesketh | yolanda: it's not timing out its failing to hit an endpoint. I had to head out though I'll look closer when I get back sorry | 08:43 |
| *** esikachev has joined #openstack-infra | 08:43 | |
| *** achanda has quit IRC | 08:45 | |
| *** mrmartin has quit IRC | 08:45 | |
| openstackgerrit | Zhang Ni proposed openstack-infra/project-config: Conveyor focues on cloning and migrating Openstack resources. https://review.openstack.org/296951 | 08:46 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Don't run releasenotes job for neutronclient till Mitaka https://review.openstack.org/296952 | 08:47 |
| *** esikachev has quit IRC | 08:48 | |
| *** GonZo2000 has quit IRC | 08:50 | |
| *** Jeffrey4l has joined #openstack-infra | 08:50 | |
| *** nadya has joined #openstack-infra | 08:51 | |
| *** niska has quit IRC | 08:51 | |
| *** jordanP has joined #openstack-infra | 08:52 | |
| *** tosky has joined #openstack-infra | 08:53 | |
| *** achanda has joined #openstack-infra | 08:56 | |
| *** esikachev has joined #openstack-infra | 08:56 | |
| *** watanabe_isao has quit IRC | 08:59 | |
| *** yaume[away] is now known as yaume | 09:00 | |
| *** roxanaghe has joined #openstack-infra | 09:02 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:02 | |
| *** ihrachys has quit IRC | 09:03 | |
| *** rossella_s has quit IRC | 09:03 | |
| *** rossella_s has joined #openstack-infra | 09:04 | |
| *** vgridnev has joined #openstack-infra | 09:05 | |
| *** ramishra has quit IRC | 09:05 | |
| *** roxanaghe has quit IRC | 09:06 | |
| *** chem has joined #openstack-infra | 09:07 | |
| *** ramishra has joined #openstack-infra | 09:07 | |
| openstackgerrit | Zhang Ni proposed openstack-infra/project-config: Conveyor focues on cloning and migrating Openstack resources. https://review.openstack.org/296951 | 09:10 |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 09:10 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Left sudo rights for ironic-multitenant-network https://review.openstack.org/296963 | 09:11 |
| *** gnuoy_ has joined #openstack-infra | 09:11 | |
| *** gnuoy_ has quit IRC | 09:11 | |
| *** lucas-dinner is now known as lucasagomes | 09:11 | |
| *** dalgaaf has joined #openstack-infra | 09:12 | |
| *** amitgandhinz has joined #openstack-infra | 09:13 | |
| *** gildub has quit IRC | 09:18 | |
| *** zeih has joined #openstack-infra | 09:22 | |
| *** amitgandhinz has quit IRC | 09:26 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Left sudo rights for strack in ironic-multitenant-network job https://review.openstack.org/296963 | 09:26 |
| *** achanda has quit IRC | 09:26 | |
| *** kzaitsev_mb has quit IRC | 09:27 | |
| openstackgerrit | Zhang Ni proposed openstack-infra/project-config: Conveyor focues on cloning and migrating Openstack resources. https://review.openstack.org/296951 | 09:30 |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/project-config: Add job to test Nova with specific libvirt and kvm packages https://review.openstack.org/289255 | 09:31 |
| markus_z | bauzas: As a heads up, we (Germany) have 2 public holidays tomorrow and Monday. In case anyone asks about me. | 09:32 |
| *** jcoufal has joined #openstack-infra | 09:33 | |
| *** sfinucan has joined #openstack-infra | 09:34 | |
| *** flepied1 has joined #openstack-infra | 09:34 | |
| *** flepied has quit IRC | 09:35 | |
| markus_z | whoops, wrong channel | 09:35 |
| *** andymaier has joined #openstack-infra | 09:36 | |
| *** Daisy has quit IRC | 09:37 | |
| *** salv-orl_ has quit IRC | 09:37 | |
| *** Daisy has joined #openstack-infra | 09:37 | |
| *** salv-orlando has joined #openstack-infra | 09:37 | |
| yolanda | AJaeger, monitoring jjb, seems it's updating jobs | 09:38 |
| *** zeih has quit IRC | 09:38 | |
| vincentll | hi, can someone from the infra team add me to both synergy-core & synergy-release groups? my handle is vincentll, thanks | 09:41 |
| *** sfinucan has quit IRC | 09:41 | |
| yolanda | hi vincentll , can you provide the link for project creation? | 09:41 |
| *** ihrachys has joined #openstack-infra | 09:41 | |
| vincentll | yolanda: https://review.openstack.org/#/c/283157/ | 09:42 |
| *** Daisy has quit IRC | 09:42 | |
| yolanda | done | 09:45 |
| vincentll | yolanda: thanks very much :) | 09:46 |
| yolanda | anytime | 09:46 |
| *** mrmartin has joined #openstack-infra | 09:47 | |
| vsaienko | hello infra team, please review https://review.openstack.org/#/c/296963/ this patch left sudo rights for stack user in ironic-multitenant-network job. | 09:48 |
| vsaienko | I've missed that sudo rights are removed by default. | 09:48 |
| *** aspiers_ is now known as aspiers | 09:49 | |
| *** _degorenko|afk is now known as degorenko | 09:51 | |
| *** niska has joined #openstack-infra | 09:55 | |
| *** markvoelker has joined #openstack-infra | 09:58 | |
| *** oanson has quit IRC | 09:59 | |
| *** tosky has quit IRC | 10:00 | |
| *** markvoelker has quit IRC | 10:03 | |
| *** zhurong has quit IRC | 10:06 | |
| *** gnuoy_ has joined #openstack-infra | 10:06 | |
| *** Jeffrey4l has quit IRC | 10:06 | |
| *** mylu has joined #openstack-infra | 10:07 | |
| *** tpsilva has joined #openstack-infra | 10:07 | |
| *** gnuoy_ has quit IRC | 10:07 | |
| *** sfinucan has joined #openstack-infra | 10:09 | |
| *** esikachev has quit IRC | 10:11 | |
| *** mylu has quit IRC | 10:12 | |
| openstackgerrit | Igor Kalnitsky proposed openstack-infra/project-config: fuel-web: use db setup for python jobs https://review.openstack.org/296989 | 10:13 |
| *** imcsk8 has quit IRC | 10:14 | |
| *** imcsk8 has joined #openstack-infra | 10:14 | |
| *** mrmartin has quit IRC | 10:14 | |
| *** MarkAtwood has joined #openstack-infra | 10:15 | |
| *** esikachev has joined #openstack-infra | 10:15 | |
| *** asettle has joined #openstack-infra | 10:16 | |
| *** Jeffrey4l has joined #openstack-infra | 10:18 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Fix voting for amqp1-centos oslo.messaging jobs https://review.openstack.org/296781 | 10:18 |
| *** ikalnitsky has joined #openstack-infra | 10:18 | |
| *** MarkAtwood has quit IRC | 10:20 | |
| *** niska has quit IRC | 10:20 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Left sudo rights for stack in ironic-multitenant-network job https://review.openstack.org/296963 | 10:20 |
| *** gongysh_ has quit IRC | 10:21 | |
| *** electrofelix has joined #openstack-infra | 10:22 | |
| *** tojuvone has quit IRC | 10:22 | |
| ikalnitsky | hey folks! could you please review that patch - https://review.openstack.org/#/c/296989/1 - it blocks our development :( | 10:22 |
| *** tojuvone has joined #openstack-infra | 10:22 | |
| ikalnitsky | AJaeger: would you be so kind to review and merge if it's ok? ^ | 10:22 |
| *** gongysh_ has joined #openstack-infra | 10:23 | |
| *** amitgandhinz has joined #openstack-infra | 10:23 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Left sudo rights for networking-generic-switch job https://review.openstack.org/296996 | 10:23 |
| *** mrmartin has joined #openstack-infra | 10:23 | |
| *** asettle has quit IRC | 10:24 | |
| *** esikachev has quit IRC | 10:24 | |
| *** sdague has joined #openstack-infra | 10:24 | |
| *** rguillebert has joined #openstack-infra | 10:26 | |
| dims | AJaeger : i fixed this one https://review.openstack.org/#/c/296781/ - i was indeed running specified bad stable/ versions in regex | 10:28 |
| *** sheeprine has quit IRC | 10:28 | |
| *** jpr1 has joined #openstack-infra | 10:30 | |
| *** jpr has quit IRC | 10:30 | |
| *** sheeprine has joined #openstack-infra | 10:30 | |
| *** niska has joined #openstack-infra | 10:33 | |
| *** jaosorior has quit IRC | 10:34 | |
| *** Jeffrey4l has quit IRC | 10:35 | |
| *** jpr1 has quit IRC | 10:35 | |
| *** amitgandhinz has quit IRC | 10:36 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: Adjust grenade branches for mitaka https://review.openstack.org/297004 | 10:37 |
| openstackgerrit | Christian Schwede proposed openstack-infra/project-config: WIP: Add grenade job for Swift rolling upgrade tests https://review.openstack.org/297005 | 10:38 |
| *** cdent has joined #openstack-infra | 10:39 | |
| *** oanson has joined #openstack-infra | 10:42 | |
| *** jaosorior has joined #openstack-infra | 10:43 | |
| *** mrmartin has quit IRC | 10:45 | |
| *** erlon has joined #openstack-infra | 10:45 | |
| *** jpr has joined #openstack-infra | 10:45 | |
| *** daemontool has quit IRC | 10:46 | |
| *** tosky has joined #openstack-infra | 10:46 | |
| *** mrmartin has joined #openstack-infra | 10:47 | |
| openstackgerrit | Thomas Herve proposed openstack-dev/pbr: Fix wsgiref script use with oslo.config https://review.openstack.org/296061 | 10:48 |
| *** roxanaghe has joined #openstack-infra | 10:50 | |
| *** gongysh_ has quit IRC | 10:51 | |
| *** claudiub|2 has quit IRC | 10:52 | |
| *** roxanaghe has quit IRC | 10:54 | |
| *** ociuhandu has quit IRC | 10:57 | |
| *** nadya has quit IRC | 10:58 | |
| markus_z | If someone could have a look at https://review.openstack.org/#/c/289255/6 , that would be great. | 11:04 |
| *** matrohon has joined #openstack-infra | 11:07 | |
| *** lifeless has joined #openstack-infra | 11:07 | |
| *** abregman has quit IRC | 11:09 | |
| *** abregman has joined #openstack-infra | 11:09 | |
| *** yamamoto_ has quit IRC | 11:10 | |
| *** baoli has joined #openstack-infra | 11:11 | |
| *** jcoufal has quit IRC | 11:13 | |
| *** flepied1 has quit IRC | 11:16 | |
| *** MarkAtwood has joined #openstack-infra | 11:16 | |
| *** mrmartin has quit IRC | 11:16 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 11:17 |
| *** ociuhandu has joined #openstack-infra | 11:18 | |
| *** baoli_ has joined #openstack-infra | 11:18 | |
| *** hashar has joined #openstack-infra | 11:19 | |
| *** baoli_ has quit IRC | 11:20 | |
| *** baoli_ has joined #openstack-infra | 11:20 | |
| *** MarkAtwood has quit IRC | 11:20 | |
| *** baoli has quit IRC | 11:21 | |
| *** hashar has quit IRC | 11:23 | |
| dulek | Hi, I've just got 502 - proxy error on Gerrit - I guess Gerrit is getting slow again. | 11:23 |
| lucasagomes | hi all, where do I can find previous logs from a post job ( https://jenkins.openstack.org/job/ironic-python-agent-buildimage-coreos/ ) | 11:24 |
| *** esikachev has joined #openstack-infra | 11:25 | |
| lucasagomes | we believe that this post job is failing because the images haven't been updated in a while http://tarballs.openstack.org/ironic-python-agent/coreos/files/ | 11:25 |
| *** electrofelix has quit IRC | 11:26 | |
| HeOS | dulek, I don't see the problem. | 11:26 |
| yolanda | lucasagomes, go through every jenkins master (from jenkins01 to jenkins07) and look at https://jenkins01.openstack.org/job/ironic-python-agent-buildimage-coreos/ | 11:28 |
| yolanda | you will see job history there and inside the console log you can get links to the full logs | 11:28 |
| yolanda | dulek, i'm not hitting that as well | 11:28 |
| yolanda | but if that continues being slow, i'll restart the service | 11:28 |
| *** flepied has joined #openstack-infra | 11:29 | |
| *** _amrith_ is now known as amrith | 11:30 | |
| *** baoli_ has quit IRC | 11:31 | |
| *** baoli has joined #openstack-infra | 11:31 | |
| *** nadya has joined #openstack-infra | 11:32 | |
| dulek | HeOS, yolanda: Sure, maybe it was just a hiccup. Thanks for checking! | 11:33 |
| *** amitgandhinz has joined #openstack-infra | 11:33 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with non-HA jobs https://review.openstack.org/297038 | 11:33 |
| *** weshay has joined #openstack-infra | 11:34 | |
| *** pradk_ is now known as pradk | 11:35 | |
| *** baoli has quit IRC | 11:35 | |
| *** baoli has joined #openstack-infra | 11:35 | |
| *** vgridnev has quit IRC | 11:36 | |
| *** vgridnev has joined #openstack-infra | 11:36 | |
| *** nadya has quit IRC | 11:37 | |
| lucasagomes | yolanda, thank you very much! I will take a look | 11:37 |
| *** esikachev has quit IRC | 11:40 | |
| *** sfinucan has quit IRC | 11:40 | |
| *** ldnunes has joined #openstack-infra | 11:40 | |
| *** otsuka has joined #openstack-infra | 11:41 | |
| *** otsuka has quit IRC | 11:41 | |
| openstackgerrit | KATO Tomoyuki proposed openstack-infra/project-config: Add release notes translation scripts https://review.openstack.org/297045 | 11:43 |
| *** thorst has joined #openstack-infra | 11:43 | |
| *** sfinucan has joined #openstack-infra | 11:43 | |
| *** amitgandhinz has quit IRC | 11:46 | |
| openstackgerrit | Christian Schwede proposed openstack-infra/project-config: WIP: Add grenade job for Swift rolling upgrade tests https://review.openstack.org/297005 | 11:46 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Fix on External Url Service https://review.openstack.org/297049 | 11:46 |
| *** annegentle has joined #openstack-infra | 11:47 | |
| *** smarcet has joined #openstack-infra | 11:48 | |
| *** aysyd has joined #openstack-infra | 11:51 | |
| *** annegentle has quit IRC | 11:52 | |
| *** rhallisey has joined #openstack-infra | 11:52 | |
| *** edmondsw has joined #openstack-infra | 11:54 | |
| *** mylu has joined #openstack-infra | 11:55 | |
| openstackgerrit | Merged openstack-infra/puppet-sudoers: Add secure_path setting to match upstream sudoers https://review.openstack.org/296598 | 11:56 |
| *** nadya has joined #openstack-infra | 11:57 | |
| *** markvoelker has joined #openstack-infra | 11:59 | |
| openstackgerrit | Merged openstack-infra/openstackid: Fix on External Url Service https://review.openstack.org/297049 | 11:59 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: Adjust grenade branches for mitaka https://review.openstack.org/297004 | 11:59 |
| *** mylu has quit IRC | 11:59 | |
| *** rfolco_ has joined #openstack-infra | 12:00 | |
| *** salv-orl_ has joined #openstack-infra | 12:02 | |
| *** salv-orl_ has quit IRC | 12:02 | |
| odyssey4me | Does the merge of https://review.openstack.org/#/c/296598/ take immediate effect, or does that just affect the next run of images built? | 12:03 |
| ikalnitsky | sdague, AJaeger, ianw: gentlemen, sorry for being insist, could you please take a look? https://review.openstack.org/#/c/296989/ we really need to address it asap. it blocks our gates | 12:03 |
| *** tqtran has joined #openstack-infra | 12:03 | |
| *** jcoufal has joined #openstack-infra | 12:03 | |
| *** electrofelix has joined #openstack-infra | 12:04 | |
| *** salv-orlando has quit IRC | 12:05 | |
| *** daemontool has joined #openstack-infra | 12:07 | |
| *** tqtran has quit IRC | 12:08 | |
| mordred | bkero: puppet module looks good, yeah. questions: what does the on-disk tree of a gem mirror look like? and is there a way we can think of to add a post-mirror command to the cron you've got there? | 12:09 |
| mordred | bkero: both questions are related to AFS - we use hashed directory structures for the other mirrors to keep the items-per-directory down. and if the mirror run is successful, we need to run "vos release" on the AFS volume to publish the changes | 12:10 |
| *** mrmartin has joined #openstack-infra | 12:10 | |
| mordred | bkero: although one could imagine that people might want to do similar things with non-AFS - like a zfs commit or something | 12:10 |
| *** cdent has quit IRC | 12:11 | |
| smarcet | fungi: morning, i am doing some load test rite now on openstackid.org | 12:14 |
| smarcet | fungi: with current apache mpm prefork model ,things are not doing so well, i think that we should move to php5-fpm | 12:14 |
| smarcet | but that will entail a refactor of the puppet manifest | 12:14 |
| *** daemontool has quit IRC | 12:15 | |
| *** MarkAtwood has joined #openstack-infra | 12:17 | |
| *** yamamoto has joined #openstack-infra | 12:17 | |
| mattt | odyssey4me: going to submit a dummy os_cinder review and see if/how it fails | 12:18 |
| pabelanger | morning | 12:20 |
| russellb | morning! | 12:21 |
| *** mriedem has joined #openstack-infra | 12:21 | |
| sbelous | dtroyer: ping | 12:21 |
| *** MarkAtwood has quit IRC | 12:22 | |
| *** kgiusti has joined #openstack-infra | 12:22 | |
| mordred | infra-root: gonna cut a shade 1.6.0 rlease - any last minute objections? | 12:23 |
| Shrews | mordred: doo eeet | 12:24 |
| yolanda | ok for me | 12:24 |
| *** dguitarbite has quit IRC | 12:24 | |
| *** mrmartin has quit IRC | 12:25 | |
| *** mtanino has joined #openstack-infra | 12:25 | |
| *** zhurong has joined #openstack-infra | 12:26 | |
| mordred | Shrews: tag pushed | 12:27 |
| yolanda | infra-root, i ran manually jjb on all jenkins masters, it was successful on all but still running on jenkins01 | 12:27 |
| mordred | yolanda: cool | 12:28 |
| *** McCorkis has quit IRC | 12:29 | |
| pabelanger | nice work | 12:30 |
| pabelanger | I think we can also do the puppet migration to ubuntu-trusty today too | 12:30 |
| pabelanger | if the image uploads clarkb did last night finished, I think we are good | 12:30 |
| *** mylu has joined #openstack-infra | 12:31 | |
| pabelanger | looks like ovh-bhs1 and rax-ord are still uploading | 12:32 |
| *** gordc has joined #openstack-infra | 12:34 | |
| *** salv-orlando has joined #openstack-infra | 12:34 | |
| *** cdent has joined #openstack-infra | 12:35 | |
| *** claudiub|2 has joined #openstack-infra | 12:36 | |
| *** Daisy has joined #openstack-infra | 12:38 | |
| *** roxanaghe has joined #openstack-infra | 12:38 | |
| yolanda | pabelanger, yes, jhesketh told that ovh one was failing finding an endpoint | 12:38 |
| *** electrofelix has quit IRC | 12:39 | |
| pabelanger | so, I actually think think the image in ovh-bhs1 has cron installed now | 12:39 |
| pabelanger | going to find a slave and just confirm | 12:39 |
| pabelanger | Ya, cron is installed | 12:40 |
| pabelanger | so I think we are safe to migrate puppet jobs | 12:41 |
| yolanda | cool, has the change the needed +2 ? | 12:41 |
| pabelanger | yolanda: AJaeger: Do you have the bandwidth this morning / afternoon to assist? | 12:41 |
| pabelanger | yolanda: not yet | 12:41 |
| dtroyer | sbelous: hey | 12:41 |
| yolanda | i will be working for a bit more, but not after lunch | 12:41 |
| pabelanger | EmilienM: we're thinking of migrating to ubuntu-trusty for puppet jobs this morning. Are you okay with that | 12:41 |
| *** Daisy has quit IRC | 12:42 | |
| *** burgerk has joined #openstack-infra | 12:42 | |
| sbelous | dtroyer: hi, Dean! could you please review that patch: https://review.openstack.org/#/c/286084/ | 12:42 |
| dtroyer | sbelous: I'll add it to the list for after the next meeting | 12:42 |
| *** knikolla has quit IRC | 12:42 | |
| sbelous | dtroyer: cool, thanks! | 12:42 |
| *** amitgandhinz has joined #openstack-infra | 12:42 | |
| *** cbader has quit IRC | 12:43 | |
| *** roxanaghe has quit IRC | 12:43 | |
| pabelanger | infra-root: EmilienM: any objections to migrating puppet jobs to ubuntu-trusty this morning? We look to be ready to land 294203 | 12:43 |
| jhesketh | mordred: yay, new shade :-) | 12:45 |
| HeOS | Hello, infra-team! Could this request be merged into the repository? https://review.openstack.org/#/c/294728/. | 12:46 |
| jhesketh | yolanda, pabelanger: ah, so all the new images are uploaded? | 12:46 |
| jhesketh | did you want me to check on the status | 12:46 |
| openstackgerrit | Yuriy Taraday proposed openstack-infra/project-config: tuning-box: Use db setup for python jobs https://review.openstack.org/297088 | 12:46 |
| yolanda | pabelanger, ovh has not uploaded? | 12:46 |
| *** oanson has quit IRC | 12:47 | |
| *** mrmartin has joined #openstack-infra | 12:47 | |
| EmilienM | pabelanger: go go go | 12:48 |
| pabelanger | yolanda: right, but it did upload on Saturday | 12:48 |
| pabelanger | which gives us the patches we need for the migration | 12:48 |
| yolanda | ah that's ok then | 12:48 |
| mordred | jhesketh: \o/ | 12:49 |
| jhesketh | pabelanger: are you poking at nodepool at the moment or do you want me to check back in with how the latest ubuntu-trusty's have gone? | 12:50 |
| *** mrmartin has quit IRC | 12:51 | |
| pabelanger | jhesketh: I looked and upload was still in progress. But feel free to do some magic if you know another way to make is successful | 12:51 |
| jhesketh | pabelanger: well ovh was giving me an error I didn't have time to debug last, but I'll take another look | 12:52 |
| *** amrith is now known as _amrith_ | 12:52 | |
| pabelanger | Ya | 12:52 |
| pabelanger | latest error I see is: | 12:52 |
| pabelanger | OpenStackCloudException: Image creation failed: Error finding address for https://image.compute.bhs1.cloud.ovh.net/v2/images/f38654fd-1662-426a-8013-7190781fada3/file: Unable to establish connection to https://image.compute.bhs1.cloud.ovh.net/v2/images/f38654fd-1662-426a-8013-7190781fada3/file | 12:52 |
| *** annegentle has joined #openstack-infra | 12:53 | |
| pabelanger | and rax-ord is giving OpenStackCloudTimeout: Timeout waiting for the image to import. | 12:53 |
| jhesketh | pabelanger: yep, just confirmed the same thing | 12:53 |
| mordred | pabelanger: yah. that's the lovely error we get | 12:53 |
| jhesketh | so I can kick the rax-ord one off again if you'd like | 12:53 |
| pabelanger | sure | 12:53 |
| mordred | it is, just to remind everyone, _not_ actually an error finding the address | 12:54 |
| jhesketh | but I don't think re-doing the ovh one will help | 12:54 |
| mordred | it's just a bad error message | 12:54 |
| jhesketh | mordred: is there something to fix it? | 12:54 |
| mordred | well, we're not sure what the actual problem is | 12:54 |
| * jhesketh kicks off the rax-ord one again | 12:54 | |
| mordred | but we do know it's not name resolution - there's a copy-pasta error message in glanceclient | 12:55 |
| *** akshai has joined #openstack-infra | 12:55 | |
| * mordred glares at flaper87 | 12:55 | |
| jhesketh | mordred: right.. is it worth retrying or waiting for somebody to have a look at it | 12:55 |
| * flaper87 looks innocent | 12:55 | |
| flaper87 | mordred: what did I do this time? | 12:55 |
| SamYaple | w00t new shade | 12:55 |
| mordred | flaper87: glance | 12:55 |
| flaper87 | iuuuhg there's a spaghetti error in glanceclient ? | 12:55 |
| mordred | flaper87: well, specifically, Image creation failed: Error finding address for ... is not actually an error finding address | 12:56 |
| *** mylu has quit IRC | 12:56 | |
| *** amitgandhinz has quit IRC | 12:56 | |
| vsaienko | infra cores, please review/merge https://review.openstack.org/#/c/296963/ and https://review.openstack.org/#/c/296996/ it is one line fix that do not remove sudo right for stack user for networking-generic-switch and ironic-multitenant-network jobs | 12:56 |
| mordred | flaper87: in glanceclient/common/http.py - there are three different errors with that error message string | 12:56 |
| mordred | flaper87: requests.exceptions.ConnectionError, socket.gaierror and ksc_exc.ConnectionRefused | 12:57 |
| flaper87 | mordred: mmh... I'll file a bug! FWIW, I hate that file :D | 12:57 |
| *** esikachev has joined #openstack-infra | 12:57 | |
| vsaienko | yolanda: thank you sir! | 12:58 |
| mordred | flaper87: I shall hand you a box of baby goats | 12:58 |
| mordred | flaper87: :) | 12:58 |
| pabelanger | mordred: SamYaple: nice, now to update shade RPM | 12:58 |
| mordred | jhesketh: I think that's the error we keep getting - lemme poke at something real quick though | 12:59 |
| jhesketh | okay | 12:59 |
| flaper87 | mordred: hahahaha, I'll take it! Mini pigs work as well. I have many pranks in mind that could use a mini pig | 12:59 |
| *** yamamoto has quit IRC | 13:00 | |
| kzaitsev_ws | hi all, I have a question about translation bot's commits and RC2. In murano-dashboard it seems that bot only commits to master — I was wondering if I should cherry-pick translations to stable/mitaka by hand. Can anyone shed some light for me on that? =) | 13:00 |
| *** oanson has joined #openstack-infra | 13:00 | |
| openstackgerrit | Tom Cocozzello proposed openstack-infra/project-config: Experimental patch_cover gate job https://review.openstack.org/296808 | 13:00 |
| vsaienko | pabelanger: thank you sir! | 13:01 |
| *** piet has joined #openstack-infra | 13:01 | |
| pabelanger | vsaienko: note, yolanda is not a sir | 13:02 |
| pabelanger | vsaienko: also, np | 13:02 |
| yolanda | miss :) | 13:02 |
| vsaienko | yolanda: I'm really sorry, thank you miss :-[ | 13:03 |
| yolanda | no problem, used to that ! | 13:03 |
| kzaitsev_ws | oh, wait. looks like I'm wrong | 13:03 |
| *** rossella_s has quit IRC | 13:03 | |
| kzaitsev_ws | nevermind ) | 13:03 |
| *** amitgandhinz has joined #openstack-infra | 13:04 | |
| *** Guest63 has joined #openstack-infra | 13:04 | |
| *** amitgandhinz has quit IRC | 13:04 | |
| *** rossella_s has joined #openstack-infra | 13:04 | |
| *** amitgandhinz has joined #openstack-infra | 13:04 | |
| *** Guest63 has quit IRC | 13:06 | |
| *** eharney has quit IRC | 13:06 | |
| *** signed8bit has joined #openstack-infra | 13:06 | |
| mordred | jhesketh: I'm attempting to upload the file contents to that image object by hand | 13:08 |
| *** jpr has quit IRC | 13:08 | |
| mordred | jhesketh: becuase I want to see either the error or learn how retries work (I have a hunch) | 13:08 |
| jhesketh | mordred: okay, anything I can do to help? | 13:08 |
| mordred | jhesketh: nah. except remind me in the past to have started the command I'm running in screen | 13:08 |
| mordred | hah! that's a better error | 13:09 |
| openstackgerrit | Merged openstack-infra/project-config: fuel-web: use db setup for python jobs https://review.openstack.org/296989 | 13:09 |
| mordred | HTTP/1.1 500 Internal Server Error | 13:09 |
| * jhesketh logs into eavesdrop to modify the irc logs | 13:09 | |
| mordred | so I'm going to guess load balancer error or something similar at ovh | 13:09 |
| openstackgerrit | Merged openstack-infra/project-config: net-bgpvpn : fix access to rc files https://review.openstack.org/296812 | 13:10 |
| openstackgerrit | Thomas Herve proposed openstack/requirements: Bump zaqarclient and barbicanclient versions https://review.openstack.org/297103 | 13:11 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs https://review.openstack.org/289445 | 13:12 |
| openstackgerrit | Merged openstack-infra/project-config: Left sudo rights for stack in ironic-multitenant-network job https://review.openstack.org/296963 | 13:12 |
| *** yamamoto has joined #openstack-infra | 13:18 | |
| *** cbader has joined #openstack-infra | 13:18 | |
| *** jsavak has joined #openstack-infra | 13:19 | |
| *** samuelBartel has joined #openstack-infra | 13:19 | |
| jhesketh | mordred: well I'm going to head off.. rax-ord is uploading from clarkb's screen still (it keeps timing out, but the others eventually got there) | 13:20 |
| jhesketh | (screen #3 if needed) | 13:20 |
| *** zz_dimtruck is now known as dimtruck | 13:20 | |
| AJaeger | pabelanger: I have five mins or so now - and then I'll probably gone again, so can't be of much help today | 13:21 |
| Sam-I-Am | sdague: moo? | 13:21 |
| pabelanger | AJaeger: sure. np. I have most of the day to dedicate to it | 13:21 |
| *** yamamoto has quit IRC | 13:22 | |
| *** _ari_ has joined #openstack-infra | 13:23 | |
| *** zeih has joined #openstack-infra | 13:24 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use the mirror server for the centos image and git repos https://review.openstack.org/285257 | 13:26 |
| *** eharney has joined #openstack-infra | 13:27 | |
| fungi | smarcet: do you think we could wrap the new and old logic in a conditional so it can be safely tested on openstackid-dev rather than in production, and just use one variable passed in from system-config to switch behavior? | 13:27 |
| *** links has quit IRC | 13:28 | |
| smarcet | fungi: i will try, to do that :) | 13:28 |
| sdague | Sam-I-Am: what's up? | 13:29 |
| Sam-I-Am | sdague: the documentation for use_neutron is vague | 13:29 |
| *** annegentle has quit IRC | 13:29 | |
| Sam-I-Am | sdague: it says it only replaces network_api_class, but in the patches to the install guide, it also replaces security_group_api ? | 13:30 |
| sdague | Sam-I-Am: ok, vague how? | 13:30 |
| Sam-I-Am | looking at the rel notes | 13:30 |
| sdague | it does replace both of them, though it happened in 2 patches | 13:31 |
| sdague | so maybe we have to update the first reno | 13:31 |
| *** ddieterly has joined #openstack-infra | 13:31 | |
| sdague | where it is introduced | 13:31 |
| Sam-I-Am | there's two notes, but in separate places | 13:31 |
| sdague | ok | 13:31 |
| Sam-I-Am | the one for network_api_class is only in upgrade nodes | 13:31 |
| Sam-I-Am | the other one is in deprecation_notes | 13:31 |
| *** tonytan4ever has joined #openstack-infra | 13:32 | |
| sdague | ok, right a bunch of class= conf variables were deprecated in nova before release | 13:32 |
| Sam-I-Am | both of them have an upgrade/deprecation impact | 13:32 |
| sdague | that is probably true | 13:32 |
| Sam-I-Am | this also happened after M3, which has made installation guide testing difficult because packagers usually release M3 packages then wait until release for the next round. | 13:33 |
| sdague | Sam-I-Am: ok, so how would you propose fixing it? | 13:33 |
| *** krtaylor has quit IRC | 13:33 | |
| Sam-I-Am | is there a way to fix release notes? | 13:33 |
| sdague | yes, they can just be updated | 13:33 |
| Sam-I-Am | since both use_neutron impacts seem to occur in rc1, i suggest grouping them together under one or both headers | 13:34 |
| Sam-I-Am | right now the impacts are split between upgrade and deprecation headers, which means the notes were probably just written differently by whomever wrote the patches | 13:35 |
| sdague | Sam-I-Am: well, it was more about how they looked in individual patch series, but I think I see the concern | 13:36 |
| sdague | let me propose something | 13:36 |
| Sam-I-Am | thanks | 13:36 |
| Sam-I-Am | relnotes have been a learning process | 13:36 |
| mordred | Sam-I-Am: out of curiosity - what's "use_neutron" ? | 13:37 |
| Sam-I-Am | does it go here... or here... or maybe here | 13:37 |
| Sam-I-Am | mordred: a new option that replaces the network_api_class and security_group_api options in nova.conf | 13:37 |
| mordred | Sam-I-Am: ah - so it's like a newer simpler way to just say "dude, I'm just using neutron, do all the things" yeah? | 13:38 |
| Sam-I-Am | yep | 13:38 |
| mordred | cool. I approve of such things | 13:38 |
| Sam-I-Am | its a good thing, just came at a bad time for the install guide updates | 13:38 |
| * mordred hands sdague a box of baby goats | 13:38 | |
| *** claudiub|2 has quit IRC | 13:39 | |
| Sam-I-Am | none of the packages available now can use it, so neutron breaks for people testing the guide | 13:39 |
| sdague | Sam-I-Am: yeh, I guess maybe I got agressive in pushing the manuals patch | 13:39 |
| sdague | I didn't realize the implications of the m3 cutoff for install guide testing | 13:39 |
| *** esikachev has quit IRC | 13:40 | |
| Sam-I-Am | sdague: well, i +2'd it, but i also didnt realize it wasnt in M3 | 13:41 |
| *** Daisy has joined #openstack-infra | 13:41 | |
| *** jsavak has quit IRC | 13:41 | |
| *** mylu has joined #openstack-infra | 13:41 | |
| Sam-I-Am | i sort of assume all large-ish/impactful config changes occur prior to RCs | 13:41 |
| sdague | Sam-I-Am: honestly, I didn't realize that either, we had a bunch of deprecation bits up and I guess we weren't paying attention to landing timing relative to when M3 actually cut | 13:41 |
| Sam-I-Am | but maybe i cant assume that either | 13:41 |
| sdague | use_neutron was inventing during the options deprecation, it wasn't in the first path | 13:42 |
| sdague | it's kind of an anomoly\ | 13:42 |
| *** jsavak has joined #openstack-infra | 13:42 | |
| Sam-I-Am | huh, ok. | 13:44 |
| *** dimtruck is now known as zz_dimtruck | 13:44 | |
| *** tonytan4ever has quit IRC | 13:44 | |
| *** krtaylor has joined #openstack-infra | 13:45 | |
| *** esikachev has joined #openstack-infra | 13:45 | |
| *** jpr has joined #openstack-infra | 13:46 | |
| *** tonytan4ever has joined #openstack-infra | 13:47 | |
| openstackgerrit | Graham Hayes proposed openstack-infra/project-config: Create 'Review-Priority' for designate repos https://review.openstack.org/295253 | 13:47 |
| pabelanger | fungi: once your back, if you want to help land 294203, I'm going to monitor that today | 13:47 |
| sdague | Sam-I-Am: https://review.openstack.org/297129 | 13:48 |
| sdague | how about that? | 13:48 |
| sdague | mordred / fungi / et al. - https://review.openstack.org/#/c/297004/ devstack-gate change to cut over for stable mitaka testing | 13:49 |
| sdague | in grenade | 13:49 |
| Sam-I-Am | sdague: should there be a deprecation notice for security_group_api too? | 13:50 |
| Sam-I-Am | or is that still sticking around for nova-net purpose | 13:50 |
| *** _skolekonov is now known as skolekonov | 13:50 | |
| sdague | Sam-I-Am: it's in there | 13:50 |
| *** achanda has joined #openstack-infra | 13:50 | |
| *** achanda has quit IRC | 13:50 | |
| sdague | http://docs.openstack.org/releasenotes/nova/unreleased.html | 13:50 |
| sdague | 4th bullet | 13:50 |
| *** yaume has quit IRC | 13:51 | |
| *** ddieterly is now known as ddieterly[away] | 13:51 | |
| Sam-I-Am | ok, your patch looks mostly good minus a word issue | 13:51 |
| *** daemontool has joined #openstack-infra | 13:52 | |
| *** mylu has quit IRC | 13:52 | |
| fungi | pabelanger: looks like it's changing job names for infra, tripleo, puppet and fuel teams. are they aware? | 13:53 |
| fungi | pabelanger: also how many jobs is it changing? will we need to babysit manual jjb updates on all the masters to avoid hitting timeouts? | 13:53 |
| sdague | Sam-I-Am: ok, just fixed that | 13:53 |
| pabelanger | fungi: Ya, EmilienM is onboard with the change. And is active in a 3 project jobs I believe | 13:54 |
| sdague | once you are good with it, I'll beat the bushes with the nova core team to get it in | 13:54 |
| pabelanger | fungi: Ya, I'll be babysitting masters to confirm things work well | 13:54 |
| *** YorikSar has joined #openstack-infra | 13:54 | |
| EmilienM | hello I'm on board | 13:54 |
| *** yaume has joined #openstack-infra | 13:54 | |
| Sam-I-Am | sdague: thx | 13:54 |
| *** annegentle has joined #openstack-infra | 13:56 | |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: Update wanted_ci_job to reflect current status of things https://review.openstack.org/294071 | 13:56 |
| YorikSar | Can someone please take a look at another DB-jobs change: https://review.openstack.org/297088 | 13:57 |
| fungi | sdague: wow, we still have icehouse and juno in there | 13:57 |
| sdague | fungi: we do | 13:57 |
| sdague | we can clean those up later | 13:57 |
| *** jaypipes has joined #openstack-infra | 13:57 | |
| fungi | sdague: yeah, not worried about it, more just surprised nobody's deleted those already | 13:58 |
| sdague | yep | 13:58 |
| jroll | hey folks, what's the best way to test a post job (we have a broken one with a potential fix) | 13:58 |
| sdague | who knows, maybe someone's even using it :) | 13:58 |
| rcarrillocruz | jroll: to my knowledge, an infra admin logging to the jenkins master and rebuilding the job | 13:58 |
| rcarrillocruz | i don't think non-admins can test post on-demand easily | 13:59 |
| *** ddieterly[away] is now known as ddieterly | 13:59 | |
| jroll | rcarrillocruz: :/ thanks | 13:59 |
| rcarrillocruz | yolanda: ^ | 13:59 |
| fungi | sdague: entirely possible, but we don't test it so grenade is entirely likely to break for those branches (if it hasn't already) | 13:59 |
| rcarrillocruz | or fungi (hey, you're here) | 13:59 |
| fungi | devstack and devstack-gate too | 13:59 |
| *** annegentle has quit IRC | 13:59 | |
| YorikSar | fungi: Thanks! | 14:00 |
| fungi | jroll: depends on how idempotent the job is (whether it's safe to rerun) | 14:00 |
| *** Daisy has quit IRC | 14:00 | |
| *** mylu has joined #openstack-infra | 14:00 | |
| jroll | fungi: it is | 14:00 |
| jroll | well, it's safe, not idempotent | 14:00 |
| fungi | jroll: what's the job? some projects run similar versions of their post jobs in experimental so they can trigger them on demand (as long as they don't upload durable artifacts anywhere important) | 14:01 |
| *** annegentle has joined #openstack-infra | 14:01 | |
| jroll | fungi: it's the ironic-python-agent build-and-upload-a-ramdisk | 14:01 |
| * jroll finds | 14:01 | |
| fungi | aha, so not something you'd want to test from a non-merged change unless you neutered it to not do the real upload at the end | 14:02 |
| jroll | https://github.com/openstack-infra/project-config/blob/ed98b617367efa2e07ddfbc36281b539a6791a67/jenkins/jobs/ironic-python-agent-jobs.yaml#L17 | 14:02 |
| jroll | right | 14:02 |
| fungi | has the fix for it already merged? | 14:02 |
| *** IlyaG has joined #openstack-infra | 14:02 | |
| jroll | no, we'd like to test it | 14:02 |
| openstackgerrit | Merged openstack-infra/project-config: Left sudo rights for networking-generic-switch job https://review.openstack.org/296996 | 14:03 |
| anteaya | matrohon: well I don't know as I agreed so much as looked at it and discussed it with clarkb | 14:03 |
| anteaya | matrohon: actually I was going to look today to read what clarkb wrote as I still don't know | 14:03 |
| *** mtanino has quit IRC | 14:03 | |
| fungi | jroll: oh, that will probably require a fair amount of manual work if the job needs to be rerun with unmerged changes | 14:03 |
| fungi | jroll: what's the proposed fix? | 14:03 |
| anteaya | matrohon: sorry if I confused you in making comments that you interpreted as agreement | 14:03 |
| jroll | fungi: yeah, if we have to merge things and see what happens, that's okay, I don't want to eat a bunch of your time | 14:04 |
| jroll | I also just noticed the potential fix is dependent on another https://review.openstack.org/#/c/297079/ | 14:05 |
| fungi | jroll: depending on where in the change is in the job or the project on which the job is running, the amount of manual work can vary | 14:05 |
| jroll | so I'm going to put up a different theory | 14:05 |
| jroll | yeah, it's in the project | 14:05 |
| matrohon | anteaya, np, tehre were two patches, one on which you both we agreeing (https://review.openstack.org/#/c/296812/) and a second one which deserves more reading | 14:06 |
| matrohon | anteaya, at least that's what I understood :) | 14:06 |
| fungi | jroll: in theory _most_ of the guts of the job should be testable locally. it's just shell scripts | 14:06 |
| jroll | fungi: well, I believe it's just a missing dep in the images used in the gate, that is there on normal ubuntu | 14:07 |
| jroll | are those images available for download/use? | 14:07 |
| *** IlyaG has quit IRC | 14:07 | |
| *** _amrith_ is now known as amrith | 14:07 | |
| fungi | jroll: we can make them available, though they don't accept generic logins so you'd have to inspect them mounted on a loopback or something (also they're large, around 6gb depending on the distro) | 14:08 |
| *** Daisy has joined #openstack-infra | 14:08 | |
| fungi | jroll: what specific dependencies do you think are missing? easy for me to check | 14:08 |
| *** tojuvone has quit IRC | 14:08 | |
| *** tojuvone has joined #openstack-infra | 14:08 | |
| jroll | fungi: meh. I think we're missing apparmor, we've already fixed this in our devstack plugin once https://review.openstack.org/#/c/294343/ | 14:09 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:09 | |
| fungi | jroll: aha, yes apparmor was preinstalled by rackspace on their ubuntu images i believe, but now that we're building our own images so that we can run those jobs in more providers than just rackspace, apparmor is not in them by default | 14:10 |
| jroll | fungi: right, so I just uploaded https://review.openstack.org/297177 , will just land that and see what happens | 14:10 |
| *** woodster_ has joined #openstack-infra | 14:11 | |
| fungi | fun times | 14:11 |
| *** asettle has joined #openstack-infra | 14:12 | |
| *** yamamoto has joined #openstack-infra | 14:12 | |
| fungi | i believe i remember the discussion for the devstack plugin too, yeah | 14:12 |
| jroll | yeah, between backporting this to devstack stable branches and now this, it's biting us hard heh | 14:12 |
| jroll | fungi: is there any way to get notifications for failed post jobs? | 14:12 |
| sdague | fungi: app-armor has been a default in ubuntu server for ever | 14:13 |
| anteaya | matrohon: oh clarkb never got a chance to review 296812 | 14:13 |
| sdague | is there as reason to not have it in the dib image? | 14:13 |
| anteaya | matrohon: well I was reading it and confirming the contents, but had wanted to hear clarkb's thoughts | 14:13 |
| anteaya | matrohon: had I agreed, I would have +2'd it | 14:13 |
| anteaya | was actually hoping to see his review on those | 14:14 |
| *** zz_dimtruck is now known as dimtruck | 14:14 | |
| smarcet | fungi: we are discussing with jpmaxman and we think that it would be better to use nginx | 14:14 |
| matrohon | anteaya, ok sorry for the misunderstanding | 14:14 |
| anteaya | no worries, sorry I wasn't clear | 14:15 |
| *** claudiub|2 has joined #openstack-infra | 14:15 | |
| smarcet | fungi: i used this puppet module puppet module install jfryman-nginx on the past, its ok if i use it for this update | 14:15 |
| smarcet | ? | 14:16 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Revert "Move puppet jobs to ubuntu-trusty"" https://review.openstack.org/294203 | 14:16 |
| pabelanger | EmilienM: ^ just landed | 14:16 |
| EmilienM | pabelanger: ack | 14:16 |
| pabelanger | EmilienM: should start running in 15mins | 14:16 |
| *** asettle has quit IRC | 14:16 | |
| EmilienM | pabelanger: I'll run recheck on the https://review.openstack.org/#/q/status:open+branch:master+topic:nodesets/trusty serie | 14:16 |
| fungi | smarcet: jpmaxman: i respect your desire to set up a high-performing web application for whatever phone-thingy it is you're writing. we wanted a simple and durable authentication server which runs in the same way we run our other services. supporting php (especially odd choice for a crypto-heavy application) is already a challenge, and i really don't think we have time to introduce an entirely | 14:18 |
| fungi | separate web server from the one we already support. "use another webserver" is usually a knee-jerk reaction from people who don't know how to tune the one they have which is why we've pushed back on it in the past | 14:18 |
| signed8bit | Anyone on that would be willing to suffer through some questions about how Gerrit submissions are setup for review.openstack.org? | 14:18 |
| *** MarkAtwood has joined #openstack-infra | 14:19 | |
| *** cdent has left #openstack-infra | 14:20 | |
| anteaya | signed8bit: ask away | 14:20 |
| anteaya | signed8bit: are you looking to submit to our gerrit at review.openstack.org? | 14:21 |
| anteaya | signed8bit: or are you trying to set up your own gerrit and are curious about our config? | 14:21 |
| sbelous | dtroyer: thank you! | 14:21 |
| signed8bit | Thanks anteaya, no more nuances of how your integration with zuul is setup | 14:21 |
| signed8bit | anteaya: we are looking to block immediate merge in our setup when a chance is submitted to gerrit and wondered how upstream was setup for that. Our world is gerrit -> jenkins though and no zuul | 14:22 |
| signed8bit | s/chance/change/ | 14:22 |
| *** rbrndt_ has joined #openstack-infra | 14:23 | |
| jpmaxman | nicely put fungi | 14:23 |
| *** knikolla has joined #openstack-infra | 14:23 | |
| jpmaxman | maybe I should explain a bit more | 14:23 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: fix stable acls for keystoneauth repository https://review.openstack.org/297182 | 14:24 |
| signed8bit | I played around which writing a pre-merge validator plugin in Gerrit which gets us closer but exposes other challenges in how that interacts with Jenkins (events, polling state etc.) | 14:24 |
| jpmaxman | OpenStack ID is part of a greater eco system of web applications that power the greater Openstack experience for users | 14:24 |
| jpmaxman | right now the other parts of that use nginx | 14:24 |
| fungi | jpmaxman: makes sense. if the design of openstackid is such that it really isn't going to perform well when run under apache, this is the point at which we should be planning to implement something else for our community systems and let openstackid just be for the marketing website | 14:24 |
| jeblair | signed8bit: there's a lot you can do with gerrit->jenkins that you don't need zuul for, but gating is very difficult, and that's the reason we wrote zuul. | 14:24 |
| jpmaxman | so the thought is to be able to get greater economies of scale | 14:25 |
| jpmaxman | by optimizing across the ecosystem of apps | 14:25 |
| anteaya | signed8bit: okay so you have your own infra set up? | 14:25 |
| signed8bit | yep jeblair for sure and we want to move to zuul but are trying to patch together something before we have time for that | 14:25 |
| jpmaxman | I think apache can be tuned to power openstack Id as well as nginx | 14:25 |
| jpmaxman | they are both web servers with their pluses and minuses on each side | 14:25 |
| anteaya | ah jeblair has joined in, he is much more knowledgeable than myself | 14:25 |
| jpmaxman | but in the name of efficiency we would like the option to move toward a single stack | 14:26 |
| jpmaxman | in the same way that you desire something similar from a systems perspective | 14:26 |
| signed8bit | yes anteaya, our pipelines are fairly long and deal with generating containers. a commit that creates a new container may take some time to complete and in the meantime a new commit will fail test if an expected container artifact isn't yet available so like jeblair mentions, we are stuck trying to solve for gating in a gerrit -> jenkins flow. | 14:27 |
| mordred | jpmaxman, fungi: it sounds like we have one of those problems where there are two different ecosystems each with their own set of needs as it relates to being internally consistent | 14:27 |
| fungi | jpmaxman: mordred: yep, my main concern for now is the disruption it's likely to cause if we scramble to switch openstackid.org from apache to nginx in the few weeks we have before the summit | 14:28 |
| jpmaxman | in an ideal world thinking long term you'd have ideal configurations for both apache and nginx so people could roll with whatever they want | 14:28 |
| mordred | I think we have different ideal worlds :) | 14:29 |
| jpmaxman | but thinking specifically of the needs of the foundation in this case and their specific use it would make sense to focus on one set of configs. | 14:29 |
| signed8bit | jeblair: Are you all doing something special in Gerrit to prevent immediate merge when a change is submitted? | 14:29 |
| *** cbader_ has joined #openstack-infra | 14:29 | |
| *** cbader has quit IRC | 14:29 | |
| jpmaxman | mordred: referring to a world where other people might want to adopt the apps for their own use | 14:30 |
| jpmaxman | mordred: a world far off I imagine ;) | 14:30 |
| jpmaxman | fungi: it is not necessarily my intention to switch anything in production between now and the summit | 14:31 |
| jeblair | signed8bit: i'm not sure what you mean by 'immediate merge'... | 14:31 |
| jpmaxman | BUT what we have needs to be optimized | 14:31 |
| *** mrmartin has joined #openstack-infra | 14:31 | |
| jpmaxman | instead of spinning the cycles to optimize Apache I'd prefer to take this out of production and optimize in dev | 14:32 |
| mordred | signed8bit: we don't let users use the submit button in our gerrit - we trigger zuul actions with a special vote category | 14:32 |
| jpmaxman | and our preference is to use nginx to do those optimizations | 14:32 |
| mordred | signed8bit: and then zuul hits the submit button when it's ready for the patch to merge | 14:32 |
| jpmaxman | our goal for production between now and the summit is stability | 14:32 |
| signed8bit | mordred jeblair: yes that was what I was after, thanks | 14:32 |
| jpmaxman | we could also use a "test" server for this and leave dev as is | 14:33 |
| signed8bit | So basically via access rules only zuul can be the submitter | 14:33 |
| mordred | signed8bit: zuul does a great job of handling making git refernces across dependencies to solve the problem you refer to above for git states - but there is no currently existing work to do the same with potential container states | 14:33 |
| *** EricGonczer_ has joined #openstack-infra | 14:34 | |
| *** Daisy has quit IRC | 14:34 | |
| mordred | in theory the logic could be reasonably similar, but work would definitely need to be done ... likely involving containers made for potential states going into a localized registry that a refernce to is passed to jobs, so that jobs that depend on a container that's tied to a not-yet-landed patch can have access to the artifact without actually publishing it | 14:35 |
| *** Daisy has joined #openstack-infra | 14:35 | |
| signed8bit | mordred: Got you. Though in our case I believe what zuul does would work since all we really want to do is prevent submission of a change until the "merge" pipeline has completed successfully. The pipeline deals with tracking and setting container state though some different tooling. | 14:35 |
| *** esker has joined #openstack-infra | 14:36 | |
| mordred | cool | 14:36 |
| *** esker has quit IRC | 14:36 | |
| *** jaosorior has quit IRC | 14:37 | |
| *** sdake has joined #openstack-infra | 14:37 | |
| *** esker has joined #openstack-infra | 14:37 | |
| *** HeOS has quit IRC | 14:37 | |
| signed8bit | mordred: Though now we are stuck with how do we do what zuul does without zuul :-P | 14:37 |
| mordred | signed8bit: well, youknow, don't try :) | 14:38 |
| signed8bit | mordred: that would be my preference but might need a band-aid until we can properly move to zuul | 14:38 |
| signed8bit | This is why I was playing around with Gerrit's pre-merge validator listener in a plugin. | 14:39 |
| *** Daisy has quit IRC | 14:39 | |
| *** abe_music has joined #openstack-infra | 14:41 | |
| *** kelepirci_ has left #openstack-infra | 14:41 | |
| clarkb | you can just require +1 in verified before submission | 14:41 |
| clarkb | then dont give humans +1 verified vote perms | 14:42 |
| mordred | clarkb: good call | 14:42 |
| *** burgerk has quit IRC | 14:42 | |
| jpmaxman | fungi mordred we have to be able to grow and scale openstackid both for the community and for the openstack marketing sites / apps. We've taken a great first step here, but there is a long way to go. | 14:43 |
| *** Nikolay_St has quit IRC | 14:43 | |
| fungi | jpmaxman: smarcet: i saw the mpm-prefork tuning merged earlier today. did you get any better numbers out of load testing with that in place? | 14:44 |
| smarcet | fungi: yes i have | 14:45 |
| signed8bit | yeah, we have considered that approach as well but still want to verify for patchet-created vs. merge... maybe a different label? The issue still subsequent commits will depend on artifacts that are generated in a merge context. | 14:45 |
| smarcet | but take in consideration that its create a child process per request | 14:45 |
| smarcet | and roof was 750 simultaneous connections | 14:45 |
| smarcet | and heavy ram load on server | 14:45 |
| smarcet | quite it became unresponsive | 14:45 |
| * fungi checks graphs | 14:45 | |
| clarkb | etherpad does way more than that | 14:48 |
| clarkb | and its something like 128 connections per process | 14:48 |
| smarcet | clarkb : i think that etherpard works on another process model | 14:48 |
| clarkb | maybe you shpuld use that one then? | 14:48 |
| smarcet | mpm worked, openstackid use mpm fork | 14:49 |
| smarcet | clarkb : i guess worth to try | 14:49 |
| *** jvrbanac_ is now known as jvrbanac | 14:49 | |
| clarkb | (it was my original suggestion since we have had to sort out etherpads high reconnect volume for summits) | 14:49 |
| *** mrmartin has quit IRC | 14:49 | |
| *** electrofelix has joined #openstack-infra | 14:49 | |
| *** electrofelix has quit IRC | 14:50 | |
| EmilienM | pabelanger: trying now, with 295819 (in the queue right now) | 14:51 |
| smarcet | clarkb: i was looking at etherpad puppet, its using also httpd puppet module, but is not specifying on any place the mpm mode of apache | 14:52 |
| pabelanger | EmilienM: might, fail. JJB still applying | 14:52 |
| smarcet | clarkb: do u know where is set that ? | 14:52 |
| fungi | smarcet: clarkb: yeah, prefork is going to be about the least efficient mpm implementation with apache | 14:52 |
| EmilienM | pabelanger: jobs did not start yet | 14:52 |
| smarcet | clarkb: also looking at httpd puppet module, and no seeing any option to set up the mpm | 14:52 |
| EmilienM | pabelanger: where do you monitor that? | 14:52 |
| pabelanger | EmilienM: I am on our puppetmaster currently, but you'll be able to see the results in puppetdb.o.o | 14:53 |
| clarkb | http://git.openstack.org/cgit/openstack-infra/puppet-etherpad_lite/tree/files/apache-connection-tuning | 14:53 |
| openstackgerrit | Tom Cocozzello proposed openstack-infra/project-config: Experimental patch_cover gate job https://review.openstack.org/296808 | 14:54 |
| *** piet has quit IRC | 14:54 | |
| EmilienM | pabelanger: it would be cool to have custom facts on http://puppetdb.openstack.org/node/jenkins01.openstack.org | 14:54 |
| EmilienM | that say the version of JJB (using SHA1 from project-config) | 14:54 |
| clarkb | and its written to disk in apache.pp | 14:54 |
| EmilienM | so we could very easily check which version of project-config we're running on nodes | 14:55 |
| EmilienM | same applies for system-config, etc | 14:55 |
| fungi | clarkb: right, smarcet already implemented similar tuning for mpm-prefork in https://review.openstack.org/296708 | 14:55 |
| clarkb | right but why prefork? | 14:56 |
| fungi | i think probably because that's what it had originally been deployed with | 14:56 |
| *** annegentle has quit IRC | 14:57 | |
| clarkb | its sort of unfair to apache to pick the worst and undefauly option, declare it bad which you already know, then use nginx | 14:57 |
| fungi | i agree a different mpm would likely work better, though also worth taking into account whether mod_php vs php-fpm will have performance differences | 14:57 |
| clarkb | default is event iirc | 14:57 |
| EmilienM | pabelanger: what do you think? | 14:58 |
| *** harlowja_at_home has joined #openstack-infra | 14:58 | |
| pabelanger | EmilienM: not sure what is involved for that. But if people find it useful, sure | 14:58 |
| pabelanger | EmilienM: okay, here come the NOT_REGISTERED puppet jobs | 14:59 |
| *** annegentle has joined #openstack-infra | 14:59 | |
| pabelanger | EmilienM: so, we are mid process right now | 14:59 |
| EmilienM | pabelanger: imagine, you go on http://puppetdb.openstack.org/node/jenkins01.openstack.org and you can see a fact that tells you which SHA1 is set project-config, would not that be cool? | 14:59 |
| *** yamahata has joined #openstack-infra | 15:00 | |
| sdague | clarkb: can I get a +A on https://review.openstack.org/#/c/297004/ | 15:00 |
| *** piet has joined #openstack-infra | 15:00 | |
| sdague | that will complete the grenade transition for mitaka | 15:00 |
| pabelanger | EmilienM: ya, I usually drop into the report page. But that just shows the latest version. I remember sometime in the paste, your would see the sha1 output | 15:01 |
| *** regXboi has joined #openstack-infra | 15:01 | |
| EmilienM | pabelanger: on the puppetdb reports, I see 10:08 AM -- what TZ is it? | 15:01 |
| pabelanger | EmilienM: EST I believe | 15:02 |
| *** yamahata has quit IRC | 15:02 | |
| *** links has joined #openstack-infra | 15:02 | |
| *** yamahata has joined #openstack-infra | 15:03 | |
| *** esikachev has quit IRC | 15:03 | |
| clarkb | sdague: done | 15:03 |
| pabelanger | will likely have to run JJB manually on a few masters, just waiting for puppet to finish up | 15:03 |
| clarkb | EmilienM: pabelanger its the timezone reported by your browser | 15:03 |
| EmilienM | clarkb: ok, thx | 15:04 |
| *** zhurong has quit IRC | 15:05 | |
| *** jsavak has quit IRC | 15:05 | |
| pabelanger | mordred: when you were giving your talk at midcycle about OVN. I remember you mentioning something about python-neutron client spamming rabbitmq, and I correct it remembering that? | 15:05 |
| pabelanger | s/and/am | 15:05 |
| mordred | uhm | 15:05 |
| mordred | ah - no, not quite | 15:06 |
| fungi | smarcet: jpmaxman: so with the current tuning, it looks like the load test at peak consumed all 15gb available ram plus 4gb swap memory, about 60% of cpu was consumed by iowait, presumably due to the sudden and constant paging activity http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=136 | 15:06 |
| sdague | clarkb: thanks | 15:06 |
| mordred | pabelanger: it's the neutron ml2 plugin agents | 15:06 |
| *** tosky_ has joined #openstack-infra | 15:06 | |
| mordred | pabelanger: they spam rabbit, and you get an agent fr each compute node, so the amount of spam increases per compute node - even if you have a completely idle neutron | 15:07 |
| smarcet | fungi: clarkb: yeah, i think that we should try at least to change to mpm worker + php-fpm | 15:07 |
| mordred | pabelanger: so your scalability is impacted even if you are not using any private networks or virtual routers | 15:07 |
| pabelanger | mordred: Ah, right. | 15:07 |
| fungi | at 750 connections that implies we were seeing about 25mb ram consumed per connection? | 15:07 |
| *** aloga_ has joined #openstack-infra | 15:08 | |
| *** aviau_ has joined #openstack-infra | 15:08 | |
| smarcet | well , first answers would said yes, but we should take statistics on server | 15:08 |
| EmilienM | clarkb: does infra have custom puppet facts already? | 15:08 |
| clarkb | EmilienM: we have had at least one not sure if it is still around | 15:08 |
| fungi | smarcet: yes, the other thing we did when tuning etherpad.openstack.org was to add the stats module and collect thread/process data from apache | 15:09 |
| EmilienM | clarkb: I've been thinking to had custom facts that could give project-config hash on jenkins nodes reports for example | 15:09 |
| EmilienM | clarkb: so just by going on http://puppetdb.openstack.org/node/jenkins01.openstack.org you would not which project-config the node is running | 15:09 |
| fungi | smarcet: one of our root admins can collect that when another load test is underway (preferably one which is large but not large enough to disrupt service) | 15:10 |
| *** Qiming has quit IRC | 15:10 | |
| *** mylu has quit IRC | 15:11 | |
| *** ruhe2 has joined #openstack-infra | 15:11 | |
| *** lucasagomes_ has joined #openstack-infra | 15:11 | |
| *** Shalmazar has joined #openstack-infra | 15:11 | |
| *** dstufft_ has joined #openstack-infra | 15:12 | |
| pabelanger | okay, starting the manual runs of JJB on jenkins.o.o | 15:12 |
| *** hughsaunders_ has joined #openstack-infra | 15:12 | |
| *** ddieterly is now known as ddieterly[away] | 15:12 | |
| *** mrunge_ has joined #openstack-infra | 15:12 | |
| *** adreznec_ has joined #openstack-infra | 15:12 | |
| *** yamamoto has quit IRC | 15:13 | |
| *** yaume has quit IRC | 15:13 | |
| *** edmondsw has quit IRC | 15:13 | |
| *** baoli has quit IRC | 15:13 | |
| *** flepied has quit IRC | 15:13 | |
| *** lifeless has quit IRC | 15:13 | |
| *** tosky has quit IRC | 15:13 | |
| *** david-lyle has quit IRC | 15:13 | |
| *** therve has quit IRC | 15:13 | |
| *** fmccrthy has quit IRC | 15:13 | |
| *** yuanying has quit IRC | 15:13 | |
| *** cody-somerville has quit IRC | 15:13 | |
| *** zzzeek has quit IRC | 15:13 | |
| *** pcrews_ has quit IRC | 15:13 | |
| *** aviau has quit IRC | 15:13 | |
| *** fbo has quit IRC | 15:13 | |
| *** markmcd1 has quit IRC | 15:13 | |
| *** lucasagomes has quit IRC | 15:13 | |
| *** ianychoi has quit IRC | 15:13 | |
| *** hughsaunders has quit IRC | 15:13 | |
| *** dstufft has quit IRC | 15:13 | |
| *** adreznec has quit IRC | 15:13 | |
| *** mrunge has quit IRC | 15:13 | |
| *** ruhe has quit IRC | 15:13 | |
| *** aloga has quit IRC | 15:13 | |
| *** clayton has quit IRC | 15:13 | |
| *** Madasi has quit IRC | 15:13 | |
| *** tobiash has quit IRC | 15:13 | |
| *** dmsimard has quit IRC | 15:13 | |
| *** diana_clarke has quit IRC | 15:13 | |
| *** skraynev has quit IRC | 15:13 | |
| *** hughsaunders_ is now known as hughsaunders | 15:13 | |
| *** ruhe2 is now known as ruhe | 15:13 | |
| *** Shalmazar is now known as Madasi | 15:13 | |
| *** dstufft_ is now known as dstufft | 15:13 | |
| *** mylu has joined #openstack-infra | 15:13 | |
| *** zzzeek has joined #openstack-infra | 15:13 | |
| *** shardy has quit IRC | 15:14 | |
| *** austin81 has joined #openstack-infra | 15:14 | |
| *** jamesmcarthur has joined #openstack-infra | 15:15 | |
| jamesmcarthur | Do you have a few minutes? | 15:15 |
| *** cdent has joined #openstack-infra | 15:15 | |
| *** Guest27982 has quit IRC | 15:17 | |
| *** rcernin has quit IRC | 15:17 | |
| *** ddieterly[away] is now known as ddieterly | 15:17 | |
| *** kzaitsev_mb has joined #openstack-infra | 15:17 | |
| *** david-lyle has joined #openstack-infra | 15:17 | |
| *** links has quit IRC | 15:18 | |
| *** IlyaG has joined #openstack-infra | 15:18 | |
| jpmaxman | Fungi smarcet 750 is OK for the time being. I don't want to make any changes to production that has any possibility of disruption of service | 15:19 |
| fungi | jamesmcarthur: smarcet: jpmaxman: also i didn't mean to belittle your work by calling whatever it is you're working on a "phone-thingy" i'm just not up on what it is or what it's intended to do, just that it seems to want to authenticate to the openid provider waaaaay more than anything else we've got | 15:19 |
| *** adreznec_ is now known as adreznec | 15:19 | |
| *** armax has quit IRC | 15:19 | |
| *** IlyaG has quit IRC | 15:20 | |
| fungi | i'm not really familiar with smartphones or what sort of apps people use them for since i don't carry one | 15:20 |
| *** esikachev has joined #openstack-infra | 15:20 | |
| *** IlyaG has joined #openstack-infra | 15:20 | |
| jpmaxman | We are about to submit the application and it will get rejected if there just happens to be some hiccup when Apple reviews it | 15:20 |
| *** armax has joined #openstack-infra | 15:20 | |
| jpmaxman | This is why I wanted to take optimization testing onto a non production server | 15:20 |
| jpmaxman | We can run load tests agaisng thag | 15:21 |
| jeblair | i fully endorse not running load tests on a production server :) | 15:21 |
| EmilienM | in case infra run unit tests on puppet modules, a ruby dep broke https://github.com/jwt/ruby-jwt/issues/133 | 15:21 |
| EmilienM | which reminds me how much I'm looking forward to have openstack rubygems mirrors, that are gated. | 15:21 |
| jeblair | EmilienM: we aren't gating any of our mirrors -- we generally use things like requirements and pins when something breaks | 15:22 |
| EmilienM | jeblair: yes, we need to find a similar system to have a common Gemfile | 15:23 |
| jeblair | EmilienM: (the theory being: it's broken for everyone, including local development) | 15:23 |
| jeblair | EmilienM: ++ | 15:23 |
| EmilienM | and changes on Gemfile would be gated | 15:23 |
| EmilienM | mwhahaha: do you think we could use https://github.com/openstack/puppet-openstack_spec_helper as a central place for the Gemfile? | 15:24 |
| *** yamamoto has joined #openstack-infra | 15:24 | |
| *** jsavak has joined #openstack-infra | 15:24 | |
| EmilienM | and in our puppet modules, add only one dep in Gemfile to have puppet-openstack_spec_helper ? would it work? | 15:24 |
| EmilienM | and since puppet-openstack_spec_helper is gated with unit tests / beaker, it would be easy to manage deps in a single place | 15:25 |
| *** esikachev has quit IRC | 15:25 | |
| *** dmsimard has joined #openstack-infra | 15:25 | |
| jeblair | i love it when ideas start with "mwhahaha". ;) | 15:25 |
| *** eharney has quit IRC | 15:25 | |
| *** fmccrthy has joined #openstack-infra | 15:25 | |
| mwhahaha | yea might be a good idea | 15:25 |
| *** baoli has joined #openstack-infra | 15:25 | |
| jpmaxman | And since we were going to have a new server to optimize against thus came the idea to go ahead and optimize nginx | 15:25 |
| *** sdake_ has joined #openstack-infra | 15:25 | |
| *** sdake has quit IRC | 15:25 | |
| bkero | mordred: on-disk tree is about 223GB. Not too bad. | 15:25 |
| EmilienM | jeblair: first time he joined our channel, I thought he was a spam. | 15:25 |
| *** yamahata has quit IRC | 15:26 | |
| EmilienM | jeblair: and now, he's core on our modules ;-) | 15:26 |
| mordred | bkero: cool. does it have a flat dir structure? or does it have any amount of dir hashing? | 15:26 |
| EmilienM | mwhahaha: ok, I'm testing it | 15:26 |
| bkero | mordred: flat :/ | 15:27 |
| bkero | $ ls -hl /data/rubygems/gems | wc -l | 15:27 |
| bkero | 675635 | 15:27 |
| EmilienM | bkero: you working on rubygems mirror? | 15:28 |
| jeblair | that's about 642868 too many :( | 15:28 |
| *** imcsk8 has quit IRC | 15:28 | |
| *** imcsk8_ has joined #openstack-infra | 15:28 | |
| *** jsavak has quit IRC | 15:29 | |
| *** Qiming has joined #openstack-infra | 15:29 | |
| *** jsavak has joined #openstack-infra | 15:29 | |
| mordred | bkero: so - sounds like we'll need an update to the mirror program to write into a hashed structure, and then potentially rewrite rules | 15:29 |
| *** HeOS has joined #openstack-infra | 15:29 | |
| *** yaume has joined #openstack-infra | 15:29 | |
| *** edmondsw has joined #openstack-infra | 15:29 | |
| *** flepied has joined #openstack-infra | 15:29 | |
| *** lifeless has joined #openstack-infra | 15:29 | |
| *** therve has joined #openstack-infra | 15:29 | |
| *** yuanying has joined #openstack-infra | 15:29 | |
| *** cody-somerville has joined #openstack-infra | 15:29 | |
| *** pcrews_ has joined #openstack-infra | 15:29 | |
| *** fbo has joined #openstack-infra | 15:29 | |
| *** markmcd1 has joined #openstack-infra | 15:29 | |
| *** ianychoi has joined #openstack-infra | 15:29 | |
| *** diana_clarke has joined #openstack-infra | 15:29 | |
| *** clayton has joined #openstack-infra | 15:29 | |
| *** tobiash has joined #openstack-infra | 15:29 | |
| *** skraynev has joined #openstack-infra | 15:29 | |
| mordred | bkero: (we had todo this for pypi mirror already) | 15:29 |
| *** Guest27982 has joined #openstack-infra | 15:30 | |
| *** lifeless has quit IRC | 15:30 | |
| *** rcernin has joined #openstack-infra | 15:30 | |
| *** e0ne has quit IRC | 15:30 | |
| *** lifeless has joined #openstack-infra | 15:30 | |
| bkero | EmilienM: yes | 15:30 |
| *** e0ne has joined #openstack-infra | 15:30 | |
| *** MED has joined #openstack-infra | 15:31 | |
| EmilienM | bkero: ♥ | 15:31 |
| bkero | mordred: The hashed directory structure isn't too difficult. Assuming the hashes are kept around in a sane way and able to be referenced | 15:31 |
| *** cdent has left #openstack-infra | 15:32 | |
| *** dalgaaf has quit IRC | 15:32 | |
| *** jpmaxman has quit IRC | 15:32 | |
| bkero | Not sure what you mean by rewrite rules. mod_rewrite stuff or perhaps something else | 15:32 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Make job networking-generic-switch-dsvm voting https://review.openstack.org/297218 | 15:32 |
| *** binbincong has quit IRC | 15:33 | |
| *** esikachev has joined #openstack-infra | 15:33 | |
| *** dalgaaf has joined #openstack-infra | 15:33 | |
| *** asettle has joined #openstack-infra | 15:34 | |
| *** jpmaxman has joined #openstack-infra | 15:34 | |
| *** tqtran has joined #openstack-infra | 15:35 | |
| jeblair | bkero: yes, mod_rewrite. we make the filesystem-level dir-hashing transparent to the pypi download protocol | 15:36 |
| *** lucasagomes_ is now known as lucasagomes | 15:36 | |
| bkero | jeblair: Oh, that's simple enough. Especially if it's already written for pypi. | 15:36 |
| jeblair | bkero: so the mirroring program knows to write out to "a/b/abc" but pip just fetches from "/abc" and gets the right file | 15:36 |
| bkero | right | 15:36 |
| jeblair | er, i think it's only one level, but you get the idea | 15:37 |
| *** yamahata has joined #openstack-infra | 15:37 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion with net-iso in periodic job https://review.openstack.org/296765 | 15:37 |
| bkero | Ok, one level is even easier :) | 15:37 |
| jeblair | bkero: here we are: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb | 15:38 |
| mordred | bkero: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/pypi.vhost.erb | 15:38 |
| mordred | heh | 15:38 |
| jeblair | hrm. we should probably delete one of those :) | 15:38 |
| *** asettle has quit IRC | 15:38 | |
| bkero | nice | 15:39 |
| openstackgerrit | Thierry Carrez proposed openstack/requirements: Precisions on the licensing requirements https://review.openstack.org/297223 | 15:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Remove old pypi vhost template https://review.openstack.org/297224 | 15:39 |
| *** tqtran has quit IRC | 15:39 | |
| *** annegentle has quit IRC | 15:39 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Fix which branch requirements bot runs https://review.openstack.org/297225 | 15:39 |
| *** binbincong has joined #openstack-infra | 15:40 | |
| mordred | jeblair: ls /afs/openstack.org/mirror/pypi/web/simple/r/ is not quick | 15:40 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs https://review.openstack.org/289445 | 15:41 |
| *** annegentle has joined #openstack-infra | 15:41 | |
| mordred | jeblair: well, it's quick the second time :) | 15:42 |
| *** shardy has joined #openstack-infra | 15:42 | |
| *** eharney has joined #openstack-infra | 15:42 | |
| jeblair | mordred: indeed :) | 15:44 |
| notmyname | I'd like to talk about setting up some swift-specific gate jobs. can someone join me in the -swift channel to talk about it? (rather than trying to have multiple conversations at once in here) | 15:45 |
| *** annegentle has quit IRC | 15:46 | |
| *** shardy has quit IRC | 15:46 | |
| *** Swami has joined #openstack-infra | 15:46 | |
| notmyname | or I can ask in here. whatever you think is best | 15:46 |
| pabelanger | infra-root: if you would mind holding off on approving any more project-config changes for JJB, jenkins will be thankful of it. Still manually processing the puppet-job change | 15:46 |
| *** doug-fish has joined #openstack-infra | 15:47 | |
| *** shardy has joined #openstack-infra | 15:47 | |
| *** yamamoto has quit IRC | 15:48 | |
| anteaya | pabelanger: ack | 15:48 |
| anteaya | I also think you should include AJaeger and sdague in that request as they aren't infra-root | 15:48 |
| *** shardy has quit IRC | 15:48 | |
| fungi | yes, project-config-core reviewers | 15:48 |
| anteaya | is anyone doing anything specific with jenkins05? | 15:48 |
| clarkb | notmyname: my I can join -swift | 15:48 |
| *** shardy has joined #openstack-infra | 15:49 | |
| pabelanger | anteaya: right, thanks for the reminder | 15:49 |
| anteaya | pabelanger: welcome | 15:49 |
| pabelanger | I am currently doing jenkins and jenkins01 updates for JJB | 15:49 |
| sdague | pabelanger: ack | 15:49 |
| *** annegentle has joined #openstack-infra | 15:49 | |
| anteaya | pabelanger: okay thanks | 15:49 |
| sdague | I will hold off on approves until you tell me otherwise | 15:49 |
| pabelanger | starting jenkins02 now | 15:49 |
| anteaya | jenkins05 seems to be off doing its own thing at the moment | 15:49 |
| fungi | yes, 05 does appear to be out to lunch. i'll see what's up with it | 15:50 |
| anteaya | fungi: thanks | 15:50 |
| *** yamamoto has joined #openstack-infra | 15:50 | |
| EmilienM | pabelanger: still NOT_REGISTERED FYI for all puppet jobs | 15:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 15:50 |
| fungi | looks like the service has ceased entirely | 15:50 |
| anteaya | fungi: :( | 15:50 |
| *** yamamoto has quit IRC | 15:50 | |
| *** angdraug has joined #openstack-infra | 15:51 | |
| anteaya | fungi: is logging an issue? | 15:51 |
| fungi | [Thu Feb 11 20:10:05 2016] Out of memory: Kill process 27318 (java) score 787 or sacrifice child | 15:51 |
| *** oanson has quit IRC | 15:51 | |
| fungi | that's the last time it actually oom'd | 15:51 |
| fungi | been more than a month | 15:51 |
| fungi | so i'm thinking something else has stopped it | 15:51 |
| anteaya | okay thanks for checking | 15:52 |
| fungi | yeah, still digging for a root cause | 15:52 |
| anteaya | cacti shows it still has 9G of space on / | 15:52 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job https://review.openstack.org/273743 | 15:52 |
| fungi | looks like it may have been a java oom rather than a kernel oom | 15:53 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1463&rra_id=all suggests it died around 14:50 utc | 15:53 |
| anteaya | I buy that interpretation | 15:53 |
| fungi | expecting to find a jvm oom in the jenkins log | 15:53 |
| fungi | but it's taking a minute to load | 15:54 |
| anteaya | :( | 15:54 |
| *** ybathia has joined #openstack-infra | 15:55 | |
| openstackgerrit | tatyana-leontovich proposed openstack-infra/project-config: fuel-ostf: use db setup for python jobs https://review.openstack.org/297236 | 15:55 |
| *** tonytan4ever has quit IRC | 15:56 | |
| *** akaszuba has quit IRC | 15:56 | |
| *** tonytan4ever has joined #openstack-infra | 15:56 | |
| *** tonytan_brb has joined #openstack-infra | 15:59 | |
| *** mikelk has quit IRC | 16:00 | |
| *** tonytan4ever has quit IRC | 16:00 | |
| *** pblaho has quit IRC | 16:00 | |
| *** ifarkas has quit IRC | 16:00 | |
| anteaya | if a channel op has a moment to refresh the topic in meeting that would be great | 16:01 |
| anteaya | orwell was bored apparently | 16:01 |
| *** nmagnezi has quit IRC | 16:02 | |
| *** Nikolay_St has joined #openstack-infra | 16:02 | |
| anteaya | the other channels don't seem affected | 16:02 |
| *** ikalnitsky has left #openstack-infra | 16:03 | |
| *** Kennan has quit IRC | 16:04 | |
| fungi | Mar 24, 2016 2:51:56 PM OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006d4880000, 421527552, 0) failed; error='Cannot allocate memory' (errno=12) | 16:04 |
| EmilienM | pabelanger, fungi: sorry to ask but NOT_REGISTERED is still expected? | 16:04 |
| EmilienM | is it related to OOM issues? | 16:04 |
| anteaya | fungi: all the used space appears to have just filled up on 05 | 16:04 |
| fungi | anteaya: that may be temporary from me opening the jenkins.log file | 16:05 |
| anteaya | EmilienM: no it isn't related, I'm going with it is due to races as pabelanger is trying to manually updated the jenkinsii | 16:05 |
| anteaya | fungi: ah okay | 16:05 |
| *** Kennan has joined #openstack-infra | 16:05 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Adjust grenade branches for mitaka https://review.openstack.org/297004 | 16:05 |
| fungi | once i closed the viewer it's reporting 17% free again | 16:05 |
| anteaya | k | 16:05 |
| *** brad_behle has joined #openstack-infra | 16:06 | |
| anteaya | I'll give cacti a few minutes to catch up | 16:06 |
| EmilienM | all puppet jobs (including puppet openstack, fuel and tripleo) are failing with NOT_REGISTERED | 16:06 |
| fungi | i'm deleting all nodes nodepool has on record for 05 and cleaning up its slaves in configuration before restarting it | 16:06 |
| anteaya | fungi: ack and thanks | 16:06 |
| *** Guest82270 has joined #openstack-infra | 16:06 | |
| *** abregman has quit IRC | 16:07 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Delete test stack without confirmation https://review.openstack.org/295047 | 16:07 |
| anteaya | EmilienM: I'd wait until pabelanger reports he has finished his updates and then see if tests are still coming back with NOT_REGISTERED | 16:07 |
| pabelanger | EmilienM: Ya, we are still landing JJB | 16:07 |
| *** Guest82270 is now known as Vivek | 16:07 | |
| *** Vivek has quit IRC | 16:07 | |
| *** Vivek has joined #openstack-infra | 16:07 | |
| EmilienM | pabelanger: ack, just asking. Please ping me when I can recheck | 16:07 |
| pabelanger | EmilienM: will do | 16:08 |
| Vivek | topic | 16:08 |
| *** apoorvad has joined #openstack-infra | 16:09 | |
| *** ifarkas has joined #openstack-infra | 16:10 | |
| *** tonytan_brb has quit IRC | 16:11 | |
| *** tonytan4ever has joined #openstack-infra | 16:11 | |
| YorikSar | Can anybody else please review my DB-jobs change https://review.openstack.org/297088 - I have one +2 from fungi already! | 16:11 |
| *** gyee has joined #openstack-infra | 16:12 | |
| anteaya | YorikSar: pabelanger is currently updating our jenkinsii and we are holding off +A'ing project-config changes until he is done | 16:13 |
| *** tphummel has joined #openstack-infra | 16:14 | |
| *** e0ne has quit IRC | 16:14 | |
| pabelanger | okay, jenkins.o.o finished, moving to jenkins03 | 16:14 |
| *** jsavak has quit IRC | 16:14 | |
| pabelanger | jenkins01 / jenkins02 still processing | 16:15 |
| anteaya | ack | 16:15 |
| *** Douhet has quit IRC | 16:17 | |
| *** zeih has quit IRC | 16:18 | |
| pabelanger | jenkins02 is going to take a while, it seems to be out sync more then other 2 masters | 16:18 |
| *** Douhet has joined #openstack-infra | 16:19 | |
| clarkb | pabelanger: ya even deleting a large number of files on it took much longer than the other masters, I think its io is just not as good | 16:19 |
| aspiers | AJaeger: still around? | 16:21 |
| *** nadya has quit IRC | 16:21 | |
| YorikSar | anteaya: Ok, thank you for information. | 16:21 |
| anteaya | YorikSar: welcome | 16:21 |
| *** mriedem is now known as mriedem_snow | 16:22 | |
| *** ddieterly is now known as ddieterly[away] | 16:24 | |
| *** sheeprine has quit IRC | 16:25 | |
| *** vgridnev has quit IRC | 16:25 | |
| *** imcsk8_ is now known as imcsk8 | 16:27 | |
| *** esikachev has quit IRC | 16:27 | |
| *** EricGonczer_ has quit IRC | 16:27 | |
| *** jsavak has joined #openstack-infra | 16:27 | |
| *** jordanP has quit IRC | 16:28 | |
| *** kzaitsev_mb has quit IRC | 16:28 | |
| *** vgridnev has joined #openstack-infra | 16:28 | |
| *** _ari_ has quit IRC | 16:29 | |
| *** sheeprine has joined #openstack-infra | 16:30 | |
| *** zzzeek has quit IRC | 16:30 | |
| *** _ari_ has joined #openstack-infra | 16:30 | |
| pabelanger | clarkb: so, I think we need to update: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/puppet-module-jobs.yaml#L358 | 16:31 |
| pabelanger | clarkb: to not be a list | 16:31 |
| clarkb | oh right | 16:31 |
| clarkb | that issue | 16:31 |
| clarkb | pabelanger: yup | 16:31 |
| clarkb | silly jjb | 16:31 |
| pabelanger | export 'BEAKER_set=nodepool-[centos7] | 16:31 |
| clarkb | pabelanger: I am distracted by all the things | 16:32 |
| pabelanger | sure | 16:32 |
| clarkb | pabelanger: but you should be able to just make it key value | 16:32 |
| clarkb | rather than key [value] | 16:32 |
| pabelanger | I can get the patch up | 16:32 |
| *** gnuoy_ has joined #openstack-infra | 16:32 | |
| pabelanger | once these jenkins run | 16:32 |
| *** bpokorny has joined #openstack-infra | 16:33 | |
| *** zzzeek has joined #openstack-infra | 16:33 | |
| *** sdake_ is now known as sdake | 16:34 | |
| pabelanger | EmilienM: so we removed devstack from the job names, so they should be nodepool-ubuntu-trusty and nodepool-centos7 | 16:35 |
| EmilienM | ok | 16:35 |
| *** jsavak has quit IRC | 16:35 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/system-config: OpenStackId 1.0.17 https://review.openstack.org/297251 | 16:35 |
| EmilienM | pabelanger: you fire up a patch or am I doing it? | 16:35 |
| pabelanger | EmilienM: Ya, was going to wait a few minutes until these jenkins finished | 16:35 |
| pabelanger | then ensure nothing else was needed | 16:36 |
| *** jsavak has joined #openstack-infra | 16:36 | |
| openstackgerrit | Andrey Nikitin proposed openstack-infra/project-config: 'fuel-abandon' group added to Fuel-related projects https://review.openstack.org/297252 | 16:36 |
| EmilienM | pabelanger: do we also change "node" for centos? it's devstack-centos7 right now | 16:36 |
| *** vgridnev has quit IRC | 16:36 | |
| pabelanger | EmilienM: not yet | 16:36 |
| pabelanger | we still have devstack-centos7 still | 16:36 |
| pabelanger | it will be ostype that needs to change | 16:37 |
| EmilienM | kk | 16:37 |
| pabelanger | to what clarkb said | 16:37 |
| pabelanger | ostype: centos7 | 16:37 |
| pabelanger | ostype: ubuntu-trusty | 16:37 |
| pabelanger | err trusty | 16:37 |
| *** gnuoy_ has quit IRC | 16:38 | |
| pabelanger | in that case, I don't think you need to patch your modules once we fix JJB | 16:38 |
| EmilienM | I'll let you do the patch ;-) and review it | 16:38 |
| pabelanger | k | 16:38 |
| EmilienM | if we could the patches I did, that would be excellent | 16:38 |
| *** ddieterly[away] is now known as ddieterly | 16:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix typo with ostype for puppet beaker jobs https://review.openstack.org/297253 | 16:40 |
| pabelanger | EmilienM: clarkb: ^ I think we need that | 16:40 |
| dims | anteaya : when you get a chance, please see https://review.openstack.org/#/c/296781/ | 16:41 |
| *** jamesmcarthur has quit IRC | 16:41 | |
| EmilienM | gerrit looks down to me | 16:41 |
| HeOS | To me too. | 16:41 |
| *** IlyaG has quit IRC | 16:42 | |
| *** ybathia has quit IRC | 16:42 | |
| pabelanger | it is getting hammered right now | 16:42 |
| pabelanger | let me check logs | 16:42 |
| anteaya | gerrit cpu usage is really high: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=25&rra_id=all | 16:42 |
| *** ajmiller has joined #openstack-infra | 16:43 | |
| anteaya | tcp connections spiked | 16:43 |
| anteaya | do we have a runaway bot? | 16:43 |
| *** jamesmcarthur has joined #openstack-infra | 16:46 | |
| AJaeger | pabelanger: sorry, saw your node on holiding approvals to late - and can't right now remove my +W on the one change I approved ;( | 16:46 |
| *** gokrokve has joined #openstack-infra | 16:46 | |
| AJaeger | Let's see whether gerrit is back sooner than jenkins finishing the test ;( | 16:46 |
| *** achanda has joined #openstack-infra | 16:47 | |
| anteaya | zuul's memory usage is climbing today: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 16:48 |
| *** annegentle has quit IRC | 16:49 | |
| *** gnuoy_ has joined #openstack-infra | 16:50 | |
| pabelanger | don't see anything specific | 16:50 |
| pabelanger | but stackalytics bot is currently running | 16:50 |
| pabelanger | which usually asks for a lot of data | 16:51 |
| pabelanger | sadly, I don't know what to look for when our CPU go up | 16:51 |
| *** yamamoto has joined #openstack-infra | 16:51 | |
| zaro | morning | 16:52 |
| clarkb | gerrit is deep into gcing | 16:52 |
| anteaya | pabelanger: does the java melody app under /monitoring show we are spending a lot of time in garbage collection | 16:52 |
| clarkb | which has spiked the active threads which means requests aren't getting served | 16:52 |
| HeOS | pabelanger, try to execute: ssh <your_login>@review.openstack.org -p 29418 gerrit show-connections -w | 16:52 |
| clarkb | I think the best bet is likely to restart the esrver | 16:52 |
| * AJaeger was successfull to remove the +W. | 16:52 | |
| anteaya | that seems to be our only way of addressing the gc abyss | 16:53 |
| anteaya | +1 restart gerrit | 16:53 |
| *** jerryz has joined #openstack-infra | 16:53 | |
| *** bpokorny_ has joined #openstack-infra | 16:53 | |
| pabelanger | anteaya: I forgot about that url | 16:54 |
| anteaya | pabelanger: no worries | 16:54 |
| pabelanger | AJaeger: np | 16:54 |
| anteaya | it takes 9 reminders to remember anything new | 16:54 |
| *** samuelBartel has quit IRC | 16:54 | |
| anteaya | 8 more to go :) | 16:54 |
| * AJaeger thought he was successfull to remove the +W, run again into gerrit timeout ;( | 16:54 | |
| *** gnuoy_ has quit IRC | 16:55 | |
| *** ddieterly has quit IRC | 16:55 | |
| *** romcheg has left #openstack-infra | 16:55 | |
| *** odyssey4me has quit IRC | 16:55 | |
| *** odyssey4me has joined #openstack-infra | 16:55 | |
| *** e0ne has joined #openstack-infra | 16:56 | |
| *** esker has quit IRC | 16:56 | |
| *** bpokorny has quit IRC | 16:56 | |
| anteaya | clarkb: so are you able to restart gerrit? | 16:56 |
| anteaya | or walk pabelanger through restarting gerrit? | 16:56 |
| clarkb | anteaya: I am still suer distracted if anyone else can that would be best | 16:57 |
| *** vincentll has quit IRC | 16:57 | |
| anteaya | pabelanger: have you ever done it before? | 16:57 |
| clarkb | its just service gerrit stop, check it stopped with ps, service gerrit start | 16:57 |
| fungi | clarkb: i want to say that after the recent jenkins upgrade you mentioned slave configuration got extracted out to a separate file/directory... where did that end up? | 16:57 |
| clarkb | fungi: /var/lib/jenkins/nodes/ | 16:57 |
| anteaya | pabelanger: and channel notifications with statusbot | 16:57 |
| pabelanger | clarkb: anteaya: I can restart gerrit. let me do that now | 16:57 |
| *** bpokorny_ has quit IRC | 16:58 | |
| pabelanger | stopping | 16:58 |
| *** jsavak has quit IRC | 16:58 | |
| pabelanger | ps good | 16:59 |
| pabelanger | starting | 16:59 |
| *** bpokorny has joined #openstack-infra | 16:59 | |
| *** sputnik13 has joined #openstack-infra | 16:59 | |
| sdake | gtting a service unavaialble at review.openstack.org | 16:59 |
| *** yamamoto has quit IRC | 16:59 | |
| pabelanger | gerrit started | 16:59 |
| EmilienM | sdake: we are in the middle of gerrit outage yes | 16:59 |
| *** jamesmcarthur has quit IRC | 16:59 | |
| sdake | EmilienM got it thanks | 16:59 |
| pabelanger | clarkb: does apache need a restart? | 16:59 |
| clarkb | pabelanger: no it should figure out soon enough gerrit is back | 17:00 |
| pabelanger | clarkb: yup, looks to be back | 17:00 |
| *** sridhar_ram has joined #openstack-infra | 17:00 | |
| pabelanger | okay, checking gerritbots | 17:00 |
| *** openstackgerrit has quit IRC | 17:01 | |
| anteaya | pabelanger: can you use statusbot to add a notification to our status wikipage that it happpened? | 17:01 |
| clarkb | pabelanger: https://review.openstack.org/monitoring?part=graph&graph=gc is what GC spike looks like in java melody | 17:01 |
| clarkb | that solid block of spending all my time looking for free memory | 17:01 |
| pabelanger | anteaya: I can try | 17:01 |
| *** openstackgerrit has joined #openstack-infra | 17:01 | |
| anteaya | thanks | 17:01 |
| *** e0ne_ has joined #openstack-infra | 17:02 | |
| pabelanger | anteaya: where can I find the syntax for the statusbot? | 17:02 |
| *** jsavak has joined #openstack-infra | 17:02 | |
| pabelanger | I only know the success bot :) | 17:02 |
| *** kzaitsev_mb has joined #openstack-infra | 17:02 | |
| *** scheuran has quit IRC | 17:02 | |
| EmilienM | AJaeger: in case you have time please look https://review.openstack.org/#/c/297253/ we're trying to bring Puppet CI back | 17:02 |
| pabelanger | clarkb: okay, good to know | 17:02 |
| pabelanger | openstackstatus: help | 17:03 |
| *** rossella_s has quit IRC | 17:03 | |
| anteaya | pabelanger: http://docs.openstack.org/infra/system-config/irc.html#statusbot | 17:03 |
| *** rossella_s has joined #openstack-infra | 17:03 | |
| pabelanger | anteaya: thanks, I was close to finding it | 17:03 |
| *** e0ne has quit IRC | 17:03 | |
| *** nadya has joined #openstack-infra | 17:04 | |
| pabelanger | #status log restarted gerrit to address GC issue | 17:04 |
| pabelanger | maybe a permission issue | 17:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add pabelanger to statusbot https://review.openstack.org/297259 | 17:05 |
| pabelanger | ^ that'll have to land first | 17:05 |
| anteaya | oh | 17:05 |
| EmilienM | well, it won't land, puppet jobs are down :-P | 17:06 |
| fungi | thanks clarkb! | 17:06 |
| pabelanger | EmilienM: indeed | 17:06 |
| anteaya | clarkb or fungi can you upstate status? | 17:06 |
| *** ybathia has joined #openstack-infra | 17:06 | |
| anteaya | pabelanger: thanks for restarting gerrit | 17:06 |
| pabelanger | EmilienM: jenkins01 / jenkins03 almost done | 17:06 |
| fungi | clarkb: so `sudo rm -rf /var/lib/jenkins/nodes/*` looks like it should be what i want. safe? | 17:06 |
| clarkb | fungi: I think that is what jeblair did on 07 | 17:06 |
| pabelanger | anteaya: np | 17:06 |
| AJaeger | EmilienM: done | 17:07 |
| *** mrmartin has joined #openstack-infra | 17:07 | |
| fungi | cool, jenkins05 cleaned up and on its way back into service now | 17:07 |
| AJaeger | off again... | 17:07 |
| anteaya | fungi: thank you | 17:07 |
| *** jsavak has quit IRC | 17:08 | |
| fungi | #status log restarted gerrit to address GC issue | 17:08 |
| openstackstatus | fungi: finished logging | 17:08 |
| anteaya | and thank you | 17:08 |
| EmilienM | AJaeger: ty | 17:08 |
| fungi | pabelanger: you'll want to propose a patch to add your nick to the statusbot config in project-config i think | 17:08 |
| pabelanger | fungi: sure, let me check | 17:08 |
| pabelanger | fungi: I did one for system-config | 17:08 |
| *** annegentle has joined #openstack-infra | 17:09 | |
| fungi | pabelanger: oh! you're not identified to nickserv | 17:10 |
| fungi | so that could also explain it | 17:10 |
| pabelanger | doh | 17:10 |
| *** jlanoux has quit IRC | 17:10 | |
| pabelanger | I thought I fixed that with znc | 17:10 |
| *** automagically has left #openstack-infra | 17:11 | |
| *** pabelanger has quit IRC | 17:11 | |
| *** pabelanger has joined #openstack-infra | 17:11 | |
| *** mylu has quit IRC | 17:11 | |
| *** mylu has joined #openstack-infra | 17:12 | |
| *** electrofelix has joined #openstack-infra | 17:13 | |
| *** tonytan_brb has joined #openstack-infra | 17:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add default value to wait parameter https://review.openstack.org/297262 | 17:13 |
| *** annegentle has quit IRC | 17:14 | |
| *** pradk_ has joined #openstack-infra | 17:14 | |
| *** rkukura has joined #openstack-infra | 17:15 | |
| mordred | Shrews: ^^ we broke somehting. it's really small and the chances that we broke a real user are almost none | 17:15 |
| mordred | Shrews: but we should fix it on general principle | 17:15 |
| *** ifarkas has quit IRC | 17:16 | |
| *** tonytan4ever has quit IRC | 17:16 | |
| *** pradk has quit IRC | 17:16 | |
| *** pradk_ is now known as pradk | 17:16 | |
| pabelanger | jenkins02 and jenkins03 are sync'd so puppet jobs should be working as expected now | 17:16 |
| pabelanger | EmilienM: ^ | 17:16 |
| *** jamesmcarthur has joined #openstack-infra | 17:16 | |
| clarkb | pabelanger: I approved the trusty/centos7 list fix | 17:17 |
| pabelanger | still working on other master | 17:17 |
| *** e0ne_ has quit IRC | 17:17 | |
| pabelanger | clarkb: okay, I'll monitor | 17:17 |
| *** eil397 has joined #openstack-infra | 17:17 | |
| *** nmagnezi has joined #openstack-infra | 17:17 | |
| *** mrmartin has quit IRC | 17:18 | |
| EmilienM | pabelanger, clarkb: thx a ton | 17:18 |
| *** tmorin has left #openstack-infra | 17:19 | |
| *** Goneri has joined #openstack-infra | 17:21 | |
| *** asettle has joined #openstack-infra | 17:22 | |
| *** jaypipes has quit IRC | 17:23 | |
| *** yaume has quit IRC | 17:23 | |
| *** mrmartin has joined #openstack-infra | 17:23 | |
| *** gyee has quit IRC | 17:23 | |
| *** MED has quit IRC | 17:24 | |
| fungi | pabelanger: it _should_ be safe to update jjb on all the masters in parallel currently, since the recent fixes seem to have reduced gearman registration impact on zuul | 17:24 |
| clarkb | rax-ord and ovh-bhs1 do not have updated ubuntu-trusty images yet, but everything else does | 17:24 |
| pabelanger | fungi: okay, doing that now | 17:25 |
| pabelanger | fungi: safe to start on jenkins05? | 17:25 |
| clarkb | fungi: that is probably a small enough portion of our quota that we could ge taway with deleting ubuntu-trusty in those two regions | 17:25 |
| *** pradk_ has joined #openstack-infra | 17:25 | |
| fungi | pabelanger: yep, it's up and working fine | 17:25 |
| *** mylu has quit IRC | 17:26 | |
| fungi | clarkb: agreed, we're also not running at max capacity for the moment | 17:26 |
| fungi | and heading into a long weekend for most of the world | 17:26 |
| *** asettle has quit IRC | 17:27 | |
| *** mylu has joined #openstack-infra | 17:27 | |
| anteaya | clarkb: yay | 17:28 |
| *** lucasagomes is now known as lucas-afk | 17:28 | |
| clarkb | fungi: I will go ahead and delete the existing old ubuntu-trusty images in those regions then | 17:28 |
| fungi | sounds good | 17:28 |
| fungi | i need to disappear and grab lunch, have friends staying through the weekend so need to pretend to be social | 17:28 |
| fungi | bbiaw | 17:28 |
| EmilienM | pabelanger: ubuntu fails https://jenkins04.openstack.org/job/gate-puppet-nova-puppet-beaker-rspec-ubuntu-trusty/1/consoleFull | 17:28 |
| anteaya | fungi: enjoy lunch and pretending to be social | 17:28 |
| EmilienM | BEAKER_set='nodepool-[trusty]' | 17:29 |
| *** gordc has quit IRC | 17:29 | |
| clarkb | oh its easter | 17:29 |
| pabelanger | EmilienM: Yup, expected | 17:29 |
| *** jamesmcarthur has quit IRC | 17:29 | |
| EmilienM | pabelanger: should I rename files? | 17:29 |
| pabelanger | until https://review.openstack.org/#/c/297253/ lands | 17:29 |
| pabelanger | that should fix it | 17:29 |
| EmilienM | I thought it was landed | 17:29 |
| EmilienM | :) | 17:29 |
| pabelanger | still processing | 17:30 |
| *** tonytan_brb is now known as tonytan4ever | 17:30 | |
| *** pradk has quit IRC | 17:30 | |
| *** mylu has quit IRC | 17:30 | |
| *** mylu has joined #openstack-infra | 17:30 | |
| *** eil397 has quit IRC | 17:30 | |
| anteaya | clarkb: yeah, happy easter | 17:30 |
| EmilienM | looking at https://jenkins07.openstack.org/job/gate-project-config-compare-xml/323/consoleFull | 17:30 |
| *** vgridnev has joined #openstack-infra | 17:31 | |
| anteaya | happy chocolate bunnies to you | 17:31 |
| EmilienM | 2016-03-24 17:01:10.502 | -export BEAKER_set=nodepool-['trusty'] | 17:31 |
| EmilienM | 2016-03-24 17:01:10.502 | +export BEAKER_set=nodepool-trusty | 17:31 |
| EmilienM | pabelanger: is it ok? ^ | 17:31 |
| *** pradk has joined #openstack-infra | 17:31 | |
| *** jsavak has joined #openstack-infra | 17:31 | |
| *** pradk_ has quit IRC | 17:31 | |
| pabelanger | EmilienM: ya, that's what the patch should do | 17:31 |
| EmilienM | ok | 17:31 |
| clarkb | I can barely keep track of what day of the week it is let alone holidays | 17:32 |
| *** earlephilhower has joined #openstack-infra | 17:32 | |
| EmilienM | clarkb: in US you have Friday off, in Canada, it depends, some of us have also Monday ;-) | 17:32 |
| anteaya | clarkb: ha ha ha | 17:32 |
| *** ybathia has quit IRC | 17:32 | |
| *** skolekonov is now known as _skolekonov | 17:33 | |
| anteaya | clarkb: good think you have fungi for the holidays part | 17:33 |
| pabelanger | Ya, I get both Friday and Monday as PTO | 17:33 |
| pabelanger | but will likely check in from time to time | 17:33 |
| EmilienM | I didn't know infra-roots had PTOs :-P | 17:34 |
| clarkb | ok ovh-bhs1 and rax-ord are without old broken ubuntu-trusty images | 17:34 |
| clarkb | we should attempt to upload to them until we succeed, in the meantime the other regions will services that demand | 17:34 |
| *** degorenko is now known as _degorenko|afk | 17:35 | |
| pabelanger | clarkb: have you looked at my nodepool patch for the upload threads yet? | 17:35 |
| *** tonytan4ever has quit IRC | 17:35 | |
| *** sridhar_ram has quit IRC | 17:35 | |
| pabelanger | should give what we need to expose them in nodepool.yaml | 17:35 |
| clarkb | pabelanger: no, that won't address broken glance servers but should help make things go faster | 17:35 |
| clarkb | I will look shortly | 17:35 |
| *** jsavak has quit IRC | 17:35 | |
| *** mylu has quit IRC | 17:35 | |
| pabelanger | clarkb: that is true | 17:35 |
| *** tonytan4ever has joined #openstack-infra | 17:36 | |
| *** aviau_ has quit IRC | 17:36 | |
| *** jsavak has joined #openstack-infra | 17:36 | |
| clarkb | the issue here is 1/10 uploads seems to succeed | 17:36 |
| *** mrmartin has quit IRC | 17:36 | |
| *** aviau has joined #openstack-infra | 17:36 | |
| clarkb | so we end up with a 17 day old image and a 5 day old image | 17:36 |
| mordred | clarkb: did you see I did some manual upload attempts to OVH earlier? | 17:36 |
| *** rcernin has quit IRC | 17:36 | |
| mordred | clarkb: HTTP/1.1 500 Internal Server Error | 17:36 |
| clarkb | mordred: no, but thats the error we are gtting from nodepool iirc | 17:36 |
| mordred | clarkb: is what you get after $time if you do the upload with curl | 17:36 |
| *** sridhar_ram has joined #openstack-infra | 17:36 | |
| clarkb | mordred: maybe you can put that together in an emial to them since your steps don't require nodepool | 17:37 |
| *** tosky_ has quit IRC | 17:37 | |
| clarkb | we don't really have any alien images mucking up the works either | 17:38 |
| *** SumitNaiksatam has joined #openstack-infra | 17:38 | |
| *** jamesmcarthur has joined #openstack-infra | 17:38 | |
| clarkb | rax-ord has a small number and thats about it | 17:38 |
| *** nadya has quit IRC | 17:38 | |
| mordred | clarkb: yah - it's super easy actually | 17:38 |
| mordred | clarkb: I just did a post the file endpoint of the image | 17:38 |
| mordred | clarkb: curl -i -X PUT -H 'X-Auth-Token: REDACTED' -H "Content-Type: application/octet-stream" -d @/opt/nodepool_dib/176676.raw 'https://image.compute.bhs1.cloud.ovh.net/v2/images/f38654fd-1662-426a-8013-7190781fada3/file' | 17:39 |
| mordred | or a put, rather | 17:39 |
| clarkb | mordred: my rought theory is some race between the image create and image put | 17:40 |
| *** ybathia has joined #openstack-infra | 17:40 | |
| clarkb | so api server says "hey ya that image has uuid foo upload to it" but the backends don't learn about it or something | 17:40 |
| *** annegentle has joined #openstack-infra | 17:40 | |
| mordred | yah- I do not believe that to be the case, actually | 17:40 |
| *** pcaruana has quit IRC | 17:40 | |
| mordred | clarkb: I tried the put to the same image several times | 17:40 |
| clarkb | mordred: over a large period of time? | 17:41 |
| mordred | not a _large_ period, but it wasn't terribly small | 17:41 |
| clarkb | ok | 17:41 |
| mordred | I can try again with a larger period of time | 17:41 |
| mordred | I mean, if it is just a larger period of time, it would be easy enough to not recreate the image record on each try | 17:42 |
| Shrews | mordred: is wait() called directly? i thought it was called only by submitTask()? | 17:42 |
| mordred | Shrews: there are other TaskManager implementations | 17:42 |
| mordred | Shrews: see: http://logs.openstack.org/51/226751/77/check/gate-dsvm-nodepool-src-shade/2e92d36/logs/screen-nodepool.txt.gz | 17:43 |
| openstackgerrit | Michael McCune proposed openstack-infra/project-config: make sahara bandit gate voting https://review.openstack.org/297271 | 17:43 |
| *** Nikolay_St has quit IRC | 17:43 | |
| *** eil397 has joined #openstack-infra | 17:44 | |
| clarkb | pabelanger: trying to decide if we should expose those worker flags on the command line | 17:44 |
| clarkb | pabelanger: the nodepool.yaml so far hasn't really been for configuring nodepool internals, just the clouds and iamges | 17:44 |
| *** bpokorny has quit IRC | 17:45 | |
| *** annegentle has quit IRC | 17:45 | |
| clarkb | alredy down to 31 images uploading woo | 17:45 |
| *** EricGonczer_ has joined #openstack-infra | 17:45 | |
| *** nmagnezi has quit IRC | 17:46 | |
| pabelanger | clarkb: I'm open to what ever people want honestly. | 17:47 |
| *** angdraug has quit IRC | 17:47 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Add multinode instances to rax https://review.openstack.org/297273 | 17:49 |
| clarkb | I notice that debian-jessie is only in rax and ovh two of the regions that have upload issues is that for any specific reason or can we add it to all the regions? | 17:49 |
| pabelanger | clarkb: Oh, I thought I had a patch up to add it to all | 17:49 |
| pabelanger | let me check | 17:50 |
| pabelanger | Ya, guess I didn't do that yet | 17:50 |
| *** kzaitsev_mb has quit IRC | 17:50 | |
| *** jamesmcarthur has quit IRC | 17:50 | |
| anteaya | pabelanger: you are still updating jenkinsii, are you not? | 17:50 |
| pabelanger | anteaya: yes, it will be a while longer | 17:50 |
| anteaya | okay thanks | 17:50 |
| pabelanger | clarkb: I can propose the debian-jessie patch, if you haven't started | 17:51 |
| clarkb | pabelanger: I havne't, go for it | 17:51 |
| *** Nikolay_St has joined #openstack-infra | 17:52 | |
| clarkb | I think by default we should do our best to get images into the clouds that don't have image upload problems as it will be easier to work through problems | 17:52 |
| clarkb | I get the reason for starting in rax because it exercises glean but we can do that out of band if we really need to | 17:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add debian-jessie to all cloud providers https://review.openstack.org/297276 | 17:53 |
| *** esikachev has joined #openstack-infra | 17:54 | |
| openstackgerrit | Michael Hoppal proposed openstack-infra/project-config: Make monasca persister tempest tests voting https://review.openstack.org/297278 | 17:54 |
| *** matrohon has quit IRC | 17:54 | |
| *** sfinucan has quit IRC | 17:54 | |
| pabelanger | clarkb: sounds fair | 17:54 |
| pabelanger | we could also do the centos-7 dib too | 17:54 |
| *** jamesmcarthur has joined #openstack-infra | 17:54 | |
| clarkb | these changes are gong to require a builder restart so lets go ahead and do as many as we are comfortable with | 17:55 |
| pabelanger | Ya, pretty sure centos-7 dib is working in other clouds | 17:55 |
| pabelanger | so I can propose that too | 17:55 |
| *** ihrachys has quit IRC | 17:55 | |
| *** mylu has joined #openstack-infra | 17:55 | |
| *** cdent has joined #openstack-infra | 17:57 | |
| clarkb | pabelanger: and hopefully real soon now we can remove devstack-trusty | 17:59 |
| pabelanger | clarkb: I think we can do it over the weekend | 17:59 |
| clarkb | excellent | 18:00 |
| openstackgerrit | Christian Schwede proposed openstack-infra/devstack-gate: Add check for partial Swift upgrade https://review.openstack.org/297281 | 18:00 |
| clarkb | the fix for puppet jjb list things is in the gate finally | 18:00 |
| clarkb | so that should be in soon, it only updates a couple hundred jobs so hopefully no manual jjb intervention necessary | 18:00 |
| AJaeger | I just checked why I missed the db setup for tuning-box - and see that codesearch does not give me results including tuning-box ;( | 18:00 |
| pabelanger | once JJB is done, we shouldn't launch another devstack-trusty. So it should be easy to monitor it | 18:00 |
| AJaeger | does anybody know what's the problem with codesearch | 18:01 |
| *** annegentle has joined #openstack-infra | 18:01 | |
| clarkb | AJaeger: I think we stopped adding new repos to it to avoid restarting it constantly | 18:01 |
| sdague | pabelanger: we are still in project-config hold, right? | 18:01 |
| clarkb | AJaeger: maybe that should be a weekend cron that runs once a week or something | 18:01 |
| pabelanger | clarkb: Ya, JJB is just working on deleting the existings jobs ATM | 18:01 |
| pabelanger | sdague: Ya, for a bit longer | 18:01 |
| elmiko | AJaeger: hey, should i remove the sahara-bandit stuff from the zuul/layout.yaml then? | 18:01 |
| AJaeger | clarkb: yeah, should be... | 18:01 |
| sdague | pabelanger: ok, cool, just checking | 18:01 |
| AJaeger | elmiko: yes, please. But first let's get my change in if you agree with it... | 18:02 |
| elmiko | AJaeger: i do, added a +2 | 18:02 |
| elmiko | but yea, let's make sure the team is ok with it | 18:02 |
| *** salv-orl_ has joined #openstack-infra | 18:02 | |
| AJaeger | elmiko: thanks. keystone and other repos already do it this way. | 18:02 |
| jamesmcarthur | fungi: what are the chances of getting openstackid-dev upgraded to the same size as production openstackid, so that we can start running appropriate load tests on the same environment? | 18:02 |
| clarkb | pabelanger: oh is the fix changes going to cause trouble? | 18:02 |
| clarkb | pabelanger: 297253 | 18:02 |
| elmiko | AJaeger: yea, makes total sense. i'll update my original review and add a depends-on for the one you made | 18:03 |
| * AJaeger disappears again... | 18:03 | |
| pabelanger | clarkb: I don't think so, we have a lock now right? | 18:03 |
| clarkb | pabelanger: maybe? it didn't work, not sure how jjb fixed | 18:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add centos-7 dib to all clouds https://review.openstack.org/297283 | 18:04 |
| *** esker has joined #openstack-infra | 18:04 | |
| *** salv-orlando has quit IRC | 18:05 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix typo with ostype for puppet beaker jobs https://review.openstack.org/297253 | 18:05 |
| openstackgerrit | Christian Schwede proposed openstack-infra/project-config: Add grenade job for partial Swift upgrade tests https://review.openstack.org/297005 | 18:05 |
| *** sambetts is now known as sambetts|afk | 18:05 | |
| clarkb | down to 30 uploads so its moving a bit quicker today, likely due to the disk pace situation being corrected | 18:06 |
| anteaya | I'm having some lunch then offline for a bit, back in a few hours | 18:06 |
| *** sdake_ has joined #openstack-infra | 18:07 | |
| *** ybathia has quit IRC | 18:07 | |
| *** resker has joined #openstack-infra | 18:08 | |
| *** pvaneck has joined #openstack-infra | 18:09 | |
| *** piet has quit IRC | 18:09 | |
| *** sdake has quit IRC | 18:10 | |
| *** shardy has quit IRC | 18:10 | |
| *** esker has quit IRC | 18:11 | |
| *** flepied has quit IRC | 18:11 | |
| *** tqtran has joined #openstack-infra | 18:14 | |
| *** piet has joined #openstack-infra | 18:16 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: update branches reported to #openstack-qa https://review.openstack.org/297290 | 18:17 |
| *** lezbar__ has quit IRC | 18:18 | |
| bswartz | grrrr | 18:18 |
| bswartz | my release job died due to a timeout | 18:18 |
| bswartz | http://logs.openstack.org/bb/bbd1717f15e2d1a05814de264252d083edad3f34/release/gate-manila-publishimage-generic/4e0d5de/console.html | 18:18 |
| bswartz | what am I supposed to do to fix this? | 18:18 |
| bswartz | just push more tags and hope it goes better next time? | 18:19 |
| *** rkukura has quit IRC | 18:19 | |
| notmyname | I'm coordinating getting rolling upgrades tested int eh gate for swift. we've got 4 patches across 4 repos in a dependency chain, ending with https://review.openstack.org/#/c/297005/. the projects are devstack, grenade, devstack-gate, and infra project-config | 18:19 |
| sdague | is anyone looking into the super high lint failure rate with horizon - http://tinyurl.com/h9h4evp | 18:19 |
| *** SumitNaiksatam_ has joined #openstack-infra | 18:19 | |
| *** SumitNaiksatam has quit IRC | 18:19 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 18:19 | |
| notmyname | I'm respectfully asking for reviews on these ASAP to have them landed by next week's TC meeting | 18:20 |
| *** rkukura has joined #openstack-infra | 18:20 | |
| *** ybathia has joined #openstack-infra | 18:20 | |
| *** mriedem_snow is now known as mriedem | 18:20 | |
| sdague | notmyname: we're trying not to do any more partial jobs, the multinode path is the direction we're pointing everyone into for mixed component upgrades | 18:20 |
| notmyname | sdague: is that a hard requirement or something to move to over time? if it's a hard requirement, then can you please leave a comment explaining how that works and what we need to look at there? | 18:21 |
| clarkb | sdague: what? | 18:21 |
| clarkb | sdague: this is exactly the thing partial exists for I thought? | 18:21 |
| clarkb | we really don't need multinode for this | 18:22 |
| clarkb | and would require swift support multinode in the gate which it currently doesn't | 18:22 |
| sdague | the complexity of partial when things go wrong is really bad | 18:22 |
| clarkb | since we only run compute services on the second node currently | 18:22 |
| sdague | partial is completely dropped in mitaka & master. | 18:23 |
| sdague | notmyname: yeh, let me provide some pointers in the reviews | 18:23 |
| pabelanger | bswartz: 2016-03-24 15:38:34.440 | sudo: unable to resolve host ubuntu-trusty-vexxhost-ca-ymq-1-9075535 | 18:23 |
| clarkb | pabelanger: that should be fine | 18:23 |
| pabelanger | that took about 15mins to resolve it looks like | 18:23 |
| clarkb | no thats just a warning that is benign | 18:24 |
| pabelanger | or something after sudo did | 18:24 |
| clarkb | the actual thing sudo is running took too long | 18:24 |
| pabelanger | also looks to be using archive.ubuntu.com | 18:24 |
| clarkb | sdague: I am not sure how not upgrading the proxy on the controller in a two node siutation is any different than what is currently proposed | 18:25 |
| clarkb | sdague: you would have old proxy on controller with new backends on controller and old backends on compute | 18:26 |
| clarkb | we already know old proxy works with old backends so youdon't need to test that | 18:26 |
| clarkb | the interesting interaction is old proxy with new backends (at least that is what is proposed) and that being on the contoller is basically the partial setup not multinode | 18:26 |
| sdague | clarkb: we're really trying to eliminate all that complexity in the devstack setup because the maintenance team is so small. If someone wants to do it as a plugin instead, that's fine. | 18:28 |
| clarkb | what is there to do in devstack? isn't it just "don't restart this service" in grenade? | 18:29 |
| clarkb | which cschwede has written | 18:29 |
| sdague | clarkb: if you would like to take ownership on devstack / grenade, you are welcome to | 18:30 |
| *** SumitNaiksatam_ has joined #openstack-infra | 18:30 | |
| clarkb | I am not trying t ofight ownership, I am just trying to understand what is involved. From what I can see it appears that swift needs an upgrade script in grenade which they have updated (so presumably can care for) and thats it | 18:31 |
| *** SumitNaiksatam has quit IRC | 18:31 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 18:31 | |
| clarkb | there is one small utility function added to devstack but I don't see that being a burden | 18:31 |
| *** tonytan4ever has quit IRC | 18:32 | |
| *** daemontool has quit IRC | 18:32 | |
| *** tonytan4ever has joined #openstack-infra | 18:32 | |
| clarkb | I just don't see how the desired change fits in the multinode case so I don't think it is fair to demand that be the only option | 18:32 |
| *** Sukhdev has joined #openstack-infra | 18:33 | |
| clarkb | at least given the way we run swift in these environments | 18:33 |
| *** tonytan_brb has joined #openstack-infra | 18:33 | |
| openstackgerrit | Merged openstack-infra/shade: Add default value to wait parameter https://review.openstack.org/297262 | 18:34 |
| clarkb | swift is sort of inverted from the infrastructure services in that their api is the thing that needs to remain up and they can update their redundant backends more easily | 18:34 |
| clarkb | whereas compared to compute for example the api upgrade is cheap but long lived cpu resources need to stay up | 18:35 |
| sdague | clarkb: like I said, if you want to take on grenade maintenance, you are welcome to | 18:35 |
| bswartz | pabelanger: so what do you suggest I do? | 18:35 |
| notmyname | (to be fair, we could do the opposite as well, but it was simpler to stop one process instead of 3) | 18:35 |
| bswartz | can the release job be resubmited without pushing another tag? | 18:35 |
| sdague | right, so the prefered model is really the multinode one | 18:35 |
| sdague | the way that works is that you setup things in the subnode that you want to not be upgraded | 18:36 |
| sdague | then everything on the master node gets upgraded | 18:36 |
| sdague | and they all happily talk to each other | 18:36 |
| *** gordc has joined #openstack-infra | 18:36 | |
| pabelanger | bswartz: I'll have to check with clarkb but think we can | 18:36 |
| clarkb | pabelanger: yes you can directly enqueue the job to the gearman queue | 18:37 |
| clarkb | pabelanger: or you can zuul-enqueue it | 18:37 |
| clarkb | pabelanger: zuul-enqueue will run all the jobs in that pipeline for the tag, direct gearman queue is per job | 18:37 |
| *** tonytan4ever has quit IRC | 18:37 | |
| pabelanger | bswartz: which job change was it again | 18:37 |
| *** tonytan_brb has quit IRC | 18:37 | |
| pabelanger | clarkb: ok, let me see if I can do it | 18:38 |
| clarkb | notmyname: I think the important thing to test is a typical rolling upgrade in a real env | 18:38 |
| *** tonytan4ever has joined #openstack-infra | 18:38 | |
| clarkb | notmyname: for nova, all the compute services remain up while the control plane is upgraded because running compute resources are not fun to take down | 18:38 |
| notmyname | clarkb: both are supported and typical-ish. | 18:38 |
| *** Sukhdev has quit IRC | 18:39 | |
| notmyname | and if the currently proposed patches land, then there are several follow-ups we can work on that will expand the number of combinations of different versions | 18:39 |
| *** vgridnev has quit IRC | 18:40 | |
| clarkb | sdagues suggestion is to get swift running on two machines using the multinode setup, then one of the machines (what is now the controller) is upgraded while the compute host is not | 18:40 |
| *** sridhar_ram1 has joined #openstack-infra | 18:40 | |
| clarkb | we do not currently run swift on both machines so that will need to be figured out | 18:40 |
| *** brad_behle has quit IRC | 18:41 | |
| notmyname | aside from "compute host" and "controller" being totally meaningless words in a swift deployment, yes, that will test the exact same thing as the currently proposed patches | 18:41 |
| sdague | right, it's not really compute host and controller | 18:41 |
| *** sridhar_ram has quit IRC | 18:41 | |
| *** oanson has joined #openstack-infra | 18:41 | |
| sdague | it's master and subnode | 18:41 |
| sdague | it's even the way that exists in the code | 18:41 |
| clarkb | right thats just how they are configred today | 18:41 |
| clarkb | its two arbitrary machines that cna be configured | 18:42 |
| sdague | right | 18:42 |
| pabelanger | EmilienM: so even though JJB is still running, I think your puppet-jobs are all updated | 18:42 |
| EmilienM | excellent, I'm testing now with 297265 | 18:42 |
| pabelanger | EmilienM: so, any failures now we can look into as new | 18:42 |
| EmilienM | good to know | 18:42 |
| notmyname | sdague: are there instructions anywhere for setting this up? | 18:43 |
| EmilienM | pabelanger: thanks for your work here | 18:43 |
| pabelanger | EmilienM: don't thank me until everything is working :D | 18:43 |
| *** asettle has joined #openstack-infra | 18:43 | |
| *** HeOS has quit IRC | 18:44 | |
| sdague | notmyname: they aren't great, we kind of assumed that we'd see folks coming and chatting in #openstack-qa or asking on the mailing list. I left a couple of pointers in the first review. But let me see if I can push up a sketch review that points out where things would need to change. | 18:44 |
| notmyname | sdague: also, if this is what you are requiring, please add a -2 to the existing patches, ideally with some guidance on what's needed to make you happy | 18:45 |
| *** mylu has quit IRC | 18:45 | |
| sdague | notmyname: how about I spend the next 15 minutes sketching an etherpad | 18:45 |
| notmyname | sdague: sure, but this is somethign that has really unfortunate timing that's accelerated the process of getting something that works | 18:45 |
| sdague | notmyname: sure, I can also WIP the TC review to give you more time | 18:46 |
| sdague | because I assume the urgency is not to have the tag drop | 18:46 |
| *** mylu has joined #openstack-infra | 18:46 | |
| *** Guest27982 has quit IRC | 18:46 | |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone eventlet job on stable branches only https://review.openstack.org/297309 | 18:46 |
| *** piet has quit IRC | 18:47 | |
| *** asettle has quit IRC | 18:48 | |
| *** gokrokve has quit IRC | 18:48 | |
| *** mtanino has joined #openstack-infra | 18:48 | |
| bswartz | pabelanger: bbd1717f15e2d1a05814de264252d083edad3f34/ | 18:48 |
| bswartz | err | 18:49 |
| sdague | notmyname: actually... what services would you want in the subnode? | 18:49 |
| pabelanger | clarkb: nothing has used devstack-trusty in almost 4 hours | 18:49 |
| bswartz | pabelanger: bbd1717f15e2d1a05814de264252d083edad3f34 | 18:49 |
| sdague | s-container,s-object ? | 18:49 |
| pabelanger | bswartz: i'll need the review from review.o.o | 18:49 |
| notmyname | sdague: the two "sets" would be (account,container,object) and (proxy) | 18:49 |
| sdague | ok, what happens if you have object at 2 different versions on 2 different nodes? | 18:50 |
| bswartz | pabelanger: it was a push of a tag | 18:50 |
| bswartz | that's why it ran in the release pipe | 18:50 |
| notmyname | sdague: what happens is that everything works | 18:50 |
| pabelanger | bswartz: Ah, right | 18:51 |
| pabelanger | bswartz: give me a few minutes | 18:51 |
| *** flepied has joined #openstack-infra | 18:51 | |
| sdague | ok, so if we do (proxy,account,container,object) upgraded on master and (account,container,object) not upgraded on subnode, all should be well? | 18:51 |
| bswartz | pabelanger: the project was openstack/manila-image-elements and the tag was 1.2.0 | 18:52 |
| notmyname | sdague: yes | 18:52 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone eventlet job on stable branches only https://review.openstack.org/297309 | 18:52 |
| notmyname | sdague: a few years ago I wrote down "normal-ish" steps for upgrading swift clusters. nothing's changed. https://swiftstack.com/blog/2013/12/20/upgrade-openstack-swift-no-downtime/ | 18:53 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: Run swift services on subnode https://review.openstack.org/297311 | 18:53 |
| *** achanda has quit IRC | 18:53 | |
| notmyname | sdague: most clusters either have everything running everywhere or proxies distinct from storage nodes. all combinations for upgrades work | 18:53 |
| sdague | ok, so let's see if it's as simple as that - https://review.openstack.org/297311 | 18:54 |
| notmyname | sdague: that's it? what does that do? is this in-lieu of cschwede's 4 patches? | 18:56 |
| sdague | it means that for the existing gate-grenade-dsvm-multinode those 3 additional services would be run on the subnode | 18:57 |
| sdague | normal swift upgrade will happen on the master of all the services | 18:57 |
| *** electrofelix has quit IRC | 18:57 | |
| sdague | notmyname: if that does all the expected things, the only thing that remains is putting gate-grenade-dsvm-multinode on swifts check/gate | 18:58 |
| notmyname | ah, ok. so we also need to add grenade multinote...right | 18:58 |
| notmyname | grenade is running tempest tests, right? | 18:58 |
| sdague | notmyname: yep | 18:58 |
| pabelanger | clarkb: how do I find the change Id for a tag in the release queue? | 18:58 |
| *** mylu has quit IRC | 18:58 | |
| sdague | the smoke tests | 18:58 |
| sdague | but there are some swift tests in there | 18:58 |
| clarkb | pabelanger: there won't be a changeid, instead you get the sha1 of the tag itself | 18:58 |
| clarkb | pabelanger: git show-ref tagname I think | 18:59 |
| pabelanger | k | 18:59 |
| sdague | notmyname: once the test results get back from that, we can figure out if it just works, or what the next steps have to be to make it work | 18:59 |
| notmyname | sdague: so the next question, though, is how those are tied together? eg is the ring updated to include the services on the subnodes? | 18:59 |
| pabelanger | clarkb: so I guess bbd1717f15e2d1a05814de264252d083edad3f34,1 ? | 19:00 |
| sdague | notmyname: that's a question that swift folks would have to answer, I don't know what other things need to be considered in this case | 19:00 |
| notmyname | maybe it just takes the IP:port for every node that has -saccount enabled and adds it to the right? | 19:00 |
| notmyname | s/right/ring | 19:00 |
| clarkb | pabelanger: you don't provide a patchset its not in a change context | 19:00 |
| sdague | notmyname: I'm not convinced that code exists today | 19:00 |
| *** mylu has joined #openstack-infra | 19:00 | |
| notmyname | sdague: where would that code live? | 19:01 |
| pabelanger | clarkb: ah, enqueue-ref | 19:01 |
| clarkb | pabelanger: zuul enqueue-ref | 19:01 |
| clarkb | ya that | 19:01 |
| EmilienM | pabelanger: it takes quite some time to get a ubuntu-trusty node | 19:01 |
| *** esikachev has quit IRC | 19:01 | |
| sdague | notmyname: when would that normally need to happen in a swift install? | 19:01 |
| *** piet has joined #openstack-infra | 19:02 | |
| sdague | then I can point you at where it probably should get added | 19:02 |
| *** Kennan has quit IRC | 19:02 | |
| clarkb | EmilienM: there are about 250 of them floating around in one state or another | 19:03 |
| *** Swami has quit IRC | 19:03 | |
| notmyname | sdague: it's an initial setup step. it's functionally similar to a config file | 19:03 |
| *** kzaitsev_mb has joined #openstack-infra | 19:03 | |
| pabelanger | clarkb: so does this look about right: sudo zuul enqueue-ref --trigger gerrit --pipeline release --project openstack/manila-image-elements --ref refs/tags/1.2.0 --newrev bbd1717f15e2d1a05814de264252d083edad3f34 | 19:03 |
| *** Kennan has joined #openstack-infra | 19:03 | |
| pabelanger | where newrev is the existing sha1 of the tag | 19:03 |
| pabelanger | docs aren't very clear | 19:04 |
| sdague | notmyname: before or after the services are up? | 19:04 |
| pabelanger | EmilienM: ya, we are down 2 regions I think. Working on new image uploads | 19:04 |
| *** cody-somerville has quit IRC | 19:04 | |
| notmyname | doesn't matter. but it's what let's the proxy know what storage nodes to talk to | 19:04 |
| EmilienM | pabelanger: ok, cool | 19:04 |
| *** rcrit_ has joined #openstack-infra | 19:04 | |
| clarkb | pabelanger: yup that looks about right, it will rerun all of the release pipeline jobs which may not be desireable if it sends email or uploads to pypi | 19:04 |
| sdague | like is a thing you can put in swift etc files before any code starts, or a command that needs to run once some things are up? | 19:04 |
| clarkb | pabelanger: if that isn't a problem you can run that, if it is a problem you should queue to gearman directly I can dig up the command for that its in zuul/tools | 19:05 |
| notmyname | sdague: in this case, it's a command that should be run before the service comes up. if you're familiar with ceph, then it's similar to building the crush map | 19:05 |
| pabelanger | clarkb: Ya, its just a single job ATM | 19:05 |
| *** mylu has quit IRC | 19:05 | |
| rcrit_ | would there be any objections if I changed SERVICE_HOST from 127.0.0.1 to localhost in devstack-vm-gate.sh? Was that done to keep things on IPv4? | 19:05 |
| clarkb | pabelanger: then ya should be fine | 19:05 |
| pabelanger | bswartz: re-queued. check status.o.o for more info | 19:06 |
| *** mylu has joined #openstack-infra | 19:06 | |
| clarkb | rcrit_: I think it was done because devstack's interface selection logic can get confused at times? | 19:06 |
| clarkb | rcrit_: you can probably check the git log | 19:06 |
| pabelanger | clarkb: command worked, thanks | 19:06 |
| thomasem | Hmmm, well now that we're using Fedora 23 for LXC gate it's even worse. I wonder how my local simulations differed. | 19:06 |
| *** annegentle has quit IRC | 19:06 | |
| *** daemontool has joined #openstack-infra | 19:07 | |
| *** vgridnev has joined #openstack-infra | 19:07 | |
| rcrit_ | I can look again | 19:07 |
| *** ybathia has quit IRC | 19:07 | |
| *** mylu has quit IRC | 19:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Set devstack-trusty to 0 https://review.openstack.org/297317 | 19:10 |
| *** mylu has joined #openstack-infra | 19:10 | |
| pabelanger | infra-root: we can likely land ^ in the next few days | 19:10 |
| clarkb | pabelanger: EmilienM does ^ mean puppet jobs are happy now? | 19:10 |
| pabelanger | that will zero out our devstack-trusty | 19:10 |
| EmilienM | still not started AFIK | 19:10 |
| pabelanger | clarkb: almost | 19:10 |
| pabelanger | setting it WIP | 19:10 |
| EmilienM | still waiting to get a ubuntu-trusty node | 19:10 |
| clarkb | looks like a puppet-trove job may have used it? | 19:11 |
| rcrit_ | AFAICT it has always been this way, SERVICE_HOST=127.0.0.1 | 19:11 |
| rcrit_ | the TLS testing isn't working so hot with an IP value | 19:12 |
| *** ybathia has joined #openstack-infra | 19:12 | |
| sdague | notmyname: ok my guess is https://github.com/openstack-dev/devstack/blob/74d82d9343431a53bcffd6554b6188278e61e0d9/lib/swift#L678-L709 is going to need changes to support specifying a subnode, today it only really seems to have a mode where it does some local multi replica thing | 19:12 |
| clarkb | its self testing :) you can switch to localhost and see what happens | 19:12 |
| clarkb | my guess its localhost is ok as long as all of the bind machinery knows to translate | 19:12 |
| pleia2 | no more meetings this week \o/ | 19:12 |
| pleia2 | (though I should get lunch soon) | 19:13 |
| AJaeger | pabelanger: can't we land 297317 directly? We can remove devstack-trusty then in a week... | 19:13 |
| AJaeger | pabelanger: 0 would still allow to run those jobs. -1 means: Do not build new nodes. | 19:13 |
| clarkb | AJaeger: pabelanger as long as its 0 I think it will be ok if demand comes in | 19:13 |
| notmyname | sdague: ah, so maybe similar to cschwede's patches, some env var would be set then this script would trigger on that? are there any variables available to that script that have info about the different nodes? | 19:14 |
| AJaeger | clarkb: agreed, so no need to WIP, we can merge... | 19:14 |
| AJaeger | ... once pabelanger allows to merge changes for project-config again ;) | 19:14 |
| openstackgerrit | Michael Hoppal proposed openstack-infra/project-config: Make monasca persister tempest tests voting https://review.openstack.org/297278 | 19:14 |
| notmyname | are the different nodes in a multinode setup different VMs? | 19:15 |
| notmyname | (perhaps they're just different venvs or something) | 19:15 |
| clarkb | yup, they are in the same cloud region but are distinct VMs | 19:15 |
| *** oanson has quit IRC | 19:15 | |
| notmyname | thanks | 19:15 |
| *** bpokorny has joined #openstack-infra | 19:15 | |
| pabelanger | AJaeger: All most | 19:16 |
| bswartz | pabelanger: SUCCESS! | 19:16 |
| bswartz | pabelanger: thanks | 19:16 |
| pabelanger | bswartz: np | 19:16 |
| pabelanger | AJaeger: clarkb Oh, I can -1 if you think that is better | 19:17 |
| pabelanger | same with devstack-trusty multinode | 19:17 |
| pabelanger | okay, jenkins03 happy | 19:17 |
| AJaeger | pabelanger: we can do 0 now - and remove completely as next step as well... | 19:17 |
| *** tonytan_brb has joined #openstack-infra | 19:18 | |
| fungi | wow, 300 lines of scrollback at lunch. i feel like i'm paraphrasing a h²g² line | 19:18 |
| pabelanger | AJaeger: okay, so I can remove my WIP then | 19:18 |
| fungi | catching up now | 19:18 |
| *** boltR has joined #openstack-infra | 19:18 | |
| sdague | notmyname: there aren't right now because in a lot of the other services it works dynamically over rabbit. https://etherpad.openstack.org/p/swift-rolling-upgrade-multinode-testing is the best guess I have at the way to do this | 19:18 |
| *** baoli has quit IRC | 19:18 | |
| sdague | with pointers to specific code chunks that are relevant | 19:18 |
| boltR | does anyone know what would happen if a job template has the same name as a job? | 19:19 |
| boltR | eg. check-{project}-requirements and check-cinder-requirements | 19:19 |
| fungi | jamesmcarthur: i'm happy to boot a 15gb replacement openstackid-dev right now, just let me catch up on scrollback first | 19:19 |
| notmyname | sdague: thank you. I'll look into that | 19:19 |
| clarkb | boltR: yes yaml parse rules come into play and one of them wins (the one parsed first iirc) | 19:19 |
| fungi | (in case someone else is already working on it) | 19:19 |
| clarkb | boltR: tl;dr don't do it | 19:19 |
| boltR | clarkb: ah I see, i was hoping to do some kind of override | 19:19 |
| *** cody-somerville has joined #openstack-infra | 19:20 | |
| sdague | notmyname: I also put a hold on the TC resolution. The intent was not to kick swift in the nuts, I was just cleaning up some things. The swift team is clearly actually working on this, so we should hold until you all get things sorted. | 19:20 |
| openstackgerrit | Michael McCune proposed openstack-infra/project-config: remove sahara-bandit job https://review.openstack.org/297271 | 19:20 |
| notmyname | sdague: thank you | 19:20 |
| pabelanger | EmilienM: keep in mind, jenkins02 is still out of sync. So, if you see failures on it, I would recheck for another manter | 19:21 |
| *** tonytan4ever has quit IRC | 19:21 | |
| pabelanger | master* | 19:21 |
| EmilienM | pabelanger: ok | 19:21 |
| EmilienM | pabelanger: still waiting for nodes :( | 19:21 |
| *** angdraug has joined #openstack-infra | 19:21 | |
| *** rhallisey has quit IRC | 19:22 | |
| jamesmcarthur | fungi: that’s great. in order to proceed with testing on dev, and to be totally thorough for apache and nginx comparison we’d like to try both. Do you think you could add nginx to the allowable puppet modules - we’ve used this one before jfryman-nginx? Would you be able to write a patch for that so it will get processed faster? | 19:22 |
| pabelanger | jenkins04 is happy | 19:23 |
| *** ihrachys has joined #openstack-infra | 19:23 | |
| pabelanger | and jenkins06 | 19:23 |
| fungi | jamesmcarthur: i'll take a look. i'm not as puppet-savvy as some on the team, but as a start let me go ahead and get the replacement machine booting and then i'll look | 19:23 |
| *** achanda has joined #openstack-infra | 19:24 | |
| jamesmcarthur | thanks sir. very much appreciated. | 19:24 |
| AJaeger | pabelanger: just confirmed that we have no jobs that require devstack-trusty, so that looks fine. | 19:24 |
| *** ybathia has quit IRC | 19:24 | |
| pabelanger | AJaeger: \o/ | 19:24 |
| *** daemontool has quit IRC | 19:25 | |
| *** rhallisey has joined #openstack-infra | 19:25 | |
| clarkb | oh hrm bare-precise might be unhappy with our iptables updates | 19:26 |
| *** mrmartin has joined #openstack-infra | 19:26 | |
| clarkb | which is odd because it should have the openstack-INPUT chain | 19:26 |
| * AJaeger also checked all other node setups and didn't find anything odd, so looks great! | 19:26 | |
| clarkb | pabelanger: fungi looks like bluebox has leaked fips | 19:26 |
| clarkb | pabelanger: I can walk you through cleaning those up if you want | 19:27 |
| pabelanger | clarkb: sure | 19:27 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished https://review.openstack.org/297328 | 19:27 |
| clarkb | pabelanger: source the openstackjenkins-bluebox-somethinorother rc file on puppetmaster | 19:28 |
| *** yamahata has quit IRC | 19:28 | |
| clarkb | I don't use openstackclient for this yet just becaues I don't have the table parse stuff sorted out for its output yet but in theory it will work. Instead I have neutronclient in a venv | 19:28 |
| *** baoli has joined #openstack-infra | 19:29 | |
| clarkb | pabelanger: I run `venv/bin/neutron floatingip-list | grep -v '10\.220\.' | sed -e '1,3d' -e '$d' | cut -d'|' -f 2` to get the uuids of the floating ips that are not associated to any hosts | 19:29 |
| clarkb | save that to a file, and rerun about five minutes later saving to a file. Any uuids that show up in a diff between the two files you don't delete because that means they ended up attached to a host, the rest you delete | 19:30 |
| pabelanger | clarkb: where do I find the rc file? never used it before | 19:30 |
| clarkb | pabelanger: oh one sec | 19:30 |
| clarkb | `cat second-fips.txt | xargs -P 1 -n 1 venv/bin/neutron floatingip-delete` is roughly how I delete | 19:30 |
| clarkb | this probably should just be a script already | 19:30 |
| pabelanger | ok | 19:30 |
| clarkb | pabelanger: source ~root/ci-launch/openstackjenkins-bluebox-sjc1.sh | 19:31 |
| fungi | i have a much more terrible one-liner than that | 19:31 |
| fungi | well, two-liner | 19:31 |
| dtroyer | clarkb: -f json or -f csv is your friend for osc output | 19:31 |
| *** MarkAtwood has quit IRC | 19:31 | |
| fungi | . ~root/ci-launch/openstackjenkins-bluebox-sjc1.sh | 19:31 |
| fungi | nova floating-ip-list | grep ' - ' | cut -d' ' -f4 | sort > fip-cruft-1 ; sleep 300 ; nova floating-ip-list | grep ' - ' | cut -d' ' -f4 | sort > fip-cruft-2 ; diff -U1000 fip-cruft-{1,2} | grep '^ ' > fip-cruft ; wc -l fip-cruft{,-1,-2} | 19:31 |
| fungi | for i in `cat fip-cruft`;do nova floating-ip-delete $i;done | 19:31 |
| fungi | i'm making dtroyer cringe | 19:31 |
| clarkb | fungi: oh don't use novaclient its slower | 19:31 |
| fungi | yeah, i expect so | 19:32 |
| *** mrmartin has quit IRC | 19:32 | |
| clarkb | because it has to proxy all of the requests | 19:32 |
| fungi | right, i'm updating my notes as we speak ;) | 19:32 |
| *** mrmartin has joined #openstack-infra | 19:32 | |
| dtroyer | devstack does have some useful functions for parsing prettytable output | 19:33 |
| clarkb | fungi: I like how our things theoretically do the same thing but use cut and sed differently :) | 19:33 |
| fungi | artistic license | 19:33 |
| *** jsavak has quit IRC | 19:34 | |
| *** davidlenwell has quit IRC | 19:35 | |
| pabelanger | clarkb: okay, pass 1 done | 19:35 |
| pabelanger | waiting | 19:36 |
| *** Swami has joined #openstack-infra | 19:37 | |
| *** piet has quit IRC | 19:37 | |
| *** MarkAtwood has joined #openstack-infra | 19:38 | |
| *** jsavak has joined #openstack-infra | 19:38 | |
| *** davidlenwell has joined #openstack-infra | 19:38 | |
| *** dalgaaf has quit IRC | 19:38 | |
| *** jamesmcarthur has quit IRC | 19:39 | |
| *** jamesmcarthur has joined #openstack-infra | 19:40 | |
| *** asettle has joined #openstack-infra | 19:40 | |
| *** EricGonczer_ has quit IRC | 19:40 | |
| *** ociuhandu has quit IRC | 19:41 | |
| *** e0ne has joined #openstack-infra | 19:41 | |
| *** baoli has quit IRC | 19:41 | |
| pabelanger | fungi: clarkb: so, no diff. Which means I haven't waited long enough? | 19:42 |
| clarkb | pabelanger: no diff means all of those uuids are leaked | 19:43 |
| clarkb | pabelanger: so you can delete them | 19:43 |
| pabelanger | ok | 19:43 |
| *** baoli has joined #openstack-infra | 19:43 | |
| clarkb | pabelanger: all the uuids that result from running the command I pasted | 19:43 |
| mordred | Shrews: ok. I'm tagging 1.6.1 to pick up that fix | 19:43 |
| clarkb | there were a small number that were attached but those are filtered out by my command above | 19:43 |
| pabelanger | clarkb: yup | 19:43 |
| clarkb | pabelanger: bluebox uses 10.220.x.y internal addrs which is where that grep -v comes from | 19:43 |
| fungi | pabelanger: yeah, the only reason i generate two lists some time apart is that i have seen them get cleaned up on their own, so if there's a diff then it contains things not getting cleaned up in a timely manner | 19:44 |
| clarkb | pabelanger: other clouds with fips like osic use other ranges but other than that its safe to reuse this system across other clouds | 19:44 |
| *** nadya has joined #openstack-infra | 19:44 | |
| Shrews | mordred: ok | 19:44 |
| fungi | if you just generate one list, then it's a poor indicator of whether there's a leak happening, since they may just be in a transitional state | 19:44 |
| Shrews | mordred: dang, we should have added a reno note for that :( | 19:45 |
| mordred | Shrews: I thought about that right after I pushed the tag | 19:45 |
| Shrews | mordred: we suck | 19:45 |
| *** asettle has quit IRC | 19:45 | |
| pabelanger | clarkb: fungi: okay, all deleted | 19:45 |
| mtreinish | mordred: do you remember the function you recommend I use on: https://review.openstack.org/#/c/285615/1 | 19:46 |
| mtreinish | because I can't :) | 19:46 |
| clarkb | pabelanger: Forbidden: IP allocation over quota in pool GATEWAY_NET. (HTTP 403) (Request-ID: req-42ddafcf-2dc1-4976-afd1-b5b6b3557ddf) from osic | 19:46 |
| clarkb | pabelanger: so you might want to try the same trick there too | 19:46 |
| *** mylu has quit IRC | 19:47 | |
| clarkb | pabelanger: just double check the internal address with a bare floating ip list first | 19:47 |
| fungi | jamesmcarthur: smarcet: jpmaxman: there's a replacement 15gb openstackid-dev booted at 2001:4800:7817:102:be76:4eff:fe06:495d and 23.253.97.175 now. let me know if you want me to switch dns over immediately or hold off | 19:47 |
| pabelanger | clarkb: sure, let me try | 19:47 |
| clarkb | I think its 10.0.1.0/24 | 19:47 |
| clarkb | pabelanger: there should be an osic cloud thing to source in that same dir | 19:47 |
| clarkb | assuming I did that correctly | 19:47 |
| clarkb | verp apparently I failed at that | 19:48 |
| *** baoli has quit IRC | 19:48 | |
| pabelanger | ya, don't see one currently | 19:48 |
| clarkb | give me a minute | 19:48 |
| pabelanger | infra-root: AJaeger: sdague: we are over the hump with JJB. Feel free to merge JJB changes again | 19:50 |
| fungi | jamesmcarthur: smarcet: jpmaxman: i've set the ttl on the a and aaaa resource records for openstackid-dev.openstack.org down to 5 minutes to make us a little more agile with swapping addresses around for this too | 19:50 |
| *** ociuhandu has joined #openstack-infra | 19:50 | |
| mordred | pabelanger: woot! | 19:51 |
| jamesmcarthur | fungi: +1. I think we are ok to switch openstackid-dev, but double checking with smarcet: | 19:51 |
| clarkb | pabelanger: its there and works now | 19:51 |
| clarkb | pabelanger: and you can confirm that 10.0.1 is the ip prefix to exclude in the grep -v | 19:52 |
| pabelanger | clarkb: great, running command now | 19:52 |
| *** EricGonczer_ has joined #openstack-infra | 19:54 | |
| AJaeger | thanks, pabelanger for taking care of jenkins! | 19:54 |
| pabelanger | AJaeger: np | 19:54 |
| *** achanda has quit IRC | 19:55 | |
| AJaeger | there's one open review for python-db jobs that needs another +2, could a project-config core review, pelase? https://review.openstack.org/#/c/297236/ | 19:56 |
| pabelanger | done | 19:56 |
| clarkb | once osic is happy I need to pop out and run errands | 19:57 |
| AJaeger | thanks, pabelanger | 19:57 |
| *** rguillebert has quit IRC | 19:57 | |
| *** mylu has joined #openstack-infra | 19:58 | |
| clarkb | pabelanger: are they deleting yet? | 19:58 |
| *** andymaier has quit IRC | 19:58 | |
| pabelanger | clarkb: no, waiting another min to diff | 19:58 |
| pabelanger | I can do that now | 19:59 |
| clarkb | its fine, was just basing on timestamps | 19:59 |
| clarkb | er irc timestamps | 19:59 |
| pabelanger | ya, no differ here too | 19:59 |
| pabelanger | 59 leaked IPs I guess | 19:59 |
| harlowja_at_home | AJaeger, i think https://review.openstack.org/#/c/295437/ might be having a side-effect in that it seems like in https://review.openstack.org/#/c/297275/ that the pypy stuff is now voting | 20:00 |
| harlowja_at_home | gate-taskflow-pypy-db in ^ | 20:00 |
| *** mylu has quit IRC | 20:01 | |
| pabelanger | clarkb: okay, deleted | 20:01 |
| harlowja_at_home | my gut feeling is that most pypy stuff won't work well if its voting | 20:01 |
| AJaeger | pabelanger: could you review https://review.openstack.org/#/c/297225 as well? Seems the release team would like to have that in... | 20:01 |
| AJaeger | harlowja_at_home: let me check | 20:01 |
| harlowja_at_home | thx | 20:02 |
| clarkb | mordred: pabelanger http://paste.openstack.org/show/491795/ may need to rethink the rabbit thing | 20:02 |
| clarkb | :( | 20:02 |
| clarkb | I have to head out though so can't pick that up | 20:02 |
| *** davidlenwell has quit IRC | 20:02 | |
| AJaeger | harlowja_at_home: you're right. Do you want to fix it or shall I? | 20:02 |
| AJaeger | harlowja_at_home: sorry for the breakage ;( | 20:02 |
| mordred | clarkb: HEADDESK | 20:03 |
| *** mylu has joined #openstack-infra | 20:03 | |
| clarkb | mordred: I have no idea why that image does not have that chain | 20:03 |
| clarkb | because all our images should have it | 20:03 |
| clarkb | best guess is something to do wtih firewalld | 20:03 |
| fungi | clarkb: huh... wow | 20:03 |
| fungi | yeah, we create that chain with puppet | 20:04 |
| openstackgerrit | Merged openstack-infra/project-config: Add debian-jessie to all cloud providers https://review.openstack.org/297276 | 20:04 |
| *** _ari_ has quit IRC | 20:04 | |
| AJaeger | project-config cores, seems dims' changes for oslo.mesaging had some unwanted side effects, I suggest to merge https://review.openstack.org/#/c/296781 soonish as well... | 20:04 |
| fungi | clarkb: does it not like that the target is omitted? | 20:05 |
| dims | thanks AJaeger | 20:05 |
| dims | ETOOMANYJOBS :) | 20:05 |
| *** amrith is now known as _amrith_ | 20:06 | |
| *** gyee has joined #openstack-infra | 20:06 | |
| fungi | AJaeger: pabelanger: safe to approve new job updates at this point, yeah? | 20:07 |
| pabelanger | fungi: Yup, we're in the clear | 20:07 |
| fungi | excellent--thanks! | 20:07 |
| fungi | (because i just did) | 20:07 |
| *** sdake has joined #openstack-infra | 20:07 | |
| pabelanger | I am still watching jenkins02 | 20:07 |
| *** sdake_ has quit IRC | 20:07 | |
| pabelanger | it was in some bad shape | 20:07 |
| fungi | it's regularly in much bad shape | 20:08 |
| *** oanson has joined #openstack-infra | 20:08 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Another ssh grenade failure fingerprint https://review.openstack.org/296854 | 20:08 |
| AJaeger | harlowja_at_home: I'll write a patch now... | 20:09 |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Create ACLs for ironic-lib stable branches https://review.openstack.org/297345 | 20:10 |
| jroll | ^ easy acl review I just realized was missing | 20:10 |
| *** sdake_ has joined #openstack-infra | 20:10 | |
| *** baoli has joined #openstack-infra | 20:10 | |
| jpmaxman | Thanks a lot fungi ! | 20:10 |
| jpmaxman | Fungi curious if you have any idea of timing for this https://review.openstack.org/#/c/297251/ | 20:11 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add release notes job for Kolla repo https://review.openstack.org/296129 | 20:11 |
| *** sdake has quit IRC | 20:12 | |
| *** davidlenwell has joined #openstack-infra | 20:12 | |
| EmilienM | pabelanger: https://jenkins05.openstack.org/job/gate-puppet-nova-puppet-beaker-rspec-ubuntu-trusty/1/console | 20:13 |
| EmilienM | let's see | 20:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Make pypy-db jobs also non-voting https://review.openstack.org/297349 | 20:14 |
| AJaeger | harlowja_at_home: this should fix the voting pypy-db jobs. project-config cores, could you review this today, please? ^ | 20:14 |
| fungi | jpmaxman: hadn't seen it until now, i guess translate-dev is authenticating okay to openstackid-dev with that state from the master branch? | 20:14 |
| smarcet | fungi: yes that is correct | 20:14 |
| fungi | my only real concern with openstackid upgrades is disrupting translator activity at the moment since theyre deep in their pre-release crunch | 20:14 |
| openstackgerrit | Merged openstack-infra/project-config: tuning-box: Use db setup for python jobs https://review.openstack.org/297088 | 20:14 |
| pabelanger | EmilienM: good, they all appear not to be using jenkins02 | 20:15 |
| jpmaxman | It's mostly ux changes for better mobile views | 20:15 |
| fungi | thanks smarcet, jpmaxman. lgtm then | 20:15 |
| fungi | yeah, had a look at the release notes in the dag message and skimmed commit messages from 1.0.16 to 1.0.17 | 20:16 |
| fungi | s/dag/tag/ | 20:16 |
| clarkb | fungi uts the exact same rule add muktinode has used forever | 20:16 |
| clarkb | fungi: maybe different versions of iptables care? | 20:16 |
| *** Swami_ has joined #openstack-infra | 20:16 | |
| fungi | clarkb: but suddenly started failing in osic only? | 20:16 |
| *** EricGonczer_ has quit IRC | 20:16 | |
| clarkb | fungi: we moved it to all hosts yesterday to fix rabbitmq | 20:17 |
| fungi | or is this everywhere now/ | 20:17 |
| AJaeger | sdake_: I'm sorry, you previous check was right. I missed that you publish already ;( | 20:17 |
| clarkb | another possibility is we are racing iptables at boot | 20:17 |
| AJaeger | sdake_: change I mean... | 20:17 |
| *** EricGonczer_ has joined #openstack-infra | 20:17 | |
| clarkb | if ssh is accepting before iptables applies | 20:17 |
| fungi | clarkb: oh, you removed the destination from the rule? | 20:17 |
| sdake_ | ok will rever thanks ;) | 20:17 |
| *** Swami__ has joined #openstack-infra | 20:17 | |
| *** jsavak has quit IRC | 20:17 | |
| clarkb | fungi: no just applied that loop to all hosts | 20:17 |
| EmilienM | pabelanger: it works fine! | 20:17 |
| clarkb | rather than only multinode | 20:18 |
| EmilienM | pabelanger: I'll abandon all my patches that renamed the nodeset | 20:18 |
| clarkb | because rabbitmq is silly | 20:18 |
| AJaeger | sdake_: will you? Ok, was just going to do it myself - so will wait. Sorry! | 20:18 |
| fungi | clarkb: now i'm on the same page, okay | 20:18 |
| openstackgerrit | Merged openstack-infra/nodepool: Log shade inner exceptions https://review.openstack.org/225361 | 20:18 |
| *** Guest39452 has quit IRC | 20:18 | |
| fungi | clarkb: so it's working for multinode ready scripts but not single node? | 20:18 |
| sdake_ | AJaeger its ok dude yo urock at rreviews | 20:18 |
| pabelanger | EmilienM: Excellent | 20:18 |
| sdake_ | i never see you make a mistake | 20:18 |
| clarkb | fungi: ya | 20:18 |
| sdake_ | so I thought I was the one in error :) | 20:18 |
| *** Hal has joined #openstack-infra | 20:18 | |
| sdake_ | which is why I didn't even argue :) | 20:18 |
| clarkb | has workedon multinode forever | 20:18 |
| AJaeger | sdake_: I'm human ;) | 20:19 |
| *** Swami has quit IRC | 20:19 | |
| *** zhiyan has quit IRC | 20:19 | |
| *** vmbrasseur has quit IRC | 20:19 | |
| *** simonmcc has quit IRC | 20:19 | |
| *** Hal is now known as Guest34499 | 20:19 | |
| *** fkautz has quit IRC | 20:19 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add release notes job for Kolla repo https://review.openstack.org/296129 | 20:19 |
| sdake_ | AJaeger its really my fault for doing two things in one commit | 20:19 |
| sdake_ | i should have reorderedin a followon commit | 20:19 |
| fungi | AJaeger: i still think you're a bot, you just may need debugging occasionally | 20:20 |
| openstackgerrit | Tom Cocozzello proposed openstack-infra/project-config: Experimental patch_cover gate job https://review.openstack.org/296808 | 20:20 |
| sdake_ | AJaeger certainly leads opentck in eviews ;-) | 20:20 |
| sdake_ | and never makes errors | 20:20 |
| *** fkautz has joined #openstack-infra | 20:20 | |
| *** simonmcc has joined #openstack-infra | 20:20 | |
| *** zhiyan has joined #openstack-infra | 20:20 | |
| sdake_ | my reviews on stuff im an SMe on have errors all the time | 20:20 |
| *** vmbrasseur has joined #openstack-infra | 20:21 | |
| *** Swami_ has quit IRC | 20:21 | |
| *** e0ne has quit IRC | 20:21 | |
| pabelanger | clarkb: mordred: not sure about the rabbit issue. Will need to research the history of it | 20:22 |
| openstackgerrit | Merged openstack-infra/project-config: fuel-ostf: use db setup for python jobs https://review.openstack.org/297236 | 20:22 |
| *** kzaitsev_mb has quit IRC | 20:22 | |
| mordred | pabelanger: it has to do with setting the hostname | 20:23 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished https://review.openstack.org/297328 | 20:23 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion with net-iso in periodic job https://review.openstack.org/296765 | 20:23 |
| mordred | pabelanger: and whether we set the hostname to the host's ip or to 127.0.1.1 | 20:23 |
| pabelanger | Ah | 20:23 |
| openstackgerrit | Merged openstack-infra/project-config: update branches reported to #openstack-qa https://review.openstack.org/297290 | 20:23 |
| mordred | pabelanger: and pretty much every choice is terrible and against someone's relgion and also breaks _something_ | 20:23 |
| mordred | pabelanger: so it's not clear what 'right' is - or which of the broken things shoudl be fixed in which direction | 20:24 |
| lifeless | all of them | 20:24 |
| *** sridhar_ram1 is now known as sridhar_ram | 20:24 | |
| EmilienM | pabelanger: when can we get jenkins02 ? | 20:24 |
| EmilienM | pabelanger: some jobs are failing | 20:24 |
| *** Sukhdev has joined #openstack-infra | 20:24 | |
| pabelanger | EmilienM: it is working still now. It has been processing JJB for 4+ hours now :) | 20:24 |
| openstackgerrit | Merged openstack-infra/project-config: Fix which branch requirements bot runs https://review.openstack.org/297225 | 20:24 |
| mordred | lifeless: yah. | 20:24 |
| openstackgerrit | Merged openstack-infra/project-config: Fix voting for amqp1-centos oslo.messaging jobs https://review.openstack.org/296781 | 20:24 |
| mordred | lifeless: I think we should fix all fo tem in all of the directions | 20:24 |
| pabelanger | EmilienM: I suspect another hour before the puppet jobs are finally updated | 20:25 |
| EmilienM | pabelanger: wow ok | 20:25 |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental job to test TLS proxy in devstack https://review.openstack.org/293090 | 20:25 |
| pabelanger | EmilienM: at puppet-horizon now | 20:25 |
| AJaeger | fungi, sdake_ ;) | 20:25 |
| *** piet has joined #openstack-infra | 20:25 | |
| *** Nikolay_St has quit IRC | 20:26 | |
| *** ihrachys has quit IRC | 20:28 | |
| *** achanda has joined #openstack-infra | 20:28 | |
| pabelanger | EmilienM: basically jenkins02 is the only master left, so keep that in mind when approving your changes | 20:29 |
| pabelanger | EmilienM: up to puppet-kibana | 20:29 |
| EmilienM | pabelanger: ok | 20:29 |
| * EmilienM | 20:29 | |
| *** sdake has joined #openstack-infra | 20:30 | |
| openstackgerrit | Tom Cocozzello proposed openstack-infra/project-config: Move `keystone-coverage-db` job to check queue https://review.openstack.org/297351 | 20:30 |
| fungi | pabelanger: i've never been afraid to put a jenkins master into prepare for shutdown in situations like this. that will prevent it from running any new jobs with the old configs until the update completes | 20:30 |
| *** achanda has quit IRC | 20:31 | |
| fungi | also, as it begins to get increasingly idle, the jjb api calls tend to speed up as well | 20:31 |
| fungi | then once it's done, just cancel shutdown | 20:31 |
| *** jsavak has joined #openstack-infra | 20:32 | |
| AJaeger | fungi, pabelanger: Is the bindep / ubuntu-trusty conversion now finished? Or do we have any open items still to do? | 20:32 |
| *** sdake_ has quit IRC | 20:33 | |
| fungi | AJaeger: we still need to stop building devstack-trusty images | 20:33 |
| fungi | but otherwise basically done | 20:34 |
| AJaeger | fungi: cleanup ;) | 20:34 |
| AJaeger | fungi, so next steps: Announcing other-requirements.txt - and documenting it? | 20:34 |
| fungi | we also need to finish moving servers off ubuntu precise so we can drop our bare-precise images. that will then be the last of non-dib imagery in nodepool | 20:34 |
| fungi | AJaeger: the change to stop using the db setup macros in the normal unit test and coverage job templates merged then? | 20:35 |
| mordred | fungi: ++ to killing precise | 20:35 |
| pabelanger | fungi: Ah, good idea | 20:35 |
| pabelanger | fungi: let me do that now | 20:35 |
| AJaeger | fungi: yes, yolanda merged them this morning... | 20:35 |
| * mordred hands boxes of baby goats to several people | 20:35 | |
| fungi | AJaeger: next step is beer, then maybe announce uses for other-requirements.txt after the weekend, yes | 20:36 |
| pabelanger | AJaeger: Ya, I think we can stop building devstack-trusty | 20:36 |
| AJaeger | fungi, yes, definitely beer for all of us - cheers! | 20:36 |
| EmilienM | pabelanger: thx because puppet jobs are failing now | 20:36 |
| fungi | i'll try to get my example nova change rebased and polished up | 20:36 |
| pabelanger | EmilienM: ya, jenkins02 in shutdown mode, so new jobs will launch there | 20:36 |
| EmilienM | excellent | 20:37 |
| EmilienM | it allows me to finish the mitaka release | 20:37 |
| *** jsavak has quit IRC | 20:37 | |
| fungi | EmilienM: did your failures run on slaves assigned to jenkins02 or where? | 20:37 |
| fungi | EmilienM: most recent failure you can link to? | 20:37 |
| fungi | just want to make sure we're not overlooking something else | 20:37 |
| EmilienM | https://review.openstack.org/#/c/297293/ | 20:37 |
| *** Nikolay_St has joined #openstack-infra | 20:38 | |
| fungi | the gate-puppet-trove-puppet-beaker-rspec-ubuntu-trusty job i guess | 20:38 |
| EmilienM | fungi: yes | 20:38 |
| *** jsavak has joined #openstack-infra | 20:38 | |
| pabelanger | fungi: EmilienM: Ya, that was before our fix landed on jenkins02 | 20:38 |
| fungi | that definitely ran on a slave of jenkins02 | 20:38 |
| pabelanger | export 'BEAKER_set=nodepool-[trusty]' | 20:38 |
| pabelanger | was the issue | 20:38 |
| EmilienM | right | 20:38 |
| pabelanger | https://review.openstack.org/#/c/297265/ should be the control patchset | 20:39 |
| pabelanger | for puppet | 20:39 |
| fungi | so at this point, if any rechecks from this point forward still fail on that job, let us know since that would be unexpected | 20:39 |
| pabelanger | which is all green on status.o.o/zuul | 20:39 |
| pabelanger | fungi: agreed | 20:39 |
| AJaeger | fungi, pabelanger, bareon-allocator is using other-requirements since yesterday, see https://review.openstack.org/#/c/296337/ - solved a problem for them. | 20:40 |
| fungi | AJaeger: excellent | 20:40 |
| pabelanger | Nice | 20:40 |
| fungi | i'll try to get 164444 and 200672 squashed together and ready to use as an example | 20:41 |
| fungi | i want to make sure our focus is first on convenience to developers, second on integration in our ci jobs | 20:41 |
| mordred | ++ | 20:42 |
| fungi | so the goal of the example is "here's how to tell you have the right packages installed to locally test nova changes with tox" (and as an added bonus, changes to this file are self-testing in our ci!) | 20:42 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/296903 | 20:42 |
| *** ybathia has joined #openstack-infra | 20:43 | |
| AJaeger | do we want to remove packages like firefox from bindep-fallback - and thus push repos needing firefox to have an other-requirements file? | 20:43 |
| AJaeger | fungi, good approach | 20:43 |
| *** Goneri has quit IRC | 20:44 | |
| fungi | AJaeger: yes, i see cleaning up the bindep-fallback.txt file as a long tail possibility, but it's not urgent | 20:44 |
| fungi | that jobs will run slightly faster for repos which bother to define a usable other-requirements.txt file is a bit of a feature actually | 20:44 |
| fungi | let's think of it as an incentive | 20:45 |
| *** kgiusti has left #openstack-infra | 20:45 | |
| pabelanger | ++ | 20:45 |
| *** annegentle has joined #openstack-infra | 20:45 | |
| *** annegentle has quit IRC | 20:45 | |
| *** annegentle has joined #openstack-infra | 20:46 | |
| pabelanger | I see project like puppet and ansible being the first to add other-requirements.txt, since their manifests / playbooks should be installing the dependencies | 20:46 |
| *** jamesmcarthur has quit IRC | 20:46 | |
| clarkb | pabelanger: good point | 20:46 |
| *** Nikolay_St has quit IRC | 20:47 | |
| *** tonytan_brb has quit IRC | 20:47 | |
| *** rossella_s has quit IRC | 20:47 | |
| *** openstackgerrit has quit IRC | 20:48 | |
| *** openstackgerrit has joined #openstack-infra | 20:48 | |
| *** kzaitsev_mb has joined #openstack-infra | 20:50 | |
| AJaeger | bindep produces a lot log lines, is there a way to reduce that? I fear we need some logging here... | 20:52 |
| *** Sukhdev has quit IRC | 20:53 | |
| clarkb | maybe log on one line the list of packages then any errors? | 20:54 |
| fungi | AJaeger: bindep actually produces almost no logging. the package managers are producing output | 20:54 |
| fungi | and yes, we could for example shunt apt-get/yum/dnf/yast/whatever output to separate files and collect those with the rest of the logs | 20:55 |
| clarkb | or apt-get -q? | 20:55 |
| fungi | these are all options | 20:56 |
| fungi | i started simple | 20:56 |
| *** rhallisey has quit IRC | 20:56 | |
| clarkb | -q produces output suitable for logging | 20:56 |
| *** HeOS has joined #openstack-infra | 20:56 | |
| fungi | if we're redirecting that output, redirecting it through something that can timestamp the lines would also be cool, though i don't know how practical that is (and whether we already have something in our bag of tricks to do it) | 20:57 |
| clarkb | according to the manpage | 20:57 |
| fungi | wfm | 20:57 |
| *** doug-fis_ has joined #openstack-infra | 20:57 | |
| *** IlyaG has joined #openstack-infra | 20:58 | |
| fungi | jpmaxman: smarcet: did you determine whether you're ready to have dns for openstackid-dev switched over to the new 15gb replacement server yet? | 20:58 |
| *** ihrachys has joined #openstack-infra | 20:58 | |
| AJaeger | clarkb: nice idea. | 20:58 |
| * AJaeger calls it a day now... | 20:59 | |
| AJaeger | Would be great if some project-config cores could review https://review.openstack.org/297349 to unbreak repos that use pypy-db..., please | 20:59 |
| AJaeger | Bye! | 20:59 |
| *** doug-fish has quit IRC | 20:59 | |
| *** sdague has quit IRC | 20:59 | |
| smarcet | fungi: go ahead :) | 21:00 |
| smarcet | fungi: tnx | 21:00 |
| smarcet | ! | 21:00 |
| *** sdake has quit IRC | 21:01 | |
| fungi | smarcet: i've changed openstackid-dev.openstack.org from 2001:4800:7813:516:3bc3:d7f6:ff05:137e and 192.237.212.222 to 2001:4800:7817:102:be76:4eff:fe06:495d and 23.253.97.175. ttl on the previous records was 5 minutes, so should be resolving for you shortly | 21:03 |
| smarcet | fungi: ok cool | 21:03 |
| smarcet | tnx! | 21:03 |
| *** kzaitsev_mb has quit IRC | 21:04 | |
| *** doug-fis_ has quit IRC | 21:05 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 21:07 | |
| *** sileht has quit IRC | 21:07 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Update the Nova Scheduler meeting https://review.openstack.org/296786 | 21:07 |
| *** SumitNaiksatam has quit IRC | 21:07 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 21:07 | |
| *** smarcet has quit IRC | 21:07 | |
| *** sileht has joined #openstack-infra | 21:08 | |
| *** regXboi has quit IRC | 21:09 | |
| *** thorst has quit IRC | 21:12 | |
| *** fawadkhaliq has joined #openstack-infra | 21:13 | |
| *** _amrith_ is now known as amrith | 21:13 | |
| *** eharney has quit IRC | 21:14 | |
| *** thorst has joined #openstack-infra | 21:14 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** jsavak has quit IRC | 21:16 | |
| *** bpokorny has quit IRC | 21:16 | |
| *** mylu has quit IRC | 21:16 | |
| *** mylu has joined #openstack-infra | 21:17 | |
| *** rfolco_ has quit IRC | 21:18 | |
| *** gordc has quit IRC | 21:18 | |
| *** thorst has quit IRC | 21:18 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:21 | |
| *** bpokorny has joined #openstack-infra | 21:22 | |
| *** baoli has quit IRC | 21:23 | |
| anteaya | pabelanger: nice job on jjb | 21:25 |
| *** oanson has quit IRC | 21:25 | |
| anteaya | pabelanger: also are you aware I am a project-config-core? | 21:26 |
| anteaya | I was just wondering as everyone else got a ping, yet I also had been waiting | 21:26 |
| *** mrmartin has quit IRC | 21:26 | |
| *** oanson has joined #openstack-infra | 21:29 | |
| *** aysyd has quit IRC | 21:29 | |
| openstackgerrit | Sean McGinnis proposed openstack/requirements: Raise upper-constraint and GR for os-brick 1.1.1 https://review.openstack.org/297373 | 21:30 |
| *** ihrachys_ has joined #openstack-infra | 21:31 | |
| *** fdegir has quit IRC | 21:32 | |
| *** zxiiro has quit IRC | 21:32 | |
| *** jsavak has joined #openstack-infra | 21:32 | |
| *** fdegir has joined #openstack-infra | 21:33 | |
| *** HeOS has quit IRC | 21:33 | |
| *** Ng has quit IRC | 21:33 | |
| *** asettle has joined #openstack-infra | 21:33 | |
| *** zxiiro has joined #openstack-infra | 21:33 | |
| *** Ng has joined #openstack-infra | 21:33 | |
| *** ihrachys has quit IRC | 21:34 | |
| *** armax has quit IRC | 21:34 | |
| *** ashtokolov_ has joined #openstack-infra | 21:34 | |
| *** jraim has quit IRC | 21:34 | |
| *** yonglihe has quit IRC | 21:34 | |
| *** mwhahaha_ has joined #openstack-infra | 21:34 | |
| *** HeOS has joined #openstack-infra | 21:34 | |
| *** patrickeast has quit IRC | 21:34 | |
| *** ashtokolov has quit IRC | 21:34 | |
| *** evgenyl has quit IRC | 21:34 | |
| *** mwhahaha has quit IRC | 21:34 | |
| *** ashtokolov_ is now known as ashtokolov | 21:34 | |
| *** harlowja_at_home has quit IRC | 21:34 | |
| *** teran has quit IRC | 21:34 | |
| *** ayoung has quit IRC | 21:34 | |
| *** armax has joined #openstack-infra | 21:34 | |
| *** mescanef has quit IRC | 21:35 | |
| *** mwhahaha_ is now known as mwhahaha | 21:35 | |
| *** yonglihe has joined #openstack-infra | 21:35 | |
| *** weshay has quit IRC | 21:35 | |
| *** sbalukoff has quit IRC | 21:35 | |
| *** devananda has quit IRC | 21:35 | |
| *** Volundr has quit IRC | 21:35 | |
| *** evgenyl has joined #openstack-infra | 21:35 | |
| *** jraim has joined #openstack-infra | 21:36 | |
| *** teran has joined #openstack-infra | 21:36 | |
| *** mylu_ has joined #openstack-infra | 21:37 | |
| *** patrickeast has joined #openstack-infra | 21:37 | |
| *** Volundr has joined #openstack-infra | 21:38 | |
| zigo | mordred: Hi there! | 21:38 |
| zigo | mordred: Were the patches you wrote about merged? | 21:39 |
| zigo | mordred: Can we resume discussion on https://review.openstack.org/#/c/294022/ ? | 21:39 |
| pabelanger | anteaya: notification slipped through by mistake. I'll use project-config-core next time | 21:39 |
| zigo | Judging by the look of the comments, its looking like Ian and Andreas didn't really understood what I was aiming for. | 21:40 |
| *** knikolla has quit IRC | 21:40 | |
| *** openstack has joined #openstack-infra | 21:49 | |
| *** fawadkhaliq has quit IRC | 21:51 | |
| *** mylu has joined #openstack-infra | 21:52 | |
| *** fawadkhaliq has joined #openstack-infra | 21:52 | |
| openstackgerrit | Merged openstack/requirements: kilo: cap diskimage-builder<1.1.2 https://review.openstack.org/296555 | 21:54 |
| *** nadya has quit IRC | 21:54 | |
| *** jsavak has joined #openstack-infra | 21:55 | |
| *** ajmiller has joined #openstack-infra | 21:56 | |
| *** mylu has quit IRC | 21:57 | |
| *** pradk has quit IRC | 21:58 | |
| openstackgerrit | Merged openstack-infra/system-config: OpenStackId 1.0.17 https://review.openstack.org/297251 | 21:59 |
| *** oanson has joined #openstack-infra | 22:00 | |
| *** annegentle has joined #openstack-infra | 22:00 | |
| *** sridhar_ram has joined #openstack-infra | 22:00 | |
| *** edmondsw has joined #openstack-infra | 22:00 | |
| *** therve has joined #openstack-infra | 22:00 | |
| *** yuanying has joined #openstack-infra | 22:00 | |
| *** pcrews_ has joined #openstack-infra | 22:00 | |
| *** fbo has joined #openstack-infra | 22:00 | |
| *** markmcd1 has joined #openstack-infra | 22:00 | |
| *** ianychoi has joined #openstack-infra | 22:00 | |
| *** diana_clarke has joined #openstack-infra | 22:00 | |
| *** clayton has joined #openstack-infra | 22:00 | |
| *** tobiash has joined #openstack-infra | 22:00 | |
| *** skraynev has joined #openstack-infra | 22:00 | |
| *** jsavak has quit IRC | 22:00 | |
| *** rcrit_ has quit IRC | 22:01 | |
| *** jsavak has joined #openstack-infra | 22:01 | |
| *** gnuoy_ has quit IRC | 22:01 | |
| *** rcrit_ has joined #openstack-infra | 22:02 | |
| *** Ng has quit IRC | 22:03 | |
| *** DuncanT has quit IRC | 22:03 | |
| *** GheRivero has quit IRC | 22:03 | |
| *** Ng has joined #openstack-infra | 22:05 | |
| *** GheRivero has joined #openstack-infra | 22:05 | |
| *** DuncanT has joined #openstack-infra | 22:06 | |
| *** baoli has joined #openstack-infra | 22:09 | |
| *** baoli has quit IRC | 22:09 | |
| *** baoli has joined #openstack-infra | 22:10 | |
| *** thorst has joined #openstack-infra | 22:11 | |
| *** |-paul-| has joined #openstack-infra | 22:11 | |
| *** fawadkhaliq has quit IRC | 22:11 | |
| *** fawadkhaliq has joined #openstack-infra | 22:12 | |
| *** asettle has quit IRC | 22:12 | |
| *** signed8bit is now known as signed8bit_Zzz | 22:14 | |
| *** EricGonczer_ has joined #openstack-infra | 22:15 | |
| *** thorst has quit IRC | 22:16 | |
| *** thorst has joined #openstack-infra | 22:16 | |
| SpamapS | mordred: what's a good way to test if a clouds.yaml file will work? | 22:17 |
| *** jpr has quit IRC | 22:17 | |
| SpamapS | mordred: like, are there cli tools that use it? Also, is ~/.config/openstack/clouds.yaml the right filename? | 22:17 |
| *** kzaitsev_mb has quit IRC | 22:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Make pypy-db jobs also non-voting https://review.openstack.org/297349 | 22:20 |
| *** ihrachys_ is now known as ihrachys | 22:20 | |
| *** jsavak has quit IRC | 22:20 | |
| *** thorst has quit IRC | 22:21 | |
| clarkb | ok back from errands | 22:21 |
| clarkb | SpamapS: yes openstackclient uses it | 22:21 |
| anteaya | clarkb: welcome back | 22:21 |
| clarkb | SpamapS: so I just openstack catalog list usually to check on a clouds.yaml | 22:21 |
| clarkb | SpamapS: you can also set some env var that specifies where a random file might be living if not in a default location | 22:22 |
| SpamapS | clarkb: no mention of clouds.yaml in the error that I get tho | 22:22 |
| mtreinish | SpamapS: you could also be crazy like me and use: https://github.com/mtreinish/mesocyclone | 22:22 |
| SpamapS | oh wait | 22:23 |
| SpamapS | n/m | 22:23 |
| clarkb | mordred: we are 3 and a half hours away from the game | 22:23 |
| SpamapS | clarkb: thanks that's what I needed | 22:23 |
| * SpamapS is missing domain_name | 22:23 | |
| clarkb | oh ya if you v3 keystone the nyou need that | 22:24 |
| clarkb | pabelanger: EmilienM everything good on the puppet front? | 22:24 |
| EmilienM | the sun is shinning | 22:25 |
| *** knikolla has joined #openstack-infra | 22:25 | |
| SpamapS | clarkb: and if I have two clouds, how do I tell openstackclient which one I mean? | 22:26 |
| *** jpr has joined #openstack-infra | 22:26 | |
| clarkb | SpamapS: there is a parameter called --os-cloud-name or something | 22:26 |
| clarkb | its --os-cloud | 22:27 |
| clarkb | openstack --os-cloud mycloud1 catalog list | 22:27 |
| *** erlon has quit IRC | 22:27 | |
| SpamapS | clarkb: cool | 22:27 |
| *** jpr has quit IRC | 22:27 | |
| *** bpokorny_ has joined #openstack-infra | 22:27 | |
| clarkb | image uploads for the day are almost done so well ahead of where we were yesterday \o/ | 22:28 |
| SpamapS | Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL. | 22:29 |
| SpamapS | <sigh> | 22:29 |
| clarkb | SpamapS: if that is the case try forcing v2 | 22:29 |
| SpamapS | but shade.. I told you to use 2 | 22:29 |
| clarkb | oh | 22:29 |
| clarkb | you already tried that | 22:29 |
| SpamapS | identity_api_version: 2 | 22:29 |
| SpamapS | should it be 2.0? | 22:30 |
| openstackgerrit | Ben Nemec proposed openstack-infra/project-config: Add stable branch jobs for tripleo-ci https://review.openstack.org/297390 | 22:30 |
| clarkb | SpamapS: try '2' | 22:30 |
| *** bpokorny has quit IRC | 22:30 | |
| clarkb | identity_api_version: '3' is what I have in a random clouds.yaml for forcing 3 | 22:30 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:31 | |
| *** dimtruck is now known as zz_dimtruck | 22:31 | |
| SpamapS | yeah no that doesn't work | 22:32 |
| SpamapS | neither '2' nor '2.0' .. | 22:32 |
| SpamapS | adding /v2.0 to the auth url does | 22:32 |
| SpamapS | but that's :( | 22:32 |
| clarkb | is it openstackclient that is failing? | 22:32 |
| SpamapS | no | 22:32 |
| SpamapS | openstackclient is working | 22:32 |
| SpamapS | shade's functional test suite is failing | 22:32 |
| clarkb | huh | 22:32 |
| SpamapS | trying to run it against my cloud :) | 22:33 |
| clarkb | you might need to also specify the auth class thing and use eg v2password | 22:33 |
| clarkb | I forget what the clouds.yaml key is for that value though | 22:33 |
| anteaya | clarkb: I have elasticsearch installed on a vm, anything special about installing kibana4? | 22:34 |
| clarkb | anteaya: nope its a unpack tarball situation iirc | 22:34 |
| SpamapS | clarkb: mm k | 22:34 |
| SpamapS | clarkb: that does make sense, but it's also frustrating. :-P | 22:34 |
| *** HeOS has quit IRC | 22:34 | |
| openstackgerrit | Christopher Aedo proposed openstack-infra/project-config: Add python-appcatalogclient repository https://review.openstack.org/297395 | 22:34 |
| anteaya | clarkb: thanks I'll look around for the tarball | 22:35 |
| clarkb | anteaya: they have it linked to on their website | 22:36 |
| *** MarkAtwood has quit IRC | 22:36 | |
| austin81 | AJaeger: How long is voting open on https://review.openstack.org/#/c/287373/ ? | 22:37 |
| anteaya | clarkb: https://www.elastic.co/downloads/kibana | 22:41 |
| anteaya | austin81: offically closed at 19:00 utc today | 22:42 |
| anteaya | austin81: but open until fungi approves and he has company at the moment | 22:42 |
| anteaya | clarkb: kibana 4.2 works with elastichsearch 2.2 but I think we are logstash 2.0, kibana 4.1 says it works with elasticsearch 1.4-1.7 | 22:43 |
| austin81 | anteaya: Ah gotcha. I thought the spec was doomed to fade into oblivion :) Thanks! | 22:44 |
| anteaya | clarkb: I know we are logstash 2.0 but I don't know what version of elasticsearch we are running | 22:44 |
| anteaya | austin81: nope not doomed | 22:44 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: cmd: Add a config file fallback to the user's home directory https://review.openstack.org/237694 | 22:44 |
| anteaya | austin81: sorry it hasn't gotten more attention | 22:44 |
| anteaya | the busier we are the behinder we get it seems | 22:45 |
| clarkb | anteaya: we are running elasticsearch 1.7 | 22:45 |
| clarkb | we will likely upgrade to 2.x as part of the xenial switch | 22:46 |
| clarkb | maybe | 22:46 |
| anteaya | okay will work with kibana 4.1 | 22:47 |
| jroll | jeblair: zuul v3 feature request: make it such that project-config changes can be tested with the patch applied, via depends-on :) | 22:49 |
| jroll | job config changes, specifically | 22:49 |
| clarkb | jroll: that is in the spec | 22:49 |
| jroll | nice \o/ | 22:49 |
| clarkb | for unprivileged jobs | 22:49 |
| jroll | sure | 22:49 |
| jroll | this thrills me greatly | 22:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add support for debian-jessie https://review.openstack.org/297419 | 22:51 |
| *** austin81 has left #openstack-infra | 22:52 | |
| *** ybathia has quit IRC | 22:52 | |
| clarkb | down to 4 image uploads, 2 of which are ovh-bhs1 and rax-ord ubuntu-trusty in a loop | 22:52 |
| *** amitgandhinz has quit IRC | 22:52 | |
| clarkb | jroll: ya ultimtaely the issue with current zuul and jenkins is jenkins has no concept of an unprivileged job | 22:52 |
| clarkb | jroll: so everything is privileged which means not good to run jobs prereview | 22:53 |
| jroll | clarkb: yeah, the difference being something like "test nova" vs "update prod gerrit"? | 22:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Make gate-infra-puppet-apply-debian-jessie to voting https://review.openstack.org/297420 | 22:53 |
| clarkb | jroll: more run unittests vs release to pypi | 22:53 |
| jroll | sure | 22:53 |
| clarkb | since release to pypi requires credential data, but unittests are all self contained | 22:53 |
| jroll | ah, right I see | 22:54 |
| clarkb | we actually had valid s3 credentials we apparently supplied to all the jobs for a long time | 22:54 |
| clarkb | for unittests? | 22:54 |
| clarkb | pretty sure we don't do that anymore | 22:54 |
| pabelanger | clarkb: EmilienM Yup, things look good here | 22:54 |
| *** annegentle has quit IRC | 22:54 | |
| jroll | heh | 22:55 |
| *** chlong has joined #openstack-infra | 22:55 | |
| *** rkukura has quit IRC | 22:56 | |
| *** tphummel has quit IRC | 22:56 | |
| *** oanson has quit IRC | 22:58 | |
| *** sdake has joined #openstack-infra | 22:58 | |
| *** Qiming_ has joined #openstack-infra | 23:00 | |
| *** cfarquhar has quit IRC | 23:00 | |
| *** akshai has quit IRC | 23:00 | |
| *** cfarquhar has joined #openstack-infra | 23:00 | |
| *** cfarquhar has quit IRC | 23:00 | |
| *** cfarquhar has joined #openstack-infra | 23:00 | |
| clarkb | mordred: I am tempted to revert the iptables change for now | 23:01 |
| *** Qiming has quit IRC | 23:01 | |
| clarkb | but there really aren't any good options here | 23:01 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Make gate-infra-puppet-apply-debian-jessie to voting https://review.openstack.org/297420 | 23:02 |
| clarkb | pabelanger: have a link to that job running somewhere? | 23:02 |
| *** thorst has joined #openstack-infra | 23:02 | |
| *** ybathia has joined #openstack-infra | 23:02 | |
| clarkb | not sure if you have check experimentaled that anywhere | 23:02 |
| pabelanger | clarkb: Ya, 1 sec | 23:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Remove experimental-openstackci-beaker-ubuntu-trusty https://review.openstack.org/297431 | 23:03 |
| clarkb | I just want to look it over | 23:03 |
| pabelanger | http://logs.openstack.org/55/239455/1/experimental/gate-infra-puppet-apply-debian-jessie/e58086f/ | 23:03 |
| pabelanger | only running on openstack_project::single_use_slave right now | 23:04 |
| *** thorst has quit IRC | 23:05 | |
| clarkb | yup | 23:05 |
| clarkb | but it ran against it which is what I wanted to see | 23:05 |
| *** rkukura has joined #openstack-infra | 23:05 | |
| *** thorst has joined #openstack-infra | 23:05 | |
| pabelanger | clarkb: we could hold off and do check experimental once debian-jessie is in other clouds besides rax | 23:05 |
| clarkb | nah | 23:05 |
| pabelanger | k | 23:05 |
| clarkb | it should be fine | 23:05 |
| clarkb | really the biggest cross cloud worries are always to do with networking | 23:06 |
| clarkb | so if jenkins can ssh in chances are its fine | 23:06 |
| pabelanger | https://review.openstack.org/#/c/297419/ also fixes bindep-fallback.txt | 23:06 |
| *** jcoufal has quit IRC | 23:06 | |
| pabelanger | for debian-jessie | 23:06 |
| *** fawadkhaliq has quit IRC | 23:06 | |
| clarkb | if we manage to get that far that jenkins sshes into it that is | 23:06 |
| pabelanger | I think locales is the right package on debian | 23:06 |
| pabelanger | clarkb: agreed | 23:06 |
| clarkb | pabelanger: re iceweasel I thought they just fixed that? | 23:06 |
| *** fawadkhaliq has joined #openstack-infra | 23:07 | |
| clarkb | maybe its not in jessie yet? | 23:07 |
| *** davidlenwell has quit IRC | 23:07 | |
| pabelanger | ya, no jessie yet | 23:07 |
| mgagne | is there a way to trigger a set of jobs from a gerrit comment without trigerring the whole pipeline? | 23:07 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: Add an option to isntall a tempest plugin repo before running tempest https://review.openstack.org/297446 | 23:07 |
| clarkb | mgagne: no there is no sub pipeline trigger mechanism | 23:07 |
| clarkb | all the triggers operate at the pipeline level | 23:08 |
| mgagne | clarkb: alright thanks | 23:08 |
| clarkb | you an do the whole skip if thing to disable specific jobs if conditions are met | 23:08 |
| anteaya | clarkb: so I downloaded the elasticsearch 1.7.5 deb following the searchlight macro but kibana is telling me that I have actually installed elasticsearch 1.3.1, elasticsearch -v says command not found | 23:08 |
| *** annegentle has joined #openstack-infra | 23:08 | |
| clarkb | anteaya: I think you can check the /var/log/elasticsearch/elasticsearch.log | 23:08 |
| mgagne | clarkb: lets say I have 10 jobs in a pipeline, I could skip jobs if it doesn't match some files so they don't all trigger? hmm | 23:08 |
| clarkb | anteaya: it prints that info at startup iirc | 23:09 |
| anteaya | clarkb: thanks | 23:09 |
| anteaya | I'll look | 23:09 |
| clarkb | mgagne: yup exactly | 23:09 |
| mgagne | clarkb: I will explore that possibility thanks! | 23:09 |
| *** bpokorny_ has quit IRC | 23:09 | |
| *** bpokorny has joined #openstack-infra | 23:09 | |
| *** rbrndt_ has quit IRC | 23:10 | |
| anteaya | clarkb: top line confirms susan richards says 1.7.5 | 23:10 |
| *** thorst has quit IRC | 23:10 | |
| mgagne | lets say you have a repo building binaries like qcow2, lot of qcow2. what would be the sane way to handle it? multiple repo? use skip as suggested? | 23:10 |
| anteaya | however I can't seem to convince kibana | 23:10 |
| clarkb | anteaya: I am not sure then | 23:11 |
| clarkb | anteaya: double check its connecting to that elasticsearch? | 23:11 |
| clarkb | mgagne: you only need some of the qcow2s based on which files were updated? | 23:11 |
| pabelanger | mgagne: look at some of the project-config jobs, we do that in some cases | 23:11 |
| clarkb | if so the skip if stuff should work | 23:12 |
| mgagne | clarkb: will check. custom pipeline with comment trigger + skip I guess | 23:12 |
| clarkb | however it might be better to map a single artifact onto each project? I know we try to do that for our ython artifacts just for sanity | 23:12 |
| anteaya | clarkb: not sure how to do that but will look around, I think I only have the one elasticsearch running on this vm | 23:12 |
| anteaya | I only installed the one deb | 23:12 |
| *** annegentle has quit IRC | 23:13 | |
| *** thorst has joined #openstack-infra | 23:14 | |
| mgagne | clarkb: I think that would be the next step I guess. trying to get "some" results after days of mental block | 23:14 |
| anteaya | ps aux only has one process for a running elasticsearch | 23:14 |
| *** davidlenwell has joined #openstack-infra | 23:14 | |
| clarkb | anteaya: I am not sure ten | 23:14 |
| clarkb | anteaya: could be a kibana bug? did you downgrade kibana? | 23:14 |
| *** tphummel has joined #openstack-infra | 23:15 | |
| pabelanger | clarkb: is 297273 safe to land or is that affected by the firewall revert you are talking about | 23:15 |
| clarkb | pabelanger: should be safe | 23:16 |
| anteaya | I did not downgrade kibana | 23:16 |
| clarkb | the firewall thing appears to only affect devstack-centos7 for whatever reason | 23:16 |
| clarkb | anteaya: you might need an older kbana 4.1? | 23:16 |
| anteaya | I have 4.1.6 | 23:16 |
| clarkb | thats the one it says is compatbile | 23:16 |
| anteaya | I'll investigate my cluster first | 23:16 |
| anteaya | yeah | 23:16 |
| *** tpsilva has quit IRC | 23:17 | |
| *** ybathia has quit IRC | 23:17 | |
| anteaya | let me look at my cluster for a minute | 23:17 |
| anteaya | I might learn something | 23:17 |
| *** thorst_ has joined #openstack-infra | 23:17 | |
| *** sdake_ has joined #openstack-infra | 23:17 | |
| *** sdake has quit IRC | 23:18 | |
| clarkb | pabelanger: for 297273 we will just have to keep an eye out for multinode jobs that fail to ssh | 23:18 |
| anteaya | clarkb: http://paste.openstack.org/show/491822/ | 23:18 |
| clarkb | pabelanger: they use eth1 on rax which is why it was failing earlier | 23:19 |
| *** thorst has quit IRC | 23:19 | |
| anteaya | I don't know who thetrainers.ca are | 23:19 |
| clarkb | that says you ahve two serversin your cluster | 23:19 |
| clarkb | maybe that one is older? | 23:19 |
| anteaya | that one isnt' my vm at all | 23:19 |
| clarkb | the ip address for both servers is printed so you can figure that out | 23:19 |
| clarkb | rogue servers on the loose | 23:20 |
| pabelanger | clarkb: ack | 23:20 |
| anteaya | this istn' my ip: http://199.19.213.180:9200/ | 23:21 |
| pabelanger | hi Joe Cartelli | 23:21 |
| anteaya | someone has elasticsearch running and my kibana found it | 23:21 |
| anteaya | and they are running 1.3.1 | 23:21 |
| *** thorst has joined #openstack-infra | 23:21 | |
| anteaya | here is the info from my ip: http://paste.openstack.org/show/491823/ | 23:22 |
| anteaya | I don't know how to tell kibana to ignore joe | 23:22 |
| *** thorst_ has quit IRC | 23:22 | |
| anteaya | I had given it my ip address | 23:22 |
| anteaya | not sure how it scooped up joe | 23:22 |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/devstack-gate: Add an option to isntall a tempest plugin repo before running tempest https://review.openstack.org/297446 | 23:23 |
| anteaya | joe has been running since 2014 | 23:23 |
| clarkb | anteaya: it joined your cluster which is how kibana found it | 23:23 |
| clarkb | elasticsearch does a multicast discovery I am guessing that somehow worked in your cloud env? | 23:24 |
| openstackgerrit | Merged openstack-infra/project-config: Add release notes job for Kolla repo https://review.openstack.org/296129 | 23:25 |
| openstackgerrit | Christopher Aedo proposed openstack-infra/project-config: Add python-appcatalogclient repository https://review.openstack.org/297395 | 23:25 |
| anteaya | I guess? | 23:25 |
| anteaya | and I had just used the default localhost, let me set the ip | 23:25 |
| clarkb | it likely won't help until you get that thing out of your cluster | 23:26 |
| clarkb | there is a config change you cna make to turn off multicast discovery then restart your node | 23:26 |
| clarkb | anteaya: discovery.zen.ping.multicast.enabled: false | 23:27 |
| clarkb | in /etc/elasticsearch/elasticsearch.yml I think | 23:27 |
| pabelanger | JJB finally finished on jenkins02 | 23:28 |
| clarkb | maybe we should retire 01 and 02 | 23:28 |
| anteaya | pabelanger: woot! | 23:28 |
| clarkb | and rebuild them if we need them | 23:28 |
| *** sdake has joined #openstack-infra | 23:28 | |
| pabelanger | it's still in shutdown mode, so I'll clean up slaves and restart | 23:28 |
| anteaya | clarkb: I'll try that, specifying my ip instead of localhost for kibana did not work | 23:28 |
| pabelanger | clarkb: I would be okay with that | 23:28 |
| *** sdake_ has quit IRC | 23:28 | |
| *** jimkenty has joined #openstack-infra | 23:29 | |
| pabelanger | jenkins02 back online | 23:32 |
| clarkb | pretty sure this is just an artifact of the flavor we booted them on | 23:32 |
| clarkb | and a rebuild would make them much happier | 23:32 |
| anteaya | clarkb: yay! | 23:32 |
| anteaya | setting multicast to false was what it needed | 23:32 |
| openstackgerrit | Merged openstack-infra/project-config: Create ACLs for ironic-lib stable branches https://review.openstack.org/297345 | 23:32 |
| anteaya | I know have one node in my cluster and it is running 1.7.2 | 23:33 |
| anteaya | sorry 1.7.5 | 23:33 |
| clarkb | yay | 23:33 |
| anteaya | so the advance button Caution: You can break stuff here | 23:33 |
| anteaya | I'm in favour of just removing the advanced button | 23:33 |
| clarkb | ya, not sure how to do that but that would be good | 23:34 |
| clarkb | maybe if we set up a proxy we can just bitbucket that path | 23:34 |
| anteaya | I don't know how to do that | 23:34 |
| anteaya | I was just going to find the css/js for the button and remove it from the html | 23:34 |
| pabelanger | and done for the day. Enjoy the PTO everybody | 23:35 |
| anteaya | have a good easter | 23:36 |
| clarkb | anteaya: thats another option if we can make that easy to deploy | 23:36 |
| *** tqtran has quit IRC | 23:37 | |
| anteaya | that would be my first direction in any case | 23:37 |
| *** IlyaG has quit IRC | 23:38 | |
| *** piet has quit IRC | 23:42 | |
| *** jimkenty has quit IRC | 23:42 | |
| mgagne | where can I find scary stats and numbers of resource consumption of infra? instance/day, disk/day, etc. | 23:44 |
| clarkb | mgagne: grafana.openstack.org is probably a pretty good place to start | 23:45 |
| mgagne | clarkb: anything readable by C-titles like people? =) | 23:46 |
| clarkb | no we don't publish clear things like that anyhwere, but its easy to get instances per day if you figure each job is one instace | 23:46 |
| clarkb | mgagne: are you interested in internap specifically or in general? | 23:46 |
| mgagne | general | 23:46 |
| mgagne | I think mordred made a presentation with some numbers and scary graphs | 23:47 |
| clarkb | http://graphite.openstack.org/render/?from=-6weeks&height=180&until=now&width=334&bgcolor=ffffff&fgcolor=000000&target=alias%28summarize%28sumSeries%28stats_counts.zuul.pipeline.*.all_jobs%29,%271d%27%29,%27All%20Jobs%27%29&title=Zuul%20Jobs%20Launched%20%28per%20Hour%29&_t=0.15331281422375265#1458863222679 is a modification of the graph on the zuul status page | 23:48 |
| *** Qiming_ has quit IRC | 23:48 | |
| *** claudiub|2 has quit IRC | 23:49 | |
| mgagne | I guess that will do | 23:49 |
| clarkb | in the lsat 6 week we peaked at 26k jobs per day | 23:49 |
| mgagne | :D | 23:49 |
| jeblair | clarkb, mgagne: ^ specifically, that is modified to be jobs per *day* :) | 23:49 |
| openstackgerrit | Merged openstack-infra/project-config: Don't run releasenotes job for neutronclient till Mitaka https://review.openstack.org/296952 | 23:49 |
| clarkb | at roughly oh ya I dind't change the title | 23:49 |
| clarkb | er | 23:49 |
| jeblair | clarkb updated the graph but not the title, and so left a trap for unsuspecting viewers :) | 23:49 |
| clarkb | at roughly 8vcpu, 8GB ram, 80gb disk per instance | 23:50 |
| docaedo | that guy is always leaving traps around :/ | 23:50 |
| openstackgerrit | Alan Pevec proposed openstack-infra/project-config: Add release tarball generation to swift3 https://review.openstack.org/297458 | 23:50 |
| mgagne | I think there was a presenation with GB logs generated by months, instances and such | 23:50 |
| jeblair | mgagne: instances == jobs | 23:50 |
| tonyb | jhesketh, clarkb, greghaynes: For the records the new ubuntu-trusty images fixed the locale issue I had Thanks! | 23:51 |
| clarkb | tonyb: awesome | 23:51 |
| jeblair | increasing the aggregation: 120,000 per week | 23:51 |
| jhesketh | tonyb: cool :-) | 23:51 |
| jeblair | so .5 million per month; 6 million per year | 23:52 |
| mgagne | nice :D | 23:53 |
| mgagne | those numbers are enough for now thanks! | 23:53 |
| *** ajmiller has quit IRC | 23:54 | |
| *** rossella_s has joined #openstack-infra | 23:55 | |
| jroll | I have an interesting question - I want to make these artifacts branch-specific; what's the best way to do that? was thinking if a missing file doesn't blow up the job, I could just list one for each branch. https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/ironic-python-agent-jobs.yaml | 23:55 |
| jroll | more generally, what I want to do is publish these files from the latest commit on each branch | 23:56 |
| mgagne | jroll: like other python projects? http://tarballs.openstack.org/nova/ they suffix the tarball with the branch name. | 23:58 |
| jeblair | jroll: there are branch-tip tarball jobs that push a build of the branch tip of projects up to tarballs.o.o after each commit lands... something like that? | 23:58 |
| mgagne | that ^ | 23:58 |
| *** esikachev has joined #openstack-infra | 23:58 | |
| greghaynes | tonyb: awesome :) | 23:58 |
| jroll | jeblair: mgagne: yes, but this is a ramdisk we're building from source. but yes similar :) | 23:59 |
| mgagne | I guess it would be a matter of using templates instead of static job and injecting the branch name somewhere | 23:59 |
| jeblair | jroll: if what mgagne and i describe sounds like what you want, the jobs are called '{project}-branch-tarball' if you want to take a look at how they work | 23:59 |
| mgagne | jroll: I'm in a very similar boat atm =) | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!