*** ildikov has joined #opendev | 00:01 | |
*** dviroel has joined #opendev | 00:01 | |
*** aannuusshhkkaa has joined #opendev | 00:01 | |
ianw | i'll double check the dib before restoring builds | 00:05 |
---|---|---|
*** aannuusshhkkaa has quit IRC | 00:13 | |
*** aannuusshhkkaa has joined #opendev | 00:16 | |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 00:23 |
*** sgw has quit IRC | 00:27 | |
ianw | $ sudo docker exec -it nodepool-builder-compose_nodepool-builder_1 disk-image-create --version | 00:30 |
ianw | 3.2.1 | 00:30 |
fungi | looks good | 00:33 |
ianw | dmsimard: re 716449 ^^ have you had any thoughts about the requirements.txt for the stable branch re ansible/ansible-base? i guess it's not possible to say "ansible | ansible-base" :/ | 00:35 |
ianw | (ara requirements, to be clear :) | 00:35 |
fungi | ftr, you can also `cd /etc/nodepool-builder-compose;sudo docker-compose exec nodepool-builder disk-image-create --version` | 00:36 |
ianw | ok, my ssh-agent socket seems to have disappeared again. something is up | 00:36 |
openstackgerrit | Merged openstack/project-config master: Revert "Pause all image builds" https://review.opendev.org/747312 | 00:40 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 01:02 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 01:31 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 01:41 |
openstackgerrit | Merged zuul/zuul-jobs master: Revert "Ensure git is installed in prepare-workspace-git role" https://review.opendev.org/747238 | 01:44 |
dmsimard | ianw: ara master has jobs that test again devel and I haven't seen them failing recently | 02:05 |
dmsimard | ansible isn't a dependency for >=1.0 but it could be an issue for 0.x | 02:06 |
dmsimard | If it doesn't work it's a bug I guess :p | 02:08 |
ianw | dmsimard: yeah it's obscure ... it's our system-config test running against ansible -devel branch. ara wants to bring in "ansible" which breaks the install from git checkout which is "ansible-base" | 02:25 |
ianw | "the install" being "the ansible install" | 02:25 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 02:47 |
openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: tox: include command output in log/error https://review.opendev.org/747325 | 02:57 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 03:06 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 03:59 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] drop test-requirements.txt https://review.opendev.org/747328 | 03:59 |
*** ykarel|away has joined #opendev | 04:26 | |
*** ykarel|away is now known as ykarel | 04:26 | |
openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] drop test-requirements.txt https://review.opendev.org/747328 | 04:31 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 04:31 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] collect tox logs https://review.opendev.org/747329 | 04:31 |
*** raukadah is now known as chandankumar | 04:41 | |
*** ysandeep|away is now known as ysandeep | 04:56 | |
*** sgw has joined #opendev | 05:15 | |
openstackgerrit | Ian Wienand proposed opendev/system-config master: tox: drop test-requirements.txt https://review.opendev.org/747328 | 06:31 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Collect tox logs https://review.opendev.org/747329 | 06:31 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes https://review.opendev.org/716449 | 06:31 |
*** sgw has quit IRC | 06:38 | |
*** weshay|ruck has quit IRC | 06:38 | |
*** dulek has quit IRC | 06:38 | |
*** danpawlik has quit IRC | 06:38 | |
*** icey has quit IRC | 06:38 | |
*** fbo|pto has quit IRC | 06:38 | |
*** SotK has quit IRC | 06:38 | |
*** noonedeadpunk has quit IRC | 06:38 | |
*** sgw has joined #opendev | 06:43 | |
*** dulek has joined #opendev | 06:43 | |
*** weshay|ruck has joined #opendev | 06:43 | |
*** fbo|pto has joined #opendev | 06:43 | |
*** danpawlik has joined #opendev | 06:43 | |
*** icey has joined #opendev | 06:43 | |
*** SotK has joined #opendev | 06:43 | |
*** noonedeadpunk has joined #opendev | 06:43 | |
frickler | ianw: are we green now with the dib/git/apt mess or is there still something I should look at? | 06:45 |
ianw | frickler: i think it's all good. revert went in, new builds done, images are building/uploading | 06:49 |
*** andrewbonney has joined #opendev | 07:07 | |
*** DSpider has joined #opendev | 07:14 | |
*** hashar has joined #opendev | 07:17 | |
yoctozepto | infra-root: could I get https://review.opendev.org/747185 in? (pretty please) :-) | 07:23 |
yoctozepto | also please let me know how to handle the launchpad part | 07:30 |
yoctozepto | ianw: thanks :-) | 07:31 |
*** dtantsur|afk is now known as dtantsur | 07:31 | |
*** priteau has joined #opendev | 07:40 | |
openstackgerrit | Merged openstack/project-config master: Add openstack/etcd3gw https://review.opendev.org/747185 | 07:49 |
AJaeger | yoctozepto: you need to configure launchpad yourself, there's nothing we do - everything is documented in Creator's Guide | 07:53 |
yoctozepto | AJaeger: ok, I'll look in that other guide then | 07:55 |
yoctozepto | AJaeger: actually no, I looked in it exactly https://docs.opendev.org/opendev/infra-manual/latest/creators.html | 07:56 |
yoctozepto | AJaeger: no mention of launchpad :-( | 07:56 |
*** moppy has quit IRC | 08:01 | |
*** moppy has joined #opendev | 08:03 | |
AJaeger | yoctozepto: ah, we removed that at one point and have only storyboard in there. | 08:09 |
AJaeger | yoctozepto: see change Ibec66e8dd952ea33a2f84578e41b33d5afab4451 for how to set it up | 08:11 |
yoctozepto | AJaeger: I know sb is the way to go, but, considering the general resistance, I guess it would be nice to restore parts of this :-) | 08:17 |
*** tosky has joined #opendev | 08:22 | |
*** lpetrut has joined #opendev | 08:22 | |
*** fressi has joined #opendev | 08:23 | |
*** fressi has quit IRC | 08:27 | |
yoctozepto | AJaeger: i think i did good: https://launchpad.net/python-etcd3gw | 08:29 |
*** hashar has quit IRC | 08:33 | |
*** hashar has joined #opendev | 08:37 | |
*** fressi has joined #opendev | 08:43 | |
AJaeger | yoctozepto: I hope so - my last setup for launchpad was ages ago. Let's see how it works ;) | 08:44 |
*** danpawlik has quit IRC | 08:45 | |
*** danpawlik has joined #opendev | 08:47 | |
*** hashar has quit IRC | 08:47 | |
yoctozepto | AJaeger: :-) would have to ask bnemec to add me as a driver and a bug supervisor to test further now | 08:49 |
yoctozepto | ;d | 08:49 |
*** hashar has joined #opendev | 08:53 | |
*** openstackgerrit has quit IRC | 08:54 | |
*** openstackgerrit has joined #opendev | 08:59 | |
openstackgerrit | Merged opendev/irc-meetings master: Update CloudKitty meeting information https://review.opendev.org/747256 | 08:59 |
yoctozepto | does it usually take long to mirror those repos? I can't see http://opendev.org/openstack/etcd3gw yet | 09:06 |
AJaeger | yoctozepto: check https://zuul.opendev.org/t/openstack/status - the deploy queue still shows the entry | 09:20 |
AJaeger | yoctozepto: at this time it takes so long due to the many periodic jobs | 09:20 |
AJaeger | including our daily infra jobs | 09:20 |
AJaeger | yoctozepto: https://zuul.opendev.org/t/openstack/status/change/747185,3 | 09:21 |
yoctozepto | AJaeger: ah, missed that one; thanks, understood | 09:55 |
yoctozepto | did ethercalc die just now? I was editing https://ethercalc.openstack.org/kolla-infra-service-matrix but now it keeps reconnecting | 09:57 |
AJaeger | infra-root, I get a 503 on ethercalc ^ | 09:59 |
frickler | AJaeger: yoctozepto: ethercalc service had crashed, restarted it | 10:18 |
frickler | #status log restarted crashed ethercalc service, log info at http://paste.openstack.org/show/797024/ | 10:19 |
openstackstatus | frickler: finished logging | 10:19 |
frickler | infra-root: from the logs before the ethercalc crash and the pads mentioned there, it looks like our service is being used by some non opendev-related consumers, not sure if we'd want to monitor that or rather don't care | 10:22 |
yoctozepto | frickler: thanks | 10:29 |
*** jhesketh has quit IRC | 10:53 | |
*** jhesketh has joined #opendev | 10:55 | |
*** hashar has quit IRC | 11:47 | |
*** ysandeep is now known as ysandeep|afk | 11:47 | |
AJaeger | thanks, frickler ! | 11:51 |
*** ysandeep|afk is now known as ysandeep | 11:59 | |
*** hashar has joined #opendev | 12:02 | |
*** rosmaita has joined #opendev | 12:45 | |
*** ysandeep is now known as ysandeep|mtg | 13:03 | |
fungi | frickler: that's the same exportCSV error we've seen twice recently | 13:10 |
fungi | likely a known bug and i wouldn't be surprised if upgrading fixes it, just haven't had time to look | 13:11 |
*** ysandeep|mtg is now known as ysandeep | 13:46 | |
fungi | i've started the ethercalc service again. it crashed once more on the same error at 13:12:10 | 13:50 |
fungi | https://github.com/audreyt/ethercalc/pull/521 | 13:51 |
fungi | assuming it's the same error (certainly looks the same) then the fix merged to master and was included in version 0.20170704.0 over three years ago | 13:52 |
fungi | unfortunately the javascript nodejs is running is compiled from livescript, so we can't just hot-patch the main.js we'll have to patch the source tree and then rebuild it | 13:57 |
*** mlavalle has joined #opendev | 14:01 | |
fungi | puppet-ethercalc says we're installing $ethercalc_version= '0.20161220.1', | 14:02 |
fungi | though apparently the author doesn't use git tags much, 0.20170704.0 is the most recent tag in the github repo but there's a 0.20200505.0 package at https://www.npmjs.com/package/ethercalc | 14:06 |
*** ykarel is now known as ykarel|away | 14:11 | |
*** lpetrut has quit IRC | 14:15 | |
*** ysandeep is now known as ysandeep|away | 14:34 | |
*** ykarel|away has quit IRC | 14:51 | |
*** chandankumar is now known as raukadah | 14:52 | |
*** fressi has quit IRC | 14:54 | |
*** qchris has quit IRC | 14:57 | |
*** ysandeep|away is now known as ysandeep | 15:06 | |
*** qchris has joined #opendev | 15:10 | |
*** ysandeep is now known as ysandeep|away | 15:10 | |
clarkb | following up on the nodepool builders all 4 are running and we have images as new as 9 hours ago | 15:10 |
clarkb | everything should be back to normal now without pauses etc | 15:10 |
clarkb | (thank you ianw for getting that done) | 15:11 |
*** tkajinam has quit IRC | 15:26 | |
openstackgerrit | Clark Boylan proposed openstack/diskimage-builder master: Don't remove packages that are requested to be installed https://review.opendev.org/747220 | 15:30 |
clarkb | yoctozepto: ^ I think there was actually a bug in the old branching behavior that I noticed when you pointed ou the redundant branch | 15:30 |
clarkb | yoctozepto: I think that ps fixes it and addresses your comment | 15:30 |
*** rosmaita has left #opendev | 15:52 | |
yoctozepto | clarkb: thanks, now I read it even differently; could you address my comments now? | 15:53 |
*** mlavalle has quit IRC | 15:59 | |
*** tosky has quit IRC | 16:00 | |
clarkb | yoctozepto: responded. Hopefully that helps | 16:01 |
*** mlavalle has joined #opendev | 16:02 | |
yoctozepto | clarkb: one clear, one to go ;d | 16:21 |
yoctozepto | I don't deny something might be escaping me | 16:21 |
clarkb | yoctozepto: hopefully latest comment helps. Basically there is a behavior that I'm trying to preserve as much as possible | 16:41 |
clarkb | whether or not that existing behavior is actually critical I don't know | 16:41 |
*** dtantsur is now known as dtantsur|afk | 16:50 | |
*** hashar has quit IRC | 16:59 | |
*** priteau has quit IRC | 17:07 | |
yoctozepto | clarkb: ok, the example was great; I guess I'm just a little tired! I left +1 (not a core yet) and some future ideas | 17:20 |
clarkb | yoctozepto: fwiw I had to run the script locaally and generate files to grok the format | 17:35 |
fungi | totally weird collision of topics today, i went looking for a geekcode decoder, and the geekcode.pm module on cpan was written by the ethercalc author | 17:39 |
clarkb | fungi: I think the ethercalc author was/is also secretary of technology of taiwan | 17:39 |
fungi | yikes | 17:40 |
clarkb | (I may have gotten the exact title wrong but cabinet level type equivalent) | 17:40 |
fungi | "taiwan's digital minister" | 17:40 |
fungi | anyway, looking at the latest ethercalc readme we need at least nodejs 0.8 and redis 2.4 | 17:42 |
fungi | which we're still easily running, it looks like | 17:44 |
yoctozepto | nodejs 0.8? lol | 17:45 |
yoctozepto | it remembers dinosaurs | 17:45 |
fungi | it recommends 4.x but supports 0.8+ | 17:45 |
yoctozepto | does it have no deps? | 17:45 |
yoctozepto | many libs dropped support for anything pre-8.x a long time ago | 17:46 |
yoctozepto | so I'd worry about its security stance | 17:46 |
fungi | https://www.npmjs.com/package/ethercalc lists 13 deps | 17:47 |
fungi | but yeah, the yarn.lock file hasn't been touched in 2 years | 17:48 |
yoctozepto | fungi: optimist was dropped a long time ago | 17:48 |
yoctozepto | I mean one of the deps | 17:48 |
fungi | even though the most recent release of ethercalc is only 3 months old | 17:48 |
yoctozepto | I do hope nobody dropped any optimists :-) | 17:48 |
yoctozepto | they deserve to live | 17:49 |
fungi | so not regenerating the deps list when making new releases, looks like | 17:49 |
yoctozepto | well, you can't really regenerate for deps which are long retired ;-) | 17:49 |
fungi | sure | 17:49 |
yoctozepto | zappajs looks bad as duck as well | 17:50 |
yoctozepto | so it's 7 years old lib and a 3 yu old lib alreay | 17:50 |
yoctozepto | uuid-pure 9 years | 17:50 |
yoctozepto | who gives more | 17:50 |
yoctozepto | just burn it | 17:50 |
fungi | anyway, my point was it looks like we can probably upgrade to latest ethercalc on the current server without touching our config management otherwise, and get the fix to the annoying csvexport crash | 17:51 |
clarkb | just because a lib is old doesn't necessarily mean it needs updates. I mean half those libs are probably a single function :P | 17:51 |
yoctozepto | clarkb: sure, cors is a known stable lib | 17:51 |
clarkb | fungi: any idea if we have to migrate the data at all? I Guess we can just dump the redis data (I think we back it up daily) then upgrade and if it doesn't work roll back | 17:51 |
yoctozepto | clarkb: but optimist is openly retired | 17:51 |
yoctozepto | clarkb: zappajs is a whole framework | 17:52 |
clarkb | yoctozepto: but also what are our choices there? turn off the service? I'm not entirely opposed | 17:52 |
clarkb | I'm not going to maintain a bunch of js libs. I don't have enough time in the day as it is | 17:52 |
yoctozepto | clarkb: true that, I mean just know the consequences | 17:52 |
fungi | it's been fairly low-maintenance until this past week when we started having someone/something triggering this 3-years-fixed-upstream bug | 17:52 |
yoctozepto | I'm using ethercalc from time to time but might move to something else if needed | 17:52 |
yoctozepto | fungi: is it used much in general? | 17:53 |
yoctozepto | i mean ethercalc | 17:53 |
yoctozepto | in opendev | 17:53 |
clarkb | yoctozepto: not like etherpad | 17:53 |
clarkb | its a much smaller niche | 17:53 |
yoctozepto | clarkb: i figured | 17:53 |
fungi | it's being used for ptg timeslot coordination, for example | 17:53 |
clarkb | ya it is used though | 17:53 |
yoctozepto | i see | 17:53 |
fungi | so i don't want to turn it off out from under people with no warning, but similarly i don't want to have to keep watching it to see if it's crashed yet again | 17:54 |
yoctozepto | let's then just cross finger and upgrade for now | 17:55 |
fungi | and fixing it today is probably better since i'm out all next week meaning even fewer people around to find out it's down and restart it | 17:55 |
clarkb | fungi: ya do a db dump first so wecan rollback if necessary and update imo | 17:55 |
openstackgerrit | Jeremy Stanley proposed opendev/puppet-ethercalc master: Upgrade to latest EtherCalc (0.20200505.0) https://review.opendev.org/747453 | 17:59 |
fungi | clarkb: yoctozepto: ^ | 17:59 |
corvus | fungi: plan lgtm | 17:59 |
fungi | now we get to find out what bitrot that module has accumulated in over a year since its last merged change | 17:59 |
fungi | i'm expecting random jobs to fail and need fixes inserted ahead of that | 18:00 |
fungi | so will keep tabs on zuul results | 18:00 |
*** shtepanie has joined #opendev | 18:02 | |
yoctozepto | fungi: I understand I should not be working on my sheet now ;d | 18:04 |
corvus | yoctozepto: i think you have a little while before excitement happens :) | 18:04 |
yoctozepto | corvus: it's been a wild CI ride week, I like excitement | 18:05 |
fungi | yeah, for starters, to completely expect to need to push more changes to fix 1.5 years of accumulated bitrot in puppet-ethercalc before we can hope to merge that | 18:07 |
fungi | hopefully they'll be straightforward issues, i expect we've seen most of them already in our other modules at this point | 18:08 |
fungi | and then once jobs are succeeding, i'll make an additional db dump before we approve | 18:08 |
fungi | as added insurance in case we have to roll back | 18:09 |
clarkb | I just fixed the beaker rspec ssh issue and that got jobs passing for another erpo | 18:12 |
clarkb | I expect that means it will be just known issues like the :: lint problem | 18:12 |
*** andrewbonney has quit IRC | 18:27 | |
openstackgerrit | Clark Boylan proposed opendev/puppet-ethercalc master: Upgrade to latest EtherCalc (0.20200505.0) https://review.opendev.org/747453 | 19:14 |
clarkb | fungi: ^ that should fix the linter errors | 19:14 |
fungi | thanks! i was just circling back around to check | 19:15 |
fungi | by the way, how awesome is it that we had a version of puppet-lint which told us we should put :: in front of everything, and then a later version which said we should remove :: from in front of everything? i don't know if i've mentioned yet how much i love undoing something which didn't seem necessary to add in the first place | 19:17 |
yoctozepto | fungi: like puppet itself | 19:19 |
yoctozepto | :-) | 19:19 |
fungi | in fairness, puppet was better than no configuration management, and i'm not sorry we used it | 19:24 |
corvus | see change #1 :) | 19:24 |
fungi | but we've well outgrown it and it's a questionable fit for our current use cases | 19:24 |
corvus | plz can haz config managed ssh? kthxbye | 19:25 |
yoctozepto | corvus: where | 19:27 |
corvus | yoctozepto: https://review.opendev.org/#/c/1/ | 19:27 |
corvus | yoctozepto: the first change in gerrit was adding some really basic puppet config | 19:27 |
corvus | because, as fungi says, it was (much) better than nothing | 19:28 |
fungi | also a prescient change to implement in light of modern situations like https://www.guardicore.com/2020/08/fritzfrog-p2p-botnet-infects-ssh-servers/ | 19:29 |
*** moppy has quit IRC | 19:29 | |
*** yoctozepto1 has joined #opendev | 19:30 | |
fungi | at this point anyone who is running a public ssh with even moderately weak passwords is completely pwn3d and has been for rather a while | 19:30 |
*** moppy has joined #opendev | 19:30 | |
yoctozepto1 | corvus: "Also Puppet should be capitalized" :D | 19:31 |
*** yoctozepto has quit IRC | 19:32 | |
*** yoctozepto1 is now known as yoctozepto | 19:32 | |
corvus | yeah, it's kind of a graffiti magnet, but that's okay :) | 19:32 |
*** jhesketh_ has joined #opendev | 19:38 | |
*** shtepanie has quit IRC | 19:42 | |
*** jhesketh has quit IRC | 19:42 | |
*** mlavalle has quit IRC | 19:42 | |
*** mlavalle has joined #opendev | 19:48 | |
clarkb | fungi: I've +2'd the ethercalc change but not approved it | 19:55 |
clarkb | fungi: I'm going to ride the bike shortly so won't be around if it lands soon | 19:56 |
clarkb | but feel free to land it when you are able to watch it (and probably do db dump?) | 19:56 |
fungi | clarkb: thanks, i'm going to start the dump now | 20:05 |
fungi | oh, actually we don't need to create the dump, redis does that continually | 20:08 |
fungi | -rw-rw---- 1 redis redis 31078548 Aug 21 20:06 /var/lib/redis/dump.rdb | 20:09 |
fungi | just need to back it up | 20:10 |
openstackgerrit | Merged opendev/system-config master: Run service-eavesdrop hourly https://review.opendev.org/746181 | 20:12 |
clarkb | ok popping out now. Back in $timeperiod | 20:17 |
*** tosky has joined #opendev | 20:30 | |
openstackgerrit | Merged opendev/puppet-ethercalc master: Upgrade to latest EtherCalc (0.20200505.0) https://review.opendev.org/747453 | 20:37 |
fungi | and now we wait for the hourly puppet-else run | 20:38 |
fungi | looks like the run which was in progress did not update it (or at least didn't restart the service) but there's a fresh one queued up as a few minutes ago | 21:04 |
fungi | hopefully it'll finish around the time i finish eating dinner | 21:05 |
*** owalsh has quit IRC | 21:25 | |
fungi | looks like it updated, but didn't restart the service | 22:05 |
fungi | made one more redis dump copy and am manually restarting ethercalc now | 22:07 |
fungi | and it's working again | 22:08 |
fungi | anyone please feel free to check existing sheets you have urls for handy, and/or try creating new sheets | 22:08 |
corvus | i just created https://ethercalc.openstack.org/wwcw19levbug | 22:09 |
fungi | current ptg scheduling sheet still has content: https://ethercalc.openstack.org/126u8ek25noy | 22:10 |
fungi | er, no that's the previous one | 22:10 |
fungi | but looks right | 22:10 |
corvus | i see lots of familiar code words so i guess that works | 22:11 |
fungi | this is the newer one, also still looks right: https://ethercalc.openstack.org/7xp2pcbh1ncb | 22:11 |
clarkb | the opendev timeslots look correct on ^ | 22:30 |
*** mlavalle has quit IRC | 23:02 | |
*** DSpider has quit IRC | 23:40 | |
*** tosky has quit IRC | 23:48 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!