Timestamps are in GMT/BST.
[20:58] * hapi (~PircBot@newton.ncc.edu) has joined #freenet
[20:58] * Topic is 'Stable: Upgrade to 5099 (5099 is mandatory) (this is a stable network reset) | Unstable: Upgrade to 60241 (60234 is mandatory) | Channel logs: http://newton.matcmp.ncc.edu/~lockej/freenet/chanlog/ | #freenet-politics and #freenet-chat are available for offtopic discussions'
[20:58] * Set by greycat on Tue Oct 26 14:47:24 EDT 2004
[21:08] * dystopia_ (admin@gaia-wan0.ipv6.freeshell.bofx.net) has joined #freenet
[21:08] * greycat (rfc1413@wooledge.org) has joined #freenet
[21:09] <greycat> RNF city. :/
[21:17] <toad_> :<
[21:17] <KenMan> simworld is much happier :o
[21:18] <KenMan> i managed to create some RNFs earlier, but very few...
[21:18] <KenMan> this was with only 5 routes/peer , so i guess a lot of looping was going on...
[21:19] * KenMan can hear toad's lips curling from here !
[21:20] <KenMan> "only 5 routes, that is so ridiculous, it doesn't even warrant an attempt"
[21:21] <toad_> did i say that?
[21:21] <KenMan> i heard you thinking it ;)
[21:21] <toad_> hmmm
[21:21] <KenMan> hey, i only joke around with you...
[21:22] <toad_> can a freesite have an icon?
[21:22] <toad_> i mean one the web browser uses?
[21:23] <KenMan> i don't think so, isn't that usually fetched from the webserver independently ?
[21:23] <toad_> anyone got CofE yet?
[21:23] * toad_ has a DNF...
[21:23] <KenMan> if he hasn't published for a month, what makes you think he did today ?
[21:24] <KenMan> perhaps you are too reliant on his reports, although I don't see many alternatives...
[21:24] <toad_> his last few publications have been commenting on builds
[21:25] <toad_> especially with regards to insert suckiness
[21:25] <KenMan> hey, when do you brits wind your clocks around ? we do it next sunday here in USA
[21:26] <toad_> wooh, i've probably got FIND...
[21:26] <toad_> KenMan: the same i think
[21:26] <KenMan> naw, that would be too convenient .
[21:27] <KenMan> I just hit it. I cut off inserts after 10000 unique keys, there are 1M store slots. Beyond this point, the number of unique keys would grow very slowly.
[21:27] <KenMan> 100 redundant key copies is a little disgusting, but hey - this is an experiment to see where success goes now.
[21:28] * dystopia_ (admin@gaia-wan0.ipv6.freeshell.bofx.net) Quit (Read error: 104 (Connection reset by peer))
[21:29] * toad_ hmmz, "coup de gueule" means "blow to the mouth"... that would be a french porn site, then?
[21:29] <KenMan> oops, my new code to dump the reference counts for each key threw an exception :( must fix & restart...
[21:29] <toad_> what's DEMASC ?
[21:29] * lolo-laptop (~lostlogic@68.251.84.226) Quit (Client Quit)
[21:29] <KenMan> did CofE send a coup de gueule at FreeNet today ?
[21:30] <KenMan> Democrats Against Social Conservatism
[21:30] <toad_> ugh, babelfish sucks... what does "Qui veut tuer son prochain dit qu'il est raciste" mean?
[21:31] <toad_> and thought crime has been updated for the first time in aaages...
[21:33] * toad_ hmmz
[21:34] <toad_> FIND links to thoughtcrime, apparently updated.. edition 5
[21:34] <toad_> :<
[21:36] <KenMan> "if you wish to say this it means you are a racist" - that's my weak translation - any better than babbellinmg fish?
[21:36] <toad_> yeah a bit
[21:38] <i2p_iip> <DrWoo> or "he who wishes to kill my words, he is a racist" ?
[21:38] <i2p_iip> <DrWoo> or "he who wishes to kill my words, he is a racist" ?
[21:39] <KenMan> i'll go with that, although which word means 'to kill' ??
[21:39] <KenMan> or to censor, or to squelch, or whatever ?
[21:39] <i2p_iip> <DrWoo> tuer
[21:40] <KenMan> thank you :)
[21:40] <KenMan> i guess i didn't quite capture the intended message, huh ?
[21:40] <toad_> as in "je voudrais le tuer" ?
[21:40] <toad_> i would like to kill it?
[21:41] <toad_> (or would that be ce)
[21:41] <toad_> whatever
[21:41] * greycat prefers spanish. much nicer language.
[21:42] * toad_ hmmz
[21:42] <toad_> hypotheses:
[21:42] <toad_> 1. the rats are abandoning the sinking ship. Newsbyte's propaganda piece did more damage than expected.
[21:43] <toad_> 2. inserts suck on the current network even worse than before.
[21:43] <toad_> 3. requests for new content suck at least as bad as before.
[21:43] <toad_> there are probably more theories...
[21:44] * goatee (~goatee@ip216-239-88-4.vif.net) has joined #freenet
[21:44] <KenMan> it is because my Super Happy Mega Average Node is not participating !
[21:44] <KenMan> hi goatie
[21:44] <toad_> but something is annoying anyway
[21:44] <toad_> 4. people didn't upgrade
[21:45] <KenMan> shouldn't inserts work better on a smaller network ? ah, no point in trying to guess that one!
[21:45] <toad_> root@amphibian:~# cat seednodes.ref | grep version= | sort | uniq -c
[21:45] <toad_> 2 version=Fred,0.5,STABLE-1.50,5070
[21:45] <toad_> 1 version=Fred,0.5,STABLE-1.50,5072
[21:45] <toad_> 11 version=Fred,0.5,STABLE-1.50,5076
[21:45] <toad_> 11 version=Fred,0.5,STABLE-1.50,5090
[21:45] <toad_> 24 version=Fred,0.5,STABLE-1.50,5091
[21:45] <toad_> 3 version=Fred,0.5,STABLE-1.50,5095
[21:45] <toad_> 79 version=Fred,0.5,STABLE-1.50,5096
[21:45] <toad_> 183 version=Fred,0.5,STABLE-1.51,5099
[21:45] <toad_> woah
[21:45] <toad_> they HAVE been upgrading
[21:46] <KenMan> hee hee
[21:46] <KenMan> yes, but do people often wipe away their node ID when doing many restarts because they don't really know what they are doing ? I mean, are there that many right now ?
[21:46] <KenMan> somebody quoted 80+ earlier - might have been the toad saying this
[21:46] <toad_> my 80 thread ISO insert id definitely not upgrading...
[21:47] <toad_> err
[21:47] <toad_> not working well
[21:47] <toad_> not succeeding
[21:47] <toad_> just goes round and round, restarting...
[21:47] <greycat> RETRY #23: Greg_Wooledge/archive.html
[21:47] <greycat> RETRY #24: Greg_Wooledge/archive.html
[21:47] <greycat> RNF: file #11/133 archive.html (154567, text/html)
[21:47] <greycat> ^^^ typical result.
[21:48] <toad_> where are all the RNFs coming from? I thought we'd gotten rid of them?
[21:48] <toad_> greycat: did it RNF on each try?
[21:48] <KenMan> heh, yeah, right !
[21:48] <greycat> <advertisement> yes, FT 0.2.5 stops after 25 tries by default, user configurable. new feature, just added today! </advertisement>
[21:48] <toad_> and is that big or small?
[21:49] <greycat> I'll start a new single-file insert with debug-level logging on
[21:49] <toad_> are there still insert bugs? or is there a major overload issue, and why? are both of the above correct?
[21:50] <greycat> it looks like a network issue more than an "insert" issue at this point.
[21:50] <toad_> have we in fact regressed?
[21:50] <KenMan> RNFs mainly come from the queueing vs mRI thingy, no ?
[21:50] <greycat> just lots and lots of RNFs, on both retrievals and insertions
[21:50] <greycat> bail: seen RouteNotFound
[21:50] <greycat> RETRY #1: filename
[21:50] <toad_> and why can i smell paint from up here ?? :)
[21:50] <greycat> super toad nose powers?
[21:50] <toad_> greycat: surreal...
[21:51] <KenMan> you've been sniffing that model airplane glue for so long, your damaged brain is confusing it with pain
[21:51] <KenMan> t
[21:51] * toad_ stops the insert
[21:51] <greycat> bail: seen RouteNotFound
[21:51] * toad_ wonders if that's it :)
[21:51] <greycat> RETRY #2: filename
[21:51] <toad_> i mean the insert
[21:51] <toad_> maybe i've been abusing the network (because nobody else does! yeah sure!)
[21:52] <greycat> it's just vigorous testing....
[21:52] <toad_> resetting the network leads to more traffic, more RNFs? or is there a bug?
[21:52] <greycat> oooh, scoexecs.png went in after only 10 retries!
[21:53] <toad_> it was working reasonably before we rolled out too :|
[21:53] <toad_> and if it's getting all the RNFs you say that's worse than 5096
[21:53] <greycat> it actually wasn't *quite* so bad about 8 hours ago
[21:53] <toad_> which I don't understand
[21:53] <toad_> who can understand the ways of freenet? ;|
[21:55] <i2p_iip> <DrWoo> toad: at least you keep a good atitude about it, you haven't stressed out heh
[21:56] <KenMan> toads are capable of perfectly concealing their stress
[21:56] <greycat> imperfectly capable.
[21:56] <KenMan> I mean, have you ever picked up a facial expression from a toad ?
[21:56] * greycat remembers some heated exchanges involving the toad on occasion :)
[21:57] <KenMan> no, toad is realistic. He knows we can't fix this all in a day. But he takes his best calculated step forward.
[21:58] <toad_> and usually discovers it's actually three steps backwards
[21:59] <toad_> 99 little bugs in the code, 99 bugs in the code... squash one bug, compile it again, 101 bugs in the code... and that's just the simple bugs, there are deeper issues too and they have similar problems
[22:00] <KenMan> that's only because you can rotate your eyes in their sockets to actually be LOOKING backwards...
[22:01] <toad_> nobody has CofE? or any of the bookmark sites with the exception of FIND?
[22:01] <KenMan> go to bed once you get 3 negative responses.
[22:01] <toad_> good idea
[22:03] <toad_> but why does it suck so much? oh well, maybe find out tomorrow?
[22:03] <toad_> bbl zzzzzzzzzzzzzz
[22:03] <KenMan> I believe that the RNF syndrome is a roaming ghost, which only preys upon a percent of the network at any given time.
[22:04] <KenMan> night toad. I hope RNF has not become a global problem.
[22:04] <greycat> one theory is that those of us who adopted 5099 early became the seed nodes for the new reset network, and now we're completely overloaded
[22:05] <toad_> KenMan: i don't see why you'd think that, but if you have a more concrete theory email me it :)
[22:05] <toad_> bbl
[22:05] <toad_> another theory is there's some sort of transfer or query multiplication bug in inserts which is causing megaload
[22:06] <KenMan> I determined that my node, and some of its peers, drifted into and out of a state where it would RNF a high percentage of requests. We tracked it down to mRI, and then built queueing in response. No ?
[22:06] <toad_> when i had the 40 thread insert running, i had nearly 300 outbound transfers, with 3 inbound
[22:06] <greycat> Connections open (Inbound/Outbound/Limit) 16 (0/16/16)
[22:06] <greycat> Transfers active (Transmitting/Receiving) 23 (10/13)
[22:06] <toad_> but afterwards, without restarting the node, it went down
[22:07] <toad_> to about that
[22:07] <toad_> Connections open (Inbound/Outbound/Limit) 77 (22/55/200)
[22:07] <toad_> Transfers active (Transmitting/Receiving) 397 (389/8)
[22:07] <toad_> woah
[22:07] <toad_> same thing again
[22:08] * toad_ wonders whether we'd save a LOT of bandwidth by waiting for the InsertReply before sending the DataInsert...
[22:08] * lostlogic (~lostlogic@node-4024215a.mdw.onnet.us.uu.net) has joined #freenet
[22:08] * toad_ wonders whether we'd save a LOT of bandwidth by waiting for the InsertReply before sending the DataInsert...
[22:09] <toad_> of course we'd have to change the timeouts again :|
[22:09] <greycat> {18214} sent [0x2d4da bytes data]
[22:09] <greycat> {18214} rcvd Restarted
[22:09] <greycat> {18214} rcvd EndMessage
[22:09] <greycat> {18214} rcvd RouteNotFound
[22:09] <greycat> {18214} rcvd EndMessage
[22:09] <greycat> bail: seen RouteNotFound
[22:09] <greycat> RETRY #19: filename
[22:09] <KenMan> I determined that my node, and some of its peers, drifted into and out of a state where it would RNF a high percentage of requests. We tracked it down to mRI, and then built queueing in response. No ?
[22:10] <KenMan> sorry, kb error
[22:10] <KenMan> well, okay, the problem was really with the user, but it's nice to shift the blame...
[22:10] <toad_> ;)
[22:11] <KenMan> what are the RNF:DNF message counts these days ?
[22:12] <KenMan> they were very comparable pre-queueing, then RNF should have gone down, DNF gone up... I never saw a significant change, but that was the expectation.
[22:13] <toad_> scenario
[22:13] <toad_> we send an insert to node A
[22:13] <toad_> which accepts it
[22:14] <toad_> we then send the DataInsert
[22:14] <toad_> it sends to B, C, D,... and eventually RNFs
[22:14] <toad_> but for many of those it will have sent much of the data
[22:14] <toad_> we then send to E, which also RNFs
[22:14] <toad_> and C, which also QRs, because of loop
[22:15] <toad_> all the nodes which don't instantly RNF get some of the data
[22:15] <toad_> if they take a while to RNF, then they get lots of it
[22:15] <toad_> and even when we cancel the sends, we have to keep the conn open while we pad the transfer out
[22:15] <toad_> until the next block
[22:15] <toad_> result:
[22:16] <toad_> a node doing lots of inserts gets a hell of a lot of transmits
[22:16] <toad_> and we have a HUGE amount of traffic from one insert
[22:16] <KenMan> that's a sucky scenario - can't you depict something more pleasant ? :p
[22:16] <greycat> but what causes the RNFs in the first place? overload?
[22:17] <toad_> well, traffic causes overload
[22:17] <toad_> so perhaps we should:
[22:17] <toad_> a) Look into not padding on a transfer failure (there may be security issues with this?)
[22:17] <toad_> b) Not send the data at all until we have somewhere to send it i.e. the InsertReply
[22:18] <KenMan> didn't we used to wait until the full chain was prepared to accept the insert ? now we only wait for the next hop to be ready ?
[22:18] <toad_> i suspect so...
[22:18] * lostlogic (~lostlogic@node-4024215a.mdw.onnet.us.uu.net) Quit ("Going to the moon")
[22:19] <KenMan> I mean, that seems fair to do, but ... you are looking at the results so far. Don't worry, you will fix. I have faith in toad.
[22:19] <toad_> although arguably there's a timing/anon issue with going all the way...
[22:19] <KenMan> certainly there is
[22:20] <KenMan> but you give up the opportunity to confirm to the user that his insert actually went 17 hops...
[22:20] <KenMan> and completed.
[22:20] * Ash-Fox (Hal-9000@aae188.neoplus.adsl.tpnet.pl) Quit (Client Quit)
[22:20] <toad_> huh?
[22:20] <greycat> SUCCESS: file #104/133 ryan2.jpg (179277, image/jpeg)
[22:20] <greycat> COLLISION: file #11/133 archive.html (154567, text/html)
[22:20] <greycat> **** INSERTING METADATA FOR MAP SPACE (19448 BYTES)
[22:20] <greycat> wow....
[22:21] <toad_> no i mean if we just don't send the data until we get an InsertReply
[22:21] <toad_> i.e. the full chain
[22:21] <toad_> the problem with that is security
[22:21] <toad_> it introduces a timing attack
[22:21] <greycat> SSK@wlUk5niRuNDUL4lzvdHu6DamhW8PAgM/Greg_Wooledge// is now inserted.
[22:21] <KenMan> yes... just store and forward. Check HTL. User doesn't get "proof" that it went all the way...
[22:22] <toad_> store on who and forward to who, when?
[22:22] <KenMan> fro each hop, bro
[22:22] <toad_> also issues with spamming
[22:22] <KenMan> each node only gets one confirmed hop. And he can't even trust that node.
[22:23] <KenMan> But, we live with it. When would a node be motivated to prematurely terminate an insert ?
[22:23] <toad_> presumably you mean don't report back to the user at all, store it, and queue it if necessary for a REALLY long time, so you don't need to do the whole tree search with data transfer thing that uses all the bandwidth
[22:23] <toad_> KenMan: well, you can't trust them NOW
[22:23] <toad_> however it might be possible to build some sort of verification protocol
[22:23] <toad_> but right now inserts are completely insecure
[22:23] <toad_> malicious node can just swallow them and make the right noises
[22:24] <toad_> no comeback
[22:24] <KenMan> once Node A has finished transferring to Node B, Node A is done. A need not wait for D to accept the insert...
[22:24] <toad_> KenMan: that means no backtracking
[22:24] <toad_> no backtracking means B will hold onto the data for as long as it takes to find somewhere to send it to
[22:25] <KenMan> yup - should work acceptably in the majority of cases, no ?
[22:25] <toad_> you would need some form of coalescing to make flooding less useful
[22:25] <KenMan> inserts deserve higher priority than requests, but at what cost ?
[22:26] <toad_> and i still think that some sort of completion notification would be useful, if only to verify with
[22:26] <KenMan> higher priority in routing accuracy anyways, maybe not transfer/bw prio
[22:26] <toad_> doing it async would be a PITA for authors
[22:26] <KenMan> because they would have to go check whether it 'got all the way there' ??
[22:27] <KenMan> ie they would have to use a 2nd node to test retrievability ?
[22:27] <toad_> because they would have no way to know when it's finished
[22:27] <toad_> and therefore you can't have an automated tool that just does everything and then finishes
[22:27] <KenMan> ok
[22:28] <KenMan> you don't want searchers to begin their search before at least HTL copies of item X exists ?
[22:29] <KenMan> as that could dirty the per node fail tables ?
[22:29] <toad_> hmmm?
[22:29] <toad_> perhaps
[22:29] <toad_> but not just that
[22:29] <KenMan> you don't want to upset the lives of the inserters. Got it.
[22:29] <toad_> it would certainly mess up the existing fts
[22:30] <toad_> right
[22:30] <KenMan> where were we, where are we, and where do we need to go ?
[22:30] <toad_> well, we think that inserts are using a hell of a lot of bandwidth
[22:30] <toad_> that's my hunch
[22:30] <toad_> it could be wrong
[22:31] <toad_> but the idea is that if the network is overloaded, inserts make it worse, and keep it worse
[22:31] <KenMan> and there is no way to confirm this, like usual... :(
[22:31] <toad_> there may be - tomorrow
[22:31] <toad_> one solution would be to not send the data until we get the InsertReply
[22:31] <toad_> but if we do that, then it opens up a timing attack against the requestor
[22:32] <KenMan> weren't we already there ?
[22:32] <toad_> the original insertor!
[22:32] <toad_> which is VERY bad
[22:32] <toad_> fire-and-forget inserts make life hard for insertors, client authors, and so on
[22:33] <toad_> because you can't verify them if you don't know when they completed!
[22:33] * verl (~verlverl@h152n2fls33o877.telia.com) Quit ()
[22:33] <KenMan> intentional delay of up to a few seconds on the Reply, per node, would hurt, help, or neither ??
[22:34] <toad_> why would it help?
[22:34] <KenMan> You don't *quite* sound committed to either approach, although you recognize the pain of fire and forget...
[22:34] <toad_> anyway it's queued
[22:34] <KenMan> it might help thwart timing attacks, bury it in the traffic...
[22:35] <toad_> no it wouldn't
[22:35] <KenMan> but it would not be much defense against an omnipotent opponent :|
[22:35] <KenMan> the all-seeing, all-powerful eyeball :)
[22:35] <toad_> the problem is that if B sends A an InsertReply and then A immediately sends a DataInsert back, B knows A is the original source of the request
[22:35] <toad_> err insert
[22:35] <toad_> KenMan: we don't have to make it proof against the Lidless Eye
[22:36] <KenMan> good
[22:37] <KenMan> yeah, and throughput will tend to decrease with hop distance also, no ?
[22:37] <toad_> one possibility would be to send the insert out in a couple of directions on a fire-and-forget basis, but first send out a couple of passive requests for the content
[22:37] <toad_> so if the passive requests are triggered, we know it's completed
[22:37] <toad_> otherwise we wait for a timeout
[22:38] <toad_> there are probably issues with that
[22:38] <KenMan> there are probably issues with every choice !!
[22:38] <toad_> yes, there are
[22:38] <toad_> i mean anon issues
[22:38] <KenMan> i know...
[22:39] * KenMan succumbs to mild hunger...
[22:39] <toad_> you could perhaps set up the passive requests way in advance...
[22:40] <toad_> but that's not very practical
[22:40] <toad_> you end up having to implement not only passive requests but also premix routing to do it safely...
[22:40] <toad_> and it's still debatable
[22:41] <toad_> it's very, very preferable to have some sort of completion notification...
[22:42] <toad_> although you could argue that delayed store and forward is important
[22:44] * dystopia_ (admin@gaia-wan0.ipv6.freeshell.bofx.net) has joined #freenet
[22:44] <toad_> i dunno, maybe this is all bull.. maybe we can just make the transfers get killed immediately rather than padding and that'll sort out half the problems
[22:45] <toad_> in terms of vast numbers of incomplete sends slowing things down
[22:45] <toad_> bbl zzz anyway
[22:47] * verl (~verlverl@h152n2fls33o877.telia.com) has joined #freenet
[22:50] * dystopia_ (admin@gaia-wan0.ipv6.freeshell.bofx.net) Quit (Remote closed the connection)
[22:52] <KenMan> ah hah! Stopping the inserts after a certain point does make success increase again.
[22:56] * dystopia_ (~admin@gaia-wan0.ipv6.freeshell.bofx.net) has joined #freenet
[22:56] <KenMan> it makes success rise slowly, while keeping the average redundancy at a very constant level. Like, with 10000 keys in 1M DS slots, success went from 63% to 77%, after an additional 1.5M requests
[22:56] <KenMan> 1000 nodes, RT of 25, HTL of 8, DS=1000keys
[22:57] <KenMan> this is with a static/fixed set of connections
[22:58] <KenMan> also, i dumped the redundancy data, so I can look and see the distribution of key redundancy per cycle.
[22:59] * plixed_ (~plixed@c160022.adsl.hansenet.de) Quit (Connection timed out)
[23:06] * dystopia_ is now known as dystopia
[23:06] * tessier_ (~treed@wsip-68-224-172-77.sd.sd.cox.net) has left #freenet
[23:07] * greycat (rfc1413@wooledge.org) Quit ("Lick Bush in '04")
[23:09] * dystopia (~admin@gaia-wan0.ipv6.freeshell.bofx.net) Quit (Remote closed the connection)
[23:13] * dystopia (admin@gaia-wan0.ipv6.freeshell.bofx.net) has joined #freenet
[23:16] * dystopia_ (admin@gaia-wan0.ipv6.freeshell.bofx.net) has joined #freenet
[23:21] * dystopia (admin@gaia-wan0.ipv6.freeshell.bofx.net) Quit (Read error: 113 (No route to host))
[23:24] * Usurp (syl@trudaine-8-82-230-34-86.fbx.proxad.net) has joined #freenet
[23:24] * lostlogic (~lostlogic@node-4024215a.mdw.onnet.us.uu.net) has joined #freenet
[23:26] * dystopia_ (admin@gaia-wan0.ipv6.freeshell.bofx.net) Quit (Read error: 54 (Connection reset by peer))
[23:27] * dystopia (admin@gaia-wan0.ipv6.freeshell.bofx.net) has joined #freenet
[23:50] * dystopia (admin@gaia-wan0.ipv6.freeshell.bofx.net) Quit (Remote closed the connection)
[23:51] * dystopia (admin@gaia-wan0.ipv6.freeshell.bofx.net) has joined #freenet
[23:52] <Redb3ard> let me find them
[23:53] <Redb3ard> yeh
These logs were automatically created by Jay Oliveri with his gimp hapi on irc.freenode.net.