Timestamps are in GMT/BST.
[0:43] * Knio (~Knio@d205-206-115-116.abhsia.telus.net) Quit (Read error: 104 (Connection reset by peer))
[1:03] * Zorix (Brandon@fl-65-41-1-233.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[1:05] * salahx (salahx@sc1-24.217.174.147.charter-stl.com) has joined #freenet
[1:41] * oierw` (mathew@cpe-69-75-126-12.hawaii.rr.com) has joined #freenet
[1:42] * oierw` (mathew@cpe-69-75-126-12.hawaii.rr.com) Quit (Remote closed the connection)
[1:42] * oierw` (mathew@cpe-69-75-126-12.hawaii.rr.com) has joined #freenet
[1:47] * Ash-Fox (Hal-9000@abd72.neoplus.adsl.tpnet.pl) has joined #FreeNET
[1:58] * Zorix (Brandon@fl-65-41-1-233.dyn.sprint-hsd.net) has joined #freenet
[2:18] * jay (jcl@ool-18bf6dac.dyn.optonline.net) Quit ("Client exiting")
[2:22] * piranha (piranha@3ffe:b80:1ca1:0:0:0:deca:fbad) Quit (Read error: 101 (Network is unreachable))
[2:36] * JustMe_ (JustMe_@cs6669187-74.houston.rr.com) Quit (Read error: 110 (Connection timed out))
[2:38] * Zorix (Brandon@fl-65-41-1-233.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[2:44] <KenMan> 5090 - recvData (436/729)=58% , sentData (633/1486)=43%
[2:44] <salahx> that good ?
[2:45] <KenMan> that probably fairly normal for a long time.
[2:46] <salahx> I sitll have void address in my RT though
[2:48] <KenMan> for once, no 'void' routes here :o also, with 5089/90 , number of routes seems to stay closer (or equal!) to number of peers
[2:49] * TLF (francisco@245.Red-81-40-116.pooles.rima-tde.net) has joined #freenet
[2:49] <salahx> i figured the automatic IP Address code was supposed to stop that void stuff ?
[2:49] <KenMan> overall , things look pretty decent on 5090. My only concern is the level of RT backoff - it averages about 90% so far !
[2:50] <KenMan> salahx: I imagine there are 5 other likely explanations for void stuff. Hopefully it gets tracked down, eventually.
[2:50] <salahx> ok
[2:50] <salahx> th'ats gonna be ahrd one to solve though
[2:50] <KenMan> yup
[2:50] <salahx> i.e. % RT backed up
[2:51] <KenMan> definitely not as simple to fix as one would like, but it is a critical system performance measurement.
[2:52] <KenMan> assuming that NGR can do *anything* more accurate than fully random routing
[2:52] <KenMan> Toad built a table of success versus accuracy of routing (regarding backoffCount) and it seemed to show that NGR actually does something.
[2:53] <KenMan> Ian refused to accept it at all. For me, it was impossible to ignore.
[2:54] <KenMan> i don't know if you remember it ? It showed that depending on the level of backoff, success went lower as backoff got higher...
[2:55] <salahx> dunno
[2:55] <salahx> I'm surei ts doing something though
[2:55] <KenMan> and the best success was achieved if NGR could get one of the top 3-4 choices , without the backoff/MRI code forcing a lower-choice route.
[2:56] <salahx> but when any routing system, is chronically overloaded (like Freenet is), requests will tend to take the less busy path, becasue there's usually nowhere else to go!
[2:56] <KenMan> thus we have success rates of 1-4 % :p
[2:56] <salahx> yep
[2:57] <KenMan> i think 10% is certainly an acheivable goal, and will reflect a key maturation of FreeNet.
[2:57] <KenMan> I'm pretty confident the project will get there. But this year, in 2005, or in 2006 ? Who knows !!
[2:58] <salahx> yep
[2:58] <KenMan> Just like Toad, and many other people, I have lots of intuitions, but it is really really hard to prove or disprove most of them.
[2:59] <salahx> there network really really help findign the bugs though
[2:59] <salahx> err network resets
[3:00] <KenMan> massive change tends to have that effect :) seems to flush some things out into the light ...
[3:01] <KenMan> we could learn an awful lot (find bugs) just by experimenting with wild configurations, with no development. For quite a long time i imagine.
[3:04] * thelema|away (~thelema@adsl-65-65-202-46.dsl.stlsmo.swbell.net) Quit (Read error: 110 (Connection timed out))
[3:06] <KenMan> incoming query success (739/23997)=3% , outgoing query success (357/21207)=1.6% :(
[3:11] <salahx> eh
[3:13] <salahx> riotuer to choice rank is better
[3:13] <salahx> err routedToChoiceRank
[3:14] <salahx> in the low 20's, was in the 80's before...
[3:18] <salahx> tha'ts off
[3:18] <salahx> errr odd
[3:18] <salahx> I jsut tried to retrive FIND and got this:
[3:18] <salahx> Warning: Unknown mime type application/zip
[3:18] <salahx>
[3:18] <salahx>
[3:18] <salahx>
[3:18] <salahx> You have retrieved some content which is not recognised by FProxy,
[3:19] <salahx> TFE is doing it too...
[3:20] <salahx> So is YoYo
[3:20] <KenMan> how many connections do you have, in order to get numbers in the 20's ?? I avg about 55 now, with 70 peers.
[3:21] <salahx> like 80 or so
[3:21] <KenMan> but this one is muddled, as it includes unconnected routes in routedToChoiceRank.
[3:21] <KenMan> you running stable or unstable ?
[3:21] <salahx> stable
[3:21] <KenMan> wow, that is confusing. outgoing bandwidth ?
[3:21] <salahx> default
[3:22] <KenMan> hmmm, that is very curious. I'm about 1.5x (18000/s) default.
[3:23] <KenMan> what you getting for searchFailedCount and backedOffCount ?
[3:23] <KenMan> wait, was you looking at rTCR / min or /hr ??
[3:23] <salahx> hr
[3:23] <KenMan> woah. that's what i assumed.
[3:24] <KenMan> what are some of your values from the max column for rTCR ?? I'm 150-160 (2x 80)
[3:25] <salahx> tis gone up - its around 3-4 (searchFailedCount)
[3:25] <salahx> backedOffCOunt is ~15
[3:26] <KenMan> sFC ~ 1.0 -1.4 here, backedOffCount ~ 27-30 ...
[3:26] <salahx> max's are almost 200
[3:27] <KenMan> our numbers don't agree. These 3 measurements should float up and down in rough sync with each other
[3:27] <salahx> well my node only been up 2 hours with 5090
[3:27] <KenMan> mine about 8 with 5090...
[3:28] <salahx> i'm trying ot download a splitfile but i think too much of it has become "unreachable"
[3:31] <salahx> I hestatie to say "fallen off" becasue data does;nt fall off Freenet easily, usually hwne htere a "netiowrk reset" suddenly data that was thoguh to ahve "fallen off" sudden is aviable again
[3:32] <KenMan> :)
[3:33] <KenMan> it seems the vast majority of queries result in RNF. Unless I am missing something, this can only be caused by two events - first, the request has looped back to a node that is already in the request chain. Second, the node has 100% of routes backed off at the time of this specific routing attempt.
[3:33] <KenMan> I expect that the second event is far more prevalent than the first, but we currently have no way to find out.
[3:34] <salahx> I think we send a QR if a quest loops back to us
[3:34] <salahx> it almost certianyl the 2nd
[3:34] <KenMan> an RNF is encoded as a QR
[3:35] <salahx> I don;t think we send an RNF in the 1st case
[3:35] <KenMan> ahh, I will have to verify then.
[3:36] <KenMan> there could be some nodes out there (dial up modem with 200 active connections) that simply can't help but RNF 99.8% of all requests, due to backoff.
[3:36] <KenMan> in other words, severe routing congestion could exist only on certain nodes, and this would hurt the net
[3:38] <KenMan> or, statistically, many nodes could be choking on a small enough percent that it results in most requests hitting backoff congestion somewhere along the 20HTL path, even if no node is congested-RNF'ing a majority of its requests.
[3:38] * theLostFloppy (~michaelku@fia41-111.dsl.hccnet.nl) has joined #freenet
[3:39] <KenMan> plus, the whole "retry 2 or 3 times on RNF" thing confuses the matter
[3:39] <KenMan> no, I am certain that request looping detection results in an RNF. Toad reminded me in an embarassing conversation several weeks ago.
[3:39] * salahx (salahx@sc1-24.217.174.147.charter-stl.com) Quit ()
[3:45] * dgrant (~dgrant@d154-20-147-177.bchsia.telus.net) has joined #freenet
[3:48] <dgrant> Is anyone able to actually access anything on freenet?
[3:52] <KenMan> someone must be, or everyone would have packed up and gone home years ago...
[3:52] <KenMan> but it does require patience. Lots and lot of patience.
[3:58] * Zorix (Brandon@fl-65-41-1-233.dyn.sprint-hsd.net) has joined #freenet
[3:59] * thelema|away (~thelema@adsl-65-65-201-216.dsl.stlsmo.swbell.net) has joined #freenet
[4:01] <dgrant> KenMan: What is the most accessible freenode site
[4:01] <KenMan> you mean freenet. freenode is this IRC system. I don't do much browsing myself, but sometimes I look at Content Of Evil.
[4:02] <KenMan> It is only updated every couple of days (lately), but I usually can get it. However, not usually on my first try.
[4:03] <KenMan> The Freedom Engine should also be accessible.
[4:03] <theLostFloppy> AAAAAHH! This stupid impostor is back.
[4:03] <theLostFloppy> Why oh why did he pick *me*
[4:03] <KenMan> Me ? or you ?
[4:03] <KenMan> o, someone done tooked your nick ??
[4:03] <theLostFloppy> Someone posting goatse.cx ascii all the time with my name@identity
[4:04] <theLostFloppy> At FROST, I mean
[4:04] <KenMan> don't worry, identity theft on FreeNet is meaningless ... HAHAHA
[4:04] <theLostFloppy> What is that suposed to mean?
[4:04] <dgrant> KenMan: yeah I meant freenet, not freenode. thanks
[4:05] <KenMan> it is just a light hearted joke. Don't get paranoid and think I'm the one impersonating you...
[4:05] <KenMan> I've run frost a few times, and seen what people are calling spam.
[4:05] <theLostFloppy> I didn't think that
[4:05] <theLostFloppy> [x] Hide unsigned messages
[4:06] <dgrant> KenMan: yeah I can get on freedom engine usually, but only a few sites from there work. trying to load content of evil now, finally loaded
[4:06] <KenMan> the solution in Frost is to sign all your messages. Problem solved.
[4:06] <theLostFloppy> I always sign
[4:06] <theLostFloppy> But he copies the NAME AND EXCEPT FOR TWO CHARACTERS MY SIG.
[4:06] <theLostFloppy> ----- LostFloppy@zqhvAzYcP0RRlQWxml6xR1V0EQU ----- 2004.08.07 - 07:34:34GMT -----
[4:06] <theLostFloppy> Oops
[4:07] <theLostFloppy> My messages appear not signed, but still by me
[4:07] <theLostFloppy> I was wrong, the part at the @ is not the sig, and he just c&p'd it
[4:09] <dgrant> KenMan: how many days would have to pass until my node becomes the best it can be
[4:10] <KenMan> currently, it's looking like it may be several hundred days. As more development is needed. With the current code, a day or two is the longest it can take.
[4:10] <dgrant> What is the best JVM to use? I'm using Blackdown right now. I'm wondering if Sun would be better for any reaosn
[4:11] <KenMan> quite prossibly Sun would give you better results. Blackdown has caused some bugs for some people, i think. Maybe it is okay though.
[4:11] * thelema|away (~thelema@adsl-65-65-201-216.dsl.stlsmo.swbell.net) Quit (Read error: 104 (Connection reset by peer))
[4:12] <KenMan> the "best" JVM to use is SUN 1.4.2 , as that is what the developers and testers are using
[4:12] <dgrant> There was a bug posted on bugs.gentoo.org about freenet not starting with blackdown. This seems to be fixed now
[4:12] <KenMan> feel free to submit bug reports with blackdown though. But be sure to indicate that JVM.
[4:13] <dgrant> KenMan: do you use gentoo?
[4:13] <KenMan> nope, sorry
[4:14] <KenMan> bbl
[4:18] * dgrant (~dgrant@d154-20-147-177.bchsia.telus.net) Quit ("Leaving")
[4:35] * Hory (~Miranda@81.196.25.110) has joined #FreeNet
[4:52] <theLostFloppy> Lotsa gentoo people here. I like that
[4:53] <theLostFloppy> Aaah load 83
[4:54] * Ash-Fox (Hal-9000@abd72.neoplus.adsl.tpnet.pl) Quit (Client Quit)
[4:54] * Ash-Fox (Hal-9000@abd72.neoplus.adsl.tpnet.pl) has joined #FreeNET
[4:59] <theLostFloppy> Mmm. Frost doesn't seem to refresh automatically anymore. Strange
[5:00] * Ash-Fox (Hal-9000@abd72.neoplus.adsl.tpnet.pl) Quit (Nick collision from services.)
[5:00] * Ash-Foxeus (Hal-9000@abd72.neoplus.adsl.tpnet.pl) has joined #FreeNET
[5:03] * Hory (~Miranda@81.196.25.110) Quit (Read error: 104 (Connection reset by peer))
[5:06] * Hory (~Miranda@81.196.25.110) has joined #FreeNet
[5:06] <iip_i2p> <mule2p> does anybody have an idea whether fred/fproxy and the machine running it can be compromized by remote access to fproxy?
[5:12] * Ash-Foxeus (Hal-9000@abd72.neoplus.adsl.tpnet.pl) Quit (Client Quit)
[5:25] * Ash-Fox (Hal-9000@abd72.neoplus.adsl.tpnet.pl) has joined #FreeNET
[5:25] <ShaunMacPherson> not sure
[5:25] <ShaunMacPherson> probably :)
[5:29] <theLostFloppy> What can I do if Frost doesn't refresh the boards automatically?
[5:38] * Hory (~Miranda@81.196.25.110) Quit (Read error: 54 (Connection reset by peer))
[6:02] * TLF (francisco@245.Red-81-40-116.pooles.rima-tde.net) Quit ("http://www.vitaliano.esp.cc || Adi?s a todos y a todas y que les vaya bien.")
[6:04] * Ash-Fox (Hal-9000@abd72.neoplus.adsl.tpnet.pl) Quit (Client Quit)
[6:05] * Ash-Fox (Hal-9000@abd72.neoplus.adsl.tpnet.pl) has joined #FreeNET
[6:09] * hobx_ (~chatzilla@ankh.math.chalmers.se) Quit (Remote closed the connection)
[6:12] * thelema|away (~thelema@adsl-65-65-201-216.dsl.stlsmo.swbell.net) has joined #freenet
[6:38] * moskau23 (~Miranda@dsl-082-082-232-068.arcor-ip.net) has joined #freenet
[7:05] * verl (~verlverl@h165n2fls33o877.telia.com) Quit ()
[7:13] * TLF (francisco@27.Red-81-40-113.pooles.rima-tde.net) has joined #freenet
[7:28] * Hory (~Miranda@81.196.25.110) has joined #FreeNet
[7:32] * Ash-Fox (Hal-9000@abd72.neoplus.adsl.tpnet.pl) Quit (Nick collision from services.)
[7:32] * Ash-Foxeus (Hal-9000@aas115.neoplus.adsl.tpnet.pl) has joined #FreeNET
[7:37] * verl (~verlverl@h165n2fls33o877.telia.com) has joined #freenet
[7:41] * TLF (francisco@27.Red-81-40-113.pooles.rima-tde.net) Quit ("http://www.vitaliano.esp.cc || Adi?s a todos y a todas y que les vaya bien.")
[8:12] * Ash-Fox (Hal-9000@aas115.neoplus.adsl.tpnet.pl) has joined #FreeNET
[8:13] * Ash-Foxeus (Hal-9000@aas115.neoplus.adsl.tpnet.pl) Quit (Success)
[8:20] * TLF (francisco@75.Red-81-40-116.pooles.rima-tde.net) has joined #freenet
[8:29] * Redb3ard (~oylerj@c-24-125-12-101.va.client2.attbi.com) Quit (Read error: 113 (No route to host))
[8:30] * ejhuff (~ejhuff@dsl-096.floodcity.net) Quit (Read error: 60 (Operation timed out))
[8:45] * Hory (~Miranda@81.196.25.110) Quit (Read error: 104 (Connection reset by peer))
[9:00] * theLostFloppy (~michaelku@fia41-111.dsl.hccnet.nl) Quit ()
[9:05] * ejhuff (~ejhuff@dsl-096.floodcity.net) has joined #FreeNet
[9:06] * hobx_ (~chatzilla@ankh.math.chalmers.se) has joined #freenet
[9:06] <hobx_> entertain me!
[9:10] * galt makes a not-too-creditable attempt at a jig
[9:16] * KenMan throws a sombrero on the ground...
[9:17] * hirvox (~hirvox@cs181027153.pp.htv.fi) Quit (Connection timed out)
[9:26] <KenMan> hobx - this one's for you - http://www.kididdles.com/mouseum/m044.html
[9:27] <mazzanet> http://s.deviantart.com/static/v4intro.swf <--- :O now that's a flash animation
[9:28] * thelema|away (~thelema@adsl-65-65-201-216.dsl.stlsmo.swbell.net) Quit (Read error: 110 (Connection timed out))
[9:45] * hirvox (~hirvox@cs181027153.pp.htv.fi) has joined #freenet
[9:47] * TLF (francisco@75.Red-81-40-116.pooles.rima-tde.net) Quit ("http://www.vitaliano.esp.cc || Adi?s a todos y a todas y que les vaya bien.")
[9:47] <hobx_> KenMan: That song was horrible. I'm scarred for life!
[9:48] <jokern> Is toad_ gone for the weekend?
[9:48] <hobx_> is he taking weekends off now? The slacker!
[9:48] * hobx_ is at work
[9:48] <jokern> haha,
[9:49] * KenMan is diligently at work, doing ... hmmm
[9:49] * jokern is at work too
[9:49] <hobx_> I'm the only person here
[9:49] <hobx_> I could take my clothes off and run around the halls...
[9:49] <KenMan> actually I don't have a job to take weekends off from :(
[9:49] * hobx_ wonders if that would be interesting
[9:49] <jokern> I am getting some errors on 5090 and it tells me to report.
[9:50] <jokern> Caught java.lang.IllegalArgumentException: illegal (low) report: -250.0! in
[9:50] <jokern> Lots of those
[9:50] <hobx_> Maybe I'll go around and sit naked on peoples chairs and take pictures
[9:51] <KenMan> then you can superimpose them with photos your students in class
[9:52] <KenMan> now that would be cool. You can teach people to relax for public speaking ... just picture yourself IN the audience, as the only one who is nude ...
[9:53] * Ash-Fox (Hal-9000@aas115.neoplus.adsl.tpnet.pl) Quit (Nick collision from services.)
[9:53] * Ash-Foxeus (Hal-9000@aas115.neoplus.adsl.tpnet.pl) has joined #FreeNET
[9:53] <jokern> Sitting here NOT nude ,,waiting for a backup to complete.
[9:55] * KenMan can't believe hobx hasn't bewitched some beautiful swedish student of his ...
[9:56] <hobx_> I think if we are going to start a get nude at work contest, we ought to choose a channel that actually has women in it...
[9:56] <hobx_> Or perhaps and IRC network.
[9:57] <hobx_> KenMan: In what sense?
[9:57] <hobx_> I think I've managed the sleep spell on a lot of them...
[9:57] <KenMan> in a sexual sense, of course
[9:57] * Ash-Fox (Hal-9000@aas115.neoplus.adsl.tpnet.pl) has joined #FreeNET
[9:58] * Ash-Foxeus (Hal-9000@aas115.neoplus.adsl.tpnet.pl) Quit (Client Quit)
[9:58] <hobx_> I'm dating an undergraduate. Not a student though, I think there are some issues with that...
[9:58] <KenMan> very well then...
[9:59] <hobx_> though of course, I could never respect a man who lived by his principles, so...
[10:00] <KenMan> wait, i thought it was the university's principles...
[10:01] <hobx_> I've never been told about any such rules.
[10:01] * pupok (~janie@81-178-89-164.dsl.pipex.com) Quit (Read error: 104 (Connection reset by peer))
[10:01] <KenMan> oh, get back to solving math problems in the nude. I'm sure it would excite some of your students.
[10:01] * pupok (~janie@81-178-89-164.dsl.pipex.com) has joined #freenet
[10:01] <hobx_> But god knows I would never dare make a pass on a student. I have been to enough forced "Gender Relations" seminars for one liftime!
[10:01] <hobx_> :-)
[10:02] <KenMan> :o
[10:04] <KenMan> what, your "blowjobs for grades" program offended the sensibilities of some other teachers ??
[10:04] <hobx_> some people just don't appreciate free enterprise...
[10:12] * thelema|away (~thelema@adsl-66-136-184-242.dsl.stlsmo.swbell.net) has joined #freenet
[10:29] * theLostFloppy (~michaelku@fia41-111.dsl.hccnet.nl) has joined #freenet
[10:30] <theLostFloppy> Aaaah. Freenode's NickServ rejects my password!
[10:30] <ShaunMacPherson> oh oh :)
[10:30] <ShaunMacPherson> chance your name to theLostPassword then ;p
[10:30] <theLostFloppy> Mmm
[10:30] <ShaunMacPherson> :)
[10:31] <theLostFloppy> Should a FUQID upload of 70 MB with default settings take more than 4 hours?
[10:32] <theLostFloppy> Even when I have good connections, I get 'Insert thread timed out' errors *all the time*.
[10:32] <theLostFloppy> By the way, how can I read the backlog?
[10:33] <hobx_> http://www.stuffmagazine.com/girls/nebraska_sextuplets/sextuplets_l1.jpg
[10:34] <ShaunMacPherson> :)
[10:35] * hobx_ remembers the "things not surf at work" electro-shock behavioural therapy...
[10:36] * hirvox (~hirvox@cs181027153.pp.htv.fi) Quit (Read error: 104 (Connection reset by peer))
[10:43] * hirvox_ (~hirvox@cs181027153.pp.htv.fi) has joined #freenet
[10:43] * hirvox_ (~hirvox@cs181027153.pp.htv.fi) Quit (Client Quit)
[10:49] * hirvox (~hirvox@cs181027153.pp.htv.fi) has joined #freenet
[10:50] <theLostFloppy> hobx_, very very nice ;-)
[10:50] * theLostFloppy 'll be back in a minute
[10:50] * theLostFloppy (~michaelku@fia41-111.dsl.hccnet.nl) Quit ()
[10:51] * thelema|away is now known as thelema
[11:02] * salahx (salahx@sc1-24.217.174.147.charter-stl.com) has joined #freenet
[11:04] <salahx> Can anyone retrieve TFE/FIND/YoYo/CofE or do you get the MIME type message too ?
[11:08] * jay (jcl@ool-18bf6dac.dyn.optonline.net) has joined #freenet
[11:30] <hobx_> Anyone think I'll be banned from Wikipedia if I fill in the entry for "Dingleberry"
[11:33] * salahx (salahx@sc1-24.217.174.147.charter-stl.com) Quit ()
[11:34] <hobx_> (Everybody can tell I'm working hard today.)
[11:39] <thelema> make sure you do dingleberry right.
[11:39] <ShaunMacPherson> they will like u for it, since cockslap made it in somehow ;)
[11:39] <ShaunMacPherson> its on the deletion page though :)
[11:42] * moskau23 (~Miranda@dsl-082-082-232-068.arcor-ip.net) Quit (Read error: 104 (Connection reset by peer))
[11:48] <jay> hobx: heh
[11:49] * Zorix (Brandon@fl-65-41-1-233.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[11:49] * Zorix (Brandon@fl-65-41-1-233.dyn.sprint-hsd.net) has joined #freenet
[11:51] <jay> so goatc.ex or whatever it is has finally made it's way to Frost
[12:11] * moskau23 (~Miranda@dsl-213-023-251-068.arcor-ip.net) has joined #freenet
[12:11] <hobx_> goatc.ex?
[12:12] * Hory (~Miranda@81.196.25.110) has joined #FreeNet
[12:14] <jay> hobx: slashdot-posted image of man bending over
[12:14] <jay> now in ascii art form on Frost
[12:14] <hobx_> no, that is goatse.cx
[12:14] <jay> ah righto
[12:14] * TLF (francisco@3.Red-81-40-113.pooles.rima-tde.net) has joined #freenet
[12:14] <hobx_> I'm not even sure .ex is a TLD
[12:15] <jay> me neither
[12:18] * theLostFloppy (~michaelku@fia41-111.dsl.hccnet.nl) has joined #freenet
[12:18] <theLostFloppy> Hi everyone
[12:21] <jay> ello
[12:21] <ShaunMacPherson> hi
[12:31] * Zorix (Brandon@fl-65-41-1-233.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[12:40] * hirvox (~hirvox@cs181027153.pp.htv.fi) Quit (Read error: 113 (No route to host))
[12:46] * jdstroy (~jdstroy@host-65-6-226-198.cae.bellsouth.net) has joined #freenet
[12:49] * jdstroy (~jdstroy@host-65-6-226-198.cae.bellsouth.net) has left #freenet
[12:52] <theLostFloppy> I've made up my mind. I'm going to install Linux on Mom&Dad's computer.
[12:52] * Sweetie (opera@dialin-145-254-058-016.arcor-ip.net) has joined #freenet
[12:52] <theLostFloppy> Hi Sweetie
[12:52] <Sweetie> hi
[12:54] <theLostFloppy> It will be a surprise. I'm sure my dad will be shocked...
[13:03] * Sweetie (opera@dialin-145-254-058-016.arcor-ip.net) has left #freenet
[13:06] <jay> your sweetie left
[13:12] <theLostFloppy> Hehe
[13:12] <theLostFloppy> Ok... deleting the Windows XP partition now - no return.
[13:13] * TLF (francisco@3.Red-81-40-113.pooles.rima-tde.net) Quit ("http://www.vitaliano.esp.cc || Adi?s a todos y a todas y que les vaya bien.")
[13:18] <thelema> theLostFloppy: was everything backed up?
[13:18] <theLostFloppy> Yup
[13:19] <jokern> k, then my playtime with freenet is over for now. 26hours uptime and I am back in that f... seednodes.ref again.
[13:19] * Ash-Fox (Hal-9000@aas115.neoplus.adsl.tpnet.pl) Quit (Client Quit)
[13:19] * Ash-Fox (Hal-9000@aas115.neoplus.adsl.tpnet.pl) has joined #FreeNET
[13:20] <jokern> Are there really so few nodes in this world that my node has to get in there every time I try to join ?
[13:21] <thelema> "get in there"? where?
[13:21] * Ash-Fox (Hal-9000@aas115.neoplus.adsl.tpnet.pl) Quit (Nick collision from services.)
[13:22] * Ash-Foxeus (Hal-9000@aas115.neoplus.adsl.tpnet.pl) has joined #FreeNET
[13:22] <thelema> there should be plenty of nodes in either the newest stable or unstable.
[13:22] <KenMan> if so, does it really matter ?? that implies there aren't that many nodes beating a path to your door ..
[13:22] <toad_> hi KenMan , thelema
[13:22] * toad_ will be back in ~ 15 mins, installing new soundcard
[13:22] <KenMan> hi toad, thelema :)
[13:22] * toad_ (toad@toad-with-underline.active.supporter.pdpc) Quit (Remote closed the connection)
[13:22] <thelema> hi KenMan, toad
[13:23] * thelema is happy to be using his laptop again, after the LCD gave up.
[13:23] <KenMan> awww, now it truly is a lap top ?
[13:23] * Ash-Foxeus is now known as Ash-Fox
[13:24] <KenMan> are you saying the screen is broke ?? do you use a external keyboard & mouse too ??
[13:25] <thelema> the LCD isn't displaying anything, but I'm using an external monitor quite successfully
[13:26] <jokern> Every time I get listed in seednodes.ref it is like beeing hit with a dos attack. Not funny at all.
[13:26] <thelema> I'm still using the built-in KB and mouse
[13:26] <KenMan> If I had a laptop, the first thing I would get would be a keyboard... maybe one of those trackball enabled ones...
[13:26] <thelema> KenMan: you don't like laptop keyboards?
[13:26] <thelema> jokern: you're saying the connection opening is too agressive?
[13:27] <jokern> That is mild understatement
[13:27] <KenMan> generally no. They don't travel down very far before you have hit the end. Sometimes I like to type agressively :)
[13:28] <thelema> KenMan: bah. I actually like laptop keyboards because I don't have to type very hard to get them to work.
[13:28] <KenMan> but, jokern, does it truly eat much of your bandwidth, or is it simply that it tickles an LED somewhere ?
[13:30] * Ash-Fox (Hal-9000@aas115.neoplus.adsl.tpnet.pl) Quit (Nick collision from services.)
[13:30] * Ash-Foxeus (Hal-9000@aas115.neoplus.adsl.tpnet.pl) has joined #FreeNET
[13:30] * thelema considers a config optoin 'seed' that turns a node into an optimal seednode.
[13:30] * KenMan wonders if the value of big seeds in the seed node exceeds zero...
[13:30] <thelema> it'd have the parameter of the computer/ip to allow to get its references
[13:30] <thelema> and it'd not limit newbie nodes.
[13:31] <KenMan> i not fully understand you, thelema...
[13:32] <thelema> and have a different connection closing sort
[13:33] <KenMan> currently, IP & port are not enough to make a connection... we need the node's public key. How this benefits us, I am not at all sure... :(
[13:33] <KenMan> these things are 'almost' free, or can be harvested ...
[13:36] <KenMan> oh, you suggest choosing from the seednodes based on suitability to the local node's state , and the same thing for closing ?
[13:36] <jokern> KenMan : Tickles a LED? Hope you ar not trying to be funny. Peeks of 300+ hits a second would make it glow quite nicely without power.
[13:37] <jokern> It makes my inetconnection about useless
[13:37] <KenMan> okay, that is the criteria then - "useless"
[13:38] <jokern> ppl not able to get their mail, webpages not accesible. ...
[13:38] * thelema realizes there;s a fundamental problem in his suggestion and withdraws it.
[13:38] <jokern> Mail not getting in here...
[13:38] <KenMan> maybe it is just that I rarely go around 'node-free' . Maybe my connection only remains useable if I RUN freenet :p
[13:40] <KenMan> jokern - the only thing you should be receiving is a tiny TCP SYN packet from each node. 300/second of those is unimaginable to me.
[13:40] <jokern> :\ Could be,, this connection works best with "normal" traffic. Before I get into that seenodes all is ok.
[13:40] <thelema> KenMan: handshaking is pretty heavy.
[13:40] <KenMan> noy if you don't answer it.
[13:40] <thelema> KenMan: fair enough.
[13:41] <jokern> So you say I should block the port , just drop the connection? k, no problem
[13:41] <theLostFloppy> 1:30 to transfer 2GiB? I wish I had gigabit ethernet...
[13:41] <KenMan> that is an option, if what you are telling us is true (300 tcpSYNs / second)
[13:42] <jokern> , done, bye Freenet
[13:42] <KenMan> "bye incoming freenet" you mean :)
[13:42] <jokern> :) Goodbye then
[13:42] <thelema> jokern: just move your node to a different port
[13:43] <KenMan> now, if that doesn't improve your situation after several days, then it didn't help.
[13:43] <KenMan> yes, that is the best solution
[13:43] <jokern> KenMan, been here before. And was into same problem a couple of months ago.
[13:44] <jokern> And it takes some hours before that traffic is slowing down. And I guess more then a year before it goes away.
[13:44] <KenMan> I guess there is no way to change your IP then ??
[13:45] <jokern> Sorry, been on this ip for years.
[13:45] <KenMan> yes, you clearly are sorry.
[13:45] <KenMan> several years of collecting derelict Freenet nodes from the whole spectrum of releases across the ages ... :)
[13:46] <KenMan> non-stop BAM BAM TAKE IT MA'AM
[13:47] <jokern> I still get 20-30 hits/hour on the port I used in september last year :) That is why I asked if there are some kind of "oldnet" using old seednodes.ref or routing-tabels
[13:51] * Zorix (Brandon@fl-65-41-1-233.dyn.sprint-hsd.net) has joined #freenet
[13:52] <jokern> Another thing to consider is to encrypt seednodes.ref. It looks like some funny guys are picking ip`s from it. Portscanning is increasing alot when I am listed in seedn.
[13:52] <jokern> Doesn really do any harm, but it is a nag-nag to get alerts.
[13:53] <thelema> well, the old connectino-opening code is really really old.
[13:54] * TLF (francisco@3.Red-81-40-113.pooles.rima-tde.net) has joined #freenet
[13:55] <thelema> jokern: encrypting it won't help; anyone would be able to decrypt it.
[13:56] * Hory (~Miranda@81.196.25.110) Quit (Read error: 54 (Connection reset by peer))
[13:56] <jokern> thelema, yes, but then they at least had to do something :) Now it is an open list
[13:59] <thelema> It might be enough to discourage things, but I don't really think it's worth the time.
[14:03] * TLF (francisco@3.Red-81-40-113.pooles.rima-tde.net) Quit ("http://www.vitaliano.esp.cc || Adi?s a todos y a todas y que les vaya bien.")
[14:04] * TLF (francisco@121.Red-81-40-113.pooles.rima-tde.net) has joined #freenet
[14:11] * TLF (francisco@121.Red-81-40-113.pooles.rima-tde.net) Quit ("http://www.vitaliano.esp.cc || Adi?s a todos y a todas y que les vaya bien.")
[14:11] * toad_ (toad@82-32-16-91.cable.ubr03.azte.blueyonder.co.uk) has joined #freenet
[14:12] <KenMan> how do your ears feel now, toad ?
[14:12] <theLostFloppy> Hihi
[14:14] <toad_> KenMan: huh?
[14:14] * thelema (~thelema@adsl-66-136-184-242.dsl.stlsmo.swbell.net) Quit (Read error: 104 (Connection reset by peer))
[14:14] <KenMan> don't you have a shiny new soundcard wrapped around your ears now ??
[14:15] * toad_ had a soundcard, now has a better one. ordered new one before realizing that the old one DID work (if use alsa drivers)
[14:15] <KenMan> :(
[14:15] <toad_> still it means i have hardware mixing :)
[14:16] <toad_> (esd has problems)
[14:16] * KenMan doesn't like the idea of soft mixing
[14:17] * toad_ doesn't like the practicalities of esound not working well
[14:17] <KenMan> i've never had the need for more than one app to generate sound at the same time
[14:17] * KenMan is a dinosaur ?
[14:18] <greycat> I do it all the time -- xmms playing music, and some game making game-sounds.
[14:18] * KenMan leverages his PC beep-speaker for anything other than xmms music
[14:19] * toad_ hasn't wired that up for upstairs
[14:19] <toad_> KenMan: any new graphs?
[14:23] <toad_> yay, Someone supporting me on the support list re upgrading
[14:24] <hobx> I'll support you
[14:25] <toad_> good, firefox 0.9.3 is in sid now
[14:27] <toad_> <mikeDOTd> there's a lot of fixes in the new build, that seem to be making a very nice improvement - thanks
[14:27] <hobx> you mean I've been surfing with the vulnerable version for several days?
[14:27] <toad_> <KenMan> 5090 - recvData (436/729)=58% , sentData (633/1486)=43% - hmm, that's not very good...
[14:27] <toad_> hobx: the day of the slashdot announcement, sid firefox was still vulnerable
[14:27] <toad_> KenMan: what about sentDataNoInserts?
[14:28] <toad_> KenMan: 43% is pretty bad really
[14:28] <KenMan> but , even if we can't explain it, at least it hasn't changed dramatically in a short time !! :)
[14:28] <toad_> :)
[14:28] <toad_> KenMan: got a graph for me of 5090?
[14:28] <toad_> <KenMan> overall , things look pretty decent on 5090. My only concern is the level of RT backoff - it averages about 90% so far ! - 90% backedOffCount is not nice
[14:29] <KenMan> i am still considering it with the artistic license of someone who doesn't have to maintain the existing code ;)
[14:29] <toad_> considering what?
[14:30] <KenMan> back off: actually I'll have results of a new experiment in a few days time, for you to consider or to laugh at, depending on the outcome.
[14:30] <KenMan> but wait for me to do it first...
[14:30] <toad_> KenMan: do you have a plain 5090 graph? i'm anxious to see how rate limiting is working
[14:30] <KenMan> hold some horses, i'm getting it. Stop stealing my window focus!
[14:31] * toad_ wonders if per node failure table will improve success %
[14:31] <toad_> <KenMan> incoming query success (739/23997)=3% , outgoing query success (357/21207)=1.6% :(
[14:31] <toad_> hmmm
[14:31] <toad_> often requestSuccessRatio is significantly better than routingSuccessRatio
[14:31] <toad_> but i don't see how that can work, on the network level...
[14:31] <KenMan> datastore perhaps
[14:32] <toad_> <salahx> errr odd
[14:32] <toad_> <salahx> I jsut tried to retrive FIND and got this:
[14:32] <toad_> <salahx> Warning: Unknown mime type application/zip
[14:32] <toad_> <salahx> You have retrieved some content which is not recognised by FProxy,
[14:32] <toad_> <salahx> TFE is doing it too...
[14:32] <toad_> <salahx> So is YoYo
[14:32] <toad_> hmm, is htat related to the NPE in FProxy?
[14:32] * TLF (francisco@121.Red-81-40-113.pooles.rima-tde.net) has joined #freenet
[14:32] <KenMan> plain stock 5090 - http://mywebpages.comcast.net/jkcorson/FreeNet/5090c.png and 5090bw.png
[14:33] <toad_> <KenMan> it seems the vast majority of queries result in RNF. Unless I am missing something, this can only be caused by two events - first, the request has looped back to a node that is already in the request chain. Second, the node has 100% of routes backed off at the time of this specific routing attempt. - okay, that does NOT happen for me
[14:34] <KenMan> i don't know why i lost 10 connections at the same time, but there are no NTP oddities in these graphs
[14:34] <toad_> KenMan: blech
[14:34] <toad_> rate limiting still doing crazy things
[14:34] <KenMan> i wonder if RNF correlates with degree of backoff for a single routing
[14:35] <toad_> was there an external stress at any point during that graph?
[14:35] <KenMan> no
[14:35] <toad_> it was reasonably stable until 08:00
[14:35] * toad_ wonders if there is a daily cycle of sizes and success probabilities...
[14:35] <KenMan> i went to bed just before that time. I did a VERY slight amount of FN-browsing then, but I doubt it is related
[14:36] <toad_> your #xfers looks reasonable...
[14:36] <KenMan> yes, definitely. Average transfer time is much lower on stable (from 5085 onwards) than it was on unstable
[14:36] <KenMan> sorry, the TIME is higher, the rate is lower on stable
[14:37] <toad_> ah
[14:37] <toad_> that's not too great
[14:37] <toad_> anyway my point: #xfers is under reasonable control
[14:37] <KenMan> yes, thank you for fixing !!
[14:38] <toad_> I have a jar you could test that will become part of 5091 (or at least unstable). It has the store-the-rate-limiting-vars-to-kill-the-startup-spike code we discusses
[14:38] <toad_> d
[14:38] <toad_> KenMan: did you say that the overwhelming majority of requests on your node RNF?
[14:38] <KenMan> cool, i hope it helps, although the spike is not so long or bad anymore
[14:39] <toad_> i can see why from that graph... :<
[14:39] <toad_> odd, i'd expect it to be longer, with longer averages
[14:39] <KenMan> on outgoings, i think it is like 2% success, 1% DNF, 97% RNF (on most nodes)...
[14:39] <toad_> hmmm
[14:39] <toad_> my backedOffCount is negligible
[14:40] <toad_> this is because I am running 2 nodes so they are constantly overloaded on CPU...
[14:41] <KenMan> you know, i just noticed that incoming xfers < outgoing xfers once again :)
[14:41] <toad_> KenMan: that's good
[14:41] * toad_ has reports that running frost causes incoming xfers to go to infinity
[14:41] <toad_> well near enough
[14:41] <KenMan> i thought you just tracked that down ?
[14:42] <toad_> no
[14:42] <toad_> i tracked down a reporting bug
[14:42] <toad_> that happens regardless of client usage
[14:42] <toad_> Contacted node references 46
[14:42] <toad_> hmmm
[14:42] <toad_> grrrr
[14:43] <toad_> that of course means my backedOffCount isn't quite so negligible
[14:43] <toad_> but it's still v. good
[14:43] <KenMan> Contacted node refs is my blue line in the 'c' graph
[14:43] <toad_> uptime: 40 minutes
[14:43] <toad_> that explains the low conns :)
[14:44] <KenMan> ah, yes
[14:44] <toad_> KenMan: what on earth happened towards the end?
[14:44] <toad_> sudden drop in connected nodes...
[14:44] <KenMan> i have no clue
[14:44] <KenMan> i just graph this $h!t ,,, ;)
[14:45] <toad_> backedOffCount on an hourly basis for the last 24h on stable is pretty respectable too...
[14:46] <KenMan> examples ? x's of y's , and we don't know what the makeup is regarding offline nodes :(
[14:46] <toad_> highest i've seen in last 24h hourly average on stable is 37.4
[14:46] <toad_> RT probably around 100 at the time
[14:46] <toad_> connected
[14:49] <KenMan> i sit around 27-30 with 80/160 routes limit, so we are comparable
[14:49] <toad_> ummm
[14:50] <toad_> that's not what your graphs say
[14:50] <toad_> your graphs say you have > ~ 90% backed off the overwhelming majority of the time
[14:50] <KenMan> I'm reading the diagnostic. The graph is a one minute sampling of RT's "Backed off nodes". You explain the diffs then :)
[14:51] <toad_> hmmm interesting...
[14:51] <toad_> well i'd expect it to be lower actually
[14:51] <toad_> i'm not sure it'd be a lot lower though...
[14:51] <KenMan> and what about the case where a request gets 2 RNFs and gets on its way on the third attempt. How is backoff and rTCR counted then ?
[14:51] <toad_> my RT backed off nodes is low though
[14:51] <KenMan> i know what they claim, but what does the code do...
[14:52] <toad_> KenMan: the total number of backed off nodes before it completes
[14:52] <toad_> Contacted node references 76
[14:52] <toad_> Backed off nodes 50
[14:52] <toad_> hmmm it's not THAT low...
[14:52] <KenMan> 62/65 backed off here
[14:52] <KenMan> now 59/65 ...
[14:53] <toad_> KenMan: you said above that most queries RNF
[14:53] <toad_> on what do you base this assertion?
[14:53] <KenMan> I thought so, no ? isn't that all that is left ??
[14:53] <toad_> might it be due simply to the RNF bug fixed in 5090?
[14:53] <toad_> Fred,0.5,STABLE-1.50,5089 |================================================================
[14:53] <toad_> Fred,0.5,STABLE-1.50,5090 |=========================================
[14:53] <toad_> hmmm
[14:54] <toad_> can't make 5090 mandatory just yet...
[14:54] <toad_> KenMan: huh?
[14:54] <KenMan> what is "Single hop overall probability of DNF given no timeout" ? it says .969 here ...
[14:54] <toad_> what it says
[14:55] <KenMan> okay. Single hop probability of search timeout = 0.768
[14:55] <toad_> eeek
[14:55] <toad_> Single hop probability of search timeout 0.716
[14:55] <toad_> hmmm
[14:55] <toad_> mine's equally bad...
[14:56] <KenMan> Lowest/Highest one hop probability of DNF = .957 , .983 (so most of my outgoings get DNFs then?)
[14:56] * toad_ ought to do another network reset
[14:56] <toad_> ;)
[14:56] <toad_> with 5090
[14:56] <KenMan> harrumph
[14:56] <toad_> no, most of them get search timeouts
[14:56] <toad_> (which are actually mostly RNFs)
[14:56] <KenMan> how is that handled and accounted for ??
[14:57] <toad_> actually, the overwhelming majority of 5089's are not connected
[14:57] <toad_> if you only take into account nodes which are connected...
[14:57] <toad_> then 5090 is quite popular
[14:57] * toad_ thinks he will make 5090 mandatory in 5089
[14:58] <KenMan> same here lots of active 5090s
[14:58] <toad_> okay
[14:58] <toad_> so how do we manage to STILL get a 70% probability of RNF?
[14:59] <toad_> even though most nodes we route to are in fact 5090 and have the BigRNFBugFix
[14:59] <KenMan> timeout counts as an RNF ?
[14:59] <toad_> ?
[14:59] <toad_> search timeout == RNF
[14:59] <toad_> they're counted as the same event
[14:59] <KenMan> ok
[15:00] <ejhuff> Build Number 5090
[15:00] <ejhuff> Uptime 14 hours 56 minutes
[15:00] <ejhuff> freenet.node.states.data.DataStateInitiator 16
[15:00] <ejhuff> Connections open (Inbound/Outbound/Limit) 177 (130/47/800)
[15:00] <ejhuff> Transfers active (Transmitting/Receiving) 105 (83/22)
[15:00] <ejhuff> Data waiting to be transmitted/received 3,564 Bytes/None
[15:00] <ejhuff> Amount of data transmitted/received over currently open connections 148 MiB/135 MiB
[15:00] <ejhuff> Total amount of data transmitted/received 487 MiB/308 MiB
[15:00] <ejhuff> # Histogram of node versions in fred's Routing table
[15:00] <ejhuff> # Aug 7, 2004 3:15:52 PM
[15:00] <ejhuff> # nodes: 1260
[15:00] <ejhuff> Fred,0.5,STABLE-1.49,5058 1
[15:00] <ejhuff> Fred,0.5,STABLE-1.49,5061 1
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5069 1
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5070 3
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5072 1
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5081 1
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5082 7
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5083 1
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5084 70
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5085 5
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5086 16
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5087 7
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5088 68
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5089 683
[15:00] <ejhuff> Fred,0.5,STABLE-1.50,5090 389
[15:00] <ejhuff> unknown 6
[15:00] <toad_> ejhuff: that's a slightly large routing table :)
[15:00] <ejhuff> bbl
[15:01] <KenMan> even by toad standards ?
[15:01] <toad_> <KenMan> there could be some nodes out there (dial up modem with 200 active connections) that simply can't help but RNF 99.8% of all requests, due to backoff.
[15:01] <toad_> <KenMan> in other words, severe routing congestion could exist only on certain nodes, and this would hurt the net
[15:01] <KenMan> ah hah, we have found a boundary ... toad thinks 800+ routes is too big...
[15:02] <toad_> KenMan: that's not realistic
[15:02] <toad_> as NGR will route around such nodes
[15:02] <KenMan> i'm just poking at straws...
[15:02] <toad_> unless it has no other choice
[15:02] <toad_> in which case the whole network has backoff problems
[15:02] <toad_> not just those few nodes
[15:02] <KenMan> that's my theory, and I'm sinking on it.
[15:03] <toad_> what? that the whole network has backoff problems?
[15:03] <KenMan> :)
[15:03] <toad_> well, duh :)
[15:04] <toad_> are there any plausible reasons why reducing the number of connections with a reset would lead to LESS backoff?
[15:04] <KenMan> no, but once we get it right it should scale with whatever number of connections exist...
[15:04] <KenMan> there is reason to believe it would increase backoff with current code
[15:04] <toad_> hmmm
[15:05] <toad_> KenMan: what's your searchFailedCount ?
[15:05] <toad_> mine's around 2
[15:05] <toad_> up to 5
[15:05] <toad_> hourly averages around 2
[15:05] <KenMan> you mean my query multiplication factor (minus 1) ? 0.5 up to 1.5 ...
[15:05] <toad_> no i mean the diagnostic
[15:06] <KenMan> i know ;)
[15:06] <toad_> http://127.0.0.1:8888/servlet/nodestatus/diagnostics/searchFailedCount/hour
[15:06] <KenMan> yes... i try make a funny
[15:06] <toad_> you checked the diagnostic?
[15:06] <KenMan> yes, it wanders from 0.5 to 1.5
[15:06] <toad_> okay
[15:06] <toad_> hypothesis:
[15:07] <toad_> it would make a difference if i added code to NGRouting such that after a QR, or other retry taking time, we recheck nodes we've already written off as backed off
[15:07] <toad_> do you think it would make much difference?
[15:07] <KenMan> yes, but please don't go there. We should correct the problem at the source (rate limiting)
[15:08] <toad_> KenMan: why?
[15:08] <toad_> IMHO it's a reasonable change, within the existing framework
[15:08] <KenMan> after a QR ? you mean, after we tried it once, try it again ?
[15:08] <toad_> i'm not going to actually retry on nodes we've already visited
[15:08] <toad_> no, check it again
[15:08] <KenMan> the existing framework has termites...
[15:08] <toad_> lets see
[15:08] <toad_> node 1 is backed off
[15:08] <toad_> node 2 is backed off
[15:08] <toad_> node 3 is backed off
[15:08] <toad_> node 4 RNFs
[15:09] <toad_> node 1 is no longer backed off, so try it again
[15:09] <toad_> node 1 QRs immediately
[15:09] <toad_> node 3 is no longer backed off...
[15:09] <toad_> etc
[15:09] <KenMan> will it bump CPU any ?? is it an efficient response to what rate limiting is causing ?? it is worth trying now, but changes to rate limiting will alleviate it in the future
[15:09] * toad_ wonders if making it into a linked list structure like that would cause memory usage problems
[15:10] <toad_> well, we'd have to recheck isAvailable on each, so it would cause some cpu usage
[15:10] <toad_> and maybe some memory usage too
[15:10] <KenMan> it would help a lot
[15:10] <toad_> KenMan: would it?
[15:10] <KenMan> top choices are likely to have smaller backoff quantities
[15:11] <toad_> no
[15:11] <KenMan> well, try it. It can't make things worse than they are, true ?
[15:11] <toad_> top nodes (by routing choice) will have LONGER backoff periods
[15:11] <KenMan> other than the CPU / memory issue... try it locally
[15:11] <toad_> well yes it can, it can cause CPU overload and OOMs ;)
[15:11] <toad_> and i don't think i can rigorously detect such problems locally
[15:12] * KenMan slowly and cautiously backs out of this conversation...
[15:12] <toad_> anyway
[15:12] <toad_> if i try that
[15:12] <toad_> we can talk about the longer term issue
[15:12] <toad_> what were you saying?
[15:12] <toad_> something is messed up somewhere?
[15:12] <KenMan> nothing... just listening to you
[15:13] <KenMan> RNFs are buggering up the network. I just realized this (after sanity's 6 months of input)...
[15:13] <KenMan> ;)
[15:13] <toad_> well, your backoff count is pretty absurd
[15:13] <toad_> on the graphs
[15:13] <toad_> does it significantly reduce with the KenManHack ?
[15:13] <KenMan> one reads the routing table at a single instant. The other (diag) averages measurements over time.
[15:14] <KenMan> I was thinking of that hack last night. Haven't turned it on for a while, wondered if there might be side effects now
[15:14] <toad_> queueing is another option but I don't actually see how it would help that much
[15:14] <toad_> well...
[15:14] <KenMan> ie- is it still a clean piece of code to use ?
[15:14] <toad_> i'd have to alter the predictive bandwidth load thingy to take it into account
[15:14] <toad_> but apart from that it'll just work with 5090
[15:14] <KenMan> to take KenManHack into account ? or queueing ?
[15:15] <toad_> KenManHack
[15:15] <toad_> otherwise load will be too high... hmm, that's good.. hmmm
[15:15] <toad_> probably it'll just work anyway
[15:15] <toad_> would be worth trying
[15:15] <KenMan> yup !
[15:15] <hobx> lib/rbtree.c: In function `rb_erase':
[15:15] <hobx> lib/rbtree.c:296: internal compiler error: Segmentation fault
[15:15] <hobx> not good
[15:15] <hobx> not good
[15:15] <KenMan> who lifted the lid on his garbage can ??
[15:16] <toad_> unfortunately for all intents and purposes i have KMH enabled and i still have lots of backed off nodes according to rt...
[15:16] <toad_> and the reason for this is quite simply that the MRIs are too <insert favourite expletive> high
[15:16] <KenMan> yes. I think I understand the matter, and will try to demonstrate it to you in a convincing manner.
[15:17] <KenMan> on the other hand, perhaps the MRIs are fine, and the query-rates are TFH
[15:17] <toad_> KenMan: by all means try
[15:17] <toad_> well
[15:17] <KenMan> ;)
[15:17] <toad_> I can't see how MRIs averaging in 6 figures (usually 3something) are fine
[15:17] <toad_> they're bound to lead to most nodes being backed off and backoff governing routing
[15:17] <KenMan> it all depends on the number of peers
[15:18] * jay (jcl@ool-18bf6dac.dyn.optonline.net) Quit ("Client exiting")
[15:18] <toad_> my incomingRequestInterval averages for the last 4 hours:
[15:18] <KenMan> if i have 800 nodes with 300s MRIs I am doing better than most of you all
[15:18] <toad_> 311590.4008781932
[15:18] <toad_> 223864.29669041373
[15:18] <toad_> 276367.1843621797
[15:18] <toad_> 239414.32257156898
[15:18] <toad_> KenMan: well, I'm reluctant to increase # conns max much further
[15:18] <toad_> and think of the impact on NGR
[15:18] <KenMan> thems numbers are encouraging to me...
[15:18] <toad_> KenMan: they do?
[15:19] <toad_> here's a question: would small fixed file sizes improve MRIs?
[15:19] <KenMan> uh huh. I know you don't like them , and I know why. But it figures into the equation better
[15:19] <toad_> and if so, why?
[15:19] <mikeDOTd> i don't know about you guys, but i haven't seen max connections be met on my node since 5084
[15:19] <KenMan> most definitely. Fixed sizes will do a hell of to make you smile more often.
[15:19] <toad_> KenMan: well, if we have 400 nodes in the rt, 200 conns max, and 100 actual conns...
[15:20] <toad_> then we can handle roughly one request every 3 seconds
[15:20] * KenMan thinks toad is going in the wrong direction. Ask 'why wouldn't it work with 50 peers?'
[15:20] <toad_> KenMan: WHY would fixed size nodes make MRIs lower? wouldn't they increase the number of requests?
[15:20] <toad_> s/nodes/files
[15:20] <KenMan> roughly, but they don't arrive in sync with the expiration times of backoffs that expire
[15:21] <KenMan> that's the tradeoff, more queries...
[15:21] <toad_> KenMan: hmm?
[15:21] <KenMan> but you have to throttle them
[15:21] <KenMan> i think the time for fixed MRI draws nearer
[15:21] <toad_> well... more, smaller queries... we could process more queries
[15:21] <KenMan> but HOW SMALL ??
[15:21] <toad_> KenMan: fixed MRIs are only useful if all nodes are identical
[15:22] <KenMan> ding ding
[15:22] <toad_> which they are not
[15:22] <KenMan> ding ding ding
[15:22] <toad_> bonnnggggg !
[15:22] <KenMan> oh crap, you asked fixed MRIs not fixed keysizes...
[15:22] <toad_> KenMan: okay, why would fixed KEY SIZES reduce backoff?
[15:23] <KenMan> i read too fast
[15:23] <KenMan> fixed keysizes would make all your maths work a little better
[15:23] <toad_> apart from indirect effects i.e. improving routing, making rate limiting operate on a shorter timescale?
[15:23] <KenMan> that is all i see
[15:23] <toad_> okay, what makes you think that would have a significant effect?
[15:24] <toad_> well lets see... if we are accepting 50kqph instead of 10kqph, our MRI will be lower, right?
[15:24] <KenMan> i think some of the math is not sufficiently reviewed for errors...
[15:24] <toad_> just by definition?
[15:24] <KenMan> it works in reverse. If we offer lower MRIs, we get more queries
[15:24] <KenMan> but yes
[15:24] <toad_> if queries are lighter weight, we can accept more queries
[15:24] <toad_> this means our MRIs are lower
[15:25] <toad_> okay, that's a pretty solid argument for fixed size keys
[15:25] <toad_> after which point, it might actually be useful to look into queuing
[15:25] <KenMan> well... you gonna have more queries for the same amount of content
[15:25] <toad_> however it'll take at least a month to implement fixed size keys
[15:25] <KenMan> hold off on queueing as long as possible. We shouldn't queue unless the query specifically says to do it.
[15:25] <toad_> KenMan: yes, and in many cases increased overall transfer because of redundancy on files that didn't used to need it...
[15:26] <KenMan> yeah, thats trye
[15:26] * toad_ disagrees, if we have light requests, the timeouts are shorter, and we can use queueing as a form of unobtanium
[15:26] <KenMan> wait let me argue that one point
[15:26] <toad_> which point?
[15:26] <KenMan> in many cases increased overall transfer because of redundancy on files
[15:26] <toad_> well
[15:27] <toad_> a typical freesite: CofE
[15:27] <toad_> can be compressed to maybe 80kB
[15:27] <toad_> that is then 3 blocks, maybe 4 blocks with redundancy
[15:27] * thelema (~thelema@adsl-66-136-184-242.dsl.stlsmo.swbell.net) has joined #freenet
[15:27] <toad_> hi thelema !
[15:27] <KenMan> the number of queries that originate at a single point (a user) increases in a small region of the network.
[15:28] <KenMan> things become imbalanced when the user gets to gobble the available outgoing quota, at the expense of peers who want to route through that user
[15:28] <toad_> thelema: what do you think of the following argument: If we have a small fixed file size, then we will have more requests, and correspondingly lower MRIs? Does that seem correct to you?
[15:28] <toad_> the question then is will the lower MRI make any difference or will the increased requests mess it up?
[15:28] <KenMan> my money is on the 'mess it up' part :)
[15:28] * toad_ thinks lower MRIs would at least give us more scope for doing intelligent tradeoffs...
[15:29] <thelema> I think that the network (mainly the protocol) isn't ready for small fixed file size
[15:29] <toad_> thelema: obviously we'd need binary presentation
[15:29] <KenMan> toad still lacks KenMan's higher-understanding of MRIs (hahaha) ;)
[15:29] <toad_> and we'd simplify the estimators slightly
[15:29] <KenMan> i jest with you
[15:29] <toad_> KenMan: what were you saying about tradeoffs?
[15:29] <thelema> the transition from 0.3 to 0.4 really sucked because 0.4 introduced many many more queries to the network (DBRs and mapfiles)
[15:29] <KenMan> but the math would work better than it does currently
[15:30] <toad_> thelema: hmmm
[15:30] <toad_> IMHO 0.4 worked better than 0.3
[15:30] <thelema> the protocol was initially designed with much lower request thoroughput in mind
[15:30] <toad_> with the exception of the DSB
[15:30] <toad_> mapfiles REDUCED the number of queries btw
[15:30] <thelema> toad_: actually, in many respects (like simple, stupid finding of data), 0.3 worked better.
[15:30] <thelema> mapfiles increased the number of queries.
[15:30] <toad_> before mapfiles we had quite hideous overheads for freesites
[15:30] <toad_> no, they didn't
[15:31] <thelema> before mapfiles, freesites used CHK@ links or direct SSK@ links
[15:31] <toad_> a freesite with 5 files on it would be 10 files without mapfiles
[15:31] <toad_> NOBODY used direct CHK links
[15:31] <toad_> with one exception
[15:31] <KenMan> so roughly 1% of my 1K qphs succeed, and the other 999 bang their heads against other failing nodes and queries. So increasing qph is the solution ??
[15:31] <toad_> now it has 6 files
[15:31] <thelema> where do you get your 10 number from?
[15:31] <toad_> CHK links don't work if you have more than one page
[15:31] <toad_> thelema: if you use SSKs, you have the SSK, and you have the CHK
[15:32] <toad_> for EVERY SINGLE FILE
[15:32] <thelema> toad_: CHK links work as long as your site is a DAG.
[15:32] <toad_> at least if it was over 32kB
[15:32] <thelema> only for files > 32K
[15:32] <toad_> including metadata
[15:32] <thelema> most HTML files were > 32K
[15:32] <thelema> most graphics > 32K were CHK'ed in.
[15:32] <toad_> thelema: well either way, 1 extra file in a medium size freesite is no problem
[15:32] <thelema> it'
[15:32] <toad_> and NOBODY USED CHK LINKS FOR IMAGES
[15:33] <toad_> KenMan: so what IS the solution then?
[15:33] <iip_i2p> <jnk> i did
[15:33] <toad_> KenMan: routing can't work as long as 90% of the RT is backed off
[15:33] <thelema> toad; anyway, the # of requests went up a *lot* between 0.3 and 0.4
[15:34] <toad_> thelema: frost?
[15:34] <thelema> I think it was caused by things other than just # of users, but it could have been that.
[15:34] <thelema> frost definitely helped, but even in the early days before frost and key-probing clients
[15:34] <toad_> when did freenet acquire FEC splitfile support?
[15:34] <thelema> late in 0.4
[15:34] <toad_> imho that will have been a watershed in terms of data transfer and perhaps queries too
[15:35] <toad_> since nonredundant splitfiles won't work for moviez
[15:35] <thelema> Splitfiles (without redundancy) were introduces at the beginning of 0.4, and they helped a lot in increasing # queries.
[15:36] <thelema> in my mind, freenet requests are basically two-phased: search and data xfer.
[15:36] <thelema> data xfer really can't be optimized any more than what we have.
[15:37] * hirvox (~hirvox@cs181027153.pp.htv.fi) has joined #freenet
[15:37] <thelema> but the search phase definitely can. (and I'm not recommending just better routing algorithms/NGR estimators)
[15:37] <thelema> (the only data xfer optimization that's even remotely feasible is data chording)
[15:37] * toad_ recommends thelema get the binary presentation out :)
[15:38] <thelema> can I get someone to work with me on it?
[15:38] <toad_> thelema: perhaps
[15:38] <thelema> I had iakin for a while, but he way overengineered something that should have been a simple hard-coded array of records.
[15:38] <toad_> i have other stuff to work on right now
[15:39] <toad_> but i will be available probably
[15:39] <toad_> what do you need to change?
[15:39] <toad_> what's the current status?
[15:39] <thelema> The code is complete but untested. It'll probably throw some exceptions at runtime letting whoever runs it know that there's some data types that need to be implemented.
[15:40] <toad_> <mule2p> does anybody have an idea whether fred/fproxy and the machine running it can be compromized by remote access to fproxy? - it depends what you mean by compromized...
[15:40] <thelema> And There's probably one or two brainfart type bugs left from initial coding.
[15:40] <thelema> it's pretty simple code.
[15:40] <KenMan> routing can't work as long as 90% of the RT is backed off - i will most certainly NOT argue with that
[15:41] <toad_> thelema: okay, so wiring it up for localtestnet would be how difficult? i assume it's back compatible?
[15:42] <KenMan> backoff is a direct consequence of rate limiting , much less (or not at all) NGR.
[15:42] <KenMan> I would work with thelema on binary presentation.
[15:42] <toad_> KenMan: well, arguably it's related to NGR having problems
[15:42] <KenMan> I would willing hammer out a spec...
[15:42] <KenMan> toad, i think we can improve on rate limiting independent of NGR
[15:42] <thelema> toad_: it's a presentation; they're negotiated during handshaking, right? and they're part of noderefs, right?
[15:43] <toad_> <hobx_> though of course, I could never respect a man who lived by his principles, so... - LOL!
[15:43] <toad_> thelema: you implemented a Presentation for it?
[15:43] <KenMan> thelema, you want to work with me at the bit level to specify FNP ?
[15:43] <toad_> in that case we just have to register both, right?
[15:44] <hobx> Sometimes I sound like my old namesake...
[15:44] <toad_> <theLostFloppy> By the way, how can I read the backlog? - umm, http://newton.matcmp.ncc.edu/~lockej/freenet/chanlog/
[15:46] <toad_> thelema: they get registed in Main, use eclipse's tools to find callers of MuxProtocol constructor
[15:47] <thelema> toad_: yes; FNCRawMessage
[15:48] <thelema> KenMan: I wouldn't mind rewriting the protocol but I think I won't have the time for it. I leave for Tanzania in september for the Peace Corps)
[15:49] <toad_> you will be missed
[15:49] <KenMan> neat. That's a shame. But it would be nice if we could incorporate your work somehow before then...
[15:49] <KenMan> I mean, it's great for Tanzania, just not Freenet !
[15:50] <thelema> as far as what I've doe for binary protocol, it's pretty simple.
[15:51] <thelema> the text parts of the protocol are all in messages of the form <MessageType><FieldSet>
[15:51] <KenMan> ahh easily extensible for others to work with then
[15:52] <thelema> there's less than 128 messageTypes used, so I encode each as 0x7F || <messageTypeByte> (simple table lookup each way)
[15:52] <KenMan> :)
[15:52] <thelema> fieldsets aren't a big deal either.
[15:52] <thelema> It's pretty easy to either have a table for each messageType or a global table
[15:53] <thelema> (at the moment it's global, but if we want to we can make one table per message type, if we run out of bytes
[15:53] <KenMan> yeah, i see where you have been headed.
[15:53] <KenMan> and it still allows new options to be initially coded as text
[15:53] <toad_> KenMan: you planning on helping thelema code-wise?
[15:53] <thelema> and just replace variable=value with <encoded_variable><encodedValue> based on the name of variable.
[15:54] <toad_> or is there something i can do right now?
[15:54] <KenMan> i think I can help him code wise... i would need to learn what he has done, and that gets me familiar with more parts too...
[15:54] <toad_> KenMan: good idea
[15:54] * toad_ will plough on with other stuff then
[15:54] <thelema> the protocol allows for future extension (as long as we don't use high ascii as the first letter of a variable or message type
[15:54] <toad_> I like binary presentation
[15:54] <thelema> KenMan: look in presentations/FNCRawMessage.java
[15:54] <KenMan> well, each FNP version would incorporate both fixed binary definitions, and extensible text support
[15:54] <toad_> it may let us reduce the padding size further, although if we do we run into issues with trailers
[15:55] <thelema> toad_: to do this right would require rewriting the protocol.
[15:55] <thelema> toad_: and not just the presentation
[15:55] <toad_> issues that we probably want to deal with anyway...
[15:55] <toad_> thelema: huh?
[15:55] <KenMan> thelema: okay, FNCRawMessage. I agree with rewriting the protocol. And specifying it !!!
[15:55] <toad_> we are talking freenet/presentation/CompressedProtocol, FNCRawMessage, and FieldSet, right?
[15:56] <thelema> my solution is just a clever hack taht cuts down # of bytes being sent. It should be reasonable to squeeze some ..
[15:56] <toad_> those are the only files to be significantly changed, right?
[15:56] <toad_> CompressedProtocol would be aka MuxProtocol
[15:56] <toad_> and would be registered as another version in Main
[15:56] <toad_> all this is not too difficult, right?
[15:56] <thelema> it shouldn't be, it just needs testing to get teh last bigs out.
[15:56] <KenMan> there are greater ramifications as you get into it, there always are with this kind of thing...
[15:56] <thelema> *bugs
[15:57] <thelema> yes, those are the only files.
[15:57] <toad_> thelema: how did you decide to deal with StoreData's excessively long field names?
[15:57] <KenMan> cool, i will eat thelema's code ...
[15:57] <thelema> although I don't think we'd need CompressedProtocol even.
[15:57] <toad_> okay, so you and KenMan can have it tested and mostly working by Monday? :)
[15:57] <toad_> thelema: why not
[15:57] <toad_> ?
[15:57] <toad_> how are you going to specify it otherwise?
[15:57] <thelema> toad_: i haven't. I'm hoping they'll go away.
[15:58] <toad_> I know you could add it to the message, or have it as a constructor arg... but it's cleaner to have it as a separate presentation protocol number, right? and it's cleaner for each one of those to have its own Protocol subclass?
[15:58] <toad_> thelema: well, either we compress it via your code, or we have a binary presentation implemented on a different level
[15:58] <toad_> for it
[15:59] <toad_> which would be more efficient of course
[15:59] <toad_> but then we have the possibility of doing that for all sorts of FieldSet producing entities...
[15:59] <thelema> It is a seperate presentation protocol number.
[16:00] <thelema> I didn't think I had to do more than rewrite FCPRawMessage.java
[16:00] <toad_> 1 byte for StoreData. 1 byte for estimator fieldset. 2 bytes for length. 1 byte for version. <highly compressed binary version produced by NodeEstimator.toBinaryFieldSet().>
[16:01] <toad_> then the other StoreData bits in the usual binary presentation format?
[16:01] <toad_> okay maybe it's 1 byte for "custom fieldset serialization format", then 1 byte for estimators, then 1 byte for estimator type, then 1 byte for estimator version
[16:02] <toad_> 2 bytes for length at the beginning
[16:02] <toad_> and finally the compressed data?
[16:02] <toad_> thelema: you mean FNPRawMessage?
[16:02] <thelema> it would be a bit ugly to have custom code for 'Estimator' as a field name for compressing...
[16:02] <thelema> toad_: oops, yes, I did mean that.
[16:02] <toad_> well it's not a field name
[16:02] <toad_> it's an entire sub-fieldset
[16:03] <thelema> because I would have to handle it on the level of a completely constructed fieldset.
[16:03] <toad_> i think that might be cool
[16:03] <thelema> you want to have a third possibility for fieldset entries?
[16:03] <toad_> thelema: ahh
[16:03] <toad_> yes that's the difficulty
[16:03] <thelema> string=value, Fieldset, or compressedFieldset?
[16:03] * toad_ thinks ideally the binary form would be the native version... or some structure close to it would be
[16:03] <thelema> where compressedFieldset has a way of being written out in a binary form nicely...
[16:04] <toad_> why Fieldset?
[16:04] <thelema> and read back in.
[16:04] <toad_> ohhh
[16:04] <toad_> yes
[16:04] <toad_> that's the idea
[16:04] <toad_> could that be made to work reasonably neatly?
[16:04] <KenMan> why not ?
[16:04] <thelema> hmm... I could extend fieldset to do that.
[16:04] * hobx (~hobx@h196n1fls21o1077.bredband.comhem.se) Quit (Remote closed the connection)
[16:04] <KenMan> :)
[16:04] <toad_> well it'd probably save us 10% or so on StoreData
[16:04] <thelema> I'd have to make sure that the compressedFieldset is capable of being traversed just like a normal fieldset
[16:04] <toad_> i suppose it's possible that it wouldn't be significant...
[16:05] <toad_> thelema: what do you mean by traversed?
[16:05] <KenMan> abstracted away...
[16:05] <toad_> CompressedFieldSet would have to be capable of being seen as an ordinary fieldset
[16:05] <thelema> hmm, I see.
[16:05] <thelema> even more interesting...
[16:05] <toad_> in fact, perhaps CompressedFieldSet would be an interface ?
[16:05] <thelema> I could just extend fieldset...
[16:06] <toad_> so we return a CompressedFieldSet, of our own custom implementation
[16:06] * Orasis (~orasis@c-66-41-30-194.mn.client2.attbi.com) Quit (Remote closed the connection)
[16:06] <toad_> we never actually converted to messy stringy formats
[16:06] <toad_> but it can be browsed that way and will make it up on the fly, when needed?
[16:06] * hobx (~hobx@h196n1fls21o1077.bredband.comhem.se) has joined #freenet
[16:06] * ChanServ sets mode +o hobx
[16:06] <toad_> does this seem completely crazy?
[16:06] <KenMan> only (de)compress at send recv times
[16:07] <KenMan> to RawMessage i suppose
[16:07] <thelema> it would be an interface that extends FieldSet. It would be exactly the methods of fieldset plus two functions for binary serialization
[16:07] <toad_> KenMan: well, making a conventional fieldset only to then turn it back into binary is pretty wasteful...
[16:07] <toad_> well, FieldSet would implement the interface
[16:07] <thelema> wait, don't we already have an interface for binary serialization?
[16:08] <toad_> we have BaseFieldSet, which isn't writable, but is traversible
[16:08] <toad_> we have StandardFieldSet, which is what FieldSet is now
[16:08] <toad_> is writable and traversible
[16:08] <toad_> and we have lots of custom compressible fieldsets derived from BaseFieldSet?
[16:09] <thelema> too many classes. don't we already have an interface that specifies that the object is able to be serialized to a datastream?
[16:09] * toad_ wonders if all this is worthwhile... there might be difficulties with versioning...
[16:09] <toad_> thelema: yeah, there's an interface...
[16:10] <toad_> current impls use an int for magic and an int for version
[16:10] <toad_> because they're intended to go to disk
[16:10] <thelema> toad_: I'll handle it.
[16:10] <toad_> if the serialization format depends on the exact code version... could be problematic
[16:10] <toad_> hmmm
[16:10] <thelema> thanks for the ideas, I'm going to ... doh, I only have 15 minutes to program this before I leave for birthday party,
[16:10] <toad_> LOL
[16:11] <thelema> toad_: try not to over OO-ify this. Keep it *SIMPLE*
[16:11] <toad_> you'll be here tomorrow?
[16:11] <toad_> (I won't)
[16:11] <KenMan> i will
[16:11] <thelema> I don't know how long I'll be here tomorrow. I expect to be busy from 10AM to 8-9PM (CST)
[16:11] * toad_ doesn't think defining appropriate interfaces is less simple than sticking lots of unrelated and complex methods on a single object
[16:11] <toad_> okay well
[16:11] <toad_> tell me what you come up with
[16:12] <toad_> it may be that what i suggested doesn't make sense, that the gains are too small
[16:12] <toad_> otoh there may be a really simple way to do it
[16:12] <toad_> e.g. CompressibleFieldSet implements serializable (or whatever it's called), extends FieldSet
[16:12] <thelema> I think there is a good way to do it.
[16:13] <toad_> okay
[16:13] <thelema> and yes, it's something like this last option.
[16:13] <toad_> tell me what you come up with
[16:13] <toad_> seeya
[16:13] <KenMan> and eat lots of cake. Tanzanian cakes will have you longing for home cakes
[16:13] <thelema> It's basically: if the sub-fieldset is easily serializable, do it.
[16:13] <toad_> thelema: how'll you handle versioning?
[16:13] <thelema> if not, serialize through iterating fields
[16:13] <thelema> versions will be static.
[16:14] <toad_> if you use the existing disk serialization methods, then you have the existing versioning code
[16:14] <toad_> which is good but bulky
[16:14] <thelema> The compression table will *need* to be fixed for a particular presentation version.
[16:14] <toad_> each RunningAverage has 8 bytes of magic+version
[16:14] <thelema> no, I won't be using disk serialization methods.
[16:14] <toad_> no i mean versioning for the ubercompressed binary format
[16:14] <thelema> I should be able to come up with something better.
[16:14] <toad_> okay good
[16:15] <thelema> for estimators, I'll just format the raw data, and read it back.
[16:15] <toad_> it needs to HAVE versioning... but it needs to be compact
[16:15] <toad_> format the raw data?
[16:15] <toad_> it seems important to me to compress StoreData as much as reasonably possible
[16:15] <toad_> that's what I want out of all this
[16:16] <toad_> after that, it may be worth thinking about making StoreData's estimator data a trailing field
[16:16] <toad_> and we can shrink the padding chunk size, perhaps (modulo concerns about trailers...)
[16:17] <ejhuff> toad_: did you notice transfers active > datastateinitiators?
[16:17] <ejhuff> Build Number 5090
[16:17] <ejhuff> Uptime 14 hours 56 minutes
[16:17] <ejhuff> freenet.node.states.data.DataStateInitiator 16
[16:17] <ejhuff> Connections open (Inbound/Outbound/Limit) 177 (130/47/800)
[16:17] <toad_> ejhuff: wtf?
[16:17] <ejhuff> Transfers active (Transmitting/Receiving) 105 (83/22)
[16:17] <toad_> aaaaaaaaaaaAAAAAAAAAAAAAAAAAAAAARrRRRRRRRrrRRREHwrehg|fs B?!
[16:18] <ejhuff> That's what my node says...
[16:18] <thelema> making it a trailing field is garbage.
[16:18] <toad_> no, never happened here
[16:18] <toad_> thelema: well, it's never going to go below 1kB
[16:18] <toad_> and all our other messages are much smaller than that
[16:18] <thelema> as far as versioning, it should suffice for old NGR's data about nodes to be ignored by new NGR implementations.
[16:18] <thelema> and vice versa
[16:19] <toad_> thelema: well, they need to know what they are dealing with
[16:19] <thelema> not if it's incompatible data
[16:19] <toad_> well yeah but they need to know WHETHER it is incompatible
[16:19] <thelema> they will.
[16:19] <toad_> how?
[16:19] <thelema> version byte + length byte
[16:20] <toad_> okay
[16:20] <toad_> make it 2 bytes for version though
[16:20] <thelema> if you go over 256 NGR implementations, increment presentation.
[16:20] <toad_> ewww
[16:20] <thelema> or just wrap, because noone should be using a version 256 old.
[16:20] <toad_> come on, this is a BIG object
[16:20] <thelema> are you planning on using more than 256 versions?
[16:21] <toad_> thelema: it gets changed slightly quite often
[16:21] <thelema> and it's up to < 10 now.
[16:21] <toad_> often the slight changes are intended to be backwards compatible
[16:21] * hobx (~hobx@h196n1fls21o1077.bredband.comhem.se) Quit (Remote closed the connection)
[16:21] <toad_> given that it's supposed to imply the factories for all the running averages, and the KeyspaceEstimators, and so on
[16:22] <toad_> 2 bytes is reasonable
[16:22] <toad_> for such a big object
[16:22] <toad_> not necessarily universally
[16:22] <thelema> like adding new fields... but that won't work so well with a network protocol, because you're not only dealing with old -> new; you have to deal with new -> old.
[16:22] <toad_> thelema: hrmm
[16:22] <toad_> yeah...
[16:22] <toad_> perhaps it's a silly idea
[16:23] <toad_> perhaps the space savings aren't that great after all
[16:23] * hobx (~hobx@h196n1fls21o1077.bredband.comhem.se) has joined #freenet
[16:23] * ChanServ sets mode +o hobx
[16:23] <toad_> or perhaps it needs to be on a finer grained level
[16:23] <thelema> in any case, I'll give you your two bytes. Hopefully there'll be enough StoreDatas to make it inefficient.
[16:23] <thelema> what needs to be finer?
[16:24] <toad_> e.g. StandardNodeEstimator { pDNF = subfieldset, rTransferRate = subfieldset, etc }
[16:24] <toad_> so we can add estimators if we need to
[16:24] <toad_> and then pDNF .. hmmm
[16:24] <toad_> it gets absurd
[16:25] <toad_> pDNF: type of pDNF estimator impl (1 byte), version of pDNF estimator impl (1 byte)
[16:25] <toad_> type and version of RunningAverage impl
[16:25] <toad_> then we have all the raw data
[16:25] <toad_> that COULD be back compatible
[16:26] <toad_> within reason anyway
[16:26] <toad_> as long as you include length bytes and add stuff on the end
[16:26] <toad_> (but that might need to be explicitly broken occasionally, in which case you just increment the impl version)
[16:26] <thelema> no, that's a little bit on the crazy side.
[16:26] <toad_> thelema: anything where you have backwards compatibility is going to be a little on the crazy side afaics
[16:27] <toad_> so perhaps it's simpler to just do what you've so far done
[16:27] <thelema> but anyway, I just need to rewrite the reading function for compressed streams so I can handle the custom writers...
[16:27] <thelema> It is much simpler. And it's got a much higher chance of working.
[16:28] <toad_> i don't think custom writers are worthwhile
[16:28] <toad_> because they necessarily break compatibility
[16:28] <toad_> anytime anything is changed
[16:28] <toad_> whereas invalid fields would just get ignored otherwise
[16:28] * hobx (~hobx@h196n1fls21o1077.bredband.comhem.se) Quit (Remote closed the connection)
[16:30] * hobx (~hobx@h196n1fls21o1077.bredband.comhem.se) has joined #freenet
[16:30] * ChanServ sets mode +o hobx
[16:32] <ejhuff> toad: Circumstances of active output xfers > datastateinitiators: I have a shell script calling "freenet.client.cli.Main put CHK@ file --skipDS --dontGuessType --noredirect --htl 20", with retry for a given key after 10 minute delay, 2 cli's running at a time for a bunch of splitfile fragments (raw CHK's, zero bytes of metadata). The cli's frequently die with EOFException caused by the 5 minute timeout in the nio code closing the connection while
[16:34] <toad_> ejhuff: aha
[16:35] * Ash-Foxeus (Hal-9000@aas115.neoplus.adsl.tpnet.pl) Quit (Client Quit)
[16:35] * KenMan (~chaziller@pcp403292pcs.mntcrm01.md.comcast.net) Quit (Remote closed the connection)
[16:35] <toad_> ejhuff: I will look into that at some point
[16:36] <toad_> it may shed some light on Frost's problems too
[16:36] <ejhuff> Presumably fred fails to decrement count of active output xfers when the connection closes under those circumstances.
[16:36] * KenMan (~chaziller@pcp403292pcs.mntcrm01.md.comcast.net) has joined #freenet
[16:36] <toad_> and generally i have heard occasionally that inserts stil timeout
[16:36] <toad_> ejhuff: OUTPUT xfers?
[16:36] <ejhuff> Well, cli is doing an insert.
[16:36] <toad_> it's the input xfers that correspond to DSIs
[16:36] <toad_> output xfers don't need DSIs
[16:37] <toad_> it's the input from the client, or something...
[16:37] <ejhuff> Hmm...
[16:37] <toad_> that leaks
[16:37] <ejhuff> Oh. I thought the bug you fixed was with output xfers....
[16:42] <ejhuff> Is there a separate way to verify the number of output xfers, or are they correct?
[16:42] <toad_> hmmm
[16:43] <toad_> how do i turn a FreenetURI into an actual Key?
[16:43] <thelema> toad; you mean a routingKey?
[16:43] <thelema> look in ClientCHK/SVK
[16:44] <ejhuff> I was looking at that, it's the base64 stuff. Done in some screwy but standard way.
[16:45] * Ash-Fox (Hal-9000@aas115.neoplus.adsl.tpnet.pl) has joined #FreeNET
[16:45] <KenMan> requestSuccessRatio/hr:
[16:45] <KenMan> 2014 59 0.02929493545183714
[16:45] <KenMan> 1992 56 0.028112449799196786
[16:45] <KenMan> 1820 58 0.031868131868131866
[16:45] <KenMan> 1963 60 0.030565461029037188
[16:45] <KenMan> 2984 66 0.022117962466487937
[16:45] <KenMan> 4814 79 0.01641046946406315
[16:45] <KenMan> 4216 75 0.017789373814041744
[16:45] <KenMan> 2657 73 0.027474595408355288
[16:45] <KenMan> there is an obvious correlation here between qph and success ratio. I wonder why. Don't you ?
[16:46] <thelema> the reason is that nodes seem to be able to handle only a certain number of successful transfers per unit time.
[16:46] <thelema> if they are getting a lot of xfers that fail, they canaccept more
[16:46] <thelema> if their xfers are succeeding, they have to accept less
[16:47] <KenMan> now, if i only took in 250 queries per hour, I wonder what my success rate would be...
[16:47] <KenMan> but if all nodes only took in 250 qph, i wonder if the current success ratio would change at all...
[16:48] <KenMan> hehe
[16:48] <toad_> KenMan: eek
[16:48] <toad_> KenMan: well, the KenManHack looks like it could be useful there!
[16:49] <KenMan> I'll give it a whirl...
[16:49] <toad_> well yes, that's the question...
[16:49] <toad_> in fact there MIGHT be some indirect feedback
[16:49] <toad_> that could lead to Really Bad Things
[16:50] * KenMan is not ashamed of his proposed change to mRI representation...
[16:50] <KenMan> it helps me to think about the problem, anyway.
[16:50] <toad_> KenMan: your what?
[16:50] <KenMan> that deal where I redefined how mRI was used for backoff...
[16:50] <toad_> huh?
[16:51] <KenMan> http://mywebpages.comcast.net/jkcorson/FreeNet/queryRateControl.png
[16:51] <toad_> ah
[16:51] <KenMan> but i'm not trying to start up a discussion about it.
[16:51] <toad_> well if it helps you understand the problem... ;)
[16:51] <KenMan> exactly
[16:52] <toad_> well
[16:52] <toad_> if all nodes take in 250qph, then MRIs will be rather high
[16:52] <toad_> and all nodes will be backed off
[16:52] <toad_> :|
[16:52] <toad_> so it's a familiar pattern
[16:52] <KenMan> you are speaking from experience with existing code.
[16:52] <toad_> beneficial to one node but catastrophic if everyone does it
[16:52] <toad_> just like having lots of conns
[16:52] <KenMan> yup
[16:52] <KenMan> uh huh
[16:52] <toad_> well no maybe not
[16:53] <toad_> lots of conns is good
[16:53] <KenMan> do you understand the ramifications of having all nodes use a constant for MRI ? I'm not arguing we do that, just asking a question...
[16:53] <KenMan> number of peers would determine query capacity
[16:53] <KenMan> right ?
[16:53] <toad_> hmmm
[16:56] <thelema> KenMan: in your proposal: why wouldn't the requesting node always send its request as soon as possible in the second scenario?
[16:57] <KenMan> that is a more-than-minor detail i left out. Nodes must adjust the incoming qph to 'match' this scheme, where 50% use (over time) would be optimal.
[16:57] <KenMan> ie, each route would be backed off 50% of the time.
[16:57] <KenMan> roughly.
[16:58] <thelema> again, that's a great goal, but I'd shoot for having routes backed off 0% (or very close to that) of the time
[16:58] <KenMan> 50% backoff should equate to 2nd choice routing for NGR.
[16:59] <thelema> 0% would equate to 1st place.
[16:59] <thelema> anyway, time for me to go. It's party time.
[16:59] <toad_> we cannot have incoming MRI affect outgoing MRI directly
[16:59] * thelema is now known as thelema|away
[16:59] <KenMan> 50% would actually equate to an average of 1.5st place. Have fun. Is it Your birthday ?? thelema|away
[16:59] <toad_> it can't have any direct impact
[16:59] <toad_> or we get freezedown
[17:00] * toad_ committing 60185
[17:00] <KenMan> then there is something wrong with the model, toad. That's just my opinion.
[17:00] * hobx (~hobx@h196n1fls21o1077.bredband.comhem.se) Quit (Remote closed the connection)
[17:00] <KenMan> RNFs mess up EVERYTHING.
[17:00] <toad_> KenMan: well, every time we've tried it, that's what has happened
[17:00] <KenMan> i know, and someday we might get it right...
[17:01] <KenMan> maybe. Or I may be living in a dream world... of theory based in non-experience :)
[17:01] <toad_> it's just possible that it was made worse by some bug we've fixed since
[17:01] <toad_> however i'd need pretty solid theoretical or simulated proof that it was going to work
[17:01] <KenMan> and exactly how many times have we tried it ? ;)
[17:02] <toad_> s/proof/evidence
[17:02] <toad_> at least 3 times iirc
[17:02] <toad_> maybe it was 2
[17:02] <KenMan> i remember one of them . Anyway, I spend much time thinking about the problem every day, soon something may click. In a way I can communicate, and demonstrate.
[17:02] * hobx (~hobx@h196n1fls21o1077.bredband.comhem.se) has joined #freenet
[17:02] * ChanServ sets mode +o hobx
[17:03] * hobx (~hobx@h196n1fls21o1077.bredband.comhem.se) Quit (Client Quit)
[17:07] <toad_> KenMan: I tried to calculate what the criteria would be for it to not cause a meltdown
[17:07] <toad_> i was unsuccessful :(
[17:11] <KenMan> okay, time for some philosophy here... if a node joins the network, and aquires the 'standard' number of peers, whatever level that is, then...
[17:12] <KenMan> the network has certain amount of resources it can offer to the new node, and the new node has a number of resources it can offer back to the network.
[17:12] <KenMan> if the number of queries going out is approximately 3x the number of queries coming in, do you see an imbalance ??
[17:13] <KenMan> i am mainly referring to qph levels (defined by MRIs) as the primary resource here.
[17:14] <KenMan> these RNFs are killing us, as we determined many moons ago.
[17:14] <KenMan> it looks like the primary cause is timeouts, huh ?
[17:14] <toad_> no RNFs
[17:15] <toad_> not real timeouts
[17:15] <toad_> well
[17:15] <toad_> maybe there are timeouts
[17:15] <toad_> but there are definitely real RNFs too quite often
[17:15] <toad_> meaning a QR received after the Accepted
[17:15] <KenMan> conceptual "box model" representation: queries in -> (rate limiting + NGR) -> queries out , and NGR is forced to be subservant to rate limiting.
[17:16] * oierw` is now known as oierw
[17:16] <KenMan> if NGR can't get decent routes with low backoff, we expect success to plummet. That is our working theory anyway.
[17:16] <toad_> KenMan: very likely
[17:17] <toad_> I'm not sure the concept of rate limiting can work if it can't propagate across the network...
[17:18] <KenMan> most definitely. And this multiplication factor screws with the current as-coded model.
[17:18] <toad_> hmmm
[17:18] <toad_> perhaps that is the problem
[17:19] <toad_> however the multiplication is limited...
[17:19] <toad_> evidently it's not limited enough?
[17:19] <toad_> what's the OVERALL multiplication factor?
[17:19] <KenMan> lets check in on OCM message counts... DataRequests sent/recvd = 2.15 here, based on DataRequest counts from OCM
[17:19] <toad_> not just searchFailed, taking into account successes from store and so on?
[17:19] <toad_> KenMan: that's overall
[17:19] <toad_> ?
[17:19] <KenMan> is that a fair evaluation of it ??
[17:20] * toad_ thinks so, very probably
[17:20] <KenMan> i like to ignore successes from store, because they will only help the matter.
[17:20] <toad_> eh?
[17:20] <KenMan> that is better than the 3:1 we had several months ago.
[17:21] <toad_> successes from store will improve matters yes
[17:21] <toad_> Number of requests (sent/received) 5120/1977
[17:21] <toad_> yikes
[17:21] <KenMan> you are still a young node.
[17:21] <KenMan> i've only been up 22 hours, but ...
[17:21] <KenMan> 113892/52967 here
[17:22] <toad_> KenMan: hmmm
[17:22] <toad_> no local traffic?
[17:23] <KenMan> none from me. Maybe 10 requests worth.
[17:23] <KenMan> Now, suppose I start browsing. How do you think the balance will be affected ??
[17:23] <KenMan> suppose I create 200 requests per hour. Remember I process around 1500 an hour.
[17:24] <KenMan> It will subtract from what I can contribute (as my queries consume a percent of available qph-out resources)
[17:24] <KenMan> it is very very hard to talk about user behavior in qph. Q/ms is more like it...
[17:25] <KenMan> or ms/Q , however you like
[17:25] <KenMan> and i only have a new route becoming available every 2000ms, on average. It doesn't average very well, because they are not quite appearing with that regularity
[17:26] <toad_> okay
[17:26] <toad_> well
[17:26] <toad_> how do we prevent RouteNotFound from multiplying our requests?
[17:26] <toad_> it seems circular
[17:26] <KenMan> local use of outbound qph quota causes higher RNFs for my peers
[17:26] * toad_ suggests the answer is queueing
[17:27] <KenMan> rate limiting local queries is a partial solution, likely to have massive (good) impact.
[17:27] <toad_> or...
[17:27] <toad_> well
[17:27] <toad_> KenMan: perhaps
[17:27] <KenMan> But users will complain that they cannot issue 10 requests per second, when they browse.
[17:27] <KenMan> so maybe THEY need to be queued / regulated
[17:27] <toad_> but suppose we don't have an overwhelming number of queries coming from the client
[17:27] <toad_> suppose that the basic problem is in fact the multiplication factor
[17:27] <toad_> how do we abolish that?
[17:28] <KenMan> that's my situation. Then we focus there. some RNFs can't be counted (as in a loop)
[17:28] <toad_> how to do? well, once we have accepted a request, we don't return an RNF just because everyone is backed off
[17:28] <toad_> or we try very very very hard not to do this
[17:28] <KenMan> we need to better characterize RNFs, by counting the different cases. A reason field would help maybe.
[17:29] <KenMan> what else can we possibly do ??
[17:29] <toad_> we can queue
[17:29] <KenMan> Queueing is an option, but can we improve without it ??
[17:29] <toad_> if we can eliminate the multiplication factor, then we might even be able to have the queue length influence outgoing MRIs
[17:29] <toad_> KenMan: we can, but probably not enough
[17:29] <toad_> anyway
[17:29] <KenMan> If requests could specify (best routing -w/queueing , or fastest routing -no queueing) that might be an option
[17:30] <toad_> thanks, I think you've enabled me to see how queueing will make a difference
[17:30] <KenMan> good enough
[17:30] <toad_> unfortunately, right now that would cause severe delays
[17:30] <KenMan> you are fighting the pop-up availability of routes , which is a random occurence
[17:30] <toad_> KenMan: is it?
[17:30] <toad_> lets assume we have a completely reset network
[17:30] <KenMan> in terms of intervals between pop-ups, yes
[17:31] <toad_> lets also assume no excessively long transfers, i.e. fixed sizes
[17:31] <KenMan> it is not exactly every 200ms anyway. It varies a lot from routing to routing.
[17:31] <toad_> our MRIs should be low enough for queueing to be practicable
[17:32] <KenMan> more peers = higher mRIs by necessity
[17:32] * |ux (~|ux@82-37-17-24.cable.ubr03.wolv.blueyonder.co.uk) has joined #freenet
[17:32] <|ux> heh
[17:32] <toad_> hmmm
[17:32] <toad_> can we get somewhere without the above?
[17:32] <toad_> especially without fixed file sizes?
[17:33] <KenMan> assuming we can only process X qph as a limitation of our CPU + net resources...
[17:33] <toad_> well
[17:33] <toad_> hypothetically
[17:33] <|ux> any updates i should know about toad_?
[17:33] <KenMan> is that unreasonable to say ? each node has a limit of X ? assuming success rates don't go wild ??
[17:34] <toad_> KenMan: hypothetically, if we abolish ALL timeouts
[17:34] <|ux> ahh 1min
[17:34] <toad_> just as a thought experiment
[17:34] <|ux> try again
[17:34] <|ux> so whats this update?
[17:34] <toad_> then it would always eventually return DNF or DF
[17:34] <toad_> right?
[17:34] <toad_> |ux: various minor improvements
[17:34] <|ux> ok
[17:34] <|ux> send again
[17:35] <|ux> i had to remove the old jar from the send dir
[17:35] <toad_> including saving the global rate limiting changes, and fixing KSK support in fproxy
[17:35] <toad_> KenMan: okay
[17:35] <toad_> thought experiment
[17:35] <toad_> we always return either a transfer, a DNF or a DF
[17:35] <toad_> therefore, our output queries are lower than our input queries
[17:36] <toad_> we track our average queue time
[17:36] <toad_> and we factor that into our load
[17:36] <toad_> our outgoing MRIs
[17:36] <toad_> somehow...
[17:36] <toad_> I'm not really attacking routing accuracy here
[17:36] <toad_> just whether it gets routed AT ALL
[17:37] <toad_> KenMan: other radical option: always fail on RNF
[17:37] <toad_> immediately
[17:37] <toad_> and implement per node failure tables to compensate
[17:37] <toad_> so no retries whatsoever, like thelema suggested for some years
[17:37] * KenMan on phone, will catch up
[17:37] <toad_> a request either ends with a transfer, a DataNotFound, or a RouteNotFound
[17:39] <|ux> 1min
[17:39] <toad_> KenMan: why is the HTL not compensating for the QRs?
[17:39] <toad_> are you sure the QRs are the reason for the higher outgoing queries than incoming?
[17:39] <toad_> surely an equal number would expire due to out of HTL?
[17:40] <|ux> toad_: still 5090
[17:40] <|ux> toad:_ ?
[17:40] <toad_> |ux: expected
[17:40] <|ux> Node Version 0.5
[17:40] <|ux> Protocol Version STABLE-1.50
[17:40] <|ux> Build Number 5090
[17:40] <|ux> CVS Revision 1.90.2.50.2.119
[17:42] <toad_> bbl
[17:42] <KenMan> good thoughts toad
[17:42] <KenMan> don't immediately fail on RNF due to looping though
[17:42] <toad_> KenMan: what is the cause of the multiplying?
[17:43] <theLostFloppy> My frost won't update automatically. Anyone have this too?
[17:43] <KenMan> i think it is backoff spread around the net in inconvenient ways
[17:43] <toad_> KenMan: surely the QR retries don't cause it? wouldn't an equal number of reqs end on each node due to the decreased htl?
[17:43] <toad_> bbiab
[17:43] <|ux> theLostFloppy: ur frost is.. Frozen!?
[17:43] <theLostFloppy> Not if I do manual update
[17:44] <KenMan> the HTL decrementing is helped with QRs, but the help is distributed, not localize as we need it to be.
[17:46] <KenMan> do you have a reason field today that explains whether a request has looped ? we would want to detect and accomodate that special condition, perhaps (which should be rare-ish).
[17:48] * Hory (~Miranda@81.196.25.110) has joined #FreeNet
[17:57] <KenMan> are you sure the QRs are the reason for the higher outgoing queries than incoming? - reasonably. Compare counts for Accepted msgs. Also, QueryRejected received to DataRequests sent.
[17:57] <KenMan> that should help interpret the situation
[18:01] <KenMan> DNFs sent is an appreciable percent of total DataRequests received, which is reason to smile.
[18:02] <|ux> ummm (from a friend): sigh, freenet was working fine until I upgraded to 5090, now it's sucking again
[18:03] <KenMan> if we really did consider trying "no retries on RNF," then we would want to penalize the routes which gave higher percentages of RNF, if they significantly stood above from the crowd.
[18:03] <KenMan> |ux: what version did they upgrade from ? 5089 ?
[18:04] <|ux> KenMan: im asking now
[18:04] <|ux> KenMan: 5089
[18:04] <KenMan> if we have (or decide to add) a reason field for RNF due to looping, could we add one for RNF due to ALL routes backed off ??
[18:04] <KenMan> |ux: do they run frost ?
[18:05] <KenMan> is their complaint with regards to frost performance, or regular HTML browsing ?
[18:05] <|ux> ahh.. kenman: he says "it's not finding anything again"
[18:06] * Hamled|Erp (~hamled@pool-68-163-62-23.phil.east.verizon.net) has joined #freenet
[18:06] <KenMan> tell him to be patient. Walk away for 30 minutes to an hour, and try it again. That's the best advice I can offer nowadays. The network resources for browsing can change quite a bit in that amount of time.
[18:06] <|ux> Kenman: do they run frost? - no, not right now
[18:07] <KenMan> then they stand a better chance of good browsing, if frost is not running.
[18:07] <|ux> ok t/y
[18:08] * ornge (~ornge@h131n2fls34o879.telia.com) has joined #freenet
[18:12] <KenMan> just looking at my incoming side of things, about 50% of requests receive a DNF 27414/52967 .
[18:14] <theLostFloppy> Who is this mysterious _*wire*_
[18:14] <KenMan> the other half sprawl into 3 out-and-back RNFs, and the other half (not DNFs) of my incoming side queries returns RNF.
[18:14] <|ux> wire?
[18:14] <KenMan> i guess they both can sprawl equally.
[18:15] <KenMan> anyway, the results returned on my incoming side is split evenly between DNFs and RNFs, so it appears.
[18:15] <theLostFloppy> Yes wire, bold and oblique
[18:17] <|ux> where?
[18:17] <KenMan> But on my outgoing side, I receive more RNFs than DNFs, 3.6 : 1 ...
[18:17] <|ux> sorry im out of the loop here
[18:17] * KenMan has no wires... :(
[18:18] * pupok (~janie@81-178-89-164.dsl.pipex.com) Quit ("Leaving")
[18:18] <KenMan> not even italic and transparent ones...
[18:18] <toad_> hi
[18:19] <toad_> <KenMan> don't immediately fail on RNF due to looping though - hmmm, good point
[18:19] <toad_> we DO have a reason field
[18:19] <KenMan> i thoguht RNF was the reason , inside a QR
[18:19] <toad_> but the general consensus is it gives away too much info and is too easily spoofed
[18:19] <toad_> hrrm
[18:19] <toad_> well yeah
[18:20] <KenMan> well, put it in for now, and take it out later...
[18:20] * |ux is completely lost *again*
[18:20] <KenMan> we don't claim to protect users anon completely, ever. Our learning is rather more important that those issues, right now.
[18:21] <toad_> KenMan: yes but making routing behaviour depend on it is Ungood
[18:21] <KenMan> okay. i support abort on looping then.
[18:21] <toad_> abort on looping sounds like a really bad idea
[18:21] <KenMan> although you are still in 'unrestricted thought' mode, not coding mode :)
[18:21] * toad_ thinks that we shouldn
[18:21] <toad_> 't abort on QR
[18:22] <|ux> KenMan: anonyimity is the MAIN reason for freenet
[18:22] <toad_> instead, we should try really really really hard not to RNF
[18:22] <toad_> so we're back to queueing
[18:22] <KenMan> Okay. So we just say that really loudly at our node ? "Try hard not to RNF"
[18:22] <KenMan> ;)
[18:22] <toad_> well
[18:22] <toad_> it comes down to queueing
[18:22] <toad_> imho
[18:23] <toad_> a limited-horizons version would be simply to queue until we can send the request to SOMEBODY
[18:23] <KenMan> would the potential bad outweight the potential good of abort on all RNFs ??
[18:23] <toad_> a more ambitious version would be to queue until we can send to somebody in the top half
[18:23] <KenMan> and can we communicate 100% backoff in any case ??
[18:23] <toad_> KenMan: 100% backoff?
[18:24] <KenMan> that reveals that the forward node is overwhelmed, regardless of MRIs offered
[18:24] <|ux> toad_: i thought the RNF thing worked?
[18:24] <KenMan> 100% backoff. 'Query cannot even leave this node'
[18:24] <toad_> |ux: yes yes, we're talking about something else...
[18:24] * Hamled (~hamled@pool-68-163-62-23.phil.east.verizon.net) Quit (Read error: 110 (Connection timed out))
[18:24] <toad_> KenMan: well, if we have zero open connections, we'd RNF
[18:25] <KenMan> yes. Let's communicate that situation backwards, so we can measure the degree of it happening
[18:25] <toad_> so we track the statistic: how long does a request remain in the queue, before going to a node?
[18:25] <KenMan> of course, with retries, well...
[18:25] <KenMan> sure
[18:25] <KenMan> that's a really good one.
[18:25] <toad_> now, what can we do with that?
[18:26] <toad_> how do we convert it into globalQuota?
[18:26] <KenMan> let it control the volume of our system, as it reports the errors using text to speech translation.
[18:26] * TLF (francisco@121.Red-81-40-113.pooles.rima-tde.net) Quit ("http://www.vitaliano.esp.cc || Adi?s a todos y a todas y que les vaya bien.")
[18:26] <KenMan> I'M REALLY BACKED UP MAN, GET ME SOME LAXITIVE, NOW !!
[18:26] <toad_> globalQuota is the maximum number of requests we can accept in an hour and still have that number less than some arbitrary setting X
[18:27] <toad_> so in fact we just factor it into load
[18:27] <KenMan> can we react quickly enough to make a difference ? are RNFs due to excess load over time ?
[18:27] <toad_> hmm?
[18:27] <toad_> do we need to react quickly?
[18:27] <KenMan> we don't know unless we can see how often an RNF is the result of 100% backoff
[18:28] <KenMan> in milliseconds...
[18:28] <toad_> what does 100% backoff have to do with it?
[18:28] <toad_> we have a stack of live requests
[18:28] <KenMan> that may be everyone's major malfunction, 100% backoff occurring SOMEWHERE on the network
[18:28] <toad_> each one will (we hope) eventually get sent to a node
[18:28] <toad_> KenMan: why?
[18:28] <toad_> that node would just be ignored, right?
[18:29] <KenMan> i would hope so, never actually had the opportunity to verify it
[18:29] <toad_> if some node returns 2E27 MRI, we'll just drop it out of our RT, no problem, right?
[18:29] <KenMan> sounds good to me :)
[18:29] <toad_> so we get some requests...
[18:29] <KenMan> except when he computes 2E4 only minutes later... :(
[18:29] <toad_> our current globalQuota is X
[18:30] <toad_> eventually we start forwarding them
[18:30] <toad_> hmmm
[18:30] * Redb3ard (~oylerj@c-24-125-12-101.va.client2.attbi.com) has joined #freenet
[18:30] <KenMan> you would get better results to queue in FCP than in fred
[18:30] <toad_> now, is there any chance of this working on the CURRENT network?
[18:30] <toad_> KenMan: no!
[18:30] <KenMan> heh, good question
[18:30] <toad_> FNP is the issue here
[18:30] <KenMan> okay, fine.
[18:30] <toad_> somebody can always hack their node anyway
[18:30] <KenMan> sure
[18:30] <toad_> we can't protect them from themselves
[18:31] <toad_> now
[18:31] <toad_> can we build something based on this on the current network?
[18:31] <KenMan> okay, so we will queue requests, to smooth over the effects of changes in routes backoff in real-time.
[18:31] <toad_> KenMan: not exactly
[18:31] <toad_> if we queue requests, we can completely avoid RNFs
[18:31] <toad_> 99% of the time
[18:32] <KenMan> yes. That sounds like a practical goal, i agree with.
[18:32] <toad_> we would only actually RNF if either a) we had NOWHERE to send the request to or b) we got it wrong and it took more than the absolute limit
[18:32] <toad_> lets see...
[18:32] <KenMan> and we will measure the average amount of queue time needed to do this, so we can measure its impact .
[18:32] <toad_> if we assume 32kB chunks
[18:32] <toad_> then the typical request time would be ~ 30 seconds + ~ 30 seconds
[18:33] <toad_> i.e. about 1 minute
[18:33] <KenMan> when would we reach the condition "NOWHERE to send it to" ? only one peer ??
[18:34] <toad_> we would have globalQuota of approximately 10*60*60/32 = 1125 successful requests per hour
[18:34] <toad_> probably more like 10k real requests per hour
[18:34] <KenMan> yes, faster changing backoffs with lower MRIs
[18:34] <KenMan> which might provide better routing
[18:35] <toad_> if we assume that's distributed evenly over 50 active nodes...
[18:35] <toad_> we get 10k/50 = 200 reqs per hour per node
[18:35] <KenMan> 1800ms
[18:35] <toad_> = 18 seconds MRI
[18:35] <toad_> KenMan: 18000ms?
[18:35] <KenMan> 18000ms, right
[18:36] <toad_> so we can send a request every 18 seconds on each node
[18:36] <KenMan> on average, if we don't let it wiggle A LOT from 18000
[18:36] <toad_> on average, yes
[18:36] <toad_> this is all on average
[18:36] <KenMan> okay
[18:37] <toad_> now, would queueing be practical?
[18:37] <toad_> well, we are receiving 10,000 reqs per hour
[18:37] <toad_> we are sending slightly less
[18:38] <KenMan> less so in that environment. One would expect (when using averages to run these numbers) more benefit in the current situation.
[18:38] <toad_> that should result in reasonable queueing times - if we're not choosy
[18:38] <toad_> KenMan: well, the problem with the current situation is that we don't want to wait forever because of the queueing
[18:38] <KenMan> not choosy, unless you permit backoffs of 1800000ms...
[18:38] <toad_> however we can wait longer for larger files... hmmm
[18:38] <toad_> eh?
[18:38] <KenMan> that makes sense too
[18:39] <KenMan> sorry, im being picky ...
[18:39] <toad_> lets assume we're searching for a 1MB file
[18:39] <toad_> it will be transferred at maybe 3kB/sec
[18:39] <toad_> so it'll take 300 seconds to transfer
[18:39] <toad_> it's reasonable to wait a reasonable time in the queue...
[18:40] <toad_> I dunno, can it work with the current situation?
[18:40] * jay (jcl@ool-18bf6dac.dyn.optonline.net) has joined #freenet
[18:40] <KenMan> yes, that makes sense. Max queueing time related to keysize of request
[18:40] <toad_> well
[18:40] <toad_> maybe it won't fully work
[18:40] <toad_> but maybe it will be helpful anyway...
[18:40] <toad_> hmmm
[18:40] <KenMan> nothing ever does around here ;)
[18:41] <toad_> well
[18:41] <toad_> by fully work
[18:41] <KenMan> we strive for partial success, as it denotes progress... haha
[18:41] <toad_> I mean more or less eliminate RNFs
[18:41] <KenMan> it would be a neat experiment, but where and how to conduct it ?
[18:41] <toad_> and allow us to propagate load
[18:41] <toad_> what is the theory here?
[18:42] <toad_> if we queue, we can reduce the number of RNFs
[18:42] <toad_> that's about it, right?
[18:42] <KenMan> better synchronizing requests with backoffed routes popping up should reduce RNF. That sounds like the theory here
[18:42] <toad_> the problem with doing it now is that MRIs are huuuuuuge
[18:42] <KenMan> you are faced with reality :)
[18:43] <toad_> KenMan: well
[18:43] <KenMan> some are some are not
[18:43] <toad_> we can't synchronize requests with backed off routes
[18:43] <toad_> if we try to, we get more RNFs
[18:43] <toad_> that's the snag
[18:43] <toad_> well
[18:43] <toad_> we can
[18:43] <toad_> but if we are getting lots of RNFs it's pretty futile
[18:44] <KenMan> not assigning queries to routes necessarily, although that would reveal if we are favoring certain routes too heavily
[18:44] <toad_> hmm?
[18:44] <KenMan> as the queues for those heavy routes would grow, and we would see part of our problem
[18:44] <KenMan> i don't know... just typing too loud
[18:44] <toad_> KenMan: why did the last attempt to synchronize requests with backed off routes fail?
[18:44] <KenMan> i don't recall that one !!
[18:45] <toad_> logically it shouldn't have produced RNFs
[18:45] <toad_> wait a minute
[18:45] <toad_> it could have
[18:45] <toad_> look
[18:45] <toad_> :
[18:45] <toad_> without queueing
[18:45] <toad_> a request comes in
[18:45] <toad_> perhaps it can go out immediately
[18:45] <toad_> perhaps it can't
[18:45] <toad_> if it can't it RNFs and gets retried somewhere else
[18:46] <toad_> so the total number of requests going out will be less than the total number coming in
[18:46] <toad_> the result of that is RNFs
[18:46] <toad_> which hmmm
[18:46] <toad_> makes it go up again
[18:46] <toad_> hmmm
[18:46] <toad_> okay..
[18:46] <toad_> here's an obvious bit:
[18:47] <toad_> why does retrying on RNFs cause us to multiply queries?
[18:47] <toad_> logically, queries only have 20 htl
[18:47] <KenMan> it is a local issue, not netwide
[18:47] <toad_> if on average we get 2 instant RNFs before getting a good route, we should reduce it by an appropriate HTL
[18:47] <toad_> this means those requests expire earlier elsewhere
[18:47] <toad_> KenMan: hmmm
[18:47] <KenMan> our retrys consume available routes that other queryes deserve to get
[18:48] <KenMan> (leading to higher backoff, which leads to...)
[18:48] <toad_> KenMan: yeah but it ought to balance out?
[18:48] <KenMan> i wouldn't bet on it, is all i'm thinking
[18:49] <KenMan> examine the local effects more closely
[18:49] <toad_> we COULD forcibly prevent multiplication
[18:49] <toad_> by always failing all the way back on a QR
[18:49] <toad_> the problem is that that would make requests pretty fragile
[18:49] <KenMan> yes. that is an option you stated earlier. but if the net suddenly started breathing...
[18:49] <toad_> e.g. looped
[18:49] <toad_> if we fail when we run into a loop, what then?
[18:50] <toad_> we're bound to do it sooner or later
[18:50] <KenMan> retry , and watch how common that event is
[18:50] <toad_> and probably sooner
[18:50] <KenMan> or just ignore it. The overall experiment is the bigger interest.
[18:50] <toad_> KenMan: hmmm
[18:51] * moskau23 (~Miranda@dsl-213-023-251-068.arcor-ip.net) Quit (Read error: 104 (Connection reset by peer))
[18:51] <toad_> how can we make propagation work?
[18:51] <KenMan> huh? prop of what ?
[18:51] <toad_> is it reasonable to say that without queueing, propagation CANNOT work?
[18:51] <toad_> KenMan: propagation of rate limiting
[18:51] <toad_> our outgoing MRIs are a function of our incoming MRIs
[18:52] <toad_> so
[18:52] <KenMan> no, that is an unreasonable statement. We just have never solved it yet.
[18:52] <toad_> we calculate the total request rate we can send out
[18:52] <toad_> and we use that as our globalQuota?
[18:52] <KenMan> that is reasonable
[18:52] <toad_> now
[18:53] <toad_> if we get a query
[18:53] <toad_> and at that instant all the routes are backed off
[18:53] <toad_> we QueryReject it
[18:53] <KenMan> ouch
[18:53] <toad_> and our predecessor is forced to retry it elsewhere
[18:53] <toad_> see?
[18:53] <KenMan> that's a good analysis
[18:53] <KenMan> it is much in line with what I was trying to propose earlier
[18:53] <toad_> so without queuing, we necessarily RNF SOME requests
[18:54] <KenMan> due to distribution of backoffs in time :)
[18:54] <toad_> but WITH queueing, we can completely avoid RNFing, by synchronizing incoming requests with outgoing requests
[18:54] <KenMan> yes
[18:54] <toad_> okay
[18:54] <toad_> so we should implement queueing
[18:54] <toad_> and then look into propagation
[18:54] <toad_> try it out initially on testnets
[18:54] <toad_> and later on larger networks
[18:54] <KenMan> i guess so. The impact is obviously that some requests might take a little longer, but many more should succeed.
[18:55] <toad_> right
[18:55] <toad_> so how do we implement queueing acceptably?
[18:55] <KenMan> it is hard to simulate backoff sets of 50 peers on local testnets
[18:55] <toad_> is it the case that without propagation, queueing is infeasible?
[18:55] <toad_> yeah...
[18:55] <toad_> well
[18:55] <KenMan> maybe i could model it, but ... how accurate the model would be is questionable
[18:55] <toad_> how about:
[18:55] <toad_> initially, and with full intention of fixing this later:
[18:55] <toad_> we take out ALL the timeouts
[18:56] <toad_> a request is allowed to take forever to complete
[18:56] <KenMan> that's a nice easy experiment.
[18:56] <toad_> but we still queue
[18:56] <toad_> and we still propagate MRI
[18:56] <KenMan> that just got harder
[18:56] <KenMan> define propagate for me again ? allowing incoming MRIs to influence our outgoing ones ??
[18:57] <toad_> yup
[18:57] <toad_> try to ensure that we only accept exactly as many requests as we can route
[18:57] <KenMan> okay. Implementing the queueing is the bulkiest part of the work, no ?
[18:57] <KenMan> yes
[18:57] <toad_> yes
[18:57] <KenMan> well, damned near to it
[18:57] <toad_> but the problem with queueing
[18:57] <toad_> how do you set the timeouts?
[18:57] <KenMan> wait, in your model, yes, exact
[18:57] <KenMan> timeouts where ?
[18:57] <toad_> well, the target is exactly equal
[18:57] <toad_> you never achieve that
[18:57] <KenMan> oh, for transfers ? no, what do you mean ?
[18:57] <toad_> KenMan: no
[18:58] <toad_> if I want to implement queueing today
[18:58] <toad_> to work on today's network
[18:58] <toad_> for best testing opportunities
[18:58] <toad_> then how would i set up timeouts?
[18:58] <|ux> biab
[18:58] <KenMan> which timeouts ???
[18:58] <|ux> (not that im actually working on anything with u guys) :p
[18:58] <toad_> how long do we allow requests to be queued before we RNF?
[18:58] <KenMan> see you soon, |ux
[18:59] <KenMan> ah, the time spent by less than 95% of queued requests
[18:59] <KenMan> something like that
[18:59] <KenMan> although the trimming messes up your search for a formula
[18:59] <KenMan> 3x the average wait time ?
[18:59] <KenMan> lord, thats a hard one
[19:00] <toad_> technically, right now, we could keep sending back QueryRestarted's indefinitely
[19:00] <KenMan> hahaha
[19:00] <KenMan> it could be attempt number one, i suppose
[19:00] <toad_> one major problem here:
[19:00] <toad_> this is PER HOP !!
[19:00] * theLostFloppy (~michaelku@fia41-111.dsl.hccnet.nl) Quit ()
[19:00] <KenMan> yeah, ouch
[19:00] <KenMan> twice the average wait time.
[19:00] <KenMan> done.
[19:00] <toad_> huh?
[19:01] <KenMan> oh, how do you count them... hmmm
[19:01] <toad_> twice the average successful wait time?
[19:01] <toad_> hmmm
[19:01] <KenMan> well, 90% of queued requests will take Xms +/- Yms
[19:01] <toad_> we'd need some sort of cap, right?
[19:01] <KenMan> yeah, i suppose you could work the numbers to CONTROL std dev of wait times, no ?
[19:01] <toad_> how about 30s per hop?
[19:01] <KenMan> i mean, that's what the cap does
[19:01] <toad_> just fix it at that?
[19:02] <toad_> at least for 1MB files?
[19:02] <KenMan> that's kinda small, but give it a try, we can adjust later.
[19:02] <toad_> well
[19:02] <toad_> if we assume a 10 hop return path
[19:02] <KenMan> so long as you force most MRIs to be NEAR 18000ms you worked out above
[19:02] <toad_> then 30 seconds per hop is still THREE HUNDRED SECONDS
[19:02] <toad_> which is 5 minutes
[19:02] <toad_> which is an awfully long time
[19:02] <KenMan> this stuff also assumes all routes are used equally
[19:03] <toad_> hmmm
[19:03] <KenMan> start off with a small queue time, and see how that improves things... (?)
[19:03] <toad_> possibly
[19:03] <KenMan> this , incidentally, is the same thing I barely began to describe with my weird MRI proposal
[19:03] <toad_> if we don't care about routing, and we're just trying to reduce RNFs, then that does rather simplify things
[19:04] <toad_> 6 seconds per hop
[19:04] <toad_> if the return path is 10 hops that makes 1 minute
[19:04] <toad_> even if it's 20 hops, it'd only be 2 minutes
[19:04] <toad_> try that arbitrary figure
[19:04] <KenMan> how do you queue them ? on what criteria ? that A route becomes available ? that the RIGHT route opens up ? this is getting confusing for me.
[19:05] <toad_> that ANY route becomes available
[19:05] <KenMan> you sound like you've got it worked out, describe the queueing just a little bit
[19:05] <KenMan> excellent !!!
[19:05] <toad_> our initial goal with queueing is to reduce the number of RNFs
[19:05] <toad_> nothing more, nothing less
[19:05] <KenMan> me likey likey
[19:05] <KenMan> :)
[19:05] <toad_> we get a request
[19:06] <toad_> we add it to the queue
[19:06] <KenMan> you will never grow beyond 10 seconds queue time in that case
[19:06] <toad_> any requests that have been on the queue more than N seconds, say 6, are RNFed
[19:06] <KenMan> if things explode on a node, that node is very poorly configured, like 2000 maxPeers for a dialup modem - heh
[19:06] <toad_> when a node becomes available for sending a request to, we send the best request for it on the queue
[19:06] <toad_> to that node
[19:07] <KenMan> better
[19:07] <KenMan> i mean, that is a better improvement
[19:07] <toad_> the problem with this is that routing has virtually no impact on where they go
[19:07] <toad_> i suppose that's no different to what we have now
[19:07] <toad_> but we can perhaps improve on it
[19:07] <KenMan> and drop/RNF those that don't get matched up within 6 seconds...
[19:07] <KenMan> heh, good point
[19:07] <toad_> request comes in
[19:07] <toad_> we sort the RT according to suitability for that request
[19:08] <KenMan> high level, it is forcing NGR to spread requests evenly.
[19:08] <KenMan> which is demanded by workable rate limiting.
[19:08] <toad_> we decide "we would like to be routed to a node in the top half, in terms of request capacity, of this list"
[19:08] <KenMan> if we can get this to work, then we could set a limit that says NGR can't favor a node by more than twice the worst one.
[19:08] <toad_> nah can't do that
[19:08] <toad_> hmmm
[19:09] <KenMan> just go with what you have, "an effort to reduce RNFs"
[19:09] <toad_> KenMan: the problem is that it causes random routing!
[19:09] <KenMan> RNF those queries we are worst suited to
[19:09] <KenMan> i see that... but it is not totally random. Consider the case where 2 or more routes ARE available
[19:09] <toad_> KenMan: good idea, implementation?
[19:10] <KenMan> i hope you didn't just talk yourself out of this completely :)
[19:10] <toad_> I don't know hrrm
[19:10] <KenMan> in some instances, NO routes are available. In some instances, more than one route is available. This is the distribution of route backoff (driven by MRIs) over time.
[19:11] <toad_> yeah...
[19:11] <KenMan> given the current model of backoff. Do you see any relation to my proposal, even though I didn't solve the problem ?
[19:11] <toad_> hmmm
[19:11] <toad_> well the difficulty is
[19:11] <toad_> is it ever sensible to wait for a good node?
[19:12] <toad_> rather than routing more quickly to a mediocre node?
[19:12] <KenMan> not if we can maintain a certain percentage of available routes
[19:12] <KenMan> the question is whether we can. We can't do ins=outs given the multiplication factor
[19:12] <toad_> obviously when the top node becomes available, we should immediately send it the best suitable request for it
[19:12] <KenMan> yes
[19:13] <toad_> but what about when the bottom node becomes available?
[19:13] <toad_> also if we CAN keep a % of the RT free
[19:13] <KenMan> maintain or average a certain percentage of available routes. Now you are balancing routing choices against backoff (which is good in this context)
[19:13] <toad_> right
[19:13] <toad_> well
[19:13] <toad_> if we can keep a % of the RT free
[19:13] <toad_> then we get a request
[19:13] <toad_> what do we do with it?
[19:14] <toad_> do we wait to be sent to the best node available, or do we send immediately to the less good node?
[19:14] <KenMan> push it out the best door, nothing more
[19:14] <KenMan> ah, it could be balanced
[19:14] <toad_> well that won't be sustainable
[19:14] <KenMan> queue or route, based on rank
[19:14] <KenMan> but you know the queue will build on the best route, while the worst goes wanting
[19:15] <KenMan> incidentally, i really like your frame of thought (focus) right now
[19:15] <toad_> we see "we could send it to route #3, in 4 seconds"
[19:15] <toad_> the problem is that there are other requests that could also be sent to that route
[19:15] <toad_> brb
[19:17] <toad_> even if it's dynamic
[19:17] <toad_> i.e. we have as many requests as we can route and no less
[19:17] <toad_> we still need to decide: will we favour good nodes somehow?
[19:17] <toad_> make sure that good nodes get good requests?
[19:18] <toad_> if we assume that time is not an immediate concern, to simplify matters
[19:18] <toad_> we have a bunch of requests
[19:18] <toad_> a node reports in, "I want another request"
[19:19] <toad_> if we're not saturated, do we take a request?
[19:19] <toad_> if we are, which one do we take?
[19:20] <toad_> if the idea is to make the best of our current request rate...
[19:20] <toad_> then each node has an ordered list of the requests it would like to have
[19:21] <toad_> for the #1 node, it's easy: when it becomes available, send the best request
[19:21] <toad_> (i'm oversimplifying here as hopefully there will be no universal #1 node)
[19:21] <|ux> isnt that lolo at the moment :p
[19:22] <toad_> for the #2 node, send the best request for #2, IF that won't go to a better node soon
[19:22] <toad_> so for node #399 of 400, we send the best request for this node that won't soon go to another node
[19:23] <toad_> KenMan: I dunno
[19:23] <KenMan> here is a simple exercise based in reality - decide which measurements to watch, collect them, then reduce the magic value of 3 retries on RNFto 2
[19:23] <toad_> KenMan: I don't see HOW we can implement queueing
[19:23] <KenMan> something should get better, and something should get worse
[19:24] <KenMan> i know. i have to go now, but I like where I might have steered your thoughts
[19:24] <toad_> I suppose it could depend on the age of the request
[19:24] <toad_> two opposing forces:
[19:24] <toad_> 1. Nodes want the best request for this node.
[19:24] <toad_> 2. Requests are picky about nodes.
[19:24] <KenMan> inMRIs determine backoff (% and makeup) at any instant in time, RNF multiplication, etc.
[19:24] <toad_> depending on their age
[19:25] <KenMan> that's a good balance
[19:25] <toad_> okay, so how do we balance them in practice?
[19:25] <KenMan> time enqued determines affinity for "best node"
[19:25] <toad_> well, 1. also includes nodes want a request, any request
[19:26] <KenMan> some inevitably get 'dropped' while some get better routed
[19:26] <toad_> if the drop time is 10 seconds, for example
[19:26] <KenMan> dropped = poorly routed, RNF, whatever
[19:26] <toad_> then when we are 9 seconds old we'll accept absolutely any node
[19:26] <toad_> whereas when we are 0 seconds old we'll accept only the best
[19:26] <KenMan> 10 seconds should increase your routing choices quite a bit, until you realize how many are competing...
[19:26] <toad_> and we have a sliding scale
[19:27] <KenMan> which is a function of how many we let through the door
[19:27] <toad_> based on our rank, and biased by the capacity
[19:27] <KenMan> yes
[19:27] <toad_> so if the top node only has 1% of the request capacity, we only give it 1% of our age
[19:27] <KenMan> that appears to show promise
[19:27] <toad_> so make the drop time 10 seconds
[19:28] <KenMan> not a bad compromise
[19:28] <toad_> after say 7 seconds, we don't care who we go to
[19:28] <toad_> we go to anyone
[19:28] <mazzanet> bring bring
[19:28] <toad_> at 0 seconds we only go to the top choice
[19:28] <toad_> sliding scale as above
[19:28] <toad_> hrrm
[19:28] <toad_> must think through ... is this implementible?
[19:28] <KenMan> but, the time you are considering will change with number of peers... or will it ? using your average math ?
[19:28] <toad_> anyway
[19:28] <toad_> must go bed too
[19:28] * Hory (~Miranda@81.196.25.110) Quit ("Miranda IM! Smaller, Faster, Easier. http://miranda-im.org")
[19:28] <KenMan> cool, chat later...
[19:28] <toad_> see you monday
[19:29] <toad_> bbl
[19:29] * toad_ (toad@82-32-16-91.cable.ubr03.azte.blueyonder.co.uk) Quit (Remote closed the connection)
[19:32] * toad_ (toad@82-32-16-91.cable.ubr03.azte.blueyonder.co.uk) has joined #freenet
[19:32] <toad_> KenMan: if you're still here...
[19:32] <toad_> we could base the decisions on the lowest normalized estimate across all nodes for all requests
[19:32] <toad_> so we have 50 requests waiting
[19:33] <toad_> we have 40 nodes waiting for a request
[19:33] <toad_> we build the matrix
[19:33] <toad_> and order it by normalized estimate
[19:34] <toad_> we work down it - find the request:node pair with the best time
[19:34] <toad_> and send it, if it's willing to be sent to that node
[19:34] <toad_> and then go down it until the reqs are no longer willing
[19:34] * bluephile (bluephile@69-160-205-193.clvdoh.adelphia.net) has joined #freenet
[19:35] <toad_> anyway bbl zzz
[19:35] * toad_ (toad@82-32-16-91.cable.ubr03.azte.blueyonder.co.uk) Quit (Client Quit)
[19:40] * Pascal (Pascal@c-24-13-17-191.client.comcast.net) has joined #Freenet
[19:40] <Pascal> howdy
[19:40] <|ux> hey
[19:41] <|ux> why would "Frost" keep suddenly "trying"... then "waiting"...
[19:41] <Pascal> amazing how well 5090 is working
[19:42] <|ux> anyone?
[19:50] * Sweetshark (~bjoern@b118147.adsl.hansenet.de) has joined #freenet
[20:02] * xChaplin (~blah@CPE000c419e6b8d-CM014460007148.cpe.net.cable.rogers.com) has joined #freenet
[20:04] <xChaplin> I keep getting "Couldn't connect to the network. Are you sure you have configured Freenet correctly? Also make sure that you are connected to the internet." How long does the node need to be up to connect properly?
[20:05] <|ux> An day
[20:05] <|ux> sometimes
[20:05] <|ux> xChaplin: check the "Open Connections" screen
[20:06] <|ux> "Connections Mode"
[20:06] <|ux> and tell me the stats there
[20:06] <|ux> for "Receiving transmitting"
[20:06] <|ux> http://1927.0.0.1:8888/servlet/nodeinfo/networking/ocm
[20:06] <xChaplin> I got 0/0
[20:06] <|ux> Connections open (Inbound/Outbound/Limit) 126 (71/55/200)
[20:06] <|ux> Transfers active (Transmitting/Receiving) 36 (18/18)
[20:06] <|ux> Data waiting to be transferred 148 Bytes
[20:06] <|ux> Total amount of data transferred 218 MiB
[20:06] <Sweetshark> im new to freenet too but it seems to take some time - esp. since there had been a kinda routing reset recently because of a version upgrade of the network ...
[20:06] <|ux> Connections Open?
[20:06] <xChaplin> Connections open (Inbound/Outbound/Limit) 0 (0/0/200)
[20:06] <xChaplin> Transfers active (Transmitting/Receiving) 0 (0/0)
[20:06] <xChaplin> Data waiting to be transferred None
[20:06] <xChaplin> Total amount of data transferred 5,724 Bytes
[20:07] <|ux> Ummm.. have you opened the port?
[20:07] <|ux> through a firewall/router/nat
[20:07] <xChaplin> I did, I've been running a node for some time now. I just upgraded to the latest
[20:07] <xChaplin> Did the default port change?
[20:08] <|ux> It shouldnt do
[20:08] <|ux> It should be in your config
[20:08] <|ux> freenet.conf
[20:08] <|ux> What version was you running?
[20:08] <xChaplin> I was running 5085. Now upgraded to 5090
[20:08] <|ux> Ok
[20:09] <|ux> How did you upgrade?
[20:09] * |ux wishes toad_ or kenman was here
[20:09] <xChaplin> uninstalled but kept the store. re-installed with web-install
[20:09] <|ux> Ok
[20:10] <|ux> remove the store, thats what i did anyways
[20:10] <|ux> lol
[20:10] <|ux> complete re-install worked for me
[20:10] <|ux> when it doubt... hehe
[20:10] <|ux> 5090 is a lot better than previous versions
[20:10] <|ux> quite a few things have been fixed
[20:10] <Sweetshark> is it normal to have freenet running 160 processes and eating huge amounts of RAM?
[20:10] <|ux> Sweetshark: Yes.. what OS?
[20:10] <xChaplin> right, I'll uninstall everything
[20:10] <|ux> xChaplin: best of luck :)
[20:10] <Sweetshark> |ux,linux
[20:11] <|ux> Sweeshark: what kernel?
[20:11] <Sweetshark> Linux lord.sinclair 2.6.6 #1 Sat Jun 5 13:02:09 CEST 2004 i686 AMD Athlon(tm) XP 2400+ AuthenticAMD GNU/Linux
[20:11] <|ux> Ok
[20:11] <|ux> how much memory do u have installed?
[20:11] <Sweetshark> 512MB
[20:11] <|ux> ok
[20:11] <|ux> are you using Gentoo
[20:11] <|ux> or another distro?
[20:12] <Sweetshark> gentoo
[20:12] <|ux> /etc/conf.d/freenet
[20:12] <|ux> should have "JAVA_ARGS" in
[20:12] <|ux> with memory size
[20:12] <|ux> is it 256?
[20:12] <Sweetshark> it is.
[20:13] <|ux> And it is using up far more than that?
[20:13] <|ux> ok... one way to find out for sure...
[20:13] <|ux> Sweetshark: go to http://127.0.0.1:8888/servlet/nodeinfo/internal/env
[20:13] <|ux> look for Memory Allocation
[20:13] <|ux> it should contain something like:
[20:13] <|ux> Maximum memory the JVM will allocate 388,864 KiB
[20:13] <|ux> Memory currently allocated by the JVM 138,596 KiB
[20:13] <|ux> Memory in use 64,420,904 Bytes
[20:13] <|ux> Estimated memory used by logger None
[20:13] <|ux> Unused allocated memory 77,499,504 Bytes
[20:13] <|ux>
[20:14] <Sweetshark> looks pretty good ...
[20:14] <Sweetshark>
[20:14] <Sweetshark> Maximum memory the JVM will allocate 320 MiB
[20:14] <Sweetshark> Memory currently allocated by the JVM 185.712 KiB
[20:14] <Sweetshark> Memory in use 104.260.240 Bytes
[20:14] <Sweetshark> Estimated memory used by logger None
[20:14] <Sweetshark> Unused allocated memory 85.906.832 Bytes
[20:15] <|ux> so its using
[20:15] <|ux> really.. 185Mb
[20:15] <|ux> Sweetshark: Are you running build 5090?
[20:15] <Sweetshark> yes.
[20:16] <|ux> 160 processes lol
[20:16] <|ux> umm
[20:16] <Sweetshark> just wasnt prepared for this kind of resource usage.
[20:16] <|ux> are you running FROST?
[20:18] <Sweetshark> FROST? dunno? but i got pretty triggerhappy in firefox opening all kinds of windows in new tabs - and this node is up for barely an hour ..
[20:18] <Sweetshark> s/windows/links/
[20:18] <|ux> ahhh
[20:19] <|ux> ok
[20:19] <|ux> slow down there boy
[20:19] <|ux> :p
[20:19] <Sweetshark> woha: Load on the webinterface says 105% ...
[20:19] <|ux> what i do is.. run freenet on my laptop.. and access it from my computer
[20:19] <|ux> Sweetshark: yea that is okay
[20:19] <|ux> Sweetshark: It will "oscillate" above and below the mark
[20:19] <Sweetshark> k
[20:19] * |ux feels like he knows something
[20:20] <|ux> Do you have Icons up yet?
[20:20] * Sweetshark is proud of |ux
[20:20] <|ux> lol dont be
[20:20] <Sweetshark> some
[20:20] <|ux> also... around this time all the boarsd "roll over"
[20:20] <|ux> so u may have icons.. then 10 mins later (about 1am British Summer time)
[20:20] <|ux> they will all have gone
[20:20] <|ux> .. its cos its hte next day, and the routes have to be reconstructed or something...
[20:20] <|ux> its really over my head
[20:21] <|ux> Remember... to ask toad_
[20:21] <|ux> or lolo (lostlogicx) or kenman
[20:21] <|ux> or verl
[20:21] <|ux> they are the people who really know whats going on
[20:21] * |ux is the guy who usually bugs them to death :p
[20:21] <Sweetshark> none right now - I guess i overloaded freenet a bit here ...
[20:21] <|ux> lol naaa
[20:22] <|ux> just ur node
[20:22] <|ux> 160 processes lol
[20:22] <|ux> u really went window crazy :p
[20:23] * xChaplin (~blah@CPE000c419e6b8d-CM014460007148.cpe.net.cable.rogers.com) Quit ()
[20:24] <Sweetshark> the first time i started freenet it was _really raising my CPU temp - it is pretty warm here (northern germany) and my CPU was already running at 56?C - it rose to 67?C and then i thought i might better stop freenet for a while. Now it is running just below 60?C.
[20:24] <|ux> my cpu has got to like 78oC b4
[20:24] <|ux> lol
[20:24] <|ux> damn summer
[20:24] <|ux> and bad heat conduction
[20:25] <Sweetshark> this machine should be used to this load - tvtime and gentoo compiles arent easy on the CPU either ..
[20:25] <|ux> nope
[20:25] <|ux> but heh
[20:25] <|ux> 160 java threads
[20:25] <|ux> is a lot
[20:25] <|ux> :p
[20:25] <Sweetshark> i have ntpl - but no p4 - that would help here i guess ...
[20:26] <Sweetshark> ht and stuff
[20:26] <|ux> yea maybe
[20:26] <|ux> 2 computers would be better :p
[20:26] <|ux> u *could* try setting it to a lower priority
[20:26] <|ux> /etc/conf.d/freenet
[20:26] <|ux> change the nice value
[20:26] <Sweetshark> just image a beowulf cluster of these ..
[20:27] <|ux> well
[20:27] <|ux> without the bandwidth
[20:27] <|ux> to support transfers
[20:27] <|ux> it would not do much
[20:27] <Sweetshark> oh system is still resonding fine - the 2.6 scheduler rules ...
[20:27] * |ux likes pre-emptible kernel :p
[20:28] <jay> can someone verify that '41156d00' is today's DBR value?
[20:28] <|ux> how?
[20:28] <jay> it's Aug 8 UTC
[20:28] <|ux> my "icons" went
[20:28] <|ux> and are back now
[20:28] * bluephile (bluephile@69-160-205-193.clvdoh.adelphia.net) Quit (Client Quit)
[20:29] <|ux> jay: how do i find out that?
[20:29] * bluephile (bluephile@69-160-205-193.clvdoh.adelphia.net) has joined #freenet
[20:29] <jay> well now's the time not to try actually
[20:29] <jay> just turned over
[20:30] <jay> unless ppl have re-inserted their DBR sites within that last 48 minutes
[20:30] <|ux> lol
[20:30] <|ux> where did u get that number?
[20:30] <jay> 48?
[20:31] <jay> it's 0:48 UTC
[20:32] <Sweetshark> load average: 2.82, 3.67, 3.88, Tasks: 222 total, 2 running, 220 sleeping, 0 stopped, 0 zombie, im using 40k bandwidth on ppp0 according to gkrellm2 - system responding fine and stable - try this with windows ...
[20:32] <|ux> '41156d00'
[20:32] <|ux> that one
[20:32] <|ux> sorry hex
[20:32] <|ux> Sweetshark: 220 sleeping?
[20:33] <jay> it's the number of days since Jan 1 1970 in Hex
[20:33] <jay> calculated by some C code i just whipped up
[20:33] <|ux> cool
[20:34] <Sweetshark> |ux, all the freenets waiting for io ..
[20:34] <|ux> Sweetshark: ok
[20:34] <jay> |ux: it's how DBR's work internally within Freenet
[20:35] <|ux> jay: ok :)
[20:35] <|ux> Sweetshark: what was ur question again.... maybe jay can tell me how wrong i was in what i said
[20:38] <Sweetshark> im just using huge amounts of cpu and RAM and threads - but im also just filling my disc with data from freenode ... im not at 166 java threads ...
[20:38] <Sweetshark> s/not/now/
[20:39] <|ux> Sweetshark: you will only fill it up to the amount of StorageSize specified
[20:40] <Sweetshark> i already got 33%
[20:41] <|ux> Sweetshark: what storage size u set?
[20:41] <|ux> Sweetshark: and that doesnt mean anything anyway.. it will not affect the speed i dont think
[20:42] <Sweetshark> Maximum size 256 MiB
[20:42] <Sweetshark> Used space 105.300 KiB
[20:42] <Sweetshark> Free space 156.844 KiB
[20:42] <Sweetshark> Percent used 40
[20:43] <Sweetshark> Pooled threads in use 110 <--- this number is faliing - good sign, i guess
[20:44] <|ux> Sweetshark I have a storage size of 2GB.. and its full
[20:44] <|ux> lol
[20:44] <Redb3ard> so whats up guys?
[20:45] <|ux> nm
[20:45] <|ux> hows ur BONE doing :p
[20:46] <Redb3ard> ?
[20:46] <|ux> lol
[20:46] <|ux> u know that network thingy
[20:46] <Redb3ard> hehe
[20:46] <Redb3ard> not bad
[20:46] <Redb3ard> its simmering more than its boiling
[20:46] <|ux> give the dog.com is it? :P [a bone]
[20:49] <Sweetshark> Current estimated load for rate limiting 200,7% [Rejecting incoming connections and requests!] <--- i guess that explains why i get nothing right now. Hmmm, patience is a virtue ...
[20:50] <|ux> Sweetshark: It happens.. go to General Information
[20:50] <|ux> Sweetshark: What is the highest percentage in there?
[20:51] <Sweetshark> 235,3% = 1176ms / 500ms > overloadLow (100%)
[20:52] <|ux> before the percentage
[20:52] <Redb3ard> heh
[20:52] <|ux> what does it say?
[20:53] <|ux> before 235.3%
[20:53] <Sweetshark> Load due to messageSendTimeRequest =
[20:53] <|ux> thought so...
[20:53] <|ux> lol
[20:53] <|ux> all the connections
[20:53] <|ux> go to the Open Connections page
[20:53] <|ux> what is ur connections?
[20:53] <|ux> Connections open (Inbound/Outbound/Limit) 130 (72/58/200)
[20:53] <|ux> Transfers active (Transmitting/Receiving) 71 (8/63)
[20:54] <Sweetshark> Connections open (Inbound/Outbound/Limit) 52 (1/51/200)
[20:54] <Sweetshark> Transfers active (Transmitting/Receiving) 77 (28/49)
[20:54] <|ux> Sweetshark: What connection are you on?
[20:55] <Sweetshark> ADSL, 3MBit down, 386 kBit up
[20:55] <|ux> Sweetshark: git! i mean er.. nice.. what "limits" have u set for Freenet?
[20:57] <Sweetshark> outputBandwidthLimit=24000, %inputBandwidthLimit=0 (none) <-- is this an "Oooops"?
[20:57] <|ux> no thats fine i think
[20:57] <|ux> yea
[20:57] <|ux> thats fine
[20:58] <jay> 0 is not set i believe
[20:58] <|ux> they are on sepereate lines like
[20:58] <jay> 0 means, "not set"
[20:58] <|ux> yea
[20:58] <|ux> and its also % commented out
[20:58] <Sweetshark> especially if commented out (%) ...;-)
[20:59] <Ash-Fox> -1
[20:59] <|ux> ?
[20:59] <Ash-Fox> !
[20:59] <|ux> Ash-Fox: wha?
[21:00] <Redb3ard> damn it
[21:00] <Redb3ard> am i the only american here?
[21:00] <|ux> Redb3ard: ?
[21:00] <|ux> Redb3ard: possibly
[21:00] <Ash-Fox> Redb3ard, nope, the bush administration is watching you
[21:00] <|ux> lol
[21:01] <Redb3ard> haha
[21:01] <Sweetshark> Redb3ard, ill visit the US soon - 26. of August ...
[21:03] <|ux> glad u didnt say 11 sept..
[21:03] <|ux> anyway im off 2 bed
[21:03] <|ux> nite all
[21:03] * |ux (~|ux@82-37-17-24.cable.ubr03.wolv.blueyonder.co.uk) Quit ("Leaving")
[21:05] <Redb3ard> well, if anyone ever wants to try something other than freenet, just ask
[21:08] <jay> other than freenet?
[21:09] <Redb3ard> yeh, im working on something a little different
[21:09] <Redb3ard> an IPv4 anonymous network
[21:14] <jay> bbl
[21:14] * jay (jcl@ool-18bf6dac.dyn.optonline.net) Quit (".")
[21:39] * Sweetshark (~bjoern@b118147.adsl.hansenet.de) has left #freenet
[21:44] * ornge (~ornge@h131n2fls34o879.telia.com) Quit ("Leaving")
[21:48] <bluephile> Redb3ard: you know about iip?
[21:49] <bluephile> erm, I mean i2p
[21:51] <Redb3ard> yeh, familiar with it
[21:58] * Zorix (Brandon@fl-65-41-1-233.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[22:01] * Pascal (Pascal@c-24-13-17-191.client.comcast.net) Quit (Read error: 110 (Connection timed out))
[22:17] * KenMan (~chaziller@pcp403292pcs.mntcrm01.md.comcast.net) Quit (Remote closed the connection)
[22:51] * salahx (salahx@sc1-24.217.174.147.charter-stl.com) has joined #freenet
[23:15] * Zorix (Brandon@fl-65-41-1-233.dyn.sprint-hsd.net) has joined #freenet
[23:43] * TheSeeker (~Fridlekh@4.27.134.106) Quit (Ping timeout: 14400 seconds)
These logs were automatically created by Jay Oliveri with his gimp hapi on irc.freenode.net.
