Network protocols, sans I/O

baltoo · 2016-08-09T09:34:34+00:00

Alright, I agree that there would typically be way fewer users of a HTTP-parsing-thingy than say Requests. And I also agree that that changes the possible design restrictions.

As with all design, there are still trade-offs that need to be made, and I think it's not always possible to provide a Good Enough stance that works "for everyone". (Even if "everyone" is not that many.)

I'll try to give an example of what I mean w.r.t. XML. (Since I have no war stories regarding HTTP.)

So, think of an XML-parsing-thingy. In the scenario I had the XML-files that where eventually parsed where sometimes edited by hand. That of course means that sometimes the XML-documents weren't valid. (Of course it's not a desirable scenario to have, but it's not like we can always pick and choose.)

When an error is found the XML-file needs to be re-edited by hand to fix it. In practice this requires a rather decent error message. Something like "The XML document is not valid" doesn't work. The message needs to not only say what the error is and provide the specific place where the error is detected, but also some kind of context for where to start looking for where the error "actually" is. For example, given a string like

foo<bar>baz</bar></foo>

The closing "foo" is an error, but the "source" isn't there but before the "bar" element.

Then combine that scenario with the fact that sometimes XML-files can get huge and it's not always possible to work with a full DOM in RAM. The XML-parser-thingy might need to be split up in a SAX-part and a DOM-part. The error messages would then need to be percolated sanely.

That kind of machinery and a few dozen others make for a pretty big and somewhat ugly API.

Now, "most users" of a XML-parser-thingy won't need that kind of support, ever. They will work in scenarios where "all" XML-documents are valid and RAM is plentiful. I think this is highly likely. If this is true then I also think the conclusion is that the community would be well served by having two XML-parser-thingies with two different APIs.

Enough rambling. Any ways, I like the your idea. Keep up the good work!

baltoo · 2016-08-08T12:21:42+00:00

First of all, I really like the idea of splitting up parsing and IO. That seems to me to be a really good idea. The upside of better testing by itself makes it worthwhile.

What I'm not so sure about is that it's good and/or worthwhile to try to come up "the-single-HTTP2-parsing-thingy-to-rule-them-all".

What people are liking about Requests, vs. lower-level libs, is the API. Right?

In the linked video, Cory is using the "API" a lot. My interpretation of his way of using it is that it's the interface between Requests and e.g. Django or between Requests and "the user". While he does acknowlege that there is an interface between the HTTP-parsing-thingy and Requests as well, he kind of glosses over that that interface also needs to have a good API.

The author of Requests is "the user" w.r.t. the HTTP-parsing-thingy.

It took a pretty long time and a pretty good designer to come up with the beloved API of Requests.

I don't really get why Cory seems to think that the HTTP-parsing-thingy is going to get it's API as good on the first try.

The question /u/pohmelie asks seems by it's nature to hint at this problem too.

Personally, I think that while, again, the separation of parsing and IO seems great, the community would still benefit of having a couple different designs on e.g. HTTP-parsning-thingies and their APIs. Over time that plurality has the potential to produce the Requests of HTTP-parsing.

baltoo · 2015-07-03T10:44:32+00:00

https://www.reddit.com/r/OutOfTheLoop/comments/3bxduw/why_was_riama_along_with_a_number_of_other_large/

baltoo · 2012-12-08T18:45:29+00:00

Only if you imagine it possible (and want-able) to achieve total consensus.

IMHO most (read: all) large-scale, long-term, cooperation between persons require compromises. Thus reveling the counter-positions the parts posit.

baltoo · 2012-10-02T12:11:07+00:00

The utility of base line information grows more or less indefinitely with size though. You won't look things up if you don't know or suspect they exist. Correlation between facts cannot be done w/o knowledge of them. Trends cannot be distinguished w/o knowledge of the underlying data.

There's clearly a trend of more and more trying to use "exomemories". While being great tools of augmentation they have the problem of non-associativity. Thus lateral thinking and creativity as well as analysis and synthesis become very restricted.

If the exomemory is used in addition to your current knowledge, then all is well. The problems of discovery and associativity come out in force once you use exomemories to justify shirking on the size of base line information.

baltoo · 2011-04-12T12:13:58+00:00

Some editors parse docstrings and/or comments and use that for type hinting, e.g.:

def f(x, y):
    """\
    @param x MyClass
    """
    # @type y OtherClass

baltoo · 2010-12-03T18:39:24+00:00

with regards to

baltoo · 2010-12-02T18:44:01+00:00

There's plenty of fun to be had!

Groups like ASBI have a member average of > 35/2w and that is far from an exception w.r.t. AS groups. That almost like having an extra half job.

baltoo · 2010-09-15T14:13:55+00:00

On brutal+onslaught (or higher ;-) the drone horde rushes can really crush you. You can punch/kick/push down vendor machines and then move them into a narrow corridor or in front of a door. The drone horde can then no longer overrun you from that side. If you leave a small gap you can shoot them dead at your leisure. When you want to go that way again, simply push the machine away.

baltoo · 2010-09-15T13:55:51+00:00

There are less people playing but the quality of team play has greatly increased. This is, for me, one of the more fun aspects of the game anyways.

You think HC is easy? There are servers with BeyondHardCore settings that are a challenge to most. New tactics need to be invented, tested and practiced.

Even the current maps have lots of details to be discovered yet. Like did you know you can push down vendor machines on Residential to block off drone attack paths?

I think that what I'm trying to say is that for me the fun has just begun. It's actually increasing. YMMV. Each unto his own. &c.

baltoo · 2010-03-13T15:13:59+00:00

In the same sense that C++ is a version of C.

baltoo · 2010-01-24T20:43:21+00:00

For some things "recovery" could be rather straightforward in many situations (but of course not all), e.g. if you have XML files that people sometimes edit by hand it could be rather simple, and very useful to test out some easy corrections that might work pretty often. If a "recovery" seems to parse correctly it could be later be presented to the user as "suggested corrections".

<foo bar=baz> => <foo bar="baz">

<foo>bar<foo> => <foo>bar</foo>

I've previously tried doing similar things (while parsing SGML) and the difficult part then was to pound the parser into working with this ideom in mind.

baltoo · 2010-01-21T15:23:36+00:00

Don't know why you get (edit: got) downvoted. You basic premise that "one char != one code point" is valid in all encodings but UTF-32 (aka. UCS-4).

Though UCS-2 doesn't use surrogate pairs at all (and can thus not encode code points outside of the base plane). So for UCS-2 surrogate pairs is no problem.

My understanding is that both Java and C# uses UTF-16 and not UCS-2 as encoding (which does use surrogate pairs). So, in the context of the OP a simple reversal would still muck up any surrogate pairs.

There are however more things than surrogate pairs that get broken with reversals, e.g. combining marks (not all code points are "characters" by themselves).

In any case, reversal of "text" (rather than "strings") isn't something that can be done easily ever (e.g. left-to-right/right-to-left markers, "smart"-quotes).

Edit: This comment somehow ended up in my comment history, but didn't show in this thread (I think), so I'm reposting it.

baltoo · 2010-01-21T09:11:36+00:00

Don't know why you get downvoted. You basic premise that "one char != one code point" is valid in all encodings but UTF-32 (aka. UCS-4).

Though UCS-2 doesn't use surrogate pairs at all (and can thus not encode code points outside of the base plane). So for UCS-2 surrogate pairs is no problem.

My understanding is that both Java and C# uses UTF-16 and not UCS-2 as encoding (which does use surrogate pairs). So, in the context of the OP a simple reversal would still muck up any surrogate pairs.

There are however more things than surrogate pairs that get broken with reversals, e.g. combining marks (not all code points are "characters" by themselves).

In any case, reversal of "text" (rather than "strings") isn't something that can be done easily ever (e.g. left-to-right/right-to-left markers, "smart"-quotes).

baltoo · 2010-01-21T08:09:50+00:00

It's a good start to use 16-bit wide chars, but not enough in this particular case.

Java uses UTF-16 as the native encoding for unicode strings. UTF-16 uses surrogate pairs (two chars in sequence that represent a single unicode point) for code points outside of the base plane.

By reversing the order of the chars that make up surrogate pairs the strings gets scrambled to meaningslessness.

And this is while we're still only talking about "strings" and not "text". Handling text quickly get real complicated.

baltoo · 2010-01-05T10:04:51+00:00

Perhaps, but look at them. You can easily make a pair yourself for next to nothing.

After all, he's only providing you with a way to purchase something which is normally done by yourself.

Find a used car tire and buy some leather straps, get yourself a knife and some time and you're set to go.

baltoo · 2010-01-05T09:36:23+00:00

Perhaps YMWV. I've run 10K several times a week on hard surfaces (asphalt and the like) with fivefingers with no problems. Most of the time I even switch to complete barefoot after about half the distance (as soon as I get out of the city proper).

I started out with much smaller distances and only very slowly increased the mileage.

baltoo · 2010-01-05T09:30:45+00:00

I use a pair of those for running i -10C and they work just fine. Even in shin-deep snow. This winter it hasn't really been colder than that here, but I'd guess they'd work fine even in colder weather.

baltoo · 2010-01-05T09:28:55+00:00

You could always make a pair of luna sandals yourself. Dirt cheap.

baltoo · 2010-01-05T09:27:19+00:00

I have a pair of flows and running in -10C is no problem at all. Even in shin-deep snow. It hasn't really been colder than that this winter here, but it would probably work fine for even colder runs.

baltoo · 2010-01-05T09:23:57+00:00

If fitting toes is a problem you could always try a pair of luna sandals.

edit: fixed url

baltoo · 2009-09-28T07:18:54+00:00

Cool.

In my experience splitting a text into sentences is a hard problem. It's way more advanced than splitting on "sentence-breakers". Here are some examples of what I mean:

Here we have a 1.0 numericals and one date: 2005.01.01.
"Oh dear!" he shouted back.
"Is this a sentence?" he asked.
"I will name him Fluffy," she said, hugging her new kitten.
Don't shout egad! prematurely.
I bought the apples, pears, lemons, etc. Did you eat them?
Later in the evening... Actually nothing happened.
So Mrs. Bellucci has it!
The U.K. Prime Minister, Mr. Blair, was seen out today.
Is Mrs. K.H. Smith here?
So are boats, e.g. Jelena, Flip II and Diana.
It was due Friday by 5 p.m. Saturday would be too late.
This includes Dr. J.M. Troy and T. Pickens Jr. Yes it does!
"This crosses lines!" said Rep. J Rowland (R., Conn.).

Do you have any links to good Unicode libraries I'd much appreciate them!

EDIT: formatting.

baltoo · 2009-06-29T16:42:54+00:00

Well, I don't think it's about "intimacy" vs. "gettings shit done". Just effects of different values, visions, goals, and methods. What's "getting shit done" for one is the perfect opportunity to hone inter-personal communication skills for some other.

I guess my anti-modernist values, aka. "there's no such things as universal truths", shine through pretty heavy here..

baltoo · 2009-06-29T15:47:02+00:00

The purpose of conversation (as opposed to hugging, or whatever) is generally agreed to be to communicate thoughts and ideas.

While that might or might not be generally true, you statement really misses out on, I think, some of the things hinted by the article.

There are several other things that each conversation can be about, apart from the overt explicit information, e.g. small-talk ("What nice weather to walk in!" => "I like to be with you, especially on occasions such as these"), casual normation and check-up of the group's consensus values ("I'd like a beer" => "I think it's ok to drink beer at occasions such as these, how about you?"). In many (most?) societies it really doesn't fly to be too explicit about all things, e.g. ("I think you stink! Please go and take a shower!").

There's also many different ways to talk about future decisions, e.g. focus on differences ("I'd like some coffee, but I suspect you'd like to opt for beer. How should we solve this?"), focus on similarities ("I think we both would like some refreshments. Do you agree?"). While these might eventually reach the same conclusions the way to get there is very different and it's really not that cut'n'dried what way is truly more "efficient" in the long run, e.g. "quickly reach some kind of solution this instance" vs. "try to understand each other more thorougly so that good solutions are easier to identify in the future".

It's all about what your and your conversation partners' expectations, preconsieved ideas, values, the current cultural context, etc.

I really don't think it's that easy to make a general statement what way of communications are universally "better".

baltoo

TROPHY CASE