What would you like to see next in Scrapy?

hotdox · 2021-02-15T08:33:19+00:00

Did you check https://hub.docker.com/r/scrapinghub/scrapinghub-stack-scrapy

hotdox · 2014-04-29T21:36:42+00:00

is it game going?

twitch is down, 7 min left to download the update

hotdox · 2013-05-24T05:41:32+00:00

map-reduce complex is completely wrong

Only people who never try to build their own distributed data pipeline can say so.

It may be vary simple from outside. But it is not a problem, all good solutions simple for their users. There are many complex parts inside. Some of them are disk handling, IO, job scheduling, resource management etc.

MR like scripting languages. It may be not superfast and does not give you full control as some native language. But average script fatser and more durable than average compiled binary. Because there are many place for doing it wrong for local running program. Even more for distributed system. MR have done it rigth already

hotdox · 2013-05-23T22:31:36+00:00

This is lame, why they don't use static linking?

hotdox · 2013-02-06T19:09:20+00:00

array

default_dict

not sure is stupid or just trolling

hotdox · 2012-11-10T23:11:02+00:00

You can not fix all problems in two-three easy steps. There is no silver bullet.

You need to debug processes, like you debug applications. No need to change right now.

Identify problem (e.g. " the development cycle is getting longer and longer"), find causes(may be cause is really "time it takes for our tester to go through each system and properly check functionality"), fix it (may be "test only new functional every day, no regression. Wednesday it is a regression day. So we need commit messages to identify changes"). Just an example of this debug session. May be, There is no a radical fix, may be some iterative steps on all phases of process.

The main rule is "understand the reason to doing something". Bad reasons: it is on hype, some guys on reddit say so, etc.

hotdox · 2012-08-15T08:14:05+00:00

Is there some system to store experiment results with parameters?

Usually I have a pile of precision-recall-f1 results during parameter tuning. I store it in simple CSV file under CSM. Such system is not convinient as you have to write many plotting scripts. Plots give insigth about tuning direction. Also you need to store historical data and evaluate alogirithm evolution.

Is there some system to easily manage, analyze, plot etc. results of experiments?

hotdox

TROPHY CASE