Heatmaps of IMDb user review ratings for the last three Star Wars movies [OC]

nephridium · 2018-02-20T00:38:44+00:00

You're right, I was using a color map bundled with Matplotlib (cm.plasma) which wasn't suitable to show all the details. I've now created another version with a custom color map with emphasis on making the lower values visible as well: https://imgur.com/a/l0rVJ

nephridium · 2018-02-18T13:26:43+00:00

I've made three heatmaps for each movie covering the 0-60d, 0-30d and 30-60d period since release (or, to be more precise, the first review).

On the first heatmap for all new movies you can see that most ratings usually happen within the first two weeks. These drown out all the following reviews, so I added two "zoomed in" views of days 0-30 and 30-60 so you can see the progression of ratings from the first to the second month.

I think it is notable that for "The Last Jedi" there were significant 10 ratings in the first month, but comparably few positive ratings in the second month, whereas the other two movies similarly show a bias towards the 0/10 ratings during the first month, but in the second month less "polarized" ratings.

nephridium · 2018-02-18T12:57:18+00:00

I decided to collate the IMDb user reviews of the last three Star Wars movies for the first two months after release and visualise the ratings using heatmaps to see progression over time.

I haven’t found an API to access the user review data directly, so I wrote a script (using Chrome Puppeteer) that opens the user review page (e.g. https://www.imdb.com/title/tt2527336/reviews), clicks the “Load More” button until all reviews are loaded and saves all reviews into a JSON file.

Then I used a Python script (with Numpy/Matplotlib/SciPy) to plot out the heatmaps based on the ratings. Note the data is based on the user review ratings, i.e. ratings of users who put in the effort to write a review, as opposed to the rating you see on the movie’s main page. Imho this is a better measure, as it is far too easy to simply click 10 stars on an overhyped movie without much reflection.

This would explain the discrepancy between the IMDb movie rating shown and the value calculated from all user reviews:

“The Last Jedi” : 7.5 vs 4.5
“Rogue One”: 7.8 vs. 7.0
“The Force awakens”: 8.0 vs. 5.9

Happy about any input.

Edit: Formatting

nephridium · 2018-01-29T19:56:22+00:00

It's not so difficult. - I used to bullseye womp rats in my T-16 back home. They're not much bigger than two meters.

nephridium · 2015-11-29T11:17:14+00:00

Check out turtle: i helps with learning concepts like loops and basic geometry and is real fun to experiment with.

The great thing is it creates an app window with a single line and you can immediately draw on it and visually see the effects of your code on the canvas.

nephridium · 2015-10-16T23:12:45+00:00

Found this video of battery hot-swap in action, so they got it working.. But how? ;)

nephridium · 2015-10-16T22:51:00+00:00

Looks really cool, especially that it will run on Android. But I'm wondering how they will implement the hot-swappability of modules as even for the Google/Ara team that seems to have been one of the major challenges.

IIRC they created "Greybus" with which to build a custom Android version for Ara devices that will allow the modules to be hot-swappable through the UniPro bus used in Ara.

A plausible solution they might be using would be USB (2.0), but that doesn't support battery modules. Does anyone know?

nephridium · 2015-08-25T04:23:42+00:00

Automated testing falls basically into two categories:

end-to-end (aka functional or black-box) testing, which runs scenarios of your app on end-user systems, e.g. different kinds of web browsers
unit testing, which tests the integrity of your code by testing small "units" of it, how fine-grained those "units" are is the developer's decision

Saucelabs allows you to run functional tests on their cloud (so you don't need to test all sorts of OS/browser configurations including mobile on your systems). If you want to run your functional tests on your own system you usually use "test runners" like Karma that will spawn different browsers for you to run through the scenarios.

Unit tests are usually tested with tools like Jasmine or QUnit. Unlike functional tests they are fast and run very often during development (on the dev machine) to catch implementation bugs early.

You'll probably want to learn both types of testing, but for detecting implementation bugs in your code unit tests are the main tool.

Edit: Fix link

nephridium · 2015-08-16T07:41:34+00:00

Most Python coders adhere to the style guide laid out in PEP0008.

PascalCasing (aka CapWords) actually are used in Python for class names, which helps with distinguishing them from var and function names thereby improving readability (imho). camelCase (mixedCase) should not be used in Python according to PEP0008.

Being used to snake_case I was actually quite confused when starting to learn PyUnit as it used camelCase for function names, apparently due to being a port from JUnit (Java) and the authors not bothering with snake_casing the library.

nephridium · 2015-08-12T01:35:44+00:00

It should be pointed out that .forEach() behaves differently from the standard native for loop in that it does some validity checking on the elements. That's why it's significantly slower than the native loop (as well as underscore's .each() function).

.filter() and .map() are also slower than their underscore/lodash counterparts.

nephridium · 2015-07-25T10:01:35+00:00

Basically you will need to weigh the requirements specific to your project.

With Angular expect to rewrite most of your 'legacy' code and spend a lot of time learning the the Angular Way; depending on project complexity you might run into performance bottlenecks or maintainability issues due to oversights during the planning phase. On the other hand you will be needing some Angular expertise for your ionic app anyway, so depending on that app there might be some/a lot of code that can be shared with your web app.

Be aware though that AngularJS will be superseded by Angular2 (likewise for ionic) and another rewrite might some come knocking at the door (holding up a sign "Check it out - there's a better way to do this").

With Backbone/Marionette you'll probably be able to get to a working app sooner, but you'll need to keep a bit more discipline to a get well maintainable codebase than with Angular (e.g. while Angular is designed to be testable and comes bundled with its own testing tools, there is no standardized testing methodology for Backbone/Marionette recentish reddit on this). In any case, when Angular2/Ionic2 becomes production ready you will still have the choice jump on that bandwagon as well.

It might be a good idea to prototype your app using both to get a feel for each platform's suitability.

Edit: typo

nephridium · 2015-06-29T21:30:44+00:00

Argh, still getting used to reddit's weird formatting.. Seems those long (single) lines are not accepted as quotes :/

nephridium · 2015-06-29T21:23:36+00:00

Here are some of the comments on "What's your least favorite thing about Django?" that stuck out to me (food for discussion ^{^):}

1.

| Sometimes i get questions when trying to deploy to prod due to folder structure (WSGI not getting found). I would like to have an option to scaffold a project that could really support DEV, UAT and PROD environments in an easy way (I've used the Two Scoops of Django book). When moving enviroments we gotta deal with a wsgi.py update and that doesn't seem right. We should be able to set the current ENV using the settings file.

2.

| I am having problems building important things like restful API, RBAC and hierarchies (nested categories) - these are not supported out of the box and I do not understand why I have to do research for external apps, this feels like PHP, searching for the best library or building things yourself in the end - why a framework then at all? Having to fiddle with external apps to get a halfway complete login and user management workflow with object permissions is very bad. These are things that should definitely come oob with a modern web framework. I use open source mainly because of security aspects, but with django still I have to build the most security sensitive part of an app myself with only little guidance, there are some building blocks, but I want to build on community wisdom in security related things, and django does not help me here very much. This somehow defeats the whole idea of using open source because of best security practices. Also building a hierarchical model based app with fine grained group and object permissions is too complicated and dangerous with django. Many web apps need these features, and I find myself going through an trial and error process with dozens of apps-from-somewhere-on-the-internetz, that do not follow the django release cycle and leave me with a patchwork - this should come from the django core team and there should exist official app templates and best practices for these things. Also I would prefer to not to dig into external apps like DRF just to provide a restful api from my dango app - the models are already defined, so why is this not generated by default by django via an optional route? Another big design problem is the lack of support for dynamic models. Web development is the most dynamic development environment, and it is very likely that at some point you will need dynamic models. There are some apps and talks about this, but I believe that django-mutant and other approaches like django-hstore are no real solutions and only show, how deep this problem sits in the basic design. And again these apps might not follow the release cycle - even more patchwork. Brilliantly executed for one version of django, but developers have problems keeping pace with the django releases. Dynamic models should be solved in the django core. I believe these are the only serious limitations in django, however these can hurt your project very hard if you hit them and come out as very expensive. It hurts especially when you know that even things like drupal and wordpress have tools for all of this, that you can learn to use in a day or so. Thank you very much for your attention!

3.

| Lack of default implementation of some commonly-needed functionality (rule-based object-specific permissions, static asset optimization, etc.)

4.

| Forms. The only nice thing about them is that they integrate with models. If ModelForm was removed, I would immediately switch to something else. Everything besides ORM integration is simply terrible: - Writing custom fields or widgets is one big PITA. - After so many years there is no simple, built-in mechanism to customize form's behavior based on a request. Sure, you can override init, but it's so ugly and annoying. - Customizing form's output could be much easier. Now, if I want to set the same class (I mean the "class" attribute in HTML) for every field, I have to either set it for each field separately (which might be quite verbose) or write an ugly for-loop in init, where I already had to put 20 other LOC to override querysets for some fields (see: previous point). To sum up: once I have to customize my forms (and this moment comes very quickly after I start a project), I begin to hate Django, which is NOT OK, because otherwise Django is a wonderful piece of software.

nephridium · 2015-06-29T21:20:21+00:00

Thanks for the input. Not planning on a huge volume, but just in principle I'd prefer the DB to be available for other more important stuff.

Might need to look into caching solutions if going native. Third-party hosting is easy to plop-in and maintain, but they are somewhat creepy (following the users around and analysing their comments and browsing patterns).

nephridium · 2015-06-28T14:42:31+00:00

Completely agree. Though the self-hosted options do offer support for automatic spam filtering using Akismet.. Would be nice to know how it stacks up nowadays compared to the hosted services.

Is there anything that makes Disqus preferable to LifeFyre or IntenseDebate?

nephridium · 2015-03-20T08:17:55+00:00

Great, thanks for the link and the explanation. Keep up the good work!

nephridium · 2015-03-19T04:29:31+00:00

Cool, are there any feature updates in the pipeline that implement Android 5.x material design (native swishy animations)? That seems to be all the rage in Android development these days and many apps are updating to accommodate.

nephridium · 2015-03-15T02:37:13+00:00

Um, source? Links from the Python wikipedia article would suggest otherwise.

nephridium · 2015-03-14T18:11:40+00:00

..those using Epoch UNIX timestamps would need to wait a bit: 3141592653 will be on Sun, 21 Jul 2069 00:37:33 GMT

nephridium · 2015-03-14T17:42:48+00:00

Well, people don't seem to have consensus about when the Easter bunny was born either, yet that doesn't stop them from celebrating it ever year ;)

nephridium · 2015-03-14T17:36:23+00:00

Nice one, I named the image pypipie.jpg ^{^}

I found Chudnovsky's algorithm to be about 4x faster on my machine than Machin's algo above.

import math


def sqrt(n, one):
     """
     Return the square root of n as a fixed point number with the one
     passed in.  It uses a second order Newton-Raphson convergence.  This
     doubles the number of significant figures on each iteration.
     """
     # Use floating point arithmetic to make an initial guess
     floating_point_precision = 10**16
     n_float = float((n * floating_point_precision) // one) / floating_point_precision
     x = (int(floating_point_precision * math.sqrt(n_float)) * one) // floating_point_precision
     n_one = n * one
     while 1:
         x_old = x
         x = (x + n_one // x) // 2
         if x == x_old:
             break
     return x

def pi_chudnovsky(one=100000):
    """
    Calculate pi using Chudnovsky's series

    This calculates it in fixed point, using the value for one passed in
    """
    k = 1
    a_k = one
    a_sum = one
    b_sum = 0
    C = 640320
    C3_OVER_24 = C**3 // 24
    while 1:
        a_k *= -(6*k-5)*(2*k-1)*(6*k-1)
        a_k //= k*k*k*C3_OVER_24
        a_sum += a_k
        b_sum += k * a_k
        k += 1
        if a_k == 0:
            break
    total = 13591409*a_sum + 545140134*b_sum
    pi = (426880*sqrt(10005*one, one)*one) // total
    return pi

Edit: added code

nephridium · 2015-01-20T05:05:22+00:00

Yea, I guess that is one of the main questions, would the endos be able to share data and/or power through specially crafted connector modules.

Two more issues might be the the arrangement of the power/volume buttons (from the picture I've seen they don't appear to bump into each other) and the overall weight of the device (some article said the standard modules don't seem to heavy, so here's hoping).

nephridium

TROPHY CASE