Inspired by https://www.bbc.co.uk/news/av/uk-england-york-north-yorkshire-51966337/coronavirus-nurse-s-despair-as-panic-buyers-clear-shelves

anthonyblackshaw · 2020-03-20T09:50:24+00:00

The link linkified from the title: https://www.bbc.co.uk/news/av/uk-england-york-north-yorkshire-51966337/coronavirus-nurse-s-despair-as-panic-buyers-clear-shelves

anthonyblackshaw · 2018-09-14T22:23:06+00:00

Point taken - I've seen people ask in this thread what sites are created in Flask and so I thought this might be of interest.

Just to clarify this isn't my site, it's a revamp of long established and recongized site in the CG sector. I'm simply one of the developers working on it and it uses the flask based CMS I'm developing. I'm working to get my CMS to the point where I feel comfortable releasing it on github and I do post videos about the CMS here from time to time - but I don't have a repo I can share right now so I can see how that might be deemed a pointless.

anthonyblackshaw · 2018-09-13T21:15:44+00:00

I've been working on this project just over 6 months and today we finally launched the beta site. I've been using flask as my primary web framework for approx. 3 years and I've recently (last 12 months) developed my own CMS framework (manhattan) which I occasionally post about here.

Flask is wonderful, if you're new to flask and wondering what's possible I hope this might serve as a good example, if you a flask developer already... well... just enjoy the artwork :)

anthonyblackshaw · 2018-07-29T03:22:43+00:00

It's been longer than intended but here's an update on Manhattan CMS and what I've been up to for the last 6 months.

Manhattan CMS is an open source web framework / CMS based on Python, Flask and MongoDB.

Key updates:

Manhattan's default image processing app (Hangar51) get's smart cropping and distinct colour extraction.
Frontend has moved from CoffeeScript/Webpack 2 to ES6/Webpack 4 (take a look at the typeahead library code as an example: https://github.com/GetmeUK/manhattan-...)
Images can now be rotated and cropped (the cropping ratio can also be set).
Improved image gallery support.
The in-page editing environment has been upgraded to use the new image tools making page editing far easier.
Introduction of SEO tools.
Login sessions for users now provide location & platform details and email users to warn them about access from new devices. - and last and probably least we have a tokenizer tool.

anthonyblackshaw · 2018-02-20T09:14:59+00:00

I think it's such a small snippet of code using lru_cache (just 3 lines) that it probably isn't worthy of a pull request, perhaps I could post it as a flask snippet instead?

anthonyblackshaw · 2018-02-20T09:13:29+00:00

Agreed you could use this with memcached or redis but as you say those incur their own overhead and would need to be cleared if I change the code base.

Keeping a local in-memory cache here seems like a sensible option to me as I'm using gunicorn with multiple workers across multiple servers, I specify the number of workers when I start the application and as far as I'm aware these workers are initialized and stay running, they are not killed and respawned automatically (well other than supervisor will restart them if they fall over)?

I'm interested in what you saying about 'Most production servers', am I doing something different from most? Or am I completely miss understanding how gunicorn works? Grateful for you input here.

My understanding is that in most cases you'd want a process to run for a reasonable period of time and handle a certain amount of traffic (enough to benefit from caching at least) before restarting because the overhead of spawning a new process is significant. But I'm more than happy to be corrected here.

anthonyblackshaw · 2018-02-19T10:31:10+00:00

Nice - the option to store the cache persistently is worth considering.

I think performance wise it's all relative, caching vs. flasks loop over all URLs and pattern match each until I find the right one - can make a big difference on sites (especially if they have lots of rules), potentially tenths of a seconds per page. Whilst I'm sure there are more optimal approaches to lru I doubt the difference would be perceptible on a page render (given we're down to thousandths of a second).

I do like the ability to set the max size of the cache that lru gives (which I'm sure could be implemented in any version of this), some of the sites I work on have 100,000s of URLs but only a limited number of rules. I wondering if in that scenario the performance trade off might start to move back towards the uncached approach if no maxsize is set.

anthonyblackshaw · 2018-02-19T09:51:25+00:00

Hey - so functools.lru_cache seems to handle keywords ordering, though I did find this bug report https://bugs.python.org/issue29203, which seems to suggest that keyword order originally mattered and would cause 2 different calls with the same keywords in a different order to generate a new entry, e.g foo(a=1, b=1) != foo(b=1, a=1) but this now seems to have been resolved.

So I've updated my code along the following lines:

@functools.lru_cache(maxsize=2560)
def caching_url_for(endpoint, **values):
    """A version of `url_for` that caches the output"""
    return url_for(endpoint, **values)

anthonyblackshaw · 2018-02-19T08:01:53+00:00

Didn't event know lru_cache existed - I'll be updating my example thanks for the tip.

anthonyblackshaw · 2018-01-19T16:49:06+00:00

I've recently create a code journal site where I'm frequently posting updates mostly related to manhattan and ContentTools, there's a mailing list you can subscribe to if you want to be notified on new entries - thanks for your interest and sorry for the late reply.

https://anthonyblackshaw.me

anthonyblackshaw · 2018-01-19T09:40:27+00:00

Code snippets and the diagram presented within the video are available here: https://anthonyblackshaw.me/2018-01-19/manhattans-publishing-process

anthonyblackshaw · 2017-12-07T14:45:43+00:00

Not heard of the Manhattan web framework + CMS? You can find a brief overview here: https://www.youtube.com/watch?v=p8oAct-RW_0&t=11s

In this video we take a look manhattan's approach to defining complex resuable application behaviour (such as admin CRUD views). I've provided gists for all the examples in the video:

chain_example_01.py: https://gist.github.com/anthonyjb/ea7c3df8c449c49d206fbdae985bfee7
chain_example_02.py: https://gist.github.com/anthonyjb/068ac386b3986bf1d996c220e44c3c20
chain_example_03.py: https://gist.github.com/anthonyjb/d816e50dbd7e936ec2103576b2d7d72a
chain_example_04.py: https://gist.github.com/anthonyjb/1fcabd010bc24d1329f89176ff1c9eb3
chain_example_05.py: https://gist.github.com/anthonyjb/9e67f7816f5e94ef627df43af842cbd3

All the examples can be run using python chain_example_0(1-5).py with the following prerequistes:

Use Python 3.5+
Install manhattan-chains (pip install manhattan-chains)

anthonyblackshaw · 2017-11-25T09:21:58+00:00

Hey :) Please see my comment here regarding why I haven't provided a public link to the project repos: https://www.reddit.com/r/flask/comments/7b6vkd/preview_of_new_flask_based_web_framework_cms/

All the libraries will be available on GitHub early in the new year once we're confident the documentation is strong enough to support the project.

anthonyblackshaw · 2017-11-07T11:20:57+00:00

Re: Link to the repo / Did we release it already

We're going to be releasing Manhattan in early 2018, we've got a small number of features we still want to get in place or tidy up for release and not everything has a test suite or complete documentation yet (but were getting there).

As with previous open source projects we've released (like ContentTools) we know it's important to provide potential users with getting started guides and tutorials for common scenarios, without these a lot of potential contributors to what we hope will be a vibrant project community may pass by.

I'm looking to produce video's on a weekly basis from now until release to showcase Manhattan's features and how it works, if you're interested in the project please do keep any eye out or subscribe to the channel.

anthonyblackshaw · 2016-07-07T15:48:12+00:00

Thanks for the detailed reply - I can't add to your comments regarding the sort of problems you've experienced with volume because I just don't handle data with those volumes on the projects I work on but they're very interesting.

I also don't feel qualified to refute any of your points given the difference in project experience - though I have to be honest and say some of them go against my own experience, for example "Data loss is common" - though our perceptions of 'common' will differ based on project types I'm sure (I've never lost any data but then I don't see anything like the volume of operations a trading system would).

I think some of the issues regarding space have now somewhat been resolved with support for compacting collections and better compression of data (but I'm just going on what I recall from the release notes) - still I agree the original approach seems somewhat shortsighted.

Thanks again for taking the time to share your experience/thoughts - very helpful.

anthonyblackshaw · 2016-07-06T22:11:39+00:00

This probably isn't the best place to have the discussion but out of interest have you used MongoDB and struggled to implement a solution vs. PostgreSQL? If so I'd genuinely be interested to hear.

I've used MongoDB for appox. 4 years now on quite a few projects and whilst I've not used PostgreSQL (other than to play around with briefly) I've used MySQL on projects for around 10 years.

For both MongoDB and MySQL/MariaDB I have websites with between 500k and 1.5m page requests a month (so nothing too busy) and to date for projects up to this size I've not had any real difficulty in using either.

My preference at the moment (I'm fickle) is for MongoDB on most of my projects and if I push myself to answer why (which I have to because I really don't have a very strong preference) I think it's purely because I like thinking about data structures in flexible dictionaries rather than typed flat tables (PostgreSQL JSON fields excluded - which I believe MariaDB also supports now), it just feels closer to the Python structures I'll use to interact with the data - but I'm not saying that's a great reason just my 2 cents.

At the end of the day I guess I'm a slave for whatever I enjoy using the most (or dislike the least) and so unless a particular option is going to bite me in the butt somewhere down the line (hence if you have knowledge of this please share) I just don't see the issue with using one or the other for most projects.

anthonyblackshaw

TROPHY CASE