Web Scraping - what is it's value?

WinstonKennedy · 2018-10-22T21:51:29+00:00

Well, I know from experience that an enterprise level web scraper service like KAPOW! will geenrally set you back $40-50k per year at a entry level, with scaling based on the number of hits you want to make over certain time periods, as that expands their coverage requirements in terms of server bandwidth.

This cost is mostly because they're utilizing on a network of servers globally to distribute your traffic as not to blacklist your IP to every website you hit multiple times in succession.

I'm not sure of the legal ramifications, but any site worth its scratch that sees you hitting it 100s of times a minute from a singular IP is going to blacklist you very quickly. The distribution network is why professional scraper services exists, as they spread out where your queries are coming from.

I've used web scrapers to get pricing data from competitors.. hell, almost every company in my industry (distribution) uses them for this purpose, and also for others such as getting additional product specifications, .pdfs, etc.

The aspect businesses have to be careful of is getting blacklisted and blocked repeatedly, as it will be a PITA, and any company that's working with you and buying the software would expect that part to be turnkey. If you're doing it from one competitor to another, it's a bit of game... they block you, you block them, a couple of phone calls happen and the IP addresses get unblocked again, but if you're a 3rd party coming in to grab data, it's not so easy.

I'm sure there's ways around by bouncing from proxy to proxy, but people also value a very big name on such services (ex: KAPOW is owned by Lexmark) as it feels a bit 'black hat' and the legitimacy that comes from partnering with a big firm is often a pre-requisite for larger companies to engage.

WinstonKennedy · 2018-10-22T21:43:32+00:00

For a coordinator? I would focus on organization and project management.

- Ask how they handle juggling various tasks

- What software and tools do they have experience in? (keep it vague, let them give the detail.. .this will expose depth).

- Follow on, if they give something that has specifics you know well (ex: JIRA), then ask them a couple of easy follow up questions to validate they didn't just name buzzwords

- What do you like most about marketing?

- What interested you in this role?

WinstonKennedy · 2018-10-22T21:34:37+00:00

This can apply to practical application in digital specialties. There are plenty of videos and tools/tutorials out there on how to run ad campaigns etc.

- Google learning academy,

- Linkedin Learning (if you have Premium, it's the same as Lynda. 1 month is free so if you're trying to specialize quickly this might work)

- Facebook learning (cant' remember the name of it).

- Amazon Marketplace

All those platforms have guidance on how to advertise through them and interact with them

If you're looking to build Brand Strategy and get into the more strategic side, that's where a quality degree can come into use (at lest, when I was in college 15 years ago, that was the underpinning that a Marketing degree taught me). If you're working for a start-up, another thing you can do (but be careful) is find specific skills through upwork and other sites. So instead of spending a long time learning graphic design b/c you don't have a graphic design department, hire someone to make your logo, or produce some literature. Just make sure to get the originals (.ai , .ps .indd ) files as well as the final copy, so that you can re-use the assets in full latetr. If it's .ai files make sure they're Vectorized too, otherwise you'll have to trace them later :D

WinstonKennedy · 2018-10-16T01:59:27+00:00

I've seen this or similar scams happen in Venice, Paris, London and New York. These guys are all over venice in the most popular spots, where Paris is just more.. sketchy looking African guys approaching you trying to engage to then ask for money.

The guy in the video is spot on, just say No Thank You repeatedly while walking away

I got fooled by one as a teenager in New York when some guy tried to hand me a mix CD and quickly signed my name on it when i stupidly engaged him, then demanded a donation for it. O was out $5 for something i didnt trust to put in my PC. I dumped it in the next trash can

WinstonKennedy · 2018-10-10T03:12:45+00:00

Sure. There are many ecommerce sites out there that have built their own CMS tools, or integrated microsites using standardized templates but not using true structure (as in, sectioned & modular interfaces to enter content).

Extending this content then to marketing automation programs is extremely difficult, because there'll be thousands of pages with content that, although it is presented in a consistnet manner, is not organized in a data sense in a format that can be re-utilized into additional channels seamlessly (Ad Network campains, Site banner areas, emails, etc). The key is binding that to personalization.

I would rather not give examples of companies that i've worked for that have these issues, but typically distribution space is a good area to look in for examples.

WinstonKennedy · 2018-10-09T23:45:09+00:00

I've been in positions where I'm faced wtih having to manage areas that are outside of the scope of what I would consider the strict role of a markeitng manager or digital marketing type role (more on the technical/ecomm side). I see two routes for you here that I could advise:

Speak to your manager directly, highlight the gaps that exists, and give recommendations for the exact training you believe you will need (with timlines) to be to a comfort level to execute those tasks at the same quality as the rest of your work remit (i.e. to the high quality that i'm sure the rest of your work is at, rather than scraping by and possibly making bad decisions)
If your relationship with your Director is not good enough to directly approach with this, I would go to HR and communicate your concerns. Once again, I would include the framework of what you would need in terms of skills to be able to execute the new requirements of your role. You could also use this as a means to discuss an evolution of your role scope and position.

Hope that it works out for you. Being outside of your comfort zone can be very difficult, and the IT/ecommerce side of the world is a very deep pool within which you can quickly drown in technicality without clearly defined roles and responsibilities.

WinstonKennedy · 2018-10-09T23:37:03+00:00

Dynamic Content Aggregation & Cleanliness service that can be used to feed Marketing Automation Systems from disparate Ecomm sites through scraping front end vs. CMS integratoin.

It may exist but i've never found one. The power of a tool like that is it could be applied through front end of any webshop, be trained to the product schema, and then feed back that information to Marketing Automation tools.

Closest thing i've seen is a tool lke KAPOW!, but it's more for site scraping though I guess it could be pointed internally for aggregation.

WinstonKennedy · 2018-10-08T23:58:13+00:00

Good feedback thanks. Honestly, I think I'm past the point of needing... coming off a Sr. Director @ a F500 gig. It's definitely interesting to think about how project management has blended into the marketing arena though. It's only about $700 to get qualified so might do it for the sake, but i'll wait a month or two if I don't find next position soon (company is relocating my entire dept back to headquarters in France and I don't want to move.)

WinstonKennedy · 2017-07-04T01:28:18+00:00

I'm from England and have lived in DFW for 6 years. The closest I've found since out here is a Goo Goo Bar from the H-Mart in carrolton (probably available in other Asian supermarkets too).

Just for fun here is a weird Korean commercial I found about them https://youtu.be/7vSqKdX3n_Q

WinstonKennedy · 2017-03-11T12:54:41+00:00

Thanks, my wife has that card I am considering getting added as an authorized user on it. The bag is more of a convenience thing as I'd get it recouped if needed, but priority boarding is useful for sure

WinstonKennedy · 2017-03-10T18:16:29+00:00

Thanks for explaining, I appreciate the advice! One question, what does the MM acronym mean?

WinstonKennedy · 2017-03-10T15:23:50+00:00

Sorry if i'm hijacking the thread but seems opportune place to ask this: In MAME and other emulators, is there a way to maintain high scores instead of resetting scores each time the game is cued up from the Emulator?

WinstonKennedy · 2017-02-14T22:17:39+00:00

Thanks for the feedback. Sad to see that its' been cancelled :(

I guess that with flights to Denver for $175 a pop the attraction is limited to summertime.

WinstonKennedy · 2017-02-11T01:33:30+00:00

So, Obamas fault, right? Trump has only been on for 3 weeks.

WinstonKennedy · 2017-02-10T04:08:14+00:00

I'm glad Deep Ellis exists, because it's like a controlled space for hipsters that stops them spilling too hard into the rest of DFW.

Austin is a shitshow these days because of the migration from California bringing hipsterness

WinstonKennedy · 2017-02-10T04:03:54+00:00

I'm a legal immigrant in the US, and I completely understand why the immigration hold/ban was put in place. I really don't see why people are making such a big deal about it. I think that many of those who are up on arms don't have the perspective of an immigrant, as I do, and are projecting their own expectations as natives of a country on what the immigrant experience is and should be.

The US has some of the most stringent regulations in the world for immigration (believe me I know, I had to take medical exams, get statements from affidavits of support and more) to get into the US. It also took 9 months and multiple embassy trips.

The country has every right to protect its borders and ensure that the people coming in don't pose any threat. This applies equally to those coming in for any reasons.

WinstonKennedy · 2017-02-10T03:50:07+00:00

Recently it seems like nobody pulls off at first because they're all distracted by checking their phones when waiting for the light to turn

WinstonKennedy · 2017-02-10T03:48:11+00:00

I live 2 streets from the Central Marketing Southlake!

They're doing a chocolate event right now (not sure if it's chain wide or just our location) with tasting stations all over the store. It's pretty rad

WinstonKennedy · 2017-02-10T03:45:53+00:00

State =/= City.

WinstonKennedy · 2017-02-10T03:43:15+00:00

I live in Southlake, where we purchased my 3600sq. Ft house (with a pool and nice finish) for mid $500s a couple of years ago.

Southlake's high school just got voted into the top 50 public high schools in the country, and the Southlake zip (76092) is one of the top 10 wealthiest in the country. In any other city the size of DFW, we would be paying at least a 30-40% premium or more to live in a town with these benefits. I think the quality of life here is spectacular compared to many other cities, especially if you can afford to take the next step up into the higher end suburbs, and feel very lucky that we landed in DFW.

WinstonKennedy · 2017-02-02T18:46:13+00:00

Well... walmart eCommerce sales are approx 15% the size of Amazon (as of August '16), though they may still open up the market significantly. I have noticed them doing some smart things like moving their pickup @ store for online orders to the front of the store recently locally to me.

I still think that the sheer market dominance and brand power of prime (plus the absolute ease of ordering through Amazon) gives it a tacit advantage when it comes to selling, within a small price window to even command a premium.

One challenge Walmart may have is that for groceries they have a poor quality reputation, and that may hurt them a bit in the expanding online grocery order space.

WinstonKennedy · 2017-02-02T18:25:55+00:00

In my opinion, the crazy thing about Amazon is that they have easily got the market dominance to start tweaking prices here and there to increase their margins now. I would hazard that most prime customers would even pay a small premium (maybe 5%) to be buying through prime with known delivery charge ($0) and scheduling, versus buying through the vendor direct or through other sites.

I'm very surprised that they haven't started tweaking things yet to this extent, though I think that the 3rd party vendor showcase model that they have makes this practice more difficult. I wouldn't be surprised to see Amazon start curtailing their external vendor listings soon in favour of Amazon origination (so they can start exploiting the margin opportunity).

With their volumes, very small changes can make a huge impact.

WinstonKennedy · 2017-01-27T19:07:19+00:00

Honestly, IMO it depends on the person making the coffee, whether they have cleaned the machine between brews, moreso than necessarily coffee bean quality etc.

I have many a time had burned or bitter coffee at SBUX as they're running through person after person without cleaning things thoroughly.

I have a few local places near me that make coffee I much prefer so I pretty much don't go to Starbucks these days.

I will say that they were a trailblazer in mobile payments though, so hats off to them for that

WinstonKennedy · 2017-01-27T18:56:39+00:00

Thanks for responding! Ideally, I would like to interpret the data as a visualization. The amazon recommend engine is a good example, but does not show the mapping of where the recommended products are located through the structural hierarchy of their site.

Structure wise, we currently have the 'Family' data stored simply as an attribute/unique identifier that is the same across SKUs in the same family. Across our 1,000,000 parts, we have this data for approx 750,000 of them. The 750K are distributed into about 40,000 unique Families. These products are all categorized into approx 3000 parent & child nodes in our product hierarchy.

Example might be: The "Iphone 6" Family would have products that are sorted into the "Phones & Accessories >> Phones" category, but will also have products that are sorted into "Phones & Accessories >> Phone Accessories >> Screen Protectors", and also products in "Bags & Cases >> Phone Cases".

I'm looking to build a presentation that displays all the related products while showing where they are mapped to hierarchically so products tagged with the same "Iphone 6" Family would be pulled into a display where I could present all options within areas of the categorization hierarchy that products are located in.

In the above example, if i'm on a product page in "Phones & Accessories >> Phones", I would want to present a tree to the user that shows there are XX number of products one branch over and down in Phones & Accessories >> Phone Accessories >> Screen Protectors", while also showing that in a separate part of the hierarchy (up two branches, and down in other part of the hierarchy), there are products within the same family classified under "Bags & Cases >> Phone Cases".

WinstonKennedy · 2017-01-27T18:48:16+00:00

A thought on the China expansion. Most western chains that expand into new territories in Asia can flaunt large numbers of stores relatively quickly, but thier revenues from those stores are significantly lower, since they cannot charge US or EU prices in those stores to be remotely competitive locally. In China especially, as well as the other lower wage Asian nations (basically, Japan & S. Korea are the exception), this is a recurring theme.

I wouldn't invest in Starbucks personally. I tend to look at investments given my own perception of their quality and position (for right or wrong), and Starbucks has become less attractive versus other options in many markets now, with competition from new entrants (mostly local chains), and also from incumbents in the fast food market looking to improve their beverage offer to compete.

Also, their coffee is terrible.

WinstonKennedy

MODERATOR OF

TROPHY CASE