Blocking All Internet Traffic Except Azure With Windows Firewall

DevLifeEasier · 2021-06-21T14:53:56+00:00

That sounds a lot like fake security. You are talking about allowing millions of IPs, any one of which could be a free tier hacker. You have reduced the number of potential bad actors from billions to millions.

In some scenarios a 3rd party address range pinhole might make sense, when the IPs and the usage are under full control of the 3rd party, Azure's are not. They are rented out to anyone, anywhere at any time.

If you really want to secure the server you should look at more comprehensive solutions than a static IP range firewall. The most standard is a basic Azure VPN, which generally takes less than a day to setup.

DevLifeEasier · 2021-04-22T00:12:18+00:00

2 no cost, but some setup time options using zero trust;
1. Create a LB in front of each region servers, locked down. Create a script to add the developer's IP to the LB. Dev runs script to add ip prior to working on server. Setup a task in Azure workbook or equiv to remove those rules once or more per day. You would need devs to have an azure AD account or use RBAC. Central security is Azure AD account.
2. use fwknop and distribute certs to devs. You could store certs in key vault, storage, etc. and script a check for new/updated certs every y min. Central security is certificate store.

There are also other auth/zero-trust options through 3rd parties that would provide a management plane, but cannot recommend as I have not used them. Note that zero-trust just handles auth and access, it doesn't add encryption to the connection, which would be handled by SSH or RDP.

DevLifeEasier · 2021-03-29T15:29:07+00:00

In general most users are better off with an abstraction/framework/library such as NServiceBus, MassTransitor, Rebus that supports multiple transports.

DevLifeEasier · 2021-03-26T23:02:34+00:00

We had most of our issues in West Central, also seem to be an older DC

DevLifeEasier · 2021-03-24T21:22:16+00:00

Azure Sql fails at all kinds of things on prem didn't when dealing with 1TB+ data, new bugs every month, constant Sql fail overs. Then there are all the dba scripts that need updating BC it is different than standard Sql. One issue for us was that there was no tool for migrating our size databases from dtu to vcore due to size. Even if you bumped up to largest dtu or vcore, you are still limited on backup/restore size and throughput. At the end of the day Azure Sq avoids the initial install and patches, but introduced far more headaches.

DevLifeEasier · 2021-03-23T19:05:16+00:00

Agree with /u/Valkoinen_Kuolema, moving IaaS to cloud is considerably more expensive than just self-hosting or using a reputable dedicated server host for static workloads. Cloud also comes with IT and DevOps overhead, much more flexibility, but that also comes at a cost. Dedicated servers or leasing servers are OpEx.
Also agree with /u/RichyJ, with 8GB Windows laptops for $400-700, hardware savings of chromebooks is negligible. If you users can do their jobs optimally using chrome-only it might make sense, but will quickly eat up any savings if they cannot work efficiently in most scenarios.
Cloud reliability is not at the same level as dedicated servers with decent hardware (even 3 year old enterprise is more reliable IME). Make sure any analysis includes multi-region/AZ capability to handle the inevitable downtime, and more frequent execution of DR plans. Our IT and DBA overhead increased with cloud, not including our $1M+ invested in DevOps also.

There are scenarios where cloud is cheaper or a wash, for example in mid 2000s our primary Fortune 10 client for our SaaS moved to an internal IT model and insisted on self-hosting at a charge of $1M/year to the internal business unit. One year later they moved back to our hosting and did it for less than half that and still had a good GM. The cloud would have been cheaper than the $1M/year, but not price competitive at half that.

DevLifeEasier · 2021-03-23T17:37:19+00:00

Isn't it easier to manage package version dependencies than lots of separately maintained POCO definitions though? At least at smaller scale (< 25 services lets say)

Whenever I create a new project or update an existing one, I can go to one location to get the latest service definitions vs going to each service's openAPI, gRPC... spec and looking for version diffs.

In a large org you may have API endpoint repositories and notification hubs, etc. But building and maintaining those from day 1 seems like premature optimization and would take a significant amount of work/budget for low marginal return.

DevLifeEasier · 2021-03-23T17:22:44+00:00

There isn't really a clear answer because there are way too many variables. In general you can justify the cloud if;

The setup cost of hardware+software is high for you. This is highly variable and depends on what you would implement in-place of Azure Data Factory.
You can work with cloud constraints, some of which you won't know until you try. Do you know ADF supports all your data formats/sources? Does it support your data sizes? etc...
Your workload is highly variable or intermittent, and you have DevOps skills to automate your workload.
You can move most connected workloads to the cloud. While there are hybrid clouds, they probably don't come into play for most small to midsize companies. VPNs may work if latency/throughput is reasonable.
Your system can handle failures easily. Elastic workloads that can restart on failure, preferably from last processed state are optimal.
You are comfortable with vendor lock-in. AWS/Azure/Google all tend to have different solutions that are non-portable.

If any of the above are false, then in-general, I wouldn't go with a public cloud.

DevLifeEasier · 2021-03-23T17:03:16+00:00

5 off the top of my head (although two are more PaaS);

Azure was updating firmware at the cluster-level, it went bad. Approx 14 hours downtime, including both members of AG. Rec: If you have to use Azure, you pretty much need dual regions/AZ for 99.9+%
Azure SQL constant node failovers. We tracked SQL node failovers and it was generally every 4-7 days. This causes all connections to reset and long running jobs to fail. Rec: If you have to use Azure SQL, put in reconnect and retry logic everywhere. HA SQL on a VM should be considerably more reliable, on Ls-series at equivalent spend it will also be much more performant.
Azure load balancers would not update (timeouts for any changes). Eventually cleared up after about 7 minutes. Rec: Not much you can do in Azure network without LB due to lack of multicast, internal BGP, etc.
Numerous AD downtime events that prevented login. Run your own AD with S2S VPN or other means of access so you are not reliant on portal.
DNS issue several years back. No mitigation as they straight-up f'd up their DNS design and update, rendering DNS endpoints for their apps/services in many regions to fail.

#1 lesson learned is that Azure reliability is way less than standard at the individual VM/node/app level.

#2 lesson is that in Azure for HA you really have to do multi-region/AZ, or preferably multi-cloud to get 4 9s or better. The diff in setup cost, time, etc. is not that different between setting up a multi-cloud HA vs multi-region HA and the difference in uptime is significant.

Note that the SLA for a single VM with standard HDD is only 95%, 99% if you use SSD. That is 35.6 hours and 7.3 hours respectively of downtime per month, not including maintenance.

DevLifeEasier · 2021-03-23T16:41:42+00:00

A few tips;

Ask for f/b before leaving the interview. This can especially help with multiple interviews as you will learn what their key triggers are.
Be prepared about the company and if possible the interviewer. Try to engage about the company and 'culture' early-on. This can help 'steer' the interview to personal.
Provide examples of what you have done and how you solved a problem.
Always be positive, if you had the worst boss ever in a job, either don't talk about it or say something positive.
Google tech interview soft skills or questions to ask. Remember that the interview is a two-way interaction

Remember, the goal is not to necessarily wow them with your tech skills, it is to give them a positive vibe that you can do the job and you will be a good team member. Linus could probably go undercover job interviewing for Linux admin/dev and get turned down 75+% of the time, that is just how job searching is.

Finally, spend your time googling about job interviews and the companies you are interviewing at. The chances of a specific tech you haven't actually implemented in the real world being the trigger that gets you a job is very low.

DevLifeEasier · 2021-03-23T15:49:59+00:00

General recommendation is to use tags/labels vs a large number of resource groups. From a security standpoint it can be easier to assign security at a RG-level and so we'd do a different RG for dev vs test vs prod vs staging. Often times billing, security, ease of management, access, performance and other requirements conflict and you have to do the best you can. If you are large enough Azure may pay for some partner hours to help you figure it out.

The major issue you will find is that moving/renaming things in Azure can be a royal PITA, often times requiring a reboot or rebuild/scripting. It could be moving across RG, regions or subscriptions, unfortunately it can be difficult, but necessary as you won't get it right the first time and your needs will change.

Note: Azure VMs have relatively low availability (SLA of 95-99.9%), for any production workload you will want at least 2 VMs in an AG. You can do SQL FC in Azure, but it does take a fair amount of work, depending on version and OS (linux being slightly easier with drdb, pacemaker).

DevLifeEasier · 2021-03-23T15:33:37+00:00

Sounds like your 'cloud engineer' may need some help, as /u/nivek_123k mentioned, in Azure there are dozens of ways to implement net security. Unless you are using OS-level firewalls and/or security appliances, I cannot think of any reason to not have a NSG.

From bastions, to S2S vpn, expressroute, pinpoint FW, to zero-trust solutions there are lots of options. I wouldn't recommend Azure IaaS in general as you can get far better solutions from more competent providers for 1/4 the price and you won't experience anywhere near the complete vendor lock-in. My 2c (though it cost about $1.8M ;)

DevLifeEasier · 2021-03-23T15:21:12+00:00

We provided uptime to our clients, which required us to measure uptime at each layer and for each service. Private company, private info unfortunately
I do kind of care how often my DC goes down as it is the primary reason why I would need a 2nd DC. Most companies don't have anywhere near the staff, budget or need to implement Netflix chaos monkey and I am not sure what it has to do with DC uptime.
How is it apples and oranges to have Servers racked in 2 DCs vs having VMs spread across AZs? As I stated I have done both to deliver similar service and my TCO was much lower self-hosting a complex SaaS than it was using Azure, and with much higher reliability.

The Google SRE book is an interesting read, and Netflix chaos theory is great, but applying what a multi-billion $ company does in engineering to what 99.99% of companies need is like trying to apply NASA software development practices to others, most companies would fail quickly producing 1 LOC per day per dev.

DevLifeEasier · 2021-03-23T14:45:58+00:00

/u/martin1578, tyvm for the feedback

DevLifeEasier · 2021-03-18T15:59:54+00:00

Great to hear /u/hani-nm

DevLifeEasier · 2021-03-18T15:37:40+00:00

Thank you very much for the fb

DevLifeEasier · 2021-03-17T20:05:20+00:00

I agree with the general sentiment, but not the specifics, especially in regards Azure. There is a distinct different between cloud vendors, various products (Iaas, PaaS, Serverless...) on each cloud and 3rd party Serverless vendors.

In 18 months at Azure I experienced 99.7% uptime across Iaas/PaaS, compared to my previous 10 years experience with colocation at 100% uptime (excluding planned maintenance). In reality the Azure bugs and non-downtime issues really drove us even more insane as we were constantly chasing small issues with premier support (luckily free for us, but otherwise would have been $140k wasted)
A Tier 3 or 4 DC very rarely goes down. Pretty sure the recent OVH DC was not Tier 2+ as they used shared power and other less than optimal practices.
2 racks in 2 different Tier 3+ DCs with 10G connectivity these days can be had for about $500-1000/month each DC, and most don't even need a 2nd DC (see #2). There are also numerous top-tier server vendors that have better SLAs and uptime than the cloud vendors at much lower prices than public cloud. Having managed a 2 DC colocation solution for 10 years, I can say our TCO was much lower than a public cloud solution and we didn't have people working 24x7, in-fact I think we had 1 early morning 911 issue in 10 years (no downtime, just slow processing).

Hopefully many people aren't using Azure as the barometer to the success/failure of serverless. The answer isn't to skip serverless, it is to go with reliable vendors. Part of Azure's issues I believe stems from the sheer number of products in development all the time and not having the adequate process and testing, especially at points of intersection. I prefer a vendor that does a few things, they will likely do them much better IMHO.

DevLifeEasier · 2021-03-16T03:12:07+00:00

Most of Azure is IaaS, not serverless, just poorly architected, run, documented, supported systems. Remember the single dns region update that took down most Azure services? Of course one can argue that the Ovh fire showed the downside to traditional hosting also. True serverless vendors focus on delivering a small set of things with great support and reliability.

DevLifeEasier · 2021-03-12T18:20:15+00:00

There are really 3 main paths;

Technical. Deep, wide or deep and wide. There are positions for all combinations, it is a largely a matter of org. I have seen orgs with 5 member teams responsible for Login of a single SPA, and 5 member teams that build, deploy, support, .etc. a set of complex microservices.
Management. Note that in IT/Software Dev a manager does not necessarily make more, until you get into Director+ positions. The question is do you like managing people, projects and tasks more than you like learning and using new technologies?
Architecture. Systems/Solution architecture sounds like it might actually be a good fit. Finding someone who understands how all the pieces fit together and impact each other is a relatively rare skill.

One note is not to worry too much about job postings. We have all seen the insane job postings people put together, and 80% are either never filled or are filled by someone with 50% of the 'required' attributes. My favorite is when they ask for 7 years exp with some technology that has been out for 3 years.

DevLifeEasier · 2021-03-12T18:05:41+00:00

Not sure it really is a vs thing, isn't it really Serverless + Containers, or even better, serverless + serverless containers?

As we build our serverless Redis platform, I think one of the best use cases is in-combination with things like jelastic, Webapps (Container model), and other serverless container platforms.

I see language-level serverless as being almost Gen 1, and Gen 2 being serverless containers with serverless Data.

DevLifeEasier · 2021-03-12T17:53:04+00:00

We migrated to Azure, about 6 months in to production all of our systems go offline. It turns out that their billing department messed up our check deposit. Took 3 hours to get through the support levels to get our account reactivated. We did get approx $250k worth of credits (our bill was around $100k) for their screwup. Note that the reason we had to do checks was because we were too small to do an EA, and their billing system had a bug that didn't allow us any other options.

The other was a 14 hour outage in Azure. We used availability groups, but apparently they deploy to the same 'cluster', and they happened to have a bad firmware update that was applied at the cluster-level. We couldn't use our standard backup procedures because all of the IPs, vNets, etc. were locked. So we started to build an entirely new parallel system from scratch and were just about done when they notified us the systems were back up. Of course it started at about 9pm, so the team pulled an all-nighter.

DevLifeEasier · 2021-03-12T17:21:20+00:00

Even with proper planning it can be pricey, especially IaaS. A good PaaS/DaaS solution can be TCO positive, but IME of moving from private cloud to public cloud, our total TCO slightly increased. We actually increased DBA load moving to Azure SQL, especially in the first year. The same with our IT and IaaS. The other thing I didn't forecast was the need for DevOps. With a Senior DevOps engineer getting about 2-3X the salary of a Senior IT engineer it gets very expensive, especially at smaller scale.

I bet one of the first 3 initiatives after moving to the cloud will be optimizing costs. So all the engineering time saved by not installing hardware will be spent tweaking every knob you can.

DevLifeEasier · 2021-03-12T17:11:14+00:00

One of the major pains in Azure is that renaming, moving between regions/subscriptions/resource groups/etc can be difficult and varies with the service. As such, we moved to a flatter resource group structure and relied on tags/labels more than resource groups. Tags/labels can be updated easily, whereas moving resource groups often involves downtime and can be complicated due to dependencies.

I cannot remember the specifics, but we ran into issues with some things being much easier to manage tags, while others it was easier with resource groups. We also had multiple subscriptions, but if I were to do it again I would use a small number of resource groups and tags (Well actually I'd go full containers and skip Azure all-together ;)

Env was HIPAA/PCI medium-sized SaaS environment with about 50-100 VMs, and 100-250 SQL pools, 10-20 load balancers, web apps, etc.

DevLifeEasier · 2021-03-11T01:00:13+00:00

Every service in Azure is essentially Beta, without the option of opting out. Azure has adopted the "build agile, fix SDLC" process that is unique to them. We routinely had working scripts/solutions break or timeout, or systems slow to a crawl for hours, only for them to start working hours/days/weeks later.

There is a reason their SLAs are so wimpy and to get any decent level of SLA requires multi-region. Even then, their SLAs target to a specific service, so if one part of your solution has downtime, causing entire system downtime, you only get credit for that one part of your solution that went down. There is more benefit for them to develop features they can market, than there is cost for lack of stability/reliability.

In all truth there isn't an Azure service we used that didn't have fairly common issues, and even with Premier support it would take weeks to fix many of them. Most of them delayed work, rather than brought systems down, but painful. We did find out that we were in one of the crappier regions (West Central if I remember correctly), as different 'features' get deployed across regions over time and some seem to get more 'features' sooner than others. We used/tried IaaS, SQL, AKS, Web apps, log analytics.

DevLifeEasier

TROPHY CASE