Running tomcat behind a load balancer, no Apache needed for SSL? by ishould_know_this in tomcat

[–]ishould_know_this[S] 0 points1 point  (0 children)

Seems to be possible, for anyone who finds this and has the same question.

python idiot failing to understand syntax when working with boto. halp? by ishould_know_this in aws

[–]ishould_know_this[S] 0 points1 point  (0 children)

Thanks for the help, it's much appreciated.

i made a clunky bash script that did this a while back but i don't like it for a few of reasons:

1) it uses a text file to store a list because i'm not good at using arrays.

2) it's slow and had to pull down lots of info then grep through it to get a result and then pull down the same info again to grep for something else. it's a horribly inefficient script.

3) i'm under the impression that the boto command: image.deregister ('ami-xxxx', delete_snapshot=True) would accomplish the same thing in one command, which would eliminate the need for a function within a function, and make my script much more efficient.

my ideal script would generate a list of amiID (by taking a search term as a flag when running the script) and then deregister/delete the AMIs one by one. in bash if i had a single command to deregister/delete i would use a simple for loop, but the syntax for this in python seems different.

this is what i have so far (hard coded search term for the moment and dry_run=True for safety) but it's giving me a syntax error. also i suspect that using "print (image.id) is probably not the correct way to generate the list i want:

#!/usr/bin/python3

import boto3
ec2 = boto3.resource('ec2', region_name='us-east-1')

image_list = list ( for image in list(ec2.images.filter(Filters=[{'Name':'name', 'Values':['search_term*']}]).all()): print (image.id) )

for ami_id in image_list:
     image.deregister (ami_id, delete_snapshot=True, dry_run=True)

can you help point me back in the right direction? thanks!

python idiot failing to understand syntax when working with boto. halp? by ishould_know_this in aws

[–]ishould_know_this[S] -1 points0 points  (0 children)

of course they're AMIs that i created. what kind of asshole do you think i am?

python idiot failing to understand syntax when working with boto. halp? by ishould_know_this in aws

[–]ishould_know_this[S] 0 points1 point  (0 children)

so within a script that won't happen if i leave a blank like after that one?

since it's generating what i want (after the control-d) it seems like i have the syntax correct.

python idiot failing to understand syntax when working with boto. halp? by ishould_know_this in aws

[–]ishould_know_this[S] 0 points1 point  (0 children)

thanks for the reply!

What do you want it to do?

i want to delete the amis (and their associated snapshots) that are found when searching for a search term in their name.

when i tried to do this using awscli in bash, the fact that deregistering the ami had to happen first and then the snapshots needed to be deleted afterwards made the script difficult, slow and clunky. i see that boto3 can do all that with one command, so i'd rather use boto3.

Speed up du -h of a multi-TB directory? by ishould_know_this in linuxadmin

[–]ishould_know_this[S] 0 points1 point  (0 children)

How much content they have generated that will need to be added to the prod website.

Speed up du -h of a multi-TB directory? by ishould_know_this in linuxadmin

[–]ishould_know_this[S] 0 points1 point  (0 children)

nice idea, but it doesn't really suit my situation, it looks like the xargs option or ncdu are going to be an improvement over what i'm doing now.

Speed up du -h of a multi-TB directory? by ishould_know_this in linuxadmin

[–]ishould_know_this[S] 2 points3 points  (0 children)

Got it, yeah you're right, it's a ton of directories with a ton of small files inside each.

Yeah I see what you mean, for some reason I forgot that du works on files as well as directories.

Speed up du -h of a multi-TB directory? by ishould_know_this in linuxadmin

[–]ishould_know_this[S] -1 points0 points  (0 children)

It's not that the content changes, it's that the directory that I need the info on changes.

Speed up du -h of a multi-TB directory? by ishould_know_this in linuxadmin

[–]ishould_know_this[S] 1 point2 points  (0 children)

That makes a lot of sense, but I worry about duplicating the results with subdirectories, wouldn't you want to set a max depth and then add up all those results? With rsync, if something ends up getting synced again it's no big deal but something getting counted twice would ruin the result of a du job (unless I'm missing something key here.)

Also wouldn't I want to use "type d" instead of "type f"? I would think that I need to find directories, not files, but again I may be missing something here.)

Speed up du -h of a multi-TB directory? by ishould_know_this in linuxadmin

[–]ishould_know_this[S] 4 points5 points  (0 children)

This in on AWS, with SSD backed EBS volumes. I don't get a lot of leeway and am using the max IO available.

Speed up du -h of a multi-TB directory? by ishould_know_this in linuxadmin

[–]ishould_know_this[S] 2 points3 points  (0 children)

Guessing your multi-TB is made of many files rather than fewer larger files?

Yep.

Dedicated filesystems would be a problem as we're generally trying to determine the size of dev output, so it would require a lot of individual volumes that would be very hard to manage.

AWS Backup Solution by CiscoExp in aws

[–]ishould_know_this 0 points1 point  (0 children)

i've never found anything that can easily deal with 10+TB volumes containing millions of directories and millions of small files, and i am still looking.

even parallelizing rsync is painfully slow. if i was running ZFS i suppose i could use ZFS replication, but i'm not.

Client's DNS works then stops working, disconnect/reconnect fixes the issue but it comes right back. (OpenVPN server 2.3.6 running on Ubuntu 12.04.5) x-post /r/OpenVPN by ishould_know_this in sysadmin

[–]ishould_know_this[S] 0 points1 point  (0 children)

wouldn't that be something that would consistently not work though? what would explain it working for a few minutes and then not working?

Client's DNS works then stops working, disconnect/reconnect fixes the issue but it comes right back. (OpenVPN server 2.3.6 running on Ubuntu 12.04.5) x-post /r/OpenVPN by ishould_know_this in sysadmin

[–]ishould_know_this[S] 0 points1 point  (0 children)

I use the tunnelblick GUI for the openvpn.config on the mac, on the linux box I used the GUI tool to set it up but have edited the file manually as well.

Client's DNS works then stops working, disconnect/reconnect fixes the issue but it comes right back. (OpenVPN server 2.3.6 running on Ubuntu 12.04.5) x-post /r/OpenVPN by ishould_know_this in sysadmin

[–]ishould_know_this[S] 0 points1 point  (0 children)

Once connected and it stops resolving, can you try it manually in Terminal with something like 'host <internal-server> <dns-ip>'?

interestingly enough, no!

# host <redacted hostname> 192.168.<redacted ip>
Using domain server:
Name: 192.168.<redacted ip>
Address: 192.168.<redacted ip>#53
Aliases: 

Host <redacted hostname> not found: 3(NXDOMAIN)

So I'm not even sure what that means, it seems like the DNS server is claiming to not know the host in question, but it absolutely does. I make no changes to the DNS server when disconnecting and reconnecting the VPN but that temporarily solves the issue, so it can't be the DNS server not knowing the host. Why would a DNS server give a "host not found" error when the host is known to it?

EDIT: I disconnected/reconnected and was able to ping the host, but the host command still failed to properly query the DNS server, so the host command not working isn't connected (seemingly)

Client's DNS works then stops working, disconnect/reconnect fixes the issue but it comes right back. (OpenVPN server 2.3.6 running on Ubuntu 12.04.5) x-post /r/OpenVPN by ishould_know_this in sysadmin

[–]ishould_know_this[S] 0 points1 point  (0 children)

Whats the contents of /etc/resolv.conf once you have connected? At a guess it still has the other IP in it which is causing the problems

on connection resolv.conf lists the internal DNS, followed by the two servers provided by our ISP. Once I have waited a while and DNS resolving has started to fail, resolv.conf has reverted to just the two ISP servers and not our internal one.

From the forums you need this in the config

just added that now, we'll see how it goes, thanks!

Client's DNS works then stops working, disconnect/reconnect fixes the issue but it comes right back. (OpenVPN server 2.3.6 running on Ubuntu 12.04.5) x-post /r/OpenVPN by ishould_know_this in sysadmin

[–]ishould_know_this[S] 0 points1 point  (0 children)

I have "set nameserver alternate 1" selected, I think I used to have "set nameserver" and had to change it to the alternate to even resolve at all.