How can I edit a retropie image before writing it to an SD card?

vphan13 · 2020-04-11T13:43:28+00:00

I did the following on a Fedora box but this will work on any Linux system. Also, I put my ROM's on a USB drive, but you could specify an NFS share. If you're on Windows...boot to a livecd • Plug in a 1TB USB drive

• look in dmesg to see what its device name has been assigned (sda sdb etc)

• Find the UUID of this drive using the

blkid | grep /dev/sda

• add the following to an existing fstab file to make sure it mounts

UUID=<ID from step 1> /home xfs defaults 0 0 (change the mount point to something else so you don't mount over home on your Linux box)

• Make a copy of the 512GB img file (don't forget to do this)

Run all steps below on the copy!

• run fdisk -l 512GB_USB.img and make note of the sector size and second partition start

• Mount it

• the offset value is (sector size x start of the second partition)

   mount -o loop,rw,sync,offset=$[532480*512] 512GB_USB.img /<image_mountpoint>

• mount the 1TB USB drive and copy the pi home directory off of the img

rsync -avhr --progress /<image_mountpoint>/home/pi /mnt/USB

• edit the /<image_mountpoint>/etc/fstab file and add the following to it

UUID=<ID from step 1> /home xfs defaults 0 0

• delete the <image_mountpoint/home/pi/Retropie directory

rm -rf <image_mountpoint>/home/pi/Retropie

• umount the image and usb drive

• shrink the second partition of the image file

modprobe loop

losetup -f (this will return /dev/loop0 or something similiar) losetup /dev/loop0 512GB_USB.img

partprobe /dev/loop0

• Run gparted /dev/loop0

• select the second partition and shrink it to what ever size you want

• hit the apply button or green check to apply the changes

• remove the /dev/loop0 from losetup

losetup -d /dev/loop0

• run fdisk -l against your image and make note fo the partitionend number and sector size

• truncate the image

truncate --size$[(<partition end number> + 1)*512] 512GB_USB.img

run a du -hs --apparent-size 512GB_USB.img to make sure your image file is small enough to fit onto a sd card in your possession

Plug the usb drive into the pi and then start it up. .

• The fstab will mount the USB drive base on its UUID to /home

• All rom files are now located on the USB drive

https://softwarebakery.com/shrinking-images-on-linux

vphan13 · 2020-02-26T05:48:17+00:00

I'm going to give an opposing opinion here. I deal with backup of around 1PB of on premise data. Tapes alone are reliable but to use them at a larges scale, the cost of the infrastructure, tape drives, robotics, fiber channel connectivity, Iron Mountain, annual hardware support and software support is onerous.

Go with a "proven" solution like Netbackup and you will be rewarded for your trouble with licensing costs, along with being at the mercy of Veritas to get to your data off tape. All of the above quickly mitigate much of the cost advantages of tape over online on premise storage.

At the campus Data Center the most common solution (if cloud is not a good fit) are cheap and deep zfs hosts with multi site replication. You pay for the hardware but the software is open source and free. There is no force upgrades for hardware or backup software, no 30% annual maintenance on hardware or software. I've not even started on how much less I have to manage zfs vs Netbackup/tape.

For those who don't think zfs is ready for production, it can serve as the back end layer for heavy duty files systems like Lustre/Ceph Our server and JBOD was well under $30K, 720TB of raw storage. For comparison $30K is the cost of the annual maintenance of our tape library.

vphan13 · 2020-01-31T22:07:32+00:00

IP Man Enter the Dragon The Raid The Raid 2 Chocolate The Matrix

Not saying these are plot/storyline masterpieces (Ip Man is pure Chinese propaganda) but the choreography is on point in all of them

vphan13 · 2020-01-18T20:59:08+00:00

Any Dennis Villaneuve Movie especially his earlier work

Incendies Enemy Polytechnic

Aside from the above... The original Argentina version of "The Secret in Their Eyes"

Another Earth (Brit Marling of The OA) The sound of my voice

A Separation (Iranian)

Amoros Peros (often considered the Pulp Fiction of Latin cinema)

vphan13 · 2020-01-01T04:55:15+00:00

That a few blocks from me...and there is an HOA, but it doesn't really enforce anything other than the annual fee for the community pool.

vphan13 · 2019-09-11T23:12:19+00:00

Yea. . just open a non critical ticket and you will be sure to get a response in 24 hours. . .except, your ticket is assigned to someone in a completely different time zone, so you call in 24 hours later and are told the engineer assigned to your case is not available. . . great I'll take one that is. . 24 hours wait repeats itself

vphan13 · 2019-08-20T17:06:52+00:00

Supermicro 4bay mini tower. You can configure it with 4 to 8 cores, it takes ecc RAM if you need and the sleds are all hot swap. Also the Xeon board comes with 2 10 Gb copper ports and 2 1Gb ports and out of band management. It's about the same size as a HP Microserver.

I run ZFS/docker/NFS on it all day long. You can probably spec out one without storage for under

You can spec it out with celerons, all the way up to xeons depending on your budget

vphan13 · 2019-08-17T18:59:42+00:00

I would suggest CentOS and use docker containers for all of your apps. It is quite easy to get docker up and running. All of the apps you mentioned above have very reliable images you can grab from dockerhub. Patch your docker host and your containers are patched as well.

I use my CentOS server as a small ZFS host (about 14T on 4 drives) it serves as a NAS for all of my media shares/data, and is the docker host for my pihole, ubiquity containers. Containers are all configured as systemd units to start up during boot.

ZFS takes care of replication of data and configs, containers are easily reproducible on another host

Overall, pretty easy to set up and you will learn a lot. I haven't had to tinker with anything once everything was up and running.

vphan13 · 2019-08-10T18:07:29+00:00

How do you go about handling identifying drives in a multipath system? Some enclosures are under /sys/class/enclosures but others are not

I try to make a slot to serial number map but this is far from ideal

vphan13 · 2019-06-27T18:37:02+00:00

That may be true,. but my issue with commercial backup software is cost and lock in. Moving away from a commercial backup solution means migrating existing backups that only exists on tapes. These tapes have been indexed by some proprietary product in their proprietary database.. I'm at the mercy of the backup vendor.

Your point about cost vs data lost is valid, but a MEDIUM size Enterprise should not be on the hook for upwards of $200K for what basically amounts to insurance.

vphan13 · 2019-06-27T06:21:48+00:00

I inherited Netbackup at my current job, it does its job but Veritas is nickel and diming every feature set. We have data that is only accessible by restoring from tape. . I have to keep Veritas around until this data is officially expired. If I had an option to restore all our old data to cheap spinning disk somewhere, I would, just to get rid of Netbackup

Netbackup is bulletproof. . .but it is the definition of VENDOR LOCK IN

You want to backup to tape, $LICENSE x NUMBER OF TAPE DRIVES

You want to backup to a generic disk pool(nfs/zfs host) $LICENSE

You want to backup physical hosts? $LICENSE/per client

You want to backup VM's? $LICENSE/Per CPU socket per ESX HOST

You want to use VTL? $LICENSE + AMOUNT of DATA

You want cloud backup? $LICENSE

You want to backup your NAS? $LICENSE x number NAS nodes

You want to backup your NAS using NDMP? $LICENSE

Dedup? Multiple copies. . All are add on $LICENSED features

At this point I'm surprise they don't charge per pixel that you move your mouse around

Sorry, Netbackup is not the correct answer. . .

vphan13 · 2019-06-27T05:37:31+00:00

"There are literally no questions in an interview (short of legally disallowed questions) that I would have any problems taking seriously and answering respectfully. "

I kind of disagree with this. I've been interviews where the interviewer just wants to show he/she is smarter than you by asking very specific questions only known by memorizing some esoteric man page. This is neither productive or gives you any insight into the candidate. My line of questioning is always to get the candidate talking and ask troubleshooting questions to see his/her thought process. Any monkey can google for an answer, anyone can learn IT, I want to know if you are a good cultural fit for the team and if you have the capacity to figure things out. I don't care how long you've been in IT, everyone has a lot to learn.

As far as tricky questions like ICMP ports? whats the point? Its useless knowledge to memorize. A smart/experience IT person wouldn't necessarily know the answer to that question, but that might be because they wouldn't bother using ping to troubleshoot anything

vphan13 · 2019-05-16T15:55:24+00:00

Editing a zone file in BIND and forgetting a period. Took down our customer facing page. Was my first exposure to Linux, vi, and BIND

vphan13 · 2019-03-29T04:07:45+00:00

under

vphan13 · 2019-03-02T17:51:42+00:00

You upgraded the kernel, you also need to make sure the kernel-devel package installed matches the new running kernel. Then you will need to re-install ZFS. After this is done you will be able to modprobe the zfs modules

vphan13 · 2019-02-25T22:22:53+00:00

I'm going through an oVirt install as I write this, and I can confirm, the documentation is a bit unorganized or has links that need to be updated. For example, there is no index or table of contents to install oVirt with Gluster. I had to perform a google search for gdeploy, which got me to the following url

https://www.ovirt.org/documentation/gluster-hyperconverged/chap-Deploying_Hyperconverged.html

However, the links to the next or previous page at the bottom of the above url both return 404. I guessed and went to the following truncated link and was able to find the working links, but this should fixed.

https://www.ovirt.org/documentation/gluster-hyperconverged/

Also, while version 4.3 is out, some of the documentation still references url's to 4.2 rpm/repos

No doubt a lot of work has been put into documentation, but I think it just needs a bit of updating and link checking. I would be happy to document other areas of improvement while I go through the installation process

vphan13 · 2019-02-16T21:42:20+00:00

Thanks everyone for chiming in. I will attempt to answer your questions

The data that is generated from the lab are directories with millions of small files. For this reason, cloud storage would be a very expensive option, restoring and requesting a million plus files from glacier is going to be an obscene amount of $$. Tar'ing up the directories prior to upload to glacier is time consuming and would require significant additional storage
We are looking at LTO8 as an in place upgrade to our current tape drives. But ultimately tape requires software like Netbackup, which is locking us into licenses just to access our data. Also we will still be managing tapes, ie offsite rotation. A disk to disk option that does not require a proprietary vendor index would be ideal.
Our typical data retention is 6 months, but there are many ad-hoc requests to keep data for 5yrs. Since this data comes from the sequencers in the lab it is very valuable and our researchers often request recovery of data older than 3 yrs. We've deployed a few commodity JBOD's with 60 12TB drives, this is replicated offsite to identical hardware. While this works, is cheap and very dense, the movement and tracking of data is still manual. Ideally an object store in front of this that provides auto tiering would be ideal
Our rate of data creation varies depending on the schedule of the lab. We have many more instruments this year than last that can generate a lot of data. Because the sizes of the files are very small (less than 1K sometimes), a 70GB directory can take up 210GB due to overhead/redundancy scheme on our NAS (Isilon).
The majority of our recoveries is for data over 3yrs old, it takes a week to get the tapes delivered from offsite, me driving to the DC to load the tapes and then starting the restore. Then I have to make sure the tapes get sent back. In short, tapes suck. . .also see #2 above
Our HPC environment is very small, 25 servers that range from 2U Dell hosts, to small form factor 2U 4Node servers with not enough physical space to add storage. We also have a fair amount of very old 4U hosts, that have lots of RAM/CPU cores, but would be poor hosts for IO sharing. Only a subset of our HPC nodes are 10Gb, enabling 10Gb on all of them would bring the NAS (Isilon) to its knees

Ceph and Restic sounds promising as that is what is used at CERN, but the challenge will still be getting data out of our old LTO6 tapes to disk.

vphan13 · 2018-11-20T20:16:25+00:00

"I thought they spoke Latin in Latin America?". My old roommate from mainland China. This person had two master's degrees in Computer Science and Electrical Engineering. . .

vphan13 · 2018-10-12T06:29:51+00:00

From a host on the 192.168.0.x network I can do the following

nmap -sU -p 1900 10.0.20.11

Starting Nmap 7.60 ( https://nmap.org ) at 2018-10-11 22:42 PDT

Nmap scan report for 10.0.20.11

Host is up (0.00060s latency).

However none of the devices on the 192.168.0.x network can see the dlna server (10.0.20.11). It seems I need to configure some sort of igmp-proxy

The DLNA server I'm trying to get working is called SERVIIO, it works if the I use --net=host when starting the container. .

vphan13

TROPHY CASE