Install a package without internet

So first of all, this isn't original.  Credit goes here.  But it's fantastic, and I wish I'd known about this a long time ago.  As usual, for my own memory/use: and actually, I'm just going to clean up what the other guy said.  He did a great job.

On the Internet-less computer:

In the terminal enter:

PACKAGENAME=<The name of the Package to install>

and then

apt-get -qqs install $PACKAGENAME | grep Inst | awk '{print $2}' | xargs apt-cache show | grep 'Filename: ' | awk '{print $2}' | while read filepath; do echo "wget \"http://archive.ubuntu.com/ubuntu/${filepath}\""; done >downloader.sh

A ready-to-use downloader for the package has now been created in the home folder.  Open your home directory in the file browser and move the file downloader.sh to the top-level directory of your flash drive.  Then eject your flash drive.

On the computer with Internet:

Insert your flash drive, and open your flash drive in the file browser.  Copy the location of your flash drive:

[CTRL]-L

[CTRL] C 

Move into the directory of the flash drive.  In a terminal this time, type:

cd [CTRL]+[SHIFT]+V 

Run the downloader:

bash ./downloader.sh

Wait for the download to complete and eject your flash drive.

Back to the Internet-less computer:

Open your flash drive in the file browser.  In the browser, type the following to copy the file location of the flash drive.

[CTRL]-L

[CTRL]-C

Move into the directory of the flash drive.  In a terminal this time, type:

cd [CTRL]+[SHIFT]+V 
sudo dpkg --install *.deb

That's it!

A Homemade Beowulf Cluster: Part 1, Hardware Assembly

A beowulf cluster lets me tie miscellaneous computers together and use their cpus like one large processor...I think.  Never done this before, still working on the details.

I'm building this with random laptops: generally i5s, and I think one's a Core Duo - it's half decent thing from 2011.  Might even throw in an RPi2 for good measure.

Make sure you read through this before starting.  You want to know what you're getting into.  Watch out, though - this is a long post.

Notes

Since we're working on multiple computers here, not everything is going to be cut-and-paste.  I will make sure that it's as clear as possible, however.  There won't be any hand-waving or assumptions of prior knowledge.

I'm doing this with Linux Mint 18 on my primary laptop.

Primary Sources

Setting up the cluster: src 1,  src 2, src 3

Hardware Ingredients

• Since the benefit of this tool is sharing computations between computers, you need a way to route that information.  Hence, an ethernet switch.  Go for a gigabit, since you don't want the switch to be your bottleneck.  I was cheap and got myself the 5-port version, and I'm already kicking myself.  Go for the 8-port version at least.  Here's the 5-port version I got, for consistency's sake. 

• A ton of ethernet cables.  You can do with short ones, but you'll want to connect the switch to your router so you can access the computers from outside their own tiny network.

• Leftover computers.  You won't be using these for anything else, so make sure you don't need them. 

Computer Preparation

Install Ubuntu Server to each computer. I used 16.04.1 LTS.  For those with a penchant for funny names (or who need to deal with annoying, obtuse colloquialisms on the web), that's the Xenial Xerus edition.

You'll need to keep track of computer names ("hostnames"), and install consistent users (so the username and password are the same on each machine).  I like to increment the computer names with letters.  For the purposes of this guide, we'll use:

hostname:    grendel-a
username:    beowulf
password:    hrunting

Create installation USB flash drive

Go here to download the 16.04.1 LTS server ISO.  That link initiates the download; here's a bit of context.

Burn the ISO to a USB flash drive. Note that you'll lose everything on the USB you use.  Run the following command twice, once before plugging in your USB and then a few seconds after.  The new entry when you run it the second time is your flash drive.

lsblk

For example, this is the output when I run that command with a flash drive plugged in.  Note on the right where it specifies the mount point (at /media/me/storage), and on the left where it shows me the name is sdb.

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 167.7G  0 disk 
├─sda1        8:1    0   487M  0 part /boot
└─sda5        8:5    0 167.2G  0 part 
  ├─mint--vg-root
  │         252:0    0 159.3G  0 lvm  /
  └─mint--vg-swap_1
            252:1    0   7.9G  0 lvm  [SWAP]
sdb           8:16   1   1.9G  0 disk 
└─sdb1        8:17   1   1.9M  0 part /media/me/storage

If your USB is mounted, it has to be unmounted first - else weird things can happen in the next step.  Trust me: once I didn't unmount a partition before copying it with dd, and my MBR was wiped out instead.  In the example above, unmounting would work like this:

umount "/media/me/storage"

If we pretend your USB is the sdb device, this is the command you'd run (I'm assuming the ISO was saved to the default "~/Downloads" location).  Swap out the 'sdb' part with what the lsblk command indicated.  Also be aware that if you mess this up, you will probably destroy whatever computer you're running the command with.  Just FYI.

sudo dd if=~/Downloads/ubuntu-16.04.1-server-amd64.iso of=/dev/sdb bs=4M status=progress

Install to each computer

You're gonna have to follow this procedure with each computer in the cluster.  There isn't a simple way around it, that I'm aware of.

Plug in the flash drive, and reboot/turn on the computer.  Make sure it's connected to the web.  Press ESC, F1, F2, F11 or F12 to choose a startup device...if those don't work, Google:

"how to choose startup device BIOS" & [your computer type]

e.g., "ThinkPad T420".

When it boots up, there's a few settings to make sure of.  Most of them should be straightforward: choosing a keyboard layout and a default language, for instance.  Just in case of problems, here's a full walkthrough.

All you want is a simple installation.

Don't bother with detecting the layout...if you're in the US or have an english keyboard, you can just stick with the defaults.  Worst case, start over.  There's a few more images after that one to do keyboard stuff, but you get the idea.

grendel-a, grendel-b, etc.  Makes it easy to keep the computers straight. After you've entered the hostname as above, press [enter].

I'm using the same thing for the full name and the username.  Makes it simple. Press [enter].

Since the username is the same as the full name, just press [enter].

That's the standard password for the computers.  I used the [down arrow] and pressed [space] to select that radio button and show the password.  Just press [enter].  It'll ask you to reenter the password for verification; do so, and press [enter] again.

Don't bother with encryption; this is a cluster designed for speed, and that just makes disk access slower.  For the next screen, check your time zone.  If it's wrong, the adjustment screen is pretty intuitive.

Press [enter] here to go with the default option.  You want to use the entire disk, and LVM could be useful (it's also an incredible pain).  Press [enter] again on the next screen after verifying the destination for the install.  It's probably the largest drive - the small one is probably your flash drive.

It asks for confirmation; [right arrow] and [enter].

I like to use a standard size for the installation that leaves lots of room to spare for other things on the drives.  Might be able to do some kind of shared network storage for the computations with the rest.  If it lets you, go for 50.0 GB.   [enter].

Another confirmation.  [right arrow] + [enter].

Unless you have weird internet, leave this blank and press [enter].  If you do have a weird internet setting, then I can't help you.

I skipped the auto updates, since they won't positively impact the performance of the cluster, and might slow it down at times.

Ok, this is the one that really matters.  Addendum: also select 'Manual Package Selection'.  For some machines with obscure drivers, this seems to ensure the drivers get installed.  You need the OpenSSH server in order to communicate with your cluster via the terminal. (at least, I'm pretty sure you do.  I didn't want to spend the time to find out for sure.)  [arrow down] to it, then [space] to select.  [enter] to move on.

Yes; you want GRUB installed to the MBR.  [enter].

...and that's it.  Reboot, remove the flash drive, and the new OS is installed.

Final Adjustment: power management

There is one more thing, though: you'll be running a bunch of these computers in the cluster, and you don't want to have to deal with them individually.  For one thing, that would be a huge mess.  So to keep them consolidated, you'll want to keep their lids shut while they're running: and if you don't change a setting, they'll just go to sleep when you close the lid.  src. 

sudo nano /etc/systemd/logind.conf

Find the line:

#HandleLidSwitch=suspend

and change it to:

HandleLidSwitch=ignore

Then restart that bit of the system:

sudo service systemd-logind restart

Assembling the Cluster Hardware

This is pretty straightforward: start with your ethernet switch.

1. Run a cable from your router to the #1 port on the switch.
2. Run a cable from each computer in the cluster to a port on the switch.  I'm not going to try and correlate port numbers and computer numbers.  Pretty sure it doesn't matter.

And that's all it takes to assemble the cluster hardware!  You should be able to ping your computers and get a response:

ping grendel-a

If you don't get replies, you have a problem.  Google is your friend.  Worked for me.

A Recurrent Neural Network to generate sentences: step-by-step assembly (Mint version)

This one predicts (ok, procedurally generates) text word-by-word.  Others do it letter-by-letter, but then you have to deal with misspellings.  src. Inspiration.

Dependencies

I'm basing this off an installation script referenced in the github readme (here).  Instead of blindly running the script, I checked its contents and discovered it fails when run on linux mint.  So I simplified it, and changed/added a thing or two since it wasn't working anyway.  In case anyone wants to blindly run my version, here it is all cleaned up.

Updating your sources is always a good idea prior to installing a bunch of software.  Also, the package below makes it easy to add new repositories (see below).

sudo apt-get update
sudo apt-get install -y python-software-properties

Now things are getting serious.  This stuff includes GCC 4.9, since it seems GCC 5 isn't compatible with Torch 7.  There was a weird error where cmake wasn't installed when I put all the packages in the same apt-get install command - I split out the packages to their own lines, ran my install script again, and it worked.  Very odd.  Hence, my inefficient code below.

sudo apt-get install -y build-essential 
sudo apt-get install -y gcc 
sudo apt-get install -y g++ 
sudo apt-get install -y curl 
sudo apt-get install -y cmake 
sudo apt-get install -y libreadline-dev 
sudo apt-get install -y git-core 
sudo apt-get install -y libqt4-core 
sudo apt-get install -y libqt4-gui 
sudo apt-get install -y libqt4-dev 
sudo apt-get install -y libjpeg-dev 
sudo apt-get install -y libpng-dev 
sudo apt-get install -y ncurses-dev 
sudo apt-get install -y imagemagick 
sudo apt-get install -y libzmq3-dev 
sudo apt-get install -y gfortran 
sudo apt-get install -y unzip 
sudo apt-get install -y gnuplot 
sudo apt-get install -y gnuplot-x11 
sudo apt-get install -y ipython
sudo apt-get install -y libpcre3-dev

These are for a program called OpenBLAS, which stands for Basic Linear Algebra Subprograms.  Here's a bit more on the concept.

sudo apt-get install -y libopenblas-dev liblapack-dev

Install Torch (finally).  Note that when running the ./install.sh command, I'm letting bash automatically enter "Y" when prompted for a user response.  It's requesting permission to add torch to the system path in the .bashrc: essentially, creating a system shortcut for each initialization of the bash terminal.

git clone https://github.com/torch/distro.git ~/torch --recursive
cd ~/torch; 
echo "Y" | ./install.sh 
source ~/.bashrc

These tools are installed with Lua, a programming language that came with Torch.

/home/$USER/torch/install/bin/luarocks install nngraph
/home/$USER/torch/install/bin/luarocks install nninit 
/home/$USER/torch/install/bin/luarocks install optim
/home/$USER/torch/install/bin/luarocks install nn
/home/$USER/torch/install/bin/luarocks install underscore.lua --from=http://marcusirven.s3.amazonaws.com/rocks/
sudo /home/$USER/torch/install/bin/luarocks install lrexlib-pcre PCRE_DIR=/lib/x86_64-linux-gnu/ PCRE_LIBDIR=/lib/x86_64-linux-gnu/ 

Make sure to add Torch to your PATH.  This ensures that when you run the command to use Torch, your computer can actually find the program.

to_path="/home/$USER/torch/install/bin"
echo "PATH=$PATH:$to_path" >> /home/$USER/.bashrc
source ~/.bashrc

With that, all dependencies should be satisfied. Time to install.

Installation

Clone the software from github.

mkdir ~/projects
cd projects
git clone https://github.com/larspars/word-rnn.git

That's actually all there is to it.  Now cd into the word-rnn directory to run the test stuff.  Before the tests and tools, though, there's a fix that you have to perform.

Extra Fixes

I'm running torch on a gpu-less computer.  There's a glitch that occurs when running the test script in that scenario.  To avert it, you have to change the name of a function from CudaTensor to Tensor.  See here for details.  

cd word-rnn/util
nano SharedDropout.lua

The third line should look like this:

SharedDropout_noise = torch.CudaTensor()

Change it to this:

SharedDropout_noise = torch.Tensor()

And save.

CTRL + O
CTRL + X

Running word-rnn

Now you can run the test function.  Be aware that on a 4-core 2.8GHz i7 processor, this command took 21 hours to complete.  BUT it's a very cool command, and the result is amazing.

th train.lua -gpuid -1

To be continued...

Running GUI programs over ssh command line

I've never run into this before, and it's really cool.

ssh -X pi@raspberrypi

This allows the remote computer to use the local computer's x server to run the gui for the remote program.  Sooo, I can use Mathematica on my Mint 18 laptop through an ssh connection to my RPi 2.  Very cool.

Setting up Raspberry Pi Zero in Headless Mode

Download the raspbian image, dd it to the sd card, unmount and remount it on the computer.

Figure out the  name of your micro sd card (I'm going to assume that it's sdb) - look for the device with a total size similar to what yours is labeled as.

lsblk

Download the image here.  Normally I'd give you a wget command, but it's a lot faster doing it on the browser.  Grab the "Raspbian Jessie with Pixel" version...I don't trust the Lite to have everything needed for future projects.

Extract the file (right-click and select either 'extract' or 'open with archive manager') and put it in the Downloads folder.  If you rename it to raspbian.img after extracting, then you can cut and paste the following command. Note that you should change 'foo' to the device name you identified above, e.g., of=/dev/sdb.  I'm keeping it as of=/dev/foo to prevent accidents.

sudo dd bs=4M if=/home/$USER/Downloads/raspbian.img of=/dev/foo status=progress

Remove the sd card from the computer and plug it back in.  That seems to allow the low-level processes to reassess the contents of the card after everything on it was changed.

Resize the partition in gparted if you're planning on doing anything that involves a significant amount of storage.  The Raspbian image doesn't leave a lot of free space, and your card probably has extra room on it.  Install gparted below, then run and resize (that's a GUI operation, so no walkthrough - sorry.  Also, I've been doing from nearly the beginning of my Linux career and don't need any reminders on how to do it).

sudo apt-get install gparted

Set up the networking - you'll need to tell the pi what the wifi password is (Source).  cd into the primary partition of the sd card - you'll find it as the folder named with a long string of meaningless text if you do:

ls /media/$USER

cd into that folder (represented by me here with x's and dashes) and then go several folders deeper to edit the wifi config file.  The whole thing is done below.

sudo nano /media/$USER/xxxx-xxxx-xxxxx/etc/wpa_supplicant/wpa_supplicant.conf

Add this to the end of the file, where "foo" is the name of your wifi network (literally, what shows up when you're choosing a network to connect to - it is case-sensitive) and "bar" is the wifi password. Both should be in quotation marks.

network={
  ssid="foo"
  psk="bar"
  proto=RSN
  key_mgmt=WPA-PSK
  pairwise=CCMP
  auth_alg=OPEN
}

Save and exit by pressing CTRL-O, ENTER and CTRL-X.

Unmount, stick the ssd in the pi, connect a wifi adapter and plug in the power.  Most of the time you don't even need to know the IP of the pi, as you can use the computer name to ssh in:

ssh pi@raspberrypi

Note the password is

raspberry

A few extra commands to be aware of:

Note that to ssh into a Mint 18 distro from the pi, you need to install openssh-server on Mint first.

sudo apt-get install openssh-server

If you know the user and computer names that you want to ssh into, you can use them instead of hard-to-keep-track-of IP addresses.  If everything is on the local network (LAN).

Copy files with scp:

scp user@computer:desired.file ~/path/to/containing/folder

And on an *ahem* completely unrelated note, found a few more dependencies that sphinxtrain needed in the install script.  I'll update vmc when I get a chance.