Announcement

**dZeus** · 2 October 2011, 01:56

Originally posted by MultimediaMan View Post

Lately, I've been messing around with iSCSI quite a bit both as a storage technology and as a replacement for conventional disks in Workstations and Application Servers; I'd been using it for years as a way to keep user profiles out of the root of C:\Documents and Settings or C:\Users by using a GPO and a couple of logon/logoff scripts; the iSCSI mount being necessary for users who needed access to their .PST files (which are not supported/ recommended to be run over an SMB/SMB2 network share).

About a year ago, I started booting Operating Systems from iSCSI using iSCSI-enabled NICs; The first iSCSI Adapters were actually hardware SCSI HBAs with a full TCP/IP Network stack built onto the board; the Operating System saw a SCSI adapter, nothing more. More modern iSCSI NICs can support both iSCSI and Network Duties. These newer NICs are technically a type of CNA (Converged Network Adapter) which allows the adapter to do double duty: operating as both a storage pathway and as a regular network adapter. These new iSCSI adapters were leveraging an ACPI call to something called the iBFT; the iSCSI Boot File Table: a memory space in which could be written by a boot kernel, and subsequently read by a full Operating System kernel.

While relatively trouble-free it took a while for OSes to catch up. CentOS/RedHat/SuSE/Ubuntu Linux have had full support for this method of booting and installation for some time, Windows platforms have not had direct parity with Linux when it comes to Booting from iSCSI.

Windows Server 2003 R2 SP2 was the first Microsoft OS to be certified for an iBFT iSCSI boot...but it was imperfect: it supported booting just fine... it did not support installation of the OS via iSCSI. WinXP could be made to boot from iSCSI, but it was not supported (But it worked).

About 2006, etherboot.org (Marty Conner and Michael Brown) started looking at the possibility of leveraging iSCSI via a PXE-Booted Kernel... thus was born gPXE.

From 2007 to 2009, gPXE progressed to the point where it was being used by mainstream vendors to replace conventional PXE Code: vendors like Emulex, HP and LSI Logic were actively developing solutions using gPXE. The gPXE project caught the attention of H. Peter Anvin (lead developer of Syslinux) and PXE binaries were compiled to leverage gPXE.

In the Spring of 2010 a codefork was precipitated because of personality conflicts between the principal developers. gPXE essentially has languished for over a year with no real changes, while the code fork iPXE has had ever more features and contributions some of which are turning heads in quite a few disciplines - particularly as it relates to virtualization.

Which brings us to my project...

The Stateless, "Soft" Desktop computer. I deliberately picked a low power system; the next iteration of it will probably be a fire-breathing gaming system of some type, just for laughs.

The Goal: Install, Configure and Run a Workstation from a single network adapter with no local storage devices of any kind (No USB, Optical Media or other temporary storage).

This obviously cannot be done without a fair amount of infrastructure planning and setup ahead of time.

Minimum Requirements:

a. A Network.
b. A DHCP Server supporting PXE.
c. A TFTP Server.
d. A Webserver (or a Broadband Internet connection)
e. An iSCSI Target.

Optional: A Linux Compile Environment.

Subsequent posts will detail the buildout...

Nice!

booting an OS through gPXE/iPXI (with no local storage) is on my long list of stuff that I want to try. Originally I looked at iSCSI because I wanted to share a networked DVD writer that was also doubling as dvd reader for my network media player (it does transparent deCSS through a fuse module called dvdreadfs). The iSCSI target I use, http://scst.sourceforge.net/SCST, seems to be very capable. However it cannot split scsi commands for cdrom targets, so burning to an USB dvd writer was out of the question.

Is SCST the iscsi target software you use? Do you create your target partitions from a local file on the server, or do you share the entire drive with scsi pass-through?

Please keep us updated on this nice project!

**MultimediaMan** · 2 October 2011, 09:48

Infrastructure:

Surprisingly, the network bandwidth required is not really that high with a couple of desktops; the real trouble with iSCSI is TCP/IP; Transmit headers and other TCP Traffic can play havoc with throughput. Most people automatically recommend Jumbo Packets to alleviate this... and while this can be an important setting, it ignores the fact that other types of communications traffic will still be present: Jumbo Frames is an all or nothing proposition. A better technique is simply creating smaller broadcast domain: if you are going to run iSCSI for more than about three or four clients simultaneously, consider private VLANs before Jumbo Frames and Gigabit; Jumbo frames is one of the things you can enable, but at the end of the day, it's all about broadcasts.

My Home network has a Gigabit backbone; backed up with Low to Mid-Level HP Switching Technology: ProCurve 1800-24G. I had the ability to to create an iSCSI VLAN with all the trimmings, but chose not to (I'm still doing packet traces to evaluate it... I know I SHOULD...I want to understand when it becomes a NEED.)

PXE Booting: A brief tutorial. You need a DHCP server which can accept Options 43, 60, 66 and 67; Option 17 is not a bad thing to have either. There are many tutorials on the web about how to setup dhcpd or dnsmasq for network booting: Here; let me Google that for you... I wrote a pretty comprehensive tutorial on how to do it in Windows a few years back, in another forum: that document is here.

Webserver: You don't actually NEED a local webserver, but it helps speed things up. My OS of choice for this project was CentOS 5.7; it natively supports network-based installations. The delivery of the Netboot ISO to the PC is the secret sauce.

iSCSI Target: I used Microsoft's iSCSI Target as part of Windows Storage Server 2008 R2. It allows the presentation of VHDs as iSCSI LUNs. Unfortunately, this is not available to the public: it is an OEM or MSDN SA Download only. That said, if I were going setup a SOHO iSCSI Target, I would take a hard look at FreeNAS.

**MultimediaMan** · 2 October 2011, 10:13

iPXE

The real magic in all of this is iPXE.

The boot sequence looks something like this...

PXE Request -> PXE Boot -> iPXE Bootloader -> Make iSCSI Disk Connection/iBFT Passthrough -> Boot to OS via iSCSI.

However, our installation process requires something a little different...

PXE Request -> PXE Boot -> iPXE Bootloader -> Make iSCSI Disk Connection/iBFT Passthrough -> Make a non-persistent ISO mount via HTTP using iPXE -> Boot to ISO.

Believe it or not, you can run PHP scripts from a Webserver to automate doing all of this...both gPXE and iPXE have a fairly robust scripting language which can be used to pass arguments to a webserver for a response. Of the two, iPXE is far more flexible because of the options around "Hooking" an iSCSI connection vs. Booting an iSCSI connection.

Deeper discussion of the minutiae of Internal Scripting and Backend Scripting can be found Here, Here, Here, and Here.

**MultimediaMan** · 2 October 2011, 12:49

Hardware Prep...

OK, on to the good stuff...

I didn't have a donor system I was comfortable with using for this project. I did some digging and found my old VIA VT310-DP Mini-ITX motherboard languishing in a cupboard: then I remembered why I didn't want to use it... it had no PXE ROM for the onboard VT6122 Gigabit Adapter...or did it? More correctly...Could it?

Backing up a little, I know all of us here at MURC have had nothing but good experiences with VIA Products over the years... ;P But in all fairness...I had not a whit of trouble with my VT310-DP in the 6 years I have owned it... In fact, it sat in the same configuration from the time I first set it up, until the time I retired it in 2010. I wasn't asking too much of it: it was running Windows Server 2003 R2 as an application server. I own three VIA-branded motherboards and have had no problems with them... but I have never owned, nor do I desire to own, an AMD System with a VIA South Bridge...

Long out of warranty...I figured some re-purposing was in order. I made a backup copy of the System BIOS and used CBROM215.exe to peek into the BIOS...

I saw there was ample room withing the 4Mb BIOS image to inject another PXE ROM... finding that PXE Rom was a bit of a problem, but I eventually located it in the driver archives over at VIA's Website; they even had a pre-compiled LOM image...

BIOS ROM Injection proceeded uneventfully. I was taking a (small) chance here on bricking my system: sometimes things don't work out too well when you casually drop another ROM into the BIOS, but I'd done this sort of thing before and knew what to expect. Backing out of a Half-Broken BIOS can be an interesting excercise... read about one of the most difficult-to-solve BIOS problems here.

I recently had a bad reaction with another manufacturers motherboard with a similar chipset, so I was steeling myself for a fight... which pleasantly, never happened...

With the warranty a distant memory, and all hope of future support a flaming ruin, I proceeded...

**MultimediaMan** · 2 October 2011, 16:17

The Testbed:

The VIA VT310-DP is a great little motherboard...if you liked low power in 2005, and like even less in 2011 (Speaking in terms of Relative CPU Power) this is definitely the ticket. The only thing usual about the VT310-DP was this was VIA most serious attempt to date at an embedded server: it has dual 1GHz Eden processors (Centaur Cores), and runs on top of a CN400 Chipset. It has onboard Unichrome4/PM800 Graphics, the VT8237 Raid Chip which supports Two SATA1 connections and a Single IDE Port. It incorporates a VT6103L 10/100 LOM (PXE enabled), a VT6122 GbE LOM and an Intel 82551QM 10/100 LOM... the latter two onboard components lack any PXE Features. Also included is the VT1616 AC'97 audio controller. The VT310-DP supports a total of 2GB of DDR1 400MHz RAM.

System Configuration: I modded the BIOS to include PXE Support for the GbE LOM; in the BIOS Setup screen I disabled the onboard SATA and IDE Connections, as well as the VT6103L, the VT1616 Audio controller as well as COM2. I put a M-Audio Revolution 7.1 (A VIA VT1724 "Envy" Chip) into the lone PCI Expansion slot.. In Windows, this card is absolutely fabulous when it comes to playback clarity and driver support. I am pleased to say this level of support continues into Linux.

Speaking of Linux Support... I when I tried to install CentOS 6 on this board, it failed... and not for the reason you may expect... the Kernel for CentOS 6 is compiled with certain CPU features which the ageing VIA Eden processors could not support. No matter: the feature set of CentOS 5.7 is very complete and well-developed. My only foible with CentOS 5.7 is that Google Chrome is somewhat broken at the moment... It has dependencies which are not in the 5.7 Kernel; I'll get around to redoing the RPMs to fix that problem in due course, but be warned with CentOS 5.7 i386: Google Chrome is broken.

Here is a picture of the wee beastie on the bench...

In the next installment, I'll talk a little about iSCSI Targets...

**Umfriend** · 2 October 2011, 22:15

I have no clue what this is about nut keep it coming, it's interesting.

VJ · 2 October 2011, 23:21

It is very interesting. It is about booting a diskless computer, with its bootdisk on an iSCSI target...

**MultimediaMan** · 3 October 2011, 00:29

iSCSI Targets...

For my Testbed, I will be using the Microsoft iSCSI Target. It is a simple applet which can create VHD file and subsequently present them as iSCSI Targets. Up to 256 iSCSI Targets are supported Each iSCSI Target can consist of 256 LUNs...oh yeah... unless you're running microscopic drives, you're gonna run out of Physical Disk Space long before you run out of LUNs or Targets.

The Microsoft iSCSI Client makes use of *.VHDs = virtual hard disks, which are files residing on an NTFS partition which contain the disk images. This allows a certain amount of flexibility when managing disks: running snapshots and outright duplicates can be made of VHD files... potentially greatly simplifying recovery and auditing.

Some weaknesses need to be pointed out here: Even though you have excellent control over the disk images... they are files on a remote server after all; you still need to think about system backups and file level restoration. It is not enough to simply back up the VHD files themselves... because if the server that supports them fails, you have to re-present Disk images as iSCSI targets. The images are simply flat files, managing them requires some degree of finesse when it comes to access control. Likewise, there is no native encryption in iSCSI, so any Enterprise implementation may require dedicated Storage VLANs and strict routing (akin to storage zoning) at the switch level to prevent data from being read remotely. iSCSI is cheap; that is the chief advantage of it... but don't expect something approaching FibreChannel performance and security without planning on spending money on Managed Layer 3 switches. iSCSI can be made pretty secure even without encryption... but you need to spend money to achieve that and have a really good network management plan.

**Chucky Cheese** · 3 October 2011, 12:53

Quick question

I don't want to interupt your chain of thought but my question is, what is the iSCSI applet in the administrator folder in control panel for? If you were going to tie that in later just ignore my question and I'll just keep on reading...this is interesting stuff for sure!

**MultimediaMan** · 3 October 2011, 22:13

iSCSI Initiators

The iSCSI Targets are now explained... we need to talk about the client side: iSCSI clients are termed Initiators: they simply provide the connection definition and filter driver for the operating system to access the volume remotely. iSCSI, being a Layer 3 protocol, runs on top of an already-present TCP/IP Stack. Put briefly, iSCSI is simply a TCP/IP wrapper for FibreChannel commands, which are in turn a subset of SCSI commands.

Putting the two together seems like a match made in heaven...but it can be pure bloody hell to get working. Particularly when dealing with multiple initiators presented the same LUN as part of a cluster. In a cluster, it is usually no problem because the clustering software prevents issues using file locking and various quorum methods. When presenting any block level storage to Initiators which are not cluster-aware (or are unaware of the cluster they may be in) corruption issues are almost certain to happen.

Back to the Initiator: The purpose of the Initiator is to login to the Target and access its LUN(s); LUNs can be queried for, but most do not allow browsing of LUNs for obvious reasons. iSCSI even supports CHAP (Initiator to Target) and Mutual CHAP (Initiator to Target/ Target to Initiatior) for authentication, though most implementations use passive authentication such as examining IQNs, IP or MAC addresses at the Target end to control access. Security at the remote storage level is a still-evolving technology.

Speaking of which: IQNs... iSCSI Qualified Names...are generated automatically by your Host Operating system, but can be modified easily. A decent explanation of IQNs and their function can be found Here: After staring at them for a while, they become second nature.

For this Testbed, all of the Initiators I am talking about are Software Initiators: they do not rely on any kind of hardware assistance to get the connection made. They DO use a little more CPU, but they are a considerably more flexible than hardware initiators. Since the advent of PCIe, there are precious few Client-Side iSCSI HBAs being produced today.

The next installment will go into setting up the Initiator and Target.

**dZeus** · 4 October 2011, 02:56

just wondering... can you enable bitlocker on the system volume when booting windows from an iscsi target?

I'll look into mutual chap which needs to be supported by the iPXE/gPXE part to prevent unauthorized users from messing up someone else's windows iscsi target. Maybe I need to restrict a single initiator per target too, if SCST supports that.

My target will run on Debian linux (freenas looks very nice, but doesn't run on ARM). I'm looking into using my router's dnsmasq DHCP/DNS server as TFTP server. I also prefer the chainloading mechanism of iPXE iscsi boot as I'm not too happy to mess with my main workstation's BIOS atm.

I'll first try booting Ubuntu from an iscsi target. Booting the main Windows installation will come after an upgrade to gigabit lan in the future.

**Jammrock** · 4 October 2011, 07:05

Originally posted by dZeus View Post

just wondering... can you enable bitlocker on the system volume when booting windows from an iscsi target?

If the host system has a TPM chip it should be possible. There are ways to boot a bitlocker system using a USB drive when a TPM is not present too.

**MultimediaMan** · 6 October 2011, 17:59

Bitlocker...

Yes, you can... Bitlocker doesn't necessarily Encrypt everything. But administering it can be difficult without a TPM...and a CA, and AD/LDAP.

**MultimediaMan** · 6 October 2011, 18:28

iPXE iSCSI Initiator and WebServer prep...

Now it was time to build an iPXE image. You have to build it in Linux... for this I built a Virtual Machine in VMware Workstation. Use your favorite distro: install gcc, mtools, syslinux and git... clone the git tree:

Code:

git clone git://git.ipxe.org/ipxe.git

cd into ipxe/src and make:

Once the core binary is built, you can customize it by embedding instructions into the binary...or calling a script. I chose to embed code into my boot image:

I had to build two customized images...

One for the setup and another for the day-to-day boot...I figured this way was simpler, because of the method I control PXE on my network. You don't need to do this..I will show another method later to do this from a boot script held on a webpage.

Setup Code:

Code:

#!ipxe
set keep-san 1
set initiator-iqn iqn.2003-12.org.centos:proteus-iscsi.olympus.bv.ar.us.local
dhcp net0
sanhook --drive 0x80 iscsi:atlas.olympus.bv.ar.us.local::::iqn.1991-05.microsoft:atlas-proteus-0-target
sanboot --drive 0x81 --no-describe http://epimetheus.olympus.bv.ar.us.local/centos/5.7/isos/i386/CentOS-5.7-i386-netinstall.iso

General Boot Code:

Code:

#!ipxe
set keep-san 1
set initiator-iqn iqn.2003-12.org.centos:proteus-iscsi.olympus.bv.ar.us.local
dhcp net0
sanboot --drive 0x80 iscsi:atlas.olympus.bv.ar.us.local::::iqn.1991-05.microsoft:atlas-proteus-0-target

Finally, I mirrored the CentOS distribution tree to my Local Webserver (Epimetheus)...

And so, all that is left to do is boot into setup...

Announcement

Stateless Desktop Computing...

Stateless Desktop Computing...

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment