IT stories and rants

[MultimediaMan]

Speaking of Cluster****s...

Working on the Domain Controller (Primary), noticed an odd message on SNMP from the iDRAC. The machine had 4 disks in the trays. Got a predictive failure, but the RAID array stayed Green. Odd. Looked a little deeper; 4-Disk RAID(f***ing) 0.

I really wasn't in the mood to rebuild a Domain Controller (we have three others); Clonezilla. One small problem was the size of the disks; going from a ~1TB array down to a 480GB array is problematic with Clonezilla (about 80GB of actual disk consumption) and UEFI disk partition positioning on the physical disk array.

Solution: Use Clonezilla in disk source and destination mode. The destination was a temporary HyperV VM with a 1.1TB Dynamic disk: Cloned the DC to the VM. Then shrank the Virtual Disk to below 480GB (mounted temporarily to do a disk check), built a new SSD-based RAID1 array on the physical DC, then cloned everything back from the VM. You avoid using Acronis/Altris/Ghost licensing, and even at 1GbE, it took less than three hours (but it felt like I could suck down about half a bottle of antacid while doing it). The DC took less than 5 minutes to sync when it came back online.

That's the good news.

Now the bad news: I checked the other Domain Controllers; there is another with an identical 4-Disk RAID(f***ing) 0 array.

But Wait, There's more: the Backup server boot array is a Single Disk RAID(f***ing) 0 array.

FML.

I know what I'm doing the next two weekends during outage windows.

****. (sorry, had to say it.)

[Umfriend]

Seriously? Who ever thought RAIDing-0 on a production environment was a sensible idea???

[dZeus]

What is single disk raid0? Is that when you want to use the raid controller (future disk expansion?) and it doesn't support JBOD ?

[MultimediaMan]

Irritation is what it is.

What is single disk raid0? Is that when you want to use the raid controller (future disk expansion?) and it doesn't support JBOD ?

The Dell PERC7 Series/x30 Series (HP G8-G9 series as well) uses an LSI SAS/SATA RAID controller; some of those SKUs can set the RAID card to non-RAID HBA (most Dells of this period do not support HBA mode, but HP's did). With the The RAID enabled "RAID-0" is what it defaults to when you only assign one disk to an array, if you go back and assign more it won't give you the option to change the RAID level: you need to start over. In fairness, the TUI/GUI for all of them is Arcane with scrolling menus not fully viewable at the default resolution, etc... (the default LSI UEFI utility is to the left of witchcraft to work with: IBM M3 and M4 series utilized the stock LSI UEFI GUI for setup, but I digress).

Whomever configured these machines didn't know all that. TL-DR will get you in the shitter right quick. And so it did in this case.

More Mayhem on the Backup Server:

Boot Array is single disk RAID-0 (needs to be fixed this coming weekend - SSDs+Trays on order).
Nearline RAID (In-Chassis Disk Storage Array) had a 12TB RAID-0 array (with almost no data in it). I just finished converting that array to RAID-5 (wanted RAID6, but the Write speed would go down dramatically).
The External RAID (SAS attached 60 Disk Storage Enclosure) was configured correctly with Windows Storage spaces, so I don't have redo everything.

Their Backup "solution" only adds to the problem. It can't keep up with the number of hosts/data we have.

SMH...

[VJ]

Slight digression... if I may... Is RAID-5 or 6 still ok with disks that are bigger than 4-8 TB?
I read that the typical "Non-recoverable Read Errors per Bits Read", which is in said to up up to around 1 per 10E15 (depending on disk) can make it impossible to rebuild the raid with such disk sizes... Just curious...

[MultimediaMan]

Slight digression... if I may... Is RAID-5 or 6 still ok with disks that are bigger than 4-8 TB?
I read that the typical "Non-recoverable Read Errors per Bits Read", which is in said to up up to around 1 per 10E15 (depending on disk) can make it impossible to rebuild the raid with such disk sizes... Just curious...

VJ, Indeed, you may! These disks are smaller; (eta) 2TB Each. You're right, though: RAID-5 error correction starts to fail past 4TB/Disk and/or ~18TB Volume sizes. RAID6 can go a LOT higher with double parity but the write costs are high.

This is why Software Defined Storage was created: To alleviate near line storage bottlenecks by creating replication groups and data locality groups to offset this (there are many caching algorithms). It's still Software RAID10 or RAID6 (or 60) at some point when the bits are actually committed to the virtual disk.

[ZokesPro]

I have an IT story of when I was a kid in high school.

When I was a kid, we used to all go to this one guys house on the same day with floppy disk racks of 20-50 diskettes and copy hacked games in his basement.

It took the whole entire weekend most times, it was geeks heaven.

This friend would spend his entire weeknights downloading from a phone line from a neighborhood bbs which I later found out that the guy running the bbs server actually lives across the street from me.

This friend had an arrangement with his dad so long as he downloaded him some porn, he could use the landline all he wanted.

So he downloaded cracked games (and porn) while creating music and ripping spice girls into mp3, which took longer to encode per mp3 than it took to actually listen to the song.

The guy running the bbs server was a kid in grade 5 who ran it with his dad, who was the one cracking the games for the server.

The kid setup and managed the bbs server and it was a 1:1 ratio, had to be original stuff, his dad financed the server and the legit games.

I had a Pentium 75 that my dad bought me, he had a dual Pentium pro, which was unheard of for decades later.

You were a on a whole new level back then, not geek level but "IT" level, just before hacker level.

Hackers really scared people more than anything back then, even more than AIDS.

Hackers were like satanic gods back then, creeping in your computer and drain you of your cash and ultimately ruin your entire financial life for ever.

They would blow up your computer and burn your house down connecting through your own your phone line using that evil electronic handshake voice of theirs, when you accidentally picked up the line.

These were evil people back then with mysterious super powers from the depths of electronic hell, well at least for the adults.

We kids loved to explore networks and such, connect all over the place, searching for free data to copy, not much more.

There were no firewalls back then, if there were, I never ran into one.

Anyways, his dad was in the military at the time and worked under my dad's command.

His dad told my dad to buy me my first computer.

Since then, the Pentium 75 was my 5th computer that my dad has bought me back in 1994.

[UtwigMU]

At 12 I had a ZX Spectrum and I was able to hack some games written in basic so I had more cash to purchase cool cars. In those days there were no anti-piracy laws and people sold game collections on audio cassettes on street stands. The covers were printed with dot-matrix printers. There was a computer mag that published walkthroughs and I remember for some game I draw a map around 1 meter in width by taping paper together. There was a student radio station that would broadcast game for Spectrum once a week and you could just record it to tape.

Then at 15 me and brother bought 386. It was AMD with 40MHz CPU, 4MB RAM and 120MB floppy and no soundblaster, just PC Speaker. We saved up around 2000 Deutschmarks - close to 2000 EUR in today's buying power. I had both the 5.25" and 3.5" floppies so I could share games with more people. We would meet at friends or bring and exchange floppies with games at school. I remember once I copied Settlers for a friend and spent several hours editing document with anti-piracy codes and drawing graphics in Paintbrush (predecessor of MS-paint). One friend would always bring my floppies back with bad sectors. Once I decided to reformat my hard drive to create doublespaced D drive for games so I could have more but still leave C non-doublespaced for better Windows performance and data integrity for my school assignments. I remember backing up all my data whole night on like 40 floppies. Then I became OS reinstall expert and everyone came to me for advice on how to reinstall. Also I had 100+ line autoexec.bat with 5 or 6 configs with variants of XMS EMMS, QEMM, caching turned on or off and one entry that would boot straight into Windows 3.1. Printing was also a nightmare. Most normal people would have an Epson Dot-Matrix but some parents had laser printers at work. I didn't have a printer so I would always save my school assignment in plain text, rtf, wordstar, Word 1.0 and Word 2.0 on a floppy. Once I was doing a history project with a lazy girl classmate who would not complete most of her part till almost deadline. Since her dad had a printer we were to merge and print assignment at her house. Of course she had earlier version of Windows (3.0) and Word (1.0) so when I brought my part in Word 2.0 it wouldn't open. Had to open as plain text and edit out the metadata and she hasn't done any of her work so we had to write her part together as well. I remember staying at her house till midnight, taking the last bus home.

Once we had school field days where we were supposed to learn languages on PCs. Room was filled with IBM XTs (8086) with 360kB floppies and green screens and there was a text based program where you had to memorize and answer new foreign words. On day 2 or 3 I reformatted some of my 1.2MB floppies in 360kB and brought some games that would work on XTs. We also discovered snake in qbasic and one of my classmates who now works for big IT companies abroad programmed in a boss key. When you would slap Enter during snake, the game would display start text from the language learning program. So we copied this to all PCs in the room and when some supervisor walked in we would all hit Enter and play innocent.

No one was as cool as Zokes Pro, I think during early 90s only one classmate had a Pentium which at the time was the holy grail. Me and most friends had 386, one still had 286 but with soundcard and he later upgraded, while some cool kids would have 486 SX or DX.

[dZeus]

wasn't the 66 MHz Pentium faster than the 75 MHz?

[UtwigMU]

Bus speed meant a lot. I remember even high end AMD 486 DX4 120 being faster or comparable to Pentium 75.

https://www.high-voltage.cz/2020/duel-486-133mhz-vs-pentium-75mhz/

Just check table if you can't understand Czech. In some synthetic benches 133 MHz 486 outperforms Pentium 75. 486 overclocked to 150MHz outperforms Pentium in almost all benches.

[ZokesPro]

wasn't the 66 MHz Pentium faster than the 75 MHz?

I actually owned a Pentium 66 a few days before i owned a Pentium 75.

We got it at a a big super store and it was rather expensive for it's measly specs.

My dad was reluctant to buy it because of the price but we didn't know much about clones back then so we bought it anyways only after asking if we could return it if we didn't like it and they said yes, no problem.

We took it home for a few days and it was an ok machine, nothing out of the ordinary but good first time boot, but not an overall fast machine, was also loaded with shareware and AOL software.

then one day my dad got some intel from his squadrons tech about a rather taboo subject at the time, computer clones.

he referred us to a store in downtown Montreal which was this old brick building that was basically an empty room with foldable tables chairs and big white boxes with specs and price printouts taped to them.

they had these boring beige looking demos machines on the table but they cost a lot less and had much better specs/hardware all around, came with and free speakers (that sounded like like amplified static goodness, all treble, no bass)

my gut told me to always ask if they could remove the case so I could look at the mobo and such and I quickly found out who the honest vendors were.

the ones with the good parts were always compliant, and that's where we bought our Pentium 75 and returned our Pentium 66.

i was the first kid in my neighborhood to have 2mb of video ram and could do 16it colors at the highest resolution my monitor could handle, which was epic at the time but no one knew to what scale cause 8 bit color was the norm, and people were happy with a few colors.(so happens it was before the time when the term epic was used on non epic events)

no one knew how much video ram impacted graphics at the time, but i chose this system not really knowing myself but more was better that much i knew and my later discovery would eventually lead to one of my best friends dad's hating me for a few years cause i was right and he was wrong about 8bit vs 16bit color. whatever you whiney old bastard your mad cause your stuck with 8 bit color.

so was it faster than a Pentium 66? who cares everything else was better plus we didn't benchmark anything per say, there was no software to benchmark with.

[Umfriend]

Bus speed meant a lot. I remember even high end AMD 486 DX4 120 being faster or comparable to Pentium 75.

https://www.high-voltage.cz/2020/duel-486-133mhz-vs-pentium-75mhz/

Just check table if you can't understand Czech. In some synthetic benches 133 MHz 486 outperforms Pentium 75. 486 overclocked to 150MHz outperforms Pentium in almost all benches.

Jup. IIRC, with the BX330 chipset, you could overclock the Celeron III(?) from 300MHz to 450MHz simply by changing the bus speed from 66MHz to 100MHz.

[UtwigMU]

You could overclock Celeron 300A (newer process) which had 4.5 multiplier and 66MHz FSB to 450MHz which was what Intel's top PII speed was at the time before PIII Katmai which went to 600 came out.

It was Intel 440BC chipset which was really good. You could run it at 66, 100 or even 133MHz.

Late boards such as Abit BX133 RAID had Socket 370 instead of SLOT1 and you could run Tualatin Celeron such as 1.1 at 1.466MHz. This was competitive with early P4s.

I had Pentium II 350 but I overclocked it to 466 by running bus at 133MHz. I had 512MB of Micron 133 CAS2 memory. G400 and Parhelia could run with overclocked AGP bus at around 87MHz but not all other cards. I was running NT 4.0 workstation for uni and freelance and Windows 98SE for gaming.

[MultimediaMan]

The Intel 440BX, 440GX, Intel 815 and the Broadcom/Serverworks Chipset were the workhorses of the late 1990's (Slot1, Slot2, Socket 370) all the way to about 2005 (Due to RDRAM debacle and teething problems with Pentium4-based Multiprocessor Xeons). Many manufactures had to get downright creative in getting Intel Server products out the door in the mid 2000's because of it. Most problems with the new Sockets and RAM were resolved by 2007-2008 but by that time AMD had made quite a dent in Chipzilla's datacenter footprint. AMD, true to form as a tragic hero, failed to continue the development cycle, and Intel got their "A" game back to come back with a vengeance...

[UtwigMU]

Do you think Intel will make a comeback again and get their stuff in order?