IT stories and rants

[UtwigMU]

Keep up the fight. Also have you done any freelancing?

This year I am close to what I was earning from freelance accounts other than the one I'm working for now full time before I employed here. I decided to develop my business to the level where I can survive financially without my current employment.

[Dilitante1]

I used to freelance, but I got burned so many times I care not to do it. 1 car dealership owes me 8+ hours that I'll never see. Another dealer blamed me for hosing all his pirated software on the company laptop and tried to pressure me into finding keys to get it all working again after I cleaned the malware out. Then the fun of saying you need to take the PC with you to scan and repair it all, that's when they freak out as they realize it isn't a quick fix and they'll have to do something else til it's fixed.

[UtwigMU]

Don't you have collectors in USA? Here if they don't pay you you send a reminder and then file court claim (can be done online, costs ~25€). Then if they don't file dispute court deducts money from their bank account and transfers it to your bank account as soon as any income arrives. It is also major blemish for their credit rating if they have a block on account. On top of that they pay court costs (25€) and bank charges them banking costs for transfer (15-25€).

So far I only had to send reminders and companies always paid me. With people you can craiglist their comp if they don't pay you.

[Dr Mordrid]

We have Judge Judy

Collectors aren't all that effective, and taking someone to Small Claims Court means at least half a day that would likely net more income elsewhere.

Kobayashi Maru

[Umfriend]

Don't you have collectors in USA? Here if they don't pay you you send a reminder and then file court claim (can be done online, costs ~25€). Then if they don't file dispute court deducts money from their bank account and transfers it to your bank account as soon as any income arrives. It is also major blemish for their credit rating if they have a block on account. On top of that they pay court costs (25€) and bank charges them banking costs for transfer (15-25€).

So far I only had to send reminders and companies always paid me. With people you can craiglist their comp if they don't pay you.

Wow. We have had two cases. It takes ages and quite a bit of money. Sure, if the court rules in your favor then you are entitled to claim expenses as well but those are prescribed and way lower than the actual expense in legal fees, cost and time.

[High_Jumbllama]

Hi everybody! Been years.
Adding a recent story to the mix:
Head of our IT Operations department comes to me and starts talking about transferring responsibility of some security alerts to me. Basically they involve potential infections of some sort. In principle, this is a good idea. I look at the alerts every morning as part of a dashboard and multiple times through the day. Oh, I work in a NOC. Ok I am am going to need some instructions on how the guy that does this now does this? His response was a more polite version of you can learn this as you go along. I told him that I need a clue as to what to do. Which alerts? His response, you can ask (we shall call him Bob) what to do. My response, how can I do that if you or he does not tell me first? His response, you need to learn as go along. You won't know everything else at first. Me: But you are not giving me any thing to go no. His Response was to tell me look at the third octet. These two values mean these are desktops, laptops, tablets, touchscreens, etc . . . . Ok. <still not liking this but this is half a clue.>

I see an alert the next morning and use nslookup. Now I do not like using this potentially the next days as of course DHCP and these machines do not have an assigned permanent addresses. I submit a ticket for a machine which unknown to me is for one of the Network infrastructure people. I get asked, why did you submit this ticket and I give a polite explanation of the above.

Response 1: Well you gave the wrong computer name. You need to look in the firewall logs.
Response 2: Why are you doing this? The initial triage belongs to infrastructure.

Well I do not have access to the firewall logs and ask your boss.

Thankfully my boos agreed and told them I will not take on this responsibility until they can ensure I have needed access and the process is written up.

[UtwigMU]

So I'm talking to CEO the other day. We have outdated CRM so I proposed we migrate to big provider. If we are selling having good CRM is not expensive.

He says to me: The new developer said he can create CRM himself. Let's make our own CRM. My response, yeah lets close our business and become CRM provider.

My report to COO: After we're done making our own CRM let's also make our own operating system. And then put it all in block chain. Next time CEO goes to South America he can bring back some powder so we will be able to work 36h / day.

Core business and sales are OK but there are more and more crazy ideas and people. Main issue is I'm too busy with side projects to change job and I have a status which allows me flexibility.

[Umfriend]

I don't see how a self built CRM can be either better or cheaper and probably neither. Surely, if you have your own developer, you let zir develop something that exploits or augments whatever it is that makes this specific company worthwhile

[UtwigMU]

I don't see how a self built CRM can be either better or cheaper and probably neither. Surely, if you have your own developer, you let zir develop something that exploits or augments whatever it is that makes this specific company worthwhile

Yeah, that idea is completely retarded. Building something is possible but making it usable and then maintaining it takes an order of magnitude more resources. It's obvious to me and smart people in and out of company. But to boss and developer it's obvious that building your own CRM is "better" and "cheaper". And this is just example. I need to find new gig before the IoT project they've been building goes in production and I get blamed for it not working.

[UtwigMU]

Half of my duckwit users cannot log in office. I've provided everyone with link to office.com and username which is name.lastname@company and password. I'm doing migration to O365 from SBS since our Exchange is unsupported and we're on the verge of being unable to send mails. I did cutover because I didn't want to link AD to Azure and to half fill new mailboxes. Most of people refuse to install authenticator on their blody phones thinking management will track them or something. 3 people don't even have a smart phone. Had I linked Azure to AD people would need phones to log into computers.

I just used a cell phone bought for messaging with overseas customers and put around 10 authenticators on it for the boomers with 2 years until retirement. When they need 2FA to check their salary in bank account and pay bills it somehow works for them.

So on cutover date I give everyone credentials and link. About half of company made it through office login site, the rest waited 2 days until I sorted out their Outlook. More than half of our people (half with degrees) cannot make it past office login screen. They are unable to function in present day.

Saga continues. Since everyone always wanted moar bigger mailboxes, there are people with 30-60GB mailboxes. Of course Outlook it's not fscking indexing your blody 30GB PST. You had the good life with your always raise limit by 5GB mailbox, now wait for it to be indexed. When I tried opening 60GB exported PST on oldish but still powerful workstation it puked out of memory. I ordered 16GB of memory and upgraded old X58 to 24GB and it still cannot open the pst. So I'm copying user's emails from old outlook profile to psts split by years. Only a single user who runs linux at home uses any of Outlook features beyond mail and calendar and is able to copy tasks from pst to inbox (drag and drop).

Also why is some Ukrainian company unable to send us emails? Turns out their server is missing or has expired SSL cert, their DKIM and DMARC records are nonexistant. Of course they cannot send emails to half of the World.

[UtwigMU]

So the shit is about to hit the fan! After new year our renowned fscking 60-year old C-suite dinosaurs acting as IT architects came out with IoT solution. Since boards are made by some Italian company, they hired a developer. The HR guy worked in dot.net shop before so he thought the dot.net is the hot shit of development. He hired dot.net Windows guy to work on Linux boards. The dot net guy (the one who was going to build own CRM, CRM is still in same state) hacked some solution with 3 servers and hacked-merged openvpn certs on Italian boards so they talk to our servers. This is because management doesn't want to pay Italians to do what is needed or start new line with different board from actually competent company.

The initialization procedure for new board takes half an hour and 4 reboots. First using USB, then copying files, manually merging certs and then manually inserting sh#t in database and hacking text files. The fscking servers have no SSL certs, passwords are all word123 and for good measure they are in sharepoint (which I didn't implement because people have problems with authenticators and we have fileserver) one phishing click away from being exposed. I had a security pentest audit last year and everything apart from SBS which I now decomissioned and web page which no one in company owns was not critical. That IoT sh#t won't ever survive any security audit and they never tested how it functions with 2k machines connecting, what happens when they run out of IPs on OpenVPN (they don't even know whether they have /16 or /24 mask and room only for 255 machines maybe).

I'm now forced to initialize 10 boards per week. Took part of two days to automate all manual file copying and cert merging steps with bash scripts which the developer who is actually here to program couldn't do. I don't have problem doing this so long they pay me but I spend around half of time dealing with user and other issues and other half for projects and infra. One half will have to go.

More likely I will go. I'm already past the initial stage interviewing with two companies for pure Linux position. Last year I managed to make minimum monthly wage on the side freelancing and I have some savings. I knew that once that sh#t goes in "production" - there is no test environment - I will have to go so I was preparing. It's tiresome to be in war mode for 2 years with all covid stress seeping from other people, while burning all my paid leaves for freelance teaching and taking only 5 days off in whole year. Government gave us 400 EUR worth of vouchers to spend on vacation locally but I haven't used them yet because time is money. Up until now I thought I'll be able to transition in pure freelance but the stress from projects managed by people who don't know what they are doing is starting to fsck with my cashflow.

Just as I was writing this post, a recruiter messaged me I have final phase of interview for remote Linux position tomorrow.

[MultimediaMan]

So the shit is about to hit the fan!

I hear you. I've been working for an apparel manufacturing company in IT Infrastructure for most of the past year. Many of the production machines use proprietary software (all Windows-Based), and my Manager is chanting "Cloud, Cloud" even though these F#¢k!n8 machines software (and ALL in-house apps) rely on SMB/CIFS shares to get to their files (no provision for SSL/Secure pipelines or Web APIs... just the Domain/OS - any guesses as to what version OS?). Domain (Admin) Credentials for some services to even function (too many lines of code to change - Apparently, UserID's and Password embedded into Code, you see..).

They had a virtual infrastructure based on Hyper-V with iSCSI storage .... NOTHING was configured correctly. No functional MPIO, Thin Virtual Disks on Thin Provisioned LUNs (No idea of real storage consumption, no de-duplication). No isolated iSCSI VLAN/Broadcast Domain, no documentation. Cabling that dated from the early 1990's. Oh, and 10 year old servers. More than a year out of date on OS patching. The only thing they had was a semi-decent AV/AM solution. Hundreds of terminated users still active in Active directory. Hundreds of stale machine accounts in Active Directory. MFA not working in house between all endpoints and IoT devices (Scanguns) - timesync issues. Spent 9 months hammering that into some recognizable shape.

Results:
Built Ubuntu NTP time server, configured "Primary" Domain Controller for NTP and an NTDS source. Group Policies deployed to enforce NTDS Time Source from Domain. Linux NTP server advertised in DHCP server for IoT: not a single call on NTP related problems since.
Physical Hosts Firmware and OS Patched to current standards.
Physical Host OOB (iDRAC, iLO, IMMs) cabled, configured and patched.
SAN Storage Firmware Patched
SAN Management/iSCSI interfaces on separate VLANs
SAN reconfigured for performance and correct failure domain for all LUNs, no thin provisioning of LUNs.
Added Storage Trays for Performance.
Recabled SAN with correct cables (not all interfaces would connect at full wire speed)
Hyper-V hosts reconfigured for Hyper-V only and common NIC/ Local filesystem layout. Added RAM to hosts to maintain 100% of VMs running even with failed Host(s).
Hyper-V guest Virtual disks converted to GPT/Dynamic disks. Group Policy settings for scheduled disk optimization (4 discrete groups to avoid stressing SAN storage).
Standalone Hyper-V hosts decommissioned or re-provisioned and joined to Hyper-V clusters.
Configured Automated vCPU/Network/Storage Consumption (Volume Sizes, Throughput and IOPS) dashboard via Graphana.
WSUS configured and deployed; patching cycles defined and working.
Migrated all VMs off of obsolete Hypervisors/Storage Arrays/SANs.
Deleted over 300 old user accounts.
Deleted over 250 machine accounts.
"Demoted" hundreds of Administrative accounts.
Built an imaging PXE server.

And my manager has the temerity to say "we" could have done more; I did all of this ALONE most of the heavy stuff during scheduled outage Windows (1 day per week).

[UtwigMU]

This is really a lot of work and looks like you brought the infra to a state where it's functioning normally without disruptive failures. Are you alone or do you have any juniors/techies?

[MultimediaMan]

Alone.

Today the subject of backups came up: we have about 9TB of storage on one File Server (Windows OS)... we had a call to restore a file from backup. I went into the Veritas NetBackup Console and the file wasn't there. Fortunately, we were able to locate a current copy of the file in userdata. My erstwhile manager wondered aloud why the file wasn't back up. So I gave it to him straight: We have multiple 2+TB drives. The drive in question has 1.8TB on it... it has over 2.7 Million files and over 120,000 Directories. How is NetBackup going to copy ~9TB of data across a single 1GbE interface and insert each object into the NetBackup Database in the allotted 8 hours? It can't. We are having a
LONG meeting tomorrow. I need to bring many, many one syllable words and a box of non-toxic crayons to explain, I'm afraid.

[UtwigMU]

I use Synology with iSCSI and Windows image backup. I have implementing secondary backup in to do but there is other sh#t that appears more urgent to higher ups.
We only have around 2TB on the file server but I have 8 Windows servers and around 5 Linux ones.