You must log in or register to comment.
Dang I haven’t seen this meme format in a while.
Vibe coders: “I don’t get it.”
What do you mean I shouldn’t have plugged Claude into the prod database?
Test environment? Everyone’s got a test environment! Although, some folks are even lucky enough to get a whole separate environment to do production in.
Actual footage of me testing in prod:
My manager quotes this to me every time we demo a bunch of random untested stuff to our clients.
Every team has a QA environment. The fancy ones have a separate production environment.
Damn, beat me by 3 hours; that’s what I get for not reading the thread first
Only the lucky few have a test env separate to the production env.
My current job provided me with a test env that has half the resourse allocation of production, and it immediately runs out of RAM when you try to use it…
Better than the new devs here finding out that the half the tools in staging touch production, giving me 5 hours of impatient sales texts
I use a sample for tests, not the full set
Mine is like 3 years out of date to prod.
this. the test environments we have need to be built manually and they haven’t been rebuilt in ages.
apparently it was a multi month project to get it setup in the first place. nothing like trying to recreate a bug in dev and finding out there’s 1000 lines of different code in the sandbox (that’s also misbehaving but in new and fascinating ways)
test? just do it and let your uptime kuma alert tell you if something’s wrong.
that’s the test
Stay borked, my friends.
Production? That’s just our pool of thousands of testers who pay for the privilege!
The hero we deserve
As a sysadmin, nothing makes me want to bitchslap devs harder. It’s shocking how many don’t even ask if we have a dev enviro. (The answer is yes)
My team made the “mistake” that all new projects need to be built and approved on our dev clusters before we allow them to be deployed to the prod clusters. We’re very explicit that dev is ephemeral, and that everything should be as code, so everything heals itself…
Despite this, whenever we reprovision the dev env, we always get emails from several different people saying that we wiped their prod databases and such. It’s crazy.
That sounds like a fun frequent task to introduce!
Get your BoFH on!
We don’t usually do it for fun, but I’m sure my teammate who has petitioned deploying a “chaosmonkey”-thingy to our prod clusters would love it 😁
Your teammate has the right approach here. People will quickly learn to stop relying on the dev environment in production deployments when the environment becomes unreliable.
I really wish we could get to that level of maturity (cries in healthcare)
Not sure if we should take the approach of biting the bullet and forcing it on ourselves or slowly growing to it (under a hostile govt that sacked ~30% of the countries ICT team)
back when my team had proper coverage, we used to do chaosmonkey style tests for hosted health systems. We’d schedule an event with the hospital IT staff. Say… 1 hour on a specific day near midnight. The local IT staff would put out a notice that for the hour the system may be inaccessible. Then we’d start a test suite, as ungracefully as possible. We’d run a abort shutdown command on one server, see how RHCS would handle it. Bring it back up stable. Silo a database instance. See how that handled. Yank out the Red1 cable from the switch, see how the network balanced the new traffic over Red2/Blue. That kind of stuff. We caught a few problems that way.
The clients hated it, but it really made the system more bulletproof for when a new recruit ran a command in the wrong window and took down an unrelated resource.
I stood up a test, staging, and prod pair of web server/dbs, got the Drupal 8 site stable on Drupal 10, hand it over, and yeah. No one uses anything but prod.
Thanks for 3x the work, ya dicks.
Tested in the school of hard knocks.
Check for security vulnerabilities? But Internet hackers will do that for free.
I literally live between ops sentiment and this sentiment
I stand shit up as a test server and then threat model before cloning a production server to a separate vlan and lock it down proper, with the added Intel of a snapshot on how the test server got tested.
