programming thread

Humans in space suits make monkeys nervous

Postby grass tacks » Mon Sep 20, 2021 12:05 pm

yeah i still haven’t figured out how the keys leaked but the server they’re on was set up 4 years ago and the version of amazon linux it’s running is obsolete and some stuff won’t update anymore, i’m guessing there’s an exploit in some service i don’t know about because nobody has accessed it through ssh.
grass tacks
 
Posts: 1172
Joined: Thu Dec 17, 2009 2:51 pm

Postby Names Bond James Bon » Mon Sep 20, 2021 12:06 pm

i should check we might not even have a cloudwatch alarm on # of emails sent, although I think there's a hard cap you have to continuously raise that starts small like 5k a day
User avatar

Names Bond James Bon
 
Posts: 2278
Joined: Mon Dec 14, 2009 2:48 pm

Postby Names Bond James Bon » Mon Sep 20, 2021 12:06 pm

the less ops I take ownership of the better, though
User avatar

Names Bond James Bon
 
Posts: 2278
Joined: Mon Dec 14, 2009 2:48 pm

Postby Names Bond James Bon » Mon Sep 20, 2021 12:07 pm

i mean let's face it if it was really our root account that got compromised and they did nothing then that screams 'false positive.'
User avatar

Names Bond James Bon
 
Posts: 2278
Joined: Mon Dec 14, 2009 2:48 pm

Postby wakeman » Mon Sep 20, 2021 12:08 pm

grass tacks wrote:yeah i still haven’t figured out how the keys leaked but the server they’re on was set up 4 years ago and the version of amazon linux it’s running is obsolete and some stuff won’t update anymore, i’m guessing there’s an exploit in some service i don’t know about because nobody has accessed it through ssh.


same boat here, I found the key they used was directly in the source code of one of the legacy apps, but not clear how they would have found it.

spending today switching all our random old dumb apps to use their own specific IAM keys with super narrow permissions. love to audit this old ass garbage. 🙈
User avatar

wakeman
 
Posts: 2358
Joined: Mon Dec 14, 2009 5:35 pm

Postby Names Bond James Bon » Mon Sep 20, 2021 12:09 pm

Was it in source control? I thought GH would warn if it thought you had keys in your repo? maybe I'm imagining that
User avatar

Names Bond James Bon
 
Posts: 2278
Joined: Mon Dec 14, 2009 2:48 pm

Postby grass tacks » Mon Sep 20, 2021 12:10 pm

for me they got access to the root account on wednesday and checked our sending capacity and verified email addresses but didn’t do anything else with it until they started sending emails friday night. so i guess it is possible. seems like there’s a lot cooler stuff you could do with an aws account than send $1 worth of emails it’s all confusing
grass tacks
 
Posts: 1172
Joined: Thu Dec 17, 2009 2:51 pm

Postby mynamerocks » Mon Sep 20, 2021 12:10 pm

Joker wrote:Was it in source control? I thought GH would warn if it thought you had keys in your repo? maybe I'm imagining that


github enterprise doesn’t, I imagine similar for other on prem ones
User avatar

mynamerocks
 
Posts: 2322
Joined: Wed May 26, 2010 4:09 am

Postby wakeman » Mon Sep 20, 2021 12:13 pm

we use Bitbucket, which yeah doesn't check.
User avatar

wakeman
 
Posts: 2358
Joined: Mon Dec 14, 2009 5:35 pm

Postby mynamerocks » Mon Sep 20, 2021 12:13 pm

Would rep https://github.com/Skyscanner/cfripper for security compliance of stacks, which can include roles then used by apps
User avatar

mynamerocks
 
Posts: 2322
Joined: Wed May 26, 2010 4:09 am

Postby Names Bond James Bon » Mon Sep 20, 2021 12:14 pm

grass tacks wrote:for me they got access to the root account on wednesday and checked our sending capacity and verified email addresses but didn’t do anything else with it until they started sending emails friday night. so i guess it is possible. seems like there’s a lot cooler stuff you could do with an aws account than send $1 worth of emails it’s all confusing


ah, interesting

yeah although without a pem you can't dig into the instances and I don't think you can backtrack any of the container shit either
User avatar

Names Bond James Bon
 
Posts: 2278
Joined: Mon Dec 14, 2009 2:48 pm

Postby mynamerocks » Mon Sep 20, 2021 12:27 pm

Joker wrote:
grass tacks wrote:for me they got access to the root account on wednesday and checked our sending capacity and verified email addresses but didn’t do anything else with it until they started sending emails friday night. so i guess it is possible. seems like there’s a lot cooler stuff you could do with an aws account than send $1 worth of emails it’s all confusing


ah, interesting

yeah although without a pem you can't dig into the instances and I don't think you can backtrack any of the container shit either


Yeah. I’ve always pushed Session Manager and not having sshd exposed, but it does mean that if a role with the right privileges gets exposed then you’re just as open
User avatar

mynamerocks
 
Posts: 2322
Joined: Wed May 26, 2010 4:09 am

Postby wakeman » Tue Sep 21, 2021 1:24 pm

christ we got hit again, SES disabled. looks like the key was not actually found in the legacy app I assumed but another app I assumed was more robust and protected. No fucking idea how they might have gotten the key/secret. In the apache log, someone tried to go to ourdomain.com/.env but obvs that didn't work
User avatar

wakeman
 
Posts: 2358
Joined: Mon Dec 14, 2009 5:35 pm

Postby grass tacks » Tue Sep 21, 2021 1:44 pm

found the source of mine and it’s embarrassing. turned out the .git folder was being served publicly and the token to access the private repo was in the git config file somehow, and the credentials were in the repo. knew i was being super lazy with the credentials but not blocking the git folder is crazy, i should go to programming jail
grass tacks
 
Posts: 1172
Joined: Thu Dec 17, 2009 2:51 pm

Postby wakeman » Tue Sep 21, 2021 1:51 pm

fucccc, that's scary shit man
User avatar

wakeman
 
Posts: 2358
Joined: Mon Dec 14, 2009 5:35 pm

Postby Names Bond James Bon » Tue Sep 21, 2021 2:33 pm

wakeman wrote:someone tried to go to ourdomain.com/.env but obvs that didn't work


If this is just an http log (sounds like it), I'd say it's a red herring.
User avatar

Names Bond James Bon
 
Posts: 2278
Joined: Mon Dec 14, 2009 2:48 pm

Postby jca » Tue Sep 21, 2021 4:37 pm

.
Last edited by jca on Wed Sep 22, 2021 12:09 am, edited 1 time in total.
User avatar

jca
 
Posts: 1289
Joined: Tue Aug 27, 2013 2:22 pm

Postby dmitry » Tue Sep 21, 2021 6:24 pm

mynamerocks wrote:how much would an average hipinioner expect to earn as a senior SW eng/almost principal level? trying to work out whether I'm being shafted and need to apply elsewhere again.


https://www.levels.fyi
Image
User avatar

dmitry
 
Posts: 9292
Joined: Thu Dec 31, 2009 1:14 pm

Postby something sensible if » Wed Sep 22, 2021 11:13 am

i went out on good terms at work like some kind of greek god ascending to the heavens after being shot through the heart in battle. nice to know i could possibly ask for my old shitty job back that i hated.

god based on levels.fyi i had a fucking average salary lol. Thought I was working myself to death for some good reason
User avatar

something sensible if
 
Posts: 3440
Joined: Mon Mar 29, 2010 12:20 am

Postby Autarch » Wed Sep 29, 2021 9:55 am

the bugs that make me mad are the ones that only occur because a user hasn't updated their browser since like 2016. Like, who are these people and shouldn't their browser just be updating automatically?
User avatar

Autarch
 
Posts: 19185
Joined: Sat Dec 17, 2016 12:13 am
Location: In these times of economy

Postby hazel » Wed Sep 29, 2021 10:16 am

that usually means their computer is from 2006 ime
User avatar

hazel
 
Posts: 3113
Joined: Thu Feb 11, 2016 11:54 pm

Postby Autarch » Wed Sep 29, 2021 10:27 am

they're on windows 7, which supports the latest version of chrome.
User avatar

Autarch
 
Posts: 19185
Joined: Sat Dec 17, 2016 12:13 am
Location: In these times of economy

Postby jack » Wed Sep 29, 2021 10:41 am

dmitry wrote:
mynamerocks wrote:how much would an average hipinioner expect to earn as a senior SW eng/almost principal level? trying to work out whether I'm being shafted and need to apply elsewhere again.


https://www.levels.fyi


wow my salary is decent but i'm not getting these huge bonuses that i guess everyone else is
User avatar

jack
 
Posts: 7308
Joined: Sat Feb 23, 2013 12:54 pm

Previous

Return to The Hot Zone

Who is online

Users browsing this forum: 31GeeWhizz, ANU, badhat, Birds vs Worms, Eyeball Kid, GonzO))), internethandle, jalapeño ranch, launchpad, Marza, mego, murderhorn, No Good Advice, outpost, Reliable Tradesman, ripersnifle, shred igniter, Slay Da Pink Fleshy Beast, smelts, trampoline, worrywort and 63 guests