Pfthb

At my job, to be able to view my paychecks, vacation hours and HR data on myself I need to log into a 3rd party website.

I'm by no means a security expert or expert programmer but I could tell (simply by trying) that I could continue to try incorrect passwords without being locked out. (brute force: viable)

After logging in I was forced to select 3 pre-determined security questions in the case of a password reset (out of a total of 8!) such as my first car's licence plate (never owned a car 3/7), my spouse's 2nd name (don't have a spouse 3/6), 2nd name of my first kid (don't have kids 3/5), birthdate, name of my highschool, favorite pet, favorite film or favorite piece of music.

Most these things you can simply get from my facebook, (which, I should note, has not been updated for years!) again showing a distinct lack of understanding in basic security practices.

I also get the feeling, from looking at the site through the developer tools they use incredibly outdated software

A JavaScript implementation of the RSA Data Security, Inc. MD5 Message
* Digest Algorithm, as defined in RFC 1321.
* Version 2.1 Copyright (C) Paul Johnston 1999 - 2002.

I reported this through my company but my superiors don't appear all that interested.

How would I go about:

A. Finding out if this site is really as insecure as I think it is?

B. if true: communicating this in an appropriate manner to the company itself
(preferably in an anonymous fashion)

To start with the easy bit: you do not have to put real information as the answers to the questions. Random strings work best if you are really paranoid and store them in a password manager just like a password.

The rest (no brute force protection, potentially outdated software) is a shame, but there is nothing that you can do, from a security perspective. I would raise the issue with HR/Payroll and ask them to investigate. If you are in Europe, then you can also talk to your DPO to suggest that their "Data Processor" has troubling account security practices that need to be investigated.

Otherwise, this is more of an internal office politics issue.

To me this says you haven't investigated enough to confidently say it's insecure. You haven't shown any particular direct exploit nor really dug into their system (in a non-hacking, poking around kind of way). Your superiors may not be interested because of the lacking direct evidence of this.

As an example, my work uses a 3rd party site for scheduling vacations, I found it allows me to "recover" my password by sending my exact password back in plain-text via email, that is a direct evidence of an issue that I can report. Likewise, a site used for some IT services (SIP Trunk) had an issue where I could change the ID's in the URL and (to my surprise) view other people's account info, again, another direct line of evidence. Right now, you just have suspicions, and not obvious ones.

As an aside, all security questions for resetting passwords are insecure, as they rely on common information. As others have suggested, you can put fake answers in here (that you can still remember) or entirely randomly generated strings. You can think of the security questions as a "String" to "String" security challenge in the most generic sense.

How would I go about:

A. Finding out if this site is really as insecure as I think it is?

You really cannot, unless you witness a concrete, exploitable problem. That's what infosec companies do, they try to actively break systems (with consent of the owners). They know many procedures and techniques to systematically take such an application appart, from simple things like using HTTP instead of HTTPS, insecure usage of cookies, XSS, SQL injection, but also other stuff like IDs being obviously simply counted up (which means you can just try them out sequentially), stuff being only checked on the browser (easily forged), and so on and so forth.

They also can combine blackbox- with whitebox-approaches (i.e., just looking from the outside, like any hacker/cracker would do, or actually studying the source code, entering the servers with provided accounts, and so on).

You could do all that yourself, but as you are asking, you obviously don't really know how. But the main problem is that if you did that, without first getting consent, you'd be at least in gray territory, if not outright breaking law.

B. if true: communicating this in an appropriate manner to the company itself (preferably in an anonymous fashion)

You cannot (anonymously, at least). If there is a dedicated CISO, he would be your first line of attack; but aside from that, especially in smaller firms, there is little you can do if your management just shrugs it off.

I guess there may be specific circumstances in which some government agency might be interested - for example if your company works for the government in an area which has strict security requirements; then obviously you might try to get an anonymous message to somewhere, but why would they care about your HR information...

But in general, for arbitrary companies, the saying "There's no fate but what we make for ourselves" would apply.

If I had to guess, I'd bet the service is also resilient to brute force attacks, by simple virtue of being under-provisioned to handle the load involved. Go too fast with the guesses, and the server will fall over, alerting admins to what's going on. Go too slow, and you're not guessing fast enough to expect success in a reasonable period of time.

Add this to the suggestion by others to use false information, and the service isn't vulnerable at all if you're careful... least, not in ways reported so far. Note I said, "If". It does seem like an unfortunate design that, as you alluded, doesn't build confidence in what else might be behind the scenes.

I wouldn't attempt to actually test anything further myself, but if this is a product that is used by many other companies, you might try to put a bug in the ear of an actual security researches, who will know how to test it in an ethical way and be better prepared to deal with any legal fallout. If you're the only technically-minded person in your area at work, you might want to also alert co-workers for how to use the service "safely".

I think you're right - it sounds like, probably, the site is potentially pretty insecure. Those are some red flags, and the whole thing sounds a bit smelly. MD5 is widely considered cryptographically insecure.

That being said, the question of "what do I do?" is one that plagues a lot of security-related scenarios. There's a cost and a benefit to any action you take, and the right choice of action depends heavily on your environment and the nuances of your individual scenario. What kind of industry do you work in? How valuable is the information you're storing on the system? Perhaps most importantly, how receptive is your company to changing security practices?

These are all the relevant questions for you to answer, and ones which none of us here on The Internet™ are able to help with. In every question of how to handle a situation involving insecure practices, knowing the subtleties of a corporate environment - individuals' work histories, the nature of their company connections, the level of risk of souring relationships - defines what to do next.

What I'm trying to get at is, it seems to me like none of us are really qualified to answer this for you. There's an infinite array of potential options, and the best course of action truly depends on your situation at a level of detail that's not really conveyable here.

Good luck!