Yes, maths and statistics courses in school, college, university would be the tried and tested route for learning the analytical and practical tools and techniques.

Forecasting rare events with any precision is almost a contradiction in terms though. When you’re down to the 1/10000 type events you need such a large dataset/sample, that there are almost always unobservable sub-populations, or unobservable historical / environmental factors that your data is likely to be missing; something important that could materially change the forecast if you were to have had complete, unbiassed data on it for you whole sample.

Practical forecasting though , i think, should be tied into the decision making, and trying to reduce the risk of choosing the course of action to take. The set of possible / feasible actions shapes the forecasting approach - you can’t really learn that pragmatic tradeoff in academic institutions - i think it’s just experience. Make some predictions, get them wrong, do a forensic analysis. Or collaborate with people who have done this for a living.

In respect of the AI, you need to check it had a reasonable concept of the population of events you want to know about. Understands its sample of observations, how that sample was drawn (i.e. it wasn’t random), and the biases in that sample or sampling method. Then it should be easy to recalculate its output, then you come up with some scenarios of the bias, or adjustments see what changes those scenarios have. A competent forecaster should have addressed major/obvious sources of bias, with ranges / scenarios etc. " how wrong might this forecast be if we assumed, X, Y, Z instead?" I don’t trust anyone who asserts they have a representative sample, it’s impossible to prove that 100% - otherwise you’d not need a sample in the first place.