You're absolutely right. All API's may be limited. But if you look again at what most if these open source libraries give you - TensorFlow, Pytorch, et .. look at what Google has done with Vertex.ai. Companies like H2o.ai, a lot of the complexity has already been abstracted for you. Even if there is stuff that you cannot do - you rarely need to go back to the drawing board for it - since someone in the Open source world would already have found a workaround. In fact most courses on Machine Learning now actually teach students how to use sklearn rather than the actual math behind it.