Design and optimization of 3D freeform waveguides with machine learning