HACKER Q&A
📣 chirau

Integrating Fortran technical analysis stage in Python trading pipeline


So I have been building a few trading bots for a while. Not super complex, but I get the gist of it. All of them have been in Python. I'd ingest both streaming and historical stock data from like an Alpaca or Polygon and do some transformations with pandas and some very basic technical analysis with Python libraries like ta and bta-lib then push trade signals with some combination of AWS lambda and/or TradingView and so on. Up until now, I have managed to do this almost completely in Python (maybe some JS here and there, but nothing significant).

Now, recently, I have been thinking about partnering with a guy who is absolutely great at data science and analysis. He is an older well seasoned guy who only operates in Fortran as his background was engineering. We are now trying to see how we can work with each other. He has worked with both large data sets and streamed sensors data. However, he has no knowledge of internet protocols or these fancy APIs or how to ultimately implement his analysis into actual order execution. We are in the process of exploring if and how we can work together.

So as we are currently thinking through it. I would be setting up the ingest pipeline and the storage infrastructure. He would be doing the technical analysis. I don't know if Fortran has some good http libraries for actual trade signaling and order execution, so I am guessing I would be taking care of that part as well. Also, I haven't worked with compiled languages in my pipelines.

So my ask is advice on how to set this up.

How can I have the Fortran technical analysis part communicate with both the ingest (especially streaming data) and the signaling and trade execution parts?

What sort of things and libraries should I be looking out for?

Would introducing microservices into my otherwise monolothic architecture help?

Or does pub-sub work better? Doing pub-sub with Fortran, is that a thing?

I couldn't find any documentation on working Fortran with AWS lambda, or exposing endpoints with Fortran, so does this mean I would have to introduce another intermediary store for generated signals and then work with those with Python?

Honestly, any criticism or advice would be greatly appreciated. I know I could try to learn Fortran, but I am trying to avoid that if possible. It takes a really long time to know the ins and outs of a language.

So even just a high level description like "ingest with this, store like that, expose with this, analyze with that, push with this, execute with that" would help greatly. I really want to understand how to work with different languages, especially compiled, at different parts of a pipeline. So I am guessing this question can also be applied to C++, GoLang or Java.

Thank you for taking time to read this.

  👤 ReleaseCandidat Accepted Answer ✓
I would just call Fortran directly from Python. That's also what's normally done by people that use Fortran and Python. There is F2py (that's part of Numpy) https://numpy.org/doc/stable/f2py/ To interchange data with Fortran you use files, mostly CSV (Fortran isn't exactly the nicest language to write parsers in). It's actually not encouraging that your post is the fourth google result of `fortran pubsub` :D

Disclaimer: I haven't used Fortran (with or without Python ;) for quite some years now, so maybe there are libraries now to do the stuff you're trying to do.

Btw: Fortran-Wiki: https://fortranwiki.org/fortran/show/HomePage

👤 mindcrime
I'm not sure how it works with Lambda as far as using compiled code, so I'm not super confident about this, but... at first blush the obvious thing to do would seem to be "call the Fortran code from Python directly" (as @ReleaseCandidat also says). There are mechanisms to do that... my only question is where or not Lambdas support that. If not, you might have to deploy a network callable service on an EC2 instance or whatever, and call into it from your Lambda or something.

Edit:

Reading this[1] it sounds like it is possible in principle to deploy Python bundles on Lambda that call compiled libs. It might take some experimenting to work out the details, but I think you could probably make this approach work.

[1]: https://aws.amazon.com/premiumsupport/knowledge-center/lambd...

👤 kregasaurusrex
Personally I've been looking for an API that includes the option greeks and doesn't cost hundreds of dollars per month. A good starting point I'd recommend is the $30/mo plan from https://twelvedata.com/pricing where you can pull down indicators that are pregenerated for a specified timeframe, and can move order execution to a faster language/platform. My background is in python which is good for EDA but there's likely to be tools made for more mature languages to interact with brokerages, and perform said signal analysis/TA with a higher throughput as well. Been busy with leetcode & interview prep so I haven't put in as much time into side projects as I could be, feel free to send an email.
👤 whinvik
There's a lot of Fortran work being done recently including a standard library and package manager. Maybe you could find something there, or just ask in their mailing lists.

https://github.com/fortran-lang